2024-09-21T09:37:41.139 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-21T09:37:41.165 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T09:37:41.267 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913663 branch: main description: upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} email: ceph-qa@ceph.com first-half-sequence: - cephadm.shell: env: - sha1 mon.a: - ceph config set mgr mgr/cephadm/daemon_cache_timeout 60 - ceph config set global log_to_journald false --force - echo wait for mgr daemons to upgrade - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - echo wait for minority of mons to upgrade - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --limit 1 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - sleep 60 - echo wait for majority of mons to upgrade - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --limit 1 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - sleep 60 - echo wait for all mons to upgrade - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - sleep 60 - echo wait for half of osds to upgrade - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 4 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done first-half-tasks: - workunit: branch: reef clients: client.0: - rbd/test_librbd.sh env: RBD_FEATURES: '61' - print: '**** done rbd/test_librbd.sh 7-workload' first_in_suite: false job_id: '7913663' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'librbd C and C++ api tests ' - desc: 'run randomized correctness test for rados operations generate write load with rados bench ' - desc: 'run basic cls tests for rbd ' - desc: 'run basic import/export cli tests for rbd ' - desc: 'librbd C and C++ api tests ' - desc: 'randomized correctness test for rados operations on a replicated pool, using only reads, writes, and deletes ' - desc: 'randomized correctness test for rados operations on a replicated pool with snapshot operations ' - desc: 'run randomized correctness test for rados operations generate write load with rados bench ' name: teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true create_rbd_pool: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - mons down - mon down - MON_DOWN - out of quorum - PG_AVAILABILITY sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: main sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 owner: scheduled_teuthology@teuthology priority: 850 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b second-half-sequence: sequential: - cephadm.shell: env: - sha1 mon.a: - sleep 60 - echo wait for upgrade to complete - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - echo upgrade complete - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 second-half-tasks: - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - print: '**** done end radosbench.yaml' seed: 5357 sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 sleep_before_teardown: 0 stress-tasks: - thrashosds: aggressive_pg_num_changes: false chance_force_recovery: 0 chance_pgnum_grow: 1 chance_pgpnum_fix: 1 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 chance_thrash_pg_upmap_items: 0 disable_objectstore_tool_tests: true timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - print: '**** done end radosbench.yaml' - workunit: branch: reef clients: client.0: - cls/test_cls_rbd.sh - print: '**** done cls/test_cls_rbd.sh 5-workload' - workunit: branch: reef clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - print: '**** done rbd/import_export.sh 5-workload' - workunit: branch: reef clients: client.0: - rbd/test_librbd.sh env: RBD_FEATURES: '61' - print: '**** done rbd/test_librbd.sh 7-workload' - full_sequential: - rados: clients: - client.0 objects: 500 op_weights: delete: 10 read: 45 write: 45 ops: 4000 write_append_excl: false - print: '**** done rados/readwrite 5-workload' - full_sequential: - rados: clients: - client.0 objects: 50 op_weights: delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 write_append_excl: false - print: '**** done rados/snaps-few-objects 5-workload' subset: 31/32 suite: upgrade suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 targets: smithi070.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBJoGuitcF4GjHdjUIPMwF7YkRx/6DmudqCtw80eyt4XZxuIk1qwtvibQaR0KWJ/Wk639H0gndp64glBRyLcJcc= smithi160.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHD2RoQkIE/Nsl7U0OsIFJGxKSTehQLFBfwT5MHKDYvnxFgzkoK7S1ZuVQ74EAhjNbBSN+G9myPF+Ni6YgoUBpM= tasks: - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install runc nvmetcli nvme-cli -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: branch: reef exclude_packages: - ceph-volume - cephadm: compiled_cephadm_branch: reef conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:reef - cephadm.shell: mon.a: - ceph fs volume create foo - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph.healthy: null - print: '**** upgrading first half of cluster, with stress ****' - parallel: - first-half-tasks - first-half-sequence - print: '**** done upgrading first half of cluster ****' - ceph.healthy: null - print: '**** applying stress + thrashing to mixed-version cluster ****' - parallel: - stress-tasks - ceph.healthy: null - print: '**** finishing upgrade ****' - parallel: - second-half-tasks - second-half-sequence - ceph.healthy: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-21_03:08:26 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-21T09:37:41.267 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa; will attempt to use it 2024-09-21T09:37:41.268 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks 2024-09-21T09:37:41.268 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-21T09:37:41.269 INFO:teuthology.task.internal:Checking packages... 2024-09-21T09:37:41.289 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '62da54b4b83a96ca9976b7e1d0c8272a564a2208' 2024-09-21T09:37:41.289 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:37:41.289 INFO:teuthology.packaging:ref: None 2024-09-21T09:37:41.289 INFO:teuthology.packaging:tag: None 2024-09-21T09:37:41.289 INFO:teuthology.packaging:branch: main 2024-09-21T09:37:41.290 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:37:41.290 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-09-21T09:37:41.498 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5092.g62da54b4 2024-09-21T09:37:41.499 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-21T09:37:41.505 INFO:teuthology.task.internal:no buildpackages task found 2024-09-21T09:37:41.505 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-21T09:37:41.530 INFO:teuthology.task.internal:Saving configuration 2024-09-21T09:37:41.552 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-21T09:37:41.559 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-21T09:37:41.580 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913663', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 09:32:35.711830', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBJoGuitcF4GjHdjUIPMwF7YkRx/6DmudqCtw80eyt4XZxuIk1qwtvibQaR0KWJ/Wk639H0gndp64glBRyLcJcc='} 2024-09-21T09:37:41.601 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi160.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913663', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 09:32:35.710510', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHD2RoQkIE/Nsl7U0OsIFJGxKSTehQLFBfwT5MHKDYvnxFgzkoK7S1ZuVQ74EAhjNbBSN+G9myPF+Ni6YgoUBpM='} 2024-09-21T09:37:41.602 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-21T09:37:41.607 INFO:teuthology.task.internal:roles: ubuntu@smithi070.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-09-21T09:37:41.607 INFO:teuthology.task.internal:roles: ubuntu@smithi160.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-09-21T09:37:41.607 INFO:teuthology.run_tasks:Running task console_log... 2024-09-21T09:37:41.686 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe6632f4160>, signals=[15]) 2024-09-21T09:37:41.687 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-21T09:37:41.693 INFO:teuthology.task.internal:Opening connections... 2024-09-21T09:37:41.693 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi070.front.sepia.ceph.com 2024-09-21T09:37:41.694 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:37:41.772 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi160.front.sepia.ceph.com 2024-09-21T09:37:41.773 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:37:41.845 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-21T09:37:41.851 DEBUG:teuthology.orchestra.run.smithi070:> uname -m 2024-09-21T09:37:41.869 INFO:teuthology.orchestra.run.smithi070.stdout:x86_64 2024-09-21T09:37:41.870 DEBUG:teuthology.orchestra.run.smithi070:> cat /etc/os-release 2024-09-21T09:37:41.927 INFO:teuthology.orchestra.run.smithi070.stdout:NAME="CentOS Stream" 2024-09-21T09:37:41.927 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION="9" 2024-09-21T09:37:41.927 INFO:teuthology.orchestra.run.smithi070.stdout:ID="centos" 2024-09-21T09:37:41.928 INFO:teuthology.orchestra.run.smithi070.stdout:ID_LIKE="rhel fedora" 2024-09-21T09:37:41.928 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION_ID="9" 2024-09-21T09:37:41.928 INFO:teuthology.orchestra.run.smithi070.stdout:PLATFORM_ID="platform:el9" 2024-09-21T09:37:41.928 INFO:teuthology.orchestra.run.smithi070.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T09:37:41.928 INFO:teuthology.orchestra.run.smithi070.stdout:ANSI_COLOR="0;31" 2024-09-21T09:37:41.928 INFO:teuthology.orchestra.run.smithi070.stdout:LOGO="fedora-logo-icon" 2024-09-21T09:37:41.928 INFO:teuthology.orchestra.run.smithi070.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T09:37:41.928 INFO:teuthology.orchestra.run.smithi070.stdout:HOME_URL="https://centos.org/" 2024-09-21T09:37:41.928 INFO:teuthology.orchestra.run.smithi070.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T09:37:41.928 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T09:37:41.928 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T09:37:41.929 INFO:teuthology.lock.ops:Updating smithi070.front.sepia.ceph.com on lock server 2024-09-21T09:37:41.968 DEBUG:teuthology.orchestra.run.smithi160:> uname -m 2024-09-21T09:37:41.986 INFO:teuthology.orchestra.run.smithi160.stdout:x86_64 2024-09-21T09:37:41.986 DEBUG:teuthology.orchestra.run.smithi160:> cat /etc/os-release 2024-09-21T09:37:42.045 INFO:teuthology.orchestra.run.smithi160.stdout:NAME="CentOS Stream" 2024-09-21T09:37:42.045 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION="9" 2024-09-21T09:37:42.045 INFO:teuthology.orchestra.run.smithi160.stdout:ID="centos" 2024-09-21T09:37:42.045 INFO:teuthology.orchestra.run.smithi160.stdout:ID_LIKE="rhel fedora" 2024-09-21T09:37:42.045 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION_ID="9" 2024-09-21T09:37:42.046 INFO:teuthology.orchestra.run.smithi160.stdout:PLATFORM_ID="platform:el9" 2024-09-21T09:37:42.046 INFO:teuthology.orchestra.run.smithi160.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T09:37:42.046 INFO:teuthology.orchestra.run.smithi160.stdout:ANSI_COLOR="0;31" 2024-09-21T09:37:42.046 INFO:teuthology.orchestra.run.smithi160.stdout:LOGO="fedora-logo-icon" 2024-09-21T09:37:42.046 INFO:teuthology.orchestra.run.smithi160.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T09:37:42.046 INFO:teuthology.orchestra.run.smithi160.stdout:HOME_URL="https://centos.org/" 2024-09-21T09:37:42.046 INFO:teuthology.orchestra.run.smithi160.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T09:37:42.046 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T09:37:42.046 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T09:37:42.046 INFO:teuthology.lock.ops:Updating smithi160.front.sepia.ceph.com on lock server 2024-09-21T09:37:42.076 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-21T09:37:42.084 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-21T09:37:42.133 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-21T09:37:42.133 DEBUG:teuthology.orchestra.run.smithi070:> test '!' -e /home/ubuntu/cephtest 2024-09-21T09:37:42.136 DEBUG:teuthology.orchestra.run.smithi160:> test '!' -e /home/ubuntu/cephtest 2024-09-21T09:37:42.153 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-21T09:37:42.180 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-21T09:37:42.180 DEBUG:teuthology.orchestra.run.smithi070:> test -z $(ls -A /var/lib/ceph) 2024-09-21T09:37:42.194 DEBUG:teuthology.orchestra.run.smithi160:> test -z $(ls -A /var/lib/ceph) 2024-09-21T09:37:42.275 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-21T09:37:42.329 INFO:teuthology.run_tasks:Running task kernel... 2024-09-21T09:37:42.341 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-21T09:37:42.342 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-21T09:37:42.342 DEBUG:teuthology.orchestra.run.smithi070:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T09:37:42.342 DEBUG:teuthology.orchestra.run.smithi160:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T09:37:42.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T09:37:42.359 DEBUG:teuthology.orchestra.run.smithi070:> uname -r 2024-09-21T09:37:42.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T09:37:42.360 DEBUG:teuthology.orchestra.run.smithi160:> uname -r 2024-09-21T09:37:42.417 INFO:teuthology.orchestra.run.smithi070.stdout:5.14.0-503.el9.x86_64 2024-09-21T09:37:42.418 INFO:teuthology.task.kernel:Running kernel on smithi070: 5.14.0-503.el9.x86_64 2024-09-21T09:37:42.418 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum install -y kernel 2024-09-21T09:37:42.419 INFO:teuthology.orchestra.run.smithi160.stdout:5.14.0-503.el9.x86_64 2024-09-21T09:37:42.419 INFO:teuthology.task.kernel:Running kernel on smithi160: 5.14.0-503.el9.x86_64 2024-09-21T09:37:42.419 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum install -y kernel 2024-09-21T09:37:45.946 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.3 MB 00:01 2024-09-21T09:37:46.038 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.3 MB 00:01 2024-09-21T09:37:47.872 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - AppStream 24 MB/s | 20 MB 00:00 2024-09-21T09:37:50.172 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - AppStream 6.6 MB/s | 20 MB 00:03 2024-09-21T09:37:52.022 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.5 MB 00:00 2024-09-21T09:37:53.473 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - Extras packages 83 kB/s | 19 kB 00:00 2024-09-21T09:37:54.594 INFO:teuthology.orchestra.run.smithi070.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:01 2024-09-21T09:37:55.648 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.5 MB 00:01 2024-09-21T09:37:57.065 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - Extras packages 99 kB/s | 19 kB 00:00 2024-09-21T09:37:57.986 INFO:teuthology.orchestra.run.smithi160.stdout:Extra Packages for Enterprise Linux 28 MB/s | 23 MB 00:00 2024-09-21T09:37:59.991 INFO:teuthology.orchestra.run.smithi070.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-21T09:38:01.897 INFO:teuthology.orchestra.run.smithi070.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T09:38:01.897 INFO:teuthology.orchestra.run.smithi070.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T09:38:01.988 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-21T09:38:01.995 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:38:01.996 INFO:teuthology.orchestra.run.smithi070.stdout: Package Architecture Version Repository Size 2024-09-21T09:38:01.996 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:38:01.996 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-21T09:38:01.996 INFO:teuthology.orchestra.run.smithi070.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T09:38:01.996 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-21T09:38:01.996 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T09:38:01.996 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T09:38:01.996 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T09:38:01.996 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:38:01.996 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-21T09:38:01.996 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:38:01.996 INFO:teuthology.orchestra.run.smithi070.stdout:Install 4 Packages 2024-09-21T09:38:01.996 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:38:01.997 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 79 M 2024-09-21T09:38:01.997 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 126 M 2024-09-21T09:38:01.997 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-21T09:38:02.359 INFO:teuthology.orchestra.run.smithi070.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 347 kB/s | 36 kB 00:00 2024-09-21T09:38:03.101 INFO:teuthology.orchestra.run.smithi070.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 19 MB/s | 16 MB 00:00 2024-09-21T09:38:03.617 INFO:teuthology.orchestra.run.smithi070.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 23 MB/s | 29 MB 00:01 2024-09-21T09:38:03.758 INFO:teuthology.orchestra.run.smithi160.stdout:lab-extras 4.5 kB/s | 1.7 kB 00:00 2024-09-21T09:38:04.314 INFO:teuthology.orchestra.run.smithi070.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 17 MB/s | 35 MB 00:02 2024-09-21T09:38:04.316 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:38:04.316 INFO:teuthology.orchestra.run.smithi070.stdout:Total 34 MB/s | 79 MB 00:02 2024-09-21T09:38:04.994 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-21T09:38:05.332 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-21T09:38:05.333 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-21T09:38:05.699 INFO:teuthology.orchestra.run.smithi160.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T09:38:05.699 INFO:teuthology.orchestra.run.smithi160.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T09:38:05.787 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T09:38:05.792 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:38:05.792 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2024-09-21T09:38:05.792 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:38:05.792 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-21T09:38:05.792 INFO:teuthology.orchestra.run.smithi160.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T09:38:05.792 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-21T09:38:05.792 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T09:38:05.792 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T09:38:05.792 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T09:38:05.792 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:38:05.792 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-21T09:38:05.792 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:38:05.793 INFO:teuthology.orchestra.run.smithi160.stdout:Install 4 Packages 2024-09-21T09:38:05.793 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:38:05.793 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 79 M 2024-09-21T09:38:05.793 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 126 M 2024-09-21T09:38:05.793 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-21T09:38:05.918 INFO:teuthology.orchestra.run.smithi160.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 456 kB/s | 36 kB 00:00 2024-09-21T09:38:06.297 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-21T09:38:06.298 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-21T09:38:08.724 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-21T09:38:09.370 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T09:38:09.451 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T09:38:09.901 INFO:teuthology.orchestra.run.smithi160.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 3.9 MB/s | 16 MB 00:04 2024-09-21T09:38:12.197 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T09:38:12.321 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T09:38:13.151 INFO:teuthology.orchestra.run.smithi160.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 4.0 MB/s | 29 MB 00:07 2024-09-21T09:38:14.859 INFO:teuthology.orchestra.run.smithi160.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 3.8 MB/s | 35 MB 00:09 2024-09-21T09:38:14.860 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:38:14.860 INFO:teuthology.orchestra.run.smithi160.stdout:Total 8.7 MB/s | 79 MB 00:09 2024-09-21T09:38:15.545 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-21T09:38:15.929 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-21T09:38:15.929 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-21T09:38:16.898 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-21T09:38:16.898 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-21T09:38:17.126 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T09:38:17.246 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:38:19.325 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-21T09:38:19.968 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T09:38:20.051 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T09:38:22.067 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:38:22.848 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T09:38:22.979 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T09:38:27.802 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T09:38:28.104 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:38:32.946 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:39:00.142 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:39:00.183 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:39:00.561 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:39:00.562 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T09:39:00.562 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T09:39:00.562 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T09:39:01.371 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:39:01.371 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:39:01.371 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-21T09:39:01.371 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T09:39:01.371 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T09:39:01.371 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T09:39:01.371 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T09:39:01.371 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:39:01.371 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-21T09:39:01.709 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T09:39:01.795 INFO:teuthology.orchestra.run.smithi070.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T09:39:01.795 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T09:39:01.796 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T09:39:01.796 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T09:39:01.796 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T09:39:01.796 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-21T09:39:01.796 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi070.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T09:39:01.796 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T09:39:01.855 INFO:teuthology.orchestra.run.smithi070.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T09:39:01.855 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T09:39:01.855 DEBUG:teuthology.orchestra.run.smithi070:> sudo rpm -qi grub2-tools 2024-09-21T09:39:01.886 INFO:teuthology.orchestra.run.smithi070.stdout:Name : grub2-tools 2024-09-21T09:39:01.886 INFO:teuthology.orchestra.run.smithi070.stdout:Epoch : 1 2024-09-21T09:39:01.886 INFO:teuthology.orchestra.run.smithi070.stdout:Version : 2.06 2024-09-21T09:39:01.886 INFO:teuthology.orchestra.run.smithi070.stdout:Release : 82.el9 2024-09-21T09:39:01.886 INFO:teuthology.orchestra.run.smithi070.stdout:Architecture: x86_64 2024-09-21T09:39:01.886 INFO:teuthology.orchestra.run.smithi070.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T09:39:01.887 INFO:teuthology.orchestra.run.smithi070.stdout:Group : Unspecified 2024-09-21T09:39:01.887 INFO:teuthology.orchestra.run.smithi070.stdout:Size : 8274670 2024-09-21T09:39:01.887 INFO:teuthology.orchestra.run.smithi070.stdout:License : GPLv3+ 2024-09-21T09:39:01.887 INFO:teuthology.orchestra.run.smithi070.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T09:39:01.887 INFO:teuthology.orchestra.run.smithi070.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T09:39:01.887 INFO:teuthology.orchestra.run.smithi070.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T09:39:01.887 INFO:teuthology.orchestra.run.smithi070.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T09:39:01.887 INFO:teuthology.orchestra.run.smithi070.stdout:Packager : builder@centos.org 2024-09-21T09:39:01.887 INFO:teuthology.orchestra.run.smithi070.stdout:Vendor : CentOS 2024-09-21T09:39:01.887 INFO:teuthology.orchestra.run.smithi070.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T09:39:01.887 INFO:teuthology.orchestra.run.smithi070.stdout:Summary : Support tools for GRUB. 2024-09-21T09:39:01.887 INFO:teuthology.orchestra.run.smithi070.stdout:Description : 2024-09-21T09:39:01.887 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:39:01.887 INFO:teuthology.orchestra.run.smithi070.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T09:39:01.888 INFO:teuthology.orchestra.run.smithi070.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T09:39:01.888 INFO:teuthology.orchestra.run.smithi070.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T09:39:01.888 INFO:teuthology.orchestra.run.smithi070.stdout:hardware devices. 2024-09-21T09:39:01.888 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:39:01.888 INFO:teuthology.orchestra.run.smithi070.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T09:39:01.888 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T09:39:01.889 INFO:teuthology.task.kernel:Updating grub on smithi070 to boot 5.14.0-508.el9.x86_64 2024-09-21T09:39:01.889 DEBUG:teuthology.orchestra.run.smithi070:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T09:39:02.522 INFO:teuthology.orchestra.run.smithi070.stderr:Generating grub configuration file ... 2024-09-21T09:39:03.831 INFO:teuthology.orchestra.run.smithi070.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T09:39:03.852 INFO:teuthology.orchestra.run.smithi070.stderr:done 2024-09-21T09:39:03.855 DEBUG:teuthology.orchestra.run.smithi070:> mktemp 2024-09-21T09:39:03.870 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/tmp.w1mpq4OdNY 2024-09-21T09:39:03.870 DEBUG:teuthology.orchestra.run.smithi070:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.w1mpq4OdNY 2024-09-21T09:39:03.954 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0666 /tmp/tmp.w1mpq4OdNY 2024-09-21T09:39:04.106 DEBUG:teuthology.orchestra.remote:smithi070:/tmp/tmp.w1mpq4OdNY is 6KB 2024-09-21T09:39:04.155 DEBUG:teuthology.orchestra.run.smithi070:> rm -fr /tmp/tmp.w1mpq4OdNY 2024-09-21T09:39:04.170 DEBUG:teuthology.orchestra.run.smithi070:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T09:39:04.232 INFO:teuthology.orchestra.run.smithi070.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T09:39:04.233 INFO:teuthology.orchestra.run.smithi070.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T09:39:04.233 INFO:teuthology.orchestra.run.smithi070.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T09:39:04.233 INFO:teuthology.orchestra.run.smithi070.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T09:39:04.234 DEBUG:teuthology.orchestra.run.smithi070:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T09:39:04.391 DEBUG:teuthology.orchestra.run.smithi070:> sudo shutdown -r now 2024-09-21T09:39:11.013 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:39:11.054 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:39:11.386 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:39:11.386 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T09:39:11.386 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T09:39:11.386 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T09:39:12.205 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:39:12.205 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:39:12.205 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-21T09:39:12.205 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T09:39:12.205 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T09:39:12.205 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T09:39:12.205 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T09:39:12.205 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:39:12.205 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T09:39:12.534 DEBUG:teuthology.orchestra.run.smithi160:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T09:39:12.623 INFO:teuthology.orchestra.run.smithi160.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T09:39:12.624 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T09:39:12.624 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T09:39:12.624 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T09:39:12.624 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T09:39:12.624 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-21T09:39:12.624 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi160.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T09:39:12.624 DEBUG:teuthology.orchestra.run.smithi160:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T09:39:12.683 INFO:teuthology.orchestra.run.smithi160.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T09:39:12.684 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T09:39:12.684 DEBUG:teuthology.orchestra.run.smithi160:> sudo rpm -qi grub2-tools 2024-09-21T09:39:12.719 INFO:teuthology.orchestra.run.smithi160.stdout:Name : grub2-tools 2024-09-21T09:39:12.720 INFO:teuthology.orchestra.run.smithi160.stdout:Epoch : 1 2024-09-21T09:39:12.720 INFO:teuthology.orchestra.run.smithi160.stdout:Version : 2.06 2024-09-21T09:39:12.720 INFO:teuthology.orchestra.run.smithi160.stdout:Release : 82.el9 2024-09-21T09:39:12.720 INFO:teuthology.orchestra.run.smithi160.stdout:Architecture: x86_64 2024-09-21T09:39:12.720 INFO:teuthology.orchestra.run.smithi160.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T09:39:12.720 INFO:teuthology.orchestra.run.smithi160.stdout:Group : Unspecified 2024-09-21T09:39:12.720 INFO:teuthology.orchestra.run.smithi160.stdout:Size : 8274670 2024-09-21T09:39:12.720 INFO:teuthology.orchestra.run.smithi160.stdout:License : GPLv3+ 2024-09-21T09:39:12.720 INFO:teuthology.orchestra.run.smithi160.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T09:39:12.720 INFO:teuthology.orchestra.run.smithi160.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T09:39:12.720 INFO:teuthology.orchestra.run.smithi160.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T09:39:12.720 INFO:teuthology.orchestra.run.smithi160.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T09:39:12.720 INFO:teuthology.orchestra.run.smithi160.stdout:Packager : builder@centos.org 2024-09-21T09:39:12.721 INFO:teuthology.orchestra.run.smithi160.stdout:Vendor : CentOS 2024-09-21T09:39:12.721 INFO:teuthology.orchestra.run.smithi160.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T09:39:12.721 INFO:teuthology.orchestra.run.smithi160.stdout:Summary : Support tools for GRUB. 2024-09-21T09:39:12.721 INFO:teuthology.orchestra.run.smithi160.stdout:Description : 2024-09-21T09:39:12.721 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:39:12.721 INFO:teuthology.orchestra.run.smithi160.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T09:39:12.721 INFO:teuthology.orchestra.run.smithi160.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T09:39:12.721 INFO:teuthology.orchestra.run.smithi160.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T09:39:12.721 INFO:teuthology.orchestra.run.smithi160.stdout:hardware devices. 2024-09-21T09:39:12.721 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:39:12.721 INFO:teuthology.orchestra.run.smithi160.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T09:39:12.722 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T09:39:12.722 INFO:teuthology.task.kernel:Updating grub on smithi160 to boot 5.14.0-508.el9.x86_64 2024-09-21T09:39:12.722 DEBUG:teuthology.orchestra.run.smithi160:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T09:39:13.364 INFO:teuthology.orchestra.run.smithi160.stderr:Generating grub configuration file ... 2024-09-21T09:39:14.746 INFO:teuthology.orchestra.run.smithi160.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T09:39:14.772 INFO:teuthology.orchestra.run.smithi160.stderr:done 2024-09-21T09:39:14.774 DEBUG:teuthology.orchestra.run.smithi160:> mktemp 2024-09-21T09:39:14.790 INFO:teuthology.orchestra.run.smithi160.stdout:/tmp/tmp.CnMvXU1GyT 2024-09-21T09:39:14.790 DEBUG:teuthology.orchestra.run.smithi160:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.CnMvXU1GyT 2024-09-21T09:39:14.857 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 0666 /tmp/tmp.CnMvXU1GyT 2024-09-21T09:39:14.993 DEBUG:teuthology.orchestra.remote:smithi160:/tmp/tmp.CnMvXU1GyT is 6KB 2024-09-21T09:39:15.041 DEBUG:teuthology.orchestra.run.smithi160:> rm -fr /tmp/tmp.CnMvXU1GyT 2024-09-21T09:39:15.057 DEBUG:teuthology.orchestra.run.smithi160:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T09:39:15.119 INFO:teuthology.orchestra.run.smithi160.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T09:39:15.120 INFO:teuthology.orchestra.run.smithi160.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T09:39:15.120 INFO:teuthology.orchestra.run.smithi160.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T09:39:15.120 INFO:teuthology.orchestra.run.smithi160.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T09:39:15.121 DEBUG:teuthology.orchestra.run.smithi160:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T09:39:15.277 DEBUG:teuthology.orchestra.run.smithi160:> sudo shutdown -r now 2024-09-21T09:39:34.413 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-21T09:39:34.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-09-21T09:39:34.414 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:39:45.290 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-21T09:39:45.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-21T09:39:45.291 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:39:52.947 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.70 2024-09-21T09:40:01.954 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-09-21T09:40:01.955 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:40:03.694 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-21T09:40:12.702 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-21T09:40:12.703 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:40:15.759 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-21T09:40:27.769 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-21T09:40:27.770 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:40:59.598 DEBUG:teuthology.orchestra.run.smithi160:> true 2024-09-21T09:41:00.176 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-21T09:41:00.176 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-21T09:41:00.176 DEBUG:teuthology.orchestra.run.smithi160:> uname -r 2024-09-21T09:41:00.191 INFO:teuthology.orchestra.run.smithi160.stdout:5.14.0-508.el9.x86_64 2024-09-21T09:41:00.191 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T09:41:00.191 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T09:41:00.191 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T09:41:01.192 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-21T09:41:01.192 DEBUG:teuthology.orchestra.run.smithi160:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T09:41:01.276 INFO:teuthology.orchestra.run.smithi160.stdout:ttyS1 2024-09-21T09:41:01.302 DEBUG:teuthology.parallel:result is None 2024-09-21T09:41:01.957 DEBUG:teuthology.orchestra.remote:timed out 2024-09-21T09:41:13.958 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-09-21T09:41:13.958 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:41:14.421 DEBUG:teuthology.orchestra.run.smithi070:> true 2024-09-21T09:41:14.736 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-09-21T09:41:14.736 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-21T09:41:14.736 DEBUG:teuthology.orchestra.run.smithi070:> uname -r 2024-09-21T09:41:14.752 INFO:teuthology.orchestra.run.smithi070.stdout:5.14.0-508.el9.x86_64 2024-09-21T09:41:14.752 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T09:41:14.752 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T09:41:14.752 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T09:41:15.753 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-21T09:41:15.753 DEBUG:teuthology.orchestra.run.smithi070:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T09:41:15.848 INFO:teuthology.orchestra.run.smithi070.stdout:ttyS1 2024-09-21T09:41:15.865 DEBUG:teuthology.parallel:result is None 2024-09-21T09:41:15.865 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-21T09:41:15.873 INFO:teuthology.task.internal:Creating test directory... 2024-09-21T09:41:15.874 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T09:41:15.877 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T09:41:15.897 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-21T09:41:15.946 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-21T09:41:15.996 INFO:teuthology.task.internal:Creating archive directory... 2024-09-21T09:41:15.996 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T09:41:15.999 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T09:41:16.104 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-21T09:41:16.110 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-21T09:41:16.111 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T09:41:16.113 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T09:41:16.146 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T09:41:16.147 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T09:41:16.157 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T09:41:16.158 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T09:41:16.159 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-21T09:41:16.166 INFO:teuthology.task.internal:Configuring sudo... 2024-09-21T09:41:16.166 DEBUG:teuthology.orchestra.run.smithi070:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T09:41:16.203 DEBUG:teuthology.orchestra.run.smithi160:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T09:41:16.231 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-21T09:41:16.239 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-21T09:41:16.239 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T09:41:16.270 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T09:41:16.288 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T09:41:16.363 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T09:41:16.416 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T09:41:16.499 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T09:41:16.539 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T09:41:16.539 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T09:41:16.603 DEBUG:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T09:41:16.650 DEBUG:teuthology.orchestra.run.smithi160:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T09:41:16.706 DEBUG:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T09:41:16.791 DEBUG:teuthology.orchestra.run.smithi160:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T09:41:16.831 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T09:41:16.832 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T09:41:16.897 DEBUG:teuthology.orchestra.run.smithi070:> sudo service rsyslog restart 2024-09-21T09:41:16.900 DEBUG:teuthology.orchestra.run.smithi160:> sudo service rsyslog restart 2024-09-21T09:41:16.947 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T09:41:16.989 INFO:teuthology.orchestra.run.smithi160.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T09:41:17.409 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-21T09:41:17.416 INFO:teuthology.task.internal:Starting timer... 2024-09-21T09:41:17.417 INFO:teuthology.run_tasks:Running task pcp... 2024-09-21T09:41:17.474 INFO:teuthology.run_tasks:Running task selinux... 2024-09-21T09:41:17.524 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-21T09:41:17.525 DEBUG:teuthology.orchestra.run.smithi070:> sudo service auditd rotate 2024-09-21T09:41:17.583 INFO:teuthology.orchestra.run.smithi070.stdout:Rotating logs: 2024-09-21T09:41:17.585 DEBUG:teuthology.orchestra.run.smithi160:> sudo service auditd rotate 2024-09-21T09:41:17.636 INFO:teuthology.orchestra.run.smithi160.stdout:Rotating logs: 2024-09-21T09:41:17.638 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-21T09:41:17.639 DEBUG:teuthology.orchestra.run.smithi070:> /usr/sbin/getenforce 2024-09-21T09:41:17.661 INFO:teuthology.orchestra.run.smithi070.stdout:Permissive 2024-09-21T09:41:17.661 DEBUG:teuthology.orchestra.run.smithi160:> /usr/sbin/getenforce 2024-09-21T09:41:17.691 INFO:teuthology.orchestra.run.smithi160.stdout:Permissive 2024-09-21T09:41:17.691 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi070.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi160.front.sepia.ceph.com': 'permissive'} 2024-09-21T09:41:17.692 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T09:41:17.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T09:41:17.726 DEBUG:teuthology.orchestra.run.smithi160:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T09:41:17.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T09:41:17.759 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-21T09:41:17.759 DEBUG:teuthology.orchestra.run.smithi070:> sudo /usr/sbin/setenforce permissive 2024-09-21T09:41:17.799 DEBUG:teuthology.orchestra.run.smithi160:> sudo /usr/sbin/setenforce permissive 2024-09-21T09:41:17.845 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-21T09:41:17.853 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-21T09:41:17.863 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-21T09:41:17.864 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi070.front.sepia.ceph.com,smithi160.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-21T09:45:04.735 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi070.front.sepia.ceph.com'), Remote(name='ubuntu@smithi160.front.sepia.ceph.com')] 2024-09-21T09:45:04.736 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-09-21T09:45:04.737 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:45:04.814 DEBUG:teuthology.orchestra.run.smithi070:> true 2024-09-21T09:45:04.851 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-09-21T09:45:04.851 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-21T09:45:04.852 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:45:04.926 DEBUG:teuthology.orchestra.run.smithi160:> true 2024-09-21T09:45:05.016 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-21T09:45:05.017 INFO:teuthology.run_tasks:Running task clock... 2024-09-21T09:45:05.026 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-21T09:45:05.027 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T09:45:05.027 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T09:45:05.030 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T09:45:05.030 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T09:45:05.061 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T09:45:05.077 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T09:45:05.090 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T09:45:05.103 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: ntpd: command not found 2024-09-21T09:45:05.106 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T09:45:05.115 INFO:teuthology.orchestra.run.smithi070.stdout:506 Cannot talk to daemon 2024-09-21T09:45:05.129 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T09:45:05.133 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: ntpd: command not found 2024-09-21T09:45:05.144 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T09:45:05.145 INFO:teuthology.orchestra.run.smithi160.stdout:506 Cannot talk to daemon 2024-09-21T09:45:05.160 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T09:45:05.175 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T09:45:05.198 INFO:teuthology.orchestra.run.smithi070.stderr:bash: line 1: ntpq: command not found 2024-09-21T09:45:05.202 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T09:45:05.202 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2024-09-21T09:45:05.202 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:45:05.202 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:45:05.202 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:45:05.203 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:45:05.233 INFO:teuthology.orchestra.run.smithi160.stderr:bash: line 1: ntpq: command not found 2024-09-21T09:45:05.237 INFO:teuthology.orchestra.run.smithi160.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T09:45:05.237 INFO:teuthology.orchestra.run.smithi160.stdout:=============================================================================== 2024-09-21T09:45:05.238 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:45:05.238 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:45:05.238 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:45:05.238 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:45:05.238 INFO:teuthology.run_tasks:Running task pexec... 2024-09-21T09:45:05.247 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-21T09:45:05.247 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi070.front.sepia.ceph.com 2024-09-21T09:45:05.247 DEBUG:teuthology.orchestra.run.smithi070:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-21T09:45:05.248 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi160.front.sepia.ceph.com 2024-09-21T09:45:05.248 DEBUG:teuthology.orchestra.run.smithi160:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-21T09:45:05.599 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-21T09:45:05.601 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:45:05.602 INFO:teuthology.orchestra.run.smithi070.stdout: Package Architecture Version Repository Size 2024-09-21T09:45:05.602 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:45:05.602 INFO:teuthology.orchestra.run.smithi070.stdout:Removing: 2024-09-21T09:45:05.602 INFO:teuthology.orchestra.run.smithi070.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-21T09:45:05.602 INFO:teuthology.orchestra.run.smithi070.stdout:Removing unused dependencies: 2024-09-21T09:45:05.602 INFO:teuthology.orchestra.run.smithi070.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-21T09:45:05.602 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:45:05.602 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-21T09:45:05.602 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:45:05.602 INFO:teuthology.orchestra.run.smithi070.stdout:Remove 2 Packages 2024-09-21T09:45:05.602 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:45:05.602 INFO:teuthology.orchestra.run.smithi070.stdout:Freed space: 5.5 M 2024-09-21T09:45:05.603 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-21T09:45:05.606 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-21T09:45:05.606 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-21T09:45:05.636 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T09:45:05.651 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:45:05.651 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2024-09-21T09:45:05.651 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:45:05.651 INFO:teuthology.orchestra.run.smithi160.stdout:Removing: 2024-09-21T09:45:05.651 INFO:teuthology.orchestra.run.smithi160.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-21T09:45:05.651 INFO:teuthology.orchestra.run.smithi160.stdout:Removing unused dependencies: 2024-09-21T09:45:05.651 INFO:teuthology.orchestra.run.smithi160.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-21T09:45:05.651 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:45:05.651 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-21T09:45:05.651 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:45:05.651 INFO:teuthology.orchestra.run.smithi160.stdout:Remove 2 Packages 2024-09-21T09:45:05.651 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:45:05.651 INFO:teuthology.orchestra.run.smithi160.stdout:Freed space: 5.5 M 2024-09-21T09:45:05.651 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-21T09:45:05.655 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-21T09:45:05.655 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-21T09:45:05.667 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-21T09:45:05.667 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-21T09:45:05.715 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-21T09:45:05.716 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-21T09:45:05.817 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-21T09:45:05.847 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-21T09:45:05.892 INFO:teuthology.orchestra.run.smithi070.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-21T09:45:05.923 INFO:teuthology.orchestra.run.smithi160.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-21T09:45:05.948 INFO:teuthology.orchestra.run.smithi070.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T09:45:05.989 INFO:teuthology.orchestra.run.smithi160.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T09:45:06.499 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T09:45:06.499 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-21T09:45:06.510 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T09:45:06.510 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-21T09:45:06.758 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-21T09:45:06.758 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:45:06.758 INFO:teuthology.orchestra.run.smithi070.stdout:Removed: 2024-09-21T09:45:06.758 INFO:teuthology.orchestra.run.smithi070.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T09:45:06.758 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:45:06.758 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-21T09:45:06.778 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-21T09:45:06.778 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:45:06.778 INFO:teuthology.orchestra.run.smithi160.stdout:Removed: 2024-09-21T09:45:06.778 INFO:teuthology.orchestra.run.smithi160.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T09:45:06.778 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:45:06.778 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T09:45:07.599 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:00:51 ago on Sat 21 Sep 2024 09:44:16 AM UTC. 2024-09-21T09:45:07.611 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:52 ago on Sat 21 Sep 2024 09:44:15 AM UTC. 2024-09-21T09:45:07.733 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-21T09:45:07.734 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:45:07.734 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-21T09:45:07.734 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:45:07.734 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-21T09:45:07.734 INFO:teuthology.orchestra.run.smithi070.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-21T09:45:07.734 INFO:teuthology.orchestra.run.smithi070.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-21T09:45:07.734 INFO:teuthology.orchestra.run.smithi070.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-21T09:45:07.734 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-21T09:45:07.734 INFO:teuthology.orchestra.run.smithi070.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-21T09:45:07.735 INFO:teuthology.orchestra.run.smithi070.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-21T09:45:07.735 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-21T09:45:07.735 INFO:teuthology.orchestra.run.smithi070.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-21T09:45:07.735 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:45:07.735 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-21T09:45:07.735 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:45:07.735 INFO:teuthology.orchestra.run.smithi070.stdout:Install 7 Packages 2024-09-21T09:45:07.735 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:45:07.747 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T09:45:07.748 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:45:07.749 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-21T09:45:07.749 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:45:07.749 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-21T09:45:07.749 INFO:teuthology.orchestra.run.smithi160.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-21T09:45:07.749 INFO:teuthology.orchestra.run.smithi160.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-21T09:45:07.749 INFO:teuthology.orchestra.run.smithi160.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-21T09:45:07.749 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-21T09:45:07.749 INFO:teuthology.orchestra.run.smithi160.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-21T09:45:07.749 INFO:teuthology.orchestra.run.smithi160.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-21T09:45:07.749 INFO:teuthology.orchestra.run.smithi160.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-21T09:45:07.749 INFO:teuthology.orchestra.run.smithi160.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-21T09:45:07.749 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:45:07.749 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-21T09:45:07.750 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:45:07.750 INFO:teuthology.orchestra.run.smithi160.stdout:Install 7 Packages 2024-09-21T09:45:07.750 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:45:07.750 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 5.2 M 2024-09-21T09:45:07.750 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 19 M 2024-09-21T09:45:07.750 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-21T09:45:07.767 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 5.2 M 2024-09-21T09:45:07.767 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 19 M 2024-09-21T09:45:07.767 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-21T09:45:07.970 INFO:teuthology.orchestra.run.smithi070.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 464 kB/s | 44 kB 00:00 2024-09-21T09:45:07.995 INFO:teuthology.orchestra.run.smithi070.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 875 kB/s | 106 kB 00:00 2024-09-21T09:45:08.021 INFO:teuthology.orchestra.run.smithi070.stdout:(3/7): python3-configshell-1.1.30-1.el9.noarch. 1.4 MB/s | 72 kB 00:00 2024-09-21T09:45:08.046 INFO:teuthology.orchestra.run.smithi070.stdout:(4/7): python3-kmod-0.9-32.el9.x86_64.rpm 1.6 MB/s | 84 kB 00:00 2024-09-21T09:45:08.079 INFO:teuthology.orchestra.run.smithi070.stdout:(5/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 4.3 MB/s | 906 kB 00:00 2024-09-21T09:45:08.135 INFO:teuthology.orchestra.run.smithi160.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 380 kB/s | 44 kB 00:00 2024-09-21T09:45:08.146 INFO:teuthology.orchestra.run.smithi070.stdout:(6/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 6.5 MB/s | 837 kB 00:00 2024-09-21T09:45:08.161 INFO:teuthology.orchestra.run.smithi160.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 746 kB/s | 106 kB 00:00 2024-09-21T09:45:08.211 INFO:teuthology.orchestra.run.smithi160.stdout:(3/7): python3-kmod-0.9-32.el9.x86_64.rpm 1.6 MB/s | 84 kB 00:00 2024-09-21T09:45:08.236 INFO:teuthology.orchestra.run.smithi160.stdout:(4/7): python3-configshell-1.1.30-1.el9.noarch. 714 kB/s | 72 kB 00:00 2024-09-21T09:45:08.295 INFO:teuthology.orchestra.run.smithi160.stdout:(5/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 3.2 MB/s | 906 kB 00:00 2024-09-21T09:45:08.339 INFO:teuthology.orchestra.run.smithi070.stdout:(7/7): runc-1.1.13-4.el9.x86_64.rpm 11 MB/s | 3.2 MB 00:00 2024-09-21T09:45:08.339 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:45:08.339 INFO:teuthology.orchestra.run.smithi070.stdout:Total 8.8 MB/s | 5.2 MB 00:00 2024-09-21T09:45:08.395 INFO:teuthology.orchestra.run.smithi160.stdout:(6/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 4.4 MB/s | 837 kB 00:00 2024-09-21T09:45:08.448 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-21T09:45:08.462 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-21T09:45:08.462 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-21T09:45:08.495 INFO:teuthology.orchestra.run.smithi160.stdout:(7/7): runc-1.1.13-4.el9.x86_64.rpm 12 MB/s | 3.2 MB 00:00 2024-09-21T09:45:08.496 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:45:08.496 INFO:teuthology.orchestra.run.smithi160.stdout:Total 7.1 MB/s | 5.2 MB 00:00 2024-09-21T09:45:08.608 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-21T09:45:08.623 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-21T09:45:08.623 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-21T09:45:08.790 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-21T09:45:08.790 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-21T09:45:08.960 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-21T09:45:08.960 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-21T09:45:09.254 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-21T09:45:09.365 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-21T09:45:09.400 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-21T09:45:09.429 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-21T09:45:09.442 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-21T09:45:09.523 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-21T09:45:09.564 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-21T09:45:09.594 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-21T09:45:09.641 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-21T09:45:09.679 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-21T09:45:09.801 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-21T09:45:09.834 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-21T09:45:10.132 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-21T09:45:10.168 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-21T09:45:10.274 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-21T09:45:10.278 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-21T09:45:10.304 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-21T09:45:10.334 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T09:45:10.418 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-21T09:45:10.481 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T09:45:11.323 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T09:45:11.324 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-21T09:45:11.324 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-21T09:45:11.324 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-21T09:45:11.324 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-21T09:45:11.324 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-21T09:45:11.324 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-21T09:45:11.439 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T09:45:11.439 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-21T09:45:11.439 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-21T09:45:11.439 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-21T09:45:11.439 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-21T09:45:11.439 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-21T09:45:11.439 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-21T09:45:11.750 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T09:45:11.750 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:45:11.750 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-21T09:45:11.750 INFO:teuthology.orchestra.run.smithi070.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T09:45:11.750 INFO:teuthology.orchestra.run.smithi070.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-21T09:45:11.750 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-21T09:45:11.750 INFO:teuthology.orchestra.run.smithi070.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-21T09:45:11.750 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:45:11.751 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-21T09:45:11.809 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T09:45:11.809 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:45:11.809 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-21T09:45:11.809 INFO:teuthology.orchestra.run.smithi160.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T09:45:11.810 INFO:teuthology.orchestra.run.smithi160.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-21T09:45:11.810 INFO:teuthology.orchestra.run.smithi160.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-21T09:45:11.810 INFO:teuthology.orchestra.run.smithi160.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-21T09:45:11.810 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:45:11.810 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T09:45:12.144 DEBUG:teuthology.parallel:result is None 2024-09-21T09:45:12.217 DEBUG:teuthology.parallel:result is None 2024-09-21T09:45:12.217 INFO:teuthology.run_tasks:Running task install... 2024-09-21T09:45:12.225 DEBUG:teuthology.task.install:project ceph 2024-09-21T09:45:12.225 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}} 2024-09-21T09:45:12.225 DEBUG:teuthology.task.install:config {'branch': 'reef', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'} 2024-09-21T09:45:12.225 INFO:teuthology.task.install:Using flavor: default 2024-09-21T09:45:12.230 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-21T09:45:12.231 INFO:teuthology.task.install:extra packages: [] 2024-09-21T09:45:12.231 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'reef', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'tag': None, 'wait_for_package': False} 2024-09-21T09:45:12.231 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:45:12.231 INFO:teuthology.packaging:ref: None 2024-09-21T09:45:12.231 INFO:teuthology.packaging:tag: None 2024-09-21T09:45:12.231 INFO:teuthology.packaging:branch: reef 2024-09-21T09:45:12.231 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:45:12.232 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-09-21T09:45:12.233 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'reef', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'tag': None, 'wait_for_package': False} 2024-09-21T09:45:12.233 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:45:12.233 INFO:teuthology.packaging:ref: None 2024-09-21T09:45:12.233 INFO:teuthology.packaging:tag: None 2024-09-21T09:45:12.234 INFO:teuthology.packaging:branch: reef 2024-09-21T09:45:12.234 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:45:12.234 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-09-21T09:45:12.472 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/reef/6a1bdf854bb5803efd58957907442aa01f41a8fc/centos/9/flavors/default/ 2024-09-21T09:45:12.473 INFO:teuthology.task.install.rpm:Package version is 18.2.4-871.g6a1bdf85 2024-09-21T09:45:12.507 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/reef/6a1bdf854bb5803efd58957907442aa01f41a8fc/centos/9/flavors/default/ 2024-09-21T09:45:12.507 INFO:teuthology.task.install.rpm:Package version is 18.2.4-871.g6a1bdf85 2024-09-21T09:45:12.582 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/reef/6a1bdf854bb5803efd58957907442aa01f41a8fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/6a1bdf854bb5803efd58957907442aa01f41a8fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/6a1bdf854bb5803efd58957907442aa01f41a8fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T09:45:12.582 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T09:45:12.582 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T09:45:12.608 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-21T09:45:12.608 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:45:12.608 INFO:teuthology.packaging:ref: None 2024-09-21T09:45:12.608 INFO:teuthology.packaging:tag: None 2024-09-21T09:45:12.608 INFO:teuthology.packaging:branch: reef 2024-09-21T09:45:12.608 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:45:12.608 DEBUG:teuthology.orchestra.run.smithi160:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/reef/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T09:45:12.612 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/reef/6a1bdf854bb5803efd58957907442aa01f41a8fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/6a1bdf854bb5803efd58957907442aa01f41a8fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/6a1bdf854bb5803efd58957907442aa01f41a8fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T09:45:12.612 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T09:45:12.612 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T09:45:12.638 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-21T09:45:12.639 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:45:12.639 INFO:teuthology.packaging:ref: None 2024-09-21T09:45:12.639 INFO:teuthology.packaging:tag: None 2024-09-21T09:45:12.639 INFO:teuthology.packaging:branch: reef 2024-09-21T09:45:12.639 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:45:12.639 DEBUG:teuthology.orchestra.run.smithi070:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/reef/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T09:45:12.702 DEBUG:teuthology.orchestra.run.smithi160:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T09:45:12.705 DEBUG:teuthology.orchestra.run.smithi070:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T09:45:12.736 DEBUG:teuthology.orchestra.run.smithi160:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T09:45:12.780 DEBUG:teuthology.orchestra.run.smithi070:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T09:45:12.800 INFO:teuthology.orchestra.run.smithi160.stdout:check_obsoletes = 1 2024-09-21T09:45:12.802 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum clean all 2024-09-21T09:45:12.844 INFO:teuthology.orchestra.run.smithi070.stdout:check_obsoletes = 1 2024-09-21T09:45:12.845 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum clean all 2024-09-21T09:45:13.028 INFO:teuthology.orchestra.run.smithi160.stdout:45 files removed 2024-09-21T09:45:13.054 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-radosgw 2024-09-21T09:45:13.089 INFO:teuthology.orchestra.run.smithi070.stdout:45 files removed 2024-09-21T09:45:13.116 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-radosgw 2024-09-21T09:45:13.601 INFO:teuthology.orchestra.run.smithi160.stdout:ceph packages for x86_64 295 kB/s | 77 kB 00:00 2024-09-21T09:45:13.672 INFO:teuthology.orchestra.run.smithi070.stdout:ceph packages for x86_64 283 kB/s | 77 kB 00:00 2024-09-21T09:45:13.876 INFO:teuthology.orchestra.run.smithi160.stdout:ceph noarch packages 49 kB/s | 11 kB 00:00 2024-09-21T09:45:13.956 INFO:teuthology.orchestra.run.smithi070.stdout:ceph noarch packages 48 kB/s | 11 kB 00:00 2024-09-21T09:45:14.153 INFO:teuthology.orchestra.run.smithi160.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-21T09:45:14.241 INFO:teuthology.orchestra.run.smithi070.stdout:ceph source packages 7.6 kB/s | 1.9 kB 00:00 2024-09-21T09:45:15.911 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.3 MB 00:01 2024-09-21T09:45:16.756 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - BaseOS 3.3 MB/s | 8.3 MB 00:02 2024-09-21T09:45:17.620 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - AppStream 32 MB/s | 20 MB 00:00 2024-09-21T09:45:20.307 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - AppStream 8.3 MB/s | 20 MB 00:02 2024-09-21T09:45:21.913 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-21T09:45:23.605 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 19 kB 00:00 2024-09-21T09:45:24.409 INFO:teuthology.orchestra.run.smithi160.stdout:Extra Packages for Enterprise Linux 33 MB/s | 23 MB 00:00 2024-09-21T09:45:25.133 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - CRB 6.9 MB/s | 6.5 MB 00:00 2024-09-21T09:45:26.675 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - Extras packages 72 kB/s | 19 kB 00:00 2024-09-21T09:45:27.546 INFO:teuthology.orchestra.run.smithi070.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-21T09:45:29.832 INFO:teuthology.orchestra.run.smithi160.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-09-21T09:45:32.064 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T09:45:32.066 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:45:32.066 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-21T09:45:32.066 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:45:32.067 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-21T09:45:32.067 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-radosgw x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 7.7 M 2024-09-21T09:45:32.067 INFO:teuthology.orchestra.run.smithi160.stdout:Upgrading: 2024-09-21T09:45:32.067 INFO:teuthology.orchestra.run.smithi160.stdout: librados2 x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 3.3 M 2024-09-21T09:45:32.067 INFO:teuthology.orchestra.run.smithi160.stdout: librbd1 x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 3.0 M 2024-09-21T09:45:32.067 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-21T09:45:32.067 INFO:teuthology.orchestra.run.smithi160.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T09:45:32.067 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-base x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 5.1 M 2024-09-21T09:45:32.067 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-common x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 18 M 2024-09-21T09:45:32.067 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-selinux x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 25 k 2024-09-21T09:45:32.067 INFO:teuthology.orchestra.run.smithi160.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-09-21T09:45:32.067 INFO:teuthology.orchestra.run.smithi160.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T09:45:32.067 INFO:teuthology.orchestra.run.smithi160.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T09:45:32.067 INFO:teuthology.orchestra.run.smithi160.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T09:45:32.067 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs2 x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 711 k 2024-09-21T09:45:32.068 INFO:teuthology.orchestra.run.smithi160.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T09:45:32.068 INFO:teuthology.orchestra.run.smithi160.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T09:45:32.068 INFO:teuthology.orchestra.run.smithi160.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T09:45:32.068 INFO:teuthology.orchestra.run.smithi160.stdout: libradosstriper1 x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 477 k 2024-09-21T09:45:32.068 INFO:teuthology.orchestra.run.smithi160.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T09:45:32.068 INFO:teuthology.orchestra.run.smithi160.stdout: librgw2 x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 4.5 M 2024-09-21T09:45:32.068 INFO:teuthology.orchestra.run.smithi160.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T09:45:32.068 INFO:teuthology.orchestra.run.smithi160.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T09:45:32.068 INFO:teuthology.orchestra.run.smithi160.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T09:45:32.068 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-argparse x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 45 k 2024-09-21T09:45:32.068 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-common x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 129 k 2024-09-21T09:45:32.068 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cephfs x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 161 k 2024-09-21T09:45:32.068 INFO:teuthology.orchestra.run.smithi160.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T09:45:32.068 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rados x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 320 k 2024-09-21T09:45:32.068 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rbd x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 300 k 2024-09-21T09:45:32.069 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rgw x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 99 k 2024-09-21T09:45:32.069 INFO:teuthology.orchestra.run.smithi160.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T09:45:32.069 INFO:teuthology.orchestra.run.smithi160.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T09:45:32.069 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:45:32.069 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-21T09:45:32.069 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:45:32.069 INFO:teuthology.orchestra.run.smithi160.stdout:Install 28 Packages 2024-09-21T09:45:32.069 INFO:teuthology.orchestra.run.smithi160.stdout:Upgrade 2 Packages 2024-09-21T09:45:32.069 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:45:32.069 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 53 M 2024-09-21T09:45:32.070 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-21T09:45:33.033 INFO:teuthology.orchestra.run.smithi160.stdout:(1/30): ceph-base-18.2.4-871.g6a1bdf85.el9.x86_ 10 MB/s | 5.1 MB 00:00 2024-09-21T09:45:33.091 INFO:teuthology.orchestra.run.smithi160.stdout:(2/30): ceph-selinux-18.2.4-871.g6a1bdf85.el9.x 433 kB/s | 25 kB 00:00 2024-09-21T09:45:33.118 INFO:teuthology.orchestra.run.smithi070.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-21T09:45:33.208 INFO:teuthology.orchestra.run.smithi160.stdout:(3/30): ceph-radosgw-18.2.4-871.g6a1bdf85.el9.x 12 MB/s | 7.7 MB 00:00 2024-09-21T09:45:33.242 INFO:teuthology.orchestra.run.smithi160.stdout:(4/30): libcephfs2-18.2.4-871.g6a1bdf85.el9.x86 4.6 MB/s | 711 kB 00:00 2024-09-21T09:45:33.267 INFO:teuthology.orchestra.run.smithi160.stdout:(5/30): libradosstriper1-18.2.4-871.g6a1bdf85.e 8.0 MB/s | 477 kB 00:00 2024-09-21T09:45:33.325 INFO:teuthology.orchestra.run.smithi160.stdout:(6/30): python3-ceph-argparse-18.2.4-871.g6a1bd 774 kB/s | 45 kB 00:00 2024-09-21T09:45:33.451 INFO:teuthology.orchestra.run.smithi160.stdout:(7/30): librgw2-18.2.4-871.g6a1bdf85.el9.x86_64 21 MB/s | 4.5 MB 00:00 2024-09-21T09:45:33.476 INFO:teuthology.orchestra.run.smithi160.stdout:(8/30): python3-ceph-common-18.2.4-871.g6a1bdf8 860 kB/s | 129 kB 00:00 2024-09-21T09:45:33.501 INFO:teuthology.orchestra.run.smithi160.stdout:(9/30): python3-cephfs-18.2.4-871.g6a1bdf85.el9 3.2 MB/s | 161 kB 00:00 2024-09-21T09:45:33.527 INFO:teuthology.orchestra.run.smithi160.stdout:(10/30): python3-rados-18.2.4-871.g6a1bdf85.el9 6.2 MB/s | 320 kB 00:00 2024-09-21T09:45:33.694 INFO:teuthology.orchestra.run.smithi160.stdout:(11/30): ceph-common-18.2.4-871.g6a1bdf85.el9.x 16 MB/s | 18 MB 00:01 2024-09-21T09:45:33.727 INFO:teuthology.orchestra.run.smithi160.stdout:(12/30): python3-rgw-18.2.4-871.g6a1bdf85.el9.x 496 kB/s | 99 kB 00:00 2024-09-21T09:45:33.753 INFO:teuthology.orchestra.run.smithi160.stdout:(13/30): python3-rbd-18.2.4-871.g6a1bdf85.el9.x 1.2 MB/s | 300 kB 00:00 2024-09-21T09:45:33.828 INFO:teuthology.orchestra.run.smithi160.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 798 kB/s | 107 kB 00:00 2024-09-21T09:45:33.853 INFO:teuthology.orchestra.run.smithi160.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 452 kB/s | 45 kB 00:00 2024-09-21T09:45:33.879 INFO:teuthology.orchestra.run.smithi160.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-09-21T09:45:33.929 INFO:teuthology.orchestra.run.smithi160.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 662 kB 00:00 2024-09-21T09:45:33.954 INFO:teuthology.orchestra.run.smithi160.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 563 kB/s | 42 kB 00:00 2024-09-21T09:45:33.980 INFO:teuthology.orchestra.run.smithi160.stdout:(19/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-09-21T09:45:34.030 INFO:teuthology.orchestra.run.smithi160.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 1.1 MB/s | 111 kB 00:00 2024-09-21T09:45:34.055 INFO:teuthology.orchestra.run.smithi160.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.1 MB/s | 26 kB 00:00 2024-09-21T09:45:34.080 INFO:teuthology.orchestra.run.smithi160.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-21T09:45:34.105 INFO:teuthology.orchestra.run.smithi160.stdout:(23/30): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.0 MB/s | 308 kB 00:00 2024-09-21T09:45:34.131 INFO:teuthology.orchestra.run.smithi160.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-21T09:45:34.173 INFO:teuthology.orchestra.run.smithi160.stdout:(25/30): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-21T09:45:34.215 INFO:teuthology.orchestra.run.smithi160.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-09-21T09:45:34.282 INFO:teuthology.orchestra.run.smithi160.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-21T09:45:34.365 INFO:teuthology.orchestra.run.smithi160.stdout:(28/30): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-21T09:45:34.449 INFO:teuthology.orchestra.run.smithi160.stdout:(29/30): librados2-18.2.4-871.g6a1bdf85.el9.x86 14 MB/s | 3.3 MB 00:00 2024-09-21T09:45:34.516 INFO:teuthology.orchestra.run.smithi160.stdout:(30/30): librbd1-18.2.4-871.g6a1bdf85.el9.x86_6 13 MB/s | 3.0 MB 00:00 2024-09-21T09:45:34.520 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:45:34.521 INFO:teuthology.orchestra.run.smithi160.stdout:Total 22 MB/s | 53 MB 00:02 2024-09-21T09:45:34.591 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-21T09:45:34.627 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-21T09:45:34.627 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-21T09:45:35.082 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-21T09:45:35.082 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-21T09:45:35.336 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-21T09:45:35.338 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:45:35.338 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-21T09:45:35.338 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:45:35.338 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-21T09:45:35.338 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-radosgw x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 7.7 M 2024-09-21T09:45:35.338 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrading: 2024-09-21T09:45:35.338 INFO:teuthology.orchestra.run.smithi070.stdout: librados2 x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 3.3 M 2024-09-21T09:45:35.338 INFO:teuthology.orchestra.run.smithi070.stdout: librbd1 x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 3.0 M 2024-09-21T09:45:35.338 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-21T09:45:35.339 INFO:teuthology.orchestra.run.smithi070.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T09:45:35.339 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 5.1 M 2024-09-21T09:45:35.339 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-common x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 18 M 2024-09-21T09:45:35.339 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-selinux x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 25 k 2024-09-21T09:45:35.339 INFO:teuthology.orchestra.run.smithi070.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-09-21T09:45:35.339 INFO:teuthology.orchestra.run.smithi070.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T09:45:35.339 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T09:45:35.339 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T09:45:35.339 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs2 x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 711 k 2024-09-21T09:45:35.339 INFO:teuthology.orchestra.run.smithi070.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T09:45:35.339 INFO:teuthology.orchestra.run.smithi070.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T09:45:35.339 INFO:teuthology.orchestra.run.smithi070.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T09:45:35.339 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1 x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 477 k 2024-09-21T09:45:35.340 INFO:teuthology.orchestra.run.smithi070.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T09:45:35.340 INFO:teuthology.orchestra.run.smithi070.stdout: librgw2 x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 4.5 M 2024-09-21T09:45:35.340 INFO:teuthology.orchestra.run.smithi070.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T09:45:35.340 INFO:teuthology.orchestra.run.smithi070.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T09:45:35.340 INFO:teuthology.orchestra.run.smithi070.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T09:45:35.340 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-argparse x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 45 k 2024-09-21T09:45:35.340 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-common x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 129 k 2024-09-21T09:45:35.340 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cephfs x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 161 k 2024-09-21T09:45:35.340 INFO:teuthology.orchestra.run.smithi070.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T09:45:35.340 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rados x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 320 k 2024-09-21T09:45:35.340 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rbd x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 300 k 2024-09-21T09:45:35.340 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rgw x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 99 k 2024-09-21T09:45:35.340 INFO:teuthology.orchestra.run.smithi070.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T09:45:35.341 INFO:teuthology.orchestra.run.smithi070.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T09:45:35.341 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:45:35.341 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-21T09:45:35.341 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:45:35.341 INFO:teuthology.orchestra.run.smithi070.stdout:Install 28 Packages 2024-09-21T09:45:35.341 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrade 2 Packages 2024-09-21T09:45:35.341 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:45:35.341 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 53 M 2024-09-21T09:45:35.341 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-21T09:45:36.244 INFO:teuthology.orchestra.run.smithi070.stdout:(1/30): ceph-base-18.2.4-871.g6a1bdf85.el9.x86_ 10 MB/s | 5.1 MB 00:00 2024-09-21T09:45:36.378 INFO:teuthology.orchestra.run.smithi070.stdout:(2/30): ceph-radosgw-18.2.4-871.g6a1bdf85.el9.x 12 MB/s | 7.7 MB 00:00 2024-09-21T09:45:36.403 INFO:teuthology.orchestra.run.smithi070.stdout:(3/30): ceph-selinux-18.2.4-871.g6a1bdf85.el9.x 158 kB/s | 25 kB 00:00 2024-09-21T09:45:36.428 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-21T09:45:36.453 INFO:teuthology.orchestra.run.smithi070.stdout:(4/30): libcephfs2-18.2.4-871.g6a1bdf85.el9.x86 14 MB/s | 711 kB 00:00 2024-09-21T09:45:36.478 INFO:teuthology.orchestra.run.smithi070.stdout:(5/30): libradosstriper1-18.2.4-871.g6a1bdf85.e 6.2 MB/s | 477 kB 00:00 2024-09-21T09:45:36.521 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-09-21T09:45:36.554 INFO:teuthology.orchestra.run.smithi070.stdout:(6/30): python3-ceph-argparse-18.2.4-871.g6a1bd 600 kB/s | 45 kB 00:00 2024-09-21T09:45:36.554 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-09-21T09:45:36.612 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-09-21T09:45:36.654 INFO:teuthology.orchestra.run.smithi070.stdout:(7/30): python3-ceph-common-18.2.4-871.g6a1bdf8 1.3 MB/s | 129 kB 00:00 2024-09-21T09:45:36.729 INFO:teuthology.orchestra.run.smithi070.stdout:(8/30): librgw2-18.2.4-871.g6a1bdf85.el9.x86_64 16 MB/s | 4.5 MB 00:00 2024-09-21T09:45:36.763 INFO:teuthology.orchestra.run.smithi070.stdout:(9/30): python3-cephfs-18.2.4-871.g6a1bdf85.el9 1.4 MB/s | 161 kB 00:00 2024-09-21T09:45:36.789 INFO:teuthology.orchestra.run.smithi070.stdout:(10/30): python3-rados-18.2.4-871.g6a1bdf85.el9 5.3 MB/s | 320 kB 00:00 2024-09-21T09:45:36.964 INFO:teuthology.orchestra.run.smithi070.stdout:(11/30): ceph-common-18.2.4-871.g6a1bdf85.el9.x 15 MB/s | 18 MB 00:01 2024-09-21T09:45:36.989 INFO:teuthology.orchestra.run.smithi070.stdout:(12/30): python3-rgw-18.2.4-871.g6a1bdf85.el9.x 497 kB/s | 99 kB 00:00 2024-09-21T09:45:36.993 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-09-21T09:45:37.014 INFO:teuthology.orchestra.run.smithi070.stdout:(13/30): python3-rbd-18.2.4-871.g6a1bdf85.el9.x 1.2 MB/s | 300 kB 00:00 2024-09-21T09:45:37.037 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : librados2-2:18.2.4-871.g6a1bdf85.el9.x86_64 5/32 2024-09-21T09:45:37.081 INFO:teuthology.orchestra.run.smithi070.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 914 kB/s | 107 kB 00:00 2024-09-21T09:45:37.115 INFO:teuthology.orchestra.run.smithi070.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 451 kB/s | 45 kB 00:00 2024-09-21T09:45:37.128 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados2-2:18.2.4-871.g6a1bdf85.el9.x86_64 5/32 2024-09-21T09:45:37.140 INFO:teuthology.orchestra.run.smithi070.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-09-21T09:45:37.182 INFO:teuthology.orchestra.run.smithi070.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 662 kB 00:00 2024-09-21T09:45:37.213 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rados-2:18.2.4-871.g6a1bdf85.el9.x86_64 6/32 2024-09-21T09:45:37.216 INFO:teuthology.orchestra.run.smithi070.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-21T09:45:37.241 INFO:teuthology.orchestra.run.smithi070.stdout:(19/30): python3-prettytable-0.7.2-27.el9.noarc 420 kB/s | 42 kB 00:00 2024-09-21T09:45:37.254 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephfs2-2:18.2.4-871.g6a1bdf85.el9.x86_64 7/32 2024-09-21T09:45:37.283 INFO:teuthology.orchestra.run.smithi070.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 1.1 MB/s | 111 kB 00:00 2024-09-21T09:45:37.300 INFO:teuthology.orchestra.run.smithi070.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.7 MB/s | 26 kB 00:00 2024-09-21T09:45:37.309 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephfs2-2:18.2.4-871.g6a1bdf85.el9.x86_64 7/32 2024-09-21T09:45:37.317 INFO:teuthology.orchestra.run.smithi070.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 2.8 MB/s | 49 kB 00:00 2024-09-21T09:45:37.342 INFO:teuthology.orchestra.run.smithi070.stdout:(23/30): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.4 MB/s | 308 kB 00:00 2024-09-21T09:45:37.346 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-09-21T09:45:37.367 INFO:teuthology.orchestra.run.smithi070.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-21T09:45:37.389 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-ceph-argparse-2:18.2.4-871.g6a1bdf85.el9.x 9/32 2024-09-21T09:45:37.401 INFO:teuthology.orchestra.run.smithi070.stdout:(25/30): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-21T09:45:37.435 INFO:teuthology.orchestra.run.smithi070.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-09-21T09:45:37.455 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cephfs-2:18.2.4-871.g6a1bdf85.el9.x86_64 10/32 2024-09-21T09:45:37.490 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libradosstriper1-2:18.2.4-871.g6a1bdf85.el9.x86_64 11/32 2024-09-21T09:45:37.493 INFO:teuthology.orchestra.run.smithi070.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-09-21T09:45:37.556 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libradosstriper1-2:18.2.4-871.g6a1bdf85.el9.x86_64 11/32 2024-09-21T09:45:37.577 INFO:teuthology.orchestra.run.smithi070.stdout:(28/30): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-09-21T09:45:37.635 INFO:teuthology.orchestra.run.smithi070.stdout:(29/30): librados2-18.2.4-871.g6a1bdf85.el9.x86 17 MB/s | 3.3 MB 00:00 2024-09-21T09:45:37.661 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/32 2024-09-21T09:45:37.697 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-09-21T09:45:37.719 INFO:teuthology.orchestra.run.smithi070.stdout:(30/30): librbd1-18.2.4-871.g6a1bdf85.el9.x86_6 13 MB/s | 3.0 MB 00:00 2024-09-21T09:45:37.724 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:45:37.724 INFO:teuthology.orchestra.run.smithi070.stdout:Total 22 MB/s | 53 MB 00:02 2024-09-21T09:45:37.737 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-09-21T09:45:37.778 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/32 2024-09-21T09:45:37.795 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-21T09:45:37.805 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-09-21T09:45:37.830 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-21T09:45:37.830 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-21T09:45:37.850 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-09-21T09:45:37.891 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-09-21T09:45:38.195 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-09-21T09:45:38.237 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : librbd1-2:18.2.4-871.g6a1bdf85.el9.x86_64 20/32 2024-09-21T09:45:38.270 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-21T09:45:38.271 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-21T09:45:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librbd1-2:18.2.4-871.g6a1bdf85.el9.x86_64 20/32 2024-09-21T09:45:38.359 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rbd-2:18.2.4-871.g6a1bdf85.el9.x86_64 21/32 2024-09-21T09:45:38.395 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-09-21T09:45:38.799 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-09-21T09:45:38.823 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librgw2-2:18.2.4-871.g6a1bdf85.el9.x86_64 24/32 2024-09-21T09:45:38.894 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librgw2-2:18.2.4-871.g6a1bdf85.el9.x86_64 24/32 2024-09-21T09:45:38.971 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rgw-2:18.2.4-871.g6a1bdf85.el9.x86_64 25/32 2024-09-21T09:45:39.001 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-ceph-common-2:18.2.4-871.g6a1bdf85.el9.x86 26/32 2024-09-21T09:45:39.649 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-21T09:45:39.739 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-09-21T09:45:39.774 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-09-21T09:45:39.824 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-09-21T09:45:40.196 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-09-21T09:45:40.256 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : librados2-2:18.2.4-871.g6a1bdf85.el9.x86_64 5/32 2024-09-21T09:45:40.353 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados2-2:18.2.4-871.g6a1bdf85.el9.x86_64 5/32 2024-09-21T09:45:40.441 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rados-2:18.2.4-871.g6a1bdf85.el9.x86_64 6/32 2024-09-21T09:45:40.486 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephfs2-2:18.2.4-871.g6a1bdf85.el9.x86_64 7/32 2024-09-21T09:45:40.548 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephfs2-2:18.2.4-871.g6a1bdf85.el9.x86_64 7/32 2024-09-21T09:45:40.585 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-09-21T09:45:40.629 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ceph-argparse-2:18.2.4-871.g6a1bdf85.el9.x 9/32 2024-09-21T09:45:40.696 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cephfs-2:18.2.4-871.g6a1bdf85.el9.x86_64 10/32 2024-09-21T09:45:40.721 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libradosstriper1-2:18.2.4-871.g6a1bdf85.el9.x86_64 11/32 2024-09-21T09:45:40.771 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libradosstriper1-2:18.2.4-871.g6a1bdf85.el9.x86_64 11/32 2024-09-21T09:45:40.876 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/32 2024-09-21T09:45:40.904 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-09-21T09:45:40.944 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-09-21T09:45:40.977 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/32 2024-09-21T09:45:40.996 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-09-21T09:45:41.032 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-09-21T09:45:41.065 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-09-21T09:45:41.087 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-common-2:18.2.4-871.g6a1bdf85.el9.x86_64 27/32 2024-09-21T09:45:41.122 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-common-2:18.2.4-871.g6a1bdf85.el9.x86_64 27/32 2024-09-21T09:45:41.373 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-09-21T09:45:41.419 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : librbd1-2:18.2.4-871.g6a1bdf85.el9.x86_64 20/32 2024-09-21T09:45:41.510 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librbd1-2:18.2.4-871.g6a1bdf85.el9.x86_64 20/32 2024-09-21T09:45:41.550 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rbd-2:18.2.4-871.g6a1bdf85.el9.x86_64 21/32 2024-09-21T09:45:41.593 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-09-21T09:45:41.663 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-common-2:18.2.4-871.g6a1bdf85.el9.x86_64 27/32 2024-09-21T09:45:41.700 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-base-2:18.2.4-871.g6a1bdf85.el9.x86_64 28/32 2024-09-21T09:45:41.804 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-base-2:18.2.4-871.g6a1bdf85.el9.x86_64 28/32 2024-09-21T09:45:41.804 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T09:45:41.805 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T09:45:41.805 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:45:41.843 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-selinux-2:18.2.4-871.g6a1bdf85.el9.x86_64 29/32 2024-09-21T09:45:42.002 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-09-21T09:45:42.030 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librgw2-2:18.2.4-871.g6a1bdf85.el9.x86_64 24/32 2024-09-21T09:45:42.086 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librgw2-2:18.2.4-871.g6a1bdf85.el9.x86_64 24/32 2024-09-21T09:45:42.161 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rgw-2:18.2.4-871.g6a1bdf85.el9.x86_64 25/32 2024-09-21T09:45:42.191 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ceph-common-2:18.2.4-871.g6a1bdf85.el9.x86 26/32 2024-09-21T09:45:44.309 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-common-2:18.2.4-871.g6a1bdf85.el9.x86_64 27/32 2024-09-21T09:45:44.337 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-common-2:18.2.4-871.g6a1bdf85.el9.x86_64 27/32 2024-09-21T09:45:44.891 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-common-2:18.2.4-871.g6a1bdf85.el9.x86_64 27/32 2024-09-21T09:45:44.924 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-base-2:18.2.4-871.g6a1bdf85.el9.x86_64 28/32 2024-09-21T09:45:45.018 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-base-2:18.2.4-871.g6a1bdf85.el9.x86_64 28/32 2024-09-21T09:45:45.018 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T09:45:45.018 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T09:45:45.018 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:45:45.042 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-selinux-2:18.2.4-871.g6a1bdf85.el9.x86_64 29/32 2024-09-21T09:45:51.796 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-selinux-2:18.2.4-871.g6a1bdf85.el9.x86_64 29/32 2024-09-21T09:45:51.796 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /sys 2024-09-21T09:45:51.796 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /proc 2024-09-21T09:45:51.796 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /mnt 2024-09-21T09:45:51.796 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /var/tmp 2024-09-21T09:45:51.796 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /home 2024-09-21T09:45:51.796 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /root 2024-09-21T09:45:51.796 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /tmp 2024-09-21T09:45:51.796 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:45:52.565 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-radosgw-2:18.2.4-871.g6a1bdf85.el9.x86_64 30/32 2024-09-21T09:45:52.591 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-871.g6a1bdf85.el9.x86_64 30/32 2024-09-21T09:45:52.591 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:45:52.591 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T09:45:52.591 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T09:45:52.591 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T09:45:52.591 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:45:52.593 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-21T09:45:52.685 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-21T09:45:52.685 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-21T09:45:53.863 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-21T09:45:53.863 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-base-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/32 2024-09-21T09:45:53.863 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-common-2:18.2.4-871.g6a1bdf85.el9.x86_64 2/32 2024-09-21T09:45:53.863 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-radosgw-2:18.2.4-871.g6a1bdf85.el9.x86_64 3/32 2024-09-21T09:45:53.863 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-selinux-2:18.2.4-871.g6a1bdf85.el9.x86_64 4/32 2024-09-21T09:45:53.863 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephfs2-2:18.2.4-871.g6a1bdf85.el9.x86_64 5/32 2024-09-21T09:45:53.863 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libradosstriper1-2:18.2.4-871.g6a1bdf85.el9.x86_64 6/32 2024-09-21T09:45:53.863 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librgw2-2:18.2.4-871.g6a1bdf85.el9.x86_64 7/32 2024-09-21T09:45:53.863 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-ceph-argparse-2:18.2.4-871.g6a1bdf85.el9.x 8/32 2024-09-21T09:45:53.863 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-ceph-common-2:18.2.4-871.g6a1bdf85.el9.x86 9/32 2024-09-21T09:45:53.863 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cephfs-2:18.2.4-871.g6a1bdf85.el9.x86_64 10/32 2024-09-21T09:45:53.864 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rados-2:18.2.4-871.g6a1bdf85.el9.x86_64 11/32 2024-09-21T09:45:53.864 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rbd-2:18.2.4-871.g6a1bdf85.el9.x86_64 12/32 2024-09-21T09:45:53.864 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rgw-2:18.2.4-871.g6a1bdf85.el9.x86_64 13/32 2024-09-21T09:45:53.864 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-09-21T09:45:53.864 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-09-21T09:45:53.864 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-09-21T09:45:53.864 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-09-21T09:45:53.864 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-09-21T09:45:53.864 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-09-21T09:45:53.864 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-09-21T09:45:53.864 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 21/32 2024-09-21T09:45:53.864 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-09-21T09:45:53.864 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-09-21T09:45:53.865 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-09-21T09:45:53.865 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-09-21T09:45:53.865 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-09-21T09:45:53.865 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : re2-1:20211101-20.el9.x86_64 27/32 2024-09-21T09:45:53.865 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-09-21T09:45:53.865 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados2-2:18.2.4-871.g6a1bdf85.el9.x86_64 29/32 2024-09-21T09:45:53.865 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-09-21T09:45:53.865 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librbd1-2:18.2.4-871.g6a1bdf85.el9.x86_64 31/32 2024-09-21T09:45:54.700 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-09-21T09:45:54.700 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:45:54.700 INFO:teuthology.orchestra.run.smithi160.stdout:Upgraded: 2024-09-21T09:45:54.700 INFO:teuthology.orchestra.run.smithi160.stdout: librados2-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:54.701 INFO:teuthology.orchestra.run.smithi160.stdout: librbd1-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:54.701 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-21T09:45:54.701 INFO:teuthology.orchestra.run.smithi160.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T09:45:54.701 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-base-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:54.701 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-common-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:54.701 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-radosgw-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:54.701 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-selinux-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:54.701 INFO:teuthology.orchestra.run.smithi160.stdout: fmt-8.1.1-5.el9.x86_64 2024-09-21T09:45:54.701 INFO:teuthology.orchestra.run.smithi160.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T09:45:54.701 INFO:teuthology.orchestra.run.smithi160.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T09:45:54.701 INFO:teuthology.orchestra.run.smithi160.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T09:45:54.701 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs2-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:54.701 INFO:teuthology.orchestra.run.smithi160.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T09:45:54.701 INFO:teuthology.orchestra.run.smithi160.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T09:45:54.702 INFO:teuthology.orchestra.run.smithi160.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T09:45:54.702 INFO:teuthology.orchestra.run.smithi160.stdout: libradosstriper1-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:54.702 INFO:teuthology.orchestra.run.smithi160.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T09:45:54.702 INFO:teuthology.orchestra.run.smithi160.stdout: librgw2-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:54.702 INFO:teuthology.orchestra.run.smithi160.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T09:45:54.702 INFO:teuthology.orchestra.run.smithi160.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T09:45:54.702 INFO:teuthology.orchestra.run.smithi160.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T09:45:54.702 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-argparse-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:54.702 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-common-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:54.702 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cephfs-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:54.702 INFO:teuthology.orchestra.run.smithi160.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T09:45:54.702 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rados-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:54.702 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rbd-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:54.703 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rgw-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:54.703 INFO:teuthology.orchestra.run.smithi160.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T09:45:54.703 INFO:teuthology.orchestra.run.smithi160.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T09:45:54.703 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:45:54.703 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T09:45:54.859 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-selinux-2:18.2.4-871.g6a1bdf85.el9.x86_64 29/32 2024-09-21T09:45:54.860 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /sys 2024-09-21T09:45:54.860 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /proc 2024-09-21T09:45:54.860 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /mnt 2024-09-21T09:45:54.860 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /var/tmp 2024-09-21T09:45:54.860 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /home 2024-09-21T09:45:54.860 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /root 2024-09-21T09:45:54.860 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /tmp 2024-09-21T09:45:54.860 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:45:55.104 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-test 2024-09-21T09:45:55.624 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-radosgw-2:18.2.4-871.g6a1bdf85.el9.x86_64 30/32 2024-09-21T09:45:55.650 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-871.g6a1bdf85.el9.x86_64 30/32 2024-09-21T09:45:55.650 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:45:55.650 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T09:45:55.650 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T09:45:55.650 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T09:45:55.650 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:45:55.651 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-21T09:45:55.684 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:26 ago on Sat 21 Sep 2024 09:45:29 AM UTC. 2024-09-21T09:45:55.735 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-21T09:45:55.735 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-21T09:45:55.815 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T09:45:55.815 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:45:55.815 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-21T09:45:55.815 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:45:55.816 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-21T09:45:55.816 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-test x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 40 M 2024-09-21T09:45:55.816 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-21T09:45:55.816 INFO:teuthology.orchestra.run.smithi160.stdout: libcephsqlite x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 166 k 2024-09-21T09:45:55.816 INFO:teuthology.orchestra.run.smithi160.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T09:45:55.816 INFO:teuthology.orchestra.run.smithi160.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T09:45:55.816 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:45:55.816 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-21T09:45:55.816 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:45:55.816 INFO:teuthology.orchestra.run.smithi160.stdout:Install 4 Packages 2024-09-21T09:45:55.816 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:45:55.817 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 40 M 2024-09-21T09:45:55.817 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 164 M 2024-09-21T09:45:55.817 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-21T09:45:55.977 INFO:teuthology.orchestra.run.smithi160.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.2 MB/s | 304 kB 00:00 2024-09-21T09:45:56.043 INFO:teuthology.orchestra.run.smithi160.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 961 kB/s | 64 kB 00:00 2024-09-21T09:45:56.068 INFO:teuthology.orchestra.run.smithi160.stdout:(3/4): libcephsqlite-18.2.4-871.g6a1bdf85.el9.x 724 kB/s | 166 kB 00:00 2024-09-21T09:45:56.874 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-21T09:45:56.874 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-base-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/32 2024-09-21T09:45:56.874 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-common-2:18.2.4-871.g6a1bdf85.el9.x86_64 2/32 2024-09-21T09:45:56.875 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-radosgw-2:18.2.4-871.g6a1bdf85.el9.x86_64 3/32 2024-09-21T09:45:56.875 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-selinux-2:18.2.4-871.g6a1bdf85.el9.x86_64 4/32 2024-09-21T09:45:56.875 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephfs2-2:18.2.4-871.g6a1bdf85.el9.x86_64 5/32 2024-09-21T09:45:56.875 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libradosstriper1-2:18.2.4-871.g6a1bdf85.el9.x86_64 6/32 2024-09-21T09:45:56.875 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librgw2-2:18.2.4-871.g6a1bdf85.el9.x86_64 7/32 2024-09-21T09:45:56.875 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ceph-argparse-2:18.2.4-871.g6a1bdf85.el9.x 8/32 2024-09-21T09:45:56.875 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ceph-common-2:18.2.4-871.g6a1bdf85.el9.x86 9/32 2024-09-21T09:45:56.875 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cephfs-2:18.2.4-871.g6a1bdf85.el9.x86_64 10/32 2024-09-21T09:45:56.875 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rados-2:18.2.4-871.g6a1bdf85.el9.x86_64 11/32 2024-09-21T09:45:56.875 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rbd-2:18.2.4-871.g6a1bdf85.el9.x86_64 12/32 2024-09-21T09:45:56.875 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rgw-2:18.2.4-871.g6a1bdf85.el9.x86_64 13/32 2024-09-21T09:45:56.875 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-09-21T09:45:56.875 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-09-21T09:45:56.876 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-09-21T09:45:56.876 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-09-21T09:45:56.876 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-09-21T09:45:56.876 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-09-21T09:45:56.876 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-09-21T09:45:56.876 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 21/32 2024-09-21T09:45:56.876 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-09-21T09:45:56.876 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-09-21T09:45:56.876 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-09-21T09:45:56.876 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-09-21T09:45:56.876 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-09-21T09:45:56.876 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : re2-1:20211101-20.el9.x86_64 27/32 2024-09-21T09:45:56.876 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-09-21T09:45:56.877 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados2-2:18.2.4-871.g6a1bdf85.el9.x86_64 29/32 2024-09-21T09:45:56.877 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-09-21T09:45:56.877 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librbd1-2:18.2.4-871.g6a1bdf85.el9.x86_64 31/32 2024-09-21T09:45:57.094 INFO:teuthology.orchestra.run.smithi160.stdout:(4/4): ceph-test-18.2.4-871.g6a1bdf85.el9.x86_6 32 MB/s | 40 MB 00:01 2024-09-21T09:45:57.096 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:45:57.096 INFO:teuthology.orchestra.run.smithi160.stdout:Total 31 MB/s | 40 MB 00:01 2024-09-21T09:45:57.123 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-21T09:45:57.143 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-21T09:45:57.143 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-21T09:45:57.462 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-21T09:45:57.463 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-21T09:45:57.903 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-09-21T09:45:57.903 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:45:57.903 INFO:teuthology.orchestra.run.smithi070.stdout:Upgraded: 2024-09-21T09:45:57.903 INFO:teuthology.orchestra.run.smithi070.stdout: librados2-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:57.903 INFO:teuthology.orchestra.run.smithi070.stdout: librbd1-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:57.903 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-21T09:45:57.903 INFO:teuthology.orchestra.run.smithi070.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T09:45:57.903 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:57.903 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-common-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:57.904 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-radosgw-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:57.904 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-selinux-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:57.904 INFO:teuthology.orchestra.run.smithi070.stdout: fmt-8.1.1-5.el9.x86_64 2024-09-21T09:45:57.904 INFO:teuthology.orchestra.run.smithi070.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T09:45:57.904 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T09:45:57.904 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T09:45:57.904 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs2-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:57.904 INFO:teuthology.orchestra.run.smithi070.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T09:45:57.904 INFO:teuthology.orchestra.run.smithi070.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T09:45:57.904 INFO:teuthology.orchestra.run.smithi070.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T09:45:57.904 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:57.904 INFO:teuthology.orchestra.run.smithi070.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T09:45:57.904 INFO:teuthology.orchestra.run.smithi070.stdout: librgw2-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:57.904 INFO:teuthology.orchestra.run.smithi070.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T09:45:57.904 INFO:teuthology.orchestra.run.smithi070.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T09:45:57.905 INFO:teuthology.orchestra.run.smithi070.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T09:45:57.905 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-argparse-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:57.905 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-common-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:57.905 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cephfs-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:57.905 INFO:teuthology.orchestra.run.smithi070.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T09:45:57.905 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rados-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:57.905 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rbd-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:57.905 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rgw-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:45:57.905 INFO:teuthology.orchestra.run.smithi070.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T09:45:57.905 INFO:teuthology.orchestra.run.smithi070.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T09:45:57.905 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:45:57.905 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-21T09:45:57.934 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-21T09:45:58.021 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-21T09:45:58.068 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-21T09:45:58.105 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephsqlite-2:18.2.4-871.g6a1bdf85.el9.x86_64 3/4 2024-09-21T09:45:58.262 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-test 2024-09-21T09:45:58.833 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:00:25 ago on Sat 21 Sep 2024 09:45:33 AM UTC. 2024-09-21T09:45:58.963 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-21T09:45:58.964 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:45:58.964 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-21T09:45:58.964 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:45:58.964 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-21T09:45:58.965 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-test x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 40 M 2024-09-21T09:45:58.965 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-21T09:45:58.965 INFO:teuthology.orchestra.run.smithi070.stdout: libcephsqlite x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 166 k 2024-09-21T09:45:58.965 INFO:teuthology.orchestra.run.smithi070.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T09:45:58.965 INFO:teuthology.orchestra.run.smithi070.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T09:45:58.965 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:45:58.965 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-21T09:45:58.965 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:45:58.965 INFO:teuthology.orchestra.run.smithi070.stdout:Install 4 Packages 2024-09-21T09:45:58.965 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:45:58.966 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 40 M 2024-09-21T09:45:58.966 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 164 M 2024-09-21T09:45:58.966 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-21T09:45:59.219 INFO:teuthology.orchestra.run.smithi070.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.4 MB/s | 304 kB 00:00 2024-09-21T09:45:59.244 INFO:teuthology.orchestra.run.smithi070.stdout:(2/4): libcephsqlite-18.2.4-871.g6a1bdf85.el9.x 715 kB/s | 166 kB 00:00 2024-09-21T09:45:59.353 INFO:teuthology.orchestra.run.smithi070.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 476 kB/s | 64 kB 00:00 2024-09-21T09:46:00.304 INFO:teuthology.orchestra.run.smithi070.stdout:(4/4): ceph-test-18.2.4-871.g6a1bdf85.el9.x86_6 31 MB/s | 40 MB 00:01 2024-09-21T09:46:00.306 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:46:00.306 INFO:teuthology.orchestra.run.smithi070.stdout:Total 30 MB/s | 40 MB 00:01 2024-09-21T09:46:00.333 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-21T09:46:00.353 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-21T09:46:00.353 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-21T09:46:00.674 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-21T09:46:00.674 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-21T09:46:01.174 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-21T09:46:01.247 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-21T09:46:01.295 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-21T09:46:01.322 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephsqlite-2:18.2.4-871.g6a1bdf85.el9.x86_64 3/4 2024-09-21T09:46:01.801 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephsqlite-2:18.2.4-871.g6a1bdf85.el9.x86_64 3/4 2024-09-21T09:46:01.878 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-test-2:18.2.4-871.g6a1bdf85.el9.x86_64 4/4 2024-09-21T09:46:03.896 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-test-2:18.2.4-871.g6a1bdf85.el9.x86_64 4/4 2024-09-21T09:46:03.896 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-test-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/4 2024-09-21T09:46:03.896 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephsqlite-2:18.2.4-871.g6a1bdf85.el9.x86_64 2/4 2024-09-21T09:46:03.896 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-21T09:46:04.242 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-21T09:46:04.242 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:04.242 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-21T09:46:04.242 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-test-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:46:04.242 INFO:teuthology.orchestra.run.smithi160.stdout: libcephsqlite-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:46:04.242 INFO:teuthology.orchestra.run.smithi160.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T09:46:04.242 INFO:teuthology.orchestra.run.smithi160.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T09:46:04.242 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:04.243 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T09:46:04.543 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph 2024-09-21T09:46:05.038 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephsqlite-2:18.2.4-871.g6a1bdf85.el9.x86_64 3/4 2024-09-21T09:46:05.111 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-test-2:18.2.4-871.g6a1bdf85.el9.x86_64 4/4 2024-09-21T09:46:05.129 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:36 ago on Sat 21 Sep 2024 09:45:29 AM UTC. 2024-09-21T09:46:05.267 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T09:46:05.269 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:46:05.269 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-21T09:46:05.269 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:46:05.269 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-21T09:46:05.269 INFO:teuthology.orchestra.run.smithi160.stdout: ceph x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 6.5 k 2024-09-21T09:46:05.269 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-21T09:46:05.270 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mds x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 2.1 M 2024-09-21T09:46:05.270 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 1.5 M 2024-09-21T09:46:05.270 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-modules-core noarch 2:18.2.4-871.g6a1bdf85.el9 ceph-noarch 246 k 2024-09-21T09:46:05.270 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mon x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 4.7 M 2024-09-21T09:46:05.270 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-osd x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 17 M 2024-09-21T09:46:05.270 INFO:teuthology.orchestra.run.smithi160.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T09:46:05.270 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T09:46:05.270 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T09:46:05.270 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T09:46:05.270 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T09:46:05.270 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T09:46:05.270 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T09:46:05.271 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-21T09:46:05.271 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T09:46:05.271 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T09:46:05.271 INFO:teuthology.orchestra.run.smithi160.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T09:46:05.271 INFO:teuthology.orchestra.run.smithi160.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T09:46:05.271 INFO:teuthology.orchestra.run.smithi160.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T09:46:05.271 INFO:teuthology.orchestra.run.smithi160.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T09:46:05.271 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T09:46:05.271 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T09:46:05.271 INFO:teuthology.orchestra.run.smithi160.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T09:46:05.271 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T09:46:05.271 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T09:46:05.271 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T09:46:05.271 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T09:46:05.271 INFO:teuthology.orchestra.run.smithi160.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T09:46:05.272 INFO:teuthology.orchestra.run.smithi160.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T09:46:05.272 INFO:teuthology.orchestra.run.smithi160.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T09:46:05.272 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T09:46:05.272 INFO:teuthology.orchestra.run.smithi160.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T09:46:05.272 INFO:teuthology.orchestra.run.smithi160.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T09:46:05.272 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:05.272 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-21T09:46:05.272 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:46:05.272 INFO:teuthology.orchestra.run.smithi160.stdout:Install 33 Packages 2024-09-21T09:46:05.272 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:05.273 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 29 M 2024-09-21T09:46:05.273 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 104 M 2024-09-21T09:46:05.273 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-21T09:46:06.278 INFO:teuthology.orchestra.run.smithi160.stdout:(1/33): ceph-18.2.4-871.g6a1bdf85.el9.x86_64.rp 51 kB/s | 6.5 kB 00:00 2024-09-21T09:46:06.504 INFO:teuthology.orchestra.run.smithi160.stdout:(2/33): ceph-mds-18.2.4-871.g6a1bdf85.el9.x86_6 5.9 MB/s | 2.1 MB 00:00 2024-09-21T09:46:06.537 INFO:teuthology.orchestra.run.smithi160.stdout:(3/33): ceph-mgr-18.2.4-871.g6a1bdf85.el9.x86_6 3.8 MB/s | 1.5 MB 00:00 2024-09-21T09:46:06.638 INFO:teuthology.orchestra.run.smithi160.stdout:(4/33): ceph-mgr-modules-core-18.2.4-871.g6a1bd 2.4 MB/s | 246 kB 00:00 2024-09-21T09:46:06.771 INFO:teuthology.orchestra.run.smithi160.stdout:(5/33): ceph-mon-18.2.4-871.g6a1bdf85.el9.x86_6 9.5 MB/s | 4.7 MB 00:00 2024-09-21T09:46:07.013 INFO:teuthology.orchestra.run.smithi160.stdout:(6/33): ceph-osd-18.2.4-871.g6a1bdf85.el9.x86_6 32 MB/s | 17 MB 00:00 2024-09-21T09:46:07.038 INFO:teuthology.orchestra.run.smithi160.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 633 kB/s | 253 kB 00:00 2024-09-21T09:46:07.155 INFO:teuthology.orchestra.run.smithi160.stdout:(8/33): python3-ply-3.11-14.el9.noarch.rpm 753 kB/s | 106 kB 00:00 2024-09-21T09:46:07.169 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-test-2:18.2.4-871.g6a1bdf85.el9.x86_64 4/4 2024-09-21T09:46:07.170 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-test-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/4 2024-09-21T09:46:07.170 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephsqlite-2:18.2.4-871.g6a1bdf85.el9.x86_64 2/4 2024-09-21T09:46:07.170 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-21T09:46:07.180 INFO:teuthology.orchestra.run.smithi160.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 954 kB/s | 135 kB 00:00 2024-09-21T09:46:07.222 INFO:teuthology.orchestra.run.smithi160.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 1.8 MB/s | 125 kB 00:00 2024-09-21T09:46:07.331 INFO:teuthology.orchestra.run.smithi160.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 2.2 MB/s | 1.2 MB 00:00 2024-09-21T09:46:07.356 INFO:teuthology.orchestra.run.smithi160.stdout:(12/33): python3-urllib3-1.26.5-6.el9.noarch.rp 1.2 MB/s | 215 kB 00:00 2024-09-21T09:46:07.423 INFO:teuthology.orchestra.run.smithi160.stdout:(13/33): python3-mako-1.1.4-6.el9.noarch.rpm 859 kB/s | 172 kB 00:00 2024-09-21T09:46:07.440 INFO:teuthology.orchestra.run.smithi160.stdout:(14/33): python3-markupsafe-1.1.1-12.el9.x86_64 320 kB/s | 35 kB 00:00 2024-09-21T09:46:07.465 INFO:teuthology.orchestra.run.smithi160.stdout:(15/33): python3-toml-0.10.2-6.el9.noarch.rpm 999 kB/s | 42 kB 00:00 2024-09-21T09:46:07.470 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-21T09:46:07.470 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:07.471 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-21T09:46:07.471 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-test-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:46:07.471 INFO:teuthology.orchestra.run.smithi070.stdout: libcephsqlite-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:46:07.471 INFO:teuthology.orchestra.run.smithi070.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T09:46:07.471 INFO:teuthology.orchestra.run.smithi070.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T09:46:07.471 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:07.471 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-21T09:46:07.490 INFO:teuthology.orchestra.run.smithi160.stdout:(16/33): python3-pytz-2021.1-5.el9.noarch.rpm 381 kB/s | 51 kB 00:00 2024-09-21T09:46:07.507 INFO:teuthology.orchestra.run.smithi160.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 647 kB/s | 43 kB 00:00 2024-09-21T09:46:07.532 INFO:teuthology.orchestra.run.smithi160.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 611 kB/s | 11 kB 00:00 2024-09-21T09:46:07.558 INFO:teuthology.orchestra.run.smithi160.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.9 MB/s | 172 kB 00:00 2024-09-21T09:46:07.583 INFO:teuthology.orchestra.run.smithi160.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 365 kB/s | 18 kB 00:00 2024-09-21T09:46:07.608 INFO:teuthology.orchestra.run.smithi160.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 940 kB/s | 23 kB 00:00 2024-09-21T09:46:07.633 INFO:teuthology.orchestra.run.smithi160.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 390 kB/s | 19 kB 00:00 2024-09-21T09:46:07.659 INFO:teuthology.orchestra.run.smithi160.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-21T09:46:07.684 INFO:teuthology.orchestra.run.smithi160.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 921 kB/s | 46 kB 00:00 2024-09-21T09:46:07.709 INFO:teuthology.orchestra.run.smithi160.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-21T09:46:07.735 INFO:teuthology.orchestra.run.smithi160.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 657 kB/s | 16 kB 00:00 2024-09-21T09:46:07.768 INFO:teuthology.orchestra.run.smithi160.stdout:(27/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-21T09:46:07.779 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph 2024-09-21T09:46:07.794 INFO:teuthology.orchestra.run.smithi160.stdout:(28/33): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-21T09:46:07.819 INFO:teuthology.orchestra.run.smithi160.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2024-09-21T09:46:07.852 INFO:teuthology.orchestra.run.smithi160.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-09-21T09:46:07.878 INFO:teuthology.orchestra.run.smithi160.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 794 kB/s | 20 kB 00:00 2024-09-21T09:46:07.903 INFO:teuthology.orchestra.run.smithi160.stdout:(32/33): python3-webob-1.8.8-2.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-09-21T09:46:07.937 INFO:teuthology.orchestra.run.smithi160.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.5 MB/s | 427 kB 00:00 2024-09-21T09:46:07.942 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:46:07.942 INFO:teuthology.orchestra.run.smithi160.stdout:Total 11 MB/s | 29 MB 00:02 2024-09-21T09:46:08.068 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-21T09:46:08.106 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-21T09:46:08.106 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-21T09:46:08.355 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:00:35 ago on Sat 21 Sep 2024 09:45:33 AM UTC. 2024-09-21T09:46:08.489 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-21T09:46:08.491 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:46:08.491 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-21T09:46:08.491 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:46:08.491 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-21T09:46:08.492 INFO:teuthology.orchestra.run.smithi070.stdout: ceph x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 6.5 k 2024-09-21T09:46:08.492 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-21T09:46:08.492 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mds x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 2.1 M 2024-09-21T09:46:08.492 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 1.5 M 2024-09-21T09:46:08.492 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core noarch 2:18.2.4-871.g6a1bdf85.el9 ceph-noarch 246 k 2024-09-21T09:46:08.492 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mon x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 4.7 M 2024-09-21T09:46:08.492 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-osd x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 17 M 2024-09-21T09:46:08.492 INFO:teuthology.orchestra.run.smithi070.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T09:46:08.492 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T09:46:08.492 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T09:46:08.492 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T09:46:08.492 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T09:46:08.492 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T09:46:08.492 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T09:46:08.493 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-21T09:46:08.493 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T09:46:08.493 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T09:46:08.493 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T09:46:08.493 INFO:teuthology.orchestra.run.smithi070.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T09:46:08.493 INFO:teuthology.orchestra.run.smithi070.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T09:46:08.493 INFO:teuthology.orchestra.run.smithi070.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T09:46:08.493 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T09:46:08.493 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T09:46:08.493 INFO:teuthology.orchestra.run.smithi070.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T09:46:08.493 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T09:46:08.493 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T09:46:08.493 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T09:46:08.493 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T09:46:08.494 INFO:teuthology.orchestra.run.smithi070.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T09:46:08.494 INFO:teuthology.orchestra.run.smithi070.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T09:46:08.494 INFO:teuthology.orchestra.run.smithi070.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T09:46:08.494 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T09:46:08.494 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T09:46:08.494 INFO:teuthology.orchestra.run.smithi070.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T09:46:08.494 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:08.494 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-21T09:46:08.494 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:46:08.494 INFO:teuthology.orchestra.run.smithi070.stdout:Install 33 Packages 2024-09-21T09:46:08.494 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:08.495 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 29 M 2024-09-21T09:46:08.495 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 104 M 2024-09-21T09:46:08.495 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-21T09:46:08.504 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-21T09:46:08.504 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-21T09:46:09.241 INFO:teuthology.orchestra.run.smithi070.stdout:(1/33): ceph-18.2.4-871.g6a1bdf85.el9.x86_64.rp 49 kB/s | 6.5 kB 00:00 2024-09-21T09:46:09.425 INFO:teuthology.orchestra.run.smithi070.stdout:(2/33): ceph-mgr-18.2.4-871.g6a1bdf85.el9.x86_6 4.6 MB/s | 1.5 MB 00:00 2024-09-21T09:46:09.475 INFO:teuthology.orchestra.run.smithi070.stdout:(3/33): ceph-mds-18.2.4-871.g6a1bdf85.el9.x86_6 5.7 MB/s | 2.1 MB 00:00 2024-09-21T09:46:09.542 INFO:teuthology.orchestra.run.smithi070.stdout:(4/33): ceph-mgr-modules-core-18.2.4-871.g6a1bd 3.6 MB/s | 246 kB 00:00 2024-09-21T09:46:09.717 INFO:teuthology.orchestra.run.smithi070.stdout:(5/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-09-21T09:46:09.727 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-21T09:46:09.806 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-09-21T09:46:09.838 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-09-21T09:46:09.871 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-09-21T09:46:09.962 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-09-21T09:46:09.984 INFO:teuthology.orchestra.run.smithi070.stdout:(6/33): ceph-osd-18.2.4-871.g6a1bdf85.el9.x86_6 30 MB/s | 17 MB 00:00 2024-09-21T09:46:10.018 INFO:teuthology.orchestra.run.smithi070.stdout:(7/33): python3-cryptography-36.0.1-4.el9.x86_6 4.1 MB/s | 1.2 MB 00:00 2024-09-21T09:46:10.049 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-09-21T09:46:10.099 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/33 2024-09-21T09:46:10.101 INFO:teuthology.orchestra.run.smithi070.stdout:(8/33): ceph-mon-18.2.4-871.g6a1bdf85.el9.x86_6 5.4 MB/s | 4.7 MB 00:00 2024-09-21T09:46:10.127 INFO:teuthology.orchestra.run.smithi070.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 1.2 MB/s | 135 kB 00:00 2024-09-21T09:46:10.134 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-09-21T09:46:10.152 INFO:teuthology.orchestra.run.smithi070.stdout:(10/33): python3-ply-3.11-14.el9.noarch.rpm 636 kB/s | 106 kB 00:00 2024-09-21T09:46:10.172 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-09-21T09:46:10.177 INFO:teuthology.orchestra.run.smithi070.stdout:(11/33): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-21T09:46:10.207 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-09-21T09:46:10.244 INFO:teuthology.orchestra.run.smithi070.stdout:(12/33): python3-requests-2.25.1-8.el9.noarch.r 877 kB/s | 125 kB 00:00 2024-09-21T09:46:10.256 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-09-21T09:46:10.278 INFO:teuthology.orchestra.run.smithi070.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 345 kB/s | 35 kB 00:00 2024-09-21T09:46:10.297 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-09-21T09:46:10.312 INFO:teuthology.orchestra.run.smithi070.stdout:(14/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-09-21T09:46:10.340 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-09-21T09:46:10.354 INFO:teuthology.orchestra.run.smithi070.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 467 kB/s | 51 kB 00:00 2024-09-21T09:46:10.375 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-09-21T09:46:10.379 INFO:teuthology.orchestra.run.smithi070.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 650 kB/s | 43 kB 00:00 2024-09-21T09:46:10.404 INFO:teuthology.orchestra.run.smithi070.stdout:(17/33): python3-toml-0.10.2-6.el9.noarch.rpm 331 kB/s | 42 kB 00:00 2024-09-21T09:46:10.413 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-09-21T09:46:10.446 INFO:teuthology.orchestra.run.smithi070.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-09-21T09:46:10.471 INFO:teuthology.orchestra.run.smithi070.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 757 kB/s | 18 kB 00:00 2024-09-21T09:46:10.486 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-09-21T09:46:10.497 INFO:teuthology.orchestra.run.smithi070.stdout:(20/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 125 kB/s | 11 kB 00:00 2024-09-21T09:46:10.522 INFO:teuthology.orchestra.run.smithi070.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-09-21T09:46:10.547 INFO:teuthology.orchestra.run.smithi070.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-09-21T09:46:10.572 INFO:teuthology.orchestra.run.smithi070.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-21T09:46:10.598 INFO:teuthology.orchestra.run.smithi070.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 925 kB/s | 46 kB 00:00 2024-09-21T09:46:10.623 INFO:teuthology.orchestra.run.smithi070.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-21T09:46:10.624 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-09-21T09:46:10.657 INFO:teuthology.orchestra.run.smithi070.stdout:(26/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-21T09:46:10.682 INFO:teuthology.orchestra.run.smithi070.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-09-21T09:46:10.703 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-09-21T09:46:10.707 INFO:teuthology.orchestra.run.smithi070.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-21T09:46:10.733 INFO:teuthology.orchestra.run.smithi070.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 711 kB/s | 36 kB 00:00 2024-09-21T09:46:10.765 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-09-21T09:46:10.766 INFO:teuthology.orchestra.run.smithi070.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-09-21T09:46:10.791 INFO:teuthology.orchestra.run.smithi070.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 803 kB/s | 20 kB 00:00 2024-09-21T09:46:10.804 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-09-21T09:46:10.825 INFO:teuthology.orchestra.run.smithi070.stdout:(32/33): python3-webob-1.8.8-2.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-09-21T09:46:10.859 INFO:teuthology.orchestra.run.smithi070.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-09-21T09:46:10.861 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-09-21T09:46:10.862 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:46:10.863 INFO:teuthology.orchestra.run.smithi070.stdout:Total 12 MB/s | 29 MB 00:02 2024-09-21T09:46:10.917 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-09-21T09:46:10.977 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-21T09:46:11.015 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-21T09:46:11.016 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-21T09:46:11.110 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-09-21T09:46:11.160 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-09-21T09:46:11.238 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-09-21T09:46:11.326 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-09-21T09:46:11.367 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-09-21T09:46:11.435 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-21T09:46:11.435 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-21T09:46:11.483 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-09-21T09:46:11.662 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-871.g6a1bdf85.el9.n 28/33 2024-09-21T09:46:11.695 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-2:18.2.4-871.g6a1bdf85.el9.x86_64 29/33 2024-09-21T09:46:11.721 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-2:18.2.4-871.g6a1bdf85.el9.x86_64 29/33 2024-09-21T09:46:11.721 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:46:11.721 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T09:46:11.721 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T09:46:11.721 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T09:46:11.721 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:12.663 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-21T09:46:12.744 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-09-21T09:46:12.776 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-09-21T09:46:12.818 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-09-21T09:46:12.910 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-09-21T09:46:12.988 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-09-21T09:46:13.045 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/33 2024-09-21T09:46:13.089 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-09-21T09:46:13.119 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-09-21T09:46:13.154 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-09-21T09:46:13.179 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-osd-2:18.2.4-871.g6a1bdf85.el9.x86_64 30/33 2024-09-21T09:46:13.187 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-09-21T09:46:13.209 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-osd-2:18.2.4-871.g6a1bdf85.el9.x86_64 30/33 2024-09-21T09:46:13.209 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:46:13.209 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T09:46:13.209 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T09:46:13.209 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T09:46:13.210 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:13.227 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-09-21T09:46:13.262 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-09-21T09:46:13.315 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-09-21T09:46:13.352 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-09-21T09:46:13.425 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-09-21T09:46:13.571 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-09-21T09:46:13.652 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-09-21T09:46:13.672 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mon-2:18.2.4-871.g6a1bdf85.el9.x86_64 31/33 2024-09-21T09:46:13.696 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mon-2:18.2.4-871.g6a1bdf85.el9.x86_64 31/33 2024-09-21T09:46:13.697 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:46:13.697 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T09:46:13.697 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T09:46:13.697 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T09:46:13.697 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:13.705 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-09-21T09:46:13.752 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-09-21T09:46:13.809 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-09-21T09:46:13.873 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-09-21T09:46:13.898 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mds-2:18.2.4-871.g6a1bdf85.el9.x86_64 32/33 2024-09-21T09:46:13.922 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mds-2:18.2.4-871.g6a1bdf85.el9.x86_64 32/33 2024-09-21T09:46:13.922 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:46:13.922 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T09:46:13.922 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T09:46:13.922 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T09:46:13.922 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:13.986 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-2:18.2.4-871.g6a1bdf85.el9.x86_64 33/33 2024-09-21T09:46:14.059 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-09-21T09:46:14.098 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-09-21T09:46:14.176 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-09-21T09:46:14.264 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-09-21T09:46:14.305 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-09-21T09:46:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-09-21T09:46:14.597 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-871.g6a1bdf85.el9.n 28/33 2024-09-21T09:46:14.624 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-2:18.2.4-871.g6a1bdf85.el9.x86_64 29/33 2024-09-21T09:46:14.650 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-2:18.2.4-871.g6a1bdf85.el9.x86_64 29/33 2024-09-21T09:46:14.650 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:46:14.650 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T09:46:14.650 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T09:46:14.650 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T09:46:14.650 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:15.667 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-2:18.2.4-871.g6a1bdf85.el9.x86_64 33/33 2024-09-21T09:46:15.667 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/33 2024-09-21T09:46:15.667 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mds-2:18.2.4-871.g6a1bdf85.el9.x86_64 2/33 2024-09-21T09:46:15.667 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-2:18.2.4-871.g6a1bdf85.el9.x86_64 3/33 2024-09-21T09:46:15.667 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mon-2:18.2.4-871.g6a1bdf85.el9.x86_64 4/33 2024-09-21T09:46:15.667 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-osd-2:18.2.4-871.g6a1bdf85.el9.x86_64 5/33 2024-09-21T09:46:15.667 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-871.g6a1bdf85.el9.n 6/33 2024-09-21T09:46:15.667 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-09-21T09:46:15.667 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-09-21T09:46:15.667 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-09-21T09:46:15.667 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-09-21T09:46:15.668 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-09-21T09:46:15.668 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-09-21T09:46:15.668 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-09-21T09:46:15.668 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-09-21T09:46:15.668 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-09-21T09:46:15.668 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-09-21T09:46:15.668 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-09-21T09:46:15.668 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-09-21T09:46:15.668 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-09-21T09:46:15.668 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-09-21T09:46:15.668 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-09-21T09:46:15.668 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-09-21T09:46:15.668 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-09-21T09:46:15.668 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-09-21T09:46:15.669 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-09-21T09:46:15.669 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-09-21T09:46:15.669 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-09-21T09:46:15.669 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-09-21T09:46:15.669 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-09-21T09:46:15.669 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-09-21T09:46:15.669 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 31/33 2024-09-21T09:46:15.669 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-09-21T09:46:16.109 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-osd-2:18.2.4-871.g6a1bdf85.el9.x86_64 30/33 2024-09-21T09:46:16.140 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-osd-2:18.2.4-871.g6a1bdf85.el9.x86_64 30/33 2024-09-21T09:46:16.140 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:46:16.140 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T09:46:16.140 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T09:46:16.141 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T09:46:16.141 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:16.610 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mon-2:18.2.4-871.g6a1bdf85.el9.x86_64 31/33 2024-09-21T09:46:16.635 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mon-2:18.2.4-871.g6a1bdf85.el9.x86_64 31/33 2024-09-21T09:46:16.635 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:46:16.635 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T09:46:16.635 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T09:46:16.636 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T09:46:16.636 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:16.698 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-09-21T09:46:16.698 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:16.698 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-21T09:46:16.698 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:46:16.698 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mds-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:46:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:46:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-modules-core-2:18.2.4-871.g6a1bdf85.el9.noarch 2024-09-21T09:46:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mon-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:46:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-osd-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:46:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T09:46:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T09:46:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T09:46:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T09:46:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T09:46:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T09:46:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T09:46:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T09:46:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T09:46:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T09:46:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T09:46:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T09:46:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T09:46:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T09:46:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T09:46:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T09:46:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T09:46:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T09:46:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T09:46:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T09:46:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T09:46:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T09:46:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T09:46:16.701 INFO:teuthology.orchestra.run.smithi160.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T09:46:16.701 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T09:46:16.701 INFO:teuthology.orchestra.run.smithi160.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T09:46:16.701 INFO:teuthology.orchestra.run.smithi160.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T09:46:16.701 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:16.701 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T09:46:16.844 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mds-2:18.2.4-871.g6a1bdf85.el9.x86_64 32/33 2024-09-21T09:46:16.868 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mds-2:18.2.4-871.g6a1bdf85.el9.x86_64 32/33 2024-09-21T09:46:16.868 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:46:16.868 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T09:46:16.868 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T09:46:16.868 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T09:46:16.868 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:16.935 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-2:18.2.4-871.g6a1bdf85.el9.x86_64 33/33 2024-09-21T09:46:17.120 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-base 2024-09-21T09:46:17.715 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:48 ago on Sat 21 Sep 2024 09:45:29 AM UTC. 2024-09-21T09:46:17.805 INFO:teuthology.orchestra.run.smithi160.stdout:Package ceph-base-2:18.2.4-871.g6a1bdf85.el9.x86_64 is already installed. 2024-09-21T09:46:17.847 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T09:46:17.848 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-09-21T09:46:17.848 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T09:46:17.928 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install cephadm 2024-09-21T09:46:18.491 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-2:18.2.4-871.g6a1bdf85.el9.x86_64 33/33 2024-09-21T09:46:18.491 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/33 2024-09-21T09:46:18.491 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mds-2:18.2.4-871.g6a1bdf85.el9.x86_64 2/33 2024-09-21T09:46:18.491 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-2:18.2.4-871.g6a1bdf85.el9.x86_64 3/33 2024-09-21T09:46:18.491 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mon-2:18.2.4-871.g6a1bdf85.el9.x86_64 4/33 2024-09-21T09:46:18.491 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-osd-2:18.2.4-871.g6a1bdf85.el9.x86_64 5/33 2024-09-21T09:46:18.492 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-871.g6a1bdf85.el9.n 6/33 2024-09-21T09:46:18.492 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-09-21T09:46:18.492 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-09-21T09:46:18.492 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-09-21T09:46:18.492 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-09-21T09:46:18.492 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-09-21T09:46:18.492 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-09-21T09:46:18.492 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-09-21T09:46:18.492 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-09-21T09:46:18.492 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-09-21T09:46:18.492 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-09-21T09:46:18.492 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-09-21T09:46:18.492 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-09-21T09:46:18.492 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-09-21T09:46:18.493 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-09-21T09:46:18.493 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-09-21T09:46:18.493 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-09-21T09:46:18.493 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-09-21T09:46:18.493 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-09-21T09:46:18.493 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-09-21T09:46:18.494 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-09-21T09:46:18.494 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-09-21T09:46:18.494 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-09-21T09:46:18.494 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-09-21T09:46:18.494 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-09-21T09:46:18.494 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 31/33 2024-09-21T09:46:18.494 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-09-21T09:46:18.504 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:49 ago on Sat 21 Sep 2024 09:45:29 AM UTC. 2024-09-21T09:46:18.637 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T09:46:18.638 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:46:18.638 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-21T09:46:18.638 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:46:18.638 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-21T09:46:18.639 INFO:teuthology.orchestra.run.smithi160.stdout: cephadm noarch 2:18.2.4-871.g6a1bdf85.el9 ceph-noarch 226 k 2024-09-21T09:46:18.639 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:18.639 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-21T09:46:18.639 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:46:18.639 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-09-21T09:46:18.639 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:18.639 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 226 k 2024-09-21T09:46:18.639 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 217 k 2024-09-21T09:46:18.639 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-21T09:46:18.870 INFO:teuthology.orchestra.run.smithi160.stdout:cephadm-18.2.4-871.g6a1bdf85.el9.noarch.rpm 980 kB/s | 226 kB 00:00 2024-09-21T09:46:18.871 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:46:18.871 INFO:teuthology.orchestra.run.smithi160.stdout:Total 972 kB/s | 226 kB 00:00 2024-09-21T09:46:18.871 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-21T09:46:18.876 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-21T09:46:18.877 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-21T09:46:18.879 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-21T09:46:18.880 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-21T09:46:18.951 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-21T09:46:19.280 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-09-21T09:46:19.280 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:19.280 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-21T09:46:19.281 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:46:19.281 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mds-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:46:19.281 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:46:19.281 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core-2:18.2.4-871.g6a1bdf85.el9.noarch 2024-09-21T09:46:19.281 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mon-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:46:19.281 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-osd-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:46:19.281 INFO:teuthology.orchestra.run.smithi070.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T09:46:19.281 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T09:46:19.281 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T09:46:19.281 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T09:46:19.281 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T09:46:19.281 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T09:46:19.281 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T09:46:19.282 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T09:46:19.282 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T09:46:19.282 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T09:46:19.282 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T09:46:19.282 INFO:teuthology.orchestra.run.smithi070.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T09:46:19.282 INFO:teuthology.orchestra.run.smithi070.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T09:46:19.282 INFO:teuthology.orchestra.run.smithi070.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T09:46:19.282 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T09:46:19.282 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T09:46:19.282 INFO:teuthology.orchestra.run.smithi070.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T09:46:19.282 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T09:46:19.282 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T09:46:19.282 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T09:46:19.283 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T09:46:19.283 INFO:teuthology.orchestra.run.smithi070.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T09:46:19.283 INFO:teuthology.orchestra.run.smithi070.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T09:46:19.283 INFO:teuthology.orchestra.run.smithi070.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T09:46:19.283 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T09:46:19.283 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T09:46:19.283 INFO:teuthology.orchestra.run.smithi070.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T09:46:19.283 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:19.283 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-21T09:46:19.417 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: cephadm-2:18.2.4-871.g6a1bdf85.el9.noarch 1/1 2024-09-21T09:46:19.506 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : cephadm-2:18.2.4-871.g6a1bdf85.el9.noarch 1/1 2024-09-21T09:46:19.677 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-base 2024-09-21T09:46:19.840 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: cephadm-2:18.2.4-871.g6a1bdf85.el9.noarch 1/1 2024-09-21T09:46:20.088 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : cephadm-2:18.2.4-871.g6a1bdf85.el9.noarch 1/1 2024-09-21T09:46:20.088 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:20.089 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-21T09:46:20.089 INFO:teuthology.orchestra.run.smithi160.stdout: cephadm-2:18.2.4-871.g6a1bdf85.el9.noarch 2024-09-21T09:46:20.089 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:20.089 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T09:46:20.265 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:00:47 ago on Sat 21 Sep 2024 09:45:33 AM UTC. 2024-09-21T09:46:20.309 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-immutable-object-cache 2024-09-21T09:46:20.353 INFO:teuthology.orchestra.run.smithi070.stdout:Package ceph-base-2:18.2.4-871.g6a1bdf85.el9.x86_64 is already installed. 2024-09-21T09:46:20.395 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-21T09:46:20.396 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-09-21T09:46:20.396 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-21T09:46:20.465 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install cephadm 2024-09-21T09:46:20.904 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:51 ago on Sat 21 Sep 2024 09:45:29 AM UTC. 2024-09-21T09:46:21.039 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T09:46:21.040 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:46:21.040 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repo Size 2024-09-21T09:46:21.040 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:46:21.040 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-21T09:46:21.040 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 142 k 2024-09-21T09:46:21.040 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:21.040 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-21T09:46:21.040 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:46:21.040 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-09-21T09:46:21.040 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:21.041 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 142 k 2024-09-21T09:46:21.041 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 395 k 2024-09-21T09:46:21.041 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-21T09:46:21.043 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:00:48 ago on Sat 21 Sep 2024 09:45:33 AM UTC. 2024-09-21T09:46:21.174 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-21T09:46:21.174 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:46:21.175 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-21T09:46:21.175 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:46:21.175 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-21T09:46:21.175 INFO:teuthology.orchestra.run.smithi070.stdout: cephadm noarch 2:18.2.4-871.g6a1bdf85.el9 ceph-noarch 226 k 2024-09-21T09:46:21.175 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:21.175 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-21T09:46:21.175 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:46:21.175 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-09-21T09:46:21.175 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:21.175 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 226 k 2024-09-21T09:46:21.175 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 217 k 2024-09-21T09:46:21.175 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-21T09:46:21.243 INFO:teuthology.orchestra.run.smithi160.stdout:ceph-immutable-object-cache-18.2.4-871.g6a1bdf8 708 kB/s | 142 kB 00:00 2024-09-21T09:46:21.243 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:46:21.244 INFO:teuthology.orchestra.run.smithi160.stdout:Total 702 kB/s | 142 kB 00:00 2024-09-21T09:46:21.244 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-21T09:46:21.251 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-21T09:46:21.251 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-21T09:46:21.310 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-21T09:46:21.310 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-21T09:46:21.411 INFO:teuthology.orchestra.run.smithi070.stdout:cephadm-18.2.4-871.g6a1bdf85.el9.noarch.rpm 960 kB/s | 226 kB 00:00 2024-09-21T09:46:21.412 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:46:21.412 INFO:teuthology.orchestra.run.smithi070.stdout:Total 953 kB/s | 226 kB 00:00 2024-09-21T09:46:21.412 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-21T09:46:21.417 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-21T09:46:21.417 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-21T09:46:21.420 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-21T09:46:21.420 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-21T09:46:21.456 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-21T09:46:21.495 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-21T09:46:21.544 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-871.g6a1bdf85.e 1/1 2024-09-21T09:46:21.569 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-871.g6a1bdf85.e 1/1 2024-09-21T09:46:21.569 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:46:21.569 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T09:46:21.570 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:21.950 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: cephadm-2:18.2.4-871.g6a1bdf85.el9.noarch 1/1 2024-09-21T09:46:22.041 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephadm-2:18.2.4-871.g6a1bdf85.el9.noarch 1/1 2024-09-21T09:46:22.400 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: cephadm-2:18.2.4-871.g6a1bdf85.el9.noarch 1/1 2024-09-21T09:46:22.504 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-871.g6a1bdf85.e 1/1 2024-09-21T09:46:22.505 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:22.505 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-21T09:46:22.505 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-immutable-object-cache-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:46:22.505 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:22.505 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T09:46:22.651 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephadm-2:18.2.4-871.g6a1bdf85.el9.noarch 1/1 2024-09-21T09:46:22.651 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:22.651 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-21T09:46:22.651 INFO:teuthology.orchestra.run.smithi070.stdout: cephadm-2:18.2.4-871.g6a1bdf85.el9.noarch 2024-09-21T09:46:22.651 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:22.651 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-21T09:46:22.738 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr 2024-09-21T09:46:22.856 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-immutable-object-cache 2024-09-21T09:46:23.336 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:54 ago on Sat 21 Sep 2024 09:45:29 AM UTC. 2024-09-21T09:46:23.430 INFO:teuthology.orchestra.run.smithi160.stdout:Package ceph-mgr-2:18.2.4-871.g6a1bdf85.el9.x86_64 is already installed. 2024-09-21T09:46:23.472 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T09:46:23.473 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-09-21T09:46:23.473 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T09:46:23.541 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:00:50 ago on Sat 21 Sep 2024 09:45:33 AM UTC. 2024-09-21T09:46:23.563 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-dashboard 2024-09-21T09:46:23.678 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-21T09:46:23.679 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:46:23.679 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repo Size 2024-09-21T09:46:23.679 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:46:23.680 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-21T09:46:23.680 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 142 k 2024-09-21T09:46:23.680 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:23.680 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-21T09:46:23.680 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:46:23.680 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-09-21T09:46:23.680 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:23.680 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 142 k 2024-09-21T09:46:23.680 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 395 k 2024-09-21T09:46:23.680 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-21T09:46:23.893 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-immutable-object-cache-18.2.4-871.g6a1bdf8 671 kB/s | 142 kB 00:00 2024-09-21T09:46:23.893 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:46:23.893 INFO:teuthology.orchestra.run.smithi070.stdout:Total 667 kB/s | 142 kB 00:00 2024-09-21T09:46:23.894 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-21T09:46:23.901 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-21T09:46:23.901 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-21T09:46:23.958 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-21T09:46:23.958 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-21T09:46:24.098 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-21T09:46:24.148 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:55 ago on Sat 21 Sep 2024 09:45:29 AM UTC. 2024-09-21T09:46:24.160 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-871.g6a1bdf85.e 1/1 2024-09-21T09:46:24.185 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-871.g6a1bdf85.e 1/1 2024-09-21T09:46:24.185 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:46:24.185 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T09:46:24.185 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:24.279 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T09:46:24.280 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:46:24.280 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-21T09:46:24.280 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:46:24.280 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-21T09:46:24.280 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-dashboard noarch 2:18.2.4-871.g6a1bdf85.el9 ceph-noarch 3.5 M 2024-09-21T09:46:24.280 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-21T09:46:24.281 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-grafana-dashboards noarch 2:18.2.4-871.g6a1bdf85.el9 ceph-noarch 24 k 2024-09-21T09:46:24.281 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-prometheus-alerts noarch 2:18.2.4-871.g6a1bdf85.el9 ceph-noarch 16 k 2024-09-21T09:46:24.281 INFO:teuthology.orchestra.run.smithi160.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T09:46:24.281 INFO:teuthology.orchestra.run.smithi160.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T09:46:24.281 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:24.281 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-21T09:46:24.281 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:46:24.281 INFO:teuthology.orchestra.run.smithi160.stdout:Install 5 Packages 2024-09-21T09:46:24.281 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:24.282 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 3.8 M 2024-09-21T09:46:24.282 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 87 M 2024-09-21T09:46:24.282 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-21T09:46:24.471 INFO:teuthology.orchestra.run.smithi160.stdout:(1/5): ceph-prometheus-alerts-18.2.4-871.g6a1bd 104 kB/s | 16 kB 00:00 2024-09-21T09:46:24.496 INFO:teuthology.orchestra.run.smithi160.stdout:(2/5): ceph-grafana-dashboards-18.2.4-871.g6a1b 133 kB/s | 24 kB 00:00 2024-09-21T09:46:24.538 INFO:teuthology.orchestra.run.smithi160.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 460 kB/s | 31 kB 00:00 2024-09-21T09:46:24.572 INFO:teuthology.orchestra.run.smithi160.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 2.4 MB/s | 188 kB 00:00 2024-09-21T09:46:24.672 INFO:teuthology.orchestra.run.smithi160.stdout:(5/5): ceph-mgr-dashboard-18.2.4-871.g6a1bdf85. 9.8 MB/s | 3.5 MB 00:00 2024-09-21T09:46:24.676 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:46:24.676 INFO:teuthology.orchestra.run.smithi160.stdout:Total 9.6 MB/s | 3.8 MB 00:00 2024-09-21T09:46:24.676 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-21T09:46:24.683 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-21T09:46:24.683 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-21T09:46:24.770 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-21T09:46:24.770 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-21T09:46:25.025 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-21T09:46:25.119 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-09-21T09:46:25.125 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-871.g6a1bdf85.e 1/1 2024-09-21T09:46:25.125 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:25.125 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-21T09:46:25.125 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-immutable-object-cache-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:46:25.126 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:25.126 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-21T09:46:25.160 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-09-21T09:46:25.189 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-871.g6a1bdf85.el9.no 3/5 2024-09-21T09:46:25.349 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr 2024-09-21T09:46:25.932 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:00:52 ago on Sat 21 Sep 2024 09:45:33 AM UTC. 2024-09-21T09:46:26.022 INFO:teuthology.orchestra.run.smithi070.stdout:Package ceph-mgr-2:18.2.4-871.g6a1bdf85.el9.x86_64 is already installed. 2024-09-21T09:46:26.063 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-21T09:46:26.064 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-09-21T09:46:26.064 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-21T09:46:26.141 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr-dashboard 2024-09-21T09:46:26.244 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-871.g6a1bdf85.el9.n 4/5 2024-09-21T09:46:26.276 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-871.g6a1bdf85.el9.noarch 5/5 2024-09-21T09:46:26.726 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:00:53 ago on Sat 21 Sep 2024 09:45:33 AM UTC. 2024-09-21T09:46:26.857 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-21T09:46:26.857 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:46:26.858 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-21T09:46:26.858 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:46:26.858 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-21T09:46:26.858 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard noarch 2:18.2.4-871.g6a1bdf85.el9 ceph-noarch 3.5 M 2024-09-21T09:46:26.858 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-21T09:46:26.858 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-grafana-dashboards noarch 2:18.2.4-871.g6a1bdf85.el9 ceph-noarch 24 k 2024-09-21T09:46:26.858 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-prometheus-alerts noarch 2:18.2.4-871.g6a1bdf85.el9 ceph-noarch 16 k 2024-09-21T09:46:26.858 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T09:46:26.858 INFO:teuthology.orchestra.run.smithi070.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T09:46:26.858 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:26.859 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-21T09:46:26.859 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:46:26.859 INFO:teuthology.orchestra.run.smithi070.stdout:Install 5 Packages 2024-09-21T09:46:26.859 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:26.859 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 3.8 M 2024-09-21T09:46:26.859 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 87 M 2024-09-21T09:46:26.859 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-21T09:46:27.344 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-871.g6a1bdf85.el9.noarch 5/5 2024-09-21T09:46:27.344 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-871.g6a1bdf85.el9.n 1/5 2024-09-21T09:46:27.344 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-871.g6a1bdf85.el9.noarch 2/5 2024-09-21T09:46:27.344 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-871.g6a1bdf85.el9.no 3/5 2024-09-21T09:46:27.344 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-09-21T09:46:27.380 INFO:teuthology.orchestra.run.smithi070.stdout:(1/5): ceph-prometheus-alerts-18.2.4-871.g6a1bd 111 kB/s | 16 kB 00:00 2024-09-21T09:46:27.405 INFO:teuthology.orchestra.run.smithi070.stdout:(2/5): ceph-grafana-dashboards-18.2.4-871.g6a1b 140 kB/s | 24 kB 00:00 2024-09-21T09:46:27.439 INFO:teuthology.orchestra.run.smithi070.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 524 kB/s | 31 kB 00:00 2024-09-21T09:46:27.464 INFO:teuthology.orchestra.run.smithi070.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 3.1 MB/s | 188 kB 00:00 2024-09-21T09:46:27.605 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-09-21T09:46:27.606 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:27.606 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-21T09:46:27.606 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-grafana-dashboards-2:18.2.4-871.g6a1bdf85.el9.noarch 2024-09-21T09:46:27.606 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-dashboard-2:18.2.4-871.g6a1bdf85.el9.noarch 2024-09-21T09:46:27.606 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-prometheus-alerts-2:18.2.4-871.g6a1bdf85.el9.noarch 2024-09-21T09:46:27.606 INFO:teuthology.orchestra.run.smithi160.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T09:46:27.606 INFO:teuthology.orchestra.run.smithi160.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T09:46:27.606 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:27.606 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T09:46:27.623 INFO:teuthology.orchestra.run.smithi070.stdout:(5/5): ceph-mgr-dashboard-18.2.4-871.g6a1bdf85. 9.0 MB/s | 3.5 MB 00:00 2024-09-21T09:46:27.626 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:46:27.626 INFO:teuthology.orchestra.run.smithi070.stdout:Total 4.9 MB/s | 3.8 MB 00:00 2024-09-21T09:46:27.627 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-21T09:46:27.634 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-21T09:46:27.634 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-21T09:46:27.720 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-21T09:46:27.721 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-21T09:46:27.843 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-21T09:46:27.974 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-21T09:46:28.070 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-09-21T09:46:28.102 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-09-21T09:46:28.131 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-871.g6a1bdf85.el9.no 3/5 2024-09-21T09:46:28.420 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:59 ago on Sat 21 Sep 2024 09:45:29 AM UTC. 2024-09-21T09:46:28.552 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T09:46:28.553 INFO:teuthology.orchestra.run.smithi160.stdout:====================================================================================== 2024-09-21T09:46:28.554 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-21T09:46:28.554 INFO:teuthology.orchestra.run.smithi160.stdout:====================================================================================== 2024-09-21T09:46:28.554 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-21T09:46:28.554 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-871.g6a1bdf85.el9 ceph-noarch 7.4 M 2024-09-21T09:46:28.554 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-21T09:46:28.554 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T09:46:28.554 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T09:46:28.554 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T09:46:28.554 INFO:teuthology.orchestra.run.smithi160.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T09:46:28.554 INFO:teuthology.orchestra.run.smithi160.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T09:46:28.554 INFO:teuthology.orchestra.run.smithi160.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T09:46:28.554 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T09:46:28.555 INFO:teuthology.orchestra.run.smithi160.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T09:46:28.555 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T09:46:28.555 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T09:46:28.555 INFO:teuthology.orchestra.run.smithi160.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T09:46:28.555 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:28.555 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-21T09:46:28.555 INFO:teuthology.orchestra.run.smithi160.stdout:====================================================================================== 2024-09-21T09:46:28.555 INFO:teuthology.orchestra.run.smithi160.stdout:Install 12 Packages 2024-09-21T09:46:28.555 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:28.556 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 38 M 2024-09-21T09:46:28.556 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 212 M 2024-09-21T09:46:28.556 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-21T09:46:29.138 INFO:teuthology.orchestra.run.smithi160.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-21T09:46:29.171 INFO:teuthology.orchestra.run.smithi160.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.6 MB/s | 801 kB 00:00 2024-09-21T09:46:29.211 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-871.g6a1bdf85.el9.n 4/5 2024-09-21T09:46:29.238 INFO:teuthology.orchestra.run.smithi160.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 327 kB/s | 33 kB 00:00 2024-09-21T09:46:29.251 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-871.g6a1bdf85.el9.noarch 5/5 2024-09-21T09:46:29.280 INFO:teuthology.orchestra.run.smithi160.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 436 kB/s | 18 kB 00:00 2024-09-21T09:46:29.339 INFO:teuthology.orchestra.run.smithi160.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 661 kB/s | 38 kB 00:00 2024-09-21T09:46:29.439 INFO:teuthology.orchestra.run.smithi160.stdout:(6/12): ceph-mgr-diskprediction-local-18.2.4-87 15 MB/s | 7.4 MB 00:00 2024-09-21T09:46:29.514 INFO:teuthology.orchestra.run.smithi160.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.7 MB/s | 3.0 MB 00:00 2024-09-21T09:46:29.639 INFO:teuthology.orchestra.run.smithi160.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 41 MB/s | 5.1 MB 00:00 2024-09-21T09:46:29.715 INFO:teuthology.orchestra.run.smithi160.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 13 MB/s | 4.9 MB 00:00 2024-09-21T09:46:29.740 INFO:teuthology.orchestra.run.smithi160.stdout:(10/12): python3-devel-3.9.19-8.el9.x86_64.rpm 814 kB/s | 245 kB 00:00 2024-09-21T09:46:29.765 INFO:teuthology.orchestra.run.smithi160.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.0 MB/s | 250 kB 00:00 2024-09-21T09:46:30.015 INFO:teuthology.orchestra.run.smithi160.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 54 MB/s | 16 MB 00:00 2024-09-21T09:46:30.016 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:46:30.017 INFO:teuthology.orchestra.run.smithi160.stdout:Total 26 MB/s | 38 MB 00:01 2024-09-21T09:46:30.371 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-21T09:46:30.397 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-21T09:46:30.397 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-21T09:46:30.411 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-871.g6a1bdf85.el9.noarch 5/5 2024-09-21T09:46:30.411 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-871.g6a1bdf85.el9.n 1/5 2024-09-21T09:46:30.411 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-871.g6a1bdf85.el9.noarch 2/5 2024-09-21T09:46:30.411 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-871.g6a1bdf85.el9.no 3/5 2024-09-21T09:46:30.411 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-09-21T09:46:30.673 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-09-21T09:46:30.673 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:30.673 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-21T09:46:30.673 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-grafana-dashboards-2:18.2.4-871.g6a1bdf85.el9.noarch 2024-09-21T09:46:30.673 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard-2:18.2.4-871.g6a1bdf85.el9.noarch 2024-09-21T09:46:30.674 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-prometheus-alerts-2:18.2.4-871.g6a1bdf85.el9.noarch 2024-09-21T09:46:30.674 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T09:46:30.674 INFO:teuthology.orchestra.run.smithi070.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T09:46:30.674 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:30.674 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-21T09:46:30.906 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-21T09:46:30.907 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-21T09:46:30.916 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-21T09:46:31.497 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:00:58 ago on Sat 21 Sep 2024 09:45:33 AM UTC. 2024-09-21T09:46:31.631 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-21T09:46:31.632 INFO:teuthology.orchestra.run.smithi070.stdout:====================================================================================== 2024-09-21T09:46:31.632 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-21T09:46:31.632 INFO:teuthology.orchestra.run.smithi070.stdout:====================================================================================== 2024-09-21T09:46:31.632 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-21T09:46:31.632 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-871.g6a1bdf85.el9 ceph-noarch 7.4 M 2024-09-21T09:46:31.632 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-21T09:46:31.632 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T09:46:31.632 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T09:46:31.632 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T09:46:31.632 INFO:teuthology.orchestra.run.smithi070.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T09:46:31.633 INFO:teuthology.orchestra.run.smithi070.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T09:46:31.633 INFO:teuthology.orchestra.run.smithi070.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T09:46:31.633 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T09:46:31.633 INFO:teuthology.orchestra.run.smithi070.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T09:46:31.633 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T09:46:31.633 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T09:46:31.633 INFO:teuthology.orchestra.run.smithi070.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T09:46:31.633 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:31.633 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-21T09:46:31.633 INFO:teuthology.orchestra.run.smithi070.stdout:====================================================================================== 2024-09-21T09:46:31.633 INFO:teuthology.orchestra.run.smithi070.stdout:Install 12 Packages 2024-09-21T09:46:31.634 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:31.634 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 38 M 2024-09-21T09:46:31.634 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 212 M 2024-09-21T09:46:31.634 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-21T09:46:31.661 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-21T09:46:31.727 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-21T09:46:31.772 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-21T09:46:31.873 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-21T09:46:31.915 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-21T09:46:31.955 INFO:teuthology.orchestra.run.smithi070.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2024-09-21T09:46:31.988 INFO:teuthology.orchestra.run.smithi070.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.6 MB/s | 801 kB 00:00 2024-09-21T09:46:32.089 INFO:teuthology.orchestra.run.smithi070.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 245 kB/s | 33 kB 00:00 2024-09-21T09:46:32.102 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-21T09:46:32.130 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-21T09:46:32.139 INFO:teuthology.orchestra.run.smithi070.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 365 kB/s | 18 kB 00:00 2024-09-21T09:46:32.217 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-21T09:46:32.222 INFO:teuthology.orchestra.run.smithi070.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 461 kB/s | 38 kB 00:00 2024-09-21T09:46:32.323 INFO:teuthology.orchestra.run.smithi070.stdout:(6/12): ceph-mgr-diskprediction-local-18.2.4-87 15 MB/s | 7.4 MB 00:00 2024-09-21T09:46:32.498 INFO:teuthology.orchestra.run.smithi070.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-09-21T09:46:32.807 INFO:teuthology.orchestra.run.smithi070.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.7 MB/s | 3.0 MB 00:00 2024-09-21T09:46:32.878 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-21T09:46:32.890 INFO:teuthology.orchestra.run.smithi070.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 250 kB 00:00 2024-09-21T09:46:33.002 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-21T09:46:33.307 INFO:teuthology.orchestra.run.smithi070.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.6 MB/s | 4.9 MB 00:01 2024-09-21T09:46:33.758 INFO:teuthology.orchestra.run.smithi070.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.0 MB/s | 5.1 MB 00:01 2024-09-21T09:46:34.254 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T09:46:35.297 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-21T09:46:35.329 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-871.g6a1bdf 12/12 2024-09-21T09:46:35.432 INFO:teuthology.orchestra.run.smithi070.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.4 MB/s | 16 MB 00:02 2024-09-21T09:46:35.433 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:46:35.434 INFO:teuthology.orchestra.run.smithi070.stdout:Total 10 MB/s | 38 MB 00:03 2024-09-21T09:46:35.795 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-21T09:46:35.821 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-21T09:46:35.821 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-21T09:46:36.346 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-21T09:46:36.346 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-21T09:46:37.138 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-21T09:46:37.221 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-21T09:46:37.272 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-21T09:46:37.370 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-21T09:46:37.415 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-21T09:46:37.601 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-21T09:46:37.631 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-21T09:46:37.708 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-21T09:46:37.937 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-871.g6a1bdf 12/12 2024-09-21T09:46:37.937 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-871.g6a1bdf 1/12 2024-09-21T09:46:37.938 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-21T09:46:37.938 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-21T09:46:37.938 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-21T09:46:37.938 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-21T09:46:37.938 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-21T09:46:37.938 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-21T09:46:37.938 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-21T09:46:37.938 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-21T09:46:37.938 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T09:46:37.938 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-21T09:46:38.346 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-21T09:46:38.346 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:38.346 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-21T09:46:38.346 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-diskprediction-local-2:18.2.4-871.g6a1bdf85.el9.noarch 2024-09-21T09:46:38.346 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T09:46:38.346 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T09:46:38.346 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T09:46:38.346 INFO:teuthology.orchestra.run.smithi160.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T09:46:38.346 INFO:teuthology.orchestra.run.smithi160.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T09:46:38.347 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T09:46:38.347 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T09:46:38.347 INFO:teuthology.orchestra.run.smithi160.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T09:46:38.347 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T09:46:38.347 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T09:46:38.347 INFO:teuthology.orchestra.run.smithi160.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T09:46:38.347 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:38.347 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T09:46:38.372 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-21T09:46:38.485 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-21T09:46:38.725 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-rook 2024-09-21T09:46:39.319 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:10 ago on Sat 21 Sep 2024 09:45:29 AM UTC. 2024-09-21T09:46:39.452 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T09:46:39.453 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:46:39.453 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-21T09:46:39.453 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:46:39.453 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-21T09:46:39.453 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-rook noarch 2:18.2.4-871.g6a1bdf85.el9 ceph-noarch 49 k 2024-09-21T09:46:39.454 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-21T09:46:39.454 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T09:46:39.454 INFO:teuthology.orchestra.run.smithi160.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T09:46:39.454 INFO:teuthology.orchestra.run.smithi160.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T09:46:39.454 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T09:46:39.454 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T09:46:39.454 INFO:teuthology.orchestra.run.smithi160.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T09:46:39.454 INFO:teuthology.orchestra.run.smithi160.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T09:46:39.454 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T09:46:39.454 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T09:46:39.454 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T09:46:39.454 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T09:46:39.454 INFO:teuthology.orchestra.run.smithi160.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T09:46:39.454 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:39.455 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-21T09:46:39.455 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:46:39.455 INFO:teuthology.orchestra.run.smithi160.stdout:Install 13 Packages 2024-09-21T09:46:39.455 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:39.455 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 2.2 M 2024-09-21T09:46:39.455 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 26 M 2024-09-21T09:46:39.456 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-21T09:46:39.661 INFO:teuthology.orchestra.run.smithi160.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 155 kB/s | 19 kB 00:00 2024-09-21T09:46:39.678 INFO:teuthology.orchestra.run.smithi160.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 182 kB/s | 26 kB 00:00 2024-09-21T09:46:39.695 INFO:teuthology.orchestra.run.smithi160.stdout:(3/13): ceph-mgr-rook-18.2.4-871.g6a1bdf85.el9. 313 kB/s | 49 kB 00:00 2024-09-21T09:46:39.737 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T09:46:39.779 INFO:teuthology.orchestra.run.smithi160.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.8 MB/s | 222 kB 00:00 2024-09-21T09:46:39.804 INFO:teuthology.orchestra.run.smithi160.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-21T09:46:39.838 INFO:teuthology.orchestra.run.smithi160.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 915 kB/s | 54 kB 00:00 2024-09-21T09:46:39.871 INFO:teuthology.orchestra.run.smithi160.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-21T09:46:39.897 INFO:teuthology.orchestra.run.smithi160.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.4 MB/s | 279 kB 00:00 2024-09-21T09:46:39.922 INFO:teuthology.orchestra.run.smithi160.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 172 kB/s | 14 kB 00:00 2024-09-21T09:46:39.955 INFO:teuthology.orchestra.run.smithi160.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-21T09:46:39.981 INFO:teuthology.orchestra.run.smithi160.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-09-21T09:46:40.006 INFO:teuthology.orchestra.run.smithi160.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-21T09:46:40.040 INFO:teuthology.orchestra.run.smithi160.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-21T09:46:40.042 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:46:40.042 INFO:teuthology.orchestra.run.smithi160.stdout:Total 3.8 MB/s | 2.2 MB 00:00 2024-09-21T09:46:40.106 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-21T09:46:40.120 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-21T09:46:40.121 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-21T09:46:40.208 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-21T09:46:40.208 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-21T09:46:40.713 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-21T09:46:40.763 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-21T09:46:40.789 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-871.g6a1bdf 12/12 2024-09-21T09:46:40.876 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-21T09:46:40.932 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-21T09:46:41.003 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-21T09:46:41.033 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-21T09:46:41.074 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-21T09:46:41.175 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-21T09:46:41.277 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-21T09:46:41.335 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-21T09:46:42.043 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-21T09:46:42.098 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-21T09:46:42.132 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-21T09:46:42.188 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-21T09:46:42.219 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-rook-2:18.2.4-871.g6a1bdf85.el9.noarch 13/13 2024-09-21T09:46:42.974 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-871.g6a1bdf85.el9.noarch 13/13 2024-09-21T09:46:42.974 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-rook-2:18.2.4-871.g6a1bdf85.el9.noarch 1/13 2024-09-21T09:46:42.974 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-21T09:46:42.974 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-21T09:46:42.974 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-21T09:46:42.974 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-21T09:46:42.974 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-21T09:46:42.975 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-21T09:46:42.975 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-21T09:46:42.975 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-21T09:46:42.975 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-21T09:46:42.975 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-21T09:46:42.975 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-21T09:46:43.320 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-21T09:46:43.320 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:43.320 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-21T09:46:43.320 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-rook-2:18.2.4-871.g6a1bdf85.el9.noarch 2024-09-21T09:46:43.320 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T09:46:43.320 INFO:teuthology.orchestra.run.smithi160.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T09:46:43.320 INFO:teuthology.orchestra.run.smithi160.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T09:46:43.320 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T09:46:43.320 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T09:46:43.320 INFO:teuthology.orchestra.run.smithi160.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T09:46:43.321 INFO:teuthology.orchestra.run.smithi160.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T09:46:43.321 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T09:46:43.321 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T09:46:43.321 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T09:46:43.321 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T09:46:43.321 INFO:teuthology.orchestra.run.smithi160.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T09:46:43.321 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:43.321 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T09:46:43.450 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-871.g6a1bdf 12/12 2024-09-21T09:46:43.450 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-871.g6a1bdf 1/12 2024-09-21T09:46:43.450 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-21T09:46:43.450 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-21T09:46:43.450 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-21T09:46:43.451 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-21T09:46:43.451 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-21T09:46:43.451 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-21T09:46:43.451 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-21T09:46:43.451 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-21T09:46:43.451 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T09:46:43.451 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-21T09:46:43.592 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-cephadm 2024-09-21T09:46:43.890 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-21T09:46:43.890 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:43.890 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-21T09:46:43.890 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-diskprediction-local-2:18.2.4-871.g6a1bdf85.el9.noarch 2024-09-21T09:46:43.891 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T09:46:43.891 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T09:46:43.891 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T09:46:43.891 INFO:teuthology.orchestra.run.smithi070.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T09:46:43.891 INFO:teuthology.orchestra.run.smithi070.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T09:46:43.891 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T09:46:43.891 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T09:46:43.891 INFO:teuthology.orchestra.run.smithi070.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T09:46:43.891 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T09:46:43.891 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T09:46:43.891 INFO:teuthology.orchestra.run.smithi070.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T09:46:43.891 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:43.891 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-21T09:46:44.182 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:15 ago on Sat 21 Sep 2024 09:45:29 AM UTC. 2024-09-21T09:46:44.275 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr-rook 2024-09-21T09:46:44.317 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T09:46:44.318 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:46:44.318 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-21T09:46:44.318 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:46:44.318 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-21T09:46:44.318 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-cephadm noarch 2:18.2.4-871.g6a1bdf85.el9 ceph-noarch 139 k 2024-09-21T09:46:44.318 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-21T09:46:44.318 INFO:teuthology.orchestra.run.smithi160.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T09:46:44.318 INFO:teuthology.orchestra.run.smithi160.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T09:46:44.318 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T09:46:44.318 INFO:teuthology.orchestra.run.smithi160.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T09:46:44.318 INFO:teuthology.orchestra.run.smithi160.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T09:46:44.318 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:44.318 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-21T09:46:44.319 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:46:44.319 INFO:teuthology.orchestra.run.smithi160.stdout:Install 6 Packages 2024-09-21T09:46:44.319 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:44.319 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 7.0 M 2024-09-21T09:46:44.319 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 33 M 2024-09-21T09:46:44.319 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-21T09:46:44.884 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:11 ago on Sat 21 Sep 2024 09:45:33 AM UTC. 2024-09-21T09:46:44.899 INFO:teuthology.orchestra.run.smithi160.stdout:(1/6): ceph-mgr-cephadm-18.2.4-871.g6a1bdf85.el 716 kB/s | 139 kB 00:00 2024-09-21T09:46:44.950 INFO:teuthology.orchestra.run.smithi160.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-21T09:46:44.966 INFO:teuthology.orchestra.run.smithi160.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 3.4 MB/s | 58 kB 00:00 2024-09-21T09:46:44.992 INFO:teuthology.orchestra.run.smithi160.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-21T09:46:45.028 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-21T09:46:45.029 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:46:45.030 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-21T09:46:45.030 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:46:45.030 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-21T09:46:45.030 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-rook noarch 2:18.2.4-871.g6a1bdf85.el9 ceph-noarch 49 k 2024-09-21T09:46:45.030 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-21T09:46:45.030 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T09:46:45.030 INFO:teuthology.orchestra.run.smithi070.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T09:46:45.030 INFO:teuthology.orchestra.run.smithi070.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T09:46:45.030 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T09:46:45.030 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T09:46:45.031 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T09:46:45.031 INFO:teuthology.orchestra.run.smithi070.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T09:46:45.031 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T09:46:45.031 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T09:46:45.031 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T09:46:45.031 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T09:46:45.031 INFO:teuthology.orchestra.run.smithi070.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T09:46:45.031 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:45.031 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-21T09:46:45.031 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:46:45.031 INFO:teuthology.orchestra.run.smithi070.stdout:Install 13 Packages 2024-09-21T09:46:45.031 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:45.032 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 2.2 M 2024-09-21T09:46:45.032 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 26 M 2024-09-21T09:46:45.032 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-21T09:46:45.234 INFO:teuthology.orchestra.run.smithi160.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 470 kB/s | 248 kB 00:00 2024-09-21T09:46:45.306 INFO:teuthology.orchestra.run.smithi070.stdout:(1/13): ceph-mgr-rook-18.2.4-871.g6a1bdf85.el9. 295 kB/s | 49 kB 00:00 2024-09-21T09:46:45.431 INFO:teuthology.orchestra.run.smithi070.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 88 kB/s | 26 kB 00:00 2024-09-21T09:46:45.456 INFO:teuthology.orchestra.run.smithi070.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 60 kB/s | 19 kB 00:00 2024-09-21T09:46:45.806 INFO:teuthology.orchestra.run.smithi070.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 423 kB/s | 159 kB 00:00 2024-09-21T09:46:45.849 INFO:teuthology.orchestra.run.smithi070.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 409 kB/s | 222 kB 00:00 2024-09-21T09:46:45.891 INFO:teuthology.orchestra.run.smithi070.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 774 kB/s | 32 kB 00:00 2024-09-21T09:46:45.916 INFO:teuthology.orchestra.run.smithi070.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 566 kB/s | 14 kB 00:00 2024-09-21T09:46:45.941 INFO:teuthology.orchestra.run.smithi070.stdout:(8/13): python3-requests-oauthlib-1.3.0-12.el9. 399 kB/s | 54 kB 00:00 2024-09-21T09:46:45.966 INFO:teuthology.orchestra.run.smithi070.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 547 kB/s | 279 kB 00:00 2024-09-21T09:46:46.008 INFO:teuthology.orchestra.run.smithi070.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.3 MB/s | 220 kB 00:00 2024-09-21T09:46:46.034 INFO:teuthology.orchestra.run.smithi070.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-09-21T09:46:46.059 INFO:teuthology.orchestra.run.smithi070.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 692 kB/s | 59 kB 00:00 2024-09-21T09:46:46.092 INFO:teuthology.orchestra.run.smithi070.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-21T09:46:46.095 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:46:46.095 INFO:teuthology.orchestra.run.smithi070.stdout:Total 2.1 MB/s | 2.2 MB 00:01 2024-09-21T09:46:46.158 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-21T09:46:46.173 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-21T09:46:46.173 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-21T09:46:46.259 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-21T09:46:46.260 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-21T09:46:46.434 INFO:teuthology.orchestra.run.smithi160.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.4 MB/s | 6.0 MB 00:01 2024-09-21T09:46:46.436 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:46:46.437 INFO:teuthology.orchestra.run.smithi160.stdout:Total 3.3 MB/s | 7.0 MB 00:02 2024-09-21T09:46:46.512 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-21T09:46:46.524 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-21T09:46:46.524 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-21T09:46:46.626 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-21T09:46:46.627 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-21T09:46:46.761 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-21T09:46:46.929 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-21T09:46:46.935 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-21T09:46:47.001 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-21T09:46:47.057 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-21T09:46:47.095 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-21T09:46:47.108 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-21T09:46:47.136 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-21T09:46:47.152 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-21T09:46:47.237 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-21T09:46:47.348 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-21T09:46:47.396 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-21T09:46:47.678 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-21T09:46:47.772 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-21T09:46:47.833 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-21T09:46:47.856 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-871.g6a1bdf85.el9.noarch 6/6 2024-09-21T09:46:48.108 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-21T09:46:48.169 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-21T09:46:48.203 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-21T09:46:48.250 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-21T09:46:48.272 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-rook-2:18.2.4-871.g6a1bdf85.el9.noarch 13/13 2024-09-21T09:46:48.618 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-871.g6a1bdf85.el9.noarch 6/6 2024-09-21T09:46:48.618 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-871.g6a1bdf85.el9.noarch 1/6 2024-09-21T09:46:48.619 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-21T09:46:48.619 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-21T09:46:48.619 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-21T09:46:48.619 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-21T09:46:48.901 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-21T09:46:48.901 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:48.901 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-21T09:46:48.901 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-cephadm-2:18.2.4-871.g6a1bdf85.el9.noarch 2024-09-21T09:46:48.901 INFO:teuthology.orchestra.run.smithi160.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T09:46:48.901 INFO:teuthology.orchestra.run.smithi160.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T09:46:48.901 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T09:46:48.901 INFO:teuthology.orchestra.run.smithi160.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T09:46:48.901 INFO:teuthology.orchestra.run.smithi160.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T09:46:48.902 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:48.902 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T09:46:49.114 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-871.g6a1bdf85.el9.noarch 13/13 2024-09-21T09:46:49.114 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-rook-2:18.2.4-871.g6a1bdf85.el9.noarch 1/13 2024-09-21T09:46:49.115 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-21T09:46:49.115 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-21T09:46:49.115 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-21T09:46:49.115 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-21T09:46:49.115 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-21T09:46:49.115 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-21T09:46:49.115 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-21T09:46:49.115 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-21T09:46:49.115 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-21T09:46:49.115 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-21T09:46:49.115 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-21T09:46:49.181 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-fuse 2024-09-21T09:46:49.444 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-21T09:46:49.444 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:49.444 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-21T09:46:49.444 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-rook-2:18.2.4-871.g6a1bdf85.el9.noarch 2024-09-21T09:46:49.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T09:46:49.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T09:46:49.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T09:46:49.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T09:46:49.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T09:46:49.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T09:46:49.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T09:46:49.445 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T09:46:49.445 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T09:46:49.445 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T09:46:49.445 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T09:46:49.445 INFO:teuthology.orchestra.run.smithi070.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T09:46:49.445 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:49.445 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-21T09:46:49.723 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr-cephadm 2024-09-21T09:46:49.761 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:20 ago on Sat 21 Sep 2024 09:45:29 AM UTC. 2024-09-21T09:46:49.894 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T09:46:49.895 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:46:49.895 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-21T09:46:49.895 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:46:49.895 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-21T09:46:49.895 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-fuse x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 853 k 2024-09-21T09:46:49.895 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-21T09:46:49.895 INFO:teuthology.orchestra.run.smithi160.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T09:46:49.895 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:49.895 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-21T09:46:49.895 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:46:49.895 INFO:teuthology.orchestra.run.smithi160.stdout:Install 2 Packages 2024-09-21T09:46:49.896 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:49.896 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 932 k 2024-09-21T09:46:49.896 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 2.7 M 2024-09-21T09:46:49.896 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-21T09:46:50.143 INFO:teuthology.orchestra.run.smithi160.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 654 kB/s | 80 kB 00:00 2024-09-21T09:46:50.301 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:17 ago on Sat 21 Sep 2024 09:45:33 AM UTC. 2024-09-21T09:46:50.302 INFO:teuthology.orchestra.run.smithi160.stdout:(2/2): ceph-fuse-18.2.4-871.g6a1bdf85.el9.x86_6 3.0 MB/s | 853 kB 00:00 2024-09-21T09:46:50.302 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:46:50.303 INFO:teuthology.orchestra.run.smithi160.stdout:Total 2.2 MB/s | 932 kB 00:00 2024-09-21T09:46:50.314 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-21T09:46:50.335 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-21T09:46:50.335 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-21T09:46:50.415 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-21T09:46:50.416 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-21T09:46:50.433 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-21T09:46:50.434 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:46:50.434 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-21T09:46:50.434 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:46:50.434 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-21T09:46:50.434 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-cephadm noarch 2:18.2.4-871.g6a1bdf85.el9 ceph-noarch 139 k 2024-09-21T09:46:50.434 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-21T09:46:50.434 INFO:teuthology.orchestra.run.smithi070.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T09:46:50.434 INFO:teuthology.orchestra.run.smithi070.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T09:46:50.434 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T09:46:50.434 INFO:teuthology.orchestra.run.smithi070.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T09:46:50.435 INFO:teuthology.orchestra.run.smithi070.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T09:46:50.435 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:50.435 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-21T09:46:50.435 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:46:50.435 INFO:teuthology.orchestra.run.smithi070.stdout:Install 6 Packages 2024-09-21T09:46:50.435 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:50.435 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 7.0 M 2024-09-21T09:46:50.436 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 33 M 2024-09-21T09:46:50.436 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-21T09:46:50.600 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-21T09:46:50.745 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-21T09:46:50.808 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-fuse-2:18.2.4-871.g6a1bdf85.el9.x86_64 2/2 2024-09-21T09:46:50.970 INFO:teuthology.orchestra.run.smithi070.stdout:(1/6): ceph-mgr-cephadm-18.2.4-871.g6a1bdf85.el 613 kB/s | 139 kB 00:00 2024-09-21T09:46:51.020 INFO:teuthology.orchestra.run.smithi070.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-21T09:46:51.046 INFO:teuthology.orchestra.run.smithi070.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-21T09:46:51.071 INFO:teuthology.orchestra.run.smithi070.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-21T09:46:51.388 INFO:teuthology.orchestra.run.smithi070.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 385 kB/s | 248 kB 00:00 2024-09-21T09:46:51.540 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-fuse-2:18.2.4-871.g6a1bdf85.el9.x86_64 2/2 2024-09-21T09:46:51.540 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-fuse-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/2 2024-09-21T09:46:51.808 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-21T09:46:51.809 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:51.809 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-21T09:46:51.809 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-fuse-2:18.2.4-871.g6a1bdf85.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-21T09:46:51.809 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:51.809 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T09:46:52.064 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install librados-devel 2024-09-21T09:46:52.080 INFO:teuthology.orchestra.run.smithi070.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.5 MB/s | 6.0 MB 00:01 2024-09-21T09:46:52.082 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:46:52.082 INFO:teuthology.orchestra.run.smithi070.stdout:Total 4.3 MB/s | 7.0 MB 00:01 2024-09-21T09:46:52.155 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-21T09:46:52.167 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-21T09:46:52.167 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-21T09:46:52.267 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-21T09:46:52.268 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-21T09:46:52.574 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-21T09:46:52.664 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:23 ago on Sat 21 Sep 2024 09:45:29 AM UTC. 2024-09-21T09:46:52.737 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-21T09:46:52.790 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-21T09:46:52.799 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T09:46:52.800 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:46:52.800 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repo Size 2024-09-21T09:46:52.800 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:46:52.800 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-21T09:46:52.800 INFO:teuthology.orchestra.run.smithi160.stdout: librados-devel x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 126 k 2024-09-21T09:46:52.800 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:52.800 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-21T09:46:52.800 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:46:52.800 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-09-21T09:46:52.800 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:52.801 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 126 k 2024-09-21T09:46:52.801 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 456 k 2024-09-21T09:46:52.801 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-21T09:46:53.006 INFO:teuthology.orchestra.run.smithi160.stdout:librados-devel-18.2.4-871.g6a1bdf85.el9.x86_64. 618 kB/s | 126 kB 00:00 2024-09-21T09:46:53.006 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:46:53.007 INFO:teuthology.orchestra.run.smithi160.stdout:Total 613 kB/s | 126 kB 00:00 2024-09-21T09:46:53.007 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-21T09:46:53.012 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-21T09:46:53.012 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-21T09:46:53.075 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-21T09:46:53.075 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-21T09:46:53.235 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-21T09:46:53.312 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-21T09:46:53.329 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librados-devel-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:46:53.410 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-21T09:46:53.480 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-21T09:46:53.511 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-871.g6a1bdf85.el9.noarch 6/6 2024-09-21T09:46:53.796 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados-devel-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:46:54.050 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados-devel-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:46:54.050 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:54.051 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-21T09:46:54.051 INFO:teuthology.orchestra.run.smithi160.stdout: librados-devel-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:46:54.051 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:54.051 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T09:46:54.281 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-871.g6a1bdf85.el9.noarch 6/6 2024-09-21T09:46:54.282 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-871.g6a1bdf85.el9.noarch 1/6 2024-09-21T09:46:54.282 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-21T09:46:54.282 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-21T09:46:54.282 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-21T09:46:54.282 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-21T09:46:54.333 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install libcephfs2 2024-09-21T09:46:54.559 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-21T09:46:54.559 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:54.559 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-21T09:46:54.559 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-cephadm-2:18.2.4-871.g6a1bdf85.el9.noarch 2024-09-21T09:46:54.559 INFO:teuthology.orchestra.run.smithi070.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T09:46:54.559 INFO:teuthology.orchestra.run.smithi070.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T09:46:54.560 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T09:46:54.560 INFO:teuthology.orchestra.run.smithi070.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T09:46:54.560 INFO:teuthology.orchestra.run.smithi070.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T09:46:54.560 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:54.560 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-21T09:46:54.835 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-fuse 2024-09-21T09:46:54.933 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:25 ago on Sat 21 Sep 2024 09:45:29 AM UTC. 2024-09-21T09:46:55.025 INFO:teuthology.orchestra.run.smithi160.stdout:Package libcephfs2-2:18.2.4-871.g6a1bdf85.el9.x86_64 is already installed. 2024-09-21T09:46:55.069 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T09:46:55.070 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-09-21T09:46:55.071 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T09:46:55.136 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install libcephfs-devel 2024-09-21T09:46:55.428 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:22 ago on Sat 21 Sep 2024 09:45:33 AM UTC. 2024-09-21T09:46:55.560 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-21T09:46:55.561 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:46:55.561 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-21T09:46:55.561 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:46:55.561 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-21T09:46:55.561 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-fuse x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 853 k 2024-09-21T09:46:55.561 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-21T09:46:55.561 INFO:teuthology.orchestra.run.smithi070.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T09:46:55.562 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:55.562 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-21T09:46:55.562 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:46:55.562 INFO:teuthology.orchestra.run.smithi070.stdout:Install 2 Packages 2024-09-21T09:46:55.562 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:55.562 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 932 k 2024-09-21T09:46:55.562 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 2.7 M 2024-09-21T09:46:55.563 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-21T09:46:55.717 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:26 ago on Sat 21 Sep 2024 09:45:29 AM UTC. 2024-09-21T09:46:55.850 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T09:46:55.851 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:46:55.851 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repo Size 2024-09-21T09:46:55.851 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:46:55.851 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-21T09:46:55.851 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs-devel x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 31 k 2024-09-21T09:46:55.851 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:55.851 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-21T09:46:55.851 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:46:55.851 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-09-21T09:46:55.851 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:55.851 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 31 k 2024-09-21T09:46:55.851 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 139 k 2024-09-21T09:46:55.852 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-21T09:46:56.009 INFO:teuthology.orchestra.run.smithi160.stdout:libcephfs-devel-18.2.4-871.g6a1bdf85.el9.x86_64 197 kB/s | 31 kB 00:00 2024-09-21T09:46:56.009 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:46:56.010 INFO:teuthology.orchestra.run.smithi160.stdout:Total 196 kB/s | 31 kB 00:00 2024-09-21T09:46:56.010 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-21T09:46:56.013 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-21T09:46:56.013 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-21T09:46:56.027 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-21T09:46:56.028 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-21T09:46:56.098 INFO:teuthology.orchestra.run.smithi070.stdout:(1/2): ceph-fuse-18.2.4-871.g6a1bdf85.el9.x86_6 3.2 MB/s | 853 kB 00:00 2024-09-21T09:46:56.115 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-21T09:46:56.222 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephfs-devel-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:46:56.282 INFO:teuthology.orchestra.run.smithi070.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 180 kB/s | 80 kB 00:00 2024-09-21T09:46:56.282 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:46:56.283 INFO:teuthology.orchestra.run.smithi070.stdout:Total 1.3 MB/s | 932 kB 00:00 2024-09-21T09:46:56.295 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-21T09:46:56.317 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-21T09:46:56.317 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-21T09:46:56.399 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-21T09:46:56.400 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-21T09:46:56.591 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-21T09:46:56.632 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:46:56.743 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-21T09:46:56.823 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-fuse-2:18.2.4-871.g6a1bdf85.el9.x86_64 2/2 2024-09-21T09:46:56.868 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephfs-devel-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:46:56.868 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:56.868 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-21T09:46:56.868 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs-devel-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:46:56.868 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:46:56.868 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T09:46:57.090 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install librados2 2024-09-21T09:46:57.523 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-fuse-2:18.2.4-871.g6a1bdf85.el9.x86_64 2/2 2024-09-21T09:46:57.523 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-fuse-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/2 2024-09-21T09:46:57.680 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:28 ago on Sat 21 Sep 2024 09:45:29 AM UTC. 2024-09-21T09:46:57.771 INFO:teuthology.orchestra.run.smithi160.stdout:Package librados2-2:18.2.4-871.g6a1bdf85.el9.x86_64 is already installed. 2024-09-21T09:46:57.809 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-21T09:46:57.809 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:57.809 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-21T09:46:57.809 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-fuse-2:18.2.4-871.g6a1bdf85.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-21T09:46:57.809 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:57.809 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-21T09:46:57.813 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T09:46:57.815 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-09-21T09:46:57.815 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T09:46:57.897 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install librbd1 2024-09-21T09:46:58.096 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install librados-devel 2024-09-21T09:46:58.486 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:29 ago on Sat 21 Sep 2024 09:45:29 AM UTC. 2024-09-21T09:46:58.575 INFO:teuthology.orchestra.run.smithi160.stdout:Package librbd1-2:18.2.4-871.g6a1bdf85.el9.x86_64 is already installed. 2024-09-21T09:46:58.618 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T09:46:58.619 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-09-21T09:46:58.620 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T09:46:58.681 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:25 ago on Sat 21 Sep 2024 09:45:33 AM UTC. 2024-09-21T09:46:58.689 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-rados 2024-09-21T09:46:58.818 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-21T09:46:58.819 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:46:58.819 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repo Size 2024-09-21T09:46:58.819 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:46:58.819 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-21T09:46:58.819 INFO:teuthology.orchestra.run.smithi070.stdout: librados-devel x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 126 k 2024-09-21T09:46:58.819 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:58.819 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-21T09:46:58.820 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:46:58.820 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-09-21T09:46:58.820 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:46:58.820 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 126 k 2024-09-21T09:46:58.820 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 456 k 2024-09-21T09:46:58.820 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-21T09:46:59.020 INFO:teuthology.orchestra.run.smithi070.stdout:librados-devel-18.2.4-871.g6a1bdf85.el9.x86_64. 632 kB/s | 126 kB 00:00 2024-09-21T09:46:59.021 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:46:59.021 INFO:teuthology.orchestra.run.smithi070.stdout:Total 627 kB/s | 126 kB 00:00 2024-09-21T09:46:59.021 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-21T09:46:59.027 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-21T09:46:59.027 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-21T09:46:59.089 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-21T09:46:59.090 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-21T09:46:59.242 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-21T09:46:59.267 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:30 ago on Sat 21 Sep 2024 09:45:29 AM UTC. 2024-09-21T09:46:59.343 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librados-devel-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:46:59.356 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-rados-2:18.2.4-871.g6a1bdf85.el9.x86_64 is already installed. 2024-09-21T09:46:59.399 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T09:46:59.400 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-09-21T09:46:59.400 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T09:46:59.482 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-rgw 2024-09-21T09:46:59.772 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados-devel-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:47:00.037 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados-devel-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:47:00.037 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:47:00.037 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-21T09:47:00.037 INFO:teuthology.orchestra.run.smithi070.stdout: librados-devel-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:47:00.037 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:47:00.038 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-21T09:47:00.064 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:31 ago on Sat 21 Sep 2024 09:45:29 AM UTC. 2024-09-21T09:47:00.153 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-rgw-2:18.2.4-871.g6a1bdf85.el9.x86_64 is already installed. 2024-09-21T09:47:00.197 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T09:47:00.198 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-09-21T09:47:00.198 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T09:47:00.271 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install libcephfs2 2024-09-21T09:47:00.290 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-cephfs 2024-09-21T09:47:00.863 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:27 ago on Sat 21 Sep 2024 09:45:33 AM UTC. 2024-09-21T09:47:00.868 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:31 ago on Sat 21 Sep 2024 09:45:29 AM UTC. 2024-09-21T09:47:00.954 INFO:teuthology.orchestra.run.smithi070.stdout:Package libcephfs2-2:18.2.4-871.g6a1bdf85.el9.x86_64 is already installed. 2024-09-21T09:47:00.958 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-cephfs-2:18.2.4-871.g6a1bdf85.el9.x86_64 is already installed. 2024-09-21T09:47:00.999 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-21T09:47:01.001 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-09-21T09:47:01.001 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-21T09:47:01.002 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T09:47:01.004 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-09-21T09:47:01.004 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T09:47:01.071 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install libcephfs-devel 2024-09-21T09:47:01.089 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-rbd 2024-09-21T09:47:01.650 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:28 ago on Sat 21 Sep 2024 09:45:33 AM UTC. 2024-09-21T09:47:01.677 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:32 ago on Sat 21 Sep 2024 09:45:29 AM UTC. 2024-09-21T09:47:01.768 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-rbd-2:18.2.4-871.g6a1bdf85.el9.x86_64 is already installed. 2024-09-21T09:47:01.780 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-21T09:47:01.781 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:47:01.781 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repo Size 2024-09-21T09:47:01.781 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:47:01.782 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-21T09:47:01.782 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs-devel x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 31 k 2024-09-21T09:47:01.782 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:47:01.782 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-21T09:47:01.782 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:47:01.782 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-09-21T09:47:01.782 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:47:01.782 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 31 k 2024-09-21T09:47:01.783 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 139 k 2024-09-21T09:47:01.783 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-21T09:47:01.811 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T09:47:01.812 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-09-21T09:47:01.812 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T09:47:01.898 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install rbd-fuse 2024-09-21T09:47:01.957 INFO:teuthology.orchestra.run.smithi070.stdout:libcephfs-devel-18.2.4-871.g6a1bdf85.el9.x86_64 177 kB/s | 31 kB 00:00 2024-09-21T09:47:01.957 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:47:01.958 INFO:teuthology.orchestra.run.smithi070.stdout:Total 176 kB/s | 31 kB 00:00 2024-09-21T09:47:01.958 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-21T09:47:01.961 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-21T09:47:01.961 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-21T09:47:01.975 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-21T09:47:01.975 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-21T09:47:02.065 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-21T09:47:02.181 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephfs-devel-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:47:02.484 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:33 ago on Sat 21 Sep 2024 09:45:29 AM UTC. 2024-09-21T09:47:02.546 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:47:02.618 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T09:47:02.618 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:47:02.618 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2024-09-21T09:47:02.618 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:47:02.619 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-21T09:47:02.619 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-fuse x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 87 k 2024-09-21T09:47:02.619 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:47:02.619 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-21T09:47:02.619 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:47:02.619 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-09-21T09:47:02.619 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:47:02.620 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 87 k 2024-09-21T09:47:02.620 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 227 k 2024-09-21T09:47:02.620 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-21T09:47:02.800 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephfs-devel-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:47:02.800 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:47:02.800 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-21T09:47:02.800 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs-devel-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:47:02.800 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:47:02.800 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-21T09:47:02.812 INFO:teuthology.orchestra.run.smithi160.stdout:rbd-fuse-18.2.4-871.g6a1bdf85.el9.x86_64.rpm 453 kB/s | 87 kB 00:00 2024-09-21T09:47:02.813 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:47:02.813 INFO:teuthology.orchestra.run.smithi160.stdout:Total 450 kB/s | 87 kB 00:00 2024-09-21T09:47:02.813 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-21T09:47:02.820 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-21T09:47:02.820 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-21T09:47:02.882 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-21T09:47:02.883 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-21T09:47:03.026 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install librados2 2024-09-21T09:47:03.026 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-21T09:47:03.125 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-fuse-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:47:03.516 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: rbd-fuse-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:47:03.604 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:30 ago on Sat 21 Sep 2024 09:45:33 AM UTC. 2024-09-21T09:47:03.693 INFO:teuthology.orchestra.run.smithi070.stdout:Package librados2-2:18.2.4-871.g6a1bdf85.el9.x86_64 is already installed. 2024-09-21T09:47:03.736 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-21T09:47:03.737 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-09-21T09:47:03.737 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-21T09:47:03.773 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-fuse-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:47:03.773 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:47:03.773 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-21T09:47:03.773 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-fuse-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:47:03.773 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:47:03.773 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T09:47:03.804 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install librbd1 2024-09-21T09:47:04.030 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install rbd-mirror 2024-09-21T09:47:04.378 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:31 ago on Sat 21 Sep 2024 09:45:33 AM UTC. 2024-09-21T09:47:04.466 INFO:teuthology.orchestra.run.smithi070.stdout:Package librbd1-2:18.2.4-871.g6a1bdf85.el9.x86_64 is already installed. 2024-09-21T09:47:04.511 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-21T09:47:04.512 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-09-21T09:47:04.512 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-21T09:47:04.590 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install python3-rados 2024-09-21T09:47:04.628 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:35 ago on Sat 21 Sep 2024 09:45:29 AM UTC. 2024-09-21T09:47:04.765 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T09:47:04.766 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:47:04.766 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repo Size 2024-09-21T09:47:04.766 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:47:04.766 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-21T09:47:04.766 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-mirror x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 3.0 M 2024-09-21T09:47:04.766 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:47:04.767 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-21T09:47:04.767 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:47:04.767 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-09-21T09:47:04.767 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:47:04.767 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 3.0 M 2024-09-21T09:47:04.767 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 12 M 2024-09-21T09:47:04.767 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-21T09:47:05.107 INFO:teuthology.orchestra.run.smithi160.stdout:rbd-mirror-18.2.4-871.g6a1bdf85.el9.x86_64.rpm 8.8 MB/s | 3.0 MB 00:00 2024-09-21T09:47:05.108 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:47:05.108 INFO:teuthology.orchestra.run.smithi160.stdout:Total 8.8 MB/s | 3.0 MB 00:00 2024-09-21T09:47:05.108 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-21T09:47:05.117 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-21T09:47:05.117 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-21T09:47:05.165 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:32 ago on Sat 21 Sep 2024 09:45:33 AM UTC. 2024-09-21T09:47:05.197 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-21T09:47:05.198 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-21T09:47:05.253 INFO:teuthology.orchestra.run.smithi070.stdout:Package python3-rados-2:18.2.4-871.g6a1bdf85.el9.x86_64 is already installed. 2024-09-21T09:47:05.296 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-21T09:47:05.297 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-09-21T09:47:05.297 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-21T09:47:05.381 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install python3-rgw 2024-09-21T09:47:05.626 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-21T09:47:05.691 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-mirror-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:47:05.716 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: rbd-mirror-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:47:05.716 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:47:05.716 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T09:47:05.716 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T09:47:05.716 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T09:47:05.717 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:47:05.964 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:32 ago on Sat 21 Sep 2024 09:45:33 AM UTC. 2024-09-21T09:47:06.052 INFO:teuthology.orchestra.run.smithi070.stdout:Package python3-rgw-2:18.2.4-871.g6a1bdf85.el9.x86_64 is already installed. 2024-09-21T09:47:06.095 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-21T09:47:06.097 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-09-21T09:47:06.097 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-21T09:47:06.162 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install python3-cephfs 2024-09-21T09:47:06.737 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:33 ago on Sat 21 Sep 2024 09:45:33 AM UTC. 2024-09-21T09:47:06.825 INFO:teuthology.orchestra.run.smithi070.stdout:Package python3-cephfs-2:18.2.4-871.g6a1bdf85.el9.x86_64 is already installed. 2024-09-21T09:47:06.848 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-mirror-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:47:06.849 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:47:06.849 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-21T09:47:06.849 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-mirror-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:47:06.849 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:47:06.849 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T09:47:06.867 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-21T09:47:06.869 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-09-21T09:47:06.869 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-21T09:47:06.940 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install python3-rbd 2024-09-21T09:47:07.093 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install rbd-nbd 2024-09-21T09:47:07.520 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:34 ago on Sat 21 Sep 2024 09:45:33 AM UTC. 2024-09-21T09:47:07.608 INFO:teuthology.orchestra.run.smithi070.stdout:Package python3-rbd-2:18.2.4-871.g6a1bdf85.el9.x86_64 is already installed. 2024-09-21T09:47:07.651 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-21T09:47:07.652 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-09-21T09:47:07.652 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-21T09:47:07.695 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:38 ago on Sat 21 Sep 2024 09:45:29 AM UTC. 2024-09-21T09:47:07.722 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install rbd-fuse 2024-09-21T09:47:07.828 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-21T09:47:07.828 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:47:07.828 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2024-09-21T09:47:07.828 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:47:07.828 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-21T09:47:07.828 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-nbd x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 172 k 2024-09-21T09:47:07.828 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:47:07.828 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-21T09:47:07.829 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-21T09:47:07.829 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-09-21T09:47:07.829 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:47:07.829 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 172 k 2024-09-21T09:47:07.829 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 494 k 2024-09-21T09:47:07.829 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-21T09:47:08.035 INFO:teuthology.orchestra.run.smithi160.stdout:rbd-nbd-18.2.4-871.g6a1bdf85.el9.x86_64.rpm 837 kB/s | 172 kB 00:00 2024-09-21T09:47:08.035 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:47:08.036 INFO:teuthology.orchestra.run.smithi160.stdout:Total 830 kB/s | 172 kB 00:00 2024-09-21T09:47:08.036 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-21T09:47:08.044 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-21T09:47:08.044 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-21T09:47:08.108 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-21T09:47:08.109 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-21T09:47:08.277 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-21T09:47:08.304 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:35 ago on Sat 21 Sep 2024 09:45:33 AM UTC. 2024-09-21T09:47:08.391 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-nbd-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:47:08.435 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-21T09:47:08.436 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:47:08.436 INFO:teuthology.orchestra.run.smithi070.stdout: Package Architecture Version Repository Size 2024-09-21T09:47:08.436 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:47:08.436 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-21T09:47:08.436 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-fuse x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 87 k 2024-09-21T09:47:08.436 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:47:08.436 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-21T09:47:08.436 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:47:08.436 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-09-21T09:47:08.436 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:47:08.437 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 87 k 2024-09-21T09:47:08.437 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 227 k 2024-09-21T09:47:08.437 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-21T09:47:08.619 INFO:teuthology.orchestra.run.smithi070.stdout:rbd-fuse-18.2.4-871.g6a1bdf85.el9.x86_64.rpm 480 kB/s | 87 kB 00:00 2024-09-21T09:47:08.620 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:47:08.620 INFO:teuthology.orchestra.run.smithi070.stdout:Total 476 kB/s | 87 kB 00:00 2024-09-21T09:47:08.620 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-21T09:47:08.627 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-21T09:47:08.627 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-21T09:47:08.690 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-21T09:47:08.690 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-21T09:47:08.825 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: rbd-nbd-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:47:08.841 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-21T09:47:08.949 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-fuse-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:47:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-nbd-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:47:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:47:09.090 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-21T09:47:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-nbd-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:47:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:47:09.091 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-21T09:47:09.332 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: rbd-fuse-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:47:09.355 DEBUG:teuthology.parallel:result is None 2024-09-21T09:47:09.614 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-fuse-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:47:09.615 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:47:09.615 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-21T09:47:09.615 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-fuse-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:47:09.615 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:47:09.615 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-21T09:47:09.852 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install rbd-mirror 2024-09-21T09:47:10.447 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:37 ago on Sat 21 Sep 2024 09:45:33 AM UTC. 2024-09-21T09:47:10.581 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-21T09:47:10.582 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:47:10.582 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repo Size 2024-09-21T09:47:10.582 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:47:10.582 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-21T09:47:10.582 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-mirror x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 3.0 M 2024-09-21T09:47:10.582 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:47:10.582 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-21T09:47:10.582 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:47:10.582 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-09-21T09:47:10.583 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:47:10.583 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 3.0 M 2024-09-21T09:47:10.583 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 12 M 2024-09-21T09:47:10.583 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-21T09:47:10.906 INFO:teuthology.orchestra.run.smithi070.stdout:rbd-mirror-18.2.4-871.g6a1bdf85.el9.x86_64.rpm 9.3 MB/s | 3.0 MB 00:00 2024-09-21T09:47:10.907 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:47:10.907 INFO:teuthology.orchestra.run.smithi070.stdout:Total 9.2 MB/s | 3.0 MB 00:00 2024-09-21T09:47:10.907 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-21T09:47:10.916 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-21T09:47:10.916 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-21T09:47:10.996 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-21T09:47:10.997 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-21T09:47:11.413 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-21T09:47:11.474 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-mirror-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:47:11.499 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: rbd-mirror-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:47:11.499 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:47:11.499 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T09:47:11.499 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T09:47:11.499 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T09:47:11.499 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:47:12.600 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-mirror-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:47:12.600 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:47:12.600 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-21T09:47:12.600 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-mirror-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:47:12.600 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:47:12.600 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-21T09:47:12.880 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install rbd-nbd 2024-09-21T09:47:13.481 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:40 ago on Sat 21 Sep 2024 09:45:33 AM UTC. 2024-09-21T09:47:13.615 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-21T09:47:13.615 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:47:13.615 INFO:teuthology.orchestra.run.smithi070.stdout: Package Architecture Version Repository Size 2024-09-21T09:47:13.615 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:47:13.615 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-21T09:47:13.615 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-nbd x86_64 2:18.2.4-871.g6a1bdf85.el9 ceph 172 k 2024-09-21T09:47:13.616 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:47:13.616 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-21T09:47:13.616 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-21T09:47:13.616 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-09-21T09:47:13.616 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:47:13.616 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 172 k 2024-09-21T09:47:13.616 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 494 k 2024-09-21T09:47:13.617 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-21T09:47:13.826 INFO:teuthology.orchestra.run.smithi070.stdout:rbd-nbd-18.2.4-871.g6a1bdf85.el9.x86_64.rpm 819 kB/s | 172 kB 00:00 2024-09-21T09:47:13.827 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:47:13.827 INFO:teuthology.orchestra.run.smithi070.stdout:Total 814 kB/s | 172 kB 00:00 2024-09-21T09:47:13.827 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-21T09:47:13.835 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-21T09:47:13.835 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-21T09:47:13.899 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-21T09:47:13.899 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-21T09:47:14.071 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-21T09:47:14.156 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-nbd-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:47:14.573 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: rbd-nbd-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:47:14.849 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-nbd-2:18.2.4-871.g6a1bdf85.el9.x86_64 1/1 2024-09-21T09:47:14.849 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:47:14.849 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-21T09:47:14.849 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-nbd-2:18.2.4-871.g6a1bdf85.el9.x86_64 2024-09-21T09:47:14.849 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:47:14.849 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-21T09:47:15.120 DEBUG:teuthology.parallel:result is None 2024-09-21T09:47:15.120 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:47:15.120 INFO:teuthology.packaging:ref: None 2024-09-21T09:47:15.120 INFO:teuthology.packaging:tag: None 2024-09-21T09:47:15.120 INFO:teuthology.packaging:branch: reef 2024-09-21T09:47:15.120 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:47:15.121 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-09-21T09:47:15.396 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T09:47:15.417 INFO:teuthology.orchestra.run.smithi070.stdout:18.2.4-871.g6a1bdf85.el9 2024-09-21T09:47:15.417 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-871.g6a1bdf85.el9 2024-09-21T09:47:15.417 INFO:teuthology.task.install:The correct ceph version 18.2.4-871.g6a1bdf85 is installed. 2024-09-21T09:47:15.419 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:47:15.419 INFO:teuthology.packaging:ref: None 2024-09-21T09:47:15.419 INFO:teuthology.packaging:tag: None 2024-09-21T09:47:15.419 INFO:teuthology.packaging:branch: reef 2024-09-21T09:47:15.419 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:47:15.419 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-09-21T09:47:15.688 DEBUG:teuthology.orchestra.run.smithi160:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T09:47:15.714 INFO:teuthology.orchestra.run.smithi160.stdout:18.2.4-871.g6a1bdf85.el9 2024-09-21T09:47:15.714 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-871.g6a1bdf85.el9 2024-09-21T09:47:15.714 INFO:teuthology.task.install:The correct ceph version 18.2.4-871.g6a1bdf85 is installed. 2024-09-21T09:47:15.716 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-21T09:47:15.716 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T09:47:15.716 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T09:47:15.742 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T09:47:15.742 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T09:47:15.783 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-21T09:47:15.783 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T09:47:15.783 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T09:47:15.822 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T09:47:15.886 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T09:47:15.887 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T09:47:15.914 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T09:47:15.982 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-21T09:47:15.983 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T09:47:15.983 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T09:47:16.008 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T09:47:16.072 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T09:47:16.072 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T09:47:16.107 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T09:47:16.173 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-21T09:47:16.173 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T09:47:16.173 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T09:47:16.197 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T09:47:16.262 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T09:47:16.262 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T09:47:16.287 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T09:47:16.352 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-21T09:47:16.440 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}}, 'image': 'quay.ceph.io/ceph-ci/ceph:reef', 'create_rbd_pool': True, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'mons down', 'mon down', 'MON_DOWN', 'out of quorum', 'PG_AVAILABILITY'], 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'} 2024-09-21T09:47:16.440 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:reef 2024-09-21T09:47:16.441 INFO:tasks.cephadm:Cluster fsid is 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T09:47:16.441 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-21T09:47:16.441 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.70', 'mon.c': '[v2:172.21.15.70:3301,v1:172.21.15.70:6790]', 'mon.b': '172.21.15.160'} 2024-09-21T09:47:16.441 INFO:tasks.cephadm:First mon is mon.a on smithi070 2024-09-21T09:47:16.441 INFO:tasks.cephadm:First mgr is y 2024-09-21T09:47:16.441 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-21T09:47:16.442 DEBUG:teuthology.orchestra.run.smithi070:> sudo hostname $(hostname -s) 2024-09-21T09:47:16.467 DEBUG:teuthology.orchestra.run.smithi160:> sudo hostname $(hostname -s) 2024-09-21T09:47:16.493 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-09-21T09:47:16.493 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:47:16.632 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/56129/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-5092-g62da54b4', 'node_name': '172.21.2.5+braggi05', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-5092.g62da54b4'}, 'url': 'https://3.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-20 21:56:15.050681', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'main', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-09-21T09:47:16.903 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref reef, sha1 6a1bdf854bb5803efd58957907442aa01f41a8fc from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-09-21T09:47:16.905 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/reef/6a1bdf854bb5803efd58957907442aa01f41a8fc/centos/9/x86_64/flavors/default/cephadm 2024-09-21T09:47:16.905 INFO:tasks.cephadm:Downloading cephadm from url: https://4.chacra.ceph.com/binaries/ceph/reef/6a1bdf854bb5803efd58957907442aa01f41a8fc/centos/9/x86_64/flavors/default/cephadm 2024-09-21T09:47:16.905 DEBUG:teuthology.orchestra.run.smithi070:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/reef/6a1bdf854bb5803efd58957907442aa01f41a8fc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T09:47:17.392 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-r--r--. 1 ubuntu ubuntu 217222 Sep 21 09:47 /home/ubuntu/cephtest/cephadm 2024-09-21T09:47:17.392 DEBUG:teuthology.orchestra.run.smithi160:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/reef/6a1bdf854bb5803efd58957907442aa01f41a8fc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T09:47:17.860 INFO:teuthology.orchestra.run.smithi160.stdout:-rw-r--r--. 1 ubuntu ubuntu 217222 Sep 21 09:47 /home/ubuntu/cephtest/cephadm 2024-09-21T09:47:17.861 DEBUG:teuthology.orchestra.run.smithi070:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T09:47:17.880 DEBUG:teuthology.orchestra.run.smithi160:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T09:47:17.906 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:reef on all hosts... 2024-09-21T09:47:17.907 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-09-21T09:47:17.922 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-09-21T09:47:19.460 INFO:teuthology.orchestra.run.smithi160.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-09-21T09:47:19.519 INFO:teuthology.orchestra.run.smithi070.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-09-21T09:47:45.263 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T09:47:45.263 INFO:teuthology.orchestra.run.smithi160.stdout: "ceph_version": "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)", 2024-09-21T09:47:45.264 INFO:teuthology.orchestra.run.smithi160.stdout: "image_id": "0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de", 2024-09-21T09:47:45.264 INFO:teuthology.orchestra.run.smithi160.stdout: "repo_digests": [ 2024-09-21T09:47:45.264 INFO:teuthology.orchestra.run.smithi160.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6" 2024-09-21T09:47:45.264 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T09:47:45.264 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T09:47:50.180 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T09:47:50.180 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_version": "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)", 2024-09-21T09:47:50.180 INFO:teuthology.orchestra.run.smithi070.stdout: "image_id": "0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de", 2024-09-21T09:47:50.180 INFO:teuthology.orchestra.run.smithi070.stdout: "repo_digests": [ 2024-09-21T09:47:50.180 INFO:teuthology.orchestra.run.smithi070.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6" 2024-09-21T09:47:50.180 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T09:47:50.180 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T09:47:50.198 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /etc/ceph 2024-09-21T09:47:50.250 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /etc/ceph 2024-09-21T09:47:50.283 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 777 /etc/ceph 2024-09-21T09:47:50.307 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 777 /etc/ceph 2024-09-21T09:47:50.352 INFO:tasks.cephadm:Writing seed config... 2024-09-21T09:47:50.353 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-09-21T09:47:50.353 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-09-21T09:47:50.353 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-21T09:47:50.353 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-21T09:47:50.353 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-21T09:47:50.353 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-09-21T09:47:50.353 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-21T09:47:50.353 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-21T09:47:50.353 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-21T09:47:50.353 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-21T09:47:50.353 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-21T09:47:50.354 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T09:47:50.354 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-21T09:47:50.369 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 7c68bfbc-77fe-11ef-baf6-efdc52797490 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-21T09:47:50.369 DEBUG:teuthology.orchestra.run.smithi070:mon.a> sudo journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mon.a.service 2024-09-21T09:47:50.412 DEBUG:teuthology.orchestra.run.smithi070:mgr.y> sudo journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mgr.y.service 2024-09-21T09:47:50.454 INFO:tasks.cephadm:Bootstrapping... 2024-09-21T09:47:50.454 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef -v bootstrap --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.70 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-21T09:47:50.605 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:47:50.606 INFO:teuthology.orchestra.run.smithi070.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:reef', '-v', 'bootstrap', '--fsid', '7c68bfbc-77fe-11ef-baf6-efdc52797490', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.70', '--skip-admin-label'] 2024-09-21T09:47:50.640 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stdout 5.2.2 2024-09-21T09:47:50.640 INFO:teuthology.orchestra.run.smithi070.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-21T09:47:50.641 INFO:teuthology.orchestra.run.smithi070.stdout:Verifying podman|docker is present... 2024-09-21T09:47:50.662 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stdout 5.2.2 2024-09-21T09:47:50.662 INFO:teuthology.orchestra.run.smithi070.stdout:Verifying lvm2 is present... 2024-09-21T09:47:50.662 INFO:teuthology.orchestra.run.smithi070.stdout:Verifying time synchronization is in place... 2024-09-21T09:47:50.670 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T09:47:50.670 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T09:47:50.676 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T09:47:50.676 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout inactive 2024-09-21T09:47:50.682 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout enabled 2024-09-21T09:47:50.689 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout active 2024-09-21T09:47:50.689 INFO:teuthology.orchestra.run.smithi070.stdout:Unit chronyd.service is enabled and running 2024-09-21T09:47:50.689 INFO:teuthology.orchestra.run.smithi070.stdout:Repeating the final host check... 2024-09-21T09:47:50.711 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stdout 5.2.2 2024-09-21T09:47:50.711 INFO:teuthology.orchestra.run.smithi070.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-21T09:47:50.711 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl is present 2024-09-21T09:47:50.711 INFO:teuthology.orchestra.run.smithi070.stdout:lvcreate is present 2024-09-21T09:47:50.717 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T09:47:50.717 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T09:47:50.723 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T09:47:50.723 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout inactive 2024-09-21T09:47:50.730 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout enabled 2024-09-21T09:47:50.736 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout active 2024-09-21T09:47:50.736 INFO:teuthology.orchestra.run.smithi070.stdout:Unit chronyd.service is enabled and running 2024-09-21T09:47:50.736 INFO:teuthology.orchestra.run.smithi070.stdout:Host looks OK 2024-09-21T09:47:50.736 INFO:teuthology.orchestra.run.smithi070.stdout:Cluster fsid: 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T09:47:50.737 INFO:teuthology.orchestra.run.smithi070.stdout:Acquiring lock 139927191826000 on /run/cephadm/7c68bfbc-77fe-11ef-baf6-efdc52797490.lock 2024-09-21T09:47:50.737 INFO:teuthology.orchestra.run.smithi070.stdout:Lock 139927191826000 acquired on /run/cephadm/7c68bfbc-77fe-11ef-baf6-efdc52797490.lock 2024-09-21T09:47:50.737 INFO:teuthology.orchestra.run.smithi070.stdout:Verifying IP 172.21.15.70 port 3300 ... 2024-09-21T09:47:50.737 INFO:teuthology.orchestra.run.smithi070.stdout:Verifying IP 172.21.15.70 port 6789 ... 2024-09-21T09:47:50.737 INFO:teuthology.orchestra.run.smithi070.stdout:Base mon IP(s) is [172.21.15.70:3300, 172.21.15.70:6789], mon addrv is [v2:172.21.15.70:3300,v1:172.21.15.70:6789] 2024-09-21T09:47:50.740 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.70 metric 100 2024-09-21T09:47:50.740 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.70 metric 100 2024-09-21T09:47:50.743 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-21T09:47:50.743 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-21T09:47:50.743 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-21T09:47:50.745 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-21T09:47:50.745 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-21T09:47:50.745 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T09:47:50.745 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-21T09:47:50.745 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d3e7/64 scope link noprefixroute 2024-09-21T09:47:50.745 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T09:47:50.746 INFO:teuthology.orchestra.run.smithi070.stdout:Mon IP `172.21.15.70` is in CIDR network `172.21.0.0/20` 2024-09-21T09:47:50.746 INFO:teuthology.orchestra.run.smithi070.stdout:Mon IP `172.21.15.70` is in CIDR network `172.21.0.0/20` 2024-09-21T09:47:50.746 INFO:teuthology.orchestra.run.smithi070.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-21T09:47:50.746 INFO:teuthology.orchestra.run.smithi070.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-21T09:47:50.747 INFO:teuthology.orchestra.run.smithi070.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-09-21T09:47:51.249 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stdout 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de 2024-09-21T09:47:51.249 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:reef... 2024-09-21T09:47:51.249 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stderr Getting image source signatures 2024-09-21T09:47:51.249 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stderr Copying blob sha256:b51f6d1e10fe8de9e06c08c0a23ce0010d2bef42d30a6f89f3ac07fd857715ba 2024-09-21T09:47:51.249 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-21T09:47:51.249 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stderr Copying config sha256:0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de 2024-09-21T09:47:51.249 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-21T09:47:52.170 INFO:teuthology.orchestra.run.smithi070.stdout:ceph: stdout ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable) 2024-09-21T09:47:52.170 INFO:teuthology.orchestra.run.smithi070.stdout:Ceph version: ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable) 2024-09-21T09:47:52.170 INFO:teuthology.orchestra.run.smithi070.stdout:Extracting ceph user uid/gid from container image... 2024-09-21T09:47:53.057 INFO:teuthology.orchestra.run.smithi070.stdout:stat: stdout 167 167 2024-09-21T09:47:53.057 INFO:teuthology.orchestra.run.smithi070.stdout:Creating initial keys... 2024-09-21T09:47:54.147 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph-authtool: stdout AQBJlu5mP0RCGRAAWizYOM2MMumAQ3VpfmsOjQ== 2024-09-21T09:47:55.050 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph-authtool: stdout AQBKlu5mi5oBHRAAXH6wulQiP5JMSl5RQnjSgQ== 2024-09-21T09:47:55.948 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph-authtool: stdout AQBLlu5mZYxXFxAA9bwdoFmyalk6qlwJOQX4yg== 2024-09-21T09:47:55.948 INFO:teuthology.orchestra.run.smithi070.stdout:Creating initial monmap... 2024-09-21T09:47:56.987 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T09:47:56.987 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-09-21T09:47:56.987 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T09:47:56.988 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T09:47:56.988 INFO:teuthology.orchestra.run.smithi070.stdout:monmaptool for a [v2:172.21.15.70:3300,v1:172.21.15.70:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T09:47:56.988 INFO:teuthology.orchestra.run.smithi070.stdout:setting min_mon_release = pacific 2024-09-21T09:47:56.988 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/monmaptool: set fsid to 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T09:47:56.988 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T09:47:56.988 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:47:56.988 INFO:teuthology.orchestra.run.smithi070.stdout:Creating mon... 2024-09-21T09:47:57.850 INFO:teuthology.orchestra.run.smithi070.stdout:create mon.a on 2024-09-21T09:47:58.049 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-21T09:47:58.185 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-21T09:47:58.357 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490.target → /etc/systemd/system/ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490.target. 2024-09-21T09:47:58.357 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490.target → /etc/systemd/system/ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490.target. 2024-09-21T09:47:58.594 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mon.a 2024-09-21T09:47:58.594 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Failed to reset failed state of unit ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mon.a.service: Unit ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mon.a.service not loaded. 2024-09-21T09:47:58.753 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490.target.wants/ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mon.a.service → /etc/systemd/system/ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@.service. 2024-09-21T09:47:59.238 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:47:59 smithi070 bash[29084]: 1b9bcddf6519f099b9f09aa4e2a3b2592e077f9443618ff8a5003c65e5ff5a64 2024-09-21T09:47:59.249 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T09:47:59.249 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout disabled 2024-09-21T09:47:59.256 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T09:47:59.256 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout inactive 2024-09-21T09:47:59.256 INFO:teuthology.orchestra.run.smithi070.stdout:firewalld.service is not enabled 2024-09-21T09:47:59.256 INFO:teuthology.orchestra.run.smithi070.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T09:47:59.257 INFO:teuthology.orchestra.run.smithi070.stdout:Waiting for mon to start... 2024-09-21T09:47:59.257 INFO:teuthology.orchestra.run.smithi070.stdout:Waiting for mon... 2024-09-21T09:47:59.564 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:47:59 smithi070 systemd[1]: Started Ceph mon.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:48:00.305 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout cluster: 2024-09-21T09:48:00.305 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout id: 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T09:48:00.305 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-21T09:48:00.305 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-09-21T09:48:00.305 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout services: 2024-09-21T09:48:00.305 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.492403s) 2024-09-21T09:48:00.305 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-21T09:48:00.306 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-21T09:48:00.306 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-09-21T09:48:00.306 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout data: 2024-09-21T09:48:00.306 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-21T09:48:00.306 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-21T09:48:00.306 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-21T09:48:00.306 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout pgs: 2024-09-21T09:48:00.306 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-09-21T09:48:00.306 INFO:teuthology.orchestra.run.smithi070.stdout:mon is available 2024-09-21T09:48:00.306 INFO:teuthology.orchestra.run.smithi070.stdout:Assimilating anything we can from ceph.conf... 2024-09-21T09:48:00.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:00 smithi070 ceph-mon[29123]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T09:48:00.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:00 smithi070 ceph-mon[29123]: monmap e1: 1 mons at {a=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:48:00.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:00 smithi070 ceph-mon[29123]: fsmap 2024-09-21T09:48:00.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:00 smithi070 ceph-mon[29123]: osdmap e1: 0 total, 0 up, 0 in 2024-09-21T09:48:00.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:00 smithi070 ceph-mon[29123]: mgrmap e1: no daemons active 2024-09-21T09:48:00.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:00 smithi070 ceph-mon[29123]: from='client.? 172.21.15.70:0/843038019' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-21T09:48:01.347 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:01 smithi070 ceph-mon[29123]: from='client.? 172.21.15.70:0/2892133704' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-21T09:48:01.347 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:01 smithi070 ceph-mon[29123]: from='client.? 172.21.15.70:0/2892133704' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-21T09:48:01.351 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-09-21T09:48:01.351 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T09:48:01.351 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout fsid = 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T09:48:01.351 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.70:3300,v1:172.21.15.70:6789] 2024-09-21T09:48:01.351 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T09:48:01.351 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T09:48:01.351 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T09:48:01.351 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T09:48:01.351 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-09-21T09:48:01.351 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T09:48:01.352 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T09:48:01.352 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-09-21T09:48:01.352 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T09:48:01.352 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T09:48:01.352 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T09:48:01.352 INFO:teuthology.orchestra.run.smithi070.stdout:Generating new minimal ceph.conf... 2024-09-21T09:48:02.381 INFO:teuthology.orchestra.run.smithi070.stdout:Restarting the monitor... 2024-09-21T09:48:02.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:02 smithi070 ceph-mon[29123]: from='client.? 172.21.15.70:0/676341631' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:02.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:02 smithi070 systemd[1]: Stopping Ceph mon.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:48:02.997 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:02 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a[29100]: 2024-09-21T09:48:02.650+0000 7fdce1615640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T09:48:02.998 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:02 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a[29100]: 2024-09-21T09:48:02.650+0000 7fdce1615640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-21T09:48:02.998 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:02 smithi070 podman[29438]: 2024-09-21 09:48:02.743439598 +0000 UTC m=+0.186866672 container died 1b9bcddf6519f099b9f09aa4e2a3b2592e077f9443618ff8a5003c65e5ff5a64 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-6a1bdf8, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS) 2024-09-21T09:48:03.372 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 podman[29438]: 2024-09-21 09:48:03.095391761 +0000 UTC m=+0.538818834 container cleanup 1b9bcddf6519f099b9f09aa4e2a3b2592e077f9443618ff8a5003c65e5ff5a64 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=reef-6a1bdf8, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-21T09:48:03.372 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 bash[29438]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a 2024-09-21T09:48:03.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 podman[29456]: 2024-09-21 09:48:03.371168426 +0000 UTC m=+0.630120325 container remove 1b9bcddf6519f099b9f09aa4e2a3b2592e077f9443618ff8a5003c65e5ff5a64 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=reef-6a1bdf8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-21T09:48:03.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-21T09:48:03.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 systemd[1]: Stopped Ceph mon.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:48:03.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 systemd[1]: Starting Ceph mon.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:48:03.951 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 podman[29551]: 2024-09-21 09:48:03.59963647 +0000 UTC m=+0.015366866 image pull 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de quay.ceph.io/ceph-ci/ceph:reef 2024-09-21T09:48:03.951 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 podman[29551]: 2024-09-21 09:48:03.730273857 +0000 UTC m=+0.146004239 container create cee8ca713d09bd95be57c60ff1eb7e4dd7cbb22f891f84b25280cbac97388331 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a, ceph=True, RELEASE=reef-6a1bdf8, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-21T09:48:03.952 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 podman[29551]: 2024-09-21 09:48:03.924077104 +0000 UTC m=+0.339807494 container init cee8ca713d09bd95be57c60ff1eb7e4dd7cbb22f891f84b25280cbac97388331 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=reef-6a1bdf8, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T09:48:03.952 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 podman[29551]: 2024-09-21 09:48:03.92885455 +0000 UTC m=+0.344584931 container start cee8ca713d09bd95be57c60ff1eb7e4dd7cbb22f891f84b25280cbac97388331 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=reef-6a1bdf8, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T09:48:03.952 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T09:48:03.952 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable), process ceph-mon, pid 7 2024-09-21T09:48:03.952 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: pidfile_write: ignore empty --pid-file 2024-09-21T09:48:04.012 INFO:teuthology.orchestra.run.smithi070.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-21T09:48:04.213 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: load: jerasure load: lrc 2024-09-21T09:48:04.213 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: RocksDB version: 7.9.2 2024-09-21T09:48:04.213 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Git sha 0 2024-09-21T09:48:04.213 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Compile date 2024-09-20 20:34:00 2024-09-21T09:48:04.213 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: DB SUMMARY 2024-09-21T09:48:04.213 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: DB Session ID: 88VTXC33C2VC35VZESI1 2024-09-21T09:48:04.213 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: CURRENT file: CURRENT 2024-09-21T09:48:04.213 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T09:48:04.214 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-21T09:48:04.214 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-21T09:48:04.214 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84540 ; 2024-09-21T09:48:04.214 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.error_if_exists: 0 2024-09-21T09:48:04.214 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.create_if_missing: 0 2024-09-21T09:48:04.214 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.paranoid_checks: 1 2024-09-21T09:48:04.214 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T09:48:04.214 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T09:48:04.214 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T09:48:04.214 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.env: 0x560eff7febc0 2024-09-21T09:48:04.214 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T09:48:04.214 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.info_log: 0x560f000fcbe0 2024-09-21T09:48:04.215 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T09:48:04.215 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.statistics: (nil) 2024-09-21T09:48:04.215 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.use_fsync: 0 2024-09-21T09:48:04.215 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.max_log_file_size: 0 2024-09-21T09:48:04.215 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T09:48:04.215 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T09:48:04.215 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T09:48:04.215 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T09:48:04.215 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.allow_fallocate: 1 2024-09-21T09:48:04.215 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T09:48:04.215 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T09:48:04.215 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.use_direct_reads: 0 2024-09-21T09:48:04.215 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T09:48:04.215 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T09:48:04.216 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.db_log_dir: 2024-09-21T09:48:04.216 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.wal_dir: 2024-09-21T09:48:04.216 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T09:48:04.216 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T09:48:04.216 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T09:48:04.216 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T09:48:04.217 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T09:48:04.217 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T09:48:04.217 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T09:48:04.217 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T09:48:04.217 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.write_buffer_manager: 0x560f0010a320 2024-09-21T09:48:04.217 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T09:48:04.217 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T09:48:04.218 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T09:48:04.218 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T09:48:04.218 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T09:48:04.218 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T09:48:04.218 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T09:48:04.218 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T09:48:04.218 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.unordered_write: 0 2024-09-21T09:48:04.218 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T09:48:04.218 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T09:48:04.218 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T09:48:04.218 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T09:48:04.218 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.row_cache: None 2024-09-21T09:48:04.218 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.wal_filter: None 2024-09-21T09:48:04.218 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T09:48:04.219 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T09:48:04.219 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.two_write_queues: 0 2024-09-21T09:48:04.219 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T09:48:04.219 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.wal_compression: 0 2024-09-21T09:48:04.219 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.atomic_flush: 0 2024-09-21T09:48:04.219 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T09:48:04.219 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T09:48:04.219 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T09:48:04.219 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.log_readahead_size: 0 2024-09-21T09:48:04.219 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T09:48:04.219 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T09:48:04.219 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T09:48:04.219 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T09:48:04.220 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T09:48:04.220 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T09:48:04.220 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T09:48:04.220 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.max_background_jobs: 2 2024-09-21T09:48:04.220 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.max_background_compactions: -1 2024-09-21T09:48:04.220 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.max_subcompactions: 1 2024-09-21T09:48:04.220 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T09:48:04.220 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T09:48:04.220 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T09:48:04.220 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T09:48:04.220 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T09:48:04.220 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T09:48:04.220 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T09:48:04.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T09:48:04.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.max_open_files: -1 2024-09-21T09:48:04.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T09:48:04.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T09:48:04.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T09:48:04.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T09:48:04.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.max_background_flushes: -1 2024-09-21T09:48:04.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Compression algorithms supported: 2024-09-21T09:48:04.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: kZSTD supported: 0 2024-09-21T09:48:04.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: kXpressCompression supported: 0 2024-09-21T09:48:04.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: kBZip2Compression supported: 0 2024-09-21T09:48:04.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T09:48:04.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: kLZ4Compression supported: 1 2024-09-21T09:48:04.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: kZlibCompression supported: 1 2024-09-21T09:48:04.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T09:48:04.222 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: kSnappyCompression supported: 1 2024-09-21T09:48:04.222 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T09:48:04.222 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T09:48:04.222 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-21T09:48:04.222 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T09:48:04.222 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T09:48:04.222 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.merge_operator: 2024-09-21T09:48:04.222 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.compaction_filter: None 2024-09-21T09:48:04.222 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T09:48:04.222 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T09:48:04.222 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T09:48:04.222 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T09:48:04.222 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560f000fc7e0) 2024-09-21T09:48:04.223 INFO:journalctl@ceph.mon.a.smithi070.stdout: cache_index_and_filter_blocks: 1 2024-09-21T09:48:04.223 INFO:journalctl@ceph.mon.a.smithi070.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T09:48:04.223 INFO:journalctl@ceph.mon.a.smithi070.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T09:48:04.223 INFO:journalctl@ceph.mon.a.smithi070.stdout: pin_top_level_index_and_filter: 1 2024-09-21T09:48:04.223 INFO:journalctl@ceph.mon.a.smithi070.stdout: index_type: 0 2024-09-21T09:48:04.223 INFO:journalctl@ceph.mon.a.smithi070.stdout: data_block_index_type: 0 2024-09-21T09:48:04.223 INFO:journalctl@ceph.mon.a.smithi070.stdout: index_shortening: 1 2024-09-21T09:48:04.223 INFO:journalctl@ceph.mon.a.smithi070.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T09:48:04.223 INFO:journalctl@ceph.mon.a.smithi070.stdout: checksum: 4 2024-09-21T09:48:04.223 INFO:journalctl@ceph.mon.a.smithi070.stdout: no_block_cache: 0 2024-09-21T09:48:04.223 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_cache: 0x560f000f1350 2024-09-21T09:48:04.223 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_cache_name: BinnedLRUCache 2024-09-21T09:48:04.223 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_cache_options: 2024-09-21T09:48:04.223 INFO:journalctl@ceph.mon.a.smithi070.stdout: capacity : 536870912 2024-09-21T09:48:04.224 INFO:journalctl@ceph.mon.a.smithi070.stdout: num_shard_bits : 4 2024-09-21T09:48:04.224 INFO:journalctl@ceph.mon.a.smithi070.stdout: strict_capacity_limit : 0 2024-09-21T09:48:04.224 INFO:journalctl@ceph.mon.a.smithi070.stdout: high_pri_pool_ratio: 0.000 2024-09-21T09:48:04.224 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_cache_compressed: (nil) 2024-09-21T09:48:04.224 INFO:journalctl@ceph.mon.a.smithi070.stdout: persistent_cache: (nil) 2024-09-21T09:48:04.224 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_size: 4096 2024-09-21T09:48:04.224 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_size_deviation: 10 2024-09-21T09:48:04.224 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_restart_interval: 16 2024-09-21T09:48:04.224 INFO:journalctl@ceph.mon.a.smithi070.stdout: index_block_restart_interval: 1 2024-09-21T09:48:04.224 INFO:journalctl@ceph.mon.a.smithi070.stdout: metadata_block_size: 4096 2024-09-21T09:48:04.224 INFO:journalctl@ceph.mon.a.smithi070.stdout: partition_filters: 0 2024-09-21T09:48:04.224 INFO:journalctl@ceph.mon.a.smithi070.stdout: use_delta_encoding: 1 2024-09-21T09:48:04.224 INFO:journalctl@ceph.mon.a.smithi070.stdout: filter_policy: bloomfilter 2024-09-21T09:48:04.224 INFO:journalctl@ceph.mon.a.smithi070.stdout: whole_key_filtering: 1 2024-09-21T09:48:04.225 INFO:journalctl@ceph.mon.a.smithi070.stdout: verify_compression: 0 2024-09-21T09:48:04.225 INFO:journalctl@ceph.mon.a.smithi070.stdout: read_amp_bytes_per_bit: 0 2024-09-21T09:48:04.225 INFO:journalctl@ceph.mon.a.smithi070.stdout: format_version: 5 2024-09-21T09:48:04.225 INFO:journalctl@ceph.mon.a.smithi070.stdout: enable_index_compression: 1 2024-09-21T09:48:04.225 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_align: 0 2024-09-21T09:48:04.225 INFO:journalctl@ceph.mon.a.smithi070.stdout: max_auto_readahead_size: 262144 2024-09-21T09:48:04.225 INFO:journalctl@ceph.mon.a.smithi070.stdout: prepopulate_block_cache: 0 2024-09-21T09:48:04.225 INFO:journalctl@ceph.mon.a.smithi070.stdout: initial_auto_readahead_size: 8192 2024-09-21T09:48:04.225 INFO:journalctl@ceph.mon.a.smithi070.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T09:48:04.225 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T09:48:04.225 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T09:48:04.225 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.compression: NoCompression 2024-09-21T09:48:04.225 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T09:48:04.225 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T09:48:04.225 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T09:48:04.226 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.num_levels: 7 2024-09-21T09:48:04.226 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T09:48:04.226 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T09:48:04.226 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T09:48:04.226 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T09:48:04.226 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T09:48:04.226 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T09:48:04.226 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T09:48:04.226 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:48:04.226 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T09:48:04.226 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T09:48:04.226 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T09:48:04.226 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T09:48:04.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T09:48:04.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T09:48:04.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T09:48:04.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T09:48:04.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:48:04.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T09:48:04.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T09:48:04.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T09:48:04.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T09:48:04.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T09:48:04.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T09:48:04.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T09:48:04.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T09:48:04.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T09:48:04.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T09:48:04.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T09:48:04.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T09:48:04.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T09:48:04.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T09:48:04.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T09:48:04.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T09:48:04.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T09:48:04.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T09:48:04.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T09:48:04.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T09:48:04.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T09:48:04.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T09:48:04.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T09:48:04.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T09:48:04.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T09:48:04.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T09:48:04.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T09:48:04.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T09:48:04.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T09:48:04.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T09:48:04.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T09:48:04.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T09:48:04.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T09:48:04.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T09:48:04.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T09:48:04.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T09:48:04.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T09:48:04.230 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.inplace_update_support: 0 2024-09-21T09:48:04.230 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T09:48:04.230 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T09:48:04.230 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T09:48:04.230 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T09:48:04.230 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.bloom_locality: 0 2024-09-21T09:48:04.230 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.max_successive_merges: 0 2024-09-21T09:48:04.230 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T09:48:04.230 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T09:48:04.230 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T09:48:04.230 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T09:48:04.230 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.ttl: 2592000 2024-09-21T09:48:04.230 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T09:48:04.230 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T09:48:04.231 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T09:48:04.231 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.enable_blob_files: false 2024-09-21T09:48:04.231 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.min_blob_size: 0 2024-09-21T09:48:04.231 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T09:48:04.231 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T09:48:04.231 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T09:48:04.231 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T09:48:04.231 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T09:48:04.231 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T09:48:04.231 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T09:48:04.231 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T09:48:04.231 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-21T09:48:04.231 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-21T09:48:04.231 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 100e7b8e-fa1d-414c-8cff-ec7e6137fb9f 2024-09-21T09:48:04.232 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912083954962, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-21T09:48:04.232 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-21T09:48:04.232 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912083955810, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81090, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 256, "table_properties": {"data_size": 79240, "index_size": 234, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10267, "raw_average_key_size": 47, "raw_value_size": 73441, "raw_average_value_size": 341, "num_data_blocks": 10, "num_entries": 215, "num_filter_entries": 215, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726912083, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "100e7b8e-fa1d-414c-8cff-ec7e6137fb9f", "db_session_id": "88VTXC33C2VC35VZESI1", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-21T09:48:04.232 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912083955904, "job": 1, "event": "recovery_finished"} 2024-09-21T09:48:04.232 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-21T09:48:04.232 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T09:48:04.232 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560f0011ce00 2024-09-21T09:48:04.232 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: DB pointer 0x560f00210000 2024-09-21T09:48:04.232 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T09:48:04.232 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T09:48:04.232 INFO:journalctl@ceph.mon.a.smithi070.stdout: ** DB Stats ** 2024-09-21T09:48:04.232 INFO:journalctl@ceph.mon.a.smithi070.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:48:04.232 INFO:journalctl@ceph.mon.a.smithi070.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T09:48:04.232 INFO:journalctl@ceph.mon.a.smithi070.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T09:48:04.233 INFO:journalctl@ceph.mon.a.smithi070.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:48:04.233 INFO:journalctl@ceph.mon.a.smithi070.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T09:48:04.233 INFO:journalctl@ceph.mon.a.smithi070.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T09:48:04.233 INFO:journalctl@ceph.mon.a.smithi070.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:48:04.233 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-09-21T09:48:04.233 INFO:journalctl@ceph.mon.a.smithi070.stdout: ** Compaction Stats [default] ** 2024-09-21T09:48:04.233 INFO:journalctl@ceph.mon.a.smithi070.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T09:48:04.233 INFO:journalctl@ceph.mon.a.smithi070.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T09:48:04.233 INFO:journalctl@ceph.mon.a.smithi070.stdout: L0 2/0 81.05 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T09:48:04.233 INFO:journalctl@ceph.mon.a.smithi070.stdout: Sum 2/0 81.05 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T09:48:04.233 INFO:journalctl@ceph.mon.a.smithi070.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T09:48:04.233 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-09-21T09:48:04.233 INFO:journalctl@ceph.mon.a.smithi070.stdout: ** Compaction Stats [default] ** 2024-09-21T09:48:04.234 INFO:journalctl@ceph.mon.a.smithi070.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T09:48:04.234 INFO:journalctl@ceph.mon.a.smithi070.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:48:04.234 INFO:journalctl@ceph.mon.a.smithi070.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 138.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T09:48:04.234 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-09-21T09:48:04.234 INFO:journalctl@ceph.mon.a.smithi070.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T09:48:04.234 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-09-21T09:48:04.234 INFO:journalctl@ceph.mon.a.smithi070.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:48:04.234 INFO:journalctl@ceph.mon.a.smithi070.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T09:48:04.234 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T09:48:04.234 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T09:48:04.234 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T09:48:04.234 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T09:48:04.234 INFO:journalctl@ceph.mon.a.smithi070.stdout: Cumulative compaction: 0.00 GB write, 26.29 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:48:04.234 INFO:journalctl@ceph.mon.a.smithi070.stdout: Interval compaction: 0.00 GB write, 26.29 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:48:04.235 INFO:journalctl@ceph.mon.a.smithi070.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T09:48:04.235 INFO:journalctl@ceph.mon.a.smithi070.stdout: Block cache BinnedLRUCache@0x560f000f1350#7 capacity: 512.00 MB usage: 1.19 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-21T09:48:04.235 INFO:journalctl@ceph.mon.a.smithi070.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.42 KB,8.04663e-05%) Misc(1,0.00 KB,0%) 2024-09-21T09:48:04.235 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-09-21T09:48:04.235 INFO:journalctl@ceph.mon.a.smithi070.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T09:48:04.235 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: starting mon.a rank 0 at public addrs [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] at bind addrs [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T09:48:04.235 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: mon.a@-1(???) e1 preinit fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T09:48:04.235 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: mon.a@-1(???).mds e1 new map 2024-09-21T09:48:04.235 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: mon.a@-1(???).mds e1 print_map 2024-09-21T09:48:04.235 INFO:journalctl@ceph.mon.a.smithi070.stdout: e1 2024-09-21T09:48:04.235 INFO:journalctl@ceph.mon.a.smithi070.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T09:48:04.235 INFO:journalctl@ceph.mon.a.smithi070.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T09:48:04.235 INFO:journalctl@ceph.mon.a.smithi070.stdout: legacy client fscid: -1 2024-09-21T09:48:04.235 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-09-21T09:48:04.236 INFO:journalctl@ceph.mon.a.smithi070.stdout: No filesystems configured 2024-09-21T09:48:04.236 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T09:48:04.236 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:48:04.236 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:48:04.236 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:48:04.236 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:03 smithi070 ceph-mon[29589]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-21T09:48:04.236 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:04 smithi070 bash[29551]: cee8ca713d09bd95be57c60ff1eb7e4dd7cbb22f891f84b25280cbac97388331 2024-09-21T09:48:04.236 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:04 smithi070 systemd[1]: Started Ceph mon.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:48:05.073 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:04 smithi070 ceph-mon[29589]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T09:48:05.073 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:04 smithi070 ceph-mon[29589]: monmap e1: 1 mons at {a=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:48:05.073 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:04 smithi070 ceph-mon[29589]: fsmap 2024-09-21T09:48:05.073 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:04 smithi070 ceph-mon[29589]: osdmap e1: 0 total, 0 up, 0 in 2024-09-21T09:48:05.073 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:04 smithi070 ceph-mon[29589]: mgrmap e1: no daemons active 2024-09-21T09:48:05.073 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:04 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/2271968170' entity='client.admin' 2024-09-21T09:48:05.075 INFO:teuthology.orchestra.run.smithi070.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-21T09:48:05.097 INFO:teuthology.orchestra.run.smithi070.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-21T09:48:05.097 INFO:teuthology.orchestra.run.smithi070.stdout:Creating mgr... 2024-09-21T09:48:05.098 INFO:teuthology.orchestra.run.smithi070.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-21T09:48:05.098 INFO:teuthology.orchestra.run.smithi070.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-21T09:48:05.344 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mgr.y 2024-09-21T09:48:05.344 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Failed to reset failed state of unit ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mgr.y.service: Unit ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mgr.y.service not loaded. 2024-09-21T09:48:05.497 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490.target.wants/ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mgr.y.service → /etc/systemd/system/ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@.service. 2024-09-21T09:48:05.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:05 smithi070 systemd[1]: Starting Ceph mgr.y for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:48:06.010 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:05 smithi070 podman[29823]: 2024-09-21 09:48:05.723325189 +0000 UTC m=+0.113216759 container create e4814f12686d04893f1b815a20904bcd2737df4a2207e438027e94239bd0327b (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-6a1bdf8, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-21T09:48:06.010 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:05 smithi070 podman[29823]: 2024-09-21 09:48:05.626504855 +0000 UTC m=+0.016396441 image pull 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de quay.ceph.io/ceph-ci/ceph:reef 2024-09-21T09:48:06.010 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:05 smithi070 podman[29823]: 2024-09-21 09:48:05.933941916 +0000 UTC m=+0.323833493 container init e4814f12686d04893f1b815a20904bcd2737df4a2207e438027e94239bd0327b (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-6a1bdf8, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-21T09:48:06.010 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:05 smithi070 podman[29823]: 2024-09-21 09:48:05.938810719 +0000 UTC m=+0.328702289 container start e4814f12686d04893f1b815a20904bcd2737df4a2207e438027e94239bd0327b (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=reef-6a1bdf8, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-21T09:48:06.024 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T09:48:06.024 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout disabled 2024-09-21T09:48:06.031 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T09:48:06.031 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout inactive 2024-09-21T09:48:06.031 INFO:teuthology.orchestra.run.smithi070.stdout:firewalld.service is not enabled 2024-09-21T09:48:06.031 INFO:teuthology.orchestra.run.smithi070.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T09:48:06.040 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T09:48:06.040 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout disabled 2024-09-21T09:48:06.047 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T09:48:06.047 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout inactive 2024-09-21T09:48:06.048 INFO:teuthology.orchestra.run.smithi070.stdout:firewalld.service is not enabled 2024-09-21T09:48:06.048 INFO:teuthology.orchestra.run.smithi070.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-21T09:48:06.048 INFO:teuthology.orchestra.run.smithi070.stdout:Waiting for mgr to start... 2024-09-21T09:48:06.048 INFO:teuthology.orchestra.run.smithi070.stdout:Waiting for mgr... 2024-09-21T09:48:06.262 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:06 smithi070 bash[29823]: e4814f12686d04893f1b815a20904bcd2737df4a2207e438027e94239bd0327b 2024-09-21T09:48:06.263 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:06 smithi070 systemd[1]: Started Ceph mgr.y for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:48:06.263 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:06 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:06.060+0000 7f6d5dbe0040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T09:48:06.263 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:06 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:06.118+0000 7f6d5dbe0040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T09:48:06.526 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:06 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:06.273+0000 7f6d5dbe0040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T09:48:06.527 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:06 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:06.341+0000 7f6d5dbe0040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:48:06.527 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:06 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:06.492+0000 7f6d5dbe0040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:48:06.782 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:06 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/3293284535' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-21T09:48:06.782 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:06 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:06.642+0000 7f6d5dbe0040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:48:07.057 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:06 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:06.957+0000 7f6d5dbe0040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:48:07.057 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:07 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:07.017+0000 7f6d5dbe0040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:48:07.065 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-09-21T09:48:07.065 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout { 2024-09-21T09:48:07.065 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "fsid": "7c68bfbc-77fe-11ef-baf6-efdc52797490", 2024-09-21T09:48:07.065 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T09:48:07.065 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T09:48:07.065 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T09:48:07.065 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T09:48:07.065 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:48:07.065 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T09:48:07.066 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T09:48:07.066 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 0 2024-09-21T09:48:07.066 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:48:07.066 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T09:48:07.066 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T09:48:07.066 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:48:07.066 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-21T09:48:07.066 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T09:48:07.066 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:48:07.066 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-21T09:48:07.066 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T09:48:07.066 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:48:07.066 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T09:48:07.066 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:48:07.067 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T09:48:07.067 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T09:48:07.067 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T09:48:07.067 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T09:48:07.067 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T09:48:07.067 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T09:48:07.067 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:48:07.067 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T09:48:07.067 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T09:48:07.067 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T09:48:07.067 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T09:48:07.067 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T09:48:07.067 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T09:48:07.067 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T09:48:07.067 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T09:48:07.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T09:48:07.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:48:07.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T09:48:07.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:48:07.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T09:48:07.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T09:48:07.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:48:07.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T09:48:07.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-21T09:48:07.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T09:48:07.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T09:48:07.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T09:48:07.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T09:48:07.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T09:48:07.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:48:07.069 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T09:48:07.069 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:48:07.069 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T09:48:07.069 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:48:07.069 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T09:47:59.217386+0000", 2024-09-21T09:48:07.069 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T09:48:07.069 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:48:07.069 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T09:48:07.069 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout } 2024-09-21T09:48:07.069 INFO:teuthology.orchestra.run.smithi070.stdout:mgr not available, waiting (1/15)... 2024-09-21T09:48:07.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:07 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:07.200+0000 7f6d5dbe0040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:48:08.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:08 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:08.062+0000 7f6d5dbe0040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:48:08.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:08 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:08.127+0000 7f6d5dbe0040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:48:09.088 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:08 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:08.831+0000 7f6d5dbe0040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:48:09.088 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:08 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:08.943+0000 7f6d5dbe0040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:48:09.088 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:09 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:09.086+0000 7f6d5dbe0040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:48:09.368 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:09 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:09.147+0000 7f6d5dbe0040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:48:09.368 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:09 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:09.294+0000 7f6d5dbe0040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:48:09.368 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:09 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:09.366+0000 7f6d5dbe0040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:48:09.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:09 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:09.425+0000 7f6d5dbe0040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:48:09.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:09 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:09.588+0000 7f6d5dbe0040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:48:09.927 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:09 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/3692023077' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-21T09:48:09.927 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:09 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:09.659+0000 7f6d5dbe0040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:48:09.927 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:09 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:09.724+0000 7f6d5dbe0040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:48:09.927 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:09 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:09.806+0000 7f6d5dbe0040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:48:09.927 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:09 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:09.867+0000 7f6d5dbe0040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:48:09.928 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:09 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:09.925+0000 7f6d5dbe0040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:48:10.254 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-09-21T09:48:10.254 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout { 2024-09-21T09:48:10.255 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "fsid": "7c68bfbc-77fe-11ef-baf6-efdc52797490", 2024-09-21T09:48:10.255 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T09:48:10.255 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T09:48:10.255 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T09:48:10.255 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T09:48:10.255 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:48:10.255 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T09:48:10.255 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T09:48:10.255 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 0 2024-09-21T09:48:10.255 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:48:10.255 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T09:48:10.255 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T09:48:10.255 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:48:10.255 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-21T09:48:10.256 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T09:48:10.256 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:48:10.256 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-21T09:48:10.256 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T09:48:10.256 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:48:10.256 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T09:48:10.256 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:48:10.256 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T09:48:10.256 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T09:48:10.256 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T09:48:10.256 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T09:48:10.256 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T09:48:10.256 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T09:48:10.256 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:48:10.257 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T09:48:10.257 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T09:48:10.257 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T09:48:10.257 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T09:48:10.257 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T09:48:10.257 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T09:48:10.257 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T09:48:10.257 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T09:48:10.257 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T09:48:10.257 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:48:10.257 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T09:48:10.257 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:48:10.257 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T09:48:10.257 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T09:48:10.258 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:48:10.258 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T09:48:10.258 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-21T09:48:10.258 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T09:48:10.258 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T09:48:10.258 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T09:48:10.258 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T09:48:10.258 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T09:48:10.258 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:48:10.258 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T09:48:10.258 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:48:10.258 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T09:48:10.258 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:48:10.258 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T09:47:59.217386+0000", 2024-09-21T09:48:10.258 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T09:48:10.259 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:48:10.259 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T09:48:10.259 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout } 2024-09-21T09:48:10.259 INFO:teuthology.orchestra.run.smithi070.stdout:mgr not available, waiting (2/15)... 2024-09-21T09:48:11.127 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:10 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:10.699+0000 7f6d5dbe0040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:48:11.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:10 smithi070 ceph-mon[29589]: Activating manager daemon y 2024-09-21T09:48:11.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:10 smithi070 ceph-mon[29589]: mgrmap e2: y(active, starting, since 0.00283946s) 2024-09-21T09:48:11.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:10 smithi070 ceph-mon[29589]: from='mgr.14100 172.21.15.70:0/2202936934' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:48:11.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:10 smithi070 ceph-mon[29589]: from='mgr.14100 172.21.15.70:0/2202936934' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:48:11.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:10 smithi070 ceph-mon[29589]: from='mgr.14100 172.21.15.70:0/2202936934' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:48:11.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:10 smithi070 ceph-mon[29589]: from='mgr.14100 172.21.15.70:0/2202936934' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:48:11.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:10 smithi070 ceph-mon[29589]: from='mgr.14100 172.21.15.70:0/2202936934' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:48:11.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:10 smithi070 ceph-mon[29589]: Manager daemon y is now available 2024-09-21T09:48:11.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:10 smithi070 ceph-mon[29589]: from='mgr.14100 172.21.15.70:0/2202936934' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:48:11.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:10 smithi070 ceph-mon[29589]: from='mgr.14100 172.21.15.70:0/2202936934' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:48:11.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:10 smithi070 ceph-mon[29589]: from='mgr.14100 172.21.15.70:0/2202936934' entity='mgr.y' 2024-09-21T09:48:11.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:10 smithi070 ceph-mon[29589]: from='mgr.14100 172.21.15.70:0/2202936934' entity='mgr.y' 2024-09-21T09:48:11.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:10 smithi070 ceph-mon[29589]: from='mgr.14100 172.21.15.70:0/2202936934' entity='mgr.y' 2024-09-21T09:48:13.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:12 smithi070 ceph-mon[29589]: mgrmap e3: y(active, since 1.00736s) 2024-09-21T09:48:14.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:13 smithi070 ceph-mon[29589]: mgrmap e4: y(active, since 2s) 2024-09-21T09:48:14.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:13 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/217642453' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-21T09:48:17.157 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-09-21T09:48:17.157 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout { 2024-09-21T09:48:17.157 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "fsid": "7c68bfbc-77fe-11ef-baf6-efdc52797490", 2024-09-21T09:48:17.157 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T09:48:17.157 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T09:48:17.157 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T09:48:17.157 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T09:48:17.157 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:48:17.157 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T09:48:17.158 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T09:48:17.158 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 0 2024-09-21T09:48:17.158 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:48:17.158 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T09:48:17.158 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T09:48:17.158 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:48:17.158 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-09-21T09:48:17.158 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T09:48:17.158 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:48:17.158 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-21T09:48:17.158 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T09:48:17.158 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:48:17.158 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T09:48:17.159 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:48:17.159 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T09:48:17.159 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T09:48:17.159 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T09:48:17.159 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T09:48:17.159 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T09:48:17.160 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T09:48:17.160 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:48:17.160 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T09:48:17.160 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T09:48:17.160 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T09:48:17.160 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T09:48:17.160 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T09:48:17.160 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T09:48:17.160 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T09:48:17.160 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T09:48:17.160 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T09:48:17.160 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:48:17.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T09:48:17.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:48:17.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T09:48:17.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T09:48:17.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:48:17.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T09:48:17.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T09:48:17.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T09:48:17.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T09:48:17.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T09:48:17.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T09:48:17.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T09:48:17.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:48:17.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T09:48:17.162 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:48:17.162 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T09:48:17.162 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:48:17.162 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T09:47:59.217386+0000", 2024-09-21T09:48:17.162 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T09:48:17.162 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:48:17.162 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T09:48:17.162 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout } 2024-09-21T09:48:17.162 INFO:teuthology.orchestra.run.smithi070.stdout:mgr is available 2024-09-21T09:48:20.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:20 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/3075321509' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-21T09:48:26.772 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-09-21T09:48:26.772 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T09:48:26.772 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout fsid = 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T09:48:26.772 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T09:48:26.772 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T09:48:26.772 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T09:48:26.772 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T09:48:26.772 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-09-21T09:48:26.772 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T09:48:26.773 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T09:48:26.773 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-09-21T09:48:26.773 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T09:48:26.773 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T09:48:26.773 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T09:48:26.773 INFO:teuthology.orchestra.run.smithi070.stdout:Enabling cephadm module... 2024-09-21T09:48:27.822 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:27 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/2670085597' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-21T09:48:27.823 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:27 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ignoring --setuser ceph since I am not root 2024-09-21T09:48:27.823 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:27 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ignoring --setgroup ceph since I am not root 2024-09-21T09:48:27.823 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:27 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:27.639+0000 7f1d9d67b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T09:48:27.823 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:27 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:27.694+0000 7f1d9d67b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T09:48:28.098 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:27 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:27.844+0000 7f1d9d67b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T09:48:28.098 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:27 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:27.910+0000 7f1d9d67b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:48:28.098 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:28 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:28.055+0000 7f1d9d67b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:48:28.356 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:28 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:28.198+0000 7f1d9d67b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:48:28.764 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:28 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/2670085597' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-21T09:48:28.764 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:28 smithi070 ceph-mon[29589]: mgrmap e5: y(active, since 16s) 2024-09-21T09:48:28.764 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:28 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:28.513+0000 7f1d9d67b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:48:28.764 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:28 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:28.577+0000 7f1d9d67b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:48:28.764 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:28 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:28.761+0000 7f1d9d67b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:48:29.232 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout { 2024-09-21T09:48:29.232 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-21T09:48:29.232 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T09:48:29.232 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-21T09:48:29.232 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T09:48:29.232 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout } 2024-09-21T09:48:29.232 INFO:teuthology.orchestra.run.smithi070.stdout:Waiting for the mgr to restart... 2024-09-21T09:48:29.232 INFO:teuthology.orchestra.run.smithi070.stdout:Waiting for mgr epoch 5... 2024-09-21T09:48:29.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:29 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/4003559259' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-21T09:48:29.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:29 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:29.654+0000 7f1d9d67b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:48:29.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:29 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:29.723+0000 7f1d9d67b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:48:30.669 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:30 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:30.416+0000 7f1d9d67b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:48:30.670 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:30 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:30.528+0000 7f1d9d67b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:48:30.940 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:30 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:30.666+0000 7f1d9d67b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:48:30.940 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:30 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:30.727+0000 7f1d9d67b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:48:30.940 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:30 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:30.870+0000 7f1d9d67b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:48:30.940 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:30 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:30.937+0000 7f1d9d67b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:48:31.215 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:30 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:30.995+0000 7f1d9d67b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:48:31.215 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:31 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:31.149+0000 7f1d9d67b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:48:31.215 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:31 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:31.212+0000 7f1d9d67b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:48:31.476 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:31 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:31.278+0000 7f1d9d67b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:48:31.477 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:31 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:31.359+0000 7f1d9d67b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:48:31.477 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:31 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:31.417+0000 7f1d9d67b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:48:31.477 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:31 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:31.474+0000 7f1d9d67b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:48:32.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:32 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:32.254+0000 7f1d9d67b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:48:32.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:32 smithi070 ceph-mon[29589]: Active manager daemon y restarted 2024-09-21T09:48:32.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:32 smithi070 ceph-mon[29589]: Activating manager daemon y 2024-09-21T09:48:32.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:32 smithi070 ceph-mon[29589]: osdmap e2: 0 total, 0 up, 0 in 2024-09-21T09:48:32.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:32 smithi070 ceph-mon[29589]: mgrmap e6: y(active, starting, since 0.0031599s) 2024-09-21T09:48:32.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:32 smithi070 ceph-mon[29589]: from='mgr.14118 172.21.15.70:0/2596913197' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:48:32.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:32 smithi070 ceph-mon[29589]: from='mgr.14118 172.21.15.70:0/2596913197' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:48:32.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:32 smithi070 ceph-mon[29589]: from='mgr.14118 172.21.15.70:0/2596913197' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:48:32.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:32 smithi070 ceph-mon[29589]: from='mgr.14118 172.21.15.70:0/2596913197' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:48:32.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:32 smithi070 ceph-mon[29589]: from='mgr.14118 172.21.15.70:0/2596913197' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:48:32.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:32 smithi070 ceph-mon[29589]: Manager daemon y is now available 2024-09-21T09:48:32.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:32 smithi070 ceph-mon[29589]: from='mgr.14118 172.21.15.70:0/2596913197' entity='mgr.y' 2024-09-21T09:48:32.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:32 smithi070 ceph-mon[29589]: from='mgr.14118 172.21.15.70:0/2596913197' entity='mgr.y' 2024-09-21T09:48:32.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:32 smithi070 ceph-mon[29589]: from='mgr.14118 172.21.15.70:0/2596913197' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:48:32.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:32 smithi070 ceph-mon[29589]: from='mgr.14118 172.21.15.70:0/2596913197' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:48:32.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:32 smithi070 ceph-mon[29589]: from='mgr.14118 172.21.15.70:0/2596913197' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:48:32.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:32 smithi070 ceph-mon[29589]: from='mgr.14118 172.21.15.70:0/2596913197' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:48:33.550 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:33 smithi070 ceph-mon[29589]: Found migration_current of "None". Setting to last migration. 2024-09-21T09:48:33.551 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:33 smithi070 ceph-mon[29589]: mgrmap e7: y(active, since 1.00585s) 2024-09-21T09:48:33.847 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout { 2024-09-21T09:48:33.847 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-21T09:48:33.847 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T09:48:33.847 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout } 2024-09-21T09:48:33.847 INFO:teuthology.orchestra.run.smithi070.stdout:mgr epoch 5 is available 2024-09-21T09:48:33.847 INFO:teuthology.orchestra.run.smithi070.stdout:Setting orchestrator backend to cephadm... 2024-09-21T09:48:34.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:34 smithi070 ceph-mon[29589]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-21T09:48:34.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:34 smithi070 ceph-mon[29589]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-21T09:48:34.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:34 smithi070 ceph-mon[29589]: from='mgr.14118 172.21.15.70:0/2596913197' entity='mgr.y' 2024-09-21T09:48:34.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:34 smithi070 ceph-mon[29589]: from='mgr.14118 172.21.15.70:0/2596913197' entity='mgr.y' 2024-09-21T09:48:35.130 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:35.035+0000 7f1d0ab66640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:09:48:35] ENGINE Error in HTTPServer.serve 2024-09-21T09:48:35.131 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: Traceback (most recent call last): 2024-09-21T09:48:35.131 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:48:35.131 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: self._connections.run(self.expiration_interval) 2024-09-21T09:48:35.131 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:48:35.131 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: self._run(expiration_interval) 2024-09-21T09:48:35.131 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:48:35.131 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:48:35.131 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:48:35.131 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:48:35.131 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:48:35.131 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: s = self.context.wrap_socket( 2024-09-21T09:48:35.131 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:48:35.131 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: return self.sslsocket_class._create( 2024-09-21T09:48:35.132 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:48:35.132 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: self.do_handshake() 2024-09-21T09:48:35.132 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:48:35.132 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: self._sslobj.do_handshake() 2024-09-21T09:48:35.132 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:48:35.132 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:35 smithi070 ceph-mon[29589]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:48:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:35 smithi070 ceph-mon[29589]: from='mgr.14118 172.21.15.70:0/2596913197' entity='mgr.y' 2024-09-21T09:48:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:35 smithi070 ceph-mon[29589]: from='mgr.14118 172.21.15.70:0/2596913197' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:48:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:35 smithi070 ceph-mon[29589]: mgrmap e8: y(active, since 2s) 2024-09-21T09:48:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:35 smithi070 ceph-mon[29589]: from='mgr.14118 172.21.15.70:0/2596913197' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:48:36.184 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-21T09:48:36.184 INFO:teuthology.orchestra.run.smithi070.stdout:Generating ssh key... 2024-09-21T09:48:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:36 smithi070 ceph-mon[29589]: [21/Sep/2024:09:48:34] ENGINE Bus STARTING 2024-09-21T09:48:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:36 smithi070 ceph-mon[29589]: [21/Sep/2024:09:48:35] ENGINE Serving on https://172.21.15.70:7150 2024-09-21T09:48:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:36 smithi070 ceph-mon[29589]: [21/Sep/2024:09:48:35] ENGINE Error in HTTPServer.serve 2024-09-21T09:48:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-09-21T09:48:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:48:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:48:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:48:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout: self._run(expiration_interval) 2024-09-21T09:48:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:48:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:48:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:48:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:48:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:48:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout: s = self.context.wrap_socket( 2024-09-21T09:48:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:48:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout: return self.sslsocket_class._create( 2024-09-21T09:48:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:48:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout: self.do_handshake() 2024-09-21T09:48:36.629 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:48:36.629 INFO:journalctl@ceph.mon.a.smithi070.stdout: self._sslobj.do_handshake() 2024-09-21T09:48:36.629 INFO:journalctl@ceph.mon.a.smithi070.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:48:36.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:36 smithi070 ceph-mon[29589]: [21/Sep/2024:09:48:35] ENGINE Serving on http://172.21.15.70:8765 2024-09-21T09:48:36.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:36 smithi070 ceph-mon[29589]: [21/Sep/2024:09:48:35] ENGINE Bus STARTED 2024-09-21T09:48:37.128 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:36 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: Generating public/private rsa key pair. 2024-09-21T09:48:37.128 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:36 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: Your identification has been saved in /tmp/tmp0l8_yz7j/key 2024-09-21T09:48:37.128 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:36 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: Your public key has been saved in /tmp/tmp0l8_yz7j/key.pub 2024-09-21T09:48:37.128 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:36 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: The key fingerprint is: 2024-09-21T09:48:37.128 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:36 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: SHA256:8aaa+IMX8XViUNuMKjyFZooJgezHQcqV2q063fbAlDE ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T09:48:37.128 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:36 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: The key's randomart image is: 2024-09-21T09:48:37.128 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:36 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: +---[RSA 3072]----+ 2024-09-21T09:48:37.128 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:36 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: |+ .o. .. | 2024-09-21T09:48:37.128 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:36 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: |oooo .. = | 2024-09-21T09:48:37.128 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:36 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: |oo+ oE+ o.o o | 2024-09-21T09:48:37.129 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:36 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: | + * *=. ++ . | 2024-09-21T09:48:37.129 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:36 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: | + oo+oSooo | 2024-09-21T09:48:37.129 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:36 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: | .o .o.o | 2024-09-21T09:48:37.129 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:36 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: | o .+ .. | 2024-09-21T09:48:37.129 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:36 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: | o ..++o | 2024-09-21T09:48:37.129 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:36 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: | . oo=o | 2024-09-21T09:48:37.129 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:36 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: +----[SHA256]-----+ 2024-09-21T09:48:37.773 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:37 smithi070 ceph-mon[29589]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:48:37.773 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:37 smithi070 ceph-mon[29589]: from='mgr.14118 172.21.15.70:0/2596913197' entity='mgr.y' 2024-09-21T09:48:37.773 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:37 smithi070 ceph-mon[29589]: from='mgr.14118 172.21.15.70:0/2596913197' entity='mgr.y' 2024-09-21T09:48:38.572 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlA/CMsHabYhr0V1gTjog9soShfhci5dpqldBBVaxWOBs7gzKh1o/E60H9/pwQUrmTCebXh/kIGomGHcbjN9dOlwvnO52pNhB0/E38uh2f+3eszG2Iem9DpMuswdeVfFYpnKhdV/nEaJOURkIrZmyZjqtoPRWfTRugSlBFiOisvSou7Oz15E3fVZEVyt/E1thVllFnP20CuiTu29cF5T156H4I6oo/3bi9DmsC4tufw403Ps0BUJb0mRP9O0SF5DJyLKQpU0aT8nsDq+N2Q/tumc1V+S5OQNvLKCTrZW7Umdh9qGd3FxfRNlexjMEm1s99cDuMbZTBVrM2hqSGGPwmsZI1YBiS+wXmZhd0O4gSyyW/HFsRFxq9mdwH7dxPrvJDLxNZyD3Vt0cfgnShWDrAWOh838WLA2s84grlQM804sXEFaCvlLedPCce49aEsxq1ewNwDv0CErXIGYL3peXNgY2v/PfGe4aBmAJqYEn9d0S0ykZ9S9SpPJ6FvB2BYsE= ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T09:48:38.572 INFO:teuthology.orchestra.run.smithi070.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-21T09:48:38.572 INFO:teuthology.orchestra.run.smithi070.stdout:Adding key to root@localhost authorized_keys... 2024-09-21T09:48:38.582 INFO:teuthology.orchestra.run.smithi070.stdout:Adding host smithi070... 2024-09-21T09:48:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:38 smithi070 ceph-mon[29589]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:48:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:38 smithi070 ceph-mon[29589]: Generating ssh key... 2024-09-21T09:48:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:39 smithi070 ceph-mon[29589]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:48:40.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:40 smithi070 ceph-mon[29589]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi070", "addr": "172.21.15.70", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:48:41.598 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:41 smithi070 ceph-mon[29589]: Deploying cephadm binary to smithi070 2024-09-21T09:48:42.445 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout Added host 'smithi070' with addr '172.21.15.70' 2024-09-21T09:48:42.445 INFO:teuthology.orchestra.run.smithi070.stdout:Deploying unmanaged mon service... 2024-09-21T09:48:43.090 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:42 smithi070 ceph-mon[29589]: from='mgr.14118 172.21.15.70:0/2596913197' entity='mgr.y' 2024-09-21T09:48:43.090 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:42 smithi070 ceph-mon[29589]: Added host smithi070 2024-09-21T09:48:43.090 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:42 smithi070 ceph-mon[29589]: from='mgr.14118 172.21.15.70:0/2596913197' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:48:44.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:44 smithi070 ceph-mon[29589]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:48:44.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:44 smithi070 ceph-mon[29589]: Saving service mon spec with placement count:5 2024-09-21T09:48:44.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:44 smithi070 ceph-mon[29589]: from='mgr.14118 172.21.15.70:0/2596913197' entity='mgr.y' 2024-09-21T09:48:44.499 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-21T09:48:44.499 INFO:teuthology.orchestra.run.smithi070.stdout:Deploying unmanaged mgr service... 2024-09-21T09:48:45.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:45 smithi070 ceph-mon[29589]: from='mgr.14118 172.21.15.70:0/2596913197' entity='mgr.y' 2024-09-21T09:48:45.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:45 smithi070 ceph-mon[29589]: from='mgr.14118 172.21.15.70:0/2596913197' entity='mgr.y' 2024-09-21T09:48:46.314 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:46 smithi070 ceph-mon[29589]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:48:46.314 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:46 smithi070 ceph-mon[29589]: Saving service mgr spec with placement count:2 2024-09-21T09:48:46.314 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:46 smithi070 ceph-mon[29589]: from='mgr.14118 172.21.15.70:0/2596913197' entity='mgr.y' 2024-09-21T09:48:46.315 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:46 smithi070 ceph-mon[29589]: from='mgr.14118 172.21.15.70:0/2596913197' entity='mgr.y' 2024-09-21T09:48:46.316 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-21T09:48:48.257 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:47 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/3099441375' entity='client.admin' 2024-09-21T09:48:48.257 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:47 smithi070 ceph-mon[29589]: from='mgr.14118 172.21.15.70:0/2596913197' entity='mgr.y' 2024-09-21T09:48:49.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:49 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/712929245' entity='client.admin' 2024-09-21T09:48:49.635 INFO:teuthology.orchestra.run.smithi070.stdout:Enabling the dashboard module... 2024-09-21T09:48:50.678 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:50 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/2967748307' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-21T09:48:50.678 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:50 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ignoring --setuser ceph since I am not root 2024-09-21T09:48:50.678 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:50 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ignoring --setgroup ceph since I am not root 2024-09-21T09:48:50.679 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:50 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:50.539+0000 7fbbd89e6040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T09:48:50.679 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:50 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:50.597+0000 7fbbd89e6040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T09:48:50.974 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:50 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:50.752+0000 7fbbd89e6040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T09:48:50.974 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:50 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:50.820+0000 7fbbd89e6040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:48:50.974 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:50 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:50.972+0000 7fbbd89e6040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:48:51.266 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:51.123+0000 7fbbd89e6040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:48:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:51 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/2967748307' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-21T09:48:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:51 smithi070 ceph-mon[29589]: mgrmap e9: y(active, since 18s) 2024-09-21T09:48:51.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:51.446+0000 7fbbd89e6040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:48:51.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:51.512+0000 7fbbd89e6040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:48:52.078 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:51.700+0000 7fbbd89e6040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:48:52.365 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout { 2024-09-21T09:48:52.365 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-21T09:48:52.365 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T09:48:52.365 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-21T09:48:52.365 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T09:48:52.365 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout } 2024-09-21T09:48:52.365 INFO:teuthology.orchestra.run.smithi070.stdout:Waiting for the mgr to restart... 2024-09-21T09:48:52.365 INFO:teuthology.orchestra.run.smithi070.stdout:Waiting for mgr epoch 9... 2024-09-21T09:48:52.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:52 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/2459763498' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-21T09:48:52.630 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:52.587+0000 7fbbd89e6040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:48:53.127 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:52.663+0000 7fbbd89e6040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:48:53.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:53.373+0000 7fbbd89e6040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:48:53.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:53.487+0000 7fbbd89e6040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:48:53.908 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:53.625+0000 7fbbd89e6040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:48:53.956 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:53.686+0000 7fbbd89e6040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:48:53.956 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:53.832+0000 7fbbd89e6040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:48:53.957 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:53.899+0000 7fbbd89e6040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:48:54.183 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:53.959+0000 7fbbd89e6040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:48:54.183 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:54 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:54.115+0000 7fbbd89e6040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:48:54.183 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:54 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:54.180+0000 7fbbd89e6040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:48:54.448 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:54 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:54.248+0000 7fbbd89e6040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:48:54.448 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:54 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:54.329+0000 7fbbd89e6040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:48:54.448 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:54 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:54.387+0000 7fbbd89e6040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:48:54.448 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:54 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:54.445+0000 7fbbd89e6040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:48:55.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:55 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:55.216+0000 7fbbd89e6040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:48:55.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:55 smithi070 ceph-mon[29589]: Active manager daemon y restarted 2024-09-21T09:48:55.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:55 smithi070 ceph-mon[29589]: Activating manager daemon y 2024-09-21T09:48:55.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:55 smithi070 ceph-mon[29589]: osdmap e3: 0 total, 0 up, 0 in 2024-09-21T09:48:55.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:55 smithi070 ceph-mon[29589]: mgrmap e10: y(active, starting, since 0.00303361s) 2024-09-21T09:48:55.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:55 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:48:55.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:55 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:48:55.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:55 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:48:55.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:55 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:48:55.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:55 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:48:55.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:55 smithi070 ceph-mon[29589]: Manager daemon y is now available 2024-09-21T09:48:55.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:55 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:48:55.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:55 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:48:55.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:55 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:48:56.821 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:56.543+0000 7fbb3ee70640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:09:48:56] ENGINE Error in HTTPServer.serve 2024-09-21T09:48:56.822 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: Traceback (most recent call last): 2024-09-21T09:48:56.822 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:48:56.822 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: self._connections.run(self.expiration_interval) 2024-09-21T09:48:56.822 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:48:56.822 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: self._run(expiration_interval) 2024-09-21T09:48:56.822 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:48:56.822 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:48:56.822 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:48:56.822 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:48:56.822 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:48:56.823 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: s = self.context.wrap_socket( 2024-09-21T09:48:56.823 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:48:56.823 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: return self.sslsocket_class._create( 2024-09-21T09:48:56.823 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:48:56.823 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: self.do_handshake() 2024-09-21T09:48:56.823 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:48:56.823 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: self._sslobj.do_handshake() 2024-09-21T09:48:56.823 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:48:56.823 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:48:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:48:57.090 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout { 2024-09-21T09:48:57.090 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-21T09:48:57.090 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T09:48:57.091 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout } 2024-09-21T09:48:57.091 INFO:teuthology.orchestra.run.smithi070.stdout:mgr epoch 9 is available 2024-09-21T09:48:57.091 INFO:teuthology.orchestra.run.smithi070.stdout:Generating a dashboard self-signed certificate... 2024-09-21T09:48:57.357 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:57 smithi070 ceph-mon[29589]: mgrmap e11: y(active, since 1.00529s) 2024-09-21T09:48:57.358 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:57 smithi070 ceph-mon[29589]: [21/Sep/2024:09:48:56] ENGINE Bus STARTING 2024-09-21T09:48:57.358 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:57 smithi070 ceph-mon[29589]: [21/Sep/2024:09:48:56] ENGINE Serving on http://172.21.15.70:8765 2024-09-21T09:48:57.358 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:57 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:48:58.336 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:58 smithi070 ceph-mon[29589]: [21/Sep/2024:09:48:56] ENGINE Serving on https://172.21.15.70:7150 2024-09-21T09:48:58.337 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:58 smithi070 ceph-mon[29589]: [21/Sep/2024:09:48:56] ENGINE Bus STARTED 2024-09-21T09:48:58.337 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:58 smithi070 ceph-mon[29589]: [21/Sep/2024:09:48:56] ENGINE Error in HTTPServer.serve 2024-09-21T09:48:58.337 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-09-21T09:48:58.337 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:48:58.337 INFO:journalctl@ceph.mon.a.smithi070.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:48:58.337 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:48:58.337 INFO:journalctl@ceph.mon.a.smithi070.stdout: self._run(expiration_interval) 2024-09-21T09:48:58.337 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:48:58.338 INFO:journalctl@ceph.mon.a.smithi070.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:48:58.338 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:48:58.338 INFO:journalctl@ceph.mon.a.smithi070.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:48:58.338 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:48:58.338 INFO:journalctl@ceph.mon.a.smithi070.stdout: s = self.context.wrap_socket( 2024-09-21T09:48:58.338 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:48:58.338 INFO:journalctl@ceph.mon.a.smithi070.stdout: return self.sslsocket_class._create( 2024-09-21T09:48:58.338 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:48:58.338 INFO:journalctl@ceph.mon.a.smithi070.stdout: self.do_handshake() 2024-09-21T09:48:58.338 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:48:58.339 INFO:journalctl@ceph.mon.a.smithi070.stdout: self._sslobj.do_handshake() 2024-09-21T09:48:58.339 INFO:journalctl@ceph.mon.a.smithi070.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:48:58.339 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:58 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:48:58.339 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:58 smithi070 ceph-mon[29589]: mgrmap e12: y(active, since 2s) 2024-09-21T09:48:58.339 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:58 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:48:58.339 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:58 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:48:58.340 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-21T09:48:58.340 INFO:teuthology.orchestra.run.smithi070.stdout:Creating initial admin user... 2024-09-21T09:48:59.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:59 smithi070 ceph-mon[29589]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:48:59.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:48:59 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:48:59.981 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$DCFOF/L6KHJfUze6qeP.beE3cXFt7oOigkWiqXL5/ljRmfwesacJG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726912139, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-21T09:48:59.981 INFO:teuthology.orchestra.run.smithi070.stdout:Fetching dashboard port number... 2024-09-21T09:49:00.331 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:00 smithi070 ceph-mon[29589]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:49:01.403 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:01 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/3185566840' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-21T09:49:01.406 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 8443 2024-09-21T09:49:01.415 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T09:49:01.415 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout disabled 2024-09-21T09:49:01.422 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T09:49:01.423 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout inactive 2024-09-21T09:49:01.423 INFO:teuthology.orchestra.run.smithi070.stdout:firewalld.service is not enabled 2024-09-21T09:49:01.423 INFO:teuthology.orchestra.run.smithi070.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-21T09:49:01.424 INFO:teuthology.orchestra.run.smithi070.stdout:Ceph Dashboard is now available at: 2024-09-21T09:49:01.424 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:49:01.424 INFO:teuthology.orchestra.run.smithi070.stdout: URL: https://smithi070.front.sepia.ceph.com:8443/ 2024-09-21T09:49:01.424 INFO:teuthology.orchestra.run.smithi070.stdout: User: admin 2024-09-21T09:49:01.424 INFO:teuthology.orchestra.run.smithi070.stdout: Password: 5vcw32segb 2024-09-21T09:49:01.424 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:49:01.425 INFO:teuthology.orchestra.run.smithi070.stdout:Saving cluster configuration to /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config directory 2024-09-21T09:49:01.425 INFO:teuthology.orchestra.run.smithi070.stdout:Enabling autotune for osd_memory_target 2024-09-21T09:49:02.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:02 smithi070 ceph-mon[29589]: mgrmap e13: y(active, since 6s) 2024-09-21T09:49:03.851 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-21T09:49:03.852 INFO:teuthology.orchestra.run.smithi070.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-21T09:49:03.852 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:49:03.852 INFO:teuthology.orchestra.run.smithi070.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-21T09:49:03.852 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:49:03.852 INFO:teuthology.orchestra.run.smithi070.stdout:Or, if you are only running a single cluster on this host: 2024-09-21T09:49:03.852 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:49:03.852 INFO:teuthology.orchestra.run.smithi070.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-21T09:49:03.852 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:49:03.852 INFO:teuthology.orchestra.run.smithi070.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-21T09:49:03.852 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:49:03.853 INFO:teuthology.orchestra.run.smithi070.stdout: ceph telemetry on 2024-09-21T09:49:03.853 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:49:03.853 INFO:teuthology.orchestra.run.smithi070.stdout:For more information see: 2024-09-21T09:49:03.853 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:49:03.853 INFO:teuthology.orchestra.run.smithi070.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-21T09:49:03.853 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:49:03.853 INFO:teuthology.orchestra.run.smithi070.stdout:Bootstrap complete. 2024-09-21T09:49:03.884 INFO:tasks.cephadm:Fetching config... 2024-09-21T09:49:03.884 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T09:49:03.884 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-21T09:49:03.900 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-21T09:49:03.900 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T09:49:03.900 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-21T09:49:03.955 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-21T09:49:03.956 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T09:49:03.956 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/keyring of=/dev/stdout 2024-09-21T09:49:04.021 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-21T09:49:04.021 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T09:49:04.021 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-21T09:49:04.076 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-21T09:49:04.076 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlA/CMsHabYhr0V1gTjog9soShfhci5dpqldBBVaxWOBs7gzKh1o/E60H9/pwQUrmTCebXh/kIGomGHcbjN9dOlwvnO52pNhB0/E38uh2f+3eszG2Iem9DpMuswdeVfFYpnKhdV/nEaJOURkIrZmyZjqtoPRWfTRugSlBFiOisvSou7Oz15E3fVZEVyt/E1thVllFnP20CuiTu29cF5T156H4I6oo/3bi9DmsC4tufw403Ps0BUJb0mRP9O0SF5DJyLKQpU0aT8nsDq+N2Q/tumc1V+S5OQNvLKCTrZW7Umdh9qGd3FxfRNlexjMEm1s99cDuMbZTBVrM2hqSGGPwmsZI1YBiS+wXmZhd0O4gSyyW/HFsRFxq9mdwH7dxPrvJDLxNZyD3Vt0cfgnShWDrAWOh838WLA2s84grlQM804sXEFaCvlLedPCce49aEsxq1ewNwDv0CErXIGYL3peXNgY2v/PfGe4aBmAJqYEn9d0S0ykZ9S9SpPJ6FvB2BYsE= ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T09:49:04.149 INFO:teuthology.orchestra.run.smithi070.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlA/CMsHabYhr0V1gTjog9soShfhci5dpqldBBVaxWOBs7gzKh1o/E60H9/pwQUrmTCebXh/kIGomGHcbjN9dOlwvnO52pNhB0/E38uh2f+3eszG2Iem9DpMuswdeVfFYpnKhdV/nEaJOURkIrZmyZjqtoPRWfTRugSlBFiOisvSou7Oz15E3fVZEVyt/E1thVllFnP20CuiTu29cF5T156H4I6oo/3bi9DmsC4tufw403Ps0BUJb0mRP9O0SF5DJyLKQpU0aT8nsDq+N2Q/tumc1V+S5OQNvLKCTrZW7Umdh9qGd3FxfRNlexjMEm1s99cDuMbZTBVrM2hqSGGPwmsZI1YBiS+wXmZhd0O4gSyyW/HFsRFxq9mdwH7dxPrvJDLxNZyD3Vt0cfgnShWDrAWOh838WLA2s84grlQM804sXEFaCvlLedPCce49aEsxq1ewNwDv0CErXIGYL3peXNgY2v/PfGe4aBmAJqYEn9d0S0ykZ9S9SpPJ6FvB2BYsE= ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T09:49:04.162 DEBUG:teuthology.orchestra.run.smithi160:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlA/CMsHabYhr0V1gTjog9soShfhci5dpqldBBVaxWOBs7gzKh1o/E60H9/pwQUrmTCebXh/kIGomGHcbjN9dOlwvnO52pNhB0/E38uh2f+3eszG2Iem9DpMuswdeVfFYpnKhdV/nEaJOURkIrZmyZjqtoPRWfTRugSlBFiOisvSou7Oz15E3fVZEVyt/E1thVllFnP20CuiTu29cF5T156H4I6oo/3bi9DmsC4tufw403Ps0BUJb0mRP9O0SF5DJyLKQpU0aT8nsDq+N2Q/tumc1V+S5OQNvLKCTrZW7Umdh9qGd3FxfRNlexjMEm1s99cDuMbZTBVrM2hqSGGPwmsZI1YBiS+wXmZhd0O4gSyyW/HFsRFxq9mdwH7dxPrvJDLxNZyD3Vt0cfgnShWDrAWOh838WLA2s84grlQM804sXEFaCvlLedPCce49aEsxq1ewNwDv0CErXIGYL3peXNgY2v/PfGe4aBmAJqYEn9d0S0ykZ9S9SpPJ6FvB2BYsE= ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T09:49:04.221 INFO:teuthology.orchestra.run.smithi160.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlA/CMsHabYhr0V1gTjog9soShfhci5dpqldBBVaxWOBs7gzKh1o/E60H9/pwQUrmTCebXh/kIGomGHcbjN9dOlwvnO52pNhB0/E38uh2f+3eszG2Iem9DpMuswdeVfFYpnKhdV/nEaJOURkIrZmyZjqtoPRWfTRugSlBFiOisvSou7Oz15E3fVZEVyt/E1thVllFnP20CuiTu29cF5T156H4I6oo/3bi9DmsC4tufw403Ps0BUJb0mRP9O0SF5DJyLKQpU0aT8nsDq+N2Q/tumc1V+S5OQNvLKCTrZW7Umdh9qGd3FxfRNlexjMEm1s99cDuMbZTBVrM2hqSGGPwmsZI1YBiS+wXmZhd0O4gSyyW/HFsRFxq9mdwH7dxPrvJDLxNZyD3Vt0cfgnShWDrAWOh838WLA2s84grlQM804sXEFaCvlLedPCce49aEsxq1ewNwDv0CErXIGYL3peXNgY2v/PfGe4aBmAJqYEn9d0S0ykZ9S9SpPJ6FvB2BYsE= ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T09:49:04.233 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-21T09:49:04.427 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:04 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/830565696' entity='client.admin' 2024-09-21T09:49:04.503 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:49:07.327 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:07 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/798003966' entity='client.admin' 2024-09-21T09:49:07.328 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:07 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:07.328 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:07 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:07.328 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:07 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi070", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:49:07.328 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:07 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:07.328 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:07 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:07.328 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:07 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:07.623 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-21T09:49:07.623 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-21T09:49:07.893 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:49:11.105 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi160 2024-09-21T09:49:11.106 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T09:49:11.106 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/etc/ceph/ceph.conf 2024-09-21T09:49:11.125 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T09:49:11.125 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:49:11.184 INFO:tasks.cephadm:Adding host smithi160 to orchestrator... 2024-09-21T09:49:11.184 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph orch host add smithi160 2024-09-21T09:49:11.374 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:49:11.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:11 smithi070 ceph-mon[29589]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:49:11.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:11 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:11.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:11 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:11.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:11 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:11.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:11 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:11.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:11 smithi070 ceph-mon[29589]: Updating smithi070:/etc/ceph/ceph.conf 2024-09-21T09:49:11.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:11 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:11.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:11 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:11.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:11 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:12.462 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:12 smithi070 ceph-mon[29589]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:49:12.462 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:12 smithi070 ceph-mon[29589]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:49:12.462 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:12 smithi070 ceph-mon[29589]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:49:13.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:13 smithi070 ceph-mon[29589]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi160", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:49:14.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:14 smithi070 ceph-mon[29589]: Deploying cephadm binary to smithi160 2024-09-21T09:49:15.714 INFO:teuthology.orchestra.run.smithi070.stdout:Added host 'smithi160' with addr '172.21.15.160' 2024-09-21T09:49:16.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:16 smithi070 ceph-mon[29589]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:16.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:16 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:16.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:16 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:16.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:16 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:16.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:16 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:16.314 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph orch host ls --format=json 2024-09-21T09:49:16.503 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:49:17.607 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:17 smithi070 ceph-mon[29589]: Added host smithi160 2024-09-21T09:49:17.607 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:17 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:18.235 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:49:18.235 INFO:teuthology.orchestra.run.smithi070.stdout:[{"addr": "172.21.15.70", "hostname": "smithi070", "labels": [], "status": ""}, {"addr": "172.21.15.160", "hostname": "smithi160", "labels": [], "status": ""}] 2024-09-21T09:49:18.537 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:18 smithi070 ceph-mon[29589]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:18.878 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-21T09:49:18.878 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd crush tunables default 2024-09-21T09:49:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:49:19.466 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:19 smithi070 ceph-mon[29589]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:49:20.502 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:20 smithi070 ceph-mon[29589]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:20.502 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:20 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:20.502 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:20 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:20.502 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:20 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:20.502 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:20 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:20.502 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:20 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi160", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:49:20.502 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:20 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:20.502 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:20 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:21.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:21 smithi070 ceph-mon[29589]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-21T09:49:21.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:21 smithi070 ceph-mon[29589]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:49:21.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:21 smithi070 ceph-mon[29589]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:49:21.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:21 smithi070 ceph-mon[29589]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:49:21.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:21 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:21.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:21 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:21.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:21 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:21.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:21 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/650154314' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-21T09:49:21.632 INFO:teuthology.orchestra.run.smithi070.stderr:adjusted tunables profile to default 2024-09-21T09:49:22.206 INFO:tasks.cephadm:Adding mon.a on smithi070 2024-09-21T09:49:22.206 INFO:tasks.cephadm:Adding mon.c on smithi070 2024-09-21T09:49:22.206 INFO:tasks.cephadm:Adding mon.b on smithi160 2024-09-21T09:49:22.206 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph orch apply mon '3;smithi070:172.21.15.70=a;smithi070:[v2:172.21.15.70:3301,v1:172.21.15.70:6790]=c;smithi160:172.21.15.160=b' 2024-09-21T09:49:22.401 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:49:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:22 smithi070 ceph-mon[29589]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:22 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/650154314' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T09:49:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:22 smithi070 ceph-mon[29589]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:49:23.992 INFO:teuthology.orchestra.run.smithi160.stdout:Scheduled mon update... 2024-09-21T09:49:24.504 DEBUG:teuthology.orchestra.run.smithi070:mon.c> sudo journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mon.c.service 2024-09-21T09:49:24.507 DEBUG:teuthology.orchestra.run.smithi160:mon.b> sudo journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mon.b.service 2024-09-21T09:49:24.509 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T09:49:24.509 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T09:49:24.532 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:24 smithi070 ceph-mon[29589]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:24.532 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:24 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:24.532 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:24 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:24.532 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:24 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:24.532 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:24 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:24.532 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:24 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:24.532 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:24 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:49:24.533 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:24 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:25.533 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:49:25.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:25 smithi070 ceph-mon[29589]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi070:172.21.15.70=a;smithi070:[v2:172.21.15.70:3301,v1:172.21.15.70:6790]=c;smithi160:172.21.15.160=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:49:25.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:25 smithi070 ceph-mon[29589]: Saving service mon spec with placement smithi070:172.21.15.70=a;smithi070:[v2:172.21.15.70:3301,v1:172.21.15.70:6790]=c;smithi160:172.21.15.160=b;count:3 2024-09-21T09:49:25.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:25 smithi070 ceph-mon[29589]: Deploying daemon mon.b on smithi160 2024-09-21T09:49:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:26 smithi070 ceph-mon[29589]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:28.257 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:49:28.257 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"7c68bfbc-77fe-11ef-baf6-efdc52797490","modified":"2024-09-21T09:47:56.277550Z","created":"2024-09-21T09:47:56.277550Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:3300","nonce":0},{"type":"v1","addr":"172.21.15.70:6789","nonce":0}]},"addr":"172.21.15.70:6789/0","public_addr":"172.21.15.70:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T09:49:28.257 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-09-21T09:49:28.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:28 smithi070 ceph-mon[29589]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:29.063 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:28 smithi160 podman[33164]: 2024-09-21 09:49:28.853185356 +0000 UTC m=+0.016518626 image pull 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6 2024-09-21T09:49:29.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 podman[33164]: 2024-09-21 09:49:29.132908909 +0000 UTC m=+0.296242177 container init 0fd16436dfa06bed80a8c8835b332485d9430f5a851d23d51af960daa8758c5b (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=reef-6a1bdf8, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T09:49:29.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 podman[33164]: 2024-09-21 09:49:29.137306088 +0000 UTC m=+0.300639348 container start 0fd16436dfa06bed80a8c8835b332485d9430f5a851d23d51af960daa8758c5b (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, RELEASE=reef-6a1bdf8, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T09:49:29.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T09:49:29.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable), process ceph-mon, pid 7 2024-09-21T09:49:29.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: pidfile_write: ignore empty --pid-file 2024-09-21T09:49:29.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: load: jerasure load: lrc 2024-09-21T09:49:29.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: RocksDB version: 7.9.2 2024-09-21T09:49:29.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Git sha 0 2024-09-21T09:49:29.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Compile date 2024-09-20 20:34:00 2024-09-21T09:49:29.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: DB SUMMARY 2024-09-21T09:49:29.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: DB Session ID: J29AAV0NUDJP6ZF8W7ZU 2024-09-21T09:49:29.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: CURRENT file: CURRENT 2024-09-21T09:49:29.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T09:49:29.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-21T09:49:29.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-09-21T09:49:29.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-09-21T09:49:29.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.error_if_exists: 0 2024-09-21T09:49:29.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.create_if_missing: 0 2024-09-21T09:49:29.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.paranoid_checks: 1 2024-09-21T09:49:29.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T09:49:29.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T09:49:29.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T09:49:29.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.env: 0x562ff9fb7bc0 2024-09-21T09:49:29.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T09:49:29.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.info_log: 0x562ffb006be0 2024-09-21T09:49:29.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T09:49:29.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.statistics: (nil) 2024-09-21T09:49:29.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.use_fsync: 0 2024-09-21T09:49:29.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.max_log_file_size: 0 2024-09-21T09:49:29.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T09:49:29.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T09:49:29.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T09:49:29.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T09:49:29.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.allow_fallocate: 1 2024-09-21T09:49:29.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T09:49:29.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T09:49:29.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.use_direct_reads: 0 2024-09-21T09:49:29.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T09:49:29.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T09:49:29.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.db_log_dir: 2024-09-21T09:49:29.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.wal_dir: 2024-09-21T09:49:29.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T09:49:29.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T09:49:29.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T09:49:29.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T09:49:29.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T09:49:29.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T09:49:29.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T09:49:29.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T09:49:29.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.write_buffer_manager: 0x562ffb014320 2024-09-21T09:49:29.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T09:49:29.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T09:49:29.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T09:49:29.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T09:49:29.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T09:49:29.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T09:49:29.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T09:49:29.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T09:49:29.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.unordered_write: 0 2024-09-21T09:49:29.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T09:49:29.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T09:49:29.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T09:49:29.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T09:49:29.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.row_cache: None 2024-09-21T09:49:29.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.wal_filter: None 2024-09-21T09:49:29.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T09:49:29.598 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T09:49:29.598 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.two_write_queues: 0 2024-09-21T09:49:29.598 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T09:49:29.598 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.wal_compression: 0 2024-09-21T09:49:29.598 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.atomic_flush: 0 2024-09-21T09:49:29.598 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T09:49:29.598 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T09:49:29.599 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T09:49:29.599 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.log_readahead_size: 0 2024-09-21T09:49:29.599 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T09:49:29.599 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T09:49:29.599 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T09:49:29.599 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T09:49:29.599 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T09:49:29.599 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T09:49:29.599 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T09:49:29.599 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.max_background_jobs: 2 2024-09-21T09:49:29.599 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.max_background_compactions: -1 2024-09-21T09:49:29.599 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.max_subcompactions: 1 2024-09-21T09:49:29.600 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T09:49:29.600 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T09:49:29.600 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T09:49:29.600 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T09:49:29.600 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T09:49:29.600 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T09:49:29.600 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T09:49:29.600 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T09:49:29.600 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.max_open_files: -1 2024-09-21T09:49:29.600 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T09:49:29.600 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T09:49:29.600 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T09:49:29.600 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T09:49:29.600 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.max_background_flushes: -1 2024-09-21T09:49:29.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Compression algorithms supported: 2024-09-21T09:49:29.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: kZSTD supported: 0 2024-09-21T09:49:29.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: kXpressCompression supported: 0 2024-09-21T09:49:29.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: kBZip2Compression supported: 0 2024-09-21T09:49:29.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T09:49:29.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: kLZ4Compression supported: 1 2024-09-21T09:49:29.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: kZlibCompression supported: 1 2024-09-21T09:49:29.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T09:49:29.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: kSnappyCompression supported: 1 2024-09-21T09:49:29.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T09:49:29.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T09:49:29.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-09-21T09:49:29.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T09:49:29.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T09:49:29.602 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.merge_operator: 2024-09-21T09:49:29.602 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.compaction_filter: None 2024-09-21T09:49:29.602 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T09:49:29.602 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T09:49:29.602 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T09:49:29.602 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T09:49:29.602 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562ffb0067e0) 2024-09-21T09:49:29.602 INFO:journalctl@ceph.mon.b.smithi160.stdout: cache_index_and_filter_blocks: 1 2024-09-21T09:49:29.602 INFO:journalctl@ceph.mon.b.smithi160.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T09:49:29.602 INFO:journalctl@ceph.mon.b.smithi160.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T09:49:29.602 INFO:journalctl@ceph.mon.b.smithi160.stdout: pin_top_level_index_and_filter: 1 2024-09-21T09:49:29.602 INFO:journalctl@ceph.mon.b.smithi160.stdout: index_type: 0 2024-09-21T09:49:29.602 INFO:journalctl@ceph.mon.b.smithi160.stdout: data_block_index_type: 0 2024-09-21T09:49:29.603 INFO:journalctl@ceph.mon.b.smithi160.stdout: index_shortening: 1 2024-09-21T09:49:29.603 INFO:journalctl@ceph.mon.b.smithi160.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T09:49:29.603 INFO:journalctl@ceph.mon.b.smithi160.stdout: checksum: 4 2024-09-21T09:49:29.603 INFO:journalctl@ceph.mon.b.smithi160.stdout: no_block_cache: 0 2024-09-21T09:49:29.603 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_cache: 0x562ffaffb350 2024-09-21T09:49:29.603 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_cache_name: BinnedLRUCache 2024-09-21T09:49:29.603 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_cache_options: 2024-09-21T09:49:29.603 INFO:journalctl@ceph.mon.b.smithi160.stdout: capacity : 536870912 2024-09-21T09:49:29.603 INFO:journalctl@ceph.mon.b.smithi160.stdout: num_shard_bits : 4 2024-09-21T09:49:29.603 INFO:journalctl@ceph.mon.b.smithi160.stdout: strict_capacity_limit : 0 2024-09-21T09:49:29.603 INFO:journalctl@ceph.mon.b.smithi160.stdout: high_pri_pool_ratio: 0.000 2024-09-21T09:49:29.603 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_cache_compressed: (nil) 2024-09-21T09:49:29.603 INFO:journalctl@ceph.mon.b.smithi160.stdout: persistent_cache: (nil) 2024-09-21T09:49:29.603 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_size: 4096 2024-09-21T09:49:29.604 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_size_deviation: 10 2024-09-21T09:49:29.604 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_restart_interval: 16 2024-09-21T09:49:29.604 INFO:journalctl@ceph.mon.b.smithi160.stdout: index_block_restart_interval: 1 2024-09-21T09:49:29.604 INFO:journalctl@ceph.mon.b.smithi160.stdout: metadata_block_size: 4096 2024-09-21T09:49:29.604 INFO:journalctl@ceph.mon.b.smithi160.stdout: partition_filters: 0 2024-09-21T09:49:29.604 INFO:journalctl@ceph.mon.b.smithi160.stdout: use_delta_encoding: 1 2024-09-21T09:49:29.604 INFO:journalctl@ceph.mon.b.smithi160.stdout: filter_policy: bloomfilter 2024-09-21T09:49:29.604 INFO:journalctl@ceph.mon.b.smithi160.stdout: whole_key_filtering: 1 2024-09-21T09:49:29.604 INFO:journalctl@ceph.mon.b.smithi160.stdout: verify_compression: 0 2024-09-21T09:49:29.604 INFO:journalctl@ceph.mon.b.smithi160.stdout: read_amp_bytes_per_bit: 0 2024-09-21T09:49:29.604 INFO:journalctl@ceph.mon.b.smithi160.stdout: format_version: 5 2024-09-21T09:49:29.604 INFO:journalctl@ceph.mon.b.smithi160.stdout: enable_index_compression: 1 2024-09-21T09:49:29.604 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_align: 0 2024-09-21T09:49:29.604 INFO:journalctl@ceph.mon.b.smithi160.stdout: max_auto_readahead_size: 262144 2024-09-21T09:49:29.605 INFO:journalctl@ceph.mon.b.smithi160.stdout: prepopulate_block_cache: 0 2024-09-21T09:49:29.605 INFO:journalctl@ceph.mon.b.smithi160.stdout: initial_auto_readahead_size: 8192 2024-09-21T09:49:29.605 INFO:journalctl@ceph.mon.b.smithi160.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T09:49:29.605 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T09:49:29.605 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T09:49:29.605 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.compression: NoCompression 2024-09-21T09:49:29.605 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T09:49:29.605 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T09:49:29.605 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T09:49:29.605 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.num_levels: 7 2024-09-21T09:49:29.605 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T09:49:29.605 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T09:49:29.605 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T09:49:29.605 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T09:49:29.606 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T09:49:29.606 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T09:49:29.606 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T09:49:29.606 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:49:29.606 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T09:49:29.606 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T09:49:29.606 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T09:49:29.606 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T09:49:29.606 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T09:49:29.606 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T09:49:29.606 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T09:49:29.606 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T09:49:29.606 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:49:29.606 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T09:49:29.607 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T09:49:29.607 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T09:49:29.607 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T09:49:29.607 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T09:49:29.607 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T09:49:29.607 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T09:49:29.607 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T09:49:29.607 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T09:49:29.607 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T09:49:29.607 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T09:49:29.607 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T09:49:29.607 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T09:49:29.607 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T09:49:29.607 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T09:49:29.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T09:49:29.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T09:49:29.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T09:49:29.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T09:49:29.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T09:49:29.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T09:49:29.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T09:49:29.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T09:49:29.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T09:49:29.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T09:49:29.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T09:49:29.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T09:49:29.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T09:49:29.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T09:49:29.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T09:49:29.609 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T09:49:29.609 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T09:49:29.609 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T09:49:29.609 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T09:49:29.609 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T09:49:29.609 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T09:49:29.609 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T09:49:29.609 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.inplace_update_support: 0 2024-09-21T09:49:29.609 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T09:49:29.609 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T09:49:29.609 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T09:49:29.609 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T09:49:29.609 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.bloom_locality: 0 2024-09-21T09:49:29.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.max_successive_merges: 0 2024-09-21T09:49:29.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T09:49:29.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T09:49:29.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T09:49:29.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T09:49:29.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.ttl: 2592000 2024-09-21T09:49:29.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T09:49:29.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T09:49:29.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T09:49:29.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.enable_blob_files: false 2024-09-21T09:49:29.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.min_blob_size: 0 2024-09-21T09:49:29.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T09:49:29.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T09:49:29.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T09:49:29.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T09:49:29.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T09:49:29.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T09:49:29.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T09:49:29.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T09:49:29.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T09:49:29.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-21T09:49:29.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 46442bcf-7bad-4411-b6e2-50f5f9dc303c 2024-09-21T09:49:29.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912169164370, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-21T09:49:29.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-21T09:49:29.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912169164777, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726912169, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "46442bcf-7bad-4411-b6e2-50f5f9dc303c", "db_session_id": "J29AAV0NUDJP6ZF8W7ZU", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-21T09:49:29.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912169164875, "job": 1, "event": "recovery_finished"} 2024-09-21T09:49:29.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-21T09:49:29.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T09:49:29.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562ffb026e00 2024-09-21T09:49:29.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: DB pointer 0x562ffb11a000 2024-09-21T09:49:29.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T09:49:29.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T09:49:29.612 INFO:journalctl@ceph.mon.b.smithi160.stdout: ** DB Stats ** 2024-09-21T09:49:29.612 INFO:journalctl@ceph.mon.b.smithi160.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:49:29.612 INFO:journalctl@ceph.mon.b.smithi160.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T09:49:29.612 INFO:journalctl@ceph.mon.b.smithi160.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T09:49:29.612 INFO:journalctl@ceph.mon.b.smithi160.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:49:29.612 INFO:journalctl@ceph.mon.b.smithi160.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T09:49:29.612 INFO:journalctl@ceph.mon.b.smithi160.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T09:49:29.612 INFO:journalctl@ceph.mon.b.smithi160.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:49:29.612 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-21T09:49:29.612 INFO:journalctl@ceph.mon.b.smithi160.stdout: ** Compaction Stats [default] ** 2024-09-21T09:49:29.613 INFO:journalctl@ceph.mon.b.smithi160.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T09:49:29.613 INFO:journalctl@ceph.mon.b.smithi160.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T09:49:29.613 INFO:journalctl@ceph.mon.b.smithi160.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T09:49:29.613 INFO:journalctl@ceph.mon.b.smithi160.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T09:49:29.613 INFO:journalctl@ceph.mon.b.smithi160.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T09:49:29.613 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-21T09:49:29.613 INFO:journalctl@ceph.mon.b.smithi160.stdout: ** Compaction Stats [default] ** 2024-09-21T09:49:29.613 INFO:journalctl@ceph.mon.b.smithi160.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T09:49:29.613 INFO:journalctl@ceph.mon.b.smithi160.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:49:29.613 INFO:journalctl@ceph.mon.b.smithi160.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T09:49:29.613 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-21T09:49:29.613 INFO:journalctl@ceph.mon.b.smithi160.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T09:49:29.613 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-21T09:49:29.613 INFO:journalctl@ceph.mon.b.smithi160.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:49:29.614 INFO:journalctl@ceph.mon.b.smithi160.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T09:49:29.614 INFO:journalctl@ceph.mon.b.smithi160.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T09:49:29.614 INFO:journalctl@ceph.mon.b.smithi160.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T09:49:29.614 INFO:journalctl@ceph.mon.b.smithi160.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T09:49:29.614 INFO:journalctl@ceph.mon.b.smithi160.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T09:49:29.614 INFO:journalctl@ceph.mon.b.smithi160.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:49:29.614 INFO:journalctl@ceph.mon.b.smithi160.stdout: Interval compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:49:29.614 INFO:journalctl@ceph.mon.b.smithi160.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T09:49:29.614 INFO:journalctl@ceph.mon.b.smithi160.stdout: Block cache BinnedLRUCache@0x562ffaffb350#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-21T09:49:29.614 INFO:journalctl@ceph.mon.b.smithi160.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-21T09:49:29.614 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-21T09:49:29.614 INFO:journalctl@ceph.mon.b.smithi160.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T09:49:29.614 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-09-21T09:49:29.615 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: using public_addr v2:172.21.15.160:0/0 -> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] 2024-09-21T09:49:29.615 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: starting mon.b rank -1 at public addrs [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] at bind addrs [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T09:49:29.615 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: mon.b@-1(???) e0 preinit fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T09:49:29.615 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: mon.b@-1(synchronizing).mds e1 new map 2024-09-21T09:49:29.615 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: mon.b@-1(synchronizing).mds e1 print_map 2024-09-21T09:49:29.615 INFO:journalctl@ceph.mon.b.smithi160.stdout: e1 2024-09-21T09:49:29.615 INFO:journalctl@ceph.mon.b.smithi160.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T09:49:29.615 INFO:journalctl@ceph.mon.b.smithi160.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T09:49:29.615 INFO:journalctl@ceph.mon.b.smithi160.stdout: legacy client fscid: -1 2024-09-21T09:49:29.615 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-21T09:49:29.615 INFO:journalctl@ceph.mon.b.smithi160.stdout: No filesystems configured 2024-09-21T09:49:29.615 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-21T09:49:29.615 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-21T09:49:29.615 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-21T09:49:29.616 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-21T09:49:29.616 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-21T09:49:29.616 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-21T09:49:29.616 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T09:49:29.616 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:49:29.616 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:49:29.616 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:49:29.616 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:29.616 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:29.616 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:29.616 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:29.616 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:29.616 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi160", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:49:29.616 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:29.617 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:29.617 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-21T09:49:29.617 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:49:29.617 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:49:29.617 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:49:29.617 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:29.617 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:29.617 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:29.617 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/650154314' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-21T09:49:29.617 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:29.617 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/650154314' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T09:49:29.617 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:49:29.617 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:29.618 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:29.618 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:29.618 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:29.618 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:29.618 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:29.618 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:49:29.618 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:29.618 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi070:172.21.15.70=a;smithi070:[v2:172.21.15.70:3301,v1:172.21.15.70:6790]=c;smithi160:172.21.15.160=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:49:29.618 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: Saving service mon spec with placement smithi070:172.21.15.70=a;smithi070:[v2:172.21.15.70:3301,v1:172.21.15.70:6790]=c;smithi160:172.21.15.160=b;count:3 2024-09-21T09:49:29.618 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: Deploying daemon mon.b on smithi160 2024-09-21T09:49:29.618 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:29.618 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:29.618 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 ceph-mon[33203]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-21T09:49:29.618 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 bash[33164]: 0fd16436dfa06bed80a8c8835b332485d9430f5a851d23d51af960daa8758c5b 2024-09-21T09:49:29.619 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:29 smithi160 systemd[1]: Started Ceph mon.b for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:49:29.830 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T09:49:29.830 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T09:49:30.035 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:49:34.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:34 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:49:34.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:34 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:49:34.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:34 smithi160 ceph-mon[33203]: mon.a calling monitor election 2024-09-21T09:49:34.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:34 smithi160 ceph-mon[33203]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:34.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:34 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:49:34.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:34 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:49:34.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:34 smithi160 ceph-mon[33203]: mon.b calling monitor election 2024-09-21T09:49:34.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:34 smithi160 ceph-mon[33203]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:34.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:34 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:49:34.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:34 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:49:34.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:34 smithi160 ceph-mon[33203]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:34.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:34 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:49:34.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:34 smithi160 ceph-mon[33203]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T09:49:34.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:34 smithi160 ceph-mon[33203]: monmap e2: 2 mons at {a=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0],b=[v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:49:34.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:34 smithi160 ceph-mon[33203]: fsmap 2024-09-21T09:49:34.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:34 smithi160 ceph-mon[33203]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:49:34.563 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:34 smithi160 ceph-mon[33203]: mgrmap e13: y(active, since 38s) 2024-09-21T09:49:34.563 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:34 smithi160 ceph-mon[33203]: overall HEALTH_OK 2024-09-21T09:49:34.563 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:34 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:34.563 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:34 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:34.563 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:34 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:34.563 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:34 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:49:34.563 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:34 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:34.577 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:34 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:49:34.577 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:34 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:49:34.577 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:34 smithi070 ceph-mon[29589]: mon.a calling monitor election 2024-09-21T09:49:34.577 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:34 smithi070 ceph-mon[29589]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:34.577 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:34 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:49:34.577 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:34 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:49:34.577 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:34 smithi070 ceph-mon[29589]: mon.b calling monitor election 2024-09-21T09:49:34.577 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:34 smithi070 ceph-mon[29589]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:34.577 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:34 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:49:34.578 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:34 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:49:34.578 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:34 smithi070 ceph-mon[29589]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:34.578 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:34 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:49:34.578 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:34 smithi070 ceph-mon[29589]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T09:49:34.578 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:34 smithi070 ceph-mon[29589]: monmap e2: 2 mons at {a=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0],b=[v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:49:34.578 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:34 smithi070 ceph-mon[29589]: fsmap 2024-09-21T09:49:34.578 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:34 smithi070 ceph-mon[29589]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:49:34.578 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:34 smithi070 ceph-mon[29589]: mgrmap e13: y(active, since 38s) 2024-09-21T09:49:34.578 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:34 smithi070 ceph-mon[29589]: overall HEALTH_OK 2024-09-21T09:49:34.578 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:34 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:34.578 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:34 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:34.578 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:34 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:34.578 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:34 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:49:34.578 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:34 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:34.749 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:49:34.749 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":2,"fsid":"7c68bfbc-77fe-11ef-baf6-efdc52797490","modified":"2024-09-21T09:49:29.188313Z","created":"2024-09-21T09:47:56.277550Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:3300","nonce":0},{"type":"v1","addr":"172.21.15.70:6789","nonce":0}]},"addr":"172.21.15.70:6789/0","public_addr":"172.21.15.70:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:3300","nonce":0},{"type":"v1","addr":"172.21.15.160:6789","nonce":0}]},"addr":"172.21.15.160:6789/0","public_addr":"172.21.15.160:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-21T09:49:34.749 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 2 2024-09-21T09:49:35.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:35 smithi160 ceph-mon[33203]: Deploying daemon mon.c on smithi070 2024-09-21T09:49:35.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:35 smithi160 ceph-mon[33203]: from='client.? 172.21.15.160:0/430499377' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:49:35.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:35 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:49:35.531 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:35 smithi070 ceph-mon[29589]: Deploying daemon mon.c on smithi070 2024-09-21T09:49:35.531 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:35 smithi070 ceph-mon[29589]: from='client.? 172.21.15.160:0/430499377' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:49:35.531 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:35 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:49:36.256 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T09:49:36.257 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T09:49:36.466 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:49:36.512 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:36 smithi070 ceph-mon[29589]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:36.578 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:36 smithi160 ceph-mon[33203]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:37.268 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 podman[41495]: 2024-09-21 09:49:37.15451055 +0000 UTC m=+0.134849285 container create fc0590f324b52865408b15b681e6c566f6ebc4944b433f0c49250a4582f9ff3f (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=reef-6a1bdf8, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-21T09:49:37.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 podman[41495]: 2024-09-21 09:49:37.373674862 +0000 UTC m=+0.354013615 container init fc0590f324b52865408b15b681e6c566f6ebc4944b433f0c49250a4582f9ff3f (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-6a1bdf8, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-21T09:49:37.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 podman[41495]: 2024-09-21 09:49:37.3784363 +0000 UTC m=+0.358775038 container start fc0590f324b52865408b15b681e6c566f6ebc4944b433f0c49250a4582f9ff3f (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-c, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=reef-6a1bdf8, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T09:49:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T09:49:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable), process ceph-mon, pid 7 2024-09-21T09:49:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: pidfile_write: ignore empty --pid-file 2024-09-21T09:49:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: load: jerasure load: lrc 2024-09-21T09:49:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: RocksDB version: 7.9.2 2024-09-21T09:49:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Git sha 0 2024-09-21T09:49:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Compile date 2024-09-20 20:34:00 2024-09-21T09:49:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: DB SUMMARY 2024-09-21T09:49:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: DB Session ID: ITWB7LW6U3F43BWTWDUG 2024-09-21T09:49:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: CURRENT file: CURRENT 2024-09-21T09:49:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T09:49:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-21T09:49:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-21T09:49:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-09-21T09:49:37.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.error_if_exists: 0 2024-09-21T09:49:37.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.create_if_missing: 0 2024-09-21T09:49:37.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.paranoid_checks: 1 2024-09-21T09:49:37.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T09:49:37.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T09:49:37.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T09:49:37.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.env: 0x5563f8608bc0 2024-09-21T09:49:37.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T09:49:37.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.info_log: 0x5563f8e74be0 2024-09-21T09:49:37.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T09:49:37.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.statistics: (nil) 2024-09-21T09:49:37.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.use_fsync: 0 2024-09-21T09:49:37.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.max_log_file_size: 0 2024-09-21T09:49:37.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T09:49:37.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T09:49:37.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T09:49:37.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T09:49:37.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.allow_fallocate: 1 2024-09-21T09:49:37.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T09:49:37.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T09:49:37.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.use_direct_reads: 0 2024-09-21T09:49:37.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T09:49:37.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T09:49:37.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.db_log_dir: 2024-09-21T09:49:37.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.wal_dir: 2024-09-21T09:49:37.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T09:49:37.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T09:49:37.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T09:49:37.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T09:49:37.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T09:49:37.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T09:49:37.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T09:49:37.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T09:49:37.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.write_buffer_manager: 0x5563f8e82320 2024-09-21T09:49:37.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T09:49:37.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T09:49:37.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T09:49:37.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T09:49:37.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T09:49:37.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T09:49:37.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T09:49:37.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T09:49:37.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.unordered_write: 0 2024-09-21T09:49:37.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T09:49:37.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T09:49:37.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T09:49:37.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T09:49:37.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.row_cache: None 2024-09-21T09:49:37.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.wal_filter: None 2024-09-21T09:49:37.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T09:49:37.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T09:49:37.634 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.two_write_queues: 0 2024-09-21T09:49:37.634 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T09:49:37.634 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.wal_compression: 0 2024-09-21T09:49:37.634 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.atomic_flush: 0 2024-09-21T09:49:37.634 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T09:49:37.634 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T09:49:37.634 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T09:49:37.634 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.log_readahead_size: 0 2024-09-21T09:49:37.634 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T09:49:37.634 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T09:49:37.634 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T09:49:37.634 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T09:49:37.634 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T09:49:37.634 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T09:49:37.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T09:49:37.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.max_background_jobs: 2 2024-09-21T09:49:37.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.max_background_compactions: -1 2024-09-21T09:49:37.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.max_subcompactions: 1 2024-09-21T09:49:37.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T09:49:37.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T09:49:37.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T09:49:37.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T09:49:37.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T09:49:37.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T09:49:37.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T09:49:37.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T09:49:37.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.max_open_files: -1 2024-09-21T09:49:37.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T09:49:37.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T09:49:37.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T09:49:37.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T09:49:37.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.max_background_flushes: -1 2024-09-21T09:49:37.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Compression algorithms supported: 2024-09-21T09:49:37.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: kZSTD supported: 0 2024-09-21T09:49:37.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: kXpressCompression supported: 0 2024-09-21T09:49:37.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: kBZip2Compression supported: 0 2024-09-21T09:49:37.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T09:49:37.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: kLZ4Compression supported: 1 2024-09-21T09:49:37.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: kZlibCompression supported: 1 2024-09-21T09:49:37.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T09:49:37.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: kSnappyCompression supported: 1 2024-09-21T09:49:37.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T09:49:37.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T09:49:37.637 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-21T09:49:37.637 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T09:49:37.637 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T09:49:37.637 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.merge_operator: 2024-09-21T09:49:37.637 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.compaction_filter: None 2024-09-21T09:49:37.637 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T09:49:37.637 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T09:49:37.637 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T09:49:37.637 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T09:49:37.637 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5563f8e747e0) 2024-09-21T09:49:37.637 INFO:journalctl@ceph.mon.c.smithi070.stdout: cache_index_and_filter_blocks: 1 2024-09-21T09:49:37.637 INFO:journalctl@ceph.mon.c.smithi070.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T09:49:37.637 INFO:journalctl@ceph.mon.c.smithi070.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T09:49:37.637 INFO:journalctl@ceph.mon.c.smithi070.stdout: pin_top_level_index_and_filter: 1 2024-09-21T09:49:37.638 INFO:journalctl@ceph.mon.c.smithi070.stdout: index_type: 0 2024-09-21T09:49:37.638 INFO:journalctl@ceph.mon.c.smithi070.stdout: data_block_index_type: 0 2024-09-21T09:49:37.638 INFO:journalctl@ceph.mon.c.smithi070.stdout: index_shortening: 1 2024-09-21T09:49:37.638 INFO:journalctl@ceph.mon.c.smithi070.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T09:49:37.638 INFO:journalctl@ceph.mon.c.smithi070.stdout: checksum: 4 2024-09-21T09:49:37.638 INFO:journalctl@ceph.mon.c.smithi070.stdout: no_block_cache: 0 2024-09-21T09:49:37.638 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_cache: 0x5563f8e691f0 2024-09-21T09:49:37.638 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_cache_name: BinnedLRUCache 2024-09-21T09:49:37.638 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_cache_options: 2024-09-21T09:49:37.638 INFO:journalctl@ceph.mon.c.smithi070.stdout: capacity : 536870912 2024-09-21T09:49:37.638 INFO:journalctl@ceph.mon.c.smithi070.stdout: num_shard_bits : 4 2024-09-21T09:49:37.638 INFO:journalctl@ceph.mon.c.smithi070.stdout: strict_capacity_limit : 0 2024-09-21T09:49:37.638 INFO:journalctl@ceph.mon.c.smithi070.stdout: high_pri_pool_ratio: 0.000 2024-09-21T09:49:37.638 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_cache_compressed: (nil) 2024-09-21T09:49:37.639 INFO:journalctl@ceph.mon.c.smithi070.stdout: persistent_cache: (nil) 2024-09-21T09:49:37.639 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_size: 4096 2024-09-21T09:49:37.639 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_size_deviation: 10 2024-09-21T09:49:37.639 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_restart_interval: 16 2024-09-21T09:49:37.639 INFO:journalctl@ceph.mon.c.smithi070.stdout: index_block_restart_interval: 1 2024-09-21T09:49:37.639 INFO:journalctl@ceph.mon.c.smithi070.stdout: metadata_block_size: 4096 2024-09-21T09:49:37.639 INFO:journalctl@ceph.mon.c.smithi070.stdout: partition_filters: 0 2024-09-21T09:49:37.639 INFO:journalctl@ceph.mon.c.smithi070.stdout: use_delta_encoding: 1 2024-09-21T09:49:37.639 INFO:journalctl@ceph.mon.c.smithi070.stdout: filter_policy: bloomfilter 2024-09-21T09:49:37.639 INFO:journalctl@ceph.mon.c.smithi070.stdout: whole_key_filtering: 1 2024-09-21T09:49:37.639 INFO:journalctl@ceph.mon.c.smithi070.stdout: verify_compression: 0 2024-09-21T09:49:37.639 INFO:journalctl@ceph.mon.c.smithi070.stdout: read_amp_bytes_per_bit: 0 2024-09-21T09:49:37.639 INFO:journalctl@ceph.mon.c.smithi070.stdout: format_version: 5 2024-09-21T09:49:37.639 INFO:journalctl@ceph.mon.c.smithi070.stdout: enable_index_compression: 1 2024-09-21T09:49:37.639 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_align: 0 2024-09-21T09:49:37.640 INFO:journalctl@ceph.mon.c.smithi070.stdout: max_auto_readahead_size: 262144 2024-09-21T09:49:37.640 INFO:journalctl@ceph.mon.c.smithi070.stdout: prepopulate_block_cache: 0 2024-09-21T09:49:37.640 INFO:journalctl@ceph.mon.c.smithi070.stdout: initial_auto_readahead_size: 8192 2024-09-21T09:49:37.640 INFO:journalctl@ceph.mon.c.smithi070.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T09:49:37.640 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T09:49:37.640 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T09:49:37.640 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.compression: NoCompression 2024-09-21T09:49:37.640 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T09:49:37.640 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T09:49:37.640 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T09:49:37.640 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.num_levels: 7 2024-09-21T09:49:37.640 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T09:49:37.640 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T09:49:37.640 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T09:49:37.640 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T09:49:37.641 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T09:49:37.641 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T09:49:37.641 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T09:49:37.641 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:49:37.641 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T09:49:37.641 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T09:49:37.641 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T09:49:37.641 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T09:49:37.641 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T09:49:37.641 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T09:49:37.641 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T09:49:37.641 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T09:49:37.641 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:49:37.641 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T09:49:37.642 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T09:49:37.642 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T09:49:37.642 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T09:49:37.642 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T09:49:37.642 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T09:49:37.642 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T09:49:37.642 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T09:49:37.642 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T09:49:37.642 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T09:49:37.642 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T09:49:37.642 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T09:49:37.642 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T09:49:37.642 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T09:49:37.642 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T09:49:37.643 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T09:49:37.643 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T09:49:37.643 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T09:49:37.643 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T09:49:37.643 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T09:49:37.643 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T09:49:37.643 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T09:49:37.643 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T09:49:37.643 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T09:49:37.643 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T09:49:37.643 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T09:49:37.643 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T09:49:37.643 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T09:49:37.643 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T09:49:37.643 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T09:49:37.644 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T09:49:37.644 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T09:49:37.644 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T09:49:37.644 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T09:49:37.644 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T09:49:37.644 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T09:49:37.644 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T09:49:37.644 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.inplace_update_support: 0 2024-09-21T09:49:37.644 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T09:49:37.644 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T09:49:37.644 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T09:49:37.644 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T09:49:37.644 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.bloom_locality: 0 2024-09-21T09:49:37.644 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.max_successive_merges: 0 2024-09-21T09:49:37.645 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T09:49:37.645 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T09:49:37.645 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T09:49:37.645 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T09:49:37.645 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.ttl: 2592000 2024-09-21T09:49:37.645 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T09:49:37.645 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T09:49:37.645 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T09:49:37.645 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.enable_blob_files: false 2024-09-21T09:49:37.645 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.min_blob_size: 0 2024-09-21T09:49:37.645 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T09:49:37.645 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T09:49:37.645 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T09:49:37.645 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T09:49:37.646 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T09:49:37.646 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T09:49:37.646 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T09:49:37.646 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T09:49:37.646 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T09:49:37.646 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-21T09:49:37.646 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a9cc8616-e551-4cf4-ade2-e75ee5bba32b 2024-09-21T09:49:37.646 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912177405144, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-21T09:49:37.646 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-21T09:49:37.646 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912177405523, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726912177, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a9cc8616-e551-4cf4-ade2-e75ee5bba32b", "db_session_id": "ITWB7LW6U3F43BWTWDUG", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-21T09:49:37.646 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912177405613, "job": 1, "event": "recovery_finished"} 2024-09-21T09:49:37.646 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-21T09:49:37.646 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T09:49:37.646 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5563f8e94e00 2024-09-21T09:49:37.647 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: DB pointer 0x5563f8f88000 2024-09-21T09:49:37.647 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T09:49:37.647 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T09:49:37.647 INFO:journalctl@ceph.mon.c.smithi070.stdout: ** DB Stats ** 2024-09-21T09:49:37.647 INFO:journalctl@ceph.mon.c.smithi070.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:49:37.647 INFO:journalctl@ceph.mon.c.smithi070.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T09:49:37.647 INFO:journalctl@ceph.mon.c.smithi070.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T09:49:37.647 INFO:journalctl@ceph.mon.c.smithi070.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:49:37.647 INFO:journalctl@ceph.mon.c.smithi070.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T09:49:37.647 INFO:journalctl@ceph.mon.c.smithi070.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T09:49:37.647 INFO:journalctl@ceph.mon.c.smithi070.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:49:37.647 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-09-21T09:49:37.647 INFO:journalctl@ceph.mon.c.smithi070.stdout: ** Compaction Stats [default] ** 2024-09-21T09:49:37.647 INFO:journalctl@ceph.mon.c.smithi070.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T09:49:37.648 INFO:journalctl@ceph.mon.c.smithi070.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T09:49:37.648 INFO:journalctl@ceph.mon.c.smithi070.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T09:49:37.648 INFO:journalctl@ceph.mon.c.smithi070.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T09:49:37.648 INFO:journalctl@ceph.mon.c.smithi070.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T09:49:37.648 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-09-21T09:49:37.648 INFO:journalctl@ceph.mon.c.smithi070.stdout: ** Compaction Stats [default] ** 2024-09-21T09:49:37.648 INFO:journalctl@ceph.mon.c.smithi070.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T09:49:37.648 INFO:journalctl@ceph.mon.c.smithi070.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:49:37.648 INFO:journalctl@ceph.mon.c.smithi070.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T09:49:37.648 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-09-21T09:49:37.648 INFO:journalctl@ceph.mon.c.smithi070.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T09:49:37.648 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-09-21T09:49:37.648 INFO:journalctl@ceph.mon.c.smithi070.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:49:37.648 INFO:journalctl@ceph.mon.c.smithi070.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T09:49:37.649 INFO:journalctl@ceph.mon.c.smithi070.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T09:49:37.649 INFO:journalctl@ceph.mon.c.smithi070.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T09:49:37.649 INFO:journalctl@ceph.mon.c.smithi070.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T09:49:37.649 INFO:journalctl@ceph.mon.c.smithi070.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T09:49:37.649 INFO:journalctl@ceph.mon.c.smithi070.stdout: Cumulative compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:49:37.649 INFO:journalctl@ceph.mon.c.smithi070.stdout: Interval compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:49:37.649 INFO:journalctl@ceph.mon.c.smithi070.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T09:49:37.649 INFO:journalctl@ceph.mon.c.smithi070.stdout: Block cache BinnedLRUCache@0x5563f8e691f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-09-21T09:49:37.649 INFO:journalctl@ceph.mon.c.smithi070.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-21T09:49:37.649 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-09-21T09:49:37.649 INFO:journalctl@ceph.mon.c.smithi070.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T09:49:37.649 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-21T09:49:37.649 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: using public_addrv [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-21T09:49:37.649 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: starting mon.c rank -1 at public addrs [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] at bind addrs [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T09:49:37.649 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: mon.c@-1(???) e0 preinit fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T09:49:37.650 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: mon.c@-1(synchronizing).mds e1 new map 2024-09-21T09:49:37.650 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-21T09:49:37.650 INFO:journalctl@ceph.mon.c.smithi070.stdout: e1 2024-09-21T09:49:37.650 INFO:journalctl@ceph.mon.c.smithi070.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T09:49:37.650 INFO:journalctl@ceph.mon.c.smithi070.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T09:49:37.650 INFO:journalctl@ceph.mon.c.smithi070.stdout: legacy client fscid: -1 2024-09-21T09:49:37.650 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-09-21T09:49:37.650 INFO:journalctl@ceph.mon.c.smithi070.stdout: No filesystems configured 2024-09-21T09:49:37.650 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-21T09:49:37.650 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-21T09:49:37.650 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-21T09:49:37.650 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-21T09:49:37.650 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-21T09:49:37.650 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-21T09:49:37.651 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T09:49:37.651 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:49:37.651 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:49:37.651 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:49:37.651 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:37.651 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:37.651 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:37.651 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:37.651 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:37.651 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi160", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:49:37.651 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:37.651 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:37.651 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-21T09:49:37.651 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:49:37.652 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:49:37.652 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:49:37.652 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:37.652 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:37.652 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:37.652 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/650154314' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-21T09:49:37.652 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:37.652 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/650154314' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T09:49:37.652 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:49:37.652 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:37.652 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:37.652 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:37.652 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:37.652 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:37.653 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:37.653 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:49:37.653 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:37.653 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi070:172.21.15.70=a;smithi070:[v2:172.21.15.70:3301,v1:172.21.15.70:6790]=c;smithi160:172.21.15.160=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:49:37.653 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: Saving service mon spec with placement smithi070:172.21.15.70=a;smithi070:[v2:172.21.15.70:3301,v1:172.21.15.70:6790]=c;smithi160:172.21.15.160=b;count:3 2024-09-21T09:49:37.653 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: Deploying daemon mon.b on smithi160 2024-09-21T09:49:37.653 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:37.653 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:37.653 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:49:37.653 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:49:37.653 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: mon.a calling monitor election 2024-09-21T09:49:37.653 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:37.653 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:49:37.654 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:49:37.654 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: mon.b calling monitor election 2024-09-21T09:49:37.654 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:37.654 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:49:37.654 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:49:37.654 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:37.654 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:49:37.654 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T09:49:37.654 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: monmap e2: 2 mons at {a=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0],b=[v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:49:37.654 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: fsmap 2024-09-21T09:49:37.654 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:49:37.654 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: mgrmap e13: y(active, since 38s) 2024-09-21T09:49:37.654 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: overall HEALTH_OK 2024-09-21T09:49:37.654 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:37.655 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:37.655 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:37.655 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:49:37.655 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:37.655 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: Deploying daemon mon.c on smithi070 2024-09-21T09:49:37.655 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='client.? 172.21.15.160:0/430499377' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:49:37.655 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:49:37.655 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:37.655 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 ceph-mon[41533]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-21T09:49:37.655 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 bash[41495]: fc0590f324b52865408b15b681e6c566f6ebc4944b433f0c49250a4582f9ff3f 2024-09-21T09:49:37.655 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:37 smithi070 systemd[1]: Started Ceph mon.c for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:49:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:42 smithi160 ceph-mon[33203]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:49:42.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:49:42.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:42 smithi160 ceph-mon[33203]: mon.a calling monitor election 2024-09-21T09:49:42.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:49:42.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:42 smithi160 ceph-mon[33203]: mon.b calling monitor election 2024-09-21T09:49:42.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:49:42.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:42 smithi160 ceph-mon[33203]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:42.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:49:42.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:42 smithi160 ceph-mon[33203]: mon.c calling monitor election 2024-09-21T09:49:42.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:49:42.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:42 smithi160 ceph-mon[33203]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:42.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:49:42.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:49:42.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:42 smithi160 ceph-mon[33203]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:49:42.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:42 smithi160 ceph-mon[33203]: monmap e3: 3 mons at {a=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0],b=[v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0],c=[v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:49:42.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:42 smithi160 ceph-mon[33203]: fsmap 2024-09-21T09:49:42.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:42 smithi160 ceph-mon[33203]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:49:42.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:42 smithi160 ceph-mon[33203]: mgrmap e13: y(active, since 47s) 2024-09-21T09:49:42.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:42 smithi160 ceph-mon[33203]: overall HEALTH_OK 2024-09-21T09:49:42.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:42.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:42.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:42.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:42.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[41533]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:42.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:49:42.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:49:42.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[41533]: mon.a calling monitor election 2024-09-21T09:49:42.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:49:42.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[41533]: mon.b calling monitor election 2024-09-21T09:49:42.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:49:42.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[41533]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:42.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:49:42.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[41533]: mon.c calling monitor election 2024-09-21T09:49:42.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:49:42.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[41533]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:42.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:49:42.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:49:42.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[41533]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:49:42.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[41533]: monmap e3: 3 mons at {a=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0],b=[v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0],c=[v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:49:42.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[41533]: fsmap 2024-09-21T09:49:42.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[41533]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:49:42.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[41533]: mgrmap e13: y(active, since 47s) 2024-09-21T09:49:42.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[41533]: overall HEALTH_OK 2024-09-21T09:49:42.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:42.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:42.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:42.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:42.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[29589]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:42.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:49:42.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:49:42.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[29589]: mon.a calling monitor election 2024-09-21T09:49:42.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:49:42.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[29589]: mon.b calling monitor election 2024-09-21T09:49:42.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:49:42.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[29589]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:42.881 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:49:42.881 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[29589]: mon.c calling monitor election 2024-09-21T09:49:42.881 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:49:42.881 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[29589]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:42.881 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:49:42.881 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:49:42.881 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[29589]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:49:42.881 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[29589]: monmap e3: 3 mons at {a=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0],b=[v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0],c=[v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:49:42.881 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[29589]: fsmap 2024-09-21T09:49:42.881 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[29589]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:49:42.881 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[29589]: mgrmap e13: y(active, since 47s) 2024-09-21T09:49:42.881 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[29589]: overall HEALTH_OK 2024-09-21T09:49:42.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:42.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:42.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:42.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:43 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:43 smithi160 ceph-mon[33203]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:43 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:49:43.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:43 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:43.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:43 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:43.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:43 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:43.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:43 smithi070 ceph-mon[29589]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:43.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:43 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:49:43.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:43 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:43.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:43 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:43.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:43 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:43.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:43 smithi070 ceph-mon[41533]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:43.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:43 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:49:43.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:43 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:43.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:43 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.227 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:49:44.227 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":3,"fsid":"7c68bfbc-77fe-11ef-baf6-efdc52797490","modified":"2024-09-21T09:49:37.426189Z","created":"2024-09-21T09:47:56.277550Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:3300","nonce":0},{"type":"v1","addr":"172.21.15.70:6789","nonce":0}]},"addr":"172.21.15.70:6789/0","public_addr":"172.21.15.70:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:3300","nonce":0},{"type":"v1","addr":"172.21.15.160:6789","nonce":0}]},"addr":"172.21.15.160:6789/0","public_addr":"172.21.15.160:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:3301","nonce":0},{"type":"v1","addr":"172.21.15.70:6790","nonce":0}]},"addr":"172.21.15.70:6790/0","public_addr":"172.21.15.70:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-21T09:49:44.227 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 3 2024-09-21T09:49:44.782 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.782 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.782 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:44.782 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:44.782 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[41533]: Updating smithi070:/etc/ceph/ceph.conf 2024-09-21T09:49:44.782 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[41533]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-21T09:49:44.783 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[41533]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:49:44.783 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[41533]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:49:44.783 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.783 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.783 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.783 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.783 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.783 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.783 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.783 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.784 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.784 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[41533]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T09:49:44.784 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:49:44.784 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:49:44.784 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:44.784 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[41533]: Reconfiguring daemon mon.a on smithi070 2024-09-21T09:49:44.784 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[41533]: from='client.? 172.21.15.160:0/1164947992' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:49:44.784 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.784 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.784 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:44.785 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:44.785 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[29589]: Updating smithi070:/etc/ceph/ceph.conf 2024-09-21T09:49:44.785 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[29589]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-21T09:49:44.785 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[29589]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:49:44.785 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[29589]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:49:44.785 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.785 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.785 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.785 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.785 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.785 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.785 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.786 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.786 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.786 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[29589]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T09:49:44.786 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:49:44.786 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:49:44.786 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:44.786 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[29589]: Reconfiguring daemon mon.a on smithi070 2024-09-21T09:49:44.786 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:44 smithi070 ceph-mon[29589]: from='client.? 172.21.15.160:0/1164947992' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:49:44.818 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:44 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.818 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:44 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.818 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:44 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:44.818 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:44 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:44.818 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:44 smithi160 ceph-mon[33203]: Updating smithi070:/etc/ceph/ceph.conf 2024-09-21T09:49:44.818 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:44 smithi160 ceph-mon[33203]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-21T09:49:44.818 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:44 smithi160 ceph-mon[33203]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:49:44.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:44 smithi160 ceph-mon[33203]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:49:44.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:44 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:44 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:44 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:44 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:44 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:44 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:44 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:44 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:44 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:44.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:44 smithi160 ceph-mon[33203]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T09:49:44.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:44 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:49:44.820 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:44 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:49:44.820 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:44 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:44.820 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:44 smithi160 ceph-mon[33203]: Reconfiguring daemon mon.a on smithi070 2024-09-21T09:49:44.820 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:44 smithi160 ceph-mon[33203]: from='client.? 172.21.15.160:0/1164947992' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:49:44.842 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-21T09:49:44.842 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph config generate-minimal-conf 2024-09-21T09:49:45.110 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:49:46.557 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:46 smithi070 ceph-mon[41533]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:46.557 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:46 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:46.557 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:46 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:46.557 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:46 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:46.557 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:46 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:49:46.558 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:46 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:49:46.558 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:46 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:46.558 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:46 smithi070 ceph-mon[29589]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:46.558 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:46 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:46.558 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:46 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:46.558 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:46 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:46.558 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:46 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:49:46.558 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:46 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:49:46.558 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:46 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:46 smithi160 ceph-mon[33203]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:46 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:46 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:46 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:46 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:49:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:46 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:49:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:46 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:47.511 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:47 smithi070 ceph-mon[29589]: Reconfiguring mon.c (monmap changed)... 2024-09-21T09:49:47.511 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:47 smithi070 ceph-mon[29589]: Reconfiguring daemon mon.c on smithi070 2024-09-21T09:49:47.511 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:47 smithi070 ceph-mon[41533]: Reconfiguring mon.c (monmap changed)... 2024-09-21T09:49:47.511 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:47 smithi070 ceph-mon[41533]: Reconfiguring daemon mon.c on smithi070 2024-09-21T09:49:47.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:47 smithi160 ceph-mon[33203]: Reconfiguring mon.c (monmap changed)... 2024-09-21T09:49:47.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:47 smithi160 ceph-mon[33203]: Reconfiguring daemon mon.c on smithi070 2024-09-21T09:49:48.341 INFO:teuthology.orchestra.run.smithi070.stdout:# minimal ceph.conf for 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T09:49:48.341 INFO:teuthology.orchestra.run.smithi070.stdout:[global] 2024-09-21T09:49:48.341 INFO:teuthology.orchestra.run.smithi070.stdout: fsid = 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T09:49:48.341 INFO:teuthology.orchestra.run.smithi070.stdout: mon_host = [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-21T09:49:48.570 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:48 smithi160 ceph-mon[33203]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:48.570 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:48 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:48.570 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:48 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:48.570 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:48 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:49:48.570 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:48 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:49:48.570 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:48 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:48.602 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:48 smithi070 ceph-mon[29589]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:48.602 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:48 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:48.602 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:48 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:48.602 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:48 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:49:48.602 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:48 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:49:48.602 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:48 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:48.602 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:48 smithi070 ceph-mon[41533]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:48.603 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:48 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:48.603 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:48 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:48.603 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:48 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:49:48.603 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:48 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:49:48.603 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:48 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:48.882 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-21T09:49:48.882 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T09:49:48.882 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T09:49:48.909 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T09:49:48.909 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:49:48.975 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T09:49:48.975 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T09:49:49.003 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T09:49:49.004 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:49:49.070 INFO:tasks.cephadm:Adding mgr.y on smithi070 2024-09-21T09:49:49.071 INFO:tasks.cephadm:Adding mgr.x on smithi160 2024-09-21T09:49:49.071 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph orch apply mgr '2;smithi070=y;smithi160=x' 2024-09-21T09:49:49.304 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:49:49.534 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:49 smithi160 ceph-mon[33203]: Reconfiguring mon.b (monmap changed)... 2024-09-21T09:49:49.534 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:49 smithi160 ceph-mon[33203]: Reconfiguring daemon mon.b on smithi160 2024-09-21T09:49:49.534 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:49 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/1582808993' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:49.534 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:49 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:49.534 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:49 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:49.534 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:49 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:49.534 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:49 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:49.534 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:49 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:49.534 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:49 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:49.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:49 smithi070 ceph-mon[29589]: Reconfiguring mon.b (monmap changed)... 2024-09-21T09:49:49.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:49 smithi070 ceph-mon[29589]: Reconfiguring daemon mon.b on smithi160 2024-09-21T09:49:49.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:49 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/1582808993' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:49.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:49 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:49.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:49 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:49.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:49 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:49.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:49 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:49.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:49 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:49.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:49 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:49.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:49 smithi070 ceph-mon[41533]: Reconfiguring mon.b (monmap changed)... 2024-09-21T09:49:49.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:49 smithi070 ceph-mon[41533]: Reconfiguring daemon mon.b on smithi160 2024-09-21T09:49:49.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:49 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/1582808993' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:49.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:49 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:49.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:49 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:49.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:49 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:49.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:49 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:49.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:49 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:49.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:49 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:50.581 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:50 smithi160 ceph-mon[33203]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:50 smithi070 ceph-mon[29589]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:50.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:50 smithi070 ceph-mon[41533]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:50.850 INFO:teuthology.orchestra.run.smithi160.stdout:Scheduled mgr update... 2024-09-21T09:49:51.444 DEBUG:teuthology.orchestra.run.smithi160:mgr.x> sudo journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mgr.x.service 2024-09-21T09:49:51.446 DEBUG:tasks.cephadm:set 0 configs 2024-09-21T09:49:51.446 INFO:tasks.cephadm:Deploying OSDs... 2024-09-21T09:49:51.446 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T09:49:51.446 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T09:49:51.463 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T09:49:51.463 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_1 2024-09-21T09:49:51.518 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T09:49:51.519 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:49:51.519 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 975 Links: 1 2024-09-21T09:49:51.519 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:49:51.519 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:49:51.519 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-09-21 09:49:36.862643366 +0000 2024-09-21T09:49:51.519 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-09-21 09:45:10.229557171 +0000 2024-09-21T09:49:51.519 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-09-21 09:45:10.229557171 +0000 2024-09-21T09:49:51.519 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-09-21 09:45:10.229557171 +0000 2024-09-21T09:49:51.519 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T09:49:51.584 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-09-21T09:49:51.584 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-09-21T09:49:51.584 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000115754 s, 4.4 MB/s 2024-09-21T09:49:51.585 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T09:49:51.641 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_2 2024-09-21T09:49:51.696 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T09:49:51.696 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:49:51.697 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 982 Links: 1 2024-09-21T09:49:51.697 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:49:51.697 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:49:51.697 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-09-21 09:49:36.862643366 +0000 2024-09-21T09:49:51.697 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-09-21 09:45:10.229557171 +0000 2024-09-21T09:49:51.697 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-09-21 09:45:10.229557171 +0000 2024-09-21T09:49:51.697 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-09-21 09:45:10.229557171 +0000 2024-09-21T09:49:51.698 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T09:49:51.759 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-09-21T09:49:51.760 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-09-21T09:49:51.760 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000105464 s, 4.9 MB/s 2024-09-21T09:49:51.761 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T09:49:51.818 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_3 2024-09-21T09:49:51.875 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T09:49:51.875 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:49:51.875 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 960 Links: 1 2024-09-21T09:49:51.875 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:49:51.875 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:49:51.875 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-09-21 09:49:36.863643336 +0000 2024-09-21T09:49:51.875 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-09-21 09:45:10.229557171 +0000 2024-09-21T09:49:51.876 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-09-21 09:45:10.229557171 +0000 2024-09-21T09:49:51.876 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-09-21 09:45:10.229557171 +0000 2024-09-21T09:49:51.876 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T09:49:51.939 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-09-21T09:49:51.939 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-09-21T09:49:51.939 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000111714 s, 4.6 MB/s 2024-09-21T09:49:51.940 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T09:49:51.997 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_4 2024-09-21T09:49:52.052 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T09:49:52.052 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:49:52.052 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 971 Links: 1 2024-09-21T09:49:52.053 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:49:52.053 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:49:52.053 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-09-21 09:49:36.863643336 +0000 2024-09-21T09:49:52.053 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-09-21 09:45:10.229557171 +0000 2024-09-21T09:49:52.053 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-09-21 09:45:10.229557171 +0000 2024-09-21T09:49:52.053 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-09-21 09:45:10.229557171 +0000 2024-09-21T09:49:52.053 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T09:49:52.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:51 smithi160 ceph-mon[33203]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi070=y;smithi160=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:49:52.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:51 smithi160 ceph-mon[33203]: Saving service mgr spec with placement smithi070=y;smithi160=x;count:2 2024-09-21T09:49:52.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:51 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:52.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:51 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:52.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:51 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:52.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:51 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:52.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:51 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:52.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:51 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:49:52.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:51 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T09:49:52.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:51 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:49:52.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:51 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:52.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:51 smithi160 ceph-mon[33203]: Deploying daemon mgr.x on smithi160 2024-09-21T09:49:52.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:51 smithi160 ceph-mon[33203]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:52.114 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:51 smithi070 ceph-mon[29589]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi070=y;smithi160=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:49:52.114 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:51 smithi070 ceph-mon[29589]: Saving service mgr spec with placement smithi070=y;smithi160=x;count:2 2024-09-21T09:49:52.115 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:51 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:52.115 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:51 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:52.115 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:51 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:52.115 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:51 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:52.115 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:51 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:52.115 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:51 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:49:52.115 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:51 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T09:49:52.115 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:51 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:49:52.116 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:51 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:52.116 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:51 smithi070 ceph-mon[29589]: Deploying daemon mgr.x on smithi160 2024-09-21T09:49:52.116 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:51 smithi070 ceph-mon[29589]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:52.116 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:51 smithi070 ceph-mon[41533]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi070=y;smithi160=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:49:52.116 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:51 smithi070 ceph-mon[41533]: Saving service mgr spec with placement smithi070=y;smithi160=x;count:2 2024-09-21T09:49:52.116 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:51 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:52.116 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:51 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:52.116 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:51 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:52.116 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:51 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:52.117 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:51 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:52.117 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:51 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:49:52.117 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:51 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T09:49:52.117 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:51 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:49:52.117 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:51 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:52.117 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:51 smithi070 ceph-mon[41533]: Deploying daemon mgr.x on smithi160 2024-09-21T09:49:52.117 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:51 smithi070 ceph-mon[41533]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:52.118 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-09-21T09:49:52.118 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-09-21T09:49:52.118 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000103413 s, 5.0 MB/s 2024-09-21T09:49:52.118 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T09:49:52.175 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T09:49:52.175 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T09:49:52.191 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T09:49:52.192 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_1 2024-09-21T09:49:52.247 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T09:49:52.247 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:49:52.247 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 969 Links: 1 2024-09-21T09:49:52.248 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:49:52.248 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:49:52.248 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-09-21 09:49:28.129513166 +0000 2024-09-21T09:49:52.248 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-09-21 09:45:10.389902546 +0000 2024-09-21T09:49:52.248 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-09-21 09:45:10.389902546 +0000 2024-09-21T09:49:52.248 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-09-21 09:45:10.389902546 +0000 2024-09-21T09:49:52.248 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T09:49:52.311 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-09-21T09:49:52.312 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-09-21T09:49:52.312 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000204966 s, 2.5 MB/s 2024-09-21T09:49:52.313 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T09:49:52.369 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_2 2024-09-21T09:49:52.424 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T09:49:52.424 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:49:52.424 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 972 Links: 1 2024-09-21T09:49:52.424 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:49:52.424 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:49:52.424 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-09-21 09:49:28.129513166 +0000 2024-09-21T09:49:52.424 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-09-21 09:45:10.389902546 +0000 2024-09-21T09:49:52.424 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-09-21 09:45:10.389902546 +0000 2024-09-21T09:49:52.424 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-09-21 09:45:10.389902546 +0000 2024-09-21T09:49:52.425 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T09:49:52.489 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-09-21T09:49:52.489 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-09-21T09:49:52.489 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000210188 s, 2.4 MB/s 2024-09-21T09:49:52.490 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T09:49:52.547 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_3 2024-09-21T09:49:52.609 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T09:49:52.609 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:49:52.610 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 982 Links: 1 2024-09-21T09:49:52.610 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:49:52.610 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:49:52.610 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-09-21 09:49:28.129513166 +0000 2024-09-21T09:49:52.610 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-09-21 09:45:10.390902635 +0000 2024-09-21T09:49:52.610 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-09-21 09:45:10.390902635 +0000 2024-09-21T09:49:52.610 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-09-21 09:45:10.390902635 +0000 2024-09-21T09:49:52.610 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T09:49:52.714 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-09-21T09:49:52.714 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-09-21T09:49:52.714 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.00026249 s, 2.0 MB/s 2024-09-21T09:49:52.715 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T09:49:52.731 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_4 2024-09-21T09:49:52.792 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T09:49:52.792 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:49:52.792 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 974 Links: 1 2024-09-21T09:49:52.792 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:49:52.792 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:49:52.792 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-09-21 09:49:52.664184595 +0000 2024-09-21T09:49:52.792 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-09-21 09:45:10.389902546 +0000 2024-09-21T09:49:52.792 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-09-21 09:45:10.389902546 +0000 2024-09-21T09:49:52.792 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-09-21 09:45:10.389902546 +0000 2024-09-21T09:49:52.793 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T09:49:52.893 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-09-21T09:49:52.893 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-09-21T09:49:52.893 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000188893 s, 2.7 MB/s 2024-09-21T09:49:52.894 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T09:49:52.910 INFO:tasks.cephadm:Deploying osd.0 on smithi070 with /dev/vg_nvme/lv_4... 2024-09-21T09:49:52.910 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T09:49:53.092 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:52 smithi160 systemd[1]: Starting Ceph mgr.x for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:49:53.106 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:49:53.450 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:53 smithi160 podman[35273]: 2024-09-21 09:49:53.026233267 +0000 UTC m=+0.016778382 image pull 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6 2024-09-21T09:49:53.450 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:53 smithi160 podman[35273]: 2024-09-21 09:49:53.123886017 +0000 UTC m=+0.114431132 container create 8f2bf49a5ce6eab267c2e874beaee8622430c1bd6e1126b0cbf829c403b2a3cf (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=reef-6a1bdf8, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-21T09:49:53.450 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:53 smithi160 podman[35273]: 2024-09-21 09:49:53.318003649 +0000 UTC m=+0.308548763 container init 8f2bf49a5ce6eab267c2e874beaee8622430c1bd6e1126b0cbf829c403b2a3cf (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=reef-6a1bdf8, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-21T09:49:53.450 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:53 smithi160 podman[35273]: 2024-09-21 09:49:53.322623069 +0000 UTC m=+0.313168190 container start 8f2bf49a5ce6eab267c2e874beaee8622430c1bd6e1126b0cbf829c403b2a3cf (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, RELEASE=reef-6a1bdf8, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-21T09:49:53.450 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:53 smithi160 ceph-mgr[35311]: -- 172.21.15.160:0/318794908 <== mon.1 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ef26d86d00 con 0x55ef26d5b400 2024-09-21T09:49:53.450 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:53 smithi160 bash[35273]: 8f2bf49a5ce6eab267c2e874beaee8622430c1bd6e1126b0cbf829c403b2a3cf 2024-09-21T09:49:53.450 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:53 smithi160 systemd[1]: Started Ceph mgr.x for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:49:53.450 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:53 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:49:53.447+0000 7fe5988e4040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T09:49:53.740 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:53 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:49:53.506+0000 7fe5988e4040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T09:49:53.741 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:53 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:49:53.661+0000 7fe5988e4040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T09:49:53.741 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:53 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:49:53.738+0000 7fe5988e4040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:49:54.039 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:53 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:49:53.897+0000 7fe5988e4040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:49:54.039 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:49:54.037+0000 7fe5988e4040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:49:54.591 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:49:54.359+0000 7fe5988e4040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:49:54.591 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:49:54.420+0000 7fe5988e4040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:49:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:54 smithi160 ceph-mon[33203]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:54 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:54.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:54 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:54.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:54 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:54.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:54 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:54.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:54 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:54.681 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:54 smithi070 ceph-mon[29589]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:54.681 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:54 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:54.681 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:54 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:54.681 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:54 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:54.681 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:54 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:54.681 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:54 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:54.682 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:54 smithi070 ceph-mon[41533]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:54.682 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:54 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:54.682 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:54 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:54.682 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:54 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:54.682 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:54 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:54.682 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:54 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:55.091 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:49:54.616+0000 7fe5988e4040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:49:55.553 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:49:55.572 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi070:vg_nvme/lv_4 2024-09-21T09:49:55.828 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:55 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:55.828 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:55 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:55.828 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:55 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:55.828 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:55 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:55.828 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:55 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:55.828 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:55 smithi070 ceph-mon[29589]: Reconfiguring mgr.y (unknown last config time)... 2024-09-21T09:49:55.828 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:55 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:49:55.828 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:55 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:49:55.828 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:55 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:55.828 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:55 smithi070 ceph-mon[29589]: Reconfiguring daemon mgr.y on smithi070 2024-09-21T09:49:55.828 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:55 smithi070 ceph-mon[29589]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:55.829 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:55 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:55.829 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:55 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:55.829 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:55 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:55.829 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:55 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:55.829 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:55 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:55.829 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:55 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:55.829 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:55 smithi070 ceph-mon[41533]: Reconfiguring mgr.y (unknown last config time)... 2024-09-21T09:49:55.829 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:55 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:49:55.829 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:55 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:49:55.829 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:55 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:55.829 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:55 smithi070 ceph-mon[41533]: Reconfiguring daemon mgr.y on smithi070 2024-09-21T09:49:55.830 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:55 smithi070 ceph-mon[41533]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:55.830 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:55 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:55.841 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:49:55.478+0000 7fe5988e4040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:49:55.841 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:49:55.544+0000 7fe5988e4040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:49:55.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:55 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:55.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:55 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:55.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:55 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:55.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:55 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:55.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:55 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:55.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:55 smithi160 ceph-mon[33203]: Reconfiguring mgr.y (unknown last config time)... 2024-09-21T09:49:55.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:55 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:49:55.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:55 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:49:55.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:55 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:55.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:55 smithi160 ceph-mon[33203]: Reconfiguring daemon mgr.y on smithi070 2024-09-21T09:49:55.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:55 smithi160 ceph-mon[33203]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:55.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:55 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:56.235 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:49:56.514 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:49:56.261+0000 7fe5988e4040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:49:56.514 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:49:56.374+0000 7fe5988e4040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:49:56.514 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:49:56.512+0000 7fe5988e4040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:49:56.792 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:49:56.571+0000 7fe5988e4040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:49:56.792 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:49:56.722+0000 7fe5988e4040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:49:56.792 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:49:56.790+0000 7fe5988e4040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:49:57.069 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:49:56.848+0000 7fe5988e4040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:49:57.069 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:49:57.003+0000 7fe5988e4040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:49:57.069 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:49:57.067+0000 7fe5988e4040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:49:57.332 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:49:57.132+0000 7fe5988e4040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:49:57.332 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:49:57.214+0000 7fe5988e4040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:49:57.332 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:49:57.272+0000 7fe5988e4040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:49:57.332 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:49:57.330+0000 7fe5988e4040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:49:57.914 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:57 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:57.914 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:57 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:57.915 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:57 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:57.915 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:57 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:57.915 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:57 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:57.915 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:57 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:57.915 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:57 smithi070 ceph-mon[41533]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:57.915 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:57 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:57.915 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:57 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:57.915 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:57 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:57.915 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:57 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:57.915 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:57 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:57.915 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:57 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:57.915 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:57 smithi070 ceph-mon[29589]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:57 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:57 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:57 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:57 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:57 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:57 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:49:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:57 smithi160 ceph-mon[33203]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:58.591 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:49:58 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:49:58.102+0000 7fe5988e4040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:49:58.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:58 smithi070 ceph-mon[41533]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:49:58.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:58 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:49:58.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:58 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:49:58.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:58 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:58.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:58 smithi070 ceph-mon[41533]: Standby manager daemon x started 2024-09-21T09:49:58.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:58 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.160:0/3425239573' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:49:58.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:58 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.160:0/3425239573' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:49:58.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:58 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.160:0/3425239573' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:49:58.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:58 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.160:0/3425239573' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:49:58.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:58 smithi070 ceph-mon[29589]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:49:58.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:58 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:49:58.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:58 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:49:58.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:58 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:58.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:58 smithi070 ceph-mon[29589]: Standby manager daemon x started 2024-09-21T09:49:58.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:58 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/3425239573' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:49:58.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:58 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/3425239573' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:49:58.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:58 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/3425239573' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:49:58.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:58 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/3425239573' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:49:59.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:58 smithi160 ceph-mon[33203]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:49:59.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:58 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:49:59.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:58 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:49:59.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:58 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:59.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:58 smithi160 ceph-mon[33203]: Standby manager daemon x started 2024-09-21T09:49:59.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:58 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.160:0/3425239573' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:49:59.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:58 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.160:0/3425239573' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:49:59.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:58 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.160:0/3425239573' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:49:59.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:58 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.160:0/3425239573' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:49:59.756 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:59 smithi070 ceph-mon[29589]: mgrmap e14: y(active, since 63s), standbys: x 2024-09-21T09:49:59.756 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:59 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:49:59.756 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:49:59 smithi070 ceph-mon[29589]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:49:59.757 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:59 smithi070 ceph-mon[41533]: mgrmap e14: y(active, since 63s), standbys: x 2024-09-21T09:49:59.757 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:59 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:49:59.757 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:49:59 smithi070 ceph-mon[41533]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:00.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:59 smithi160 ceph-mon[33203]: mgrmap e14: y(active, since 63s), standbys: x 2024-09-21T09:50:00.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:59 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:50:00.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:49:59 smithi160 ceph-mon[33203]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:00.839 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:00 smithi070 ceph-mon[29589]: overall HEALTH_OK 2024-09-21T09:50:00.839 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:00 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/390511047' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6870ff8-14eb-4110-957a-00c73645b181"}]: dispatch 2024-09-21T09:50:00.839 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:00 smithi070 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6870ff8-14eb-4110-957a-00c73645b181"}]: dispatch 2024-09-21T09:50:00.839 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:00 smithi070 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f6870ff8-14eb-4110-957a-00c73645b181"}]': finished 2024-09-21T09:50:00.839 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:00 smithi070 ceph-mon[29589]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T09:50:00.839 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:00 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:50:00.840 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:00 smithi070 ceph-mon[41533]: overall HEALTH_OK 2024-09-21T09:50:00.840 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:00 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/390511047' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6870ff8-14eb-4110-957a-00c73645b181"}]: dispatch 2024-09-21T09:50:00.840 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:00 smithi070 ceph-mon[41533]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6870ff8-14eb-4110-957a-00c73645b181"}]: dispatch 2024-09-21T09:50:00.840 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:00 smithi070 ceph-mon[41533]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f6870ff8-14eb-4110-957a-00c73645b181"}]': finished 2024-09-21T09:50:00.840 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:00 smithi070 ceph-mon[41533]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T09:50:00.840 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:00 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:50:01.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:00 smithi160 ceph-mon[33203]: overall HEALTH_OK 2024-09-21T09:50:01.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:00 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/390511047' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6870ff8-14eb-4110-957a-00c73645b181"}]: dispatch 2024-09-21T09:50:01.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:00 smithi160 ceph-mon[33203]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6870ff8-14eb-4110-957a-00c73645b181"}]: dispatch 2024-09-21T09:50:01.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:00 smithi160 ceph-mon[33203]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f6870ff8-14eb-4110-957a-00c73645b181"}]': finished 2024-09-21T09:50:01.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:00 smithi160 ceph-mon[33203]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T09:50:01.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:00 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:50:01.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:01 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/3989251963' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:50:01.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:01 smithi070 ceph-mon[29589]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:01.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:01 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/3989251963' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:50:01.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:01 smithi070 ceph-mon[41533]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:02.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:01 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/3989251963' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:50:02.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:01 smithi160 ceph-mon[33203]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:04.436 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:04 smithi070 ceph-mon[29589]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:04.436 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:04 smithi070 ceph-mon[41533]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:04 smithi160 ceph-mon[33203]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:06 smithi160 ceph-mon[33203]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:06 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-21T09:50:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:06 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:06.599 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:06 smithi070 ceph-mon[29589]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:06.599 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:06 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-21T09:50:06.599 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:06 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:06.599 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:06 smithi070 ceph-mon[41533]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:06.600 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:06 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-21T09:50:06.600 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:06 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:07.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:07 smithi160 ceph-mon[33203]: Deploying daemon osd.0 on smithi070 2024-09-21T09:50:07.595 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:07 smithi070 ceph-mon[41533]: Deploying daemon osd.0 on smithi070 2024-09-21T09:50:07.595 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:07 smithi070 ceph-mon[29589]: Deploying daemon osd.0 on smithi070 2024-09-21T09:50:08.435 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:08 smithi070 ceph-mon[29589]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:08.435 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:08 smithi070 ceph-mon[41533]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:08 smithi160 ceph-mon[33203]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:10 smithi160 ceph-mon[33203]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:10.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:10 smithi070 ceph-mon[29589]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:10.610 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:10 smithi070 ceph-mon[41533]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:11.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:11 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:11.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:11 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:11.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:11 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:11.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:11 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:11.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:11 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:11.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:11 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:11.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:11 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:11.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:11 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:11.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:11 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:12 smithi160 ceph-mon[33203]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:12 smithi070 ceph-mon[29589]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:12.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:12 smithi070 ceph-mon[41533]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:14.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:14 smithi070 ceph-mon[29589]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:14.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:14 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:14.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:14 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:14.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:14 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:14.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:14 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:50:14.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:14 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:14.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:14 smithi070 ceph-mon[29589]: from='osd.0 [v2:172.21.15.70:6802/2896279311,v1:172.21.15.70:6803/2896279311]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T09:50:14.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:14 smithi070 ceph-mon[41533]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:14.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:14 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:14.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:14 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:14.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:14 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:14.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:14 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:50:14.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:14 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:14.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:14 smithi070 ceph-mon[41533]: from='osd.0 [v2:172.21.15.70:6802/2896279311,v1:172.21.15.70:6803/2896279311]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T09:50:14.444 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 0 on host 'smithi070' 2024-09-21T09:50:14.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:14 smithi160 ceph-mon[33203]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:14.637 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:14 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:14.637 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:14 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:14.637 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:14 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:14.637 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:14 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:50:14.637 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:14 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:14.637 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:14 smithi160 ceph-mon[33203]: from='osd.0 [v2:172.21.15.70:6802/2896279311,v1:172.21.15.70:6803/2896279311]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T09:50:15.041 DEBUG:teuthology.orchestra.run.smithi070:osd.0> sudo journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.0.service 2024-09-21T09:50:15.044 INFO:tasks.cephadm:Deploying osd.1 on smithi070 with /dev/vg_nvme/lv_3... 2024-09-21T09:50:15.044 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T09:50:15.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:15 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:15.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:15 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:15.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:15 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:15.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:15 smithi070 ceph-mon[41533]: from='osd.0 [v2:172.21.15.70:6802/2896279311,v1:172.21.15.70:6803/2896279311]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T09:50:15.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:15 smithi070 ceph-mon[41533]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T09:50:15.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:15 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:50:15.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:15 smithi070 ceph-mon[41533]: from='osd.0 [v2:172.21.15.70:6802/2896279311,v1:172.21.15.70:6803/2896279311]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-09-21T09:50:15.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:15 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:15.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:15 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:15.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:15 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:15.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:15 smithi070 ceph-mon[29589]: from='osd.0 [v2:172.21.15.70:6802/2896279311,v1:172.21.15.70:6803/2896279311]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T09:50:15.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:15 smithi070 ceph-mon[29589]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T09:50:15.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:15 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:50:15.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:15 smithi070 ceph-mon[29589]: from='osd.0 [v2:172.21.15.70:6802/2896279311,v1:172.21.15.70:6803/2896279311]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-09-21T09:50:15.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:15 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:15.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:15 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:15.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:15 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:15.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:15 smithi160 ceph-mon[33203]: from='osd.0 [v2:172.21.15.70:6802/2896279311,v1:172.21.15.70:6803/2896279311]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T09:50:15.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:15 smithi160 ceph-mon[33203]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T09:50:15.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:15 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:50:15.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:15 smithi160 ceph-mon[33203]: from='osd.0 [v2:172.21.15.70:6802/2896279311,v1:172.21.15.70:6803/2896279311]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-09-21T09:50:15.872 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:50:16.239 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 09:50:15 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0[45840]: 2024-09-21T09:50:15.944+0000 7fceba56c640 -1 osd.0 0 waiting for initial osdmap 2024-09-21T09:50:16.240 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 09:50:15 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0[45840]: 2024-09-21T09:50:15.946+0000 7fceb6395640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:50:16.531 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:16 smithi070 ceph-mon[29589]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:16.531 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:16 smithi070 ceph-mon[29589]: from='osd.0 [v2:172.21.15.70:6802/2896279311,v1:172.21.15.70:6803/2896279311]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-09-21T09:50:16.531 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:16 smithi070 ceph-mon[29589]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T09:50:16.531 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:16 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:50:16.531 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:16 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:50:16.531 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:16 smithi070 ceph-mon[41533]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:16.532 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:16 smithi070 ceph-mon[41533]: from='osd.0 [v2:172.21.15.70:6802/2896279311,v1:172.21.15.70:6803/2896279311]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-09-21T09:50:16.532 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:16 smithi070 ceph-mon[41533]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T09:50:16.532 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:16 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:50:16.532 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:16 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:50:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:16 smithi160 ceph-mon[33203]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:16 smithi160 ceph-mon[33203]: from='osd.0 [v2:172.21.15.70:6802/2896279311,v1:172.21.15.70:6803/2896279311]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-09-21T09:50:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:16 smithi160 ceph-mon[33203]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T09:50:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:16 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:50:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:16 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:50:18.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:17 smithi070 ceph-mon[29589]: purged_snaps scrub starts 2024-09-21T09:50:18.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:17 smithi070 ceph-mon[29589]: purged_snaps scrub ok 2024-09-21T09:50:18.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:17 smithi070 ceph-mon[29589]: osd.0 [v2:172.21.15.70:6802/2896279311,v1:172.21.15.70:6803/2896279311] boot 2024-09-21T09:50:18.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:17 smithi070 ceph-mon[29589]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T09:50:18.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:17 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:50:18.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:17 smithi070 ceph-mon[29589]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:18.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:17 smithi070 ceph-mon[41533]: purged_snaps scrub starts 2024-09-21T09:50:18.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:17 smithi070 ceph-mon[41533]: purged_snaps scrub ok 2024-09-21T09:50:18.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:17 smithi070 ceph-mon[41533]: osd.0 [v2:172.21.15.70:6802/2896279311,v1:172.21.15.70:6803/2896279311] boot 2024-09-21T09:50:18.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:17 smithi070 ceph-mon[41533]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T09:50:18.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:17 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:50:18.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:17 smithi070 ceph-mon[41533]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:17 smithi160 ceph-mon[33203]: purged_snaps scrub starts 2024-09-21T09:50:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:17 smithi160 ceph-mon[33203]: purged_snaps scrub ok 2024-09-21T09:50:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:17 smithi160 ceph-mon[33203]: osd.0 [v2:172.21.15.70:6802/2896279311,v1:172.21.15.70:6803/2896279311] boot 2024-09-21T09:50:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:17 smithi160 ceph-mon[33203]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T09:50:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:17 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:50:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:17 smithi160 ceph-mon[33203]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:50:19.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:18 smithi070 ceph-mon[41533]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T09:50:19.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:18 smithi070 ceph-mon[41533]: Detected new or changed devices on smithi070 2024-09-21T09:50:19.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:18 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:19.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:18 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:19.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:18 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:19.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:18 smithi070 ceph-mon[41533]: Adjusting osd_memory_target on smithi070 to 16064M 2024-09-21T09:50:19.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:18 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:19.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:18 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:19.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:18 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:50:19.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:18 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:19.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:18 smithi070 ceph-mon[29589]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T09:50:19.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:18 smithi070 ceph-mon[29589]: Detected new or changed devices on smithi070 2024-09-21T09:50:19.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:18 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:19.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:18 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:19.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:18 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:19.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:18 smithi070 ceph-mon[29589]: Adjusting osd_memory_target on smithi070 to 16064M 2024-09-21T09:50:19.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:18 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:19.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:18 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:19.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:18 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:50:19.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:18 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:19.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:18 smithi160 ceph-mon[33203]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T09:50:19.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:18 smithi160 ceph-mon[33203]: Detected new or changed devices on smithi070 2024-09-21T09:50:19.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:18 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:19.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:18 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:19.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:18 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:19.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:18 smithi160 ceph-mon[33203]: Adjusting osd_memory_target on smithi070 to 16064M 2024-09-21T09:50:19.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:18 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:19.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:18 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:19.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:18 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:50:19.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:18 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:19.810 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:50:19.834 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi070:vg_nvme/lv_3 2024-09-21T09:50:20.027 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:50:20.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:19 smithi070 ceph-mon[29589]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:20.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:19 smithi070 ceph-mon[41533]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:20.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:19 smithi160 ceph-mon[33203]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:22.382 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:22 smithi070 ceph-mon[29589]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:22.382 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:22 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:50:22.382 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:22 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:50:22.382 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:22 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:22.382 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:22 smithi070 ceph-mon[41533]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:22.382 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:22 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:50:22.382 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:22 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:50:22.383 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:22 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:22.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:22 smithi160 ceph-mon[33203]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:22.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:22 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:50:22.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:22 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:50:22.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:22 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:23.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:23 smithi070 ceph-mon[29589]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:23.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:23 smithi070 ceph-mon[41533]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:23.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:23 smithi160 ceph-mon[33203]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:24 smithi160 ceph-mon[33203]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:24 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/2806787447' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b1349e31-7493-489f-823a-55edb866d25f"}]: dispatch 2024-09-21T09:50:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:24 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/2806787447' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b1349e31-7493-489f-823a-55edb866d25f"}]': finished 2024-09-21T09:50:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:24 smithi160 ceph-mon[33203]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T09:50:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:24 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:50:24.623 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:24 smithi070 ceph-mon[29589]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:24.623 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:24 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/2806787447' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b1349e31-7493-489f-823a-55edb866d25f"}]: dispatch 2024-09-21T09:50:24.623 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:24 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/2806787447' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b1349e31-7493-489f-823a-55edb866d25f"}]': finished 2024-09-21T09:50:24.623 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:24 smithi070 ceph-mon[29589]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T09:50:24.623 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:24 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:50:24.624 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:24 smithi070 ceph-mon[41533]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:24.624 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:24 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/2806787447' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b1349e31-7493-489f-823a-55edb866d25f"}]: dispatch 2024-09-21T09:50:24.624 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:24 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/2806787447' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b1349e31-7493-489f-823a-55edb866d25f"}]': finished 2024-09-21T09:50:24.624 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:24 smithi070 ceph-mon[41533]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T09:50:24.624 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:24 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:50:25.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:25 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/1393395500' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:50:25.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:25 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/1393395500' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:50:25.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:25 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/1393395500' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:50:26.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:26 smithi160 ceph-mon[33203]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:26 smithi070 ceph-mon[29589]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:26.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:26 smithi070 ceph-mon[41533]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:28.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:28 smithi070 ceph-mon[29589]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:28.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:28 smithi070 ceph-mon[41533]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:28.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:28 smithi160 ceph-mon[33203]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:30.568 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:30 smithi070 ceph-mon[29589]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:30.568 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:30 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-21T09:50:30.568 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:30 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:30.568 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:30 smithi070 ceph-mon[41533]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:30.568 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:30 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-21T09:50:30.568 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:30 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:30 smithi160 ceph-mon[33203]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:30 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-21T09:50:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:30 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:31.330 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:31 smithi070 ceph-mon[29589]: Deploying daemon osd.1 on smithi070 2024-09-21T09:50:31.330 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:31 smithi070 ceph-mon[41533]: Deploying daemon osd.1 on smithi070 2024-09-21T09:50:31.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:31 smithi160 ceph-mon[33203]: Deploying daemon osd.1 on smithi070 2024-09-21T09:50:32.536 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:32 smithi070 ceph-mon[29589]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:32.537 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:32 smithi070 ceph-mon[41533]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:32.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:32 smithi160 ceph-mon[33203]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:34.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:34 smithi160 ceph-mon[33203]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:34.619 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:34 smithi070 ceph-mon[29589]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:34.619 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:34 smithi070 ceph-mon[41533]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:35.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:35 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:35.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:35 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:35.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:35 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:35.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:35 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:35.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:35 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:35.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:35 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:35.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:35 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:35.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:35 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:35.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:35 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:36 smithi160 ceph-mon[33203]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:36.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:36 smithi070 ceph-mon[41533]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:36 smithi070 ceph-mon[29589]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:38.449 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 1 on host 'smithi070' 2024-09-21T09:50:38.450 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:38 smithi070 ceph-mon[41533]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:38.450 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:38 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:38.450 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:38 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:38.450 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:38 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:38.450 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:38 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:50:38.450 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:38 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:38.450 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:38 smithi070 ceph-mon[41533]: from='osd.1 [v2:172.21.15.70:6810/937875216,v1:172.21.15.70:6811/937875216]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-21T09:50:38.450 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:38 smithi070 ceph-mon[41533]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-21T09:50:38.451 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:38 smithi070 ceph-mon[29589]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:38.451 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:38 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:38.451 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:38 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:38.451 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:38 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:38.451 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:38 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:50:38.451 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:38 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:38.451 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:38 smithi070 ceph-mon[29589]: from='osd.1 [v2:172.21.15.70:6810/937875216,v1:172.21.15.70:6811/937875216]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-21T09:50:38.451 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:38 smithi070 ceph-mon[29589]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-21T09:50:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:38 smithi160 ceph-mon[33203]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:38 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:38 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:38 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:38.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:38 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:50:38.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:38 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:38.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:38 smithi160 ceph-mon[33203]: from='osd.1 [v2:172.21.15.70:6810/937875216,v1:172.21.15.70:6811/937875216]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-21T09:50:38.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:38 smithi160 ceph-mon[33203]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-21T09:50:39.055 DEBUG:teuthology.orchestra.run.smithi070:osd.1> sudo journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.1.service 2024-09-21T09:50:39.057 INFO:tasks.cephadm:Deploying osd.2 on smithi070 with /dev/vg_nvme/lv_2... 2024-09-21T09:50:39.057 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T09:50:39.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:39 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:39.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:39 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:39.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:39 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:39.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:39 smithi160 ceph-mon[33203]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T09:50:39.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:39 smithi160 ceph-mon[33203]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T09:50:39.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:39 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:50:39.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:39 smithi160 ceph-mon[33203]: from='osd.1 [v2:172.21.15.70:6810/937875216,v1:172.21.15.70:6811/937875216]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-09-21T09:50:39.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:39 smithi160 ceph-mon[33203]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-09-21T09:50:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:39 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:39 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:39 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:39 smithi070 ceph-mon[29589]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T09:50:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:39 smithi070 ceph-mon[29589]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T09:50:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:39 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:50:39.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:39 smithi070 ceph-mon[29589]: from='osd.1 [v2:172.21.15.70:6810/937875216,v1:172.21.15.70:6811/937875216]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-09-21T09:50:39.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:39 smithi070 ceph-mon[29589]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-09-21T09:50:39.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:39 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:39.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:39 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:39.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:39 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:39.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:39 smithi070 ceph-mon[41533]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T09:50:39.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:39 smithi070 ceph-mon[41533]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T09:50:39.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:39 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:50:39.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:39 smithi070 ceph-mon[41533]: from='osd.1 [v2:172.21.15.70:6810/937875216,v1:172.21.15.70:6811/937875216]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-09-21T09:50:39.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:39 smithi070 ceph-mon[41533]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-09-21T09:50:39.764 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:50:40.121 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 09:50:39 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1[51853]: 2024-09-21T09:50:39.778+0000 7f8378b28640 -1 osd.1 0 waiting for initial osdmap 2024-09-21T09:50:40.122 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 09:50:39 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1[51853]: 2024-09-21T09:50:39.781+0000 7f8374951640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:50:40.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:40 smithi070 ceph-mon[29589]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:40.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:40 smithi070 ceph-mon[29589]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-09-21T09:50:40.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:40 smithi070 ceph-mon[29589]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T09:50:40.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:40 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:50:40.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:40 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:50:40.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:40 smithi070 ceph-mon[41533]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:40.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:40 smithi070 ceph-mon[41533]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-09-21T09:50:40.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:40 smithi070 ceph-mon[41533]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T09:50:40.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:40 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:50:40.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:40 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:50:40.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:40 smithi160 ceph-mon[33203]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:50:40.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:40 smithi160 ceph-mon[33203]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-09-21T09:50:40.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:40 smithi160 ceph-mon[33203]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T09:50:40.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:40 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:50:40.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:40 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:50:42.090 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:41 smithi070 ceph-mon[29589]: purged_snaps scrub starts 2024-09-21T09:50:42.091 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:41 smithi070 ceph-mon[29589]: purged_snaps scrub ok 2024-09-21T09:50:42.091 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:41 smithi070 ceph-mon[29589]: osd.1 [v2:172.21.15.70:6810/937875216,v1:172.21.15.70:6811/937875216] boot 2024-09-21T09:50:42.091 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:41 smithi070 ceph-mon[29589]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T09:50:42.091 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:41 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:50:42.091 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:41 smithi070 ceph-mon[29589]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:50:42.091 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:41 smithi070 ceph-mon[41533]: purged_snaps scrub starts 2024-09-21T09:50:42.091 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:41 smithi070 ceph-mon[41533]: purged_snaps scrub ok 2024-09-21T09:50:42.091 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:41 smithi070 ceph-mon[41533]: osd.1 [v2:172.21.15.70:6810/937875216,v1:172.21.15.70:6811/937875216] boot 2024-09-21T09:50:42.091 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:41 smithi070 ceph-mon[41533]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T09:50:42.091 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:41 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:50:42.092 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:41 smithi070 ceph-mon[41533]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:50:42.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:41 smithi160 ceph-mon[33203]: purged_snaps scrub starts 2024-09-21T09:50:42.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:41 smithi160 ceph-mon[33203]: purged_snaps scrub ok 2024-09-21T09:50:42.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:41 smithi160 ceph-mon[33203]: osd.1 [v2:172.21.15.70:6810/937875216,v1:172.21.15.70:6811/937875216] boot 2024-09-21T09:50:42.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:41 smithi160 ceph-mon[33203]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T09:50:42.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:41 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:50:42.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:41 smithi160 ceph-mon[33203]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:50:43.061 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:42 smithi070 ceph-mon[41533]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T09:50:43.061 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:42 smithi070 ceph-mon[41533]: Detected new or changed devices on smithi070 2024-09-21T09:50:43.061 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:43.062 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:43.062 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:43.062 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:43.062 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:43.062 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:43.062 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:50:43.062 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:43.062 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:42 smithi070 ceph-mon[29589]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T09:50:43.062 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:42 smithi070 ceph-mon[29589]: Detected new or changed devices on smithi070 2024-09-21T09:50:43.062 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:43.062 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:43.062 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:43.063 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:43.063 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:43.063 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:43.063 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:50:43.063 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:42 smithi160 ceph-mon[33203]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T09:50:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:42 smithi160 ceph-mon[33203]: Detected new or changed devices on smithi070 2024-09-21T09:50:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:43.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:43.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:43.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:50:43.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:43.684 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:50:43.701 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi070:vg_nvme/lv_2 2024-09-21T09:50:43.897 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:50:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:43 smithi160 ceph-mon[33203]: Adjusting osd_memory_target on smithi070 to 8032M 2024-09-21T09:50:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:43 smithi160 ceph-mon[33203]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:50:44.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:43 smithi070 ceph-mon[29589]: Adjusting osd_memory_target on smithi070 to 8032M 2024-09-21T09:50:44.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:43 smithi070 ceph-mon[29589]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:50:44.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:43 smithi070 ceph-mon[41533]: Adjusting osd_memory_target on smithi070 to 8032M 2024-09-21T09:50:44.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:43 smithi070 ceph-mon[41533]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:50:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:46 smithi160 ceph-mon[33203]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:50:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:46 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:50:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:46 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:50:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:46 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:46.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:46 smithi070 ceph-mon[29589]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:50:46.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:46 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:50:46.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:46 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:50:46.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:46 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:46.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:46 smithi070 ceph-mon[41533]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:50:46.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:46 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:50:46.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:46 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:50:46.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:46 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:47.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:47 smithi160 ceph-mon[33203]: from='client.24176 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:47.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:47 smithi070 ceph-mon[29589]: from='client.24176 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:47.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:47 smithi070 ceph-mon[41533]: from='client.24176 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:48.535 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:48 smithi070 ceph-mon[29589]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:50:48.535 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:48 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/1821865332' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ac4496ba-a860-4b09-bb38-7f433e081541"}]: dispatch 2024-09-21T09:50:48.535 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:48 smithi070 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ac4496ba-a860-4b09-bb38-7f433e081541"}]: dispatch 2024-09-21T09:50:48.535 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:48 smithi070 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ac4496ba-a860-4b09-bb38-7f433e081541"}]': finished 2024-09-21T09:50:48.535 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:48 smithi070 ceph-mon[29589]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T09:50:48.535 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:48 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:50:48.535 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:48 smithi070 ceph-mon[41533]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:50:48.535 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:48 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/1821865332' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ac4496ba-a860-4b09-bb38-7f433e081541"}]: dispatch 2024-09-21T09:50:48.535 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:48 smithi070 ceph-mon[41533]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ac4496ba-a860-4b09-bb38-7f433e081541"}]: dispatch 2024-09-21T09:50:48.535 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:48 smithi070 ceph-mon[41533]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ac4496ba-a860-4b09-bb38-7f433e081541"}]': finished 2024-09-21T09:50:48.536 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:48 smithi070 ceph-mon[41533]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T09:50:48.536 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:48 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:50:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:48 smithi160 ceph-mon[33203]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:50:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:48 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/1821865332' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ac4496ba-a860-4b09-bb38-7f433e081541"}]: dispatch 2024-09-21T09:50:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:48 smithi160 ceph-mon[33203]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ac4496ba-a860-4b09-bb38-7f433e081541"}]: dispatch 2024-09-21T09:50:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:48 smithi160 ceph-mon[33203]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ac4496ba-a860-4b09-bb38-7f433e081541"}]': finished 2024-09-21T09:50:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:48 smithi160 ceph-mon[33203]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T09:50:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:48 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:50:49.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:49 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/2500157194' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:50:49.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:49 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/2500157194' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:50:49.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:49 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/2500157194' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:50:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:50 smithi160 ceph-mon[33203]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:50:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:50 smithi070 ceph-mon[29589]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:50:50.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:50 smithi070 ceph-mon[41533]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:50:52.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:52 smithi070 ceph-mon[29589]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:50:52.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:52 smithi070 ceph-mon[41533]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:50:52.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:52 smithi160 ceph-mon[33203]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:50:54.493 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:54 smithi070 ceph-mon[41533]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:50:54.493 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:54 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-21T09:50:54.493 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:54 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:54.493 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:54 smithi070 ceph-mon[29589]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:50:54.493 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:54 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-21T09:50:54.493 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:54 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:54 smithi160 ceph-mon[33203]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:50:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:54 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-21T09:50:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:54 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:55.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:55 smithi070 ceph-mon[29589]: Deploying daemon osd.2 on smithi070 2024-09-21T09:50:55.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:55 smithi070 ceph-mon[41533]: Deploying daemon osd.2 on smithi070 2024-09-21T09:50:55.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:55 smithi160 ceph-mon[33203]: Deploying daemon osd.2 on smithi070 2024-09-21T09:50:56.517 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:56 smithi070 ceph-mon[29589]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:50:56.517 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:56 smithi070 ceph-mon[41533]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:50:56.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:56 smithi160 ceph-mon[33203]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:50:58.518 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:58 smithi070 ceph-mon[29589]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:50:58.518 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:58 smithi070 ceph-mon[41533]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:50:58.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:58 smithi160 ceph-mon[33203]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:50:59.398 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:59 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:59.398 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:59 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:59.398 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:50:59 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:59.398 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:59 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:59.398 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:59 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:59.398 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:50:59 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:59.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:59 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:59.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:59 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:50:59.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:50:59 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:00.569 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:00 smithi070 ceph-mon[29589]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:51:00.569 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:00 smithi070 ceph-mon[41533]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:51:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:00 smithi160 ceph-mon[33203]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:51:01.842 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 2 on host 'smithi070' 2024-09-21T09:51:02.382 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:02 smithi070 ceph-mon[29589]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:51:02.382 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:02 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:02.382 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:02 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:02.382 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:02 smithi070 ceph-mon[29589]: from='osd.2 [v2:172.21.15.70:6818/940406420,v1:172.21.15.70:6819/940406420]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-21T09:51:02.383 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:02 smithi070 ceph-mon[41533]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:51:02.383 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:02 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:02.383 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:02 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:02.383 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:02 smithi070 ceph-mon[41533]: from='osd.2 [v2:172.21.15.70:6818/940406420,v1:172.21.15.70:6819/940406420]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-21T09:51:02.384 DEBUG:teuthology.orchestra.run.smithi070:osd.2> sudo journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.2.service 2024-09-21T09:51:02.386 INFO:tasks.cephadm:Deploying osd.3 on smithi070 with /dev/vg_nvme/lv_1... 2024-09-21T09:51:02.386 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T09:51:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:02 smithi160 ceph-mon[33203]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:51:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:02 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:02 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:02 smithi160 ceph-mon[33203]: from='osd.2 [v2:172.21.15.70:6818/940406420,v1:172.21.15.70:6819/940406420]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-21T09:51:02.625 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:03.719 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:03 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:03.719 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:03 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:03.719 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:03 smithi070 ceph-mon[29589]: from='osd.2 [v2:172.21.15.70:6818/940406420,v1:172.21.15.70:6819/940406420]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T09:51:03.719 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:03 smithi070 ceph-mon[29589]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T09:51:03.719 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:03 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:51:03.719 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:03 smithi070 ceph-mon[29589]: from='osd.2 [v2:172.21.15.70:6818/940406420,v1:172.21.15.70:6819/940406420]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-09-21T09:51:03.719 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:03 smithi070 ceph-mon[29589]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:51:03.719 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:03 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:03.720 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:03 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:03.720 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:03 smithi070 ceph-mon[41533]: from='osd.2 [v2:172.21.15.70:6818/940406420,v1:172.21.15.70:6819/940406420]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T09:51:03.720 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:03 smithi070 ceph-mon[41533]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T09:51:03.720 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:03 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:51:03.720 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:03 smithi070 ceph-mon[41533]: from='osd.2 [v2:172.21.15.70:6818/940406420,v1:172.21.15.70:6819/940406420]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-09-21T09:51:03.720 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:03 smithi070 ceph-mon[41533]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:51:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:03 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:03 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:03 smithi160 ceph-mon[33203]: from='osd.2 [v2:172.21.15.70:6818/940406420,v1:172.21.15.70:6819/940406420]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T09:51:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:03 smithi160 ceph-mon[33203]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T09:51:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:03 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:51:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:03 smithi160 ceph-mon[33203]: from='osd.2 [v2:172.21.15.70:6818/940406420,v1:172.21.15.70:6819/940406420]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-09-21T09:51:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:03 smithi160 ceph-mon[33203]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:51:04.274 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 09:51:04 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2[58230]: 2024-09-21T09:51:04.095+0000 7f3a28446640 -1 osd.2 0 waiting for initial osdmap 2024-09-21T09:51:04.274 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 09:51:04 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2[58230]: 2024-09-21T09:51:04.099+0000 7f3a20267640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:51:05.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:04 smithi070 ceph-mon[29589]: from='osd.2 [v2:172.21.15.70:6818/940406420,v1:172.21.15.70:6819/940406420]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-09-21T09:51:05.047 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:04 smithi070 ceph-mon[29589]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T09:51:05.047 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:04 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:51:05.047 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:04 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:51:05.047 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:04 smithi070 ceph-mon[29589]: from='osd.2 [v2:172.21.15.70:6818/940406420,v1:172.21.15.70:6819/940406420]' entity='osd.2' 2024-09-21T09:51:05.047 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:04 smithi070 ceph-mon[41533]: from='osd.2 [v2:172.21.15.70:6818/940406420,v1:172.21.15.70:6819/940406420]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-09-21T09:51:05.047 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:04 smithi070 ceph-mon[41533]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T09:51:05.047 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:04 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:51:05.047 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:04 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:51:05.047 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:04 smithi070 ceph-mon[41533]: from='osd.2 [v2:172.21.15.70:6818/940406420,v1:172.21.15.70:6819/940406420]' entity='osd.2' 2024-09-21T09:51:05.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:04 smithi160 ceph-mon[33203]: from='osd.2 [v2:172.21.15.70:6818/940406420,v1:172.21.15.70:6819/940406420]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-09-21T09:51:05.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:04 smithi160 ceph-mon[33203]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T09:51:05.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:04 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:51:05.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:04 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:51:05.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:04 smithi160 ceph-mon[33203]: from='osd.2 [v2:172.21.15.70:6818/940406420,v1:172.21.15.70:6819/940406420]' entity='osd.2' 2024-09-21T09:51:05.784 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:51:05.803 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi070:vg_nvme/lv_1 2024-09-21T09:51:06.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:05 smithi070 ceph-mon[29589]: purged_snaps scrub starts 2024-09-21T09:51:06.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:05 smithi070 ceph-mon[29589]: purged_snaps scrub ok 2024-09-21T09:51:06.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:05 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:51:06.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:05 smithi070 ceph-mon[29589]: osd.2 [v2:172.21.15.70:6818/940406420,v1:172.21.15.70:6819/940406420] boot 2024-09-21T09:51:06.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:05 smithi070 ceph-mon[29589]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T09:51:06.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:05 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:51:06.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:05 smithi070 ceph-mon[29589]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:51:06.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:05 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:51:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:05 smithi070 ceph-mon[41533]: purged_snaps scrub starts 2024-09-21T09:51:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:05 smithi070 ceph-mon[41533]: purged_snaps scrub ok 2024-09-21T09:51:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:05 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:51:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:05 smithi070 ceph-mon[41533]: osd.2 [v2:172.21.15.70:6818/940406420,v1:172.21.15.70:6819/940406420] boot 2024-09-21T09:51:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:05 smithi070 ceph-mon[41533]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T09:51:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:05 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:51:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:05 smithi070 ceph-mon[41533]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:51:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:05 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:51:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:05 smithi160 ceph-mon[33203]: purged_snaps scrub starts 2024-09-21T09:51:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:05 smithi160 ceph-mon[33203]: purged_snaps scrub ok 2024-09-21T09:51:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:05 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:51:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:05 smithi160 ceph-mon[33203]: osd.2 [v2:172.21.15.70:6818/940406420,v1:172.21.15.70:6819/940406420] boot 2024-09-21T09:51:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:05 smithi160 ceph-mon[33203]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T09:51:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:05 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:51:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:05 smithi160 ceph-mon[33203]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T09:51:06.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:05 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:51:06.374 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:07.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:07 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T09:51:07.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:07 smithi070 ceph-mon[29589]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T09:51:07.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:07 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:51:07.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:07 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T09:51:07.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:07 smithi070 ceph-mon[41533]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T09:51:07.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:07 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:51:07.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:07 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T09:51:07.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:07 smithi160 ceph-mon[33203]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T09:51:07.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:07 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:51:07.689 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 09:51:07 smithi070 sudo[62224]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T09:51:07.689 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 09:51:07 smithi070 sudo[62224]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:51:07.689 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 09:51:07 smithi070 sudo[62224]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:51:07.689 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 09:51:07 smithi070 sudo[62224]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:51:07.963 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 09:51:07 smithi070 sudo[62227]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T09:51:07.963 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 09:51:07 smithi070 sudo[62227]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:51:07.963 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 09:51:07 smithi070 sudo[62227]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:51:07.963 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 09:51:07 smithi070 sudo[62227]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:51:08.226 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 09:51:07 smithi070 sudo[62300]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T09:51:08.226 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 09:51:07 smithi070 sudo[62300]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:51:08.226 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 09:51:07 smithi070 sudo[62300]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:51:08.227 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 09:51:07 smithi070 sudo[62300]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:51:08.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T09:51:08.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[29589]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T09:51:08.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[29589]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:08.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:08.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:08.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:08.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:08.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:08.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:08.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:08.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:08.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:08.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:08.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:08.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:08.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:08.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:08.228 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T09:51:08.228 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[41533]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T09:51:08.228 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[41533]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:08.228 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:08.229 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:08.229 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:08.229 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:08.229 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:08.229 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:08.229 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:08.229 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:08.229 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:08.229 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:08.229 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:08.229 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:08.229 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:08.229 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:08 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:08.479 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 09:51:08 smithi070 sudo[62324]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T09:51:08.479 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 09:51:08 smithi070 sudo[62324]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:51:08.479 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 09:51:08 smithi070 sudo[62324]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:51:08.479 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 09:51:08 smithi070 sudo[62324]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:51:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:08 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T09:51:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:08 smithi160 ceph-mon[33203]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T09:51:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:08 smithi160 ceph-mon[33203]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:08 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:08 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:08 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:08 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:08 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:08.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:08 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:08.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:08 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:08.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:08 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:08.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:08 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:08.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:08 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:08.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:08 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:08.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:08 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:08.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:08 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:08.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:08 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:08.745 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 09:51:08 smithi070 sudo[62350]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T09:51:08.745 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 09:51:08 smithi070 sudo[62350]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:51:08.745 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 09:51:08 smithi070 sudo[62350]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:51:08.745 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 09:51:08 smithi070 sudo[62350]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:51:09.006 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 09:51:08 smithi070 sudo[62379]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T09:51:09.007 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 09:51:08 smithi070 sudo[62379]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:51:09.007 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 09:51:08 smithi070 sudo[62379]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:51:09.007 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 09:51:08 smithi070 sudo[62379]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:51:09.007 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:08 smithi070 sudo[62383]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T09:51:09.007 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:08 smithi070 sudo[62383]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:51:09.007 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:08 smithi070 sudo[62383]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:51:09.007 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:09 smithi070 sudo[62383]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:51:09.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:09 smithi070 ceph-mon[41533]: Detected new or changed devices on smithi070 2024-09-21T09:51:09.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:09 smithi070 ceph-mon[41533]: Adjusting osd_memory_target on smithi070 to 5354M 2024-09-21T09:51:09.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:09 smithi070 ceph-mon[41533]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T09:51:09.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:09 smithi070 ceph-mon[41533]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-21T09:51:09.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:09 smithi070 ceph-mon[41533]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:51:09.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:09 smithi070 sudo[62386]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T09:51:09.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:09 smithi070 sudo[62386]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:51:09.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:09 smithi070 sudo[62386]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:51:09.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:09 smithi070 sudo[62386]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:51:09.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:09 smithi070 ceph-mon[29589]: Detected new or changed devices on smithi070 2024-09-21T09:51:09.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:09 smithi070 ceph-mon[29589]: Adjusting osd_memory_target on smithi070 to 5354M 2024-09-21T09:51:09.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:09 smithi070 ceph-mon[29589]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T09:51:09.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:09 smithi070 ceph-mon[29589]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-21T09:51:09.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:09 smithi070 ceph-mon[29589]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:51:09.513 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:09 smithi160 ceph-mon[33203]: Detected new or changed devices on smithi070 2024-09-21T09:51:09.513 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:09 smithi160 ceph-mon[33203]: Adjusting osd_memory_target on smithi070 to 5354M 2024-09-21T09:51:09.514 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:09 smithi160 ceph-mon[33203]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T09:51:09.514 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:09 smithi160 ceph-mon[33203]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-21T09:51:09.514 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:09 smithi160 ceph-mon[33203]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:51:09.514 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:09 smithi160 sudo[35964]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T09:51:09.514 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:09 smithi160 sudo[35964]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:51:09.514 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:09 smithi160 sudo[35964]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:51:09.514 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:09 smithi160 sudo[35964]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:51:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:09 smithi160 sudo[35967]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T09:51:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:09 smithi160 sudo[35967]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:51:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:09 smithi160 sudo[35967]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:51:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:09 smithi160 sudo[35967]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:51:10.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:10 smithi070 sudo[62549]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T09:51:10.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:10 smithi070 sudo[62549]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:51:10.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:10 smithi070 sudo[62549]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:51:10.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:10 smithi070 sudo[62549]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:51:10.560 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[29589]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:10.560 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[29589]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:10.560 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[29589]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:51:10.560 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:51:10.560 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:10.560 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:10.560 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:51:10.560 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:51:10.560 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:10.560 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[29589]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:51:10.561 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:51:10.561 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:10.561 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:10.561 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[29589]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:51:10.561 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:10.561 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:51:10.561 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:10.561 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:10.561 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:10 smithi070 sudo[62558]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T09:51:10.561 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:10 smithi070 sudo[62558]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:51:10.561 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:10 smithi070 sudo[62558]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:51:10.562 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:10 smithi070 sudo[62558]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:51:10.562 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[41533]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:10.562 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[41533]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:10.562 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[41533]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:51:10.562 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:51:10.562 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:10.562 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:10.562 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:51:10.562 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:51:10.562 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:10.562 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[41533]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:51:10.562 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:51:10.562 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:10.562 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:10.563 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[41533]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:51:10.563 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:10.563 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:51:10.563 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:10.563 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:10 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:10 smithi160 ceph-mon[33203]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:10 smithi160 ceph-mon[33203]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:10 smithi160 ceph-mon[33203]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:51:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:10 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:51:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:10 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:10 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:10 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:51:10.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:10 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:51:10.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:10 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:10.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:10 smithi160 ceph-mon[33203]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:51:10.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:10 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:51:10.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:10 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:10.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:10 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:10.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:10 smithi160 ceph-mon[33203]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:51:10.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:10 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:10.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:10 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:51:10.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:10 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:10.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:10 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:11.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:11 smithi160 ceph-mon[33203]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:51:11.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:11 smithi160 ceph-mon[33203]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:51:11.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:11 smithi070 ceph-mon[29589]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:51:11.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:11 smithi070 ceph-mon[29589]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:51:11.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:11 smithi070 ceph-mon[41533]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:51:11.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:11 smithi070 ceph-mon[41533]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:51:12.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:12 smithi160 ceph-mon[33203]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:12.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:12 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/679083359' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1eedf4ce-057c-4576-ae5b-51d3595df195"}]: dispatch 2024-09-21T09:51:12.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:12 smithi160 ceph-mon[33203]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1eedf4ce-057c-4576-ae5b-51d3595df195"}]: dispatch 2024-09-21T09:51:12.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:12 smithi160 ceph-mon[33203]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1eedf4ce-057c-4576-ae5b-51d3595df195"}]': finished 2024-09-21T09:51:12.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:12 smithi160 ceph-mon[33203]: osdmap e22: 4 total, 3 up, 4 in 2024-09-21T09:51:12.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:12 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:51:12.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:12 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/3023125094' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:51:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:12 smithi070 ceph-mon[29589]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:12 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/679083359' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1eedf4ce-057c-4576-ae5b-51d3595df195"}]: dispatch 2024-09-21T09:51:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:12 smithi070 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1eedf4ce-057c-4576-ae5b-51d3595df195"}]: dispatch 2024-09-21T09:51:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:12 smithi070 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1eedf4ce-057c-4576-ae5b-51d3595df195"}]': finished 2024-09-21T09:51:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:12 smithi070 ceph-mon[29589]: osdmap e22: 4 total, 3 up, 4 in 2024-09-21T09:51:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:12 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:51:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:12 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/3023125094' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:51:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:12 smithi070 ceph-mon[41533]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:12 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/679083359' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1eedf4ce-057c-4576-ae5b-51d3595df195"}]: dispatch 2024-09-21T09:51:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:12 smithi070 ceph-mon[41533]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1eedf4ce-057c-4576-ae5b-51d3595df195"}]: dispatch 2024-09-21T09:51:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:12 smithi070 ceph-mon[41533]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1eedf4ce-057c-4576-ae5b-51d3595df195"}]': finished 2024-09-21T09:51:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:12 smithi070 ceph-mon[41533]: osdmap e22: 4 total, 3 up, 4 in 2024-09-21T09:51:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:12 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:51:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:12 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/3023125094' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:51:14.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:14 smithi160 ceph-mon[33203]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:14 smithi070 ceph-mon[29589]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:14.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:14 smithi070 ceph-mon[41533]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:16.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:16 smithi070 ceph-mon[29589]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:16.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:16 smithi070 ceph-mon[41533]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:16 smithi160 ceph-mon[33203]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:18 smithi160 ceph-mon[33203]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:18 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-21T09:51:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:18 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:18 smithi070 ceph-mon[29589]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:18 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-21T09:51:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:18 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:18.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:18 smithi070 ceph-mon[41533]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:18.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:18 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-21T09:51:18.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:18 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:19.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:19 smithi070 ceph-mon[29589]: Deploying daemon osd.3 on smithi070 2024-09-21T09:51:19.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:19 smithi070 ceph-mon[41533]: Deploying daemon osd.3 on smithi070 2024-09-21T09:51:19.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:19 smithi160 ceph-mon[33203]: Deploying daemon osd.3 on smithi070 2024-09-21T09:51:20.539 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:20 smithi070 ceph-mon[29589]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:20.539 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:20 smithi070 ceph-mon[41533]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:20.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:20 smithi160 ceph-mon[33203]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:22.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:22 smithi160 ceph-mon[33203]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:22 smithi070 ceph-mon[29589]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:22.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:22 smithi070 ceph-mon[41533]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:23.540 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:23 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:23.540 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:23 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:23.540 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:23 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:23.540 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:23 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:23.540 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:23 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:23.540 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:23 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:23.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:23 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:23.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:23 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:23.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:23 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:24.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:24 smithi070 ceph-mon[29589]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:24.359 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:24 smithi070 ceph-mon[41533]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:24 smithi160 ceph-mon[33203]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:25.929 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 3 on host 'smithi070' 2024-09-21T09:51:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:26 smithi070 ceph-mon[29589]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:26 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:26 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:26 smithi070 ceph-mon[29589]: from='osd.3 [v2:172.21.15.70:6826/1341239112,v1:172.21.15.70:6827/1341239112]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-21T09:51:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:26 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:26.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:26 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:26.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:26 smithi070 ceph-mon[41533]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:26.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:26 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:26.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:26 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:26.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:26 smithi070 ceph-mon[41533]: from='osd.3 [v2:172.21.15.70:6826/1341239112,v1:172.21.15.70:6827/1341239112]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-21T09:51:26.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:26 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:26.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:26 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:26.676 DEBUG:teuthology.orchestra.run.smithi070:osd.3> sudo journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.3.service 2024-09-21T09:51:26.678 INFO:tasks.cephadm:Deploying osd.4 on smithi160 with /dev/vg_nvme/lv_4... 2024-09-21T09:51:26.678 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T09:51:26.707 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:26 smithi160 ceph-mon[33203]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:26.707 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:26 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:26.707 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:26 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:26.707 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:26 smithi160 ceph-mon[33203]: from='osd.3 [v2:172.21.15.70:6826/1341239112,v1:172.21.15.70:6827/1341239112]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-21T09:51:26.707 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:26 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:26.707 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:26 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:26.884 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:51:27.424 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:27 smithi070 ceph-mon[41533]: from='osd.3 [v2:172.21.15.70:6826/1341239112,v1:172.21.15.70:6827/1341239112]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T09:51:27.425 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:27 smithi070 ceph-mon[41533]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T09:51:27.425 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:27 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:51:27.425 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:27 smithi070 ceph-mon[41533]: from='osd.3 [v2:172.21.15.70:6826/1341239112,v1:172.21.15.70:6827/1341239112]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-09-21T09:51:27.425 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:27 smithi070 ceph-mon[29589]: from='osd.3 [v2:172.21.15.70:6826/1341239112,v1:172.21.15.70:6827/1341239112]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T09:51:27.425 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:27 smithi070 ceph-mon[29589]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T09:51:27.425 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:27 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:51:27.426 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:27 smithi070 ceph-mon[29589]: from='osd.3 [v2:172.21.15.70:6826/1341239112,v1:172.21.15.70:6827/1341239112]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-09-21T09:51:27.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:27 smithi160 ceph-mon[33203]: from='osd.3 [v2:172.21.15.70:6826/1341239112,v1:172.21.15.70:6827/1341239112]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T09:51:27.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:27 smithi160 ceph-mon[33203]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T09:51:27.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:27 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:51:27.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:27 smithi160 ceph-mon[33203]: from='osd.3 [v2:172.21.15.70:6826/1341239112,v1:172.21.15.70:6827/1341239112]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-09-21T09:51:27.792 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 09:51:27 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3[64453]: 2024-09-21T09:51:27.422+0000 7f8182418640 -1 osd.3 0 waiting for initial osdmap 2024-09-21T09:51:27.793 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 09:51:27 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3[64453]: 2024-09-21T09:51:27.426+0000 7f817e241640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:51:28.642 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:28 smithi160 ceph-mon[33203]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:28.643 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:28 smithi160 ceph-mon[33203]: from='osd.3 [v2:172.21.15.70:6826/1341239112,v1:172.21.15.70:6827/1341239112]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-09-21T09:51:28.643 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:28 smithi160 ceph-mon[33203]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T09:51:28.643 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:28 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:51:28.643 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:28 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:51:28.737 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:28 smithi070 ceph-mon[29589]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:28.737 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:28 smithi070 ceph-mon[29589]: from='osd.3 [v2:172.21.15.70:6826/1341239112,v1:172.21.15.70:6827/1341239112]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-09-21T09:51:28.737 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:28 smithi070 ceph-mon[29589]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T09:51:28.737 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:28 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:51:28.738 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:28 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:51:28.738 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:28 smithi070 ceph-mon[41533]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:28.738 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:28 smithi070 ceph-mon[41533]: from='osd.3 [v2:172.21.15.70:6826/1341239112,v1:172.21.15.70:6827/1341239112]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-09-21T09:51:28.738 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:28 smithi070 ceph-mon[41533]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T09:51:28.738 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:28 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:51:28.738 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:28 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:51:29.139 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:51:29.158 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi160:vg_nvme/lv_4 2024-09-21T09:51:29.391 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:51:29.512 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:29 smithi160 ceph-mon[33203]: purged_snaps scrub starts 2024-09-21T09:51:29.512 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:29 smithi160 ceph-mon[33203]: purged_snaps scrub ok 2024-09-21T09:51:29.512 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:29 smithi160 ceph-mon[33203]: osd.3 [v2:172.21.15.70:6826/1341239112,v1:172.21.15.70:6827/1341239112] boot 2024-09-21T09:51:29.513 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:29 smithi160 ceph-mon[33203]: osdmap e25: 4 total, 4 up, 4 in 2024-09-21T09:51:29.513 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:51:29.513 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:29.513 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:29.513 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:29.513 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:29.513 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:29.513 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:29.513 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:29.513 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:29.513 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:29.513 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:29 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:29.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[41533]: purged_snaps scrub starts 2024-09-21T09:51:29.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[41533]: purged_snaps scrub ok 2024-09-21T09:51:29.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[41533]: osd.3 [v2:172.21.15.70:6826/1341239112,v1:172.21.15.70:6827/1341239112] boot 2024-09-21T09:51:29.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[41533]: osdmap e25: 4 total, 4 up, 4 in 2024-09-21T09:51:29.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:51:29.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:29.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:29.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:29.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:29.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:29.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:29.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:29.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:29.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:29.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:29.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[29589]: purged_snaps scrub starts 2024-09-21T09:51:29.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[29589]: purged_snaps scrub ok 2024-09-21T09:51:29.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[29589]: osd.3 [v2:172.21.15.70:6826/1341239112,v1:172.21.15.70:6827/1341239112] boot 2024-09-21T09:51:29.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[29589]: osdmap e25: 4 total, 4 up, 4 in 2024-09-21T09:51:29.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:51:29.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:29.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:29.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:29.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:29.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:29.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:29.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:29.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:29.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:29.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:29 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:30.543 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:30 smithi160 ceph-mon[33203]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:30.543 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:30 smithi160 ceph-mon[33203]: Detected new or changed devices on smithi070 2024-09-21T09:51:30.543 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:30 smithi160 ceph-mon[33203]: Adjusting osd_memory_target on smithi070 to 4016M 2024-09-21T09:51:30.543 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:30 smithi160 ceph-mon[33203]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T09:51:30.543 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:30 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:30.544 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:30 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:30.544 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:30 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:30.544 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:30 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:30.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:30 smithi070 ceph-mon[29589]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:30.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:30 smithi070 ceph-mon[29589]: Detected new or changed devices on smithi070 2024-09-21T09:51:30.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:30 smithi070 ceph-mon[29589]: Adjusting osd_memory_target on smithi070 to 4016M 2024-09-21T09:51:30.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:30 smithi070 ceph-mon[29589]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T09:51:30.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:30 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:30.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:30 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:30.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:30 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:30.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:30 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:30.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:30 smithi070 ceph-mon[41533]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-21T09:51:30.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:30 smithi070 ceph-mon[41533]: Detected new or changed devices on smithi070 2024-09-21T09:51:30.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:30 smithi070 ceph-mon[41533]: Adjusting osd_memory_target on smithi070 to 4016M 2024-09-21T09:51:30.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:30 smithi070 ceph-mon[41533]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T09:51:30.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:30 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:30.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:30 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:30.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:30 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:30.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:30 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:31.788 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:31 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:51:31.788 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:31 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:51:31.788 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:31 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:31 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:51:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:31 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:51:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:31 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:31.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:31 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:51:31.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:31 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:51:31.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:31 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:32.535 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:32 smithi160 ceph-mon[33203]: from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:32.535 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:32 smithi160 ceph-mon[33203]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:32.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:32 smithi070 ceph-mon[29589]: from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:32.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:32 smithi070 ceph-mon[29589]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:32.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:32 smithi070 ceph-mon[41533]: from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:32.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:32 smithi070 ceph-mon[41533]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:33 smithi160 ceph-mon[33203]: from='client.? 172.21.15.160:0/2070009595' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4b728b8-97a4-478f-8a11-ba590c0213e9"}]: dispatch 2024-09-21T09:51:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:33 smithi160 ceph-mon[33203]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4b728b8-97a4-478f-8a11-ba590c0213e9"}]: dispatch 2024-09-21T09:51:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:33 smithi160 ceph-mon[33203]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a4b728b8-97a4-478f-8a11-ba590c0213e9"}]': finished 2024-09-21T09:51:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:33 smithi160 ceph-mon[33203]: osdmap e27: 5 total, 4 up, 5 in 2024-09-21T09:51:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:33 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:51:33.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:33 smithi070 ceph-mon[29589]: from='client.? 172.21.15.160:0/2070009595' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4b728b8-97a4-478f-8a11-ba590c0213e9"}]: dispatch 2024-09-21T09:51:33.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:33 smithi070 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4b728b8-97a4-478f-8a11-ba590c0213e9"}]: dispatch 2024-09-21T09:51:33.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:33 smithi070 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a4b728b8-97a4-478f-8a11-ba590c0213e9"}]': finished 2024-09-21T09:51:33.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:33 smithi070 ceph-mon[29589]: osdmap e27: 5 total, 4 up, 5 in 2024-09-21T09:51:33.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:33 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:51:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:33 smithi070 ceph-mon[41533]: from='client.? 172.21.15.160:0/2070009595' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4b728b8-97a4-478f-8a11-ba590c0213e9"}]: dispatch 2024-09-21T09:51:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:33 smithi070 ceph-mon[41533]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4b728b8-97a4-478f-8a11-ba590c0213e9"}]: dispatch 2024-09-21T09:51:33.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:33 smithi070 ceph-mon[41533]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a4b728b8-97a4-478f-8a11-ba590c0213e9"}]': finished 2024-09-21T09:51:33.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:33 smithi070 ceph-mon[41533]: osdmap e27: 5 total, 4 up, 5 in 2024-09-21T09:51:33.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:33 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:51:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:34 smithi160 ceph-mon[33203]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:34 smithi160 ceph-mon[33203]: from='client.? 172.21.15.160:0/3223241313' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:51:34.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:34 smithi070 ceph-mon[29589]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:34.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:34 smithi070 ceph-mon[29589]: from='client.? 172.21.15.160:0/3223241313' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:51:34.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:34 smithi070 ceph-mon[41533]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:34.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:34 smithi070 ceph-mon[41533]: from='client.? 172.21.15.160:0/3223241313' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:51:36.751 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:36 smithi160 ceph-mon[33203]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:36.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:36 smithi070 ceph-mon[29589]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:36.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:36 smithi070 ceph-mon[41533]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:38.618 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:38 smithi160 ceph-mon[33203]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:38.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:38 smithi070 ceph-mon[29589]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:38.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:38 smithi070 ceph-mon[41533]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:39.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:39 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-21T09:51:39.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:39 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:39.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:39 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-21T09:51:39.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:39 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:39.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:39 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-21T09:51:39.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:39 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:40.796 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:40 smithi160 ceph-mon[33203]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:40.796 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:40 smithi160 ceph-mon[33203]: Deploying daemon osd.4 on smithi160 2024-09-21T09:51:40.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:40 smithi070 ceph-mon[29589]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:40.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:40 smithi070 ceph-mon[29589]: Deploying daemon osd.4 on smithi160 2024-09-21T09:51:40.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:40 smithi070 ceph-mon[41533]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:40.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:40 smithi070 ceph-mon[41533]: Deploying daemon osd.4 on smithi160 2024-09-21T09:51:42.748 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:42 smithi160 ceph-mon[33203]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:42.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:42 smithi070 ceph-mon[29589]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:42.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:42 smithi070 ceph-mon[41533]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:43.684 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:43 smithi160 ceph-mon[33203]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:43.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:43 smithi070 ceph-mon[29589]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:43.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:43 smithi070 ceph-mon[41533]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:44.790 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:44 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:44.790 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:44 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:44.790 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:44 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:44.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:44 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:44.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:44 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:44.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:44 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:44.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:44 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:44.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:44 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:44.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:44 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:45.786 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:45 smithi160 ceph-mon[33203]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:45.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:45 smithi070 ceph-mon[29589]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:45.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:45 smithi070 ceph-mon[41533]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:47.385 INFO:teuthology.orchestra.run.smithi160.stdout:Created osd(s) 4 on host 'smithi160' 2024-09-21T09:51:47.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:47 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:47.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:47 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:47.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:47 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:47.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:47 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:47.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:47 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:47.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:47 smithi070 ceph-mon[29589]: from='osd.4 [v2:172.21.15.160:6800/3634505869,v1:172.21.15.160:6801/3634505869]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T09:51:47.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:47 smithi070 ceph-mon[29589]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T09:51:47.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:47 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:47.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:47 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:47.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:47 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:47.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:47 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:47.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:47 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:47.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:47 smithi070 ceph-mon[41533]: from='osd.4 [v2:172.21.15.160:6800/3634505869,v1:172.21.15.160:6801/3634505869]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T09:51:47.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:47 smithi070 ceph-mon[41533]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T09:51:47.706 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:47 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:47.707 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:47 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:47.707 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:47 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:47.707 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:47 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:47.707 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:47 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:47.707 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:47 smithi160 ceph-mon[33203]: from='osd.4 [v2:172.21.15.160:6800/3634505869,v1:172.21.15.160:6801/3634505869]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T09:51:47.707 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:47 smithi160 ceph-mon[33203]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T09:51:48.002 DEBUG:teuthology.orchestra.run.smithi160:osd.4> sudo journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.4.service 2024-09-21T09:51:48.004 INFO:tasks.cephadm:Deploying osd.5 on smithi160 with /dev/vg_nvme/lv_3... 2024-09-21T09:51:48.004 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T09:51:48.576 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:48 smithi160 ceph-mon[33203]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:48.576 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:48 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:48.576 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:48 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:48.576 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:48 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:48.576 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:48 smithi160 ceph-mon[33203]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T09:51:48.576 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:48 smithi160 ceph-mon[33203]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T09:51:48.576 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:48 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:51:48.576 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:48 smithi160 ceph-mon[33203]: from='osd.4 [v2:172.21.15.160:6800/3634505869,v1:172.21.15.160:6801/3634505869]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-21T09:51:48.576 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:48 smithi160 ceph-mon[33203]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-21T09:51:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:48 smithi070 ceph-mon[29589]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:48 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:48 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:48 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:48 smithi070 ceph-mon[29589]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T09:51:48.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:48 smithi070 ceph-mon[29589]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T09:51:48.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:48 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:51:48.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:48 smithi070 ceph-mon[29589]: from='osd.4 [v2:172.21.15.160:6800/3634505869,v1:172.21.15.160:6801/3634505869]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-21T09:51:48.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:48 smithi070 ceph-mon[29589]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-21T09:51:48.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:48 smithi070 ceph-mon[41533]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:48.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:48 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:48.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:48 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:48.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:48 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:48.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:48 smithi070 ceph-mon[41533]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T09:51:48.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:48 smithi070 ceph-mon[41533]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T09:51:48.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:48 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:51:48.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:48 smithi070 ceph-mon[41533]: from='osd.4 [v2:172.21.15.160:6800/3634505869,v1:172.21.15.160:6801/3634505869]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-21T09:51:48.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:48 smithi070 ceph-mon[41533]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-21T09:51:48.841 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 21 09:51:48 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-4[38539]: 2024-09-21T09:51:48.728+0000 7f48b67ab640 -1 osd.4 0 waiting for initial osdmap 2024-09-21T09:51:48.841 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 21 09:51:48 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-4[38539]: 2024-09-21T09:51:48.732+0000 7f48b25d4640 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:51:49.216 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:51:50.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:49 smithi160 ceph-mon[33203]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-21T09:51:50.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:49 smithi160 ceph-mon[33203]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T09:51:50.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:49 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:51:50.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:49 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:51:50.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:49 smithi160 ceph-mon[33203]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:50.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:49 smithi160 ceph-mon[33203]: osd.4 [v2:172.21.15.160:6800/3634505869,v1:172.21.15.160:6801/3634505869] boot 2024-09-21T09:51:50.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:49 smithi160 ceph-mon[33203]: osdmap e30: 5 total, 5 up, 5 in 2024-09-21T09:51:50.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:49 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:51:50.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:49 smithi070 ceph-mon[41533]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-21T09:51:50.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:49 smithi070 ceph-mon[41533]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T09:51:50.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:49 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:51:50.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:49 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:51:50.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:49 smithi070 ceph-mon[41533]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:50.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:49 smithi070 ceph-mon[41533]: osd.4 [v2:172.21.15.160:6800/3634505869,v1:172.21.15.160:6801/3634505869] boot 2024-09-21T09:51:50.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:49 smithi070 ceph-mon[41533]: osdmap e30: 5 total, 5 up, 5 in 2024-09-21T09:51:50.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:49 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:51:50.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:49 smithi070 ceph-mon[29589]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-21T09:51:50.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:49 smithi070 ceph-mon[29589]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T09:51:50.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:49 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:51:50.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:49 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:51:50.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:49 smithi070 ceph-mon[29589]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-21T09:51:50.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:49 smithi070 ceph-mon[29589]: osd.4 [v2:172.21.15.160:6800/3634505869,v1:172.21.15.160:6801/3634505869] boot 2024-09-21T09:51:50.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:49 smithi070 ceph-mon[29589]: osdmap e30: 5 total, 5 up, 5 in 2024-09-21T09:51:50.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:49 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:51:51.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:51 smithi160 ceph-mon[33203]: purged_snaps scrub starts 2024-09-21T09:51:51.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:51 smithi160 ceph-mon[33203]: purged_snaps scrub ok 2024-09-21T09:51:51.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:51 smithi160 ceph-mon[33203]: osdmap e31: 5 total, 5 up, 5 in 2024-09-21T09:51:51.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:51 smithi070 ceph-mon[41533]: purged_snaps scrub starts 2024-09-21T09:51:51.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:51 smithi070 ceph-mon[41533]: purged_snaps scrub ok 2024-09-21T09:51:51.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:51 smithi070 ceph-mon[41533]: osdmap e31: 5 total, 5 up, 5 in 2024-09-21T09:51:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:51 smithi070 ceph-mon[29589]: purged_snaps scrub starts 2024-09-21T09:51:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:51 smithi070 ceph-mon[29589]: purged_snaps scrub ok 2024-09-21T09:51:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:51 smithi070 ceph-mon[29589]: osdmap e31: 5 total, 5 up, 5 in 2024-09-21T09:51:52.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:52 smithi160 ceph-mon[33203]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-21T09:51:52.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:52 smithi160 ceph-mon[33203]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T09:51:52.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:52 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:52.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:52 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:52.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:52 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:52.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:52 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:52.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:52 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:52.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:52 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:52.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:52 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:52.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:52 smithi070 ceph-mon[29589]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-21T09:51:52.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:52 smithi070 ceph-mon[29589]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T09:51:52.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:52 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:52.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:52 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:52.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:52 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:52.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:52 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:52.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:52 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:52.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:52 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:52.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:52 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:52.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:52 smithi070 ceph-mon[41533]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-21T09:51:52.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:52 smithi070 ceph-mon[41533]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T09:51:52.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:52 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:52.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:52 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:52.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:52 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:52.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:52 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:52.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:52 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:52.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:52 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:52.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:52 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:51:52.861 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:51:52.879 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi160:vg_nvme/lv_3 2024-09-21T09:51:53.072 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:51:53.519 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:53 smithi160 ceph-mon[33203]: Detected new or changed devices on smithi160 2024-09-21T09:51:53.519 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:53 smithi160 ceph-mon[33203]: Adjusting osd_memory_target on smithi160 to 17088M 2024-09-21T09:51:53.519 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:53 smithi160 ceph-mon[33203]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T09:51:53.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:53 smithi070 ceph-mon[29589]: Detected new or changed devices on smithi160 2024-09-21T09:51:53.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:53 smithi070 ceph-mon[29589]: Adjusting osd_memory_target on smithi160 to 17088M 2024-09-21T09:51:53.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:53 smithi070 ceph-mon[29589]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T09:51:53.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:53 smithi070 ceph-mon[41533]: Detected new or changed devices on smithi160 2024-09-21T09:51:53.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:53 smithi070 ceph-mon[41533]: Adjusting osd_memory_target on smithi160 to 17088M 2024-09-21T09:51:53.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:53 smithi070 ceph-mon[41533]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T09:51:54.483 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:54 smithi160 ceph-mon[33203]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-21T09:51:54.483 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:54 smithi160 ceph-mon[33203]: osdmap e34: 5 total, 5 up, 5 in 2024-09-21T09:51:54.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:54 smithi070 ceph-mon[29589]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-21T09:51:54.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:54 smithi070 ceph-mon[29589]: osdmap e34: 5 total, 5 up, 5 in 2024-09-21T09:51:54.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:54 smithi070 ceph-mon[41533]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-21T09:51:54.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:54 smithi070 ceph-mon[41533]: osdmap e34: 5 total, 5 up, 5 in 2024-09-21T09:51:55.478 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:55 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:51:55.478 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:55 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:51:55.479 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:55 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:55.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:55 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:51:55.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:55 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:51:55.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:55 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:55.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:55 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:51:55.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:55 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:51:55.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:55 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:56.507 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:56 smithi160 ceph-mon[33203]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:56.507 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:56 smithi160 ceph-mon[33203]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail 2024-09-21T09:51:56.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:56 smithi070 ceph-mon[29589]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:56.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:56 smithi070 ceph-mon[29589]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail 2024-09-21T09:51:56.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:56 smithi070 ceph-mon[41533]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:56.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:56 smithi070 ceph-mon[41533]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail 2024-09-21T09:51:57.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:57 smithi160 ceph-mon[33203]: from='client.? 172.21.15.160:0/3697563655' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56915347-9254-4e94-b202-1f16d09e7946"}]: dispatch 2024-09-21T09:51:57.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:57 smithi160 ceph-mon[33203]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56915347-9254-4e94-b202-1f16d09e7946"}]: dispatch 2024-09-21T09:51:57.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:57 smithi160 ceph-mon[33203]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56915347-9254-4e94-b202-1f16d09e7946"}]': finished 2024-09-21T09:51:57.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:57 smithi160 ceph-mon[33203]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T09:51:57.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:57 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:51:57.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:57 smithi070 ceph-mon[29589]: from='client.? 172.21.15.160:0/3697563655' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56915347-9254-4e94-b202-1f16d09e7946"}]: dispatch 2024-09-21T09:51:57.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:57 smithi070 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56915347-9254-4e94-b202-1f16d09e7946"}]: dispatch 2024-09-21T09:51:57.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:57 smithi070 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56915347-9254-4e94-b202-1f16d09e7946"}]': finished 2024-09-21T09:51:57.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:57 smithi070 ceph-mon[29589]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T09:51:57.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:57 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:51:57.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:57 smithi070 ceph-mon[41533]: from='client.? 172.21.15.160:0/3697563655' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56915347-9254-4e94-b202-1f16d09e7946"}]: dispatch 2024-09-21T09:51:57.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:57 smithi070 ceph-mon[41533]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56915347-9254-4e94-b202-1f16d09e7946"}]: dispatch 2024-09-21T09:51:57.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:57 smithi070 ceph-mon[41533]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56915347-9254-4e94-b202-1f16d09e7946"}]': finished 2024-09-21T09:51:57.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:57 smithi070 ceph-mon[41533]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T09:51:57.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:57 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:51:58.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:58 smithi160 ceph-mon[33203]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T09:51:58.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:51:58 smithi160 ceph-mon[33203]: from='client.? 172.21.15.160:0/1876560695' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:51:58.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:58 smithi070 ceph-mon[29589]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T09:51:58.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:51:58 smithi070 ceph-mon[29589]: from='client.? 172.21.15.160:0/1876560695' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:51:58.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:58 smithi070 ceph-mon[41533]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T09:51:58.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:51:58 smithi070 ceph-mon[41533]: from='client.? 172.21.15.160:0/1876560695' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:52:00.581 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:00 smithi160 ceph-mon[33203]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-21T09:52:00.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:00 smithi070 ceph-mon[29589]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-21T09:52:00.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:00 smithi070 ceph-mon[41533]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-21T09:52:02.518 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:02 smithi160 ceph-mon[33203]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T09:52:02.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:02 smithi070 ceph-mon[41533]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T09:52:02.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:02 smithi070 ceph-mon[29589]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T09:52:03.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:03 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-21T09:52:03.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:03 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:03.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:03 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-21T09:52:03.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:03 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:03.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:03 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-21T09:52:03.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:03 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:04.483 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:04 smithi160 ceph-mon[33203]: Deploying daemon osd.5 on smithi160 2024-09-21T09:52:04.483 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:04 smithi160 ceph-mon[33203]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:52:04.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:04 smithi070 ceph-mon[29589]: Deploying daemon osd.5 on smithi160 2024-09-21T09:52:04.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:04 smithi070 ceph-mon[29589]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:52:04.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:04 smithi070 ceph-mon[41533]: Deploying daemon osd.5 on smithi160 2024-09-21T09:52:04.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:04 smithi070 ceph-mon[41533]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:52:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:06 smithi160 ceph-mon[33203]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:52:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:06 smithi070 ceph-mon[29589]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:52:06.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:06 smithi070 ceph-mon[41533]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:52:08.555 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:08 smithi160 ceph-mon[33203]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-21T09:52:08.555 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:08 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:08.556 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:08 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:08.556 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:08 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:08 smithi070 ceph-mon[29589]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-21T09:52:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:08 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:08 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:08 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:08.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:08 smithi070 ceph-mon[41533]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-21T09:52:08.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:08 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:08.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:08 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:08.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:08 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:10.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:10 smithi160 ceph-mon[33203]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-21T09:52:10.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:10 smithi070 ceph-mon[29589]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-21T09:52:10.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:10 smithi070 ceph-mon[41533]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-21T09:52:11.264 INFO:teuthology.orchestra.run.smithi160.stdout:Created osd(s) 5 on host 'smithi160' 2024-09-21T09:52:11.836 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:11 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:11.836 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:11 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:11.836 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:11 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:11.836 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:11 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:11.837 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:11 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:11.837 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:11 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:11.837 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:11 smithi160 ceph-mon[33203]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-21T09:52:11.837 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:11 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:11.837 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:11 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:11.837 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:11 smithi160 ceph-mon[33203]: from='osd.5 [v2:172.21.15.160:6808/2006621188,v1:172.21.15.160:6809/2006621188]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T09:52:11.837 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:11 smithi160 ceph-mon[33203]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T09:52:11.855 DEBUG:teuthology.orchestra.run.smithi160:osd.5> sudo journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.5.service 2024-09-21T09:52:11.857 INFO:tasks.cephadm:Deploying osd.6 on smithi160 with /dev/vg_nvme/lv_2... 2024-09-21T09:52:11.857 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T09:52:11.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:11 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:11.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:11 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:11.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:11 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:11.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:11 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:11.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:11 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:11.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:11 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:11.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:11 smithi070 ceph-mon[29589]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-21T09:52:11.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:11 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:11.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:11 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:11.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:11 smithi070 ceph-mon[29589]: from='osd.5 [v2:172.21.15.160:6808/2006621188,v1:172.21.15.160:6809/2006621188]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T09:52:11.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:11 smithi070 ceph-mon[29589]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T09:52:11.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:11 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:11.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:11 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:11.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:11 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:11.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:11 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:11.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:11 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:11.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:11 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:11.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:11 smithi070 ceph-mon[41533]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-21T09:52:11.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:11 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:11.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:11 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:11.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:11 smithi070 ceph-mon[41533]: from='osd.5 [v2:172.21.15.160:6808/2006621188,v1:172.21.15.160:6809/2006621188]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T09:52:11.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:11 smithi070 ceph-mon[41533]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T09:52:12.595 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:52:13.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:13 smithi160 ceph-mon[33203]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T09:52:13.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:13 smithi160 ceph-mon[33203]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T09:52:13.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:13 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:52:13.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:13 smithi160 ceph-mon[33203]: from='osd.5 [v2:172.21.15.160:6808/2006621188,v1:172.21.15.160:6809/2006621188]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-21T09:52:13.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:13 smithi160 ceph-mon[33203]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-21T09:52:13.593 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 09:52:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[44885]: 2024-09-21T09:52:13.265+0000 7f1b87b11640 -1 osd.5 0 waiting for initial osdmap 2024-09-21T09:52:13.593 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 09:52:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[44885]: 2024-09-21T09:52:13.273+0000 7f1b8393a640 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:52:13.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:13 smithi070 ceph-mon[29589]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T09:52:13.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:13 smithi070 ceph-mon[29589]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T09:52:13.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:13 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:52:13.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:13 smithi070 ceph-mon[29589]: from='osd.5 [v2:172.21.15.160:6808/2006621188,v1:172.21.15.160:6809/2006621188]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-21T09:52:13.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:13 smithi070 ceph-mon[29589]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-21T09:52:13.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:13 smithi070 ceph-mon[41533]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T09:52:13.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:13 smithi070 ceph-mon[41533]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T09:52:13.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:13 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:52:13.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:13 smithi070 ceph-mon[41533]: from='osd.5 [v2:172.21.15.160:6808/2006621188,v1:172.21.15.160:6809/2006621188]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-21T09:52:13.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:13 smithi070 ceph-mon[41533]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-21T09:52:14.542 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:14 smithi160 ceph-mon[33203]: purged_snaps scrub starts 2024-09-21T09:52:14.542 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:14 smithi160 ceph-mon[33203]: purged_snaps scrub ok 2024-09-21T09:52:14.542 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:14 smithi160 ceph-mon[33203]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-21T09:52:14.543 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:14 smithi160 ceph-mon[33203]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-21T09:52:14.543 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:14 smithi160 ceph-mon[33203]: osdmap e37: 6 total, 5 up, 6 in 2024-09-21T09:52:14.543 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:14 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:52:14.543 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:14 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:52:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:14 smithi070 ceph-mon[29589]: purged_snaps scrub starts 2024-09-21T09:52:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:14 smithi070 ceph-mon[29589]: purged_snaps scrub ok 2024-09-21T09:52:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:14 smithi070 ceph-mon[29589]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-21T09:52:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:14 smithi070 ceph-mon[29589]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-21T09:52:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:14 smithi070 ceph-mon[29589]: osdmap e37: 6 total, 5 up, 6 in 2024-09-21T09:52:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:14 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:52:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:14 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:52:14.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:14 smithi070 ceph-mon[41533]: purged_snaps scrub starts 2024-09-21T09:52:14.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:14 smithi070 ceph-mon[41533]: purged_snaps scrub ok 2024-09-21T09:52:14.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:14 smithi070 ceph-mon[41533]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-21T09:52:14.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:14 smithi070 ceph-mon[41533]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-21T09:52:14.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:14 smithi070 ceph-mon[41533]: osdmap e37: 6 total, 5 up, 6 in 2024-09-21T09:52:14.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:14 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:52:14.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:14 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:52:15.487 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:15 smithi160 ceph-mon[33203]: osd.5 [v2:172.21.15.160:6808/2006621188,v1:172.21.15.160:6809/2006621188] boot 2024-09-21T09:52:15.488 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:15 smithi160 ceph-mon[33203]: osdmap e38: 6 total, 6 up, 6 in 2024-09-21T09:52:15.488 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:15 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:52:15.488 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:15 smithi160 ceph-mon[33203]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T09:52:15.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:15 smithi070 ceph-mon[29589]: osd.5 [v2:172.21.15.160:6808/2006621188,v1:172.21.15.160:6809/2006621188] boot 2024-09-21T09:52:15.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:15 smithi070 ceph-mon[29589]: osdmap e38: 6 total, 6 up, 6 in 2024-09-21T09:52:15.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:15 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:52:15.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:15 smithi070 ceph-mon[29589]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T09:52:15.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:15 smithi070 ceph-mon[41533]: osd.5 [v2:172.21.15.160:6808/2006621188,v1:172.21.15.160:6809/2006621188] boot 2024-09-21T09:52:15.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:15 smithi070 ceph-mon[41533]: osdmap e38: 6 total, 6 up, 6 in 2024-09-21T09:52:15.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:15 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:52:15.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:15 smithi070 ceph-mon[41533]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T09:52:16.375 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:16 smithi160 ceph-mon[33203]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:16.375 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:16 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:16.375 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:16 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:16.375 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:16 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:52:16.375 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:16 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:52:16.375 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:16 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:16.376 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:16 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:16.376 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:16 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:16.376 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:16 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:16.376 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:16 smithi160 ceph-mon[33203]: osdmap e40: 6 total, 6 up, 6 in 2024-09-21T09:52:16.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:16 smithi070 ceph-mon[29589]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:16.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:16 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:16.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:16 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:16.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:16 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:52:16.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:16 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:52:16.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:16 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:16.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:16 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:16.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:16 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:16.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:16 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:16.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:16 smithi070 ceph-mon[29589]: osdmap e40: 6 total, 6 up, 6 in 2024-09-21T09:52:16.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:16 smithi070 ceph-mon[41533]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:16.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:16 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:16.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:16 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:16.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:16 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:52:16.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:16 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:52:16.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:16 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:16.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:16 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:16.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:16 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:16.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:16 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:16.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:16 smithi070 ceph-mon[41533]: osdmap e40: 6 total, 6 up, 6 in 2024-09-21T09:52:17.148 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:52:17.167 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi160:vg_nvme/lv_2 2024-09-21T09:52:17.366 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:52:17.485 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:17 smithi160 ceph-mon[33203]: Detected new or changed devices on smithi160 2024-09-21T09:52:17.486 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:17 smithi160 ceph-mon[33203]: Adjusting osd_memory_target on smithi160 to 8544M 2024-09-21T09:52:17.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:17 smithi070 ceph-mon[29589]: Detected new or changed devices on smithi160 2024-09-21T09:52:17.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:17 smithi070 ceph-mon[29589]: Adjusting osd_memory_target on smithi160 to 8544M 2024-09-21T09:52:17.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:17 smithi070 ceph-mon[41533]: Detected new or changed devices on smithi160 2024-09-21T09:52:17.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:17 smithi070 ceph-mon[41533]: Adjusting osd_memory_target on smithi160 to 8544M 2024-09-21T09:52:18.509 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:18 smithi160 ceph-mon[33203]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:18 smithi070 ceph-mon[29589]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:18.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:18 smithi070 ceph-mon[41533]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:19.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:19 smithi160 ceph-mon[33203]: osdmap e41: 6 total, 6 up, 6 in 2024-09-21T09:52:19.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:19 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:52:19.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:19 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:52:19.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:19 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:19.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:19 smithi070 ceph-mon[41533]: osdmap e41: 6 total, 6 up, 6 in 2024-09-21T09:52:19.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:19 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:52:19.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:19 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:52:19.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:19 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:19.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:19 smithi070 ceph-mon[29589]: osdmap e41: 6 total, 6 up, 6 in 2024-09-21T09:52:19.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:19 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:52:19.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:19 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:52:19.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:19 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:20.485 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:20 smithi160 ceph-mon[33203]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:20.485 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:20 smithi160 ceph-mon[33203]: pgmap v133: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:20.485 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:20 smithi160 ceph-mon[33203]: osdmap e42: 6 total, 6 up, 6 in 2024-09-21T09:52:20.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:20 smithi070 ceph-mon[41533]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:20.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:20 smithi070 ceph-mon[41533]: pgmap v133: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:20.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:20 smithi070 ceph-mon[41533]: osdmap e42: 6 total, 6 up, 6 in 2024-09-21T09:52:20.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:20 smithi070 ceph-mon[29589]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:20.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:20 smithi070 ceph-mon[29589]: pgmap v133: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:20.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:20 smithi070 ceph-mon[29589]: osdmap e42: 6 total, 6 up, 6 in 2024-09-21T09:52:22.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:22 smithi160 ceph-mon[33203]: pgmap v135: 1 pgs: 1 peering; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:22.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:22 smithi160 ceph-mon[33203]: from='client.? 172.21.15.160:0/190277859' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79e6de8c-b78b-40e2-9229-0275a873d40e"}]: dispatch 2024-09-21T09:52:22.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:22 smithi160 ceph-mon[33203]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79e6de8c-b78b-40e2-9229-0275a873d40e"}]: dispatch 2024-09-21T09:52:22.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:22 smithi160 ceph-mon[33203]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "79e6de8c-b78b-40e2-9229-0275a873d40e"}]': finished 2024-09-21T09:52:22.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:22 smithi160 ceph-mon[33203]: osdmap e43: 7 total, 6 up, 7 in 2024-09-21T09:52:22.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:22 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:52:22.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:22 smithi160 ceph-mon[33203]: from='client.? 172.21.15.160:0/1127187839' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:52:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:22 smithi070 ceph-mon[29589]: pgmap v135: 1 pgs: 1 peering; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:22 smithi070 ceph-mon[29589]: from='client.? 172.21.15.160:0/190277859' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79e6de8c-b78b-40e2-9229-0275a873d40e"}]: dispatch 2024-09-21T09:52:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:22 smithi070 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79e6de8c-b78b-40e2-9229-0275a873d40e"}]: dispatch 2024-09-21T09:52:22.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:22 smithi070 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "79e6de8c-b78b-40e2-9229-0275a873d40e"}]': finished 2024-09-21T09:52:22.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:22 smithi070 ceph-mon[29589]: osdmap e43: 7 total, 6 up, 7 in 2024-09-21T09:52:22.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:22 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:52:22.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:22 smithi070 ceph-mon[29589]: from='client.? 172.21.15.160:0/1127187839' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:52:22.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:22 smithi070 ceph-mon[41533]: pgmap v135: 1 pgs: 1 peering; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:22.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:22 smithi070 ceph-mon[41533]: from='client.? 172.21.15.160:0/190277859' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79e6de8c-b78b-40e2-9229-0275a873d40e"}]: dispatch 2024-09-21T09:52:22.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:22 smithi070 ceph-mon[41533]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79e6de8c-b78b-40e2-9229-0275a873d40e"}]: dispatch 2024-09-21T09:52:22.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:22 smithi070 ceph-mon[41533]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "79e6de8c-b78b-40e2-9229-0275a873d40e"}]': finished 2024-09-21T09:52:22.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:22 smithi070 ceph-mon[41533]: osdmap e43: 7 total, 6 up, 7 in 2024-09-21T09:52:22.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:22 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:52:22.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:22 smithi070 ceph-mon[41533]: from='client.? 172.21.15.160:0/1127187839' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:52:24.543 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:24 smithi160 ceph-mon[33203]: pgmap v137: 1 pgs: 1 peering; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:24.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:24 smithi070 ceph-mon[29589]: pgmap v137: 1 pgs: 1 peering; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:24.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:24 smithi070 ceph-mon[41533]: pgmap v137: 1 pgs: 1 peering; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:26.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:26 smithi160 ceph-mon[33203]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:26 smithi070 ceph-mon[29589]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:26.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:26 smithi070 ceph-mon[41533]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:28.486 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:28 smithi160 ceph-mon[33203]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:28.486 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:28 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-21T09:52:28.486 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:28 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:28.486 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:28 smithi160 ceph-mon[33203]: Deploying daemon osd.6 on smithi160 2024-09-21T09:52:28.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:28 smithi070 ceph-mon[29589]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:28.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:28 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-21T09:52:28.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:28 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:28.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:28 smithi070 ceph-mon[29589]: Deploying daemon osd.6 on smithi160 2024-09-21T09:52:28.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:28 smithi070 ceph-mon[41533]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:28.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:28 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-21T09:52:28.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:28 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:28.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:28 smithi070 ceph-mon[41533]: Deploying daemon osd.6 on smithi160 2024-09-21T09:52:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:30 smithi160 ceph-mon[33203]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:30.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:30 smithi070 ceph-mon[29589]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:30.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:30 smithi070 ceph-mon[41533]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:32.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:32 smithi160 ceph-mon[33203]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:32.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:32 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:32.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:32 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:32.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:32 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:32.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:32 smithi070 ceph-mon[29589]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:32.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:32 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:32.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:32 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:32.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:32 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:32.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:32 smithi070 ceph-mon[41533]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:32.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:32 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:32.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:32 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:32.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:32 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:34.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:34 smithi160 ceph-mon[33203]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:34.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:34 smithi070 ceph-mon[29589]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:34.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:34 smithi070 ceph-mon[41533]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:35.252 INFO:teuthology.orchestra.run.smithi160.stdout:Created osd(s) 6 on host 'smithi160' 2024-09-21T09:52:35.870 DEBUG:teuthology.orchestra.run.smithi160:osd.6> sudo journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.6.service 2024-09-21T09:52:35.873 INFO:tasks.cephadm:Deploying osd.7 on smithi160 with /dev/vg_nvme/lv_1... 2024-09-21T09:52:35.873 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T09:52:36.113 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:52:36.503 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:36 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:36.503 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:36 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:36.503 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:36 smithi160 ceph-mon[33203]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:36.503 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:36 smithi160 ceph-mon[33203]: from='osd.6 [v2:172.21.15.160:6816/1949075400,v1:172.21.15.160:6817/1949075400]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T09:52:36.503 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:36 smithi160 ceph-mon[33203]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T09:52:36.503 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:36 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:36.503 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:36 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:36 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:36 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:36 smithi070 ceph-mon[29589]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:36 smithi070 ceph-mon[29589]: from='osd.6 [v2:172.21.15.160:6816/1949075400,v1:172.21.15.160:6817/1949075400]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T09:52:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:36 smithi070 ceph-mon[29589]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T09:52:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:36 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:36 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:36 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:36 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:36 smithi070 ceph-mon[41533]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:36.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:36 smithi070 ceph-mon[41533]: from='osd.6 [v2:172.21.15.160:6816/1949075400,v1:172.21.15.160:6817/1949075400]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T09:52:36.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:36 smithi070 ceph-mon[41533]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T09:52:36.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:36 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:36.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:36 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:37.459 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:37 smithi160 ceph-mon[33203]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T09:52:37.459 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:37 smithi160 ceph-mon[33203]: osdmap e44: 7 total, 6 up, 7 in 2024-09-21T09:52:37.459 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:37 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:52:37.459 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:37 smithi160 ceph-mon[33203]: from='osd.6 [v2:172.21.15.160:6816/1949075400,v1:172.21.15.160:6817/1949075400]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-21T09:52:37.459 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:37 smithi160 ceph-mon[33203]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-21T09:52:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:37 smithi070 ceph-mon[29589]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T09:52:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:37 smithi070 ceph-mon[29589]: osdmap e44: 7 total, 6 up, 7 in 2024-09-21T09:52:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:37 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:52:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:37 smithi070 ceph-mon[29589]: from='osd.6 [v2:172.21.15.160:6816/1949075400,v1:172.21.15.160:6817/1949075400]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-21T09:52:37.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:37 smithi070 ceph-mon[29589]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-21T09:52:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:37 smithi070 ceph-mon[41533]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T09:52:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:37 smithi070 ceph-mon[41533]: osdmap e44: 7 total, 6 up, 7 in 2024-09-21T09:52:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:37 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:52:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:37 smithi070 ceph-mon[41533]: from='osd.6 [v2:172.21.15.160:6816/1949075400,v1:172.21.15.160:6817/1949075400]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-21T09:52:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:37 smithi070 ceph-mon[41533]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-21T09:52:37.714 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 09:52:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[51187]: 2024-09-21T09:52:37.690+0000 7f81d68ca640 -1 osd.6 0 waiting for initial osdmap 2024-09-21T09:52:37.714 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 09:52:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[51187]: 2024-09-21T09:52:37.699+0000 7f81ce6eb640 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:52:38.514 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:38 smithi160 ceph-mon[33203]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-21T09:52:38.514 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:38 smithi160 ceph-mon[33203]: osdmap e45: 7 total, 6 up, 7 in 2024-09-21T09:52:38.514 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:38 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:52:38.514 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:38 smithi160 ceph-mon[33203]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:38.514 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:38 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:52:38.515 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:38 smithi160 ceph-mon[33203]: from='osd.6 ' entity='osd.6' 2024-09-21T09:52:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:38 smithi070 ceph-mon[29589]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-21T09:52:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:38 smithi070 ceph-mon[29589]: osdmap e45: 7 total, 6 up, 7 in 2024-09-21T09:52:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:38 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:52:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:38 smithi070 ceph-mon[29589]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:38.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:38 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:52:38.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:38 smithi070 ceph-mon[29589]: from='osd.6 ' entity='osd.6' 2024-09-21T09:52:38.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:38 smithi070 ceph-mon[41533]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-21T09:52:38.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:38 smithi070 ceph-mon[41533]: osdmap e45: 7 total, 6 up, 7 in 2024-09-21T09:52:38.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:38 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:52:38.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:38 smithi070 ceph-mon[41533]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-21T09:52:38.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:38 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:52:38.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:38 smithi070 ceph-mon[41533]: from='osd.6 ' entity='osd.6' 2024-09-21T09:52:39.273 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T09:52:39.291 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi160:vg_nvme/lv_1 2024-09-21T09:52:39.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:39 smithi160 ceph-mon[33203]: purged_snaps scrub starts 2024-09-21T09:52:39.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:39 smithi160 ceph-mon[33203]: purged_snaps scrub ok 2024-09-21T09:52:39.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:39 smithi160 ceph-mon[33203]: osd.6 [v2:172.21.15.160:6816/1949075400,v1:172.21.15.160:6817/1949075400] boot 2024-09-21T09:52:39.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:39 smithi160 ceph-mon[33203]: osdmap e46: 7 total, 7 up, 7 in 2024-09-21T09:52:39.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:39 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:52:39.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:39 smithi160 ceph-mon[33203]: osdmap e47: 7 total, 7 up, 7 in 2024-09-21T09:52:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:39 smithi070 ceph-mon[29589]: purged_snaps scrub starts 2024-09-21T09:52:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:39 smithi070 ceph-mon[29589]: purged_snaps scrub ok 2024-09-21T09:52:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:39 smithi070 ceph-mon[29589]: osd.6 [v2:172.21.15.160:6816/1949075400,v1:172.21.15.160:6817/1949075400] boot 2024-09-21T09:52:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:39 smithi070 ceph-mon[29589]: osdmap e46: 7 total, 7 up, 7 in 2024-09-21T09:52:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:39 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:52:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:39 smithi070 ceph-mon[29589]: osdmap e47: 7 total, 7 up, 7 in 2024-09-21T09:52:39.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:39 smithi070 ceph-mon[41533]: purged_snaps scrub starts 2024-09-21T09:52:39.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:39 smithi070 ceph-mon[41533]: purged_snaps scrub ok 2024-09-21T09:52:39.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:39 smithi070 ceph-mon[41533]: osd.6 [v2:172.21.15.160:6816/1949075400,v1:172.21.15.160:6817/1949075400] boot 2024-09-21T09:52:39.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:39 smithi070 ceph-mon[41533]: osdmap e46: 7 total, 7 up, 7 in 2024-09-21T09:52:39.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:39 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:52:39.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:39 smithi070 ceph-mon[41533]: osdmap e47: 7 total, 7 up, 7 in 2024-09-21T09:52:39.847 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:52:40.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:40 smithi160 ceph-mon[33203]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 625 GiB / 626 GiB avail 2024-09-21T09:52:40.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:40 smithi160 ceph-mon[33203]: osdmap e48: 7 total, 7 up, 7 in 2024-09-21T09:52:40.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:40 smithi070 ceph-mon[41533]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 625 GiB / 626 GiB avail 2024-09-21T09:52:40.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:40 smithi070 ceph-mon[41533]: osdmap e48: 7 total, 7 up, 7 in 2024-09-21T09:52:40.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:40 smithi070 ceph-mon[29589]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 625 GiB / 626 GiB avail 2024-09-21T09:52:40.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:40 smithi070 ceph-mon[29589]: osdmap e48: 7 total, 7 up, 7 in 2024-09-21T09:52:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:42 smithi160 ceph-mon[33203]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 625 GiB / 626 GiB avail 2024-09-21T09:52:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:42 smithi160 ceph-mon[33203]: osdmap e49: 7 total, 7 up, 7 in 2024-09-21T09:52:42.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:42 smithi160 ceph-mon[33203]: Detected new or changed devices on smithi160 2024-09-21T09:52:42.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:42.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:42.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:52:42.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:52:42.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:52:42.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:42 smithi160 ceph-mon[33203]: Adjusting osd_memory_target on smithi160 to 5696M 2024-09-21T09:52:42.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:42.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:42.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:42.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:42.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:42.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:42.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:42.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:42.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:52:42.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:52:42.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:42 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:42.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[29589]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 625 GiB / 626 GiB avail 2024-09-21T09:52:42.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[29589]: osdmap e49: 7 total, 7 up, 7 in 2024-09-21T09:52:42.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[29589]: Detected new or changed devices on smithi160 2024-09-21T09:52:42.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:42.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:42.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:52:42.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:52:42.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:52:42.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[29589]: Adjusting osd_memory_target on smithi160 to 5696M 2024-09-21T09:52:42.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:42.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:42.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:42.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:42.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:42.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:42.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:42.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:42.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:52:42.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:52:42.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:42.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[41533]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 625 GiB / 626 GiB avail 2024-09-21T09:52:42.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[41533]: osdmap e49: 7 total, 7 up, 7 in 2024-09-21T09:52:42.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[41533]: Detected new or changed devices on smithi160 2024-09-21T09:52:42.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:42.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:42.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:52:42.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:52:42.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:52:42.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[41533]: Adjusting osd_memory_target on smithi160 to 5696M 2024-09-21T09:52:42.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:42.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:42.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:42.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:42.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:42.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:42.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:42.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:42.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:52:42.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:52:42.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:42 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:43.382 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:43 smithi160 ceph-mon[33203]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:43.383 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:43 smithi160 ceph-mon[33203]: osdmap e50: 7 total, 7 up, 7 in 2024-09-21T09:52:43.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:43 smithi070 ceph-mon[29589]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:43.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:43 smithi070 ceph-mon[29589]: osdmap e50: 7 total, 7 up, 7 in 2024-09-21T09:52:43.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:43 smithi070 ceph-mon[41533]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:43.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:43 smithi070 ceph-mon[41533]: osdmap e50: 7 total, 7 up, 7 in 2024-09-21T09:52:44.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:44 smithi160 ceph-mon[33203]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 625 GiB / 626 GiB avail 2024-09-21T09:52:44.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:44 smithi070 ceph-mon[29589]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 625 GiB / 626 GiB avail 2024-09-21T09:52:44.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:44 smithi070 ceph-mon[41533]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 625 GiB / 626 GiB avail 2024-09-21T09:52:45.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:45 smithi160 ceph-mon[33203]: from='client.? 172.21.15.160:0/3797119716' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72245c8a-2fdb-4445-b378-16569f337fcd"}]: dispatch 2024-09-21T09:52:45.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:45 smithi160 ceph-mon[33203]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72245c8a-2fdb-4445-b378-16569f337fcd"}]: dispatch 2024-09-21T09:52:45.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:45 smithi160 ceph-mon[33203]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72245c8a-2fdb-4445-b378-16569f337fcd"}]': finished 2024-09-21T09:52:45.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:45 smithi160 ceph-mon[33203]: osdmap e51: 8 total, 7 up, 8 in 2024-09-21T09:52:45.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:45 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:52:45.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:45 smithi160 ceph-mon[33203]: from='client.? 172.21.15.160:0/3930322459' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:52:45.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:45 smithi070 ceph-mon[29589]: from='client.? 172.21.15.160:0/3797119716' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72245c8a-2fdb-4445-b378-16569f337fcd"}]: dispatch 2024-09-21T09:52:45.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:45 smithi070 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72245c8a-2fdb-4445-b378-16569f337fcd"}]: dispatch 2024-09-21T09:52:45.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:45 smithi070 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72245c8a-2fdb-4445-b378-16569f337fcd"}]': finished 2024-09-21T09:52:45.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:45 smithi070 ceph-mon[29589]: osdmap e51: 8 total, 7 up, 8 in 2024-09-21T09:52:45.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:45 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:52:45.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:45 smithi070 ceph-mon[29589]: from='client.? 172.21.15.160:0/3930322459' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:52:45.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:45 smithi070 ceph-mon[41533]: from='client.? 172.21.15.160:0/3797119716' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72245c8a-2fdb-4445-b378-16569f337fcd"}]: dispatch 2024-09-21T09:52:45.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:45 smithi070 ceph-mon[41533]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72245c8a-2fdb-4445-b378-16569f337fcd"}]: dispatch 2024-09-21T09:52:45.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:45 smithi070 ceph-mon[41533]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72245c8a-2fdb-4445-b378-16569f337fcd"}]': finished 2024-09-21T09:52:45.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:45 smithi070 ceph-mon[41533]: osdmap e51: 8 total, 7 up, 8 in 2024-09-21T09:52:45.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:45 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:52:45.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:45 smithi070 ceph-mon[41533]: from='client.? 172.21.15.160:0/3930322459' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:52:46.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:46 smithi070 ceph-mon[29589]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-09-21T09:52:46.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:46 smithi070 ceph-mon[41533]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-09-21T09:52:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:46 smithi160 ceph-mon[33203]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-09-21T09:52:48.511 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:48 smithi160 ceph-mon[33203]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T09:52:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:48 smithi070 ceph-mon[29589]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T09:52:48.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:48 smithi070 ceph-mon[41533]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T09:52:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:50 smithi070 ceph-mon[29589]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T09:52:50.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:50 smithi070 ceph-mon[41533]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T09:52:50.671 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:50 smithi160 ceph-mon[33203]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T09:52:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:51 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-21T09:52:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:51 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:51 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-21T09:52:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:51 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:51.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:51 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-21T09:52:51.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:51 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:52.455 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:52 smithi160 ceph-mon[33203]: Deploying daemon osd.7 on smithi160 2024-09-21T09:52:52.455 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:52 smithi160 ceph-mon[33203]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-09-21T09:52:52.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:52 smithi070 ceph-mon[29589]: Deploying daemon osd.7 on smithi160 2024-09-21T09:52:52.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:52 smithi070 ceph-mon[29589]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-09-21T09:52:52.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:52 smithi070 ceph-mon[41533]: Deploying daemon osd.7 on smithi160 2024-09-21T09:52:52.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:52 smithi070 ceph-mon[41533]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-09-21T09:52:54.448 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:54 smithi160 ceph-mon[33203]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:52:54.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:54 smithi070 ceph-mon[29589]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:52:54.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:54 smithi070 ceph-mon[41533]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:52:56.447 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:56 smithi160 ceph-mon[33203]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-21T09:52:56.447 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:56 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:56.448 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:56 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:56.448 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:56 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:56.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:56 smithi070 ceph-mon[29589]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-21T09:52:56.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:56 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:56.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:56 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:56.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:56 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:56.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:56 smithi070 ceph-mon[41533]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-21T09:52:56.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:56 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:56.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:56 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:56.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:56 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:58.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:58 smithi070 ceph-mon[29589]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-21T09:52:58.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:58 smithi070 ceph-mon[41533]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-21T09:52:58.644 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:58 smithi160 ceph-mon[33203]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-21T09:52:58.677 INFO:teuthology.orchestra.run.smithi160.stdout:Created osd(s) 7 on host 'smithi160' 2024-09-21T09:52:59.235 DEBUG:teuthology.orchestra.run.smithi160:osd.7> sudo journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.7.service 2024-09-21T09:52:59.237 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-21T09:52:59.237 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd stat -f json 2024-09-21T09:52:59.445 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:52:59.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:59 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:59.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:59 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:59.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:59 smithi070 ceph-mon[29589]: from='osd.7 [v2:172.21.15.160:6824/601673919,v1:172.21.15.160:6825/601673919]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T09:52:59.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:59 smithi070 ceph-mon[29589]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T09:52:59.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:59 smithi070 ceph-mon[29589]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-21T09:52:59.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:59 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:59.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:52:59 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:59.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:59 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:59.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:59 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:59.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:59 smithi070 ceph-mon[41533]: from='osd.7 [v2:172.21.15.160:6824/601673919,v1:172.21.15.160:6825/601673919]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T09:52:59.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:59 smithi070 ceph-mon[41533]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T09:52:59.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:59 smithi070 ceph-mon[41533]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-21T09:52:59.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:59 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:52:59.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:52:59 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:00.065 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:59 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:00.065 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:59 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:00.065 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:59 smithi160 ceph-mon[33203]: from='osd.7 [v2:172.21.15.160:6824/601673919,v1:172.21.15.160:6825/601673919]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T09:53:00.065 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:59 smithi160 ceph-mon[33203]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T09:53:00.065 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:59 smithi160 ceph-mon[33203]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-21T09:53:00.065 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:59 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:00.066 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:52:59 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:00.752 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:00 smithi070 ceph-mon[29589]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T09:53:00.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:00 smithi070 ceph-mon[29589]: osdmap e52: 8 total, 7 up, 8 in 2024-09-21T09:53:00.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:00 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:53:00.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:00 smithi070 ceph-mon[29589]: from='osd.7 [v2:172.21.15.160:6824/601673919,v1:172.21.15.160:6825/601673919]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-21T09:53:00.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:00 smithi070 ceph-mon[29589]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-21T09:53:00.753 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:00 smithi070 ceph-mon[41533]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T09:53:00.753 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:00 smithi070 ceph-mon[41533]: osdmap e52: 8 total, 7 up, 8 in 2024-09-21T09:53:00.753 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:00 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:53:00.753 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:00 smithi070 ceph-mon[41533]: from='osd.7 [v2:172.21.15.160:6824/601673919,v1:172.21.15.160:6825/601673919]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-21T09:53:00.753 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:00 smithi070 ceph-mon[41533]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-21T09:53:01.028 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:00 smithi160 ceph-mon[33203]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T09:53:01.029 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:00 smithi160 ceph-mon[33203]: osdmap e52: 8 total, 7 up, 8 in 2024-09-21T09:53:01.029 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:00 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:53:01.029 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:00 smithi160 ceph-mon[33203]: from='osd.7 [v2:172.21.15.160:6824/601673919,v1:172.21.15.160:6825/601673919]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-21T09:53:01.029 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:00 smithi160 ceph-mon[33203]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-21T09:53:01.157 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:53:01.290 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 09:53:01 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T09:53:01.136+0000 7fe7021bb640 -1 osd.7 0 waiting for initial osdmap 2024-09-21T09:53:01.290 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 09:53:01 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T09:53:01.145+0000 7fe6f9fdc640 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:53:01.711 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":53,"num_osds":8,"num_up_osds":7,"osd_up_since":1726912358,"num_in_osds":8,"osd_in_since":1726912364,"num_remapped_pgs":0} 2024-09-21T09:53:02.010 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:01 smithi160 ceph-mon[33203]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-21T09:53:02.010 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:01 smithi160 ceph-mon[33203]: osdmap e53: 8 total, 7 up, 8 in 2024-09-21T09:53:02.011 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:01 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:53:02.011 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:01 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:53:02.011 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:01 smithi160 ceph-mon[33203]: from='osd.7 ' entity='osd.7' 2024-09-21T09:53:02.011 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:01 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/68731623' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T09:53:02.011 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:01 smithi160 ceph-mon[33203]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-21T09:53:02.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:01 smithi070 ceph-mon[29589]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-21T09:53:02.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:01 smithi070 ceph-mon[29589]: osdmap e53: 8 total, 7 up, 8 in 2024-09-21T09:53:02.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:01 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:53:02.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:01 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:53:02.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:01 smithi070 ceph-mon[29589]: from='osd.7 ' entity='osd.7' 2024-09-21T09:53:02.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:01 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/68731623' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T09:53:02.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:01 smithi070 ceph-mon[29589]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-21T09:53:02.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:01 smithi070 ceph-mon[41533]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-21T09:53:02.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:01 smithi070 ceph-mon[41533]: osdmap e53: 8 total, 7 up, 8 in 2024-09-21T09:53:02.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:01 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:53:02.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:01 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:53:02.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:01 smithi070 ceph-mon[41533]: from='osd.7 ' entity='osd.7' 2024-09-21T09:53:02.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:01 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/68731623' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T09:53:02.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:01 smithi070 ceph-mon[41533]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-21T09:53:02.713 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd stat -f json 2024-09-21T09:53:02.933 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:53:03.053 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:02 smithi070 ceph-mon[29589]: purged_snaps scrub starts 2024-09-21T09:53:03.053 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:02 smithi070 ceph-mon[29589]: purged_snaps scrub ok 2024-09-21T09:53:03.053 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:02 smithi070 ceph-mon[29589]: osd.7 [v2:172.21.15.160:6824/601673919,v1:172.21.15.160:6825/601673919] boot 2024-09-21T09:53:03.054 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:02 smithi070 ceph-mon[29589]: osdmap e54: 8 total, 8 up, 8 in 2024-09-21T09:53:03.054 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:02 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:53:03.054 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:02 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:03.054 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:02 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:03.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:02 smithi070 ceph-mon[41533]: purged_snaps scrub starts 2024-09-21T09:53:03.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:02 smithi070 ceph-mon[41533]: purged_snaps scrub ok 2024-09-21T09:53:03.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:02 smithi070 ceph-mon[41533]: osd.7 [v2:172.21.15.160:6824/601673919,v1:172.21.15.160:6825/601673919] boot 2024-09-21T09:53:03.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:02 smithi070 ceph-mon[41533]: osdmap e54: 8 total, 8 up, 8 in 2024-09-21T09:53:03.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:02 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:53:03.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:02 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:03.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:02 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:02 smithi160 ceph-mon[33203]: purged_snaps scrub starts 2024-09-21T09:53:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:02 smithi160 ceph-mon[33203]: purged_snaps scrub ok 2024-09-21T09:53:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:02 smithi160 ceph-mon[33203]: osd.7 [v2:172.21.15.160:6824/601673919,v1:172.21.15.160:6825/601673919] boot 2024-09-21T09:53:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:02 smithi160 ceph-mon[33203]: osdmap e54: 8 total, 8 up, 8 in 2024-09-21T09:53:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:02 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:53:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:02 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:02 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:04.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:03 smithi160 ceph-mon[33203]: Detected new or changed devices on smithi160 2024-09-21T09:53:04.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:03 smithi160 ceph-mon[33203]: osdmap e55: 8 total, 8 up, 8 in 2024-09-21T09:53:04.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:03 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:04.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:03 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:04.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:03 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:04.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:03 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:04.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:03 smithi160 ceph-mon[33203]: Adjusting osd_memory_target on smithi160 to 4272M 2024-09-21T09:53:04.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:03 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:04.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:03 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:04.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:03 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:04.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:03 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:04.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:03 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:04.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:03 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:04.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:03 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:04.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:03 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:04.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:03 smithi160 ceph-mon[33203]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 436 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:04.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[29589]: Detected new or changed devices on smithi160 2024-09-21T09:53:04.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[29589]: osdmap e55: 8 total, 8 up, 8 in 2024-09-21T09:53:04.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[29589]: Adjusting osd_memory_target on smithi160 to 4272M 2024-09-21T09:53:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[29589]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 436 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:04.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[41533]: Detected new or changed devices on smithi160 2024-09-21T09:53:04.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[41533]: osdmap e55: 8 total, 8 up, 8 in 2024-09-21T09:53:04.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:04.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:04.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:04.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:04.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[41533]: Adjusting osd_memory_target on smithi160 to 4272M 2024-09-21T09:53:04.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:04.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:04.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:04.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:04.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:04.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:04.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:04.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:04.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:03 smithi070 ceph-mon[41533]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 436 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:04.861 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:53:04.862 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:04 smithi070 ceph-mon[29589]: osdmap e56: 8 total, 8 up, 8 in 2024-09-21T09:53:04.862 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:04 smithi070 ceph-mon[29589]: osdmap e57: 8 total, 8 up, 8 in 2024-09-21T09:53:04.862 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:04 smithi070 ceph-mon[41533]: osdmap e56: 8 total, 8 up, 8 in 2024-09-21T09:53:04.862 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:04 smithi070 ceph-mon[41533]: osdmap e57: 8 total, 8 up, 8 in 2024-09-21T09:53:05.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:04 smithi160 ceph-mon[33203]: osdmap e56: 8 total, 8 up, 8 in 2024-09-21T09:53:05.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:04 smithi160 ceph-mon[33203]: osdmap e57: 8 total, 8 up, 8 in 2024-09-21T09:53:05.474 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":57,"num_osds":8,"num_up_osds":8,"osd_up_since":1726912381,"num_in_osds":8,"osd_in_since":1726912364,"num_remapped_pgs":0} 2024-09-21T09:53:05.474 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T09:53:05.666 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:53:05.770 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:05 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/2499951962' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T09:53:05.770 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:05 smithi070 ceph-mon[29589]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 436 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:05.770 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:05 smithi070 ceph-mon[29589]: osdmap e58: 8 total, 8 up, 8 in 2024-09-21T09:53:05.770 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:05 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/2499951962' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T09:53:05.770 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:05 smithi070 ceph-mon[41533]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 436 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:05.770 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:05 smithi070 ceph-mon[41533]: osdmap e58: 8 total, 8 up, 8 in 2024-09-21T09:53:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:05 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/2499951962' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T09:53:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:05 smithi160 ceph-mon[33203]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 436 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:05 smithi160 ceph-mon[33203]: osdmap e58: 8 total, 8 up, 8 in 2024-09-21T09:53:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:53:07.399 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":58,"fsid":"7c68bfbc-77fe-11ef-baf6-efdc52797490","created":"2024-09-21T09:47:59.216934+0000","modified":"2024-09-21T09:53:05.269397+0000","last_up_change":"2024-09-21T09:53:01.680149+0000","last_in_change":"2024-09-21T09:52:44.345943+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:51:05.514472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6870ff8-14eb-4110-957a-00c73645b181","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2896279311},{"type":"v1","addr":"172.21.15.70:6803","nonce":2896279311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2896279311},{"type":"v1","addr":"172.21.15.70:6805","nonce":2896279311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":2896279311},{"type":"v1","addr":"172.21.15.70:6809","nonce":2896279311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2896279311},{"type":"v1","addr":"172.21.15.70:6807","nonce":2896279311}]},"public_addr":"172.21.15.70:6803/2896279311","cluster_addr":"172.21.15.70:6805/2896279311","heartbeat_back_addr":"172.21.15.70:6809/2896279311","heartbeat_front_addr":"172.21.15.70:6807/2896279311","state":["exists","up"]},{"osd":1,"uuid":"b1349e31-7493-489f-823a-55edb866d25f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":937875216},{"type":"v1","addr":"172.21.15.70:6811","nonce":937875216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":937875216},{"type":"v1","addr":"172.21.15.70:6813","nonce":937875216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":937875216},{"type":"v1","addr":"172.21.15.70:6817","nonce":937875216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":937875216},{"type":"v1","addr":"172.21.15.70:6815","nonce":937875216}]},"public_addr":"172.21.15.70:6811/937875216","cluster_addr":"172.21.15.70:6813/937875216","heartbeat_back_addr":"172.21.15.70:6817/937875216","heartbeat_front_addr":"172.21.15.70:6815/937875216","state":["exists","up"]},{"osd":2,"uuid":"ac4496ba-a860-4b09-bb38-7f433e081541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":940406420},{"type":"v1","addr":"172.21.15.70:6819","nonce":940406420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":940406420},{"type":"v1","addr":"172.21.15.70:6821","nonce":940406420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":940406420},{"type":"v1","addr":"172.21.15.70:6825","nonce":940406420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":940406420},{"type":"v1","addr":"172.21.15.70:6823","nonce":940406420}]},"public_addr":"172.21.15.70:6819/940406420","cluster_addr":"172.21.15.70:6821/940406420","heartbeat_back_addr":"172.21.15.70:6825/940406420","heartbeat_front_addr":"172.21.15.70:6823/940406420","state":["exists","up"]},{"osd":3,"uuid":"1eedf4ce-057c-4576-ae5b-51d3595df195","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1341239112},{"type":"v1","addr":"172.21.15.70:6827","nonce":1341239112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1341239112},{"type":"v1","addr":"172.21.15.70:6829","nonce":1341239112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1341239112},{"type":"v1","addr":"172.21.15.70:6833","nonce":1341239112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1341239112},{"type":"v1","addr":"172.21.15.70:6831","nonce":1341239112}]},"public_addr":"172.21.15.70:6827/1341239112","cluster_addr":"172.21.15.70:6829/1341239112","heartbeat_back_addr":"172.21.15.70:6833/1341239112","heartbeat_front_addr":"172.21.15.70:6831/1341239112","state":["exists","up"]},{"osd":4,"uuid":"a4b728b8-97a4-478f-8a11-ba590c0213e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6801","nonce":3634505869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6803","nonce":3634505869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6807","nonce":3634505869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6805","nonce":3634505869}]},"public_addr":"172.21.15.160:6801/3634505869","cluster_addr":"172.21.15.160:6803/3634505869","heartbeat_back_addr":"172.21.15.160:6807/3634505869","heartbeat_front_addr":"172.21.15.160:6805/3634505869","state":["exists","up"]},{"osd":5,"uuid":"56915347-9254-4e94-b202-1f16d09e7946","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6809","nonce":2006621188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6811","nonce":2006621188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6815","nonce":2006621188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6813","nonce":2006621188}]},"public_addr":"172.21.15.160:6809/2006621188","cluster_addr":"172.21.15.160:6811/2006621188","heartbeat_back_addr":"172.21.15.160:6815/2006621188","heartbeat_front_addr":"172.21.15.160:6813/2006621188","state":["exists","up"]},{"osd":6,"uuid":"79e6de8c-b78b-40e2-9229-0275a873d40e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6817","nonce":1949075400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6819","nonce":1949075400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6823","nonce":1949075400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6821","nonce":1949075400}]},"public_addr":"172.21.15.160:6817/1949075400","cluster_addr":"172.21.15.160:6819/1949075400","heartbeat_back_addr":"172.21.15.160:6823/1949075400","heartbeat_front_addr":"172.21.15.160:6821/1949075400","state":["exists","up"]},{"osd":7,"uuid":"72245c8a-2fdb-4445-b378-16569f337fcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6825","nonce":601673919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6827","nonce":601673919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6831","nonce":601673919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6829","nonce":601673919}]},"public_addr":"172.21.15.160:6825/601673919","cluster_addr":"172.21.15.160:6827/601673919","heartbeat_back_addr":"172.21.15.160:6831/601673919","heartbeat_front_addr":"172.21.15.160:6829/601673919","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:15.134515+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:39.191791+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:51:02.934581+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:51:26.514393+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:51:48.329249+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:12.268576+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:36.370859+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:59.888857+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:6801/2142025682":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/3159615643":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/2918782890":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/749465179":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/4150778424":"2024-09-22T09:48:32.256720+0000","172.21.15.70:0/1012316198":"2024-09-22T09:48:32.256720+0000","172.21.15.70:6801/177993484":"2024-09-22T09:48:32.256720+0000","172.21.15.70:6800/177993484":"2024-09-22T09:48:32.256720+0000","172.21.15.70:6800/2142025682":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/2521911561":"2024-09-22T09:48:32.256720+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T09:53:08.071 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T09:51:05.514472+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-21T09:53:08.071 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool get .mgr pg_num 2024-09-21T09:53:08.266 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:53:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:08 smithi160 ceph-mon[33203]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-09-21T09:53:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:08 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/1124484040' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:53:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:08 smithi070 ceph-mon[29589]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-09-21T09:53:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:08 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/1124484040' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:53:08.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:08 smithi070 ceph-mon[41533]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-09-21T09:53:08.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:08 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/1124484040' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:53:10.135 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-09-21T09:53:10.494 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:10 smithi070 ceph-mon[29589]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T09:53:10.494 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:10 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/3013753217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-21T09:53:10.494 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:10 smithi070 ceph-mon[41533]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T09:53:10.494 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:10 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/3013753217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-21T09:53:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:10 smithi160 ceph-mon[33203]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T09:53:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:10 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/3013753217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-21T09:53:10.789 INFO:tasks.cephadm:Adding prometheus.a on smithi160 2024-09-21T09:53:10.790 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph orch apply prometheus '1;smithi160=a' 2024-09-21T09:53:10.985 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:53:12.384 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:12 smithi160 ceph-mon[33203]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-21T09:53:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:12 smithi070 ceph-mon[29589]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-21T09:53:12.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:12 smithi070 ceph-mon[41533]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-21T09:53:12.758 INFO:teuthology.orchestra.run.smithi160.stdout:Scheduled prometheus update... 2024-09-21T09:53:13.485 DEBUG:teuthology.orchestra.run.smithi160:prometheus.a> sudo journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@prometheus.a.service 2024-09-21T09:53:13.487 INFO:tasks.cephadm:Adding node-exporter.a on smithi070 2024-09-21T09:53:13.487 INFO:tasks.cephadm:Adding node-exporter.b on smithi160 2024-09-21T09:53:13.487 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph orch apply node-exporter '2;smithi070=a;smithi160=b' 2024-09-21T09:53:13.730 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:53:14.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:13 smithi160 ceph-mon[33203]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi160=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:14.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:13 smithi160 ceph-mon[33203]: Saving service prometheus spec with placement smithi160=a;count:1 2024-09-21T09:53:14.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:13 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:14.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:13 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:14.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:13 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:14.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:13 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:14.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:13 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:14.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:13 smithi160 ceph-mon[33203]: Deploying daemon prometheus.a on smithi160 2024-09-21T09:53:14.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:13 smithi160 ceph-mon[33203]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-09-21T09:53:14.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:13 smithi070 ceph-mon[29589]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi160=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:14.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:13 smithi070 ceph-mon[29589]: Saving service prometheus spec with placement smithi160=a;count:1 2024-09-21T09:53:14.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:13 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:14.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:13 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:14.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:13 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:14.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:13 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:14.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:13 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:14.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:13 smithi070 ceph-mon[29589]: Deploying daemon prometheus.a on smithi160 2024-09-21T09:53:14.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:13 smithi070 ceph-mon[29589]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-09-21T09:53:14.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:13 smithi070 ceph-mon[41533]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi160=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:14.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:13 smithi070 ceph-mon[41533]: Saving service prometheus spec with placement smithi160=a;count:1 2024-09-21T09:53:14.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:13 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:14.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:13 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:14.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:13 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:14.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:13 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:14.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:13 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:14.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:13 smithi070 ceph-mon[41533]: Deploying daemon prometheus.a on smithi160 2024-09-21T09:53:14.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:13 smithi070 ceph-mon[41533]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-09-21T09:53:16.455 INFO:teuthology.orchestra.run.smithi160.stdout:Scheduled node-exporter update... 2024-09-21T09:53:16.455 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:16 smithi160 ceph-mon[33203]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:53:16.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:16 smithi070 ceph-mon[41533]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:53:16.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:16 smithi070 ceph-mon[29589]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:53:17.364 DEBUG:teuthology.orchestra.run.smithi070:node-exporter.a> sudo journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@node-exporter.a.service 2024-09-21T09:53:17.367 DEBUG:teuthology.orchestra.run.smithi160:node-exporter.b> sudo journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@node-exporter.b.service 2024-09-21T09:53:17.369 INFO:tasks.cephadm:Adding alertmanager.a on smithi070 2024-09-21T09:53:17.370 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph orch apply alertmanager '1;smithi070=a' 2024-09-21T09:53:17.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:17 smithi160 ceph-mon[33203]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi070=a;smithi160=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:17.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:17 smithi160 ceph-mon[33203]: Saving service node-exporter spec with placement smithi070=a;smithi160=b;count:2 2024-09-21T09:53:17.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:17 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:17.608 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:53:17.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:17 smithi070 ceph-mon[29589]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi070=a;smithi160=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:17.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:17 smithi070 ceph-mon[29589]: Saving service node-exporter spec with placement smithi070=a;smithi160=b;count:2 2024-09-21T09:53:17.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:17 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:17.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:17 smithi070 ceph-mon[41533]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi070=a;smithi160=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:17.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:17 smithi070 ceph-mon[41533]: Saving service node-exporter spec with placement smithi070=a;smithi160=b;count:2 2024-09-21T09:53:17.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:17 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:18.776 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:18 smithi160 ceph-mon[33203]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T09:53:18.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:18 smithi070 ceph-mon[29589]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T09:53:18.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:18 smithi070 ceph-mon[41533]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T09:53:20.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:20 smithi160 ceph-mon[33203]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:20.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:20 smithi070 ceph-mon[29589]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:20.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:20 smithi070 ceph-mon[41533]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:21.623 INFO:teuthology.orchestra.run.smithi160.stdout:Scheduled alertmanager update... 2024-09-21T09:53:22.502 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:22 smithi160 ceph-mon[33203]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:22.502 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:22 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:22.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:22 smithi070 ceph-mon[29589]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:22.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:22 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:22.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:22 smithi070 ceph-mon[41533]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:22.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:22 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:23.550 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:23 smithi160 ceph-mon[33203]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi070=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:23.550 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:23 smithi160 ceph-mon[33203]: Saving service alertmanager spec with placement smithi070=a;count:1 2024-09-21T09:53:23.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:23 smithi070 ceph-mon[41533]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi070=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:23.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:23 smithi070 ceph-mon[41533]: Saving service alertmanager spec with placement smithi070=a;count:1 2024-09-21T09:53:23.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:23 smithi070 ceph-mon[29589]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi070=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:23.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:23 smithi070 ceph-mon[29589]: Saving service alertmanager spec with placement smithi070=a;count:1 2024-09-21T09:53:24.749 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:24 smithi160 ceph-mon[33203]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:24.774 DEBUG:teuthology.orchestra.run.smithi070:alertmanager.a> sudo journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@alertmanager.a.service 2024-09-21T09:53:24.777 INFO:tasks.cephadm:Adding grafana.a on smithi160 2024-09-21T09:53:24.777 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph orch apply grafana '1;smithi160=a' 2024-09-21T09:53:24.802 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:24 smithi070 ceph-mon[29589]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:24.802 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:24 smithi070 ceph-mon[41533]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:25.001 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:53:24 smithi160 podman[61980]: 2024-09-21 09:53:24.888989204 +0000 UTC m=+0.018387671 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-21T09:53:25.001 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:53:24 smithi160 podman[61980]: 2024-09-21 09:53:24.999465741 +0000 UTC m=+0.128864265 container create b6e4ec2e67697798891cc34de74d2e26e1af350be38e92ea41e72b2d0a77b480 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:53:25.265 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:53:25 smithi160 podman[61980]: 2024-09-21 09:53:25.235298136 +0000 UTC m=+0.364696604 container init b6e4ec2e67697798891cc34de74d2e26e1af350be38e92ea41e72b2d0a77b480 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:53:25.266 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:53:25 smithi160 podman[61980]: 2024-09-21 09:53:25.240237749 +0000 UTC m=+0.369636208 container start b6e4ec2e67697798891cc34de74d2e26e1af350be38e92ea41e72b2d0a77b480 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:53:25.363 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:53:25.563 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:53:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:53:25.264Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-21T09:53:25.563 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:53:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:53:25.264Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-21T09:53:25.563 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:53:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:53:25.264Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi160 (none))" 2024-09-21T09:53:25.563 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:53:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:53:25.264Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-21T09:53:25.563 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:53:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:53:25.264Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-21T09:53:25.563 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:53:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:53:25.265Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-21T09:53:25.563 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:53:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:53:25.266Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-21T09:53:25.563 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:53:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:53:25.267Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-21T09:53:25.563 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:53:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:53:25.267Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-21T09:53:25.563 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:53:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:53:25.269Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-21T09:53:25.563 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:53:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:53:25.269Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.237µs 2024-09-21T09:53:25.564 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:53:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:53:25.269Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-21T09:53:25.564 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:53:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:53:25.270Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-09-21T09:53:25.564 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:53:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:53:25.270Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=26.97µs wal_replay_duration=191.357µs wbl_replay_duration=129ns total_replay_duration=244.214µs 2024-09-21T09:53:25.564 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:53:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:53:25.270Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-21T09:53:25.564 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:53:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:53:25.270Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-21T09:53:25.564 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:53:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:53:25.270Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-21T09:53:25.564 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:53:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:53:25.291Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.918233ms db_storage=743ns remote_storage=1.112µs web_handler=393ns query_engine=511ns scrape=438.344µs scrape_sd=82.609µs notify=667ns notify_sd=1.233µs rules=20.130801ms tracing=5.76µs 2024-09-21T09:53:25.564 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:53:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:53:25.291Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-21T09:53:25.564 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:53:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:53:25.291Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-21T09:53:25.564 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:53:25 smithi160 bash[61980]: b6e4ec2e67697798891cc34de74d2e26e1af350be38e92ea41e72b2d0a77b480 2024-09-21T09:53:25.564 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:53:25 smithi160 systemd[1]: Started Ceph prometheus.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:53:26.526 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:26 smithi160 ceph-mon[33203]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:26.526 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:26 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:26.526 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:26 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:26.526 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:26 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:26.526 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:26 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-21T09:53:26.527 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:26 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: ignoring --setuser ceph since I am not root 2024-09-21T09:53:26.527 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:26 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: ignoring --setgroup ceph since I am not root 2024-09-21T09:53:26.527 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:26 smithi160 ceph-mgr[35311]: -- 172.21.15.160:0/2427743989 <== mon.1 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bccb694d00 con 0x55bccb669000 2024-09-21T09:53:26.755 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:26 smithi070 ceph-mon[29589]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:26.756 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:26 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:26.756 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:26 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:26.756 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:26 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:26.756 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:26 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-21T09:53:26.756 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:26 smithi070 ceph-mon[41533]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:26.756 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:26 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:26.756 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:26 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:26.756 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:26 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' 2024-09-21T09:53:26.756 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:26 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-21T09:53:26.757 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:26 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ignoring --setuser ceph since I am not root 2024-09-21T09:53:26.757 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:26 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ignoring --setgroup ceph since I am not root 2024-09-21T09:53:26.757 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:26 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:26.552+0000 7fb74e825040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T09:53:26.757 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:26 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:26.607+0000 7fb74e825040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T09:53:26.757 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:26 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:26.752+0000 7fb74e825040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T09:53:26.781 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:26 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:53:26.550+0000 7f3f72648040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T09:53:26.781 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:26 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:53:26.610+0000 7f3f72648040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T09:53:26.781 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:26 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:53:26.758+0000 7f3f72648040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T09:53:27.091 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:26 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:53:26.831+0000 7f3f72648040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:53:27.091 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:26 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:53:26.978+0000 7f3f72648040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:53:27.106 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:26 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:26.818+0000 7fb74e825040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:53:27.106 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:26 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:26.962+0000 7fb74e825040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:53:27.106 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:27 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:27.103+0000 7fb74e825040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:53:27.409 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:27 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:53:27.124+0000 7f3f72648040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:53:27.671 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:27 smithi160 ceph-mon[33203]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T09:53:27.671 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:27 smithi160 ceph-mon[33203]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-21T09:53:27.672 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:27 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:53:27.430+0000 7f3f72648040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:53:27.672 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:27 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:53:27.490+0000 7f3f72648040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:53:27.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:27 smithi070 ceph-mon[29589]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T09:53:27.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:27 smithi070 ceph-mon[29589]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-21T09:53:27.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:27 smithi070 ceph-mon[41533]: from='mgr.14150 172.21.15.70:0/3725338588' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T09:53:27.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:27 smithi070 ceph-mon[41533]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-21T09:53:27.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:27 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:27.413+0000 7fb74e825040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:53:27.878 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:27 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:27.473+0000 7fb74e825040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:53:27.878 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:27 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:27.654+0000 7fb74e825040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:53:28.841 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:27 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:53:27.669+0000 7f3f72648040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:53:28.841 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:28 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:53:28.528+0000 7f3f72648040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:53:28.841 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:28 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:53:28.592+0000 7f3f72648040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:53:28.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:28 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:28.520+0000 7fb74e825040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:53:28.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:28 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:28.587+0000 7fb74e825040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:53:29.541 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:29 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:29.285+0000 7fb74e825040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:53:29.541 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:29 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:29.399+0000 7fb74e825040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:53:29.542 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:29 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:53:29.288+0000 7f3f72648040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:53:29.542 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:29 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:53:29.401+0000 7f3f72648040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:53:29.811 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:29 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:53:29.540+0000 7f3f72648040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:53:29.811 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:29 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:53:29.600+0000 7f3f72648040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:53:29.811 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:29 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:53:29.743+0000 7f3f72648040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:53:29.811 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:29 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:53:29.809+0000 7f3f72648040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:53:29.813 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:29 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:29.539+0000 7fb74e825040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:53:29.813 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:29 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:29.599+0000 7fb74e825040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:53:29.813 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:29 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:29.743+0000 7fb74e825040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:53:29.813 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:29 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:29.810+0000 7fb74e825040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:53:30.086 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:29 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:53:29.867+0000 7f3f72648040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:53:30.086 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:30 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:53:30.020+0000 7f3f72648040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:53:30.086 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:30 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:53:30.084+0000 7f3f72648040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:53:30.092 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:29 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:29.870+0000 7fb74e825040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:53:30.092 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:30 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:30.025+0000 7fb74e825040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:53:30.092 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:30 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:30.089+0000 7fb74e825040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:53:30.341 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:30 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:53:30.149+0000 7f3f72648040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:53:30.341 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:30 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:53:30.231+0000 7f3f72648040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:53:30.341 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:30 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:53:30.289+0000 7f3f72648040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:53:30.359 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:30 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:30.156+0000 7fb74e825040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:53:30.360 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:30 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:30.238+0000 7fb74e825040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:53:30.360 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:30 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:30.298+0000 7fb74e825040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:53:30.360 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:30 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:30.357+0000 7fb74e825040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:53:30.841 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:30 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:53:30.347+0000 7f3f72648040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:53:31.384 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:31.131+0000 7fb74e825040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:53:31.384 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:53:31] ENGINE Bus STARTING 2024-09-21T09:53:31.384 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: CherryPy Checker: 2024-09-21T09:53:31.384 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: The Application mounted at '' has an empty config. 2024-09-21T09:53:31.384 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:31.384 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:53:31] ENGINE Serving on http://:::9283 2024-09-21T09:53:31.384 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:53:31] ENGINE Bus STARTED 2024-09-21T09:53:31.385 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[29589]: Standby manager daemon x restarted 2024-09-21T09:53:31.385 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[29589]: Standby manager daemon x started 2024-09-21T09:53:31.385 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/1513521605' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:53:31.385 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/1513521605' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:53:31.385 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/1513521605' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:53:31.385 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/1513521605' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:53:31.385 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[29589]: Active manager daemon y restarted 2024-09-21T09:53:31.385 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[29589]: Activating manager daemon y 2024-09-21T09:53:31.385 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[29589]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T09:53:31.385 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[29589]: mgrmap e17: y(active, starting, since 0.00692044s), standbys: x 2024-09-21T09:53:31.385 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:53:31.386 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:53:31.386 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:53:31.386 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:53:31.386 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:53:31.386 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:53:31.386 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:53:31.386 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:53:31.386 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:53:31.386 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:53:31.386 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:53:31.386 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:53:31.386 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:53:31.386 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:53:31.386 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:53:31.387 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:53:31.387 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[29589]: Manager daemon y is now available 2024-09-21T09:53:31.387 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[41533]: Standby manager daemon x restarted 2024-09-21T09:53:31.387 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[41533]: Standby manager daemon x started 2024-09-21T09:53:31.387 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.160:0/1513521605' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:53:31.387 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.160:0/1513521605' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:53:31.387 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.160:0/1513521605' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:53:31.387 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.160:0/1513521605' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:53:31.387 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[41533]: Active manager daemon y restarted 2024-09-21T09:53:31.387 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[41533]: Activating manager daemon y 2024-09-21T09:53:31.387 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[41533]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T09:53:31.387 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[41533]: mgrmap e17: y(active, starting, since 0.00692044s), standbys: x 2024-09-21T09:53:31.388 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:53:31.388 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:53:31.388 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:53:31.388 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:53:31.388 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:53:31.388 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:53:31.389 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:53:31.389 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:53:31.389 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:53:31.390 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:53:31.390 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:53:31.390 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:53:31.390 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:53:31.390 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:53:31.390 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:53:31.390 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:53:31.390 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:31 smithi070 ceph-mon[41533]: Manager daemon y is now available 2024-09-21T09:53:31.514 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:53:31.113+0000 7f3f72648040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:53:31.514 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: [21/Sep/2024:09:53:31] ENGINE Bus STARTING 2024-09-21T09:53:31.514 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: CherryPy Checker: 2024-09-21T09:53:31.514 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: The Application mounted at '' has an empty config. 2024-09-21T09:53:31.514 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: 2024-09-21T09:53:31.514 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: [21/Sep/2024:09:53:31] ENGINE Serving on http://:::9283 2024-09-21T09:53:31.514 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[35288]: [21/Sep/2024:09:53:31] ENGINE Bus STARTED 2024-09-21T09:53:31.515 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-mon[33203]: Standby manager daemon x restarted 2024-09-21T09:53:31.515 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-mon[33203]: Standby manager daemon x started 2024-09-21T09:53:31.515 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.160:0/1513521605' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:53:31.515 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.160:0/1513521605' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:53:31.515 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.160:0/1513521605' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:53:31.515 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.160:0/1513521605' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:53:31.515 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-mon[33203]: Active manager daemon y restarted 2024-09-21T09:53:31.515 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-mon[33203]: Activating manager daemon y 2024-09-21T09:53:31.515 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-mon[33203]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T09:53:31.515 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-mon[33203]: mgrmap e17: y(active, starting, since 0.00692044s), standbys: x 2024-09-21T09:53:31.515 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:53:31.515 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:53:31.515 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:53:31.515 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:53:31.516 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:53:31.516 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:53:31.516 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:53:31.516 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:53:31.516 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:53:31.516 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:53:31.516 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:53:31.516 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:53:31.516 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:53:31.516 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:53:31.516 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:53:31.516 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:53:31.516 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:31 smithi160 ceph-mon[33203]: Manager daemon y is now available 2024-09-21T09:53:32.159 INFO:teuthology.orchestra.run.smithi160.stdout:Scheduled grafana update... 2024-09-21T09:53:32.209 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:32.209 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:32.209 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:53:32.209 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:53:32.209 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:53:32.209 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-mon[29589]: mgrmap e18: y(active, since 1.01221s), standbys: x 2024-09-21T09:53:32.210 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:32.210 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:32.210 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:32.210 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:53:32.210 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:53:32.210 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:53:32.210 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-mon[41533]: mgrmap e18: y(active, since 1.01221s), standbys: x 2024-09-21T09:53:32.210 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:32.419 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:32 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:32.419 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:32 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:32.419 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:32 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:53:32.419 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:32 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:53:32.419 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:32 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:53:32.420 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:32 smithi160 ceph-mon[33203]: mgrmap e18: y(active, since 1.01221s), standbys: x 2024-09-21T09:53:32.420 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:32 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:32.707 DEBUG:teuthology.orchestra.run.smithi160:grafana.a> sudo journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@grafana.a.service 2024-09-21T09:53:32.710 INFO:tasks.cephadm:Setting up client nodes... 2024-09-21T09:53:32.710 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-21T09:53:32.916 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:53:33.041 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:32.667+0000 7fb6a6d94640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:09:53:32] ENGINE Error in HTTPServer.serve 2024-09-21T09:53:33.041 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: Traceback (most recent call last): 2024-09-21T09:53:33.041 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:53:33.041 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: self._connections.run(self.expiration_interval) 2024-09-21T09:53:33.041 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:53:33.041 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: self._run(expiration_interval) 2024-09-21T09:53:33.042 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:53:33.042 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:53:33.042 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:53:33.042 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:53:33.042 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:53:33.042 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: s = self.context.wrap_socket( 2024-09-21T09:53:33.042 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:53:33.042 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: return self.sslsocket_class._create( 2024-09-21T09:53:33.042 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:53:33.042 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: self.do_handshake() 2024-09-21T09:53:33.042 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:53:33.042 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: self._sslobj.do_handshake() 2024-09-21T09:53:33.042 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:53:33.042 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:32 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:53:33.331 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:33 smithi070 ceph-mon[41533]: Saving service grafana spec with placement smithi160=a;count:1 2024-09-21T09:53:33.331 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:33 smithi070 ceph-mon[41533]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:33.331 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:33 smithi070 ceph-mon[41533]: [21/Sep/2024:09:53:32] ENGINE Bus STARTING 2024-09-21T09:53:33.331 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:33 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:33.331 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:33 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:33.331 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:33 smithi070 ceph-mon[29589]: Saving service grafana spec with placement smithi160=a;count:1 2024-09-21T09:53:33.331 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:33 smithi070 ceph-mon[29589]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:33.331 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:33 smithi070 ceph-mon[29589]: [21/Sep/2024:09:53:32] ENGINE Bus STARTING 2024-09-21T09:53:33.332 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:33 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:33.332 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:33 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:33.581 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:33 smithi160 ceph-mon[33203]: Saving service grafana spec with placement smithi160=a;count:1 2024-09-21T09:53:33.581 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:33 smithi160 ceph-mon[33203]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:33.581 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:33 smithi160 ceph-mon[33203]: [21/Sep/2024:09:53:32] ENGINE Bus STARTING 2024-09-21T09:53:33.582 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:33 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:33.582 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:33 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:34.438 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:34 smithi070 ceph-mon[29589]: [21/Sep/2024:09:53:32] ENGINE Serving on http://172.21.15.70:8765 2024-09-21T09:53:34.438 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:34 smithi070 ceph-mon[29589]: [21/Sep/2024:09:53:32] ENGINE Serving on https://172.21.15.70:7150 2024-09-21T09:53:34.438 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:34 smithi070 ceph-mon[29589]: [21/Sep/2024:09:53:32] ENGINE Bus STARTED 2024-09-21T09:53:34.438 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:34 smithi070 ceph-mon[29589]: [21/Sep/2024:09:53:32] ENGINE Error in HTTPServer.serve 2024-09-21T09:53:34.438 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-09-21T09:53:34.438 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:53:34.438 INFO:journalctl@ceph.mon.a.smithi070.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:53:34.438 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:53:34.439 INFO:journalctl@ceph.mon.a.smithi070.stdout: self._run(expiration_interval) 2024-09-21T09:53:34.439 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:53:34.439 INFO:journalctl@ceph.mon.a.smithi070.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:53:34.439 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:53:34.439 INFO:journalctl@ceph.mon.a.smithi070.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:53:34.439 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:53:34.439 INFO:journalctl@ceph.mon.a.smithi070.stdout: s = self.context.wrap_socket( 2024-09-21T09:53:34.439 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:53:34.439 INFO:journalctl@ceph.mon.a.smithi070.stdout: return self.sslsocket_class._create( 2024-09-21T09:53:34.439 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:53:34.439 INFO:journalctl@ceph.mon.a.smithi070.stdout: self.do_handshake() 2024-09-21T09:53:34.439 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:53:34.439 INFO:journalctl@ceph.mon.a.smithi070.stdout: self._sslobj.do_handshake() 2024-09-21T09:53:34.439 INFO:journalctl@ceph.mon.a.smithi070.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:53:34.440 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:34 smithi070 ceph-mon[29589]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:34.440 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:34 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:34.440 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:34 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:34.440 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:34 smithi070 ceph-mon[29589]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-21T09:53:34.440 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:34 smithi070 ceph-mon[41533]: [21/Sep/2024:09:53:32] ENGINE Serving on http://172.21.15.70:8765 2024-09-21T09:53:34.440 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:34 smithi070 ceph-mon[41533]: [21/Sep/2024:09:53:32] ENGINE Serving on https://172.21.15.70:7150 2024-09-21T09:53:34.440 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:34 smithi070 ceph-mon[41533]: [21/Sep/2024:09:53:32] ENGINE Bus STARTED 2024-09-21T09:53:34.440 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:34 smithi070 ceph-mon[41533]: [21/Sep/2024:09:53:32] ENGINE Error in HTTPServer.serve 2024-09-21T09:53:34.440 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-09-21T09:53:34.440 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:53:34.440 INFO:journalctl@ceph.mon.c.smithi070.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:53:34.440 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:53:34.440 INFO:journalctl@ceph.mon.c.smithi070.stdout: self._run(expiration_interval) 2024-09-21T09:53:34.441 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:53:34.441 INFO:journalctl@ceph.mon.c.smithi070.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:53:34.441 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:53:34.441 INFO:journalctl@ceph.mon.c.smithi070.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:53:34.441 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:53:34.441 INFO:journalctl@ceph.mon.c.smithi070.stdout: s = self.context.wrap_socket( 2024-09-21T09:53:34.441 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:53:34.441 INFO:journalctl@ceph.mon.c.smithi070.stdout: return self.sslsocket_class._create( 2024-09-21T09:53:34.441 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:53:34.441 INFO:journalctl@ceph.mon.c.smithi070.stdout: self.do_handshake() 2024-09-21T09:53:34.441 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:53:34.441 INFO:journalctl@ceph.mon.c.smithi070.stdout: self._sslobj.do_handshake() 2024-09-21T09:53:34.441 INFO:journalctl@ceph.mon.c.smithi070.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:53:34.441 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:34 smithi070 ceph-mon[41533]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:34.441 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:34 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:34.442 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:34 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:34.442 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:34 smithi070 ceph-mon[41533]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-21T09:53:34.463 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:34 smithi160 ceph-mon[33203]: [21/Sep/2024:09:53:32] ENGINE Serving on http://172.21.15.70:8765 2024-09-21T09:53:34.464 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:34 smithi160 ceph-mon[33203]: [21/Sep/2024:09:53:32] ENGINE Serving on https://172.21.15.70:7150 2024-09-21T09:53:34.464 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:34 smithi160 ceph-mon[33203]: [21/Sep/2024:09:53:32] ENGINE Bus STARTED 2024-09-21T09:53:34.464 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:34 smithi160 ceph-mon[33203]: [21/Sep/2024:09:53:32] ENGINE Error in HTTPServer.serve 2024-09-21T09:53:34.464 INFO:journalctl@ceph.mon.b.smithi160.stdout: Traceback (most recent call last): 2024-09-21T09:53:34.464 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:53:34.464 INFO:journalctl@ceph.mon.b.smithi160.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:53:34.464 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:53:34.464 INFO:journalctl@ceph.mon.b.smithi160.stdout: self._run(expiration_interval) 2024-09-21T09:53:34.464 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:53:34.464 INFO:journalctl@ceph.mon.b.smithi160.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:53:34.464 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:53:34.464 INFO:journalctl@ceph.mon.b.smithi160.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:53:34.464 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:53:34.465 INFO:journalctl@ceph.mon.b.smithi160.stdout: s = self.context.wrap_socket( 2024-09-21T09:53:34.465 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:53:34.465 INFO:journalctl@ceph.mon.b.smithi160.stdout: return self.sslsocket_class._create( 2024-09-21T09:53:34.465 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:53:34.465 INFO:journalctl@ceph.mon.b.smithi160.stdout: self.do_handshake() 2024-09-21T09:53:34.465 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:53:34.465 INFO:journalctl@ceph.mon.b.smithi160.stdout: self._sslobj.do_handshake() 2024-09-21T09:53:34.465 INFO:journalctl@ceph.mon.b.smithi160.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:53:34.465 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:34 smithi160 ceph-mon[33203]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:34.465 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:34 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:34.465 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:34 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:34.465 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:34 smithi160 ceph-mon[33203]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-21T09:53:34.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:34 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:53:34] "GET /metrics HTTP/1.1" 200 23679 "" "Prometheus/2.43.0" 2024-09-21T09:53:34.903 INFO:teuthology.orchestra.run.smithi070.stdout:[client.0] 2024-09-21T09:53:34.903 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQCel+5mP0SHNRAAuQudPbmkuRzGUouQn22l2A== 2024-09-21T09:53:35.549 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:35 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/516740055' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:53:35.549 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:35 smithi070 ceph-mon[29589]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:53:35.549 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:35 smithi070 ceph-mon[29589]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T09:53:35.549 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:35 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/516740055' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:53:35.549 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:35 smithi070 ceph-mon[41533]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:53:35.550 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:35 smithi070 ceph-mon[41533]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T09:53:35.570 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T09:53:35.570 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-21T09:53:35.570 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-21T09:53:35.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:35 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/516740055' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:53:35.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:35 smithi160 ceph-mon[33203]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:53:35.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:35 smithi160 ceph-mon[33203]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T09:53:35.607 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-21T09:53:35.809 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:53:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:36 smithi160 ceph-mon[33203]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:36 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:36 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:36 smithi160 ceph-mon[33203]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-21T09:53:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:36 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:36 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:36 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:36.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:36 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:36.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:36 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:36.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:36 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:36.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:36 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:36.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:36 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:36.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:36 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:36 smithi070 ceph-mon[29589]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:36 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:36 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:36 smithi070 ceph-mon[29589]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-21T09:53:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:36 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:36 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:36 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:36 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:36 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:36 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:36 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:36 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:36 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:36 smithi070 ceph-mon[41533]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:36 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:36 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:36 smithi070 ceph-mon[41533]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-21T09:53:36.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:36 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:36.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:36 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:36.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:36 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:36.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:36 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:36.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:36 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:36.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:36 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:36.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:36 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:36.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:36 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:36.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:36 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:37.279 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:37 smithi160 ceph-mon[33203]: Adjusting osd_memory_target on smithi160 to 4016M 2024-09-21T09:53:37.279 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:37 smithi160 ceph-mon[33203]: Updating smithi070:/etc/ceph/ceph.conf 2024-09-21T09:53:37.279 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:37 smithi160 ceph-mon[33203]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-21T09:53:37.279 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:37 smithi160 ceph-mon[33203]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:53:37.279 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:37 smithi160 ceph-mon[33203]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:53:37.279 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:37 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:37.279 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:37 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:37.279 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:37 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:37.279 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:37 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:37.280 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:37 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:37 smithi070 ceph-mon[29589]: Adjusting osd_memory_target on smithi160 to 4016M 2024-09-21T09:53:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:37 smithi070 ceph-mon[29589]: Updating smithi070:/etc/ceph/ceph.conf 2024-09-21T09:53:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:37 smithi070 ceph-mon[29589]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-21T09:53:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:37 smithi070 ceph-mon[29589]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:53:37.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:37 smithi070 ceph-mon[29589]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:53:37.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:37 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:37.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:37 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:37.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:37 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:37.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:37 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:37.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:37 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:37 smithi070 ceph-mon[41533]: Adjusting osd_memory_target on smithi160 to 4016M 2024-09-21T09:53:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:37 smithi070 ceph-mon[41533]: Updating smithi070:/etc/ceph/ceph.conf 2024-09-21T09:53:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:37 smithi070 ceph-mon[41533]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-21T09:53:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:37 smithi070 ceph-mon[41533]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:53:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:37 smithi070 ceph-mon[41533]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:53:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:37 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:37.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:37 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:37.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:37 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:37.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:37 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:37.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:37 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:37.713 INFO:teuthology.orchestra.run.smithi160.stdout:[client.1] 2024-09-21T09:53:37.713 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQChl+5mOhdCKhAAv0Wed18wqh8rYcGNUc+BPg== 2024-09-21T09:53:38.299 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:38 smithi160 ceph-mon[33203]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:53:38.299 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:38 smithi160 ceph-mon[33203]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:53:38.299 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:38 smithi160 ceph-mon[33203]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:53:38.300 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:38 smithi160 ceph-mon[33203]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:53:38.300 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:38 smithi160 ceph-mon[33203]: Deploying daemon alertmanager.a on smithi070 2024-09-21T09:53:38.300 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:38 smithi160 ceph-mon[33203]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:38.300 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:38 smithi160 ceph-mon[33203]: from='client.? 172.21.15.160:0/1720764898' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:53:38.300 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:38 smithi160 ceph-mon[33203]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:53:38.300 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:38 smithi160 ceph-mon[33203]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T09:53:38.320 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T09:53:38.321 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-21T09:53:38.321 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-21T09:53:38.357 INFO:tasks.cephadm:Creating RBD pool 2024-09-21T09:53:38.358 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool create rbd 8 2024-09-21T09:53:38.563 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:53:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:38 smithi070 ceph-mon[29589]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:53:38.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:38 smithi070 ceph-mon[29589]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:53:38.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:38 smithi070 ceph-mon[29589]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:53:38.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:38 smithi070 ceph-mon[29589]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:53:38.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:38 smithi070 ceph-mon[29589]: Deploying daemon alertmanager.a on smithi070 2024-09-21T09:53:38.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:38 smithi070 ceph-mon[29589]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:38.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:38 smithi070 ceph-mon[29589]: from='client.? 172.21.15.160:0/1720764898' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:53:38.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:38 smithi070 ceph-mon[29589]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:53:38.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:38 smithi070 ceph-mon[29589]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T09:53:38.740 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:38 smithi070 ceph-mon[41533]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:53:38.740 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:38 smithi070 ceph-mon[41533]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:53:38.740 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:38 smithi070 ceph-mon[41533]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:53:38.740 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:38 smithi070 ceph-mon[41533]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:53:38.740 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:38 smithi070 ceph-mon[41533]: Deploying daemon alertmanager.a on smithi070 2024-09-21T09:53:38.740 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:38 smithi070 ceph-mon[41533]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:38.741 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:38 smithi070 ceph-mon[41533]: from='client.? 172.21.15.160:0/1720764898' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:53:38.741 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:38 smithi070 ceph-mon[41533]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:53:38.741 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:38 smithi070 ceph-mon[41533]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T09:53:40.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:40 smithi070 ceph-mon[41533]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-21T09:53:40.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:40 smithi070 ceph-mon[29589]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-21T09:53:40.571 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:40 smithi160 ceph-mon[33203]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-21T09:53:42.186 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'rbd' created 2024-09-21T09:53:42.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:42 smithi070 ceph-mon[29589]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T09:53:42.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:42 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/4145897567' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-21T09:53:42.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:42 smithi070 ceph-mon[29589]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-21T09:53:42.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:42 smithi070 ceph-mon[41533]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T09:53:42.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:42 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/4145897567' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-21T09:53:42.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:42 smithi070 ceph-mon[41533]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-21T09:53:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:42 smithi160 ceph-mon[33203]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T09:53:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:42 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/4145897567' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-21T09:53:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:42 smithi160 ceph-mon[33203]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-21T09:53:43.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:43 smithi070 ceph-mon[41533]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-09-21T09:53:43.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:43 smithi070 ceph-mon[41533]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T09:53:43.324 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:43 smithi070 ceph-mon[29589]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-09-21T09:53:43.324 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:43 smithi070 ceph-mon[29589]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T09:53:43.348 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- rbd pool init rbd 2024-09-21T09:53:43.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:43 smithi160 ceph-mon[33203]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-09-21T09:53:43.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:43 smithi160 ceph-mon[33203]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T09:53:44.338 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:44 smithi070 ceph-mon[41533]: pgmap v10: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T09:53:44.338 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:44 smithi070 ceph-mon[41533]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T09:53:44.338 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:44 smithi070 ceph-mon[41533]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:53:44.339 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:44 smithi070 ceph-mon[41533]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T09:53:44.339 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:44 smithi070 ceph-mon[29589]: pgmap v10: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T09:53:44.339 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:44 smithi070 ceph-mon[29589]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T09:53:44.339 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:44 smithi070 ceph-mon[29589]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:53:44.339 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:44 smithi070 ceph-mon[29589]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T09:53:44.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:44 smithi160 ceph-mon[33203]: pgmap v10: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T09:53:44.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:44 smithi160 ceph-mon[33203]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T09:53:44.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:44 smithi160 ceph-mon[33203]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:53:44.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:44 smithi160 ceph-mon[33203]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T09:53:44.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:53:44] "GET /metrics HTTP/1.1" 200 23679 "" "Prometheus/2.43.0" 2024-09-21T09:53:44.684 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:53:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:46 smithi160 ceph-mon[33203]: pgmap v13: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:46 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:46 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:53:46.592 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:46 smithi070 ceph-mon[29589]: pgmap v13: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:46.592 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:46 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:46.592 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:46 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:53:46.592 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:46 smithi070 ceph-mon[41533]: pgmap v13: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:46.592 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:46 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:46.592 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:46 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:53:47.128 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:53:47 smithi070 podman[71923]: 2024-09-21 09:53:46.998024901 +0000 UTC m=+0.120597646 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-21T09:53:47.553 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:53:47 smithi070 podman[71923]: 2024-09-21 09:53:47.219524252 +0000 UTC m=+0.342096993 volume create be72394d0c22878060752511a3a96a5774cba527f65f9897d14a74d5f49eaf36 2024-09-21T09:53:47.553 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:53:47 smithi070 podman[71923]: 2024-09-21 09:53:47.311665293 +0000 UTC m=+0.434238097 container create 714e53dbe751756c09eacf78ae1c83c8ab35e4edd3f5647e4bc46596a86faac9 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:53:47.554 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:47 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/3111200407' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-21T09:53:47.554 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:47 smithi070 ceph-mon[29589]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-21T09:53:47.554 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:47 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/3111200407' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-21T09:53:47.554 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:47 smithi070 ceph-mon[41533]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-21T09:53:47.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:47 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/3111200407' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-21T09:53:47.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:47 smithi160 ceph-mon[33203]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-21T09:53:48.127 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:53:47 smithi070 podman[71923]: 2024-09-21 09:53:47.873011058 +0000 UTC m=+0.995583801 container init 714e53dbe751756c09eacf78ae1c83c8ab35e4edd3f5647e4bc46596a86faac9 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:53:48.127 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:53:47 smithi070 podman[71923]: 2024-09-21 09:53:47.878774515 +0000 UTC m=+1.001347254 container start 714e53dbe751756c09eacf78ae1c83c8ab35e4edd3f5647e4bc46596a86faac9 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:53:48.127 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:53:47 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[71967]: ts=2024-09-21T09:53:47.891Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-21T09:53:48.127 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:53:47 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[71967]: ts=2024-09-21T09:53:47.891Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-21T09:53:48.128 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:53:47 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[71967]: ts=2024-09-21T09:53:47.893Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.70 port=9094 2024-09-21T09:53:48.128 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:53:47 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[71967]: ts=2024-09-21T09:53:47.894Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-21T09:53:48.128 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:53:47 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[71967]: ts=2024-09-21T09:53:47.918Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T09:53:48.128 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:53:47 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[71967]: ts=2024-09-21T09:53:47.919Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T09:53:48.128 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:53:47 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[71967]: ts=2024-09-21T09:53:47.921Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-21T09:53:48.128 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:53:47 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[71967]: ts=2024-09-21T09:53:47.921Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-21T09:53:48.128 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:53:47 smithi070 bash[71923]: 714e53dbe751756c09eacf78ae1c83c8ab35e4edd3f5647e4bc46596a86faac9 2024-09-21T09:53:48.128 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:53:47 smithi070 systemd[1]: Started Ceph alertmanager.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:53:48.564 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:48 smithi070 ceph-mon[29589]: pgmap v14: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:48.564 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:48 smithi070 ceph-mon[29589]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-09-21T09:53:48.564 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:48 smithi070 ceph-mon[29589]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T09:53:48.564 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:48 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:48.564 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:48 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:48.564 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:48 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:48.564 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:48 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:48.565 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:48 smithi070 ceph-mon[41533]: pgmap v14: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:48.565 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:48 smithi070 ceph-mon[41533]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-09-21T09:53:48.565 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:48 smithi070 ceph-mon[41533]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T09:53:48.565 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:48 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:48.565 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:48 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:48.565 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:48 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:48.565 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:48 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:48 smithi160 ceph-mon[33203]: pgmap v14: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:48 smithi160 ceph-mon[33203]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-09-21T09:53:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:48 smithi160 ceph-mon[33203]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T09:53:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:48 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:48 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:48 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:48.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:48 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:48.877 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:48 smithi070 systemd[1]: Starting Ceph node-exporter.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:53:49.212 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:48 smithi070 bash[72235]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-21T09:53:49.571 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:49 smithi070 ceph-mon[29589]: Deploying daemon node-exporter.a on smithi070 2024-09-21T09:53:49.572 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:49 smithi070 ceph-mon[29589]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T09:53:49.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:49 smithi070 ceph-mon[41533]: Deploying daemon node-exporter.a on smithi070 2024-09-21T09:53:49.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:49 smithi070 ceph-mon[41533]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T09:53:49.572 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:49 smithi070 bash[72235]: Getting image source signatures 2024-09-21T09:53:49.572 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:49 smithi070 bash[72235]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-21T09:53:49.572 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:49 smithi070 bash[72235]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-21T09:53:49.572 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:49 smithi070 bash[72235]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-21T09:53:49.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:49 smithi160 ceph-mon[33203]: Deploying daemon node-exporter.a on smithi070 2024-09-21T09:53:49.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:49 smithi160 ceph-mon[33203]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T09:53:50.166 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:53:49 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[71967]: ts=2024-09-21T09:53:49.895Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000842383s 2024-09-21T09:53:50.186 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph config log 1 --format=json 2024-09-21T09:53:50.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:50 smithi160 ceph-mon[33203]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:50.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:50 smithi160 ceph-mon[33203]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:53:50.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:50 smithi160 ceph-mon[33203]: Cluster is now healthy 2024-09-21T09:53:50.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:50 smithi160 ceph-mon[33203]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T09:53:50.382 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:53:50.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:50 smithi070 ceph-mon[29589]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:50.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:50 smithi070 ceph-mon[29589]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:53:50.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:50 smithi070 ceph-mon[29589]: Cluster is now healthy 2024-09-21T09:53:50.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:50 smithi070 ceph-mon[29589]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T09:53:50.503 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:50 smithi070 ceph-mon[41533]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:50.504 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:50 smithi070 ceph-mon[41533]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:53:50.504 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:50 smithi070 ceph-mon[41533]: Cluster is now healthy 2024-09-21T09:53:50.504 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:50 smithi070 ceph-mon[41533]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T09:53:51.399 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:51 smithi070 bash[72235]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-21T09:53:51.681 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:51 smithi070 bash[72235]: Writing manifest to image destination 2024-09-21T09:53:52.214 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-mon[29589]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:52.214 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:52.215 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-mon[41533]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:52.215 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:52.215 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:51 smithi070 podman[72235]: 2024-09-21 09:53:51.959455491 +0000 UTC m=+3.123662256 container create 2f7c5b1e36ebf02e4400257f3911224013acf9f8aa91aaac458e35202a04ad1a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T09:53:52.215 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:51 smithi070 podman[72235]: 2024-09-21 09:53:51.886171164 +0000 UTC m=+3.050377945 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-21T09:53:52.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:52 smithi160 ceph-mon[33203]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:53:52.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:52 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:52.877 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 podman[72235]: 2024-09-21 09:53:52.666004752 +0000 UTC m=+3.830211518 container init 2f7c5b1e36ebf02e4400257f3911224013acf9f8aa91aaac458e35202a04ad1a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T09:53:52.877 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 podman[72235]: 2024-09-21 09:53:52.672344811 +0000 UTC m=+3.836551584 container start 2f7c5b1e36ebf02e4400257f3911224013acf9f8aa91aaac458e35202a04ad1a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T09:53:52.878 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.676Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-21T09:53:52.878 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.676Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-21T09:53:52.878 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.676Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-21T09:53:52.878 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.676Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-21T09:53:52.878 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-21T09:53:52.878 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-21T09:53:52.878 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-21T09:53:52.878 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=arp 2024-09-21T09:53:52.878 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-21T09:53:52.878 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-21T09:53:52.878 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-21T09:53:52.878 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-21T09:53:52.878 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-21T09:53:52.879 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-21T09:53:52.879 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-21T09:53:52.879 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-21T09:53:52.879 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=edac 2024-09-21T09:53:52.879 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-21T09:53:52.879 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-21T09:53:52.879 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-21T09:53:52.880 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-21T09:53:52.880 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-21T09:53:52.880 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-21T09:53:52.880 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-21T09:53:52.880 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-21T09:53:52.880 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-21T09:53:52.880 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-21T09:53:52.880 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-21T09:53:52.880 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-21T09:53:52.880 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-21T09:53:52.880 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-21T09:53:52.880 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-21T09:53:52.880 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-21T09:53:52.880 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=os 2024-09-21T09:53:52.881 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-21T09:53:52.881 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-21T09:53:52.881 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-21T09:53:52.881 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-21T09:53:52.881 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-21T09:53:52.881 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-21T09:53:52.881 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-21T09:53:52.881 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=stat 2024-09-21T09:53:52.881 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-21T09:53:52.881 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-21T09:53:52.881 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-21T09:53:52.881 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=time 2024-09-21T09:53:52.881 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-21T09:53:52.881 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=uname 2024-09-21T09:53:52.882 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-21T09:53:52.882 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-21T09:53:52.882 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.677Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-21T09:53:52.882 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.678Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-21T09:53:52.882 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[72452]: ts=2024-09-21T09:53:52.678Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-21T09:53:52.882 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 bash[72235]: 2f7c5b1e36ebf02e4400257f3911224013acf9f8aa91aaac458e35202a04ad1a 2024-09-21T09:53:52.882 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:53:52 smithi070 systemd[1]: Started Ceph node-exporter.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:53:53.401 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:53:53.809 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:53 smithi160 bash[64141]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-21T09:53:53.956 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:53 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:53.956 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:53 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:53.956 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:53 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:53.956 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:53 smithi070 ceph-mon[29589]: Deploying daemon node-exporter.b on smithi160 2024-09-21T09:53:53.957 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:53 smithi070 ceph-mon[29589]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-09-21T09:53:53.957 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:53 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/2966973665' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-21T09:53:53.957 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:53 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:53.957 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:53 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:53.957 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:53 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:53.957 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:53 smithi070 ceph-mon[41533]: Deploying daemon node-exporter.b on smithi160 2024-09-21T09:53:53.957 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:53 smithi070 ceph-mon[41533]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-09-21T09:53:53.957 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:53 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/2966973665' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-21T09:53:53.977 INFO:teuthology.orchestra.run.smithi070.stdout:[{"version":21,"timestamp":"2024-09-21T09:53:36.047419+0000","name":"","changes":[{"name":"osd/host:smithi160/osd_memory_target","previous_value":"4479548313","new_value":"4211112857"}]}] 2024-09-21T09:53:53.977 INFO:tasks.ceph_manager:config epoch is 21 2024-09-21T09:53:53.978 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T09:53:53.978 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-21T09:53:53.978 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph mgr dump --format=json 2024-09-21T09:53:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:53 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:53 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:53 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:53 smithi160 ceph-mon[33203]: Deploying daemon node-exporter.b on smithi160 2024-09-21T09:53:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:53 smithi160 ceph-mon[33203]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-09-21T09:53:54.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:53 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/2966973665' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-21T09:53:54.176 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:53:54.585 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:54 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:53:54] "GET /metrics HTTP/1.1" 200 28557 "" "Prometheus/2.43.0" 2024-09-21T09:53:54.591 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:54 smithi160 bash[64141]: Getting image source signatures 2024-09-21T09:53:54.591 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:54 smithi160 bash[64141]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-21T09:53:54.591 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:54 smithi160 bash[64141]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-21T09:53:54.591 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:54 smithi160 bash[64141]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-21T09:53:55.356 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:55 smithi070 ceph-mon[29589]: osdmap e66: 8 total, 8 up, 8 in 2024-09-21T09:53:55.356 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:55 smithi070 ceph-mon[41533]: osdmap e66: 8 total, 8 up, 8 in 2024-09-21T09:53:55.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:55 smithi160 ceph-mon[33203]: osdmap e66: 8 total, 8 up, 8 in 2024-09-21T09:53:55.887 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:55 smithi160 bash[64141]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-21T09:53:55.887 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:55 smithi160 bash[64141]: Writing manifest to image destination 2024-09-21T09:53:56.084 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:53:56.232 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 podman[64141]: 2024-09-21 09:53:56.082564265 +0000 UTC m=+2.476980181 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-21T09:53:56.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:56 smithi070 ceph-mon[29589]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 294 B/s wr, 0 op/s 2024-09-21T09:53:56.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:56 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/2533216337' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T09:53:56.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:56 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:56.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:56 smithi070 ceph-mon[41533]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 294 B/s wr, 0 op/s 2024-09-21T09:53:56.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:56 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/2533216337' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T09:53:56.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:56 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:56.551 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-mon[33203]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 294 B/s wr, 0 op/s 2024-09-21T09:53:56.551 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/2533216337' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T09:53:56.551 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:56.551 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 podman[64141]: 2024-09-21 09:53:56.231436103 +0000 UTC m=+2.625852061 container create e0fa2f0d997571dc65be2942ff3dfe519314ddfbca6146db71abf769f37dc4aa (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:53:56.552 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 podman[64141]: 2024-09-21 09:53:56.441636948 +0000 UTC m=+2.836052863 container init e0fa2f0d997571dc65be2942ff3dfe519314ddfbca6146db71abf769f37dc4aa (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:53:56.552 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 podman[64141]: 2024-09-21 09:53:56.446827385 +0000 UTC m=+2.841243302 container start e0fa2f0d997571dc65be2942ff3dfe519314ddfbca6146db71abf769f37dc4aa (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:53:56.552 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.450Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-21T09:53:56.552 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.450Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-21T09:53:56.552 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.451Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-21T09:53:56.552 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.451Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-21T09:53:56.552 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.451Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-21T09:53:56.552 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.451Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-21T09:53:56.552 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-21T09:53:56.552 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=arp 2024-09-21T09:53:56.552 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-21T09:53:56.552 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-21T09:53:56.553 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-21T09:53:56.553 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-21T09:53:56.553 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-21T09:53:56.553 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-21T09:53:56.553 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-21T09:53:56.553 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-21T09:53:56.553 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=edac 2024-09-21T09:53:56.553 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-21T09:53:56.554 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-21T09:53:56.554 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-21T09:53:56.554 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-21T09:53:56.554 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-21T09:53:56.554 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-21T09:53:56.554 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-21T09:53:56.554 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-21T09:53:56.554 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-21T09:53:56.554 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-21T09:53:56.554 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-21T09:53:56.554 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-21T09:53:56.554 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-21T09:53:56.554 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-21T09:53:56.555 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-21T09:53:56.555 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-21T09:53:56.555 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=os 2024-09-21T09:53:56.555 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-21T09:53:56.555 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-21T09:53:56.555 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-21T09:53:56.555 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-21T09:53:56.555 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-21T09:53:56.555 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-21T09:53:56.555 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-21T09:53:56.555 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=stat 2024-09-21T09:53:56.555 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-21T09:53:56.555 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-21T09:53:56.555 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-21T09:53:56.556 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=time 2024-09-21T09:53:56.556 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-21T09:53:56.556 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=uname 2024-09-21T09:53:56.556 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-21T09:53:56.556 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-21T09:53:56.556 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.452Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-21T09:53:56.556 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.453Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-21T09:53:56.556 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[64214]: ts=2024-09-21T09:53:56.453Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-21T09:53:56.556 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 bash[64141]: e0fa2f0d997571dc65be2942ff3dfe519314ddfbca6146db71abf769f37dc4aa 2024-09-21T09:53:56.816 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":20,"flags":0,"active_gid":24341,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":3639085022},{"type":"v1","addr":"172.21.15.70:6801","nonce":3639085022}]},"active_addr":"172.21.15.70:6801/3639085022","active_change":"2024-09-21T09:53:31.134686+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24334,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.70:8443/","prometheus":"http://172.21.15.70:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":59,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":595094674}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":2386870039}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":2110605013}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":47418078}]}]} 2024-09-21T09:53:56.821 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-21T09:53:56.821 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T09:53:56.822 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T09:53:56.841 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:53:56 smithi160 systemd[1]: Started Ceph node-exporter.b for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:53:56.993 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:53:56] ENGINE Bus STOPPING 2024-09-21T09:53:56.993 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:53:56] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:53:57.015 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:53:57.307 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:53:56] ENGINE Bus STOPPED 2024-09-21T09:53:57.307 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:53:56] ENGINE Bus STARTING 2024-09-21T09:53:57.307 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:53:57] ENGINE Serving on http://:::9283 2024-09-21T09:53:57.307 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:53:57] ENGINE Bus STARTED 2024-09-21T09:53:57.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:57.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:57.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:57.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:57.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:57.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-mon[29589]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-21T09:53:57.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:57.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:57.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T09:53:57.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-mon[29589]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T09:53:57.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:57.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-mon[29589]: Deploying daemon grafana.a on smithi160 2024-09-21T09:53:57.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-mon[29589]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T09:53:57.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:57.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:57.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:57.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:57.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:57.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-mon[41533]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-21T09:53:57.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:57.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:57.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T09:53:57.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-mon[41533]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T09:53:57.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:57.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-mon[41533]: Deploying daemon grafana.a on smithi160 2024-09-21T09:53:57.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-mon[41533]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T09:53:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:57 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:57 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:57 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:57 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:57 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:57 smithi160 ceph-mon[33203]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-21T09:53:58.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:57 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:58.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:57 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:58.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:57 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T09:53:58.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:57 smithi160 ceph-mon[33203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T09:53:58.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:57 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:53:58.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:57 smithi160 ceph-mon[33203]: Deploying daemon grafana.a on smithi160 2024-09-21T09:53:58.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:57 smithi160 ceph-mon[33203]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T09:53:58.329 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:53:57 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[71967]: ts=2024-09-21T09:53:57.898Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003728562s 2024-09-21T09:53:59.095 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:53:59.095 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":66,"fsid":"7c68bfbc-77fe-11ef-baf6-efdc52797490","created":"2024-09-21T09:47:59.216934+0000","modified":"2024-09-21T09:53:54.275767+0000","last_up_change":"2024-09-21T09:53:01.680149+0000","last_in_change":"2024-09-21T09:52:44.345943+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:51:05.514472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:53:41.830233+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6870ff8-14eb-4110-957a-00c73645b181","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2896279311},{"type":"v1","addr":"172.21.15.70:6803","nonce":2896279311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2896279311},{"type":"v1","addr":"172.21.15.70:6805","nonce":2896279311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":2896279311},{"type":"v1","addr":"172.21.15.70:6809","nonce":2896279311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2896279311},{"type":"v1","addr":"172.21.15.70:6807","nonce":2896279311}]},"public_addr":"172.21.15.70:6803/2896279311","cluster_addr":"172.21.15.70:6805/2896279311","heartbeat_back_addr":"172.21.15.70:6809/2896279311","heartbeat_front_addr":"172.21.15.70:6807/2896279311","state":["exists","up"]},{"osd":1,"uuid":"b1349e31-7493-489f-823a-55edb866d25f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":937875216},{"type":"v1","addr":"172.21.15.70:6811","nonce":937875216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":937875216},{"type":"v1","addr":"172.21.15.70:6813","nonce":937875216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":937875216},{"type":"v1","addr":"172.21.15.70:6817","nonce":937875216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":937875216},{"type":"v1","addr":"172.21.15.70:6815","nonce":937875216}]},"public_addr":"172.21.15.70:6811/937875216","cluster_addr":"172.21.15.70:6813/937875216","heartbeat_back_addr":"172.21.15.70:6817/937875216","heartbeat_front_addr":"172.21.15.70:6815/937875216","state":["exists","up"]},{"osd":2,"uuid":"ac4496ba-a860-4b09-bb38-7f433e081541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":940406420},{"type":"v1","addr":"172.21.15.70:6819","nonce":940406420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":940406420},{"type":"v1","addr":"172.21.15.70:6821","nonce":940406420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":940406420},{"type":"v1","addr":"172.21.15.70:6825","nonce":940406420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":940406420},{"type":"v1","addr":"172.21.15.70:6823","nonce":940406420}]},"public_addr":"172.21.15.70:6819/940406420","cluster_addr":"172.21.15.70:6821/940406420","heartbeat_back_addr":"172.21.15.70:6825/940406420","heartbeat_front_addr":"172.21.15.70:6823/940406420","state":["exists","up"]},{"osd":3,"uuid":"1eedf4ce-057c-4576-ae5b-51d3595df195","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1341239112},{"type":"v1","addr":"172.21.15.70:6827","nonce":1341239112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1341239112},{"type":"v1","addr":"172.21.15.70:6829","nonce":1341239112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1341239112},{"type":"v1","addr":"172.21.15.70:6833","nonce":1341239112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1341239112},{"type":"v1","addr":"172.21.15.70:6831","nonce":1341239112}]},"public_addr":"172.21.15.70:6827/1341239112","cluster_addr":"172.21.15.70:6829/1341239112","heartbeat_back_addr":"172.21.15.70:6833/1341239112","heartbeat_front_addr":"172.21.15.70:6831/1341239112","state":["exists","up"]},{"osd":4,"uuid":"a4b728b8-97a4-478f-8a11-ba590c0213e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6801","nonce":3634505869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6803","nonce":3634505869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6807","nonce":3634505869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6805","nonce":3634505869}]},"public_addr":"172.21.15.160:6801/3634505869","cluster_addr":"172.21.15.160:6803/3634505869","heartbeat_back_addr":"172.21.15.160:6807/3634505869","heartbeat_front_addr":"172.21.15.160:6805/3634505869","state":["exists","up"]},{"osd":5,"uuid":"56915347-9254-4e94-b202-1f16d09e7946","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6809","nonce":2006621188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6811","nonce":2006621188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6815","nonce":2006621188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6813","nonce":2006621188}]},"public_addr":"172.21.15.160:6809/2006621188","cluster_addr":"172.21.15.160:6811/2006621188","heartbeat_back_addr":"172.21.15.160:6815/2006621188","heartbeat_front_addr":"172.21.15.160:6813/2006621188","state":["exists","up"]},{"osd":6,"uuid":"79e6de8c-b78b-40e2-9229-0275a873d40e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6817","nonce":1949075400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6819","nonce":1949075400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6823","nonce":1949075400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6821","nonce":1949075400}]},"public_addr":"172.21.15.160:6817/1949075400","cluster_addr":"172.21.15.160:6819/1949075400","heartbeat_back_addr":"172.21.15.160:6823/1949075400","heartbeat_front_addr":"172.21.15.160:6821/1949075400","state":["exists","up"]},{"osd":7,"uuid":"72245c8a-2fdb-4445-b378-16569f337fcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6825","nonce":601673919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6827","nonce":601673919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6831","nonce":601673919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6829","nonce":601673919}]},"public_addr":"172.21.15.160:6825/601673919","cluster_addr":"172.21.15.160:6827/601673919","heartbeat_back_addr":"172.21.15.160:6831/601673919","heartbeat_front_addr":"172.21.15.160:6829/601673919","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:15.134515+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:39.191791+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:51:02.934581+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:51:26.514393+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:51:48.329249+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:12.268576+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:36.370859+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:59.888857+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/1905911777":"2024-09-22T09:53:31.134530+0000","172.21.15.70:0/2521911561":"2024-09-22T09:48:32.256720+0000","172.21.15.70:6800/2142025682":"2024-09-22T09:48:55.219103+0000","172.21.15.70:6800/177993484":"2024-09-22T09:48:32.256720+0000","172.21.15.70:6801/177993484":"2024-09-22T09:48:32.256720+0000","172.21.15.70:0/1012316198":"2024-09-22T09:48:32.256720+0000","172.21.15.70:0/4150778424":"2024-09-22T09:48:32.256720+0000","172.21.15.70:0/749465179":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/1066794139":"2024-09-22T09:53:31.134530+0000","172.21.15.70:0/3159615643":"2024-09-22T09:48:55.219103+0000","172.21.15.70:6801/1520336634":"2024-09-22T09:53:31.134530+0000","172.21.15.70:6801/2142025682":"2024-09-22T09:48:55.219103+0000","172.21.15.70:6800/1520336634":"2024-09-22T09:53:31.134530+0000","172.21.15.70:0/1469701388":"2024-09-22T09:53:31.134530+0000","172.21.15.70:0/2918782890":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/2141342771":"2024-09-22T09:53:31.134530+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T09:53:59.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:53:59 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/586104111' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:53:59.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:53:59 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/586104111' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:53:59.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:53:59 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/586104111' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:53:59.790 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T09:53:59.790 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T09:53:59.983 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:54:00.328 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:00 smithi070 ceph-mon[29589]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 206 B/s wr, 0 op/s 2024-09-21T09:54:00.329 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:00 smithi070 ceph-mon[41533]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 206 B/s wr, 0 op/s 2024-09-21T09:54:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:00 smithi160 ceph-mon[33203]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 206 B/s wr, 0 op/s 2024-09-21T09:54:01.682 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:54:01.682 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":66,"fsid":"7c68bfbc-77fe-11ef-baf6-efdc52797490","created":"2024-09-21T09:47:59.216934+0000","modified":"2024-09-21T09:53:54.275767+0000","last_up_change":"2024-09-21T09:53:01.680149+0000","last_in_change":"2024-09-21T09:52:44.345943+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:51:05.514472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:53:41.830233+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6870ff8-14eb-4110-957a-00c73645b181","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2896279311},{"type":"v1","addr":"172.21.15.70:6803","nonce":2896279311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2896279311},{"type":"v1","addr":"172.21.15.70:6805","nonce":2896279311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":2896279311},{"type":"v1","addr":"172.21.15.70:6809","nonce":2896279311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2896279311},{"type":"v1","addr":"172.21.15.70:6807","nonce":2896279311}]},"public_addr":"172.21.15.70:6803/2896279311","cluster_addr":"172.21.15.70:6805/2896279311","heartbeat_back_addr":"172.21.15.70:6809/2896279311","heartbeat_front_addr":"172.21.15.70:6807/2896279311","state":["exists","up"]},{"osd":1,"uuid":"b1349e31-7493-489f-823a-55edb866d25f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":937875216},{"type":"v1","addr":"172.21.15.70:6811","nonce":937875216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":937875216},{"type":"v1","addr":"172.21.15.70:6813","nonce":937875216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":937875216},{"type":"v1","addr":"172.21.15.70:6817","nonce":937875216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":937875216},{"type":"v1","addr":"172.21.15.70:6815","nonce":937875216}]},"public_addr":"172.21.15.70:6811/937875216","cluster_addr":"172.21.15.70:6813/937875216","heartbeat_back_addr":"172.21.15.70:6817/937875216","heartbeat_front_addr":"172.21.15.70:6815/937875216","state":["exists","up"]},{"osd":2,"uuid":"ac4496ba-a860-4b09-bb38-7f433e081541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":940406420},{"type":"v1","addr":"172.21.15.70:6819","nonce":940406420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":940406420},{"type":"v1","addr":"172.21.15.70:6821","nonce":940406420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":940406420},{"type":"v1","addr":"172.21.15.70:6825","nonce":940406420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":940406420},{"type":"v1","addr":"172.21.15.70:6823","nonce":940406420}]},"public_addr":"172.21.15.70:6819/940406420","cluster_addr":"172.21.15.70:6821/940406420","heartbeat_back_addr":"172.21.15.70:6825/940406420","heartbeat_front_addr":"172.21.15.70:6823/940406420","state":["exists","up"]},{"osd":3,"uuid":"1eedf4ce-057c-4576-ae5b-51d3595df195","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1341239112},{"type":"v1","addr":"172.21.15.70:6827","nonce":1341239112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1341239112},{"type":"v1","addr":"172.21.15.70:6829","nonce":1341239112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1341239112},{"type":"v1","addr":"172.21.15.70:6833","nonce":1341239112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1341239112},{"type":"v1","addr":"172.21.15.70:6831","nonce":1341239112}]},"public_addr":"172.21.15.70:6827/1341239112","cluster_addr":"172.21.15.70:6829/1341239112","heartbeat_back_addr":"172.21.15.70:6833/1341239112","heartbeat_front_addr":"172.21.15.70:6831/1341239112","state":["exists","up"]},{"osd":4,"uuid":"a4b728b8-97a4-478f-8a11-ba590c0213e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6801","nonce":3634505869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6803","nonce":3634505869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6807","nonce":3634505869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6805","nonce":3634505869}]},"public_addr":"172.21.15.160:6801/3634505869","cluster_addr":"172.21.15.160:6803/3634505869","heartbeat_back_addr":"172.21.15.160:6807/3634505869","heartbeat_front_addr":"172.21.15.160:6805/3634505869","state":["exists","up"]},{"osd":5,"uuid":"56915347-9254-4e94-b202-1f16d09e7946","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6809","nonce":2006621188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6811","nonce":2006621188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6815","nonce":2006621188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6813","nonce":2006621188}]},"public_addr":"172.21.15.160:6809/2006621188","cluster_addr":"172.21.15.160:6811/2006621188","heartbeat_back_addr":"172.21.15.160:6815/2006621188","heartbeat_front_addr":"172.21.15.160:6813/2006621188","state":["exists","up"]},{"osd":6,"uuid":"79e6de8c-b78b-40e2-9229-0275a873d40e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6817","nonce":1949075400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6819","nonce":1949075400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6823","nonce":1949075400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6821","nonce":1949075400}]},"public_addr":"172.21.15.160:6817/1949075400","cluster_addr":"172.21.15.160:6819/1949075400","heartbeat_back_addr":"172.21.15.160:6823/1949075400","heartbeat_front_addr":"172.21.15.160:6821/1949075400","state":["exists","up"]},{"osd":7,"uuid":"72245c8a-2fdb-4445-b378-16569f337fcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6825","nonce":601673919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6827","nonce":601673919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6831","nonce":601673919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6829","nonce":601673919}]},"public_addr":"172.21.15.160:6825/601673919","cluster_addr":"172.21.15.160:6827/601673919","heartbeat_back_addr":"172.21.15.160:6831/601673919","heartbeat_front_addr":"172.21.15.160:6829/601673919","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:15.134515+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:39.191791+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:51:02.934581+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:51:26.514393+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:51:48.329249+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:12.268576+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:36.370859+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:59.888857+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/1905911777":"2024-09-22T09:53:31.134530+0000","172.21.15.70:0/2521911561":"2024-09-22T09:48:32.256720+0000","172.21.15.70:6800/2142025682":"2024-09-22T09:48:55.219103+0000","172.21.15.70:6800/177993484":"2024-09-22T09:48:32.256720+0000","172.21.15.70:6801/177993484":"2024-09-22T09:48:32.256720+0000","172.21.15.70:0/1012316198":"2024-09-22T09:48:32.256720+0000","172.21.15.70:0/4150778424":"2024-09-22T09:48:32.256720+0000","172.21.15.70:0/749465179":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/1066794139":"2024-09-22T09:53:31.134530+0000","172.21.15.70:0/3159615643":"2024-09-22T09:48:55.219103+0000","172.21.15.70:6801/1520336634":"2024-09-22T09:53:31.134530+0000","172.21.15.70:6801/2142025682":"2024-09-22T09:48:55.219103+0000","172.21.15.70:6800/1520336634":"2024-09-22T09:53:31.134530+0000","172.21.15.70:0/1469701388":"2024-09-22T09:53:31.134530+0000","172.21.15.70:0/2918782890":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/2141342771":"2024-09-22T09:53:31.134530+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T09:54:02.409 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T09:54:02.409 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T09:54:02.411 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T09:54:02.412 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T09:54:02.413 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T09:54:02.414 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T09:54:02.414 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T09:54:02.415 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T09:54:02.454 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:02 smithi070 ceph-mon[29589]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 204 B/s wr, 0 op/s 2024-09-21T09:54:02.455 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:02 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:02.455 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:02 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:54:02.455 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:02 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:02.455 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:02 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/4221854811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:54:02.456 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:02 smithi070 ceph-mon[41533]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 204 B/s wr, 0 op/s 2024-09-21T09:54:02.456 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:02 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:02.456 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:02 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:54:02.456 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:02 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:02.456 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:02 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/4221854811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:54:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:02 smithi160 ceph-mon[33203]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 204 B/s wr, 0 op/s 2024-09-21T09:54:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:02 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:02.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:02 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:54:02.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:02 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:02.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:02 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/4221854811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:54:02.868 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:54:02.880 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:54:02.881 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:54:02.881 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:54:02.881 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:54:02.882 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:54:02.882 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:54:02.906 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:54:04.223 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:04 smithi160 ceph-mon[33203]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:04.485 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:04 smithi070 ceph-mon[29589]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:04.486 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:04 smithi070 ceph-mon[41533]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:04.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:54:04 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:54:04] "GET /metrics HTTP/1.1" 200 28581 "" "Prometheus/2.43.0" 2024-09-21T09:54:06.338 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:06 smithi160 ceph-mon[33203]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:06.476 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:06 smithi070 ceph-mon[29589]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:06.476 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:06 smithi070 ceph-mon[41533]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:06.591 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:06 smithi160 systemd[1]: Starting Ceph grafana.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:54:07.012 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:06 smithi160 podman[64614]: 2024-09-21 09:54:06.725104409 +0000 UTC m=+0.109607508 container create 8b9850d5203c7f74df753c11ce8e2f577d266071bb6669ce07aa57a3b8ab144e (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, version=8.5, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, architecture=x86_64, release=239.1651231664, vendor=Red Hat, Inc., io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-04-29T11:48:11.055897, vcs-type=git, io.buildah.version=1.32.0) 2024-09-21T09:54:07.012 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:06 smithi160 podman[64614]: 2024-09-21 09:54:06.630620747 +0000 UTC m=+0.015123856 image pull 954c08fa618887fe9a6fe572ae541197c77e625b05f4193df7898f5ac649ada9 quay.io/ceph/ceph-grafana:9.4.7 2024-09-21T09:54:07.012 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 podman[64614]: 2024-09-21 09:54:07.010375059 +0000 UTC m=+0.394878162 container init 8b9850d5203c7f74df753c11ce8e2f577d266071bb6669ce07aa57a3b8ab144e (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.32.0, name=ubi8, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, architecture=x86_64, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, release=239.1651231664, build-date=2022-04-29T11:48:11.055897, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, maintainer=Paul Cuzner ) 2024-09-21T09:54:07.262 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 podman[64614]: 2024-09-21 09:54:07.015869416 +0000 UTC m=+0.400372518 container start 8b9850d5203c7f74df753c11ce8e2f577d266071bb6669ce07aa57a3b8ab144e (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, io.openshift.tags=base rhel8, description=Ceph Grafana Container, release=239.1651231664, distribution-scope=public, io.openshift.expose-services=, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, maintainer=Paul Cuzner , name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-04-29T11:48:11.055897, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664) 2024-09-21T09:54:07.262 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 bash[64614]: 8b9850d5203c7f74df753c11ce8e2f577d266071bb6669ce07aa57a3b8ab144e 2024-09-21T09:54:07.262 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 systemd[1]: Started Ceph grafana.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:54:07.263 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=settings t=2024-09-21T09:54:07.173490732Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-21T09:54:07.263 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=settings t=2024-09-21T09:54:07.17367398Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-21T09:54:07.263 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=settings t=2024-09-21T09:54:07.173704507Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-21T09:54:07.263 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=settings t=2024-09-21T09:54:07.17371561Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-21T09:54:07.263 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=settings t=2024-09-21T09:54:07.173721959Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-21T09:54:07.263 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=settings t=2024-09-21T09:54:07.17372816Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-21T09:54:07.263 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=settings t=2024-09-21T09:54:07.173734229Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-21T09:54:07.263 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=settings t=2024-09-21T09:54:07.173740229Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-21T09:54:07.263 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=settings t=2024-09-21T09:54:07.17374674Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-21T09:54:07.263 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=settings t=2024-09-21T09:54:07.173752675Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-21T09:54:07.264 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=settings t=2024-09-21T09:54:07.173758371Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-21T09:54:07.264 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=settings t=2024-09-21T09:54:07.173763897Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-21T09:54:07.264 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=settings t=2024-09-21T09:54:07.173769945Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-21T09:54:07.264 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=settings t=2024-09-21T09:54:07.17377589Z level=info msg="App mode production" 2024-09-21T09:54:07.264 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=sqlstore t=2024-09-21T09:54:07.173821334Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-21T09:54:07.264 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=sqlstore t=2024-09-21T09:54:07.173838378Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-21T09:54:07.264 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.174677373Z level=info msg="Starting DB migrations" 2024-09-21T09:54:07.264 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.174770437Z level=info msg="Executing migration" id="create migration_log table" 2024-09-21T09:54:07.264 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.175274404Z level=info msg="Executing migration" id="create user table" 2024-09-21T09:54:07.264 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.17625813Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-21T09:54:07.264 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.1767066Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-21T09:54:07.265 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.177140652Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-21T09:54:07.265 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.177954663Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-21T09:54:07.265 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.178387265Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-21T09:54:07.265 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.179123827Z level=info msg="Executing migration" id="create user table v2" 2024-09-21T09:54:07.265 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.179980187Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-21T09:54:07.265 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.180907888Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-21T09:54:07.265 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.181709037Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-21T09:54:07.265 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.182088617Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-21T09:54:07.265 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.18292056Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-21T09:54:07.265 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.18335818Z level=info msg="Executing migration" id="Update user table charset" 2024-09-21T09:54:07.265 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.183760581Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-21T09:54:07.265 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.184193568Z level=info msg="Executing migration" id="Add missing user data" 2024-09-21T09:54:07.265 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.184595129Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-21T09:54:07.265 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.185038176Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-21T09:54:07.266 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.185453316Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-21T09:54:07.266 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.186406394Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-21T09:54:07.266 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.187716338Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-21T09:54:07.266 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.18858571Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-21T09:54:07.266 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.191330638Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-21T09:54:07.266 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.19429866Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-21T09:54:07.266 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.197031029Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-21T09:54:07.266 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.197436065Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-21T09:54:07.266 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.197708535Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-21T09:54:07.266 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.198137339Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-21T09:54:07.266 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.198580198Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-21T09:54:07.266 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.199006634Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-21T09:54:07.266 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.19941706Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-21T09:54:07.266 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.200110173Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-21T09:54:07.267 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.200533335Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-21T09:54:07.267 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.200949833Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-21T09:54:07.267 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.201357461Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-21T09:54:07.267 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.201732708Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-21T09:54:07.267 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.202156104Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-21T09:54:07.267 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.202472952Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-21T09:54:07.268 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.202805955Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-21T09:54:07.268 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.203150152Z level=info msg="Executing migration" id="create star table" 2024-09-21T09:54:07.268 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.203505146Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-21T09:54:07.268 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.203942596Z level=info msg="Executing migration" id="create org table v1" 2024-09-21T09:54:07.268 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.204341251Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-21T09:54:07.268 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.204754841Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-21T09:54:07.268 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.20517907Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-21T09:54:07.268 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.2055903Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-21T09:54:07.268 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.206016469Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-21T09:54:07.268 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.206422856Z level=info msg="Executing migration" id="Update org table charset" 2024-09-21T09:54:07.268 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.206701734Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-21T09:54:07.268 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.207002656Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-21T09:54:07.268 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.207299664Z level=info msg="Executing migration" id="create dashboard table" 2024-09-21T09:54:07.268 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.207682752Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-21T09:54:07.269 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.208144013Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-21T09:54:07.269 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.208561285Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-21T09:54:07.269 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.208964099Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-21T09:54:07.269 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.209383727Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-21T09:54:07.270 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.209770438Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-21T09:54:07.270 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.210623527Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-21T09:54:07.270 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.211025735Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-21T09:54:07.270 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.211448927Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-21T09:54:07.270 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.211904071Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-21T09:54:07.270 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.212240936Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-21T09:54:07.270 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.212605341Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-21T09:54:07.270 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.213006176Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-21T09:54:07.270 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.213498338Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-21T09:54:07.270 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.214000622Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-21T09:54:07.271 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.214491266Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-21T09:54:07.271 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.214933939Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-21T09:54:07.271 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.215414244Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-21T09:54:07.271 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.215885871Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-21T09:54:07.271 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.216667916Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-21T09:54:07.271 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.216981903Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-21T09:54:07.271 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.217266127Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-21T09:54:07.271 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.217782473Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-21T09:54:07.271 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.218287448Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-21T09:54:07.271 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.218776998Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-21T09:54:07.271 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.219299005Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-21T09:54:07.271 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.221653266Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-21T09:54:07.271 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.222112463Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-21T09:54:07.271 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.224491262Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-21T09:54:07.272 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.224803894Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-21T09:54:07.272 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.22746646Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-21T09:54:07.272 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.227915678Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-21T09:54:07.272 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.230467902Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-21T09:54:07.272 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.233010139Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-21T09:54:07.272 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.235549158Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-21T09:54:07.272 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.236013759Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-21T09:54:07.272 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.238556843Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-21T09:54:07.272 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.241122063Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-21T09:54:07.272 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.243710808Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-21T09:54:07.272 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.246134431Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-21T09:54:07.273 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.248816759Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-21T09:54:07.273 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.251291105Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-21T09:54:07.273 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.254056788Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-21T09:54:07.273 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.254757243Z level=info msg="Executing migration" id="create data_source table" 2024-09-21T09:54:07.273 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.257400106Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-21T09:54:07.273 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.259886282Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-21T09:54:07.273 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.260329636Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-21T09:54:07.273 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.260732672Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-21T09:54:07.273 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.261444464Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-21T09:54:07.273 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.263303597Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-21T09:54:07.273 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.264118182Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-21T09:54:07.273 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.264930076Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-21T09:54:07.273 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.26549675Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-21T09:54:07.514 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.266314546Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-21T09:54:07.514 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.267175723Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-21T09:54:07.514 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.268011222Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-21T09:54:07.514 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.268366039Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-21T09:54:07.514 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.269182397Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-21T09:54:07.514 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.269735873Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-21T09:54:07.514 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.270069404Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-21T09:54:07.514 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.270917503Z level=info msg="Executing migration" id="Add uid column" 2024-09-21T09:54:07.515 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.271484267Z level=info msg="Executing migration" id="Update uid value" 2024-09-21T09:54:07.515 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.27178013Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-21T09:54:07.515 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.272218765Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-21T09:54:07.515 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.27266771Z level=info msg="Executing migration" id="create api_key table" 2024-09-21T09:54:07.515 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.273510546Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-21T09:54:07.515 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.274383097Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-21T09:54:07.515 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.274865462Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-21T09:54:07.515 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.275760494Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-21T09:54:07.515 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.276601757Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-21T09:54:07.515 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.277479973Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-21T09:54:07.515 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.27837102Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-21T09:54:07.516 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.279451136Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-21T09:54:07.516 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.280342597Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-21T09:54:07.516 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.281208701Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-21T09:54:07.516 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.282075714Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-21T09:54:07.516 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.282939122Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-21T09:54:07.516 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.283835026Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-21T09:54:07.516 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.284325868Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-21T09:54:07.517 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.285161319Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-21T09:54:07.517 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.28572724Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-21T09:54:07.517 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.286280148Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-21T09:54:07.517 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.287105872Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-21T09:54:07.517 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.287686994Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-21T09:54:07.517 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.288271488Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-21T09:54:07.517 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.289123117Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-21T09:54:07.517 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.289529473Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-21T09:54:07.517 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.292079199Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-21T09:54:07.517 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.294984519Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-21T09:54:07.517 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.297757492Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-21T09:54:07.518 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.300562234Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-21T09:54:07.518 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.303354251Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-21T09:54:07.518 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.306165806Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-21T09:54:07.518 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.30903018Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-21T09:54:07.518 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.311850126Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-21T09:54:07.518 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.314614089Z level=info msg="Executing migration" id="create quota table v1" 2024-09-21T09:54:07.518 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.317495606Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-21T09:54:07.518 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.320368274Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-21T09:54:07.518 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.323251795Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-21T09:54:07.518 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.326185687Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-21T09:54:07.518 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.329042905Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-21T09:54:07.519 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.331910957Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-21T09:54:07.519 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.334760368Z level=info msg="Executing migration" id="create session table" 2024-09-21T09:54:07.519 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.337578532Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-21T09:54:07.519 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.340134678Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-21T09:54:07.519 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.342664984Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-21T09:54:07.519 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.345204756Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-21T09:54:07.519 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.347611117Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-21T09:54:07.519 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.350097625Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-21T09:54:07.519 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.352732624Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-21T09:54:07.519 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.355395753Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-21T09:54:07.519 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.357846625Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-21T09:54:07.519 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.360412436Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-21T09:54:07.519 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.36294685Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-21T09:54:07.520 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.365665168Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-21T09:54:07.520 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.368402853Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-21T09:54:07.520 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.370995276Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-21T09:54:07.520 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.373401319Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-21T09:54:07.520 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.374405638Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-21T09:54:07.520 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.377107238Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-21T09:54:07.520 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.379852129Z level=info msg="Executing migration" id="create alert table v1" 2024-09-21T09:54:07.520 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.38262995Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-21T09:54:07.520 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.385361999Z level=info msg="Executing migration" id="add index alert state" 2024-09-21T09:54:07.520 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.388052019Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-21T09:54:07.520 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.390762943Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-21T09:54:07.520 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.393391618Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-21T09:54:07.520 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.396105513Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-21T09:54:07.520 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.39886017Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-21T09:54:07.520 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.401421835Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-21T09:54:07.521 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.404081094Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-21T09:54:07.521 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.406792733Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-21T09:54:07.521 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.409420012Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-21T09:54:07.521 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.41220641Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-21T09:54:07.522 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.414773765Z level=info msg="Executing migration" id="Add column is_default" 2024-09-21T09:54:07.522 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.417421808Z level=info msg="Executing migration" id="Add column frequency" 2024-09-21T09:54:07.522 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.420132803Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-21T09:54:07.522 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.422721127Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-21T09:54:07.522 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.42552542Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-21T09:54:07.522 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.428172896Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-21T09:54:07.522 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.43075858Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-21T09:54:07.522 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.433423363Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-21T09:54:07.522 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.436162712Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-21T09:54:07.522 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.438686026Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-21T09:54:07.522 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.441330862Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-21T09:54:07.522 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.444076983Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-21T09:54:07.522 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.446725532Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-21T09:54:07.523 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.447517628Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-21T09:54:07.523 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.448343167Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-21T09:54:07.523 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.449161305Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-21T09:54:07.523 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.44996303Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-21T09:54:07.523 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.450778673Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-21T09:54:07.523 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.451556228Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-21T09:54:07.523 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.452347759Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-21T09:54:07.523 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.453134477Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-21T09:54:07.523 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.453960878Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-21T09:54:07.523 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.454843414Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-21T09:54:07.523 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.455583992Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-21T09:54:07.523 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.456392925Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-21T09:54:07.523 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.457163692Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-21T09:54:07.524 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.457949981Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-21T09:54:07.524 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.4587467Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-21T09:54:07.524 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.459572294Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-21T09:54:07.524 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.460349019Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-21T09:54:07.524 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.461117989Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-21T09:54:07.524 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.461960729Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-21T09:54:07.524 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.462816571Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-21T09:54:07.524 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.463688455Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-21T09:54:07.524 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.464554191Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-21T09:54:07.524 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.465379321Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-21T09:54:07.524 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.46698102Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-21T09:54:07.524 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.467806792Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-21T09:54:07.525 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.46866579Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-21T09:54:07.525 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.469465166Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-21T09:54:07.525 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.470303221Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-21T09:54:07.525 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.470639688Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-21T09:54:07.525 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.47136574Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-21T09:54:07.525 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.472195882Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-21T09:54:07.525 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.473062207Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-21T09:54:07.525 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.473842881Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-21T09:54:07.525 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.474622791Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-21T09:54:07.525 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.475488326Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-21T09:54:07.525 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.478322435Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-21T09:54:07.525 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.481093817Z level=info msg="Executing migration" id="Move region to single row" 2024-09-21T09:54:07.525 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.483863353Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-21T09:54:07.526 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.486567478Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-21T09:54:07.526 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.489340402Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-21T09:54:07.526 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.49205849Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-21T09:54:07.526 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.494796355Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-21T09:54:07.526 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.497566749Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-21T09:54:07.526 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.500347963Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-21T09:54:07.526 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.500723295Z level=info msg="Executing migration" id="create test_data table" 2024-09-21T09:54:07.526 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.503315771Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-21T09:54:07.526 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.506072904Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-21T09:54:07.526 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.508798756Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-21T09:54:07.526 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.509571143Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-21T09:54:07.526 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.512247766Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-21T09:54:07.527 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.514932364Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-21T09:54:07.527 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.517547007Z level=info msg="Executing migration" id="create team table" 2024-09-21T09:54:07.765 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.518226165Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-21T09:54:07.765 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.520967058Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-21T09:54:07.766 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.523807502Z level=info msg="Executing migration" id="create team member table" 2024-09-21T09:54:07.766 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.526408508Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-21T09:54:07.766 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.526901302Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-21T09:54:07.766 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.529556628Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-21T09:54:07.766 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.532204199Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-21T09:54:07.766 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.534761975Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-21T09:54:07.766 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.535508005Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-21T09:54:07.766 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.538211371Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-21T09:54:07.766 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.540983737Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-21T09:54:07.766 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.543317456Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-21T09:54:07.766 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.543812511Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-21T09:54:07.767 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.546523575Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-21T09:54:07.767 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.549105801Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-21T09:54:07.767 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.551751578Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-21T09:54:07.767 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.554329583Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-21T09:54:07.767 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.556904646Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-21T09:54:07.767 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.557704602Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-21T09:54:07.767 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.55851828Z level=info msg="Executing migration" id="create tag table" 2024-09-21T09:54:07.767 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.559356141Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-21T09:54:07.767 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.560138159Z level=info msg="Executing migration" id="create login attempt table" 2024-09-21T09:54:07.767 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.560911138Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-21T09:54:07.767 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.561686003Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-21T09:54:07.767 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.56250011Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-21T09:54:07.768 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.564608653Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-21T09:54:07.768 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.565356729Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-21T09:54:07.768 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.566200552Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-21T09:54:07.768 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.566991097Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-21T09:54:07.768 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.567800449Z level=info msg="Executing migration" id="create user auth table" 2024-09-21T09:54:07.768 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.568569938Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-21T09:54:07.768 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.569389635Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-21T09:54:07.768 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.570207952Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-21T09:54:07.768 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.571086342Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-21T09:54:07.768 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.57208372Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-21T09:54:07.768 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.573185285Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-21T09:54:07.769 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.574344722Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-21T09:54:07.769 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.575217487Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-21T09:54:07.770 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.576382929Z level=info msg="Executing migration" id="create server_lock table" 2024-09-21T09:54:07.770 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.577209947Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-21T09:54:07.770 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.578056802Z level=info msg="Executing migration" id="create user auth token table" 2024-09-21T09:54:07.770 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.578970043Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-21T09:54:07.770 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.582387859Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-21T09:54:07.770 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.583097517Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-21T09:54:07.771 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.583956319Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-21T09:54:07.771 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.584809367Z level=info msg="Executing migration" id="create cache_data table" 2024-09-21T09:54:07.771 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.585642159Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-21T09:54:07.771 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.586548496Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-21T09:54:07.771 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.588583656Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-21T09:54:07.771 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.591127714Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-21T09:54:07.771 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.593765209Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-21T09:54:07.771 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.596508222Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-21T09:54:07.771 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.599179263Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-21T09:54:07.771 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.601574112Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-21T09:54:07.771 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.604409686Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-21T09:54:07.771 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.607073505Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-21T09:54:07.771 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.609815863Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-21T09:54:07.772 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.61226972Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-21T09:54:07.772 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.615015763Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-21T09:54:07.772 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.615722599Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-21T09:54:07.772 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.616513881Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-21T09:54:07.772 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.616955516Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-21T09:54:07.772 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.617266906Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-21T09:54:07.772 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.617690742Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-21T09:54:07.772 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.618145796Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-21T09:54:07.772 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.61857839Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-21T09:54:07.772 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.618909106Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-21T09:54:07.772 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.619333741Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-21T09:54:07.773 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.619778619Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-21T09:54:07.773 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.620273374Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-21T09:54:07.773 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.620737755Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-21T09:54:07.773 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.621521251Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-21T09:54:07.773 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.621976872Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-21T09:54:07.773 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.622406033Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-21T09:54:07.773 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.62603692Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-21T09:54:07.773 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.629525001Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-21T09:54:07.773 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.629988978Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-21T09:54:07.773 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.630423453Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-21T09:54:07.773 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.631255223Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-21T09:54:07.774 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.631717214Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-21T09:54:07.774 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.632180749Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-21T09:54:07.774 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.632636122Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-21T09:54:07.774 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.633081005Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-21T09:54:07.774 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.633401963Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-21T09:54:07.774 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.63425626Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-21T09:54:07.774 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.635190784Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-21T09:54:07.774 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.636062981Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-21T09:54:07.774 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.636530271Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-21T09:54:07.774 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.637010115Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-21T09:54:07.774 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.637861312Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-21T09:54:07.774 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.638656408Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-21T09:54:07.774 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.639157831Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-21T09:54:07.775 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.640013541Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-21T09:54:07.775 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.640842502Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-21T09:54:07.775 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.641165115Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-21T09:54:07.775 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.641612247Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-21T09:54:07.775 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.642065917Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-21T09:54:07.775 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.64252474Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-21T09:54:07.775 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.642882651Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-21T09:54:07.775 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.64375139Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-21T09:54:07.775 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.646416778Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-21T09:54:07.775 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.647291721Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-21T09:54:07.775 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.648186754Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-21T09:54:07.775 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.649070644Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-21T09:54:07.775 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.649422139Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-21T09:54:07.776 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.649846808Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-21T09:54:07.776 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.650688167Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-21T09:54:07.776 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.651021835Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-21T09:54:07.776 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.651903877Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-21T09:54:07.776 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.654877047Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-21T09:54:07.776 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.655732975Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-21T09:54:07.776 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.656152591Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-21T09:54:07.776 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.656626118Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-21T09:54:07.776 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.657546497Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-21T09:54:07.776 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.658013528Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-21T09:54:07.776 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.658506451Z level=info msg="Executing migration" id="create alert_image table" 2024-09-21T09:54:07.776 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.658945903Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-21T09:54:07.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.659373071Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-21T09:54:07.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.659662147Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-21T09:54:07.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.660085678Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-21T09:54:07.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.660491703Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-21T09:54:07.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.660579459Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-21T09:54:07.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.66088395Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-21T09:54:07.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.661368033Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-21T09:54:07.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.66219815Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-09-21T09:54:07.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.662346271Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-09-21T09:54:07.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.662467618Z level=info msg="alerts found to migrate" alerts=0 2024-09-21T09:54:07.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.662903015Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-21T09:54:07.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.663401295Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-21T09:54:07.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.664229553Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-21T09:54:07.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.665055287Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-21T09:54:07.778 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.666033539Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-21T09:54:07.778 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.666771422Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-21T09:54:07.778 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.667596659Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-21T09:54:07.778 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.668537134Z level=info msg="Executing migration" id="create data_keys table" 2024-09-21T09:54:07.778 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.669476623Z level=info msg="Executing migration" id="create secrets table" 2024-09-21T09:54:07.778 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.670232541Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-21T09:54:07.778 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.674585175Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-21T09:54:07.778 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.675510662Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-21T09:54:07.778 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.676349302Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-21T09:54:07.778 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.680622283Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-21T09:54:07.778 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.684844323Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-21T09:54:07.778 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.685888284Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-21T09:54:07.778 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.686617082Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-21T09:54:07.779 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.687597671Z level=info msg="Executing migration" id="create permission table" 2024-09-21T09:54:07.779 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.688398643Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-21T09:54:07.779 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.689329951Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-21T09:54:07.779 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.690062545Z level=info msg="Executing migration" id="create role table" 2024-09-21T09:54:07.779 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.690943989Z level=info msg="Executing migration" id="add column display_name" 2024-09-21T09:54:07.779 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.691875868Z level=info msg="Executing migration" id="add column group_name" 2024-09-21T09:54:07.779 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.692774519Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-21T09:54:07.779 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.693655375Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-21T09:54:07.779 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.694777524Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-21T09:54:07.779 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.69552552Z level=info msg="Executing migration" id="create team role table" 2024-09-21T09:54:07.779 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.696558836Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-21T09:54:07.779 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.69728883Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-21T09:54:07.779 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.69827837Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-21T09:54:07.779 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.698965942Z level=info msg="Executing migration" id="create user role table" 2024-09-21T09:54:07.780 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.699863215Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-21T09:54:07.780 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.702742913Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-21T09:54:07.780 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.705718226Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-21T09:54:07.780 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.708503386Z level=info msg="Executing migration" id="create builtin role table" 2024-09-21T09:54:07.780 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.711342837Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-21T09:54:07.780 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.714020432Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-21T09:54:07.780 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.71696486Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-21T09:54:07.780 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.719619332Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-21T09:54:07.780 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.720134816Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-21T09:54:07.780 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.722931766Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-21T09:54:07.780 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.723363714Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-21T09:54:07.780 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.723763565Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-21T09:54:07.780 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.72417676Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-21T09:54:07.781 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.726656437Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-21T09:54:07.781 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.727661532Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-21T09:54:07.781 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.728069487Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-21T09:54:07.781 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.728489711Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-21T09:54:07.781 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.728760738Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-21T09:54:07.781 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.730395328Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-21T09:54:07.781 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.730873395Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-21T09:54:07.781 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.731237968Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-21T09:54:07.781 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.731518165Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-21T09:54:07.781 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.731882592Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-21T09:54:07.781 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.732290204Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-21T09:54:07.781 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.732726934Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-21T09:54:07.781 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.733690323Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-21T09:54:07.781 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.734016948Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-21T09:54:07.782 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.734421531Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-21T09:54:07.782 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.734855326Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-21T09:54:07.782 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.735276602Z level=info msg="Executing migration" id="add correlation config column" 2024-09-21T09:54:07.782 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.736271637Z level=info msg="Executing migration" id="create entity_events table" 2024-09-21T09:54:07.782 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.7372136Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-21T09:54:07.782 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.737834455Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-21T09:54:07.782 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.73791666Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-21T09:54:07.782 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.738656727Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-21T09:54:07.782 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.738765003Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-21T09:54:07.782 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.739510809Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-21T09:54:07.782 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.740190003Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-21T09:54:07.782 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.740977493Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-21T09:54:07.782 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.741754275Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-21T09:54:07.783 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.744518753Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-21T09:54:07.783 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.747444932Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-21T09:54:07.783 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.750220471Z level=info msg="Executing migration" id="Drop public config table" 2024-09-21T09:54:07.783 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.752836452Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-21T09:54:07.783 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.755590906Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-21T09:54:07.783 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.758327738Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-21T09:54:07.783 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.761102261Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-21T09:54:07.783 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.763829925Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-21T09:54:07.783 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.766938942Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-21T09:54:08.092 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.769823462Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-21T09:54:08.092 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.772297844Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-21T09:54:08.092 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.775229046Z level=info msg="Executing migration" id="add share column" 2024-09-21T09:54:08.092 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.777900436Z level=info msg="Executing migration" id="create default alerting folders" 2024-09-21T09:54:08.092 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.780522847Z level=info msg="Executing migration" id="create file table" 2024-09-21T09:54:08.092 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.78329727Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-21T09:54:08.092 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.785955174Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-21T09:54:08.092 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.788462596Z level=info msg="Executing migration" id="create file_meta table" 2024-09-21T09:54:08.092 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.79085022Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-21T09:54:08.092 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.793526443Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-21T09:54:08.092 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.796292604Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-21T09:54:08.093 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.79891655Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-21T09:54:08.093 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.801621474Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-21T09:54:08.093 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.804288396Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-21T09:54:08.093 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.806893042Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-21T09:54:08.093 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.809741257Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-21T09:54:08.093 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.812392013Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-21T09:54:08.093 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.814936296Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-21T09:54:08.093 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.817724197Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-21T09:54:08.094 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.820285846Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-21T09:54:08.094 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.823014946Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-21T09:54:08.094 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.825638078Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-21T09:54:08.094 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.828323185Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-21T09:54:08.094 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.83850549Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-21T09:54:08.094 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.841034124Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-21T09:54:08.094 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.84368236Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-21T09:54:08.094 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.847236259Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-21T09:54:08.094 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.847988009Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-21T09:54:08.094 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.848777575Z level=info msg="Executing migration" id="create folder table" 2024-09-21T09:54:08.094 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.849504994Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-21T09:54:08.094 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.850217785Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-21T09:54:08.095 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.850936669Z level=info msg="Executing migration" id="Update folder title length" 2024-09-21T09:54:08.095 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.851687728Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-21T09:54:08.095 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=migrator t=2024-09-21T09:54:07.852470406Z level=info msg="migrations completed" performed=485 skipped=0 duration=677.725883ms 2024-09-21T09:54:08.095 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=sqlstore t=2024-09-21T09:54:07.852914891Z level=info msg="Created default organization" 2024-09-21T09:54:08.095 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=plugin.loader t=2024-09-21T09:54:07.933626012Z level=info msg="Plugin registered" pluginID=input 2024-09-21T09:54:08.095 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=plugin.loader t=2024-09-21T09:54:07.963235595Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-21T09:54:08.095 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=plugin.loader t=2024-09-21T09:54:07.963266802Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-21T09:54:08.095 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=secrets t=2024-09-21T09:54:07.963300026Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-21T09:54:08.095 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=query_data t=2024-09-21T09:54:07.964943868Z level=info msg="Query Service initialization" 2024-09-21T09:54:08.095 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:07 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=live.push_http t=2024-09-21T09:54:07.967504531Z level=info msg="Live Push Gateway initialization" 2024-09-21T09:54:08.521 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:08 smithi070 ceph-mon[29589]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:08.521 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:08 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:08.522 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:08 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:08.522 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:08 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:08.522 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:08 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:08.522 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:08 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:08.522 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:08 smithi070 ceph-mon[41533]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:08.522 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:08 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:08.523 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:08 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:08.523 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:08 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:08.523 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:08 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:08.523 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:08 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:08 smithi160 ceph-mon[33203]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:08 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:08 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:08 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:08 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:08 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:09.032 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:08 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=infra.usagestats.collector t=2024-09-21T09:54:08.714571809Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-21T09:54:09.032 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:08 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=provisioning.datasources t=2024-09-21T09:54:08.76991291Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-21T09:54:09.032 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:08 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=provisioning.datasources t=2024-09-21T09:54:08.801917646Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-09-21T09:54:09.032 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:08 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=provisioning.alerting t=2024-09-21T09:54:08.823217208Z level=info msg="starting to provision alerting" 2024-09-21T09:54:09.032 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:08 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=provisioning.alerting t=2024-09-21T09:54:08.823255224Z level=info msg="finished to provision alerting" 2024-09-21T09:54:09.032 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:08 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=grafanaStorageLogger t=2024-09-21T09:54:08.823456084Z level=info msg="storage starting" 2024-09-21T09:54:09.032 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:08 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=ngalert.state.manager t=2024-09-21T09:54:08.823464348Z level=info msg="Warming state cache for startup" 2024-09-21T09:54:09.032 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:08 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=http.server t=2024-09-21T09:54:08.827506078Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-21T09:54:09.032 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:08 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=sqlstore.transactions t=2024-09-21T09:54:08.834336991Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T09:54:09.033 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:08 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=ngalert.state.manager t=2024-09-21T09:54:08.834462036Z level=info msg="State cache has been initialized" states=0 duration=10.996574ms 2024-09-21T09:54:09.033 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:08 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=ticker t=2024-09-21T09:54:08.834538152Z level=info msg=starting first_tick=2024-09-21T09:54:10Z 2024-09-21T09:54:09.033 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:08 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=ngalert.multiorg.alertmanager t=2024-09-21T09:54:08.834531898Z level=info msg="starting MultiOrg Alertmanager" 2024-09-21T09:54:09.033 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:08 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=sqlstore.transactions t=2024-09-21T09:54:08.88951122Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T09:54:09.033 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:54:08 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[64628]: logger=sqlstore.transactions t=2024-09-21T09:54:08.90045236Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-21T09:54:10.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:10 smithi070 ceph-mon[29589]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:10.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:10 smithi070 ceph-mon[41533]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:10.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:10 smithi160 ceph-mon[33203]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:11.501 INFO:teuthology.orchestra.run.smithi070.stdout:55834574892 2024-09-21T09:54:11.502 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T09:54:12.325 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:12 smithi070 ceph-mon[29589]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:12.325 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:12 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:12.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:12 smithi070 ceph-mon[41533]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:12.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:12 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:12 smithi160 ceph-mon[33203]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:12 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:14.268 INFO:teuthology.orchestra.run.smithi070.stdout:34359738417 2024-09-21T09:54:14.268 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T09:54:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:14 smithi160 ceph-mon[33203]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:14.484 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:14 smithi070 ceph-mon[29589]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:14.484 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:14 smithi070 ceph-mon[41533]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:14.757 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:54:14 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:54:14] "GET /metrics HTTP/1.1" 200 28581 "" "Prometheus/2.43.0" 2024-09-21T09:54:15.682 INFO:teuthology.orchestra.run.smithi070.stdout:231928234000 2024-09-21T09:54:15.683 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T09:54:16.249 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:15 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:16.249 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:15 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:16.249 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:15 smithi070 ceph-mon[41533]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:16.250 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:15 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:16.250 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:15 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:16.250 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:15 smithi070 ceph-mon[29589]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:16.270 INFO:teuthology.orchestra.run.smithi070.stdout:107374182435 2024-09-21T09:54:16.270 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T09:54:16.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:15 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:16.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:15 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:16.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:15 smithi160 ceph-mon[33203]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:16.980 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:16 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:54:16.980 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:16 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:54:17.267 INFO:teuthology.orchestra.run.smithi070.stdout:128849018910 2024-09-21T09:54:17.267 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T09:54:17.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:16 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:54:17.616 INFO:teuthology.orchestra.run.smithi070.stdout:77309411369 2024-09-21T09:54:17.616 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T09:54:17.627 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:54:17.633 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:54:18.123 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:17 smithi070 ceph-mon[29589]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:18.123 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:17 smithi070 ceph-mon[41533]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:17 smithi160 ceph-mon[33203]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:20.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:20 smithi070 ceph-mon[29589]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:20.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:20 smithi070 ceph-mon[41533]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:20.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:20 smithi160 ceph-mon[33203]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:20.899 INFO:teuthology.orchestra.run.smithi070.stdout:163208757275 2024-09-21T09:54:20.900 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T09:54:21.545 INFO:teuthology.orchestra.run.smithi070.stdout:197568495638 2024-09-21T09:54:21.545 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T09:54:22.571 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:22 smithi070 ceph-mon[29589]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:22.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:22 smithi070 ceph-mon[41533]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:22.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:22 smithi160 ceph-mon[33203]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:22.601 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:54:22.603 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:54:23.379 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:54:23.386 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:54:23.861 INFO:teuthology.orchestra.run.smithi070.stdout:34359738420 2024-09-21T09:54:24.334 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:24 smithi070 ceph-mon[41533]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:24.334 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:24 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/2117155736' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T09:54:24.334 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:24 smithi070 ceph-mon[29589]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:24.334 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:24 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/2117155736' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T09:54:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:24 smithi160 ceph-mon[33203]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:24 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/2117155736' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T09:54:24.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:54:24 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:54:24] "GET /metrics HTTP/1.1" 200 28583 "" "Prometheus/2.43.0" 2024-09-21T09:54:24.709 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738417 got 34359738420 for osd.0 2024-09-21T09:54:24.709 DEBUG:teuthology.parallel:result is None 2024-09-21T09:54:25.253 INFO:teuthology.orchestra.run.smithi070.stdout:55834574895 2024-09-21T09:54:26.568 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:26 smithi070 ceph-mon[29589]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:26.568 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:26 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/2326432104' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T09:54:26.568 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:26 smithi070 ceph-mon[41533]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:26.568 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:26 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/2326432104' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T09:54:26.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:26 smithi160 ceph-mon[33203]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:26.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:26 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/2326432104' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T09:54:27.240 INFO:teuthology.orchestra.run.smithi070.stdout:107374182438 2024-09-21T09:54:28.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:28 smithi070 ceph-mon[41533]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:28.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:28 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/3051245309' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T09:54:28.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:28 smithi070 ceph-mon[29589]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:28.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:28 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/3051245309' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T09:54:28.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:28 smithi160 ceph-mon[33203]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:28.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:28 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/3051245309' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T09:54:28.785 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:54:28.785 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:54:29.194 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574892 got 55834574895 for osd.1 2024-09-21T09:54:29.195 DEBUG:teuthology.parallel:result is None 2024-09-21T09:54:29.846 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182435 got 107374182438 for osd.3 2024-09-21T09:54:29.847 DEBUG:teuthology.parallel:result is None 2024-09-21T09:54:30.579 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:30 smithi070 ceph-mon[29589]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:30.580 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:30 smithi070 ceph-mon[41533]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:30 smithi160 ceph-mon[33203]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:31.218 INFO:teuthology.orchestra.run.smithi070.stdout:128849018913 2024-09-21T09:54:31.304 INFO:teuthology.orchestra.run.smithi070.stdout:231928234003 2024-09-21T09:54:31.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:31 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:54:31.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:31 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/1205961301' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T09:54:31.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:31 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:54:31.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:31 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/1205961301' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T09:54:31.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:31 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:54:31.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:31 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/1205961301' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T09:54:32.268 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018910 got 128849018913 for osd.4 2024-09-21T09:54:32.269 DEBUG:teuthology.parallel:result is None 2024-09-21T09:54:32.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:32 smithi160 ceph-mon[33203]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:32.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:32 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/2118819579' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T09:54:32.599 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:32 smithi070 ceph-mon[41533]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:32.599 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:32 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/2118819579' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T09:54:32.600 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:32 smithi070 ceph-mon[29589]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:32.600 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:32 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/2118819579' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T09:54:33.261 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234000 got 231928234003 for osd.7 2024-09-21T09:54:33.261 DEBUG:teuthology.parallel:result is None 2024-09-21T09:54:34.484 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:34 smithi070 ceph-mon[41533]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:34.485 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:34 smithi070 ceph-mon[29589]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:34.526 INFO:teuthology.orchestra.run.smithi070.stdout:77309411373 2024-09-21T09:54:34.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:34 smithi160 ceph-mon[33203]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:34.878 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:54:34 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:54:34] "GET /metrics HTTP/1.1" 200 28580 "" "Prometheus/2.43.0" 2024-09-21T09:54:35.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:35 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/3054058968' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T09:54:35.611 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:35 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/3054058968' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T09:54:35.611 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:35 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/3054058968' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T09:54:36.011 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411369 got 77309411373 for osd.2 2024-09-21T09:54:36.011 DEBUG:teuthology.parallel:result is None 2024-09-21T09:54:36.243 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:36 smithi070 ceph-mon[29589]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:36.244 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:36 smithi070 ceph-mon[41533]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:36 smithi160 ceph-mon[33203]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:36.935 INFO:teuthology.orchestra.run.smithi070.stdout:197568495641 2024-09-21T09:54:37.146 INFO:teuthology.orchestra.run.smithi070.stdout:163208757278 2024-09-21T09:54:37.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:37 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/3340676573' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T09:54:37.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:37 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/2202409371' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T09:54:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:37 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/3340676573' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T09:54:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:37 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/2202409371' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T09:54:37.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:37 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/3340676573' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T09:54:37.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:37 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/2202409371' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T09:54:38.075 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495638 got 197568495641 for osd.6 2024-09-21T09:54:38.075 DEBUG:teuthology.parallel:result is None 2024-09-21T09:54:38.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:38 smithi070 ceph-mon[29589]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:38.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:38 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:38.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:38 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:38.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:38 smithi070 ceph-mon[41533]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:38.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:38 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:38.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:38 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:38.492 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:38 smithi160 ceph-mon[33203]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:38.493 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:38 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:38.493 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:38 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:38.587 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757275 got 163208757278 for osd.5 2024-09-21T09:54:38.588 DEBUG:teuthology.parallel:result is None 2024-09-21T09:54:38.588 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T09:54:38.588 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T09:54:39.262 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:39 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:39.262 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:39 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:54:39.262 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:39 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:39.262 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:39 smithi070 ceph-mon[29589]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T09:54:39.262 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:39 smithi070 ceph-mon[29589]: Reconfiguring daemon alertmanager.a on smithi070 2024-09-21T09:54:39.262 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:39 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:39.262 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:39 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:54:39.263 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:39 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:39.263 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:39 smithi070 ceph-mon[41533]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T09:54:39.263 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:39 smithi070 ceph-mon[41533]: Reconfiguring daemon alertmanager.a on smithi070 2024-09-21T09:54:39.515 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:54:39.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:39 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:39.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:39 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:54:39.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:39 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:39.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:39 smithi160 ceph-mon[33203]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T09:54:39.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:39 smithi160 ceph-mon[33203]: Reconfiguring daemon alertmanager.a on smithi070 2024-09-21T09:54:40.566 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:40 smithi070 ceph-mon[29589]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:40.566 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:40 smithi070 ceph-mon[41533]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:40.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:40 smithi160 ceph-mon[33203]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:42.583 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:42 smithi070 ceph-mon[29589]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:42.583 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:54:42 smithi070 systemd[1]: Stopping Ceph alertmanager.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:54:42.584 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:42 smithi070 ceph-mon[41533]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:42 smithi160 ceph-mon[33203]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:43.052 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:54:43.052 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T09:54:43.260 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:54:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[71967]: ts=2024-09-21T09:54:43.048Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-21T09:54:43.542 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:54:43 smithi070 podman[77505]: 2024-09-21 09:54:43.25871693 +0000 UTC m=+0.338713716 container died 714e53dbe751756c09eacf78ae1c83c8ab35e4edd3f5647e4bc46596a86faac9 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:54:43.877 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:54:43 smithi070 podman[77505]: 2024-09-21 09:54:43.647926637 +0000 UTC m=+0.727923380 container cleanup 714e53dbe751756c09eacf78ae1c83c8ab35e4edd3f5647e4bc46596a86faac9 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:54:43.877 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:54:43 smithi070 bash[77505]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a 2024-09-21T09:54:44.055 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-09-21T09:54:41.157288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":181,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131,"ondisk_log_size":131,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":227252,"kb_used_data":6996,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":749733964,"statfs":{"total":767960285184,"available":767727579136,"internally_reserved":0,"allocated":7163904,"data_stored":4035574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12741,"internal_metadata":225431099},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001697"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:53:55.012487+0000","last_change":"2024-09-21T09:53:49.220984+0000","last_active":"2024-09-21T09:53:55.012487+0000","last_peered":"2024-09-21T09:53:55.012487+0000","last_clean":"2024-09-21T09:53:55.012487+0000","last_became_active":"2024-09-21T09:53:43.190004+0000","last_became_peered":"2024-09-21T09:53:43.190004+0000","last_unstale":"2024-09-21T09:53:55.012487+0000","last_undegraded":"2024-09-21T09:53:55.012487+0000","last_fullsized":"2024-09-21T09:53:55.012487+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:26:21.202894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:53:54.278381+0000","last_change":"2024-09-21T09:53:49.211757+0000","last_active":"2024-09-21T09:53:54.278381+0000","last_peered":"2024-09-21T09:53:54.278381+0000","last_clean":"2024-09-21T09:53:54.278381+0000","last_became_active":"2024-09-21T09:53:43.197526+0000","last_became_peered":"2024-09-21T09:53:43.197526+0000","last_unstale":"2024-09-21T09:53:54.278381+0000","last_undegraded":"2024-09-21T09:53:54.278381+0000","last_fullsized":"2024-09-21T09:53:54.278381+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:23:27.608603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035589600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:53:55.012790+0000","last_change":"2024-09-21T09:53:49.374824+0000","last_active":"2024-09-21T09:53:55.012790+0000","last_peered":"2024-09-21T09:53:55.012790+0000","last_clean":"2024-09-21T09:53:55.012790+0000","last_became_active":"2024-09-21T09:53:43.197315+0000","last_became_peered":"2024-09-21T09:53:43.197315+0000","last_unstale":"2024-09-21T09:53:55.012790+0000","last_undegraded":"2024-09-21T09:53:55.012790+0000","last_fullsized":"2024-09-21T09:53:55.012790+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:49:20.376464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:53:54.278346+0000","last_change":"2024-09-21T09:53:49.211745+0000","last_active":"2024-09-21T09:53:54.278346+0000","last_peered":"2024-09-21T09:53:54.278346+0000","last_clean":"2024-09-21T09:53:54.278346+0000","last_became_active":"2024-09-21T09:53:43.189852+0000","last_became_peered":"2024-09-21T09:53:43.189852+0000","last_unstale":"2024-09-21T09:53:54.278346+0000","last_undegraded":"2024-09-21T09:53:54.278346+0000","last_fullsized":"2024-09-21T09:53:54.278346+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:16.083394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":22,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:53:55.012402+0000","last_change":"2024-09-21T09:53:49.219893+0000","last_active":"2024-09-21T09:53:55.012402+0000","last_peered":"2024-09-21T09:53:55.012402+0000","last_clean":"2024-09-21T09:53:55.012402+0000","last_became_active":"2024-09-21T09:53:43.189073+0000","last_became_peered":"2024-09-21T09:53:43.189073+0000","last_unstale":"2024-09-21T09:53:55.012402+0000","last_undegraded":"2024-09-21T09:53:55.012402+0000","last_fullsized":"2024-09-21T09:53:55.012402+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:48.257597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084477399999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:53:55.012242+0000","last_change":"2024-09-21T09:53:49.211722+0000","last_active":"2024-09-21T09:53:55.012242+0000","last_peered":"2024-09-21T09:53:55.012242+0000","last_clean":"2024-09-21T09:53:55.012242+0000","last_became_active":"2024-09-21T09:53:43.196820+0000","last_became_peered":"2024-09-21T09:53:43.196820+0000","last_unstale":"2024-09-21T09:53:55.012242+0000","last_undegraded":"2024-09-21T09:53:55.012242+0000","last_fullsized":"2024-09-21T09:53:55.012242+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:24.782773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:53:55.012940+0000","last_change":"2024-09-21T09:53:49.374842+0000","last_active":"2024-09-21T09:53:55.012940+0000","last_peered":"2024-09-21T09:53:55.012940+0000","last_clean":"2024-09-21T09:53:55.012940+0000","last_became_active":"2024-09-21T09:53:43.191101+0000","last_became_peered":"2024-09-21T09:53:43.191101+0000","last_unstale":"2024-09-21T09:53:55.012940+0000","last_undegraded":"2024-09-21T09:53:55.012940+0000","last_fullsized":"2024-09-21T09:53:55.012940+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:24:33.044234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":21,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:53:55.012415+0000","last_change":"2024-09-21T09:53:49.219350+0000","last_active":"2024-09-21T09:53:55.012415+0000","last_peered":"2024-09-21T09:53:55.012415+0000","last_clean":"2024-09-21T09:53:55.012415+0000","last_became_active":"2024-09-21T09:53:43.189467+0000","last_became_peered":"2024-09-21T09:53:43.189467+0000","last_unstale":"2024-09-21T09:53:55.012415+0000","last_undegraded":"2024-09-21T09:53:55.012415+0000","last_fullsized":"2024-09-21T09:53:55.012415+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:37.943058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"59'128","reported_seq":261,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:53:49.374628+0000","last_change":"2024-09-21T09:53:05.275142+0000","last_active":"2024-09-21T09:53:49.374628+0000","last_peered":"2024-09-21T09:53:49.374628+0000","last_clean":"2024-09-21T09:53:49.374628+0000","last_became_active":"2024-09-21T09:53:05.274635+0000","last_became_peered":"2024-09-21T09:53:05.274635+0000","last_unstale":"2024-09-21T09:53:49.374628+0000","last_undegraded":"2024-09-21T09:53:49.374628+0000","last_fullsized":"2024-09-21T09:53:49.374628+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:31.035143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234005,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28256,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716896,"statfs":{"total":95995035648,"available":95966101504,"internally_reserved":0,"allocated":1265664,"data_stored":873461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:54:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.64100000000000001}]},{"osd":1,"last update":"Sat Sep 21 09:54:03 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.68000000000000005}]},{"osd":2,"last update":"Sat Sep 21 09:54:03 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.628}]},{"osd":3,"last update":"Sat Sep 21 09:54:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.65400000000000003}]},{"osd":4,"last update":"Sat Sep 21 09:54:03 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.61099999999999999}]},{"osd":5,"last update":"Sat Sep 21 09:54:03 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.504},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.59699999999999998}]},{"osd":6,"last update":"Sat Sep 21 09:54:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.54600000000000004}]}]},{"osd":6,"up_from":46,"seq":197568495642,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28260,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716892,"statfs":{"total":95995035648,"available":95966097408,"internally_reserved":0,"allocated":1269760,"data_stored":873480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.55000000000000004}]},{"osd":1,"last update":"Sat Sep 21 09:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.50800000000000001}]},{"osd":2,"last update":"Sat Sep 21 09:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.433},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.65500000000000003}]},{"osd":3,"last update":"Sat Sep 21 09:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.59499999999999997}]},{"osd":4,"last update":"Sat Sep 21 09:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56899999999999995}]},{"osd":5,"last update":"Sat Sep 21 09:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.40999999999999998}]},{"osd":7,"last update":"Sat Sep 21 09:54:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.626}]}]},{"osd":5,"up_from":38,"seq":163208757279,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":282781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:54:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52000000000000002,"15min":0.51900000000000002},"min":{"1min":0.34200000000000003,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.54900000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.79200000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.54800000000000004}]},{"osd":1,"last update":"Sat Sep 21 09:54:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56699999999999995,"15min":0.57299999999999995},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.74199999999999999,"5min":0.755,"15min":0.755},"last":0.49099999999999999}]},{"osd":2,"last update":"Sat Sep 21 09:54:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.55600000000000005,"15min":0.55200000000000005},"min":{"1min":0.373,"5min":0.31,"15min":0.31},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55900000000000005,"15min":0.55700000000000005},"min":{"1min":0.40899999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.79600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.56399999999999995}]},{"osd":3,"last update":"Sat Sep 21 09:54:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54800000000000004,"15min":0.55000000000000004},"min":{"1min":0.34799999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72499999999999998,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.435},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.42599999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80200000000000005,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.59499999999999997}]},{"osd":4,"last update":"Sat Sep 21 09:54:17 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.40400000000000003,"15min":0.39600000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.504,"5min":0.41999999999999998,"15min":0.40600000000000003},"min":{"1min":0.30599999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.46899999999999997}]},{"osd":6,"last update":"Sat Sep 21 09:53:40 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.58099999999999996}]},{"osd":7,"last update":"Sat Sep 21 09:54:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.41899999999999998}]}]},{"osd":4,"up_from":30,"seq":128849018915,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":283093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:53:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.499,"15min":0.495},"min":{"1min":0.35199999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.64500000000000002,"5min":0.755,"15min":0.755},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.55000000000000004,"15min":0.54300000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.65400000000000003}]},{"osd":1,"last update":"Sat Sep 21 09:53:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.67900000000000005,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.52500000000000002,"15min":0.51900000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.54900000000000004}]},{"osd":2,"last update":"Sat Sep 21 09:53:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.53800000000000003,"15min":0.53300000000000003},"min":{"1min":0.38700000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.75,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.498},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.51600000000000001,"15min":0.51000000000000001},"min":{"1min":0.42499999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.53600000000000003}]},{"osd":3,"last update":"Sat Sep 21 09:53:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.53200000000000003,"15min":0.52400000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.51700000000000002,"15min":0.51100000000000001},"min":{"1min":0.36499999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.59999999999999998}]},{"osd":5,"last update":"Sat Sep 21 09:54:19 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.375,"15min":0.36299999999999999},"min":{"1min":0.29799999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.377,"15min":0.36699999999999999},"min":{"1min":0.29999999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.628}]},{"osd":6,"last update":"Sat Sep 21 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.52200000000000002}]},{"osd":7,"last update":"Sat Sep 21 09:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.48199999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182440,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":283093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.35899999999999999,"15min":0.36899999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.436,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.39900000000000002,"15min":0.40500000000000003},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.46500000000000002,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.34100000000000003}]},{"osd":1,"last update":"Sat Sep 21 09:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.40899999999999997,"15min":0.41599999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.43099999999999999,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.372,"15min":0.378},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.45200000000000001,"5min":0.52400000000000002,"15min":0.52400000000000002},"last":0.32500000000000001}]},{"osd":2,"last update":"Sat Sep 21 09:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.40300000000000002,"15min":0.41099999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.42699999999999999,"5min":0.627,"15min":0.627},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.375,"15min":0.377},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.48199999999999998,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.377}]},{"osd":4,"last update":"Sat Sep 21 09:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.53400000000000003,"15min":0.52500000000000002},"min":{"1min":0.442,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":1.0580000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.51700000000000002,"15min":0.50600000000000001},"min":{"1min":0.36799999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.63400000000000001}]},{"osd":5,"last update":"Sat Sep 21 09:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54900000000000004,"15min":0.55200000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.70199999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":1.2210000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56200000000000006,"15min":0.56699999999999995},"min":{"1min":0.42799999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.70899999999999996}]},{"osd":6,"last update":"Sat Sep 21 09:53:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":1.1200000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":1.236}]},{"osd":7,"last update":"Sat Sep 21 09:54:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":1.1499999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31772,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713380,"statfs":{"total":95995035648,"available":95962501120,"internally_reserved":0,"allocated":671744,"data_stored":283093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:54:12 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.38600000000000001,"15min":0.372},"min":{"1min":0.252,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.55700000000000005,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.442}]},{"osd":1,"last update":"Sat Sep 21 09:54:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.39500000000000002,"15min":0.39100000000000001},"min":{"1min":0.28000000000000003,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41199999999999998,"15min":0.40400000000000003},"min":{"1min":0.29999999999999999,"5min":0.251,"15min":0.251},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.47199999999999998}]},{"osd":3,"last update":"Sat Sep 21 09:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.42899999999999999,"15min":0.432},"min":{"1min":0.28699999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.73099999999999998,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.41799999999999998,"15min":0.42599999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.48099999999999998}]},{"osd":4,"last update":"Sat Sep 21 09:53:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.56200000000000006,"15min":0.56399999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.81899999999999995,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.57099999999999995,"15min":0.56499999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.749}]},{"osd":5,"last update":"Sat Sep 21 09:54:16 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.629,"15min":0.63600000000000001},"min":{"1min":0.47299999999999998,"5min":0.443,"15min":0.443},"max":{"1min":0.81399999999999995,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60199999999999998,"15min":0.60399999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.71199999999999997}]},{"osd":6,"last update":"Sat Sep 21 09:53:40 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.72599999999999998}]},{"osd":7,"last update":"Sat Sep 21 09:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.76600000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27680,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717472,"statfs":{"total":95995035648,"available":95966691328,"internally_reserved":0,"allocated":675840,"data_stored":283112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:53:45 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.32600000000000001,"15min":0.314},"min":{"1min":0.246,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.46500000000000002,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.34300000000000003,"15min":0.32100000000000001},"min":{"1min":0.249,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.51100000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.27800000000000002}]},{"osd":2,"last update":"Sat Sep 21 09:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.34399999999999997,"15min":0.33900000000000002},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.44700000000000001,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.38300000000000001,"15min":0.38100000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.42699999999999999,"5min":0.623,"15min":0.623},"last":0.34999999999999998}]},{"osd":3,"last update":"Sat Sep 21 09:53:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.41299999999999998,"15min":0.41499999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.40899999999999997,"15min":0.41099999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.29799999999999999}]},{"osd":4,"last update":"Sat Sep 21 09:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.54500000000000004,"15min":0.54800000000000004},"min":{"1min":0.40999999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.64500000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.629},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54800000000000004,"15min":0.55100000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.69399999999999995,"5min":1.022,"15min":1.022},"last":0.61399999999999999}]},{"osd":5,"last update":"Sat Sep 21 09:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.59099999999999997,"15min":0.60299999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66800000000000004,"5min":1,"15min":1},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.59199999999999997,"15min":0.60299999999999998},"min":{"1min":0.42399999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.69899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.65400000000000003}]},{"osd":6,"last update":"Sat Sep 21 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.69799999999999995}]},{"osd":7,"last update":"Sat Sep 21 09:54:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.63900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738423,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28256,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716896,"statfs":{"total":95995035648,"available":95966101504,"internally_reserved":0,"allocated":1265664,"data_stored":873461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 09:53:44 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.312,"15min":0.29199999999999998},"min":{"1min":0.22900000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.44500000000000001,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.246},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.34300000000000003,"15min":0.317},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.32400000000000001}]},{"osd":2,"last update":"Sat Sep 21 09:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.378,"15min":0.35699999999999998},"min":{"1min":0.28799999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.35699999999999998,"15min":0.34200000000000003},"min":{"1min":0.26900000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.30599999999999999}]},{"osd":3,"last update":"Sat Sep 21 09:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.42899999999999999,"15min":0.435},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.41599999999999998,"15min":0.41899999999999998},"min":{"1min":0.27900000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.35899999999999999}]},{"osd":4,"last update":"Sat Sep 21 09:53:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55700000000000005,"15min":0.55900000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80000000000000004,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.81999999999999995,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.46700000000000003}]},{"osd":5,"last update":"Sat Sep 21 09:54:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56200000000000006,"15min":0.56299999999999994},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.78900000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.83299999999999996,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.54800000000000004}]},{"osd":6,"last update":"Sat Sep 21 09:53:40 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.502}]},{"osd":7,"last update":"Sat Sep 21 09:54:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.55600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T09:54:44.057 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T09:54:44.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:44 smithi070 ceph-mon[29589]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:54:44.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:44 smithi070 ceph-mon[29589]: pgmap v46: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:44.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:44 smithi070 ceph-mon[41533]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:54:44.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:44 smithi070 ceph-mon[41533]: pgmap v46: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:44.590 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:54:44.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:44 smithi160 ceph-mon[33203]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:54:44.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:44 smithi160 ceph-mon[33203]: pgmap v46: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:44.712 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:54:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:54:44] "GET /metrics HTTP/1.1" 200 28580 "" "Prometheus/2.43.0" 2024-09-21T09:54:44.712 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:54:44 smithi070 podman[77536]: 2024-09-21 09:54:44.410383154 +0000 UTC m=+1.325755546 container remove 714e53dbe751756c09eacf78ae1c83c8ab35e4edd3f5647e4bc46596a86faac9 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:54:44.712 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:54:44 smithi070 podman[77536]: 2024-09-21 09:54:44.494253361 +0000 UTC m=+1.409625750 volume remove be72394d0c22878060752511a3a96a5774cba527f65f9897d14a74d5f49eaf36 2024-09-21T09:54:44.712 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:54:44 smithi070 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-21T09:54:44.712 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:54:44 smithi070 systemd[1]: Stopped Ceph alertmanager.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:54:44.712 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:54:44 smithi070 systemd[1]: Starting Ceph alertmanager.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:54:45.763 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:54:45 smithi070 podman[77751]: 2024-09-21 09:54:45.32893246 +0000 UTC m=+0.017216901 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-21T09:54:45.763 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:54:45 smithi070 podman[77751]: 2024-09-21 09:54:45.761472394 +0000 UTC m=+0.449756821 volume create b9885b62cbf488fd51528b952d815a5f97ee369f2308150c7e2164eef109418d 2024-09-21T09:54:46.049 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:54:45 smithi070 podman[77751]: 2024-09-21 09:54:45.845110748 +0000 UTC m=+0.533395172 container create df28d76f17955124c71a93e1ee8ece1dc111a8336fe256b388d8134370bd271d (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:54:46.385 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:46 smithi070 ceph-mon[29589]: pgmap v47: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:46.385 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:46 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:54:46.385 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:46 smithi070 ceph-mon[41533]: pgmap v47: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:46.386 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:46 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:54:46.386 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:54:46 smithi070 podman[77751]: 2024-09-21 09:54:46.366180227 +0000 UTC m=+1.054464663 container init df28d76f17955124c71a93e1ee8ece1dc111a8336fe256b388d8134370bd271d (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:54:46.386 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:54:46 smithi070 podman[77751]: 2024-09-21 09:54:46.370963889 +0000 UTC m=+1.059248317 container start df28d76f17955124c71a93e1ee8ece1dc111a8336fe256b388d8134370bd271d (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:54:46.386 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:54:46 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[77792]: ts=2024-09-21T09:54:46.383Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-21T09:54:46.386 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:54:46 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[77792]: ts=2024-09-21T09:54:46.383Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-21T09:54:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:46 smithi160 ceph-mon[33203]: pgmap v47: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:46 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:54:46.652 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:54:46 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[77792]: ts=2024-09-21T09:54:46.384Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.70 port=9094 2024-09-21T09:54:46.652 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:54:46 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[77792]: ts=2024-09-21T09:54:46.385Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-21T09:54:46.653 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:54:46 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[77792]: ts=2024-09-21T09:54:46.408Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T09:54:46.653 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:54:46 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[77792]: ts=2024-09-21T09:54:46.409Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T09:54:46.653 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:54:46 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[77792]: ts=2024-09-21T09:54:46.411Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-21T09:54:46.653 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:54:46 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[77792]: ts=2024-09-21T09:54:46.411Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-21T09:54:46.653 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:54:46 smithi070 bash[77751]: df28d76f17955124c71a93e1ee8ece1dc111a8336fe256b388d8134370bd271d 2024-09-21T09:54:46.653 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:54:46 smithi070 systemd[1]: Started Ceph alertmanager.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:54:46.815 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:54:46.815 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T09:54:47.552 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":47,"stamp":"2024-09-21T09:54:45.157825+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":181,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131,"ondisk_log_size":131,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":227252,"kb_used_data":6996,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":749733964,"statfs":{"total":767960285184,"available":767727579136,"internally_reserved":0,"allocated":7163904,"data_stored":4035574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12741,"internal_metadata":225431099},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001676"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:53:55.012487+0000","last_change":"2024-09-21T09:53:49.220984+0000","last_active":"2024-09-21T09:53:55.012487+0000","last_peered":"2024-09-21T09:53:55.012487+0000","last_clean":"2024-09-21T09:53:55.012487+0000","last_became_active":"2024-09-21T09:53:43.190004+0000","last_became_peered":"2024-09-21T09:53:43.190004+0000","last_unstale":"2024-09-21T09:53:55.012487+0000","last_undegraded":"2024-09-21T09:53:55.012487+0000","last_fullsized":"2024-09-21T09:53:55.012487+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:26:21.202894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:53:54.278381+0000","last_change":"2024-09-21T09:53:49.211757+0000","last_active":"2024-09-21T09:53:54.278381+0000","last_peered":"2024-09-21T09:53:54.278381+0000","last_clean":"2024-09-21T09:53:54.278381+0000","last_became_active":"2024-09-21T09:53:43.197526+0000","last_became_peered":"2024-09-21T09:53:43.197526+0000","last_unstale":"2024-09-21T09:53:54.278381+0000","last_undegraded":"2024-09-21T09:53:54.278381+0000","last_fullsized":"2024-09-21T09:53:54.278381+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:23:27.608603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035589600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:53:55.012790+0000","last_change":"2024-09-21T09:53:49.374824+0000","last_active":"2024-09-21T09:53:55.012790+0000","last_peered":"2024-09-21T09:53:55.012790+0000","last_clean":"2024-09-21T09:53:55.012790+0000","last_became_active":"2024-09-21T09:53:43.197315+0000","last_became_peered":"2024-09-21T09:53:43.197315+0000","last_unstale":"2024-09-21T09:53:55.012790+0000","last_undegraded":"2024-09-21T09:53:55.012790+0000","last_fullsized":"2024-09-21T09:53:55.012790+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:49:20.376464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:53:54.278346+0000","last_change":"2024-09-21T09:53:49.211745+0000","last_active":"2024-09-21T09:53:54.278346+0000","last_peered":"2024-09-21T09:53:54.278346+0000","last_clean":"2024-09-21T09:53:54.278346+0000","last_became_active":"2024-09-21T09:53:43.189852+0000","last_became_peered":"2024-09-21T09:53:43.189852+0000","last_unstale":"2024-09-21T09:53:54.278346+0000","last_undegraded":"2024-09-21T09:53:54.278346+0000","last_fullsized":"2024-09-21T09:53:54.278346+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:16.083394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":22,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:53:55.012402+0000","last_change":"2024-09-21T09:53:49.219893+0000","last_active":"2024-09-21T09:53:55.012402+0000","last_peered":"2024-09-21T09:53:55.012402+0000","last_clean":"2024-09-21T09:53:55.012402+0000","last_became_active":"2024-09-21T09:53:43.189073+0000","last_became_peered":"2024-09-21T09:53:43.189073+0000","last_unstale":"2024-09-21T09:53:55.012402+0000","last_undegraded":"2024-09-21T09:53:55.012402+0000","last_fullsized":"2024-09-21T09:53:55.012402+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:48.257597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084477399999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:53:55.012242+0000","last_change":"2024-09-21T09:53:49.211722+0000","last_active":"2024-09-21T09:53:55.012242+0000","last_peered":"2024-09-21T09:53:55.012242+0000","last_clean":"2024-09-21T09:53:55.012242+0000","last_became_active":"2024-09-21T09:53:43.196820+0000","last_became_peered":"2024-09-21T09:53:43.196820+0000","last_unstale":"2024-09-21T09:53:55.012242+0000","last_undegraded":"2024-09-21T09:53:55.012242+0000","last_fullsized":"2024-09-21T09:53:55.012242+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:24.782773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:53:55.012940+0000","last_change":"2024-09-21T09:53:49.374842+0000","last_active":"2024-09-21T09:53:55.012940+0000","last_peered":"2024-09-21T09:53:55.012940+0000","last_clean":"2024-09-21T09:53:55.012940+0000","last_became_active":"2024-09-21T09:53:43.191101+0000","last_became_peered":"2024-09-21T09:53:43.191101+0000","last_unstale":"2024-09-21T09:53:55.012940+0000","last_undegraded":"2024-09-21T09:53:55.012940+0000","last_fullsized":"2024-09-21T09:53:55.012940+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:24:33.044234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":21,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:53:55.012415+0000","last_change":"2024-09-21T09:53:49.219350+0000","last_active":"2024-09-21T09:53:55.012415+0000","last_peered":"2024-09-21T09:53:55.012415+0000","last_clean":"2024-09-21T09:53:55.012415+0000","last_became_active":"2024-09-21T09:53:43.189467+0000","last_became_peered":"2024-09-21T09:53:43.189467+0000","last_unstale":"2024-09-21T09:53:55.012415+0000","last_undegraded":"2024-09-21T09:53:55.012415+0000","last_fullsized":"2024-09-21T09:53:55.012415+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:37.943058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"59'128","reported_seq":261,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:53:49.374628+0000","last_change":"2024-09-21T09:53:05.275142+0000","last_active":"2024-09-21T09:53:49.374628+0000","last_peered":"2024-09-21T09:53:49.374628+0000","last_clean":"2024-09-21T09:53:49.374628+0000","last_became_active":"2024-09-21T09:53:05.274635+0000","last_became_peered":"2024-09-21T09:53:05.274635+0000","last_unstale":"2024-09-21T09:53:49.374628+0000","last_undegraded":"2024-09-21T09:53:49.374628+0000","last_fullsized":"2024-09-21T09:53:49.374628+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:31.035143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234006,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28256,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716896,"statfs":{"total":95995035648,"available":95966101504,"internally_reserved":0,"allocated":1265664,"data_stored":873461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:54:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.60199999999999998}]},{"osd":1,"last update":"Sat Sep 21 09:54:03 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.627}]},{"osd":2,"last update":"Sat Sep 21 09:54:03 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.64700000000000002}]},{"osd":3,"last update":"Sat Sep 21 09:54:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.71399999999999997}]},{"osd":4,"last update":"Sat Sep 21 09:54:03 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.49299999999999999}]},{"osd":5,"last update":"Sat Sep 21 09:54:03 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.51600000000000001}]},{"osd":6,"last update":"Sat Sep 21 09:54:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.58699999999999997}]}]},{"osd":6,"up_from":46,"seq":197568495643,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28260,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716892,"statfs":{"total":95995035648,"available":95966097408,"internally_reserved":0,"allocated":1269760,"data_stored":873480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.78900000000000003}]},{"osd":1,"last update":"Sat Sep 21 09:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.83099999999999996}]},{"osd":2,"last update":"Sat Sep 21 09:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.76400000000000001}]},{"osd":3,"last update":"Sat Sep 21 09:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.68999999999999995}]},{"osd":4,"last update":"Sat Sep 21 09:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.77900000000000003}]},{"osd":5,"last update":"Sat Sep 21 09:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.65300000000000002}]},{"osd":7,"last update":"Sat Sep 21 09:54:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.63500000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757280,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":282781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:54:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52000000000000002,"15min":0.51900000000000002},"min":{"1min":0.34200000000000003,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.54900000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.79200000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.63700000000000001}]},{"osd":1,"last update":"Sat Sep 21 09:54:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56699999999999995,"15min":0.57299999999999995},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.74199999999999999,"5min":0.755,"15min":0.755},"last":0.67300000000000004}]},{"osd":2,"last update":"Sat Sep 21 09:54:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.55600000000000005,"15min":0.55200000000000005},"min":{"1min":0.373,"5min":0.31,"15min":0.31},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55900000000000005,"15min":0.55700000000000005},"min":{"1min":0.40899999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.79600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.56999999999999995}]},{"osd":3,"last update":"Sat Sep 21 09:54:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54800000000000004,"15min":0.55000000000000004},"min":{"1min":0.34799999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72499999999999998,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.42599999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80200000000000005,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.62}]},{"osd":4,"last update":"Sat Sep 21 09:54:17 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.40400000000000003,"15min":0.39600000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.504,"5min":0.41999999999999998,"15min":0.40600000000000003},"min":{"1min":0.30599999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.377}]},{"osd":6,"last update":"Sat Sep 21 09:54:40 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.441,"15min":0.436},"min":{"1min":0.33000000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.47999999999999998,"15min":0.47499999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.76700000000000002,"5min":1.21,"15min":1.21},"last":0.49099999999999999}]},{"osd":7,"last update":"Sat Sep 21 09:54:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.50600000000000001}]}]},{"osd":4,"up_from":30,"seq":128849018916,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":283093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:53:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.499,"15min":0.495},"min":{"1min":0.35199999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.64500000000000002,"5min":0.755,"15min":0.755},"last":0.503},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.55000000000000004,"15min":0.54300000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.42699999999999999}]},{"osd":1,"last update":"Sat Sep 21 09:53:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.67900000000000005,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.52500000000000002,"15min":0.51900000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.47199999999999998}]},{"osd":2,"last update":"Sat Sep 21 09:53:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.53800000000000003,"15min":0.53300000000000003},"min":{"1min":0.38700000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.75,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.437},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.51600000000000001,"15min":0.51000000000000001},"min":{"1min":0.42499999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.48999999999999999}]},{"osd":3,"last update":"Sat Sep 21 09:53:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.53200000000000003,"15min":0.52400000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.51700000000000002,"15min":0.51100000000000001},"min":{"1min":0.36499999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.45500000000000002}]},{"osd":5,"last update":"Sat Sep 21 09:54:19 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.375,"15min":0.36299999999999999},"min":{"1min":0.29799999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.377,"15min":0.36699999999999999},"min":{"1min":0.29999999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.33200000000000002}]},{"osd":6,"last update":"Sat Sep 21 09:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.438,"15min":0.43099999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.41599999999999998,"15min":0.40999999999999998},"min":{"1min":0.318,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.376}]},{"osd":7,"last update":"Sat Sep 21 09:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.34100000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182441,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":283093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.35899999999999999,"15min":0.36899999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.436,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.39900000000000002,"15min":0.40500000000000003},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.46500000000000002,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.42699999999999999}]},{"osd":1,"last update":"Sat Sep 21 09:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.40899999999999997,"15min":0.41599999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.43099999999999999,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.376},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.372,"15min":0.378},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.45200000000000001,"5min":0.52400000000000002,"15min":0.52400000000000002},"last":0.30299999999999999}]},{"osd":2,"last update":"Sat Sep 21 09:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.40300000000000002,"15min":0.41099999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.42699999999999999,"5min":0.627,"15min":0.627},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.375,"15min":0.377},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.48199999999999998,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.34699999999999998}]},{"osd":4,"last update":"Sat Sep 21 09:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.53400000000000003,"15min":0.52500000000000002},"min":{"1min":0.442,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.51700000000000002,"15min":0.50600000000000001},"min":{"1min":0.36799999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.41399999999999998}]},{"osd":5,"last update":"Sat Sep 21 09:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54900000000000004,"15min":0.55200000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.70199999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56200000000000006,"15min":0.56699999999999995},"min":{"1min":0.42799999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.51300000000000001}]},{"osd":6,"last update":"Sat Sep 21 09:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54700000000000004,"15min":0.54300000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.43099999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.496}]},{"osd":7,"last update":"Sat Sep 21 09:54:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.436}]}]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31772,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713380,"statfs":{"total":95995035648,"available":95962501120,"internally_reserved":0,"allocated":671744,"data_stored":283093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:54:12 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.38600000000000001,"15min":0.372},"min":{"1min":0.252,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.55700000000000005,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.39500000000000002}]},{"osd":1,"last update":"Sat Sep 21 09:54:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.39500000000000002,"15min":0.39100000000000001},"min":{"1min":0.28000000000000003,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41199999999999998,"15min":0.40400000000000003},"min":{"1min":0.29999999999999999,"5min":0.251,"15min":0.251},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.46899999999999997}]},{"osd":3,"last update":"Sat Sep 21 09:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.42899999999999999,"15min":0.432},"min":{"1min":0.28699999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.73099999999999998,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.435},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.41799999999999998,"15min":0.42599999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.42099999999999999}]},{"osd":4,"last update":"Sat Sep 21 09:53:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.56200000000000006,"15min":0.56399999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.81899999999999995,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.57099999999999995,"15min":0.56499999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.52700000000000002}]},{"osd":5,"last update":"Sat Sep 21 09:54:16 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.629,"15min":0.63600000000000001},"min":{"1min":0.47299999999999998,"5min":0.443,"15min":0.443},"max":{"1min":0.81399999999999995,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60199999999999998,"15min":0.60399999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.51500000000000001}]},{"osd":6,"last update":"Sat Sep 21 09:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.63600000000000001,"15min":0.64400000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79200000000000004,"5min":1.202,"15min":1.202},"last":0.50700000000000001}]},{"osd":7,"last update":"Sat Sep 21 09:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.56000000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27680,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717472,"statfs":{"total":95995035648,"available":95966691328,"internally_reserved":0,"allocated":675840,"data_stored":283112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:53:45 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.32600000000000001,"15min":0.314},"min":{"1min":0.246,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.46500000000000002,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.34300000000000003,"15min":0.32100000000000001},"min":{"1min":0.249,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.51100000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.48499999999999999}]},{"osd":2,"last update":"Sat Sep 21 09:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.34399999999999997,"15min":0.33900000000000002},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.44700000000000001,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.38300000000000001,"15min":0.38100000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.42699999999999999,"5min":0.623,"15min":0.623},"last":0.439}]},{"osd":3,"last update":"Sat Sep 21 09:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.39800000000000002,"15min":0.40999999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.47499999999999998,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.39400000000000002,"15min":0.40600000000000003},"min":{"1min":0.246,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.48999999999999999,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.42099999999999999}]},{"osd":4,"last update":"Sat Sep 21 09:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.54500000000000004,"15min":0.54800000000000004},"min":{"1min":0.40999999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.64500000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54800000000000004,"15min":0.55100000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.69399999999999995,"5min":1.022,"15min":1.022},"last":0.69499999999999995}]},{"osd":5,"last update":"Sat Sep 21 09:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.59099999999999997,"15min":0.60299999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66800000000000004,"5min":1,"15min":1},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.59199999999999997,"15min":0.60299999999999998},"min":{"1min":0.42399999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.69899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.77300000000000002}]},{"osd":6,"last update":"Sat Sep 21 09:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.56699999999999995,"15min":0.57499999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.67100000000000004,"5min":1.038,"15min":1.038},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.40000000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.69799999999999995,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.79500000000000004}]},{"osd":7,"last update":"Sat Sep 21 09:54:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.78600000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28256,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716896,"statfs":{"total":95995035648,"available":95966101504,"internally_reserved":0,"allocated":1265664,"data_stored":873461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 09:53:44 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.312,"15min":0.29199999999999998},"min":{"1min":0.22900000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.44500000000000001,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.34300000000000003,"15min":0.317},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.46999999999999997}]},{"osd":2,"last update":"Sat Sep 21 09:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.378,"15min":0.35699999999999998},"min":{"1min":0.28799999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.35699999999999998,"15min":0.34200000000000003},"min":{"1min":0.26900000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.48699999999999999}]},{"osd":3,"last update":"Sat Sep 21 09:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.42899999999999999,"15min":0.435},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.498},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.41599999999999998,"15min":0.41899999999999998},"min":{"1min":0.27900000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.51700000000000002}]},{"osd":4,"last update":"Sat Sep 21 09:53:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55700000000000005,"15min":0.55900000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80000000000000004,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.81999999999999995,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.66000000000000003}]},{"osd":5,"last update":"Sat Sep 21 09:54:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56200000000000006,"15min":0.56299999999999994},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.78900000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.83299999999999996,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.752}]},{"osd":6,"last update":"Sat Sep 21 09:53:40 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.75900000000000001}]},{"osd":7,"last update":"Sat Sep 21 09:54:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.80200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T09:54:47.554 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T09:54:47.554 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T09:54:47.554 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-21T09:54:47.555 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph health --format=json 2024-09-21T09:54:47.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:47 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:47.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:47 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:47.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:47 smithi160 ceph-mon[33203]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:54:47.750 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:54:47.870 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:47 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:47.870 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:47 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:47.870 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:47 smithi070 ceph-mon[29589]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:54:47.870 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:47 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:47.871 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:47 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:47.871 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:47 smithi070 ceph-mon[41533]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:54:48.554 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:48 smithi160 ceph-mon[33203]: Reconfiguring daemon prometheus.a on smithi160 2024-09-21T09:54:48.554 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:48 smithi160 ceph-mon[33203]: from='client.24527 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:54:48.554 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:48 smithi160 ceph-mon[33203]: pgmap v48: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:48.877 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:54:48 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[77792]: ts=2024-09-21T09:54:48.386Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000425088s 2024-09-21T09:54:48.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:48 smithi070 ceph-mon[29589]: Reconfiguring daemon prometheus.a on smithi160 2024-09-21T09:54:48.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:48 smithi070 ceph-mon[29589]: from='client.24527 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:54:48.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:48 smithi070 ceph-mon[29589]: pgmap v48: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:48.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:48 smithi070 ceph-mon[41533]: Reconfiguring daemon prometheus.a on smithi160 2024-09-21T09:54:48.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:48 smithi070 ceph-mon[41533]: from='client.24527 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:54:48.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:48 smithi070 ceph-mon[41533]: pgmap v48: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:49.590 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:54:49.590 INFO:teuthology.orchestra.run.smithi070.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T09:54:50.016 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:49 smithi160 systemd[1]: Stopping Ceph prometheus.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:54:50.248 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-21T09:54:50.248 INFO:tasks.cephadm:Setup complete, yielding 2024-09-21T09:54:50.249 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-21T09:54:50.267 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi070.front.sepia.ceph.com 2024-09-21T09:54:50.267 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- bash -c 'ceph fs volume create foo' 2024-09-21T09:54:50.341 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:54:50.014Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T09:54:50.342 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:54:50.015Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-21T09:54:50.342 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:54:50.015Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-21T09:54:50.342 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:54:50.015Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T09:54:50.342 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:54:50.015Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-21T09:54:50.342 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:54:50.015Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T09:54:50.342 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:54:50.015Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-21T09:54:50.342 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:54:50.015Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-21T09:54:50.342 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:54:50.015Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-21T09:54:50.342 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:54:50.015Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T09:54:50.343 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:54:50.015Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-21T09:54:50.343 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[62007]: ts=2024-09-21T09:54:50.015Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-21T09:54:50.343 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:50 smithi160 podman[65529]: 2024-09-21 09:54:50.025271751 +0000 UTC m=+0.116407995 container died b6e4ec2e67697798891cc34de74d2e26e1af350be38e92ea41e72b2d0a77b480 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:54:50.460 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:54:50.581 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:50 smithi070 ceph-mon[41533]: pgmap v49: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:50.581 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:50 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/3042378086' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T09:54:50.581 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:50 smithi070 ceph-mon[29589]: pgmap v49: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:50.582 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:50 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/3042378086' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T09:54:50.665 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:50 smithi160 ceph-mon[33203]: pgmap v49: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:50.665 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:50 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/3042378086' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T09:54:50.666 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:50 smithi160 podman[65529]: 2024-09-21 09:54:50.370910677 +0000 UTC m=+0.462046919 container cleanup b6e4ec2e67697798891cc34de74d2e26e1af350be38e92ea41e72b2d0a77b480 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:54:50.666 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:50 smithi160 bash[65529]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T09:54:51.033 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:50 smithi160 podman[65548]: 2024-09-21 09:54:50.663993343 +0000 UTC m=+0.641321700 container remove b6e4ec2e67697798891cc34de74d2e26e1af350be38e92ea41e72b2d0a77b480 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:54:51.033 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:50 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-21T09:54:51.033 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:50 smithi160 systemd[1]: Stopped Ceph prometheus.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:54:51.033 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:50 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@prometheus.a.service: Consumed 1.179s CPU time. 2024-09-21T09:54:51.033 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:50 smithi160 systemd[1]: Starting Ceph prometheus.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:54:51.284 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:51 smithi160 podman[65641]: 2024-09-21 09:54:51.031591613 +0000 UTC m=+0.110722491 container create 136d3e422755ba98b78f46ff8e6a3bebc7310387de232dbcd5ccfe811fa84210 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:54:51.284 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:51 smithi160 podman[65641]: 2024-09-21 09:54:50.937659334 +0000 UTC m=+0.016790169 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-21T09:54:51.284 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:51 smithi160 podman[65641]: 2024-09-21 09:54:51.249972415 +0000 UTC m=+0.329103258 container init 136d3e422755ba98b78f46ff8e6a3bebc7310387de232dbcd5ccfe811fa84210 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:54:51.284 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:51 smithi160 podman[65641]: 2024-09-21 09:54:51.254617526 +0000 UTC m=+0.333748364 container start 136d3e422755ba98b78f46ff8e6a3bebc7310387de232dbcd5ccfe811fa84210 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:54:51.284 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:51 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:54:51.278Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-21T09:54:51.284 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:51 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:54:51.278Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-21T09:54:51.284 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:51 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:54:51.278Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi160 (none))" 2024-09-21T09:54:51.284 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:51 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:54:51.278Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-21T09:54:51.284 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:51 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:54:51.278Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-21T09:54:51.284 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:51 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:54:51.279Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-21T09:54:51.284 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:51 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:54:51.280Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-21T09:54:51.285 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:51 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:54:51.282Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-21T09:54:51.285 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:51 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:54:51.282Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-21T09:54:51.591 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:51 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:54:51.283Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-21T09:54:51.591 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:51 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:54:51.283Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.073µs 2024-09-21T09:54:51.591 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:51 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:54:51.283Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-21T09:54:51.591 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:51 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:54:51.285Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-21T09:54:51.591 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:51 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:54:51.285Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-21T09:54:51.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:51 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:54:51.285Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=20.499µs wal_replay_duration=1.976885ms wbl_replay_duration=120ns total_replay_duration=2.018068ms 2024-09-21T09:54:51.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:51 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:54:51.286Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-21T09:54:51.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:51 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:54:51.286Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-21T09:54:51.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:51 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:54:51.286Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-21T09:54:51.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:51 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:54:51.308Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=22.024117ms db_storage=742ns remote_storage=1.006µs web_handler=381ns query_engine=667ns scrape=481.42µs scrape_sd=111.215µs notify=23.136µs notify_sd=46.065µs rules=20.968206ms tracing=5.966µs 2024-09-21T09:54:51.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:51 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:54:51.308Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-21T09:54:51.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:51 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:54:51.308Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-21T09:54:51.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:51 smithi160 bash[65641]: 136d3e422755ba98b78f46ff8e6a3bebc7310387de232dbcd5ccfe811fa84210 2024-09-21T09:54:51.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:54:51 smithi160 systemd[1]: Started Ceph prometheus.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:54:51.616 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:54:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:54:51] ENGINE Bus STOPPING 2024-09-21T09:54:51.617 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:54:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:54:51] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:54:51.617 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:54:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:54:51] ENGINE Bus STOPPED 2024-09-21T09:54:51.617 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:54:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:54:51] ENGINE Bus STARTING 2024-09-21T09:54:51.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:54:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:54:51] ENGINE Serving on http://:::9283 2024-09-21T09:54:51.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:54:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:54:51] ENGINE Bus STARTED 2024-09-21T09:54:51.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:54:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:54:51] ENGINE Bus STOPPING 2024-09-21T09:54:52.230 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:54:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:54:52.230 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:54:52] ENGINE Bus STOPPED 2024-09-21T09:54:52.230 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:54:52] ENGINE Bus STARTING 2024-09-21T09:54:52.230 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:54:52] ENGINE Serving on http://:::9283 2024-09-21T09:54:52.230 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:54:52] ENGINE Bus STARTED 2024-09-21T09:54:52.479 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:54:52] ENGINE Bus STOPPING 2024-09-21T09:54:52.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[29589]: pgmap v50: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:52.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:52.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:52.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:54:52.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[29589]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:54:52.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi070.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T09:54:52.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[29589]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi070.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T09:54:52.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:52.736 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:54:52.736 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[29589]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:54:52.736 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi160.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T09:54:52.736 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[29589]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi160.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T09:54:52.736 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:52.736 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:54:52.736 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[29589]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:54:52.736 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi160.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T09:54:52.736 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[29589]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi160.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T09:54:52.736 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:52.736 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:52.736 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[41533]: pgmap v50: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:52.737 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:52.737 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:52.737 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:54:52.737 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[41533]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:54:52.737 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi070.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T09:54:52.737 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[41533]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi070.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T09:54:52.737 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:52.737 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:54:52.737 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[41533]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:54:52.737 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi160.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T09:54:52.737 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[41533]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi160.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T09:54:52.737 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:52.737 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:54:52.738 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[41533]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:54:52.738 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi160.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T09:54:52.738 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[41533]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi160.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T09:54:52.738 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:52.738 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:52 smithi160 ceph-mon[33203]: pgmap v50: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:52 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:52 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:52.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:52 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:54:52.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:52 smithi160 ceph-mon[33203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:54:52.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:52 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi070.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T09:54:52.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:52 smithi160 ceph-mon[33203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi070.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T09:54:52.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:52 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:52.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:52 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:54:52.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:52 smithi160 ceph-mon[33203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:54:52.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:52 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi160.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T09:54:52.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:52 smithi160 ceph-mon[33203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi160.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T09:54:52.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:52 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:52.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:52 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:54:52.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:52 smithi160 ceph-mon[33203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:54:52.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:52 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi160.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T09:54:52.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:52 smithi160 ceph-mon[33203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi160.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T09:54:52.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:52 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:52.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:52 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:53.127 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:54:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:54:53.127 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:54:52] ENGINE Bus STOPPED 2024-09-21T09:54:53.127 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:54:52] ENGINE Bus STARTING 2024-09-21T09:54:53.127 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:54:52] ENGINE Serving on http://:::9283 2024-09-21T09:54:53.127 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:54:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:54:52] ENGINE Bus STARTED 2024-09-21T09:54:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:53 smithi160 ceph-mon[33203]: from='client.24539 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:53 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-21T09:54:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:53 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-21T09:54:53.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:53 smithi160 ceph-mon[33203]: osdmap e67: 8 total, 8 up, 8 in 2024-09-21T09:54:53.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:53 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-21T09:54:53.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:53 smithi160 ceph-mon[33203]: pgmap v52: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:53.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:53 smithi070 ceph-mon[41533]: from='client.24539 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:53.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:53 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-21T09:54:53.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:53 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-21T09:54:53.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:53 smithi070 ceph-mon[41533]: osdmap e67: 8 total, 8 up, 8 in 2024-09-21T09:54:53.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:53 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-21T09:54:53.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:53 smithi070 ceph-mon[41533]: pgmap v52: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:53.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:53 smithi070 ceph-mon[29589]: from='client.24539 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:53.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:53 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-21T09:54:53.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:53 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-21T09:54:53.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:53 smithi070 ceph-mon[29589]: osdmap e67: 8 total, 8 up, 8 in 2024-09-21T09:54:53.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:53 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-21T09:54:53.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:53 smithi070 ceph-mon[29589]: pgmap v52: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:53.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a[29566]: 2024-09-21T09:54:53.620+0000 7f7e3149e640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T09:54:54.678 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-09-21T09:54:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:54 smithi160 ceph-mon[33203]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:54:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:54 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-21T09:54:54.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:54 smithi160 ceph-mon[33203]: osdmap e68: 8 total, 8 up, 8 in 2024-09-21T09:54:54.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:54 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-21T09:54:54.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:54 smithi160 ceph-mon[33203]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T09:54:54.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:54 smithi160 ceph-mon[33203]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T09:54:54.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:54 smithi160 ceph-mon[33203]: osdmap e69: 8 total, 8 up, 8 in 2024-09-21T09:54:54.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:54 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-21T09:54:54.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:54 smithi160 ceph-mon[33203]: fsmap foo:0 2024-09-21T09:54:54.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:54 smithi160 ceph-mon[33203]: Saving service mds.foo spec with placement count:2 2024-09-21T09:54:54.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:54 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:54.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:54 smithi070 ceph-mon[29589]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:54:54.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:54 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-21T09:54:54.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:54 smithi070 ceph-mon[29589]: osdmap e68: 8 total, 8 up, 8 in 2024-09-21T09:54:54.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:54 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-21T09:54:54.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:54 smithi070 ceph-mon[29589]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T09:54:54.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:54 smithi070 ceph-mon[29589]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T09:54:54.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:54 smithi070 ceph-mon[29589]: osdmap e69: 8 total, 8 up, 8 in 2024-09-21T09:54:54.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:54 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-21T09:54:54.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:54 smithi070 ceph-mon[29589]: fsmap foo:0 2024-09-21T09:54:54.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:54 smithi070 ceph-mon[29589]: Saving service mds.foo spec with placement count:2 2024-09-21T09:54:54.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:54 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:54.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:54 smithi070 ceph-mon[41533]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:54:54.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:54 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-21T09:54:54.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:54 smithi070 ceph-mon[41533]: osdmap e68: 8 total, 8 up, 8 in 2024-09-21T09:54:54.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:54 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-21T09:54:54.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:54 smithi070 ceph-mon[41533]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T09:54:54.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:54 smithi070 ceph-mon[41533]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T09:54:54.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:54 smithi070 ceph-mon[41533]: osdmap e69: 8 total, 8 up, 8 in 2024-09-21T09:54:54.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:54 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-21T09:54:54.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:54 smithi070 ceph-mon[41533]: fsmap foo:0 2024-09-21T09:54:54.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:54 smithi070 ceph-mon[41533]: Saving service mds.foo spec with placement count:2 2024-09-21T09:54:54.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:54 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:55.409 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:54:55.844 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:55 smithi070 ceph-mon[29589]: osdmap e70: 8 total, 8 up, 8 in 2024-09-21T09:54:55.845 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:55 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:55.845 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:55 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:55.845 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:55 smithi070 ceph-mon[29589]: pgmap v56: 73 pgs: 64 unknown, 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:55.845 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:55 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:55.845 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:55 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:55.845 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:55 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:55.845 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:55 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:54:55.845 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:55 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:55.845 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:55 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi070.gansda", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:54:55.845 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:55 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi070.gansda", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T09:54:55.845 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:55 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:55.845 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:55 smithi070 ceph-mon[29589]: Deploying daemon mds.foo.smithi070.gansda on smithi070 2024-09-21T09:54:55.846 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:55 smithi070 ceph-mon[41533]: osdmap e70: 8 total, 8 up, 8 in 2024-09-21T09:54:55.846 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:55 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:55.846 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:55 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:55.846 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:55 smithi070 ceph-mon[41533]: pgmap v56: 73 pgs: 64 unknown, 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:55.846 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:55 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:55.846 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:55 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:55.846 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:55 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:55.846 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:55 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:54:55.846 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:55 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:55.846 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:55 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi070.gansda", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:54:55.846 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:55 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi070.gansda", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T09:54:55.846 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:55 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:55.847 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:55 smithi070 ceph-mon[41533]: Deploying daemon mds.foo.smithi070.gansda on smithi070 2024-09-21T09:54:56.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:55 smithi160 ceph-mon[33203]: osdmap e70: 8 total, 8 up, 8 in 2024-09-21T09:54:56.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:55 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:56.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:55 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:56.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:55 smithi160 ceph-mon[33203]: pgmap v56: 73 pgs: 64 unknown, 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:56.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:55 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:56.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:55 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:56.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:55 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:56.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:55 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:54:56.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:55 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:54:56.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:55 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi070.gansda", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:54:56.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:55 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi070.gansda", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T09:54:56.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:55 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:56.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:55 smithi160 ceph-mon[33203]: Deploying daemon mds.foo.smithi070.gansda on smithi070 2024-09-21T09:54:56.622 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:54:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[77792]: ts=2024-09-21T09:54:56.388Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003060606s 2024-09-21T09:54:56.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:56 smithi070 ceph-mon[29589]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T09:54:56.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:56 smithi070 ceph-mon[29589]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:54:56.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:56 smithi070 ceph-mon[41533]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T09:54:56.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:56 smithi070 ceph-mon[41533]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:54:57.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:56 smithi160 ceph-mon[33203]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T09:54:57.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:56 smithi160 ceph-mon[33203]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:54:58.031 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:54:57 smithi070 ceph-mon[29589]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:58.031 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:54:57 smithi070 ceph-mon[41533]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:54:57 smithi160 ceph-mon[33203]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:54:59.127 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-09-21T09:54:59.609 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:00.320 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:00 smithi160 ceph-mon[33203]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:00.321 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:00 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:00.321 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:00 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:00.321 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:00 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:00.321 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:00 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi160.byyhfz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:55:00.321 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:00 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi160.byyhfz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T09:55:00.321 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:00 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:00.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:00 smithi070 ceph-mon[41533]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:00.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:00 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:00.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:00 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:00.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:00 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:00.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:00 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi160.byyhfz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:55:00.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:00 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi160.byyhfz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T09:55:00.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:00 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:00.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:00 smithi070 ceph-mon[29589]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:00.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:00 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:00.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:00 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:00.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:00 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:00.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:00 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi160.byyhfz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:55:00.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:00 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi160.byyhfz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T09:55:00.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:00 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:01.289 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:01 smithi160 ceph-mon[33203]: Deploying daemon mds.foo.smithi160.byyhfz on smithi160 2024-09-21T09:55:01.289 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:01 smithi160 ceph-mon[33203]: mds.? [v2:172.21.15.70:6834/1361282039,v1:172.21.15.70:6835/1361282039] up:boot 2024-09-21T09:55:01.289 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:01 smithi160 ceph-mon[33203]: daemon mds.foo.smithi070.gansda assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-21T09:55:01.289 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:01 smithi160 ceph-mon[33203]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T09:55:01.289 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:01 smithi160 ceph-mon[33203]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T09:55:01.290 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:01 smithi160 ceph-mon[33203]: Cluster is now healthy 2024-09-21T09:55:01.290 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:01 smithi160 ceph-mon[33203]: fsmap foo:0 1 up:standby 2024-09-21T09:55:01.290 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:01 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi070.gansda"}]: dispatch 2024-09-21T09:55:01.290 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:01 smithi160 ceph-mon[33203]: fsmap foo:1 {0=foo.smithi070.gansda=up:creating} 2024-09-21T09:55:01.290 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:01 smithi160 ceph-mon[33203]: daemon mds.foo.smithi070.gansda is now active in filesystem foo as rank 0 2024-09-21T09:55:01.290 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:01 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:01.526 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:01 smithi070 ceph-mon[29589]: Deploying daemon mds.foo.smithi160.byyhfz on smithi160 2024-09-21T09:55:01.526 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:01 smithi070 ceph-mon[29589]: mds.? [v2:172.21.15.70:6834/1361282039,v1:172.21.15.70:6835/1361282039] up:boot 2024-09-21T09:55:01.526 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:01 smithi070 ceph-mon[29589]: daemon mds.foo.smithi070.gansda assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-21T09:55:01.526 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:01 smithi070 ceph-mon[29589]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T09:55:01.527 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:01 smithi070 ceph-mon[29589]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T09:55:01.527 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:01 smithi070 ceph-mon[29589]: Cluster is now healthy 2024-09-21T09:55:01.527 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:01 smithi070 ceph-mon[29589]: fsmap foo:0 1 up:standby 2024-09-21T09:55:01.527 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:01 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi070.gansda"}]: dispatch 2024-09-21T09:55:01.527 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:01 smithi070 ceph-mon[29589]: fsmap foo:1 {0=foo.smithi070.gansda=up:creating} 2024-09-21T09:55:01.527 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:01 smithi070 ceph-mon[29589]: daemon mds.foo.smithi070.gansda is now active in filesystem foo as rank 0 2024-09-21T09:55:01.527 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:01 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:01.527 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:01 smithi070 ceph-mon[41533]: Deploying daemon mds.foo.smithi160.byyhfz on smithi160 2024-09-21T09:55:01.527 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:01 smithi070 ceph-mon[41533]: mds.? [v2:172.21.15.70:6834/1361282039,v1:172.21.15.70:6835/1361282039] up:boot 2024-09-21T09:55:01.527 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:01 smithi070 ceph-mon[41533]: daemon mds.foo.smithi070.gansda assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-21T09:55:01.527 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:01 smithi070 ceph-mon[41533]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T09:55:01.528 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:01 smithi070 ceph-mon[41533]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T09:55:01.528 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:01 smithi070 ceph-mon[41533]: Cluster is now healthy 2024-09-21T09:55:01.528 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:01 smithi070 ceph-mon[41533]: fsmap foo:0 1 up:standby 2024-09-21T09:55:01.528 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:01 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi070.gansda"}]: dispatch 2024-09-21T09:55:01.528 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:01 smithi070 ceph-mon[41533]: fsmap foo:1 {0=foo.smithi070.gansda=up:creating} 2024-09-21T09:55:01.528 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:01 smithi070 ceph-mon[41533]: daemon mds.foo.smithi070.gansda is now active in filesystem foo as rank 0 2024-09-21T09:55:01.528 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:01 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:02.143 INFO:teuthology.run_tasks:Running task ceph.healthy... 2024-09-21T09:55:02.151 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T09:55:02.152 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-21T09:55:02.152 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph mgr dump --format=json 2024-09-21T09:55:02.363 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:02.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:02 smithi070 ceph-mon[41533]: pgmap v60: 73 pgs: 73 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:02.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:02 smithi070 ceph-mon[41533]: mds.? [v2:172.21.15.70:6834/1361282039,v1:172.21.15.70:6835/1361282039] up:active 2024-09-21T09:55:02.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:02 smithi070 ceph-mon[41533]: fsmap foo:1 {0=foo.smithi070.gansda=up:active} 2024-09-21T09:55:02.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:02 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:02.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:02 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:02.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:02 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:02.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:02 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:02.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:02 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:02.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:02 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:02.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:02 smithi070 ceph-mon[29589]: pgmap v60: 73 pgs: 73 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:02.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:02 smithi070 ceph-mon[29589]: mds.? [v2:172.21.15.70:6834/1361282039,v1:172.21.15.70:6835/1361282039] up:active 2024-09-21T09:55:02.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:02 smithi070 ceph-mon[29589]: fsmap foo:1 {0=foo.smithi070.gansda=up:active} 2024-09-21T09:55:02.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:02 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:02.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:02 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:02.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:02 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:02.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:02 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:02.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:02 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:02.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:02 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:02 smithi160 ceph-mon[33203]: pgmap v60: 73 pgs: 73 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:02 smithi160 ceph-mon[33203]: mds.? [v2:172.21.15.70:6834/1361282039,v1:172.21.15.70:6835/1361282039] up:active 2024-09-21T09:55:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:02 smithi160 ceph-mon[33203]: fsmap foo:1 {0=foo.smithi070.gansda=up:active} 2024-09-21T09:55:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:02 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:02 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:02 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:02.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:02 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:02.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:02 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:02.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:02 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:03.549 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:03 smithi070 ceph-mon[29589]: mds.? [v2:172.21.15.160:6832/2992374343,v1:172.21.15.160:6833/2992374343] up:boot 2024-09-21T09:55:03.549 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:03 smithi070 ceph-mon[29589]: fsmap foo:1 {0=foo.smithi070.gansda=up:active} 1 up:standby 2024-09-21T09:55:03.549 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:03 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi160.byyhfz"}]: dispatch 2024-09-21T09:55:03.549 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:03 smithi070 ceph-mon[29589]: fsmap foo:1 {0=foo.smithi070.gansda=up:active} 1 up:standby 2024-09-21T09:55:03.549 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:03 smithi070 ceph-mon[41533]: mds.? [v2:172.21.15.160:6832/2992374343,v1:172.21.15.160:6833/2992374343] up:boot 2024-09-21T09:55:03.550 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:03 smithi070 ceph-mon[41533]: fsmap foo:1 {0=foo.smithi070.gansda=up:active} 1 up:standby 2024-09-21T09:55:03.550 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:03 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi160.byyhfz"}]: dispatch 2024-09-21T09:55:03.550 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:03 smithi070 ceph-mon[41533]: fsmap foo:1 {0=foo.smithi070.gansda=up:active} 1 up:standby 2024-09-21T09:55:03.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:03 smithi160 ceph-mon[33203]: mds.? [v2:172.21.15.160:6832/2992374343,v1:172.21.15.160:6833/2992374343] up:boot 2024-09-21T09:55:03.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:03 smithi160 ceph-mon[33203]: fsmap foo:1 {0=foo.smithi070.gansda=up:active} 1 up:standby 2024-09-21T09:55:03.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:03 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi160.byyhfz"}]: dispatch 2024-09-21T09:55:03.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:03 smithi160 ceph-mon[33203]: fsmap foo:1 {0=foo.smithi070.gansda=up:active} 1 up:standby 2024-09-21T09:55:04.240 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:55:04.485 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:04 smithi070 ceph-mon[29589]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s wr, 5 op/s 2024-09-21T09:55:04.485 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:04 smithi070 ceph-mon[41533]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s wr, 5 op/s 2024-09-21T09:55:04.584 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:04 smithi160 ceph-mon[33203]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s wr, 5 op/s 2024-09-21T09:55:04.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:55:04 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:55:04] "GET /metrics HTTP/1.1" 200 33486 "" "Prometheus/2.43.0" 2024-09-21T09:55:05.004 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":20,"flags":0,"active_gid":24341,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":3639085022},{"type":"v1","addr":"172.21.15.70:6801","nonce":3639085022}]},"active_addr":"172.21.15.70:6801/3639085022","active_change":"2024-09-21T09:53:31.134686+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24334,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.70:8443/","prometheus":"http://172.21.15.70:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":59,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":595094674}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":2386870039}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":2110605013}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":47418078}]}]} 2024-09-21T09:55:05.007 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-21T09:55:05.007 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T09:55:05.008 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T09:55:05.318 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:05.556 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:05 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/3961298076' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T09:55:05.556 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:05 smithi070 ceph-mon[41533]: mds.? [v2:172.21.15.70:6834/1361282039,v1:172.21.15.70:6835/1361282039] up:active 2024-09-21T09:55:05.556 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:05 smithi070 ceph-mon[41533]: fsmap foo:1 {0=foo.smithi070.gansda=up:active} 1 up:standby 2024-09-21T09:55:05.556 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:05 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/3961298076' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T09:55:05.556 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:05 smithi070 ceph-mon[29589]: mds.? [v2:172.21.15.70:6834/1361282039,v1:172.21.15.70:6835/1361282039] up:active 2024-09-21T09:55:05.556 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:05 smithi070 ceph-mon[29589]: fsmap foo:1 {0=foo.smithi070.gansda=up:active} 1 up:standby 2024-09-21T09:55:05.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:05 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/3961298076' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T09:55:05.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:05 smithi160 ceph-mon[33203]: mds.? [v2:172.21.15.70:6834/1361282039,v1:172.21.15.70:6835/1361282039] up:active 2024-09-21T09:55:05.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:05 smithi160 ceph-mon[33203]: fsmap foo:1 {0=foo.smithi070.gansda=up:active} 1 up:standby 2024-09-21T09:55:06.489 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:06 smithi070 ceph-mon[29589]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-09-21T09:55:06.489 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:06 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:06.490 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:06 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:06.490 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:06 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:06.490 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:06 smithi070 ceph-mon[41533]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-09-21T09:55:06.490 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:06 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:06.490 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:06 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:06.490 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:06 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:06 smithi160 ceph-mon[33203]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-09-21T09:55:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:06 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:06 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:06 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:07.161 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:55:07.162 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":71,"fsid":"7c68bfbc-77fe-11ef-baf6-efdc52797490","created":"2024-09-21T09:47:59.216934+0000","modified":"2024-09-21T09:54:55.624032+0000","last_up_change":"2024-09-21T09:53:01.680149+0000","last_in_change":"2024-09-21T09:52:44.345943+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:51:05.514472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:53:41.830233+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-21T09:54:52.551642+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-21T09:54:52.861791+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6870ff8-14eb-4110-957a-00c73645b181","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2896279311},{"type":"v1","addr":"172.21.15.70:6803","nonce":2896279311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2896279311},{"type":"v1","addr":"172.21.15.70:6805","nonce":2896279311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":2896279311},{"type":"v1","addr":"172.21.15.70:6809","nonce":2896279311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2896279311},{"type":"v1","addr":"172.21.15.70:6807","nonce":2896279311}]},"public_addr":"172.21.15.70:6803/2896279311","cluster_addr":"172.21.15.70:6805/2896279311","heartbeat_back_addr":"172.21.15.70:6809/2896279311","heartbeat_front_addr":"172.21.15.70:6807/2896279311","state":["exists","up"]},{"osd":1,"uuid":"b1349e31-7493-489f-823a-55edb866d25f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":937875216},{"type":"v1","addr":"172.21.15.70:6811","nonce":937875216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":937875216},{"type":"v1","addr":"172.21.15.70:6813","nonce":937875216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":937875216},{"type":"v1","addr":"172.21.15.70:6817","nonce":937875216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":937875216},{"type":"v1","addr":"172.21.15.70:6815","nonce":937875216}]},"public_addr":"172.21.15.70:6811/937875216","cluster_addr":"172.21.15.70:6813/937875216","heartbeat_back_addr":"172.21.15.70:6817/937875216","heartbeat_front_addr":"172.21.15.70:6815/937875216","state":["exists","up"]},{"osd":2,"uuid":"ac4496ba-a860-4b09-bb38-7f433e081541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":940406420},{"type":"v1","addr":"172.21.15.70:6819","nonce":940406420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":940406420},{"type":"v1","addr":"172.21.15.70:6821","nonce":940406420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":940406420},{"type":"v1","addr":"172.21.15.70:6825","nonce":940406420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":940406420},{"type":"v1","addr":"172.21.15.70:6823","nonce":940406420}]},"public_addr":"172.21.15.70:6819/940406420","cluster_addr":"172.21.15.70:6821/940406420","heartbeat_back_addr":"172.21.15.70:6825/940406420","heartbeat_front_addr":"172.21.15.70:6823/940406420","state":["exists","up"]},{"osd":3,"uuid":"1eedf4ce-057c-4576-ae5b-51d3595df195","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1341239112},{"type":"v1","addr":"172.21.15.70:6827","nonce":1341239112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1341239112},{"type":"v1","addr":"172.21.15.70:6829","nonce":1341239112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1341239112},{"type":"v1","addr":"172.21.15.70:6833","nonce":1341239112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1341239112},{"type":"v1","addr":"172.21.15.70:6831","nonce":1341239112}]},"public_addr":"172.21.15.70:6827/1341239112","cluster_addr":"172.21.15.70:6829/1341239112","heartbeat_back_addr":"172.21.15.70:6833/1341239112","heartbeat_front_addr":"172.21.15.70:6831/1341239112","state":["exists","up"]},{"osd":4,"uuid":"a4b728b8-97a4-478f-8a11-ba590c0213e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6801","nonce":3634505869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6803","nonce":3634505869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6807","nonce":3634505869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6805","nonce":3634505869}]},"public_addr":"172.21.15.160:6801/3634505869","cluster_addr":"172.21.15.160:6803/3634505869","heartbeat_back_addr":"172.21.15.160:6807/3634505869","heartbeat_front_addr":"172.21.15.160:6805/3634505869","state":["exists","up"]},{"osd":5,"uuid":"56915347-9254-4e94-b202-1f16d09e7946","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6809","nonce":2006621188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6811","nonce":2006621188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6815","nonce":2006621188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6813","nonce":2006621188}]},"public_addr":"172.21.15.160:6809/2006621188","cluster_addr":"172.21.15.160:6811/2006621188","heartbeat_back_addr":"172.21.15.160:6815/2006621188","heartbeat_front_addr":"172.21.15.160:6813/2006621188","state":["exists","up"]},{"osd":6,"uuid":"79e6de8c-b78b-40e2-9229-0275a873d40e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6817","nonce":1949075400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6819","nonce":1949075400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6823","nonce":1949075400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6821","nonce":1949075400}]},"public_addr":"172.21.15.160:6817/1949075400","cluster_addr":"172.21.15.160:6819/1949075400","heartbeat_back_addr":"172.21.15.160:6823/1949075400","heartbeat_front_addr":"172.21.15.160:6821/1949075400","state":["exists","up"]},{"osd":7,"uuid":"72245c8a-2fdb-4445-b378-16569f337fcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6825","nonce":601673919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6827","nonce":601673919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6831","nonce":601673919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6829","nonce":601673919}]},"public_addr":"172.21.15.160:6825/601673919","cluster_addr":"172.21.15.160:6827/601673919","heartbeat_back_addr":"172.21.15.160:6831/601673919","heartbeat_front_addr":"172.21.15.160:6829/601673919","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:15.134515+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:39.191791+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:51:02.934581+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:51:26.514393+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:51:48.329249+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:12.268576+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:36.370859+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:59.888857+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/1905911777":"2024-09-22T09:53:31.134530+0000","172.21.15.70:0/2521911561":"2024-09-22T09:48:32.256720+0000","172.21.15.70:6800/2142025682":"2024-09-22T09:48:55.219103+0000","172.21.15.70:6800/177993484":"2024-09-22T09:48:32.256720+0000","172.21.15.70:6801/177993484":"2024-09-22T09:48:32.256720+0000","172.21.15.70:0/1012316198":"2024-09-22T09:48:32.256720+0000","172.21.15.70:0/4150778424":"2024-09-22T09:48:32.256720+0000","172.21.15.70:0/749465179":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/1066794139":"2024-09-22T09:53:31.134530+0000","172.21.15.70:0/3159615643":"2024-09-22T09:48:55.219103+0000","172.21.15.70:6801/1520336634":"2024-09-22T09:53:31.134530+0000","172.21.15.70:6801/2142025682":"2024-09-22T09:48:55.219103+0000","172.21.15.70:6800/1520336634":"2024-09-22T09:53:31.134530+0000","172.21.15.70:0/1469701388":"2024-09-22T09:53:31.134530+0000","172.21.15.70:0/2918782890":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/2141342771":"2024-09-22T09:53:31.134530+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T09:55:07.693 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T09:55:07.693 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T09:55:07.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:07 smithi070 ceph-mon[29589]: mds.? [v2:172.21.15.160:6832/2992374343,v1:172.21.15.160:6833/2992374343] up:standby 2024-09-21T09:55:07.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:07 smithi070 ceph-mon[29589]: fsmap foo:1 {0=foo.smithi070.gansda=up:active} 1 up:standby 2024-09-21T09:55:07.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:07 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/3201636727' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:55:07.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:07 smithi070 ceph-mon[29589]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-21T09:55:07.716 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:07 smithi070 ceph-mon[41533]: mds.? [v2:172.21.15.160:6832/2992374343,v1:172.21.15.160:6833/2992374343] up:standby 2024-09-21T09:55:07.716 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:07 smithi070 ceph-mon[41533]: fsmap foo:1 {0=foo.smithi070.gansda=up:active} 1 up:standby 2024-09-21T09:55:07.716 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:07 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/3201636727' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:55:07.716 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:07 smithi070 ceph-mon[41533]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-21T09:55:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:07 smithi160 ceph-mon[33203]: mds.? [v2:172.21.15.160:6832/2992374343,v1:172.21.15.160:6833/2992374343] up:standby 2024-09-21T09:55:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:07 smithi160 ceph-mon[33203]: fsmap foo:1 {0=foo.smithi070.gansda=up:active} 1 up:standby 2024-09-21T09:55:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:07 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/3201636727' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:55:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:07 smithi160 ceph-mon[33203]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-21T09:55:08.881 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:10.455 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:10 smithi070 ceph-mon[41533]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-21T09:55:10.455 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:10 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:10.455 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:10 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:10.455 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:10 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:10.455 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:10 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:55:10.455 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:10 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:10.455 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:10 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:10.456 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:10 smithi070 ceph-mon[29589]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-21T09:55:10.456 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:10 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:10.456 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:10 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:10.456 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:10 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:10.456 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:10 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:55:10.456 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:10 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:10.456 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:10 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:10.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:10 smithi160 ceph-mon[33203]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-21T09:55:10.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:10 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:10.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:10 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:10.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:10 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:10.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:10 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:55:10.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:10 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:10.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:10 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:11.185 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:55:11.185 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":71,"fsid":"7c68bfbc-77fe-11ef-baf6-efdc52797490","created":"2024-09-21T09:47:59.216934+0000","modified":"2024-09-21T09:54:55.624032+0000","last_up_change":"2024-09-21T09:53:01.680149+0000","last_in_change":"2024-09-21T09:52:44.345943+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:51:05.514472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:53:41.830233+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-21T09:54:52.551642+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-21T09:54:52.861791+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6870ff8-14eb-4110-957a-00c73645b181","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2896279311},{"type":"v1","addr":"172.21.15.70:6803","nonce":2896279311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2896279311},{"type":"v1","addr":"172.21.15.70:6805","nonce":2896279311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":2896279311},{"type":"v1","addr":"172.21.15.70:6809","nonce":2896279311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2896279311},{"type":"v1","addr":"172.21.15.70:6807","nonce":2896279311}]},"public_addr":"172.21.15.70:6803/2896279311","cluster_addr":"172.21.15.70:6805/2896279311","heartbeat_back_addr":"172.21.15.70:6809/2896279311","heartbeat_front_addr":"172.21.15.70:6807/2896279311","state":["exists","up"]},{"osd":1,"uuid":"b1349e31-7493-489f-823a-55edb866d25f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":937875216},{"type":"v1","addr":"172.21.15.70:6811","nonce":937875216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":937875216},{"type":"v1","addr":"172.21.15.70:6813","nonce":937875216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":937875216},{"type":"v1","addr":"172.21.15.70:6817","nonce":937875216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":937875216},{"type":"v1","addr":"172.21.15.70:6815","nonce":937875216}]},"public_addr":"172.21.15.70:6811/937875216","cluster_addr":"172.21.15.70:6813/937875216","heartbeat_back_addr":"172.21.15.70:6817/937875216","heartbeat_front_addr":"172.21.15.70:6815/937875216","state":["exists","up"]},{"osd":2,"uuid":"ac4496ba-a860-4b09-bb38-7f433e081541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":940406420},{"type":"v1","addr":"172.21.15.70:6819","nonce":940406420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":940406420},{"type":"v1","addr":"172.21.15.70:6821","nonce":940406420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":940406420},{"type":"v1","addr":"172.21.15.70:6825","nonce":940406420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":940406420},{"type":"v1","addr":"172.21.15.70:6823","nonce":940406420}]},"public_addr":"172.21.15.70:6819/940406420","cluster_addr":"172.21.15.70:6821/940406420","heartbeat_back_addr":"172.21.15.70:6825/940406420","heartbeat_front_addr":"172.21.15.70:6823/940406420","state":["exists","up"]},{"osd":3,"uuid":"1eedf4ce-057c-4576-ae5b-51d3595df195","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1341239112},{"type":"v1","addr":"172.21.15.70:6827","nonce":1341239112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1341239112},{"type":"v1","addr":"172.21.15.70:6829","nonce":1341239112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1341239112},{"type":"v1","addr":"172.21.15.70:6833","nonce":1341239112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1341239112},{"type":"v1","addr":"172.21.15.70:6831","nonce":1341239112}]},"public_addr":"172.21.15.70:6827/1341239112","cluster_addr":"172.21.15.70:6829/1341239112","heartbeat_back_addr":"172.21.15.70:6833/1341239112","heartbeat_front_addr":"172.21.15.70:6831/1341239112","state":["exists","up"]},{"osd":4,"uuid":"a4b728b8-97a4-478f-8a11-ba590c0213e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6801","nonce":3634505869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6803","nonce":3634505869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6807","nonce":3634505869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6805","nonce":3634505869}]},"public_addr":"172.21.15.160:6801/3634505869","cluster_addr":"172.21.15.160:6803/3634505869","heartbeat_back_addr":"172.21.15.160:6807/3634505869","heartbeat_front_addr":"172.21.15.160:6805/3634505869","state":["exists","up"]},{"osd":5,"uuid":"56915347-9254-4e94-b202-1f16d09e7946","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6809","nonce":2006621188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6811","nonce":2006621188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6815","nonce":2006621188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6813","nonce":2006621188}]},"public_addr":"172.21.15.160:6809/2006621188","cluster_addr":"172.21.15.160:6811/2006621188","heartbeat_back_addr":"172.21.15.160:6815/2006621188","heartbeat_front_addr":"172.21.15.160:6813/2006621188","state":["exists","up"]},{"osd":6,"uuid":"79e6de8c-b78b-40e2-9229-0275a873d40e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6817","nonce":1949075400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6819","nonce":1949075400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6823","nonce":1949075400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6821","nonce":1949075400}]},"public_addr":"172.21.15.160:6817/1949075400","cluster_addr":"172.21.15.160:6819/1949075400","heartbeat_back_addr":"172.21.15.160:6823/1949075400","heartbeat_front_addr":"172.21.15.160:6821/1949075400","state":["exists","up"]},{"osd":7,"uuid":"72245c8a-2fdb-4445-b378-16569f337fcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6825","nonce":601673919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6827","nonce":601673919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6831","nonce":601673919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6829","nonce":601673919}]},"public_addr":"172.21.15.160:6825/601673919","cluster_addr":"172.21.15.160:6827/601673919","heartbeat_back_addr":"172.21.15.160:6831/601673919","heartbeat_front_addr":"172.21.15.160:6829/601673919","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:15.134515+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:39.191791+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:51:02.934581+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:51:26.514393+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:51:48.329249+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:12.268576+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:36.370859+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:59.888857+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/1905911777":"2024-09-22T09:53:31.134530+0000","172.21.15.70:0/2521911561":"2024-09-22T09:48:32.256720+0000","172.21.15.70:6800/2142025682":"2024-09-22T09:48:55.219103+0000","172.21.15.70:6800/177993484":"2024-09-22T09:48:32.256720+0000","172.21.15.70:6801/177993484":"2024-09-22T09:48:32.256720+0000","172.21.15.70:0/1012316198":"2024-09-22T09:48:32.256720+0000","172.21.15.70:0/4150778424":"2024-09-22T09:48:32.256720+0000","172.21.15.70:0/749465179":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/1066794139":"2024-09-22T09:53:31.134530+0000","172.21.15.70:0/3159615643":"2024-09-22T09:48:55.219103+0000","172.21.15.70:6801/1520336634":"2024-09-22T09:53:31.134530+0000","172.21.15.70:6801/2142025682":"2024-09-22T09:48:55.219103+0000","172.21.15.70:6800/1520336634":"2024-09-22T09:53:31.134530+0000","172.21.15.70:0/1469701388":"2024-09-22T09:53:31.134530+0000","172.21.15.70:0/2918782890":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/2141342771":"2024-09-22T09:53:31.134530+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T09:55:11.892 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T09:55:11.892 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T09:55:11.893 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T09:55:11.893 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T09:55:11.893 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T09:55:11.894 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T09:55:11.894 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T09:55:11.894 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T09:55:12.417 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:12.421 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:12.422 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:12.423 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:12.427 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:12.429 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:12.437 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:12.531 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:12 smithi160 ceph-mon[33203]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-21T09:55:12.531 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:12 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/2707442268' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:55:12.555 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:12 smithi070 ceph-mon[29589]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-21T09:55:12.555 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:12 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/2707442268' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:55:12.555 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:12 smithi070 ceph-mon[41533]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-21T09:55:12.555 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:12 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/2707442268' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:55:13.617 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:14.030 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:13 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:14.030 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:13 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:14.031 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:13 smithi070 ceph-mon[41533]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-21T09:55:14.031 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:13 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:14.031 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:13 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:14.031 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:13 smithi070 ceph-mon[29589]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-21T09:55:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:13 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:13 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:13 smithi160 ceph-mon[33203]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-21T09:55:14.775 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:55:14 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:55:14] "GET /metrics HTTP/1.1" 200 33486 "" "Prometheus/2.43.0" 2024-09-21T09:55:16.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:16 smithi070 ceph-mon[29589]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-21T09:55:16.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:16 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:16.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:16 smithi070 ceph-mon[41533]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-21T09:55:16.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:16 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:16 smithi160 ceph-mon[33203]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-21T09:55:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:16 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:18.356 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:18 smithi070 ceph-mon[41533]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-21T09:55:18.357 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:18 smithi070 ceph-mon[29589]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-21T09:55:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:18 smithi160 ceph-mon[33203]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-21T09:55:20.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:20 smithi160 ceph-mon[33203]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:20.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:20 smithi070 ceph-mon[29589]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:20.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:20 smithi070 ceph-mon[41533]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:21.473 INFO:teuthology.orchestra.run.smithi070.stdout:107374182449 2024-09-21T09:55:21.474 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T09:55:22.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:22 smithi160 ceph-mon[33203]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:22 smithi070 ceph-mon[29589]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:22.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:22 smithi070 ceph-mon[41533]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:23.265 INFO:teuthology.orchestra.run.smithi070.stdout:163208757288 2024-09-21T09:55:23.266 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T09:55:24.267 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:24 smithi070 ceph-mon[29589]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:24.267 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:24 smithi070 ceph-mon[41533]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:24.557 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:55:24 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:55:24] "GET /metrics HTTP/1.1" 200 33778 "" "Prometheus/2.43.0" 2024-09-21T09:55:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:24 smithi160 ceph-mon[33203]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:25.645 INFO:teuthology.orchestra.run.smithi070.stdout:77309411384 2024-09-21T09:55:25.646 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T09:55:26.517 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:26 smithi070 ceph-mon[29589]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:26.517 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:26 smithi070 ceph-mon[41533]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:26.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:26 smithi160 ceph-mon[33203]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:27.579 INFO:teuthology.orchestra.run.smithi070.stdout:231928234015 2024-09-21T09:55:27.579 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T09:55:28.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:28 smithi070 ceph-mon[29589]: pgmap v73: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:28.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:28 smithi070 ceph-mon[41533]: pgmap v73: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:28.425 INFO:teuthology.orchestra.run.smithi070.stdout:128849018926 2024-09-21T09:55:28.426 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T09:55:28.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:28 smithi160 ceph-mon[33203]: pgmap v73: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:28.818 INFO:teuthology.orchestra.run.smithi070.stdout:55834574908 2024-09-21T09:55:28.819 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T09:55:29.837 INFO:teuthology.orchestra.run.smithi070.stdout:34359738433 2024-09-21T09:55:29.838 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T09:55:30.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:30 smithi070 ceph-mon[29589]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:30.360 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:30 smithi070 ceph-mon[41533]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:30 smithi160 ceph-mon[33203]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:31.048 INFO:teuthology.orchestra.run.smithi070.stdout:197568495653 2024-09-21T09:55:31.048 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T09:55:31.093 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:31.093 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:31.335 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:31 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-09-21T09:55:31.336 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:31 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]: dispatch 2024-09-21T09:55:31.336 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:31 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-09-21T09:55:31.336 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:31 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:31.336 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:31 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-09-21T09:55:31.336 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:31 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]: dispatch 2024-09-21T09:55:31.336 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:31 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-09-21T09:55:31.336 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:31 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:31.586 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:31.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:31 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-09-21T09:55:31.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:31 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]: dispatch 2024-09-21T09:55:31.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:31 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-09-21T09:55:31.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:31 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:31.776 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:32.268 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:32 smithi070 ceph-mon[29589]: pgmap v75: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:32.268 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:32 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-09-21T09:55:32.268 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:32 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]': finished 2024-09-21T09:55:32.268 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:32 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-21T09:55:32.268 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:32 smithi070 ceph-mon[29589]: osdmap e72: 8 total, 8 up, 8 in 2024-09-21T09:55:32.268 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:32 smithi070 ceph-mon[41533]: pgmap v75: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:32.268 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:32 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-09-21T09:55:32.268 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:32 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]': finished 2024-09-21T09:55:32.268 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:32 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-21T09:55:32.268 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:32 smithi070 ceph-mon[41533]: osdmap e72: 8 total, 8 up, 8 in 2024-09-21T09:55:32.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:32 smithi160 ceph-mon[33203]: pgmap v75: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:32.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:32 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-09-21T09:55:32.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:32 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]': finished 2024-09-21T09:55:32.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:32 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-21T09:55:32.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:32 smithi160 ceph-mon[33203]: osdmap e72: 8 total, 8 up, 8 in 2024-09-21T09:55:33.400 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:33.412 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:33.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:33 smithi070 ceph-mon[29589]: osdmap e73: 8 total, 8 up, 8 in 2024-09-21T09:55:33.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:33 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:33.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:33 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:33.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:33 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:33.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:33 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:55:33.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:33 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:33.505 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:33 smithi070 ceph-mon[41533]: osdmap e73: 8 total, 8 up, 8 in 2024-09-21T09:55:33.505 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:33 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:33.506 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:33 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:33.506 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:33 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:33.506 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:33 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:55:33.506 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:33 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:33.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:33 smithi160 ceph-mon[33203]: osdmap e73: 8 total, 8 up, 8 in 2024-09-21T09:55:33.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:33 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:33.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:33 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:33.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:33 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:33.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:33 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:55:33.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:33 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:55:34.432 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:34 smithi070 ceph-mon[29589]: pgmap v78: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:34.432 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:34 smithi070 ceph-mon[41533]: pgmap v78: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:34.803 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:55:34 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:55:34] "GET /metrics HTTP/1.1" 200 33778 "" "Prometheus/2.43.0" 2024-09-21T09:55:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:34 smithi160 ceph-mon[33203]: pgmap v78: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:36.133 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:36.823 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:36 smithi070 ceph-mon[29589]: pgmap v79: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:36.824 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:36 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/2449928670' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T09:55:36.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:36 smithi070 ceph-mon[41533]: pgmap v79: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:36.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:36 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/2449928670' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T09:55:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:36 smithi160 ceph-mon[33203]: pgmap v79: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:36 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/2449928670' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T09:55:37.493 INFO:teuthology.orchestra.run.smithi070.stdout:77309411386 2024-09-21T09:55:38.651 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:38 smithi070 ceph-mon[41533]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:38.652 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:38 smithi070 ceph-mon[29589]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:38 smithi160 ceph-mon[33203]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:39.162 INFO:teuthology.orchestra.run.smithi070.stdout:55834574911 2024-09-21T09:55:39.453 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:39 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/2060708275' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T09:55:39.453 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:39 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/2060708275' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T09:55:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:39 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/2060708275' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T09:55:40.617 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:40 smithi070 ceph-mon[29589]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:40.617 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:40 smithi070 ceph-mon[41533]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:40.637 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411384 got 77309411386 for osd.2 2024-09-21T09:55:40.637 DEBUG:teuthology.parallel:result is None 2024-09-21T09:55:40.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:40 smithi160 ceph-mon[33203]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:40.783 INFO:teuthology.orchestra.run.smithi070.stdout:163208757292 2024-09-21T09:55:41.231 INFO:teuthology.orchestra.run.smithi070.stdout:231928234018 2024-09-21T09:55:41.758 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:41 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/1503058471' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T09:55:41.758 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:41 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/1743471282' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T09:55:41.758 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:41 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/1013814008' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T09:55:41.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:41 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/1503058471' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T09:55:41.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:41 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/1743471282' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T09:55:41.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:41 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/1013814008' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T09:55:41.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:41 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/1503058471' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T09:55:41.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:41 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/1743471282' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T09:55:41.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:41 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/1013814008' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T09:55:42.006 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574908 got 55834574911 for osd.1 2024-09-21T09:55:42.006 DEBUG:teuthology.parallel:result is None 2024-09-21T09:55:42.316 INFO:teuthology.orchestra.run.smithi070.stdout:34359738436 2024-09-21T09:55:42.346 INFO:teuthology.orchestra.run.smithi070.stdout:107374182453 2024-09-21T09:55:42.604 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:42 smithi070 ceph-mon[29589]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:42.605 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:42 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/159623787' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T09:55:42.605 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:42 smithi070 ceph-mon[41533]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:42.605 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:42 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/159623787' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T09:55:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:42 smithi160 ceph-mon[33203]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:42 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/159623787' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T09:55:43.507 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182449 got 107374182453 for osd.3 2024-09-21T09:55:43.507 DEBUG:teuthology.parallel:result is None 2024-09-21T09:55:43.607 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738433 got 34359738436 for osd.0 2024-09-21T09:55:43.607 DEBUG:teuthology.parallel:result is None 2024-09-21T09:55:44.484 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:44 smithi070 ceph-mon[41533]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:44.484 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:44 smithi070 ceph-mon[29589]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:44.574 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234015 got 231928234018 for osd.7 2024-09-21T09:55:44.575 DEBUG:teuthology.parallel:result is None 2024-09-21T09:55:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:44 smithi160 ceph-mon[33203]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:44.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:55:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:55:44] "GET /metrics HTTP/1.1" 200 33778 "" "Prometheus/2.43.0" 2024-09-21T09:55:45.364 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757288 got 163208757292 for osd.5 2024-09-21T09:55:45.364 DEBUG:teuthology.parallel:result is None 2024-09-21T09:55:45.929 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:46.050 INFO:teuthology.orchestra.run.smithi070.stdout:197568495656 2024-09-21T09:55:46.692 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:46 smithi070 ceph-mon[29589]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:46.692 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:46 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/1509640829' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T09:55:46.693 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:46 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:46.693 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:46 smithi070 ceph-mon[41533]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:46.693 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:46 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/1509640829' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T09:55:46.693 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:46 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:46 smithi160 ceph-mon[33203]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:46 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/1509640829' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T09:55:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:46 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:47.030 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495653 got 197568495656 for osd.6 2024-09-21T09:55:47.030 DEBUG:teuthology.parallel:result is None 2024-09-21T09:55:48.224 INFO:teuthology.orchestra.run.smithi070.stdout:128849018930 2024-09-21T09:55:48.785 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:48 smithi070 ceph-mon[29589]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:48.785 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:48 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/1534326820' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T09:55:48.785 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:48 smithi070 ceph-mon[41533]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:48.785 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:48 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/1534326820' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T09:55:48.809 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018926 got 128849018930 for osd.4 2024-09-21T09:55:48.809 DEBUG:teuthology.parallel:result is None 2024-09-21T09:55:48.809 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T09:55:48.809 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T09:55:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:48 smithi160 ceph-mon[33203]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:48 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/1534326820' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T09:55:49.010 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:50.677 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:50 smithi070 ceph-mon[29589]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:50.677 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:50 smithi070 ceph-mon[41533]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:50.842 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:55:50.843 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T09:55:50.845 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:50 smithi160 ceph-mon[33203]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:51.517 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2024-09-21T09:55:49.170896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592749,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":226,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":170,"ondisk_log_size":170,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":228040,"kb_used_data":7764,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":749733176,"statfs":{"total":767960285184,"available":767726772224,"internally_reserved":0,"allocated":7950336,"data_stored":4446684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12741,"internal_metadata":225431099},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002894"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.249270+0000","last_change":"2024-09-21T09:54:54.633334+0000","last_active":"2024-09-21T09:55:31.249270+0000","last_peered":"2024-09-21T09:55:31.249270+0000","last_clean":"2024-09-21T09:55:31.249270+0000","last_became_active":"2024-09-21T09:54:54.632922+0000","last_became_peered":"2024-09-21T09:54:54.632922+0000","last_unstale":"2024-09-21T09:55:31.249270+0000","last_undegraded":"2024-09-21T09:55:31.249270+0000","last_fullsized":"2024-09-21T09:55:31.249270+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:56.286918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.249308+0000","last_change":"2024-09-21T09:54:53.624613+0000","last_active":"2024-09-21T09:55:31.249308+0000","last_peered":"2024-09-21T09:55:31.249308+0000","last_clean":"2024-09-21T09:55:31.249308+0000","last_became_active":"2024-09-21T09:54:53.624346+0000","last_became_peered":"2024-09-21T09:54:53.624346+0000","last_unstale":"2024-09-21T09:55:31.249308+0000","last_undegraded":"2024-09-21T09:55:31.249308+0000","last_fullsized":"2024-09-21T09:55:31.249308+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:37:37.666312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.252080+0000","last_change":"2024-09-21T09:54:54.631581+0000","last_active":"2024-09-21T09:55:31.252080+0000","last_peered":"2024-09-21T09:55:31.252080+0000","last_clean":"2024-09-21T09:55:31.252080+0000","last_became_active":"2024-09-21T09:54:54.631271+0000","last_became_peered":"2024-09-21T09:54:54.631271+0000","last_unstale":"2024-09-21T09:55:31.252080+0000","last_undegraded":"2024-09-21T09:55:31.252080+0000","last_fullsized":"2024-09-21T09:55:31.252080+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:44:16.275479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.242097+0000","last_change":"2024-09-21T09:54:53.623601+0000","last_active":"2024-09-21T09:55:31.242097+0000","last_peered":"2024-09-21T09:55:31.242097+0000","last_clean":"2024-09-21T09:55:31.242097+0000","last_became_active":"2024-09-21T09:54:53.623319+0000","last_became_peered":"2024-09-21T09:54:53.623319+0000","last_unstale":"2024-09-21T09:55:31.242097+0000","last_undegraded":"2024-09-21T09:55:31.242097+0000","last_fullsized":"2024-09-21T09:55:31.242097+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:33:11.417229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.242947+0000","last_change":"2024-09-21T09:54:54.632027+0000","last_active":"2024-09-21T09:55:31.242947+0000","last_peered":"2024-09-21T09:55:31.242947+0000","last_clean":"2024-09-21T09:55:31.242947+0000","last_became_active":"2024-09-21T09:54:54.631708+0000","last_became_peered":"2024-09-21T09:54:54.631708+0000","last_unstale":"2024-09-21T09:55:31.242947+0000","last_undegraded":"2024-09-21T09:55:31.242947+0000","last_fullsized":"2024-09-21T09:55:31.242947+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:13:11.557171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.249379+0000","last_change":"2024-09-21T09:54:53.625016+0000","last_active":"2024-09-21T09:55:31.249379+0000","last_peered":"2024-09-21T09:55:31.249379+0000","last_clean":"2024-09-21T09:55:31.249379+0000","last_became_active":"2024-09-21T09:54:53.624750+0000","last_became_peered":"2024-09-21T09:54:53.624750+0000","last_unstale":"2024-09-21T09:55:31.249379+0000","last_undegraded":"2024-09-21T09:55:31.249379+0000","last_fullsized":"2024-09-21T09:55:31.249379+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:34:26.823544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.249679+0000","last_change":"2024-09-21T09:54:54.633218+0000","last_active":"2024-09-21T09:55:31.249679+0000","last_peered":"2024-09-21T09:55:31.249679+0000","last_clean":"2024-09-21T09:55:31.249679+0000","last_became_active":"2024-09-21T09:54:54.632912+0000","last_became_peered":"2024-09-21T09:54:54.632912+0000","last_unstale":"2024-09-21T09:55:31.249679+0000","last_undegraded":"2024-09-21T09:55:31.249679+0000","last_fullsized":"2024-09-21T09:55:31.249679+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:24:09.344461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":23,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.247193+0000","last_change":"2024-09-21T09:54:53.625692+0000","last_active":"2024-09-21T09:55:31.247193+0000","last_peered":"2024-09-21T09:55:31.247193+0000","last_clean":"2024-09-21T09:55:31.247193+0000","last_became_active":"2024-09-21T09:54:53.625372+0000","last_became_peered":"2024-09-21T09:54:53.625372+0000","last_unstale":"2024-09-21T09:55:31.247193+0000","last_undegraded":"2024-09-21T09:55:31.247193+0000","last_fullsized":"2024-09-21T09:55:31.247193+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:32.289423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.249845+0000","last_change":"2024-09-21T09:54:53.625967+0000","last_active":"2024-09-21T09:55:31.249845+0000","last_peered":"2024-09-21T09:55:31.249845+0000","last_clean":"2024-09-21T09:55:31.249845+0000","last_became_active":"2024-09-21T09:54:53.625650+0000","last_became_peered":"2024-09-21T09:54:53.625650+0000","last_unstale":"2024-09-21T09:55:31.249845+0000","last_undegraded":"2024-09-21T09:55:31.249845+0000","last_fullsized":"2024-09-21T09:55:31.249845+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:42:10.159809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.251843+0000","last_change":"2024-09-21T09:54:54.677572+0000","last_active":"2024-09-21T09:55:31.251843+0000","last_peered":"2024-09-21T09:55:31.251843+0000","last_clean":"2024-09-21T09:55:31.251843+0000","last_became_active":"2024-09-21T09:54:54.677253+0000","last_became_peered":"2024-09-21T09:54:54.677253+0000","last_unstale":"2024-09-21T09:55:31.251843+0000","last_undegraded":"2024-09-21T09:55:31.251843+0000","last_fullsized":"2024-09-21T09:55:31.251843+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:31:40.629299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.249755+0000","last_change":"2024-09-21T09:54:53.624164+0000","last_active":"2024-09-21T09:55:31.249755+0000","last_peered":"2024-09-21T09:55:31.249755+0000","last_clean":"2024-09-21T09:55:31.249755+0000","last_became_active":"2024-09-21T09:54:53.623875+0000","last_became_peered":"2024-09-21T09:54:53.623875+0000","last_unstale":"2024-09-21T09:55:31.249755+0000","last_undegraded":"2024-09-21T09:55:31.249755+0000","last_fullsized":"2024-09-21T09:55:31.249755+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:11:56.692776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.246037+0000","last_change":"2024-09-21T09:54:53.721961+0000","last_active":"2024-09-21T09:55:31.246037+0000","last_peered":"2024-09-21T09:55:31.246037+0000","last_clean":"2024-09-21T09:55:31.246037+0000","last_became_active":"2024-09-21T09:54:53.721676+0000","last_became_peered":"2024-09-21T09:54:53.721676+0000","last_unstale":"2024-09-21T09:55:31.246037+0000","last_undegraded":"2024-09-21T09:55:31.246037+0000","last_fullsized":"2024-09-21T09:55:31.246037+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:14:40.284754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.249624+0000","last_change":"2024-09-21T09:54:53.628923+0000","last_active":"2024-09-21T09:55:31.249624+0000","last_peered":"2024-09-21T09:55:31.249624+0000","last_clean":"2024-09-21T09:55:31.249624+0000","last_became_active":"2024-09-21T09:54:53.627939+0000","last_became_peered":"2024-09-21T09:54:53.627939+0000","last_unstale":"2024-09-21T09:55:31.249624+0000","last_undegraded":"2024-09-21T09:55:31.249624+0000","last_fullsized":"2024-09-21T09:55:31.249624+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:07:02.820689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T09:55:32.413399+0000","last_change":"2024-09-21T09:55:32.413399+0000","last_active":"2024-09-21T09:55:32.413399+0000","last_peered":"2024-09-21T09:55:32.413399+0000","last_clean":"2024-09-21T09:55:32.413399+0000","last_became_active":"2024-09-21T09:55:32.413050+0000","last_became_peered":"2024-09-21T09:55:32.413050+0000","last_unstale":"2024-09-21T09:55:32.413399+0000","last_undegraded":"2024-09-21T09:55:32.413399+0000","last_fullsized":"2024-09-21T09:55:32.413399+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:30.986060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.252037+0000","last_change":"2024-09-21T09:54:53.632932+0000","last_active":"2024-09-21T09:55:31.252037+0000","last_peered":"2024-09-21T09:55:31.252037+0000","last_clean":"2024-09-21T09:55:31.252037+0000","last_became_active":"2024-09-21T09:54:53.632652+0000","last_became_peered":"2024-09-21T09:54:53.632652+0000","last_unstale":"2024-09-21T09:55:31.252037+0000","last_undegraded":"2024-09-21T09:55:31.252037+0000","last_fullsized":"2024-09-21T09:55:31.252037+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:01.436815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.242451+0000","last_change":"2024-09-21T09:54:54.645001+0000","last_active":"2024-09-21T09:55:31.242451+0000","last_peered":"2024-09-21T09:55:31.242451+0000","last_clean":"2024-09-21T09:55:31.242451+0000","last_became_active":"2024-09-21T09:54:54.644635+0000","last_became_peered":"2024-09-21T09:54:54.644635+0000","last_unstale":"2024-09-21T09:55:31.242451+0000","last_undegraded":"2024-09-21T09:55:31.242451+0000","last_fullsized":"2024-09-21T09:55:31.242451+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:46:24.237543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.249097+0000","last_change":"2024-09-21T09:54:53.631134+0000","last_active":"2024-09-21T09:55:31.249097+0000","last_peered":"2024-09-21T09:55:31.249097+0000","last_clean":"2024-09-21T09:55:31.249097+0000","last_became_active":"2024-09-21T09:54:53.630853+0000","last_became_peered":"2024-09-21T09:54:53.630853+0000","last_unstale":"2024-09-21T09:55:31.249097+0000","last_undegraded":"2024-09-21T09:55:31.249097+0000","last_fullsized":"2024-09-21T09:55:31.249097+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:48:45.503176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.246182+0000","last_change":"2024-09-21T09:54:53.723089+0000","last_active":"2024-09-21T09:55:31.246182+0000","last_peered":"2024-09-21T09:55:31.246182+0000","last_clean":"2024-09-21T09:55:31.246182+0000","last_became_active":"2024-09-21T09:54:53.722813+0000","last_became_peered":"2024-09-21T09:54:53.722813+0000","last_unstale":"2024-09-21T09:55:31.246182+0000","last_undegraded":"2024-09-21T09:55:31.246182+0000","last_fullsized":"2024-09-21T09:55:31.246182+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:23.685574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.247988+0000","last_change":"2024-09-21T09:54:53.626364+0000","last_active":"2024-09-21T09:55:31.247988+0000","last_peered":"2024-09-21T09:55:31.247988+0000","last_clean":"2024-09-21T09:55:31.247988+0000","last_became_active":"2024-09-21T09:54:53.626082+0000","last_became_peered":"2024-09-21T09:54:53.626082+0000","last_unstale":"2024-09-21T09:55:31.247988+0000","last_undegraded":"2024-09-21T09:55:31.247988+0000","last_fullsized":"2024-09-21T09:55:31.247988+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:22:16.136523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.246788+0000","last_change":"2024-09-21T09:54:53.638354+0000","last_active":"2024-09-21T09:55:31.246788+0000","last_peered":"2024-09-21T09:55:31.246788+0000","last_clean":"2024-09-21T09:55:31.246788+0000","last_became_active":"2024-09-21T09:54:53.638077+0000","last_became_peered":"2024-09-21T09:54:53.638077+0000","last_unstale":"2024-09-21T09:55:31.246788+0000","last_undegraded":"2024-09-21T09:55:31.246788+0000","last_fullsized":"2024-09-21T09:55:31.246788+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:07:36.028409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.246273+0000","last_change":"2024-09-21T09:54:53.625145+0000","last_active":"2024-09-21T09:55:31.246273+0000","last_peered":"2024-09-21T09:55:31.246273+0000","last_clean":"2024-09-21T09:55:31.246273+0000","last_became_active":"2024-09-21T09:54:53.624845+0000","last_became_peered":"2024-09-21T09:54:53.624845+0000","last_unstale":"2024-09-21T09:55:31.246273+0000","last_undegraded":"2024-09-21T09:55:31.246273+0000","last_fullsized":"2024-09-21T09:55:31.246273+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:44.551367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.246537+0000","last_change":"2024-09-21T09:54:53.638897+0000","last_active":"2024-09-21T09:55:31.246537+0000","last_peered":"2024-09-21T09:55:31.246537+0000","last_clean":"2024-09-21T09:55:31.246537+0000","last_became_active":"2024-09-21T09:54:53.638624+0000","last_became_peered":"2024-09-21T09:54:53.638624+0000","last_unstale":"2024-09-21T09:55:31.246537+0000","last_undegraded":"2024-09-21T09:55:31.246537+0000","last_fullsized":"2024-09-21T09:55:31.246537+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:48:33.305248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.247453+0000","last_change":"2024-09-21T09:54:53.625098+0000","last_active":"2024-09-21T09:55:31.247453+0000","last_peered":"2024-09-21T09:55:31.247453+0000","last_clean":"2024-09-21T09:55:31.247453+0000","last_became_active":"2024-09-21T09:54:53.624771+0000","last_became_peered":"2024-09-21T09:54:53.624771+0000","last_unstale":"2024-09-21T09:55:31.247453+0000","last_undegraded":"2024-09-21T09:55:31.247453+0000","last_fullsized":"2024-09-21T09:55:31.247453+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:54:59.433791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.244910+0000","last_change":"2024-09-21T09:54:53.722001+0000","last_active":"2024-09-21T09:55:31.244910+0000","last_peered":"2024-09-21T09:55:31.244910+0000","last_clean":"2024-09-21T09:55:31.244910+0000","last_became_active":"2024-09-21T09:54:53.721570+0000","last_became_peered":"2024-09-21T09:54:53.721570+0000","last_unstale":"2024-09-21T09:55:31.244910+0000","last_undegraded":"2024-09-21T09:55:31.244910+0000","last_fullsized":"2024-09-21T09:55:31.244910+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:49:10.088299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"59'128","reported_seq":269,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.249432+0000","last_change":"2024-09-21T09:53:05.275142+0000","last_active":"2024-09-21T09:55:31.249432+0000","last_peered":"2024-09-21T09:55:31.249432+0000","last_clean":"2024-09-21T09:55:31.249432+0000","last_became_active":"2024-09-21T09:53:05.274635+0000","last_became_peered":"2024-09-21T09:53:05.274635+0000","last_unstale":"2024-09-21T09:55:31.249432+0000","last_undegraded":"2024-09-21T09:55:31.249432+0000","last_fullsized":"2024-09-21T09:55:31.249432+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:31.035143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":28,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.251965+0000","last_change":"2024-09-21T09:53:49.219350+0000","last_active":"2024-09-21T09:55:31.251965+0000","last_peered":"2024-09-21T09:55:31.251965+0000","last_clean":"2024-09-21T09:55:31.251965+0000","last_became_active":"2024-09-21T09:53:43.189467+0000","last_became_peered":"2024-09-21T09:53:43.189467+0000","last_unstale":"2024-09-21T09:55:31.251965+0000","last_undegraded":"2024-09-21T09:55:31.251965+0000","last_fullsized":"2024-09-21T09:55:31.251965+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:37.943058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.248961+0000","last_change":"2024-09-21T09:54:53.638100+0000","last_active":"2024-09-21T09:55:31.248961+0000","last_peered":"2024-09-21T09:55:31.248961+0000","last_clean":"2024-09-21T09:55:31.248961+0000","last_became_active":"2024-09-21T09:54:53.636403+0000","last_became_peered":"2024-09-21T09:54:53.636403+0000","last_unstale":"2024-09-21T09:55:31.248961+0000","last_undegraded":"2024-09-21T09:55:31.248961+0000","last_fullsized":"2024-09-21T09:55:31.248961+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:31:07.327629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.247147+0000","last_change":"2024-09-21T09:54:53.722213+0000","last_active":"2024-09-21T09:55:31.247147+0000","last_peered":"2024-09-21T09:55:31.247147+0000","last_clean":"2024-09-21T09:55:31.247147+0000","last_became_active":"2024-09-21T09:54:53.721437+0000","last_became_peered":"2024-09-21T09:54:53.721437+0000","last_unstale":"2024-09-21T09:55:31.247147+0000","last_undegraded":"2024-09-21T09:55:31.247147+0000","last_fullsized":"2024-09-21T09:55:31.247147+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:01:41.456731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T09:55:32.412392+0000","last_change":"2024-09-21T09:53:49.374842+0000","last_active":"2024-09-21T09:55:32.412392+0000","last_peered":"2024-09-21T09:55:32.412392+0000","last_clean":"2024-09-21T09:55:32.412392+0000","last_became_active":"2024-09-21T09:53:43.191101+0000","last_became_peered":"2024-09-21T09:53:43.191101+0000","last_unstale":"2024-09-21T09:55:32.412392+0000","last_undegraded":"2024-09-21T09:55:32.412392+0000","last_fullsized":"2024-09-21T09:55:32.412392+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:24:33.044234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.242119+0000","last_change":"2024-09-21T09:54:53.626864+0000","last_active":"2024-09-21T09:55:31.242119+0000","last_peered":"2024-09-21T09:55:31.242119+0000","last_clean":"2024-09-21T09:55:31.242119+0000","last_became_active":"2024-09-21T09:54:53.626612+0000","last_became_peered":"2024-09-21T09:54:53.626612+0000","last_unstale":"2024-09-21T09:55:31.242119+0000","last_undegraded":"2024-09-21T09:55:31.242119+0000","last_fullsized":"2024-09-21T09:55:31.242119+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:00:19.195266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.242870+0000","last_change":"2024-09-21T09:54:54.677306+0000","last_active":"2024-09-21T09:55:31.242870+0000","last_peered":"2024-09-21T09:55:31.242870+0000","last_clean":"2024-09-21T09:55:31.242870+0000","last_became_active":"2024-09-21T09:54:54.677012+0000","last_became_peered":"2024-09-21T09:54:54.677012+0000","last_unstale":"2024-09-21T09:55:31.242870+0000","last_undegraded":"2024-09-21T09:55:31.242870+0000","last_fullsized":"2024-09-21T09:55:31.242870+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:21:18.119360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.252842+0000","last_change":"2024-09-21T09:53:49.211722+0000","last_active":"2024-09-21T09:55:31.252842+0000","last_peered":"2024-09-21T09:55:31.252842+0000","last_clean":"2024-09-21T09:55:31.252842+0000","last_became_active":"2024-09-21T09:53:43.196820+0000","last_became_peered":"2024-09-21T09:53:43.196820+0000","last_unstale":"2024-09-21T09:55:31.252842+0000","last_undegraded":"2024-09-21T09:55:31.252842+0000","last_fullsized":"2024-09-21T09:55:31.252842+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:24.782773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.243621+0000","last_change":"2024-09-21T09:54:53.622210+0000","last_active":"2024-09-21T09:55:31.243621+0000","last_peered":"2024-09-21T09:55:31.243621+0000","last_clean":"2024-09-21T09:55:31.243621+0000","last_became_active":"2024-09-21T09:54:53.621855+0000","last_became_peered":"2024-09-21T09:54:53.621855+0000","last_unstale":"2024-09-21T09:55:31.243621+0000","last_undegraded":"2024-09-21T09:55:31.243621+0000","last_fullsized":"2024-09-21T09:55:31.243621+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:47:17.990851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.244978+0000","last_change":"2024-09-21T09:54:53.635976+0000","last_active":"2024-09-21T09:55:31.244978+0000","last_peered":"2024-09-21T09:55:31.244978+0000","last_clean":"2024-09-21T09:55:31.244978+0000","last_became_active":"2024-09-21T09:54:53.635592+0000","last_became_peered":"2024-09-21T09:54:53.635592+0000","last_unstale":"2024-09-21T09:55:31.244978+0000","last_undegraded":"2024-09-21T09:55:31.244978+0000","last_fullsized":"2024-09-21T09:55:31.244978+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:44.327770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":29,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.251933+0000","last_change":"2024-09-21T09:53:49.219893+0000","last_active":"2024-09-21T09:55:31.251933+0000","last_peered":"2024-09-21T09:55:31.251933+0000","last_clean":"2024-09-21T09:55:31.251933+0000","last_became_active":"2024-09-21T09:53:43.189073+0000","last_became_peered":"2024-09-21T09:53:43.189073+0000","last_unstale":"2024-09-21T09:55:31.251933+0000","last_undegraded":"2024-09-21T09:55:31.251933+0000","last_fullsized":"2024-09-21T09:55:31.251933+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:48.257597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084477399999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.246441+0000","last_change":"2024-09-21T09:54:53.626386+0000","last_active":"2024-09-21T09:55:31.246441+0000","last_peered":"2024-09-21T09:55:31.246441+0000","last_clean":"2024-09-21T09:55:31.246441+0000","last_became_active":"2024-09-21T09:54:53.625344+0000","last_became_peered":"2024-09-21T09:54:53.625344+0000","last_unstale":"2024-09-21T09:55:31.246441+0000","last_undegraded":"2024-09-21T09:55:31.246441+0000","last_fullsized":"2024-09-21T09:55:31.246441+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:28:40.301087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.244358+0000","last_change":"2024-09-21T09:54:53.724007+0000","last_active":"2024-09-21T09:55:31.244358+0000","last_peered":"2024-09-21T09:55:31.244358+0000","last_clean":"2024-09-21T09:55:31.244358+0000","last_became_active":"2024-09-21T09:54:53.723726+0000","last_became_peered":"2024-09-21T09:54:53.723726+0000","last_unstale":"2024-09-21T09:55:31.244358+0000","last_undegraded":"2024-09-21T09:55:31.244358+0000","last_fullsized":"2024-09-21T09:55:31.244358+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:08:37.741014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.252164+0000","last_change":"2024-09-21T09:54:53.633398+0000","last_active":"2024-09-21T09:55:31.252164+0000","last_peered":"2024-09-21T09:55:31.252164+0000","last_clean":"2024-09-21T09:55:31.252164+0000","last_became_active":"2024-09-21T09:54:53.633146+0000","last_became_peered":"2024-09-21T09:54:53.633146+0000","last_unstale":"2024-09-21T09:55:31.252164+0000","last_undegraded":"2024-09-21T09:55:31.252164+0000","last_fullsized":"2024-09-21T09:55:31.252164+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:10:19.183476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.244512+0000","last_change":"2024-09-21T09:53:49.211745+0000","last_active":"2024-09-21T09:55:31.244512+0000","last_peered":"2024-09-21T09:55:31.244512+0000","last_clean":"2024-09-21T09:55:31.244512+0000","last_became_active":"2024-09-21T09:53:43.189852+0000","last_became_peered":"2024-09-21T09:53:43.189852+0000","last_unstale":"2024-09-21T09:55:31.244512+0000","last_undegraded":"2024-09-21T09:55:31.244512+0000","last_fullsized":"2024-09-21T09:55:31.244512+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:16.083394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.243928+0000","last_change":"2024-09-21T09:54:53.635401+0000","last_active":"2024-09-21T09:55:31.243928+0000","last_peered":"2024-09-21T09:55:31.243928+0000","last_clean":"2024-09-21T09:55:31.243928+0000","last_became_active":"2024-09-21T09:54:53.635044+0000","last_became_peered":"2024-09-21T09:54:53.635044+0000","last_unstale":"2024-09-21T09:55:31.243928+0000","last_undegraded":"2024-09-21T09:55:31.243928+0000","last_fullsized":"2024-09-21T09:55:31.243928+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:40:49.188614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.244445+0000","last_change":"2024-09-21T09:54:53.624476+0000","last_active":"2024-09-21T09:55:31.244445+0000","last_peered":"2024-09-21T09:55:31.244445+0000","last_clean":"2024-09-21T09:55:31.244445+0000","last_became_active":"2024-09-21T09:54:53.624223+0000","last_became_peered":"2024-09-21T09:54:53.624223+0000","last_unstale":"2024-09-21T09:55:31.244445+0000","last_undegraded":"2024-09-21T09:55:31.244445+0000","last_fullsized":"2024-09-21T09:55:31.244445+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:47:05.700214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.249645+0000","last_change":"2024-09-21T09:53:49.374824+0000","last_active":"2024-09-21T09:55:31.249645+0000","last_peered":"2024-09-21T09:55:31.249645+0000","last_clean":"2024-09-21T09:55:31.249645+0000","last_became_active":"2024-09-21T09:53:43.197315+0000","last_became_peered":"2024-09-21T09:53:43.197315+0000","last_unstale":"2024-09-21T09:55:31.249645+0000","last_undegraded":"2024-09-21T09:55:31.249645+0000","last_fullsized":"2024-09-21T09:55:31.249645+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:49:20.376464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.242523+0000","last_change":"2024-09-21T09:54:54.630089+0000","last_active":"2024-09-21T09:55:31.242523+0000","last_peered":"2024-09-21T09:55:31.242523+0000","last_clean":"2024-09-21T09:55:31.242523+0000","last_became_active":"2024-09-21T09:54:54.629786+0000","last_became_peered":"2024-09-21T09:54:54.629786+0000","last_unstale":"2024-09-21T09:55:31.242523+0000","last_undegraded":"2024-09-21T09:55:31.242523+0000","last_fullsized":"2024-09-21T09:55:31.242523+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:01:39.916299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.249796+0000","last_change":"2024-09-21T09:54:53.627677+0000","last_active":"2024-09-21T09:55:31.249796+0000","last_peered":"2024-09-21T09:55:31.249796+0000","last_clean":"2024-09-21T09:55:31.249796+0000","last_became_active":"2024-09-21T09:54:53.627142+0000","last_became_peered":"2024-09-21T09:54:53.627142+0000","last_unstale":"2024-09-21T09:55:31.249796+0000","last_undegraded":"2024-09-21T09:55:31.249796+0000","last_fullsized":"2024-09-21T09:55:31.249796+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:36.818257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.244909+0000","last_change":"2024-09-21T09:53:49.211757+0000","last_active":"2024-09-21T09:55:31.244909+0000","last_peered":"2024-09-21T09:55:31.244909+0000","last_clean":"2024-09-21T09:55:31.244909+0000","last_became_active":"2024-09-21T09:53:43.197526+0000","last_became_peered":"2024-09-21T09:53:43.197526+0000","last_unstale":"2024-09-21T09:55:31.244909+0000","last_undegraded":"2024-09-21T09:55:31.244909+0000","last_fullsized":"2024-09-21T09:55:31.244909+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:23:27.608603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035589600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.247301+0000","last_change":"2024-09-21T09:54:54.631615+0000","last_active":"2024-09-21T09:55:31.247301+0000","last_peered":"2024-09-21T09:55:31.247301+0000","last_clean":"2024-09-21T09:55:31.247301+0000","last_became_active":"2024-09-21T09:54:54.631222+0000","last_became_peered":"2024-09-21T09:54:54.631222+0000","last_unstale":"2024-09-21T09:55:31.247301+0000","last_undegraded":"2024-09-21T09:55:31.247301+0000","last_fullsized":"2024-09-21T09:55:31.247301+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:16:19.613693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.243035+0000","last_change":"2024-09-21T09:54:53.626192+0000","last_active":"2024-09-21T09:55:31.243035+0000","last_peered":"2024-09-21T09:55:31.243035+0000","last_clean":"2024-09-21T09:55:31.243035+0000","last_became_active":"2024-09-21T09:54:53.625916+0000","last_became_peered":"2024-09-21T09:54:53.625916+0000","last_unstale":"2024-09-21T09:55:31.243035+0000","last_undegraded":"2024-09-21T09:55:31.243035+0000","last_fullsized":"2024-09-21T09:55:31.243035+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:57:48.369381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.247388+0000","last_change":"2024-09-21T09:53:49.220984+0000","last_active":"2024-09-21T09:55:31.247388+0000","last_peered":"2024-09-21T09:55:31.247388+0000","last_clean":"2024-09-21T09:55:31.247388+0000","last_became_active":"2024-09-21T09:53:43.190004+0000","last_became_peered":"2024-09-21T09:53:43.190004+0000","last_unstale":"2024-09-21T09:55:31.247388+0000","last_undegraded":"2024-09-21T09:55:31.247388+0000","last_fullsized":"2024-09-21T09:55:31.247388+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:26:21.202894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.246041+0000","last_change":"2024-09-21T09:54:53.636902+0000","last_active":"2024-09-21T09:55:31.246041+0000","last_peered":"2024-09-21T09:55:31.246041+0000","last_clean":"2024-09-21T09:55:31.246041+0000","last_became_active":"2024-09-21T09:54:53.636616+0000","last_became_peered":"2024-09-21T09:54:53.636616+0000","last_unstale":"2024-09-21T09:55:31.246041+0000","last_undegraded":"2024-09-21T09:55:31.246041+0000","last_fullsized":"2024-09-21T09:55:31.246041+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:46:47.694495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.246319+0000","last_change":"2024-09-21T09:54:53.638962+0000","last_active":"2024-09-21T09:55:31.246319+0000","last_peered":"2024-09-21T09:55:31.246319+0000","last_clean":"2024-09-21T09:55:31.246319+0000","last_became_active":"2024-09-21T09:54:53.638098+0000","last_became_peered":"2024-09-21T09:54:53.638098+0000","last_unstale":"2024-09-21T09:55:31.246319+0000","last_undegraded":"2024-09-21T09:55:31.246319+0000","last_fullsized":"2024-09-21T09:55:31.246319+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:30:32.467009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"71'6","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.246282+0000","last_change":"2024-09-21T09:54:53.624887+0000","last_active":"2024-09-21T09:55:31.246282+0000","last_peered":"2024-09-21T09:55:31.246282+0000","last_clean":"2024-09-21T09:55:31.246282+0000","last_became_active":"2024-09-21T09:54:53.624525+0000","last_became_peered":"2024-09-21T09:54:53.624525+0000","last_unstale":"2024-09-21T09:55:31.246282+0000","last_undegraded":"2024-09-21T09:55:31.246282+0000","last_fullsized":"2024-09-21T09:55:31.246282+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:05:59.605316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.244044+0000","last_change":"2024-09-21T09:54:53.637302+0000","last_active":"2024-09-21T09:55:31.244044+0000","last_peered":"2024-09-21T09:55:31.244044+0000","last_clean":"2024-09-21T09:55:31.244044+0000","last_became_active":"2024-09-21T09:54:53.637019+0000","last_became_peered":"2024-09-21T09:54:53.637019+0000","last_unstale":"2024-09-21T09:55:31.244044+0000","last_undegraded":"2024-09-21T09:55:31.244044+0000","last_fullsized":"2024-09-21T09:55:31.244044+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:03:01.499169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":18,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.249544+0000","last_change":"2024-09-21T09:54:53.632794+0000","last_active":"2024-09-21T09:55:31.249544+0000","last_peered":"2024-09-21T09:55:31.249544+0000","last_clean":"2024-09-21T09:55:31.249544+0000","last_became_active":"2024-09-21T09:54:53.632493+0000","last_became_peered":"2024-09-21T09:54:53.632493+0000","last_unstale":"2024-09-21T09:55:31.249544+0000","last_undegraded":"2024-09-21T09:55:31.249544+0000","last_fullsized":"2024-09-21T09:55:31.249544+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:15:30.340221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.252131+0000","last_change":"2024-09-21T09:54:53.628259+0000","last_active":"2024-09-21T09:55:31.252131+0000","last_peered":"2024-09-21T09:55:31.252131+0000","last_clean":"2024-09-21T09:55:31.252131+0000","last_became_active":"2024-09-21T09:54:53.627998+0000","last_became_peered":"2024-09-21T09:54:53.627998+0000","last_unstale":"2024-09-21T09:55:31.252131+0000","last_undegraded":"2024-09-21T09:55:31.252131+0000","last_fullsized":"2024-09-21T09:55:31.252131+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:47:26.663885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.248575+0000","last_change":"2024-09-21T09:54:54.677644+0000","last_active":"2024-09-21T09:55:31.248575+0000","last_peered":"2024-09-21T09:55:31.248575+0000","last_clean":"2024-09-21T09:55:31.248575+0000","last_became_active":"2024-09-21T09:54:54.677043+0000","last_became_peered":"2024-09-21T09:54:54.677043+0000","last_unstale":"2024-09-21T09:55:31.248575+0000","last_undegraded":"2024-09-21T09:55:31.248575+0000","last_fullsized":"2024-09-21T09:55:31.248575+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:47:09.833413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.242515+0000","last_change":"2024-09-21T09:54:53.628040+0000","last_active":"2024-09-21T09:55:31.242515+0000","last_peered":"2024-09-21T09:55:31.242515+0000","last_clean":"2024-09-21T09:55:31.242515+0000","last_became_active":"2024-09-21T09:54:53.627803+0000","last_became_peered":"2024-09-21T09:54:53.627803+0000","last_unstale":"2024-09-21T09:55:31.242515+0000","last_undegraded":"2024-09-21T09:55:31.242515+0000","last_fullsized":"2024-09-21T09:55:31.242515+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:34:24.332639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T09:55:32.436468+0000","last_change":"2024-09-21T09:55:32.436468+0000","last_active":"2024-09-21T09:55:32.436468+0000","last_peered":"2024-09-21T09:55:32.436468+0000","last_clean":"2024-09-21T09:55:32.436468+0000","last_became_active":"2024-09-21T09:55:32.436208+0000","last_became_peered":"2024-09-21T09:55:32.436208+0000","last_unstale":"2024-09-21T09:55:32.436468+0000","last_undegraded":"2024-09-21T09:55:32.436468+0000","last_fullsized":"2024-09-21T09:55:32.436468+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:54:14.558869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.247391+0000","last_change":"2024-09-21T09:54:53.638131+0000","last_active":"2024-09-21T09:55:31.247391+0000","last_peered":"2024-09-21T09:55:31.247391+0000","last_clean":"2024-09-21T09:55:31.247391+0000","last_became_active":"2024-09-21T09:54:53.637677+0000","last_became_peered":"2024-09-21T09:54:53.637677+0000","last_unstale":"2024-09-21T09:55:31.247391+0000","last_undegraded":"2024-09-21T09:55:31.247391+0000","last_fullsized":"2024-09-21T09:55:31.247391+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:35.298342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.247383+0000","last_change":"2024-09-21T09:54:53.629173+0000","last_active":"2024-09-21T09:55:31.247383+0000","last_peered":"2024-09-21T09:55:31.247383+0000","last_clean":"2024-09-21T09:55:31.247383+0000","last_became_active":"2024-09-21T09:54:53.628897+0000","last_became_peered":"2024-09-21T09:54:53.628897+0000","last_unstale":"2024-09-21T09:55:31.247383+0000","last_undegraded":"2024-09-21T09:55:31.247383+0000","last_fullsized":"2024-09-21T09:55:31.247383+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:23:55.120748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.242789+0000","last_change":"2024-09-21T09:54:54.631770+0000","last_active":"2024-09-21T09:55:31.242789+0000","last_peered":"2024-09-21T09:55:31.242789+0000","last_clean":"2024-09-21T09:55:31.242789+0000","last_became_active":"2024-09-21T09:54:54.631481+0000","last_became_peered":"2024-09-21T09:54:54.631481+0000","last_unstale":"2024-09-21T09:55:31.242789+0000","last_undegraded":"2024-09-21T09:55:31.242789+0000","last_fullsized":"2024-09-21T09:55:31.242789+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:13:55.396337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.243784+0000","last_change":"2024-09-21T09:54:53.630346+0000","last_active":"2024-09-21T09:55:31.243784+0000","last_peered":"2024-09-21T09:55:31.243784+0000","last_clean":"2024-09-21T09:55:31.243784+0000","last_became_active":"2024-09-21T09:54:53.629976+0000","last_became_peered":"2024-09-21T09:54:53.629976+0000","last_unstale":"2024-09-21T09:55:31.243784+0000","last_undegraded":"2024-09-21T09:55:31.243784+0000","last_fullsized":"2024-09-21T09:55:31.243784+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:03:20.424882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T09:55:32.469886+0000","last_change":"2024-09-21T09:55:32.469886+0000","last_active":"2024-09-21T09:55:32.469886+0000","last_peered":"2024-09-21T09:55:32.469886+0000","last_clean":"2024-09-21T09:55:32.469886+0000","last_became_active":"2024-09-21T09:55:32.469481+0000","last_became_peered":"2024-09-21T09:55:32.469481+0000","last_unstale":"2024-09-21T09:55:32.469886+0000","last_undegraded":"2024-09-21T09:55:32.469886+0000","last_fullsized":"2024-09-21T09:55:32.469886+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:59.955781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.246229+0000","last_change":"2024-09-21T09:54:53.624732+0000","last_active":"2024-09-21T09:55:31.246229+0000","last_peered":"2024-09-21T09:55:31.246229+0000","last_clean":"2024-09-21T09:55:31.246229+0000","last_became_active":"2024-09-21T09:54:53.624453+0000","last_became_peered":"2024-09-21T09:54:53.624453+0000","last_unstale":"2024-09-21T09:55:31.246229+0000","last_undegraded":"2024-09-21T09:55:31.246229+0000","last_fullsized":"2024-09-21T09:55:31.246229+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:09:20.809064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.252953+0000","last_change":"2024-09-21T09:54:53.722977+0000","last_active":"2024-09-21T09:55:31.252953+0000","last_peered":"2024-09-21T09:55:31.252953+0000","last_clean":"2024-09-21T09:55:31.252953+0000","last_became_active":"2024-09-21T09:54:53.722691+0000","last_became_peered":"2024-09-21T09:54:53.722691+0000","last_unstale":"2024-09-21T09:55:31.252953+0000","last_undegraded":"2024-09-21T09:55:31.252953+0000","last_fullsized":"2024-09-21T09:55:31.252953+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:49:11.950594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.242600+0000","last_change":"2024-09-21T09:54:53.626810+0000","last_active":"2024-09-21T09:55:31.242600+0000","last_peered":"2024-09-21T09:55:31.242600+0000","last_clean":"2024-09-21T09:55:31.242600+0000","last_became_active":"2024-09-21T09:54:53.626533+0000","last_became_peered":"2024-09-21T09:54:53.626533+0000","last_unstale":"2024-09-21T09:55:31.242600+0000","last_undegraded":"2024-09-21T09:55:31.242600+0000","last_fullsized":"2024-09-21T09:55:31.242600+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:43:30.732225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.246333+0000","last_change":"2024-09-21T09:54:53.722232+0000","last_active":"2024-09-21T09:55:31.246333+0000","last_peered":"2024-09-21T09:55:31.246333+0000","last_clean":"2024-09-21T09:55:31.246333+0000","last_became_active":"2024-09-21T09:54:53.721954+0000","last_became_peered":"2024-09-21T09:54:53.721954+0000","last_unstale":"2024-09-21T09:55:31.246333+0000","last_undegraded":"2024-09-21T09:55:31.246333+0000","last_fullsized":"2024-09-21T09:55:31.246333+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:03:01.512522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.252004+0000","last_change":"2024-09-21T09:54:53.634082+0000","last_active":"2024-09-21T09:55:31.252004+0000","last_peered":"2024-09-21T09:55:31.252004+0000","last_clean":"2024-09-21T09:55:31.252004+0000","last_became_active":"2024-09-21T09:54:53.633812+0000","last_became_peered":"2024-09-21T09:54:53.633812+0000","last_unstale":"2024-09-21T09:55:31.252004+0000","last_undegraded":"2024-09-21T09:55:31.252004+0000","last_fullsized":"2024-09-21T09:55:31.252004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:47:20.955775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.245960+0000","last_change":"2024-09-21T09:54:53.723103+0000","last_active":"2024-09-21T09:55:31.245960+0000","last_peered":"2024-09-21T09:55:31.245960+0000","last_clean":"2024-09-21T09:55:31.245960+0000","last_became_active":"2024-09-21T09:54:53.722859+0000","last_became_peered":"2024-09-21T09:54:53.722859+0000","last_unstale":"2024-09-21T09:55:31.245960+0000","last_undegraded":"2024-09-21T09:55:31.245960+0000","last_fullsized":"2024-09-21T09:55:31.245960+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:03:58.582877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.252175+0000","last_change":"2024-09-21T09:54:53.634508+0000","last_active":"2024-09-21T09:55:31.252175+0000","last_peered":"2024-09-21T09:55:31.252175+0000","last_clean":"2024-09-21T09:55:31.252175+0000","last_became_active":"2024-09-21T09:54:53.634254+0000","last_became_peered":"2024-09-21T09:54:53.634254+0000","last_unstale":"2024-09-21T09:55:31.252175+0000","last_undegraded":"2024-09-21T09:55:31.252175+0000","last_fullsized":"2024-09-21T09:55:31.252175+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:18.122107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.248747+0000","last_change":"2024-09-21T09:54:54.677321+0000","last_active":"2024-09-21T09:55:31.248747+0000","last_peered":"2024-09-21T09:55:31.248747+0000","last_clean":"2024-09-21T09:55:31.248747+0000","last_became_active":"2024-09-21T09:54:54.676952+0000","last_became_peered":"2024-09-21T09:54:54.676952+0000","last_unstale":"2024-09-21T09:55:31.248747+0000","last_undegraded":"2024-09-21T09:55:31.248747+0000","last_fullsized":"2024-09-21T09:55:31.248747+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:21:06.467629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.249546+0000","last_change":"2024-09-21T09:54:53.619622+0000","last_active":"2024-09-21T09:55:31.249546+0000","last_peered":"2024-09-21T09:55:31.249546+0000","last_clean":"2024-09-21T09:55:31.249546+0000","last_became_active":"2024-09-21T09:54:53.619323+0000","last_became_peered":"2024-09-21T09:54:53.619323+0000","last_unstale":"2024-09-21T09:55:31.249546+0000","last_undegraded":"2024-09-21T09:55:31.249546+0000","last_fullsized":"2024-09-21T09:55:31.249546+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:25:19.988045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.245915+0000","last_change":"2024-09-21T09:54:53.638406+0000","last_active":"2024-09-21T09:55:31.245915+0000","last_peered":"2024-09-21T09:55:31.245915+0000","last_clean":"2024-09-21T09:55:31.245915+0000","last_became_active":"2024-09-21T09:54:53.638117+0000","last_became_peered":"2024-09-21T09:54:53.638117+0000","last_unstale":"2024-09-21T09:55:31.245915+0000","last_undegraded":"2024-09-21T09:55:31.245915+0000","last_fullsized":"2024-09-21T09:55:31.245915+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:15:06.633281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":19,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.242435+0000","last_change":"2024-09-21T09:54:53.627562+0000","last_active":"2024-09-21T09:55:31.242435+0000","last_peered":"2024-09-21T09:55:31.242435+0000","last_clean":"2024-09-21T09:55:31.242435+0000","last_became_active":"2024-09-21T09:54:53.627303+0000","last_became_peered":"2024-09-21T09:54:53.627303+0000","last_unstale":"2024-09-21T09:55:31.242435+0000","last_undegraded":"2024-09-21T09:55:31.242435+0000","last_fullsized":"2024-09-21T09:55:31.242435+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:03:15.883024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234020,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28372,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716780,"statfs":{"total":95995035648,"available":95965982720,"internally_reserved":0,"allocated":1380352,"data_stored":925744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52900000000000003,"15min":0.53100000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.70499999999999996,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.48899999999999999,"15min":0.48499999999999999},"min":{"1min":0.41099999999999998,"5min":0.214,"15min":0.214},"max":{"1min":0.65500000000000003,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.56699999999999995}]},{"osd":1,"last update":"Sat Sep 21 09:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.50600000000000001,"15min":0.504},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.36699999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.71199999999999997,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.47999999999999998}]},{"osd":2,"last update":"Sat Sep 21 09:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.5,"15min":0.498},"min":{"1min":0.376,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.69599999999999995,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.40999999999999998}]},{"osd":3,"last update":"Sat Sep 21 09:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.52000000000000002,"15min":0.51600000000000001},"min":{"1min":0.372,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.66500000000000004,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.377},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.52400000000000002,"15min":0.52200000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.72899999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.499}]},{"osd":4,"last update":"Sat Sep 21 09:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.42099999999999999},"min":{"1min":0.29499999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.499,"5min":0.45000000000000001,"15min":0.442},"min":{"1min":0.309,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.58999999999999997}]},{"osd":5,"last update":"Sat Sep 21 09:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.46400000000000002,"15min":0.46200000000000002},"min":{"1min":0.32400000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.58599999999999997,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.49199999999999999,"15min":0.48899999999999999},"min":{"1min":0.33600000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.67500000000000004,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.66600000000000004}]},{"osd":6,"last update":"Sat Sep 21 09:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.48699999999999999,"15min":0.48299999999999998},"min":{"1min":0.36499999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.65200000000000002,"5min":0.752,"15min":0.752},"last":0.624},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.501,"15min":0.495},"min":{"1min":0.35999999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66600000000000004,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.53300000000000003}]}]},{"osd":6,"up_from":46,"seq":197568495657,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28360,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716792,"statfs":{"total":95995035648,"available":95965995008,"internally_reserved":0,"allocated":1372160,"data_stored":925617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:54:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55200000000000005,"15min":0.55400000000000005},"min":{"1min":0.39500000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.34399999999999997,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.53300000000000003}]},{"osd":1,"last update":"Sat Sep 21 09:54:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54100000000000004,"15min":0.54400000000000004},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.68000000000000005,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.48999999999999999}]},{"osd":2,"last update":"Sat Sep 21 09:54:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.53600000000000003,"15min":0.53700000000000003},"min":{"1min":0.35699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.67800000000000005,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.54900000000000004,"15min":0.54800000000000004},"min":{"1min":0.39200000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.58199999999999996}]},{"osd":3,"last update":"Sat Sep 21 09:54:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.55100000000000005,"15min":0.55300000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.40300000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.59699999999999998}]},{"osd":4,"last update":"Sat Sep 21 09:54:43 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.42899999999999999,"15min":0.41599999999999998},"min":{"1min":0.32500000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.45800000000000002,"15min":0.44800000000000001},"min":{"1min":0.318,"5min":0.25,"15min":0.25},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.55700000000000005}]},{"osd":5,"last update":"Sat Sep 21 09:54:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.35999999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.628},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.47999999999999998,"15min":0.47199999999999998},"min":{"1min":0.36799999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.65400000000000003}]},{"osd":7,"last update":"Sat Sep 21 09:55:05 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48499999999999999,"15min":0.48399999999999999},"min":{"1min":0.30399999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.74299999999999999,"5min":0.745,"15min":0.745},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.53700000000000003,"15min":0.53300000000000003},"min":{"1min":0.34200000000000003,"5min":0.318,"15min":0.318},"max":{"1min":0.70499999999999996,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.67900000000000005}]}]},{"osd":5,"up_from":38,"seq":163208757294,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27772,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717380,"statfs":{"total":95995035648,"available":95966597120,"internally_reserved":0,"allocated":765952,"data_stored":333936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.28699999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.53900000000000003,"15min":0.54700000000000004},"min":{"1min":0.34699999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69099999999999995,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.57599999999999996}]},{"osd":1,"last update":"Sat Sep 21 09:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56000000000000005,"15min":0.56999999999999995},"min":{"1min":0.38300000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70699999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53900000000000003,"15min":0.54100000000000004},"min":{"1min":0.41499999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.69199999999999995,"5min":0.755,"15min":0.755},"last":0.54800000000000004}]},{"osd":2,"last update":"Sat Sep 21 09:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.39500000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.71999999999999997,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.40699999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.80300000000000005,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.59799999999999998}]},{"osd":3,"last update":"Sat Sep 21 09:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.54300000000000004,"15min":0.54800000000000004},"min":{"1min":0.42699999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.70199999999999996,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.62},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56799999999999995,"15min":0.57299999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72399999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.70999999999999996}]},{"osd":4,"last update":"Sat Sep 21 09:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.41799999999999998,"15min":0.40100000000000002},"min":{"1min":0.309,"5min":0.217,"15min":0.217},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.42799999999999999,"15min":0.40899999999999997},"min":{"1min":0.29499999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.63,"5min":1.24,"15min":1.24},"last":0.53500000000000003}]},{"osd":6,"last update":"Sat Sep 21 09:55:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.44800000000000001,"15min":0.438},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.66700000000000004,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.37},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.48399999999999999,"15min":0.47599999999999998},"min":{"1min":0.35199999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.72099999999999997,"5min":1.21,"15min":1.21},"last":0.38800000000000001}]},{"osd":7,"last update":"Sat Sep 21 09:55:05 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.55600000000000005,"15min":0.56399999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.67800000000000005,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.46999999999999997}]}]},{"osd":4,"up_from":30,"seq":128849018930,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27788,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717364,"statfs":{"total":95995035648,"available":95966580736,"internally_reserved":0,"allocated":782336,"data_stored":335306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.504,"15min":0.497},"min":{"1min":0.33400000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55600000000000005,"15min":0.54500000000000004},"min":{"1min":0.41899999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.73199999999999998}]},{"osd":1,"last update":"Sat Sep 21 09:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51100000000000001,"15min":0.51500000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.55900000000000005,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.52900000000000003,"15min":0.52100000000000002},"min":{"1min":0.39400000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.64300000000000002}]},{"osd":2,"last update":"Sat Sep 21 09:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.53700000000000003,"15min":0.53300000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52000000000000002,"15min":0.51100000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.68200000000000005,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.70699999999999996}]},{"osd":3,"last update":"Sat Sep 21 09:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53300000000000003,"15min":0.52500000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.72299999999999998,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.52600000000000002,"15min":0.51400000000000001},"min":{"1min":0.38300000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.60499999999999998}]},{"osd":5,"last update":"Sat Sep 21 09:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.39200000000000002,"15min":0.36899999999999999},"min":{"1min":0.32300000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.63,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.40000000000000002,"15min":0.375},"min":{"1min":0.29799999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.623}]},{"osd":6,"last update":"Sat Sep 21 09:55:41 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45300000000000001,"15min":0.436},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.42699999999999999,"15min":0.41299999999999998},"min":{"1min":0.30399999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.68700000000000006}]},{"osd":7,"last update":"Sat Sep 21 09:55:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.53600000000000003,"15min":0.53800000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71399999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.46500000000000002,"15min":0.46100000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.66000000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27764,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":761856,"data_stored":333686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:55:34 2024","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.34399999999999997,"15min":0.36399999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.45700000000000002,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.38800000000000001,"15min":0.40100000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.51700000000000002,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.504}]},{"osd":1,"last update":"Sat Sep 21 09:55:34 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.39600000000000002,"15min":0.41199999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.497,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.36099999999999999,"15min":0.374},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.44,"5min":0.52400000000000002,"15min":0.52400000000000002},"last":0.42599999999999999}]},{"osd":2,"last update":"Sat Sep 21 09:55:34 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.39300000000000002,"15min":0.40799999999999997},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.498,"5min":0.627,"15min":0.627},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.36799999999999999,"15min":0.375},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.45200000000000001,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.48099999999999998}]},{"osd":4,"last update":"Sat Sep 21 09:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.54500000000000004,"15min":0.52900000000000003},"min":{"1min":0.39900000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52600000000000002,"15min":0.50900000000000001},"min":{"1min":0.39700000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.68100000000000005}]},{"osd":5,"last update":"Sat Sep 21 09:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.56599999999999995,"15min":0.55800000000000005},"min":{"1min":0.39200000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.432,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.73599999999999999}]},{"osd":6,"last update":"Sat Sep 21 09:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.56200000000000006,"15min":0.54800000000000004},"min":{"1min":0.441,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.84499999999999997,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.58899999999999997,"15min":0.57999999999999996},"min":{"1min":0.48799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.95099999999999996,"5min":1.236,"15min":1.236},"last":0.69999999999999996}]},{"osd":7,"last update":"Sat Sep 21 09:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.58499999999999996,"15min":0.59099999999999997},"min":{"1min":0.42499999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.56499999999999995,"15min":0.55700000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.66500000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31868,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713284,"statfs":{"total":95995035648,"available":95962402816,"internally_reserved":0,"allocated":765952,"data_stored":334212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.23899999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.41399999999999998,"5min":1.125,"15min":1.125},"last":0.217},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.38300000000000001,"15min":0.371},"min":{"1min":0.24199999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.48899999999999999,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.245}]},{"osd":1,"last update":"Sat Sep 21 09:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.38500000000000001,"15min":0.38800000000000001},"min":{"1min":0.26200000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.44600000000000001,"5min":1.161,"15min":1.161},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.40500000000000003,"15min":0.40100000000000002},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.51500000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.28399999999999997}]},{"osd":3,"last update":"Sat Sep 21 09:55:31 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.41699999999999998,"15min":0.42799999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.51400000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.318},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.40200000000000002,"15min":0.41999999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.48099999999999998,"5min":1.052,"15min":1.052},"last":0.30599999999999999}]},{"osd":4,"last update":"Sat Sep 21 09:54:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56299999999999994,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57699999999999996,"15min":0.56699999999999995},"min":{"1min":0.432,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.83499999999999996,"5min":1.012,"15min":1.012},"last":0.35699999999999998}]},{"osd":5,"last update":"Sat Sep 21 09:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.623,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77100000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.48099999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85199999999999998,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.442}]},{"osd":6,"last update":"Sat Sep 21 09:55:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.80400000000000005,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.373},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.621,"15min":0.63900000000000001},"min":{"1min":0.42999999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79400000000000004,"5min":1.202,"15min":1.202},"last":0.43099999999999999}]},{"osd":7,"last update":"Sat Sep 21 09:55:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.59299999999999997,"15min":0.59499999999999997},"min":{"1min":0.46400000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.83599999999999997,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.64200000000000002,"15min":0.65200000000000002},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.77200000000000002,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.38400000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27764,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":761856,"data_stored":333615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:54:46 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.32900000000000001,"15min":0.315},"min":{"1min":0.224,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.47399999999999998,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.34999999999999998,"15min":0.32300000000000001},"min":{"1min":0.20699999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.437}]},{"osd":2,"last update":"Sat Sep 21 09:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34699999999999998,"15min":0.34000000000000002},"min":{"1min":0.247,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.38600000000000001,"15min":0.38200000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.33900000000000002}]},{"osd":3,"last update":"Sat Sep 21 09:55:42 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.39100000000000001,"15min":0.40699999999999997},"min":{"1min":0.28799999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.39600000000000002,"15min":0.40600000000000003},"min":{"1min":0.27100000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.44800000000000001}]},{"osd":4,"last update":"Sat Sep 21 09:54:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54300000000000004,"15min":0.54900000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.71499999999999997,"5min":1.022,"15min":1.022},"last":0.67400000000000004}]},{"osd":5,"last update":"Sat Sep 21 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.59299999999999997,"15min":0.60399999999999998},"min":{"1min":0.42599999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.59899999999999998,"15min":0.60499999999999998},"min":{"1min":0.42699999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.64700000000000002}]},{"osd":6,"last update":"Sat Sep 21 09:55:42 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.57099999999999995,"15min":0.57699999999999996},"min":{"1min":0.43099999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.56000000000000005,"15min":0.55200000000000005},"min":{"1min":0.439,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.71699999999999997}]},{"osd":7,"last update":"Sat Sep 21 09:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.54000000000000004,"15min":0.53500000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.51000000000000001,"15min":0.502},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.65500000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28352,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716800,"statfs":{"total":95995035648,"available":95966003200,"internally_reserved":0,"allocated":1359872,"data_stored":924568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 09:54:49 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.29399999999999998},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.45500000000000002,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.35499999999999998,"15min":0.32100000000000001},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.60699999999999998}]},{"osd":2,"last update":"Sat Sep 21 09:55:11 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.38200000000000001,"15min":0.35799999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.56200000000000006,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.36199999999999999,"15min":0.34399999999999997},"min":{"1min":0.25700000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.51000000000000001,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.45900000000000002}]},{"osd":3,"last update":"Sat Sep 21 09:55:32 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.42299999999999999,"15min":0.433},"min":{"1min":0.30099999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53100000000000003,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.40999999999999998,"15min":0.41699999999999998},"min":{"1min":0.311,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.53300000000000003,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.51500000000000001}]},{"osd":4,"last update":"Sat Sep 21 09:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55300000000000005,"15min":0.55700000000000005},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.79700000000000004,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.90000000000000002,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.627}]},{"osd":5,"last update":"Sat Sep 21 09:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.81100000000000005,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59499999999999997,"15min":0.59999999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.83299999999999996,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.68200000000000005}]},{"osd":6,"last update":"Sat Sep 21 09:54:44 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.83899999999999997,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57699999999999996,"15min":0.57599999999999996},"min":{"1min":0.432,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.75900000000000001,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.53100000000000003}]},{"osd":7,"last update":"Sat Sep 21 09:55:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.57399999999999995,"15min":0.57799999999999996},"min":{"1min":0.38600000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73099999999999998,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60099999999999998,"15min":0.59999999999999998},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84499999999999997,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.56499999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T09:55:51.522 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T09:55:51.721 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:52.764 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:52 smithi070 ceph-mon[29589]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:55:52.764 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:52 smithi070 ceph-mon[29589]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:52.764 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:52 smithi070 ceph-mon[41533]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:55:52.764 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:52 smithi070 ceph-mon[41533]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:52 smithi160 ceph-mon[33203]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:55:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:52 smithi160 ceph-mon[33203]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:53.642 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:55:53.644 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T09:55:54.351 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2024-09-21T09:55:53.171591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592749,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":226,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":170,"ondisk_log_size":170,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":228040,"kb_used_data":7764,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":749733176,"statfs":{"total":767960285184,"available":767726772224,"internally_reserved":0,"allocated":7950336,"data_stored":4446684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12741,"internal_metadata":225431099},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002615"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.249270+0000","last_change":"2024-09-21T09:54:54.633334+0000","last_active":"2024-09-21T09:55:31.249270+0000","last_peered":"2024-09-21T09:55:31.249270+0000","last_clean":"2024-09-21T09:55:31.249270+0000","last_became_active":"2024-09-21T09:54:54.632922+0000","last_became_peered":"2024-09-21T09:54:54.632922+0000","last_unstale":"2024-09-21T09:55:31.249270+0000","last_undegraded":"2024-09-21T09:55:31.249270+0000","last_fullsized":"2024-09-21T09:55:31.249270+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:56.286918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.249308+0000","last_change":"2024-09-21T09:54:53.624613+0000","last_active":"2024-09-21T09:55:31.249308+0000","last_peered":"2024-09-21T09:55:31.249308+0000","last_clean":"2024-09-21T09:55:31.249308+0000","last_became_active":"2024-09-21T09:54:53.624346+0000","last_became_peered":"2024-09-21T09:54:53.624346+0000","last_unstale":"2024-09-21T09:55:31.249308+0000","last_undegraded":"2024-09-21T09:55:31.249308+0000","last_fullsized":"2024-09-21T09:55:31.249308+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:37:37.666312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.252080+0000","last_change":"2024-09-21T09:54:54.631581+0000","last_active":"2024-09-21T09:55:31.252080+0000","last_peered":"2024-09-21T09:55:31.252080+0000","last_clean":"2024-09-21T09:55:31.252080+0000","last_became_active":"2024-09-21T09:54:54.631271+0000","last_became_peered":"2024-09-21T09:54:54.631271+0000","last_unstale":"2024-09-21T09:55:31.252080+0000","last_undegraded":"2024-09-21T09:55:31.252080+0000","last_fullsized":"2024-09-21T09:55:31.252080+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:44:16.275479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.242097+0000","last_change":"2024-09-21T09:54:53.623601+0000","last_active":"2024-09-21T09:55:31.242097+0000","last_peered":"2024-09-21T09:55:31.242097+0000","last_clean":"2024-09-21T09:55:31.242097+0000","last_became_active":"2024-09-21T09:54:53.623319+0000","last_became_peered":"2024-09-21T09:54:53.623319+0000","last_unstale":"2024-09-21T09:55:31.242097+0000","last_undegraded":"2024-09-21T09:55:31.242097+0000","last_fullsized":"2024-09-21T09:55:31.242097+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:33:11.417229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.242947+0000","last_change":"2024-09-21T09:54:54.632027+0000","last_active":"2024-09-21T09:55:31.242947+0000","last_peered":"2024-09-21T09:55:31.242947+0000","last_clean":"2024-09-21T09:55:31.242947+0000","last_became_active":"2024-09-21T09:54:54.631708+0000","last_became_peered":"2024-09-21T09:54:54.631708+0000","last_unstale":"2024-09-21T09:55:31.242947+0000","last_undegraded":"2024-09-21T09:55:31.242947+0000","last_fullsized":"2024-09-21T09:55:31.242947+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:13:11.557171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.249379+0000","last_change":"2024-09-21T09:54:53.625016+0000","last_active":"2024-09-21T09:55:31.249379+0000","last_peered":"2024-09-21T09:55:31.249379+0000","last_clean":"2024-09-21T09:55:31.249379+0000","last_became_active":"2024-09-21T09:54:53.624750+0000","last_became_peered":"2024-09-21T09:54:53.624750+0000","last_unstale":"2024-09-21T09:55:31.249379+0000","last_undegraded":"2024-09-21T09:55:31.249379+0000","last_fullsized":"2024-09-21T09:55:31.249379+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:34:26.823544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.249679+0000","last_change":"2024-09-21T09:54:54.633218+0000","last_active":"2024-09-21T09:55:31.249679+0000","last_peered":"2024-09-21T09:55:31.249679+0000","last_clean":"2024-09-21T09:55:31.249679+0000","last_became_active":"2024-09-21T09:54:54.632912+0000","last_became_peered":"2024-09-21T09:54:54.632912+0000","last_unstale":"2024-09-21T09:55:31.249679+0000","last_undegraded":"2024-09-21T09:55:31.249679+0000","last_fullsized":"2024-09-21T09:55:31.249679+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:24:09.344461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":23,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.247193+0000","last_change":"2024-09-21T09:54:53.625692+0000","last_active":"2024-09-21T09:55:31.247193+0000","last_peered":"2024-09-21T09:55:31.247193+0000","last_clean":"2024-09-21T09:55:31.247193+0000","last_became_active":"2024-09-21T09:54:53.625372+0000","last_became_peered":"2024-09-21T09:54:53.625372+0000","last_unstale":"2024-09-21T09:55:31.247193+0000","last_undegraded":"2024-09-21T09:55:31.247193+0000","last_fullsized":"2024-09-21T09:55:31.247193+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:32.289423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.249845+0000","last_change":"2024-09-21T09:54:53.625967+0000","last_active":"2024-09-21T09:55:31.249845+0000","last_peered":"2024-09-21T09:55:31.249845+0000","last_clean":"2024-09-21T09:55:31.249845+0000","last_became_active":"2024-09-21T09:54:53.625650+0000","last_became_peered":"2024-09-21T09:54:53.625650+0000","last_unstale":"2024-09-21T09:55:31.249845+0000","last_undegraded":"2024-09-21T09:55:31.249845+0000","last_fullsized":"2024-09-21T09:55:31.249845+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:42:10.159809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.251843+0000","last_change":"2024-09-21T09:54:54.677572+0000","last_active":"2024-09-21T09:55:31.251843+0000","last_peered":"2024-09-21T09:55:31.251843+0000","last_clean":"2024-09-21T09:55:31.251843+0000","last_became_active":"2024-09-21T09:54:54.677253+0000","last_became_peered":"2024-09-21T09:54:54.677253+0000","last_unstale":"2024-09-21T09:55:31.251843+0000","last_undegraded":"2024-09-21T09:55:31.251843+0000","last_fullsized":"2024-09-21T09:55:31.251843+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:31:40.629299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.249755+0000","last_change":"2024-09-21T09:54:53.624164+0000","last_active":"2024-09-21T09:55:31.249755+0000","last_peered":"2024-09-21T09:55:31.249755+0000","last_clean":"2024-09-21T09:55:31.249755+0000","last_became_active":"2024-09-21T09:54:53.623875+0000","last_became_peered":"2024-09-21T09:54:53.623875+0000","last_unstale":"2024-09-21T09:55:31.249755+0000","last_undegraded":"2024-09-21T09:55:31.249755+0000","last_fullsized":"2024-09-21T09:55:31.249755+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:11:56.692776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.246037+0000","last_change":"2024-09-21T09:54:53.721961+0000","last_active":"2024-09-21T09:55:31.246037+0000","last_peered":"2024-09-21T09:55:31.246037+0000","last_clean":"2024-09-21T09:55:31.246037+0000","last_became_active":"2024-09-21T09:54:53.721676+0000","last_became_peered":"2024-09-21T09:54:53.721676+0000","last_unstale":"2024-09-21T09:55:31.246037+0000","last_undegraded":"2024-09-21T09:55:31.246037+0000","last_fullsized":"2024-09-21T09:55:31.246037+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:14:40.284754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.249624+0000","last_change":"2024-09-21T09:54:53.628923+0000","last_active":"2024-09-21T09:55:31.249624+0000","last_peered":"2024-09-21T09:55:31.249624+0000","last_clean":"2024-09-21T09:55:31.249624+0000","last_became_active":"2024-09-21T09:54:53.627939+0000","last_became_peered":"2024-09-21T09:54:53.627939+0000","last_unstale":"2024-09-21T09:55:31.249624+0000","last_undegraded":"2024-09-21T09:55:31.249624+0000","last_fullsized":"2024-09-21T09:55:31.249624+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:07:02.820689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T09:55:32.413399+0000","last_change":"2024-09-21T09:55:32.413399+0000","last_active":"2024-09-21T09:55:32.413399+0000","last_peered":"2024-09-21T09:55:32.413399+0000","last_clean":"2024-09-21T09:55:32.413399+0000","last_became_active":"2024-09-21T09:55:32.413050+0000","last_became_peered":"2024-09-21T09:55:32.413050+0000","last_unstale":"2024-09-21T09:55:32.413399+0000","last_undegraded":"2024-09-21T09:55:32.413399+0000","last_fullsized":"2024-09-21T09:55:32.413399+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:30.986060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.252037+0000","last_change":"2024-09-21T09:54:53.632932+0000","last_active":"2024-09-21T09:55:31.252037+0000","last_peered":"2024-09-21T09:55:31.252037+0000","last_clean":"2024-09-21T09:55:31.252037+0000","last_became_active":"2024-09-21T09:54:53.632652+0000","last_became_peered":"2024-09-21T09:54:53.632652+0000","last_unstale":"2024-09-21T09:55:31.252037+0000","last_undegraded":"2024-09-21T09:55:31.252037+0000","last_fullsized":"2024-09-21T09:55:31.252037+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:01.436815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.242451+0000","last_change":"2024-09-21T09:54:54.645001+0000","last_active":"2024-09-21T09:55:31.242451+0000","last_peered":"2024-09-21T09:55:31.242451+0000","last_clean":"2024-09-21T09:55:31.242451+0000","last_became_active":"2024-09-21T09:54:54.644635+0000","last_became_peered":"2024-09-21T09:54:54.644635+0000","last_unstale":"2024-09-21T09:55:31.242451+0000","last_undegraded":"2024-09-21T09:55:31.242451+0000","last_fullsized":"2024-09-21T09:55:31.242451+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:46:24.237543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.249097+0000","last_change":"2024-09-21T09:54:53.631134+0000","last_active":"2024-09-21T09:55:31.249097+0000","last_peered":"2024-09-21T09:55:31.249097+0000","last_clean":"2024-09-21T09:55:31.249097+0000","last_became_active":"2024-09-21T09:54:53.630853+0000","last_became_peered":"2024-09-21T09:54:53.630853+0000","last_unstale":"2024-09-21T09:55:31.249097+0000","last_undegraded":"2024-09-21T09:55:31.249097+0000","last_fullsized":"2024-09-21T09:55:31.249097+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:48:45.503176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.246182+0000","last_change":"2024-09-21T09:54:53.723089+0000","last_active":"2024-09-21T09:55:31.246182+0000","last_peered":"2024-09-21T09:55:31.246182+0000","last_clean":"2024-09-21T09:55:31.246182+0000","last_became_active":"2024-09-21T09:54:53.722813+0000","last_became_peered":"2024-09-21T09:54:53.722813+0000","last_unstale":"2024-09-21T09:55:31.246182+0000","last_undegraded":"2024-09-21T09:55:31.246182+0000","last_fullsized":"2024-09-21T09:55:31.246182+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:23.685574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.247988+0000","last_change":"2024-09-21T09:54:53.626364+0000","last_active":"2024-09-21T09:55:31.247988+0000","last_peered":"2024-09-21T09:55:31.247988+0000","last_clean":"2024-09-21T09:55:31.247988+0000","last_became_active":"2024-09-21T09:54:53.626082+0000","last_became_peered":"2024-09-21T09:54:53.626082+0000","last_unstale":"2024-09-21T09:55:31.247988+0000","last_undegraded":"2024-09-21T09:55:31.247988+0000","last_fullsized":"2024-09-21T09:55:31.247988+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:22:16.136523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.246788+0000","last_change":"2024-09-21T09:54:53.638354+0000","last_active":"2024-09-21T09:55:31.246788+0000","last_peered":"2024-09-21T09:55:31.246788+0000","last_clean":"2024-09-21T09:55:31.246788+0000","last_became_active":"2024-09-21T09:54:53.638077+0000","last_became_peered":"2024-09-21T09:54:53.638077+0000","last_unstale":"2024-09-21T09:55:31.246788+0000","last_undegraded":"2024-09-21T09:55:31.246788+0000","last_fullsized":"2024-09-21T09:55:31.246788+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:07:36.028409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.246273+0000","last_change":"2024-09-21T09:54:53.625145+0000","last_active":"2024-09-21T09:55:31.246273+0000","last_peered":"2024-09-21T09:55:31.246273+0000","last_clean":"2024-09-21T09:55:31.246273+0000","last_became_active":"2024-09-21T09:54:53.624845+0000","last_became_peered":"2024-09-21T09:54:53.624845+0000","last_unstale":"2024-09-21T09:55:31.246273+0000","last_undegraded":"2024-09-21T09:55:31.246273+0000","last_fullsized":"2024-09-21T09:55:31.246273+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:44.551367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.246537+0000","last_change":"2024-09-21T09:54:53.638897+0000","last_active":"2024-09-21T09:55:31.246537+0000","last_peered":"2024-09-21T09:55:31.246537+0000","last_clean":"2024-09-21T09:55:31.246537+0000","last_became_active":"2024-09-21T09:54:53.638624+0000","last_became_peered":"2024-09-21T09:54:53.638624+0000","last_unstale":"2024-09-21T09:55:31.246537+0000","last_undegraded":"2024-09-21T09:55:31.246537+0000","last_fullsized":"2024-09-21T09:55:31.246537+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:48:33.305248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.247453+0000","last_change":"2024-09-21T09:54:53.625098+0000","last_active":"2024-09-21T09:55:31.247453+0000","last_peered":"2024-09-21T09:55:31.247453+0000","last_clean":"2024-09-21T09:55:31.247453+0000","last_became_active":"2024-09-21T09:54:53.624771+0000","last_became_peered":"2024-09-21T09:54:53.624771+0000","last_unstale":"2024-09-21T09:55:31.247453+0000","last_undegraded":"2024-09-21T09:55:31.247453+0000","last_fullsized":"2024-09-21T09:55:31.247453+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:54:59.433791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.244910+0000","last_change":"2024-09-21T09:54:53.722001+0000","last_active":"2024-09-21T09:55:31.244910+0000","last_peered":"2024-09-21T09:55:31.244910+0000","last_clean":"2024-09-21T09:55:31.244910+0000","last_became_active":"2024-09-21T09:54:53.721570+0000","last_became_peered":"2024-09-21T09:54:53.721570+0000","last_unstale":"2024-09-21T09:55:31.244910+0000","last_undegraded":"2024-09-21T09:55:31.244910+0000","last_fullsized":"2024-09-21T09:55:31.244910+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:49:10.088299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"59'128","reported_seq":269,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.249432+0000","last_change":"2024-09-21T09:53:05.275142+0000","last_active":"2024-09-21T09:55:31.249432+0000","last_peered":"2024-09-21T09:55:31.249432+0000","last_clean":"2024-09-21T09:55:31.249432+0000","last_became_active":"2024-09-21T09:53:05.274635+0000","last_became_peered":"2024-09-21T09:53:05.274635+0000","last_unstale":"2024-09-21T09:55:31.249432+0000","last_undegraded":"2024-09-21T09:55:31.249432+0000","last_fullsized":"2024-09-21T09:55:31.249432+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:31.035143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T09:55:32.435325+0000","last_change":"2024-09-21T09:53:49.219350+0000","last_active":"2024-09-21T09:55:32.435325+0000","last_peered":"2024-09-21T09:55:32.435325+0000","last_clean":"2024-09-21T09:55:32.435325+0000","last_became_active":"2024-09-21T09:53:43.189467+0000","last_became_peered":"2024-09-21T09:53:43.189467+0000","last_unstale":"2024-09-21T09:55:32.435325+0000","last_undegraded":"2024-09-21T09:55:32.435325+0000","last_fullsized":"2024-09-21T09:55:32.435325+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:37.943058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.248961+0000","last_change":"2024-09-21T09:54:53.638100+0000","last_active":"2024-09-21T09:55:31.248961+0000","last_peered":"2024-09-21T09:55:31.248961+0000","last_clean":"2024-09-21T09:55:31.248961+0000","last_became_active":"2024-09-21T09:54:53.636403+0000","last_became_peered":"2024-09-21T09:54:53.636403+0000","last_unstale":"2024-09-21T09:55:31.248961+0000","last_undegraded":"2024-09-21T09:55:31.248961+0000","last_fullsized":"2024-09-21T09:55:31.248961+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:31:07.327629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.247147+0000","last_change":"2024-09-21T09:54:53.722213+0000","last_active":"2024-09-21T09:55:31.247147+0000","last_peered":"2024-09-21T09:55:31.247147+0000","last_clean":"2024-09-21T09:55:31.247147+0000","last_became_active":"2024-09-21T09:54:53.721437+0000","last_became_peered":"2024-09-21T09:54:53.721437+0000","last_unstale":"2024-09-21T09:55:31.247147+0000","last_undegraded":"2024-09-21T09:55:31.247147+0000","last_fullsized":"2024-09-21T09:55:31.247147+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:01:41.456731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T09:55:32.412392+0000","last_change":"2024-09-21T09:53:49.374842+0000","last_active":"2024-09-21T09:55:32.412392+0000","last_peered":"2024-09-21T09:55:32.412392+0000","last_clean":"2024-09-21T09:55:32.412392+0000","last_became_active":"2024-09-21T09:53:43.191101+0000","last_became_peered":"2024-09-21T09:53:43.191101+0000","last_unstale":"2024-09-21T09:55:32.412392+0000","last_undegraded":"2024-09-21T09:55:32.412392+0000","last_fullsized":"2024-09-21T09:55:32.412392+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:24:33.044234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.242119+0000","last_change":"2024-09-21T09:54:53.626864+0000","last_active":"2024-09-21T09:55:31.242119+0000","last_peered":"2024-09-21T09:55:31.242119+0000","last_clean":"2024-09-21T09:55:31.242119+0000","last_became_active":"2024-09-21T09:54:53.626612+0000","last_became_peered":"2024-09-21T09:54:53.626612+0000","last_unstale":"2024-09-21T09:55:31.242119+0000","last_undegraded":"2024-09-21T09:55:31.242119+0000","last_fullsized":"2024-09-21T09:55:31.242119+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:00:19.195266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.242870+0000","last_change":"2024-09-21T09:54:54.677306+0000","last_active":"2024-09-21T09:55:31.242870+0000","last_peered":"2024-09-21T09:55:31.242870+0000","last_clean":"2024-09-21T09:55:31.242870+0000","last_became_active":"2024-09-21T09:54:54.677012+0000","last_became_peered":"2024-09-21T09:54:54.677012+0000","last_unstale":"2024-09-21T09:55:31.242870+0000","last_undegraded":"2024-09-21T09:55:31.242870+0000","last_fullsized":"2024-09-21T09:55:31.242870+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:21:18.119360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T09:55:32.243192+0000","last_change":"2024-09-21T09:53:49.211722+0000","last_active":"2024-09-21T09:55:32.243192+0000","last_peered":"2024-09-21T09:55:32.243192+0000","last_clean":"2024-09-21T09:55:32.243192+0000","last_became_active":"2024-09-21T09:53:43.196820+0000","last_became_peered":"2024-09-21T09:53:43.196820+0000","last_unstale":"2024-09-21T09:55:32.243192+0000","last_undegraded":"2024-09-21T09:55:32.243192+0000","last_fullsized":"2024-09-21T09:55:32.243192+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:24.782773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.243621+0000","last_change":"2024-09-21T09:54:53.622210+0000","last_active":"2024-09-21T09:55:31.243621+0000","last_peered":"2024-09-21T09:55:31.243621+0000","last_clean":"2024-09-21T09:55:31.243621+0000","last_became_active":"2024-09-21T09:54:53.621855+0000","last_became_peered":"2024-09-21T09:54:53.621855+0000","last_unstale":"2024-09-21T09:55:31.243621+0000","last_undegraded":"2024-09-21T09:55:31.243621+0000","last_fullsized":"2024-09-21T09:55:31.243621+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:47:17.990851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.244978+0000","last_change":"2024-09-21T09:54:53.635976+0000","last_active":"2024-09-21T09:55:31.244978+0000","last_peered":"2024-09-21T09:55:31.244978+0000","last_clean":"2024-09-21T09:55:31.244978+0000","last_became_active":"2024-09-21T09:54:53.635592+0000","last_became_peered":"2024-09-21T09:54:53.635592+0000","last_unstale":"2024-09-21T09:55:31.244978+0000","last_undegraded":"2024-09-21T09:55:31.244978+0000","last_fullsized":"2024-09-21T09:55:31.244978+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:44.327770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":30,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T09:55:32.435370+0000","last_change":"2024-09-21T09:53:49.219893+0000","last_active":"2024-09-21T09:55:32.435370+0000","last_peered":"2024-09-21T09:55:32.435370+0000","last_clean":"2024-09-21T09:55:32.435370+0000","last_became_active":"2024-09-21T09:53:43.189073+0000","last_became_peered":"2024-09-21T09:53:43.189073+0000","last_unstale":"2024-09-21T09:55:32.435370+0000","last_undegraded":"2024-09-21T09:55:32.435370+0000","last_fullsized":"2024-09-21T09:55:32.435370+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:48.257597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084477399999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.246441+0000","last_change":"2024-09-21T09:54:53.626386+0000","last_active":"2024-09-21T09:55:31.246441+0000","last_peered":"2024-09-21T09:55:31.246441+0000","last_clean":"2024-09-21T09:55:31.246441+0000","last_became_active":"2024-09-21T09:54:53.625344+0000","last_became_peered":"2024-09-21T09:54:53.625344+0000","last_unstale":"2024-09-21T09:55:31.246441+0000","last_undegraded":"2024-09-21T09:55:31.246441+0000","last_fullsized":"2024-09-21T09:55:31.246441+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:28:40.301087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.244358+0000","last_change":"2024-09-21T09:54:53.724007+0000","last_active":"2024-09-21T09:55:31.244358+0000","last_peered":"2024-09-21T09:55:31.244358+0000","last_clean":"2024-09-21T09:55:31.244358+0000","last_became_active":"2024-09-21T09:54:53.723726+0000","last_became_peered":"2024-09-21T09:54:53.723726+0000","last_unstale":"2024-09-21T09:55:31.244358+0000","last_undegraded":"2024-09-21T09:55:31.244358+0000","last_fullsized":"2024-09-21T09:55:31.244358+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:08:37.741014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.252164+0000","last_change":"2024-09-21T09:54:53.633398+0000","last_active":"2024-09-21T09:55:31.252164+0000","last_peered":"2024-09-21T09:55:31.252164+0000","last_clean":"2024-09-21T09:55:31.252164+0000","last_became_active":"2024-09-21T09:54:53.633146+0000","last_became_peered":"2024-09-21T09:54:53.633146+0000","last_unstale":"2024-09-21T09:55:31.252164+0000","last_undegraded":"2024-09-21T09:55:31.252164+0000","last_fullsized":"2024-09-21T09:55:31.252164+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:10:19.183476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T09:55:32.244430+0000","last_change":"2024-09-21T09:53:49.211745+0000","last_active":"2024-09-21T09:55:32.244430+0000","last_peered":"2024-09-21T09:55:32.244430+0000","last_clean":"2024-09-21T09:55:32.244430+0000","last_became_active":"2024-09-21T09:53:43.189852+0000","last_became_peered":"2024-09-21T09:53:43.189852+0000","last_unstale":"2024-09-21T09:55:32.244430+0000","last_undegraded":"2024-09-21T09:55:32.244430+0000","last_fullsized":"2024-09-21T09:55:32.244430+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:16.083394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.243928+0000","last_change":"2024-09-21T09:54:53.635401+0000","last_active":"2024-09-21T09:55:31.243928+0000","last_peered":"2024-09-21T09:55:31.243928+0000","last_clean":"2024-09-21T09:55:31.243928+0000","last_became_active":"2024-09-21T09:54:53.635044+0000","last_became_peered":"2024-09-21T09:54:53.635044+0000","last_unstale":"2024-09-21T09:55:31.243928+0000","last_undegraded":"2024-09-21T09:55:31.243928+0000","last_fullsized":"2024-09-21T09:55:31.243928+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:40:49.188614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.244445+0000","last_change":"2024-09-21T09:54:53.624476+0000","last_active":"2024-09-21T09:55:31.244445+0000","last_peered":"2024-09-21T09:55:31.244445+0000","last_clean":"2024-09-21T09:55:31.244445+0000","last_became_active":"2024-09-21T09:54:53.624223+0000","last_became_peered":"2024-09-21T09:54:53.624223+0000","last_unstale":"2024-09-21T09:55:31.244445+0000","last_undegraded":"2024-09-21T09:55:31.244445+0000","last_fullsized":"2024-09-21T09:55:31.244445+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:47:05.700214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T09:55:32.412617+0000","last_change":"2024-09-21T09:53:49.374824+0000","last_active":"2024-09-21T09:55:32.412617+0000","last_peered":"2024-09-21T09:55:32.412617+0000","last_clean":"2024-09-21T09:55:32.412617+0000","last_became_active":"2024-09-21T09:53:43.197315+0000","last_became_peered":"2024-09-21T09:53:43.197315+0000","last_unstale":"2024-09-21T09:55:32.412617+0000","last_undegraded":"2024-09-21T09:55:32.412617+0000","last_fullsized":"2024-09-21T09:55:32.412617+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:49:20.376464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.242523+0000","last_change":"2024-09-21T09:54:54.630089+0000","last_active":"2024-09-21T09:55:31.242523+0000","last_peered":"2024-09-21T09:55:31.242523+0000","last_clean":"2024-09-21T09:55:31.242523+0000","last_became_active":"2024-09-21T09:54:54.629786+0000","last_became_peered":"2024-09-21T09:54:54.629786+0000","last_unstale":"2024-09-21T09:55:31.242523+0000","last_undegraded":"2024-09-21T09:55:31.242523+0000","last_fullsized":"2024-09-21T09:55:31.242523+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:01:39.916299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.249796+0000","last_change":"2024-09-21T09:54:53.627677+0000","last_active":"2024-09-21T09:55:31.249796+0000","last_peered":"2024-09-21T09:55:31.249796+0000","last_clean":"2024-09-21T09:55:31.249796+0000","last_became_active":"2024-09-21T09:54:53.627142+0000","last_became_peered":"2024-09-21T09:54:53.627142+0000","last_unstale":"2024-09-21T09:55:31.249796+0000","last_undegraded":"2024-09-21T09:55:31.249796+0000","last_fullsized":"2024-09-21T09:55:31.249796+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:36.818257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.244909+0000","last_change":"2024-09-21T09:53:49.211757+0000","last_active":"2024-09-21T09:55:31.244909+0000","last_peered":"2024-09-21T09:55:31.244909+0000","last_clean":"2024-09-21T09:55:31.244909+0000","last_became_active":"2024-09-21T09:53:43.197526+0000","last_became_peered":"2024-09-21T09:53:43.197526+0000","last_unstale":"2024-09-21T09:55:31.244909+0000","last_undegraded":"2024-09-21T09:55:31.244909+0000","last_fullsized":"2024-09-21T09:55:31.244909+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:23:27.608603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035589600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.247301+0000","last_change":"2024-09-21T09:54:54.631615+0000","last_active":"2024-09-21T09:55:31.247301+0000","last_peered":"2024-09-21T09:55:31.247301+0000","last_clean":"2024-09-21T09:55:31.247301+0000","last_became_active":"2024-09-21T09:54:54.631222+0000","last_became_peered":"2024-09-21T09:54:54.631222+0000","last_unstale":"2024-09-21T09:55:31.247301+0000","last_undegraded":"2024-09-21T09:55:31.247301+0000","last_fullsized":"2024-09-21T09:55:31.247301+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:16:19.613693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.243035+0000","last_change":"2024-09-21T09:54:53.626192+0000","last_active":"2024-09-21T09:55:31.243035+0000","last_peered":"2024-09-21T09:55:31.243035+0000","last_clean":"2024-09-21T09:55:31.243035+0000","last_became_active":"2024-09-21T09:54:53.625916+0000","last_became_peered":"2024-09-21T09:54:53.625916+0000","last_unstale":"2024-09-21T09:55:31.243035+0000","last_undegraded":"2024-09-21T09:55:31.243035+0000","last_fullsized":"2024-09-21T09:55:31.243035+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:57:48.369381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.247388+0000","last_change":"2024-09-21T09:53:49.220984+0000","last_active":"2024-09-21T09:55:31.247388+0000","last_peered":"2024-09-21T09:55:31.247388+0000","last_clean":"2024-09-21T09:55:31.247388+0000","last_became_active":"2024-09-21T09:53:43.190004+0000","last_became_peered":"2024-09-21T09:53:43.190004+0000","last_unstale":"2024-09-21T09:55:31.247388+0000","last_undegraded":"2024-09-21T09:55:31.247388+0000","last_fullsized":"2024-09-21T09:55:31.247388+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:26:21.202894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.246041+0000","last_change":"2024-09-21T09:54:53.636902+0000","last_active":"2024-09-21T09:55:31.246041+0000","last_peered":"2024-09-21T09:55:31.246041+0000","last_clean":"2024-09-21T09:55:31.246041+0000","last_became_active":"2024-09-21T09:54:53.636616+0000","last_became_peered":"2024-09-21T09:54:53.636616+0000","last_unstale":"2024-09-21T09:55:31.246041+0000","last_undegraded":"2024-09-21T09:55:31.246041+0000","last_fullsized":"2024-09-21T09:55:31.246041+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:46:47.694495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.246319+0000","last_change":"2024-09-21T09:54:53.638962+0000","last_active":"2024-09-21T09:55:31.246319+0000","last_peered":"2024-09-21T09:55:31.246319+0000","last_clean":"2024-09-21T09:55:31.246319+0000","last_became_active":"2024-09-21T09:54:53.638098+0000","last_became_peered":"2024-09-21T09:54:53.638098+0000","last_unstale":"2024-09-21T09:55:31.246319+0000","last_undegraded":"2024-09-21T09:55:31.246319+0000","last_fullsized":"2024-09-21T09:55:31.246319+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:30:32.467009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"71'6","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.246282+0000","last_change":"2024-09-21T09:54:53.624887+0000","last_active":"2024-09-21T09:55:31.246282+0000","last_peered":"2024-09-21T09:55:31.246282+0000","last_clean":"2024-09-21T09:55:31.246282+0000","last_became_active":"2024-09-21T09:54:53.624525+0000","last_became_peered":"2024-09-21T09:54:53.624525+0000","last_unstale":"2024-09-21T09:55:31.246282+0000","last_undegraded":"2024-09-21T09:55:31.246282+0000","last_fullsized":"2024-09-21T09:55:31.246282+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:05:59.605316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.244044+0000","last_change":"2024-09-21T09:54:53.637302+0000","last_active":"2024-09-21T09:55:31.244044+0000","last_peered":"2024-09-21T09:55:31.244044+0000","last_clean":"2024-09-21T09:55:31.244044+0000","last_became_active":"2024-09-21T09:54:53.637019+0000","last_became_peered":"2024-09-21T09:54:53.637019+0000","last_unstale":"2024-09-21T09:55:31.244044+0000","last_undegraded":"2024-09-21T09:55:31.244044+0000","last_fullsized":"2024-09-21T09:55:31.244044+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:03:01.499169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":18,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.249544+0000","last_change":"2024-09-21T09:54:53.632794+0000","last_active":"2024-09-21T09:55:31.249544+0000","last_peered":"2024-09-21T09:55:31.249544+0000","last_clean":"2024-09-21T09:55:31.249544+0000","last_became_active":"2024-09-21T09:54:53.632493+0000","last_became_peered":"2024-09-21T09:54:53.632493+0000","last_unstale":"2024-09-21T09:55:31.249544+0000","last_undegraded":"2024-09-21T09:55:31.249544+0000","last_fullsized":"2024-09-21T09:55:31.249544+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:15:30.340221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.252131+0000","last_change":"2024-09-21T09:54:53.628259+0000","last_active":"2024-09-21T09:55:31.252131+0000","last_peered":"2024-09-21T09:55:31.252131+0000","last_clean":"2024-09-21T09:55:31.252131+0000","last_became_active":"2024-09-21T09:54:53.627998+0000","last_became_peered":"2024-09-21T09:54:53.627998+0000","last_unstale":"2024-09-21T09:55:31.252131+0000","last_undegraded":"2024-09-21T09:55:31.252131+0000","last_fullsized":"2024-09-21T09:55:31.252131+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:47:26.663885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.248575+0000","last_change":"2024-09-21T09:54:54.677644+0000","last_active":"2024-09-21T09:55:31.248575+0000","last_peered":"2024-09-21T09:55:31.248575+0000","last_clean":"2024-09-21T09:55:31.248575+0000","last_became_active":"2024-09-21T09:54:54.677043+0000","last_became_peered":"2024-09-21T09:54:54.677043+0000","last_unstale":"2024-09-21T09:55:31.248575+0000","last_undegraded":"2024-09-21T09:55:31.248575+0000","last_fullsized":"2024-09-21T09:55:31.248575+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:47:09.833413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.242515+0000","last_change":"2024-09-21T09:54:53.628040+0000","last_active":"2024-09-21T09:55:31.242515+0000","last_peered":"2024-09-21T09:55:31.242515+0000","last_clean":"2024-09-21T09:55:31.242515+0000","last_became_active":"2024-09-21T09:54:53.627803+0000","last_became_peered":"2024-09-21T09:54:53.627803+0000","last_unstale":"2024-09-21T09:55:31.242515+0000","last_undegraded":"2024-09-21T09:55:31.242515+0000","last_fullsized":"2024-09-21T09:55:31.242515+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:34:24.332639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T09:55:32.436468+0000","last_change":"2024-09-21T09:55:32.436468+0000","last_active":"2024-09-21T09:55:32.436468+0000","last_peered":"2024-09-21T09:55:32.436468+0000","last_clean":"2024-09-21T09:55:32.436468+0000","last_became_active":"2024-09-21T09:55:32.436208+0000","last_became_peered":"2024-09-21T09:55:32.436208+0000","last_unstale":"2024-09-21T09:55:32.436468+0000","last_undegraded":"2024-09-21T09:55:32.436468+0000","last_fullsized":"2024-09-21T09:55:32.436468+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:54:14.558869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.247391+0000","last_change":"2024-09-21T09:54:53.638131+0000","last_active":"2024-09-21T09:55:31.247391+0000","last_peered":"2024-09-21T09:55:31.247391+0000","last_clean":"2024-09-21T09:55:31.247391+0000","last_became_active":"2024-09-21T09:54:53.637677+0000","last_became_peered":"2024-09-21T09:54:53.637677+0000","last_unstale":"2024-09-21T09:55:31.247391+0000","last_undegraded":"2024-09-21T09:55:31.247391+0000","last_fullsized":"2024-09-21T09:55:31.247391+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:35.298342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.247383+0000","last_change":"2024-09-21T09:54:53.629173+0000","last_active":"2024-09-21T09:55:31.247383+0000","last_peered":"2024-09-21T09:55:31.247383+0000","last_clean":"2024-09-21T09:55:31.247383+0000","last_became_active":"2024-09-21T09:54:53.628897+0000","last_became_peered":"2024-09-21T09:54:53.628897+0000","last_unstale":"2024-09-21T09:55:31.247383+0000","last_undegraded":"2024-09-21T09:55:31.247383+0000","last_fullsized":"2024-09-21T09:55:31.247383+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:23:55.120748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.242789+0000","last_change":"2024-09-21T09:54:54.631770+0000","last_active":"2024-09-21T09:55:31.242789+0000","last_peered":"2024-09-21T09:55:31.242789+0000","last_clean":"2024-09-21T09:55:31.242789+0000","last_became_active":"2024-09-21T09:54:54.631481+0000","last_became_peered":"2024-09-21T09:54:54.631481+0000","last_unstale":"2024-09-21T09:55:31.242789+0000","last_undegraded":"2024-09-21T09:55:31.242789+0000","last_fullsized":"2024-09-21T09:55:31.242789+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:13:55.396337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.243784+0000","last_change":"2024-09-21T09:54:53.630346+0000","last_active":"2024-09-21T09:55:31.243784+0000","last_peered":"2024-09-21T09:55:31.243784+0000","last_clean":"2024-09-21T09:55:31.243784+0000","last_became_active":"2024-09-21T09:54:53.629976+0000","last_became_peered":"2024-09-21T09:54:53.629976+0000","last_unstale":"2024-09-21T09:55:31.243784+0000","last_undegraded":"2024-09-21T09:55:31.243784+0000","last_fullsized":"2024-09-21T09:55:31.243784+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:03:20.424882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T09:55:32.469886+0000","last_change":"2024-09-21T09:55:32.469886+0000","last_active":"2024-09-21T09:55:32.469886+0000","last_peered":"2024-09-21T09:55:32.469886+0000","last_clean":"2024-09-21T09:55:32.469886+0000","last_became_active":"2024-09-21T09:55:32.469481+0000","last_became_peered":"2024-09-21T09:55:32.469481+0000","last_unstale":"2024-09-21T09:55:32.469886+0000","last_undegraded":"2024-09-21T09:55:32.469886+0000","last_fullsized":"2024-09-21T09:55:32.469886+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:59.955781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.246229+0000","last_change":"2024-09-21T09:54:53.624732+0000","last_active":"2024-09-21T09:55:31.246229+0000","last_peered":"2024-09-21T09:55:31.246229+0000","last_clean":"2024-09-21T09:55:31.246229+0000","last_became_active":"2024-09-21T09:54:53.624453+0000","last_became_peered":"2024-09-21T09:54:53.624453+0000","last_unstale":"2024-09-21T09:55:31.246229+0000","last_undegraded":"2024-09-21T09:55:31.246229+0000","last_fullsized":"2024-09-21T09:55:31.246229+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:09:20.809064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.252953+0000","last_change":"2024-09-21T09:54:53.722977+0000","last_active":"2024-09-21T09:55:31.252953+0000","last_peered":"2024-09-21T09:55:31.252953+0000","last_clean":"2024-09-21T09:55:31.252953+0000","last_became_active":"2024-09-21T09:54:53.722691+0000","last_became_peered":"2024-09-21T09:54:53.722691+0000","last_unstale":"2024-09-21T09:55:31.252953+0000","last_undegraded":"2024-09-21T09:55:31.252953+0000","last_fullsized":"2024-09-21T09:55:31.252953+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:49:11.950594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.242600+0000","last_change":"2024-09-21T09:54:53.626810+0000","last_active":"2024-09-21T09:55:31.242600+0000","last_peered":"2024-09-21T09:55:31.242600+0000","last_clean":"2024-09-21T09:55:31.242600+0000","last_became_active":"2024-09-21T09:54:53.626533+0000","last_became_peered":"2024-09-21T09:54:53.626533+0000","last_unstale":"2024-09-21T09:55:31.242600+0000","last_undegraded":"2024-09-21T09:55:31.242600+0000","last_fullsized":"2024-09-21T09:55:31.242600+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:43:30.732225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.246333+0000","last_change":"2024-09-21T09:54:53.722232+0000","last_active":"2024-09-21T09:55:31.246333+0000","last_peered":"2024-09-21T09:55:31.246333+0000","last_clean":"2024-09-21T09:55:31.246333+0000","last_became_active":"2024-09-21T09:54:53.721954+0000","last_became_peered":"2024-09-21T09:54:53.721954+0000","last_unstale":"2024-09-21T09:55:31.246333+0000","last_undegraded":"2024-09-21T09:55:31.246333+0000","last_fullsized":"2024-09-21T09:55:31.246333+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:03:01.512522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.252004+0000","last_change":"2024-09-21T09:54:53.634082+0000","last_active":"2024-09-21T09:55:31.252004+0000","last_peered":"2024-09-21T09:55:31.252004+0000","last_clean":"2024-09-21T09:55:31.252004+0000","last_became_active":"2024-09-21T09:54:53.633812+0000","last_became_peered":"2024-09-21T09:54:53.633812+0000","last_unstale":"2024-09-21T09:55:31.252004+0000","last_undegraded":"2024-09-21T09:55:31.252004+0000","last_fullsized":"2024-09-21T09:55:31.252004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:47:20.955775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.245960+0000","last_change":"2024-09-21T09:54:53.723103+0000","last_active":"2024-09-21T09:55:31.245960+0000","last_peered":"2024-09-21T09:55:31.245960+0000","last_clean":"2024-09-21T09:55:31.245960+0000","last_became_active":"2024-09-21T09:54:53.722859+0000","last_became_peered":"2024-09-21T09:54:53.722859+0000","last_unstale":"2024-09-21T09:55:31.245960+0000","last_undegraded":"2024-09-21T09:55:31.245960+0000","last_fullsized":"2024-09-21T09:55:31.245960+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:03:58.582877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.252175+0000","last_change":"2024-09-21T09:54:53.634508+0000","last_active":"2024-09-21T09:55:31.252175+0000","last_peered":"2024-09-21T09:55:31.252175+0000","last_clean":"2024-09-21T09:55:31.252175+0000","last_became_active":"2024-09-21T09:54:53.634254+0000","last_became_peered":"2024-09-21T09:54:53.634254+0000","last_unstale":"2024-09-21T09:55:31.252175+0000","last_undegraded":"2024-09-21T09:55:31.252175+0000","last_fullsized":"2024-09-21T09:55:31.252175+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:18.122107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.248747+0000","last_change":"2024-09-21T09:54:54.677321+0000","last_active":"2024-09-21T09:55:31.248747+0000","last_peered":"2024-09-21T09:55:31.248747+0000","last_clean":"2024-09-21T09:55:31.248747+0000","last_became_active":"2024-09-21T09:54:54.676952+0000","last_became_peered":"2024-09-21T09:54:54.676952+0000","last_unstale":"2024-09-21T09:55:31.248747+0000","last_undegraded":"2024-09-21T09:55:31.248747+0000","last_fullsized":"2024-09-21T09:55:31.248747+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:21:06.467629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.249546+0000","last_change":"2024-09-21T09:54:53.619622+0000","last_active":"2024-09-21T09:55:31.249546+0000","last_peered":"2024-09-21T09:55:31.249546+0000","last_clean":"2024-09-21T09:55:31.249546+0000","last_became_active":"2024-09-21T09:54:53.619323+0000","last_became_peered":"2024-09-21T09:54:53.619323+0000","last_unstale":"2024-09-21T09:55:31.249546+0000","last_undegraded":"2024-09-21T09:55:31.249546+0000","last_fullsized":"2024-09-21T09:55:31.249546+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:25:19.988045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.245915+0000","last_change":"2024-09-21T09:54:53.638406+0000","last_active":"2024-09-21T09:55:31.245915+0000","last_peered":"2024-09-21T09:55:31.245915+0000","last_clean":"2024-09-21T09:55:31.245915+0000","last_became_active":"2024-09-21T09:54:53.638117+0000","last_became_peered":"2024-09-21T09:54:53.638117+0000","last_unstale":"2024-09-21T09:55:31.245915+0000","last_undegraded":"2024-09-21T09:55:31.245915+0000","last_fullsized":"2024-09-21T09:55:31.245915+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:15:06.633281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":19,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T09:55:31.242435+0000","last_change":"2024-09-21T09:54:53.627562+0000","last_active":"2024-09-21T09:55:31.242435+0000","last_peered":"2024-09-21T09:55:31.242435+0000","last_clean":"2024-09-21T09:55:31.242435+0000","last_became_active":"2024-09-21T09:54:53.627303+0000","last_became_peered":"2024-09-21T09:54:53.627303+0000","last_unstale":"2024-09-21T09:55:31.242435+0000","last_undegraded":"2024-09-21T09:55:31.242435+0000","last_fullsized":"2024-09-21T09:55:31.242435+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:03:15.883024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234021,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28372,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716780,"statfs":{"total":95995035648,"available":95965982720,"internally_reserved":0,"allocated":1380352,"data_stored":925744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52900000000000003,"15min":0.53100000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.70499999999999996,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.48899999999999999,"15min":0.48499999999999999},"min":{"1min":0.41099999999999998,"5min":0.214,"15min":0.214},"max":{"1min":0.65500000000000003,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.39800000000000002}]},{"osd":1,"last update":"Sat Sep 21 09:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.50600000000000001,"15min":0.504},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.36699999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.71199999999999997,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.49299999999999999}]},{"osd":2,"last update":"Sat Sep 21 09:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.5,"15min":0.498},"min":{"1min":0.376,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.69599999999999995,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.52300000000000002}]},{"osd":3,"last update":"Sat Sep 21 09:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.52000000000000002,"15min":0.51600000000000001},"min":{"1min":0.372,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.66500000000000004,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.52400000000000002,"15min":0.52200000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.72899999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.47599999999999998}]},{"osd":4,"last update":"Sat Sep 21 09:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.42099999999999999},"min":{"1min":0.29499999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.499,"5min":0.45000000000000001,"15min":0.442},"min":{"1min":0.309,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.40999999999999998}]},{"osd":5,"last update":"Sat Sep 21 09:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.46400000000000002,"15min":0.46200000000000002},"min":{"1min":0.32400000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.58599999999999997,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.378},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.49199999999999999,"15min":0.48899999999999999},"min":{"1min":0.33600000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.67500000000000004,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.443}]},{"osd":6,"last update":"Sat Sep 21 09:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.48699999999999999,"15min":0.48299999999999998},"min":{"1min":0.36499999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.65200000000000002,"5min":0.752,"15min":0.752},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.501,"15min":0.495},"min":{"1min":0.35999999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66600000000000004,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.46100000000000002}]}]},{"osd":6,"up_from":46,"seq":197568495658,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28360,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716792,"statfs":{"total":95995035648,"available":95965995008,"internally_reserved":0,"allocated":1372160,"data_stored":925617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.54300000000000004,"15min":0.55100000000000005},"min":{"1min":0.35399999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.72799999999999998,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.377,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.70099999999999996}]},{"osd":1,"last update":"Sat Sep 21 09:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.53300000000000003,"15min":0.54100000000000004},"min":{"1min":0.35099999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.71499999999999997,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.39700000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.74099999999999999,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.752}]},{"osd":2,"last update":"Sat Sep 21 09:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.53200000000000003,"15min":0.53600000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.65700000000000003,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54600000000000004,"15min":0.54700000000000004},"min":{"1min":0.38900000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.63500000000000001}]},{"osd":3,"last update":"Sat Sep 21 09:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54800000000000004,"15min":0.55200000000000005},"min":{"1min":0.38300000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.39500000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.64500000000000002}]},{"osd":4,"last update":"Sat Sep 21 09:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.434,"15min":0.41799999999999998},"min":{"1min":0.28599999999999998,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.46000000000000002,"15min":0.44800000000000001},"min":{"1min":0.32000000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.69599999999999995,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.50800000000000001}]},{"osd":5,"last update":"Sat Sep 21 09:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46800000000000003,"15min":0.45300000000000001},"min":{"1min":0.30499999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.628,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.47999999999999998,"15min":0.47199999999999998},"min":{"1min":0.32300000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.66300000000000003}]},{"osd":7,"last update":"Sat Sep 21 09:55:05 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48499999999999999,"15min":0.48399999999999999},"min":{"1min":0.30399999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.74299999999999999,"5min":0.745,"15min":0.745},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.53700000000000003,"15min":0.53300000000000003},"min":{"1min":0.34200000000000003,"5min":0.318,"15min":0.318},"max":{"1min":0.70499999999999996,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.68100000000000005}]}]},{"osd":5,"up_from":38,"seq":163208757295,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27772,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717380,"statfs":{"total":95995035648,"available":95966597120,"internally_reserved":0,"allocated":765952,"data_stored":333936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.28699999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.53900000000000003,"15min":0.54700000000000004},"min":{"1min":0.34699999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69099999999999995,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.47699999999999998}]},{"osd":1,"last update":"Sat Sep 21 09:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56000000000000005,"15min":0.56999999999999995},"min":{"1min":0.38300000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70699999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53900000000000003,"15min":0.54100000000000004},"min":{"1min":0.41499999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.69199999999999995,"5min":0.755,"15min":0.755},"last":0.51600000000000001}]},{"osd":2,"last update":"Sat Sep 21 09:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.39500000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.71999999999999997,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.40699999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.80300000000000005,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.48799999999999999}]},{"osd":3,"last update":"Sat Sep 21 09:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.54300000000000004,"15min":0.54800000000000004},"min":{"1min":0.42699999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.70199999999999996,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56799999999999995,"15min":0.57299999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72399999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.59899999999999998}]},{"osd":4,"last update":"Sat Sep 21 09:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.41799999999999998,"15min":0.40100000000000002},"min":{"1min":0.309,"5min":0.217,"15min":0.217},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.44,"5min":0.42799999999999999,"15min":0.40899999999999997},"min":{"1min":0.29499999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.63,"5min":1.24,"15min":1.24},"last":0.52800000000000002}]},{"osd":6,"last update":"Sat Sep 21 09:55:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.44800000000000001,"15min":0.438},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.66700000000000004,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.48399999999999999,"15min":0.47599999999999998},"min":{"1min":0.35199999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.72099999999999997,"5min":1.21,"15min":1.21},"last":0.45700000000000002}]},{"osd":7,"last update":"Sat Sep 21 09:55:05 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.55600000000000005,"15min":0.56399999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.67800000000000005,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.501},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.437}]}]},{"osd":4,"up_from":30,"seq":128849018931,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27788,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717364,"statfs":{"total":95995035648,"available":95966580736,"internally_reserved":0,"allocated":782336,"data_stored":335306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.504,"15min":0.497},"min":{"1min":0.33400000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55600000000000005,"15min":0.54500000000000004},"min":{"1min":0.41899999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.65100000000000002}]},{"osd":1,"last update":"Sat Sep 21 09:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51100000000000001,"15min":0.51500000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.55900000000000005,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.52900000000000003,"15min":0.52100000000000002},"min":{"1min":0.39400000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.66500000000000004}]},{"osd":2,"last update":"Sat Sep 21 09:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.53700000000000003,"15min":0.53300000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52000000000000002,"15min":0.51100000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.68200000000000005,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.53900000000000003}]},{"osd":3,"last update":"Sat Sep 21 09:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53300000000000003,"15min":0.52500000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.72299999999999998,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.52600000000000002,"15min":0.51400000000000001},"min":{"1min":0.38300000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.59399999999999997}]},{"osd":5,"last update":"Sat Sep 21 09:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.39200000000000002,"15min":0.36899999999999999},"min":{"1min":0.32300000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.63,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.40000000000000002,"15min":0.375},"min":{"1min":0.29799999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.41099999999999998}]},{"osd":6,"last update":"Sat Sep 21 09:55:41 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45300000000000001,"15min":0.436},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.42699999999999999,"15min":0.41299999999999998},"min":{"1min":0.30399999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.63600000000000001}]},{"osd":7,"last update":"Sat Sep 21 09:55:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.53600000000000003,"15min":0.53800000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71399999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.46500000000000002,"15min":0.46100000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.499}]}]},{"osd":3,"up_from":25,"seq":107374182456,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27764,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":761856,"data_stored":333686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:55:34 2024","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.34399999999999997,"15min":0.36399999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.45700000000000002,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.38800000000000001,"15min":0.40100000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.51700000000000002,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.52800000000000002}]},{"osd":1,"last update":"Sat Sep 21 09:55:34 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.39600000000000002,"15min":0.41199999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.497,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.36099999999999999,"15min":0.374},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.44,"5min":0.52400000000000002,"15min":0.52400000000000002},"last":0.437}]},{"osd":2,"last update":"Sat Sep 21 09:55:34 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.39300000000000002,"15min":0.40799999999999997},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.498,"5min":0.627,"15min":0.627},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.36799999999999999,"15min":0.375},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.45200000000000001,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.44900000000000001}]},{"osd":4,"last update":"Sat Sep 21 09:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.54500000000000004,"15min":0.52900000000000003},"min":{"1min":0.39900000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52600000000000002,"15min":0.50900000000000001},"min":{"1min":0.39700000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.65700000000000003}]},{"osd":5,"last update":"Sat Sep 21 09:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.56599999999999995,"15min":0.55800000000000005},"min":{"1min":0.39200000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.432,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.58699999999999997}]},{"osd":6,"last update":"Sat Sep 21 09:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.56200000000000006,"15min":0.54800000000000004},"min":{"1min":0.441,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.84499999999999997,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.58899999999999997,"15min":0.57999999999999996},"min":{"1min":0.48799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.95099999999999996,"5min":1.236,"15min":1.236},"last":0.70299999999999996}]},{"osd":7,"last update":"Sat Sep 21 09:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.58499999999999996,"15min":0.59099999999999997},"min":{"1min":0.42499999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.56499999999999995,"15min":0.55700000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.67400000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31868,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713284,"statfs":{"total":95995035648,"available":95962402816,"internally_reserved":0,"allocated":765952,"data_stored":334212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.23899999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.41399999999999998,"5min":1.125,"15min":1.125},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.38300000000000001,"15min":0.371},"min":{"1min":0.24199999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.48899999999999999,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.42099999999999999}]},{"osd":1,"last update":"Sat Sep 21 09:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.38500000000000001,"15min":0.38800000000000001},"min":{"1min":0.26200000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.44600000000000001,"5min":1.161,"15min":1.161},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.40500000000000003,"15min":0.40100000000000002},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.51500000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.36899999999999999}]},{"osd":3,"last update":"Sat Sep 21 09:55:31 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.41699999999999998,"15min":0.42799999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.51400000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.40200000000000002,"15min":0.41999999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.48099999999999998,"5min":1.052,"15min":1.052},"last":0.33600000000000002}]},{"osd":4,"last update":"Sat Sep 21 09:54:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56299999999999994,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57699999999999996,"15min":0.56699999999999995},"min":{"1min":0.432,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.83499999999999996,"5min":1.012,"15min":1.012},"last":0.70299999999999996}]},{"osd":5,"last update":"Sat Sep 21 09:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.623,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77100000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.48099999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85199999999999998,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.77400000000000002}]},{"osd":6,"last update":"Sat Sep 21 09:55:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.80400000000000005,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.621,"15min":0.63900000000000001},"min":{"1min":0.42999999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79400000000000004,"5min":1.202,"15min":1.202},"last":0.73599999999999999}]},{"osd":7,"last update":"Sat Sep 21 09:55:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.59299999999999997,"15min":0.59499999999999997},"min":{"1min":0.46400000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.83599999999999997,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.64200000000000002,"15min":0.65200000000000002},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.77200000000000002,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.753}]}]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27764,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":761856,"data_stored":333615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.35399999999999998,"15min":0.32300000000000001},"min":{"1min":0.23499999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.373,"15min":0.33100000000000002},"min":{"1min":0.22500000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.798,"5min":1.798,"15min":1.798},"last":0.26700000000000002}]},{"osd":2,"last update":"Sat Sep 21 09:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34699999999999998,"15min":0.34000000000000002},"min":{"1min":0.247,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.38600000000000001,"15min":0.38200000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.34100000000000003}]},{"osd":3,"last update":"Sat Sep 21 09:55:42 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.39100000000000001,"15min":0.40699999999999997},"min":{"1min":0.28799999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.39600000000000002,"15min":0.40600000000000003},"min":{"1min":0.27100000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.28499999999999998}]},{"osd":4,"last update":"Sat Sep 21 09:54:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54300000000000004,"15min":0.54900000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.71499999999999997,"5min":1.022,"15min":1.022},"last":0.628}]},{"osd":5,"last update":"Sat Sep 21 09:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.59299999999999997,"15min":0.60399999999999998},"min":{"1min":0.42599999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.59899999999999998,"15min":0.60499999999999998},"min":{"1min":0.42699999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.65100000000000002}]},{"osd":6,"last update":"Sat Sep 21 09:55:42 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.57099999999999995,"15min":0.57699999999999996},"min":{"1min":0.43099999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.56000000000000005,"15min":0.55200000000000005},"min":{"1min":0.439,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.54900000000000004}]},{"osd":7,"last update":"Sat Sep 21 09:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.54000000000000004,"15min":0.53500000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.51000000000000001,"15min":0.502},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.66100000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28352,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716800,"statfs":{"total":95995035648,"available":95966003200,"internally_reserved":0,"allocated":1359872,"data_stored":924568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 09:54:49 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.29399999999999998},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.45500000000000002,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.35499999999999998,"15min":0.32100000000000001},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.49399999999999999}]},{"osd":2,"last update":"Sat Sep 21 09:55:11 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.38200000000000001,"15min":0.35799999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.56200000000000006,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.36199999999999999,"15min":0.34399999999999997},"min":{"1min":0.25700000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.51000000000000001,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.51000000000000001}]},{"osd":3,"last update":"Sat Sep 21 09:55:32 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.42299999999999999,"15min":0.433},"min":{"1min":0.30099999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53100000000000003,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.40999999999999998,"15min":0.41699999999999998},"min":{"1min":0.311,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.53300000000000003,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.54800000000000004}]},{"osd":4,"last update":"Sat Sep 21 09:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55300000000000005,"15min":0.55700000000000005},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.79700000000000004,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.90000000000000002,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.81299999999999994}]},{"osd":5,"last update":"Sat Sep 21 09:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.81100000000000005,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59499999999999997,"15min":0.59999999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.83299999999999996,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.82599999999999996}]},{"osd":6,"last update":"Sat Sep 21 09:55:48 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58499999999999996,"15min":0.58199999999999996},"min":{"1min":0.41299999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.57399999999999995,"15min":0.57499999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.748,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.68300000000000005}]},{"osd":7,"last update":"Sat Sep 21 09:55:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.57399999999999995,"15min":0.57799999999999996},"min":{"1min":0.38600000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73099999999999998,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60099999999999998,"15min":0.59999999999999998},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84499999999999997,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.86299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T09:55:54.355 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T09:55:54.355 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T09:55:54.355 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-21T09:55:54.355 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph health --format=json 2024-09-21T09:55:54.557 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:54.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:54 smithi070 ceph-mon[41533]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:54.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:54 smithi070 ceph-mon[29589]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:54.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:55:54 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:55:54] "GET /metrics HTTP/1.1" 200 33791 "" "Prometheus/2.43.0" 2024-09-21T09:55:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:54 smithi160 ceph-mon[33203]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:55.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:55 smithi070 ceph-mon[29589]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:55:55.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:55 smithi070 ceph-mon[41533]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:55:55.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:55 smithi160 ceph-mon[33203]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:55:56.368 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T09:55:56.368 INFO:teuthology.orchestra.run.smithi070.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T09:55:56.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:56 smithi070 ceph-mon[41533]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:56.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:56 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/890192962' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T09:55:56.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:56 smithi070 ceph-mon[29589]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:56.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:56 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/890192962' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T09:55:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:56 smithi160 ceph-mon[33203]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:56 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/890192962' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T09:55:57.019 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-21T09:55:57.019 INFO:teuthology.run_tasks:Running task print... 2024-09-21T09:55:57.028 INFO:teuthology.task.print:**** upgrading first half of cluster, with stress **** 2024-09-21T09:55:57.028 INFO:teuthology.run_tasks:Running task parallel... 2024-09-21T09:55:57.061 INFO:teuthology.task.parallel:starting parallel... 2024-09-21T09:55:57.062 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-21T09:55:57.062 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-09-21T09:55:57.067 INFO:tasks.workunit:Pulling workunits from ref reef 2024-09-21T09:55:57.067 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-21T09:55:57.067 DEBUG:teuthology.orchestra.run.smithi070:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-21T09:55:57.067 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-21T09:55:57.068 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-09-21T09:55:57.068 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi070.front.sepia.ceph.com 2024-09-21T09:55:57.068 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph config set mgr mgr/cephadm/daemon_cache_timeout 60' 2024-09-21T09:55:57.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T09:55:57.085 INFO:teuthology.orchestra.run.smithi070.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-21T09:55:57.085 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-21T09:55:57.139 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-21T09:55:57.139 DEBUG:teuthology.orchestra.run.smithi070:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-21T09:55:57.194 INFO:tasks.workunit:timeout=3h 2024-09-21T09:55:57.195 INFO:tasks.workunit:cleanup=True 2024-09-21T09:55:57.195 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch reef https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2024-09-21T09:55:57.250 INFO:tasks.workunit.client.0.smithi070.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-21T09:55:57.270 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:58.705 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:55:58 smithi070 ceph-mon[29589]: pgmap v90: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:58.705 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:55:58 smithi070 ceph-mon[41533]: pgmap v90: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:55:58 smithi160 ceph-mon[33203]: pgmap v90: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:55:59.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:55:59 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:55:59] ENGINE Bus STOPPING 2024-09-21T09:55:59.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:55:59 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:55:59] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:55:59.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:55:59 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:55:59] ENGINE Bus STOPPED 2024-09-21T09:55:59.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:55:59 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:55:59] ENGINE Bus STARTING 2024-09-21T09:55:59.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:55:59 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:55:59] ENGINE Serving on http://:::9283 2024-09-21T09:55:59.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:55:59 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:55:59] ENGINE Bus STARTED 2024-09-21T09:55:59.959 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph config set global log_to_journald false --force' 2024-09-21T09:56:00.200 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:00.462 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:00 smithi070 ceph-mon[29589]: pgmap v91: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:00.462 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:00 smithi070 ceph-mon[29589]: from='client.? ' entity='client.admin' 2024-09-21T09:56:00.462 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:00 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:56:00.462 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:00 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:56:00.462 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:00 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:56:00.462 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:00 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:00.462 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:00 smithi070 ceph-mon[41533]: pgmap v91: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:00.462 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:00 smithi070 ceph-mon[41533]: from='client.? ' entity='client.admin' 2024-09-21T09:56:00.462 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:00 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:56:00.463 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:00 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:56:00.463 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:00 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:56:00.463 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:00 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:00 smithi160 ceph-mon[33203]: pgmap v91: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:00 smithi160 ceph-mon[33203]: from='client.? ' entity='client.admin' 2024-09-21T09:56:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:00 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:56:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:00 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:56:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:00 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:56:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:00 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:01.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:01 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:01.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:01 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:01.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:01 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:02.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:02 smithi070 ceph-mon[29589]: pgmap v92: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:02.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:02 smithi070 ceph-mon[41533]: pgmap v92: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:02 smithi160 ceph-mon[33203]: pgmap v92: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:03.282 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'echo wait for mgr daemons to upgrade' 2024-09-21T09:56:03.457 DEBUG:teuthology.orchestra.run.smithi070:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-21T09:56:03.482 INFO:tasks.workunit.client.0.smithi070.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-21T09:56:03.484 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:03.484 INFO:tasks.workunit.client.0.smithi070.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T09:56:03.484 INFO:tasks.workunit.client.0.smithi070.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-21T09:56:04.484 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:04 smithi070 ceph-mon[41533]: pgmap v93: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:04.485 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:04 smithi070 ceph-mon[29589]: pgmap v93: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:04 smithi160 ceph-mon[33203]: pgmap v93: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:04.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:56:04 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:56:04] "GET /metrics HTTP/1.1" 200 33784 "" "Prometheus/2.43.0" 2024-09-21T09:56:05.607 INFO:tasks.workunit.client.0.smithi070.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-21T09:56:05.763 INFO:tasks.workunit.client.0.smithi070.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-21T09:56:05.813 INFO:tasks.workunit.client.0.smithi070.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T09:56:05.815 INFO:tasks.workunit.client.0.smithi070.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T09:56:05.815 INFO:tasks.workunit.client.0.smithi070.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-21T09:56:05.857 INFO:tasks.workunit.client.0.smithi070.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T09:56:05.860 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T09:56:05.861 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-21T09:56:05.916 INFO:tasks.workunit:Running workunits matching rbd/test_librbd.sh on client.0... 2024-09-21T09:56:05.917 INFO:tasks.workunit:Running workunit rbd/test_librbd.sh... 2024-09-21T09:56:05.917 DEBUG:teuthology.orchestra.run.smithi070:workunit test rbd/test_librbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh 2024-09-21T09:56:05.931 INFO:teuthology.orchestra.run.smithi070.stdout:wait for mgr daemons to upgrade 2024-09-21T09:56:06.010 INFO:tasks.workunit.client.0.smithi070.stderr:warning: failed to enable lockdep 2024-09-21T09:56:06.011 INFO:tasks.workunit.client.0.smithi070.stdout:seed 86016 2024-09-21T09:56:06.011 INFO:tasks.workunit.client.0.smithi070.stdout:[==========] Running 490 tests from 24 test suites. 2024-09-21T09:56:06.011 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] Global test environment set-up. 2024-09-21T09:56:06.011 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 1 test from TestJournalStress 2024-09-21T09:56:06.585 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:06 smithi070 ceph-mon[29589]: pgmap v94: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:06.585 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:06 smithi070 ceph-mon[41533]: pgmap v94: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:06 smithi160 ceph-mon[33203]: pgmap v94: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:06.606 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-09-21T09:56:06.800 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:07.336 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestJournalStress.DiscardWithPruneWriteOverlap 2024-09-21T09:56:07.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:07 smithi160 ceph-mon[33203]: osdmap e74: 8 total, 8 up, 8 in 2024-09-21T09:56:07.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:07 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/1941844133' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:07.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:07 smithi070 ceph-mon[29589]: osdmap e74: 8 total, 8 up, 8 in 2024-09-21T09:56:07.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:07 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/1941844133' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:07.608 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:07 smithi070 ceph-mon[41533]: osdmap e74: 8 total, 8 up, 8 in 2024-09-21T09:56:07.608 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:07 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/1941844133' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:08.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:08 smithi160 ceph-mon[33203]: pgmap v96: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:08.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:08 smithi160 ceph-mon[33203]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:56:08.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:08 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/1941844133' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:56:08.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:08 smithi160 ceph-mon[33203]: osdmap e75: 8 total, 8 up, 8 in 2024-09-21T09:56:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:08 smithi070 ceph-mon[29589]: pgmap v96: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:08 smithi070 ceph-mon[29589]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:56:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:08 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/1941844133' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:56:08.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:08 smithi070 ceph-mon[29589]: osdmap e75: 8 total, 8 up, 8 in 2024-09-21T09:56:08.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:08 smithi070 ceph-mon[41533]: pgmap v96: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:08.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:08 smithi070 ceph-mon[41533]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:56:08.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:08 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/1941844133' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:56:08.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:08 smithi070 ceph-mon[41533]: osdmap e75: 8 total, 8 up, 8 in 2024-09-21T09:56:09.358 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/journal/test_Stress.cc:37: Skipped 2024-09-21T09:56:09.359 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T09:56:09.359 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalStress.DiscardWithPruneWriteOverlap (2022 ms) 2024-09-21T09:56:09.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:09 smithi160 ceph-mon[33203]: osdmap e76: 8 total, 8 up, 8 in 2024-09-21T09:56:09.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:09 smithi070 ceph-mon[29589]: osdmap e76: 8 total, 8 up, 8 in 2024-09-21T09:56:09.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:09 smithi070 ceph-mon[41533]: osdmap e76: 8 total, 8 up, 8 in 2024-09-21T09:56:10.350 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 1 test from TestJournalStress (2022 ms total) 2024-09-21T09:56:10.350 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T09:56:10.350 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 16 tests from TestJournalReplay 2024-09-21T09:56:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:10 smithi160 ceph-mon[33203]: from='client.14832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:10 smithi160 ceph-mon[33203]: pgmap v99: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:10.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:10 smithi160 ceph-mon[33203]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:56:10.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:10 smithi160 ceph-mon[33203]: Cluster is now healthy 2024-09-21T09:56:10.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:10 smithi160 ceph-mon[33203]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T09:56:10.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:10 smithi070 ceph-mon[29589]: from='client.14832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:10.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:10 smithi070 ceph-mon[29589]: pgmap v99: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:10.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:10 smithi070 ceph-mon[29589]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:56:10.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:10 smithi070 ceph-mon[29589]: Cluster is now healthy 2024-09-21T09:56:10.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:10 smithi070 ceph-mon[29589]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T09:56:10.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:10 smithi070 ceph-mon[41533]: from='client.14832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:10.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:10 smithi070 ceph-mon[41533]: pgmap v99: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:10.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:10 smithi070 ceph-mon[41533]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:56:10.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:10 smithi070 ceph-mon[41533]: Cluster is now healthy 2024-09-21T09:56:10.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:10 smithi070 ceph-mon[41533]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T09:56:11.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:11 smithi070 ceph-mon[29589]: osdmap e78: 8 total, 8 up, 8 in 2024-09-21T09:56:11.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:11 smithi070 ceph-mon[41533]: osdmap e78: 8 total, 8 up, 8 in 2024-09-21T09:56:11.721 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:11 smithi160 ceph-mon[33203]: osdmap e78: 8 total, 8 up, 8 in 2024-09-21T09:56:12.353 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestJournalReplay.AioDiscardEvent 2024-09-21T09:56:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:12 smithi070 ceph-mon[29589]: pgmap v102: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:12 smithi070 ceph-mon[29589]: osdmap e79: 8 total, 8 up, 8 in 2024-09-21T09:56:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:12 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/2708080876' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:12.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:12 smithi070 ceph-mon[41533]: pgmap v102: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:12 smithi070 ceph-mon[41533]: osdmap e79: 8 total, 8 up, 8 in 2024-09-21T09:56:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:12 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/2708080876' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:12 smithi160 ceph-mon[33203]: pgmap v102: 73 pgs: 73 active+clean; 579 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:12 smithi160 ceph-mon[33203]: osdmap e79: 8 total, 8 up, 8 in 2024-09-21T09:56:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:12 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/2708080876' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:13.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:13 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/2708080876' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:56:13.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:13 smithi070 ceph-mon[29589]: osdmap e80: 8 total, 8 up, 8 in 2024-09-21T09:56:13.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:13 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/2708080876' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:56:13.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:13 smithi070 ceph-mon[41533]: osdmap e80: 8 total, 8 up, 8 in 2024-09-21T09:56:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:13 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/2708080876' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:56:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:13 smithi160 ceph-mon[33203]: osdmap e80: 8 total, 8 up, 8 in 2024-09-21T09:56:14.387 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/journal/test_Replay.cc:109: Skipped 2024-09-21T09:56:14.387 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T09:56:14.387 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.AioDiscardEvent (2033 ms) 2024-09-21T09:56:14.387 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestJournalReplay.AioWriteEvent 2024-09-21T09:56:14.397 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/journal/test_Replay.cc:200: Skipped 2024-09-21T09:56:14.397 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T09:56:14.397 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.AioWriteEvent (11 ms) 2024-09-21T09:56:14.397 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestJournalReplay.AioFlushEvent 2024-09-21T09:56:14.405 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/journal/test_Replay.cc:261: Skipped 2024-09-21T09:56:14.405 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T09:56:14.405 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.AioFlushEvent (8 ms) 2024-09-21T09:56:14.405 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestJournalReplay.SnapCreate 2024-09-21T09:56:14.414 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/journal/test_Replay.cc:307: Skipped 2024-09-21T09:56:14.414 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T09:56:14.414 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.SnapCreate (9 ms) 2024-09-21T09:56:14.414 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestJournalReplay.SnapProtect 2024-09-21T09:56:14.422 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/journal/test_Replay.cc:348: Skipped 2024-09-21T09:56:14.422 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T09:56:14.422 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.SnapProtect (8 ms) 2024-09-21T09:56:14.422 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestJournalReplay.SnapUnprotect 2024-09-21T09:56:14.430 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/journal/test_Replay.cc:394: Skipped 2024-09-21T09:56:14.430 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T09:56:14.430 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.SnapUnprotect (8 ms) 2024-09-21T09:56:14.430 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestJournalReplay.SnapRename 2024-09-21T09:56:14.438 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/journal/test_Replay.cc:450: Skipped 2024-09-21T09:56:14.438 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T09:56:14.438 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.SnapRename (8 ms) 2024-09-21T09:56:14.438 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestJournalReplay.SnapRollback 2024-09-21T09:56:14.445 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/journal/test_Replay.cc:500: Skipped 2024-09-21T09:56:14.445 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T09:56:14.445 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.SnapRollback (7 ms) 2024-09-21T09:56:14.445 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestJournalReplay.SnapRemove 2024-09-21T09:56:14.453 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/journal/test_Replay.cc:541: Skipped 2024-09-21T09:56:14.453 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T09:56:14.453 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.SnapRemove (8 ms) 2024-09-21T09:56:14.453 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestJournalReplay.Rename 2024-09-21T09:56:14.460 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/journal/test_Replay.cc:590: Skipped 2024-09-21T09:56:14.460 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T09:56:14.460 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.Rename (7 ms) 2024-09-21T09:56:14.461 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestJournalReplay.Resize 2024-09-21T09:56:14.468 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/journal/test_Replay.cc:624: Skipped 2024-09-21T09:56:14.468 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T09:56:14.468 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.Resize (8 ms) 2024-09-21T09:56:14.469 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestJournalReplay.Flatten 2024-09-21T09:56:14.476 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/journal/test_Replay.cc:657: Skipped 2024-09-21T09:56:14.476 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T09:56:14.476 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.Flatten (8 ms) 2024-09-21T09:56:14.476 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestJournalReplay.UpdateFeatures 2024-09-21T09:56:14.483 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/journal/test_Replay.cc:704: Skipped 2024-09-21T09:56:14.483 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T09:56:14.484 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.UpdateFeatures (7 ms) 2024-09-21T09:56:14.484 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestJournalReplay.MetadataSet 2024-09-21T09:56:14.491 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/journal/test_Replay.cc:741: Skipped 2024-09-21T09:56:14.491 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T09:56:14.491 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.MetadataSet (8 ms) 2024-09-21T09:56:14.491 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestJournalReplay.MetadataRemove 2024-09-21T09:56:14.498 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/journal/test_Replay.cc:781: Skipped 2024-09-21T09:56:14.498 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T09:56:14.498 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.MetadataRemove (7 ms) 2024-09-21T09:56:14.499 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestJournalReplay.ObjectPosition 2024-09-21T09:56:14.506 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/journal/test_Replay.cc:824: Skipped 2024-09-21T09:56:14.506 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T09:56:14.506 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.ObjectPosition (8 ms) 2024-09-21T09:56:14.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:56:14 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:56:14] "GET /metrics HTTP/1.1" 200 33784 "" "Prometheus/2.43.0" 2024-09-21T09:56:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:14 smithi070 ceph-mon[29589]: pgmap v105: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:14.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:14 smithi070 ceph-mon[29589]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T09:56:14.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:14 smithi070 ceph-mon[41533]: pgmap v105: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:14.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:14 smithi070 ceph-mon[41533]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T09:56:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:14 smithi160 ceph-mon[33203]: pgmap v105: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:14 smithi160 ceph-mon[33203]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T09:56:15.386 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 16 tests from TestJournalReplay (2153 ms total) 2024-09-21T09:56:15.386 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T09:56:15.386 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 4 tests from TestJournalEntries 2024-09-21T09:56:15.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:15 smithi070 ceph-mon[29589]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T09:56:15.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:15 smithi070 ceph-mon[41533]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T09:56:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:15 smithi160 ceph-mon[33203]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T09:56:16.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:16 smithi160 ceph-mon[33203]: pgmap v108: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:16.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:16 smithi160 ceph-mon[33203]: osdmap e83: 8 total, 8 up, 8 in 2024-09-21T09:56:16.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:16 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:16.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:16 smithi070 ceph-mon[29589]: pgmap v108: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:16.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:16 smithi070 ceph-mon[29589]: osdmap e83: 8 total, 8 up, 8 in 2024-09-21T09:56:16.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:16 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:16.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:16 smithi070 ceph-mon[41533]: pgmap v108: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:16.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:16 smithi070 ceph-mon[41533]: osdmap e83: 8 total, 8 up, 8 in 2024-09-21T09:56:16.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:16 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:17.394 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestJournalEntries.AioWrite 2024-09-21T09:56:17.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:17 smithi160 ceph-mon[33203]: osdmap e84: 8 total, 8 up, 8 in 2024-09-21T09:56:17.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:17 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/521009019' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:17.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:17 smithi070 ceph-mon[29589]: osdmap e84: 8 total, 8 up, 8 in 2024-09-21T09:56:17.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:17 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/521009019' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:17.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:17 smithi070 ceph-mon[41533]: osdmap e84: 8 total, 8 up, 8 in 2024-09-21T09:56:17.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:17 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/521009019' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:18 smithi160 ceph-mon[33203]: pgmap v111: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:18 smithi160 ceph-mon[33203]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:56:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:18 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/521009019' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:56:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:18 smithi160 ceph-mon[33203]: osdmap e85: 8 total, 8 up, 8 in 2024-09-21T09:56:18.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:18 smithi070 ceph-mon[29589]: pgmap v111: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:18.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:18 smithi070 ceph-mon[29589]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:56:18.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:18 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/521009019' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:56:18.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:18 smithi070 ceph-mon[29589]: osdmap e85: 8 total, 8 up, 8 in 2024-09-21T09:56:18.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:18 smithi070 ceph-mon[41533]: pgmap v111: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:18.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:18 smithi070 ceph-mon[41533]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:56:18.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:18 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/521009019' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:56:18.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:18 smithi070 ceph-mon[41533]: osdmap e85: 8 total, 8 up, 8 in 2024-09-21T09:56:19.417 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/journal/test_Entries.cc:118: Skipped 2024-09-21T09:56:19.417 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T09:56:19.417 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalEntries.AioWrite (2023 ms) 2024-09-21T09:56:19.417 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestJournalEntries.AioDiscard 2024-09-21T09:56:19.426 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/journal/test_Entries.cc:163: Skipped 2024-09-21T09:56:19.426 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T09:56:19.426 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalEntries.AioDiscard (9 ms) 2024-09-21T09:56:19.426 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestJournalEntries.AioDiscardWithPrune 2024-09-21T09:56:19.434 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/journal/test_Entries.cc:200: Skipped 2024-09-21T09:56:19.434 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T09:56:19.434 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalEntries.AioDiscardWithPrune (8 ms) 2024-09-21T09:56:19.434 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestJournalEntries.AioFlush 2024-09-21T09:56:19.442 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/journal/test_Entries.cc:263: Skipped 2024-09-21T09:56:19.442 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T09:56:19.442 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalEntries.AioFlush (8 ms) 2024-09-21T09:56:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:19 smithi160 ceph-mon[33203]: osdmap e86: 8 total, 8 up, 8 in 2024-09-21T09:56:19.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:19 smithi070 ceph-mon[41533]: osdmap e86: 8 total, 8 up, 8 in 2024-09-21T09:56:19.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:19 smithi070 ceph-mon[29589]: osdmap e86: 8 total, 8 up, 8 in 2024-09-21T09:56:20.410 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 4 tests from TestJournalEntries (2048 ms total) 2024-09-21T09:56:20.410 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T09:56:20.410 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 2 tests from TestTrash 2024-09-21T09:56:20.777 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:20 smithi070 ceph-mon[29589]: pgmap v114: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:20.778 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:20 smithi070 ceph-mon[29589]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:56:20.778 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:20 smithi070 ceph-mon[29589]: Cluster is now healthy 2024-09-21T09:56:20.778 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:20 smithi070 ceph-mon[29589]: osdmap e87: 8 total, 8 up, 8 in 2024-09-21T09:56:20.778 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:20 smithi070 ceph-mon[41533]: pgmap v114: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:20.778 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:20 smithi070 ceph-mon[41533]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:56:20.778 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:20 smithi070 ceph-mon[41533]: Cluster is now healthy 2024-09-21T09:56:20.778 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:20 smithi070 ceph-mon[41533]: osdmap e87: 8 total, 8 up, 8 in 2024-09-21T09:56:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:20 smithi160 ceph-mon[33203]: pgmap v114: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:20 smithi160 ceph-mon[33203]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:56:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:20 smithi160 ceph-mon[33203]: Cluster is now healthy 2024-09-21T09:56:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:20 smithi160 ceph-mon[33203]: osdmap e87: 8 total, 8 up, 8 in 2024-09-21T09:56:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:21 smithi160 ceph-mon[33203]: osdmap e88: 8 total, 8 up, 8 in 2024-09-21T09:56:21.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:21 smithi070 ceph-mon[29589]: osdmap e88: 8 total, 8 up, 8 in 2024-09-21T09:56:21.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:21 smithi070 ceph-mon[41533]: osdmap e88: 8 total, 8 up, 8 in 2024-09-21T09:56:22.416 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:22 smithi070 ceph-mon[29589]: pgmap v117: 73 pgs: 73 active+clean; 579 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:22.416 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:22 smithi070 ceph-mon[29589]: osdmap e89: 8 total, 8 up, 8 in 2024-09-21T09:56:22.416 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:22 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/726224530' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:22.419 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestTrash.UserRemovingSource 2024-09-21T09:56:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:22 smithi160 ceph-mon[33203]: pgmap v117: 73 pgs: 73 active+clean; 579 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:22 smithi160 ceph-mon[33203]: osdmap e89: 8 total, 8 up, 8 in 2024-09-21T09:56:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:22 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/726224530' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:22.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:22 smithi070 ceph-mon[41533]: pgmap v117: 73 pgs: 73 active+clean; 579 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:22.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:22 smithi070 ceph-mon[41533]: osdmap e89: 8 total, 8 up, 8 in 2024-09-21T09:56:22.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:22 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/726224530' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:23 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/726224530' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:56:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:23 smithi160 ceph-mon[33203]: osdmap e90: 8 total, 8 up, 8 in 2024-09-21T09:56:23.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:23 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/726224530' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:56:23.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:23 smithi070 ceph-mon[41533]: osdmap e90: 8 total, 8 up, 8 in 2024-09-21T09:56:23.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:23 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/726224530' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:56:23.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:23 smithi070 ceph-mon[29589]: osdmap e90: 8 total, 8 up, 8 in 2024-09-21T09:56:24.506 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestTrash.UserRemovingSource (2087 ms) 2024-09-21T09:56:24.506 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestTrash.RestoreMirroringSource 2024-09-21T09:56:24.542 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestTrash.RestoreMirroringSource (35 ms) 2024-09-21T09:56:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:24 smithi160 ceph-mon[33203]: pgmap v120: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:24 smithi160 ceph-mon[33203]: osdmap e91: 8 total, 8 up, 8 in 2024-09-21T09:56:24.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:24 smithi070 ceph-mon[29589]: pgmap v120: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:24.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:24 smithi070 ceph-mon[29589]: osdmap e91: 8 total, 8 up, 8 in 2024-09-21T09:56:24.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:56:24 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:56:24] "GET /metrics HTTP/1.1" 200 33790 "" "Prometheus/2.43.0" 2024-09-21T09:56:24.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:24 smithi070 ceph-mon[41533]: pgmap v120: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:24.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:24 smithi070 ceph-mon[41533]: osdmap e91: 8 total, 8 up, 8 in 2024-09-21T09:56:25.438 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 2 tests from TestTrash (2123 ms total) 2024-09-21T09:56:25.438 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T09:56:25.439 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 1 test from TestOperations 2024-09-21T09:56:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:25 smithi160 ceph-mon[33203]: osdmap e92: 8 total, 8 up, 8 in 2024-09-21T09:56:25.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:25 smithi070 ceph-mon[29589]: osdmap e92: 8 total, 8 up, 8 in 2024-09-21T09:56:25.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:25 smithi070 ceph-mon[41533]: osdmap e92: 8 total, 8 up, 8 in 2024-09-21T09:56:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:26 smithi160 ceph-mon[33203]: pgmap v123: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:26 smithi160 ceph-mon[33203]: osdmap e93: 8 total, 8 up, 8 in 2024-09-21T09:56:26.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:26 smithi070 ceph-mon[29589]: pgmap v123: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:26.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:26 smithi070 ceph-mon[29589]: osdmap e93: 8 total, 8 up, 8 in 2024-09-21T09:56:26.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:26 smithi070 ceph-mon[41533]: pgmap v123: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:26.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:26 smithi070 ceph-mon[41533]: osdmap e93: 8 total, 8 up, 8 in 2024-09-21T09:56:27.445 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestOperations.DisableJournalingCorrupt 2024-09-21T09:56:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:27 smithi160 ceph-mon[33203]: osdmap e94: 8 total, 8 up, 8 in 2024-09-21T09:56:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:27 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/1990594074' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:27 smithi160 ceph-mon[33203]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:27.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:27 smithi070 ceph-mon[29589]: osdmap e94: 8 total, 8 up, 8 in 2024-09-21T09:56:27.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:27 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/1990594074' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:27.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:27 smithi070 ceph-mon[29589]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:27.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:27 smithi070 ceph-mon[41533]: osdmap e94: 8 total, 8 up, 8 in 2024-09-21T09:56:27.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:27 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/1990594074' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:27.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:27 smithi070 ceph-mon[41533]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:28 smithi160 ceph-mon[33203]: pgmap v126: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:28 smithi160 ceph-mon[33203]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:56:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:28 smithi160 ceph-mon[33203]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:56:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:28 smithi160 ceph-mon[33203]: osdmap e95: 8 total, 8 up, 8 in 2024-09-21T09:56:28.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:28 smithi070 ceph-mon[29589]: pgmap v126: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:28.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:28 smithi070 ceph-mon[29589]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:56:28.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:28 smithi070 ceph-mon[29589]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:56:28.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:28 smithi070 ceph-mon[29589]: osdmap e95: 8 total, 8 up, 8 in 2024-09-21T09:56:28.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:28 smithi070 ceph-mon[41533]: pgmap v126: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:28.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:28 smithi070 ceph-mon[41533]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:56:28.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:28 smithi070 ceph-mon[41533]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:56:28.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:28 smithi070 ceph-mon[41533]: osdmap e95: 8 total, 8 up, 8 in 2024-09-21T09:56:29.466 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/test_Operations.cc:18: Skipped 2024-09-21T09:56:29.466 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T09:56:29.466 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestOperations.DisableJournalingCorrupt (2021 ms) 2024-09-21T09:56:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:29 smithi160 ceph-mon[33203]: osdmap e96: 8 total, 8 up, 8 in 2024-09-21T09:56:29.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:29 smithi070 ceph-mon[29589]: osdmap e96: 8 total, 8 up, 8 in 2024-09-21T09:56:29.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:29 smithi070 ceph-mon[41533]: osdmap e96: 8 total, 8 up, 8 in 2024-09-21T09:56:30.464 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 1 test from TestOperations (2021 ms total) 2024-09-21T09:56:30.464 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T09:56:30.464 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 4 tests from TestObjectMap 2024-09-21T09:56:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:30 smithi160 ceph-mon[33203]: pgmap v129: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:30 smithi160 ceph-mon[33203]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:56:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:30 smithi160 ceph-mon[33203]: Cluster is now healthy 2024-09-21T09:56:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:30 smithi160 ceph-mon[33203]: osdmap e97: 8 total, 8 up, 8 in 2024-09-21T09:56:30.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:30 smithi070 ceph-mon[29589]: pgmap v129: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:30.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:30 smithi070 ceph-mon[29589]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:56:30.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:30 smithi070 ceph-mon[29589]: Cluster is now healthy 2024-09-21T09:56:30.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:30 smithi070 ceph-mon[29589]: osdmap e97: 8 total, 8 up, 8 in 2024-09-21T09:56:30.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:30 smithi070 ceph-mon[41533]: pgmap v129: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:30.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:30 smithi070 ceph-mon[41533]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:56:30.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:30 smithi070 ceph-mon[41533]: Cluster is now healthy 2024-09-21T09:56:30.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:30 smithi070 ceph-mon[41533]: osdmap e97: 8 total, 8 up, 8 in 2024-09-21T09:56:31.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:31 smithi160 ceph-mon[33203]: osdmap e98: 8 total, 8 up, 8 in 2024-09-21T09:56:31.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:31 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:31 smithi070 ceph-mon[29589]: osdmap e98: 8 total, 8 up, 8 in 2024-09-21T09:56:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:31 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:31.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:31 smithi070 ceph-mon[41533]: osdmap e98: 8 total, 8 up, 8 in 2024-09-21T09:56:31.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:31 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:32.478 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestObjectMap.RefreshInvalidatesWhenCorrupt 2024-09-21T09:56:32.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:32 smithi070 ceph-mon[41533]: pgmap v132: 73 pgs: 73 active+clean; 579 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:32.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:32 smithi070 ceph-mon[41533]: osdmap e99: 8 total, 8 up, 8 in 2024-09-21T09:56:32.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:32 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/2138714756' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:32.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:32 smithi070 ceph-mon[41533]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:32.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:32 smithi070 ceph-mon[29589]: pgmap v132: 73 pgs: 73 active+clean; 579 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:32.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:32 smithi070 ceph-mon[29589]: osdmap e99: 8 total, 8 up, 8 in 2024-09-21T09:56:32.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:32 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/2138714756' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:32.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:32 smithi070 ceph-mon[29589]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:32 smithi160 ceph-mon[33203]: pgmap v132: 73 pgs: 73 active+clean; 579 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:32 smithi160 ceph-mon[33203]: osdmap e99: 8 total, 8 up, 8 in 2024-09-21T09:56:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:32 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/2138714756' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:32.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:32 smithi160 ceph-mon[33203]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:33 smithi160 ceph-mon[33203]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:56:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:33 smithi160 ceph-mon[33203]: osdmap e100: 8 total, 8 up, 8 in 2024-09-21T09:56:33.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:33 smithi070 ceph-mon[29589]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:56:33.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:33 smithi070 ceph-mon[29589]: osdmap e100: 8 total, 8 up, 8 in 2024-09-21T09:56:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:33 smithi070 ceph-mon[41533]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:56:33.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:33 smithi070 ceph-mon[41533]: osdmap e100: 8 total, 8 up, 8 in 2024-09-21T09:56:34.514 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestObjectMap.RefreshInvalidatesWhenCorrupt (2036 ms) 2024-09-21T09:56:34.514 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestObjectMap.RefreshInvalidatesWhenTooSmall 2024-09-21T09:56:34.541 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestObjectMap.RefreshInvalidatesWhenTooSmall (27 ms) 2024-09-21T09:56:34.542 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestObjectMap.InvalidateFlagOnDisk 2024-09-21T09:56:34.579 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestObjectMap.InvalidateFlagOnDisk (38 ms) 2024-09-21T09:56:34.579 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestObjectMap.AcquireLockInvalidatesWhenTooSmall 2024-09-21T09:56:34.604 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestObjectMap.AcquireLockInvalidatesWhenTooSmall (25 ms) 2024-09-21T09:56:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:34 smithi160 ceph-mon[33203]: pgmap v135: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:34 smithi160 ceph-mon[33203]: osdmap e101: 8 total, 8 up, 8 in 2024-09-21T09:56:34.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:56:34 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:56:34] "GET /metrics HTTP/1.1" 200 33781 "" "Prometheus/2.43.0" 2024-09-21T09:56:34.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:34 smithi070 ceph-mon[41533]: pgmap v135: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:34.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:34 smithi070 ceph-mon[41533]: osdmap e101: 8 total, 8 up, 8 in 2024-09-21T09:56:34.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:34 smithi070 ceph-mon[29589]: pgmap v135: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:34.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:34 smithi070 ceph-mon[29589]: osdmap e101: 8 total, 8 up, 8 in 2024-09-21T09:56:35.491 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 4 tests from TestObjectMap (2126 ms total) 2024-09-21T09:56:35.491 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T09:56:35.491 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 2 tests from TestMirroringWatcher 2024-09-21T09:56:35.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:35 smithi160 ceph-mon[33203]: osdmap e102: 8 total, 8 up, 8 in 2024-09-21T09:56:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:35 smithi070 ceph-mon[29589]: osdmap e102: 8 total, 8 up, 8 in 2024-09-21T09:56:35.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:35 smithi070 ceph-mon[41533]: osdmap e102: 8 total, 8 up, 8 in 2024-09-21T09:56:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:36 smithi160 ceph-mon[33203]: pgmap v138: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:36 smithi160 ceph-mon[33203]: osdmap e103: 8 total, 8 up, 8 in 2024-09-21T09:56:36.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:36 smithi070 ceph-mon[29589]: pgmap v138: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:36.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:36 smithi070 ceph-mon[29589]: osdmap e103: 8 total, 8 up, 8 in 2024-09-21T09:56:36.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:36 smithi070 ceph-mon[41533]: pgmap v138: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:36.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:36 smithi070 ceph-mon[41533]: osdmap e103: 8 total, 8 up, 8 in 2024-09-21T09:56:37.498 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroringWatcher.ModeUpdated 2024-09-21T09:56:37.606 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:37 smithi070 ceph-mon[29589]: osdmap e104: 8 total, 8 up, 8 in 2024-09-21T09:56:37.606 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:37 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/1747635210' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:37.606 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:37 smithi070 ceph-mon[29589]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:37.607 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:37 smithi070 ceph-mon[41533]: osdmap e104: 8 total, 8 up, 8 in 2024-09-21T09:56:37.607 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:37 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/1747635210' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:37.607 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:37 smithi070 ceph-mon[41533]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:37.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:37 smithi160 ceph-mon[33203]: osdmap e104: 8 total, 8 up, 8 in 2024-09-21T09:56:37.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:37 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/1747635210' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:37.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:37 smithi160 ceph-mon[33203]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:38 smithi160 ceph-mon[33203]: pgmap v141: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:38 smithi160 ceph-mon[33203]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:56:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:38 smithi160 ceph-mon[33203]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:56:38.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:38 smithi160 ceph-mon[33203]: osdmap e105: 8 total, 8 up, 8 in 2024-09-21T09:56:38.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:38 smithi070 ceph-mon[29589]: pgmap v141: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:38.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:38 smithi070 ceph-mon[29589]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:56:38.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:38 smithi070 ceph-mon[29589]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:56:38.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:38 smithi070 ceph-mon[29589]: osdmap e105: 8 total, 8 up, 8 in 2024-09-21T09:56:38.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:38 smithi070 ceph-mon[41533]: pgmap v141: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:38.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:38 smithi070 ceph-mon[41533]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:56:38.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:38 smithi070 ceph-mon[41533]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:56:38.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:38 smithi070 ceph-mon[41533]: osdmap e105: 8 total, 8 up, 8 in 2024-09-21T09:56:39.535 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroringWatcher.ModeUpdated (2037 ms) 2024-09-21T09:56:39.535 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroringWatcher.ImageStatusUpdated 2024-09-21T09:56:39.555 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroringWatcher.ImageStatusUpdated (20 ms) 2024-09-21T09:56:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:39 smithi160 ceph-mon[33203]: osdmap e106: 8 total, 8 up, 8 in 2024-09-21T09:56:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:39 smithi160 ceph-mon[33203]: pgmap v144: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:39.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:39 smithi070 ceph-mon[29589]: osdmap e106: 8 total, 8 up, 8 in 2024-09-21T09:56:39.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:39 smithi070 ceph-mon[29589]: pgmap v144: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:39.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:39 smithi070 ceph-mon[41533]: osdmap e106: 8 total, 8 up, 8 in 2024-09-21T09:56:39.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:39 smithi070 ceph-mon[41533]: pgmap v144: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:40.515 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 2 tests from TestMirroringWatcher (2057 ms total) 2024-09-21T09:56:40.515 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T09:56:40.515 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 57 tests from TestMigration 2024-09-21T09:56:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:40 smithi160 ceph-mon[33203]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:56:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:40 smithi160 ceph-mon[33203]: Cluster is now healthy 2024-09-21T09:56:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:40 smithi160 ceph-mon[33203]: osdmap e107: 8 total, 8 up, 8 in 2024-09-21T09:56:40.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:40 smithi070 ceph-mon[29589]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:56:40.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:40 smithi070 ceph-mon[29589]: Cluster is now healthy 2024-09-21T09:56:40.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:40 smithi070 ceph-mon[29589]: osdmap e107: 8 total, 8 up, 8 in 2024-09-21T09:56:40.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:40 smithi070 ceph-mon[41533]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:56:40.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:40 smithi070 ceph-mon[41533]: Cluster is now healthy 2024-09-21T09:56:40.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:40 smithi070 ceph-mon[41533]: osdmap e107: 8 total, 8 up, 8 in 2024-09-21T09:56:41.076 INFO:teuthology.orchestra.run.smithi070.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:56:41.813 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:41 smithi070 ceph-mon[29589]: osdmap e108: 8 total, 8 up, 8 in 2024-09-21T09:56:41.813 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:41 smithi070 ceph-mon[29589]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:56:41.813 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:41 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:41.813 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:41 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:56:41.813 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:41 smithi070 ceph-mon[29589]: pgmap v147: 73 pgs: 73 active+clean; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:41.814 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:41 smithi070 ceph-mon[41533]: osdmap e108: 8 total, 8 up, 8 in 2024-09-21T09:56:41.814 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:41 smithi070 ceph-mon[41533]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:56:41.814 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:41 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:41.814 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:41 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:56:41.814 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:41 smithi070 ceph-mon[41533]: pgmap v147: 73 pgs: 73 active+clean; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:41.833 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-09-21T09:56:41.837 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:41 smithi160 ceph-mon[33203]: osdmap e108: 8 total, 8 up, 8 in 2024-09-21T09:56:41.838 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:41 smithi160 ceph-mon[33203]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:56:41.838 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:41 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:41.838 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:41 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:56:41.838 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:41 smithi160 ceph-mon[33203]: pgmap v147: 73 pgs: 73 active+clean; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:42.395 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:42.591 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:42 smithi070 ceph-mon[29589]: osdmap e109: 8 total, 8 up, 8 in 2024-09-21T09:56:42.591 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:42 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/2753619917' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:42.591 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:42 smithi070 ceph-mon[29589]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:42.591 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:42 smithi070 ceph-mon[41533]: osdmap e109: 8 total, 8 up, 8 in 2024-09-21T09:56:42.591 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:42 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/2753619917' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:42.591 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:42 smithi070 ceph-mon[41533]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:42 smithi160 ceph-mon[33203]: osdmap e109: 8 total, 8 up, 8 in 2024-09-21T09:56:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:42 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/2753619917' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:42 smithi160 ceph-mon[33203]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:56:43.535 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:43 smithi070 ceph-mon[29589]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:56:43.535 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:43 smithi070 ceph-mon[29589]: osdmap e110: 8 total, 8 up, 8 in 2024-09-21T09:56:43.535 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:43 smithi070 ceph-mon[29589]: pgmap v150: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:43.540 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.Empty 2024-09-21T09:56:43.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:43 smithi160 ceph-mon[33203]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:56:43.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:43 smithi160 ceph-mon[33203]: osdmap e110: 8 total, 8 up, 8 in 2024-09-21T09:56:43.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:43 smithi160 ceph-mon[33203]: pgmap v150: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:43.856 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:43 smithi070 ceph-mon[41533]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:56:43.856 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:43 smithi070 ceph-mon[41533]: osdmap e110: 8 total, 8 up, 8 in 2024-09-21T09:56:43.856 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:43 smithi070 ceph-mon[41533]: pgmap v150: 105 pgs: 32 creating+peering, 73 active+clean; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:44.567 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-09-21T09:56:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:44 smithi160 ceph-mon[33203]: osdmap e111: 8 total, 8 up, 8 in 2024-09-21T09:56:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:44 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:44 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:44.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:56:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:56:44] "GET /metrics HTTP/1.1" 200 33781 "" "Prometheus/2.43.0" 2024-09-21T09:56:44.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:44 smithi070 ceph-mon[29589]: osdmap e111: 8 total, 8 up, 8 in 2024-09-21T09:56:44.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:44 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:44.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:44 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:44.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:44 smithi070 ceph-mon[41533]: osdmap e111: 8 total, 8 up, 8 in 2024-09-21T09:56:44.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:44 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:44.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:44 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:45.123 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T09:56:45.123 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (119s) 72s ago 2m 24.7M - 0.25.0 c8568f914cd2 df28d76f1795 2024-09-21T09:56:45.123 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi160 *:3000 running (2m) 0s ago 2m 81.6M - 9.4.7 954c08fa6188 8b9850d5203c 2024-09-21T09:56:45.123 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi070.gansda smithi070 running (105s) 72s ago 105s 14.2M - 18.2.4-871-g6a1bdf85 0adab276c97e 6016b2501c8a 2024-09-21T09:56:45.123 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi160.byyhfz smithi160 running (103s) 0s ago 103s 13.3M - 18.2.4-871-g6a1bdf85 0adab276c97e 93e108d91e65 2024-09-21T09:56:45.123 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi160 *:8443,8765 running (6m) 0s ago 6m 438M - 18.2.4-871-g6a1bdf85 0adab276c97e 8f2bf49a5ce6 2024-09-21T09:56:45.123 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:9283,8765 running (8m) 72s ago 8m 483M - 18.2.4-871-g6a1bdf85 0adab276c97e e4814f12686d 2024-09-21T09:56:45.123 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (8m) 72s ago 8m 44.2M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e cee8ca713d09 2024-09-21T09:56:45.123 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi160 running (7m) 0s ago 7m 44.4M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e 0fd16436dfa0 2024-09-21T09:56:45.124 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (7m) 72s ago 7m 37.1M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e fc0590f324b5 2024-09-21T09:56:45.124 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (2m) 72s ago 2m 16.6M - 1.5.0 0da6a335fe13 2f7c5b1e36eb 2024-09-21T09:56:45.124 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi160 *:9100 running (2m) 0s ago 2m 16.0M - 1.5.0 0da6a335fe13 e0fa2f0d9975 2024-09-21T09:56:45.124 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (6m) 72s ago 6m 44.1M 4016M 18.2.4-871-g6a1bdf85 0adab276c97e 10580e80f007 2024-09-21T09:56:45.124 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (6m) 72s ago 6m 42.9M 4016M 18.2.4-871-g6a1bdf85 0adab276c97e b7c0d1169d1d 2024-09-21T09:56:45.124 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (5m) 72s ago 5m 48.7M 4016M 18.2.4-871-g6a1bdf85 0adab276c97e 58a11b8ddb81 2024-09-21T09:56:45.124 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (5m) 72s ago 5m 41.0M 4016M 18.2.4-871-g6a1bdf85 0adab276c97e aa0c8a732a41 2024-09-21T09:56:45.124 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi160 running (5m) 0s ago 5m 48.1M 4016M 18.2.4-871-g6a1bdf85 0adab276c97e 7325a075c488 2024-09-21T09:56:45.124 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi160 running (4m) 0s ago 4m 47.6M 4016M 18.2.4-871-g6a1bdf85 0adab276c97e f74026345641 2024-09-21T09:56:45.124 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi160 running (4m) 0s ago 4m 56.7M 4016M 18.2.4-871-g6a1bdf85 0adab276c97e 03e6bb54f8b2 2024-09-21T09:56:45.124 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi160 running (3m) 0s ago 3m 55.2M 4016M 18.2.4-871-g6a1bdf85 0adab276c97e e405b21a6146 2024-09-21T09:56:45.124 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi160 *:9095 running (114s) 0s ago 3m 41.8M - 2.43.0 a07b618ecd1d 136d3e422755 2024-09-21T09:56:45.455 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T09:56:45.455 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-09-21T09:56:45.455 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 3 2024-09-21T09:56:45.455 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:56:45.455 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-09-21T09:56:45.455 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 2 2024-09-21T09:56:45.455 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:56:45.455 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-09-21T09:56:45.455 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 8 2024-09-21T09:56:45.455 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:56:45.455 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": { 2024-09-21T09:56:45.456 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 2 2024-09-21T09:56:45.456 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:56:45.456 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-09-21T09:56:45.456 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 15 2024-09-21T09:56:45.456 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T09:56:45.456 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T09:56:45.589 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:56:45.599 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:56:45.741 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:56:45.772 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T09:56:45.772 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208", 2024-09-21T09:56:45.772 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-09-21T09:56:45.772 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-09-21T09:56:45.772 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-09-21T09:56:45.772 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "", 2024-09-21T09:56:45.773 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "", 2024-09-21T09:56:45.773 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-09-21T09:56:45.773 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T09:56:45.773 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:45 smithi070 ceph-mon[41533]: osdmap e112: 8 total, 8 up, 8 in 2024-09-21T09:56:45.773 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:45 smithi070 ceph-mon[41533]: from='client.24821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:45.773 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:45 smithi070 ceph-mon[41533]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:45.773 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:45 smithi070 ceph-mon[41533]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:45.773 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:45 smithi070 ceph-mon[41533]: pgmap v153: 137 pgs: 32 unknown, 32 creating+peering, 73 active+clean; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:45.773 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:45 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/2132833823' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:56:45.774 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:45 smithi070 ceph-mon[29589]: osdmap e112: 8 total, 8 up, 8 in 2024-09-21T09:56:45.774 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:45 smithi070 ceph-mon[29589]: from='client.24821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:45.774 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:45 smithi070 ceph-mon[29589]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:45.774 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:45 smithi070 ceph-mon[29589]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:45.774 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:45 smithi070 ceph-mon[29589]: pgmap v153: 137 pgs: 32 unknown, 32 creating+peering, 73 active+clean; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:45.774 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:45 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/2132833823' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:56:45.815 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:56:45.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:45 smithi160 ceph-mon[33203]: osdmap e112: 8 total, 8 up, 8 in 2024-09-21T09:56:45.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:45 smithi160 ceph-mon[33203]: from='client.24821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:45.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:45 smithi160 ceph-mon[33203]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:45.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:45 smithi160 ceph-mon[33203]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:45.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:45 smithi160 ceph-mon[33203]: pgmap v153: 137 pgs: 32 unknown, 32 creating+peering, 73 active+clean; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-21T09:56:45.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:45 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/2132833823' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:56:45.892 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.Empty (2352 ms) 2024-09-21T09:56:45.892 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.OtherName 2024-09-21T09:56:45.920 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:56:45.930 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:56:46.055 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:56:46.113 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:56:46.191 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.OtherName (299 ms) 2024-09-21T09:56:46.191 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.OtherPool 2024-09-21T09:56:46.222 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:56:46.232 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:56:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:46 smithi160 ceph-mon[33203]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:56:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:46 smithi160 ceph-mon[33203]: osdmap e113: 8 total, 8 up, 8 in 2024-09-21T09:56:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:46 smithi160 ceph-mon[33203]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:46 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:46 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:46.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:46 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:46.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:46 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:46.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:46 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:56:46.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:46 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:56:46.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:46 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:46.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:46 smithi160 ceph-mon[33203]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:56:46.865 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:46 smithi070 ceph-mon[29589]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:56:46.865 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:46 smithi070 ceph-mon[29589]: osdmap e113: 8 total, 8 up, 8 in 2024-09-21T09:56:46.865 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:46 smithi070 ceph-mon[29589]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:46.865 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:46 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:46.865 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:46 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:46.865 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:46 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:46.865 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:46 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:46.865 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:46 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:56:46.866 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:46 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:56:46.866 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:46 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:46.866 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:46 smithi070 ceph-mon[29589]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:56:46.866 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:46 smithi070 ceph-mon[41533]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:56:46.866 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:46 smithi070 ceph-mon[41533]: osdmap e113: 8 total, 8 up, 8 in 2024-09-21T09:56:46.866 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:46 smithi070 ceph-mon[41533]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:46.866 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:46 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:46.866 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:46 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:46.866 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:46 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:46.866 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:46 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:46.867 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:46 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:56:46.867 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:46 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:56:46.867 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:46 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:46.867 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:46 smithi070 ceph-mon[41533]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:56:48.315 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:56:48.371 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:56:48.450 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.OtherPool (2258 ms) 2024-09-21T09:56:48.450 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.OtherNamespace 2024-09-21T09:56:48.480 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:56:48.493 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:56:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:48 smithi160 ceph-mon[33203]: pgmap v155: 137 pgs: 137 active+clean; 579 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 305 KiB/s rd, 18 KiB/s wr, 387 op/s 2024-09-21T09:56:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:48 smithi160 ceph-mon[33203]: osdmap e114: 8 total, 8 up, 8 in 2024-09-21T09:56:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:48 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:48 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:56:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:48 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:48.625 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:56:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:48 smithi070 ceph-mon[29589]: pgmap v155: 137 pgs: 137 active+clean; 579 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 305 KiB/s rd, 18 KiB/s wr, 387 op/s 2024-09-21T09:56:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:48 smithi070 ceph-mon[29589]: osdmap e114: 8 total, 8 up, 8 in 2024-09-21T09:56:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:48 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:48 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:56:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:48 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:48.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:48 smithi070 ceph-mon[41533]: pgmap v155: 137 pgs: 137 active+clean; 579 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 305 KiB/s rd, 18 KiB/s wr, 387 op/s 2024-09-21T09:56:48.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:48 smithi070 ceph-mon[41533]: osdmap e114: 8 total, 8 up, 8 in 2024-09-21T09:56:48.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:48 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:48.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:48 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:56:48.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:48 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:48.685 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:56:48.762 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.OtherNamespace (313 ms) 2024-09-21T09:56:48.762 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.DataPool 2024-09-21T09:56:48.788 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:56:48.798 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:56:48.944 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:56:49.016 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:56:49.099 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.DataPool (336 ms) 2024-09-21T09:56:49.099 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.AbortAfterPrepare 2024-09-21T09:56:49.126 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:56:49.136 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:56:49.261 INFO:tasks.workunit.client.0.smithi070.stdout:migration_abort 2024-09-21T09:56:49.362 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.AbortAfterPrepare (263 ms) 2024-09-21T09:56:49.362 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.AbortAfterFailedPrepare 2024-09-21T09:56:49.389 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:56:49.398 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:56:49.490 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.AbortAfterFailedPrepare (128 ms) 2024-09-21T09:56:49.491 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.AbortAfterExecute 2024-09-21T09:56:49.518 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:56:49.528 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:56:49.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:49 smithi160 ceph-mon[33203]: Upgrade: Target is version 19.3.0-5092-g62da54b4 (unknown) 2024-09-21T09:56:49.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:49 smithi160 ceph-mon[33203]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, digests ['quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47'] 2024-09-21T09:56:49.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:49 smithi160 ceph-mon[33203]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-21T09:56:49.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:49 smithi160 ceph-mon[33203]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 on smithi160 2024-09-21T09:56:49.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:49 smithi160 ceph-mon[33203]: osdmap e115: 8 total, 8 up, 8 in 2024-09-21T09:56:49.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:49 smithi070 ceph-mon[29589]: Upgrade: Target is version 19.3.0-5092-g62da54b4 (unknown) 2024-09-21T09:56:49.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:49 smithi070 ceph-mon[29589]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, digests ['quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47'] 2024-09-21T09:56:49.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:49 smithi070 ceph-mon[29589]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-21T09:56:49.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:49 smithi070 ceph-mon[29589]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 on smithi160 2024-09-21T09:56:49.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:49 smithi070 ceph-mon[29589]: osdmap e115: 8 total, 8 up, 8 in 2024-09-21T09:56:49.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:49 smithi070 ceph-mon[41533]: Upgrade: Target is version 19.3.0-5092-g62da54b4 (unknown) 2024-09-21T09:56:49.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:49 smithi070 ceph-mon[41533]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, digests ['quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47'] 2024-09-21T09:56:49.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:49 smithi070 ceph-mon[41533]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-21T09:56:49.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:49 smithi070 ceph-mon[41533]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 on smithi160 2024-09-21T09:56:49.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:49 smithi070 ceph-mon[41533]: osdmap e115: 8 total, 8 up, 8 in 2024-09-21T09:56:49.657 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:56:49.718 INFO:tasks.workunit.client.0.smithi070.stdout:migration_abort 2024-09-21T09:56:49.808 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.AbortAfterExecute (318 ms) 2024-09-21T09:56:49.809 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.OtherPoolAbortAfterExecute 2024-09-21T09:56:49.834 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:56:49.843 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:56:49.968 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:56:50.029 INFO:tasks.workunit.client.0.smithi070.stdout:migration_abort 2024-09-21T09:56:50.125 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.OtherPoolAbortAfterExecute (317 ms) 2024-09-21T09:56:50.125 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.OtherNamespaceAbortAfterExecute 2024-09-21T09:56:50.153 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:56:50.166 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:56:50.284 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:56:50.342 INFO:tasks.workunit.client.0.smithi070.stdout:migration_abort 2024-09-21T09:56:50.444 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.OtherNamespaceAbortAfterExecute (318 ms) 2024-09-21T09:56:50.444 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.MirroringSamePool 2024-09-21T09:56:50.470 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:56:50.479 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/test_Migration.cc:635: Skipped 2024-09-21T09:56:50.479 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T09:56:50.485 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestMigration.MirroringSamePool (42 ms) 2024-09-21T09:56:50.485 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.MirroringAbort 2024-09-21T09:56:50.512 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:56:50.521 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/test_Migration.cc:653: Skipped 2024-09-21T09:56:50.521 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T09:56:50.527 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestMigration.MirroringAbort (42 ms) 2024-09-21T09:56:50.527 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.MirroringOtherPoolDisabled 2024-09-21T09:56:50.559 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:56:50.569 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/test_Migration.cc:676: Skipped 2024-09-21T09:56:50.569 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T09:56:50.575 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestMigration.MirroringOtherPoolDisabled (48 ms) 2024-09-21T09:56:50.576 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.MirroringOtherPoolEnabled 2024-09-21T09:56:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:50 smithi160 ceph-mon[33203]: pgmap v158: 137 pgs: 137 active+clean; 579 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 306 KiB/s rd, 18 KiB/s wr, 388 op/s 2024-09-21T09:56:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:50 smithi160 ceph-mon[33203]: osdmap e116: 8 total, 8 up, 8 in 2024-09-21T09:56:50.603 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:56:50.613 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/test_Migration.cc:694: Skipped 2024-09-21T09:56:50.613 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T09:56:50.619 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestMigration.MirroringOtherPoolEnabled (44 ms) 2024-09-21T09:56:50.619 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.MirroringPool 2024-09-21T09:56:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:50 smithi070 ceph-mon[29589]: pgmap v158: 137 pgs: 137 active+clean; 579 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 306 KiB/s rd, 18 KiB/s wr, 388 op/s 2024-09-21T09:56:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:50 smithi070 ceph-mon[29589]: osdmap e116: 8 total, 8 up, 8 in 2024-09-21T09:56:50.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:50 smithi070 ceph-mon[41533]: pgmap v158: 137 pgs: 137 active+clean; 579 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 306 KiB/s rd, 18 KiB/s wr, 388 op/s 2024-09-21T09:56:50.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:50 smithi070 ceph-mon[41533]: osdmap e116: 8 total, 8 up, 8 in 2024-09-21T09:56:50.647 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:56:50.658 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/test_Migration.cc:714: Skipped 2024-09-21T09:56:50.658 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T09:56:50.664 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestMigration.MirroringPool (45 ms) 2024-09-21T09:56:50.664 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.Group 2024-09-21T09:56:50.690 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:56:50.708 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:56:50.841 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:56:50.902 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:56:50.995 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.Group (331 ms) 2024-09-21T09:56:50.996 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.GroupAbort 2024-09-21T09:56:51.023 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:56:51.044 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:56:51.181 INFO:tasks.workunit.client.0.smithi070.stdout:migration_abort 2024-09-21T09:56:51.290 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.GroupAbort (295 ms) 2024-09-21T09:56:51.291 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.NoSnaps 2024-09-21T09:56:51.335 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:56:51.344 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 0~8388609 2024-09-21T09:56:51.409 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 16777218~8388609 2024-09-21T09:56:51.460 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:56:51.636 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:56:51.733 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:56:51.844 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.NoSnaps (554 ms) 2024-09-21T09:56:51.845 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.NoSnapsOtherPool 2024-09-21T09:56:51.871 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:56:51.881 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 0~8388609 2024-09-21T09:56:51.934 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 16777218~8388609 2024-09-21T09:56:51.987 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 0~8388609 2024-09-21T09:56:51.992 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 16777218~8388609 2024-09-21T09:56:52.047 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:56:52.219 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:56:52.332 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:56:52.460 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.NoSnapsOtherPool (616 ms) 2024-09-21T09:56:52.460 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.NoSnapsDataPool 2024-09-21T09:56:52.490 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:56:52.499 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 0~8388609 2024-09-21T09:56:52.558 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 16777218~8388609 2024-09-21T09:56:52.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:52 smithi160 ceph-mon[33203]: pgmap v160: 137 pgs: 137 active+clean; 579 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 252 KiB/s rd, 15 KiB/s wr, 320 op/s 2024-09-21T09:56:52.603 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:56:52.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:52 smithi070 ceph-mon[41533]: pgmap v160: 137 pgs: 137 active+clean; 579 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 252 KiB/s rd, 15 KiB/s wr, 320 op/s 2024-09-21T09:56:52.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:52 smithi070 ceph-mon[29589]: pgmap v160: 137 pgs: 137 active+clean; 579 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 252 KiB/s rd, 15 KiB/s wr, 320 op/s 2024-09-21T09:56:52.790 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:56:52.915 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:56:53.045 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.NoSnapsDataPool (585 ms) 2024-09-21T09:56:53.046 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.NoSnapsShrinkAfterPrepare 2024-09-21T09:56:53.073 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:56:53.082 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 0~8388609 2024-09-21T09:56:53.146 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 16777218~8388609 2024-09-21T09:56:53.197 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:56:53.372 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 41943040 2024-09-21T09:56:53.380 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:56:53.485 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:56:53.586 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.NoSnapsShrinkAfterPrepare (541 ms) 2024-09-21T09:56:53.586 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.NoSnapsShrinkToZeroBeforePrepare 2024-09-21T09:56:53.611 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:56:53.620 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 0~8388609 2024-09-21T09:56:53.671 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 16777218~8388609 2024-09-21T09:56:53.727 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 0 2024-09-21T09:56:53.740 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:56:53.837 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:56:53.901 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:56:53.943 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.NoSnapsShrinkToZeroBeforePrepare (357 ms) 2024-09-21T09:56:53.943 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.NoSnapsShrinkToZeroAfterPrepare 2024-09-21T09:56:53.974 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:56:53.983 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 0~8388609 2024-09-21T09:56:54.028 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 16777218~8388609 2024-09-21T09:56:54.080 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:56:54.246 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 0 2024-09-21T09:56:54.258 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:56:54.322 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:56:54.374 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.NoSnapsShrinkToZeroAfterPrepare (431 ms) 2024-09-21T09:56:54.374 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.NoSnapsExpandAfterPrepare 2024-09-21T09:56:54.401 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:56:54.411 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 0~8388609 2024-09-21T09:56:54.468 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 16777218~8388609 2024-09-21T09:56:54.530 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:56:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:54 smithi160 ceph-mon[33203]: pgmap v161: 137 pgs: 137 active+clean; 65 MiB data, 432 MiB used, 715 GiB / 715 GiB avail; 20 MiB/s rd, 19 MiB/s wr, 1.58k op/s 2024-09-21T09:56:54.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:54 smithi070 ceph-mon[41533]: pgmap v161: 137 pgs: 137 active+clean; 65 MiB data, 432 MiB used, 715 GiB / 715 GiB avail; 20 MiB/s rd, 19 MiB/s wr, 1.58k op/s 2024-09-21T09:56:54.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:54 smithi070 ceph-mon[29589]: pgmap v161: 137 pgs: 137 active+clean; 65 MiB data, 432 MiB used, 715 GiB / 715 GiB avail; 20 MiB/s rd, 19 MiB/s wr, 1.58k op/s 2024-09-21T09:56:54.628 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:56:54 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:56:54] "GET /metrics HTTP/1.1" 200 38515 "" "Prometheus/2.43.0" 2024-09-21T09:56:54.710 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 167772160 2024-09-21T09:56:54.718 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:56:54.823 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:56:54.962 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.NoSnapsExpandAfterPrepare (588 ms) 2024-09-21T09:56:54.962 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.NoSnapsSnapAfterPrepare 2024-09-21T09:56:54.994 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:56:55.004 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 0~8388609 2024-09-21T09:56:55.050 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 16777218~8388609 2024-09-21T09:56:55.098 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:56:55.281 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: after_prepare_snap 2024-09-21T09:56:55.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:55 smithi160 ceph-mon[33203]: osdmap e117: 8 total, 8 up, 8 in 2024-09-21T09:56:55.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:55 smithi070 ceph-mon[41533]: osdmap e117: 8 total, 8 up, 8 in 2024-09-21T09:56:55.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:55 smithi070 ceph-mon[29589]: osdmap e117: 8 total, 8 up, 8 in 2024-09-21T09:56:56.339 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 41943040 2024-09-21T09:56:56.362 INFO:tasks.workunit.client.0.smithi070.stdout:write: * 0~1000 2024-09-21T09:56:56.368 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:56:56.481 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:56:56.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:56 smithi160 ceph-mon[33203]: pgmap v163: 137 pgs: 137 active+clean; 65 MiB data, 432 MiB used, 715 GiB / 715 GiB avail; 17 MiB/s rd, 16 MiB/s wr, 1.36k op/s 2024-09-21T09:56:56.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:56 smithi160 ceph-mon[33203]: osdmap e118: 8 total, 8 up, 8 in 2024-09-21T09:56:56.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:56 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:56.621 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.NoSnapsSnapAfterPrepare (1659 ms) 2024-09-21T09:56:56.621 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.Snaps 2024-09-21T09:56:56.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:56 smithi070 ceph-mon[29589]: pgmap v163: 137 pgs: 137 active+clean; 65 MiB data, 432 MiB used, 715 GiB / 715 GiB avail; 17 MiB/s rd, 16 MiB/s wr, 1.36k op/s 2024-09-21T09:56:56.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:56 smithi070 ceph-mon[29589]: osdmap e118: 8 total, 8 up, 8 in 2024-09-21T09:56:56.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:56 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:56.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:56 smithi070 ceph-mon[41533]: pgmap v163: 137 pgs: 137 active+clean; 65 MiB data, 432 MiB used, 715 GiB / 715 GiB avail; 17 MiB/s rd, 16 MiB/s wr, 1.36k op/s 2024-09-21T09:56:56.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:56 smithi070 ceph-mon[41533]: osdmap e118: 8 total, 8 up, 8 in 2024-09-21T09:56:56.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:56 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:56:56.648 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:56:56.658 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 0~8388609 2024-09-21T09:56:56.704 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap1 2024-09-21T09:56:57.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:57 smithi160 ceph-mon[33203]: osdmap e119: 8 total, 8 up, 8 in 2024-09-21T09:56:57.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:57 smithi070 ceph-mon[41533]: osdmap e119: 8 total, 8 up, 8 in 2024-09-21T09:56:57.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:57 smithi070 ceph-mon[29589]: osdmap e119: 8 total, 8 up, 8 in 2024-09-21T09:56:58.351 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 8388609~8388609 2024-09-21T09:56:58.368 INFO:tasks.workunit.client.0.smithi070.stdout:write: X 0~1000 2024-09-21T09:56:58.368 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 1010~1000 2024-09-21T09:56:58.368 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap2 2024-09-21T09:56:58.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:58 smithi160 ceph-mon[33203]: pgmap v166: 137 pgs: 137 active+clean; 209 MiB data, 897 MiB used, 714 GiB / 715 GiB avail; 93 MiB/s rd, 67 MiB/s wr, 2.88k op/s 2024-09-21T09:56:58.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:58 smithi160 ceph-mon[33203]: osdmap e120: 8 total, 8 up, 8 in 2024-09-21T09:56:58.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:58 smithi070 ceph-mon[29589]: pgmap v166: 137 pgs: 137 active+clean; 209 MiB data, 897 MiB used, 714 GiB / 715 GiB avail; 93 MiB/s rd, 67 MiB/s wr, 2.88k op/s 2024-09-21T09:56:58.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:58 smithi070 ceph-mon[29589]: osdmap e120: 8 total, 8 up, 8 in 2024-09-21T09:56:58.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:58 smithi070 ceph-mon[41533]: pgmap v166: 137 pgs: 137 active+clean; 209 MiB data, 897 MiB used, 714 GiB / 715 GiB avail; 93 MiB/s rd, 67 MiB/s wr, 2.88k op/s 2024-09-21T09:56:58.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:58 smithi070 ceph-mon[41533]: osdmap e120: 8 total, 8 up, 8 in 2024-09-21T09:56:59.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:56:59 smithi070 ceph-mon[29589]: osdmap e121: 8 total, 8 up, 8 in 2024-09-21T09:56:59.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:56:59 smithi070 ceph-mon[41533]: osdmap e121: 8 total, 8 up, 8 in 2024-09-21T09:56:59.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:56:59 smithi160 ceph-mon[33203]: osdmap e121: 8 total, 8 up, 8 in 2024-09-21T09:57:00.358 INFO:tasks.workunit.client.0.smithi070.stdout:write: X 8388609~1000 2024-09-21T09:57:00.365 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 16777228~1000 2024-09-21T09:57:00.365 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 167772160 2024-09-21T09:57:00.373 INFO:tasks.workunit.client.0.smithi070.stdout:write: 2 83886079~8388609 2024-09-21T09:57:00.398 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap3 2024-09-21T09:57:00.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:00 smithi070 ceph-mon[41533]: pgmap v169: 137 pgs: 137 active+clean; 209 MiB data, 897 MiB used, 714 GiB / 715 GiB avail; 109 MiB/s rd, 72 MiB/s wr, 1.95k op/s 2024-09-21T09:57:00.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:00 smithi070 ceph-mon[41533]: osdmap e122: 8 total, 8 up, 8 in 2024-09-21T09:57:00.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:00 smithi070 ceph-mon[29589]: pgmap v169: 137 pgs: 137 active+clean; 209 MiB data, 897 MiB used, 714 GiB / 715 GiB avail; 109 MiB/s rd, 72 MiB/s wr, 1.95k op/s 2024-09-21T09:57:00.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:00 smithi070 ceph-mon[29589]: osdmap e122: 8 total, 8 up, 8 in 2024-09-21T09:57:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:00 smithi160 ceph-mon[33203]: pgmap v169: 137 pgs: 137 active+clean; 209 MiB data, 897 MiB used, 714 GiB / 715 GiB avail; 109 MiB/s rd, 72 MiB/s wr, 1.95k op/s 2024-09-21T09:57:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:00 smithi160 ceph-mon[33203]: osdmap e122: 8 total, 8 up, 8 in 2024-09-21T09:57:01.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:01 smithi070 ceph-mon[29589]: osdmap e123: 8 total, 8 up, 8 in 2024-09-21T09:57:01.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:01 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:01.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:01 smithi070 ceph-mon[41533]: osdmap e123: 8 total, 8 up, 8 in 2024-09-21T09:57:01.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:01 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:01 smithi160 ceph-mon[33203]: osdmap e123: 8 total, 8 up, 8 in 2024-09-21T09:57:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:01 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:02.365 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:57:02.386 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 83886079~1 2024-09-21T09:57:02.393 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:57:02.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:02 smithi070 ceph-mon[29589]: pgmap v172: 137 pgs: 137 active+clean; 209 MiB data, 897 MiB used, 714 GiB / 715 GiB avail 2024-09-21T09:57:02.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:02 smithi070 ceph-mon[29589]: osdmap e124: 8 total, 8 up, 8 in 2024-09-21T09:57:02.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:02 smithi070 ceph-mon[41533]: pgmap v172: 137 pgs: 137 active+clean; 209 MiB data, 897 MiB used, 714 GiB / 715 GiB avail 2024-09-21T09:57:02.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:02 smithi070 ceph-mon[41533]: osdmap e124: 8 total, 8 up, 8 in 2024-09-21T09:57:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:02 smithi160 ceph-mon[33203]: pgmap v172: 137 pgs: 137 active+clean; 209 MiB data, 897 MiB used, 714 GiB / 715 GiB avail 2024-09-21T09:57:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:02 smithi160 ceph-mon[33203]: osdmap e124: 8 total, 8 up, 8 in 2024-09-21T09:57:03.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:03 smithi070 ceph-mon[29589]: osdmap e125: 8 total, 8 up, 8 in 2024-09-21T09:57:03.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:03 smithi070 ceph-mon[41533]: osdmap e125: 8 total, 8 up, 8 in 2024-09-21T09:57:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:03 smithi160 ceph-mon[33203]: osdmap e125: 8 total, 8 up, 8 in 2024-09-21T09:57:04.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:04 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:57:04] "GET /metrics HTTP/1.1" 200 38643 "" "Prometheus/2.43.0" 2024-09-21T09:57:04.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:04 smithi070 ceph-mon[29589]: pgmap v175: 137 pgs: 137 active+clean; 249 MiB data, 1001 MiB used, 714 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 MiB/s wr, 103 op/s 2024-09-21T09:57:04.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:04 smithi070 ceph-mon[29589]: osdmap e126: 8 total, 8 up, 8 in 2024-09-21T09:57:04.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:04 smithi070 ceph-mon[41533]: pgmap v175: 137 pgs: 137 active+clean; 249 MiB data, 1001 MiB used, 714 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 MiB/s wr, 103 op/s 2024-09-21T09:57:04.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:04 smithi070 ceph-mon[41533]: osdmap e126: 8 total, 8 up, 8 in 2024-09-21T09:57:04.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:04 smithi160 ceph-mon[33203]: pgmap v175: 137 pgs: 137 active+clean; 249 MiB data, 1001 MiB used, 714 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 MiB/s wr, 103 op/s 2024-09-21T09:57:04.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:04 smithi160 ceph-mon[33203]: osdmap e126: 8 total, 8 up, 8 in 2024-09-21T09:57:05.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:05 smithi070 ceph-mon[29589]: osdmap e127: 8 total, 8 up, 8 in 2024-09-21T09:57:05.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:05 smithi070 ceph-mon[41533]: osdmap e127: 8 total, 8 up, 8 in 2024-09-21T09:57:05.748 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:57:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:05 smithi160 ceph-mon[33203]: osdmap e127: 8 total, 8 up, 8 in 2024-09-21T09:57:05.882 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:57:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:06 smithi070 ceph-mon[29589]: pgmap v178: 137 pgs: 137 active+clean; 249 MiB data, 1001 MiB used, 714 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 MiB/s wr, 103 op/s 2024-09-21T09:57:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:06 smithi070 ceph-mon[29589]: osdmap e128: 8 total, 8 up, 8 in 2024-09-21T09:57:06.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:06 smithi070 ceph-mon[41533]: pgmap v178: 137 pgs: 137 active+clean; 249 MiB data, 1001 MiB used, 714 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 MiB/s wr, 103 op/s 2024-09-21T09:57:06.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:06 smithi070 ceph-mon[41533]: osdmap e128: 8 total, 8 up, 8 in 2024-09-21T09:57:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:06 smithi160 ceph-mon[33203]: pgmap v178: 137 pgs: 137 active+clean; 249 MiB data, 1001 MiB used, 714 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 MiB/s wr, 103 op/s 2024-09-21T09:57:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:06 smithi160 ceph-mon[33203]: osdmap e128: 8 total, 8 up, 8 in 2024-09-21T09:57:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:07 smithi160 ceph-mon[33203]: osdmap e129: 8 total, 8 up, 8 in 2024-09-21T09:57:07.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:07 smithi070 ceph-mon[41533]: osdmap e129: 8 total, 8 up, 8 in 2024-09-21T09:57:07.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:07 smithi070 ceph-mon[29589]: osdmap e129: 8 total, 8 up, 8 in 2024-09-21T09:57:08.676 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.Snaps (12055 ms) 2024-09-21T09:57:08.676 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.SnapsOtherPool 2024-09-21T09:57:08.705 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:57:08.716 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 0~8388609 2024-09-21T09:57:08.768 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap1 2024-09-21T09:57:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:08 smithi160 ceph-mon[33203]: pgmap v181: 137 pgs: 137 active+clean; 269 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 501 op/s 2024-09-21T09:57:08.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:08 smithi160 ceph-mon[33203]: osdmap e130: 8 total, 8 up, 8 in 2024-09-21T09:57:08.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:08 smithi070 ceph-mon[41533]: pgmap v181: 137 pgs: 137 active+clean; 269 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 501 op/s 2024-09-21T09:57:08.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:08 smithi070 ceph-mon[41533]: osdmap e130: 8 total, 8 up, 8 in 2024-09-21T09:57:08.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:08 smithi070 ceph-mon[29589]: pgmap v181: 137 pgs: 137 active+clean; 269 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 501 op/s 2024-09-21T09:57:08.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:08 smithi070 ceph-mon[29589]: osdmap e130: 8 total, 8 up, 8 in 2024-09-21T09:57:09.848 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:09 smithi160 ceph-mon[33203]: osdmap e131: 8 total, 8 up, 8 in 2024-09-21T09:57:09.848 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:09 smithi160 ceph-mon[33203]: osdmap e132: 8 total, 8 up, 8 in 2024-09-21T09:57:09.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:09 smithi070 ceph-mon[29589]: osdmap e131: 8 total, 8 up, 8 in 2024-09-21T09:57:09.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:09 smithi070 ceph-mon[29589]: osdmap e132: 8 total, 8 up, 8 in 2024-09-21T09:57:09.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:09 smithi070 ceph-mon[41533]: osdmap e131: 8 total, 8 up, 8 in 2024-09-21T09:57:09.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:09 smithi070 ceph-mon[41533]: osdmap e132: 8 total, 8 up, 8 in 2024-09-21T09:57:10.327 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 8388609~8388609 2024-09-21T09:57:10.341 INFO:tasks.workunit.client.0.smithi070.stdout:write: X 0~1000 2024-09-21T09:57:10.341 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 1010~1000 2024-09-21T09:57:10.341 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap2 2024-09-21T09:57:10.743 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:10 smithi160 ceph-mon[33203]: pgmap v184: 137 pgs: 137 active+clean; 269 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 501 op/s 2024-09-21T09:57:10.743 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:10 smithi160 ceph-mon[33203]: osdmap e133: 8 total, 8 up, 8 in 2024-09-21T09:57:10.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:10 smithi070 ceph-mon[29589]: pgmap v184: 137 pgs: 137 active+clean; 269 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 501 op/s 2024-09-21T09:57:10.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:10 smithi070 ceph-mon[29589]: osdmap e133: 8 total, 8 up, 8 in 2024-09-21T09:57:10.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:10 smithi070 ceph-mon[41533]: pgmap v184: 137 pgs: 137 active+clean; 269 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 501 op/s 2024-09-21T09:57:10.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:10 smithi070 ceph-mon[41533]: osdmap e133: 8 total, 8 up, 8 in 2024-09-21T09:57:12.333 INFO:tasks.workunit.client.0.smithi070.stdout:write: X 8388609~1000 2024-09-21T09:57:12.340 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 16777228~1000 2024-09-21T09:57:12.340 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 167772160 2024-09-21T09:57:12.349 INFO:tasks.workunit.client.0.smithi070.stdout:write: 2 83886079~8388609 2024-09-21T09:57:12.367 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap3 2024-09-21T09:57:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:12 smithi160 ceph-mon[33203]: pgmap v187: 137 pgs: 137 active+clean; 269 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-09-21T09:57:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:12 smithi160 ceph-mon[33203]: osdmap e134: 8 total, 8 up, 8 in 2024-09-21T09:57:12.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:12 smithi070 ceph-mon[41533]: pgmap v187: 137 pgs: 137 active+clean; 269 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-09-21T09:57:12.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:12 smithi070 ceph-mon[41533]: osdmap e134: 8 total, 8 up, 8 in 2024-09-21T09:57:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:12 smithi070 ceph-mon[29589]: pgmap v187: 137 pgs: 137 active+clean; 269 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-09-21T09:57:12.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:12 smithi070 ceph-mon[29589]: osdmap e134: 8 total, 8 up, 8 in 2024-09-21T09:57:13.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:13 smithi160 ceph-mon[33203]: osdmap e135: 8 total, 8 up, 8 in 2024-09-21T09:57:13.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:13 smithi070 ceph-mon[41533]: osdmap e135: 8 total, 8 up, 8 in 2024-09-21T09:57:13.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:13 smithi070 ceph-mon[29589]: osdmap e135: 8 total, 8 up, 8 in 2024-09-21T09:57:14.334 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:57:14.357 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 83886079~1 2024-09-21T09:57:14.366 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 0~8388609 2024-09-21T09:57:14.376 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 16777218~8388609 2024-09-21T09:57:14.424 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:57:14.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:14 smithi160 ceph-mon[33203]: pgmap v190: 137 pgs: 137 active+clean; 281 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 32 MiB/s rd, 8.0 MiB/s wr, 134 op/s 2024-09-21T09:57:14.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:14 smithi160 ceph-mon[33203]: osdmap e136: 8 total, 8 up, 8 in 2024-09-21T09:57:14.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:14 smithi160 ceph-mon[33203]: osdmap e137: 8 total, 8 up, 8 in 2024-09-21T09:57:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:14 smithi070 ceph-mon[29589]: pgmap v190: 137 pgs: 137 active+clean; 281 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 32 MiB/s rd, 8.0 MiB/s wr, 134 op/s 2024-09-21T09:57:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:14 smithi070 ceph-mon[29589]: osdmap e136: 8 total, 8 up, 8 in 2024-09-21T09:57:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:14 smithi070 ceph-mon[29589]: osdmap e137: 8 total, 8 up, 8 in 2024-09-21T09:57:14.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:14 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:57:14] "GET /metrics HTTP/1.1" 200 38643 "" "Prometheus/2.43.0" 2024-09-21T09:57:14.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:14 smithi070 ceph-mon[41533]: pgmap v190: 137 pgs: 137 active+clean; 281 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 32 MiB/s rd, 8.0 MiB/s wr, 134 op/s 2024-09-21T09:57:14.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:14 smithi070 ceph-mon[41533]: osdmap e136: 8 total, 8 up, 8 in 2024-09-21T09:57:14.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:14 smithi070 ceph-mon[41533]: osdmap e137: 8 total, 8 up, 8 in 2024-09-21T09:57:16.070 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-09-21T09:57:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:16 smithi160 ceph-mon[33203]: pgmap v193: 137 pgs: 137 active+clean; 281 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 32 MiB/s rd, 8.0 MiB/s wr, 134 op/s 2024-09-21T09:57:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:16 smithi160 ceph-mon[33203]: osdmap e138: 8 total, 8 up, 8 in 2024-09-21T09:57:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:16 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:16.601 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T09:57:16.601 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (2m) 30s ago 3m 26.8M - 0.25.0 c8568f914cd2 df28d76f1795 2024-09-21T09:57:16.601 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi160 *:3000 running (3m) 32s ago 3m 81.6M - 9.4.7 954c08fa6188 8b9850d5203c 2024-09-21T09:57:16.601 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi070.gansda smithi070 running (2m) 30s ago 2m 15.0M - 18.2.4-871-g6a1bdf85 0adab276c97e 6016b2501c8a 2024-09-21T09:57:16.601 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi160.byyhfz smithi160 running (2m) 32s ago 2m 13.3M - 18.2.4-871-g6a1bdf85 0adab276c97e 93e108d91e65 2024-09-21T09:57:16.601 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi160 *:8443,8765 running (7m) 32s ago 7m 438M - 18.2.4-871-g6a1bdf85 0adab276c97e 8f2bf49a5ce6 2024-09-21T09:57:16.601 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:9283,8765 running (9m) 30s ago 9m 489M - 18.2.4-871-g6a1bdf85 0adab276c97e e4814f12686d 2024-09-21T09:57:16.601 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (9m) 30s ago 9m 47.9M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e cee8ca713d09 2024-09-21T09:57:16.601 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi160 running (7m) 32s ago 7m 44.4M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e 0fd16436dfa0 2024-09-21T09:57:16.601 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (7m) 30s ago 7m 40.5M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e fc0590f324b5 2024-09-21T09:57:16.601 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (3m) 30s ago 3m 16.7M - 1.5.0 0da6a335fe13 2f7c5b1e36eb 2024-09-21T09:57:16.601 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi160 *:9100 running (3m) 32s ago 3m 16.0M - 1.5.0 0da6a335fe13 e0fa2f0d9975 2024-09-21T09:57:16.602 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (7m) 30s ago 7m 50.0M 4016M 18.2.4-871-g6a1bdf85 0adab276c97e 10580e80f007 2024-09-21T09:57:16.602 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (6m) 30s ago 6m 50.0M 4016M 18.2.4-871-g6a1bdf85 0adab276c97e b7c0d1169d1d 2024-09-21T09:57:16.602 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (6m) 30s ago 6m 55.8M 4016M 18.2.4-871-g6a1bdf85 0adab276c97e 58a11b8ddb81 2024-09-21T09:57:16.602 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (5m) 30s ago 5m 48.0M 4016M 18.2.4-871-g6a1bdf85 0adab276c97e aa0c8a732a41 2024-09-21T09:57:16.602 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi160 running (5m) 32s ago 5m 48.1M 4016M 18.2.4-871-g6a1bdf85 0adab276c97e 7325a075c488 2024-09-21T09:57:16.602 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi160 running (5m) 32s ago 5m 47.6M 4016M 18.2.4-871-g6a1bdf85 0adab276c97e f74026345641 2024-09-21T09:57:16.602 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi160 running (4m) 32s ago 4m 56.7M 4016M 18.2.4-871-g6a1bdf85 0adab276c97e 03e6bb54f8b2 2024-09-21T09:57:16.602 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi160 running (4m) 32s ago 4m 55.2M 4016M 18.2.4-871-g6a1bdf85 0adab276c97e e405b21a6146 2024-09-21T09:57:16.602 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi160 *:9095 running (2m) 32s ago 3m 41.8M - 2.43.0 a07b618ecd1d 136d3e422755 2024-09-21T09:57:16.602 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:16 smithi070 ceph-mon[29589]: pgmap v193: 137 pgs: 137 active+clean; 281 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 32 MiB/s rd, 8.0 MiB/s wr, 134 op/s 2024-09-21T09:57:16.602 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:16 smithi070 ceph-mon[29589]: osdmap e138: 8 total, 8 up, 8 in 2024-09-21T09:57:16.603 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:16 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:16.603 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:16 smithi070 ceph-mon[41533]: pgmap v193: 137 pgs: 137 active+clean; 281 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 32 MiB/s rd, 8.0 MiB/s wr, 134 op/s 2024-09-21T09:57:16.603 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:16 smithi070 ceph-mon[41533]: osdmap e138: 8 total, 8 up, 8 in 2024-09-21T09:57:16.603 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:16 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:16.916 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T09:57:16.916 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-09-21T09:57:16.916 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 3 2024-09-21T09:57:16.916 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:57:16.916 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-09-21T09:57:16.916 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 2 2024-09-21T09:57:16.916 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:57:16.916 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-09-21T09:57:16.916 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 8 2024-09-21T09:57:16.916 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:57:16.916 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": { 2024-09-21T09:57:16.916 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 2 2024-09-21T09:57:16.917 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:57:16.917 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-09-21T09:57:16.917 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 15 2024-09-21T09:57:16.917 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T09:57:16.917 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T09:57:17.184 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T09:57:17.184 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47", 2024-09-21T09:57:17.184 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-09-21T09:57:17.185 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-09-21T09:57:17.185 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-09-21T09:57:17.185 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "0/2 daemons upgraded", 2024-09-21T09:57:17.185 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 image on host smithi160", 2024-09-21T09:57:17.185 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-09-21T09:57:17.185 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T09:57:17.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:17 smithi160 ceph-mon[33203]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:17.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:17 smithi160 ceph-mon[33203]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:17.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:17 smithi160 ceph-mon[33203]: osdmap e139: 8 total, 8 up, 8 in 2024-09-21T09:57:17.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:17 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/1508982226' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:57:17.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:17 smithi070 ceph-mon[41533]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:17.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:17 smithi070 ceph-mon[41533]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:17.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:17 smithi070 ceph-mon[41533]: osdmap e139: 8 total, 8 up, 8 in 2024-09-21T09:57:17.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:17 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/1508982226' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:57:17.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:17 smithi070 ceph-mon[29589]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:17.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:17 smithi070 ceph-mon[29589]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:17.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:17 smithi070 ceph-mon[29589]: osdmap e139: 8 total, 8 up, 8 in 2024-09-21T09:57:17.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:17 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/1508982226' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:57:17.725 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:57:17.901 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:57:18.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:18 smithi160 ceph-mon[33203]: from='client.24863 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:18.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:18 smithi160 ceph-mon[33203]: from='client.14973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:18.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:18 smithi160 ceph-mon[33203]: pgmap v196: 137 pgs: 137 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 12 MiB/s wr, 196 op/s 2024-09-21T09:57:18.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:18 smithi160 ceph-mon[33203]: osdmap e140: 8 total, 8 up, 8 in 2024-09-21T09:57:18.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:18 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:18.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:18 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:57:18.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:18 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:57:18.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:18 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:18.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:18 smithi070 ceph-mon[41533]: from='client.24863 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:18.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:18 smithi070 ceph-mon[41533]: from='client.14973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:18.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:18 smithi070 ceph-mon[41533]: pgmap v196: 137 pgs: 137 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 12 MiB/s wr, 196 op/s 2024-09-21T09:57:18.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:18 smithi070 ceph-mon[41533]: osdmap e140: 8 total, 8 up, 8 in 2024-09-21T09:57:18.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:18 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:18.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:18 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:57:18.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:18 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:57:18.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:18 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:18.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:18 smithi070 ceph-mon[29589]: from='client.24863 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:18.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:18 smithi070 ceph-mon[29589]: from='client.14973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:18.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:18 smithi070 ceph-mon[29589]: pgmap v196: 137 pgs: 137 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 12 MiB/s wr, 196 op/s 2024-09-21T09:57:18.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:18 smithi070 ceph-mon[29589]: osdmap e140: 8 total, 8 up, 8 in 2024-09-21T09:57:18.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:18 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:18.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:18 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:57:18.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:18 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:57:18.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:18 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:19.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:19 smithi070 ceph-mon[29589]: Upgrade: Updating mgr.x 2024-09-21T09:57:19.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:19 smithi070 ceph-mon[29589]: Deploying daemon mgr.x on smithi160 2024-09-21T09:57:19.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:19 smithi070 ceph-mon[29589]: osdmap e141: 8 total, 8 up, 8 in 2024-09-21T09:57:19.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:19 smithi070 ceph-mon[41533]: Upgrade: Updating mgr.x 2024-09-21T09:57:19.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:19 smithi070 ceph-mon[41533]: Deploying daemon mgr.x on smithi160 2024-09-21T09:57:19.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:19 smithi070 ceph-mon[41533]: osdmap e141: 8 total, 8 up, 8 in 2024-09-21T09:57:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:19 smithi160 ceph-mon[33203]: Upgrade: Updating mgr.x 2024-09-21T09:57:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:19 smithi160 ceph-mon[33203]: Deploying daemon mgr.x on smithi160 2024-09-21T09:57:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:19 smithi160 ceph-mon[33203]: osdmap e141: 8 total, 8 up, 8 in 2024-09-21T09:57:20.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:20 smithi070 ceph-mon[41533]: pgmap v199: 137 pgs: 137 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 12 MiB/s wr, 196 op/s 2024-09-21T09:57:20.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:20 smithi070 ceph-mon[41533]: osdmap e142: 8 total, 8 up, 8 in 2024-09-21T09:57:20.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:20 smithi070 ceph-mon[29589]: pgmap v199: 137 pgs: 137 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 12 MiB/s wr, 196 op/s 2024-09-21T09:57:20.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:20 smithi070 ceph-mon[29589]: osdmap e142: 8 total, 8 up, 8 in 2024-09-21T09:57:20.703 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.SnapsOtherPool (12027 ms) 2024-09-21T09:57:20.703 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.SnapsDataPool 2024-09-21T09:57:20.740 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:57:20.750 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 0~8388609 2024-09-21T09:57:20.817 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap1 2024-09-21T09:57:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:20 smithi160 ceph-mon[33203]: pgmap v199: 137 pgs: 137 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 12 MiB/s wr, 196 op/s 2024-09-21T09:57:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:20 smithi160 ceph-mon[33203]: osdmap e142: 8 total, 8 up, 8 in 2024-09-21T09:57:21.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:21 smithi070 ceph-mon[29589]: osdmap e143: 8 total, 8 up, 8 in 2024-09-21T09:57:21.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:21 smithi070 ceph-mon[41533]: osdmap e143: 8 total, 8 up, 8 in 2024-09-21T09:57:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:21 smithi160 ceph-mon[33203]: osdmap e143: 8 total, 8 up, 8 in 2024-09-21T09:57:22.386 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 8388609~8388609 2024-09-21T09:57:22.405 INFO:tasks.workunit.client.0.smithi070.stdout:write: X 0~1000 2024-09-21T09:57:22.405 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 1010~1000 2024-09-21T09:57:22.405 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap2 2024-09-21T09:57:22.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:22 smithi070 ceph-mon[41533]: pgmap v202: 137 pgs: 137 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-21T09:57:22.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:22 smithi070 ceph-mon[41533]: osdmap e144: 8 total, 8 up, 8 in 2024-09-21T09:57:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:22 smithi070 ceph-mon[29589]: pgmap v202: 137 pgs: 137 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-21T09:57:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:22 smithi070 ceph-mon[29589]: osdmap e144: 8 total, 8 up, 8 in 2024-09-21T09:57:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:22 smithi160 ceph-mon[33203]: pgmap v202: 137 pgs: 137 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-21T09:57:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:22 smithi160 ceph-mon[33203]: osdmap e144: 8 total, 8 up, 8 in 2024-09-21T09:57:23.718 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:23 smithi160 ceph-mon[33203]: osdmap e145: 8 total, 8 up, 8 in 2024-09-21T09:57:23.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:23 smithi070 ceph-mon[29589]: osdmap e145: 8 total, 8 up, 8 in 2024-09-21T09:57:23.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:23 smithi070 ceph-mon[41533]: osdmap e145: 8 total, 8 up, 8 in 2024-09-21T09:57:24.246 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:23 smithi160 systemd[1]: Stopping Ceph mgr.x for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:57:24.331 INFO:tasks.workunit.client.0.smithi070.stdout:write: X 8388609~1000 2024-09-21T09:57:24.336 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 16777228~1000 2024-09-21T09:57:24.336 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 167772160 2024-09-21T09:57:24.346 INFO:tasks.workunit.client.0.smithi070.stdout:write: 2 83886079~8388609 2024-09-21T09:57:24.369 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap3 2024-09-21T09:57:24.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:24 smithi160 ceph-mon[33203]: pgmap v205: 137 pgs: 137 active+clean; 353 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 82 MiB/s rd, 14 MiB/s wr, 464 op/s 2024-09-21T09:57:24.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:24 smithi160 ceph-mon[33203]: osdmap e146: 8 total, 8 up, 8 in 2024-09-21T09:57:24.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:24 smithi160 ceph-mon[33203]: osdmap e147: 8 total, 8 up, 8 in 2024-09-21T09:57:24.558 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:24 smithi160 podman[69080]: 2024-09-21 09:57:24.244135863 +0000 UTC m=+0.122615253 container died 8f2bf49a5ce6eab267c2e874beaee8622430c1bd6e1126b0cbf829c403b2a3cf (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x, RELEASE=reef-6a1bdf8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T09:57:24.841 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:24 smithi160 podman[69080]: 2024-09-21 09:57:24.704751441 +0000 UTC m=+0.583230820 container cleanup 8f2bf49a5ce6eab267c2e874beaee8622430c1bd6e1126b0cbf829c403b2a3cf (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-6a1bdf8, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-21T09:57:24.841 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:24 smithi160 bash[69080]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x 2024-09-21T09:57:24.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:24 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:57:24] "GET /metrics HTTP/1.1" 200 38643 "" "Prometheus/2.43.0" 2024-09-21T09:57:24.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:24 smithi070 ceph-mon[41533]: pgmap v205: 137 pgs: 137 active+clean; 353 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 82 MiB/s rd, 14 MiB/s wr, 464 op/s 2024-09-21T09:57:24.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:24 smithi070 ceph-mon[41533]: osdmap e146: 8 total, 8 up, 8 in 2024-09-21T09:57:24.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:24 smithi070 ceph-mon[41533]: osdmap e147: 8 total, 8 up, 8 in 2024-09-21T09:57:24.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:24 smithi070 ceph-mon[29589]: pgmap v205: 137 pgs: 137 active+clean; 353 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 82 MiB/s rd, 14 MiB/s wr, 464 op/s 2024-09-21T09:57:24.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:24 smithi070 ceph-mon[29589]: osdmap e146: 8 total, 8 up, 8 in 2024-09-21T09:57:24.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:24 smithi070 ceph-mon[29589]: osdmap e147: 8 total, 8 up, 8 in 2024-09-21T09:57:25.330 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:24 smithi160 podman[69097]: 2024-09-21 09:57:24.973015347 +0000 UTC m=+0.717021553 container remove 8f2bf49a5ce6eab267c2e874beaee8622430c1bd6e1126b0cbf829c403b2a3cf (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=reef-6a1bdf8, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T09:57:25.331 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:24 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-21T09:57:25.331 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:25 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mgr.x.service: Failed with result 'exit-code'. 2024-09-21T09:57:25.331 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:25 smithi160 systemd[1]: Stopped Ceph mgr.x for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:57:25.331 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:25 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mgr.x.service: Consumed 10.811s CPU time. 2024-09-21T09:57:25.331 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:25 smithi160 systemd[1]: Starting Ceph mgr.x for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:57:25.591 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:25 smithi160 podman[69231]: 2024-09-21 09:57:25.441119554 +0000 UTC m=+0.016109221 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:57:25.591 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:25 smithi160 podman[69231]: 2024-09-21 09:57:25.574143985 +0000 UTC m=+0.149133650 container create 0f7e42b65de50fb5b62087bb81eb6927b9714c842ecd49f74fe4bcd7f01517bc (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True) 2024-09-21T09:57:26.341 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:25 smithi160 podman[69231]: 2024-09-21 09:57:25.868499143 +0000 UTC m=+0.443488806 container init 0f7e42b65de50fb5b62087bb81eb6927b9714c842ecd49f74fe4bcd7f01517bc (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x, RELEASE=main-62da54b, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T09:57:26.341 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:25 smithi160 podman[69231]: 2024-09-21 09:57:25.873344324 +0000 UTC m=+0.448333983 container start 0f7e42b65de50fb5b62087bb81eb6927b9714c842ecd49f74fe4bcd7f01517bc (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-21T09:57:26.341 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:25 smithi160 ceph-mgr[69268]: -- 172.21.15.160:0/2973217503 <== mon.1 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e2cba130e0 con 0x55e2cb9f0800 2024-09-21T09:57:26.341 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:25 smithi160 bash[69231]: 0f7e42b65de50fb5b62087bb81eb6927b9714c842ecd49f74fe4bcd7f01517bc 2024-09-21T09:57:26.341 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:25 smithi160 systemd[1]: Started Ceph mgr.x for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:57:26.398 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:57:26.417 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 83886079~1 2024-09-21T09:57:26.426 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:57:26.763 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:26 smithi160 ceph-mon[33203]: pgmap v208: 137 pgs: 137 active+clean; 353 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 82 MiB/s rd, 14 MiB/s wr, 464 op/s 2024-09-21T09:57:26.763 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:26 smithi160 ceph-mon[33203]: osdmap e148: 8 total, 8 up, 8 in 2024-09-21T09:57:26.763 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:26 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:26.763 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:26 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:26.764 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:26 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:57:26.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:26 smithi070 ceph-mon[41533]: pgmap v208: 137 pgs: 137 active+clean; 353 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 82 MiB/s rd, 14 MiB/s wr, 464 op/s 2024-09-21T09:57:26.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:26 smithi070 ceph-mon[41533]: osdmap e148: 8 total, 8 up, 8 in 2024-09-21T09:57:26.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:26 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:26.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:26 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:26.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:26 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:57:26.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:26 smithi070 ceph-mon[29589]: pgmap v208: 137 pgs: 137 active+clean; 353 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 82 MiB/s rd, 14 MiB/s wr, 464 op/s 2024-09-21T09:57:26.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:26 smithi070 ceph-mon[29589]: osdmap e148: 8 total, 8 up, 8 in 2024-09-21T09:57:26.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:26 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:26.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:26 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:26.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:26 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:57:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:27 smithi160 ceph-mon[33203]: osdmap e149: 8 total, 8 up, 8 in 2024-09-21T09:57:27.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:27 smithi070 ceph-mon[41533]: osdmap e149: 8 total, 8 up, 8 in 2024-09-21T09:57:27.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:27 smithi070 ceph-mon[29589]: osdmap e149: 8 total, 8 up, 8 in 2024-09-21T09:57:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:28 smithi160 ceph-mon[33203]: pgmap v211: 137 pgs: 137 active+clean; 393 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 140 KiB/s rd, 8.0 MiB/s wr, 201 op/s 2024-09-21T09:57:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:28 smithi160 ceph-mon[33203]: osdmap e150: 8 total, 8 up, 8 in 2024-09-21T09:57:28.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:28 smithi070 ceph-mon[29589]: pgmap v211: 137 pgs: 137 active+clean; 393 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 140 KiB/s rd, 8.0 MiB/s wr, 201 op/s 2024-09-21T09:57:28.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:28 smithi070 ceph-mon[29589]: osdmap e150: 8 total, 8 up, 8 in 2024-09-21T09:57:28.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:28 smithi070 ceph-mon[41533]: pgmap v211: 137 pgs: 137 active+clean; 393 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 140 KiB/s rd, 8.0 MiB/s wr, 201 op/s 2024-09-21T09:57:28.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:28 smithi070 ceph-mon[41533]: osdmap e150: 8 total, 8 up, 8 in 2024-09-21T09:57:29.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:29 smithi160 ceph-mon[33203]: osdmap e151: 8 total, 8 up, 8 in 2024-09-21T09:57:29.823 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:57:29.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:29 smithi070 ceph-mon[29589]: osdmap e151: 8 total, 8 up, 8 in 2024-09-21T09:57:29.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:29 smithi070 ceph-mon[41533]: osdmap e151: 8 total, 8 up, 8 in 2024-09-21T09:57:29.983 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:57:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:30 smithi160 ceph-mon[33203]: pgmap v214: 137 pgs: 137 active+clean; 393 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 140 KiB/s rd, 8.0 MiB/s wr, 201 op/s 2024-09-21T09:57:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:30 smithi160 ceph-mon[33203]: osdmap e152: 8 total, 8 up, 8 in 2024-09-21T09:57:30.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:30 smithi070 ceph-mon[29589]: pgmap v214: 137 pgs: 137 active+clean; 393 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 140 KiB/s rd, 8.0 MiB/s wr, 201 op/s 2024-09-21T09:57:30.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:30 smithi070 ceph-mon[29589]: osdmap e152: 8 total, 8 up, 8 in 2024-09-21T09:57:30.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:30 smithi070 ceph-mon[41533]: pgmap v214: 137 pgs: 137 active+clean; 393 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 140 KiB/s rd, 8.0 MiB/s wr, 201 op/s 2024-09-21T09:57:30.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:30 smithi070 ceph-mon[41533]: osdmap e152: 8 total, 8 up, 8 in 2024-09-21T09:57:31.536 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:31 smithi160 ceph-mon[33203]: osdmap e153: 8 total, 8 up, 8 in 2024-09-21T09:57:31.536 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:31 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:31.536 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:31 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:31.536 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:31 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:31.536 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:31 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:31.536 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:31 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:31.536 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:31 smithi160 ceph-mon[33203]: Standby manager daemon x restarted 2024-09-21T09:57:31.536 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:31 smithi160 ceph-mon[33203]: Standby manager daemon x started 2024-09-21T09:57:31.537 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:31 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:57:31.537 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:31 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:57:31.537 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:31 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:57:31.537 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:31 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:57:31.537 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:31 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: [21/Sep/2024:09:57:31] ENGINE Bus STARTING 2024-09-21T09:57:31.537 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:31 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: CherryPy Checker: 2024-09-21T09:57:31.537 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:31 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: The Application mounted at '' has an empty config. 2024-09-21T09:57:31.537 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:31 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: 2024-09-21T09:57:31.537 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:31 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: [21/Sep/2024:09:57:31] ENGINE Serving on http://:::9283 2024-09-21T09:57:31.537 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:31 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: [21/Sep/2024:09:57:31] ENGINE Bus STARTED 2024-09-21T09:57:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:31 smithi070 ceph-mon[29589]: osdmap e153: 8 total, 8 up, 8 in 2024-09-21T09:57:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:31 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:31 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:31.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:31 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:31.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:31 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:31.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:31 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:31.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:31 smithi070 ceph-mon[29589]: Standby manager daemon x restarted 2024-09-21T09:57:31.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:31 smithi070 ceph-mon[29589]: Standby manager daemon x started 2024-09-21T09:57:31.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:31 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:57:31.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:31 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:57:31.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:31 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:57:31.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:31 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:57:31.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:31 smithi070 ceph-mon[41533]: osdmap e153: 8 total, 8 up, 8 in 2024-09-21T09:57:31.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:31 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:31.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:31 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:31.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:31 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:31.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:31 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:31.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:31 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:31.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:31 smithi070 ceph-mon[41533]: Standby manager daemon x restarted 2024-09-21T09:57:31.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:31 smithi070 ceph-mon[41533]: Standby manager daemon x started 2024-09-21T09:57:31.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:31 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:57:31.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:31 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:57:31.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:31 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:57:31.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:31 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:57:32.698 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:32 smithi160 ceph-mon[33203]: pgmap v217: 137 pgs: 137 active+clean; 393 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T09:57:32.699 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:32 smithi160 ceph-mon[33203]: osdmap e154: 8 total, 8 up, 8 in 2024-09-21T09:57:32.699 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:32 smithi160 ceph-mon[33203]: mgrmap e21: y(active, since 4m), standbys: x 2024-09-21T09:57:32.748 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.SnapsDataPool (12045 ms) 2024-09-21T09:57:32.748 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.SnapsShrinkAfterPrepare 2024-09-21T09:57:32.782 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:57:32.792 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 0~8388609 2024-09-21T09:57:32.851 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap1 2024-09-21T09:57:32.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:32 smithi070 ceph-mon[29589]: pgmap v217: 137 pgs: 137 active+clean; 393 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T09:57:32.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:32 smithi070 ceph-mon[29589]: osdmap e154: 8 total, 8 up, 8 in 2024-09-21T09:57:32.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:32 smithi070 ceph-mon[29589]: mgrmap e21: y(active, since 4m), standbys: x 2024-09-21T09:57:32.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:32 smithi070 ceph-mon[41533]: pgmap v217: 137 pgs: 137 active+clean; 393 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T09:57:32.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:32 smithi070 ceph-mon[41533]: osdmap e154: 8 total, 8 up, 8 in 2024-09-21T09:57:32.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:32 smithi070 ceph-mon[41533]: mgrmap e21: y(active, since 4m), standbys: x 2024-09-21T09:57:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:33 smithi160 ceph-mon[33203]: osdmap e155: 8 total, 8 up, 8 in 2024-09-21T09:57:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:33 smithi070 ceph-mon[41533]: osdmap e155: 8 total, 8 up, 8 in 2024-09-21T09:57:33.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:33 smithi070 ceph-mon[29589]: osdmap e155: 8 total, 8 up, 8 in 2024-09-21T09:57:34.331 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 8388609~8388609 2024-09-21T09:57:34.352 INFO:tasks.workunit.client.0.smithi070.stdout:write: X 0~1000 2024-09-21T09:57:34.352 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 1010~1000 2024-09-21T09:57:34.352 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap2 2024-09-21T09:57:34.591 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:34 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: [21/Sep/2024:09:57:34] ENGINE Bus STOPPING 2024-09-21T09:57:34.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:34 smithi160 ceph-mon[33203]: pgmap v220: 137 pgs: 137 active+clean; 413 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 414 op/s 2024-09-21T09:57:34.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:34 smithi160 ceph-mon[33203]: osdmap e156: 8 total, 8 up, 8 in 2024-09-21T09:57:34.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:34 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:34.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:34 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:34.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:34 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:34.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:34 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:57:34.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:34 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:34.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:34 smithi160 ceph-mon[33203]: osdmap e157: 8 total, 8 up, 8 in 2024-09-21T09:57:34.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:34 smithi070 ceph-mon[29589]: pgmap v220: 137 pgs: 137 active+clean; 413 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 414 op/s 2024-09-21T09:57:34.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:34 smithi070 ceph-mon[29589]: osdmap e156: 8 total, 8 up, 8 in 2024-09-21T09:57:34.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:34 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:34.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:34 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:34.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:34 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:34.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:34 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:57:34.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:34 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:34.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:34 smithi070 ceph-mon[29589]: osdmap e157: 8 total, 8 up, 8 in 2024-09-21T09:57:34.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:34 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:57:34.449+0000 7fb7497c5640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-21T09:57:34.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:34 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ignoring --setuser ceph since I am not root 2024-09-21T09:57:34.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:34 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ignoring --setgroup ceph since I am not root 2024-09-21T09:57:34.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:34 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:57:34.589+0000 7f2be7e21040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T09:57:34.794 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:34 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:57:34.644+0000 7f2be7e21040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T09:57:34.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:34 smithi070 ceph-mon[41533]: pgmap v220: 137 pgs: 137 active+clean; 413 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 414 op/s 2024-09-21T09:57:34.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:34 smithi070 ceph-mon[41533]: osdmap e156: 8 total, 8 up, 8 in 2024-09-21T09:57:34.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:34 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:34.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:34 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:34.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:34 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:34.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:34 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:57:34.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:34 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' 2024-09-21T09:57:34.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:34 smithi070 ceph-mon[41533]: osdmap e157: 8 total, 8 up, 8 in 2024-09-21T09:57:35.127 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:34 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:57:34.790+0000 7f2be7e21040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T09:57:35.127 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:34 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:57:34.856+0000 7f2be7e21040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:57:35.127 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:57:35.001+0000 7f2be7e21040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:57:35.341 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:34 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: [21/Sep/2024:09:57:34] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:57:35.341 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:34 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: [21/Sep/2024:09:57:34] ENGINE Bus STOPPED 2024-09-21T09:57:35.430 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:57:35.139+0000 7f2be7e21040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:57:35.464 INFO:tasks.workunit.client.0.smithi070.stdout:write: X 8388609~1000 2024-09-21T09:57:35.470 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 16777228~1000 2024-09-21T09:57:35.470 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 167772160 2024-09-21T09:57:35.477 INFO:tasks.workunit.client.0.smithi070.stdout:write: 2 83886079~8388609 2024-09-21T09:57:35.499 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap3 2024-09-21T09:57:35.710 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:57:35.445+0000 7f2be7e21040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:57:35.710 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:57:35.526+0000 7f2be7e21040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:57:35.710 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:57:35.711 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-21T09:57:35.711 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[29589]: Activating manager daemon x 2024-09-21T09:57:35.711 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[29589]: osdmap e158: 8 total, 8 up, 8 in 2024-09-21T09:57:35.711 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:57:35.711 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:57:35.711 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:57:35.711 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi070.gansda"}]: dispatch 2024-09-21T09:57:35.711 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[29589]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-21T09:57:35.711 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[29589]: mgrmap e22: x(active, starting, since 0.00858812s) 2024-09-21T09:57:35.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi160.byyhfz"}]: dispatch 2024-09-21T09:57:35.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:57:35.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:57:35.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:57:35.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:57:35.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:57:35.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:57:35.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:57:35.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:57:35.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:57:35.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:57:35.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:57:35.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:57:35.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[29589]: Manager daemon x is now available 2024-09-21T09:57:35.713 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:57:35.713 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-21T09:57:35.713 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[41533]: Activating manager daemon x 2024-09-21T09:57:35.713 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[41533]: osdmap e158: 8 total, 8 up, 8 in 2024-09-21T09:57:35.713 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:57:35.713 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:57:35.713 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:57:35.713 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi070.gansda"}]: dispatch 2024-09-21T09:57:35.713 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[41533]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-21T09:57:35.713 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[41533]: mgrmap e22: x(active, starting, since 0.00858812s) 2024-09-21T09:57:35.713 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi160.byyhfz"}]: dispatch 2024-09-21T09:57:35.713 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:57:35.714 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:57:35.714 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:57:35.714 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:57:35.714 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:57:35.714 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:57:35.714 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:57:35.714 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:57:35.714 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:57:35.714 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:57:35.714 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:57:35.714 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:57:35.714 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-mon[41533]: Manager daemon x is now available 2024-09-21T09:57:35.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:35 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:57:35.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:35 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-21T09:57:35.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:35 smithi160 ceph-mon[33203]: Activating manager daemon x 2024-09-21T09:57:35.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:35 smithi160 ceph-mon[33203]: osdmap e158: 8 total, 8 up, 8 in 2024-09-21T09:57:35.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:35 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:57:35.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:35 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:57:35.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:35 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:57:35.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:35 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi070.gansda"}]: dispatch 2024-09-21T09:57:35.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:35 smithi160 ceph-mon[33203]: from='mgr.24341 172.21.15.70:0/3233530482' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-21T09:57:35.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:35 smithi160 ceph-mon[33203]: mgrmap e22: x(active, starting, since 0.00858812s) 2024-09-21T09:57:35.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:35 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi160.byyhfz"}]: dispatch 2024-09-21T09:57:35.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:35 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:57:35.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:35 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:57:35.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:35 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:57:35.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:35 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:57:35.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:35 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:57:35.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:35 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:57:35.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:35 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:57:35.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:35 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:57:35.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:35 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:57:35.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:35 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:57:35.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:35 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:57:35.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:35 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:57:35.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:35 smithi160 ceph-mon[33203]: Manager daemon x is now available 2024-09-21T09:57:36.127 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:57:35.708+0000 7f2be7e21040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:57:36.239 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:35 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: [21/Sep/2024:09:57:35] ENGINE Bus STARTING 2024-09-21T09:57:36.239 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:36 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: 2024-09-21T09:57:36.007+0000 7fb1c53fb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:57:36.239 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:36 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: 2024-09-21T09:57:36.007+0000 7fb1c53fb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:57:36.239 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:36 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: 2024-09-21T09:57:36.007+0000 7fb1c53fb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:57:36.239 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:36 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: 2024-09-21T09:57:36.007+0000 7fb1c53fb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:57:36.239 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:36 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: 2024-09-21T09:57:36.007+0000 7fb1c53fb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:57:36.239 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:36 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: 2024-09-21T09:57:36.007+0000 7fb1c73ff640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:57:36.239 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:36 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: 2024-09-21T09:57:36.007+0000 7fb1c73ff640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:57:36.239 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:36 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: 2024-09-21T09:57:36.007+0000 7fb1c73ff640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:57:36.239 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:36 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: 2024-09-21T09:57:36.007+0000 7fb1c73ff640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:57:36.240 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:36 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: 2024-09-21T09:57:36.007+0000 7fb1c73ff640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:57:36.240 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:36 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: [21/Sep/2024:09:57:36] ENGINE Serving on http://:::9283 2024-09-21T09:57:36.240 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:36 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: [21/Sep/2024:09:57:36] ENGINE Bus STARTED 2024-09-21T09:57:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:36 smithi160 ceph-mon[33203]: osdmap e159: 8 total, 8 up, 8 in 2024-09-21T09:57:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:36 smithi160 ceph-mon[33203]: mgrmap e23: x(active, since 1.01269s) 2024-09-21T09:57:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:36 smithi160 ceph-mon[33203]: pgmap v4: 137 pgs: 137 active+clean; 425 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T09:57:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:36 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:36.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:36 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:36.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:36 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:36.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:36 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:36.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:36 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:36.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:36 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:57:36.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:36 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:36.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:36 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:57:36.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:36 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:57:36.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:36 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:57:36.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:36 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T09:57:36.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:36 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T09:57:36.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:57:36.595+0000 7f2be7e21040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:57:36.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:57:36.661+0000 7f2be7e21040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:57:36.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[29589]: osdmap e159: 8 total, 8 up, 8 in 2024-09-21T09:57:36.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[29589]: mgrmap e23: x(active, since 1.01269s) 2024-09-21T09:57:36.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[29589]: pgmap v4: 137 pgs: 137 active+clean; 425 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T09:57:36.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:36.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:36.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:36.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:36.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:36.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:57:36.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:36.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:57:36.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:57:36.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:57:36.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T09:57:36.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T09:57:36.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[41533]: osdmap e159: 8 total, 8 up, 8 in 2024-09-21T09:57:36.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[41533]: mgrmap e23: x(active, since 1.01269s) 2024-09-21T09:57:36.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[41533]: pgmap v4: 137 pgs: 137 active+clean; 425 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T09:57:36.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:36.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:36.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:36.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:36.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:36.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:57:36.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:36.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:57:36.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:57:36.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:57:36.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T09:57:36.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:36 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T09:57:37.475 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:57:37.500 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 83886079~1 2024-09-21T09:57:37.509 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:57:37.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:37 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:57:37.372+0000 7f2be7e21040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:57:37.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:37 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:57:37.496+0000 7f2be7e21040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:57:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:37 smithi070 ceph-mon[29589]: Checking for cert/key for grafana.a 2024-09-21T09:57:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:37 smithi070 ceph-mon[29589]: Migrating grafana.a cert to cert store 2024-09-21T09:57:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:37 smithi070 ceph-mon[29589]: Migrating grafana.a key to cert store 2024-09-21T09:57:37.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:37 smithi070 ceph-mon[29589]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:57:37.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:37 smithi070 ceph-mon[29589]: Deploying cephadm binary to smithi160 2024-09-21T09:57:37.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:37 smithi070 ceph-mon[29589]: pgmap v5: 137 pgs: 137 active+clean; 425 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T09:57:37.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:37 smithi070 ceph-mon[29589]: osdmap e160: 8 total, 8 up, 8 in 2024-09-21T09:57:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:37 smithi070 ceph-mon[41533]: Checking for cert/key for grafana.a 2024-09-21T09:57:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:37 smithi070 ceph-mon[41533]: Migrating grafana.a cert to cert store 2024-09-21T09:57:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:37 smithi070 ceph-mon[41533]: Migrating grafana.a key to cert store 2024-09-21T09:57:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:37 smithi070 ceph-mon[41533]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:57:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:37 smithi070 ceph-mon[41533]: Deploying cephadm binary to smithi160 2024-09-21T09:57:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:37 smithi070 ceph-mon[41533]: pgmap v5: 137 pgs: 137 active+clean; 425 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T09:57:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:37 smithi070 ceph-mon[41533]: osdmap e160: 8 total, 8 up, 8 in 2024-09-21T09:57:37.834 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:37 smithi160 ceph-mon[33203]: Checking for cert/key for grafana.a 2024-09-21T09:57:37.834 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:37 smithi160 ceph-mon[33203]: Migrating grafana.a cert to cert store 2024-09-21T09:57:37.834 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:37 smithi160 ceph-mon[33203]: Migrating grafana.a key to cert store 2024-09-21T09:57:37.834 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:37 smithi160 ceph-mon[33203]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:57:37.834 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:37 smithi160 ceph-mon[33203]: Deploying cephadm binary to smithi160 2024-09-21T09:57:37.834 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:37 smithi160 ceph-mon[33203]: pgmap v5: 137 pgs: 137 active+clean; 425 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T09:57:37.834 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:37 smithi160 ceph-mon[33203]: osdmap e160: 8 total, 8 up, 8 in 2024-09-21T09:57:37.834 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: 2024-09-21T09:57:37.469+0000 7fb1ab94b640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:09:57:37] ENGINE Error in HTTPServer.serve 2024-09-21T09:57:37.834 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: Traceback (most recent call last): 2024-09-21T09:57:37.834 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:57:37.835 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: self._connections.run(self.expiration_interval) 2024-09-21T09:57:37.835 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:57:37.835 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: self._run(expiration_interval) 2024-09-21T09:57:37.835 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:57:37.835 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:57:37.835 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:57:37.835 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:57:37.835 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:57:37.835 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: s = self.context.wrap_socket( 2024-09-21T09:57:37.835 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:57:37.835 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: return self.sslsocket_class._create( 2024-09-21T09:57:37.836 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:57:37.836 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: self.do_handshake() 2024-09-21T09:57:37.836 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:57:37.836 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: self._sslobj.do_handshake() 2024-09-21T09:57:37.836 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:57:37.836 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:57:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: 2024-09-21T09:57:37.925 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:37 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:57:37.647+0000 7f2be7e21040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:57:37.925 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:37 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:57:37.708+0000 7f2be7e21040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:57:37.925 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:37 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:57:37.856+0000 7f2be7e21040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:57:37.925 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:37 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:57:37.923+0000 7f2be7e21040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:57:38.213 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:37 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:57:37.983+0000 7f2be7e21040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:57:38.213 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:38 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:57:38.143+0000 7f2be7e21040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:57:38.213 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:38 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:57:38.211+0000 7f2be7e21040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:57:38.466 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:38 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:57:38.281+0000 7f2be7e21040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:57:38.467 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:38 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:57:38.379+0000 7f2be7e21040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:57:38.467 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:38 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:57:38.445+0000 7f2be7e21040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:57:38.732 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:38 smithi160 ceph-mon[33203]: [21/Sep/2024:09:57:37] ENGINE Bus STARTING 2024-09-21T09:57:38.732 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:38 smithi160 ceph-mon[33203]: Deploying cephadm binary to smithi070 2024-09-21T09:57:38.732 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:38 smithi160 ceph-mon[33203]: [21/Sep/2024:09:57:37] ENGINE Serving on http://172.21.15.160:8765 2024-09-21T09:57:38.732 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:38 smithi160 ceph-mon[33203]: [21/Sep/2024:09:57:37] ENGINE Serving on https://172.21.15.160:7150 2024-09-21T09:57:38.732 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:38 smithi160 ceph-mon[33203]: [21/Sep/2024:09:57:37] ENGINE Bus STARTED 2024-09-21T09:57:38.732 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:38 smithi160 ceph-mon[33203]: osdmap e161: 8 total, 8 up, 8 in 2024-09-21T09:57:38.732 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:38 smithi160 ceph-mon[33203]: mgrmap e24: x(active, since 3s) 2024-09-21T09:57:38.732 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:38 smithi160 ceph-mon[33203]: [21/Sep/2024:09:57:37] ENGINE Error in HTTPServer.serve 2024-09-21T09:57:38.732 INFO:journalctl@ceph.mon.b.smithi160.stdout: Traceback (most recent call last): 2024-09-21T09:57:38.732 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:57:38.732 INFO:journalctl@ceph.mon.b.smithi160.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:57:38.733 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:57:38.733 INFO:journalctl@ceph.mon.b.smithi160.stdout: self._run(expiration_interval) 2024-09-21T09:57:38.733 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:57:38.733 INFO:journalctl@ceph.mon.b.smithi160.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:57:38.733 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:57:38.733 INFO:journalctl@ceph.mon.b.smithi160.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:57:38.733 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:57:38.733 INFO:journalctl@ceph.mon.b.smithi160.stdout: s = self.context.wrap_socket( 2024-09-21T09:57:38.733 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:57:38.733 INFO:journalctl@ceph.mon.b.smithi160.stdout: return self.sslsocket_class._create( 2024-09-21T09:57:38.733 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:57:38.733 INFO:journalctl@ceph.mon.b.smithi160.stdout: self.do_handshake() 2024-09-21T09:57:38.734 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:57:38.734 INFO:journalctl@ceph.mon.b.smithi160.stdout: self._sslobj.do_handshake() 2024-09-21T09:57:38.734 INFO:journalctl@ceph.mon.b.smithi160.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:57:38.813 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:38 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:57:38.508+0000 7f2be7e21040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:57:38.813 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:38 smithi070 ceph-mon[29589]: [21/Sep/2024:09:57:37] ENGINE Bus STARTING 2024-09-21T09:57:38.813 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:38 smithi070 ceph-mon[29589]: Deploying cephadm binary to smithi070 2024-09-21T09:57:38.813 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:38 smithi070 ceph-mon[29589]: [21/Sep/2024:09:57:37] ENGINE Serving on http://172.21.15.160:8765 2024-09-21T09:57:38.813 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:38 smithi070 ceph-mon[29589]: [21/Sep/2024:09:57:37] ENGINE Serving on https://172.21.15.160:7150 2024-09-21T09:57:38.813 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:38 smithi070 ceph-mon[29589]: [21/Sep/2024:09:57:37] ENGINE Bus STARTED 2024-09-21T09:57:38.813 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:38 smithi070 ceph-mon[29589]: osdmap e161: 8 total, 8 up, 8 in 2024-09-21T09:57:38.813 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:38 smithi070 ceph-mon[29589]: mgrmap e24: x(active, since 3s) 2024-09-21T09:57:38.813 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:38 smithi070 ceph-mon[29589]: [21/Sep/2024:09:57:37] ENGINE Error in HTTPServer.serve 2024-09-21T09:57:38.814 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-09-21T09:57:38.814 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:57:38.814 INFO:journalctl@ceph.mon.a.smithi070.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:57:38.814 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:57:38.814 INFO:journalctl@ceph.mon.a.smithi070.stdout: self._run(expiration_interval) 2024-09-21T09:57:38.814 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:57:38.814 INFO:journalctl@ceph.mon.a.smithi070.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:57:38.814 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:57:38.814 INFO:journalctl@ceph.mon.a.smithi070.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:57:38.814 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:57:38.814 INFO:journalctl@ceph.mon.a.smithi070.stdout: s = self.context.wrap_socket( 2024-09-21T09:57:38.814 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:57:38.814 INFO:journalctl@ceph.mon.a.smithi070.stdout: return self.sslsocket_class._create( 2024-09-21T09:57:38.814 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:57:38.815 INFO:journalctl@ceph.mon.a.smithi070.stdout: self.do_handshake() 2024-09-21T09:57:38.815 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:57:38.815 INFO:journalctl@ceph.mon.a.smithi070.stdout: self._sslobj.do_handshake() 2024-09-21T09:57:38.815 INFO:journalctl@ceph.mon.a.smithi070.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:57:38.815 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:38 smithi070 ceph-mon[41533]: [21/Sep/2024:09:57:37] ENGINE Bus STARTING 2024-09-21T09:57:38.815 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:38 smithi070 ceph-mon[41533]: Deploying cephadm binary to smithi070 2024-09-21T09:57:38.815 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:38 smithi070 ceph-mon[41533]: [21/Sep/2024:09:57:37] ENGINE Serving on http://172.21.15.160:8765 2024-09-21T09:57:38.815 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:38 smithi070 ceph-mon[41533]: [21/Sep/2024:09:57:37] ENGINE Serving on https://172.21.15.160:7150 2024-09-21T09:57:38.815 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:38 smithi070 ceph-mon[41533]: [21/Sep/2024:09:57:37] ENGINE Bus STARTED 2024-09-21T09:57:38.815 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:38 smithi070 ceph-mon[41533]: osdmap e161: 8 total, 8 up, 8 in 2024-09-21T09:57:38.815 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:38 smithi070 ceph-mon[41533]: mgrmap e24: x(active, since 3s) 2024-09-21T09:57:38.815 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:38 smithi070 ceph-mon[41533]: [21/Sep/2024:09:57:37] ENGINE Error in HTTPServer.serve 2024-09-21T09:57:38.816 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-09-21T09:57:38.816 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:57:38.816 INFO:journalctl@ceph.mon.c.smithi070.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:57:38.816 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:57:38.816 INFO:journalctl@ceph.mon.c.smithi070.stdout: self._run(expiration_interval) 2024-09-21T09:57:38.816 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:57:38.816 INFO:journalctl@ceph.mon.c.smithi070.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:57:38.816 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:57:38.816 INFO:journalctl@ceph.mon.c.smithi070.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:57:38.816 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:57:38.816 INFO:journalctl@ceph.mon.c.smithi070.stdout: s = self.context.wrap_socket( 2024-09-21T09:57:38.816 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:57:38.816 INFO:journalctl@ceph.mon.c.smithi070.stdout: return self.sslsocket_class._create( 2024-09-21T09:57:38.816 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:57:38.817 INFO:journalctl@ceph.mon.c.smithi070.stdout: self.do_handshake() 2024-09-21T09:57:38.817 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:57:38.817 INFO:journalctl@ceph.mon.c.smithi070.stdout: self._sslobj.do_handshake() 2024-09-21T09:57:38.817 INFO:journalctl@ceph.mon.c.smithi070.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:57:39.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:39 smithi070 ceph-mon[41533]: pgmap v8: 137 pgs: 137 active+clean; 425 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T09:57:39.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:39 smithi070 ceph-mon[41533]: osdmap e162: 8 total, 8 up, 8 in 2024-09-21T09:57:39.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:39 smithi070 ceph-mon[41533]: osdmap e163: 8 total, 8 up, 8 in 2024-09-21T09:57:39.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:39 smithi070 ceph-mon[41533]: Standby manager daemon y started 2024-09-21T09:57:39.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:39 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.70:0/654498851' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T09:57:39.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:39 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.70:0/654498851' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:57:39.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:39 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.70:0/654498851' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T09:57:39.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:39 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.70:0/654498851' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:57:39.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:39 smithi070 ceph-mon[29589]: pgmap v8: 137 pgs: 137 active+clean; 425 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T09:57:39.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:39 smithi070 ceph-mon[29589]: osdmap e162: 8 total, 8 up, 8 in 2024-09-21T09:57:39.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:39 smithi070 ceph-mon[29589]: osdmap e163: 8 total, 8 up, 8 in 2024-09-21T09:57:39.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:39 smithi070 ceph-mon[29589]: Standby manager daemon y started 2024-09-21T09:57:39.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:39 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.70:0/654498851' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T09:57:39.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:39 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.70:0/654498851' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:57:39.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:39 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.70:0/654498851' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T09:57:39.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:39 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.70:0/654498851' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:57:39.629 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:39 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:57:39.329+0000 7f2be7e21040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:57:39.629 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:39 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:57:39] ENGINE Bus STARTING 2024-09-21T09:57:39.629 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:39 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: CherryPy Checker: 2024-09-21T09:57:39.629 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:39 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: The Application mounted at '' has an empty config. 2024-09-21T09:57:39.629 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:39 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: 2024-09-21T09:57:39.629 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:39 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:57:39] ENGINE Serving on http://:::9283 2024-09-21T09:57:39.629 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:39 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: [21/Sep/2024:09:57:39] ENGINE Bus STARTED 2024-09-21T09:57:39.734 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:39 smithi160 ceph-mon[33203]: pgmap v8: 137 pgs: 137 active+clean; 425 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T09:57:39.735 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:39 smithi160 ceph-mon[33203]: osdmap e162: 8 total, 8 up, 8 in 2024-09-21T09:57:39.735 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:39 smithi160 ceph-mon[33203]: osdmap e163: 8 total, 8 up, 8 in 2024-09-21T09:57:39.735 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:39 smithi160 ceph-mon[33203]: Standby manager daemon y started 2024-09-21T09:57:39.735 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:39 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.70:0/654498851' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T09:57:39.735 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:39 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.70:0/654498851' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:57:39.736 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:39 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.70:0/654498851' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T09:57:39.736 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:39 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.70:0/654498851' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:57:40.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:40 smithi160 ceph-mon[33203]: mgrmap e25: x(active, since 5s), standbys: y 2024-09-21T09:57:40.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:40 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:57:40.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:40 smithi160 ceph-mon[33203]: osdmap e164: 8 total, 8 up, 8 in 2024-09-21T09:57:40.710 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 41943040 2024-09-21T09:57:40.726 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:57:40.862 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:57:40.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:40 smithi070 ceph-mon[29589]: mgrmap e25: x(active, since 5s), standbys: y 2024-09-21T09:57:40.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:40 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:57:40.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:40 smithi070 ceph-mon[29589]: osdmap e164: 8 total, 8 up, 8 in 2024-09-21T09:57:40.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:40 smithi070 ceph-mon[41533]: mgrmap e25: x(active, since 5s), standbys: y 2024-09-21T09:57:40.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:40 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:57:40.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:40 smithi070 ceph-mon[41533]: osdmap e164: 8 total, 8 up, 8 in 2024-09-21T09:57:41.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:41 smithi160 ceph-mon[33203]: pgmap v12: 137 pgs: 137 active+clean; 425 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T09:57:41.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:41 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:41.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:41 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:41.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:41 smithi070 ceph-mon[29589]: pgmap v12: 137 pgs: 137 active+clean; 425 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T09:57:41.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:41 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:41.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:41 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:41.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:41 smithi070 ceph-mon[41533]: pgmap v12: 137 pgs: 137 active+clean; 425 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T09:57:41.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:41 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:41.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:41 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:42.747 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:42 smithi160 ceph-mon[33203]: osdmap e165: 8 total, 8 up, 8 in 2024-09-21T09:57:42.747 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:42 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:42.747 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:42 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:42.747 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:42 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:42.747 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:42 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:42.747 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:42 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:42.747 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:42 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:42.747 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:42 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:42.748 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:42 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:42.748 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:42 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:42.748 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:42 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:42.748 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:42 smithi160 ceph-mon[33203]: Adjusting osd_memory_target on smithi070 to 2480M 2024-09-21T09:57:42.748 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:42 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:42.748 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:42 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:42.748 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:42 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:42.748 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:42 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:42.748 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:42 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:42.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[29589]: osdmap e165: 8 total, 8 up, 8 in 2024-09-21T09:57:42.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:42.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:42.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:42.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:42.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:42.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:42.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:42.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:42.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:42.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:42.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[29589]: Adjusting osd_memory_target on smithi070 to 2480M 2024-09-21T09:57:42.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:42.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:42.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:42.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:42.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:42.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[41533]: osdmap e165: 8 total, 8 up, 8 in 2024-09-21T09:57:42.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:42.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:42.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:42.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:42.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:42.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:42.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:42.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:42.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:42.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:42.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[41533]: Adjusting osd_memory_target on smithi070 to 2480M 2024-09-21T09:57:42.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:42.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:42.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:42.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:42.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:42 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:43 smithi160 ceph-mon[33203]: pgmap v14: 137 pgs: 137 active+clean; 449 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.0 MiB/s wr, 269 op/s 2024-09-21T09:57:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:43 smithi160 ceph-mon[33203]: osdmap e166: 8 total, 8 up, 8 in 2024-09-21T09:57:43.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:43 smithi070 ceph-mon[41533]: pgmap v14: 137 pgs: 137 active+clean; 449 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.0 MiB/s wr, 269 op/s 2024-09-21T09:57:43.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:43 smithi070 ceph-mon[41533]: osdmap e166: 8 total, 8 up, 8 in 2024-09-21T09:57:43.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:43 smithi070 ceph-mon[29589]: pgmap v14: 137 pgs: 137 active+clean; 449 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.0 MiB/s wr, 269 op/s 2024-09-21T09:57:43.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:43 smithi070 ceph-mon[29589]: osdmap e166: 8 total, 8 up, 8 in 2024-09-21T09:57:43.879 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.SnapsShrinkAfterPrepare (11130 ms) 2024-09-21T09:57:43.879 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.SnapsShrinkToZeroBeforePrepare 2024-09-21T09:57:43.908 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:57:43.917 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 0~8388609 2024-09-21T09:57:43.973 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap1 2024-09-21T09:57:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:44 smithi160 ceph-mon[33203]: osdmap e167: 8 total, 8 up, 8 in 2024-09-21T09:57:44.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:44 smithi070 ceph-mon[29589]: osdmap e167: 8 total, 8 up, 8 in 2024-09-21T09:57:44.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:57:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:57:44.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:44 smithi070 ceph-mon[41533]: osdmap e167: 8 total, 8 up, 8 in 2024-09-21T09:57:45.519 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 8388609~8388609 2024-09-21T09:57:45.545 INFO:tasks.workunit.client.0.smithi070.stdout:write: X 0~1000 2024-09-21T09:57:45.545 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 1010~1000 2024-09-21T09:57:45.545 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap2 2024-09-21T09:57:45.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:45 smithi160 ceph-mon[33203]: pgmap v17: 137 pgs: 137 active+clean; 449 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 3.9 MiB/s wr, 260 op/s 2024-09-21T09:57:45.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:45 smithi160 ceph-mon[33203]: osdmap e168: 8 total, 8 up, 8 in 2024-09-21T09:57:45.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:45 smithi070 ceph-mon[29589]: pgmap v17: 137 pgs: 137 active+clean; 449 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 3.9 MiB/s wr, 260 op/s 2024-09-21T09:57:45.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:45 smithi070 ceph-mon[29589]: osdmap e168: 8 total, 8 up, 8 in 2024-09-21T09:57:45.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:45 smithi070 ceph-mon[41533]: pgmap v17: 137 pgs: 137 active+clean; 449 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 3.9 MiB/s wr, 260 op/s 2024-09-21T09:57:45.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:45 smithi070 ceph-mon[41533]: osdmap e168: 8 total, 8 up, 8 in 2024-09-21T09:57:46.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:46 smithi160 ceph-mon[33203]: osdmap e169: 8 total, 8 up, 8 in 2024-09-21T09:57:46.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:46 smithi070 ceph-mon[29589]: osdmap e169: 8 total, 8 up, 8 in 2024-09-21T09:57:46.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:46 smithi070 ceph-mon[41533]: osdmap e169: 8 total, 8 up, 8 in 2024-09-21T09:57:47.493 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-09-21T09:57:47.539 INFO:tasks.workunit.client.0.smithi070.stdout:write: X 8388609~1000 2024-09-21T09:57:47.545 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 16777228~1000 2024-09-21T09:57:47.545 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 167772160 2024-09-21T09:57:47.553 INFO:tasks.workunit.client.0.smithi070.stdout:write: 2 83886079~8388609 2024-09-21T09:57:47.566 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap3 2024-09-21T09:57:47.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:47 smithi160 ceph-mon[33203]: pgmap v20: 137 pgs: 137 active+clean; 465 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 58 MiB/s rd, 10 MiB/s wr, 440 op/s 2024-09-21T09:57:47.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:47 smithi160 ceph-mon[33203]: osdmap e170: 8 total, 8 up, 8 in 2024-09-21T09:57:47.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:47 smithi070 ceph-mon[41533]: pgmap v20: 137 pgs: 137 active+clean; 465 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 58 MiB/s rd, 10 MiB/s wr, 440 op/s 2024-09-21T09:57:47.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:47 smithi070 ceph-mon[41533]: osdmap e170: 8 total, 8 up, 8 in 2024-09-21T09:57:47.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:47 smithi070 ceph-mon[29589]: pgmap v20: 137 pgs: 137 active+clean; 465 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 58 MiB/s rd, 10 MiB/s wr, 440 op/s 2024-09-21T09:57:47.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:47 smithi070 ceph-mon[29589]: osdmap e170: 8 total, 8 up, 8 in 2024-09-21T09:57:48.074 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T09:57:48.074 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (3m) 7s ago 4m 26.8M - 0.25.0 c8568f914cd2 df28d76f1795 2024-09-21T09:57:48.074 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi160 *:3000 running (3m) 6s ago 3m 86.5M - 9.4.7 954c08fa6188 8b9850d5203c 2024-09-21T09:57:48.074 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi070.gansda smithi070 running (2m) 7s ago 2m 16.1M - 18.2.4-871-g6a1bdf85 0adab276c97e 6016b2501c8a 2024-09-21T09:57:48.074 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi160.byyhfz smithi160 running (2m) 6s ago 2m 14.0M - 18.2.4-871-g6a1bdf85 0adab276c97e 93e108d91e65 2024-09-21T09:57:48.074 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi160 *:8443,9283,8765 running (22s) 6s ago 7m 498M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 0f7e42b65de5 2024-09-21T09:57:48.074 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:9283,8765 running (9m) 7s ago 9m 357M - 18.2.4-871-g6a1bdf85 0adab276c97e e4814f12686d 2024-09-21T09:57:48.074 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (9m) 7s ago 9m 51.1M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e cee8ca713d09 2024-09-21T09:57:48.075 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi160 running (8m) 6s ago 8m 47.1M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e 0fd16436dfa0 2024-09-21T09:57:48.075 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (8m) 7s ago 8m 42.0M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e fc0590f324b5 2024-09-21T09:57:48.075 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (3m) 7s ago 3m 16.3M - 1.5.0 0da6a335fe13 2f7c5b1e36eb 2024-09-21T09:57:48.075 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi160 *:9100 running (3m) 6s ago 3m 16.0M - 1.5.0 0da6a335fe13 e0fa2f0d9975 2024-09-21T09:57:48.075 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (7m) 7s ago 7m 109M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 10580e80f007 2024-09-21T09:57:48.075 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (7m) 7s ago 7m 95.7M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e b7c0d1169d1d 2024-09-21T09:57:48.075 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (6m) 7s ago 6m 94.5M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 58a11b8ddb81 2024-09-21T09:57:48.075 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (6m) 7s ago 6m 79.0M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e aa0c8a732a41 2024-09-21T09:57:48.075 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi160 running (6m) 6s ago 6m 106M 4016M 18.2.4-871-g6a1bdf85 0adab276c97e 7325a075c488 2024-09-21T09:57:48.075 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi160 running (5m) 6s ago 5m 80.2M 4016M 18.2.4-871-g6a1bdf85 0adab276c97e f74026345641 2024-09-21T09:57:48.075 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi160 running (5m) 6s ago 5m 126M 4016M 18.2.4-871-g6a1bdf85 0adab276c97e 03e6bb54f8b2 2024-09-21T09:57:48.075 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi160 running (4m) 6s ago 4m 102M 4016M 18.2.4-871-g6a1bdf85 0adab276c97e e405b21a6146 2024-09-21T09:57:48.075 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi160 *:9095 running (2m) 6s ago 4m 42.8M - 2.43.0 a07b618ecd1d 136d3e422755 2024-09-21T09:57:48.408 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T09:57:48.409 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-09-21T09:57:48.409 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 3 2024-09-21T09:57:48.409 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:57:48.409 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-09-21T09:57:48.409 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 1, 2024-09-21T09:57:48.409 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 1 2024-09-21T09:57:48.409 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:57:48.409 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-09-21T09:57:48.409 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 8 2024-09-21T09:57:48.409 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:57:48.410 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": { 2024-09-21T09:57:48.410 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 2 2024-09-21T09:57:48.410 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:57:48.410 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-09-21T09:57:48.410 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 14, 2024-09-21T09:57:48.410 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 1 2024-09-21T09:57:48.410 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T09:57:48.410 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T09:57:48.699 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T09:57:48.699 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47", 2024-09-21T09:57:48.700 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-09-21T09:57:48.700 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-09-21T09:57:48.700 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-09-21T09:57:48.700 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "1/2 daemons upgraded", 2024-09-21T09:57:48.700 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "", 2024-09-21T09:57:48.700 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-09-21T09:57:48.700 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T09:57:48.700 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:48 smithi070 ceph-mon[29589]: from='client.24926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:48.700 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:48 smithi070 ceph-mon[29589]: osdmap e171: 8 total, 8 up, 8 in 2024-09-21T09:57:48.700 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:48 smithi070 ceph-mon[29589]: from='client.15033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:48.701 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:48 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/1105542569' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:57:48.701 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:48 smithi070 ceph-mon[41533]: from='client.24926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:48.701 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:48 smithi070 ceph-mon[41533]: osdmap e171: 8 total, 8 up, 8 in 2024-09-21T09:57:48.701 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:48 smithi070 ceph-mon[41533]: from='client.15033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:48.701 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:48 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/1105542569' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:57:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:48 smithi160 ceph-mon[33203]: from='client.24926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:48 smithi160 ceph-mon[33203]: osdmap e171: 8 total, 8 up, 8 in 2024-09-21T09:57:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:48 smithi160 ceph-mon[33203]: from='client.15033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:48 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/1105542569' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:57:49.335 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:57:49.357 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 83886079~1 2024-09-21T09:57:49.364 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 0 2024-09-21T09:57:49.380 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:57:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:49 smithi160 ceph-mon[33203]: from='client.15039 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:49 smithi160 ceph-mon[33203]: pgmap v23: 137 pgs: 137 active+clean; 465 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 58 MiB/s rd, 10 MiB/s wr, 440 op/s 2024-09-21T09:57:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:49 smithi160 ceph-mon[33203]: osdmap e172: 8 total, 8 up, 8 in 2024-09-21T09:57:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:49 smithi160 ceph-mon[33203]: from='client.24950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:49 smithi160 ceph-mon[33203]: osdmap e173: 8 total, 8 up, 8 in 2024-09-21T09:57:49.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:49 smithi070 ceph-mon[29589]: from='client.15039 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:49.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:49 smithi070 ceph-mon[29589]: pgmap v23: 137 pgs: 137 active+clean; 465 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 58 MiB/s rd, 10 MiB/s wr, 440 op/s 2024-09-21T09:57:49.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:49 smithi070 ceph-mon[29589]: osdmap e172: 8 total, 8 up, 8 in 2024-09-21T09:57:49.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:49 smithi070 ceph-mon[29589]: from='client.24950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:49.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:49 smithi070 ceph-mon[29589]: osdmap e173: 8 total, 8 up, 8 in 2024-09-21T09:57:49.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:49 smithi070 ceph-mon[41533]: from='client.15039 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:49.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:49 smithi070 ceph-mon[41533]: pgmap v23: 137 pgs: 137 active+clean; 465 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 58 MiB/s rd, 10 MiB/s wr, 440 op/s 2024-09-21T09:57:49.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:49 smithi070 ceph-mon[41533]: osdmap e172: 8 total, 8 up, 8 in 2024-09-21T09:57:49.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:49 smithi070 ceph-mon[41533]: from='client.24950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:49.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:49 smithi070 ceph-mon[41533]: osdmap e173: 8 total, 8 up, 8 in 2024-09-21T09:57:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:51 smithi160 ceph-mon[33203]: osdmap e174: 8 total, 8 up, 8 in 2024-09-21T09:57:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:51 smithi160 ceph-mon[33203]: pgmap v27: 137 pgs: 137 active+clean; 465 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2024-09-21T09:57:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:51 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:51.591 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:51 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:57:51.290Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.70:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.70:8765: connect: connection refused" 2024-09-21T09:57:51.591 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:51 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:57:51.293Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.70:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.70:8765: connect: connection refused" 2024-09-21T09:57:51.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:51 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:57:51.293Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.70:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.70:8765: connect: connection refused" 2024-09-21T09:57:51.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:51 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:57:51.293Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.70:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.70:8765: connect: connection refused" 2024-09-21T09:57:51.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:51 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:57:51.294Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.70:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.70:8765: connect: connection refused" 2024-09-21T09:57:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:51 smithi070 ceph-mon[29589]: osdmap e174: 8 total, 8 up, 8 in 2024-09-21T09:57:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:51 smithi070 ceph-mon[29589]: pgmap v27: 137 pgs: 137 active+clean; 465 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2024-09-21T09:57:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:51 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:51.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:51 smithi070 ceph-mon[41533]: osdmap e174: 8 total, 8 up, 8 in 2024-09-21T09:57:51.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:51 smithi070 ceph-mon[41533]: pgmap v27: 137 pgs: 137 active+clean; 465 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2024-09-21T09:57:51.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:51 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:52.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:52 smithi160 ceph-mon[33203]: osdmap e175: 8 total, 8 up, 8 in 2024-09-21T09:57:52.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:52 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:52.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:52 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:52.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:52 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:52.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:52 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:52.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:52 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:52.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:52 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:52.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:52 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:52.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:52 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:52.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:52 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:52.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:52 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:52.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:52 smithi160 ceph-mon[33203]: Adjusting osd_memory_target on smithi160 to 2480M 2024-09-21T09:57:52.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:52 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:52.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:52 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:52.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:52 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:57:52.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:52 smithi160 ceph-mon[33203]: Updating smithi070:/etc/ceph/ceph.conf 2024-09-21T09:57:52.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:52 smithi160 ceph-mon[33203]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-21T09:57:52.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:52 smithi160 ceph-mon[33203]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:57:52.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:52 smithi160 ceph-mon[33203]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:57:52.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:52 smithi160 ceph-mon[33203]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:57:52.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:52 smithi160 ceph-mon[33203]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:57:52.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:52 smithi160 ceph-mon[33203]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:57:52.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:52 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:52.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:52 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:52.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:52 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:52.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:52 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:52.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:52 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:52.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[41533]: osdmap e175: 8 total, 8 up, 8 in 2024-09-21T09:57:52.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:52.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:52.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:52.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:52.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:52.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:52.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:52.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:52.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:52.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:52.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[41533]: Adjusting osd_memory_target on smithi160 to 2480M 2024-09-21T09:57:52.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:52.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:52.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:57:52.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[41533]: Updating smithi070:/etc/ceph/ceph.conf 2024-09-21T09:57:52.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[41533]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-21T09:57:52.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[41533]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:57:52.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[41533]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:57:52.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[41533]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:57:52.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[41533]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:57:52.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[41533]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:57:52.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:52.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:52.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:52.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:52.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:52.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[29589]: osdmap e175: 8 total, 8 up, 8 in 2024-09-21T09:57:52.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:52.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:52.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:52.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:52.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:52.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:52.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:52.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:52.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:52.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:57:52.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[29589]: Adjusting osd_memory_target on smithi160 to 2480M 2024-09-21T09:57:52.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:52.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:52.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:57:52.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[29589]: Updating smithi070:/etc/ceph/ceph.conf 2024-09-21T09:57:52.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[29589]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-21T09:57:52.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[29589]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:57:52.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[29589]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:57:52.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[29589]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:57:52.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[29589]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:57:52.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[29589]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:57:52.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:52.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:52.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:52.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:52.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:52 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:52.664 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:57:52.808 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:57:53.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:53 smithi160 ceph-mon[33203]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:57:53.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:53 smithi160 ceph-mon[33203]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:57:53.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:53 smithi160 ceph-mon[33203]: osdmap e176: 8 total, 8 up, 8 in 2024-09-21T09:57:53.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:53 smithi160 ceph-mon[33203]: pgmap v30: 137 pgs: 137 active+clean; 505 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 152 KiB/s rd, 8.2 MiB/s wr, 222 op/s 2024-09-21T09:57:53.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:53 smithi160 ceph-mon[33203]: Reconfiguring daemon prometheus.a on smithi160 2024-09-21T09:57:53.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:53 smithi070 ceph-mon[29589]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:57:53.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:53 smithi070 ceph-mon[29589]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:57:53.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:53 smithi070 ceph-mon[29589]: osdmap e176: 8 total, 8 up, 8 in 2024-09-21T09:57:53.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:53 smithi070 ceph-mon[29589]: pgmap v30: 137 pgs: 137 active+clean; 505 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 152 KiB/s rd, 8.2 MiB/s wr, 222 op/s 2024-09-21T09:57:53.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:53 smithi070 ceph-mon[29589]: Reconfiguring daemon prometheus.a on smithi160 2024-09-21T09:57:53.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:53 smithi070 ceph-mon[41533]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:57:53.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:53 smithi070 ceph-mon[41533]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:57:53.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:53 smithi070 ceph-mon[41533]: osdmap e176: 8 total, 8 up, 8 in 2024-09-21T09:57:53.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:53 smithi070 ceph-mon[41533]: pgmap v30: 137 pgs: 137 active+clean; 505 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 152 KiB/s rd, 8.2 MiB/s wr, 222 op/s 2024-09-21T09:57:53.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:53 smithi070 ceph-mon[41533]: Reconfiguring daemon prometheus.a on smithi160 2024-09-21T09:57:54.474 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:54 smithi160 ceph-mon[33203]: osdmap e177: 8 total, 8 up, 8 in 2024-09-21T09:57:54.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:54 smithi070 ceph-mon[29589]: osdmap e177: 8 total, 8 up, 8 in 2024-09-21T09:57:54.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:57:54 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[29838]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:57:54] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:57:54.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:54 smithi070 ceph-mon[41533]: osdmap e177: 8 total, 8 up, 8 in 2024-09-21T09:57:55.473 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:55 smithi160 ceph-mon[33203]: osdmap e178: 8 total, 8 up, 8 in 2024-09-21T09:57:55.474 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:55 smithi160 ceph-mon[33203]: pgmap v33: 137 pgs: 137 active+clean; 505 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 149 KiB/s rd, 8.0 MiB/s wr, 219 op/s 2024-09-21T09:57:55.600 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.SnapsShrinkToZeroBeforePrepare (11722 ms) 2024-09-21T09:57:55.600 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.SnapsShrinkToZeroAfterPrepare 2024-09-21T09:57:55.634 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:55 smithi070 ceph-mon[29589]: osdmap e178: 8 total, 8 up, 8 in 2024-09-21T09:57:55.634 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:55 smithi070 ceph-mon[29589]: pgmap v33: 137 pgs: 137 active+clean; 505 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 149 KiB/s rd, 8.0 MiB/s wr, 219 op/s 2024-09-21T09:57:55.634 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:55 smithi070 ceph-mon[41533]: osdmap e178: 8 total, 8 up, 8 in 2024-09-21T09:57:55.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:55 smithi070 ceph-mon[41533]: pgmap v33: 137 pgs: 137 active+clean; 505 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 149 KiB/s rd, 8.0 MiB/s wr, 219 op/s 2024-09-21T09:57:55.635 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:57:55.640 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 0~8388609 2024-09-21T09:57:55.694 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap1 2024-09-21T09:57:55.843 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:55 smithi160 systemd[1]: Stopping Ceph prometheus.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:57:56.138 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:57:55.877Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T09:57:56.138 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:57:55.878Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-21T09:57:56.139 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:57:55.878Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-21T09:57:56.139 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:57:55.878Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T09:57:56.139 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:57:55.878Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-21T09:57:56.139 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:57:55.878Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-21T09:57:56.139 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:57:55.878Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T09:57:56.139 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:57:55.878Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-21T09:57:56.139 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:57:55.878Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-21T09:57:56.139 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:57:55.878Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T09:57:56.139 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:57:55.878Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-21T09:57:56.139 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[65656]: ts=2024-09-21T09:57:55.878Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-21T09:57:56.139 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:55 smithi160 podman[77136]: 2024-09-21 09:57:55.888422207 +0000 UTC m=+0.114472282 container died 136d3e422755ba98b78f46ff8e6a3bebc7310387de232dbcd5ccfe811fa84210 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:57:56.539 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:56 smithi160 ceph-mon[33203]: osdmap e179: 8 total, 8 up, 8 in 2024-09-21T09:57:56.539 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:56 smithi160 podman[77136]: 2024-09-21 09:57:56.30957416 +0000 UTC m=+0.535624270 container cleanup 136d3e422755ba98b78f46ff8e6a3bebc7310387de232dbcd5ccfe811fa84210 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:57:56.539 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:56 smithi160 bash[77136]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T09:57:56.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:56 smithi070 ceph-mon[29589]: osdmap e179: 8 total, 8 up, 8 in 2024-09-21T09:57:56.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:56 smithi070 ceph-mon[41533]: osdmap e179: 8 total, 8 up, 8 in 2024-09-21T09:57:56.789 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:56 smithi160 podman[77154]: 2024-09-21 09:57:56.537174707 +0000 UTC m=+0.648718690 container remove 136d3e422755ba98b78f46ff8e6a3bebc7310387de232dbcd5ccfe811fa84210 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:57:57.091 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:56 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-21T09:57:57.091 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:56 smithi160 systemd[1]: Stopped Ceph prometheus.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:57:57.091 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:56 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@prometheus.a.service: Consumed 1.794s CPU time. 2024-09-21T09:57:57.091 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:56 smithi160 systemd[1]: Starting Ceph prometheus.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:57:57.091 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:56 smithi160 podman[77249]: 2024-09-21 09:57:56.898014099 +0000 UTC m=+0.017665217 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-21T09:57:57.092 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:57 smithi160 podman[77249]: 2024-09-21 09:57:57.038276075 +0000 UTC m=+0.157927166 container create fb7be2db65a2e3668178c1533c0cc5fd68b8287a679bfbf4c02dd66390d03635 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:57:57.377 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 8388609~8388609 2024-09-21T09:57:57.399 INFO:tasks.workunit.client.0.smithi070.stdout:write: X 0~1000 2024-09-21T09:57:57.399 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 1010~1000 2024-09-21T09:57:57.399 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap2 2024-09-21T09:57:57.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:57 smithi160 ceph-mon[33203]: osdmap e180: 8 total, 8 up, 8 in 2024-09-21T09:57:57.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:57 smithi160 ceph-mon[33203]: pgmap v36: 137 pgs: 137 active+clean; 505 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 39 MiB/s rd, 6.0 MiB/s wr, 419 op/s 2024-09-21T09:57:57.591 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:57 smithi160 podman[77249]: 2024-09-21 09:57:57.273515346 +0000 UTC m=+0.393166457 container init fb7be2db65a2e3668178c1533c0cc5fd68b8287a679bfbf4c02dd66390d03635 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:57:57.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:57 smithi160 podman[77249]: 2024-09-21 09:57:57.278890517 +0000 UTC m=+0.398541608 container start fb7be2db65a2e3668178c1533c0cc5fd68b8287a679bfbf4c02dd66390d03635 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:57:57.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:57:57.302Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-21T09:57:57.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:57:57.302Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-21T09:57:57.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:57:57.302Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi160 (none))" 2024-09-21T09:57:57.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:57:57.302Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-21T09:57:57.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:57:57.302Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-21T09:57:57.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:57:57.304Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-21T09:57:57.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:57:57.304Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-21T09:57:57.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:57:57.306Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-21T09:57:57.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:57:57.306Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-21T09:57:57.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:57:57.307Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-21T09:57:57.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:57:57.307Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.13µs 2024-09-21T09:57:57.593 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:57:57.307Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-21T09:57:57.593 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:57:57.309Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-09-21T09:57:57.593 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:57:57.316Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-09-21T09:57:57.593 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:57:57.316Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-09-21T09:57:57.593 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:57:57.316Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=28.816µs wal_replay_duration=9.192031ms wbl_replay_duration=108ns total_replay_duration=9.247973ms 2024-09-21T09:57:57.593 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:57:57.318Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-21T09:57:57.593 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:57:57.318Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-21T09:57:57.593 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:57:57.318Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-21T09:57:57.593 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:57:57.339Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=21.423876ms db_storage=786ns remote_storage=1.144µs web_handler=445ns query_engine=614ns scrape=419.751µs scrape_sd=131.983µs notify=28.105µs notify_sd=24.612µs rules=20.442281ms tracing=6.13µs 2024-09-21T09:57:57.593 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:57:57.339Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-21T09:57:57.594 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:57 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:57:57.339Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-21T09:57:57.594 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:57 smithi160 bash[77249]: fb7be2db65a2e3668178c1533c0cc5fd68b8287a679bfbf4c02dd66390d03635 2024-09-21T09:57:57.594 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:57:57 smithi160 systemd[1]: Started Ceph prometheus.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:57:57.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:57 smithi070 ceph-mon[29589]: osdmap e180: 8 total, 8 up, 8 in 2024-09-21T09:57:57.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:57 smithi070 ceph-mon[29589]: pgmap v36: 137 pgs: 137 active+clean; 505 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 39 MiB/s rd, 6.0 MiB/s wr, 419 op/s 2024-09-21T09:57:57.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:57 smithi070 ceph-mon[41533]: osdmap e180: 8 total, 8 up, 8 in 2024-09-21T09:57:57.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:57 smithi070 ceph-mon[41533]: pgmap v36: 137 pgs: 137 active+clean; 505 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 39 MiB/s rd, 6.0 MiB/s wr, 419 op/s 2024-09-21T09:57:58.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:58 smithi070 ceph-mon[29589]: osdmap e181: 8 total, 8 up, 8 in 2024-09-21T09:57:58.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:58 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:58.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:58 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:58.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:58 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:57:58.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:58 smithi070 ceph-mon[29589]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:57:58.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:58 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:57:58.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:58 smithi070 ceph-mon[29589]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-21T09:57:58.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:58 smithi070 ceph-mon[29589]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-21T09:57:58.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:58 smithi070 ceph-mon[41533]: osdmap e181: 8 total, 8 up, 8 in 2024-09-21T09:57:58.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:58 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:58.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:58 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:58.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:58 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:57:58.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:58 smithi070 ceph-mon[41533]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:57:58.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:58 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:57:58.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:58 smithi070 ceph-mon[41533]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-21T09:57:58.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:58 smithi070 ceph-mon[41533]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-21T09:57:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:58 smithi160 ceph-mon[33203]: osdmap e181: 8 total, 8 up, 8 in 2024-09-21T09:57:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:58 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:58 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:58 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:57:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:58 smithi160 ceph-mon[33203]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:57:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:58 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:57:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:58 smithi160 ceph-mon[33203]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-21T09:57:58.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:58 smithi160 ceph-mon[33203]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-21T09:57:59.341 INFO:tasks.workunit.client.0.smithi070.stdout:write: X 8388609~1000 2024-09-21T09:57:59.346 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 16777228~1000 2024-09-21T09:57:59.346 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 167772160 2024-09-21T09:57:59.354 INFO:tasks.workunit.client.0.smithi070.stdout:write: 2 83886079~8388609 2024-09-21T09:57:59.372 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap3 2024-09-21T09:57:59.401 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:59 smithi070 ceph-mon[29589]: osdmap e182: 8 total, 8 up, 8 in 2024-09-21T09:57:59.401 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:59 smithi070 ceph-mon[29589]: pgmap v39: 137 pgs: 137 active+clean; 505 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 39 MiB/s rd, 6.0 MiB/s wr, 419 op/s 2024-09-21T09:57:59.401 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:59 smithi070 ceph-mon[29589]: Upgrade: Updating mgr.y 2024-09-21T09:57:59.401 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:59 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:59.401 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:59 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:57:59.401 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:59 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:57:59.401 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:59 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:57:59.401 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:59 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:59.401 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:59 smithi070 ceph-mon[29589]: Deploying daemon mgr.y on smithi070 2024-09-21T09:57:59.402 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:57:59 smithi070 ceph-mon[29589]: osdmap e183: 8 total, 8 up, 8 in 2024-09-21T09:57:59.402 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:59 smithi070 ceph-mon[41533]: osdmap e182: 8 total, 8 up, 8 in 2024-09-21T09:57:59.402 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:59 smithi070 ceph-mon[41533]: pgmap v39: 137 pgs: 137 active+clean; 505 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 39 MiB/s rd, 6.0 MiB/s wr, 419 op/s 2024-09-21T09:57:59.402 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:59 smithi070 ceph-mon[41533]: Upgrade: Updating mgr.y 2024-09-21T09:57:59.402 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:59 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:59.402 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:59 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:57:59.403 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:59 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:57:59.403 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:59 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:57:59.403 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:59 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:59.403 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:59 smithi070 ceph-mon[41533]: Deploying daemon mgr.y on smithi070 2024-09-21T09:57:59.403 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:57:59 smithi070 ceph-mon[41533]: osdmap e183: 8 total, 8 up, 8 in 2024-09-21T09:57:59.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:59 smithi160 ceph-mon[33203]: osdmap e182: 8 total, 8 up, 8 in 2024-09-21T09:57:59.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:59 smithi160 ceph-mon[33203]: pgmap v39: 137 pgs: 137 active+clean; 505 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 39 MiB/s rd, 6.0 MiB/s wr, 419 op/s 2024-09-21T09:57:59.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:59 smithi160 ceph-mon[33203]: Upgrade: Updating mgr.y 2024-09-21T09:57:59.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:59 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:57:59.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:59 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:57:59.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:59 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:57:59.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:59 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:57:59.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:59 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:59.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:59 smithi160 ceph-mon[33203]: Deploying daemon mgr.y on smithi070 2024-09-21T09:57:59.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:57:59 smithi160 ceph-mon[33203]: osdmap e183: 8 total, 8 up, 8 in 2024-09-21T09:58:00.812 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:00 smithi070 systemd[1]: Stopping Ceph mgr.y for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:58:01.096 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:00 smithi070 podman[89610]: 2024-09-21 09:58:00.809774326 +0000 UTC m=+0.153673395 container died e4814f12686d04893f1b815a20904bcd2737df4a2207e438027e94239bd0327b (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=reef-6a1bdf8, org.label-schema.schema-version=1.0) 2024-09-21T09:58:01.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:01 smithi070 podman[89610]: 2024-09-21 09:58:01.251805802 +0000 UTC m=+0.595704859 container cleanup e4814f12686d04893f1b815a20904bcd2737df4a2207e438027e94239bd0327b (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, RELEASE=reef-6a1bdf8, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-21T09:58:01.378 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:01 smithi070 bash[89610]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y 2024-09-21T09:58:01.410 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:58:01.429 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 83886079~1 2024-09-21T09:58:01.435 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:58:01.713 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:01 smithi070 ceph-mon[29589]: osdmap e184: 8 total, 8 up, 8 in 2024-09-21T09:58:01.713 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:01 smithi070 ceph-mon[29589]: pgmap v42: 137 pgs: 137 active+clean; 513 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.0 MiB/s wr, 1 op/s 2024-09-21T09:58:01.713 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:01 smithi070 ceph-mon[41533]: osdmap e184: 8 total, 8 up, 8 in 2024-09-21T09:58:01.713 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:01 smithi070 ceph-mon[41533]: pgmap v42: 137 pgs: 137 active+clean; 513 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.0 MiB/s wr, 1 op/s 2024-09-21T09:58:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:01 smithi160 ceph-mon[33203]: osdmap e184: 8 total, 8 up, 8 in 2024-09-21T09:58:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:01 smithi160 ceph-mon[33203]: pgmap v42: 137 pgs: 137 active+clean; 513 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.0 MiB/s wr, 1 op/s 2024-09-21T09:58:02.127 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:01 smithi070 podman[89627]: 2024-09-21 09:58:01.711635293 +0000 UTC m=+0.905993858 container remove e4814f12686d04893f1b815a20904bcd2737df4a2207e438027e94239bd0327b (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=reef-6a1bdf8, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-21T09:58:02.127 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:01 smithi070 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-09-21T09:58:02.127 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:01 smithi070 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mgr.y.service: Failed with result 'exit-code'. 2024-09-21T09:58:02.127 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:01 smithi070 systemd[1]: Stopped Ceph mgr.y for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:58:02.128 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:01 smithi070 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mgr.y.service: Consumed 42.656s CPU time. 2024-09-21T09:58:02.396 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:02 smithi070 systemd[1]: Starting Ceph mgr.y for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:58:02.396 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:02 smithi070 podman[89760]: 2024-09-21 09:58:02.242184706 +0000 UTC m=+0.015844037 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:58:02.397 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:02 smithi070 podman[89760]: 2024-09-21 09:58:02.346140447 +0000 UTC m=+0.119799765 container create e3d390893cff7dfefe7f03b68d7aea1d181d3d58a08908234a15f82c29160fc9 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-62da54b, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-21T09:58:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:02 smithi160 ceph-mon[33203]: osdmap e185: 8 total, 8 up, 8 in 2024-09-21T09:58:02.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:02 smithi070 ceph-mon[41533]: osdmap e185: 8 total, 8 up, 8 in 2024-09-21T09:58:02.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:02 smithi070 ceph-mon[29589]: osdmap e185: 8 total, 8 up, 8 in 2024-09-21T09:58:02.878 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:02 smithi070 podman[89760]: 2024-09-21 09:58:02.556810014 +0000 UTC m=+0.330469341 container init e3d390893cff7dfefe7f03b68d7aea1d181d3d58a08908234a15f82c29160fc9 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, RELEASE=main-62da54b, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-21T09:58:02.878 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:02 smithi070 podman[89760]: 2024-09-21 09:58:02.561648266 +0000 UTC m=+0.335307585 container start e3d390893cff7dfefe7f03b68d7aea1d181d3d58a08908234a15f82c29160fc9 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True) 2024-09-21T09:58:02.878 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:02 smithi070 ceph-mgr[89799]: -- 172.21.15.70:0/2989687027 <== mon.2 v2:172.21.15.70:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556e4f9ad0e0 con 0x556e4f98b000 2024-09-21T09:58:02.878 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:02 smithi070 bash[89760]: e3d390893cff7dfefe7f03b68d7aea1d181d3d58a08908234a15f82c29160fc9 2024-09-21T09:58:02.878 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:02 smithi070 systemd[1]: Started Ceph mgr.y for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:58:03.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:03 smithi160 ceph-mon[33203]: osdmap e186: 8 total, 8 up, 8 in 2024-09-21T09:58:03.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:03 smithi160 ceph-mon[33203]: pgmap v45: 137 pgs: 137 active+clean; 561 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 12 MiB/s wr, 158 op/s 2024-09-21T09:58:03.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:03 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:03.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:03 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:03.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:03 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:03.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:03 smithi070 ceph-mon[29589]: osdmap e186: 8 total, 8 up, 8 in 2024-09-21T09:58:03.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:03 smithi070 ceph-mon[29589]: pgmap v45: 137 pgs: 137 active+clean; 561 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 12 MiB/s wr, 158 op/s 2024-09-21T09:58:03.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:03 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:03.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:03 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:03.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:03 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:03.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:03 smithi070 ceph-mon[41533]: osdmap e186: 8 total, 8 up, 8 in 2024-09-21T09:58:03.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:03 smithi070 ceph-mon[41533]: pgmap v45: 137 pgs: 137 active+clean; 561 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 12 MiB/s wr, 158 op/s 2024-09-21T09:58:03.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:03 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:03.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:03 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:03.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:03 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:04 smithi160 ceph-mon[33203]: osdmap e187: 8 total, 8 up, 8 in 2024-09-21T09:58:04.837 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 0 2024-09-21T09:58:04.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:04 smithi070 ceph-mon[41533]: osdmap e187: 8 total, 8 up, 8 in 2024-09-21T09:58:04.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:04 smithi070 ceph-mon[29589]: osdmap e187: 8 total, 8 up, 8 in 2024-09-21T09:58:04.905 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:58:05.033 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:58:05.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:05 smithi160 ceph-mon[33203]: osdmap e188: 8 total, 8 up, 8 in 2024-09-21T09:58:05.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:05 smithi160 ceph-mon[33203]: pgmap v48: 137 pgs: 137 active+clean; 561 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 11 MiB/s wr, 157 op/s 2024-09-21T09:58:05.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:05 smithi070 ceph-mon[29589]: osdmap e188: 8 total, 8 up, 8 in 2024-09-21T09:58:05.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:05 smithi070 ceph-mon[29589]: pgmap v48: 137 pgs: 137 active+clean; 561 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 11 MiB/s wr, 157 op/s 2024-09-21T09:58:05.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:05 smithi070 ceph-mon[41533]: osdmap e188: 8 total, 8 up, 8 in 2024-09-21T09:58:05.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:05 smithi070 ceph-mon[41533]: pgmap v48: 137 pgs: 137 active+clean; 561 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 11 MiB/s wr, 157 op/s 2024-09-21T09:58:06.679 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:06 smithi160 ceph-mon[33203]: osdmap e189: 8 total, 8 up, 8 in 2024-09-21T09:58:06.679 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:06 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:58:06.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:06 smithi070 ceph-mon[41533]: osdmap e189: 8 total, 8 up, 8 in 2024-09-21T09:58:06.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:06 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:58:06.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:06 smithi070 ceph-mon[29589]: osdmap e189: 8 total, 8 up, 8 in 2024-09-21T09:58:06.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:06 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:58:07.668 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.SnapsShrinkToZeroAfterPrepare (12068 ms) 2024-09-21T09:58:07.668 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.SnapsExpandAfterPrepare 2024-09-21T09:58:07.697 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:58:07.707 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 0~8388609 2024-09-21T09:58:07.758 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap1 2024-09-21T09:58:07.760 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:07 smithi070 ceph-mon[29589]: osdmap e190: 8 total, 8 up, 8 in 2024-09-21T09:58:07.760 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:07 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:07.760 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:07 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:07.760 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:07 smithi070 ceph-mon[29589]: pgmap v51: 137 pgs: 137 active+clean; 581 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 42 MiB/s rd, 6.0 MiB/s wr, 541 op/s 2024-09-21T09:58:07.761 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:07 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:07.761 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:07 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:07.761 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:07 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:07.761 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:07 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:07.761 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:07 smithi070 ceph-mon[41533]: osdmap e190: 8 total, 8 up, 8 in 2024-09-21T09:58:07.761 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:07 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:07.761 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:07 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:07.761 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:07 smithi070 ceph-mon[41533]: pgmap v51: 137 pgs: 137 active+clean; 581 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 42 MiB/s rd, 6.0 MiB/s wr, 541 op/s 2024-09-21T09:58:07.761 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:07 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:07.761 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:07 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:07.761 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:07 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:07.762 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:07 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:07 smithi160 ceph-mon[33203]: osdmap e190: 8 total, 8 up, 8 in 2024-09-21T09:58:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:07 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:07 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:07 smithi160 ceph-mon[33203]: pgmap v51: 137 pgs: 137 active+clean; 581 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 42 MiB/s rd, 6.0 MiB/s wr, 541 op/s 2024-09-21T09:58:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:07 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:07 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:07 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:07.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:07 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:08.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:08 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:09:58:08] ENGINE Bus STARTING 2024-09-21T09:58:08.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:08 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: CherryPy Checker: 2024-09-21T09:58:08.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:08 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: The Application mounted at '' has an empty config. 2024-09-21T09:58:08.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:08 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T09:58:08.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:08 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:09:58:08] ENGINE Serving on http://:::9283 2024-09-21T09:58:08.378 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:08 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:09:58:08] ENGINE Bus STARTED 2024-09-21T09:58:08.426 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:08 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:58:08] "GET /metrics HTTP/1.1" 200 38726 "" "Prometheus/2.43.0" 2024-09-21T09:58:08.758 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:08 smithi070 ceph-mon[29589]: osdmap e191: 8 total, 8 up, 8 in 2024-09-21T09:58:08.758 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:08 smithi070 ceph-mon[29589]: Standby manager daemon y restarted 2024-09-21T09:58:08.758 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:08 smithi070 ceph-mon[29589]: Standby manager daemon y started 2024-09-21T09:58:08.758 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:08 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T09:58:08.758 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:08 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:58:08.758 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:08 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T09:58:08.758 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:08 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:58:08.759 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:08 smithi070 ceph-mon[41533]: osdmap e191: 8 total, 8 up, 8 in 2024-09-21T09:58:08.759 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:08 smithi070 ceph-mon[41533]: Standby manager daemon y restarted 2024-09-21T09:58:08.759 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:08 smithi070 ceph-mon[41533]: Standby manager daemon y started 2024-09-21T09:58:08.759 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:08 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T09:58:08.759 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:08 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:58:08.759 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:08 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T09:58:08.759 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:08 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:58:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:08 smithi160 ceph-mon[33203]: osdmap e191: 8 total, 8 up, 8 in 2024-09-21T09:58:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:08 smithi160 ceph-mon[33203]: Standby manager daemon y restarted 2024-09-21T09:58:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:08 smithi160 ceph-mon[33203]: Standby manager daemon y started 2024-09-21T09:58:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:08 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T09:58:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:08 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:58:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:08 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T09:58:08.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:08 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:58:09.339 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 8388609~8388609 2024-09-21T09:58:09.350 INFO:tasks.workunit.client.0.smithi070.stdout:write: X 0~1000 2024-09-21T09:58:09.350 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 1010~1000 2024-09-21T09:58:09.350 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap2 2024-09-21T09:58:09.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:09 smithi070 ceph-mon[41533]: pgmap v53: 137 pgs: 137 active+clean; 581 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 42 MiB/s rd, 5.9 MiB/s wr, 534 op/s 2024-09-21T09:58:09.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:09 smithi070 ceph-mon[41533]: mgrmap e26: x(active, since 34s), standbys: y 2024-09-21T09:58:09.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:09 smithi070 ceph-mon[41533]: osdmap e192: 8 total, 8 up, 8 in 2024-09-21T09:58:09.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:09 smithi070 ceph-mon[41533]: osdmap e193: 8 total, 8 up, 8 in 2024-09-21T09:58:09.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:09 smithi070 ceph-mon[29589]: pgmap v53: 137 pgs: 137 active+clean; 581 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 42 MiB/s rd, 5.9 MiB/s wr, 534 op/s 2024-09-21T09:58:09.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:09 smithi070 ceph-mon[29589]: mgrmap e26: x(active, since 34s), standbys: y 2024-09-21T09:58:09.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:09 smithi070 ceph-mon[29589]: osdmap e192: 8 total, 8 up, 8 in 2024-09-21T09:58:09.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:09 smithi070 ceph-mon[29589]: osdmap e193: 8 total, 8 up, 8 in 2024-09-21T09:58:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:09 smithi160 ceph-mon[33203]: pgmap v53: 137 pgs: 137 active+clean; 581 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 42 MiB/s rd, 5.9 MiB/s wr, 534 op/s 2024-09-21T09:58:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:09 smithi160 ceph-mon[33203]: mgrmap e26: x(active, since 34s), standbys: y 2024-09-21T09:58:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:09 smithi160 ceph-mon[33203]: osdmap e192: 8 total, 8 up, 8 in 2024-09-21T09:58:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:09 smithi160 ceph-mon[33203]: osdmap e193: 8 total, 8 up, 8 in 2024-09-21T09:58:10.520 INFO:tasks.workunit.client.0.smithi070.stdout:write: X 8388609~1000 2024-09-21T09:58:10.525 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 16777228~1000 2024-09-21T09:58:10.525 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 167772160 2024-09-21T09:58:10.533 INFO:tasks.workunit.client.0.smithi070.stdout:write: 2 83886079~8388609 2024-09-21T09:58:10.549 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap3 2024-09-21T09:58:10.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:10 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:09:58:10] ENGINE Bus STOPPING 2024-09-21T09:58:10.841 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:10 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: ignoring --setuser ceph since I am not root 2024-09-21T09:58:10.841 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:10 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: ignoring --setgroup ceph since I am not root 2024-09-21T09:58:10.841 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:10 smithi160 ceph-mgr[69268]: -- 172.21.15.160:0/4164884738 <== mon.1 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5606d58cb0e0 con 0x5606d58a8800 2024-09-21T09:58:11.064 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:10 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:09:58:10] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:58:11.064 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:10 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:09:58:10] ENGINE Bus STOPPED 2024-09-21T09:58:11.064 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T09:58:11.061+0000 7fd2b953c640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:58:11.331 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T09:58:11.061+0000 7fd2b953c640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:58:11.331 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T09:58:11.061+0000 7fd2b953c640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:58:11.331 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T09:58:11.061+0000 7fd2b953c640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:58:11.331 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T09:58:11.062+0000 7fd2b953c640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:58:11.331 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T09:58:11.061+0000 7fd2bc542640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:58:11.331 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T09:58:11.062+0000 7fd2bc542640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:58:11.331 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T09:58:11.062+0000 7fd2bc542640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:58:11.331 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T09:58:11.062+0000 7fd2bc542640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:58:11.332 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T09:58:11.062+0000 7fd2bc542640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:58:11.428 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: osdmap e194: 8 total, 8 up, 8 in 2024-09-21T09:58:11.429 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:11.429 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:11.429 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:58:11.429 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:58:11.429 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:11.429 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:11.430 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-21T09:58:11.430 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-21T09:58:11.430 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: Activating manager daemon y 2024-09-21T09:58:11.430 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: osdmap e195: 8 total, 8 up, 8 in 2024-09-21T09:58:11.430 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24658 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-21T09:58:11.430 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: mgrmap e27: y(active, starting, since 0.00657059s) 2024-09-21T09:58:11.431 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:58:11.431 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:58:11.431 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:58:11.431 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi070.gansda"}]: dispatch 2024-09-21T09:58:11.431 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi160.byyhfz"}]: dispatch 2024-09-21T09:58:11.431 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:58:11.432 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:58:11.432 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:58:11.432 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:58:11.432 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:58:11.432 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:58:11.432 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:58:11.433 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:58:11.433 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:58:11.433 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:58:11.433 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:58:11.433 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:58:11.433 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: Manager daemon y is now available 2024-09-21T09:58:11.434 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:11.434 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:58:11.434 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:58:11.434 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:58:11.434 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:11 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:58:11.626 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:09:58:11] ENGINE Bus STARTING 2024-09-21T09:58:11.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: osdmap e194: 8 total, 8 up, 8 in 2024-09-21T09:58:11.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:11.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:11.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:58:11.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:58:11.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:11.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:11.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-21T09:58:11.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-21T09:58:11.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: Activating manager daemon y 2024-09-21T09:58:11.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: osdmap e195: 8 total, 8 up, 8 in 2024-09-21T09:58:11.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24658 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-21T09:58:11.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: mgrmap e27: y(active, starting, since 0.00657059s) 2024-09-21T09:58:11.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:58:11.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:58:11.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:58:11.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi070.gansda"}]: dispatch 2024-09-21T09:58:11.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi160.byyhfz"}]: dispatch 2024-09-21T09:58:11.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:58:11.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:58:11.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:58:11.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:58:11.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:58:11.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:58:11.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:58:11.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:58:11.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:58:11.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:58:11.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:58:11.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:58:11.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: osdmap e194: 8 total, 8 up, 8 in 2024-09-21T09:58:11.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:11.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:11.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:58:11.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:58:11.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' 2024-09-21T09:58:11.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:11.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24658 172.21.15.160:0/1619576225' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-21T09:58:11.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-21T09:58:11.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: Activating manager daemon y 2024-09-21T09:58:11.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: osdmap e195: 8 total, 8 up, 8 in 2024-09-21T09:58:11.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24658 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-21T09:58:11.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: mgrmap e27: y(active, starting, since 0.00657059s) 2024-09-21T09:58:11.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:58:11.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:58:11.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:58:11.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi070.gansda"}]: dispatch 2024-09-21T09:58:11.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi160.byyhfz"}]: dispatch 2024-09-21T09:58:11.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:58:11.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:58:11.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:58:11.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:58:11.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:58:11.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:58:11.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:58:11.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:58:11.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:58:11.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:58:11.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:58:11.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:58:11.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: Manager daemon y is now available 2024-09-21T09:58:11.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:11.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:58:11.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:58:11.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:58:11.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:58:11.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: Manager daemon y is now available 2024-09-21T09:58:11.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:11.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:58:11.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:58:11.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:58:11.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:58:11.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:09:58:11] ENGINE Serving on http://:::9283 2024-09-21T09:58:11.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:11 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:09:58:11] ENGINE Bus STARTED 2024-09-21T09:58:12.505 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T09:58:12.293+0000 7fd29facc640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:09:58:12] ENGINE Error in HTTPServer.serve 2024-09-21T09:58:12.505 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: Traceback (most recent call last): 2024-09-21T09:58:12.505 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:58:12.506 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: self._connections.run(self.expiration_interval) 2024-09-21T09:58:12.506 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:58:12.506 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: self._run(expiration_interval) 2024-09-21T09:58:12.506 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:58:12.506 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:58:12.506 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:58:12.506 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:58:12.506 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:58:12.506 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: s = self.context.wrap_socket( 2024-09-21T09:58:12.506 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:58:12.506 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: return self.sslsocket_class._create( 2024-09-21T09:58:12.506 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:58:12.506 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: self.do_handshake() 2024-09-21T09:58:12.506 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:58:12.507 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: self._sslobj.do_handshake() 2024-09-21T09:58:12.507 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:58:12.507 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T09:58:12.532 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:58:12.552 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 83886079~1 2024-09-21T09:58:12.561 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:58:12.837 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-mon[29589]: mgrmap e28: y(active, since 1.00974s) 2024-09-21T09:58:12.838 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-mon[29589]: osdmap e196: 8 total, 8 up, 8 in 2024-09-21T09:58:12.838 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-mon[29589]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:58:12.838 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-mon[29589]: pgmap v4: 137 pgs: 137 active+clean; 617 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:12.838 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-mon[41533]: mgrmap e28: y(active, since 1.00974s) 2024-09-21T09:58:12.838 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-mon[41533]: osdmap e196: 8 total, 8 up, 8 in 2024-09-21T09:58:12.838 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-mon[41533]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:58:12.838 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:12 smithi070 ceph-mon[41533]: pgmap v4: 137 pgs: 137 active+clean; 617 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:12.839 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:12 smithi160 ceph-mon[33203]: mgrmap e28: y(active, since 1.00974s) 2024-09-21T09:58:12.839 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:12 smithi160 ceph-mon[33203]: osdmap e196: 8 total, 8 up, 8 in 2024-09-21T09:58:12.839 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:12 smithi160 ceph-mon[33203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:58:12.839 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:12 smithi160 ceph-mon[33203]: pgmap v4: 137 pgs: 137 active+clean; 617 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:13.808 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:13 smithi070 ceph-mon[29589]: [21/Sep/2024:09:58:12] ENGINE Bus STARTING 2024-09-21T09:58:13.808 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:13 smithi070 ceph-mon[29589]: [21/Sep/2024:09:58:12] ENGINE Serving on https://172.21.15.70:7150 2024-09-21T09:58:13.808 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:13 smithi070 ceph-mon[29589]: [21/Sep/2024:09:58:12] ENGINE Error in HTTPServer.serve 2024-09-21T09:58:13.808 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-09-21T09:58:13.808 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:58:13.809 INFO:journalctl@ceph.mon.a.smithi070.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:58:13.809 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:58:13.809 INFO:journalctl@ceph.mon.a.smithi070.stdout: self._run(expiration_interval) 2024-09-21T09:58:13.809 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:58:13.809 INFO:journalctl@ceph.mon.a.smithi070.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:58:13.809 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:58:13.809 INFO:journalctl@ceph.mon.a.smithi070.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:58:13.809 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:58:13.809 INFO:journalctl@ceph.mon.a.smithi070.stdout: s = self.context.wrap_socket( 2024-09-21T09:58:13.809 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:58:13.809 INFO:journalctl@ceph.mon.a.smithi070.stdout: return self.sslsocket_class._create( 2024-09-21T09:58:13.809 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:58:13.809 INFO:journalctl@ceph.mon.a.smithi070.stdout: self.do_handshake() 2024-09-21T09:58:13.809 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:58:13.809 INFO:journalctl@ceph.mon.a.smithi070.stdout: self._sslobj.do_handshake() 2024-09-21T09:58:13.810 INFO:journalctl@ceph.mon.a.smithi070.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:58:13.810 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:13 smithi070 ceph-mon[29589]: [21/Sep/2024:09:58:12] ENGINE Serving on http://172.21.15.70:8765 2024-09-21T09:58:13.810 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:13 smithi070 ceph-mon[29589]: [21/Sep/2024:09:58:12] ENGINE Bus STARTED 2024-09-21T09:58:13.810 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:13 smithi070 ceph-mon[29589]: pgmap v5: 137 pgs: 137 active+clean; 617 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:13.810 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:13 smithi070 ceph-mon[29589]: mgrmap e29: y(active, since 2s) 2024-09-21T09:58:13.810 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:13 smithi070 ceph-mon[29589]: osdmap e197: 8 total, 8 up, 8 in 2024-09-21T09:58:13.810 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:13 smithi070 ceph-mon[41533]: [21/Sep/2024:09:58:12] ENGINE Bus STARTING 2024-09-21T09:58:13.810 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:13 smithi070 ceph-mon[41533]: [21/Sep/2024:09:58:12] ENGINE Serving on https://172.21.15.70:7150 2024-09-21T09:58:13.810 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:13 smithi070 ceph-mon[41533]: [21/Sep/2024:09:58:12] ENGINE Error in HTTPServer.serve 2024-09-21T09:58:13.810 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-09-21T09:58:13.810 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:58:13.810 INFO:journalctl@ceph.mon.c.smithi070.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:58:13.811 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:58:13.811 INFO:journalctl@ceph.mon.c.smithi070.stdout: self._run(expiration_interval) 2024-09-21T09:58:13.811 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:58:13.811 INFO:journalctl@ceph.mon.c.smithi070.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:58:13.811 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:58:13.811 INFO:journalctl@ceph.mon.c.smithi070.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:58:13.811 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:58:13.811 INFO:journalctl@ceph.mon.c.smithi070.stdout: s = self.context.wrap_socket( 2024-09-21T09:58:13.811 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:58:13.811 INFO:journalctl@ceph.mon.c.smithi070.stdout: return self.sslsocket_class._create( 2024-09-21T09:58:13.811 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:58:13.811 INFO:journalctl@ceph.mon.c.smithi070.stdout: self.do_handshake() 2024-09-21T09:58:13.811 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:58:13.812 INFO:journalctl@ceph.mon.c.smithi070.stdout: self._sslobj.do_handshake() 2024-09-21T09:58:13.812 INFO:journalctl@ceph.mon.c.smithi070.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:58:13.812 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:13 smithi070 ceph-mon[41533]: [21/Sep/2024:09:58:12] ENGINE Serving on http://172.21.15.70:8765 2024-09-21T09:58:13.812 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:13 smithi070 ceph-mon[41533]: [21/Sep/2024:09:58:12] ENGINE Bus STARTED 2024-09-21T09:58:13.812 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:13 smithi070 ceph-mon[41533]: pgmap v5: 137 pgs: 137 active+clean; 617 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:13.812 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:13 smithi070 ceph-mon[41533]: mgrmap e29: y(active, since 2s) 2024-09-21T09:58:13.812 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:13 smithi070 ceph-mon[41533]: osdmap e197: 8 total, 8 up, 8 in 2024-09-21T09:58:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:13 smithi160 ceph-mon[33203]: [21/Sep/2024:09:58:12] ENGINE Bus STARTING 2024-09-21T09:58:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:13 smithi160 ceph-mon[33203]: [21/Sep/2024:09:58:12] ENGINE Serving on https://172.21.15.70:7150 2024-09-21T09:58:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:13 smithi160 ceph-mon[33203]: [21/Sep/2024:09:58:12] ENGINE Error in HTTPServer.serve 2024-09-21T09:58:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout: Traceback (most recent call last): 2024-09-21T09:58:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:58:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:58:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:58:13.842 INFO:journalctl@ceph.mon.b.smithi160.stdout: self._run(expiration_interval) 2024-09-21T09:58:13.842 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:58:13.842 INFO:journalctl@ceph.mon.b.smithi160.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:58:13.842 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:58:13.842 INFO:journalctl@ceph.mon.b.smithi160.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:58:13.842 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:58:13.842 INFO:journalctl@ceph.mon.b.smithi160.stdout: s = self.context.wrap_socket( 2024-09-21T09:58:13.842 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:58:13.842 INFO:journalctl@ceph.mon.b.smithi160.stdout: return self.sslsocket_class._create( 2024-09-21T09:58:13.842 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:58:13.842 INFO:journalctl@ceph.mon.b.smithi160.stdout: self.do_handshake() 2024-09-21T09:58:13.842 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:58:13.842 INFO:journalctl@ceph.mon.b.smithi160.stdout: self._sslobj.do_handshake() 2024-09-21T09:58:13.843 INFO:journalctl@ceph.mon.b.smithi160.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:58:13.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:13 smithi160 ceph-mon[33203]: [21/Sep/2024:09:58:12] ENGINE Serving on http://172.21.15.70:8765 2024-09-21T09:58:13.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:13 smithi160 ceph-mon[33203]: [21/Sep/2024:09:58:12] ENGINE Bus STARTED 2024-09-21T09:58:13.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:13 smithi160 ceph-mon[33203]: pgmap v5: 137 pgs: 137 active+clean; 617 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:13.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:13 smithi160 ceph-mon[33203]: mgrmap e29: y(active, since 2s) 2024-09-21T09:58:13.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:13 smithi160 ceph-mon[33203]: osdmap e197: 8 total, 8 up, 8 in 2024-09-21T09:58:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:14 smithi160 ceph-mon[33203]: osdmap e198: 8 total, 8 up, 8 in 2024-09-21T09:58:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:14 smithi160 ceph-mon[33203]: osdmap e199: 8 total, 8 up, 8 in 2024-09-21T09:58:14.846 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:14 smithi070 ceph-mon[29589]: osdmap e198: 8 total, 8 up, 8 in 2024-09-21T09:58:14.846 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:14 smithi070 ceph-mon[29589]: osdmap e199: 8 total, 8 up, 8 in 2024-09-21T09:58:14.846 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:14 smithi070 ceph-mon[41533]: osdmap e198: 8 total, 8 up, 8 in 2024-09-21T09:58:14.846 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:14 smithi070 ceph-mon[41533]: osdmap e199: 8 total, 8 up, 8 in 2024-09-21T09:58:15.739 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 167772160 2024-09-21T09:58:15.750 INFO:tasks.workunit.client.0.smithi070.stdout:write: * 83886080~1000 2024-09-21T09:58:15.782 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:58:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:15 smithi160 ceph-mon[33203]: pgmap v9: 137 pgs: 137 active+clean; 617 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:15 smithi160 ceph-mon[33203]: mgrmap e30: y(active, since 4s) 2024-09-21T09:58:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:15 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:15 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:15.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:15 smithi160 ceph-mon[33203]: osdmap e200: 8 total, 8 up, 8 in 2024-09-21T09:58:15.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:15 smithi070 ceph-mon[29589]: pgmap v9: 137 pgs: 137 active+clean; 617 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:15.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:15 smithi070 ceph-mon[29589]: mgrmap e30: y(active, since 4s) 2024-09-21T09:58:15.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:15 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:15.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:15 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:15.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:15 smithi070 ceph-mon[29589]: osdmap e200: 8 total, 8 up, 8 in 2024-09-21T09:58:15.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:15 smithi070 ceph-mon[41533]: pgmap v9: 137 pgs: 137 active+clean; 617 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:15.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:15 smithi070 ceph-mon[41533]: mgrmap e30: y(active, since 4s) 2024-09-21T09:58:15.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:15 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:15.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:15 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:15.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:15 smithi070 ceph-mon[41533]: osdmap e200: 8 total, 8 up, 8 in 2024-09-21T09:58:15.920 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:58:16.341 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:15 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: [21/Sep/2024:09:58:15] ENGINE Bus STARTING 2024-09-21T09:58:16.341 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:15 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: CherryPy Checker: 2024-09-21T09:58:16.341 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:15 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: The Application mounted at '' has an empty config. 2024-09-21T09:58:16.342 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:15 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: 2024-09-21T09:58:16.342 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:16 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: [21/Sep/2024:09:58:16] ENGINE Serving on http://:::9283 2024-09-21T09:58:16.342 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:16 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: [21/Sep/2024:09:58:16] ENGINE Bus STARTED 2024-09-21T09:58:16.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:16 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:16.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:16 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:16.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:16 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:16.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:16 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:16.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:16 smithi070 ceph-mon[41533]: Standby manager daemon x started 2024-09-21T09:58:16.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:16 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.160:0/728523658' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:58:16.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:16 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.160:0/728523658' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:58:16.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:16 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.160:0/728523658' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:58:16.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:16 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.160:0/728523658' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:58:16.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:16 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:16.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:16 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:16.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:16 smithi070 ceph-mon[41533]: pgmap v11: 137 pgs: 137 active+clean; 629 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.2 MiB/s rd, 3.0 MiB/s wr, 204 op/s 2024-09-21T09:58:16.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:16 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:16.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:16 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:16.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:16 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:16.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:16 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:16.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:16 smithi070 ceph-mon[29589]: Standby manager daemon x started 2024-09-21T09:58:16.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:16 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/728523658' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:58:16.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:16 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/728523658' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:58:16.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:16 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/728523658' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:58:16.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:16 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/728523658' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:58:16.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:16 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:16.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:16 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:16.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:16 smithi070 ceph-mon[29589]: pgmap v11: 137 pgs: 137 active+clean; 629 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.2 MiB/s rd, 3.0 MiB/s wr, 204 op/s 2024-09-21T09:58:17.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:16 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:17.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:16 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:17.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:16 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:17.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:16 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:17.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:16 smithi160 ceph-mon[33203]: Standby manager daemon x started 2024-09-21T09:58:17.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:16 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.160:0/728523658' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:58:17.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:16 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.160:0/728523658' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:58:17.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:16 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.160:0/728523658' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:58:17.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:16 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.160:0/728523658' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:58:17.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:16 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:17.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:16 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:17.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:16 smithi160 ceph-mon[33203]: pgmap v11: 137 pgs: 137 active+clean; 629 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.2 MiB/s rd, 3.0 MiB/s wr, 204 op/s 2024-09-21T09:58:18.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:17 smithi160 ceph-mon[33203]: mgrmap e31: y(active, since 6s), standbys: x 2024-09-21T09:58:18.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:17 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:58:18.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:17 smithi160 ceph-mon[33203]: osdmap e201: 8 total, 8 up, 8 in 2024-09-21T09:58:18.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:17 smithi070 ceph-mon[29589]: mgrmap e31: y(active, since 6s), standbys: x 2024-09-21T09:58:18.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:17 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:58:18.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:17 smithi070 ceph-mon[29589]: osdmap e201: 8 total, 8 up, 8 in 2024-09-21T09:58:18.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:17 smithi070 ceph-mon[41533]: mgrmap e31: y(active, since 6s), standbys: x 2024-09-21T09:58:18.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:17 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:58:18.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:17 smithi070 ceph-mon[41533]: osdmap e201: 8 total, 8 up, 8 in 2024-09-21T09:58:18.591 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:18 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[69245]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:58:18] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:58:19.089 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-09-21T09:58:19.090 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:18 smithi070 ceph-mon[29589]: osdmap e202: 8 total, 8 up, 8 in 2024-09-21T09:58:19.090 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:18 smithi070 ceph-mon[29589]: pgmap v14: 137 pgs: 137 active+clean; 645 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 37 MiB/s rd, 5.8 MiB/s wr, 514 op/s 2024-09-21T09:58:19.090 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:18 smithi070 ceph-mon[41533]: osdmap e202: 8 total, 8 up, 8 in 2024-09-21T09:58:19.090 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:18 smithi070 ceph-mon[41533]: pgmap v14: 137 pgs: 137 active+clean; 645 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 37 MiB/s rd, 5.8 MiB/s wr, 514 op/s 2024-09-21T09:58:19.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:18 smithi160 ceph-mon[33203]: osdmap e202: 8 total, 8 up, 8 in 2024-09-21T09:58:19.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:18 smithi160 ceph-mon[33203]: pgmap v14: 137 pgs: 137 active+clean; 645 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 37 MiB/s rd, 5.8 MiB/s wr, 514 op/s 2024-09-21T09:58:19.174 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.SnapsExpandAfterPrepare (11506 ms) 2024-09-21T09:58:19.174 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.SnapsExpandAfterPrepare2 2024-09-21T09:58:19.209 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:58:19.220 INFO:tasks.workunit.client.0.smithi070.stdout:write: X 41943040~10 2024-09-21T09:58:19.230 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap1 2024-09-21T09:58:19.732 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T09:58:19.732 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (3m) 4s ago 4m 26.8M - 0.25.0 c8568f914cd2 df28d76f1795 2024-09-21T09:58:19.732 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi160 *:3000 running (4m) 3s ago 4m 86.4M - 9.4.7 954c08fa6188 8b9850d5203c 2024-09-21T09:58:19.732 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi070.gansda smithi070 running (3m) 4s ago 3m 16.9M - 18.2.4-871-g6a1bdf85 0adab276c97e 6016b2501c8a 2024-09-21T09:58:19.732 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi160.byyhfz smithi160 running (3m) 3s ago 3m 14.3M - 18.2.4-871-g6a1bdf85 0adab276c97e 93e108d91e65 2024-09-21T09:58:19.732 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi160 *:8443,9283,8765 running (54s) 3s ago 8m 112M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 0f7e42b65de5 2024-09-21T09:58:19.732 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (17s) 4s ago 10m 491M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 e3d390893cff 2024-09-21T09:58:19.732 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (10m) 4s ago 10m 53.0M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e cee8ca713d09 2024-09-21T09:58:19.732 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi160 running (8m) 3s ago 8m 50.5M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e 0fd16436dfa0 2024-09-21T09:58:19.732 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (8m) 4s ago 8m 45.4M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e fc0590f324b5 2024-09-21T09:58:19.732 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (4m) 4s ago 4m 16.0M - 1.5.0 0da6a335fe13 2f7c5b1e36eb 2024-09-21T09:58:19.732 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi160 *:9100 running (4m) 3s ago 4m 15.9M - 1.5.0 0da6a335fe13 e0fa2f0d9975 2024-09-21T09:58:19.733 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (8m) 4s ago 8m 107M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 10580e80f007 2024-09-21T09:58:19.733 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (7m) 4s ago 7m 98.0M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e b7c0d1169d1d 2024-09-21T09:58:19.733 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (7m) 4s ago 7m 95.9M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 58a11b8ddb81 2024-09-21T09:58:19.733 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (6m) 4s ago 6m 76.3M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e aa0c8a732a41 2024-09-21T09:58:19.733 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi160 running (6m) 3s ago 6m 117M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 7325a075c488 2024-09-21T09:58:19.733 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi160 running (6m) 3s ago 6m 85.6M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e f74026345641 2024-09-21T09:58:19.733 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi160 running (5m) 3s ago 5m 122M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 03e6bb54f8b2 2024-09-21T09:58:19.733 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi160 running (5m) 3s ago 5m 106M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e e405b21a6146 2024-09-21T09:58:19.733 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi160 *:9095 running (22s) 3s ago 4m 38.3M - 2.43.0 a07b618ecd1d fb7be2db65a2 2024-09-21T09:58:20.078 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T09:58:20.079 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-09-21T09:58:20.079 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 3 2024-09-21T09:58:20.079 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:58:20.079 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-09-21T09:58:20.079 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T09:58:20.079 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:58:20.079 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-09-21T09:58:20.079 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 8 2024-09-21T09:58:20.079 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:58:20.079 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": { 2024-09-21T09:58:20.079 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 2 2024-09-21T09:58:20.079 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:58:20.080 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-09-21T09:58:20.080 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 13, 2024-09-21T09:58:20.080 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T09:58:20.080 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T09:58:20.080 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T09:58:20.080 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[29589]: osdmap e203: 8 total, 8 up, 8 in 2024-09-21T09:58:20.080 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:20.080 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:20.080 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:58:20.081 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:58:20.081 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[29589]: Updating smithi070:/etc/ceph/ceph.conf 2024-09-21T09:58:20.081 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[29589]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-21T09:58:20.081 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[29589]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:58:20.081 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[29589]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:58:20.081 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[29589]: from='client.24995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:20.081 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[29589]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:58:20.081 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[29589]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:58:20.081 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[29589]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:58:20.081 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[29589]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:58:20.081 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[29589]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:20.081 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:20.081 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:20.081 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:20.082 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:20.082 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:20.082 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[29589]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:58:20.082 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[41533]: osdmap e203: 8 total, 8 up, 8 in 2024-09-21T09:58:20.082 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:20.082 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:20.082 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:58:20.082 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:58:20.082 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[41533]: Updating smithi070:/etc/ceph/ceph.conf 2024-09-21T09:58:20.082 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[41533]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-21T09:58:20.082 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[41533]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:58:20.082 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[41533]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:58:20.083 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[41533]: from='client.24995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:20.083 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[41533]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:58:20.083 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[41533]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:58:20.083 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[41533]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:58:20.083 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[41533]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:58:20.083 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[41533]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:20.083 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:20.083 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:20.083 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:20.083 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:20.083 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:20.083 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:19 smithi070 ceph-mon[41533]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:58:20.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:19 smithi160 ceph-mon[33203]: osdmap e203: 8 total, 8 up, 8 in 2024-09-21T09:58:20.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:19 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:20.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:19 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:20.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:19 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:58:20.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:19 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:58:20.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:19 smithi160 ceph-mon[33203]: Updating smithi070:/etc/ceph/ceph.conf 2024-09-21T09:58:20.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:19 smithi160 ceph-mon[33203]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-21T09:58:20.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:19 smithi160 ceph-mon[33203]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:58:20.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:19 smithi160 ceph-mon[33203]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:58:20.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:19 smithi160 ceph-mon[33203]: from='client.24995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:20.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:19 smithi160 ceph-mon[33203]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:58:20.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:19 smithi160 ceph-mon[33203]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:58:20.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:19 smithi160 ceph-mon[33203]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:58:20.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:19 smithi160 ceph-mon[33203]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:58:20.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:19 smithi160 ceph-mon[33203]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:20.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:19 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:20.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:19 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:20.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:19 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:20.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:19 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:20.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:19 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:20.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:19 smithi160 ceph-mon[33203]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:58:20.366 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T09:58:20.366 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47", 2024-09-21T09:58:20.366 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-09-21T09:58:20.366 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-09-21T09:58:20.366 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [ 2024-09-21T09:58:20.367 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr" 2024-09-21T09:58:20.367 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-09-21T09:58:20.367 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "2/2 daemons upgraded", 2024-09-21T09:58:20.367 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "", 2024-09-21T09:58:20.367 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-09-21T09:58:20.367 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T09:58:20.895 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 41943040 2024-09-21T09:58:20.923 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:58:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:20 smithi160 ceph-mon[33203]: from='client.25007 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:20 smithi160 ceph-mon[33203]: Reconfiguring daemon prometheus.a on smithi160 2024-09-21T09:58:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:20 smithi160 ceph-mon[33203]: osdmap e204: 8 total, 8 up, 8 in 2024-09-21T09:58:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:20 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/2708202778' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:20 smithi160 ceph-mon[33203]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:20 smithi160 ceph-mon[33203]: pgmap v17: 137 pgs: 137 active+clean; 645 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 33 MiB/s rd, 3.0 MiB/s wr, 333 op/s 2024-09-21T09:58:21.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:20 smithi070 ceph-mon[29589]: from='client.25007 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:21.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:20 smithi070 ceph-mon[29589]: Reconfiguring daemon prometheus.a on smithi160 2024-09-21T09:58:21.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:20 smithi070 ceph-mon[29589]: osdmap e204: 8 total, 8 up, 8 in 2024-09-21T09:58:21.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:20 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/2708202778' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:21.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:20 smithi070 ceph-mon[29589]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:21.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:20 smithi070 ceph-mon[29589]: pgmap v17: 137 pgs: 137 active+clean; 645 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 33 MiB/s rd, 3.0 MiB/s wr, 333 op/s 2024-09-21T09:58:21.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:20 smithi070 ceph-mon[41533]: from='client.25007 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:21.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:20 smithi070 ceph-mon[41533]: Reconfiguring daemon prometheus.a on smithi160 2024-09-21T09:58:21.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:20 smithi070 ceph-mon[41533]: osdmap e204: 8 total, 8 up, 8 in 2024-09-21T09:58:21.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:20 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/2708202778' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:21.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:20 smithi070 ceph-mon[41533]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:21.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:20 smithi070 ceph-mon[41533]: pgmap v17: 137 pgs: 137 active+clean; 645 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 33 MiB/s rd, 3.0 MiB/s wr, 333 op/s 2024-09-21T09:58:22.024 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:58:22.034 INFO:tasks.workunit.client.0.smithi070.stdout:write: Y 41943040~5 2024-09-21T09:58:22.042 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:21 smithi160 ceph-mon[33203]: osdmap e205: 8 total, 8 up, 8 in 2024-09-21T09:58:22.132 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:58:22.203 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:58:22.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:21 smithi070 ceph-mon[29589]: osdmap e205: 8 total, 8 up, 8 in 2024-09-21T09:58:22.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:21 smithi070 ceph-mon[41533]: osdmap e205: 8 total, 8 up, 8 in 2024-09-21T09:58:22.986 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.SnapsExpandAfterPrepare2 (3812 ms) 2024-09-21T09:58:22.987 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.SnapsSnapAfterPrepare 2024-09-21T09:58:23.017 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:58:23.026 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 0~8388609 2024-09-21T09:58:23.093 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap1 2024-09-21T09:58:23.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:22 smithi070 ceph-mon[29589]: osdmap e206: 8 total, 8 up, 8 in 2024-09-21T09:58:23.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:22 smithi070 ceph-mon[29589]: pgmap v20: 137 pgs: 137 active+clean; 617 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 32 MiB/s rd, 15 KiB/s wr, 167 op/s 2024-09-21T09:58:23.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:22 smithi070 ceph-mon[41533]: osdmap e206: 8 total, 8 up, 8 in 2024-09-21T09:58:23.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:22 smithi070 ceph-mon[41533]: pgmap v20: 137 pgs: 137 active+clean; 617 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 32 MiB/s rd, 15 KiB/s wr, 167 op/s 2024-09-21T09:58:23.295 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:22 smithi160 ceph-mon[33203]: osdmap e206: 8 total, 8 up, 8 in 2024-09-21T09:58:23.295 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:22 smithi160 ceph-mon[33203]: pgmap v20: 137 pgs: 137 active+clean; 617 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 32 MiB/s rd, 15 KiB/s wr, 167 op/s 2024-09-21T09:58:23.295 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:23 smithi160 systemd[1]: Stopping Ceph prometheus.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:58:23.591 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:23 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:58:23.293Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T09:58:23.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:23 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:58:23.293Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-21T09:58:23.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:23 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:58:23.293Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-21T09:58:23.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:23 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:58:23.293Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T09:58:23.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:23 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:58:23.293Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-21T09:58:23.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:23 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:58:23.293Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-21T09:58:23.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:23 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:58:23.293Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T09:58:23.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:23 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:58:23.293Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-21T09:58:23.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:23 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:58:23.295Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-21T09:58:23.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:23 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:58:23.295Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T09:58:23.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:23 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:58:23.295Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-21T09:58:23.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:23 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[77264]: ts=2024-09-21T09:58:23.295Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-21T09:58:23.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:23 smithi160 podman[79306]: 2024-09-21 09:58:23.303765338 +0000 UTC m=+0.128202439 container died fb7be2db65a2e3668178c1533c0cc5fd68b8287a679bfbf4c02dd66390d03635 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:58:24.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:23 smithi160 ceph-mon[33203]: osdmap e207: 8 total, 8 up, 8 in 2024-09-21T09:58:24.091 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:23 smithi160 podman[79306]: 2024-09-21 09:58:23.890713483 +0000 UTC m=+0.715150594 container cleanup fb7be2db65a2e3668178c1533c0cc5fd68b8287a679bfbf4c02dd66390d03635 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:58:24.091 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:23 smithi160 bash[79306]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T09:58:24.345 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 8388609~8388609 2024-09-21T09:58:24.359 INFO:tasks.workunit.client.0.smithi070.stdout:write: X 0~1000 2024-09-21T09:58:24.359 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 1010~1000 2024-09-21T09:58:24.359 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap2 2024-09-21T09:58:24.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:23 smithi070 ceph-mon[29589]: osdmap e207: 8 total, 8 up, 8 in 2024-09-21T09:58:24.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:23 smithi070 ceph-mon[41533]: osdmap e207: 8 total, 8 up, 8 in 2024-09-21T09:58:24.525 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 podman[79325]: 2024-09-21 09:58:24.134054429 +0000 UTC m=+0.831725519 container remove fb7be2db65a2e3668178c1533c0cc5fd68b8287a679bfbf4c02dd66390d03635 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:58:24.525 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-21T09:58:24.525 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 systemd[1]: Stopped Ceph prometheus.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:58:24.525 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@prometheus.a.service: Consumed 1.124s CPU time. 2024-09-21T09:58:24.525 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 systemd[1]: Starting Ceph prometheus.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:58:24.780 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 podman[79422]: 2024-09-21 09:58:24.4277745 +0000 UTC m=+0.017022401 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-21T09:58:24.780 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 podman[79422]: 2024-09-21 09:58:24.534939934 +0000 UTC m=+0.124187821 container create fa2811145bbba0596267d86d52909ed3fb420f7452a06b802745f7b659e08cf9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:58:24.780 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 podman[79422]: 2024-09-21 09:58:24.778505943 +0000 UTC m=+0.367753834 container init fa2811145bbba0596267d86d52909ed3fb420f7452a06b802745f7b659e08cf9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:58:25.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:24 smithi160 ceph-mon[33203]: osdmap e208: 8 total, 8 up, 8 in 2024-09-21T09:58:25.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:24 smithi160 ceph-mon[33203]: osdmap e209: 8 total, 8 up, 8 in 2024-09-21T09:58:25.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:24 smithi160 ceph-mon[33203]: pgmap v24: 137 pgs: 137 active+clean; 617 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 35 MiB/s rd, 17 KiB/s wr, 184 op/s 2024-09-21T09:58:25.092 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 podman[79422]: 2024-09-21 09:58:24.783692546 +0000 UTC m=+0.372940434 container start fa2811145bbba0596267d86d52909ed3fb420f7452a06b802745f7b659e08cf9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:58:25.092 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:58:24.808Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-21T09:58:25.092 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:58:24.808Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-21T09:58:25.092 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:58:24.808Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi160 (none))" 2024-09-21T09:58:25.092 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:58:24.808Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-21T09:58:25.092 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:58:24.808Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-21T09:58:25.092 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:58:24.809Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-21T09:58:25.092 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:58:24.810Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-21T09:58:25.092 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:58:24.811Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-21T09:58:25.092 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:58:24.811Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-21T09:58:25.092 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:58:24.813Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-21T09:58:25.093 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:58:24.813Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.689µs 2024-09-21T09:58:25.093 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:58:24.813Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-21T09:58:25.093 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:58:24.814Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-09-21T09:58:25.093 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:58:24.821Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-09-21T09:58:25.093 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:58:24.823Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-09-21T09:58:25.093 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:58:24.824Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-09-21T09:58:25.093 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:58:24.824Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=21.268µs wal_replay_duration=10.807504ms wbl_replay_duration=107ns total_replay_duration=10.846816ms 2024-09-21T09:58:25.093 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:58:24.825Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-21T09:58:25.093 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:58:24.825Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-21T09:58:25.093 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:58:24.825Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-21T09:58:25.093 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:58:24.845Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.078583ms db_storage=836ns remote_storage=1.094µs web_handler=449ns query_engine=557ns scrape=426.416µs scrape_sd=114.254µs notify=16.381µs notify_sd=31.338µs rules=19.13597ms tracing=8.287µs 2024-09-21T09:58:25.093 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:58:24.845Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-21T09:58:25.093 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:58:24.846Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-21T09:58:25.093 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 bash[79422]: fa2811145bbba0596267d86d52909ed3fb420f7452a06b802745f7b659e08cf9 2024-09-21T09:58:25.094 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:58:24 smithi160 systemd[1]: Started Ceph prometheus.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:58:25.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:24 smithi070 ceph-mon[29589]: osdmap e208: 8 total, 8 up, 8 in 2024-09-21T09:58:25.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:24 smithi070 ceph-mon[29589]: osdmap e209: 8 total, 8 up, 8 in 2024-09-21T09:58:25.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:24 smithi070 ceph-mon[29589]: pgmap v24: 137 pgs: 137 active+clean; 617 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 35 MiB/s rd, 17 KiB/s wr, 184 op/s 2024-09-21T09:58:25.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:24 smithi070 ceph-mon[41533]: osdmap e208: 8 total, 8 up, 8 in 2024-09-21T09:58:25.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:24 smithi070 ceph-mon[41533]: osdmap e209: 8 total, 8 up, 8 in 2024-09-21T09:58:25.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:24 smithi070 ceph-mon[41533]: pgmap v24: 137 pgs: 137 active+clean; 617 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 35 MiB/s rd, 17 KiB/s wr, 184 op/s 2024-09-21T09:58:26.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:25 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:26.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:25 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:26.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:25 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:58:26.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:25 smithi160 ceph-mon[33203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:58:26.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:25 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:26.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:25 smithi160 ceph-mon[33203]: osdmap e210: 8 total, 8 up, 8 in 2024-09-21T09:58:26.352 INFO:tasks.workunit.client.0.smithi070.stdout:write: X 8388609~1000 2024-09-21T09:58:26.359 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 16777228~1000 2024-09-21T09:58:26.360 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 167772160 2024-09-21T09:58:26.368 INFO:tasks.workunit.client.0.smithi070.stdout:write: 2 83886079~8388609 2024-09-21T09:58:26.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:25 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:26.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:25 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:26.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:25 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:58:26.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:25 smithi070 ceph-mon[29589]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:58:26.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:25 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:26.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:25 smithi070 ceph-mon[29589]: osdmap e210: 8 total, 8 up, 8 in 2024-09-21T09:58:26.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:25 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:26.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:25 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:26.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:25 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:58:26.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:25 smithi070 ceph-mon[41533]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:58:26.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:25 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:26.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:25 smithi070 ceph-mon[41533]: osdmap e210: 8 total, 8 up, 8 in 2024-09-21T09:58:26.395 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap3 2024-09-21T09:58:26.993 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:26 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:58:26.993 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:26 smithi160 ceph-mon[33203]: osdmap e211: 8 total, 8 up, 8 in 2024-09-21T09:58:26.994 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:26 smithi160 ceph-mon[33203]: Upgrade: Updating mgr.x 2024-09-21T09:58:26.994 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:26 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:58:26.994 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:26 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:58:26.994 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:26 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:58:26.994 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:26 smithi160 ceph-mon[33203]: Deploying daemon mgr.x on smithi160 2024-09-21T09:58:26.994 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:26 smithi160 ceph-mon[33203]: pgmap v27: 137 pgs: 137 active+clean; 621 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 111 KiB/s rd, 1.1 MiB/s wr, 134 op/s 2024-09-21T09:58:27.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:26 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:58:27.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:26 smithi070 ceph-mon[41533]: osdmap e211: 8 total, 8 up, 8 in 2024-09-21T09:58:27.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:26 smithi070 ceph-mon[41533]: Upgrade: Updating mgr.x 2024-09-21T09:58:27.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:26 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:58:27.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:26 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:58:27.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:26 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:58:27.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:26 smithi070 ceph-mon[41533]: Deploying daemon mgr.x on smithi160 2024-09-21T09:58:27.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:26 smithi070 ceph-mon[41533]: pgmap v27: 137 pgs: 137 active+clean; 621 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 111 KiB/s rd, 1.1 MiB/s wr, 134 op/s 2024-09-21T09:58:27.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:26 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:58:27.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:26 smithi070 ceph-mon[29589]: osdmap e211: 8 total, 8 up, 8 in 2024-09-21T09:58:27.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:26 smithi070 ceph-mon[29589]: Upgrade: Updating mgr.x 2024-09-21T09:58:27.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:26 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:58:27.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:26 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:58:27.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:26 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:58:27.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:26 smithi070 ceph-mon[29589]: Deploying daemon mgr.x on smithi160 2024-09-21T09:58:27.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:26 smithi070 ceph-mon[29589]: pgmap v27: 137 pgs: 137 active+clean; 621 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 111 KiB/s rd, 1.1 MiB/s wr, 134 op/s 2024-09-21T09:58:28.280 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:28 smithi160 systemd[1]: Stopping Ceph mgr.x for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:58:28.360 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:58:28.382 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 83886079~1 2024-09-21T09:58:28.390 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:58:28.581 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:28 smithi160 ceph-mon[33203]: osdmap e212: 8 total, 8 up, 8 in 2024-09-21T09:58:28.582 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:28 smithi160 podman[79762]: 2024-09-21 09:58:28.278444541 +0000 UTC m=+0.131737798 container died 0f7e42b65de50fb5b62087bb81eb6927b9714c842ecd49f74fe4bcd7f01517bc (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-21T09:58:28.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:28 smithi070 ceph-mon[41533]: osdmap e212: 8 total, 8 up, 8 in 2024-09-21T09:58:28.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:28 smithi070 ceph-mon[29589]: osdmap e212: 8 total, 8 up, 8 in 2024-09-21T09:58:28.841 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:28 smithi160 podman[79762]: 2024-09-21 09:58:28.661905213 +0000 UTC m=+0.515198470 container cleanup 0f7e42b65de50fb5b62087bb81eb6927b9714c842ecd49f74fe4bcd7f01517bc (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T09:58:28.841 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:28 smithi160 bash[79762]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x 2024-09-21T09:58:29.299 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:28 smithi160 podman[79782]: 2024-09-21 09:58:28.921446093 +0000 UTC m=+0.640510077 container remove 0f7e42b65de50fb5b62087bb81eb6927b9714c842ecd49f74fe4bcd7f01517bc (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-21T09:58:29.299 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:28 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-21T09:58:29.299 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:29 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mgr.x.service: Failed with result 'exit-code'. 2024-09-21T09:58:29.299 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:29 smithi160 systemd[1]: Stopped Ceph mgr.x for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:58:29.299 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:29 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mgr.x.service: Consumed 17.928s CPU time. 2024-09-21T09:58:29.299 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:29 smithi160 systemd[1]: Starting Ceph mgr.x for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:58:29.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:29 smithi160 ceph-mon[33203]: osdmap e213: 8 total, 8 up, 8 in 2024-09-21T09:58:29.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:29 smithi160 ceph-mon[33203]: pgmap v30: 137 pgs: 137 active+clean; 673 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 324 KiB/s rd, 12 MiB/s wr, 425 op/s 2024-09-21T09:58:29.591 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:29 smithi160 podman[79912]: 2024-09-21 09:58:29.497371325 +0000 UTC m=+0.103296527 container create d952652a9683dd915b748da0bf7687e8f6d5e67da3c90433a88b3a5358afa4c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-21T09:58:29.591 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:29 smithi160 podman[79912]: 2024-09-21 09:58:29.410352818 +0000 UTC m=+0.016278037 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:58:29.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:29 smithi070 ceph-mon[29589]: osdmap e213: 8 total, 8 up, 8 in 2024-09-21T09:58:29.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:29 smithi070 ceph-mon[29589]: pgmap v30: 137 pgs: 137 active+clean; 673 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 324 KiB/s rd, 12 MiB/s wr, 425 op/s 2024-09-21T09:58:29.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:29 smithi070 ceph-mon[41533]: osdmap e213: 8 total, 8 up, 8 in 2024-09-21T09:58:29.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:29 smithi070 ceph-mon[41533]: pgmap v30: 137 pgs: 137 active+clean; 673 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 324 KiB/s rd, 12 MiB/s wr, 425 op/s 2024-09-21T09:58:30.091 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:29 smithi160 podman[79912]: 2024-09-21 09:58:29.742054007 +0000 UTC m=+0.347979231 container init d952652a9683dd915b748da0bf7687e8f6d5e67da3c90433a88b3a5358afa4c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-21T09:58:30.091 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:29 smithi160 podman[79912]: 2024-09-21 09:58:29.746594205 +0000 UTC m=+0.352519409 container start d952652a9683dd915b748da0bf7687e8f6d5e67da3c90433a88b3a5358afa4c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-21T09:58:30.091 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:29 smithi160 ceph-mgr[79950]: -- 172.21.15.160:0/1004759006 <== mon.1 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55deb4f350e0 con 0x55deb4f13400 2024-09-21T09:58:30.091 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:29 smithi160 bash[79912]: d952652a9683dd915b748da0bf7687e8f6d5e67da3c90433a88b3a5358afa4c8 2024-09-21T09:58:30.092 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:29 smithi160 systemd[1]: Started Ceph mgr.x for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:58:30.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:30 smithi070 ceph-mon[29589]: osdmap e214: 8 total, 8 up, 8 in 2024-09-21T09:58:30.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:30 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:30.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:30 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:30.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:30 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:30.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:30 smithi070 ceph-mon[41533]: osdmap e214: 8 total, 8 up, 8 in 2024-09-21T09:58:30.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:30 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:30.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:30 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:30.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:30 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:30.671 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:30 smithi160 ceph-mon[33203]: osdmap e214: 8 total, 8 up, 8 in 2024-09-21T09:58:30.671 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:30 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:30.672 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:30 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:30.672 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:30 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:31.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:31 smithi160 ceph-mon[33203]: osdmap e215: 8 total, 8 up, 8 in 2024-09-21T09:58:31.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:31 smithi160 ceph-mon[33203]: pgmap v33: 137 pgs: 137 active+clean; 673 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 223 KiB/s rd, 11 MiB/s wr, 303 op/s 2024-09-21T09:58:31.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:31 smithi070 ceph-mon[29589]: osdmap e215: 8 total, 8 up, 8 in 2024-09-21T09:58:31.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:31 smithi070 ceph-mon[29589]: pgmap v33: 137 pgs: 137 active+clean; 673 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 223 KiB/s rd, 11 MiB/s wr, 303 op/s 2024-09-21T09:58:31.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:31 smithi070 ceph-mon[41533]: osdmap e215: 8 total, 8 up, 8 in 2024-09-21T09:58:31.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:31 smithi070 ceph-mon[41533]: pgmap v33: 137 pgs: 137 active+clean; 673 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 223 KiB/s rd, 11 MiB/s wr, 303 op/s 2024-09-21T09:58:31.912 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: after_prepare_snap 2024-09-21T09:58:32.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:32 smithi160 ceph-mon[33203]: osdmap e216: 8 total, 8 up, 8 in 2024-09-21T09:58:32.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:32 smithi070 ceph-mon[41533]: osdmap e216: 8 total, 8 up, 8 in 2024-09-21T09:58:32.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:32 smithi070 ceph-mon[29589]: osdmap e216: 8 total, 8 up, 8 in 2024-09-21T09:58:33.395 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 41943040 2024-09-21T09:58:33.427 INFO:tasks.workunit.client.0.smithi070.stdout:write: * 0~1000 2024-09-21T09:58:33.440 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:58:33.572 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:58:33.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:33 smithi160 ceph-mon[33203]: osdmap e217: 8 total, 8 up, 8 in 2024-09-21T09:58:33.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:33 smithi160 ceph-mon[33203]: pgmap v36: 137 pgs: 137 active+clean; 673 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 12 KiB/s wr, 192 op/s 2024-09-21T09:58:33.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:33 smithi070 ceph-mon[41533]: osdmap e217: 8 total, 8 up, 8 in 2024-09-21T09:58:33.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:33 smithi070 ceph-mon[41533]: pgmap v36: 137 pgs: 137 active+clean; 673 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 12 KiB/s wr, 192 op/s 2024-09-21T09:58:33.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:33 smithi070 ceph-mon[29589]: osdmap e217: 8 total, 8 up, 8 in 2024-09-21T09:58:33.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:33 smithi070 ceph-mon[29589]: pgmap v36: 137 pgs: 137 active+clean; 673 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 12 KiB/s wr, 192 op/s 2024-09-21T09:58:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:34 smithi160 ceph-mon[33203]: osdmap e218: 8 total, 8 up, 8 in 2024-09-21T09:58:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:34 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:34 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:34 smithi160 ceph-mon[33203]: osdmap e219: 8 total, 8 up, 8 in 2024-09-21T09:58:34.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:34 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:58:34] "GET /metrics HTTP/1.1" 200 38721 "" "Prometheus/2.43.0" 2024-09-21T09:58:34.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:34 smithi070 ceph-mon[41533]: osdmap e218: 8 total, 8 up, 8 in 2024-09-21T09:58:34.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:34 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:34.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:34 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:34.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:34 smithi070 ceph-mon[41533]: osdmap e219: 8 total, 8 up, 8 in 2024-09-21T09:58:34.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:34 smithi070 ceph-mon[29589]: osdmap e218: 8 total, 8 up, 8 in 2024-09-21T09:58:34.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:34 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:34.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:34 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:34.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:34 smithi070 ceph-mon[29589]: osdmap e219: 8 total, 8 up, 8 in 2024-09-21T09:58:35.508 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:35 smithi160 ceph-mon[33203]: pgmap v39: 137 pgs: 137 active+clean; 673 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 12 KiB/s wr, 192 op/s 2024-09-21T09:58:35.508 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:35 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:35.508 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:35 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:35.508 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:35 smithi160 ceph-mon[33203]: Standby manager daemon x restarted 2024-09-21T09:58:35.508 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:35 smithi160 ceph-mon[33203]: Standby manager daemon x started 2024-09-21T09:58:35.508 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:35 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.160:0/4130382849' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:58:35.508 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:35 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.160:0/4130382849' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:58:35.508 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:35 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.160:0/4130382849' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:58:35.508 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:35 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.160:0/4130382849' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:58:35.508 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:35 smithi160 ceph-mon[33203]: osdmap e220: 8 total, 8 up, 8 in 2024-09-21T09:58:35.509 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:35 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[79927]: [21/Sep/2024:09:58:35] ENGINE Bus STARTING 2024-09-21T09:58:35.509 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:35 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[79927]: CherryPy Checker: 2024-09-21T09:58:35.509 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:35 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[79927]: The Application mounted at '' has an empty config. 2024-09-21T09:58:35.509 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:35 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[79927]: 2024-09-21T09:58:35.509 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:35 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[79927]: [21/Sep/2024:09:58:35] ENGINE Serving on http://:::9283 2024-09-21T09:58:35.509 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 09:58:35 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[79927]: [21/Sep/2024:09:58:35] ENGINE Bus STARTED 2024-09-21T09:58:35.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:35 smithi070 ceph-mon[41533]: pgmap v39: 137 pgs: 137 active+clean; 673 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 12 KiB/s wr, 192 op/s 2024-09-21T09:58:35.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:35 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:35.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:35 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:35.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:35 smithi070 ceph-mon[41533]: Standby manager daemon x restarted 2024-09-21T09:58:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:35 smithi070 ceph-mon[41533]: Standby manager daemon x started 2024-09-21T09:58:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:35 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.160:0/4130382849' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:58:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:35 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.160:0/4130382849' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:58:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:35 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.160:0/4130382849' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:58:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:35 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.160:0/4130382849' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:58:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:35 smithi070 ceph-mon[41533]: osdmap e220: 8 total, 8 up, 8 in 2024-09-21T09:58:35.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:35 smithi070 ceph-mon[29589]: pgmap v39: 137 pgs: 137 active+clean; 673 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 12 KiB/s wr, 192 op/s 2024-09-21T09:58:35.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:35 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:35.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:35 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:35.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:35 smithi070 ceph-mon[29589]: Standby manager daemon x restarted 2024-09-21T09:58:35.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:35 smithi070 ceph-mon[29589]: Standby manager daemon x started 2024-09-21T09:58:35.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:35 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/4130382849' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:58:35.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:35 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/4130382849' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:58:35.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:35 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/4130382849' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:58:35.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:35 smithi070 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/4130382849' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:58:35.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:35 smithi070 ceph-mon[29589]: osdmap e220: 8 total, 8 up, 8 in 2024-09-21T09:58:36.681 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.SnapsSnapAfterPrepare (13695 ms) 2024-09-21T09:58:36.681 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.SnapsSnapExpandAfterPrepare 2024-09-21T09:58:36.712 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:58:36.721 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 0~8388609 2024-09-21T09:58:36.771 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap1 2024-09-21T09:58:37.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:36 smithi160 ceph-mon[33203]: mgrmap e32: y(active, since 25s), standbys: x 2024-09-21T09:58:37.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:36 smithi160 ceph-mon[33203]: osdmap e221: 8 total, 8 up, 8 in 2024-09-21T09:58:37.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:36 smithi160 ceph-mon[33203]: pgmap v42: 137 pgs: 137 active+clean; 681 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.0 MiB/s rd, 2.0 MiB/s wr, 31 op/s 2024-09-21T09:58:37.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:36 smithi070 ceph-mon[41533]: mgrmap e32: y(active, since 25s), standbys: x 2024-09-21T09:58:37.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:36 smithi070 ceph-mon[41533]: osdmap e221: 8 total, 8 up, 8 in 2024-09-21T09:58:37.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:36 smithi070 ceph-mon[41533]: pgmap v42: 137 pgs: 137 active+clean; 681 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.0 MiB/s rd, 2.0 MiB/s wr, 31 op/s 2024-09-21T09:58:37.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:36 smithi070 ceph-mon[29589]: mgrmap e32: y(active, since 25s), standbys: x 2024-09-21T09:58:37.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:36 smithi070 ceph-mon[29589]: osdmap e221: 8 total, 8 up, 8 in 2024-09-21T09:58:37.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:36 smithi070 ceph-mon[29589]: pgmap v42: 137 pgs: 137 active+clean; 681 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.0 MiB/s rd, 2.0 MiB/s wr, 31 op/s 2024-09-21T09:58:38.360 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 8388609~8388609 2024-09-21T09:58:38.372 INFO:tasks.workunit.client.0.smithi070.stdout:write: X 0~1000 2024-09-21T09:58:38.372 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 1010~1000 2024-09-21T09:58:38.372 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap2 2024-09-21T09:58:38.611 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: osdmap e222: 8 total, 8 up, 8 in 2024-09-21T09:58:38.611 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.611 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.611 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:58:38.611 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:58:38.611 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.611 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:38.611 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:38.611 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.611 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-21T09:58:38.612 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-21T09:58:38.612 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-21T09:58:38.612 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-21T09:58:38.612 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:38.612 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.612 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:38.612 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.612 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:38.612 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.612 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:38.612 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.612 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:38.612 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.612 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:38.613 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.613 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:38.613 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.613 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:38.613 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.613 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: osdmap e222: 8 total, 8 up, 8 in 2024-09-21T09:58:38.613 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.613 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.613 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:58:38.613 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:58:38.613 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.613 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:38.614 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:38.614 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.614 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-21T09:58:38.614 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-21T09:58:38.614 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-21T09:58:38.614 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-21T09:58:38.614 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:38.614 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.614 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:38.614 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.614 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:38.614 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.614 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:38.615 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.615 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:38.615 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.615 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:38.615 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.615 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:38.615 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.615 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:38.615 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: osdmap e222: 8 total, 8 up, 8 in 2024-09-21T09:58:38.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:58:38.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:58:38.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:38.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:38.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-21T09:58:38.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-21T09:58:38.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-21T09:58:38.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-21T09:58:38.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:38.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:38.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:38.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:38.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:38.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:38.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:38.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:38.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:38.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:39.625 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:39 smithi070 ceph-mon[29589]: Upgrade: Setting container_image for all mgr 2024-09-21T09:58:39.625 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:39 smithi070 ceph-mon[29589]: Upgrade: Setting container_image for all crash 2024-09-21T09:58:39.625 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:39 smithi070 ceph-mon[29589]: Upgrade: Setting container_image for all rgw 2024-09-21T09:58:39.625 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:39 smithi070 ceph-mon[29589]: Upgrade: Setting container_image for all rbd-mirror 2024-09-21T09:58:39.625 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:39 smithi070 ceph-mon[29589]: Upgrade: Setting container_image for all cephfs-mirror 2024-09-21T09:58:39.625 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:39 smithi070 ceph-mon[29589]: Upgrade: Setting container_image for all ceph-exporter 2024-09-21T09:58:39.625 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:39 smithi070 ceph-mon[29589]: Upgrade: Setting container_image for all iscsi 2024-09-21T09:58:39.626 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:39 smithi070 ceph-mon[29589]: Upgrade: Setting container_image for all nfs 2024-09-21T09:58:39.626 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:39 smithi070 ceph-mon[29589]: Upgrade: Setting container_image for all nvmeof 2024-09-21T09:58:39.626 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:39 smithi070 ceph-mon[29589]: osdmap e223: 8 total, 8 up, 8 in 2024-09-21T09:58:39.626 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:39 smithi070 ceph-mon[29589]: pgmap v45: 137 pgs: 137 active+clean; 673 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 60 MiB/s rd, 6.0 MiB/s wr, 502 op/s 2024-09-21T09:58:39.626 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:39 smithi070 ceph-mon[41533]: Upgrade: Setting container_image for all mgr 2024-09-21T09:58:39.626 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:39 smithi070 ceph-mon[41533]: Upgrade: Setting container_image for all crash 2024-09-21T09:58:39.626 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:39 smithi070 ceph-mon[41533]: Upgrade: Setting container_image for all rgw 2024-09-21T09:58:39.626 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:39 smithi070 ceph-mon[41533]: Upgrade: Setting container_image for all rbd-mirror 2024-09-21T09:58:39.626 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:39 smithi070 ceph-mon[41533]: Upgrade: Setting container_image for all cephfs-mirror 2024-09-21T09:58:39.626 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:39 smithi070 ceph-mon[41533]: Upgrade: Setting container_image for all ceph-exporter 2024-09-21T09:58:39.626 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:39 smithi070 ceph-mon[41533]: Upgrade: Setting container_image for all iscsi 2024-09-21T09:58:39.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:39 smithi070 ceph-mon[41533]: Upgrade: Setting container_image for all nfs 2024-09-21T09:58:39.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:39 smithi070 ceph-mon[41533]: Upgrade: Setting container_image for all nvmeof 2024-09-21T09:58:39.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:39 smithi070 ceph-mon[41533]: osdmap e223: 8 total, 8 up, 8 in 2024-09-21T09:58:39.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:39 smithi070 ceph-mon[41533]: pgmap v45: 137 pgs: 137 active+clean; 673 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 60 MiB/s rd, 6.0 MiB/s wr, 502 op/s 2024-09-21T09:58:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:39 smithi160 ceph-mon[33203]: Upgrade: Setting container_image for all mgr 2024-09-21T09:58:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:39 smithi160 ceph-mon[33203]: Upgrade: Setting container_image for all crash 2024-09-21T09:58:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:39 smithi160 ceph-mon[33203]: Upgrade: Setting container_image for all rgw 2024-09-21T09:58:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:39 smithi160 ceph-mon[33203]: Upgrade: Setting container_image for all rbd-mirror 2024-09-21T09:58:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:39 smithi160 ceph-mon[33203]: Upgrade: Setting container_image for all cephfs-mirror 2024-09-21T09:58:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:39 smithi160 ceph-mon[33203]: Upgrade: Setting container_image for all ceph-exporter 2024-09-21T09:58:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:39 smithi160 ceph-mon[33203]: Upgrade: Setting container_image for all iscsi 2024-09-21T09:58:39.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:39 smithi160 ceph-mon[33203]: Upgrade: Setting container_image for all nfs 2024-09-21T09:58:39.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:39 smithi160 ceph-mon[33203]: Upgrade: Setting container_image for all nvmeof 2024-09-21T09:58:39.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:39 smithi160 ceph-mon[33203]: osdmap e223: 8 total, 8 up, 8 in 2024-09-21T09:58:39.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:39 smithi160 ceph-mon[33203]: pgmap v45: 137 pgs: 137 active+clean; 673 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 60 MiB/s rd, 6.0 MiB/s wr, 502 op/s 2024-09-21T09:58:40.353 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:40 smithi070 systemd[1]: Stopping Ceph node-exporter.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:58:40.368 INFO:tasks.workunit.client.0.smithi070.stdout:write: X 8388609~1000 2024-09-21T09:58:40.374 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 16777228~1000 2024-09-21T09:58:40.374 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 167772160 2024-09-21T09:58:40.382 INFO:tasks.workunit.client.0.smithi070.stdout:write: 2 83886079~8388609 2024-09-21T09:58:40.400 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap3 2024-09-21T09:58:40.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:40 smithi070 ceph-mon[29589]: osdmap e224: 8 total, 8 up, 8 in 2024-09-21T09:58:40.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:40 smithi070 ceph-mon[41533]: osdmap e224: 8 total, 8 up, 8 in 2024-09-21T09:58:40.627 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:40 smithi070 podman[97503]: 2024-09-21 09:58:40.598739358 +0000 UTC m=+0.343436246 container died 2f7c5b1e36ebf02e4400257f3911224013acf9f8aa91aaac458e35202a04ad1a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T09:58:40.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:40 smithi160 ceph-mon[33203]: osdmap e224: 8 total, 8 up, 8 in 2024-09-21T09:58:41.105 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:40 smithi070 podman[97503]: 2024-09-21 09:58:40.927983372 +0000 UTC m=+0.672680298 container cleanup 2f7c5b1e36ebf02e4400257f3911224013acf9f8aa91aaac458e35202a04ad1a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T09:58:41.105 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:40 smithi070 bash[97503]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a 2024-09-21T09:58:41.357 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:41 smithi070 podman[97520]: 2024-09-21 09:58:41.104051109 +0000 UTC m=+0.509056933 container remove 2f7c5b1e36ebf02e4400257f3911224013acf9f8aa91aaac458e35202a04ad1a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T09:58:41.357 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:41 smithi070 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-21T09:58:41.357 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:41 smithi070 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-21T09:58:41.358 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:41 smithi070 systemd[1]: Stopped Ceph node-exporter.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:58:41.358 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:41 smithi070 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@node-exporter.a.service: Consumed 2.235s CPU time. 2024-09-21T09:58:41.627 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:41 smithi070 systemd[1]: Starting Ceph node-exporter.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:58:41.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:41 smithi070 ceph-mon[29589]: Upgrade: Updating node-exporter.a (1/2) 2024-09-21T09:58:41.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:41 smithi070 ceph-mon[29589]: Deploying daemon node-exporter.a on smithi070 2024-09-21T09:58:41.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:41 smithi070 ceph-mon[29589]: osdmap e225: 8 total, 8 up, 8 in 2024-09-21T09:58:41.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:41 smithi070 ceph-mon[29589]: pgmap v48: 137 pgs: 137 active+clean; 673 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 51 MiB/s rd, 4.0 MiB/s wr, 470 op/s 2024-09-21T09:58:41.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:41 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:58:41.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:41 smithi070 ceph-mon[41533]: Upgrade: Updating node-exporter.a (1/2) 2024-09-21T09:58:41.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:41 smithi070 ceph-mon[41533]: Deploying daemon node-exporter.a on smithi070 2024-09-21T09:58:41.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:41 smithi070 ceph-mon[41533]: osdmap e225: 8 total, 8 up, 8 in 2024-09-21T09:58:41.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:41 smithi070 ceph-mon[41533]: pgmap v48: 137 pgs: 137 active+clean; 673 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 51 MiB/s rd, 4.0 MiB/s wr, 470 op/s 2024-09-21T09:58:41.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:41 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:58:41.762 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:41 smithi160 ceph-mon[33203]: Upgrade: Updating node-exporter.a (1/2) 2024-09-21T09:58:41.762 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:41 smithi160 ceph-mon[33203]: Deploying daemon node-exporter.a on smithi070 2024-09-21T09:58:41.762 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:41 smithi160 ceph-mon[33203]: osdmap e225: 8 total, 8 up, 8 in 2024-09-21T09:58:41.762 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:41 smithi160 ceph-mon[33203]: pgmap v48: 137 pgs: 137 active+clean; 673 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 51 MiB/s rd, 4.0 MiB/s wr, 470 op/s 2024-09-21T09:58:41.762 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:41 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:58:42.117 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:41 smithi070 bash[97654]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-21T09:58:42.117 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:42 smithi070 bash[97654]: Getting image source signatures 2024-09-21T09:58:42.118 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:42 smithi070 bash[97654]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-21T09:58:42.374 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:58:42.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:42 smithi070 ceph-mon[29589]: osdmap e226: 8 total, 8 up, 8 in 2024-09-21T09:58:42.377 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:42 smithi070 bash[97654]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-21T09:58:42.377 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:42 smithi070 bash[97654]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-21T09:58:42.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:42 smithi070 ceph-mon[41533]: osdmap e226: 8 total, 8 up, 8 in 2024-09-21T09:58:42.395 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 83886079~1 2024-09-21T09:58:42.403 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:58:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:42 smithi160 ceph-mon[33203]: osdmap e226: 8 total, 8 up, 8 in 2024-09-21T09:58:43.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:43 smithi070 ceph-mon[41533]: osdmap e227: 8 total, 8 up, 8 in 2024-09-21T09:58:43.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:43 smithi070 ceph-mon[41533]: pgmap v51: 137 pgs: 137 active+clean; 729 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 12 MiB/s wr, 160 op/s 2024-09-21T09:58:43.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:43 smithi070 ceph-mon[29589]: osdmap e227: 8 total, 8 up, 8 in 2024-09-21T09:58:43.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:43 smithi070 ceph-mon[29589]: pgmap v51: 137 pgs: 137 active+clean; 729 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 12 MiB/s wr, 160 op/s 2024-09-21T09:58:43.628 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:43 smithi070 bash[97654]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-21T09:58:43.628 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:43 smithi070 bash[97654]: Writing manifest to image destination 2024-09-21T09:58:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:43 smithi160 ceph-mon[33203]: osdmap e227: 8 total, 8 up, 8 in 2024-09-21T09:58:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:43 smithi160 ceph-mon[33203]: pgmap v51: 137 pgs: 137 active+clean; 729 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 12 MiB/s wr, 160 op/s 2024-09-21T09:58:44.058 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:43 smithi070 podman[97654]: 2024-09-21 09:58:43.739785424 +0000 UTC m=+2.104563338 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-21T09:58:44.058 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:43 smithi070 podman[97654]: 2024-09-21 09:58:43.813022405 +0000 UTC m=+2.177800305 container create 5d5d4b026070e1daa98aaabd67bb75cb9b6117c727068865f51afce896f70b8c (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T09:58:44.058 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 podman[97654]: 2024-09-21 09:58:44.056577377 +0000 UTC m=+2.421355280 container init 5d5d4b026070e1daa98aaabd67bb75cb9b6117c727068865f51afce896f70b8c (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T09:58:44.373 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 podman[97654]: 2024-09-21 09:58:44.061695761 +0000 UTC m=+2.426473659 container start 5d5d4b026070e1daa98aaabd67bb75cb9b6117c727068865f51afce896f70b8c (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T09:58:44.373 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.064Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-21T09:58:44.373 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.065Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-21T09:58:44.373 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.065Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-21T09:58:44.373 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.065Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-21T09:58:44.373 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-21T09:58:44.373 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-21T09:58:44.373 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-21T09:58:44.373 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=arp 2024-09-21T09:58:44.374 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-21T09:58:44.374 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-21T09:58:44.374 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-21T09:58:44.374 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-21T09:58:44.374 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-21T09:58:44.374 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-21T09:58:44.374 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-21T09:58:44.374 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-21T09:58:44.374 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=edac 2024-09-21T09:58:44.375 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-21T09:58:44.375 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-21T09:58:44.375 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-21T09:58:44.375 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-21T09:58:44.375 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-21T09:58:44.375 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-21T09:58:44.375 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-21T09:58:44.375 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-21T09:58:44.375 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-21T09:58:44.375 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-21T09:58:44.375 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-21T09:58:44.375 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-21T09:58:44.375 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-21T09:58:44.376 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-21T09:58:44.376 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-21T09:58:44.376 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-21T09:58:44.376 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=os 2024-09-21T09:58:44.376 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-21T09:58:44.376 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-21T09:58:44.376 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-21T09:58:44.376 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-21T09:58:44.376 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-21T09:58:44.376 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-21T09:58:44.376 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-21T09:58:44.376 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=stat 2024-09-21T09:58:44.376 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-21T09:58:44.377 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-21T09:58:44.377 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-21T09:58:44.377 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=time 2024-09-21T09:58:44.377 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-21T09:58:44.377 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=uname 2024-09-21T09:58:44.377 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-21T09:58:44.377 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-21T09:58:44.377 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-21T09:58:44.377 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-21T09:58:44.377 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a[97730]: ts=2024-09-21T09:58:44.066Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-21T09:58:44.377 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 bash[97654]: 5d5d4b026070e1daa98aaabd67bb75cb9b6117c727068865f51afce896f70b8c 2024-09-21T09:58:44.377 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 09:58:44 smithi070 systemd[1]: Started Ceph node-exporter.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:58:44.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:58:44] "GET /metrics HTTP/1.1" 200 38722 "" "Prometheus/2.43.0" 2024-09-21T09:58:44.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-mon[29589]: osdmap e228: 8 total, 8 up, 8 in 2024-09-21T09:58:44.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:44.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:44.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-mon[29589]: osdmap e229: 8 total, 8 up, 8 in 2024-09-21T09:58:44.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-mon[41533]: osdmap e228: 8 total, 8 up, 8 in 2024-09-21T09:58:44.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:44.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:44.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:44 smithi070 ceph-mon[41533]: osdmap e229: 8 total, 8 up, 8 in 2024-09-21T09:58:44.678 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:44 smithi160 ceph-mon[33203]: osdmap e228: 8 total, 8 up, 8 in 2024-09-21T09:58:44.678 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:44 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:44.678 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:44 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:44.678 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:44 smithi160 ceph-mon[33203]: osdmap e229: 8 total, 8 up, 8 in 2024-09-21T09:58:45.451 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:45 smithi160 ceph-mon[33203]: pgmap v54: 137 pgs: 137 active+clean; 729 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 12 MiB/s wr, 160 op/s 2024-09-21T09:58:45.451 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:45 smithi160 ceph-mon[33203]: osdmap e230: 8 total, 8 up, 8 in 2024-09-21T09:58:45.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:45 smithi070 ceph-mon[29589]: pgmap v54: 137 pgs: 137 active+clean; 729 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 12 MiB/s wr, 160 op/s 2024-09-21T09:58:45.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:45 smithi070 ceph-mon[29589]: osdmap e230: 8 total, 8 up, 8 in 2024-09-21T09:58:45.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:45 smithi070 ceph-mon[41533]: pgmap v54: 137 pgs: 137 active+clean; 729 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 12 MiB/s wr, 160 op/s 2024-09-21T09:58:45.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:45 smithi070 ceph-mon[41533]: osdmap e230: 8 total, 8 up, 8 in 2024-09-21T09:58:45.723 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: after_prepare_snap 2024-09-21T09:58:46.091 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:45 smithi160 systemd[1]: Stopping Ceph node-exporter.b for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:58:46.490 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:46 smithi160 ceph-mon[33203]: Upgrade: Updating node-exporter.b (2/2) 2024-09-21T09:58:46.490 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:46 smithi160 ceph-mon[33203]: Deploying daemon node-exporter.b on smithi160 2024-09-21T09:58:46.490 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:46 smithi160 podman[83694]: 2024-09-21 09:58:46.224372828 +0000 UTC m=+0.094790652 container died e0fa2f0d997571dc65be2942ff3dfe519314ddfbca6146db71abf769f37dc4aa (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:58:46.841 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:46 smithi160 podman[83694]: 2024-09-21 09:58:46.586656317 +0000 UTC m=+0.457074144 container cleanup e0fa2f0d997571dc65be2942ff3dfe519314ddfbca6146db71abf769f37dc4aa (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:58:46.841 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:46 smithi160 bash[83694]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b 2024-09-21T09:58:46.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:46 smithi070 ceph-mon[29589]: Upgrade: Updating node-exporter.b (2/2) 2024-09-21T09:58:46.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:46 smithi070 ceph-mon[29589]: Deploying daemon node-exporter.b on smithi160 2024-09-21T09:58:46.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:46 smithi070 ceph-mon[41533]: Upgrade: Updating node-exporter.b (2/2) 2024-09-21T09:58:46.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:46 smithi070 ceph-mon[41533]: Deploying daemon node-exporter.b on smithi160 2024-09-21T09:58:47.211 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:46 smithi160 podman[83711]: 2024-09-21 09:58:46.875900687 +0000 UTC m=+0.651899701 container remove e0fa2f0d997571dc65be2942ff3dfe519314ddfbca6146db71abf769f37dc4aa (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:58:47.211 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:46 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-21T09:58:47.211 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:47 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-21T09:58:47.211 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:47 smithi160 systemd[1]: Stopped Ceph node-exporter.b for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:58:47.211 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:47 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@node-exporter.b.service: Consumed 2.313s CPU time. 2024-09-21T09:58:47.401 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 167772160 2024-09-21T09:58:47.414 INFO:tasks.workunit.client.0.smithi070.stdout:write: * 83886080~1000 2024-09-21T09:58:47.422 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:58:47.553 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:58:47.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:47 smithi160 ceph-mon[33203]: osdmap e231: 8 total, 8 up, 8 in 2024-09-21T09:58:47.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:47 smithi160 ceph-mon[33203]: pgmap v57: 137 pgs: 137 active+clean; 729 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.1 MiB/s rd, 2.2 KiB/s wr, 146 op/s 2024-09-21T09:58:47.591 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:47 smithi160 systemd[1]: Starting Ceph node-exporter.b for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:58:47.592 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:47 smithi160 bash[83843]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-21T09:58:47.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:47 smithi070 ceph-mon[29589]: osdmap e231: 8 total, 8 up, 8 in 2024-09-21T09:58:47.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:47 smithi070 ceph-mon[29589]: pgmap v57: 137 pgs: 137 active+clean; 729 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.1 MiB/s rd, 2.2 KiB/s wr, 146 op/s 2024-09-21T09:58:47.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:47 smithi070 ceph-mon[41533]: osdmap e231: 8 total, 8 up, 8 in 2024-09-21T09:58:47.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:47 smithi070 ceph-mon[41533]: pgmap v57: 137 pgs: 137 active+clean; 729 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.1 MiB/s rd, 2.2 KiB/s wr, 146 op/s 2024-09-21T09:58:48.091 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:47 smithi160 bash[83843]: Getting image source signatures 2024-09-21T09:58:48.091 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:47 smithi160 bash[83843]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-21T09:58:48.091 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:47 smithi160 bash[83843]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-21T09:58:48.091 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:47 smithi160 bash[83843]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-21T09:58:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:48 smithi160 ceph-mon[33203]: osdmap e232: 8 total, 8 up, 8 in 2024-09-21T09:58:48.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:48 smithi070 ceph-mon[29589]: osdmap e232: 8 total, 8 up, 8 in 2024-09-21T09:58:48.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:48 smithi070 ceph-mon[41533]: osdmap e232: 8 total, 8 up, 8 in 2024-09-21T09:58:49.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:49 smithi160 ceph-mon[33203]: osdmap e233: 8 total, 8 up, 8 in 2024-09-21T09:58:49.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:49 smithi160 ceph-mon[33203]: pgmap v60: 137 pgs: 137 active+clean; 729 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 32 MiB/s rd, 12 KiB/s wr, 279 op/s 2024-09-21T09:58:49.591 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:49 smithi160 bash[83843]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-21T09:58:49.592 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:49 smithi160 bash[83843]: Writing manifest to image destination 2024-09-21T09:58:49.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:49 smithi070 ceph-mon[29589]: osdmap e233: 8 total, 8 up, 8 in 2024-09-21T09:58:49.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:49 smithi070 ceph-mon[29589]: pgmap v60: 137 pgs: 137 active+clean; 729 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 32 MiB/s rd, 12 KiB/s wr, 279 op/s 2024-09-21T09:58:49.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:49 smithi070 ceph-mon[41533]: osdmap e233: 8 total, 8 up, 8 in 2024-09-21T09:58:49.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:49 smithi070 ceph-mon[41533]: pgmap v60: 137 pgs: 137 active+clean; 729 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 32 MiB/s rd, 12 KiB/s wr, 279 op/s 2024-09-21T09:58:50.091 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:49 smithi160 podman[83843]: 2024-09-21 09:58:49.739383395 +0000 UTC m=+2.438633088 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-21T09:58:50.092 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:49 smithi160 podman[83843]: 2024-09-21 09:58:49.837800155 +0000 UTC m=+2.537049830 container create 19b8e0e8ee0c3481a50904ebdbdd7cc113b8158dc83b644d97b69573f5338ede (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:58:50.092 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 podman[83843]: 2024-09-21 09:58:50.042312851 +0000 UTC m=+2.741562525 container init 19b8e0e8ee0c3481a50904ebdbdd7cc113b8158dc83b644d97b69573f5338ede (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:58:50.092 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 podman[83843]: 2024-09-21 09:58:50.047368738 +0000 UTC m=+2.746618414 container start 19b8e0e8ee0c3481a50904ebdbdd7cc113b8158dc83b644d97b69573f5338ede (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:58:50.092 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.050Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-21T09:58:50.092 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.050Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-21T09:58:50.092 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.051Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-21T09:58:50.092 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.051Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-21T09:58:50.092 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.051Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-21T09:58:50.092 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-21T09:58:50.092 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-21T09:58:50.093 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=arp 2024-09-21T09:58:50.093 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-21T09:58:50.093 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-21T09:58:50.093 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-21T09:58:50.093 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-21T09:58:50.094 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-21T09:58:50.094 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-21T09:58:50.094 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-21T09:58:50.094 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-21T09:58:50.094 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=edac 2024-09-21T09:58:50.094 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-21T09:58:50.094 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-21T09:58:50.094 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-21T09:58:50.094 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-21T09:58:50.094 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-21T09:58:50.095 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-21T09:58:50.095 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-21T09:58:50.095 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-21T09:58:50.095 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-21T09:58:50.095 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-21T09:58:50.095 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-21T09:58:50.095 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-21T09:58:50.095 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-21T09:58:50.095 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-21T09:58:50.095 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-21T09:58:50.095 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-21T09:58:50.096 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=os 2024-09-21T09:58:50.096 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-21T09:58:50.096 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-21T09:58:50.096 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-21T09:58:50.096 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-21T09:58:50.096 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-21T09:58:50.096 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-21T09:58:50.096 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-21T09:58:50.096 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=stat 2024-09-21T09:58:50.096 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-21T09:58:50.096 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-21T09:58:50.096 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-21T09:58:50.097 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=time 2024-09-21T09:58:50.097 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-21T09:58:50.097 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=uname 2024-09-21T09:58:50.097 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-21T09:58:50.097 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-21T09:58:50.097 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-21T09:58:50.097 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-21T09:58:50.097 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b[83918]: ts=2024-09-21T09:58:50.052Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-21T09:58:50.591 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 bash[83843]: 19b8e0e8ee0c3481a50904ebdbdd7cc113b8158dc83b644d97b69573f5338ede 2024-09-21T09:58:50.606 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 09:58:50 smithi160 systemd[1]: Started Ceph node-exporter.b for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:58:50.607 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-mon[33203]: osdmap e234: 8 total, 8 up, 8 in 2024-09-21T09:58:50.607 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:50.607 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:50.607 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:50 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:50.736 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:50 smithi070 ceph-mon[29589]: osdmap e234: 8 total, 8 up, 8 in 2024-09-21T09:58:50.736 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:50 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:50.737 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:50 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:50.737 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:50 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:50.737 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:50 smithi070 ceph-mon[41533]: osdmap e234: 8 total, 8 up, 8 in 2024-09-21T09:58:50.737 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:50 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:50.737 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:50 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:50.737 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:50 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:50.737 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-09-21T09:58:50.834 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.SnapsSnapExpandAfterPrepare (14152 ms) 2024-09-21T09:58:50.834 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.Clone 2024-09-21T09:58:50.868 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:58:50.876 INFO:tasks.workunit.client.0.smithi070.stdout:write: X 0~8388609 2024-09-21T09:58:50.932 INFO:tasks.workunit.client.0.smithi070.stdout:write: X 16777218~8388609 2024-09-21T09:58:50.978 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap 2024-09-21T09:58:51.385 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T09:58:51.386 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (4m) 36s ago 5m 26.8M - 0.25.0 c8568f914cd2 df28d76f1795 2024-09-21T09:58:51.386 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi160 *:3000 running (4m) 17s ago 4m 86.4M - 9.4.7 954c08fa6188 8b9850d5203c 2024-09-21T09:58:51.386 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi070.gansda smithi070 running (3m) 36s ago 3m 16.9M - 18.2.4-871-g6a1bdf85 0adab276c97e 6016b2501c8a 2024-09-21T09:58:51.386 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi160.byyhfz smithi160 running (3m) 17s ago 3m 14.5M - 18.2.4-871-g6a1bdf85 0adab276c97e 93e108d91e65 2024-09-21T09:58:51.386 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi160 *:8443,9283,8765 running (21s) 17s ago 8m 63.9M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 d952652a9683 2024-09-21T09:58:51.386 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (49s) 36s ago 10m 491M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 e3d390893cff 2024-09-21T09:58:51.386 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (10m) 36s ago 10m 53.0M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e cee8ca713d09 2024-09-21T09:58:51.386 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi160 running (9m) 17s ago 9m 48.5M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e 0fd16436dfa0 2024-09-21T09:58:51.386 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (9m) 36s ago 9m 45.4M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e fc0590f324b5 2024-09-21T09:58:51.386 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 starting - - - - 2024-09-21T09:58:51.387 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi160 *:9100 starting - - - - 2024-09-21T09:58:51.387 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (8m) 36s ago 8m 107M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 10580e80f007 2024-09-21T09:58:51.387 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (8m) 36s ago 8m 98.0M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e b7c0d1169d1d 2024-09-21T09:58:51.387 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (7m) 36s ago 7m 95.9M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 58a11b8ddb81 2024-09-21T09:58:51.387 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (7m) 36s ago 7m 76.3M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e aa0c8a732a41 2024-09-21T09:58:51.387 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi160 running (7m) 17s ago 7m 110M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 7325a075c488 2024-09-21T09:58:51.387 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi160 running (6m) 17s ago 6m 88.1M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e f74026345641 2024-09-21T09:58:51.387 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi160 running (6m) 17s ago 6m 127M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 03e6bb54f8b2 2024-09-21T09:58:51.387 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi160 running (5m) 17s ago 5m 103M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e e405b21a6146 2024-09-21T09:58:51.387 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi160 *:9095 running (26s) 17s ago 5m 33.2M - 2.43.0 a07b618ecd1d fa2811145bbb 2024-09-21T09:58:51.639 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:51 smithi070 ceph-mon[29589]: osdmap e235: 8 total, 8 up, 8 in 2024-09-21T09:58:51.639 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:51 smithi070 ceph-mon[29589]: pgmap v63: 137 pgs: 137 active+clean; 729 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 26 MiB/s rd, 9.5 KiB/s wr, 133 op/s 2024-09-21T09:58:51.639 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:51 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:51.639 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:51 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:51.639 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:51 smithi070 ceph-mon[41533]: osdmap e235: 8 total, 8 up, 8 in 2024-09-21T09:58:51.639 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:51 smithi070 ceph-mon[41533]: pgmap v63: 137 pgs: 137 active+clean; 729 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 26 MiB/s rd, 9.5 KiB/s wr, 133 op/s 2024-09-21T09:58:51.639 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:51 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:51.640 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:51 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:51.726 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:51 smithi160 ceph-mon[33203]: osdmap e235: 8 total, 8 up, 8 in 2024-09-21T09:58:51.727 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:51 smithi160 ceph-mon[33203]: pgmap v63: 137 pgs: 137 active+clean; 729 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 26 MiB/s rd, 9.5 KiB/s wr, 133 op/s 2024-09-21T09:58:51.727 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:51 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:51.727 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:51 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:51.756 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T09:58:51.757 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-09-21T09:58:51.757 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 3 2024-09-21T09:58:51.757 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:58:51.757 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-09-21T09:58:51.757 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T09:58:51.757 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:58:51.757 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-09-21T09:58:51.757 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 8 2024-09-21T09:58:51.757 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:58:51.757 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": { 2024-09-21T09:58:51.757 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 2 2024-09-21T09:58:51.757 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:58:51.757 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-09-21T09:58:51.757 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 13, 2024-09-21T09:58:51.758 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T09:58:51.758 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T09:58:51.758 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T09:58:52.073 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T09:58:52.073 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47", 2024-09-21T09:58:52.073 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-09-21T09:58:52.073 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-09-21T09:58:52.073 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [ 2024-09-21T09:58:52.073 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr" 2024-09-21T09:58:52.073 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-09-21T09:58:52.074 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "2/2 daemons upgraded", 2024-09-21T09:58:52.074 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading node-exporter daemons", 2024-09-21T09:58:52.074 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-09-21T09:58:52.074 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T09:58:52.550 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap 2024-09-21T09:58:52.562 INFO:tasks.workunit.client.0.smithi070.stdout:clone image105 -> image108 2024-09-21T09:58:52.653 INFO:tasks.workunit.client.0.smithi070.stdout:write: X 0~1000 2024-09-21T09:58:52.711 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 1010~1000 2024-09-21T09:58:52.713 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap 2024-09-21T09:58:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:52 smithi160 ceph-mon[33203]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:52 smithi160 ceph-mon[33203]: from='client.24802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:52 smithi160 ceph-mon[33203]: from='client.25040 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:52 smithi160 ceph-mon[33203]: osdmap e236: 8 total, 8 up, 8 in 2024-09-21T09:58:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:52 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/3802694102' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:52.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:52 smithi070 ceph-mon[29589]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:52.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:52 smithi070 ceph-mon[29589]: from='client.24802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:52.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:52 smithi070 ceph-mon[29589]: from='client.25040 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:52.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:52 smithi070 ceph-mon[29589]: osdmap e236: 8 total, 8 up, 8 in 2024-09-21T09:58:52.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:52 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/3802694102' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:52.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:52 smithi070 ceph-mon[41533]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:52.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:52 smithi070 ceph-mon[41533]: from='client.24802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:52.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:52 smithi070 ceph-mon[41533]: from='client.25040 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:52.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:52 smithi070 ceph-mon[41533]: osdmap e236: 8 total, 8 up, 8 in 2024-09-21T09:58:52.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:52 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/3802694102' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:53.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:53 smithi160 ceph-mon[33203]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:53.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:53 smithi160 ceph-mon[33203]: pgmap v65: 137 pgs: 137 active+clean; 761 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 45 MiB/s rd, 14 MiB/s wr, 418 op/s 2024-09-21T09:58:53.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:53 smithi160 ceph-mon[33203]: osdmap e237: 8 total, 8 up, 8 in 2024-09-21T09:58:53.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:53 smithi070 ceph-mon[29589]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:53.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:53 smithi070 ceph-mon[29589]: pgmap v65: 137 pgs: 137 active+clean; 761 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 45 MiB/s rd, 14 MiB/s wr, 418 op/s 2024-09-21T09:58:53.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:53 smithi070 ceph-mon[29589]: osdmap e237: 8 total, 8 up, 8 in 2024-09-21T09:58:53.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:53 smithi070 ceph-mon[41533]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:53.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:53 smithi070 ceph-mon[41533]: pgmap v65: 137 pgs: 137 active+clean; 761 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 45 MiB/s rd, 14 MiB/s wr, 418 op/s 2024-09-21T09:58:53.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:53 smithi070 ceph-mon[41533]: osdmap e237: 8 total, 8 up, 8 in 2024-09-21T09:58:54.351 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap 2024-09-21T09:58:54.362 INFO:tasks.workunit.client.0.smithi070.stdout:clone image108 -> image110 2024-09-21T09:58:54.473 INFO:tasks.workunit.client.0.smithi070.stdout:write: X 1000~1000 2024-09-21T09:58:54.523 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 2010~1000 2024-09-21T09:58:54.525 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:58:54.823 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:58:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:54 smithi160 ceph-mon[33203]: osdmap e238: 8 total, 8 up, 8 in 2024-09-21T09:58:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:54 smithi160 ceph-mon[33203]: osdmap e239: 8 total, 8 up, 8 in 2024-09-21T09:58:54.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:54 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:54.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:54 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:54.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:54 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:54.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:58:54 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:58:54] "GET /metrics HTTP/1.1" 200 38722 "" "Prometheus/2.43.0" 2024-09-21T09:58:54.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:54 smithi070 ceph-mon[41533]: osdmap e238: 8 total, 8 up, 8 in 2024-09-21T09:58:54.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:54 smithi070 ceph-mon[41533]: osdmap e239: 8 total, 8 up, 8 in 2024-09-21T09:58:54.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:54 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:54.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:54 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:54.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:54 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:54.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:54 smithi070 ceph-mon[29589]: osdmap e238: 8 total, 8 up, 8 in 2024-09-21T09:58:54.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:54 smithi070 ceph-mon[29589]: osdmap e239: 8 total, 8 up, 8 in 2024-09-21T09:58:54.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:54 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:54.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:54 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:54.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:54 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:54.999 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:58:55.170 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.Clone (4335 ms) 2024-09-21T09:58:55.170 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.CloneParent 2024-09-21T09:58:55.198 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:58:55.211 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap 2024-09-21T09:58:55.721 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:55 smithi070 ceph-mon[29589]: pgmap v69: 137 pgs: 137 active+clean; 761 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 46 MiB/s rd, 14 MiB/s wr, 431 op/s 2024-09-21T09:58:55.721 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:55 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:55.721 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:55 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:55.721 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:55 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:55.721 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:55 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:55.721 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:55 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:55.721 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:55 smithi070 ceph-mon[41533]: pgmap v69: 137 pgs: 137 active+clean; 761 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 46 MiB/s rd, 14 MiB/s wr, 431 op/s 2024-09-21T09:58:55.721 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:55 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:55.721 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:55 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:55.721 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:55 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:55.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:55 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:55.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:55 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:55.807 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:55 smithi160 ceph-mon[33203]: pgmap v69: 137 pgs: 137 active+clean; 761 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 46 MiB/s rd, 14 MiB/s wr, 431 op/s 2024-09-21T09:58:55.807 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:55 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:55.807 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:55 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:55.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:55 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:55.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:55 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:55.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:55 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:56.822 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:56 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:58:56.822 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:56 smithi070 ceph-mon[29589]: osdmap e240: 8 total, 8 up, 8 in 2024-09-21T09:58:56.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:56 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:58:56.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:56 smithi070 ceph-mon[41533]: osdmap e240: 8 total, 8 up, 8 in 2024-09-21T09:58:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:56 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:58:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:56 smithi160 ceph-mon[33203]: osdmap e240: 8 total, 8 up, 8 in 2024-09-21T09:58:57.162 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap 2024-09-21T09:58:57.171 INFO:tasks.workunit.client.0.smithi070.stdout:clone image111 -> image114 2024-09-21T09:58:57.261 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:58:57.415 INFO:tasks.workunit.client.0.smithi070.stdout:migration_abort 2024-09-21T09:58:57.549 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.CloneParent (2380 ms) 2024-09-21T09:58:57.549 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.CloneUpdateAfterPrepare 2024-09-21T09:58:57.579 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:58:57.590 INFO:tasks.workunit.client.0.smithi070.stdout:write: X 0~10 2024-09-21T09:58:57.597 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap 2024-09-21T09:58:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:57 smithi160 ceph-mon[33203]: pgmap v71: 137 pgs: 137 active+clean; 761 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.1 MiB/s rd, 1.0 MiB/s wr, 89 op/s 2024-09-21T09:58:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:57 smithi160 ceph-mon[33203]: osdmap e241: 8 total, 8 up, 8 in 2024-09-21T09:58:57.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:57 smithi070 ceph-mon[29589]: pgmap v71: 137 pgs: 137 active+clean; 761 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.1 MiB/s rd, 1.0 MiB/s wr, 89 op/s 2024-09-21T09:58:57.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:57 smithi070 ceph-mon[29589]: osdmap e241: 8 total, 8 up, 8 in 2024-09-21T09:58:57.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:57 smithi070 ceph-mon[41533]: pgmap v71: 137 pgs: 137 active+clean; 761 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.1 MiB/s rd, 1.0 MiB/s wr, 89 op/s 2024-09-21T09:58:57.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:57 smithi070 ceph-mon[41533]: osdmap e241: 8 total, 8 up, 8 in 2024-09-21T09:58:59.566 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap 2024-09-21T09:58:59.577 INFO:tasks.workunit.client.0.smithi070.stdout:clone image115 -> image118 2024-09-21T09:58:59.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:59 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:59.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:59 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:59.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:59 smithi160 ceph-mon[33203]: pgmap v73: 137 pgs: 137 active+clean; 777 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 4.0 MiB/s wr, 663 op/s 2024-09-21T09:58:59.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:59 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:59.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:59 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:59.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:59 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:58:59.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:59 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:58:59.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:59 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:59.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:59 smithi160 ceph-mon[33203]: osdmap e242: 8 total, 8 up, 8 in 2024-09-21T09:58:59.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:59 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:59.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:59 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:59 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:59 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:59 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:59 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:59 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:59 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:59 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:59 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:58:59 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.670 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:58:59.822 INFO:tasks.workunit.client.0.smithi070.stdout:write: Y 0~1 2024-09-21T09:58:59.854 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:58:59.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:59.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:59.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[41533]: pgmap v73: 137 pgs: 137 active+clean; 777 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 4.0 MiB/s wr, 663 op/s 2024-09-21T09:58:59.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:59.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:59.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:58:59.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:58:59.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:59.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[41533]: osdmap e242: 8 total, 8 up, 8 in 2024-09-21T09:58:59.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:59.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:59.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:59.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[29589]: pgmap v73: 137 pgs: 137 active+clean; 777 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 4.0 MiB/s wr, 663 op/s 2024-09-21T09:58:59.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:59.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:59.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:58:59.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:58:59.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:58:59.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[29589]: osdmap e242: 8 total, 8 up, 8 in 2024-09-21T09:58:59.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:59.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:58:59 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:58:59.927 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:59:00.026 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.CloneUpdateAfterPrepare (2477 ms) 2024-09-21T09:59:00.026 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.TriggerAssertSnapcSeq 2024-09-21T09:59:00.054 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:59:00.064 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 41943040~10 2024-09-21T09:59:00.071 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap1 2024-09-21T09:59:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:00 smithi160 ceph-mon[33203]: osdmap e243: 8 total, 8 up, 8 in 2024-09-21T09:59:00.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:00 smithi070 ceph-mon[29589]: osdmap e243: 8 total, 8 up, 8 in 2024-09-21T09:59:00.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:00 smithi070 ceph-mon[41533]: osdmap e243: 8 total, 8 up, 8 in 2024-09-21T09:59:01.580 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 41943041~10 2024-09-21T09:59:01.585 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:59:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:01 smithi160 ceph-mon[33203]: Upgrade: Updating prometheus.a 2024-09-21T09:59:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:01 smithi160 ceph-mon[33203]: pgmap v76: 137 pgs: 137 active+clean; 777 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 4.6 MiB/s wr, 755 op/s 2024-09-21T09:59:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:01 smithi160 ceph-mon[33203]: Deploying daemon prometheus.a on smithi160 2024-09-21T09:59:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:01 smithi160 ceph-mon[33203]: osdmap e244: 8 total, 8 up, 8 in 2024-09-21T09:59:01.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:01 smithi070 ceph-mon[29589]: Upgrade: Updating prometheus.a 2024-09-21T09:59:01.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:01 smithi070 ceph-mon[29589]: pgmap v76: 137 pgs: 137 active+clean; 777 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 4.6 MiB/s wr, 755 op/s 2024-09-21T09:59:01.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:01 smithi070 ceph-mon[29589]: Deploying daemon prometheus.a on smithi160 2024-09-21T09:59:01.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:01 smithi070 ceph-mon[29589]: osdmap e244: 8 total, 8 up, 8 in 2024-09-21T09:59:01.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:01 smithi070 ceph-mon[41533]: Upgrade: Updating prometheus.a 2024-09-21T09:59:01.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:01 smithi070 ceph-mon[41533]: pgmap v76: 137 pgs: 137 active+clean; 777 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 4.6 MiB/s wr, 755 op/s 2024-09-21T09:59:01.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:01 smithi070 ceph-mon[41533]: Deploying daemon prometheus.a on smithi160 2024-09-21T09:59:01.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:01 smithi070 ceph-mon[41533]: osdmap e244: 8 total, 8 up, 8 in 2024-09-21T09:59:02.708 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 41943042~10 2024-09-21T09:59:02.720 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap2 2024-09-21T09:59:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:02 smithi160 ceph-mon[33203]: osdmap e245: 8 total, 8 up, 8 in 2024-09-21T09:59:02.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:02 smithi070 ceph-mon[29589]: osdmap e245: 8 total, 8 up, 8 in 2024-09-21T09:59:02.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:02 smithi070 ceph-mon[41533]: osdmap e245: 8 total, 8 up, 8 in 2024-09-21T09:59:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:03 smithi160 ceph-mon[33203]: pgmap v79: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.0 MiB/s wr, 748 op/s 2024-09-21T09:59:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:03 smithi160 ceph-mon[33203]: osdmap e246: 8 total, 8 up, 8 in 2024-09-21T09:59:03.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:03 smithi070 ceph-mon[29589]: pgmap v79: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.0 MiB/s wr, 748 op/s 2024-09-21T09:59:03.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:03 smithi070 ceph-mon[29589]: osdmap e246: 8 total, 8 up, 8 in 2024-09-21T09:59:03.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:03 smithi070 ceph-mon[41533]: pgmap v79: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.0 MiB/s wr, 748 op/s 2024-09-21T09:59:03.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:03 smithi070 ceph-mon[41533]: osdmap e246: 8 total, 8 up, 8 in 2024-09-21T09:59:04.597 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 41943040 2024-09-21T09:59:04.630 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:59:04.698 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:59:04.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:04 smithi160 ceph-mon[33203]: osdmap e247: 8 total, 8 up, 8 in 2024-09-21T09:59:04.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:04 smithi070 ceph-mon[41533]: osdmap e247: 8 total, 8 up, 8 in 2024-09-21T09:59:04.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:04 smithi070 ceph-mon[29589]: osdmap e247: 8 total, 8 up, 8 in 2024-09-21T09:59:04.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:59:04 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:59:04] "GET /metrics HTTP/1.1" 200 38728 "" "Prometheus/2.43.0" 2024-09-21T09:59:05.717 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.TriggerAssertSnapcSeq (5691 ms) 2024-09-21T09:59:05.717 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.SnapTrimBeforePrepare 2024-09-21T09:59:05.748 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:59:05.758 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 41943040~10 2024-09-21T09:59:05.766 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap1 2024-09-21T09:59:05.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:05 smithi070 ceph-mon[29589]: pgmap v82: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.0 MiB/s wr, 748 op/s 2024-09-21T09:59:05.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:05 smithi070 ceph-mon[29589]: osdmap e248: 8 total, 8 up, 8 in 2024-09-21T09:59:05.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:05 smithi070 ceph-mon[41533]: pgmap v82: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.0 MiB/s wr, 748 op/s 2024-09-21T09:59:05.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:05 smithi070 ceph-mon[41533]: osdmap e248: 8 total, 8 up, 8 in 2024-09-21T09:59:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:05 smithi160 ceph-mon[33203]: pgmap v82: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.0 MiB/s wr, 748 op/s 2024-09-21T09:59:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:05 smithi160 ceph-mon[33203]: osdmap e248: 8 total, 8 up, 8 in 2024-09-21T09:59:06.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:06 smithi070 ceph-mon[29589]: osdmap e249: 8 total, 8 up, 8 in 2024-09-21T09:59:06.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:06 smithi070 ceph-mon[41533]: osdmap e249: 8 total, 8 up, 8 in 2024-09-21T09:59:07.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:06 smithi160 ceph-mon[33203]: osdmap e249: 8 total, 8 up, 8 in 2024-09-21T09:59:07.620 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 41943040 2024-09-21T09:59:07.641 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:59:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:07 smithi160 ceph-mon[33203]: pgmap v85: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 767 B/s wr, 5 op/s 2024-09-21T09:59:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:07 smithi160 ceph-mon[33203]: osdmap e250: 8 total, 8 up, 8 in 2024-09-21T09:59:07.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:07 smithi070 ceph-mon[29589]: pgmap v85: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 767 B/s wr, 5 op/s 2024-09-21T09:59:07.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:07 smithi070 ceph-mon[29589]: osdmap e250: 8 total, 8 up, 8 in 2024-09-21T09:59:07.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:07 smithi070 ceph-mon[41533]: pgmap v85: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 767 B/s wr, 5 op/s 2024-09-21T09:59:07.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:07 smithi070 ceph-mon[41533]: osdmap e250: 8 total, 8 up, 8 in 2024-09-21T09:59:08.730 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:59:08.741 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap3 2024-09-21T09:59:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:08 smithi160 ceph-mon[33203]: osdmap e251: 8 total, 8 up, 8 in 2024-09-21T09:59:08.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:08 smithi160 ceph-mon[33203]: pgmap v88: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 256 KiB/s rd, 23 KiB/s wr, 347 op/s 2024-09-21T09:59:08.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:08 smithi070 ceph-mon[29589]: osdmap e251: 8 total, 8 up, 8 in 2024-09-21T09:59:08.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:08 smithi070 ceph-mon[29589]: pgmap v88: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 256 KiB/s rd, 23 KiB/s wr, 347 op/s 2024-09-21T09:59:08.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:08 smithi070 ceph-mon[41533]: osdmap e251: 8 total, 8 up, 8 in 2024-09-21T09:59:08.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:08 smithi070 ceph-mon[41533]: pgmap v88: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 256 KiB/s rd, 23 KiB/s wr, 347 op/s 2024-09-21T09:59:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:09 smithi160 ceph-mon[33203]: osdmap e252: 8 total, 8 up, 8 in 2024-09-21T09:59:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:09 smithi160 ceph-mon[33203]: osdmap e253: 8 total, 8 up, 8 in 2024-09-21T09:59:09.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:09 smithi070 ceph-mon[29589]: osdmap e252: 8 total, 8 up, 8 in 2024-09-21T09:59:09.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:09 smithi070 ceph-mon[29589]: osdmap e253: 8 total, 8 up, 8 in 2024-09-21T09:59:09.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:09 smithi070 ceph-mon[41533]: osdmap e252: 8 total, 8 up, 8 in 2024-09-21T09:59:09.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:09 smithi070 ceph-mon[41533]: osdmap e253: 8 total, 8 up, 8 in 2024-09-21T09:59:10.355 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 41943040~10 2024-09-21T09:59:10.361 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap4 2024-09-21T09:59:11.091 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:10 smithi160 systemd[1]: Stopping Ceph prometheus.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:59:11.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:11 smithi070 ceph-mon[29589]: osdmap e254: 8 total, 8 up, 8 in 2024-09-21T09:59:11.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:11 smithi070 ceph-mon[29589]: pgmap v92: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:11.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:11 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:59:11.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:11 smithi070 ceph-mon[41533]: osdmap e254: 8 total, 8 up, 8 in 2024-09-21T09:59:11.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:11 smithi070 ceph-mon[41533]: pgmap v92: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:11.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:11 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:59:11.649 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:11 smithi160 ceph-mon[33203]: osdmap e254: 8 total, 8 up, 8 in 2024-09-21T09:59:11.649 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:11 smithi160 ceph-mon[33203]: pgmap v92: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:11.649 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:11 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:59:11.649 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:11 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:59:11.418Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T09:59:11.649 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:11 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:59:11.419Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-21T09:59:11.649 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:11 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:59:11.419Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-21T09:59:11.650 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:11 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:59:11.419Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T09:59:11.650 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:11 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:59:11.419Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-21T09:59:11.650 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:11 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:59:11.419Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-21T09:59:11.650 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:11 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:59:11.419Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T09:59:11.650 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:11 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:59:11.419Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-21T09:59:11.650 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:11 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:59:11.419Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-21T09:59:11.650 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:11 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:59:11.419Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T09:59:11.650 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:11 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:59:11.419Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-21T09:59:11.650 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:11 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[79437]: ts=2024-09-21T09:59:11.419Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-21T09:59:11.650 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:11 smithi160 podman[87827]: 2024-09-21 09:59:11.429475406 +0000 UTC m=+0.560119711 container died fa2811145bbba0596267d86d52909ed3fb420f7452a06b802745f7b659e08cf9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:59:11.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:11 smithi160 podman[87827]: 2024-09-21 09:59:11.817663788 +0000 UTC m=+0.948308097 container cleanup fa2811145bbba0596267d86d52909ed3fb420f7452a06b802745f7b659e08cf9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:59:11.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:11 smithi160 bash[87827]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T09:59:12.244 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:12 smithi160 podman[87846]: 2024-09-21 09:59:12.096606775 +0000 UTC m=+0.664494807 container remove fa2811145bbba0596267d86d52909ed3fb420f7452a06b802745f7b659e08cf9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:59:12.367 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 41943040 2024-09-21T09:59:12.388 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:59:12.448 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:59:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:12 smithi160 ceph-mon[33203]: osdmap e255: 8 total, 8 up, 8 in 2024-09-21T09:59:12.591 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:12 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-21T09:59:12.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:12 smithi160 systemd[1]: Stopped Ceph prometheus.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:59:12.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:12 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@prometheus.a.service: Consumed 1.216s CPU time. 2024-09-21T09:59:12.592 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:12 smithi160 systemd[1]: Starting Ceph prometheus.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:59:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:12 smithi070 ceph-mon[29589]: osdmap e255: 8 total, 8 up, 8 in 2024-09-21T09:59:12.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:12 smithi070 ceph-mon[41533]: osdmap e255: 8 total, 8 up, 8 in 2024-09-21T09:59:12.943 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:12 smithi160 podman[87986]: 2024-09-21 09:59:12.588538742 +0000 UTC m=+0.017082516 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-21T09:59:12.943 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:12 smithi160 podman[87986]: 2024-09-21 09:59:12.794981247 +0000 UTC m=+0.223525005 container create 648952e57ad7c6f8ab76d6b076f2f02f51e6d413608e94503b4161582b451c94 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:59:13.341 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 podman[87986]: 2024-09-21 09:59:13.047979671 +0000 UTC m=+0.476523446 container init 648952e57ad7c6f8ab76d6b076f2f02f51e6d413608e94503b4161582b451c94 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:59:13.342 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 podman[87986]: 2024-09-21 09:59:13.053183858 +0000 UTC m=+0.481727614 container start 648952e57ad7c6f8ab76d6b076f2f02f51e6d413608e94503b4161582b451c94 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:59:13.342 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T09:59:13.074Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-21T09:59:13.342 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T09:59:13.074Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-21T09:59:13.342 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T09:59:13.074Z caller=main.go:623 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi160 (none))" 2024-09-21T09:59:13.342 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T09:59:13.074Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-21T09:59:13.342 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T09:59:13.074Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-21T09:59:13.342 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T09:59:13.076Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-21T09:59:13.342 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T09:59:13.077Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-21T09:59:13.342 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T09:59:13.079Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-21T09:59:13.342 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T09:59:13.079Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-21T09:59:13.342 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T09:59:13.080Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-21T09:59:13.342 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T09:59:13.080Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.712µs 2024-09-21T09:59:13.343 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T09:59:13.080Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-21T09:59:13.343 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T09:59:13.082Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=4 2024-09-21T09:59:13.343 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T09:59:13.088Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=4 2024-09-21T09:59:13.343 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T09:59:13.089Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=4 2024-09-21T09:59:13.343 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T09:59:13.090Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=4 2024-09-21T09:59:13.343 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T09:59:13.090Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=4 2024-09-21T09:59:13.343 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T09:59:13.090Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=26.186µs wal_replay_duration=9.551076ms wbl_replay_duration=190ns total_replay_duration=9.596129ms 2024-09-21T09:59:13.343 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T09:59:13.092Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-21T09:59:13.343 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T09:59:13.092Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-21T09:59:13.343 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T09:59:13.092Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-21T09:59:13.343 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T09:59:13.108Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.818627ms db_storage=854ns remote_storage=1.235µs web_handler=410ns query_engine=549ns scrape=414.922µs scrape_sd=98.574µs notify=9.892µs notify_sd=8.056µs rules=14.938565ms tracing=5.196µs 2024-09-21T09:59:13.343 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T09:59:13.108Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-21T09:59:13.343 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T09:59:13.108Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-21T09:59:13.344 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 bash[87986]: 648952e57ad7c6f8ab76d6b076f2f02f51e6d413608e94503b4161582b451c94 2024-09-21T09:59:13.344 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 09:59:13 smithi160 systemd[1]: Started Ceph prometheus.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:59:13.506 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.SnapTrimBeforePrepare (7789 ms) 2024-09-21T09:59:13.507 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.AbortInUseImage 2024-09-21T09:59:13.536 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:59:13.546 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:59:13.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:13 smithi070 ceph-mon[29589]: osdmap e256: 8 total, 8 up, 8 in 2024-09-21T09:59:13.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:13 smithi070 ceph-mon[29589]: pgmap v95: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 140 KiB/s rd, 15 KiB/s wr, 187 op/s 2024-09-21T09:59:13.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:13 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:13.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:13 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:13.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:13 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:59:13.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:13 smithi070 ceph-mon[41533]: osdmap e256: 8 total, 8 up, 8 in 2024-09-21T09:59:13.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:13 smithi070 ceph-mon[41533]: pgmap v95: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 140 KiB/s rd, 15 KiB/s wr, 187 op/s 2024-09-21T09:59:13.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:13 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:13.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:13 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:13.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:13 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:59:13.710 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.AbortInUseImage (204 ms) 2024-09-21T09:59:13.711 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.AbortWithoutSnapshots 2024-09-21T09:59:13.741 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:59:13.751 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 0~8388609 2024-09-21T09:59:13.804 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 16777218~8388609 2024-09-21T09:59:13.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-mon[33203]: osdmap e256: 8 total, 8 up, 8 in 2024-09-21T09:59:13.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-mon[33203]: pgmap v95: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 140 KiB/s rd, 15 KiB/s wr, 187 op/s 2024-09-21T09:59:13.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:13.820 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:13.820 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:13 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:59:13.847 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:59:14.024 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 0~8388609 2024-09-21T09:59:14.066 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 16777218~8388609 2024-09-21T09:59:14.108 INFO:tasks.workunit.client.0.smithi070.stdout:migration_abort 2024-09-21T09:59:14.280 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.AbortWithoutSnapshots (570 ms) 2024-09-21T09:59:14.281 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.AbortWithSnapshots 2024-09-21T09:59:14.311 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:59:14.321 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 0~8388609 2024-09-21T09:59:14.389 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap1 2024-09-21T09:59:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:14 smithi070 ceph-mon[29589]: osdmap e257: 8 total, 8 up, 8 in 2024-09-21T09:59:14.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:14 smithi070 ceph-mon[41533]: osdmap e257: 8 total, 8 up, 8 in 2024-09-21T09:59:14.766 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:14 smithi160 ceph-mon[33203]: osdmap e257: 8 total, 8 up, 8 in 2024-09-21T09:59:15.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:15 smithi070 ceph-mon[29589]: pgmap v97: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 131 KiB/s rd, 14 KiB/s wr, 175 op/s 2024-09-21T09:59:15.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:15 smithi070 ceph-mon[41533]: pgmap v97: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 131 KiB/s rd, 14 KiB/s wr, 175 op/s 2024-09-21T09:59:15.744 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:15 smithi160 ceph-mon[33203]: pgmap v97: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 131 KiB/s rd, 14 KiB/s wr, 175 op/s 2024-09-21T09:59:16.388 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 8388609~8388609 2024-09-21T09:59:16.400 INFO:tasks.workunit.client.0.smithi070.stdout:write: X 0~1000 2024-09-21T09:59:16.401 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 1010~1000 2024-09-21T09:59:16.401 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap2 2024-09-21T09:59:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:16 smithi160 ceph-mon[33203]: osdmap e258: 8 total, 8 up, 8 in 2024-09-21T09:59:16.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:16 smithi070 ceph-mon[29589]: osdmap e258: 8 total, 8 up, 8 in 2024-09-21T09:59:16.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:16 smithi070 ceph-mon[41533]: osdmap e258: 8 total, 8 up, 8 in 2024-09-21T09:59:17.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:17 smithi160 ceph-mon[33203]: osdmap e259: 8 total, 8 up, 8 in 2024-09-21T09:59:17.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:17 smithi160 ceph-mon[33203]: pgmap v100: 137 pgs: 137 active+clean; 793 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 4.9 MiB/s rd, 4.8 MiB/s wr, 121 op/s 2024-09-21T09:59:17.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:17 smithi070 ceph-mon[29589]: osdmap e259: 8 total, 8 up, 8 in 2024-09-21T09:59:17.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:17 smithi070 ceph-mon[29589]: pgmap v100: 137 pgs: 137 active+clean; 793 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 4.9 MiB/s rd, 4.8 MiB/s wr, 121 op/s 2024-09-21T09:59:17.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:17 smithi070 ceph-mon[41533]: osdmap e259: 8 total, 8 up, 8 in 2024-09-21T09:59:17.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:17 smithi070 ceph-mon[41533]: pgmap v100: 137 pgs: 137 active+clean; 793 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 4.9 MiB/s rd, 4.8 MiB/s wr, 121 op/s 2024-09-21T09:59:18.401 INFO:tasks.workunit.client.0.smithi070.stdout:write: X 8388609~1000 2024-09-21T09:59:18.408 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 16777228~1000 2024-09-21T09:59:18.408 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 167772160 2024-09-21T09:59:18.417 INFO:tasks.workunit.client.0.smithi070.stdout:write: 2 83886079~8388609 2024-09-21T09:59:18.431 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap3 2024-09-21T09:59:18.673 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:18 smithi160 ceph-mon[33203]: osdmap e260: 8 total, 8 up, 8 in 2024-09-21T09:59:18.673 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:18 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:18.673 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:18 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:18.674 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:18 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:18.674 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:18 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:18.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:18 smithi070 ceph-mon[41533]: osdmap e260: 8 total, 8 up, 8 in 2024-09-21T09:59:18.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:18 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:18.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:18 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:18.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:18 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:18.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:18 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:18.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:18 smithi070 ceph-mon[29589]: osdmap e260: 8 total, 8 up, 8 in 2024-09-21T09:59:18.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:18 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:18.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:18 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:18.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:18 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:18.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:18 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:19.714 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:19 smithi160 ceph-mon[33203]: osdmap e261: 8 total, 8 up, 8 in 2024-09-21T09:59:19.714 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:19 smithi160 ceph-mon[33203]: pgmap v103: 137 pgs: 137 active+clean; 845 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 29 MiB/s wr, 647 op/s 2024-09-21T09:59:19.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:19 smithi070 ceph-mon[29589]: osdmap e261: 8 total, 8 up, 8 in 2024-09-21T09:59:19.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:19 smithi070 ceph-mon[29589]: pgmap v103: 137 pgs: 137 active+clean; 845 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 29 MiB/s wr, 647 op/s 2024-09-21T09:59:19.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:19 smithi070 ceph-mon[41533]: osdmap e261: 8 total, 8 up, 8 in 2024-09-21T09:59:19.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:19 smithi070 ceph-mon[41533]: pgmap v103: 137 pgs: 137 active+clean; 845 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 29 MiB/s wr, 647 op/s 2024-09-21T09:59:20.410 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:59:20.431 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 83886079~1 2024-09-21T09:59:20.439 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:59:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:20 smithi160 ceph-mon[33203]: osdmap e262: 8 total, 8 up, 8 in 2024-09-21T09:59:20.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:20 smithi070 ceph-mon[29589]: osdmap e262: 8 total, 8 up, 8 in 2024-09-21T09:59:20.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:20 smithi070 ceph-mon[41533]: osdmap e262: 8 total, 8 up, 8 in 2024-09-21T09:59:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:21 smithi160 ceph-mon[33203]: osdmap e263: 8 total, 8 up, 8 in 2024-09-21T09:59:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:21 smithi160 ceph-mon[33203]: pgmap v106: 137 pgs: 137 active+clean; 845 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 24 MiB/s wr, 520 op/s 2024-09-21T09:59:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:21 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:21 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:21 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:59:21.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:21 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:59:21.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:21 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:21.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:21 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:59:21.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:21 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:59:21.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:21 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:21.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:21 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:21.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:21 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:21.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:21 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:21.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:21 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:21.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:21 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:21.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:21 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:21.747 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[29589]: osdmap e263: 8 total, 8 up, 8 in 2024-09-21T09:59:21.747 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[29589]: pgmap v106: 137 pgs: 137 active+clean; 845 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 24 MiB/s wr, 520 op/s 2024-09-21T09:59:21.747 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:21.747 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:21.747 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:59:21.747 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:59:21.747 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:21.747 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:59:21.747 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:59:21.747 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:21.747 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:21.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:21.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:21.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:21.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:21.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:21.748 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[41533]: osdmap e263: 8 total, 8 up, 8 in 2024-09-21T09:59:21.748 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[41533]: pgmap v106: 137 pgs: 137 active+clean; 845 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 24 MiB/s wr, 520 op/s 2024-09-21T09:59:21.748 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:21.748 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:21.748 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:59:21.748 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:59:21.749 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:21.749 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:59:21.749 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:59:21.749 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:21.749 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:21.749 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:21.749 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:21.749 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:21.749 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:21.749 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:21 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:22.395 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-09-21T09:59:22.768 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:22 smithi070 ceph-mon[41533]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:59:22.768 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:22 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:22.769 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:22 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:22.769 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:22 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:22.769 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:22 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:22.769 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:22 smithi070 ceph-mon[41533]: osdmap e264: 8 total, 8 up, 8 in 2024-09-21T09:59:22.770 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:22 smithi070 ceph-mon[29589]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:59:22.770 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:22 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:22.770 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:22 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:22.770 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:22 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:22.770 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:22 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:22.771 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:22 smithi070 ceph-mon[29589]: osdmap e264: 8 total, 8 up, 8 in 2024-09-21T09:59:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:22 smithi160 ceph-mon[33203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:59:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:22 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:22 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:22.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:22 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:22.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:22 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:22.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:22 smithi160 ceph-mon[33203]: osdmap e264: 8 total, 8 up, 8 in 2024-09-21T09:59:22.996 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T09:59:22.996 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (4m) 28s ago 5m 25.2M - 0.25.0 c8568f914cd2 df28d76f1795 2024-09-21T09:59:22.996 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi160 *:3000 running (5m) 5s ago 5m 86.6M - 9.4.7 954c08fa6188 8b9850d5203c 2024-09-21T09:59:22.996 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi070.gansda smithi070 running (4m) 28s ago 4m 17.5M - 18.2.4-871-g6a1bdf85 0adab276c97e 6016b2501c8a 2024-09-21T09:59:22.996 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi160.byyhfz smithi160 running (4m) 5s ago 4m 14.9M - 18.2.4-871-g6a1bdf85 0adab276c97e 93e108d91e65 2024-09-21T09:59:22.996 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi160 *:8443,9283,8765 running (53s) 5s ago 9m 453M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 d952652a9683 2024-09-21T09:59:22.997 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (80s) 28s ago 11m 512M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 e3d390893cff 2024-09-21T09:59:22.997 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (11m) 28s ago 11m 55.7M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e cee8ca713d09 2024-09-21T09:59:22.997 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi160 running (9m) 5s ago 9m 53.0M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e 0fd16436dfa0 2024-09-21T09:59:22.997 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (9m) 28s ago 9m 45.2M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e fc0590f324b5 2024-09-21T09:59:22.997 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (39s) 28s ago 5m 7306k - 1.7.0 72c9c2088986 5d5d4b026070 2024-09-21T09:59:22.997 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi160 *:9100 running (33s) 5s ago 5m 11.9M - 1.7.0 72c9c2088986 19b8e0e8ee0c 2024-09-21T09:59:22.997 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (9m) 28s ago 9m 120M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 10580e80f007 2024-09-21T09:59:22.997 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (8m) 28s ago 8m 96.3M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e b7c0d1169d1d 2024-09-21T09:59:22.997 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (8m) 28s ago 8m 99.7M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 58a11b8ddb81 2024-09-21T09:59:22.997 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (8m) 28s ago 8m 99.0M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e aa0c8a732a41 2024-09-21T09:59:22.997 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi160 running (7m) 5s ago 7m 112M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 7325a075c488 2024-09-21T09:59:22.997 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi160 running (7m) 5s ago 7m 88.9M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e f74026345641 2024-09-21T09:59:22.997 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi160 running (6m) 5s ago 6m 131M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 03e6bb54f8b2 2024-09-21T09:59:22.997 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi160 running (6m) 5s ago 6m 107M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e e405b21a6146 2024-09-21T09:59:22.998 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi160 *:9095 running (10s) 5s ago 5m 32.9M - 2.51.0 1d3b7f56885b 648952e57ad7 2024-09-21T09:59:23.407 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T09:59:23.407 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-09-21T09:59:23.407 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 3 2024-09-21T09:59:23.407 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:59:23.407 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-09-21T09:59:23.407 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T09:59:23.407 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:59:23.407 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-09-21T09:59:23.407 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 8 2024-09-21T09:59:23.407 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:59:23.408 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": { 2024-09-21T09:59:23.408 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 2 2024-09-21T09:59:23.408 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:59:23.408 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-09-21T09:59:23.408 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 13, 2024-09-21T09:59:23.408 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T09:59:23.408 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T09:59:23.408 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T09:59:23.408 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:59:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:59:23] "GET /metrics HTTP/1.1" 200 38726 "" "Prometheus/2.51.0" 2024-09-21T09:59:23.740 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T09:59:23.740 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47", 2024-09-21T09:59:23.740 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-09-21T09:59:23.741 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-09-21T09:59:23.741 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [ 2024-09-21T09:59:23.741 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr" 2024-09-21T09:59:23.741 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-09-21T09:59:23.741 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "2/2 daemons upgraded", 2024-09-21T09:59:23.741 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading alertmanager daemons", 2024-09-21T09:59:23.741 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-09-21T09:59:23.741 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T09:59:23.741 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:23 smithi070 ceph-mon[29589]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:23.741 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:23 smithi070 ceph-mon[29589]: osdmap e265: 8 total, 8 up, 8 in 2024-09-21T09:59:23.741 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:23 smithi070 ceph-mon[29589]: pgmap v109: 137 pgs: 137 active+clean; 869 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 170 op/s 2024-09-21T09:59:23.741 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:23 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/1562524679' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:23.742 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:23 smithi070 ceph-mon[41533]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:23.742 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:23 smithi070 ceph-mon[41533]: osdmap e265: 8 total, 8 up, 8 in 2024-09-21T09:59:23.742 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:23 smithi070 ceph-mon[41533]: pgmap v109: 137 pgs: 137 active+clean; 869 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 170 op/s 2024-09-21T09:59:23.742 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:23 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/1562524679' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:23 smithi160 ceph-mon[33203]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:23 smithi160 ceph-mon[33203]: osdmap e265: 8 total, 8 up, 8 in 2024-09-21T09:59:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:23 smithi160 ceph-mon[33203]: pgmap v109: 137 pgs: 137 active+clean; 869 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 170 op/s 2024-09-21T09:59:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:23 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/1562524679' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:23.847 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 0~8388609 2024-09-21T09:59:23.934 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 16777218~8388609 2024-09-21T09:59:24.356 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 0~8388609 2024-09-21T09:59:24.370 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 16777218~8388609 2024-09-21T09:59:24.442 INFO:tasks.workunit.client.0.smithi070.stdout:migration_abort 2024-09-21T09:59:24.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:24 smithi160 ceph-mon[33203]: from='client.25064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:24.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:24 smithi160 ceph-mon[33203]: Upgrade: Updating alertmanager.a 2024-09-21T09:59:24.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:24 smithi160 ceph-mon[33203]: Deploying daemon alertmanager.a on smithi070 2024-09-21T09:59:24.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:24 smithi160 ceph-mon[33203]: from='client.25070 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:24.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:24 smithi160 ceph-mon[33203]: osdmap e266: 8 total, 8 up, 8 in 2024-09-21T09:59:24.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:24 smithi160 ceph-mon[33203]: osdmap e267: 8 total, 8 up, 8 in 2024-09-21T09:59:24.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:24 smithi070 ceph-mon[29589]: from='client.25064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:24.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:24 smithi070 ceph-mon[29589]: Upgrade: Updating alertmanager.a 2024-09-21T09:59:24.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:24 smithi070 ceph-mon[29589]: Deploying daemon alertmanager.a on smithi070 2024-09-21T09:59:24.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:24 smithi070 ceph-mon[29589]: from='client.25070 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:24.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:24 smithi070 ceph-mon[29589]: osdmap e266: 8 total, 8 up, 8 in 2024-09-21T09:59:24.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:24 smithi070 ceph-mon[29589]: osdmap e267: 8 total, 8 up, 8 in 2024-09-21T09:59:24.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:24 smithi070 ceph-mon[41533]: from='client.25064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:24.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:24 smithi070 ceph-mon[41533]: Upgrade: Updating alertmanager.a 2024-09-21T09:59:24.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:24 smithi070 ceph-mon[41533]: Deploying daemon alertmanager.a on smithi070 2024-09-21T09:59:24.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:24 smithi070 ceph-mon[41533]: from='client.25070 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:24.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:24 smithi070 ceph-mon[41533]: osdmap e266: 8 total, 8 up, 8 in 2024-09-21T09:59:24.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:24 smithi070 ceph-mon[41533]: osdmap e267: 8 total, 8 up, 8 in 2024-09-21T09:59:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:25 smithi160 ceph-mon[33203]: from='client.15189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:25 smithi160 ceph-mon[33203]: pgmap v112: 137 pgs: 137 active+clean; 869 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 170 op/s 2024-09-21T09:59:25.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:25 smithi070 ceph-mon[29589]: from='client.15189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:25.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:25 smithi070 ceph-mon[29589]: pgmap v112: 137 pgs: 137 active+clean; 869 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 170 op/s 2024-09-21T09:59:25.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:25 smithi070 ceph-mon[41533]: from='client.15189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:25.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:25 smithi070 ceph-mon[41533]: pgmap v112: 137 pgs: 137 active+clean; 869 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 170 op/s 2024-09-21T09:59:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:26 smithi160 ceph-mon[33203]: osdmap e268: 8 total, 8 up, 8 in 2024-09-21T09:59:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:26 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:59:26.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:26 smithi070 ceph-mon[29589]: osdmap e268: 8 total, 8 up, 8 in 2024-09-21T09:59:26.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:26 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:59:26.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:26 smithi070 ceph-mon[41533]: osdmap e268: 8 total, 8 up, 8 in 2024-09-21T09:59:26.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:26 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:59:27.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:27 smithi070 ceph-mon[29589]: osdmap e269: 8 total, 8 up, 8 in 2024-09-21T09:59:27.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:27 smithi070 ceph-mon[29589]: pgmap v115: 137 pgs: 137 active+clean; 869 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.0 MiB/s rd, 3.0 KiB/s wr, 14 op/s 2024-09-21T09:59:27.715 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:27 smithi070 ceph-mon[41533]: osdmap e269: 8 total, 8 up, 8 in 2024-09-21T09:59:27.715 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:27 smithi070 ceph-mon[41533]: pgmap v115: 137 pgs: 137 active+clean; 869 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.0 MiB/s rd, 3.0 KiB/s wr, 14 op/s 2024-09-21T09:59:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:27 smithi160 ceph-mon[33203]: osdmap e269: 8 total, 8 up, 8 in 2024-09-21T09:59:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:27 smithi160 ceph-mon[33203]: pgmap v115: 137 pgs: 137 active+clean; 869 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.0 MiB/s rd, 3.0 KiB/s wr, 14 op/s 2024-09-21T09:59:28.598 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:28 smithi070 ceph-mon[29589]: osdmap e270: 8 total, 8 up, 8 in 2024-09-21T09:59:28.598 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:28 smithi070 ceph-mon[41533]: osdmap e270: 8 total, 8 up, 8 in 2024-09-21T09:59:28.814 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.AbortWithSnapshots (14534 ms) 2024-09-21T09:59:28.814 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.CloneV1Parent 2024-09-21T09:59:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:28 smithi160 ceph-mon[33203]: osdmap e270: 8 total, 8 up, 8 in 2024-09-21T09:59:28.844 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:59:28.853 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 0~10 2024-09-21T09:59:28.861 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap1 2024-09-21T09:59:29.591 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:29 smithi070 ceph-mon[29589]: osdmap e271: 8 total, 8 up, 8 in 2024-09-21T09:59:29.591 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:29 smithi070 ceph-mon[29589]: pgmap v118: 137 pgs: 137 active+clean; 945 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 42 MiB/s rd, 23 MiB/s wr, 389 op/s 2024-09-21T09:59:29.591 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:29 smithi070 ceph-mon[41533]: osdmap e271: 8 total, 8 up, 8 in 2024-09-21T09:59:29.591 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:29 smithi070 ceph-mon[41533]: pgmap v118: 137 pgs: 137 active+clean; 945 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 42 MiB/s rd, 23 MiB/s wr, 389 op/s 2024-09-21T09:59:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:29 smithi160 ceph-mon[33203]: osdmap e271: 8 total, 8 up, 8 in 2024-09-21T09:59:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:29 smithi160 ceph-mon[33203]: pgmap v118: 137 pgs: 137 active+clean; 945 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 42 MiB/s rd, 23 MiB/s wr, 389 op/s 2024-09-21T09:59:30.462 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap1 2024-09-21T09:59:30.526 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:59:30.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:30 smithi070 ceph-mon[29589]: osdmap e272: 8 total, 8 up, 8 in 2024-09-21T09:59:30.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:30 smithi070 ceph-mon[41533]: osdmap e272: 8 total, 8 up, 8 in 2024-09-21T09:59:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:30 smithi160 ceph-mon[33203]: osdmap e272: 8 total, 8 up, 8 in 2024-09-21T09:59:31.273 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:30 smithi070 systemd[1]: Stopping Ceph alertmanager.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:59:31.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:31 smithi070 ceph-mon[29589]: osdmap e273: 8 total, 8 up, 8 in 2024-09-21T09:59:31.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:31 smithi070 ceph-mon[29589]: pgmap v121: 137 pgs: 137 active+clean; 945 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 37 MiB/s rd, 23 MiB/s wr, 375 op/s 2024-09-21T09:59:31.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:31 smithi070 ceph-mon[41533]: osdmap e273: 8 total, 8 up, 8 in 2024-09-21T09:59:31.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:31 smithi070 ceph-mon[41533]: pgmap v121: 137 pgs: 137 active+clean; 945 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 37 MiB/s rd, 23 MiB/s wr, 375 op/s 2024-09-21T09:59:31.628 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:31 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[77792]: ts=2024-09-21T09:59:31.270Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-21T09:59:31.629 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:31 smithi070 podman[102105]: 2024-09-21 09:59:31.311026282 +0000 UTC m=+0.173446955 container died df28d76f17955124c71a93e1ee8ece1dc111a8336fe256b388d8134370bd271d (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:59:31.641 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:59:31.710 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:59:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:31 smithi160 ceph-mon[33203]: osdmap e273: 8 total, 8 up, 8 in 2024-09-21T09:59:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:31 smithi160 ceph-mon[33203]: pgmap v121: 137 pgs: 137 active+clean; 945 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 37 MiB/s rd, 23 MiB/s wr, 375 op/s 2024-09-21T09:59:32.127 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:31 smithi070 podman[102105]: 2024-09-21 09:59:31.819407986 +0000 UTC m=+0.681828659 container cleanup df28d76f17955124c71a93e1ee8ece1dc111a8336fe256b388d8134370bd271d (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:59:32.128 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:31 smithi070 bash[102105]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a 2024-09-21T09:59:32.457 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:32 smithi070 podman[102121]: 2024-09-21 09:59:32.199731786 +0000 UTC m=+0.881853821 container remove df28d76f17955124c71a93e1ee8ece1dc111a8336fe256b388d8134370bd271d (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:59:32.577 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.CloneV1Parent (3762 ms) 2024-09-21T09:59:32.577 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.CloneV2Parent 2024-09-21T09:59:32.611 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:59:32.619 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 0~10 2024-09-21T09:59:32.627 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap1 2024-09-21T09:59:32.779 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:32 smithi070 podman[102121]: 2024-09-21 09:59:32.665634731 +0000 UTC m=+1.347756766 volume remove b9885b62cbf488fd51528b952d815a5f97ee369f2308150c7e2164eef109418d 2024-09-21T09:59:32.779 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:32 smithi070 ceph-mon[29589]: osdmap e274: 8 total, 8 up, 8 in 2024-09-21T09:59:32.779 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:32 smithi070 ceph-mon[41533]: osdmap e274: 8 total, 8 up, 8 in 2024-09-21T09:59:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:32 smithi160 ceph-mon[33203]: osdmap e274: 8 total, 8 up, 8 in 2024-09-21T09:59:33.127 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:32 smithi070 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-21T09:59:33.127 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:32 smithi070 systemd[1]: Stopped Ceph alertmanager.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:59:33.128 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:32 smithi070 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@alertmanager.a.service: Consumed 1.310s CPU time. 2024-09-21T09:59:33.128 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:32 smithi070 systemd[1]: Starting Ceph alertmanager.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:59:33.467 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:59:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:59:33] "GET /metrics HTTP/1.1" 200 38720 "" "Prometheus/2.51.0" 2024-09-21T09:59:33.467 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:33 smithi070 podman[102257]: 2024-09-21 09:59:33.092391844 +0000 UTC m=+0.016031085 image pull 11f11916f8cdfb17c1da74b20deb2075d778357b4d60474d82453303f8d461f5 quay.io/prometheus/alertmanager:v0.27.0 2024-09-21T09:59:33.468 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:33 smithi070 podman[102257]: 2024-09-21 09:59:33.200273115 +0000 UTC m=+0.123912346 volume create 901253e1476e56a94630dcca7c87830472087b89ed5d8cce13b6869131348e87 2024-09-21T09:59:33.468 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:33 smithi070 podman[102257]: 2024-09-21 09:59:33.292408696 +0000 UTC m=+0.216047928 container create 89ae95182419b722e9c485b51402bf6e6d9ebfa1c9224996f6e5a432fdb6bcc5 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:59:33.729 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:33 smithi070 ceph-mon[29589]: osdmap e275: 8 total, 8 up, 8 in 2024-09-21T09:59:33.730 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:33 smithi070 ceph-mon[29589]: pgmap v124: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 36 MiB/s rd, 16 KiB/s wr, 287 op/s 2024-09-21T09:59:33.730 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:33 smithi070 ceph-mon[41533]: osdmap e275: 8 total, 8 up, 8 in 2024-09-21T09:59:33.730 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:33 smithi070 ceph-mon[41533]: pgmap v124: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 36 MiB/s rd, 16 KiB/s wr, 287 op/s 2024-09-21T09:59:33.731 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:33 smithi070 podman[102257]: 2024-09-21 09:59:33.620053966 +0000 UTC m=+0.543693203 container init 89ae95182419b722e9c485b51402bf6e6d9ebfa1c9224996f6e5a432fdb6bcc5 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:59:33.731 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:33 smithi070 podman[102257]: 2024-09-21 09:59:33.625215461 +0000 UTC m=+0.548854693 container start 89ae95182419b722e9c485b51402bf6e6d9ebfa1c9224996f6e5a432fdb6bcc5 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:59:33.731 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[102272]: ts=2024-09-21T09:59:33.636Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-09-21T09:59:33.731 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[102272]: ts=2024-09-21T09:59:33.636Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-09-21T09:59:33.731 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[102272]: ts=2024-09-21T09:59:33.637Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.70 port=9094 2024-09-21T09:59:33.731 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[102272]: ts=2024-09-21T09:59:33.638Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-21T09:59:33.731 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[102272]: ts=2024-09-21T09:59:33.660Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T09:59:33.731 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[102272]: ts=2024-09-21T09:59:33.661Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T09:59:33.731 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[102272]: ts=2024-09-21T09:59:33.662Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-09-21T09:59:33.731 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[102272]: ts=2024-09-21T09:59:33.662Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-21T09:59:33.731 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:33 smithi070 bash[102257]: 89ae95182419b722e9c485b51402bf6e6d9ebfa1c9224996f6e5a432fdb6bcc5 2024-09-21T09:59:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:33 smithi160 ceph-mon[33203]: osdmap e275: 8 total, 8 up, 8 in 2024-09-21T09:59:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:33 smithi160 ceph-mon[33203]: pgmap v124: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 36 MiB/s rd, 16 KiB/s wr, 287 op/s 2024-09-21T09:59:34.127 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:33 smithi070 systemd[1]: Started Ceph alertmanager.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:59:34.359 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap1 2024-09-21T09:59:34.413 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:59:34.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:34 smithi070 ceph-mon[29589]: osdmap e276: 8 total, 8 up, 8 in 2024-09-21T09:59:34.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:34 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:34.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:34 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:34.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:34 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:59:34.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:34 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:34.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:34 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:34.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:34 smithi070 ceph-mon[29589]: osdmap e277: 8 total, 8 up, 8 in 2024-09-21T09:59:34.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:34 smithi070 ceph-mon[41533]: osdmap e276: 8 total, 8 up, 8 in 2024-09-21T09:59:34.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:34 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:34.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:34 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:34.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:34 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:59:34.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:34 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:34.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:34 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:34.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:34 smithi070 ceph-mon[41533]: osdmap e277: 8 total, 8 up, 8 in 2024-09-21T09:59:34.788 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:34 smithi160 ceph-mon[33203]: osdmap e276: 8 total, 8 up, 8 in 2024-09-21T09:59:34.788 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:34 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:34.788 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:34 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:34.788 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:34 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:59:34.788 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:34 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:34.788 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:34 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:34.788 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:34 smithi160 ceph-mon[33203]: osdmap e277: 8 total, 8 up, 8 in 2024-09-21T09:59:35.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:35 smithi070 ceph-mon[29589]: pgmap v127: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 36 MiB/s rd, 16 KiB/s wr, 287 op/s 2024-09-21T09:59:35.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:35 smithi070 ceph-mon[41533]: pgmap v127: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 36 MiB/s rd, 16 KiB/s wr, 287 op/s 2024-09-21T09:59:35.652 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T09:59:35.735 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T09:59:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:35 smithi160 ceph-mon[33203]: pgmap v127: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 36 MiB/s rd, 16 KiB/s wr, 287 op/s 2024-09-21T09:59:36.127 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[102272]: ts=2024-09-21T09:59:35.639Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000284831s 2024-09-21T09:59:36.580 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.CloneV2Parent (4004 ms) 2024-09-21T09:59:36.581 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.CloneV1ParentAbort 2024-09-21T09:59:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:36 smithi160 ceph-mon[33203]: osdmap e278: 8 total, 8 up, 8 in 2024-09-21T09:59:36.610 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:59:36.621 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 0~10 2024-09-21T09:59:36.631 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap1 2024-09-21T09:59:36.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:36 smithi070 ceph-mon[29589]: osdmap e278: 8 total, 8 up, 8 in 2024-09-21T09:59:36.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:36 smithi070 ceph-mon[41533]: osdmap e278: 8 total, 8 up, 8 in 2024-09-21T09:59:37.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:37 smithi160 ceph-mon[33203]: osdmap e279: 8 total, 8 up, 8 in 2024-09-21T09:59:37.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:37 smithi160 ceph-mon[33203]: pgmap v130: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 146 KiB/s rd, 10 KiB/s wr, 185 op/s 2024-09-21T09:59:37.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:37 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:37.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:37 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:37.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:37 smithi070 ceph-mon[29589]: osdmap e279: 8 total, 8 up, 8 in 2024-09-21T09:59:37.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:37 smithi070 ceph-mon[29589]: pgmap v130: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 146 KiB/s rd, 10 KiB/s wr, 185 op/s 2024-09-21T09:59:37.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:37 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:37.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:37 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:37.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:37 smithi070 ceph-mon[41533]: osdmap e279: 8 total, 8 up, 8 in 2024-09-21T09:59:37.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:37 smithi070 ceph-mon[41533]: pgmap v130: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 146 KiB/s rd, 10 KiB/s wr, 185 op/s 2024-09-21T09:59:37.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:37 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:37.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:37 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:38.499 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap1 2024-09-21T09:59:38.552 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:59:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:38 smithi070 ceph-mon[29589]: osdmap e280: 8 total, 8 up, 8 in 2024-09-21T09:59:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:38.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:38.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:38 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:38.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:38 smithi070 ceph-mon[41533]: osdmap e280: 8 total, 8 up, 8 in 2024-09-21T09:59:38.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:38.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:38.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:38.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:38 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:38 smithi160 ceph-mon[33203]: osdmap e280: 8 total, 8 up, 8 in 2024-09-21T09:59:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:38 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:39.662 INFO:tasks.workunit.client.0.smithi070.stdout:migration_abort 2024-09-21T09:59:39.779 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:39 smithi070 ceph-mon[29589]: osdmap e281: 8 total, 8 up, 8 in 2024-09-21T09:59:39.779 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:39 smithi070 ceph-mon[29589]: pgmap v133: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 487 KiB/s rd, 28 KiB/s wr, 610 op/s 2024-09-21T09:59:39.780 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:39 smithi070 ceph-mon[41533]: osdmap e281: 8 total, 8 up, 8 in 2024-09-21T09:59:39.780 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:39 smithi070 ceph-mon[41533]: pgmap v133: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 487 KiB/s rd, 28 KiB/s wr, 610 op/s 2024-09-21T09:59:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:39 smithi160 ceph-mon[33203]: osdmap e281: 8 total, 8 up, 8 in 2024-09-21T09:59:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:39 smithi160 ceph-mon[33203]: pgmap v133: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 487 KiB/s rd, 28 KiB/s wr, 610 op/s 2024-09-21T09:59:40.618 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.CloneV1ParentAbort (4038 ms) 2024-09-21T09:59:40.618 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.CloneV2ParentAbort 2024-09-21T09:59:40.649 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:59:40.663 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 0~10 2024-09-21T09:59:40.672 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap1 2024-09-21T09:59:40.743 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:40 smithi160 ceph-mon[33203]: osdmap e282: 8 total, 8 up, 8 in 2024-09-21T09:59:40.828 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:40 smithi070 ceph-mon[29589]: osdmap e282: 8 total, 8 up, 8 in 2024-09-21T09:59:40.828 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:40 smithi070 ceph-mon[41533]: osdmap e282: 8 total, 8 up, 8 in 2024-09-21T09:59:41.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:41 smithi160 ceph-mon[33203]: osdmap e283: 8 total, 8 up, 8 in 2024-09-21T09:59:41.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:41 smithi160 ceph-mon[33203]: pgmap v136: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 341 KiB/s rd, 18 KiB/s wr, 425 op/s 2024-09-21T09:59:41.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:41 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:59:41.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:41 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:41.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:41 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:41.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:41 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:59:41.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:41 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:59:41.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:41 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:41.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:41 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:59:41.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:41 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:59:41.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:41 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:41 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:41 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:41 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:41 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:41 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:41 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:41 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:41 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:41 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:41 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:41 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[29589]: osdmap e283: 8 total, 8 up, 8 in 2024-09-21T09:59:41.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[29589]: pgmap v136: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 341 KiB/s rd, 18 KiB/s wr, 425 op/s 2024-09-21T09:59:41.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:59:41.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:41.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:41.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:59:41.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:59:41.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:41.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:59:41.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:59:41.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[41533]: osdmap e283: 8 total, 8 up, 8 in 2024-09-21T09:59:41.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[41533]: pgmap v136: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 341 KiB/s rd, 18 KiB/s wr, 425 op/s 2024-09-21T09:59:41.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:59:41.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:41.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:41.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:59:41.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:59:41.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:41.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:59:41.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:59:41.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:41.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:41 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:42.527 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap1 2024-09-21T09:59:42.580 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:59:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:42 smithi160 ceph-mon[33203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:59:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:42 smithi160 ceph-mon[33203]: osdmap e284: 8 total, 8 up, 8 in 2024-09-21T09:59:42.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:42 smithi070 ceph-mon[29589]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:59:42.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:42 smithi070 ceph-mon[29589]: osdmap e284: 8 total, 8 up, 8 in 2024-09-21T09:59:42.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:42 smithi070 ceph-mon[41533]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:59:42.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:42 smithi070 ceph-mon[41533]: osdmap e284: 8 total, 8 up, 8 in 2024-09-21T09:59:43.513 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:59:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:59:43] "GET /metrics HTTP/1.1" 200 38719 "" "Prometheus/2.51.0" 2024-09-21T09:59:43.663 INFO:tasks.workunit.client.0.smithi070.stdout:migration_abort 2024-09-21T09:59:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:43 smithi160 ceph-mon[33203]: osdmap e285: 8 total, 8 up, 8 in 2024-09-21T09:59:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:43 smithi160 ceph-mon[33203]: mgrmap e33: y(active, since 92s), standbys: x 2024-09-21T09:59:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:43 smithi160 ceph-mon[33203]: pgmap v139: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 299 KiB/s rd, 23 KiB/s wr, 391 op/s 2024-09-21T09:59:43.877 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 09:59:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[102272]: ts=2024-09-21T09:59:43.642Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003852597s 2024-09-21T09:59:43.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:43 smithi070 ceph-mon[29589]: osdmap e285: 8 total, 8 up, 8 in 2024-09-21T09:59:43.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:43 smithi070 ceph-mon[29589]: mgrmap e33: y(active, since 92s), standbys: x 2024-09-21T09:59:43.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:43 smithi070 ceph-mon[29589]: pgmap v139: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 299 KiB/s rd, 23 KiB/s wr, 391 op/s 2024-09-21T09:59:43.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:43 smithi070 ceph-mon[41533]: osdmap e285: 8 total, 8 up, 8 in 2024-09-21T09:59:43.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:43 smithi070 ceph-mon[41533]: mgrmap e33: y(active, since 92s), standbys: x 2024-09-21T09:59:43.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:43 smithi070 ceph-mon[41533]: pgmap v139: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 299 KiB/s rd, 23 KiB/s wr, 391 op/s 2024-09-21T09:59:44.469 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.CloneV2ParentAbort (3851 ms) 2024-09-21T09:59:44.469 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.CloneV1ParentAbortFixIncompleteChildReattach 2024-09-21T09:59:44.497 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:59:44.507 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 0~10 2024-09-21T09:59:44.515 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap1 2024-09-21T09:59:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:44 smithi160 ceph-mon[33203]: Upgrade: Updating grafana.a 2024-09-21T09:59:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:44 smithi160 ceph-mon[33203]: Deploying daemon grafana.a on smithi160 2024-09-21T09:59:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:44 smithi160 ceph-mon[33203]: osdmap e286: 8 total, 8 up, 8 in 2024-09-21T09:59:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:44 smithi160 ceph-mon[33203]: osdmap e287: 8 total, 8 up, 8 in 2024-09-21T09:59:44.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:44 smithi070 ceph-mon[29589]: Upgrade: Updating grafana.a 2024-09-21T09:59:44.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:44 smithi070 ceph-mon[29589]: Deploying daemon grafana.a on smithi160 2024-09-21T09:59:44.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:44 smithi070 ceph-mon[29589]: osdmap e286: 8 total, 8 up, 8 in 2024-09-21T09:59:44.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:44 smithi070 ceph-mon[29589]: osdmap e287: 8 total, 8 up, 8 in 2024-09-21T09:59:44.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:44 smithi070 ceph-mon[41533]: Upgrade: Updating grafana.a 2024-09-21T09:59:44.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:44 smithi070 ceph-mon[41533]: Deploying daemon grafana.a on smithi160 2024-09-21T09:59:44.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:44 smithi070 ceph-mon[41533]: osdmap e286: 8 total, 8 up, 8 in 2024-09-21T09:59:44.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:44 smithi070 ceph-mon[41533]: osdmap e287: 8 total, 8 up, 8 in 2024-09-21T09:59:45.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:45 smithi160 ceph-mon[33203]: pgmap v142: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 299 KiB/s rd, 23 KiB/s wr, 391 op/s 2024-09-21T09:59:45.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:45 smithi070 ceph-mon[29589]: pgmap v142: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 299 KiB/s rd, 23 KiB/s wr, 391 op/s 2024-09-21T09:59:45.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:45 smithi070 ceph-mon[41533]: pgmap v142: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 299 KiB/s rd, 23 KiB/s wr, 391 op/s 2024-09-21T09:59:46.536 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap1 2024-09-21T09:59:46.589 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:59:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:46 smithi160 ceph-mon[33203]: osdmap e288: 8 total, 8 up, 8 in 2024-09-21T09:59:46.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:46 smithi070 ceph-mon[29589]: osdmap e288: 8 total, 8 up, 8 in 2024-09-21T09:59:46.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:46 smithi070 ceph-mon[41533]: osdmap e288: 8 total, 8 up, 8 in 2024-09-21T09:59:47.700 INFO:tasks.workunit.client.0.smithi070.stdout:migration_abort 2024-09-21T09:59:47.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:47 smithi160 ceph-mon[33203]: osdmap e289: 8 total, 8 up, 8 in 2024-09-21T09:59:47.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:47 smithi160 ceph-mon[33203]: pgmap v145: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 30 KiB/s rd, 4.0 KiB/s wr, 46 op/s 2024-09-21T09:59:47.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:47 smithi070 ceph-mon[29589]: osdmap e289: 8 total, 8 up, 8 in 2024-09-21T09:59:47.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:47 smithi070 ceph-mon[29589]: pgmap v145: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 30 KiB/s rd, 4.0 KiB/s wr, 46 op/s 2024-09-21T09:59:47.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:47 smithi070 ceph-mon[41533]: osdmap e289: 8 total, 8 up, 8 in 2024-09-21T09:59:47.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:47 smithi070 ceph-mon[41533]: pgmap v145: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 30 KiB/s rd, 4.0 KiB/s wr, 46 op/s 2024-09-21T09:59:48.658 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.CloneV1ParentAbortFixIncompleteChildReattach (4188 ms) 2024-09-21T09:59:48.658 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.CloneV1ParentAbortFixParentReattach 2024-09-21T09:59:48.687 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:59:48.698 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 0~10 2024-09-21T09:59:48.706 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap1 2024-09-21T09:59:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:48 smithi160 ceph-mon[33203]: osdmap e290: 8 total, 8 up, 8 in 2024-09-21T09:59:48.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:48 smithi070 ceph-mon[41533]: osdmap e290: 8 total, 8 up, 8 in 2024-09-21T09:59:48.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:48 smithi070 ceph-mon[29589]: osdmap e290: 8 total, 8 up, 8 in 2024-09-21T09:59:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:49 smithi160 ceph-mon[33203]: pgmap v147: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 229 KiB/s rd, 20 KiB/s wr, 300 op/s 2024-09-21T09:59:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:49 smithi160 ceph-mon[33203]: osdmap e291: 8 total, 8 up, 8 in 2024-09-21T09:59:49.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:49 smithi070 ceph-mon[29589]: pgmap v147: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 229 KiB/s rd, 20 KiB/s wr, 300 op/s 2024-09-21T09:59:49.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:49 smithi070 ceph-mon[29589]: osdmap e291: 8 total, 8 up, 8 in 2024-09-21T09:59:49.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:49 smithi070 ceph-mon[41533]: pgmap v147: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 229 KiB/s rd, 20 KiB/s wr, 300 op/s 2024-09-21T09:59:49.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:49 smithi070 ceph-mon[41533]: osdmap e291: 8 total, 8 up, 8 in 2024-09-21T09:59:50.564 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap1 2024-09-21T09:59:50.620 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:59:50.734 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:50 smithi160 ceph-mon[33203]: osdmap e292: 8 total, 8 up, 8 in 2024-09-21T09:59:50.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:50 smithi070 ceph-mon[29589]: osdmap e292: 8 total, 8 up, 8 in 2024-09-21T09:59:50.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:50 smithi070 ceph-mon[41533]: osdmap e292: 8 total, 8 up, 8 in 2024-09-21T09:59:51.724 INFO:tasks.workunit.client.0.smithi070.stdout:migration_abort 2024-09-21T09:59:51.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:51 smithi160 ceph-mon[33203]: pgmap v150: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 226 KiB/s rd, 21 KiB/s wr, 294 op/s 2024-09-21T09:59:51.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:51 smithi160 ceph-mon[33203]: osdmap e293: 8 total, 8 up, 8 in 2024-09-21T09:59:51.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:51 smithi070 ceph-mon[41533]: pgmap v150: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 226 KiB/s rd, 21 KiB/s wr, 294 op/s 2024-09-21T09:59:51.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:51 smithi070 ceph-mon[41533]: osdmap e293: 8 total, 8 up, 8 in 2024-09-21T09:59:51.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:51 smithi070 ceph-mon[29589]: pgmap v150: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 226 KiB/s rd, 21 KiB/s wr, 294 op/s 2024-09-21T09:59:51.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:51 smithi070 ceph-mon[29589]: osdmap e293: 8 total, 8 up, 8 in 2024-09-21T09:59:52.694 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.CloneV1ParentAbortFixParentReattach (4036 ms) 2024-09-21T09:59:52.694 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.CloneV1ParentAbortRelinkNotNeeded 2024-09-21T09:59:52.695 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:52 smithi160 ceph-mon[33203]: osdmap e294: 8 total, 8 up, 8 in 2024-09-21T09:59:52.723 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:59:52.734 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 0~10 2024-09-21T09:59:52.742 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap1 2024-09-21T09:59:52.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:52 smithi070 ceph-mon[29589]: osdmap e294: 8 total, 8 up, 8 in 2024-09-21T09:59:52.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:52 smithi070 ceph-mon[41533]: osdmap e294: 8 total, 8 up, 8 in 2024-09-21T09:59:53.238 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:52 smithi160 systemd[1]: Stopping Ceph grafana.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:59:53.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 09:59:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:09:59:53] "GET /metrics HTTP/1.1" 200 38719 "" "Prometheus/2.51.0" 2024-09-21T09:59:53.526 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:53 smithi160 podman[92823]: 2024-09-21 09:59:53.235856373 +0000 UTC m=+0.107465139 container died 8b9850d5203c7f74df753c11ce8e2f577d266071bb6669ce07aa57a3b8ab144e (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vendor=Red Hat, Inc., architecture=x86_64, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, build-date=2022-04-29T11:48:11.055897, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, io.openshift.expose-services=, io.openshift.tags=base rhel8, name=ubi8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, version=8.5, distribution-scope=public, maintainer=Paul Cuzner ) 2024-09-21T09:59:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:53 smithi160 ceph-mon[33203]: pgmap v153: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 362 KiB/s rd, 20 KiB/s wr, 461 op/s 2024-09-21T09:59:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:53 smithi160 ceph-mon[33203]: osdmap e295: 8 total, 8 up, 8 in 2024-09-21T09:59:53.841 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:53 smithi160 podman[92823]: 2024-09-21 09:59:53.59816536 +0000 UTC m=+0.469774122 container cleanup 8b9850d5203c7f74df753c11ce8e2f577d266071bb6669ce07aa57a3b8ab144e (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a, io.buildah.version=1.32.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, version=8.5, description=Ceph Grafana Container, io.openshift.tags=base rhel8, architecture=x86_64, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897, name=ubi8, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-09-21T09:59:53.841 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:53 smithi160 bash[92823]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a 2024-09-21T09:59:53.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:53 smithi070 ceph-mon[29589]: pgmap v153: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 362 KiB/s rd, 20 KiB/s wr, 461 op/s 2024-09-21T09:59:53.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:53 smithi070 ceph-mon[29589]: osdmap e295: 8 total, 8 up, 8 in 2024-09-21T09:59:53.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:53 smithi070 ceph-mon[41533]: pgmap v153: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 362 KiB/s rd, 20 KiB/s wr, 461 op/s 2024-09-21T09:59:53.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:53 smithi070 ceph-mon[41533]: osdmap e295: 8 total, 8 up, 8 in 2024-09-21T09:59:54.053 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-09-21T09:59:54.209 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:53 smithi160 podman[92840]: 2024-09-21 09:59:53.874853233 +0000 UTC m=+0.641220912 container remove 8b9850d5203c7f74df753c11ce8e2f577d266071bb6669ce07aa57a3b8ab144e (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a, vcs-type=git, build-date=2022-04-29T11:48:11.055897, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, com.redhat.component=ubi8-container, description=Ceph Grafana Container, io.openshift.expose-services=, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, version=8.5, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-09-21T09:59:54.209 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:53 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-21T09:59:54.209 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@grafana.a.service: Failed with result 'exit-code'. 2024-09-21T09:59:54.209 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 systemd[1]: Stopped Ceph grafana.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:59:54.209 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@grafana.a.service: Consumed 7.677s CPU time. 2024-09-21T09:59:54.209 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 systemd[1]: Starting Ceph grafana.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:59:54.364 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap1 2024-09-21T09:59:54.413 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:59:54.524 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 podman[92972]: 2024-09-21 09:59:54.3927718 +0000 UTC m=+0.093653711 container create daa2d8dfab48af09378046f2da6d57eaf11d749ac73ce79ab86f59b17549b177 (image=quay.io/ceph/grafana:10.4.8, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-21T09:59:54.524 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 podman[92972]: 2024-09-21 09:59:54.315705717 +0000 UTC m=+0.016587648 image pull 5aad1d7cf004c80a4acb170e1c80fb85781b022f00f08c6bcd22608882b4c748 quay.io/ceph/grafana:10.4.8 2024-09-21T09:59:54.632 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T09:59:54.632 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (21s) 17s ago 6m 16.5M - 0.27.0 11f11916f8cd 89ae95182419 2024-09-21T09:59:54.632 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi160 *:3000 running (5m) 16s ago 5m 87.3M - 9.4.7 954c08fa6188 8b9850d5203c 2024-09-21T09:59:54.633 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi070.gansda smithi070 running (4m) 17s ago 4m 17.9M - 18.2.4-871-g6a1bdf85 0adab276c97e 6016b2501c8a 2024-09-21T09:59:54.633 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi160.byyhfz smithi160 running (4m) 16s ago 4m 15.0M - 18.2.4-871-g6a1bdf85 0adab276c97e 93e108d91e65 2024-09-21T09:59:54.633 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi160 *:8443,9283,8765 running (85s) 16s ago 10m 454M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 d952652a9683 2024-09-21T09:59:54.633 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (112s) 17s ago 11m 517M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 e3d390893cff 2024-09-21T09:59:54.633 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (11m) 17s ago 11m 58.1M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e cee8ca713d09 2024-09-21T09:59:54.633 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi160 running (10m) 16s ago 10m 54.6M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e 0fd16436dfa0 2024-09-21T09:59:54.633 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (10m) 17s ago 10m 48.6M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e fc0590f324b5 2024-09-21T09:59:54.633 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (70s) 17s ago 6m 9625k - 1.7.0 72c9c2088986 5d5d4b026070 2024-09-21T09:59:54.633 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi160 *:9100 running (64s) 16s ago 5m 11.9M - 1.7.0 72c9c2088986 19b8e0e8ee0c 2024-09-21T09:59:54.633 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (9m) 17s ago 9m 125M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 10580e80f007 2024-09-21T09:59:54.633 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (9m) 17s ago 9m 99.7M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e b7c0d1169d1d 2024-09-21T09:59:54.633 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (8m) 17s ago 8m 106M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 58a11b8ddb81 2024-09-21T09:59:54.633 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (8m) 17s ago 8m 99.2M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e aa0c8a732a41 2024-09-21T09:59:54.634 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi160 running (8m) 16s ago 8m 118M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 7325a075c488 2024-09-21T09:59:54.634 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi160 running (7m) 16s ago 7m 92.3M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e f74026345641 2024-09-21T09:59:54.634 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi160 running (7m) 16s ago 7m 140M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 03e6bb54f8b2 2024-09-21T09:59:54.634 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi160 running (6m) 16s ago 6m 113M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e e405b21a6146 2024-09-21T09:59:54.634 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi160 *:9095 running (42s) 16s ago 6m 37.6M - 2.51.0 1d3b7f56885b 648952e57ad7 2024-09-21T09:59:54.828 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-mon[33203]: osdmap e296: 8 total, 8 up, 8 in 2024-09-21T09:59:54.828 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-mon[33203]: osdmap e297: 8 total, 8 up, 8 in 2024-09-21T09:59:54.829 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 podman[92972]: 2024-09-21 09:59:54.653569861 +0000 UTC m=+0.354451790 container init daa2d8dfab48af09378046f2da6d57eaf11d749ac73ce79ab86f59b17549b177 (image=quay.io/ceph/grafana:10.4.8, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-21T09:59:54.829 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 podman[92972]: 2024-09-21 09:59:54.658117886 +0000 UTC m=+0.358999797 container start daa2d8dfab48af09378046f2da6d57eaf11d749ac73ce79ab86f59b17549b177 (image=quay.io/ceph/grafana:10.4.8, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-21T09:59:54.829 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 bash[92972]: daa2d8dfab48af09378046f2da6d57eaf11d749ac73ce79ab86f59b17549b177 2024-09-21T09:59:54.829 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 systemd[1]: Started Ceph grafana.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:59:54.829 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=settings t=2024-09-21T09:59:54.786029749Z level=info msg="Starting Grafana" version=10.4.8 commit=8c384b71bde142a25ddc01ed5a183ca572b2b7f7 branch=v10.4.x compiled=2024-09-21T09:59:54Z 2024-09-21T09:59:54.829 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=settings t=2024-09-21T09:59:54.786268883Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-21T09:59:54.829 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=settings t=2024-09-21T09:59:54.786277892Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-21T09:59:54.829 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=settings t=2024-09-21T09:59:54.786281157Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-09-21T09:59:54.829 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=settings t=2024-09-21T09:59:54.786284211Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-09-21T09:59:54.829 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=settings t=2024-09-21T09:59:54.786286891Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-09-21T09:59:54.829 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=settings t=2024-09-21T09:59:54.786290402Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-09-21T09:59:54.830 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=settings t=2024-09-21T09:59:54.78629304Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-09-21T09:59:54.830 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=settings t=2024-09-21T09:59:54.786296106Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-21T09:59:54.830 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=settings t=2024-09-21T09:59:54.786298922Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-21T09:59:54.830 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=settings t=2024-09-21T09:59:54.786301496Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-21T09:59:54.830 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=settings t=2024-09-21T09:59:54.786304535Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-21T09:59:54.830 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=settings t=2024-09-21T09:59:54.786307412Z level=info msg=Target target=[all] 2024-09-21T09:59:54.830 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=settings t=2024-09-21T09:59:54.786313866Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-21T09:59:54.830 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=settings t=2024-09-21T09:59:54.786316671Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-21T09:59:54.830 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=settings t=2024-09-21T09:59:54.786319156Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-21T09:59:54.831 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=settings t=2024-09-21T09:59:54.786321614Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-21T09:59:54.831 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=settings t=2024-09-21T09:59:54.786324287Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-21T09:59:54.831 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=settings t=2024-09-21T09:59:54.786326912Z level=info msg="App mode production" 2024-09-21T09:59:54.831 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=sqlstore t=2024-09-21T09:59:54.78657243Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-21T09:59:54.831 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=sqlstore t=2024-09-21T09:59:54.786586332Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-21T09:59:54.831 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.787035505Z level=info msg="Starting DB migrations" 2024-09-21T09:59:54.831 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.804055914Z level=info msg="Executing migration" id="Add uid column to user" 2024-09-21T09:59:54.831 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.809662261Z level=info msg="Migration successfully executed" id="Add uid column to user" duration=5.604749ms 2024-09-21T09:59:54.831 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.810093871Z level=info msg="Executing migration" id="Update uid column values for users" 2024-09-21T09:59:54.831 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.810225894Z level=info msg="Migration successfully executed" id="Update uid column values for users" duration=132.086µs 2024-09-21T09:59:54.831 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.812608087Z level=info msg="Executing migration" id="Add unique index user_uid" 2024-09-21T09:59:54.831 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.813340994Z level=info msg="Migration successfully executed" id="Add unique index user_uid" duration=732.625µs 2024-09-21T09:59:54.831 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.813720322Z level=info msg="Executing migration" id="update login field with orgid to allow for multiple service accounts with same name across orgs" 2024-09-21T09:59:54.831 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.813951165Z level=info msg="Migration successfully executed" id="update login field with orgid to allow for multiple service accounts with same name across orgs" duration=231.7µs 2024-09-21T09:59:54.832 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.816585553Z level=info msg="Executing migration" id="Add playlist column created_at" 2024-09-21T09:59:54.832 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.821687441Z level=info msg="Migration successfully executed" id="Add playlist column created_at" duration=5.101328ms 2024-09-21T09:59:54.832 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.822091014Z level=info msg="Executing migration" id="Add playlist column updated_at" 2024-09-21T09:59:54.832 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.827334044Z level=info msg="Migration successfully executed" id="Add playlist column updated_at" duration=5.238781ms 2024-09-21T09:59:54.832 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.827971702Z level=info msg="Executing migration" id="Add column uid in team" 2024-09-21T09:59:54.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:54 smithi070 ceph-mon[29589]: osdmap e296: 8 total, 8 up, 8 in 2024-09-21T09:59:54.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:54 smithi070 ceph-mon[29589]: osdmap e297: 8 total, 8 up, 8 in 2024-09-21T09:59:54.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:54 smithi070 ceph-mon[41533]: osdmap e296: 8 total, 8 up, 8 in 2024-09-21T09:59:54.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:54 smithi070 ceph-mon[41533]: osdmap e297: 8 total, 8 up, 8 in 2024-09-21T09:59:54.971 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T09:59:54.971 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-09-21T09:59:54.971 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 3 2024-09-21T09:59:54.972 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:59:54.972 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-09-21T09:59:54.972 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T09:59:54.972 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:59:54.972 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-09-21T09:59:54.972 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 8 2024-09-21T09:59:54.972 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:59:54.972 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": { 2024-09-21T09:59:54.972 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 2 2024-09-21T09:59:54.972 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T09:59:54.973 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-09-21T09:59:54.973 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 13, 2024-09-21T09:59:54.973 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T09:59:54.973 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T09:59:54.973 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T09:59:55.080 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.833650075Z level=info msg="Migration successfully executed" id="Add column uid in team" duration=5.677437ms 2024-09-21T09:59:55.080 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.834082759Z level=info msg="Executing migration" id="Update uid column values in team" 2024-09-21T09:59:55.080 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.834216575Z level=info msg="Migration successfully executed" id="Update uid column values in team" duration=133.978µs 2024-09-21T09:59:55.080 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.834608002Z level=info msg="Executing migration" id="Add unique index team_org_id_uid" 2024-09-21T09:59:55.080 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.835382345Z level=info msg="Migration successfully executed" id="Add unique index team_org_id_uid" duration=774.513µs 2024-09-21T09:59:55.081 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.837519894Z level=info msg="Executing migration" id="add index user_auth_token.revoked_at" 2024-09-21T09:59:55.081 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.838255699Z level=info msg="Migration successfully executed" id="add index user_auth_token.revoked_at" duration=735.549µs 2024-09-21T09:59:55.081 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.840317069Z level=info msg="Executing migration" id="add result_fingerprint column to alert_instance" 2024-09-21T09:59:55.081 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.845735239Z level=info msg="Migration successfully executed" id="add result_fingerprint column to alert_instance" duration=5.417815ms 2024-09-21T09:59:55.081 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.846176172Z level=info msg="Executing migration" id="add last_applied column to alert_configuration_history" 2024-09-21T09:59:55.081 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.851156514Z level=info msg="Migration successfully executed" id="add last_applied column to alert_configuration_history" duration=4.964052ms 2024-09-21T09:59:55.081 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.851536267Z level=info msg="Executing migration" id="alter library_element model to mediumtext" 2024-09-21T09:59:55.081 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.8515756Z level=info msg="Migration successfully executed" id="alter library_element model to mediumtext" duration=39.227µs 2024-09-21T09:59:55.081 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.854324342Z level=info msg="Executing migration" id="add library_element folder uid" 2024-09-21T09:59:55.081 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.859452002Z level=info msg="Migration successfully executed" id="add library_element folder uid" duration=5.12652ms 2024-09-21T09:59:55.082 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.860004502Z level=info msg="Executing migration" id="populate library_element folder_uid" 2024-09-21T09:59:55.082 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.860304613Z level=info msg="Migration successfully executed" id="populate library_element folder_uid" duration=300.283µs 2024-09-21T09:59:55.083 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.862639031Z level=info msg="Executing migration" id="add index library_element org_id-folder_uid-name-kind" 2024-09-21T09:59:55.083 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.863500596Z level=info msg="Migration successfully executed" id="add index library_element org_id-folder_uid-name-kind" duration=861.736µs 2024-09-21T09:59:55.083 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.86395892Z level=info msg="Executing migration" id="permission kind migration" 2024-09-21T09:59:55.083 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.869058571Z level=info msg="Migration successfully executed" id="permission kind migration" duration=5.099283ms 2024-09-21T09:59:55.083 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.869473964Z level=info msg="Executing migration" id="permission attribute migration" 2024-09-21T09:59:55.083 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.874614302Z level=info msg="Migration successfully executed" id="permission attribute migration" duration=5.139914ms 2024-09-21T09:59:55.083 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.875019538Z level=info msg="Executing migration" id="permission identifier migration" 2024-09-21T09:59:55.083 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.88006012Z level=info msg="Migration successfully executed" id="permission identifier migration" duration=5.04027ms 2024-09-21T09:59:55.083 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.880438913Z level=info msg="Executing migration" id="add permission identifier index" 2024-09-21T09:59:55.083 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.881226703Z level=info msg="Migration successfully executed" id="add permission identifier index" duration=787.826µs 2024-09-21T09:59:55.084 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.881580741Z level=info msg="Executing migration" id="add permission action scope role_id index" 2024-09-21T09:59:55.084 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.882360466Z level=info msg="Migration successfully executed" id="add permission action scope role_id index" duration=779.888µs 2024-09-21T09:59:55.084 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.884305734Z level=info msg="Executing migration" id="remove permission role_id action scope index" 2024-09-21T09:59:55.084 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.885036861Z level=info msg="Migration successfully executed" id="remove permission role_id action scope index" duration=731.218µs 2024-09-21T09:59:55.084 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.886979865Z level=info msg="Executing migration" id="rbac disabled migrator" 2024-09-21T09:59:55.084 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.887006353Z level=info msg="Migration successfully executed" id="rbac disabled migrator" duration=26.968µs 2024-09-21T09:59:55.084 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.889526052Z level=info msg="Executing migration" id="drop index IDX_correlation_uid - v1" 2024-09-21T09:59:55.084 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.890288532Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_uid - v1" duration=762.351µs 2024-09-21T09:59:55.084 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.890657156Z level=info msg="Executing migration" id="drop index IDX_correlation_source_uid - v1" 2024-09-21T09:59:55.084 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.891421409Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_source_uid - v1" duration=764.538µs 2024-09-21T09:59:55.084 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.893321679Z level=info msg="Executing migration" id="Rename table correlation to correlation_tmp_qwerty - v1" 2024-09-21T09:59:55.084 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.910094589Z level=info msg="Migration successfully executed" id="Rename table correlation to correlation_tmp_qwerty - v1" duration=16.772011ms 2024-09-21T09:59:55.084 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.910524047Z level=info msg="Executing migration" id="create correlation v2" 2024-09-21T09:59:55.085 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.911255596Z level=info msg="Migration successfully executed" id="create correlation v2" duration=727.784µs 2024-09-21T09:59:55.085 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.913199248Z level=info msg="Executing migration" id="create index IDX_correlation_uid - v2" 2024-09-21T09:59:55.085 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.913905685Z level=info msg="Migration successfully executed" id="create index IDX_correlation_uid - v2" duration=704.62µs 2024-09-21T09:59:55.085 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.914253409Z level=info msg="Executing migration" id="create index IDX_correlation_source_uid - v2" 2024-09-21T09:59:55.085 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.915011915Z level=info msg="Migration successfully executed" id="create index IDX_correlation_source_uid - v2" duration=758.111µs 2024-09-21T09:59:55.085 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.916991782Z level=info msg="Executing migration" id="create index IDX_correlation_org_id - v2" 2024-09-21T09:59:55.085 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.917669305Z level=info msg="Migration successfully executed" id="create index IDX_correlation_org_id - v2" duration=677.166µs 2024-09-21T09:59:55.085 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.919726181Z level=info msg="Executing migration" id="copy correlation v1 to v2" 2024-09-21T09:59:55.085 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.919896542Z level=info msg="Migration successfully executed" id="copy correlation v1 to v2" duration=170.322µs 2024-09-21T09:59:55.085 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.922454162Z level=info msg="Executing migration" id="drop correlation_tmp_qwerty" 2024-09-21T09:59:55.085 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.923074771Z level=info msg="Migration successfully executed" id="drop correlation_tmp_qwerty" duration=620.385µs 2024-09-21T09:59:55.085 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.923437678Z level=info msg="Executing migration" id="add provisioning column" 2024-09-21T09:59:55.085 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.928733458Z level=info msg="Migration successfully executed" id="add provisioning column" duration=5.295149ms 2024-09-21T09:59:55.085 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.929114936Z level=info msg="Executing migration" id="backfill empty share column fields with default of public" 2024-09-21T09:59:55.086 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.929257648Z level=info msg="Migration successfully executed" id="backfill empty share column fields with default of public" duration=142.354µs 2024-09-21T09:59:55.086 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.929946529Z level=info msg="Executing migration" id="migrate contents column to mediumblob for MySQL" 2024-09-21T09:59:55.086 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.929991364Z level=info msg="Migration successfully executed" id="migrate contents column to mediumblob for MySQL" duration=44.588µs 2024-09-21T09:59:55.086 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.930811958Z level=info msg="Executing migration" id="add origin column to seed_assignment" 2024-09-21T09:59:55.086 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.936081722Z level=info msg="Migration successfully executed" id="add origin column to seed_assignment" duration=5.268783ms 2024-09-21T09:59:55.086 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.936488245Z level=info msg="Executing migration" id="add origin to plugin seed_assignment" 2024-09-21T09:59:55.086 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.936686293Z level=info msg="Migration successfully executed" id="add origin to plugin seed_assignment" duration=197.824µs 2024-09-21T09:59:55.086 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.937329757Z level=info msg="Executing migration" id="prevent seeding OnCall access" 2024-09-21T09:59:55.086 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.937452665Z level=info msg="Migration successfully executed" id="prevent seeding OnCall access" duration=122.83µs 2024-09-21T09:59:55.086 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.938215603Z level=info msg="Executing migration" id="managed folder permissions library panel actions migration" 2024-09-21T09:59:55.086 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.939168473Z level=info msg="Migration successfully executed" id="managed folder permissions library panel actions migration" duration=952.466µs 2024-09-21T09:59:55.086 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.939608927Z level=info msg="Executing migration" id="Remove unique index for folder.title and folder.parent_uid" 2024-09-21T09:59:55.086 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.940323439Z level=info msg="Migration successfully executed" id="Remove unique index for folder.title and folder.parent_uid" duration=715.261µs 2024-09-21T09:59:55.087 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.940663707Z level=info msg="Executing migration" id="Add unique index for title, parent_uid, and org_id" 2024-09-21T09:59:55.087 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.941424367Z level=info msg="Migration successfully executed" id="Add unique index for title, parent_uid, and org_id" duration=760.918µs 2024-09-21T09:59:55.087 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.941809119Z level=info msg="Executing migration" id="Sync dashboard and folder table" 2024-09-21T09:59:55.087 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.942226112Z level=info msg="Migration successfully executed" id="Sync dashboard and folder table" duration=417.346µs 2024-09-21T09:59:55.087 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.942627931Z level=info msg="Executing migration" id="Remove ghost folders from the folder table" 2024-09-21T09:59:55.087 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.942839899Z level=info msg="Migration successfully executed" id="Remove ghost folders from the folder table" duration=212.298µs 2024-09-21T09:59:55.087 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.943531572Z level=info msg="Executing migration" id="Remove unique index UQE_folder_uid_org_id" 2024-09-21T09:59:55.087 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.94423582Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_uid_org_id" duration=705.036µs 2024-09-21T09:59:55.087 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.944600365Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_uid" 2024-09-21T09:59:55.087 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.94537709Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_uid" duration=776.869µs 2024-09-21T09:59:55.087 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.945758125Z level=info msg="Executing migration" id="Remove unique index UQE_folder_title_parent_uid_org_id" 2024-09-21T09:59:55.087 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.946445461Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_title_parent_uid_org_id" duration=686.83µs 2024-09-21T09:59:55.087 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.946802943Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_parent_uid_title" 2024-09-21T09:59:55.087 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.94754698Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_parent_uid_title" duration=744.194µs 2024-09-21T09:59:55.088 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.947917472Z level=info msg="Executing migration" id="Remove index IDX_folder_parent_uid_org_id" 2024-09-21T09:59:55.088 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.948595806Z level=info msg="Migration successfully executed" id="Remove index IDX_folder_parent_uid_org_id" duration=678.979µs 2024-09-21T09:59:55.088 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.948951974Z level=info msg="Executing migration" id="create anon_device table" 2024-09-21T09:59:55.088 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.949527213Z level=info msg="Migration successfully executed" id="create anon_device table" duration=574.978µs 2024-09-21T09:59:55.088 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.949876294Z level=info msg="Executing migration" id="add unique index anon_device.device_id" 2024-09-21T09:59:55.088 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.950558864Z level=info msg="Migration successfully executed" id="add unique index anon_device.device_id" duration=682.312µs 2024-09-21T09:59:55.088 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.950907263Z level=info msg="Executing migration" id="add index anon_device.updated_at" 2024-09-21T09:59:55.089 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.951587221Z level=info msg="Migration successfully executed" id="add index anon_device.updated_at" duration=679.657µs 2024-09-21T09:59:55.089 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.951936818Z level=info msg="Executing migration" id="create signing_key table" 2024-09-21T09:59:55.089 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.952574167Z level=info msg="Migration successfully executed" id="create signing_key table" duration=636.007µs 2024-09-21T09:59:55.089 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.952941239Z level=info msg="Executing migration" id="add unique index signing_key.key_id" 2024-09-21T09:59:55.089 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.953606726Z level=info msg="Migration successfully executed" id="add unique index signing_key.key_id" duration=665.484µs 2024-09-21T09:59:55.089 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.953957172Z level=info msg="Executing migration" id="set legacy alert migration status in kvstore" 2024-09-21T09:59:55.089 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.954591608Z level=info msg="Migration successfully executed" id="set legacy alert migration status in kvstore" duration=634.357µs 2024-09-21T09:59:55.089 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.956870257Z level=info msg="Executing migration" id="migrate record of created folders during legacy migration to kvstore" 2024-09-21T09:59:55.089 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.957278607Z level=info msg="Migration successfully executed" id="migrate record of created folders during legacy migration to kvstore" duration=408.873µs 2024-09-21T09:59:55.089 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.959737567Z level=info msg="Executing migration" id="Add folder_uid for dashboard" 2024-09-21T09:59:55.089 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.965265755Z level=info msg="Migration successfully executed" id="Add folder_uid for dashboard" duration=5.510717ms 2024-09-21T09:59:55.089 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.965691577Z level=info msg="Executing migration" id="Populate dashboard folder_uid column" 2024-09-21T09:59:55.089 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.96763281Z level=info msg="Migration successfully executed" id="Populate dashboard folder_uid column" duration=1.940814ms 2024-09-21T09:59:55.090 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.968537566Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title" 2024-09-21T09:59:55.090 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.968550388Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title" duration=11.846µs 2024-09-21T09:59:55.090 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.971302144Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_id_title" 2024-09-21T09:59:55.090 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.972100102Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_id_title" duration=797.979µs 2024-09-21T09:59:55.090 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.972441863Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_uid_title" 2024-09-21T09:59:55.090 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.972451309Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_uid_title" duration=9.879µs 2024-09-21T09:59:55.090 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.972867403Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" 2024-09-21T09:59:55.090 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.973726849Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" duration=858.685µs 2024-09-21T09:59:55.090 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.974234201Z level=info msg="Executing migration" id="Restore index for dashboard_org_id_folder_id_title" 2024-09-21T09:59:55.090 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.975077461Z level=info msg="Migration successfully executed" id="Restore index for dashboard_org_id_folder_id_title" duration=844.172µs 2024-09-21T09:59:55.090 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.977076471Z level=info msg="Executing migration" id="create sso_setting table" 2024-09-21T09:59:55.090 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.977834401Z level=info msg="Migration successfully executed" id="create sso_setting table" duration=757.777µs 2024-09-21T09:59:55.091 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.980017594Z level=info msg="Executing migration" id="copy kvstore migration status to each org" 2024-09-21T09:59:55.091 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.980671245Z level=info msg="Migration successfully executed" id="copy kvstore migration status to each org" duration=652.969µs 2024-09-21T09:59:55.091 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.981001781Z level=info msg="Executing migration" id="add back entry for orgid=0 migrated status" 2024-09-21T09:59:55.091 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.98138381Z level=info msg="Migration successfully executed" id="add back entry for orgid=0 migrated status" duration=381.845µs 2024-09-21T09:59:55.091 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.981712017Z level=info msg="Executing migration" id="alter kv_store.value to longtext" 2024-09-21T09:59:55.091 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.981750127Z level=info msg="Migration successfully executed" id="alter kv_store.value to longtext" duration=38.397µs 2024-09-21T09:59:55.091 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.984534031Z level=info msg="Executing migration" id="add notification_settings column to alert_rule table" 2024-09-21T09:59:55.091 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.989875659Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule table" duration=5.333773ms 2024-09-21T09:59:55.091 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.990403578Z level=info msg="Executing migration" id="add notification_settings column to alert_rule_version table" 2024-09-21T09:59:55.091 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.996428065Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule_version table" duration=6.02314ms 2024-09-21T09:59:55.091 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.996929274Z level=info msg="Executing migration" id="removing scope from alert.instances:read action migration" 2024-09-21T09:59:55.091 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.997191855Z level=info msg="Migration successfully executed" id="removing scope from alert.instances:read action migration" duration=262.87µs 2024-09-21T09:59:55.091 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=migrator t=2024-09-21T09:59:54.997575839Z level=info msg="migrations completed" performed=69 skipped=482 duration=193.591507ms 2024-09-21T09:59:55.091 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:54 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=sqlstore t=2024-09-21T09:59:54.998347501Z level=info msg="Created default organization" 2024-09-21T09:59:55.092 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=secrets t=2024-09-21T09:59:55.000769189Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-21T09:59:55.092 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=plugin.store t=2024-09-21T09:59:55.016451024Z level=info msg="Loading plugins..." 2024-09-21T09:59:55.092 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=local.finder t=2024-09-21T09:59:55.061422025Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-09-21T09:59:55.092 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=plugin.store t=2024-09-21T09:59:55.061544813Z level=info msg="Plugins loaded" count=55 duration=45.093394ms 2024-09-21T09:59:55.092 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=query_data t=2024-09-21T09:59:55.065262509Z level=info msg="Query Service initialization" 2024-09-21T09:59:55.092 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=live.push_http t=2024-09-21T09:59:55.068032971Z level=info msg="Live Push Gateway initialization" 2024-09-21T09:59:55.092 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=ngalert.migration t=2024-09-21T09:59:55.06909271Z level=info msg=Starting 2024-09-21T09:59:55.092 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=ngalert.state.manager t=2024-09-21T09:59:55.074558848Z level=info msg="Running in alternative execution of Error/NoData mode" 2024-09-21T09:59:55.092 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=infra.usagestats.collector t=2024-09-21T09:59:55.075980492Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-21T09:59:55.092 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=provisioning.datasources t=2024-09-21T09:59:55.078454807Z level=info msg="deleted datasource based on configuration" name=Dashboard1 2024-09-21T09:59:55.092 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=provisioning.datasources t=2024-09-21T09:59:55.078792863Z level=info msg="inserting datasource from configuration" name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-21T09:59:55.253 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T09:59:55.253 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47", 2024-09-21T09:59:55.253 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-09-21T09:59:55.254 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-09-21T09:59:55.254 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [ 2024-09-21T09:59:55.254 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr" 2024-09-21T09:59:55.254 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-09-21T09:59:55.254 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "2/2 daemons upgraded", 2024-09-21T09:59:55.254 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading grafana daemons", 2024-09-21T09:59:55.254 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-09-21T09:59:55.254 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T09:59:55.341 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=provisioning.alerting t=2024-09-21T09:59:55.115796281Z level=info msg="starting to provision alerting" 2024-09-21T09:59:55.341 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=provisioning.alerting t=2024-09-21T09:59:55.115813362Z level=info msg="finished to provision alerting" 2024-09-21T09:59:55.342 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=grafanaStorageLogger t=2024-09-21T09:59:55.116173889Z level=info msg="Storage starting" 2024-09-21T09:59:55.342 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=ngalert.state.manager t=2024-09-21T09:59:55.116345988Z level=info msg="Warming state cache for startup" 2024-09-21T09:59:55.342 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=http.server t=2024-09-21T09:59:55.117999613Z level=info msg="HTTP Server TLS settings" MinTLSVersion=TLS1.2 configuredciphers=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA 2024-09-21T09:59:55.342 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=http.server t=2024-09-21T09:59:55.118248614Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-21T09:59:55.342 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=ngalert.multiorg.alertmanager t=2024-09-21T09:59:55.133939268Z level=info msg="Starting MultiOrg Alertmanager" 2024-09-21T09:59:55.342 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=ngalert.state.manager t=2024-09-21T09:59:55.154400409Z level=info msg="State cache has been initialized" states=0 duration=38.050891ms 2024-09-21T09:59:55.342 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=ngalert.scheduler t=2024-09-21T09:59:55.154450735Z level=info msg="Starting scheduler" tickInterval=10s maxAttempts=1 2024-09-21T09:59:55.342 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=ticker t=2024-09-21T09:59:55.154507214Z level=info msg=starting first_tick=2024-09-21T10:00:00Z 2024-09-21T09:59:55.342 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=provisioning.dashboard t=2024-09-21T09:59:55.160204826Z level=info msg="starting to provision dashboards" 2024-09-21T09:59:55.342 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=grafana.update.checker t=2024-09-21T09:59:55.20034113Z level=info msg="Update check succeeded" duration=84.400532ms 2024-09-21T09:59:55.342 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=plugins.update.checker t=2024-09-21T09:59:55.205486578Z level=info msg="Update check succeeded" duration=89.5642ms 2024-09-21T09:59:55.342 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=sqlstore.transactions t=2024-09-21T09:59:55.240856593Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T09:59:55.342 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=sqlstore.transactions t=2024-09-21T09:59:55.25138843Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-21T09:59:55.343 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=provisioning.dashboard t=2024-09-21T09:59:55.323311815Z level=info msg="finished to provision dashboards" 2024-09-21T09:59:55.513 INFO:tasks.workunit.client.0.smithi070.stdout:migration_abort 2024-09-21T09:59:55.841 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=grafana-apiserver t=2024-09-21T09:59:55.425364945Z level=info msg="Adding GroupVersion playlist.grafana.app v0alpha1 to ResourceManager" 2024-09-21T09:59:55.841 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=grafana-apiserver t=2024-09-21T09:59:55.425948578Z level=info msg="Adding GroupVersion featuretoggle.grafana.app v0alpha1 to ResourceManager" 2024-09-21T09:59:55.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-mon[33203]: from='client.25088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:55.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-mon[33203]: from='client.15201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:55.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-mon[33203]: pgmap v157: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 363 KiB/s rd, 21 KiB/s wr, 463 op/s 2024-09-21T09:59:55.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:55.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:55.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:59:55.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/437709085' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:55.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:55 smithi160 ceph-mon[33203]: osdmap e298: 8 total, 8 up, 8 in 2024-09-21T09:59:55.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:55 smithi070 ceph-mon[41533]: from='client.25088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:55.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:55 smithi070 ceph-mon[41533]: from='client.15201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:55.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:55 smithi070 ceph-mon[41533]: pgmap v157: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 363 KiB/s rd, 21 KiB/s wr, 463 op/s 2024-09-21T09:59:55.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:55 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:55.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:55 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:55.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:55 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:59:55.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:55 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/437709085' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:55.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:55 smithi070 ceph-mon[41533]: osdmap e298: 8 total, 8 up, 8 in 2024-09-21T09:59:55.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:55 smithi070 ceph-mon[29589]: from='client.25088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:55.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:55 smithi070 ceph-mon[29589]: from='client.15201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:55.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:55 smithi070 ceph-mon[29589]: pgmap v157: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 363 KiB/s rd, 21 KiB/s wr, 463 op/s 2024-09-21T09:59:55.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:55 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:55.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:55 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:55.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:55 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:59:55.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:55 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/437709085' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:59:55.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:55 smithi070 ceph-mon[29589]: osdmap e298: 8 total, 8 up, 8 in 2024-09-21T09:59:56.484 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.CloneV1ParentAbortRelinkNotNeeded (3790 ms) 2024-09-21T09:59:56.484 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.CloneV2ParentAbortFixIncompleteChildReattach 2024-09-21T09:59:56.514 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T09:59:56.524 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 0~10 2024-09-21T09:59:56.531 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap1 2024-09-21T09:59:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:56 smithi160 ceph-mon[33203]: from='client.15207 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:56 smithi160 ceph-mon[33203]: from='client.25112 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:56 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:59:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:56 smithi160 ceph-mon[33203]: osdmap e299: 8 total, 8 up, 8 in 2024-09-21T09:59:56.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:56 smithi070 ceph-mon[41533]: from='client.15207 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:56.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:56 smithi070 ceph-mon[41533]: from='client.25112 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:56.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:56 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:59:56.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:56 smithi070 ceph-mon[41533]: osdmap e299: 8 total, 8 up, 8 in 2024-09-21T09:59:56.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:56 smithi070 ceph-mon[29589]: from='client.15207 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:56.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:56 smithi070 ceph-mon[29589]: from='client.25112 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:56.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:56 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:59:56.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:56 smithi070 ceph-mon[29589]: osdmap e299: 8 total, 8 up, 8 in 2024-09-21T09:59:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:57 smithi160 ceph-mon[33203]: pgmap v160: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 172 KiB/s rd, 13 KiB/s wr, 220 op/s 2024-09-21T09:59:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:57 smithi160 ceph-mon[33203]: osdmap e300: 8 total, 8 up, 8 in 2024-09-21T09:59:57.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:57 smithi070 ceph-mon[29589]: pgmap v160: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 172 KiB/s rd, 13 KiB/s wr, 220 op/s 2024-09-21T09:59:57.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:57 smithi070 ceph-mon[29589]: osdmap e300: 8 total, 8 up, 8 in 2024-09-21T09:59:57.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:57 smithi070 ceph-mon[41533]: pgmap v160: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 172 KiB/s rd, 13 KiB/s wr, 220 op/s 2024-09-21T09:59:57.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:57 smithi070 ceph-mon[41533]: osdmap e300: 8 total, 8 up, 8 in 2024-09-21T09:59:58.376 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap1 2024-09-21T09:59:58.431 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T09:59:59.526 INFO:tasks.workunit.client.0.smithi070.stdout:migration_abort 2024-09-21T09:59:59.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:59 smithi070 ceph-mon[41533]: osdmap e301: 8 total, 8 up, 8 in 2024-09-21T09:59:59.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:59 smithi070 ceph-mon[41533]: pgmap v163: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 442 KiB/s rd, 27 KiB/s wr, 565 op/s 2024-09-21T09:59:59.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:59 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:59.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 09:59:59 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:59.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:59 smithi070 ceph-mon[29589]: osdmap e301: 8 total, 8 up, 8 in 2024-09-21T09:59:59.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:59 smithi070 ceph-mon[29589]: pgmap v163: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 442 KiB/s rd, 27 KiB/s wr, 565 op/s 2024-09-21T09:59:59.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:59 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:59.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 09:59:59 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:59.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:59 smithi160 ceph-mon[33203]: osdmap e301: 8 total, 8 up, 8 in 2024-09-21T09:59:59.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:59 smithi160 ceph-mon[33203]: pgmap v163: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 442 KiB/s rd, 27 KiB/s wr, 565 op/s 2024-09-21T09:59:59.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:59 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T09:59:59.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 09:59:59 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:00 smithi160 ceph-mon[33203]: osdmap e302: 8 total, 8 up, 8 in 2024-09-21T10:00:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:00 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:00 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:00 smithi160 ceph-mon[33203]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-21T10:00:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:00 smithi160 ceph-mon[33203]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-21T10:00:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:00 smithi160 ceph-mon[33203]: application not enabled on pool 'test-librbd-smithi070-86016-9' 2024-09-21T10:00:00.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:00 smithi160 ceph-mon[33203]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-21T10:00:00.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:00 smithi070 ceph-mon[29589]: osdmap e302: 8 total, 8 up, 8 in 2024-09-21T10:00:00.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:00 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:00.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:00 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:00.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:00 smithi070 ceph-mon[29589]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-21T10:00:00.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:00 smithi070 ceph-mon[29589]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-21T10:00:00.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:00 smithi070 ceph-mon[29589]: application not enabled on pool 'test-librbd-smithi070-86016-9' 2024-09-21T10:00:00.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:00 smithi070 ceph-mon[29589]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-21T10:00:00.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:00 smithi070 ceph-mon[41533]: osdmap e302: 8 total, 8 up, 8 in 2024-09-21T10:00:00.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:00 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:00.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:00 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:00.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:00 smithi070 ceph-mon[41533]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-21T10:00:00.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:00 smithi070 ceph-mon[41533]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-21T10:00:00.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:00 smithi070 ceph-mon[41533]: application not enabled on pool 'test-librbd-smithi070-86016-9' 2024-09-21T10:00:00.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:00 smithi070 ceph-mon[41533]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-21T10:00:00.718 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.CloneV2ParentAbortFixIncompleteChildReattach (4234 ms) 2024-09-21T10:00:00.719 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.CloneV2ParentAbortFixParentReattach 2024-09-21T10:00:00.748 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T10:00:00.757 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 0~10 2024-09-21T10:00:00.765 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap1 2024-09-21T10:00:01.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:01 smithi070 ceph-mon[29589]: pgmap v165: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 423 KiB/s rd, 26 KiB/s wr, 541 op/s 2024-09-21T10:00:01.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:01 smithi070 ceph-mon[29589]: osdmap e303: 8 total, 8 up, 8 in 2024-09-21T10:00:01.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:01 smithi070 ceph-mon[41533]: pgmap v165: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 423 KiB/s rd, 26 KiB/s wr, 541 op/s 2024-09-21T10:00:01.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:01 smithi070 ceph-mon[41533]: osdmap e303: 8 total, 8 up, 8 in 2024-09-21T10:00:02.081 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:01 smithi160 ceph-mon[33203]: pgmap v165: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 423 KiB/s rd, 26 KiB/s wr, 541 op/s 2024-09-21T10:00:02.081 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:01 smithi160 ceph-mon[33203]: osdmap e303: 8 total, 8 up, 8 in 2024-09-21T10:00:02.622 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap1 2024-09-21T10:00:02.680 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T10:00:02.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:02 smithi070 ceph-mon[41533]: osdmap e304: 8 total, 8 up, 8 in 2024-09-21T10:00:02.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:02 smithi070 ceph-mon[29589]: osdmap e304: 8 total, 8 up, 8 in 2024-09-21T10:00:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:02 smithi160 ceph-mon[33203]: osdmap e304: 8 total, 8 up, 8 in 2024-09-21T10:00:03.499 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:00:03] "GET /metrics HTTP/1.1" 200 38722 "" "Prometheus/2.51.0" 2024-09-21T10:00:03.770 INFO:tasks.workunit.client.0.smithi070.stdout:migration_abort 2024-09-21T10:00:03.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: pgmap v168: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 241 KiB/s rd, 20 KiB/s wr, 312 op/s 2024-09-21T10:00:03.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: osdmap e305: 8 total, 8 up, 8 in 2024-09-21T10:00:03.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:03.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:03.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:00:03.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:00:03.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:03.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T10:00:03.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T10:00:03.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:03.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: Upgrade: Finalizing container_image settings 2024-09-21T10:00:03.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:03.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-21T10:00:03.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-21T10:00:03.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:00:03.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-21T10:00:03.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-21T10:00:03.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-21T10:00:03.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-21T10:00:03.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-21T10:00:03.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-21T10:00:03.845 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-09-21T10:00:03.845 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-21T10:00:03.845 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-09-21T10:00:03.845 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:00:03.845 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-21T10:00:03.845 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-09-21T10:00:03.845 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-21T10:00:03.845 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-09-21T10:00:03.845 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-21T10:00:03.845 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-09-21T10:00:03.845 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:03 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-21T10:00:03.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: pgmap v168: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 241 KiB/s rd, 20 KiB/s wr, 312 op/s 2024-09-21T10:00:03.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: osdmap e305: 8 total, 8 up, 8 in 2024-09-21T10:00:03.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:03.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:03.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:00:03.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:00:03.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:03.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T10:00:03.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T10:00:03.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:03.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: Upgrade: Finalizing container_image settings 2024-09-21T10:00:03.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:03.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-21T10:00:03.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-21T10:00:03.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: pgmap v168: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 241 KiB/s rd, 20 KiB/s wr, 312 op/s 2024-09-21T10:00:03.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: osdmap e305: 8 total, 8 up, 8 in 2024-09-21T10:00:03.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:03.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:03.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:00:03.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:00:03.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:03.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T10:00:03.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T10:00:03.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:03.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:03.882 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: Upgrade: Finalizing container_image settings 2024-09-21T10:00:03.882 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:03.882 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-21T10:00:03.882 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-21T10:00:03.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:00:03.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-21T10:00:03.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-21T10:00:03.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-21T10:00:03.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-21T10:00:03.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-21T10:00:03.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-21T10:00:03.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-09-21T10:00:03.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-21T10:00:03.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-09-21T10:00:03.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:00:03.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-21T10:00:03.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-09-21T10:00:03.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-21T10:00:03.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-09-21T10:00:03.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-21T10:00:03.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-09-21T10:00:03.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-21T10:00:03.883 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:00:03.883 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-21T10:00:03.884 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-21T10:00:03.884 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-21T10:00:03.884 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-21T10:00:03.884 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-21T10:00:03.884 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-21T10:00:03.884 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-09-21T10:00:03.884 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-21T10:00:03.884 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-09-21T10:00:03.884 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:00:03.884 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-21T10:00:03.884 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-09-21T10:00:03.884 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-21T10:00:03.884 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-09-21T10:00:03.884 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-21T10:00:03.884 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-09-21T10:00:03.885 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:03 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-21T10:00:04.463 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.CloneV2ParentAbortFixParentReattach (3745 ms) 2024-09-21T10:00:04.463 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.CloneV2ParentAbortRelinkNotNeeded 2024-09-21T10:00:04.493 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T10:00:04.502 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 0~10 2024-09-21T10:00:04.510 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap1 2024-09-21T10:00:04.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-21T10:00:04.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:00:04.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[41533]: osdmap e306: 8 total, 8 up, 8 in 2024-09-21T10:00:04.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:00:04.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:00:04.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:00:04.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:00:04.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:00:04.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[41533]: Upgrade: Complete! 2024-09-21T10:00:04.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-21T10:00:04.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-21T10:00:04.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:04.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:00:04.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:00:04.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:04.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:04.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:00:04.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:00:04.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:04.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[41533]: osdmap e307: 8 total, 8 up, 8 in 2024-09-21T10:00:04.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[41533]: pgmap v172: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 255 KiB/s rd, 21 KiB/s wr, 330 op/s 2024-09-21T10:00:04.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-21T10:00:04.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:00:04.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[29589]: osdmap e306: 8 total, 8 up, 8 in 2024-09-21T10:00:04.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:00:04.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:00:04.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:00:04.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:00:04.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:00:04.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[29589]: Upgrade: Complete! 2024-09-21T10:00:04.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-21T10:00:04.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-21T10:00:04.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:04.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:00:04.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:00:04.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:04.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:04.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:00:04.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:00:04.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:04.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[29589]: osdmap e307: 8 total, 8 up, 8 in 2024-09-21T10:00:04.881 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:04 smithi070 ceph-mon[29589]: pgmap v172: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 255 KiB/s rd, 21 KiB/s wr, 330 op/s 2024-09-21T10:00:05.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:04 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-21T10:00:05.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:04 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:00:05.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:04 smithi160 ceph-mon[33203]: osdmap e306: 8 total, 8 up, 8 in 2024-09-21T10:00:05.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:04 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:00:05.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:04 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:00:05.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:04 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:00:05.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:04 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:00:05.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:04 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:00:05.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:04 smithi160 ceph-mon[33203]: Upgrade: Complete! 2024-09-21T10:00:05.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:04 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-21T10:00:05.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:04 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-21T10:00:05.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:04 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:05.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:04 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:00:05.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:04 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:00:05.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:04 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:05.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:04 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:05.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:04 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:00:05.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:04 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:00:05.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:04 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:05.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:04 smithi160 ceph-mon[33203]: osdmap e307: 8 total, 8 up, 8 in 2024-09-21T10:00:05.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:04 smithi160 ceph-mon[33203]: pgmap v172: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 255 KiB/s rd, 21 KiB/s wr, 330 op/s 2024-09-21T10:00:06.372 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap1 2024-09-21T10:00:06.431 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T10:00:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:06 smithi070 ceph-mon[29589]: osdmap e308: 8 total, 8 up, 8 in 2024-09-21T10:00:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:06 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:06.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:06 smithi070 ceph-mon[41533]: osdmap e308: 8 total, 8 up, 8 in 2024-09-21T10:00:06.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:06 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:06 smithi160 ceph-mon[33203]: osdmap e308: 8 total, 8 up, 8 in 2024-09-21T10:00:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:06 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:07.553 INFO:tasks.workunit.client.0.smithi070.stdout:migration_abort 2024-09-21T10:00:07.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:07 smithi070 ceph-mon[29589]: osdmap e309: 8 total, 8 up, 8 in 2024-09-21T10:00:07.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:07 smithi070 ceph-mon[29589]: pgmap v175: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 41 KiB/s rd, 6.9 KiB/s wr, 57 op/s 2024-09-21T10:00:07.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:07 smithi070 ceph-mon[41533]: osdmap e309: 8 total, 8 up, 8 in 2024-09-21T10:00:07.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:07 smithi070 ceph-mon[41533]: pgmap v175: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 41 KiB/s rd, 6.9 KiB/s wr, 57 op/s 2024-09-21T10:00:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:07 smithi160 ceph-mon[33203]: osdmap e309: 8 total, 8 up, 8 in 2024-09-21T10:00:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:07 smithi160 ceph-mon[33203]: pgmap v175: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 41 KiB/s rd, 6.9 KiB/s wr, 57 op/s 2024-09-21T10:00:08.504 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.CloneV2ParentAbortRelinkNotNeeded (4040 ms) 2024-09-21T10:00:08.504 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.StressNoMigrate 2024-09-21T10:00:08.534 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T10:00:08.547 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 30370675~5199236 2024-09-21T10:00:08.581 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 450084~8128949 2024-09-21T10:00:08.581 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 75670785~8215295 2024-09-21T10:00:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:08 smithi070 ceph-mon[29589]: osdmap e310: 8 total, 8 up, 8 in 2024-09-21T10:00:08.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:08 smithi070 ceph-mon[41533]: osdmap e310: 8 total, 8 up, 8 in 2024-09-21T10:00:08.634 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 33542640~7176432 2024-09-21T10:00:08.641 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 16979991~8119441 2024-09-21T10:00:08.693 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 33143453~3637225 2024-09-21T10:00:08.696 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 33563431~5649723 2024-09-21T10:00:08.723 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 31356110~23869 2024-09-21T10:00:08.725 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap0 2024-09-21T10:00:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:08 smithi160 ceph-mon[33203]: osdmap e310: 8 total, 8 up, 8 in 2024-09-21T10:00:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:09 smithi160 ceph-mon[33203]: osdmap e311: 8 total, 8 up, 8 in 2024-09-21T10:00:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:09 smithi160 ceph-mon[33203]: pgmap v178: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 324 KiB/s rd, 27 KiB/s wr, 423 op/s 2024-09-21T10:00:09.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:09 smithi070 ceph-mon[29589]: osdmap e311: 8 total, 8 up, 8 in 2024-09-21T10:00:09.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:09 smithi070 ceph-mon[29589]: pgmap v178: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 324 KiB/s rd, 27 KiB/s wr, 423 op/s 2024-09-21T10:00:09.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:09 smithi070 ceph-mon[41533]: osdmap e311: 8 total, 8 up, 8 in 2024-09-21T10:00:09.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:09 smithi070 ceph-mon[41533]: pgmap v178: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 324 KiB/s rd, 27 KiB/s wr, 423 op/s 2024-09-21T10:00:10.397 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap0 2024-09-21T10:00:10.409 INFO:tasks.workunit.client.0.smithi070.stdout:clone image159 -> image162 2024-09-21T10:00:10.505 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 139007671 2024-09-21T10:00:10.517 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 31325969~6322972 2024-09-21T10:00:10.573 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 1546363~7884513 2024-09-21T10:00:10.581 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 133030592~5977079 2024-09-21T10:00:10.623 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 25195249~7325724 2024-09-21T10:00:10.627 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 134093070~4914601 2024-09-21T10:00:10.662 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 92004746~3852700 2024-09-21T10:00:10.662 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 58197921~3084765 2024-09-21T10:00:10.678 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 55082606~5426789 2024-09-21T10:00:10.680 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap1 2024-09-21T10:00:10.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:10 smithi160 ceph-mon[33203]: osdmap e312: 8 total, 8 up, 8 in 2024-09-21T10:00:10.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:10 smithi070 ceph-mon[41533]: osdmap e312: 8 total, 8 up, 8 in 2024-09-21T10:00:10.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:10 smithi070 ceph-mon[29589]: osdmap e312: 8 total, 8 up, 8 in 2024-09-21T10:00:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:11 smithi160 ceph-mon[33203]: osdmap e313: 8 total, 8 up, 8 in 2024-09-21T10:00:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:11 smithi160 ceph-mon[33203]: pgmap v181: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 283 KiB/s rd, 21 KiB/s wr, 367 op/s 2024-09-21T10:00:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:11 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:00:11.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:11 smithi070 ceph-mon[41533]: osdmap e313: 8 total, 8 up, 8 in 2024-09-21T10:00:11.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:11 smithi070 ceph-mon[41533]: pgmap v181: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 283 KiB/s rd, 21 KiB/s wr, 367 op/s 2024-09-21T10:00:11.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:11 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:00:11.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:11 smithi070 ceph-mon[29589]: osdmap e313: 8 total, 8 up, 8 in 2024-09-21T10:00:11.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:11 smithi070 ceph-mon[29589]: pgmap v181: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 283 KiB/s rd, 21 KiB/s wr, 367 op/s 2024-09-21T10:00:11.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:11 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:00:12.410 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap1 2024-09-21T10:00:12.423 INFO:tasks.workunit.client.0.smithi070.stdout:clone image162 -> image164 2024-09-21T10:00:12.551 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 99412136 2024-09-21T10:00:12.566 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 21484654~483839 2024-09-21T10:00:12.610 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 38045009~2525449 2024-09-21T10:00:12.634 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 84042156~2306275 2024-09-21T10:00:12.656 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 31988472~4130064 2024-09-21T10:00:12.682 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 339177~5976850 2024-09-21T10:00:12.718 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 42557929~2829431 2024-09-21T10:00:12.724 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 29675087~1657509 2024-09-21T10:00:12.736 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 88341112~1874090 2024-09-21T10:00:12.740 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap2 2024-09-21T10:00:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:12 smithi160 ceph-mon[33203]: osdmap e314: 8 total, 8 up, 8 in 2024-09-21T10:00:12.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:12 smithi070 ceph-mon[29589]: osdmap e314: 8 total, 8 up, 8 in 2024-09-21T10:00:12.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:12 smithi070 ceph-mon[41533]: osdmap e314: 8 total, 8 up, 8 in 2024-09-21T10:00:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:00:13] "GET /metrics HTTP/1.1" 200 38720 "" "Prometheus/2.51.0" 2024-09-21T10:00:13.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:13 smithi070 ceph-mon[29589]: osdmap e315: 8 total, 8 up, 8 in 2024-09-21T10:00:13.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:13 smithi070 ceph-mon[29589]: pgmap v184: 137 pgs: 137 active+clean; 993 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.6 MiB/s rd, 23 MiB/s wr, 447 op/s 2024-09-21T10:00:13.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:13 smithi070 ceph-mon[41533]: osdmap e315: 8 total, 8 up, 8 in 2024-09-21T10:00:13.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:13 smithi070 ceph-mon[41533]: pgmap v184: 137 pgs: 137 active+clean; 993 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.6 MiB/s rd, 23 MiB/s wr, 447 op/s 2024-09-21T10:00:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:13 smithi160 ceph-mon[33203]: osdmap e315: 8 total, 8 up, 8 in 2024-09-21T10:00:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:13 smithi160 ceph-mon[33203]: pgmap v184: 137 pgs: 137 active+clean; 993 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.6 MiB/s rd, 23 MiB/s wr, 447 op/s 2024-09-21T10:00:14.369 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap2 2024-09-21T10:00:14.381 INFO:tasks.workunit.client.0.smithi070.stdout:clone image164 -> image166 2024-09-21T10:00:14.542 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 144235189 2024-09-21T10:00:14.552 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 57966080~5366533 2024-09-21T10:00:14.602 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 65170469~26517 2024-09-21T10:00:14.604 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 122669333~941537 2024-09-21T10:00:14.615 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 37153738~1499569 2024-09-21T10:00:14.634 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 102658496~6646132 2024-09-21T10:00:14.674 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 7584296~966139 2024-09-21T10:00:14.698 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 117866199~2479039 2024-09-21T10:00:14.715 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 132917910~3105647 2024-09-21T10:00:14.715 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap3 2024-09-21T10:00:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:14 smithi160 ceph-mon[33203]: osdmap e316: 8 total, 8 up, 8 in 2024-09-21T10:00:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:14 smithi160 ceph-mon[33203]: osdmap e317: 8 total, 8 up, 8 in 2024-09-21T10:00:14.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:14 smithi070 ceph-mon[41533]: osdmap e316: 8 total, 8 up, 8 in 2024-09-21T10:00:14.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:14 smithi070 ceph-mon[41533]: osdmap e317: 8 total, 8 up, 8 in 2024-09-21T10:00:14.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:14 smithi070 ceph-mon[29589]: osdmap e316: 8 total, 8 up, 8 in 2024-09-21T10:00:14.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:14 smithi070 ceph-mon[29589]: osdmap e317: 8 total, 8 up, 8 in 2024-09-21T10:00:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:15 smithi160 ceph-mon[33203]: pgmap v187: 137 pgs: 137 active+clean; 993 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.6 MiB/s rd, 23 MiB/s wr, 447 op/s 2024-09-21T10:00:15.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:15 smithi070 ceph-mon[41533]: pgmap v187: 137 pgs: 137 active+clean; 993 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.6 MiB/s rd, 23 MiB/s wr, 447 op/s 2024-09-21T10:00:15.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:15 smithi070 ceph-mon[29589]: pgmap v187: 137 pgs: 137 active+clean; 993 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.6 MiB/s rd, 23 MiB/s wr, 447 op/s 2024-09-21T10:00:16.424 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap3 2024-09-21T10:00:16.435 INFO:tasks.workunit.client.0.smithi070.stdout:clone image166 -> image168 2024-09-21T10:00:16.610 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 84598787 2024-09-21T10:00:16.768 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.StressNoMigrate (8265 ms) 2024-09-21T10:00:16.768 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.Stress 2024-09-21T10:00:16.799 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T10:00:16.809 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 42423493~8319813 2024-09-21T10:00:16.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:16 smithi160 ceph-mon[33203]: osdmap e318: 8 total, 8 up, 8 in 2024-09-21T10:00:16.861 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 2760637~6064705 2024-09-21T10:00:16.861 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 58053863~2858638 2024-09-21T10:00:16.877 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 46675769~8151548 2024-09-21T10:00:16.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:16 smithi070 ceph-mon[29589]: osdmap e318: 8 total, 8 up, 8 in 2024-09-21T10:00:16.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:16 smithi070 ceph-mon[41533]: osdmap e318: 8 total, 8 up, 8 in 2024-09-21T10:00:16.884 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 11130931~2537479 2024-09-21T10:00:16.897 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 38232900~3461824 2024-09-21T10:00:16.900 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 49048561~8108857 2024-09-21T10:00:16.935 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 8939482~4704986 2024-09-21T10:00:16.938 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap0 2024-09-21T10:00:17.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:17 smithi160 ceph-mon[33203]: osdmap e319: 8 total, 8 up, 8 in 2024-09-21T10:00:17.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:17 smithi160 ceph-mon[33203]: pgmap v190: 137 pgs: 137 active+clean; 1.0 GiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 2.3 MiB/s rd, 6.9 MiB/s wr, 104 op/s 2024-09-21T10:00:17.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:17 smithi070 ceph-mon[29589]: osdmap e319: 8 total, 8 up, 8 in 2024-09-21T10:00:17.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:17 smithi070 ceph-mon[29589]: pgmap v190: 137 pgs: 137 active+clean; 1.0 GiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 2.3 MiB/s rd, 6.9 MiB/s wr, 104 op/s 2024-09-21T10:00:17.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:17 smithi070 ceph-mon[41533]: osdmap e319: 8 total, 8 up, 8 in 2024-09-21T10:00:17.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:17 smithi070 ceph-mon[41533]: pgmap v190: 137 pgs: 137 active+clean; 1.0 GiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 2.3 MiB/s rd, 6.9 MiB/s wr, 104 op/s 2024-09-21T10:00:18.435 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap0 2024-09-21T10:00:18.448 INFO:tasks.workunit.client.0.smithi070.stdout:clone image169 -> image172 2024-09-21T10:00:18.549 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 14415384~2050443 2024-09-21T10:00:18.578 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 12434251~6073792 2024-09-21T10:00:18.584 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 73762854~6992915 2024-09-21T10:00:18.628 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 69396565~6524683 2024-09-21T10:00:18.632 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 29902003~1371029 2024-09-21T10:00:18.645 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 37817672~8277370 2024-09-21T10:00:18.660 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 41763007~7077740 2024-09-21T10:00:18.711 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 79142387~4743693 2024-09-21T10:00:18.713 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap1 2024-09-21T10:00:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:18 smithi160 ceph-mon[33203]: osdmap e320: 8 total, 8 up, 8 in 2024-09-21T10:00:18.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:18 smithi070 ceph-mon[29589]: osdmap e320: 8 total, 8 up, 8 in 2024-09-21T10:00:18.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:18 smithi070 ceph-mon[41533]: osdmap e320: 8 total, 8 up, 8 in 2024-09-21T10:00:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:19 smithi160 ceph-mon[33203]: osdmap e321: 8 total, 8 up, 8 in 2024-09-21T10:00:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:19 smithi160 ceph-mon[33203]: pgmap v193: 137 pgs: 137 active+clean; 1.1 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.2 MiB/s rd, 17 MiB/s wr, 603 op/s 2024-09-21T10:00:19.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:19 smithi070 ceph-mon[29589]: osdmap e321: 8 total, 8 up, 8 in 2024-09-21T10:00:19.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:19 smithi070 ceph-mon[29589]: pgmap v193: 137 pgs: 137 active+clean; 1.1 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.2 MiB/s rd, 17 MiB/s wr, 603 op/s 2024-09-21T10:00:19.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:19 smithi070 ceph-mon[41533]: osdmap e321: 8 total, 8 up, 8 in 2024-09-21T10:00:19.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:19 smithi070 ceph-mon[41533]: pgmap v193: 137 pgs: 137 active+clean; 1.1 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.2 MiB/s rd, 17 MiB/s wr, 603 op/s 2024-09-21T10:00:20.451 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 35897977~4087444 2024-09-21T10:00:20.459 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 48567964~3946453 2024-09-21T10:00:20.459 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 57739925~5085868 2024-09-21T10:00:20.465 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 21583333~2914784 2024-09-21T10:00:20.465 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 24396986~6125191 2024-09-21T10:00:20.488 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 30949465~5356992 2024-09-21T10:00:20.493 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 57739083~7047431 2024-09-21T10:00:20.561 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 46675655~3421810 2024-09-21T10:00:20.561 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap2 2024-09-21T10:00:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:20 smithi160 ceph-mon[33203]: osdmap e322: 8 total, 8 up, 8 in 2024-09-21T10:00:20.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:20 smithi070 ceph-mon[29589]: osdmap e322: 8 total, 8 up, 8 in 2024-09-21T10:00:20.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:20 smithi070 ceph-mon[41533]: osdmap e322: 8 total, 8 up, 8 in 2024-09-21T10:00:21.779 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:21 smithi070 ceph-mon[29589]: osdmap e323: 8 total, 8 up, 8 in 2024-09-21T10:00:21.779 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:21 smithi070 ceph-mon[29589]: pgmap v196: 137 pgs: 137 active+clean; 1.1 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.9 MiB/s rd, 10 MiB/s wr, 498 op/s 2024-09-21T10:00:21.779 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:21 smithi070 ceph-mon[41533]: osdmap e323: 8 total, 8 up, 8 in 2024-09-21T10:00:21.779 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:21 smithi070 ceph-mon[41533]: pgmap v196: 137 pgs: 137 active+clean; 1.1 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.9 MiB/s rd, 10 MiB/s wr, 498 op/s 2024-09-21T10:00:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:21 smithi160 ceph-mon[33203]: osdmap e323: 8 total, 8 up, 8 in 2024-09-21T10:00:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:21 smithi160 ceph-mon[33203]: pgmap v196: 137 pgs: 137 active+clean; 1.1 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.9 MiB/s rd, 10 MiB/s wr, 498 op/s 2024-09-21T10:00:22.462 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap2 2024-09-21T10:00:22.474 INFO:tasks.workunit.client.0.smithi070.stdout:clone image172 -> image174 2024-09-21T10:00:22.595 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 153992942 2024-09-21T10:00:22.606 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 19274876~1200546 2024-09-21T10:00:22.623 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 89217419~4369027 2024-09-21T10:00:22.623 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 112870016~4202580 2024-09-21T10:00:22.656 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 149367085~4625857 2024-09-21T10:00:22.656 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 14770305~4987353 2024-09-21T10:00:22.681 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 44327959~6603152 2024-09-21T10:00:22.713 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 131172433~5078531 2024-09-21T10:00:22.745 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 94984149~8147530 2024-09-21T10:00:22.745 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap3 2024-09-21T10:00:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:22 smithi160 ceph-mon[33203]: osdmap e324: 8 total, 8 up, 8 in 2024-09-21T10:00:22.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:22 smithi070 ceph-mon[29589]: osdmap e324: 8 total, 8 up, 8 in 2024-09-21T10:00:22.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:22 smithi070 ceph-mon[41533]: osdmap e324: 8 total, 8 up, 8 in 2024-09-21T10:00:23.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:00:23] "GET /metrics HTTP/1.1" 200 38720 "" "Prometheus/2.51.0" 2024-09-21T10:00:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:23 smithi160 ceph-mon[33203]: osdmap e325: 8 total, 8 up, 8 in 2024-09-21T10:00:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:23 smithi160 ceph-mon[33203]: pgmap v199: 137 pgs: 137 active+clean; 1.2 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 24 MiB/s rd, 35 MiB/s wr, 472 op/s 2024-09-21T10:00:23.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:23 smithi070 ceph-mon[29589]: osdmap e325: 8 total, 8 up, 8 in 2024-09-21T10:00:23.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:23 smithi070 ceph-mon[29589]: pgmap v199: 137 pgs: 137 active+clean; 1.2 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 24 MiB/s rd, 35 MiB/s wr, 472 op/s 2024-09-21T10:00:23.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:23 smithi070 ceph-mon[41533]: osdmap e325: 8 total, 8 up, 8 in 2024-09-21T10:00:23.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:23 smithi070 ceph-mon[41533]: pgmap v199: 137 pgs: 137 active+clean; 1.2 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 24 MiB/s rd, 35 MiB/s wr, 472 op/s 2024-09-21T10:00:24.476 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap3 2024-09-21T10:00:24.485 INFO:tasks.workunit.client.0.smithi070.stdout:clone image174 -> image176 2024-09-21T10:00:24.638 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T10:00:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:24 smithi160 ceph-mon[33203]: osdmap e326: 8 total, 8 up, 8 in 2024-09-21T10:00:24.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:24 smithi070 ceph-mon[29589]: osdmap e326: 8 total, 8 up, 8 in 2024-09-21T10:00:24.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:24 smithi070 ceph-mon[41533]: osdmap e326: 8 total, 8 up, 8 in 2024-09-21T10:00:25.076 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T10:00:25.253 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T10:00:25.553 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.Stress (8785 ms) 2024-09-21T10:00:25.553 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.Stress2 2024-09-21T10:00:25.583 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T10:00:25.594 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 44004507~6229632 2024-09-21T10:00:25.633 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 30626618~4903783 2024-09-21T10:00:25.633 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 61019072~7418459 2024-09-21T10:00:25.679 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 31603111~4119617 2024-09-21T10:00:25.679 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 67572489~5720626 2024-09-21T10:00:25.713 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 82772930~1113150 2024-09-21T10:00:25.713 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 69422560~3461623 2024-09-21T10:00:25.731 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 74939514~3671954 2024-09-21T10:00:25.733 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap0 2024-09-21T10:00:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:25 smithi160 ceph-mon[33203]: osdmap e327: 8 total, 8 up, 8 in 2024-09-21T10:00:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:25 smithi160 ceph-mon[33203]: pgmap v202: 137 pgs: 137 active+clean; 1.2 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 24 MiB/s rd, 35 MiB/s wr, 472 op/s 2024-09-21T10:00:25.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:25 smithi070 ceph-mon[29589]: osdmap e327: 8 total, 8 up, 8 in 2024-09-21T10:00:25.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:25 smithi070 ceph-mon[29589]: pgmap v202: 137 pgs: 137 active+clean; 1.2 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 24 MiB/s rd, 35 MiB/s wr, 472 op/s 2024-09-21T10:00:25.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:25 smithi070 ceph-mon[41533]: osdmap e327: 8 total, 8 up, 8 in 2024-09-21T10:00:25.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:25 smithi070 ceph-mon[41533]: pgmap v202: 137 pgs: 137 active+clean; 1.2 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 24 MiB/s rd, 35 MiB/s wr, 472 op/s 2024-09-21T10:00:26.307 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'echo wait for minority of mons to upgrade' 2024-09-21T10:00:26.508 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:00:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:26 smithi070 ceph-mon[29589]: from='client.24865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:26 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:00:26.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:26 smithi070 ceph-mon[41533]: from='client.24865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:26.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:26 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:00:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:26 smithi160 ceph-mon[33203]: from='client.24865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:26 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:00:27.495 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap0 2024-09-21T10:00:27.525 INFO:tasks.workunit.client.0.smithi070.stdout:clone image177 -> image180 2024-09-21T10:00:27.604 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 167457862 2024-09-21T10:00:27.613 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 114131869~4487671 2024-09-21T10:00:27.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:27 smithi070 ceph-mon[29589]: osdmap e328: 8 total, 8 up, 8 in 2024-09-21T10:00:27.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:27 smithi070 ceph-mon[29589]: pgmap v204: 137 pgs: 137 active+clean; 1.3 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 21 MiB/s rd, 7.4 MiB/s wr, 474 op/s 2024-09-21T10:00:27.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:27 smithi070 ceph-mon[41533]: osdmap e328: 8 total, 8 up, 8 in 2024-09-21T10:00:27.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:27 smithi070 ceph-mon[41533]: pgmap v204: 137 pgs: 137 active+clean; 1.3 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 21 MiB/s rd, 7.4 MiB/s wr, 474 op/s 2024-09-21T10:00:27.648 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 157102298~4673056 2024-09-21T10:00:27.648 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 129143327~1723019 2024-09-21T10:00:27.660 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 62572855~2168681 2024-09-21T10:00:27.695 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 17643713~244448 2024-09-21T10:00:27.702 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 49983284~7925067 2024-09-21T10:00:27.748 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 128369940~3324553 2024-09-21T10:00:27.764 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 1653931~3926823 2024-09-21T10:00:27.770 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap1 2024-09-21T10:00:27.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:27 smithi160 ceph-mon[33203]: osdmap e328: 8 total, 8 up, 8 in 2024-09-21T10:00:27.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:27 smithi160 ceph-mon[33203]: pgmap v204: 137 pgs: 137 active+clean; 1.3 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 21 MiB/s rd, 7.4 MiB/s wr, 474 op/s 2024-09-21T10:00:28.272 INFO:teuthology.orchestra.run.smithi070.stdout:wait for minority of mons to upgrade 2024-09-21T10:00:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:28 smithi160 ceph-mon[33203]: osdmap e329: 8 total, 8 up, 8 in 2024-09-21T10:00:28.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:28 smithi070 ceph-mon[29589]: osdmap e329: 8 total, 8 up, 8 in 2024-09-21T10:00:28.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:28 smithi070 ceph-mon[41533]: osdmap e329: 8 total, 8 up, 8 in 2024-09-21T10:00:28.957 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --limit 1' 2024-09-21T10:00:29.155 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:00:29.504 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap1 2024-09-21T10:00:29.515 INFO:tasks.workunit.client.0.smithi070.stdout:clone image180 -> image182 2024-09-21T10:00:29.538 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:29 smithi070 ceph-mon[41533]: osdmap e330: 8 total, 8 up, 8 in 2024-09-21T10:00:29.539 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:29 smithi070 ceph-mon[41533]: pgmap v207: 137 pgs: 137 active+clean; 1.3 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 44 MiB/s rd, 20 MiB/s wr, 1.15k op/s 2024-09-21T10:00:29.539 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:29 smithi070 ceph-mon[29589]: osdmap e330: 8 total, 8 up, 8 in 2024-09-21T10:00:29.539 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:29 smithi070 ceph-mon[29589]: pgmap v207: 137 pgs: 137 active+clean; 1.3 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 44 MiB/s rd, 20 MiB/s wr, 1.15k op/s 2024-09-21T10:00:29.637 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 241597467 2024-09-21T10:00:29.648 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 52189358~7065065 2024-09-21T10:00:29.713 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 93736460~4979982 2024-09-21T10:00:29.719 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 25191038~2915794 2024-09-21T10:00:29.747 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 140805476~2103303 2024-09-21T10:00:29.753 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 12025889~7883647 2024-09-21T10:00:29.800 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 103188368~6073675 2024-09-21T10:00:29.807 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 111917611~6452787 2024-09-21T10:00:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:29 smithi160 ceph-mon[33203]: osdmap e330: 8 total, 8 up, 8 in 2024-09-21T10:00:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:29 smithi160 ceph-mon[33203]: pgmap v207: 137 pgs: 137 active+clean; 1.3 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 44 MiB/s rd, 20 MiB/s wr, 1.15k op/s 2024-09-21T10:00:29.845 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 102506285~3853798 2024-09-21T10:00:29.848 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap2 2024-09-21T10:00:30.621 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:30 smithi070 ceph-mon[29589]: osdmap e331: 8 total, 8 up, 8 in 2024-09-21T10:00:30.621 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:30 smithi070 ceph-mon[41533]: osdmap e331: 8 total, 8 up, 8 in 2024-09-21T10:00:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:30 smithi160 ceph-mon[33203]: osdmap e331: 8 total, 8 up, 8 in 2024-09-21T10:00:31.521 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap2 2024-09-21T10:00:31.530 INFO:tasks.workunit.client.0.smithi070.stdout:clone image182 -> image184 2024-09-21T10:00:31.674 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 233876200 2024-09-21T10:00:31.690 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 112280835~5197671 2024-09-21T10:00:31.740 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 68478506~2806540 2024-09-21T10:00:31.768 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 174908077~3606811 2024-09-21T10:00:31.795 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 53495970~6332056 2024-09-21T10:00:31.822 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 182770586~6527832 2024-09-21T10:00:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:31 smithi160 ceph-mon[33203]: osdmap e332: 8 total, 8 up, 8 in 2024-09-21T10:00:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:31 smithi160 ceph-mon[33203]: pgmap v210: 137 pgs: 137 active+clean; 1.3 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 23 MiB/s rd, 13 MiB/s wr, 682 op/s 2024-09-21T10:00:31.867 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 28826229~5370666 2024-09-21T10:00:31.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:31 smithi070 ceph-mon[41533]: osdmap e332: 8 total, 8 up, 8 in 2024-09-21T10:00:31.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:31 smithi070 ceph-mon[41533]: pgmap v210: 137 pgs: 137 active+clean; 1.3 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 23 MiB/s rd, 13 MiB/s wr, 682 op/s 2024-09-21T10:00:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:31 smithi070 ceph-mon[29589]: osdmap e332: 8 total, 8 up, 8 in 2024-09-21T10:00:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:31 smithi070 ceph-mon[29589]: pgmap v210: 137 pgs: 137 active+clean; 1.3 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 23 MiB/s rd, 13 MiB/s wr, 682 op/s 2024-09-21T10:00:31.896 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 153544308~5636291 2024-09-21T10:00:31.937 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 181236650~7635563 2024-09-21T10:00:31.940 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap3 2024-09-21T10:00:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:32 smithi160 ceph-mon[33203]: from='client.25124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208", "daemon_types": "mon", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:32 smithi160 ceph-mon[33203]: osdmap e333: 8 total, 8 up, 8 in 2024-09-21T10:00:32.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:32 smithi070 ceph-mon[29589]: from='client.25124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208", "daemon_types": "mon", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:32.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:32 smithi070 ceph-mon[29589]: osdmap e333: 8 total, 8 up, 8 in 2024-09-21T10:00:32.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:32 smithi070 ceph-mon[41533]: from='client.25124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208", "daemon_types": "mon", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:32.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:32 smithi070 ceph-mon[41533]: osdmap e333: 8 total, 8 up, 8 in 2024-09-21T10:00:33.349 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:00:33] "GET /metrics HTTP/1.1" 200 38734 "" "Prometheus/2.51.0" 2024-09-21T10:00:33.382 INFO:teuthology.orchestra.run.smithi070.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:00:33.526 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap3 2024-09-21T10:00:33.537 INFO:tasks.workunit.client.0.smithi070.stdout:clone image184 -> image186 2024-09-21T10:00:33.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:33 smithi070 ceph-mon[29589]: osdmap e334: 8 total, 8 up, 8 in 2024-09-21T10:00:33.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:33 smithi070 ceph-mon[29589]: pgmap v213: 137 pgs: 137 active+clean; 1.5 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.6 MiB/s rd, 21 MiB/s wr, 466 op/s 2024-09-21T10:00:33.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:33 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:33.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:33 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:33.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:33 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:00:33.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:33 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:00:33.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:33 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:33.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:33 smithi070 ceph-mon[41533]: osdmap e334: 8 total, 8 up, 8 in 2024-09-21T10:00:33.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:33 smithi070 ceph-mon[41533]: pgmap v213: 137 pgs: 137 active+clean; 1.5 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.6 MiB/s rd, 21 MiB/s wr, 466 op/s 2024-09-21T10:00:33.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:33 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:33.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:33 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:33.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:33 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:00:33.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:33 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:00:33.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:33 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:33.718 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T10:00:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:33 smithi160 ceph-mon[33203]: osdmap e334: 8 total, 8 up, 8 in 2024-09-21T10:00:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:33 smithi160 ceph-mon[33203]: pgmap v213: 137 pgs: 137 active+clean; 1.5 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.6 MiB/s rd, 21 MiB/s wr, 466 op/s 2024-09-21T10:00:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:33 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:33 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:33 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:00:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:33 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:00:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:33 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:34.301 INFO:tasks.workunit.client.0.smithi070.stdout:write: a 171436458~7529201 2024-09-21T10:00:34.315 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-09-21T10:00:34.379 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 177266478~7183600 2024-09-21T10:00:34.402 INFO:tasks.workunit.client.0.smithi070.stdout:write: a 131843446~7728326 2024-09-21T10:00:34.462 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 160921636~7558497 2024-09-21T10:00:34.484 INFO:tasks.workunit.client.0.smithi070.stdout:write: a 185907363~1829532 2024-09-21T10:00:34.514 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 227287135~3449822 2024-09-21T10:00:34.516 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 10:00:34 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=infra.usagestats t=2024-09-21T10:00:34.122175793Z level=info msg="Usage stats are ready to report" 2024-09-21T10:00:34.543 INFO:tasks.workunit.client.0.smithi070.stdout:write: a 159642437~35838 2024-09-21T10:00:34.545 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 104510498~2364019 2024-09-21T10:00:34.566 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: user0 2024-09-21T10:00:34.569 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:34 smithi070 ceph-mon[29589]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:00:34.569 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:34 smithi070 ceph-mon[29589]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:00:34.569 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:34 smithi070 ceph-mon[29589]: osdmap e335: 8 total, 8 up, 8 in 2024-09-21T10:00:34.569 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:34 smithi070 ceph-mon[41533]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:00:34.569 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:34 smithi070 ceph-mon[41533]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:00:34.570 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:34 smithi070 ceph-mon[41533]: osdmap e335: 8 total, 8 up, 8 in 2024-09-21T10:00:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:34 smithi160 ceph-mon[33203]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:00:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:34 smithi160 ceph-mon[33203]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:00:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:34 smithi160 ceph-mon[33203]: osdmap e335: 8 total, 8 up, 8 in 2024-09-21T10:00:35.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:35 smithi070 ceph-mon[29589]: pgmap v215: 137 pgs: 137 active+clean; 1.5 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.5 MiB/s rd, 21 MiB/s wr, 461 op/s 2024-09-21T10:00:35.520 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:35 smithi070 ceph-mon[41533]: pgmap v215: 137 pgs: 137 active+clean; 1.5 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.5 MiB/s rd, 21 MiB/s wr, 461 op/s 2024-09-21T10:00:35.677 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:00:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:35 smithi160 ceph-mon[33203]: pgmap v215: 137 pgs: 137 active+clean; 1.5 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.5 MiB/s rd, 21 MiB/s wr, 461 op/s 2024-09-21T10:00:36.539 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 283799718 2024-09-21T10:00:36.553 INFO:tasks.workunit.client.0.smithi070.stdout:write: b 267222174~584863 2024-09-21T10:00:36.559 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 124642323~5115857 2024-09-21T10:00:36.559 INFO:tasks.workunit.client.0.smithi070.stdout:write: b 257995017~1507690 2024-09-21T10:00:36.559 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 26747106~1379669 2024-09-21T10:00:36.560 INFO:tasks.workunit.client.0.smithi070.stdout:write: b 42569664~2637631 2024-09-21T10:00:36.563 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 81590613~4181272 2024-09-21T10:00:36.565 INFO:tasks.workunit.client.0.smithi070.stdout:write: b 138952548~8315185 2024-09-21T10:00:36.577 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 77044949~7747791 2024-09-21T10:00:36.680 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: user1 2024-09-21T10:00:36.831 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:36 smithi070 ceph-mon[29589]: osdmap e336: 8 total, 8 up, 8 in 2024-09-21T10:00:36.832 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:36 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:36.832 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:36 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:36.832 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:36 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:36.832 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:36 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:36.832 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:36 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-21T10:00:36.832 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:36 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-21T10:00:36.832 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:36 smithi070 ceph-mon[41533]: osdmap e336: 8 total, 8 up, 8 in 2024-09-21T10:00:36.832 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:36 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:36.832 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:36 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:36.832 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:36 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:36.832 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:36 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:36.833 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:36 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-21T10:00:36.833 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:36 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-21T10:00:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:36 smithi160 ceph-mon[33203]: osdmap e336: 8 total, 8 up, 8 in 2024-09-21T10:00:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:36 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:36 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:36 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:36 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:36 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-21T10:00:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:36 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-21T10:00:37.511 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-09-21T10:00:37.787 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:37 smithi070 ceph-mon[29589]: Upgrade: Target is version 19.3.0-5092-g62da54b4 (squid) 2024-09-21T10:00:37.788 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:37 smithi070 ceph-mon[29589]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, digests ['quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47'] 2024-09-21T10:00:37.788 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:37 smithi070 ceph-mon[29589]: Upgrade: Setting container_image for all mgr 2024-09-21T10:00:37.788 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:37 smithi070 ceph-mon[29589]: Upgrade: It appears safe to stop mon.a 2024-09-21T10:00:37.788 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:37 smithi070 ceph-mon[29589]: osdmap e337: 8 total, 8 up, 8 in 2024-09-21T10:00:37.788 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:37 smithi070 ceph-mon[29589]: pgmap v218: 137 pgs: 137 active+clean; 1.6 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 31 MiB/s rd, 33 MiB/s wr, 912 op/s 2024-09-21T10:00:37.788 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:37 smithi070 ceph-mon[41533]: Upgrade: Target is version 19.3.0-5092-g62da54b4 (squid) 2024-09-21T10:00:37.788 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:37 smithi070 ceph-mon[41533]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, digests ['quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47'] 2024-09-21T10:00:37.788 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:37 smithi070 ceph-mon[41533]: Upgrade: Setting container_image for all mgr 2024-09-21T10:00:37.788 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:37 smithi070 ceph-mon[41533]: Upgrade: It appears safe to stop mon.a 2024-09-21T10:00:37.788 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:37 smithi070 ceph-mon[41533]: osdmap e337: 8 total, 8 up, 8 in 2024-09-21T10:00:37.789 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:37 smithi070 ceph-mon[41533]: pgmap v218: 137 pgs: 137 active+clean; 1.6 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 31 MiB/s rd, 33 MiB/s wr, 912 op/s 2024-09-21T10:00:37.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:37 smithi160 ceph-mon[33203]: Upgrade: Target is version 19.3.0-5092-g62da54b4 (squid) 2024-09-21T10:00:37.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:37 smithi160 ceph-mon[33203]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, digests ['quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47'] 2024-09-21T10:00:37.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:37 smithi160 ceph-mon[33203]: Upgrade: Setting container_image for all mgr 2024-09-21T10:00:37.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:37 smithi160 ceph-mon[33203]: Upgrade: It appears safe to stop mon.a 2024-09-21T10:00:37.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:37 smithi160 ceph-mon[33203]: osdmap e337: 8 total, 8 up, 8 in 2024-09-21T10:00:37.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:37 smithi160 ceph-mon[33203]: pgmap v218: 137 pgs: 137 active+clean; 1.6 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 31 MiB/s rd, 33 MiB/s wr, 912 op/s 2024-09-21T10:00:38.089 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T10:00:38.089 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (64s) 60s ago 6m 16.5M - 0.27.0 11f11916f8cd 89ae95182419 2024-09-21T10:00:38.089 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi160 *:3000 running (43s) 39s ago 6m 42.1M - 10.4.8 5aad1d7cf004 daa2d8dfab48 2024-09-21T10:00:38.089 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi070.gansda smithi070 running (5m) 60s ago 5m 17.9M - 18.2.4-871-g6a1bdf85 0adab276c97e 6016b2501c8a 2024-09-21T10:00:38.089 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi160.byyhfz smithi160 running (5m) 39s ago 5m 15.1M - 18.2.4-871-g6a1bdf85 0adab276c97e 93e108d91e65 2024-09-21T10:00:38.089 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi160 *:8443,9283,8765 running (2m) 39s ago 10m 454M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 d952652a9683 2024-09-21T10:00:38.089 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (2m) 60s ago 12m 517M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 e3d390893cff 2024-09-21T10:00:38.090 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (12m) 60s ago 12m 58.1M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e cee8ca713d09 2024-09-21T10:00:38.090 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi160 running (11m) 39s ago 11m 56.9M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e 0fd16436dfa0 2024-09-21T10:00:38.090 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (11m) 60s ago 11m 48.6M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e fc0590f324b5 2024-09-21T10:00:38.090 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (114s) 60s ago 6m 9625k - 1.7.0 72c9c2088986 5d5d4b026070 2024-09-21T10:00:38.090 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi160 *:9100 running (108s) 39s ago 6m 11.7M - 1.7.0 72c9c2088986 19b8e0e8ee0c 2024-09-21T10:00:38.090 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (10m) 60s ago 10m 125M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 10580e80f007 2024-09-21T10:00:38.090 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (10m) 60s ago 10m 99.7M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e b7c0d1169d1d 2024-09-21T10:00:38.090 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (9m) 60s ago 9m 106M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 58a11b8ddb81 2024-09-21T10:00:38.090 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (9m) 60s ago 9m 99.2M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e aa0c8a732a41 2024-09-21T10:00:38.090 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi160 running (8m) 39s ago 8m 119M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 7325a075c488 2024-09-21T10:00:38.090 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi160 running (8m) 39s ago 8m 94.7M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e f74026345641 2024-09-21T10:00:38.090 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi160 running (8m) 39s ago 8m 142M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 03e6bb54f8b2 2024-09-21T10:00:38.090 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi160 running (7m) 39s ago 7m 114M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e e405b21a6146 2024-09-21T10:00:38.091 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi160 *:9095 running (85s) 39s ago 7m 38.5M - 2.51.0 1d3b7f56885b 648952e57ad7 2024-09-21T10:00:38.434 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:00:38.435 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-09-21T10:00:38.435 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 3 2024-09-21T10:00:38.435 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:00:38.435 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-09-21T10:00:38.435 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T10:00:38.435 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:00:38.435 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-09-21T10:00:38.435 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 8 2024-09-21T10:00:38.435 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:00:38.435 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": { 2024-09-21T10:00:38.435 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 2 2024-09-21T10:00:38.435 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:00:38.436 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-09-21T10:00:38.436 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 13, 2024-09-21T10:00:38.436 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T10:00:38.436 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:00:38.436 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:00:38.549 INFO:tasks.workunit.client.0.smithi070.stdout:write: c 216214065~7087509 2024-09-21T10:00:38.569 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 21661194~3597465 2024-09-21T10:00:38.570 INFO:tasks.workunit.client.0.smithi070.stdout:write: c 178321314~6915443 2024-09-21T10:00:38.579 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 90083840~2681984 2024-09-21T10:00:38.579 INFO:tasks.workunit.client.0.smithi070.stdout:write: c 70049734~6346130 2024-09-21T10:00:38.663 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 16253278~7059709 2024-09-21T10:00:38.680 INFO:tasks.workunit.client.0.smithi070.stdout:write: c 126321306~7933885 2024-09-21T10:00:38.704 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 154889639~1384763 2024-09-21T10:00:38.704 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: user2 2024-09-21T10:00:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:38 smithi160 ceph-mon[33203]: from='client.25130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:38 smithi160 ceph-mon[33203]: osdmap e338: 8 total, 8 up, 8 in 2024-09-21T10:00:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:38 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/362880418' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:38.842 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:00:38.843 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47", 2024-09-21T10:00:38.843 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-09-21T10:00:38.843 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) mon. Upgrade limited to 1 daemons (1 remaining).", 2024-09-21T10:00:38.843 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-09-21T10:00:38.843 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "0/3 daemons upgraded", 2024-09-21T10:00:38.843 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading mon daemons", 2024-09-21T10:00:38.843 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-09-21T10:00:38.843 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:00:38.843 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:38 smithi070 ceph-mon[29589]: from='client.25130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:38.843 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:38 smithi070 ceph-mon[29589]: osdmap e338: 8 total, 8 up, 8 in 2024-09-21T10:00:38.844 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:38 smithi070 ceph-mon[29589]: from='client.? 172.21.15.70:0/362880418' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:38.844 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:38 smithi070 ceph-mon[41533]: from='client.25130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:38.844 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:38 smithi070 ceph-mon[41533]: osdmap e338: 8 total, 8 up, 8 in 2024-09-21T10:00:38.844 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:38 smithi070 ceph-mon[41533]: from='client.? 172.21.15.70:0/362880418' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:00:39.758 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:39 smithi070 ceph-mon[29589]: from='client.25136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:39.758 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:39 smithi070 ceph-mon[29589]: from='client.25142 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:39.758 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:39 smithi070 ceph-mon[29589]: osdmap e339: 8 total, 8 up, 8 in 2024-09-21T10:00:39.758 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:39 smithi070 ceph-mon[29589]: pgmap v221: 137 pgs: 137 active+clean; 1.6 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 31 MiB/s rd, 34 MiB/s wr, 920 op/s 2024-09-21T10:00:39.758 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:39 smithi070 ceph-mon[29589]: Upgrade: Updating mon.a 2024-09-21T10:00:39.758 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:39 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:39.758 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:39 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T10:00:39.758 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:39 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T10:00:39.759 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:39 smithi070 ceph-mon[29589]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:00:39.759 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:39 smithi070 ceph-mon[29589]: Deploying daemon mon.a on smithi070 2024-09-21T10:00:39.759 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:39 smithi070 ceph-mon[41533]: from='client.25136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:39.759 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:39 smithi070 ceph-mon[41533]: from='client.25142 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:39.759 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:39 smithi070 ceph-mon[41533]: osdmap e339: 8 total, 8 up, 8 in 2024-09-21T10:00:39.759 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:39 smithi070 ceph-mon[41533]: pgmap v221: 137 pgs: 137 active+clean; 1.6 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 31 MiB/s rd, 34 MiB/s wr, 920 op/s 2024-09-21T10:00:39.759 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:39 smithi070 ceph-mon[41533]: Upgrade: Updating mon.a 2024-09-21T10:00:39.759 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:39 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:39.759 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:39 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T10:00:39.759 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:39 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T10:00:39.759 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:39 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:00:39.760 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:39 smithi070 ceph-mon[41533]: Deploying daemon mon.a on smithi070 2024-09-21T10:00:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:39 smithi160 ceph-mon[33203]: from='client.25136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:39 smithi160 ceph-mon[33203]: from='client.25142 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:39 smithi160 ceph-mon[33203]: osdmap e339: 8 total, 8 up, 8 in 2024-09-21T10:00:39.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:39 smithi160 ceph-mon[33203]: pgmap v221: 137 pgs: 137 active+clean; 1.6 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 31 MiB/s rd, 34 MiB/s wr, 920 op/s 2024-09-21T10:00:39.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:39 smithi160 ceph-mon[33203]: Upgrade: Updating mon.a 2024-09-21T10:00:39.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:39 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' 2024-09-21T10:00:39.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:39 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T10:00:39.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:39 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T10:00:39.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:39 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:00:39.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:39 smithi160 ceph-mon[33203]: Deploying daemon mon.a on smithi070 2024-09-21T10:00:40.586 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 393457719 2024-09-21T10:00:40.600 INFO:tasks.workunit.client.0.smithi070.stdout:write: d 265934515~3989432 2024-09-21T10:00:40.610 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 90665171~1488520 2024-09-21T10:00:40.610 INFO:tasks.workunit.client.0.smithi070.stdout:write: d 273778028~6104727 2024-09-21T10:00:40.618 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 324190078~6926078 2024-09-21T10:00:40.618 INFO:tasks.workunit.client.0.smithi070.stdout:write: d 45416090~2041721 2024-09-21T10:00:40.619 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 146597054~3168443 2024-09-21T10:00:40.619 INFO:tasks.workunit.client.0.smithi070.stdout:write: d 53005850~6286912 2024-09-21T10:00:40.628 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 245572259~5849201 2024-09-21T10:00:40.634 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: user3 2024-09-21T10:00:40.832 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:40 smithi070 systemd[1]: Stopping Ceph mon.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:00:40.832 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:40 smithi070 ceph-mon[29589]: from='client.25154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:40.832 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:40 smithi070 ceph-mon[29589]: osdmap e340: 8 total, 8 up, 8 in 2024-09-21T10:00:40.832 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:40 smithi070 ceph-mon[41533]: from='client.25154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:40.832 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:40 smithi070 ceph-mon[41533]: osdmap e340: 8 total, 8 up, 8 in 2024-09-21T10:00:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:40 smithi160 ceph-mon[33203]: from='client.25154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:40 smithi160 ceph-mon[33203]: osdmap e340: 8 total, 8 up, 8 in 2024-09-21T10:00:41.086 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:40 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:00:40] ENGINE Bus STOPPING 2024-09-21T10:00:41.086 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:40 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a[29566]: 2024-09-21T10:00:40.828+0000 7f7e36ca9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:00:41.086 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:40 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a[29566]: 2024-09-21T10:00:40.828+0000 7f7e36ca9640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-21T10:00:41.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:41 smithi070 podman[107185]: 2024-09-21 10:00:41.084704951 +0000 UTC m=+0.484961271 container died cee8ca713d09bd95be57c60ff1eb7e4dd7cbb22f891f84b25280cbac97388331 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=reef-6a1bdf8, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-21T10:00:41.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:41 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:00:41] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T10:00:41.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:41 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:00:41] ENGINE Bus STOPPED 2024-09-21T10:00:41.378 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:41 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:00:41] ENGINE Bus STARTING 2024-09-21T10:00:41.378 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:41 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:00:41] ENGINE Serving on http://:::9283 2024-09-21T10:00:41.378 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:41 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:00:41] ENGINE Bus STARTED 2024-09-21T10:00:41.869 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:41 smithi070 podman[107185]: 2024-09-21 10:00:41.511026158 +0000 UTC m=+0.911282468 container cleanup cee8ca713d09bd95be57c60ff1eb7e4dd7cbb22f891f84b25280cbac97388331 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-6a1bdf8, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-21T10:00:41.869 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:41 smithi070 bash[107185]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a 2024-09-21T10:00:41.869 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:41 smithi070 podman[107201]: 2024-09-21 10:00:41.737559504 +0000 UTC m=+0.650297136 container remove cee8ca713d09bd95be57c60ff1eb7e4dd7cbb22f891f84b25280cbac97388331 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-6a1bdf8, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD) 2024-09-21T10:00:42.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:41 smithi070 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-21T10:00:42.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:41 smithi070 systemd[1]: Stopped Ceph mon.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:00:42.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:41 smithi070 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mon.a.service: Consumed 16.748s CPU time. 2024-09-21T10:00:42.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 systemd[1]: Starting Ceph mon.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:00:42.559 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 podman[107347]: 2024-09-21 10:00:42.195203533 +0000 UTC m=+0.015529680 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:00:42.559 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 podman[107347]: 2024-09-21 10:00:42.330095995 +0000 UTC m=+0.150422141 container create 4a7b667a403311f40b95be0411d7385e1161f266e1871482d304082744d55eed (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b, CEPH_POINT_RELEASE=) 2024-09-21T10:00:42.559 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 podman[107347]: 2024-09-21 10:00:42.557941328 +0000 UTC m=+0.378267475 container init 4a7b667a403311f40b95be0411d7385e1161f266e1871482d304082744d55eed (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-21T10:00:42.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 podman[107347]: 2024-09-21 10:00:42.562232113 +0000 UTC m=+0.382558263 container start 4a7b667a403311f40b95be0411d7385e1161f266e1871482d304082744d55eed (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a, GIT_CLEAN=True, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-21T10:00:42.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T10:00:42.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev), process ceph-mon, pid 7 2024-09-21T10:00:42.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: pidfile_write: ignore empty --pid-file 2024-09-21T10:00:42.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: load: jerasure load: lrc 2024-09-21T10:00:42.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: RocksDB version: 7.9.2 2024-09-21T10:00:42.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Git sha 0 2024-09-21T10:00:42.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Compile date 2024-09-20 20:32:28 2024-09-21T10:00:42.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: DB SUMMARY 2024-09-21T10:00:42.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: DB Session ID: ARBALPM1J5MTYNYEA301 2024-09-21T10:00:42.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: CURRENT file: CURRENT 2024-09-21T10:00:42.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T10:00:42.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1520 Bytes 2024-09-21T10:00:42.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000032.sst 2024-09-21T10:00:42.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000030.log size: 6072830 ; 2024-09-21T10:00:42.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.error_if_exists: 0 2024-09-21T10:00:42.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.create_if_missing: 0 2024-09-21T10:00:42.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.paranoid_checks: 1 2024-09-21T10:00:42.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T10:00:42.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T10:00:42.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T10:00:42.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.env: 0x55aebf4c5120 2024-09-21T10:00:42.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T10:00:42.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.info_log: 0x55aec0cc3700 2024-09-21T10:00:42.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T10:00:42.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.statistics: (nil) 2024-09-21T10:00:42.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.use_fsync: 0 2024-09-21T10:00:42.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.max_log_file_size: 0 2024-09-21T10:00:42.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T10:00:42.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T10:00:42.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T10:00:42.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T10:00:42.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.allow_fallocate: 1 2024-09-21T10:00:42.881 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T10:00:42.881 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T10:00:42.881 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.use_direct_reads: 0 2024-09-21T10:00:42.881 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T10:00:42.881 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T10:00:42.881 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.db_log_dir: 2024-09-21T10:00:42.881 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.wal_dir: 2024-09-21T10:00:42.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T10:00:42.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T10:00:42.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T10:00:42.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T10:00:42.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T10:00:42.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T10:00:42.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T10:00:42.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T10:00:42.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.write_buffer_manager: 0x55aec0cc61e0 2024-09-21T10:00:42.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T10:00:42.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T10:00:42.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T10:00:42.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T10:00:42.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T10:00:42.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T10:00:42.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T10:00:42.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T10:00:42.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.unordered_write: 0 2024-09-21T10:00:42.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T10:00:42.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T10:00:42.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T10:00:42.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T10:00:42.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.row_cache: None 2024-09-21T10:00:42.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.wal_filter: None 2024-09-21T10:00:42.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T10:00:42.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T10:00:42.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.two_write_queues: 0 2024-09-21T10:00:42.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T10:00:42.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.wal_compression: 0 2024-09-21T10:00:42.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.atomic_flush: 0 2024-09-21T10:00:42.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T10:00:42.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T10:00:42.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T10:00:42.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.log_readahead_size: 0 2024-09-21T10:00:42.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T10:00:42.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T10:00:42.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T10:00:42.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T10:00:42.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T10:00:42.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T10:00:42.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T10:00:42.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.max_background_jobs: 2 2024-09-21T10:00:42.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.max_background_compactions: -1 2024-09-21T10:00:42.885 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.max_subcompactions: 1 2024-09-21T10:00:42.885 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T10:00:42.885 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T10:00:42.885 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T10:00:42.885 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T10:00:42.885 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T10:00:42.885 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T10:00:42.885 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T10:00:42.885 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T10:00:42.885 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.max_open_files: -1 2024-09-21T10:00:42.885 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T10:00:42.885 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T10:00:42.885 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T10:00:42.885 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T10:00:42.885 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.max_background_flushes: -1 2024-09-21T10:00:42.886 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Compression algorithms supported: 2024-09-21T10:00:42.886 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: kZSTD supported: 0 2024-09-21T10:00:42.886 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: kXpressCompression supported: 0 2024-09-21T10:00:42.886 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: kBZip2Compression supported: 0 2024-09-21T10:00:42.886 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T10:00:42.886 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: kLZ4Compression supported: 1 2024-09-21T10:00:42.886 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: kZlibCompression supported: 1 2024-09-21T10:00:42.886 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T10:00:42.886 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: kSnappyCompression supported: 1 2024-09-21T10:00:42.886 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T10:00:42.886 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T10:00:42.886 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-09-21T10:00:42.886 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T10:00:42.886 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T10:00:42.887 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.merge_operator: 2024-09-21T10:00:42.887 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.compaction_filter: None 2024-09-21T10:00:42.887 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T10:00:42.887 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T10:00:42.887 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T10:00:42.887 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T10:00:42.887 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55aec0cc3320) 2024-09-21T10:00:42.887 INFO:journalctl@ceph.mon.a.smithi070.stdout: cache_index_and_filter_blocks: 1 2024-09-21T10:00:42.887 INFO:journalctl@ceph.mon.a.smithi070.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T10:00:42.887 INFO:journalctl@ceph.mon.a.smithi070.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T10:00:42.887 INFO:journalctl@ceph.mon.a.smithi070.stdout: pin_top_level_index_and_filter: 1 2024-09-21T10:00:42.887 INFO:journalctl@ceph.mon.a.smithi070.stdout: index_type: 0 2024-09-21T10:00:42.887 INFO:journalctl@ceph.mon.a.smithi070.stdout: data_block_index_type: 0 2024-09-21T10:00:42.887 INFO:journalctl@ceph.mon.a.smithi070.stdout: index_shortening: 1 2024-09-21T10:00:42.887 INFO:journalctl@ceph.mon.a.smithi070.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T10:00:42.888 INFO:journalctl@ceph.mon.a.smithi070.stdout: checksum: 4 2024-09-21T10:00:42.888 INFO:journalctl@ceph.mon.a.smithi070.stdout: no_block_cache: 0 2024-09-21T10:00:42.888 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_cache: 0x55aec0cb7610 2024-09-21T10:00:42.888 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_cache_name: BinnedLRUCache 2024-09-21T10:00:42.888 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_cache_options: 2024-09-21T10:00:42.888 INFO:journalctl@ceph.mon.a.smithi070.stdout: capacity : 536870912 2024-09-21T10:00:42.888 INFO:journalctl@ceph.mon.a.smithi070.stdout: num_shard_bits : 4 2024-09-21T10:00:42.888 INFO:journalctl@ceph.mon.a.smithi070.stdout: strict_capacity_limit : 0 2024-09-21T10:00:42.888 INFO:journalctl@ceph.mon.a.smithi070.stdout: high_pri_pool_ratio: 0.000 2024-09-21T10:00:42.888 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_cache_compressed: (nil) 2024-09-21T10:00:42.888 INFO:journalctl@ceph.mon.a.smithi070.stdout: persistent_cache: (nil) 2024-09-21T10:00:42.888 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_size: 4096 2024-09-21T10:00:42.888 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_size_deviation: 10 2024-09-21T10:00:42.888 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_restart_interval: 16 2024-09-21T10:00:42.888 INFO:journalctl@ceph.mon.a.smithi070.stdout: index_block_restart_interval: 1 2024-09-21T10:00:42.888 INFO:journalctl@ceph.mon.a.smithi070.stdout: metadata_block_size: 4096 2024-09-21T10:00:42.889 INFO:journalctl@ceph.mon.a.smithi070.stdout: partition_filters: 0 2024-09-21T10:00:42.889 INFO:journalctl@ceph.mon.a.smithi070.stdout: use_delta_encoding: 1 2024-09-21T10:00:42.889 INFO:journalctl@ceph.mon.a.smithi070.stdout: filter_policy: bloomfilter 2024-09-21T10:00:42.889 INFO:journalctl@ceph.mon.a.smithi070.stdout: whole_key_filtering: 1 2024-09-21T10:00:42.889 INFO:journalctl@ceph.mon.a.smithi070.stdout: verify_compression: 0 2024-09-21T10:00:42.889 INFO:journalctl@ceph.mon.a.smithi070.stdout: read_amp_bytes_per_bit: 0 2024-09-21T10:00:42.889 INFO:journalctl@ceph.mon.a.smithi070.stdout: format_version: 5 2024-09-21T10:00:42.889 INFO:journalctl@ceph.mon.a.smithi070.stdout: enable_index_compression: 1 2024-09-21T10:00:42.889 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_align: 0 2024-09-21T10:00:42.889 INFO:journalctl@ceph.mon.a.smithi070.stdout: max_auto_readahead_size: 262144 2024-09-21T10:00:42.889 INFO:journalctl@ceph.mon.a.smithi070.stdout: prepopulate_block_cache: 0 2024-09-21T10:00:42.889 INFO:journalctl@ceph.mon.a.smithi070.stdout: initial_auto_readahead_size: 8192 2024-09-21T10:00:42.889 INFO:journalctl@ceph.mon.a.smithi070.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T10:00:42.889 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T10:00:42.889 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T10:00:42.890 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.compression: NoCompression 2024-09-21T10:00:42.890 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T10:00:42.890 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T10:00:42.890 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T10:00:42.890 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.num_levels: 7 2024-09-21T10:00:42.890 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T10:00:42.890 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T10:00:42.890 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T10:00:42.890 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T10:00:42.890 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T10:00:42.890 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T10:00:42.890 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T10:00:42.890 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T10:00:42.890 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T10:00:42.891 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T10:00:42.891 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T10:00:42.891 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T10:00:42.891 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T10:00:42.891 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T10:00:42.891 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T10:00:42.891 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T10:00:42.891 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T10:00:42.891 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T10:00:42.891 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T10:00:42.891 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T10:00:42.891 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T10:00:42.891 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T10:00:42.891 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T10:00:42.891 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T10:00:42.892 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T10:00:42.892 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T10:00:42.892 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T10:00:42.892 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T10:00:42.892 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T10:00:42.892 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T10:00:42.892 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T10:00:42.892 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T10:00:42.892 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T10:00:42.892 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T10:00:42.892 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T10:00:42.892 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T10:00:42.892 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T10:00:42.892 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T10:00:42.893 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T10:00:42.893 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T10:00:42.893 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T10:00:42.893 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T10:00:42.893 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T10:00:42.893 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T10:00:42.893 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T10:00:42.893 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T10:00:42.893 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T10:00:42.893 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T10:00:42.893 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T10:00:42.893 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T10:00:42.893 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T10:00:42.893 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T10:00:42.894 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T10:00:42.894 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T10:00:42.894 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.inplace_update_support: 0 2024-09-21T10:00:42.894 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T10:00:42.894 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T10:00:42.894 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T10:00:42.894 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T10:00:42.894 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.bloom_locality: 0 2024-09-21T10:00:42.894 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.max_successive_merges: 0 2024-09-21T10:00:42.894 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T10:00:42.894 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T10:00:42.894 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T10:00:42.894 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T10:00:42.895 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.ttl: 2592000 2024-09-21T10:00:42.895 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T10:00:42.895 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T10:00:42.895 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T10:00:42.895 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.enable_blob_files: false 2024-09-21T10:00:42.895 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.min_blob_size: 0 2024-09-21T10:00:42.895 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T10:00:42.895 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T10:00:42.895 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T10:00:42.895 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T10:00:42.895 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T10:00:42.895 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T10:00:42.895 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T10:00:42.895 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T10:00:42.895 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-21T10:00:42.896 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 34, last_sequence is 12816, log_number is 30,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 30 2024-09-21T10:00:42.896 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 30 2024-09-21T10:00:42.896 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 100e7b8e-fa1d-414c-8cff-ec7e6137fb9f 2024-09-21T10:00:42.896 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912842588642, "job": 1, "event": "recovery_started", "wal_files": [30]} 2024-09-21T10:00:42.896 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #30 mode 2 2024-09-21T10:00:42.896 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912842605636, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 35, "file_size": 5648936, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12818, "largest_seqno": 16833, "table_properties": {"data_size": 5629673, "index_size": 13608, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 4613, "raw_key_size": 41834, "raw_average_key_size": 22, "raw_value_size": 5589897, "raw_average_value_size": 3042, "num_data_blocks": 589, "num_entries": 1837, "num_filter_entries": 1837, "num_deletions": 268, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726912842, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "100e7b8e-fa1d-414c-8cff-ec7e6137fb9f", "db_session_id": "ARBALPM1J5MTYNYEA301", "orig_file_number": 35, "seqno_to_time_mapping": "N/A"}} 2024-09-21T10:00:42.896 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912842605780, "job": 1, "event": "recovery_finished"} 2024-09-21T10:00:42.896 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: [db/version_set.cc:5047] Creating manifest 37 2024-09-21T10:00:42.896 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-21T10:00:42.896 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000030.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T10:00:42.896 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55aec0ce6e00 2024-09-21T10:00:42.896 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: DB pointer 0x55aec0de8000 2024-09-21T10:00:42.896 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T10:00:42.896 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T10:00:42.897 INFO:journalctl@ceph.mon.a.smithi070.stdout: ** DB Stats ** 2024-09-21T10:00:42.897 INFO:journalctl@ceph.mon.a.smithi070.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T10:00:42.897 INFO:journalctl@ceph.mon.a.smithi070.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T10:00:42.897 INFO:journalctl@ceph.mon.a.smithi070.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T10:00:42.897 INFO:journalctl@ceph.mon.a.smithi070.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T10:00:42.897 INFO:journalctl@ceph.mon.a.smithi070.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T10:00:42.897 INFO:journalctl@ceph.mon.a.smithi070.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T10:00:42.897 INFO:journalctl@ceph.mon.a.smithi070.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T10:00:42.897 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-09-21T10:00:42.897 INFO:journalctl@ceph.mon.a.smithi070.stdout: ** Compaction Stats [default] ** 2024-09-21T10:00:42.897 INFO:journalctl@ceph.mon.a.smithi070.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T10:00:42.897 INFO:journalctl@ceph.mon.a.smithi070.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T10:00:42.897 INFO:journalctl@ceph.mon.a.smithi070.stdout: L0 1/0 5.39 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 667.9 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-09-21T10:00:42.897 INFO:journalctl@ceph.mon.a.smithi070.stdout: L6 1/0 8.98 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-21T10:00:42.898 INFO:journalctl@ceph.mon.a.smithi070.stdout: Sum 2/0 14.37 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 667.9 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-09-21T10:00:42.898 INFO:journalctl@ceph.mon.a.smithi070.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 667.9 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-09-21T10:00:42.898 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-09-21T10:00:42.898 INFO:journalctl@ceph.mon.a.smithi070.stdout: ** Compaction Stats [default] ** 2024-09-21T10:00:42.898 INFO:journalctl@ceph.mon.a.smithi070.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T10:00:42.898 INFO:journalctl@ceph.mon.a.smithi070.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T10:00:42.898 INFO:journalctl@ceph.mon.a.smithi070.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 667.9 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-09-21T10:00:42.898 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-09-21T10:00:42.898 INFO:journalctl@ceph.mon.a.smithi070.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T10:00:42.898 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-09-21T10:00:42.898 INFO:journalctl@ceph.mon.a.smithi070.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T10:00:42.898 INFO:journalctl@ceph.mon.a.smithi070.stdout: Flush(GB): cumulative 0.005, interval 0.005 2024-09-21T10:00:42.898 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T10:00:42.898 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T10:00:42.899 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T10:00:42.899 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T10:00:42.899 INFO:journalctl@ceph.mon.a.smithi070.stdout: Cumulative compaction: 0.01 GB write, 273.98 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T10:00:42.899 INFO:journalctl@ceph.mon.a.smithi070.stdout: Interval compaction: 0.01 GB write, 273.98 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T10:00:42.899 INFO:journalctl@ceph.mon.a.smithi070.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T10:00:42.899 INFO:journalctl@ceph.mon.a.smithi070.stdout: Block cache BinnedLRUCache@0x55aec0cb7610#7 capacity: 512.00 MB usage: 21.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-21T10:00:42.899 INFO:journalctl@ceph.mon.a.smithi070.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,5.03 KB,0.000959635%) IndexBlock(1,16.08 KB,0.00306666%) Misc(1,0.00 KB,0%) 2024-09-21T10:00:42.899 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-09-21T10:00:42.899 INFO:journalctl@ceph.mon.a.smithi070.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T10:00:42.899 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: starting mon.a rank 0 at public addrs [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] at bind addrs [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T10:00:42.899 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: mon.a@-1(???) e3 preinit fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T10:00:42.899 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: mon.a@-1(???).mds e9 new map 2024-09-21T10:00:42.899 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: mon.a@-1(???).mds e9 print_map 2024-09-21T10:00:42.899 INFO:journalctl@ceph.mon.a.smithi070.stdout: e9 2024-09-21T10:00:42.900 INFO:journalctl@ceph.mon.a.smithi070.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-21T10:00:42.900 INFO:journalctl@ceph.mon.a.smithi070.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T10:00:42.900 INFO:journalctl@ceph.mon.a.smithi070.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T10:00:42.900 INFO:journalctl@ceph.mon.a.smithi070.stdout: legacy client fscid: 1 2024-09-21T10:00:42.900 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-09-21T10:00:42.900 INFO:journalctl@ceph.mon.a.smithi070.stdout: Filesystem 'foo' (1) 2024-09-21T10:00:42.900 INFO:journalctl@ceph.mon.a.smithi070.stdout: fs_name foo 2024-09-21T10:00:42.900 INFO:journalctl@ceph.mon.a.smithi070.stdout: epoch 8 2024-09-21T10:00:42.900 INFO:journalctl@ceph.mon.a.smithi070.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-21T10:00:42.900 INFO:journalctl@ceph.mon.a.smithi070.stdout: created 2024-09-21T09:54:53.620516+0000 2024-09-21T10:00:42.900 INFO:journalctl@ceph.mon.a.smithi070.stdout: modified 2024-09-21T09:55:04.281129+0000 2024-09-21T10:00:42.900 INFO:journalctl@ceph.mon.a.smithi070.stdout: tableserver 0 2024-09-21T10:00:42.900 INFO:journalctl@ceph.mon.a.smithi070.stdout: root 0 2024-09-21T10:00:42.900 INFO:journalctl@ceph.mon.a.smithi070.stdout: session_timeout 60 2024-09-21T10:00:42.900 INFO:journalctl@ceph.mon.a.smithi070.stdout: session_autoclose 300 2024-09-21T10:00:42.900 INFO:journalctl@ceph.mon.a.smithi070.stdout: max_file_size 1099511627776 2024-09-21T10:00:42.901 INFO:journalctl@ceph.mon.a.smithi070.stdout: max_xattr_size 65536 2024-09-21T10:00:42.901 INFO:journalctl@ceph.mon.a.smithi070.stdout: required_client_features {} 2024-09-21T10:00:42.901 INFO:journalctl@ceph.mon.a.smithi070.stdout: last_failure 0 2024-09-21T10:00:42.901 INFO:journalctl@ceph.mon.a.smithi070.stdout: last_failure_osd_epoch 0 2024-09-21T10:00:42.901 INFO:journalctl@ceph.mon.a.smithi070.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T10:00:42.901 INFO:journalctl@ceph.mon.a.smithi070.stdout: max_mds 1 2024-09-21T10:00:42.901 INFO:journalctl@ceph.mon.a.smithi070.stdout: in 0 2024-09-21T10:00:42.901 INFO:journalctl@ceph.mon.a.smithi070.stdout: up {0=14652} 2024-09-21T10:00:42.901 INFO:journalctl@ceph.mon.a.smithi070.stdout: failed 2024-09-21T10:00:42.901 INFO:journalctl@ceph.mon.a.smithi070.stdout: damaged 2024-09-21T10:00:42.901 INFO:journalctl@ceph.mon.a.smithi070.stdout: stopped 2024-09-21T10:00:42.901 INFO:journalctl@ceph.mon.a.smithi070.stdout: data_pools [4] 2024-09-21T10:00:42.901 INFO:journalctl@ceph.mon.a.smithi070.stdout: metadata_pool 3 2024-09-21T10:00:42.901 INFO:journalctl@ceph.mon.a.smithi070.stdout: inline_data disabled 2024-09-21T10:00:42.901 INFO:journalctl@ceph.mon.a.smithi070.stdout: balancer 2024-09-21T10:00:42.902 INFO:journalctl@ceph.mon.a.smithi070.stdout: bal_rank_mask -1 2024-09-21T10:00:42.902 INFO:journalctl@ceph.mon.a.smithi070.stdout: standby_count_wanted 1 2024-09-21T10:00:42.902 INFO:journalctl@ceph.mon.a.smithi070.stdout: qdb_cluster leader: 0 members: 2024-09-21T10:00:42.902 INFO:journalctl@ceph.mon.a.smithi070.stdout: [mds.foo.smithi070.gansda{0:14652} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.70:6834/1361282039,v1:172.21.15.70:6835/1361282039] compat {c=[1],r=[1],i=[7ff]}] 2024-09-21T10:00:42.902 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-09-21T10:00:42.902 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-09-21T10:00:42.902 INFO:journalctl@ceph.mon.a.smithi070.stdout: Standby daemons: 2024-09-21T10:00:42.902 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-09-21T10:00:42.902 INFO:journalctl@ceph.mon.a.smithi070.stdout: [mds.foo.smithi160.byyhfz{-1:24457} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.160:6832/2992374343,v1:172.21.15.160:6833/2992374343] compat {c=[1],r=[1],i=[7ff]}] 2024-09-21T10:00:42.902 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: mon.a@-1(???).osd e341 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-21T10:00:42.902 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: mon.a@-1(???).osd e341 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T10:00:42.902 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: mon.a@-1(???).osd e341 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T10:00:42.902 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: mon.a@-1(???).osd e341 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T10:00:42.902 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 ceph-mon[107385]: mon.a@-1(???).paxosservice(auth 1..19) refresh upgraded, format 0 -> 3 2024-09-21T10:00:42.902 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 bash[107347]: 4a7b667a403311f40b95be0411d7385e1161f266e1871482d304082744d55eed 2024-09-21T10:00:42.903 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:42 smithi070 systemd[1]: Started Ceph mon.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:00:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:00:43] "GET /metrics HTTP/1.1" 200 38730 "" "Prometheus/2.51.0" 2024-09-21T10:00:44.341 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[79927]: ignoring --setuser ceph since I am not root 2024-09-21T10:00:44.341 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[79927]: ignoring --setgroup ceph since I am not root 2024-09-21T10:00:44.341 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mgr[79950]: -- 172.21.15.160:0/3463916282 <== mon.1 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5599f76a10e0 con 0x5599f767f000 2024-09-21T10:00:44.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T10:00:44.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T10:00:44.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T10:00:44.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:00:44.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mon[33203]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:00:44.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mon[33203]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:00:44.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mon[33203]: pgmap v225: 137 pgs: 137 active+clean; 1.9 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 14 MiB/s rd, 50 MiB/s wr, 434 op/s 2024-09-21T10:00:44.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mon[33203]: mon.a calling monitor election 2024-09-21T10:00:44.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mon[33203]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T10:00:44.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mon[33203]: monmap epoch 3 2024-09-21T10:00:44.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mon[33203]: fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T10:00:44.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mon[33203]: last_changed 2024-09-21T09:49:37.426189+0000 2024-09-21T10:00:44.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mon[33203]: created 2024-09-21T09:47:56.277550+0000 2024-09-21T10:00:44.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mon[33203]: min_mon_release 18 (reef) 2024-09-21T10:00:44.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mon[33203]: election_strategy: 1 2024-09-21T10:00:44.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mon[33203]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-21T10:00:44.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mon[33203]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.b 2024-09-21T10:00:44.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mon[33203]: 2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-21T10:00:44.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mon[33203]: fsmap foo:1 {0=foo.smithi070.gansda=up:active} 1 up:standby 2024-09-21T10:00:44.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mon[33203]: osdmap e341: 8 total, 8 up, 8 in 2024-09-21T10:00:44.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mon[33203]: mgrmap e33: y(active, since 2m), standbys: x 2024-09-21T10:00:44.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mon[33203]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-21T10:00:44.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mon[33203]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-21T10:00:44.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mon[33203]: application not enabled on pool 'test-librbd-smithi070-86016-9' 2024-09-21T10:00:44.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mon[33203]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-21T10:00:44.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mon[33203]: from='mgr.24956 ' entity='' 2024-09-21T10:00:44.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:44 smithi160 ceph-mon[33203]: mgrmap e34: y(active, since 2m), standbys: x 2024-09-21T10:00:44.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ignoring --setuser ceph since I am not root 2024-09-21T10:00:44.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ignoring --setgroup ceph since I am not root 2024-09-21T10:00:44.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mgr[89799]: -- 172.21.15.70:0/945712044 <== mon.2 v2:172.21.15.70:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5590a616f0e0 con 0x5590a614c800 2024-09-21T10:00:44.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[107385]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T10:00:44.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[107385]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T10:00:44.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[107385]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T10:00:44.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[107385]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:00:44.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[107385]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:00:44.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[107385]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:00:44.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[107385]: pgmap v225: 137 pgs: 137 active+clean; 1.9 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 14 MiB/s rd, 50 MiB/s wr, 434 op/s 2024-09-21T10:00:44.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[107385]: mon.a calling monitor election 2024-09-21T10:00:44.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[107385]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T10:00:44.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[107385]: monmap epoch 3 2024-09-21T10:00:44.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[107385]: fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T10:00:44.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[107385]: last_changed 2024-09-21T09:49:37.426189+0000 2024-09-21T10:00:44.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[107385]: created 2024-09-21T09:47:56.277550+0000 2024-09-21T10:00:44.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[107385]: min_mon_release 18 (reef) 2024-09-21T10:00:44.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[107385]: election_strategy: 1 2024-09-21T10:00:44.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[107385]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-21T10:00:44.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[107385]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.b 2024-09-21T10:00:44.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[107385]: 2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-21T10:00:44.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[107385]: fsmap foo:1 {0=foo.smithi070.gansda=up:active} 1 up:standby 2024-09-21T10:00:44.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[107385]: osdmap e341: 8 total, 8 up, 8 in 2024-09-21T10:00:44.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[107385]: mgrmap e33: y(active, since 2m), standbys: x 2024-09-21T10:00:44.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[107385]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-21T10:00:44.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[107385]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-21T10:00:44.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[107385]: application not enabled on pool 'test-librbd-smithi070-86016-9' 2024-09-21T10:00:44.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[107385]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-21T10:00:44.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[107385]: from='mgr.24956 ' entity='' 2024-09-21T10:00:44.380 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[107385]: mgrmap e34: y(active, since 2m), standbys: x 2024-09-21T10:00:44.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T10:00:44.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T10:00:44.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T10:00:44.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:00:44.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[41533]: from='mgr.24956 172.21.15.70:0/3742189505' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:00:44.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[41533]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:00:44.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[41533]: pgmap v225: 137 pgs: 137 active+clean; 1.9 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 14 MiB/s rd, 50 MiB/s wr, 434 op/s 2024-09-21T10:00:44.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[41533]: mon.a calling monitor election 2024-09-21T10:00:44.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[41533]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T10:00:44.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[41533]: monmap epoch 3 2024-09-21T10:00:44.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[41533]: fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T10:00:44.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[41533]: last_changed 2024-09-21T09:49:37.426189+0000 2024-09-21T10:00:44.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[41533]: created 2024-09-21T09:47:56.277550+0000 2024-09-21T10:00:44.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[41533]: min_mon_release 18 (reef) 2024-09-21T10:00:44.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[41533]: election_strategy: 1 2024-09-21T10:00:44.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[41533]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-21T10:00:44.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[41533]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.b 2024-09-21T10:00:44.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[41533]: 2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-21T10:00:44.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[41533]: fsmap foo:1 {0=foo.smithi070.gansda=up:active} 1 up:standby 2024-09-21T10:00:44.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[41533]: osdmap e341: 8 total, 8 up, 8 in 2024-09-21T10:00:44.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[41533]: mgrmap e33: y(active, since 2m), standbys: x 2024-09-21T10:00:44.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[41533]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-21T10:00:44.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[41533]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-21T10:00:44.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[41533]: application not enabled on pool 'test-librbd-smithi070-86016-9' 2024-09-21T10:00:44.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[41533]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-21T10:00:44.382 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[41533]: from='mgr.24956 ' entity='' 2024-09-21T10:00:44.382 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:44 smithi070 ceph-mon[41533]: mgrmap e34: y(active, since 2m), standbys: x 2024-09-21T10:00:45.108 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 274227239 2024-09-21T10:00:45.142 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 27422723~6855680 2024-09-21T10:00:45.145 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 29136643~3427840 2024-09-21T10:00:45.203 INFO:tasks.workunit.client.0.smithi070.stdout:write: 2 54845447~6855680 2024-09-21T10:00:45.205 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 56559367~3427840 2024-09-21T10:00:45.251 INFO:tasks.workunit.client.0.smithi070.stdout:write: 3 82268171~6855680 2024-09-21T10:00:45.253 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 83982091~3427840 2024-09-21T10:00:45.288 INFO:tasks.workunit.client.0.smithi070.stdout:write: 4 109690895~6855680 2024-09-21T10:00:45.291 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 111404815~3427840 2024-09-21T10:00:45.351 INFO:tasks.workunit.client.0.smithi070.stdout:write: 5 137113619~6855680 2024-09-21T10:00:45.353 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 138827539~3427840 2024-09-21T10:00:45.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:45 smithi070 ceph-mon[41533]: osdmap e342: 8 total, 8 up, 8 in 2024-09-21T10:00:45.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:45 smithi070 ceph-mon[107385]: osdmap e342: 8 total, 8 up, 8 in 2024-09-21T10:00:45.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:45 smithi160 ceph-mon[33203]: osdmap e342: 8 total, 8 up, 8 in 2024-09-21T10:00:46.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:46 smithi070 ceph-mon[107385]: osdmap e343: 8 total, 8 up, 8 in 2024-09-21T10:00:46.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:46 smithi070 ceph-mon[41533]: osdmap e343: 8 total, 8 up, 8 in 2024-09-21T10:00:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:46 smithi160 ceph-mon[33203]: osdmap e343: 8 total, 8 up, 8 in 2024-09-21T10:00:47.488 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T10:00:47.759 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T10:00:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:49 smithi160 ceph-mon[33203]: Standby manager daemon x restarted 2024-09-21T10:00:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:49 smithi160 ceph-mon[33203]: Standby manager daemon x started 2024-09-21T10:00:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:49 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.160:0/2255750356' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T10:00:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:49 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.160:0/2255750356' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T10:00:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:49 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.160:0/2255750356' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T10:00:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:49 smithi160 ceph-mon[33203]: from='mgr.? 172.21.15.160:0/2255750356' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T10:00:49.842 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 10:00:49 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[79927]: [21/Sep/2024:10:00:49] ENGINE Bus STARTING 2024-09-21T10:00:49.842 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 10:00:49 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[79927]: CherryPy Checker: 2024-09-21T10:00:49.842 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 10:00:49 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[79927]: The Application mounted at '' has an empty config. 2024-09-21T10:00:49.842 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 10:00:49 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[79927]: 2024-09-21T10:00:49.842 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 10:00:49 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[79927]: [21/Sep/2024:10:00:49] ENGINE Serving on http://:::9283 2024-09-21T10:00:49.842 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 10:00:49 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x[79927]: [21/Sep/2024:10:00:49] ENGINE Bus STARTED 2024-09-21T10:00:49.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:49 smithi070 ceph-mon[41533]: Standby manager daemon x restarted 2024-09-21T10:00:49.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:49 smithi070 ceph-mon[41533]: Standby manager daemon x started 2024-09-21T10:00:49.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:49 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.160:0/2255750356' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T10:00:49.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:49 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.160:0/2255750356' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T10:00:49.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:49 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.160:0/2255750356' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T10:00:49.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:49 smithi070 ceph-mon[41533]: from='mgr.? 172.21.15.160:0/2255750356' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T10:00:49.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:49 smithi070 ceph-mon[107385]: Standby manager daemon x restarted 2024-09-21T10:00:49.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:49 smithi070 ceph-mon[107385]: Standby manager daemon x started 2024-09-21T10:00:49.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:49 smithi070 ceph-mon[107385]: from='mgr.? 172.21.15.160:0/2255750356' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T10:00:49.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:49 smithi070 ceph-mon[107385]: from='mgr.? 172.21.15.160:0/2255750356' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T10:00:49.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:49 smithi070 ceph-mon[107385]: from='mgr.? 172.21.15.160:0/2255750356' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T10:00:49.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:49 smithi070 ceph-mon[107385]: from='mgr.? 172.21.15.160:0/2255750356' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T10:00:49.944 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.Stress2 (24391 ms) 2024-09-21T10:00:49.944 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMigration.StressLive 2024-09-21T10:00:49.977 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 83886080 2024-09-21T10:00:49.987 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 3781491~7806062 2024-09-21T10:00:50.033 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 30929409~683119 2024-09-21T10:00:50.033 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 16131436~6722361 2024-09-21T10:00:50.075 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 64987895~6563485 2024-09-21T10:00:50.075 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 75057113~3996152 2024-09-21T10:00:50.101 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 22142549~360772 2024-09-21T10:00:50.104 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 71459068~8057100 2024-09-21T10:00:50.151 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 34711444~443821 2024-09-21T10:00:50.151 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap0 2024-09-21T10:00:50.628 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T10:00:50.301+0000 7fd0599b1640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:00:50.628 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T10:00:50.301+0000 7fd0599b1640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:00:50.628 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T10:00:50.301+0000 7fd0599b1640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:00:50.628 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T10:00:50.301+0000 7fd0599b1640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:00:50.628 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T10:00:50.301+0000 7fd0599b1640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:00:50.628 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T10:00:50.303+0000 7fd0569ab640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:00:50.628 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T10:00:50.303+0000 7fd0569ab640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:00:50.628 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T10:00:50.303+0000 7fd0569ab640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:00:50.628 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T10:00:50.303+0000 7fd0569ab640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:00:50.628 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T10:00:50.303+0000 7fd0569ab640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:00:50.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: mgrmap e35: y(active, since 2m), standbys: x 2024-09-21T10:00:50.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: Active manager daemon y restarted 2024-09-21T10:00:50.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: Activating manager daemon y 2024-09-21T10:00:50.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: osdmap e344: 8 total, 8 up, 8 in 2024-09-21T10:00:50.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: mgrmap e36: y(active, starting, since 0.00888835s), standbys: x 2024-09-21T10:00:50.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T10:00:50.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T10:00:50.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T10:00:50.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi070.gansda"}]: dispatch 2024-09-21T10:00:50.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi160.byyhfz"}]: dispatch 2024-09-21T10:00:50.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T10:00:50.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T10:00:50.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T10:00:50.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T10:00:50.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T10:00:50.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T10:00:50.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T10:00:50.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T10:00:50.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T10:00:50.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T10:00:50.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T10:00:50.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T10:00:50.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T10:00:50.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: Manager daemon y is now available 2024-09-21T10:00:50.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:50.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:00:50.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:00:50.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T10:00:50.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T10:00:50.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T10:00:50.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[41533]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T10:00:50.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: mgrmap e35: y(active, since 2m), standbys: x 2024-09-21T10:00:50.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: Active manager daemon y restarted 2024-09-21T10:00:50.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: Activating manager daemon y 2024-09-21T10:00:50.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: osdmap e344: 8 total, 8 up, 8 in 2024-09-21T10:00:50.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: mgrmap e36: y(active, starting, since 0.00888835s), standbys: x 2024-09-21T10:00:50.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T10:00:50.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T10:00:50.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T10:00:50.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi070.gansda"}]: dispatch 2024-09-21T10:00:50.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi160.byyhfz"}]: dispatch 2024-09-21T10:00:50.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T10:00:50.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T10:00:50.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T10:00:50.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T10:00:50.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T10:00:50.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T10:00:50.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T10:00:50.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T10:00:50.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T10:00:50.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T10:00:50.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T10:00:50.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T10:00:50.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T10:00:50.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: Manager daemon y is now available 2024-09-21T10:00:50.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:50.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:00:50.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:00:50.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T10:00:50.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T10:00:50.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T10:00:50.634 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T10:00:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: mgrmap e35: y(active, since 2m), standbys: x 2024-09-21T10:00:50.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: Active manager daemon y restarted 2024-09-21T10:00:50.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: Activating manager daemon y 2024-09-21T10:00:50.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: osdmap e344: 8 total, 8 up, 8 in 2024-09-21T10:00:50.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: mgrmap e36: y(active, starting, since 0.00888835s), standbys: x 2024-09-21T10:00:50.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T10:00:50.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T10:00:50.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T10:00:50.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi070.gansda"}]: dispatch 2024-09-21T10:00:50.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi160.byyhfz"}]: dispatch 2024-09-21T10:00:50.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T10:00:50.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T10:00:50.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T10:00:50.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T10:00:50.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T10:00:50.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T10:00:50.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T10:00:50.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T10:00:50.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T10:00:50.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T10:00:50.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T10:00:50.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T10:00:50.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T10:00:50.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: Manager daemon y is now available 2024-09-21T10:00:50.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:50.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:00:50.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:00:50.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T10:00:50.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T10:00:50.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T10:00:50.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:50 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T10:00:51.211 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T10:00:50.914+0000 7fd088d1d640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-21T10:00:51.212 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:00:50] ENGINE Bus STARTING 2024-09-21T10:00:51.212 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: CherryPy Checker: 2024-09-21T10:00:51.212 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: The Application mounted at '' has an empty config. 2024-09-21T10:00:51.212 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:50 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T10:00:51.212 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:00:51] ENGINE Serving on http://:::9283 2024-09-21T10:00:51.212 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:00:51] ENGINE Bus STARTED 2024-09-21T10:00:51.212 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T10:00:51.208+0000 7fd03c7fd640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:10:00:51] ENGINE Error in HTTPServer.serve 2024-09-21T10:00:51.212 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: Traceback (most recent call last): 2024-09-21T10:00:51.212 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T10:00:51.212 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: self._connections.run(self.expiration_interval) 2024-09-21T10:00:51.212 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T10:00:51.212 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: self._run(expiration_interval) 2024-09-21T10:00:51.212 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T10:00:51.470 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T10:00:51.470 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T10:00:51.471 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T10:00:51.471 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T10:00:51.471 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: s = self.context.wrap_socket( 2024-09-21T10:00:51.471 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T10:00:51.471 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: return self.sslsocket_class._create( 2024-09-21T10:00:51.471 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T10:00:51.471 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: self.do_handshake() 2024-09-21T10:00:51.471 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T10:00:51.471 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: self._sslobj.do_handshake() 2024-09-21T10:00:51.471 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T10:00:51.471 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T10:00:51.936 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap0 2024-09-21T10:00:51.947 INFO:tasks.workunit.client.0.smithi070.stdout:clone image187 -> image190 2024-09-21T10:00:52.038 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 55964129~3272156 2024-09-21T10:00:52.070 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 5313878~1969200 2024-09-21T10:00:52.104 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 55469292~7491817 2024-09-21T10:00:52.152 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 5390122~6077168 2024-09-21T10:00:52.166 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 37092194~7004989 2024-09-21T10:00:52.195 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-mon[107385]: osdmap e345: 8 total, 8 up, 8 in 2024-09-21T10:00:52.195 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-mon[107385]: mgrmap e37: y(active, since 1.01661s), standbys: x 2024-09-21T10:00:52.196 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-mon[107385]: [21/Sep/2024:10:00:51] ENGINE Bus STARTING 2024-09-21T10:00:52.196 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-mon[41533]: osdmap e345: 8 total, 8 up, 8 in 2024-09-21T10:00:52.196 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-mon[41533]: mgrmap e37: y(active, since 1.01661s), standbys: x 2024-09-21T10:00:52.196 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:51 smithi070 ceph-mon[41533]: [21/Sep/2024:10:00:51] ENGINE Bus STARTING 2024-09-21T10:00:52.206 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 41658928~6885892 2024-09-21T10:00:52.211 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 75972282~7913798 2024-09-21T10:00:52.269 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 8596917~3497379 2024-09-21T10:00:52.271 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap1 2024-09-21T10:00:52.280 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:51 smithi160 ceph-mon[33203]: osdmap e345: 8 total, 8 up, 8 in 2024-09-21T10:00:52.280 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:51 smithi160 ceph-mon[33203]: mgrmap e37: y(active, since 1.01661s), standbys: x 2024-09-21T10:00:52.280 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:51 smithi160 ceph-mon[33203]: [21/Sep/2024:10:00:51] ENGINE Bus STARTING 2024-09-21T10:00:53.227 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:00:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:00:53] "GET /metrics HTTP/1.1" 200 35885 "" "Prometheus/2.51.0" 2024-09-21T10:00:53.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:52 smithi070 ceph-mon[107385]: [21/Sep/2024:10:00:51] ENGINE Serving on https://172.21.15.70:7150 2024-09-21T10:00:53.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:52 smithi070 ceph-mon[107385]: [21/Sep/2024:10:00:51] ENGINE Error in HTTPServer.serve 2024-09-21T10:00:53.228 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-09-21T10:00:53.228 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T10:00:53.228 INFO:journalctl@ceph.mon.a.smithi070.stdout: self._connections.run(self.expiration_interval) 2024-09-21T10:00:53.228 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T10:00:53.228 INFO:journalctl@ceph.mon.a.smithi070.stdout: self._run(expiration_interval) 2024-09-21T10:00:53.228 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T10:00:53.228 INFO:journalctl@ceph.mon.a.smithi070.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T10:00:53.228 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T10:00:53.228 INFO:journalctl@ceph.mon.a.smithi070.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T10:00:53.229 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T10:00:53.229 INFO:journalctl@ceph.mon.a.smithi070.stdout: s = self.context.wrap_socket( 2024-09-21T10:00:53.229 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T10:00:53.229 INFO:journalctl@ceph.mon.a.smithi070.stdout: return self.sslsocket_class._create( 2024-09-21T10:00:53.229 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T10:00:53.229 INFO:journalctl@ceph.mon.a.smithi070.stdout: self.do_handshake() 2024-09-21T10:00:53.229 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T10:00:53.229 INFO:journalctl@ceph.mon.a.smithi070.stdout: self._sslobj.do_handshake() 2024-09-21T10:00:53.229 INFO:journalctl@ceph.mon.a.smithi070.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T10:00:53.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:52 smithi070 ceph-mon[107385]: [21/Sep/2024:10:00:51] ENGINE Serving on http://172.21.15.70:8765 2024-09-21T10:00:53.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:52 smithi070 ceph-mon[107385]: [21/Sep/2024:10:00:51] ENGINE Bus STARTED 2024-09-21T10:00:53.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:52 smithi070 ceph-mon[107385]: pgmap v5: 137 pgs: 137 active+clean; 2.0 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:00:53.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:52 smithi070 ceph-mon[107385]: mgrmap e38: y(active, since 2s), standbys: x 2024-09-21T10:00:53.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:52 smithi070 ceph-mon[107385]: osdmap e346: 8 total, 8 up, 8 in 2024-09-21T10:00:53.230 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:52 smithi070 ceph-mon[41533]: [21/Sep/2024:10:00:51] ENGINE Serving on https://172.21.15.70:7150 2024-09-21T10:00:53.230 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:52 smithi070 ceph-mon[41533]: [21/Sep/2024:10:00:51] ENGINE Error in HTTPServer.serve 2024-09-21T10:00:53.230 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-09-21T10:00:53.230 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T10:00:53.230 INFO:journalctl@ceph.mon.c.smithi070.stdout: self._connections.run(self.expiration_interval) 2024-09-21T10:00:53.230 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T10:00:53.230 INFO:journalctl@ceph.mon.c.smithi070.stdout: self._run(expiration_interval) 2024-09-21T10:00:53.230 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T10:00:53.230 INFO:journalctl@ceph.mon.c.smithi070.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T10:00:53.230 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T10:00:53.230 INFO:journalctl@ceph.mon.c.smithi070.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T10:00:53.230 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T10:00:53.230 INFO:journalctl@ceph.mon.c.smithi070.stdout: s = self.context.wrap_socket( 2024-09-21T10:00:53.231 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T10:00:53.231 INFO:journalctl@ceph.mon.c.smithi070.stdout: return self.sslsocket_class._create( 2024-09-21T10:00:53.231 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T10:00:53.231 INFO:journalctl@ceph.mon.c.smithi070.stdout: self.do_handshake() 2024-09-21T10:00:53.231 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T10:00:53.231 INFO:journalctl@ceph.mon.c.smithi070.stdout: self._sslobj.do_handshake() 2024-09-21T10:00:53.231 INFO:journalctl@ceph.mon.c.smithi070.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T10:00:53.231 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:52 smithi070 ceph-mon[41533]: [21/Sep/2024:10:00:51] ENGINE Serving on http://172.21.15.70:8765 2024-09-21T10:00:53.231 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:52 smithi070 ceph-mon[41533]: [21/Sep/2024:10:00:51] ENGINE Bus STARTED 2024-09-21T10:00:53.231 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:52 smithi070 ceph-mon[41533]: pgmap v5: 137 pgs: 137 active+clean; 2.0 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:00:53.231 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:52 smithi070 ceph-mon[41533]: mgrmap e38: y(active, since 2s), standbys: x 2024-09-21T10:00:53.231 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:52 smithi070 ceph-mon[41533]: osdmap e346: 8 total, 8 up, 8 in 2024-09-21T10:00:53.234 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:52 smithi160 ceph-mon[33203]: [21/Sep/2024:10:00:51] ENGINE Serving on https://172.21.15.70:7150 2024-09-21T10:00:53.234 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:52 smithi160 ceph-mon[33203]: [21/Sep/2024:10:00:51] ENGINE Error in HTTPServer.serve 2024-09-21T10:00:53.234 INFO:journalctl@ceph.mon.b.smithi160.stdout: Traceback (most recent call last): 2024-09-21T10:00:53.234 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T10:00:53.235 INFO:journalctl@ceph.mon.b.smithi160.stdout: self._connections.run(self.expiration_interval) 2024-09-21T10:00:53.235 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T10:00:53.235 INFO:journalctl@ceph.mon.b.smithi160.stdout: self._run(expiration_interval) 2024-09-21T10:00:53.235 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T10:00:53.235 INFO:journalctl@ceph.mon.b.smithi160.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T10:00:53.235 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T10:00:53.235 INFO:journalctl@ceph.mon.b.smithi160.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T10:00:53.235 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T10:00:53.235 INFO:journalctl@ceph.mon.b.smithi160.stdout: s = self.context.wrap_socket( 2024-09-21T10:00:53.235 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T10:00:53.235 INFO:journalctl@ceph.mon.b.smithi160.stdout: return self.sslsocket_class._create( 2024-09-21T10:00:53.235 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T10:00:53.235 INFO:journalctl@ceph.mon.b.smithi160.stdout: self.do_handshake() 2024-09-21T10:00:53.235 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T10:00:53.236 INFO:journalctl@ceph.mon.b.smithi160.stdout: self._sslobj.do_handshake() 2024-09-21T10:00:53.236 INFO:journalctl@ceph.mon.b.smithi160.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T10:00:53.236 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:52 smithi160 ceph-mon[33203]: [21/Sep/2024:10:00:51] ENGINE Serving on http://172.21.15.70:8765 2024-09-21T10:00:53.236 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:52 smithi160 ceph-mon[33203]: [21/Sep/2024:10:00:51] ENGINE Bus STARTED 2024-09-21T10:00:53.236 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:52 smithi160 ceph-mon[33203]: pgmap v5: 137 pgs: 137 active+clean; 2.0 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:00:53.236 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:52 smithi160 ceph-mon[33203]: mgrmap e38: y(active, since 2s), standbys: x 2024-09-21T10:00:53.236 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:52 smithi160 ceph-mon[33203]: osdmap e346: 8 total, 8 up, 8 in 2024-09-21T10:00:53.943 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap1 2024-09-21T10:00:53.952 INFO:tasks.workunit.client.0.smithi070.stdout:clone image190 -> image192 2024-09-21T10:00:54.065 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 142168866 2024-09-21T10:00:54.076 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 87970944~1853201 2024-09-21T10:00:54.099 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 15051090~3558682 2024-09-21T10:00:54.132 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 69394152~70936 2024-09-21T10:00:54.139 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 83855843~1081334 2024-09-21T10:00:54.194 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 121428738~979229 2024-09-21T10:00:54.197 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:53 smithi160 ceph-mon[33203]: osdmap e347: 8 total, 8 up, 8 in 2024-09-21T10:00:54.205 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 113440714~3127543 2024-09-21T10:00:54.205 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 52890414~5510502 2024-09-21T10:00:54.254 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 92008962~1108370 2024-09-21T10:00:54.256 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap2 2024-09-21T10:00:54.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:53 smithi070 ceph-mon[107385]: osdmap e347: 8 total, 8 up, 8 in 2024-09-21T10:00:54.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:53 smithi070 ceph-mon[41533]: osdmap e347: 8 total, 8 up, 8 in 2024-09-21T10:00:55.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:54 smithi160 ceph-mon[33203]: pgmap v8: 137 pgs: 137 active+clean; 2.0 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:00:55.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:54 smithi160 ceph-mon[33203]: mgrmap e39: y(active, since 4s), standbys: x 2024-09-21T10:00:55.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:54 smithi160 ceph-mon[33203]: osdmap e348: 8 total, 8 up, 8 in 2024-09-21T10:00:55.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:54 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:55.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:54 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:55.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:54 smithi070 ceph-mon[107385]: pgmap v8: 137 pgs: 137 active+clean; 2.0 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:00:55.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:54 smithi070 ceph-mon[107385]: mgrmap e39: y(active, since 4s), standbys: x 2024-09-21T10:00:55.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:54 smithi070 ceph-mon[107385]: osdmap e348: 8 total, 8 up, 8 in 2024-09-21T10:00:55.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:54 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:55.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:54 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:55.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:54 smithi070 ceph-mon[41533]: pgmap v8: 137 pgs: 137 active+clean; 2.0 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:00:55.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:54 smithi070 ceph-mon[41533]: mgrmap e39: y(active, since 4s), standbys: x 2024-09-21T10:00:55.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:54 smithi070 ceph-mon[41533]: osdmap e348: 8 total, 8 up, 8 in 2024-09-21T10:00:55.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:54 smithi070 ceph-mon[41533]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:55.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:54 smithi070 ceph-mon[41533]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:55.961 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap2 2024-09-21T10:00:55.970 INFO:tasks.workunit.client.0.smithi070.stdout:clone image192 -> image194 2024-09-21T10:00:56.113 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 101514136~124333 2024-09-21T10:00:56.125 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 57930973~4862395 2024-09-21T10:00:56.167 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 76061847~3362032 2024-09-21T10:00:56.207 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:55 smithi160 ceph-mon[33203]: osdmap e349: 8 total, 8 up, 8 in 2024-09-21T10:00:56.208 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:55 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:56.208 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:55 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:56.208 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:55 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:56.208 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:55 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:56.208 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:55 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:56.208 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:55 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:56.209 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 77211712~3187285 2024-09-21T10:00:56.248 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 104058891~8075663 2024-09-21T10:00:56.289 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 130409598~4955094 2024-09-21T10:00:56.295 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 20446543~2958125 2024-09-21T10:00:56.331 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 111554803~1228352 2024-09-21T10:00:56.333 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: snap3 2024-09-21T10:00:56.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:55 smithi070 ceph-mon[107385]: osdmap e349: 8 total, 8 up, 8 in 2024-09-21T10:00:56.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:55 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:56.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:55 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:56.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:55 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:56.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:55 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:56.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:55 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:56.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:55 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:56.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:55 smithi070 ceph-mon[41533]: osdmap e349: 8 total, 8 up, 8 in 2024-09-21T10:00:56.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:55 smithi070 ceph-mon[41533]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:56.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:55 smithi070 ceph-mon[41533]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:56.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:55 smithi070 ceph-mon[41533]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:56.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:55 smithi070 ceph-mon[41533]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:56.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:55 smithi070 ceph-mon[41533]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:56.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:55 smithi070 ceph-mon[41533]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:57.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:56 smithi070 ceph-mon[41533]: pgmap v11: 137 pgs: 137 active+clean; 2.0 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:00:57.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:56 smithi070 ceph-mon[41533]: osdmap e350: 8 total, 8 up, 8 in 2024-09-21T10:00:57.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:56 smithi070 ceph-mon[107385]: pgmap v11: 137 pgs: 137 active+clean; 2.0 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:00:57.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:56 smithi070 ceph-mon[107385]: osdmap e350: 8 total, 8 up, 8 in 2024-09-21T10:00:57.205 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:56 smithi160 ceph-mon[33203]: pgmap v11: 137 pgs: 137 active+clean; 2.0 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:00:57.205 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:56 smithi160 ceph-mon[33203]: osdmap e350: 8 total, 8 up, 8 in 2024-09-21T10:00:57.966 INFO:tasks.workunit.client.0.smithi070.stdout:snap_protect: snap3 2024-09-21T10:00:57.976 INFO:tasks.workunit.client.0.smithi070.stdout:clone image194 -> image196 2024-09-21T10:00:58.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:57 smithi070 ceph-mon[41533]: osdmap e351: 8 total, 8 up, 8 in 2024-09-21T10:00:58.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:57 smithi070 ceph-mon[107385]: osdmap e351: 8 total, 8 up, 8 in 2024-09-21T10:00:58.151 INFO:tasks.workunit.client.0.smithi070.stdout:migration_prepare 2024-09-21T10:00:58.247 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:57 smithi160 ceph-mon[33203]: osdmap e351: 8 total, 8 up, 8 in 2024-09-21T10:00:58.625 INFO:tasks.workunit.client.0.smithi070.stdout:migration_execute 2024-09-21T10:00:58.625 INFO:tasks.workunit.client.0.smithi070.stdout:write: a 102889569~7052215 2024-09-21T10:00:58.686 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 26996011~5720429 2024-09-21T10:00:58.716 INFO:tasks.workunit.client.0.smithi070.stdout:write: a 51228483~3152209 2024-09-21T10:00:58.755 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 70007787~7170615 2024-09-21T10:00:58.797 INFO:tasks.workunit.client.0.smithi070.stdout:write: a 8716455~830781 2024-09-21T10:00:58.816 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 104905933~2709876 2024-09-21T10:00:58.829 INFO:tasks.workunit.client.0.smithi070.stdout:write: a 28437503~6136392 2024-09-21T10:00:58.893 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 65705135~5170155 2024-09-21T10:00:58.905 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: user0 2024-09-21T10:00:59.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:58 smithi160 ceph-mon[33203]: pgmap v14: 137 pgs: 137 active+clean; 2.1 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 12 MiB/s rd, 21 MiB/s wr, 414 op/s 2024-09-21T10:00:59.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:58 smithi160 ceph-mon[33203]: osdmap e352: 8 total, 8 up, 8 in 2024-09-21T10:00:59.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:58 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:59.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:58 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:59.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:58 smithi070 ceph-mon[107385]: pgmap v14: 137 pgs: 137 active+clean; 2.1 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 12 MiB/s rd, 21 MiB/s wr, 414 op/s 2024-09-21T10:00:59.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:58 smithi070 ceph-mon[107385]: osdmap e352: 8 total, 8 up, 8 in 2024-09-21T10:00:59.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:58 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:59.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:58 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:59.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:58 smithi070 ceph-mon[41533]: pgmap v14: 137 pgs: 137 active+clean; 2.1 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 12 MiB/s rd, 21 MiB/s wr, 414 op/s 2024-09-21T10:00:59.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:58 smithi070 ceph-mon[41533]: osdmap e352: 8 total, 8 up, 8 in 2024-09-21T10:00:59.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:58 smithi070 ceph-mon[41533]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:00:59.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:58 smithi070 ceph-mon[41533]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:00.220 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:00.220 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:00.220 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:01:00.220 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:01:00.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[107385]: Updating smithi070:/etc/ceph/ceph.conf 2024-09-21T10:01:00.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[107385]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-21T10:01:00.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[107385]: osdmap e353: 8 total, 8 up, 8 in 2024-09-21T10:01:00.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:00.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:00.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:00.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:00.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:00.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:01:00.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:00.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-21T10:01:00.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-21T10:01:00.222 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[41533]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:00.222 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[41533]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:00.222 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:01:00.222 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:01:00.222 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[41533]: Updating smithi070:/etc/ceph/ceph.conf 2024-09-21T10:01:00.222 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[41533]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-21T10:01:00.222 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[41533]: osdmap e353: 8 total, 8 up, 8 in 2024-09-21T10:01:00.222 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[41533]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:00.222 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[41533]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:00.222 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[41533]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:00.222 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[41533]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:00.222 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[41533]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:00.222 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:01:00.222 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:00.223 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-21T10:01:00.223 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:00:59 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-21T10:01:00.313 INFO:tasks.workunit.client.0.smithi070.stdout:write: b 126816325~143579 2024-09-21T10:01:00.318 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 99819618~1294981 2024-09-21T10:01:00.320 INFO:tasks.workunit.client.0.smithi070.stdout:write: b 51883256~6875518 2024-09-21T10:01:00.323 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 51211686~6375444 2024-09-21T10:01:00.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:59 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:00.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:59 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:00.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:59 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:01:00.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:59 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:01:00.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:59 smithi160 ceph-mon[33203]: Updating smithi070:/etc/ceph/ceph.conf 2024-09-21T10:01:00.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:59 smithi160 ceph-mon[33203]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-21T10:01:00.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:59 smithi160 ceph-mon[33203]: osdmap e353: 8 total, 8 up, 8 in 2024-09-21T10:01:00.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:59 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:00.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:59 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:00.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:59 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:00.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:59 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:00.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:59 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:00.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:59 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:01:00.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:59 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:00.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:59 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-21T10:01:00.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:00:59 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-21T10:01:00.368 INFO:tasks.workunit.client.0.smithi070.stdout:write: b 51550341~7648701 2024-09-21T10:01:00.373 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 99801469~1233450 2024-09-21T10:01:00.373 INFO:tasks.workunit.client.0.smithi070.stdout:write: b 25434534~4627660 2024-09-21T10:01:00.378 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 17867361~4440508 2024-09-21T10:01:00.378 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: user1 2024-09-21T10:01:01.239 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:00 smithi160 ceph-mon[33203]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T10:01:01.239 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:00 smithi160 ceph-mon[33203]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T10:01:01.239 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:00 smithi160 ceph-mon[33203]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-09-21T10:01:01.239 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:00 smithi160 ceph-mon[33203]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-21T10:01:01.239 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:00 smithi160 ceph-mon[33203]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T10:01:01.239 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:00 smithi160 ceph-mon[33203]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T10:01:01.239 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:00 smithi160 ceph-mon[33203]: Upgrade: It appears safe to stop mon.c 2024-09-21T10:01:01.239 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:00 smithi160 ceph-mon[33203]: pgmap v17: 137 pgs: 137 active+clean; 2.1 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 12 MiB/s rd, 21 MiB/s wr, 414 op/s 2024-09-21T10:01:01.240 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:00 smithi160 ceph-mon[33203]: osdmap e354: 8 total, 8 up, 8 in 2024-09-21T10:01:01.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:00 smithi070 ceph-mon[41533]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T10:01:01.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:00 smithi070 ceph-mon[41533]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T10:01:01.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:00 smithi070 ceph-mon[41533]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-09-21T10:01:01.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:00 smithi070 ceph-mon[41533]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-21T10:01:01.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:00 smithi070 ceph-mon[41533]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T10:01:01.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:00 smithi070 ceph-mon[41533]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T10:01:01.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:00 smithi070 ceph-mon[41533]: Upgrade: It appears safe to stop mon.c 2024-09-21T10:01:01.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:00 smithi070 ceph-mon[41533]: pgmap v17: 137 pgs: 137 active+clean; 2.1 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 12 MiB/s rd, 21 MiB/s wr, 414 op/s 2024-09-21T10:01:01.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:00 smithi070 ceph-mon[41533]: osdmap e354: 8 total, 8 up, 8 in 2024-09-21T10:01:01.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:00 smithi070 ceph-mon[107385]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T10:01:01.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:00 smithi070 ceph-mon[107385]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T10:01:01.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:00 smithi070 ceph-mon[107385]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-09-21T10:01:01.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:00 smithi070 ceph-mon[107385]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-21T10:01:01.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:00 smithi070 ceph-mon[107385]: Updating smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T10:01:01.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:00 smithi070 ceph-mon[107385]: Updating smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T10:01:01.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:00 smithi070 ceph-mon[107385]: Upgrade: It appears safe to stop mon.c 2024-09-21T10:01:01.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:00 smithi070 ceph-mon[107385]: pgmap v17: 137 pgs: 137 active+clean; 2.1 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 12 MiB/s rd, 21 MiB/s wr, 414 op/s 2024-09-21T10:01:01.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:00 smithi070 ceph-mon[107385]: osdmap e354: 8 total, 8 up, 8 in 2024-09-21T10:01:02.322 INFO:tasks.workunit.client.0.smithi070.stdout:write: c 85066144~4083345 2024-09-21T10:01:02.335 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 133944112~323925 2024-09-21T10:01:02.335 INFO:tasks.workunit.client.0.smithi070.stdout:write: c 5529406~579227 2024-09-21T10:01:02.336 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 17802291~2755730 2024-09-21T10:01:02.336 INFO:tasks.workunit.client.0.smithi070.stdout:write: c 72424548~1460114 2024-09-21T10:01:02.336 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 2616361~6123312 2024-09-21T10:01:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:02 smithi160 ceph-mon[33203]: Upgrade: Updating mon.c 2024-09-21T10:01:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:02 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:02 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T10:01:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:02 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T10:01:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:02 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:01:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:02 smithi160 ceph-mon[33203]: Deploying daemon mon.c on smithi070 2024-09-21T10:01:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:02 smithi160 ceph-mon[33203]: osdmap e355: 8 total, 8 up, 8 in 2024-09-21T10:01:02.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:02 smithi070 ceph-mon[107385]: Upgrade: Updating mon.c 2024-09-21T10:01:02.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:02 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:02.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:02 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T10:01:02.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:02 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T10:01:02.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:02 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:01:02.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:02 smithi070 ceph-mon[107385]: Deploying daemon mon.c on smithi070 2024-09-21T10:01:02.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:02 smithi070 ceph-mon[107385]: osdmap e355: 8 total, 8 up, 8 in 2024-09-21T10:01:02.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:02 smithi070 ceph-mon[41533]: Upgrade: Updating mon.c 2024-09-21T10:01:02.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:02 smithi070 ceph-mon[41533]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:02.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:02 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T10:01:02.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:02 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T10:01:02.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:02 smithi070 ceph-mon[41533]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:01:02.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:02 smithi070 ceph-mon[41533]: Deploying daemon mon.c on smithi070 2024-09-21T10:01:02.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:02 smithi070 ceph-mon[41533]: osdmap e355: 8 total, 8 up, 8 in 2024-09-21T10:01:02.401 INFO:tasks.workunit.client.0.smithi070.stdout:write: c 117657612~3113693 2024-09-21T10:01:02.403 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 107826484~6773849 2024-09-21T10:01:02.404 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: user2 2024-09-21T10:01:03.057 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:02 smithi070 systemd[1]: Stopping Ceph mon.c for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:01:03.363 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:01:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:01:03] ENGINE Bus STOPPING 2024-09-21T10:01:03.363 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:01:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:01:03] "GET /metrics HTTP/1.1" 200 35885 "" "Prometheus/2.51.0" 2024-09-21T10:01:03.363 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:01:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:01:03] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T10:01:03.363 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:01:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:01:03] ENGINE Bus STOPPED 2024-09-21T10:01:03.363 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-c[41510]: 2024-09-21T10:01:03.054+0000 7f258c0e9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:01:03.363 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-c[41510]: 2024-09-21T10:01:03.054+0000 7f258c0e9640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-21T10:01:03.363 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:03 smithi070 podman[111951]: 2024-09-21 10:01:03.075489105 +0000 UTC m=+0.128149834 container died fc0590f324b52865408b15b681e6c566f6ebc4944b433f0c49250a4582f9ff3f (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-c, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=reef-6a1bdf8, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:01:03.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:03 smithi070 podman[111951]: 2024-09-21 10:01:03.460468351 +0000 UTC m=+0.513129073 container cleanup fc0590f324b52865408b15b681e6c566f6ebc4944b433f0c49250a4582f9ff3f (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-c, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, RELEASE=reef-6a1bdf8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-21T10:01:03.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:03 smithi070 bash[111951]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-c 2024-09-21T10:01:04.038 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:03 smithi070 podman[111969]: 2024-09-21 10:01:03.703241121 +0000 UTC m=+0.627671520 container remove fc0590f324b52865408b15b681e6c566f6ebc4944b433f0c49250a4582f9ff3f (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-c, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-6a1bdf8, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:01:04.038 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:03 smithi070 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mon.c.service: Deactivated successfully. 2024-09-21T10:01:04.038 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:03 smithi070 systemd[1]: Stopped Ceph mon.c for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:01:04.038 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:03 smithi070 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mon.c.service: Consumed 8.425s CPU time. 2024-09-21T10:01:04.038 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 systemd[1]: Starting Ceph mon.c for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:01:04.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 podman[112100]: 2024-09-21 10:01:04.143554623 +0000 UTC m=+0.015572831 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:01:04.659 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 podman[112100]: 2024-09-21 10:01:04.396254214 +0000 UTC m=+0.268272456 container create e154380b80a2464f25105dce2a8896a64240f04aa62fcb8628f82ad9410e8275 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-c, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-21T10:01:04.659 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 podman[112100]: 2024-09-21 10:01:04.598721601 +0000 UTC m=+0.470739806 container init e154380b80a2464f25105dce2a8896a64240f04aa62fcb8628f82ad9410e8275 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-c, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:01:04.660 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 podman[112100]: 2024-09-21 10:01:04.603475392 +0000 UTC m=+0.475493592 container start e154380b80a2464f25105dce2a8896a64240f04aa62fcb8628f82ad9410e8275 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-c, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-21T10:01:04.660 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T10:01:04.660 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev), process ceph-mon, pid 7 2024-09-21T10:01:04.660 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: pidfile_write: ignore empty --pid-file 2024-09-21T10:01:04.660 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: load: jerasure load: lrc 2024-09-21T10:01:04.660 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: RocksDB version: 7.9.2 2024-09-21T10:01:04.660 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Git sha 0 2024-09-21T10:01:04.660 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Compile date 2024-09-20 20:32:28 2024-09-21T10:01:04.660 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: DB SUMMARY 2024-09-21T10:01:04.660 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: DB Session ID: 17DI05OTB91RY637YWY8 2024-09-21T10:01:04.660 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: CURRENT file: CURRENT 2024-09-21T10:01:04.660 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T10:01:04.660 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: MANIFEST file: MANIFEST-000010 size: 1413 Bytes 2024-09-21T10:01:04.660 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-09-21T10:01:04.661 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 12739008 ; 2024-09-21T10:01:04.661 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.error_if_exists: 0 2024-09-21T10:01:04.661 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.create_if_missing: 0 2024-09-21T10:01:04.661 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.paranoid_checks: 1 2024-09-21T10:01:04.662 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T10:01:04.662 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T10:01:04.662 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T10:01:04.662 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.env: 0x55f25fb9b120 2024-09-21T10:01:04.662 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T10:01:04.662 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.info_log: 0x55f261e5b700 2024-09-21T10:01:04.662 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T10:01:04.662 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.statistics: (nil) 2024-09-21T10:01:04.662 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.use_fsync: 0 2024-09-21T10:01:04.662 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.max_log_file_size: 0 2024-09-21T10:01:04.663 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T10:01:04.663 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T10:01:04.663 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T10:01:04.663 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T10:01:04.663 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.allow_fallocate: 1 2024-09-21T10:01:04.663 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T10:01:04.663 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T10:01:04.663 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.use_direct_reads: 0 2024-09-21T10:01:04.663 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T10:01:04.663 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T10:01:04.663 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.db_log_dir: 2024-09-21T10:01:04.663 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.wal_dir: 2024-09-21T10:01:04.663 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T10:01:04.663 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T10:01:04.664 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T10:01:04.664 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T10:01:04.664 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T10:01:04.664 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T10:01:04.664 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T10:01:04.664 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T10:01:04.664 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.write_buffer_manager: 0x55f261e5e1e0 2024-09-21T10:01:04.664 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T10:01:04.664 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T10:01:04.664 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T10:01:04.664 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T10:01:04.664 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T10:01:04.664 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T10:01:04.664 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T10:01:04.665 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T10:01:04.665 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.unordered_write: 0 2024-09-21T10:01:04.665 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T10:01:04.665 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T10:01:04.665 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T10:01:04.665 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T10:01:04.665 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.row_cache: None 2024-09-21T10:01:04.665 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.wal_filter: None 2024-09-21T10:01:04.665 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T10:01:04.665 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T10:01:04.665 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.two_write_queues: 0 2024-09-21T10:01:04.665 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T10:01:04.665 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.wal_compression: 0 2024-09-21T10:01:04.665 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.atomic_flush: 0 2024-09-21T10:01:04.666 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T10:01:04.666 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T10:01:04.666 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T10:01:04.666 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.log_readahead_size: 0 2024-09-21T10:01:04.666 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T10:01:04.666 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T10:01:04.666 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T10:01:04.666 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T10:01:04.666 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T10:01:04.666 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T10:01:04.666 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T10:01:04.666 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.max_background_jobs: 2 2024-09-21T10:01:04.666 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.max_background_compactions: -1 2024-09-21T10:01:04.667 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.max_subcompactions: 1 2024-09-21T10:01:04.667 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T10:01:04.667 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T10:01:04.667 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T10:01:04.667 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T10:01:04.667 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T10:01:04.667 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T10:01:04.667 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T10:01:04.667 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T10:01:04.667 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.max_open_files: -1 2024-09-21T10:01:04.667 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T10:01:04.667 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T10:01:04.667 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T10:01:04.668 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T10:01:04.668 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.max_background_flushes: -1 2024-09-21T10:01:04.668 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Compression algorithms supported: 2024-09-21T10:01:04.668 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: kZSTD supported: 0 2024-09-21T10:01:04.668 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: kXpressCompression supported: 0 2024-09-21T10:01:04.668 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: kBZip2Compression supported: 0 2024-09-21T10:01:04.668 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T10:01:04.668 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: kLZ4Compression supported: 1 2024-09-21T10:01:04.668 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: kZlibCompression supported: 1 2024-09-21T10:01:04.668 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T10:01:04.668 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: kSnappyCompression supported: 1 2024-09-21T10:01:04.668 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T10:01:04.668 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T10:01:04.668 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000010 2024-09-21T10:01:04.669 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T10:01:04.669 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T10:01:04.669 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.merge_operator: 2024-09-21T10:01:04.669 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.compaction_filter: None 2024-09-21T10:01:04.669 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T10:01:04.669 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T10:01:04.669 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T10:01:04.669 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T10:01:04.669 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f261e5b320) 2024-09-21T10:01:04.669 INFO:journalctl@ceph.mon.c.smithi070.stdout: cache_index_and_filter_blocks: 1 2024-09-21T10:01:04.669 INFO:journalctl@ceph.mon.c.smithi070.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T10:01:04.669 INFO:journalctl@ceph.mon.c.smithi070.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T10:01:04.669 INFO:journalctl@ceph.mon.c.smithi070.stdout: pin_top_level_index_and_filter: 1 2024-09-21T10:01:04.669 INFO:journalctl@ceph.mon.c.smithi070.stdout: index_type: 0 2024-09-21T10:01:04.669 INFO:journalctl@ceph.mon.c.smithi070.stdout: data_block_index_type: 0 2024-09-21T10:01:04.670 INFO:journalctl@ceph.mon.c.smithi070.stdout: index_shortening: 1 2024-09-21T10:01:04.670 INFO:journalctl@ceph.mon.c.smithi070.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T10:01:04.670 INFO:journalctl@ceph.mon.c.smithi070.stdout: checksum: 4 2024-09-21T10:01:04.670 INFO:journalctl@ceph.mon.c.smithi070.stdout: no_block_cache: 0 2024-09-21T10:01:04.670 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_cache: 0x55f261e4f610 2024-09-21T10:01:04.670 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_cache_name: BinnedLRUCache 2024-09-21T10:01:04.670 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_cache_options: 2024-09-21T10:01:04.670 INFO:journalctl@ceph.mon.c.smithi070.stdout: capacity : 536870912 2024-09-21T10:01:04.670 INFO:journalctl@ceph.mon.c.smithi070.stdout: num_shard_bits : 4 2024-09-21T10:01:04.670 INFO:journalctl@ceph.mon.c.smithi070.stdout: strict_capacity_limit : 0 2024-09-21T10:01:04.670 INFO:journalctl@ceph.mon.c.smithi070.stdout: high_pri_pool_ratio: 0.000 2024-09-21T10:01:04.670 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_cache_compressed: (nil) 2024-09-21T10:01:04.670 INFO:journalctl@ceph.mon.c.smithi070.stdout: persistent_cache: (nil) 2024-09-21T10:01:04.670 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_size: 4096 2024-09-21T10:01:04.670 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_size_deviation: 10 2024-09-21T10:01:04.671 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_restart_interval: 16 2024-09-21T10:01:04.671 INFO:journalctl@ceph.mon.c.smithi070.stdout: index_block_restart_interval: 1 2024-09-21T10:01:04.671 INFO:journalctl@ceph.mon.c.smithi070.stdout: metadata_block_size: 4096 2024-09-21T10:01:04.671 INFO:journalctl@ceph.mon.c.smithi070.stdout: partition_filters: 0 2024-09-21T10:01:04.671 INFO:journalctl@ceph.mon.c.smithi070.stdout: use_delta_encoding: 1 2024-09-21T10:01:04.671 INFO:journalctl@ceph.mon.c.smithi070.stdout: filter_policy: bloomfilter 2024-09-21T10:01:04.671 INFO:journalctl@ceph.mon.c.smithi070.stdout: whole_key_filtering: 1 2024-09-21T10:01:04.671 INFO:journalctl@ceph.mon.c.smithi070.stdout: verify_compression: 0 2024-09-21T10:01:04.671 INFO:journalctl@ceph.mon.c.smithi070.stdout: read_amp_bytes_per_bit: 0 2024-09-21T10:01:04.671 INFO:journalctl@ceph.mon.c.smithi070.stdout: format_version: 5 2024-09-21T10:01:04.671 INFO:journalctl@ceph.mon.c.smithi070.stdout: enable_index_compression: 1 2024-09-21T10:01:04.671 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_align: 0 2024-09-21T10:01:04.671 INFO:journalctl@ceph.mon.c.smithi070.stdout: max_auto_readahead_size: 262144 2024-09-21T10:01:04.671 INFO:journalctl@ceph.mon.c.smithi070.stdout: prepopulate_block_cache: 0 2024-09-21T10:01:04.671 INFO:journalctl@ceph.mon.c.smithi070.stdout: initial_auto_readahead_size: 8192 2024-09-21T10:01:04.672 INFO:journalctl@ceph.mon.c.smithi070.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T10:01:04.672 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T10:01:04.672 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T10:01:04.673 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.compression: NoCompression 2024-09-21T10:01:04.673 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T10:01:04.673 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T10:01:04.673 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T10:01:04.673 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.num_levels: 7 2024-09-21T10:01:04.673 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T10:01:04.674 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T10:01:04.674 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T10:01:04.674 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T10:01:04.674 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T10:01:04.674 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T10:01:04.674 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T10:01:04.674 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T10:01:04.674 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T10:01:04.674 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T10:01:04.674 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T10:01:04.674 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T10:01:04.674 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T10:01:04.674 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T10:01:04.675 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T10:01:04.675 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T10:01:04.675 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T10:01:04.675 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T10:01:04.675 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T10:01:04.675 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T10:01:04.675 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T10:01:04.675 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T10:01:04.675 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T10:01:04.675 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T10:01:04.675 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T10:01:04.675 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T10:01:04.675 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T10:01:04.675 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T10:01:04.675 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T10:01:04.676 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T10:01:04.676 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T10:01:04.676 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T10:01:04.676 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T10:01:04.676 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T10:01:04.676 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T10:01:04.676 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T10:01:04.676 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T10:01:04.676 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T10:01:04.676 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T10:01:04.676 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T10:01:04.676 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T10:01:04.676 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T10:01:04.676 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T10:01:04.676 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T10:01:04.676 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T10:01:04.677 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T10:01:04.677 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T10:01:04.677 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T10:01:04.677 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T10:01:04.677 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T10:01:04.677 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T10:01:04.677 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T10:01:04.677 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T10:01:04.677 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T10:01:04.677 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.inplace_update_support: 0 2024-09-21T10:01:04.677 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T10:01:04.677 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T10:01:04.677 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T10:01:04.677 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T10:01:04.677 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.bloom_locality: 0 2024-09-21T10:01:04.678 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.max_successive_merges: 0 2024-09-21T10:01:04.678 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T10:01:04.678 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T10:01:04.678 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T10:01:04.678 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T10:01:04.678 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.ttl: 2592000 2024-09-21T10:01:04.678 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T10:01:04.678 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T10:01:04.678 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T10:01:04.678 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.enable_blob_files: false 2024-09-21T10:01:04.678 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.min_blob_size: 0 2024-09-21T10:01:04.678 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T10:01:04.678 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T10:01:04.679 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T10:01:04.679 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T10:01:04.679 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T10:01:04.679 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T10:01:04.679 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T10:01:04.679 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T10:01:04.679 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-21T10:01:04.679 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 29, last_sequence is 13429, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 25 2024-09-21T10:01:04.679 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-09-21T10:01:04.679 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a9cc8616-e551-4cf4-ade2-e75ee5bba32b 2024-09-21T10:01:04.679 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912864629870, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-09-21T10:01:04.679 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-09-21T10:01:04.679 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912864658065, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 7953307, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 13436, "largest_seqno": 18473, "table_properties": {"data_size": 7930982, "index_size": 15966, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 5317, "raw_key_size": 48208, "raw_average_key_size": 22, "raw_value_size": 7885081, "raw_average_value_size": 3731, "num_data_blocks": 687, "num_entries": 2113, "num_filter_entries": 2113, "num_deletions": 268, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726912864, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a9cc8616-e551-4cf4-ade2-e75ee5bba32b", "db_session_id": "17DI05OTB91RY637YWY8", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-09-21T10:01:04.679 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912864658417, "job": 1, "event": "recovery_finished"} 2024-09-21T10:01:04.679 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-09-21T10:01:04.680 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-21T10:01:04.680 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T10:01:04.680 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f261e7ee00 2024-09-21T10:01:04.680 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: DB pointer 0x55f261f80000 2024-09-21T10:01:05.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T10:01:05.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T10:01:05.128 INFO:journalctl@ceph.mon.c.smithi070.stdout: ** DB Stats ** 2024-09-21T10:01:05.128 INFO:journalctl@ceph.mon.c.smithi070.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T10:01:05.128 INFO:journalctl@ceph.mon.c.smithi070.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T10:01:05.128 INFO:journalctl@ceph.mon.c.smithi070.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T10:01:05.128 INFO:journalctl@ceph.mon.c.smithi070.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T10:01:05.128 INFO:journalctl@ceph.mon.c.smithi070.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T10:01:05.128 INFO:journalctl@ceph.mon.c.smithi070.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T10:01:05.128 INFO:journalctl@ceph.mon.c.smithi070.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T10:01:05.128 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-09-21T10:01:05.128 INFO:journalctl@ceph.mon.c.smithi070.stdout: ** Compaction Stats [default] ** 2024-09-21T10:01:05.129 INFO:journalctl@ceph.mon.c.smithi070.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T10:01:05.129 INFO:journalctl@ceph.mon.c.smithi070.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T10:01:05.129 INFO:journalctl@ceph.mon.c.smithi070.stdout: L0 1/0 7.58 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 694.1 0.01 0.00 1 0.011 0 0 0.0 0.0 2024-09-21T10:01:05.129 INFO:journalctl@ceph.mon.c.smithi070.stdout: L6 1/0 8.98 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-21T10:01:05.130 INFO:journalctl@ceph.mon.c.smithi070.stdout: Sum 2/0 16.57 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 694.1 0.01 0.00 1 0.011 0 0 0.0 0.0 2024-09-21T10:01:05.131 INFO:journalctl@ceph.mon.c.smithi070.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 694.1 0.01 0.00 1 0.011 0 0 0.0 0.0 2024-09-21T10:01:05.131 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-09-21T10:01:05.131 INFO:journalctl@ceph.mon.c.smithi070.stdout: ** Compaction Stats [default] ** 2024-09-21T10:01:05.131 INFO:journalctl@ceph.mon.c.smithi070.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T10:01:05.131 INFO:journalctl@ceph.mon.c.smithi070.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T10:01:05.131 INFO:journalctl@ceph.mon.c.smithi070.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 694.1 0.01 0.00 1 0.011 0 0 0.0 0.0 2024-09-21T10:01:05.131 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-09-21T10:01:05.131 INFO:journalctl@ceph.mon.c.smithi070.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T10:01:05.131 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-09-21T10:01:05.131 INFO:journalctl@ceph.mon.c.smithi070.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T10:01:05.131 INFO:journalctl@ceph.mon.c.smithi070.stdout: Flush(GB): cumulative 0.007, interval 0.007 2024-09-21T10:01:05.131 INFO:journalctl@ceph.mon.c.smithi070.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T10:01:05.131 INFO:journalctl@ceph.mon.c.smithi070.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T10:01:05.132 INFO:journalctl@ceph.mon.c.smithi070.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T10:01:05.132 INFO:journalctl@ceph.mon.c.smithi070.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T10:01:05.132 INFO:journalctl@ceph.mon.c.smithi070.stdout: Cumulative compaction: 0.01 GB write, 237.06 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T10:01:05.132 INFO:journalctl@ceph.mon.c.smithi070.stdout: Interval compaction: 0.01 GB write, 237.06 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T10:01:05.132 INFO:journalctl@ceph.mon.c.smithi070.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T10:01:05.132 INFO:journalctl@ceph.mon.c.smithi070.stdout: Block cache BinnedLRUCache@0x55f261e4f610#7 capacity: 512.00 MB usage: 22.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-09-21T10:01:05.132 INFO:journalctl@ceph.mon.c.smithi070.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,6.03 KB,0.00115037%) IndexBlock(1,16.08 KB,0.00306666%) Misc(1,0.00 KB,0%) 2024-09-21T10:01:05.132 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-09-21T10:01:05.132 INFO:journalctl@ceph.mon.c.smithi070.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T10:01:05.132 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: starting mon.c rank 2 at public addrs [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] at bind addrs [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T10:01:05.132 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: mon.c@-1(???) e3 preinit fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T10:01:05.132 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: mon.c@-1(???).mds e9 new map 2024-09-21T10:01:05.133 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: mon.c@-1(???).mds e9 print_map 2024-09-21T10:01:05.133 INFO:journalctl@ceph.mon.c.smithi070.stdout: e9 2024-09-21T10:01:05.133 INFO:journalctl@ceph.mon.c.smithi070.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-21T10:01:05.133 INFO:journalctl@ceph.mon.c.smithi070.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T10:01:05.133 INFO:journalctl@ceph.mon.c.smithi070.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T10:01:05.133 INFO:journalctl@ceph.mon.c.smithi070.stdout: legacy client fscid: 1 2024-09-21T10:01:05.133 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-09-21T10:01:05.133 INFO:journalctl@ceph.mon.c.smithi070.stdout: Filesystem 'foo' (1) 2024-09-21T10:01:05.133 INFO:journalctl@ceph.mon.c.smithi070.stdout: fs_name foo 2024-09-21T10:01:05.133 INFO:journalctl@ceph.mon.c.smithi070.stdout: epoch 8 2024-09-21T10:01:05.133 INFO:journalctl@ceph.mon.c.smithi070.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-21T10:01:05.133 INFO:journalctl@ceph.mon.c.smithi070.stdout: created 2024-09-21T09:54:53.620516+0000 2024-09-21T10:01:05.133 INFO:journalctl@ceph.mon.c.smithi070.stdout: modified 2024-09-21T09:55:04.281129+0000 2024-09-21T10:01:05.134 INFO:journalctl@ceph.mon.c.smithi070.stdout: tableserver 0 2024-09-21T10:01:05.134 INFO:journalctl@ceph.mon.c.smithi070.stdout: root 0 2024-09-21T10:01:05.134 INFO:journalctl@ceph.mon.c.smithi070.stdout: session_timeout 60 2024-09-21T10:01:05.134 INFO:journalctl@ceph.mon.c.smithi070.stdout: session_autoclose 300 2024-09-21T10:01:05.134 INFO:journalctl@ceph.mon.c.smithi070.stdout: max_file_size 1099511627776 2024-09-21T10:01:05.134 INFO:journalctl@ceph.mon.c.smithi070.stdout: max_xattr_size 65536 2024-09-21T10:01:05.134 INFO:journalctl@ceph.mon.c.smithi070.stdout: required_client_features {} 2024-09-21T10:01:05.134 INFO:journalctl@ceph.mon.c.smithi070.stdout: last_failure 0 2024-09-21T10:01:05.134 INFO:journalctl@ceph.mon.c.smithi070.stdout: last_failure_osd_epoch 0 2024-09-21T10:01:05.134 INFO:journalctl@ceph.mon.c.smithi070.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T10:01:05.134 INFO:journalctl@ceph.mon.c.smithi070.stdout: max_mds 1 2024-09-21T10:01:05.134 INFO:journalctl@ceph.mon.c.smithi070.stdout: in 0 2024-09-21T10:01:05.134 INFO:journalctl@ceph.mon.c.smithi070.stdout: up {0=14652} 2024-09-21T10:01:05.135 INFO:journalctl@ceph.mon.c.smithi070.stdout: failed 2024-09-21T10:01:05.135 INFO:journalctl@ceph.mon.c.smithi070.stdout: damaged 2024-09-21T10:01:05.135 INFO:journalctl@ceph.mon.c.smithi070.stdout: stopped 2024-09-21T10:01:05.135 INFO:journalctl@ceph.mon.c.smithi070.stdout: data_pools [4] 2024-09-21T10:01:05.135 INFO:journalctl@ceph.mon.c.smithi070.stdout: metadata_pool 3 2024-09-21T10:01:05.135 INFO:journalctl@ceph.mon.c.smithi070.stdout: inline_data disabled 2024-09-21T10:01:05.135 INFO:journalctl@ceph.mon.c.smithi070.stdout: balancer 2024-09-21T10:01:05.135 INFO:journalctl@ceph.mon.c.smithi070.stdout: bal_rank_mask -1 2024-09-21T10:01:05.135 INFO:journalctl@ceph.mon.c.smithi070.stdout: standby_count_wanted 1 2024-09-21T10:01:05.135 INFO:journalctl@ceph.mon.c.smithi070.stdout: qdb_cluster leader: 0 members: 2024-09-21T10:01:05.135 INFO:journalctl@ceph.mon.c.smithi070.stdout: [mds.foo.smithi070.gansda{0:14652} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.70:6834/1361282039,v1:172.21.15.70:6835/1361282039] compat {c=[1],r=[1],i=[7ff]}] 2024-09-21T10:01:05.135 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-09-21T10:01:05.135 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-09-21T10:01:05.136 INFO:journalctl@ceph.mon.c.smithi070.stdout: Standby daemons: 2024-09-21T10:01:05.136 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-09-21T10:01:05.136 INFO:journalctl@ceph.mon.c.smithi070.stdout: [mds.foo.smithi160.byyhfz{-1:24457} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.160:6832/2992374343,v1:172.21.15.160:6833/2992374343] compat {c=[1],r=[1],i=[7ff]}] 2024-09-21T10:01:05.136 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: mon.c@-1(???).osd e356 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-21T10:01:05.136 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: mon.c@-1(???).osd e356 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T10:01:05.136 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: mon.c@-1(???).osd e356 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T10:01:05.136 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: mon.c@-1(???).osd e356 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T10:01:05.136 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 ceph-mon[112139]: mon.c@-1(???).paxosservice(auth 1..20) refresh upgraded, format 0 -> 3 2024-09-21T10:01:05.136 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 bash[112100]: e154380b80a2464f25105dce2a8896a64240f04aa62fcb8628f82ad9410e8275 2024-09-21T10:01:05.136 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:04 smithi070 systemd[1]: Started Ceph mon.c for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:01:10.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:09 smithi160 ceph-mon[33203]: pgmap v20: 137 pgs: 137 active+clean; 2.3 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 35 MiB/s rd, 45 MiB/s wr, 1.18k op/s 2024-09-21T10:01:10.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:09 smithi160 ceph-mon[33203]: osdmap e356: 8 total, 8 up, 8 in 2024-09-21T10:01:10.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:09 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T10:01:10.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:09 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T10:01:10.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:09 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T10:01:10.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:09 smithi070 ceph-mon[107385]: pgmap v20: 137 pgs: 137 active+clean; 2.3 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 35 MiB/s rd, 45 MiB/s wr, 1.18k op/s 2024-09-21T10:01:10.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:09 smithi070 ceph-mon[107385]: osdmap e356: 8 total, 8 up, 8 in 2024-09-21T10:01:10.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:09 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T10:01:10.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:09 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T10:01:10.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:09 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T10:01:10.378 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:01:09 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:01:09] ENGINE Bus STARTING 2024-09-21T10:01:10.378 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:01:09 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:01:09] ENGINE Serving on http://:::9283 2024-09-21T10:01:10.378 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:01:09 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:01:09] ENGINE Bus STARTED 2024-09-21T10:01:10.563 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-09-21T10:01:11.202 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T10:01:11.202 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (98s) 16s ago 7m 20.4M - 0.27.0 11f11916f8cd 89ae95182419 2024-09-21T10:01:11.202 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi160 *:3000 running (76s) 15s ago 7m 82.9M - 10.4.8 5aad1d7cf004 daa2d8dfab48 2024-09-21T10:01:11.202 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi070.gansda smithi070 running (6m) 16s ago 6m 19.3M - 18.2.4-871-g6a1bdf85 0adab276c97e 6016b2501c8a 2024-09-21T10:01:11.202 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi160.byyhfz smithi160 running (6m) 15s ago 6m 15.4M - 18.2.4-871-g6a1bdf85 0adab276c97e 93e108d91e65 2024-09-21T10:01:11.202 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi160 *:8443,9283,8765 running (2m) 15s ago 11m 453M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 d952652a9683 2024-09-21T10:01:11.202 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (3m) 16s ago 13m 489M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 e3d390893cff 2024-09-21T10:01:11.202 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (28s) 16s ago 13m 32.8M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 4a7b667a4033 2024-09-21T10:01:11.202 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi160 running (11m) 15s ago 11m 65.3M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e 0fd16436dfa0 2024-09-21T10:01:11.202 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 starting - - - 2048M 2024-09-21T10:01:11.202 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (2m) 16s ago 7m 11.6M - 1.7.0 72c9c2088986 5d5d4b026070 2024-09-21T10:01:11.203 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi160 *:9100 running (2m) 15s ago 7m 11.5M - 1.7.0 72c9c2088986 19b8e0e8ee0c 2024-09-21T10:01:11.203 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (11m) 16s ago 11m 161M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 10580e80f007 2024-09-21T10:01:11.203 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (10m) 16s ago 10m 133M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e b7c0d1169d1d 2024-09-21T10:01:11.203 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (10m) 16s ago 10m 132M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 58a11b8ddb81 2024-09-21T10:01:11.203 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (9m) 16s ago 9m 127M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e aa0c8a732a41 2024-09-21T10:01:11.203 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi160 running (9m) 15s ago 9m 138M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 7325a075c488 2024-09-21T10:01:11.203 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi160 running (9m) 15s ago 9m 110M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e f74026345641 2024-09-21T10:01:11.203 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi160 running (8m) 15s ago 8m 168M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 03e6bb54f8b2 2024-09-21T10:01:11.203 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi160 running (8m) 15s ago 8m 136M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e e405b21a6146 2024-09-21T10:01:11.203 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi160 *:9095 running (118s) 15s ago 7m 44.9M - 2.51.0 1d3b7f56885b 648952e57ad7 2024-09-21T10:01:11.203 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[107385]: mon.c calling monitor election 2024-09-21T10:01:11.204 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[107385]: mon.b calling monitor election 2024-09-21T10:01:11.204 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[107385]: mon.a calling monitor election 2024-09-21T10:01:11.204 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[107385]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T10:01:11.204 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[107385]: monmap epoch 3 2024-09-21T10:01:11.204 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[107385]: fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T10:01:11.204 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[107385]: last_changed 2024-09-21T09:49:37.426189+0000 2024-09-21T10:01:11.204 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[107385]: created 2024-09-21T09:47:56.277550+0000 2024-09-21T10:01:11.204 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[107385]: min_mon_release 18 (reef) 2024-09-21T10:01:11.204 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[107385]: election_strategy: 1 2024-09-21T10:01:11.204 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[107385]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-21T10:01:11.204 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[107385]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.b 2024-09-21T10:01:11.204 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[107385]: 2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-21T10:01:11.204 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[107385]: fsmap foo:1 {0=foo.smithi070.gansda=up:active} 1 up:standby 2024-09-21T10:01:11.205 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[107385]: osdmap e356: 8 total, 8 up, 8 in 2024-09-21T10:01:11.205 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[107385]: mgrmap e39: y(active, since 20s), standbys: x 2024-09-21T10:01:11.205 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[107385]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-21T10:01:11.205 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[107385]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-21T10:01:11.205 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[107385]: application not enabled on pool 'test-librbd-smithi070-86016-9' 2024-09-21T10:01:11.205 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[107385]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-21T10:01:11.205 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[112139]: pgmap v20: 137 pgs: 137 active+clean; 2.3 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 35 MiB/s rd, 45 MiB/s wr, 1.18k op/s 2024-09-21T10:01:11.205 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[112139]: osdmap e356: 8 total, 8 up, 8 in 2024-09-21T10:01:11.205 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T10:01:11.205 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T10:01:11.205 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T10:01:11.205 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[112139]: mon.c calling monitor election 2024-09-21T10:01:11.206 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[112139]: mon.b calling monitor election 2024-09-21T10:01:11.206 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[112139]: mon.a calling monitor election 2024-09-21T10:01:11.206 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[112139]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T10:01:11.206 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[112139]: monmap epoch 3 2024-09-21T10:01:11.206 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[112139]: fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T10:01:11.206 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[112139]: last_changed 2024-09-21T09:49:37.426189+0000 2024-09-21T10:01:11.206 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[112139]: created 2024-09-21T09:47:56.277550+0000 2024-09-21T10:01:11.206 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[112139]: min_mon_release 18 (reef) 2024-09-21T10:01:11.206 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[112139]: election_strategy: 1 2024-09-21T10:01:11.206 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[112139]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-21T10:01:11.206 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[112139]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.b 2024-09-21T10:01:11.206 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[112139]: 2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-21T10:01:11.206 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[112139]: fsmap foo:1 {0=foo.smithi070.gansda=up:active} 1 up:standby 2024-09-21T10:01:11.206 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[112139]: osdmap e356: 8 total, 8 up, 8 in 2024-09-21T10:01:11.207 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[112139]: mgrmap e39: y(active, since 20s), standbys: x 2024-09-21T10:01:11.207 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[112139]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-21T10:01:11.207 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[112139]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-21T10:01:11.207 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[112139]: application not enabled on pool 'test-librbd-smithi070-86016-9' 2024-09-21T10:01:11.207 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:10 smithi070 ceph-mon[112139]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-21T10:01:11.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:10 smithi160 ceph-mon[33203]: mon.c calling monitor election 2024-09-21T10:01:11.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:10 smithi160 ceph-mon[33203]: mon.b calling monitor election 2024-09-21T10:01:11.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:10 smithi160 ceph-mon[33203]: mon.a calling monitor election 2024-09-21T10:01:11.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:10 smithi160 ceph-mon[33203]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T10:01:11.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:10 smithi160 ceph-mon[33203]: monmap epoch 3 2024-09-21T10:01:11.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:10 smithi160 ceph-mon[33203]: fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T10:01:11.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:10 smithi160 ceph-mon[33203]: last_changed 2024-09-21T09:49:37.426189+0000 2024-09-21T10:01:11.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:10 smithi160 ceph-mon[33203]: created 2024-09-21T09:47:56.277550+0000 2024-09-21T10:01:11.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:10 smithi160 ceph-mon[33203]: min_mon_release 18 (reef) 2024-09-21T10:01:11.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:10 smithi160 ceph-mon[33203]: election_strategy: 1 2024-09-21T10:01:11.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:10 smithi160 ceph-mon[33203]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-21T10:01:11.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:10 smithi160 ceph-mon[33203]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.b 2024-09-21T10:01:11.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:10 smithi160 ceph-mon[33203]: 2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-21T10:01:11.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:10 smithi160 ceph-mon[33203]: fsmap foo:1 {0=foo.smithi070.gansda=up:active} 1 up:standby 2024-09-21T10:01:11.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:10 smithi160 ceph-mon[33203]: osdmap e356: 8 total, 8 up, 8 in 2024-09-21T10:01:11.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:10 smithi160 ceph-mon[33203]: mgrmap e39: y(active, since 20s), standbys: x 2024-09-21T10:01:11.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:10 smithi160 ceph-mon[33203]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-21T10:01:11.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:10 smithi160 ceph-mon[33203]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-21T10:01:11.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:10 smithi160 ceph-mon[33203]: application not enabled on pool 'test-librbd-smithi070-86016-9' 2024-09-21T10:01:11.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:10 smithi160 ceph-mon[33203]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-21T10:01:11.551 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:01:11.551 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-09-21T10:01:11.551 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 1, 2024-09-21T10:01:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T10:01:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:01:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-09-21T10:01:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T10:01:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:01:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-09-21T10:01:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 8 2024-09-21T10:01:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:01:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": { 2024-09-21T10:01:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 2 2024-09-21T10:01:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:01:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-09-21T10:01:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 11, 2024-09-21T10:01:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 4 2024-09-21T10:01:11.553 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:01:11.553 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:01:11.852 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:01:11.852 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47", 2024-09-21T10:01:11.852 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-09-21T10:01:11.853 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) mon. Upgrade limited to 1 daemons (0 remaining).", 2024-09-21T10:01:11.853 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-09-21T10:01:11.853 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "1/3 daemons upgraded", 2024-09-21T10:01:11.853 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading mon daemons", 2024-09-21T10:01:11.853 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-09-21T10:01:11.853 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:01:11.944 INFO:tasks.workunit.client.0.smithi070.stdout:write: d 28628572~8194199 2024-09-21T10:01:11.955 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 126246367~1827958 2024-09-21T10:01:11.956 INFO:tasks.workunit.client.0.smithi070.stdout:write: d 90661242~405556 2024-09-21T10:01:11.956 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 35193438~2587558 2024-09-21T10:01:11.992 INFO:tasks.workunit.client.0.smithi070.stdout:write: d 77823842~6864953 2024-09-21T10:01:11.999 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 14805221~4171745 2024-09-21T10:01:11.999 INFO:tasks.workunit.client.0.smithi070.stdout:write: d 38028207~4368133 2024-09-21T10:01:12.008 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 23637823~7000199 2024-09-21T10:01:12.009 INFO:tasks.workunit.client.0.smithi070.stdout:snap_create: user3 2024-09-21T10:01:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:11 smithi070 ceph-mon[107385]: from='client.34143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:11 smithi070 ceph-mon[107385]: from='client.34149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:11 smithi070 ceph-mon[107385]: osdmap e357: 8 total, 8 up, 8 in 2024-09-21T10:01:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:11 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2031815828' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:01:12.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:11 smithi070 ceph-mon[112139]: from='client.34143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:12.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:11 smithi070 ceph-mon[112139]: from='client.34149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:12.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:11 smithi070 ceph-mon[112139]: osdmap e357: 8 total, 8 up, 8 in 2024-09-21T10:01:12.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:11 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2031815828' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:01:12.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:11 smithi160 ceph-mon[33203]: from='client.34143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:12.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:11 smithi160 ceph-mon[33203]: from='client.34149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:12.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:11 smithi160 ceph-mon[33203]: osdmap e357: 8 total, 8 up, 8 in 2024-09-21T10:01:12.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:11 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/2031815828' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:01:13.255 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:01:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:01:13] "GET /metrics HTTP/1.1" 200 38723 "" "Prometheus/2.51.0" 2024-09-21T10:01:13.256 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:12 smithi070 ceph-mon[112139]: from='client.34155 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:13.256 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:12 smithi070 ceph-mon[112139]: from='client.34167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:13.256 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:12 smithi070 ceph-mon[112139]: pgmap v27: 137 pgs: 137 active+clean; 2.4 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 759 KiB/s rd, 6.0 MiB/s wr, 53 op/s 2024-09-21T10:01:13.256 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:12 smithi070 ceph-mon[112139]: osdmap e358: 8 total, 8 up, 8 in 2024-09-21T10:01:13.256 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:12 smithi070 ceph-mon[107385]: from='client.34155 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:13.256 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:12 smithi070 ceph-mon[107385]: from='client.34167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:13.256 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:12 smithi070 ceph-mon[107385]: pgmap v27: 137 pgs: 137 active+clean; 2.4 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 759 KiB/s rd, 6.0 MiB/s wr, 53 op/s 2024-09-21T10:01:13.256 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:12 smithi070 ceph-mon[107385]: osdmap e358: 8 total, 8 up, 8 in 2024-09-21T10:01:13.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:12 smithi160 ceph-mon[33203]: from='client.34155 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:13.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:12 smithi160 ceph-mon[33203]: from='client.34167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:13.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:12 smithi160 ceph-mon[33203]: pgmap v27: 137 pgs: 137 active+clean; 2.4 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 759 KiB/s rd, 6.0 MiB/s wr, 53 op/s 2024-09-21T10:01:13.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:12 smithi160 ceph-mon[33203]: osdmap e358: 8 total, 8 up, 8 in 2024-09-21T10:01:13.967 INFO:tasks.workunit.client.0.smithi070.stdout:write: 1 14216886~3554221 2024-09-21T10:01:13.970 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 15105441~1777110 2024-09-21T10:01:13.988 INFO:tasks.workunit.client.0.smithi070.stdout:write: 2 28433773~3554221 2024-09-21T10:01:13.989 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 29322328~1777110 2024-09-21T10:01:14.008 INFO:tasks.workunit.client.0.smithi070.stdout:write: 3 42650659~3554221 2024-09-21T10:01:14.009 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 43539214~1777110 2024-09-21T10:01:14.037 INFO:tasks.workunit.client.0.smithi070.stdout:write: 4 56867546~3554221 2024-09-21T10:01:14.037 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 57756101~1777110 2024-09-21T10:01:14.055 INFO:tasks.workunit.client.0.smithi070.stdout:write: 5 71084433~3554221 2024-09-21T10:01:14.056 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 71972988~1777110 2024-09-21T10:01:14.175 INFO:tasks.workunit.client.0.smithi070.stdout:migration_commit 2024-09-21T10:01:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:13 smithi160 ceph-mon[33203]: osdmap e359: 8 total, 8 up, 8 in 2024-09-21T10:01:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:13 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:13 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:14.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:13 smithi070 ceph-mon[107385]: osdmap e359: 8 total, 8 up, 8 in 2024-09-21T10:01:14.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:13 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:14.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:13 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:14.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:13 smithi070 ceph-mon[112139]: osdmap e359: 8 total, 8 up, 8 in 2024-09-21T10:01:14.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:13 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:14.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:13 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:14.934 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:01:14 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T10:01:14.669+0000 7fd088d1d640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-21T10:01:15.270 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMigration.StressLive (25326 ms) 2024-09-21T10:01:15.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:14 smithi160 ceph-mon[33203]: pgmap v30: 137 pgs: 137 active+clean; 2.4 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail 2024-09-21T10:01:15.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:14 smithi160 ceph-mon[33203]: osdmap e360: 8 total, 8 up, 8 in 2024-09-21T10:01:15.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:14 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:15.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:14 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:15.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:14 smithi070 ceph-mon[107385]: pgmap v30: 137 pgs: 137 active+clean; 2.4 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail 2024-09-21T10:01:15.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:14 smithi070 ceph-mon[107385]: osdmap e360: 8 total, 8 up, 8 in 2024-09-21T10:01:15.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:14 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:15.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:14 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:15.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:14 smithi070 ceph-mon[112139]: pgmap v30: 137 pgs: 137 active+clean; 2.4 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail 2024-09-21T10:01:15.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:14 smithi070 ceph-mon[112139]: osdmap e360: 8 total, 8 up, 8 in 2024-09-21T10:01:15.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:14 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:15.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:14 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:16.962 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 57 tests from TestMigration (271730 ms total) 2024-09-21T10:01:16.962 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:01:16.962 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 6 tests from TestGroup 2024-09-21T10:01:17.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:16 smithi160 ceph-mon[33203]: pgmap v32: 137 pgs: 137 active+clean; 2.4 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail 2024-09-21T10:01:17.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:16 smithi160 ceph-mon[33203]: osdmap e361: 8 total, 8 up, 8 in 2024-09-21T10:01:17.348 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:16 smithi070 ceph-mon[107385]: pgmap v32: 137 pgs: 137 active+clean; 2.4 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail 2024-09-21T10:01:17.348 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:16 smithi070 ceph-mon[107385]: osdmap e361: 8 total, 8 up, 8 in 2024-09-21T10:01:17.348 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:16 smithi070 ceph-mon[112139]: pgmap v32: 137 pgs: 137 active+clean; 2.4 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail 2024-09-21T10:01:17.349 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:16 smithi070 ceph-mon[112139]: osdmap e361: 8 total, 8 up, 8 in 2024-09-21T10:01:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: osdmap e362: 8 total, 8 up, 8 in 2024-09-21T10:01:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:18.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:18.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:01:18.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:01:18.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:18.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:01:18.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:18.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:18.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:18.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:18.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:18.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-21T10:01:18.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-21T10:01:18.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-21T10:01:18.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-21T10:01:18.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-21T10:01:18.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-21T10:01:18.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:18.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-21T10:01:18.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-21T10:01:18.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-21T10:01:18.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-21T10:01:18.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:18.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-21T10:01:18.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-21T10:01:18.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-21T10:01:18.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-21T10:01:18.344 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:18.344 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-21T10:01:18.344 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-21T10:01:18.344 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-21T10:01:18.344 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-21T10:01:18.344 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:18.345 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-21T10:01:18.345 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-21T10:01:18.345 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-21T10:01:18.345 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-21T10:01:18.345 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:18.345 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-21T10:01:18.345 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-21T10:01:18.345 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-21T10:01:18.345 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-21T10:01:18.345 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-21T10:01:18.345 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-21T10:01:18.345 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-21T10:01:18.346 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:01:18.346 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:01:18.346 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-21T10:01:18.346 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-21T10:01:18.346 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-21T10:01:18.346 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-21T10:01:18.346 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-21T10:01:18.346 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-21T10:01:18.346 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-21T10:01:18.346 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-21T10:01:18.346 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-21T10:01:18.346 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-21T10:01:18.346 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-21T10:01:18.347 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:01:18.347 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:01:18.347 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-21T10:01:18.347 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-21T10:01:18.347 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-21T10:01:18.347 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-21T10:01:18.347 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-21T10:01:18.347 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-21T10:01:18.347 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-21T10:01:18.347 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-21T10:01:18.347 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-21T10:01:18.347 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:01:18.347 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:01:18.347 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:01:18.348 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:01:18.348 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:01:18.348 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:01:18.348 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:01:18.348 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:01:18.348 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:01:18.348 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:01:18.348 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:01:18.348 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:01:18.348 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-21T10:01:18.348 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-21T10:01:18.348 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-21T10:01:18.348 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:01:18.348 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:01:18.349 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:01:18.349 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:17 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:18.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: osdmap e362: 8 total, 8 up, 8 in 2024-09-21T10:01:18.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:18.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:18.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:01:18.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:01:18.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:18.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:01:18.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:18.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:18.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:18.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:18.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:18.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-21T10:01:18.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-21T10:01:18.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-21T10:01:18.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-21T10:01:18.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-21T10:01:18.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-21T10:01:18.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:18.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-21T10:01:18.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-21T10:01:18.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-21T10:01:18.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-21T10:01:18.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:18.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-21T10:01:18.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-21T10:01:18.380 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-21T10:01:18.380 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-21T10:01:18.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: osdmap e362: 8 total, 8 up, 8 in 2024-09-21T10:01:18.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:18.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:18.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:01:18.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:01:18.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:18.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:01:18.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:18.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:18.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:18.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:18.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:18.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-21T10:01:18.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-21T10:01:18.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-21T10:01:18.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-21T10:01:18.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-21T10:01:18.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-21T10:01:18.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:18.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-21T10:01:18.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-21T10:01:18.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-21T10:01:18.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-21T10:01:18.382 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:18.382 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-21T10:01:18.382 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-21T10:01:18.382 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-21T10:01:18.382 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-21T10:01:18.382 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:18.382 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-21T10:01:18.382 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-21T10:01:18.382 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-21T10:01:18.382 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-21T10:01:18.383 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:18.383 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-21T10:01:18.383 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-21T10:01:18.383 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-21T10:01:18.383 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-21T10:01:18.383 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:18.383 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-21T10:01:18.383 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-21T10:01:18.383 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-21T10:01:18.383 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-21T10:01:18.383 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-21T10:01:18.383 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-21T10:01:18.383 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-21T10:01:18.383 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:01:18.384 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:01:18.384 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-21T10:01:18.384 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-21T10:01:18.384 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-21T10:01:18.384 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-21T10:01:18.384 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:18.384 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-21T10:01:18.384 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-21T10:01:18.384 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-21T10:01:18.384 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-21T10:01:18.385 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:18.385 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-21T10:01:18.385 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-21T10:01:18.385 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-21T10:01:18.385 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-21T10:01:18.385 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:18.385 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-21T10:01:18.385 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-21T10:01:18.385 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-21T10:01:18.385 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-21T10:01:18.385 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-21T10:01:18.385 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-21T10:01:18.385 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-21T10:01:18.386 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:01:18.386 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:01:18.386 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-21T10:01:18.386 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-21T10:01:18.386 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-21T10:01:18.386 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-21T10:01:18.386 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-21T10:01:18.387 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-21T10:01:18.387 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-21T10:01:18.387 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-21T10:01:18.387 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-21T10:01:18.387 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-21T10:01:18.387 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-21T10:01:18.387 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:01:18.387 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:01:18.387 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-21T10:01:18.387 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-21T10:01:18.387 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-21T10:01:18.387 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-21T10:01:18.387 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-21T10:01:18.388 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-21T10:01:18.388 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-21T10:01:18.388 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-21T10:01:18.388 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-21T10:01:18.388 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:01:18.388 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:01:18.388 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:01:18.388 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:01:18.388 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:01:18.388 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:01:18.388 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:01:18.388 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:01:18.389 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:01:18.389 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:01:18.389 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:01:18.389 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:01:18.389 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-21T10:01:18.389 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-21T10:01:18.389 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-21T10:01:18.389 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:01:18.389 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:01:18.389 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:01:18.389 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:18.390 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-21T10:01:18.390 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-21T10:01:18.390 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-21T10:01:18.390 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-21T10:01:18.390 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-21T10:01:18.390 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-21T10:01:18.390 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-21T10:01:18.390 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:01:18.390 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:01:18.390 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-21T10:01:18.390 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-21T10:01:18.390 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-21T10:01:18.391 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-21T10:01:18.391 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-21T10:01:18.391 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-21T10:01:18.391 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-21T10:01:18.391 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-21T10:01:18.391 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-21T10:01:18.391 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:01:18.391 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:01:18.391 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:01:18.391 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:01:18.391 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:01:18.391 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:01:18.391 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:01:18.391 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:01:18.392 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:01:18.392 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:01:18.392 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-21T10:01:18.392 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:01:18.392 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-21T10:01:18.392 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-21T10:01:18.392 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-21T10:01:18.392 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:01:18.392 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:01:18.392 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:01:18.392 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:17 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:18.972 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestGroup.group_create 2024-09-21T10:01:19.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:18 smithi160 ceph-mon[33203]: Upgrade: Setting container_image for all nvmeof 2024-09-21T10:01:19.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:18 smithi160 ceph-mon[33203]: Upgrade: Setting container_image for all node-exporter 2024-09-21T10:01:19.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:18 smithi160 ceph-mon[33203]: Upgrade: Setting container_image for all prometheus 2024-09-21T10:01:19.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:18 smithi160 ceph-mon[33203]: Upgrade: Setting container_image for all alertmanager 2024-09-21T10:01:19.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:18 smithi160 ceph-mon[33203]: Upgrade: Setting container_image for all grafana 2024-09-21T10:01:19.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:18 smithi160 ceph-mon[33203]: Upgrade: Setting container_image for all loki 2024-09-21T10:01:19.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:18 smithi160 ceph-mon[33203]: Upgrade: Setting container_image for all promtail 2024-09-21T10:01:19.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:18 smithi160 ceph-mon[33203]: Upgrade: Finalizing container_image settings 2024-09-21T10:01:19.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:18 smithi160 ceph-mon[33203]: Upgrade: Complete! 2024-09-21T10:01:19.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:18 smithi160 ceph-mon[33203]: pgmap v35: 73 pgs: 73 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-09-21T10:01:19.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:18 smithi160 ceph-mon[33203]: osdmap e363: 8 total, 8 up, 8 in 2024-09-21T10:01:19.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:18 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/989885603' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:01:19.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:18 smithi160 ceph-mon[33203]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:01:19.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:18 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:01:19.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:18 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:01:19.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:18 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:01:19.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:18 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:19.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:18 smithi160 ceph-mon[33203]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:01:19.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:18 smithi160 ceph-mon[33203]: Cluster is now healthy 2024-09-21T10:01:19.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all nvmeof 2024-09-21T10:01:19.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all node-exporter 2024-09-21T10:01:19.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all prometheus 2024-09-21T10:01:19.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all alertmanager 2024-09-21T10:01:19.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all grafana 2024-09-21T10:01:19.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all loki 2024-09-21T10:01:19.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all promtail 2024-09-21T10:01:19.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[107385]: Upgrade: Finalizing container_image settings 2024-09-21T10:01:19.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[107385]: Upgrade: Complete! 2024-09-21T10:01:19.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[107385]: pgmap v35: 73 pgs: 73 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-09-21T10:01:19.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[107385]: osdmap e363: 8 total, 8 up, 8 in 2024-09-21T10:01:19.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/989885603' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:01:19.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:01:19.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:01:19.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:01:19.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:01:19.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:19.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[107385]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:01:19.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[107385]: Cluster is now healthy 2024-09-21T10:01:19.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all nvmeof 2024-09-21T10:01:19.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all node-exporter 2024-09-21T10:01:19.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all prometheus 2024-09-21T10:01:19.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all alertmanager 2024-09-21T10:01:19.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all grafana 2024-09-21T10:01:19.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all loki 2024-09-21T10:01:19.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all promtail 2024-09-21T10:01:19.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[112139]: Upgrade: Finalizing container_image settings 2024-09-21T10:01:19.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[112139]: Upgrade: Complete! 2024-09-21T10:01:19.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[112139]: pgmap v35: 73 pgs: 73 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-09-21T10:01:19.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[112139]: osdmap e363: 8 total, 8 up, 8 in 2024-09-21T10:01:19.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/989885603' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:01:19.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:01:19.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:01:19.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:01:19.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:01:19.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:19.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[112139]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:01:19.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:18 smithi070 ceph-mon[112139]: Cluster is now healthy 2024-09-21T10:01:20.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:19 smithi160 ceph-mon[33203]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:01:20.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:19 smithi160 ceph-mon[33203]: osdmap e364: 8 total, 8 up, 8 in 2024-09-21T10:01:20.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:19 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:01:20.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:19 smithi070 ceph-mon[107385]: osdmap e364: 8 total, 8 up, 8 in 2024-09-21T10:01:20.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:19 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:01:20.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:19 smithi070 ceph-mon[112139]: osdmap e364: 8 total, 8 up, 8 in 2024-09-21T10:01:20.999 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestGroup.group_create (2027 ms) 2024-09-21T10:01:20.999 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestGroup.group_createPP 2024-09-21T10:01:21.019 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestGroup.group_createPP (20 ms) 2024-09-21T10:01:21.020 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestGroup.add_image 2024-09-21T10:01:21.058 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestGroup.add_image (39 ms) 2024-09-21T10:01:21.058 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestGroup.add_imagePP 2024-09-21T10:01:21.092 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestGroup.add_imagePP (34 ms) 2024-09-21T10:01:21.092 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestGroup.add_snapshot 2024-09-21T10:01:21.279 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:20 smithi070 ceph-mon[107385]: pgmap v38: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-09-21T10:01:21.279 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:20 smithi070 ceph-mon[107385]: osdmap e365: 8 total, 8 up, 8 in 2024-09-21T10:01:21.279 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:20 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:21.279 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:20 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:21.279 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:01:21.280 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:20 smithi070 ceph-mon[112139]: pgmap v38: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-09-21T10:01:21.280 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:20 smithi070 ceph-mon[112139]: osdmap e365: 8 total, 8 up, 8 in 2024-09-21T10:01:21.280 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:20 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:21.280 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:20 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:21.280 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:01:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:20 smithi160 ceph-mon[33203]: pgmap v38: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-09-21T10:01:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:20 smithi160 ceph-mon[33203]: osdmap e365: 8 total, 8 up, 8 in 2024-09-21T10:01:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:20 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:20 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:01:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:20 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:01:22.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:21 smithi070 ceph-mon[107385]: osdmap e366: 8 total, 8 up, 8 in 2024-09-21T10:01:22.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:21 smithi070 ceph-mon[112139]: osdmap e366: 8 total, 8 up, 8 in 2024-09-21T10:01:22.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:21 smithi160 ceph-mon[33203]: osdmap e366: 8 total, 8 up, 8 in 2024-09-21T10:01:23.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:22 smithi070 ceph-mon[107385]: pgmap v41: 105 pgs: 105 active+clean; 582 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T10:01:23.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:22 smithi070 ceph-mon[107385]: osdmap e367: 8 total, 8 up, 8 in 2024-09-21T10:01:23.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:22 smithi070 ceph-mon[112139]: pgmap v41: 105 pgs: 105 active+clean; 582 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T10:01:23.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:22 smithi070 ceph-mon[112139]: osdmap e367: 8 total, 8 up, 8 in 2024-09-21T10:01:23.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:22 smithi160 ceph-mon[33203]: pgmap v41: 105 pgs: 105 active+clean; 582 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T10:01:23.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:22 smithi160 ceph-mon[33203]: osdmap e367: 8 total, 8 up, 8 in 2024-09-21T10:01:23.496 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:01:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:01:23] "GET /metrics HTTP/1.1" 200 36217 "" "Prometheus/2.51.0" 2024-09-21T10:01:24.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:23 smithi160 ceph-mon[33203]: osdmap e368: 8 total, 8 up, 8 in 2024-09-21T10:01:24.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:23 smithi070 ceph-mon[107385]: osdmap e368: 8 total, 8 up, 8 in 2024-09-21T10:01:24.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:23 smithi070 ceph-mon[112139]: osdmap e368: 8 total, 8 up, 8 in 2024-09-21T10:01:25.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:24 smithi160 ceph-mon[33203]: pgmap v44: 105 pgs: 105 active+clean; 582 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T10:01:25.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:24 smithi160 ceph-mon[33203]: osdmap e369: 8 total, 8 up, 8 in 2024-09-21T10:01:25.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:24 smithi070 ceph-mon[107385]: pgmap v44: 105 pgs: 105 active+clean; 582 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T10:01:25.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:24 smithi070 ceph-mon[107385]: osdmap e369: 8 total, 8 up, 8 in 2024-09-21T10:01:25.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:24 smithi070 ceph-mon[112139]: pgmap v44: 105 pgs: 105 active+clean; 582 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T10:01:25.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:24 smithi070 ceph-mon[112139]: osdmap e369: 8 total, 8 up, 8 in 2024-09-21T10:01:26.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:26 smithi160 ceph-mon[33203]: osdmap e370: 8 total, 8 up, 8 in 2024-09-21T10:01:26.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:26 smithi070 ceph-mon[107385]: osdmap e370: 8 total, 8 up, 8 in 2024-09-21T10:01:26.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:26 smithi070 ceph-mon[112139]: osdmap e370: 8 total, 8 up, 8 in 2024-09-21T10:01:27.043 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestGroup.add_snapshot (5951 ms) 2024-09-21T10:01:27.043 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestGroup.add_snapshotPP 2024-09-21T10:01:27.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:27 smithi160 ceph-mon[33203]: pgmap v47: 105 pgs: 105 active+clean; 582 KiB data, 358 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:01:27.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:27 smithi160 ceph-mon[33203]: osdmap e371: 8 total, 8 up, 8 in 2024-09-21T10:01:27.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:27 smithi070 ceph-mon[107385]: pgmap v47: 105 pgs: 105 active+clean; 582 KiB data, 358 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:01:27.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:27 smithi070 ceph-mon[107385]: osdmap e371: 8 total, 8 up, 8 in 2024-09-21T10:01:27.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:27 smithi070 ceph-mon[112139]: pgmap v47: 105 pgs: 105 active+clean; 582 KiB data, 358 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:01:27.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:27 smithi070 ceph-mon[112139]: osdmap e371: 8 total, 8 up, 8 in 2024-09-21T10:01:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:28 smithi160 ceph-mon[33203]: osdmap e372: 8 total, 8 up, 8 in 2024-09-21T10:01:28.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:28 smithi070 ceph-mon[107385]: osdmap e372: 8 total, 8 up, 8 in 2024-09-21T10:01:28.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:28 smithi070 ceph-mon[112139]: osdmap e372: 8 total, 8 up, 8 in 2024-09-21T10:01:29.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:29 smithi160 ceph-mon[33203]: pgmap v50: 105 pgs: 105 active+clean; 582 KiB data, 359 MiB used, 715 GiB / 715 GiB avail; 164 KiB/s rd, 18 KiB/s wr, 221 op/s 2024-09-21T10:01:29.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:29 smithi160 ceph-mon[33203]: osdmap e373: 8 total, 8 up, 8 in 2024-09-21T10:01:29.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:29 smithi070 ceph-mon[107385]: pgmap v50: 105 pgs: 105 active+clean; 582 KiB data, 359 MiB used, 715 GiB / 715 GiB avail; 164 KiB/s rd, 18 KiB/s wr, 221 op/s 2024-09-21T10:01:29.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:29 smithi070 ceph-mon[107385]: osdmap e373: 8 total, 8 up, 8 in 2024-09-21T10:01:29.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:29 smithi070 ceph-mon[112139]: pgmap v50: 105 pgs: 105 active+clean; 582 KiB data, 359 MiB used, 715 GiB / 715 GiB avail; 164 KiB/s rd, 18 KiB/s wr, 221 op/s 2024-09-21T10:01:29.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:29 smithi070 ceph-mon[112139]: osdmap e373: 8 total, 8 up, 8 in 2024-09-21T10:01:30.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:30 smithi160 ceph-mon[33203]: osdmap e374: 8 total, 8 up, 8 in 2024-09-21T10:01:30.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:30 smithi070 ceph-mon[112139]: osdmap e374: 8 total, 8 up, 8 in 2024-09-21T10:01:30.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:30 smithi070 ceph-mon[107385]: osdmap e374: 8 total, 8 up, 8 in 2024-09-21T10:01:31.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:31 smithi160 ceph-mon[33203]: pgmap v53: 105 pgs: 105 active+clean; 582 KiB data, 359 MiB used, 715 GiB / 715 GiB avail; 164 KiB/s rd, 18 KiB/s wr, 221 op/s 2024-09-21T10:01:31.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:31 smithi160 ceph-mon[33203]: osdmap e375: 8 total, 8 up, 8 in 2024-09-21T10:01:31.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:31 smithi070 ceph-mon[112139]: pgmap v53: 105 pgs: 105 active+clean; 582 KiB data, 359 MiB used, 715 GiB / 715 GiB avail; 164 KiB/s rd, 18 KiB/s wr, 221 op/s 2024-09-21T10:01:31.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:31 smithi070 ceph-mon[112139]: osdmap e375: 8 total, 8 up, 8 in 2024-09-21T10:01:31.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:31 smithi070 ceph-mon[107385]: pgmap v53: 105 pgs: 105 active+clean; 582 KiB data, 359 MiB used, 715 GiB / 715 GiB avail; 164 KiB/s rd, 18 KiB/s wr, 221 op/s 2024-09-21T10:01:31.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:31 smithi070 ceph-mon[107385]: osdmap e375: 8 total, 8 up, 8 in 2024-09-21T10:01:32.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:32 smithi160 ceph-mon[33203]: osdmap e376: 8 total, 8 up, 8 in 2024-09-21T10:01:32.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:32 smithi070 ceph-mon[107385]: osdmap e376: 8 total, 8 up, 8 in 2024-09-21T10:01:32.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:32 smithi070 ceph-mon[112139]: osdmap e376: 8 total, 8 up, 8 in 2024-09-21T10:01:32.650 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestGroup.add_snapshotPP (5607 ms) 2024-09-21T10:01:33.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:33 smithi160 ceph-mon[33203]: pgmap v56: 105 pgs: 105 active+clean; 583 KiB data, 360 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 11 KiB/s wr, 148 op/s 2024-09-21T10:01:33.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:33 smithi160 ceph-mon[33203]: osdmap e377: 8 total, 8 up, 8 in 2024-09-21T10:01:33.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:33 smithi160 ceph-mon[33203]: osdmap e378: 8 total, 8 up, 8 in 2024-09-21T10:01:33.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:33 smithi070 ceph-mon[107385]: pgmap v56: 105 pgs: 105 active+clean; 583 KiB data, 360 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 11 KiB/s wr, 148 op/s 2024-09-21T10:01:33.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:33 smithi070 ceph-mon[107385]: osdmap e377: 8 total, 8 up, 8 in 2024-09-21T10:01:33.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:33 smithi070 ceph-mon[107385]: osdmap e378: 8 total, 8 up, 8 in 2024-09-21T10:01:33.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:01:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:01:33] "GET /metrics HTTP/1.1" 200 36217 "" "Prometheus/2.51.0" 2024-09-21T10:01:33.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:33 smithi070 ceph-mon[112139]: pgmap v56: 105 pgs: 105 active+clean; 583 KiB data, 360 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 11 KiB/s wr, 148 op/s 2024-09-21T10:01:33.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:33 smithi070 ceph-mon[112139]: osdmap e377: 8 total, 8 up, 8 in 2024-09-21T10:01:33.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:33 smithi070 ceph-mon[112139]: osdmap e378: 8 total, 8 up, 8 in 2024-09-21T10:01:33.633 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 6 tests from TestGroup (13678 ms total) 2024-09-21T10:01:33.633 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:01:33.633 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 31 tests from TestDeepCopy 2024-09-21T10:01:35.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:34 smithi160 ceph-mon[33203]: osdmap e379: 8 total, 8 up, 8 in 2024-09-21T10:01:35.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:34 smithi160 ceph-mon[33203]: pgmap v60: 73 pgs: 73 active+clean; 582 KiB data, 360 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:01:35.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:34 smithi070 ceph-mon[112139]: osdmap e379: 8 total, 8 up, 8 in 2024-09-21T10:01:35.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:34 smithi070 ceph-mon[112139]: pgmap v60: 73 pgs: 73 active+clean; 582 KiB data, 360 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:01:35.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:34 smithi070 ceph-mon[107385]: osdmap e379: 8 total, 8 up, 8 in 2024-09-21T10:01:35.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:34 smithi070 ceph-mon[107385]: pgmap v60: 73 pgs: 73 active+clean; 582 KiB data, 360 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:01:35.637 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.Empty 2024-09-21T10:01:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:35 smithi160 ceph-mon[33203]: osdmap e380: 8 total, 8 up, 8 in 2024-09-21T10:01:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:35 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/3401947325' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:01:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:35 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:01:36.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:35 smithi070 ceph-mon[107385]: osdmap e380: 8 total, 8 up, 8 in 2024-09-21T10:01:36.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:35 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3401947325' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:01:36.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:01:36.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:35 smithi070 ceph-mon[112139]: osdmap e380: 8 total, 8 up, 8 in 2024-09-21T10:01:36.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:35 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3401947325' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:01:36.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:01:37.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:36 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/3401947325' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:01:37.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:36 smithi160 ceph-mon[33203]: osdmap e381: 8 total, 8 up, 8 in 2024-09-21T10:01:37.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:36 smithi160 ceph-mon[33203]: pgmap v63: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 360 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:01:37.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:36 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3401947325' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:01:37.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:36 smithi070 ceph-mon[107385]: osdmap e381: 8 total, 8 up, 8 in 2024-09-21T10:01:37.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:36 smithi070 ceph-mon[107385]: pgmap v63: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 360 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:01:37.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:36 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3401947325' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:01:37.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:36 smithi070 ceph-mon[112139]: osdmap e381: 8 total, 8 up, 8 in 2024-09-21T10:01:37.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:36 smithi070 ceph-mon[112139]: pgmap v63: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 360 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:01:37.767 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestDeepCopy.Empty (2130 ms) 2024-09-21T10:01:37.767 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.NoSnaps 2024-09-21T10:01:38.012 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestDeepCopy.NoSnaps (245 ms) 2024-09-21T10:01:38.012 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.Snaps 2024-09-21T10:01:38.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:37 smithi160 ceph-mon[33203]: osdmap e382: 8 total, 8 up, 8 in 2024-09-21T10:01:38.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:37 smithi070 ceph-mon[112139]: osdmap e382: 8 total, 8 up, 8 in 2024-09-21T10:01:38.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:37 smithi070 ceph-mon[107385]: osdmap e382: 8 total, 8 up, 8 in 2024-09-21T10:01:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:38 smithi160 ceph-mon[33203]: osdmap e383: 8 total, 8 up, 8 in 2024-09-21T10:01:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:38 smithi160 ceph-mon[33203]: pgmap v66: 105 pgs: 105 active+clean; 582 KiB data, 360 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:01:39.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:38 smithi070 ceph-mon[112139]: osdmap e383: 8 total, 8 up, 8 in 2024-09-21T10:01:39.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:38 smithi070 ceph-mon[112139]: pgmap v66: 105 pgs: 105 active+clean; 582 KiB data, 360 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:01:39.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:38 smithi070 ceph-mon[107385]: osdmap e383: 8 total, 8 up, 8 in 2024-09-21T10:01:39.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:38 smithi070 ceph-mon[107385]: pgmap v66: 105 pgs: 105 active+clean; 582 KiB data, 360 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:01:40.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:39 smithi160 ceph-mon[33203]: osdmap e384: 8 total, 8 up, 8 in 2024-09-21T10:01:40.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:39 smithi070 ceph-mon[112139]: osdmap e384: 8 total, 8 up, 8 in 2024-09-21T10:01:40.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:39 smithi070 ceph-mon[107385]: osdmap e384: 8 total, 8 up, 8 in 2024-09-21T10:01:40.992 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:40 smithi160 ceph-mon[33203]: osdmap e385: 8 total, 8 up, 8 in 2024-09-21T10:01:40.993 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:40 smithi160 ceph-mon[33203]: pgmap v69: 105 pgs: 105 active+clean; 582 KiB data, 360 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:01:41.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:40 smithi070 ceph-mon[107385]: osdmap e385: 8 total, 8 up, 8 in 2024-09-21T10:01:41.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:40 smithi070 ceph-mon[107385]: pgmap v69: 105 pgs: 105 active+clean; 582 KiB data, 360 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:01:41.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:40 smithi070 ceph-mon[112139]: osdmap e385: 8 total, 8 up, 8 in 2024-09-21T10:01:41.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:40 smithi070 ceph-mon[112139]: pgmap v69: 105 pgs: 105 active+clean; 582 KiB data, 360 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:01:41.968 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestDeepCopy.Snaps (3956 ms) 2024-09-21T10:01:41.968 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.SnapDiscard 2024-09-21T10:01:42.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:41 smithi160 ceph-mon[33203]: osdmap e386: 8 total, 8 up, 8 in 2024-09-21T10:01:42.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:41 smithi070 ceph-mon[107385]: osdmap e386: 8 total, 8 up, 8 in 2024-09-21T10:01:42.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:41 smithi070 ceph-mon[112139]: osdmap e386: 8 total, 8 up, 8 in 2024-09-21T10:01:42.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:42 smithi070 ceph-mon[107385]: osdmap e387: 8 total, 8 up, 8 in 2024-09-21T10:01:42.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:42 smithi070 ceph-mon[107385]: pgmap v72: 105 pgs: 105 active+clean; 49 MiB data, 509 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 12 MiB/s wr, 176 op/s 2024-09-21T10:01:42.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:42 smithi070 ceph-mon[112139]: osdmap e387: 8 total, 8 up, 8 in 2024-09-21T10:01:42.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:42 smithi070 ceph-mon[112139]: pgmap v72: 105 pgs: 105 active+clean; 49 MiB data, 509 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 12 MiB/s wr, 176 op/s 2024-09-21T10:01:42.989 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'sleep 60' 2024-09-21T10:01:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:42 smithi160 ceph-mon[33203]: osdmap e387: 8 total, 8 up, 8 in 2024-09-21T10:01:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:42 smithi160 ceph-mon[33203]: pgmap v72: 105 pgs: 105 active+clean; 49 MiB data, 509 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 12 MiB/s wr, 176 op/s 2024-09-21T10:01:43.189 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:01:43.309 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:01:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:01:43] "GET /metrics HTTP/1.1" 200 36209 "" "Prometheus/2.51.0" 2024-09-21T10:01:43.781 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestDeepCopy.SnapDiscard (1812 ms) 2024-09-21T10:01:43.781 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.CloneDiscard 2024-09-21T10:01:43.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:43 smithi070 ceph-mon[107385]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:43.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:43 smithi070 ceph-mon[107385]: osdmap e388: 8 total, 8 up, 8 in 2024-09-21T10:01:43.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:43 smithi070 ceph-mon[112139]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:43.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:43 smithi070 ceph-mon[112139]: osdmap e388: 8 total, 8 up, 8 in 2024-09-21T10:01:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:43 smithi160 ceph-mon[33203]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:43 smithi160 ceph-mon[33203]: osdmap e388: 8 total, 8 up, 8 in 2024-09-21T10:01:44.860 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestDeepCopy.CloneDiscard (1079 ms) 2024-09-21T10:01:44.860 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.CloneShrink 2024-09-21T10:01:44.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:44 smithi070 ceph-mon[107385]: osdmap e389: 8 total, 8 up, 8 in 2024-09-21T10:01:44.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:44 smithi070 ceph-mon[107385]: pgmap v75: 105 pgs: 105 active+clean; 49 MiB data, 509 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 12 MiB/s wr, 176 op/s 2024-09-21T10:01:44.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:44 smithi070 ceph-mon[112139]: osdmap e389: 8 total, 8 up, 8 in 2024-09-21T10:01:44.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:44 smithi070 ceph-mon[112139]: pgmap v75: 105 pgs: 105 active+clean; 49 MiB data, 509 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 12 MiB/s wr, 176 op/s 2024-09-21T10:01:45.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:44 smithi160 ceph-mon[33203]: osdmap e389: 8 total, 8 up, 8 in 2024-09-21T10:01:45.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:44 smithi160 ceph-mon[33203]: pgmap v75: 105 pgs: 105 active+clean; 49 MiB data, 509 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 12 MiB/s wr, 176 op/s 2024-09-21T10:01:45.865 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestDeepCopy.CloneShrink (1005 ms) 2024-09-21T10:01:45.865 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.CloneExpand 2024-09-21T10:01:46.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:45 smithi160 ceph-mon[33203]: osdmap e390: 8 total, 8 up, 8 in 2024-09-21T10:01:46.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:45 smithi070 ceph-mon[112139]: osdmap e390: 8 total, 8 up, 8 in 2024-09-21T10:01:46.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:45 smithi070 ceph-mon[107385]: osdmap e390: 8 total, 8 up, 8 in 2024-09-21T10:01:46.910 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestDeepCopy.CloneExpand (1046 ms) 2024-09-21T10:01:46.910 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.CloneHideParent 2024-09-21T10:01:47.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:46 smithi160 ceph-mon[33203]: osdmap e391: 8 total, 8 up, 8 in 2024-09-21T10:01:47.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:46 smithi160 ceph-mon[33203]: pgmap v78: 105 pgs: 105 active+clean; 49 MiB data, 509 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:01:47.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:46 smithi070 ceph-mon[107385]: osdmap e391: 8 total, 8 up, 8 in 2024-09-21T10:01:47.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:46 smithi070 ceph-mon[107385]: pgmap v78: 105 pgs: 105 active+clean; 49 MiB data, 509 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:01:47.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:46 smithi070 ceph-mon[112139]: osdmap e391: 8 total, 8 up, 8 in 2024-09-21T10:01:47.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:46 smithi070 ceph-mon[112139]: pgmap v78: 105 pgs: 105 active+clean; 49 MiB data, 509 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:01:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:47 smithi160 ceph-mon[33203]: osdmap e392: 8 total, 8 up, 8 in 2024-09-21T10:01:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:47 smithi160 ceph-mon[33203]: osdmap e393: 8 total, 8 up, 8 in 2024-09-21T10:01:48.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:47 smithi070 ceph-mon[107385]: osdmap e392: 8 total, 8 up, 8 in 2024-09-21T10:01:48.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:47 smithi070 ceph-mon[107385]: osdmap e393: 8 total, 8 up, 8 in 2024-09-21T10:01:48.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:47 smithi070 ceph-mon[112139]: osdmap e392: 8 total, 8 up, 8 in 2024-09-21T10:01:48.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:47 smithi070 ceph-mon[112139]: osdmap e393: 8 total, 8 up, 8 in 2024-09-21T10:01:49.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:48 smithi160 ceph-mon[33203]: pgmap v81: 105 pgs: 105 active+clean; 65 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 24 MiB/s rd, 4.0 MiB/s wr, 437 op/s 2024-09-21T10:01:49.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:48 smithi070 ceph-mon[107385]: pgmap v81: 105 pgs: 105 active+clean; 65 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 24 MiB/s rd, 4.0 MiB/s wr, 437 op/s 2024-09-21T10:01:49.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:48 smithi070 ceph-mon[112139]: pgmap v81: 105 pgs: 105 active+clean; 65 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 24 MiB/s rd, 4.0 MiB/s wr, 437 op/s 2024-09-21T10:01:50.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:49 smithi160 ceph-mon[33203]: osdmap e394: 8 total, 8 up, 8 in 2024-09-21T10:01:50.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:49 smithi070 ceph-mon[107385]: osdmap e394: 8 total, 8 up, 8 in 2024-09-21T10:01:50.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:49 smithi070 ceph-mon[112139]: osdmap e394: 8 total, 8 up, 8 in 2024-09-21T10:01:51.029 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:50 smithi070 ceph-mon[107385]: osdmap e395: 8 total, 8 up, 8 in 2024-09-21T10:01:51.029 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:50 smithi070 ceph-mon[107385]: pgmap v84: 105 pgs: 105 active+clean; 65 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 24 MiB/s rd, 4.0 MiB/s wr, 437 op/s 2024-09-21T10:01:51.029 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:01:51.029 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:50 smithi070 ceph-mon[112139]: osdmap e395: 8 total, 8 up, 8 in 2024-09-21T10:01:51.029 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:50 smithi070 ceph-mon[112139]: pgmap v84: 105 pgs: 105 active+clean; 65 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 24 MiB/s rd, 4.0 MiB/s wr, 437 op/s 2024-09-21T10:01:51.029 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:01:51.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:50 smithi160 ceph-mon[33203]: osdmap e395: 8 total, 8 up, 8 in 2024-09-21T10:01:51.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:50 smithi160 ceph-mon[33203]: pgmap v84: 105 pgs: 105 active+clean; 65 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 24 MiB/s rd, 4.0 MiB/s wr, 437 op/s 2024-09-21T10:01:51.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:50 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:01:52.048 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:51 smithi070 ceph-mon[107385]: osdmap e396: 8 total, 8 up, 8 in 2024-09-21T10:01:52.048 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:51 smithi070 ceph-mon[112139]: osdmap e396: 8 total, 8 up, 8 in 2024-09-21T10:01:52.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:51 smithi160 ceph-mon[33203]: osdmap e396: 8 total, 8 up, 8 in 2024-09-21T10:01:53.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:52 smithi160 ceph-mon[33203]: osdmap e397: 8 total, 8 up, 8 in 2024-09-21T10:01:53.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:52 smithi160 ceph-mon[33203]: pgmap v87: 105 pgs: 105 active+clean; 65 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 239 KiB/s rd, 18 KiB/s wr, 327 op/s 2024-09-21T10:01:53.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:52 smithi070 ceph-mon[107385]: osdmap e397: 8 total, 8 up, 8 in 2024-09-21T10:01:53.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:52 smithi070 ceph-mon[107385]: pgmap v87: 105 pgs: 105 active+clean; 65 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 239 KiB/s rd, 18 KiB/s wr, 327 op/s 2024-09-21T10:01:53.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:52 smithi070 ceph-mon[112139]: osdmap e397: 8 total, 8 up, 8 in 2024-09-21T10:01:53.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:52 smithi070 ceph-mon[112139]: pgmap v87: 105 pgs: 105 active+clean; 65 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 239 KiB/s rd, 18 KiB/s wr, 327 op/s 2024-09-21T10:01:53.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:01:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:01:53] "GET /metrics HTTP/1.1" 200 36277 "" "Prometheus/2.51.0" 2024-09-21T10:01:53.847 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestDeepCopy.CloneHideParent (6937 ms) 2024-09-21T10:01:53.847 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.Clone 2024-09-21T10:01:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:53 smithi160 ceph-mon[33203]: osdmap e398: 8 total, 8 up, 8 in 2024-09-21T10:01:54.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:53 smithi070 ceph-mon[107385]: osdmap e398: 8 total, 8 up, 8 in 2024-09-21T10:01:54.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:53 smithi070 ceph-mon[112139]: osdmap e398: 8 total, 8 up, 8 in 2024-09-21T10:01:55.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:54 smithi160 ceph-mon[33203]: osdmap e399: 8 total, 8 up, 8 in 2024-09-21T10:01:55.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:54 smithi160 ceph-mon[33203]: pgmap v90: 105 pgs: 105 active+clean; 65 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 239 KiB/s rd, 18 KiB/s wr, 327 op/s 2024-09-21T10:01:55.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:54 smithi070 ceph-mon[107385]: osdmap e399: 8 total, 8 up, 8 in 2024-09-21T10:01:55.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:54 smithi070 ceph-mon[107385]: pgmap v90: 105 pgs: 105 active+clean; 65 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 239 KiB/s rd, 18 KiB/s wr, 327 op/s 2024-09-21T10:01:55.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:54 smithi070 ceph-mon[112139]: osdmap e399: 8 total, 8 up, 8 in 2024-09-21T10:01:55.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:54 smithi070 ceph-mon[112139]: pgmap v90: 105 pgs: 105 active+clean; 65 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 239 KiB/s rd, 18 KiB/s wr, 327 op/s 2024-09-21T10:01:56.083 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestDeepCopy.Clone (2235 ms) 2024-09-21T10:01:56.083 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.CloneFlatten 2024-09-21T10:01:56.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:55 smithi160 ceph-mon[33203]: osdmap e400: 8 total, 8 up, 8 in 2024-09-21T10:01:56.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:55 smithi070 ceph-mon[107385]: osdmap e400: 8 total, 8 up, 8 in 2024-09-21T10:01:56.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:55 smithi070 ceph-mon[112139]: osdmap e400: 8 total, 8 up, 8 in 2024-09-21T10:01:57.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:56 smithi160 ceph-mon[33203]: osdmap e401: 8 total, 8 up, 8 in 2024-09-21T10:01:57.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:56 smithi160 ceph-mon[33203]: pgmap v93: 105 pgs: 105 active+clean; 85 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 2.1 MiB/s rd, 5.0 MiB/s wr, 170 op/s 2024-09-21T10:01:57.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:56 smithi070 ceph-mon[107385]: osdmap e401: 8 total, 8 up, 8 in 2024-09-21T10:01:57.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:56 smithi070 ceph-mon[107385]: pgmap v93: 105 pgs: 105 active+clean; 85 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 2.1 MiB/s rd, 5.0 MiB/s wr, 170 op/s 2024-09-21T10:01:57.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:56 smithi070 ceph-mon[112139]: osdmap e401: 8 total, 8 up, 8 in 2024-09-21T10:01:57.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:56 smithi070 ceph-mon[112139]: pgmap v93: 105 pgs: 105 active+clean; 85 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 2.1 MiB/s rd, 5.0 MiB/s wr, 170 op/s 2024-09-21T10:01:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:57 smithi160 ceph-mon[33203]: osdmap e402: 8 total, 8 up, 8 in 2024-09-21T10:01:58.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:57 smithi070 ceph-mon[107385]: osdmap e402: 8 total, 8 up, 8 in 2024-09-21T10:01:58.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:57 smithi070 ceph-mon[112139]: osdmap e402: 8 total, 8 up, 8 in 2024-09-21T10:01:58.227 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestDeepCopy.CloneFlatten (2145 ms) 2024-09-21T10:01:58.227 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.Stress 2024-09-21T10:01:58.248 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 13641092~1476943 2024-09-21T10:01:58.258 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 21009889~5514936 2024-09-21T10:01:58.258 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 6975050~152980 2024-09-21T10:01:58.260 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 73385156~4736187 2024-09-21T10:01:58.260 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 16238489~5216712 2024-09-21T10:01:58.272 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 11990561~2118693 2024-09-21T10:01:58.274 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 41199115~1374821 2024-09-21T10:01:58.278 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 35329712~3407544 2024-09-21T10:01:58.279 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap0 2024-09-21T10:01:58.786 INFO:tasks.workunit.client.0.smithi070.stdout:clone image242 -> image243 2024-09-21T10:01:58.832 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 86066916 2024-09-21T10:01:58.837 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 10837193~2172085 2024-09-21T10:01:58.849 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 82247073~3819843 2024-09-21T10:01:58.852 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 18226296~2614139 2024-09-21T10:01:58.873 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 46075004~3565284 2024-09-21T10:01:58.881 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 39423507~3639285 2024-09-21T10:01:58.892 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 74104171~3266921 2024-09-21T10:01:58.895 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 57512109~3653818 2024-09-21T10:01:58.905 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 51340896~5125843 2024-09-21T10:01:58.907 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap1 2024-09-21T10:01:59.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:58 smithi160 ceph-mon[33203]: osdmap e403: 8 total, 8 up, 8 in 2024-09-21T10:01:59.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:58 smithi160 ceph-mon[33203]: pgmap v96: 105 pgs: 105 active+clean; 93 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 7.0 MiB/s wr, 260 op/s 2024-09-21T10:01:59.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:58 smithi070 ceph-mon[107385]: osdmap e403: 8 total, 8 up, 8 in 2024-09-21T10:01:59.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:58 smithi070 ceph-mon[107385]: pgmap v96: 105 pgs: 105 active+clean; 93 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 7.0 MiB/s wr, 260 op/s 2024-09-21T10:01:59.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:58 smithi070 ceph-mon[112139]: osdmap e403: 8 total, 8 up, 8 in 2024-09-21T10:01:59.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:58 smithi070 ceph-mon[112139]: pgmap v96: 105 pgs: 105 active+clean; 93 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 7.0 MiB/s wr, 260 op/s 2024-09-21T10:01:59.792 INFO:tasks.workunit.client.0.smithi070.stdout:clone image243 -> image244 2024-09-21T10:01:59.859 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 81832845~4234071 2024-09-21T10:01:59.873 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 32553368~540390 2024-09-21T10:01:59.876 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 28973431~290155 2024-09-21T10:01:59.880 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 60112586~5070166 2024-09-21T10:01:59.890 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 55503414~7549887 2024-09-21T10:01:59.905 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 70492842~630263 2024-09-21T10:01:59.908 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 32502265~637674 2024-09-21T10:01:59.911 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 81801089~4148779 2024-09-21T10:01:59.912 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap2 2024-09-21T10:02:00.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:01:59 smithi160 ceph-mon[33203]: osdmap e404: 8 total, 8 up, 8 in 2024-09-21T10:02:00.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:01:59 smithi070 ceph-mon[107385]: osdmap e404: 8 total, 8 up, 8 in 2024-09-21T10:02:00.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:01:59 smithi070 ceph-mon[112139]: osdmap e404: 8 total, 8 up, 8 in 2024-09-21T10:02:00.802 INFO:tasks.workunit.client.0.smithi070.stdout:clone image244 -> image245 2024-09-21T10:02:00.879 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 69093783~4060497 2024-09-21T10:02:00.898 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 40250006~5356582 2024-09-21T10:02:00.912 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 80935931~804330 2024-09-21T10:02:00.917 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 21235365~7310689 2024-09-21T10:02:00.923 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 36598149~365692 2024-09-21T10:02:00.927 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 26322450~3956033 2024-09-21T10:02:00.933 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 13728404~7760211 2024-09-21T10:02:00.964 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 75581885~7209540 2024-09-21T10:02:00.967 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap3 2024-09-21T10:02:01.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:00 smithi160 ceph-mon[33203]: osdmap e405: 8 total, 8 up, 8 in 2024-09-21T10:02:01.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:00 smithi160 ceph-mon[33203]: pgmap v99: 105 pgs: 105 active+clean; 93 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 2.0 MiB/s wr, 89 op/s 2024-09-21T10:02:01.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:00 smithi070 ceph-mon[107385]: osdmap e405: 8 total, 8 up, 8 in 2024-09-21T10:02:01.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:00 smithi070 ceph-mon[107385]: pgmap v99: 105 pgs: 105 active+clean; 93 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 2.0 MiB/s wr, 89 op/s 2024-09-21T10:02:01.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:00 smithi070 ceph-mon[112139]: osdmap e405: 8 total, 8 up, 8 in 2024-09-21T10:02:01.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:00 smithi070 ceph-mon[112139]: pgmap v99: 105 pgs: 105 active+clean; 93 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 2.0 MiB/s wr, 89 op/s 2024-09-21T10:02:01.803 INFO:tasks.workunit.client.0.smithi070.stdout:clone image245 -> image246 2024-09-21T10:02:01.887 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 106563732 2024-09-21T10:02:02.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:01 smithi160 ceph-mon[33203]: osdmap e406: 8 total, 8 up, 8 in 2024-09-21T10:02:02.114 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestDeepCopy.Stress (3887 ms) 2024-09-21T10:02:02.114 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.NoSnaps_LargerDstObjSize 2024-09-21T10:02:02.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:01 smithi070 ceph-mon[107385]: osdmap e406: 8 total, 8 up, 8 in 2024-09-21T10:02:02.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:01 smithi070 ceph-mon[112139]: osdmap e406: 8 total, 8 up, 8 in 2024-09-21T10:02:02.345 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestDeepCopy.NoSnaps_LargerDstObjSize (231 ms) 2024-09-21T10:02:02.345 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.Snaps_LargerDstObjSize 2024-09-21T10:02:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:02 smithi160 ceph-mon[33203]: osdmap e407: 8 total, 8 up, 8 in 2024-09-21T10:02:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:02 smithi160 ceph-mon[33203]: pgmap v102: 105 pgs: 105 active+clean; 298 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 48 MiB/s rd, 40 MiB/s wr, 655 op/s 2024-09-21T10:02:03.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:02 smithi070 ceph-mon[107385]: osdmap e407: 8 total, 8 up, 8 in 2024-09-21T10:02:03.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:02 smithi070 ceph-mon[107385]: pgmap v102: 105 pgs: 105 active+clean; 298 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 48 MiB/s rd, 40 MiB/s wr, 655 op/s 2024-09-21T10:02:03.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:02 smithi070 ceph-mon[112139]: osdmap e407: 8 total, 8 up, 8 in 2024-09-21T10:02:03.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:02 smithi070 ceph-mon[112139]: pgmap v102: 105 pgs: 105 active+clean; 298 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 48 MiB/s rd, 40 MiB/s wr, 655 op/s 2024-09-21T10:02:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:02:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:02:03] "GET /metrics HTTP/1.1" 200 36277 "" "Prometheus/2.51.0" 2024-09-21T10:02:04.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:03 smithi160 ceph-mon[33203]: osdmap e408: 8 total, 8 up, 8 in 2024-09-21T10:02:04.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:03 smithi070 ceph-mon[112139]: osdmap e408: 8 total, 8 up, 8 in 2024-09-21T10:02:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:03 smithi070 ceph-mon[107385]: osdmap e408: 8 total, 8 up, 8 in 2024-09-21T10:02:05.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:04 smithi160 ceph-mon[33203]: osdmap e409: 8 total, 8 up, 8 in 2024-09-21T10:02:05.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:04 smithi160 ceph-mon[33203]: pgmap v105: 105 pgs: 105 active+clean; 298 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 48 MiB/s rd, 40 MiB/s wr, 655 op/s 2024-09-21T10:02:05.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:04 smithi070 ceph-mon[107385]: osdmap e409: 8 total, 8 up, 8 in 2024-09-21T10:02:05.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:04 smithi070 ceph-mon[107385]: pgmap v105: 105 pgs: 105 active+clean; 298 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 48 MiB/s rd, 40 MiB/s wr, 655 op/s 2024-09-21T10:02:05.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:04 smithi070 ceph-mon[112139]: osdmap e409: 8 total, 8 up, 8 in 2024-09-21T10:02:05.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:04 smithi070 ceph-mon[112139]: pgmap v105: 105 pgs: 105 active+clean; 298 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 48 MiB/s rd, 40 MiB/s wr, 655 op/s 2024-09-21T10:02:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:05 smithi160 ceph-mon[33203]: osdmap e410: 8 total, 8 up, 8 in 2024-09-21T10:02:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:05 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:02:06.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:05 smithi070 ceph-mon[107385]: osdmap e410: 8 total, 8 up, 8 in 2024-09-21T10:02:06.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:02:06.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:05 smithi070 ceph-mon[112139]: osdmap e410: 8 total, 8 up, 8 in 2024-09-21T10:02:06.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:02:06.142 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestDeepCopy.Snaps_LargerDstObjSize (3797 ms) 2024-09-21T10:02:06.142 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.Clone_LargerDstObjSize 2024-09-21T10:02:07.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:06 smithi160 ceph-mon[33203]: osdmap e411: 8 total, 8 up, 8 in 2024-09-21T10:02:07.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:06 smithi160 ceph-mon[33203]: pgmap v108: 105 pgs: 105 active+clean; 347 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 13 MiB/s wr, 359 op/s 2024-09-21T10:02:07.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:06 smithi070 ceph-mon[107385]: osdmap e411: 8 total, 8 up, 8 in 2024-09-21T10:02:07.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:06 smithi070 ceph-mon[107385]: pgmap v108: 105 pgs: 105 active+clean; 347 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 13 MiB/s wr, 359 op/s 2024-09-21T10:02:07.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:06 smithi070 ceph-mon[112139]: osdmap e411: 8 total, 8 up, 8 in 2024-09-21T10:02:07.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:06 smithi070 ceph-mon[112139]: pgmap v108: 105 pgs: 105 active+clean; 347 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 13 MiB/s wr, 359 op/s 2024-09-21T10:02:08.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:07 smithi160 ceph-mon[33203]: osdmap e412: 8 total, 8 up, 8 in 2024-09-21T10:02:08.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:07 smithi070 ceph-mon[107385]: osdmap e412: 8 total, 8 up, 8 in 2024-09-21T10:02:08.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:07 smithi070 ceph-mon[112139]: osdmap e412: 8 total, 8 up, 8 in 2024-09-21T10:02:08.242 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestDeepCopy.Clone_LargerDstObjSize (2100 ms) 2024-09-21T10:02:08.242 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.CloneFlatten_LargerDstObjSize 2024-09-21T10:02:09.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:08 smithi160 ceph-mon[33203]: osdmap e413: 8 total, 8 up, 8 in 2024-09-21T10:02:09.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:08 smithi160 ceph-mon[33203]: pgmap v111: 105 pgs: 105 active+clean; 365 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 31 MiB/s rd, 18 MiB/s wr, 459 op/s 2024-09-21T10:02:09.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:08 smithi070 ceph-mon[112139]: osdmap e413: 8 total, 8 up, 8 in 2024-09-21T10:02:09.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:08 smithi070 ceph-mon[112139]: pgmap v111: 105 pgs: 105 active+clean; 365 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 31 MiB/s rd, 18 MiB/s wr, 459 op/s 2024-09-21T10:02:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:08 smithi070 ceph-mon[107385]: osdmap e413: 8 total, 8 up, 8 in 2024-09-21T10:02:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:08 smithi070 ceph-mon[107385]: pgmap v111: 105 pgs: 105 active+clean; 365 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 31 MiB/s rd, 18 MiB/s wr, 459 op/s 2024-09-21T10:02:10.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:09 smithi160 ceph-mon[33203]: osdmap e414: 8 total, 8 up, 8 in 2024-09-21T10:02:10.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:09 smithi070 ceph-mon[112139]: osdmap e414: 8 total, 8 up, 8 in 2024-09-21T10:02:10.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:09 smithi070 ceph-mon[107385]: osdmap e414: 8 total, 8 up, 8 in 2024-09-21T10:02:10.369 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestDeepCopy.CloneFlatten_LargerDstObjSize (2122 ms) 2024-09-21T10:02:10.369 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.Stress_LargerDstObjSize 2024-09-21T10:02:10.388 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 58007836~7023845 2024-09-21T10:02:10.415 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 74770383~3576865 2024-09-21T10:02:10.415 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 7725644~2089929 2024-09-21T10:02:10.423 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 5540709~3730429 2024-09-21T10:02:10.424 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 58122574~6099954 2024-09-21T10:02:10.446 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 36622360~4168164 2024-09-21T10:02:10.447 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 21749711~7102101 2024-09-21T10:02:10.463 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 73133046~3302823 2024-09-21T10:02:10.463 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap0 2024-09-21T10:02:10.849 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 85569579 2024-09-21T10:02:10.854 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 22634265~4998069 2024-09-21T10:02:10.856 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 29965389~1055102 2024-09-21T10:02:10.856 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 26075145~3138917 2024-09-21T10:02:10.869 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 17285947~8296406 2024-09-21T10:02:10.870 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 10071643~5021431 2024-09-21T10:02:10.870 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 66841754~907321 2024-09-21T10:02:10.874 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 58799610~7110042 2024-09-21T10:02:10.874 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 29865279~3425303 2024-09-21T10:02:10.910 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap1 2024-09-21T10:02:11.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:10 smithi160 ceph-mon[33203]: osdmap e415: 8 total, 8 up, 8 in 2024-09-21T10:02:11.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:10 smithi160 ceph-mon[33203]: pgmap v114: 105 pgs: 105 active+clean; 365 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 5.0 MiB/s wr, 99 op/s 2024-09-21T10:02:11.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:10 smithi070 ceph-mon[112139]: osdmap e415: 8 total, 8 up, 8 in 2024-09-21T10:02:11.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:10 smithi070 ceph-mon[112139]: pgmap v114: 105 pgs: 105 active+clean; 365 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 5.0 MiB/s wr, 99 op/s 2024-09-21T10:02:11.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:10 smithi070 ceph-mon[107385]: osdmap e415: 8 total, 8 up, 8 in 2024-09-21T10:02:11.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:10 smithi070 ceph-mon[107385]: pgmap v114: 105 pgs: 105 active+clean; 365 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 5.0 MiB/s wr, 99 op/s 2024-09-21T10:02:11.866 INFO:tasks.workunit.client.0.smithi070.stdout:clone image265 -> image266 2024-09-21T10:02:11.917 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 155942282 2024-09-21T10:02:11.926 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 84788899~7825540 2024-09-21T10:02:11.959 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 82972973~1127093 2024-09-21T10:02:11.962 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 37694556~3541613 2024-09-21T10:02:11.978 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 108366000~188984 2024-09-21T10:02:11.979 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 122112802~5864769 2024-09-21T10:02:11.997 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 122405778~2124344 2024-09-21T10:02:11.999 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 140012257~6229089 2024-09-21T10:02:12.016 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 137716042~214763 2024-09-21T10:02:12.016 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap2 2024-09-21T10:02:12.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:11 smithi160 ceph-mon[33203]: osdmap e416: 8 total, 8 up, 8 in 2024-09-21T10:02:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:11 smithi070 ceph-mon[107385]: osdmap e416: 8 total, 8 up, 8 in 2024-09-21T10:02:12.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:11 smithi070 ceph-mon[112139]: osdmap e416: 8 total, 8 up, 8 in 2024-09-21T10:02:12.877 INFO:tasks.workunit.client.0.smithi070.stdout:clone image266 -> image267 2024-09-21T10:02:12.939 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 119806221~2547845 2024-09-21T10:02:12.956 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 97702200~6203290 2024-09-21T10:02:12.960 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 456590~4349965 2024-09-21T10:02:12.975 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 65200325~1355477 2024-09-21T10:02:13.000 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 147332631~1088424 2024-09-21T10:02:13.006 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 99995855~6829698 2024-09-21T10:02:13.008 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 23902115~1074598 2024-09-21T10:02:13.017 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 14460703~4324449 2024-09-21T10:02:13.031 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap3 2024-09-21T10:02:13.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:12 smithi070 ceph-mon[107385]: osdmap e417: 8 total, 8 up, 8 in 2024-09-21T10:02:13.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:12 smithi070 ceph-mon[107385]: pgmap v117: 105 pgs: 105 active+clean; 554 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 66 MiB/s rd, 48 MiB/s wr, 565 op/s 2024-09-21T10:02:13.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:12 smithi070 ceph-mon[112139]: osdmap e417: 8 total, 8 up, 8 in 2024-09-21T10:02:13.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:12 smithi070 ceph-mon[112139]: pgmap v117: 105 pgs: 105 active+clean; 554 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 66 MiB/s rd, 48 MiB/s wr, 565 op/s 2024-09-21T10:02:13.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:12 smithi160 ceph-mon[33203]: osdmap e417: 8 total, 8 up, 8 in 2024-09-21T10:02:13.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:12 smithi160 ceph-mon[33203]: pgmap v117: 105 pgs: 105 active+clean; 554 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 66 MiB/s rd, 48 MiB/s wr, 565 op/s 2024-09-21T10:02:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:02:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:02:13] "GET /metrics HTTP/1.1" 200 36284 "" "Prometheus/2.51.0" 2024-09-21T10:02:14.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:13 smithi070 ceph-mon[107385]: osdmap e418: 8 total, 8 up, 8 in 2024-09-21T10:02:14.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:13 smithi070 ceph-mon[112139]: osdmap e418: 8 total, 8 up, 8 in 2024-09-21T10:02:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:13 smithi160 ceph-mon[33203]: osdmap e418: 8 total, 8 up, 8 in 2024-09-21T10:02:15.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:14 smithi070 ceph-mon[107385]: osdmap e419: 8 total, 8 up, 8 in 2024-09-21T10:02:15.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:14 smithi070 ceph-mon[107385]: pgmap v120: 105 pgs: 105 active+clean; 554 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 66 MiB/s rd, 48 MiB/s wr, 565 op/s 2024-09-21T10:02:15.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:14 smithi070 ceph-mon[112139]: osdmap e419: 8 total, 8 up, 8 in 2024-09-21T10:02:15.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:14 smithi070 ceph-mon[112139]: pgmap v120: 105 pgs: 105 active+clean; 554 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 66 MiB/s rd, 48 MiB/s wr, 565 op/s 2024-09-21T10:02:15.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:14 smithi160 ceph-mon[33203]: osdmap e419: 8 total, 8 up, 8 in 2024-09-21T10:02:15.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:14 smithi160 ceph-mon[33203]: pgmap v120: 105 pgs: 105 active+clean; 554 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 66 MiB/s rd, 48 MiB/s wr, 565 op/s 2024-09-21T10:02:15.499 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestDeepCopy.Stress_LargerDstObjSize (5135 ms) 2024-09-21T10:02:15.500 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.NoSnaps_SmallerDstObjSize 2024-09-21T10:02:15.712 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestDeepCopy.NoSnaps_SmallerDstObjSize (213 ms) 2024-09-21T10:02:15.712 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.Snaps_SmallerDstObjSize 2024-09-21T10:02:16.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:15 smithi070 ceph-mon[107385]: osdmap e420: 8 total, 8 up, 8 in 2024-09-21T10:02:16.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:15 smithi070 ceph-mon[112139]: osdmap e420: 8 total, 8 up, 8 in 2024-09-21T10:02:16.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:15 smithi160 ceph-mon[33203]: osdmap e420: 8 total, 8 up, 8 in 2024-09-21T10:02:17.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:16 smithi070 ceph-mon[107385]: osdmap e421: 8 total, 8 up, 8 in 2024-09-21T10:02:17.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:16 smithi070 ceph-mon[107385]: pgmap v123: 105 pgs: 105 active+clean; 706 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 63 MiB/s rd, 30 MiB/s wr, 471 op/s 2024-09-21T10:02:17.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:16 smithi070 ceph-mon[112139]: osdmap e421: 8 total, 8 up, 8 in 2024-09-21T10:02:17.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:16 smithi070 ceph-mon[112139]: pgmap v123: 105 pgs: 105 active+clean; 706 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 63 MiB/s rd, 30 MiB/s wr, 471 op/s 2024-09-21T10:02:17.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:16 smithi160 ceph-mon[33203]: osdmap e421: 8 total, 8 up, 8 in 2024-09-21T10:02:17.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:16 smithi160 ceph-mon[33203]: pgmap v123: 105 pgs: 105 active+clean; 706 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 63 MiB/s rd, 30 MiB/s wr, 471 op/s 2024-09-21T10:02:18.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:17 smithi070 ceph-mon[107385]: osdmap e422: 8 total, 8 up, 8 in 2024-09-21T10:02:18.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:17 smithi070 ceph-mon[112139]: osdmap e422: 8 total, 8 up, 8 in 2024-09-21T10:02:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:17 smithi160 ceph-mon[33203]: osdmap e422: 8 total, 8 up, 8 in 2024-09-21T10:02:19.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:18 smithi160 ceph-mon[33203]: osdmap e423: 8 total, 8 up, 8 in 2024-09-21T10:02:19.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:18 smithi160 ceph-mon[33203]: pgmap v126: 105 pgs: 105 active+clean; 765 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 87 MiB/s rd, 41 MiB/s wr, 515 op/s 2024-09-21T10:02:19.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:18 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:02:19.158 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestDeepCopy.Snaps_SmallerDstObjSize (3446 ms) 2024-09-21T10:02:19.158 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.Clone_SmallerDstObjSize 2024-09-21T10:02:19.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:18 smithi070 ceph-mon[107385]: osdmap e423: 8 total, 8 up, 8 in 2024-09-21T10:02:19.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:18 smithi070 ceph-mon[107385]: pgmap v126: 105 pgs: 105 active+clean; 765 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 87 MiB/s rd, 41 MiB/s wr, 515 op/s 2024-09-21T10:02:19.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:18 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:02:19.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:18 smithi070 ceph-mon[112139]: osdmap e423: 8 total, 8 up, 8 in 2024-09-21T10:02:19.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:18 smithi070 ceph-mon[112139]: pgmap v126: 105 pgs: 105 active+clean; 765 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 87 MiB/s rd, 41 MiB/s wr, 515 op/s 2024-09-21T10:02:19.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:18 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:02:20.249 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:19 smithi070 ceph-mon[107385]: osdmap e424: 8 total, 8 up, 8 in 2024-09-21T10:02:20.250 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:19 smithi070 ceph-mon[112139]: osdmap e424: 8 total, 8 up, 8 in 2024-09-21T10:02:20.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:19 smithi160 ceph-mon[33203]: osdmap e424: 8 total, 8 up, 8 in 2024-09-21T10:02:21.179 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:20 smithi070 ceph-mon[107385]: osdmap e425: 8 total, 8 up, 8 in 2024-09-21T10:02:21.179 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:20 smithi070 ceph-mon[107385]: pgmap v129: 105 pgs: 105 active+clean; 765 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 24 MiB/s rd, 11 MiB/s wr, 44 op/s 2024-09-21T10:02:21.179 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:02:21.179 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:20 smithi070 ceph-mon[112139]: osdmap e425: 8 total, 8 up, 8 in 2024-09-21T10:02:21.179 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:20 smithi070 ceph-mon[112139]: pgmap v129: 105 pgs: 105 active+clean; 765 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 24 MiB/s rd, 11 MiB/s wr, 44 op/s 2024-09-21T10:02:21.180 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:02:21.205 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestDeepCopy.Clone_SmallerDstObjSize (2047 ms) 2024-09-21T10:02:21.206 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.CloneFlatten_SmallerDstObjSize 2024-09-21T10:02:21.272 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:20 smithi160 ceph-mon[33203]: osdmap e425: 8 total, 8 up, 8 in 2024-09-21T10:02:21.272 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:20 smithi160 ceph-mon[33203]: pgmap v129: 105 pgs: 105 active+clean; 765 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 24 MiB/s rd, 11 MiB/s wr, 44 op/s 2024-09-21T10:02:21.272 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:20 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:02:22.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:21 smithi070 ceph-mon[107385]: osdmap e426: 8 total, 8 up, 8 in 2024-09-21T10:02:22.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:21 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:22.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:21 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:22.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:21 smithi070 ceph-mon[112139]: osdmap e426: 8 total, 8 up, 8 in 2024-09-21T10:02:22.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:21 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:22.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:21 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:22.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:21 smithi160 ceph-mon[33203]: osdmap e426: 8 total, 8 up, 8 in 2024-09-21T10:02:22.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:21 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:22.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:21 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:23.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:22 smithi070 ceph-mon[112139]: osdmap e427: 8 total, 8 up, 8 in 2024-09-21T10:02:23.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:22 smithi070 ceph-mon[112139]: pgmap v132: 105 pgs: 105 active+clean; 801 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 25 MiB/s rd, 9.0 MiB/s wr, 236 op/s 2024-09-21T10:02:23.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:22 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:23.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:22 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:23.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:22 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:02:23.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:22 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:02:23.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:22 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:23.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:22 smithi070 ceph-mon[107385]: osdmap e427: 8 total, 8 up, 8 in 2024-09-21T10:02:23.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:22 smithi070 ceph-mon[107385]: pgmap v132: 105 pgs: 105 active+clean; 801 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 25 MiB/s rd, 9.0 MiB/s wr, 236 op/s 2024-09-21T10:02:23.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:22 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:23.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:22 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:23.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:22 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:02:23.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:22 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:02:23.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:22 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:23.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:22 smithi160 ceph-mon[33203]: osdmap e427: 8 total, 8 up, 8 in 2024-09-21T10:02:23.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:22 smithi160 ceph-mon[33203]: pgmap v132: 105 pgs: 105 active+clean; 801 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 25 MiB/s rd, 9.0 MiB/s wr, 236 op/s 2024-09-21T10:02:23.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:22 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:23.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:22 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:23.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:22 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:02:23.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:22 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:02:23.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:22 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:23.381 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestDeepCopy.CloneFlatten_SmallerDstObjSize (2176 ms) 2024-09-21T10:02:23.381 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.Stress_SmallerDstObjSize 2024-09-21T10:02:23.405 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 44846164~7907047 2024-09-21T10:02:23.423 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 32232531~57627 2024-09-21T10:02:23.423 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 56831604~743605 2024-09-21T10:02:23.427 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 3383651~7281470 2024-09-21T10:02:23.428 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 58794977~7692595 2024-09-21T10:02:23.446 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 44689671~5931496 2024-09-21T10:02:23.450 INFO:tasks.workunit.client.0.smithi070.stdout:write: A 8565539~6278011 2024-09-21T10:02:23.469 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 20718347~7096596 2024-09-21T10:02:23.469 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap0 2024-09-21T10:02:23.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:02:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:02:23] "GET /metrics HTTP/1.1" 200 36285 "" "Prometheus/2.51.0" 2024-09-21T10:02:23.919 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 164413107 2024-09-21T10:02:23.924 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 28566369~882231 2024-09-21T10:02:23.927 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 38870314~491711 2024-09-21T10:02:23.928 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 156452531~7960576 2024-09-21T10:02:23.928 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 83288715~4158520 2024-09-21T10:02:23.929 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 64957152~4293614 2024-09-21T10:02:23.929 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 22381215~2727192 2024-09-21T10:02:23.934 INFO:tasks.workunit.client.0.smithi070.stdout:write: B 69416225~5719237 2024-09-21T10:02:23.934 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 113449241~2284524 2024-09-21T10:02:23.983 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap1 2024-09-21T10:02:24.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:23 smithi160 ceph-mon[33203]: mgrmap e40: y(active, since 93s), standbys: x 2024-09-21T10:02:24.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:23 smithi160 ceph-mon[33203]: osdmap e428: 8 total, 8 up, 8 in 2024-09-21T10:02:24.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:23 smithi070 ceph-mon[112139]: mgrmap e40: y(active, since 93s), standbys: x 2024-09-21T10:02:24.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:23 smithi070 ceph-mon[112139]: osdmap e428: 8 total, 8 up, 8 in 2024-09-21T10:02:24.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:23 smithi070 ceph-mon[107385]: mgrmap e40: y(active, since 93s), standbys: x 2024-09-21T10:02:24.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:23 smithi070 ceph-mon[107385]: osdmap e428: 8 total, 8 up, 8 in 2024-09-21T10:02:24.935 INFO:tasks.workunit.client.0.smithi070.stdout:clone image286 -> image287 2024-09-21T10:02:24.992 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 109019148 2024-09-21T10:02:25.000 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 50522848~1368965 2024-09-21T10:02:25.014 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 32761089~3364920 2024-09-21T10:02:25.017 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 18195319~2446491 2024-09-21T10:02:25.026 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 51682663~7477392 2024-09-21T10:02:25.047 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 42564232~5195455 2024-09-21T10:02:25.068 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 52462854~4407334 2024-09-21T10:02:25.069 INFO:tasks.workunit.client.0.smithi070.stdout:write: C 56858756~1503423 2024-09-21T10:02:25.074 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 79514576~2328113 2024-09-21T10:02:25.077 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap2 2024-09-21T10:02:25.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:24 smithi160 ceph-mon[33203]: osdmap e429: 8 total, 8 up, 8 in 2024-09-21T10:02:25.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:24 smithi160 ceph-mon[33203]: pgmap v135: 105 pgs: 105 active+clean; 801 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 25 MiB/s rd, 9.0 MiB/s wr, 236 op/s 2024-09-21T10:02:25.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:24 smithi070 ceph-mon[107385]: osdmap e429: 8 total, 8 up, 8 in 2024-09-21T10:02:25.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:24 smithi070 ceph-mon[107385]: pgmap v135: 105 pgs: 105 active+clean; 801 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 25 MiB/s rd, 9.0 MiB/s wr, 236 op/s 2024-09-21T10:02:25.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:24 smithi070 ceph-mon[112139]: osdmap e429: 8 total, 8 up, 8 in 2024-09-21T10:02:25.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:24 smithi070 ceph-mon[112139]: pgmap v135: 105 pgs: 105 active+clean; 801 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 25 MiB/s rd, 9.0 MiB/s wr, 236 op/s 2024-09-21T10:02:25.941 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 150487157 2024-09-21T10:02:25.950 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 129893974~6728337 2024-09-21T10:02:25.953 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 144688831~5430963 2024-09-21T10:02:25.955 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 67067506~5276177 2024-09-21T10:02:25.955 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 45177430~6697238 2024-09-21T10:02:25.960 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 8744847~7158930 2024-09-21T10:02:25.963 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 139832758~7956575 2024-09-21T10:02:25.975 INFO:tasks.workunit.client.0.smithi070.stdout:write: D 54169558~6051516 2024-09-21T10:02:25.976 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 57833953~5425442 2024-09-21T10:02:26.039 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap3 2024-09-21T10:02:26.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:25 smithi160 ceph-mon[33203]: osdmap e430: 8 total, 8 up, 8 in 2024-09-21T10:02:26.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:25 smithi070 ceph-mon[107385]: osdmap e430: 8 total, 8 up, 8 in 2024-09-21T10:02:26.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:25 smithi070 ceph-mon[112139]: osdmap e430: 8 total, 8 up, 8 in 2024-09-21T10:02:26.955 INFO:tasks.workunit.client.0.smithi070.stdout:clone image287 -> image288 2024-09-21T10:02:27.022 INFO:tasks.workunit.client.0.smithi070.stdout:resize: 211979977 2024-09-21T10:02:27.278 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestDeepCopy.Stress_SmallerDstObjSize (3897 ms) 2024-09-21T10:02:27.278 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.NoSnaps_StrippingLargerDstObjSize 2024-09-21T10:02:27.302 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/test_DeepCopy.cc:621: Skipped 2024-09-21T10:02:27.302 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T10:02:27.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:26 smithi160 ceph-mon[33203]: osdmap e431: 8 total, 8 up, 8 in 2024-09-21T10:02:27.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:26 smithi160 ceph-mon[33203]: pgmap v138: 105 pgs: 105 active+clean; 977 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 45 MiB/s rd, 41 MiB/s wr, 506 op/s 2024-09-21T10:02:27.376 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestDeepCopy.NoSnaps_StrippingLargerDstObjSize (98 ms) 2024-09-21T10:02:27.376 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.Snaps_StrippingLargerDstObjSize 2024-09-21T10:02:27.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:26 smithi070 ceph-mon[107385]: osdmap e431: 8 total, 8 up, 8 in 2024-09-21T10:02:27.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:26 smithi070 ceph-mon[107385]: pgmap v138: 105 pgs: 105 active+clean; 977 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 45 MiB/s rd, 41 MiB/s wr, 506 op/s 2024-09-21T10:02:27.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:26 smithi070 ceph-mon[112139]: osdmap e431: 8 total, 8 up, 8 in 2024-09-21T10:02:27.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:26 smithi070 ceph-mon[112139]: pgmap v138: 105 pgs: 105 active+clean; 977 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 45 MiB/s rd, 41 MiB/s wr, 506 op/s 2024-09-21T10:02:27.397 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/test_DeepCopy.cc:635: Skipped 2024-09-21T10:02:27.397 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T10:02:27.468 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestDeepCopy.Snaps_StrippingLargerDstObjSize (92 ms) 2024-09-21T10:02:27.468 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.Clone_StrippingLargerDstObjSize 2024-09-21T10:02:27.489 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/test_DeepCopy.cc:649: Skipped 2024-09-21T10:02:27.489 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T10:02:27.558 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestDeepCopy.Clone_StrippingLargerDstObjSize (90 ms) 2024-09-21T10:02:27.558 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.CloneFlatten_StrippingLargerDstObjSize 2024-09-21T10:02:27.579 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/test_DeepCopy.cc:663: Skipped 2024-09-21T10:02:27.579 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T10:02:27.647 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestDeepCopy.CloneFlatten_StrippingLargerDstObjSize (89 ms) 2024-09-21T10:02:27.648 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.Stress_StrippingLargerDstObjSize 2024-09-21T10:02:27.670 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/test_DeepCopy.cc:679: Skipped 2024-09-21T10:02:27.670 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T10:02:27.738 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestDeepCopy.Stress_StrippingLargerDstObjSize (91 ms) 2024-09-21T10:02:27.739 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.NoSnaps_StrippingSmallerDstObjSize 2024-09-21T10:02:27.761 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/test_DeepCopy.cc:693: Skipped 2024-09-21T10:02:27.762 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T10:02:27.828 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestDeepCopy.NoSnaps_StrippingSmallerDstObjSize (90 ms) 2024-09-21T10:02:27.829 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.Snaps_StrippingSmallerDstObjSize 2024-09-21T10:02:27.850 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/test_DeepCopy.cc:707: Skipped 2024-09-21T10:02:27.850 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T10:02:27.918 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestDeepCopy.Snaps_StrippingSmallerDstObjSize (90 ms) 2024-09-21T10:02:27.918 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.Clone_StrippingSmallerDstObjSize 2024-09-21T10:02:27.940 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/test_DeepCopy.cc:721: Skipped 2024-09-21T10:02:27.940 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T10:02:28.008 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestDeepCopy.Clone_StrippingSmallerDstObjSize (89 ms) 2024-09-21T10:02:28.008 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.CloneFlatten_StrippingSmallerDstObjSize 2024-09-21T10:02:28.029 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/test_DeepCopy.cc:735: Skipped 2024-09-21T10:02:28.029 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T10:02:28.098 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestDeepCopy.CloneFlatten_StrippingSmallerDstObjSize (91 ms) 2024-09-21T10:02:28.098 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestDeepCopy.Stress_StrippingSmallerDstObjSize 2024-09-21T10:02:28.119 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/test_DeepCopy.cc:751: Skipped 2024-09-21T10:02:28.119 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T10:02:28.190 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestDeepCopy.Stress_StrippingSmallerDstObjSize (92 ms) 2024-09-21T10:02:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:27 smithi160 ceph-mon[33203]: osdmap e432: 8 total, 8 up, 8 in 2024-09-21T10:02:28.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:27 smithi070 ceph-mon[112139]: osdmap e432: 8 total, 8 up, 8 in 2024-09-21T10:02:28.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:27 smithi070 ceph-mon[107385]: osdmap e432: 8 total, 8 up, 8 in 2024-09-21T10:02:28.940 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 31 tests from TestDeepCopy (52553 ms total) 2024-09-21T10:02:28.940 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:02:28.940 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 51 tests from TestMirroring 2024-09-21T10:02:29.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:28 smithi160 ceph-mon[33203]: pgmap v140: 105 pgs: 105 active+clean; 999 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 47 MiB/s rd, 44 MiB/s wr, 635 op/s 2024-09-21T10:02:29.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:28 smithi070 ceph-mon[107385]: pgmap v140: 105 pgs: 105 active+clean; 999 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 47 MiB/s rd, 44 MiB/s wr, 635 op/s 2024-09-21T10:02:29.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:28 smithi070 ceph-mon[112139]: pgmap v140: 105 pgs: 105 active+clean; 999 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 47 MiB/s rd, 44 MiB/s wr, 635 op/s 2024-09-21T10:02:30.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:29 smithi160 ceph-mon[33203]: osdmap e433: 8 total, 8 up, 8 in 2024-09-21T10:02:30.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:29 smithi070 ceph-mon[107385]: osdmap e433: 8 total, 8 up, 8 in 2024-09-21T10:02:30.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:29 smithi070 ceph-mon[112139]: osdmap e433: 8 total, 8 up, 8 in 2024-09-21T10:02:30.945 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeImage 2024-09-21T10:02:31.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:30 smithi160 ceph-mon[33203]: pgmap v142: 73 pgs: 73 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:02:31.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:30 smithi160 ceph-mon[33203]: osdmap e434: 8 total, 8 up, 8 in 2024-09-21T10:02:31.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:30 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-12","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:02:31.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:30 smithi070 ceph-mon[112139]: pgmap v142: 73 pgs: 73 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:02:31.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:30 smithi070 ceph-mon[112139]: osdmap e434: 8 total, 8 up, 8 in 2024-09-21T10:02:31.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:30 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-12","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:02:31.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:30 smithi070 ceph-mon[107385]: pgmap v142: 73 pgs: 73 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:02:31.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:30 smithi070 ceph-mon[107385]: osdmap e434: 8 total, 8 up, 8 in 2024-09-21T10:02:31.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:30 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-12","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:02:32.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:31 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:02:32.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:31 smithi160 ceph-mon[33203]: osdmap e435: 8 total, 8 up, 8 in 2024-09-21T10:02:32.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:31 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:02:32.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:31 smithi070 ceph-mon[107385]: osdmap e435: 8 total, 8 up, 8 in 2024-09-21T10:02:32.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:31 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:02:32.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:31 smithi070 ceph-mon[112139]: osdmap e435: 8 total, 8 up, 8 in 2024-09-21T10:02:33.066 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeImage (2120 ms) 2024-09-21T10:02:33.066 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModePool 2024-09-21T10:02:33.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:32 smithi070 ceph-mon[112139]: pgmap v145: 105 pgs: 32 creating+peering, 73 active+clean; 582 KiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:02:33.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:32 smithi070 ceph-mon[112139]: osdmap e436: 8 total, 8 up, 8 in 2024-09-21T10:02:33.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:32 smithi070 ceph-mon[107385]: pgmap v145: 105 pgs: 32 creating+peering, 73 active+clean; 582 KiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:02:33.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:32 smithi070 ceph-mon[107385]: osdmap e436: 8 total, 8 up, 8 in 2024-09-21T10:02:33.168 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModePool (103 ms) 2024-09-21T10:02:33.168 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeDisabled 2024-09-21T10:02:33.246 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeDisabled (77 ms) 2024-09-21T10:02:33.246 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.DisableImageMirror_In_MirrorModeImage 2024-09-21T10:02:33.337 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.DisableImageMirror_In_MirrorModeImage (92 ms) 2024-09-21T10:02:33.337 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.DisableImageMirror_In_MirrorModeImage_NoObjectMap 2024-09-21T10:02:33.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:32 smithi160 ceph-mon[33203]: pgmap v145: 105 pgs: 32 creating+peering, 73 active+clean; 582 KiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:02:33.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:32 smithi160 ceph-mon[33203]: osdmap e436: 8 total, 8 up, 8 in 2024-09-21T10:02:33.427 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.DisableImageMirror_In_MirrorModeImage_NoObjectMap (90 ms) 2024-09-21T10:02:33.427 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.DisableImageMirror_In_MirrorModePool 2024-09-21T10:02:33.519 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.DisableImageMirror_In_MirrorModePool (92 ms) 2024-09-21T10:02:33.519 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.DisableImageMirror_In_MirrorModeDisabled 2024-09-21T10:02:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:02:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:02:33] "GET /metrics HTTP/1.1" 200 36285 "" "Prometheus/2.51.0" 2024-09-21T10:02:33.628 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.DisableImageMirror_In_MirrorModeDisabled (108 ms) 2024-09-21T10:02:33.628 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.DisableImageMirrorWithPeer 2024-09-21T10:02:33.628 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/test_mirroring.cc:482: Skipped 2024-09-21T10:02:33.628 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T10:02:33.628 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestMirroring.DisableImageMirrorWithPeer (0 ms) 2024-09-21T10:02:33.628 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.DisableJournalingWithPeer 2024-09-21T10:02:33.628 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/test_mirroring.cc:520: Skipped 2024-09-21T10:02:33.628 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T10:02:33.628 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestMirroring.DisableJournalingWithPeer (0 ms) 2024-09-21T10:02:33.628 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeDisabled_WithoutJournaling 2024-09-21T10:02:33.693 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeDisabled_WithoutJournaling (66 ms) 2024-09-21T10:02:33.693 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModePool_WithoutJournaling 2024-09-21T10:02:33.764 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModePool_WithoutJournaling (71 ms) 2024-09-21T10:02:33.765 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeImage_WithoutJournaling 2024-09-21T10:02:34.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:33 smithi160 ceph-mon[33203]: osdmap e437: 8 total, 8 up, 8 in 2024-09-21T10:02:34.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:33 smithi070 ceph-mon[107385]: osdmap e437: 8 total, 8 up, 8 in 2024-09-21T10:02:34.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:33 smithi070 ceph-mon[112139]: osdmap e437: 8 total, 8 up, 8 in 2024-09-21T10:02:35.023 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeImage_WithoutJournaling (1259 ms) 2024-09-21T10:02:35.023 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeImage_WithoutExclusiveLock 2024-09-21T10:02:35.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:34 smithi160 ceph-mon[33203]: pgmap v148: 105 pgs: 32 creating+peering, 73 active+clean; 582 KiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:02:35.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:34 smithi160 ceph-mon[33203]: osdmap e438: 8 total, 8 up, 8 in 2024-09-21T10:02:35.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:34 smithi070 ceph-mon[107385]: pgmap v148: 105 pgs: 32 creating+peering, 73 active+clean; 582 KiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:02:35.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:34 smithi070 ceph-mon[107385]: osdmap e438: 8 total, 8 up, 8 in 2024-09-21T10:02:35.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:34 smithi070 ceph-mon[112139]: pgmap v148: 105 pgs: 32 creating+peering, 73 active+clean; 582 KiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:02:35.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:34 smithi070 ceph-mon[112139]: osdmap e438: 8 total, 8 up, 8 in 2024-09-21T10:02:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:35 smithi160 ceph-mon[33203]: osdmap e439: 8 total, 8 up, 8 in 2024-09-21T10:02:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:35 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:02:36.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:35 smithi070 ceph-mon[107385]: osdmap e439: 8 total, 8 up, 8 in 2024-09-21T10:02:36.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:02:36.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:35 smithi070 ceph-mon[112139]: osdmap e439: 8 total, 8 up, 8 in 2024-09-21T10:02:36.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:02:37.029 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeImage_WithoutExclusiveLock (2006 ms) 2024-09-21T10:02:37.029 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModeDisabled 2024-09-21T10:02:37.112 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModeDisabled (82 ms) 2024-09-21T10:02:37.112 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModeImage 2024-09-21T10:02:37.196 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModeImage (84 ms) 2024-09-21T10:02:37.196 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModePool 2024-09-21T10:02:37.286 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModePool (90 ms) 2024-09-21T10:02:37.286 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModePool_WithoutJournaling 2024-09-21T10:02:37.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:36 smithi160 ceph-mon[33203]: pgmap v151: 105 pgs: 7 creating+peering, 98 active+clean; 582 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 354 KiB/s rd, 48 KiB/s wr, 517 op/s 2024-09-21T10:02:37.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:36 smithi160 ceph-mon[33203]: osdmap e440: 8 total, 8 up, 8 in 2024-09-21T10:02:37.362 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModePool_WithoutJournaling (76 ms) 2024-09-21T10:02:37.362 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModeImage_WithoutJournaling 2024-09-21T10:02:37.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:36 smithi070 ceph-mon[107385]: pgmap v151: 105 pgs: 7 creating+peering, 98 active+clean; 582 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 354 KiB/s rd, 48 KiB/s wr, 517 op/s 2024-09-21T10:02:37.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:36 smithi070 ceph-mon[107385]: osdmap e440: 8 total, 8 up, 8 in 2024-09-21T10:02:37.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:36 smithi070 ceph-mon[112139]: pgmap v151: 105 pgs: 7 creating+peering, 98 active+clean; 582 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 354 KiB/s rd, 48 KiB/s wr, 517 op/s 2024-09-21T10:02:37.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:36 smithi070 ceph-mon[112139]: osdmap e440: 8 total, 8 up, 8 in 2024-09-21T10:02:37.437 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModeImage_WithoutJournaling (75 ms) 2024-09-21T10:02:37.437 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.EnableJournaling_In_MirrorModeDisabled 2024-09-21T10:02:37.523 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.EnableJournaling_In_MirrorModeDisabled (86 ms) 2024-09-21T10:02:37.523 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.EnableJournaling_In_MirrorModeImage 2024-09-21T10:02:37.620 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.EnableJournaling_In_MirrorModeImage (97 ms) 2024-09-21T10:02:37.621 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.EnableJournaling_In_MirrorModeImage_SnapshotMirroringEnabled 2024-09-21T10:02:38.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:37 smithi160 ceph-mon[33203]: osdmap e441: 8 total, 8 up, 8 in 2024-09-21T10:02:38.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:37 smithi070 ceph-mon[107385]: osdmap e441: 8 total, 8 up, 8 in 2024-09-21T10:02:38.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:37 smithi070 ceph-mon[112139]: osdmap e441: 8 total, 8 up, 8 in 2024-09-21T10:02:39.053 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.EnableJournaling_In_MirrorModeImage_SnapshotMirroringEnabled (1432 ms) 2024-09-21T10:02:39.053 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.EnableJournaling_In_MirrorModePool 2024-09-21T10:02:39.156 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.EnableJournaling_In_MirrorModePool (104 ms) 2024-09-21T10:02:39.156 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.DisableJournaling_In_MirrorModePool 2024-09-21T10:02:39.249 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.DisableJournaling_In_MirrorModePool (93 ms) 2024-09-21T10:02:39.249 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.DisableJournaling_In_MirrorModeImage 2024-09-21T10:02:39.339 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.DisableJournaling_In_MirrorModeImage (89 ms) 2024-09-21T10:02:39.339 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.MirrorModeSet_DisabledMode_To_PoolMode 2024-09-21T10:02:39.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:38 smithi160 ceph-mon[33203]: pgmap v154: 105 pgs: 105 active+clean; 582 KiB data, 466 MiB used, 715 GiB / 715 GiB avail; 411 KiB/s rd, 55 KiB/s wr, 595 op/s 2024-09-21T10:02:39.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:38 smithi160 ceph-mon[33203]: osdmap e442: 8 total, 8 up, 8 in 2024-09-21T10:02:39.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:38 smithi070 ceph-mon[107385]: pgmap v154: 105 pgs: 105 active+clean; 582 KiB data, 466 MiB used, 715 GiB / 715 GiB avail; 411 KiB/s rd, 55 KiB/s wr, 595 op/s 2024-09-21T10:02:39.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:38 smithi070 ceph-mon[107385]: osdmap e442: 8 total, 8 up, 8 in 2024-09-21T10:02:39.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:38 smithi070 ceph-mon[112139]: pgmap v154: 105 pgs: 105 active+clean; 582 KiB data, 466 MiB used, 715 GiB / 715 GiB avail; 411 KiB/s rd, 55 KiB/s wr, 595 op/s 2024-09-21T10:02:39.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:38 smithi070 ceph-mon[112139]: osdmap e442: 8 total, 8 up, 8 in 2024-09-21T10:02:39.479 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.MirrorModeSet_DisabledMode_To_PoolMode (140 ms) 2024-09-21T10:02:39.479 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.MirrorModeSet_PoolMode_To_DisabledMode 2024-09-21T10:02:39.621 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.MirrorModeSet_PoolMode_To_DisabledMode (143 ms) 2024-09-21T10:02:39.621 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.MirrorModeSet_DisabledMode_To_ImageMode 2024-09-21T10:02:39.747 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.MirrorModeSet_DisabledMode_To_ImageMode (126 ms) 2024-09-21T10:02:39.747 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.MirrorModeSet_PoolMode_To_ImageMode 2024-09-21T10:02:39.880 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.MirrorModeSet_PoolMode_To_ImageMode (132 ms) 2024-09-21T10:02:39.880 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.MirrorModeSet_ImageMode_To_PoolMode 2024-09-21T10:02:40.021 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.MirrorModeSet_ImageMode_To_PoolMode (142 ms) 2024-09-21T10:02:40.021 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.MirrorModeSet_ImageMode_To_DisabledMode 2024-09-21T10:02:40.178 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.MirrorModeSet_ImageMode_To_DisabledMode (156 ms) 2024-09-21T10:02:40.178 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.RemoveImage_With_MirrorImageEnabled 2024-09-21T10:02:40.257 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.RemoveImage_With_MirrorImageEnabled (79 ms) 2024-09-21T10:02:40.258 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.RemoveImage_With_MirrorImageDisabled 2024-09-21T10:02:40.258 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:39 smithi160 ceph-mon[33203]: osdmap e443: 8 total, 8 up, 8 in 2024-09-21T10:02:40.327 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.RemoveImage_With_MirrorImageDisabled (71 ms) 2024-09-21T10:02:40.327 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.RemoveImage_With_ImageWithoutJournal 2024-09-21T10:02:40.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:39 smithi070 ceph-mon[107385]: osdmap e443: 8 total, 8 up, 8 in 2024-09-21T10:02:40.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:39 smithi070 ceph-mon[112139]: osdmap e443: 8 total, 8 up, 8 in 2024-09-21T10:02:40.395 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.RemoveImage_With_ImageWithoutJournal (68 ms) 2024-09-21T10:02:40.395 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.RemoveImage_With_MirrorImageDemoted 2024-09-21T10:02:40.506 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.RemoveImage_With_MirrorImageDemoted (111 ms) 2024-09-21T10:02:40.506 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.TrashMoveRestore_PoolMode 2024-09-21T10:02:40.626 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.TrashMoveRestore_PoolMode (120 ms) 2024-09-21T10:02:40.626 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.TrashMoveRestore_ImageMode_MirroringDisabled 2024-09-21T10:02:40.725 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.TrashMoveRestore_ImageMode_MirroringDisabled (99 ms) 2024-09-21T10:02:40.725 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.TrashMoveRestore_ImageMode_MirroringEnabled 2024-09-21T10:02:40.834 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.TrashMoveRestore_ImageMode_MirroringEnabled (109 ms) 2024-09-21T10:02:40.834 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.MirrorStatusList 2024-09-21T10:02:40.903 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.MirrorStatusList (69 ms) 2024-09-21T10:02:40.903 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.RemoveBootstrapped 2024-09-21T10:02:40.991 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.RemoveBootstrapped (88 ms) 2024-09-21T10:02:40.992 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.AioPromoteDemote 2024-09-21T10:02:41.326 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.AioPromoteDemote (334 ms) 2024-09-21T10:02:41.326 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.AioGetInfo 2024-09-21T10:02:41.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:40 smithi160 ceph-mon[33203]: pgmap v157: 105 pgs: 105 active+clean; 582 KiB data, 466 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 6.7 KiB/s wr, 77 op/s 2024-09-21T10:02:41.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:40 smithi070 ceph-mon[107385]: pgmap v157: 105 pgs: 105 active+clean; 582 KiB data, 466 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 6.7 KiB/s wr, 77 op/s 2024-09-21T10:02:41.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:40 smithi070 ceph-mon[112139]: pgmap v157: 105 pgs: 105 active+clean; 582 KiB data, 466 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 6.7 KiB/s wr, 77 op/s 2024-09-21T10:02:41.643 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.AioGetInfo (318 ms) 2024-09-21T10:02:41.643 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.AioGetStatus 2024-09-21T10:02:41.855 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.AioGetStatus (211 ms) 2024-09-21T10:02:41.855 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.SiteName 2024-09-21T10:02:41.861 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.SiteName (7 ms) 2024-09-21T10:02:41.861 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.Bootstrap 2024-09-21T10:02:42.625 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.Bootstrap (764 ms) 2024-09-21T10:02:42.625 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.PeerDirection 2024-09-21T10:02:42.633 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.PeerDirection (8 ms) 2024-09-21T10:02:42.633 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.Snapshot 2024-09-21T10:02:43.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' 2024-09-21T10:02:43.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/site_name"} : dispatch 2024-09-21T10:02:43.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/site_name"} : dispatch 2024-09-21T10:02:43.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]': finished 2024-09-21T10:02:43.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/site_name"} : dispatch 2024-09-21T10:02:43.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[107385]: pgmap v158: 105 pgs: 105 active+clean; 582 KiB data, 462 MiB used, 715 GiB / 715 GiB avail; 846 KiB/s rd, 117 KiB/s wr, 1.20k op/s 2024-09-21T10:02:43.130 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/peer_client_id"} : dispatch 2024-09-21T10:02:43.130 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' 2024-09-21T10:02:43.130 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"} : dispatch 2024-09-21T10:02:43.130 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]': finished 2024-09-21T10:02:43.130 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/site_name"} : dispatch 2024-09-21T10:02:43.130 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/267ce723-2415-454d-9ec9-426773ef0b76"} : dispatch 2024-09-21T10:02:43.130 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/267ce723-2415-454d-9ec9-426773ef0b76"}]': finished 2024-09-21T10:02:43.130 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[107385]: osdmap e444: 8 total, 8 up, 8 in 2024-09-21T10:02:43.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' 2024-09-21T10:02:43.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/site_name"} : dispatch 2024-09-21T10:02:43.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/site_name"} : dispatch 2024-09-21T10:02:43.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]': finished 2024-09-21T10:02:43.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/site_name"} : dispatch 2024-09-21T10:02:43.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[112139]: pgmap v158: 105 pgs: 105 active+clean; 582 KiB data, 462 MiB used, 715 GiB / 715 GiB avail; 846 KiB/s rd, 117 KiB/s wr, 1.20k op/s 2024-09-21T10:02:43.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/peer_client_id"} : dispatch 2024-09-21T10:02:43.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' 2024-09-21T10:02:43.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"} : dispatch 2024-09-21T10:02:43.132 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]': finished 2024-09-21T10:02:43.132 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/site_name"} : dispatch 2024-09-21T10:02:43.132 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/267ce723-2415-454d-9ec9-426773ef0b76"} : dispatch 2024-09-21T10:02:43.132 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/267ce723-2415-454d-9ec9-426773ef0b76"}]': finished 2024-09-21T10:02:43.132 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:42 smithi070 ceph-mon[112139]: osdmap e444: 8 total, 8 up, 8 in 2024-09-21T10:02:43.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:42 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' 2024-09-21T10:02:43.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:42 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/site_name"} : dispatch 2024-09-21T10:02:43.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:42 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/site_name"} : dispatch 2024-09-21T10:02:43.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:42 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]': finished 2024-09-21T10:02:43.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:42 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/site_name"} : dispatch 2024-09-21T10:02:43.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:42 smithi160 ceph-mon[33203]: pgmap v158: 105 pgs: 105 active+clean; 582 KiB data, 462 MiB used, 715 GiB / 715 GiB avail; 846 KiB/s rd, 117 KiB/s wr, 1.20k op/s 2024-09-21T10:02:43.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:42 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/peer_client_id"} : dispatch 2024-09-21T10:02:43.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:42 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' 2024-09-21T10:02:43.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:42 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"} : dispatch 2024-09-21T10:02:43.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:42 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]': finished 2024-09-21T10:02:43.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:42 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/site_name"} : dispatch 2024-09-21T10:02:43.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:42 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/267ce723-2415-454d-9ec9-426773ef0b76"} : dispatch 2024-09-21T10:02:43.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:42 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/267ce723-2415-454d-9ec9-426773ef0b76"}]': finished 2024-09-21T10:02:43.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:42 smithi160 ceph-mon[33203]: osdmap e444: 8 total, 8 up, 8 in 2024-09-21T10:02:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:02:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:02:43] "GET /metrics HTTP/1.1" 200 36213 "" "Prometheus/2.51.0" 2024-09-21T10:02:45.015 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:44 smithi070 ceph-mon[112139]: osdmap e445: 8 total, 8 up, 8 in 2024-09-21T10:02:45.015 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:44 smithi070 ceph-mon[112139]: pgmap v161: 105 pgs: 105 active+clean; 582 KiB data, 462 MiB used, 715 GiB / 715 GiB avail; 847 KiB/s rd, 117 KiB/s wr, 1.20k op/s 2024-09-21T10:02:45.016 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:44 smithi070 ceph-mon[107385]: osdmap e445: 8 total, 8 up, 8 in 2024-09-21T10:02:45.016 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:44 smithi070 ceph-mon[107385]: pgmap v161: 105 pgs: 105 active+clean; 582 KiB data, 462 MiB used, 715 GiB / 715 GiB avail; 847 KiB/s rd, 117 KiB/s wr, 1.20k op/s 2024-09-21T10:02:45.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:44 smithi160 ceph-mon[33203]: osdmap e445: 8 total, 8 up, 8 in 2024-09-21T10:02:45.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:44 smithi160 ceph-mon[33203]: pgmap v161: 105 pgs: 105 active+clean; 582 KiB data, 462 MiB used, 715 GiB / 715 GiB avail; 847 KiB/s rd, 117 KiB/s wr, 1.20k op/s 2024-09-21T10:02:45.415 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'echo wait for majority of mons to upgrade' 2024-09-21T10:02:45.615 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:45.908 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:45 smithi070 ceph-mon[107385]: osdmap e446: 8 total, 8 up, 8 in 2024-09-21T10:02:45.908 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:45 smithi070 ceph-mon[112139]: osdmap e446: 8 total, 8 up, 8 in 2024-09-21T10:02:46.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:45 smithi160 ceph-mon[33203]: osdmap e446: 8 total, 8 up, 8 in 2024-09-21T10:02:46.789 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:46 smithi070 ceph-mon[107385]: osdmap e447: 8 total, 8 up, 8 in 2024-09-21T10:02:46.789 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:46 smithi070 ceph-mon[107385]: pgmap v164: 105 pgs: 105 active+clean; 583 KiB data, 467 MiB used, 715 GiB / 715 GiB avail; 1007 KiB/s rd, 171 KiB/s wr, 1.36k op/s 2024-09-21T10:02:46.789 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:46 smithi070 ceph-mon[112139]: osdmap e447: 8 total, 8 up, 8 in 2024-09-21T10:02:46.789 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:46 smithi070 ceph-mon[112139]: pgmap v164: 105 pgs: 105 active+clean; 583 KiB data, 467 MiB used, 715 GiB / 715 GiB avail; 1007 KiB/s rd, 171 KiB/s wr, 1.36k op/s 2024-09-21T10:02:47.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:46 smithi160 ceph-mon[33203]: osdmap e447: 8 total, 8 up, 8 in 2024-09-21T10:02:47.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:46 smithi160 ceph-mon[33203]: pgmap v164: 105 pgs: 105 active+clean; 583 KiB data, 467 MiB used, 715 GiB / 715 GiB avail; 1007 KiB/s rd, 171 KiB/s wr, 1.36k op/s 2024-09-21T10:02:47.284 INFO:teuthology.orchestra.run.smithi070.stdout:wait for majority of mons to upgrade 2024-09-21T10:02:47.921 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:47 smithi070 ceph-mon[107385]: osdmap e448: 8 total, 8 up, 8 in 2024-09-21T10:02:47.921 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:47 smithi070 ceph-mon[107385]: osdmap e449: 8 total, 8 up, 8 in 2024-09-21T10:02:47.921 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:47 smithi070 ceph-mon[112139]: osdmap e448: 8 total, 8 up, 8 in 2024-09-21T10:02:47.921 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:47 smithi070 ceph-mon[112139]: osdmap e449: 8 total, 8 up, 8 in 2024-09-21T10:02:47.943 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --limit 1' 2024-09-21T10:02:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:47 smithi160 ceph-mon[33203]: osdmap e448: 8 total, 8 up, 8 in 2024-09-21T10:02:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:47 smithi160 ceph-mon[33203]: osdmap e449: 8 total, 8 up, 8 in 2024-09-21T10:02:48.137 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:48.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:48 smithi070 ceph-mon[112139]: pgmap v167: 105 pgs: 105 active+clean; 583 KiB data, 471 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 181 KiB/s wr, 1.59k op/s 2024-09-21T10:02:48.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:48 smithi070 ceph-mon[107385]: pgmap v167: 105 pgs: 105 active+clean; 583 KiB data, 471 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 181 KiB/s wr, 1.59k op/s 2024-09-21T10:02:49.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:48 smithi160 ceph-mon[33203]: pgmap v167: 105 pgs: 105 active+clean; 583 KiB data, 471 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 181 KiB/s wr, 1.59k op/s 2024-09-21T10:02:49.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:49 smithi070 ceph-mon[107385]: osdmap e450: 8 total, 8 up, 8 in 2024-09-21T10:02:49.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:49 smithi070 ceph-mon[112139]: osdmap e450: 8 total, 8 up, 8 in 2024-09-21T10:02:50.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:49 smithi160 ceph-mon[33203]: osdmap e450: 8 total, 8 up, 8 in 2024-09-21T10:02:51.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:50 smithi070 ceph-mon[107385]: osdmap e451: 8 total, 8 up, 8 in 2024-09-21T10:02:51.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:50 smithi070 ceph-mon[107385]: pgmap v170: 105 pgs: 105 active+clean; 583 KiB data, 471 MiB used, 715 GiB / 715 GiB avail; 187 KiB/s rd, 10 KiB/s wr, 229 op/s 2024-09-21T10:02:51.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:50 smithi070 ceph-mon[107385]: from='client.34221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208", "daemon_types": "mon", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:51.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:02:51.003 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:50 smithi070 ceph-mon[112139]: osdmap e451: 8 total, 8 up, 8 in 2024-09-21T10:02:51.003 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:50 smithi070 ceph-mon[112139]: pgmap v170: 105 pgs: 105 active+clean; 583 KiB data, 471 MiB used, 715 GiB / 715 GiB avail; 187 KiB/s rd, 10 KiB/s wr, 229 op/s 2024-09-21T10:02:51.004 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:50 smithi070 ceph-mon[112139]: from='client.34221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208", "daemon_types": "mon", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:51.004 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:02:51.044 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:50 smithi160 ceph-mon[33203]: osdmap e451: 8 total, 8 up, 8 in 2024-09-21T10:02:51.044 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:50 smithi160 ceph-mon[33203]: pgmap v170: 105 pgs: 105 active+clean; 583 KiB data, 471 MiB used, 715 GiB / 715 GiB avail; 187 KiB/s rd, 10 KiB/s wr, 229 op/s 2024-09-21T10:02:51.044 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:50 smithi160 ceph-mon[33203]: from='client.34221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208", "daemon_types": "mon", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:51.045 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:50 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:02:52.066 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:51 smithi070 ceph-mon[107385]: osdmap e452: 8 total, 8 up, 8 in 2024-09-21T10:02:52.067 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:51 smithi070 ceph-mon[112139]: osdmap e452: 8 total, 8 up, 8 in 2024-09-21T10:02:52.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:51 smithi160 ceph-mon[33203]: osdmap e452: 8 total, 8 up, 8 in 2024-09-21T10:02:52.103 INFO:teuthology.orchestra.run.smithi070.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:02:52.770 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:52 smithi070 ceph-mon[112139]: osdmap e453: 8 total, 8 up, 8 in 2024-09-21T10:02:52.771 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:52 smithi070 ceph-mon[112139]: pgmap v173: 105 pgs: 105 active+clean; 583 KiB data, 472 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 7.0 KiB/s wr, 85 op/s 2024-09-21T10:02:52.771 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:52 smithi070 ceph-mon[112139]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:02:52.771 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:52 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:52.771 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:52 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:02:52.771 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:52 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:02:52.771 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:52 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:02:52.771 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:52 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:52.771 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:52 smithi070 ceph-mon[107385]: osdmap e453: 8 total, 8 up, 8 in 2024-09-21T10:02:52.771 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:52 smithi070 ceph-mon[107385]: pgmap v173: 105 pgs: 105 active+clean; 583 KiB data, 472 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 7.0 KiB/s wr, 85 op/s 2024-09-21T10:02:52.771 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:52 smithi070 ceph-mon[107385]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:02:52.771 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:52 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:52.772 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:52 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:02:52.772 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:52 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:02:52.772 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:52 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:02:52.772 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:52 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:52.790 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-09-21T10:02:53.003 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:53.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:52 smithi160 ceph-mon[33203]: osdmap e453: 8 total, 8 up, 8 in 2024-09-21T10:02:53.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:52 smithi160 ceph-mon[33203]: pgmap v173: 105 pgs: 105 active+clean; 583 KiB data, 472 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 7.0 KiB/s wr, 85 op/s 2024-09-21T10:02:53.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:52 smithi160 ceph-mon[33203]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:02:53.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:52 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:53.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:52 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:02:53.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:52 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:02:53.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:52 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:02:53.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:52 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:53.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:02:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:02:53] "GET /metrics HTTP/1.1" 200 36264 "" "Prometheus/2.51.0" 2024-09-21T10:02:54.013 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:53 smithi070 ceph-mon[107385]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:02:54.013 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:53 smithi070 ceph-mon[107385]: osdmap e454: 8 total, 8 up, 8 in 2024-09-21T10:02:54.013 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:53 smithi070 ceph-mon[112139]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:02:54.013 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:53 smithi070 ceph-mon[112139]: osdmap e454: 8 total, 8 up, 8 in 2024-09-21T10:02:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:53 smithi160 ceph-mon[33203]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:02:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:53 smithi160 ceph-mon[33203]: osdmap e454: 8 total, 8 up, 8 in 2024-09-21T10:02:54.833 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:54 smithi070 ceph-mon[107385]: osdmap e455: 8 total, 8 up, 8 in 2024-09-21T10:02:54.834 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:54 smithi070 ceph-mon[107385]: pgmap v176: 105 pgs: 105 active+clean; 583 KiB data, 472 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 7.0 KiB/s wr, 85 op/s 2024-09-21T10:02:54.834 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:54 smithi070 ceph-mon[112139]: osdmap e455: 8 total, 8 up, 8 in 2024-09-21T10:02:54.834 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:54 smithi070 ceph-mon[112139]: pgmap v176: 105 pgs: 105 active+clean; 583 KiB data, 472 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 7.0 KiB/s wr, 85 op/s 2024-09-21T10:02:55.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:54 smithi160 ceph-mon[33203]: osdmap e455: 8 total, 8 up, 8 in 2024-09-21T10:02:55.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:54 smithi160 ceph-mon[33203]: pgmap v176: 105 pgs: 105 active+clean; 583 KiB data, 472 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 7.0 KiB/s wr, 85 op/s 2024-09-21T10:02:55.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:55 smithi070 ceph-mon[107385]: osdmap e456: 8 total, 8 up, 8 in 2024-09-21T10:02:55.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:55 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:55.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:55 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:02:55.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:55 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:02:55.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:55 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:55.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:55 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-21T10:02:55.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:55 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-09-21T10:02:55.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:55 smithi070 ceph-mon[112139]: osdmap e456: 8 total, 8 up, 8 in 2024-09-21T10:02:55.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:55 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:55.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:55 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:02:55.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:55 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:02:55.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:55 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:55.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:55 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-21T10:02:55.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:55 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-09-21T10:02:56.022 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:55 smithi160 ceph-mon[33203]: osdmap e456: 8 total, 8 up, 8 in 2024-09-21T10:02:56.022 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:55 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:56.022 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:55 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:02:56.022 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:55 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:02:56.023 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:55 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:56.023 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:55 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-21T10:02:56.023 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:55 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-09-21T10:02:56.232 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-09-21T10:02:56.732 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T10:02:56.732 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (3m) 35s ago 9m 20.4M - 0.27.0 11f11916f8cd 89ae95182419 2024-09-21T10:02:56.732 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi160 *:3000 running (3m) 34s ago 8m 82.5M - 10.4.8 5aad1d7cf004 daa2d8dfab48 2024-09-21T10:02:56.732 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi070.gansda smithi070 running (7m) 35s ago 7m 20.1M - 18.2.4-871-g6a1bdf85 0adab276c97e 6016b2501c8a 2024-09-21T10:02:56.732 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi160.byyhfz smithi160 running (7m) 34s ago 7m 15.8M - 18.2.4-871-g6a1bdf85 0adab276c97e 93e108d91e65 2024-09-21T10:02:56.732 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi160 *:8443,9283,8765 running (4m) 34s ago 13m 455M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 d952652a9683 2024-09-21T10:02:56.732 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (4m) 35s ago 14m 509M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 e3d390893cff 2024-09-21T10:02:56.732 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (2m) 35s ago 14m 47.2M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 4a7b667a4033 2024-09-21T10:02:56.733 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi160 running (13m) 34s ago 13m 63.0M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e 0fd16436dfa0 2024-09-21T10:02:56.733 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (112s) 35s ago 13m 39.2M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 e154380b80a2 2024-09-21T10:02:56.733 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (4m) 35s ago 9m 11.3M - 1.7.0 72c9c2088986 5d5d4b026070 2024-09-21T10:02:56.733 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi160 *:9100 running (4m) 34s ago 9m 11.2M - 1.7.0 72c9c2088986 19b8e0e8ee0c 2024-09-21T10:02:56.733 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (12m) 35s ago 12m 115M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 10580e80f007 2024-09-21T10:02:56.733 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (12m) 35s ago 12m 125M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e b7c0d1169d1d 2024-09-21T10:02:56.733 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (11m) 35s ago 11m 161M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 58a11b8ddb81 2024-09-21T10:02:56.733 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (11m) 35s ago 11m 131M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e aa0c8a732a41 2024-09-21T10:02:56.733 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi160 running (11m) 34s ago 11m 134M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 7325a075c488 2024-09-21T10:02:56.733 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi160 running (10m) 34s ago 10m 101M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e f74026345641 2024-09-21T10:02:56.733 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi160 running (10m) 34s ago 10m 156M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 03e6bb54f8b2 2024-09-21T10:02:56.733 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi160 running (10m) 34s ago 10m 120M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e e405b21a6146 2024-09-21T10:02:56.733 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi160 *:9095 running (3m) 34s ago 9m 53.6M - 2.51.0 1d3b7f56885b 648952e57ad7 2024-09-21T10:02:57.002 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:56 smithi160 ceph-mon[33203]: Upgrade: Target is version 19.3.0-5092-g62da54b4 (squid) 2024-09-21T10:02:57.002 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:56 smithi160 ceph-mon[33203]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, digests ['quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47'] 2024-09-21T10:02:57.002 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:56 smithi160 ceph-mon[33203]: Upgrade: Setting container_image for all mgr 2024-09-21T10:02:57.003 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:56 smithi160 ceph-mon[33203]: Upgrade: It appears safe to stop mon.b 2024-09-21T10:02:57.003 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:56 smithi160 ceph-mon[33203]: osdmap e457: 8 total, 8 up, 8 in 2024-09-21T10:02:57.003 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:56 smithi160 ceph-mon[33203]: pgmap v179: 105 pgs: 105 active+clean; 583 KiB data, 473 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 5.2 KiB/s wr, 88 op/s 2024-09-21T10:02:57.071 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:02:57.072 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-09-21T10:02:57.072 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 1, 2024-09-21T10:02:57.072 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T10:02:57.072 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:02:57.072 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-09-21T10:02:57.072 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T10:02:57.072 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:02:57.072 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-09-21T10:02:57.072 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 8 2024-09-21T10:02:57.073 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:02:57.073 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": { 2024-09-21T10:02:57.073 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 2 2024-09-21T10:02:57.073 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:02:57.073 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-09-21T10:02:57.073 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 11, 2024-09-21T10:02:57.073 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 4 2024-09-21T10:02:57.073 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:02:57.073 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:02:57.074 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:56 smithi070 ceph-mon[107385]: Upgrade: Target is version 19.3.0-5092-g62da54b4 (squid) 2024-09-21T10:02:57.074 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:56 smithi070 ceph-mon[107385]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, digests ['quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47'] 2024-09-21T10:02:57.074 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:56 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all mgr 2024-09-21T10:02:57.074 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:56 smithi070 ceph-mon[107385]: Upgrade: It appears safe to stop mon.b 2024-09-21T10:02:57.074 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:56 smithi070 ceph-mon[107385]: osdmap e457: 8 total, 8 up, 8 in 2024-09-21T10:02:57.074 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:56 smithi070 ceph-mon[107385]: pgmap v179: 105 pgs: 105 active+clean; 583 KiB data, 473 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 5.2 KiB/s wr, 88 op/s 2024-09-21T10:02:57.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:56 smithi070 ceph-mon[112139]: Upgrade: Target is version 19.3.0-5092-g62da54b4 (squid) 2024-09-21T10:02:57.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:56 smithi070 ceph-mon[112139]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, digests ['quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47'] 2024-09-21T10:02:57.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:56 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all mgr 2024-09-21T10:02:57.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:56 smithi070 ceph-mon[112139]: Upgrade: It appears safe to stop mon.b 2024-09-21T10:02:57.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:56 smithi070 ceph-mon[112139]: osdmap e457: 8 total, 8 up, 8 in 2024-09-21T10:02:57.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:56 smithi070 ceph-mon[112139]: pgmap v179: 105 pgs: 105 active+clean; 583 KiB data, 473 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 5.2 KiB/s wr, 88 op/s 2024-09-21T10:02:57.365 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:02:57.365 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47", 2024-09-21T10:02:57.365 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-09-21T10:02:57.365 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) mon. Upgrade limited to 1 daemons (1 remaining).", 2024-09-21T10:02:57.366 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-09-21T10:02:57.366 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "2/3 daemons upgraded", 2024-09-21T10:02:57.366 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading mon daemons", 2024-09-21T10:02:57.366 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-09-21T10:02:57.366 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:02:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:57 smithi160 ceph-mon[33203]: from='client.44156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:57 smithi160 ceph-mon[33203]: from='client.34233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:57 smithi160 ceph-mon[33203]: osdmap e458: 8 total, 8 up, 8 in 2024-09-21T10:02:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:57 smithi160 ceph-mon[33203]: from='client.44168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:57.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:57 smithi160 ceph-mon[33203]: from='client.? 172.21.15.70:0/870631390' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:02:57.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:57 smithi160 ceph-mon[33203]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:57.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:57 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T10:02:57.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:57 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T10:02:57.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:57 smithi160 ceph-mon[33203]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:02:57.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:57 smithi160 ceph-mon[33203]: osdmap e459: 8 total, 8 up, 8 in 2024-09-21T10:02:58.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:57 smithi070 ceph-mon[107385]: from='client.44156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:58.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:57 smithi070 ceph-mon[107385]: from='client.34233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:58.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:57 smithi070 ceph-mon[107385]: osdmap e458: 8 total, 8 up, 8 in 2024-09-21T10:02:58.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:57 smithi070 ceph-mon[107385]: from='client.44168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:58.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:57 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/870631390' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:02:58.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:57 smithi070 ceph-mon[107385]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:58.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:57 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T10:02:58.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:57 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T10:02:58.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:57 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:02:58.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:57 smithi070 ceph-mon[107385]: osdmap e459: 8 total, 8 up, 8 in 2024-09-21T10:02:58.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:57 smithi070 ceph-mon[112139]: from='client.44156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:58.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:57 smithi070 ceph-mon[112139]: from='client.34233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:58.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:57 smithi070 ceph-mon[112139]: osdmap e458: 8 total, 8 up, 8 in 2024-09-21T10:02:58.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:57 smithi070 ceph-mon[112139]: from='client.44168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:58.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:57 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/870631390' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:02:58.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:57 smithi070 ceph-mon[112139]: from='mgr.34107 ' entity='mgr.y' 2024-09-21T10:02:58.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:57 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T10:02:58.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:57 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T10:02:58.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:57 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:02:58.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:57 smithi070 ceph-mon[112139]: osdmap e459: 8 total, 8 up, 8 in 2024-09-21T10:02:58.729 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.Snapshot (16096 ms) 2024-09-21T10:02:58.729 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.SnapshotRemoveOnDisable 2024-09-21T10:02:59.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:58 smithi160 ceph-mon[33203]: from='client.34251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:59.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:58 smithi160 ceph-mon[33203]: Upgrade: Updating mon.b 2024-09-21T10:02:59.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:58 smithi160 ceph-mon[33203]: Deploying daemon mon.b on smithi160 2024-09-21T10:02:59.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:58 smithi160 ceph-mon[33203]: pgmap v182: 105 pgs: 105 active+clean; 583 KiB data, 473 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 5.2 KiB/s wr, 88 op/s 2024-09-21T10:02:59.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:58 smithi160 ceph-mon[33203]: osdmap e460: 8 total, 8 up, 8 in 2024-09-21T10:02:59.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:58 smithi070 ceph-mon[107385]: from='client.34251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:59.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:58 smithi070 ceph-mon[107385]: Upgrade: Updating mon.b 2024-09-21T10:02:59.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:58 smithi070 ceph-mon[107385]: Deploying daemon mon.b on smithi160 2024-09-21T10:02:59.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:58 smithi070 ceph-mon[107385]: pgmap v182: 105 pgs: 105 active+clean; 583 KiB data, 473 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 5.2 KiB/s wr, 88 op/s 2024-09-21T10:02:59.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:02:58 smithi070 ceph-mon[107385]: osdmap e460: 8 total, 8 up, 8 in 2024-09-21T10:02:59.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:58 smithi070 ceph-mon[112139]: from='client.34251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:59.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:58 smithi070 ceph-mon[112139]: Upgrade: Updating mon.b 2024-09-21T10:02:59.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:58 smithi070 ceph-mon[112139]: Deploying daemon mon.b on smithi160 2024-09-21T10:02:59.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:58 smithi070 ceph-mon[112139]: pgmap v182: 105 pgs: 105 active+clean; 583 KiB data, 473 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 5.2 KiB/s wr, 88 op/s 2024-09-21T10:02:59.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:02:58 smithi070 ceph-mon[112139]: osdmap e460: 8 total, 8 up, 8 in 2024-09-21T10:02:59.677 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:59 smithi160 systemd[1]: Stopping Ceph mon.b for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:02:59.677 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:59 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-b[33180]: 2024-09-21T10:02:59.655+0000 7f0ea6fc3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:02:59.677 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:59 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-b[33180]: 2024-09-21T10:02:59.655+0000 7f0ea6fc3640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-21T10:03:00.023 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:02:59 smithi160 podman[101688]: 2024-09-21 10:02:59.676152942 +0000 UTC m=+0.111397433 container died 0fd16436dfa06bed80a8c8835b332485d9430f5a851d23d51af960daa8758c5b (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=reef-6a1bdf8, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-21T10:03:00.127 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:02:59 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:02:59] ENGINE Bus STOPPING 2024-09-21T10:03:00.339 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:00 smithi160 podman[101688]: 2024-09-21 10:03:00.103563088 +0000 UTC m=+0.538807576 container cleanup 0fd16436dfa06bed80a8c8835b332485d9430f5a851d23d51af960daa8758c5b (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-b, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=reef-6a1bdf8, GIT_BRANCH=HEAD) 2024-09-21T10:03:00.340 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:00 smithi160 bash[101688]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-b 2024-09-21T10:03:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:00 smithi160 podman[101706]: 2024-09-21 10:03:00.338283311 +0000 UTC m=+0.655572002 container remove 0fd16436dfa06bed80a8c8835b332485d9430f5a851d23d51af960daa8758c5b (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-b, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-6a1bdf8, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:03:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:00 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mon.b.service: Deactivated successfully. 2024-09-21T10:03:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:00 smithi160 systemd[1]: Stopped Ceph mon.b for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:03:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:00 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mon.b.service: Consumed 11.798s CPU time. 2024-09-21T10:03:00.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:03:00 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:03:00] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T10:03:00.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:03:00 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:03:00] ENGINE Bus STOPPED 2024-09-21T10:03:00.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:03:00 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:03:00] ENGINE Bus STARTING 2024-09-21T10:03:00.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:03:00 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:03:00] ENGINE Serving on http://:::9283 2024-09-21T10:03:00.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:03:00 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:03:00] ENGINE Bus STARTED 2024-09-21T10:03:01.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:00 smithi160 systemd[1]: Starting Ceph mon.b for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:03:01.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:00 smithi160 podman[101841]: 2024-09-21 10:03:00.882892041 +0000 UTC m=+0.015825104 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:03:01.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:00 smithi160 podman[101841]: 2024-09-21 10:03:00.980880195 +0000 UTC m=+0.113813258 container create 86e88af21f363c6d8bfd177fd8ebb63fb182a6bd6d5acfa40ef9d00474af8514 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-b, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-62da54b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-21T10:03:01.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 podman[101841]: 2024-09-21 10:03:01.233662171 +0000 UTC m=+0.366595238 container init 86e88af21f363c6d8bfd177fd8ebb63fb182a6bd6d5acfa40ef9d00474af8514 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-b, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:03:01.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 podman[101841]: 2024-09-21 10:03:01.238077357 +0000 UTC m=+0.371010423 container start 86e88af21f363c6d8bfd177fd8ebb63fb182a6bd6d5acfa40ef9d00474af8514 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS) 2024-09-21T10:03:01.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T10:03:01.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev), process ceph-mon, pid 7 2024-09-21T10:03:01.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: pidfile_write: ignore empty --pid-file 2024-09-21T10:03:01.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: load: jerasure load: lrc 2024-09-21T10:03:01.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: RocksDB version: 7.9.2 2024-09-21T10:03:01.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Git sha 0 2024-09-21T10:03:01.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Compile date 2024-09-20 20:32:28 2024-09-21T10:03:01.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: DB SUMMARY 2024-09-21T10:03:01.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: DB Session ID: Q9EDKJQ30F0QHSZ16EG5 2024-09-21T10:03:01.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: CURRENT file: CURRENT 2024-09-21T10:03:01.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T10:03:01.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: MANIFEST file: MANIFEST-000010 size: 1656 Bytes 2024-09-21T10:03:01.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000030.sst 2024-09-21T10:03:01.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000028.log size: 4133317 ; 2024-09-21T10:03:01.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.error_if_exists: 0 2024-09-21T10:03:01.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.create_if_missing: 0 2024-09-21T10:03:01.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.paranoid_checks: 1 2024-09-21T10:03:01.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T10:03:01.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T10:03:01.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T10:03:01.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.env: 0x55beeb630120 2024-09-21T10:03:01.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T10:03:01.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.info_log: 0x55beedb9b700 2024-09-21T10:03:01.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T10:03:01.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.statistics: (nil) 2024-09-21T10:03:01.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.use_fsync: 0 2024-09-21T10:03:01.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.max_log_file_size: 0 2024-09-21T10:03:01.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T10:03:01.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T10:03:01.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T10:03:01.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T10:03:01.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.allow_fallocate: 1 2024-09-21T10:03:01.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T10:03:01.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T10:03:01.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.use_direct_reads: 0 2024-09-21T10:03:01.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T10:03:01.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T10:03:01.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.db_log_dir: 2024-09-21T10:03:01.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.wal_dir: 2024-09-21T10:03:01.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T10:03:01.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T10:03:01.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T10:03:01.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T10:03:01.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T10:03:01.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T10:03:01.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T10:03:01.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T10:03:01.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.write_buffer_manager: 0x55beedb9e1e0 2024-09-21T10:03:01.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T10:03:01.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T10:03:01.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T10:03:01.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T10:03:01.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T10:03:01.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T10:03:01.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T10:03:01.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T10:03:01.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.unordered_write: 0 2024-09-21T10:03:01.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T10:03:01.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T10:03:01.598 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T10:03:01.598 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T10:03:01.598 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.row_cache: None 2024-09-21T10:03:01.598 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.wal_filter: None 2024-09-21T10:03:01.599 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T10:03:01.599 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T10:03:01.600 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.two_write_queues: 0 2024-09-21T10:03:01.600 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T10:03:01.600 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.wal_compression: 0 2024-09-21T10:03:01.600 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.atomic_flush: 0 2024-09-21T10:03:01.600 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T10:03:01.600 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T10:03:01.600 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T10:03:01.600 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.log_readahead_size: 0 2024-09-21T10:03:01.600 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T10:03:01.600 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T10:03:01.600 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T10:03:01.600 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T10:03:01.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T10:03:01.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T10:03:01.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T10:03:01.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.max_background_jobs: 2 2024-09-21T10:03:01.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.max_background_compactions: -1 2024-09-21T10:03:01.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.max_subcompactions: 1 2024-09-21T10:03:01.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T10:03:01.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T10:03:01.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T10:03:01.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T10:03:01.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T10:03:01.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T10:03:01.602 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T10:03:01.602 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T10:03:01.602 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.max_open_files: -1 2024-09-21T10:03:01.602 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T10:03:01.602 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T10:03:01.602 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T10:03:01.602 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T10:03:01.602 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.max_background_flushes: -1 2024-09-21T10:03:01.602 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Compression algorithms supported: 2024-09-21T10:03:01.602 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: kZSTD supported: 0 2024-09-21T10:03:01.602 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: kXpressCompression supported: 0 2024-09-21T10:03:01.603 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: kBZip2Compression supported: 0 2024-09-21T10:03:01.603 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T10:03:01.603 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: kLZ4Compression supported: 1 2024-09-21T10:03:01.603 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: kZlibCompression supported: 1 2024-09-21T10:03:01.603 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T10:03:01.603 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: kSnappyCompression supported: 1 2024-09-21T10:03:01.603 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T10:03:01.603 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T10:03:01.603 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000010 2024-09-21T10:03:01.603 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T10:03:01.603 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T10:03:01.603 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.merge_operator: 2024-09-21T10:03:01.604 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.compaction_filter: None 2024-09-21T10:03:01.604 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T10:03:01.604 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T10:03:01.604 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T10:03:01.604 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T10:03:01.604 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55beedb9b320) 2024-09-21T10:03:01.604 INFO:journalctl@ceph.mon.b.smithi160.stdout: cache_index_and_filter_blocks: 1 2024-09-21T10:03:01.604 INFO:journalctl@ceph.mon.b.smithi160.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T10:03:01.604 INFO:journalctl@ceph.mon.b.smithi160.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T10:03:01.604 INFO:journalctl@ceph.mon.b.smithi160.stdout: pin_top_level_index_and_filter: 1 2024-09-21T10:03:01.604 INFO:journalctl@ceph.mon.b.smithi160.stdout: index_type: 0 2024-09-21T10:03:01.604 INFO:journalctl@ceph.mon.b.smithi160.stdout: data_block_index_type: 0 2024-09-21T10:03:01.605 INFO:journalctl@ceph.mon.b.smithi160.stdout: index_shortening: 1 2024-09-21T10:03:01.605 INFO:journalctl@ceph.mon.b.smithi160.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T10:03:01.605 INFO:journalctl@ceph.mon.b.smithi160.stdout: checksum: 4 2024-09-21T10:03:01.605 INFO:journalctl@ceph.mon.b.smithi160.stdout: no_block_cache: 0 2024-09-21T10:03:01.605 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_cache: 0x55beedb8f610 2024-09-21T10:03:01.605 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_cache_name: BinnedLRUCache 2024-09-21T10:03:01.605 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_cache_options: 2024-09-21T10:03:01.605 INFO:journalctl@ceph.mon.b.smithi160.stdout: capacity : 536870912 2024-09-21T10:03:01.605 INFO:journalctl@ceph.mon.b.smithi160.stdout: num_shard_bits : 4 2024-09-21T10:03:01.605 INFO:journalctl@ceph.mon.b.smithi160.stdout: strict_capacity_limit : 0 2024-09-21T10:03:01.605 INFO:journalctl@ceph.mon.b.smithi160.stdout: high_pri_pool_ratio: 0.000 2024-09-21T10:03:01.605 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_cache_compressed: (nil) 2024-09-21T10:03:01.606 INFO:journalctl@ceph.mon.b.smithi160.stdout: persistent_cache: (nil) 2024-09-21T10:03:01.606 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_size: 4096 2024-09-21T10:03:01.606 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_size_deviation: 10 2024-09-21T10:03:01.606 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_restart_interval: 16 2024-09-21T10:03:01.606 INFO:journalctl@ceph.mon.b.smithi160.stdout: index_block_restart_interval: 1 2024-09-21T10:03:01.606 INFO:journalctl@ceph.mon.b.smithi160.stdout: metadata_block_size: 4096 2024-09-21T10:03:01.606 INFO:journalctl@ceph.mon.b.smithi160.stdout: partition_filters: 0 2024-09-21T10:03:01.606 INFO:journalctl@ceph.mon.b.smithi160.stdout: use_delta_encoding: 1 2024-09-21T10:03:01.606 INFO:journalctl@ceph.mon.b.smithi160.stdout: filter_policy: bloomfilter 2024-09-21T10:03:01.606 INFO:journalctl@ceph.mon.b.smithi160.stdout: whole_key_filtering: 1 2024-09-21T10:03:01.606 INFO:journalctl@ceph.mon.b.smithi160.stdout: verify_compression: 0 2024-09-21T10:03:01.606 INFO:journalctl@ceph.mon.b.smithi160.stdout: read_amp_bytes_per_bit: 0 2024-09-21T10:03:01.607 INFO:journalctl@ceph.mon.b.smithi160.stdout: format_version: 5 2024-09-21T10:03:01.607 INFO:journalctl@ceph.mon.b.smithi160.stdout: enable_index_compression: 1 2024-09-21T10:03:01.607 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_align: 0 2024-09-21T10:03:01.607 INFO:journalctl@ceph.mon.b.smithi160.stdout: max_auto_readahead_size: 262144 2024-09-21T10:03:01.607 INFO:journalctl@ceph.mon.b.smithi160.stdout: prepopulate_block_cache: 0 2024-09-21T10:03:01.607 INFO:journalctl@ceph.mon.b.smithi160.stdout: initial_auto_readahead_size: 8192 2024-09-21T10:03:01.607 INFO:journalctl@ceph.mon.b.smithi160.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T10:03:01.607 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T10:03:01.607 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T10:03:01.607 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.compression: NoCompression 2024-09-21T10:03:01.607 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T10:03:01.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T10:03:01.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T10:03:01.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.num_levels: 7 2024-09-21T10:03:01.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T10:03:01.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T10:03:01.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T10:03:01.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T10:03:01.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T10:03:01.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T10:03:01.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T10:03:01.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T10:03:01.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T10:03:01.609 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T10:03:01.609 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T10:03:01.609 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T10:03:01.609 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T10:03:01.609 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T10:03:01.609 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T10:03:01.609 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T10:03:01.609 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T10:03:01.609 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T10:03:01.609 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T10:03:01.609 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T10:03:01.609 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T10:03:01.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T10:03:01.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T10:03:01.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T10:03:01.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T10:03:01.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T10:03:01.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T10:03:01.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T10:03:01.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T10:03:01.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T10:03:01.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T10:03:01.610 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T10:03:01.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T10:03:01.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T10:03:01.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T10:03:01.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T10:03:01.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T10:03:01.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T10:03:01.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T10:03:01.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T10:03:01.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T10:03:01.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T10:03:01.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T10:03:01.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T10:03:01.611 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T10:03:01.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T10:03:01.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T10:03:01.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T10:03:01.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T10:03:01.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T10:03:01.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T10:03:01.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T10:03:01.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T10:03:01.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T10:03:01.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.inplace_update_support: 0 2024-09-21T10:03:01.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T10:03:01.612 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T10:03:01.613 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T10:03:01.613 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T10:03:01.613 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.bloom_locality: 0 2024-09-21T10:03:01.613 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.max_successive_merges: 0 2024-09-21T10:03:01.613 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T10:03:01.613 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T10:03:01.613 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T10:03:01.613 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T10:03:01.613 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.ttl: 2592000 2024-09-21T10:03:01.613 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T10:03:01.613 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T10:03:01.613 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T10:03:01.614 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.enable_blob_files: false 2024-09-21T10:03:01.614 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.min_blob_size: 0 2024-09-21T10:03:01.614 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T10:03:01.614 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T10:03:01.614 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T10:03:01.614 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T10:03:01.614 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T10:03:01.614 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T10:03:01.614 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T10:03:01.615 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T10:03:01.615 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-21T10:03:01.615 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 32, last_sequence is 19592, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 28 2024-09-21T10:03:01.615 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-09-21T10:03:01.615 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 46442bcf-7bad-4411-b6e2-50f5f9dc303c 2024-09-21T10:03:01.615 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912981264915, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-09-21T10:03:01.615 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-09-21T10:03:01.615 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912981275257, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 2693251, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 19598, "largest_seqno": 21670, "table_properties": {"data_size": 2683188, "index_size": 7097, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1925, "raw_key_size": 16913, "raw_average_key_size": 22, "raw_value_size": 2665080, "raw_average_value_size": 3520, "num_data_blocks": 308, "num_entries": 757, "num_filter_entries": 757, "num_deletions": 5, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726912981, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "46442bcf-7bad-4411-b6e2-50f5f9dc303c", "db_session_id": "Q9EDKJQ30F0QHSZ16EG5", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-09-21T10:03:01.615 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912981275368, "job": 1, "event": "recovery_finished"} 2024-09-21T10:03:01.615 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-09-21T10:03:01.615 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-21T10:03:01.616 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T10:03:01.616 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55beedbbee00 2024-09-21T10:03:01.616 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: DB pointer 0x55beedcc0000 2024-09-21T10:03:01.616 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T10:03:01.616 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T10:03:01.616 INFO:journalctl@ceph.mon.b.smithi160.stdout: ** DB Stats ** 2024-09-21T10:03:01.616 INFO:journalctl@ceph.mon.b.smithi160.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T10:03:01.616 INFO:journalctl@ceph.mon.b.smithi160.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T10:03:01.616 INFO:journalctl@ceph.mon.b.smithi160.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T10:03:01.616 INFO:journalctl@ceph.mon.b.smithi160.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T10:03:01.616 INFO:journalctl@ceph.mon.b.smithi160.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T10:03:01.617 INFO:journalctl@ceph.mon.b.smithi160.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T10:03:01.617 INFO:journalctl@ceph.mon.b.smithi160.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T10:03:01.617 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-21T10:03:01.617 INFO:journalctl@ceph.mon.b.smithi160.stdout: ** Compaction Stats [default] ** 2024-09-21T10:03:01.617 INFO:journalctl@ceph.mon.b.smithi160.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T10:03:01.617 INFO:journalctl@ceph.mon.b.smithi160.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T10:03:01.617 INFO:journalctl@ceph.mon.b.smithi160.stdout: L0 1/0 2.57 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 593.7 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-21T10:03:01.617 INFO:journalctl@ceph.mon.b.smithi160.stdout: L6 1/0 12.98 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-21T10:03:01.617 INFO:journalctl@ceph.mon.b.smithi160.stdout: Sum 2/0 15.55 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 593.7 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-21T10:03:01.617 INFO:journalctl@ceph.mon.b.smithi160.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 593.7 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-21T10:03:01.617 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-21T10:03:01.617 INFO:journalctl@ceph.mon.b.smithi160.stdout: ** Compaction Stats [default] ** 2024-09-21T10:03:01.618 INFO:journalctl@ceph.mon.b.smithi160.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T10:03:01.618 INFO:journalctl@ceph.mon.b.smithi160.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T10:03:01.618 INFO:journalctl@ceph.mon.b.smithi160.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 593.7 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-21T10:03:01.618 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-21T10:03:01.618 INFO:journalctl@ceph.mon.b.smithi160.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T10:03:01.618 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-21T10:03:01.618 INFO:journalctl@ceph.mon.b.smithi160.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T10:03:01.618 INFO:journalctl@ceph.mon.b.smithi160.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-09-21T10:03:01.618 INFO:journalctl@ceph.mon.b.smithi160.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T10:03:01.619 INFO:journalctl@ceph.mon.b.smithi160.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T10:03:01.619 INFO:journalctl@ceph.mon.b.smithi160.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T10:03:01.619 INFO:journalctl@ceph.mon.b.smithi160.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T10:03:01.619 INFO:journalctl@ceph.mon.b.smithi160.stdout: Cumulative compaction: 0.00 GB write, 176.37 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T10:03:01.619 INFO:journalctl@ceph.mon.b.smithi160.stdout: Interval compaction: 0.00 GB write, 176.37 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T10:03:01.619 INFO:journalctl@ceph.mon.b.smithi160.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T10:03:01.619 INFO:journalctl@ceph.mon.b.smithi160.stdout: Block cache BinnedLRUCache@0x55beedb8f610#7 capacity: 512.00 MB usage: 10.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-21T10:03:01.619 INFO:journalctl@ceph.mon.b.smithi160.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.03 KB,0.00038743%) IndexBlock(1,8.08 KB,0.00154078%) Misc(1,0.00 KB,0%) 2024-09-21T10:03:01.619 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-21T10:03:01.619 INFO:journalctl@ceph.mon.b.smithi160.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T10:03:01.619 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: starting mon.b rank 1 at public addrs [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] at bind addrs [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T10:03:01.619 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: mon.b@-1(???) e3 preinit fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T10:03:01.620 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: mon.b@-1(???).mds e9 new map 2024-09-21T10:03:01.620 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: mon.b@-1(???).mds e9 print_map 2024-09-21T10:03:01.620 INFO:journalctl@ceph.mon.b.smithi160.stdout: e9 2024-09-21T10:03:01.620 INFO:journalctl@ceph.mon.b.smithi160.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-21T10:03:01.620 INFO:journalctl@ceph.mon.b.smithi160.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T10:03:01.620 INFO:journalctl@ceph.mon.b.smithi160.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T10:03:01.620 INFO:journalctl@ceph.mon.b.smithi160.stdout: legacy client fscid: 1 2024-09-21T10:03:01.620 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-21T10:03:01.620 INFO:journalctl@ceph.mon.b.smithi160.stdout: Filesystem 'foo' (1) 2024-09-21T10:03:01.620 INFO:journalctl@ceph.mon.b.smithi160.stdout: fs_name foo 2024-09-21T10:03:01.620 INFO:journalctl@ceph.mon.b.smithi160.stdout: epoch 8 2024-09-21T10:03:01.621 INFO:journalctl@ceph.mon.b.smithi160.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-21T10:03:01.621 INFO:journalctl@ceph.mon.b.smithi160.stdout: created 2024-09-21T09:54:53.620516+0000 2024-09-21T10:03:01.621 INFO:journalctl@ceph.mon.b.smithi160.stdout: modified 2024-09-21T09:55:04.281129+0000 2024-09-21T10:03:01.621 INFO:journalctl@ceph.mon.b.smithi160.stdout: tableserver 0 2024-09-21T10:03:01.621 INFO:journalctl@ceph.mon.b.smithi160.stdout: root 0 2024-09-21T10:03:01.621 INFO:journalctl@ceph.mon.b.smithi160.stdout: session_timeout 60 2024-09-21T10:03:01.621 INFO:journalctl@ceph.mon.b.smithi160.stdout: session_autoclose 300 2024-09-21T10:03:01.621 INFO:journalctl@ceph.mon.b.smithi160.stdout: max_file_size 1099511627776 2024-09-21T10:03:01.621 INFO:journalctl@ceph.mon.b.smithi160.stdout: max_xattr_size 65536 2024-09-21T10:03:01.621 INFO:journalctl@ceph.mon.b.smithi160.stdout: required_client_features {} 2024-09-21T10:03:01.621 INFO:journalctl@ceph.mon.b.smithi160.stdout: last_failure 0 2024-09-21T10:03:01.621 INFO:journalctl@ceph.mon.b.smithi160.stdout: last_failure_osd_epoch 0 2024-09-21T10:03:01.622 INFO:journalctl@ceph.mon.b.smithi160.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T10:03:01.622 INFO:journalctl@ceph.mon.b.smithi160.stdout: max_mds 1 2024-09-21T10:03:01.622 INFO:journalctl@ceph.mon.b.smithi160.stdout: in 0 2024-09-21T10:03:01.622 INFO:journalctl@ceph.mon.b.smithi160.stdout: up {0=14652} 2024-09-21T10:03:01.622 INFO:journalctl@ceph.mon.b.smithi160.stdout: failed 2024-09-21T10:03:01.622 INFO:journalctl@ceph.mon.b.smithi160.stdout: damaged 2024-09-21T10:03:01.622 INFO:journalctl@ceph.mon.b.smithi160.stdout: stopped 2024-09-21T10:03:01.622 INFO:journalctl@ceph.mon.b.smithi160.stdout: data_pools [4] 2024-09-21T10:03:01.622 INFO:journalctl@ceph.mon.b.smithi160.stdout: metadata_pool 3 2024-09-21T10:03:01.622 INFO:journalctl@ceph.mon.b.smithi160.stdout: inline_data disabled 2024-09-21T10:03:01.622 INFO:journalctl@ceph.mon.b.smithi160.stdout: balancer 2024-09-21T10:03:01.623 INFO:journalctl@ceph.mon.b.smithi160.stdout: bal_rank_mask -1 2024-09-21T10:03:01.623 INFO:journalctl@ceph.mon.b.smithi160.stdout: standby_count_wanted 1 2024-09-21T10:03:01.623 INFO:journalctl@ceph.mon.b.smithi160.stdout: qdb_cluster leader: 0 members: 2024-09-21T10:03:01.623 INFO:journalctl@ceph.mon.b.smithi160.stdout: [mds.foo.smithi070.gansda{0:14652} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.70:6834/1361282039,v1:172.21.15.70:6835/1361282039] compat {c=[1],r=[1],i=[7ff]}] 2024-09-21T10:03:01.623 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-21T10:03:01.623 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-21T10:03:01.623 INFO:journalctl@ceph.mon.b.smithi160.stdout: Standby daemons: 2024-09-21T10:03:01.623 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-21T10:03:01.623 INFO:journalctl@ceph.mon.b.smithi160.stdout: [mds.foo.smithi160.byyhfz{-1:24457} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.160:6832/2992374343,v1:172.21.15.160:6833/2992374343] compat {c=[1],r=[1],i=[7ff]}] 2024-09-21T10:03:01.623 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: mon.b@-1(???).osd e460 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-21T10:03:01.623 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: mon.b@-1(???).osd e460 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T10:03:01.624 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: mon.b@-1(???).osd e460 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T10:03:01.624 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: mon.b@-1(???).osd e460 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T10:03:01.624 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 ceph-mon[101879]: mon.b@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2024-09-21T10:03:01.624 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 bash[101841]: 86e88af21f363c6d8bfd177fd8ebb63fb182a6bd6d5acfa40ef9d00474af8514 2024-09-21T10:03:01.624 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:01 smithi160 systemd[1]: Started Ceph mon.b for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:03:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:02 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/a51fd572-868d-4ed8-9509-5d4e09431914"} : dispatch 2024-09-21T10:03:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:02 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/a51fd572-868d-4ed8-9509-5d4e09431914"}]': finished 2024-09-21T10:03:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:02 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T10:03:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:02 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T10:03:03.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:02 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T10:03:03.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:02 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/a51fd572-868d-4ed8-9509-5d4e09431914"} : dispatch 2024-09-21T10:03:03.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:02 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/a51fd572-868d-4ed8-9509-5d4e09431914"}]': finished 2024-09-21T10:03:03.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:02 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T10:03:03.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:02 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T10:03:03.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:02 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T10:03:03.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:02 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/a51fd572-868d-4ed8-9509-5d4e09431914"} : dispatch 2024-09-21T10:03:03.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:02 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/a51fd572-868d-4ed8-9509-5d4e09431914"}]': finished 2024-09-21T10:03:03.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:02 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T10:03:03.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:02 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T10:03:03.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:02 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T10:03:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:03:03] "GET /metrics HTTP/1.1" 200 36264 "" "Prometheus/2.51.0" 2024-09-21T10:03:04.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: fsmap foo:1 {0=foo.smithi070.gansda=up:active} 1 up:standby 2024-09-21T10:03:04.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: osdmap e460: 8 total, 8 up, 8 in 2024-09-21T10:03:04.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: mgrmap e40: y(active, since 2m), standbys: x 2024-09-21T10:03:04.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: overall HEALTH_OK 2024-09-21T10:03:04.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: mon.b calling monitor election 2024-09-21T10:03:04.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: mon.c calling monitor election 2024-09-21T10:03:04.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T10:03:04.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T10:03:04.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T10:03:04.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: mon.a calling monitor election 2024-09-21T10:03:04.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T10:03:04.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: monmap epoch 4 2024-09-21T10:03:04.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T10:03:04.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: last_changed 2024-09-21T10:03:02.725320+0000 2024-09-21T10:03:04.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: created 2024-09-21T09:47:56.277550+0000 2024-09-21T10:03:04.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: min_mon_release 19 (squid) 2024-09-21T10:03:04.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: election_strategy: 1 2024-09-21T10:03:04.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-21T10:03:04.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.b 2024-09-21T10:03:04.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: 2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-21T10:03:04.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: fsmap foo:1 {0=foo.smithi070.gansda=up:active} 1 up:standby 2024-09-21T10:03:04.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: osdmap e460: 8 total, 8 up, 8 in 2024-09-21T10:03:04.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: mgrmap e40: y(active, since 2m), standbys: x 2024-09-21T10:03:04.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: overall HEALTH_OK 2024-09-21T10:03:04.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:04.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:04.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:04.094 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:03 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:03:04.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: fsmap foo:1 {0=foo.smithi070.gansda=up:active} 1 up:standby 2024-09-21T10:03:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: osdmap e460: 8 total, 8 up, 8 in 2024-09-21T10:03:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: mgrmap e40: y(active, since 2m), standbys: x 2024-09-21T10:03:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: overall HEALTH_OK 2024-09-21T10:03:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: mon.b calling monitor election 2024-09-21T10:03:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: mon.c calling monitor election 2024-09-21T10:03:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T10:03:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T10:03:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T10:03:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: mon.a calling monitor election 2024-09-21T10:03:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T10:03:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: monmap epoch 4 2024-09-21T10:03:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T10:03:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: last_changed 2024-09-21T10:03:02.725320+0000 2024-09-21T10:03:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: created 2024-09-21T09:47:56.277550+0000 2024-09-21T10:03:04.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: min_mon_release 19 (squid) 2024-09-21T10:03:04.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: election_strategy: 1 2024-09-21T10:03:04.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-21T10:03:04.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.b 2024-09-21T10:03:04.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: 2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-21T10:03:04.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: fsmap foo:1 {0=foo.smithi070.gansda=up:active} 1 up:standby 2024-09-21T10:03:04.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: osdmap e460: 8 total, 8 up, 8 in 2024-09-21T10:03:04.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: mgrmap e40: y(active, since 2m), standbys: x 2024-09-21T10:03:04.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: overall HEALTH_OK 2024-09-21T10:03:04.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:04.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:04.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:04.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:03:04.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: fsmap foo:1 {0=foo.smithi070.gansda=up:active} 1 up:standby 2024-09-21T10:03:04.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: osdmap e460: 8 total, 8 up, 8 in 2024-09-21T10:03:04.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: mgrmap e40: y(active, since 2m), standbys: x 2024-09-21T10:03:04.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: overall HEALTH_OK 2024-09-21T10:03:04.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: mon.b calling monitor election 2024-09-21T10:03:04.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: mon.c calling monitor election 2024-09-21T10:03:04.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T10:03:04.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T10:03:04.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T10:03:04.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: mon.a calling monitor election 2024-09-21T10:03:04.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T10:03:04.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: monmap epoch 4 2024-09-21T10:03:04.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T10:03:04.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: last_changed 2024-09-21T10:03:02.725320+0000 2024-09-21T10:03:04.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: created 2024-09-21T09:47:56.277550+0000 2024-09-21T10:03:04.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: min_mon_release 19 (squid) 2024-09-21T10:03:04.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: election_strategy: 1 2024-09-21T10:03:04.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-21T10:03:04.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.b 2024-09-21T10:03:04.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: 2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-21T10:03:04.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: fsmap foo:1 {0=foo.smithi070.gansda=up:active} 1 up:standby 2024-09-21T10:03:04.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: osdmap e460: 8 total, 8 up, 8 in 2024-09-21T10:03:04.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: mgrmap e40: y(active, since 2m), standbys: x 2024-09-21T10:03:04.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: overall HEALTH_OK 2024-09-21T10:03:04.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:04.132 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:04.132 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:04.132 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:03 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:03:05.069 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:04 smithi160 ceph-mon[101879]: osdmap e461: 8 total, 8 up, 8 in 2024-09-21T10:03:05.069 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:04 smithi160 ceph-mon[101879]: pgmap v187: 105 pgs: 105 active+clean; 583 KiB data, 473 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 3.3 KiB/s wr, 51 op/s 2024-09-21T10:03:05.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:04 smithi070 ceph-mon[112139]: osdmap e461: 8 total, 8 up, 8 in 2024-09-21T10:03:05.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:04 smithi070 ceph-mon[112139]: pgmap v187: 105 pgs: 105 active+clean; 583 KiB data, 473 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 3.3 KiB/s wr, 51 op/s 2024-09-21T10:03:05.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:04 smithi070 ceph-mon[107385]: osdmap e461: 8 total, 8 up, 8 in 2024-09-21T10:03:05.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:04 smithi070 ceph-mon[107385]: pgmap v187: 105 pgs: 105 active+clean; 583 KiB data, 473 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 3.3 KiB/s wr, 51 op/s 2024-09-21T10:03:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:05 smithi160 ceph-mon[101879]: osdmap e462: 8 total, 8 up, 8 in 2024-09-21T10:03:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:06.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:05 smithi070 ceph-mon[107385]: osdmap e462: 8 total, 8 up, 8 in 2024-09-21T10:03:06.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:06.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:05 smithi070 ceph-mon[112139]: osdmap e462: 8 total, 8 up, 8 in 2024-09-21T10:03:06.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:06.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:03:06 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T10:03:06.288+0000 7fd088d1d640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-21T10:03:06.838 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.SnapshotRemoveOnDisable (8109 ms) 2024-09-21T10:03:06.838 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.SnapshotUnlinkPeer 2024-09-21T10:03:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:06 smithi160 ceph-mon[101879]: osdmap e463: 8 total, 8 up, 8 in 2024-09-21T10:03:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:06 smithi160 ceph-mon[101879]: pgmap v190: 105 pgs: 105 active+clean; 583 KiB data, 473 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 7.0 KiB/s wr, 82 op/s 2024-09-21T10:03:07.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:06 smithi070 ceph-mon[107385]: osdmap e463: 8 total, 8 up, 8 in 2024-09-21T10:03:07.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:06 smithi070 ceph-mon[107385]: pgmap v190: 105 pgs: 105 active+clean; 583 KiB data, 473 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 7.0 KiB/s wr, 82 op/s 2024-09-21T10:03:07.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:06 smithi070 ceph-mon[112139]: osdmap e463: 8 total, 8 up, 8 in 2024-09-21T10:03:07.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:06 smithi070 ceph-mon[112139]: pgmap v190: 105 pgs: 105 active+clean; 583 KiB data, 473 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 7.0 KiB/s wr, 82 op/s 2024-09-21T10:03:08.013 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:07 smithi160 ceph-mon[101879]: osdmap e464: 8 total, 8 up, 8 in 2024-09-21T10:03:08.013 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:07 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/941259f0-116f-4ce8-8251-a942d540f938"} : dispatch 2024-09-21T10:03:08.013 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:07 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/941259f0-116f-4ce8-8251-a942d540f938"}]': finished 2024-09-21T10:03:08.013 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:08.014 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:08.014 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:08.014 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:08.014 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:07 smithi160 ceph-mon[101879]: osdmap e465: 8 total, 8 up, 8 in 2024-09-21T10:03:08.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:07 smithi070 ceph-mon[107385]: osdmap e464: 8 total, 8 up, 8 in 2024-09-21T10:03:08.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:07 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/941259f0-116f-4ce8-8251-a942d540f938"} : dispatch 2024-09-21T10:03:08.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:07 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/941259f0-116f-4ce8-8251-a942d540f938"}]': finished 2024-09-21T10:03:08.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:08.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:08.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:08.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:08.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:07 smithi070 ceph-mon[107385]: osdmap e465: 8 total, 8 up, 8 in 2024-09-21T10:03:08.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:07 smithi070 ceph-mon[112139]: osdmap e464: 8 total, 8 up, 8 in 2024-09-21T10:03:08.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:07 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/941259f0-116f-4ce8-8251-a942d540f938"} : dispatch 2024-09-21T10:03:08.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:07 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/941259f0-116f-4ce8-8251-a942d540f938"}]': finished 2024-09-21T10:03:08.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:08.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:08.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:08.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:08.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:07 smithi070 ceph-mon[112139]: osdmap e465: 8 total, 8 up, 8 in 2024-09-21T10:03:08.926 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:08 smithi160 ceph-mon[101879]: pgmap v193: 105 pgs: 105 active+clean; 583 KiB data, 473 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 5.2 KiB/s wr, 43 op/s 2024-09-21T10:03:08.927 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:08 smithi160 ceph-mon[101879]: osdmap e466: 8 total, 8 up, 8 in 2024-09-21T10:03:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:08 smithi070 ceph-mon[107385]: pgmap v193: 105 pgs: 105 active+clean; 583 KiB data, 473 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 5.2 KiB/s wr, 43 op/s 2024-09-21T10:03:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:08 smithi070 ceph-mon[107385]: osdmap e466: 8 total, 8 up, 8 in 2024-09-21T10:03:09.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:08 smithi070 ceph-mon[112139]: pgmap v193: 105 pgs: 105 active+clean; 583 KiB data, 473 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 5.2 KiB/s wr, 43 op/s 2024-09-21T10:03:09.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:08 smithi070 ceph-mon[112139]: osdmap e466: 8 total, 8 up, 8 in 2024-09-21T10:03:10.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:10 smithi160 ceph-mon[101879]: osdmap e467: 8 total, 8 up, 8 in 2024-09-21T10:03:10.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:10 smithi160 ceph-mon[101879]: pgmap v196: 105 pgs: 105 active+clean; 583 KiB data, 473 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:03:10.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:10 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:10.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:10 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:10.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:10 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:10.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:10 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:03:10.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:10 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:10.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:10 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T10:03:10.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:10 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T10:03:10.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:10 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:10.929 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:10 smithi070 ceph-mon[107385]: osdmap e467: 8 total, 8 up, 8 in 2024-09-21T10:03:10.929 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:10 smithi070 ceph-mon[107385]: pgmap v196: 105 pgs: 105 active+clean; 583 KiB data, 473 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:03:10.929 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:10 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:10.929 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:10 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:10.929 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:10 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:10.929 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:10 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:03:10.929 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:10 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:10.929 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:10 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T10:03:10.930 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:10 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T10:03:10.930 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:10 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:10.930 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:10 smithi070 ceph-mon[112139]: osdmap e467: 8 total, 8 up, 8 in 2024-09-21T10:03:10.930 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:10 smithi070 ceph-mon[112139]: pgmap v196: 105 pgs: 105 active+clean; 583 KiB data, 473 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:03:10.930 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:10 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:10.930 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:10 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:10.930 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:10 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:10.930 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:10 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:03:10.930 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:10 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:10.930 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:10 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T10:03:10.930 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:10 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T10:03:10.931 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:10 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:11.862 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:11 smithi160 ceph-mon[101879]: Reconfiguring mon.a (monmap changed)... 2024-09-21T10:03:11.862 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:11 smithi160 ceph-mon[101879]: Reconfiguring daemon mon.a on smithi070 2024-09-21T10:03:11.862 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:11 smithi160 ceph-mon[101879]: osdmap e468: 8 total, 8 up, 8 in 2024-09-21T10:03:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:11 smithi070 ceph-mon[107385]: Reconfiguring mon.a (monmap changed)... 2024-09-21T10:03:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:11 smithi070 ceph-mon[107385]: Reconfiguring daemon mon.a on smithi070 2024-09-21T10:03:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:11 smithi070 ceph-mon[107385]: osdmap e468: 8 total, 8 up, 8 in 2024-09-21T10:03:12.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:11 smithi070 ceph-mon[112139]: Reconfiguring mon.a (monmap changed)... 2024-09-21T10:03:12.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:11 smithi070 ceph-mon[112139]: Reconfiguring daemon mon.a on smithi070 2024-09-21T10:03:12.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:11 smithi070 ceph-mon[112139]: osdmap e468: 8 total, 8 up, 8 in 2024-09-21T10:03:12.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:12 smithi070 ceph-mon[107385]: osdmap e469: 8 total, 8 up, 8 in 2024-09-21T10:03:12.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:12 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/665e5415-6e2a-4fc2-b55c-981f833eeae3"} : dispatch 2024-09-21T10:03:12.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:12 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/665e5415-6e2a-4fc2-b55c-981f833eeae3"}]': finished 2024-09-21T10:03:12.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:12 smithi070 ceph-mon[107385]: pgmap v199: 105 pgs: 105 active+clean; 583 KiB data, 458 MiB used, 715 GiB / 715 GiB avail; 116 KiB/s rd, 11 KiB/s wr, 158 op/s 2024-09-21T10:03:12.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:12 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:12.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:12 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:12.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:12 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T10:03:12.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:12 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T10:03:12.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:12 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:12.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:12 smithi070 ceph-mon[112139]: osdmap e469: 8 total, 8 up, 8 in 2024-09-21T10:03:12.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:12 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/665e5415-6e2a-4fc2-b55c-981f833eeae3"} : dispatch 2024-09-21T10:03:12.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:12 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/665e5415-6e2a-4fc2-b55c-981f833eeae3"}]': finished 2024-09-21T10:03:12.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:12 smithi070 ceph-mon[112139]: pgmap v199: 105 pgs: 105 active+clean; 583 KiB data, 458 MiB used, 715 GiB / 715 GiB avail; 116 KiB/s rd, 11 KiB/s wr, 158 op/s 2024-09-21T10:03:12.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:12 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:12.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:12 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:12.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:12 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T10:03:12.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:12 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T10:03:12.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:12 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:13.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:12 smithi160 ceph-mon[101879]: osdmap e469: 8 total, 8 up, 8 in 2024-09-21T10:03:13.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:12 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/665e5415-6e2a-4fc2-b55c-981f833eeae3"} : dispatch 2024-09-21T10:03:13.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:12 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/665e5415-6e2a-4fc2-b55c-981f833eeae3"}]': finished 2024-09-21T10:03:13.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:12 smithi160 ceph-mon[101879]: pgmap v199: 105 pgs: 105 active+clean; 583 KiB data, 458 MiB used, 715 GiB / 715 GiB avail; 116 KiB/s rd, 11 KiB/s wr, 158 op/s 2024-09-21T10:03:13.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:12 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:13.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:12 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:13.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:12 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T10:03:13.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:12 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T10:03:13.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:12 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:13.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:03:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:03:13] "GET /metrics HTTP/1.1" 200 36207 "" "Prometheus/2.51.0" 2024-09-21T10:03:14.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:13 smithi070 ceph-mon[107385]: Reconfiguring mgr.y (monmap changed)... 2024-09-21T10:03:14.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:13 smithi070 ceph-mon[107385]: Reconfiguring daemon mgr.y on smithi070 2024-09-21T10:03:14.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:13 smithi070 ceph-mon[107385]: osdmap e470: 8 total, 8 up, 8 in 2024-09-21T10:03:14.045 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:13 smithi070 ceph-mon[112139]: Reconfiguring mgr.y (monmap changed)... 2024-09-21T10:03:14.045 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:13 smithi070 ceph-mon[112139]: Reconfiguring daemon mgr.y on smithi070 2024-09-21T10:03:14.046 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:13 smithi070 ceph-mon[112139]: osdmap e470: 8 total, 8 up, 8 in 2024-09-21T10:03:14.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:13 smithi160 ceph-mon[101879]: Reconfiguring mgr.y (monmap changed)... 2024-09-21T10:03:14.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:13 smithi160 ceph-mon[101879]: Reconfiguring daemon mgr.y on smithi070 2024-09-21T10:03:14.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:13 smithi160 ceph-mon[101879]: osdmap e470: 8 total, 8 up, 8 in 2024-09-21T10:03:15.036 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:14 smithi070 ceph-mon[107385]: osdmap e471: 8 total, 8 up, 8 in 2024-09-21T10:03:15.036 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:14 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:15.036 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:14 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:15.037 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:14 smithi070 ceph-mon[107385]: Reconfiguring mon.c (monmap changed)... 2024-09-21T10:03:15.037 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:14 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T10:03:15.037 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:14 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T10:03:15.037 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:14 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:15.037 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:14 smithi070 ceph-mon[107385]: Reconfiguring daemon mon.c on smithi070 2024-09-21T10:03:15.037 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:14 smithi070 ceph-mon[107385]: pgmap v202: 105 pgs: 105 active+clean; 583 KiB data, 458 MiB used, 715 GiB / 715 GiB avail; 116 KiB/s rd, 11 KiB/s wr, 158 op/s 2024-09-21T10:03:15.037 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:14 smithi070 ceph-mon[112139]: osdmap e471: 8 total, 8 up, 8 in 2024-09-21T10:03:15.037 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:14 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:15.037 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:14 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:15.037 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:14 smithi070 ceph-mon[112139]: Reconfiguring mon.c (monmap changed)... 2024-09-21T10:03:15.037 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:14 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T10:03:15.038 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:14 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T10:03:15.038 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:14 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:15.038 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:14 smithi070 ceph-mon[112139]: Reconfiguring daemon mon.c on smithi070 2024-09-21T10:03:15.038 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:14 smithi070 ceph-mon[112139]: pgmap v202: 105 pgs: 105 active+clean; 583 KiB data, 458 MiB used, 715 GiB / 715 GiB avail; 116 KiB/s rd, 11 KiB/s wr, 158 op/s 2024-09-21T10:03:15.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:14 smithi160 ceph-mon[101879]: osdmap e471: 8 total, 8 up, 8 in 2024-09-21T10:03:15.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:14 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:15.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:14 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:15.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:14 smithi160 ceph-mon[101879]: Reconfiguring mon.c (monmap changed)... 2024-09-21T10:03:15.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:14 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T10:03:15.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:14 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T10:03:15.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:14 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:15.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:14 smithi160 ceph-mon[101879]: Reconfiguring daemon mon.c on smithi070 2024-09-21T10:03:15.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:14 smithi160 ceph-mon[101879]: pgmap v202: 105 pgs: 105 active+clean; 583 KiB data, 458 MiB used, 715 GiB / 715 GiB avail; 116 KiB/s rd, 11 KiB/s wr, 158 op/s 2024-09-21T10:03:16.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:15 smithi160 ceph-mon[101879]: osdmap e472: 8 total, 8 up, 8 in 2024-09-21T10:03:16.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:15 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:16.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:15 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:16.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:15 smithi160 ceph-mon[101879]: Reconfiguring osd.0 (monmap changed)... 2024-09-21T10:03:16.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:15 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T10:03:16.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:15 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:16.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:15 smithi160 ceph-mon[101879]: Reconfiguring daemon osd.0 on smithi070 2024-09-21T10:03:16.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:15 smithi070 ceph-mon[107385]: osdmap e472: 8 total, 8 up, 8 in 2024-09-21T10:03:16.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:15 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:16.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:15 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:16.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:15 smithi070 ceph-mon[107385]: Reconfiguring osd.0 (monmap changed)... 2024-09-21T10:03:16.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:15 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T10:03:16.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:15 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:16.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:15 smithi070 ceph-mon[107385]: Reconfiguring daemon osd.0 on smithi070 2024-09-21T10:03:16.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:15 smithi070 ceph-mon[112139]: osdmap e472: 8 total, 8 up, 8 in 2024-09-21T10:03:16.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:15 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:16.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:15 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:16.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:15 smithi070 ceph-mon[112139]: Reconfiguring osd.0 (monmap changed)... 2024-09-21T10:03:16.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:15 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T10:03:16.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:15 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:16.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:15 smithi070 ceph-mon[112139]: Reconfiguring daemon osd.0 on smithi070 2024-09-21T10:03:16.791 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.SnapshotUnlinkPeer (9952 ms) 2024-09-21T10:03:16.791 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.SnapshotImageState 2024-09-21T10:03:16.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:16 smithi070 ceph-mon[107385]: osdmap e473: 8 total, 8 up, 8 in 2024-09-21T10:03:16.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:16 smithi070 ceph-mon[107385]: pgmap v205: 105 pgs: 105 active+clean; 583 KiB data, 459 MiB used, 715 GiB / 715 GiB avail; 161 KiB/s rd, 9.0 KiB/s wr, 215 op/s 2024-09-21T10:03:16.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:16.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:16.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T10:03:16.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:16.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:16 smithi070 ceph-mon[112139]: osdmap e473: 8 total, 8 up, 8 in 2024-09-21T10:03:16.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:16 smithi070 ceph-mon[112139]: pgmap v205: 105 pgs: 105 active+clean; 583 KiB data, 459 MiB used, 715 GiB / 715 GiB avail; 161 KiB/s rd, 9.0 KiB/s wr, 215 op/s 2024-09-21T10:03:16.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:16.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:16.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T10:03:16.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:17.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:16 smithi160 ceph-mon[101879]: osdmap e473: 8 total, 8 up, 8 in 2024-09-21T10:03:17.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:16 smithi160 ceph-mon[101879]: pgmap v205: 105 pgs: 105 active+clean; 583 KiB data, 459 MiB used, 715 GiB / 715 GiB avail; 161 KiB/s rd, 9.0 KiB/s wr, 215 op/s 2024-09-21T10:03:17.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:17.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:17.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T10:03:17.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:18.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:17 smithi160 ceph-mon[101879]: Reconfiguring osd.1 (monmap changed)... 2024-09-21T10:03:18.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:17 smithi160 ceph-mon[101879]: Reconfiguring daemon osd.1 on smithi070 2024-09-21T10:03:18.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:17 smithi160 ceph-mon[101879]: osdmap e474: 8 total, 8 up, 8 in 2024-09-21T10:03:18.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:17 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/6b1287b5-9f41-4140-b72a-94039429e47a"} : dispatch 2024-09-21T10:03:18.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:17 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/6b1287b5-9f41-4140-b72a-94039429e47a"}]': finished 2024-09-21T10:03:18.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:17 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/fbcfcca8-6db3-4e87-862d-401e753d4519"} : dispatch 2024-09-21T10:03:18.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:17 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/fbcfcca8-6db3-4e87-862d-401e753d4519"}]': finished 2024-09-21T10:03:18.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:17 smithi160 ceph-mon[101879]: osdmap e475: 8 total, 8 up, 8 in 2024-09-21T10:03:18.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:17 smithi070 ceph-mon[112139]: Reconfiguring osd.1 (monmap changed)... 2024-09-21T10:03:18.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:17 smithi070 ceph-mon[112139]: Reconfiguring daemon osd.1 on smithi070 2024-09-21T10:03:18.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:17 smithi070 ceph-mon[112139]: osdmap e474: 8 total, 8 up, 8 in 2024-09-21T10:03:18.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:17 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/6b1287b5-9f41-4140-b72a-94039429e47a"} : dispatch 2024-09-21T10:03:18.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:17 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/6b1287b5-9f41-4140-b72a-94039429e47a"}]': finished 2024-09-21T10:03:18.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:17 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/fbcfcca8-6db3-4e87-862d-401e753d4519"} : dispatch 2024-09-21T10:03:18.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:17 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/fbcfcca8-6db3-4e87-862d-401e753d4519"}]': finished 2024-09-21T10:03:18.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:17 smithi070 ceph-mon[112139]: osdmap e475: 8 total, 8 up, 8 in 2024-09-21T10:03:18.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:17 smithi070 ceph-mon[107385]: Reconfiguring osd.1 (monmap changed)... 2024-09-21T10:03:18.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:17 smithi070 ceph-mon[107385]: Reconfiguring daemon osd.1 on smithi070 2024-09-21T10:03:18.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:17 smithi070 ceph-mon[107385]: osdmap e474: 8 total, 8 up, 8 in 2024-09-21T10:03:18.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:17 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/6b1287b5-9f41-4140-b72a-94039429e47a"} : dispatch 2024-09-21T10:03:18.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:17 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/6b1287b5-9f41-4140-b72a-94039429e47a"}]': finished 2024-09-21T10:03:18.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:17 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/fbcfcca8-6db3-4e87-862d-401e753d4519"} : dispatch 2024-09-21T10:03:18.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:17 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/fbcfcca8-6db3-4e87-862d-401e753d4519"}]': finished 2024-09-21T10:03:18.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:17 smithi070 ceph-mon[107385]: osdmap e475: 8 total, 8 up, 8 in 2024-09-21T10:03:18.836 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:18 smithi070 ceph-mon[107385]: pgmap v208: 105 pgs: 105 active+clean; 583 KiB data, 459 MiB used, 715 GiB / 715 GiB avail; 161 KiB/s rd, 9.0 KiB/s wr, 215 op/s 2024-09-21T10:03:18.836 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:18 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:18.837 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:18 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:18.837 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:18 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T10:03:18.837 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:18 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:18.837 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:18 smithi070 ceph-mon[107385]: osdmap e476: 8 total, 8 up, 8 in 2024-09-21T10:03:18.837 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:18 smithi070 ceph-mon[112139]: pgmap v208: 105 pgs: 105 active+clean; 583 KiB data, 459 MiB used, 715 GiB / 715 GiB avail; 161 KiB/s rd, 9.0 KiB/s wr, 215 op/s 2024-09-21T10:03:18.837 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:18 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:18.837 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:18 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:18.837 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:18 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T10:03:18.837 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:18 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:18.837 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:18 smithi070 ceph-mon[112139]: osdmap e476: 8 total, 8 up, 8 in 2024-09-21T10:03:19.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:18 smithi160 ceph-mon[101879]: pgmap v208: 105 pgs: 105 active+clean; 583 KiB data, 459 MiB used, 715 GiB / 715 GiB avail; 161 KiB/s rd, 9.0 KiB/s wr, 215 op/s 2024-09-21T10:03:19.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:18 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:19.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:18 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:19.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:18 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T10:03:19.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:18 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:19.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:18 smithi160 ceph-mon[101879]: osdmap e476: 8 total, 8 up, 8 in 2024-09-21T10:03:20.066 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:19 smithi070 ceph-mon[107385]: Reconfiguring osd.2 (monmap changed)... 2024-09-21T10:03:20.066 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:19 smithi070 ceph-mon[107385]: Reconfiguring daemon osd.2 on smithi070 2024-09-21T10:03:20.067 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:19 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:20.067 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:19 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:20.067 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:19 smithi070 ceph-mon[112139]: Reconfiguring osd.2 (monmap changed)... 2024-09-21T10:03:20.067 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:19 smithi070 ceph-mon[112139]: Reconfiguring daemon osd.2 on smithi070 2024-09-21T10:03:20.067 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:19 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:20.067 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:19 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:20.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:19 smithi160 ceph-mon[101879]: Reconfiguring osd.2 (monmap changed)... 2024-09-21T10:03:20.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:19 smithi160 ceph-mon[101879]: Reconfiguring daemon osd.2 on smithi070 2024-09-21T10:03:20.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:19 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:20.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:19 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:20.787 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.SnapshotImageState (3997 ms) 2024-09-21T10:03:20.787 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.SnapshotPromoteDemote 2024-09-21T10:03:21.064 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:20 smithi070 ceph-mon[107385]: Reconfiguring osd.3 (monmap changed)... 2024-09-21T10:03:21.064 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T10:03:21.064 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:21.064 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:20 smithi070 ceph-mon[107385]: Reconfiguring daemon osd.3 on smithi070 2024-09-21T10:03:21.064 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:20 smithi070 ceph-mon[107385]: osdmap e477: 8 total, 8 up, 8 in 2024-09-21T10:03:21.065 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:20 smithi070 ceph-mon[107385]: pgmap v211: 105 pgs: 105 active+clean; 583 KiB data, 459 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:03:21.065 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:21.065 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:20 smithi070 ceph-mon[112139]: Reconfiguring osd.3 (monmap changed)... 2024-09-21T10:03:21.065 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T10:03:21.065 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:21.065 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:20 smithi070 ceph-mon[112139]: Reconfiguring daemon osd.3 on smithi070 2024-09-21T10:03:21.065 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:20 smithi070 ceph-mon[112139]: osdmap e477: 8 total, 8 up, 8 in 2024-09-21T10:03:21.065 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:20 smithi070 ceph-mon[112139]: pgmap v211: 105 pgs: 105 active+clean; 583 KiB data, 459 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:03:21.065 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:21.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:20 smithi160 ceph-mon[101879]: Reconfiguring osd.3 (monmap changed)... 2024-09-21T10:03:21.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T10:03:21.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:21.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:20 smithi160 ceph-mon[101879]: Reconfiguring daemon osd.3 on smithi070 2024-09-21T10:03:21.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:20 smithi160 ceph-mon[101879]: osdmap e477: 8 total, 8 up, 8 in 2024-09-21T10:03:21.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:20 smithi160 ceph-mon[101879]: pgmap v211: 105 pgs: 105 active+clean; 583 KiB data, 459 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:03:21.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:22.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:21 smithi160 ceph-mon[101879]: osdmap e478: 8 total, 8 up, 8 in 2024-09-21T10:03:22.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:22.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:22.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi070.gansda", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T10:03:22.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:22.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:21 smithi070 ceph-mon[112139]: osdmap e478: 8 total, 8 up, 8 in 2024-09-21T10:03:22.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:22.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:22.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi070.gansda", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T10:03:22.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:22.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:21 smithi070 ceph-mon[107385]: osdmap e478: 8 total, 8 up, 8 in 2024-09-21T10:03:22.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:22.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:22.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi070.gansda", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T10:03:22.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:22.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:22 smithi070 ceph-mon[107385]: Reconfiguring mds.foo.smithi070.gansda (monmap changed)... 2024-09-21T10:03:22.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:22 smithi070 ceph-mon[107385]: Reconfiguring daemon mds.foo.smithi070.gansda on smithi070 2024-09-21T10:03:22.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:22 smithi070 ceph-mon[107385]: osdmap e479: 8 total, 8 up, 8 in 2024-09-21T10:03:22.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:22 smithi070 ceph-mon[107385]: pgmap v214: 105 pgs: 105 active+clean; 583 KiB data, 463 MiB used, 715 GiB / 715 GiB avail; 171 KiB/s rd, 26 KiB/s wr, 250 op/s 2024-09-21T10:03:22.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:22 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:22.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:22 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:22.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:22 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T10:03:22.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:22 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T10:03:22.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:22 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:22.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:22 smithi070 ceph-mon[112139]: Reconfiguring mds.foo.smithi070.gansda (monmap changed)... 2024-09-21T10:03:22.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:22 smithi070 ceph-mon[112139]: Reconfiguring daemon mds.foo.smithi070.gansda on smithi070 2024-09-21T10:03:22.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:22 smithi070 ceph-mon[112139]: osdmap e479: 8 total, 8 up, 8 in 2024-09-21T10:03:22.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:22 smithi070 ceph-mon[112139]: pgmap v214: 105 pgs: 105 active+clean; 583 KiB data, 463 MiB used, 715 GiB / 715 GiB avail; 171 KiB/s rd, 26 KiB/s wr, 250 op/s 2024-09-21T10:03:22.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:22 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:22.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:22 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:22.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:22 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T10:03:22.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:22 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T10:03:22.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:22 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:23.034 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:22 smithi160 ceph-mon[101879]: Reconfiguring mds.foo.smithi070.gansda (monmap changed)... 2024-09-21T10:03:23.035 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:22 smithi160 ceph-mon[101879]: Reconfiguring daemon mds.foo.smithi070.gansda on smithi070 2024-09-21T10:03:23.035 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:22 smithi160 ceph-mon[101879]: osdmap e479: 8 total, 8 up, 8 in 2024-09-21T10:03:23.035 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:22 smithi160 ceph-mon[101879]: pgmap v214: 105 pgs: 105 active+clean; 583 KiB data, 463 MiB used, 715 GiB / 715 GiB avail; 171 KiB/s rd, 26 KiB/s wr, 250 op/s 2024-09-21T10:03:23.035 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:22 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:23.035 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:22 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:23.035 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:22 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T10:03:23.035 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:22 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T10:03:23.035 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:22 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:23.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:03:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:03:23] "GET /metrics HTTP/1.1" 200 36273 "" "Prometheus/2.51.0" 2024-09-21T10:03:24.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:23 smithi160 ceph-mon[101879]: Reconfiguring mon.b (monmap changed)... 2024-09-21T10:03:24.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:23 smithi160 ceph-mon[101879]: Reconfiguring daemon mon.b on smithi160 2024-09-21T10:03:24.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:23 smithi160 ceph-mon[101879]: osdmap e480: 8 total, 8 up, 8 in 2024-09-21T10:03:24.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:23 smithi070 ceph-mon[107385]: Reconfiguring mon.b (monmap changed)... 2024-09-21T10:03:24.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:23 smithi070 ceph-mon[107385]: Reconfiguring daemon mon.b on smithi160 2024-09-21T10:03:24.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:23 smithi070 ceph-mon[107385]: osdmap e480: 8 total, 8 up, 8 in 2024-09-21T10:03:24.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:23 smithi070 ceph-mon[112139]: Reconfiguring mon.b (monmap changed)... 2024-09-21T10:03:24.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:23 smithi070 ceph-mon[112139]: Reconfiguring daemon mon.b on smithi160 2024-09-21T10:03:24.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:23 smithi070 ceph-mon[112139]: osdmap e480: 8 total, 8 up, 8 in 2024-09-21T10:03:24.926 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:24 smithi160 ceph-mon[101879]: osdmap e481: 8 total, 8 up, 8 in 2024-09-21T10:03:24.926 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:24 smithi160 ceph-mon[101879]: pgmap v217: 105 pgs: 105 active+clean; 583 KiB data, 463 MiB used, 715 GiB / 715 GiB avail; 171 KiB/s rd, 26 KiB/s wr, 250 op/s 2024-09-21T10:03:24.926 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:24 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:24.926 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:24 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:24.926 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:24 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T10:03:24.927 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:24 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T10:03:24.927 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:24 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:25.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:24 smithi070 ceph-mon[107385]: osdmap e481: 8 total, 8 up, 8 in 2024-09-21T10:03:25.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:24 smithi070 ceph-mon[107385]: pgmap v217: 105 pgs: 105 active+clean; 583 KiB data, 463 MiB used, 715 GiB / 715 GiB avail; 171 KiB/s rd, 26 KiB/s wr, 250 op/s 2024-09-21T10:03:25.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:24 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:25.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:24 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:25.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:24 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T10:03:25.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:24 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T10:03:25.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:24 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:25.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:24 smithi070 ceph-mon[112139]: osdmap e481: 8 total, 8 up, 8 in 2024-09-21T10:03:25.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:24 smithi070 ceph-mon[112139]: pgmap v217: 105 pgs: 105 active+clean; 583 KiB data, 463 MiB used, 715 GiB / 715 GiB avail; 171 KiB/s rd, 26 KiB/s wr, 250 op/s 2024-09-21T10:03:25.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:24 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:25.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:24 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:25.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:24 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T10:03:25.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:24 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T10:03:25.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:24 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:26.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:25 smithi160 ceph-mon[101879]: Reconfiguring mgr.x (monmap changed)... 2024-09-21T10:03:26.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:25 smithi160 ceph-mon[101879]: Reconfiguring daemon mgr.x on smithi160 2024-09-21T10:03:26.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:25 smithi160 ceph-mon[101879]: osdmap e482: 8 total, 8 up, 8 in 2024-09-21T10:03:26.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:25 smithi070 ceph-mon[107385]: Reconfiguring mgr.x (monmap changed)... 2024-09-21T10:03:26.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:25 smithi070 ceph-mon[107385]: Reconfiguring daemon mgr.x on smithi160 2024-09-21T10:03:26.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:25 smithi070 ceph-mon[107385]: osdmap e482: 8 total, 8 up, 8 in 2024-09-21T10:03:26.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:25 smithi070 ceph-mon[112139]: Reconfiguring mgr.x (monmap changed)... 2024-09-21T10:03:26.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:25 smithi070 ceph-mon[112139]: Reconfiguring daemon mgr.x on smithi160 2024-09-21T10:03:26.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:25 smithi070 ceph-mon[112139]: osdmap e482: 8 total, 8 up, 8 in 2024-09-21T10:03:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:26 smithi160 ceph-mon[101879]: osdmap e483: 8 total, 8 up, 8 in 2024-09-21T10:03:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:26 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:26.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:26 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:26.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:26 smithi160 ceph-mon[101879]: Reconfiguring osd.4 (monmap changed)... 2024-09-21T10:03:26.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:26 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T10:03:26.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:26 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:26.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:26 smithi160 ceph-mon[101879]: Reconfiguring daemon osd.4 on smithi160 2024-09-21T10:03:26.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:26 smithi160 ceph-mon[101879]: pgmap v220: 105 pgs: 105 active+clean; 583 KiB data, 464 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s rd, 9.0 KiB/s wr, 164 op/s 2024-09-21T10:03:27.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:26 smithi070 ceph-mon[107385]: osdmap e483: 8 total, 8 up, 8 in 2024-09-21T10:03:27.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:26 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:27.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:26 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:27.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:26 smithi070 ceph-mon[107385]: Reconfiguring osd.4 (monmap changed)... 2024-09-21T10:03:27.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:26 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T10:03:27.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:26 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:27.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:26 smithi070 ceph-mon[107385]: Reconfiguring daemon osd.4 on smithi160 2024-09-21T10:03:27.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:26 smithi070 ceph-mon[107385]: pgmap v220: 105 pgs: 105 active+clean; 583 KiB data, 464 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s rd, 9.0 KiB/s wr, 164 op/s 2024-09-21T10:03:27.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:26 smithi070 ceph-mon[112139]: osdmap e483: 8 total, 8 up, 8 in 2024-09-21T10:03:27.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:26 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:27.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:26 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:27.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:26 smithi070 ceph-mon[112139]: Reconfiguring osd.4 (monmap changed)... 2024-09-21T10:03:27.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:26 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T10:03:27.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:26 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:27.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:26 smithi070 ceph-mon[112139]: Reconfiguring daemon osd.4 on smithi160 2024-09-21T10:03:27.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:26 smithi070 ceph-mon[112139]: pgmap v220: 105 pgs: 105 active+clean; 583 KiB data, 464 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s rd, 9.0 KiB/s wr, 164 op/s 2024-09-21T10:03:27.679 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-09-21T10:03:27.983 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:27 smithi160 ceph-mon[101879]: osdmap e484: 8 total, 8 up, 8 in 2024-09-21T10:03:27.984 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:27 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:27.984 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:27 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:27.984 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:27 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T10:03:27.984 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:27 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:27.984 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:27 smithi160 ceph-mon[101879]: osdmap e485: 8 total, 8 up, 8 in 2024-09-21T10:03:28.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:27 smithi070 ceph-mon[107385]: osdmap e484: 8 total, 8 up, 8 in 2024-09-21T10:03:28.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:27 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:28.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:27 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:28.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:27 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T10:03:28.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:27 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:28.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:27 smithi070 ceph-mon[107385]: osdmap e485: 8 total, 8 up, 8 in 2024-09-21T10:03:28.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:27 smithi070 ceph-mon[112139]: osdmap e484: 8 total, 8 up, 8 in 2024-09-21T10:03:28.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:27 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:28.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:27 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:28.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:27 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T10:03:28.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:27 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:28.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:27 smithi070 ceph-mon[112139]: osdmap e485: 8 total, 8 up, 8 in 2024-09-21T10:03:28.237 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T10:03:28.237 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (3m) 66s ago 9m 20.4M - 0.27.0 11f11916f8cd 89ae95182419 2024-09-21T10:03:28.237 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi160 *:3000 running (3m) 21s ago 9m 84.7M - 10.4.8 5aad1d7cf004 daa2d8dfab48 2024-09-21T10:03:28.237 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi070.gansda smithi070 running (8m) 66s ago 8m 20.1M - 18.2.4-871-g6a1bdf85 0adab276c97e 6016b2501c8a 2024-09-21T10:03:28.238 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi160.byyhfz smithi160 running (8m) 21s ago 8m 16.4M - 18.2.4-871-g6a1bdf85 0adab276c97e 93e108d91e65 2024-09-21T10:03:28.238 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi160 *:8443,9283,8765 running (4m) 21s ago 13m 456M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 d952652a9683 2024-09-21T10:03:28.238 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (5m) 66s ago 15m 509M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 e3d390893cff 2024-09-21T10:03:28.238 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (2m) 66s ago 15m 47.2M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 4a7b667a4033 2024-09-21T10:03:28.238 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi160 running (27s) 21s ago 13m 21.8M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 86e88af21f36 2024-09-21T10:03:28.238 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (2m) 66s ago 13m 39.2M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 e154380b80a2 2024-09-21T10:03:28.238 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (4m) 66s ago 9m 11.3M - 1.7.0 72c9c2088986 5d5d4b026070 2024-09-21T10:03:28.238 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi160 *:9100 running (4m) 21s ago 9m 11.1M - 1.7.0 72c9c2088986 19b8e0e8ee0c 2024-09-21T10:03:28.238 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (13m) 66s ago 13m 115M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 10580e80f007 2024-09-21T10:03:28.238 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (12m) 66s ago 12m 125M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e b7c0d1169d1d 2024-09-21T10:03:28.238 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (12m) 66s ago 12m 161M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 58a11b8ddb81 2024-09-21T10:03:28.238 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (12m) 66s ago 12m 131M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e aa0c8a732a41 2024-09-21T10:03:28.238 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi160 running (11m) 21s ago 11m 120M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 7325a075c488 2024-09-21T10:03:28.239 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi160 running (11m) 21s ago 11m 106M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e f74026345641 2024-09-21T10:03:28.239 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi160 running (10m) 21s ago 10m 144M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 03e6bb54f8b2 2024-09-21T10:03:28.239 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi160 running (10m) 21s ago 10m 112M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e e405b21a6146 2024-09-21T10:03:28.239 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi160 *:9095 running (4m) 21s ago 10m 53.6M - 2.51.0 1d3b7f56885b 648952e57ad7 2024-09-21T10:03:28.580 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:03:28.580 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-09-21T10:03:28.580 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 3 2024-09-21T10:03:28.580 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:03:28.580 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-09-21T10:03:28.580 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T10:03:28.580 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:03:28.581 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-09-21T10:03:28.581 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 8 2024-09-21T10:03:28.581 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:03:28.581 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": { 2024-09-21T10:03:28.581 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 2 2024-09-21T10:03:28.581 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:03:28.581 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-09-21T10:03:28.581 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 10, 2024-09-21T10:03:28.581 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 5 2024-09-21T10:03:28.581 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:03:28.581 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:03:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:28 smithi160 ceph-mon[101879]: Reconfiguring osd.5 (monmap changed)... 2024-09-21T10:03:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:28 smithi160 ceph-mon[101879]: Reconfiguring daemon osd.5 on smithi160 2024-09-21T10:03:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:28 smithi160 ceph-mon[101879]: from='client.44186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:03:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:28 smithi160 ceph-mon[101879]: pgmap v223: 105 pgs: 105 active+clean; 583 KiB data, 464 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s rd, 9.0 KiB/s wr, 164 op/s 2024-09-21T10:03:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:28 smithi160 ceph-mon[101879]: from='client.34263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:03:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:28 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3967546509' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:03:28.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:28 smithi160 ceph-mon[101879]: osdmap e486: 8 total, 8 up, 8 in 2024-09-21T10:03:28.871 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:03:28.872 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47", 2024-09-21T10:03:28.872 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-09-21T10:03:28.872 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) mon. Upgrade limited to 1 daemons (0 remaining).", 2024-09-21T10:03:28.872 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [ 2024-09-21T10:03:28.872 INFO:teuthology.orchestra.run.smithi070.stdout: "mon" 2024-09-21T10:03:28.872 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-09-21T10:03:28.872 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "3/3 daemons upgraded", 2024-09-21T10:03:28.872 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading mon daemons", 2024-09-21T10:03:28.872 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-09-21T10:03:28.872 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:03:28.873 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:28 smithi070 ceph-mon[112139]: Reconfiguring osd.5 (monmap changed)... 2024-09-21T10:03:28.873 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:28 smithi070 ceph-mon[112139]: Reconfiguring daemon osd.5 on smithi160 2024-09-21T10:03:28.873 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:28 smithi070 ceph-mon[112139]: from='client.44186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:03:28.873 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:28 smithi070 ceph-mon[112139]: pgmap v223: 105 pgs: 105 active+clean; 583 KiB data, 464 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s rd, 9.0 KiB/s wr, 164 op/s 2024-09-21T10:03:28.873 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:28 smithi070 ceph-mon[112139]: from='client.34263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:03:28.873 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:28 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3967546509' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:03:28.873 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:28 smithi070 ceph-mon[112139]: osdmap e486: 8 total, 8 up, 8 in 2024-09-21T10:03:28.874 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:28 smithi070 ceph-mon[107385]: Reconfiguring osd.5 (monmap changed)... 2024-09-21T10:03:28.874 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:28 smithi070 ceph-mon[107385]: Reconfiguring daemon osd.5 on smithi160 2024-09-21T10:03:28.874 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:28 smithi070 ceph-mon[107385]: from='client.44186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:03:28.874 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:28 smithi070 ceph-mon[107385]: pgmap v223: 105 pgs: 105 active+clean; 583 KiB data, 464 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s rd, 9.0 KiB/s wr, 164 op/s 2024-09-21T10:03:28.874 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:28 smithi070 ceph-mon[107385]: from='client.34263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:03:28.874 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:28 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3967546509' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:03:28.874 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:28 smithi070 ceph-mon[107385]: osdmap e486: 8 total, 8 up, 8 in 2024-09-21T10:03:30.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:30 smithi070 ceph-mon[107385]: from='client.44198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:03:30.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:30 smithi070 ceph-mon[107385]: from='client.44210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:03:30.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:30 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:30.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:30 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:30.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:30 smithi070 ceph-mon[107385]: Reconfiguring osd.6 (monmap changed)... 2024-09-21T10:03:30.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:30 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T10:03:30.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:30 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:30.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:30 smithi070 ceph-mon[107385]: Reconfiguring daemon osd.6 on smithi160 2024-09-21T10:03:30.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:30 smithi070 ceph-mon[107385]: osdmap e487: 8 total, 8 up, 8 in 2024-09-21T10:03:30.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:30 smithi070 ceph-mon[112139]: from='client.44198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:03:30.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:30 smithi070 ceph-mon[112139]: from='client.44210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:03:30.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:30 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:30.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:30 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:30.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:30 smithi070 ceph-mon[112139]: Reconfiguring osd.6 (monmap changed)... 2024-09-21T10:03:30.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:30 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T10:03:30.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:30 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:30.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:30 smithi070 ceph-mon[112139]: Reconfiguring daemon osd.6 on smithi160 2024-09-21T10:03:30.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:30 smithi070 ceph-mon[112139]: osdmap e487: 8 total, 8 up, 8 in 2024-09-21T10:03:30.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:30 smithi160 ceph-mon[101879]: from='client.44198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:03:30.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:30 smithi160 ceph-mon[101879]: from='client.44210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:03:30.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:30 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:30.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:30 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:30.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:30 smithi160 ceph-mon[101879]: Reconfiguring osd.6 (monmap changed)... 2024-09-21T10:03:30.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:30 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T10:03:30.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:30 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:30.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:30 smithi160 ceph-mon[101879]: Reconfiguring daemon osd.6 on smithi160 2024-09-21T10:03:30.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:30 smithi160 ceph-mon[101879]: osdmap e487: 8 total, 8 up, 8 in 2024-09-21T10:03:30.706 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.SnapshotPromoteDemote (9919 ms) 2024-09-21T10:03:30.707 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestMirroring.AioSnapshotCreate 2024-09-21T10:03:31.181 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:31 smithi160 ceph-mon[101879]: pgmap v226: 105 pgs: 105 active+clean; 583 KiB data, 464 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:03:31.181 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:31.181 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:31.181 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T10:03:31.181 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:31.181 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:31 smithi160 ceph-mon[101879]: osdmap e488: 8 total, 8 up, 8 in 2024-09-21T10:03:31.181 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:31 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/a314ffb8-2273-40e7-b70e-e2843296de4b"} : dispatch 2024-09-21T10:03:31.181 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:31 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/a314ffb8-2273-40e7-b70e-e2843296de4b"}]': finished 2024-09-21T10:03:31.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:31 smithi070 ceph-mon[107385]: pgmap v226: 105 pgs: 105 active+clean; 583 KiB data, 464 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:03:31.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:31.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:31.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T10:03:31.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:31.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:31 smithi070 ceph-mon[107385]: osdmap e488: 8 total, 8 up, 8 in 2024-09-21T10:03:31.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:31 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/a314ffb8-2273-40e7-b70e-e2843296de4b"} : dispatch 2024-09-21T10:03:31.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:31 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/a314ffb8-2273-40e7-b70e-e2843296de4b"}]': finished 2024-09-21T10:03:31.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:31 smithi070 ceph-mon[112139]: pgmap v226: 105 pgs: 105 active+clean; 583 KiB data, 464 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:03:31.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:31.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:31.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T10:03:31.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:31.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:31 smithi070 ceph-mon[112139]: osdmap e488: 8 total, 8 up, 8 in 2024-09-21T10:03:31.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:31 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/a314ffb8-2273-40e7-b70e-e2843296de4b"} : dispatch 2024-09-21T10:03:31.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:31 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/a314ffb8-2273-40e7-b70e-e2843296de4b"}]': finished 2024-09-21T10:03:32.317 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:32 smithi160 ceph-mon[101879]: Reconfiguring osd.7 (monmap changed)... 2024-09-21T10:03:32.318 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:32 smithi160 ceph-mon[101879]: Reconfiguring daemon osd.7 on smithi160 2024-09-21T10:03:32.318 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:32 smithi160 ceph-mon[101879]: osdmap e489: 8 total, 8 up, 8 in 2024-09-21T10:03:32.318 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:32 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:32.318 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:32 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:32.318 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:32 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi160.byyhfz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T10:03:32.318 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:32 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:32.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:32 smithi070 ceph-mon[107385]: Reconfiguring osd.7 (monmap changed)... 2024-09-21T10:03:32.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:32 smithi070 ceph-mon[107385]: Reconfiguring daemon osd.7 on smithi160 2024-09-21T10:03:32.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:32 smithi070 ceph-mon[107385]: osdmap e489: 8 total, 8 up, 8 in 2024-09-21T10:03:32.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:32 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:32.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:32 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:32.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:32 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi160.byyhfz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T10:03:32.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:32 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:32.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:32 smithi070 ceph-mon[112139]: Reconfiguring osd.7 (monmap changed)... 2024-09-21T10:03:32.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:32 smithi070 ceph-mon[112139]: Reconfiguring daemon osd.7 on smithi160 2024-09-21T10:03:32.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:32 smithi070 ceph-mon[112139]: osdmap e489: 8 total, 8 up, 8 in 2024-09-21T10:03:32.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:32 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:32.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:32 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:32.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:32 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi160.byyhfz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T10:03:32.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:32 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:33.286 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:33 smithi160 ceph-mon[101879]: pgmap v229: 105 pgs: 105 active+clean; 583 KiB data, 472 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 6.7 KiB/s wr, 56 op/s 2024-09-21T10:03:33.286 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:33 smithi160 ceph-mon[101879]: Reconfiguring mds.foo.smithi160.byyhfz (monmap changed)... 2024-09-21T10:03:33.286 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:33 smithi160 ceph-mon[101879]: Reconfiguring daemon mds.foo.smithi160.byyhfz on smithi160 2024-09-21T10:03:33.286 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:33 smithi160 ceph-mon[101879]: osdmap e490: 8 total, 8 up, 8 in 2024-09-21T10:03:33.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:03:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:03:33] "GET /metrics HTTP/1.1" 200 36273 "" "Prometheus/2.51.0" 2024-09-21T10:03:33.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:33 smithi070 ceph-mon[112139]: pgmap v229: 105 pgs: 105 active+clean; 583 KiB data, 472 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 6.7 KiB/s wr, 56 op/s 2024-09-21T10:03:33.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:33 smithi070 ceph-mon[112139]: Reconfiguring mds.foo.smithi160.byyhfz (monmap changed)... 2024-09-21T10:03:33.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:33 smithi070 ceph-mon[112139]: Reconfiguring daemon mds.foo.smithi160.byyhfz on smithi160 2024-09-21T10:03:33.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:33 smithi070 ceph-mon[112139]: osdmap e490: 8 total, 8 up, 8 in 2024-09-21T10:03:33.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:33 smithi070 ceph-mon[107385]: pgmap v229: 105 pgs: 105 active+clean; 583 KiB data, 472 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 6.7 KiB/s wr, 56 op/s 2024-09-21T10:03:33.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:33 smithi070 ceph-mon[107385]: Reconfiguring mds.foo.smithi160.byyhfz (monmap changed)... 2024-09-21T10:03:33.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:33 smithi070 ceph-mon[107385]: Reconfiguring daemon mds.foo.smithi160.byyhfz on smithi160 2024-09-21T10:03:33.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:33 smithi070 ceph-mon[107385]: osdmap e490: 8 total, 8 up, 8 in 2024-09-21T10:03:34.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:34.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:34.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:03:34.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:03:34.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:03:34.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: Upgrade: Setting container_image for all nvmeof 2024-09-21T10:03:34.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:34.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:03:34.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: Upgrade: Setting container_image for all node-exporter 2024-09-21T10:03:34.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:34.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-21T10:03:34.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-21T10:03:34.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:03:34.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: Upgrade: Setting container_image for all prometheus 2024-09-21T10:03:34.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-21T10:03:34.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:03:34.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: Upgrade: Setting container_image for all alertmanager 2024-09-21T10:03:34.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-21T10:03:34.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:03:34.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: Upgrade: Setting container_image for all grafana 2024-09-21T10:03:34.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-21T10:03:34.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:03:34.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: Upgrade: Setting container_image for all loki 2024-09-21T10:03:34.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-21T10:03:34.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:03:34.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: Upgrade: Setting container_image for all promtail 2024-09-21T10:03:34.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-21T10:03:34.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: Upgrade: Finalizing container_image settings 2024-09-21T10:03:34.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-21T10:03:34.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-21T10:03:34.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:03:34.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-21T10:03:34.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-21T10:03:34.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-21T10:03:34.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-21T10:03:34.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-21T10:03:34.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-21T10:03:34.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:03:34.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-21T10:03:34.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-21T10:03:34.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-21T10:03:34.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-21T10:03:34.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-21T10:03:34.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:03:34.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:03:34.597 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:03:34.598 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:03:34.598 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:03:34.598 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:03:34.598 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: Upgrade: Complete! 2024-09-21T10:03:34.598 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-21T10:03:34.598 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-21T10:03:34.599 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:03:34.599 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: osdmap e491: 8 total, 8 up, 8 in 2024-09-21T10:03:34.599 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:34 smithi160 ceph-mon[101879]: pgmap v232: 105 pgs: 105 active+clean; 583 KiB data, 472 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 6.7 KiB/s wr, 56 op/s 2024-09-21T10:03:34.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:34.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:34.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:03:34.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:03:34.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:03:34.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all nvmeof 2024-09-21T10:03:34.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:34.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:03:34.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all node-exporter 2024-09-21T10:03:34.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:34.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-21T10:03:34.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-21T10:03:34.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:03:34.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all prometheus 2024-09-21T10:03:34.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-21T10:03:34.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:03:34.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all alertmanager 2024-09-21T10:03:34.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-21T10:03:34.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:03:34.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all grafana 2024-09-21T10:03:34.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-21T10:03:34.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:03:34.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all loki 2024-09-21T10:03:34.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-21T10:03:34.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:03:34.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all promtail 2024-09-21T10:03:34.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-21T10:03:34.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: Upgrade: Finalizing container_image settings 2024-09-21T10:03:34.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-21T10:03:34.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:34.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:34.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:03:34.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:03:34.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:03:34.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all nvmeof 2024-09-21T10:03:34.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:34.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:03:34.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all node-exporter 2024-09-21T10:03:34.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:34.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-21T10:03:34.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-21T10:03:34.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:03:34.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all prometheus 2024-09-21T10:03:34.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-21T10:03:34.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:03:34.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all alertmanager 2024-09-21T10:03:34.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-21T10:03:34.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:03:34.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all grafana 2024-09-21T10:03:34.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-21T10:03:34.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:03:34.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all loki 2024-09-21T10:03:34.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-21T10:03:34.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:03:34.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all promtail 2024-09-21T10:03:34.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-21T10:03:34.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: Upgrade: Finalizing container_image settings 2024-09-21T10:03:34.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-21T10:03:34.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-21T10:03:34.634 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:03:34.634 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-21T10:03:34.634 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-21T10:03:34.634 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-21T10:03:34.634 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-21T10:03:34.634 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-21T10:03:34.634 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-21T10:03:34.634 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:03:34.634 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-21T10:03:34.634 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-21T10:03:34.634 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-21T10:03:34.634 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-21T10:03:34.634 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-21T10:03:34.634 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:03:34.635 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:03:34.635 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:03:34.635 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:03:34.635 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:03:34.635 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:03:34.635 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: Upgrade: Complete! 2024-09-21T10:03:34.635 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-21T10:03:34.635 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-21T10:03:34.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-21T10:03:34.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:03:34.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-21T10:03:34.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-21T10:03:34.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-21T10:03:34.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-21T10:03:34.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-21T10:03:34.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-21T10:03:34.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:03:34.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-21T10:03:34.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-21T10:03:34.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-21T10:03:34.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-21T10:03:34.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-21T10:03:34.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:03:34.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:03:34.637 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:03:34.637 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:03:34.637 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:03:34.637 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:03:34.637 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: Upgrade: Complete! 2024-09-21T10:03:34.637 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-21T10:03:34.637 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-21T10:03:34.637 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:03:34.638 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: osdmap e491: 8 total, 8 up, 8 in 2024-09-21T10:03:34.638 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[107385]: pgmap v232: 105 pgs: 105 active+clean; 583 KiB data, 472 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 6.7 KiB/s wr, 56 op/s 2024-09-21T10:03:34.638 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:03:34.638 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: osdmap e491: 8 total, 8 up, 8 in 2024-09-21T10:03:34.638 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:34 smithi070 ceph-mon[112139]: pgmap v232: 105 pgs: 105 active+clean; 583 KiB data, 472 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 6.7 KiB/s wr, 56 op/s 2024-09-21T10:03:36.023 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:35 smithi070 ceph-mon[107385]: osdmap e492: 8 total, 8 up, 8 in 2024-09-21T10:03:36.024 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:36.024 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:36.024 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:35 smithi070 ceph-mon[112139]: osdmap e492: 8 total, 8 up, 8 in 2024-09-21T10:03:36.024 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:36.024 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:35 smithi160 ceph-mon[101879]: osdmap e492: 8 total, 8 up, 8 in 2024-09-21T10:03:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:36.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:36 smithi070 ceph-mon[107385]: osdmap e493: 8 total, 8 up, 8 in 2024-09-21T10:03:36.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:36 smithi070 ceph-mon[107385]: pgmap v235: 105 pgs: 105 active+clean; 583 KiB data, 477 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 14 KiB/s wr, 119 op/s 2024-09-21T10:03:36.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:36 smithi070 ceph-mon[112139]: osdmap e493: 8 total, 8 up, 8 in 2024-09-21T10:03:36.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:36 smithi070 ceph-mon[112139]: pgmap v235: 105 pgs: 105 active+clean; 583 KiB data, 477 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 14 KiB/s wr, 119 op/s 2024-09-21T10:03:37.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:36 smithi160 ceph-mon[101879]: osdmap e493: 8 total, 8 up, 8 in 2024-09-21T10:03:37.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:36 smithi160 ceph-mon[101879]: pgmap v235: 105 pgs: 105 active+clean; 583 KiB data, 477 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 14 KiB/s wr, 119 op/s 2024-09-21T10:03:38.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:37 smithi160 ceph-mon[101879]: osdmap e494: 8 total, 8 up, 8 in 2024-09-21T10:03:38.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:37 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:38.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:37 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:38.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:37 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:38.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:37 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:03:38.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:37 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:38.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:37 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:03:38.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:37 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:38.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:37 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:03:38.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:37 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:38.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:37 smithi160 ceph-mon[101879]: osdmap e495: 8 total, 8 up, 8 in 2024-09-21T10:03:38.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:37 smithi070 ceph-mon[107385]: osdmap e494: 8 total, 8 up, 8 in 2024-09-21T10:03:38.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:37 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:38.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:37 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:38.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:37 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:38.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:37 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:03:38.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:37 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:38.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:37 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:03:38.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:37 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:38.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:37 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:03:38.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:37 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:38.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:37 smithi070 ceph-mon[107385]: osdmap e495: 8 total, 8 up, 8 in 2024-09-21T10:03:38.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:37 smithi070 ceph-mon[112139]: osdmap e494: 8 total, 8 up, 8 in 2024-09-21T10:03:38.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:37 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:38.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:37 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:38.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:37 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:38.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:37 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:03:38.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:37 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:38.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:37 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:03:38.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:37 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:38.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:37 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:03:38.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:37 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:03:38.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:37 smithi070 ceph-mon[112139]: osdmap e495: 8 total, 8 up, 8 in 2024-09-21T10:03:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:38 smithi160 ceph-mon[101879]: pgmap v238: 105 pgs: 105 active+clean; 583 KiB data, 477 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 14 KiB/s wr, 119 op/s 2024-09-21T10:03:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:38 smithi160 ceph-mon[101879]: osdmap e496: 8 total, 8 up, 8 in 2024-09-21T10:03:39.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:38 smithi070 ceph-mon[112139]: pgmap v238: 105 pgs: 105 active+clean; 583 KiB data, 477 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 14 KiB/s wr, 119 op/s 2024-09-21T10:03:39.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:38 smithi070 ceph-mon[112139]: osdmap e496: 8 total, 8 up, 8 in 2024-09-21T10:03:39.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:38 smithi070 ceph-mon[107385]: pgmap v238: 105 pgs: 105 active+clean; 583 KiB data, 477 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 14 KiB/s wr, 119 op/s 2024-09-21T10:03:39.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:38 smithi070 ceph-mon[107385]: osdmap e496: 8 total, 8 up, 8 in 2024-09-21T10:03:41.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:40 smithi160 ceph-mon[101879]: osdmap e497: 8 total, 8 up, 8 in 2024-09-21T10:03:41.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:40 smithi160 ceph-mon[101879]: pgmap v241: 105 pgs: 105 active+clean; 583 KiB data, 477 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:03:41.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:40 smithi070 ceph-mon[112139]: osdmap e497: 8 total, 8 up, 8 in 2024-09-21T10:03:41.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:40 smithi070 ceph-mon[112139]: pgmap v241: 105 pgs: 105 active+clean; 583 KiB data, 477 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:03:41.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:40 smithi070 ceph-mon[107385]: osdmap e497: 8 total, 8 up, 8 in 2024-09-21T10:03:41.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:40 smithi070 ceph-mon[107385]: pgmap v241: 105 pgs: 105 active+clean; 583 KiB data, 477 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:03:42.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:41 smithi160 ceph-mon[101879]: osdmap e498: 8 total, 8 up, 8 in 2024-09-21T10:03:42.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:41 smithi070 ceph-mon[107385]: osdmap e498: 8 total, 8 up, 8 in 2024-09-21T10:03:42.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:41 smithi070 ceph-mon[112139]: osdmap e498: 8 total, 8 up, 8 in 2024-09-21T10:03:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:42 smithi160 ceph-mon[101879]: osdmap e499: 8 total, 8 up, 8 in 2024-09-21T10:03:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:42 smithi160 ceph-mon[101879]: pgmap v244: 105 pgs: 105 active+clean; 584 KiB data, 478 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 13 KiB/s wr, 112 op/s 2024-09-21T10:03:43.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:42 smithi070 ceph-mon[107385]: osdmap e499: 8 total, 8 up, 8 in 2024-09-21T10:03:43.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:42 smithi070 ceph-mon[107385]: pgmap v244: 105 pgs: 105 active+clean; 584 KiB data, 478 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 13 KiB/s wr, 112 op/s 2024-09-21T10:03:43.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:42 smithi070 ceph-mon[112139]: osdmap e499: 8 total, 8 up, 8 in 2024-09-21T10:03:43.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:42 smithi070 ceph-mon[112139]: pgmap v244: 105 pgs: 105 active+clean; 584 KiB data, 478 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 13 KiB/s wr, 112 op/s 2024-09-21T10:03:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:03:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:03:43] "GET /metrics HTTP/1.1" 200 36272 "" "Prometheus/2.51.0" 2024-09-21T10:03:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:43 smithi160 ceph-mon[101879]: osdmap e500: 8 total, 8 up, 8 in 2024-09-21T10:03:44.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:43 smithi070 ceph-mon[107385]: osdmap e500: 8 total, 8 up, 8 in 2024-09-21T10:03:44.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:43 smithi070 ceph-mon[112139]: osdmap e500: 8 total, 8 up, 8 in 2024-09-21T10:03:45.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:44 smithi160 ceph-mon[101879]: osdmap e501: 8 total, 8 up, 8 in 2024-09-21T10:03:45.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:44 smithi160 ceph-mon[101879]: pgmap v247: 105 pgs: 105 active+clean; 584 KiB data, 478 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 13 KiB/s wr, 112 op/s 2024-09-21T10:03:45.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:44 smithi070 ceph-mon[107385]: osdmap e501: 8 total, 8 up, 8 in 2024-09-21T10:03:45.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:44 smithi070 ceph-mon[107385]: pgmap v247: 105 pgs: 105 active+clean; 584 KiB data, 478 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 13 KiB/s wr, 112 op/s 2024-09-21T10:03:45.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:44 smithi070 ceph-mon[112139]: osdmap e501: 8 total, 8 up, 8 in 2024-09-21T10:03:45.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:44 smithi070 ceph-mon[112139]: pgmap v247: 105 pgs: 105 active+clean; 584 KiB data, 478 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 13 KiB/s wr, 112 op/s 2024-09-21T10:03:46.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:45 smithi160 ceph-mon[101879]: osdmap e502: 8 total, 8 up, 8 in 2024-09-21T10:03:46.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:45 smithi070 ceph-mon[107385]: osdmap e502: 8 total, 8 up, 8 in 2024-09-21T10:03:46.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:45 smithi070 ceph-mon[112139]: osdmap e502: 8 total, 8 up, 8 in 2024-09-21T10:03:47.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:46 smithi160 ceph-mon[101879]: osdmap e503: 8 total, 8 up, 8 in 2024-09-21T10:03:47.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:46 smithi160 ceph-mon[101879]: pgmap v250: 105 pgs: 105 active+clean; 584 KiB data, 479 MiB used, 715 GiB / 715 GiB avail; 195 KiB/s rd, 20 KiB/s wr, 267 op/s 2024-09-21T10:03:47.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:46 smithi070 ceph-mon[107385]: osdmap e503: 8 total, 8 up, 8 in 2024-09-21T10:03:47.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:46 smithi070 ceph-mon[107385]: pgmap v250: 105 pgs: 105 active+clean; 584 KiB data, 479 MiB used, 715 GiB / 715 GiB avail; 195 KiB/s rd, 20 KiB/s wr, 267 op/s 2024-09-21T10:03:47.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:46 smithi070 ceph-mon[112139]: osdmap e503: 8 total, 8 up, 8 in 2024-09-21T10:03:47.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:46 smithi070 ceph-mon[112139]: pgmap v250: 105 pgs: 105 active+clean; 584 KiB data, 479 MiB used, 715 GiB / 715 GiB avail; 195 KiB/s rd, 20 KiB/s wr, 267 op/s 2024-09-21T10:03:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:47 smithi160 ceph-mon[101879]: osdmap e504: 8 total, 8 up, 8 in 2024-09-21T10:03:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:47 smithi160 ceph-mon[101879]: osdmap e505: 8 total, 8 up, 8 in 2024-09-21T10:03:48.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:47 smithi070 ceph-mon[107385]: osdmap e504: 8 total, 8 up, 8 in 2024-09-21T10:03:48.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:47 smithi070 ceph-mon[107385]: osdmap e505: 8 total, 8 up, 8 in 2024-09-21T10:03:48.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:47 smithi070 ceph-mon[112139]: osdmap e504: 8 total, 8 up, 8 in 2024-09-21T10:03:48.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:47 smithi070 ceph-mon[112139]: osdmap e505: 8 total, 8 up, 8 in 2024-09-21T10:03:49.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:48 smithi160 ceph-mon[101879]: pgmap v253: 105 pgs: 105 active+clean; 584 KiB data, 479 MiB used, 715 GiB / 715 GiB avail; 195 KiB/s rd, 20 KiB/s wr, 267 op/s 2024-09-21T10:03:49.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:48 smithi070 ceph-mon[107385]: pgmap v253: 105 pgs: 105 active+clean; 584 KiB data, 479 MiB used, 715 GiB / 715 GiB avail; 195 KiB/s rd, 20 KiB/s wr, 267 op/s 2024-09-21T10:03:49.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:48 smithi070 ceph-mon[112139]: pgmap v253: 105 pgs: 105 active+clean; 584 KiB data, 479 MiB used, 715 GiB / 715 GiB avail; 195 KiB/s rd, 20 KiB/s wr, 267 op/s 2024-09-21T10:03:50.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:49 smithi160 ceph-mon[101879]: osdmap e506: 8 total, 8 up, 8 in 2024-09-21T10:03:50.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:49 smithi070 ceph-mon[107385]: osdmap e506: 8 total, 8 up, 8 in 2024-09-21T10:03:50.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:49 smithi070 ceph-mon[112139]: osdmap e506: 8 total, 8 up, 8 in 2024-09-21T10:03:51.029 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:50 smithi070 ceph-mon[107385]: osdmap e507: 8 total, 8 up, 8 in 2024-09-21T10:03:51.029 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:50 smithi070 ceph-mon[107385]: pgmap v256: 105 pgs: 105 active+clean; 584 KiB data, 479 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:03:51.029 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:51.029 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:50 smithi070 ceph-mon[112139]: osdmap e507: 8 total, 8 up, 8 in 2024-09-21T10:03:51.029 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:50 smithi070 ceph-mon[112139]: pgmap v256: 105 pgs: 105 active+clean; 584 KiB data, 479 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:03:51.029 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:51.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:50 smithi160 ceph-mon[101879]: osdmap e507: 8 total, 8 up, 8 in 2024-09-21T10:03:51.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:50 smithi160 ceph-mon[101879]: pgmap v256: 105 pgs: 105 active+clean; 584 KiB data, 479 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:03:51.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:52.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:51 smithi070 ceph-mon[107385]: osdmap e508: 8 total, 8 up, 8 in 2024-09-21T10:03:52.045 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:51 smithi070 ceph-mon[112139]: osdmap e508: 8 total, 8 up, 8 in 2024-09-21T10:03:52.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:51 smithi160 ceph-mon[101879]: osdmap e508: 8 total, 8 up, 8 in 2024-09-21T10:03:52.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:52 smithi070 ceph-mon[107385]: osdmap e509: 8 total, 8 up, 8 in 2024-09-21T10:03:52.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:52 smithi070 ceph-mon[107385]: pgmap v259: 105 pgs: 105 active+clean; 584 KiB data, 483 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 7.5 KiB/s wr, 116 op/s 2024-09-21T10:03:52.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:52 smithi070 ceph-mon[112139]: osdmap e509: 8 total, 8 up, 8 in 2024-09-21T10:03:52.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:52 smithi070 ceph-mon[112139]: pgmap v259: 105 pgs: 105 active+clean; 584 KiB data, 483 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 7.5 KiB/s wr, 116 op/s 2024-09-21T10:03:53.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:52 smithi160 ceph-mon[101879]: osdmap e509: 8 total, 8 up, 8 in 2024-09-21T10:03:53.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:52 smithi160 ceph-mon[101879]: pgmap v259: 105 pgs: 105 active+clean; 584 KiB data, 483 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 7.5 KiB/s wr, 116 op/s 2024-09-21T10:03:53.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:03:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:03:53] "GET /metrics HTTP/1.1" 200 36275 "" "Prometheus/2.51.0" 2024-09-21T10:03:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:53 smithi160 ceph-mon[101879]: osdmap e510: 8 total, 8 up, 8 in 2024-09-21T10:03:54.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:53 smithi070 ceph-mon[107385]: osdmap e510: 8 total, 8 up, 8 in 2024-09-21T10:03:54.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:53 smithi070 ceph-mon[112139]: osdmap e510: 8 total, 8 up, 8 in 2024-09-21T10:03:55.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:54 smithi160 ceph-mon[101879]: osdmap e511: 8 total, 8 up, 8 in 2024-09-21T10:03:55.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:54 smithi160 ceph-mon[101879]: pgmap v262: 105 pgs: 105 active+clean; 584 KiB data, 483 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 7.5 KiB/s wr, 116 op/s 2024-09-21T10:03:55.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:54 smithi070 ceph-mon[107385]: osdmap e511: 8 total, 8 up, 8 in 2024-09-21T10:03:55.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:54 smithi070 ceph-mon[107385]: pgmap v262: 105 pgs: 105 active+clean; 584 KiB data, 483 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 7.5 KiB/s wr, 116 op/s 2024-09-21T10:03:55.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:54 smithi070 ceph-mon[112139]: osdmap e511: 8 total, 8 up, 8 in 2024-09-21T10:03:55.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:54 smithi070 ceph-mon[112139]: pgmap v262: 105 pgs: 105 active+clean; 584 KiB data, 483 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 7.5 KiB/s wr, 116 op/s 2024-09-21T10:03:56.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:55 smithi160 ceph-mon[101879]: osdmap e512: 8 total, 8 up, 8 in 2024-09-21T10:03:56.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:55 smithi070 ceph-mon[112139]: osdmap e512: 8 total, 8 up, 8 in 2024-09-21T10:03:56.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:55 smithi070 ceph-mon[107385]: osdmap e512: 8 total, 8 up, 8 in 2024-09-21T10:03:57.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:56 smithi160 ceph-mon[101879]: osdmap e513: 8 total, 8 up, 8 in 2024-09-21T10:03:57.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:56 smithi160 ceph-mon[101879]: pgmap v265: 105 pgs: 105 active+clean; 583 KiB data, 472 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 8.7 KiB/s wr, 160 op/s 2024-09-21T10:03:57.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:56 smithi070 ceph-mon[112139]: osdmap e513: 8 total, 8 up, 8 in 2024-09-21T10:03:57.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:56 smithi070 ceph-mon[112139]: pgmap v265: 105 pgs: 105 active+clean; 583 KiB data, 472 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 8.7 KiB/s wr, 160 op/s 2024-09-21T10:03:57.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:56 smithi070 ceph-mon[107385]: osdmap e513: 8 total, 8 up, 8 in 2024-09-21T10:03:57.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:56 smithi070 ceph-mon[107385]: pgmap v265: 105 pgs: 105 active+clean; 583 KiB data, 472 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 8.7 KiB/s wr, 160 op/s 2024-09-21T10:03:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:57 smithi160 ceph-mon[101879]: osdmap e514: 8 total, 8 up, 8 in 2024-09-21T10:03:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:57 smithi160 ceph-mon[101879]: osdmap e515: 8 total, 8 up, 8 in 2024-09-21T10:03:58.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:57 smithi070 ceph-mon[112139]: osdmap e514: 8 total, 8 up, 8 in 2024-09-21T10:03:58.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:57 smithi070 ceph-mon[112139]: osdmap e515: 8 total, 8 up, 8 in 2024-09-21T10:03:58.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:57 smithi070 ceph-mon[107385]: osdmap e514: 8 total, 8 up, 8 in 2024-09-21T10:03:58.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:57 smithi070 ceph-mon[107385]: osdmap e515: 8 total, 8 up, 8 in 2024-09-21T10:03:59.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:58 smithi160 ceph-mon[101879]: pgmap v268: 105 pgs: 105 active+clean; 583 KiB data, 472 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 8.7 KiB/s wr, 160 op/s 2024-09-21T10:03:59.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:03:58 smithi160 ceph-mon[101879]: osdmap e516: 8 total, 8 up, 8 in 2024-09-21T10:03:59.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:58 smithi070 ceph-mon[107385]: pgmap v268: 105 pgs: 105 active+clean; 583 KiB data, 472 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 8.7 KiB/s wr, 160 op/s 2024-09-21T10:03:59.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:03:58 smithi070 ceph-mon[107385]: osdmap e516: 8 total, 8 up, 8 in 2024-09-21T10:03:59.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:58 smithi070 ceph-mon[112139]: pgmap v268: 105 pgs: 105 active+clean; 583 KiB data, 472 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 8.7 KiB/s wr, 160 op/s 2024-09-21T10:03:59.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:03:58 smithi070 ceph-mon[112139]: osdmap e516: 8 total, 8 up, 8 in 2024-09-21T10:03:59.885 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'sleep 60' 2024-09-21T10:04:00.082 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:00.771 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:00 smithi070 ceph-mon[107385]: from='client.44216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:04:00.771 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:00 smithi070 ceph-mon[107385]: osdmap e517: 8 total, 8 up, 8 in 2024-09-21T10:04:00.771 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:00 smithi070 ceph-mon[107385]: pgmap v271: 105 pgs: 105 active+clean; 583 KiB data, 472 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:00.771 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:00 smithi070 ceph-mon[112139]: from='client.44216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:04:00.771 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:00 smithi070 ceph-mon[112139]: osdmap e517: 8 total, 8 up, 8 in 2024-09-21T10:04:00.771 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:00 smithi070 ceph-mon[112139]: pgmap v271: 105 pgs: 105 active+clean; 583 KiB data, 472 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:01.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:00 smithi160 ceph-mon[101879]: from='client.44216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:04:01.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:00 smithi160 ceph-mon[101879]: osdmap e517: 8 total, 8 up, 8 in 2024-09-21T10:04:01.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:00 smithi160 ceph-mon[101879]: pgmap v271: 105 pgs: 105 active+clean; 583 KiB data, 472 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:01.725 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestMirroring.AioSnapshotCreate (31019 ms) 2024-09-21T10:04:02.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:01 smithi160 ceph-mon[101879]: osdmap e518: 8 total, 8 up, 8 in 2024-09-21T10:04:02.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:01 smithi070 ceph-mon[107385]: osdmap e518: 8 total, 8 up, 8 in 2024-09-21T10:04:02.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:01 smithi070 ceph-mon[112139]: osdmap e518: 8 total, 8 up, 8 in 2024-09-21T10:04:02.729 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 51 tests from TestMirroring (90780 ms total) 2024-09-21T10:04:02.729 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:04:02.729 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 34 tests from TestInternal 2024-09-21T10:04:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:02 smithi160 ceph-mon[101879]: osdmap e519: 8 total, 8 up, 8 in 2024-09-21T10:04:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:02 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/38193273-7b6f-41f0-b037-2c07dff85a4a"} : dispatch 2024-09-21T10:04:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:02 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/38193273-7b6f-41f0-b037-2c07dff85a4a"}]': finished 2024-09-21T10:04:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:02 smithi160 ceph-mon[101879]: pgmap v274: 105 pgs: 105 active+clean; 583 KiB data, 456 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 7.5 KiB/s wr, 116 op/s 2024-09-21T10:04:03.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:02 smithi070 ceph-mon[112139]: osdmap e519: 8 total, 8 up, 8 in 2024-09-21T10:04:03.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:02 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/38193273-7b6f-41f0-b037-2c07dff85a4a"} : dispatch 2024-09-21T10:04:03.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:02 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/38193273-7b6f-41f0-b037-2c07dff85a4a"}]': finished 2024-09-21T10:04:03.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:02 smithi070 ceph-mon[112139]: pgmap v274: 105 pgs: 105 active+clean; 583 KiB data, 456 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 7.5 KiB/s wr, 116 op/s 2024-09-21T10:04:03.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:02 smithi070 ceph-mon[107385]: osdmap e519: 8 total, 8 up, 8 in 2024-09-21T10:04:03.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:02 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/38193273-7b6f-41f0-b037-2c07dff85a4a"} : dispatch 2024-09-21T10:04:03.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:02 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041281077' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/38193273-7b6f-41f0-b037-2c07dff85a4a"}]': finished 2024-09-21T10:04:03.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:02 smithi070 ceph-mon[107385]: pgmap v274: 105 pgs: 105 active+clean; 583 KiB data, 456 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 7.5 KiB/s wr, 116 op/s 2024-09-21T10:04:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:04:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:04:03] "GET /metrics HTTP/1.1" 200 36275 "" "Prometheus/2.51.0" 2024-09-21T10:04:04.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:03 smithi160 ceph-mon[101879]: osdmap e520: 8 total, 8 up, 8 in 2024-09-21T10:04:04.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:03 smithi070 ceph-mon[107385]: osdmap e520: 8 total, 8 up, 8 in 2024-09-21T10:04:04.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:03 smithi070 ceph-mon[112139]: osdmap e520: 8 total, 8 up, 8 in 2024-09-21T10:04:04.744 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.OpenByID 2024-09-21T10:04:05.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:04 smithi160 ceph-mon[101879]: osdmap e521: 8 total, 8 up, 8 in 2024-09-21T10:04:05.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:04 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2124964304' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-13","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:04:05.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:04 smithi160 ceph-mon[101879]: pgmap v277: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 456 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:05.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:04 smithi070 ceph-mon[107385]: osdmap e521: 8 total, 8 up, 8 in 2024-09-21T10:04:05.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:04 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2124964304' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-13","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:04:05.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:04 smithi070 ceph-mon[107385]: pgmap v277: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 456 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:05.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:04 smithi070 ceph-mon[112139]: osdmap e521: 8 total, 8 up, 8 in 2024-09-21T10:04:05.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:04 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2124964304' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-13","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:04:05.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:04 smithi070 ceph-mon[112139]: pgmap v277: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 456 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:05 smithi160 ceph-mon[101879]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:04:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:05 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2124964304' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:04:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:05 smithi160 ceph-mon[101879]: osdmap e522: 8 total, 8 up, 8 in 2024-09-21T10:04:06.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:04:06.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:06.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:05 smithi070 ceph-mon[107385]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:04:06.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:05 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2124964304' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:04:06.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:05 smithi070 ceph-mon[107385]: osdmap e522: 8 total, 8 up, 8 in 2024-09-21T10:04:06.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:04:06.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:05 smithi070 ceph-mon[112139]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:04:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:05 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2124964304' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:04:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:05 smithi070 ceph-mon[112139]: osdmap e522: 8 total, 8 up, 8 in 2024-09-21T10:04:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:04:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:06.782 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.OpenByID (2039 ms) 2024-09-21T10:04:06.783 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.OpenSnapDNE 2024-09-21T10:04:06.806 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.OpenSnapDNE (23 ms) 2024-09-21T10:04:06.806 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.IsExclusiveLockOwner 2024-09-21T10:04:06.828 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.IsExclusiveLockOwner (23 ms) 2024-09-21T10:04:06.828 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.ResizeLocksImage 2024-09-21T10:04:06.855 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.ResizeLocksImage (26 ms) 2024-09-21T10:04:06.855 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.ResizeFailsToLockImage 2024-09-21T10:04:06.871 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.ResizeFailsToLockImage (17 ms) 2024-09-21T10:04:06.871 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.SnapCreateLocksImage 2024-09-21T10:04:07.094 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:06 smithi160 ceph-mon[101879]: osdmap e523: 8 total, 8 up, 8 in 2024-09-21T10:04:07.094 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:06 smithi160 ceph-mon[101879]: pgmap v280: 105 pgs: 105 active+clean; 582 KiB data, 456 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:07.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:06 smithi070 ceph-mon[112139]: osdmap e523: 8 total, 8 up, 8 in 2024-09-21T10:04:07.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:06 smithi070 ceph-mon[112139]: pgmap v280: 105 pgs: 105 active+clean; 582 KiB data, 456 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:07.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:06 smithi070 ceph-mon[107385]: osdmap e523: 8 total, 8 up, 8 in 2024-09-21T10:04:07.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:06 smithi070 ceph-mon[107385]: pgmap v280: 105 pgs: 105 active+clean; 582 KiB data, 456 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:08.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:07 smithi160 ceph-mon[101879]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:04:08.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:07 smithi160 ceph-mon[101879]: Cluster is now healthy 2024-09-21T10:04:08.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:07 smithi160 ceph-mon[101879]: osdmap e524: 8 total, 8 up, 8 in 2024-09-21T10:04:08.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:07 smithi070 ceph-mon[107385]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:04:08.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:07 smithi070 ceph-mon[107385]: Cluster is now healthy 2024-09-21T10:04:08.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:07 smithi070 ceph-mon[107385]: osdmap e524: 8 total, 8 up, 8 in 2024-09-21T10:04:08.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:07 smithi070 ceph-mon[112139]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:04:08.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:07 smithi070 ceph-mon[112139]: Cluster is now healthy 2024-09-21T10:04:08.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:07 smithi070 ceph-mon[112139]: osdmap e524: 8 total, 8 up, 8 in 2024-09-21T10:04:08.772 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.SnapCreateLocksImage (1901 ms) 2024-09-21T10:04:08.772 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.SnapCreateFailsToLockImage 2024-09-21T10:04:08.796 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.SnapCreateFailsToLockImage (24 ms) 2024-09-21T10:04:08.796 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.SnapRollbackLocksImage 2024-09-21T10:04:09.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:08 smithi160 ceph-mon[101879]: osdmap e525: 8 total, 8 up, 8 in 2024-09-21T10:04:09.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:08 smithi160 ceph-mon[101879]: pgmap v283: 105 pgs: 105 active+clean; 582 KiB data, 456 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T10:04:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:08 smithi070 ceph-mon[107385]: osdmap e525: 8 total, 8 up, 8 in 2024-09-21T10:04:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:08 smithi070 ceph-mon[107385]: pgmap v283: 105 pgs: 105 active+clean; 582 KiB data, 456 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T10:04:09.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:08 smithi070 ceph-mon[112139]: osdmap e525: 8 total, 8 up, 8 in 2024-09-21T10:04:09.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:08 smithi070 ceph-mon[112139]: pgmap v283: 105 pgs: 105 active+clean; 582 KiB data, 456 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T10:04:10.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:09 smithi160 ceph-mon[101879]: osdmap e526: 8 total, 8 up, 8 in 2024-09-21T10:04:10.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:09 smithi070 ceph-mon[107385]: osdmap e526: 8 total, 8 up, 8 in 2024-09-21T10:04:10.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:09 smithi070 ceph-mon[112139]: osdmap e526: 8 total, 8 up, 8 in 2024-09-21T10:04:10.793 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.SnapRollbackLocksImage (1996 ms) 2024-09-21T10:04:10.793 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.SnapRollbackFailsToLockImage 2024-09-21T10:04:11.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:10 smithi160 ceph-mon[101879]: osdmap e527: 8 total, 8 up, 8 in 2024-09-21T10:04:11.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:10 smithi160 ceph-mon[101879]: pgmap v286: 105 pgs: 105 active+clean; 582 KiB data, 456 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:11.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:10 smithi070 ceph-mon[107385]: osdmap e527: 8 total, 8 up, 8 in 2024-09-21T10:04:11.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:10 smithi070 ceph-mon[107385]: pgmap v286: 105 pgs: 105 active+clean; 582 KiB data, 456 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:11.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:10 smithi070 ceph-mon[112139]: osdmap e527: 8 total, 8 up, 8 in 2024-09-21T10:04:11.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:10 smithi070 ceph-mon[112139]: pgmap v286: 105 pgs: 105 active+clean; 582 KiB data, 456 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:12.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:11 smithi160 ceph-mon[101879]: osdmap e528: 8 total, 8 up, 8 in 2024-09-21T10:04:12.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:11 smithi070 ceph-mon[112139]: osdmap e528: 8 total, 8 up, 8 in 2024-09-21T10:04:12.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:11 smithi070 ceph-mon[107385]: osdmap e528: 8 total, 8 up, 8 in 2024-09-21T10:04:12.793 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.SnapRollbackFailsToLockImage (2001 ms) 2024-09-21T10:04:12.793 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.SnapSetReleasesLock 2024-09-21T10:04:13.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:12 smithi160 ceph-mon[101879]: osdmap e529: 8 total, 8 up, 8 in 2024-09-21T10:04:13.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:12 smithi160 ceph-mon[101879]: pgmap v289: 105 pgs: 105 active+clean; 582 KiB data, 457 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 13 KiB/s wr, 172 op/s 2024-09-21T10:04:13.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:12 smithi070 ceph-mon[107385]: osdmap e529: 8 total, 8 up, 8 in 2024-09-21T10:04:13.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:12 smithi070 ceph-mon[107385]: pgmap v289: 105 pgs: 105 active+clean; 582 KiB data, 457 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 13 KiB/s wr, 172 op/s 2024-09-21T10:04:13.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:12 smithi070 ceph-mon[112139]: osdmap e529: 8 total, 8 up, 8 in 2024-09-21T10:04:13.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:12 smithi070 ceph-mon[112139]: pgmap v289: 105 pgs: 105 active+clean; 582 KiB data, 457 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 13 KiB/s wr, 172 op/s 2024-09-21T10:04:13.630 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:04:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:04:13] "GET /metrics HTTP/1.1" 200 36220 "" "Prometheus/2.51.0" 2024-09-21T10:04:14.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:13 smithi160 ceph-mon[101879]: osdmap e530: 8 total, 8 up, 8 in 2024-09-21T10:04:14.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:13 smithi070 ceph-mon[107385]: osdmap e530: 8 total, 8 up, 8 in 2024-09-21T10:04:14.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:13 smithi070 ceph-mon[112139]: osdmap e530: 8 total, 8 up, 8 in 2024-09-21T10:04:14.807 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.SnapSetReleasesLock (2012 ms) 2024-09-21T10:04:14.807 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.FlattenLocksImage 2024-09-21T10:04:15.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:14 smithi160 ceph-mon[101879]: osdmap e531: 8 total, 8 up, 8 in 2024-09-21T10:04:15.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:14 smithi160 ceph-mon[101879]: pgmap v292: 105 pgs: 105 active+clean; 582 KiB data, 457 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 13 KiB/s wr, 172 op/s 2024-09-21T10:04:15.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:14 smithi070 ceph-mon[107385]: osdmap e531: 8 total, 8 up, 8 in 2024-09-21T10:04:15.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:14 smithi070 ceph-mon[107385]: pgmap v292: 105 pgs: 105 active+clean; 582 KiB data, 457 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 13 KiB/s wr, 172 op/s 2024-09-21T10:04:15.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:14 smithi070 ceph-mon[112139]: osdmap e531: 8 total, 8 up, 8 in 2024-09-21T10:04:15.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:14 smithi070 ceph-mon[112139]: pgmap v292: 105 pgs: 105 active+clean; 582 KiB data, 457 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 13 KiB/s wr, 172 op/s 2024-09-21T10:04:16.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:15 smithi160 ceph-mon[101879]: osdmap e532: 8 total, 8 up, 8 in 2024-09-21T10:04:16.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:15 smithi070 ceph-mon[107385]: osdmap e532: 8 total, 8 up, 8 in 2024-09-21T10:04:16.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:15 smithi070 ceph-mon[112139]: osdmap e532: 8 total, 8 up, 8 in 2024-09-21T10:04:16.828 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.FlattenLocksImage (2023 ms) 2024-09-21T10:04:16.828 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.FlattenFailsToLockImage 2024-09-21T10:04:17.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:16 smithi160 ceph-mon[101879]: osdmap e533: 8 total, 8 up, 8 in 2024-09-21T10:04:17.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:16 smithi160 ceph-mon[101879]: pgmap v295: 105 pgs: 105 active+clean; 583 KiB data, 462 MiB used, 715 GiB / 715 GiB avail; 127 KiB/s rd, 9.2 KiB/s wr, 173 op/s 2024-09-21T10:04:17.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:16 smithi070 ceph-mon[107385]: osdmap e533: 8 total, 8 up, 8 in 2024-09-21T10:04:17.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:16 smithi070 ceph-mon[107385]: pgmap v295: 105 pgs: 105 active+clean; 583 KiB data, 462 MiB used, 715 GiB / 715 GiB avail; 127 KiB/s rd, 9.2 KiB/s wr, 173 op/s 2024-09-21T10:04:17.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:16 smithi070 ceph-mon[112139]: osdmap e533: 8 total, 8 up, 8 in 2024-09-21T10:04:17.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:16 smithi070 ceph-mon[112139]: pgmap v295: 105 pgs: 105 active+clean; 583 KiB data, 462 MiB used, 715 GiB / 715 GiB avail; 127 KiB/s rd, 9.2 KiB/s wr, 173 op/s 2024-09-21T10:04:18.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:17 smithi160 ceph-mon[101879]: osdmap e534: 8 total, 8 up, 8 in 2024-09-21T10:04:18.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:17 smithi160 ceph-mon[101879]: osdmap e535: 8 total, 8 up, 8 in 2024-09-21T10:04:18.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:17 smithi070 ceph-mon[107385]: osdmap e534: 8 total, 8 up, 8 in 2024-09-21T10:04:18.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:17 smithi070 ceph-mon[107385]: osdmap e535: 8 total, 8 up, 8 in 2024-09-21T10:04:18.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:17 smithi070 ceph-mon[112139]: osdmap e534: 8 total, 8 up, 8 in 2024-09-21T10:04:18.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:17 smithi070 ceph-mon[112139]: osdmap e535: 8 total, 8 up, 8 in 2024-09-21T10:04:18.826 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.FlattenFailsToLockImage (1998 ms) 2024-09-21T10:04:18.827 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.WriteFailsToLockImageBlocklisted 2024-09-21T10:04:19.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:18 smithi160 ceph-mon[101879]: pgmap v298: 105 pgs: 105 active+clean; 583 KiB data, 462 MiB used, 715 GiB / 715 GiB avail; 127 KiB/s rd, 9.2 KiB/s wr, 173 op/s 2024-09-21T10:04:19.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:18 smithi070 ceph-mon[107385]: pgmap v298: 105 pgs: 105 active+clean; 583 KiB data, 462 MiB used, 715 GiB / 715 GiB avail; 127 KiB/s rd, 9.2 KiB/s wr, 173 op/s 2024-09-21T10:04:19.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:18 smithi070 ceph-mon[112139]: pgmap v298: 105 pgs: 105 active+clean; 583 KiB data, 462 MiB used, 715 GiB / 715 GiB avail; 127 KiB/s rd, 9.2 KiB/s wr, 173 op/s 2024-09-21T10:04:19.821 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.WriteFailsToLockImageBlocklisted (995 ms) 2024-09-21T10:04:19.821 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.WriteFailsToLockImageBlocklistedWatch 2024-09-21T10:04:20.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:19 smithi160 ceph-mon[101879]: osdmap e536: 8 total, 8 up, 8 in 2024-09-21T10:04:20.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:19 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/694065139' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.70:0/694065139"} : dispatch 2024-09-21T10:04:20.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:19 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.70:0/694065139"} : dispatch 2024-09-21T10:04:20.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:19 smithi070 ceph-mon[107385]: osdmap e536: 8 total, 8 up, 8 in 2024-09-21T10:04:20.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:19 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/694065139' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.70:0/694065139"} : dispatch 2024-09-21T10:04:20.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:19 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.70:0/694065139"} : dispatch 2024-09-21T10:04:20.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:19 smithi070 ceph-mon[112139]: osdmap e536: 8 total, 8 up, 8 in 2024-09-21T10:04:20.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:19 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/694065139' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.70:0/694065139"} : dispatch 2024-09-21T10:04:20.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:19 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.70:0/694065139"} : dispatch 2024-09-21T10:04:21.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:20 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.70:0/694065139"}]': finished 2024-09-21T10:04:21.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:20 smithi160 ceph-mon[101879]: osdmap e537: 8 total, 8 up, 8 in 2024-09-21T10:04:21.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:20 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4293543511' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.70:0/4293543511"} : dispatch 2024-09-21T10:04:21.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:20 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.70:0/4293543511"} : dispatch 2024-09-21T10:04:21.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:20 smithi160 ceph-mon[101879]: pgmap v301: 105 pgs: 105 active+clean; 583 KiB data, 462 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:21.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:04:21.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:21.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:20 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.70:0/694065139"}]': finished 2024-09-21T10:04:21.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:20 smithi070 ceph-mon[107385]: osdmap e537: 8 total, 8 up, 8 in 2024-09-21T10:04:21.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:20 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4293543511' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.70:0/4293543511"} : dispatch 2024-09-21T10:04:21.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:20 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.70:0/4293543511"} : dispatch 2024-09-21T10:04:21.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:20 smithi070 ceph-mon[107385]: pgmap v301: 105 pgs: 105 active+clean; 583 KiB data, 462 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:21.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:04:21.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:21.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:20 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.70:0/694065139"}]': finished 2024-09-21T10:04:21.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:20 smithi070 ceph-mon[112139]: osdmap e537: 8 total, 8 up, 8 in 2024-09-21T10:04:21.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:20 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4293543511' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.70:0/4293543511"} : dispatch 2024-09-21T10:04:21.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:20 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.70:0/4293543511"} : dispatch 2024-09-21T10:04:21.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:20 smithi070 ceph-mon[112139]: pgmap v301: 105 pgs: 105 active+clean; 583 KiB data, 462 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:21.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:04:21.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:21.823 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.WriteFailsToLockImageBlocklistedWatch (2002 ms) 2024-09-21T10:04:21.823 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.AioWriteRequestsLock 2024-09-21T10:04:21.857 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.AioWriteRequestsLock (34 ms) 2024-09-21T10:04:21.857 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.AioDiscardRequestsLock 2024-09-21T10:04:21.883 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.AioDiscardRequestsLock (26 ms) 2024-09-21T10:04:21.883 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.CancelAsyncResize 2024-09-21T10:04:21.922 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.CancelAsyncResize (39 ms) 2024-09-21T10:04:21.922 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.MultipleResize 2024-09-21T10:04:21.972 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.MultipleResize (50 ms) 2024-09-21T10:04:21.972 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.Metadata 2024-09-21T10:04:22.015 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.Metadata (43 ms) 2024-09-21T10:04:22.015 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.MetadataConfApply 2024-09-21T10:04:22.042 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.MetadataConfApply (27 ms) 2024-09-21T10:04:22.042 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.SnapshotCopyup 2024-09-21T10:04:22.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:21 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.70:0/4293543511"}]': finished 2024-09-21T10:04:22.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:21 smithi160 ceph-mon[101879]: osdmap e538: 8 total, 8 up, 8 in 2024-09-21T10:04:22.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:21 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.70:0/4293543511"}]': finished 2024-09-21T10:04:22.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:21 smithi070 ceph-mon[107385]: osdmap e538: 8 total, 8 up, 8 in 2024-09-21T10:04:22.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:21 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.70:0/4293543511"}]': finished 2024-09-21T10:04:22.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:21 smithi070 ceph-mon[112139]: osdmap e538: 8 total, 8 up, 8 in 2024-09-21T10:04:23.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:22 smithi160 ceph-mon[101879]: pgmap v303: 105 pgs: 105 active+clean; 583 KiB data, 467 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 7.4 KiB/s wr, 147 op/s 2024-09-21T10:04:23.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:22 smithi070 ceph-mon[112139]: pgmap v303: 105 pgs: 105 active+clean; 583 KiB data, 467 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 7.4 KiB/s wr, 147 op/s 2024-09-21T10:04:23.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:22 smithi070 ceph-mon[107385]: pgmap v303: 105 pgs: 105 active+clean; 583 KiB data, 467 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 7.4 KiB/s wr, 147 op/s 2024-09-21T10:04:23.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:04:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:04:23] "GET /metrics HTTP/1.1" 200 36267 "" "Prometheus/2.51.0" 2024-09-21T10:04:24.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:23 smithi160 ceph-mon[101879]: osdmap e539: 8 total, 8 up, 8 in 2024-09-21T10:04:24.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:23 smithi070 ceph-mon[112139]: osdmap e539: 8 total, 8 up, 8 in 2024-09-21T10:04:24.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:23 smithi070 ceph-mon[107385]: osdmap e539: 8 total, 8 up, 8 in 2024-09-21T10:04:24.933 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.SnapshotCopyup (2891 ms) 2024-09-21T10:04:24.934 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.SnapshotCopyupZeros 2024-09-21T10:04:25.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:24 smithi160 ceph-mon[101879]: osdmap e540: 8 total, 8 up, 8 in 2024-09-21T10:04:25.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:24 smithi160 ceph-mon[101879]: pgmap v306: 105 pgs: 105 active+clean; 583 KiB data, 467 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 7.7 KiB/s wr, 152 op/s 2024-09-21T10:04:25.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:24 smithi070 ceph-mon[107385]: osdmap e540: 8 total, 8 up, 8 in 2024-09-21T10:04:25.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:24 smithi070 ceph-mon[107385]: pgmap v306: 105 pgs: 105 active+clean; 583 KiB data, 467 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 7.7 KiB/s wr, 152 op/s 2024-09-21T10:04:25.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:24 smithi070 ceph-mon[112139]: osdmap e540: 8 total, 8 up, 8 in 2024-09-21T10:04:25.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:24 smithi070 ceph-mon[112139]: pgmap v306: 105 pgs: 105 active+clean; 583 KiB data, 467 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 7.7 KiB/s wr, 152 op/s 2024-09-21T10:04:26.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:25 smithi070 ceph-mon[107385]: osdmap e541: 8 total, 8 up, 8 in 2024-09-21T10:04:26.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:25 smithi070 ceph-mon[112139]: osdmap e541: 8 total, 8 up, 8 in 2024-09-21T10:04:26.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:25 smithi160 ceph-mon[101879]: osdmap e541: 8 total, 8 up, 8 in 2024-09-21T10:04:26.886 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.SnapshotCopyupZeros (1953 ms) 2024-09-21T10:04:26.886 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.SnapshotCopyupZerosMigration 2024-09-21T10:04:27.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:26 smithi070 ceph-mon[107385]: osdmap e542: 8 total, 8 up, 8 in 2024-09-21T10:04:27.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:26 smithi070 ceph-mon[107385]: pgmap v309: 105 pgs: 105 active+clean; 586 KiB data, 467 MiB used, 715 GiB / 715 GiB avail; 299 KiB/s rd, 32 KiB/s wr, 416 op/s 2024-09-21T10:04:27.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:26 smithi070 ceph-mon[112139]: osdmap e542: 8 total, 8 up, 8 in 2024-09-21T10:04:27.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:26 smithi070 ceph-mon[112139]: pgmap v309: 105 pgs: 105 active+clean; 586 KiB data, 467 MiB used, 715 GiB / 715 GiB avail; 299 KiB/s rd, 32 KiB/s wr, 416 op/s 2024-09-21T10:04:27.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:26 smithi160 ceph-mon[101879]: osdmap e542: 8 total, 8 up, 8 in 2024-09-21T10:04:27.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:26 smithi160 ceph-mon[101879]: pgmap v309: 105 pgs: 105 active+clean; 586 KiB data, 467 MiB used, 715 GiB / 715 GiB avail; 299 KiB/s rd, 32 KiB/s wr, 416 op/s 2024-09-21T10:04:27.692 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.SnapshotCopyupZerosMigration (806 ms) 2024-09-21T10:04:27.692 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.ResizeCopyup 2024-09-21T10:04:28.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:27 smithi070 ceph-mon[107385]: osdmap e543: 8 total, 8 up, 8 in 2024-09-21T10:04:28.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:27 smithi070 ceph-mon[107385]: osdmap e544: 8 total, 8 up, 8 in 2024-09-21T10:04:28.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:27 smithi070 ceph-mon[112139]: osdmap e543: 8 total, 8 up, 8 in 2024-09-21T10:04:28.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:27 smithi070 ceph-mon[112139]: osdmap e544: 8 total, 8 up, 8 in 2024-09-21T10:04:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:27 smithi160 ceph-mon[101879]: osdmap e543: 8 total, 8 up, 8 in 2024-09-21T10:04:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:27 smithi160 ceph-mon[101879]: osdmap e544: 8 total, 8 up, 8 in 2024-09-21T10:04:29.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:28 smithi070 ceph-mon[107385]: pgmap v312: 105 pgs: 105 active+clean; 586 KiB data, 467 MiB used, 715 GiB / 715 GiB avail; 299 KiB/s rd, 32 KiB/s wr, 416 op/s 2024-09-21T10:04:29.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:28 smithi070 ceph-mon[107385]: osdmap e545: 8 total, 8 up, 8 in 2024-09-21T10:04:29.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:28 smithi070 ceph-mon[112139]: pgmap v312: 105 pgs: 105 active+clean; 586 KiB data, 467 MiB used, 715 GiB / 715 GiB avail; 299 KiB/s rd, 32 KiB/s wr, 416 op/s 2024-09-21T10:04:29.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:28 smithi070 ceph-mon[112139]: osdmap e545: 8 total, 8 up, 8 in 2024-09-21T10:04:29.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:28 smithi160 ceph-mon[101879]: pgmap v312: 105 pgs: 105 active+clean; 586 KiB data, 467 MiB used, 715 GiB / 715 GiB avail; 299 KiB/s rd, 32 KiB/s wr, 416 op/s 2024-09-21T10:04:29.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:28 smithi160 ceph-mon[101879]: osdmap e545: 8 total, 8 up, 8 in 2024-09-21T10:04:29.723 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.ResizeCopyup (2031 ms) 2024-09-21T10:04:29.723 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.DiscardCopyup 2024-09-21T10:04:29.732 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/test_internal.cc:1056: Skipped 2024-09-21T10:04:29.732 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T10:04:29.732 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestInternal.DiscardCopyup (9 ms) 2024-09-21T10:04:29.733 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.ImageOptions 2024-09-21T10:04:29.740 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.ImageOptions (8 ms) 2024-09-21T10:04:29.741 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.WriteFullCopyup 2024-09-21T10:04:31.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:30 smithi160 ceph-mon[101879]: osdmap e546: 8 total, 8 up, 8 in 2024-09-21T10:04:31.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:30 smithi160 ceph-mon[101879]: pgmap v315: 105 pgs: 105 active+clean; 586 KiB data, 467 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:31.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:30 smithi070 ceph-mon[112139]: osdmap e546: 8 total, 8 up, 8 in 2024-09-21T10:04:31.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:30 smithi070 ceph-mon[112139]: pgmap v315: 105 pgs: 105 active+clean; 586 KiB data, 467 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:31.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:30 smithi070 ceph-mon[107385]: osdmap e546: 8 total, 8 up, 8 in 2024-09-21T10:04:31.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:30 smithi070 ceph-mon[107385]: pgmap v315: 105 pgs: 105 active+clean; 586 KiB data, 467 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:32.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:31 smithi160 ceph-mon[101879]: osdmap e547: 8 total, 8 up, 8 in 2024-09-21T10:04:32.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:31 smithi070 ceph-mon[112139]: osdmap e547: 8 total, 8 up, 8 in 2024-09-21T10:04:32.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:31 smithi070 ceph-mon[107385]: osdmap e547: 8 total, 8 up, 8 in 2024-09-21T10:04:33.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:32 smithi160 ceph-mon[101879]: osdmap e548: 8 total, 8 up, 8 in 2024-09-21T10:04:33.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:32 smithi160 ceph-mon[101879]: pgmap v318: 105 pgs: 105 active+clean; 4.6 MiB data, 468 MiB used, 715 GiB / 715 GiB avail; 226 KiB/s rd, 1.0 MiB/s wr, 314 op/s 2024-09-21T10:04:33.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:32 smithi070 ceph-mon[107385]: osdmap e548: 8 total, 8 up, 8 in 2024-09-21T10:04:33.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:32 smithi070 ceph-mon[107385]: pgmap v318: 105 pgs: 105 active+clean; 4.6 MiB data, 468 MiB used, 715 GiB / 715 GiB avail; 226 KiB/s rd, 1.0 MiB/s wr, 314 op/s 2024-09-21T10:04:33.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:32 smithi070 ceph-mon[112139]: osdmap e548: 8 total, 8 up, 8 in 2024-09-21T10:04:33.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:32 smithi070 ceph-mon[112139]: pgmap v318: 105 pgs: 105 active+clean; 4.6 MiB data, 468 MiB used, 715 GiB / 715 GiB avail; 226 KiB/s rd, 1.0 MiB/s wr, 314 op/s 2024-09-21T10:04:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:04:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:04:33] "GET /metrics HTTP/1.1" 200 36267 "" "Prometheus/2.51.0" 2024-09-21T10:04:33.708 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.WriteFullCopyup (3967 ms) 2024-09-21T10:04:33.708 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.DiffIterateCloneOverwrite 2024-09-21T10:04:34.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:33 smithi160 ceph-mon[101879]: osdmap e549: 8 total, 8 up, 8 in 2024-09-21T10:04:34.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:33 smithi070 ceph-mon[112139]: osdmap e549: 8 total, 8 up, 8 in 2024-09-21T10:04:34.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:33 smithi070 ceph-mon[107385]: osdmap e549: 8 total, 8 up, 8 in 2024-09-21T10:04:35.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:34 smithi160 ceph-mon[101879]: osdmap e550: 8 total, 8 up, 8 in 2024-09-21T10:04:35.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:34 smithi160 ceph-mon[101879]: pgmap v321: 105 pgs: 105 active+clean; 4.6 MiB data, 468 MiB used, 715 GiB / 715 GiB avail; 226 KiB/s rd, 1.0 MiB/s wr, 314 op/s 2024-09-21T10:04:35.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:34 smithi070 ceph-mon[107385]: osdmap e550: 8 total, 8 up, 8 in 2024-09-21T10:04:35.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:34 smithi070 ceph-mon[107385]: pgmap v321: 105 pgs: 105 active+clean; 4.6 MiB data, 468 MiB used, 715 GiB / 715 GiB avail; 226 KiB/s rd, 1.0 MiB/s wr, 314 op/s 2024-09-21T10:04:35.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:34 smithi070 ceph-mon[112139]: osdmap e550: 8 total, 8 up, 8 in 2024-09-21T10:04:35.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:34 smithi070 ceph-mon[112139]: pgmap v321: 105 pgs: 105 active+clean; 4.6 MiB data, 468 MiB used, 715 GiB / 715 GiB avail; 226 KiB/s rd, 1.0 MiB/s wr, 314 op/s 2024-09-21T10:04:35.726 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.DiffIterateCloneOverwrite (2018 ms) 2024-09-21T10:04:35.726 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.TestCoR 2024-09-21T10:04:35.735 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/test_internal.cc:1313: Skipped 2024-09-21T10:04:35.735 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to disabled copy-on-read 2024-09-21T10:04:35.735 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestInternal.TestCoR (9 ms) 2024-09-21T10:04:35.735 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.FlattenNoEmptyObjects 2024-09-21T10:04:35.940 INFO:tasks.workunit.client.0.smithi070.stdout:made parent image "image399": 4194304K (1024 * 4K) 2024-09-21T10:04:35.940 INFO:tasks.workunit.client.0.smithi070.stdout:generated random write map: 2024-09-21T10:04:35.941 INFO:tasks.workunit.client.0.smithi070.stdout: [8 , 34110 ] 2024-09-21T10:04:35.941 INFO:tasks.workunit.client.0.smithi070.stdout: [9 , 37659 ] 2024-09-21T10:04:35.941 INFO:tasks.workunit.client.0.smithi070.stdout: [16 , 68788 ] 2024-09-21T10:04:35.941 INFO:tasks.workunit.client.0.smithi070.stdout: [23 , 96949 ] 2024-09-21T10:04:35.941 INFO:tasks.workunit.client.0.smithi070.stdout: [49 , 202005 ] 2024-09-21T10:04:35.941 INFO:tasks.workunit.client.0.smithi070.stdout: [61 , 250858 ] 2024-09-21T10:04:35.941 INFO:tasks.workunit.client.0.smithi070.stdout: [63 , 260040 ] 2024-09-21T10:04:35.941 INFO:tasks.workunit.client.0.smithi070.stdout: [71 , 293985 ] 2024-09-21T10:04:35.941 INFO:tasks.workunit.client.0.smithi070.stdout: [78 , 322555 ] 2024-09-21T10:04:35.941 INFO:tasks.workunit.client.0.smithi070.stdout: [81 , 332937 ] 2024-09-21T10:04:35.941 INFO:tasks.workunit.client.0.smithi070.stdout: [91 , 375503 ] 2024-09-21T10:04:35.941 INFO:tasks.workunit.client.0.smithi070.stdout: [100 , 412907 ] 2024-09-21T10:04:35.941 INFO:tasks.workunit.client.0.smithi070.stdout: [104 , 427312 ] 2024-09-21T10:04:35.941 INFO:tasks.workunit.client.0.smithi070.stdout: [106 , 436745 ] 2024-09-21T10:04:35.942 INFO:tasks.workunit.client.0.smithi070.stdout: [113 , 463760 ] 2024-09-21T10:04:35.942 INFO:tasks.workunit.client.0.smithi070.stdout: [119 , 489033 ] 2024-09-21T10:04:35.942 INFO:tasks.workunit.client.0.smithi070.stdout: [125 , 513985 ] 2024-09-21T10:04:35.942 INFO:tasks.workunit.client.0.smithi070.stdout: [158 , 648677 ] 2024-09-21T10:04:35.942 INFO:tasks.workunit.client.0.smithi070.stdout: [159 , 654755 ] 2024-09-21T10:04:35.942 INFO:tasks.workunit.client.0.smithi070.stdout: [167 , 684851 ] 2024-09-21T10:04:35.942 INFO:tasks.workunit.client.0.smithi070.stdout: [183 , 750771 ] 2024-09-21T10:04:35.942 INFO:tasks.workunit.client.0.smithi070.stdout: [198 , 812287 ] 2024-09-21T10:04:35.942 INFO:tasks.workunit.client.0.smithi070.stdout: [201 , 826763 ] 2024-09-21T10:04:35.942 INFO:tasks.workunit.client.0.smithi070.stdout: [206 , 846227 ] 2024-09-21T10:04:35.942 INFO:tasks.workunit.client.0.smithi070.stdout: [207 , 848166 ] 2024-09-21T10:04:35.942 INFO:tasks.workunit.client.0.smithi070.stdout: [217 , 891502 ] 2024-09-21T10:04:35.943 INFO:tasks.workunit.client.0.smithi070.stdout: [251 , 1029048 ] 2024-09-21T10:04:35.943 INFO:tasks.workunit.client.0.smithi070.stdout: [295 , 1209062 ] 2024-09-21T10:04:35.943 INFO:tasks.workunit.client.0.smithi070.stdout: [301 , 1232958 ] 2024-09-21T10:04:35.943 INFO:tasks.workunit.client.0.smithi070.stdout: [307 , 1258151 ] 2024-09-21T10:04:35.943 INFO:tasks.workunit.client.0.smithi070.stdout: [309 , 1266390 ] 2024-09-21T10:04:35.943 INFO:tasks.workunit.client.0.smithi070.stdout: [318 , 1302612 ] 2024-09-21T10:04:35.943 INFO:tasks.workunit.client.0.smithi070.stdout: [320 , 1313844 ] 2024-09-21T10:04:35.943 INFO:tasks.workunit.client.0.smithi070.stdout: [322 , 1320267 ] 2024-09-21T10:04:35.943 INFO:tasks.workunit.client.0.smithi070.stdout: [326 , 1337146 ] 2024-09-21T10:04:35.943 INFO:tasks.workunit.client.0.smithi070.stdout: [355 , 1454203 ] 2024-09-21T10:04:35.943 INFO:tasks.workunit.client.0.smithi070.stdout: [365 , 1495470 ] 2024-09-21T10:04:35.943 INFO:tasks.workunit.client.0.smithi070.stdout: [370 , 1517253 ] 2024-09-21T10:04:35.943 INFO:tasks.workunit.client.0.smithi070.stdout: [373 , 1529880 ] 2024-09-21T10:04:35.943 INFO:tasks.workunit.client.0.smithi070.stdout: [377 , 1544204 ] 2024-09-21T10:04:35.944 INFO:tasks.workunit.client.0.smithi070.stdout: [387 , 1588430 ] 2024-09-21T10:04:35.944 INFO:tasks.workunit.client.0.smithi070.stdout: [400 , 1640817 ] 2024-09-21T10:04:35.944 INFO:tasks.workunit.client.0.smithi070.stdout: [402 , 1649211 ] 2024-09-21T10:04:35.944 INFO:tasks.workunit.client.0.smithi070.stdout: [406 , 1666109 ] 2024-09-21T10:04:35.944 INFO:tasks.workunit.client.0.smithi070.stdout: [421 , 1726320 ] 2024-09-21T10:04:35.944 INFO:tasks.workunit.client.0.smithi070.stdout: [427 , 1750173 ] 2024-09-21T10:04:35.944 INFO:tasks.workunit.client.0.smithi070.stdout: [453 , 1856543 ] 2024-09-21T10:04:35.944 INFO:tasks.workunit.client.0.smithi070.stdout: [467 , 1914325 ] 2024-09-21T10:04:35.944 INFO:tasks.workunit.client.0.smithi070.stdout: [492 , 2016623 ] 2024-09-21T10:04:35.944 INFO:tasks.workunit.client.0.smithi070.stdout: [501 , 2053150 ] 2024-09-21T10:04:35.944 INFO:tasks.workunit.client.0.smithi070.stdout: [504 , 2067530 ] 2024-09-21T10:04:35.944 INFO:tasks.workunit.client.0.smithi070.stdout: [511 , 2093947 ] 2024-09-21T10:04:35.944 INFO:tasks.workunit.client.0.smithi070.stdout: [515 , 2112281 ] 2024-09-21T10:04:35.945 INFO:tasks.workunit.client.0.smithi070.stdout: [518 , 2122855 ] 2024-09-21T10:04:35.945 INFO:tasks.workunit.client.0.smithi070.stdout: [522 , 2141163 ] 2024-09-21T10:04:35.945 INFO:tasks.workunit.client.0.smithi070.stdout: [525 , 2151906 ] 2024-09-21T10:04:35.945 INFO:tasks.workunit.client.0.smithi070.stdout: [528 , 2165091 ] 2024-09-21T10:04:35.945 INFO:tasks.workunit.client.0.smithi070.stdout: [530 , 2172247 ] 2024-09-21T10:04:35.945 INFO:tasks.workunit.client.0.smithi070.stdout: [551 , 2260230 ] 2024-09-21T10:04:35.945 INFO:tasks.workunit.client.0.smithi070.stdout: [553 , 2267684 ] 2024-09-21T10:04:35.945 INFO:tasks.workunit.client.0.smithi070.stdout: [559 , 2292565 ] 2024-09-21T10:04:35.945 INFO:tasks.workunit.client.0.smithi070.stdout: [565 , 2316133 ] 2024-09-21T10:04:35.945 INFO:tasks.workunit.client.0.smithi070.stdout: [576 , 2361893 ] 2024-09-21T10:04:35.945 INFO:tasks.workunit.client.0.smithi070.stdout: [585 , 2398287 ] 2024-09-21T10:04:35.945 INFO:tasks.workunit.client.0.smithi070.stdout: [612 , 2509547 ] 2024-09-21T10:04:35.945 INFO:tasks.workunit.client.0.smithi070.stdout: [633 , 2593506 ] 2024-09-21T10:04:35.945 INFO:tasks.workunit.client.0.smithi070.stdout: [646 , 2648553 ] 2024-09-21T10:04:35.945 INFO:tasks.workunit.client.0.smithi070.stdout: [648 , 2657260 ] 2024-09-21T10:04:35.946 INFO:tasks.workunit.client.0.smithi070.stdout: [661 , 2710169 ] 2024-09-21T10:04:35.946 INFO:tasks.workunit.client.0.smithi070.stdout: [671 , 2751279 ] 2024-09-21T10:04:35.946 INFO:tasks.workunit.client.0.smithi070.stdout: [687 , 2817429 ] 2024-09-21T10:04:35.946 INFO:tasks.workunit.client.0.smithi070.stdout: [692 , 2834987 ] 2024-09-21T10:04:35.946 INFO:tasks.workunit.client.0.smithi070.stdout: [693 , 2839125 ] 2024-09-21T10:04:35.946 INFO:tasks.workunit.client.0.smithi070.stdout: [706 , 2894305 ] 2024-09-21T10:04:35.946 INFO:tasks.workunit.client.0.smithi070.stdout: [713 , 2922517 ] 2024-09-21T10:04:35.946 INFO:tasks.workunit.client.0.smithi070.stdout: [732 , 2999568 ] 2024-09-21T10:04:35.946 INFO:tasks.workunit.client.0.smithi070.stdout: [746 , 3056485 ] 2024-09-21T10:04:35.946 INFO:tasks.workunit.client.0.smithi070.stdout: [754 , 3091091 ] 2024-09-21T10:04:35.946 INFO:tasks.workunit.client.0.smithi070.stdout: [776 , 3180462 ] 2024-09-21T10:04:35.946 INFO:tasks.workunit.client.0.smithi070.stdout: [783 , 3209340 ] 2024-09-21T10:04:35.946 INFO:tasks.workunit.client.0.smithi070.stdout: [789 , 3233584 ] 2024-09-21T10:04:35.946 INFO:tasks.workunit.client.0.smithi070.stdout: [797 , 3266937 ] 2024-09-21T10:04:35.947 INFO:tasks.workunit.client.0.smithi070.stdout: [800 , 3276977 ] 2024-09-21T10:04:35.947 INFO:tasks.workunit.client.0.smithi070.stdout: [812 , 3329357 ] 2024-09-21T10:04:35.947 INFO:tasks.workunit.client.0.smithi070.stdout: [823 , 3371198 ] 2024-09-21T10:04:35.947 INFO:tasks.workunit.client.0.smithi070.stdout: [842 , 3449014 ] 2024-09-21T10:04:35.947 INFO:tasks.workunit.client.0.smithi070.stdout: [862 , 3533824 ] 2024-09-21T10:04:35.947 INFO:tasks.workunit.client.0.smithi070.stdout: [868 , 3556784 ] 2024-09-21T10:04:35.947 INFO:tasks.workunit.client.0.smithi070.stdout: [882 , 3612705 ] 2024-09-21T10:04:35.947 INFO:tasks.workunit.client.0.smithi070.stdout: [910 , 3729071 ] 2024-09-21T10:04:35.947 INFO:tasks.workunit.client.0.smithi070.stdout: [922 , 3779864 ] 2024-09-21T10:04:35.947 INFO:tasks.workunit.client.0.smithi070.stdout: [924 , 3785031 ] 2024-09-21T10:04:35.947 INFO:tasks.workunit.client.0.smithi070.stdout: [946 , 3878326 ] 2024-09-21T10:04:35.947 INFO:tasks.workunit.client.0.smithi070.stdout: [983 , 4027719 ] 2024-09-21T10:04:35.947 INFO:tasks.workunit.client.0.smithi070.stdout: [996 , 4083049 ] 2024-09-21T10:04:35.947 INFO:tasks.workunit.client.0.smithi070.stdout: [1000 , 4099574 ] 2024-09-21T10:04:35.947 INFO:tasks.workunit.client.0.smithi070.stdout: [1004 , 4114397 ] 2024-09-21T10:04:35.948 INFO:tasks.workunit.client.0.smithi070.stdout:write data based on random map 2024-09-21T10:04:35.948 INFO:tasks.workunit.client.0.smithi070.stdout: write object-8 2024-09-21T10:04:35.948 INFO:tasks.workunit.client.0.smithi070.stdout: write object-9 2024-09-21T10:04:35.948 INFO:tasks.workunit.client.0.smithi070.stdout: write object-16 2024-09-21T10:04:35.948 INFO:tasks.workunit.client.0.smithi070.stdout: write object-23 2024-09-21T10:04:35.948 INFO:tasks.workunit.client.0.smithi070.stdout: write object-49 2024-09-21T10:04:35.948 INFO:tasks.workunit.client.0.smithi070.stdout: write object-61 2024-09-21T10:04:35.948 INFO:tasks.workunit.client.0.smithi070.stdout: write object-63 2024-09-21T10:04:35.948 INFO:tasks.workunit.client.0.smithi070.stdout: write object-71 2024-09-21T10:04:35.948 INFO:tasks.workunit.client.0.smithi070.stdout: write object-78 2024-09-21T10:04:35.948 INFO:tasks.workunit.client.0.smithi070.stdout: write object-81 2024-09-21T10:04:35.948 INFO:tasks.workunit.client.0.smithi070.stdout: write object-91 2024-09-21T10:04:35.948 INFO:tasks.workunit.client.0.smithi070.stdout: write object-100 2024-09-21T10:04:35.948 INFO:tasks.workunit.client.0.smithi070.stdout: write object-104 2024-09-21T10:04:35.948 INFO:tasks.workunit.client.0.smithi070.stdout: write object-106 2024-09-21T10:04:35.949 INFO:tasks.workunit.client.0.smithi070.stdout: write object-113 2024-09-21T10:04:35.949 INFO:tasks.workunit.client.0.smithi070.stdout: write object-119 2024-09-21T10:04:35.949 INFO:tasks.workunit.client.0.smithi070.stdout: write object-125 2024-09-21T10:04:35.949 INFO:tasks.workunit.client.0.smithi070.stdout: write object-158 2024-09-21T10:04:35.949 INFO:tasks.workunit.client.0.smithi070.stdout: write object-159 2024-09-21T10:04:35.949 INFO:tasks.workunit.client.0.smithi070.stdout: write object-167 2024-09-21T10:04:35.949 INFO:tasks.workunit.client.0.smithi070.stdout: write object-183 2024-09-21T10:04:35.949 INFO:tasks.workunit.client.0.smithi070.stdout: write object-198 2024-09-21T10:04:35.949 INFO:tasks.workunit.client.0.smithi070.stdout: write object-201 2024-09-21T10:04:35.949 INFO:tasks.workunit.client.0.smithi070.stdout: write object-206 2024-09-21T10:04:35.949 INFO:tasks.workunit.client.0.smithi070.stdout: write object-207 2024-09-21T10:04:35.949 INFO:tasks.workunit.client.0.smithi070.stdout: write object-217 2024-09-21T10:04:35.949 INFO:tasks.workunit.client.0.smithi070.stdout: write object-251 2024-09-21T10:04:35.949 INFO:tasks.workunit.client.0.smithi070.stdout: write object-295 2024-09-21T10:04:35.950 INFO:tasks.workunit.client.0.smithi070.stdout: write object-301 2024-09-21T10:04:35.950 INFO:tasks.workunit.client.0.smithi070.stdout: write object-307 2024-09-21T10:04:35.950 INFO:tasks.workunit.client.0.smithi070.stdout: write object-309 2024-09-21T10:04:35.950 INFO:tasks.workunit.client.0.smithi070.stdout: write object-318 2024-09-21T10:04:35.950 INFO:tasks.workunit.client.0.smithi070.stdout: write object-320 2024-09-21T10:04:35.950 INFO:tasks.workunit.client.0.smithi070.stdout: write object-322 2024-09-21T10:04:35.950 INFO:tasks.workunit.client.0.smithi070.stdout: write object-326 2024-09-21T10:04:35.950 INFO:tasks.workunit.client.0.smithi070.stdout: write object-355 2024-09-21T10:04:35.950 INFO:tasks.workunit.client.0.smithi070.stdout: write object-365 2024-09-21T10:04:35.950 INFO:tasks.workunit.client.0.smithi070.stdout: write object-370 2024-09-21T10:04:35.950 INFO:tasks.workunit.client.0.smithi070.stdout: write object-373 2024-09-21T10:04:35.950 INFO:tasks.workunit.client.0.smithi070.stdout: write object-377 2024-09-21T10:04:35.950 INFO:tasks.workunit.client.0.smithi070.stdout: write object-387 2024-09-21T10:04:35.950 INFO:tasks.workunit.client.0.smithi070.stdout: write object-400 2024-09-21T10:04:35.950 INFO:tasks.workunit.client.0.smithi070.stdout: write object-402 2024-09-21T10:04:35.951 INFO:tasks.workunit.client.0.smithi070.stdout: write object-406 2024-09-21T10:04:35.951 INFO:tasks.workunit.client.0.smithi070.stdout: write object-421 2024-09-21T10:04:35.951 INFO:tasks.workunit.client.0.smithi070.stdout: write object-427 2024-09-21T10:04:35.951 INFO:tasks.workunit.client.0.smithi070.stdout: write object-453 2024-09-21T10:04:35.951 INFO:tasks.workunit.client.0.smithi070.stdout: write object-467 2024-09-21T10:04:35.951 INFO:tasks.workunit.client.0.smithi070.stdout: write object-492 2024-09-21T10:04:35.951 INFO:tasks.workunit.client.0.smithi070.stdout: write object-501 2024-09-21T10:04:35.951 INFO:tasks.workunit.client.0.smithi070.stdout: write object-504 2024-09-21T10:04:35.951 INFO:tasks.workunit.client.0.smithi070.stdout: write object-511 2024-09-21T10:04:35.951 INFO:tasks.workunit.client.0.smithi070.stdout: write object-515 2024-09-21T10:04:35.951 INFO:tasks.workunit.client.0.smithi070.stdout: write object-518 2024-09-21T10:04:35.951 INFO:tasks.workunit.client.0.smithi070.stdout: write object-522 2024-09-21T10:04:35.951 INFO:tasks.workunit.client.0.smithi070.stdout: write object-525 2024-09-21T10:04:35.951 INFO:tasks.workunit.client.0.smithi070.stdout: write object-528 2024-09-21T10:04:35.951 INFO:tasks.workunit.client.0.smithi070.stdout: write object-530 2024-09-21T10:04:35.952 INFO:tasks.workunit.client.0.smithi070.stdout: write object-551 2024-09-21T10:04:35.952 INFO:tasks.workunit.client.0.smithi070.stdout: write object-553 2024-09-21T10:04:35.952 INFO:tasks.workunit.client.0.smithi070.stdout: write object-559 2024-09-21T10:04:35.952 INFO:tasks.workunit.client.0.smithi070.stdout: write object-565 2024-09-21T10:04:35.952 INFO:tasks.workunit.client.0.smithi070.stdout: write object-576 2024-09-21T10:04:35.952 INFO:tasks.workunit.client.0.smithi070.stdout: write object-585 2024-09-21T10:04:35.952 INFO:tasks.workunit.client.0.smithi070.stdout: write object-612 2024-09-21T10:04:35.952 INFO:tasks.workunit.client.0.smithi070.stdout: write object-633 2024-09-21T10:04:35.952 INFO:tasks.workunit.client.0.smithi070.stdout: write object-646 2024-09-21T10:04:35.952 INFO:tasks.workunit.client.0.smithi070.stdout: write object-648 2024-09-21T10:04:35.952 INFO:tasks.workunit.client.0.smithi070.stdout: write object-661 2024-09-21T10:04:35.952 INFO:tasks.workunit.client.0.smithi070.stdout: write object-671 2024-09-21T10:04:35.952 INFO:tasks.workunit.client.0.smithi070.stdout: write object-687 2024-09-21T10:04:35.952 INFO:tasks.workunit.client.0.smithi070.stdout: write object-692 2024-09-21T10:04:35.952 INFO:tasks.workunit.client.0.smithi070.stdout: write object-693 2024-09-21T10:04:35.953 INFO:tasks.workunit.client.0.smithi070.stdout: write object-706 2024-09-21T10:04:35.953 INFO:tasks.workunit.client.0.smithi070.stdout: write object-713 2024-09-21T10:04:35.953 INFO:tasks.workunit.client.0.smithi070.stdout: write object-732 2024-09-21T10:04:35.953 INFO:tasks.workunit.client.0.smithi070.stdout: write object-746 2024-09-21T10:04:35.953 INFO:tasks.workunit.client.0.smithi070.stdout: write object-754 2024-09-21T10:04:35.953 INFO:tasks.workunit.client.0.smithi070.stdout: write object-776 2024-09-21T10:04:35.953 INFO:tasks.workunit.client.0.smithi070.stdout: write object-783 2024-09-21T10:04:35.953 INFO:tasks.workunit.client.0.smithi070.stdout: write object-789 2024-09-21T10:04:35.953 INFO:tasks.workunit.client.0.smithi070.stdout: write object-797 2024-09-21T10:04:35.953 INFO:tasks.workunit.client.0.smithi070.stdout: write object-800 2024-09-21T10:04:35.953 INFO:tasks.workunit.client.0.smithi070.stdout: write object-812 2024-09-21T10:04:35.953 INFO:tasks.workunit.client.0.smithi070.stdout: write object-823 2024-09-21T10:04:35.953 INFO:tasks.workunit.client.0.smithi070.stdout: write object-842 2024-09-21T10:04:35.953 INFO:tasks.workunit.client.0.smithi070.stdout: write object-862 2024-09-21T10:04:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:35 smithi160 ceph-mon[101879]: osdmap e551: 8 total, 8 up, 8 in 2024-09-21T10:04:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:36.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:35 smithi070 ceph-mon[107385]: osdmap e551: 8 total, 8 up, 8 in 2024-09-21T10:04:36.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:36.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:35 smithi070 ceph-mon[112139]: osdmap e551: 8 total, 8 up, 8 in 2024-09-21T10:04:36.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:36.875 INFO:tasks.workunit.client.0.smithi070.stdout: write object-868 2024-09-21T10:04:36.875 INFO:tasks.workunit.client.0.smithi070.stdout: write object-882 2024-09-21T10:04:36.875 INFO:tasks.workunit.client.0.smithi070.stdout: write object-910 2024-09-21T10:04:36.876 INFO:tasks.workunit.client.0.smithi070.stdout: write object-922 2024-09-21T10:04:36.876 INFO:tasks.workunit.client.0.smithi070.stdout: write object-924 2024-09-21T10:04:36.876 INFO:tasks.workunit.client.0.smithi070.stdout: write object-946 2024-09-21T10:04:36.876 INFO:tasks.workunit.client.0.smithi070.stdout: write object-983 2024-09-21T10:04:36.876 INFO:tasks.workunit.client.0.smithi070.stdout: write object-996 2024-09-21T10:04:36.876 INFO:tasks.workunit.client.0.smithi070.stdout: write object-1000 2024-09-21T10:04:36.876 INFO:tasks.workunit.client.0.smithi070.stdout: write object-1004 2024-09-21T10:04:36.876 INFO:tasks.workunit.client.0.smithi070.stdout:verify written data by reading 2024-09-21T10:04:36.876 INFO:tasks.workunit.client.0.smithi070.stdout: read object-8 2024-09-21T10:04:36.876 INFO:tasks.workunit.client.0.smithi070.stdout:made snapshot "image399@parent_snap" and protect it 2024-09-21T10:04:36.876 INFO:tasks.workunit.client.0.smithi070.stdout:made and opened clone "image400" 2024-09-21T10:04:36.876 INFO:tasks.workunit.client.0.smithi070.stdout:flattening clone: "image400" 2024-09-21T10:04:36.876 INFO:tasks.workunit.client.0.smithi070.stdout:check whether child image has the same set of objects as parent 2024-09-21T10:04:36.876 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.FlattenNoEmptyObjects (1140 ms) 2024-09-21T10:04:36.877 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.PoolMetadataConfApply 2024-09-21T10:04:36.943 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.PoolMetadataConfApply (68 ms) 2024-09-21T10:04:36.943 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.Sparsify 2024-09-21T10:04:36.963 INFO:tasks.workunit.client.0.smithi070.stdout:sparsify_supported=1 2024-09-21T10:04:36.963 INFO:tasks.workunit.client.0.smithi070.stdout:sparse_read_supported=1 2024-09-21T10:04:36.989 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.Sparsify (46 ms) 2024-09-21T10:04:36.989 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.SparsifyClone 2024-09-21T10:04:37.005 INFO:tasks.workunit.client.0.smithi070.stdout:sparsify_supported=1 2024-09-21T10:04:37.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:36 smithi160 ceph-mon[101879]: osdmap e552: 8 total, 8 up, 8 in 2024-09-21T10:04:37.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:36 smithi160 ceph-mon[101879]: pgmap v324: 105 pgs: 105 active+clean; 4.8 MiB data, 469 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 404 op/s 2024-09-21T10:04:37.127 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:04:36 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:04:36] ENGINE Bus STOPPING 2024-09-21T10:04:37.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:36 smithi070 ceph-mon[107385]: osdmap e552: 8 total, 8 up, 8 in 2024-09-21T10:04:37.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:36 smithi070 ceph-mon[107385]: pgmap v324: 105 pgs: 105 active+clean; 4.8 MiB data, 469 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 404 op/s 2024-09-21T10:04:37.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:36 smithi070 ceph-mon[112139]: osdmap e552: 8 total, 8 up, 8 in 2024-09-21T10:04:37.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:36 smithi070 ceph-mon[112139]: pgmap v324: 105 pgs: 105 active+clean; 4.8 MiB data, 469 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 404 op/s 2024-09-21T10:04:37.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:04:37 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:04:37] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T10:04:37.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:04:37 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:04:37] ENGINE Bus STOPPED 2024-09-21T10:04:37.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:04:37 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:04:37] ENGINE Bus STARTING 2024-09-21T10:04:37.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:04:37 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:04:37] ENGINE Serving on http://:::9283 2024-09-21T10:04:37.628 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:04:37 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:04:37] ENGINE Bus STARTED 2024-09-21T10:04:37.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:37 smithi160 ceph-mon[101879]: osdmap e553: 8 total, 8 up, 8 in 2024-09-21T10:04:37.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:37 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2124964304' entity='client.admin' 2024-09-21T10:04:37.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:37 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:04:37.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:37 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:04:37.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:37 smithi160 ceph-mon[101879]: osdmap e554: 8 total, 8 up, 8 in 2024-09-21T10:04:38.131 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:37 smithi070 ceph-mon[107385]: osdmap e553: 8 total, 8 up, 8 in 2024-09-21T10:04:38.131 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:37 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2124964304' entity='client.admin' 2024-09-21T10:04:38.131 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:37 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:04:38.131 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:37 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:04:38.131 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:37 smithi070 ceph-mon[107385]: osdmap e554: 8 total, 8 up, 8 in 2024-09-21T10:04:38.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:37 smithi070 ceph-mon[112139]: osdmap e553: 8 total, 8 up, 8 in 2024-09-21T10:04:38.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:37 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2124964304' entity='client.admin' 2024-09-21T10:04:38.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:37 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:04:38.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:37 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:04:38.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:37 smithi070 ceph-mon[112139]: osdmap e554: 8 total, 8 up, 8 in 2024-09-21T10:04:38.725 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.SparsifyClone (1736 ms) 2024-09-21T10:04:38.725 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.MissingDataPool 2024-09-21T10:04:39.081 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:38 smithi160 ceph-mon[101879]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:04:39.081 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:38 smithi160 ceph-mon[101879]: pgmap v327: 105 pgs: 105 active+clean; 4.8 MiB data, 469 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 404 op/s 2024-09-21T10:04:39.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:38 smithi070 ceph-mon[107385]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:04:39.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:38 smithi070 ceph-mon[107385]: pgmap v327: 105 pgs: 105 active+clean; 4.8 MiB data, 469 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 404 op/s 2024-09-21T10:04:39.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:38 smithi070 ceph-mon[112139]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:04:39.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:38 smithi070 ceph-mon[112139]: pgmap v327: 105 pgs: 105 active+clean; 4.8 MiB data, 469 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 404 op/s 2024-09-21T10:04:39.831 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.MissingDataPool (1106 ms) 2024-09-21T10:04:40.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:39 smithi160 ceph-mon[101879]: osdmap e555: 8 total, 8 up, 8 in 2024-09-21T10:04:40.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:39 smithi070 ceph-mon[107385]: osdmap e555: 8 total, 8 up, 8 in 2024-09-21T10:04:40.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:39 smithi070 ceph-mon[112139]: osdmap e555: 8 total, 8 up, 8 in 2024-09-21T10:04:40.732 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 34 tests from TestInternal (35088 ms total) 2024-09-21T10:04:40.732 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:04:40.732 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 17 tests from TestImageWatcher 2024-09-21T10:04:41.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:40 smithi160 ceph-mon[101879]: osdmap e556: 8 total, 8 up, 8 in 2024-09-21T10:04:41.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:40 smithi160 ceph-mon[101879]: pgmap v330: 105 pgs: 105 active+clean; 4.8 MiB data, 469 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:41.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:40 smithi070 ceph-mon[107385]: osdmap e556: 8 total, 8 up, 8 in 2024-09-21T10:04:41.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:40 smithi070 ceph-mon[107385]: pgmap v330: 105 pgs: 105 active+clean; 4.8 MiB data, 469 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:41.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:40 smithi070 ceph-mon[112139]: osdmap e556: 8 total, 8 up, 8 in 2024-09-21T10:04:41.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:40 smithi070 ceph-mon[112139]: pgmap v330: 105 pgs: 105 active+clean; 4.8 MiB data, 469 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:42.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:41 smithi160 ceph-mon[101879]: osdmap e557: 8 total, 8 up, 8 in 2024-09-21T10:04:42.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:41 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:04:42.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:41 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:04:42.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:41 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:42.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:41 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:04:42.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:41 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:04:42.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:41 smithi070 ceph-mon[112139]: osdmap e557: 8 total, 8 up, 8 in 2024-09-21T10:04:42.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:41 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:04:42.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:41 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:04:42.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:41 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:42.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:41 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:04:42.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:41 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:04:42.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:41 smithi070 ceph-mon[107385]: osdmap e557: 8 total, 8 up, 8 in 2024-09-21T10:04:42.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:41 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:04:42.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:41 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:04:42.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:41 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:42.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:41 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:04:42.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:41 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:04:42.740 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifyHeaderUpdate 2024-09-21T10:04:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:42 smithi160 ceph-mon[101879]: osdmap e558: 8 total, 8 up, 8 in 2024-09-21T10:04:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:42 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1339602900' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-14","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:04:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:42 smithi160 ceph-mon[101879]: pgmap v333: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 482 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:43.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:42 smithi070 ceph-mon[107385]: osdmap e558: 8 total, 8 up, 8 in 2024-09-21T10:04:43.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:42 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1339602900' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-14","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:04:43.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:42 smithi070 ceph-mon[107385]: pgmap v333: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 482 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:43.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:42 smithi070 ceph-mon[112139]: osdmap e558: 8 total, 8 up, 8 in 2024-09-21T10:04:43.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:42 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1339602900' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-14","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:04:43.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:42 smithi070 ceph-mon[112139]: pgmap v333: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 482 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:04:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:04:43] "GET /metrics HTTP/1.1" 200 36281 "" "Prometheus/2.51.0" 2024-09-21T10:04:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:43 smithi160 ceph-mon[101879]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:04:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:43 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1339602900' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:04:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:43 smithi160 ceph-mon[101879]: osdmap e559: 8 total, 8 up, 8 in 2024-09-21T10:04:44.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:43 smithi070 ceph-mon[107385]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:04:44.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:43 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1339602900' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:04:44.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:43 smithi070 ceph-mon[107385]: osdmap e559: 8 total, 8 up, 8 in 2024-09-21T10:04:44.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:43 smithi070 ceph-mon[112139]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:04:44.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:43 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1339602900' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:04:44.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:43 smithi070 ceph-mon[112139]: osdmap e559: 8 total, 8 up, 8 in 2024-09-21T10:04:44.778 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifyHeaderUpdate (2037 ms) 2024-09-21T10:04:44.778 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifyFlatten 2024-09-21T10:04:44.799 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifyFlatten (22 ms) 2024-09-21T10:04:44.800 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifyResize 2024-09-21T10:04:44.822 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifyResize (23 ms) 2024-09-21T10:04:44.822 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifyRebuildObjectMap 2024-09-21T10:04:44.843 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifyRebuildObjectMap (21 ms) 2024-09-21T10:04:44.843 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifyUpdateFeatures 2024-09-21T10:04:44.862 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifyUpdateFeatures (19 ms) 2024-09-21T10:04:44.862 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifySnapCreate 2024-09-21T10:04:44.884 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifySnapCreate (22 ms) 2024-09-21T10:04:44.884 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifySnapCreateError 2024-09-21T10:04:44.902 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifySnapCreateError (18 ms) 2024-09-21T10:04:44.902 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifySnapRename 2024-09-21T10:04:44.922 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifySnapRename (20 ms) 2024-09-21T10:04:44.922 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifySnapRenameError 2024-09-21T10:04:44.939 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifySnapRenameError (17 ms) 2024-09-21T10:04:44.939 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifySnapRemove 2024-09-21T10:04:44.958 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifySnapRemove (19 ms) 2024-09-21T10:04:44.958 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifySnapProtect 2024-09-21T10:04:44.978 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifySnapProtect (20 ms) 2024-09-21T10:04:44.978 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifySnapUnprotect 2024-09-21T10:04:44.998 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifySnapUnprotect (20 ms) 2024-09-21T10:04:44.998 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifyRename 2024-09-21T10:04:45.017 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifyRename (19 ms) 2024-09-21T10:04:45.017 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifyAsyncTimedOut 2024-09-21T10:04:45.034 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifyAsyncTimedOut (17 ms) 2024-09-21T10:04:45.034 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifyAsyncError 2024-09-21T10:04:45.053 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifyAsyncError (19 ms) 2024-09-21T10:04:45.053 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifyAsyncCompleteError 2024-09-21T10:04:45.072 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifyAsyncCompleteError (19 ms) 2024-09-21T10:04:45.072 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifyAsyncRequestTimedOut 2024-09-21T10:04:45.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:44 smithi160 ceph-mon[101879]: osdmap e560: 8 total, 8 up, 8 in 2024-09-21T10:04:45.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:44 smithi160 ceph-mon[101879]: pgmap v336: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 482 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:45.091 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifyAsyncRequestTimedOut (19 ms) 2024-09-21T10:04:45.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:44 smithi070 ceph-mon[107385]: osdmap e560: 8 total, 8 up, 8 in 2024-09-21T10:04:45.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:44 smithi070 ceph-mon[107385]: pgmap v336: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 482 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:45.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:44 smithi070 ceph-mon[112139]: osdmap e560: 8 total, 8 up, 8 in 2024-09-21T10:04:45.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:44 smithi070 ceph-mon[112139]: pgmap v336: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 482 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:45.757 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 17 tests from TestImageWatcher (2351 ms total) 2024-09-21T10:04:45.757 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:04:45.757 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 170 tests from TestLibRBD 2024-09-21T10:04:45.787 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.CreateAndStat 2024-09-21T10:04:46.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:45 smithi160 ceph-mon[101879]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:04:46.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:45 smithi160 ceph-mon[101879]: Cluster is now healthy 2024-09-21T10:04:46.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:45 smithi160 ceph-mon[101879]: osdmap e561: 8 total, 8 up, 8 in 2024-09-21T10:04:46.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:45 smithi070 ceph-mon[107385]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:04:46.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:45 smithi070 ceph-mon[107385]: Cluster is now healthy 2024-09-21T10:04:46.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:45 smithi070 ceph-mon[107385]: osdmap e561: 8 total, 8 up, 8 in 2024-09-21T10:04:46.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:45 smithi070 ceph-mon[112139]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:04:46.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:45 smithi070 ceph-mon[112139]: Cluster is now healthy 2024-09-21T10:04:46.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:45 smithi070 ceph-mon[112139]: osdmap e561: 8 total, 8 up, 8 in 2024-09-21T10:04:47.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:46 smithi160 ceph-mon[101879]: osdmap e562: 8 total, 8 up, 8 in 2024-09-21T10:04:47.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:46 smithi160 ceph-mon[101879]: pgmap v339: 73 pgs: 73 active+clean; 582 KiB data, 471 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:47.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:46 smithi070 ceph-mon[107385]: osdmap e562: 8 total, 8 up, 8 in 2024-09-21T10:04:47.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:46 smithi070 ceph-mon[107385]: pgmap v339: 73 pgs: 73 active+clean; 582 KiB data, 471 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:47.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:46 smithi070 ceph-mon[112139]: osdmap e562: 8 total, 8 up, 8 in 2024-09-21T10:04:47.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:46 smithi070 ceph-mon[112139]: pgmap v339: 73 pgs: 73 active+clean; 582 KiB data, 471 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:47.771 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:04:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:47 smithi160 ceph-mon[101879]: osdmap e563: 8 total, 8 up, 8 in 2024-09-21T10:04:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:47 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1060922746' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:04:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:47 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:04:48.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:47 smithi070 ceph-mon[107385]: osdmap e563: 8 total, 8 up, 8 in 2024-09-21T10:04:48.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:47 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1060922746' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:04:48.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:47 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:04:48.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:47 smithi070 ceph-mon[112139]: osdmap e563: 8 total, 8 up, 8 in 2024-09-21T10:04:48.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:47 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1060922746' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:04:48.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:47 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:04:49.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:48 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:04:49.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:48 smithi160 ceph-mon[101879]: osdmap e564: 8 total, 8 up, 8 in 2024-09-21T10:04:49.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:48 smithi160 ceph-mon[101879]: pgmap v342: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 471 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:49.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:48 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:04:49.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:48 smithi070 ceph-mon[112139]: osdmap e564: 8 total, 8 up, 8 in 2024-09-21T10:04:49.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:48 smithi070 ceph-mon[112139]: pgmap v342: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 471 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:49.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:48 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:04:49.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:48 smithi070 ceph-mon[107385]: osdmap e564: 8 total, 8 up, 8 in 2024-09-21T10:04:49.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:48 smithi070 ceph-mon[107385]: pgmap v342: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 471 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:49.802 INFO:tasks.workunit.client.0.smithi070.stdout:image has size 2097152 and order 22 2024-09-21T10:04:49.802 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.CreateAndStat (4014 ms) 2024-09-21T10:04:49.802 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.CreateWithSameDataPool 2024-09-21T10:04:49.802 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:04:49.819 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.CreateWithSameDataPool (18 ms) 2024-09-21T10:04:49.819 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.CreateAndStatPP 2024-09-21T10:04:49.819 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:04:49.836 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.CreateAndStatPP (17 ms) 2024-09-21T10:04:49.837 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.GetId 2024-09-21T10:04:49.837 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:04:49.859 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.GetId (23 ms) 2024-09-21T10:04:49.859 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.GetIdPP 2024-09-21T10:04:49.859 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:04:49.885 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.GetIdPP (26 ms) 2024-09-21T10:04:49.885 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefix 2024-09-21T10:04:49.885 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:04:49.902 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.GetBlockNamePrefix (17 ms) 2024-09-21T10:04:49.902 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefixPP 2024-09-21T10:04:49.902 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:04:49.916 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.GetBlockNamePrefixPP (14 ms) 2024-09-21T10:04:49.916 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestGetCreateTimestamp 2024-09-21T10:04:49.916 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:04:49.931 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestGetCreateTimestamp (15 ms) 2024-09-21T10:04:49.931 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.GetCreateTimestampPP 2024-09-21T10:04:49.931 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:04:49.947 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.GetCreateTimestampPP (15 ms) 2024-09-21T10:04:49.947 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.OpenAio 2024-09-21T10:04:49.947 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:04:49.961 INFO:tasks.workunit.client.0.smithi070.stdout:image has size 2097152 and order 22 2024-09-21T10:04:49.962 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.OpenAio (15 ms) 2024-09-21T10:04:49.962 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.OpenAioFail 2024-09-21T10:04:49.965 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.OpenAioFail (4 ms) 2024-09-21T10:04:49.965 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.OpenAioPP 2024-09-21T10:04:49.965 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:04:49.987 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.OpenAioPP (22 ms) 2024-09-21T10:04:49.988 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.OpenAioFailPP 2024-09-21T10:04:49.990 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.OpenAioFailPP (3 ms) 2024-09-21T10:04:49.990 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ResizeAndStat 2024-09-21T10:04:49.991 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:04:50.019 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ResizeAndStat (29 ms) 2024-09-21T10:04:50.020 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ResizeAndStatPP 2024-09-21T10:04:50.020 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:04:50.046 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ResizeAndStatPP (27 ms) 2024-09-21T10:04:50.046 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResize 2024-09-21T10:04:50.046 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:04:50.076 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.UpdateWatchAndResize (30 ms) 2024-09-21T10:04:50.076 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResizePP 2024-09-21T10:04:50.076 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:04:50.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:49 smithi160 ceph-mon[101879]: osdmap e565: 8 total, 8 up, 8 in 2024-09-21T10:04:50.109 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.UpdateWatchAndResizePP (33 ms) 2024-09-21T10:04:50.109 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCreateLsDelete 2024-09-21T10:04:50.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:49 smithi070 ceph-mon[107385]: osdmap e565: 8 total, 8 up, 8 in 2024-09-21T10:04:50.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:49 smithi070 ceph-mon[112139]: osdmap e565: 8 total, 8 up, 8 in 2024-09-21T10:04:51.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:50 smithi160 ceph-mon[101879]: osdmap e566: 8 total, 8 up, 8 in 2024-09-21T10:04:51.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:50 smithi160 ceph-mon[101879]: pgmap v345: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 471 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:51.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:51.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:50 smithi070 ceph-mon[107385]: osdmap e566: 8 total, 8 up, 8 in 2024-09-21T10:04:51.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:50 smithi070 ceph-mon[107385]: pgmap v345: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 471 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:51.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:51.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:50 smithi070 ceph-mon[112139]: osdmap e566: 8 total, 8 up, 8 in 2024-09-21T10:04:51.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:50 smithi070 ceph-mon[112139]: pgmap v345: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 471 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:04:51.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:51.801 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:04:52.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:51 smithi070 ceph-mon[112139]: osdmap e567: 8 total, 8 up, 8 in 2024-09-21T10:04:52.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:51 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2394310220' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-16","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:04:52.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:51 smithi070 ceph-mon[107385]: osdmap e567: 8 total, 8 up, 8 in 2024-09-21T10:04:52.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:51 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2394310220' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-16","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:04:52.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:51 smithi160 ceph-mon[101879]: osdmap e567: 8 total, 8 up, 8 in 2024-09-21T10:04:52.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:51 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2394310220' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-16","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:04:53.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:52 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2394310220' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:04:53.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:52 smithi160 ceph-mon[101879]: osdmap e568: 8 total, 8 up, 8 in 2024-09-21T10:04:53.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:52 smithi160 ceph-mon[101879]: pgmap v348: 137 pgs: 32 creating+peering, 105 active+clean; 583 KiB data, 480 MiB used, 715 GiB / 715 GiB avail; 181 KiB/s rd, 20 KiB/s wr, 250 op/s 2024-09-21T10:04:53.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:52 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2394310220' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:04:53.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:52 smithi070 ceph-mon[107385]: osdmap e568: 8 total, 8 up, 8 in 2024-09-21T10:04:53.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:52 smithi070 ceph-mon[107385]: pgmap v348: 137 pgs: 32 creating+peering, 105 active+clean; 583 KiB data, 480 MiB used, 715 GiB / 715 GiB avail; 181 KiB/s rd, 20 KiB/s wr, 250 op/s 2024-09-21T10:04:53.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:52 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2394310220' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:04:53.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:52 smithi070 ceph-mon[112139]: osdmap e568: 8 total, 8 up, 8 in 2024-09-21T10:04:53.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:52 smithi070 ceph-mon[112139]: pgmap v348: 137 pgs: 32 creating+peering, 105 active+clean; 583 KiB data, 480 MiB used, 715 GiB / 715 GiB avail; 181 KiB/s rd, 20 KiB/s wr, 250 op/s 2024-09-21T10:04:53.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:04:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:04:53] "GET /metrics HTTP/1.1" 200 36150 "" "Prometheus/2.51.0" 2024-09-21T10:04:53.825 INFO:tasks.workunit.client.0.smithi070.stdout:image: image18 2024-09-21T10:04:53.825 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image18 2024-09-21T10:04:53.825 INFO:tasks.workunit.client.0.smithi070.stdout:found image18 2024-09-21T10:04:53.825 INFO:tasks.workunit.client.0.smithi070.stdout:erased image18 2024-09-21T10:04:53.825 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:04:53.901 INFO:tasks.workunit.client.0.smithi070.stdout:image: image18 2024-09-21T10:04:53.902 INFO:tasks.workunit.client.0.smithi070.stdout:image: image19 2024-09-21T10:04:53.902 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image18 2024-09-21T10:04:53.902 INFO:tasks.workunit.client.0.smithi070.stdout:found image18 2024-09-21T10:04:53.902 INFO:tasks.workunit.client.0.smithi070.stdout:erased image18 2024-09-21T10:04:53.902 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image19 2024-09-21T10:04:53.902 INFO:tasks.workunit.client.0.smithi070.stdout:found image19 2024-09-21T10:04:53.902 INFO:tasks.workunit.client.0.smithi070.stdout:erased image19 2024-09-21T10:04:53.902 INFO:tasks.workunit.client.0.smithi070.stdout:image: image19 2024-09-21T10:04:53.902 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image19 2024-09-21T10:04:53.902 INFO:tasks.workunit.client.0.smithi070.stdout:found image19 2024-09-21T10:04:53.902 INFO:tasks.workunit.client.0.smithi070.stdout:erased image19 2024-09-21T10:04:53.903 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCreateLsDelete (3792 ms) 2024-09-21T10:04:53.903 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCreateLsDeletePP 2024-09-21T10:04:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:53 smithi160 ceph-mon[101879]: osdmap e569: 8 total, 8 up, 8 in 2024-09-21T10:04:54.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:53 smithi070 ceph-mon[107385]: osdmap e569: 8 total, 8 up, 8 in 2024-09-21T10:04:54.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:53 smithi070 ceph-mon[112139]: osdmap e569: 8 total, 8 up, 8 in 2024-09-21T10:04:55.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:54 smithi160 ceph-mon[101879]: osdmap e570: 8 total, 8 up, 8 in 2024-09-21T10:04:55.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:54 smithi160 ceph-mon[101879]: pgmap v351: 137 pgs: 32 creating+peering, 105 active+clean; 583 KiB data, 480 MiB used, 715 GiB / 715 GiB avail; 181 KiB/s rd, 20 KiB/s wr, 250 op/s 2024-09-21T10:04:55.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:54 smithi070 ceph-mon[107385]: osdmap e570: 8 total, 8 up, 8 in 2024-09-21T10:04:55.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:54 smithi070 ceph-mon[107385]: pgmap v351: 137 pgs: 32 creating+peering, 105 active+clean; 583 KiB data, 480 MiB used, 715 GiB / 715 GiB avail; 181 KiB/s rd, 20 KiB/s wr, 250 op/s 2024-09-21T10:04:55.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:54 smithi070 ceph-mon[112139]: osdmap e570: 8 total, 8 up, 8 in 2024-09-21T10:04:55.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:54 smithi070 ceph-mon[112139]: pgmap v351: 137 pgs: 32 creating+peering, 105 active+clean; 583 KiB data, 480 MiB used, 715 GiB / 715 GiB avail; 181 KiB/s rd, 20 KiB/s wr, 250 op/s 2024-09-21T10:04:55.828 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:04:56.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:55 smithi160 ceph-mon[101879]: osdmap e571: 8 total, 8 up, 8 in 2024-09-21T10:04:56.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:55 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1810692593' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-17","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:04:56.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:55 smithi070 ceph-mon[107385]: osdmap e571: 8 total, 8 up, 8 in 2024-09-21T10:04:56.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:55 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1810692593' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-17","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:04:56.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:55 smithi070 ceph-mon[112139]: osdmap e571: 8 total, 8 up, 8 in 2024-09-21T10:04:56.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:55 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1810692593' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-17","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:04:57.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:56 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1810692593' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:04:57.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:56 smithi160 ceph-mon[101879]: osdmap e572: 8 total, 8 up, 8 in 2024-09-21T10:04:57.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:56 smithi160 ceph-mon[101879]: pgmap v354: 169 pgs: 169 active+clean; 583 KiB data, 484 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 3.2 KiB/s wr, 41 op/s 2024-09-21T10:04:57.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:56 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1810692593' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:04:57.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:56 smithi070 ceph-mon[107385]: osdmap e572: 8 total, 8 up, 8 in 2024-09-21T10:04:57.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:56 smithi070 ceph-mon[107385]: pgmap v354: 169 pgs: 169 active+clean; 583 KiB data, 484 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 3.2 KiB/s wr, 41 op/s 2024-09-21T10:04:57.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:56 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1810692593' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:04:57.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:56 smithi070 ceph-mon[112139]: osdmap e572: 8 total, 8 up, 8 in 2024-09-21T10:04:57.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:56 smithi070 ceph-mon[112139]: pgmap v354: 169 pgs: 169 active+clean; 583 KiB data, 484 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 3.2 KiB/s wr, 41 op/s 2024-09-21T10:04:57.692 INFO:tasks.workunit.client.0.smithi070.stdout:num images is: 1 2024-09-21T10:04:57.692 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-21T10:04:57.692 INFO:tasks.workunit.client.0.smithi070.stdout:image: image20 2024-09-21T10:04:57.692 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image20 2024-09-21T10:04:57.692 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:04:57.701 INFO:tasks.workunit.client.0.smithi070.stdout:num images is: 2 2024-09-21T10:04:57.701 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 2 2024-09-21T10:04:57.701 INFO:tasks.workunit.client.0.smithi070.stdout:image: image20 2024-09-21T10:04:57.701 INFO:tasks.workunit.client.0.smithi070.stdout:image: image21 2024-09-21T10:04:57.701 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image20 2024-09-21T10:04:57.701 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image21 2024-09-21T10:04:57.760 INFO:tasks.workunit.client.0.smithi070.stdout:num images is: 1 2024-09-21T10:04:57.760 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-21T10:04:57.760 INFO:tasks.workunit.client.0.smithi070.stdout:image: image21 2024-09-21T10:04:57.760 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image21 2024-09-21T10:04:57.760 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCreateLsDeletePP (3859 ms) 2024-09-21T10:04:57.760 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCopy 2024-09-21T10:04:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:57 smithi160 ceph-mon[101879]: osdmap e573: 8 total, 8 up, 8 in 2024-09-21T10:04:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:57 smithi160 ceph-mon[101879]: osdmap e574: 8 total, 8 up, 8 in 2024-09-21T10:04:58.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:57 smithi070 ceph-mon[112139]: osdmap e573: 8 total, 8 up, 8 in 2024-09-21T10:04:58.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:57 smithi070 ceph-mon[112139]: osdmap e574: 8 total, 8 up, 8 in 2024-09-21T10:04:58.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:57 smithi070 ceph-mon[107385]: osdmap e573: 8 total, 8 up, 8 in 2024-09-21T10:04:58.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:57 smithi070 ceph-mon[107385]: osdmap e574: 8 total, 8 up, 8 in 2024-09-21T10:04:59.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:58 smithi160 ceph-mon[101879]: pgmap v357: 169 pgs: 169 active+clean; 583 KiB data, 484 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 3.2 KiB/s wr, 41 op/s 2024-09-21T10:04:59.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:58 smithi160 ceph-mon[101879]: osdmap e575: 8 total, 8 up, 8 in 2024-09-21T10:04:59.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:04:58 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3716122247' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-18","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:04:59.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:58 smithi070 ceph-mon[107385]: pgmap v357: 169 pgs: 169 active+clean; 583 KiB data, 484 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 3.2 KiB/s wr, 41 op/s 2024-09-21T10:04:59.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:58 smithi070 ceph-mon[107385]: osdmap e575: 8 total, 8 up, 8 in 2024-09-21T10:04:59.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:04:58 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3716122247' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-18","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:04:59.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:58 smithi070 ceph-mon[112139]: pgmap v357: 169 pgs: 169 active+clean; 583 KiB data, 484 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 3.2 KiB/s wr, 41 op/s 2024-09-21T10:04:59.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:58 smithi070 ceph-mon[112139]: osdmap e575: 8 total, 8 up, 8 in 2024-09-21T10:04:59.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:04:58 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3716122247' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-18","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:04:59.682 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:05:01.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:00 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3716122247' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:05:01.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:00 smithi160 ceph-mon[101879]: osdmap e576: 8 total, 8 up, 8 in 2024-09-21T10:05:01.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:00 smithi160 ceph-mon[101879]: pgmap v360: 201 pgs: 32 unknown, 169 active+clean; 583 KiB data, 484 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:05:01.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:00 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3716122247' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:05:01.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:00 smithi070 ceph-mon[107385]: osdmap e576: 8 total, 8 up, 8 in 2024-09-21T10:05:01.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:00 smithi070 ceph-mon[107385]: pgmap v360: 201 pgs: 32 unknown, 169 active+clean; 583 KiB data, 484 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:05:01.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:00 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3716122247' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:05:01.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:00 smithi070 ceph-mon[112139]: osdmap e576: 8 total, 8 up, 8 in 2024-09-21T10:05:01.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:00 smithi070 ceph-mon[112139]: pgmap v360: 201 pgs: 32 unknown, 169 active+clean; 583 KiB data, 484 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:05:01.803 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:01 smithi070 ceph-mon[107385]: osdmap e577: 8 total, 8 up, 8 in 2024-09-21T10:05:01.804 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:01 smithi070 ceph-mon[112139]: osdmap e577: 8 total, 8 up, 8 in 2024-09-21T10:05:02.063 INFO:tasks.workunit.client.0.smithi070.stdout:image: image22 2024-09-21T10:05:02.063 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image22 2024-09-21T10:05:02.063 INFO:tasks.workunit.client.0.smithi070.stdout:found image22 2024-09-21T10:05:02.063 INFO:tasks.workunit.client.0.smithi070.stdout:erased image22 2024-09-21T10:05:02.063 INFO:tasks.workunit.client.0.smithi070.stdout:image: image22 2024-09-21T10:05:02.063 INFO:tasks.workunit.client.0.smithi070.stdout:image: image23 2024-09-21T10:05:02.063 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image22 2024-09-21T10:05:02.063 INFO:tasks.workunit.client.0.smithi070.stdout:found image22 2024-09-21T10:05:02.063 INFO:tasks.workunit.client.0.smithi070.stdout:erased image22 2024-09-21T10:05:02.063 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image23 2024-09-21T10:05:02.063 INFO:tasks.workunit.client.0.smithi070.stdout:found image23 2024-09-21T10:05:02.064 INFO:tasks.workunit.client.0.smithi070.stdout:erased image23 2024-09-21T10:05:02.064 INFO:tasks.workunit.client.0.smithi070.stdout:100.00% done 2024-09-21T10:05:02.064 INFO:tasks.workunit.client.0.smithi070.stdout:image: image22 2024-09-21T10:05:02.064 INFO:tasks.workunit.client.0.smithi070.stdout:image: image23 2024-09-21T10:05:02.064 INFO:tasks.workunit.client.0.smithi070.stdout:image: image24 2024-09-21T10:05:02.064 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image22 2024-09-21T10:05:02.064 INFO:tasks.workunit.client.0.smithi070.stdout:found image22 2024-09-21T10:05:02.064 INFO:tasks.workunit.client.0.smithi070.stdout:erased image22 2024-09-21T10:05:02.064 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image23 2024-09-21T10:05:02.064 INFO:tasks.workunit.client.0.smithi070.stdout:found image23 2024-09-21T10:05:02.064 INFO:tasks.workunit.client.0.smithi070.stdout:erased image23 2024-09-21T10:05:02.064 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image24 2024-09-21T10:05:02.064 INFO:tasks.workunit.client.0.smithi070.stdout:found image24 2024-09-21T10:05:02.064 INFO:tasks.workunit.client.0.smithi070.stdout:erased image24 2024-09-21T10:05:02.064 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCopy (4302 ms) 2024-09-21T10:05:02.065 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCopyPP 2024-09-21T10:05:02.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:01 smithi160 ceph-mon[101879]: osdmap e577: 8 total, 8 up, 8 in 2024-09-21T10:05:02.216 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'echo wait for all mons to upgrade' 2024-09-21T10:05:02.414 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:02.790 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:02 smithi070 ceph-mon[107385]: osdmap e578: 8 total, 8 up, 8 in 2024-09-21T10:05:02.790 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:02 smithi070 ceph-mon[107385]: pgmap v363: 201 pgs: 201 active+clean; 583 KiB data, 489 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 3.5 KiB/s wr, 41 op/s 2024-09-21T10:05:02.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:02 smithi070 ceph-mon[112139]: osdmap e578: 8 total, 8 up, 8 in 2024-09-21T10:05:02.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:02 smithi070 ceph-mon[112139]: pgmap v363: 201 pgs: 201 active+clean; 583 KiB data, 489 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 3.5 KiB/s wr, 41 op/s 2024-09-21T10:05:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:02 smithi160 ceph-mon[101879]: osdmap e578: 8 total, 8 up, 8 in 2024-09-21T10:05:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:02 smithi160 ceph-mon[101879]: pgmap v363: 201 pgs: 201 active+clean; 583 KiB data, 489 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 3.5 KiB/s wr, 41 op/s 2024-09-21T10:05:03.527 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:05:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:05:03] "GET /metrics HTTP/1.1" 200 36150 "" "Prometheus/2.51.0" 2024-09-21T10:05:03.705 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:05:03.847 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:03 smithi070 ceph-mon[107385]: osdmap e579: 8 total, 8 up, 8 in 2024-09-21T10:05:03.847 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:03 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2608005061' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:03.847 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:03 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:03.847 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:03 smithi070 ceph-mon[112139]: osdmap e579: 8 total, 8 up, 8 in 2024-09-21T10:05:03.847 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:03 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2608005061' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:03.847 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:03 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:04.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:03 smithi160 ceph-mon[101879]: osdmap e579: 8 total, 8 up, 8 in 2024-09-21T10:05:04.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:03 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2608005061' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:04.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:03 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:04.097 INFO:teuthology.orchestra.run.smithi070.stdout:wait for all mons to upgrade 2024-09-21T10:05:04.704 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:04 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:05:04.704 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:04 smithi070 ceph-mon[107385]: osdmap e580: 8 total, 8 up, 8 in 2024-09-21T10:05:04.704 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:04 smithi070 ceph-mon[107385]: pgmap v366: 233 pgs: 32 unknown, 201 active+clean; 583 KiB data, 489 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 3.5 KiB/s wr, 41 op/s 2024-09-21T10:05:04.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:04 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:05:04.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:04 smithi070 ceph-mon[112139]: osdmap e580: 8 total, 8 up, 8 in 2024-09-21T10:05:04.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:04 smithi070 ceph-mon[112139]: pgmap v366: 233 pgs: 32 unknown, 201 active+clean; 583 KiB data, 489 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 3.5 KiB/s wr, 41 op/s 2024-09-21T10:05:04.726 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon' 2024-09-21T10:05:04.922 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:05.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:04 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:05:05.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:04 smithi160 ceph-mon[101879]: osdmap e580: 8 total, 8 up, 8 in 2024-09-21T10:05:05.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:04 smithi160 ceph-mon[101879]: pgmap v366: 233 pgs: 32 unknown, 201 active+clean; 583 KiB data, 489 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 3.5 KiB/s wr, 41 op/s 2024-09-21T10:05:05.971 INFO:tasks.workunit.client.0.smithi070.stdout:num images is: 1 2024-09-21T10:05:05.982 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-21T10:05:05.982 INFO:tasks.workunit.client.0.smithi070.stdout:image: image25 2024-09-21T10:05:05.982 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image25 2024-09-21T10:05:05.988 INFO:tasks.workunit.client.0.smithi070.stdout:num images is: 2 2024-09-21T10:05:05.988 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 2 2024-09-21T10:05:05.988 INFO:tasks.workunit.client.0.smithi070.stdout:image: image25 2024-09-21T10:05:05.988 INFO:tasks.workunit.client.0.smithi070.stdout:image: image26 2024-09-21T10:05:05.988 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image25 2024-09-21T10:05:05.988 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image26 2024-09-21T10:05:06.027 INFO:tasks.workunit.client.0.smithi070.stdout:100.00% done 2024-09-21T10:05:06.027 INFO:tasks.workunit.client.0.smithi070.stdout:num images is: 3 2024-09-21T10:05:06.027 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 3 2024-09-21T10:05:06.027 INFO:tasks.workunit.client.0.smithi070.stdout:image: image25 2024-09-21T10:05:06.027 INFO:tasks.workunit.client.0.smithi070.stdout:image: image26 2024-09-21T10:05:06.029 INFO:tasks.workunit.client.0.smithi070.stdout:image: image27 2024-09-21T10:05:06.029 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image25 2024-09-21T10:05:06.031 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image26 2024-09-21T10:05:06.032 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image27 2024-09-21T10:05:06.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:05 smithi070 ceph-mon[107385]: osdmap e581: 8 total, 8 up, 8 in 2024-09-21T10:05:06.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:06.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:05 smithi070 ceph-mon[112139]: osdmap e581: 8 total, 8 up, 8 in 2024-09-21T10:05:06.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:06.056 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCopyPP (3994 ms) 2024-09-21T10:05:06.056 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestDeepCopy 2024-09-21T10:05:06.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:05 smithi160 ceph-mon[101879]: osdmap e581: 8 total, 8 up, 8 in 2024-09-21T10:05:06.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:07.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:06 smithi160 ceph-mon[101879]: osdmap e582: 8 total, 8 up, 8 in 2024-09-21T10:05:07.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:06 smithi160 ceph-mon[101879]: pgmap v369: 233 pgs: 233 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 599 KiB/s rd, 39 KiB/s wr, 702 op/s 2024-09-21T10:05:07.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:06 smithi070 ceph-mon[107385]: osdmap e582: 8 total, 8 up, 8 in 2024-09-21T10:05:07.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:06 smithi070 ceph-mon[107385]: pgmap v369: 233 pgs: 233 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 599 KiB/s rd, 39 KiB/s wr, 702 op/s 2024-09-21T10:05:07.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:06 smithi070 ceph-mon[112139]: osdmap e582: 8 total, 8 up, 8 in 2024-09-21T10:05:07.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:06 smithi070 ceph-mon[112139]: pgmap v369: 233 pgs: 233 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 599 KiB/s rd, 39 KiB/s wr, 702 op/s 2024-09-21T10:05:07.686 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:05:07.998 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:07 smithi070 ceph-mon[112139]: osdmap e583: 8 total, 8 up, 8 in 2024-09-21T10:05:07.998 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:07 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1605235103' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-20","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:07.998 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:07 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-20","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:07.998 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:07 smithi070 ceph-mon[112139]: from='client.34377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208", "daemon_types": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:07.998 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:07 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:05:07.998 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:07 smithi070 ceph-mon[112139]: osdmap e584: 8 total, 8 up, 8 in 2024-09-21T10:05:07.998 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:07 smithi070 ceph-mon[107385]: osdmap e583: 8 total, 8 up, 8 in 2024-09-21T10:05:07.998 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:07 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1605235103' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-20","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:07.999 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:07 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-20","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:07.999 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:07 smithi070 ceph-mon[107385]: from='client.34377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208", "daemon_types": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:07.999 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:07 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:05:07.999 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:07 smithi070 ceph-mon[107385]: osdmap e584: 8 total, 8 up, 8 in 2024-09-21T10:05:08.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:07 smithi160 ceph-mon[101879]: osdmap e583: 8 total, 8 up, 8 in 2024-09-21T10:05:08.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:07 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1605235103' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-20","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:08.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:07 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-20","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:08.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:07 smithi160 ceph-mon[101879]: from='client.34377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208", "daemon_types": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:08.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:07 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:05:08.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:07 smithi160 ceph-mon[101879]: osdmap e584: 8 total, 8 up, 8 in 2024-09-21T10:05:08.862 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:08 smithi070 ceph-mon[107385]: pgmap v372: 265 pgs: 32 unknown, 233 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 599 KiB/s rd, 39 KiB/s wr, 702 op/s 2024-09-21T10:05:08.862 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:08 smithi070 ceph-mon[107385]: osdmap e585: 8 total, 8 up, 8 in 2024-09-21T10:05:08.863 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:08 smithi070 ceph-mon[112139]: pgmap v372: 265 pgs: 32 unknown, 233 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 599 KiB/s rd, 39 KiB/s wr, 702 op/s 2024-09-21T10:05:08.863 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:08 smithi070 ceph-mon[112139]: osdmap e585: 8 total, 8 up, 8 in 2024-09-21T10:05:08.896 INFO:teuthology.orchestra.run.smithi070.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:05:09.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:08 smithi160 ceph-mon[101879]: pgmap v372: 265 pgs: 32 unknown, 233 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail; 599 KiB/s rd, 39 KiB/s wr, 702 op/s 2024-09-21T10:05:09.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:08 smithi160 ceph-mon[101879]: osdmap e585: 8 total, 8 up, 8 in 2024-09-21T10:05:09.528 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-09-21T10:05:10.106 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:10.227 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:09 smithi070 ceph-mon[112139]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:05:10.227 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:09 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:10.227 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:09 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:05:10.227 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:09 smithi070 ceph-mon[112139]: osdmap e586: 8 total, 8 up, 8 in 2024-09-21T10:05:10.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:09 smithi070 ceph-mon[107385]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:05:10.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:09 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:10.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:09 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:05:10.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:09 smithi070 ceph-mon[107385]: osdmap e586: 8 total, 8 up, 8 in 2024-09-21T10:05:10.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:09 smithi160 ceph-mon[101879]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:05:10.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:09 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:10.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:09 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:05:10.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:09 smithi160 ceph-mon[101879]: osdmap e586: 8 total, 8 up, 8 in 2024-09-21T10:05:11.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:10 smithi070 ceph-mon[107385]: pgmap v375: 265 pgs: 32 unknown, 233 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:05:11.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:10 smithi070 ceph-mon[112139]: pgmap v375: 265 pgs: 32 unknown, 233 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:05:11.242 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:10 smithi160 ceph-mon[101879]: pgmap v375: 265 pgs: 32 unknown, 233 active+clean; 583 KiB data, 490 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:05:11.982 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:11 smithi070 ceph-mon[107385]: osdmap e587: 8 total, 8 up, 8 in 2024-09-21T10:05:11.983 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:11 smithi070 ceph-mon[112139]: osdmap e587: 8 total, 8 up, 8 in 2024-09-21T10:05:12.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:11 smithi160 ceph-mon[101879]: osdmap e587: 8 total, 8 up, 8 in 2024-09-21T10:05:12.984 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-09-21T10:05:12.984 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:12 smithi070 ceph-mon[107385]: osdmap e588: 8 total, 8 up, 8 in 2024-09-21T10:05:12.984 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:12 smithi070 ceph-mon[107385]: pgmap v378: 265 pgs: 265 active+clean; 583 KiB data, 494 MiB used, 715 GiB / 715 GiB avail; 458 KiB/s rd, 28 KiB/s wr, 538 op/s 2024-09-21T10:05:12.984 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:12 smithi070 ceph-mon[112139]: osdmap e588: 8 total, 8 up, 8 in 2024-09-21T10:05:12.985 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:12 smithi070 ceph-mon[112139]: pgmap v378: 265 pgs: 265 active+clean; 583 KiB data, 494 MiB used, 715 GiB / 715 GiB avail; 458 KiB/s rd, 28 KiB/s wr, 538 op/s 2024-09-21T10:05:13.308 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:05:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:05:13] "GET /metrics HTTP/1.1" 200 45826 "" "Prometheus/2.51.0" 2024-09-21T10:05:13.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:12 smithi160 ceph-mon[101879]: osdmap e588: 8 total, 8 up, 8 in 2024-09-21T10:05:13.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:12 smithi160 ceph-mon[101879]: pgmap v378: 265 pgs: 265 active+clean; 583 KiB data, 494 MiB used, 715 GiB / 715 GiB avail; 458 KiB/s rd, 28 KiB/s wr, 538 op/s 2024-09-21T10:05:13.563 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T10:05:13.563 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (5m) 96s ago 11m 20.4M - 0.27.0 11f11916f8cd 89ae95182419 2024-09-21T10:05:13.563 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi160 *:3000 running (5m) 32s ago 11m 84.5M - 10.4.8 5aad1d7cf004 daa2d8dfab48 2024-09-21T10:05:13.563 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi070.gansda smithi070 running (10m) 96s ago 10m 21.2M - 18.2.4-871-g6a1bdf85 0adab276c97e 6016b2501c8a 2024-09-21T10:05:13.563 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi160.byyhfz smithi160 running (10m) 32s ago 10m 16.9M - 18.2.4-871-g6a1bdf85 0adab276c97e 93e108d91e65 2024-09-21T10:05:13.563 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi160 *:8443,9283,8765 running (6m) 32s ago 15m 457M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 d952652a9683 2024-09-21T10:05:13.563 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (7m) 96s ago 17m 512M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 e3d390893cff 2024-09-21T10:05:13.563 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (4m) 96s ago 17m 56.0M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 4a7b667a4033 2024-09-21T10:05:13.564 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi160 running (2m) 32s ago 15m 38.6M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 86e88af21f36 2024-09-21T10:05:13.564 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (4m) 96s ago 15m 45.4M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 e154380b80a2 2024-09-21T10:05:13.564 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (6m) 96s ago 11m 11.0M - 1.7.0 72c9c2088986 5d5d4b026070 2024-09-21T10:05:13.564 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi160 *:9100 running (6m) 32s ago 11m 11.1M - 1.7.0 72c9c2088986 19b8e0e8ee0c 2024-09-21T10:05:13.564 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (15m) 96s ago 15m 114M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 10580e80f007 2024-09-21T10:05:13.564 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (14m) 96s ago 14m 117M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e b7c0d1169d1d 2024-09-21T10:05:13.564 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (14m) 96s ago 14m 140M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 58a11b8ddb81 2024-09-21T10:05:13.564 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (13m) 96s ago 13m 123M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e aa0c8a732a41 2024-09-21T10:05:13.564 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi160 running (13m) 32s ago 13m 123M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 7325a075c488 2024-09-21T10:05:13.564 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi160 running (13m) 32s ago 13m 112M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e f74026345641 2024-09-21T10:05:13.564 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi160 running (12m) 32s ago 12m 142M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 03e6bb54f8b2 2024-09-21T10:05:13.564 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi160 running (12m) 32s ago 12m 119M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e e405b21a6146 2024-09-21T10:05:13.564 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi160 *:9095 running (6m) 32s ago 11m 55.9M - 2.51.0 1d3b7f56885b 648952e57ad7 2024-09-21T10:05:13.932 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:05:13.932 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-09-21T10:05:13.932 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 3 2024-09-21T10:05:13.932 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:05:13.932 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-09-21T10:05:13.933 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T10:05:13.933 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:05:13.933 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-09-21T10:05:13.933 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 8 2024-09-21T10:05:13.933 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:05:13.933 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": { 2024-09-21T10:05:13.933 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 2 2024-09-21T10:05:13.933 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:05:13.933 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-09-21T10:05:13.933 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 10, 2024-09-21T10:05:13.933 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 5 2024-09-21T10:05:13.933 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:05:13.933 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:05:13.989 INFO:tasks.workunit.client.0.smithi070.stdout:image: image28 2024-09-21T10:05:13.989 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image28 2024-09-21T10:05:13.989 INFO:tasks.workunit.client.0.smithi070.stdout:found image28 2024-09-21T10:05:13.989 INFO:tasks.workunit.client.0.smithi070.stdout:erased image28 2024-09-21T10:05:13.989 INFO:tasks.workunit.client.0.smithi070.stdout:image: image28 2024-09-21T10:05:13.990 INFO:tasks.workunit.client.0.smithi070.stdout:image: image29 2024-09-21T10:05:13.990 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image28 2024-09-21T10:05:13.990 INFO:tasks.workunit.client.0.smithi070.stdout:found image28 2024-09-21T10:05:13.990 INFO:tasks.workunit.client.0.smithi070.stdout:erased image28 2024-09-21T10:05:13.990 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image29 2024-09-21T10:05:13.990 INFO:tasks.workunit.client.0.smithi070.stdout:found image29 2024-09-21T10:05:13.990 INFO:tasks.workunit.client.0.smithi070.stdout:erased image29 2024-09-21T10:05:13.990 INFO:tasks.workunit.client.0.smithi070.stdout:100.00% done 2024-09-21T10:05:13.990 INFO:tasks.workunit.client.0.smithi070.stdout:image: image28 2024-09-21T10:05:13.990 INFO:tasks.workunit.client.0.smithi070.stdout:image: image29 2024-09-21T10:05:13.990 INFO:tasks.workunit.client.0.smithi070.stdout:image: image30 2024-09-21T10:05:13.990 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image28 2024-09-21T10:05:13.990 INFO:tasks.workunit.client.0.smithi070.stdout:found image28 2024-09-21T10:05:13.991 INFO:tasks.workunit.client.0.smithi070.stdout:erased image28 2024-09-21T10:05:13.991 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image29 2024-09-21T10:05:13.991 INFO:tasks.workunit.client.0.smithi070.stdout:found image29 2024-09-21T10:05:13.991 INFO:tasks.workunit.client.0.smithi070.stdout:erased image29 2024-09-21T10:05:13.991 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image30 2024-09-21T10:05:13.991 INFO:tasks.workunit.client.0.smithi070.stdout:found image30 2024-09-21T10:05:13.991 INFO:tasks.workunit.client.0.smithi070.stdout:erased image30 2024-09-21T10:05:13.991 INFO:tasks.workunit.client.0.smithi070.stdout:image: image28 2024-09-21T10:05:13.991 INFO:tasks.workunit.client.0.smithi070.stdout:image: image29 2024-09-21T10:05:13.991 INFO:tasks.workunit.client.0.smithi070.stdout:image: image30 2024-09-21T10:05:13.991 INFO:tasks.workunit.client.0.smithi070.stdout:image: image31 2024-09-21T10:05:13.991 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image28 2024-09-21T10:05:13.991 INFO:tasks.workunit.client.0.smithi070.stdout:found image28 2024-09-21T10:05:13.991 INFO:tasks.workunit.client.0.smithi070.stdout:erased image28 2024-09-21T10:05:13.992 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image29 2024-09-21T10:05:13.992 INFO:tasks.workunit.client.0.smithi070.stdout:found image29 2024-09-21T10:05:13.992 INFO:tasks.workunit.client.0.smithi070.stdout:erased image29 2024-09-21T10:05:13.992 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image30 2024-09-21T10:05:13.992 INFO:tasks.workunit.client.0.smithi070.stdout:found image30 2024-09-21T10:05:13.992 INFO:tasks.workunit.client.0.smithi070.stdout:erased image30 2024-09-21T10:05:13.992 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image31 2024-09-21T10:05:13.992 INFO:tasks.workunit.client.0.smithi070.stdout:found image31 2024-09-21T10:05:13.992 INFO:tasks.workunit.client.0.smithi070.stdout:erased image31 2024-09-21T10:05:13.992 INFO:tasks.workunit.client.0.smithi070.stdout:image: image28 2024-09-21T10:05:13.992 INFO:tasks.workunit.client.0.smithi070.stdout:image: image29 2024-09-21T10:05:13.992 INFO:tasks.workunit.client.0.smithi070.stdout:image: image30 2024-09-21T10:05:13.992 INFO:tasks.workunit.client.0.smithi070.stdout:image: image31 2024-09-21T10:05:13.992 INFO:tasks.workunit.client.0.smithi070.stdout:image: image32 2024-09-21T10:05:13.993 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image28 2024-09-21T10:05:13.993 INFO:tasks.workunit.client.0.smithi070.stdout:found image28 2024-09-21T10:05:13.993 INFO:tasks.workunit.client.0.smithi070.stdout:erased image28 2024-09-21T10:05:13.993 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image29 2024-09-21T10:05:13.993 INFO:tasks.workunit.client.0.smithi070.stdout:found image29 2024-09-21T10:05:13.993 INFO:tasks.workunit.client.0.smithi070.stdout:erased image29 2024-09-21T10:05:13.993 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image30 2024-09-21T10:05:13.993 INFO:tasks.workunit.client.0.smithi070.stdout:found image30 2024-09-21T10:05:13.993 INFO:tasks.workunit.client.0.smithi070.stdout:erased image30 2024-09-21T10:05:13.993 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image31 2024-09-21T10:05:13.993 INFO:tasks.workunit.client.0.smithi070.stdout:found image31 2024-09-21T10:05:13.993 INFO:tasks.workunit.client.0.smithi070.stdout:erased image31 2024-09-21T10:05:13.993 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image32 2024-09-21T10:05:13.993 INFO:tasks.workunit.client.0.smithi070.stdout:found image32 2024-09-21T10:05:13.993 INFO:tasks.workunit.client.0.smithi070.stdout:erased image32 2024-09-21T10:05:13.994 INFO:tasks.workunit.client.0.smithi070.stdout:100.00% done 2024-09-21T10:05:13.994 INFO:tasks.workunit.client.0.smithi070.stdout:image: image28 2024-09-21T10:05:13.994 INFO:tasks.workunit.client.0.smithi070.stdout:image: image29 2024-09-21T10:05:13.994 INFO:tasks.workunit.client.0.smithi070.stdout:image: image30 2024-09-21T10:05:13.994 INFO:tasks.workunit.client.0.smithi070.stdout:image: image31 2024-09-21T10:05:13.994 INFO:tasks.workunit.client.0.smithi070.stdout:image: image32 2024-09-21T10:05:13.994 INFO:tasks.workunit.client.0.smithi070.stdout:image: image33 2024-09-21T10:05:13.994 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image28 2024-09-21T10:05:13.994 INFO:tasks.workunit.client.0.smithi070.stdout:found image28 2024-09-21T10:05:13.994 INFO:tasks.workunit.client.0.smithi070.stdout:erased image28 2024-09-21T10:05:13.994 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image29 2024-09-21T10:05:13.994 INFO:tasks.workunit.client.0.smithi070.stdout:found image29 2024-09-21T10:05:13.994 INFO:tasks.workunit.client.0.smithi070.stdout:erased image29 2024-09-21T10:05:13.994 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image30 2024-09-21T10:05:13.995 INFO:tasks.workunit.client.0.smithi070.stdout:found image30 2024-09-21T10:05:13.995 INFO:tasks.workunit.client.0.smithi070.stdout:erased image30 2024-09-21T10:05:13.995 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image31 2024-09-21T10:05:13.995 INFO:tasks.workunit.client.0.smithi070.stdout:found image31 2024-09-21T10:05:13.995 INFO:tasks.workunit.client.0.smithi070.stdout:erased image31 2024-09-21T10:05:13.995 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image32 2024-09-21T10:05:13.995 INFO:tasks.workunit.client.0.smithi070.stdout:found image32 2024-09-21T10:05:13.995 INFO:tasks.workunit.client.0.smithi070.stdout:erased image32 2024-09-21T10:05:13.995 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image33 2024-09-21T10:05:13.995 INFO:tasks.workunit.client.0.smithi070.stdout:found image33 2024-09-21T10:05:13.995 INFO:tasks.workunit.client.0.smithi070.stdout:erased image33 2024-09-21T10:05:13.995 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestDeepCopy (7933 ms) 2024-09-21T10:05:13.995 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestDeepCopyPP 2024-09-21T10:05:14.192 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:13 smithi070 ceph-mon[107385]: osdmap e589: 8 total, 8 up, 8 in 2024-09-21T10:05:14.192 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:13 smithi070 ceph-mon[107385]: from='client.44312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:14.192 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:13 smithi070 ceph-mon[112139]: osdmap e589: 8 total, 8 up, 8 in 2024-09-21T10:05:14.192 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:13 smithi070 ceph-mon[112139]: from='client.44312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:14.235 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:05:14.235 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208", 2024-09-21T10:05:14.235 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-09-21T10:05:14.235 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) mon", 2024-09-21T10:05:14.235 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-09-21T10:05:14.235 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "", 2024-09-21T10:05:14.235 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading mon daemons", 2024-09-21T10:05:14.235 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-09-21T10:05:14.235 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:05:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:13 smithi160 ceph-mon[101879]: osdmap e589: 8 total, 8 up, 8 in 2024-09-21T10:05:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:13 smithi160 ceph-mon[101879]: from='client.44312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:15.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:14 smithi160 ceph-mon[101879]: from='client.34389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:15.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:14 smithi160 ceph-mon[101879]: from='client.34395 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:15.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:14 smithi160 ceph-mon[101879]: osdmap e590: 8 total, 8 up, 8 in 2024-09-21T10:05:15.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:14 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2605167053' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:15.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:14 smithi160 ceph-mon[101879]: pgmap v381: 265 pgs: 265 active+clean; 583 KiB data, 494 MiB used, 715 GiB / 715 GiB avail; 458 KiB/s rd, 28 KiB/s wr, 538 op/s 2024-09-21T10:05:15.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:14 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:15.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:14 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:15.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:14 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:15.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:14 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:05:15.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:14 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:15.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:14 smithi070 ceph-mon[107385]: from='client.34389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:15.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:14 smithi070 ceph-mon[107385]: from='client.34395 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:15.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:14 smithi070 ceph-mon[107385]: osdmap e590: 8 total, 8 up, 8 in 2024-09-21T10:05:15.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:14 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2605167053' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:15.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:14 smithi070 ceph-mon[107385]: pgmap v381: 265 pgs: 265 active+clean; 583 KiB data, 494 MiB used, 715 GiB / 715 GiB avail; 458 KiB/s rd, 28 KiB/s wr, 538 op/s 2024-09-21T10:05:15.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:14 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:15.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:14 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:15.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:14 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:15.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:14 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:05:15.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:14 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:15.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:14 smithi070 ceph-mon[112139]: from='client.34389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:15.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:14 smithi070 ceph-mon[112139]: from='client.34395 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:15.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:14 smithi070 ceph-mon[112139]: osdmap e590: 8 total, 8 up, 8 in 2024-09-21T10:05:15.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:14 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2605167053' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:15.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:14 smithi070 ceph-mon[112139]: pgmap v381: 265 pgs: 265 active+clean; 583 KiB data, 494 MiB used, 715 GiB / 715 GiB avail; 458 KiB/s rd, 28 KiB/s wr, 538 op/s 2024-09-21T10:05:15.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:14 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:15.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:14 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:15.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:14 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:15.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:14 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:05:15.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:14 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:15.933 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:05:16.082 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:15 smithi070 ceph-mon[112139]: from='client.44342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:16.082 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:15 smithi070 ceph-mon[112139]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:05:16.082 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:15 smithi070 ceph-mon[112139]: osdmap e591: 8 total, 8 up, 8 in 2024-09-21T10:05:16.082 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:15 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3723265293' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:16.082 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:15 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:16.082 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:15 smithi070 ceph-mon[107385]: from='client.44342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:16.083 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:15 smithi070 ceph-mon[107385]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:05:16.083 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:15 smithi070 ceph-mon[107385]: osdmap e591: 8 total, 8 up, 8 in 2024-09-21T10:05:16.083 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:15 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3723265293' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:16.083 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:15 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:16.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:15 smithi160 ceph-mon[101879]: from='client.44342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:16.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:15 smithi160 ceph-mon[101879]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:05:16.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:15 smithi160 ceph-mon[101879]: osdmap e591: 8 total, 8 up, 8 in 2024-09-21T10:05:16.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:15 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3723265293' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:16.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:15 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:17.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:05:17.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: osdmap e592: 8 total, 8 up, 8 in 2024-09-21T10:05:17.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: pgmap v384: 297 pgs: 297 active+clean; 584 KiB data, 495 MiB used, 715 GiB / 715 GiB avail; 173 KiB/s rd, 8.2 KiB/s wr, 206 op/s 2024-09-21T10:05:17.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:05:17.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.130 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.130 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.130 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.130 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.130 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.130 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.130 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.130 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.130 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-21T10:05:17.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:05:17.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: osdmap e592: 8 total, 8 up, 8 in 2024-09-21T10:05:17.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: pgmap v384: 297 pgs: 297 active+clean; 584 KiB data, 495 MiB used, 715 GiB / 715 GiB avail; 173 KiB/s rd, 8.2 KiB/s wr, 206 op/s 2024-09-21T10:05:17.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:05:17.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.132 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.132 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.132 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.132 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.132 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.132 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.132 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.132 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.132 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.132 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.132 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.132 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.132 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.132 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.132 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.133 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-21T10:05:17.133 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-21T10:05:17.133 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:05:17.133 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-21T10:05:17.133 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-21T10:05:17.133 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-21T10:05:17.133 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-21T10:05:17.134 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-21T10:05:17.134 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-21T10:05:17.134 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-09-21T10:05:17.134 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-21T10:05:17.134 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-09-21T10:05:17.134 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:05:17.134 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-21T10:05:17.134 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-09-21T10:05:17.134 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-21T10:05:17.134 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-09-21T10:05:17.134 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-21T10:05:17.135 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-09-21T10:05:17.135 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-21T10:05:17.135 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-21T10:05:17.135 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:05:17.135 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:05:17.135 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:05:17.135 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:05:17.135 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:05:17.135 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:05:17.135 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-21T10:05:17.135 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-21T10:05:17.135 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:05:17.135 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:17.136 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:05:17.136 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.136 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-21T10:05:17.136 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:05:17.136 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-21T10:05:17.136 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-21T10:05:17.136 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-21T10:05:17.136 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-21T10:05:17.137 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-21T10:05:17.137 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-21T10:05:17.137 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-09-21T10:05:17.137 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-21T10:05:17.137 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-09-21T10:05:17.137 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:05:17.137 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-21T10:05:17.137 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-09-21T10:05:17.137 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-21T10:05:17.137 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-09-21T10:05:17.137 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-21T10:05:17.137 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-09-21T10:05:17.137 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-21T10:05:17.138 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-21T10:05:17.138 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:05:17.138 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:05:17.138 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:05:17.138 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:05:17.138 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:05:17.138 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:05:17.138 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-21T10:05:17.138 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-21T10:05:17.138 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:05:17.138 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:17.138 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:05:17.138 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:05:17.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: osdmap e592: 8 total, 8 up, 8 in 2024-09-21T10:05:17.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: pgmap v384: 297 pgs: 297 active+clean; 584 KiB data, 495 MiB used, 715 GiB / 715 GiB avail; 173 KiB/s rd, 8.2 KiB/s wr, 206 op/s 2024-09-21T10:05:17.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:05:17.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.344 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.344 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.344 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.344 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.344 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.344 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.344 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.344 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:17.344 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-21T10:05:17.344 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-21T10:05:17.344 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:05:17.344 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-21T10:05:17.344 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-21T10:05:17.344 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-21T10:05:17.345 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-21T10:05:17.345 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-21T10:05:17.345 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-21T10:05:17.345 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-09-21T10:05:17.345 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-21T10:05:17.345 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-09-21T10:05:17.345 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:05:17.345 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-21T10:05:17.345 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-09-21T10:05:17.345 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-21T10:05:17.345 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-09-21T10:05:17.345 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-21T10:05:17.345 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-09-21T10:05:17.345 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-21T10:05:17.346 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-21T10:05:17.346 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:05:17.346 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:05:17.346 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:05:17.346 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:05:17.346 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:05:17.346 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:05:17.346 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-21T10:05:17.346 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-21T10:05:17.346 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:05:17.346 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:17.346 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:05:17.346 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:17.937 INFO:tasks.workunit.client.0.smithi070.stdout:num images is: 1 2024-09-21T10:05:17.938 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-21T10:05:17.938 INFO:tasks.workunit.client.0.smithi070.stdout:image: image34 2024-09-21T10:05:17.938 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image34 2024-09-21T10:05:17.961 INFO:tasks.workunit.client.0.smithi070.stdout:num images is: 2 2024-09-21T10:05:17.962 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 2 2024-09-21T10:05:17.962 INFO:tasks.workunit.client.0.smithi070.stdout:image: image34 2024-09-21T10:05:17.962 INFO:tasks.workunit.client.0.smithi070.stdout:image: image35 2024-09-21T10:05:17.962 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image34 2024-09-21T10:05:17.962 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image35 2024-09-21T10:05:18.017 INFO:tasks.workunit.client.0.smithi070.stdout:100.00% done 2024-09-21T10:05:18.017 INFO:tasks.workunit.client.0.smithi070.stdout:num images is: 3 2024-09-21T10:05:18.017 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 3 2024-09-21T10:05:18.017 INFO:tasks.workunit.client.0.smithi070.stdout:image: image34 2024-09-21T10:05:18.017 INFO:tasks.workunit.client.0.smithi070.stdout:image: image35 2024-09-21T10:05:18.017 INFO:tasks.workunit.client.0.smithi070.stdout:image: image36 2024-09-21T10:05:18.017 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image34 2024-09-21T10:05:18.017 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image35 2024-09-21T10:05:18.018 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image36 2024-09-21T10:05:18.048 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestDeepCopyPP (4059 ms) 2024-09-21T10:05:18.048 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnap 2024-09-21T10:05:18.048 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:05:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:17 smithi160 ceph-mon[101879]: Upgrade: Target is version 19.3.0-5092-g62da54b4 (squid) 2024-09-21T10:05:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:17 smithi160 ceph-mon[101879]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, digests ['quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47'] 2024-09-21T10:05:18.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:17 smithi160 ceph-mon[101879]: Upgrade: Setting container_image for all mgr 2024-09-21T10:05:18.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:17 smithi160 ceph-mon[101879]: Upgrade: Setting container_image for all mon 2024-09-21T10:05:18.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:17 smithi160 ceph-mon[101879]: Upgrade: Setting container_image for all crash 2024-09-21T10:05:18.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:17 smithi160 ceph-mon[101879]: Upgrade: Setting container_image for all rgw 2024-09-21T10:05:18.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:17 smithi160 ceph-mon[101879]: Upgrade: Setting container_image for all rbd-mirror 2024-09-21T10:05:18.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:17 smithi160 ceph-mon[101879]: Upgrade: Setting container_image for all cephfs-mirror 2024-09-21T10:05:18.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:17 smithi160 ceph-mon[101879]: Upgrade: Setting container_image for all ceph-exporter 2024-09-21T10:05:18.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:17 smithi160 ceph-mon[101879]: Upgrade: Setting container_image for all iscsi 2024-09-21T10:05:18.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:17 smithi160 ceph-mon[101879]: Upgrade: Setting container_image for all nfs 2024-09-21T10:05:18.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:17 smithi160 ceph-mon[101879]: Upgrade: Setting container_image for all nvmeof 2024-09-21T10:05:18.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:17 smithi160 ceph-mon[101879]: Upgrade: Setting container_image for all node-exporter 2024-09-21T10:05:18.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:17 smithi160 ceph-mon[101879]: Upgrade: Setting container_image for all prometheus 2024-09-21T10:05:18.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:17 smithi160 ceph-mon[101879]: Upgrade: Setting container_image for all alertmanager 2024-09-21T10:05:18.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:17 smithi160 ceph-mon[101879]: Upgrade: Setting container_image for all grafana 2024-09-21T10:05:18.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:17 smithi160 ceph-mon[101879]: Upgrade: Setting container_image for all loki 2024-09-21T10:05:18.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:17 smithi160 ceph-mon[101879]: Upgrade: Setting container_image for all promtail 2024-09-21T10:05:18.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:17 smithi160 ceph-mon[101879]: Upgrade: Finalizing container_image settings 2024-09-21T10:05:18.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:17 smithi160 ceph-mon[101879]: Upgrade: Complete! 2024-09-21T10:05:18.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:17 smithi160 ceph-mon[101879]: osdmap e593: 8 total, 8 up, 8 in 2024-09-21T10:05:18.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:17 smithi160 ceph-mon[101879]: osdmap e594: 8 total, 8 up, 8 in 2024-09-21T10:05:18.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[112139]: Upgrade: Target is version 19.3.0-5092-g62da54b4 (squid) 2024-09-21T10:05:18.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[112139]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, digests ['quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47'] 2024-09-21T10:05:18.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all mgr 2024-09-21T10:05:18.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all mon 2024-09-21T10:05:18.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all crash 2024-09-21T10:05:18.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all rgw 2024-09-21T10:05:18.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all rbd-mirror 2024-09-21T10:05:18.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all cephfs-mirror 2024-09-21T10:05:18.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all ceph-exporter 2024-09-21T10:05:18.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all iscsi 2024-09-21T10:05:18.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all nfs 2024-09-21T10:05:18.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all nvmeof 2024-09-21T10:05:18.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all node-exporter 2024-09-21T10:05:18.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all prometheus 2024-09-21T10:05:18.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all alertmanager 2024-09-21T10:05:18.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all grafana 2024-09-21T10:05:18.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all loki 2024-09-21T10:05:18.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all promtail 2024-09-21T10:05:18.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[112139]: Upgrade: Finalizing container_image settings 2024-09-21T10:05:18.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[112139]: Upgrade: Complete! 2024-09-21T10:05:18.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[112139]: osdmap e593: 8 total, 8 up, 8 in 2024-09-21T10:05:18.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[112139]: osdmap e594: 8 total, 8 up, 8 in 2024-09-21T10:05:18.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[107385]: Upgrade: Target is version 19.3.0-5092-g62da54b4 (squid) 2024-09-21T10:05:18.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[107385]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, digests ['quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47'] 2024-09-21T10:05:18.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all mgr 2024-09-21T10:05:18.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all mon 2024-09-21T10:05:18.380 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all crash 2024-09-21T10:05:18.380 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all rgw 2024-09-21T10:05:18.380 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all rbd-mirror 2024-09-21T10:05:18.380 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all cephfs-mirror 2024-09-21T10:05:18.380 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all ceph-exporter 2024-09-21T10:05:18.380 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all iscsi 2024-09-21T10:05:18.380 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all nfs 2024-09-21T10:05:18.380 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all nvmeof 2024-09-21T10:05:18.380 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all node-exporter 2024-09-21T10:05:18.380 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all prometheus 2024-09-21T10:05:18.380 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all alertmanager 2024-09-21T10:05:18.380 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all grafana 2024-09-21T10:05:18.380 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all loki 2024-09-21T10:05:18.380 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all promtail 2024-09-21T10:05:18.381 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[107385]: Upgrade: Finalizing container_image settings 2024-09-21T10:05:18.381 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[107385]: Upgrade: Complete! 2024-09-21T10:05:18.381 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[107385]: osdmap e593: 8 total, 8 up, 8 in 2024-09-21T10:05:18.381 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:17 smithi070 ceph-mon[107385]: osdmap e594: 8 total, 8 up, 8 in 2024-09-21T10:05:19.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:18 smithi160 ceph-mon[101879]: pgmap v387: 297 pgs: 297 active+clean; 584 KiB data, 495 MiB used, 715 GiB / 715 GiB avail; 173 KiB/s rd, 8.2 KiB/s wr, 206 op/s 2024-09-21T10:05:19.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:18 smithi070 ceph-mon[107385]: pgmap v387: 297 pgs: 297 active+clean; 584 KiB data, 495 MiB used, 715 GiB / 715 GiB avail; 173 KiB/s rd, 8.2 KiB/s wr, 206 op/s 2024-09-21T10:05:19.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:18 smithi070 ceph-mon[112139]: pgmap v387: 297 pgs: 297 active+clean; 584 KiB data, 495 MiB used, 715 GiB / 715 GiB avail; 173 KiB/s rd, 8.2 KiB/s wr, 206 op/s 2024-09-21T10:05:20.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:19 smithi160 ceph-mon[101879]: osdmap e595: 8 total, 8 up, 8 in 2024-09-21T10:05:20.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:19 smithi070 ceph-mon[107385]: osdmap e595: 8 total, 8 up, 8 in 2024-09-21T10:05:20.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:19 smithi070 ceph-mon[112139]: osdmap e595: 8 total, 8 up, 8 in 2024-09-21T10:05:21.279 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:20 smithi070 ceph-mon[112139]: osdmap e596: 8 total, 8 up, 8 in 2024-09-21T10:05:21.279 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:20 smithi070 ceph-mon[112139]: pgmap v390: 297 pgs: 297 active+clean; 584 KiB data, 495 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:05:21.279 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:21.279 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:21.279 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:20 smithi070 ceph-mon[107385]: osdmap e596: 8 total, 8 up, 8 in 2024-09-21T10:05:21.279 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:20 smithi070 ceph-mon[107385]: pgmap v390: 297 pgs: 297 active+clean; 584 KiB data, 495 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:05:21.279 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:21.279 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:20 smithi160 ceph-mon[101879]: osdmap e596: 8 total, 8 up, 8 in 2024-09-21T10:05:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:20 smithi160 ceph-mon[101879]: pgmap v390: 297 pgs: 297 active+clean; 584 KiB data, 495 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:05:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:21.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:21.971 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 1 2024-09-21T10:05:21.971 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-21T10:05:21.971 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap1 2024-09-21T10:05:21.971 INFO:tasks.workunit.client.0.smithi070.stdout:found snap1 with size 2097152 2024-09-21T10:05:21.972 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 2 2024-09-21T10:05:21.972 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 2 2024-09-21T10:05:21.972 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap1 2024-09-21T10:05:21.972 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap2 2024-09-21T10:05:21.972 INFO:tasks.workunit.client.0.smithi070.stdout:found snap1 with size 2097152 2024-09-21T10:05:21.972 INFO:tasks.workunit.client.0.smithi070.stdout:found snap2 with size 4194304 2024-09-21T10:05:21.972 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 1 2024-09-21T10:05:21.972 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-21T10:05:21.972 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap2 2024-09-21T10:05:21.972 INFO:tasks.workunit.client.0.smithi070.stdout:found snap2 with size 4194304 2024-09-21T10:05:21.972 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 0 2024-09-21T10:05:21.972 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 0 2024-09-21T10:05:21.972 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnap (3923 ms) 2024-09-21T10:05:21.972 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestGetSnapShotTimeStamp 2024-09-21T10:05:21.973 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:05:22.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:21 smithi160 ceph-mon[101879]: osdmap e597: 8 total, 8 up, 8 in 2024-09-21T10:05:22.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:21 smithi070 ceph-mon[107385]: osdmap e597: 8 total, 8 up, 8 in 2024-09-21T10:05:22.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:21 smithi070 ceph-mon[112139]: osdmap e597: 8 total, 8 up, 8 in 2024-09-21T10:05:23.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:22 smithi160 ceph-mon[101879]: osdmap e598: 8 total, 8 up, 8 in 2024-09-21T10:05:23.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:22 smithi160 ceph-mon[101879]: pgmap v393: 297 pgs: 297 active+clean; 584 KiB data, 500 MiB used, 715 GiB / 715 GiB avail; 376 KiB/s rd, 26 KiB/s wr, 450 op/s 2024-09-21T10:05:23.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:22 smithi070 ceph-mon[112139]: osdmap e598: 8 total, 8 up, 8 in 2024-09-21T10:05:23.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:22 smithi070 ceph-mon[112139]: pgmap v393: 297 pgs: 297 active+clean; 584 KiB data, 500 MiB used, 715 GiB / 715 GiB avail; 376 KiB/s rd, 26 KiB/s wr, 450 op/s 2024-09-21T10:05:23.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:22 smithi070 ceph-mon[107385]: osdmap e598: 8 total, 8 up, 8 in 2024-09-21T10:05:23.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:22 smithi070 ceph-mon[107385]: pgmap v393: 297 pgs: 297 active+clean; 584 KiB data, 500 MiB used, 715 GiB / 715 GiB avail; 376 KiB/s rd, 26 KiB/s wr, 450 op/s 2024-09-21T10:05:23.378 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:05:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:05:23] "GET /metrics HTTP/1.1" 200 50650 "" "Prometheus/2.51.0" 2024-09-21T10:05:23.989 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestGetSnapShotTimeStamp (2018 ms) 2024-09-21T10:05:23.989 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnapPP 2024-09-21T10:05:23.989 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:05:24.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:23 smithi160 ceph-mon[101879]: osdmap e599: 8 total, 8 up, 8 in 2024-09-21T10:05:24.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:23 smithi070 ceph-mon[112139]: osdmap e599: 8 total, 8 up, 8 in 2024-09-21T10:05:24.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:23 smithi070 ceph-mon[107385]: osdmap e599: 8 total, 8 up, 8 in 2024-09-21T10:05:24.989 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 1 2024-09-21T10:05:24.989 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-21T10:05:24.990 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap1 2024-09-21T10:05:24.990 INFO:tasks.workunit.client.0.smithi070.stdout:found snap1 with size 2097152 2024-09-21T10:05:25.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:24 smithi160 ceph-mon[101879]: osdmap e600: 8 total, 8 up, 8 in 2024-09-21T10:05:25.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:24 smithi160 ceph-mon[101879]: pgmap v396: 297 pgs: 297 active+clean; 584 KiB data, 500 MiB used, 715 GiB / 715 GiB avail; 376 KiB/s rd, 26 KiB/s wr, 450 op/s 2024-09-21T10:05:25.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:24 smithi070 ceph-mon[107385]: osdmap e600: 8 total, 8 up, 8 in 2024-09-21T10:05:25.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:24 smithi070 ceph-mon[107385]: pgmap v396: 297 pgs: 297 active+clean; 584 KiB data, 500 MiB used, 715 GiB / 715 GiB avail; 376 KiB/s rd, 26 KiB/s wr, 450 op/s 2024-09-21T10:05:25.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:24 smithi070 ceph-mon[112139]: osdmap e600: 8 total, 8 up, 8 in 2024-09-21T10:05:25.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:24 smithi070 ceph-mon[112139]: pgmap v396: 297 pgs: 297 active+clean; 584 KiB data, 500 MiB used, 715 GiB / 715 GiB avail; 376 KiB/s rd, 26 KiB/s wr, 450 op/s 2024-09-21T10:05:25.998 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 2 2024-09-21T10:05:25.998 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 2 2024-09-21T10:05:25.998 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap1 2024-09-21T10:05:25.998 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap2 2024-09-21T10:05:25.998 INFO:tasks.workunit.client.0.smithi070.stdout:found snap1 with size 2097152 2024-09-21T10:05:25.998 INFO:tasks.workunit.client.0.smithi070.stdout:found snap2 with size 4194304 2024-09-21T10:05:26.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:25 smithi160 ceph-mon[101879]: osdmap e601: 8 total, 8 up, 8 in 2024-09-21T10:05:26.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:25 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:26.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:25 smithi070 ceph-mon[107385]: osdmap e601: 8 total, 8 up, 8 in 2024-09-21T10:05:26.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:25 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:26.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:25 smithi070 ceph-mon[112139]: osdmap e601: 8 total, 8 up, 8 in 2024-09-21T10:05:26.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:25 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:26.996 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 1 2024-09-21T10:05:26.996 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-21T10:05:26.996 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap2 2024-09-21T10:05:26.996 INFO:tasks.workunit.client.0.smithi070.stdout:found snap2 with size 4194304 2024-09-21T10:05:27.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:26 smithi160 ceph-mon[101879]: pgmap v399: 297 pgs: 297 active+clean; 584 KiB data, 501 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 5.7 KiB/s wr, 77 op/s 2024-09-21T10:05:27.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:26 smithi160 ceph-mon[101879]: osdmap e602: 8 total, 8 up, 8 in 2024-09-21T10:05:27.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:26 smithi070 ceph-mon[107385]: pgmap v399: 297 pgs: 297 active+clean; 584 KiB data, 501 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 5.7 KiB/s wr, 77 op/s 2024-09-21T10:05:27.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:26 smithi070 ceph-mon[107385]: osdmap e602: 8 total, 8 up, 8 in 2024-09-21T10:05:27.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:26 smithi070 ceph-mon[112139]: pgmap v399: 297 pgs: 297 active+clean; 584 KiB data, 501 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 5.7 KiB/s wr, 77 op/s 2024-09-21T10:05:27.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:26 smithi070 ceph-mon[112139]: osdmap e602: 8 total, 8 up, 8 in 2024-09-21T10:05:27.694 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 0 2024-09-21T10:05:27.694 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 0 2024-09-21T10:05:27.701 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnapPP (3712 ms) 2024-09-21T10:05:27.701 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestGetNameIdSnapPP 2024-09-21T10:05:27.702 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:05:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:27 smithi160 ceph-mon[101879]: osdmap e603: 8 total, 8 up, 8 in 2024-09-21T10:05:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:27 smithi160 ceph-mon[101879]: osdmap e604: 8 total, 8 up, 8 in 2024-09-21T10:05:28.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:27 smithi070 ceph-mon[107385]: osdmap e603: 8 total, 8 up, 8 in 2024-09-21T10:05:28.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:27 smithi070 ceph-mon[107385]: osdmap e604: 8 total, 8 up, 8 in 2024-09-21T10:05:28.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:27 smithi070 ceph-mon[112139]: osdmap e603: 8 total, 8 up, 8 in 2024-09-21T10:05:28.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:27 smithi070 ceph-mon[112139]: osdmap e604: 8 total, 8 up, 8 in 2024-09-21T10:05:29.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:28 smithi160 ceph-mon[101879]: pgmap v402: 297 pgs: 297 active+clean; 584 KiB data, 501 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 5.7 KiB/s wr, 77 op/s 2024-09-21T10:05:29.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:28 smithi160 ceph-mon[101879]: osdmap e605: 8 total, 8 up, 8 in 2024-09-21T10:05:29.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:28 smithi070 ceph-mon[107385]: pgmap v402: 297 pgs: 297 active+clean; 584 KiB data, 501 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 5.7 KiB/s wr, 77 op/s 2024-09-21T10:05:29.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:28 smithi070 ceph-mon[107385]: osdmap e605: 8 total, 8 up, 8 in 2024-09-21T10:05:29.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:28 smithi070 ceph-mon[112139]: pgmap v402: 297 pgs: 297 active+clean; 584 KiB data, 501 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 5.7 KiB/s wr, 77 op/s 2024-09-21T10:05:29.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:28 smithi070 ceph-mon[112139]: osdmap e605: 8 total, 8 up, 8 in 2024-09-21T10:05:31.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:30 smithi160 ceph-mon[101879]: osdmap e606: 8 total, 8 up, 8 in 2024-09-21T10:05:31.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:30 smithi160 ceph-mon[101879]: pgmap v405: 297 pgs: 297 active+clean; 584 KiB data, 501 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:05:31.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:30 smithi070 ceph-mon[107385]: osdmap e606: 8 total, 8 up, 8 in 2024-09-21T10:05:31.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:30 smithi070 ceph-mon[107385]: pgmap v405: 297 pgs: 297 active+clean; 584 KiB data, 501 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:05:31.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:30 smithi070 ceph-mon[112139]: osdmap e606: 8 total, 8 up, 8 in 2024-09-21T10:05:31.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:30 smithi070 ceph-mon[112139]: pgmap v405: 297 pgs: 297 active+clean; 584 KiB data, 501 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:05:32.010 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:31 smithi070 ceph-mon[107385]: osdmap e607: 8 total, 8 up, 8 in 2024-09-21T10:05:32.011 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:31 smithi070 ceph-mon[112139]: osdmap e607: 8 total, 8 up, 8 in 2024-09-21T10:05:32.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:31 smithi160 ceph-mon[101879]: osdmap e607: 8 total, 8 up, 8 in 2024-09-21T10:05:32.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:32 smithi070 ceph-mon[112139]: osdmap e608: 8 total, 8 up, 8 in 2024-09-21T10:05:32.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:32 smithi070 ceph-mon[112139]: pgmap v408: 297 pgs: 297 active+clean; 584 KiB data, 501 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 6.7 KiB/s wr, 80 op/s 2024-09-21T10:05:32.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:32 smithi070 ceph-mon[107385]: osdmap e608: 8 total, 8 up, 8 in 2024-09-21T10:05:32.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:32 smithi070 ceph-mon[107385]: pgmap v408: 297 pgs: 297 active+clean; 584 KiB data, 501 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 6.7 KiB/s wr, 80 op/s 2024-09-21T10:05:33.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:32 smithi160 ceph-mon[101879]: osdmap e608: 8 total, 8 up, 8 in 2024-09-21T10:05:33.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:32 smithi160 ceph-mon[101879]: pgmap v408: 297 pgs: 297 active+clean; 584 KiB data, 501 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 6.7 KiB/s wr, 80 op/s 2024-09-21T10:05:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:05:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:05:33] "GET /metrics HTTP/1.1" 200 50650 "" "Prometheus/2.51.0" 2024-09-21T10:05:33.723 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 0 2024-09-21T10:05:33.723 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 0 2024-09-21T10:05:33.730 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestGetNameIdSnapPP (6028 ms) 2024-09-21T10:05:33.730 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCreateLsRenameSnapPP 2024-09-21T10:05:33.730 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:05:34.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:33 smithi160 ceph-mon[101879]: osdmap e609: 8 total, 8 up, 8 in 2024-09-21T10:05:34.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:33 smithi070 ceph-mon[112139]: osdmap e609: 8 total, 8 up, 8 in 2024-09-21T10:05:34.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:33 smithi070 ceph-mon[107385]: osdmap e609: 8 total, 8 up, 8 in 2024-09-21T10:05:34.738 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 1 2024-09-21T10:05:34.738 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-21T10:05:34.738 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap1 2024-09-21T10:05:34.738 INFO:tasks.workunit.client.0.smithi070.stdout:found snap1 with size 2097152 2024-09-21T10:05:35.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:34 smithi160 ceph-mon[101879]: osdmap e610: 8 total, 8 up, 8 in 2024-09-21T10:05:35.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:34 smithi160 ceph-mon[101879]: pgmap v411: 297 pgs: 297 active+clean; 584 KiB data, 501 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 6.7 KiB/s wr, 80 op/s 2024-09-21T10:05:35.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:34 smithi070 ceph-mon[112139]: osdmap e610: 8 total, 8 up, 8 in 2024-09-21T10:05:35.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:34 smithi070 ceph-mon[112139]: pgmap v411: 297 pgs: 297 active+clean; 584 KiB data, 501 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 6.7 KiB/s wr, 80 op/s 2024-09-21T10:05:35.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:34 smithi070 ceph-mon[107385]: osdmap e610: 8 total, 8 up, 8 in 2024-09-21T10:05:35.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:34 smithi070 ceph-mon[107385]: pgmap v411: 297 pgs: 297 active+clean; 584 KiB data, 501 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 6.7 KiB/s wr, 80 op/s 2024-09-21T10:05:35.736 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 2 2024-09-21T10:05:35.736 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 2 2024-09-21T10:05:35.736 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap1 2024-09-21T10:05:35.736 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap2 2024-09-21T10:05:35.736 INFO:tasks.workunit.client.0.smithi070.stdout:found snap1 with size 2097152 2024-09-21T10:05:35.736 INFO:tasks.workunit.client.0.smithi070.stdout:found snap2 with size 4194304 2024-09-21T10:05:35.740 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 2 2024-09-21T10:05:35.740 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 2 2024-09-21T10:05:35.740 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap1-rename 2024-09-21T10:05:35.740 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap2 2024-09-21T10:05:35.740 INFO:tasks.workunit.client.0.smithi070.stdout:found snap1-rename with size 2097152 2024-09-21T10:05:35.740 INFO:tasks.workunit.client.0.smithi070.stdout:found snap2 with size 4194304 2024-09-21T10:05:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:35 smithi160 ceph-mon[101879]: osdmap e611: 8 total, 8 up, 8 in 2024-09-21T10:05:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:36.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:35 smithi070 ceph-mon[107385]: osdmap e611: 8 total, 8 up, 8 in 2024-09-21T10:05:36.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:36.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:35 smithi070 ceph-mon[112139]: osdmap e611: 8 total, 8 up, 8 in 2024-09-21T10:05:36.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:36.742 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 1 2024-09-21T10:05:36.742 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-21T10:05:36.743 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap2-rename 2024-09-21T10:05:36.743 INFO:tasks.workunit.client.0.smithi070.stdout:found snap2-rename with size 4194304 2024-09-21T10:05:37.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:36 smithi160 ceph-mon[101879]: osdmap e612: 8 total, 8 up, 8 in 2024-09-21T10:05:37.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:36 smithi160 ceph-mon[101879]: pgmap v414: 297 pgs: 297 active+clean; 584 KiB data, 502 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 6.2 KiB/s wr, 82 op/s 2024-09-21T10:05:37.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:36 smithi070 ceph-mon[112139]: osdmap e612: 8 total, 8 up, 8 in 2024-09-21T10:05:37.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:36 smithi070 ceph-mon[112139]: pgmap v414: 297 pgs: 297 active+clean; 584 KiB data, 502 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 6.2 KiB/s wr, 82 op/s 2024-09-21T10:05:37.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:36 smithi070 ceph-mon[107385]: osdmap e612: 8 total, 8 up, 8 in 2024-09-21T10:05:37.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:36 smithi070 ceph-mon[107385]: pgmap v414: 297 pgs: 297 active+clean; 584 KiB data, 502 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 6.2 KiB/s wr, 82 op/s 2024-09-21T10:05:37.696 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 0 2024-09-21T10:05:37.696 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 0 2024-09-21T10:05:37.704 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCreateLsRenameSnapPP (3975 ms) 2024-09-21T10:05:37.704 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ConcurrentCreatesUnvalidatedPool 2024-09-21T10:05:38.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:37 smithi160 ceph-mon[101879]: osdmap e613: 8 total, 8 up, 8 in 2024-09-21T10:05:38.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:37 smithi160 ceph-mon[101879]: osdmap e614: 8 total, 8 up, 8 in 2024-09-21T10:05:38.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:37 smithi070 ceph-mon[107385]: osdmap e613: 8 total, 8 up, 8 in 2024-09-21T10:05:38.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:37 smithi070 ceph-mon[107385]: osdmap e614: 8 total, 8 up, 8 in 2024-09-21T10:05:38.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:37 smithi070 ceph-mon[112139]: osdmap e613: 8 total, 8 up, 8 in 2024-09-21T10:05:38.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:37 smithi070 ceph-mon[112139]: osdmap e614: 8 total, 8 up, 8 in 2024-09-21T10:05:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:38 smithi160 ceph-mon[101879]: pgmap v417: 297 pgs: 297 active+clean; 584 KiB data, 502 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 6.2 KiB/s wr, 82 op/s 2024-09-21T10:05:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:38 smithi160 ceph-mon[101879]: osdmap e615: 8 total, 8 up, 8 in 2024-09-21T10:05:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:38 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3704718082' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-22","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:39.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:38 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-22","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:39.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:38 smithi070 ceph-mon[107385]: pgmap v417: 297 pgs: 297 active+clean; 584 KiB data, 502 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 6.2 KiB/s wr, 82 op/s 2024-09-21T10:05:39.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:38 smithi070 ceph-mon[107385]: osdmap e615: 8 total, 8 up, 8 in 2024-09-21T10:05:39.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:38 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3704718082' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-22","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:39.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:38 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-22","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:39.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:38 smithi070 ceph-mon[112139]: pgmap v417: 297 pgs: 297 active+clean; 584 KiB data, 502 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 6.2 KiB/s wr, 82 op/s 2024-09-21T10:05:39.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:38 smithi070 ceph-mon[112139]: osdmap e615: 8 total, 8 up, 8 in 2024-09-21T10:05:39.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:38 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3704718082' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-22","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:39.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:38 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-22","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:39.695 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:05:39.696 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:05:39.696 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:05:39.696 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:05:41.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:40 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:05:41.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:40 smithi160 ceph-mon[101879]: osdmap e616: 8 total, 8 up, 8 in 2024-09-21T10:05:41.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:40 smithi160 ceph-mon[101879]: pgmap v420: 329 pgs: 32 unknown, 297 active+clean; 584 KiB data, 502 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:05:41.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:40 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:05:41.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:40 smithi070 ceph-mon[107385]: osdmap e616: 8 total, 8 up, 8 in 2024-09-21T10:05:41.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:40 smithi070 ceph-mon[107385]: pgmap v420: 329 pgs: 32 unknown, 297 active+clean; 584 KiB data, 502 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:05:41.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:40 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:05:41.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:40 smithi070 ceph-mon[112139]: osdmap e616: 8 total, 8 up, 8 in 2024-09-21T10:05:41.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:40 smithi070 ceph-mon[112139]: pgmap v420: 329 pgs: 32 unknown, 297 active+clean; 584 KiB data, 502 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:05:41.939 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ConcurrentCreatesUnvalidatedPool (4235 ms) 2024-09-21T10:05:41.940 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.RemoveFullTry 2024-09-21T10:05:41.993 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:41 smithi160 ceph-mon[101879]: osdmap e617: 8 total, 8 up, 8 in 2024-09-21T10:05:42.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:41 smithi070 ceph-mon[107385]: osdmap e617: 8 total, 8 up, 8 in 2024-09-21T10:05:42.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:41 smithi070 ceph-mon[112139]: osdmap e617: 8 total, 8 up, 8 in 2024-09-21T10:05:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:42 smithi160 ceph-mon[101879]: osdmap e618: 8 total, 8 up, 8 in 2024-09-21T10:05:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:42 smithi160 ceph-mon[101879]: pgmap v423: 329 pgs: 329 active+clean; 584 KiB data, 503 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 1.7 KiB/s wr, 25 op/s 2024-09-21T10:05:43.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:42 smithi070 ceph-mon[107385]: osdmap e618: 8 total, 8 up, 8 in 2024-09-21T10:05:43.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:42 smithi070 ceph-mon[107385]: pgmap v423: 329 pgs: 329 active+clean; 584 KiB data, 503 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 1.7 KiB/s wr, 25 op/s 2024-09-21T10:05:43.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:42 smithi070 ceph-mon[112139]: osdmap e618: 8 total, 8 up, 8 in 2024-09-21T10:05:43.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:42 smithi070 ceph-mon[112139]: pgmap v423: 329 pgs: 329 active+clean; 584 KiB data, 503 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 1.7 KiB/s wr, 25 op/s 2024-09-21T10:05:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:05:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:05:43] "GET /metrics HTTP/1.1" 200 50702 "" "Prometheus/2.51.0" 2024-09-21T10:05:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:43 smithi160 ceph-mon[101879]: osdmap e619: 8 total, 8 up, 8 in 2024-09-21T10:05:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:43 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1331636661' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:44.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:43 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:44.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:43 smithi070 ceph-mon[107385]: osdmap e619: 8 total, 8 up, 8 in 2024-09-21T10:05:44.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:43 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1331636661' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:44.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:43 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:44.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:43 smithi070 ceph-mon[112139]: osdmap e619: 8 total, 8 up, 8 in 2024-09-21T10:05:44.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:43 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1331636661' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:44.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:43 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:44.835 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:44 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:05:44.835 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:44 smithi070 ceph-mon[107385]: osdmap e620: 8 total, 8 up, 8 in 2024-09-21T10:05:44.835 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:44 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-23","app": "rbd","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:44.835 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:44 smithi070 ceph-mon[107385]: pgmap v426: 361 pgs: 32 unknown, 329 active+clean; 584 KiB data, 503 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 1.7 KiB/s wr, 25 op/s 2024-09-21T10:05:44.835 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:44 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:05:44.835 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:44 smithi070 ceph-mon[112139]: osdmap e620: 8 total, 8 up, 8 in 2024-09-21T10:05:44.835 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:44 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-23","app": "rbd","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:44.835 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:44 smithi070 ceph-mon[112139]: pgmap v426: 361 pgs: 32 unknown, 329 active+clean; 584 KiB data, 503 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 1.7 KiB/s wr, 25 op/s 2024-09-21T10:05:45.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:44 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:05:45.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:44 smithi160 ceph-mon[101879]: osdmap e620: 8 total, 8 up, 8 in 2024-09-21T10:05:45.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:44 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-23","app": "rbd","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:45.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:44 smithi160 ceph-mon[101879]: pgmap v426: 361 pgs: 32 unknown, 329 active+clean; 584 KiB data, 503 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 1.7 KiB/s wr, 25 op/s 2024-09-21T10:05:45.223 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'sleep 60' 2024-09-21T10:05:45.420 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:45.822 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:45 smithi070 ceph-mon[107385]: from='client.34431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:45.822 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:45 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-23","app": "rbd","yes_i_really_mean_it": true}]': finished 2024-09-21T10:05:45.822 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:45 smithi070 ceph-mon[107385]: osdmap e621: 8 total, 8 up, 8 in 2024-09-21T10:05:45.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:45 smithi070 ceph-mon[112139]: from='client.34431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:45.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:45 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-23","app": "rbd","yes_i_really_mean_it": true}]': finished 2024-09-21T10:05:45.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:45 smithi070 ceph-mon[112139]: osdmap e621: 8 total, 8 up, 8 in 2024-09-21T10:05:46.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:45 smithi160 ceph-mon[101879]: from='client.34431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:46.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:45 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-23","app": "rbd","yes_i_really_mean_it": true}]': finished 2024-09-21T10:05:46.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:45 smithi160 ceph-mon[101879]: osdmap e621: 8 total, 8 up, 8 in 2024-09-21T10:05:46.745 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:05:46.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:05:46 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:05:46] ENGINE Bus STOPPING 2024-09-21T10:05:46.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:46 smithi070 ceph-mon[107385]: osdmap e622: 8 total, 8 up, 8 in 2024-09-21T10:05:46.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:46 smithi070 ceph-mon[107385]: pgmap v429: 361 pgs: 361 active+clean; 584 KiB data, 503 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 8.2 KiB/s wr, 144 op/s 2024-09-21T10:05:46.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:46 smithi070 ceph-mon[112139]: osdmap e622: 8 total, 8 up, 8 in 2024-09-21T10:05:46.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:46 smithi070 ceph-mon[112139]: pgmap v429: 361 pgs: 361 active+clean; 584 KiB data, 503 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 8.2 KiB/s wr, 144 op/s 2024-09-21T10:05:47.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:46 smithi160 ceph-mon[101879]: osdmap e622: 8 total, 8 up, 8 in 2024-09-21T10:05:47.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:46 smithi160 ceph-mon[101879]: pgmap v429: 361 pgs: 361 active+clean; 584 KiB data, 503 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 8.2 KiB/s wr, 144 op/s 2024-09-21T10:05:47.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:05:47 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:05:47] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T10:05:47.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:05:47 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:05:47] ENGINE Bus STOPPED 2024-09-21T10:05:47.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:05:47 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:05:47] ENGINE Bus STARTING 2024-09-21T10:05:47.628 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:05:47 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:05:47] ENGINE Serving on http://:::9283 2024-09-21T10:05:47.628 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:05:47 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:05:47] ENGINE Bus STARTED 2024-09-21T10:05:48.066 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:47 smithi160 ceph-mon[101879]: osdmap e623: 8 total, 8 up, 8 in 2024-09-21T10:05:48.066 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:47 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' 2024-09-21T10:05:48.067 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:47 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:05:48.067 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:47 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-23", "field": "max_bytes", "val": "10485760"} : dispatch 2024-09-21T10:05:48.067 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:47 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:05:48.067 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:47 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-23", "field": "max_bytes", "val": "10485760"}]': finished 2024-09-21T10:05:48.067 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:47 smithi160 ceph-mon[101879]: osdmap e624: 8 total, 8 up, 8 in 2024-09-21T10:05:48.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:47 smithi070 ceph-mon[107385]: osdmap e623: 8 total, 8 up, 8 in 2024-09-21T10:05:48.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:47 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' 2024-09-21T10:05:48.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:47 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:05:48.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:47 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-23", "field": "max_bytes", "val": "10485760"} : dispatch 2024-09-21T10:05:48.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:47 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:05:48.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:47 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-23", "field": "max_bytes", "val": "10485760"}]': finished 2024-09-21T10:05:48.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:47 smithi070 ceph-mon[107385]: osdmap e624: 8 total, 8 up, 8 in 2024-09-21T10:05:48.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:47 smithi070 ceph-mon[112139]: osdmap e623: 8 total, 8 up, 8 in 2024-09-21T10:05:48.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:47 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' 2024-09-21T10:05:48.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:47 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:05:48.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:47 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-23", "field": "max_bytes", "val": "10485760"} : dispatch 2024-09-21T10:05:48.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:47 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:05:48.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:47 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-23", "field": "max_bytes", "val": "10485760"}]': finished 2024-09-21T10:05:48.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:47 smithi070 ceph-mon[112139]: osdmap e624: 8 total, 8 up, 8 in 2024-09-21T10:05:49.012 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:48 smithi160 ceph-mon[101879]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:05:49.012 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:48 smithi160 ceph-mon[101879]: pgmap v432: 361 pgs: 361 active+clean; 584 KiB data, 503 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 8.2 KiB/s wr, 144 op/s 2024-09-21T10:05:49.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:48 smithi070 ceph-mon[107385]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:05:49.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:48 smithi070 ceph-mon[107385]: pgmap v432: 361 pgs: 361 active+clean; 584 KiB data, 503 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 8.2 KiB/s wr, 144 op/s 2024-09-21T10:05:49.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:48 smithi070 ceph-mon[112139]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:05:49.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:48 smithi070 ceph-mon[112139]: pgmap v432: 361 pgs: 361 active+clean; 584 KiB data, 503 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 8.2 KiB/s wr, 144 op/s 2024-09-21T10:05:50.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.8", "id": [7, 4]} : dispatch 2024-09-21T10:05:50.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.17", "id": [7, 0]} : dispatch 2024-09-21T10:05:50.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.18", "id": [7, 5]} : dispatch 2024-09-21T10:05:50.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.1e", "id": [7, 4]} : dispatch 2024-09-21T10:05:50.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.5", "id": [7, 0]} : dispatch 2024-09-21T10:05:50.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.14", "id": [7, 0]} : dispatch 2024-09-21T10:05:50.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1a", "id": [7, 2]} : dispatch 2024-09-21T10:05:50.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.d", "id": [0, 2]} : dispatch 2024-09-21T10:05:50.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.15", "id": [0, 2]} : dispatch 2024-09-21T10:05:50.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.8", "id": [7, 4]} : dispatch 2024-09-21T10:05:50.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.17", "id": [7, 0]} : dispatch 2024-09-21T10:05:50.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.18", "id": [7, 5]} : dispatch 2024-09-21T10:05:50.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.1e", "id": [7, 4]} : dispatch 2024-09-21T10:05:50.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.5", "id": [7, 0]} : dispatch 2024-09-21T10:05:50.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.14", "id": [7, 0]} : dispatch 2024-09-21T10:05:50.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1a", "id": [7, 2]} : dispatch 2024-09-21T10:05:50.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.d", "id": [0, 2]} : dispatch 2024-09-21T10:05:50.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.15", "id": [0, 2]} : dispatch 2024-09-21T10:05:50.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.8", "id": [7, 4]} : dispatch 2024-09-21T10:05:50.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.17", "id": [7, 0]} : dispatch 2024-09-21T10:05:50.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.18", "id": [7, 5]} : dispatch 2024-09-21T10:05:50.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.1e", "id": [7, 4]} : dispatch 2024-09-21T10:05:50.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.5", "id": [7, 0]} : dispatch 2024-09-21T10:05:50.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.14", "id": [7, 0]} : dispatch 2024-09-21T10:05:50.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1a", "id": [7, 2]} : dispatch 2024-09-21T10:05:50.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.d", "id": [0, 2]} : dispatch 2024-09-21T10:05:50.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.15", "id": [0, 2]} : dispatch 2024-09-21T10:05:51.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[107385]: pgmap v433: 361 pgs: 361 active+clean; 584 KiB data, 503 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 6.3 KiB/s wr, 109 op/s 2024-09-21T10:05:51.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.8", "id": [7, 4]}]': finished 2024-09-21T10:05:51.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.17", "id": [7, 0]}]': finished 2024-09-21T10:05:51.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.18", "id": [7, 5]}]': finished 2024-09-21T10:05:51.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.1e", "id": [7, 4]}]': finished 2024-09-21T10:05:51.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.5", "id": [7, 0]}]': finished 2024-09-21T10:05:51.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.14", "id": [7, 0]}]': finished 2024-09-21T10:05:51.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1a", "id": [7, 2]}]': finished 2024-09-21T10:05:51.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.d", "id": [0, 2]}]': finished 2024-09-21T10:05:51.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.15", "id": [0, 2]}]': finished 2024-09-21T10:05:51.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[107385]: osdmap e625: 8 total, 8 up, 8 in 2024-09-21T10:05:51.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:51.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:51.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:51.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[112139]: pgmap v433: 361 pgs: 361 active+clean; 584 KiB data, 503 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 6.3 KiB/s wr, 109 op/s 2024-09-21T10:05:51.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.8", "id": [7, 4]}]': finished 2024-09-21T10:05:51.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.17", "id": [7, 0]}]': finished 2024-09-21T10:05:51.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.18", "id": [7, 5]}]': finished 2024-09-21T10:05:51.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.1e", "id": [7, 4]}]': finished 2024-09-21T10:05:51.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.5", "id": [7, 0]}]': finished 2024-09-21T10:05:51.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.14", "id": [7, 0]}]': finished 2024-09-21T10:05:51.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1a", "id": [7, 2]}]': finished 2024-09-21T10:05:51.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.d", "id": [0, 2]}]': finished 2024-09-21T10:05:51.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.15", "id": [0, 2]}]': finished 2024-09-21T10:05:51.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[112139]: osdmap e625: 8 total, 8 up, 8 in 2024-09-21T10:05:51.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:51.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:51.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:51 smithi160 ceph-mon[101879]: pgmap v433: 361 pgs: 361 active+clean; 584 KiB data, 503 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 6.3 KiB/s wr, 109 op/s 2024-09-21T10:05:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.8", "id": [7, 4]}]': finished 2024-09-21T10:05:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.17", "id": [7, 0]}]': finished 2024-09-21T10:05:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.18", "id": [7, 5]}]': finished 2024-09-21T10:05:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.1e", "id": [7, 4]}]': finished 2024-09-21T10:05:51.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.5", "id": [7, 0]}]': finished 2024-09-21T10:05:51.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.14", "id": [7, 0]}]': finished 2024-09-21T10:05:51.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1a", "id": [7, 2]}]': finished 2024-09-21T10:05:51.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.d", "id": [0, 2]}]': finished 2024-09-21T10:05:51.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.15", "id": [0, 2]}]': finished 2024-09-21T10:05:51.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:51 smithi160 ceph-mon[101879]: osdmap e625: 8 total, 8 up, 8 in 2024-09-21T10:05:51.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:51.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:51.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:52.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:52 smithi160 ceph-mon[101879]: osdmap e626: 8 total, 8 up, 8 in 2024-09-21T10:05:52.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:52 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:52.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:52 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:05:52.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:52 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:52.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:52 smithi070 ceph-mon[107385]: osdmap e626: 8 total, 8 up, 8 in 2024-09-21T10:05:52.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:52 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:52.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:52 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:05:52.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:52 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:52.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:52 smithi070 ceph-mon[112139]: osdmap e626: 8 total, 8 up, 8 in 2024-09-21T10:05:52.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:52 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:52.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:52 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:05:52.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:52 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:05:53.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:53 smithi160 ceph-mon[101879]: pgmap v436: 361 pgs: 9 peering, 352 active+clean; 4.6 MiB data, 534 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 781 KiB/s wr, 11 op/s 2024-09-21T10:05:53.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:53 smithi160 ceph-mon[101879]: osdmap e627: 8 total, 8 up, 8 in 2024-09-21T10:05:53.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:53 smithi070 ceph-mon[107385]: pgmap v436: 361 pgs: 9 peering, 352 active+clean; 4.6 MiB data, 534 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 781 KiB/s wr, 11 op/s 2024-09-21T10:05:53.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:53 smithi070 ceph-mon[107385]: osdmap e627: 8 total, 8 up, 8 in 2024-09-21T10:05:53.378 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:05:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:05:53] "GET /metrics HTTP/1.1" 200 55509 "" "Prometheus/2.51.0" 2024-09-21T10:05:53.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:53 smithi070 ceph-mon[112139]: pgmap v436: 361 pgs: 9 peering, 352 active+clean; 4.6 MiB data, 534 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 781 KiB/s wr, 11 op/s 2024-09-21T10:05:53.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:53 smithi070 ceph-mon[112139]: osdmap e627: 8 total, 8 up, 8 in 2024-09-21T10:05:55.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:55 smithi160 ceph-mon[101879]: pgmap v438: 361 pgs: 9 peering, 352 active+clean; 4.6 MiB data, 534 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 684 KiB/s wr, 10 op/s 2024-09-21T10:05:55.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:55 smithi070 ceph-mon[112139]: pgmap v438: 361 pgs: 9 peering, 352 active+clean; 4.6 MiB data, 534 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 684 KiB/s wr, 10 op/s 2024-09-21T10:05:55.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:55 smithi070 ceph-mon[107385]: pgmap v438: 361 pgs: 9 peering, 352 active+clean; 4.6 MiB data, 534 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 684 KiB/s wr, 10 op/s 2024-09-21T10:05:57.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:57 smithi160 ceph-mon[101879]: pgmap v439: 361 pgs: 361 active+clean; 9.6 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 1.5 MiB/s wr, 13 op/s; 5 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:05:57.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:57 smithi070 ceph-mon[107385]: pgmap v439: 361 pgs: 361 active+clean; 9.6 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 1.5 MiB/s wr, 13 op/s; 5 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:05:57.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:57 smithi070 ceph-mon[112139]: pgmap v439: 361 pgs: 361 active+clean; 9.6 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 1.5 MiB/s wr, 13 op/s; 5 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:05:59.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:05:59 smithi160 ceph-mon[101879]: pgmap v440: 361 pgs: 361 active+clean; 9.6 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 1.1 MiB/s wr, 10 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:05:59.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:05:59 smithi070 ceph-mon[107385]: pgmap v440: 361 pgs: 361 active+clean; 9.6 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 1.1 MiB/s wr, 10 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:05:59.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:05:59 smithi070 ceph-mon[112139]: pgmap v440: 361 pgs: 361 active+clean; 9.6 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 1.1 MiB/s wr, 10 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:06:01.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:01 smithi160 ceph-mon[101879]: pgmap v441: 361 pgs: 361 active+clean; 9.6 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 572 KiB/s wr, 2 op/s; 3 B/s, 0 keys/s, 0 objects/s recovering 2024-09-21T10:06:01.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:01 smithi070 ceph-mon[107385]: pgmap v441: 361 pgs: 361 active+clean; 9.6 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 572 KiB/s wr, 2 op/s; 3 B/s, 0 keys/s, 0 objects/s recovering 2024-09-21T10:06:01.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:01 smithi070 ceph-mon[112139]: pgmap v441: 361 pgs: 361 active+clean; 9.6 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 572 KiB/s wr, 2 op/s; 3 B/s, 0 keys/s, 0 objects/s recovering 2024-09-21T10:06:02.858 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.RemoveFullTry (20918 ms) 2024-09-21T10:06:02.858 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.RemoveFullTryDataPool 2024-09-21T10:06:03.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:03 smithi160 ceph-mon[101879]: pgmap v442: 361 pgs: 361 active+clean; 15 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.0 MiB/s wr, 3 op/s; 3 B/s, 0 keys/s, 0 objects/s recovering 2024-09-21T10:06:03.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:03 smithi160 ceph-mon[101879]: pool 'test-librbd-smithi070-86016-23' is full (reached quota's max_bytes: 10 MiB) 2024-09-21T10:06:03.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:03 smithi160 ceph-mon[101879]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-21T10:06:03.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:03 smithi160 ceph-mon[101879]: osdmap e628: 8 total, 8 up, 8 in 2024-09-21T10:06:03.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:06:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:06:03] "GET /metrics HTTP/1.1" 200 55509 "" "Prometheus/2.51.0" 2024-09-21T10:06:03.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:03 smithi070 ceph-mon[107385]: pgmap v442: 361 pgs: 361 active+clean; 15 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.0 MiB/s wr, 3 op/s; 3 B/s, 0 keys/s, 0 objects/s recovering 2024-09-21T10:06:03.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:03 smithi070 ceph-mon[107385]: pool 'test-librbd-smithi070-86016-23' is full (reached quota's max_bytes: 10 MiB) 2024-09-21T10:06:03.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:03 smithi070 ceph-mon[107385]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-21T10:06:03.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:03 smithi070 ceph-mon[107385]: osdmap e628: 8 total, 8 up, 8 in 2024-09-21T10:06:03.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:03 smithi070 ceph-mon[112139]: pgmap v442: 361 pgs: 361 active+clean; 15 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.0 MiB/s wr, 3 op/s; 3 B/s, 0 keys/s, 0 objects/s recovering 2024-09-21T10:06:03.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:03 smithi070 ceph-mon[112139]: pool 'test-librbd-smithi070-86016-23' is full (reached quota's max_bytes: 10 MiB) 2024-09-21T10:06:03.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:03 smithi070 ceph-mon[112139]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-21T10:06:03.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:03 smithi070 ceph-mon[112139]: osdmap e628: 8 total, 8 up, 8 in 2024-09-21T10:06:05.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:04 smithi160 ceph-mon[101879]: osdmap e629: 8 total, 8 up, 8 in 2024-09-21T10:06:05.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:04 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3205822190' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:05.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:04 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:05.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:04 smithi160 ceph-mon[101879]: pgmap v445: 393 pgs: 32 unknown, 361 active+clean; 15 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 640 KiB/s wr, 1 op/s 2024-09-21T10:06:05.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:04 smithi070 ceph-mon[107385]: osdmap e629: 8 total, 8 up, 8 in 2024-09-21T10:06:05.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:04 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3205822190' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:05.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:04 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:05.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:04 smithi070 ceph-mon[107385]: pgmap v445: 393 pgs: 32 unknown, 361 active+clean; 15 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 640 KiB/s wr, 1 op/s 2024-09-21T10:06:05.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:04 smithi070 ceph-mon[112139]: osdmap e629: 8 total, 8 up, 8 in 2024-09-21T10:06:05.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:04 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3205822190' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:05.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:04 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:05.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:04 smithi070 ceph-mon[112139]: pgmap v445: 393 pgs: 32 unknown, 361 active+clean; 15 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 640 KiB/s wr, 1 op/s 2024-09-21T10:06:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:05 smithi160 ceph-mon[101879]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:06:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:05 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:06:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:05 smithi160 ceph-mon[101879]: osdmap e630: 8 total, 8 up, 8 in 2024-09-21T10:06:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:06.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:05 smithi070 ceph-mon[107385]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:06:06.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:05 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:06:06.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:05 smithi070 ceph-mon[107385]: osdmap e630: 8 total, 8 up, 8 in 2024-09-21T10:06:06.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:06.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:05 smithi070 ceph-mon[112139]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:06:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:05 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:06:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:05 smithi070 ceph-mon[112139]: osdmap e630: 8 total, 8 up, 8 in 2024-09-21T10:06:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:07.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:06 smithi160 ceph-mon[101879]: osdmap e631: 8 total, 8 up, 8 in 2024-09-21T10:06:07.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:06 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2808758407' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:07.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:06 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:07.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:06 smithi160 ceph-mon[101879]: pgmap v448: 425 pgs: 32 creating+peering, 393 active+clean; 584 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 258 KiB/s wr, 39 op/s 2024-09-21T10:06:07.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:06 smithi070 ceph-mon[107385]: osdmap e631: 8 total, 8 up, 8 in 2024-09-21T10:06:07.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:06 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2808758407' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:07.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:06 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:07.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:06 smithi070 ceph-mon[107385]: pgmap v448: 425 pgs: 32 creating+peering, 393 active+clean; 584 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 258 KiB/s wr, 39 op/s 2024-09-21T10:06:07.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:06 smithi070 ceph-mon[112139]: osdmap e631: 8 total, 8 up, 8 in 2024-09-21T10:06:07.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:06 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2808758407' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:07.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:06 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:07.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:06 smithi070 ceph-mon[112139]: pgmap v448: 425 pgs: 32 creating+peering, 393 active+clean; 584 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 258 KiB/s wr, 39 op/s 2024-09-21T10:06:08.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:07 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:06:08.094 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:07 smithi160 ceph-mon[101879]: osdmap e632: 8 total, 8 up, 8 in 2024-09-21T10:06:08.094 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:07 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-24","app": "rbd","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:08.094 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:07 smithi160 ceph-mon[101879]: pool 'test-librbd-smithi070-86016-23' no longer out of quota; removing NO_QUOTA flag 2024-09-21T10:06:08.094 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:07 smithi160 ceph-mon[101879]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-21T10:06:08.094 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:07 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-24","app": "rbd","yes_i_really_mean_it": true}]': finished 2024-09-21T10:06:08.094 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:07 smithi160 ceph-mon[101879]: osdmap e633: 8 total, 8 up, 8 in 2024-09-21T10:06:08.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:07 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:06:08.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:07 smithi070 ceph-mon[107385]: osdmap e632: 8 total, 8 up, 8 in 2024-09-21T10:06:08.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:07 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-24","app": "rbd","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:08.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:07 smithi070 ceph-mon[107385]: pool 'test-librbd-smithi070-86016-23' no longer out of quota; removing NO_QUOTA flag 2024-09-21T10:06:08.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:07 smithi070 ceph-mon[107385]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-21T10:06:08.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:07 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-24","app": "rbd","yes_i_really_mean_it": true}]': finished 2024-09-21T10:06:08.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:07 smithi070 ceph-mon[107385]: osdmap e633: 8 total, 8 up, 8 in 2024-09-21T10:06:08.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:07 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:06:08.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:07 smithi070 ceph-mon[112139]: osdmap e632: 8 total, 8 up, 8 in 2024-09-21T10:06:08.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:07 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-24","app": "rbd","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:08.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:07 smithi070 ceph-mon[112139]: pool 'test-librbd-smithi070-86016-23' no longer out of quota; removing NO_QUOTA flag 2024-09-21T10:06:08.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:07 smithi070 ceph-mon[112139]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-21T10:06:08.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:07 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-24","app": "rbd","yes_i_really_mean_it": true}]': finished 2024-09-21T10:06:08.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:07 smithi070 ceph-mon[112139]: osdmap e633: 8 total, 8 up, 8 in 2024-09-21T10:06:09.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:08 smithi160 ceph-mon[101879]: pgmap v451: 425 pgs: 32 creating+peering, 393 active+clean; 584 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 258 KiB/s wr, 39 op/s 2024-09-21T10:06:09.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:08 smithi160 ceph-mon[101879]: osdmap e634: 8 total, 8 up, 8 in 2024-09-21T10:06:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:08 smithi070 ceph-mon[107385]: pgmap v451: 425 pgs: 32 creating+peering, 393 active+clean; 584 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 258 KiB/s wr, 39 op/s 2024-09-21T10:06:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:08 smithi070 ceph-mon[107385]: osdmap e634: 8 total, 8 up, 8 in 2024-09-21T10:06:09.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:08 smithi070 ceph-mon[112139]: pgmap v451: 425 pgs: 32 creating+peering, 393 active+clean; 584 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 258 KiB/s wr, 39 op/s 2024-09-21T10:06:09.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:08 smithi070 ceph-mon[112139]: osdmap e634: 8 total, 8 up, 8 in 2024-09-21T10:06:09.715 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:06:10.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:09 smithi160 ceph-mon[101879]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:06:10.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:09 smithi160 ceph-mon[101879]: Cluster is now healthy 2024-09-21T10:06:10.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:09 smithi160 ceph-mon[101879]: osdmap e635: 8 total, 8 up, 8 in 2024-09-21T10:06:10.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:09 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' 2024-09-21T10:06:10.127 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:06:09 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:06:09] ENGINE Bus STOPPING 2024-09-21T10:06:10.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:09 smithi070 ceph-mon[107385]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:06:10.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:09 smithi070 ceph-mon[107385]: Cluster is now healthy 2024-09-21T10:06:10.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:09 smithi070 ceph-mon[107385]: osdmap e635: 8 total, 8 up, 8 in 2024-09-21T10:06:10.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:09 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' 2024-09-21T10:06:10.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:09 smithi070 ceph-mon[112139]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:06:10.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:09 smithi070 ceph-mon[112139]: Cluster is now healthy 2024-09-21T10:06:10.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:09 smithi070 ceph-mon[112139]: osdmap e635: 8 total, 8 up, 8 in 2024-09-21T10:06:10.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:09 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' 2024-09-21T10:06:10.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:06:10 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:06:10] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T10:06:10.628 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:06:10 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:06:10] ENGINE Bus STOPPED 2024-09-21T10:06:10.628 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:06:10 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:06:10] ENGINE Bus STARTING 2024-09-21T10:06:10.628 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:06:10 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:06:10] ENGINE Serving on http://:::9283 2024-09-21T10:06:10.628 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:06:10 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:06:10] ENGINE Bus STARTED 2024-09-21T10:06:11.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:10 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:06:11.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:10 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:06:11.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:10 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:06:11.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:10 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:11.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:10 smithi160 ceph-mon[101879]: pgmap v454: 425 pgs: 32 creating+peering, 393 active+clean; 584 KiB data, 538 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:06:11.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:10 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:06:11.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:10 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:06:11.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:10 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:06:11.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:10 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:06:11.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:10 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:11.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:10 smithi070 ceph-mon[107385]: pgmap v454: 425 pgs: 32 creating+peering, 393 active+clean; 584 KiB data, 538 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:06:11.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:10 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:06:11.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:10 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:06:11.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:10 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:06:11.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:10 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:06:11.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:10 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:11.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:10 smithi070 ceph-mon[112139]: pgmap v454: 425 pgs: 32 creating+peering, 393 active+clean; 584 KiB data, 538 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:06:11.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:10 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:06:11.992 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:11 smithi160 ceph-mon[101879]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:06:11.993 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:11 smithi160 ceph-mon[101879]: osdmap e636: 8 total, 8 up, 8 in 2024-09-21T10:06:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:11 smithi070 ceph-mon[107385]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:06:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:11 smithi070 ceph-mon[107385]: osdmap e636: 8 total, 8 up, 8 in 2024-09-21T10:06:12.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:11 smithi070 ceph-mon[112139]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:06:12.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:11 smithi070 ceph-mon[112139]: osdmap e636: 8 total, 8 up, 8 in 2024-09-21T10:06:13.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:12 smithi160 ceph-mon[101879]: osdmap e637: 8 total, 8 up, 8 in 2024-09-21T10:06:13.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:12 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-25", "field": "max_bytes", "val": "10485760"} : dispatch 2024-09-21T10:06:13.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:12 smithi160 ceph-mon[101879]: pgmap v457: 425 pgs: 425 active+clean; 584 KiB data, 539 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 1023 B/s wr, 1 op/s 2024-09-21T10:06:13.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:12 smithi070 ceph-mon[112139]: osdmap e637: 8 total, 8 up, 8 in 2024-09-21T10:06:13.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:12 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-25", "field": "max_bytes", "val": "10485760"} : dispatch 2024-09-21T10:06:13.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:12 smithi070 ceph-mon[112139]: pgmap v457: 425 pgs: 425 active+clean; 584 KiB data, 539 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 1023 B/s wr, 1 op/s 2024-09-21T10:06:13.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:12 smithi070 ceph-mon[107385]: osdmap e637: 8 total, 8 up, 8 in 2024-09-21T10:06:13.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:12 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-25", "field": "max_bytes", "val": "10485760"} : dispatch 2024-09-21T10:06:13.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:12 smithi070 ceph-mon[107385]: pgmap v457: 425 pgs: 425 active+clean; 584 KiB data, 539 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 1023 B/s wr, 1 op/s 2024-09-21T10:06:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:06:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:06:13] "GET /metrics HTTP/1.1" 200 57952 "" "Prometheus/2.51.0" 2024-09-21T10:06:14.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:13 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-25", "field": "max_bytes", "val": "10485760"}]': finished 2024-09-21T10:06:14.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:13 smithi160 ceph-mon[101879]: osdmap e638: 8 total, 8 up, 8 in 2024-09-21T10:06:14.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:13 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-25", "field": "max_bytes", "val": "10485760"}]': finished 2024-09-21T10:06:14.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:13 smithi070 ceph-mon[112139]: osdmap e638: 8 total, 8 up, 8 in 2024-09-21T10:06:14.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:13 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-25", "field": "max_bytes", "val": "10485760"}]': finished 2024-09-21T10:06:14.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:13 smithi070 ceph-mon[107385]: osdmap e638: 8 total, 8 up, 8 in 2024-09-21T10:06:15.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:14 smithi160 ceph-mon[101879]: pgmap v459: 425 pgs: 425 active+clean; 584 KiB data, 539 MiB used, 715 GiB / 715 GiB avail; 237 B/s rd, 951 B/s wr, 1 op/s 2024-09-21T10:06:15.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:14 smithi070 ceph-mon[112139]: pgmap v459: 425 pgs: 425 active+clean; 584 KiB data, 539 MiB used, 715 GiB / 715 GiB avail; 237 B/s rd, 951 B/s wr, 1 op/s 2024-09-21T10:06:15.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:14 smithi070 ceph-mon[107385]: pgmap v459: 425 pgs: 425 active+clean; 584 KiB data, 539 MiB used, 715 GiB / 715 GiB avail; 237 B/s rd, 951 B/s wr, 1 op/s 2024-09-21T10:06:17.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:17 smithi160 ceph-mon[101879]: pgmap v460: 425 pgs: 425 active+clean; 4.6 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 684 KiB/s wr, 10 op/s 2024-09-21T10:06:17.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:17 smithi070 ceph-mon[107385]: pgmap v460: 425 pgs: 425 active+clean; 4.6 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 684 KiB/s wr, 10 op/s 2024-09-21T10:06:17.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:17 smithi070 ceph-mon[112139]: pgmap v460: 425 pgs: 425 active+clean; 4.6 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 684 KiB/s wr, 10 op/s 2024-09-21T10:06:19.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:18 smithi160 ceph-mon[101879]: osdmap e639: 8 total, 8 up, 8 in 2024-09-21T10:06:19.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:18 smithi160 ceph-mon[101879]: pgmap v462: 425 pgs: 425 active+clean; 4.6 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 655 KiB/s wr, 9 op/s 2024-09-21T10:06:19.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:18 smithi070 ceph-mon[107385]: osdmap e639: 8 total, 8 up, 8 in 2024-09-21T10:06:19.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:18 smithi070 ceph-mon[107385]: pgmap v462: 425 pgs: 425 active+clean; 4.6 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 655 KiB/s wr, 9 op/s 2024-09-21T10:06:19.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:18 smithi070 ceph-mon[112139]: osdmap e639: 8 total, 8 up, 8 in 2024-09-21T10:06:19.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:18 smithi070 ceph-mon[112139]: pgmap v462: 425 pgs: 425 active+clean; 4.6 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 655 KiB/s wr, 9 op/s 2024-09-21T10:06:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:21 smithi160 ceph-mon[101879]: pgmap v463: 425 pgs: 425 active+clean; 4.6 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 513 KiB/s wr, 7 op/s 2024-09-21T10:06:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:21.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:21 smithi070 ceph-mon[107385]: pgmap v463: 425 pgs: 425 active+clean; 4.6 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 513 KiB/s wr, 7 op/s 2024-09-21T10:06:21.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:21.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:21.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:21 smithi070 ceph-mon[112139]: pgmap v463: 425 pgs: 425 active+clean; 4.6 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 513 KiB/s wr, 7 op/s 2024-09-21T10:06:21.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:21.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:23.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:23 smithi160 ceph-mon[101879]: pgmap v464: 425 pgs: 425 active+clean; 9.6 MiB data, 563 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 996 KiB/s wr, 8 op/s 2024-09-21T10:06:23.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:23 smithi070 ceph-mon[107385]: pgmap v464: 425 pgs: 425 active+clean; 9.6 MiB data, 563 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 996 KiB/s wr, 8 op/s 2024-09-21T10:06:23.378 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:06:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:06:23] "GET /metrics HTTP/1.1" 200 60396 "" "Prometheus/2.51.0" 2024-09-21T10:06:23.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:23 smithi070 ceph-mon[112139]: pgmap v464: 425 pgs: 425 active+clean; 9.6 MiB data, 563 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 996 KiB/s wr, 8 op/s 2024-09-21T10:06:25.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:25 smithi160 ceph-mon[101879]: pgmap v465: 425 pgs: 425 active+clean; 9.6 MiB data, 563 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 922 KiB/s wr, 7 op/s 2024-09-21T10:06:25.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:25 smithi070 ceph-mon[107385]: pgmap v465: 425 pgs: 425 active+clean; 9.6 MiB data, 563 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 922 KiB/s wr, 7 op/s 2024-09-21T10:06:25.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:25 smithi070 ceph-mon[112139]: pgmap v465: 425 pgs: 425 active+clean; 9.6 MiB data, 563 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 922 KiB/s wr, 7 op/s 2024-09-21T10:06:27.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:27 smithi160 ceph-mon[101879]: pgmap v466: 425 pgs: 425 active+clean; 15 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.0 MiB/s wr, 3 op/s 2024-09-21T10:06:27.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:27 smithi070 ceph-mon[107385]: pgmap v466: 425 pgs: 425 active+clean; 15 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.0 MiB/s wr, 3 op/s 2024-09-21T10:06:27.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:27 smithi070 ceph-mon[112139]: pgmap v466: 425 pgs: 425 active+clean; 15 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.0 MiB/s wr, 3 op/s 2024-09-21T10:06:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:28 smithi160 ceph-mon[101879]: pool 'test-librbd-smithi070-86016-25' is full (reached quota's max_bytes: 10 MiB) 2024-09-21T10:06:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:28 smithi160 ceph-mon[101879]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-21T10:06:28.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:28 smithi160 ceph-mon[101879]: osdmap e640: 8 total, 8 up, 8 in 2024-09-21T10:06:28.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:28 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-24", "field": "max_objects", "val": "6"} : dispatch 2024-09-21T10:06:28.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:28 smithi070 ceph-mon[107385]: pool 'test-librbd-smithi070-86016-25' is full (reached quota's max_bytes: 10 MiB) 2024-09-21T10:06:28.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:28 smithi070 ceph-mon[107385]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-21T10:06:28.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:28 smithi070 ceph-mon[107385]: osdmap e640: 8 total, 8 up, 8 in 2024-09-21T10:06:28.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:28 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-24", "field": "max_objects", "val": "6"} : dispatch 2024-09-21T10:06:28.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:28 smithi070 ceph-mon[112139]: pool 'test-librbd-smithi070-86016-25' is full (reached quota's max_bytes: 10 MiB) 2024-09-21T10:06:28.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:28 smithi070 ceph-mon[112139]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-21T10:06:28.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:28 smithi070 ceph-mon[112139]: osdmap e640: 8 total, 8 up, 8 in 2024-09-21T10:06:28.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:28 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-24", "field": "max_objects", "val": "6"} : dispatch 2024-09-21T10:06:28.698 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:06:29.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:29 smithi160 ceph-mon[101879]: pgmap v468: 425 pgs: 425 active+clean; 15 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.0 MiB/s wr, 3 op/s 2024-09-21T10:06:29.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:29 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-24", "field": "max_objects", "val": "6"}]': finished 2024-09-21T10:06:29.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:29 smithi160 ceph-mon[101879]: osdmap e641: 8 total, 8 up, 8 in 2024-09-21T10:06:29.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:29 smithi070 ceph-mon[112139]: pgmap v468: 425 pgs: 425 active+clean; 15 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.0 MiB/s wr, 3 op/s 2024-09-21T10:06:29.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:29 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-24", "field": "max_objects", "val": "6"}]': finished 2024-09-21T10:06:29.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:29 smithi070 ceph-mon[112139]: osdmap e641: 8 total, 8 up, 8 in 2024-09-21T10:06:29.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:29 smithi070 ceph-mon[107385]: pgmap v468: 425 pgs: 425 active+clean; 15 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.0 MiB/s wr, 3 op/s 2024-09-21T10:06:29.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:29 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-24", "field": "max_objects", "val": "6"}]': finished 2024-09-21T10:06:29.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:29 smithi070 ceph-mon[107385]: osdmap e641: 8 total, 8 up, 8 in 2024-09-21T10:06:29.721 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:06:30.732 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:06:31.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:31 smithi160 ceph-mon[101879]: pgmap v470: 425 pgs: 425 active+clean; 15 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 640 KiB/s wr, 1 op/s 2024-09-21T10:06:31.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:31 smithi070 ceph-mon[112139]: pgmap v470: 425 pgs: 425 active+clean; 15 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 640 KiB/s wr, 1 op/s 2024-09-21T10:06:31.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:31 smithi070 ceph-mon[107385]: pgmap v470: 425 pgs: 425 active+clean; 15 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 640 KiB/s wr, 1 op/s 2024-09-21T10:06:31.743 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:06:32.753 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:06:32.819 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.RemoveFullTryDataPool (29961 ms) 2024-09-21T10:06:32.819 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.RemoveFullTryNamespace 2024-09-21T10:06:33.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:33 smithi160 ceph-mon[101879]: pgmap v471: 425 pgs: 425 active+clean; 16 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 769 KiB/s wr, 6 op/s 2024-09-21T10:06:33.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:33 smithi160 ceph-mon[101879]: pool 'test-librbd-smithi070-86016-24' is full (reached quota's max_objects: 6) 2024-09-21T10:06:33.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:33 smithi160 ceph-mon[101879]: Health check update: 2 pool(s) full (POOL_FULL) 2024-09-21T10:06:33.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:33 smithi160 ceph-mon[101879]: osdmap e642: 8 total, 8 up, 8 in 2024-09-21T10:06:33.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:06:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:06:33] "GET /metrics HTTP/1.1" 200 60396 "" "Prometheus/2.51.0" 2024-09-21T10:06:33.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:33 smithi070 ceph-mon[107385]: pgmap v471: 425 pgs: 425 active+clean; 16 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 769 KiB/s wr, 6 op/s 2024-09-21T10:06:33.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:33 smithi070 ceph-mon[107385]: pool 'test-librbd-smithi070-86016-24' is full (reached quota's max_objects: 6) 2024-09-21T10:06:33.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:33 smithi070 ceph-mon[107385]: Health check update: 2 pool(s) full (POOL_FULL) 2024-09-21T10:06:33.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:33 smithi070 ceph-mon[107385]: osdmap e642: 8 total, 8 up, 8 in 2024-09-21T10:06:33.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:33 smithi070 ceph-mon[112139]: pgmap v471: 425 pgs: 425 active+clean; 16 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 769 KiB/s wr, 6 op/s 2024-09-21T10:06:33.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:33 smithi070 ceph-mon[112139]: pool 'test-librbd-smithi070-86016-24' is full (reached quota's max_objects: 6) 2024-09-21T10:06:33.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:33 smithi070 ceph-mon[112139]: Health check update: 2 pool(s) full (POOL_FULL) 2024-09-21T10:06:33.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:33 smithi070 ceph-mon[112139]: osdmap e642: 8 total, 8 up, 8 in 2024-09-21T10:06:34.749 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:06:35.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:34 smithi160 ceph-mon[101879]: osdmap e643: 8 total, 8 up, 8 in 2024-09-21T10:06:35.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:34 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4252773918' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-26","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:35.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:34 smithi160 ceph-mon[101879]: pgmap v474: 457 pgs: 32 unknown, 425 active+clean; 16 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 173 KiB/s wr, 6 op/s 2024-09-21T10:06:35.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:34 smithi070 ceph-mon[107385]: osdmap e643: 8 total, 8 up, 8 in 2024-09-21T10:06:35.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:34 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4252773918' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-26","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:35.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:34 smithi070 ceph-mon[107385]: pgmap v474: 457 pgs: 32 unknown, 425 active+clean; 16 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 173 KiB/s wr, 6 op/s 2024-09-21T10:06:35.128 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:06:34 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:06:34] ENGINE Bus STOPPING 2024-09-21T10:06:35.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:34 smithi070 ceph-mon[112139]: osdmap e643: 8 total, 8 up, 8 in 2024-09-21T10:06:35.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:34 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4252773918' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-26","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:35.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:34 smithi070 ceph-mon[112139]: pgmap v474: 457 pgs: 32 unknown, 425 active+clean; 16 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 173 KiB/s wr, 6 op/s 2024-09-21T10:06:35.493 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:06:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:06:35] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T10:06:35.493 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:06:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:06:35] ENGINE Bus STOPPED 2024-09-21T10:06:35.493 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:06:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:06:35] ENGINE Bus STARTING 2024-09-21T10:06:35.493 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:06:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:06:35] ENGINE Serving on http://:::9283 2024-09-21T10:06:35.493 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:06:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:06:35] ENGINE Bus STARTED 2024-09-21T10:06:35.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:35 smithi070 ceph-mon[112139]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:06:35.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:35 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4252773918' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:06:35.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:35 smithi070 ceph-mon[112139]: osdmap e644: 8 total, 8 up, 8 in 2024-09-21T10:06:35.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:35 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' 2024-09-21T10:06:35.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:06:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:06:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:35.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:35 smithi070 ceph-mon[107385]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:06:35.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:35 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4252773918' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:06:35.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:35 smithi070 ceph-mon[107385]: osdmap e644: 8 total, 8 up, 8 in 2024-09-21T10:06:35.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:35 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' 2024-09-21T10:06:35.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:06:35.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:06:35.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:35.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:35 smithi160 ceph-mon[101879]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:06:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:35 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4252773918' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:06:36.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:35 smithi160 ceph-mon[101879]: osdmap e644: 8 total, 8 up, 8 in 2024-09-21T10:06:36.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:35 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' 2024-09-21T10:06:36.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:06:36.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:06:36.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:36.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:37.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:36 smithi160 ceph-mon[101879]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:06:37.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:36 smithi160 ceph-mon[101879]: osdmap e645: 8 total, 8 up, 8 in 2024-09-21T10:06:37.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:36 smithi160 ceph-mon[101879]: pgmap v477: 457 pgs: 457 active+clean; 584 KiB data, 528 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 3.7 KiB/s wr, 45 op/s 2024-09-21T10:06:37.097 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:36 smithi070 ceph-mon[107385]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:06:37.098 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:36 smithi070 ceph-mon[107385]: osdmap e645: 8 total, 8 up, 8 in 2024-09-21T10:06:37.098 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:36 smithi070 ceph-mon[107385]: pgmap v477: 457 pgs: 457 active+clean; 584 KiB data, 528 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 3.7 KiB/s wr, 45 op/s 2024-09-21T10:06:37.098 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:36 smithi070 ceph-mon[112139]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:06:37.098 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:36 smithi070 ceph-mon[112139]: osdmap e645: 8 total, 8 up, 8 in 2024-09-21T10:06:37.098 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:36 smithi070 ceph-mon[112139]: pgmap v477: 457 pgs: 457 active+clean; 584 KiB data, 528 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 3.7 KiB/s wr, 45 op/s 2024-09-21T10:06:38.060 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:37 smithi070 ceph-mon[107385]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:06:38.060 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:37 smithi070 ceph-mon[107385]: osdmap e646: 8 total, 8 up, 8 in 2024-09-21T10:06:38.060 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:37 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-26", "field": "max_bytes", "val": "10485760"} : dispatch 2024-09-21T10:06:38.060 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:37 smithi070 ceph-mon[107385]: pool 'test-librbd-smithi070-86016-25' no longer out of quota; removing NO_QUOTA flag 2024-09-21T10:06:38.060 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:37 smithi070 ceph-mon[107385]: Health check update: 1 pool(s) full (POOL_FULL) 2024-09-21T10:06:38.060 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:37 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-26", "field": "max_bytes", "val": "10485760"}]': finished 2024-09-21T10:06:38.060 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:37 smithi070 ceph-mon[107385]: osdmap e647: 8 total, 8 up, 8 in 2024-09-21T10:06:38.061 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:37 smithi070 ceph-mon[112139]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:06:38.061 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:37 smithi070 ceph-mon[112139]: osdmap e646: 8 total, 8 up, 8 in 2024-09-21T10:06:38.061 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:37 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-26", "field": "max_bytes", "val": "10485760"} : dispatch 2024-09-21T10:06:38.061 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:37 smithi070 ceph-mon[112139]: pool 'test-librbd-smithi070-86016-25' no longer out of quota; removing NO_QUOTA flag 2024-09-21T10:06:38.061 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:37 smithi070 ceph-mon[112139]: Health check update: 1 pool(s) full (POOL_FULL) 2024-09-21T10:06:38.061 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:37 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-26", "field": "max_bytes", "val": "10485760"}]': finished 2024-09-21T10:06:38.061 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:37 smithi070 ceph-mon[112139]: osdmap e647: 8 total, 8 up, 8 in 2024-09-21T10:06:38.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:37 smithi160 ceph-mon[101879]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:06:38.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:37 smithi160 ceph-mon[101879]: osdmap e646: 8 total, 8 up, 8 in 2024-09-21T10:06:38.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:37 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-26", "field": "max_bytes", "val": "10485760"} : dispatch 2024-09-21T10:06:38.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:37 smithi160 ceph-mon[101879]: pool 'test-librbd-smithi070-86016-25' no longer out of quota; removing NO_QUOTA flag 2024-09-21T10:06:38.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:37 smithi160 ceph-mon[101879]: Health check update: 1 pool(s) full (POOL_FULL) 2024-09-21T10:06:38.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:37 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-26", "field": "max_bytes", "val": "10485760"}]': finished 2024-09-21T10:06:38.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:37 smithi160 ceph-mon[101879]: osdmap e647: 8 total, 8 up, 8 in 2024-09-21T10:06:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:38 smithi160 ceph-mon[101879]: pgmap v480: 457 pgs: 457 active+clean; 584 KiB data, 528 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 3.7 KiB/s wr, 45 op/s 2024-09-21T10:06:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:38 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:38 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:39.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:38 smithi070 ceph-mon[107385]: pgmap v480: 457 pgs: 457 active+clean; 584 KiB data, 528 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 3.7 KiB/s wr, 45 op/s 2024-09-21T10:06:39.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:38 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:39.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:38 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:39.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:38 smithi070 ceph-mon[112139]: pgmap v480: 457 pgs: 457 active+clean; 584 KiB data, 528 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 3.7 KiB/s wr, 45 op/s 2024-09-21T10:06:39.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:38 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:39.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:38 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:40.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:39 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:06:40.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:39 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:06:40.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:39 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:40.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:39 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:06:40.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:39 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:06:40.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:39 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:40.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:39 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:06:40.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:39 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:06:40.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:39 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:41.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:40 smithi160 ceph-mon[101879]: pgmap v481: 457 pgs: 457 active+clean; 584 KiB data, 528 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 2.8 KiB/s wr, 34 op/s 2024-09-21T10:06:41.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:40 smithi070 ceph-mon[107385]: pgmap v481: 457 pgs: 457 active+clean; 584 KiB data, 528 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 2.8 KiB/s wr, 34 op/s 2024-09-21T10:06:41.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:40 smithi070 ceph-mon[112139]: pgmap v481: 457 pgs: 457 active+clean; 584 KiB data, 528 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 2.8 KiB/s wr, 34 op/s 2024-09-21T10:06:43.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:43 smithi160 ceph-mon[101879]: pgmap v482: 457 pgs: 457 active+clean; 4.6 MiB data, 538 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 655 KiB/s wr, 38 op/s 2024-09-21T10:06:43.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:43 smithi160 ceph-mon[101879]: osdmap e648: 8 total, 8 up, 8 in 2024-09-21T10:06:43.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:43 smithi070 ceph-mon[107385]: pgmap v482: 457 pgs: 457 active+clean; 4.6 MiB data, 538 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 655 KiB/s wr, 38 op/s 2024-09-21T10:06:43.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:43 smithi070 ceph-mon[107385]: osdmap e648: 8 total, 8 up, 8 in 2024-09-21T10:06:43.378 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:06:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:06:43] "GET /metrics HTTP/1.1" 200 62768 "" "Prometheus/2.51.0" 2024-09-21T10:06:43.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:43 smithi070 ceph-mon[112139]: pgmap v482: 457 pgs: 457 active+clean; 4.6 MiB data, 538 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 655 KiB/s wr, 38 op/s 2024-09-21T10:06:43.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:43 smithi070 ceph-mon[112139]: osdmap e648: 8 total, 8 up, 8 in 2024-09-21T10:06:45.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:45 smithi160 ceph-mon[101879]: pgmap v484: 457 pgs: 457 active+clean; 4.6 MiB data, 538 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 563 KiB/s wr, 8 op/s 2024-09-21T10:06:45.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:45 smithi070 ceph-mon[107385]: pgmap v484: 457 pgs: 457 active+clean; 4.6 MiB data, 538 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 563 KiB/s wr, 8 op/s 2024-09-21T10:06:45.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:45 smithi070 ceph-mon[112139]: pgmap v484: 457 pgs: 457 active+clean; 4.6 MiB data, 538 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 563 KiB/s wr, 8 op/s 2024-09-21T10:06:47.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:47 smithi160 ceph-mon[101879]: pgmap v485: 457 pgs: 457 active+clean; 9.6 MiB data, 537 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-21T10:06:47.372 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:47 smithi070 ceph-mon[107385]: pgmap v485: 457 pgs: 457 active+clean; 9.6 MiB data, 537 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-21T10:06:47.372 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:47 smithi070 ceph-mon[112139]: pgmap v485: 457 pgs: 457 active+clean; 9.6 MiB data, 537 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-21T10:06:47.754 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'echo wait for half of osds to upgrade' 2024-09-21T10:06:47.958 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:49.228 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:49 smithi070 ceph-mon[112139]: pgmap v486: 457 pgs: 457 active+clean; 9.6 MiB data, 537 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 922 KiB/s wr, 7 op/s 2024-09-21T10:06:49.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:49 smithi070 ceph-mon[107385]: pgmap v486: 457 pgs: 457 active+clean; 9.6 MiB data, 537 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 922 KiB/s wr, 7 op/s 2024-09-21T10:06:49.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:49 smithi160 ceph-mon[101879]: pgmap v486: 457 pgs: 457 active+clean; 9.6 MiB data, 537 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 922 KiB/s wr, 7 op/s 2024-09-21T10:06:49.797 INFO:teuthology.orchestra.run.smithi070.stdout:wait for half of osds to upgrade 2024-09-21T10:06:50.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.7", "id": [6, 4]} : dispatch 2024-09-21T10:06:50.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "29.1a", "id": [6, 0]} : dispatch 2024-09-21T10:06:50.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.7", "id": [6, 4]} : dispatch 2024-09-21T10:06:50.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "29.1a", "id": [6, 0]} : dispatch 2024-09-21T10:06:50.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.7", "id": [6, 4]} : dispatch 2024-09-21T10:06:50.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "29.1a", "id": [6, 0]} : dispatch 2024-09-21T10:06:50.490 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 4' 2024-09-21T10:06:50.684 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:51.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:51 smithi070 ceph-mon[107385]: pgmap v487: 457 pgs: 457 active+clean; 9.6 MiB data, 537 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 922 KiB/s wr, 7 op/s 2024-09-21T10:06:51.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.7", "id": [6, 4]}]': finished 2024-09-21T10:06:51.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "29.1a", "id": [6, 0]}]': finished 2024-09-21T10:06:51.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:51 smithi070 ceph-mon[107385]: osdmap e649: 8 total, 8 up, 8 in 2024-09-21T10:06:51.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:51.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:51.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:51 smithi070 ceph-mon[112139]: pgmap v487: 457 pgs: 457 active+clean; 9.6 MiB data, 537 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 922 KiB/s wr, 7 op/s 2024-09-21T10:06:51.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.7", "id": [6, 4]}]': finished 2024-09-21T10:06:51.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "29.1a", "id": [6, 0]}]': finished 2024-09-21T10:06:51.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:51 smithi070 ceph-mon[112139]: osdmap e649: 8 total, 8 up, 8 in 2024-09-21T10:06:51.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:51.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:51 smithi160 ceph-mon[101879]: pgmap v487: 457 pgs: 457 active+clean; 9.6 MiB data, 537 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 922 KiB/s wr, 7 op/s 2024-09-21T10:06:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.7", "id": [6, 4]}]': finished 2024-09-21T10:06:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "29.1a", "id": [6, 0]}]': finished 2024-09-21T10:06:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:51 smithi160 ceph-mon[101879]: osdmap e649: 8 total, 8 up, 8 in 2024-09-21T10:06:51.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:51.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:52.343 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:52 smithi070 ceph-mon[107385]: osdmap e650: 8 total, 8 up, 8 in 2024-09-21T10:06:52.343 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:52 smithi070 ceph-mon[112139]: osdmap e650: 8 total, 8 up, 8 in 2024-09-21T10:06:52.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:52 smithi160 ceph-mon[101879]: osdmap e650: 8 total, 8 up, 8 in 2024-09-21T10:06:52.860 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.RemoveFullTryNamespace (20041 ms) 2024-09-21T10:06:52.860 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.RemoveFullTryNamespaceDataPool 2024-09-21T10:06:53.352 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:06:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:06:53] "GET /metrics HTTP/1.1" 200 62811 "" "Prometheus/2.51.0" 2024-09-21T10:06:53.352 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:53 smithi070 ceph-mon[107385]: pgmap v490: 457 pgs: 2 peering, 455 active+clean; 15 MiB data, 552 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.3 MiB/s wr, 4 op/s 2024-09-21T10:06:53.352 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:53 smithi070 ceph-mon[107385]: pool 'test-librbd-smithi070-86016-26' is full (reached quota's max_bytes: 10 MiB) 2024-09-21T10:06:53.352 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:53 smithi070 ceph-mon[107385]: Health check update: 2 pool(s) full (POOL_FULL) 2024-09-21T10:06:53.352 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:53 smithi070 ceph-mon[107385]: osdmap e651: 8 total, 8 up, 8 in 2024-09-21T10:06:53.353 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:53 smithi070 ceph-mon[112139]: pgmap v490: 457 pgs: 2 peering, 455 active+clean; 15 MiB data, 552 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.3 MiB/s wr, 4 op/s 2024-09-21T10:06:53.353 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:53 smithi070 ceph-mon[112139]: pool 'test-librbd-smithi070-86016-26' is full (reached quota's max_bytes: 10 MiB) 2024-09-21T10:06:53.353 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:53 smithi070 ceph-mon[112139]: Health check update: 2 pool(s) full (POOL_FULL) 2024-09-21T10:06:53.353 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:53 smithi070 ceph-mon[112139]: osdmap e651: 8 total, 8 up, 8 in 2024-09-21T10:06:53.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:53 smithi160 ceph-mon[101879]: pgmap v490: 457 pgs: 2 peering, 455 active+clean; 15 MiB data, 552 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.3 MiB/s wr, 4 op/s 2024-09-21T10:06:53.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:53 smithi160 ceph-mon[101879]: pool 'test-librbd-smithi070-86016-26' is full (reached quota's max_bytes: 10 MiB) 2024-09-21T10:06:53.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:53 smithi160 ceph-mon[101879]: Health check update: 2 pool(s) full (POOL_FULL) 2024-09-21T10:06:53.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:53 smithi160 ceph-mon[101879]: osdmap e651: 8 total, 8 up, 8 in 2024-09-21T10:06:54.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:54 smithi070 ceph-mon[107385]: from='client.34455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208", "daemon_types": "osd", "limit": 4, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:54.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:54 smithi070 ceph-mon[107385]: osdmap e652: 8 total, 8 up, 8 in 2024-09-21T10:06:54.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:54 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2288631877' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:54.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:54 smithi070 ceph-mon[112139]: from='client.34455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208", "daemon_types": "osd", "limit": 4, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:54.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:54 smithi070 ceph-mon[112139]: osdmap e652: 8 total, 8 up, 8 in 2024-09-21T10:06:54.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:54 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2288631877' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:54.550 INFO:teuthology.orchestra.run.smithi070.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:06:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:54 smithi160 ceph-mon[101879]: from='client.34455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208", "daemon_types": "osd", "limit": 4, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:54 smithi160 ceph-mon[101879]: osdmap e652: 8 total, 8 up, 8 in 2024-09-21T10:06:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:54 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2288631877' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:55.167 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:55 smithi070 ceph-mon[107385]: pgmap v493: 489 pgs: 32 unknown, 2 peering, 455 active+clean; 15 MiB data, 552 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.3 MiB/s wr, 3 op/s 2024-09-21T10:06:55.167 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:55 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:55.167 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:55 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:06:55.167 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:55 smithi070 ceph-mon[107385]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:06:55.167 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:55 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2288631877' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:06:55.167 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:55 smithi070 ceph-mon[107385]: osdmap e653: 8 total, 8 up, 8 in 2024-09-21T10:06:55.168 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:55 smithi070 ceph-mon[112139]: pgmap v493: 489 pgs: 32 unknown, 2 peering, 455 active+clean; 15 MiB data, 552 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.3 MiB/s wr, 3 op/s 2024-09-21T10:06:55.168 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:55 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:55.168 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:55 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:06:55.168 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:55 smithi070 ceph-mon[112139]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:06:55.168 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:55 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2288631877' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:06:55.168 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:55 smithi070 ceph-mon[112139]: osdmap e653: 8 total, 8 up, 8 in 2024-09-21T10:06:55.169 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-09-21T10:06:55.367 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:55.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:55 smithi160 ceph-mon[101879]: pgmap v493: 489 pgs: 32 unknown, 2 peering, 455 active+clean; 15 MiB data, 552 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.3 MiB/s wr, 3 op/s 2024-09-21T10:06:55.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:55 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:55.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:55 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:06:55.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:55 smithi160 ceph-mon[101879]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:06:55.530 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:55 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2288631877' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:06:55.530 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:55 smithi160 ceph-mon[101879]: osdmap e653: 8 total, 8 up, 8 in 2024-09-21T10:06:56.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:56 smithi160 ceph-mon[101879]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:06:56.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:56 smithi160 ceph-mon[101879]: osdmap e654: 8 total, 8 up, 8 in 2024-09-21T10:06:56.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:56 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/331854476' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-28","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:56.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:56 smithi070 ceph-mon[107385]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:06:56.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:56 smithi070 ceph-mon[107385]: osdmap e654: 8 total, 8 up, 8 in 2024-09-21T10:06:56.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:56 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/331854476' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-28","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:56.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:56 smithi070 ceph-mon[112139]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:06:56.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:56 smithi070 ceph-mon[112139]: osdmap e654: 8 total, 8 up, 8 in 2024-09-21T10:06:56.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:56 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/331854476' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-28","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:56.750 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:06:56.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:06:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:06:56] ENGINE Bus STOPPING 2024-09-21T10:06:57.168 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:57 smithi070 ceph-mon[107385]: pgmap v496: 521 pgs: 32 creating+peering, 489 active+clean; 584 KiB data, 514 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 258 KiB/s wr, 39 op/s; 4 B/s, 0 objects/s recovering 2024-09-21T10:06:57.168 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:57 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/331854476' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:06:57.168 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:57 smithi070 ceph-mon[107385]: osdmap e655: 8 total, 8 up, 8 in 2024-09-21T10:06:57.168 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:57 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' 2024-09-21T10:06:57.169 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:57 smithi070 ceph-mon[112139]: pgmap v496: 521 pgs: 32 creating+peering, 489 active+clean; 584 KiB data, 514 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 258 KiB/s wr, 39 op/s; 4 B/s, 0 objects/s recovering 2024-09-21T10:06:57.169 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:57 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/331854476' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:06:57.169 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:57 smithi070 ceph-mon[112139]: osdmap e655: 8 total, 8 up, 8 in 2024-09-21T10:06:57.169 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:57 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' 2024-09-21T10:06:57.329 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-09-21T10:06:57.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:57 smithi160 ceph-mon[101879]: pgmap v496: 521 pgs: 32 creating+peering, 489 active+clean; 584 KiB data, 514 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 258 KiB/s wr, 39 op/s; 4 B/s, 0 objects/s recovering 2024-09-21T10:06:57.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:57 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/331854476' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:06:57.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:57 smithi160 ceph-mon[101879]: osdmap e655: 8 total, 8 up, 8 in 2024-09-21T10:06:57.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:57 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' 2024-09-21T10:06:57.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:06:57 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:06:57] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T10:06:57.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:06:57 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:06:57] ENGINE Bus STOPPED 2024-09-21T10:06:57.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:06:57 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:06:57] ENGINE Bus STARTING 2024-09-21T10:06:57.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:06:57 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:06:57] ENGINE Serving on http://:::9283 2024-09-21T10:06:57.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:06:57 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:06:57] ENGINE Bus STARTED 2024-09-21T10:06:57.902 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T10:06:57.902 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (7m) 19s ago 13m 19.3M - 0.27.0 11f11916f8cd 89ae95182419 2024-09-21T10:06:57.902 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi160 *:3000 running (7m) 66s ago 12m 85.0M - 10.4.8 5aad1d7cf004 daa2d8dfab48 2024-09-21T10:06:57.902 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi070.gansda smithi070 running (11m) 19s ago 11m 23.1M - 18.2.4-871-g6a1bdf85 0adab276c97e 6016b2501c8a 2024-09-21T10:06:57.902 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi160.byyhfz smithi160 running (11m) 66s ago 11m 17.1M - 18.2.4-871-g6a1bdf85 0adab276c97e 93e108d91e65 2024-09-21T10:06:57.903 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi160 *:8443,9283,8765 running (8m) 66s ago 17m 457M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 d952652a9683 2024-09-21T10:06:57.903 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (8m) 19s ago 18m 527M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 e3d390893cff 2024-09-21T10:06:57.903 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (6m) 19s ago 18m 72.5M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 4a7b667a4033 2024-09-21T10:06:57.903 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi160 running (3m) 66s ago 17m 44.2M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 86e88af21f36 2024-09-21T10:06:57.903 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (5m) 19s ago 17m 54.1M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 e154380b80a2 2024-09-21T10:06:57.903 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (8m) 19s ago 13m 10.7M - 1.7.0 72c9c2088986 5d5d4b026070 2024-09-21T10:06:57.903 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi160 *:9100 running (8m) 66s ago 13m 10.9M - 1.7.0 72c9c2088986 19b8e0e8ee0c 2024-09-21T10:06:57.903 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (16m) 19s ago 16m 120M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 10580e80f007 2024-09-21T10:06:57.903 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (16m) 19s ago 16m 125M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e b7c0d1169d1d 2024-09-21T10:06:57.903 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (15m) 19s ago 15m 143M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 58a11b8ddb81 2024-09-21T10:06:57.903 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (15m) 19s ago 15m 134M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e aa0c8a732a41 2024-09-21T10:06:57.903 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi160 running (15m) 66s ago 15m 127M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 7325a075c488 2024-09-21T10:06:57.903 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi160 running (14m) 66s ago 14m 116M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e f74026345641 2024-09-21T10:06:57.903 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi160 running (14m) 66s ago 14m 144M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 03e6bb54f8b2 2024-09-21T10:06:57.903 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi160 running (14m) 66s ago 14m 123M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e e405b21a6146 2024-09-21T10:06:57.904 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi160 *:9095 running (7m) 66s ago 13m 56.0M - 2.51.0 1d3b7f56885b 648952e57ad7 2024-09-21T10:06:58.240 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:06:58.240 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-09-21T10:06:58.240 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 3 2024-09-21T10:06:58.240 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:06:58.240 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-09-21T10:06:58.240 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T10:06:58.240 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:06:58.240 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-09-21T10:06:58.240 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 8 2024-09-21T10:06:58.240 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:06:58.241 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": { 2024-09-21T10:06:58.241 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 2 2024-09-21T10:06:58.241 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:06:58.241 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-09-21T10:06:58.241 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 10, 2024-09-21T10:06:58.241 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 5 2024-09-21T10:06:58.241 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:06:58.241 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:06:58.241 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:58 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:06:58.241 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:58 smithi070 ceph-mon[107385]: pool 'test-librbd-smithi070-86016-26' no longer out of quota; removing NO_QUOTA flag 2024-09-21T10:06:58.241 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:58 smithi070 ceph-mon[107385]: Health check update: 1 pool(s) full (POOL_FULL) 2024-09-21T10:06:58.241 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:58 smithi070 ceph-mon[107385]: osdmap e656: 8 total, 8 up, 8 in 2024-09-21T10:06:58.242 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:58 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:06:58.242 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:58 smithi070 ceph-mon[112139]: pool 'test-librbd-smithi070-86016-26' no longer out of quota; removing NO_QUOTA flag 2024-09-21T10:06:58.242 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:58 smithi070 ceph-mon[112139]: Health check update: 1 pool(s) full (POOL_FULL) 2024-09-21T10:06:58.242 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:58 smithi070 ceph-mon[112139]: osdmap e656: 8 total, 8 up, 8 in 2024-09-21T10:06:58.382 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:58 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:06:58.382 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:58 smithi160 ceph-mon[101879]: pool 'test-librbd-smithi070-86016-26' no longer out of quota; removing NO_QUOTA flag 2024-09-21T10:06:58.382 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:58 smithi160 ceph-mon[101879]: Health check update: 1 pool(s) full (POOL_FULL) 2024-09-21T10:06:58.382 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:58 smithi160 ceph-mon[101879]: osdmap e656: 8 total, 8 up, 8 in 2024-09-21T10:06:58.530 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:06:58.531 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208", 2024-09-21T10:06:58.531 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-09-21T10:06:58.531 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 4 daemons (4 remaining).", 2024-09-21T10:06:58.531 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-09-21T10:06:58.531 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "", 2024-09-21T10:06:58.531 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 image", 2024-09-21T10:06:58.531 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-09-21T10:06:58.531 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:06:59.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:59 smithi070 ceph-mon[107385]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:06:59.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:59 smithi070 ceph-mon[107385]: from='client.34473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:59.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:59 smithi070 ceph-mon[107385]: from='client.34479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:59.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:59 smithi070 ceph-mon[107385]: from='client.44414 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:59.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:59 smithi070 ceph-mon[107385]: pgmap v499: 521 pgs: 32 creating+peering, 489 active+clean; 584 KiB data, 514 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 258 KiB/s wr, 39 op/s; 4 B/s, 0 objects/s recovering 2024-09-21T10:06:59.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:59 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/305285146' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:06:59.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:59 smithi070 ceph-mon[107385]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:06:59.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:59 smithi070 ceph-mon[107385]: osdmap e657: 8 total, 8 up, 8 in 2024-09-21T10:06:59.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:59 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-28", "field": "max_bytes", "val": "10485760"} : dispatch 2024-09-21T10:06:59.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:59.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:06:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:59.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:59 smithi070 ceph-mon[112139]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:06:59.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:59 smithi070 ceph-mon[112139]: from='client.34473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:59.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:59 smithi070 ceph-mon[112139]: from='client.34479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:59.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:59 smithi070 ceph-mon[112139]: from='client.44414 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:59.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:59 smithi070 ceph-mon[112139]: pgmap v499: 521 pgs: 32 creating+peering, 489 active+clean; 584 KiB data, 514 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 258 KiB/s wr, 39 op/s; 4 B/s, 0 objects/s recovering 2024-09-21T10:06:59.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:59 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/305285146' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:06:59.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:59 smithi070 ceph-mon[112139]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:06:59.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:59 smithi070 ceph-mon[112139]: osdmap e657: 8 total, 8 up, 8 in 2024-09-21T10:06:59.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:59 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-28", "field": "max_bytes", "val": "10485760"} : dispatch 2024-09-21T10:06:59.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:59 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:59.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:06:59 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:59.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:59 smithi160 ceph-mon[101879]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:06:59.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:59 smithi160 ceph-mon[101879]: from='client.34473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:59.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:59 smithi160 ceph-mon[101879]: from='client.34479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:59.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:59 smithi160 ceph-mon[101879]: from='client.44414 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:59.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:59 smithi160 ceph-mon[101879]: pgmap v499: 521 pgs: 32 creating+peering, 489 active+clean; 584 KiB data, 514 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 258 KiB/s wr, 39 op/s; 4 B/s, 0 objects/s recovering 2024-09-21T10:06:59.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:59 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/305285146' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:06:59.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:59 smithi160 ceph-mon[101879]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:06:59.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:59 smithi160 ceph-mon[101879]: osdmap e657: 8 total, 8 up, 8 in 2024-09-21T10:06:59.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:59 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-28", "field": "max_bytes", "val": "10485760"} : dispatch 2024-09-21T10:06:59.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:59 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:06:59.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:06:59 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:00.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:00 smithi070 ceph-mon[107385]: from='client.44426 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:00.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:00 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:07:00.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:00 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:07:00.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:00 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:00.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:00 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-28", "field": "max_bytes", "val": "10485760"}]': finished 2024-09-21T10:07:00.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:00 smithi070 ceph-mon[107385]: osdmap e658: 8 total, 8 up, 8 in 2024-09-21T10:07:00.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:00 smithi070 ceph-mon[112139]: from='client.44426 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:00.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:00 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:07:00.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:00 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:07:00.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:00 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:00.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:00 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-28", "field": "max_bytes", "val": "10485760"}]': finished 2024-09-21T10:07:00.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:00 smithi070 ceph-mon[112139]: osdmap e658: 8 total, 8 up, 8 in 2024-09-21T10:07:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:00 smithi160 ceph-mon[101879]: from='client.44426 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:00 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:07:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:00 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:07:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:00 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:00 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-28", "field": "max_bytes", "val": "10485760"}]': finished 2024-09-21T10:07:00.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:00 smithi160 ceph-mon[101879]: osdmap e658: 8 total, 8 up, 8 in 2024-09-21T10:07:01.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:01 smithi070 ceph-mon[107385]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:07:01.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:01 smithi070 ceph-mon[107385]: pgmap v502: 521 pgs: 32 creating+peering, 489 active+clean; 584 KiB data, 514 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:07:01.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:01 smithi070 ceph-mon[112139]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:07:01.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:01 smithi070 ceph-mon[112139]: pgmap v502: 521 pgs: 32 creating+peering, 489 active+clean; 584 KiB data, 514 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:07:01.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:01 smithi160 ceph-mon[101879]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T10:07:01.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:01 smithi160 ceph-mon[101879]: pgmap v502: 521 pgs: 32 creating+peering, 489 active+clean; 584 KiB data, 514 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:07:03.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:07:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:07:03] "GET /metrics HTTP/1.1" 200 62811 "" "Prometheus/2.51.0" 2024-09-21T10:07:03.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:03 smithi070 ceph-mon[107385]: pgmap v503: 521 pgs: 521 active+clean; 2.6 MiB data, 521 MiB used, 715 GiB / 715 GiB avail; 6.6 KiB/s rd, 389 KiB/s wr, 12 op/s 2024-09-21T10:07:03.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:03 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:03.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:03 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:03.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:03 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:03.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:03 smithi070 ceph-mon[107385]: osdmap e659: 8 total, 8 up, 8 in 2024-09-21T10:07:03.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:03 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:03.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:03 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:03.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:03 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:03.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:03 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:03.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:03 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:03.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:03 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-21T10:07:03.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:03 smithi070 ceph-mon[112139]: pgmap v503: 521 pgs: 521 active+clean; 2.6 MiB data, 521 MiB used, 715 GiB / 715 GiB avail; 6.6 KiB/s rd, 389 KiB/s wr, 12 op/s 2024-09-21T10:07:03.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:03 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:03.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:03 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:03.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:03 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:03.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:03 smithi070 ceph-mon[112139]: osdmap e659: 8 total, 8 up, 8 in 2024-09-21T10:07:03.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:03 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:03.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:03 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:03.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:03 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:03.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:03 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:03.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:03 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:03.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:03 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-21T10:07:03.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:03 smithi160 ceph-mon[101879]: pgmap v503: 521 pgs: 521 active+clean; 2.6 MiB data, 521 MiB used, 715 GiB / 715 GiB avail; 6.6 KiB/s rd, 389 KiB/s wr, 12 op/s 2024-09-21T10:07:03.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:03 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:03.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:03 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:03.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:03 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:03.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:03 smithi160 ceph-mon[101879]: osdmap e659: 8 total, 8 up, 8 in 2024-09-21T10:07:03.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:03 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:03.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:03 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:03.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:03 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:03.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:03 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:03.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:03 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:03.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:03 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-21T10:07:04.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:04 smithi070 ceph-mon[107385]: Upgrade: Target is version 19.3.0-5092-g62da54b4 (squid) 2024-09-21T10:07:04.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:04 smithi070 ceph-mon[107385]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, digests ['quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47'] 2024-09-21T10:07:04.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:04 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all mgr 2024-09-21T10:07:04.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:04 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all mon 2024-09-21T10:07:04.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:04 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all crash 2024-09-21T10:07:04.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:04 smithi070 ceph-mon[107385]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-21T10:07:04.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:04 smithi070 ceph-mon[107385]: Upgrade: osd.0 is safe to restart 2024-09-21T10:07:04.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:04 smithi070 ceph-mon[112139]: Upgrade: Target is version 19.3.0-5092-g62da54b4 (squid) 2024-09-21T10:07:04.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:04 smithi070 ceph-mon[112139]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, digests ['quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47'] 2024-09-21T10:07:04.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:04 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all mgr 2024-09-21T10:07:04.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:04 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all mon 2024-09-21T10:07:04.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:04 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all crash 2024-09-21T10:07:04.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:04 smithi070 ceph-mon[112139]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-21T10:07:04.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:04 smithi070 ceph-mon[112139]: Upgrade: osd.0 is safe to restart 2024-09-21T10:07:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:04 smithi160 ceph-mon[101879]: Upgrade: Target is version 19.3.0-5092-g62da54b4 (squid) 2024-09-21T10:07:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:04 smithi160 ceph-mon[101879]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, digests ['quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47'] 2024-09-21T10:07:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:04 smithi160 ceph-mon[101879]: Upgrade: Setting container_image for all mgr 2024-09-21T10:07:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:04 smithi160 ceph-mon[101879]: Upgrade: Setting container_image for all mon 2024-09-21T10:07:04.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:04 smithi160 ceph-mon[101879]: Upgrade: Setting container_image for all crash 2024-09-21T10:07:04.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:04 smithi160 ceph-mon[101879]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-21T10:07:04.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:04 smithi160 ceph-mon[101879]: Upgrade: osd.0 is safe to restart 2024-09-21T10:07:05.419 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:05 smithi070 ceph-mon[107385]: pgmap v505: 521 pgs: 521 active+clean; 2.6 MiB data, 521 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 343 KiB/s wr, 10 op/s 2024-09-21T10:07:05.419 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:05 smithi070 ceph-mon[107385]: Upgrade: Updating osd.0 2024-09-21T10:07:05.419 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:05.419 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T10:07:05.419 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:07:05.419 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:05 smithi070 ceph-mon[107385]: Deploying daemon osd.0 on smithi070 2024-09-21T10:07:05.419 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:05 smithi070 ceph-mon[112139]: pgmap v505: 521 pgs: 521 active+clean; 2.6 MiB data, 521 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 343 KiB/s wr, 10 op/s 2024-09-21T10:07:05.419 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:05 smithi070 ceph-mon[112139]: Upgrade: Updating osd.0 2024-09-21T10:07:05.419 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:05.420 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T10:07:05.420 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:07:05.420 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:05 smithi070 ceph-mon[112139]: Deploying daemon osd.0 on smithi070 2024-09-21T10:07:05.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:05 smithi160 ceph-mon[101879]: pgmap v505: 521 pgs: 521 active+clean; 2.6 MiB data, 521 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 343 KiB/s wr, 10 op/s 2024-09-21T10:07:05.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:05 smithi160 ceph-mon[101879]: Upgrade: Updating osd.0 2024-09-21T10:07:05.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:05.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T10:07:05.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:07:05.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:05 smithi160 ceph-mon[101879]: Deploying daemon osd.0 on smithi070 2024-09-21T10:07:06.305 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:06.306 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:07.351 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:07 smithi070 ceph-mon[107385]: pgmap v506: 521 pgs: 521 active+clean; 7.6 MiB data, 537 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 982 KiB/s wr, 10 op/s 2024-09-21T10:07:07.351 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:07 smithi070 ceph-mon[112139]: pgmap v506: 521 pgs: 521 active+clean; 7.6 MiB data, 537 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 982 KiB/s wr, 10 op/s 2024-09-21T10:07:07.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:07 smithi160 ceph-mon[101879]: pgmap v506: 521 pgs: 521 active+clean; 7.6 MiB data, 537 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 982 KiB/s wr, 10 op/s 2024-09-21T10:07:07.624 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:07 smithi070 systemd[1]: Stopping Ceph osd.0 for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:07:07.877 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:07 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0[45840]: 2024-09-21T10:07:07.621+0000 7fcebb584640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:07:07.877 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:07 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0[45840]: 2024-09-21T10:07:07.621+0000 7fcebb584640 -1 osd.0 659 *** Got signal Terminated *** 2024-09-21T10:07:07.877 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:07 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0[45840]: 2024-09-21T10:07:07.621+0000 7fcebb584640 -1 osd.0 659 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:07:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:08 smithi160 ceph-mon[101879]: osd.0 marked itself down and dead 2024-09-21T10:07:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:08 smithi070 ceph-mon[107385]: osd.0 marked itself down and dead 2024-09-21T10:07:08.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:08 smithi070 ceph-mon[112139]: osd.0 marked itself down and dead 2024-09-21T10:07:08.981 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:08 smithi070 podman[124461]: 2024-09-21 10:07:08.726361447 +0000 UTC m=+1.233683156 container died 10580e80f007091b6150cd0d27021fe7913d83dba8363d63c078122bd3970b7c (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=reef-6a1bdf8, CEPH_POINT_RELEASE=) 2024-09-21T10:07:09.263 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:09 smithi070 podman[124461]: 2024-09-21 10:07:09.060623736 +0000 UTC m=+1.567945439 container cleanup 10580e80f007091b6150cd0d27021fe7913d83dba8363d63c078122bd3970b7c (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=reef-6a1bdf8, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:07:09.263 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:09 smithi070 bash[124461]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0 2024-09-21T10:07:09.264 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:09 smithi070 ceph-mon[107385]: pgmap v507: 521 pgs: 521 active+clean; 7.6 MiB data, 537 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 864 KiB/s wr, 9 op/s 2024-09-21T10:07:09.264 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:09 smithi070 ceph-mon[107385]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:07:09.264 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:09 smithi070 ceph-mon[107385]: osdmap e660: 8 total, 7 up, 8 in 2024-09-21T10:07:09.264 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:09 smithi070 ceph-mon[107385]: osd.0 marked itself dead as of e660 2024-09-21T10:07:09.264 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:09 smithi070 ceph-mon[112139]: pgmap v507: 521 pgs: 521 active+clean; 7.6 MiB data, 537 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 864 KiB/s wr, 9 op/s 2024-09-21T10:07:09.264 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:09 smithi070 ceph-mon[112139]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:07:09.264 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:09 smithi070 ceph-mon[112139]: osdmap e660: 8 total, 7 up, 8 in 2024-09-21T10:07:09.264 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:09 smithi070 ceph-mon[112139]: osd.0 marked itself dead as of e660 2024-09-21T10:07:09.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:09 smithi160 ceph-mon[101879]: pgmap v507: 521 pgs: 521 active+clean; 7.6 MiB data, 537 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 864 KiB/s wr, 9 op/s 2024-09-21T10:07:09.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:09 smithi160 ceph-mon[101879]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:07:09.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:09 smithi160 ceph-mon[101879]: osdmap e660: 8 total, 7 up, 8 in 2024-09-21T10:07:09.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:09 smithi160 ceph-mon[101879]: osd.0 marked itself dead as of e660 2024-09-21T10:07:09.627 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:09 smithi070 podman[124482]: 2024-09-21 10:07:09.261963753 +0000 UTC m=+0.534505144 container remove 10580e80f007091b6150cd0d27021fe7913d83dba8363d63c078122bd3970b7c (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=reef-6a1bdf8, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-21T10:07:09.933 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:09 smithi070 podman[124574]: 2024-09-21 10:07:09.565843614 +0000 UTC m=+0.015743835 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:07:09.933 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:09 smithi070 podman[124574]: 2024-09-21 10:07:09.671150955 +0000 UTC m=+0.121051212 container create 858352879af011e727ad96b2492392239b64b9662605ce08025f0bae7c0e0083 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-21T10:07:09.933 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:09 smithi070 podman[124574]: 2024-09-21 10:07:09.931771773 +0000 UTC m=+0.381671997 container init 858352879af011e727ad96b2492392239b64b9662605ce08025f0bae7c0e0083 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-21T10:07:10.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:10 smithi070 ceph-mon[107385]: osdmap e661: 8 total, 7 up, 8 in 2024-09-21T10:07:10.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:10 smithi070 ceph-mon[107385]: pgmap v510: 521 pgs: 62 stale+active+clean, 459 active+clean; 7.6 MiB data, 537 MiB used, 715 GiB / 715 GiB avail; 699 B/s rd, 700 KiB/s wr, 1 op/s 2024-09-21T10:07:10.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:10 smithi070 ceph-mon[112139]: osdmap e661: 8 total, 7 up, 8 in 2024-09-21T10:07:10.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:10 smithi070 ceph-mon[112139]: pgmap v510: 521 pgs: 62 stale+active+clean, 459 active+clean; 7.6 MiB data, 537 MiB used, 715 GiB / 715 GiB avail; 699 B/s rd, 700 KiB/s wr, 1 op/s 2024-09-21T10:07:10.378 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:09 smithi070 podman[124574]: 2024-09-21 10:07:09.936433069 +0000 UTC m=+0.386333284 container start 858352879af011e727ad96b2492392239b64b9662605ce08025f0bae7c0e0083 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, GIT_CLEAN=True, RELEASE=main-62da54b, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-21T10:07:10.378 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:10 smithi070 podman[124574]: 2024-09-21 10:07:10.023318539 +0000 UTC m=+0.473218773 container attach 858352879af011e727ad96b2492392239b64b9662605ce08025f0bae7c0e0083 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731) 2024-09-21T10:07:10.378 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:10 smithi070 podman[124574]: 2024-09-21 10:07:10.123069772 +0000 UTC m=+0.572969996 container died 858352879af011e727ad96b2492392239b64b9662605ce08025f0bae7c0e0083 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-21T10:07:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:10 smithi160 ceph-mon[101879]: osdmap e661: 8 total, 7 up, 8 in 2024-09-21T10:07:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:10 smithi160 ceph-mon[101879]: pgmap v510: 521 pgs: 62 stale+active+clean, 459 active+clean; 7.6 MiB data, 537 MiB used, 715 GiB / 715 GiB avail; 699 B/s rd, 700 KiB/s wr, 1 op/s 2024-09-21T10:07:11.327 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:10 smithi070 podman[124617]: 2024-09-21 10:07:10.993794514 +0000 UTC m=+0.864772720 container remove 858352879af011e727ad96b2492392239b64b9662605ce08025f0bae7c0e0083 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux ) 2024-09-21T10:07:11.327 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:11 smithi070 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.0.service: Deactivated successfully. 2024-09-21T10:07:11.327 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:11 smithi070 systemd[1]: Stopped Ceph osd.0 for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:07:11.327 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:11 smithi070 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.0.service: Consumed 30.786s CPU time. 2024-09-21T10:07:11.327 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:11 smithi070 systemd[1]: Starting Ceph osd.0 for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:07:11.582 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:11 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:11.582 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:11 smithi070 podman[124720]: 2024-09-21 10:07:11.331009833 +0000 UTC m=+0.016686854 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:07:11.582 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:11 smithi070 podman[124720]: 2024-09-21 10:07:11.444622941 +0000 UTC m=+0.130299949 container create 65293f58d6927113263497192f3717bd2e7d994a380bb5cb54b5539a213b03f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0-activate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-21T10:07:11.583 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:11 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:11.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:11 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:11.965 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:11 smithi070 podman[124720]: 2024-09-21 10:07:11.705534278 +0000 UTC m=+0.391211293 container init 65293f58d6927113263497192f3717bd2e7d994a380bb5cb54b5539a213b03f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0-activate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:07:11.965 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:11 smithi070 podman[124720]: 2024-09-21 10:07:11.710444757 +0000 UTC m=+0.396121766 container start 65293f58d6927113263497192f3717bd2e7d994a380bb5cb54b5539a213b03f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=main-62da54b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-21T10:07:11.965 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:11 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0-activate[124736]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:07:11.965 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:11 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0-activate[124736]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:07:12.322 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:11 smithi070 bash[124720]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:07:12.322 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:11 smithi070 bash[124720]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:07:12.323 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:11 smithi070 podman[124720]: 2024-09-21 10:07:11.963688739 +0000 UTC m=+0.649365764 container attach 65293f58d6927113263497192f3717bd2e7d994a380bb5cb54b5539a213b03f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0-activate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=main-62da54b, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-21T10:07:12.323 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0-activate[124736]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T10:07:12.323 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:12 smithi070 bash[124720]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T10:07:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:12 smithi160 ceph-mon[101879]: pgmap v511: 521 pgs: 133 active+undersized, 51 active+undersized+degraded, 337 active+clean; 13 MiB data, 552 MiB used, 715 GiB / 715 GiB avail; 1.3 MiB/s wr, 1 op/s; 75/591 objects degraded (12.690%) 2024-09-21T10:07:12.611 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:12 smithi070 ceph-mon[107385]: pgmap v511: 521 pgs: 133 active+undersized, 51 active+undersized+degraded, 337 active+clean; 13 MiB data, 552 MiB used, 715 GiB / 715 GiB avail; 1.3 MiB/s wr, 1 op/s; 75/591 objects degraded (12.690%) 2024-09-21T10:07:12.611 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:12 smithi070 ceph-mon[112139]: pgmap v511: 521 pgs: 133 active+undersized, 51 active+undersized+degraded, 337 active+clean; 13 MiB data, 552 MiB used, 715 GiB / 715 GiB avail; 1.3 MiB/s wr, 1 op/s; 75/591 objects degraded (12.690%) 2024-09-21T10:07:12.612 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0-activate[124736]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-21T10:07:12.612 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:12 smithi070 bash[124720]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-21T10:07:12.612 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0-activate[124736]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-21T10:07:12.612 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:12 smithi070 bash[124720]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-21T10:07:12.878 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0-activate[124736]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-21T10:07:12.878 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:12 smithi070 bash[124720]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-21T10:07:12.878 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0-activate[124736]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-21T10:07:12.878 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:12 smithi070 bash[124720]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-21T10:07:12.878 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0-activate[124736]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T10:07:12.878 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:12 smithi070 bash[124720]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T10:07:12.878 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0-activate[124736]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-21T10:07:12.878 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:12 smithi070 bash[124720]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-21T10:07:12.878 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:12 smithi070 podman[124720]: 2024-09-21 10:07:12.640166657 +0000 UTC m=+1.325843672 container died 65293f58d6927113263497192f3717bd2e7d994a380bb5cb54b5539a213b03f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0-activate, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b) 2024-09-21T10:07:13.260 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:07:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:07:13] "GET /metrics HTTP/1.1" 200 67617 "" "Prometheus/2.51.0" 2024-09-21T10:07:13.587 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:13 smithi070 ceph-mon[107385]: Health check failed: Degraded data redundancy: 75/591 objects degraded (12.690%), 51 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:13.587 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:13 smithi070 ceph-mon[107385]: pool 'test-librbd-smithi070-86016-28' is full (reached quota's max_bytes: 10 MiB) 2024-09-21T10:07:13.587 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:13 smithi070 ceph-mon[107385]: Health check update: 2 pool(s) full (POOL_FULL) 2024-09-21T10:07:13.587 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:13 smithi070 ceph-mon[107385]: osdmap e662: 8 total, 7 up, 8 in 2024-09-21T10:07:13.587 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:13 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-27", "field": "max_objects", "val": "7"} : dispatch 2024-09-21T10:07:13.587 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:13 smithi070 podman[124892]: 2024-09-21 10:07:13.258852732 +0000 UTC m=+0.612068141 container remove 65293f58d6927113263497192f3717bd2e7d994a380bb5cb54b5539a213b03f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0-activate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-21T10:07:13.588 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:13 smithi070 podman[124958]: 2024-09-21 10:07:13.36461039 +0000 UTC m=+0.015990592 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:07:13.588 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:13 smithi070 podman[124958]: 2024-09-21 10:07:13.468055168 +0000 UTC m=+0.119435370 container create eedacb586f7cd3a818326f1341db47909276349880368abce57bf2152e01a536 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:07:13.588 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:13 smithi070 ceph-mon[112139]: Health check failed: Degraded data redundancy: 75/591 objects degraded (12.690%), 51 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:13.588 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:13 smithi070 ceph-mon[112139]: pool 'test-librbd-smithi070-86016-28' is full (reached quota's max_bytes: 10 MiB) 2024-09-21T10:07:13.588 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:13 smithi070 ceph-mon[112139]: Health check update: 2 pool(s) full (POOL_FULL) 2024-09-21T10:07:13.588 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:13 smithi070 ceph-mon[112139]: osdmap e662: 8 total, 7 up, 8 in 2024-09-21T10:07:13.588 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:13 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-27", "field": "max_objects", "val": "7"} : dispatch 2024-09-21T10:07:13.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:13 smithi160 ceph-mon[101879]: Health check failed: Degraded data redundancy: 75/591 objects degraded (12.690%), 51 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:13.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:13 smithi160 ceph-mon[101879]: pool 'test-librbd-smithi070-86016-28' is full (reached quota's max_bytes: 10 MiB) 2024-09-21T10:07:13.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:13 smithi160 ceph-mon[101879]: Health check update: 2 pool(s) full (POOL_FULL) 2024-09-21T10:07:13.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:13 smithi160 ceph-mon[101879]: osdmap e662: 8 total, 7 up, 8 in 2024-09-21T10:07:13.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:13 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-27", "field": "max_objects", "val": "7"} : dispatch 2024-09-21T10:07:13.708 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:07:14.127 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:13 smithi070 podman[124958]: 2024-09-21 10:07:13.695125991 +0000 UTC m=+0.346506200 container init eedacb586f7cd3a818326f1341db47909276349880368abce57bf2152e01a536 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True) 2024-09-21T10:07:14.128 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:13 smithi070 podman[124958]: 2024-09-21 10:07:13.699738781 +0000 UTC m=+0.351118993 container start eedacb586f7cd3a818326f1341db47909276349880368abce57bf2152e01a536 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-21T10:07:14.128 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:13 smithi070 ceph-osd[124995]: -- 172.21.15.70:0/2092015211 <== mon.0 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5621946bfa40 con 0x56219548c400 2024-09-21T10:07:14.128 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:13 smithi070 bash[124958]: eedacb586f7cd3a818326f1341db47909276349880368abce57bf2152e01a536 2024-09-21T10:07:14.128 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:13 smithi070 systemd[1]: Started Ceph osd.0 for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:07:14.731 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:07:15.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:14 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-27", "field": "max_objects", "val": "7"}]': finished 2024-09-21T10:07:15.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:14 smithi160 ceph-mon[101879]: osdmap e663: 8 total, 7 up, 8 in 2024-09-21T10:07:15.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:14 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:15.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:14 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:15.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:14 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:15.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:14 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:15.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:14 smithi160 ceph-mon[101879]: pgmap v514: 521 pgs: 133 active+undersized, 51 active+undersized+degraded, 337 active+clean; 13 MiB data, 552 MiB used, 715 GiB / 715 GiB avail; 874 KiB/s wr, 1 op/s; 75/591 objects degraded (12.690%) 2024-09-21T10:07:15.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:14 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-27", "field": "max_objects", "val": "7"}]': finished 2024-09-21T10:07:15.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:14 smithi070 ceph-mon[107385]: osdmap e663: 8 total, 7 up, 8 in 2024-09-21T10:07:15.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:14 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:15.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:14 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:15.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:14 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:15.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:14 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:15.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:14 smithi070 ceph-mon[107385]: pgmap v514: 521 pgs: 133 active+undersized, 51 active+undersized+degraded, 337 active+clean; 13 MiB data, 552 MiB used, 715 GiB / 715 GiB avail; 874 KiB/s wr, 1 op/s; 75/591 objects degraded (12.690%) 2024-09-21T10:07:15.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:14 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi070-86016-27", "field": "max_objects", "val": "7"}]': finished 2024-09-21T10:07:15.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:14 smithi070 ceph-mon[112139]: osdmap e663: 8 total, 7 up, 8 in 2024-09-21T10:07:15.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:14 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:15.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:14 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:15.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:14 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:15.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:14 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:15.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:14 smithi070 ceph-mon[112139]: pgmap v514: 521 pgs: 133 active+undersized, 51 active+undersized+degraded, 337 active+clean; 13 MiB data, 552 MiB used, 715 GiB / 715 GiB avail; 874 KiB/s wr, 1 op/s; 75/591 objects degraded (12.690%) 2024-09-21T10:07:15.629 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:15 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0[124972]: 2024-09-21T10:07:15.366+0000 7fa7af912640 -1 Falling back to public interface 2024-09-21T10:07:15.739 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:07:16.749 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:07:17.168 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:17 smithi070 ceph-mon[107385]: pgmap v515: 521 pgs: 133 active+undersized, 51 active+undersized+degraded, 337 active+clean; 14 MiB data, 558 MiB used, 715 GiB / 715 GiB avail; 898 KiB/s wr, 3 op/s; 77/621 objects degraded (12.399%) 2024-09-21T10:07:17.169 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:17 smithi070 ceph-mon[112139]: pgmap v515: 521 pgs: 133 active+undersized, 51 active+undersized+degraded, 337 active+clean; 14 MiB data, 558 MiB used, 715 GiB / 715 GiB avail; 898 KiB/s wr, 3 op/s; 77/621 objects degraded (12.399%) 2024-09-21T10:07:17.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:17 smithi160 ceph-mon[101879]: pgmap v515: 521 pgs: 133 active+undersized, 51 active+undersized+degraded, 337 active+clean; 14 MiB data, 558 MiB used, 715 GiB / 715 GiB avail; 898 KiB/s wr, 3 op/s; 77/621 objects degraded (12.399%) 2024-09-21T10:07:17.759 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:07:17.825 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.RemoveFullTryNamespaceDataPool (24966 ms) 2024-09-21T10:07:17.826 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestIO 2024-09-21T10:07:17.826 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:07:18.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:18 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:18.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:18 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:18.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:18 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 77/621 objects degraded (12.399%), 51 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:18.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:18 smithi070 ceph-mon[107385]: pool 'test-librbd-smithi070-86016-27' is full (reached quota's max_objects: 7) 2024-09-21T10:07:18.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:18 smithi070 ceph-mon[107385]: Health check update: 3 pool(s) full (POOL_FULL) 2024-09-21T10:07:18.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:18 smithi070 ceph-mon[107385]: osdmap e664: 8 total, 7 up, 8 in 2024-09-21T10:07:18.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:18 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:18.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:18 smithi070 ceph-mon[107385]: pgmap v517: 521 pgs: 133 active+undersized, 51 active+undersized+degraded, 337 active+clean; 14 MiB data, 558 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 173 KiB/s wr, 8 op/s; 77/621 objects degraded (12.399%) 2024-09-21T10:07:18.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:18 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:18.740 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:18 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:18.740 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:18 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:18.740 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:18 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 77/621 objects degraded (12.399%), 51 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:18.740 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:18 smithi070 ceph-mon[112139]: pool 'test-librbd-smithi070-86016-27' is full (reached quota's max_objects: 7) 2024-09-21T10:07:18.741 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:18 smithi070 ceph-mon[112139]: Health check update: 3 pool(s) full (POOL_FULL) 2024-09-21T10:07:18.741 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:18 smithi070 ceph-mon[112139]: osdmap e664: 8 total, 7 up, 8 in 2024-09-21T10:07:18.741 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:18 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:18.741 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:18 smithi070 ceph-mon[112139]: pgmap v517: 521 pgs: 133 active+undersized, 51 active+undersized+degraded, 337 active+clean; 14 MiB data, 558 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 173 KiB/s wr, 8 op/s; 77/621 objects degraded (12.399%) 2024-09-21T10:07:18.741 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:18 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:18 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:18 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:18 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 77/621 objects degraded (12.399%), 51 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:18 smithi160 ceph-mon[101879]: pool 'test-librbd-smithi070-86016-27' is full (reached quota's max_objects: 7) 2024-09-21T10:07:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:18 smithi160 ceph-mon[101879]: Health check update: 3 pool(s) full (POOL_FULL) 2024-09-21T10:07:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:18 smithi160 ceph-mon[101879]: osdmap e664: 8 total, 7 up, 8 in 2024-09-21T10:07:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:18 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:18.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:18 smithi160 ceph-mon[101879]: pgmap v517: 521 pgs: 133 active+undersized, 51 active+undersized+degraded, 337 active+clean; 14 MiB data, 558 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 173 KiB/s wr, 8 op/s; 77/621 objects degraded (12.399%) 2024-09-21T10:07:18.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:18 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:19.127 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:18 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0[124972]: 2024-09-21T10:07:18.805+0000 7fa7af912640 -1 osd.0 660 log_to_monitors true 2024-09-21T10:07:19.499 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:19 smithi070 ceph-mon[107385]: from='osd.0 [v2:172.21.15.70:6802/452014366,v1:172.21.15.70:6803/452014366]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T10:07:19.499 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:19 smithi070 ceph-mon[107385]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T10:07:19.499 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:19 smithi070 ceph-mon[112139]: from='osd.0 [v2:172.21.15.70:6802/452014366,v1:172.21.15.70:6803/452014366]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T10:07:19.499 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:19 smithi070 ceph-mon[112139]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T10:07:19.540 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:07:19.540 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:07:19.540 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:07:19.540 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:07:19.540 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:07:19.540 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:19.540 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:07:19.540 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:19.540 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:19.541 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:07:19.541 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:19.541 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:07:19.541 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:19.541 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:19.541 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:07:19.541 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:19.541 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:07:19.541 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:19.541 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:19.541 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:07:19.541 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:19.541 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:07:19.541 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:19.542 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:19.542 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:07:19.542 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:19.542 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:07:19.542 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:19.542 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:19.542 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:07:19.542 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-21T10:07:19.542 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-21T10:07:19.542 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-21T10:07:19.542 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-21T10:07:19.542 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-21T10:07:19.543 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-21T10:07:19.543 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-21T10:07:19.543 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-21T10:07:19.543 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-21T10:07:19.543 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-21T10:07:19.543 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:19.543 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-21T10:07:19.543 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:19.543 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:19.543 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-21T10:07:19.543 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:19.543 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-21T10:07:19.543 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:19.543 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:19.543 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-21T10:07:19.544 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:19.544 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-21T10:07:19.544 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:19.544 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:19.544 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-21T10:07:19.544 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:19.544 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-21T10:07:19.544 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:19.544 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:19.544 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-21T10:07:19.544 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:19.544 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-21T10:07:19.544 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:19.544 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:19.544 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-21T10:07:19.545 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:19.545 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:19.545 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:19.545 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:19.545 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:19.545 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:19.545 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:07:19.545 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:07:19.545 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:07:19.545 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:19.545 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:07:19.545 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:07:19.545 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:07:19.546 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:19.546 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:07:19.546 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:07:19.546 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:07:19.546 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:19.546 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:07:19.546 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:07:19.546 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:07:19.546 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:19.546 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:07:19.546 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:07:19.546 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:07:19.546 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 512~512 = 512 2024-09-21T10:07:19.547 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:19.547 INFO:tasks.workunit.client.0.smithi070.stdout:aio discard: 1536~512 = 0 2024-09-21T10:07:19.547 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:19.547 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:19.547 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:19.547 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:19.547 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:19.547 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 0 2024-09-21T10:07:19.547 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:19.547 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:19.547 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 0 2024-09-21T10:07:19.547 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:19.548 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:19.548 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 16384 2024-09-21T10:07:19.548 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:19.548 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:19.548 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 16384 2024-09-21T10:07:19.548 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:19.548 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:19.548 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:19.548 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:19.548 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 49152 2024-09-21T10:07:19.548 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:19.548 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:19.548 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 49152 2024-09-21T10:07:19.548 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:19.549 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:19.549 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 65536 2024-09-21T10:07:19.549 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:19.549 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:19.549 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 65536 2024-09-21T10:07:19.549 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:19.549 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:19.549 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:19.549 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:19.549 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 98304 2024-09-21T10:07:19.549 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:19.549 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:19.549 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 98304 2024-09-21T10:07:19.549 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:19.549 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:19.550 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 114688 2024-09-21T10:07:19.550 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:19.550 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:19.550 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 114688 2024-09-21T10:07:19.550 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:19.550 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:19.550 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:19.550 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:19.550 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 147456 2024-09-21T10:07:19.550 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:19.550 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:19.550 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 147456 2024-09-21T10:07:19.550 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:19.550 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:19.551 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 163840 2024-09-21T10:07:19.551 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:19.551 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:19.551 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 163840 2024-09-21T10:07:19.551 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:19.551 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:19.551 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:19.551 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:19.551 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 196608 2024-09-21T10:07:19.551 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:19.551 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:19.551 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 196608 2024-09-21T10:07:19.551 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:19.551 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:19.552 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 212992 2024-09-21T10:07:19.552 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:19.552 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:19.552 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 212992 2024-09-21T10:07:19.552 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:19.552 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:19.552 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:19.552 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:19.552 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:19.552 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:19.552 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:19.552 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:19.552 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:19.552 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:19.553 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:19.553 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:19.553 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:19.553 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:19.553 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:19.553 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:19.553 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:19.553 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:19.553 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:19.553 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:19.553 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:19.553 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:19.553 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:19.553 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:19.553 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:19.553 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:19.554 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:19.554 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:19.554 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:19.554 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:19.554 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:19.554 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:19.554 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:19.554 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:19.554 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:19.554 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:19.554 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:19.554 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:19.554 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:19.554 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:19.555 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:19.555 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:19.555 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:19.555 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:19.555 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:19.555 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:19.555 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:19.555 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:19.555 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:19.555 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:19.555 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:19.555 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:19.555 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:19.822 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 10:07:19 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0[124972]: 2024-09-21T10:07:19.496+0000 7fa7a76bc640 -1 osd.0 660 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T10:07:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:19 smithi160 ceph-mon[101879]: from='osd.0 [v2:172.21.15.70:6802/452014366,v1:172.21.15.70:6803/452014366]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T10:07:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:19 smithi160 ceph-mon[101879]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T10:07:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:20 smithi160 ceph-mon[101879]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T10:07:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:20 smithi160 ceph-mon[101879]: osdmap e665: 8 total, 7 up, 8 in 2024-09-21T10:07:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:20 smithi160 ceph-mon[101879]: from='osd.0 [v2:172.21.15.70:6802/452014366,v1:172.21.15.70:6803/452014366]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]} : dispatch 2024-09-21T10:07:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:20 smithi160 ceph-mon[101879]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]} : dispatch 2024-09-21T10:07:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:20 smithi160 ceph-mon[101879]: pgmap v519: 521 pgs: 133 active+undersized, 51 active+undersized+degraded, 337 active+clean; 14 MiB data, 529 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 164 KiB/s wr, 8 op/s; 77/621 objects degraded (12.399%) 2024-09-21T10:07:20.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:20.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:20.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:20 smithi070 ceph-mon[107385]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T10:07:20.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:20 smithi070 ceph-mon[107385]: osdmap e665: 8 total, 7 up, 8 in 2024-09-21T10:07:20.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:20 smithi070 ceph-mon[107385]: from='osd.0 [v2:172.21.15.70:6802/452014366,v1:172.21.15.70:6803/452014366]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]} : dispatch 2024-09-21T10:07:20.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:20 smithi070 ceph-mon[107385]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]} : dispatch 2024-09-21T10:07:20.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:20 smithi070 ceph-mon[107385]: pgmap v519: 521 pgs: 133 active+undersized, 51 active+undersized+degraded, 337 active+clean; 14 MiB data, 529 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 164 KiB/s wr, 8 op/s; 77/621 objects degraded (12.399%) 2024-09-21T10:07:20.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:20.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:20.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:20 smithi070 ceph-mon[112139]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T10:07:20.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:20 smithi070 ceph-mon[112139]: osdmap e665: 8 total, 7 up, 8 in 2024-09-21T10:07:20.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:20 smithi070 ceph-mon[112139]: from='osd.0 [v2:172.21.15.70:6802/452014366,v1:172.21.15.70:6803/452014366]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]} : dispatch 2024-09-21T10:07:20.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:20 smithi070 ceph-mon[112139]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]} : dispatch 2024-09-21T10:07:20.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:20 smithi070 ceph-mon[112139]: pgmap v519: 521 pgs: 133 active+undersized, 51 active+undersized+degraded, 337 active+clean; 14 MiB data, 529 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 164 KiB/s wr, 8 op/s; 77/621 objects degraded (12.399%) 2024-09-21T10:07:20.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:20.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:21.530 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:21.530 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:21.530 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:21.530 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:21.530 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:21.530 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:21.530 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:21.530 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:21.530 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:21.531 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:21.531 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:21.531 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:21.531 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:21.531 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:21.531 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:21.531 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:21.531 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:21.531 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:21.531 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:21.531 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:21.531 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:21.531 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:21.531 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:21.532 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:21.532 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:21.532 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:21.532 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:21.532 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:21.532 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:21.532 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:21.532 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:21.532 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:21.532 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:21.532 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:21.532 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:21.532 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:21.533 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:21.533 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:21.533 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:21.533 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:21.533 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:21.533 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:21.533 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:21.533 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:21.533 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:21.533 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:21.533 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:21.533 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:21.534 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:21.534 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:21.534 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:21.534 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:21.534 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:21.534 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:21.534 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:21.534 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:21.534 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:21.534 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:21.534 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:21.534 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:21.534 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:21.535 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:21.535 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:21.535 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:21.535 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:21.535 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:21.535 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:21.535 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:21.535 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:21.535 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:21.535 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:21.535 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:21.535 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:21.535 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:21.535 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:21.535 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:21.536 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:21.536 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:21.536 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:21.536 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:21.536 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:21.536 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:21.536 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:21.536 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:21.536 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:21.536 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:21.536 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:21.537 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:21.537 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:21.537 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:21.537 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:21.537 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:21.537 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:21.537 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:21.537 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:21.537 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:21.537 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:21.537 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:21.537 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:21.538 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:21.538 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:21.538 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:21.538 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:21.538 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:21.538 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:21.538 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:21.538 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:21.538 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:21.538 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:21.538 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:21.539 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:21.539 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:21.539 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:21.539 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:21.539 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:21.539 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:21.539 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:21.539 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:21.539 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:21.539 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:21.539 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:21.539 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:07:21.539 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:07:21.540 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:07:21.540 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:07:21.540 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestIO (3704 ms) 2024-09-21T10:07:21.540 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS1 2024-09-21T10:07:21.540 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:07:21.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T10:07:21.363+0000 7fd088d1d640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (47 PGs are or would become offline) 2024-09-21T10:07:21.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[107385]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:07:21.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[107385]: osd.0 [v2:172.21.15.70:6802/452014366,v1:172.21.15.70:6803/452014366] boot 2024-09-21T10:07:21.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[107385]: osdmap e666: 8 total, 8 up, 8 in 2024-09-21T10:07:21.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T10:07:21.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:21.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:21.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:07:21.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:07:21.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:21.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:21.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:21.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:21.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:21.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-21T10:07:21.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[107385]: osdmap e667: 8 total, 8 up, 8 in 2024-09-21T10:07:21.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[112139]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:07:21.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[112139]: osd.0 [v2:172.21.15.70:6802/452014366,v1:172.21.15.70:6803/452014366] boot 2024-09-21T10:07:21.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[112139]: osdmap e666: 8 total, 8 up, 8 in 2024-09-21T10:07:21.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T10:07:21.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:21.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:21.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:07:21.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:07:21.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:21.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:21.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:21.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:21.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:21.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-21T10:07:21.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:21 smithi070 ceph-mon[112139]: osdmap e667: 8 total, 8 up, 8 in 2024-09-21T10:07:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:21 smithi160 ceph-mon[101879]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:07:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:21 smithi160 ceph-mon[101879]: osd.0 [v2:172.21.15.70:6802/452014366,v1:172.21.15.70:6803/452014366] boot 2024-09-21T10:07:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:21 smithi160 ceph-mon[101879]: osdmap e666: 8 total, 8 up, 8 in 2024-09-21T10:07:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T10:07:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:21.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:21.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:07:21.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:07:21.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:21.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:21.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:21.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:21.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:21.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-21T10:07:21.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:21 smithi160 ceph-mon[101879]: osdmap e667: 8 total, 8 up, 8 in 2024-09-21T10:07:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:22 smithi070 ceph-mon[107385]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-21T10:07:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:22 smithi070 ceph-mon[107385]: Upgrade: unsafe to stop osd(s) at this time (47 PGs are or would become offline) 2024-09-21T10:07:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:22 smithi070 ceph-mon[107385]: pgmap v522: 521 pgs: 122 peering, 48 active+undersized, 14 active+undersized+degraded, 337 active+clean; 585 KiB data, 496 MiB used, 715 GiB / 715 GiB avail; 1004 KiB/s rd, 490 KiB/s wr, 1.93k op/s; 17/618 objects degraded (2.751%) 2024-09-21T10:07:22.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:22 smithi070 ceph-mon[112139]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-21T10:07:22.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:22 smithi070 ceph-mon[112139]: Upgrade: unsafe to stop osd(s) at this time (47 PGs are or would become offline) 2024-09-21T10:07:22.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:22 smithi070 ceph-mon[112139]: pgmap v522: 521 pgs: 122 peering, 48 active+undersized, 14 active+undersized+degraded, 337 active+clean; 585 KiB data, 496 MiB used, 715 GiB / 715 GiB avail; 1004 KiB/s rd, 490 KiB/s wr, 1.93k op/s; 17/618 objects degraded (2.751%) 2024-09-21T10:07:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:22 smithi160 ceph-mon[101879]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-21T10:07:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:22 smithi160 ceph-mon[101879]: Upgrade: unsafe to stop osd(s) at this time (47 PGs are or would become offline) 2024-09-21T10:07:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:22 smithi160 ceph-mon[101879]: pgmap v522: 521 pgs: 122 peering, 48 active+undersized, 14 active+undersized+degraded, 337 active+clean; 585 KiB data, 496 MiB used, 715 GiB / 715 GiB avail; 1004 KiB/s rd, 490 KiB/s wr, 1.93k op/s; 17/618 objects degraded (2.751%) 2024-09-21T10:07:23.472 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:07:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:07:23] "GET /metrics HTTP/1.1" 200 67628 "" "Prometheus/2.51.0" 2024-09-21T10:07:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:23 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 17/618 objects degraded (2.751%), 14 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:23 smithi160 ceph-mon[101879]: pool 'test-librbd-smithi070-86016-28' no longer out of quota; removing NO_QUOTA flag 2024-09-21T10:07:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:23 smithi160 ceph-mon[101879]: Health check update: 2 pool(s) full (POOL_FULL) 2024-09-21T10:07:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:23 smithi160 ceph-mon[101879]: osdmap e668: 8 total, 8 up, 8 in 2024-09-21T10:07:23.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:23 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 17/618 objects degraded (2.751%), 14 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:23.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:23 smithi070 ceph-mon[112139]: pool 'test-librbd-smithi070-86016-28' no longer out of quota; removing NO_QUOTA flag 2024-09-21T10:07:23.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:23 smithi070 ceph-mon[112139]: Health check update: 2 pool(s) full (POOL_FULL) 2024-09-21T10:07:23.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:23 smithi070 ceph-mon[112139]: osdmap e668: 8 total, 8 up, 8 in 2024-09-21T10:07:23.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:23 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 17/618 objects degraded (2.751%), 14 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:23.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:23 smithi070 ceph-mon[107385]: pool 'test-librbd-smithi070-86016-28' no longer out of quota; removing NO_QUOTA flag 2024-09-21T10:07:23.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:23 smithi070 ceph-mon[107385]: Health check update: 2 pool(s) full (POOL_FULL) 2024-09-21T10:07:23.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:23 smithi070 ceph-mon[107385]: osdmap e668: 8 total, 8 up, 8 in 2024-09-21T10:07:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:24 smithi160 ceph-mon[101879]: pgmap v524: 521 pgs: 122 peering, 48 active+undersized, 14 active+undersized+degraded, 337 active+clean; 585 KiB data, 496 MiB used, 715 GiB / 715 GiB avail; 947 KiB/s rd, 462 KiB/s wr, 1.82k op/s; 17/618 objects degraded (2.751%) 2024-09-21T10:07:24.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:24 smithi070 ceph-mon[112139]: pgmap v524: 521 pgs: 122 peering, 48 active+undersized, 14 active+undersized+degraded, 337 active+clean; 585 KiB data, 496 MiB used, 715 GiB / 715 GiB avail; 947 KiB/s rd, 462 KiB/s wr, 1.82k op/s; 17/618 objects degraded (2.751%) 2024-09-21T10:07:24.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:24 smithi070 ceph-mon[107385]: pgmap v524: 521 pgs: 122 peering, 48 active+undersized, 14 active+undersized+degraded, 337 active+clean; 585 KiB data, 496 MiB used, 715 GiB / 715 GiB avail; 947 KiB/s rd, 462 KiB/s wr, 1.82k op/s; 17/618 objects degraded (2.751%) 2024-09-21T10:07:26.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:26 smithi160 ceph-mon[101879]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 17/618 objects degraded (2.751%), 14 pgs degraded) 2024-09-21T10:07:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:26 smithi070 ceph-mon[107385]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 17/618 objects degraded (2.751%), 14 pgs degraded) 2024-09-21T10:07:26.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:26 smithi070 ceph-mon[112139]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 17/618 objects degraded (2.751%), 14 pgs degraded) 2024-09-21T10:07:26.655 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:07:26.655 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:07:26.655 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:07:26.655 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:07:26.655 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:07:26.655 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:26.655 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:07:26.655 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:26.655 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:26.655 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:07:26.656 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:26.656 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:07:26.656 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:26.656 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:26.656 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:07:26.656 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:26.656 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:07:26.656 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:26.656 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:26.656 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:07:26.656 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:26.656 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:07:26.656 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:26.656 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:26.657 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:07:26.657 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:26.657 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:07:26.657 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:26.657 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:26.657 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:07:26.657 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-21T10:07:26.657 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-21T10:07:26.657 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-21T10:07:26.657 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-21T10:07:26.657 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-21T10:07:26.657 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-21T10:07:26.657 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-21T10:07:26.657 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-21T10:07:26.657 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-21T10:07:26.658 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-21T10:07:26.658 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:26.658 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-21T10:07:26.658 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:26.658 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:26.658 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-21T10:07:26.658 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:26.658 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-21T10:07:26.658 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:26.658 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:26.658 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-21T10:07:26.658 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:26.659 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-21T10:07:26.659 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:26.659 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:26.659 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-21T10:07:26.659 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:26.659 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-21T10:07:26.659 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:26.659 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:26.659 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-21T10:07:26.659 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:26.659 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-21T10:07:26.659 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:26.659 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:26.659 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-21T10:07:26.659 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:26.660 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:26.660 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:26.660 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:26.660 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:26.660 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:26.660 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:07:26.660 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:07:26.660 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:07:26.660 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:26.660 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:07:26.660 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:07:26.660 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:07:26.660 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:26.661 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:07:26.661 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:07:26.661 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:07:26.661 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:26.661 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:07:26.661 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:07:26.661 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:07:26.661 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:26.661 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:07:26.661 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:07:26.661 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:07:26.661 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 512~512 = 512 2024-09-21T10:07:26.661 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:26.661 INFO:tasks.workunit.client.0.smithi070.stdout:aio discard: 1536~512 = 0 2024-09-21T10:07:26.662 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:26.662 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:26.662 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:26.662 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:26.662 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:26.662 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 0 2024-09-21T10:07:26.662 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:26.662 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:26.662 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 0 2024-09-21T10:07:26.662 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:26.662 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:26.662 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 16384 2024-09-21T10:07:26.662 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:26.662 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:26.663 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 16384 2024-09-21T10:07:26.663 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:26.663 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:26.663 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:26.663 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:26.663 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 49152 2024-09-21T10:07:26.663 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:26.663 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:26.663 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 49152 2024-09-21T10:07:26.663 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:26.663 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:26.663 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 65536 2024-09-21T10:07:26.663 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:26.663 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:26.664 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 65536 2024-09-21T10:07:26.664 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:26.664 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:26.664 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:26.664 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:26.664 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 98304 2024-09-21T10:07:26.664 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:26.664 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:26.664 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 98304 2024-09-21T10:07:26.664 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:26.664 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:26.664 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 114688 2024-09-21T10:07:26.664 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:26.664 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:26.664 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 114688 2024-09-21T10:07:26.665 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:26.665 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:26.665 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:26.665 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:26.665 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 147456 2024-09-21T10:07:26.665 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:26.665 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:26.665 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 147456 2024-09-21T10:07:26.665 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:26.665 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:26.665 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 163840 2024-09-21T10:07:26.665 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:26.665 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:26.665 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 163840 2024-09-21T10:07:26.665 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:26.666 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:26.666 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:26.666 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:26.666 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 196608 2024-09-21T10:07:26.666 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:26.666 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:26.666 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 196608 2024-09-21T10:07:26.666 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:26.666 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:26.666 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 212992 2024-09-21T10:07:26.666 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:26.666 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:26.666 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 212992 2024-09-21T10:07:26.667 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:26.667 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:26.667 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:26.667 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:26.667 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:26.667 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:26.667 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:26.667 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:26.667 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:26.667 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:26.667 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:26.667 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:26.667 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:26.667 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:26.667 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:26.668 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:26.668 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:26.668 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:26.668 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:26.668 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:26.668 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:26.668 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:26.668 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:26.668 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:26.668 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:26.668 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:26.668 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:26.668 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:26.668 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:26.669 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:26.669 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:26.669 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:26.669 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:26.669 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:26.669 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:26.669 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:26.669 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:26.669 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:26.669 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:26.669 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:26.669 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:26.669 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:26.669 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:26.670 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:26.670 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:26.670 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:26.670 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:26.670 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:26.670 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:26.670 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:26.670 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:26.670 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:26.670 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:27.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:27 smithi160 ceph-mon[101879]: pgmap v525: 521 pgs: 521 active+clean; 6.8 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 1.90k op/s; 18 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:07:27.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:27 smithi070 ceph-mon[107385]: pgmap v525: 521 pgs: 521 active+clean; 6.8 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 1.90k op/s; 18 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:07:27.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:27 smithi070 ceph-mon[112139]: pgmap v525: 521 pgs: 521 active+clean; 6.8 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 1.90k op/s; 18 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:07:28.844 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-09-21T10:07:29.458 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T10:07:29.458 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (7m) 12s ago 13m 19.3M - 0.27.0 11f11916f8cd 89ae95182419 2024-09-21T10:07:29.458 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi160 *:3000 running (7m) 30s ago 13m 81.2M - 10.4.8 5aad1d7cf004 daa2d8dfab48 2024-09-21T10:07:29.459 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi070.gansda smithi070 running (12m) 12s ago 12m 23.4M - 18.2.4-871-g6a1bdf85 0adab276c97e 6016b2501c8a 2024-09-21T10:07:29.459 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi160.byyhfz smithi160 running (12m) 30s ago 12m 17.4M - 18.2.4-871-g6a1bdf85 0adab276c97e 93e108d91e65 2024-09-21T10:07:29.459 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi160 *:8443,9283,8765 running (9m) 30s ago 17m 458M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 d952652a9683 2024-09-21T10:07:29.459 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (9m) 12s ago 19m 532M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 e3d390893cff 2024-09-21T10:07:29.459 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (6m) 12s ago 19m 72.9M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 4a7b667a4033 2024-09-21T10:07:29.459 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi160 running (4m) 30s ago 18m 46.4M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 86e88af21f36 2024-09-21T10:07:29.459 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (6m) 12s ago 17m 58.4M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 e154380b80a2 2024-09-21T10:07:29.459 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (8m) 12s ago 13m 10.5M - 1.7.0 72c9c2088986 5d5d4b026070 2024-09-21T10:07:29.459 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi160 *:9100 running (8m) 30s ago 13m 10.2M - 1.7.0 72c9c2088986 19b8e0e8ee0c 2024-09-21T10:07:29.459 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (16s) 12s ago 17m 11.5M 2480M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 eedacb586f7c 2024-09-21T10:07:29.459 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (16m) 12s ago 16m 127M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e b7c0d1169d1d 2024-09-21T10:07:29.459 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (16m) 12s ago 16m 145M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 58a11b8ddb81 2024-09-21T10:07:29.459 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (16m) 12s ago 16m 135M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e aa0c8a732a41 2024-09-21T10:07:29.460 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi160 running (15m) 30s ago 15m 124M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 7325a075c488 2024-09-21T10:07:29.460 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi160 running (15m) 30s ago 15m 117M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e f74026345641 2024-09-21T10:07:29.460 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi160 running (14m) 30s ago 14m 143M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 03e6bb54f8b2 2024-09-21T10:07:29.460 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi160 running (14m) 30s ago 14m 126M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e e405b21a6146 2024-09-21T10:07:29.460 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi160 *:9095 running (8m) 30s ago 14m 58.1M - 2.51.0 1d3b7f56885b 648952e57ad7 2024-09-21T10:07:29.460 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:29 smithi070 ceph-mon[112139]: pgmap v526: 521 pgs: 521 active+clean; 6.8 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 931 KiB/s rd, 1019 KiB/s wr, 1.50k op/s; 14 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:07:29.460 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:29 smithi070 ceph-mon[107385]: pgmap v526: 521 pgs: 521 active+clean; 6.8 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 931 KiB/s rd, 1019 KiB/s wr, 1.50k op/s; 14 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:07:29.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:29 smithi160 ceph-mon[101879]: pgmap v526: 521 pgs: 521 active+clean; 6.8 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 931 KiB/s rd, 1019 KiB/s wr, 1.50k op/s; 14 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:07:29.817 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:07:29.817 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-09-21T10:07:29.817 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 3 2024-09-21T10:07:29.817 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:07:29.817 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-09-21T10:07:29.817 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T10:07:29.818 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:07:29.818 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-09-21T10:07:29.818 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 7, 2024-09-21T10:07:29.818 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 1 2024-09-21T10:07:29.818 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:07:29.818 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": { 2024-09-21T10:07:29.818 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 2 2024-09-21T10:07:29.818 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:07:29.818 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-09-21T10:07:29.818 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 9, 2024-09-21T10:07:29.818 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 6 2024-09-21T10:07:29.818 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:07:29.818 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:07:30.106 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:07:30.106 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47", 2024-09-21T10:07:30.106 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-09-21T10:07:30.106 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 4 daemons (3 remaining).", 2024-09-21T10:07:30.106 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-09-21T10:07:30.106 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "1/8 daemons upgraded", 2024-09-21T10:07:30.106 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading osd daemons", 2024-09-21T10:07:30.107 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-09-21T10:07:30.107 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:07:30.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:30 smithi070 ceph-mon[107385]: from='client.34509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:30.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:30 smithi070 ceph-mon[107385]: from='client.34515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:30.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:30 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1309249859' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:30.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:30 smithi070 ceph-mon[112139]: from='client.34509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:30.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:30 smithi070 ceph-mon[112139]: from='client.34515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:30.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:30 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1309249859' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:30 smithi160 ceph-mon[101879]: from='client.34509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:30 smithi160 ceph-mon[101879]: from='client.34515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:30 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1309249859' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:31.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:31 smithi160 ceph-mon[101879]: from='client.44450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:31.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:31 smithi160 ceph-mon[101879]: pgmap v527: 521 pgs: 521 active+clean; 6.8 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 317 KiB/s rd, 654 KiB/s wr, 357 op/s; 12 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:07:31.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:31 smithi160 ceph-mon[101879]: from='client.34533 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:31.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:31 smithi070 ceph-mon[107385]: from='client.44450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:31.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:31 smithi070 ceph-mon[107385]: pgmap v527: 521 pgs: 521 active+clean; 6.8 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 317 KiB/s rd, 654 KiB/s wr, 357 op/s; 12 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:07:31.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:31 smithi070 ceph-mon[107385]: from='client.34533 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:31.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:31 smithi070 ceph-mon[112139]: from='client.44450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:31.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:31 smithi070 ceph-mon[112139]: pgmap v527: 521 pgs: 521 active+clean; 6.8 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 317 KiB/s rd, 654 KiB/s wr, 357 op/s; 12 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:07:31.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:31 smithi070 ceph-mon[112139]: from='client.34533 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:32.243 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:32.243 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:32.243 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:32.243 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:32.243 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:32.243 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:32.243 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:32.243 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:32.243 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:32.244 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:32.244 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:32.244 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:32.244 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:32.244 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:32.244 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:32.244 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:32.244 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:32.244 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:32.244 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:32.244 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:32.244 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:32.244 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:32.244 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:32.244 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:32.245 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:32.245 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:32.245 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:32.245 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:32.245 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:32.245 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:32.245 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:32.245 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:32.245 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:32.245 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:32.245 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:32.245 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:32.245 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:32.245 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:32.245 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:32.246 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:32.246 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:32.246 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:32.246 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:32.246 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:32.246 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:32.246 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:32.246 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:32.246 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:32.246 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:32.246 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:32.246 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:32.246 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:32.247 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:32.247 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:32.247 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:32.247 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:32.247 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:32.247 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:32.247 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:32.247 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:32.247 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:32.247 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:32.247 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:32.247 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:32.247 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:32.247 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:32.247 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:32.248 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:32.248 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:32.248 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:32.248 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:32.248 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:32.248 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:32.248 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:32.248 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:32.248 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:32.248 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:32.248 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:32.248 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:32.248 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:32.248 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:32.248 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:32.249 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:32.249 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:32.249 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:32.249 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:32.249 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:32.249 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:32.249 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:32.249 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:32.249 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:32.249 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:32.249 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:32.249 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:32.249 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:32.249 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:32.249 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:32.250 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:32.250 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:32.250 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:32.250 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:32.250 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:32.250 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:32.250 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:32.250 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:32.250 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:32.250 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:32.250 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:32.250 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:32.250 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:32.250 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:32.250 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:32.251 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:32.251 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:32.251 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:32.251 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:32.251 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:32.251 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:32.251 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:32.251 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:32.251 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:32.251 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:07:32.251 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:07:32.251 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:07:32.251 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:07:32.251 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 4 2024-09-21T10:07:32.251 INFO:tasks.workunit.client.0.smithi070.stdout:read: 4 2024-09-21T10:07:32.252 INFO:tasks.workunit.client.0.smithi070.stdout:read: 4 2024-09-21T10:07:32.252 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestEncryptionLUKS1 (10713 ms) 2024-09-21T10:07:32.252 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS2 2024-09-21T10:07:32.252 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:07:33.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:33 smithi070 ceph-mon[107385]: pgmap v528: 521 pgs: 521 active+clean; 11 MiB data, 528 MiB used, 715 GiB / 715 GiB avail; 1.7 MiB/s rd, 831 KiB/s wr, 887 op/s; 11 B/s, 0 keys/s, 0 objects/s recovering 2024-09-21T10:07:33.378 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:07:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:07:33] "GET /metrics HTTP/1.1" 200 67628 "" "Prometheus/2.51.0" 2024-09-21T10:07:33.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:33 smithi070 ceph-mon[112139]: pgmap v528: 521 pgs: 521 active+clean; 11 MiB data, 528 MiB used, 715 GiB / 715 GiB avail; 1.7 MiB/s rd, 831 KiB/s wr, 887 op/s; 11 B/s, 0 keys/s, 0 objects/s recovering 2024-09-21T10:07:33.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:33 smithi160 ceph-mon[101879]: pgmap v528: 521 pgs: 521 active+clean; 11 MiB data, 528 MiB used, 715 GiB / 715 GiB avail; 1.7 MiB/s rd, 831 KiB/s wr, 887 op/s; 11 B/s, 0 keys/s, 0 objects/s recovering 2024-09-21T10:07:35.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:35 smithi160 ceph-mon[101879]: pgmap v529: 521 pgs: 521 active+clean; 11 MiB data, 528 MiB used, 715 GiB / 715 GiB avail; 1.5 MiB/s rd, 734 KiB/s wr, 784 op/s; 9 B/s, 0 keys/s, 0 objects/s recovering 2024-09-21T10:07:35.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:35 smithi070 ceph-mon[107385]: pgmap v529: 521 pgs: 521 active+clean; 11 MiB data, 528 MiB used, 715 GiB / 715 GiB avail; 1.5 MiB/s rd, 734 KiB/s wr, 784 op/s; 9 B/s, 0 keys/s, 0 objects/s recovering 2024-09-21T10:07:35.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:35 smithi070 ceph-mon[112139]: pgmap v529: 521 pgs: 521 active+clean; 11 MiB data, 528 MiB used, 715 GiB / 715 GiB avail; 1.5 MiB/s rd, 734 KiB/s wr, 784 op/s; 9 B/s, 0 keys/s, 0 objects/s recovering 2024-09-21T10:07:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:36.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:36.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:36.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:36 smithi160 ceph-mon[101879]: pgmap v530: 521 pgs: 521 active+clean; 11 MiB data, 525 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 693 KiB/s wr, 742 op/s; 9 B/s, 0 keys/s, 0 objects/s recovering 2024-09-21T10:07:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:36 smithi070 ceph-mon[107385]: pgmap v530: 521 pgs: 521 active+clean; 11 MiB data, 525 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 693 KiB/s wr, 742 op/s; 9 B/s, 0 keys/s, 0 objects/s recovering 2024-09-21T10:07:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:36 smithi070 ceph-mon[112139]: pgmap v530: 521 pgs: 521 active+clean; 11 MiB data, 525 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 693 KiB/s wr, 742 op/s; 9 B/s, 0 keys/s, 0 objects/s recovering 2024-09-21T10:07:37.585 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:37 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-21T10:07:37.585 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:37 smithi070 ceph-mon[107385]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-21T10:07:37.585 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:37 smithi070 ceph-mon[107385]: Upgrade: osd.1 is safe to restart 2024-09-21T10:07:37.585 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:37 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-21T10:07:37.585 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:37 smithi070 ceph-mon[112139]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-21T10:07:37.585 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:37 smithi070 ceph-mon[112139]: Upgrade: osd.1 is safe to restart 2024-09-21T10:07:37.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:37 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-21T10:07:37.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:37 smithi160 ceph-mon[101879]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-21T10:07:37.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:37 smithi160 ceph-mon[101879]: Upgrade: osd.1 is safe to restart 2024-09-21T10:07:38.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:38 smithi070 ceph-mon[112139]: pgmap v531: 521 pgs: 521 active+clean; 11 MiB data, 525 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 226 KiB/s wr, 487 op/s 2024-09-21T10:07:38.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:38 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:38.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:38 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T10:07:38.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:38 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:07:38.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:38 smithi070 ceph-mon[107385]: pgmap v531: 521 pgs: 521 active+clean; 11 MiB data, 525 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 226 KiB/s wr, 487 op/s 2024-09-21T10:07:38.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:38 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:38.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:38 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T10:07:38.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:38 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:07:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:38 smithi160 ceph-mon[101879]: pgmap v531: 521 pgs: 521 active+clean; 11 MiB data, 525 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 226 KiB/s wr, 487 op/s 2024-09-21T10:07:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:38 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:38 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T10:07:38.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:38 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:07:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:39 smithi070 ceph-mon[107385]: Upgrade: Updating osd.1 2024-09-21T10:07:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:39 smithi070 ceph-mon[107385]: Deploying daemon osd.1 on smithi070 2024-09-21T10:07:39.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:39 smithi070 ceph-mon[112139]: Upgrade: Updating osd.1 2024-09-21T10:07:39.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:39 smithi070 ceph-mon[112139]: Deploying daemon osd.1 on smithi070 2024-09-21T10:07:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:39 smithi160 ceph-mon[101879]: Upgrade: Updating osd.1 2024-09-21T10:07:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:39 smithi160 ceph-mon[101879]: Deploying daemon osd.1 on smithi070 2024-09-21T10:07:39.888 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:07:39.888 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:07:39.888 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:07:39.888 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:07:39.888 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:07:39.889 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:39.889 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:07:39.889 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:39.889 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:39.889 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:07:39.889 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:39.889 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:07:39.889 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:39.889 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:39.889 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:07:39.890 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:39.890 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:07:39.890 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:39.890 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:39.890 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:07:39.890 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:39.890 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:07:39.890 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:39.890 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:39.890 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:07:39.890 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:39.891 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:07:39.891 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:39.891 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:39.891 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:07:39.891 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-21T10:07:39.891 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-21T10:07:39.891 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-21T10:07:39.891 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-21T10:07:39.891 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-21T10:07:39.891 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-21T10:07:39.891 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-21T10:07:39.891 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-21T10:07:39.891 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-21T10:07:39.892 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-21T10:07:39.892 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:39.892 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-21T10:07:39.892 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:39.892 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:39.892 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-21T10:07:39.892 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:39.892 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-21T10:07:39.892 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:39.892 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:39.892 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-21T10:07:39.892 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:39.892 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-21T10:07:39.893 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:39.893 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:39.893 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-21T10:07:39.893 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:39.893 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-21T10:07:39.893 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:39.893 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:39.893 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-21T10:07:39.893 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:39.893 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-21T10:07:39.893 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:39.893 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:39.893 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-21T10:07:39.893 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:39.894 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:39.894 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:39.894 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:39.894 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:39.894 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:39.894 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:07:39.894 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:07:39.894 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:07:39.894 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:39.894 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:07:39.894 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:07:39.894 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:07:39.894 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:39.895 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:07:39.895 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:07:39.895 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:07:39.895 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:39.895 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:07:39.895 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:07:39.895 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:07:39.895 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:39.895 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:07:39.895 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:07:39.895 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:07:39.895 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 512~512 = 512 2024-09-21T10:07:39.896 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:39.896 INFO:tasks.workunit.client.0.smithi070.stdout:aio discard: 1536~512 = 0 2024-09-21T10:07:39.896 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:39.896 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:39.896 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:39.896 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:39.896 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:07:39.896 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 0 2024-09-21T10:07:39.896 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:39.896 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:39.896 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 0 2024-09-21T10:07:39.896 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:39.897 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:39.897 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 16384 2024-09-21T10:07:39.897 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:39.897 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:39.897 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 16384 2024-09-21T10:07:39.897 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:39.897 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:39.897 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:39.897 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:39.897 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 49152 2024-09-21T10:07:39.897 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:39.897 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:39.897 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 49152 2024-09-21T10:07:39.897 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:39.898 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:39.898 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 65536 2024-09-21T10:07:39.898 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:39.898 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:39.898 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 65536 2024-09-21T10:07:39.898 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:39.898 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:39.898 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:39.898 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:39.898 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 98304 2024-09-21T10:07:39.898 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:39.898 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:39.898 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 98304 2024-09-21T10:07:39.899 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:39.899 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:39.899 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 114688 2024-09-21T10:07:39.899 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:39.899 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:39.899 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 114688 2024-09-21T10:07:39.899 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:39.899 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:39.899 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:39.899 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:39.899 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 147456 2024-09-21T10:07:39.899 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:39.899 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:39.900 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 147456 2024-09-21T10:07:39.900 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:39.900 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:39.900 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 163840 2024-09-21T10:07:39.900 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:39.900 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:39.900 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 163840 2024-09-21T10:07:39.900 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:39.900 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:39.900 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:39.900 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:39.900 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 196608 2024-09-21T10:07:39.900 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:39.900 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:39.900 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 196608 2024-09-21T10:07:39.901 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:39.901 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:39.901 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 212992 2024-09-21T10:07:39.901 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:39.901 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:39.901 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 212992 2024-09-21T10:07:39.901 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:39.901 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:39.901 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:39.901 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:39.901 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:39.901 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:39.902 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:39.902 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:39.902 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:39.902 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:39.902 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:39.902 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:39.902 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:39.902 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:39.903 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:39.903 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:39.903 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:39.903 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:39.903 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:39.903 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:39.903 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:39.903 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:39.903 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:39.904 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:39.904 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:39.904 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:39.904 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:39.904 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:39.904 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:39.904 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:39.904 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:39.904 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:39.905 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:39.905 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:39.905 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:39.905 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:39.905 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:39.905 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:39.905 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:39.905 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:39.905 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:39.906 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:39.906 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:39.906 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:39.906 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:39.906 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:39.906 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:39.906 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:39.906 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:39.907 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:39.907 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:39.907 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:39.907 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:40.584 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:40 smithi070 ceph-mon[107385]: pgmap v532: 521 pgs: 521 active+clean; 11 MiB data, 525 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 226 KiB/s wr, 487 op/s 2024-09-21T10:07:40.584 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:40 smithi070 ceph-mon[112139]: pgmap v532: 521 pgs: 521 active+clean; 11 MiB data, 525 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 226 KiB/s wr, 487 op/s 2024-09-21T10:07:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:40 smithi160 ceph-mon[101879]: pgmap v532: 521 pgs: 521 active+clean; 11 MiB data, 525 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 226 KiB/s wr, 487 op/s 2024-09-21T10:07:41.627 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:41 smithi070 systemd[1]: Stopping Ceph osd.1 for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:07:42.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:41 smithi160 ceph-mon[101879]: osd.1 marked itself down and dead 2024-09-21T10:07:42.127 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:41 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1[51853]: 2024-09-21T10:07:41.713+0000 7f8379b40640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:07:42.127 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:41 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1[51853]: 2024-09-21T10:07:41.713+0000 7f8379b40640 -1 osd.1 668 *** Got signal Terminated *** 2024-09-21T10:07:42.127 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:41 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1[51853]: 2024-09-21T10:07:41.713+0000 7f8379b40640 -1 osd.1 668 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:07:42.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:41 smithi070 ceph-mon[107385]: osd.1 marked itself down and dead 2024-09-21T10:07:42.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:41 smithi070 ceph-mon[112139]: osd.1 marked itself down and dead 2024-09-21T10:07:42.627 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:42 smithi070 podman[129520]: 2024-09-21 10:07:42.345793731 +0000 UTC m=+0.764515994 container died b7c0d1169d1dd9d735a32686ec7c1726f5cbbf2c2da384695e10856f5b1b4e35 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=reef-6a1bdf8, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-21T10:07:43.089 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:42 smithi070 podman[129520]: 2024-09-21 10:07:42.877808961 +0000 UTC m=+1.296531221 container cleanup b7c0d1169d1dd9d735a32686ec7c1726f5cbbf2c2da384695e10856f5b1b4e35 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , RELEASE=reef-6a1bdf8, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:07:43.089 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:42 smithi070 bash[129520]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1 2024-09-21T10:07:43.090 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:43 smithi070 podman[129575]: 2024-09-21 10:07:43.087557036 +0000 UTC m=+0.734479313 container remove b7c0d1169d1dd9d735a32686ec7c1726f5cbbf2c2da384695e10856f5b1b4e35 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=reef-6a1bdf8) 2024-09-21T10:07:43.090 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:42 smithi070 ceph-mon[107385]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:07:43.090 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:42 smithi070 ceph-mon[107385]: osdmap e669: 8 total, 7 up, 8 in 2024-09-21T10:07:43.090 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:42 smithi070 ceph-mon[107385]: osd.1 marked itself dead as of e669 2024-09-21T10:07:43.090 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:42 smithi070 ceph-mon[107385]: pgmap v534: 521 pgs: 76 stale+active+clean, 445 active+clean; 27 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 701 op/s 2024-09-21T10:07:43.090 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:42 smithi070 ceph-mon[107385]: osdmap e670: 8 total, 7 up, 8 in 2024-09-21T10:07:43.090 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:42 smithi070 ceph-mon[112139]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:07:43.090 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:42 smithi070 ceph-mon[112139]: osdmap e669: 8 total, 7 up, 8 in 2024-09-21T10:07:43.091 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:42 smithi070 ceph-mon[112139]: osd.1 marked itself dead as of e669 2024-09-21T10:07:43.091 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:42 smithi070 ceph-mon[112139]: pgmap v534: 521 pgs: 76 stale+active+clean, 445 active+clean; 27 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 701 op/s 2024-09-21T10:07:43.091 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:42 smithi070 ceph-mon[112139]: osdmap e670: 8 total, 7 up, 8 in 2024-09-21T10:07:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:42 smithi160 ceph-mon[101879]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:07:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:42 smithi160 ceph-mon[101879]: osdmap e669: 8 total, 7 up, 8 in 2024-09-21T10:07:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:42 smithi160 ceph-mon[101879]: osd.1 marked itself dead as of e669 2024-09-21T10:07:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:42 smithi160 ceph-mon[101879]: pgmap v534: 521 pgs: 76 stale+active+clean, 445 active+clean; 27 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 701 op/s 2024-09-21T10:07:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:42 smithi160 ceph-mon[101879]: osdmap e670: 8 total, 7 up, 8 in 2024-09-21T10:07:43.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:07:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:07:43] "GET /metrics HTTP/1.1" 200 67774 "" "Prometheus/2.51.0" 2024-09-21T10:07:43.877 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:43 smithi070 podman[129721]: 2024-09-21 10:07:43.492129506 +0000 UTC m=+0.021019525 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:07:43.878 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:43 smithi070 podman[129721]: 2024-09-21 10:07:43.613447479 +0000 UTC m=+0.142337480 container create 4b2ea9823f91f0860ff4522db24a446a8f7efc864be7b1c92a434082e338b12b (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=main-62da54b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-21T10:07:44.377 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:43 smithi070 podman[129721]: 2024-09-21 10:07:43.883069994 +0000 UTC m=+0.411960008 container init 4b2ea9823f91f0860ff4522db24a446a8f7efc864be7b1c92a434082e338b12b (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, org.label-schema.build-date=20240731, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True) 2024-09-21T10:07:44.378 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:43 smithi070 podman[129721]: 2024-09-21 10:07:43.890540353 +0000 UTC m=+0.419430362 container start 4b2ea9823f91f0860ff4522db24a446a8f7efc864be7b1c92a434082e338b12b (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=main-62da54b, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-21T10:07:44.378 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:43 smithi070 podman[129721]: 2024-09-21 10:07:43.965882323 +0000 UTC m=+0.494772326 container attach 4b2ea9823f91f0860ff4522db24a446a8f7efc864be7b1c92a434082e338b12b (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, ceph=True, GIT_CLEAN=True, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:07:44.378 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:44 smithi070 podman[129721]: 2024-09-21 10:07:44.126826266 +0000 UTC m=+0.655716279 container died 4b2ea9823f91f0860ff4522db24a446a8f7efc864be7b1c92a434082e338b12b (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=main-62da54b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-21T10:07:45.148 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:44 smithi070 podman[129783]: 2024-09-21 10:07:44.886644046 +0000 UTC m=+0.754070794 container remove 4b2ea9823f91f0860ff4522db24a446a8f7efc864be7b1c92a434082e338b12b (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True) 2024-09-21T10:07:45.149 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:44 smithi070 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.1.service: Deactivated successfully. 2024-09-21T10:07:45.149 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:44 smithi070 systemd[1]: Stopped Ceph osd.1 for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:07:45.149 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:44 smithi070 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.1.service: Consumed 27.361s CPU time. 2024-09-21T10:07:45.149 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:45 smithi070 systemd[1]: Starting Ceph osd.1 for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:07:45.149 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:45 smithi070 ceph-mon[107385]: pgmap v536: 521 pgs: 76 stale+active+clean, 445 active+clean; 27 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.4 MiB/s wr, 872 op/s 2024-09-21T10:07:45.149 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:45 smithi070 ceph-mon[112139]: pgmap v536: 521 pgs: 76 stale+active+clean, 445 active+clean; 27 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.4 MiB/s wr, 872 op/s 2024-09-21T10:07:45.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:45 smithi160 ceph-mon[101879]: pgmap v536: 521 pgs: 76 stale+active+clean, 445 active+clean; 27 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.4 MiB/s wr, 872 op/s 2024-09-21T10:07:45.492 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:45 smithi070 podman[129942]: 2024-09-21 10:07:45.27710626 +0000 UTC m=+0.016810698 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:07:45.492 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:45 smithi070 podman[129942]: 2024-09-21 10:07:45.379132155 +0000 UTC m=+0.118836579 container create 41ebe9d1d330747a365333724ab5eb776158c68fb96ec6d2489a5faae532af1f (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1-activate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-21T10:07:45.769 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:45 smithi070 podman[129942]: 2024-09-21 10:07:45.60712806 +0000 UTC m=+0.346832507 container init 41ebe9d1d330747a365333724ab5eb776158c68fb96ec6d2489a5faae532af1f (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1-activate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-21T10:07:45.769 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:45 smithi070 podman[129942]: 2024-09-21 10:07:45.614951208 +0000 UTC m=+0.354655638 container start 41ebe9d1d330747a365333724ab5eb776158c68fb96ec6d2489a5faae532af1f (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1-activate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True) 2024-09-21T10:07:45.769 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:45 smithi070 podman[129942]: 2024-09-21 10:07:45.714895941 +0000 UTC m=+0.454600367 container attach 41ebe9d1d330747a365333724ab5eb776158c68fb96ec6d2489a5faae532af1f (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1-activate, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-21T10:07:45.815 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:45.815 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:45.815 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:45.816 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:45.816 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:45.816 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:45.816 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:45.816 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:45.816 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:45.816 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:45.816 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:45.816 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:45.816 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:45.816 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:45.816 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:45.817 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:45.817 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:45.817 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:45.817 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:45.817 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:45.817 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:45.817 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:45.817 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:45.817 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:45.817 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:45.817 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:45.817 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:45.817 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:45.817 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:45.818 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:45.818 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:45.818 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:45.818 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:45.818 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:45.818 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:45.818 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:45.818 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:45.818 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:45.818 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:45.818 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:45.818 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:45.818 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:45.818 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:45.819 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:45.819 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:45.819 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:45.819 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:45.819 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:45.819 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:45.819 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:45.819 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:45.819 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:45.819 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:45.819 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:45.819 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:45.819 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:45.820 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:45.820 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:45.820 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:45.820 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:45.820 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:45.820 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:45.820 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:45.820 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:45.820 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:45.820 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:45.820 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:45.820 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:45.820 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:45.821 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:45.821 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:45.821 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:45.821 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:45.821 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:45.821 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:45.821 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:45.821 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:45.821 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:45.821 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:45.821 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:45.821 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:45.821 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:45.821 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:45.821 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:45.822 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:45.822 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:45.822 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:45.822 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:45.822 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:45.822 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:45.822 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:45.822 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:45.822 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:45.822 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:45.822 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:45.822 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:45.823 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:45.823 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:45.823 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:45.823 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:45.823 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:45.823 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:45.823 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:45.823 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:45.823 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:45.823 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:45.823 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:45.823 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:45.823 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:45.823 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:45.824 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:07:45.824 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:07:45.824 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:07:45.824 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:07:45.824 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:07:45.824 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:45.824 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:45.824 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:45.824 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:07:45.824 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:07:45.824 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:07:45.824 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:07:45.824 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:07:45.825 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:07:45.825 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:07:45.825 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 4 2024-09-21T10:07:45.825 INFO:tasks.workunit.client.0.smithi070.stdout:read: 4 2024-09-21T10:07:45.825 INFO:tasks.workunit.client.0.smithi070.stdout:read: 4 2024-09-21T10:07:45.825 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestEncryptionLUKS2 (13573 ms) 2024-09-21T10:07:45.825 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCloneEncryption 2024-09-21T10:07:45.825 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:07:46.127 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:45 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1-activate[129969]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:07:46.127 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:45 smithi070 bash[129942]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:07:46.127 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:45 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1-activate[129969]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:07:46.127 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:45 smithi070 bash[129942]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:07:46.532 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:46 smithi070 ceph-mon[112139]: Health check failed: Degraded data redundancy: 87/672 objects degraded (12.946%), 53 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:46.532 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:46 smithi070 ceph-mon[112139]: osdmap e671: 8 total, 7 up, 8 in 2024-09-21T10:07:46.533 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:46 smithi070 ceph-mon[107385]: Health check failed: Degraded data redundancy: 87/672 objects degraded (12.946%), 53 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:46.533 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:46 smithi070 ceph-mon[107385]: osdmap e671: 8 total, 7 up, 8 in 2024-09-21T10:07:46.533 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:46 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1-activate[129969]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-21T10:07:46.533 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:46 smithi070 bash[129942]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-21T10:07:46.533 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:46 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1-activate[129969]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-21T10:07:46.533 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:46 smithi070 bash[129942]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-21T10:07:46.533 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:46 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1-activate[129969]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-21T10:07:46.533 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:46 smithi070 bash[129942]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-21T10:07:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:46 smithi160 ceph-mon[101879]: Health check failed: Degraded data redundancy: 87/672 objects degraded (12.946%), 53 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:46 smithi160 ceph-mon[101879]: osdmap e671: 8 total, 7 up, 8 in 2024-09-21T10:07:46.877 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:46 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1-activate[129969]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-21T10:07:46.877 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:46 smithi070 bash[129942]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-21T10:07:46.877 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:46 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1-activate[129969]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-21T10:07:46.877 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:46 smithi070 bash[129942]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-21T10:07:46.878 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:46 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1-activate[129969]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-21T10:07:46.878 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:46 smithi070 bash[129942]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-21T10:07:46.878 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:46 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1-activate[129969]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-21T10:07:46.878 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:46 smithi070 bash[129942]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-21T10:07:46.878 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:46 smithi070 podman[129942]: 2024-09-21 10:07:46.558355297 +0000 UTC m=+1.298059732 container died 41ebe9d1d330747a365333724ab5eb776158c68fb96ec6d2489a5faae532af1f (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1-activate, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-09-21T10:07:47.320 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:47 smithi070 ceph-mon[107385]: pgmap v537: 521 pgs: 135 active+undersized, 53 active+undersized+degraded, 333 active+clean; 31 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 4.9 MiB/s rd, 2.5 MiB/s wr, 1.06k op/s; 87/672 objects degraded (12.946%) 2024-09-21T10:07:47.320 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:47 smithi070 ceph-mon[112139]: pgmap v537: 521 pgs: 135 active+undersized, 53 active+undersized+degraded, 333 active+clean; 31 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 4.9 MiB/s rd, 2.5 MiB/s wr, 1.06k op/s; 87/672 objects degraded (12.946%) 2024-09-21T10:07:47.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:47 smithi160 ceph-mon[101879]: pgmap v537: 521 pgs: 135 active+undersized, 53 active+undersized+degraded, 333 active+clean; 31 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 4.9 MiB/s rd, 2.5 MiB/s wr, 1.06k op/s; 87/672 objects degraded (12.946%) 2024-09-21T10:07:47.627 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:47 smithi070 podman[130138]: 2024-09-21 10:07:47.318283157 +0000 UTC m=+0.754568471 container remove 41ebe9d1d330747a365333724ab5eb776158c68fb96ec6d2489a5faae532af1f (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, RELEASE=main-62da54b, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2) 2024-09-21T10:07:47.627 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:47 smithi070 podman[130201]: 2024-09-21 10:07:47.424255542 +0000 UTC m=+0.016808935 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:07:47.628 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:47 smithi070 podman[130201]: 2024-09-21 10:07:47.527685424 +0000 UTC m=+0.120255596 container create 853f3bb7fa25dc85edf4d6051f583ebb6d21c87e669766fed690887303f3d031 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-21T10:07:48.127 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:47 smithi070 podman[130201]: 2024-09-21 10:07:47.872039782 +0000 UTC m=+0.464593190 container init 853f3bb7fa25dc85edf4d6051f583ebb6d21c87e669766fed690887303f3d031 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-21T10:07:48.127 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:47 smithi070 podman[130201]: 2024-09-21 10:07:47.878475324 +0000 UTC m=+0.471028707 container start 853f3bb7fa25dc85edf4d6051f583ebb6d21c87e669766fed690887303f3d031 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True) 2024-09-21T10:07:48.128 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:47 smithi070 ceph-osd[130238]: -- 172.21.15.70:0/4276872858 <== mon.2 v2:172.21.15.70:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55905eaefa40 con 0x55905f8be000 2024-09-21T10:07:48.128 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:47 smithi070 bash[130201]: 853f3bb7fa25dc85edf4d6051f583ebb6d21c87e669766fed690887303f3d031 2024-09-21T10:07:48.128 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:47 smithi070 systemd[1]: Started Ceph osd.1 for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:07:49.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:49 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:49.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:49 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:49.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:49 smithi160 ceph-mon[101879]: pgmap v539: 521 pgs: 135 active+undersized, 53 active+undersized+degraded, 333 active+clean; 31 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.7 MiB/s rd, 133 KiB/s wr, 246 op/s; 87/672 objects degraded (12.946%) 2024-09-21T10:07:49.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:49 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:49.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:49 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:49.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:49 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:49.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:49 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:49.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:49 smithi070 ceph-mon[107385]: pgmap v539: 521 pgs: 135 active+undersized, 53 active+undersized+degraded, 333 active+clean; 31 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.7 MiB/s rd, 133 KiB/s wr, 246 op/s; 87/672 objects degraded (12.946%) 2024-09-21T10:07:49.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:49 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:49.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:49 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:49.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:49 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:49.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:49 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:49.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:49 smithi070 ceph-mon[112139]: pgmap v539: 521 pgs: 135 active+undersized, 53 active+undersized+degraded, 333 active+clean; 31 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.7 MiB/s rd, 133 KiB/s wr, 246 op/s; 87/672 objects degraded (12.946%) 2024-09-21T10:07:49.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:49 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:49.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:49 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:50.127 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:49 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1[130215]: 2024-09-21T10:07:49.779+0000 7eff7e54a640 -1 Falling back to public interface 2024-09-21T10:07:51.353 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:51 smithi070 ceph-mon[107385]: pgmap v540: 521 pgs: 135 active+undersized, 53 active+undersized+degraded, 333 active+clean; 31 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 105 KiB/s wr, 194 op/s; 89/678 objects degraded (13.127%) 2024-09-21T10:07:51.353 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:51.353 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:51.353 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:51 smithi070 ceph-mon[112139]: pgmap v540: 521 pgs: 135 active+undersized, 53 active+undersized+degraded, 333 active+clean; 31 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 105 KiB/s wr, 194 op/s; 89/678 objects degraded (13.127%) 2024-09-21T10:07:51.353 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:51.353 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:51 smithi160 ceph-mon[101879]: pgmap v540: 521 pgs: 135 active+undersized, 53 active+undersized+degraded, 333 active+clean; 31 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 105 KiB/s wr, 194 op/s; 89/678 objects degraded (13.127%) 2024-09-21T10:07:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:52.849 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:52 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:52.849 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:52 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:52.849 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:52 smithi070 ceph-mon[112139]: pgmap v541: 521 pgs: 135 active+undersized, 53 active+undersized+degraded, 333 active+clean; 35 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 534 KiB/s wr, 192 op/s; 92/696 objects degraded (13.218%) 2024-09-21T10:07:52.850 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:52 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:52.850 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:52 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:52.850 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:52 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:52.850 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:52 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:52.850 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:52 smithi070 ceph-mon[107385]: pgmap v541: 521 pgs: 135 active+undersized, 53 active+undersized+degraded, 333 active+clean; 35 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 534 KiB/s wr, 192 op/s; 92/696 objects degraded (13.218%) 2024-09-21T10:07:52.850 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:52 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:52.850 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:52 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:53.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:52 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:53.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:52 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:53.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:52 smithi160 ceph-mon[101879]: pgmap v541: 521 pgs: 135 active+undersized, 53 active+undersized+degraded, 333 active+clean; 35 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 MiB/s rd, 534 KiB/s wr, 192 op/s; 92/696 objects degraded (13.218%) 2024-09-21T10:07:53.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:52 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:53.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:52 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:53.457 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:07:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:07:53] "GET /metrics HTTP/1.1" 200 67778 "" "Prometheus/2.51.0" 2024-09-21T10:07:53.866 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1[130215]: 2024-09-21T10:07:53.455+0000 7eff7e54a640 -1 osd.1 669 log_to_monitors true 2024-09-21T10:07:53.866 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:53 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 92/696 objects degraded (13.218%), 53 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:53.866 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:53 smithi070 ceph-mon[107385]: osdmap e672: 8 total, 7 up, 8 in 2024-09-21T10:07:53.866 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:53 smithi070 ceph-mon[107385]: from='osd.1 [v2:172.21.15.70:6810/3938187175,v1:172.21.15.70:6811/3938187175]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T10:07:53.867 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:53 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 92/696 objects degraded (13.218%), 53 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:53.867 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:53 smithi070 ceph-mon[112139]: osdmap e672: 8 total, 7 up, 8 in 2024-09-21T10:07:53.867 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:53 smithi070 ceph-mon[112139]: from='osd.1 [v2:172.21.15.70:6810/3938187175,v1:172.21.15.70:6811/3938187175]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T10:07:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:53 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 92/696 objects degraded (13.218%), 53 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:53 smithi160 ceph-mon[101879]: osdmap e672: 8 total, 7 up, 8 in 2024-09-21T10:07:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:53 smithi160 ceph-mon[101879]: from='osd.1 [v2:172.21.15.70:6810/3938187175,v1:172.21.15.70:6811/3938187175]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T10:07:54.127 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 10:07:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1[130215]: 2024-09-21T10:07:53.934+0000 7eff722ec640 -1 osd.1 669 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T10:07:54.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:54 smithi070 ceph-mon[107385]: from='osd.1 [v2:172.21.15.70:6810/3938187175,v1:172.21.15.70:6811/3938187175]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T10:07:54.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:54 smithi070 ceph-mon[107385]: osdmap e673: 8 total, 7 up, 8 in 2024-09-21T10:07:54.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:54 smithi070 ceph-mon[107385]: from='osd.1 [v2:172.21.15.70:6810/3938187175,v1:172.21.15.70:6811/3938187175]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]} : dispatch 2024-09-21T10:07:54.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:54 smithi070 ceph-mon[107385]: from='osd.1 [v2:172.21.15.70:6810/3938187175,v1:172.21.15.70:6811/3938187175]' entity='osd.1' 2024-09-21T10:07:54.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:54 smithi070 ceph-mon[107385]: pgmap v544: 521 pgs: 135 active+undersized, 53 active+undersized+degraded, 333 active+clean; 35 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 517 KiB/s wr, 30 op/s; 92/696 objects degraded (13.218%) 2024-09-21T10:07:54.882 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:54 smithi070 ceph-mon[112139]: from='osd.1 [v2:172.21.15.70:6810/3938187175,v1:172.21.15.70:6811/3938187175]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T10:07:54.882 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:54 smithi070 ceph-mon[112139]: osdmap e673: 8 total, 7 up, 8 in 2024-09-21T10:07:54.882 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:54 smithi070 ceph-mon[112139]: from='osd.1 [v2:172.21.15.70:6810/3938187175,v1:172.21.15.70:6811/3938187175]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]} : dispatch 2024-09-21T10:07:54.883 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:54 smithi070 ceph-mon[112139]: from='osd.1 [v2:172.21.15.70:6810/3938187175,v1:172.21.15.70:6811/3938187175]' entity='osd.1' 2024-09-21T10:07:54.883 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:54 smithi070 ceph-mon[112139]: pgmap v544: 521 pgs: 135 active+undersized, 53 active+undersized+degraded, 333 active+clean; 35 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 517 KiB/s wr, 30 op/s; 92/696 objects degraded (13.218%) 2024-09-21T10:07:55.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:54 smithi160 ceph-mon[101879]: from='osd.1 [v2:172.21.15.70:6810/3938187175,v1:172.21.15.70:6811/3938187175]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T10:07:55.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:54 smithi160 ceph-mon[101879]: osdmap e673: 8 total, 7 up, 8 in 2024-09-21T10:07:55.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:54 smithi160 ceph-mon[101879]: from='osd.1 [v2:172.21.15.70:6810/3938187175,v1:172.21.15.70:6811/3938187175]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]} : dispatch 2024-09-21T10:07:55.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:54 smithi160 ceph-mon[101879]: from='osd.1 [v2:172.21.15.70:6810/3938187175,v1:172.21.15.70:6811/3938187175]' entity='osd.1' 2024-09-21T10:07:55.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:54 smithi160 ceph-mon[101879]: pgmap v544: 521 pgs: 135 active+undersized, 53 active+undersized+degraded, 333 active+clean; 35 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 517 KiB/s wr, 30 op/s; 92/696 objects degraded (13.218%) 2024-09-21T10:07:55.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:07:55 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T10:07:55.787+0000 7fd088d1d640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (65 PGs are or would become offline) 2024-09-21T10:07:55.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:55 smithi070 ceph-mon[107385]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:07:55.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:55 smithi070 ceph-mon[107385]: osd.1 [v2:172.21.15.70:6810/3938187175,v1:172.21.15.70:6811/3938187175] boot 2024-09-21T10:07:55.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:55 smithi070 ceph-mon[107385]: osdmap e674: 8 total, 8 up, 8 in 2024-09-21T10:07:55.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:55 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T10:07:55.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:55 smithi070 ceph-mon[112139]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:07:55.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:55 smithi070 ceph-mon[112139]: osd.1 [v2:172.21.15.70:6810/3938187175,v1:172.21.15.70:6811/3938187175] boot 2024-09-21T10:07:55.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:55 smithi070 ceph-mon[112139]: osdmap e674: 8 total, 8 up, 8 in 2024-09-21T10:07:55.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:55 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T10:07:56.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:55 smithi160 ceph-mon[101879]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:07:56.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:55 smithi160 ceph-mon[101879]: osd.1 [v2:172.21.15.70:6810/3938187175,v1:172.21.15.70:6811/3938187175] boot 2024-09-21T10:07:56.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:55 smithi160 ceph-mon[101879]: osdmap e674: 8 total, 8 up, 8 in 2024-09-21T10:07:56.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:55 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T10:07:57.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:56 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:57.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:56 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:57.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:56 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:07:57.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:56 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:07:57.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:56 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:57.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:56 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:57.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:56 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:57.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:56 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:57.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:56 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:57.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:56 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T10:07:57.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:56 smithi160 ceph-mon[101879]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T10:07:57.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:56 smithi160 ceph-mon[101879]: Upgrade: unsafe to stop osd(s) at this time (65 PGs are or would become offline) 2024-09-21T10:07:57.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:56 smithi160 ceph-mon[101879]: osdmap e675: 8 total, 8 up, 8 in 2024-09-21T10:07:57.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:56 smithi160 ceph-mon[101879]: pgmap v547: 521 pgs: 2 activating+degraded, 101 activating, 58 active+undersized, 27 active+undersized+degraded, 333 active+clean; 35 MiB data, 566 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.0 KiB/s wr, 58 op/s; 55/708 objects degraded (7.768%) 2024-09-21T10:07:57.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:57.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:57.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:07:57.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:07:57.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:57.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:57.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:57.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:57.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:57.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T10:07:57.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[112139]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T10:07:57.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[112139]: Upgrade: unsafe to stop osd(s) at this time (65 PGs are or would become offline) 2024-09-21T10:07:57.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[112139]: osdmap e675: 8 total, 8 up, 8 in 2024-09-21T10:07:57.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[112139]: pgmap v547: 521 pgs: 2 activating+degraded, 101 activating, 58 active+undersized, 27 active+undersized+degraded, 333 active+clean; 35 MiB data, 566 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.0 KiB/s wr, 58 op/s; 55/708 objects degraded (7.768%) 2024-09-21T10:07:57.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:57.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:57.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:07:57.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:07:57.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:07:57.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:57.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:57.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:57.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:57.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T10:07:57.130 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[107385]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T10:07:57.130 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[107385]: Upgrade: unsafe to stop osd(s) at this time (65 PGs are or would become offline) 2024-09-21T10:07:57.130 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[107385]: osdmap e675: 8 total, 8 up, 8 in 2024-09-21T10:07:57.130 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:56 smithi070 ceph-mon[107385]: pgmap v547: 521 pgs: 2 activating+degraded, 101 activating, 58 active+undersized, 27 active+undersized+degraded, 333 active+clean; 35 MiB data, 566 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.0 KiB/s wr, 58 op/s; 55/708 objects degraded (7.768%) 2024-09-21T10:07:58.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:57 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 55/708 objects degraded (7.768%), 29 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:58.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:57 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 55/708 objects degraded (7.768%), 29 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:58.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:57 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 55/708 objects degraded (7.768%), 29 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:59.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:07:58 smithi160 ceph-mon[101879]: pgmap v548: 521 pgs: 2 activating+degraded, 101 activating, 58 active+undersized, 27 active+undersized+degraded, 333 active+clean; 35 MiB data, 566 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 2.2 KiB/s wr, 43 op/s; 55/708 objects degraded (7.768%) 2024-09-21T10:07:59.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:07:58 smithi070 ceph-mon[107385]: pgmap v548: 521 pgs: 2 activating+degraded, 101 activating, 58 active+undersized, 27 active+undersized+degraded, 333 active+clean; 35 MiB data, 566 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 2.2 KiB/s wr, 43 op/s; 55/708 objects degraded (7.768%) 2024-09-21T10:07:59.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:07:58 smithi070 ceph-mon[112139]: pgmap v548: 521 pgs: 2 activating+degraded, 101 activating, 58 active+undersized, 27 active+undersized+degraded, 333 active+clean; 35 MiB data, 566 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 2.2 KiB/s wr, 43 op/s; 55/708 objects degraded (7.768%) 2024-09-21T10:08:00.458 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-09-21T10:08:01.083 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T10:08:01.083 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (8m) 9s ago 14m 19.3M - 0.27.0 11f11916f8cd 89ae95182419 2024-09-21T10:08:01.083 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi160 *:3000 running (8m) 62s ago 13m 81.2M - 10.4.8 5aad1d7cf004 daa2d8dfab48 2024-09-21T10:08:01.083 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi070.gansda smithi070 running (13m) 9s ago 13m 23.7M - 18.2.4-871-g6a1bdf85 0adab276c97e 6016b2501c8a 2024-09-21T10:08:01.084 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi160.byyhfz smithi160 running (12m) 62s ago 12m 17.4M - 18.2.4-871-g6a1bdf85 0adab276c97e 93e108d91e65 2024-09-21T10:08:01.084 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi160 *:8443,9283,8765 running (9m) 62s ago 18m 458M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 d952652a9683 2024-09-21T10:08:01.084 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (9m) 9s ago 19m 534M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 e3d390893cff 2024-09-21T10:08:01.084 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (7m) 9s ago 20m 75.4M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 4a7b667a4033 2024-09-21T10:08:01.084 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi160 running (5m) 62s ago 18m 46.4M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 86e88af21f36 2024-09-21T10:08:01.084 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (6m) 9s ago 18m 60.9M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 e154380b80a2 2024-09-21T10:08:01.084 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (9m) 9s ago 14m 10.5M - 1.7.0 72c9c2088986 5d5d4b026070 2024-09-21T10:08:01.084 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi160 *:9100 running (9m) 62s ago 14m 10.2M - 1.7.0 72c9c2088986 19b8e0e8ee0c 2024-09-21T10:08:01.084 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (47s) 9s ago 17m 54.3M 2480M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 eedacb586f7c 2024-09-21T10:08:01.084 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (13s) 9s ago 17m 11.3M 2480M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 853f3bb7fa25 2024-09-21T10:08:01.084 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (17m) 9s ago 17m 148M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 58a11b8ddb81 2024-09-21T10:08:01.084 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (16m) 9s ago 16m 139M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e aa0c8a732a41 2024-09-21T10:08:01.084 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi160 running (16m) 62s ago 16m 124M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 7325a075c488 2024-09-21T10:08:01.085 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi160 running (15m) 62s ago 15m 117M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e f74026345641 2024-09-21T10:08:01.085 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi160 running (15m) 62s ago 15m 143M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 03e6bb54f8b2 2024-09-21T10:08:01.085 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi160 running (15m) 62s ago 15m 126M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e e405b21a6146 2024-09-21T10:08:01.085 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi160 *:9095 running (8m) 62s ago 14m 58.1M - 2.51.0 1d3b7f56885b 648952e57ad7 2024-09-21T10:08:01.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:01 smithi070 ceph-mon[107385]: pgmap v549: 521 pgs: 2 activating+degraded, 101 activating, 418 active+clean; 35 MiB data, 571 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2.2 KiB/s wr, 40 op/s; 3/711 objects degraded (0.422%); 18 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:08:01.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:01 smithi070 ceph-mon[112139]: pgmap v549: 521 pgs: 2 activating+degraded, 101 activating, 418 active+clean; 35 MiB data, 571 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2.2 KiB/s wr, 40 op/s; 3/711 objects degraded (0.422%); 18 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:08:01.449 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:08:01.449 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-09-21T10:08:01.449 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 3 2024-09-21T10:08:01.449 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:08:01.450 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-09-21T10:08:01.450 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T10:08:01.450 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:08:01.450 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-09-21T10:08:01.450 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 6, 2024-09-21T10:08:01.450 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T10:08:01.451 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:08:01.451 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": { 2024-09-21T10:08:01.451 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 2 2024-09-21T10:08:01.451 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:08:01.451 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-09-21T10:08:01.451 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 8, 2024-09-21T10:08:01.452 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 7 2024-09-21T10:08:01.452 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:08:01.452 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:08:01.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:01 smithi160 ceph-mon[101879]: pgmap v549: 521 pgs: 2 activating+degraded, 101 activating, 418 active+clean; 35 MiB data, 571 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2.2 KiB/s wr, 40 op/s; 3/711 objects degraded (0.422%); 18 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:08:01.808 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:08:01.808 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47", 2024-09-21T10:08:01.808 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-09-21T10:08:01.808 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 4 daemons (2 remaining).", 2024-09-21T10:08:01.808 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-09-21T10:08:01.808 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "2/8 daemons upgraded", 2024-09-21T10:08:01.808 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading osd daemons", 2024-09-21T10:08:01.809 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-09-21T10:08:01.809 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:08:02.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:02 smithi070 ceph-mon[107385]: from='client.34545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:02.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:02 smithi070 ceph-mon[107385]: from='client.34551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:02.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:02 smithi070 ceph-mon[107385]: from='client.34557 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:02.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:02 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3159056135' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:02.111 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:02 smithi070 ceph-mon[112139]: from='client.34545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:02.111 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:02 smithi070 ceph-mon[112139]: from='client.34551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:02.111 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:02 smithi070 ceph-mon[112139]: from='client.34557 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:02.111 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:02 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3159056135' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:02 smithi160 ceph-mon[101879]: from='client.34545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:02 smithi160 ceph-mon[101879]: from='client.34551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:02 smithi160 ceph-mon[101879]: from='client.34557 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:02.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:02 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3159056135' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:03.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:08:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:08:03] "GET /metrics HTTP/1.1" 200 67778 "" "Prometheus/2.51.0" 2024-09-21T10:08:03.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:03 smithi070 ceph-mon[107385]: from='client.34569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:03.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:03 smithi070 ceph-mon[107385]: pgmap v550: 521 pgs: 521 active+clean; 35 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 1.7 KiB/s wr, 32 op/s; 21 B/s, 3 keys/s, 0 objects/s recovering 2024-09-21T10:08:03.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:03 smithi070 ceph-mon[107385]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/711 objects degraded (0.422%), 2 pgs degraded) 2024-09-21T10:08:03.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:03 smithi070 ceph-mon[112139]: from='client.34569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:03.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:03 smithi070 ceph-mon[112139]: pgmap v550: 521 pgs: 521 active+clean; 35 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 1.7 KiB/s wr, 32 op/s; 21 B/s, 3 keys/s, 0 objects/s recovering 2024-09-21T10:08:03.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:03 smithi070 ceph-mon[112139]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/711 objects degraded (0.422%), 2 pgs degraded) 2024-09-21T10:08:03.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:03 smithi160 ceph-mon[101879]: from='client.34569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:03.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:03 smithi160 ceph-mon[101879]: pgmap v550: 521 pgs: 521 active+clean; 35 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 1.7 KiB/s wr, 32 op/s; 21 B/s, 3 keys/s, 0 objects/s recovering 2024-09-21T10:08:03.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:03 smithi160 ceph-mon[101879]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/711 objects degraded (0.422%), 2 pgs degraded) 2024-09-21T10:08:05.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:05 smithi070 ceph-mon[107385]: pgmap v551: 521 pgs: 521 active+clean; 43 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 901 KiB/s wr, 28 op/s; 18 B/s, 3 keys/s, 0 objects/s recovering 2024-09-21T10:08:05.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:05 smithi070 ceph-mon[112139]: pgmap v551: 521 pgs: 521 active+clean; 43 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 901 KiB/s wr, 28 op/s; 18 B/s, 3 keys/s, 0 objects/s recovering 2024-09-21T10:08:05.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:05 smithi160 ceph-mon[101879]: pgmap v551: 521 pgs: 521 active+clean; 43 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 901 KiB/s wr, 28 op/s; 18 B/s, 3 keys/s, 0 objects/s recovering 2024-09-21T10:08:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:06 smithi160 ceph-mon[101879]: pgmap v552: 521 pgs: 521 active+clean; 51 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 1.6 MiB/s wr, 29 op/s; 16 B/s, 2 keys/s, 0 objects/s recovering 2024-09-21T10:08:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:06 smithi070 ceph-mon[107385]: pgmap v552: 521 pgs: 521 active+clean; 51 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 1.6 MiB/s wr, 29 op/s; 16 B/s, 2 keys/s, 0 objects/s recovering 2024-09-21T10:08:06.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:06.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:06.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:06 smithi070 ceph-mon[112139]: pgmap v552: 521 pgs: 521 active+clean; 51 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 1.6 MiB/s wr, 29 op/s; 16 B/s, 2 keys/s, 0 objects/s recovering 2024-09-21T10:08:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:08 smithi160 ceph-mon[101879]: osdmap e676: 8 total, 8 up, 8 in 2024-09-21T10:08:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:08 smithi160 ceph-mon[101879]: pgmap v554: 521 pgs: 521 active+clean; 51 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 1.6 MiB/s wr, 6 op/s; 16 B/s, 2 keys/s, 0 objects/s recovering 2024-09-21T10:08:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:08 smithi070 ceph-mon[107385]: osdmap e676: 8 total, 8 up, 8 in 2024-09-21T10:08:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:08 smithi070 ceph-mon[107385]: pgmap v554: 521 pgs: 521 active+clean; 51 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 1.6 MiB/s wr, 6 op/s; 16 B/s, 2 keys/s, 0 objects/s recovering 2024-09-21T10:08:08.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:08 smithi070 ceph-mon[112139]: osdmap e676: 8 total, 8 up, 8 in 2024-09-21T10:08:08.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:08 smithi070 ceph-mon[112139]: pgmap v554: 521 pgs: 521 active+clean; 51 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 1.6 MiB/s wr, 6 op/s; 16 B/s, 2 keys/s, 0 objects/s recovering 2024-09-21T10:08:11.370 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:11 smithi070 ceph-mon[107385]: pgmap v555: 521 pgs: 521 active+clean; 51 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 1.6 MiB/s wr, 17 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:08:11.370 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:11 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:11.370 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:11 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T10:08:11.370 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:11 smithi070 ceph-mon[112139]: pgmap v555: 521 pgs: 521 active+clean; 51 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 1.6 MiB/s wr, 17 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:08:11.370 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:11 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:11.371 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:11 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T10:08:11.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:11 smithi160 ceph-mon[101879]: pgmap v555: 521 pgs: 521 active+clean; 51 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 1.6 MiB/s wr, 17 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:08:11.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:11 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:11.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:11 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T10:08:12.285 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:12 smithi070 ceph-mon[107385]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T10:08:12.285 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:12 smithi070 ceph-mon[107385]: Upgrade: osd.2 is safe to restart 2024-09-21T10:08:12.285 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:12 smithi070 ceph-mon[112139]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T10:08:12.285 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:12 smithi070 ceph-mon[112139]: Upgrade: osd.2 is safe to restart 2024-09-21T10:08:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:12 smithi160 ceph-mon[101879]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T10:08:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:12 smithi160 ceph-mon[101879]: Upgrade: osd.2 is safe to restart 2024-09-21T10:08:13.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:08:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:08:13] "GET /metrics HTTP/1.1" 200 67765 "" "Prometheus/2.51.0" 2024-09-21T10:08:13.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:13 smithi070 ceph-mon[107385]: pgmap v556: 521 pgs: 521 active+clean; 55 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 2.0 MiB/s wr, 40 op/s 2024-09-21T10:08:13.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:13 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:13.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:13 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T10:08:13.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:13 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:08:13.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:13 smithi070 ceph-mon[112139]: pgmap v556: 521 pgs: 521 active+clean; 55 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 2.0 MiB/s wr, 40 op/s 2024-09-21T10:08:13.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:13 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:13.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:13 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T10:08:13.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:13 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:08:13.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:13 smithi160 ceph-mon[101879]: pgmap v556: 521 pgs: 521 active+clean; 55 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 2.0 MiB/s wr, 40 op/s 2024-09-21T10:08:13.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:13 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:13.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:13 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T10:08:13.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:13 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:08:14.306 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:14 smithi070 ceph-mon[107385]: Upgrade: Updating osd.2 2024-09-21T10:08:14.306 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:14 smithi070 ceph-mon[107385]: Deploying daemon osd.2 on smithi070 2024-09-21T10:08:14.307 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:14 smithi070 ceph-mon[112139]: Upgrade: Updating osd.2 2024-09-21T10:08:14.307 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:14 smithi070 ceph-mon[112139]: Deploying daemon osd.2 on smithi070 2024-09-21T10:08:14.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:14 smithi160 ceph-mon[101879]: Upgrade: Updating osd.2 2024-09-21T10:08:14.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:14 smithi160 ceph-mon[101879]: Deploying daemon osd.2 on smithi070 2024-09-21T10:08:15.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:15 smithi070 ceph-mon[107385]: pgmap v557: 521 pgs: 521 active+clean; 55 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 1.2 MiB/s wr, 40 op/s 2024-09-21T10:08:15.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:15 smithi070 ceph-mon[112139]: pgmap v557: 521 pgs: 521 active+clean; 55 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 1.2 MiB/s wr, 40 op/s 2024-09-21T10:08:15.492 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:15 smithi160 ceph-mon[101879]: pgmap v557: 521 pgs: 521 active+clean; 55 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 1.2 MiB/s wr, 40 op/s 2024-09-21T10:08:16.127 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:15 smithi070 systemd[1]: Stopping Ceph osd.2 for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:08:16.627 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:16 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2[58230]: 2024-09-21T10:08:16.171+0000 7f3a2945e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:08:16.628 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:16 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2[58230]: 2024-09-21T10:08:16.171+0000 7f3a2945e640 -1 osd.2 676 *** Got signal Terminated *** 2024-09-21T10:08:16.628 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:16 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2[58230]: 2024-09-21T10:08:16.171+0000 7f3a2945e640 -1 osd.2 676 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:08:17.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:17 smithi070 ceph-mon[107385]: pgmap v558: 521 pgs: 521 active+clean; 55 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 411 KiB/s wr, 36 op/s 2024-09-21T10:08:17.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:17 smithi070 ceph-mon[107385]: osd.2 marked itself down and dead 2024-09-21T10:08:17.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:17 smithi070 ceph-mon[112139]: pgmap v558: 521 pgs: 521 active+clean; 55 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 411 KiB/s wr, 36 op/s 2024-09-21T10:08:17.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:17 smithi070 ceph-mon[112139]: osd.2 marked itself down and dead 2024-09-21T10:08:17.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:17 smithi160 ceph-mon[101879]: pgmap v558: 521 pgs: 521 active+clean; 55 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 411 KiB/s wr, 36 op/s 2024-09-21T10:08:17.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:17 smithi160 ceph-mon[101879]: osd.2 marked itself down and dead 2024-09-21T10:08:18.126 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:17 smithi070 podman[135993]: 2024-09-21 10:08:17.716192479 +0000 UTC m=+1.689711862 container died 58a11b8ddb81dc1f69c478c054599ad618207faa30437027040d0f24e3d85f9c (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2, GIT_BRANCH=HEAD, RELEASE=reef-6a1bdf8, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-21T10:08:18.126 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:18 smithi070 ceph-mon[107385]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:08:18.126 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:18 smithi070 ceph-mon[107385]: osdmap e677: 8 total, 7 up, 8 in 2024-09-21T10:08:18.126 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:18 smithi070 ceph-mon[112139]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:08:18.126 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:18 smithi070 ceph-mon[112139]: osdmap e677: 8 total, 7 up, 8 in 2024-09-21T10:08:18.377 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:18 smithi070 podman[135993]: 2024-09-21 10:08:18.316913826 +0000 UTC m=+2.290433190 container cleanup 58a11b8ddb81dc1f69c478c054599ad618207faa30437027040d0f24e3d85f9c (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-6a1bdf8) 2024-09-21T10:08:18.377 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:18 smithi070 bash[135993]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2 2024-09-21T10:08:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:18 smithi160 ceph-mon[101879]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:08:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:18 smithi160 ceph-mon[101879]: osdmap e677: 8 total, 7 up, 8 in 2024-09-21T10:08:18.877 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:18 smithi070 podman[136099]: 2024-09-21 10:08:18.551596812 +0000 UTC m=+0.831823157 container remove 58a11b8ddb81dc1f69c478c054599ad618207faa30437027040d0f24e3d85f9c (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-6a1bdf8, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731) 2024-09-21T10:08:19.257 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:18 smithi070 podman[136232]: 2024-09-21 10:08:18.807686997 +0000 UTC m=+0.020508532 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:08:19.257 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:19 smithi070 podman[136232]: 2024-09-21 10:08:19.027491631 +0000 UTC m=+0.240313162 container create b67f51b4b890fb5074af95ec71c2bdfd73294bb91e7203fa0906e0806bebea81 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-62da54b) 2024-09-21T10:08:19.257 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:19 smithi070 podman[136232]: 2024-09-21 10:08:19.255429999 +0000 UTC m=+0.468251544 container init b67f51b4b890fb5074af95ec71c2bdfd73294bb91e7203fa0906e0806bebea81 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=main-62da54b, org.label-schema.schema-version=1.0) 2024-09-21T10:08:19.258 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:19 smithi070 ceph-mon[107385]: pgmap v560: 521 pgs: 53 stale+active+clean, 468 active+clean; 55 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 411 KiB/s wr, 36 op/s 2024-09-21T10:08:19.258 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:19 smithi070 ceph-mon[107385]: osdmap e678: 8 total, 7 up, 8 in 2024-09-21T10:08:19.258 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:19 smithi070 ceph-mon[112139]: pgmap v560: 521 pgs: 53 stale+active+clean, 468 active+clean; 55 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 411 KiB/s wr, 36 op/s 2024-09-21T10:08:19.258 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:19 smithi070 ceph-mon[112139]: osdmap e678: 8 total, 7 up, 8 in 2024-09-21T10:08:19.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:19 smithi160 ceph-mon[101879]: pgmap v560: 521 pgs: 53 stale+active+clean, 468 active+clean; 55 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 411 KiB/s wr, 36 op/s 2024-09-21T10:08:19.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:19 smithi160 ceph-mon[101879]: osdmap e678: 8 total, 7 up, 8 in 2024-09-21T10:08:19.627 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:19 smithi070 podman[136232]: 2024-09-21 10:08:19.262235148 +0000 UTC m=+0.475056687 container start b67f51b4b890fb5074af95ec71c2bdfd73294bb91e7203fa0906e0806bebea81 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, RELEASE=main-62da54b, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:08:19.627 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:19 smithi070 podman[136232]: 2024-09-21 10:08:19.385578568 +0000 UTC m=+0.598400107 container attach b67f51b4b890fb5074af95ec71c2bdfd73294bb91e7203fa0906e0806bebea81 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:08:19.627 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:19 smithi070 podman[136232]: 2024-09-21 10:08:19.494638545 +0000 UTC m=+0.707460083 container died b67f51b4b890fb5074af95ec71c2bdfd73294bb91e7203fa0906e0806bebea81 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.build-date=20240731, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:08:20.144 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:20 smithi070 podman[136313]: 2024-09-21 10:08:20.125436523 +0000 UTC m=+0.622140745 container remove b67f51b4b890fb5074af95ec71c2bdfd73294bb91e7203fa0906e0806bebea81 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-21T10:08:20.144 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:20 smithi070 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.2.service: Deactivated successfully. 2024-09-21T10:08:20.144 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:20 smithi070 systemd[1]: Stopped Ceph osd.2 for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:08:20.401 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:20 smithi070 ceph-mon[107385]: pgmap v562: 521 pgs: 47 active+undersized, 38 stale+active+clean, 16 active+undersized+degraded, 420 active+clean; 59 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 512 KiB/s wr, 0 op/s; 36/741 objects degraded (4.858%) 2024-09-21T10:08:20.401 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:20 smithi070 ceph-mon[107385]: Health check failed: Degraded data redundancy: 36/741 objects degraded (4.858%), 16 pgs degraded (PG_DEGRADED) 2024-09-21T10:08:20.402 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:20 smithi070 ceph-mon[112139]: pgmap v562: 521 pgs: 47 active+undersized, 38 stale+active+clean, 16 active+undersized+degraded, 420 active+clean; 59 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 512 KiB/s wr, 0 op/s; 36/741 objects degraded (4.858%) 2024-09-21T10:08:20.402 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:20 smithi070 ceph-mon[112139]: Health check failed: Degraded data redundancy: 36/741 objects degraded (4.858%), 16 pgs degraded (PG_DEGRADED) 2024-09-21T10:08:20.402 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:20 smithi070 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.2.service: Consumed 29.939s CPU time. 2024-09-21T10:08:20.402 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:20 smithi070 systemd[1]: Starting Ceph osd.2 for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:08:20.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:20 smithi160 ceph-mon[101879]: pgmap v562: 521 pgs: 47 active+undersized, 38 stale+active+clean, 16 active+undersized+degraded, 420 active+clean; 59 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 512 KiB/s wr, 0 op/s; 36/741 objects degraded (4.858%) 2024-09-21T10:08:20.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:20 smithi160 ceph-mon[101879]: Health check failed: Degraded data redundancy: 36/741 objects degraded (4.858%), 16 pgs degraded (PG_DEGRADED) 2024-09-21T10:08:20.757 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:20 smithi070 podman[136471]: 2024-09-21 10:08:20.532481146 +0000 UTC m=+0.017317623 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:08:20.757 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:20 smithi070 podman[136471]: 2024-09-21 10:08:20.63455199 +0000 UTC m=+0.119388461 container create 4e56b8fbe21352945cd587f5868c0b2364ecd91542165ad85a1857e502e09c0e (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2-activate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=main-62da54b) 2024-09-21T10:08:21.028 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:20 smithi070 podman[136471]: 2024-09-21 10:08:20.878818345 +0000 UTC m=+0.363654808 container init 4e56b8fbe21352945cd587f5868c0b2364ecd91542165ad85a1857e502e09c0e (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2-activate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b) 2024-09-21T10:08:21.028 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:20 smithi070 podman[136471]: 2024-09-21 10:08:20.884059256 +0000 UTC m=+0.368895720 container start 4e56b8fbe21352945cd587f5868c0b2364ecd91542165ad85a1857e502e09c0e (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2-activate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-62da54b, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-21T10:08:21.028 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:20 smithi070 podman[136471]: 2024-09-21 10:08:20.961709045 +0000 UTC m=+0.446545508 container attach 4e56b8fbe21352945cd587f5868c0b2364ecd91542165ad85a1857e502e09c0e (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2-activate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-21T10:08:21.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:21.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:21.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:21.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:21.378 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:21 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2-activate[136501]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:08:21.378 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:21 smithi070 bash[136471]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:08:21.378 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:21 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2-activate[136501]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:08:21.378 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:21 smithi070 bash[136471]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:08:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:21.866 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:21 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2-activate[136501]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T10:08:21.866 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:21 smithi070 bash[136471]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T10:08:21.866 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:21 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2-activate[136501]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-21T10:08:21.866 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:21 smithi070 bash[136471]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-21T10:08:21.866 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:21 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2-activate[136501]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-21T10:08:21.866 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:21 smithi070 bash[136471]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-21T10:08:22.127 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:21 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2-activate[136501]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-21T10:08:22.128 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:21 smithi070 bash[136471]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-21T10:08:22.128 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:21 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2-activate[136501]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-21T10:08:22.128 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:21 smithi070 bash[136471]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-21T10:08:22.128 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:21 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2-activate[136501]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T10:08:22.128 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:21 smithi070 bash[136471]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T10:08:22.128 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:21 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2-activate[136501]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-21T10:08:22.128 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:21 smithi070 bash[136471]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-21T10:08:22.128 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:21 smithi070 podman[136471]: 2024-09-21 10:08:21.919276212 +0000 UTC m=+1.404112692 container died 4e56b8fbe21352945cd587f5868c0b2364ecd91542165ad85a1857e502e09c0e (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2-activate, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:08:22.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:22 smithi160 ceph-mon[101879]: pgmap v563: 521 pgs: 148 active+undersized, 53 active+undersized+degraded, 320 active+clean; 71 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s wr, 1 op/s; 103/750 objects degraded (13.733%) 2024-09-21T10:08:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:22 smithi070 ceph-mon[107385]: pgmap v563: 521 pgs: 148 active+undersized, 53 active+undersized+degraded, 320 active+clean; 71 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s wr, 1 op/s; 103/750 objects degraded (13.733%) 2024-09-21T10:08:22.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:22 smithi070 ceph-mon[112139]: pgmap v563: 521 pgs: 148 active+undersized, 53 active+undersized+degraded, 320 active+clean; 71 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s wr, 1 op/s; 103/750 objects degraded (13.733%) 2024-09-21T10:08:22.950 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:22 smithi070 podman[136710]: 2024-09-21 10:08:22.649181795 +0000 UTC m=+0.723299276 container remove 4e56b8fbe21352945cd587f5868c0b2364ecd91542165ad85a1857e502e09c0e (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2-activate, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-21T10:08:22.951 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:22 smithi070 podman[136821]: 2024-09-21 10:08:22.786300346 +0000 UTC m=+0.023198419 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:08:22.951 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:22 smithi070 podman[136821]: 2024-09-21 10:08:22.891856539 +0000 UTC m=+0.128754613 container create 014c9efbc75d658bac4169810fb03f1d72b3958c43f05cc8da67d59f9e854ffd (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:08:23.250 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:08:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:08:23] "GET /metrics HTTP/1.1" 200 67775 "" "Prometheus/2.51.0" 2024-09-21T10:08:23.251 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:23 smithi070 podman[136821]: 2024-09-21 10:08:23.153354148 +0000 UTC m=+0.390252223 container init 014c9efbc75d658bac4169810fb03f1d72b3958c43f05cc8da67d59f9e854ffd (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True) 2024-09-21T10:08:23.251 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:23 smithi070 podman[136821]: 2024-09-21 10:08:23.161738188 +0000 UTC m=+0.398636256 container start 014c9efbc75d658bac4169810fb03f1d72b3958c43f05cc8da67d59f9e854ffd (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:08:23.251 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:23 smithi070 ceph-osd[136871]: -- 172.21.15.70:0/1613706926 <== mon.2 v2:172.21.15.70:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564098095a40 con 0x564098e64000 2024-09-21T10:08:23.527 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:23 smithi070 bash[136821]: 014c9efbc75d658bac4169810fb03f1d72b3958c43f05cc8da67d59f9e854ffd 2024-09-21T10:08:23.527 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:23 smithi070 systemd[1]: Started Ceph osd.2 for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:08:24.342 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:24 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2[136847]: 2024-09-21T10:08:24.075+0000 7fd66372d640 -1 Falling back to public interface 2024-09-21T10:08:24.342 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:24 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:24.342 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:24 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:24.342 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:24 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:24.343 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:24 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:08:24.343 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:24 smithi070 ceph-mon[107385]: pgmap v564: 521 pgs: 148 active+undersized, 53 active+undersized+degraded, 320 active+clean; 71 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s wr, 1 op/s; 103/750 objects degraded (13.733%) 2024-09-21T10:08:24.343 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:24 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:24 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:24 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:24 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:24 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:08:24.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:24 smithi160 ceph-mon[101879]: pgmap v564: 521 pgs: 148 active+undersized, 53 active+undersized+degraded, 320 active+clean; 71 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s wr, 1 op/s; 103/750 objects degraded (13.733%) 2024-09-21T10:08:24.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:24 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:24.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:24 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:24.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:24 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:08:24.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:24 smithi070 ceph-mon[112139]: pgmap v564: 521 pgs: 148 active+undersized, 53 active+undersized+degraded, 320 active+clean; 71 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s wr, 1 op/s; 103/750 objects degraded (13.733%) 2024-09-21T10:08:27.254 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:27 smithi160 ceph-mon[101879]: pgmap v565: 521 pgs: 148 active+undersized, 53 active+undersized+degraded, 320 active+clean; 71 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.8 MiB/s wr, 1 op/s; 103/750 objects degraded (13.733%) 2024-09-21T10:08:27.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:27 smithi070 ceph-mon[107385]: pgmap v565: 521 pgs: 148 active+undersized, 53 active+undersized+degraded, 320 active+clean; 71 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.8 MiB/s wr, 1 op/s; 103/750 objects degraded (13.733%) 2024-09-21T10:08:27.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:27 smithi070 ceph-mon[112139]: pgmap v565: 521 pgs: 148 active+undersized, 53 active+undersized+degraded, 320 active+clean; 71 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.8 MiB/s wr, 1 op/s; 103/750 objects degraded (13.733%) 2024-09-21T10:08:28.369 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:27 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2[136847]: 2024-09-21T10:08:27.938+0000 7fd66372d640 -1 osd.2 677 log_to_monitors true 2024-09-21T10:08:28.369 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:28 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:28.369 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:28 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:28.370 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:28 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:28.370 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:28 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:28.370 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:28 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 103/750 objects degraded (13.733%), 53 pgs degraded (PG_DEGRADED) 2024-09-21T10:08:28.370 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:28 smithi070 ceph-mon[107385]: from='osd.2 [v2:172.21.15.70:6818/890146607,v1:172.21.15.70:6819/890146607]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T10:08:28.370 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:28 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:28.370 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:28 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:28.370 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:28 smithi070 ceph-mon[107385]: pgmap v566: 521 pgs: 148 active+undersized, 53 active+undersized+degraded, 320 active+clean; 71 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.6 MiB/s wr, 1 op/s; 103/750 objects degraded (13.733%) 2024-09-21T10:08:28.370 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:28 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:28.370 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:28 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:28.370 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:28 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:28.370 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:28 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:28.371 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:28 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 103/750 objects degraded (13.733%), 53 pgs degraded (PG_DEGRADED) 2024-09-21T10:08:28.371 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:28 smithi070 ceph-mon[112139]: from='osd.2 [v2:172.21.15.70:6818/890146607,v1:172.21.15.70:6819/890146607]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T10:08:28.371 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:28 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:28.371 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:28 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:28.371 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:28 smithi070 ceph-mon[112139]: pgmap v566: 521 pgs: 148 active+undersized, 53 active+undersized+degraded, 320 active+clean; 71 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.6 MiB/s wr, 1 op/s; 103/750 objects degraded (13.733%) 2024-09-21T10:08:28.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:28 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:28.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:28 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:28.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:28 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:28.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:28 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:28.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:28 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 103/750 objects degraded (13.733%), 53 pgs degraded (PG_DEGRADED) 2024-09-21T10:08:28.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:28 smithi160 ceph-mon[101879]: from='osd.2 [v2:172.21.15.70:6818/890146607,v1:172.21.15.70:6819/890146607]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T10:08:28.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:28 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:28.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:28 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:28.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:28 smithi160 ceph-mon[101879]: pgmap v566: 521 pgs: 148 active+undersized, 53 active+undersized+degraded, 320 active+clean; 71 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.6 MiB/s wr, 1 op/s; 103/750 objects degraded (13.733%) 2024-09-21T10:08:28.627 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 10:08:28 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2[136847]: 2024-09-21T10:08:28.577+0000 7fd65b4d7640 -1 osd.2 677 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T10:08:29.785 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:29 smithi070 ceph-mon[107385]: from='osd.2 [v2:172.21.15.70:6818/890146607,v1:172.21.15.70:6819/890146607]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T10:08:29.785 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:29 smithi070 ceph-mon[107385]: osdmap e679: 8 total, 7 up, 8 in 2024-09-21T10:08:29.785 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:29 smithi070 ceph-mon[107385]: from='osd.2 [v2:172.21.15.70:6818/890146607,v1:172.21.15.70:6819/890146607]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]} : dispatch 2024-09-21T10:08:29.785 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:29 smithi070 ceph-mon[112139]: from='osd.2 [v2:172.21.15.70:6818/890146607,v1:172.21.15.70:6819/890146607]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T10:08:29.785 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:29 smithi070 ceph-mon[112139]: osdmap e679: 8 total, 7 up, 8 in 2024-09-21T10:08:29.785 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:29 smithi070 ceph-mon[112139]: from='osd.2 [v2:172.21.15.70:6818/890146607,v1:172.21.15.70:6819/890146607]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]} : dispatch 2024-09-21T10:08:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:29 smithi160 ceph-mon[101879]: from='osd.2 [v2:172.21.15.70:6818/890146607,v1:172.21.15.70:6819/890146607]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T10:08:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:29 smithi160 ceph-mon[101879]: osdmap e679: 8 total, 7 up, 8 in 2024-09-21T10:08:29.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:29 smithi160 ceph-mon[101879]: from='osd.2 [v2:172.21.15.70:6818/890146607,v1:172.21.15.70:6819/890146607]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]} : dispatch 2024-09-21T10:08:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:30 smithi160 ceph-mon[101879]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:08:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:30 smithi160 ceph-mon[101879]: osd.2 [v2:172.21.15.70:6818/890146607,v1:172.21.15.70:6819/890146607] boot 2024-09-21T10:08:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:30 smithi160 ceph-mon[101879]: osdmap e680: 8 total, 8 up, 8 in 2024-09-21T10:08:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:30 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T10:08:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:30 smithi160 ceph-mon[101879]: pgmap v569: 521 pgs: 148 active+undersized, 53 active+undersized+degraded, 320 active+clean; 71 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.5 MiB/s rd, 0 op/s; 103/750 objects degraded (13.733%) 2024-09-21T10:08:30.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:30 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:30.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:30 smithi070 ceph-mon[107385]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:08:30.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:30 smithi070 ceph-mon[107385]: osd.2 [v2:172.21.15.70:6818/890146607,v1:172.21.15.70:6819/890146607] boot 2024-09-21T10:08:30.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:30 smithi070 ceph-mon[107385]: osdmap e680: 8 total, 8 up, 8 in 2024-09-21T10:08:30.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:30 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T10:08:30.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:30 smithi070 ceph-mon[107385]: pgmap v569: 521 pgs: 148 active+undersized, 53 active+undersized+degraded, 320 active+clean; 71 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.5 MiB/s rd, 0 op/s; 103/750 objects degraded (13.733%) 2024-09-21T10:08:30.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:30 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:30.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:30 smithi070 ceph-mon[112139]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:08:30.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:30 smithi070 ceph-mon[112139]: osd.2 [v2:172.21.15.70:6818/890146607,v1:172.21.15.70:6819/890146607] boot 2024-09-21T10:08:30.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:30 smithi070 ceph-mon[112139]: osdmap e680: 8 total, 8 up, 8 in 2024-09-21T10:08:30.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:30 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T10:08:30.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:30 smithi070 ceph-mon[112139]: pgmap v569: 521 pgs: 148 active+undersized, 53 active+undersized+degraded, 320 active+clean; 71 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.5 MiB/s rd, 0 op/s; 103/750 objects degraded (13.733%) 2024-09-21T10:08:30.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:30 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:31.558 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:08:31 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T10:08:31.358+0000 7fd088d1d640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (66 PGs are or would become offline) 2024-09-21T10:08:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:31 smithi160 ceph-mon[101879]: osdmap e681: 8 total, 8 up, 8 in 2024-09-21T10:08:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:31.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:08:31.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:08:31.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:31.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:08:31.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:31.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:31.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:31.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T10:08:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:31 smithi070 ceph-mon[107385]: osdmap e681: 8 total, 8 up, 8 in 2024-09-21T10:08:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:31.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:08:31.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:08:31.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:31.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:08:31.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:31.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:31.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:31.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T10:08:31.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:31 smithi070 ceph-mon[112139]: osdmap e681: 8 total, 8 up, 8 in 2024-09-21T10:08:31.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:31.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:31.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:08:31.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:08:31.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:31.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:08:31.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:31.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:31.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:31.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T10:08:32.153 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-09-21T10:08:32.570 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:32 smithi070 ceph-mon[107385]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T10:08:32.570 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:32 smithi070 ceph-mon[107385]: Upgrade: unsafe to stop osd(s) at this time (66 PGs are or would become offline) 2024-09-21T10:08:32.570 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:32 smithi070 ceph-mon[107385]: osdmap e682: 8 total, 8 up, 8 in 2024-09-21T10:08:32.570 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:32 smithi070 ceph-mon[107385]: pgmap v572: 521 pgs: 76 active+undersized, 25 active+undersized+degraded, 420 active+clean; 71 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 145 KiB/s rd, 0 op/s; 51/750 objects degraded (6.800%) 2024-09-21T10:08:32.570 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:32 smithi070 ceph-mon[107385]: from='client.34581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:32.571 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:32 smithi070 ceph-mon[112139]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T10:08:32.571 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:32 smithi070 ceph-mon[112139]: Upgrade: unsafe to stop osd(s) at this time (66 PGs are or would become offline) 2024-09-21T10:08:32.571 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:32 smithi070 ceph-mon[112139]: osdmap e682: 8 total, 8 up, 8 in 2024-09-21T10:08:32.571 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:32 smithi070 ceph-mon[112139]: pgmap v572: 521 pgs: 76 active+undersized, 25 active+undersized+degraded, 420 active+clean; 71 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 145 KiB/s rd, 0 op/s; 51/750 objects degraded (6.800%) 2024-09-21T10:08:32.571 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:32 smithi070 ceph-mon[112139]: from='client.34581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:32.800 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T10:08:32.800 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (8m) 5s ago 14m 19.3M - 0.27.0 11f11916f8cd 89ae95182419 2024-09-21T10:08:32.800 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi160 *:3000 running (8m) 5s ago 14m 83.2M - 10.4.8 5aad1d7cf004 daa2d8dfab48 2024-09-21T10:08:32.800 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi070.gansda smithi070 running (13m) 5s ago 13m 23.9M - 18.2.4-871-g6a1bdf85 0adab276c97e 6016b2501c8a 2024-09-21T10:08:32.801 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi160.byyhfz smithi160 running (13m) 5s ago 13m 17.8M - 18.2.4-871-g6a1bdf85 0adab276c97e 93e108d91e65 2024-09-21T10:08:32.801 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi160 *:8443,9283,8765 running (10m) 5s ago 18m 459M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 d952652a9683 2024-09-21T10:08:32.801 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (10m) 5s ago 20m 536M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 e3d390893cff 2024-09-21T10:08:32.801 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (7m) 5s ago 20m 78.6M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 4a7b667a4033 2024-09-21T10:08:32.801 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi160 running (5m) 5s ago 19m 52.5M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 86e88af21f36 2024-09-21T10:08:32.801 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (7m) 5s ago 18m 63.7M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 e154380b80a2 2024-09-21T10:08:32.801 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (9m) 5s ago 14m 10.7M - 1.7.0 72c9c2088986 5d5d4b026070 2024-09-21T10:08:32.801 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi160 *:9100 running (9m) 5s ago 14m 11.3M - 1.7.0 72c9c2088986 19b8e0e8ee0c 2024-09-21T10:08:32.801 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (79s) 5s ago 18m 56.9M 2480M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 eedacb586f7c 2024-09-21T10:08:32.801 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (45s) 5s ago 17m 68.9M 2480M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 853f3bb7fa25 2024-09-21T10:08:32.801 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (10s) 5s ago 17m 11.8M 2480M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 014c9efbc75d 2024-09-21T10:08:32.801 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (17m) 5s ago 17m 142M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e aa0c8a732a41 2024-09-21T10:08:32.801 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi160 running (16m) 5s ago 16m 136M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 7325a075c488 2024-09-21T10:08:32.801 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi160 running (16m) 5s ago 16m 120M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e f74026345641 2024-09-21T10:08:32.802 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi160 running (16m) 5s ago 16m 150M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 03e6bb54f8b2 2024-09-21T10:08:32.802 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi160 running (15m) 5s ago 15m 134M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e e405b21a6146 2024-09-21T10:08:32.802 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi160 *:9095 running (9m) 5s ago 15m 58.3M - 2.51.0 1d3b7f56885b 648952e57ad7 2024-09-21T10:08:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:32 smithi160 ceph-mon[101879]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T10:08:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:32 smithi160 ceph-mon[101879]: Upgrade: unsafe to stop osd(s) at this time (66 PGs are or would become offline) 2024-09-21T10:08:32.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:32 smithi160 ceph-mon[101879]: osdmap e682: 8 total, 8 up, 8 in 2024-09-21T10:08:32.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:32 smithi160 ceph-mon[101879]: pgmap v572: 521 pgs: 76 active+undersized, 25 active+undersized+degraded, 420 active+clean; 71 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 145 KiB/s rd, 0 op/s; 51/750 objects degraded (6.800%) 2024-09-21T10:08:32.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:32 smithi160 ceph-mon[101879]: from='client.34581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:33.166 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:08:33.166 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-09-21T10:08:33.166 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 3 2024-09-21T10:08:33.166 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:08:33.166 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-09-21T10:08:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T10:08:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:08:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-09-21T10:08:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 5, 2024-09-21T10:08:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 3 2024-09-21T10:08:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:08:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": { 2024-09-21T10:08:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 2 2024-09-21T10:08:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:08:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-09-21T10:08:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 7, 2024-09-21T10:08:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 8 2024-09-21T10:08:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:08:33.167 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:08:33.514 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:08:33.515 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47", 2024-09-21T10:08:33.515 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-09-21T10:08:33.515 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 4 daemons (1 remaining).", 2024-09-21T10:08:33.515 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-09-21T10:08:33.515 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "3/8 daemons upgraded", 2024-09-21T10:08:33.515 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading osd daemons", 2024-09-21T10:08:33.515 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-09-21T10:08:33.515 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:08:33.515 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:08:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:08:33] "GET /metrics HTTP/1.1" 200 67775 "" "Prometheus/2.51.0" 2024-09-21T10:08:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:33 smithi160 ceph-mon[101879]: from='client.44516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:33 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 51/750 objects degraded (6.800%), 25 pgs degraded (PG_DEGRADED) 2024-09-21T10:08:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:33 smithi160 ceph-mon[101879]: from='client.44522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:33.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:33 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3009645627' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:33 smithi070 ceph-mon[112139]: from='client.44516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:33 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 51/750 objects degraded (6.800%), 25 pgs degraded (PG_DEGRADED) 2024-09-21T10:08:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:33 smithi070 ceph-mon[112139]: from='client.44522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:33 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3009645627' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:33.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:33 smithi070 ceph-mon[107385]: from='client.44516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:33.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:33 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 51/750 objects degraded (6.800%), 25 pgs degraded (PG_DEGRADED) 2024-09-21T10:08:33.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:33 smithi070 ceph-mon[107385]: from='client.44522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:33.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:33 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3009645627' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:34.579 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:34 smithi070 ceph-mon[107385]: from='client.44534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:34 smithi160 ceph-mon[101879]: from='client.44534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:34 smithi160 ceph-mon[101879]: pgmap v573: 521 pgs: 16 active+undersized, 5 active+undersized+degraded, 500 active+clean; 71 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s rd, 1.1 KiB/s wr, 37 op/s; 6/756 objects degraded (0.794%) 2024-09-21T10:08:34.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:34 smithi070 ceph-mon[112139]: from='client.44534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:34.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:34 smithi070 ceph-mon[112139]: pgmap v573: 521 pgs: 16 active+undersized, 5 active+undersized+degraded, 500 active+clean; 71 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s rd, 1.1 KiB/s wr, 37 op/s; 6/756 objects degraded (0.794%) 2024-09-21T10:08:34.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:34 smithi070 ceph-mon[107385]: pgmap v573: 521 pgs: 16 active+undersized, 5 active+undersized+degraded, 500 active+clean; 71 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s rd, 1.1 KiB/s wr, 37 op/s; 6/756 objects degraded (0.794%) 2024-09-21T10:08:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:36 smithi160 ceph-mon[101879]: pgmap v574: 521 pgs: 521 active+clean; 71 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 141 KiB/s rd, 1.7 KiB/s wr, 64 op/s 2024-09-21T10:08:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:36 smithi070 ceph-mon[107385]: pgmap v574: 521 pgs: 521 active+clean; 71 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 141 KiB/s rd, 1.7 KiB/s wr, 64 op/s 2024-09-21T10:08:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:36 smithi070 ceph-mon[112139]: pgmap v574: 521 pgs: 521 active+clean; 71 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 141 KiB/s rd, 1.7 KiB/s wr, 64 op/s 2024-09-21T10:08:37.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:37 smithi160 ceph-mon[101879]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/756 objects degraded (0.794%), 5 pgs degraded) 2024-09-21T10:08:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:37 smithi070 ceph-mon[107385]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/756 objects degraded (0.794%), 5 pgs degraded) 2024-09-21T10:08:37.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:37 smithi070 ceph-mon[112139]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/756 objects degraded (0.794%), 5 pgs degraded) 2024-09-21T10:08:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:38 smithi160 ceph-mon[101879]: pgmap v575: 521 pgs: 521 active+clean; 71 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 1.4 KiB/s wr, 52 op/s 2024-09-21T10:08:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:38 smithi070 ceph-mon[107385]: pgmap v575: 521 pgs: 521 active+clean; 71 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 1.4 KiB/s wr, 52 op/s 2024-09-21T10:08:38.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:38 smithi070 ceph-mon[112139]: pgmap v575: 521 pgs: 521 active+clean; 71 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 1.4 KiB/s wr, 52 op/s 2024-09-21T10:08:41.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:41 smithi160 ceph-mon[101879]: pgmap v576: 521 pgs: 521 active+clean; 71 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 127 KiB/s rd, 1.2 KiB/s wr, 44 op/s 2024-09-21T10:08:41.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:41 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:41.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:41 smithi070 ceph-mon[107385]: pgmap v576: 521 pgs: 521 active+clean; 71 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 127 KiB/s rd, 1.2 KiB/s wr, 44 op/s 2024-09-21T10:08:41.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:41 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:41.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:41 smithi070 ceph-mon[112139]: pgmap v576: 521 pgs: 521 active+clean; 71 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 127 KiB/s rd, 1.2 KiB/s wr, 44 op/s 2024-09-21T10:08:41.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:41 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:43.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:43 smithi160 ceph-mon[101879]: pgmap v577: 521 pgs: 521 active+clean; 251 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 928 KiB/s rd, 17 MiB/s wr, 105 op/s 2024-09-21T10:08:43.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:43 smithi070 ceph-mon[107385]: pgmap v577: 521 pgs: 521 active+clean; 251 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 928 KiB/s rd, 17 MiB/s wr, 105 op/s 2024-09-21T10:08:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:08:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:08:43] "GET /metrics HTTP/1.1" 200 67769 "" "Prometheus/2.51.0" 2024-09-21T10:08:43.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:43 smithi070 ceph-mon[112139]: pgmap v577: 521 pgs: 521 active+clean; 251 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 928 KiB/s rd, 17 MiB/s wr, 105 op/s 2024-09-21T10:08:44.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:44 smithi160 ceph-mon[101879]: osdmap e683: 8 total, 8 up, 8 in 2024-09-21T10:08:44.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:44 smithi070 ceph-mon[107385]: osdmap e683: 8 total, 8 up, 8 in 2024-09-21T10:08:44.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:44 smithi070 ceph-mon[112139]: osdmap e683: 8 total, 8 up, 8 in 2024-09-21T10:08:45.492 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:45 smithi160 ceph-mon[101879]: pgmap v579: 521 pgs: 521 active+clean; 247 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1004 KiB/s rd, 26 MiB/s wr, 120 op/s 2024-09-21T10:08:45.493 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:45 smithi160 ceph-mon[101879]: osdmap e684: 8 total, 8 up, 8 in 2024-09-21T10:08:45.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:45 smithi070 ceph-mon[107385]: pgmap v579: 521 pgs: 521 active+clean; 247 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1004 KiB/s rd, 26 MiB/s wr, 120 op/s 2024-09-21T10:08:45.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:45 smithi070 ceph-mon[107385]: osdmap e684: 8 total, 8 up, 8 in 2024-09-21T10:08:45.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:45 smithi070 ceph-mon[112139]: pgmap v579: 521 pgs: 521 active+clean; 247 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1004 KiB/s rd, 26 MiB/s wr, 120 op/s 2024-09-21T10:08:45.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:45 smithi070 ceph-mon[112139]: osdmap e684: 8 total, 8 up, 8 in 2024-09-21T10:08:46.258 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 4 2024-09-21T10:08:46.259 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 4 2024-09-21T10:08:46.259 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 4 2024-09-21T10:08:46.259 INFO:tasks.workunit.client.0.smithi070.stdout:read: 4 2024-09-21T10:08:46.259 INFO:tasks.workunit.client.0.smithi070.stdout:read: 4 2024-09-21T10:08:46.259 INFO:tasks.workunit.client.0.smithi070.stdout:read: 4 2024-09-21T10:08:46.259 INFO:tasks.workunit.client.0.smithi070.stdout:read: 4 2024-09-21T10:08:46.259 INFO:tasks.workunit.client.0.smithi070.stdout:read: 4 2024-09-21T10:08:46.259 INFO:tasks.workunit.client.0.smithi070.stdout:read: 4 2024-09-21T10:08:46.259 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCloneEncryption (60443 ms) 2024-09-21T10:08:46.259 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.LUKS1UnderLUKS2WithoutResize 2024-09-21T10:08:46.260 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:08:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:46 smithi160 ceph-mon[101879]: osdmap e685: 8 total, 8 up, 8 in 2024-09-21T10:08:46.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:46 smithi070 ceph-mon[107385]: osdmap e685: 8 total, 8 up, 8 in 2024-09-21T10:08:46.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:46 smithi070 ceph-mon[112139]: osdmap e685: 8 total, 8 up, 8 in 2024-09-21T10:08:47.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:47 smithi070 ceph-mon[107385]: pgmap v582: 521 pgs: 521 active+clean; 39 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.6 MiB/s rd, 43 MiB/s wr, 269 op/s 2024-09-21T10:08:47.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:47 smithi070 ceph-mon[107385]: osdmap e686: 8 total, 8 up, 8 in 2024-09-21T10:08:47.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:47 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T10:08:47.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:47 smithi070 ceph-mon[112139]: pgmap v582: 521 pgs: 521 active+clean; 39 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.6 MiB/s rd, 43 MiB/s wr, 269 op/s 2024-09-21T10:08:47.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:47 smithi070 ceph-mon[112139]: osdmap e686: 8 total, 8 up, 8 in 2024-09-21T10:08:47.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:47 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T10:08:47.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:47 smithi160 ceph-mon[101879]: pgmap v582: 521 pgs: 521 active+clean; 39 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.6 MiB/s rd, 43 MiB/s wr, 269 op/s 2024-09-21T10:08:47.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:47 smithi160 ceph-mon[101879]: osdmap e686: 8 total, 8 up, 8 in 2024-09-21T10:08:47.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:47 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T10:08:48.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:48 smithi070 ceph-mon[107385]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T10:08:48.472 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:48 smithi070 ceph-mon[107385]: Upgrade: osd.3 is safe to restart 2024-09-21T10:08:48.472 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:48 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:48.472 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:48 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T10:08:48.473 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:48 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:08:48.473 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:48 smithi070 ceph-mon[107385]: osdmap e687: 8 total, 8 up, 8 in 2024-09-21T10:08:48.473 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:48 smithi070 ceph-mon[112139]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T10:08:48.474 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:48 smithi070 ceph-mon[112139]: Upgrade: osd.3 is safe to restart 2024-09-21T10:08:48.474 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:48 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:48.475 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:48 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T10:08:48.475 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:48 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:08:48.475 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:48 smithi070 ceph-mon[112139]: osdmap e687: 8 total, 8 up, 8 in 2024-09-21T10:08:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:48 smithi160 ceph-mon[101879]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T10:08:48.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:48 smithi160 ceph-mon[101879]: Upgrade: osd.3 is safe to restart 2024-09-21T10:08:48.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:48 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:48.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:48 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T10:08:48.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:48 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:08:48.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:48 smithi160 ceph-mon[101879]: osdmap e687: 8 total, 8 up, 8 in 2024-09-21T10:08:49.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:49 smithi070 ceph-mon[107385]: Upgrade: Updating osd.3 2024-09-21T10:08:49.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:49 smithi070 ceph-mon[107385]: Deploying daemon osd.3 on smithi070 2024-09-21T10:08:49.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:49 smithi070 ceph-mon[107385]: pgmap v585: 521 pgs: 521 active+clean; 39 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 11 KiB/s wr, 156 op/s 2024-09-21T10:08:49.359 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:49 smithi070 ceph-mon[112139]: Upgrade: Updating osd.3 2024-09-21T10:08:49.360 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:49 smithi070 ceph-mon[112139]: Deploying daemon osd.3 on smithi070 2024-09-21T10:08:49.360 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:49 smithi070 ceph-mon[112139]: pgmap v585: 521 pgs: 521 active+clean; 39 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 11 KiB/s wr, 156 op/s 2024-09-21T10:08:49.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:49 smithi160 ceph-mon[101879]: Upgrade: Updating osd.3 2024-09-21T10:08:49.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:49 smithi160 ceph-mon[101879]: Deploying daemon osd.3 on smithi070 2024-09-21T10:08:49.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:49 smithi160 ceph-mon[101879]: pgmap v585: 521 pgs: 521 active+clean; 39 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 11 KiB/s wr, 156 op/s 2024-09-21T10:08:51.181 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:51 smithi070 ceph-mon[107385]: pgmap v586: 521 pgs: 521 active+clean; 31 MiB data, 538 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 7.8 KiB/s wr, 129 op/s 2024-09-21T10:08:51.181 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:51 smithi070 ceph-mon[107385]: osdmap e688: 8 total, 8 up, 8 in 2024-09-21T10:08:51.182 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:51.182 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:51.182 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:50 smithi070 systemd[1]: Stopping Ceph osd.3 for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:08:51.182 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:51 smithi070 ceph-mon[112139]: pgmap v586: 521 pgs: 521 active+clean; 31 MiB data, 538 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 7.8 KiB/s wr, 129 op/s 2024-09-21T10:08:51.182 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:51 smithi070 ceph-mon[112139]: osdmap e688: 8 total, 8 up, 8 in 2024-09-21T10:08:51.182 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:51.182 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:51 smithi160 ceph-mon[101879]: pgmap v586: 521 pgs: 521 active+clean; 31 MiB data, 538 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 7.8 KiB/s wr, 129 op/s 2024-09-21T10:08:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:51 smithi160 ceph-mon[101879]: osdmap e688: 8 total, 8 up, 8 in 2024-09-21T10:08:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:51.627 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3[64453]: 2024-09-21T10:08:51.245+0000 7f8183430640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:08:51.627 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3[64453]: 2024-09-21T10:08:51.245+0000 7f8183430640 -1 osd.3 688 *** Got signal Terminated *** 2024-09-21T10:08:51.628 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:51 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3[64453]: 2024-09-21T10:08:51.245+0000 7f8183430640 -1 osd.3 688 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:08:52.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:52 smithi070 ceph-mon[107385]: osd.3 marked itself down and dead 2024-09-21T10:08:52.499 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:52 smithi070 ceph-mon[112139]: osd.3 marked itself down and dead 2024-09-21T10:08:52.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:52 smithi160 ceph-mon[101879]: osd.3 marked itself down and dead 2024-09-21T10:08:53.104 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:52 smithi070 podman[141957]: 2024-09-21 10:08:52.834245962 +0000 UTC m=+1.715250649 container died aa0c8a732a412c9326184d2a9e1a9368ee7972968fd44ce916a8b0b05dbefeba (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-6a1bdf8) 2024-09-21T10:08:53.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:08:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:08:53] "GET /metrics HTTP/1.1" 200 67774 "" "Prometheus/2.51.0" 2024-09-21T10:08:53.378 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:53 smithi070 podman[141957]: 2024-09-21 10:08:53.192438844 +0000 UTC m=+2.073443524 container cleanup aa0c8a732a412c9326184d2a9e1a9368ee7972968fd44ce916a8b0b05dbefeba (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, RELEASE=reef-6a1bdf8, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True) 2024-09-21T10:08:53.378 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:53 smithi070 bash[141957]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3 2024-09-21T10:08:53.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:53 smithi070 ceph-mon[107385]: pgmap v588: 521 pgs: 521 active+clean; 56 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 4.2 MiB/s wr, 66 op/s 2024-09-21T10:08:53.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:53 smithi070 ceph-mon[107385]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:08:53.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:53 smithi070 ceph-mon[107385]: osdmap e689: 8 total, 7 up, 8 in 2024-09-21T10:08:53.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:53 smithi070 ceph-mon[107385]: osd.3 marked itself dead as of e689 2024-09-21T10:08:53.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:53 smithi070 ceph-mon[107385]: osdmap e690: 8 total, 7 up, 8 in 2024-09-21T10:08:53.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:53 smithi070 ceph-mon[112139]: pgmap v588: 521 pgs: 521 active+clean; 56 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 4.2 MiB/s wr, 66 op/s 2024-09-21T10:08:53.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:53 smithi070 ceph-mon[112139]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:08:53.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:53 smithi070 ceph-mon[112139]: osdmap e689: 8 total, 7 up, 8 in 2024-09-21T10:08:53.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:53 smithi070 ceph-mon[112139]: osd.3 marked itself dead as of e689 2024-09-21T10:08:53.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:53 smithi070 ceph-mon[112139]: osdmap e690: 8 total, 7 up, 8 in 2024-09-21T10:08:53.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:53 smithi160 ceph-mon[101879]: pgmap v588: 521 pgs: 521 active+clean; 56 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 4.2 MiB/s wr, 66 op/s 2024-09-21T10:08:53.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:53 smithi160 ceph-mon[101879]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:08:53.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:53 smithi160 ceph-mon[101879]: osdmap e689: 8 total, 7 up, 8 in 2024-09-21T10:08:53.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:53 smithi160 ceph-mon[101879]: osd.3 marked itself dead as of e689 2024-09-21T10:08:53.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:53 smithi160 ceph-mon[101879]: osdmap e690: 8 total, 7 up, 8 in 2024-09-21T10:08:53.873 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:53 smithi070 podman[142379]: 2024-09-21 10:08:53.527202094 +0000 UTC m=+0.688590588 container remove aa0c8a732a412c9326184d2a9e1a9368ee7972968fd44ce916a8b0b05dbefeba (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=reef-6a1bdf8) 2024-09-21T10:08:53.873 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:53 smithi070 podman[142503]: 2024-09-21 10:08:53.774738559 +0000 UTC m=+0.021825819 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:08:54.127 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:53 smithi070 podman[142503]: 2024-09-21 10:08:53.90557752 +0000 UTC m=+0.152664781 container create fb5818501c15c3fa8073744c4eaa0ce86db8765d66fc61565cab6ea73825db00 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, RELEASE=main-62da54b, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:08:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:54 smithi160 ceph-mon[101879]: pgmap v591: 521 pgs: 48 active+undersized, 55 stale+active+clean, 22 active+undersized+degraded, 396 active+clean; 60 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 4.8 MiB/s wr, 81 op/s; 45/717 objects degraded (6.276%) 2024-09-21T10:08:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:54 smithi160 ceph-mon[101879]: Health check failed: Degraded data redundancy: 45/717 objects degraded (6.276%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T10:08:54.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:54 smithi070 ceph-mon[107385]: pgmap v591: 521 pgs: 48 active+undersized, 55 stale+active+clean, 22 active+undersized+degraded, 396 active+clean; 60 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 4.8 MiB/s wr, 81 op/s; 45/717 objects degraded (6.276%) 2024-09-21T10:08:54.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:54 smithi070 ceph-mon[107385]: Health check failed: Degraded data redundancy: 45/717 objects degraded (6.276%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T10:08:54.628 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:54 smithi070 podman[142503]: 2024-09-21 10:08:54.214804471 +0000 UTC m=+0.461891750 container init fb5818501c15c3fa8073744c4eaa0ce86db8765d66fc61565cab6ea73825db00 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, CEPH_POINT_RELEASE=) 2024-09-21T10:08:54.628 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:54 smithi070 podman[142503]: 2024-09-21 10:08:54.222720404 +0000 UTC m=+0.469807671 container start fb5818501c15c3fa8073744c4eaa0ce86db8765d66fc61565cab6ea73825db00 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-21T10:08:54.629 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:54 smithi070 podman[142503]: 2024-09-21 10:08:54.330949249 +0000 UTC m=+0.578036511 container attach fb5818501c15c3fa8073744c4eaa0ce86db8765d66fc61565cab6ea73825db00 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-21T10:08:54.629 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:54 smithi070 podman[142503]: 2024-09-21 10:08:54.441013864 +0000 UTC m=+0.688101123 container died fb5818501c15c3fa8073744c4eaa0ce86db8765d66fc61565cab6ea73825db00 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-21T10:08:54.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:54 smithi070 ceph-mon[112139]: pgmap v591: 521 pgs: 48 active+undersized, 55 stale+active+clean, 22 active+undersized+degraded, 396 active+clean; 60 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 4.8 MiB/s wr, 81 op/s; 45/717 objects degraded (6.276%) 2024-09-21T10:08:54.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:54 smithi070 ceph-mon[112139]: Health check failed: Degraded data redundancy: 45/717 objects degraded (6.276%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T10:08:55.112 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:55 smithi070 podman[142586]: 2024-09-21 10:08:55.109354858 +0000 UTC m=+0.662033380 container remove fb5818501c15c3fa8073744c4eaa0ce86db8765d66fc61565cab6ea73825db00 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:08:55.377 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:55 smithi070 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.3.service: Deactivated successfully. 2024-09-21T10:08:55.378 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:55 smithi070 systemd[1]: Stopped Ceph osd.3 for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:08:55.378 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:55 smithi070 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.3.service: Consumed 34.268s CPU time. 2024-09-21T10:08:55.754 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:55 smithi070 systemd[1]: Starting Ceph osd.3 for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:08:55.754 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:55 smithi070 podman[142734]: 2024-09-21 10:08:55.63110632 +0000 UTC m=+0.019026211 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:08:55.754 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:55 smithi070 podman[142734]: 2024-09-21 10:08:55.752048936 +0000 UTC m=+0.139968828 container create 493c7123a695434e7c0de740dd3e154777fe89656b2a81d51f8e13e7e6e9dedb (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3-activate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-21T10:08:56.082 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:55 smithi070 podman[142734]: 2024-09-21 10:08:55.963514106 +0000 UTC m=+0.351434013 container init 493c7123a695434e7c0de740dd3e154777fe89656b2a81d51f8e13e7e6e9dedb (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3-activate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-21T10:08:56.082 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:55 smithi070 podman[142734]: 2024-09-21 10:08:55.971198683 +0000 UTC m=+0.359118569 container start 493c7123a695434e7c0de740dd3e154777fe89656b2a81d51f8e13e7e6e9dedb (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-21T10:08:56.082 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:56 smithi070 podman[142734]: 2024-09-21 10:08:56.079709963 +0000 UTC m=+0.467629852 container attach 493c7123a695434e7c0de740dd3e154777fe89656b2a81d51f8e13e7e6e9dedb (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3-activate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-21T10:08:56.377 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3-activate[142765]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:08:56.378 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:56 smithi070 bash[142734]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:08:56.378 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3-activate[142765]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:08:56.378 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:56 smithi070 bash[142734]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:08:56.878 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3-activate[142765]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-21T10:08:56.878 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:56 smithi070 bash[142734]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-21T10:08:56.878 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3-activate[142765]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-21T10:08:56.878 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:56 smithi070 bash[142734]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-21T10:08:57.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:57 smithi070 ceph-mon[107385]: pgmap v592: 521 pgs: 160 active+undersized, 60 active+undersized+degraded, 301 active+clean; 60 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 4.8 MiB/s wr, 50 op/s; 101/717 objects degraded (14.086%) 2024-09-21T10:08:57.303 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3-activate[142765]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-21T10:08:57.303 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:56 smithi070 bash[142734]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-21T10:08:57.303 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3-activate[142765]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-21T10:08:57.303 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:56 smithi070 bash[142734]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-21T10:08:57.303 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3-activate[142765]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-21T10:08:57.303 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:56 smithi070 bash[142734]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-21T10:08:57.303 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3-activate[142765]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-21T10:08:57.303 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:56 smithi070 bash[142734]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-21T10:08:57.303 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:56 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3-activate[142765]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-21T10:08:57.304 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:56 smithi070 bash[142734]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-21T10:08:57.304 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:56 smithi070 podman[142734]: 2024-09-21 10:08:56.966563735 +0000 UTC m=+1.354483622 container died 493c7123a695434e7c0de740dd3e154777fe89656b2a81d51f8e13e7e6e9dedb (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3-activate, io.buildah.version=1.37.2, RELEASE=main-62da54b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2) 2024-09-21T10:08:57.304 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:57 smithi070 ceph-mon[112139]: pgmap v592: 521 pgs: 160 active+undersized, 60 active+undersized+degraded, 301 active+clean; 60 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 4.8 MiB/s wr, 50 op/s; 101/717 objects degraded (14.086%) 2024-09-21T10:08:57.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:57 smithi160 ceph-mon[101879]: pgmap v592: 521 pgs: 160 active+undersized, 60 active+undersized+degraded, 301 active+clean; 60 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 4.8 MiB/s wr, 50 op/s; 101/717 objects degraded (14.086%) 2024-09-21T10:08:57.852 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:57 smithi070 podman[142958]: 2024-09-21 10:08:57.591480907 +0000 UTC m=+0.618504610 container remove 493c7123a695434e7c0de740dd3e154777fe89656b2a81d51f8e13e7e6e9dedb (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3-activate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=main-62da54b, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:08:57.853 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:57 smithi070 podman[143062]: 2024-09-21 10:08:57.730468942 +0000 UTC m=+0.023250362 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:08:57.853 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:57 smithi070 podman[143062]: 2024-09-21 10:08:57.850689873 +0000 UTC m=+0.143471292 container create 862f41fda2c3a1ede15feb1b50d8d410e975f82fec81188500e80680ac209fd2 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True) 2024-09-21T10:08:58.127 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:58 smithi070 podman[143062]: 2024-09-21 10:08:58.020037394 +0000 UTC m=+0.312818816 container init 862f41fda2c3a1ede15feb1b50d8d410e975f82fec81188500e80680ac209fd2 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=main-62da54b, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:08:58.128 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:58 smithi070 podman[143062]: 2024-09-21 10:08:58.025508571 +0000 UTC m=+0.318289986 container start 862f41fda2c3a1ede15feb1b50d8d410e975f82fec81188500e80680ac209fd2 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-62da54b, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD) 2024-09-21T10:08:58.128 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:58 smithi070 ceph-osd[143112]: -- 172.21.15.70:0/2338073107 <== mon.0 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5629b030da40 con 0x5629b10dc400 2024-09-21T10:08:58.128 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:58 smithi070 bash[143062]: 862f41fda2c3a1ede15feb1b50d8d410e975f82fec81188500e80680ac209fd2 2024-09-21T10:08:58.128 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:58 smithi070 systemd[1]: Started Ceph osd.3 for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:08:59.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:59 smithi160 ceph-mon[101879]: pgmap v593: 521 pgs: 160 active+undersized, 60 active+undersized+degraded, 301 active+clean; 60 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 3.7 MiB/s wr, 38 op/s; 101/717 objects degraded (14.086%) 2024-09-21T10:08:59.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:59 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:59.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:59 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:59.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:59 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:59.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:59 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:08:59.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:59 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:59.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:08:59 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:59.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:59 smithi070 ceph-mon[107385]: pgmap v593: 521 pgs: 160 active+undersized, 60 active+undersized+degraded, 301 active+clean; 60 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 3.7 MiB/s wr, 38 op/s; 101/717 objects degraded (14.086%) 2024-09-21T10:08:59.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:59.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:59.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:59.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:08:59.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:59.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:08:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:59.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:59 smithi070 ceph-mon[112139]: pgmap v593: 521 pgs: 160 active+undersized, 60 active+undersized+degraded, 301 active+clean; 60 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 3.7 MiB/s wr, 38 op/s; 101/717 objects degraded (14.086%) 2024-09-21T10:08:59.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:59 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:59.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:59 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:59.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:59 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:59.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:59 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:08:59.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:59 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:08:59.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:08:59 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:00.230 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:00 smithi070 ceph-mon[107385]: pgmap v594: 521 pgs: 160 active+undersized, 60 active+undersized+degraded, 301 active+clean; 64 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.0 MiB/s wr, 10 op/s; 101/720 objects degraded (14.028%) 2024-09-21T10:09:00.230 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:00 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 101/720 objects degraded (14.028%), 60 pgs degraded (PG_DEGRADED) 2024-09-21T10:09:00.231 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:08:59 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3[143088]: 2024-09-21T10:08:59.930+0000 7fa7cceb6640 -1 Falling back to public interface 2024-09-21T10:09:00.231 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:00 smithi070 ceph-mon[112139]: pgmap v594: 521 pgs: 160 active+undersized, 60 active+undersized+degraded, 301 active+clean; 64 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.0 MiB/s wr, 10 op/s; 101/720 objects degraded (14.028%) 2024-09-21T10:09:00.231 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:00 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 101/720 objects degraded (14.028%), 60 pgs degraded (PG_DEGRADED) 2024-09-21T10:09:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:00 smithi160 ceph-mon[101879]: pgmap v594: 521 pgs: 160 active+undersized, 60 active+undersized+degraded, 301 active+clean; 64 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.0 MiB/s wr, 10 op/s; 101/720 objects degraded (14.028%) 2024-09-21T10:09:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:00 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 101/720 objects degraded (14.028%), 60 pgs degraded (PG_DEGRADED) 2024-09-21T10:09:02.130 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.LUKS1UnderLUKS2WithoutResize (15872 ms) 2024-09-21T10:09:02.130 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.LUKS2UnderLUKS1WithoutResize 2024-09-21T10:09:02.131 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:09:03.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:03 smithi070 ceph-mon[107385]: pgmap v595: 521 pgs: 160 active+undersized, 60 active+undersized+degraded, 301 active+clean; 76 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2.0 MiB/s wr, 9 op/s; 102/729 objects degraded (13.992%) 2024-09-21T10:09:03.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:03 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:03.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:03 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:03.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:03 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:03.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:03 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:03.378 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:09:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:09:03] "GET /metrics HTTP/1.1" 200 67774 "" "Prometheus/2.51.0" 2024-09-21T10:09:03.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:03 smithi070 ceph-mon[112139]: pgmap v595: 521 pgs: 160 active+undersized, 60 active+undersized+degraded, 301 active+clean; 76 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2.0 MiB/s wr, 9 op/s; 102/729 objects degraded (13.992%) 2024-09-21T10:09:03.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:03 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:03.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:03 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:03.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:03 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:03.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:03 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:03.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:03 smithi160 ceph-mon[101879]: pgmap v595: 521 pgs: 160 active+undersized, 60 active+undersized+degraded, 301 active+clean; 76 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2.0 MiB/s wr, 9 op/s; 102/729 objects degraded (13.992%) 2024-09-21T10:09:03.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:03 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:03.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:03 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:03.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:03 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:03.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:03 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:03.954 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-09-21T10:09:04.127 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:09:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3[143088]: 2024-09-21T10:09:03.729+0000 7fa7cceb6640 -1 osd.3 689 log_to_monitors true 2024-09-21T10:09:04.489 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:04 smithi070 ceph-mon[107385]: from='osd.3 [v2:172.21.15.70:6826/3858266770,v1:172.21.15.70:6827/3858266770]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T10:09:04.489 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:04 smithi070 ceph-mon[107385]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T10:09:04.490 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:04 smithi070 ceph-mon[112139]: from='osd.3 [v2:172.21.15.70:6826/3858266770,v1:172.21.15.70:6827/3858266770]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T10:09:04.490 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:04 smithi070 ceph-mon[112139]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T10:09:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:04 smithi160 ceph-mon[101879]: from='osd.3 [v2:172.21.15.70:6826/3858266770,v1:172.21.15.70:6827/3858266770]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T10:09:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:04 smithi160 ceph-mon[101879]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T10:09:04.715 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T10:09:04.715 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (9m) 2s ago 15m 19.5M - 0.27.0 11f11916f8cd 89ae95182419 2024-09-21T10:09:04.715 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi160 *:3000 running (9m) 37s ago 14m 83.2M - 10.4.8 5aad1d7cf004 daa2d8dfab48 2024-09-21T10:09:04.715 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi070.gansda smithi070 running (14m) 2s ago 14m 24.2M - 18.2.4-871-g6a1bdf85 0adab276c97e 6016b2501c8a 2024-09-21T10:09:04.715 INFO:teuthology.orchestra.run.smithi070.stdout:mds.foo.smithi160.byyhfz smithi160 running (14m) 37s ago 14m 17.8M - 18.2.4-871-g6a1bdf85 0adab276c97e 93e108d91e65 2024-09-21T10:09:04.715 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi160 *:8443,9283,8765 running (10m) 37s ago 19m 459M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 d952652a9683 2024-09-21T10:09:04.715 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (11m) 2s ago 20m 538M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 e3d390893cff 2024-09-21T10:09:04.715 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (8m) 2s ago 21m 81.7M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 4a7b667a4033 2024-09-21T10:09:04.715 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi160 running (6m) 37s ago 19m 52.5M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 86e88af21f36 2024-09-21T10:09:04.716 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (8m) 2s ago 19m 64.5M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 e154380b80a2 2024-09-21T10:09:04.716 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (10m) 2s ago 15m 11.2M - 1.7.0 72c9c2088986 5d5d4b026070 2024-09-21T10:09:04.716 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi160 *:9100 running (10m) 37s ago 15m 11.3M - 1.7.0 72c9c2088986 19b8e0e8ee0c 2024-09-21T10:09:04.716 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (111s) 2s ago 18m 74.6M 2480M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 eedacb586f7c 2024-09-21T10:09:04.716 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (77s) 2s ago 18m 69.7M 2480M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 853f3bb7fa25 2024-09-21T10:09:04.716 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (41s) 2s ago 18m 78.3M 2480M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 014c9efbc75d 2024-09-21T10:09:04.716 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (6s) 2s ago 17m 11.2M 2480M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 862f41fda2c3 2024-09-21T10:09:04.716 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi160 running (17m) 37s ago 17m 136M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 7325a075c488 2024-09-21T10:09:04.716 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi160 running (16m) 37s ago 16m 120M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e f74026345641 2024-09-21T10:09:04.716 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi160 running (16m) 37s ago 16m 150M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e 03e6bb54f8b2 2024-09-21T10:09:04.716 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi160 running (16m) 37s ago 16m 134M 2480M 18.2.4-871-g6a1bdf85 0adab276c97e e405b21a6146 2024-09-21T10:09:04.716 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi160 *:9095 running (9m) 37s ago 15m 58.3M - 2.51.0 1d3b7f56885b 648952e57ad7 2024-09-21T10:09:04.764 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 10:09:04 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3[143088]: 2024-09-21T10:09:04.486+0000 7fa7c0c58640 -1 osd.3 689 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T10:09:05.118 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:09:05.119 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-09-21T10:09:05.119 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 3 2024-09-21T10:09:05.119 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:09:05.119 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-09-21T10:09:05.119 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T10:09:05.119 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:09:05.119 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-09-21T10:09:05.119 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 4, 2024-09-21T10:09:05.119 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 3 2024-09-21T10:09:05.119 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:09:05.120 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": { 2024-09-21T10:09:05.120 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 2 2024-09-21T10:09:05.120 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:09:05.120 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-09-21T10:09:05.120 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 6, 2024-09-21T10:09:05.120 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 8 2024-09-21T10:09:05.120 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:09:05.120 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:09:05.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:05 smithi070 ceph-mon[107385]: from='client.34611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:09:05.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:05 smithi070 ceph-mon[107385]: pgmap v596: 521 pgs: 160 active+undersized, 60 active+undersized+degraded, 301 active+clean; 76 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 1.8 MiB/s wr, 8 op/s; 103/732 objects degraded (14.071%) 2024-09-21T10:09:05.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:05 smithi070 ceph-mon[107385]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T10:09:05.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:05 smithi070 ceph-mon[107385]: osdmap e691: 8 total, 7 up, 8 in 2024-09-21T10:09:05.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:05 smithi070 ceph-mon[107385]: from='osd.3 [v2:172.21.15.70:6826/3858266770,v1:172.21.15.70:6827/3858266770]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]} : dispatch 2024-09-21T10:09:05.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:05 smithi070 ceph-mon[107385]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]} : dispatch 2024-09-21T10:09:05.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:05 smithi070 ceph-mon[107385]: from='osd.3 ' entity='osd.3' 2024-09-21T10:09:05.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:05 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/79023735' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:05.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:05 smithi070 ceph-mon[112139]: from='client.34611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:09:05.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:05 smithi070 ceph-mon[112139]: pgmap v596: 521 pgs: 160 active+undersized, 60 active+undersized+degraded, 301 active+clean; 76 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 1.8 MiB/s wr, 8 op/s; 103/732 objects degraded (14.071%) 2024-09-21T10:09:05.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:05 smithi070 ceph-mon[112139]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T10:09:05.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:05 smithi070 ceph-mon[112139]: osdmap e691: 8 total, 7 up, 8 in 2024-09-21T10:09:05.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:05 smithi070 ceph-mon[112139]: from='osd.3 [v2:172.21.15.70:6826/3858266770,v1:172.21.15.70:6827/3858266770]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]} : dispatch 2024-09-21T10:09:05.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:05 smithi070 ceph-mon[112139]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]} : dispatch 2024-09-21T10:09:05.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:05 smithi070 ceph-mon[112139]: from='osd.3 ' entity='osd.3' 2024-09-21T10:09:05.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:05 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/79023735' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:05.575 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:09:05.575 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47", 2024-09-21T10:09:05.575 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-09-21T10:09:05.575 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 4 daemons (0 remaining).", 2024-09-21T10:09:05.575 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-09-21T10:09:05.576 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "4/8 daemons upgraded", 2024-09-21T10:09:05.576 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading osd daemons", 2024-09-21T10:09:05.576 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-09-21T10:09:05.576 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:09:05.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:05 smithi160 ceph-mon[101879]: from='client.34611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:09:05.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:05 smithi160 ceph-mon[101879]: pgmap v596: 521 pgs: 160 active+undersized, 60 active+undersized+degraded, 301 active+clean; 76 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 1.8 MiB/s wr, 8 op/s; 103/732 objects degraded (14.071%) 2024-09-21T10:09:05.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:05 smithi160 ceph-mon[101879]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T10:09:05.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:05 smithi160 ceph-mon[101879]: osdmap e691: 8 total, 7 up, 8 in 2024-09-21T10:09:05.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:05 smithi160 ceph-mon[101879]: from='osd.3 [v2:172.21.15.70:6826/3858266770,v1:172.21.15.70:6827/3858266770]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]} : dispatch 2024-09-21T10:09:05.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:05 smithi160 ceph-mon[101879]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]} : dispatch 2024-09-21T10:09:05.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:05 smithi160 ceph-mon[101879]: from='osd.3 ' entity='osd.3' 2024-09-21T10:09:05.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:05 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/79023735' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:06.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:06 smithi070 ceph-mon[107385]: from='client.44552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:09:06.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:06 smithi070 ceph-mon[107385]: from='client.44558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:09:06.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:06.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:06.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:06 smithi070 ceph-mon[107385]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:09:06.502 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:06 smithi070 ceph-mon[107385]: osd.3 [v2:172.21.15.70:6826/3858266770,v1:172.21.15.70:6827/3858266770] boot 2024-09-21T10:09:06.502 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:06 smithi070 ceph-mon[107385]: osdmap e692: 8 total, 8 up, 8 in 2024-09-21T10:09:06.502 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T10:09:06.502 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:06 smithi070 ceph-mon[112139]: from='client.44552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:09:06.502 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:06 smithi070 ceph-mon[112139]: from='client.44558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:09:06.502 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:06.503 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:06.503 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:06 smithi070 ceph-mon[112139]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:09:06.503 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:06 smithi070 ceph-mon[112139]: osd.3 [v2:172.21.15.70:6826/3858266770,v1:172.21.15.70:6827/3858266770] boot 2024-09-21T10:09:06.503 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:06 smithi070 ceph-mon[112139]: osdmap e692: 8 total, 8 up, 8 in 2024-09-21T10:09:06.503 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T10:09:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:06 smithi160 ceph-mon[101879]: from='client.44552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:09:06.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:06 smithi160 ceph-mon[101879]: from='client.44558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:09:06.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:06.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:06.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:06 smithi160 ceph-mon[101879]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:09:06.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:06 smithi160 ceph-mon[101879]: osd.3 [v2:172.21.15.70:6826/3858266770,v1:172.21.15.70:6827/3858266770] boot 2024-09-21T10:09:06.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:06 smithi160 ceph-mon[101879]: osdmap e692: 8 total, 8 up, 8 in 2024-09-21T10:09:06.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T10:09:07.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:09:07.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: pgmap v599: 521 pgs: 78 peering, 104 active+undersized, 38 active+undersized+degraded, 301 active+clean; 76 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 11 op/s; 68/738 objects degraded (9.214%) 2024-09-21T10:09:07.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 68/738 objects degraded (9.214%), 38 pgs degraded (PG_DEGRADED) 2024-09-21T10:09:07.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: osdmap e693: 8 total, 8 up, 8 in 2024-09-21T10:09:07.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:07.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:07.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:09:07.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:09:07.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:07.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:09:07.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:07.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:07.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:07.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:07.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:07.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:07.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:07.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:07.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:07.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-21T10:09:07.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-21T10:09:07.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:09:07.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-21T10:09:07.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-21T10:09:07.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-21T10:09:07.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-21T10:09:07.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-21T10:09:07.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-21T10:09:07.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-21T10:09:07.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:09:07.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-21T10:09:07.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-21T10:09:07.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-21T10:09:07.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-21T10:09:07.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-21T10:09:07.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:09:07.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:09:07.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:09:07.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:09:07.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:09:07.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:09:07.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-21T10:09:07.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-21T10:09:07.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:09:07.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:09:07.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:09:07.595 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:07.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:09:07.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:09:07.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:09:07.596 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:07.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:09:07.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: pgmap v599: 521 pgs: 78 peering, 104 active+undersized, 38 active+undersized+degraded, 301 active+clean; 76 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 11 op/s; 68/738 objects degraded (9.214%) 2024-09-21T10:09:07.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 68/738 objects degraded (9.214%), 38 pgs degraded (PG_DEGRADED) 2024-09-21T10:09:07.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: osdmap e693: 8 total, 8 up, 8 in 2024-09-21T10:09:07.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:07.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:07.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:09:07.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:09:07.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:07.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:09:07.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:07.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:07.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:07.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:07.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:07.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:07.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:07.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:07.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:07.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-21T10:09:07.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-21T10:09:07.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:09:07.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-21T10:09:07.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-21T10:09:07.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-21T10:09:07.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-21T10:09:07.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:09:07.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: pgmap v599: 521 pgs: 78 peering, 104 active+undersized, 38 active+undersized+degraded, 301 active+clean; 76 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 11 op/s; 68/738 objects degraded (9.214%) 2024-09-21T10:09:07.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 68/738 objects degraded (9.214%), 38 pgs degraded (PG_DEGRADED) 2024-09-21T10:09:07.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: osdmap e693: 8 total, 8 up, 8 in 2024-09-21T10:09:07.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:07.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:07.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:09:07.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:09:07.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:07.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:09:07.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:07.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:07.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:07.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:07.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:07.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:07.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:07.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:07.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:07.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-21T10:09:07.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-21T10:09:07.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:09:07.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-21T10:09:07.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-21T10:09:07.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-21T10:09:07.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-21T10:09:07.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-21T10:09:07.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-21T10:09:07.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-21T10:09:07.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:09:07.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-21T10:09:07.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-21T10:09:07.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-21T10:09:07.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-21T10:09:07.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-21T10:09:07.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:09:07.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:09:07.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:09:07.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:09:07.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:09:07.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:09:07.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-21T10:09:07.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-21T10:09:07.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:09:07.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:09:07.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:09:07.634 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:07.634 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:09:07.634 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:09:07.634 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:09:07.634 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:07.634 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-21T10:09:07.634 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-21T10:09:07.634 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-21T10:09:07.634 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:09:07.634 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-21T10:09:07.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-21T10:09:07.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-21T10:09:07.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-21T10:09:07.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-21T10:09:07.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:09:07.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:09:07.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:09:07.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:09:07.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:09:07.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-21T10:09:07.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-21T10:09:07.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-21T10:09:07.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:09:07.635 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:09:07.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:09:07.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:07.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:09:07.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:09:07.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:09:07.636 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:08 smithi160 ceph-mon[101879]: Upgrade: Setting container_image for all nvmeof 2024-09-21T10:09:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:08 smithi160 ceph-mon[101879]: Upgrade: Finalizing container_image settings 2024-09-21T10:09:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:08 smithi160 ceph-mon[101879]: Upgrade: Complete! 2024-09-21T10:09:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:08 smithi160 ceph-mon[101879]: osdmap e694: 8 total, 8 up, 8 in 2024-09-21T10:09:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:08 smithi070 ceph-mon[107385]: Upgrade: Setting container_image for all nvmeof 2024-09-21T10:09:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:08 smithi070 ceph-mon[107385]: Upgrade: Finalizing container_image settings 2024-09-21T10:09:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:08 smithi070 ceph-mon[107385]: Upgrade: Complete! 2024-09-21T10:09:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:08 smithi070 ceph-mon[107385]: osdmap e694: 8 total, 8 up, 8 in 2024-09-21T10:09:08.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:08 smithi070 ceph-mon[112139]: Upgrade: Setting container_image for all nvmeof 2024-09-21T10:09:08.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:08 smithi070 ceph-mon[112139]: Upgrade: Finalizing container_image settings 2024-09-21T10:09:08.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:08 smithi070 ceph-mon[112139]: Upgrade: Complete! 2024-09-21T10:09:08.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:08 smithi070 ceph-mon[112139]: osdmap e694: 8 total, 8 up, 8 in 2024-09-21T10:09:09.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:09 smithi160 ceph-mon[101879]: pgmap v602: 521 pgs: 78 peering, 104 active+undersized, 38 active+undersized+degraded, 301 active+clean; 76 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 3.0 MiB/s rd, 767 B/s wr, 9 op/s; 68/738 objects degraded (9.214%) 2024-09-21T10:09:09.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:09 smithi070 ceph-mon[107385]: pgmap v602: 521 pgs: 78 peering, 104 active+undersized, 38 active+undersized+degraded, 301 active+clean; 76 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 3.0 MiB/s rd, 767 B/s wr, 9 op/s; 68/738 objects degraded (9.214%) 2024-09-21T10:09:09.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:09 smithi070 ceph-mon[112139]: pgmap v602: 521 pgs: 78 peering, 104 active+undersized, 38 active+undersized+degraded, 301 active+clean; 76 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 3.0 MiB/s rd, 767 B/s wr, 9 op/s; 68/738 objects degraded (9.214%) 2024-09-21T10:09:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:10 smithi160 ceph-mon[101879]: pgmap v603: 521 pgs: 78 peering, 443 active+clean; 101 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 4.2 MiB/s wr, 37 op/s; 7 B/s, 0 objects/s recovering 2024-09-21T10:09:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:10 smithi160 ceph-mon[101879]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 68/738 objects degraded (9.214%), 38 pgs degraded) 2024-09-21T10:09:10.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:10 smithi070 ceph-mon[107385]: pgmap v603: 521 pgs: 78 peering, 443 active+clean; 101 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 4.2 MiB/s wr, 37 op/s; 7 B/s, 0 objects/s recovering 2024-09-21T10:09:10.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:10 smithi070 ceph-mon[107385]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 68/738 objects degraded (9.214%), 38 pgs degraded) 2024-09-21T10:09:10.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:10 smithi070 ceph-mon[112139]: pgmap v603: 521 pgs: 78 peering, 443 active+clean; 101 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 4.2 MiB/s wr, 37 op/s; 7 B/s, 0 objects/s recovering 2024-09-21T10:09:10.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:10 smithi070 ceph-mon[112139]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 68/738 objects degraded (9.214%), 38 pgs degraded) 2024-09-21T10:09:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:11 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:11.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:11 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:11.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:11 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:12 smithi160 ceph-mon[101879]: pgmap v604: 521 pgs: 521 active+clean; 117 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 6.2 MiB/s wr, 59 op/s; 620 KiB/s, 7 keys/s, 0 objects/s recovering 2024-09-21T10:09:12.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:12 smithi070 ceph-mon[112139]: pgmap v604: 521 pgs: 521 active+clean; 117 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 6.2 MiB/s wr, 59 op/s; 620 KiB/s, 7 keys/s, 0 objects/s recovering 2024-09-21T10:09:12.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:12 smithi070 ceph-mon[107385]: pgmap v604: 521 pgs: 521 active+clean; 117 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 6.2 MiB/s wr, 59 op/s; 620 KiB/s, 7 keys/s, 0 objects/s recovering 2024-09-21T10:09:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:09:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:09:13] "GET /metrics HTTP/1.1" 200 67643 "" "Prometheus/2.51.0" 2024-09-21T10:09:14.541 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.LUKS2UnderLUKS1WithoutResize (12411 ms) 2024-09-21T10:09:14.542 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.EncryptionFormatNoData 2024-09-21T10:09:14.542 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:09:15.544 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:15 smithi160 ceph-mon[101879]: pgmap v605: 521 pgs: 521 active+clean; 117 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 159 KiB/s rd, 5.1 MiB/s wr, 43 op/s; 512 KiB/s, 6 keys/s, 0 objects/s recovering 2024-09-21T10:09:15.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:15 smithi070 ceph-mon[107385]: pgmap v605: 521 pgs: 521 active+clean; 117 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 159 KiB/s rd, 5.1 MiB/s wr, 43 op/s; 512 KiB/s, 6 keys/s, 0 objects/s recovering 2024-09-21T10:09:15.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:15 smithi070 ceph-mon[112139]: pgmap v605: 521 pgs: 521 active+clean; 117 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 159 KiB/s rd, 5.1 MiB/s wr, 43 op/s; 512 KiB/s, 6 keys/s, 0 objects/s recovering 2024-09-21T10:09:17.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:17 smithi160 ceph-mon[101879]: pgmap v606: 521 pgs: 521 active+clean; 121 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.4 MiB/s rd, 4.7 MiB/s wr, 45 op/s; 426 KiB/s, 5 keys/s, 0 objects/s recovering 2024-09-21T10:09:17.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:17 smithi070 ceph-mon[107385]: pgmap v606: 521 pgs: 521 active+clean; 121 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.4 MiB/s rd, 4.7 MiB/s wr, 45 op/s; 426 KiB/s, 5 keys/s, 0 objects/s recovering 2024-09-21T10:09:17.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:17 smithi070 ceph-mon[112139]: pgmap v606: 521 pgs: 521 active+clean; 121 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.4 MiB/s rd, 4.7 MiB/s wr, 45 op/s; 426 KiB/s, 5 keys/s, 0 objects/s recovering 2024-09-21T10:09:18.711 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.EncryptionFormatNoData (4170 ms) 2024-09-21T10:09:18.711 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.EncryptionLoadBadSize 2024-09-21T10:09:18.711 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:09:19.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:19 smithi160 ceph-mon[101879]: pgmap v607: 521 pgs: 521 active+clean; 121 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 42 op/s; 394 KiB/s, 4 keys/s, 0 objects/s recovering 2024-09-21T10:09:19.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:19 smithi070 ceph-mon[107385]: pgmap v607: 521 pgs: 521 active+clean; 121 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 42 op/s; 394 KiB/s, 4 keys/s, 0 objects/s recovering 2024-09-21T10:09:19.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:19 smithi070 ceph-mon[112139]: pgmap v607: 521 pgs: 521 active+clean; 121 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 42 op/s; 394 KiB/s, 4 keys/s, 0 objects/s recovering 2024-09-21T10:09:21.429 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:21 smithi070 ceph-mon[107385]: pgmap v608: 521 pgs: 521 active+clean; 121 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.8 MiB/s wr, 39 op/s; 341 KiB/s, 4 keys/s, 0 objects/s recovering 2024-09-21T10:09:21.429 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:21.429 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:21.429 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:21.429 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:21 smithi070 ceph-mon[112139]: pgmap v608: 521 pgs: 521 active+clean; 121 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.8 MiB/s wr, 39 op/s; 341 KiB/s, 4 keys/s, 0 objects/s recovering 2024-09-21T10:09:21.429 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:21.429 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:21.430 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:21 smithi160 ceph-mon[101879]: pgmap v608: 521 pgs: 521 active+clean; 121 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 3.8 MiB/s wr, 39 op/s; 341 KiB/s, 4 keys/s, 0 objects/s recovering 2024-09-21T10:09:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:09:23.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:23 smithi160 ceph-mon[101879]: pgmap v609: 521 pgs: 521 active+clean; 125 MiB data, 792 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 26 op/s; 341 KiB/s, 4 keys/s, 0 objects/s recovering 2024-09-21T10:09:23.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:23 smithi070 ceph-mon[107385]: pgmap v609: 521 pgs: 521 active+clean; 125 MiB data, 792 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 26 op/s; 341 KiB/s, 4 keys/s, 0 objects/s recovering 2024-09-21T10:09:23.628 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:09:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:09:23] "GET /metrics HTTP/1.1" 200 67788 "" "Prometheus/2.51.0" 2024-09-21T10:09:23.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:23 smithi070 ceph-mon[112139]: pgmap v609: 521 pgs: 521 active+clean; 125 MiB data, 792 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 26 op/s; 341 KiB/s, 4 keys/s, 0 objects/s recovering 2024-09-21T10:09:24.309 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.EncryptionLoadBadSize (5598 ms) 2024-09-21T10:09:24.310 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.EncryptionLoadBadStripePattern 2024-09-21T10:09:24.310 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/test_librbd.cc:2905: Skipped 2024-09-21T10:09:24.310 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T10:09:24.310 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadBadStripePattern (0 ms) 2024-09-21T10:09:24.310 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.EncryptionLoadFormatMismatch 2024-09-21T10:09:24.310 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:09:25.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:25 smithi160 ceph-mon[101879]: pgmap v610: 521 pgs: 521 active+clean; 125 MiB data, 800 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 684 KiB/s wr, 18 op/s 2024-09-21T10:09:25.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:25 smithi070 ceph-mon[107385]: pgmap v610: 521 pgs: 521 active+clean; 125 MiB data, 800 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 684 KiB/s wr, 18 op/s 2024-09-21T10:09:25.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:25 smithi070 ceph-mon[112139]: pgmap v610: 521 pgs: 521 active+clean; 125 MiB data, 800 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 684 KiB/s wr, 18 op/s 2024-09-21T10:09:27.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:27 smithi160 ceph-mon[101879]: pgmap v611: 521 pgs: 521 active+clean; 129 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.0 MiB/s wr, 19 op/s 2024-09-21T10:09:27.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:27 smithi070 ceph-mon[107385]: pgmap v611: 521 pgs: 521 active+clean; 129 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.0 MiB/s wr, 19 op/s 2024-09-21T10:09:27.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:27 smithi070 ceph-mon[112139]: pgmap v611: 521 pgs: 521 active+clean; 129 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.0 MiB/s wr, 19 op/s 2024-09-21T10:09:29.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:29 smithi160 ceph-mon[101879]: pgmap v612: 521 pgs: 521 active+clean; 129 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 683 KiB/s wr, 12 op/s 2024-09-21T10:09:29.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:29 smithi160 ceph-mon[101879]: osdmap e695: 8 total, 8 up, 8 in 2024-09-21T10:09:29.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:29 smithi070 ceph-mon[107385]: pgmap v612: 521 pgs: 521 active+clean; 129 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 683 KiB/s wr, 12 op/s 2024-09-21T10:09:29.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:29 smithi070 ceph-mon[107385]: osdmap e695: 8 total, 8 up, 8 in 2024-09-21T10:09:29.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:29 smithi070 ceph-mon[112139]: pgmap v612: 521 pgs: 521 active+clean; 129 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 683 KiB/s wr, 12 op/s 2024-09-21T10:09:29.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:29 smithi070 ceph-mon[112139]: osdmap e695: 8 total, 8 up, 8 in 2024-09-21T10:09:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:30 smithi160 ceph-mon[101879]: osdmap e696: 8 total, 8 up, 8 in 2024-09-21T10:09:30.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:30 smithi070 ceph-mon[112139]: osdmap e696: 8 total, 8 up, 8 in 2024-09-21T10:09:30.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:30 smithi070 ceph-mon[107385]: osdmap e696: 8 total, 8 up, 8 in 2024-09-21T10:09:31.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:31 smithi160 ceph-mon[101879]: pgmap v615: 521 pgs: 521 active+clean; 129 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 112 KiB/s rd, 514 KiB/s wr, 52 op/s 2024-09-21T10:09:31.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:31 smithi070 ceph-mon[107385]: pgmap v615: 521 pgs: 521 active+clean; 129 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 112 KiB/s rd, 514 KiB/s wr, 52 op/s 2024-09-21T10:09:31.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:31 smithi070 ceph-mon[112139]: pgmap v615: 521 pgs: 521 active+clean; 129 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 112 KiB/s rd, 514 KiB/s wr, 52 op/s 2024-09-21T10:09:33.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:33 smithi160 ceph-mon[101879]: pgmap v616: 521 pgs: 521 active+clean; 133 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 115 KiB/s rd, 1.0 MiB/s wr, 58 op/s 2024-09-21T10:09:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:09:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:09:33] "GET /metrics HTTP/1.1" 200 67788 "" "Prometheus/2.51.0" 2024-09-21T10:09:33.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:33 smithi070 ceph-mon[112139]: pgmap v616: 521 pgs: 521 active+clean; 133 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 115 KiB/s rd, 1.0 MiB/s wr, 58 op/s 2024-09-21T10:09:33.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:33 smithi070 ceph-mon[107385]: pgmap v616: 521 pgs: 521 active+clean; 133 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 115 KiB/s rd, 1.0 MiB/s wr, 58 op/s 2024-09-21T10:09:35.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:35 smithi160 ceph-mon[101879]: pgmap v617: 521 pgs: 521 active+clean; 133 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 62 KiB/s rd, 515 KiB/s wr, 79 op/s 2024-09-21T10:09:35.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:35 smithi070 ceph-mon[107385]: pgmap v617: 521 pgs: 521 active+clean; 133 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 62 KiB/s rd, 515 KiB/s wr, 79 op/s 2024-09-21T10:09:35.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:35 smithi070 ceph-mon[112139]: pgmap v617: 521 pgs: 521 active+clean; 133 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 62 KiB/s rd, 515 KiB/s wr, 79 op/s 2024-09-21T10:09:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:36.690 DEBUG:teuthology.parallel:result is None 2024-09-21T10:09:37.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:37 smithi160 ceph-mon[101879]: from='client.44576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:09:37.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:37 smithi160 ceph-mon[101879]: pgmap v618: 521 pgs: 521 active+clean; 133 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 62 KiB/s rd, 515 KiB/s wr, 79 op/s 2024-09-21T10:09:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:37 smithi070 ceph-mon[107385]: from='client.44576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:09:37.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:37 smithi070 ceph-mon[107385]: pgmap v618: 521 pgs: 521 active+clean; 133 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 62 KiB/s rd, 515 KiB/s wr, 79 op/s 2024-09-21T10:09:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:37 smithi070 ceph-mon[112139]: from='client.44576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:09:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:37 smithi070 ceph-mon[112139]: pgmap v618: 521 pgs: 521 active+clean; 133 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 62 KiB/s rd, 515 KiB/s wr, 79 op/s 2024-09-21T10:09:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:38 smithi160 ceph-mon[101879]: pgmap v619: 521 pgs: 521 active+clean; 133 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 50 KiB/s rd, 415 KiB/s wr, 64 op/s 2024-09-21T10:09:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:38 smithi070 ceph-mon[107385]: pgmap v619: 521 pgs: 521 active+clean; 133 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 50 KiB/s rd, 415 KiB/s wr, 64 op/s 2024-09-21T10:09:38.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:38 smithi070 ceph-mon[112139]: pgmap v619: 521 pgs: 521 active+clean; 133 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 50 KiB/s rd, 415 KiB/s wr, 64 op/s 2024-09-21T10:09:41.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:41 smithi160 ceph-mon[101879]: pgmap v620: 521 pgs: 521 active+clean; 137 MiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 751 KiB/s wr, 38 op/s 2024-09-21T10:09:41.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:41 smithi070 ceph-mon[107385]: pgmap v620: 521 pgs: 521 active+clean; 137 MiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 751 KiB/s wr, 38 op/s 2024-09-21T10:09:41.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:41 smithi070 ceph-mon[112139]: pgmap v620: 521 pgs: 521 active+clean; 137 MiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 751 KiB/s wr, 38 op/s 2024-09-21T10:09:43.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:43 smithi160 ceph-mon[101879]: pgmap v621: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 39 op/s 2024-09-21T10:09:43.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:43 smithi070 ceph-mon[112139]: pgmap v621: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 39 op/s 2024-09-21T10:09:43.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:43 smithi070 ceph-mon[107385]: pgmap v621: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 39 op/s 2024-09-21T10:09:43.628 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:09:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:09:43] "GET /metrics HTTP/1.1" 200 67780 "" "Prometheus/2.51.0" 2024-09-21T10:09:45.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:45 smithi160 ceph-mon[101879]: pgmap v622: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.3 MiB/s wr, 28 op/s 2024-09-21T10:09:45.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:45 smithi070 ceph-mon[107385]: pgmap v622: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.3 MiB/s wr, 28 op/s 2024-09-21T10:09:45.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:45 smithi070 ceph-mon[112139]: pgmap v622: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.3 MiB/s wr, 28 op/s 2024-09-21T10:09:47.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:47 smithi160 ceph-mon[101879]: pgmap v623: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2024-09-21T10:09:47.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:47 smithi070 ceph-mon[112139]: pgmap v623: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2024-09-21T10:09:47.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:47 smithi070 ceph-mon[107385]: pgmap v623: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2024-09-21T10:09:49.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:49 smithi160 ceph-mon[101879]: pgmap v624: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2024-09-21T10:09:49.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:49 smithi070 ceph-mon[112139]: pgmap v624: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2024-09-21T10:09:49.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:49 smithi070 ceph-mon[107385]: pgmap v624: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2024-09-21T10:09:51.429 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:51 smithi070 ceph-mon[107385]: pgmap v625: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-09-21T10:09:51.429 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:51.429 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:51 smithi070 ceph-mon[112139]: pgmap v625: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-09-21T10:09:51.429 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:51 smithi160 ceph-mon[101879]: pgmap v625: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-09-21T10:09:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:53.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:53 smithi160 ceph-mon[101879]: pgmap v626: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 1024 KiB/s wr, 9 op/s 2024-09-21T10:09:53.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:53 smithi070 ceph-mon[107385]: pgmap v626: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 1024 KiB/s wr, 9 op/s 2024-09-21T10:09:53.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:09:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:09:53] "GET /metrics HTTP/1.1" 200 67778 "" "Prometheus/2.51.0" 2024-09-21T10:09:53.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:53 smithi070 ceph-mon[112139]: pgmap v626: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 1024 KiB/s wr, 9 op/s 2024-09-21T10:09:55.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:55 smithi160 ceph-mon[101879]: pgmap v627: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 4.8 MiB/s rd, 2 op/s 2024-09-21T10:09:55.592 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 10:09:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=cleanup t=2024-09-21T10:09:55.1334402Z level=info msg="Completed cleanup jobs" duration=16.953995ms 2024-09-21T10:09:55.592 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 10:09:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=plugins.update.checker t=2024-09-21T10:09:55.283731189Z level=info msg="Update check succeeded" duration=78.211869ms 2024-09-21T10:09:55.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:55 smithi070 ceph-mon[107385]: pgmap v627: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 4.8 MiB/s rd, 2 op/s 2024-09-21T10:09:55.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:55 smithi070 ceph-mon[112139]: pgmap v627: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 4.8 MiB/s rd, 2 op/s 2024-09-21T10:09:57.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:57 smithi160 ceph-mon[101879]: pgmap v628: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 3 op/s 2024-09-21T10:09:57.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:57 smithi070 ceph-mon[107385]: pgmap v628: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 3 op/s 2024-09-21T10:09:57.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:57 smithi070 ceph-mon[112139]: pgmap v628: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 3 op/s 2024-09-21T10:09:59.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:09:59 smithi160 ceph-mon[101879]: pgmap v629: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.4 MiB/s rd, 1 op/s 2024-09-21T10:09:59.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:09:59 smithi070 ceph-mon[107385]: pgmap v629: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.4 MiB/s rd, 1 op/s 2024-09-21T10:09:59.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:09:59 smithi070 ceph-mon[112139]: pgmap v629: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.4 MiB/s rd, 1 op/s 2024-09-21T10:10:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:00 smithi160 ceph-mon[101879]: Health detail: HEALTH_WARN 2 pool(s) full 2024-09-21T10:10:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:00 smithi160 ceph-mon[101879]: [WRN] POOL_FULL: 2 pool(s) full 2024-09-21T10:10:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:00 smithi160 ceph-mon[101879]: pool 'test-librbd-smithi070-86016-24' is full (running out of quota) 2024-09-21T10:10:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:00 smithi160 ceph-mon[101879]: pool 'test-librbd-smithi070-86016-27' is full (running out of quota) 2024-09-21T10:10:00.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:00 smithi070 ceph-mon[107385]: Health detail: HEALTH_WARN 2 pool(s) full 2024-09-21T10:10:00.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:00 smithi070 ceph-mon[107385]: [WRN] POOL_FULL: 2 pool(s) full 2024-09-21T10:10:00.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:00 smithi070 ceph-mon[107385]: pool 'test-librbd-smithi070-86016-24' is full (running out of quota) 2024-09-21T10:10:00.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:00 smithi070 ceph-mon[107385]: pool 'test-librbd-smithi070-86016-27' is full (running out of quota) 2024-09-21T10:10:00.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:00 smithi070 ceph-mon[112139]: Health detail: HEALTH_WARN 2 pool(s) full 2024-09-21T10:10:00.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:00 smithi070 ceph-mon[112139]: [WRN] POOL_FULL: 2 pool(s) full 2024-09-21T10:10:00.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:00 smithi070 ceph-mon[112139]: pool 'test-librbd-smithi070-86016-24' is full (running out of quota) 2024-09-21T10:10:00.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:00 smithi070 ceph-mon[112139]: pool 'test-librbd-smithi070-86016-27' is full (running out of quota) 2024-09-21T10:10:01.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:01 smithi160 ceph-mon[101879]: pgmap v630: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 1 op/s 2024-09-21T10:10:01.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:01 smithi070 ceph-mon[107385]: pgmap v630: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 1 op/s 2024-09-21T10:10:01.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:01 smithi070 ceph-mon[112139]: pgmap v630: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 1 op/s 2024-09-21T10:10:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:02 smithi160 ceph-mon[101879]: pgmap v631: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 1 op/s 2024-09-21T10:10:02.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:02 smithi070 ceph-mon[107385]: pgmap v631: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 1 op/s 2024-09-21T10:10:02.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:02 smithi070 ceph-mon[112139]: pgmap v631: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 1 op/s 2024-09-21T10:10:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:10:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:10:03] "GET /metrics HTTP/1.1" 200 67778 "" "Prometheus/2.51.0" 2024-09-21T10:10:05.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:05 smithi160 ceph-mon[101879]: pgmap v632: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 0 op/s 2024-09-21T10:10:05.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:05 smithi070 ceph-mon[107385]: pgmap v632: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 0 op/s 2024-09-21T10:10:05.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:05 smithi070 ceph-mon[112139]: pgmap v632: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 0 op/s 2024-09-21T10:10:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:06.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:07.506 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:07 smithi070 ceph-mon[107385]: pgmap v633: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s rd, 1 op/s 2024-09-21T10:10:07.506 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:10:07.507 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:10:07.507 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:10:07.507 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:07 smithi070 ceph-mon[112139]: pgmap v633: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s rd, 1 op/s 2024-09-21T10:10:07.507 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:10:07.507 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:10:07.507 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:10:07.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:07 smithi160 ceph-mon[101879]: pgmap v633: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s rd, 1 op/s 2024-09-21T10:10:07.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:10:07.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:10:07.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:10:09.440 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:09 smithi160 ceph-mon[101879]: pgmap v634: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 1 op/s 2024-09-21T10:10:09.497 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:09 smithi070 ceph-mon[107385]: pgmap v634: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 1 op/s 2024-09-21T10:10:09.497 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:09 smithi070 ceph-mon[112139]: pgmap v634: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 1 op/s 2024-09-21T10:10:11.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:11 smithi160 ceph-mon[101879]: pgmap v635: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 1 op/s 2024-09-21T10:10:11.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:11 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:10:11.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:11 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:10:11.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:11 smithi070 ceph-mon[107385]: pgmap v635: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 1 op/s 2024-09-21T10:10:11.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:11 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:10:11.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:11 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:10:11.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:11 smithi070 ceph-mon[112139]: pgmap v635: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 1 op/s 2024-09-21T10:10:11.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:11 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:10:11.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:11 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:10:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:12 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:10:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:12 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:10:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:12 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:10:12.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:12 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:10:12.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:12 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:10:12.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:12 smithi160 ceph-mon[101879]: pgmap v636: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 1 op/s 2024-09-21T10:10:12.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:12 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:10:12.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:12 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:10:12.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:12 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:10:12.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:12 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:10:12.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:12 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:10:12.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:12 smithi070 ceph-mon[107385]: pgmap v636: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 1 op/s 2024-09-21T10:10:12.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:12 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:10:12.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:12 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:10:12.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:12 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:10:12.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:12 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:10:12.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:12 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:10:12.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:12 smithi070 ceph-mon[112139]: pgmap v636: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 1 op/s 2024-09-21T10:10:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:10:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:10:13] "GET /metrics HTTP/1.1" 200 67779 "" "Prometheus/2.51.0" 2024-09-21T10:10:15.545 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:15 smithi160 ceph-mon[101879]: pgmap v637: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 0 op/s 2024-09-21T10:10:15.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:15 smithi070 ceph-mon[107385]: pgmap v637: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 0 op/s 2024-09-21T10:10:15.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:15 smithi070 ceph-mon[112139]: pgmap v637: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 0 op/s 2024-09-21T10:10:17.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:17 smithi160 ceph-mon[101879]: pgmap v638: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s rd, 4 op/s 2024-09-21T10:10:17.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:17 smithi070 ceph-mon[107385]: pgmap v638: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s rd, 4 op/s 2024-09-21T10:10:17.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:17 smithi070 ceph-mon[112139]: pgmap v638: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s rd, 4 op/s 2024-09-21T10:10:19.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:19 smithi160 ceph-mon[101879]: pgmap v639: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 464 KiB/s rd, 3 op/s 2024-09-21T10:10:19.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:19 smithi070 ceph-mon[112139]: pgmap v639: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 464 KiB/s rd, 3 op/s 2024-09-21T10:10:19.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:19 smithi070 ceph-mon[107385]: pgmap v639: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 464 KiB/s rd, 3 op/s 2024-09-21T10:10:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:21 smithi160 ceph-mon[101879]: pgmap v640: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 468 KiB/s rd, 8 op/s 2024-09-21T10:10:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:21.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:21 smithi070 ceph-mon[107385]: pgmap v640: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 468 KiB/s rd, 8 op/s 2024-09-21T10:10:21.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:21.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:21 smithi070 ceph-mon[112139]: pgmap v640: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 468 KiB/s rd, 8 op/s 2024-09-21T10:10:21.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:23.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:23 smithi160 ceph-mon[101879]: pgmap v641: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 470 KiB/s rd, 11 op/s 2024-09-21T10:10:23.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:10:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:10:23] "GET /metrics HTTP/1.1" 200 67777 "" "Prometheus/2.51.0" 2024-09-21T10:10:23.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:23 smithi070 ceph-mon[112139]: pgmap v641: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 470 KiB/s rd, 11 op/s 2024-09-21T10:10:23.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:23 smithi070 ceph-mon[107385]: pgmap v641: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 470 KiB/s rd, 11 op/s 2024-09-21T10:10:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:24 smithi160 ceph-mon[101879]: pgmap v642: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 109 KiB/s rd, 15 op/s 2024-09-21T10:10:24.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:24 smithi070 ceph-mon[107385]: pgmap v642: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 109 KiB/s rd, 15 op/s 2024-09-21T10:10:24.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:24 smithi070 ceph-mon[112139]: pgmap v642: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 109 KiB/s rd, 15 op/s 2024-09-21T10:10:27.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:27 smithi160 ceph-mon[101879]: pgmap v643: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 184 KiB/s rd, 19 op/s 2024-09-21T10:10:27.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:27 smithi070 ceph-mon[112139]: pgmap v643: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 184 KiB/s rd, 19 op/s 2024-09-21T10:10:27.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:27 smithi070 ceph-mon[107385]: pgmap v643: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 184 KiB/s rd, 19 op/s 2024-09-21T10:10:29.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:29 smithi160 ceph-mon[101879]: pgmap v644: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 109 KiB/s rd, 15 op/s 2024-09-21T10:10:29.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:29 smithi070 ceph-mon[107385]: pgmap v644: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 109 KiB/s rd, 15 op/s 2024-09-21T10:10:29.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:29 smithi070 ceph-mon[112139]: pgmap v644: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 109 KiB/s rd, 15 op/s 2024-09-21T10:10:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:30 smithi160 ceph-mon[101879]: pgmap v645: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 138 KiB/s rd, 85 B/s wr, 21 op/s 2024-09-21T10:10:30.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:30 smithi070 ceph-mon[107385]: pgmap v645: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 138 KiB/s rd, 85 B/s wr, 21 op/s 2024-09-21T10:10:30.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:30 smithi070 ceph-mon[112139]: pgmap v645: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 138 KiB/s rd, 85 B/s wr, 21 op/s 2024-09-21T10:10:33.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:33 smithi160 ceph-mon[101879]: pgmap v646: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 182 KiB/s rd, 85 B/s wr, 16 op/s 2024-09-21T10:10:33.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:33 smithi070 ceph-mon[107385]: pgmap v646: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 182 KiB/s rd, 85 B/s wr, 16 op/s 2024-09-21T10:10:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:10:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:10:33] "GET /metrics HTTP/1.1" 200 67777 "" "Prometheus/2.51.0" 2024-09-21T10:10:33.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:33 smithi070 ceph-mon[112139]: pgmap v646: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 182 KiB/s rd, 85 B/s wr, 16 op/s 2024-09-21T10:10:35.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:35 smithi160 ceph-mon[101879]: pgmap v647: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 160 KiB/s rd, 170 B/s wr, 19 op/s 2024-09-21T10:10:35.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:35 smithi070 ceph-mon[107385]: pgmap v647: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 160 KiB/s rd, 170 B/s wr, 19 op/s 2024-09-21T10:10:35.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:35 smithi070 ceph-mon[112139]: pgmap v647: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 160 KiB/s rd, 170 B/s wr, 19 op/s 2024-09-21T10:10:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:36 smithi160 ceph-mon[101879]: pgmap v648: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 230 KiB/s rd, 255 B/s wr, 17 op/s 2024-09-21T10:10:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:36 smithi070 ceph-mon[107385]: pgmap v648: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 230 KiB/s rd, 255 B/s wr, 17 op/s 2024-09-21T10:10:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:36 smithi070 ceph-mon[112139]: pgmap v648: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 230 KiB/s rd, 255 B/s wr, 17 op/s 2024-09-21T10:10:39.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:39 smithi160 ceph-mon[101879]: pgmap v649: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 155 KiB/s rd, 255 B/s wr, 14 op/s 2024-09-21T10:10:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:39 smithi070 ceph-mon[107385]: pgmap v649: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 155 KiB/s rd, 255 B/s wr, 14 op/s 2024-09-21T10:10:39.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:39 smithi070 ceph-mon[112139]: pgmap v649: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 155 KiB/s rd, 255 B/s wr, 14 op/s 2024-09-21T10:10:40.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:40 smithi160 ceph-mon[101879]: pgmap v650: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 184 KiB/s rd, 255 B/s wr, 20 op/s 2024-09-21T10:10:40.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:40 smithi070 ceph-mon[112139]: pgmap v650: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 184 KiB/s rd, 255 B/s wr, 20 op/s 2024-09-21T10:10:40.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:40 smithi070 ceph-mon[107385]: pgmap v650: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 184 KiB/s rd, 255 B/s wr, 20 op/s 2024-09-21T10:10:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:42 smithi160 ceph-mon[101879]: pgmap v651: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 182 KiB/s rd, 170 B/s wr, 16 op/s 2024-09-21T10:10:42.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:42 smithi070 ceph-mon[107385]: pgmap v651: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 182 KiB/s rd, 170 B/s wr, 16 op/s 2024-09-21T10:10:42.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:42 smithi070 ceph-mon[112139]: pgmap v651: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 182 KiB/s rd, 170 B/s wr, 16 op/s 2024-09-21T10:10:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:10:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:10:43] "GET /metrics HTTP/1.1" 200 67775 "" "Prometheus/2.51.0" 2024-09-21T10:10:44.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:44 smithi160 ceph-mon[101879]: pgmap v652: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 138 KiB/s rd, 170 B/s wr, 21 op/s 2024-09-21T10:10:44.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:44 smithi070 ceph-mon[107385]: pgmap v652: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 138 KiB/s rd, 170 B/s wr, 21 op/s 2024-09-21T10:10:44.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:44 smithi070 ceph-mon[112139]: pgmap v652: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 138 KiB/s rd, 170 B/s wr, 21 op/s 2024-09-21T10:10:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:46 smithi160 ceph-mon[101879]: pgmap v653: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 208 KiB/s rd, 85 B/s wr, 19 op/s 2024-09-21T10:10:46.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:46 smithi070 ceph-mon[107385]: pgmap v653: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 208 KiB/s rd, 85 B/s wr, 19 op/s 2024-09-21T10:10:46.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:46 smithi070 ceph-mon[112139]: pgmap v653: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 208 KiB/s rd, 85 B/s wr, 19 op/s 2024-09-21T10:10:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:48 smithi160 ceph-mon[101879]: pgmap v654: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-09-21T10:10:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:48 smithi070 ceph-mon[107385]: pgmap v654: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-09-21T10:10:48.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:48 smithi070 ceph-mon[112139]: pgmap v654: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-09-21T10:10:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:50 smithi160 ceph-mon[101879]: pgmap v655: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-09-21T10:10:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:50 smithi070 ceph-mon[107385]: pgmap v655: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-09-21T10:10:50.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:50 smithi070 ceph-mon[112139]: pgmap v655: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-09-21T10:10:51.528 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:51.529 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:52.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:52 smithi160 ceph-mon[101879]: pgmap v656: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 131 KiB/s rd, 13 op/s 2024-09-21T10:10:52.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:52 smithi070 ceph-mon[107385]: pgmap v656: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 131 KiB/s rd, 13 op/s 2024-09-21T10:10:52.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:52 smithi070 ceph-mon[112139]: pgmap v656: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 131 KiB/s rd, 13 op/s 2024-09-21T10:10:53.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:10:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:10:53] "GET /metrics HTTP/1.1" 200 67772 "" "Prometheus/2.51.0" 2024-09-21T10:10:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:54 smithi160 ceph-mon[101879]: pgmap v657: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 109 KiB/s rd, 15 op/s 2024-09-21T10:10:54.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:54 smithi070 ceph-mon[107385]: pgmap v657: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 109 KiB/s rd, 15 op/s 2024-09-21T10:10:54.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:54 smithi070 ceph-mon[112139]: pgmap v657: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 109 KiB/s rd, 15 op/s 2024-09-21T10:10:56.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:56 smithi160 ceph-mon[101879]: pgmap v658: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 179 KiB/s rd, 13 op/s 2024-09-21T10:10:56.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:56 smithi070 ceph-mon[107385]: pgmap v658: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 179 KiB/s rd, 13 op/s 2024-09-21T10:10:56.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:56 smithi070 ceph-mon[112139]: pgmap v658: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 179 KiB/s rd, 13 op/s 2024-09-21T10:10:58.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:10:58 smithi160 ceph-mon[101879]: pgmap v659: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 105 KiB/s rd, 10 op/s 2024-09-21T10:10:58.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:10:58 smithi070 ceph-mon[107385]: pgmap v659: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 105 KiB/s rd, 10 op/s 2024-09-21T10:10:58.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:10:58 smithi070 ceph-mon[112139]: pgmap v659: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 105 KiB/s rd, 10 op/s 2024-09-21T10:11:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:00 smithi160 ceph-mon[101879]: pgmap v660: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-09-21T10:11:00.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:00 smithi070 ceph-mon[107385]: pgmap v660: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-09-21T10:11:00.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:00 smithi070 ceph-mon[112139]: pgmap v660: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-09-21T10:11:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:02 smithi160 ceph-mon[101879]: pgmap v661: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 182 KiB/s rd, 16 op/s 2024-09-21T10:11:02.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:02 smithi070 ceph-mon[107385]: pgmap v661: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 182 KiB/s rd, 16 op/s 2024-09-21T10:11:02.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:02 smithi070 ceph-mon[112139]: pgmap v661: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 182 KiB/s rd, 16 op/s 2024-09-21T10:11:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:11:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:11:03] "GET /metrics HTTP/1.1" 200 67772 "" "Prometheus/2.51.0" 2024-09-21T10:11:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:04 smithi160 ceph-mon[101879]: pgmap v662: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 160 KiB/s rd, 19 op/s 2024-09-21T10:11:04.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:04 smithi070 ceph-mon[112139]: pgmap v662: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 160 KiB/s rd, 19 op/s 2024-09-21T10:11:04.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:04 smithi070 ceph-mon[107385]: pgmap v662: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 160 KiB/s rd, 19 op/s 2024-09-21T10:11:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:05.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:05.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:06 smithi160 ceph-mon[101879]: pgmap v663: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 230 KiB/s rd, 17 op/s 2024-09-21T10:11:06.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:06 smithi070 ceph-mon[107385]: pgmap v663: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 230 KiB/s rd, 17 op/s 2024-09-21T10:11:06.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:06 smithi070 ceph-mon[112139]: pgmap v663: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 230 KiB/s rd, 17 op/s 2024-09-21T10:11:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:08 smithi160 ceph-mon[101879]: pgmap v664: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 155 KiB/s rd, 13 op/s 2024-09-21T10:11:08.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:08 smithi070 ceph-mon[112139]: pgmap v664: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 155 KiB/s rd, 13 op/s 2024-09-21T10:11:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:08 smithi070 ceph-mon[107385]: pgmap v664: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 155 KiB/s rd, 13 op/s 2024-09-21T10:11:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:10 smithi160 ceph-mon[101879]: pgmap v665: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 184 KiB/s rd, 19 op/s 2024-09-21T10:11:10.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:10 smithi070 ceph-mon[107385]: pgmap v665: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 184 KiB/s rd, 19 op/s 2024-09-21T10:11:10.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:10 smithi070 ceph-mon[112139]: pgmap v665: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 184 KiB/s rd, 19 op/s 2024-09-21T10:11:12.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:11 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:11:12.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:11 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:11:12.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:11 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:11:13.199 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:12 smithi160 ceph-mon[101879]: pgmap v666: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 206 KiB/s rd, 16 op/s 2024-09-21T10:11:13.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:12 smithi070 ceph-mon[112139]: pgmap v666: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 206 KiB/s rd, 16 op/s 2024-09-21T10:11:13.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:12 smithi070 ceph-mon[107385]: pgmap v666: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 206 KiB/s rd, 16 op/s 2024-09-21T10:11:13.378 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:11:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:11:13] "GET /metrics HTTP/1.1" 200 67776 "" "Prometheus/2.51.0" 2024-09-21T10:11:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:14 smithi160 ceph-mon[101879]: pgmap v667: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 162 KiB/s rd, 21 op/s 2024-09-21T10:11:14.618 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:14 smithi070 ceph-mon[107385]: pgmap v667: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 162 KiB/s rd, 21 op/s 2024-09-21T10:11:14.619 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:14 smithi070 ceph-mon[112139]: pgmap v667: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 162 KiB/s rd, 21 op/s 2024-09-21T10:11:17.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:11:17.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:11:17.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:11:17.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:11:17.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:16 smithi160 ceph-mon[101879]: pgmap v668: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 208 KiB/s rd, 19 op/s 2024-09-21T10:11:17.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:11:17.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:11:17.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:11:17.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:11:17.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:11:17.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:11:17.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:11:17.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:16 smithi070 ceph-mon[112139]: pgmap v668: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 208 KiB/s rd, 19 op/s 2024-09-21T10:11:17.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:11:17.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:11:17.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:11:17.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:11:17.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:11:17.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:11:17.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:11:17.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:16 smithi070 ceph-mon[107385]: pgmap v668: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 208 KiB/s rd, 19 op/s 2024-09-21T10:11:17.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:11:17.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:11:17.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:11:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:18 smithi160 ceph-mon[101879]: pgmap v669: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-09-21T10:11:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:18 smithi070 ceph-mon[107385]: pgmap v669: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-09-21T10:11:18.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:18 smithi070 ceph-mon[112139]: pgmap v669: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-09-21T10:11:20.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:20 smithi160 ceph-mon[101879]: pgmap v670: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 158 KiB/s rd, 16 op/s 2024-09-21T10:11:20.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:20 smithi070 ceph-mon[107385]: pgmap v670: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 158 KiB/s rd, 16 op/s 2024-09-21T10:11:20.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:20 smithi070 ceph-mon[112139]: pgmap v670: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 158 KiB/s rd, 16 op/s 2024-09-21T10:11:21.528 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:21.529 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:22.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:22 smithi160 ceph-mon[101879]: pgmap v671: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 179 KiB/s rd, 13 op/s 2024-09-21T10:11:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:22 smithi070 ceph-mon[107385]: pgmap v671: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 179 KiB/s rd, 13 op/s 2024-09-21T10:11:22.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:22 smithi070 ceph-mon[112139]: pgmap v671: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 179 KiB/s rd, 13 op/s 2024-09-21T10:11:23.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:11:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:11:23] "GET /metrics HTTP/1.1" 200 67777 "" "Prometheus/2.51.0" 2024-09-21T10:11:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:24 smithi160 ceph-mon[101879]: pgmap v672: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-09-21T10:11:24.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:24 smithi070 ceph-mon[107385]: pgmap v672: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-09-21T10:11:24.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:24 smithi070 ceph-mon[112139]: pgmap v672: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-09-21T10:11:26.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:26 smithi160 ceph-mon[101879]: pgmap v673: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 155 KiB/s rd, 13 op/s 2024-09-21T10:11:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:26 smithi070 ceph-mon[107385]: pgmap v673: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 155 KiB/s rd, 13 op/s 2024-09-21T10:11:26.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:26 smithi070 ceph-mon[112139]: pgmap v673: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 155 KiB/s rd, 13 op/s 2024-09-21T10:11:28.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:28 smithi160 ceph-mon[101879]: pgmap v674: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 105 KiB/s rd, 10 op/s 2024-09-21T10:11:28.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:28 smithi070 ceph-mon[107385]: pgmap v674: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 105 KiB/s rd, 10 op/s 2024-09-21T10:11:28.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:28 smithi070 ceph-mon[112139]: pgmap v674: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 105 KiB/s rd, 10 op/s 2024-09-21T10:11:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:30 smithi160 ceph-mon[101879]: pgmap v675: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-09-21T10:11:30.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:30 smithi070 ceph-mon[107385]: pgmap v675: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-09-21T10:11:30.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:30 smithi070 ceph-mon[112139]: pgmap v675: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-09-21T10:11:31.041 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.EncryptionLoadFormatMismatch (126732 ms) 2024-09-21T10:11:31.041 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.EncryptedResize 2024-09-21T10:11:31.041 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:11:32.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:32 smithi160 ceph-mon[101879]: pgmap v676: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 157 KiB/s rd, 16 op/s 2024-09-21T10:11:32.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:32 smithi070 ceph-mon[107385]: pgmap v676: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 157 KiB/s rd, 16 op/s 2024-09-21T10:11:32.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:32 smithi070 ceph-mon[112139]: pgmap v676: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 157 KiB/s rd, 16 op/s 2024-09-21T10:11:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:11:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:11:33] "GET /metrics HTTP/1.1" 200 67777 "" "Prometheus/2.51.0" 2024-09-21T10:11:34.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:34 smithi160 ceph-mon[101879]: pgmap v677: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 107 KiB/s rd, 170 B/s wr, 14 op/s 2024-09-21T10:11:34.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:34 smithi070 ceph-mon[107385]: pgmap v677: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 107 KiB/s rd, 170 B/s wr, 14 op/s 2024-09-21T10:11:34.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:34 smithi070 ceph-mon[112139]: pgmap v677: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 107 KiB/s rd, 170 B/s wr, 14 op/s 2024-09-21T10:11:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:36 smithi160 ceph-mon[101879]: pgmap v678: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 105 KiB/s rd, 426 B/s wr, 11 op/s 2024-09-21T10:11:36.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:36 smithi070 ceph-mon[107385]: pgmap v678: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 105 KiB/s rd, 426 B/s wr, 11 op/s 2024-09-21T10:11:36.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:36 smithi070 ceph-mon[112139]: pgmap v678: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 105 KiB/s rd, 426 B/s wr, 11 op/s 2024-09-21T10:11:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:38 smithi160 ceph-mon[101879]: pgmap v679: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 79 KiB/s rd, 426 B/s wr, 9 op/s 2024-09-21T10:11:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:38 smithi070 ceph-mon[107385]: pgmap v679: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 79 KiB/s rd, 426 B/s wr, 9 op/s 2024-09-21T10:11:38.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:38 smithi070 ceph-mon[112139]: pgmap v679: 521 pgs: 521 active+clean; 149 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 79 KiB/s rd, 426 B/s wr, 9 op/s 2024-09-21T10:11:40.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:40 smithi160 ceph-mon[101879]: pgmap v680: 521 pgs: 521 active+clean; 157 MiB data, 888 MiB used, 714 GiB / 715 GiB avail; 79 KiB/s rd, 683 KiB/s wr, 9 op/s 2024-09-21T10:11:40.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:40 smithi070 ceph-mon[107385]: pgmap v680: 521 pgs: 521 active+clean; 157 MiB data, 888 MiB used, 714 GiB / 715 GiB avail; 79 KiB/s rd, 683 KiB/s wr, 9 op/s 2024-09-21T10:11:40.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:40 smithi070 ceph-mon[112139]: pgmap v680: 521 pgs: 521 active+clean; 157 MiB data, 888 MiB used, 714 GiB / 715 GiB avail; 79 KiB/s rd, 683 KiB/s wr, 9 op/s 2024-09-21T10:11:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:42 smithi160 ceph-mon[101879]: pgmap v681: 521 pgs: 521 active+clean; 169 MiB data, 908 MiB used, 714 GiB / 715 GiB avail; 108 KiB/s rd, 1.6 MiB/s wr, 17 op/s 2024-09-21T10:11:42.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:42 smithi070 ceph-mon[107385]: pgmap v681: 521 pgs: 521 active+clean; 169 MiB data, 908 MiB used, 714 GiB / 715 GiB avail; 108 KiB/s rd, 1.6 MiB/s wr, 17 op/s 2024-09-21T10:11:42.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:42 smithi070 ceph-mon[112139]: pgmap v681: 521 pgs: 521 active+clean; 169 MiB data, 908 MiB used, 714 GiB / 715 GiB avail; 108 KiB/s rd, 1.6 MiB/s wr, 17 op/s 2024-09-21T10:11:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:11:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:11:43] "GET /metrics HTTP/1.1" 200 67775 "" "Prometheus/2.51.0" 2024-09-21T10:11:44.395 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.EncryptedResize (13354 ms) 2024-09-21T10:11:44.396 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.EncryptedFlattenSmallData 2024-09-21T10:11:44.396 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:11:44.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:44 smithi160 ceph-mon[101879]: pgmap v682: 521 pgs: 521 active+clean; 169 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 60 KiB/s rd, 1.6 MiB/s wr, 17 op/s 2024-09-21T10:11:44.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:44 smithi070 ceph-mon[107385]: pgmap v682: 521 pgs: 521 active+clean; 169 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 60 KiB/s rd, 1.6 MiB/s wr, 17 op/s 2024-09-21T10:11:44.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:44 smithi070 ceph-mon[112139]: pgmap v682: 521 pgs: 521 active+clean; 169 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 60 KiB/s rd, 1.6 MiB/s wr, 17 op/s 2024-09-21T10:11:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:46 smithi160 ceph-mon[101879]: pgmap v683: 521 pgs: 521 active+clean; 165 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 95 KiB/s rd, 1.6 MiB/s wr, 31 op/s 2024-09-21T10:11:46.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:46 smithi070 ceph-mon[107385]: pgmap v683: 521 pgs: 521 active+clean; 165 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 95 KiB/s rd, 1.6 MiB/s wr, 31 op/s 2024-09-21T10:11:46.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:46 smithi070 ceph-mon[112139]: pgmap v683: 521 pgs: 521 active+clean; 165 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 95 KiB/s rd, 1.6 MiB/s wr, 31 op/s 2024-09-21T10:11:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:48 smithi160 ceph-mon[101879]: pgmap v684: 521 pgs: 521 active+clean; 165 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 93 KiB/s rd, 1.6 MiB/s wr, 28 op/s 2024-09-21T10:11:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:48 smithi070 ceph-mon[107385]: pgmap v684: 521 pgs: 521 active+clean; 165 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 93 KiB/s rd, 1.6 MiB/s wr, 28 op/s 2024-09-21T10:11:48.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:48 smithi070 ceph-mon[112139]: pgmap v684: 521 pgs: 521 active+clean; 165 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 93 KiB/s rd, 1.6 MiB/s wr, 28 op/s 2024-09-21T10:11:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:50 smithi160 ceph-mon[101879]: pgmap v685: 521 pgs: 521 active+clean; 165 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 93 KiB/s rd, 1.6 MiB/s wr, 29 op/s 2024-09-21T10:11:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:50 smithi070 ceph-mon[107385]: pgmap v685: 521 pgs: 521 active+clean; 165 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 93 KiB/s rd, 1.6 MiB/s wr, 29 op/s 2024-09-21T10:11:50.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:50 smithi070 ceph-mon[112139]: pgmap v685: 521 pgs: 521 active+clean; 165 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 93 KiB/s rd, 1.6 MiB/s wr, 29 op/s 2024-09-21T10:11:51.528 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:51.528 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:52.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:52 smithi160 ceph-mon[101879]: pgmap v686: 521 pgs: 521 active+clean; 165 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 93 KiB/s rd, 940 KiB/s wr, 28 op/s 2024-09-21T10:11:52.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:52 smithi070 ceph-mon[107385]: pgmap v686: 521 pgs: 521 active+clean; 165 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 93 KiB/s rd, 940 KiB/s wr, 28 op/s 2024-09-21T10:11:52.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:52 smithi070 ceph-mon[112139]: pgmap v686: 521 pgs: 521 active+clean; 165 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 93 KiB/s rd, 940 KiB/s wr, 28 op/s 2024-09-21T10:11:53.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:53 smithi160 ceph-mon[101879]: osdmap e697: 8 total, 8 up, 8 in 2024-09-21T10:11:53.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:11:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:11:53] "GET /metrics HTTP/1.1" 200 67780 "" "Prometheus/2.51.0" 2024-09-21T10:11:53.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:53 smithi070 ceph-mon[107385]: osdmap e697: 8 total, 8 up, 8 in 2024-09-21T10:11:53.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:53 smithi070 ceph-mon[112139]: osdmap e697: 8 total, 8 up, 8 in 2024-09-21T10:11:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:54 smithi160 ceph-mon[101879]: pgmap v688: 521 pgs: 521 active+clean; 173 MiB data, 932 MiB used, 714 GiB / 715 GiB avail; 42 KiB/s rd, 820 KiB/s wr, 18 op/s 2024-09-21T10:11:54.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:54 smithi070 ceph-mon[107385]: pgmap v688: 521 pgs: 521 active+clean; 173 MiB data, 932 MiB used, 714 GiB / 715 GiB avail; 42 KiB/s rd, 820 KiB/s wr, 18 op/s 2024-09-21T10:11:54.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:54 smithi070 ceph-mon[112139]: pgmap v688: 521 pgs: 521 active+clean; 173 MiB data, 932 MiB used, 714 GiB / 715 GiB avail; 42 KiB/s rd, 820 KiB/s wr, 18 op/s 2024-09-21T10:11:56.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:56 smithi160 ceph-mon[101879]: pgmap v689: 521 pgs: 521 active+clean; 181 MiB data, 964 MiB used, 714 GiB / 715 GiB avail; 80 KiB/s rd, 1.6 MiB/s wr, 30 op/s 2024-09-21T10:11:56.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:56 smithi070 ceph-mon[107385]: pgmap v689: 521 pgs: 521 active+clean; 181 MiB data, 964 MiB used, 714 GiB / 715 GiB avail; 80 KiB/s rd, 1.6 MiB/s wr, 30 op/s 2024-09-21T10:11:56.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:56 smithi070 ceph-mon[112139]: pgmap v689: 521 pgs: 521 active+clean; 181 MiB data, 964 MiB used, 714 GiB / 715 GiB avail; 80 KiB/s rd, 1.6 MiB/s wr, 30 op/s 2024-09-21T10:11:58.407 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.EncryptedFlattenSmallData (14012 ms) 2024-09-21T10:11:58.407 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestIOWithIOHint 2024-09-21T10:11:58.407 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:11:58.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:11:58 smithi160 ceph-mon[101879]: pgmap v690: 521 pgs: 521 active+clean; 181 MiB data, 964 MiB used, 714 GiB / 715 GiB avail; 80 KiB/s rd, 1.6 MiB/s wr, 30 op/s 2024-09-21T10:11:58.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:11:58 smithi070 ceph-mon[107385]: pgmap v690: 521 pgs: 521 active+clean; 181 MiB data, 964 MiB used, 714 GiB / 715 GiB avail; 80 KiB/s rd, 1.6 MiB/s wr, 30 op/s 2024-09-21T10:11:58.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:11:58 smithi070 ceph-mon[112139]: pgmap v690: 521 pgs: 521 active+clean; 181 MiB data, 964 MiB used, 714 GiB / 715 GiB avail; 80 KiB/s rd, 1.6 MiB/s wr, 30 op/s 2024-09-21T10:12:00.433 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:12:00.433 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:12:00.433 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:12:00.433 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:12:00.433 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:12:00.433 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:00.433 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:12:00.433 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:00.433 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:00.434 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:12:00.434 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:00.434 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:12:00.434 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:00.434 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:00.434 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:12:00.434 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:00.434 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:12:00.434 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:00.434 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:00.434 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:12:00.434 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:00.434 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:12:00.435 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:00.435 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:00.435 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:12:00.435 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:00.435 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:12:00.435 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:00.435 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:00.435 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:12:00.435 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-21T10:12:00.435 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-21T10:12:00.435 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-21T10:12:00.435 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-21T10:12:00.435 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-21T10:12:00.435 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-21T10:12:00.436 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-21T10:12:00.436 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-21T10:12:00.436 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-21T10:12:00.436 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-21T10:12:00.436 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:00.436 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-21T10:12:00.436 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:00.436 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:00.436 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-21T10:12:00.436 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:00.436 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-21T10:12:00.436 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:00.436 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:00.436 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-21T10:12:00.437 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:00.437 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-21T10:12:00.437 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:00.437 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:00.437 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-21T10:12:00.437 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:00.437 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-21T10:12:00.437 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:00.437 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:00.437 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-21T10:12:00.437 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:00.437 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-21T10:12:00.437 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:00.437 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:00.438 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-21T10:12:00.438 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:00.438 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:00.438 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:00.438 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:00.438 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:00.438 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:00.438 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:12:00.438 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:00.438 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:12:00.438 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:00.438 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:12:00.438 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:00.438 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:12:00.439 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:00.439 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:12:00.439 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:00.439 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:12:00.439 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:00.439 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:12:00.439 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:00.439 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:12:00.439 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:00.439 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:12:00.439 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:00.439 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:12:00.439 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 512~512 = 512 2024-09-21T10:12:00.439 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:00.439 INFO:tasks.workunit.client.0.smithi070.stdout:aio discard: 1536~512 = 0 2024-09-21T10:12:00.440 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:00.440 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:00.440 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:00.440 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:00.440 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:00.440 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 0 2024-09-21T10:12:00.440 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:00.440 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:00.440 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 0 2024-09-21T10:12:00.440 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:00.440 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:00.440 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 16384 2024-09-21T10:12:00.440 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:00.440 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:00.441 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 16384 2024-09-21T10:12:00.441 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:00.441 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:00.441 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:00.441 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:00.441 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 49152 2024-09-21T10:12:00.441 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:00.441 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:00.441 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 49152 2024-09-21T10:12:00.441 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:00.441 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:00.441 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 65536 2024-09-21T10:12:00.441 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:00.441 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:00.441 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 65536 2024-09-21T10:12:00.442 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:00.442 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:00.442 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:00.442 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:00.442 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 98304 2024-09-21T10:12:00.442 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:00.442 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:00.442 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 98304 2024-09-21T10:12:00.442 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:00.442 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:00.442 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 114688 2024-09-21T10:12:00.442 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:00.442 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:00.442 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 114688 2024-09-21T10:12:00.442 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:00.443 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:00.443 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:00.443 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:00.443 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 147456 2024-09-21T10:12:00.443 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:00.443 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:00.443 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 147456 2024-09-21T10:12:00.443 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:00.443 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:00.443 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 163840 2024-09-21T10:12:00.443 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:00.443 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:00.443 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 163840 2024-09-21T10:12:00.443 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:00.444 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:00.444 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:00.444 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:00.444 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 196608 2024-09-21T10:12:00.444 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:00.444 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:00.444 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 196608 2024-09-21T10:12:00.444 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:00.444 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:00.444 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 212992 2024-09-21T10:12:00.444 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:00.444 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:00.444 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 212992 2024-09-21T10:12:00.444 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:00.445 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:00.445 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:00.445 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:00.445 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:00.445 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:00.445 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:00.445 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:00.445 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:00.445 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:00.445 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:00.445 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:00.445 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:00.445 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:00.445 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:00.446 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:00.446 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:00.446 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:00.446 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:00.446 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:00.446 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:00.446 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:00.446 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:00.446 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:00.446 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:00.446 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:00.446 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:00.446 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:00.446 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:00.446 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:00.447 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:00.447 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:00.447 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:00.447 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:00.447 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:00.447 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:00.447 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:00.447 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:00.447 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:00.447 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:00.447 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:00.447 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:00.447 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:00.447 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:00.447 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:00.448 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:00.448 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:00.448 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:00.448 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:00.448 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:00.448 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:00.448 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:00.448 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:00 smithi160 ceph-mon[101879]: pgmap v691: 521 pgs: 521 active+clean; 189 MiB data, 984 MiB used, 714 GiB / 715 GiB avail; 933 KiB/s rd, 2.4 MiB/s wr, 37 op/s 2024-09-21T10:12:00.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:00 smithi070 ceph-mon[107385]: pgmap v691: 521 pgs: 521 active+clean; 189 MiB data, 984 MiB used, 714 GiB / 715 GiB avail; 933 KiB/s rd, 2.4 MiB/s wr, 37 op/s 2024-09-21T10:12:00.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:00 smithi070 ceph-mon[112139]: pgmap v691: 521 pgs: 521 active+clean; 189 MiB data, 984 MiB used, 714 GiB / 715 GiB avail; 933 KiB/s rd, 2.4 MiB/s wr, 37 op/s 2024-09-21T10:12:02.167 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:02.167 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:02.168 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:02.168 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:02.168 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:02.168 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:02.168 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:02.168 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:02.168 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:02.168 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:02.168 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:02.168 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:02.168 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:02.168 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:02.169 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:02.169 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:02.169 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:02.169 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:02.169 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:02.169 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:02.169 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:02.169 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:02.169 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:02.169 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:02.169 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:02.170 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:02.170 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:02.170 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:02.170 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:02.170 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:02.170 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:02.170 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:02.170 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:02.170 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:02.170 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:02.170 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:02.171 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:02.171 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:02.171 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:02.171 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:02.171 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:02.171 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:02.171 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:02.171 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:02.171 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:02.171 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:02.171 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:02.171 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:02.171 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:02.172 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:02.172 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:02.172 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:02.172 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:02.172 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:02.172 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:02.172 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:02.172 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:02.172 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:02.172 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:02.172 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:02.173 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:02.173 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:02.173 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:02.173 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:02.173 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:02.173 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:02.173 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:02.173 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:02.173 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:02.173 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:02.173 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:02.173 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:02.174 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:02.174 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:02.174 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:02.174 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:02.174 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:02.174 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:02.174 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:02.174 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:02.174 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:02.174 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:02.174 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:02.174 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:02.175 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:02.175 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:02.175 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:02.175 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:02.175 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:02.175 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:02.175 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:02.175 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:02.175 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:02.175 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:02.175 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:02.176 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:02.176 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:02.176 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:02.176 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:02.176 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:02.176 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:02.176 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:02.176 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:02.176 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:02.176 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:02.176 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:02.177 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:02.177 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:02.177 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:02.177 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:02.177 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:02.177 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:02.177 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:02.177 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:02.177 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:02.177 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:02.177 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:02.177 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:02.178 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:02.178 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:02.178 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:02.178 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:02.178 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:12:02.178 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:02.178 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestIOWithIOHint (3760 ms) 2024-09-21T10:12:02.178 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestDataPoolIO 2024-09-21T10:12:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:02 smithi160 ceph-mon[101879]: pgmap v692: 521 pgs: 521 active+clean; 197 MiB data, 1014 MiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.4 MiB/s wr, 626 op/s 2024-09-21T10:12:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:02 smithi160 ceph-mon[101879]: osdmap e698: 8 total, 8 up, 8 in 2024-09-21T10:12:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:02 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2446351814' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-29","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:02.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:02 smithi070 ceph-mon[107385]: pgmap v692: 521 pgs: 521 active+clean; 197 MiB data, 1014 MiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.4 MiB/s wr, 626 op/s 2024-09-21T10:12:02.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:02 smithi070 ceph-mon[107385]: osdmap e698: 8 total, 8 up, 8 in 2024-09-21T10:12:02.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:02 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2446351814' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-29","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:02.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:02 smithi070 ceph-mon[112139]: pgmap v692: 521 pgs: 521 active+clean; 197 MiB data, 1014 MiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.4 MiB/s wr, 626 op/s 2024-09-21T10:12:02.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:02 smithi070 ceph-mon[112139]: osdmap e698: 8 total, 8 up, 8 in 2024-09-21T10:12:02.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:02 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2446351814' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-29","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:03.251 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:03.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:03 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-29","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:03.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:03 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:12:03.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:03 smithi160 ceph-mon[101879]: osdmap e699: 8 total, 8 up, 8 in 2024-09-21T10:12:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:12:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:12:03] "GET /metrics HTTP/1.1" 200 67780 "" "Prometheus/2.51.0" 2024-09-21T10:12:03.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:03 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-29","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:03.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:03 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:12:03.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:03 smithi070 ceph-mon[107385]: osdmap e699: 8 total, 8 up, 8 in 2024-09-21T10:12:03.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:03 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-29","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:03.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:03 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:12:03.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:03 smithi070 ceph-mon[112139]: osdmap e699: 8 total, 8 up, 8 in 2024-09-21T10:12:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:04 smithi160 ceph-mon[101879]: pgmap v695: 553 pgs: 19 unknown, 534 active+clean; 197 MiB data, 1015 MiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 745 op/s 2024-09-21T10:12:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:04 smithi160 ceph-mon[101879]: osdmap e700: 8 total, 8 up, 8 in 2024-09-21T10:12:04.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:04 smithi070 ceph-mon[107385]: pgmap v695: 553 pgs: 19 unknown, 534 active+clean; 197 MiB data, 1015 MiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 745 op/s 2024-09-21T10:12:04.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:04 smithi070 ceph-mon[107385]: osdmap e700: 8 total, 8 up, 8 in 2024-09-21T10:12:04.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:04 smithi070 ceph-mon[112139]: pgmap v695: 553 pgs: 19 unknown, 534 active+clean; 197 MiB data, 1015 MiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 745 op/s 2024-09-21T10:12:04.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:04 smithi070 ceph-mon[112139]: osdmap e700: 8 total, 8 up, 8 in 2024-09-21T10:12:05.303 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:12:05.303 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:12:05.303 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:12:05.303 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:12:05.303 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:12:05.304 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:05.304 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:12:05.304 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:05.304 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:05.304 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:12:05.304 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:05.304 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:12:05.304 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:05.304 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:05.304 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:12:05.305 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:05.305 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:12:05.305 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:05.305 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:05.305 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:12:05.305 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:05.305 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:12:05.305 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:05.306 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:05.306 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:12:05.306 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:05.306 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:12:05.306 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:05.306 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:05.306 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:12:05.306 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:05.306 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:05.306 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:05.306 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:05.306 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:05.306 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:05.306 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:12:05.307 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:05.307 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:12:05.307 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:05.307 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:12:05.307 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:05.307 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:12:05.307 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:05.307 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:12:05.307 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:05.307 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:12:05.308 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:05.308 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:12:05.308 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:05.308 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:12:05.308 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:05.308 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:12:05.308 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:05.308 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:12:05.308 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 512~512 = 512 2024-09-21T10:12:05.308 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:05.308 INFO:tasks.workunit.client.0.smithi070.stdout:aio discard: 1536~512 = 0 2024-09-21T10:12:05.308 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:05.308 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:05.308 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:05.309 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:05.309 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:05.309 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:05.309 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:05.309 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestDataPoolIO (3135 ms) 2024-09-21T10:12:05.309 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatch 2024-09-21T10:12:05.309 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:05.328 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatch (26 ms) 2024-09-21T10:12:05.329 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatch 2024-09-21T10:12:05.329 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:05.355 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatch (26 ms) 2024-09-21T10:12:05.355 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccess 2024-09-21T10:12:05.355 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:05.379 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccess (25 ms) 2024-09-21T10:12:05.379 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccess 2024-09-21T10:12:05.380 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:05.407 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccess (28 ms) 2024-09-21T10:12:05.408 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitUnaligned 2024-09-21T10:12:05.408 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:05.478 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitUnaligned (71 ms) 2024-09-21T10:12:05.478 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnaligned 2024-09-21T10:12:05.478 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:05.548 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnaligned (70 ms) 2024-09-21T10:12:05.549 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteTooLarge 2024-09-21T10:12:05.549 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:05.601 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteTooLarge (53 ms) 2024-09-21T10:12:05.602 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteTooLarge 2024-09-21T10:12:05.602 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:05.661 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteTooLarge (60 ms) 2024-09-21T10:12:05.661 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitSuccess 2024-09-21T10:12:05.661 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:06.020 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitSuccess (358 ms) 2024-09-21T10:12:06.020 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccess 2024-09-21T10:12:06.020 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:06.253 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccess (234 ms) 2024-09-21T10:12:06.253 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVIovecLenDiffers 2024-09-21T10:12:06.253 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:06.283 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVIovecLenDiffers (30 ms) 2024-09-21T10:12:06.283 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVMismatch 2024-09-21T10:12:06.284 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:06.329 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVMismatch (45 ms) 2024-09-21T10:12:06.329 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVSuccess 2024-09-21T10:12:06.329 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:06.356 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVSuccess (28 ms) 2024-09-21T10:12:06.356 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestScatterGatherIO 2024-09-21T10:12:06.356 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:06.385 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestScatterGatherIO (29 ms) 2024-09-21T10:12:06.385 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestEmptyDiscard 2024-09-21T10:12:06.385 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:06.404 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:06.404 INFO:tasks.workunit.client.0.smithi070.stdout:aio discard: 0~1048576 = 0 2024-09-21T10:12:06.404 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:06.404 INFO:tasks.workunit.client.0.smithi070.stdout:aio discard: 0~4194304 = 0 2024-09-21T10:12:06.404 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:06.404 INFO:tasks.workunit.client.0.smithi070.stdout:aio discard: 3145728~1048576 = 0 2024-09-21T10:12:06.404 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestEmptyDiscard (19 ms) 2024-09-21T10:12:06.405 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestFUA 2024-09-21T10:12:06.405 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:06.506 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:12:06.506 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:12:06.506 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:12:06.506 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:12:06.506 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:12:06.506 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:06.506 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:06.506 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:06.506 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:06.506 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:06.507 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:06.507 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:12:06.507 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:06.507 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:06.507 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:12:06.507 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:06.507 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:12:06.507 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:06.507 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:06.507 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:12:06.507 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:06.507 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:12:06.507 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:06.507 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:06.507 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:12:06.508 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:06.508 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:12:06.508 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:06.508 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:06.508 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:12:06.508 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:06.508 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:12:06.508 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:06.508 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:06.508 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:12:06.508 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:06.508 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:12:06.508 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:06.508 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:12:06.508 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:06.508 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:12:06.509 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:06.509 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:12:06.509 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:06.509 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:12:06.509 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:06.509 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:12:06.509 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:06.509 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:12:06.509 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:06.509 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:12:06.509 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:06.509 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:12:06.509 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:06.509 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:12:06.509 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestFUA (101 ms) 2024-09-21T10:12:06.510 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestIOPP 2024-09-21T10:12:06.510 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:06.523 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:12:06.523 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:12:06.523 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:12:06.523 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:12:06.523 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:12:06.523 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:06.523 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:12:06.524 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:06.524 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:06.524 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:12:06.524 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:06.524 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:12:06.524 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:06.524 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:06.524 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:12:06.524 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:06.524 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:12:06.524 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:06.524 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:06.524 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:12:06.525 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:06.525 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:12:06.525 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:06.525 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:06.525 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:12:06.525 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:06.525 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:12:06.525 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:06.538 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:06.538 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:12:06.538 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-21T10:12:06.538 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-21T10:12:06.538 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-21T10:12:06.538 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-21T10:12:06.538 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-21T10:12:06.539 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-21T10:12:06.539 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-21T10:12:06.539 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-21T10:12:06.539 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-21T10:12:06.539 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-21T10:12:06.539 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:06.539 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-21T10:12:06.539 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:06.539 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:06.539 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-21T10:12:06.540 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:06.540 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-21T10:12:06.540 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:06.540 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:06.540 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-21T10:12:06.540 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:06.540 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-21T10:12:06.540 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:06.540 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:06.540 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-21T10:12:06.540 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:06.541 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-21T10:12:06.541 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:06.541 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:06.541 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-21T10:12:06.541 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:06.541 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-21T10:12:06.542 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:06.543 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:06.543 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-21T10:12:06.543 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:06.543 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:06.543 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:06.543 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:06.543 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:06.543 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:06.543 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:12:06.543 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:06.543 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:12:06.544 INFO:tasks.workunit.client.0.smithi070.stdout:finished read 2024-09-21T10:12:06.544 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:06.544 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:12:06.544 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:06.544 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:12:06.544 INFO:tasks.workunit.client.0.smithi070.stdout:finished read 2024-09-21T10:12:06.545 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:06.545 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:12:06.545 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:06.545 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:12:06.545 INFO:tasks.workunit.client.0.smithi070.stdout:finished read 2024-09-21T10:12:06.545 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:06.545 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:12:06.545 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:06.545 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:12:06.545 INFO:tasks.workunit.client.0.smithi070.stdout:finished read 2024-09-21T10:12:06.545 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:06.545 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:12:06.545 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:06.545 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:12:06.546 INFO:tasks.workunit.client.0.smithi070.stdout:finished read 2024-09-21T10:12:06.546 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 512~512 2024-09-21T10:12:06.546 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:06 smithi160 ceph-mon[101879]: osdmap e701: 8 total, 8 up, 8 in 2024-09-21T10:12:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:06 smithi160 ceph-mon[101879]: pgmap v698: 553 pgs: 553 active+clean; 231 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 9.3 MiB/s rd, 9.1 MiB/s wr, 760 op/s 2024-09-21T10:12:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:06 smithi070 ceph-mon[107385]: osdmap e701: 8 total, 8 up, 8 in 2024-09-21T10:12:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:06 smithi070 ceph-mon[107385]: pgmap v698: 553 pgs: 553 active+clean; 231 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 9.3 MiB/s rd, 9.1 MiB/s wr, 760 op/s 2024-09-21T10:12:06.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:06 smithi070 ceph-mon[112139]: osdmap e701: 8 total, 8 up, 8 in 2024-09-21T10:12:06.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:06.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:06 smithi070 ceph-mon[112139]: pgmap v698: 553 pgs: 553 active+clean; 231 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 9.3 MiB/s rd, 9.1 MiB/s wr, 760 op/s 2024-09-21T10:12:07.404 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:07.404 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:07.404 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:07.404 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:07.404 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:07.404 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 0 2024-09-21T10:12:07.404 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.404 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.404 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 0 2024-09-21T10:12:07.405 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.405 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.405 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 16384 2024-09-21T10:12:07.405 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.405 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.405 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 16384 2024-09-21T10:12:07.405 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.405 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.405 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:07.405 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:07.405 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 49152 2024-09-21T10:12:07.405 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.405 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.405 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 49152 2024-09-21T10:12:07.406 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.406 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.406 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 65536 2024-09-21T10:12:07.406 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.406 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.406 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 65536 2024-09-21T10:12:07.406 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.406 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.406 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:07.406 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:07.406 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 98304 2024-09-21T10:12:07.406 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.406 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.406 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 98304 2024-09-21T10:12:07.407 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.407 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.407 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 114688 2024-09-21T10:12:07.407 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.407 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.407 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 114688 2024-09-21T10:12:07.407 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.407 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.407 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:07.407 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:07.407 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 147456 2024-09-21T10:12:07.407 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.407 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.407 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 147456 2024-09-21T10:12:07.408 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.408 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.408 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 163840 2024-09-21T10:12:07.408 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.408 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.408 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 163840 2024-09-21T10:12:07.408 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.408 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.408 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:07.408 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:07.408 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 196608 2024-09-21T10:12:07.408 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.408 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.408 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 196608 2024-09-21T10:12:07.409 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.409 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.409 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 212992 2024-09-21T10:12:07.409 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.409 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.409 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 212992 2024-09-21T10:12:07.409 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.409 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.409 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:07.409 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:07.409 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:07.409 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:07.409 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:07.409 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:07.410 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:07.410 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.410 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.410 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:07.410 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:07.410 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:07.410 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:07.410 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:07.410 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.410 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.410 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:07.410 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:07.410 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:07.411 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:07.411 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:07.411 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.411 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.412 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:07.412 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:07.412 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:07.419 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:07.419 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:07.419 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.419 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.419 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:07.420 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:07.420 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:07.420 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:07.420 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:07.420 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:07.420 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:07.420 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:07.421 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:07.438 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:07.439 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:07.439 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.439 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.439 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:07.439 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:07.439 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:07.458 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:07.458 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:07.458 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.459 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.459 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:07.459 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:07.459 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:07.484 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:07.485 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:07.485 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.485 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.485 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:07.485 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:07.485 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:07.510 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:07.510 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:07.511 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.511 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.511 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:07.511 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:07.511 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:07.511 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:07.511 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:07.511 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:07.511 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:07.511 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:07.511 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:07.550 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:07.550 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:07.550 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.550 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.551 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:07.551 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:07.551 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:07.587 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:07.587 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:07.587 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.587 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.587 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:07.587 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:07.587 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:07.630 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:07.630 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:07.630 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.630 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.630 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:07.631 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:07.631 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:07.673 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:07.673 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:07.673 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.673 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.673 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:07.673 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:07.673 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:07.673 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:07.673 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:07.674 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:07.674 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:07.674 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:07.674 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:07.738 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:07.738 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:07.738 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.739 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.739 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:07.739 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:07.739 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:07.794 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:07.794 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:07.794 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.795 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.795 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:07.795 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:07.795 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:07.859 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:07.859 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:07.859 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.859 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.859 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:07.859 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:07.860 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:07.919 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:07.919 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:07.919 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:07.919 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:07.920 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:07.920 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:07.920 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:07.920 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:07.920 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:07.920 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:07.920 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:07.920 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:07.920 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:08.010 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:08.010 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:08.010 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:08.010 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:08.010 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:08.010 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:08.010 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:08.080 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:08.080 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:08.080 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:08.081 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:08.081 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:08.081 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:08.081 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:08.169 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:08.170 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:08.170 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:08.170 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:08.170 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:08.170 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:08.170 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:08.256 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:08.256 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:08.257 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:08.257 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:08.257 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:08.257 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:08.257 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:08.257 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:08.257 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:08.257 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:08.257 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestIOPP (1751 ms) 2024-09-21T10:12:08.257 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteCompareTooSmallPP 2024-09-21T10:12:08.257 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:08.280 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteCompareTooSmallPP (23 ms) 2024-09-21T10:12:08.280 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteCompareTooSmallPP 2024-09-21T10:12:08.280 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:08.302 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteCompareTooSmallPP (22 ms) 2024-09-21T10:12:08.302 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteWriteTooSmallPP 2024-09-21T10:12:08.302 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:08.327 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteWriteTooSmallPP (25 ms) 2024-09-21T10:12:08.327 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteWriteTooSmallPP 2024-09-21T10:12:08.327 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:08.353 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteWriteTooSmallPP (27 ms) 2024-09-21T10:12:08.354 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatchPP 2024-09-21T10:12:08.354 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:08.379 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatchPP (25 ms) 2024-09-21T10:12:08.379 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatchPP 2024-09-21T10:12:08.379 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:08.402 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:08.405 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatchPP (27 ms) 2024-09-21T10:12:08.405 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatchBufferlistGreaterLenPP 2024-09-21T10:12:08.406 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:08.427 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatchBufferlistGreaterLenPP (22 ms) 2024-09-21T10:12:08.427 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatchBufferlistGreaterLenPP 2024-09-21T10:12:08.427 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:08.446 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:08.450 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatchBufferlistGreaterLenPP (22 ms) 2024-09-21T10:12:08.450 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccessPP 2024-09-21T10:12:08.450 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:08.476 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccessPP (26 ms) 2024-09-21T10:12:08.476 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccessPP 2024-09-21T10:12:08.476 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:08.495 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:08.498 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccessPP (23 ms) 2024-09-21T10:12:08.498 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccessBufferlistGreaterLenPP 2024-09-21T10:12:08.498 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:08.522 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccessBufferlistGreaterLenPP (23 ms) 2024-09-21T10:12:08.522 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccessBufferlistGreaterLenPP 2024-09-21T10:12:08.522 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:08.543 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:08.547 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccessBufferlistGreaterLenPP (26 ms) 2024-09-21T10:12:08.547 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitUnalignedPP 2024-09-21T10:12:08.548 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:08 smithi160 ceph-mon[101879]: pgmap v699: 553 pgs: 553 active+clean; 231 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 6.1 MiB/s wr, 510 op/s 2024-09-21T10:12:08.608 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitUnalignedPP (61 ms) 2024-09-21T10:12:08.608 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnalignedPP 2024-09-21T10:12:08.609 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:08 smithi070 ceph-mon[107385]: pgmap v699: 553 pgs: 553 active+clean; 231 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 6.1 MiB/s wr, 510 op/s 2024-09-21T10:12:08.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:08 smithi070 ceph-mon[112139]: pgmap v699: 553 pgs: 553 active+clean; 231 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.3 MiB/s rd, 6.1 MiB/s wr, 510 op/s 2024-09-21T10:12:08.659 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:08.680 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnalignedPP (72 ms) 2024-09-21T10:12:08.680 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteTooLargePP 2024-09-21T10:12:08.680 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:08.750 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteTooLargePP (69 ms) 2024-09-21T10:12:08.750 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteTooLargePP 2024-09-21T10:12:08.750 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:08.792 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:08.807 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteTooLargePP (58 ms) 2024-09-21T10:12:08.807 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitSuccessPP 2024-09-21T10:12:08.807 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:09.134 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitSuccessPP (326 ms) 2024-09-21T10:12:09.134 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccessPP 2024-09-21T10:12:09.134 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:09.432 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:09.455 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccessPP (321 ms) 2024-09-21T10:12:09.455 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestIOPPWithIOHint 2024-09-21T10:12:09.455 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:09.484 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:12:09.484 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:12:09.484 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:12:09.484 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:12:09.484 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:12:09.484 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:09.484 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:12:09.484 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:09.485 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:09.485 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:12:09.485 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:09.485 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:12:09.485 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:09.486 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:09.486 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:12:09.486 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:09.486 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:12:09.486 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:09.487 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:09.487 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:12:09.487 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:09.487 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:12:09.487 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:09.490 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:09.490 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:12:09.490 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:09.490 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:12:09.490 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:09.491 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:09.491 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:12:09.491 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-21T10:12:09.491 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:09.491 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:12:09.491 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:09.492 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:12:09.492 INFO:tasks.workunit.client.0.smithi070.stdout:finished read 2024-09-21T10:12:09.492 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:09.492 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:12:09.492 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:09.492 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:12:09.492 INFO:tasks.workunit.client.0.smithi070.stdout:finished read 2024-09-21T10:12:09.492 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:09.492 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:12:09.492 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:09.492 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:12:09.492 INFO:tasks.workunit.client.0.smithi070.stdout:finished read 2024-09-21T10:12:09.492 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:09.493 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:12:09.493 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:09.493 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:12:09.493 INFO:tasks.workunit.client.0.smithi070.stdout:finished read 2024-09-21T10:12:09.493 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:09.493 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:12:09.493 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:12:10.320 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:12:10.320 INFO:tasks.workunit.client.0.smithi070.stdout:finished read 2024-09-21T10:12:10.320 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 0 2024-09-21T10:12:10.320 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.320 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.321 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 0 2024-09-21T10:12:10.321 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.321 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.321 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 16384 2024-09-21T10:12:10.321 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.321 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.321 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 16384 2024-09-21T10:12:10.321 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.321 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.321 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:10.321 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:10.321 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 49152 2024-09-21T10:12:10.321 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.321 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.322 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 49152 2024-09-21T10:12:10.322 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.322 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.322 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 65536 2024-09-21T10:12:10.322 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.322 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.322 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 65536 2024-09-21T10:12:10.322 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.322 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.322 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:10.322 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:10.322 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 98304 2024-09-21T10:12:10.322 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.323 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.323 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 98304 2024-09-21T10:12:10.323 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.323 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.323 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 114688 2024-09-21T10:12:10.323 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.323 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.323 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 114688 2024-09-21T10:12:10.323 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.323 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.323 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:10.323 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:10.323 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 147456 2024-09-21T10:12:10.323 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.324 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.324 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 147456 2024-09-21T10:12:10.324 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.324 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.324 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 163840 2024-09-21T10:12:10.324 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.324 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.324 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 163840 2024-09-21T10:12:10.324 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.324 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.324 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:10.325 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:10.325 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 196608 2024-09-21T10:12:10.325 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.325 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.325 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 196608 2024-09-21T10:12:10.325 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.325 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.325 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 212992 2024-09-21T10:12:10.325 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.325 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.325 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 212992 2024-09-21T10:12:10.325 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.325 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.325 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:10.326 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:10.326 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:10.326 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:10.326 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:10.326 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:10.326 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:10.326 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.326 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.326 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:10.326 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:10.326 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:10.327 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:10.327 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:10.327 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.327 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.327 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:10.327 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:10.327 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:10.329 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:10.329 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:10.330 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.330 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.330 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:10.330 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:10.330 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:10.336 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:10.337 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:10.337 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.337 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.337 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:10.337 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:10.337 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:10.337 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:10.337 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:10.337 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:10.337 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:10.337 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:10.337 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:10.356 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:10.356 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:10.356 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.356 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.356 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:10.356 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:10.356 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:10.375 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:10.375 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:10.375 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.375 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.375 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:10.375 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:10.375 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:10.400 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:10.400 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:10.400 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.400 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.400 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:10.400 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:10.401 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:10.424 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:10.424 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:10.424 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.424 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.424 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:10.424 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:10.424 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:10.424 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:10.425 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:10.425 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:10.425 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:10.425 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:10.425 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:10.463 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:10.463 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:10.463 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.463 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.464 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:10.464 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:10.464 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:10.500 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:10.501 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:10.501 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.501 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.501 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:10.501 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:10.501 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:10.550 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:10.550 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:10.550 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.550 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.550 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:10.551 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:10.551 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:10 smithi160 ceph-mon[101879]: pgmap v700: 553 pgs: 553 active+clean; 257 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 9.6 MiB/s wr, 1.79k op/s 2024-09-21T10:12:10.608 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:10.608 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:10.608 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.608 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.608 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:10.608 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:10.608 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:10.608 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:10.608 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:10.608 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:10.608 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:10.608 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:10.608 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:10.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:10 smithi070 ceph-mon[107385]: pgmap v700: 553 pgs: 553 active+clean; 257 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 9.6 MiB/s wr, 1.79k op/s 2024-09-21T10:12:10.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:10 smithi070 ceph-mon[112139]: pgmap v700: 553 pgs: 553 active+clean; 257 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 9.6 MiB/s wr, 1.79k op/s 2024-09-21T10:12:10.678 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:10.678 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:10.678 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.678 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.678 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:10.678 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:10.679 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:10.731 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:10.731 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:10.731 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.731 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.731 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:10.731 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:10.732 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:10.791 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:10.791 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:10.791 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.791 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.791 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:10.792 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:10.792 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:10.854 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:10.854 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:10.854 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.854 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.854 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:10.854 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:10.854 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:10.854 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:10.854 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:10.854 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:10.855 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:10.855 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:10.855 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:10.943 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:10.943 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:10.943 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:10.943 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:10.944 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:10.944 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:10.944 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:11.015 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:11.015 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:11.015 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:11.015 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:11.015 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:11.016 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:11.016 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:11.098 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:11.098 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:11.098 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:11.098 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:11.098 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:11.098 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:11.098 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:12:11.178 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:12:11.179 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-21T10:12:11.179 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-21T10:12:11.179 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-21T10:12:11.179 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:11.179 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:11.179 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:11.179 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:12:11.179 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-21T10:12:11.179 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-21T10:12:11.179 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestIOPPWithIOHint (1723 ms) 2024-09-21T10:12:11.179 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestIOToSnapshot 2024-09-21T10:12:11.179 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:12.293 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 0 2024-09-21T10:12:12.293 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 0 2024-09-21T10:12:12.293 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 1 2024-09-21T10:12:12.294 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-21T10:12:12.294 INFO:tasks.workunit.client.0.smithi070.stdout:snap: orig 2024-09-21T10:12:12.294 INFO:tasks.workunit.client.0.smithi070.stdout:found orig with size 2097152 2024-09-21T10:12:12.294 INFO:tasks.workunit.client.0.smithi070.stdout:read: 80 2024-09-21T10:12:12.294 INFO:tasks.workunit.client.0.smithi070.stdout:write test data! 2024-09-21T10:12:12.294 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 80 2024-09-21T10:12:12.294 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 2 2024-09-21T10:12:12.294 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 2 2024-09-21T10:12:12.294 INFO:tasks.workunit.client.0.smithi070.stdout:snap: orig 2024-09-21T10:12:12.294 INFO:tasks.workunit.client.0.smithi070.stdout:snap: written 2024-09-21T10:12:12.294 INFO:tasks.workunit.client.0.smithi070.stdout:found orig with size 2097152 2024-09-21T10:12:12.294 INFO:tasks.workunit.client.0.smithi070.stdout:found written with size 2097152 2024-09-21T10:12:12.295 INFO:tasks.workunit.client.0.smithi070.stdout:read: 80 2024-09-21T10:12:12.295 INFO:tasks.workunit.client.0.smithi070.stdout:read: 80 2024-09-21T10:12:12.295 INFO:tasks.workunit.client.0.smithi070.stdout:read: 80 2024-09-21T10:12:12.295 INFO:tasks.workunit.client.0.smithi070.stdout:write to snapshot returned -30 2024-09-21T10:12:12.295 INFO:tasks.workunit.client.0.smithi070.stdout:Read-only file system 2024-09-21T10:12:12.316 INFO:tasks.workunit.client.0.smithi070.stdout:read: 80 2024-09-21T10:12:12.316 INFO:tasks.workunit.client.0.smithi070.stdout:read: 80 2024-09-21T10:12:12.316 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 80 2024-09-21T10:12:12.316 INFO:tasks.workunit.client.0.smithi070.stdout:opening testimg@orig 2024-09-21T10:12:12.316 INFO:tasks.workunit.client.0.smithi070.stdout:read: 80 2024-09-21T10:12:12.316 INFO:tasks.workunit.client.0.smithi070.stdout:write to snapshot returned -30 2024-09-21T10:12:12.316 INFO:tasks.workunit.client.0.smithi070.stdout:Read-only file system 2024-09-21T10:12:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:12 smithi160 ceph-mon[101879]: osdmap e702: 8 total, 8 up, 8 in 2024-09-21T10:12:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:12 smithi160 ceph-mon[101879]: pgmap v702: 553 pgs: 553 active+clean; 297 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 14 MiB/s wr, 1.70k op/s 2024-09-21T10:12:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:12 smithi070 ceph-mon[107385]: osdmap e702: 8 total, 8 up, 8 in 2024-09-21T10:12:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:12 smithi070 ceph-mon[107385]: pgmap v702: 553 pgs: 553 active+clean; 297 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 14 MiB/s wr, 1.70k op/s 2024-09-21T10:12:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:12 smithi070 ceph-mon[112139]: osdmap e702: 8 total, 8 up, 8 in 2024-09-21T10:12:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:12 smithi070 ceph-mon[112139]: pgmap v702: 553 pgs: 553 active+clean; 297 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 14 MiB/s wr, 1.70k op/s 2024-09-21T10:12:13.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:13 smithi160 ceph-mon[101879]: osdmap e703: 8 total, 8 up, 8 in 2024-09-21T10:12:13.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:13 smithi160 ceph-mon[101879]: osdmap e704: 8 total, 8 up, 8 in 2024-09-21T10:12:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:12:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:12:13] "GET /metrics HTTP/1.1" 200 70170 "" "Prometheus/2.51.0" 2024-09-21T10:12:13.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:13 smithi070 ceph-mon[107385]: osdmap e703: 8 total, 8 up, 8 in 2024-09-21T10:12:13.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:13 smithi070 ceph-mon[107385]: osdmap e704: 8 total, 8 up, 8 in 2024-09-21T10:12:13.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:13 smithi070 ceph-mon[112139]: osdmap e703: 8 total, 8 up, 8 in 2024-09-21T10:12:13.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:13 smithi070 ceph-mon[112139]: osdmap e704: 8 total, 8 up, 8 in 2024-09-21T10:12:13.790 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 2 2024-09-21T10:12:13.790 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 2 2024-09-21T10:12:13.791 INFO:tasks.workunit.client.0.smithi070.stdout:snap: orig 2024-09-21T10:12:13.791 INFO:tasks.workunit.client.0.smithi070.stdout:snap: written 2024-09-21T10:12:13.791 INFO:tasks.workunit.client.0.smithi070.stdout:found orig with size 2097152 2024-09-21T10:12:13.791 INFO:tasks.workunit.client.0.smithi070.stdout:found written with size 2097152 2024-09-21T10:12:13.791 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 1 2024-09-21T10:12:13.791 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-21T10:12:13.791 INFO:tasks.workunit.client.0.smithi070.stdout:snap: orig 2024-09-21T10:12:13.791 INFO:tasks.workunit.client.0.smithi070.stdout:found orig with size 2097152 2024-09-21T10:12:13.791 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 0 2024-09-21T10:12:13.791 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 0 2024-09-21T10:12:13.791 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestIOToSnapshot (2612 ms) 2024-09-21T10:12:13.791 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestSnapshotDeletedIo 2024-09-21T10:12:13.791 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:15.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:14 smithi160 ceph-mon[101879]: osdmap e705: 8 total, 8 up, 8 in 2024-09-21T10:12:15.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:14 smithi160 ceph-mon[101879]: pgmap v706: 553 pgs: 1 active+clean+snaptrim, 552 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 13 MiB/s wr, 2.39k op/s 2024-09-21T10:12:15.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:14 smithi070 ceph-mon[107385]: osdmap e705: 8 total, 8 up, 8 in 2024-09-21T10:12:15.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:14 smithi070 ceph-mon[107385]: pgmap v706: 553 pgs: 1 active+clean+snaptrim, 552 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 13 MiB/s wr, 2.39k op/s 2024-09-21T10:12:15.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:14 smithi070 ceph-mon[112139]: osdmap e705: 8 total, 8 up, 8 in 2024-09-21T10:12:15.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:14 smithi070 ceph-mon[112139]: pgmap v706: 553 pgs: 1 active+clean+snaptrim, 552 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 13 MiB/s wr, 2.39k op/s 2024-09-21T10:12:15.801 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestSnapshotDeletedIo (2010 ms) 2024-09-21T10:12:15.801 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestClone 2024-09-21T10:12:15.801 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:16.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:15 smithi160 ceph-mon[101879]: osdmap e706: 8 total, 8 up, 8 in 2024-09-21T10:12:16.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:15 smithi070 ceph-mon[107385]: osdmap e706: 8 total, 8 up, 8 in 2024-09-21T10:12:16.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:15 smithi070 ceph-mon[112139]: osdmap e706: 8 total, 8 up, 8 in 2024-09-21T10:12:17.051 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:16 smithi070 ceph-mon[107385]: osdmap e707: 8 total, 8 up, 8 in 2024-09-21T10:12:17.051 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:16 smithi070 ceph-mon[107385]: pgmap v709: 553 pgs: 1 active+clean+snaptrim, 552 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 542 KiB/s wr, 2.25k op/s 2024-09-21T10:12:17.051 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:16 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:12:17.051 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:16 smithi070 ceph-mon[112139]: osdmap e707: 8 total, 8 up, 8 in 2024-09-21T10:12:17.051 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:16 smithi070 ceph-mon[112139]: pgmap v709: 553 pgs: 1 active+clean+snaptrim, 552 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 542 KiB/s wr, 2.25k op/s 2024-09-21T10:12:17.051 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:16 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:12:17.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:16 smithi160 ceph-mon[101879]: osdmap e707: 8 total, 8 up, 8 in 2024-09-21T10:12:17.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:16 smithi160 ceph-mon[101879]: pgmap v709: 553 pgs: 1 active+clean+snaptrim, 552 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 542 KiB/s wr, 2.25k op/s 2024-09-21T10:12:17.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:16 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:12:17.788 INFO:tasks.workunit.client.0.smithi070.stdout:made parent image "parent" 2024-09-21T10:12:17.788 INFO:tasks.workunit.client.0.smithi070.stdout:parent has no parent info 2024-09-21T10:12:17.788 INFO:tasks.workunit.client.0.smithi070.stdout:made snapshot "parent@parent_snap" 2024-09-21T10:12:17.788 INFO:tasks.workunit.client.0.smithi070.stdout:can't unprotect an unprotected snap 2024-09-21T10:12:17.788 INFO:tasks.workunit.client.0.smithi070.stdout:can't protect a protected snap 2024-09-21T10:12:17.788 INFO:tasks.workunit.client.0.smithi070.stdout:made and opened clone "child" 2024-09-21T10:12:17.788 INFO:tasks.workunit.client.0.smithi070.stdout:read: 8 2024-09-21T10:12:17.788 INFO:tasks.workunit.client.0.smithi070.stdout:read: 8 2024-09-21T10:12:17.788 INFO:tasks.workunit.client.0.smithi070.stdout:read: 8 2024-09-21T10:12:17.788 INFO:tasks.workunit.client.0.smithi070.stdout:sizes and overlaps are good between parent and child 2024-09-21T10:12:17.788 INFO:tasks.workunit.client.0.smithi070.stdout:child image successfully cloned all image-meta pairs 2024-09-21T10:12:17.789 INFO:tasks.workunit.client.0.smithi070.stdout:sized down clone, changed overlap 2024-09-21T10:12:17.789 INFO:tasks.workunit.client.0.smithi070.stdout:parent info: size 4194304 obj_size 4194304 parent_pool 18446744073709551615 2024-09-21T10:12:17.789 INFO:tasks.workunit.client.0.smithi070.stdout:sized up clone, changed size but not overlap or parent's size 2024-09-21T10:12:17.789 INFO:tasks.workunit.client.0.smithi070.stdout:can't remove parent while child still exists 2024-09-21T10:12:17.789 INFO:tasks.workunit.client.0.smithi070.stdout:can't remove parent while still protected 2024-09-21T10:12:17.789 INFO:tasks.workunit.client.0.smithi070.stdout:removed parent snap after unprotecting 2024-09-21T10:12:17.789 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestClone (1987 ms) 2024-09-21T10:12:17.789 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestClone2 2024-09-21T10:12:17.789 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:18.088 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:17 smithi070 ceph-mon[107385]: osdmap e708: 8 total, 8 up, 8 in 2024-09-21T10:12:18.088 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:17 smithi070 ceph-mon[107385]: osdmap e709: 8 total, 8 up, 8 in 2024-09-21T10:12:18.088 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:17 smithi070 ceph-mon[112139]: osdmap e708: 8 total, 8 up, 8 in 2024-09-21T10:12:18.088 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:17 smithi070 ceph-mon[112139]: osdmap e709: 8 total, 8 up, 8 in 2024-09-21T10:12:18.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:17 smithi160 ceph-mon[101879]: osdmap e708: 8 total, 8 up, 8 in 2024-09-21T10:12:18.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:17 smithi160 ceph-mon[101879]: osdmap e709: 8 total, 8 up, 8 in 2024-09-21T10:12:18.908 INFO:tasks.workunit.client.0.smithi070.stdout:made parent image "parent" 2024-09-21T10:12:18.908 INFO:tasks.workunit.client.0.smithi070.stdout:parent has no parent info 2024-09-21T10:12:18.908 INFO:tasks.workunit.client.0.smithi070.stdout:made snapshot "parent@parent_snap" 2024-09-21T10:12:18.908 INFO:tasks.workunit.client.0.smithi070.stdout:made and opened clone "child" 2024-09-21T10:12:18.908 INFO:tasks.workunit.client.0.smithi070.stdout:child image successfully cloned all image-meta pairs 2024-09-21T10:12:18.908 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestClone2 (1120 ms) 2024-09-21T10:12:18.908 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ListChildren 2024-09-21T10:12:19.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:18 smithi160 ceph-mon[101879]: pgmap v712: 553 pgs: 1 active+clean+snaptrim, 552 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 63 KiB/s rd, 7.7 KiB/s wr, 90 op/s 2024-09-21T10:12:19.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:18 smithi070 ceph-mon[107385]: pgmap v712: 553 pgs: 1 active+clean+snaptrim, 552 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 63 KiB/s rd, 7.7 KiB/s wr, 90 op/s 2024-09-21T10:12:19.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:18 smithi070 ceph-mon[112139]: pgmap v712: 553 pgs: 1 active+clean+snaptrim, 552 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 63 KiB/s rd, 7.7 KiB/s wr, 90 op/s 2024-09-21T10:12:19.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:19 smithi070 ceph-mon[107385]: osdmap e710: 8 total, 8 up, 8 in 2024-09-21T10:12:19.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:19 smithi070 ceph-mon[112139]: osdmap e710: 8 total, 8 up, 8 in 2024-09-21T10:12:20.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:19 smithi160 ceph-mon[101879]: osdmap e710: 8 total, 8 up, 8 in 2024-09-21T10:12:21.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:20 smithi160 ceph-mon[101879]: osdmap e711: 8 total, 8 up, 8 in 2024-09-21T10:12:21.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:20 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/834163645' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-30","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:21.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:12:21.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:12:21.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:20 smithi160 ceph-mon[101879]: pgmap v715: 585 pgs: 32 unknown, 553 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 700 KiB/s rd, 43 KiB/s wr, 818 op/s 2024-09-21T10:12:21.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:21.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:12:21.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:12:21.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:20 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/834163645' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:12:21.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:20 smithi160 ceph-mon[101879]: osdmap e712: 8 total, 8 up, 8 in 2024-09-21T10:12:21.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:20 smithi070 ceph-mon[107385]: osdmap e711: 8 total, 8 up, 8 in 2024-09-21T10:12:21.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:20 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/834163645' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-30","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:21.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:12:21.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:12:21.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:20 smithi070 ceph-mon[107385]: pgmap v715: 585 pgs: 32 unknown, 553 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 700 KiB/s rd, 43 KiB/s wr, 818 op/s 2024-09-21T10:12:21.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:21.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:12:21.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:12:21.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:20 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/834163645' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:12:21.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:20 smithi070 ceph-mon[107385]: osdmap e712: 8 total, 8 up, 8 in 2024-09-21T10:12:21.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:20 smithi070 ceph-mon[112139]: osdmap e711: 8 total, 8 up, 8 in 2024-09-21T10:12:21.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:20 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/834163645' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-30","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:21.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:12:21.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:12:21.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:20 smithi070 ceph-mon[112139]: pgmap v715: 585 pgs: 32 unknown, 553 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 700 KiB/s rd, 43 KiB/s wr, 818 op/s 2024-09-21T10:12:21.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:21.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:12:21.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:12:21.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:20 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/834163645' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:12:21.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:20 smithi070 ceph-mon[112139]: osdmap e712: 8 total, 8 up, 8 in 2024-09-21T10:12:22.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:12:22.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:12:22.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:12:22.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:21 smithi160 ceph-mon[101879]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:12:22.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:12:22.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:12:22.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:12:22.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:21 smithi070 ceph-mon[107385]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:12:22.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:12:22.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:12:22.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:12:22.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:21 smithi070 ceph-mon[112139]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:12:22.774 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:23.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:22 smithi160 ceph-mon[101879]: osdmap e713: 8 total, 8 up, 8 in 2024-09-21T10:12:23.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:22 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3622112396' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-31","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:23.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:22 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-31","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:23.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:22 smithi160 ceph-mon[101879]: pgmap v718: 617 pgs: 50 unknown, 567 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 754 KiB/s rd, 53 KiB/s wr, 899 op/s 2024-09-21T10:12:23.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:22 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:12:23.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:22 smithi160 ceph-mon[101879]: osdmap e714: 8 total, 8 up, 8 in 2024-09-21T10:12:23.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:22 smithi070 ceph-mon[107385]: osdmap e713: 8 total, 8 up, 8 in 2024-09-21T10:12:23.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:22 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3622112396' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-31","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:23.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:22 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-31","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:23.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:22 smithi070 ceph-mon[107385]: pgmap v718: 617 pgs: 50 unknown, 567 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 754 KiB/s rd, 53 KiB/s wr, 899 op/s 2024-09-21T10:12:23.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:22 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:12:23.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:22 smithi070 ceph-mon[107385]: osdmap e714: 8 total, 8 up, 8 in 2024-09-21T10:12:23.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:22 smithi070 ceph-mon[112139]: osdmap e713: 8 total, 8 up, 8 in 2024-09-21T10:12:23.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:22 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3622112396' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-31","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:23.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:22 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-31","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:23.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:22 smithi070 ceph-mon[112139]: pgmap v718: 617 pgs: 50 unknown, 567 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 754 KiB/s rd, 53 KiB/s wr, 899 op/s 2024-09-21T10:12:23.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:22 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:12:23.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:22 smithi070 ceph-mon[112139]: osdmap e714: 8 total, 8 up, 8 in 2024-09-21T10:12:23.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:12:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:12:23] "GET /metrics HTTP/1.1" 200 72548 "" "Prometheus/2.51.0" 2024-09-21T10:12:25.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:24 smithi160 ceph-mon[101879]: osdmap e715: 8 total, 8 up, 8 in 2024-09-21T10:12:25.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:24 smithi160 ceph-mon[101879]: pgmap v721: 617 pgs: 39 unknown, 578 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 89 KiB/s rd, 12 KiB/s wr, 123 op/s 2024-09-21T10:12:25.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:24 smithi070 ceph-mon[112139]: osdmap e715: 8 total, 8 up, 8 in 2024-09-21T10:12:25.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:24 smithi070 ceph-mon[112139]: pgmap v721: 617 pgs: 39 unknown, 578 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 89 KiB/s rd, 12 KiB/s wr, 123 op/s 2024-09-21T10:12:25.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:24 smithi070 ceph-mon[107385]: osdmap e715: 8 total, 8 up, 8 in 2024-09-21T10:12:25.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:24 smithi070 ceph-mon[107385]: pgmap v721: 617 pgs: 39 unknown, 578 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 89 KiB/s rd, 12 KiB/s wr, 123 op/s 2024-09-21T10:12:26.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:25 smithi160 ceph-mon[101879]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:12:26.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:25 smithi160 ceph-mon[101879]: osdmap e716: 8 total, 8 up, 8 in 2024-09-21T10:12:26.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:25 smithi070 ceph-mon[112139]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:12:26.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:25 smithi070 ceph-mon[112139]: osdmap e716: 8 total, 8 up, 8 in 2024-09-21T10:12:26.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:25 smithi070 ceph-mon[107385]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:12:26.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:25 smithi070 ceph-mon[107385]: osdmap e716: 8 total, 8 up, 8 in 2024-09-21T10:12:27.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:26 smithi160 ceph-mon[101879]: osdmap e717: 8 total, 8 up, 8 in 2024-09-21T10:12:27.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:26 smithi160 ceph-mon[101879]: pgmap v724: 617 pgs: 617 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 65 KiB/s rd, 5.0 KiB/s wr, 85 op/s 2024-09-21T10:12:27.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:26 smithi070 ceph-mon[107385]: osdmap e717: 8 total, 8 up, 8 in 2024-09-21T10:12:27.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:26 smithi070 ceph-mon[107385]: pgmap v724: 617 pgs: 617 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 65 KiB/s rd, 5.0 KiB/s wr, 85 op/s 2024-09-21T10:12:27.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:26 smithi070 ceph-mon[112139]: osdmap e717: 8 total, 8 up, 8 in 2024-09-21T10:12:27.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:26 smithi070 ceph-mon[112139]: pgmap v724: 617 pgs: 617 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 65 KiB/s rd, 5.0 KiB/s wr, 85 op/s 2024-09-21T10:12:28.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:27 smithi160 ceph-mon[101879]: osdmap e718: 8 total, 8 up, 8 in 2024-09-21T10:12:28.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:27 smithi160 ceph-mon[101879]: osdmap e719: 8 total, 8 up, 8 in 2024-09-21T10:12:28.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:27 smithi070 ceph-mon[107385]: osdmap e718: 8 total, 8 up, 8 in 2024-09-21T10:12:28.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:27 smithi070 ceph-mon[107385]: osdmap e719: 8 total, 8 up, 8 in 2024-09-21T10:12:28.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:27 smithi070 ceph-mon[112139]: osdmap e718: 8 total, 8 up, 8 in 2024-09-21T10:12:28.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:27 smithi070 ceph-mon[112139]: osdmap e719: 8 total, 8 up, 8 in 2024-09-21T10:12:28.263 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.263 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-31/image125 2024-09-21T10:12:28.264 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-31/image125 2024-09-21T10:12:28.264 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image125 2024-09-21T10:12:28.264 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.264 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 1 2024-09-21T10:12:28.264 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-21T10:12:28.264 INFO:tasks.workunit.client.0.smithi070.stdout:child: image125 2024-09-21T10:12:28.264 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image125/86198bbe93fc 2024-09-21T10:12:28.264 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.264 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.264 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-31/image125 2024-09-21T10:12:28.264 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.265 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-31/image125 2024-09-21T10:12:28.265 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image125 2024-09-21T10:12:28.265 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.265 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.265 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.265 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.265 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.265 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.265 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 2 2024-09-21T10:12:28.265 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 2 2024-09-21T10:12:28.265 INFO:tasks.workunit.client.0.smithi070.stdout:child: image126 2024-09-21T10:12:28.265 INFO:tasks.workunit.client.0.smithi070.stdout:child: image125 2024-09-21T10:12:28.265 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image125/86198bbe93fc 2024-09-21T10:12:28.265 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.266 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-30/image126/86191910a208 2024-09-21T10:12:28.266 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.266 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.266 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-31/image125 2024-09-21T10:12:28.266 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.266 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-31/image125 2024-09-21T10:12:28.266 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image125 2024-09-21T10:12:28.266 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.266 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.266 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.266 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.266 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.266 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.267 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.267 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-31/image127 2024-09-21T10:12:28.267 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.267 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-31/image125 2024-09-21T10:12:28.267 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-31/image127 2024-09-21T10:12:28.267 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image127 2024-09-21T10:12:28.267 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.267 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 3 2024-09-21T10:12:28.267 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 3 2024-09-21T10:12:28.267 INFO:tasks.workunit.client.0.smithi070.stdout:child: image126 2024-09-21T10:12:28.267 INFO:tasks.workunit.client.0.smithi070.stdout:child: image125 2024-09-21T10:12:28.267 INFO:tasks.workunit.client.0.smithi070.stdout:child: image127 2024-09-21T10:12:28.267 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image125/86198bbe93fc 2024-09-21T10:12:28.268 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.268 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-30/image126/86191910a208 2024-09-21T10:12:28.268 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.268 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image127/86197d9eb137 2024-09-21T10:12:28.268 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.268 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.268 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-31/image125 2024-09-21T10:12:28.268 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.268 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-31/image125 2024-09-21T10:12:28.268 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image125 2024-09-21T10:12:28.268 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.268 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.268 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.268 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.269 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.269 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.269 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 3 2024-09-21T10:12:28.269 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 3 2024-09-21T10:12:28.269 INFO:tasks.workunit.client.0.smithi070.stdout:child: image126 2024-09-21T10:12:28.269 INFO:tasks.workunit.client.0.smithi070.stdout:child: image125 2024-09-21T10:12:28.269 INFO:tasks.workunit.client.0.smithi070.stdout:child: image127 2024-09-21T10:12:28.269 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image125/86198bbe93fc 2024-09-21T10:12:28.269 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.269 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-30/image126/86191910a208 2024-09-21T10:12:28.269 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.269 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image127/86197d9eb137 2024-09-21T10:12:28.269 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.269 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.269 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-31/image125 2024-09-21T10:12:28.269 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.270 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-31/image125 2024-09-21T10:12:28.270 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image125 2024-09-21T10:12:28.270 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.270 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.270 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.270 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.270 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.270 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.270 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.270 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-31/image128 2024-09-21T10:12:28.270 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.270 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-31/image125 2024-09-21T10:12:28.270 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-31/image128 2024-09-21T10:12:28.271 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image128 2024-09-21T10:12:28.271 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.271 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 4 2024-09-21T10:12:28.271 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 4 2024-09-21T10:12:28.271 INFO:tasks.workunit.client.0.smithi070.stdout:child: image126 2024-09-21T10:12:28.271 INFO:tasks.workunit.client.0.smithi070.stdout:child: image125 2024-09-21T10:12:28.271 INFO:tasks.workunit.client.0.smithi070.stdout:child: image127 2024-09-21T10:12:28.271 INFO:tasks.workunit.client.0.smithi070.stdout:child: image128 2024-09-21T10:12:28.271 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image125/86198bbe93fc 2024-09-21T10:12:28.271 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.271 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-30/image126/86191910a208 2024-09-21T10:12:28.271 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.271 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image127/86197d9eb137 2024-09-21T10:12:28.271 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.272 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image128/86198abeee52 2024-09-21T10:12:28.272 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.272 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.272 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-31/image125 2024-09-21T10:12:28.272 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.272 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-31/image125 2024-09-21T10:12:28.272 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image125 2024-09-21T10:12:28.272 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.272 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.272 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.272 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.272 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.272 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.272 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.273 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-31/image127 2024-09-21T10:12:28.273 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.273 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-31/image125 2024-09-21T10:12:28.273 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-31/image127 2024-09-21T10:12:28.273 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image127 2024-09-21T10:12:28.273 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.273 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.273 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-31/image128 2024-09-21T10:12:28.273 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.873 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-31/image125 2024-09-21T10:12:28.873 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-31/image127 2024-09-21T10:12:28.873 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-31/image128 2024-09-21T10:12:28.873 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image128 2024-09-21T10:12:28.873 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.873 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 4 2024-09-21T10:12:28.873 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 4 2024-09-21T10:12:28.873 INFO:tasks.workunit.client.0.smithi070.stdout:child: image126 2024-09-21T10:12:28.874 INFO:tasks.workunit.client.0.smithi070.stdout:child: image125 2024-09-21T10:12:28.874 INFO:tasks.workunit.client.0.smithi070.stdout:child: image127 2024-09-21T10:12:28.874 INFO:tasks.workunit.client.0.smithi070.stdout:child: image128 2024-09-21T10:12:28.874 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image125/86198bbe93fc 2024-09-21T10:12:28.874 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.874 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-30/image126/86191910a208 2024-09-21T10:12:28.874 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.874 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image127/86197d9eb137 2024-09-21T10:12:28.874 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.874 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image128/86198abeee52 2024-09-21T10:12:28.874 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.874 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.874 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.874 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.874 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.875 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.875 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.875 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-31/image127 2024-09-21T10:12:28.875 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.875 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-31/image127 2024-09-21T10:12:28.875 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image127 2024-09-21T10:12:28.875 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.875 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.875 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-31/image128 2024-09-21T10:12:28.875 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.875 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-31/image127 2024-09-21T10:12:28.875 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-31/image128 2024-09-21T10:12:28.875 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image128 2024-09-21T10:12:28.875 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.875 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 3 2024-09-21T10:12:28.875 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 3 2024-09-21T10:12:28.876 INFO:tasks.workunit.client.0.smithi070.stdout:child: image126 2024-09-21T10:12:28.876 INFO:tasks.workunit.client.0.smithi070.stdout:child: image127 2024-09-21T10:12:28.876 INFO:tasks.workunit.client.0.smithi070.stdout:child: image128 2024-09-21T10:12:28.876 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-30/image126/86191910a208 2024-09-21T10:12:28.876 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.876 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image127/86197d9eb137 2024-09-21T10:12:28.876 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.876 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image128/86198abeee52 2024-09-21T10:12:28.876 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.876 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.876 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.876 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.876 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.876 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.876 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.877 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-31/image128 2024-09-21T10:12:28.877 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.877 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-31/image128 2024-09-21T10:12:28.877 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image128 2024-09-21T10:12:28.877 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.877 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 2 2024-09-21T10:12:28.877 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 2 2024-09-21T10:12:28.877 INFO:tasks.workunit.client.0.smithi070.stdout:child: image126 2024-09-21T10:12:28.877 INFO:tasks.workunit.client.0.smithi070.stdout:child: image128 2024-09-21T10:12:28.877 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-30/image126/86191910a208 2024-09-21T10:12:28.877 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.877 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-31/image128/86198abeee52 2024-09-21T10:12:28.877 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.877 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.877 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.878 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.878 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-30/image126 2024-09-21T10:12:28.878 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.878 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 1 2024-09-21T10:12:28.878 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-21T10:12:28.878 INFO:tasks.workunit.client.0.smithi070.stdout:child: image126 2024-09-21T10:12:28.878 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-30/image126/86191910a208 2024-09-21T10:12:28.878 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:28.878 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 0 2024-09-21T10:12:28.878 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 0 2024-09-21T10:12:28.878 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ListChildren (9965 ms) 2024-09-21T10:12:28.878 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ListChildrenTiered 2024-09-21T10:12:29.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:28 smithi160 ceph-mon[101879]: pgmap v727: 617 pgs: 617 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 2.5 KiB/s wr, 42 op/s 2024-09-21T10:12:29.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:28 smithi070 ceph-mon[107385]: pgmap v727: 617 pgs: 617 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 2.5 KiB/s wr, 42 op/s 2024-09-21T10:12:29.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:28 smithi070 ceph-mon[112139]: pgmap v727: 617 pgs: 617 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 2.5 KiB/s wr, 42 op/s 2024-09-21T10:12:30.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:29 smithi160 ceph-mon[101879]: osdmap e720: 8 total, 8 up, 8 in 2024-09-21T10:12:30.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:29 smithi070 ceph-mon[107385]: osdmap e720: 8 total, 8 up, 8 in 2024-09-21T10:12:30.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:29 smithi070 ceph-mon[112139]: osdmap e720: 8 total, 8 up, 8 in 2024-09-21T10:12:31.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:30 smithi160 ceph-mon[101879]: osdmap e721: 8 total, 8 up, 8 in 2024-09-21T10:12:31.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:30 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4279500303' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-32","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:31.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:30 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-32","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:31.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:30 smithi160 ceph-mon[101879]: pgmap v730: 649 pgs: 32 unknown, 617 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 138 KiB/s rd, 4.2 KiB/s wr, 164 op/s 2024-09-21T10:12:31.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:30 smithi070 ceph-mon[107385]: osdmap e721: 8 total, 8 up, 8 in 2024-09-21T10:12:31.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:30 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4279500303' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-32","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:31.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:30 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-32","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:31.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:30 smithi070 ceph-mon[107385]: pgmap v730: 649 pgs: 32 unknown, 617 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 138 KiB/s rd, 4.2 KiB/s wr, 164 op/s 2024-09-21T10:12:31.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:30 smithi070 ceph-mon[112139]: osdmap e721: 8 total, 8 up, 8 in 2024-09-21T10:12:31.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:30 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4279500303' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-32","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:31.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:30 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-32","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:31.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:30 smithi070 ceph-mon[112139]: pgmap v730: 649 pgs: 32 unknown, 617 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 138 KiB/s rd, 4.2 KiB/s wr, 164 op/s 2024-09-21T10:12:32.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:31 smithi160 ceph-mon[101879]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:12:32.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:31 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:12:32.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:31 smithi160 ceph-mon[101879]: osdmap e722: 8 total, 8 up, 8 in 2024-09-21T10:12:32.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:31 smithi070 ceph-mon[107385]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:12:32.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:31 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:12:32.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:31 smithi070 ceph-mon[107385]: osdmap e722: 8 total, 8 up, 8 in 2024-09-21T10:12:32.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:31 smithi070 ceph-mon[112139]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:12:32.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:31 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:12:32.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:31 smithi070 ceph-mon[112139]: osdmap e722: 8 total, 8 up, 8 in 2024-09-21T10:12:33.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:32 smithi160 ceph-mon[101879]: osdmap e723: 8 total, 8 up, 8 in 2024-09-21T10:12:33.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:32 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3390487014' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:33.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:32 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:33.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:32 smithi160 ceph-mon[101879]: pgmap v733: 681 pgs: 52 unknown, 629 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 564 KiB/s rd, 14 KiB/s wr, 687 op/s 2024-09-21T10:12:33.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:32 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:12:33.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:32 smithi160 ceph-mon[101879]: osdmap e724: 8 total, 8 up, 8 in 2024-09-21T10:12:33.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:32 smithi070 ceph-mon[107385]: osdmap e723: 8 total, 8 up, 8 in 2024-09-21T10:12:33.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:32 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3390487014' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:33.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:32 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:33.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:32 smithi070 ceph-mon[107385]: pgmap v733: 681 pgs: 52 unknown, 629 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 564 KiB/s rd, 14 KiB/s wr, 687 op/s 2024-09-21T10:12:33.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:32 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:12:33.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:32 smithi070 ceph-mon[107385]: osdmap e724: 8 total, 8 up, 8 in 2024-09-21T10:12:33.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:32 smithi070 ceph-mon[112139]: osdmap e723: 8 total, 8 up, 8 in 2024-09-21T10:12:33.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:32 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3390487014' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:33.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:32 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:33.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:32 smithi070 ceph-mon[112139]: pgmap v733: 681 pgs: 52 unknown, 629 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 564 KiB/s rd, 14 KiB/s wr, 687 op/s 2024-09-21T10:12:33.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:32 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:12:33.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:32 smithi070 ceph-mon[112139]: osdmap e724: 8 total, 8 up, 8 in 2024-09-21T10:12:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:12:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:12:33] "GET /metrics HTTP/1.1" 200 72548 "" "Prometheus/2.51.0" 2024-09-21T10:12:35.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:34 smithi160 ceph-mon[101879]: osdmap e725: 8 total, 8 up, 8 in 2024-09-21T10:12:35.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:34 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/192679063' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-34","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:35.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:34 smithi160 ceph-mon[101879]: pgmap v736: 713 pgs: 62 unknown, 651 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 426 KiB/s rd, 9.5 KiB/s wr, 522 op/s 2024-09-21T10:12:35.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:34 smithi070 ceph-mon[107385]: osdmap e725: 8 total, 8 up, 8 in 2024-09-21T10:12:35.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:34 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/192679063' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-34","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:35.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:34 smithi070 ceph-mon[107385]: pgmap v736: 713 pgs: 62 unknown, 651 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 426 KiB/s rd, 9.5 KiB/s wr, 522 op/s 2024-09-21T10:12:35.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:34 smithi070 ceph-mon[112139]: osdmap e725: 8 total, 8 up, 8 in 2024-09-21T10:12:35.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:34 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/192679063' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-34","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:35.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:34 smithi070 ceph-mon[112139]: pgmap v736: 713 pgs: 62 unknown, 651 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 426 KiB/s rd, 9.5 KiB/s wr, 522 op/s 2024-09-21T10:12:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:35 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/192679063' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:12:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:35 smithi160 ceph-mon[101879]: osdmap e726: 8 total, 8 up, 8 in 2024-09-21T10:12:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:35 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-librbd-smithi070-86016-32", "tierpool":"test-librbd-smithi070-86016-34", "force_nonempty":""} : dispatch 2024-09-21T10:12:36.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:12:36.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:36.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:35 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/192679063' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:12:36.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:35 smithi070 ceph-mon[107385]: osdmap e726: 8 total, 8 up, 8 in 2024-09-21T10:12:36.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:35 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-librbd-smithi070-86016-32", "tierpool":"test-librbd-smithi070-86016-34", "force_nonempty":""} : dispatch 2024-09-21T10:12:36.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:12:36.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:36.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:35 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/192679063' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:12:36.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:35 smithi070 ceph-mon[112139]: osdmap e726: 8 total, 8 up, 8 in 2024-09-21T10:12:36.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:35 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-librbd-smithi070-86016-32", "tierpool":"test-librbd-smithi070-86016-34", "force_nonempty":""} : dispatch 2024-09-21T10:12:36.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:12:36.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:37.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:36 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-librbd-smithi070-86016-32", "tierpool":"test-librbd-smithi070-86016-34", "force_nonempty":""}]': finished 2024-09-21T10:12:37.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:36 smithi160 ceph-mon[101879]: osdmap e727: 8 total, 8 up, 8 in 2024-09-21T10:12:37.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:36 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-librbd-smithi070-86016-34", "mode":"writeback"} : dispatch 2024-09-21T10:12:37.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:36 smithi160 ceph-mon[101879]: pgmap v739: 713 pgs: 12 creating+peering, 3 unknown, 698 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:12:37.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:36 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-librbd-smithi070-86016-32", "tierpool":"test-librbd-smithi070-86016-34", "force_nonempty":""}]': finished 2024-09-21T10:12:37.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:36 smithi070 ceph-mon[107385]: osdmap e727: 8 total, 8 up, 8 in 2024-09-21T10:12:37.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:36 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-librbd-smithi070-86016-34", "mode":"writeback"} : dispatch 2024-09-21T10:12:37.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:36 smithi070 ceph-mon[107385]: pgmap v739: 713 pgs: 12 creating+peering, 3 unknown, 698 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:12:37.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:36 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-librbd-smithi070-86016-32", "tierpool":"test-librbd-smithi070-86016-34", "force_nonempty":""}]': finished 2024-09-21T10:12:37.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:36 smithi070 ceph-mon[112139]: osdmap e727: 8 total, 8 up, 8 in 2024-09-21T10:12:37.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:36 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-librbd-smithi070-86016-34", "mode":"writeback"} : dispatch 2024-09-21T10:12:37.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:36 smithi070 ceph-mon[112139]: pgmap v739: 713 pgs: 12 creating+peering, 3 unknown, 698 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:12:37.780 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:38.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:37 smithi160 ceph-mon[101879]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:12:38.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:37 smithi160 ceph-mon[101879]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-21T10:12:38.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:37 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-librbd-smithi070-86016-34", "mode":"writeback"}]': finished 2024-09-21T10:12:38.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:37 smithi160 ceph-mon[101879]: osdmap e728: 8 total, 8 up, 8 in 2024-09-21T10:12:38.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:37 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-librbd-smithi070-86016-32", "overlaypool":"test-librbd-smithi070-86016-34"} : dispatch 2024-09-21T10:12:38.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:37 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-librbd-smithi070-86016-32", "overlaypool":"test-librbd-smithi070-86016-34"}]': finished 2024-09-21T10:12:38.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:37 smithi160 ceph-mon[101879]: osdmap e729: 8 total, 8 up, 8 in 2024-09-21T10:12:38.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:37 smithi070 ceph-mon[107385]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:12:38.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:37 smithi070 ceph-mon[107385]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-21T10:12:38.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:37 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-librbd-smithi070-86016-34", "mode":"writeback"}]': finished 2024-09-21T10:12:38.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:37 smithi070 ceph-mon[107385]: osdmap e728: 8 total, 8 up, 8 in 2024-09-21T10:12:38.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:37 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-librbd-smithi070-86016-32", "overlaypool":"test-librbd-smithi070-86016-34"} : dispatch 2024-09-21T10:12:38.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:37 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-librbd-smithi070-86016-32", "overlaypool":"test-librbd-smithi070-86016-34"}]': finished 2024-09-21T10:12:38.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:37 smithi070 ceph-mon[107385]: osdmap e729: 8 total, 8 up, 8 in 2024-09-21T10:12:38.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:37 smithi070 ceph-mon[112139]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:12:38.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:37 smithi070 ceph-mon[112139]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-21T10:12:38.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:37 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-librbd-smithi070-86016-34", "mode":"writeback"}]': finished 2024-09-21T10:12:38.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:37 smithi070 ceph-mon[112139]: osdmap e728: 8 total, 8 up, 8 in 2024-09-21T10:12:38.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:37 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-librbd-smithi070-86016-32", "overlaypool":"test-librbd-smithi070-86016-34"} : dispatch 2024-09-21T10:12:38.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:37 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-librbd-smithi070-86016-32", "overlaypool":"test-librbd-smithi070-86016-34"}]': finished 2024-09-21T10:12:38.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:37 smithi070 ceph-mon[112139]: osdmap e729: 8 total, 8 up, 8 in 2024-09-21T10:12:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:38 smithi160 ceph-mon[101879]: pgmap v742: 713 pgs: 12 creating+peering, 3 unknown, 698 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:12:39.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:38 smithi070 ceph-mon[107385]: pgmap v742: 713 pgs: 12 creating+peering, 3 unknown, 698 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:12:39.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:38 smithi070 ceph-mon[112139]: pgmap v742: 713 pgs: 12 creating+peering, 3 unknown, 698 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:12:40.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:39 smithi160 ceph-mon[101879]: osdmap e730: 8 total, 8 up, 8 in 2024-09-21T10:12:40.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:39 smithi070 ceph-mon[107385]: osdmap e730: 8 total, 8 up, 8 in 2024-09-21T10:12:40.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:39 smithi070 ceph-mon[112139]: osdmap e730: 8 total, 8 up, 8 in 2024-09-21T10:12:41.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:40 smithi160 ceph-mon[101879]: osdmap e731: 8 total, 8 up, 8 in 2024-09-21T10:12:41.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:40 smithi160 ceph-mon[101879]: pgmap v745: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:12:41.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:40 smithi070 ceph-mon[107385]: osdmap e731: 8 total, 8 up, 8 in 2024-09-21T10:12:41.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:40 smithi070 ceph-mon[107385]: pgmap v745: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:12:41.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:40 smithi070 ceph-mon[112139]: osdmap e731: 8 total, 8 up, 8 in 2024-09-21T10:12:41.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:40 smithi070 ceph-mon[112139]: pgmap v745: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:12:42.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:41 smithi160 ceph-mon[101879]: osdmap e732: 8 total, 8 up, 8 in 2024-09-21T10:12:42.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:41 smithi070 ceph-mon[112139]: osdmap e732: 8 total, 8 up, 8 in 2024-09-21T10:12:42.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:41 smithi070 ceph-mon[107385]: osdmap e732: 8 total, 8 up, 8 in 2024-09-21T10:12:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:42 smithi160 ceph-mon[101879]: osdmap e733: 8 total, 8 up, 8 in 2024-09-21T10:12:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:42 smithi160 ceph-mon[101879]: pgmap v748: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 2.2 KiB/s wr, 39 op/s 2024-09-21T10:12:43.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:42 smithi070 ceph-mon[107385]: osdmap e733: 8 total, 8 up, 8 in 2024-09-21T10:12:43.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:42 smithi070 ceph-mon[107385]: pgmap v748: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 2.2 KiB/s wr, 39 op/s 2024-09-21T10:12:43.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:42 smithi070 ceph-mon[112139]: osdmap e733: 8 total, 8 up, 8 in 2024-09-21T10:12:43.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:42 smithi070 ceph-mon[112139]: pgmap v748: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 2.2 KiB/s wr, 39 op/s 2024-09-21T10:12:43.336 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.336 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-33/image130 2024-09-21T10:12:43.337 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-33/image130 2024-09-21T10:12:43.337 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image130 2024-09-21T10:12:43.337 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.337 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 1 2024-09-21T10:12:43.337 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-21T10:12:43.337 INFO:tasks.workunit.client.0.smithi070.stdout:child: image130 2024-09-21T10:12:43.337 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image130/861960e17a28 2024-09-21T10:12:43.337 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.337 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.337 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-33/image130 2024-09-21T10:12:43.337 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:43.338 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-33/image130 2024-09-21T10:12:43.338 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image130 2024-09-21T10:12:43.338 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.338 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.338 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:43.338 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:43.338 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:43.338 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.338 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 2 2024-09-21T10:12:43.338 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 2 2024-09-21T10:12:43.338 INFO:tasks.workunit.client.0.smithi070.stdout:child: image131 2024-09-21T10:12:43.338 INFO:tasks.workunit.client.0.smithi070.stdout:child: image130 2024-09-21T10:12:43.338 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image130/861960e17a28 2024-09-21T10:12:43.338 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.339 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-32/image131/8619377cf470 2024-09-21T10:12:43.339 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.339 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.339 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-33/image130 2024-09-21T10:12:43.339 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:43.339 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-33/image130 2024-09-21T10:12:43.339 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image130 2024-09-21T10:12:43.339 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.339 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.339 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:43.339 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:43.339 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:43.340 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.340 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.340 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-33/image132 2024-09-21T10:12:43.340 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:43.340 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-33/image130 2024-09-21T10:12:43.340 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-33/image132 2024-09-21T10:12:43.340 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image132 2024-09-21T10:12:43.340 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.340 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 3 2024-09-21T10:12:43.340 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 3 2024-09-21T10:12:43.340 INFO:tasks.workunit.client.0.smithi070.stdout:child: image131 2024-09-21T10:12:43.340 INFO:tasks.workunit.client.0.smithi070.stdout:child: image130 2024-09-21T10:12:43.340 INFO:tasks.workunit.client.0.smithi070.stdout:child: image132 2024-09-21T10:12:43.340 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image130/861960e17a28 2024-09-21T10:12:43.341 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.341 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-32/image131/8619377cf470 2024-09-21T10:12:43.341 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.341 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image132/8619997e61d0 2024-09-21T10:12:43.341 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.341 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.341 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-33/image130 2024-09-21T10:12:43.341 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:43.341 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-33/image130 2024-09-21T10:12:43.341 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image130 2024-09-21T10:12:43.341 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.341 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.341 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:43.341 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:43.342 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:43.342 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.342 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 3 2024-09-21T10:12:43.342 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 3 2024-09-21T10:12:43.342 INFO:tasks.workunit.client.0.smithi070.stdout:child: image131 2024-09-21T10:12:43.342 INFO:tasks.workunit.client.0.smithi070.stdout:child: image130 2024-09-21T10:12:43.342 INFO:tasks.workunit.client.0.smithi070.stdout:child: image132 2024-09-21T10:12:43.342 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image130/861960e17a28 2024-09-21T10:12:43.342 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.342 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-32/image131/8619377cf470 2024-09-21T10:12:43.342 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.342 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image132/8619997e61d0 2024-09-21T10:12:43.342 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.343 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.343 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-33/image130 2024-09-21T10:12:43.343 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:43.343 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-33/image130 2024-09-21T10:12:43.343 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image130 2024-09-21T10:12:43.343 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.343 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.343 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:43.343 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:43.343 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:43.343 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.343 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.343 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-33/image133 2024-09-21T10:12:43.343 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:43.344 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-33/image130 2024-09-21T10:12:43.344 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-33/image133 2024-09-21T10:12:43.344 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image133 2024-09-21T10:12:43.344 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.344 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 4 2024-09-21T10:12:43.344 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 4 2024-09-21T10:12:43.344 INFO:tasks.workunit.client.0.smithi070.stdout:child: image131 2024-09-21T10:12:43.344 INFO:tasks.workunit.client.0.smithi070.stdout:child: image130 2024-09-21T10:12:43.344 INFO:tasks.workunit.client.0.smithi070.stdout:child: image132 2024-09-21T10:12:43.344 INFO:tasks.workunit.client.0.smithi070.stdout:child: image133 2024-09-21T10:12:43.344 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image130/861960e17a28 2024-09-21T10:12:43.344 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.344 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-32/image131/8619377cf470 2024-09-21T10:12:43.344 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.345 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image132/8619997e61d0 2024-09-21T10:12:43.345 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.345 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image133/86192f2261ab 2024-09-21T10:12:43.345 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.345 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.345 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-33/image130 2024-09-21T10:12:43.345 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:43.345 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-33/image130 2024-09-21T10:12:43.345 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image130 2024-09-21T10:12:43.345 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.345 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.345 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:43.345 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:43.345 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:43.346 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.346 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.346 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-33/image132 2024-09-21T10:12:43.346 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:43.346 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-33/image130 2024-09-21T10:12:43.346 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-33/image132 2024-09-21T10:12:43.346 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image132 2024-09-21T10:12:43.346 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.346 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:43.346 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-33/image133 2024-09-21T10:12:43.346 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:12:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:12:43] "GET /metrics HTTP/1.1" 200 82089 "" "Prometheus/2.51.0" 2024-09-21T10:12:44.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:43 smithi070 ceph-mon[107385]: osdmap e734: 8 total, 8 up, 8 in 2024-09-21T10:12:44.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:43 smithi070 ceph-mon[112139]: osdmap e734: 8 total, 8 up, 8 in 2024-09-21T10:12:44.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:43 smithi160 ceph-mon[101879]: osdmap e734: 8 total, 8 up, 8 in 2024-09-21T10:12:45.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:44 smithi070 ceph-mon[107385]: osdmap e735: 8 total, 8 up, 8 in 2024-09-21T10:12:45.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:44 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-librbd-smithi070-86016-32"} : dispatch 2024-09-21T10:12:45.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:44 smithi070 ceph-mon[107385]: pgmap v751: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 150 KiB/s rd, 6.7 KiB/s wr, 185 op/s 2024-09-21T10:12:45.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:44 smithi070 ceph-mon[112139]: osdmap e735: 8 total, 8 up, 8 in 2024-09-21T10:12:45.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:44 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-librbd-smithi070-86016-32"} : dispatch 2024-09-21T10:12:45.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:44 smithi070 ceph-mon[112139]: pgmap v751: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 150 KiB/s rd, 6.7 KiB/s wr, 185 op/s 2024-09-21T10:12:45.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:44 smithi160 ceph-mon[101879]: osdmap e735: 8 total, 8 up, 8 in 2024-09-21T10:12:45.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:44 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-librbd-smithi070-86016-32"} : dispatch 2024-09-21T10:12:45.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:44 smithi160 ceph-mon[101879]: pgmap v751: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 150 KiB/s rd, 6.7 KiB/s wr, 185 op/s 2024-09-21T10:12:45.847 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-33/image130 2024-09-21T10:12:45.848 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-33/image132 2024-09-21T10:12:45.848 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-33/image133 2024-09-21T10:12:45.848 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image133 2024-09-21T10:12:45.848 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:45.848 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 4 2024-09-21T10:12:45.848 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 4 2024-09-21T10:12:45.848 INFO:tasks.workunit.client.0.smithi070.stdout:child: image131 2024-09-21T10:12:45.848 INFO:tasks.workunit.client.0.smithi070.stdout:child: image130 2024-09-21T10:12:45.848 INFO:tasks.workunit.client.0.smithi070.stdout:child: image132 2024-09-21T10:12:45.848 INFO:tasks.workunit.client.0.smithi070.stdout:child: image133 2024-09-21T10:12:45.848 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image130/861960e17a28 2024-09-21T10:12:45.848 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:45.849 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-32/image131/8619377cf470 2024-09-21T10:12:45.849 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:45.849 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image132/8619997e61d0 2024-09-21T10:12:45.849 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:45.849 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image133/86192f2261ab 2024-09-21T10:12:45.849 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:45.849 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:45.849 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:45.849 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:45.849 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:45.849 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:45.849 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:45.849 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-33/image132 2024-09-21T10:12:45.850 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:45.850 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-33/image132 2024-09-21T10:12:45.850 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image132 2024-09-21T10:12:45.850 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:45.850 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:45.850 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-33/image133 2024-09-21T10:12:45.850 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:45.850 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-33/image132 2024-09-21T10:12:45.850 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-33/image133 2024-09-21T10:12:45.850 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image133 2024-09-21T10:12:45.850 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:45.850 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 3 2024-09-21T10:12:45.850 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 3 2024-09-21T10:12:45.851 INFO:tasks.workunit.client.0.smithi070.stdout:child: image131 2024-09-21T10:12:45.851 INFO:tasks.workunit.client.0.smithi070.stdout:child: image132 2024-09-21T10:12:45.851 INFO:tasks.workunit.client.0.smithi070.stdout:child: image133 2024-09-21T10:12:45.851 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-32/image131/8619377cf470 2024-09-21T10:12:45.851 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:45.851 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image132/8619997e61d0 2024-09-21T10:12:45.851 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:45.851 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image133/86192f2261ab 2024-09-21T10:12:45.851 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:45.851 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:45.851 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:45.851 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:45.851 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:45.851 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:45.852 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:45.852 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-33/image133 2024-09-21T10:12:45.852 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:45.852 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-33/image133 2024-09-21T10:12:45.852 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image133 2024-09-21T10:12:45.852 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:45.852 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 2 2024-09-21T10:12:45.852 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 2 2024-09-21T10:12:45.852 INFO:tasks.workunit.client.0.smithi070.stdout:child: image131 2024-09-21T10:12:45.852 INFO:tasks.workunit.client.0.smithi070.stdout:child: image133 2024-09-21T10:12:45.852 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-32/image131/8619377cf470 2024-09-21T10:12:45.852 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:45.852 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-33/image133/86192f2261ab 2024-09-21T10:12:45.853 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:45.853 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:45.853 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:45.853 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:45.853 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-32/image131 2024-09-21T10:12:45.853 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:45.853 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 1 2024-09-21T10:12:45.853 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-21T10:12:45.853 INFO:tasks.workunit.client.0.smithi070.stdout:child: image131 2024-09-21T10:12:45.853 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-32/image131/8619377cf470 2024-09-21T10:12:45.853 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:12:45.853 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 0 2024-09-21T10:12:45.853 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 0 2024-09-21T10:12:45.853 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ListChildrenTiered (16975 ms) 2024-09-21T10:12:45.854 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.LockingPP 2024-09-21T10:12:45.854 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:45.915 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.LockingPP (67 ms) 2024-09-21T10:12:45.915 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.FlushAio 2024-09-21T10:12:45.915 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:46.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:45 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-librbd-smithi070-86016-32"}]': finished 2024-09-21T10:12:46.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:45 smithi070 ceph-mon[107385]: osdmap e736: 8 total, 8 up, 8 in 2024-09-21T10:12:46.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:45 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-librbd-smithi070-86016-32", "tierpool":"test-librbd-smithi070-86016-34"} : dispatch 2024-09-21T10:12:46.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:45 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-librbd-smithi070-86016-32"}]': finished 2024-09-21T10:12:46.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:45 smithi070 ceph-mon[112139]: osdmap e736: 8 total, 8 up, 8 in 2024-09-21T10:12:46.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:45 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-librbd-smithi070-86016-32", "tierpool":"test-librbd-smithi070-86016-34"} : dispatch 2024-09-21T10:12:46.137 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.FlushAio (223 ms) 2024-09-21T10:12:46.137 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.FlushAioPP 2024-09-21T10:12:46.137 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:46.324 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.FlushAioPP (187 ms) 2024-09-21T10:12:46.324 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ZeroLengthWrite 2024-09-21T10:12:46.324 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:46.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:45 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-librbd-smithi070-86016-32"}]': finished 2024-09-21T10:12:46.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:45 smithi160 ceph-mon[101879]: osdmap e736: 8 total, 8 up, 8 in 2024-09-21T10:12:46.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:45 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-librbd-smithi070-86016-32", "tierpool":"test-librbd-smithi070-86016-34"} : dispatch 2024-09-21T10:12:46.345 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ZeroLengthWrite (21 ms) 2024-09-21T10:12:46.345 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ZeroLengthDiscard 2024-09-21T10:12:46.345 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:46.373 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ZeroLengthDiscard (28 ms) 2024-09-21T10:12:46.373 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ZeroLengthRead 2024-09-21T10:12:46.373 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:46.390 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ZeroLengthRead (16 ms) 2024-09-21T10:12:46.390 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.LargeCacheRead 2024-09-21T10:12:46.390 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:46.425 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.LargeCacheRead (35 ms) 2024-09-21T10:12:46.425 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestPendingAio 2024-09-21T10:12:46.425 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:46.644 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestPendingAio (218 ms) 2024-09-21T10:12:46.644 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.CompareAndWriteCopyup 2024-09-21T10:12:46.644 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:46.950 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:47.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:46 smithi070 ceph-mon[107385]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-21T10:12:47.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:46 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-librbd-smithi070-86016-32", "tierpool":"test-librbd-smithi070-86016-34"}]': finished 2024-09-21T10:12:47.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:46 smithi070 ceph-mon[107385]: osdmap e737: 8 total, 8 up, 8 in 2024-09-21T10:12:47.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:46 smithi070 ceph-mon[107385]: pgmap v754: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 580 KiB/s rd, 15 KiB/s wr, 710 op/s 2024-09-21T10:12:47.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:46 smithi070 ceph-mon[112139]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-21T10:12:47.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:46 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-librbd-smithi070-86016-32", "tierpool":"test-librbd-smithi070-86016-34"}]': finished 2024-09-21T10:12:47.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:46 smithi070 ceph-mon[112139]: osdmap e737: 8 total, 8 up, 8 in 2024-09-21T10:12:47.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:46 smithi070 ceph-mon[112139]: pgmap v754: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 580 KiB/s rd, 15 KiB/s wr, 710 op/s 2024-09-21T10:12:47.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:46 smithi160 ceph-mon[101879]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-21T10:12:47.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:46 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-librbd-smithi070-86016-32", "tierpool":"test-librbd-smithi070-86016-34"}]': finished 2024-09-21T10:12:47.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:46 smithi160 ceph-mon[101879]: osdmap e737: 8 total, 8 up, 8 in 2024-09-21T10:12:47.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:46 smithi160 ceph-mon[101879]: pgmap v754: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 580 KiB/s rd, 15 KiB/s wr, 710 op/s 2024-09-21T10:12:48.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:47 smithi070 ceph-mon[107385]: osdmap e738: 8 total, 8 up, 8 in 2024-09-21T10:12:48.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:47 smithi070 ceph-mon[107385]: osdmap e739: 8 total, 8 up, 8 in 2024-09-21T10:12:48.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:47 smithi070 ceph-mon[112139]: osdmap e738: 8 total, 8 up, 8 in 2024-09-21T10:12:48.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:47 smithi070 ceph-mon[112139]: osdmap e739: 8 total, 8 up, 8 in 2024-09-21T10:12:48.242 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:47 smithi160 ceph-mon[101879]: osdmap e738: 8 total, 8 up, 8 in 2024-09-21T10:12:48.243 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:47 smithi160 ceph-mon[101879]: osdmap e739: 8 total, 8 up, 8 in 2024-09-21T10:12:48.818 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.CompareAndWriteCopyup (2174 ms) 2024-09-21T10:12:48.818 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.CompareAndWriteCopyupMismatch 2024-09-21T10:12:48.818 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:49.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:48 smithi070 ceph-mon[107385]: pgmap v757: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 459 KiB/s rd, 10 KiB/s wr, 563 op/s 2024-09-21T10:12:49.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:48 smithi070 ceph-mon[107385]: osdmap e740: 8 total, 8 up, 8 in 2024-09-21T10:12:49.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:48 smithi070 ceph-mon[112139]: pgmap v757: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 459 KiB/s rd, 10 KiB/s wr, 563 op/s 2024-09-21T10:12:49.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:48 smithi070 ceph-mon[112139]: osdmap e740: 8 total, 8 up, 8 in 2024-09-21T10:12:49.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:48 smithi160 ceph-mon[101879]: pgmap v757: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 459 KiB/s rd, 10 KiB/s wr, 563 op/s 2024-09-21T10:12:49.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:48 smithi160 ceph-mon[101879]: osdmap e740: 8 total, 8 up, 8 in 2024-09-21T10:12:49.867 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:51.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:50 smithi160 ceph-mon[101879]: osdmap e741: 8 total, 8 up, 8 in 2024-09-21T10:12:51.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:12:51.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:51.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.1", "id": [2, 4]} : dispatch 2024-09-21T10:12:51.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.b", "id": [3, 4]} : dispatch 2024-09-21T10:12:51.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.f", "id": [2, 1]} : dispatch 2024-09-21T10:12:51.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "35.1d", "id": [4, 2]} : dispatch 2024-09-21T10:12:51.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "36.b", "id": [6, 3]} : dispatch 2024-09-21T10:12:51.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.13", "id": [0, 6]} : dispatch 2024-09-21T10:12:51.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.1f", "id": [0, 3]} : dispatch 2024-09-21T10:12:51.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.4", "id": [7, 4]} : dispatch 2024-09-21T10:12:51.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.5", "id": [0, 1]} : dispatch 2024-09-21T10:12:51.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.16", "id": [0, 1]} : dispatch 2024-09-21T10:12:51.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:50 smithi070 ceph-mon[107385]: osdmap e741: 8 total, 8 up, 8 in 2024-09-21T10:12:51.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:12:51.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:51.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.1", "id": [2, 4]} : dispatch 2024-09-21T10:12:51.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.b", "id": [3, 4]} : dispatch 2024-09-21T10:12:51.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.f", "id": [2, 1]} : dispatch 2024-09-21T10:12:51.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "35.1d", "id": [4, 2]} : dispatch 2024-09-21T10:12:51.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "36.b", "id": [6, 3]} : dispatch 2024-09-21T10:12:51.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.13", "id": [0, 6]} : dispatch 2024-09-21T10:12:51.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.1f", "id": [0, 3]} : dispatch 2024-09-21T10:12:51.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.4", "id": [7, 4]} : dispatch 2024-09-21T10:12:51.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.5", "id": [0, 1]} : dispatch 2024-09-21T10:12:51.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.16", "id": [0, 1]} : dispatch 2024-09-21T10:12:51.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:50 smithi070 ceph-mon[112139]: osdmap e741: 8 total, 8 up, 8 in 2024-09-21T10:12:51.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:12:51.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:51.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.1", "id": [2, 4]} : dispatch 2024-09-21T10:12:51.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.b", "id": [3, 4]} : dispatch 2024-09-21T10:12:51.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.f", "id": [2, 1]} : dispatch 2024-09-21T10:12:51.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "35.1d", "id": [4, 2]} : dispatch 2024-09-21T10:12:51.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "36.b", "id": [6, 3]} : dispatch 2024-09-21T10:12:51.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.13", "id": [0, 6]} : dispatch 2024-09-21T10:12:51.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.1f", "id": [0, 3]} : dispatch 2024-09-21T10:12:51.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.4", "id": [7, 4]} : dispatch 2024-09-21T10:12:51.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.5", "id": [0, 1]} : dispatch 2024-09-21T10:12:51.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.16", "id": [0, 1]} : dispatch 2024-09-21T10:12:51.830 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.CompareAndWriteCopyupMismatch (3013 ms) 2024-09-21T10:12:51.831 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.Flatten 2024-09-21T10:12:51.831 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:52.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:51 smithi160 ceph-mon[101879]: pgmap v760: 713 pgs: 713 active+clean; 299 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 148 KiB/s rd, 137 KiB/s wr, 324 op/s 2024-09-21T10:12:52.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.1", "id": [2, 4]}]': finished 2024-09-21T10:12:52.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.b", "id": [3, 4]}]': finished 2024-09-21T10:12:52.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.f", "id": [2, 1]}]': finished 2024-09-21T10:12:52.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "35.1d", "id": [4, 2]}]': finished 2024-09-21T10:12:52.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "36.b", "id": [6, 3]}]': finished 2024-09-21T10:12:52.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.13", "id": [0, 6]}]': finished 2024-09-21T10:12:52.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.1f", "id": [0, 3]}]': finished 2024-09-21T10:12:52.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.4", "id": [7, 4]}]': finished 2024-09-21T10:12:52.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.5", "id": [0, 1]}]': finished 2024-09-21T10:12:52.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.16", "id": [0, 1]}]': finished 2024-09-21T10:12:52.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:51 smithi160 ceph-mon[101879]: osdmap e742: 8 total, 8 up, 8 in 2024-09-21T10:12:52.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:51 smithi070 ceph-mon[107385]: pgmap v760: 713 pgs: 713 active+clean; 299 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 148 KiB/s rd, 137 KiB/s wr, 324 op/s 2024-09-21T10:12:52.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.1", "id": [2, 4]}]': finished 2024-09-21T10:12:52.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.b", "id": [3, 4]}]': finished 2024-09-21T10:12:52.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.f", "id": [2, 1]}]': finished 2024-09-21T10:12:52.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "35.1d", "id": [4, 2]}]': finished 2024-09-21T10:12:52.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "36.b", "id": [6, 3]}]': finished 2024-09-21T10:12:52.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.13", "id": [0, 6]}]': finished 2024-09-21T10:12:52.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.1f", "id": [0, 3]}]': finished 2024-09-21T10:12:52.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.4", "id": [7, 4]}]': finished 2024-09-21T10:12:52.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.5", "id": [0, 1]}]': finished 2024-09-21T10:12:52.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.16", "id": [0, 1]}]': finished 2024-09-21T10:12:52.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:51 smithi070 ceph-mon[107385]: osdmap e742: 8 total, 8 up, 8 in 2024-09-21T10:12:52.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:51 smithi070 ceph-mon[112139]: pgmap v760: 713 pgs: 713 active+clean; 299 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 148 KiB/s rd, 137 KiB/s wr, 324 op/s 2024-09-21T10:12:52.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.1", "id": [2, 4]}]': finished 2024-09-21T10:12:52.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.b", "id": [3, 4]}]': finished 2024-09-21T10:12:52.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.f", "id": [2, 1]}]': finished 2024-09-21T10:12:52.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "35.1d", "id": [4, 2]}]': finished 2024-09-21T10:12:52.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "36.b", "id": [6, 3]}]': finished 2024-09-21T10:12:52.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.13", "id": [0, 6]}]': finished 2024-09-21T10:12:52.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.1f", "id": [0, 3]}]': finished 2024-09-21T10:12:52.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.4", "id": [7, 4]}]': finished 2024-09-21T10:12:52.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.5", "id": [0, 1]}]': finished 2024-09-21T10:12:52.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.16", "id": [0, 1]}]': finished 2024-09-21T10:12:52.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:51 smithi070 ceph-mon[112139]: osdmap e742: 8 total, 8 up, 8 in 2024-09-21T10:12:52.878 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.Flatten (1047 ms) 2024-09-21T10:12:52.878 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.Sparsify 2024-09-21T10:12:52.878 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:52.909 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 32768 2024-09-21T10:12:52.909 INFO:tasks.workunit.client.0.smithi070.stdout:read: 32768 2024-09-21T10:12:52.909 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.Sparsify (31 ms) 2024-09-21T10:12:52.909 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.SparsifyPP 2024-09-21T10:12:52.909 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:52.943 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.SparsifyPP (35 ms) 2024-09-21T10:12:52.943 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.SnapshotLimit 2024-09-21T10:12:52.943 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:53.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:52 smithi160 ceph-mon[101879]: osdmap e743: 8 total, 8 up, 8 in 2024-09-21T10:12:53.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:52 smithi160 ceph-mon[101879]: osdmap e744: 8 total, 8 up, 8 in 2024-09-21T10:12:53.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:52 smithi070 ceph-mon[107385]: osdmap e743: 8 total, 8 up, 8 in 2024-09-21T10:12:53.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:52 smithi070 ceph-mon[107385]: osdmap e744: 8 total, 8 up, 8 in 2024-09-21T10:12:53.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:52 smithi070 ceph-mon[112139]: osdmap e743: 8 total, 8 up, 8 in 2024-09-21T10:12:53.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:52 smithi070 ceph-mon[112139]: osdmap e744: 8 total, 8 up, 8 in 2024-09-21T10:12:53.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:12:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:12:53] "GET /metrics HTTP/1.1" 200 82167 "" "Prometheus/2.51.0" 2024-09-21T10:12:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:53 smithi160 ceph-mon[101879]: pgmap v763: 713 pgs: 6 peering, 707 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 875 KiB/s rd, 737 KiB/s wr, 672 op/s 2024-09-21T10:12:54.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:53 smithi070 ceph-mon[112139]: pgmap v763: 713 pgs: 6 peering, 707 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 875 KiB/s rd, 737 KiB/s wr, 672 op/s 2024-09-21T10:12:54.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:53 smithi070 ceph-mon[107385]: pgmap v763: 713 pgs: 6 peering, 707 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 875 KiB/s rd, 737 KiB/s wr, 672 op/s 2024-09-21T10:12:55.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:54 smithi160 ceph-mon[101879]: osdmap e745: 8 total, 8 up, 8 in 2024-09-21T10:12:55.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:54 smithi070 ceph-mon[107385]: osdmap e745: 8 total, 8 up, 8 in 2024-09-21T10:12:55.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:54 smithi070 ceph-mon[112139]: osdmap e745: 8 total, 8 up, 8 in 2024-09-21T10:12:56.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:55 smithi160 ceph-mon[101879]: pgmap v766: 713 pgs: 6 peering, 707 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 810 KiB/s rd, 619 KiB/s wr, 441 op/s 2024-09-21T10:12:56.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:55 smithi160 ceph-mon[101879]: osdmap e746: 8 total, 8 up, 8 in 2024-09-21T10:12:56.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:55 smithi070 ceph-mon[107385]: pgmap v766: 713 pgs: 6 peering, 707 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 810 KiB/s rd, 619 KiB/s wr, 441 op/s 2024-09-21T10:12:56.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:55 smithi070 ceph-mon[107385]: osdmap e746: 8 total, 8 up, 8 in 2024-09-21T10:12:56.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:55 smithi070 ceph-mon[112139]: pgmap v766: 713 pgs: 6 peering, 707 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 810 KiB/s rd, 619 KiB/s wr, 441 op/s 2024-09-21T10:12:56.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:55 smithi070 ceph-mon[112139]: osdmap e746: 8 total, 8 up, 8 in 2024-09-21T10:12:57.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:56 smithi160 ceph-mon[101879]: osdmap e747: 8 total, 8 up, 8 in 2024-09-21T10:12:57.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:56 smithi070 ceph-mon[107385]: osdmap e747: 8 total, 8 up, 8 in 2024-09-21T10:12:57.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:56 smithi070 ceph-mon[112139]: osdmap e747: 8 total, 8 up, 8 in 2024-09-21T10:12:57.864 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.SnapshotLimit (4921 ms) 2024-09-21T10:12:57.864 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.SnapshotLimitPP 2024-09-21T10:12:57.864 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:12:58.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:57 smithi070 ceph-mon[107385]: pgmap v769: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 192 KiB/s rd, 37 KiB/s wr, 239 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:12:58.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:57 smithi070 ceph-mon[107385]: osdmap e748: 8 total, 8 up, 8 in 2024-09-21T10:12:58.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:57 smithi070 ceph-mon[112139]: pgmap v769: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 192 KiB/s rd, 37 KiB/s wr, 239 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:12:58.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:57 smithi070 ceph-mon[112139]: osdmap e748: 8 total, 8 up, 8 in 2024-09-21T10:12:58.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:57 smithi160 ceph-mon[101879]: pgmap v769: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 192 KiB/s rd, 37 KiB/s wr, 239 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:12:58.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:57 smithi160 ceph-mon[101879]: osdmap e748: 8 total, 8 up, 8 in 2024-09-21T10:12:59.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:58 smithi070 ceph-mon[107385]: osdmap e749: 8 total, 8 up, 8 in 2024-09-21T10:12:59.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:58 smithi070 ceph-mon[112139]: osdmap e749: 8 total, 8 up, 8 in 2024-09-21T10:12:59.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:58 smithi160 ceph-mon[101879]: osdmap e749: 8 total, 8 up, 8 in 2024-09-21T10:13:00.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:59 smithi070 ceph-mon[107385]: pgmap v772: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 110 KiB/s rd, 18 KiB/s wr, 146 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:13:00.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:12:59 smithi070 ceph-mon[107385]: osdmap e750: 8 total, 8 up, 8 in 2024-09-21T10:13:00.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:59 smithi070 ceph-mon[112139]: pgmap v772: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 110 KiB/s rd, 18 KiB/s wr, 146 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:13:00.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:12:59 smithi070 ceph-mon[112139]: osdmap e750: 8 total, 8 up, 8 in 2024-09-21T10:13:00.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:59 smithi160 ceph-mon[101879]: pgmap v772: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 110 KiB/s rd, 18 KiB/s wr, 146 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:13:00.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:12:59 smithi160 ceph-mon[101879]: osdmap e750: 8 total, 8 up, 8 in 2024-09-21T10:13:01.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:00 smithi070 ceph-mon[107385]: osdmap e751: 8 total, 8 up, 8 in 2024-09-21T10:13:01.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:00 smithi070 ceph-mon[112139]: osdmap e751: 8 total, 8 up, 8 in 2024-09-21T10:13:01.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:00 smithi160 ceph-mon[101879]: osdmap e751: 8 total, 8 up, 8 in 2024-09-21T10:13:02.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:01 smithi070 ceph-mon[107385]: pgmap v775: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 2.5 KiB/s wr, 24 op/s 2024-09-21T10:13:02.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:01 smithi070 ceph-mon[107385]: osdmap e752: 8 total, 8 up, 8 in 2024-09-21T10:13:02.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:01 smithi070 ceph-mon[112139]: pgmap v775: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 2.5 KiB/s wr, 24 op/s 2024-09-21T10:13:02.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:01 smithi070 ceph-mon[112139]: osdmap e752: 8 total, 8 up, 8 in 2024-09-21T10:13:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:01 smithi160 ceph-mon[101879]: pgmap v775: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 2.5 KiB/s wr, 24 op/s 2024-09-21T10:13:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:01 smithi160 ceph-mon[101879]: osdmap e752: 8 total, 8 up, 8 in 2024-09-21T10:13:02.899 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.SnapshotLimitPP (5035 ms) 2024-09-21T10:13:02.899 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.RebuildObjectMapViaLockOwner 2024-09-21T10:13:02.900 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:02.953 INFO:tasks.workunit.client.0.smithi070.stdout:0.00% done 2024-09-21T10:13:02.953 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.RebuildObjectMapViaLockOwner (53 ms) 2024-09-21T10:13:02.953 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.RenameViaLockOwner 2024-09-21T10:13:02.953 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:03.021 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.RenameViaLockOwner (68 ms) 2024-09-21T10:13:03.021 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.SnapCreateViaLockOwner 2024-09-21T10:13:03.022 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:03.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:02 smithi070 ceph-mon[112139]: osdmap e753: 8 total, 8 up, 8 in 2024-09-21T10:13:03.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:02 smithi070 ceph-mon[107385]: osdmap e753: 8 total, 8 up, 8 in 2024-09-21T10:13:03.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:02 smithi160 ceph-mon[101879]: osdmap e753: 8 total, 8 up, 8 in 2024-09-21T10:13:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:13:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:13:03] "GET /metrics HTTP/1.1" 200 82167 "" "Prometheus/2.51.0" 2024-09-21T10:13:03.913 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.SnapCreateViaLockOwner (893 ms) 2024-09-21T10:13:03.913 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.SnapRemoveViaLockOwner 2024-09-21T10:13:03.913 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:04.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:03 smithi070 ceph-mon[107385]: pgmap v778: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 39 KiB/s rd, 4.5 KiB/s wr, 52 op/s 2024-09-21T10:13:04.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:03 smithi070 ceph-mon[107385]: osdmap e754: 8 total, 8 up, 8 in 2024-09-21T10:13:04.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:03 smithi070 ceph-mon[112139]: pgmap v778: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 39 KiB/s rd, 4.5 KiB/s wr, 52 op/s 2024-09-21T10:13:04.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:03 smithi070 ceph-mon[112139]: osdmap e754: 8 total, 8 up, 8 in 2024-09-21T10:13:04.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:03 smithi160 ceph-mon[101879]: pgmap v778: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 39 KiB/s rd, 4.5 KiB/s wr, 52 op/s 2024-09-21T10:13:04.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:03 smithi160 ceph-mon[101879]: osdmap e754: 8 total, 8 up, 8 in 2024-09-21T10:13:05.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:04 smithi160 ceph-mon[101879]: osdmap e755: 8 total, 8 up, 8 in 2024-09-21T10:13:05.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:04 smithi070 ceph-mon[107385]: osdmap e755: 8 total, 8 up, 8 in 2024-09-21T10:13:05.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:04 smithi070 ceph-mon[112139]: osdmap e755: 8 total, 8 up, 8 in 2024-09-21T10:13:05.923 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.SnapRemoveViaLockOwner (2010 ms) 2024-09-21T10:13:05.923 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.UpdateFeaturesViaLockOwner 2024-09-21T10:13:05.923 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:05.973 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.UpdateFeaturesViaLockOwner (49 ms) 2024-09-21T10:13:05.973 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.EnableJournalingViaLockOwner 2024-09-21T10:13:05.973 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/test_librbd.cc:9736: Skipped 2024-09-21T10:13:05.973 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T10:13:05.973 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestLibRBD.EnableJournalingViaLockOwner (0 ms) 2024-09-21T10:13:05.973 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.SnapRemove2 2024-09-21T10:13:05.973 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:05 smithi160 ceph-mon[101879]: pgmap v781: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 3.5 KiB/s wr, 51 op/s 2024-09-21T10:13:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:05 smithi160 ceph-mon[101879]: osdmap e756: 8 total, 8 up, 8 in 2024-09-21T10:13:06.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:06.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:05 smithi070 ceph-mon[107385]: pgmap v781: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 3.5 KiB/s wr, 51 op/s 2024-09-21T10:13:06.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:05 smithi070 ceph-mon[107385]: osdmap e756: 8 total, 8 up, 8 in 2024-09-21T10:13:06.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:06.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:05 smithi070 ceph-mon[112139]: pgmap v781: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 3.5 KiB/s wr, 51 op/s 2024-09-21T10:13:06.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:05 smithi070 ceph-mon[112139]: osdmap e756: 8 total, 8 up, 8 in 2024-09-21T10:13:06.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:07.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:06 smithi160 ceph-mon[101879]: osdmap e757: 8 total, 8 up, 8 in 2024-09-21T10:13:07.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:06 smithi070 ceph-mon[107385]: osdmap e757: 8 total, 8 up, 8 in 2024-09-21T10:13:07.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:06 smithi070 ceph-mon[112139]: osdmap e757: 8 total, 8 up, 8 in 2024-09-21T10:13:07.812 INFO:tasks.workunit.client.0.smithi070.stdout:100.00% done 2024-09-21T10:13:07.812 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.SnapRemove2 (1839 ms) 2024-09-21T10:13:07.813 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.SnapRenameViaLockOwner 2024-09-21T10:13:07.813 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:08.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:07 smithi160 ceph-mon[101879]: pgmap v784: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 139 KiB/s rd, 13 KiB/s wr, 194 op/s 2024-09-21T10:13:08.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:07 smithi160 ceph-mon[101879]: osdmap e758: 8 total, 8 up, 8 in 2024-09-21T10:13:08.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:07 smithi160 ceph-mon[101879]: osdmap e759: 8 total, 8 up, 8 in 2024-09-21T10:13:08.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:07 smithi070 ceph-mon[107385]: pgmap v784: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 139 KiB/s rd, 13 KiB/s wr, 194 op/s 2024-09-21T10:13:08.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:07 smithi070 ceph-mon[107385]: osdmap e758: 8 total, 8 up, 8 in 2024-09-21T10:13:08.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:07 smithi070 ceph-mon[107385]: osdmap e759: 8 total, 8 up, 8 in 2024-09-21T10:13:08.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:07 smithi070 ceph-mon[112139]: pgmap v784: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 139 KiB/s rd, 13 KiB/s wr, 194 op/s 2024-09-21T10:13:08.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:07 smithi070 ceph-mon[112139]: osdmap e758: 8 total, 8 up, 8 in 2024-09-21T10:13:08.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:07 smithi070 ceph-mon[112139]: osdmap e759: 8 total, 8 up, 8 in 2024-09-21T10:13:08.829 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.SnapRenameViaLockOwner (1017 ms) 2024-09-21T10:13:08.829 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.SnapProtectViaLockOwner 2024-09-21T10:13:08.829 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:09.834 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.SnapProtectViaLockOwner (1005 ms) 2024-09-21T10:13:09.834 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.SnapUnprotectViaLockOwner 2024-09-21T10:13:09.835 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:10.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:09 smithi160 ceph-mon[101879]: pgmap v787: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 124 KiB/s rd, 12 KiB/s wr, 170 op/s 2024-09-21T10:13:10.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:09 smithi160 ceph-mon[101879]: osdmap e760: 8 total, 8 up, 8 in 2024-09-21T10:13:10.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:09 smithi070 ceph-mon[107385]: pgmap v787: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 124 KiB/s rd, 12 KiB/s wr, 170 op/s 2024-09-21T10:13:10.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:09 smithi070 ceph-mon[107385]: osdmap e760: 8 total, 8 up, 8 in 2024-09-21T10:13:10.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:09 smithi070 ceph-mon[112139]: pgmap v787: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 124 KiB/s rd, 12 KiB/s wr, 170 op/s 2024-09-21T10:13:10.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:09 smithi070 ceph-mon[112139]: osdmap e760: 8 total, 8 up, 8 in 2024-09-21T10:13:10.850 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.SnapUnprotectViaLockOwner (1016 ms) 2024-09-21T10:13:10.850 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.FlattenViaLockOwner 2024-09-21T10:13:10.851 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:11.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:10 smithi160 ceph-mon[101879]: osdmap e761: 8 total, 8 up, 8 in 2024-09-21T10:13:11.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:10 smithi070 ceph-mon[112139]: osdmap e761: 8 total, 8 up, 8 in 2024-09-21T10:13:11.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:10 smithi070 ceph-mon[107385]: osdmap e761: 8 total, 8 up, 8 in 2024-09-21T10:13:11.910 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.FlattenViaLockOwner (1059 ms) 2024-09-21T10:13:11.910 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ResizeViaLockOwner 2024-09-21T10:13:11.910 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:11.947 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ResizeViaLockOwner (37 ms) 2024-09-21T10:13:11.947 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.SparsifyViaLockOwner 2024-09-21T10:13:11.947 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:11.981 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.SparsifyViaLockOwner (34 ms) 2024-09-21T10:13:11.981 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ObjectMapConsistentSnap 2024-09-21T10:13:11.981 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:12.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:11 smithi160 ceph-mon[101879]: pgmap v790: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 40 KiB/s rd, 2.7 KiB/s wr, 54 op/s 2024-09-21T10:13:12.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:11 smithi160 ceph-mon[101879]: osdmap e762: 8 total, 8 up, 8 in 2024-09-21T10:13:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:11 smithi070 ceph-mon[107385]: pgmap v790: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 40 KiB/s rd, 2.7 KiB/s wr, 54 op/s 2024-09-21T10:13:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:11 smithi070 ceph-mon[107385]: osdmap e762: 8 total, 8 up, 8 in 2024-09-21T10:13:12.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:11 smithi070 ceph-mon[112139]: pgmap v790: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 40 KiB/s rd, 2.7 KiB/s wr, 54 op/s 2024-09-21T10:13:12.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:11 smithi070 ceph-mon[112139]: osdmap e762: 8 total, 8 up, 8 in 2024-09-21T10:13:13.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:12 smithi160 ceph-mon[101879]: osdmap e763: 8 total, 8 up, 8 in 2024-09-21T10:13:13.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:12 smithi160 ceph-mon[101879]: osdmap e764: 8 total, 8 up, 8 in 2024-09-21T10:13:13.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:12 smithi070 ceph-mon[107385]: osdmap e763: 8 total, 8 up, 8 in 2024-09-21T10:13:13.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:12 smithi070 ceph-mon[107385]: osdmap e764: 8 total, 8 up, 8 in 2024-09-21T10:13:13.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:12 smithi070 ceph-mon[112139]: osdmap e763: 8 total, 8 up, 8 in 2024-09-21T10:13:13.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:12 smithi070 ceph-mon[112139]: osdmap e764: 8 total, 8 up, 8 in 2024-09-21T10:13:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:13:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:13:13] "GET /metrics HTTP/1.1" 200 82166 "" "Prometheus/2.51.0" 2024-09-21T10:13:14.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:13 smithi160 ceph-mon[101879]: pgmap v793: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 181 KiB/s rd, 11 KiB/s wr, 235 op/s 2024-09-21T10:13:14.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:13 smithi070 ceph-mon[107385]: pgmap v793: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 181 KiB/s rd, 11 KiB/s wr, 235 op/s 2024-09-21T10:13:14.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:13 smithi070 ceph-mon[112139]: pgmap v793: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 181 KiB/s rd, 11 KiB/s wr, 235 op/s 2024-09-21T10:13:15.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:14 smithi160 ceph-mon[101879]: osdmap e765: 8 total, 8 up, 8 in 2024-09-21T10:13:15.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:14 smithi070 ceph-mon[107385]: osdmap e765: 8 total, 8 up, 8 in 2024-09-21T10:13:15.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:14 smithi070 ceph-mon[112139]: osdmap e765: 8 total, 8 up, 8 in 2024-09-21T10:13:16.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:15 smithi160 ceph-mon[101879]: pgmap v796: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 170 KiB/s rd, 12 KiB/s wr, 230 op/s 2024-09-21T10:13:16.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:15 smithi160 ceph-mon[101879]: osdmap e766: 8 total, 8 up, 8 in 2024-09-21T10:13:16.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:15 smithi070 ceph-mon[107385]: pgmap v796: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 170 KiB/s rd, 12 KiB/s wr, 230 op/s 2024-09-21T10:13:16.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:15 smithi070 ceph-mon[107385]: osdmap e766: 8 total, 8 up, 8 in 2024-09-21T10:13:16.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:15 smithi070 ceph-mon[112139]: pgmap v796: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 170 KiB/s rd, 12 KiB/s wr, 230 op/s 2024-09-21T10:13:16.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:15 smithi070 ceph-mon[112139]: osdmap e766: 8 total, 8 up, 8 in 2024-09-21T10:13:17.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:16 smithi160 ceph-mon[101879]: osdmap e767: 8 total, 8 up, 8 in 2024-09-21T10:13:17.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:16 smithi070 ceph-mon[107385]: osdmap e767: 8 total, 8 up, 8 in 2024-09-21T10:13:17.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:16 smithi070 ceph-mon[112139]: osdmap e767: 8 total, 8 up, 8 in 2024-09-21T10:13:18.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:17 smithi160 ceph-mon[101879]: pgmap v799: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 155 KiB/s rd, 10 KiB/s wr, 211 op/s 2024-09-21T10:13:18.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:17 smithi160 ceph-mon[101879]: osdmap e768: 8 total, 8 up, 8 in 2024-09-21T10:13:18.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:17 smithi160 ceph-mon[101879]: osdmap e769: 8 total, 8 up, 8 in 2024-09-21T10:13:18.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:17 smithi070 ceph-mon[107385]: pgmap v799: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 155 KiB/s rd, 10 KiB/s wr, 211 op/s 2024-09-21T10:13:18.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:17 smithi070 ceph-mon[107385]: osdmap e768: 8 total, 8 up, 8 in 2024-09-21T10:13:18.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:17 smithi070 ceph-mon[107385]: osdmap e769: 8 total, 8 up, 8 in 2024-09-21T10:13:18.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:17 smithi070 ceph-mon[112139]: pgmap v799: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 155 KiB/s rd, 10 KiB/s wr, 211 op/s 2024-09-21T10:13:18.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:17 smithi070 ceph-mon[112139]: osdmap e768: 8 total, 8 up, 8 in 2024-09-21T10:13:18.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:17 smithi070 ceph-mon[112139]: osdmap e769: 8 total, 8 up, 8 in 2024-09-21T10:13:20.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:19 smithi160 ceph-mon[101879]: pgmap v802: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 126 KiB/s rd, 7.0 KiB/s wr, 161 op/s 2024-09-21T10:13:20.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:19 smithi160 ceph-mon[101879]: osdmap e770: 8 total, 8 up, 8 in 2024-09-21T10:13:20.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:19 smithi070 ceph-mon[107385]: pgmap v802: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 126 KiB/s rd, 7.0 KiB/s wr, 161 op/s 2024-09-21T10:13:20.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:19 smithi070 ceph-mon[107385]: osdmap e770: 8 total, 8 up, 8 in 2024-09-21T10:13:20.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:19 smithi070 ceph-mon[112139]: pgmap v802: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 126 KiB/s rd, 7.0 KiB/s wr, 161 op/s 2024-09-21T10:13:20.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:19 smithi070 ceph-mon[112139]: osdmap e770: 8 total, 8 up, 8 in 2024-09-21T10:13:21.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:20 smithi160 ceph-mon[101879]: osdmap e771: 8 total, 8 up, 8 in 2024-09-21T10:13:21.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:21.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:20 smithi070 ceph-mon[112139]: osdmap e771: 8 total, 8 up, 8 in 2024-09-21T10:13:21.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:21.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:20 smithi070 ceph-mon[107385]: osdmap e771: 8 total, 8 up, 8 in 2024-09-21T10:13:21.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:21.925 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:21 smithi070 ceph-mon[107385]: pgmap v805: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-21T10:13:21.925 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:21 smithi070 ceph-mon[107385]: osdmap e772: 8 total, 8 up, 8 in 2024-09-21T10:13:21.925 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:13:21.925 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:21 smithi070 ceph-mon[112139]: pgmap v805: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-21T10:13:21.926 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:21 smithi070 ceph-mon[112139]: osdmap e772: 8 total, 8 up, 8 in 2024-09-21T10:13:21.926 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:13:22.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:21 smithi160 ceph-mon[101879]: pgmap v805: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-21T10:13:22.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:21 smithi160 ceph-mon[101879]: osdmap e772: 8 total, 8 up, 8 in 2024-09-21T10:13:22.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:13:22.406 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ObjectMapConsistentSnap (10425 ms) 2024-09-21T10:13:22.406 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.Metadata 2024-09-21T10:13:22.406 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:22.920 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.Metadata (515 ms) 2024-09-21T10:13:22.920 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.MetadataPP 2024-09-21T10:13:22.920 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:23.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:22 smithi160 ceph-mon[101879]: osdmap e773: 8 total, 8 up, 8 in 2024-09-21T10:13:23.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:22 smithi070 ceph-mon[107385]: osdmap e773: 8 total, 8 up, 8 in 2024-09-21T10:13:23.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:22 smithi070 ceph-mon[112139]: osdmap e773: 8 total, 8 up, 8 in 2024-09-21T10:13:23.395 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:13:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:13:23] "GET /metrics HTTP/1.1" 200 82162 "" "Prometheus/2.51.0" 2024-09-21T10:13:23.923 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.MetadataPP (1002 ms) 2024-09-21T10:13:23.923 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.UpdateFeatures 2024-09-21T10:13:23.923 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:24.018 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.UpdateFeatures (95 ms) 2024-09-21T10:13:24.018 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.FeaturesBitmaskString 2024-09-21T10:13:24.018 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.FeaturesBitmaskString (1 ms) 2024-09-21T10:13:24.018 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.RebuildObjectMap 2024-09-21T10:13:24.018 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:24.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:23 smithi160 ceph-mon[101879]: pgmap v808: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 74 KiB/s rd, 3.7 KiB/s wr, 98 op/s 2024-09-21T10:13:24.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:23 smithi160 ceph-mon[101879]: osdmap e774: 8 total, 8 up, 8 in 2024-09-21T10:13:24.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:23 smithi070 ceph-mon[112139]: pgmap v808: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 74 KiB/s rd, 3.7 KiB/s wr, 98 op/s 2024-09-21T10:13:24.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:23 smithi070 ceph-mon[112139]: osdmap e774: 8 total, 8 up, 8 in 2024-09-21T10:13:24.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:23 smithi070 ceph-mon[107385]: pgmap v808: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 74 KiB/s rd, 3.7 KiB/s wr, 98 op/s 2024-09-21T10:13:24.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:23 smithi070 ceph-mon[107385]: osdmap e774: 8 total, 8 up, 8 in 2024-09-21T10:13:24.889 INFO:tasks.workunit.client.0.smithi070.stdout:0.00% done 2024-09-21T10:13:24.889 INFO:tasks.workunit.client.0.smithi070.stdout:25.00% done 2024-09-21T10:13:24.889 INFO:tasks.workunit.client.0.smithi070.stdout:50.00% done 2024-09-21T10:13:24.889 INFO:tasks.workunit.client.0.smithi070.stdout:75.00% done 2024-09-21T10:13:24.889 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.RebuildObjectMap (870 ms) 2024-09-21T10:13:24.889 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.RebuildNewObjectMap 2024-09-21T10:13:24.927 INFO:tasks.workunit.client.0.smithi070.stdout:0.00% done 2024-09-21T10:13:24.927 INFO:tasks.workunit.client.0.smithi070.stdout:25.00% done 2024-09-21T10:13:24.927 INFO:tasks.workunit.client.0.smithi070.stdout:50.00% done 2024-09-21T10:13:24.927 INFO:tasks.workunit.client.0.smithi070.stdout:75.00% done 2024-09-21T10:13:24.927 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.RebuildNewObjectMap (37 ms) 2024-09-21T10:13:24.927 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.CheckObjectMap 2024-09-21T10:13:24.927 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:25.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:24 smithi160 ceph-mon[101879]: osdmap e775: 8 total, 8 up, 8 in 2024-09-21T10:13:25.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:24 smithi070 ceph-mon[107385]: osdmap e775: 8 total, 8 up, 8 in 2024-09-21T10:13:25.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:24 smithi070 ceph-mon[112139]: osdmap e775: 8 total, 8 up, 8 in 2024-09-21T10:13:25.939 INFO:tasks.workunit.client.0.smithi070.stdout:0.00% done 2024-09-21T10:13:25.940 INFO:tasks.workunit.client.0.smithi070.stdout:25.00% done 2024-09-21T10:13:25.940 INFO:tasks.workunit.client.0.smithi070.stdout:50.00% done 2024-09-21T10:13:25.940 INFO:tasks.workunit.client.0.smithi070.stdout:75.00% done 2024-09-21T10:13:25.940 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.CheckObjectMap (1012 ms) 2024-09-21T10:13:25.940 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.BlockingAIO 2024-09-21T10:13:25.940 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:25.967 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.BlockingAIO (27 ms) 2024-09-21T10:13:25.967 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ExclusiveLockTransition 2024-09-21T10:13:25.967 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:26.087 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ExclusiveLockTransition (121 ms) 2024-09-21T10:13:26.088 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ExclusiveLockReadTransition 2024-09-21T10:13:26.088 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/test_librbd.cc:10793: Skipped 2024-09-21T10:13:26.088 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T10:13:26.088 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLockReadTransition (0 ms) 2024-09-21T10:13:26.088 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.CacheMayCopyOnWrite 2024-09-21T10:13:26.088 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:26.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:25 smithi160 ceph-mon[101879]: pgmap v811: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 97 KiB/s rd, 22 KiB/s wr, 145 op/s 2024-09-21T10:13:26.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:25 smithi160 ceph-mon[101879]: osdmap e776: 8 total, 8 up, 8 in 2024-09-21T10:13:26.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:25 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:13:26.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:25 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:13:26.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:25 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:13:26.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:25 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:13:26.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:25 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:13:26.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:25 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:13:26.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:25 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:13:26.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:25 smithi070 ceph-mon[107385]: pgmap v811: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 97 KiB/s rd, 22 KiB/s wr, 145 op/s 2024-09-21T10:13:26.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:25 smithi070 ceph-mon[107385]: osdmap e776: 8 total, 8 up, 8 in 2024-09-21T10:13:26.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:25 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:13:26.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:25 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:13:26.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:25 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:13:26.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:25 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:13:26.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:25 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:13:26.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:25 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:13:26.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:25 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:13:26.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:25 smithi070 ceph-mon[112139]: pgmap v811: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 97 KiB/s rd, 22 KiB/s wr, 145 op/s 2024-09-21T10:13:26.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:25 smithi070 ceph-mon[112139]: osdmap e776: 8 total, 8 up, 8 in 2024-09-21T10:13:26.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:25 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:13:26.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:25 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:13:26.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:25 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:13:26.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:25 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:13:26.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:25 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:13:26.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:25 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:13:26.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:25 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:13:26.957 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.CacheMayCopyOnWrite (869 ms) 2024-09-21T10:13:26.957 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.FlushEmptyOpsOnExternalSnapshot 2024-09-21T10:13:26.957 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:27.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:26 smithi160 ceph-mon[101879]: osdmap e777: 8 total, 8 up, 8 in 2024-09-21T10:13:27.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:26 smithi070 ceph-mon[112139]: osdmap e777: 8 total, 8 up, 8 in 2024-09-21T10:13:27.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:26 smithi070 ceph-mon[107385]: osdmap e777: 8 total, 8 up, 8 in 2024-09-21T10:13:27.924 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.FlushEmptyOpsOnExternalSnapshot (968 ms) 2024-09-21T10:13:27.925 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestImageOptions 2024-09-21T10:13:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:27 smithi160 ceph-mon[101879]: pgmap v814: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 600 KiB/s rd, 248 KiB/s wr, 959 op/s 2024-09-21T10:13:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:27 smithi160 ceph-mon[101879]: osdmap e778: 8 total, 8 up, 8 in 2024-09-21T10:13:28.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:27 smithi070 ceph-mon[112139]: pgmap v814: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 600 KiB/s rd, 248 KiB/s wr, 959 op/s 2024-09-21T10:13:28.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:27 smithi070 ceph-mon[112139]: osdmap e778: 8 total, 8 up, 8 in 2024-09-21T10:13:28.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:27 smithi070 ceph-mon[107385]: pgmap v814: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 600 KiB/s rd, 248 KiB/s wr, 959 op/s 2024-09-21T10:13:28.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:27 smithi070 ceph-mon[107385]: osdmap e778: 8 total, 8 up, 8 in 2024-09-21T10:13:29.020 INFO:tasks.workunit.client.0.smithi070.stdout:0.00% done 2024-09-21T10:13:29.020 INFO:tasks.workunit.client.0.smithi070.stdout:100.00% done 2024-09-21T10:13:29.021 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestImageOptions (1096 ms) 2024-09-21T10:13:29.021 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestImageOptionsPP 2024-09-21T10:13:29.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:28 smithi160 ceph-mon[101879]: osdmap e779: 8 total, 8 up, 8 in 2024-09-21T10:13:29.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:28 smithi070 ceph-mon[107385]: osdmap e779: 8 total, 8 up, 8 in 2024-09-21T10:13:29.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:28 smithi070 ceph-mon[112139]: osdmap e779: 8 total, 8 up, 8 in 2024-09-21T10:13:30.004 INFO:tasks.workunit.client.0.smithi070.stdout:0.00% done 2024-09-21T10:13:30.045 INFO:tasks.workunit.client.0.smithi070.stdout:100.00% done 2024-09-21T10:13:30.045 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestImageOptionsPP (979 ms) 2024-09-21T10:13:30.045 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.EventSocketPipe 2024-09-21T10:13:30.046 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.EventSocketPipe (0 ms) 2024-09-21T10:13:30.046 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.EventSocketEventfd 2024-09-21T10:13:30.046 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.EventSocketEventfd (0 ms) 2024-09-21T10:13:30.046 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ImagePollIO 2024-09-21T10:13:30.046 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:30.047 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:13:30.047 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:13:30.047 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:13:30.047 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:13:30.047 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-21T10:13:30.047 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:13:30.047 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:13:30.047 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:13:30.047 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:13:30.048 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:13:30.048 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:13:30.048 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:13:30.048 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:13:30.048 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:13:30.048 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:13:30.048 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:13:30.048 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:13:30.048 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:13:30.048 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:13:30.048 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:13:30.049 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:13:30.049 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:13:30.049 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:13:30.049 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:13:30.049 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:13:30.049 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:13:30.049 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-21T10:13:30.049 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-21T10:13:30.049 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-21T10:13:30.049 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-21T10:13:30.049 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:13:30.049 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:13:30.050 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:13:30.050 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:13:30.050 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:13:30.050 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:13:30.050 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:13:30.050 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:13:30.050 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:13:30.050 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:13:30.050 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:13:30.050 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:13:30.050 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:13:30.051 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:13:30.051 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:13:30.051 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:13:30.051 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-21T10:13:30.051 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-21T10:13:30.051 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-21T10:13:30.051 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-21T10:13:30.051 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ImagePollIO (35 ms) 2024-09-21T10:13:30.051 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.Mirror 2024-09-21T10:13:30.051 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:30.052 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:30.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:29 smithi160 ceph-mon[101879]: pgmap v817: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 575 KiB/s rd, 229 KiB/s wr, 910 op/s 2024-09-21T10:13:30.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:29 smithi160 ceph-mon[101879]: osdmap e780: 8 total, 8 up, 8 in 2024-09-21T10:13:30.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:29 smithi070 ceph-mon[112139]: pgmap v817: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 575 KiB/s rd, 229 KiB/s wr, 910 op/s 2024-09-21T10:13:30.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:29 smithi070 ceph-mon[112139]: osdmap e780: 8 total, 8 up, 8 in 2024-09-21T10:13:30.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:29 smithi070 ceph-mon[107385]: pgmap v817: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 575 KiB/s rd, 229 KiB/s wr, 910 op/s 2024-09-21T10:13:30.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:29 smithi070 ceph-mon[107385]: osdmap e780: 8 total, 8 up, 8 in 2024-09-21T10:13:31.051 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.Mirror (1016 ms) 2024-09-21T10:13:31.051 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.MirrorPeerAttributes 2024-09-21T10:13:31.065 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.MirrorPeerAttributes (15 ms) 2024-09-21T10:13:31.065 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.CreateWithMirrorEnabled 2024-09-21T10:13:31.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:30 smithi160 ceph-mon[101879]: osdmap e781: 8 total, 8 up, 8 in 2024-09-21T10:13:31.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:30 smithi070 ceph-mon[107385]: osdmap e781: 8 total, 8 up, 8 in 2024-09-21T10:13:31.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:30 smithi070 ceph-mon[112139]: osdmap e781: 8 total, 8 up, 8 in 2024-09-21T10:13:32.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:31 smithi160 ceph-mon[101879]: pgmap v820: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 291 KiB/s rd, 79 KiB/s wr, 383 op/s 2024-09-21T10:13:32.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:31 smithi160 ceph-mon[101879]: osdmap e782: 8 total, 8 up, 8 in 2024-09-21T10:13:32.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:31 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/uuid4"} : dispatch 2024-09-21T10:13:32.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:31 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/uuid4"}]': finished 2024-09-21T10:13:32.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:31 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/ca9e0996-84cf-4ed3-ae5a-718a68e506a7"} : dispatch 2024-09-21T10:13:32.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:31 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/ca9e0996-84cf-4ed3-ae5a-718a68e506a7"}]': finished 2024-09-21T10:13:32.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:31 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/a9e2f58d-a46f-4ba5-9e61-b96e92992d09"} : dispatch 2024-09-21T10:13:32.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:31 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/a9e2f58d-a46f-4ba5-9e61-b96e92992d09"}]': finished 2024-09-21T10:13:32.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:31 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/b5a4acb2-a0f4-4f00-8d8f-bf6bfd57ac4b"} : dispatch 2024-09-21T10:13:32.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:31 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/b5a4acb2-a0f4-4f00-8d8f-bf6bfd57ac4b"}]': finished 2024-09-21T10:13:32.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:31 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/peer/19/178d8fb9-292a-4611-974a-0bc302e72606"} : dispatch 2024-09-21T10:13:32.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:31 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' 2024-09-21T10:13:32.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:31 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/peer/19/178d8fb9-292a-4611-974a-0bc302e72606"} : dispatch 2024-09-21T10:13:32.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:31 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/178d8fb9-292a-4611-974a-0bc302e72606"} : dispatch 2024-09-21T10:13:32.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:31 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/178d8fb9-292a-4611-974a-0bc302e72606"}]': finished 2024-09-21T10:13:32.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[107385]: pgmap v820: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 291 KiB/s rd, 79 KiB/s wr, 383 op/s 2024-09-21T10:13:32.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[107385]: osdmap e782: 8 total, 8 up, 8 in 2024-09-21T10:13:32.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/uuid4"} : dispatch 2024-09-21T10:13:32.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/uuid4"}]': finished 2024-09-21T10:13:32.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/ca9e0996-84cf-4ed3-ae5a-718a68e506a7"} : dispatch 2024-09-21T10:13:32.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/ca9e0996-84cf-4ed3-ae5a-718a68e506a7"}]': finished 2024-09-21T10:13:32.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/a9e2f58d-a46f-4ba5-9e61-b96e92992d09"} : dispatch 2024-09-21T10:13:32.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/a9e2f58d-a46f-4ba5-9e61-b96e92992d09"}]': finished 2024-09-21T10:13:32.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/b5a4acb2-a0f4-4f00-8d8f-bf6bfd57ac4b"} : dispatch 2024-09-21T10:13:32.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/b5a4acb2-a0f4-4f00-8d8f-bf6bfd57ac4b"}]': finished 2024-09-21T10:13:32.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/peer/19/178d8fb9-292a-4611-974a-0bc302e72606"} : dispatch 2024-09-21T10:13:32.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' 2024-09-21T10:13:32.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/peer/19/178d8fb9-292a-4611-974a-0bc302e72606"} : dispatch 2024-09-21T10:13:32.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/178d8fb9-292a-4611-974a-0bc302e72606"} : dispatch 2024-09-21T10:13:32.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/178d8fb9-292a-4611-974a-0bc302e72606"}]': finished 2024-09-21T10:13:32.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[112139]: pgmap v820: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 291 KiB/s rd, 79 KiB/s wr, 383 op/s 2024-09-21T10:13:32.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[112139]: osdmap e782: 8 total, 8 up, 8 in 2024-09-21T10:13:32.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/uuid4"} : dispatch 2024-09-21T10:13:32.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/uuid4"}]': finished 2024-09-21T10:13:32.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/ca9e0996-84cf-4ed3-ae5a-718a68e506a7"} : dispatch 2024-09-21T10:13:32.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/ca9e0996-84cf-4ed3-ae5a-718a68e506a7"}]': finished 2024-09-21T10:13:32.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/a9e2f58d-a46f-4ba5-9e61-b96e92992d09"} : dispatch 2024-09-21T10:13:32.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/a9e2f58d-a46f-4ba5-9e61-b96e92992d09"}]': finished 2024-09-21T10:13:32.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/b5a4acb2-a0f4-4f00-8d8f-bf6bfd57ac4b"} : dispatch 2024-09-21T10:13:32.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/b5a4acb2-a0f4-4f00-8d8f-bf6bfd57ac4b"}]': finished 2024-09-21T10:13:32.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/peer/19/178d8fb9-292a-4611-974a-0bc302e72606"} : dispatch 2024-09-21T10:13:32.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' 2024-09-21T10:13:32.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/peer/19/178d8fb9-292a-4611-974a-0bc302e72606"} : dispatch 2024-09-21T10:13:32.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/178d8fb9-292a-4611-974a-0bc302e72606"} : dispatch 2024-09-21T10:13:32.381 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:31 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1197859384' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/178d8fb9-292a-4611-974a-0bc302e72606"}]': finished 2024-09-21T10:13:33.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:33 smithi160 ceph-mon[101879]: osdmap e783: 8 total, 8 up, 8 in 2024-09-21T10:13:33.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:33 smithi070 ceph-mon[107385]: osdmap e783: 8 total, 8 up, 8 in 2024-09-21T10:13:33.378 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:13:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:13:33] "GET /metrics HTTP/1.1" 200 82162 "" "Prometheus/2.51.0" 2024-09-21T10:13:33.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:33 smithi070 ceph-mon[112139]: osdmap e783: 8 total, 8 up, 8 in 2024-09-21T10:13:34.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:34 smithi160 ceph-mon[101879]: pgmap v823: 713 pgs: 713 active+clean; 311 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 433 KiB/s rd, 1.6 MiB/s wr, 554 op/s 2024-09-21T10:13:34.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:34 smithi160 ceph-mon[101879]: osdmap e784: 8 total, 8 up, 8 in 2024-09-21T10:13:34.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:34 smithi070 ceph-mon[112139]: pgmap v823: 713 pgs: 713 active+clean; 311 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 433 KiB/s rd, 1.6 MiB/s wr, 554 op/s 2024-09-21T10:13:34.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:34 smithi070 ceph-mon[112139]: osdmap e784: 8 total, 8 up, 8 in 2024-09-21T10:13:34.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:34 smithi070 ceph-mon[107385]: pgmap v823: 713 pgs: 713 active+clean; 311 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 433 KiB/s rd, 1.6 MiB/s wr, 554 op/s 2024-09-21T10:13:34.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:34 smithi070 ceph-mon[107385]: osdmap e784: 8 total, 8 up, 8 in 2024-09-21T10:13:35.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:35 smithi160 ceph-mon[101879]: osdmap e785: 8 total, 8 up, 8 in 2024-09-21T10:13:35.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:35 smithi070 ceph-mon[107385]: osdmap e785: 8 total, 8 up, 8 in 2024-09-21T10:13:35.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:35 smithi070 ceph-mon[112139]: osdmap e785: 8 total, 8 up, 8 in 2024-09-21T10:13:36.129 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.CreateWithMirrorEnabled (5064 ms) 2024-09-21T10:13:36.129 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot 2024-09-21T10:13:36.129 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:36.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:36 smithi070 ceph-mon[107385]: pgmap v826: 713 pgs: 713 active+clean; 313 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 226 KiB/s rd, 2.0 MiB/s wr, 287 op/s 2024-09-21T10:13:36.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:36 smithi070 ceph-mon[107385]: osdmap e786: 8 total, 8 up, 8 in 2024-09-21T10:13:36.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:36.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:36 smithi070 ceph-mon[112139]: pgmap v826: 713 pgs: 713 active+clean; 313 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 226 KiB/s rd, 2.0 MiB/s wr, 287 op/s 2024-09-21T10:13:36.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:36 smithi070 ceph-mon[112139]: osdmap e786: 8 total, 8 up, 8 in 2024-09-21T10:13:36.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:36 smithi160 ceph-mon[101879]: pgmap v826: 713 pgs: 713 active+clean; 313 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 226 KiB/s rd, 2.0 MiB/s wr, 287 op/s 2024-09-21T10:13:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:36 smithi160 ceph-mon[101879]: osdmap e786: 8 total, 8 up, 8 in 2024-09-21T10:13:36.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:37.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:37 smithi070 ceph-mon[107385]: osdmap e787: 8 total, 8 up, 8 in 2024-09-21T10:13:37.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:37 smithi070 ceph-mon[112139]: osdmap e787: 8 total, 8 up, 8 in 2024-09-21T10:13:37.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:37 smithi160 ceph-mon[101879]: osdmap e787: 8 total, 8 up, 8 in 2024-09-21T10:13:38.146 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot (2017 ms) 2024-09-21T10:13:38.146 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ExclusiveLock 2024-09-21T10:13:38.146 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:38.197 INFO:tasks.workunit.client.0.smithi070.stdout:1: acquiring exclusive lock 2024-09-21T10:13:38.197 INFO:tasks.workunit.client.0.smithi070.stdout:2: acquiring exclusive lock 2024-09-21T10:13:38.200 INFO:tasks.workunit.client.0.smithi070.stdout:2: exclusive lock acquired 2024-09-21T10:13:38.236 INFO:tasks.workunit.client.0.smithi070.stdout:2: releasing exclusive lock 2024-09-21T10:13:38.238 INFO:tasks.workunit.client.0.smithi070.stdout:2: exclusive lock released 2024-09-21T10:13:38.238 INFO:tasks.workunit.client.0.smithi070.stdout:2: acquiring exclusive lock 2024-09-21T10:13:38.241 INFO:tasks.workunit.client.0.smithi070.stdout:1: exclusive lock acquired 2024-09-21T10:13:38.258 INFO:tasks.workunit.client.0.smithi070.stdout:1: releasing exclusive lock 2024-09-21T10:13:38.261 INFO:tasks.workunit.client.0.smithi070.stdout:1: exclusive lock released 2024-09-21T10:13:38.261 INFO:tasks.workunit.client.0.smithi070.stdout:1: acquiring exclusive lock 2024-09-21T10:13:38.265 INFO:tasks.workunit.client.0.smithi070.stdout:2: exclusive lock acquired 2024-09-21T10:13:38.300 INFO:tasks.workunit.client.0.smithi070.stdout:2: releasing exclusive lock 2024-09-21T10:13:38.303 INFO:tasks.workunit.client.0.smithi070.stdout:2: exclusive lock released 2024-09-21T10:13:38.303 INFO:tasks.workunit.client.0.smithi070.stdout:2: acquiring exclusive lock 2024-09-21T10:13:38.307 INFO:tasks.workunit.client.0.smithi070.stdout:1: exclusive lock acquired 2024-09-21T10:13:38.322 INFO:tasks.workunit.client.0.smithi070.stdout:1: releasing exclusive lock 2024-09-21T10:13:38.325 INFO:tasks.workunit.client.0.smithi070.stdout:1: exclusive lock released 2024-09-21T10:13:38.325 INFO:tasks.workunit.client.0.smithi070.stdout:1: acquiring exclusive lock 2024-09-21T10:13:38.329 INFO:tasks.workunit.client.0.smithi070.stdout:1: exclusive lock acquired 2024-09-21T10:13:38.363 INFO:tasks.workunit.client.0.smithi070.stdout:1: releasing exclusive lock 2024-09-21T10:13:38.366 INFO:tasks.workunit.client.0.smithi070.stdout:1: exclusive lock released 2024-09-21T10:13:38.366 INFO:tasks.workunit.client.0.smithi070.stdout:1: acquiring exclusive lock 2024-09-21T10:13:38.369 INFO:tasks.workunit.client.0.smithi070.stdout:2: exclusive lock acquired 2024-09-21T10:13:38.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:38 smithi070 ceph-mon[107385]: pgmap v829: 713 pgs: 713 active+clean; 313 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 241 KiB/s rd, 533 KiB/s wr, 319 op/s 2024-09-21T10:13:38.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:38 smithi070 ceph-mon[107385]: osdmap e788: 8 total, 8 up, 8 in 2024-09-21T10:13:38.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:38 smithi070 ceph-mon[112139]: pgmap v829: 713 pgs: 713 active+clean; 313 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 241 KiB/s rd, 533 KiB/s wr, 319 op/s 2024-09-21T10:13:38.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:38 smithi070 ceph-mon[112139]: osdmap e788: 8 total, 8 up, 8 in 2024-09-21T10:13:38.386 INFO:tasks.workunit.client.0.smithi070.stdout:2: releasing exclusive lock 2024-09-21T10:13:38.389 INFO:tasks.workunit.client.0.smithi070.stdout:2: exclusive lock released 2024-09-21T10:13:38.389 INFO:tasks.workunit.client.0.smithi070.stdout:2: acquiring exclusive lock 2024-09-21T10:13:38.393 INFO:tasks.workunit.client.0.smithi070.stdout:2: exclusive lock acquired 2024-09-21T10:13:38.425 INFO:tasks.workunit.client.0.smithi070.stdout:2: releasing exclusive lock 2024-09-21T10:13:38.428 INFO:tasks.workunit.client.0.smithi070.stdout:2: exclusive lock released 2024-09-21T10:13:38.428 INFO:tasks.workunit.client.0.smithi070.stdout:2: acquiring exclusive lock 2024-09-21T10:13:38.431 INFO:tasks.workunit.client.0.smithi070.stdout:1: exclusive lock acquired 2024-09-21T10:13:38.432 INFO:tasks.workunit.client.0.smithi070.stdout:1: releasing exclusive lock 2024-09-21T10:13:38.435 INFO:tasks.workunit.client.0.smithi070.stdout:1: exclusive lock released 2024-09-21T10:13:38.435 INFO:tasks.workunit.client.0.smithi070.stdout:1: acquiring exclusive lock 2024-09-21T10:13:38.439 INFO:tasks.workunit.client.0.smithi070.stdout:2: exclusive lock acquired 2024-09-21T10:13:38.461 INFO:tasks.workunit.client.0.smithi070.stdout:2: releasing exclusive lock 2024-09-21T10:13:38.464 INFO:tasks.workunit.client.0.smithi070.stdout:2: exclusive lock released 2024-09-21T10:13:38.468 INFO:tasks.workunit.client.0.smithi070.stdout:1: exclusive lock acquired 2024-09-21T10:13:38.481 INFO:tasks.workunit.client.0.smithi070.stdout:1: releasing exclusive lock 2024-09-21T10:13:38.484 INFO:tasks.workunit.client.0.smithi070.stdout:1: exclusive lock released 2024-09-21T10:13:38.497 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ExclusiveLock (351 ms) 2024-09-21T10:13:38.497 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.BreakLock 2024-09-21T10:13:38.512 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:38 smithi160 ceph-mon[101879]: pgmap v829: 713 pgs: 713 active+clean; 313 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 241 KiB/s rd, 533 KiB/s wr, 319 op/s 2024-09-21T10:13:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:38 smithi160 ceph-mon[101879]: osdmap e788: 8 total, 8 up, 8 in 2024-09-21T10:13:39.157 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.BreakLock (659 ms) 2024-09-21T10:13:39.157 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.DiscardAfterWrite 2024-09-21T10:13:39.157 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:39.179 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.DiscardAfterWrite (23 ms) 2024-09-21T10:13:39.179 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.DefaultFeatures 2024-09-21T10:13:39.179 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.DefaultFeatures (0 ms) 2024-09-21T10:13:39.179 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndPurge 2024-09-21T10:13:39.179 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:39.239 INFO:tasks.workunit.client.0.smithi070.stdout:0.00% done 2024-09-21T10:13:39.239 INFO:tasks.workunit.client.0.smithi070.stdout:1.56% done 2024-09-21T10:13:39.239 INFO:tasks.workunit.client.0.smithi070.stdout:3.12% done 2024-09-21T10:13:39.239 INFO:tasks.workunit.client.0.smithi070.stdout:4.69% done 2024-09-21T10:13:39.239 INFO:tasks.workunit.client.0.smithi070.stdout:6.25% done 2024-09-21T10:13:39.240 INFO:tasks.workunit.client.0.smithi070.stdout:7.81% done 2024-09-21T10:13:39.240 INFO:tasks.workunit.client.0.smithi070.stdout:9.38% done 2024-09-21T10:13:39.240 INFO:tasks.workunit.client.0.smithi070.stdout:10.94% done 2024-09-21T10:13:39.240 INFO:tasks.workunit.client.0.smithi070.stdout:12.50% done 2024-09-21T10:13:39.240 INFO:tasks.workunit.client.0.smithi070.stdout:14.06% done 2024-09-21T10:13:39.240 INFO:tasks.workunit.client.0.smithi070.stdout:15.62% done 2024-09-21T10:13:39.240 INFO:tasks.workunit.client.0.smithi070.stdout:17.19% done 2024-09-21T10:13:39.240 INFO:tasks.workunit.client.0.smithi070.stdout:18.75% done 2024-09-21T10:13:39.240 INFO:tasks.workunit.client.0.smithi070.stdout:20.31% done 2024-09-21T10:13:39.240 INFO:tasks.workunit.client.0.smithi070.stdout:21.88% done 2024-09-21T10:13:39.240 INFO:tasks.workunit.client.0.smithi070.stdout:23.44% done 2024-09-21T10:13:39.240 INFO:tasks.workunit.client.0.smithi070.stdout:25.00% done 2024-09-21T10:13:39.240 INFO:tasks.workunit.client.0.smithi070.stdout:26.56% done 2024-09-21T10:13:39.240 INFO:tasks.workunit.client.0.smithi070.stdout:28.12% done 2024-09-21T10:13:39.240 INFO:tasks.workunit.client.0.smithi070.stdout:29.69% done 2024-09-21T10:13:39.240 INFO:tasks.workunit.client.0.smithi070.stdout:31.25% done 2024-09-21T10:13:39.241 INFO:tasks.workunit.client.0.smithi070.stdout:32.81% done 2024-09-21T10:13:39.241 INFO:tasks.workunit.client.0.smithi070.stdout:34.38% done 2024-09-21T10:13:39.241 INFO:tasks.workunit.client.0.smithi070.stdout:35.94% done 2024-09-21T10:13:39.241 INFO:tasks.workunit.client.0.smithi070.stdout:37.50% done 2024-09-21T10:13:39.241 INFO:tasks.workunit.client.0.smithi070.stdout:39.06% done 2024-09-21T10:13:39.241 INFO:tasks.workunit.client.0.smithi070.stdout:40.62% done 2024-09-21T10:13:39.241 INFO:tasks.workunit.client.0.smithi070.stdout:42.19% done 2024-09-21T10:13:39.241 INFO:tasks.workunit.client.0.smithi070.stdout:43.75% done 2024-09-21T10:13:39.241 INFO:tasks.workunit.client.0.smithi070.stdout:45.31% done 2024-09-21T10:13:39.241 INFO:tasks.workunit.client.0.smithi070.stdout:46.88% done 2024-09-21T10:13:39.241 INFO:tasks.workunit.client.0.smithi070.stdout:48.44% done 2024-09-21T10:13:39.241 INFO:tasks.workunit.client.0.smithi070.stdout:50.00% done 2024-09-21T10:13:39.241 INFO:tasks.workunit.client.0.smithi070.stdout:51.56% done 2024-09-21T10:13:39.241 INFO:tasks.workunit.client.0.smithi070.stdout:53.12% done 2024-09-21T10:13:39.241 INFO:tasks.workunit.client.0.smithi070.stdout:54.69% done 2024-09-21T10:13:39.241 INFO:tasks.workunit.client.0.smithi070.stdout:56.25% done 2024-09-21T10:13:39.242 INFO:tasks.workunit.client.0.smithi070.stdout:57.81% done 2024-09-21T10:13:39.242 INFO:tasks.workunit.client.0.smithi070.stdout:59.38% done 2024-09-21T10:13:39.242 INFO:tasks.workunit.client.0.smithi070.stdout:60.94% done 2024-09-21T10:13:39.242 INFO:tasks.workunit.client.0.smithi070.stdout:62.50% done 2024-09-21T10:13:39.242 INFO:tasks.workunit.client.0.smithi070.stdout:64.06% done 2024-09-21T10:13:39.242 INFO:tasks.workunit.client.0.smithi070.stdout:65.62% done 2024-09-21T10:13:39.242 INFO:tasks.workunit.client.0.smithi070.stdout:67.19% done 2024-09-21T10:13:39.242 INFO:tasks.workunit.client.0.smithi070.stdout:68.75% done 2024-09-21T10:13:39.242 INFO:tasks.workunit.client.0.smithi070.stdout:70.31% done 2024-09-21T10:13:39.242 INFO:tasks.workunit.client.0.smithi070.stdout:71.88% done 2024-09-21T10:13:39.242 INFO:tasks.workunit.client.0.smithi070.stdout:73.44% done 2024-09-21T10:13:39.242 INFO:tasks.workunit.client.0.smithi070.stdout:75.00% done 2024-09-21T10:13:39.242 INFO:tasks.workunit.client.0.smithi070.stdout:76.56% done 2024-09-21T10:13:39.242 INFO:tasks.workunit.client.0.smithi070.stdout:78.12% done 2024-09-21T10:13:39.242 INFO:tasks.workunit.client.0.smithi070.stdout:79.69% done 2024-09-21T10:13:39.243 INFO:tasks.workunit.client.0.smithi070.stdout:81.25% done 2024-09-21T10:13:39.243 INFO:tasks.workunit.client.0.smithi070.stdout:82.81% done 2024-09-21T10:13:39.243 INFO:tasks.workunit.client.0.smithi070.stdout:84.38% done 2024-09-21T10:13:39.243 INFO:tasks.workunit.client.0.smithi070.stdout:85.94% done 2024-09-21T10:13:39.243 INFO:tasks.workunit.client.0.smithi070.stdout:87.50% done 2024-09-21T10:13:39.243 INFO:tasks.workunit.client.0.smithi070.stdout:89.06% done 2024-09-21T10:13:39.243 INFO:tasks.workunit.client.0.smithi070.stdout:90.62% done 2024-09-21T10:13:39.243 INFO:tasks.workunit.client.0.smithi070.stdout:92.19% done 2024-09-21T10:13:39.243 INFO:tasks.workunit.client.0.smithi070.stdout:93.75% done 2024-09-21T10:13:39.243 INFO:tasks.workunit.client.0.smithi070.stdout:95.31% done 2024-09-21T10:13:39.243 INFO:tasks.workunit.client.0.smithi070.stdout:96.88% done 2024-09-21T10:13:39.243 INFO:tasks.workunit.client.0.smithi070.stdout:98.44% done 2024-09-21T10:13:39.243 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestTrashMoveAndPurge (60 ms) 2024-09-21T10:13:39.243 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay 2024-09-21T10:13:39.243 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:39.293 INFO:tasks.workunit.client.0.smithi070.stdout:0.00% done 2024-09-21T10:13:39.293 INFO:tasks.workunit.client.0.smithi070.stdout:1.56% done 2024-09-21T10:13:39.293 INFO:tasks.workunit.client.0.smithi070.stdout:3.12% done 2024-09-21T10:13:39.294 INFO:tasks.workunit.client.0.smithi070.stdout:4.69% done 2024-09-21T10:13:39.294 INFO:tasks.workunit.client.0.smithi070.stdout:6.25% done 2024-09-21T10:13:39.294 INFO:tasks.workunit.client.0.smithi070.stdout:7.81% done 2024-09-21T10:13:39.294 INFO:tasks.workunit.client.0.smithi070.stdout:9.38% done 2024-09-21T10:13:39.294 INFO:tasks.workunit.client.0.smithi070.stdout:10.94% done 2024-09-21T10:13:39.294 INFO:tasks.workunit.client.0.smithi070.stdout:12.50% done 2024-09-21T10:13:39.294 INFO:tasks.workunit.client.0.smithi070.stdout:14.06% done 2024-09-21T10:13:39.294 INFO:tasks.workunit.client.0.smithi070.stdout:15.62% done 2024-09-21T10:13:39.294 INFO:tasks.workunit.client.0.smithi070.stdout:17.19% done 2024-09-21T10:13:39.294 INFO:tasks.workunit.client.0.smithi070.stdout:18.75% done 2024-09-21T10:13:39.294 INFO:tasks.workunit.client.0.smithi070.stdout:20.31% done 2024-09-21T10:13:39.294 INFO:tasks.workunit.client.0.smithi070.stdout:21.88% done 2024-09-21T10:13:39.294 INFO:tasks.workunit.client.0.smithi070.stdout:23.44% done 2024-09-21T10:13:39.294 INFO:tasks.workunit.client.0.smithi070.stdout:25.00% done 2024-09-21T10:13:39.294 INFO:tasks.workunit.client.0.smithi070.stdout:26.56% done 2024-09-21T10:13:39.295 INFO:tasks.workunit.client.0.smithi070.stdout:28.12% done 2024-09-21T10:13:39.295 INFO:tasks.workunit.client.0.smithi070.stdout:29.69% done 2024-09-21T10:13:39.295 INFO:tasks.workunit.client.0.smithi070.stdout:31.25% done 2024-09-21T10:13:39.295 INFO:tasks.workunit.client.0.smithi070.stdout:32.81% done 2024-09-21T10:13:39.295 INFO:tasks.workunit.client.0.smithi070.stdout:34.38% done 2024-09-21T10:13:39.295 INFO:tasks.workunit.client.0.smithi070.stdout:35.94% done 2024-09-21T10:13:39.295 INFO:tasks.workunit.client.0.smithi070.stdout:37.50% done 2024-09-21T10:13:39.295 INFO:tasks.workunit.client.0.smithi070.stdout:39.06% done 2024-09-21T10:13:39.295 INFO:tasks.workunit.client.0.smithi070.stdout:40.62% done 2024-09-21T10:13:39.295 INFO:tasks.workunit.client.0.smithi070.stdout:42.19% done 2024-09-21T10:13:39.295 INFO:tasks.workunit.client.0.smithi070.stdout:43.75% done 2024-09-21T10:13:39.295 INFO:tasks.workunit.client.0.smithi070.stdout:45.31% done 2024-09-21T10:13:39.295 INFO:tasks.workunit.client.0.smithi070.stdout:46.88% done 2024-09-21T10:13:39.295 INFO:tasks.workunit.client.0.smithi070.stdout:48.44% done 2024-09-21T10:13:39.295 INFO:tasks.workunit.client.0.smithi070.stdout:50.00% done 2024-09-21T10:13:39.296 INFO:tasks.workunit.client.0.smithi070.stdout:51.56% done 2024-09-21T10:13:39.296 INFO:tasks.workunit.client.0.smithi070.stdout:53.12% done 2024-09-21T10:13:39.296 INFO:tasks.workunit.client.0.smithi070.stdout:54.69% done 2024-09-21T10:13:39.296 INFO:tasks.workunit.client.0.smithi070.stdout:56.25% done 2024-09-21T10:13:39.296 INFO:tasks.workunit.client.0.smithi070.stdout:57.81% done 2024-09-21T10:13:39.296 INFO:tasks.workunit.client.0.smithi070.stdout:59.38% done 2024-09-21T10:13:39.296 INFO:tasks.workunit.client.0.smithi070.stdout:60.94% done 2024-09-21T10:13:39.296 INFO:tasks.workunit.client.0.smithi070.stdout:62.50% done 2024-09-21T10:13:39.296 INFO:tasks.workunit.client.0.smithi070.stdout:64.06% done 2024-09-21T10:13:39.296 INFO:tasks.workunit.client.0.smithi070.stdout:65.62% done 2024-09-21T10:13:39.296 INFO:tasks.workunit.client.0.smithi070.stdout:67.19% done 2024-09-21T10:13:39.296 INFO:tasks.workunit.client.0.smithi070.stdout:68.75% done 2024-09-21T10:13:39.296 INFO:tasks.workunit.client.0.smithi070.stdout:70.31% done 2024-09-21T10:13:39.296 INFO:tasks.workunit.client.0.smithi070.stdout:71.88% done 2024-09-21T10:13:39.296 INFO:tasks.workunit.client.0.smithi070.stdout:73.44% done 2024-09-21T10:13:39.296 INFO:tasks.workunit.client.0.smithi070.stdout:75.00% done 2024-09-21T10:13:39.297 INFO:tasks.workunit.client.0.smithi070.stdout:76.56% done 2024-09-21T10:13:39.297 INFO:tasks.workunit.client.0.smithi070.stdout:78.12% done 2024-09-21T10:13:39.297 INFO:tasks.workunit.client.0.smithi070.stdout:79.69% done 2024-09-21T10:13:39.297 INFO:tasks.workunit.client.0.smithi070.stdout:81.25% done 2024-09-21T10:13:39.297 INFO:tasks.workunit.client.0.smithi070.stdout:82.81% done 2024-09-21T10:13:39.297 INFO:tasks.workunit.client.0.smithi070.stdout:84.38% done 2024-09-21T10:13:39.297 INFO:tasks.workunit.client.0.smithi070.stdout:85.94% done 2024-09-21T10:13:39.297 INFO:tasks.workunit.client.0.smithi070.stdout:87.50% done 2024-09-21T10:13:39.297 INFO:tasks.workunit.client.0.smithi070.stdout:89.06% done 2024-09-21T10:13:39.297 INFO:tasks.workunit.client.0.smithi070.stdout:90.62% done 2024-09-21T10:13:39.297 INFO:tasks.workunit.client.0.smithi070.stdout:92.19% done 2024-09-21T10:13:39.297 INFO:tasks.workunit.client.0.smithi070.stdout:93.75% done 2024-09-21T10:13:39.297 INFO:tasks.workunit.client.0.smithi070.stdout:95.31% done 2024-09-21T10:13:39.297 INFO:tasks.workunit.client.0.smithi070.stdout:96.88% done 2024-09-21T10:13:39.297 INFO:tasks.workunit.client.0.smithi070.stdout:98.44% done 2024-09-21T10:13:39.298 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay (54 ms) 2024-09-21T10:13:39.298 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestTrashPurge 2024-09-21T10:13:39.298 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:39.301 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:39.413 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestTrashPurge (120 ms) 2024-09-21T10:13:39.413 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndRestore 2024-09-21T10:13:39.413 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:39.457 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestTrashMoveAndRestore (43 ms) 2024-09-21T10:13:39.457 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestListWatchers 2024-09-21T10:13:39.457 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:39.489 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestListWatchers (33 ms) 2024-09-21T10:13:39.490 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestSetSnapById 2024-09-21T10:13:39.490 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:39.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:39 smithi160 ceph-mon[101879]: osdmap e789: 8 total, 8 up, 8 in 2024-09-21T10:13:39.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:39 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.70:0/1102418977"} : dispatch 2024-09-21T10:13:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:39 smithi070 ceph-mon[107385]: osdmap e789: 8 total, 8 up, 8 in 2024-09-21T10:13:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:39 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.70:0/1102418977"} : dispatch 2024-09-21T10:13:39.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:39 smithi070 ceph-mon[112139]: osdmap e789: 8 total, 8 up, 8 in 2024-09-21T10:13:39.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:39 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.70:0/1102418977"} : dispatch 2024-09-21T10:13:40.164 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestSetSnapById (674 ms) 2024-09-21T10:13:40.164 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.Namespaces 2024-09-21T10:13:40.184 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.Namespaces (21 ms) 2024-09-21T10:13:40.184 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.NamespacesPP 2024-09-21T10:13:40.213 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:40.218 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:40.292 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.NamespacesPP (107 ms) 2024-09-21T10:13:40.292 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.Migration 2024-09-21T10:13:40.292 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:40.292 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:40.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:40 smithi160 ceph-mon[101879]: pgmap v832: 713 pgs: 713 active+clean; 313 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 156 KiB/s rd, 13 KiB/s wr, 203 op/s 2024-09-21T10:13:40.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:40 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.70:0/1102418977"}]': finished 2024-09-21T10:13:40.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:40 smithi160 ceph-mon[101879]: osdmap e790: 8 total, 8 up, 8 in 2024-09-21T10:13:40.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:40 smithi070 ceph-mon[112139]: pgmap v832: 713 pgs: 713 active+clean; 313 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 156 KiB/s rd, 13 KiB/s wr, 203 op/s 2024-09-21T10:13:40.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:40 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.70:0/1102418977"}]': finished 2024-09-21T10:13:40.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:40 smithi070 ceph-mon[112139]: osdmap e790: 8 total, 8 up, 8 in 2024-09-21T10:13:40.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:40 smithi070 ceph-mon[107385]: pgmap v832: 713 pgs: 713 active+clean; 313 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 156 KiB/s rd, 13 KiB/s wr, 203 op/s 2024-09-21T10:13:40.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:40 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.70:0/1102418977"}]': finished 2024-09-21T10:13:40.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:40 smithi070 ceph-mon[107385]: osdmap e790: 8 total, 8 up, 8 in 2024-09-21T10:13:40.685 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.Migration (394 ms) 2024-09-21T10:13:40.685 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.MigrationPP 2024-09-21T10:13:40.685 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:40.685 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:41.066 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.MigrationPP (380 ms) 2024-09-21T10:13:41.066 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestGetAccessTimestamp 2024-09-21T10:13:41.066 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:41.084 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestGetAccessTimestamp (19 ms) 2024-09-21T10:13:41.084 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestGetModifyTimestamp 2024-09-21T10:13:41.084 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:41.101 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestGetModifyTimestamp (17 ms) 2024-09-21T10:13:41.102 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ZeroOverlapFlatten 2024-09-21T10:13:41.102 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:41.222 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ZeroOverlapFlatten (121 ms) 2024-09-21T10:13:41.223 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.PoolMetadata 2024-09-21T10:13:41.248 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.PoolMetadata (26 ms) 2024-09-21T10:13:41.248 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.PoolMetadataPP 2024-09-21T10:13:41.267 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.PoolMetadataPP (18 ms) 2024-09-21T10:13:41.267 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.Config 2024-09-21T10:13:41.271 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:41.307 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.Config (41 ms) 2024-09-21T10:13:41.307 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ConfigPP 2024-09-21T10:13:41.311 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:41.345 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ConfigPP (37 ms) 2024-09-21T10:13:41.345 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.PoolStatsPP 2024-09-21T10:13:41.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:41 smithi160 ceph-mon[101879]: osdmap e791: 8 total, 8 up, 8 in 2024-09-21T10:13:41.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:41 smithi070 ceph-mon[107385]: osdmap e791: 8 total, 8 up, 8 in 2024-09-21T10:13:41.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:13:41 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:13:41] ENGINE Bus STOPPING 2024-09-21T10:13:41.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:41 smithi070 ceph-mon[112139]: osdmap e791: 8 total, 8 up, 8 in 2024-09-21T10:13:42.128 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:13:41 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:13:41] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T10:13:42.128 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:13:41 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:13:41] ENGINE Bus STOPPED 2024-09-21T10:13:42.128 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:13:41 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:13:41] ENGINE Bus STARTING 2024-09-21T10:13:42.128 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:13:41 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:13:41] ENGINE Serving on http://:::9283 2024-09-21T10:13:42.128 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:13:41 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:10:13:41] ENGINE Bus STARTED 2024-09-21T10:13:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:42 smithi160 ceph-mon[101879]: pgmap v835: 713 pgs: 713 active+clean; 314 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 96 KiB/s rd, 67 KiB/s wr, 142 op/s 2024-09-21T10:13:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:42 smithi160 ceph-mon[101879]: osdmap e792: 8 total, 8 up, 8 in 2024-09-21T10:13:42.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:42 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' 2024-09-21T10:13:42.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:42 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:13:42.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:42 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:13:42.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:42 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:13:42.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:42 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:13:42.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:42 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:13:42.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:42 smithi070 ceph-mon[112139]: pgmap v835: 713 pgs: 713 active+clean; 314 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 96 KiB/s rd, 67 KiB/s wr, 142 op/s 2024-09-21T10:13:42.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:42 smithi070 ceph-mon[112139]: osdmap e792: 8 total, 8 up, 8 in 2024-09-21T10:13:42.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:42 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' 2024-09-21T10:13:42.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:42 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:13:42.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:42 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:13:42.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:42 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:13:42.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:42 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:13:42.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:42 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:13:42.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:42 smithi070 ceph-mon[107385]: pgmap v835: 713 pgs: 713 active+clean; 314 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 96 KiB/s rd, 67 KiB/s wr, 142 op/s 2024-09-21T10:13:42.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:42 smithi070 ceph-mon[107385]: osdmap e792: 8 total, 8 up, 8 in 2024-09-21T10:13:42.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:42 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1178607346' entity='client.admin' 2024-09-21T10:13:42.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:42 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:13:42.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:42 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:13:42.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:42 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:13:42.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:42 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:13:42.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:42 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:13:42.809 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:43.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:43 smithi160 ceph-mon[101879]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:13:43.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:43 smithi160 ceph-mon[101879]: osdmap e793: 8 total, 8 up, 8 in 2024-09-21T10:13:43.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:43 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3776666492' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-35","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:13:43.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:43 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-35","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:13:43.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:43 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:13:43.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:43 smithi160 ceph-mon[101879]: osdmap e794: 8 total, 8 up, 8 in 2024-09-21T10:13:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:13:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:13:43] "GET /metrics HTTP/1.1" 200 82159 "" "Prometheus/2.51.0" 2024-09-21T10:13:43.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:43 smithi070 ceph-mon[107385]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:13:43.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:43 smithi070 ceph-mon[107385]: osdmap e793: 8 total, 8 up, 8 in 2024-09-21T10:13:43.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:43 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3776666492' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-35","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:13:43.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:43 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-35","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:13:43.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:43 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:13:43.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:43 smithi070 ceph-mon[107385]: osdmap e794: 8 total, 8 up, 8 in 2024-09-21T10:13:43.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:43 smithi070 ceph-mon[112139]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:13:43.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:43 smithi070 ceph-mon[112139]: osdmap e793: 8 total, 8 up, 8 in 2024-09-21T10:13:43.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:43 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3776666492' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-35","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:13:43.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:43 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-35","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:13:43.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:43 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:13:43.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:43 smithi070 ceph-mon[112139]: osdmap e794: 8 total, 8 up, 8 in 2024-09-21T10:13:44.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:44 smithi160 ceph-mon[101879]: pgmap v837: 713 pgs: 713 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 551 KiB/s rd, 159 KiB/s wr, 690 op/s 2024-09-21T10:13:44.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:44 smithi160 ceph-mon[101879]: osdmap e795: 8 total, 8 up, 8 in 2024-09-21T10:13:44.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:44 smithi070 ceph-mon[107385]: pgmap v837: 713 pgs: 713 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 551 KiB/s rd, 159 KiB/s wr, 690 op/s 2024-09-21T10:13:44.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:44 smithi070 ceph-mon[107385]: osdmap e795: 8 total, 8 up, 8 in 2024-09-21T10:13:44.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:44 smithi070 ceph-mon[112139]: pgmap v837: 713 pgs: 713 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 551 KiB/s rd, 159 KiB/s wr, 690 op/s 2024-09-21T10:13:44.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:44 smithi070 ceph-mon[112139]: osdmap e795: 8 total, 8 up, 8 in 2024-09-21T10:13:44.824 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:44.833 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:44.841 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:45.870 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.PoolStatsPP (4524 ms) 2024-09-21T10:13:45.870 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ImageSpec 2024-09-21T10:13:46.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:45 smithi160 ceph-mon[101879]: pgmap v841: 745 pgs: 26 unknown, 719 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 735 KiB/s rd, 110 KiB/s wr, 909 op/s 2024-09-21T10:13:46.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:45 smithi160 ceph-mon[101879]: osdmap e796: 8 total, 8 up, 8 in 2024-09-21T10:13:46.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:45 smithi070 ceph-mon[112139]: pgmap v841: 745 pgs: 26 unknown, 719 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 735 KiB/s rd, 110 KiB/s wr, 909 op/s 2024-09-21T10:13:46.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:45 smithi070 ceph-mon[112139]: osdmap e796: 8 total, 8 up, 8 in 2024-09-21T10:13:46.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:45 smithi070 ceph-mon[107385]: pgmap v841: 745 pgs: 26 unknown, 719 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 735 KiB/s rd, 110 KiB/s wr, 909 op/s 2024-09-21T10:13:46.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:45 smithi070 ceph-mon[107385]: osdmap e796: 8 total, 8 up, 8 in 2024-09-21T10:13:47.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:46 smithi160 ceph-mon[101879]: osdmap e797: 8 total, 8 up, 8 in 2024-09-21T10:13:47.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:46 smithi070 ceph-mon[107385]: osdmap e797: 8 total, 8 up, 8 in 2024-09-21T10:13:47.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:46 smithi070 ceph-mon[112139]: osdmap e797: 8 total, 8 up, 8 in 2024-09-21T10:13:47.803 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:47 smithi160 ceph-mon[101879]: pgmap v844: 745 pgs: 745 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 507 KiB/s rd, 32 KiB/s wr, 652 op/s 2024-09-21T10:13:48.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:47 smithi160 ceph-mon[101879]: osdmap e798: 8 total, 8 up, 8 in 2024-09-21T10:13:48.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:47 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1059510681' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-36","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:13:48.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:47 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-36","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:13:48.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:47 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:13:48.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:47 smithi160 ceph-mon[101879]: osdmap e799: 8 total, 8 up, 8 in 2024-09-21T10:13:48.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:47 smithi070 ceph-mon[107385]: pgmap v844: 745 pgs: 745 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 507 KiB/s rd, 32 KiB/s wr, 652 op/s 2024-09-21T10:13:48.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:47 smithi070 ceph-mon[107385]: osdmap e798: 8 total, 8 up, 8 in 2024-09-21T10:13:48.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:47 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1059510681' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-36","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:13:48.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:47 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-36","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:13:48.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:47 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:13:48.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:47 smithi070 ceph-mon[107385]: osdmap e799: 8 total, 8 up, 8 in 2024-09-21T10:13:48.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:47 smithi070 ceph-mon[112139]: pgmap v844: 745 pgs: 745 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 507 KiB/s rd, 32 KiB/s wr, 652 op/s 2024-09-21T10:13:48.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:47 smithi070 ceph-mon[112139]: osdmap e798: 8 total, 8 up, 8 in 2024-09-21T10:13:48.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:47 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1059510681' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-36","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:13:48.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:47 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-36","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:13:48.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:47 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:13:48.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:47 smithi070 ceph-mon[112139]: osdmap e799: 8 total, 8 up, 8 in 2024-09-21T10:13:50.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:49 smithi160 ceph-mon[101879]: pgmap v847: 777 pgs: 32 unknown, 745 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 241 KiB/s rd, 17 KiB/s wr, 307 op/s 2024-09-21T10:13:50.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:49 smithi160 ceph-mon[101879]: osdmap e800: 8 total, 8 up, 8 in 2024-09-21T10:13:50.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:49 smithi070 ceph-mon[107385]: pgmap v847: 777 pgs: 32 unknown, 745 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 241 KiB/s rd, 17 KiB/s wr, 307 op/s 2024-09-21T10:13:50.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:49 smithi070 ceph-mon[107385]: osdmap e800: 8 total, 8 up, 8 in 2024-09-21T10:13:50.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:49 smithi070 ceph-mon[112139]: pgmap v847: 777 pgs: 32 unknown, 745 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 241 KiB/s rd, 17 KiB/s wr, 307 op/s 2024-09-21T10:13:50.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:49 smithi070 ceph-mon[112139]: osdmap e800: 8 total, 8 up, 8 in 2024-09-21T10:13:51.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:50 smithi160 ceph-mon[101879]: osdmap e801: 8 total, 8 up, 8 in 2024-09-21T10:13:51.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:51.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:50 smithi070 ceph-mon[107385]: osdmap e801: 8 total, 8 up, 8 in 2024-09-21T10:13:51.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:51.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:50 smithi070 ceph-mon[112139]: osdmap e801: 8 total, 8 up, 8 in 2024-09-21T10:13:51.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:51.962 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ImageSpec (6093 ms) 2024-09-21T10:13:51.962 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.SnapRemoveWithChildMissing 2024-09-21T10:13:52.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:51 smithi160 ceph-mon[101879]: pgmap v850: 777 pgs: 15 unknown, 762 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 3.0 KiB/s wr, 21 op/s 2024-09-21T10:13:52.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:51 smithi160 ceph-mon[101879]: osdmap e802: 8 total, 8 up, 8 in 2024-09-21T10:13:52.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:51 smithi070 ceph-mon[107385]: pgmap v850: 777 pgs: 15 unknown, 762 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 3.0 KiB/s wr, 21 op/s 2024-09-21T10:13:52.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:51 smithi070 ceph-mon[107385]: osdmap e802: 8 total, 8 up, 8 in 2024-09-21T10:13:52.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:51 smithi070 ceph-mon[112139]: pgmap v850: 777 pgs: 15 unknown, 762 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 3.0 KiB/s wr, 21 op/s 2024-09-21T10:13:52.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:51 smithi070 ceph-mon[112139]: osdmap e802: 8 total, 8 up, 8 in 2024-09-21T10:13:53.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:52 smithi160 ceph-mon[101879]: osdmap e803: 8 total, 8 up, 8 in 2024-09-21T10:13:53.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:52 smithi160 ceph-mon[101879]: osdmap e804: 8 total, 8 up, 8 in 2024-09-21T10:13:53.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:52 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1328063739' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-37","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:13:53.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:52 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-37","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:13:53.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:52 smithi070 ceph-mon[107385]: osdmap e803: 8 total, 8 up, 8 in 2024-09-21T10:13:53.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:52 smithi070 ceph-mon[107385]: osdmap e804: 8 total, 8 up, 8 in 2024-09-21T10:13:53.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:52 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1328063739' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-37","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:13:53.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:52 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-37","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:13:53.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:52 smithi070 ceph-mon[112139]: osdmap e803: 8 total, 8 up, 8 in 2024-09-21T10:13:53.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:52 smithi070 ceph-mon[112139]: osdmap e804: 8 total, 8 up, 8 in 2024-09-21T10:13:53.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:52 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1328063739' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-37","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:13:53.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:52 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-37","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:13:53.494 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:13:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:13:53] "GET /metrics HTTP/1.1" 200 86957 "" "Prometheus/2.51.0" 2024-09-21T10:13:53.801 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:13:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:53 smithi160 ceph-mon[101879]: pgmap v853: 777 pgs: 777 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 70 KiB/s rd, 5.5 KiB/s wr, 91 op/s 2024-09-21T10:13:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:53 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:13:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:53 smithi160 ceph-mon[101879]: osdmap e805: 8 total, 8 up, 8 in 2024-09-21T10:13:54.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:53 smithi070 ceph-mon[107385]: pgmap v853: 777 pgs: 777 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 70 KiB/s rd, 5.5 KiB/s wr, 91 op/s 2024-09-21T10:13:54.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:53 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:13:54.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:53 smithi070 ceph-mon[107385]: osdmap e805: 8 total, 8 up, 8 in 2024-09-21T10:13:54.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:53 smithi070 ceph-mon[112139]: pgmap v853: 777 pgs: 777 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 70 KiB/s rd, 5.5 KiB/s wr, 91 op/s 2024-09-21T10:13:54.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:53 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:13:54.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:53 smithi070 ceph-mon[112139]: osdmap e805: 8 total, 8 up, 8 in 2024-09-21T10:13:56.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:55 smithi160 ceph-mon[101879]: pgmap v856: 809 pgs: 5 creating+peering, 27 unknown, 777 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 57 KiB/s rd, 3.0 KiB/s wr, 74 op/s 2024-09-21T10:13:56.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:55 smithi160 ceph-mon[101879]: osdmap e806: 8 total, 8 up, 8 in 2024-09-21T10:13:56.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:55 smithi070 ceph-mon[112139]: pgmap v856: 809 pgs: 5 creating+peering, 27 unknown, 777 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 57 KiB/s rd, 3.0 KiB/s wr, 74 op/s 2024-09-21T10:13:56.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:55 smithi070 ceph-mon[112139]: osdmap e806: 8 total, 8 up, 8 in 2024-09-21T10:13:56.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:55 smithi070 ceph-mon[107385]: pgmap v856: 809 pgs: 5 creating+peering, 27 unknown, 777 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 57 KiB/s rd, 3.0 KiB/s wr, 74 op/s 2024-09-21T10:13:56.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:55 smithi070 ceph-mon[107385]: osdmap e806: 8 total, 8 up, 8 in 2024-09-21T10:13:57.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:56 smithi160 ceph-mon[101879]: osdmap e807: 8 total, 8 up, 8 in 2024-09-21T10:13:57.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:56 smithi070 ceph-mon[107385]: osdmap e807: 8 total, 8 up, 8 in 2024-09-21T10:13:57.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:56 smithi070 ceph-mon[112139]: osdmap e807: 8 total, 8 up, 8 in 2024-09-21T10:13:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:57 smithi160 ceph-mon[101879]: pgmap v859: 809 pgs: 5 creating+peering, 804 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 85 KiB/s rd, 5.0 KiB/s wr, 110 op/s 2024-09-21T10:13:58.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:57 smithi160 ceph-mon[101879]: osdmap e808: 8 total, 8 up, 8 in 2024-09-21T10:13:58.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:57 smithi160 ceph-mon[101879]: osdmap e809: 8 total, 8 up, 8 in 2024-09-21T10:13:58.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:57 smithi070 ceph-mon[112139]: pgmap v859: 809 pgs: 5 creating+peering, 804 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 85 KiB/s rd, 5.0 KiB/s wr, 110 op/s 2024-09-21T10:13:58.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:57 smithi070 ceph-mon[112139]: osdmap e808: 8 total, 8 up, 8 in 2024-09-21T10:13:58.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:57 smithi070 ceph-mon[112139]: osdmap e809: 8 total, 8 up, 8 in 2024-09-21T10:13:58.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:57 smithi070 ceph-mon[107385]: pgmap v859: 809 pgs: 5 creating+peering, 804 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 85 KiB/s rd, 5.0 KiB/s wr, 110 op/s 2024-09-21T10:13:58.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:57 smithi070 ceph-mon[107385]: osdmap e808: 8 total, 8 up, 8 in 2024-09-21T10:13:58.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:57 smithi070 ceph-mon[107385]: osdmap e809: 8 total, 8 up, 8 in 2024-09-21T10:14:00.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:59 smithi160 ceph-mon[101879]: pgmap v862: 809 pgs: 5 creating+peering, 804 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 83 KiB/s rd, 4.5 KiB/s wr, 106 op/s 2024-09-21T10:14:00.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:13:59 smithi160 ceph-mon[101879]: osdmap e810: 8 total, 8 up, 8 in 2024-09-21T10:14:00.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:59 smithi070 ceph-mon[112139]: pgmap v862: 809 pgs: 5 creating+peering, 804 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 83 KiB/s rd, 4.5 KiB/s wr, 106 op/s 2024-09-21T10:14:00.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:13:59 smithi070 ceph-mon[112139]: osdmap e810: 8 total, 8 up, 8 in 2024-09-21T10:14:00.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:59 smithi070 ceph-mon[107385]: pgmap v862: 809 pgs: 5 creating+peering, 804 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 83 KiB/s rd, 4.5 KiB/s wr, 106 op/s 2024-09-21T10:14:00.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:13:59 smithi070 ceph-mon[107385]: osdmap e810: 8 total, 8 up, 8 in 2024-09-21T10:14:00.876 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 3 2024-09-21T10:14:00.877 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 3 2024-09-21T10:14:00.877 INFO:tasks.workunit.client.0.smithi070.stdout:child: image230 2024-09-21T10:14:00.877 INFO:tasks.workunit.client.0.smithi070.stdout:child: image232 2024-09-21T10:14:00.877 INFO:tasks.workunit.client.0.smithi070.stdout:child: image231 2024-09-21T10:14:00.877 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-15/image230/861964091f59 2024-09-21T10:14:00.877 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:14:00.877 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-37/image231/861915852bee 2024-09-21T10:14:00.877 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:14:00.877 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-86016-15/image232/8619161a5970 2024-09-21T10:14:00.877 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:14:00.877 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 0 2024-09-21T10:14:00.877 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 0 2024-09-21T10:14:00.877 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 0 2024-09-21T10:14:00.877 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 0 2024-09-21T10:14:00.878 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.SnapRemoveWithChildMissing (8914 ms) 2024-09-21T10:14:00.878 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.QuiesceWatch 2024-09-21T10:14:01.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:00 smithi070 ceph-mon[112139]: osdmap e811: 8 total, 8 up, 8 in 2024-09-21T10:14:01.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:00 smithi070 ceph-mon[107385]: osdmap e811: 8 total, 8 up, 8 in 2024-09-21T10:14:01.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:00 smithi160 ceph-mon[101879]: osdmap e811: 8 total, 8 up, 8 in 2024-09-21T10:14:02.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:01 smithi070 ceph-mon[112139]: pgmap v865: 777 pgs: 777 active+clean; 2.6 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:14:02.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:01 smithi070 ceph-mon[112139]: osdmap e812: 8 total, 8 up, 8 in 2024-09-21T10:14:02.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:01 smithi070 ceph-mon[107385]: pgmap v865: 777 pgs: 777 active+clean; 2.6 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:14:02.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:01 smithi070 ceph-mon[107385]: osdmap e812: 8 total, 8 up, 8 in 2024-09-21T10:14:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:01 smithi160 ceph-mon[101879]: pgmap v865: 777 pgs: 777 active+clean; 2.6 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:14:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:01 smithi160 ceph-mon[101879]: osdmap e812: 8 total, 8 up, 8 in 2024-09-21T10:14:02.805 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:14:03.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:02 smithi070 ceph-mon[112139]: osdmap e813: 8 total, 8 up, 8 in 2024-09-21T10:14:03.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:02 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/323340276' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-38","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:14:03.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:02 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-38","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:14:03.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:02 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:14:03.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:02 smithi070 ceph-mon[112139]: osdmap e814: 8 total, 8 up, 8 in 2024-09-21T10:14:03.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:02 smithi070 ceph-mon[107385]: osdmap e813: 8 total, 8 up, 8 in 2024-09-21T10:14:03.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:02 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/323340276' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-38","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:14:03.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:02 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-38","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:14:03.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:02 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:14:03.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:02 smithi070 ceph-mon[107385]: osdmap e814: 8 total, 8 up, 8 in 2024-09-21T10:14:03.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:02 smithi160 ceph-mon[101879]: osdmap e813: 8 total, 8 up, 8 in 2024-09-21T10:14:03.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:02 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/323340276' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-38","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:14:03.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:02 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-38","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:14:03.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:02 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:14:03.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:02 smithi160 ceph-mon[101879]: osdmap e814: 8 total, 8 up, 8 in 2024-09-21T10:14:03.628 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:14:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:14:03] "GET /metrics HTTP/1.1" 200 86957 "" "Prometheus/2.51.0" 2024-09-21T10:14:04.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:03 smithi070 ceph-mon[112139]: pgmap v868: 809 pgs: 32 unknown, 777 active+clean; 2.6 MiB data, 793 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:14:04.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:03 smithi070 ceph-mon[112139]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:14:04.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:03 smithi070 ceph-mon[112139]: osdmap e815: 8 total, 8 up, 8 in 2024-09-21T10:14:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:03 smithi070 ceph-mon[107385]: pgmap v868: 809 pgs: 32 unknown, 777 active+clean; 2.6 MiB data, 793 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:14:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:03 smithi070 ceph-mon[107385]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:14:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:03 smithi070 ceph-mon[107385]: osdmap e815: 8 total, 8 up, 8 in 2024-09-21T10:14:04.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:03 smithi160 ceph-mon[101879]: pgmap v868: 809 pgs: 32 unknown, 777 active+clean; 2.6 MiB data, 793 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:14:04.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:03 smithi160 ceph-mon[101879]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:14:04.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:03 smithi160 ceph-mon[101879]: osdmap e815: 8 total, 8 up, 8 in 2024-09-21T10:14:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:05 smithi160 ceph-mon[101879]: pgmap v871: 809 pgs: 22 unknown, 787 active+clean; 2.6 MiB data, 532 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 1.7 KiB/s wr, 58 op/s 2024-09-21T10:14:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:05 smithi160 ceph-mon[101879]: osdmap e816: 8 total, 8 up, 8 in 2024-09-21T10:14:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:05 smithi160 ceph-mon[101879]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:14:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:06.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:05 smithi070 ceph-mon[107385]: pgmap v871: 809 pgs: 22 unknown, 787 active+clean; 2.6 MiB data, 532 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 1.7 KiB/s wr, 58 op/s 2024-09-21T10:14:06.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:05 smithi070 ceph-mon[107385]: osdmap e816: 8 total, 8 up, 8 in 2024-09-21T10:14:06.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:05 smithi070 ceph-mon[107385]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:14:06.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:05 smithi070 ceph-mon[112139]: pgmap v871: 809 pgs: 22 unknown, 787 active+clean; 2.6 MiB data, 532 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 1.7 KiB/s wr, 58 op/s 2024-09-21T10:14:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:05 smithi070 ceph-mon[112139]: osdmap e816: 8 total, 8 up, 8 in 2024-09-21T10:14:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:05 smithi070 ceph-mon[112139]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:14:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:07.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:06 smithi160 ceph-mon[101879]: osdmap e817: 8 total, 8 up, 8 in 2024-09-21T10:14:07.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:06 smithi070 ceph-mon[112139]: osdmap e817: 8 total, 8 up, 8 in 2024-09-21T10:14:07.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:06 smithi070 ceph-mon[107385]: osdmap e817: 8 total, 8 up, 8 in 2024-09-21T10:14:08.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:07 smithi160 ceph-mon[101879]: pgmap v874: 809 pgs: 809 active+clean; 2.6 MiB data, 532 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 4.2 KiB/s wr, 60 op/s 2024-09-21T10:14:08.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:07 smithi160 ceph-mon[101879]: osdmap e818: 8 total, 8 up, 8 in 2024-09-21T10:14:08.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:07 smithi160 ceph-mon[101879]: osdmap e819: 8 total, 8 up, 8 in 2024-09-21T10:14:08.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:07 smithi070 ceph-mon[112139]: pgmap v874: 809 pgs: 809 active+clean; 2.6 MiB data, 532 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 4.2 KiB/s wr, 60 op/s 2024-09-21T10:14:08.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:07 smithi070 ceph-mon[112139]: osdmap e818: 8 total, 8 up, 8 in 2024-09-21T10:14:08.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:07 smithi070 ceph-mon[112139]: osdmap e819: 8 total, 8 up, 8 in 2024-09-21T10:14:08.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:07 smithi070 ceph-mon[107385]: pgmap v874: 809 pgs: 809 active+clean; 2.6 MiB data, 532 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 4.2 KiB/s wr, 60 op/s 2024-09-21T10:14:08.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:07 smithi070 ceph-mon[107385]: osdmap e818: 8 total, 8 up, 8 in 2024-09-21T10:14:08.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:07 smithi070 ceph-mon[107385]: osdmap e819: 8 total, 8 up, 8 in 2024-09-21T10:14:10.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:09 smithi160 ceph-mon[101879]: pgmap v877: 809 pgs: 809 active+clean; 2.6 MiB data, 532 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 4.2 KiB/s wr, 60 op/s 2024-09-21T10:14:10.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:09 smithi160 ceph-mon[101879]: osdmap e820: 8 total, 8 up, 8 in 2024-09-21T10:14:10.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:09 smithi070 ceph-mon[112139]: pgmap v877: 809 pgs: 809 active+clean; 2.6 MiB data, 532 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 4.2 KiB/s wr, 60 op/s 2024-09-21T10:14:10.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:09 smithi070 ceph-mon[112139]: osdmap e820: 8 total, 8 up, 8 in 2024-09-21T10:14:10.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:09 smithi070 ceph-mon[107385]: pgmap v877: 809 pgs: 809 active+clean; 2.6 MiB data, 532 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 4.2 KiB/s wr, 60 op/s 2024-09-21T10:14:10.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:09 smithi070 ceph-mon[107385]: osdmap e820: 8 total, 8 up, 8 in 2024-09-21T10:14:10.917 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.QuiesceWatch (10041 ms) 2024-09-21T10:14:10.918 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.QuiesceWatchPP 2024-09-21T10:14:10.918 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:14:11.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:10 smithi160 ceph-mon[101879]: osdmap e821: 8 total, 8 up, 8 in 2024-09-21T10:14:11.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:10 smithi070 ceph-mon[107385]: osdmap e821: 8 total, 8 up, 8 in 2024-09-21T10:14:11.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:10 smithi070 ceph-mon[112139]: osdmap e821: 8 total, 8 up, 8 in 2024-09-21T10:14:12.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:11 smithi160 ceph-mon[101879]: pgmap v880: 809 pgs: 809 active+clean; 2.6 MiB data, 532 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-21T10:14:12.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:11 smithi160 ceph-mon[101879]: osdmap e822: 8 total, 8 up, 8 in 2024-09-21T10:14:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:11 smithi070 ceph-mon[107385]: pgmap v880: 809 pgs: 809 active+clean; 2.6 MiB data, 532 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-21T10:14:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:11 smithi070 ceph-mon[107385]: osdmap e822: 8 total, 8 up, 8 in 2024-09-21T10:14:12.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:11 smithi070 ceph-mon[112139]: pgmap v880: 809 pgs: 809 active+clean; 2.6 MiB data, 532 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-21T10:14:12.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:11 smithi070 ceph-mon[112139]: osdmap e822: 8 total, 8 up, 8 in 2024-09-21T10:14:13.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:12 smithi070 ceph-mon[107385]: osdmap e823: 8 total, 8 up, 8 in 2024-09-21T10:14:13.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:12 smithi070 ceph-mon[107385]: osdmap e824: 8 total, 8 up, 8 in 2024-09-21T10:14:13.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:12 smithi070 ceph-mon[112139]: osdmap e823: 8 total, 8 up, 8 in 2024-09-21T10:14:13.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:12 smithi070 ceph-mon[112139]: osdmap e824: 8 total, 8 up, 8 in 2024-09-21T10:14:13.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:12 smithi160 ceph-mon[101879]: osdmap e823: 8 total, 8 up, 8 in 2024-09-21T10:14:13.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:12 smithi160 ceph-mon[101879]: osdmap e824: 8 total, 8 up, 8 in 2024-09-21T10:14:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:14:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:14:13] "GET /metrics HTTP/1.1" 200 89332 "" "Prometheus/2.51.0" 2024-09-21T10:14:14.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:13 smithi070 ceph-mon[107385]: pgmap v883: 809 pgs: 809 active+clean; 2.6 MiB data, 533 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 6.5 KiB/s wr, 99 op/s 2024-09-21T10:14:14.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:13 smithi070 ceph-mon[107385]: osdmap e825: 8 total, 8 up, 8 in 2024-09-21T10:14:14.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:13 smithi070 ceph-mon[112139]: pgmap v883: 809 pgs: 809 active+clean; 2.6 MiB data, 533 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 6.5 KiB/s wr, 99 op/s 2024-09-21T10:14:14.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:13 smithi070 ceph-mon[112139]: osdmap e825: 8 total, 8 up, 8 in 2024-09-21T10:14:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:13 smithi160 ceph-mon[101879]: pgmap v883: 809 pgs: 809 active+clean; 2.6 MiB data, 533 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 6.5 KiB/s wr, 99 op/s 2024-09-21T10:14:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:13 smithi160 ceph-mon[101879]: osdmap e825: 8 total, 8 up, 8 in 2024-09-21T10:14:16.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:15 smithi160 ceph-mon[101879]: pgmap v886: 809 pgs: 809 active+clean; 2.6 MiB data, 533 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 5.7 KiB/s wr, 103 op/s 2024-09-21T10:14:16.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:15 smithi160 ceph-mon[101879]: osdmap e826: 8 total, 8 up, 8 in 2024-09-21T10:14:16.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:15 smithi070 ceph-mon[107385]: pgmap v886: 809 pgs: 809 active+clean; 2.6 MiB data, 533 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 5.7 KiB/s wr, 103 op/s 2024-09-21T10:14:16.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:15 smithi070 ceph-mon[107385]: osdmap e826: 8 total, 8 up, 8 in 2024-09-21T10:14:16.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:15 smithi070 ceph-mon[112139]: pgmap v886: 809 pgs: 809 active+clean; 2.6 MiB data, 533 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 5.7 KiB/s wr, 103 op/s 2024-09-21T10:14:16.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:15 smithi070 ceph-mon[112139]: osdmap e826: 8 total, 8 up, 8 in 2024-09-21T10:14:16.905 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.QuiesceWatchPP (5988 ms) 2024-09-21T10:14:16.905 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.QuiesceWatchError 2024-09-21T10:14:16.905 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:14:17.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:16 smithi160 ceph-mon[101879]: osdmap e827: 8 total, 8 up, 8 in 2024-09-21T10:14:17.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:16 smithi070 ceph-mon[107385]: osdmap e827: 8 total, 8 up, 8 in 2024-09-21T10:14:17.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:16 smithi070 ceph-mon[112139]: osdmap e827: 8 total, 8 up, 8 in 2024-09-21T10:14:18.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:17 smithi160 ceph-mon[101879]: pgmap v889: 809 pgs: 809 active+clean; 2.6 MiB data, 534 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 5.5 KiB/s wr, 74 op/s 2024-09-21T10:14:18.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:17 smithi160 ceph-mon[101879]: osdmap e828: 8 total, 8 up, 8 in 2024-09-21T10:14:18.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:17 smithi160 ceph-mon[101879]: osdmap e829: 8 total, 8 up, 8 in 2024-09-21T10:14:18.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:17 smithi070 ceph-mon[107385]: pgmap v889: 809 pgs: 809 active+clean; 2.6 MiB data, 534 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 5.5 KiB/s wr, 74 op/s 2024-09-21T10:14:18.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:17 smithi070 ceph-mon[107385]: osdmap e828: 8 total, 8 up, 8 in 2024-09-21T10:14:18.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:17 smithi070 ceph-mon[107385]: osdmap e829: 8 total, 8 up, 8 in 2024-09-21T10:14:18.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:17 smithi070 ceph-mon[112139]: pgmap v889: 809 pgs: 809 active+clean; 2.6 MiB data, 534 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 5.5 KiB/s wr, 74 op/s 2024-09-21T10:14:18.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:17 smithi070 ceph-mon[112139]: osdmap e828: 8 total, 8 up, 8 in 2024-09-21T10:14:18.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:17 smithi070 ceph-mon[112139]: osdmap e829: 8 total, 8 up, 8 in 2024-09-21T10:14:20.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:19 smithi160 ceph-mon[101879]: pgmap v892: 809 pgs: 809 active+clean; 2.6 MiB data, 534 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 5.2 KiB/s wr, 67 op/s 2024-09-21T10:14:20.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:19 smithi160 ceph-mon[101879]: osdmap e830: 8 total, 8 up, 8 in 2024-09-21T10:14:20.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:19 smithi070 ceph-mon[107385]: pgmap v892: 809 pgs: 809 active+clean; 2.6 MiB data, 534 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 5.2 KiB/s wr, 67 op/s 2024-09-21T10:14:20.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:19 smithi070 ceph-mon[107385]: osdmap e830: 8 total, 8 up, 8 in 2024-09-21T10:14:20.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:19 smithi070 ceph-mon[112139]: pgmap v892: 809 pgs: 809 active+clean; 2.6 MiB data, 534 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 5.2 KiB/s wr, 67 op/s 2024-09-21T10:14:20.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:19 smithi070 ceph-mon[112139]: osdmap e830: 8 total, 8 up, 8 in 2024-09-21T10:14:21.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:20 smithi070 ceph-mon[107385]: osdmap e831: 8 total, 8 up, 8 in 2024-09-21T10:14:21.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:21.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:20 smithi070 ceph-mon[112139]: osdmap e831: 8 total, 8 up, 8 in 2024-09-21T10:14:21.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:20 smithi160 ceph-mon[101879]: osdmap e831: 8 total, 8 up, 8 in 2024-09-21T10:14:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:22.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:21 smithi070 ceph-mon[107385]: pgmap v895: 809 pgs: 809 active+clean; 2.6 MiB data, 534 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 511 B/s wr, 8 op/s 2024-09-21T10:14:22.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:21 smithi070 ceph-mon[107385]: osdmap e832: 8 total, 8 up, 8 in 2024-09-21T10:14:22.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:21 smithi070 ceph-mon[112139]: pgmap v895: 809 pgs: 809 active+clean; 2.6 MiB data, 534 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 511 B/s wr, 8 op/s 2024-09-21T10:14:22.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:21 smithi070 ceph-mon[112139]: osdmap e832: 8 total, 8 up, 8 in 2024-09-21T10:14:22.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:21 smithi160 ceph-mon[101879]: pgmap v895: 809 pgs: 809 active+clean; 2.6 MiB data, 534 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 511 B/s wr, 8 op/s 2024-09-21T10:14:22.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:21 smithi160 ceph-mon[101879]: osdmap e832: 8 total, 8 up, 8 in 2024-09-21T10:14:22.897 INFO:tasks.workunit.client.0.smithi070.stdout:10.00% done 2024-09-21T10:14:22.897 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.QuiesceWatchError (5991 ms) 2024-09-21T10:14:22.897 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.QuiesceWatchTimeout 2024-09-21T10:14:22.897 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:14:22.910 INFO:tasks.workunit.client.0.smithi070.stderr:test quiesce is not long enough to time out 2024-09-21T10:14:23.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:22 smithi070 ceph-mon[107385]: osdmap e833: 8 total, 8 up, 8 in 2024-09-21T10:14:23.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:22 smithi070 ceph-mon[107385]: osdmap e834: 8 total, 8 up, 8 in 2024-09-21T10:14:23.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:22 smithi070 ceph-mon[112139]: osdmap e833: 8 total, 8 up, 8 in 2024-09-21T10:14:23.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:22 smithi070 ceph-mon[112139]: osdmap e834: 8 total, 8 up, 8 in 2024-09-21T10:14:23.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:22 smithi160 ceph-mon[101879]: osdmap e833: 8 total, 8 up, 8 in 2024-09-21T10:14:23.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:22 smithi160 ceph-mon[101879]: osdmap e834: 8 total, 8 up, 8 in 2024-09-21T10:14:23.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:14:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:14:23] "GET /metrics HTTP/1.1" 200 89344 "" "Prometheus/2.51.0" 2024-09-21T10:14:24.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:23 smithi070 ceph-mon[107385]: pgmap v898: 809 pgs: 809 active+clean; 2.6 MiB data, 535 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 5.7 KiB/s wr, 108 op/s 2024-09-21T10:14:24.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:23 smithi070 ceph-mon[112139]: pgmap v898: 809 pgs: 809 active+clean; 2.6 MiB data, 535 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 5.7 KiB/s wr, 108 op/s 2024-09-21T10:14:24.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:23 smithi160 ceph-mon[101879]: pgmap v898: 809 pgs: 809 active+clean; 2.6 MiB data, 535 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 5.7 KiB/s wr, 108 op/s 2024-09-21T10:14:26.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:25 smithi070 ceph-mon[107385]: pgmap v900: 809 pgs: 809 active+clean; 2.6 MiB data, 535 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 5.2 KiB/s wr, 104 op/s 2024-09-21T10:14:26.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:25 smithi070 ceph-mon[112139]: pgmap v900: 809 pgs: 809 active+clean; 2.6 MiB data, 535 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 5.2 KiB/s wr, 104 op/s 2024-09-21T10:14:26.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:25 smithi160 ceph-mon[101879]: pgmap v900: 809 pgs: 809 active+clean; 2.6 MiB data, 535 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 5.2 KiB/s wr, 104 op/s 2024-09-21T10:14:28.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:27 smithi070 ceph-mon[107385]: pgmap v901: 809 pgs: 809 active+clean; 2.6 MiB data, 539 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 5.7 KiB/s wr, 109 op/s 2024-09-21T10:14:28.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:27 smithi070 ceph-mon[107385]: osdmap e835: 8 total, 8 up, 8 in 2024-09-21T10:14:28.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:27 smithi070 ceph-mon[112139]: pgmap v901: 809 pgs: 809 active+clean; 2.6 MiB data, 539 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 5.7 KiB/s wr, 109 op/s 2024-09-21T10:14:28.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:27 smithi070 ceph-mon[112139]: osdmap e835: 8 total, 8 up, 8 in 2024-09-21T10:14:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:27 smithi160 ceph-mon[101879]: pgmap v901: 809 pgs: 809 active+clean; 2.6 MiB data, 539 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 5.7 KiB/s wr, 109 op/s 2024-09-21T10:14:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:27 smithi160 ceph-mon[101879]: osdmap e835: 8 total, 8 up, 8 in 2024-09-21T10:14:30.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:29 smithi070 ceph-mon[107385]: pgmap v903: 809 pgs: 809 active+clean; 2.6 MiB data, 539 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 2.0 KiB/s wr, 39 op/s 2024-09-21T10:14:30.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:29 smithi070 ceph-mon[112139]: pgmap v903: 809 pgs: 809 active+clean; 2.6 MiB data, 539 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 2.0 KiB/s wr, 39 op/s 2024-09-21T10:14:30.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:29 smithi160 ceph-mon[101879]: pgmap v903: 809 pgs: 809 active+clean; 2.6 MiB data, 539 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 2.0 KiB/s wr, 39 op/s 2024-09-21T10:14:31.903 INFO:tasks.workunit.client.0.smithi070.stderr:test quiesce is timed out 2024-09-21T10:14:31.905 INFO:tasks.workunit.client.0.smithi070.stderr:waiting for timed out ... 0 2024-09-21T10:14:32.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:31 smithi160 ceph-mon[101879]: pgmap v904: 809 pgs: 809 active+clean; 2.6 MiB data, 539 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 1.6 KiB/s wr, 31 op/s 2024-09-21T10:14:32.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:31 smithi070 ceph-mon[107385]: pgmap v904: 809 pgs: 809 active+clean; 2.6 MiB data, 539 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 1.6 KiB/s wr, 31 op/s 2024-09-21T10:14:32.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:31 smithi070 ceph-mon[112139]: pgmap v904: 809 pgs: 809 active+clean; 2.6 MiB data, 539 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 1.6 KiB/s wr, 31 op/s 2024-09-21T10:14:32.906 INFO:tasks.workunit.client.0.smithi070.stderr:waiting for timed out ... 1 2024-09-21T10:14:33.154 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:32 smithi070 ceph-mon[107385]: osdmap e836: 8 total, 8 up, 8 in 2024-09-21T10:14:33.155 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:32 smithi070 ceph-mon[112139]: osdmap e836: 8 total, 8 up, 8 in 2024-09-21T10:14:33.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:32 smithi160 ceph-mon[101879]: osdmap e836: 8 total, 8 up, 8 in 2024-09-21T10:14:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:14:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:14:33] "GET /metrics HTTP/1.1" 200 89344 "" "Prometheus/2.51.0" 2024-09-21T10:14:33.906 INFO:tasks.workunit.client.0.smithi070.stderr:waiting for timed out ... 2 2024-09-21T10:14:34.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:33 smithi160 ceph-mon[101879]: pgmap v906: 809 pgs: 809 active+clean; 2.6 MiB data, 539 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 1.4 KiB/s wr, 24 op/s 2024-09-21T10:14:34.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:33 smithi070 ceph-mon[107385]: pgmap v906: 809 pgs: 809 active+clean; 2.6 MiB data, 539 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 1.4 KiB/s wr, 24 op/s 2024-09-21T10:14:34.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:33 smithi070 ceph-mon[112139]: pgmap v906: 809 pgs: 809 active+clean; 2.6 MiB data, 539 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 1.4 KiB/s wr, 24 op/s 2024-09-21T10:14:34.906 INFO:tasks.workunit.client.0.smithi070.stderr:waiting for timed out ... 3 2024-09-21T10:14:35.906 INFO:tasks.workunit.client.0.smithi070.stderr:waiting for timed out ... 4 2024-09-21T10:14:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:35 smithi160 ceph-mon[101879]: pgmap v907: 809 pgs: 809 active+clean; 2.6 MiB data, 539 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 127 B/s wr, 0 op/s 2024-09-21T10:14:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:36.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:35 smithi070 ceph-mon[107385]: pgmap v907: 809 pgs: 809 active+clean; 2.6 MiB data, 539 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 127 B/s wr, 0 op/s 2024-09-21T10:14:36.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:36.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:35 smithi070 ceph-mon[112139]: pgmap v907: 809 pgs: 809 active+clean; 2.6 MiB data, 539 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 127 B/s wr, 0 op/s 2024-09-21T10:14:36.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:36.906 INFO:tasks.workunit.client.0.smithi070.stderr:waiting for timed out ... 5 2024-09-21T10:14:37.906 INFO:tasks.workunit.client.0.smithi070.stderr:waiting for timed out ... 6 2024-09-21T10:14:38.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:37 smithi160 ceph-mon[101879]: pgmap v908: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 363 B/s wr, 4 op/s 2024-09-21T10:14:38.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:37 smithi070 ceph-mon[112139]: pgmap v908: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 363 B/s wr, 4 op/s 2024-09-21T10:14:38.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:37 smithi070 ceph-mon[107385]: pgmap v908: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 363 B/s wr, 4 op/s 2024-09-21T10:14:38.906 INFO:tasks.workunit.client.0.smithi070.stderr:waiting for timed out ... 7 2024-09-21T10:14:39.906 INFO:tasks.workunit.client.0.smithi070.stderr:waiting for timed out ... 8 2024-09-21T10:14:40.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:39 smithi160 ceph-mon[101879]: pgmap v909: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 306 B/s wr, 3 op/s 2024-09-21T10:14:40.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:39 smithi070 ceph-mon[107385]: pgmap v909: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 306 B/s wr, 3 op/s 2024-09-21T10:14:40.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:39 smithi070 ceph-mon[112139]: pgmap v909: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 306 B/s wr, 3 op/s 2024-09-21T10:14:40.906 INFO:tasks.workunit.client.0.smithi070.stderr:waiting for timed out ... 9 2024-09-21T10:14:41.907 INFO:tasks.workunit.client.0.smithi070.stderr:waiting for timed out ... 10 2024-09-21T10:14:42.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:41 smithi160 ceph-mon[101879]: pgmap v910: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 306 B/s wr, 3 op/s 2024-09-21T10:14:42.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:41 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:14:42.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:41 smithi070 ceph-mon[107385]: pgmap v910: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 306 B/s wr, 3 op/s 2024-09-21T10:14:42.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:41 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:14:42.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:41 smithi070 ceph-mon[112139]: pgmap v910: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 306 B/s wr, 3 op/s 2024-09-21T10:14:42.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:41 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:14:43.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:14:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:14:43] "GET /metrics HTTP/1.1" 200 89349 "" "Prometheus/2.51.0" 2024-09-21T10:14:44.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:43 smithi070 ceph-mon[107385]: pgmap v911: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 295 B/s wr, 3 op/s 2024-09-21T10:14:44.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:43 smithi070 ceph-mon[112139]: pgmap v911: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 295 B/s wr, 3 op/s 2024-09-21T10:14:44.234 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:43 smithi160 ceph-mon[101879]: pgmap v911: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 295 B/s wr, 3 op/s 2024-09-21T10:14:46.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:45 smithi160 ceph-mon[101879]: pgmap v912: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 255 B/s wr, 3 op/s 2024-09-21T10:14:46.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:45 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:14:46.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:45 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:14:46.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:45 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:14:46.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:45 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:14:46.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:45 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:14:46.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:45 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:14:46.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:45 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:14:46.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:45 smithi070 ceph-mon[107385]: pgmap v912: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 255 B/s wr, 3 op/s 2024-09-21T10:14:46.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:45 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:14:46.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:45 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:14:46.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:45 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:14:46.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:45 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:14:46.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:45 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:14:46.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:45 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:14:46.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:45 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:14:46.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:45 smithi070 ceph-mon[112139]: pgmap v912: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 255 B/s wr, 3 op/s 2024-09-21T10:14:46.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:45 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:14:46.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:45 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:14:46.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:45 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:14:46.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:45 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:14:46.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:45 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:14:46.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:45 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:14:46.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:45 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:14:48.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:47 smithi160 ceph-mon[101879]: pgmap v913: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 170 B/s wr, 2 op/s 2024-09-21T10:14:48.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:47 smithi070 ceph-mon[107385]: pgmap v913: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 170 B/s wr, 2 op/s 2024-09-21T10:14:48.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:47 smithi070 ceph-mon[112139]: pgmap v913: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 170 B/s wr, 2 op/s 2024-09-21T10:14:50.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:49 smithi160 ceph-mon[101879]: pgmap v914: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:14:50.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:49 smithi070 ceph-mon[107385]: pgmap v914: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:14:50.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:49 smithi070 ceph-mon[112139]: pgmap v914: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:14:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:51.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:51.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:52.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:51 smithi160 ceph-mon[101879]: pgmap v915: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:14:52.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.18", "id": [4, 3]} : dispatch 2024-09-21T10:14:52.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.8", "id": [3, 2]} : dispatch 2024-09-21T10:14:52.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.d", "id": [3, 1]} : dispatch 2024-09-21T10:14:52.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.13", "id": [3, 1]} : dispatch 2024-09-21T10:14:52.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.19", "id": [6, 1]} : dispatch 2024-09-21T10:14:52.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:51 smithi070 ceph-mon[112139]: pgmap v915: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:14:52.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.18", "id": [4, 3]} : dispatch 2024-09-21T10:14:52.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.8", "id": [3, 2]} : dispatch 2024-09-21T10:14:52.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.d", "id": [3, 1]} : dispatch 2024-09-21T10:14:52.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.13", "id": [3, 1]} : dispatch 2024-09-21T10:14:52.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.19", "id": [6, 1]} : dispatch 2024-09-21T10:14:52.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:51 smithi070 ceph-mon[107385]: pgmap v915: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:14:52.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.18", "id": [4, 3]} : dispatch 2024-09-21T10:14:52.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.8", "id": [3, 2]} : dispatch 2024-09-21T10:14:52.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.d", "id": [3, 1]} : dispatch 2024-09-21T10:14:52.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.13", "id": [3, 1]} : dispatch 2024-09-21T10:14:52.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.19", "id": [6, 1]} : dispatch 2024-09-21T10:14:52.907 INFO:tasks.workunit.client.0.smithi070.stderr:test retry succeeds 2024-09-21T10:14:53.279 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:14:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:14:53] "GET /metrics HTTP/1.1" 200 89345 "" "Prometheus/2.51.0" 2024-09-21T10:14:53.279 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:52 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.18", "id": [4, 3]}]': finished 2024-09-21T10:14:53.279 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:52 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.8", "id": [3, 2]}]': finished 2024-09-21T10:14:53.279 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:52 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.d", "id": [3, 1]}]': finished 2024-09-21T10:14:53.279 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:52 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.13", "id": [3, 1]}]': finished 2024-09-21T10:14:53.280 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:52 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.19", "id": [6, 1]}]': finished 2024-09-21T10:14:53.280 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:52 smithi070 ceph-mon[107385]: osdmap e837: 8 total, 8 up, 8 in 2024-09-21T10:14:53.280 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:53 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.18", "id": [4, 3]}]': finished 2024-09-21T10:14:53.280 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:53 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.8", "id": [3, 2]}]': finished 2024-09-21T10:14:53.280 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:53 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.d", "id": [3, 1]}]': finished 2024-09-21T10:14:53.280 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:53 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.13", "id": [3, 1]}]': finished 2024-09-21T10:14:53.280 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:53 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.19", "id": [6, 1]}]': finished 2024-09-21T10:14:53.280 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:53 smithi070 ceph-mon[112139]: osdmap e837: 8 total, 8 up, 8 in 2024-09-21T10:14:53.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:52 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.18", "id": [4, 3]}]': finished 2024-09-21T10:14:53.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:52 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.8", "id": [3, 2]}]': finished 2024-09-21T10:14:53.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:52 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.d", "id": [3, 1]}]': finished 2024-09-21T10:14:53.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:52 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.13", "id": [3, 1]}]': finished 2024-09-21T10:14:53.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:52 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.19", "id": [6, 1]}]': finished 2024-09-21T10:14:53.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:52 smithi160 ceph-mon[101879]: osdmap e837: 8 total, 8 up, 8 in 2024-09-21T10:14:54.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:53 smithi070 ceph-mon[107385]: pgmap v917: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:14:54.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:53 smithi070 ceph-mon[107385]: osdmap e838: 8 total, 8 up, 8 in 2024-09-21T10:14:54.299 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:53 smithi070 ceph-mon[112139]: pgmap v917: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:14:54.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:53 smithi070 ceph-mon[112139]: osdmap e838: 8 total, 8 up, 8 in 2024-09-21T10:14:54.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:53 smithi160 ceph-mon[101879]: pgmap v917: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:14:54.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:53 smithi160 ceph-mon[101879]: osdmap e838: 8 total, 8 up, 8 in 2024-09-21T10:14:55.092 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.QuiesceWatchTimeout (32196 ms) 2024-09-21T10:14:55.092 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.WriteZeroes 2024-09-21T10:14:55.092 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:14:55.155 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.WriteZeroes (62 ms) 2024-09-21T10:14:55.155 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.WriteZeroesThickProvision 2024-09-21T10:14:55.155 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:14:55.223 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.WriteZeroesThickProvision (69 ms) 2024-09-21T10:14:55.224 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ConcurentOperations 2024-09-21T10:14:55.224 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:14:55.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:55 smithi160 ceph-mon[101879]: osdmap e839: 8 total, 8 up, 8 in 2024-09-21T10:14:55.342 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 10:14:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=sqlstore.transactions t=2024-09-21T10:14:55.082920876Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T10:14:55.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:55 smithi070 ceph-mon[107385]: osdmap e839: 8 total, 8 up, 8 in 2024-09-21T10:14:55.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:55 smithi070 ceph-mon[112139]: osdmap e839: 8 total, 8 up, 8 in 2024-09-21T10:14:56.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:56 smithi160 ceph-mon[101879]: pgmap v920: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 B/s wr, 0 op/s 2024-09-21T10:14:56.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:56 smithi160 ceph-mon[101879]: osdmap e840: 8 total, 8 up, 8 in 2024-09-21T10:14:56.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:56 smithi070 ceph-mon[107385]: pgmap v920: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 B/s wr, 0 op/s 2024-09-21T10:14:56.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:56 smithi070 ceph-mon[107385]: osdmap e840: 8 total, 8 up, 8 in 2024-09-21T10:14:56.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:56 smithi070 ceph-mon[112139]: pgmap v920: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 B/s wr, 0 op/s 2024-09-21T10:14:56.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:56 smithi070 ceph-mon[112139]: osdmap e840: 8 total, 8 up, 8 in 2024-09-21T10:14:57.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:57 smithi160 ceph-mon[101879]: osdmap e841: 8 total, 8 up, 8 in 2024-09-21T10:14:57.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:57 smithi070 ceph-mon[107385]: osdmap e841: 8 total, 8 up, 8 in 2024-09-21T10:14:57.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:57 smithi070 ceph-mon[112139]: osdmap e841: 8 total, 8 up, 8 in 2024-09-21T10:14:58.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:58 smithi160 ceph-mon[101879]: pgmap v923: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 7.0 KiB/s wr, 115 op/s 2024-09-21T10:14:58.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:58 smithi160 ceph-mon[101879]: osdmap e842: 8 total, 8 up, 8 in 2024-09-21T10:14:58.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:58 smithi070 ceph-mon[107385]: pgmap v923: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 7.0 KiB/s wr, 115 op/s 2024-09-21T10:14:58.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:58 smithi070 ceph-mon[107385]: osdmap e842: 8 total, 8 up, 8 in 2024-09-21T10:14:58.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:58 smithi070 ceph-mon[112139]: pgmap v923: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 7.0 KiB/s wr, 115 op/s 2024-09-21T10:14:58.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:58 smithi070 ceph-mon[112139]: osdmap e842: 8 total, 8 up, 8 in 2024-09-21T10:14:59.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:14:59 smithi160 ceph-mon[101879]: osdmap e843: 8 total, 8 up, 8 in 2024-09-21T10:14:59.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:14:59 smithi070 ceph-mon[107385]: osdmap e843: 8 total, 8 up, 8 in 2024-09-21T10:14:59.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:14:59 smithi070 ceph-mon[112139]: osdmap e843: 8 total, 8 up, 8 in 2024-09-21T10:15:00.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:00 smithi160 ceph-mon[101879]: pgmap v926: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 7.0 KiB/s wr, 114 op/s 2024-09-21T10:15:00.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:00 smithi160 ceph-mon[101879]: osdmap e844: 8 total, 8 up, 8 in 2024-09-21T10:15:00.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:00 smithi070 ceph-mon[107385]: pgmap v926: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 7.0 KiB/s wr, 114 op/s 2024-09-21T10:15:00.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:00 smithi070 ceph-mon[107385]: osdmap e844: 8 total, 8 up, 8 in 2024-09-21T10:15:00.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:00 smithi070 ceph-mon[112139]: pgmap v926: 809 pgs: 809 active+clean; 2.6 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 7.0 KiB/s wr, 114 op/s 2024-09-21T10:15:00.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:00 smithi070 ceph-mon[112139]: osdmap e844: 8 total, 8 up, 8 in 2024-09-21T10:15:01.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:01 smithi160 ceph-mon[101879]: osdmap e845: 8 total, 8 up, 8 in 2024-09-21T10:15:01.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:01 smithi070 ceph-mon[107385]: osdmap e845: 8 total, 8 up, 8 in 2024-09-21T10:15:01.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:01 smithi070 ceph-mon[112139]: osdmap e845: 8 total, 8 up, 8 in 2024-09-21T10:15:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:02 smithi160 ceph-mon[101879]: pgmap v929: 809 pgs: 809 active+clean; 2.6 MiB data, 541 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 3.0 KiB/s wr, 64 op/s 2024-09-21T10:15:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:02 smithi160 ceph-mon[101879]: osdmap e846: 8 total, 8 up, 8 in 2024-09-21T10:15:02.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:02 smithi070 ceph-mon[107385]: pgmap v929: 809 pgs: 809 active+clean; 2.6 MiB data, 541 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 3.0 KiB/s wr, 64 op/s 2024-09-21T10:15:02.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:02 smithi070 ceph-mon[107385]: osdmap e846: 8 total, 8 up, 8 in 2024-09-21T10:15:02.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:02 smithi070 ceph-mon[112139]: pgmap v929: 809 pgs: 809 active+clean; 2.6 MiB data, 541 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 3.0 KiB/s wr, 64 op/s 2024-09-21T10:15:02.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:02 smithi070 ceph-mon[112139]: osdmap e846: 8 total, 8 up, 8 in 2024-09-21T10:15:03.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:03 smithi160 ceph-mon[101879]: osdmap e847: 8 total, 8 up, 8 in 2024-09-21T10:15:03.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:03 smithi160 ceph-mon[101879]: osdmap e848: 8 total, 8 up, 8 in 2024-09-21T10:15:03.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:03 smithi070 ceph-mon[107385]: osdmap e847: 8 total, 8 up, 8 in 2024-09-21T10:15:03.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:03 smithi070 ceph-mon[107385]: osdmap e848: 8 total, 8 up, 8 in 2024-09-21T10:15:03.378 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:15:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:15:03] "GET /metrics HTTP/1.1" 200 89345 "" "Prometheus/2.51.0" 2024-09-21T10:15:03.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:03 smithi070 ceph-mon[112139]: osdmap e847: 8 total, 8 up, 8 in 2024-09-21T10:15:03.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:03 smithi070 ceph-mon[112139]: osdmap e848: 8 total, 8 up, 8 in 2024-09-21T10:15:04.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:04 smithi160 ceph-mon[101879]: pgmap v932: 809 pgs: 809 active+clean; 2.6 MiB data, 541 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 6.5 KiB/s wr, 80 op/s 2024-09-21T10:15:04.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:04 smithi160 ceph-mon[101879]: osdmap e849: 8 total, 8 up, 8 in 2024-09-21T10:15:04.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:04 smithi070 ceph-mon[107385]: pgmap v932: 809 pgs: 809 active+clean; 2.6 MiB data, 541 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 6.5 KiB/s wr, 80 op/s 2024-09-21T10:15:04.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:04 smithi070 ceph-mon[107385]: osdmap e849: 8 total, 8 up, 8 in 2024-09-21T10:15:04.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:04 smithi070 ceph-mon[112139]: pgmap v932: 809 pgs: 809 active+clean; 2.6 MiB data, 541 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 6.5 KiB/s wr, 80 op/s 2024-09-21T10:15:04.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:04 smithi070 ceph-mon[112139]: osdmap e849: 8 total, 8 up, 8 in 2024-09-21T10:15:05.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:05 smithi070 ceph-mon[107385]: pgmap v935: 809 pgs: 809 active+clean; 2.6 MiB data, 542 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 4.2 KiB/s wr, 16 op/s 2024-09-21T10:15:05.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:05 smithi070 ceph-mon[107385]: osdmap e850: 8 total, 8 up, 8 in 2024-09-21T10:15:05.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:05.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:05 smithi070 ceph-mon[112139]: pgmap v935: 809 pgs: 809 active+clean; 2.6 MiB data, 542 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 4.2 KiB/s wr, 16 op/s 2024-09-21T10:15:05.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:05 smithi070 ceph-mon[112139]: osdmap e850: 8 total, 8 up, 8 in 2024-09-21T10:15:05.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:05 smithi160 ceph-mon[101879]: pgmap v935: 809 pgs: 809 active+clean; 2.6 MiB data, 542 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 4.2 KiB/s wr, 16 op/s 2024-09-21T10:15:06.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:05 smithi160 ceph-mon[101879]: osdmap e850: 8 total, 8 up, 8 in 2024-09-21T10:15:06.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:07.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:06 smithi160 ceph-mon[101879]: osdmap e851: 8 total, 8 up, 8 in 2024-09-21T10:15:07.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:06 smithi070 ceph-mon[107385]: osdmap e851: 8 total, 8 up, 8 in 2024-09-21T10:15:07.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:06 smithi070 ceph-mon[112139]: osdmap e851: 8 total, 8 up, 8 in 2024-09-21T10:15:08.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:07 smithi160 ceph-mon[101879]: pgmap v938: 809 pgs: 809 active+clean; 2.6 MiB data, 542 MiB used, 715 GiB / 715 GiB avail; 175 KiB/s rd, 4.7 KiB/s wr, 238 op/s 2024-09-21T10:15:08.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:07 smithi160 ceph-mon[101879]: osdmap e852: 8 total, 8 up, 8 in 2024-09-21T10:15:08.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:07 smithi070 ceph-mon[107385]: pgmap v938: 809 pgs: 809 active+clean; 2.6 MiB data, 542 MiB used, 715 GiB / 715 GiB avail; 175 KiB/s rd, 4.7 KiB/s wr, 238 op/s 2024-09-21T10:15:08.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:07 smithi070 ceph-mon[107385]: osdmap e852: 8 total, 8 up, 8 in 2024-09-21T10:15:08.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:07 smithi070 ceph-mon[112139]: pgmap v938: 809 pgs: 809 active+clean; 2.6 MiB data, 542 MiB used, 715 GiB / 715 GiB avail; 175 KiB/s rd, 4.7 KiB/s wr, 238 op/s 2024-09-21T10:15:08.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:07 smithi070 ceph-mon[112139]: osdmap e852: 8 total, 8 up, 8 in 2024-09-21T10:15:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:08 smithi070 ceph-mon[107385]: osdmap e853: 8 total, 8 up, 8 in 2024-09-21T10:15:09.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:08 smithi070 ceph-mon[112139]: osdmap e853: 8 total, 8 up, 8 in 2024-09-21T10:15:09.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:08 smithi160 ceph-mon[101879]: osdmap e853: 8 total, 8 up, 8 in 2024-09-21T10:15:10.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:09 smithi070 ceph-mon[107385]: pgmap v941: 809 pgs: 809 active+clean; 2.6 MiB data, 542 MiB used, 715 GiB / 715 GiB avail; 175 KiB/s rd, 4.0 KiB/s wr, 237 op/s 2024-09-21T10:15:10.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:09 smithi070 ceph-mon[107385]: osdmap e854: 8 total, 8 up, 8 in 2024-09-21T10:15:10.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:09 smithi070 ceph-mon[112139]: pgmap v941: 809 pgs: 809 active+clean; 2.6 MiB data, 542 MiB used, 715 GiB / 715 GiB avail; 175 KiB/s rd, 4.0 KiB/s wr, 237 op/s 2024-09-21T10:15:10.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:09 smithi070 ceph-mon[112139]: osdmap e854: 8 total, 8 up, 8 in 2024-09-21T10:15:10.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:09 smithi160 ceph-mon[101879]: pgmap v941: 809 pgs: 809 active+clean; 2.6 MiB data, 542 MiB used, 715 GiB / 715 GiB avail; 175 KiB/s rd, 4.0 KiB/s wr, 237 op/s 2024-09-21T10:15:10.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:09 smithi160 ceph-mon[101879]: osdmap e854: 8 total, 8 up, 8 in 2024-09-21T10:15:11.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:10 smithi070 ceph-mon[107385]: osdmap e855: 8 total, 8 up, 8 in 2024-09-21T10:15:11.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:10 smithi070 ceph-mon[112139]: osdmap e855: 8 total, 8 up, 8 in 2024-09-21T10:15:11.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:10 smithi160 ceph-mon[101879]: osdmap e855: 8 total, 8 up, 8 in 2024-09-21T10:15:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:11 smithi070 ceph-mon[107385]: pgmap v944: 809 pgs: 809 active+clean; 2.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-21T10:15:12.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:11 smithi070 ceph-mon[107385]: osdmap e856: 8 total, 8 up, 8 in 2024-09-21T10:15:12.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:11 smithi070 ceph-mon[112139]: pgmap v944: 809 pgs: 809 active+clean; 2.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-21T10:15:12.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:11 smithi070 ceph-mon[112139]: osdmap e856: 8 total, 8 up, 8 in 2024-09-21T10:15:12.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:11 smithi160 ceph-mon[101879]: pgmap v944: 809 pgs: 809 active+clean; 2.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-21T10:15:12.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:11 smithi160 ceph-mon[101879]: osdmap e856: 8 total, 8 up, 8 in 2024-09-21T10:15:13.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:12 smithi070 ceph-mon[107385]: osdmap e857: 8 total, 8 up, 8 in 2024-09-21T10:15:13.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:12 smithi070 ceph-mon[107385]: osdmap e858: 8 total, 8 up, 8 in 2024-09-21T10:15:13.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:12 smithi070 ceph-mon[112139]: osdmap e857: 8 total, 8 up, 8 in 2024-09-21T10:15:13.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:12 smithi070 ceph-mon[112139]: osdmap e858: 8 total, 8 up, 8 in 2024-09-21T10:15:13.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:12 smithi160 ceph-mon[101879]: osdmap e857: 8 total, 8 up, 8 in 2024-09-21T10:15:13.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:12 smithi160 ceph-mon[101879]: osdmap e858: 8 total, 8 up, 8 in 2024-09-21T10:15:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:15:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:15:13] "GET /metrics HTTP/1.1" 200 89353 "" "Prometheus/2.51.0" 2024-09-21T10:15:14.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:13 smithi070 ceph-mon[107385]: pgmap v947: 809 pgs: 809 active+clean; 2.6 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 5.2 KiB/s wr, 28 op/s 2024-09-21T10:15:14.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:13 smithi070 ceph-mon[107385]: osdmap e859: 8 total, 8 up, 8 in 2024-09-21T10:15:14.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:13 smithi070 ceph-mon[112139]: pgmap v947: 809 pgs: 809 active+clean; 2.6 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 5.2 KiB/s wr, 28 op/s 2024-09-21T10:15:14.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:13 smithi070 ceph-mon[112139]: osdmap e859: 8 total, 8 up, 8 in 2024-09-21T10:15:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:13 smithi160 ceph-mon[101879]: pgmap v947: 809 pgs: 809 active+clean; 2.6 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 5.2 KiB/s wr, 28 op/s 2024-09-21T10:15:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:13 smithi160 ceph-mon[101879]: osdmap e859: 8 total, 8 up, 8 in 2024-09-21T10:15:16.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:15 smithi160 ceph-mon[101879]: pgmap v950: 809 pgs: 809 active+clean; 2.6 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 5.0 KiB/s wr, 23 op/s 2024-09-21T10:15:16.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:15 smithi160 ceph-mon[101879]: osdmap e860: 8 total, 8 up, 8 in 2024-09-21T10:15:16.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:15 smithi070 ceph-mon[112139]: pgmap v950: 809 pgs: 809 active+clean; 2.6 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 5.0 KiB/s wr, 23 op/s 2024-09-21T10:15:16.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:15 smithi070 ceph-mon[112139]: osdmap e860: 8 total, 8 up, 8 in 2024-09-21T10:15:16.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:15 smithi070 ceph-mon[107385]: pgmap v950: 809 pgs: 809 active+clean; 2.6 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 5.0 KiB/s wr, 23 op/s 2024-09-21T10:15:16.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:15 smithi070 ceph-mon[107385]: osdmap e860: 8 total, 8 up, 8 in 2024-09-21T10:15:17.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:16 smithi070 ceph-mon[112139]: osdmap e861: 8 total, 8 up, 8 in 2024-09-21T10:15:17.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:16 smithi070 ceph-mon[107385]: osdmap e861: 8 total, 8 up, 8 in 2024-09-21T10:15:17.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:16 smithi160 ceph-mon[101879]: osdmap e861: 8 total, 8 up, 8 in 2024-09-21T10:15:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:17 smithi160 ceph-mon[101879]: pgmap v953: 809 pgs: 809 active+clean; 2.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 4.0 KiB/s wr, 44 op/s 2024-09-21T10:15:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:17 smithi160 ceph-mon[101879]: osdmap e862: 8 total, 8 up, 8 in 2024-09-21T10:15:18.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:17 smithi160 ceph-mon[101879]: osdmap e863: 8 total, 8 up, 8 in 2024-09-21T10:15:18.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:17 smithi070 ceph-mon[107385]: pgmap v953: 809 pgs: 809 active+clean; 2.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 4.0 KiB/s wr, 44 op/s 2024-09-21T10:15:18.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:17 smithi070 ceph-mon[107385]: osdmap e862: 8 total, 8 up, 8 in 2024-09-21T10:15:18.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:17 smithi070 ceph-mon[107385]: osdmap e863: 8 total, 8 up, 8 in 2024-09-21T10:15:18.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:17 smithi070 ceph-mon[112139]: pgmap v953: 809 pgs: 809 active+clean; 2.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 4.0 KiB/s wr, 44 op/s 2024-09-21T10:15:18.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:17 smithi070 ceph-mon[112139]: osdmap e862: 8 total, 8 up, 8 in 2024-09-21T10:15:18.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:17 smithi070 ceph-mon[112139]: osdmap e863: 8 total, 8 up, 8 in 2024-09-21T10:15:20.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:19 smithi160 ceph-mon[101879]: pgmap v956: 809 pgs: 809 active+clean; 2.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 4.0 KiB/s wr, 42 op/s 2024-09-21T10:15:20.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:19 smithi160 ceph-mon[101879]: osdmap e864: 8 total, 8 up, 8 in 2024-09-21T10:15:20.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:19 smithi070 ceph-mon[112139]: pgmap v956: 809 pgs: 809 active+clean; 2.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 4.0 KiB/s wr, 42 op/s 2024-09-21T10:15:20.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:19 smithi070 ceph-mon[112139]: osdmap e864: 8 total, 8 up, 8 in 2024-09-21T10:15:20.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:19 smithi070 ceph-mon[107385]: pgmap v956: 809 pgs: 809 active+clean; 2.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 4.0 KiB/s wr, 42 op/s 2024-09-21T10:15:20.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:19 smithi070 ceph-mon[107385]: osdmap e864: 8 total, 8 up, 8 in 2024-09-21T10:15:20.924 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ConcurentOperations (25701 ms) 2024-09-21T10:15:21.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:20 smithi160 ceph-mon[101879]: osdmap e865: 8 total, 8 up, 8 in 2024-09-21T10:15:21.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:21.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:20 smithi070 ceph-mon[107385]: osdmap e865: 8 total, 8 up, 8 in 2024-09-21T10:15:21.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:21.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:20 smithi070 ceph-mon[112139]: osdmap e865: 8 total, 8 up, 8 in 2024-09-21T10:15:21.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:22.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:21 smithi160 ceph-mon[101879]: pgmap v959: 809 pgs: 809 active+clean; 2.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 255 B/s wr, 4 op/s 2024-09-21T10:15:22.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:21 smithi160 ceph-mon[101879]: osdmap e866: 8 total, 8 up, 8 in 2024-09-21T10:15:22.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:21 smithi070 ceph-mon[107385]: pgmap v959: 809 pgs: 809 active+clean; 2.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 255 B/s wr, 4 op/s 2024-09-21T10:15:22.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:21 smithi070 ceph-mon[107385]: osdmap e866: 8 total, 8 up, 8 in 2024-09-21T10:15:22.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:21 smithi070 ceph-mon[112139]: pgmap v959: 809 pgs: 809 active+clean; 2.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 255 B/s wr, 4 op/s 2024-09-21T10:15:22.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:21 smithi070 ceph-mon[112139]: osdmap e866: 8 total, 8 up, 8 in 2024-09-21T10:15:23.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:22 smithi070 ceph-mon[107385]: osdmap e867: 8 total, 8 up, 8 in 2024-09-21T10:15:23.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:22 smithi070 ceph-mon[107385]: osdmap e868: 8 total, 8 up, 8 in 2024-09-21T10:15:23.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:22 smithi070 ceph-mon[112139]: osdmap e867: 8 total, 8 up, 8 in 2024-09-21T10:15:23.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:22 smithi070 ceph-mon[112139]: osdmap e868: 8 total, 8 up, 8 in 2024-09-21T10:15:23.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:22 smithi160 ceph-mon[101879]: osdmap e867: 8 total, 8 up, 8 in 2024-09-21T10:15:23.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:22 smithi160 ceph-mon[101879]: osdmap e868: 8 total, 8 up, 8 in 2024-09-21T10:15:23.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:15:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:15:23] "GET /metrics HTTP/1.1" 200 89360 "" "Prometheus/2.51.0" 2024-09-21T10:15:24.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:23 smithi070 ceph-mon[107385]: pgmap v962: 777 pgs: 777 active+clean; 2.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 2.0 KiB/s wr, 72 op/s 2024-09-21T10:15:24.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:23 smithi070 ceph-mon[107385]: osdmap e869: 8 total, 8 up, 8 in 2024-09-21T10:15:24.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:23 smithi070 ceph-mon[112139]: pgmap v962: 777 pgs: 777 active+clean; 2.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 2.0 KiB/s wr, 72 op/s 2024-09-21T10:15:24.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:23 smithi070 ceph-mon[112139]: osdmap e869: 8 total, 8 up, 8 in 2024-09-21T10:15:24.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:23 smithi160 ceph-mon[101879]: pgmap v962: 777 pgs: 777 active+clean; 2.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 2.0 KiB/s wr, 72 op/s 2024-09-21T10:15:24.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:23 smithi160 ceph-mon[101879]: osdmap e869: 8 total, 8 up, 8 in 2024-09-21T10:15:26.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:25 smithi160 ceph-mon[101879]: pgmap v965: 713 pgs: 713 active+clean; 2.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:26.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:25 smithi160 ceph-mon[101879]: osdmap e870: 8 total, 8 up, 8 in 2024-09-21T10:15:26.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:25 smithi070 ceph-mon[107385]: pgmap v965: 713 pgs: 713 active+clean; 2.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:26.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:25 smithi070 ceph-mon[107385]: osdmap e870: 8 total, 8 up, 8 in 2024-09-21T10:15:26.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:25 smithi070 ceph-mon[112139]: pgmap v965: 713 pgs: 713 active+clean; 2.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:26.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:25 smithi070 ceph-mon[112139]: osdmap e870: 8 total, 8 up, 8 in 2024-09-21T10:15:27.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:26 smithi070 ceph-mon[107385]: osdmap e871: 8 total, 8 up, 8 in 2024-09-21T10:15:27.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:26 smithi070 ceph-mon[112139]: osdmap e871: 8 total, 8 up, 8 in 2024-09-21T10:15:27.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:26 smithi160 ceph-mon[101879]: osdmap e871: 8 total, 8 up, 8 in 2024-09-21T10:15:28.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:27 smithi070 ceph-mon[107385]: pgmap v968: 649 pgs: 649 active+clean; 2.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:28.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:27 smithi070 ceph-mon[107385]: osdmap e872: 8 total, 8 up, 8 in 2024-09-21T10:15:28.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:27 smithi070 ceph-mon[107385]: osdmap e873: 8 total, 8 up, 8 in 2024-09-21T10:15:28.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:27 smithi070 ceph-mon[112139]: pgmap v968: 649 pgs: 649 active+clean; 2.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:28.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:27 smithi070 ceph-mon[112139]: osdmap e872: 8 total, 8 up, 8 in 2024-09-21T10:15:28.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:27 smithi070 ceph-mon[112139]: osdmap e873: 8 total, 8 up, 8 in 2024-09-21T10:15:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:27 smithi160 ceph-mon[101879]: pgmap v968: 649 pgs: 649 active+clean; 2.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:27 smithi160 ceph-mon[101879]: osdmap e872: 8 total, 8 up, 8 in 2024-09-21T10:15:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:27 smithi160 ceph-mon[101879]: osdmap e873: 8 total, 8 up, 8 in 2024-09-21T10:15:30.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:29 smithi160 ceph-mon[101879]: pgmap v971: 585 pgs: 585 active+clean; 2.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:30.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:29 smithi160 ceph-mon[101879]: osdmap e874: 8 total, 8 up, 8 in 2024-09-21T10:15:30.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:29 smithi070 ceph-mon[112139]: pgmap v971: 585 pgs: 585 active+clean; 2.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:30.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:29 smithi070 ceph-mon[112139]: osdmap e874: 8 total, 8 up, 8 in 2024-09-21T10:15:30.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:29 smithi070 ceph-mon[107385]: pgmap v971: 585 pgs: 585 active+clean; 2.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:30.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:29 smithi070 ceph-mon[107385]: osdmap e874: 8 total, 8 up, 8 in 2024-09-21T10:15:31.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:30 smithi070 ceph-mon[107385]: Health check update: 1 pool(s) full (POOL_FULL) 2024-09-21T10:15:31.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:30 smithi070 ceph-mon[107385]: osdmap e875: 8 total, 8 up, 8 in 2024-09-21T10:15:31.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:30 smithi070 ceph-mon[107385]: osdmap e876: 8 total, 8 up, 8 in 2024-09-21T10:15:31.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:30 smithi070 ceph-mon[112139]: Health check update: 1 pool(s) full (POOL_FULL) 2024-09-21T10:15:31.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:30 smithi070 ceph-mon[112139]: osdmap e875: 8 total, 8 up, 8 in 2024-09-21T10:15:31.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:30 smithi070 ceph-mon[112139]: osdmap e876: 8 total, 8 up, 8 in 2024-09-21T10:15:31.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:30 smithi160 ceph-mon[101879]: Health check update: 1 pool(s) full (POOL_FULL) 2024-09-21T10:15:31.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:30 smithi160 ceph-mon[101879]: osdmap e875: 8 total, 8 up, 8 in 2024-09-21T10:15:31.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:30 smithi160 ceph-mon[101879]: osdmap e876: 8 total, 8 up, 8 in 2024-09-21T10:15:32.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:31 smithi070 ceph-mon[112139]: pgmap v974: 521 pgs: 521 active+clean; 2.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:32.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:31 smithi070 ceph-mon[112139]: osdmap e877: 8 total, 8 up, 8 in 2024-09-21T10:15:32.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:31 smithi070 ceph-mon[107385]: pgmap v974: 521 pgs: 521 active+clean; 2.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:32.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:31 smithi070 ceph-mon[107385]: osdmap e877: 8 total, 8 up, 8 in 2024-09-21T10:15:32.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:31 smithi160 ceph-mon[101879]: pgmap v974: 521 pgs: 521 active+clean; 2.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:32.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:31 smithi160 ceph-mon[101879]: osdmap e877: 8 total, 8 up, 8 in 2024-09-21T10:15:33.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:32 smithi070 ceph-mon[107385]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-21T10:15:33.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:32 smithi070 ceph-mon[107385]: Cluster is now healthy 2024-09-21T10:15:33.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:32 smithi070 ceph-mon[107385]: osdmap e878: 8 total, 8 up, 8 in 2024-09-21T10:15:33.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:32 smithi070 ceph-mon[112139]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-21T10:15:33.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:32 smithi070 ceph-mon[112139]: Cluster is now healthy 2024-09-21T10:15:33.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:32 smithi070 ceph-mon[112139]: osdmap e878: 8 total, 8 up, 8 in 2024-09-21T10:15:33.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:32 smithi160 ceph-mon[101879]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-21T10:15:33.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:32 smithi160 ceph-mon[101879]: Cluster is now healthy 2024-09-21T10:15:33.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:32 smithi160 ceph-mon[101879]: osdmap e878: 8 total, 8 up, 8 in 2024-09-21T10:15:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:15:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:15:33] "GET /metrics HTTP/1.1" 200 89360 "" "Prometheus/2.51.0" 2024-09-21T10:15:34.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:33 smithi070 ceph-mon[107385]: pgmap v977: 457 pgs: 457 active+clean; 2.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:34.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:33 smithi070 ceph-mon[107385]: osdmap e879: 8 total, 8 up, 8 in 2024-09-21T10:15:34.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:33 smithi070 ceph-mon[112139]: pgmap v977: 457 pgs: 457 active+clean; 2.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:34.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:33 smithi070 ceph-mon[112139]: osdmap e879: 8 total, 8 up, 8 in 2024-09-21T10:15:34.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:33 smithi160 ceph-mon[101879]: pgmap v977: 457 pgs: 457 active+clean; 2.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:34.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:33 smithi160 ceph-mon[101879]: osdmap e879: 8 total, 8 up, 8 in 2024-09-21T10:15:36.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:35 smithi070 ceph-mon[112139]: pgmap v980: 393 pgs: 393 active+clean; 2.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:36.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:35 smithi070 ceph-mon[112139]: osdmap e880: 8 total, 8 up, 8 in 2024-09-21T10:15:36.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:15:36.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:36.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:35 smithi070 ceph-mon[107385]: pgmap v980: 393 pgs: 393 active+clean; 2.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:36.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:35 smithi070 ceph-mon[107385]: osdmap e880: 8 total, 8 up, 8 in 2024-09-21T10:15:36.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:15:36.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:35 smithi160 ceph-mon[101879]: pgmap v980: 393 pgs: 393 active+clean; 2.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:35 smithi160 ceph-mon[101879]: osdmap e880: 8 total, 8 up, 8 in 2024-09-21T10:15:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:15:36.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:37.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:36 smithi070 ceph-mon[107385]: osdmap e881: 8 total, 8 up, 8 in 2024-09-21T10:15:37.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:36 smithi070 ceph-mon[107385]: osdmap e882: 8 total, 8 up, 8 in 2024-09-21T10:15:37.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:36 smithi070 ceph-mon[112139]: osdmap e881: 8 total, 8 up, 8 in 2024-09-21T10:15:37.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:36 smithi070 ceph-mon[112139]: osdmap e882: 8 total, 8 up, 8 in 2024-09-21T10:15:37.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:36 smithi160 ceph-mon[101879]: osdmap e881: 8 total, 8 up, 8 in 2024-09-21T10:15:37.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:36 smithi160 ceph-mon[101879]: osdmap e882: 8 total, 8 up, 8 in 2024-09-21T10:15:38.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:37 smithi070 ceph-mon[107385]: pgmap v983: 329 pgs: 329 active+clean; 587 KiB data, 569 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:38.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:37 smithi070 ceph-mon[107385]: osdmap e883: 8 total, 8 up, 8 in 2024-09-21T10:15:38.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:37 smithi070 ceph-mon[112139]: pgmap v983: 329 pgs: 329 active+clean; 587 KiB data, 569 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:38.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:37 smithi070 ceph-mon[112139]: osdmap e883: 8 total, 8 up, 8 in 2024-09-21T10:15:38.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:37 smithi160 ceph-mon[101879]: pgmap v983: 329 pgs: 329 active+clean; 587 KiB data, 569 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:38.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:37 smithi160 ceph-mon[101879]: osdmap e883: 8 total, 8 up, 8 in 2024-09-21T10:15:40.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:39 smithi070 ceph-mon[107385]: pgmap v986: 265 pgs: 265 active+clean; 587 KiB data, 569 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:40.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:39 smithi070 ceph-mon[107385]: osdmap e884: 8 total, 8 up, 8 in 2024-09-21T10:15:40.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:39 smithi070 ceph-mon[112139]: pgmap v986: 265 pgs: 265 active+clean; 587 KiB data, 569 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:40.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:39 smithi070 ceph-mon[112139]: osdmap e884: 8 total, 8 up, 8 in 2024-09-21T10:15:40.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:39 smithi160 ceph-mon[101879]: pgmap v986: 265 pgs: 265 active+clean; 587 KiB data, 569 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:40.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:39 smithi160 ceph-mon[101879]: osdmap e884: 8 total, 8 up, 8 in 2024-09-21T10:15:41.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:40 smithi070 ceph-mon[107385]: osdmap e885: 8 total, 8 up, 8 in 2024-09-21T10:15:41.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:40 smithi070 ceph-mon[112139]: osdmap e885: 8 total, 8 up, 8 in 2024-09-21T10:15:41.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:40 smithi160 ceph-mon[101879]: osdmap e885: 8 total, 8 up, 8 in 2024-09-21T10:15:42.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:41 smithi070 ceph-mon[107385]: pgmap v989: 201 pgs: 201 active+clean; 587 KiB data, 573 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:42.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:41 smithi070 ceph-mon[107385]: osdmap e886: 8 total, 8 up, 8 in 2024-09-21T10:15:42.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:41 smithi070 ceph-mon[107385]: osdmap e887: 8 total, 8 up, 8 in 2024-09-21T10:15:42.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:41 smithi070 ceph-mon[112139]: pgmap v989: 201 pgs: 201 active+clean; 587 KiB data, 573 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:42.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:41 smithi070 ceph-mon[112139]: osdmap e886: 8 total, 8 up, 8 in 2024-09-21T10:15:42.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:41 smithi070 ceph-mon[112139]: osdmap e887: 8 total, 8 up, 8 in 2024-09-21T10:15:42.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:41 smithi160 ceph-mon[101879]: pgmap v989: 201 pgs: 201 active+clean; 587 KiB data, 573 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:42.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:41 smithi160 ceph-mon[101879]: osdmap e886: 8 total, 8 up, 8 in 2024-09-21T10:15:42.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:41 smithi160 ceph-mon[101879]: osdmap e887: 8 total, 8 up, 8 in 2024-09-21T10:15:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:15:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:15:43] "GET /metrics HTTP/1.1" 200 55643 "" "Prometheus/2.51.0" 2024-09-21T10:15:43.856 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 170 tests from TestLibRBD (635137 ms total) 2024-09-21T10:15:43.856 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:15:43.857 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/0, where TypeParam = PlaintextUnderLUKS1 2024-09-21T10:15:43.887 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/0.Simple 2024-09-21T10:15:44.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:43 smithi070 ceph-mon[107385]: pgmap v992: 137 pgs: 137 active+clean; 586 KiB data, 578 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:44.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:43 smithi070 ceph-mon[107385]: osdmap e888: 8 total, 8 up, 8 in 2024-09-21T10:15:44.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:43 smithi070 ceph-mon[112139]: pgmap v992: 137 pgs: 137 active+clean; 586 KiB data, 578 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:44.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:43 smithi070 ceph-mon[112139]: osdmap e888: 8 total, 8 up, 8 in 2024-09-21T10:15:44.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:43 smithi160 ceph-mon[101879]: pgmap v992: 137 pgs: 137 active+clean; 586 KiB data, 578 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:44.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:43 smithi160 ceph-mon[101879]: osdmap e888: 8 total, 8 up, 8 in 2024-09-21T10:15:45.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:44 smithi070 ceph-mon[107385]: osdmap e889: 8 total, 8 up, 8 in 2024-09-21T10:15:45.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:44 smithi070 ceph-mon[112139]: osdmap e889: 8 total, 8 up, 8 in 2024-09-21T10:15:45.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:44 smithi160 ceph-mon[101879]: osdmap e889: 8 total, 8 up, 8 in 2024-09-21T10:15:45.873 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:15:46.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:45 smithi070 ceph-mon[107385]: pgmap v995: 73 pgs: 73 active+clean; 582 KiB data, 578 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:46.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:45 smithi070 ceph-mon[107385]: osdmap e890: 8 total, 8 up, 8 in 2024-09-21T10:15:46.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:45 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3992787904' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-39","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:15:46.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:45 smithi070 ceph-mon[112139]: pgmap v995: 73 pgs: 73 active+clean; 582 KiB data, 578 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:46.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:45 smithi070 ceph-mon[112139]: osdmap e890: 8 total, 8 up, 8 in 2024-09-21T10:15:46.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:45 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3992787904' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-39","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:15:46.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:45 smithi160 ceph-mon[101879]: pgmap v995: 73 pgs: 73 active+clean; 582 KiB data, 578 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:46.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:45 smithi160 ceph-mon[101879]: osdmap e890: 8 total, 8 up, 8 in 2024-09-21T10:15:46.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:45 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3992787904' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-39","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:15:47.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:46 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:15:47.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:46 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3992787904' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:15:47.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:46 smithi070 ceph-mon[112139]: osdmap e891: 8 total, 8 up, 8 in 2024-09-21T10:15:47.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:46 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:15:47.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:46 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3992787904' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:15:47.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:46 smithi070 ceph-mon[107385]: osdmap e891: 8 total, 8 up, 8 in 2024-09-21T10:15:47.233 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:46 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:15:47.233 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:46 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3992787904' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:15:47.233 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:46 smithi160 ceph-mon[101879]: osdmap e891: 8 total, 8 up, 8 in 2024-09-21T10:15:48.154 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:47 smithi160 ceph-mon[101879]: pgmap v998: 105 pgs: 13 unknown, 92 active+clean; 582 KiB data, 582 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:48.154 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:47 smithi160 ceph-mon[101879]: osdmap e892: 8 total, 8 up, 8 in 2024-09-21T10:15:48.269 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:47 smithi070 ceph-mon[107385]: pgmap v998: 105 pgs: 13 unknown, 92 active+clean; 582 KiB data, 582 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:48.269 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:47 smithi070 ceph-mon[107385]: osdmap e892: 8 total, 8 up, 8 in 2024-09-21T10:15:48.269 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:47 smithi070 ceph-mon[112139]: pgmap v998: 105 pgs: 13 unknown, 92 active+clean; 582 KiB data, 582 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:48.270 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:47 smithi070 ceph-mon[112139]: osdmap e892: 8 total, 8 up, 8 in 2024-09-21T10:15:49.165 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:48 smithi070 ceph-mon[107385]: osdmap e893: 8 total, 8 up, 8 in 2024-09-21T10:15:49.165 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:48 smithi070 ceph-mon[112139]: osdmap e893: 8 total, 8 up, 8 in 2024-09-21T10:15:49.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:48 smithi160 ceph-mon[101879]: osdmap e893: 8 total, 8 up, 8 in 2024-09-21T10:15:50.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:49 smithi160 ceph-mon[101879]: pgmap v1001: 105 pgs: 13 unknown, 92 active+clean; 582 KiB data, 582 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:50.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:49 smithi160 ceph-mon[101879]: osdmap e894: 8 total, 8 up, 8 in 2024-09-21T10:15:50.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:49 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:15:50.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:49 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:15:50.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:49 smithi070 ceph-mon[107385]: pgmap v1001: 105 pgs: 13 unknown, 92 active+clean; 582 KiB data, 582 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:50.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:49 smithi070 ceph-mon[107385]: osdmap e894: 8 total, 8 up, 8 in 2024-09-21T10:15:50.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:49 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:15:50.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:49 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:15:50.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:49 smithi070 ceph-mon[112139]: pgmap v1001: 105 pgs: 13 unknown, 92 active+clean; 582 KiB data, 582 MiB used, 715 GiB / 715 GiB avail 2024-09-21T10:15:50.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:49 smithi070 ceph-mon[112139]: osdmap e894: 8 total, 8 up, 8 in 2024-09-21T10:15:50.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:49 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:15:50.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:49 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:15:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:15:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:15:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:15:51.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:15:51.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:15:51.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:15:51.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:15:51.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:51.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:15:51.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:15:51.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:15:51.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:15:51.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:15:51.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:51.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:15:51.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:15:51.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:15:52.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:52 smithi160 ceph-mon[101879]: pgmap v1003: 105 pgs: 105 active+clean; 9.6 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 2.0 MiB/s wr, 70 op/s 2024-09-21T10:15:52.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:52 smithi070 ceph-mon[107385]: pgmap v1003: 105 pgs: 105 active+clean; 9.6 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 2.0 MiB/s wr, 70 op/s 2024-09-21T10:15:52.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:52 smithi070 ceph-mon[112139]: pgmap v1003: 105 pgs: 105 active+clean; 9.6 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 2.0 MiB/s wr, 70 op/s 2024-09-21T10:15:53.547 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:15:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:15:53] "GET /metrics HTTP/1.1" 200 36455 "" "Prometheus/2.51.0" 2024-09-21T10:15:54.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:54 smithi070 ceph-mon[107385]: pgmap v1004: 105 pgs: 105 active+clean; 26 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 4.2 MiB/s wr, 60 op/s 2024-09-21T10:15:54.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:54 smithi070 ceph-mon[112139]: pgmap v1004: 105 pgs: 105 active+clean; 26 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 4.2 MiB/s wr, 60 op/s 2024-09-21T10:15:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:54 smithi160 ceph-mon[101879]: pgmap v1004: 105 pgs: 105 active+clean; 26 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 4.2 MiB/s wr, 60 op/s 2024-09-21T10:15:56.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:56 smithi070 ceph-mon[107385]: pgmap v1005: 105 pgs: 105 active+clean; 30 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 3.9 MiB/s wr, 52 op/s 2024-09-21T10:15:56.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:56 smithi070 ceph-mon[112139]: pgmap v1005: 105 pgs: 105 active+clean; 30 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 3.9 MiB/s wr, 52 op/s 2024-09-21T10:15:56.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:56 smithi160 ceph-mon[101879]: pgmap v1005: 105 pgs: 105 active+clean; 30 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 3.9 MiB/s wr, 52 op/s 2024-09-21T10:15:56.743 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/0.Simple (12856 ms) 2024-09-21T10:15:56.744 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/0.Grow 2024-09-21T10:15:56.744 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:15:58.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:58 smithi070 ceph-mon[107385]: pgmap v1006: 105 pgs: 105 active+clean; 51 MiB data, 724 MiB used, 715 GiB / 715 GiB avail; 6.4 MiB/s rd, 5.9 MiB/s wr, 59 op/s 2024-09-21T10:15:58.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:58 smithi070 ceph-mon[107385]: osdmap e895: 8 total, 8 up, 8 in 2024-09-21T10:15:58.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:15:58 smithi070 ceph-mon[107385]: osdmap e896: 8 total, 8 up, 8 in 2024-09-21T10:15:58.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:58 smithi070 ceph-mon[112139]: pgmap v1006: 105 pgs: 105 active+clean; 51 MiB data, 724 MiB used, 715 GiB / 715 GiB avail; 6.4 MiB/s rd, 5.9 MiB/s wr, 59 op/s 2024-09-21T10:15:58.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:58 smithi070 ceph-mon[112139]: osdmap e895: 8 total, 8 up, 8 in 2024-09-21T10:15:58.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:15:58 smithi070 ceph-mon[112139]: osdmap e896: 8 total, 8 up, 8 in 2024-09-21T10:15:58.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:58 smithi160 ceph-mon[101879]: pgmap v1006: 105 pgs: 105 active+clean; 51 MiB data, 724 MiB used, 715 GiB / 715 GiB avail; 6.4 MiB/s rd, 5.9 MiB/s wr, 59 op/s 2024-09-21T10:15:58.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:58 smithi160 ceph-mon[101879]: osdmap e895: 8 total, 8 up, 8 in 2024-09-21T10:15:58.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:15:58 smithi160 ceph-mon[101879]: osdmap e896: 8 total, 8 up, 8 in 2024-09-21T10:16:00.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:00 smithi070 ceph-mon[107385]: pgmap v1009: 105 pgs: 105 active+clean; 51 MiB data, 724 MiB used, 715 GiB / 715 GiB avail; 6.7 MiB/s rd, 5.1 MiB/s wr, 22 op/s 2024-09-21T10:16:00.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:00 smithi070 ceph-mon[112139]: pgmap v1009: 105 pgs: 105 active+clean; 51 MiB data, 724 MiB used, 715 GiB / 715 GiB avail; 6.7 MiB/s rd, 5.1 MiB/s wr, 22 op/s 2024-09-21T10:16:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:00 smithi160 ceph-mon[101879]: pgmap v1009: 105 pgs: 105 active+clean; 51 MiB data, 724 MiB used, 715 GiB / 715 GiB avail; 6.7 MiB/s rd, 5.1 MiB/s wr, 22 op/s 2024-09-21T10:16:02.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:02 smithi070 ceph-mon[112139]: pgmap v1010: 105 pgs: 105 active+clean; 68 MiB data, 786 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 42 op/s 2024-09-21T10:16:02.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:02 smithi070 ceph-mon[107385]: pgmap v1010: 105 pgs: 105 active+clean; 68 MiB data, 786 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 42 op/s 2024-09-21T10:16:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:02 smithi160 ceph-mon[101879]: pgmap v1010: 105 pgs: 105 active+clean; 68 MiB data, 786 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 42 op/s 2024-09-21T10:16:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:16:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:16:03] "GET /metrics HTTP/1.1" 200 36455 "" "Prometheus/2.51.0" 2024-09-21T10:16:04.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:04 smithi070 ceph-mon[107385]: pgmap v1011: 105 pgs: 105 active+clean; 84 MiB data, 828 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 7.3 MiB/s wr, 87 op/s 2024-09-21T10:16:04.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:04 smithi070 ceph-mon[112139]: pgmap v1011: 105 pgs: 105 active+clean; 84 MiB data, 828 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 7.3 MiB/s wr, 87 op/s 2024-09-21T10:16:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:04 smithi160 ceph-mon[101879]: pgmap v1011: 105 pgs: 105 active+clean; 84 MiB data, 828 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 7.3 MiB/s wr, 87 op/s 2024-09-21T10:16:04.904 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/0.Grow (8161 ms) 2024-09-21T10:16:04.904 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/0.Shrink 2024-09-21T10:16:04.904 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:16:06.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:06 smithi070 ceph-mon[107385]: pgmap v1012: 105 pgs: 105 active+clean; 100 MiB data, 852 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.6 MiB/s wr, 76 op/s 2024-09-21T10:16:06.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:06 smithi070 ceph-mon[107385]: osdmap e897: 8 total, 8 up, 8 in 2024-09-21T10:16:06.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:06.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:06 smithi070 ceph-mon[112139]: pgmap v1012: 105 pgs: 105 active+clean; 100 MiB data, 852 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.6 MiB/s wr, 76 op/s 2024-09-21T10:16:06.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:06 smithi070 ceph-mon[112139]: osdmap e897: 8 total, 8 up, 8 in 2024-09-21T10:16:06.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:06 smithi160 ceph-mon[101879]: pgmap v1012: 105 pgs: 105 active+clean; 100 MiB data, 852 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.6 MiB/s wr, 76 op/s 2024-09-21T10:16:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:06 smithi160 ceph-mon[101879]: osdmap e897: 8 total, 8 up, 8 in 2024-09-21T10:16:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:08.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:08 smithi070 ceph-mon[112139]: pgmap v1014: 105 pgs: 105 active+clean; 122 MiB data, 920 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 9.3 MiB/s wr, 97 op/s 2024-09-21T10:16:08.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:08 smithi070 ceph-mon[107385]: pgmap v1014: 105 pgs: 105 active+clean; 122 MiB data, 920 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 9.3 MiB/s wr, 97 op/s 2024-09-21T10:16:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:08 smithi160 ceph-mon[101879]: pgmap v1014: 105 pgs: 105 active+clean; 122 MiB data, 920 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 9.3 MiB/s wr, 97 op/s 2024-09-21T10:16:10.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:10 smithi070 ceph-mon[107385]: pgmap v1015: 105 pgs: 105 active+clean; 122 MiB data, 920 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 7.9 MiB/s wr, 82 op/s 2024-09-21T10:16:10.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:10 smithi070 ceph-mon[112139]: pgmap v1015: 105 pgs: 105 active+clean; 122 MiB data, 920 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 7.9 MiB/s wr, 82 op/s 2024-09-21T10:16:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:10 smithi160 ceph-mon[101879]: pgmap v1015: 105 pgs: 105 active+clean; 122 MiB data, 920 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 7.9 MiB/s wr, 82 op/s 2024-09-21T10:16:12.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:12 smithi070 ceph-mon[107385]: pgmap v1016: 105 pgs: 105 active+clean; 134 MiB data, 974 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 7.0 MiB/s wr, 102 op/s 2024-09-21T10:16:12.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:12 smithi070 ceph-mon[112139]: pgmap v1016: 105 pgs: 105 active+clean; 134 MiB data, 974 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 7.0 MiB/s wr, 102 op/s 2024-09-21T10:16:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:12 smithi160 ceph-mon[101879]: pgmap v1016: 105 pgs: 105 active+clean; 134 MiB data, 974 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 7.0 MiB/s wr, 102 op/s 2024-09-21T10:16:12.912 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/0.Shrink (8007 ms) 2024-09-21T10:16:12.912 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/0.ShrinkToOne 2024-09-21T10:16:12.912 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:16:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:16:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:16:13] "GET /metrics HTTP/1.1" 200 36465 "" "Prometheus/2.51.0" 2024-09-21T10:16:14.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:14 smithi070 ceph-mon[112139]: pgmap v1017: 105 pgs: 105 active+clean; 159 MiB data, 1011 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 8.3 MiB/s wr, 70 op/s 2024-09-21T10:16:14.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:14 smithi070 ceph-mon[112139]: osdmap e898: 8 total, 8 up, 8 in 2024-09-21T10:16:14.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:14 smithi070 ceph-mon[107385]: pgmap v1017: 105 pgs: 105 active+clean; 159 MiB data, 1011 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 8.3 MiB/s wr, 70 op/s 2024-09-21T10:16:14.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:14 smithi070 ceph-mon[107385]: osdmap e898: 8 total, 8 up, 8 in 2024-09-21T10:16:14.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:14 smithi160 ceph-mon[101879]: pgmap v1017: 105 pgs: 105 active+clean; 159 MiB data, 1011 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 8.3 MiB/s wr, 70 op/s 2024-09-21T10:16:14.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:14 smithi160 ceph-mon[101879]: osdmap e898: 8 total, 8 up, 8 in 2024-09-21T10:16:16.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:16 smithi070 ceph-mon[107385]: pgmap v1019: 105 pgs: 105 active+clean; 171 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 8.2 MiB/s wr, 89 op/s 2024-09-21T10:16:16.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:16 smithi070 ceph-mon[112139]: pgmap v1019: 105 pgs: 105 active+clean; 171 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 8.2 MiB/s wr, 89 op/s 2024-09-21T10:16:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:16 smithi160 ceph-mon[101879]: pgmap v1019: 105 pgs: 105 active+clean; 171 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 8.2 MiB/s wr, 89 op/s 2024-09-21T10:16:18.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:18 smithi070 ceph-mon[107385]: pgmap v1020: 105 pgs: 105 active+clean; 188 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 7.0 MiB/s wr, 99 op/s 2024-09-21T10:16:18.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:18 smithi070 ceph-mon[112139]: pgmap v1020: 105 pgs: 105 active+clean; 188 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 7.0 MiB/s wr, 99 op/s 2024-09-21T10:16:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:18 smithi160 ceph-mon[101879]: pgmap v1020: 105 pgs: 105 active+clean; 188 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 7.0 MiB/s wr, 99 op/s 2024-09-21T10:16:20.393 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:20 smithi160 ceph-mon[101879]: pgmap v1021: 105 pgs: 105 active+clean; 188 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 7.0 MiB/s wr, 99 op/s 2024-09-21T10:16:20.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:20 smithi070 ceph-mon[107385]: pgmap v1021: 105 pgs: 105 active+clean; 188 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 7.0 MiB/s wr, 99 op/s 2024-09-21T10:16:20.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:20 smithi070 ceph-mon[112139]: pgmap v1021: 105 pgs: 105 active+clean; 188 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 7.0 MiB/s wr, 99 op/s 2024-09-21T10:16:20.863 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/0.ShrinkToOne (7951 ms) 2024-09-21T10:16:20.863 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/0.ShrinkToOneAfterSnapshot 2024-09-21T10:16:20.863 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:16:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:21.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:21.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:22.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:22 smithi160 ceph-mon[101879]: pgmap v1022: 105 pgs: 105 active+clean; 188 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 58 op/s 2024-09-21T10:16:22.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:22 smithi160 ceph-mon[101879]: osdmap e899: 8 total, 8 up, 8 in 2024-09-21T10:16:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:22 smithi070 ceph-mon[107385]: pgmap v1022: 105 pgs: 105 active+clean; 188 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 58 op/s 2024-09-21T10:16:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:22 smithi070 ceph-mon[107385]: osdmap e899: 8 total, 8 up, 8 in 2024-09-21T10:16:22.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:22 smithi070 ceph-mon[112139]: pgmap v1022: 105 pgs: 105 active+clean; 188 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 58 op/s 2024-09-21T10:16:22.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:22 smithi070 ceph-mon[112139]: osdmap e899: 8 total, 8 up, 8 in 2024-09-21T10:16:23.528 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:16:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:16:23] "GET /metrics HTTP/1.1" 200 36470 "" "Prometheus/2.51.0" 2024-09-21T10:16:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:24 smithi160 ceph-mon[101879]: pgmap v1024: 105 pgs: 105 active+clean; 200 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.8 MiB/s rd, 5.3 MiB/s wr, 79 op/s 2024-09-21T10:16:24.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:24 smithi070 ceph-mon[112139]: pgmap v1024: 105 pgs: 105 active+clean; 200 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.8 MiB/s rd, 5.3 MiB/s wr, 79 op/s 2024-09-21T10:16:24.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:24 smithi070 ceph-mon[107385]: pgmap v1024: 105 pgs: 105 active+clean; 200 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.8 MiB/s rd, 5.3 MiB/s wr, 79 op/s 2024-09-21T10:16:26.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:26 smithi160 ceph-mon[101879]: pgmap v1025: 105 pgs: 105 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.9 MiB/s rd, 4.5 MiB/s wr, 66 op/s 2024-09-21T10:16:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:26 smithi070 ceph-mon[107385]: pgmap v1025: 105 pgs: 105 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.9 MiB/s rd, 4.5 MiB/s wr, 66 op/s 2024-09-21T10:16:26.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:26 smithi070 ceph-mon[112139]: pgmap v1025: 105 pgs: 105 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.9 MiB/s rd, 4.5 MiB/s wr, 66 op/s 2024-09-21T10:16:28.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:28 smithi160 ceph-mon[101879]: pgmap v1026: 105 pgs: 105 active+clean; 225 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 979 KiB/s rd, 4.5 MiB/s wr, 61 op/s 2024-09-21T10:16:28.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:28 smithi160 ceph-mon[101879]: osdmap e900: 8 total, 8 up, 8 in 2024-09-21T10:16:28.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:28 smithi070 ceph-mon[107385]: pgmap v1026: 105 pgs: 105 active+clean; 225 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 979 KiB/s rd, 4.5 MiB/s wr, 61 op/s 2024-09-21T10:16:28.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:28 smithi070 ceph-mon[107385]: osdmap e900: 8 total, 8 up, 8 in 2024-09-21T10:16:28.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:28 smithi070 ceph-mon[112139]: pgmap v1026: 105 pgs: 105 active+clean; 225 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 979 KiB/s rd, 4.5 MiB/s wr, 61 op/s 2024-09-21T10:16:28.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:28 smithi070 ceph-mon[112139]: osdmap e900: 8 total, 8 up, 8 in 2024-09-21T10:16:29.431 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/0.ShrinkToOneAfterSnapshot (8568 ms) 2024-09-21T10:16:29.431 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/0.MinOverlap 2024-09-21T10:16:29.431 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:16:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:30 smithi160 ceph-mon[101879]: pgmap v1028: 105 pgs: 105 active+clean; 225 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.6 MiB/s wr, 76 op/s 2024-09-21T10:16:30.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:30 smithi070 ceph-mon[107385]: pgmap v1028: 105 pgs: 105 active+clean; 225 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.6 MiB/s wr, 76 op/s 2024-09-21T10:16:30.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:30 smithi070 ceph-mon[112139]: pgmap v1028: 105 pgs: 105 active+clean; 225 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s rd, 5.6 MiB/s wr, 76 op/s 2024-09-21T10:16:31.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:31 smithi160 ceph-mon[101879]: osdmap e901: 8 total, 8 up, 8 in 2024-09-21T10:16:31.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:31 smithi070 ceph-mon[107385]: osdmap e901: 8 total, 8 up, 8 in 2024-09-21T10:16:31.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:31 smithi070 ceph-mon[112139]: osdmap e901: 8 total, 8 up, 8 in 2024-09-21T10:16:32.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:32 smithi160 ceph-mon[101879]: pgmap v1030: 105 pgs: 105 active+clean; 249 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 6.1 MiB/s wr, 65 op/s 2024-09-21T10:16:32.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:32 smithi070 ceph-mon[107385]: pgmap v1030: 105 pgs: 105 active+clean; 249 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 6.1 MiB/s wr, 65 op/s 2024-09-21T10:16:32.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:32 smithi070 ceph-mon[112139]: pgmap v1030: 105 pgs: 105 active+clean; 249 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 6.1 MiB/s wr, 65 op/s 2024-09-21T10:16:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:16:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:16:33] "GET /metrics HTTP/1.1" 200 36470 "" "Prometheus/2.51.0" 2024-09-21T10:16:34.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:34 smithi160 ceph-mon[101879]: pgmap v1031: 105 pgs: 105 active+clean; 298 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 11 MiB/s wr, 128 op/s 2024-09-21T10:16:34.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:34 smithi070 ceph-mon[107385]: pgmap v1031: 105 pgs: 105 active+clean; 298 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 11 MiB/s wr, 128 op/s 2024-09-21T10:16:34.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:34 smithi070 ceph-mon[112139]: pgmap v1031: 105 pgs: 105 active+clean; 298 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 11 MiB/s wr, 128 op/s 2024-09-21T10:16:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:36 smithi160 ceph-mon[101879]: pgmap v1032: 105 pgs: 105 active+clean; 303 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 9.8 MiB/s wr, 87 op/s 2024-09-21T10:16:36.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:36 smithi070 ceph-mon[107385]: pgmap v1032: 105 pgs: 105 active+clean; 303 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 9.8 MiB/s wr, 87 op/s 2024-09-21T10:16:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:36.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:36 smithi070 ceph-mon[112139]: pgmap v1032: 105 pgs: 105 active+clean; 303 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 9.8 MiB/s wr, 87 op/s 2024-09-21T10:16:36.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:37.931 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/0.MinOverlap (8500 ms) 2024-09-21T10:16:37.931 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/0.ZeroOverlap 2024-09-21T10:16:37.931 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:16:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:38 smithi160 ceph-mon[101879]: pgmap v1033: 105 pgs: 105 active+clean; 311 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.3 MiB/s rd, 9.8 MiB/s wr, 98 op/s 2024-09-21T10:16:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:38 smithi070 ceph-mon[107385]: pgmap v1033: 105 pgs: 105 active+clean; 311 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.3 MiB/s rd, 9.8 MiB/s wr, 98 op/s 2024-09-21T10:16:38.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:38 smithi070 ceph-mon[112139]: pgmap v1033: 105 pgs: 105 active+clean; 311 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.3 MiB/s rd, 9.8 MiB/s wr, 98 op/s 2024-09-21T10:16:39.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:39 smithi160 ceph-mon[101879]: osdmap e902: 8 total, 8 up, 8 in 2024-09-21T10:16:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:39 smithi070 ceph-mon[107385]: osdmap e902: 8 total, 8 up, 8 in 2024-09-21T10:16:39.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:39 smithi070 ceph-mon[112139]: osdmap e902: 8 total, 8 up, 8 in 2024-09-21T10:16:40.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:40 smithi160 ceph-mon[101879]: pgmap v1035: 105 pgs: 105 active+clean; 311 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 8.1 MiB/s wr, 96 op/s 2024-09-21T10:16:40.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:40 smithi070 ceph-mon[112139]: pgmap v1035: 105 pgs: 105 active+clean; 311 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 8.1 MiB/s wr, 96 op/s 2024-09-21T10:16:40.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:40 smithi070 ceph-mon[107385]: pgmap v1035: 105 pgs: 105 active+clean; 311 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 8.1 MiB/s wr, 96 op/s 2024-09-21T10:16:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:42 smithi160 ceph-mon[101879]: pgmap v1036: 105 pgs: 105 active+clean; 339 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 9.8 MiB/s wr, 84 op/s 2024-09-21T10:16:42.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:42 smithi070 ceph-mon[107385]: pgmap v1036: 105 pgs: 105 active+clean; 339 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 9.8 MiB/s wr, 84 op/s 2024-09-21T10:16:42.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:42 smithi070 ceph-mon[112139]: pgmap v1036: 105 pgs: 105 active+clean; 339 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 9.8 MiB/s wr, 84 op/s 2024-09-21T10:16:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:16:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:16:43] "GET /metrics HTTP/1.1" 200 36471 "" "Prometheus/2.51.0" 2024-09-21T10:16:44.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:44 smithi160 ceph-mon[101879]: pgmap v1037: 105 pgs: 105 active+clean; 340 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 5.0 MiB/s wr, 60 op/s 2024-09-21T10:16:44.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:44 smithi070 ceph-mon[107385]: pgmap v1037: 105 pgs: 105 active+clean; 340 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 5.0 MiB/s wr, 60 op/s 2024-09-21T10:16:44.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:44 smithi070 ceph-mon[112139]: pgmap v1037: 105 pgs: 105 active+clean; 340 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 5.0 MiB/s wr, 60 op/s 2024-09-21T10:16:45.916 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/0.ZeroOverlap (7986 ms) 2024-09-21T10:16:46.198 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/0 (62029 ms total) 2024-09-21T10:16:46.198 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:16:46.198 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/1, where TypeParam = PlaintextUnderLUKS2 2024-09-21T10:16:46.229 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/1.Simple 2024-09-21T10:16:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:46 smithi160 ceph-mon[101879]: pgmap v1038: 105 pgs: 105 active+clean; 340 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 60 op/s 2024-09-21T10:16:46.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:46 smithi070 ceph-mon[112139]: pgmap v1038: 105 pgs: 105 active+clean; 340 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 60 op/s 2024-09-21T10:16:46.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:46 smithi070 ceph-mon[107385]: pgmap v1038: 105 pgs: 105 active+clean; 340 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 60 op/s 2024-09-21T10:16:47.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:47 smithi160 ceph-mon[101879]: osdmap e903: 8 total, 8 up, 8 in 2024-09-21T10:16:47.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:47 smithi070 ceph-mon[107385]: osdmap e903: 8 total, 8 up, 8 in 2024-09-21T10:16:47.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:47 smithi070 ceph-mon[112139]: osdmap e903: 8 total, 8 up, 8 in 2024-09-21T10:16:48.213 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:16:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:48 smithi160 ceph-mon[101879]: pgmap v1040: 73 pgs: 73 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:16:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:48 smithi160 ceph-mon[101879]: osdmap e904: 8 total, 8 up, 8 in 2024-09-21T10:16:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:48 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3720991639' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-40","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:16:48.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:48 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-40","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:16:48.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:48 smithi070 ceph-mon[112139]: pgmap v1040: 73 pgs: 73 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:16:48.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:48 smithi070 ceph-mon[112139]: osdmap e904: 8 total, 8 up, 8 in 2024-09-21T10:16:48.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:48 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3720991639' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-40","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:16:48.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:48 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-40","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:16:48.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:48 smithi070 ceph-mon[107385]: pgmap v1040: 73 pgs: 73 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:16:48.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:48 smithi070 ceph-mon[107385]: osdmap e904: 8 total, 8 up, 8 in 2024-09-21T10:16:48.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:48 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3720991639' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-40","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:16:48.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:48 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-40","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:16:49.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:49 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:16:49.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:49 smithi160 ceph-mon[101879]: osdmap e905: 8 total, 8 up, 8 in 2024-09-21T10:16:49.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:49 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:16:49.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:49 smithi070 ceph-mon[107385]: osdmap e905: 8 total, 8 up, 8 in 2024-09-21T10:16:49.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:49 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:16:49.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:49 smithi070 ceph-mon[112139]: osdmap e905: 8 total, 8 up, 8 in 2024-09-21T10:16:50.492 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:50 smithi160 ceph-mon[101879]: pgmap v1043: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:16:50.493 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:50 smithi160 ceph-mon[101879]: osdmap e906: 8 total, 8 up, 8 in 2024-09-21T10:16:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:50 smithi070 ceph-mon[107385]: pgmap v1043: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:16:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:50 smithi070 ceph-mon[107385]: osdmap e906: 8 total, 8 up, 8 in 2024-09-21T10:16:50.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:50 smithi070 ceph-mon[112139]: pgmap v1043: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:16:50.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:50 smithi070 ceph-mon[112139]: osdmap e906: 8 total, 8 up, 8 in 2024-09-21T10:16:51.336 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:51 smithi160 ceph-mon[101879]: osdmap e907: 8 total, 8 up, 8 in 2024-09-21T10:16:51.337 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:51.337 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:16:51.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:51 smithi070 ceph-mon[112139]: osdmap e907: 8 total, 8 up, 8 in 2024-09-21T10:16:51.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:51.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:16:51.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:51 smithi070 ceph-mon[107385]: osdmap e907: 8 total, 8 up, 8 in 2024-09-21T10:16:51.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:51.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:16:52.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:52 smithi160 ceph-mon[101879]: pgmap v1046: 105 pgs: 16 unknown, 89 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:16:52.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:52 smithi160 ceph-mon[101879]: osdmap e908: 8 total, 8 up, 8 in 2024-09-21T10:16:52.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:52 smithi070 ceph-mon[107385]: pgmap v1046: 105 pgs: 16 unknown, 89 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:16:52.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:52 smithi070 ceph-mon[107385]: osdmap e908: 8 total, 8 up, 8 in 2024-09-21T10:16:52.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:52 smithi070 ceph-mon[112139]: pgmap v1046: 105 pgs: 16 unknown, 89 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:16:52.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:52 smithi070 ceph-mon[112139]: osdmap e908: 8 total, 8 up, 8 in 2024-09-21T10:16:53.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:16:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:16:53] "GET /metrics HTTP/1.1" 200 36407 "" "Prometheus/2.51.0" 2024-09-21T10:16:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:54 smithi160 ceph-mon[101879]: pgmap v1048: 105 pgs: 105 active+clean; 17 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 9.7 KiB/s rd, 3.9 MiB/s wr, 14 op/s 2024-09-21T10:16:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:54 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:16:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:54 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:16:54.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:54 smithi070 ceph-mon[112139]: pgmap v1048: 105 pgs: 105 active+clean; 17 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 9.7 KiB/s rd, 3.9 MiB/s wr, 14 op/s 2024-09-21T10:16:54.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:54 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:16:54.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:54 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:16:54.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:54 smithi070 ceph-mon[107385]: pgmap v1048: 105 pgs: 105 active+clean; 17 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 9.7 KiB/s rd, 3.9 MiB/s wr, 14 op/s 2024-09-21T10:16:54.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:54 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:16:54.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:54 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:16:56.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:55 smithi160 ceph-mon[101879]: pgmap v1049: 105 pgs: 105 active+clean; 26 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.2 MiB/s wr, 11 op/s 2024-09-21T10:16:56.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:55 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:16:56.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:55 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:16:56.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:55 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:16:56.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:55 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:16:56.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:55 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:16:56.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:55 smithi070 ceph-mon[107385]: pgmap v1049: 105 pgs: 105 active+clean; 26 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.2 MiB/s wr, 11 op/s 2024-09-21T10:16:56.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:55 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:16:56.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:55 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:16:56.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:55 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:16:56.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:55 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:16:56.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:55 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:16:56.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:55 smithi070 ceph-mon[112139]: pgmap v1049: 105 pgs: 105 active+clean; 26 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.2 MiB/s wr, 11 op/s 2024-09-21T10:16:56.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:55 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:16:56.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:55 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:16:56.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:55 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:16:56.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:55 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:16:56.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:55 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:16:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:57 smithi160 ceph-mon[101879]: pgmap v1050: 105 pgs: 105 active+clean; 26 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 3.5 MiB/s wr, 52 op/s 2024-09-21T10:16:58.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:57 smithi070 ceph-mon[107385]: pgmap v1050: 105 pgs: 105 active+clean; 26 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 3.5 MiB/s wr, 52 op/s 2024-09-21T10:16:58.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:57 smithi070 ceph-mon[112139]: pgmap v1050: 105 pgs: 105 active+clean; 26 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 3.5 MiB/s wr, 52 op/s 2024-09-21T10:16:59.130 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:58 smithi070 ceph-mon[107385]: osdmap e909: 8 total, 8 up, 8 in 2024-09-21T10:16:59.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:58 smithi070 ceph-mon[112139]: osdmap e909: 8 total, 8 up, 8 in 2024-09-21T10:16:59.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:58 smithi160 ceph-mon[101879]: osdmap e909: 8 total, 8 up, 8 in 2024-09-21T10:17:00.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:16:59 smithi070 ceph-mon[107385]: pgmap v1052: 105 pgs: 105 active+clean; 26 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 3.1 MiB/s wr, 46 op/s 2024-09-21T10:17:00.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:16:59 smithi070 ceph-mon[112139]: pgmap v1052: 105 pgs: 105 active+clean; 26 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 3.1 MiB/s wr, 46 op/s 2024-09-21T10:17:00.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:16:59 smithi160 ceph-mon[101879]: pgmap v1052: 105 pgs: 105 active+clean; 26 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 3.1 MiB/s wr, 46 op/s 2024-09-21T10:17:01.422 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/1.Simple (15193 ms) 2024-09-21T10:17:01.423 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/1.Grow 2024-09-21T10:17:01.423 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:17:02.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:01 smithi070 ceph-mon[107385]: pgmap v1053: 105 pgs: 105 active+clean; 42 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 508 KiB/s rd, 3.2 MiB/s wr, 35 op/s 2024-09-21T10:17:02.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:01 smithi070 ceph-mon[112139]: pgmap v1053: 105 pgs: 105 active+clean; 42 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 508 KiB/s rd, 3.2 MiB/s wr, 35 op/s 2024-09-21T10:17:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:01 smithi160 ceph-mon[101879]: pgmap v1053: 105 pgs: 105 active+clean; 42 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 508 KiB/s rd, 3.2 MiB/s wr, 35 op/s 2024-09-21T10:17:03.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:02 smithi070 ceph-mon[107385]: osdmap e910: 8 total, 8 up, 8 in 2024-09-21T10:17:03.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:02 smithi070 ceph-mon[112139]: osdmap e910: 8 total, 8 up, 8 in 2024-09-21T10:17:03.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:02 smithi160 ceph-mon[101879]: osdmap e910: 8 total, 8 up, 8 in 2024-09-21T10:17:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:17:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:17:03] "GET /metrics HTTP/1.1" 200 36407 "" "Prometheus/2.51.0" 2024-09-21T10:17:04.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:03 smithi070 ceph-mon[107385]: pgmap v1055: 105 pgs: 105 active+clean; 62 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 5.0 MiB/s wr, 70 op/s 2024-09-21T10:17:04.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:03 smithi070 ceph-mon[112139]: pgmap v1055: 105 pgs: 105 active+clean; 62 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 5.0 MiB/s wr, 70 op/s 2024-09-21T10:17:04.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:03 smithi160 ceph-mon[101879]: pgmap v1055: 105 pgs: 105 active+clean; 62 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 5.0 MiB/s wr, 70 op/s 2024-09-21T10:17:06.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:05 smithi070 ceph-mon[112139]: pgmap v1056: 105 pgs: 105 active+clean; 62 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.5 MiB/s wr, 44 op/s 2024-09-21T10:17:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:06.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:05 smithi070 ceph-mon[107385]: pgmap v1056: 105 pgs: 105 active+clean; 62 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.5 MiB/s wr, 44 op/s 2024-09-21T10:17:06.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:05 smithi160 ceph-mon[101879]: pgmap v1056: 105 pgs: 105 active+clean; 62 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.5 MiB/s wr, 44 op/s 2024-09-21T10:17:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:08.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:07 smithi070 ceph-mon[107385]: pgmap v1057: 105 pgs: 105 active+clean; 92 MiB data, 867 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 9.7 MiB/s wr, 78 op/s 2024-09-21T10:17:08.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:07 smithi070 ceph-mon[112139]: pgmap v1057: 105 pgs: 105 active+clean; 92 MiB data, 867 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 9.7 MiB/s wr, 78 op/s 2024-09-21T10:17:08.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:07 smithi160 ceph-mon[101879]: pgmap v1057: 105 pgs: 105 active+clean; 92 MiB data, 867 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 9.7 MiB/s wr, 78 op/s 2024-09-21T10:17:10.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:09 smithi070 ceph-mon[107385]: pgmap v1058: 105 pgs: 105 active+clean; 92 MiB data, 867 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 66 op/s 2024-09-21T10:17:10.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:09 smithi070 ceph-mon[112139]: pgmap v1058: 105 pgs: 105 active+clean; 92 MiB data, 867 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 66 op/s 2024-09-21T10:17:10.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:09 smithi160 ceph-mon[101879]: pgmap v1058: 105 pgs: 105 active+clean; 92 MiB data, 867 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 66 op/s 2024-09-21T10:17:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:11 smithi070 ceph-mon[107385]: pgmap v1059: 105 pgs: 105 active+clean; 96 MiB data, 867 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.6 MiB/s wr, 71 op/s 2024-09-21T10:17:12.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:11 smithi070 ceph-mon[112139]: pgmap v1059: 105 pgs: 105 active+clean; 96 MiB data, 867 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.6 MiB/s wr, 71 op/s 2024-09-21T10:17:12.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:11 smithi160 ceph-mon[101879]: pgmap v1059: 105 pgs: 105 active+clean; 96 MiB data, 867 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.6 MiB/s wr, 71 op/s 2024-09-21T10:17:13.434 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/1.Grow (12012 ms) 2024-09-21T10:17:13.434 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/1.Shrink 2024-09-21T10:17:13.435 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:17:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:17:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:17:13] "GET /metrics HTTP/1.1" 200 36461 "" "Prometheus/2.51.0" 2024-09-21T10:17:14.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:13 smithi070 ceph-mon[107385]: pgmap v1060: 105 pgs: 105 active+clean; 104 MiB data, 899 MiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 5.2 MiB/s wr, 48 op/s 2024-09-21T10:17:14.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:13 smithi070 ceph-mon[112139]: pgmap v1060: 105 pgs: 105 active+clean; 104 MiB data, 899 MiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 5.2 MiB/s wr, 48 op/s 2024-09-21T10:17:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:13 smithi160 ceph-mon[101879]: pgmap v1060: 105 pgs: 105 active+clean; 104 MiB data, 899 MiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 5.2 MiB/s wr, 48 op/s 2024-09-21T10:17:15.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:14 smithi160 ceph-mon[101879]: osdmap e911: 8 total, 8 up, 8 in 2024-09-21T10:17:15.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:14 smithi070 ceph-mon[107385]: osdmap e911: 8 total, 8 up, 8 in 2024-09-21T10:17:15.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:14 smithi070 ceph-mon[112139]: osdmap e911: 8 total, 8 up, 8 in 2024-09-21T10:17:16.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:15 smithi160 ceph-mon[101879]: pgmap v1062: 105 pgs: 105 active+clean; 116 MiB data, 928 MiB used, 714 GiB / 715 GiB avail; 4.3 MiB/s rd, 5.8 MiB/s wr, 43 op/s 2024-09-21T10:17:16.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:15 smithi070 ceph-mon[107385]: pgmap v1062: 105 pgs: 105 active+clean; 116 MiB data, 928 MiB used, 714 GiB / 715 GiB avail; 4.3 MiB/s rd, 5.8 MiB/s wr, 43 op/s 2024-09-21T10:17:16.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:15 smithi070 ceph-mon[112139]: pgmap v1062: 105 pgs: 105 active+clean; 116 MiB data, 928 MiB used, 714 GiB / 715 GiB avail; 4.3 MiB/s rd, 5.8 MiB/s wr, 43 op/s 2024-09-21T10:17:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:17 smithi160 ceph-mon[101879]: pgmap v1063: 105 pgs: 105 active+clean; 158 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 68 op/s 2024-09-21T10:17:18.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:17 smithi070 ceph-mon[107385]: pgmap v1063: 105 pgs: 105 active+clean; 158 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 68 op/s 2024-09-21T10:17:18.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:17 smithi070 ceph-mon[112139]: pgmap v1063: 105 pgs: 105 active+clean; 158 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 68 op/s 2024-09-21T10:17:20.144 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:19 smithi160 ceph-mon[101879]: pgmap v1064: 105 pgs: 105 active+clean; 158 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 68 op/s 2024-09-21T10:17:20.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:19 smithi070 ceph-mon[107385]: pgmap v1064: 105 pgs: 105 active+clean; 158 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 68 op/s 2024-09-21T10:17:20.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:19 smithi070 ceph-mon[112139]: pgmap v1064: 105 pgs: 105 active+clean; 158 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 68 op/s 2024-09-21T10:17:21.156 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:21.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:21.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:22.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:21 smithi160 ceph-mon[101879]: pgmap v1065: 105 pgs: 105 active+clean; 158 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.8 MiB/s wr, 62 op/s 2024-09-21T10:17:22.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:21 smithi070 ceph-mon[107385]: pgmap v1065: 105 pgs: 105 active+clean; 158 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.8 MiB/s wr, 62 op/s 2024-09-21T10:17:22.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:21 smithi070 ceph-mon[112139]: pgmap v1065: 105 pgs: 105 active+clean; 158 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.8 MiB/s wr, 62 op/s 2024-09-21T10:17:23.579 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:17:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:17:23] "GET /metrics HTTP/1.1" 200 36469 "" "Prometheus/2.51.0" 2024-09-21T10:17:24.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:23 smithi160 ceph-mon[101879]: pgmap v1066: 105 pgs: 105 active+clean; 158 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 57 op/s 2024-09-21T10:17:24.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:23 smithi070 ceph-mon[107385]: pgmap v1066: 105 pgs: 105 active+clean; 158 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 57 op/s 2024-09-21T10:17:24.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:23 smithi070 ceph-mon[112139]: pgmap v1066: 105 pgs: 105 active+clean; 158 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 57 op/s 2024-09-21T10:17:25.787 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/1.Shrink (12352 ms) 2024-09-21T10:17:25.787 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/1.ShrinkToOne 2024-09-21T10:17:25.787 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:17:26.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:25 smithi160 ceph-mon[101879]: pgmap v1067: 105 pgs: 105 active+clean; 166 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 6.3 MiB/s wr, 64 op/s 2024-09-21T10:17:26.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:25 smithi070 ceph-mon[107385]: pgmap v1067: 105 pgs: 105 active+clean; 166 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 6.3 MiB/s wr, 64 op/s 2024-09-21T10:17:26.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:25 smithi070 ceph-mon[112139]: pgmap v1067: 105 pgs: 105 active+clean; 166 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 6.3 MiB/s wr, 64 op/s 2024-09-21T10:17:27.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:26 smithi160 ceph-mon[101879]: osdmap e912: 8 total, 8 up, 8 in 2024-09-21T10:17:27.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:26 smithi070 ceph-mon[107385]: osdmap e912: 8 total, 8 up, 8 in 2024-09-21T10:17:27.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:26 smithi070 ceph-mon[112139]: osdmap e912: 8 total, 8 up, 8 in 2024-09-21T10:17:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:27 smithi160 ceph-mon[101879]: pgmap v1069: 105 pgs: 105 active+clean; 207 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.5 MiB/s wr, 60 op/s 2024-09-21T10:17:28.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:27 smithi070 ceph-mon[107385]: pgmap v1069: 105 pgs: 105 active+clean; 207 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.5 MiB/s wr, 60 op/s 2024-09-21T10:17:28.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:27 smithi070 ceph-mon[112139]: pgmap v1069: 105 pgs: 105 active+clean; 207 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.5 MiB/s wr, 60 op/s 2024-09-21T10:17:30.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:29 smithi160 ceph-mon[101879]: pgmap v1070: 105 pgs: 105 active+clean; 207 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.5 MiB/s wr, 60 op/s 2024-09-21T10:17:30.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:29 smithi070 ceph-mon[107385]: pgmap v1070: 105 pgs: 105 active+clean; 207 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.5 MiB/s wr, 60 op/s 2024-09-21T10:17:30.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:29 smithi070 ceph-mon[112139]: pgmap v1070: 105 pgs: 105 active+clean; 207 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.5 MiB/s wr, 60 op/s 2024-09-21T10:17:32.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:31 smithi160 ceph-mon[101879]: pgmap v1071: 105 pgs: 105 active+clean; 224 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 69 op/s 2024-09-21T10:17:32.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:31 smithi070 ceph-mon[107385]: pgmap v1071: 105 pgs: 105 active+clean; 224 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 69 op/s 2024-09-21T10:17:32.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:31 smithi070 ceph-mon[112139]: pgmap v1071: 105 pgs: 105 active+clean; 224 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 69 op/s 2024-09-21T10:17:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:17:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:17:33] "GET /metrics HTTP/1.1" 200 36469 "" "Prometheus/2.51.0" 2024-09-21T10:17:34.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:33 smithi160 ceph-mon[101879]: pgmap v1072: 105 pgs: 105 active+clean; 224 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 69 op/s 2024-09-21T10:17:34.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:33 smithi070 ceph-mon[107385]: pgmap v1072: 105 pgs: 105 active+clean; 224 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 69 op/s 2024-09-21T10:17:34.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:33 smithi070 ceph-mon[112139]: pgmap v1072: 105 pgs: 105 active+clean; 224 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 69 op/s 2024-09-21T10:17:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:35 smithi160 ceph-mon[101879]: pgmap v1073: 105 pgs: 105 active+clean; 224 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 7.0 MiB/s wr, 60 op/s 2024-09-21T10:17:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:36.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:35 smithi070 ceph-mon[107385]: pgmap v1073: 105 pgs: 105 active+clean; 224 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 7.0 MiB/s wr, 60 op/s 2024-09-21T10:17:36.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:36.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:35 smithi070 ceph-mon[112139]: pgmap v1073: 105 pgs: 105 active+clean; 224 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 7.0 MiB/s wr, 60 op/s 2024-09-21T10:17:36.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:37.452 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/1.ShrinkToOne (11665 ms) 2024-09-21T10:17:37.452 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/1.ShrinkToOneAfterSnapshot 2024-09-21T10:17:37.452 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:17:38.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:37 smithi160 ceph-mon[101879]: pgmap v1074: 105 pgs: 105 active+clean; 240 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 8.6 MiB/s wr, 65 op/s 2024-09-21T10:17:38.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:37 smithi070 ceph-mon[107385]: pgmap v1074: 105 pgs: 105 active+clean; 240 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 8.6 MiB/s wr, 65 op/s 2024-09-21T10:17:38.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:37 smithi070 ceph-mon[112139]: pgmap v1074: 105 pgs: 105 active+clean; 240 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 8.6 MiB/s wr, 65 op/s 2024-09-21T10:17:39.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:38 smithi160 ceph-mon[101879]: osdmap e913: 8 total, 8 up, 8 in 2024-09-21T10:17:39.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:38 smithi070 ceph-mon[107385]: osdmap e913: 8 total, 8 up, 8 in 2024-09-21T10:17:39.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:38 smithi070 ceph-mon[112139]: osdmap e913: 8 total, 8 up, 8 in 2024-09-21T10:17:40.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:39 smithi160 ceph-mon[101879]: pgmap v1076: 105 pgs: 105 active+clean; 240 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 22 op/s 2024-09-21T10:17:40.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:39 smithi070 ceph-mon[107385]: pgmap v1076: 105 pgs: 105 active+clean; 240 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 22 op/s 2024-09-21T10:17:40.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:39 smithi070 ceph-mon[112139]: pgmap v1076: 105 pgs: 105 active+clean; 240 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 22 op/s 2024-09-21T10:17:42.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:41 smithi160 ceph-mon[101879]: pgmap v1077: 105 pgs: 105 active+clean; 264 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.0 MiB/s wr, 52 op/s 2024-09-21T10:17:42.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:41 smithi070 ceph-mon[107385]: pgmap v1077: 105 pgs: 105 active+clean; 264 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.0 MiB/s wr, 52 op/s 2024-09-21T10:17:42.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:41 smithi070 ceph-mon[112139]: pgmap v1077: 105 pgs: 105 active+clean; 264 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.0 MiB/s wr, 52 op/s 2024-09-21T10:17:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:17:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:17:43] "GET /metrics HTTP/1.1" 200 36470 "" "Prometheus/2.51.0" 2024-09-21T10:17:44.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:43 smithi160 ceph-mon[101879]: pgmap v1078: 105 pgs: 105 active+clean; 269 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 60 op/s 2024-09-21T10:17:44.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:43 smithi070 ceph-mon[107385]: pgmap v1078: 105 pgs: 105 active+clean; 269 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 60 op/s 2024-09-21T10:17:44.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:43 smithi070 ceph-mon[112139]: pgmap v1078: 105 pgs: 105 active+clean; 269 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 60 op/s 2024-09-21T10:17:46.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:45 smithi160 ceph-mon[101879]: pgmap v1079: 105 pgs: 105 active+clean; 269 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 57 op/s 2024-09-21T10:17:46.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:45 smithi070 ceph-mon[112139]: pgmap v1079: 105 pgs: 105 active+clean; 269 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 57 op/s 2024-09-21T10:17:46.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:45 smithi070 ceph-mon[107385]: pgmap v1079: 105 pgs: 105 active+clean; 269 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 57 op/s 2024-09-21T10:17:48.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:47 smithi160 ceph-mon[101879]: pgmap v1080: 105 pgs: 105 active+clean; 273 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 916 KiB/s rd, 4.5 MiB/s wr, 52 op/s 2024-09-21T10:17:48.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:47 smithi070 ceph-mon[107385]: pgmap v1080: 105 pgs: 105 active+clean; 273 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 916 KiB/s rd, 4.5 MiB/s wr, 52 op/s 2024-09-21T10:17:48.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:47 smithi070 ceph-mon[112139]: pgmap v1080: 105 pgs: 105 active+clean; 273 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 916 KiB/s rd, 4.5 MiB/s wr, 52 op/s 2024-09-21T10:17:49.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:48 smithi160 ceph-mon[101879]: osdmap e914: 8 total, 8 up, 8 in 2024-09-21T10:17:49.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:48 smithi070 ceph-mon[107385]: osdmap e914: 8 total, 8 up, 8 in 2024-09-21T10:17:49.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:48 smithi070 ceph-mon[112139]: osdmap e914: 8 total, 8 up, 8 in 2024-09-21T10:17:50.191 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/1.ShrinkToOneAfterSnapshot (12739 ms) 2024-09-21T10:17:50.191 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/1.MinOverlap 2024-09-21T10:17:50.191 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:17:50.242 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:49 smithi160 ceph-mon[101879]: pgmap v1082: 105 pgs: 105 active+clean; 273 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 916 KiB/s rd, 4.5 MiB/s wr, 52 op/s 2024-09-21T10:17:50.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:49 smithi070 ceph-mon[107385]: pgmap v1082: 105 pgs: 105 active+clean; 273 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 916 KiB/s rd, 4.5 MiB/s wr, 52 op/s 2024-09-21T10:17:50.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:49 smithi070 ceph-mon[112139]: pgmap v1082: 105 pgs: 105 active+clean; 273 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 916 KiB/s rd, 4.5 MiB/s wr, 52 op/s 2024-09-21T10:17:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:51.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:51.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:52.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:51 smithi160 ceph-mon[101879]: pgmap v1083: 105 pgs: 105 active+clean; 334 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s rd, 7.0 MiB/s wr, 27 op/s 2024-09-21T10:17:52.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:51 smithi160 ceph-mon[101879]: osdmap e915: 8 total, 8 up, 8 in 2024-09-21T10:17:52.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:51 smithi070 ceph-mon[107385]: pgmap v1083: 105 pgs: 105 active+clean; 334 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s rd, 7.0 MiB/s wr, 27 op/s 2024-09-21T10:17:52.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:51 smithi070 ceph-mon[107385]: osdmap e915: 8 total, 8 up, 8 in 2024-09-21T10:17:52.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:51 smithi070 ceph-mon[112139]: pgmap v1083: 105 pgs: 105 active+clean; 334 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s rd, 7.0 MiB/s wr, 27 op/s 2024-09-21T10:17:52.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:51 smithi070 ceph-mon[112139]: osdmap e915: 8 total, 8 up, 8 in 2024-09-21T10:17:53.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:17:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:17:53] "GET /metrics HTTP/1.1" 200 36470 "" "Prometheus/2.51.0" 2024-09-21T10:17:54.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:53 smithi160 ceph-mon[101879]: pgmap v1085: 105 pgs: 105 active+clean; 362 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 12 MiB/s wr, 60 op/s 2024-09-21T10:17:54.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:53 smithi070 ceph-mon[107385]: pgmap v1085: 105 pgs: 105 active+clean; 362 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 12 MiB/s wr, 60 op/s 2024-09-21T10:17:54.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:53 smithi070 ceph-mon[112139]: pgmap v1085: 105 pgs: 105 active+clean; 362 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 12 MiB/s wr, 60 op/s 2024-09-21T10:17:56.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:55 smithi070 ceph-mon[112139]: pgmap v1086: 105 pgs: 105 active+clean; 370 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 12 MiB/s wr, 59 op/s 2024-09-21T10:17:56.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:55 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:17:56.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:55 smithi070 ceph-mon[107385]: pgmap v1086: 105 pgs: 105 active+clean; 370 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 12 MiB/s wr, 59 op/s 2024-09-21T10:17:56.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:55 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:17:56.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:55 smithi160 ceph-mon[101879]: pgmap v1086: 105 pgs: 105 active+clean; 370 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 12 MiB/s wr, 59 op/s 2024-09-21T10:17:56.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:55 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:17:58.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:57 smithi160 ceph-mon[101879]: pgmap v1087: 105 pgs: 105 active+clean; 379 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.0 MiB/s rd, 13 MiB/s wr, 89 op/s 2024-09-21T10:17:58.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:57 smithi070 ceph-mon[107385]: pgmap v1087: 105 pgs: 105 active+clean; 379 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.0 MiB/s rd, 13 MiB/s wr, 89 op/s 2024-09-21T10:17:58.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:57 smithi070 ceph-mon[112139]: pgmap v1087: 105 pgs: 105 active+clean; 379 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.0 MiB/s rd, 13 MiB/s wr, 89 op/s 2024-09-21T10:18:00.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:59 smithi160 ceph-mon[101879]: pgmap v1088: 105 pgs: 105 active+clean; 379 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 11 MiB/s wr, 74 op/s 2024-09-21T10:18:00.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:59 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:18:00.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:59 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:18:00.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:59 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:18:00.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:59 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:18:00.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:59 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:18:00.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:59 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:18:00.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:17:59 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:18:00.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:59 smithi070 ceph-mon[107385]: pgmap v1088: 105 pgs: 105 active+clean; 379 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 11 MiB/s wr, 74 op/s 2024-09-21T10:18:00.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:18:00.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:18:00.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:18:00.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:18:00.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:18:00.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:18:00.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:17:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:18:00.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:59 smithi070 ceph-mon[112139]: pgmap v1088: 105 pgs: 105 active+clean; 379 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 11 MiB/s wr, 74 op/s 2024-09-21T10:18:00.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:59 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:18:00.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:59 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:18:00.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:59 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:18:00.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:59 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:18:00.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:59 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:18:00.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:59 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:18:00.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:17:59 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:18:01.378 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/1.MinOverlap (11188 ms) 2024-09-21T10:18:01.378 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/1.ZeroOverlap 2024-09-21T10:18:01.378 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:18:02.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:01 smithi070 ceph-mon[107385]: pgmap v1089: 105 pgs: 105 active+clean; 391 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 5.7 MiB/s wr, 69 op/s 2024-09-21T10:18:02.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:01 smithi070 ceph-mon[112139]: pgmap v1089: 105 pgs: 105 active+clean; 391 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 5.7 MiB/s wr, 69 op/s 2024-09-21T10:18:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:01 smithi160 ceph-mon[101879]: pgmap v1089: 105 pgs: 105 active+clean; 391 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 5.7 MiB/s wr, 69 op/s 2024-09-21T10:18:03.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:02 smithi070 ceph-mon[107385]: osdmap e916: 8 total, 8 up, 8 in 2024-09-21T10:18:03.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:02 smithi070 ceph-mon[112139]: osdmap e916: 8 total, 8 up, 8 in 2024-09-21T10:18:03.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:02 smithi160 ceph-mon[101879]: osdmap e916: 8 total, 8 up, 8 in 2024-09-21T10:18:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:18:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:18:03] "GET /metrics HTTP/1.1" 200 36470 "" "Prometheus/2.51.0" 2024-09-21T10:18:04.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:03 smithi070 ceph-mon[107385]: pgmap v1091: 105 pgs: 105 active+clean; 399 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 49 op/s 2024-09-21T10:18:04.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:03 smithi070 ceph-mon[112139]: pgmap v1091: 105 pgs: 105 active+clean; 399 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 49 op/s 2024-09-21T10:18:04.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:03 smithi160 ceph-mon[101879]: pgmap v1091: 105 pgs: 105 active+clean; 399 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 49 op/s 2024-09-21T10:18:06.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:05 smithi070 ceph-mon[107385]: pgmap v1092: 105 pgs: 105 active+clean; 403 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 52 op/s 2024-09-21T10:18:06.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:05 smithi070 ceph-mon[112139]: pgmap v1092: 105 pgs: 105 active+clean; 403 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 52 op/s 2024-09-21T10:18:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:05 smithi160 ceph-mon[101879]: pgmap v1092: 105 pgs: 105 active+clean; 403 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 52 op/s 2024-09-21T10:18:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:08.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:07 smithi070 ceph-mon[107385]: pgmap v1093: 105 pgs: 105 active+clean; 424 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 6.5 MiB/s wr, 62 op/s 2024-09-21T10:18:08.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:07 smithi070 ceph-mon[112139]: pgmap v1093: 105 pgs: 105 active+clean; 424 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 6.5 MiB/s wr, 62 op/s 2024-09-21T10:18:08.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:07 smithi160 ceph-mon[101879]: pgmap v1093: 105 pgs: 105 active+clean; 424 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 6.5 MiB/s wr, 62 op/s 2024-09-21T10:18:10.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:09 smithi070 ceph-mon[107385]: pgmap v1094: 105 pgs: 105 active+clean; 424 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 6.5 MiB/s wr, 62 op/s 2024-09-21T10:18:10.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:09 smithi070 ceph-mon[112139]: pgmap v1094: 105 pgs: 105 active+clean; 424 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 6.5 MiB/s wr, 62 op/s 2024-09-21T10:18:10.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:09 smithi160 ceph-mon[101879]: pgmap v1094: 105 pgs: 105 active+clean; 424 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 6.5 MiB/s wr, 62 op/s 2024-09-21T10:18:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:11 smithi070 ceph-mon[107385]: pgmap v1095: 105 pgs: 105 active+clean; 428 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 5.7 MiB/s wr, 55 op/s 2024-09-21T10:18:12.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:11 smithi070 ceph-mon[112139]: pgmap v1095: 105 pgs: 105 active+clean; 428 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 5.7 MiB/s wr, 55 op/s 2024-09-21T10:18:12.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:11 smithi160 ceph-mon[101879]: pgmap v1095: 105 pgs: 105 active+clean; 428 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 5.7 MiB/s wr, 55 op/s 2024-09-21T10:18:13.304 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/1.ZeroOverlap (11926 ms) 2024-09-21T10:18:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:18:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:18:13] "GET /metrics HTTP/1.1" 200 36474 "" "Prometheus/2.51.0" 2024-09-21T10:18:13.883 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/1 (87075 ms total) 2024-09-21T10:18:13.883 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:18:13.884 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/2, where TypeParam = UnformattedLUKS1 2024-09-21T10:18:13.912 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/2.Simple 2024-09-21T10:18:14.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:13 smithi070 ceph-mon[107385]: pgmap v1096: 105 pgs: 105 active+clean; 432 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 51 op/s 2024-09-21T10:18:14.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:13 smithi070 ceph-mon[112139]: pgmap v1096: 105 pgs: 105 active+clean; 432 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 51 op/s 2024-09-21T10:18:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:13 smithi160 ceph-mon[101879]: pgmap v1096: 105 pgs: 105 active+clean; 432 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 51 op/s 2024-09-21T10:18:15.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:14 smithi160 ceph-mon[101879]: osdmap e917: 8 total, 8 up, 8 in 2024-09-21T10:18:15.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:14 smithi070 ceph-mon[107385]: osdmap e917: 8 total, 8 up, 8 in 2024-09-21T10:18:15.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:14 smithi070 ceph-mon[112139]: osdmap e917: 8 total, 8 up, 8 in 2024-09-21T10:18:15.893 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:18:16.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:15 smithi160 ceph-mon[101879]: pgmap v1098: 73 pgs: 73 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:18:16.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:15 smithi160 ceph-mon[101879]: osdmap e918: 8 total, 8 up, 8 in 2024-09-21T10:18:16.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:15 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2125232960' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-41","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:18:16.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:15 smithi070 ceph-mon[107385]: pgmap v1098: 73 pgs: 73 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:18:16.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:15 smithi070 ceph-mon[107385]: osdmap e918: 8 total, 8 up, 8 in 2024-09-21T10:18:16.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:15 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2125232960' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-41","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:18:16.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:15 smithi070 ceph-mon[112139]: pgmap v1098: 73 pgs: 73 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:18:16.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:15 smithi070 ceph-mon[112139]: osdmap e918: 8 total, 8 up, 8 in 2024-09-21T10:18:16.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:15 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2125232960' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-41","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:18:17.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:16 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2125232960' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:18:17.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:16 smithi160 ceph-mon[101879]: osdmap e919: 8 total, 8 up, 8 in 2024-09-21T10:18:17.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:16 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2125232960' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:18:17.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:16 smithi070 ceph-mon[107385]: osdmap e919: 8 total, 8 up, 8 in 2024-09-21T10:18:17.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:16 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2125232960' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:18:17.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:16 smithi070 ceph-mon[112139]: osdmap e919: 8 total, 8 up, 8 in 2024-09-21T10:18:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:17 smithi160 ceph-mon[101879]: pgmap v1101: 105 pgs: 18 unknown, 87 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:18:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:17 smithi160 ceph-mon[101879]: osdmap e920: 8 total, 8 up, 8 in 2024-09-21T10:18:18.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:17 smithi070 ceph-mon[112139]: pgmap v1101: 105 pgs: 18 unknown, 87 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:18:18.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:17 smithi070 ceph-mon[112139]: osdmap e920: 8 total, 8 up, 8 in 2024-09-21T10:18:18.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:17 smithi070 ceph-mon[107385]: pgmap v1101: 105 pgs: 18 unknown, 87 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:18:18.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:17 smithi070 ceph-mon[107385]: osdmap e920: 8 total, 8 up, 8 in 2024-09-21T10:18:19.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:18 smithi160 ceph-mon[101879]: osdmap e921: 8 total, 8 up, 8 in 2024-09-21T10:18:19.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:18 smithi070 ceph-mon[107385]: osdmap e921: 8 total, 8 up, 8 in 2024-09-21T10:18:19.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:18 smithi070 ceph-mon[112139]: osdmap e921: 8 total, 8 up, 8 in 2024-09-21T10:18:20.243 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:19 smithi160 ceph-mon[101879]: pgmap v1104: 105 pgs: 18 unknown, 87 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:18:20.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:19 smithi070 ceph-mon[107385]: pgmap v1104: 105 pgs: 18 unknown, 87 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:18:20.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:19 smithi070 ceph-mon[112139]: pgmap v1104: 105 pgs: 18 unknown, 87 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:18:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:21.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:21.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:22.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:21 smithi160 ceph-mon[101879]: pgmap v1105: 105 pgs: 105 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 563 B/s rd, 563 B/s wr, 1 op/s 2024-09-21T10:18:22.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:21 smithi070 ceph-mon[112139]: pgmap v1105: 105 pgs: 105 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 563 B/s rd, 563 B/s wr, 1 op/s 2024-09-21T10:18:22.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:21 smithi070 ceph-mon[107385]: pgmap v1105: 105 pgs: 105 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 563 B/s rd, 563 B/s wr, 1 op/s 2024-09-21T10:18:23.278 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:18:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:18:23] "GET /metrics HTTP/1.1" 200 36436 "" "Prometheus/2.51.0" 2024-09-21T10:18:23.278 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:22 smithi070 ceph-mon[107385]: osdmap e922: 8 total, 8 up, 8 in 2024-09-21T10:18:23.278 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:22 smithi070 ceph-mon[112139]: osdmap e922: 8 total, 8 up, 8 in 2024-09-21T10:18:23.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:22 smithi160 ceph-mon[101879]: osdmap e922: 8 total, 8 up, 8 in 2024-09-21T10:18:24.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:23 smithi070 ceph-mon[107385]: pgmap v1107: 105 pgs: 105 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-09-21T10:18:24.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:23 smithi070 ceph-mon[112139]: pgmap v1107: 105 pgs: 105 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-09-21T10:18:24.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:23 smithi160 ceph-mon[101879]: pgmap v1107: 105 pgs: 105 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-09-21T10:18:26.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:25 smithi160 ceph-mon[101879]: pgmap v1108: 105 pgs: 105 active+clean; 4.6 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 7.4 KiB/s rd, 552 KiB/s wr, 12 op/s 2024-09-21T10:18:26.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:25 smithi070 ceph-mon[112139]: pgmap v1108: 105 pgs: 105 active+clean; 4.6 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 7.4 KiB/s rd, 552 KiB/s wr, 12 op/s 2024-09-21T10:18:26.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:25 smithi070 ceph-mon[107385]: pgmap v1108: 105 pgs: 105 active+clean; 4.6 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 7.4 KiB/s rd, 552 KiB/s wr, 12 op/s 2024-09-21T10:18:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:27 smithi160 ceph-mon[101879]: pgmap v1109: 105 pgs: 105 active+clean; 30 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 3.4 MiB/s wr, 53 op/s 2024-09-21T10:18:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:27 smithi160 ceph-mon[101879]: osdmap e923: 8 total, 8 up, 8 in 2024-09-21T10:18:28.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:27 smithi070 ceph-mon[107385]: pgmap v1109: 105 pgs: 105 active+clean; 30 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 3.4 MiB/s wr, 53 op/s 2024-09-21T10:18:28.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:27 smithi070 ceph-mon[107385]: osdmap e923: 8 total, 8 up, 8 in 2024-09-21T10:18:28.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:27 smithi070 ceph-mon[112139]: pgmap v1109: 105 pgs: 105 active+clean; 30 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 3.4 MiB/s wr, 53 op/s 2024-09-21T10:18:28.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:27 smithi070 ceph-mon[112139]: osdmap e923: 8 total, 8 up, 8 in 2024-09-21T10:18:28.481 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/2.Simple (14570 ms) 2024-09-21T10:18:28.481 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/2.Grow 2024-09-21T10:18:28.481 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:18:30.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:29 smithi160 ceph-mon[101879]: pgmap v1111: 105 pgs: 105 active+clean; 30 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 3.6 MiB/s wr, 55 op/s 2024-09-21T10:18:30.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:29 smithi070 ceph-mon[112139]: pgmap v1111: 105 pgs: 105 active+clean; 30 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 3.6 MiB/s wr, 55 op/s 2024-09-21T10:18:30.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:29 smithi070 ceph-mon[107385]: pgmap v1111: 105 pgs: 105 active+clean; 30 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 3.6 MiB/s wr, 55 op/s 2024-09-21T10:18:32.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:31 smithi160 ceph-mon[101879]: pgmap v1112: 105 pgs: 105 active+clean; 42 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 4.9 MiB/s wr, 69 op/s 2024-09-21T10:18:32.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:31 smithi070 ceph-mon[107385]: pgmap v1112: 105 pgs: 105 active+clean; 42 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 4.9 MiB/s wr, 69 op/s 2024-09-21T10:18:32.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:31 smithi070 ceph-mon[112139]: pgmap v1112: 105 pgs: 105 active+clean; 42 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 4.9 MiB/s wr, 69 op/s 2024-09-21T10:18:33.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:32 smithi160 ceph-mon[101879]: osdmap e924: 8 total, 8 up, 8 in 2024-09-21T10:18:33.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:32 smithi070 ceph-mon[107385]: osdmap e924: 8 total, 8 up, 8 in 2024-09-21T10:18:33.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:18:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:18:33] "GET /metrics HTTP/1.1" 200 36436 "" "Prometheus/2.51.0" 2024-09-21T10:18:33.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:32 smithi070 ceph-mon[112139]: osdmap e924: 8 total, 8 up, 8 in 2024-09-21T10:18:34.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:33 smithi160 ceph-mon[101879]: pgmap v1114: 105 pgs: 105 active+clean; 59 MiB data, 796 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 6.8 MiB/s wr, 73 op/s 2024-09-21T10:18:34.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:33 smithi070 ceph-mon[107385]: pgmap v1114: 105 pgs: 105 active+clean; 59 MiB data, 796 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 6.8 MiB/s wr, 73 op/s 2024-09-21T10:18:34.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:33 smithi070 ceph-mon[112139]: pgmap v1114: 105 pgs: 105 active+clean; 59 MiB data, 796 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 6.8 MiB/s wr, 73 op/s 2024-09-21T10:18:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:35 smithi160 ceph-mon[101879]: pgmap v1115: 105 pgs: 105 active+clean; 71 MiB data, 817 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 5.1 MiB/s wr, 58 op/s 2024-09-21T10:18:36.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:36.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:35 smithi070 ceph-mon[112139]: pgmap v1115: 105 pgs: 105 active+clean; 71 MiB data, 817 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 5.1 MiB/s wr, 58 op/s 2024-09-21T10:18:36.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:36.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:35 smithi070 ceph-mon[107385]: pgmap v1115: 105 pgs: 105 active+clean; 71 MiB data, 817 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 5.1 MiB/s wr, 58 op/s 2024-09-21T10:18:36.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:38.058 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/2.Grow (9576 ms) 2024-09-21T10:18:38.058 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/2.Shrink 2024-09-21T10:18:38.058 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:18:38.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:37 smithi160 ceph-mon[101879]: pgmap v1116: 105 pgs: 105 active+clean; 96 MiB data, 883 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 7.8 MiB/s wr, 78 op/s 2024-09-21T10:18:38.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:37 smithi070 ceph-mon[107385]: pgmap v1116: 105 pgs: 105 active+clean; 96 MiB data, 883 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 7.8 MiB/s wr, 78 op/s 2024-09-21T10:18:38.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:37 smithi070 ceph-mon[112139]: pgmap v1116: 105 pgs: 105 active+clean; 96 MiB data, 883 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 7.8 MiB/s wr, 78 op/s 2024-09-21T10:18:40.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:39 smithi160 ceph-mon[101879]: pgmap v1117: 105 pgs: 105 active+clean; 96 MiB data, 883 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 6.6 MiB/s wr, 66 op/s 2024-09-21T10:18:40.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:39 smithi070 ceph-mon[107385]: pgmap v1117: 105 pgs: 105 active+clean; 96 MiB data, 883 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 6.6 MiB/s wr, 66 op/s 2024-09-21T10:18:40.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:39 smithi070 ceph-mon[112139]: pgmap v1117: 105 pgs: 105 active+clean; 96 MiB data, 883 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 6.6 MiB/s wr, 66 op/s 2024-09-21T10:18:42.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:41 smithi160 ceph-mon[101879]: pgmap v1118: 105 pgs: 105 active+clean; 105 MiB data, 920 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 6.3 MiB/s wr, 61 op/s 2024-09-21T10:18:42.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:41 smithi070 ceph-mon[107385]: pgmap v1118: 105 pgs: 105 active+clean; 105 MiB data, 920 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 6.3 MiB/s wr, 61 op/s 2024-09-21T10:18:42.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:41 smithi070 ceph-mon[112139]: pgmap v1118: 105 pgs: 105 active+clean; 105 MiB data, 920 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 6.3 MiB/s wr, 61 op/s 2024-09-21T10:18:43.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:42 smithi160 ceph-mon[101879]: osdmap e925: 8 total, 8 up, 8 in 2024-09-21T10:18:43.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:18:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:18:43] "GET /metrics HTTP/1.1" 200 36461 "" "Prometheus/2.51.0" 2024-09-21T10:18:43.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:42 smithi070 ceph-mon[107385]: osdmap e925: 8 total, 8 up, 8 in 2024-09-21T10:18:43.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:42 smithi070 ceph-mon[112139]: osdmap e925: 8 total, 8 up, 8 in 2024-09-21T10:18:44.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:43 smithi160 ceph-mon[101879]: pgmap v1120: 105 pgs: 105 active+clean; 117 MiB data, 970 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-09-21T10:18:44.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:43 smithi070 ceph-mon[107385]: pgmap v1120: 105 pgs: 105 active+clean; 117 MiB data, 970 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-09-21T10:18:44.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:43 smithi070 ceph-mon[112139]: pgmap v1120: 105 pgs: 105 active+clean; 117 MiB data, 970 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-09-21T10:18:46.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:45 smithi160 ceph-mon[101879]: pgmap v1121: 105 pgs: 105 active+clean; 130 MiB data, 995 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.9 MiB/s wr, 53 op/s 2024-09-21T10:18:46.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:45 smithi070 ceph-mon[107385]: pgmap v1121: 105 pgs: 105 active+clean; 130 MiB data, 995 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.9 MiB/s wr, 53 op/s 2024-09-21T10:18:46.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:45 smithi070 ceph-mon[112139]: pgmap v1121: 105 pgs: 105 active+clean; 130 MiB data, 995 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.9 MiB/s wr, 53 op/s 2024-09-21T10:18:48.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:47 smithi160 ceph-mon[101879]: pgmap v1122: 105 pgs: 105 active+clean; 146 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.6 MiB/s rd, 5.0 MiB/s wr, 61 op/s 2024-09-21T10:18:48.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:47 smithi070 ceph-mon[107385]: pgmap v1122: 105 pgs: 105 active+clean; 146 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.6 MiB/s rd, 5.0 MiB/s wr, 61 op/s 2024-09-21T10:18:48.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:47 smithi070 ceph-mon[112139]: pgmap v1122: 105 pgs: 105 active+clean; 146 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.6 MiB/s rd, 5.0 MiB/s wr, 61 op/s 2024-09-21T10:18:48.390 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/2.Shrink (10333 ms) 2024-09-21T10:18:48.390 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/2.ShrinkToOne 2024-09-21T10:18:48.391 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:18:50.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:49 smithi160 ceph-mon[101879]: pgmap v1123: 105 pgs: 105 active+clean; 146 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.6 MiB/s rd, 5.0 MiB/s wr, 61 op/s 2024-09-21T10:18:50.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:49 smithi070 ceph-mon[107385]: pgmap v1123: 105 pgs: 105 active+clean; 146 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.6 MiB/s rd, 5.0 MiB/s wr, 61 op/s 2024-09-21T10:18:50.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:49 smithi070 ceph-mon[112139]: pgmap v1123: 105 pgs: 105 active+clean; 146 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.6 MiB/s rd, 5.0 MiB/s wr, 61 op/s 2024-09-21T10:18:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:51.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:51.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:52.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:51 smithi160 ceph-mon[101879]: pgmap v1124: 105 pgs: 105 active+clean; 166 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.1 MiB/s wr, 67 op/s 2024-09-21T10:18:52.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:51 smithi070 ceph-mon[107385]: pgmap v1124: 105 pgs: 105 active+clean; 166 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.1 MiB/s wr, 67 op/s 2024-09-21T10:18:52.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:51 smithi070 ceph-mon[112139]: pgmap v1124: 105 pgs: 105 active+clean; 166 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.1 MiB/s wr, 67 op/s 2024-09-21T10:18:53.278 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:52 smithi070 ceph-mon[107385]: osdmap e926: 8 total, 8 up, 8 in 2024-09-21T10:18:53.279 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:18:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:18:53] "GET /metrics HTTP/1.1" 200 36468 "" "Prometheus/2.51.0" 2024-09-21T10:18:53.279 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:52 smithi070 ceph-mon[112139]: osdmap e926: 8 total, 8 up, 8 in 2024-09-21T10:18:53.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:52 smithi160 ceph-mon[101879]: osdmap e926: 8 total, 8 up, 8 in 2024-09-21T10:18:54.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:53 smithi160 ceph-mon[101879]: pgmap v1126: 105 pgs: 105 active+clean; 175 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-09-21T10:18:54.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:53 smithi070 ceph-mon[107385]: pgmap v1126: 105 pgs: 105 active+clean; 175 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-09-21T10:18:54.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:53 smithi070 ceph-mon[112139]: pgmap v1126: 105 pgs: 105 active+clean; 175 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-09-21T10:18:56.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:55 smithi160 ceph-mon[101879]: pgmap v1127: 105 pgs: 105 active+clean; 183 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.3 MiB/s wr, 63 op/s 2024-09-21T10:18:56.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:55 smithi070 ceph-mon[107385]: pgmap v1127: 105 pgs: 105 active+clean; 183 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.3 MiB/s wr, 63 op/s 2024-09-21T10:18:56.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:55 smithi070 ceph-mon[112139]: pgmap v1127: 105 pgs: 105 active+clean; 183 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.3 MiB/s wr, 63 op/s 2024-09-21T10:18:58.316 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/2.ShrinkToOne (9926 ms) 2024-09-21T10:18:58.316 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/2.ShrinkToOneAfterSnapshot 2024-09-21T10:18:58.316 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:18:58.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:57 smithi160 ceph-mon[101879]: pgmap v1128: 105 pgs: 105 active+clean; 204 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-09-21T10:18:58.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:57 smithi070 ceph-mon[112139]: pgmap v1128: 105 pgs: 105 active+clean; 204 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-09-21T10:18:58.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:57 smithi070 ceph-mon[107385]: pgmap v1128: 105 pgs: 105 active+clean; 204 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-09-21T10:19:00.263 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:59 smithi160 ceph-mon[101879]: pgmap v1129: 105 pgs: 105 active+clean; 204 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-09-21T10:19:00.263 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:59 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:19:00.263 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:59 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:19:00.263 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:18:59 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:19:00.279 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:59 smithi070 ceph-mon[107385]: pgmap v1129: 105 pgs: 105 active+clean; 204 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-09-21T10:19:00.279 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:19:00.279 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:19:00.279 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:18:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:19:00.279 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:59 smithi070 ceph-mon[112139]: pgmap v1129: 105 pgs: 105 active+clean; 204 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-09-21T10:19:00.279 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:59 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:19:00.280 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:59 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:19:00.280 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:18:59 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:19:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:01 smithi160 ceph-mon[101879]: pgmap v1130: 105 pgs: 105 active+clean; 212 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.8 MiB/s rd, 5.0 MiB/s wr, 55 op/s 2024-09-21T10:19:02.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:01 smithi070 ceph-mon[107385]: pgmap v1130: 105 pgs: 105 active+clean; 212 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.8 MiB/s rd, 5.0 MiB/s wr, 55 op/s 2024-09-21T10:19:02.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:01 smithi070 ceph-mon[112139]: pgmap v1130: 105 pgs: 105 active+clean; 212 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.8 MiB/s rd, 5.0 MiB/s wr, 55 op/s 2024-09-21T10:19:03.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:02 smithi070 ceph-mon[107385]: osdmap e927: 8 total, 8 up, 8 in 2024-09-21T10:19:03.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:02 smithi070 ceph-mon[112139]: osdmap e927: 8 total, 8 up, 8 in 2024-09-21T10:19:03.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:02 smithi160 ceph-mon[101879]: osdmap e927: 8 total, 8 up, 8 in 2024-09-21T10:19:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:19:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:19:03] "GET /metrics HTTP/1.1" 200 36468 "" "Prometheus/2.51.0" 2024-09-21T10:19:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:04 smithi160 ceph-mon[101879]: pgmap v1132: 105 pgs: 105 active+clean; 212 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 946 KiB/s rd, 4.1 MiB/s wr, 56 op/s 2024-09-21T10:19:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:04 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:19:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:04 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:19:04.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:04 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:19:04.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:04 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:19:04.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:04 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:19:04.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:04 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:19:04.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:04 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:19:04.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:04 smithi070 ceph-mon[107385]: pgmap v1132: 105 pgs: 105 active+clean; 212 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 946 KiB/s rd, 4.1 MiB/s wr, 56 op/s 2024-09-21T10:19:04.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:04 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:19:04.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:04 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:19:04.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:04 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:19:04.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:04 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:19:04.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:04 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:19:04.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:04 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:19:04.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:04 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:19:04.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:04 smithi070 ceph-mon[112139]: pgmap v1132: 105 pgs: 105 active+clean; 212 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 946 KiB/s rd, 4.1 MiB/s wr, 56 op/s 2024-09-21T10:19:04.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:04 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:19:04.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:04 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:19:04.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:04 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:19:04.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:04 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:19:04.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:04 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:19:04.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:04 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:19:04.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:04 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:19:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:06 smithi160 ceph-mon[101879]: pgmap v1133: 105 pgs: 105 active+clean; 216 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 942 KiB/s rd, 3.7 MiB/s wr, 48 op/s 2024-09-21T10:19:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:06 smithi070 ceph-mon[107385]: pgmap v1133: 105 pgs: 105 active+clean; 216 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 942 KiB/s rd, 3.7 MiB/s wr, 48 op/s 2024-09-21T10:19:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:06.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:06 smithi070 ceph-mon[112139]: pgmap v1133: 105 pgs: 105 active+clean; 216 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 942 KiB/s rd, 3.7 MiB/s wr, 48 op/s 2024-09-21T10:19:06.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:08 smithi160 ceph-mon[101879]: pgmap v1134: 105 pgs: 105 active+clean; 241 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 946 KiB/s rd, 4.1 MiB/s wr, 56 op/s 2024-09-21T10:19:08.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:08 smithi160 ceph-mon[101879]: osdmap e928: 8 total, 8 up, 8 in 2024-09-21T10:19:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:08 smithi070 ceph-mon[107385]: pgmap v1134: 105 pgs: 105 active+clean; 241 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 946 KiB/s rd, 4.1 MiB/s wr, 56 op/s 2024-09-21T10:19:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:08 smithi070 ceph-mon[107385]: osdmap e928: 8 total, 8 up, 8 in 2024-09-21T10:19:08.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:08 smithi070 ceph-mon[112139]: pgmap v1134: 105 pgs: 105 active+clean; 241 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 946 KiB/s rd, 4.1 MiB/s wr, 56 op/s 2024-09-21T10:19:08.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:08 smithi070 ceph-mon[112139]: osdmap e928: 8 total, 8 up, 8 in 2024-09-21T10:19:09.431 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/2.ShrinkToOneAfterSnapshot (11115 ms) 2024-09-21T10:19:09.432 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/2.MinOverlap 2024-09-21T10:19:09.432 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:19:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:10 smithi160 ceph-mon[101879]: pgmap v1136: 105 pgs: 105 active+clean; 241 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 116 KiB/s rd, 3.6 MiB/s wr, 58 op/s 2024-09-21T10:19:10.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:10 smithi070 ceph-mon[107385]: pgmap v1136: 105 pgs: 105 active+clean; 241 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 116 KiB/s rd, 3.6 MiB/s wr, 58 op/s 2024-09-21T10:19:10.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:10 smithi070 ceph-mon[112139]: pgmap v1136: 105 pgs: 105 active+clean; 241 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 116 KiB/s rd, 3.6 MiB/s wr, 58 op/s 2024-09-21T10:19:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:12 smithi160 ceph-mon[101879]: pgmap v1137: 105 pgs: 105 active+clean; 265 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 6.4 MiB/s wr, 66 op/s 2024-09-21T10:19:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:12 smithi070 ceph-mon[107385]: pgmap v1137: 105 pgs: 105 active+clean; 265 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 6.4 MiB/s wr, 66 op/s 2024-09-21T10:19:12.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:12 smithi070 ceph-mon[112139]: pgmap v1137: 105 pgs: 105 active+clean; 265 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 6.4 MiB/s wr, 66 op/s 2024-09-21T10:19:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:19:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:19:13] "GET /metrics HTTP/1.1" 200 36472 "" "Prometheus/2.51.0" 2024-09-21T10:19:14.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:14 smithi160 ceph-mon[101879]: pgmap v1138: 105 pgs: 105 active+clean; 302 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 9.0 MiB/s wr, 71 op/s 2024-09-21T10:19:14.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:14 smithi160 ceph-mon[101879]: osdmap e929: 8 total, 8 up, 8 in 2024-09-21T10:19:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:14 smithi070 ceph-mon[107385]: pgmap v1138: 105 pgs: 105 active+clean; 302 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 9.0 MiB/s wr, 71 op/s 2024-09-21T10:19:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:14 smithi070 ceph-mon[107385]: osdmap e929: 8 total, 8 up, 8 in 2024-09-21T10:19:14.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:14 smithi070 ceph-mon[112139]: pgmap v1138: 105 pgs: 105 active+clean; 302 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 9.0 MiB/s wr, 71 op/s 2024-09-21T10:19:14.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:14 smithi070 ceph-mon[112139]: osdmap e929: 8 total, 8 up, 8 in 2024-09-21T10:19:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:16 smithi160 ceph-mon[101879]: pgmap v1140: 105 pgs: 105 active+clean; 314 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 9.1 MiB/s wr, 73 op/s 2024-09-21T10:19:16.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:16 smithi070 ceph-mon[112139]: pgmap v1140: 105 pgs: 105 active+clean; 314 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 9.1 MiB/s wr, 73 op/s 2024-09-21T10:19:16.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:16 smithi070 ceph-mon[107385]: pgmap v1140: 105 pgs: 105 active+clean; 314 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 9.1 MiB/s wr, 73 op/s 2024-09-21T10:19:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:18 smithi160 ceph-mon[101879]: pgmap v1141: 105 pgs: 105 active+clean; 331 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 9.8 MiB/s wr, 82 op/s 2024-09-21T10:19:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:18 smithi070 ceph-mon[107385]: pgmap v1141: 105 pgs: 105 active+clean; 331 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 9.8 MiB/s wr, 82 op/s 2024-09-21T10:19:18.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:18 smithi070 ceph-mon[112139]: pgmap v1141: 105 pgs: 105 active+clean; 331 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 9.8 MiB/s wr, 82 op/s 2024-09-21T10:19:19.608 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/2.MinOverlap (10177 ms) 2024-09-21T10:19:19.608 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/2.ZeroOverlap 2024-09-21T10:19:19.608 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:19:20.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:20 smithi160 ceph-mon[101879]: pgmap v1142: 105 pgs: 105 active+clean; 331 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 9.0 MiB/s wr, 75 op/s 2024-09-21T10:19:20.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:20 smithi070 ceph-mon[107385]: pgmap v1142: 105 pgs: 105 active+clean; 331 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 9.0 MiB/s wr, 75 op/s 2024-09-21T10:19:20.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:20 smithi070 ceph-mon[112139]: pgmap v1142: 105 pgs: 105 active+clean; 331 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 9.0 MiB/s wr, 75 op/s 2024-09-21T10:19:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:21.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:21.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:22.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:22 smithi160 ceph-mon[101879]: pgmap v1143: 105 pgs: 105 active+clean; 335 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 7.4 MiB/s wr, 64 op/s 2024-09-21T10:19:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:22 smithi070 ceph-mon[107385]: pgmap v1143: 105 pgs: 105 active+clean; 335 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 7.4 MiB/s wr, 64 op/s 2024-09-21T10:19:22.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:22 smithi070 ceph-mon[112139]: pgmap v1143: 105 pgs: 105 active+clean; 335 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 7.4 MiB/s wr, 64 op/s 2024-09-21T10:19:23.528 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:19:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:19:23] "GET /metrics HTTP/1.1" 200 36474 "" "Prometheus/2.51.0" 2024-09-21T10:19:24.540 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:24 smithi070 ceph-mon[107385]: pgmap v1144: 105 pgs: 105 active+clean; 339 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 60 op/s 2024-09-21T10:19:24.540 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:24 smithi070 ceph-mon[107385]: osdmap e930: 8 total, 8 up, 8 in 2024-09-21T10:19:24.540 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:24 smithi070 ceph-mon[112139]: pgmap v1144: 105 pgs: 105 active+clean; 339 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 60 op/s 2024-09-21T10:19:24.540 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:24 smithi070 ceph-mon[112139]: osdmap e930: 8 total, 8 up, 8 in 2024-09-21T10:19:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:24 smithi160 ceph-mon[101879]: pgmap v1144: 105 pgs: 105 active+clean; 339 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 60 op/s 2024-09-21T10:19:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:24 smithi160 ceph-mon[101879]: osdmap e930: 8 total, 8 up, 8 in 2024-09-21T10:19:26.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:26 smithi160 ceph-mon[101879]: pgmap v1146: 105 pgs: 105 active+clean; 351 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 45 op/s 2024-09-21T10:19:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:26 smithi070 ceph-mon[107385]: pgmap v1146: 105 pgs: 105 active+clean; 351 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 45 op/s 2024-09-21T10:19:26.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:26 smithi070 ceph-mon[112139]: pgmap v1146: 105 pgs: 105 active+clean; 351 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 45 op/s 2024-09-21T10:19:28.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:28 smithi160 ceph-mon[101879]: pgmap v1147: 105 pgs: 105 active+clean; 368 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 4.1 MiB/s wr, 58 op/s 2024-09-21T10:19:28.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:28 smithi070 ceph-mon[107385]: pgmap v1147: 105 pgs: 105 active+clean; 368 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 4.1 MiB/s wr, 58 op/s 2024-09-21T10:19:28.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:28 smithi070 ceph-mon[112139]: pgmap v1147: 105 pgs: 105 active+clean; 368 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 4.1 MiB/s wr, 58 op/s 2024-09-21T10:19:29.616 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/2.ZeroOverlap (10008 ms) 2024-09-21T10:19:30.240 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/2 (75705 ms total) 2024-09-21T10:19:30.241 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:19:30.241 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/3, where TypeParam = LUKS1UnderLUKS1 2024-09-21T10:19:30.272 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/3.Simple 2024-09-21T10:19:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:30 smithi160 ceph-mon[101879]: pgmap v1148: 105 pgs: 105 active+clean; 368 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 4.1 MiB/s wr, 58 op/s 2024-09-21T10:19:30.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:30 smithi070 ceph-mon[112139]: pgmap v1148: 105 pgs: 105 active+clean; 368 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 4.1 MiB/s wr, 58 op/s 2024-09-21T10:19:30.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:30 smithi070 ceph-mon[107385]: pgmap v1148: 105 pgs: 105 active+clean; 368 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 4.1 MiB/s wr, 58 op/s 2024-09-21T10:19:31.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:31 smithi160 ceph-mon[101879]: osdmap e931: 8 total, 8 up, 8 in 2024-09-21T10:19:31.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:31 smithi070 ceph-mon[112139]: osdmap e931: 8 total, 8 up, 8 in 2024-09-21T10:19:31.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:31 smithi070 ceph-mon[107385]: osdmap e931: 8 total, 8 up, 8 in 2024-09-21T10:19:32.259 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:19:32.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:32 smithi160 ceph-mon[101879]: pgmap v1150: 73 pgs: 73 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:19:32.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:32 smithi160 ceph-mon[101879]: osdmap e932: 8 total, 8 up, 8 in 2024-09-21T10:19:32.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:32 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/429099683' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-42","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:19:32.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:32 smithi070 ceph-mon[112139]: pgmap v1150: 73 pgs: 73 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:19:32.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:32 smithi070 ceph-mon[112139]: osdmap e932: 8 total, 8 up, 8 in 2024-09-21T10:19:32.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:32 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/429099683' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-42","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:19:32.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:32 smithi070 ceph-mon[107385]: pgmap v1150: 73 pgs: 73 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:19:32.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:32 smithi070 ceph-mon[107385]: osdmap e932: 8 total, 8 up, 8 in 2024-09-21T10:19:32.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:32 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/429099683' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-42","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:19:33.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:33 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/429099683' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:19:33.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:33 smithi160 ceph-mon[101879]: osdmap e933: 8 total, 8 up, 8 in 2024-09-21T10:19:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:19:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:19:33] "GET /metrics HTTP/1.1" 200 36474 "" "Prometheus/2.51.0" 2024-09-21T10:19:33.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:33 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/429099683' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:19:33.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:33 smithi070 ceph-mon[107385]: osdmap e933: 8 total, 8 up, 8 in 2024-09-21T10:19:33.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:33 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/429099683' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:19:33.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:33 smithi070 ceph-mon[112139]: osdmap e933: 8 total, 8 up, 8 in 2024-09-21T10:19:34.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:34 smithi160 ceph-mon[101879]: pgmap v1153: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:19:34.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:34 smithi160 ceph-mon[101879]: osdmap e934: 8 total, 8 up, 8 in 2024-09-21T10:19:34.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:34 smithi070 ceph-mon[107385]: pgmap v1153: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:19:34.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:34 smithi070 ceph-mon[107385]: osdmap e934: 8 total, 8 up, 8 in 2024-09-21T10:19:34.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:34 smithi070 ceph-mon[112139]: pgmap v1153: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:19:34.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:34 smithi070 ceph-mon[112139]: osdmap e934: 8 total, 8 up, 8 in 2024-09-21T10:19:35.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:35 smithi160 ceph-mon[101879]: osdmap e935: 8 total, 8 up, 8 in 2024-09-21T10:19:35.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:35 smithi070 ceph-mon[107385]: osdmap e935: 8 total, 8 up, 8 in 2024-09-21T10:19:35.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:35 smithi070 ceph-mon[112139]: osdmap e935: 8 total, 8 up, 8 in 2024-09-21T10:19:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:36 smithi160 ceph-mon[101879]: pgmap v1156: 105 pgs: 21 unknown, 84 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T10:19:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:36 smithi070 ceph-mon[107385]: pgmap v1156: 105 pgs: 21 unknown, 84 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T10:19:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:36.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:36 smithi070 ceph-mon[112139]: pgmap v1156: 105 pgs: 21 unknown, 84 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T10:19:36.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:38 smithi160 ceph-mon[101879]: pgmap v1157: 105 pgs: 105 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1002 B/s wr, 3 op/s 2024-09-21T10:19:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:38 smithi070 ceph-mon[107385]: pgmap v1157: 105 pgs: 105 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1002 B/s wr, 3 op/s 2024-09-21T10:19:38.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:38 smithi070 ceph-mon[112139]: pgmap v1157: 105 pgs: 105 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1002 B/s wr, 3 op/s 2024-09-21T10:19:39.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:39 smithi160 ceph-mon[101879]: osdmap e936: 8 total, 8 up, 8 in 2024-09-21T10:19:39.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:39 smithi160 ceph-mon[101879]: pgmap v1159: 105 pgs: 105 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 853 B/s wr, 3 op/s 2024-09-21T10:19:39.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:39 smithi070 ceph-mon[107385]: osdmap e936: 8 total, 8 up, 8 in 2024-09-21T10:19:39.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:39 smithi070 ceph-mon[107385]: pgmap v1159: 105 pgs: 105 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 853 B/s wr, 3 op/s 2024-09-21T10:19:39.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:39 smithi070 ceph-mon[112139]: osdmap e936: 8 total, 8 up, 8 in 2024-09-21T10:19:39.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:39 smithi070 ceph-mon[112139]: pgmap v1159: 105 pgs: 105 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 853 B/s wr, 3 op/s 2024-09-21T10:19:41.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:41 smithi160 ceph-mon[101879]: pgmap v1160: 105 pgs: 105 active+clean; 25 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 3.4 MiB/s wr, 42 op/s 2024-09-21T10:19:41.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:41 smithi070 ceph-mon[107385]: pgmap v1160: 105 pgs: 105 active+clean; 25 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 3.4 MiB/s wr, 42 op/s 2024-09-21T10:19:41.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:41 smithi070 ceph-mon[112139]: pgmap v1160: 105 pgs: 105 active+clean; 25 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 3.4 MiB/s wr, 42 op/s 2024-09-21T10:19:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:19:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:19:43] "GET /metrics HTTP/1.1" 200 36432 "" "Prometheus/2.51.0" 2024-09-21T10:19:44.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:43 smithi070 ceph-mon[107385]: pgmap v1161: 105 pgs: 105 active+clean; 30 MiB data, 718 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 3.6 MiB/s wr, 46 op/s 2024-09-21T10:19:44.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:43 smithi070 ceph-mon[107385]: osdmap e937: 8 total, 8 up, 8 in 2024-09-21T10:19:44.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:43 smithi070 ceph-mon[112139]: pgmap v1161: 105 pgs: 105 active+clean; 30 MiB data, 718 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 3.6 MiB/s wr, 46 op/s 2024-09-21T10:19:44.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:43 smithi070 ceph-mon[112139]: osdmap e937: 8 total, 8 up, 8 in 2024-09-21T10:19:44.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:43 smithi160 ceph-mon[101879]: pgmap v1161: 105 pgs: 105 active+clean; 30 MiB data, 718 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 3.6 MiB/s wr, 46 op/s 2024-09-21T10:19:44.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:43 smithi160 ceph-mon[101879]: osdmap e937: 8 total, 8 up, 8 in 2024-09-21T10:19:46.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:45 smithi070 ceph-mon[107385]: pgmap v1163: 105 pgs: 105 active+clean; 30 MiB data, 722 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 3.6 MiB/s wr, 49 op/s 2024-09-21T10:19:46.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:45 smithi070 ceph-mon[112139]: pgmap v1163: 105 pgs: 105 active+clean; 30 MiB data, 722 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 3.6 MiB/s wr, 49 op/s 2024-09-21T10:19:46.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:45 smithi160 ceph-mon[101879]: pgmap v1163: 105 pgs: 105 active+clean; 30 MiB data, 722 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 3.6 MiB/s wr, 49 op/s 2024-09-21T10:19:48.115 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/3.Simple (17843 ms) 2024-09-21T10:19:48.115 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/3.Grow 2024-09-21T10:19:48.115 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:19:48.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:47 smithi070 ceph-mon[107385]: pgmap v1164: 105 pgs: 105 active+clean; 38 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 1.8 MiB/s rd, 4.6 MiB/s wr, 63 op/s 2024-09-21T10:19:48.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:47 smithi070 ceph-mon[112139]: pgmap v1164: 105 pgs: 105 active+clean; 38 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 1.8 MiB/s rd, 4.6 MiB/s wr, 63 op/s 2024-09-21T10:19:48.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:47 smithi160 ceph-mon[101879]: pgmap v1164: 105 pgs: 105 active+clean; 38 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 1.8 MiB/s rd, 4.6 MiB/s wr, 63 op/s 2024-09-21T10:19:50.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:49 smithi070 ceph-mon[107385]: pgmap v1165: 105 pgs: 105 active+clean; 38 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 51 op/s 2024-09-21T10:19:50.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:49 smithi070 ceph-mon[112139]: pgmap v1165: 105 pgs: 105 active+clean; 38 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 51 op/s 2024-09-21T10:19:50.143 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:49 smithi160 ceph-mon[101879]: pgmap v1165: 105 pgs: 105 active+clean; 38 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 51 op/s 2024-09-21T10:19:51.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:51.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:52.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:51 smithi070 ceph-mon[107385]: pgmap v1166: 105 pgs: 105 active+clean; 55 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 9.3 MiB/s rd, 3.4 MiB/s wr, 27 op/s 2024-09-21T10:19:52.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:51 smithi070 ceph-mon[112139]: pgmap v1166: 105 pgs: 105 active+clean; 55 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 9.3 MiB/s rd, 3.4 MiB/s wr, 27 op/s 2024-09-21T10:19:52.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:51 smithi160 ceph-mon[101879]: pgmap v1166: 105 pgs: 105 active+clean; 55 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 9.3 MiB/s rd, 3.4 MiB/s wr, 27 op/s 2024-09-21T10:19:53.155 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:52 smithi070 ceph-mon[112139]: osdmap e938: 8 total, 8 up, 8 in 2024-09-21T10:19:53.155 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:52 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.0", "id": [0, 4]} : dispatch 2024-09-21T10:19:53.155 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:52 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.d", "id": [0, 4]} : dispatch 2024-09-21T10:19:53.155 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:52 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.f", "id": [0, 2]} : dispatch 2024-09-21T10:19:53.155 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:52 smithi070 ceph-mon[107385]: osdmap e938: 8 total, 8 up, 8 in 2024-09-21T10:19:53.155 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:52 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.0", "id": [0, 4]} : dispatch 2024-09-21T10:19:53.155 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:52 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.d", "id": [0, 4]} : dispatch 2024-09-21T10:19:53.155 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:52 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.f", "id": [0, 2]} : dispatch 2024-09-21T10:19:53.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:52 smithi160 ceph-mon[101879]: osdmap e938: 8 total, 8 up, 8 in 2024-09-21T10:19:53.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:52 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.0", "id": [0, 4]} : dispatch 2024-09-21T10:19:53.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:52 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.d", "id": [0, 4]} : dispatch 2024-09-21T10:19:53.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:52 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.f", "id": [0, 2]} : dispatch 2024-09-21T10:19:53.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:19:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:19:53] "GET /metrics HTTP/1.1" 200 36462 "" "Prometheus/2.51.0" 2024-09-21T10:19:54.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:53 smithi160 ceph-mon[101879]: pgmap v1168: 105 pgs: 105 active+clean; 59 MiB data, 806 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 33 op/s 2024-09-21T10:19:54.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:53 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.0", "id": [0, 4]}]': finished 2024-09-21T10:19:54.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:53 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.d", "id": [0, 4]}]': finished 2024-09-21T10:19:54.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:53 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.f", "id": [0, 2]}]': finished 2024-09-21T10:19:54.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:53 smithi160 ceph-mon[101879]: osdmap e939: 8 total, 8 up, 8 in 2024-09-21T10:19:54.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:53 smithi070 ceph-mon[112139]: pgmap v1168: 105 pgs: 105 active+clean; 59 MiB data, 806 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 33 op/s 2024-09-21T10:19:54.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:53 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.0", "id": [0, 4]}]': finished 2024-09-21T10:19:54.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:53 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.d", "id": [0, 4]}]': finished 2024-09-21T10:19:54.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:53 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.f", "id": [0, 2]}]': finished 2024-09-21T10:19:54.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:53 smithi070 ceph-mon[112139]: osdmap e939: 8 total, 8 up, 8 in 2024-09-21T10:19:54.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:53 smithi070 ceph-mon[107385]: pgmap v1168: 105 pgs: 105 active+clean; 59 MiB data, 806 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 33 op/s 2024-09-21T10:19:54.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:53 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.0", "id": [0, 4]}]': finished 2024-09-21T10:19:54.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:53 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.d", "id": [0, 4]}]': finished 2024-09-21T10:19:54.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:53 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.f", "id": [0, 2]}]': finished 2024-09-21T10:19:54.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:53 smithi070 ceph-mon[107385]: osdmap e939: 8 total, 8 up, 8 in 2024-09-21T10:19:55.267 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:54 smithi160 ceph-mon[101879]: osdmap e940: 8 total, 8 up, 8 in 2024-09-21T10:19:55.267 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 10:19:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=cleanup t=2024-09-21T10:19:55.129424677Z level=info msg="Completed cleanup jobs" duration=12.98357ms 2024-09-21T10:19:55.267 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 10:19:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=plugins.update.checker t=2024-09-21T10:19:55.265446169Z level=info msg="Update check succeeded" duration=58.825566ms 2024-09-21T10:19:55.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:54 smithi070 ceph-mon[107385]: osdmap e940: 8 total, 8 up, 8 in 2024-09-21T10:19:55.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:54 smithi070 ceph-mon[112139]: osdmap e940: 8 total, 8 up, 8 in 2024-09-21T10:19:56.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:55 smithi160 ceph-mon[101879]: pgmap v1171: 105 pgs: 1 peering, 104 active+clean; 64 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 5.0 MiB/s wr, 38 op/s 2024-09-21T10:19:56.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:55 smithi070 ceph-mon[107385]: pgmap v1171: 105 pgs: 1 peering, 104 active+clean; 64 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 5.0 MiB/s wr, 38 op/s 2024-09-21T10:19:56.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:55 smithi070 ceph-mon[112139]: pgmap v1171: 105 pgs: 1 peering, 104 active+clean; 64 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 5.0 MiB/s wr, 38 op/s 2024-09-21T10:19:58.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:57 smithi160 ceph-mon[101879]: pgmap v1172: 105 pgs: 1 peering, 104 active+clean; 92 MiB data, 923 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.2 MiB/s wr, 82 op/s; 1.3 MiB/s, 0 objects/s recovering 2024-09-21T10:19:58.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:57 smithi070 ceph-mon[107385]: pgmap v1172: 105 pgs: 1 peering, 104 active+clean; 92 MiB data, 923 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.2 MiB/s wr, 82 op/s; 1.3 MiB/s, 0 objects/s recovering 2024-09-21T10:19:58.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:57 smithi070 ceph-mon[112139]: pgmap v1172: 105 pgs: 1 peering, 104 active+clean; 92 MiB data, 923 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.2 MiB/s wr, 82 op/s; 1.3 MiB/s, 0 objects/s recovering 2024-09-21T10:20:00.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:19:59 smithi160 ceph-mon[101879]: pgmap v1173: 105 pgs: 1 peering, 104 active+clean; 92 MiB data, 923 MiB used, 714 GiB / 715 GiB avail; 88 KiB/s rd, 5.1 MiB/s wr, 58 op/s; 1.2 MiB/s, 0 objects/s recovering 2024-09-21T10:20:00.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:19:59 smithi070 ceph-mon[112139]: pgmap v1173: 105 pgs: 1 peering, 104 active+clean; 92 MiB data, 923 MiB used, 714 GiB / 715 GiB avail; 88 KiB/s rd, 5.1 MiB/s wr, 58 op/s; 1.2 MiB/s, 0 objects/s recovering 2024-09-21T10:20:00.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:19:59 smithi070 ceph-mon[107385]: pgmap v1173: 105 pgs: 1 peering, 104 active+clean; 92 MiB data, 923 MiB used, 714 GiB / 715 GiB avail; 88 KiB/s rd, 5.1 MiB/s wr, 58 op/s; 1.2 MiB/s, 0 objects/s recovering 2024-09-21T10:20:01.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:00 smithi160 ceph-mon[101879]: overall HEALTH_OK 2024-09-21T10:20:01.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:00 smithi070 ceph-mon[112139]: overall HEALTH_OK 2024-09-21T10:20:01.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:00 smithi070 ceph-mon[107385]: overall HEALTH_OK 2024-09-21T10:20:01.735 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/3.Grow (13620 ms) 2024-09-21T10:20:01.736 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/3.Shrink 2024-09-21T10:20:01.736 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:20:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:01 smithi160 ceph-mon[101879]: pgmap v1174: 105 pgs: 105 active+clean; 92 MiB data, 919 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 52 op/s; 2.0 MiB/s, 0 objects/s recovering 2024-09-21T10:20:02.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:01 smithi070 ceph-mon[107385]: pgmap v1174: 105 pgs: 105 active+clean; 92 MiB data, 919 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 52 op/s; 2.0 MiB/s, 0 objects/s recovering 2024-09-21T10:20:02.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:01 smithi070 ceph-mon[112139]: pgmap v1174: 105 pgs: 105 active+clean; 92 MiB data, 919 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 52 op/s; 2.0 MiB/s, 0 objects/s recovering 2024-09-21T10:20:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:20:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:20:03] "GET /metrics HTTP/1.1" 200 36462 "" "Prometheus/2.51.0" 2024-09-21T10:20:04.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:03 smithi070 ceph-mon[107385]: pgmap v1175: 105 pgs: 105 active+clean; 105 MiB data, 972 MiB used, 714 GiB / 715 GiB avail; 6.1 MiB/s rd, 5.3 MiB/s wr, 54 op/s; 1.7 MiB/s, 0 objects/s recovering 2024-09-21T10:20:04.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:03 smithi070 ceph-mon[112139]: pgmap v1175: 105 pgs: 105 active+clean; 105 MiB data, 972 MiB used, 714 GiB / 715 GiB avail; 6.1 MiB/s rd, 5.3 MiB/s wr, 54 op/s; 1.7 MiB/s, 0 objects/s recovering 2024-09-21T10:20:04.312 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:03 smithi160 ceph-mon[101879]: pgmap v1175: 105 pgs: 105 active+clean; 105 MiB data, 972 MiB used, 714 GiB / 715 GiB avail; 6.1 MiB/s rd, 5.3 MiB/s wr, 54 op/s; 1.7 MiB/s, 0 objects/s recovering 2024-09-21T10:20:05.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:04 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:20:05.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:04 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:20:05.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:04 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:20:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:05 smithi160 ceph-mon[101879]: pgmap v1176: 105 pgs: 105 active+clean; 105 MiB data, 972 MiB used, 714 GiB / 715 GiB avail; 5.7 MiB/s rd, 4.3 MiB/s wr, 49 op/s; 1.5 MiB/s, 0 objects/s recovering 2024-09-21T10:20:06.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:06.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:05 smithi070 ceph-mon[107385]: pgmap v1176: 105 pgs: 105 active+clean; 105 MiB data, 972 MiB used, 714 GiB / 715 GiB avail; 5.7 MiB/s rd, 4.3 MiB/s wr, 49 op/s; 1.5 MiB/s, 0 objects/s recovering 2024-09-21T10:20:06.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:06.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:05 smithi070 ceph-mon[112139]: pgmap v1176: 105 pgs: 105 active+clean; 105 MiB data, 972 MiB used, 714 GiB / 715 GiB avail; 5.7 MiB/s rd, 4.3 MiB/s wr, 49 op/s; 1.5 MiB/s, 0 objects/s recovering 2024-09-21T10:20:06.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:07.189 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:06 smithi160 ceph-mon[101879]: osdmap e941: 8 total, 8 up, 8 in 2024-09-21T10:20:07.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:06 smithi070 ceph-mon[107385]: osdmap e941: 8 total, 8 up, 8 in 2024-09-21T10:20:07.361 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:06 smithi070 ceph-mon[112139]: osdmap e941: 8 total, 8 up, 8 in 2024-09-21T10:20:08.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:07 smithi160 ceph-mon[101879]: pgmap v1178: 105 pgs: 105 active+clean; 129 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 4.1 MiB/s wr, 52 op/s; 819 KiB/s, 0 objects/s recovering 2024-09-21T10:20:08.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:20:08.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:20:08.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:07 smithi070 ceph-mon[107385]: pgmap v1178: 105 pgs: 105 active+clean; 129 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 4.1 MiB/s wr, 52 op/s; 819 KiB/s, 0 objects/s recovering 2024-09-21T10:20:08.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:20:08.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:20:08.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:07 smithi070 ceph-mon[112139]: pgmap v1178: 105 pgs: 105 active+clean; 129 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 4.1 MiB/s wr, 52 op/s; 819 KiB/s, 0 objects/s recovering 2024-09-21T10:20:08.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:20:08.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:20:09.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:09 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:20:09.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:09 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:20:09.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:09 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:20:09.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:09 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:20:09.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:09 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:20:09.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:09 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:20:09.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:09 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:20:09.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:09 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:20:09.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:09 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:20:09.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:09 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:20:09.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:09 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:20:09.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:09 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:20:09.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:09 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:20:09.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:09 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:20:09.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:09 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:20:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:10 smithi160 ceph-mon[101879]: pgmap v1179: 105 pgs: 105 active+clean; 129 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 4.1 MiB/s wr, 52 op/s; 819 KiB/s, 0 objects/s recovering 2024-09-21T10:20:10.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:10 smithi070 ceph-mon[112139]: pgmap v1179: 105 pgs: 105 active+clean; 129 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 4.1 MiB/s wr, 52 op/s; 819 KiB/s, 0 objects/s recovering 2024-09-21T10:20:10.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:10 smithi070 ceph-mon[107385]: pgmap v1179: 105 pgs: 105 active+clean; 129 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 4.1 MiB/s wr, 52 op/s; 819 KiB/s, 0 objects/s recovering 2024-09-21T10:20:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:12 smithi160 ceph-mon[101879]: pgmap v1180: 105 pgs: 105 active+clean; 146 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 5.8 MiB/s wr, 59 op/s 2024-09-21T10:20:12.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:12 smithi070 ceph-mon[112139]: pgmap v1180: 105 pgs: 105 active+clean; 146 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 5.8 MiB/s wr, 59 op/s 2024-09-21T10:20:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:12 smithi070 ceph-mon[107385]: pgmap v1180: 105 pgs: 105 active+clean; 146 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 5.8 MiB/s wr, 59 op/s 2024-09-21T10:20:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:20:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:20:13] "GET /metrics HTTP/1.1" 200 36486 "" "Prometheus/2.51.0" 2024-09-21T10:20:14.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:14 smithi160 ceph-mon[101879]: pgmap v1181: 105 pgs: 105 active+clean; 146 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 4.1 MiB/s wr, 54 op/s 2024-09-21T10:20:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:14 smithi070 ceph-mon[107385]: pgmap v1181: 105 pgs: 105 active+clean; 146 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 4.1 MiB/s wr, 54 op/s 2024-09-21T10:20:14.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:14 smithi070 ceph-mon[112139]: pgmap v1181: 105 pgs: 105 active+clean; 146 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 4.1 MiB/s wr, 54 op/s 2024-09-21T10:20:15.575 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/3.Shrink (13840 ms) 2024-09-21T10:20:15.576 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/3.ShrinkToOne 2024-09-21T10:20:15.576 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:20:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:16 smithi160 ceph-mon[101879]: pgmap v1182: 105 pgs: 105 active+clean; 159 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.4 MiB/s rd, 5.8 MiB/s wr, 58 op/s 2024-09-21T10:20:16.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:16 smithi070 ceph-mon[107385]: pgmap v1182: 105 pgs: 105 active+clean; 159 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.4 MiB/s rd, 5.8 MiB/s wr, 58 op/s 2024-09-21T10:20:16.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:16 smithi070 ceph-mon[112139]: pgmap v1182: 105 pgs: 105 active+clean; 159 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.4 MiB/s rd, 5.8 MiB/s wr, 58 op/s 2024-09-21T10:20:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:18 smithi160 ceph-mon[101879]: pgmap v1183: 105 pgs: 105 active+clean; 175 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 5.9 MiB/s wr, 67 op/s 2024-09-21T10:20:18.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:18 smithi070 ceph-mon[112139]: pgmap v1183: 105 pgs: 105 active+clean; 175 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 5.9 MiB/s wr, 67 op/s 2024-09-21T10:20:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:18 smithi070 ceph-mon[107385]: pgmap v1183: 105 pgs: 105 active+clean; 175 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 5.9 MiB/s wr, 67 op/s 2024-09-21T10:20:20.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:20 smithi160 ceph-mon[101879]: pgmap v1184: 105 pgs: 105 active+clean; 175 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.3 MiB/s rd, 4.2 MiB/s wr, 33 op/s 2024-09-21T10:20:20.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:20 smithi160 ceph-mon[101879]: osdmap e942: 8 total, 8 up, 8 in 2024-09-21T10:20:20.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:20 smithi070 ceph-mon[112139]: pgmap v1184: 105 pgs: 105 active+clean; 175 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.3 MiB/s rd, 4.2 MiB/s wr, 33 op/s 2024-09-21T10:20:20.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:20 smithi070 ceph-mon[112139]: osdmap e942: 8 total, 8 up, 8 in 2024-09-21T10:20:20.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:20 smithi070 ceph-mon[107385]: pgmap v1184: 105 pgs: 105 active+clean; 175 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.3 MiB/s rd, 4.2 MiB/s wr, 33 op/s 2024-09-21T10:20:20.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:20 smithi070 ceph-mon[107385]: osdmap e942: 8 total, 8 up, 8 in 2024-09-21T10:20:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:21.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:21.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:22.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:22 smithi160 ceph-mon[101879]: pgmap v1186: 105 pgs: 105 active+clean; 179 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 3.7 MiB/s wr, 31 op/s 2024-09-21T10:20:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:22 smithi070 ceph-mon[107385]: pgmap v1186: 105 pgs: 105 active+clean; 179 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 3.7 MiB/s wr, 31 op/s 2024-09-21T10:20:22.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:22 smithi070 ceph-mon[112139]: pgmap v1186: 105 pgs: 105 active+clean; 179 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 3.7 MiB/s wr, 31 op/s 2024-09-21T10:20:23.528 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:20:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:20:23] "GET /metrics HTTP/1.1" 200 36472 "" "Prometheus/2.51.0" 2024-09-21T10:20:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:24 smithi160 ceph-mon[101879]: pgmap v1187: 105 pgs: 105 active+clean; 192 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.0 MiB/s wr, 51 op/s 2024-09-21T10:20:24.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:24 smithi070 ceph-mon[107385]: pgmap v1187: 105 pgs: 105 active+clean; 192 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.0 MiB/s wr, 51 op/s 2024-09-21T10:20:24.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:24 smithi070 ceph-mon[112139]: pgmap v1187: 105 pgs: 105 active+clean; 192 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.0 MiB/s wr, 51 op/s 2024-09-21T10:20:26.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:26 smithi160 ceph-mon[101879]: pgmap v1188: 105 pgs: 105 active+clean; 204 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.6 MiB/s rd, 4.5 MiB/s wr, 48 op/s 2024-09-21T10:20:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:26 smithi070 ceph-mon[107385]: pgmap v1188: 105 pgs: 105 active+clean; 204 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.6 MiB/s rd, 4.5 MiB/s wr, 48 op/s 2024-09-21T10:20:26.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:26 smithi070 ceph-mon[112139]: pgmap v1188: 105 pgs: 105 active+clean; 204 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.6 MiB/s rd, 4.5 MiB/s wr, 48 op/s 2024-09-21T10:20:28.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:28 smithi160 ceph-mon[101879]: pgmap v1189: 105 pgs: 105 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 43 op/s 2024-09-21T10:20:28.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:28 smithi070 ceph-mon[107385]: pgmap v1189: 105 pgs: 105 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 43 op/s 2024-09-21T10:20:28.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:28 smithi070 ceph-mon[112139]: pgmap v1189: 105 pgs: 105 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 43 op/s 2024-09-21T10:20:28.922 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/3.ShrinkToOne (13347 ms) 2024-09-21T10:20:28.922 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/3.ShrinkToOneAfterSnapshot 2024-09-21T10:20:28.922 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:20:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:30 smithi160 ceph-mon[101879]: pgmap v1190: 105 pgs: 105 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 43 op/s 2024-09-21T10:20:30.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:30 smithi070 ceph-mon[107385]: pgmap v1190: 105 pgs: 105 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 43 op/s 2024-09-21T10:20:30.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:30 smithi070 ceph-mon[112139]: pgmap v1190: 105 pgs: 105 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 43 op/s 2024-09-21T10:20:32.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:32 smithi160 ceph-mon[101879]: pgmap v1191: 105 pgs: 105 active+clean; 212 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 40 op/s 2024-09-21T10:20:32.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:32 smithi070 ceph-mon[112139]: pgmap v1191: 105 pgs: 105 active+clean; 212 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 40 op/s 2024-09-21T10:20:32.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:32 smithi070 ceph-mon[107385]: pgmap v1191: 105 pgs: 105 active+clean; 212 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 40 op/s 2024-09-21T10:20:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:20:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:20:33] "GET /metrics HTTP/1.1" 200 36472 "" "Prometheus/2.51.0" 2024-09-21T10:20:34.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:34 smithi160 ceph-mon[101879]: pgmap v1192: 105 pgs: 105 active+clean; 212 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 789 KiB/s rd, 3.4 MiB/s wr, 47 op/s 2024-09-21T10:20:34.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:34 smithi160 ceph-mon[101879]: osdmap e943: 8 total, 8 up, 8 in 2024-09-21T10:20:34.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:34 smithi070 ceph-mon[107385]: pgmap v1192: 105 pgs: 105 active+clean; 212 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 789 KiB/s rd, 3.4 MiB/s wr, 47 op/s 2024-09-21T10:20:34.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:34 smithi070 ceph-mon[107385]: osdmap e943: 8 total, 8 up, 8 in 2024-09-21T10:20:34.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:34 smithi070 ceph-mon[112139]: pgmap v1192: 105 pgs: 105 active+clean; 212 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 789 KiB/s rd, 3.4 MiB/s wr, 47 op/s 2024-09-21T10:20:34.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:34 smithi070 ceph-mon[112139]: osdmap e943: 8 total, 8 up, 8 in 2024-09-21T10:20:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:36 smithi160 ceph-mon[101879]: pgmap v1194: 105 pgs: 105 active+clean; 220 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 923 KiB/s rd, 2.4 MiB/s wr, 25 op/s 2024-09-21T10:20:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:36.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:36 smithi070 ceph-mon[112139]: pgmap v1194: 105 pgs: 105 active+clean; 220 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 923 KiB/s rd, 2.4 MiB/s wr, 25 op/s 2024-09-21T10:20:36.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:36 smithi070 ceph-mon[107385]: pgmap v1194: 105 pgs: 105 active+clean; 220 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 923 KiB/s rd, 2.4 MiB/s wr, 25 op/s 2024-09-21T10:20:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:38 smithi160 ceph-mon[101879]: pgmap v1195: 105 pgs: 105 active+clean; 241 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 912 KiB/s rd, 4.1 MiB/s wr, 48 op/s 2024-09-21T10:20:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:38 smithi070 ceph-mon[107385]: pgmap v1195: 105 pgs: 105 active+clean; 241 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 912 KiB/s rd, 4.1 MiB/s wr, 48 op/s 2024-09-21T10:20:38.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:38 smithi070 ceph-mon[112139]: pgmap v1195: 105 pgs: 105 active+clean; 241 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 912 KiB/s rd, 4.1 MiB/s wr, 48 op/s 2024-09-21T10:20:40.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:40 smithi160 ceph-mon[101879]: pgmap v1196: 105 pgs: 105 active+clean; 241 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 912 KiB/s rd, 4.1 MiB/s wr, 48 op/s 2024-09-21T10:20:40.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:40 smithi070 ceph-mon[107385]: pgmap v1196: 105 pgs: 105 active+clean; 241 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 912 KiB/s rd, 4.1 MiB/s wr, 48 op/s 2024-09-21T10:20:40.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:40 smithi070 ceph-mon[112139]: pgmap v1196: 105 pgs: 105 active+clean; 241 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 912 KiB/s rd, 4.1 MiB/s wr, 48 op/s 2024-09-21T10:20:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:42 smithi160 ceph-mon[101879]: pgmap v1197: 105 pgs: 105 active+clean; 241 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 502 KiB/s rd, 3.3 MiB/s wr, 47 op/s 2024-09-21T10:20:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:42 smithi160 ceph-mon[101879]: osdmap e944: 8 total, 8 up, 8 in 2024-09-21T10:20:42.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:42 smithi070 ceph-mon[112139]: pgmap v1197: 105 pgs: 105 active+clean; 241 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 502 KiB/s rd, 3.3 MiB/s wr, 47 op/s 2024-09-21T10:20:42.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:42 smithi070 ceph-mon[112139]: osdmap e944: 8 total, 8 up, 8 in 2024-09-21T10:20:42.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:42 smithi070 ceph-mon[107385]: pgmap v1197: 105 pgs: 105 active+clean; 241 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 502 KiB/s rd, 3.3 MiB/s wr, 47 op/s 2024-09-21T10:20:42.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:42 smithi070 ceph-mon[107385]: osdmap e944: 8 total, 8 up, 8 in 2024-09-21T10:20:43.435 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/3.ShrinkToOneAfterSnapshot (14513 ms) 2024-09-21T10:20:43.435 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/3.MinOverlap 2024-09-21T10:20:43.435 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:20:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:20:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:20:43] "GET /metrics HTTP/1.1" 200 36468 "" "Prometheus/2.51.0" 2024-09-21T10:20:44.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:44 smithi160 ceph-mon[101879]: pgmap v1199: 105 pgs: 105 active+clean; 245 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 97 KiB/s rd, 3.6 MiB/s wr, 46 op/s 2024-09-21T10:20:44.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:44 smithi070 ceph-mon[107385]: pgmap v1199: 105 pgs: 105 active+clean; 245 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 97 KiB/s rd, 3.6 MiB/s wr, 46 op/s 2024-09-21T10:20:44.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:44 smithi070 ceph-mon[112139]: pgmap v1199: 105 pgs: 105 active+clean; 245 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 97 KiB/s rd, 3.6 MiB/s wr, 46 op/s 2024-09-21T10:20:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:46 smithi160 ceph-mon[101879]: pgmap v1200: 105 pgs: 105 active+clean; 253 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 915 KiB/s rd, 3.3 MiB/s wr, 55 op/s 2024-09-21T10:20:46.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:46 smithi070 ceph-mon[107385]: pgmap v1200: 105 pgs: 105 active+clean; 253 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 915 KiB/s rd, 3.3 MiB/s wr, 55 op/s 2024-09-21T10:20:46.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:46 smithi070 ceph-mon[112139]: pgmap v1200: 105 pgs: 105 active+clean; 253 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 915 KiB/s rd, 3.3 MiB/s wr, 55 op/s 2024-09-21T10:20:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:48 smithi160 ceph-mon[101879]: pgmap v1201: 105 pgs: 105 active+clean; 302 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 6.1 MiB/s wr, 44 op/s 2024-09-21T10:20:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:48 smithi160 ceph-mon[101879]: osdmap e945: 8 total, 8 up, 8 in 2024-09-21T10:20:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:48 smithi070 ceph-mon[107385]: pgmap v1201: 105 pgs: 105 active+clean; 302 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 6.1 MiB/s wr, 44 op/s 2024-09-21T10:20:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:48 smithi070 ceph-mon[107385]: osdmap e945: 8 total, 8 up, 8 in 2024-09-21T10:20:48.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:48 smithi070 ceph-mon[112139]: pgmap v1201: 105 pgs: 105 active+clean; 302 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 6.1 MiB/s wr, 44 op/s 2024-09-21T10:20:48.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:48 smithi070 ceph-mon[112139]: osdmap e945: 8 total, 8 up, 8 in 2024-09-21T10:20:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:50 smithi160 ceph-mon[101879]: pgmap v1203: 105 pgs: 105 active+clean; 302 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 7.6 MiB/s wr, 52 op/s 2024-09-21T10:20:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:50 smithi070 ceph-mon[107385]: pgmap v1203: 105 pgs: 105 active+clean; 302 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 7.6 MiB/s wr, 52 op/s 2024-09-21T10:20:50.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:50 smithi070 ceph-mon[112139]: pgmap v1203: 105 pgs: 105 active+clean; 302 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 7.6 MiB/s wr, 52 op/s 2024-09-21T10:20:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:51.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:52.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:52 smithi160 ceph-mon[101879]: pgmap v1204: 105 pgs: 105 active+clean; 310 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 7.1 MiB/s wr, 66 op/s 2024-09-21T10:20:52.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:52 smithi070 ceph-mon[107385]: pgmap v1204: 105 pgs: 105 active+clean; 310 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 7.1 MiB/s wr, 66 op/s 2024-09-21T10:20:52.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:52 smithi070 ceph-mon[112139]: pgmap v1204: 105 pgs: 105 active+clean; 310 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 7.1 MiB/s wr, 66 op/s 2024-09-21T10:20:53.528 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:20:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:20:53] "GET /metrics HTTP/1.1" 200 36472 "" "Prometheus/2.51.0" 2024-09-21T10:20:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:54 smithi160 ceph-mon[101879]: pgmap v1205: 105 pgs: 105 active+clean; 331 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 8.6 MiB/s wr, 71 op/s 2024-09-21T10:20:54.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:54 smithi070 ceph-mon[107385]: pgmap v1205: 105 pgs: 105 active+clean; 331 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 8.6 MiB/s wr, 71 op/s 2024-09-21T10:20:54.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:54 smithi070 ceph-mon[112139]: pgmap v1205: 105 pgs: 105 active+clean; 331 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 8.6 MiB/s wr, 71 op/s 2024-09-21T10:20:56.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:56 smithi160 ceph-mon[101879]: pgmap v1206: 105 pgs: 105 active+clean; 335 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 8.2 MiB/s wr, 58 op/s 2024-09-21T10:20:56.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:56 smithi070 ceph-mon[107385]: pgmap v1206: 105 pgs: 105 active+clean; 335 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 8.2 MiB/s wr, 58 op/s 2024-09-21T10:20:56.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:56 smithi070 ceph-mon[112139]: pgmap v1206: 105 pgs: 105 active+clean; 335 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 8.2 MiB/s wr, 58 op/s 2024-09-21T10:20:57.021 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/3.MinOverlap (13586 ms) 2024-09-21T10:20:57.021 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/3.ZeroOverlap 2024-09-21T10:20:57.022 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:20:58.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:20:58 smithi160 ceph-mon[101879]: pgmap v1207: 105 pgs: 105 active+clean; 335 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 506 KiB/s rd, 3.3 MiB/s wr, 52 op/s 2024-09-21T10:20:58.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:20:58 smithi070 ceph-mon[107385]: pgmap v1207: 105 pgs: 105 active+clean; 335 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 506 KiB/s rd, 3.3 MiB/s wr, 52 op/s 2024-09-21T10:20:58.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:20:58 smithi070 ceph-mon[112139]: pgmap v1207: 105 pgs: 105 active+clean; 335 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 506 KiB/s rd, 3.3 MiB/s wr, 52 op/s 2024-09-21T10:21:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:00 smithi160 ceph-mon[101879]: pgmap v1208: 105 pgs: 105 active+clean; 335 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 454 KiB/s rd, 3.0 MiB/s wr, 47 op/s 2024-09-21T10:21:00.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:00 smithi070 ceph-mon[107385]: pgmap v1208: 105 pgs: 105 active+clean; 335 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 454 KiB/s rd, 3.0 MiB/s wr, 47 op/s 2024-09-21T10:21:00.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:00 smithi070 ceph-mon[112139]: pgmap v1208: 105 pgs: 105 active+clean; 335 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 454 KiB/s rd, 3.0 MiB/s wr, 47 op/s 2024-09-21T10:21:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:02 smithi160 ceph-mon[101879]: pgmap v1209: 105 pgs: 105 active+clean; 339 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 48 op/s 2024-09-21T10:21:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:02 smithi160 ceph-mon[101879]: osdmap e946: 8 total, 8 up, 8 in 2024-09-21T10:21:02.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:02 smithi070 ceph-mon[112139]: pgmap v1209: 105 pgs: 105 active+clean; 339 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 48 op/s 2024-09-21T10:21:02.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:02 smithi070 ceph-mon[112139]: osdmap e946: 8 total, 8 up, 8 in 2024-09-21T10:21:02.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:02 smithi070 ceph-mon[107385]: pgmap v1209: 105 pgs: 105 active+clean; 339 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 48 op/s 2024-09-21T10:21:02.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:02 smithi070 ceph-mon[107385]: osdmap e946: 8 total, 8 up, 8 in 2024-09-21T10:21:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:21:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:21:03] "GET /metrics HTTP/1.1" 200 36472 "" "Prometheus/2.51.0" 2024-09-21T10:21:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:04 smithi160 ceph-mon[101879]: pgmap v1211: 105 pgs: 105 active+clean; 355 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.2 MiB/s wr, 29 op/s 2024-09-21T10:21:04.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:04 smithi070 ceph-mon[107385]: pgmap v1211: 105 pgs: 105 active+clean; 355 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.2 MiB/s wr, 29 op/s 2024-09-21T10:21:04.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:04 smithi070 ceph-mon[112139]: pgmap v1211: 105 pgs: 105 active+clean; 355 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.2 MiB/s wr, 29 op/s 2024-09-21T10:21:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:06 smithi160 ceph-mon[101879]: pgmap v1212: 105 pgs: 105 active+clean; 368 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 46 op/s 2024-09-21T10:21:06.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:06 smithi070 ceph-mon[107385]: pgmap v1212: 105 pgs: 105 active+clean; 368 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 46 op/s 2024-09-21T10:21:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:06.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:06 smithi070 ceph-mon[112139]: pgmap v1212: 105 pgs: 105 active+clean; 368 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 46 op/s 2024-09-21T10:21:06.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:08 smithi160 ceph-mon[101879]: pgmap v1213: 105 pgs: 105 active+clean; 368 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 43 op/s 2024-09-21T10:21:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:08 smithi070 ceph-mon[107385]: pgmap v1213: 105 pgs: 105 active+clean; 368 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 43 op/s 2024-09-21T10:21:08.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:08 smithi070 ceph-mon[112139]: pgmap v1213: 105 pgs: 105 active+clean; 368 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 43 op/s 2024-09-21T10:21:09.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:09 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:21:09.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:09 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:21:09.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:09 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:21:09.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:09 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:21:09.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:09 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:21:09.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:09 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:21:09.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:09 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:21:09.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:09 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:21:09.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:09 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:21:10.520 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:10 smithi160 ceph-mon[101879]: pgmap v1214: 105 pgs: 105 active+clean; 368 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 43 op/s 2024-09-21T10:21:10.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:10 smithi070 ceph-mon[107385]: pgmap v1214: 105 pgs: 105 active+clean; 368 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 43 op/s 2024-09-21T10:21:10.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:10 smithi070 ceph-mon[112139]: pgmap v1214: 105 pgs: 105 active+clean; 368 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 43 op/s 2024-09-21T10:21:11.212 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/3.ZeroOverlap (14191 ms) 2024-09-21T10:21:11.273 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/3 (100940 ms total) 2024-09-21T10:21:11.273 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:21:11.273 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/4, where TypeParam = LUKS1UnderLUKS2 2024-09-21T10:21:11.301 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/4.Simple 2024-09-21T10:21:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:12 smithi160 ceph-mon[101879]: pgmap v1215: 105 pgs: 105 active+clean; 372 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 61 KiB/s rd, 3.3 MiB/s wr, 44 op/s 2024-09-21T10:21:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:12 smithi160 ceph-mon[101879]: osdmap e947: 8 total, 8 up, 8 in 2024-09-21T10:21:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:12 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:21:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:12 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:21:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:12 smithi070 ceph-mon[107385]: pgmap v1215: 105 pgs: 105 active+clean; 372 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 61 KiB/s rd, 3.3 MiB/s wr, 44 op/s 2024-09-21T10:21:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:12 smithi070 ceph-mon[107385]: osdmap e947: 8 total, 8 up, 8 in 2024-09-21T10:21:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:12 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:21:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:12 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:21:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:12 smithi070 ceph-mon[112139]: pgmap v1215: 105 pgs: 105 active+clean; 372 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 61 KiB/s rd, 3.3 MiB/s wr, 44 op/s 2024-09-21T10:21:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:12 smithi070 ceph-mon[112139]: osdmap e947: 8 total, 8 up, 8 in 2024-09-21T10:21:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:12 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:21:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:12 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:21:13.282 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:21:13.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:13 smithi160 ceph-mon[101879]: osdmap e948: 8 total, 8 up, 8 in 2024-09-21T10:21:13.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:13 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1574267589' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-43","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:21:13.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:13 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-43","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:21:13.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:13 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:21:13.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:13 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:21:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:21:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:21:13] "GET /metrics HTTP/1.1" 200 36473 "" "Prometheus/2.51.0" 2024-09-21T10:21:13.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:13 smithi070 ceph-mon[107385]: osdmap e948: 8 total, 8 up, 8 in 2024-09-21T10:21:13.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:13 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1574267589' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-43","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:21:13.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:13 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-43","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:21:13.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:13 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:21:13.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:13 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:21:13.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:13 smithi070 ceph-mon[112139]: osdmap e948: 8 total, 8 up, 8 in 2024-09-21T10:21:13.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:13 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1574267589' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-43","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:21:13.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:13 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-43","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:21:13.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:13 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:21:13.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:13 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:21:14.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:14 smithi160 ceph-mon[101879]: pgmap v1218: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:21:14.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:14 smithi160 ceph-mon[101879]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:21:14.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:14 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:21:14.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:14 smithi160 ceph-mon[101879]: osdmap e949: 8 total, 8 up, 8 in 2024-09-21T10:21:14.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:14 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:21:14.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:14 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:21:14.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:14 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:21:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:14 smithi070 ceph-mon[107385]: pgmap v1218: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:21:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:14 smithi070 ceph-mon[107385]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:21:14.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:14 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:21:14.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:14 smithi070 ceph-mon[107385]: osdmap e949: 8 total, 8 up, 8 in 2024-09-21T10:21:14.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:14 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:21:14.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:14 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:21:14.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:14 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:21:14.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:14 smithi070 ceph-mon[112139]: pgmap v1218: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:21:14.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:14 smithi070 ceph-mon[112139]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:21:14.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:14 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:21:14.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:14 smithi070 ceph-mon[112139]: osdmap e949: 8 total, 8 up, 8 in 2024-09-21T10:21:14.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:14 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:21:14.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:14 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:21:14.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:14 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:21:15.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:15 smithi160 ceph-mon[101879]: osdmap e950: 8 total, 8 up, 8 in 2024-09-21T10:21:15.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:15 smithi160 ceph-mon[101879]: pgmap v1221: 105 pgs: 21 unknown, 84 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:21:15.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:15 smithi070 ceph-mon[107385]: osdmap e950: 8 total, 8 up, 8 in 2024-09-21T10:21:15.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:15 smithi070 ceph-mon[107385]: pgmap v1221: 105 pgs: 21 unknown, 84 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:21:15.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:15 smithi070 ceph-mon[112139]: osdmap e950: 8 total, 8 up, 8 in 2024-09-21T10:21:15.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:15 smithi070 ceph-mon[112139]: pgmap v1221: 105 pgs: 21 unknown, 84 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:21:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:16 smithi160 ceph-mon[101879]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:21:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:16 smithi160 ceph-mon[101879]: Cluster is now healthy 2024-09-21T10:21:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:16 smithi160 ceph-mon[101879]: osdmap e951: 8 total, 8 up, 8 in 2024-09-21T10:21:16.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:16 smithi070 ceph-mon[107385]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:21:16.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:16 smithi070 ceph-mon[107385]: Cluster is now healthy 2024-09-21T10:21:16.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:16 smithi070 ceph-mon[107385]: osdmap e951: 8 total, 8 up, 8 in 2024-09-21T10:21:16.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:16 smithi070 ceph-mon[112139]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:21:16.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:16 smithi070 ceph-mon[112139]: Cluster is now healthy 2024-09-21T10:21:16.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:16 smithi070 ceph-mon[112139]: osdmap e951: 8 total, 8 up, 8 in 2024-09-21T10:21:17.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:17 smithi160 ceph-mon[101879]: pgmap v1223: 105 pgs: 105 active+clean; 582 KiB data, 672 MiB used, 715 GiB / 715 GiB avail; 5.9 KiB/s rd, 999 B/s wr, 8 op/s 2024-09-21T10:21:17.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:17 smithi070 ceph-mon[107385]: pgmap v1223: 105 pgs: 105 active+clean; 582 KiB data, 672 MiB used, 715 GiB / 715 GiB avail; 5.9 KiB/s rd, 999 B/s wr, 8 op/s 2024-09-21T10:21:17.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:17 smithi070 ceph-mon[112139]: pgmap v1223: 105 pgs: 105 active+clean; 582 KiB data, 672 MiB used, 715 GiB / 715 GiB avail; 5.9 KiB/s rd, 999 B/s wr, 8 op/s 2024-09-21T10:21:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:19 smithi160 ceph-mon[101879]: pgmap v1224: 105 pgs: 105 active+clean; 582 KiB data, 672 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-09-21T10:21:19.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:19 smithi070 ceph-mon[107385]: pgmap v1224: 105 pgs: 105 active+clean; 582 KiB data, 672 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-09-21T10:21:19.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:19 smithi070 ceph-mon[112139]: pgmap v1224: 105 pgs: 105 active+clean; 582 KiB data, 672 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-09-21T10:21:20.743 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:20 smithi160 ceph-mon[101879]: osdmap e952: 8 total, 8 up, 8 in 2024-09-21T10:21:20.743 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:20.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:20 smithi070 ceph-mon[107385]: osdmap e952: 8 total, 8 up, 8 in 2024-09-21T10:21:20.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:20.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:20 smithi070 ceph-mon[112139]: osdmap e952: 8 total, 8 up, 8 in 2024-09-21T10:21:20.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:21.759 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:21 smithi160 ceph-mon[101879]: pgmap v1226: 105 pgs: 105 active+clean; 9.6 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 1.5 MiB/s wr, 20 op/s 2024-09-21T10:21:21.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:21 smithi070 ceph-mon[107385]: pgmap v1226: 105 pgs: 105 active+clean; 9.6 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 1.5 MiB/s wr, 20 op/s 2024-09-21T10:21:21.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:21 smithi070 ceph-mon[112139]: pgmap v1226: 105 pgs: 105 active+clean; 9.6 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 1.5 MiB/s wr, 20 op/s 2024-09-21T10:21:23.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:21:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:21:23] "GET /metrics HTTP/1.1" 200 36452 "" "Prometheus/2.51.0" 2024-09-21T10:21:24.160 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:23 smithi070 ceph-mon[107385]: pgmap v1227: 105 pgs: 105 active+clean; 30 MiB data, 751 MiB used, 714 GiB / 715 GiB avail; 35 KiB/s rd, 3.6 MiB/s wr, 47 op/s 2024-09-21T10:21:24.160 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:23 smithi070 ceph-mon[107385]: osdmap e953: 8 total, 8 up, 8 in 2024-09-21T10:21:24.160 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:23 smithi070 ceph-mon[112139]: pgmap v1227: 105 pgs: 105 active+clean; 30 MiB data, 751 MiB used, 714 GiB / 715 GiB avail; 35 KiB/s rd, 3.6 MiB/s wr, 47 op/s 2024-09-21T10:21:24.160 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:23 smithi070 ceph-mon[112139]: osdmap e953: 8 total, 8 up, 8 in 2024-09-21T10:21:24.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:23 smithi160 ceph-mon[101879]: pgmap v1227: 105 pgs: 105 active+clean; 30 MiB data, 751 MiB used, 714 GiB / 715 GiB avail; 35 KiB/s rd, 3.6 MiB/s wr, 47 op/s 2024-09-21T10:21:24.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:23 smithi160 ceph-mon[101879]: osdmap e953: 8 total, 8 up, 8 in 2024-09-21T10:21:26.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:25 smithi160 ceph-mon[101879]: pgmap v1229: 105 pgs: 105 active+clean; 30 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 3.6 MiB/s wr, 46 op/s 2024-09-21T10:21:26.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:25 smithi070 ceph-mon[107385]: pgmap v1229: 105 pgs: 105 active+clean; 30 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 3.6 MiB/s wr, 46 op/s 2024-09-21T10:21:26.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:25 smithi070 ceph-mon[112139]: pgmap v1229: 105 pgs: 105 active+clean; 30 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 3.6 MiB/s wr, 46 op/s 2024-09-21T10:21:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:27 smithi160 ceph-mon[101879]: pgmap v1230: 105 pgs: 105 active+clean; 30 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 3.6 MiB/s wr, 46 op/s 2024-09-21T10:21:28.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:27 smithi070 ceph-mon[107385]: pgmap v1230: 105 pgs: 105 active+clean; 30 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 3.6 MiB/s wr, 46 op/s 2024-09-21T10:21:28.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:27 smithi070 ceph-mon[112139]: pgmap v1230: 105 pgs: 105 active+clean; 30 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 3.6 MiB/s wr, 46 op/s 2024-09-21T10:21:30.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:29 smithi160 ceph-mon[101879]: pgmap v1231: 105 pgs: 105 active+clean; 30 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.8 MiB/s wr, 32 op/s 2024-09-21T10:21:30.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:29 smithi070 ceph-mon[107385]: pgmap v1231: 105 pgs: 105 active+clean; 30 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.8 MiB/s wr, 32 op/s 2024-09-21T10:21:30.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:29 smithi070 ceph-mon[112139]: pgmap v1231: 105 pgs: 105 active+clean; 30 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.8 MiB/s wr, 32 op/s 2024-09-21T10:21:32.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:31 smithi160 ceph-mon[101879]: pgmap v1232: 105 pgs: 105 active+clean; 30 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 2.0 MiB/s wr, 27 op/s 2024-09-21T10:21:32.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:31 smithi070 ceph-mon[107385]: pgmap v1232: 105 pgs: 105 active+clean; 30 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 2.0 MiB/s wr, 27 op/s 2024-09-21T10:21:32.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:31 smithi070 ceph-mon[112139]: pgmap v1232: 105 pgs: 105 active+clean; 30 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 2.0 MiB/s wr, 27 op/s 2024-09-21T10:21:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:21:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:21:33] "GET /metrics HTTP/1.1" 200 36452 "" "Prometheus/2.51.0" 2024-09-21T10:21:34.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:33 smithi160 ceph-mon[101879]: pgmap v1233: 105 pgs: 105 active+clean; 30 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-21T10:21:34.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:33 smithi070 ceph-mon[107385]: pgmap v1233: 105 pgs: 105 active+clean; 30 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-21T10:21:34.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:33 smithi070 ceph-mon[112139]: pgmap v1233: 105 pgs: 105 active+clean; 30 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-21T10:21:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:35 smithi160 ceph-mon[101879]: pgmap v1234: 105 pgs: 105 active+clean; 42 MiB data, 780 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 1.0 MiB/s wr, 6 op/s 2024-09-21T10:21:36.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:36.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:35 smithi070 ceph-mon[112139]: pgmap v1234: 105 pgs: 105 active+clean; 42 MiB data, 780 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 1.0 MiB/s wr, 6 op/s 2024-09-21T10:21:36.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:36.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:35 smithi070 ceph-mon[107385]: pgmap v1234: 105 pgs: 105 active+clean; 42 MiB data, 780 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 1.0 MiB/s wr, 6 op/s 2024-09-21T10:21:36.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:38.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:37 smithi160 ceph-mon[101879]: pgmap v1235: 105 pgs: 105 active+clean; 46 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 54 KiB/s rd, 1.3 MiB/s wr, 9 op/s 2024-09-21T10:21:38.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:37 smithi070 ceph-mon[107385]: pgmap v1235: 105 pgs: 105 active+clean; 46 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 54 KiB/s rd, 1.3 MiB/s wr, 9 op/s 2024-09-21T10:21:38.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:37 smithi070 ceph-mon[112139]: pgmap v1235: 105 pgs: 105 active+clean; 46 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 54 KiB/s rd, 1.3 MiB/s wr, 9 op/s 2024-09-21T10:21:39.314 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/4.Simple (28012 ms) 2024-09-21T10:21:39.314 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/4.Grow 2024-09-21T10:21:39.314 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:21:40.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:39 smithi160 ceph-mon[101879]: pgmap v1236: 105 pgs: 105 active+clean; 46 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 54 KiB/s rd, 1.3 MiB/s wr, 9 op/s 2024-09-21T10:21:40.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:39 smithi070 ceph-mon[107385]: pgmap v1236: 105 pgs: 105 active+clean; 46 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 54 KiB/s rd, 1.3 MiB/s wr, 9 op/s 2024-09-21T10:21:40.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:39 smithi070 ceph-mon[112139]: pgmap v1236: 105 pgs: 105 active+clean; 46 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 54 KiB/s rd, 1.3 MiB/s wr, 9 op/s 2024-09-21T10:21:42.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:41 smithi160 ceph-mon[101879]: pgmap v1237: 105 pgs: 105 active+clean; 55 MiB data, 847 MiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 3.1 MiB/s wr, 17 op/s 2024-09-21T10:21:42.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:41 smithi070 ceph-mon[112139]: pgmap v1237: 105 pgs: 105 active+clean; 55 MiB data, 847 MiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 3.1 MiB/s wr, 17 op/s 2024-09-21T10:21:42.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:41 smithi070 ceph-mon[107385]: pgmap v1237: 105 pgs: 105 active+clean; 55 MiB data, 847 MiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 3.1 MiB/s wr, 17 op/s 2024-09-21T10:21:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:21:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:21:43] "GET /metrics HTTP/1.1" 200 36456 "" "Prometheus/2.51.0" 2024-09-21T10:21:44.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:43 smithi160 ceph-mon[101879]: pgmap v1238: 105 pgs: 105 active+clean; 71 MiB data, 891 MiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 4.8 MiB/s wr, 28 op/s 2024-09-21T10:21:44.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:43 smithi160 ceph-mon[101879]: osdmap e954: 8 total, 8 up, 8 in 2024-09-21T10:21:44.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:43 smithi070 ceph-mon[107385]: pgmap v1238: 105 pgs: 105 active+clean; 71 MiB data, 891 MiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 4.8 MiB/s wr, 28 op/s 2024-09-21T10:21:44.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:43 smithi070 ceph-mon[107385]: osdmap e954: 8 total, 8 up, 8 in 2024-09-21T10:21:44.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:43 smithi070 ceph-mon[112139]: pgmap v1238: 105 pgs: 105 active+clean; 71 MiB data, 891 MiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 4.8 MiB/s wr, 28 op/s 2024-09-21T10:21:44.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:43 smithi070 ceph-mon[112139]: osdmap e954: 8 total, 8 up, 8 in 2024-09-21T10:21:46.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:45 smithi160 ceph-mon[101879]: pgmap v1240: 105 pgs: 105 active+clean; 79 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 5.3 MiB/s wr, 31 op/s 2024-09-21T10:21:46.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:45 smithi070 ceph-mon[107385]: pgmap v1240: 105 pgs: 105 active+clean; 79 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 5.3 MiB/s wr, 31 op/s 2024-09-21T10:21:46.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:45 smithi070 ceph-mon[112139]: pgmap v1240: 105 pgs: 105 active+clean; 79 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 5.3 MiB/s wr, 31 op/s 2024-09-21T10:21:48.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:47 smithi160 ceph-mon[101879]: pgmap v1241: 105 pgs: 105 active+clean; 100 MiB data, 974 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 58 op/s 2024-09-21T10:21:48.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:47 smithi070 ceph-mon[107385]: pgmap v1241: 105 pgs: 105 active+clean; 100 MiB data, 974 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 58 op/s 2024-09-21T10:21:48.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:47 smithi070 ceph-mon[112139]: pgmap v1241: 105 pgs: 105 active+clean; 100 MiB data, 974 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 58 op/s 2024-09-21T10:21:50.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:49 smithi160 ceph-mon[101879]: pgmap v1242: 105 pgs: 105 active+clean; 100 MiB data, 974 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 58 op/s 2024-09-21T10:21:50.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:49 smithi070 ceph-mon[107385]: pgmap v1242: 105 pgs: 105 active+clean; 100 MiB data, 974 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 58 op/s 2024-09-21T10:21:50.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:49 smithi070 ceph-mon[112139]: pgmap v1242: 105 pgs: 105 active+clean; 100 MiB data, 974 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 58 op/s 2024-09-21T10:21:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:51.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:51.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:52.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:51 smithi160 ceph-mon[101879]: pgmap v1243: 105 pgs: 105 active+clean; 100 MiB data, 974 MiB used, 714 GiB / 715 GiB avail; 7.2 MiB/s rd, 4.9 MiB/s wr, 48 op/s 2024-09-21T10:21:52.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:51 smithi070 ceph-mon[112139]: pgmap v1243: 105 pgs: 105 active+clean; 100 MiB data, 974 MiB used, 714 GiB / 715 GiB avail; 7.2 MiB/s rd, 4.9 MiB/s wr, 48 op/s 2024-09-21T10:21:52.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:51 smithi070 ceph-mon[107385]: pgmap v1243: 105 pgs: 105 active+clean; 100 MiB data, 974 MiB used, 714 GiB / 715 GiB avail; 7.2 MiB/s rd, 4.9 MiB/s wr, 48 op/s 2024-09-21T10:21:53.278 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:21:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:21:53] "GET /metrics HTTP/1.1" 200 36465 "" "Prometheus/2.51.0" 2024-09-21T10:21:53.279 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:52 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "47.1f", "id": [1, 2]} : dispatch 2024-09-21T10:21:53.279 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:52 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "47.1f", "id": [1, 2]} : dispatch 2024-09-21T10:21:53.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:52 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "47.1f", "id": [1, 2]} : dispatch 2024-09-21T10:21:54.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:53 smithi160 ceph-mon[101879]: pgmap v1244: 105 pgs: 105 active+clean; 108 MiB data, 994 MiB used, 714 GiB / 715 GiB avail; 441 KiB/s rd, 3.7 MiB/s wr, 43 op/s 2024-09-21T10:21:54.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:53 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "47.1f", "id": [1, 2]}]': finished 2024-09-21T10:21:54.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:53 smithi160 ceph-mon[101879]: osdmap e955: 8 total, 8 up, 8 in 2024-09-21T10:21:54.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:53 smithi070 ceph-mon[107385]: pgmap v1244: 105 pgs: 105 active+clean; 108 MiB data, 994 MiB used, 714 GiB / 715 GiB avail; 441 KiB/s rd, 3.7 MiB/s wr, 43 op/s 2024-09-21T10:21:54.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:53 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "47.1f", "id": [1, 2]}]': finished 2024-09-21T10:21:54.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:53 smithi070 ceph-mon[107385]: osdmap e955: 8 total, 8 up, 8 in 2024-09-21T10:21:54.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:53 smithi070 ceph-mon[112139]: pgmap v1244: 105 pgs: 105 active+clean; 108 MiB data, 994 MiB used, 714 GiB / 715 GiB avail; 441 KiB/s rd, 3.7 MiB/s wr, 43 op/s 2024-09-21T10:21:54.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:53 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "47.1f", "id": [1, 2]}]': finished 2024-09-21T10:21:54.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:53 smithi070 ceph-mon[112139]: osdmap e955: 8 total, 8 up, 8 in 2024-09-21T10:21:55.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:54 smithi160 ceph-mon[101879]: osdmap e956: 8 total, 8 up, 8 in 2024-09-21T10:21:55.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:54 smithi070 ceph-mon[107385]: osdmap e956: 8 total, 8 up, 8 in 2024-09-21T10:21:55.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:54 smithi070 ceph-mon[112139]: osdmap e956: 8 total, 8 up, 8 in 2024-09-21T10:21:56.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:55 smithi160 ceph-mon[101879]: pgmap v1247: 105 pgs: 105 active+clean; 116 MiB data, 1014 MiB used, 714 GiB / 715 GiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 10 op/s 2024-09-21T10:21:56.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:55 smithi070 ceph-mon[107385]: pgmap v1247: 105 pgs: 105 active+clean; 116 MiB data, 1014 MiB used, 714 GiB / 715 GiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 10 op/s 2024-09-21T10:21:56.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:55 smithi070 ceph-mon[112139]: pgmap v1247: 105 pgs: 105 active+clean; 116 MiB data, 1014 MiB used, 714 GiB / 715 GiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 10 op/s 2024-09-21T10:21:56.522 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/4.Grow (17209 ms) 2024-09-21T10:21:56.522 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/4.Shrink 2024-09-21T10:21:56.522 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:21:58.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:57 smithi160 ceph-mon[101879]: pgmap v1248: 105 pgs: 105 active+clean; 137 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 5.6 MiB/s wr, 36 op/s; 1024 KiB/s, 0 objects/s recovering 2024-09-21T10:21:58.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:57 smithi070 ceph-mon[107385]: pgmap v1248: 105 pgs: 105 active+clean; 137 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 5.6 MiB/s wr, 36 op/s; 1024 KiB/s, 0 objects/s recovering 2024-09-21T10:21:58.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:57 smithi070 ceph-mon[112139]: pgmap v1248: 105 pgs: 105 active+clean; 137 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 5.6 MiB/s wr, 36 op/s; 1024 KiB/s, 0 objects/s recovering 2024-09-21T10:22:00.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:21:59 smithi160 ceph-mon[101879]: pgmap v1249: 105 pgs: 105 active+clean; 137 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 5.6 MiB/s wr, 36 op/s; 1024 KiB/s, 0 objects/s recovering 2024-09-21T10:22:00.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:21:59 smithi070 ceph-mon[107385]: pgmap v1249: 105 pgs: 105 active+clean; 137 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 5.6 MiB/s wr, 36 op/s; 1024 KiB/s, 0 objects/s recovering 2024-09-21T10:22:00.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:21:59 smithi070 ceph-mon[112139]: pgmap v1249: 105 pgs: 105 active+clean; 137 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 5.6 MiB/s wr, 36 op/s; 1024 KiB/s, 0 objects/s recovering 2024-09-21T10:22:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:01 smithi160 ceph-mon[101879]: pgmap v1250: 105 pgs: 105 active+clean; 141 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 6.1 MiB/s wr, 29 op/s; 1024 KiB/s, 0 objects/s recovering 2024-09-21T10:22:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:01 smithi160 ceph-mon[101879]: osdmap e957: 8 total, 8 up, 8 in 2024-09-21T10:22:02.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:01 smithi070 ceph-mon[112139]: pgmap v1250: 105 pgs: 105 active+clean; 141 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 6.1 MiB/s wr, 29 op/s; 1024 KiB/s, 0 objects/s recovering 2024-09-21T10:22:02.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:01 smithi070 ceph-mon[112139]: osdmap e957: 8 total, 8 up, 8 in 2024-09-21T10:22:02.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:01 smithi070 ceph-mon[107385]: pgmap v1250: 105 pgs: 105 active+clean; 141 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 6.1 MiB/s wr, 29 op/s; 1024 KiB/s, 0 objects/s recovering 2024-09-21T10:22:02.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:01 smithi070 ceph-mon[107385]: osdmap e957: 8 total, 8 up, 8 in 2024-09-21T10:22:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:22:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:22:03] "GET /metrics HTTP/1.1" 200 36465 "" "Prometheus/2.51.0" 2024-09-21T10:22:04.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:03 smithi160 ceph-mon[101879]: pgmap v1252: 105 pgs: 105 active+clean; 153 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 6.3 MiB/s wr, 54 op/s; 970 KiB/s, 0 objects/s recovering 2024-09-21T10:22:04.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:03 smithi070 ceph-mon[107385]: pgmap v1252: 105 pgs: 105 active+clean; 153 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 6.3 MiB/s wr, 54 op/s; 970 KiB/s, 0 objects/s recovering 2024-09-21T10:22:04.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:03 smithi070 ceph-mon[112139]: pgmap v1252: 105 pgs: 105 active+clean; 153 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 6.3 MiB/s wr, 54 op/s; 970 KiB/s, 0 objects/s recovering 2024-09-21T10:22:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:05 smithi160 ceph-mon[101879]: pgmap v1253: 105 pgs: 105 active+clean; 162 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.2 MiB/s wr, 47 op/s; 819 KiB/s, 0 objects/s recovering 2024-09-21T10:22:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:06.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:05 smithi070 ceph-mon[107385]: pgmap v1253: 105 pgs: 105 active+clean; 162 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.2 MiB/s wr, 47 op/s; 819 KiB/s, 0 objects/s recovering 2024-09-21T10:22:06.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:06.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:05 smithi070 ceph-mon[112139]: pgmap v1253: 105 pgs: 105 active+clean; 162 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.2 MiB/s wr, 47 op/s; 819 KiB/s, 0 objects/s recovering 2024-09-21T10:22:06.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:08.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:07 smithi160 ceph-mon[101879]: pgmap v1254: 105 pgs: 105 active+clean; 170 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.6 MiB/s rd, 4.1 MiB/s wr, 42 op/s 2024-09-21T10:22:08.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:07 smithi070 ceph-mon[107385]: pgmap v1254: 105 pgs: 105 active+clean; 170 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.6 MiB/s rd, 4.1 MiB/s wr, 42 op/s 2024-09-21T10:22:08.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:07 smithi070 ceph-mon[112139]: pgmap v1254: 105 pgs: 105 active+clean; 170 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.6 MiB/s rd, 4.1 MiB/s wr, 42 op/s 2024-09-21T10:22:10.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:09 smithi160 ceph-mon[101879]: pgmap v1255: 105 pgs: 105 active+clean; 170 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.6 MiB/s rd, 4.1 MiB/s wr, 42 op/s 2024-09-21T10:22:10.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:09 smithi070 ceph-mon[107385]: pgmap v1255: 105 pgs: 105 active+clean; 170 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.6 MiB/s rd, 4.1 MiB/s wr, 42 op/s 2024-09-21T10:22:10.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:09 smithi070 ceph-mon[112139]: pgmap v1255: 105 pgs: 105 active+clean; 170 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.6 MiB/s rd, 4.1 MiB/s wr, 42 op/s 2024-09-21T10:22:12.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:11 smithi160 ceph-mon[101879]: pgmap v1256: 105 pgs: 105 active+clean; 174 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 40 op/s 2024-09-21T10:22:12.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:11 smithi070 ceph-mon[107385]: pgmap v1256: 105 pgs: 105 active+clean; 174 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 40 op/s 2024-09-21T10:22:12.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:11 smithi070 ceph-mon[112139]: pgmap v1256: 105 pgs: 105 active+clean; 174 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 40 op/s 2024-09-21T10:22:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:22:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:22:13] "GET /metrics HTTP/1.1" 200 36478 "" "Prometheus/2.51.0" 2024-09-21T10:22:14.222 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:13 smithi070 ceph-mon[107385]: pgmap v1257: 105 pgs: 105 active+clean; 186 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 45 op/s 2024-09-21T10:22:14.222 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:13 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:22:14.222 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:13 smithi070 ceph-mon[112139]: pgmap v1257: 105 pgs: 105 active+clean; 186 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 45 op/s 2024-09-21T10:22:14.222 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:13 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:22:14.281 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:13 smithi160 ceph-mon[101879]: pgmap v1257: 105 pgs: 105 active+clean; 186 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 45 op/s 2024-09-21T10:22:14.281 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:13 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:22:14.877 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/4.Shrink (18354 ms) 2024-09-21T10:22:14.877 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/4.ShrinkToOne 2024-09-21T10:22:14.877 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:22:16.228 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:15 smithi070 ceph-mon[107385]: pgmap v1258: 105 pgs: 105 active+clean; 187 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 25 op/s 2024-09-21T10:22:16.228 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:15 smithi070 ceph-mon[112139]: pgmap v1258: 105 pgs: 105 active+clean; 187 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 25 op/s 2024-09-21T10:22:16.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:15 smithi160 ceph-mon[101879]: pgmap v1258: 105 pgs: 105 active+clean; 187 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 25 op/s 2024-09-21T10:22:18.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:18 smithi070 ceph-mon[107385]: pgmap v1259: 105 pgs: 105 active+clean; 211 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.3 MiB/s rd, 5.4 MiB/s wr, 40 op/s 2024-09-21T10:22:18.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:18 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:22:18.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:18 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:22:18.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:18 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:22:18.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:18 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:22:18.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:18 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:22:18.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:18 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:22:18.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:18 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:22:18.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:18 smithi070 ceph-mon[112139]: pgmap v1259: 105 pgs: 105 active+clean; 211 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.3 MiB/s rd, 5.4 MiB/s wr, 40 op/s 2024-09-21T10:22:18.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:18 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:22:18.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:18 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:22:18.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:18 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:22:18.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:18 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:22:18.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:18 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:22:18.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:18 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:22:18.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:18 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:22:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:18 smithi160 ceph-mon[101879]: pgmap v1259: 105 pgs: 105 active+clean; 211 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.3 MiB/s rd, 5.4 MiB/s wr, 40 op/s 2024-09-21T10:22:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:18 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:22:18.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:18 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:22:18.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:18 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:22:18.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:18 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:22:18.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:18 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:22:18.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:18 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:22:18.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:18 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:22:20.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:20 smithi070 ceph-mon[112139]: pgmap v1260: 105 pgs: 105 active+clean; 211 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.3 MiB/s rd, 4.8 MiB/s wr, 27 op/s 2024-09-21T10:22:20.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:20 smithi070 ceph-mon[112139]: osdmap e958: 8 total, 8 up, 8 in 2024-09-21T10:22:20.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:20 smithi070 ceph-mon[107385]: pgmap v1260: 105 pgs: 105 active+clean; 211 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.3 MiB/s rd, 4.8 MiB/s wr, 27 op/s 2024-09-21T10:22:20.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:20 smithi070 ceph-mon[107385]: osdmap e958: 8 total, 8 up, 8 in 2024-09-21T10:22:20.394 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:20 smithi160 ceph-mon[101879]: pgmap v1260: 105 pgs: 105 active+clean; 211 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.3 MiB/s rd, 4.8 MiB/s wr, 27 op/s 2024-09-21T10:22:20.394 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:20 smithi160 ceph-mon[101879]: osdmap e958: 8 total, 8 up, 8 in 2024-09-21T10:22:21.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:21.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:21.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:22.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:22 smithi160 ceph-mon[101879]: pgmap v1262: 105 pgs: 105 active+clean; 224 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 6.6 MiB/s wr, 61 op/s 2024-09-21T10:22:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:22 smithi070 ceph-mon[107385]: pgmap v1262: 105 pgs: 105 active+clean; 224 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 6.6 MiB/s wr, 61 op/s 2024-09-21T10:22:22.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:22 smithi070 ceph-mon[112139]: pgmap v1262: 105 pgs: 105 active+clean; 224 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 6.6 MiB/s wr, 61 op/s 2024-09-21T10:22:23.528 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:22:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:22:23] "GET /metrics HTTP/1.1" 200 36473 "" "Prometheus/2.51.0" 2024-09-21T10:22:24.540 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:24 smithi070 ceph-mon[107385]: pgmap v1263: 105 pgs: 105 active+clean; 240 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 60 op/s 2024-09-21T10:22:24.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:24 smithi070 ceph-mon[112139]: pgmap v1263: 105 pgs: 105 active+clean; 240 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 60 op/s 2024-09-21T10:22:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:24 smithi160 ceph-mon[101879]: pgmap v1263: 105 pgs: 105 active+clean; 240 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 60 op/s 2024-09-21T10:22:26.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:26 smithi070 ceph-mon[107385]: pgmap v1264: 105 pgs: 105 active+clean; 240 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 6.5 MiB/s wr, 59 op/s 2024-09-21T10:22:26.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:26 smithi070 ceph-mon[112139]: pgmap v1264: 105 pgs: 105 active+clean; 240 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 6.5 MiB/s wr, 59 op/s 2024-09-21T10:22:26.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:26 smithi160 ceph-mon[101879]: pgmap v1264: 105 pgs: 105 active+clean; 240 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 6.5 MiB/s wr, 59 op/s 2024-09-21T10:22:28.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:28 smithi160 ceph-mon[101879]: pgmap v1265: 105 pgs: 105 active+clean; 240 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 541 KiB/s rd, 2.9 MiB/s wr, 39 op/s 2024-09-21T10:22:28.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:28 smithi070 ceph-mon[107385]: pgmap v1265: 105 pgs: 105 active+clean; 240 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 541 KiB/s rd, 2.9 MiB/s wr, 39 op/s 2024-09-21T10:22:28.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:28 smithi070 ceph-mon[112139]: pgmap v1265: 105 pgs: 105 active+clean; 240 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 541 KiB/s rd, 2.9 MiB/s wr, 39 op/s 2024-09-21T10:22:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:30 smithi160 ceph-mon[101879]: pgmap v1266: 105 pgs: 105 active+clean; 240 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 541 KiB/s rd, 2.9 MiB/s wr, 39 op/s 2024-09-21T10:22:30.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:30 smithi070 ceph-mon[107385]: pgmap v1266: 105 pgs: 105 active+clean; 240 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 541 KiB/s rd, 2.9 MiB/s wr, 39 op/s 2024-09-21T10:22:30.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:30 smithi070 ceph-mon[112139]: pgmap v1266: 105 pgs: 105 active+clean; 240 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 541 KiB/s rd, 2.9 MiB/s wr, 39 op/s 2024-09-21T10:22:32.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:32 smithi160 ceph-mon[101879]: pgmap v1267: 105 pgs: 105 active+clean; 252 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 419 KiB/s rd, 2.8 MiB/s wr, 39 op/s 2024-09-21T10:22:32.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:32 smithi070 ceph-mon[107385]: pgmap v1267: 105 pgs: 105 active+clean; 252 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 419 KiB/s rd, 2.8 MiB/s wr, 39 op/s 2024-09-21T10:22:32.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:32 smithi070 ceph-mon[112139]: pgmap v1267: 105 pgs: 105 active+clean; 252 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 419 KiB/s rd, 2.8 MiB/s wr, 39 op/s 2024-09-21T10:22:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:22:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:22:33] "GET /metrics HTTP/1.1" 200 36473 "" "Prometheus/2.51.0" 2024-09-21T10:22:33.751 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/4.ShrinkToOne (18874 ms) 2024-09-21T10:22:33.751 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/4.ShrinkToOneAfterSnapshot 2024-09-21T10:22:33.751 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:22:34.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:34 smithi160 ceph-mon[101879]: pgmap v1268: 105 pgs: 105 active+clean; 256 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 36 KiB/s rd, 2.7 MiB/s wr, 18 op/s 2024-09-21T10:22:34.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:34 smithi070 ceph-mon[107385]: pgmap v1268: 105 pgs: 105 active+clean; 256 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 36 KiB/s rd, 2.7 MiB/s wr, 18 op/s 2024-09-21T10:22:34.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:34 smithi070 ceph-mon[112139]: pgmap v1268: 105 pgs: 105 active+clean; 256 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 36 KiB/s rd, 2.7 MiB/s wr, 18 op/s 2024-09-21T10:22:36.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:36 smithi070 ceph-mon[112139]: pgmap v1269: 105 pgs: 105 active+clean; 256 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 17 op/s 2024-09-21T10:22:36.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:36.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:36 smithi070 ceph-mon[107385]: pgmap v1269: 105 pgs: 105 active+clean; 256 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 17 op/s 2024-09-21T10:22:36.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:36 smithi160 ceph-mon[101879]: pgmap v1269: 105 pgs: 105 active+clean; 256 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 17 op/s 2024-09-21T10:22:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:38 smithi160 ceph-mon[101879]: pgmap v1270: 105 pgs: 105 active+clean; 260 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 23 op/s 2024-09-21T10:22:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:38 smithi070 ceph-mon[107385]: pgmap v1270: 105 pgs: 105 active+clean; 260 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 23 op/s 2024-09-21T10:22:38.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:38 smithi070 ceph-mon[112139]: pgmap v1270: 105 pgs: 105 active+clean; 260 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 23 op/s 2024-09-21T10:22:39.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:39 smithi160 ceph-mon[101879]: osdmap e959: 8 total, 8 up, 8 in 2024-09-21T10:22:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:39 smithi070 ceph-mon[107385]: osdmap e959: 8 total, 8 up, 8 in 2024-09-21T10:22:39.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:39 smithi070 ceph-mon[112139]: osdmap e959: 8 total, 8 up, 8 in 2024-09-21T10:22:40.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:40 smithi160 ceph-mon[101879]: pgmap v1272: 105 pgs: 105 active+clean; 260 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.0 MiB/s wr, 28 op/s 2024-09-21T10:22:40.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:40 smithi070 ceph-mon[107385]: pgmap v1272: 105 pgs: 105 active+clean; 260 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.0 MiB/s wr, 28 op/s 2024-09-21T10:22:40.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:40 smithi070 ceph-mon[112139]: pgmap v1272: 105 pgs: 105 active+clean; 260 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.0 MiB/s wr, 28 op/s 2024-09-21T10:22:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:42 smithi160 ceph-mon[101879]: pgmap v1273: 105 pgs: 105 active+clean; 273 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 30 op/s 2024-09-21T10:22:42.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:42 smithi070 ceph-mon[107385]: pgmap v1273: 105 pgs: 105 active+clean; 273 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 30 op/s 2024-09-21T10:22:42.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:42 smithi070 ceph-mon[112139]: pgmap v1273: 105 pgs: 105 active+clean; 273 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 30 op/s 2024-09-21T10:22:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:22:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:22:43] "GET /metrics HTTP/1.1" 200 36466 "" "Prometheus/2.51.0" 2024-09-21T10:22:44.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:44 smithi160 ceph-mon[101879]: pgmap v1274: 105 pgs: 105 active+clean; 289 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 51 op/s 2024-09-21T10:22:44.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:44 smithi070 ceph-mon[107385]: pgmap v1274: 105 pgs: 105 active+clean; 289 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 51 op/s 2024-09-21T10:22:44.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:44 smithi070 ceph-mon[112139]: pgmap v1274: 105 pgs: 105 active+clean; 289 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 51 op/s 2024-09-21T10:22:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:46 smithi160 ceph-mon[101879]: pgmap v1275: 105 pgs: 105 active+clean; 289 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 880 KiB/s rd, 4.1 MiB/s wr, 42 op/s 2024-09-21T10:22:46.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:46 smithi070 ceph-mon[107385]: pgmap v1275: 105 pgs: 105 active+clean; 289 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 880 KiB/s rd, 4.1 MiB/s wr, 42 op/s 2024-09-21T10:22:46.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:46 smithi070 ceph-mon[112139]: pgmap v1275: 105 pgs: 105 active+clean; 289 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 880 KiB/s rd, 4.1 MiB/s wr, 42 op/s 2024-09-21T10:22:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:48 smithi160 ceph-mon[101879]: pgmap v1276: 105 pgs: 105 active+clean; 305 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 60 KiB/s rd, 4.5 MiB/s wr, 41 op/s 2024-09-21T10:22:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:48 smithi070 ceph-mon[107385]: pgmap v1276: 105 pgs: 105 active+clean; 305 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 60 KiB/s rd, 4.5 MiB/s wr, 41 op/s 2024-09-21T10:22:48.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:48 smithi070 ceph-mon[112139]: pgmap v1276: 105 pgs: 105 active+clean; 305 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 60 KiB/s rd, 4.5 MiB/s wr, 41 op/s 2024-09-21T10:22:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:50 smithi160 ceph-mon[101879]: pgmap v1277: 105 pgs: 105 active+clean; 305 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 4.4 MiB/s wr, 41 op/s 2024-09-21T10:22:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:50 smithi070 ceph-mon[107385]: pgmap v1277: 105 pgs: 105 active+clean; 305 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 4.4 MiB/s wr, 41 op/s 2024-09-21T10:22:50.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:50 smithi070 ceph-mon[112139]: pgmap v1277: 105 pgs: 105 active+clean; 305 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 4.4 MiB/s wr, 41 op/s 2024-09-21T10:22:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:51 smithi160 ceph-mon[101879]: osdmap e960: 8 total, 8 up, 8 in 2024-09-21T10:22:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:51 smithi070 ceph-mon[107385]: osdmap e960: 8 total, 8 up, 8 in 2024-09-21T10:22:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:51.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:51 smithi070 ceph-mon[112139]: osdmap e960: 8 total, 8 up, 8 in 2024-09-21T10:22:51.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:52.420 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/4.ShrinkToOneAfterSnapshot (18669 ms) 2024-09-21T10:22:52.420 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/4.MinOverlap 2024-09-21T10:22:52.420 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:22:52.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:52 smithi160 ceph-mon[101879]: pgmap v1279: 105 pgs: 105 active+clean; 305 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 58 KiB/s rd, 3.2 MiB/s wr, 37 op/s 2024-09-21T10:22:52.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:52 smithi070 ceph-mon[112139]: pgmap v1279: 105 pgs: 105 active+clean; 305 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 58 KiB/s rd, 3.2 MiB/s wr, 37 op/s 2024-09-21T10:22:52.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:52 smithi070 ceph-mon[107385]: pgmap v1279: 105 pgs: 105 active+clean; 305 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 58 KiB/s rd, 3.2 MiB/s wr, 37 op/s 2024-09-21T10:22:53.528 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:22:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:22:53] "GET /metrics HTTP/1.1" 200 36470 "" "Prometheus/2.51.0" 2024-09-21T10:22:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:54 smithi160 ceph-mon[101879]: pgmap v1280: 105 pgs: 105 active+clean; 358 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 6.9 MiB/s wr, 32 op/s 2024-09-21T10:22:54.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:54 smithi070 ceph-mon[107385]: pgmap v1280: 105 pgs: 105 active+clean; 358 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 6.9 MiB/s wr, 32 op/s 2024-09-21T10:22:54.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:54 smithi070 ceph-mon[112139]: pgmap v1280: 105 pgs: 105 active+clean; 358 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 6.9 MiB/s wr, 32 op/s 2024-09-21T10:22:56.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:56 smithi160 ceph-mon[101879]: pgmap v1281: 105 pgs: 105 active+clean; 374 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 8.5 MiB/s wr, 36 op/s 2024-09-21T10:22:56.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:56 smithi070 ceph-mon[112139]: pgmap v1281: 105 pgs: 105 active+clean; 374 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 8.5 MiB/s wr, 36 op/s 2024-09-21T10:22:56.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:56 smithi070 ceph-mon[107385]: pgmap v1281: 105 pgs: 105 active+clean; 374 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 8.5 MiB/s wr, 36 op/s 2024-09-21T10:22:57.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:57 smithi160 ceph-mon[101879]: osdmap e961: 8 total, 8 up, 8 in 2024-09-21T10:22:57.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:57 smithi070 ceph-mon[107385]: osdmap e961: 8 total, 8 up, 8 in 2024-09-21T10:22:57.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:57 smithi070 ceph-mon[112139]: osdmap e961: 8 total, 8 up, 8 in 2024-09-21T10:22:58.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:22:58 smithi160 ceph-mon[101879]: pgmap v1283: 105 pgs: 105 active+clean; 394 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.2 MiB/s rd, 11 MiB/s wr, 67 op/s 2024-09-21T10:22:58.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:22:58 smithi070 ceph-mon[107385]: pgmap v1283: 105 pgs: 105 active+clean; 394 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.2 MiB/s rd, 11 MiB/s wr, 67 op/s 2024-09-21T10:22:58.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:22:58 smithi070 ceph-mon[112139]: pgmap v1283: 105 pgs: 105 active+clean; 394 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.2 MiB/s rd, 11 MiB/s wr, 67 op/s 2024-09-21T10:23:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:00 smithi160 ceph-mon[101879]: pgmap v1284: 105 pgs: 105 active+clean; 394 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 11 MiB/s wr, 59 op/s 2024-09-21T10:23:00.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:00 smithi070 ceph-mon[107385]: pgmap v1284: 105 pgs: 105 active+clean; 394 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 11 MiB/s wr, 59 op/s 2024-09-21T10:23:00.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:00 smithi070 ceph-mon[112139]: pgmap v1284: 105 pgs: 105 active+clean; 394 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 11 MiB/s wr, 59 op/s 2024-09-21T10:23:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:02 smithi160 ceph-mon[101879]: pgmap v1285: 105 pgs: 105 active+clean; 403 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 9.8 MiB/s wr, 51 op/s 2024-09-21T10:23:02.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:02 smithi070 ceph-mon[107385]: pgmap v1285: 105 pgs: 105 active+clean; 403 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 9.8 MiB/s wr, 51 op/s 2024-09-21T10:23:02.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:02 smithi070 ceph-mon[112139]: pgmap v1285: 105 pgs: 105 active+clean; 403 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 9.8 MiB/s wr, 51 op/s 2024-09-21T10:23:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:23:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:23:03] "GET /metrics HTTP/1.1" 200 36470 "" "Prometheus/2.51.0" 2024-09-21T10:23:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:04 smithi160 ceph-mon[101879]: pgmap v1286: 105 pgs: 105 active+clean; 403 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.5 MiB/s wr, 52 op/s 2024-09-21T10:23:04.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:04 smithi070 ceph-mon[107385]: pgmap v1286: 105 pgs: 105 active+clean; 403 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.5 MiB/s wr, 52 op/s 2024-09-21T10:23:04.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:04 smithi070 ceph-mon[112139]: pgmap v1286: 105 pgs: 105 active+clean; 403 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.5 MiB/s wr, 52 op/s 2024-09-21T10:23:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:06 smithi160 ceph-mon[101879]: pgmap v1287: 105 pgs: 105 active+clean; 411 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 66 KiB/s rd, 3.7 MiB/s wr, 53 op/s 2024-09-21T10:23:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:06 smithi070 ceph-mon[107385]: pgmap v1287: 105 pgs: 105 active+clean; 411 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 66 KiB/s rd, 3.7 MiB/s wr, 53 op/s 2024-09-21T10:23:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:06.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:06 smithi070 ceph-mon[112139]: pgmap v1287: 105 pgs: 105 active+clean; 411 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 66 KiB/s rd, 3.7 MiB/s wr, 53 op/s 2024-09-21T10:23:06.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:08 smithi160 ceph-mon[101879]: pgmap v1288: 105 pgs: 105 active+clean; 419 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 54 KiB/s rd, 2.5 MiB/s wr, 34 op/s 2024-09-21T10:23:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:08 smithi070 ceph-mon[107385]: pgmap v1288: 105 pgs: 105 active+clean; 419 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 54 KiB/s rd, 2.5 MiB/s wr, 34 op/s 2024-09-21T10:23:08.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:08 smithi070 ceph-mon[112139]: pgmap v1288: 105 pgs: 105 active+clean; 419 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 54 KiB/s rd, 2.5 MiB/s wr, 34 op/s 2024-09-21T10:23:09.731 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/4.MinOverlap (17311 ms) 2024-09-21T10:23:09.731 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/4.ZeroOverlap 2024-09-21T10:23:09.731 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:23:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:10 smithi160 ceph-mon[101879]: pgmap v1289: 105 pgs: 105 active+clean; 419 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 46 KiB/s rd, 2.1 MiB/s wr, 29 op/s 2024-09-21T10:23:10.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:10 smithi070 ceph-mon[107385]: pgmap v1289: 105 pgs: 105 active+clean; 419 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 46 KiB/s rd, 2.1 MiB/s wr, 29 op/s 2024-09-21T10:23:10.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:10 smithi070 ceph-mon[112139]: pgmap v1289: 105 pgs: 105 active+clean; 419 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 46 KiB/s rd, 2.1 MiB/s wr, 29 op/s 2024-09-21T10:23:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:12 smithi160 ceph-mon[101879]: pgmap v1290: 105 pgs: 105 active+clean; 419 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.1 MiB/s wr, 32 op/s 2024-09-21T10:23:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:12 smithi070 ceph-mon[107385]: pgmap v1290: 105 pgs: 105 active+clean; 419 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.1 MiB/s wr, 32 op/s 2024-09-21T10:23:12.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:12 smithi070 ceph-mon[112139]: pgmap v1290: 105 pgs: 105 active+clean; 419 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.1 MiB/s wr, 32 op/s 2024-09-21T10:23:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:23:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:23:13] "GET /metrics HTTP/1.1" 200 36470 "" "Prometheus/2.51.0" 2024-09-21T10:23:14.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:14 smithi160 ceph-mon[101879]: pgmap v1291: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 39 op/s 2024-09-21T10:23:14.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:14 smithi160 ceph-mon[101879]: osdmap e962: 8 total, 8 up, 8 in 2024-09-21T10:23:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:14 smithi070 ceph-mon[107385]: pgmap v1291: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 39 op/s 2024-09-21T10:23:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:14 smithi070 ceph-mon[107385]: osdmap e962: 8 total, 8 up, 8 in 2024-09-21T10:23:14.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:14 smithi070 ceph-mon[112139]: pgmap v1291: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 39 op/s 2024-09-21T10:23:14.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:14 smithi070 ceph-mon[112139]: osdmap e962: 8 total, 8 up, 8 in 2024-09-21T10:23:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:16 smithi160 ceph-mon[101879]: pgmap v1293: 105 pgs: 105 active+clean; 443 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 5.2 MiB/s wr, 53 op/s 2024-09-21T10:23:16.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:16 smithi070 ceph-mon[107385]: pgmap v1293: 105 pgs: 105 active+clean; 443 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 5.2 MiB/s wr, 53 op/s 2024-09-21T10:23:16.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:16 smithi070 ceph-mon[112139]: pgmap v1293: 105 pgs: 105 active+clean; 443 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 5.2 MiB/s wr, 53 op/s 2024-09-21T10:23:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:18 smithi160 ceph-mon[101879]: pgmap v1294: 105 pgs: 105 active+clean; 452 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 5.3 MiB/s wr, 54 op/s 2024-09-21T10:23:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:18 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:23:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:18 smithi070 ceph-mon[107385]: pgmap v1294: 105 pgs: 105 active+clean; 452 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 5.3 MiB/s wr, 54 op/s 2024-09-21T10:23:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:18 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:23:18.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:18 smithi070 ceph-mon[112139]: pgmap v1294: 105 pgs: 105 active+clean; 452 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 5.3 MiB/s wr, 54 op/s 2024-09-21T10:23:18.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:18 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:23:20.241 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:20 smithi070 ceph-mon[107385]: pgmap v1295: 105 pgs: 105 active+clean; 452 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 5.3 MiB/s wr, 54 op/s 2024-09-21T10:23:20.241 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:20 smithi070 ceph-mon[112139]: pgmap v1295: 105 pgs: 105 active+clean; 452 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 5.3 MiB/s wr, 54 op/s 2024-09-21T10:23:20.535 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:20 smithi160 ceph-mon[101879]: pgmap v1295: 105 pgs: 105 active+clean; 452 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 5.3 MiB/s wr, 54 op/s 2024-09-21T10:23:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:21.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:21.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:22.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:22 smithi160 ceph-mon[101879]: pgmap v1296: 105 pgs: 105 active+clean; 452 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 4.1 MiB/s wr, 50 op/s 2024-09-21T10:23:22.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:22 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:23:22.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:22 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:23:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:22 smithi070 ceph-mon[107385]: pgmap v1296: 105 pgs: 105 active+clean; 452 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 4.1 MiB/s wr, 50 op/s 2024-09-21T10:23:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:22 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:23:22.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:22 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:23:22.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:22 smithi070 ceph-mon[112139]: pgmap v1296: 105 pgs: 105 active+clean; 452 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 4.1 MiB/s wr, 50 op/s 2024-09-21T10:23:22.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:22 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:23:22.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:22 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:23:23.407 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:23:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:23:23] "GET /metrics HTTP/1.1" 200 36473 "" "Prometheus/2.51.0" 2024-09-21T10:23:23.664 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:23 smithi070 ceph-mon[112139]: pgmap v1297: 105 pgs: 105 active+clean; 464 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 62 KiB/s rd, 4.1 MiB/s wr, 47 op/s 2024-09-21T10:23:23.664 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:23 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:23:23.664 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:23 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:23:23.664 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:23 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:23:23.665 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:23 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:23:23.665 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:23 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:23:23.665 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:23 smithi070 ceph-mon[107385]: pgmap v1297: 105 pgs: 105 active+clean; 464 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 62 KiB/s rd, 4.1 MiB/s wr, 47 op/s 2024-09-21T10:23:23.665 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:23 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:23:23.665 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:23 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:23:23.665 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:23 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:23:23.665 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:23 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:23:23.665 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:23 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:23:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:23 smithi160 ceph-mon[101879]: pgmap v1297: 105 pgs: 105 active+clean; 464 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 62 KiB/s rd, 4.1 MiB/s wr, 47 op/s 2024-09-21T10:23:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:23 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:23:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:23 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:23:23.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:23 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:23:23.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:23 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:23:23.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:23 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:23:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:25 smithi160 ceph-mon[101879]: pgmap v1298: 105 pgs: 105 active+clean; 468 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 84 KiB/s rd, 4.0 MiB/s wr, 46 op/s 2024-09-21T10:23:25.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:25 smithi070 ceph-mon[112139]: pgmap v1298: 105 pgs: 105 active+clean; 468 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 84 KiB/s rd, 4.0 MiB/s wr, 46 op/s 2024-09-21T10:23:25.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:25 smithi070 ceph-mon[107385]: pgmap v1298: 105 pgs: 105 active+clean; 468 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 84 KiB/s rd, 4.0 MiB/s wr, 46 op/s 2024-09-21T10:23:27.498 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/4.ZeroOverlap (17768 ms) 2024-09-21T10:23:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:27 smithi160 ceph-mon[101879]: pgmap v1299: 105 pgs: 105 active+clean; 468 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.1 MiB/s wr, 25 op/s 2024-09-21T10:23:27.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:27 smithi070 ceph-mon[107385]: pgmap v1299: 105 pgs: 105 active+clean; 468 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.1 MiB/s wr, 25 op/s 2024-09-21T10:23:27.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:27 smithi070 ceph-mon[112139]: pgmap v1299: 105 pgs: 105 active+clean; 468 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.1 MiB/s wr, 25 op/s 2024-09-21T10:23:28.425 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/4 (136197 ms total) 2024-09-21T10:23:28.425 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:23:28.425 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/5, where TypeParam = UnformattedLUKS2 2024-09-21T10:23:28.455 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/5.Simple 2024-09-21T10:23:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:29 smithi160 ceph-mon[101879]: pgmap v1300: 105 pgs: 105 active+clean; 468 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 18 op/s 2024-09-21T10:23:29.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:29 smithi160 ceph-mon[101879]: osdmap e963: 8 total, 8 up, 8 in 2024-09-21T10:23:29.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:29 smithi070 ceph-mon[107385]: pgmap v1300: 105 pgs: 105 active+clean; 468 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 18 op/s 2024-09-21T10:23:29.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:29 smithi070 ceph-mon[107385]: osdmap e963: 8 total, 8 up, 8 in 2024-09-21T10:23:29.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:29 smithi070 ceph-mon[112139]: pgmap v1300: 105 pgs: 105 active+clean; 468 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 18 op/s 2024-09-21T10:23:29.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:29 smithi070 ceph-mon[112139]: osdmap e963: 8 total, 8 up, 8 in 2024-09-21T10:23:30.440 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:23:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:30 smithi160 ceph-mon[101879]: osdmap e964: 8 total, 8 up, 8 in 2024-09-21T10:23:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:30 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4229070166' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-44","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:23:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:30 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-44","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:23:30.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:30 smithi070 ceph-mon[107385]: osdmap e964: 8 total, 8 up, 8 in 2024-09-21T10:23:30.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:30 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4229070166' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-44","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:23:30.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:30 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-44","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:23:30.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:30 smithi070 ceph-mon[112139]: osdmap e964: 8 total, 8 up, 8 in 2024-09-21T10:23:30.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:30 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4229070166' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-44","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:23:30.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:30 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-44","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:23:31.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:31 smithi160 ceph-mon[101879]: pgmap v1303: 105 pgs: 3 creating+peering, 29 unknown, 73 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:23:31.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:31 smithi160 ceph-mon[101879]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:23:31.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:31 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:23:31.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:31 smithi160 ceph-mon[101879]: osdmap e965: 8 total, 8 up, 8 in 2024-09-21T10:23:31.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:31 smithi070 ceph-mon[112139]: pgmap v1303: 105 pgs: 3 creating+peering, 29 unknown, 73 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:23:31.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:31 smithi070 ceph-mon[112139]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:23:31.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:31 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:23:31.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:31 smithi070 ceph-mon[112139]: osdmap e965: 8 total, 8 up, 8 in 2024-09-21T10:23:31.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:31 smithi070 ceph-mon[107385]: pgmap v1303: 105 pgs: 3 creating+peering, 29 unknown, 73 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:23:31.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:31 smithi070 ceph-mon[107385]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:23:31.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:31 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:23:31.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:31 smithi070 ceph-mon[107385]: osdmap e965: 8 total, 8 up, 8 in 2024-09-21T10:23:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:32 smithi160 ceph-mon[101879]: osdmap e966: 8 total, 8 up, 8 in 2024-09-21T10:23:32.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:32 smithi070 ceph-mon[107385]: osdmap e966: 8 total, 8 up, 8 in 2024-09-21T10:23:32.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:32 smithi070 ceph-mon[112139]: osdmap e966: 8 total, 8 up, 8 in 2024-09-21T10:23:33.443 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:23:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:23:33] "GET /metrics HTTP/1.1" 200 36473 "" "Prometheus/2.51.0" 2024-09-21T10:23:33.443 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:33 smithi070 ceph-mon[107385]: pgmap v1306: 105 pgs: 3 creating+peering, 13 unknown, 89 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:33.444 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:33 smithi070 ceph-mon[107385]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:23:33.444 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:33 smithi070 ceph-mon[107385]: Cluster is now healthy 2024-09-21T10:23:33.444 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:33 smithi070 ceph-mon[107385]: osdmap e967: 8 total, 8 up, 8 in 2024-09-21T10:23:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:33 smithi160 ceph-mon[101879]: pgmap v1306: 105 pgs: 3 creating+peering, 13 unknown, 89 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:33 smithi160 ceph-mon[101879]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:23:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:33 smithi160 ceph-mon[101879]: Cluster is now healthy 2024-09-21T10:23:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:33 smithi160 ceph-mon[101879]: osdmap e967: 8 total, 8 up, 8 in 2024-09-21T10:23:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:33 smithi070 ceph-mon[112139]: pgmap v1306: 105 pgs: 3 creating+peering, 13 unknown, 89 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:33.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:33 smithi070 ceph-mon[112139]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:23:33.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:33 smithi070 ceph-mon[112139]: Cluster is now healthy 2024-09-21T10:23:33.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:33 smithi070 ceph-mon[112139]: osdmap e967: 8 total, 8 up, 8 in 2024-09-21T10:23:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:35 smithi160 ceph-mon[101879]: pgmap v1308: 105 pgs: 3 creating+peering, 102 active+clean; 582 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 616 B/s rd, 411 B/s wr, 2 op/s 2024-09-21T10:23:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:35 smithi070 ceph-mon[107385]: pgmap v1308: 105 pgs: 3 creating+peering, 102 active+clean; 582 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 616 B/s rd, 411 B/s wr, 2 op/s 2024-09-21T10:23:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:35.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:35 smithi070 ceph-mon[112139]: pgmap v1308: 105 pgs: 3 creating+peering, 102 active+clean; 582 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 616 B/s rd, 411 B/s wr, 2 op/s 2024-09-21T10:23:35.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:37.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:37 smithi160 ceph-mon[101879]: pgmap v1309: 105 pgs: 105 active+clean; 582 KiB data, 687 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-09-21T10:23:37.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:37 smithi070 ceph-mon[107385]: pgmap v1309: 105 pgs: 105 active+clean; 582 KiB data, 687 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-09-21T10:23:37.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:37 smithi070 ceph-mon[112139]: pgmap v1309: 105 pgs: 105 active+clean; 582 KiB data, 687 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-09-21T10:23:39.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:38 smithi160 ceph-mon[101879]: osdmap e968: 8 total, 8 up, 8 in 2024-09-21T10:23:39.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:38 smithi070 ceph-mon[112139]: osdmap e968: 8 total, 8 up, 8 in 2024-09-21T10:23:39.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:38 smithi070 ceph-mon[107385]: osdmap e968: 8 total, 8 up, 8 in 2024-09-21T10:23:40.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:39 smithi160 ceph-mon[101879]: pgmap v1311: 105 pgs: 105 active+clean; 582 KiB data, 687 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-09-21T10:23:40.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:39 smithi070 ceph-mon[107385]: pgmap v1311: 105 pgs: 105 active+clean; 582 KiB data, 687 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-09-21T10:23:40.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:39 smithi070 ceph-mon[112139]: pgmap v1311: 105 pgs: 105 active+clean; 582 KiB data, 687 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-09-21T10:23:42.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:41 smithi160 ceph-mon[101879]: pgmap v1312: 105 pgs: 105 active+clean; 21 MiB data, 748 MiB used, 714 GiB / 715 GiB avail; 56 KiB/s rd, 2.5 MiB/s wr, 30 op/s 2024-09-21T10:23:42.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:41 smithi070 ceph-mon[107385]: pgmap v1312: 105 pgs: 105 active+clean; 21 MiB data, 748 MiB used, 714 GiB / 715 GiB avail; 56 KiB/s rd, 2.5 MiB/s wr, 30 op/s 2024-09-21T10:23:42.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:41 smithi070 ceph-mon[112139]: pgmap v1312: 105 pgs: 105 active+clean; 21 MiB data, 748 MiB used, 714 GiB / 715 GiB avail; 56 KiB/s rd, 2.5 MiB/s wr, 30 op/s 2024-09-21T10:23:43.537 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/5.Simple (15082 ms) 2024-09-21T10:23:43.538 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/5.Grow 2024-09-21T10:23:43.538 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:23:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:23:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:23:43] "GET /metrics HTTP/1.1" 200 36433 "" "Prometheus/2.51.0" 2024-09-21T10:23:44.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:43 smithi160 ceph-mon[101879]: pgmap v1313: 105 pgs: 105 active+clean; 42 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 63 KiB/s rd, 4.1 MiB/s wr, 47 op/s 2024-09-21T10:23:44.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:43 smithi070 ceph-mon[107385]: pgmap v1313: 105 pgs: 105 active+clean; 42 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 63 KiB/s rd, 4.1 MiB/s wr, 47 op/s 2024-09-21T10:23:44.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:43 smithi070 ceph-mon[112139]: pgmap v1313: 105 pgs: 105 active+clean; 42 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 63 KiB/s rd, 4.1 MiB/s wr, 47 op/s 2024-09-21T10:23:46.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:45 smithi160 ceph-mon[101879]: pgmap v1314: 105 pgs: 105 active+clean; 50 MiB data, 850 MiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 4.9 MiB/s wr, 56 op/s 2024-09-21T10:23:46.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:45 smithi070 ceph-mon[107385]: pgmap v1314: 105 pgs: 105 active+clean; 50 MiB data, 850 MiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 4.9 MiB/s wr, 56 op/s 2024-09-21T10:23:46.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:45 smithi070 ceph-mon[112139]: pgmap v1314: 105 pgs: 105 active+clean; 50 MiB data, 850 MiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 4.9 MiB/s wr, 56 op/s 2024-09-21T10:23:48.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:47 smithi160 ceph-mon[101879]: pgmap v1315: 105 pgs: 105 active+clean; 83 MiB data, 933 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 68 op/s 2024-09-21T10:23:48.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:47 smithi070 ceph-mon[107385]: pgmap v1315: 105 pgs: 105 active+clean; 83 MiB data, 933 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 68 op/s 2024-09-21T10:23:48.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:47 smithi070 ceph-mon[112139]: pgmap v1315: 105 pgs: 105 active+clean; 83 MiB data, 933 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 68 op/s 2024-09-21T10:23:50.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:49 smithi160 ceph-mon[101879]: pgmap v1316: 105 pgs: 105 active+clean; 83 MiB data, 933 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 7.8 MiB/s wr, 65 op/s 2024-09-21T10:23:50.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:49 smithi070 ceph-mon[107385]: pgmap v1316: 105 pgs: 105 active+clean; 83 MiB data, 933 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 7.8 MiB/s wr, 65 op/s 2024-09-21T10:23:50.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:49 smithi070 ceph-mon[112139]: pgmap v1316: 105 pgs: 105 active+clean; 83 MiB data, 933 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 7.8 MiB/s wr, 65 op/s 2024-09-21T10:23:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:51.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:51.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:52.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:51 smithi160 ceph-mon[101879]: pgmap v1317: 105 pgs: 105 active+clean; 83 MiB data, 933 MiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 6.8 MiB/s wr, 57 op/s 2024-09-21T10:23:52.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:51 smithi070 ceph-mon[107385]: pgmap v1317: 105 pgs: 105 active+clean; 83 MiB data, 933 MiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 6.8 MiB/s wr, 57 op/s 2024-09-21T10:23:52.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:51 smithi070 ceph-mon[112139]: pgmap v1317: 105 pgs: 105 active+clean; 83 MiB data, 933 MiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 6.8 MiB/s wr, 57 op/s 2024-09-21T10:23:53.278 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:52 smithi070 ceph-mon[107385]: osdmap e969: 8 total, 8 up, 8 in 2024-09-21T10:23:53.278 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:23:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:23:53] "GET /metrics HTTP/1.1" 200 36465 "" "Prometheus/2.51.0" 2024-09-21T10:23:53.278 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:52 smithi070 ceph-mon[112139]: osdmap e969: 8 total, 8 up, 8 in 2024-09-21T10:23:53.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:52 smithi160 ceph-mon[101879]: osdmap e969: 8 total, 8 up, 8 in 2024-09-21T10:23:54.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:53 smithi160 ceph-mon[101879]: pgmap v1319: 105 pgs: 105 active+clean; 83 MiB data, 933 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 26 op/s 2024-09-21T10:23:54.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:53 smithi070 ceph-mon[107385]: pgmap v1319: 105 pgs: 105 active+clean; 83 MiB data, 933 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 26 op/s 2024-09-21T10:23:54.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:53 smithi070 ceph-mon[112139]: pgmap v1319: 105 pgs: 105 active+clean; 83 MiB data, 933 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 26 op/s 2024-09-21T10:23:56.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:55 smithi160 ceph-mon[101879]: pgmap v1320: 105 pgs: 105 active+clean; 91 MiB data, 969 MiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 4.1 MiB/s wr, 36 op/s 2024-09-21T10:23:56.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:55 smithi070 ceph-mon[112139]: pgmap v1320: 105 pgs: 105 active+clean; 91 MiB data, 969 MiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 4.1 MiB/s wr, 36 op/s 2024-09-21T10:23:56.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:55 smithi070 ceph-mon[107385]: pgmap v1320: 105 pgs: 105 active+clean; 91 MiB data, 969 MiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 4.1 MiB/s wr, 36 op/s 2024-09-21T10:23:58.243 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/5.Grow (14706 ms) 2024-09-21T10:23:58.243 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/5.Shrink 2024-09-21T10:23:58.243 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:23:58.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:57 smithi160 ceph-mon[101879]: pgmap v1321: 105 pgs: 105 active+clean; 132 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 4.9 MiB/s wr, 51 op/s 2024-09-21T10:23:58.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:57 smithi070 ceph-mon[107385]: pgmap v1321: 105 pgs: 105 active+clean; 132 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 4.9 MiB/s wr, 51 op/s 2024-09-21T10:23:58.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:57 smithi070 ceph-mon[112139]: pgmap v1321: 105 pgs: 105 active+clean; 132 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 4.9 MiB/s wr, 51 op/s 2024-09-21T10:24:00.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:23:59 smithi160 ceph-mon[101879]: pgmap v1322: 105 pgs: 105 active+clean; 132 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 4.9 MiB/s wr, 51 op/s 2024-09-21T10:24:00.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:23:59 smithi070 ceph-mon[107385]: pgmap v1322: 105 pgs: 105 active+clean; 132 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 4.9 MiB/s wr, 51 op/s 2024-09-21T10:24:00.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:23:59 smithi070 ceph-mon[112139]: pgmap v1322: 105 pgs: 105 active+clean; 132 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 4.9 MiB/s wr, 51 op/s 2024-09-21T10:24:01.950 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:01 smithi070 ceph-mon[112139]: pgmap v1323: 105 pgs: 105 active+clean; 157 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.5 MiB/s rd, 7.4 MiB/s wr, 56 op/s 2024-09-21T10:24:01.951 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:01 smithi070 ceph-mon[107385]: pgmap v1323: 105 pgs: 105 active+clean; 157 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.5 MiB/s rd, 7.4 MiB/s wr, 56 op/s 2024-09-21T10:24:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:01 smithi160 ceph-mon[101879]: pgmap v1323: 105 pgs: 105 active+clean; 157 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.5 MiB/s rd, 7.4 MiB/s wr, 56 op/s 2024-09-21T10:24:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:24:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:24:03] "GET /metrics HTTP/1.1" 200 36465 "" "Prometheus/2.51.0" 2024-09-21T10:24:04.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:03 smithi160 ceph-mon[101879]: pgmap v1324: 105 pgs: 105 active+clean; 165 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 7.8 MiB/s wr, 67 op/s 2024-09-21T10:24:04.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:03 smithi070 ceph-mon[107385]: pgmap v1324: 105 pgs: 105 active+clean; 165 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 7.8 MiB/s wr, 67 op/s 2024-09-21T10:24:04.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:03 smithi070 ceph-mon[112139]: pgmap v1324: 105 pgs: 105 active+clean; 165 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 7.8 MiB/s wr, 67 op/s 2024-09-21T10:24:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:05 smithi160 ceph-mon[101879]: pgmap v1325: 105 pgs: 105 active+clean; 165 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 6.8 MiB/s wr, 58 op/s 2024-09-21T10:24:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:06.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:05 smithi070 ceph-mon[107385]: pgmap v1325: 105 pgs: 105 active+clean; 165 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 6.8 MiB/s wr, 58 op/s 2024-09-21T10:24:06.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:06.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:05 smithi070 ceph-mon[112139]: pgmap v1325: 105 pgs: 105 active+clean; 165 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 6.8 MiB/s wr, 58 op/s 2024-09-21T10:24:06.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:07.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:06 smithi160 ceph-mon[101879]: osdmap e970: 8 total, 8 up, 8 in 2024-09-21T10:24:07.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:06 smithi070 ceph-mon[107385]: osdmap e970: 8 total, 8 up, 8 in 2024-09-21T10:24:07.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:06 smithi070 ceph-mon[112139]: osdmap e970: 8 total, 8 up, 8 in 2024-09-21T10:24:08.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:07 smithi160 ceph-mon[101879]: pgmap v1327: 105 pgs: 105 active+clean; 185 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 5.3 MiB/s wr, 51 op/s 2024-09-21T10:24:08.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:07 smithi070 ceph-mon[107385]: pgmap v1327: 105 pgs: 105 active+clean; 185 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 5.3 MiB/s wr, 51 op/s 2024-09-21T10:24:08.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:07 smithi070 ceph-mon[112139]: pgmap v1327: 105 pgs: 105 active+clean; 185 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 5.3 MiB/s wr, 51 op/s 2024-09-21T10:24:10.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:09 smithi160 ceph-mon[101879]: pgmap v1328: 105 pgs: 105 active+clean; 185 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 5.3 MiB/s wr, 51 op/s 2024-09-21T10:24:10.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:09 smithi070 ceph-mon[107385]: pgmap v1328: 105 pgs: 105 active+clean; 185 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 5.3 MiB/s wr, 51 op/s 2024-09-21T10:24:10.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:09 smithi070 ceph-mon[112139]: pgmap v1328: 105 pgs: 105 active+clean; 185 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 5.3 MiB/s wr, 51 op/s 2024-09-21T10:24:12.245 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/5.Shrink (14002 ms) 2024-09-21T10:24:12.246 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/5.ShrinkToOne 2024-09-21T10:24:12.246 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:24:12.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:11 smithi160 ceph-mon[101879]: pgmap v1329: 105 pgs: 105 active+clean; 206 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s rd, 4.9 MiB/s wr, 56 op/s 2024-09-21T10:24:12.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:11 smithi070 ceph-mon[107385]: pgmap v1329: 105 pgs: 105 active+clean; 206 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s rd, 4.9 MiB/s wr, 56 op/s 2024-09-21T10:24:12.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:11 smithi070 ceph-mon[112139]: pgmap v1329: 105 pgs: 105 active+clean; 206 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s rd, 4.9 MiB/s wr, 56 op/s 2024-09-21T10:24:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:24:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:24:13] "GET /metrics HTTP/1.1" 200 36469 "" "Prometheus/2.51.0" 2024-09-21T10:24:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:13 smithi160 ceph-mon[101879]: pgmap v1330: 105 pgs: 105 active+clean; 226 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.5 MiB/s rd, 6.1 MiB/s wr, 45 op/s 2024-09-21T10:24:14.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:13 smithi070 ceph-mon[107385]: pgmap v1330: 105 pgs: 105 active+clean; 226 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.5 MiB/s rd, 6.1 MiB/s wr, 45 op/s 2024-09-21T10:24:14.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:13 smithi070 ceph-mon[112139]: pgmap v1330: 105 pgs: 105 active+clean; 226 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.5 MiB/s rd, 6.1 MiB/s wr, 45 op/s 2024-09-21T10:24:16.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:15 smithi160 ceph-mon[101879]: pgmap v1331: 105 pgs: 105 active+clean; 242 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 7.7 MiB/s wr, 64 op/s 2024-09-21T10:24:16.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:15 smithi070 ceph-mon[107385]: pgmap v1331: 105 pgs: 105 active+clean; 242 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 7.7 MiB/s wr, 64 op/s 2024-09-21T10:24:16.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:15 smithi070 ceph-mon[112139]: pgmap v1331: 105 pgs: 105 active+clean; 242 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 7.7 MiB/s wr, 64 op/s 2024-09-21T10:24:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:17 smithi160 ceph-mon[101879]: pgmap v1332: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 7.8 MiB/s wr, 68 op/s 2024-09-21T10:24:18.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:17 smithi070 ceph-mon[107385]: pgmap v1332: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 7.8 MiB/s wr, 68 op/s 2024-09-21T10:24:18.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:17 smithi070 ceph-mon[112139]: pgmap v1332: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 7.8 MiB/s wr, 68 op/s 2024-09-21T10:24:20.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:19 smithi160 ceph-mon[101879]: pgmap v1333: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 5.2 MiB/s wr, 32 op/s 2024-09-21T10:24:20.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:19 smithi070 ceph-mon[107385]: pgmap v1333: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 5.2 MiB/s wr, 32 op/s 2024-09-21T10:24:20.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:19 smithi070 ceph-mon[112139]: pgmap v1333: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 5.2 MiB/s wr, 32 op/s 2024-09-21T10:24:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:21.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:21.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:22.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:21 smithi160 ceph-mon[101879]: pgmap v1334: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 5.2 MiB/s wr, 32 op/s 2024-09-21T10:24:22.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:21 smithi160 ceph-mon[101879]: osdmap e971: 8 total, 8 up, 8 in 2024-09-21T10:24:22.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:21 smithi070 ceph-mon[107385]: pgmap v1334: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 5.2 MiB/s wr, 32 op/s 2024-09-21T10:24:22.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:21 smithi070 ceph-mon[107385]: osdmap e971: 8 total, 8 up, 8 in 2024-09-21T10:24:22.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:21 smithi070 ceph-mon[112139]: pgmap v1334: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 5.2 MiB/s wr, 32 op/s 2024-09-21T10:24:22.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:21 smithi070 ceph-mon[112139]: osdmap e971: 8 total, 8 up, 8 in 2024-09-21T10:24:23.278 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:22 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:24:23.278 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:24:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:24:23] "GET /metrics HTTP/1.1" 200 36470 "" "Prometheus/2.51.0" 2024-09-21T10:24:23.278 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:22 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:24:23.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:22 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:24:24.321 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:23 smithi160 ceph-mon[101879]: pgmap v1336: 105 pgs: 105 active+clean; 259 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 7.2 MiB/s rd, 3.3 MiB/s wr, 45 op/s 2024-09-21T10:24:24.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:23 smithi070 ceph-mon[107385]: pgmap v1336: 105 pgs: 105 active+clean; 259 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 7.2 MiB/s rd, 3.3 MiB/s wr, 45 op/s 2024-09-21T10:24:24.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:23 smithi070 ceph-mon[112139]: pgmap v1336: 105 pgs: 105 active+clean; 259 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 7.2 MiB/s rd, 3.3 MiB/s wr, 45 op/s 2024-09-21T10:24:26.275 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:25 smithi070 ceph-mon[107385]: pgmap v1337: 105 pgs: 105 active+clean; 280 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 29 op/s 2024-09-21T10:24:26.275 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:25 smithi070 ceph-mon[112139]: pgmap v1337: 105 pgs: 105 active+clean; 280 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 29 op/s 2024-09-21T10:24:26.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:25 smithi160 ceph-mon[101879]: pgmap v1337: 105 pgs: 105 active+clean; 280 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 29 op/s 2024-09-21T10:24:27.879 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/5.ShrinkToOne (15634 ms) 2024-09-21T10:24:27.879 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/5.ShrinkToOneAfterSnapshot 2024-09-21T10:24:27.879 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:24:28.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:27 smithi160 ceph-mon[101879]: pgmap v1338: 105 pgs: 105 active+clean; 300 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 505 KiB/s rd, 5.7 MiB/s wr, 50 op/s 2024-09-21T10:24:28.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:27 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:24:28.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:27 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:24:28.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:27 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:24:28.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:27 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:24:28.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:27 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:24:28.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:27 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:24:28.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:27 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:24:28.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:27 smithi070 ceph-mon[112139]: pgmap v1338: 105 pgs: 105 active+clean; 300 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 505 KiB/s rd, 5.7 MiB/s wr, 50 op/s 2024-09-21T10:24:28.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:27 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:24:28.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:27 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:24:28.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:27 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:24:28.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:27 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:24:28.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:27 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:24:28.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:27 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:24:28.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:27 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:24:28.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:27 smithi070 ceph-mon[107385]: pgmap v1338: 105 pgs: 105 active+clean; 300 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 505 KiB/s rd, 5.7 MiB/s wr, 50 op/s 2024-09-21T10:24:28.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:27 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:24:28.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:27 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:24:28.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:27 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:24:28.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:27 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:24:28.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:27 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:24:28.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:27 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:24:28.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:27 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:24:30.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:29 smithi160 ceph-mon[101879]: pgmap v1339: 105 pgs: 105 active+clean; 300 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 505 KiB/s rd, 5.7 MiB/s wr, 50 op/s 2024-09-21T10:24:30.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:29 smithi070 ceph-mon[107385]: pgmap v1339: 105 pgs: 105 active+clean; 300 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 505 KiB/s rd, 5.7 MiB/s wr, 50 op/s 2024-09-21T10:24:30.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:29 smithi070 ceph-mon[112139]: pgmap v1339: 105 pgs: 105 active+clean; 300 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 505 KiB/s rd, 5.7 MiB/s wr, 50 op/s 2024-09-21T10:24:32.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:31 smithi160 ceph-mon[101879]: pgmap v1340: 105 pgs: 105 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 57 op/s 2024-09-21T10:24:32.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:31 smithi070 ceph-mon[107385]: pgmap v1340: 105 pgs: 105 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 57 op/s 2024-09-21T10:24:32.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:31 smithi070 ceph-mon[112139]: pgmap v1340: 105 pgs: 105 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 57 op/s 2024-09-21T10:24:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:24:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:24:33] "GET /metrics HTTP/1.1" 200 36470 "" "Prometheus/2.51.0" 2024-09-21T10:24:34.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:33 smithi160 ceph-mon[101879]: pgmap v1341: 105 pgs: 105 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s rd, 5.3 MiB/s wr, 54 op/s 2024-09-21T10:24:34.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:33 smithi070 ceph-mon[107385]: pgmap v1341: 105 pgs: 105 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s rd, 5.3 MiB/s wr, 54 op/s 2024-09-21T10:24:34.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:33 smithi070 ceph-mon[112139]: pgmap v1341: 105 pgs: 105 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s rd, 5.3 MiB/s wr, 54 op/s 2024-09-21T10:24:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:35 smithi160 ceph-mon[101879]: pgmap v1342: 105 pgs: 105 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 35 op/s 2024-09-21T10:24:36.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:36.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:35 smithi070 ceph-mon[107385]: pgmap v1342: 105 pgs: 105 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 35 op/s 2024-09-21T10:24:36.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:36.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:35 smithi070 ceph-mon[112139]: pgmap v1342: 105 pgs: 105 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 35 op/s 2024-09-21T10:24:36.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:37.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:36 smithi160 ceph-mon[101879]: osdmap e972: 8 total, 8 up, 8 in 2024-09-21T10:24:37.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:36 smithi070 ceph-mon[107385]: osdmap e972: 8 total, 8 up, 8 in 2024-09-21T10:24:37.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:36 smithi070 ceph-mon[112139]: osdmap e972: 8 total, 8 up, 8 in 2024-09-21T10:24:38.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:37 smithi160 ceph-mon[101879]: pgmap v1344: 105 pgs: 105 active+clean; 325 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 33 op/s 2024-09-21T10:24:38.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:37 smithi070 ceph-mon[107385]: pgmap v1344: 105 pgs: 105 active+clean; 325 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 33 op/s 2024-09-21T10:24:38.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:37 smithi070 ceph-mon[112139]: pgmap v1344: 105 pgs: 105 active+clean; 325 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 33 op/s 2024-09-21T10:24:40.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:39 smithi160 ceph-mon[101879]: pgmap v1345: 105 pgs: 105 active+clean; 325 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 33 op/s 2024-09-21T10:24:40.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:39 smithi070 ceph-mon[107385]: pgmap v1345: 105 pgs: 105 active+clean; 325 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 33 op/s 2024-09-21T10:24:40.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:39 smithi070 ceph-mon[112139]: pgmap v1345: 105 pgs: 105 active+clean; 325 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 33 op/s 2024-09-21T10:24:42.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:41 smithi160 ceph-mon[101879]: pgmap v1346: 105 pgs: 105 active+clean; 349 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 92 KiB/s rd, 4.1 MiB/s wr, 46 op/s 2024-09-21T10:24:42.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:41 smithi160 ceph-mon[101879]: osdmap e973: 8 total, 8 up, 8 in 2024-09-21T10:24:42.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:41 smithi070 ceph-mon[112139]: pgmap v1346: 105 pgs: 105 active+clean; 349 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 92 KiB/s rd, 4.1 MiB/s wr, 46 op/s 2024-09-21T10:24:42.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:41 smithi070 ceph-mon[112139]: osdmap e973: 8 total, 8 up, 8 in 2024-09-21T10:24:42.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:41 smithi070 ceph-mon[107385]: pgmap v1346: 105 pgs: 105 active+clean; 349 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 92 KiB/s rd, 4.1 MiB/s wr, 46 op/s 2024-09-21T10:24:42.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:41 smithi070 ceph-mon[107385]: osdmap e973: 8 total, 8 up, 8 in 2024-09-21T10:24:43.124 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/5.ShrinkToOneAfterSnapshot (15244 ms) 2024-09-21T10:24:43.124 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/5.MinOverlap 2024-09-21T10:24:43.124 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:24:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:24:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:24:43] "GET /metrics HTTP/1.1" 200 36468 "" "Prometheus/2.51.0" 2024-09-21T10:24:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:43 smithi160 ceph-mon[101879]: pgmap v1348: 105 pgs: 105 active+clean; 381 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 9.1 MiB/s wr, 77 op/s 2024-09-21T10:24:44.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:43 smithi070 ceph-mon[107385]: pgmap v1348: 105 pgs: 105 active+clean; 381 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 9.1 MiB/s wr, 77 op/s 2024-09-21T10:24:44.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:43 smithi070 ceph-mon[112139]: pgmap v1348: 105 pgs: 105 active+clean; 381 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 9.1 MiB/s wr, 77 op/s 2024-09-21T10:24:46.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:45 smithi160 ceph-mon[101879]: pgmap v1349: 105 pgs: 105 active+clean; 418 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.9 MiB/s rd, 13 MiB/s wr, 79 op/s 2024-09-21T10:24:46.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:45 smithi070 ceph-mon[107385]: pgmap v1349: 105 pgs: 105 active+clean; 418 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.9 MiB/s rd, 13 MiB/s wr, 79 op/s 2024-09-21T10:24:46.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:45 smithi070 ceph-mon[112139]: pgmap v1349: 105 pgs: 105 active+clean; 418 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.9 MiB/s rd, 13 MiB/s wr, 79 op/s 2024-09-21T10:24:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:47 smithi160 ceph-mon[101879]: pgmap v1350: 105 pgs: 105 active+clean; 434 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.8 MiB/s rd, 11 MiB/s wr, 62 op/s 2024-09-21T10:24:48.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:47 smithi070 ceph-mon[107385]: pgmap v1350: 105 pgs: 105 active+clean; 434 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.8 MiB/s rd, 11 MiB/s wr, 62 op/s 2024-09-21T10:24:48.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:47 smithi070 ceph-mon[112139]: pgmap v1350: 105 pgs: 105 active+clean; 434 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.8 MiB/s rd, 11 MiB/s wr, 62 op/s 2024-09-21T10:24:50.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:49 smithi160 ceph-mon[101879]: pgmap v1351: 105 pgs: 105 active+clean; 434 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.8 MiB/s rd, 11 MiB/s wr, 62 op/s 2024-09-21T10:24:50.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:49 smithi070 ceph-mon[107385]: pgmap v1351: 105 pgs: 105 active+clean; 434 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.8 MiB/s rd, 11 MiB/s wr, 62 op/s 2024-09-21T10:24:50.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:49 smithi070 ceph-mon[112139]: pgmap v1351: 105 pgs: 105 active+clean; 434 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.8 MiB/s rd, 11 MiB/s wr, 62 op/s 2024-09-21T10:24:51.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:51.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:51.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:52.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:51 smithi160 ceph-mon[101879]: pgmap v1352: 105 pgs: 105 active+clean; 434 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.8 MiB/s rd, 8.5 MiB/s wr, 42 op/s 2024-09-21T10:24:52.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:51 smithi160 ceph-mon[101879]: osdmap e974: 8 total, 8 up, 8 in 2024-09-21T10:24:52.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:51 smithi070 ceph-mon[107385]: pgmap v1352: 105 pgs: 105 active+clean; 434 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.8 MiB/s rd, 8.5 MiB/s wr, 42 op/s 2024-09-21T10:24:52.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:51 smithi070 ceph-mon[107385]: osdmap e974: 8 total, 8 up, 8 in 2024-09-21T10:24:52.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:51 smithi070 ceph-mon[112139]: pgmap v1352: 105 pgs: 105 active+clean; 434 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.8 MiB/s rd, 8.5 MiB/s wr, 42 op/s 2024-09-21T10:24:52.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:51 smithi070 ceph-mon[112139]: osdmap e974: 8 total, 8 up, 8 in 2024-09-21T10:24:53.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:24:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:24:53] "GET /metrics HTTP/1.1" 200 36475 "" "Prometheus/2.51.0" 2024-09-21T10:24:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:53 smithi160 ceph-mon[101879]: pgmap v1354: 105 pgs: 105 active+clean; 455 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.4 MiB/s rd, 7.4 MiB/s wr, 36 op/s 2024-09-21T10:24:54.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:53 smithi070 ceph-mon[112139]: pgmap v1354: 105 pgs: 105 active+clean; 455 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.4 MiB/s rd, 7.4 MiB/s wr, 36 op/s 2024-09-21T10:24:54.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:53 smithi070 ceph-mon[107385]: pgmap v1354: 105 pgs: 105 active+clean; 455 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.4 MiB/s rd, 7.4 MiB/s wr, 36 op/s 2024-09-21T10:24:56.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:55 smithi160 ceph-mon[101879]: pgmap v1355: 105 pgs: 105 active+clean; 471 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 5.3 MiB/s wr, 57 op/s 2024-09-21T10:24:56.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:55 smithi070 ceph-mon[107385]: pgmap v1355: 105 pgs: 105 active+clean; 471 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 5.3 MiB/s wr, 57 op/s 2024-09-21T10:24:56.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:55 smithi070 ceph-mon[112139]: pgmap v1355: 105 pgs: 105 active+clean; 471 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 5.3 MiB/s wr, 57 op/s 2024-09-21T10:24:57.361 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/5.MinOverlap (14238 ms) 2024-09-21T10:24:57.361 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/5.ZeroOverlap 2024-09-21T10:24:57.361 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:24:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:57 smithi160 ceph-mon[101879]: pgmap v1356: 105 pgs: 105 active+clean; 491 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.1 MiB/s wr, 49 op/s 2024-09-21T10:24:58.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:57 smithi070 ceph-mon[107385]: pgmap v1356: 105 pgs: 105 active+clean; 491 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.1 MiB/s wr, 49 op/s 2024-09-21T10:24:58.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:57 smithi070 ceph-mon[112139]: pgmap v1356: 105 pgs: 105 active+clean; 491 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.1 MiB/s wr, 49 op/s 2024-09-21T10:25:00.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:24:59 smithi160 ceph-mon[101879]: pgmap v1357: 105 pgs: 105 active+clean; 491 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.1 MiB/s wr, 49 op/s 2024-09-21T10:25:00.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:24:59 smithi070 ceph-mon[107385]: pgmap v1357: 105 pgs: 105 active+clean; 491 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.1 MiB/s wr, 49 op/s 2024-09-21T10:25:00.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:24:59 smithi070 ceph-mon[112139]: pgmap v1357: 105 pgs: 105 active+clean; 491 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.1 MiB/s wr, 49 op/s 2024-09-21T10:25:02.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:01 smithi160 ceph-mon[101879]: pgmap v1358: 105 pgs: 105 active+clean; 495 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 6.5 MiB/s wr, 62 op/s 2024-09-21T10:25:02.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:01 smithi070 ceph-mon[107385]: pgmap v1358: 105 pgs: 105 active+clean; 495 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 6.5 MiB/s wr, 62 op/s 2024-09-21T10:25:02.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:01 smithi070 ceph-mon[112139]: pgmap v1358: 105 pgs: 105 active+clean; 495 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 6.5 MiB/s wr, 62 op/s 2024-09-21T10:25:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:25:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:25:03] "GET /metrics HTTP/1.1" 200 36475 "" "Prometheus/2.51.0" 2024-09-21T10:25:04.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:03 smithi160 ceph-mon[101879]: pgmap v1359: 105 pgs: 105 active+clean; 495 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 5.6 MiB/s wr, 55 op/s 2024-09-21T10:25:04.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:03 smithi070 ceph-mon[107385]: pgmap v1359: 105 pgs: 105 active+clean; 495 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 5.6 MiB/s wr, 55 op/s 2024-09-21T10:25:04.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:03 smithi070 ceph-mon[112139]: pgmap v1359: 105 pgs: 105 active+clean; 495 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 5.6 MiB/s wr, 55 op/s 2024-09-21T10:25:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:05 smithi160 ceph-mon[101879]: pgmap v1360: 105 pgs: 105 active+clean; 495 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.7 MiB/s wr, 41 op/s 2024-09-21T10:25:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:06.099 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:05 smithi070 ceph-mon[107385]: pgmap v1360: 105 pgs: 105 active+clean; 495 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.7 MiB/s wr, 41 op/s 2024-09-21T10:25:06.099 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:06.099 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:05 smithi070 ceph-mon[112139]: pgmap v1360: 105 pgs: 105 active+clean; 495 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 3.7 MiB/s wr, 41 op/s 2024-09-21T10:25:06.100 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:08.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:07 smithi160 ceph-mon[101879]: pgmap v1361: 105 pgs: 105 active+clean; 519 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 4.3 MiB/s wr, 26 op/s 2024-09-21T10:25:08.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:07 smithi160 ceph-mon[101879]: osdmap e975: 8 total, 8 up, 8 in 2024-09-21T10:25:08.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:07 smithi070 ceph-mon[107385]: pgmap v1361: 105 pgs: 105 active+clean; 519 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 4.3 MiB/s wr, 26 op/s 2024-09-21T10:25:08.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:07 smithi070 ceph-mon[107385]: osdmap e975: 8 total, 8 up, 8 in 2024-09-21T10:25:08.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:07 smithi070 ceph-mon[112139]: pgmap v1361: 105 pgs: 105 active+clean; 519 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.7 MiB/s rd, 4.3 MiB/s wr, 26 op/s 2024-09-21T10:25:08.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:07 smithi070 ceph-mon[112139]: osdmap e975: 8 total, 8 up, 8 in 2024-09-21T10:25:10.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:09 smithi070 ceph-mon[107385]: pgmap v1363: 105 pgs: 105 active+clean; 519 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 24 op/s 2024-09-21T10:25:10.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:09 smithi070 ceph-mon[112139]: pgmap v1363: 105 pgs: 105 active+clean; 519 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 24 op/s 2024-09-21T10:25:10.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:09 smithi160 ceph-mon[101879]: pgmap v1363: 105 pgs: 105 active+clean; 519 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 24 op/s 2024-09-21T10:25:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:11 smithi070 ceph-mon[107385]: pgmap v1364: 105 pgs: 105 active+clean; 536 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 433 KiB/s rd, 4.1 MiB/s wr, 32 op/s 2024-09-21T10:25:12.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:11 smithi070 ceph-mon[112139]: pgmap v1364: 105 pgs: 105 active+clean; 536 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 433 KiB/s rd, 4.1 MiB/s wr, 32 op/s 2024-09-21T10:25:12.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:11 smithi160 ceph-mon[101879]: pgmap v1364: 105 pgs: 105 active+clean; 536 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 433 KiB/s rd, 4.1 MiB/s wr, 32 op/s 2024-09-21T10:25:13.008 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/5.ZeroOverlap (15647 ms) 2024-09-21T10:25:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:25:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:25:13] "GET /metrics HTTP/1.1" 200 36471 "" "Prometheus/2.51.0" 2024-09-21T10:25:13.856 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/5 (104553 ms total) 2024-09-21T10:25:13.856 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:25:13.856 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/6, where TypeParam = LUKS2UnderLUKS2 2024-09-21T10:25:13.885 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/6.Simple 2024-09-21T10:25:14.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:13 smithi070 ceph-mon[107385]: pgmap v1365: 105 pgs: 105 active+clean; 544 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 52 op/s 2024-09-21T10:25:14.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:13 smithi070 ceph-mon[112139]: pgmap v1365: 105 pgs: 105 active+clean; 544 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 52 op/s 2024-09-21T10:25:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:13 smithi160 ceph-mon[101879]: pgmap v1365: 105 pgs: 105 active+clean; 544 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 52 op/s 2024-09-21T10:25:15.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:14 smithi070 ceph-mon[107385]: osdmap e976: 8 total, 8 up, 8 in 2024-09-21T10:25:15.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:14 smithi070 ceph-mon[112139]: osdmap e976: 8 total, 8 up, 8 in 2024-09-21T10:25:15.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:14 smithi160 ceph-mon[101879]: osdmap e976: 8 total, 8 up, 8 in 2024-09-21T10:25:15.872 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:25:16.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:15 smithi070 ceph-mon[107385]: pgmap v1367: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:25:16.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:15 smithi070 ceph-mon[107385]: osdmap e977: 8 total, 8 up, 8 in 2024-09-21T10:25:16.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:15 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3111611846' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-45","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:25:16.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:15 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-45","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:25:16.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:15 smithi070 ceph-mon[112139]: pgmap v1367: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:25:16.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:15 smithi070 ceph-mon[112139]: osdmap e977: 8 total, 8 up, 8 in 2024-09-21T10:25:16.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:15 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3111611846' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-45","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:25:16.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:15 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-45","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:25:16.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:15 smithi160 ceph-mon[101879]: pgmap v1367: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:25:16.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:15 smithi160 ceph-mon[101879]: osdmap e977: 8 total, 8 up, 8 in 2024-09-21T10:25:16.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:15 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3111611846' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-45","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:25:16.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:15 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-45","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:25:17.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:16 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:25:17.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:16 smithi070 ceph-mon[107385]: osdmap e978: 8 total, 8 up, 8 in 2024-09-21T10:25:17.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:16 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:25:17.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:16 smithi070 ceph-mon[112139]: osdmap e978: 8 total, 8 up, 8 in 2024-09-21T10:25:17.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:16 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:25:17.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:16 smithi160 ceph-mon[101879]: osdmap e978: 8 total, 8 up, 8 in 2024-09-21T10:25:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:17 smithi160 ceph-mon[101879]: pgmap v1370: 105 pgs: 14 unknown, 91 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:25:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:17 smithi160 ceph-mon[101879]: osdmap e979: 8 total, 8 up, 8 in 2024-09-21T10:25:18.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:17 smithi070 ceph-mon[107385]: pgmap v1370: 105 pgs: 14 unknown, 91 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:25:18.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:17 smithi070 ceph-mon[107385]: osdmap e979: 8 total, 8 up, 8 in 2024-09-21T10:25:18.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:17 smithi070 ceph-mon[112139]: pgmap v1370: 105 pgs: 14 unknown, 91 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:25:18.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:17 smithi070 ceph-mon[112139]: osdmap e979: 8 total, 8 up, 8 in 2024-09-21T10:25:19.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:18 smithi160 ceph-mon[101879]: osdmap e980: 8 total, 8 up, 8 in 2024-09-21T10:25:19.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:18 smithi070 ceph-mon[107385]: osdmap e980: 8 total, 8 up, 8 in 2024-09-21T10:25:19.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:18 smithi070 ceph-mon[112139]: osdmap e980: 8 total, 8 up, 8 in 2024-09-21T10:25:20.242 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:19 smithi160 ceph-mon[101879]: pgmap v1373: 105 pgs: 14 unknown, 91 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:25:20.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:19 smithi070 ceph-mon[107385]: pgmap v1373: 105 pgs: 14 unknown, 91 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:25:20.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:19 smithi070 ceph-mon[112139]: pgmap v1373: 105 pgs: 14 unknown, 91 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:25:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:21.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:21.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:22.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:21 smithi160 ceph-mon[101879]: pgmap v1374: 105 pgs: 105 active+clean; 582 KiB data, 723 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 734 B/s wr, 3 op/s 2024-09-21T10:25:22.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:21 smithi070 ceph-mon[107385]: pgmap v1374: 105 pgs: 105 active+clean; 582 KiB data, 723 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 734 B/s wr, 3 op/s 2024-09-21T10:25:22.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:21 smithi070 ceph-mon[112139]: pgmap v1374: 105 pgs: 105 active+clean; 582 KiB data, 723 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 734 B/s wr, 3 op/s 2024-09-21T10:25:23.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:25:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:25:23] "GET /metrics HTTP/1.1" 200 36407 "" "Prometheus/2.51.0" 2024-09-21T10:25:24.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:23 smithi160 ceph-mon[101879]: pgmap v1375: 105 pgs: 105 active+clean; 582 KiB data, 723 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-09-21T10:25:24.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:23 smithi160 ceph-mon[101879]: osdmap e981: 8 total, 8 up, 8 in 2024-09-21T10:25:24.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:23 smithi070 ceph-mon[107385]: pgmap v1375: 105 pgs: 105 active+clean; 582 KiB data, 723 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-09-21T10:25:24.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:23 smithi070 ceph-mon[107385]: osdmap e981: 8 total, 8 up, 8 in 2024-09-21T10:25:24.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:23 smithi070 ceph-mon[112139]: pgmap v1375: 105 pgs: 105 active+clean; 582 KiB data, 723 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-09-21T10:25:24.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:23 smithi070 ceph-mon[112139]: osdmap e981: 8 total, 8 up, 8 in 2024-09-21T10:25:26.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:25 smithi160 ceph-mon[101879]: pgmap v1377: 105 pgs: 105 active+clean; 582 KiB data, 723 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-09-21T10:25:26.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:25 smithi070 ceph-mon[107385]: pgmap v1377: 105 pgs: 105 active+clean; 582 KiB data, 723 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-09-21T10:25:26.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:25 smithi070 ceph-mon[112139]: pgmap v1377: 105 pgs: 105 active+clean; 582 KiB data, 723 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-09-21T10:25:27.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:26 smithi160 ceph-mon[101879]: osdmap e982: 8 total, 8 up, 8 in 2024-09-21T10:25:27.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:26 smithi070 ceph-mon[107385]: osdmap e982: 8 total, 8 up, 8 in 2024-09-21T10:25:27.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:26 smithi070 ceph-mon[112139]: osdmap e982: 8 total, 8 up, 8 in 2024-09-21T10:25:28.197 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:27 smithi160 ceph-mon[101879]: pgmap v1379: 105 pgs: 105 active+clean; 26 MiB data, 794 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 3.1 MiB/s wr, 28 op/s 2024-09-21T10:25:28.197 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:27 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:25:28.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:27 smithi070 ceph-mon[107385]: pgmap v1379: 105 pgs: 105 active+clean; 26 MiB data, 794 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 3.1 MiB/s wr, 28 op/s 2024-09-21T10:25:28.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:27 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:25:28.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:27 smithi070 ceph-mon[112139]: pgmap v1379: 105 pgs: 105 active+clean; 26 MiB data, 794 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 3.1 MiB/s wr, 28 op/s 2024-09-21T10:25:28.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:27 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:25:30.257 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:29 smithi160 ceph-mon[101879]: pgmap v1380: 105 pgs: 105 active+clean; 26 MiB data, 794 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 3.1 MiB/s wr, 26 op/s 2024-09-21T10:25:30.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:29 smithi070 ceph-mon[112139]: pgmap v1380: 105 pgs: 105 active+clean; 26 MiB data, 794 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 3.1 MiB/s wr, 26 op/s 2024-09-21T10:25:30.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:29 smithi070 ceph-mon[107385]: pgmap v1380: 105 pgs: 105 active+clean; 26 MiB data, 794 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 3.1 MiB/s wr, 26 op/s 2024-09-21T10:25:32.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:31 smithi160 ceph-mon[101879]: pgmap v1381: 105 pgs: 105 active+clean; 42 MiB data, 847 MiB used, 714 GiB / 715 GiB avail; 32 KiB/s rd, 5.1 MiB/s wr, 44 op/s 2024-09-21T10:25:32.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:25:32.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:25:32.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:25:32.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:25:32.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:25:32.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:25:32.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:25:32.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:31 smithi070 ceph-mon[107385]: pgmap v1381: 105 pgs: 105 active+clean; 42 MiB data, 847 MiB used, 714 GiB / 715 GiB avail; 32 KiB/s rd, 5.1 MiB/s wr, 44 op/s 2024-09-21T10:25:32.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:25:32.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:25:32.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:25:32.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:25:32.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:25:32.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:25:32.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:25:32.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:31 smithi070 ceph-mon[112139]: pgmap v1381: 105 pgs: 105 active+clean; 42 MiB data, 847 MiB used, 714 GiB / 715 GiB avail; 32 KiB/s rd, 5.1 MiB/s wr, 44 op/s 2024-09-21T10:25:32.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:25:32.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:25:32.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:25:32.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:25:32.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:25:32.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:25:32.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:25:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:25:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:25:33] "GET /metrics HTTP/1.1" 200 36407 "" "Prometheus/2.51.0" 2024-09-21T10:25:34.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:33 smithi160 ceph-mon[101879]: pgmap v1382: 105 pgs: 105 active+clean; 42 MiB data, 847 MiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 4.3 MiB/s wr, 37 op/s 2024-09-21T10:25:34.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:33 smithi070 ceph-mon[107385]: pgmap v1382: 105 pgs: 105 active+clean; 42 MiB data, 847 MiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 4.3 MiB/s wr, 37 op/s 2024-09-21T10:25:34.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:33 smithi070 ceph-mon[112139]: pgmap v1382: 105 pgs: 105 active+clean; 42 MiB data, 847 MiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 4.3 MiB/s wr, 37 op/s 2024-09-21T10:25:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:35 smithi160 ceph-mon[101879]: pgmap v1383: 105 pgs: 105 active+clean; 50 MiB data, 871 MiB used, 714 GiB / 715 GiB avail; 88 KiB/s rd, 4.9 MiB/s wr, 43 op/s 2024-09-21T10:25:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:36.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:35 smithi070 ceph-mon[107385]: pgmap v1383: 105 pgs: 105 active+clean; 50 MiB data, 871 MiB used, 714 GiB / 715 GiB avail; 88 KiB/s rd, 4.9 MiB/s wr, 43 op/s 2024-09-21T10:25:36.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:36.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:35 smithi070 ceph-mon[112139]: pgmap v1383: 105 pgs: 105 active+clean; 50 MiB data, 871 MiB used, 714 GiB / 715 GiB avail; 88 KiB/s rd, 4.9 MiB/s wr, 43 op/s 2024-09-21T10:25:36.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:37.349 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/6.Simple (23463 ms) 2024-09-21T10:25:37.355 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/6.Grow 2024-09-21T10:25:37.356 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:25:38.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:37 smithi160 ceph-mon[101879]: pgmap v1384: 105 pgs: 105 active+clean; 67 MiB data, 932 MiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 6.7 MiB/s wr, 47 op/s 2024-09-21T10:25:38.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:37 smithi070 ceph-mon[112139]: pgmap v1384: 105 pgs: 105 active+clean; 67 MiB data, 932 MiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 6.7 MiB/s wr, 47 op/s 2024-09-21T10:25:38.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:37 smithi070 ceph-mon[107385]: pgmap v1384: 105 pgs: 105 active+clean; 67 MiB data, 932 MiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 6.7 MiB/s wr, 47 op/s 2024-09-21T10:25:40.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:39 smithi160 ceph-mon[101879]: pgmap v1385: 105 pgs: 105 active+clean; 67 MiB data, 932 MiB used, 714 GiB / 715 GiB avail; 4.6 MiB/s rd, 3.8 MiB/s wr, 26 op/s 2024-09-21T10:25:40.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:39 smithi070 ceph-mon[107385]: pgmap v1385: 105 pgs: 105 active+clean; 67 MiB data, 932 MiB used, 714 GiB / 715 GiB avail; 4.6 MiB/s rd, 3.8 MiB/s wr, 26 op/s 2024-09-21T10:25:40.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:39 smithi070 ceph-mon[112139]: pgmap v1385: 105 pgs: 105 active+clean; 67 MiB data, 932 MiB used, 714 GiB / 715 GiB avail; 4.6 MiB/s rd, 3.8 MiB/s wr, 26 op/s 2024-09-21T10:25:42.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:41 smithi160 ceph-mon[101879]: pgmap v1386: 105 pgs: 105 active+clean; 83 MiB data, 965 MiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 6.1 MiB/s wr, 40 op/s 2024-09-21T10:25:42.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:41 smithi070 ceph-mon[107385]: pgmap v1386: 105 pgs: 105 active+clean; 83 MiB data, 965 MiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 6.1 MiB/s wr, 40 op/s 2024-09-21T10:25:42.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:41 smithi070 ceph-mon[112139]: pgmap v1386: 105 pgs: 105 active+clean; 83 MiB data, 965 MiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 6.1 MiB/s wr, 40 op/s 2024-09-21T10:25:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:25:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:25:43] "GET /metrics HTTP/1.1" 200 36471 "" "Prometheus/2.51.0" 2024-09-21T10:25:44.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:43 smithi160 ceph-mon[101879]: pgmap v1387: 105 pgs: 105 active+clean; 83 MiB data, 970 MiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 28 op/s 2024-09-21T10:25:44.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:43 smithi070 ceph-mon[107385]: pgmap v1387: 105 pgs: 105 active+clean; 83 MiB data, 970 MiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 28 op/s 2024-09-21T10:25:44.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:43 smithi070 ceph-mon[112139]: pgmap v1387: 105 pgs: 105 active+clean; 83 MiB data, 970 MiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 28 op/s 2024-09-21T10:25:46.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:45 smithi160 ceph-mon[101879]: pgmap v1388: 105 pgs: 105 active+clean; 83 MiB data, 970 MiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 4.8 MiB/s wr, 28 op/s 2024-09-21T10:25:46.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:45 smithi160 ceph-mon[101879]: osdmap e983: 8 total, 8 up, 8 in 2024-09-21T10:25:46.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:45 smithi070 ceph-mon[107385]: pgmap v1388: 105 pgs: 105 active+clean; 83 MiB data, 970 MiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 4.8 MiB/s wr, 28 op/s 2024-09-21T10:25:46.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:45 smithi070 ceph-mon[107385]: osdmap e983: 8 total, 8 up, 8 in 2024-09-21T10:25:46.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:45 smithi070 ceph-mon[112139]: pgmap v1388: 105 pgs: 105 active+clean; 83 MiB data, 970 MiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 4.8 MiB/s wr, 28 op/s 2024-09-21T10:25:46.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:45 smithi070 ceph-mon[112139]: osdmap e983: 8 total, 8 up, 8 in 2024-09-21T10:25:48.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:47 smithi160 ceph-mon[101879]: pgmap v1390: 105 pgs: 105 active+clean; 116 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 6.1 MiB/s wr, 43 op/s 2024-09-21T10:25:48.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:47 smithi070 ceph-mon[107385]: pgmap v1390: 105 pgs: 105 active+clean; 116 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 6.1 MiB/s wr, 43 op/s 2024-09-21T10:25:48.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:47 smithi070 ceph-mon[112139]: pgmap v1390: 105 pgs: 105 active+clean; 116 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 6.1 MiB/s wr, 43 op/s 2024-09-21T10:25:50.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:49 smithi160 ceph-mon[101879]: pgmap v1391: 105 pgs: 105 active+clean; 116 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 6.1 MiB/s wr, 43 op/s 2024-09-21T10:25:50.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:49 smithi070 ceph-mon[112139]: pgmap v1391: 105 pgs: 105 active+clean; 116 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 6.1 MiB/s wr, 43 op/s 2024-09-21T10:25:50.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:49 smithi070 ceph-mon[107385]: pgmap v1391: 105 pgs: 105 active+clean; 116 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 6.1 MiB/s wr, 43 op/s 2024-09-21T10:25:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:51.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:51.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:52.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:51 smithi160 ceph-mon[101879]: pgmap v1392: 105 pgs: 105 active+clean; 124 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 950 KiB/s rd, 4.1 MiB/s wr, 39 op/s 2024-09-21T10:25:52.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:51 smithi070 ceph-mon[112139]: pgmap v1392: 105 pgs: 105 active+clean; 124 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 950 KiB/s rd, 4.1 MiB/s wr, 39 op/s 2024-09-21T10:25:52.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:51 smithi070 ceph-mon[107385]: pgmap v1392: 105 pgs: 105 active+clean; 124 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 950 KiB/s rd, 4.1 MiB/s wr, 39 op/s 2024-09-21T10:25:53.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:25:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:25:53] "GET /metrics HTTP/1.1" 200 36485 "" "Prometheus/2.51.0" 2024-09-21T10:25:54.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:53 smithi070 ceph-mon[107385]: pgmap v1393: 105 pgs: 105 active+clean; 124 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:25:54.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:53 smithi070 ceph-mon[112139]: pgmap v1393: 105 pgs: 105 active+clean; 124 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:25:54.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:53 smithi160 ceph-mon[101879]: pgmap v1393: 105 pgs: 105 active+clean; 124 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:25:55.978 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:55 smithi070 ceph-mon[107385]: pgmap v1394: 105 pgs: 105 active+clean; 124 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:25:56.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:55 smithi160 ceph-mon[101879]: pgmap v1394: 105 pgs: 105 active+clean; 124 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:25:56.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:55 smithi070 ceph-mon[112139]: pgmap v1394: 105 pgs: 105 active+clean; 124 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:25:58.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:57 smithi160 ceph-mon[101879]: pgmap v1395: 105 pgs: 105 active+clean; 140 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 79 KiB/s rd, 5.0 MiB/s wr, 40 op/s 2024-09-21T10:25:58.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:57 smithi070 ceph-mon[107385]: pgmap v1395: 105 pgs: 105 active+clean; 140 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 79 KiB/s rd, 5.0 MiB/s wr, 40 op/s 2024-09-21T10:25:58.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:57 smithi070 ceph-mon[112139]: pgmap v1395: 105 pgs: 105 active+clean; 140 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 79 KiB/s rd, 5.0 MiB/s wr, 40 op/s 2024-09-21T10:25:58.901 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/6.Grow (21553 ms) 2024-09-21T10:25:58.901 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/6.Shrink 2024-09-21T10:25:58.902 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:26:00.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:25:59 smithi160 ceph-mon[101879]: pgmap v1396: 105 pgs: 105 active+clean; 140 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 60 KiB/s rd, 2.0 MiB/s wr, 19 op/s 2024-09-21T10:26:00.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:25:59 smithi070 ceph-mon[107385]: pgmap v1396: 105 pgs: 105 active+clean; 140 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 60 KiB/s rd, 2.0 MiB/s wr, 19 op/s 2024-09-21T10:26:00.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:25:59 smithi070 ceph-mon[112139]: pgmap v1396: 105 pgs: 105 active+clean; 140 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 60 KiB/s rd, 2.0 MiB/s wr, 19 op/s 2024-09-21T10:26:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:01 smithi160 ceph-mon[101879]: pgmap v1397: 105 pgs: 105 active+clean; 148 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 20 op/s 2024-09-21T10:26:02.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:01 smithi070 ceph-mon[107385]: pgmap v1397: 105 pgs: 105 active+clean; 148 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 20 op/s 2024-09-21T10:26:02.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:01 smithi070 ceph-mon[112139]: pgmap v1397: 105 pgs: 105 active+clean; 148 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 20 op/s 2024-09-21T10:26:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:26:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:26:03] "GET /metrics HTTP/1.1" 200 36485 "" "Prometheus/2.51.0" 2024-09-21T10:26:04.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:03 smithi160 ceph-mon[101879]: pgmap v1398: 105 pgs: 105 active+clean; 165 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 4.8 MiB/s wr, 29 op/s 2024-09-21T10:26:04.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:03 smithi070 ceph-mon[107385]: pgmap v1398: 105 pgs: 105 active+clean; 165 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 4.8 MiB/s wr, 29 op/s 2024-09-21T10:26:04.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:03 smithi070 ceph-mon[112139]: pgmap v1398: 105 pgs: 105 active+clean; 165 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 4.8 MiB/s wr, 29 op/s 2024-09-21T10:26:06.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:05 smithi070 ceph-mon[112139]: pgmap v1399: 105 pgs: 105 active+clean; 165 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 30 op/s 2024-09-21T10:26:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:06.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:05 smithi070 ceph-mon[107385]: pgmap v1399: 105 pgs: 105 active+clean; 165 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 30 op/s 2024-09-21T10:26:06.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:05 smithi160 ceph-mon[101879]: pgmap v1399: 105 pgs: 105 active+clean; 165 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 30 op/s 2024-09-21T10:26:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:08.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:07 smithi160 ceph-mon[101879]: pgmap v1400: 105 pgs: 105 active+clean; 165 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 30 op/s 2024-09-21T10:26:08.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:07 smithi160 ceph-mon[101879]: osdmap e984: 8 total, 8 up, 8 in 2024-09-21T10:26:08.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:07 smithi070 ceph-mon[112139]: pgmap v1400: 105 pgs: 105 active+clean; 165 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 30 op/s 2024-09-21T10:26:08.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:07 smithi070 ceph-mon[112139]: osdmap e984: 8 total, 8 up, 8 in 2024-09-21T10:26:08.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:07 smithi070 ceph-mon[107385]: pgmap v1400: 105 pgs: 105 active+clean; 165 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 30 op/s 2024-09-21T10:26:08.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:07 smithi070 ceph-mon[107385]: osdmap e984: 8 total, 8 up, 8 in 2024-09-21T10:26:10.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:09 smithi160 ceph-mon[101879]: pgmap v1402: 105 pgs: 105 active+clean; 165 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 25 op/s 2024-09-21T10:26:10.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:09 smithi070 ceph-mon[107385]: pgmap v1402: 105 pgs: 105 active+clean; 165 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 25 op/s 2024-09-21T10:26:10.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:09 smithi070 ceph-mon[112139]: pgmap v1402: 105 pgs: 105 active+clean; 165 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 25 op/s 2024-09-21T10:26:12.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:12 smithi160 ceph-mon[101879]: pgmap v1403: 105 pgs: 105 active+clean; 186 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 5.4 MiB/s wr, 25 op/s 2024-09-21T10:26:12.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:12 smithi070 ceph-mon[112139]: pgmap v1403: 105 pgs: 105 active+clean; 186 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 5.4 MiB/s wr, 25 op/s 2024-09-21T10:26:12.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:12 smithi070 ceph-mon[107385]: pgmap v1403: 105 pgs: 105 active+clean; 186 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 5.4 MiB/s wr, 25 op/s 2024-09-21T10:26:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:26:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:26:13] "GET /metrics HTTP/1.1" 200 36485 "" "Prometheus/2.51.0" 2024-09-21T10:26:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:14 smithi160 ceph-mon[101879]: pgmap v1404: 105 pgs: 105 active+clean; 206 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 845 KiB/s rd, 4.1 MiB/s wr, 36 op/s 2024-09-21T10:26:14.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:14 smithi070 ceph-mon[107385]: pgmap v1404: 105 pgs: 105 active+clean; 206 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 845 KiB/s rd, 4.1 MiB/s wr, 36 op/s 2024-09-21T10:26:14.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:14 smithi070 ceph-mon[112139]: pgmap v1404: 105 pgs: 105 active+clean; 206 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 845 KiB/s rd, 4.1 MiB/s wr, 36 op/s 2024-09-21T10:26:16.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:16 smithi160 ceph-mon[101879]: pgmap v1405: 105 pgs: 105 active+clean; 206 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:26:16.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:16 smithi070 ceph-mon[107385]: pgmap v1405: 105 pgs: 105 active+clean; 206 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:26:16.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:16 smithi070 ceph-mon[112139]: pgmap v1405: 105 pgs: 105 active+clean; 206 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:26:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:18 smithi160 ceph-mon[101879]: pgmap v1406: 105 pgs: 105 active+clean; 210 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 62 KiB/s rd, 4.5 MiB/s wr, 46 op/s 2024-09-21T10:26:18.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:18 smithi070 ceph-mon[107385]: pgmap v1406: 105 pgs: 105 active+clean; 210 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 62 KiB/s rd, 4.5 MiB/s wr, 46 op/s 2024-09-21T10:26:18.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:18 smithi070 ceph-mon[112139]: pgmap v1406: 105 pgs: 105 active+clean; 210 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 62 KiB/s rd, 4.5 MiB/s wr, 46 op/s 2024-09-21T10:26:20.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:20 smithi160 ceph-mon[101879]: pgmap v1407: 105 pgs: 105 active+clean; 210 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 54 KiB/s rd, 3.9 MiB/s wr, 40 op/s 2024-09-21T10:26:20.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:20 smithi070 ceph-mon[112139]: pgmap v1407: 105 pgs: 105 active+clean; 210 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 54 KiB/s rd, 3.9 MiB/s wr, 40 op/s 2024-09-21T10:26:20.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:20 smithi070 ceph-mon[107385]: pgmap v1407: 105 pgs: 105 active+clean; 210 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 54 KiB/s rd, 3.9 MiB/s wr, 40 op/s 2024-09-21T10:26:20.529 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/6.Shrink (21628 ms) 2024-09-21T10:26:20.529 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/6.ShrinkToOne 2024-09-21T10:26:20.529 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:26:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:21.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:21.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:22.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:22 smithi160 ceph-mon[101879]: pgmap v1408: 105 pgs: 105 active+clean; 230 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 6.4 MiB/s wr, 40 op/s 2024-09-21T10:26:22.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:22 smithi070 ceph-mon[107385]: pgmap v1408: 105 pgs: 105 active+clean; 230 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 6.4 MiB/s wr, 40 op/s 2024-09-21T10:26:22.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:22 smithi070 ceph-mon[112139]: pgmap v1408: 105 pgs: 105 active+clean; 230 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 6.4 MiB/s wr, 40 op/s 2024-09-21T10:26:23.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:26:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:26:23] "GET /metrics HTTP/1.1" 200 36487 "" "Prometheus/2.51.0" 2024-09-21T10:26:24.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:24 smithi160 ceph-mon[101879]: pgmap v1409: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 6.4 MiB/s wr, 55 op/s 2024-09-21T10:26:24.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:24 smithi070 ceph-mon[107385]: pgmap v1409: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 6.4 MiB/s wr, 55 op/s 2024-09-21T10:26:24.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:24 smithi070 ceph-mon[112139]: pgmap v1409: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 6.4 MiB/s wr, 55 op/s 2024-09-21T10:26:26.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:26 smithi160 ceph-mon[101879]: pgmap v1410: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 31 op/s 2024-09-21T10:26:26.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:26 smithi070 ceph-mon[107385]: pgmap v1410: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 31 op/s 2024-09-21T10:26:26.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:26 smithi070 ceph-mon[112139]: pgmap v1410: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 31 op/s 2024-09-21T10:26:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:28 smithi160 ceph-mon[101879]: pgmap v1411: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 31 op/s 2024-09-21T10:26:28.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:28 smithi070 ceph-mon[112139]: pgmap v1411: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 31 op/s 2024-09-21T10:26:28.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:28 smithi070 ceph-mon[107385]: pgmap v1411: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 31 op/s 2024-09-21T10:26:29.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:29 smithi160 ceph-mon[101879]: osdmap e985: 8 total, 8 up, 8 in 2024-09-21T10:26:29.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:29 smithi070 ceph-mon[107385]: osdmap e985: 8 total, 8 up, 8 in 2024-09-21T10:26:29.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:29 smithi070 ceph-mon[112139]: osdmap e985: 8 total, 8 up, 8 in 2024-09-21T10:26:30.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:30 smithi160 ceph-mon[101879]: pgmap v1413: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 5.3 MiB/s wr, 26 op/s 2024-09-21T10:26:30.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:30 smithi070 ceph-mon[107385]: pgmap v1413: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 5.3 MiB/s wr, 26 op/s 2024-09-21T10:26:30.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:30 smithi070 ceph-mon[112139]: pgmap v1413: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 5.3 MiB/s wr, 26 op/s 2024-09-21T10:26:32.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:32 smithi160 ceph-mon[101879]: pgmap v1414: 105 pgs: 105 active+clean; 268 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 7.2 MiB/s rd, 4.2 MiB/s wr, 57 op/s 2024-09-21T10:26:32.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:32 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:26:32.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:32 smithi070 ceph-mon[112139]: pgmap v1414: 105 pgs: 105 active+clean; 268 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 7.2 MiB/s rd, 4.2 MiB/s wr, 57 op/s 2024-09-21T10:26:32.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:32 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:26:32.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:32 smithi070 ceph-mon[107385]: pgmap v1414: 105 pgs: 105 active+clean; 268 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 7.2 MiB/s rd, 4.2 MiB/s wr, 57 op/s 2024-09-21T10:26:32.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:32 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:26:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:26:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:26:33] "GET /metrics HTTP/1.1" 200 36487 "" "Prometheus/2.51.0" 2024-09-21T10:26:34.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:34 smithi070 ceph-mon[112139]: pgmap v1415: 105 pgs: 105 active+clean; 288 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 848 KiB/s rd, 4.1 MiB/s wr, 39 op/s 2024-09-21T10:26:34.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:34 smithi070 ceph-mon[107385]: pgmap v1415: 105 pgs: 105 active+clean; 288 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 848 KiB/s rd, 4.1 MiB/s wr, 39 op/s 2024-09-21T10:26:34.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:34 smithi160 ceph-mon[101879]: pgmap v1415: 105 pgs: 105 active+clean; 288 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 848 KiB/s rd, 4.1 MiB/s wr, 39 op/s 2024-09-21T10:26:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:36 smithi160 ceph-mon[101879]: pgmap v1416: 105 pgs: 105 active+clean; 288 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:26:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:26:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:26:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:26:36.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:36 smithi070 ceph-mon[107385]: pgmap v1416: 105 pgs: 105 active+clean; 288 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:26:36.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:36.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:26:36.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:26:36.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:26:36.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:36 smithi070 ceph-mon[112139]: pgmap v1416: 105 pgs: 105 active+clean; 288 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:26:36.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:36.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:26:36.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:26:36.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:26:37.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:37 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:26:37.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:37 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:26:37.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:37 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:26:37.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:37 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:26:37.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:37 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:26:37.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:37 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:26:37.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:37 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:26:37.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:37 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:26:37.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:37 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:26:37.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:37 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:26:37.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:37 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:26:37.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:37 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:26:38.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:38 smithi160 ceph-mon[101879]: pgmap v1417: 105 pgs: 105 active+clean; 288 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:26:38.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:38 smithi070 ceph-mon[107385]: pgmap v1417: 105 pgs: 105 active+clean; 288 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:26:38.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:38 smithi070 ceph-mon[112139]: pgmap v1417: 105 pgs: 105 active+clean; 288 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:26:40.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:40 smithi160 ceph-mon[101879]: pgmap v1418: 105 pgs: 105 active+clean; 288 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 3.9 MiB/s wr, 33 op/s 2024-09-21T10:26:40.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:40 smithi070 ceph-mon[107385]: pgmap v1418: 105 pgs: 105 active+clean; 288 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 3.9 MiB/s wr, 33 op/s 2024-09-21T10:26:40.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:40 smithi070 ceph-mon[112139]: pgmap v1418: 105 pgs: 105 active+clean; 288 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 3.9 MiB/s wr, 33 op/s 2024-09-21T10:26:42.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:42 smithi160 ceph-mon[101879]: pgmap v1419: 105 pgs: 105 active+clean; 296 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 50 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:26:42.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:42 smithi070 ceph-mon[107385]: pgmap v1419: 105 pgs: 105 active+clean; 296 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 50 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:26:42.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:42 smithi070 ceph-mon[112139]: pgmap v1419: 105 pgs: 105 active+clean; 296 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 50 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:26:43.099 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/6.ShrinkToOne (22570 ms) 2024-09-21T10:26:43.099 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/6.ShrinkToOneAfterSnapshot 2024-09-21T10:26:43.100 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:26:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:26:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:26:43] "GET /metrics HTTP/1.1" 200 36489 "" "Prometheus/2.51.0" 2024-09-21T10:26:44.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:44 smithi160 ceph-mon[101879]: pgmap v1420: 105 pgs: 105 active+clean; 304 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 3.0 MiB/s wr, 11 op/s 2024-09-21T10:26:44.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:44 smithi070 ceph-mon[112139]: pgmap v1420: 105 pgs: 105 active+clean; 304 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 3.0 MiB/s wr, 11 op/s 2024-09-21T10:26:44.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:44 smithi070 ceph-mon[107385]: pgmap v1420: 105 pgs: 105 active+clean; 304 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 3.0 MiB/s wr, 11 op/s 2024-09-21T10:26:46.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:46 smithi160 ceph-mon[101879]: pgmap v1421: 105 pgs: 105 active+clean; 304 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 19 op/s 2024-09-21T10:26:46.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:46 smithi070 ceph-mon[107385]: pgmap v1421: 105 pgs: 105 active+clean; 304 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 19 op/s 2024-09-21T10:26:46.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:46 smithi070 ceph-mon[112139]: pgmap v1421: 105 pgs: 105 active+clean; 304 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 19 op/s 2024-09-21T10:26:48.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:48 smithi160 ceph-mon[101879]: pgmap v1422: 105 pgs: 105 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 23 op/s 2024-09-21T10:26:48.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:48 smithi070 ceph-mon[107385]: pgmap v1422: 105 pgs: 105 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 23 op/s 2024-09-21T10:26:48.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:48 smithi070 ceph-mon[112139]: pgmap v1422: 105 pgs: 105 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 23 op/s 2024-09-21T10:26:50.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:50 smithi160 ceph-mon[101879]: pgmap v1423: 105 pgs: 105 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 23 op/s 2024-09-21T10:26:50.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:50 smithi070 ceph-mon[112139]: pgmap v1423: 105 pgs: 105 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 23 op/s 2024-09-21T10:26:50.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:50 smithi070 ceph-mon[107385]: pgmap v1423: 105 pgs: 105 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 23 op/s 2024-09-21T10:26:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:51.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:51.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:52.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:52 smithi160 ceph-mon[101879]: pgmap v1424: 105 pgs: 105 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 23 op/s 2024-09-21T10:26:52.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:52 smithi070 ceph-mon[107385]: pgmap v1424: 105 pgs: 105 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 23 op/s 2024-09-21T10:26:52.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:52 smithi070 ceph-mon[112139]: pgmap v1424: 105 pgs: 105 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 23 op/s 2024-09-21T10:26:53.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:53 smithi160 ceph-mon[101879]: osdmap e986: 8 total, 8 up, 8 in 2024-09-21T10:26:53.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:26:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:26:53] "GET /metrics HTTP/1.1" 200 36479 "" "Prometheus/2.51.0" 2024-09-21T10:26:53.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:53 smithi070 ceph-mon[112139]: osdmap e986: 8 total, 8 up, 8 in 2024-09-21T10:26:53.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:53 smithi070 ceph-mon[107385]: osdmap e986: 8 total, 8 up, 8 in 2024-09-21T10:26:54.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:54 smithi160 ceph-mon[101879]: pgmap v1426: 105 pgs: 105 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2024-09-21T10:26:54.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:54 smithi070 ceph-mon[112139]: pgmap v1426: 105 pgs: 105 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2024-09-21T10:26:54.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:54 smithi070 ceph-mon[107385]: pgmap v1426: 105 pgs: 105 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2024-09-21T10:26:56.087 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:56 smithi070 ceph-mon[112139]: pgmap v1427: 105 pgs: 105 active+clean; 320 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 842 KiB/s rd, 2.8 MiB/s wr, 29 op/s 2024-09-21T10:26:56.087 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:56 smithi070 ceph-mon[107385]: pgmap v1427: 105 pgs: 105 active+clean; 320 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 842 KiB/s rd, 2.8 MiB/s wr, 29 op/s 2024-09-21T10:26:56.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:56 smithi160 ceph-mon[101879]: pgmap v1427: 105 pgs: 105 active+clean; 320 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 842 KiB/s rd, 2.8 MiB/s wr, 29 op/s 2024-09-21T10:26:58.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:26:58 smithi160 ceph-mon[101879]: pgmap v1428: 105 pgs: 105 active+clean; 349 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:26:58.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:26:58 smithi070 ceph-mon[107385]: pgmap v1428: 105 pgs: 105 active+clean; 349 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:26:58.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:26:58 smithi070 ceph-mon[112139]: pgmap v1428: 105 pgs: 105 active+clean; 349 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:27:00.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:00 smithi160 ceph-mon[101879]: pgmap v1429: 105 pgs: 105 active+clean; 349 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:27:00.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:00 smithi070 ceph-mon[107385]: pgmap v1429: 105 pgs: 105 active+clean; 349 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:27:00.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:00 smithi070 ceph-mon[112139]: pgmap v1429: 105 pgs: 105 active+clean; 349 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:27:02.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:02 smithi070 ceph-mon[107385]: pgmap v1430: 105 pgs: 105 active+clean; 357 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 4.9 MiB/s wr, 42 op/s 2024-09-21T10:27:02.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:02 smithi070 ceph-mon[112139]: pgmap v1430: 105 pgs: 105 active+clean; 357 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 4.9 MiB/s wr, 42 op/s 2024-09-21T10:27:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:02 smithi160 ceph-mon[101879]: pgmap v1430: 105 pgs: 105 active+clean; 357 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 4.9 MiB/s wr, 42 op/s 2024-09-21T10:27:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:27:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:27:03] "GET /metrics HTTP/1.1" 200 36479 "" "Prometheus/2.51.0" 2024-09-21T10:27:04.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:04 smithi070 ceph-mon[107385]: pgmap v1431: 105 pgs: 105 active+clean; 365 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 60 KiB/s rd, 5.5 MiB/s wr, 45 op/s 2024-09-21T10:27:04.414 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:04 smithi070 ceph-mon[107385]: osdmap e987: 8 total, 8 up, 8 in 2024-09-21T10:27:04.414 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:04 smithi070 ceph-mon[112139]: pgmap v1431: 105 pgs: 105 active+clean; 365 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 60 KiB/s rd, 5.5 MiB/s wr, 45 op/s 2024-09-21T10:27:04.414 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:04 smithi070 ceph-mon[112139]: osdmap e987: 8 total, 8 up, 8 in 2024-09-21T10:27:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:04 smithi160 ceph-mon[101879]: pgmap v1431: 105 pgs: 105 active+clean; 365 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 60 KiB/s rd, 5.5 MiB/s wr, 45 op/s 2024-09-21T10:27:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:04 smithi160 ceph-mon[101879]: osdmap e987: 8 total, 8 up, 8 in 2024-09-21T10:27:05.116 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/6.ShrinkToOneAfterSnapshot (22017 ms) 2024-09-21T10:27:05.116 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/6.MinOverlap 2024-09-21T10:27:05.117 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:27:06.103 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:06 smithi070 ceph-mon[112139]: pgmap v1433: 105 pgs: 105 active+clean; 389 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 6.9 MiB/s wr, 37 op/s 2024-09-21T10:27:06.104 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:06.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:06 smithi070 ceph-mon[107385]: pgmap v1433: 105 pgs: 105 active+clean; 389 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 6.9 MiB/s wr, 37 op/s 2024-09-21T10:27:06.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:06 smithi160 ceph-mon[101879]: pgmap v1433: 105 pgs: 105 active+clean; 389 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 6.9 MiB/s wr, 37 op/s 2024-09-21T10:27:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:08.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:08 smithi070 ceph-mon[107385]: pgmap v1434: 105 pgs: 105 active+clean; 434 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 8.5 MiB/s wr, 46 op/s 2024-09-21T10:27:08.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:08 smithi070 ceph-mon[112139]: pgmap v1434: 105 pgs: 105 active+clean; 434 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 8.5 MiB/s wr, 46 op/s 2024-09-21T10:27:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:08 smithi160 ceph-mon[101879]: pgmap v1434: 105 pgs: 105 active+clean; 434 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 8.5 MiB/s wr, 46 op/s 2024-09-21T10:27:10.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:10 smithi070 ceph-mon[107385]: pgmap v1435: 105 pgs: 105 active+clean; 434 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 8.5 MiB/s wr, 46 op/s 2024-09-21T10:27:10.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:10 smithi070 ceph-mon[112139]: pgmap v1435: 105 pgs: 105 active+clean; 434 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 8.5 MiB/s wr, 46 op/s 2024-09-21T10:27:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:10 smithi160 ceph-mon[101879]: pgmap v1435: 105 pgs: 105 active+clean; 434 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 8.5 MiB/s wr, 46 op/s 2024-09-21T10:27:12.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:12 smithi070 ceph-mon[107385]: pgmap v1436: 105 pgs: 105 active+clean; 434 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 7.7 MiB/s wr, 43 op/s 2024-09-21T10:27:12.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:12 smithi070 ceph-mon[112139]: pgmap v1436: 105 pgs: 105 active+clean; 434 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 7.7 MiB/s wr, 43 op/s 2024-09-21T10:27:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:12 smithi160 ceph-mon[101879]: pgmap v1436: 105 pgs: 105 active+clean; 434 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 7.7 MiB/s wr, 43 op/s 2024-09-21T10:27:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:27:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:27:13] "GET /metrics HTTP/1.1" 200 36484 "" "Prometheus/2.51.0" 2024-09-21T10:27:14.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:14 smithi070 ceph-mon[107385]: pgmap v1437: 105 pgs: 105 active+clean; 434 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 6.9 MiB/s wr, 38 op/s 2024-09-21T10:27:14.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:14 smithi070 ceph-mon[107385]: osdmap e988: 8 total, 8 up, 8 in 2024-09-21T10:27:14.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:14 smithi070 ceph-mon[112139]: pgmap v1437: 105 pgs: 105 active+clean; 434 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 6.9 MiB/s wr, 38 op/s 2024-09-21T10:27:14.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:14 smithi070 ceph-mon[112139]: osdmap e988: 8 total, 8 up, 8 in 2024-09-21T10:27:14.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:14 smithi160 ceph-mon[101879]: pgmap v1437: 105 pgs: 105 active+clean; 434 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 6.9 MiB/s wr, 38 op/s 2024-09-21T10:27:14.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:14 smithi160 ceph-mon[101879]: osdmap e988: 8 total, 8 up, 8 in 2024-09-21T10:27:16.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:16 smithi070 ceph-mon[107385]: pgmap v1439: 105 pgs: 105 active+clean; 450 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 6.1 MiB/s wr, 42 op/s 2024-09-21T10:27:16.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:16 smithi070 ceph-mon[112139]: pgmap v1439: 105 pgs: 105 active+clean; 450 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 6.1 MiB/s wr, 42 op/s 2024-09-21T10:27:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:16 smithi160 ceph-mon[101879]: pgmap v1439: 105 pgs: 105 active+clean; 450 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 6.1 MiB/s wr, 42 op/s 2024-09-21T10:27:18.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:18 smithi070 ceph-mon[107385]: pgmap v1440: 105 pgs: 105 active+clean; 475 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 4.1 MiB/s wr, 39 op/s 2024-09-21T10:27:18.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:18 smithi070 ceph-mon[112139]: pgmap v1440: 105 pgs: 105 active+clean; 475 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 4.1 MiB/s wr, 39 op/s 2024-09-21T10:27:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:18 smithi160 ceph-mon[101879]: pgmap v1440: 105 pgs: 105 active+clean; 475 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 4.1 MiB/s wr, 39 op/s 2024-09-21T10:27:20.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:20 smithi070 ceph-mon[107385]: pgmap v1441: 105 pgs: 105 active+clean; 475 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 4.1 MiB/s wr, 39 op/s 2024-09-21T10:27:20.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:20 smithi070 ceph-mon[112139]: pgmap v1441: 105 pgs: 105 active+clean; 475 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 4.1 MiB/s wr, 39 op/s 2024-09-21T10:27:20.393 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:20 smithi160 ceph-mon[101879]: pgmap v1441: 105 pgs: 105 active+clean; 475 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 4.1 MiB/s wr, 39 op/s 2024-09-21T10:27:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:21.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:21.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:22.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:22 smithi160 ceph-mon[101879]: pgmap v1442: 105 pgs: 105 active+clean; 475 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:27:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:22 smithi070 ceph-mon[107385]: pgmap v1442: 105 pgs: 105 active+clean; 475 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:27:22.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:22 smithi070 ceph-mon[112139]: pgmap v1442: 105 pgs: 105 active+clean; 475 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:27:23.528 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:27:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:27:23] "GET /metrics HTTP/1.1" 200 36481 "" "Prometheus/2.51.0" 2024-09-21T10:27:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:24 smithi160 ceph-mon[101879]: pgmap v1443: 105 pgs: 105 active+clean; 487 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 57 KiB/s rd, 5.3 MiB/s wr, 39 op/s 2024-09-21T10:27:24.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:24 smithi070 ceph-mon[107385]: pgmap v1443: 105 pgs: 105 active+clean; 487 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 57 KiB/s rd, 5.3 MiB/s wr, 39 op/s 2024-09-21T10:27:24.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:24 smithi070 ceph-mon[112139]: pgmap v1443: 105 pgs: 105 active+clean; 487 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 57 KiB/s rd, 5.3 MiB/s wr, 39 op/s 2024-09-21T10:27:26.530 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/6.MinOverlap (21414 ms) 2024-09-21T10:27:26.531 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/6.ZeroOverlap 2024-09-21T10:27:26.531 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:27:26.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:26 smithi160 ceph-mon[101879]: pgmap v1444: 105 pgs: 105 active+clean; 487 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 4.7 MiB/s wr, 41 op/s 2024-09-21T10:27:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:26 smithi070 ceph-mon[107385]: pgmap v1444: 105 pgs: 105 active+clean; 487 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 4.7 MiB/s wr, 41 op/s 2024-09-21T10:27:26.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:26 smithi070 ceph-mon[112139]: pgmap v1444: 105 pgs: 105 active+clean; 487 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 4.7 MiB/s wr, 41 op/s 2024-09-21T10:27:28.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:28 smithi160 ceph-mon[101879]: pgmap v1445: 105 pgs: 105 active+clean; 491 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 32 op/s 2024-09-21T10:27:28.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:28 smithi070 ceph-mon[112139]: pgmap v1445: 105 pgs: 105 active+clean; 491 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 32 op/s 2024-09-21T10:27:28.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:28 smithi070 ceph-mon[107385]: pgmap v1445: 105 pgs: 105 active+clean; 491 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 32 op/s 2024-09-21T10:27:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:30 smithi160 ceph-mon[101879]: pgmap v1446: 105 pgs: 105 active+clean; 491 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 18 op/s 2024-09-21T10:27:30.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:30 smithi070 ceph-mon[107385]: pgmap v1446: 105 pgs: 105 active+clean; 491 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 18 op/s 2024-09-21T10:27:30.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:30 smithi070 ceph-mon[112139]: pgmap v1446: 105 pgs: 105 active+clean; 491 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 18 op/s 2024-09-21T10:27:32.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:32 smithi160 ceph-mon[101879]: pgmap v1447: 105 pgs: 105 active+clean; 495 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 24 op/s 2024-09-21T10:27:32.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:32 smithi070 ceph-mon[107385]: pgmap v1447: 105 pgs: 105 active+clean; 495 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 24 op/s 2024-09-21T10:27:32.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:32 smithi070 ceph-mon[112139]: pgmap v1447: 105 pgs: 105 active+clean; 495 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 24 op/s 2024-09-21T10:27:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:27:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:27:33] "GET /metrics HTTP/1.1" 200 36481 "" "Prometheus/2.51.0" 2024-09-21T10:27:34.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:34 smithi160 ceph-mon[101879]: pgmap v1448: 105 pgs: 105 active+clean; 495 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 25 op/s 2024-09-21T10:27:34.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:34 smithi070 ceph-mon[107385]: pgmap v1448: 105 pgs: 105 active+clean; 495 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 25 op/s 2024-09-21T10:27:34.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:34 smithi070 ceph-mon[112139]: pgmap v1448: 105 pgs: 105 active+clean; 495 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 25 op/s 2024-09-21T10:27:35.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:35 smithi160 ceph-mon[101879]: osdmap e989: 8 total, 8 up, 8 in 2024-09-21T10:27:35.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:35 smithi070 ceph-mon[107385]: osdmap e989: 8 total, 8 up, 8 in 2024-09-21T10:27:35.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:35 smithi070 ceph-mon[112139]: osdmap e989: 8 total, 8 up, 8 in 2024-09-21T10:27:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:36 smithi160 ceph-mon[101879]: pgmap v1450: 105 pgs: 105 active+clean; 495 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.8 MiB/s wr, 30 op/s 2024-09-21T10:27:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:36 smithi070 ceph-mon[107385]: pgmap v1450: 105 pgs: 105 active+clean; 495 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.8 MiB/s wr, 30 op/s 2024-09-21T10:27:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:36 smithi070 ceph-mon[112139]: pgmap v1450: 105 pgs: 105 active+clean; 495 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 2.8 MiB/s wr, 30 op/s 2024-09-21T10:27:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:37.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:37 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:27:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:37 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:27:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:37 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:27:38.502 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:38 smithi070 ceph-mon[107385]: pgmap v1451: 105 pgs: 105 active+clean; 519 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 22 op/s 2024-09-21T10:27:38.502 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:38 smithi070 ceph-mon[112139]: pgmap v1451: 105 pgs: 105 active+clean; 519 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 22 op/s 2024-09-21T10:27:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:38 smithi160 ceph-mon[101879]: pgmap v1451: 105 pgs: 105 active+clean; 519 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 22 op/s 2024-09-21T10:27:40.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:40 smithi070 ceph-mon[107385]: pgmap v1452: 105 pgs: 105 active+clean; 519 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 22 op/s 2024-09-21T10:27:40.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:40 smithi070 ceph-mon[112139]: pgmap v1452: 105 pgs: 105 active+clean; 519 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 22 op/s 2024-09-21T10:27:40.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:40 smithi160 ceph-mon[101879]: pgmap v1452: 105 pgs: 105 active+clean; 519 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 22 op/s 2024-09-21T10:27:41.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:41 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:27:41.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:41 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:27:41.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:41 smithi160 ceph-mon[101879]: pgmap v1453: 105 pgs: 105 active+clean; 536 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 36 op/s 2024-09-21T10:27:41.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:41 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:27:41.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:41 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:27:41.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:41 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:27:41.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:41 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:27:41.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:41 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:27:41.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:41 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:27:41.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:41 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:27:41.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:41 smithi070 ceph-mon[107385]: pgmap v1453: 105 pgs: 105 active+clean; 536 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 36 op/s 2024-09-21T10:27:41.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:41 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:27:41.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:41 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:27:41.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:41 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:27:41.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:41 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:27:41.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:41 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:27:41.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:41 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:27:41.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:41 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:27:41.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:41 smithi070 ceph-mon[112139]: pgmap v1453: 105 pgs: 105 active+clean; 536 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 36 op/s 2024-09-21T10:27:41.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:41 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:27:41.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:41 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:27:41.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:41 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:27:41.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:41 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:27:41.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:41 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:27:43.530 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:27:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:27:43] "GET /metrics HTTP/1.1" 200 36483 "" "Prometheus/2.51.0" 2024-09-21T10:27:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:43 smithi160 ceph-mon[101879]: pgmap v1454: 105 pgs: 105 active+clean; 548 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 5.3 MiB/s wr, 40 op/s 2024-09-21T10:27:43.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:43 smithi070 ceph-mon[107385]: pgmap v1454: 105 pgs: 105 active+clean; 548 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 5.3 MiB/s wr, 40 op/s 2024-09-21T10:27:43.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:43 smithi070 ceph-mon[112139]: pgmap v1454: 105 pgs: 105 active+clean; 548 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 5.3 MiB/s wr, 40 op/s 2024-09-21T10:27:45.536 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/6.ZeroOverlap (19006 ms) 2024-09-21T10:27:45.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:45 smithi160 ceph-mon[101879]: pgmap v1455: 105 pgs: 105 active+clean; 552 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 486 KiB/s rd, 5.5 MiB/s wr, 47 op/s 2024-09-21T10:27:45.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:45 smithi070 ceph-mon[107385]: pgmap v1455: 105 pgs: 105 active+clean; 552 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 486 KiB/s rd, 5.5 MiB/s wr, 47 op/s 2024-09-21T10:27:45.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:45 smithi070 ceph-mon[112139]: pgmap v1455: 105 pgs: 105 active+clean; 552 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 486 KiB/s rd, 5.5 MiB/s wr, 47 op/s 2024-09-21T10:27:46.547 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/6 (151651 ms total) 2024-09-21T10:27:46.547 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:27:46.547 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/7, where TypeParam = LUKS2UnderLUKS1 2024-09-21T10:27:46.577 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/7.Simple 2024-09-21T10:27:47.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:47 smithi160 ceph-mon[101879]: pgmap v1456: 105 pgs: 105 active+clean; 552 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 MiB/s rd, 5.8 MiB/s wr, 35 op/s 2024-09-21T10:27:47.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:47 smithi160 ceph-mon[101879]: osdmap e990: 8 total, 8 up, 8 in 2024-09-21T10:27:47.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:47 smithi070 ceph-mon[107385]: pgmap v1456: 105 pgs: 105 active+clean; 552 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 MiB/s rd, 5.8 MiB/s wr, 35 op/s 2024-09-21T10:27:47.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:47 smithi070 ceph-mon[107385]: osdmap e990: 8 total, 8 up, 8 in 2024-09-21T10:27:47.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:47 smithi070 ceph-mon[112139]: pgmap v1456: 105 pgs: 105 active+clean; 552 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 MiB/s rd, 5.8 MiB/s wr, 35 op/s 2024-09-21T10:27:47.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:47 smithi070 ceph-mon[112139]: osdmap e990: 8 total, 8 up, 8 in 2024-09-21T10:27:48.558 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:27:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:48 smithi160 ceph-mon[101879]: osdmap e991: 8 total, 8 up, 8 in 2024-09-21T10:27:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:48 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/13797202' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-46","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:27:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:48 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-46","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:27:48.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:48 smithi070 ceph-mon[107385]: osdmap e991: 8 total, 8 up, 8 in 2024-09-21T10:27:48.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:48 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/13797202' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-46","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:27:48.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:48 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-46","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:27:48.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:48 smithi070 ceph-mon[112139]: osdmap e991: 8 total, 8 up, 8 in 2024-09-21T10:27:48.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:48 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/13797202' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-46","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:27:48.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:48 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-46","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:27:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:49 smithi160 ceph-mon[101879]: pgmap v1459: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:27:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:49 smithi160 ceph-mon[101879]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:27:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:49 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-46","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:27:49.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:49 smithi160 ceph-mon[101879]: osdmap e992: 8 total, 8 up, 8 in 2024-09-21T10:27:49.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:49 smithi070 ceph-mon[107385]: pgmap v1459: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:27:49.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:49 smithi070 ceph-mon[107385]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:27:49.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:49 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-46","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:27:49.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:49 smithi070 ceph-mon[107385]: osdmap e992: 8 total, 8 up, 8 in 2024-09-21T10:27:49.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:49 smithi070 ceph-mon[112139]: pgmap v1459: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:27:49.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:49 smithi070 ceph-mon[112139]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:27:49.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:49 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-46","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:27:49.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:49 smithi070 ceph-mon[112139]: osdmap e992: 8 total, 8 up, 8 in 2024-09-21T10:27:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:50 smithi160 ceph-mon[101879]: osdmap e993: 8 total, 8 up, 8 in 2024-09-21T10:27:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:27:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:50.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:50 smithi070 ceph-mon[107385]: osdmap e993: 8 total, 8 up, 8 in 2024-09-21T10:27:50.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:27:50.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:50.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:50 smithi070 ceph-mon[112139]: osdmap e993: 8 total, 8 up, 8 in 2024-09-21T10:27:50.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:27:50.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:51 smithi160 ceph-mon[101879]: pgmap v1462: 105 pgs: 21 unknown, 84 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:27:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:51 smithi160 ceph-mon[101879]: osdmap e994: 8 total, 8 up, 8 in 2024-09-21T10:27:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:51 smithi160 ceph-mon[101879]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:27:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:51 smithi160 ceph-mon[101879]: Cluster is now healthy 2024-09-21T10:27:51.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:51 smithi070 ceph-mon[107385]: pgmap v1462: 105 pgs: 21 unknown, 84 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:27:51.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:51 smithi070 ceph-mon[107385]: osdmap e994: 8 total, 8 up, 8 in 2024-09-21T10:27:51.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:51 smithi070 ceph-mon[107385]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:27:51.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:51 smithi070 ceph-mon[107385]: Cluster is now healthy 2024-09-21T10:27:51.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:51 smithi070 ceph-mon[112139]: pgmap v1462: 105 pgs: 21 unknown, 84 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:27:51.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:51 smithi070 ceph-mon[112139]: osdmap e994: 8 total, 8 up, 8 in 2024-09-21T10:27:51.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:51 smithi070 ceph-mon[112139]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:27:51.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:51 smithi070 ceph-mon[112139]: Cluster is now healthy 2024-09-21T10:27:53.569 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:53 smithi070 ceph-mon[107385]: pgmap v1464: 105 pgs: 105 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-09-21T10:27:53.569 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:53 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "50.14", "id": [6, 1]} : dispatch 2024-09-21T10:27:53.569 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:27:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:27:53] "GET /metrics HTTP/1.1" 200 36421 "" "Prometheus/2.51.0" 2024-09-21T10:27:53.570 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:53 smithi070 ceph-mon[112139]: pgmap v1464: 105 pgs: 105 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-09-21T10:27:53.570 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:53 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "50.14", "id": [6, 1]} : dispatch 2024-09-21T10:27:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:53 smithi160 ceph-mon[101879]: pgmap v1464: 105 pgs: 105 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-09-21T10:27:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:53 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "50.14", "id": [6, 1]} : dispatch 2024-09-21T10:27:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:54 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "50.14", "id": [6, 1]}]': finished 2024-09-21T10:27:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:54 smithi160 ceph-mon[101879]: osdmap e995: 8 total, 8 up, 8 in 2024-09-21T10:27:54.863 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:54 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "50.14", "id": [6, 1]}]': finished 2024-09-21T10:27:54.863 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:54 smithi070 ceph-mon[107385]: osdmap e995: 8 total, 8 up, 8 in 2024-09-21T10:27:54.864 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:54 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "50.14", "id": [6, 1]}]': finished 2024-09-21T10:27:54.864 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:54 smithi070 ceph-mon[112139]: osdmap e995: 8 total, 8 up, 8 in 2024-09-21T10:27:55.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:55 smithi160 ceph-mon[101879]: pgmap v1466: 105 pgs: 1 peering, 104 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-09-21T10:27:55.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:55 smithi160 ceph-mon[101879]: osdmap e996: 8 total, 8 up, 8 in 2024-09-21T10:27:55.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:55 smithi070 ceph-mon[107385]: pgmap v1466: 105 pgs: 1 peering, 104 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-09-21T10:27:55.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:55 smithi070 ceph-mon[107385]: osdmap e996: 8 total, 8 up, 8 in 2024-09-21T10:27:55.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:55 smithi070 ceph-mon[112139]: pgmap v1466: 105 pgs: 1 peering, 104 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-09-21T10:27:55.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:55 smithi070 ceph-mon[112139]: osdmap e996: 8 total, 8 up, 8 in 2024-09-21T10:27:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:57 smithi160 ceph-mon[101879]: pgmap v1468: 105 pgs: 1 peering, 104 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-09-21T10:27:57.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:57 smithi070 ceph-mon[107385]: pgmap v1468: 105 pgs: 1 peering, 104 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-09-21T10:27:57.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:57 smithi070 ceph-mon[112139]: pgmap v1468: 105 pgs: 1 peering, 104 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-09-21T10:27:59.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:58 smithi160 ceph-mon[101879]: osdmap e997: 8 total, 8 up, 8 in 2024-09-21T10:27:59.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:58 smithi070 ceph-mon[107385]: osdmap e997: 8 total, 8 up, 8 in 2024-09-21T10:27:59.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:58 smithi070 ceph-mon[112139]: osdmap e997: 8 total, 8 up, 8 in 2024-09-21T10:28:00.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:27:59 smithi160 ceph-mon[101879]: pgmap v1470: 105 pgs: 1 peering, 104 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 511 B/s wr, 5 op/s 2024-09-21T10:28:00.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:27:59 smithi070 ceph-mon[107385]: pgmap v1470: 105 pgs: 1 peering, 104 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 511 B/s wr, 5 op/s 2024-09-21T10:28:00.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:27:59 smithi070 ceph-mon[112139]: pgmap v1470: 105 pgs: 1 peering, 104 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 511 B/s wr, 5 op/s 2024-09-21T10:28:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:01 smithi160 ceph-mon[101879]: pgmap v1471: 105 pgs: 105 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 444 B/s wr, 4 op/s 2024-09-21T10:28:02.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:01 smithi070 ceph-mon[112139]: pgmap v1471: 105 pgs: 105 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 444 B/s wr, 4 op/s 2024-09-21T10:28:02.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:01 smithi070 ceph-mon[107385]: pgmap v1471: 105 pgs: 105 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 444 B/s wr, 4 op/s 2024-09-21T10:28:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:28:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:28:03] "GET /metrics HTTP/1.1" 200 36421 "" "Prometheus/2.51.0" 2024-09-21T10:28:04.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:03 smithi160 ceph-mon[101879]: pgmap v1472: 105 pgs: 105 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 127 B/s wr, 0 op/s 2024-09-21T10:28:04.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:03 smithi070 ceph-mon[107385]: pgmap v1472: 105 pgs: 105 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 127 B/s wr, 0 op/s 2024-09-21T10:28:04.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:03 smithi070 ceph-mon[112139]: pgmap v1472: 105 pgs: 105 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 127 B/s wr, 0 op/s 2024-09-21T10:28:05.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:04 smithi160 ceph-mon[101879]: osdmap e998: 8 total, 8 up, 8 in 2024-09-21T10:28:05.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:04 smithi070 ceph-mon[107385]: osdmap e998: 8 total, 8 up, 8 in 2024-09-21T10:28:05.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:04 smithi070 ceph-mon[112139]: osdmap e998: 8 total, 8 up, 8 in 2024-09-21T10:28:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:05 smithi160 ceph-mon[101879]: pgmap v1474: 105 pgs: 105 active+clean; 5.6 MiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 640 KiB/s wr, 2 op/s 2024-09-21T10:28:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:28:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:06.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:05 smithi070 ceph-mon[107385]: pgmap v1474: 105 pgs: 105 active+clean; 5.6 MiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 640 KiB/s wr, 2 op/s 2024-09-21T10:28:06.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:28:06.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:06.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:05 smithi070 ceph-mon[112139]: pgmap v1474: 105 pgs: 105 active+clean; 5.6 MiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 640 KiB/s wr, 2 op/s 2024-09-21T10:28:06.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:28:06.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:08.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:07 smithi160 ceph-mon[101879]: pgmap v1475: 105 pgs: 105 active+clean; 42 MiB data, 799 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 4.8 MiB/s wr, 24 op/s 2024-09-21T10:28:08.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:07 smithi070 ceph-mon[112139]: pgmap v1475: 105 pgs: 105 active+clean; 42 MiB data, 799 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 4.8 MiB/s wr, 24 op/s 2024-09-21T10:28:08.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:07 smithi070 ceph-mon[107385]: pgmap v1475: 105 pgs: 105 active+clean; 42 MiB data, 799 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 4.8 MiB/s wr, 24 op/s 2024-09-21T10:28:10.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:09 smithi160 ceph-mon[101879]: pgmap v1476: 105 pgs: 105 active+clean; 42 MiB data, 799 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 4.1 MiB/s wr, 20 op/s 2024-09-21T10:28:10.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:09 smithi070 ceph-mon[107385]: pgmap v1476: 105 pgs: 105 active+clean; 42 MiB data, 799 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 4.1 MiB/s wr, 20 op/s 2024-09-21T10:28:10.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:09 smithi070 ceph-mon[112139]: pgmap v1476: 105 pgs: 105 active+clean; 42 MiB data, 799 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 4.1 MiB/s wr, 20 op/s 2024-09-21T10:28:12.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:11 smithi160 ceph-mon[101879]: pgmap v1477: 105 pgs: 105 active+clean; 46 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 4.5 MiB/s wr, 35 op/s 2024-09-21T10:28:12.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:11 smithi070 ceph-mon[107385]: pgmap v1477: 105 pgs: 105 active+clean; 46 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 4.5 MiB/s wr, 35 op/s 2024-09-21T10:28:12.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:11 smithi070 ceph-mon[112139]: pgmap v1477: 105 pgs: 105 active+clean; 46 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 4.5 MiB/s wr, 35 op/s 2024-09-21T10:28:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:28:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:28:13] "GET /metrics HTTP/1.1" 200 36464 "" "Prometheus/2.51.0" 2024-09-21T10:28:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:13 smithi160 ceph-mon[101879]: pgmap v1478: 105 pgs: 105 active+clean; 46 MiB data, 816 MiB used, 714 GiB / 715 GiB avail; 84 KiB/s rd, 4.5 MiB/s wr, 37 op/s 2024-09-21T10:28:14.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:13 smithi070 ceph-mon[107385]: pgmap v1478: 105 pgs: 105 active+clean; 46 MiB data, 816 MiB used, 714 GiB / 715 GiB avail; 84 KiB/s rd, 4.5 MiB/s wr, 37 op/s 2024-09-21T10:28:14.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:13 smithi070 ceph-mon[112139]: pgmap v1478: 105 pgs: 105 active+clean; 46 MiB data, 816 MiB used, 714 GiB / 715 GiB avail; 84 KiB/s rd, 4.5 MiB/s wr, 37 op/s 2024-09-21T10:28:16.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:15 smithi160 ceph-mon[101879]: pgmap v1479: 105 pgs: 105 active+clean; 46 MiB data, 816 MiB used, 714 GiB / 715 GiB avail; 206 KiB/s rd, 3.8 MiB/s wr, 36 op/s 2024-09-21T10:28:16.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:15 smithi070 ceph-mon[107385]: pgmap v1479: 105 pgs: 105 active+clean; 46 MiB data, 816 MiB used, 714 GiB / 715 GiB avail; 206 KiB/s rd, 3.8 MiB/s wr, 36 op/s 2024-09-21T10:28:16.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:15 smithi070 ceph-mon[112139]: pgmap v1479: 105 pgs: 105 active+clean; 46 MiB data, 816 MiB used, 714 GiB / 715 GiB avail; 206 KiB/s rd, 3.8 MiB/s wr, 36 op/s 2024-09-21T10:28:17.752 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/7.Simple (31175 ms) 2024-09-21T10:28:17.753 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/7.Grow 2024-09-21T10:28:17.753 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:28:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:17 smithi160 ceph-mon[101879]: pgmap v1480: 105 pgs: 105 active+clean; 66 MiB data, 840 MiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 5.0 MiB/s wr, 48 op/s 2024-09-21T10:28:18.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:17 smithi070 ceph-mon[112139]: pgmap v1480: 105 pgs: 105 active+clean; 66 MiB data, 840 MiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 5.0 MiB/s wr, 48 op/s 2024-09-21T10:28:18.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:17 smithi070 ceph-mon[107385]: pgmap v1480: 105 pgs: 105 active+clean; 66 MiB data, 840 MiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 5.0 MiB/s wr, 48 op/s 2024-09-21T10:28:20.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:19 smithi160 ceph-mon[101879]: pgmap v1481: 105 pgs: 105 active+clean; 66 MiB data, 840 MiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.0 MiB/s wr, 32 op/s 2024-09-21T10:28:20.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:19 smithi070 ceph-mon[107385]: pgmap v1481: 105 pgs: 105 active+clean; 66 MiB data, 840 MiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.0 MiB/s wr, 32 op/s 2024-09-21T10:28:20.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:19 smithi070 ceph-mon[112139]: pgmap v1481: 105 pgs: 105 active+clean; 66 MiB data, 840 MiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 2.0 MiB/s wr, 32 op/s 2024-09-21T10:28:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:21.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:21.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:22.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:21 smithi160 ceph-mon[101879]: pgmap v1482: 105 pgs: 105 active+clean; 74 MiB data, 872 MiB used, 714 GiB / 715 GiB avail; 7.4 MiB/s rd, 3.0 MiB/s wr, 35 op/s 2024-09-21T10:28:22.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:21 smithi070 ceph-mon[107385]: pgmap v1482: 105 pgs: 105 active+clean; 74 MiB data, 872 MiB used, 714 GiB / 715 GiB avail; 7.4 MiB/s rd, 3.0 MiB/s wr, 35 op/s 2024-09-21T10:28:22.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:21 smithi070 ceph-mon[112139]: pgmap v1482: 105 pgs: 105 active+clean; 74 MiB data, 872 MiB used, 714 GiB / 715 GiB avail; 7.4 MiB/s rd, 3.0 MiB/s wr, 35 op/s 2024-09-21T10:28:23.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:28:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:28:23] "GET /metrics HTTP/1.1" 200 36473 "" "Prometheus/2.51.0" 2024-09-21T10:28:24.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:23 smithi160 ceph-mon[101879]: pgmap v1483: 105 pgs: 105 active+clean; 74 MiB data, 900 MiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 2.7 MiB/s wr, 26 op/s 2024-09-21T10:28:24.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:23 smithi070 ceph-mon[107385]: pgmap v1483: 105 pgs: 105 active+clean; 74 MiB data, 900 MiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 2.7 MiB/s wr, 26 op/s 2024-09-21T10:28:24.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:23 smithi070 ceph-mon[112139]: pgmap v1483: 105 pgs: 105 active+clean; 74 MiB data, 900 MiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 2.7 MiB/s wr, 26 op/s 2024-09-21T10:28:26.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:25 smithi160 ceph-mon[101879]: pgmap v1484: 105 pgs: 105 active+clean; 74 MiB data, 900 MiB used, 714 GiB / 715 GiB avail; 8.7 MiB/s rd, 2.7 MiB/s wr, 25 op/s 2024-09-21T10:28:26.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:25 smithi070 ceph-mon[107385]: pgmap v1484: 105 pgs: 105 active+clean; 74 MiB data, 900 MiB used, 714 GiB / 715 GiB avail; 8.7 MiB/s rd, 2.7 MiB/s wr, 25 op/s 2024-09-21T10:28:26.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:25 smithi070 ceph-mon[112139]: pgmap v1484: 105 pgs: 105 active+clean; 74 MiB data, 900 MiB used, 714 GiB / 715 GiB avail; 8.7 MiB/s rd, 2.7 MiB/s wr, 25 op/s 2024-09-21T10:28:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:28 smithi160 ceph-mon[101879]: pgmap v1485: 105 pgs: 105 active+clean; 74 MiB data, 900 MiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 2.7 MiB/s wr, 22 op/s 2024-09-21T10:28:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:28 smithi160 ceph-mon[101879]: osdmap e999: 8 total, 8 up, 8 in 2024-09-21T10:28:28.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:28 smithi070 ceph-mon[107385]: pgmap v1485: 105 pgs: 105 active+clean; 74 MiB data, 900 MiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 2.7 MiB/s wr, 22 op/s 2024-09-21T10:28:28.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:28 smithi070 ceph-mon[107385]: osdmap e999: 8 total, 8 up, 8 in 2024-09-21T10:28:28.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:28 smithi070 ceph-mon[112139]: pgmap v1485: 105 pgs: 105 active+clean; 74 MiB data, 900 MiB used, 714 GiB / 715 GiB avail; 8.6 MiB/s rd, 2.7 MiB/s wr, 22 op/s 2024-09-21T10:28:28.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:28 smithi070 ceph-mon[112139]: osdmap e999: 8 total, 8 up, 8 in 2024-09-21T10:28:30.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:30 smithi160 ceph-mon[101879]: pgmap v1487: 105 pgs: 105 active+clean; 74 MiB data, 900 MiB used, 714 GiB / 715 GiB avail; 6.1 MiB/s rd, 1.2 MiB/s wr, 7 op/s 2024-09-21T10:28:30.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:30 smithi070 ceph-mon[107385]: pgmap v1487: 105 pgs: 105 active+clean; 74 MiB data, 900 MiB used, 714 GiB / 715 GiB avail; 6.1 MiB/s rd, 1.2 MiB/s wr, 7 op/s 2024-09-21T10:28:30.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:30 smithi070 ceph-mon[112139]: pgmap v1487: 105 pgs: 105 active+clean; 74 MiB data, 900 MiB used, 714 GiB / 715 GiB avail; 6.1 MiB/s rd, 1.2 MiB/s wr, 7 op/s 2024-09-21T10:28:32.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:32 smithi160 ceph-mon[101879]: pgmap v1488: 105 pgs: 105 active+clean; 99 MiB data, 949 MiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 21 op/s 2024-09-21T10:28:32.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:32 smithi070 ceph-mon[107385]: pgmap v1488: 105 pgs: 105 active+clean; 99 MiB data, 949 MiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 21 op/s 2024-09-21T10:28:32.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:32 smithi070 ceph-mon[112139]: pgmap v1488: 105 pgs: 105 active+clean; 99 MiB data, 949 MiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 21 op/s 2024-09-21T10:28:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:28:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:28:33] "GET /metrics HTTP/1.1" 200 36473 "" "Prometheus/2.51.0" 2024-09-21T10:28:34.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:34 smithi160 ceph-mon[101879]: pgmap v1489: 105 pgs: 105 active+clean; 119 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 4.5 MiB/s wr, 36 op/s 2024-09-21T10:28:34.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:34 smithi070 ceph-mon[107385]: pgmap v1489: 105 pgs: 105 active+clean; 119 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 4.5 MiB/s wr, 36 op/s 2024-09-21T10:28:34.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:34 smithi070 ceph-mon[112139]: pgmap v1489: 105 pgs: 105 active+clean; 119 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 4.5 MiB/s wr, 36 op/s 2024-09-21T10:28:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:36 smithi160 ceph-mon[101879]: pgmap v1490: 105 pgs: 105 active+clean; 119 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 84 KiB/s rd, 4.5 MiB/s wr, 36 op/s 2024-09-21T10:28:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:36.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:36 smithi070 ceph-mon[107385]: pgmap v1490: 105 pgs: 105 active+clean; 119 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 84 KiB/s rd, 4.5 MiB/s wr, 36 op/s 2024-09-21T10:28:36.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:36.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:36 smithi070 ceph-mon[112139]: pgmap v1490: 105 pgs: 105 active+clean; 119 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 84 KiB/s rd, 4.5 MiB/s wr, 36 op/s 2024-09-21T10:28:36.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:38.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:38 smithi160 ceph-mon[101879]: pgmap v1491: 105 pgs: 105 active+clean; 123 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 119 KiB/s rd, 4.9 MiB/s wr, 45 op/s 2024-09-21T10:28:38.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:38 smithi070 ceph-mon[107385]: pgmap v1491: 105 pgs: 105 active+clean; 123 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 119 KiB/s rd, 4.9 MiB/s wr, 45 op/s 2024-09-21T10:28:38.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:38 smithi070 ceph-mon[112139]: pgmap v1491: 105 pgs: 105 active+clean; 123 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 119 KiB/s rd, 4.9 MiB/s wr, 45 op/s 2024-09-21T10:28:40.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:40 smithi160 ceph-mon[101879]: pgmap v1492: 105 pgs: 105 active+clean; 123 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 104 KiB/s rd, 4.3 MiB/s wr, 39 op/s 2024-09-21T10:28:40.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:40 smithi070 ceph-mon[112139]: pgmap v1492: 105 pgs: 105 active+clean; 123 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 104 KiB/s rd, 4.3 MiB/s wr, 39 op/s 2024-09-21T10:28:40.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:40 smithi070 ceph-mon[107385]: pgmap v1492: 105 pgs: 105 active+clean; 123 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 104 KiB/s rd, 4.3 MiB/s wr, 39 op/s 2024-09-21T10:28:40.793 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/7.Grow (23039 ms) 2024-09-21T10:28:40.793 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/7.Shrink 2024-09-21T10:28:40.793 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:28:41.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:41 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:28:41.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:41 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:28:41.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:41 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:28:42.313 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:42 smithi160 ceph-mon[101879]: pgmap v1493: 105 pgs: 105 active+clean; 135 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 5.1 MiB/s wr, 45 op/s 2024-09-21T10:28:42.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:42 smithi070 ceph-mon[112139]: pgmap v1493: 105 pgs: 105 active+clean; 135 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 5.1 MiB/s wr, 45 op/s 2024-09-21T10:28:42.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:42 smithi070 ceph-mon[107385]: pgmap v1493: 105 pgs: 105 active+clean; 135 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 5.1 MiB/s wr, 45 op/s 2024-09-21T10:28:43.437 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:28:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:28:43] "GET /metrics HTTP/1.1" 200 36485 "" "Prometheus/2.51.0" 2024-09-21T10:28:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:44 smithi160 ceph-mon[101879]: pgmap v1494: 105 pgs: 105 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.8 MiB/s rd, 4.3 MiB/s wr, 43 op/s 2024-09-21T10:28:44.365 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:44 smithi070 ceph-mon[107385]: pgmap v1494: 105 pgs: 105 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.8 MiB/s rd, 4.3 MiB/s wr, 43 op/s 2024-09-21T10:28:44.365 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:44 smithi070 ceph-mon[112139]: pgmap v1494: 105 pgs: 105 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.8 MiB/s rd, 4.3 MiB/s wr, 43 op/s 2024-09-21T10:28:46.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:45 smithi160 ceph-mon[101879]: pgmap v1495: 105 pgs: 105 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.7 MiB/s wr, 27 op/s 2024-09-21T10:28:46.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:45 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:28:46.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:45 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:28:46.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:45 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:28:46.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:45 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:28:46.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:45 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:28:46.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:45 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:28:46.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:45 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:28:46.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:45 smithi070 ceph-mon[107385]: pgmap v1495: 105 pgs: 105 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.7 MiB/s wr, 27 op/s 2024-09-21T10:28:46.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:45 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:28:46.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:45 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:28:46.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:45 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:28:46.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:45 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:28:46.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:45 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:28:46.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:45 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:28:46.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:45 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:28:46.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:45 smithi070 ceph-mon[112139]: pgmap v1495: 105 pgs: 105 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.7 MiB/s wr, 27 op/s 2024-09-21T10:28:46.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:45 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:28:46.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:45 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:28:46.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:45 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:28:46.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:45 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:28:46.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:45 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:28:46.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:45 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:28:46.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:45 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:28:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:47 smithi160 ceph-mon[101879]: pgmap v1496: 105 pgs: 105 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 2.7 MiB/s wr, 27 op/s 2024-09-21T10:28:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:47 smithi160 ceph-mon[101879]: osdmap e1000: 8 total, 8 up, 8 in 2024-09-21T10:28:48.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:47 smithi070 ceph-mon[107385]: pgmap v1496: 105 pgs: 105 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 2.7 MiB/s wr, 27 op/s 2024-09-21T10:28:48.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:47 smithi070 ceph-mon[107385]: osdmap e1000: 8 total, 8 up, 8 in 2024-09-21T10:28:48.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:47 smithi070 ceph-mon[112139]: pgmap v1496: 105 pgs: 105 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 2.7 MiB/s wr, 27 op/s 2024-09-21T10:28:48.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:47 smithi070 ceph-mon[112139]: osdmap e1000: 8 total, 8 up, 8 in 2024-09-21T10:28:50.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:49 smithi160 ceph-mon[101879]: pgmap v1498: 105 pgs: 105 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 2.8 MiB/s wr, 24 op/s 2024-09-21T10:28:50.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:49 smithi070 ceph-mon[107385]: pgmap v1498: 105 pgs: 105 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 2.8 MiB/s wr, 24 op/s 2024-09-21T10:28:50.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:49 smithi070 ceph-mon[112139]: pgmap v1498: 105 pgs: 105 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 2.8 MiB/s wr, 24 op/s 2024-09-21T10:28:51.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:51.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:51.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:52.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:51 smithi160 ceph-mon[101879]: pgmap v1499: 105 pgs: 105 active+clean; 163 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.7 MiB/s rd, 3.2 MiB/s wr, 21 op/s 2024-09-21T10:28:52.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:51 smithi070 ceph-mon[112139]: pgmap v1499: 105 pgs: 105 active+clean; 163 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.7 MiB/s rd, 3.2 MiB/s wr, 21 op/s 2024-09-21T10:28:52.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:51 smithi070 ceph-mon[107385]: pgmap v1499: 105 pgs: 105 active+clean; 163 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 6.7 MiB/s rd, 3.2 MiB/s wr, 21 op/s 2024-09-21T10:28:53.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:28:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:28:53] "GET /metrics HTTP/1.1" 200 36483 "" "Prometheus/2.51.0" 2024-09-21T10:28:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:53 smithi160 ceph-mon[101879]: pgmap v1500: 105 pgs: 105 active+clean; 188 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s rd, 4.1 MiB/s wr, 36 op/s 2024-09-21T10:28:54.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:53 smithi070 ceph-mon[107385]: pgmap v1500: 105 pgs: 105 active+clean; 188 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s rd, 4.1 MiB/s wr, 36 op/s 2024-09-21T10:28:54.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:53 smithi070 ceph-mon[112139]: pgmap v1500: 105 pgs: 105 active+clean; 188 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s rd, 4.1 MiB/s wr, 36 op/s 2024-09-21T10:28:56.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:55 smithi160 ceph-mon[101879]: pgmap v1501: 105 pgs: 105 active+clean; 192 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.5 MiB/s wr, 36 op/s 2024-09-21T10:28:56.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:55 smithi070 ceph-mon[107385]: pgmap v1501: 105 pgs: 105 active+clean; 192 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.5 MiB/s wr, 36 op/s 2024-09-21T10:28:56.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:55 smithi070 ceph-mon[112139]: pgmap v1501: 105 pgs: 105 active+clean; 192 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.5 MiB/s wr, 36 op/s 2024-09-21T10:28:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:57 smithi160 ceph-mon[101879]: pgmap v1502: 105 pgs: 105 active+clean; 196 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 193 KiB/s rd, 4.9 MiB/s wr, 46 op/s 2024-09-21T10:28:58.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:57 smithi070 ceph-mon[107385]: pgmap v1502: 105 pgs: 105 active+clean; 196 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 193 KiB/s rd, 4.9 MiB/s wr, 46 op/s 2024-09-21T10:28:58.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:57 smithi070 ceph-mon[112139]: pgmap v1502: 105 pgs: 105 active+clean; 196 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 193 KiB/s rd, 4.9 MiB/s wr, 46 op/s 2024-09-21T10:29:00.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:28:59 smithi160 ceph-mon[101879]: pgmap v1503: 105 pgs: 105 active+clean; 196 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 165 KiB/s rd, 4.2 MiB/s wr, 39 op/s 2024-09-21T10:29:00.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:28:59 smithi070 ceph-mon[107385]: pgmap v1503: 105 pgs: 105 active+clean; 196 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 165 KiB/s rd, 4.2 MiB/s wr, 39 op/s 2024-09-21T10:29:00.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:28:59 smithi070 ceph-mon[112139]: pgmap v1503: 105 pgs: 105 active+clean; 196 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 165 KiB/s rd, 4.2 MiB/s wr, 39 op/s 2024-09-21T10:29:00.272 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/7.Shrink (19481 ms) 2024-09-21T10:29:00.272 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/7.ShrinkToOne 2024-09-21T10:29:00.272 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:29:02.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:01 smithi160 ceph-mon[101879]: pgmap v1504: 105 pgs: 105 active+clean; 200 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 4.8 MiB/s wr, 50 op/s 2024-09-21T10:29:02.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:01 smithi070 ceph-mon[107385]: pgmap v1504: 105 pgs: 105 active+clean; 200 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 4.8 MiB/s wr, 50 op/s 2024-09-21T10:29:02.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:01 smithi070 ceph-mon[112139]: pgmap v1504: 105 pgs: 105 active+clean; 200 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 4.8 MiB/s wr, 50 op/s 2024-09-21T10:29:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:29:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:29:03] "GET /metrics HTTP/1.1" 200 36483 "" "Prometheus/2.51.0" 2024-09-21T10:29:04.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:03 smithi160 ceph-mon[101879]: pgmap v1505: 105 pgs: 105 active+clean; 220 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.5 MiB/s rd, 5.1 MiB/s wr, 52 op/s 2024-09-21T10:29:04.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:03 smithi070 ceph-mon[107385]: pgmap v1505: 105 pgs: 105 active+clean; 220 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.5 MiB/s rd, 5.1 MiB/s wr, 52 op/s 2024-09-21T10:29:04.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:03 smithi070 ceph-mon[112139]: pgmap v1505: 105 pgs: 105 active+clean; 220 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.5 MiB/s rd, 5.1 MiB/s wr, 52 op/s 2024-09-21T10:29:05.824 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:05 smithi070 ceph-mon[107385]: pgmap v1506: 105 pgs: 105 active+clean; 220 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 3.0 MiB/s wr, 28 op/s 2024-09-21T10:29:05.824 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:05.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:05 smithi070 ceph-mon[112139]: pgmap v1506: 105 pgs: 105 active+clean; 220 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 3.0 MiB/s wr, 28 op/s 2024-09-21T10:29:05.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:05 smithi160 ceph-mon[101879]: pgmap v1506: 105 pgs: 105 active+clean; 220 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 3.0 MiB/s wr, 28 op/s 2024-09-21T10:29:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:08.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:07 smithi160 ceph-mon[101879]: pgmap v1507: 105 pgs: 105 active+clean; 220 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 2.7 MiB/s wr, 28 op/s 2024-09-21T10:29:08.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:07 smithi070 ceph-mon[107385]: pgmap v1507: 105 pgs: 105 active+clean; 220 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 2.7 MiB/s wr, 28 op/s 2024-09-21T10:29:08.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:07 smithi070 ceph-mon[112139]: pgmap v1507: 105 pgs: 105 active+clean; 220 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 2.7 MiB/s wr, 28 op/s 2024-09-21T10:29:09.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:08 smithi160 ceph-mon[101879]: osdmap e1001: 8 total, 8 up, 8 in 2024-09-21T10:29:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:08 smithi070 ceph-mon[107385]: osdmap e1001: 8 total, 8 up, 8 in 2024-09-21T10:29:09.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:08 smithi070 ceph-mon[112139]: osdmap e1001: 8 total, 8 up, 8 in 2024-09-21T10:29:10.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:09 smithi160 ceph-mon[101879]: pgmap v1509: 105 pgs: 105 active+clean; 220 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 2.8 MiB/s wr, 24 op/s 2024-09-21T10:29:10.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:09 smithi070 ceph-mon[107385]: pgmap v1509: 105 pgs: 105 active+clean; 220 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 2.8 MiB/s wr, 24 op/s 2024-09-21T10:29:10.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:09 smithi070 ceph-mon[112139]: pgmap v1509: 105 pgs: 105 active+clean; 220 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 2.8 MiB/s wr, 24 op/s 2024-09-21T10:29:12.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:11 smithi160 ceph-mon[101879]: pgmap v1510: 105 pgs: 105 active+clean; 225 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 2.5 MiB/s wr, 10 op/s 2024-09-21T10:29:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:11 smithi070 ceph-mon[107385]: pgmap v1510: 105 pgs: 105 active+clean; 225 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 2.5 MiB/s wr, 10 op/s 2024-09-21T10:29:12.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:11 smithi070 ceph-mon[112139]: pgmap v1510: 105 pgs: 105 active+clean; 225 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 2.5 MiB/s wr, 10 op/s 2024-09-21T10:29:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:29:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:29:13] "GET /metrics HTTP/1.1" 200 36484 "" "Prometheus/2.51.0" 2024-09-21T10:29:14.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:13 smithi160 ceph-mon[101879]: pgmap v1511: 105 pgs: 105 active+clean; 261 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 435 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:29:14.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:13 smithi070 ceph-mon[107385]: pgmap v1511: 105 pgs: 105 active+clean; 261 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 435 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:29:14.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:13 smithi070 ceph-mon[112139]: pgmap v1511: 105 pgs: 105 active+clean; 261 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 435 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-09-21T10:29:16.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:15 smithi070 ceph-mon[107385]: pgmap v1512: 105 pgs: 105 active+clean; 261 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 4.1 MiB/s wr, 36 op/s 2024-09-21T10:29:16.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:15 smithi070 ceph-mon[112139]: pgmap v1512: 105 pgs: 105 active+clean; 261 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 4.1 MiB/s wr, 36 op/s 2024-09-21T10:29:16.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:15 smithi160 ceph-mon[101879]: pgmap v1512: 105 pgs: 105 active+clean; 261 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 4.1 MiB/s wr, 36 op/s 2024-09-21T10:29:18.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:17 smithi070 ceph-mon[107385]: pgmap v1513: 105 pgs: 105 active+clean; 269 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 118 KiB/s rd, 4.9 MiB/s wr, 44 op/s 2024-09-21T10:29:18.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:17 smithi070 ceph-mon[112139]: pgmap v1513: 105 pgs: 105 active+clean; 269 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 118 KiB/s rd, 4.9 MiB/s wr, 44 op/s 2024-09-21T10:29:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:17 smithi160 ceph-mon[101879]: pgmap v1513: 105 pgs: 105 active+clean; 269 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 118 KiB/s rd, 4.9 MiB/s wr, 44 op/s 2024-09-21T10:29:20.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:19 smithi070 ceph-mon[107385]: pgmap v1514: 105 pgs: 105 active+clean; 269 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 111 KiB/s rd, 4.6 MiB/s wr, 41 op/s 2024-09-21T10:29:20.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:19 smithi070 ceph-mon[112139]: pgmap v1514: 105 pgs: 105 active+clean; 269 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 111 KiB/s rd, 4.6 MiB/s wr, 41 op/s 2024-09-21T10:29:20.143 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:19 smithi160 ceph-mon[101879]: pgmap v1514: 105 pgs: 105 active+clean; 269 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 111 KiB/s rd, 4.6 MiB/s wr, 41 op/s 2024-09-21T10:29:21.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:21.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:21.662 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/7.ShrinkToOne (21390 ms) 2024-09-21T10:29:21.662 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/7.ShrinkToOneAfterSnapshot 2024-09-21T10:29:21.662 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:29:22.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:21 smithi070 ceph-mon[112139]: pgmap v1515: 105 pgs: 105 active+clean; 269 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 185 KiB/s rd, 4.1 MiB/s wr, 38 op/s 2024-09-21T10:29:22.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:21 smithi070 ceph-mon[107385]: pgmap v1515: 105 pgs: 105 active+clean; 269 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 185 KiB/s rd, 4.1 MiB/s wr, 38 op/s 2024-09-21T10:29:22.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:21 smithi160 ceph-mon[101879]: pgmap v1515: 105 pgs: 105 active+clean; 269 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 185 KiB/s rd, 4.1 MiB/s wr, 38 op/s 2024-09-21T10:29:23.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:29:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:29:23] "GET /metrics HTTP/1.1" 200 36480 "" "Prometheus/2.51.0" 2024-09-21T10:29:24.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:23 smithi070 ceph-mon[107385]: pgmap v1516: 105 pgs: 105 active+clean; 265 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 579 KiB/s rd, 4.0 MiB/s wr, 44 op/s 2024-09-21T10:29:24.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:23 smithi070 ceph-mon[112139]: pgmap v1516: 105 pgs: 105 active+clean; 265 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 579 KiB/s rd, 4.0 MiB/s wr, 44 op/s 2024-09-21T10:29:24.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:23 smithi160 ceph-mon[101879]: pgmap v1516: 105 pgs: 105 active+clean; 265 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 579 KiB/s rd, 4.0 MiB/s wr, 44 op/s 2024-09-21T10:29:26.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:25 smithi070 ceph-mon[107385]: pgmap v1517: 105 pgs: 105 active+clean; 269 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 901 KiB/s rd, 1.7 MiB/s wr, 20 op/s 2024-09-21T10:29:26.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:25 smithi070 ceph-mon[112139]: pgmap v1517: 105 pgs: 105 active+clean; 269 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 901 KiB/s rd, 1.7 MiB/s wr, 20 op/s 2024-09-21T10:29:26.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:25 smithi160 ceph-mon[101879]: pgmap v1517: 105 pgs: 105 active+clean; 269 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 901 KiB/s rd, 1.7 MiB/s wr, 20 op/s 2024-09-21T10:29:28.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:27 smithi070 ceph-mon[107385]: pgmap v1518: 105 pgs: 105 active+clean; 269 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 903 KiB/s rd, 1.7 MiB/s wr, 23 op/s 2024-09-21T10:29:28.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:27 smithi070 ceph-mon[112139]: pgmap v1518: 105 pgs: 105 active+clean; 269 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 903 KiB/s rd, 1.7 MiB/s wr, 23 op/s 2024-09-21T10:29:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:27 smithi160 ceph-mon[101879]: pgmap v1518: 105 pgs: 105 active+clean; 269 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 903 KiB/s rd, 1.7 MiB/s wr, 23 op/s 2024-09-21T10:29:30.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:29 smithi070 ceph-mon[107385]: pgmap v1519: 105 pgs: 105 active+clean; 269 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 827 KiB/s rd, 1.0 MiB/s wr, 16 op/s 2024-09-21T10:29:30.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:29 smithi070 ceph-mon[112139]: pgmap v1519: 105 pgs: 105 active+clean; 269 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 827 KiB/s rd, 1.0 MiB/s wr, 16 op/s 2024-09-21T10:29:30.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:29 smithi160 ceph-mon[101879]: pgmap v1519: 105 pgs: 105 active+clean; 269 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 827 KiB/s rd, 1.0 MiB/s wr, 16 op/s 2024-09-21T10:29:31.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:30 smithi070 ceph-mon[107385]: osdmap e1002: 8 total, 8 up, 8 in 2024-09-21T10:29:31.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:30 smithi070 ceph-mon[112139]: osdmap e1002: 8 total, 8 up, 8 in 2024-09-21T10:29:31.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:30 smithi160 ceph-mon[101879]: osdmap e1002: 8 total, 8 up, 8 in 2024-09-21T10:29:32.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:31 smithi070 ceph-mon[107385]: pgmap v1521: 105 pgs: 105 active+clean; 274 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 889 KiB/s rd, 1.7 MiB/s wr, 18 op/s 2024-09-21T10:29:32.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:31 smithi070 ceph-mon[112139]: pgmap v1521: 105 pgs: 105 active+clean; 274 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 889 KiB/s rd, 1.7 MiB/s wr, 18 op/s 2024-09-21T10:29:32.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:31 smithi160 ceph-mon[101879]: pgmap v1521: 105 pgs: 105 active+clean; 274 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 889 KiB/s rd, 1.7 MiB/s wr, 18 op/s 2024-09-21T10:29:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:29:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:29:33] "GET /metrics HTTP/1.1" 200 36480 "" "Prometheus/2.51.0" 2024-09-21T10:29:34.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:33 smithi070 ceph-mon[107385]: pgmap v1522: 105 pgs: 105 active+clean; 306 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 434 KiB/s rd, 4.5 MiB/s wr, 34 op/s 2024-09-21T10:29:34.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:33 smithi070 ceph-mon[112139]: pgmap v1522: 105 pgs: 105 active+clean; 306 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 434 KiB/s rd, 4.5 MiB/s wr, 34 op/s 2024-09-21T10:29:34.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:33 smithi160 ceph-mon[101879]: pgmap v1522: 105 pgs: 105 active+clean; 306 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 434 KiB/s rd, 4.5 MiB/s wr, 34 op/s 2024-09-21T10:29:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:35 smithi160 ceph-mon[101879]: pgmap v1523: 105 pgs: 105 active+clean; 310 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 4.1 MiB/s wr, 39 op/s 2024-09-21T10:29:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:36.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:35 smithi070 ceph-mon[107385]: pgmap v1523: 105 pgs: 105 active+clean; 310 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 4.1 MiB/s wr, 39 op/s 2024-09-21T10:29:36.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:36.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:35 smithi070 ceph-mon[112139]: pgmap v1523: 105 pgs: 105 active+clean; 310 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 4.1 MiB/s wr, 39 op/s 2024-09-21T10:29:36.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:38.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:37 smithi160 ceph-mon[101879]: pgmap v1524: 105 pgs: 105 active+clean; 314 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 84 KiB/s rd, 4.5 MiB/s wr, 37 op/s 2024-09-21T10:29:38.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:37 smithi070 ceph-mon[107385]: pgmap v1524: 105 pgs: 105 active+clean; 314 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 84 KiB/s rd, 4.5 MiB/s wr, 37 op/s 2024-09-21T10:29:38.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:37 smithi070 ceph-mon[112139]: pgmap v1524: 105 pgs: 105 active+clean; 314 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 84 KiB/s rd, 4.5 MiB/s wr, 37 op/s 2024-09-21T10:29:40.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:39 smithi160 ceph-mon[101879]: pgmap v1525: 105 pgs: 105 active+clean; 314 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 84 KiB/s rd, 4.5 MiB/s wr, 37 op/s 2024-09-21T10:29:40.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:39 smithi070 ceph-mon[107385]: pgmap v1525: 105 pgs: 105 active+clean; 314 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 84 KiB/s rd, 4.5 MiB/s wr, 37 op/s 2024-09-21T10:29:40.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:39 smithi070 ceph-mon[112139]: pgmap v1525: 105 pgs: 105 active+clean; 314 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 84 KiB/s rd, 4.5 MiB/s wr, 37 op/s 2024-09-21T10:29:42.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:41 smithi160 ceph-mon[101879]: pgmap v1526: 105 pgs: 105 active+clean; 314 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 175 KiB/s rd, 3.8 MiB/s wr, 35 op/s 2024-09-21T10:29:42.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:41 smithi070 ceph-mon[107385]: pgmap v1526: 105 pgs: 105 active+clean; 314 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 175 KiB/s rd, 3.8 MiB/s wr, 35 op/s 2024-09-21T10:29:42.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:41 smithi070 ceph-mon[112139]: pgmap v1526: 105 pgs: 105 active+clean; 314 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 175 KiB/s rd, 3.8 MiB/s wr, 35 op/s 2024-09-21T10:29:43.154 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:42 smithi070 ceph-mon[107385]: osdmap e1003: 8 total, 8 up, 8 in 2024-09-21T10:29:43.155 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:42 smithi070 ceph-mon[112139]: osdmap e1003: 8 total, 8 up, 8 in 2024-09-21T10:29:43.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:42 smithi160 ceph-mon[101879]: osdmap e1003: 8 total, 8 up, 8 in 2024-09-21T10:29:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:29:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:29:43] "GET /metrics HTTP/1.1" 200 36482 "" "Prometheus/2.51.0" 2024-09-21T10:29:44.158 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/7.ShrinkToOneAfterSnapshot (22496 ms) 2024-09-21T10:29:44.158 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/7.MinOverlap 2024-09-21T10:29:44.158 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:29:44.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:43 smithi160 ceph-mon[101879]: pgmap v1528: 105 pgs: 105 active+clean; 318 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 231 KiB/s rd, 1.2 MiB/s wr, 21 op/s 2024-09-21T10:29:44.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:43 smithi070 ceph-mon[107385]: pgmap v1528: 105 pgs: 105 active+clean; 318 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 231 KiB/s rd, 1.2 MiB/s wr, 21 op/s 2024-09-21T10:29:44.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:43 smithi070 ceph-mon[112139]: pgmap v1528: 105 pgs: 105 active+clean; 318 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 231 KiB/s rd, 1.2 MiB/s wr, 21 op/s 2024-09-21T10:29:46.178 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:45 smithi160 ceph-mon[101879]: pgmap v1529: 105 pgs: 105 active+clean; 326 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-21T10:29:46.178 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:45 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:29:46.178 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:45 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:29:46.178 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:45 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:29:46.361 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:45 smithi070 ceph-mon[107385]: pgmap v1529: 105 pgs: 105 active+clean; 326 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-21T10:29:46.362 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:45 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:29:46.362 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:45 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:29:46.362 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:45 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:29:46.362 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:45 smithi070 ceph-mon[112139]: pgmap v1529: 105 pgs: 105 active+clean; 326 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-21T10:29:46.362 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:45 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:29:46.362 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:45 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:29:46.362 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:45 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:29:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:47 smithi160 ceph-mon[101879]: pgmap v1530: 105 pgs: 105 active+clean; 374 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 6.0 MiB/s wr, 44 op/s 2024-09-21T10:29:48.326 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:47 smithi070 ceph-mon[107385]: pgmap v1530: 105 pgs: 105 active+clean; 374 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 6.0 MiB/s wr, 44 op/s 2024-09-21T10:29:48.326 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:47 smithi070 ceph-mon[112139]: pgmap v1530: 105 pgs: 105 active+clean; 374 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 6.0 MiB/s wr, 44 op/s 2024-09-21T10:29:50.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:49 smithi160 ceph-mon[101879]: pgmap v1531: 105 pgs: 105 active+clean; 374 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 6.0 MiB/s wr, 44 op/s 2024-09-21T10:29:50.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:49 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:29:50.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:49 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:29:50.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:49 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:29:50.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:49 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:29:50.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:49 smithi070 ceph-mon[107385]: pgmap v1531: 105 pgs: 105 active+clean; 374 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 6.0 MiB/s wr, 44 op/s 2024-09-21T10:29:50.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:49 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:29:50.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:49 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:29:50.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:49 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:29:50.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:49 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:29:50.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:49 smithi070 ceph-mon[112139]: pgmap v1531: 105 pgs: 105 active+clean; 374 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s rd, 6.0 MiB/s wr, 44 op/s 2024-09-21T10:29:50.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:49 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:29:50.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:49 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:29:50.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:49 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:29:50.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:49 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:29:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:29:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:29:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:29:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:51.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:29:51.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:29:51.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:29:51.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:51.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:29:51.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:29:51.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:29:51.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:52.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:51 smithi160 ceph-mon[101879]: pgmap v1532: 105 pgs: 105 active+clean; 374 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 6.0 MiB/s wr, 43 op/s 2024-09-21T10:29:52.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:51 smithi160 ceph-mon[101879]: osdmap e1004: 8 total, 8 up, 8 in 2024-09-21T10:29:52.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:51 smithi070 ceph-mon[107385]: pgmap v1532: 105 pgs: 105 active+clean; 374 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 6.0 MiB/s wr, 43 op/s 2024-09-21T10:29:52.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:51 smithi070 ceph-mon[107385]: osdmap e1004: 8 total, 8 up, 8 in 2024-09-21T10:29:52.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:51 smithi070 ceph-mon[112139]: pgmap v1532: 105 pgs: 105 active+clean; 374 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 6.0 MiB/s wr, 43 op/s 2024-09-21T10:29:52.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:51 smithi070 ceph-mon[112139]: osdmap e1004: 8 total, 8 up, 8 in 2024-09-21T10:29:53.537 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:29:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:29:53] "GET /metrics HTTP/1.1" 200 36480 "" "Prometheus/2.51.0" 2024-09-21T10:29:54.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:53 smithi160 ceph-mon[101879]: pgmap v1534: 105 pgs: 105 active+clean; 394 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 7.6 MiB/s wr, 67 op/s 2024-09-21T10:29:54.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:53 smithi070 ceph-mon[107385]: pgmap v1534: 105 pgs: 105 active+clean; 394 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 7.6 MiB/s wr, 67 op/s 2024-09-21T10:29:54.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:53 smithi070 ceph-mon[112139]: pgmap v1534: 105 pgs: 105 active+clean; 394 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 7.6 MiB/s wr, 67 op/s 2024-09-21T10:29:55.394 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 10:29:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=cleanup t=2024-09-21T10:29:55.129818371Z level=info msg="Completed cleanup jobs" duration=13.159502ms 2024-09-21T10:29:55.841 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 10:29:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=plugins.update.checker t=2024-09-21T10:29:55.393018571Z level=info msg="Update check succeeded" duration=186.804939ms 2024-09-21T10:29:56.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:55 smithi160 ceph-mon[101879]: pgmap v1535: 105 pgs: 105 active+clean; 410 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 8.4 MiB/s wr, 68 op/s 2024-09-21T10:29:56.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:55 smithi070 ceph-mon[107385]: pgmap v1535: 105 pgs: 105 active+clean; 410 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 8.4 MiB/s wr, 68 op/s 2024-09-21T10:29:56.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:55 smithi070 ceph-mon[112139]: pgmap v1535: 105 pgs: 105 active+clean; 410 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 8.4 MiB/s wr, 68 op/s 2024-09-21T10:29:58.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:57 smithi160 ceph-mon[101879]: pgmap v1536: 105 pgs: 105 active+clean; 419 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 4.5 MiB/s wr, 37 op/s 2024-09-21T10:29:58.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:57 smithi070 ceph-mon[107385]: pgmap v1536: 105 pgs: 105 active+clean; 419 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 4.5 MiB/s wr, 37 op/s 2024-09-21T10:29:58.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:57 smithi070 ceph-mon[112139]: pgmap v1536: 105 pgs: 105 active+clean; 419 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 4.5 MiB/s wr, 37 op/s 2024-09-21T10:30:00.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:29:59 smithi160 ceph-mon[101879]: pgmap v1537: 105 pgs: 105 active+clean; 419 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 4.5 MiB/s wr, 37 op/s 2024-09-21T10:30:00.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:29:59 smithi070 ceph-mon[107385]: pgmap v1537: 105 pgs: 105 active+clean; 419 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 4.5 MiB/s wr, 37 op/s 2024-09-21T10:30:00.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:29:59 smithi070 ceph-mon[112139]: pgmap v1537: 105 pgs: 105 active+clean; 419 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 4.5 MiB/s wr, 37 op/s 2024-09-21T10:30:01.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:00 smithi160 ceph-mon[101879]: overall HEALTH_OK 2024-09-21T10:30:01.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:00 smithi070 ceph-mon[107385]: overall HEALTH_OK 2024-09-21T10:30:01.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:00 smithi070 ceph-mon[112139]: overall HEALTH_OK 2024-09-21T10:30:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:01 smithi160 ceph-mon[101879]: pgmap v1538: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 187 KiB/s rd, 4.9 MiB/s wr, 38 op/s 2024-09-21T10:30:02.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:01 smithi070 ceph-mon[107385]: pgmap v1538: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 187 KiB/s rd, 4.9 MiB/s wr, 38 op/s 2024-09-21T10:30:02.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:01 smithi070 ceph-mon[112139]: pgmap v1538: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 187 KiB/s rd, 4.9 MiB/s wr, 38 op/s 2024-09-21T10:30:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:30:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:30:03] "GET /metrics HTTP/1.1" 200 36480 "" "Prometheus/2.51.0" 2024-09-21T10:30:04.292 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/7.MinOverlap (20134 ms) 2024-09-21T10:30:04.292 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/7.ZeroOverlap 2024-09-21T10:30:04.292 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:30:04.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:03 smithi160 ceph-mon[101879]: pgmap v1539: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 191 KiB/s rd, 4.2 MiB/s wr, 39 op/s 2024-09-21T10:30:04.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:03 smithi070 ceph-mon[112139]: pgmap v1539: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 191 KiB/s rd, 4.2 MiB/s wr, 39 op/s 2024-09-21T10:30:04.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:03 smithi070 ceph-mon[107385]: pgmap v1539: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 191 KiB/s rd, 4.2 MiB/s wr, 39 op/s 2024-09-21T10:30:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:05 smithi160 ceph-mon[101879]: pgmap v1540: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 847 KiB/s rd, 3.1 MiB/s wr, 12 op/s 2024-09-21T10:30:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:06.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:05 smithi070 ceph-mon[107385]: pgmap v1540: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 847 KiB/s rd, 3.1 MiB/s wr, 12 op/s 2024-09-21T10:30:06.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:06.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:05 smithi070 ceph-mon[112139]: pgmap v1540: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 847 KiB/s rd, 3.1 MiB/s wr, 12 op/s 2024-09-21T10:30:06.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:08.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:07 smithi160 ceph-mon[101879]: pgmap v1541: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 23 op/s 2024-09-21T10:30:08.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:07 smithi070 ceph-mon[107385]: pgmap v1541: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 23 op/s 2024-09-21T10:30:08.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:07 smithi070 ceph-mon[112139]: pgmap v1541: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 23 op/s 2024-09-21T10:30:10.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:09 smithi160 ceph-mon[101879]: pgmap v1542: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 20 op/s 2024-09-21T10:30:10.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:09 smithi070 ceph-mon[107385]: pgmap v1542: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 20 op/s 2024-09-21T10:30:10.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:09 smithi070 ceph-mon[112139]: pgmap v1542: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 20 op/s 2024-09-21T10:30:12.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:11 smithi160 ceph-mon[101879]: pgmap v1543: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 24 op/s 2024-09-21T10:30:12.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:11 smithi070 ceph-mon[107385]: pgmap v1543: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 24 op/s 2024-09-21T10:30:12.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:11 smithi070 ceph-mon[112139]: pgmap v1543: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 24 op/s 2024-09-21T10:30:13.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:12 smithi160 ceph-mon[101879]: osdmap e1005: 8 total, 8 up, 8 in 2024-09-21T10:30:13.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:12 smithi070 ceph-mon[107385]: osdmap e1005: 8 total, 8 up, 8 in 2024-09-21T10:30:13.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:30:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:30:13] "GET /metrics HTTP/1.1" 200 36483 "" "Prometheus/2.51.0" 2024-09-21T10:30:13.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:12 smithi070 ceph-mon[112139]: osdmap e1005: 8 total, 8 up, 8 in 2024-09-21T10:30:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:13 smithi160 ceph-mon[101879]: pgmap v1545: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.2 MiB/s wr, 20 op/s 2024-09-21T10:30:14.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:13 smithi070 ceph-mon[107385]: pgmap v1545: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.2 MiB/s wr, 20 op/s 2024-09-21T10:30:14.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:13 smithi070 ceph-mon[112139]: pgmap v1545: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.2 MiB/s wr, 20 op/s 2024-09-21T10:30:16.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:15 smithi160 ceph-mon[101879]: pgmap v1546: 105 pgs: 105 active+clean; 439 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 45 op/s 2024-09-21T10:30:16.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:15 smithi070 ceph-mon[107385]: pgmap v1546: 105 pgs: 105 active+clean; 439 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 45 op/s 2024-09-21T10:30:16.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:15 smithi070 ceph-mon[112139]: pgmap v1546: 105 pgs: 105 active+clean; 439 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 45 op/s 2024-09-21T10:30:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:17 smithi160 ceph-mon[101879]: pgmap v1547: 105 pgs: 105 active+clean; 464 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 438 KiB/s rd, 4.1 MiB/s wr, 41 op/s 2024-09-21T10:30:18.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:17 smithi070 ceph-mon[107385]: pgmap v1547: 105 pgs: 105 active+clean; 464 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 438 KiB/s rd, 4.1 MiB/s wr, 41 op/s 2024-09-21T10:30:18.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:17 smithi070 ceph-mon[112139]: pgmap v1547: 105 pgs: 105 active+clean; 464 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 438 KiB/s rd, 4.1 MiB/s wr, 41 op/s 2024-09-21T10:30:20.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:19 smithi160 ceph-mon[101879]: pgmap v1548: 105 pgs: 105 active+clean; 464 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 438 KiB/s rd, 4.1 MiB/s wr, 41 op/s 2024-09-21T10:30:20.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:19 smithi070 ceph-mon[107385]: pgmap v1548: 105 pgs: 105 active+clean; 464 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 438 KiB/s rd, 4.1 MiB/s wr, 41 op/s 2024-09-21T10:30:20.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:19 smithi070 ceph-mon[112139]: pgmap v1548: 105 pgs: 105 active+clean; 464 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 438 KiB/s rd, 4.1 MiB/s wr, 41 op/s 2024-09-21T10:30:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:21.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:21.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:22.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:21 smithi160 ceph-mon[101879]: pgmap v1549: 105 pgs: 105 active+clean; 468 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 4.5 MiB/s wr, 37 op/s 2024-09-21T10:30:22.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:21 smithi070 ceph-mon[107385]: pgmap v1549: 105 pgs: 105 active+clean; 468 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 4.5 MiB/s wr, 37 op/s 2024-09-21T10:30:22.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:21 smithi070 ceph-mon[112139]: pgmap v1549: 105 pgs: 105 active+clean; 468 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 4.5 MiB/s wr, 37 op/s 2024-09-21T10:30:23.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:30:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:30:23] "GET /metrics HTTP/1.1" 200 36483 "" "Prometheus/2.51.0" 2024-09-21T10:30:24.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:23 smithi160 ceph-mon[101879]: pgmap v1550: 105 pgs: 105 active+clean; 472 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 180 KiB/s rd, 4.6 MiB/s wr, 40 op/s 2024-09-21T10:30:24.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:23 smithi070 ceph-mon[112139]: pgmap v1550: 105 pgs: 105 active+clean; 472 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 180 KiB/s rd, 4.6 MiB/s wr, 40 op/s 2024-09-21T10:30:24.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:23 smithi070 ceph-mon[107385]: pgmap v1550: 105 pgs: 105 active+clean; 472 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 180 KiB/s rd, 4.6 MiB/s wr, 40 op/s 2024-09-21T10:30:25.632 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/7.ZeroOverlap (21339 ms) 2024-09-21T10:30:25.980 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/7 (159054 ms total) 2024-09-21T10:30:25.980 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:30:25.980 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 19 tests from DiffIterateTest/0, where TypeParam = DiffIterateParams 2024-09-21T10:30:26.008 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/0.DiffIterate 2024-09-21T10:30:26.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:25 smithi160 ceph-mon[101879]: pgmap v1551: 105 pgs: 105 active+clean; 472 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 550 KiB/s rd, 4.4 MiB/s wr, 39 op/s 2024-09-21T10:30:26.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:25 smithi070 ceph-mon[107385]: pgmap v1551: 105 pgs: 105 active+clean; 472 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 550 KiB/s rd, 4.4 MiB/s wr, 39 op/s 2024-09-21T10:30:26.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:25 smithi070 ceph-mon[112139]: pgmap v1551: 105 pgs: 105 active+clean; 472 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 550 KiB/s rd, 4.4 MiB/s wr, 39 op/s 2024-09-21T10:30:27.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:26 smithi160 ceph-mon[101879]: osdmap e1006: 8 total, 8 up, 8 in 2024-09-21T10:30:27.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:26 smithi070 ceph-mon[107385]: osdmap e1006: 8 total, 8 up, 8 in 2024-09-21T10:30:27.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:26 smithi070 ceph-mon[112139]: osdmap e1006: 8 total, 8 up, 8 in 2024-09-21T10:30:27.998 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:30:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:27 smithi160 ceph-mon[101879]: pgmap v1553: 73 pgs: 73 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:30:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:27 smithi160 ceph-mon[101879]: osdmap e1007: 8 total, 8 up, 8 in 2024-09-21T10:30:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:27 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3054775367' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-47","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:30:28.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:27 smithi070 ceph-mon[107385]: pgmap v1553: 73 pgs: 73 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:30:28.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:27 smithi070 ceph-mon[107385]: osdmap e1007: 8 total, 8 up, 8 in 2024-09-21T10:30:28.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:27 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3054775367' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-47","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:30:28.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:27 smithi070 ceph-mon[112139]: pgmap v1553: 73 pgs: 73 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:30:28.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:27 smithi070 ceph-mon[112139]: osdmap e1007: 8 total, 8 up, 8 in 2024-09-21T10:30:28.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:27 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3054775367' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-47","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:30:29.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:28 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3054775367' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:30:29.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:28 smithi160 ceph-mon[101879]: osdmap e1008: 8 total, 8 up, 8 in 2024-09-21T10:30:29.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:28 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3054775367' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:30:29.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:28 smithi070 ceph-mon[107385]: osdmap e1008: 8 total, 8 up, 8 in 2024-09-21T10:30:29.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:28 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3054775367' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:30:29.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:28 smithi070 ceph-mon[112139]: osdmap e1008: 8 total, 8 up, 8 in 2024-09-21T10:30:30.028 INFO:tasks.workunit.client.0.smithi070.stdout: wrote [984405~92360,1760277~76298,2049385~67997,2786077~12675,6385007~79205,8923358~50850,15027429~61490,15185745~100741,18158314~21633,19538630~72692] 2024-09-21T10:30:30.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:30 smithi160 ceph-mon[101879]: pgmap v1556: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:30:30.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:30 smithi160 ceph-mon[101879]: osdmap e1009: 8 total, 8 up, 8 in 2024-09-21T10:30:30.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:30 smithi070 ceph-mon[107385]: pgmap v1556: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:30:30.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:30 smithi070 ceph-mon[107385]: osdmap e1009: 8 total, 8 up, 8 in 2024-09-21T10:30:30.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:30 smithi070 ceph-mon[112139]: pgmap v1556: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:30:30.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:30 smithi070 ceph-mon[112139]: osdmap e1009: 8 total, 8 up, 8 in 2024-09-21T10:30:31.031 INFO:tasks.workunit.client.0.smithi070.stdout: wrote [906500~42452,1719537~56659,4916634~59586,6390566~62961,15523978~49191,16466520~2815,18494126~8614,19350925~43248,19508274~34462] 2024-09-21T10:30:31.034 INFO:tasks.workunit.client.0.smithi070.stdout: diff was [906500~42452,1719537~56659,4916634~59586,6390566~62961,15286486~1182849,18494126~8614,19350925~43248,19508274~34462] 2024-09-21T10:30:31.038 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/0.DiffIterate (5030 ms) 2024-09-21T10:30:31.038 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/0.DiffIterateDeterministic 2024-09-21T10:30:31.038 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:30:31.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:31 smithi160 ceph-mon[101879]: osdmap e1010: 8 total, 8 up, 8 in 2024-09-21T10:30:31.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:31 smithi070 ceph-mon[107385]: osdmap e1010: 8 total, 8 up, 8 in 2024-09-21T10:30:31.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:31 smithi070 ceph-mon[112139]: osdmap e1010: 8 total, 8 up, 8 in 2024-09-21T10:30:32.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:32 smithi160 ceph-mon[101879]: pgmap v1559: 105 pgs: 16 unknown, 89 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T10:30:32.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:32 smithi160 ceph-mon[101879]: osdmap e1011: 8 total, 8 up, 8 in 2024-09-21T10:30:32.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:32 smithi070 ceph-mon[107385]: pgmap v1559: 105 pgs: 16 unknown, 89 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T10:30:32.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:32 smithi070 ceph-mon[107385]: osdmap e1011: 8 total, 8 up, 8 in 2024-09-21T10:30:32.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:32 smithi070 ceph-mon[112139]: pgmap v1559: 105 pgs: 16 unknown, 89 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T10:30:32.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:32 smithi070 ceph-mon[112139]: osdmap e1011: 8 total, 8 up, 8 in 2024-09-21T10:30:33.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:33 smithi160 ceph-mon[101879]: osdmap e1012: 8 total, 8 up, 8 in 2024-09-21T10:30:33.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:33 smithi070 ceph-mon[107385]: osdmap e1012: 8 total, 8 up, 8 in 2024-09-21T10:30:33.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:30:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:30:33] "GET /metrics HTTP/1.1" 200 36483 "" "Prometheus/2.51.0" 2024-09-21T10:30:33.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:33 smithi070 ceph-mon[112139]: osdmap e1012: 8 total, 8 up, 8 in 2024-09-21T10:30:34.049 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:30:34.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:34 smithi160 ceph-mon[101879]: pgmap v1562: 105 pgs: 105 active+clean; 9.9 MiB data, 692 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 205 KiB/s wr, 27 op/s 2024-09-21T10:30:34.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:34 smithi160 ceph-mon[101879]: osdmap e1013: 8 total, 8 up, 8 in 2024-09-21T10:30:34.342 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 10:30:34 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=infra.usagestats t=2024-09-21T10:30:34.129890504Z level=info msg="Usage stats are ready to report" 2024-09-21T10:30:34.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:34 smithi070 ceph-mon[107385]: pgmap v1562: 105 pgs: 105 active+clean; 9.9 MiB data, 692 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 205 KiB/s wr, 27 op/s 2024-09-21T10:30:34.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:34 smithi070 ceph-mon[107385]: osdmap e1013: 8 total, 8 up, 8 in 2024-09-21T10:30:34.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:34 smithi070 ceph-mon[112139]: pgmap v1562: 105 pgs: 105 active+clean; 9.9 MiB data, 692 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 205 KiB/s wr, 27 op/s 2024-09-21T10:30:34.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:34 smithi070 ceph-mon[112139]: osdmap e1013: 8 total, 8 up, 8 in 2024-09-21T10:30:35.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:35 smithi160 ceph-mon[101879]: osdmap e1014: 8 total, 8 up, 8 in 2024-09-21T10:30:35.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:35 smithi070 ceph-mon[107385]: osdmap e1014: 8 total, 8 up, 8 in 2024-09-21T10:30:35.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:35 smithi070 ceph-mon[112139]: osdmap e1014: 8 total, 8 up, 8 in 2024-09-21T10:30:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:36 smithi160 ceph-mon[101879]: pgmap v1565: 105 pgs: 105 active+clean; 9.9 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 208 KiB/s wr, 41 op/s 2024-09-21T10:30:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:36 smithi160 ceph-mon[101879]: osdmap e1015: 8 total, 8 up, 8 in 2024-09-21T10:30:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:36.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:36 smithi070 ceph-mon[112139]: pgmap v1565: 105 pgs: 105 active+clean; 9.9 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 208 KiB/s wr, 41 op/s 2024-09-21T10:30:36.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:36 smithi070 ceph-mon[112139]: osdmap e1015: 8 total, 8 up, 8 in 2024-09-21T10:30:36.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:36.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:36 smithi070 ceph-mon[107385]: pgmap v1565: 105 pgs: 105 active+clean; 9.9 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 208 KiB/s wr, 41 op/s 2024-09-21T10:30:36.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:36 smithi070 ceph-mon[107385]: osdmap e1015: 8 total, 8 up, 8 in 2024-09-21T10:30:36.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:37.072 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:30:37.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:37 smithi160 ceph-mon[101879]: osdmap e1016: 8 total, 8 up, 8 in 2024-09-21T10:30:37.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:37 smithi070 ceph-mon[107385]: osdmap e1016: 8 total, 8 up, 8 in 2024-09-21T10:30:37.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:37 smithi070 ceph-mon[112139]: osdmap e1016: 8 total, 8 up, 8 in 2024-09-21T10:30:38.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:38 smithi160 ceph-mon[101879]: pgmap v1568: 105 pgs: 105 active+clean; 13 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 58 KiB/s wr, 112 op/s 2024-09-21T10:30:38.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:38 smithi160 ceph-mon[101879]: osdmap e1017: 8 total, 8 up, 8 in 2024-09-21T10:30:38.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:38 smithi160 ceph-mon[101879]: osdmap e1018: 8 total, 8 up, 8 in 2024-09-21T10:30:38.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:38 smithi070 ceph-mon[107385]: pgmap v1568: 105 pgs: 105 active+clean; 13 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 58 KiB/s wr, 112 op/s 2024-09-21T10:30:38.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:38 smithi070 ceph-mon[107385]: osdmap e1017: 8 total, 8 up, 8 in 2024-09-21T10:30:38.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:38 smithi070 ceph-mon[107385]: osdmap e1018: 8 total, 8 up, 8 in 2024-09-21T10:30:38.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:38 smithi070 ceph-mon[112139]: pgmap v1568: 105 pgs: 105 active+clean; 13 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 58 KiB/s wr, 112 op/s 2024-09-21T10:30:38.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:38 smithi070 ceph-mon[112139]: osdmap e1017: 8 total, 8 up, 8 in 2024-09-21T10:30:38.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:38 smithi070 ceph-mon[112139]: osdmap e1018: 8 total, 8 up, 8 in 2024-09-21T10:30:40.013 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:30:40.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:39 smithi160 ceph-mon[101879]: pgmap v1571: 105 pgs: 105 active+clean; 13 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 55 KiB/s wr, 97 op/s 2024-09-21T10:30:40.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:39 smithi160 ceph-mon[101879]: osdmap e1019: 8 total, 8 up, 8 in 2024-09-21T10:30:40.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:39 smithi070 ceph-mon[112139]: pgmap v1571: 105 pgs: 105 active+clean; 13 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 55 KiB/s wr, 97 op/s 2024-09-21T10:30:40.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:39 smithi070 ceph-mon[112139]: osdmap e1019: 8 total, 8 up, 8 in 2024-09-21T10:30:40.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:39 smithi070 ceph-mon[107385]: pgmap v1571: 105 pgs: 105 active+clean; 13 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 55 KiB/s wr, 97 op/s 2024-09-21T10:30:40.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:39 smithi070 ceph-mon[107385]: osdmap e1019: 8 total, 8 up, 8 in 2024-09-21T10:30:41.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:40 smithi160 ceph-mon[101879]: osdmap e1020: 8 total, 8 up, 8 in 2024-09-21T10:30:41.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:40 smithi070 ceph-mon[107385]: osdmap e1020: 8 total, 8 up, 8 in 2024-09-21T10:30:41.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:40 smithi070 ceph-mon[112139]: osdmap e1020: 8 total, 8 up, 8 in 2024-09-21T10:30:42.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:41 smithi160 ceph-mon[101879]: pgmap v1574: 105 pgs: 105 active+clean; 15 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 2.7 KiB/s wr, 32 op/s 2024-09-21T10:30:42.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:41 smithi160 ceph-mon[101879]: osdmap e1021: 8 total, 8 up, 8 in 2024-09-21T10:30:42.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:41 smithi070 ceph-mon[107385]: pgmap v1574: 105 pgs: 105 active+clean; 15 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 2.7 KiB/s wr, 32 op/s 2024-09-21T10:30:42.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:41 smithi070 ceph-mon[107385]: osdmap e1021: 8 total, 8 up, 8 in 2024-09-21T10:30:42.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:41 smithi070 ceph-mon[112139]: pgmap v1574: 105 pgs: 105 active+clean; 15 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 2.7 KiB/s wr, 32 op/s 2024-09-21T10:30:42.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:41 smithi070 ceph-mon[112139]: osdmap e1021: 8 total, 8 up, 8 in 2024-09-21T10:30:43.033 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:30:43.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:43 smithi160 ceph-mon[101879]: osdmap e1022: 8 total, 8 up, 8 in 2024-09-21T10:30:43.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:30:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:30:43] "GET /metrics HTTP/1.1" 200 36463 "" "Prometheus/2.51.0" 2024-09-21T10:30:43.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:43 smithi070 ceph-mon[112139]: osdmap e1022: 8 total, 8 up, 8 in 2024-09-21T10:30:43.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:43 smithi070 ceph-mon[107385]: osdmap e1022: 8 total, 8 up, 8 in 2024-09-21T10:30:44.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:44 smithi160 ceph-mon[101879]: pgmap v1577: 105 pgs: 105 active+clean; 18 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 8.0 KiB/s wr, 81 op/s 2024-09-21T10:30:44.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:44 smithi160 ceph-mon[101879]: osdmap e1023: 8 total, 8 up, 8 in 2024-09-21T10:30:44.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:44 smithi070 ceph-mon[112139]: pgmap v1577: 105 pgs: 105 active+clean; 18 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 8.0 KiB/s wr, 81 op/s 2024-09-21T10:30:44.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:44 smithi070 ceph-mon[112139]: osdmap e1023: 8 total, 8 up, 8 in 2024-09-21T10:30:44.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:44 smithi070 ceph-mon[107385]: pgmap v1577: 105 pgs: 105 active+clean; 18 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 8.0 KiB/s wr, 81 op/s 2024-09-21T10:30:44.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:44 smithi070 ceph-mon[107385]: osdmap e1023: 8 total, 8 up, 8 in 2024-09-21T10:30:45.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:45 smithi160 ceph-mon[101879]: osdmap e1024: 8 total, 8 up, 8 in 2024-09-21T10:30:45.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:45 smithi070 ceph-mon[107385]: osdmap e1024: 8 total, 8 up, 8 in 2024-09-21T10:30:45.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:45 smithi070 ceph-mon[112139]: osdmap e1024: 8 total, 8 up, 8 in 2024-09-21T10:30:46.049 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/0.DiffIterateDeterministic (15010 ms) 2024-09-21T10:30:46.049 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/0.DiffIterateDeterministicPP 2024-09-21T10:30:46.049 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:30:46.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:46 smithi160 ceph-mon[101879]: pgmap v1580: 105 pgs: 105 active+clean; 18 MiB data, 700 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 7.7 KiB/s wr, 74 op/s 2024-09-21T10:30:46.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:46 smithi160 ceph-mon[101879]: osdmap e1025: 8 total, 8 up, 8 in 2024-09-21T10:30:46.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:46 smithi070 ceph-mon[107385]: pgmap v1580: 105 pgs: 105 active+clean; 18 MiB data, 700 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 7.7 KiB/s wr, 74 op/s 2024-09-21T10:30:46.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:46 smithi070 ceph-mon[107385]: osdmap e1025: 8 total, 8 up, 8 in 2024-09-21T10:30:46.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:46 smithi070 ceph-mon[112139]: pgmap v1580: 105 pgs: 105 active+clean; 18 MiB data, 700 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 7.7 KiB/s wr, 74 op/s 2024-09-21T10:30:46.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:46 smithi070 ceph-mon[112139]: osdmap e1025: 8 total, 8 up, 8 in 2024-09-21T10:30:47.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:47 smithi160 ceph-mon[101879]: osdmap e1026: 8 total, 8 up, 8 in 2024-09-21T10:30:47.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:47 smithi070 ceph-mon[107385]: osdmap e1026: 8 total, 8 up, 8 in 2024-09-21T10:30:47.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:47 smithi070 ceph-mon[112139]: osdmap e1026: 8 total, 8 up, 8 in 2024-09-21T10:30:48.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:48 smithi160 ceph-mon[101879]: pgmap v1583: 105 pgs: 105 active+clean; 23 MiB data, 700 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 9.5 KiB/s wr, 110 op/s 2024-09-21T10:30:48.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:48 smithi160 ceph-mon[101879]: osdmap e1027: 8 total, 8 up, 8 in 2024-09-21T10:30:48.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:48 smithi070 ceph-mon[107385]: pgmap v1583: 105 pgs: 105 active+clean; 23 MiB data, 700 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 9.5 KiB/s wr, 110 op/s 2024-09-21T10:30:48.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:48 smithi070 ceph-mon[107385]: osdmap e1027: 8 total, 8 up, 8 in 2024-09-21T10:30:48.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:48 smithi070 ceph-mon[112139]: pgmap v1583: 105 pgs: 105 active+clean; 23 MiB data, 700 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 9.5 KiB/s wr, 110 op/s 2024-09-21T10:30:48.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:48 smithi070 ceph-mon[112139]: osdmap e1027: 8 total, 8 up, 8 in 2024-09-21T10:30:49.065 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:30:49.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:49 smithi160 ceph-mon[101879]: osdmap e1028: 8 total, 8 up, 8 in 2024-09-21T10:30:49.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:49 smithi070 ceph-mon[107385]: osdmap e1028: 8 total, 8 up, 8 in 2024-09-21T10:30:49.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:49 smithi070 ceph-mon[112139]: osdmap e1028: 8 total, 8 up, 8 in 2024-09-21T10:30:50.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:50 smithi160 ceph-mon[101879]: pgmap v1586: 105 pgs: 105 active+clean; 23 MiB data, 700 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 7.0 KiB/s wr, 84 op/s 2024-09-21T10:30:50.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:50 smithi160 ceph-mon[101879]: osdmap e1029: 8 total, 8 up, 8 in 2024-09-21T10:30:50.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:50 smithi070 ceph-mon[107385]: pgmap v1586: 105 pgs: 105 active+clean; 23 MiB data, 700 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 7.0 KiB/s wr, 84 op/s 2024-09-21T10:30:50.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:50 smithi070 ceph-mon[107385]: osdmap e1029: 8 total, 8 up, 8 in 2024-09-21T10:30:50.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:50 smithi070 ceph-mon[112139]: pgmap v1586: 105 pgs: 105 active+clean; 23 MiB data, 700 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 7.0 KiB/s wr, 84 op/s 2024-09-21T10:30:50.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:50 smithi070 ceph-mon[112139]: osdmap e1029: 8 total, 8 up, 8 in 2024-09-21T10:30:51.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:51 smithi070 ceph-mon[107385]: osdmap e1030: 8 total, 8 up, 8 in 2024-09-21T10:30:51.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:30:51.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:51 smithi160 ceph-mon[101879]: osdmap e1030: 8 total, 8 up, 8 in 2024-09-21T10:30:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:30:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:51.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:51 smithi070 ceph-mon[112139]: osdmap e1030: 8 total, 8 up, 8 in 2024-09-21T10:30:51.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:30:51.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:52.074 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:30:52.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:52 smithi160 ceph-mon[101879]: pgmap v1589: 105 pgs: 105 active+clean; 27 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 5.0 KiB/s wr, 58 op/s 2024-09-21T10:30:52.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:52 smithi160 ceph-mon[101879]: osdmap e1031: 8 total, 8 up, 8 in 2024-09-21T10:30:52.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:52 smithi070 ceph-mon[107385]: pgmap v1589: 105 pgs: 105 active+clean; 27 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 5.0 KiB/s wr, 58 op/s 2024-09-21T10:30:52.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:52 smithi070 ceph-mon[107385]: osdmap e1031: 8 total, 8 up, 8 in 2024-09-21T10:30:52.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:52 smithi070 ceph-mon[112139]: pgmap v1589: 105 pgs: 105 active+clean; 27 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 5.0 KiB/s wr, 58 op/s 2024-09-21T10:30:52.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:52 smithi070 ceph-mon[112139]: osdmap e1031: 8 total, 8 up, 8 in 2024-09-21T10:30:53.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:53 smithi070 ceph-mon[107385]: osdmap e1032: 8 total, 8 up, 8 in 2024-09-21T10:30:53.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:53 smithi070 ceph-mon[112139]: osdmap e1032: 8 total, 8 up, 8 in 2024-09-21T10:30:53.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:53 smithi160 ceph-mon[101879]: osdmap e1032: 8 total, 8 up, 8 in 2024-09-21T10:30:53.385 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:30:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:30:53] "GET /metrics HTTP/1.1" 200 36468 "" "Prometheus/2.51.0" 2024-09-21T10:30:54.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:54 smithi160 ceph-mon[101879]: pgmap v1592: 105 pgs: 105 active+clean; 30 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 6.7 KiB/s wr, 80 op/s 2024-09-21T10:30:54.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:54 smithi160 ceph-mon[101879]: osdmap e1033: 8 total, 8 up, 8 in 2024-09-21T10:30:54.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:54 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:30:54.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:54 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:30:54.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:54 smithi070 ceph-mon[107385]: pgmap v1592: 105 pgs: 105 active+clean; 30 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 6.7 KiB/s wr, 80 op/s 2024-09-21T10:30:54.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:54 smithi070 ceph-mon[107385]: osdmap e1033: 8 total, 8 up, 8 in 2024-09-21T10:30:54.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:54 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:30:54.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:54 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:30:54.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:54 smithi070 ceph-mon[112139]: pgmap v1592: 105 pgs: 105 active+clean; 30 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 6.7 KiB/s wr, 80 op/s 2024-09-21T10:30:54.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:54 smithi070 ceph-mon[112139]: osdmap e1033: 8 total, 8 up, 8 in 2024-09-21T10:30:54.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:54 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:30:54.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:54 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:30:55.093 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:30:55.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:55 smithi160 ceph-mon[101879]: osdmap e1034: 8 total, 8 up, 8 in 2024-09-21T10:30:55.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:55 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:30:55.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:55 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:30:55.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:55 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:30:55.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:55 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:30:55.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:55 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:30:55.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:55 smithi070 ceph-mon[107385]: osdmap e1034: 8 total, 8 up, 8 in 2024-09-21T10:30:55.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:55 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:30:55.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:55 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:30:55.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:55 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:30:55.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:55 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:30:55.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:55 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:30:55.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:55 smithi070 ceph-mon[112139]: osdmap e1034: 8 total, 8 up, 8 in 2024-09-21T10:30:55.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:55 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:30:55.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:55 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:30:55.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:55 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:30:55.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:55 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:30:55.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:55 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:30:56.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:56 smithi160 ceph-mon[101879]: pgmap v1595: 105 pgs: 105 active+clean; 36 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 5.2 KiB/s wr, 52 op/s 2024-09-21T10:30:56.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:56 smithi160 ceph-mon[101879]: osdmap e1035: 8 total, 8 up, 8 in 2024-09-21T10:30:56.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:56 smithi070 ceph-mon[107385]: pgmap v1595: 105 pgs: 105 active+clean; 36 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 5.2 KiB/s wr, 52 op/s 2024-09-21T10:30:56.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:56 smithi070 ceph-mon[107385]: osdmap e1035: 8 total, 8 up, 8 in 2024-09-21T10:30:56.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:56 smithi070 ceph-mon[112139]: pgmap v1595: 105 pgs: 105 active+clean; 36 MiB data, 701 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 5.2 KiB/s wr, 52 op/s 2024-09-21T10:30:56.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:56 smithi070 ceph-mon[112139]: osdmap e1035: 8 total, 8 up, 8 in 2024-09-21T10:30:57.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:57 smithi160 ceph-mon[101879]: osdmap e1036: 8 total, 8 up, 8 in 2024-09-21T10:30:57.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:57 smithi070 ceph-mon[107385]: osdmap e1036: 8 total, 8 up, 8 in 2024-09-21T10:30:57.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:57 smithi070 ceph-mon[112139]: osdmap e1036: 8 total, 8 up, 8 in 2024-09-21T10:30:58.115 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:30:58.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:58 smithi160 ceph-mon[101879]: pgmap v1598: 105 pgs: 105 active+clean; 39 MiB data, 702 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 8.5 KiB/s wr, 81 op/s 2024-09-21T10:30:58.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:58 smithi160 ceph-mon[101879]: osdmap e1037: 8 total, 8 up, 8 in 2024-09-21T10:30:58.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:58 smithi070 ceph-mon[112139]: pgmap v1598: 105 pgs: 105 active+clean; 39 MiB data, 702 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 8.5 KiB/s wr, 81 op/s 2024-09-21T10:30:58.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:58 smithi070 ceph-mon[112139]: osdmap e1037: 8 total, 8 up, 8 in 2024-09-21T10:30:58.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:58 smithi070 ceph-mon[107385]: pgmap v1598: 105 pgs: 105 active+clean; 39 MiB data, 702 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 8.5 KiB/s wr, 81 op/s 2024-09-21T10:30:58.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:58 smithi070 ceph-mon[107385]: osdmap e1037: 8 total, 8 up, 8 in 2024-09-21T10:30:59.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:30:59 smithi070 ceph-mon[107385]: osdmap e1038: 8 total, 8 up, 8 in 2024-09-21T10:30:59.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:30:59 smithi070 ceph-mon[112139]: osdmap e1038: 8 total, 8 up, 8 in 2024-09-21T10:30:59.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:30:59 smithi160 ceph-mon[101879]: osdmap e1038: 8 total, 8 up, 8 in 2024-09-21T10:31:00.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:00 smithi070 ceph-mon[107385]: pgmap v1601: 105 pgs: 105 active+clean; 39 MiB data, 702 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 5.0 KiB/s wr, 51 op/s 2024-09-21T10:31:00.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:00 smithi070 ceph-mon[107385]: osdmap e1039: 8 total, 8 up, 8 in 2024-09-21T10:31:00.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:00 smithi070 ceph-mon[112139]: pgmap v1601: 105 pgs: 105 active+clean; 39 MiB data, 702 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 5.0 KiB/s wr, 51 op/s 2024-09-21T10:31:00.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:00 smithi070 ceph-mon[112139]: osdmap e1039: 8 total, 8 up, 8 in 2024-09-21T10:31:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:00 smithi160 ceph-mon[101879]: pgmap v1601: 105 pgs: 105 active+clean; 39 MiB data, 702 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 5.0 KiB/s wr, 51 op/s 2024-09-21T10:31:00.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:00 smithi160 ceph-mon[101879]: osdmap e1039: 8 total, 8 up, 8 in 2024-09-21T10:31:01.135 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/0.DiffIterateDeterministicPP (15087 ms) 2024-09-21T10:31:01.135 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/0.DiffIterateDeterministicLUKS1 2024-09-21T10:31:01.135 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:31:01.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:01 smithi070 ceph-mon[107385]: osdmap e1040: 8 total, 8 up, 8 in 2024-09-21T10:31:01.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:01 smithi070 ceph-mon[112139]: osdmap e1040: 8 total, 8 up, 8 in 2024-09-21T10:31:01.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:01 smithi160 ceph-mon[101879]: osdmap e1040: 8 total, 8 up, 8 in 2024-09-21T10:31:02.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:02 smithi070 ceph-mon[112139]: pgmap v1604: 105 pgs: 105 active+clean; 43 MiB data, 702 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 3.2 KiB/s wr, 42 op/s 2024-09-21T10:31:02.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:02 smithi070 ceph-mon[112139]: osdmap e1041: 8 total, 8 up, 8 in 2024-09-21T10:31:02.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:02 smithi070 ceph-mon[107385]: pgmap v1604: 105 pgs: 105 active+clean; 43 MiB data, 702 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 3.2 KiB/s wr, 42 op/s 2024-09-21T10:31:02.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:02 smithi070 ceph-mon[107385]: osdmap e1041: 8 total, 8 up, 8 in 2024-09-21T10:31:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:02 smithi160 ceph-mon[101879]: pgmap v1604: 105 pgs: 105 active+clean; 43 MiB data, 702 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 3.2 KiB/s wr, 42 op/s 2024-09-21T10:31:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:02 smithi160 ceph-mon[101879]: osdmap e1041: 8 total, 8 up, 8 in 2024-09-21T10:31:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:31:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:31:03] "GET /metrics HTTP/1.1" 200 36468 "" "Prometheus/2.51.0" 2024-09-21T10:31:04.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:04 smithi070 ceph-mon[107385]: pgmap v1606: 105 pgs: 105 active+clean; 49 MiB data, 702 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 7.0 KiB/s wr, 80 op/s 2024-09-21T10:31:04.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:04 smithi070 ceph-mon[112139]: pgmap v1606: 105 pgs: 105 active+clean; 49 MiB data, 702 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 7.0 KiB/s wr, 80 op/s 2024-09-21T10:31:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:04 smithi160 ceph-mon[101879]: pgmap v1606: 105 pgs: 105 active+clean; 49 MiB data, 702 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 7.0 KiB/s wr, 80 op/s 2024-09-21T10:31:06.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:06 smithi070 ceph-mon[107385]: pgmap v1607: 105 pgs: 105 active+clean; 53 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 7.2 KiB/s wr, 90 op/s 2024-09-21T10:31:06.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:06 smithi070 ceph-mon[107385]: osdmap e1042: 8 total, 8 up, 8 in 2024-09-21T10:31:06.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:06.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:06 smithi070 ceph-mon[112139]: pgmap v1607: 105 pgs: 105 active+clean; 53 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 7.2 KiB/s wr, 90 op/s 2024-09-21T10:31:06.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:06 smithi070 ceph-mon[112139]: osdmap e1042: 8 total, 8 up, 8 in 2024-09-21T10:31:06.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:06 smithi160 ceph-mon[101879]: pgmap v1607: 105 pgs: 105 active+clean; 53 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 7.2 KiB/s wr, 90 op/s 2024-09-21T10:31:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:06 smithi160 ceph-mon[101879]: osdmap e1042: 8 total, 8 up, 8 in 2024-09-21T10:31:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:07.154 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:31:07.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:07 smithi070 ceph-mon[112139]: osdmap e1043: 8 total, 8 up, 8 in 2024-09-21T10:31:07.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:07 smithi070 ceph-mon[107385]: osdmap e1043: 8 total, 8 up, 8 in 2024-09-21T10:31:07.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:07 smithi160 ceph-mon[101879]: osdmap e1043: 8 total, 8 up, 8 in 2024-09-21T10:31:08.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:08 smithi070 ceph-mon[112139]: pgmap v1610: 105 pgs: 105 active+clean; 57 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 689 KiB/s wr, 68 op/s 2024-09-21T10:31:08.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:08 smithi070 ceph-mon[112139]: osdmap e1044: 8 total, 8 up, 8 in 2024-09-21T10:31:08.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:08 smithi070 ceph-mon[107385]: pgmap v1610: 105 pgs: 105 active+clean; 57 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 689 KiB/s wr, 68 op/s 2024-09-21T10:31:08.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:08 smithi070 ceph-mon[107385]: osdmap e1044: 8 total, 8 up, 8 in 2024-09-21T10:31:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:08 smithi160 ceph-mon[101879]: pgmap v1610: 105 pgs: 105 active+clean; 57 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 689 KiB/s wr, 68 op/s 2024-09-21T10:31:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:08 smithi160 ceph-mon[101879]: osdmap e1044: 8 total, 8 up, 8 in 2024-09-21T10:31:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:10 smithi160 ceph-mon[101879]: pgmap v1612: 105 pgs: 105 active+clean; 57 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 686 KiB/s wr, 37 op/s 2024-09-21T10:31:10.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:10 smithi070 ceph-mon[107385]: pgmap v1612: 105 pgs: 105 active+clean; 57 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 686 KiB/s wr, 37 op/s 2024-09-21T10:31:10.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:10 smithi070 ceph-mon[112139]: pgmap v1612: 105 pgs: 105 active+clean; 57 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 686 KiB/s wr, 37 op/s 2024-09-21T10:31:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:12 smithi160 ceph-mon[101879]: pgmap v1613: 105 pgs: 105 active+clean; 57 MiB data, 719 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-09-21T10:31:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:12 smithi160 ceph-mon[101879]: osdmap e1045: 8 total, 8 up, 8 in 2024-09-21T10:31:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:12 smithi070 ceph-mon[107385]: pgmap v1613: 105 pgs: 105 active+clean; 57 MiB data, 719 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-09-21T10:31:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:12 smithi070 ceph-mon[107385]: osdmap e1045: 8 total, 8 up, 8 in 2024-09-21T10:31:12.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:12 smithi070 ceph-mon[112139]: pgmap v1613: 105 pgs: 105 active+clean; 57 MiB data, 719 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-09-21T10:31:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:12 smithi070 ceph-mon[112139]: osdmap e1045: 8 total, 8 up, 8 in 2024-09-21T10:31:13.177 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:31:13.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:13 smithi160 ceph-mon[101879]: osdmap e1046: 8 total, 8 up, 8 in 2024-09-21T10:31:13.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:13 smithi070 ceph-mon[107385]: osdmap e1046: 8 total, 8 up, 8 in 2024-09-21T10:31:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:31:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:31:13] "GET /metrics HTTP/1.1" 200 36467 "" "Prometheus/2.51.0" 2024-09-21T10:31:13.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:13 smithi070 ceph-mon[112139]: osdmap e1046: 8 total, 8 up, 8 in 2024-09-21T10:31:14.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:14 smithi160 ceph-mon[101879]: pgmap v1616: 105 pgs: 105 active+clean; 57 MiB data, 719 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 3.5 KiB/s wr, 37 op/s 2024-09-21T10:31:14.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:14 smithi160 ceph-mon[101879]: osdmap e1047: 8 total, 8 up, 8 in 2024-09-21T10:31:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:14 smithi070 ceph-mon[107385]: pgmap v1616: 105 pgs: 105 active+clean; 57 MiB data, 719 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 3.5 KiB/s wr, 37 op/s 2024-09-21T10:31:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:14 smithi070 ceph-mon[107385]: osdmap e1047: 8 total, 8 up, 8 in 2024-09-21T10:31:14.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:14 smithi070 ceph-mon[112139]: pgmap v1616: 105 pgs: 105 active+clean; 57 MiB data, 719 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 3.5 KiB/s wr, 37 op/s 2024-09-21T10:31:14.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:14 smithi070 ceph-mon[112139]: osdmap e1047: 8 total, 8 up, 8 in 2024-09-21T10:31:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:16 smithi160 ceph-mon[101879]: pgmap v1618: 105 pgs: 105 active+clean; 61 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 688 KiB/s wr, 45 op/s 2024-09-21T10:31:16.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:16 smithi070 ceph-mon[107385]: pgmap v1618: 105 pgs: 105 active+clean; 61 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 688 KiB/s wr, 45 op/s 2024-09-21T10:31:16.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:16 smithi070 ceph-mon[112139]: pgmap v1618: 105 pgs: 105 active+clean; 61 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 688 KiB/s wr, 45 op/s 2024-09-21T10:31:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:18 smithi160 ceph-mon[101879]: pgmap v1619: 105 pgs: 105 active+clean; 63 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 687 KiB/s wr, 44 op/s 2024-09-21T10:31:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:18 smithi160 ceph-mon[101879]: osdmap e1048: 8 total, 8 up, 8 in 2024-09-21T10:31:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:18 smithi070 ceph-mon[107385]: pgmap v1619: 105 pgs: 105 active+clean; 63 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 687 KiB/s wr, 44 op/s 2024-09-21T10:31:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:18 smithi070 ceph-mon[107385]: osdmap e1048: 8 total, 8 up, 8 in 2024-09-21T10:31:18.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:18 smithi070 ceph-mon[112139]: pgmap v1619: 105 pgs: 105 active+clean; 63 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 687 KiB/s wr, 44 op/s 2024-09-21T10:31:18.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:18 smithi070 ceph-mon[112139]: osdmap e1048: 8 total, 8 up, 8 in 2024-09-21T10:31:19.197 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:31:19.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:19 smithi160 ceph-mon[101879]: osdmap e1049: 8 total, 8 up, 8 in 2024-09-21T10:31:19.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:19 smithi070 ceph-mon[107385]: osdmap e1049: 8 total, 8 up, 8 in 2024-09-21T10:31:19.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:19 smithi070 ceph-mon[112139]: osdmap e1049: 8 total, 8 up, 8 in 2024-09-21T10:31:20.493 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:20 smithi160 ceph-mon[101879]: pgmap v1622: 105 pgs: 105 active+clean; 63 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 686 KiB/s wr, 40 op/s 2024-09-21T10:31:20.493 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:20 smithi160 ceph-mon[101879]: osdmap e1050: 8 total, 8 up, 8 in 2024-09-21T10:31:20.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:20 smithi070 ceph-mon[107385]: pgmap v1622: 105 pgs: 105 active+clean; 63 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 686 KiB/s wr, 40 op/s 2024-09-21T10:31:20.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:20 smithi070 ceph-mon[107385]: osdmap e1050: 8 total, 8 up, 8 in 2024-09-21T10:31:20.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:20 smithi070 ceph-mon[112139]: pgmap v1622: 105 pgs: 105 active+clean; 63 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 686 KiB/s wr, 40 op/s 2024-09-21T10:31:20.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:20 smithi070 ceph-mon[112139]: osdmap e1050: 8 total, 8 up, 8 in 2024-09-21T10:31:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:21.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:21.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:22.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:22 smithi160 ceph-mon[101879]: pgmap v1624: 105 pgs: 105 active+clean; 67 MiB data, 720 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 688 KiB/s wr, 61 op/s 2024-09-21T10:31:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:22 smithi070 ceph-mon[107385]: pgmap v1624: 105 pgs: 105 active+clean; 67 MiB data, 720 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 688 KiB/s wr, 61 op/s 2024-09-21T10:31:22.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:22 smithi070 ceph-mon[112139]: pgmap v1624: 105 pgs: 105 active+clean; 67 MiB data, 720 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 688 KiB/s wr, 61 op/s 2024-09-21T10:31:23.528 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:31:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:31:23] "GET /metrics HTTP/1.1" 200 36470 "" "Prometheus/2.51.0" 2024-09-21T10:31:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:24 smithi160 ceph-mon[101879]: pgmap v1625: 105 pgs: 105 active+clean; 69 MiB data, 728 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 39 op/s 2024-09-21T10:31:24.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:24 smithi160 ceph-mon[101879]: osdmap e1051: 8 total, 8 up, 8 in 2024-09-21T10:31:24.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:24 smithi070 ceph-mon[112139]: pgmap v1625: 105 pgs: 105 active+clean; 69 MiB data, 728 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 39 op/s 2024-09-21T10:31:24.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:24 smithi070 ceph-mon[112139]: osdmap e1051: 8 total, 8 up, 8 in 2024-09-21T10:31:24.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:24 smithi070 ceph-mon[107385]: pgmap v1625: 105 pgs: 105 active+clean; 69 MiB data, 728 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 39 op/s 2024-09-21T10:31:24.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:24 smithi070 ceph-mon[107385]: osdmap e1051: 8 total, 8 up, 8 in 2024-09-21T10:31:25.219 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:31:25.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:25 smithi160 ceph-mon[101879]: osdmap e1052: 8 total, 8 up, 8 in 2024-09-21T10:31:25.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:25 smithi070 ceph-mon[112139]: osdmap e1052: 8 total, 8 up, 8 in 2024-09-21T10:31:25.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:25 smithi070 ceph-mon[107385]: osdmap e1052: 8 total, 8 up, 8 in 2024-09-21T10:31:26.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:26 smithi160 ceph-mon[101879]: pgmap v1628: 105 pgs: 105 active+clean; 75 MiB data, 740 MiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 1.3 MiB/s wr, 47 op/s 2024-09-21T10:31:26.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:26 smithi160 ceph-mon[101879]: osdmap e1053: 8 total, 8 up, 8 in 2024-09-21T10:31:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:26 smithi070 ceph-mon[107385]: pgmap v1628: 105 pgs: 105 active+clean; 75 MiB data, 740 MiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 1.3 MiB/s wr, 47 op/s 2024-09-21T10:31:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:26 smithi070 ceph-mon[107385]: osdmap e1053: 8 total, 8 up, 8 in 2024-09-21T10:31:26.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:26 smithi070 ceph-mon[112139]: pgmap v1628: 105 pgs: 105 active+clean; 75 MiB data, 740 MiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 1.3 MiB/s wr, 47 op/s 2024-09-21T10:31:26.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:26 smithi070 ceph-mon[112139]: osdmap e1053: 8 total, 8 up, 8 in 2024-09-21T10:31:28.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:28 smithi160 ceph-mon[101879]: pgmap v1630: 105 pgs: 105 active+clean; 75 MiB data, 740 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 45 op/s 2024-09-21T10:31:28.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:28 smithi070 ceph-mon[107385]: pgmap v1630: 105 pgs: 105 active+clean; 75 MiB data, 740 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 45 op/s 2024-09-21T10:31:28.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:28 smithi070 ceph-mon[112139]: pgmap v1630: 105 pgs: 105 active+clean; 75 MiB data, 740 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 45 op/s 2024-09-21T10:31:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:30 smithi160 ceph-mon[101879]: pgmap v1631: 105 pgs: 105 active+clean; 75 MiB data, 740 MiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 686 KiB/s wr, 34 op/s 2024-09-21T10:31:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:30 smithi160 ceph-mon[101879]: osdmap e1054: 8 total, 8 up, 8 in 2024-09-21T10:31:30.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:30 smithi070 ceph-mon[107385]: pgmap v1631: 105 pgs: 105 active+clean; 75 MiB data, 740 MiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 686 KiB/s wr, 34 op/s 2024-09-21T10:31:30.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:30 smithi070 ceph-mon[107385]: osdmap e1054: 8 total, 8 up, 8 in 2024-09-21T10:31:30.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:30 smithi070 ceph-mon[112139]: pgmap v1631: 105 pgs: 105 active+clean; 75 MiB data, 740 MiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 686 KiB/s wr, 34 op/s 2024-09-21T10:31:30.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:30 smithi070 ceph-mon[112139]: osdmap e1054: 8 total, 8 up, 8 in 2024-09-21T10:31:31.243 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/0.DiffIterateDeterministicLUKS1 (30108 ms) 2024-09-21T10:31:31.243 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/0.DiffIterateDeterministicLUKS1PP 2024-09-21T10:31:31.243 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:31:31.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:31 smithi160 ceph-mon[101879]: osdmap e1055: 8 total, 8 up, 8 in 2024-09-21T10:31:31.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:31 smithi070 ceph-mon[107385]: osdmap e1055: 8 total, 8 up, 8 in 2024-09-21T10:31:31.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:31 smithi070 ceph-mon[112139]: osdmap e1055: 8 total, 8 up, 8 in 2024-09-21T10:31:32.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:32 smithi160 ceph-mon[101879]: pgmap v1634: 105 pgs: 105 active+clean; 83 MiB data, 744 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 685 KiB/s wr, 33 op/s 2024-09-21T10:31:32.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:32 smithi160 ceph-mon[101879]: osdmap e1056: 8 total, 8 up, 8 in 2024-09-21T10:31:32.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:32 smithi070 ceph-mon[107385]: pgmap v1634: 105 pgs: 105 active+clean; 83 MiB data, 744 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 685 KiB/s wr, 33 op/s 2024-09-21T10:31:32.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:32 smithi070 ceph-mon[107385]: osdmap e1056: 8 total, 8 up, 8 in 2024-09-21T10:31:32.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:32 smithi070 ceph-mon[112139]: pgmap v1634: 105 pgs: 105 active+clean; 83 MiB data, 744 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 685 KiB/s wr, 33 op/s 2024-09-21T10:31:32.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:32 smithi070 ceph-mon[112139]: osdmap e1056: 8 total, 8 up, 8 in 2024-09-21T10:31:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:31:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:31:33] "GET /metrics HTTP/1.1" 200 36470 "" "Prometheus/2.51.0" 2024-09-21T10:31:34.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:34 smithi160 ceph-mon[101879]: pgmap v1636: 105 pgs: 105 active+clean; 87 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 685 KiB/s wr, 25 op/s 2024-09-21T10:31:34.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:34 smithi070 ceph-mon[107385]: pgmap v1636: 105 pgs: 105 active+clean; 87 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 685 KiB/s wr, 25 op/s 2024-09-21T10:31:34.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:34 smithi070 ceph-mon[112139]: pgmap v1636: 105 pgs: 105 active+clean; 87 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 685 KiB/s wr, 25 op/s 2024-09-21T10:31:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:36 smithi160 ceph-mon[101879]: pgmap v1637: 105 pgs: 105 active+clean; 87 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 686 KiB/s wr, 28 op/s 2024-09-21T10:31:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:36 smithi160 ceph-mon[101879]: osdmap e1057: 8 total, 8 up, 8 in 2024-09-21T10:31:36.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:36 smithi070 ceph-mon[107385]: pgmap v1637: 105 pgs: 105 active+clean; 87 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 686 KiB/s wr, 28 op/s 2024-09-21T10:31:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:36 smithi070 ceph-mon[107385]: osdmap e1057: 8 total, 8 up, 8 in 2024-09-21T10:31:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:36.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:36 smithi070 ceph-mon[112139]: pgmap v1637: 105 pgs: 105 active+clean; 87 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 686 KiB/s wr, 28 op/s 2024-09-21T10:31:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:36 smithi070 ceph-mon[112139]: osdmap e1057: 8 total, 8 up, 8 in 2024-09-21T10:31:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:37.265 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:31:37.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:37 smithi160 ceph-mon[101879]: osdmap e1058: 8 total, 8 up, 8 in 2024-09-21T10:31:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:37 smithi070 ceph-mon[107385]: osdmap e1058: 8 total, 8 up, 8 in 2024-09-21T10:31:37.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:37 smithi070 ceph-mon[112139]: osdmap e1058: 8 total, 8 up, 8 in 2024-09-21T10:31:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:38 smithi160 ceph-mon[101879]: pgmap v1640: 105 pgs: 105 active+clean; 87 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 4.8 KiB/s wr, 44 op/s 2024-09-21T10:31:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:38 smithi160 ceph-mon[101879]: osdmap e1059: 8 total, 8 up, 8 in 2024-09-21T10:31:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:38 smithi070 ceph-mon[107385]: pgmap v1640: 105 pgs: 105 active+clean; 87 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 4.8 KiB/s wr, 44 op/s 2024-09-21T10:31:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:38 smithi070 ceph-mon[107385]: osdmap e1059: 8 total, 8 up, 8 in 2024-09-21T10:31:38.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:38 smithi070 ceph-mon[112139]: pgmap v1640: 105 pgs: 105 active+clean; 87 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 4.8 KiB/s wr, 44 op/s 2024-09-21T10:31:38.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:38 smithi070 ceph-mon[112139]: osdmap e1059: 8 total, 8 up, 8 in 2024-09-21T10:31:40.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:40 smithi160 ceph-mon[101879]: pgmap v1642: 105 pgs: 105 active+clean; 87 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 2.5 KiB/s wr, 25 op/s 2024-09-21T10:31:40.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:40 smithi070 ceph-mon[107385]: pgmap v1642: 105 pgs: 105 active+clean; 87 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 2.5 KiB/s wr, 25 op/s 2024-09-21T10:31:40.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:40 smithi070 ceph-mon[112139]: pgmap v1642: 105 pgs: 105 active+clean; 87 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 2.5 KiB/s wr, 25 op/s 2024-09-21T10:31:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:42 smithi160 ceph-mon[101879]: pgmap v1643: 105 pgs: 105 active+clean; 91 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 34 KiB/s rd, 687 KiB/s wr, 49 op/s 2024-09-21T10:31:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:42 smithi160 ceph-mon[101879]: osdmap e1060: 8 total, 8 up, 8 in 2024-09-21T10:31:42.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:42 smithi070 ceph-mon[107385]: pgmap v1643: 105 pgs: 105 active+clean; 91 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 34 KiB/s rd, 687 KiB/s wr, 49 op/s 2024-09-21T10:31:42.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:42 smithi070 ceph-mon[107385]: osdmap e1060: 8 total, 8 up, 8 in 2024-09-21T10:31:42.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:42 smithi070 ceph-mon[112139]: pgmap v1643: 105 pgs: 105 active+clean; 91 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 34 KiB/s rd, 687 KiB/s wr, 49 op/s 2024-09-21T10:31:42.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:42 smithi070 ceph-mon[112139]: osdmap e1060: 8 total, 8 up, 8 in 2024-09-21T10:31:43.285 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:31:43.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:43 smithi160 ceph-mon[101879]: osdmap e1061: 8 total, 8 up, 8 in 2024-09-21T10:31:43.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:43 smithi070 ceph-mon[107385]: osdmap e1061: 8 total, 8 up, 8 in 2024-09-21T10:31:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:31:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:31:43] "GET /metrics HTTP/1.1" 200 36470 "" "Prometheus/2.51.0" 2024-09-21T10:31:43.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:43 smithi070 ceph-mon[112139]: osdmap e1061: 8 total, 8 up, 8 in 2024-09-21T10:31:44.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:44 smithi160 ceph-mon[101879]: pgmap v1646: 105 pgs: 105 active+clean; 91 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 686 KiB/s wr, 35 op/s 2024-09-21T10:31:44.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:44 smithi160 ceph-mon[101879]: osdmap e1062: 8 total, 8 up, 8 in 2024-09-21T10:31:44.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:44 smithi070 ceph-mon[107385]: pgmap v1646: 105 pgs: 105 active+clean; 91 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 686 KiB/s wr, 35 op/s 2024-09-21T10:31:44.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:44 smithi070 ceph-mon[107385]: osdmap e1062: 8 total, 8 up, 8 in 2024-09-21T10:31:44.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:44 smithi070 ceph-mon[112139]: pgmap v1646: 105 pgs: 105 active+clean; 91 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 686 KiB/s wr, 35 op/s 2024-09-21T10:31:44.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:44 smithi070 ceph-mon[112139]: osdmap e1062: 8 total, 8 up, 8 in 2024-09-21T10:31:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:46 smithi160 ceph-mon[101879]: pgmap v1648: 105 pgs: 105 active+clean; 98 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 43 KiB/s rd, 1.3 MiB/s wr, 66 op/s 2024-09-21T10:31:46.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:46 smithi070 ceph-mon[107385]: pgmap v1648: 105 pgs: 105 active+clean; 98 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 43 KiB/s rd, 1.3 MiB/s wr, 66 op/s 2024-09-21T10:31:46.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:46 smithi070 ceph-mon[112139]: pgmap v1648: 105 pgs: 105 active+clean; 98 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 43 KiB/s rd, 1.3 MiB/s wr, 66 op/s 2024-09-21T10:31:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:48 smithi160 ceph-mon[101879]: pgmap v1649: 105 pgs: 105 active+clean; 101 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 688 KiB/s wr, 48 op/s 2024-09-21T10:31:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:48 smithi160 ceph-mon[101879]: osdmap e1063: 8 total, 8 up, 8 in 2024-09-21T10:31:48.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:48 smithi070 ceph-mon[112139]: pgmap v1649: 105 pgs: 105 active+clean; 101 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 688 KiB/s wr, 48 op/s 2024-09-21T10:31:48.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:48 smithi070 ceph-mon[112139]: osdmap e1063: 8 total, 8 up, 8 in 2024-09-21T10:31:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:48 smithi070 ceph-mon[107385]: pgmap v1649: 105 pgs: 105 active+clean; 101 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 688 KiB/s wr, 48 op/s 2024-09-21T10:31:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:48 smithi070 ceph-mon[107385]: osdmap e1063: 8 total, 8 up, 8 in 2024-09-21T10:31:49.301 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:31:49.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:49 smithi160 ceph-mon[101879]: osdmap e1064: 8 total, 8 up, 8 in 2024-09-21T10:31:49.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:49 smithi070 ceph-mon[107385]: osdmap e1064: 8 total, 8 up, 8 in 2024-09-21T10:31:49.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:49 smithi070 ceph-mon[112139]: osdmap e1064: 8 total, 8 up, 8 in 2024-09-21T10:31:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:50 smithi160 ceph-mon[101879]: pgmap v1652: 105 pgs: 105 active+clean; 101 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 686 KiB/s wr, 40 op/s 2024-09-21T10:31:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:50 smithi160 ceph-mon[101879]: osdmap e1065: 8 total, 8 up, 8 in 2024-09-21T10:31:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:50 smithi070 ceph-mon[107385]: pgmap v1652: 105 pgs: 105 active+clean; 101 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 686 KiB/s wr, 40 op/s 2024-09-21T10:31:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:50 smithi070 ceph-mon[107385]: osdmap e1065: 8 total, 8 up, 8 in 2024-09-21T10:31:50.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:50 smithi070 ceph-mon[112139]: pgmap v1652: 105 pgs: 105 active+clean; 101 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 686 KiB/s wr, 40 op/s 2024-09-21T10:31:50.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:50 smithi070 ceph-mon[112139]: osdmap e1065: 8 total, 8 up, 8 in 2024-09-21T10:31:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:51.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:52.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:52 smithi160 ceph-mon[101879]: pgmap v1654: 105 pgs: 105 active+clean; 108 MiB data, 782 MiB used, 714 GiB / 715 GiB avail; 6.5 KiB/s rd, 684 KiB/s wr, 12 op/s 2024-09-21T10:31:52.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:52 smithi070 ceph-mon[107385]: pgmap v1654: 105 pgs: 105 active+clean; 108 MiB data, 782 MiB used, 714 GiB / 715 GiB avail; 6.5 KiB/s rd, 684 KiB/s wr, 12 op/s 2024-09-21T10:31:52.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:52 smithi070 ceph-mon[112139]: pgmap v1654: 105 pgs: 105 active+clean; 108 MiB data, 782 MiB used, 714 GiB / 715 GiB avail; 6.5 KiB/s rd, 684 KiB/s wr, 12 op/s 2024-09-21T10:31:53.528 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:31:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:31:53] "GET /metrics HTTP/1.1" 200 36472 "" "Prometheus/2.51.0" 2024-09-21T10:31:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:54 smithi160 ceph-mon[101879]: pgmap v1655: 105 pgs: 105 active+clean; 111 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 686 KiB/s wr, 38 op/s 2024-09-21T10:31:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:54 smithi160 ceph-mon[101879]: osdmap e1066: 8 total, 8 up, 8 in 2024-09-21T10:31:54.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:54 smithi070 ceph-mon[112139]: pgmap v1655: 105 pgs: 105 active+clean; 111 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 686 KiB/s wr, 38 op/s 2024-09-21T10:31:54.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:54 smithi070 ceph-mon[112139]: osdmap e1066: 8 total, 8 up, 8 in 2024-09-21T10:31:54.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:54 smithi070 ceph-mon[107385]: pgmap v1655: 105 pgs: 105 active+clean; 111 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 686 KiB/s wr, 38 op/s 2024-09-21T10:31:54.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:54 smithi070 ceph-mon[107385]: osdmap e1066: 8 total, 8 up, 8 in 2024-09-21T10:31:55.321 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:31:55.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:55 smithi160 ceph-mon[101879]: osdmap e1067: 8 total, 8 up, 8 in 2024-09-21T10:31:55.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:55 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:31:55.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:55 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:31:55.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:55 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:31:55.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:55 smithi070 ceph-mon[107385]: osdmap e1067: 8 total, 8 up, 8 in 2024-09-21T10:31:55.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:55 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:31:55.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:55 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:31:55.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:55 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:31:55.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:55 smithi070 ceph-mon[112139]: osdmap e1067: 8 total, 8 up, 8 in 2024-09-21T10:31:55.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:55 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:31:55.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:55 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:31:55.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:55 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:31:56.568 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:56 smithi070 ceph-mon[107385]: pgmap v1658: 105 pgs: 105 active+clean; 111 MiB data, 794 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-09-21T10:31:56.568 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:56 smithi070 ceph-mon[107385]: osdmap e1068: 8 total, 8 up, 8 in 2024-09-21T10:31:56.568 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:56 smithi070 ceph-mon[112139]: pgmap v1658: 105 pgs: 105 active+clean; 111 MiB data, 794 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-09-21T10:31:56.568 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:56 smithi070 ceph-mon[112139]: osdmap e1068: 8 total, 8 up, 8 in 2024-09-21T10:31:56.582 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:56 smithi160 ceph-mon[101879]: pgmap v1658: 105 pgs: 105 active+clean; 111 MiB data, 794 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-09-21T10:31:56.582 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:56 smithi160 ceph-mon[101879]: osdmap e1068: 8 total, 8 up, 8 in 2024-09-21T10:31:58.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:58 smithi160 ceph-mon[101879]: pgmap v1660: 105 pgs: 105 active+clean; 121 MiB data, 802 MiB used, 714 GiB / 715 GiB avail; 48 KiB/s rd, 690 KiB/s wr, 73 op/s 2024-09-21T10:31:58.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:58 smithi070 ceph-mon[107385]: pgmap v1660: 105 pgs: 105 active+clean; 121 MiB data, 802 MiB used, 714 GiB / 715 GiB avail; 48 KiB/s rd, 690 KiB/s wr, 73 op/s 2024-09-21T10:31:58.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:58 smithi070 ceph-mon[112139]: pgmap v1660: 105 pgs: 105 active+clean; 121 MiB data, 802 MiB used, 714 GiB / 715 GiB avail; 48 KiB/s rd, 690 KiB/s wr, 73 op/s 2024-09-21T10:31:59.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:59 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:31:59.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:59 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:31:59.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:59 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:31:59.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:31:59 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:31:59.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:31:59.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:31:59.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:31:59.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:31:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:31:59.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:59 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:31:59.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:59 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:31:59.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:59 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:31:59.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:31:59 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:32:00.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:00 smithi070 ceph-mon[107385]: pgmap v1661: 105 pgs: 105 active+clean; 121 MiB data, 802 MiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 687 KiB/s wr, 38 op/s 2024-09-21T10:32:00.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:00 smithi070 ceph-mon[107385]: osdmap e1069: 8 total, 8 up, 8 in 2024-09-21T10:32:00.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:00 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:32:00.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:00 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:32:00.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:00 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:32:00.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:00 smithi070 ceph-mon[112139]: pgmap v1661: 105 pgs: 105 active+clean; 121 MiB data, 802 MiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 687 KiB/s wr, 38 op/s 2024-09-21T10:32:00.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:00 smithi070 ceph-mon[112139]: osdmap e1069: 8 total, 8 up, 8 in 2024-09-21T10:32:00.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:00 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:32:00.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:00 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:32:00.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:00 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:32:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:00 smithi160 ceph-mon[101879]: pgmap v1661: 105 pgs: 105 active+clean; 121 MiB data, 802 MiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 687 KiB/s wr, 38 op/s 2024-09-21T10:32:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:00 smithi160 ceph-mon[101879]: osdmap e1069: 8 total, 8 up, 8 in 2024-09-21T10:32:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:00 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:32:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:00 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:32:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:00 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:32:01.383 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/0.DiffIterateDeterministicLUKS1PP (30140 ms) 2024-09-21T10:32:01.383 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/0.DiffIterateDeterministicLUKS2 2024-09-21T10:32:01.383 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:32:01.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:01 smithi070 ceph-mon[107385]: osdmap e1070: 8 total, 8 up, 8 in 2024-09-21T10:32:01.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:01 smithi070 ceph-mon[112139]: osdmap e1070: 8 total, 8 up, 8 in 2024-09-21T10:32:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:01 smithi160 ceph-mon[101879]: osdmap e1070: 8 total, 8 up, 8 in 2024-09-21T10:32:02.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:02 smithi070 ceph-mon[107385]: pgmap v1664: 105 pgs: 105 active+clean; 121 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-09-21T10:32:02.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:02 smithi070 ceph-mon[107385]: osdmap e1071: 8 total, 8 up, 8 in 2024-09-21T10:32:02.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:02 smithi070 ceph-mon[112139]: pgmap v1664: 105 pgs: 105 active+clean; 121 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-09-21T10:32:02.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:02 smithi070 ceph-mon[112139]: osdmap e1071: 8 total, 8 up, 8 in 2024-09-21T10:32:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:02 smithi160 ceph-mon[101879]: pgmap v1664: 105 pgs: 105 active+clean; 121 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-09-21T10:32:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:02 smithi160 ceph-mon[101879]: osdmap e1071: 8 total, 8 up, 8 in 2024-09-21T10:32:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:32:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:32:03] "GET /metrics HTTP/1.1" 200 36472 "" "Prometheus/2.51.0" 2024-09-21T10:32:04.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:04 smithi070 ceph-mon[107385]: pgmap v1666: 105 pgs: 105 active+clean; 133 MiB data, 811 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 685 KiB/s wr, 23 op/s 2024-09-21T10:32:04.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:04 smithi070 ceph-mon[112139]: pgmap v1666: 105 pgs: 105 active+clean; 133 MiB data, 811 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 685 KiB/s wr, 23 op/s 2024-09-21T10:32:04.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:04 smithi160 ceph-mon[101879]: pgmap v1666: 105 pgs: 105 active+clean; 133 MiB data, 811 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 685 KiB/s wr, 23 op/s 2024-09-21T10:32:05.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:05.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:06 smithi070 ceph-mon[107385]: pgmap v1667: 105 pgs: 105 active+clean; 133 MiB data, 815 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 686 KiB/s wr, 27 op/s 2024-09-21T10:32:06.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:06 smithi070 ceph-mon[112139]: pgmap v1667: 105 pgs: 105 active+clean; 133 MiB data, 815 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 686 KiB/s wr, 27 op/s 2024-09-21T10:32:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:06 smithi160 ceph-mon[101879]: pgmap v1667: 105 pgs: 105 active+clean; 133 MiB data, 815 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 686 KiB/s wr, 27 op/s 2024-09-21T10:32:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:08 smithi070 ceph-mon[107385]: pgmap v1668: 105 pgs: 105 active+clean; 133 MiB data, 815 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 573 KiB/s wr, 36 op/s 2024-09-21T10:32:08.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:08 smithi070 ceph-mon[112139]: pgmap v1668: 105 pgs: 105 active+clean; 133 MiB data, 815 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 573 KiB/s wr, 36 op/s 2024-09-21T10:32:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:08 smithi160 ceph-mon[101879]: pgmap v1668: 105 pgs: 105 active+clean; 133 MiB data, 815 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 573 KiB/s wr, 36 op/s 2024-09-21T10:32:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:09 smithi160 ceph-mon[101879]: pgmap v1669: 105 pgs: 105 active+clean; 133 MiB data, 815 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 503 KiB/s wr, 29 op/s 2024-09-21T10:32:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:09 smithi160 ceph-mon[101879]: osdmap e1072: 8 total, 8 up, 8 in 2024-09-21T10:32:09.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:09 smithi070 ceph-mon[107385]: pgmap v1669: 105 pgs: 105 active+clean; 133 MiB data, 815 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 503 KiB/s wr, 29 op/s 2024-09-21T10:32:09.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:09 smithi070 ceph-mon[107385]: osdmap e1072: 8 total, 8 up, 8 in 2024-09-21T10:32:09.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:09 smithi070 ceph-mon[112139]: pgmap v1669: 105 pgs: 105 active+clean; 133 MiB data, 815 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 503 KiB/s wr, 29 op/s 2024-09-21T10:32:09.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:09 smithi070 ceph-mon[112139]: osdmap e1072: 8 total, 8 up, 8 in 2024-09-21T10:32:11.410 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:32:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:11 smithi160 ceph-mon[101879]: osdmap e1073: 8 total, 8 up, 8 in 2024-09-21T10:32:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:11 smithi160 ceph-mon[101879]: pgmap v1672: 105 pgs: 105 active+clean; 141 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 1.0 MiB/s wr, 16 op/s 2024-09-21T10:32:11.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:11 smithi070 ceph-mon[107385]: osdmap e1073: 8 total, 8 up, 8 in 2024-09-21T10:32:11.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:11 smithi070 ceph-mon[107385]: pgmap v1672: 105 pgs: 105 active+clean; 141 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 1.0 MiB/s wr, 16 op/s 2024-09-21T10:32:11.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:11 smithi070 ceph-mon[112139]: osdmap e1073: 8 total, 8 up, 8 in 2024-09-21T10:32:11.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:11 smithi070 ceph-mon[112139]: pgmap v1672: 105 pgs: 105 active+clean; 141 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 1.0 MiB/s wr, 16 op/s 2024-09-21T10:32:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:12 smithi160 ceph-mon[101879]: osdmap e1074: 8 total, 8 up, 8 in 2024-09-21T10:32:12.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:12 smithi070 ceph-mon[112139]: osdmap e1074: 8 total, 8 up, 8 in 2024-09-21T10:32:12.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:12 smithi070 ceph-mon[107385]: osdmap e1074: 8 total, 8 up, 8 in 2024-09-21T10:32:13.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:13 smithi070 ceph-mon[107385]: pgmap v1674: 105 pgs: 105 active+clean; 149 MiB data, 851 MiB used, 714 GiB / 715 GiB avail; 6.3 KiB/s rd, 2.7 MiB/s wr, 12 op/s 2024-09-21T10:32:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:32:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:32:13] "GET /metrics HTTP/1.1" 200 36471 "" "Prometheus/2.51.0" 2024-09-21T10:32:13.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:13 smithi070 ceph-mon[112139]: pgmap v1674: 105 pgs: 105 active+clean; 149 MiB data, 851 MiB used, 714 GiB / 715 GiB avail; 6.3 KiB/s rd, 2.7 MiB/s wr, 12 op/s 2024-09-21T10:32:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:13 smithi160 ceph-mon[101879]: pgmap v1674: 105 pgs: 105 active+clean; 149 MiB data, 851 MiB used, 714 GiB / 715 GiB avail; 6.3 KiB/s rd, 2.7 MiB/s wr, 12 op/s 2024-09-21T10:32:15.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:15 smithi070 ceph-mon[107385]: pgmap v1675: 105 pgs: 105 active+clean; 149 MiB data, 863 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 2.7 MiB/s wr, 32 op/s 2024-09-21T10:32:15.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:15 smithi070 ceph-mon[112139]: pgmap v1675: 105 pgs: 105 active+clean; 149 MiB data, 863 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 2.7 MiB/s wr, 32 op/s 2024-09-21T10:32:16.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:15 smithi160 ceph-mon[101879]: pgmap v1675: 105 pgs: 105 active+clean; 149 MiB data, 863 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 2.7 MiB/s wr, 32 op/s 2024-09-21T10:32:17.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:17 smithi070 ceph-mon[107385]: pgmap v1676: 105 pgs: 105 active+clean; 149 MiB data, 864 MiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 1.7 MiB/s wr, 36 op/s 2024-09-21T10:32:17.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:17 smithi070 ceph-mon[112139]: pgmap v1676: 105 pgs: 105 active+clean; 149 MiB data, 864 MiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 1.7 MiB/s wr, 36 op/s 2024-09-21T10:32:18.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:17 smithi160 ceph-mon[101879]: pgmap v1676: 105 pgs: 105 active+clean; 149 MiB data, 864 MiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 1.7 MiB/s wr, 36 op/s 2024-09-21T10:32:19.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:19 smithi070 ceph-mon[107385]: pgmap v1677: 105 pgs: 105 active+clean; 149 MiB data, 864 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 1007 KiB/s wr, 31 op/s 2024-09-21T10:32:19.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:19 smithi070 ceph-mon[112139]: pgmap v1677: 105 pgs: 105 active+clean; 149 MiB data, 864 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 1007 KiB/s wr, 31 op/s 2024-09-21T10:32:20.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:19 smithi160 ceph-mon[101879]: pgmap v1677: 105 pgs: 105 active+clean; 149 MiB data, 864 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 1007 KiB/s wr, 31 op/s 2024-09-21T10:32:20.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:20 smithi070 ceph-mon[107385]: osdmap e1075: 8 total, 8 up, 8 in 2024-09-21T10:32:20.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:20.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:20 smithi070 ceph-mon[112139]: osdmap e1075: 8 total, 8 up, 8 in 2024-09-21T10:32:20.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:20.992 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:20 smithi160 ceph-mon[101879]: osdmap e1075: 8 total, 8 up, 8 in 2024-09-21T10:32:20.993 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:21.641 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:32:21.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:21 smithi070 ceph-mon[107385]: pgmap v1679: 105 pgs: 105 active+clean; 161 MiB data, 884 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 1.3 MiB/s wr, 25 op/s 2024-09-21T10:32:21.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:21 smithi070 ceph-mon[107385]: osdmap e1076: 8 total, 8 up, 8 in 2024-09-21T10:32:21.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:21 smithi070 ceph-mon[112139]: pgmap v1679: 105 pgs: 105 active+clean; 161 MiB data, 884 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 1.3 MiB/s wr, 25 op/s 2024-09-21T10:32:21.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:21 smithi070 ceph-mon[112139]: osdmap e1076: 8 total, 8 up, 8 in 2024-09-21T10:32:22.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:21 smithi160 ceph-mon[101879]: pgmap v1679: 105 pgs: 105 active+clean; 161 MiB data, 884 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 1.3 MiB/s wr, 25 op/s 2024-09-21T10:32:22.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:21 smithi160 ceph-mon[101879]: osdmap e1076: 8 total, 8 up, 8 in 2024-09-21T10:32:22.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:22 smithi070 ceph-mon[107385]: osdmap e1077: 8 total, 8 up, 8 in 2024-09-21T10:32:22.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:22 smithi070 ceph-mon[112139]: osdmap e1077: 8 total, 8 up, 8 in 2024-09-21T10:32:23.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:22 smithi160 ceph-mon[101879]: osdmap e1077: 8 total, 8 up, 8 in 2024-09-21T10:32:23.624 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:32:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:32:23] "GET /metrics HTTP/1.1" 200 36473 "" "Prometheus/2.51.0" 2024-09-21T10:32:23.624 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:23 smithi070 ceph-mon[107385]: pgmap v1682: 105 pgs: 105 active+clean; 167 MiB data, 912 MiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 2.7 MiB/s wr, 19 op/s 2024-09-21T10:32:23.624 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:23 smithi070 ceph-mon[112139]: pgmap v1682: 105 pgs: 105 active+clean; 167 MiB data, 912 MiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 2.7 MiB/s wr, 19 op/s 2024-09-21T10:32:24.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:23 smithi160 ceph-mon[101879]: pgmap v1682: 105 pgs: 105 active+clean; 167 MiB data, 912 MiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 2.7 MiB/s wr, 19 op/s 2024-09-21T10:32:25.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:25 smithi070 ceph-mon[107385]: pgmap v1683: 105 pgs: 105 active+clean; 167 MiB data, 912 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 25 op/s 2024-09-21T10:32:25.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:25 smithi070 ceph-mon[112139]: pgmap v1683: 105 pgs: 105 active+clean; 167 MiB data, 912 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 25 op/s 2024-09-21T10:32:26.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:25 smithi160 ceph-mon[101879]: pgmap v1683: 105 pgs: 105 active+clean; 167 MiB data, 912 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 25 op/s 2024-09-21T10:32:28.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:27 smithi160 ceph-mon[101879]: pgmap v1684: 105 pgs: 105 active+clean; 167 MiB data, 912 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 594 KiB/s wr, 33 op/s 2024-09-21T10:32:28.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:27 smithi070 ceph-mon[107385]: pgmap v1684: 105 pgs: 105 active+clean; 167 MiB data, 912 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 594 KiB/s wr, 33 op/s 2024-09-21T10:32:28.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:27 smithi070 ceph-mon[112139]: pgmap v1684: 105 pgs: 105 active+clean; 167 MiB data, 912 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 594 KiB/s wr, 33 op/s 2024-09-21T10:32:30.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:29 smithi160 ceph-mon[101879]: pgmap v1685: 105 pgs: 105 active+clean; 167 MiB data, 912 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 515 KiB/s wr, 28 op/s 2024-09-21T10:32:30.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:29 smithi070 ceph-mon[107385]: pgmap v1685: 105 pgs: 105 active+clean; 167 MiB data, 912 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 515 KiB/s wr, 28 op/s 2024-09-21T10:32:30.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:29 smithi070 ceph-mon[112139]: pgmap v1685: 105 pgs: 105 active+clean; 167 MiB data, 912 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 515 KiB/s wr, 28 op/s 2024-09-21T10:32:31.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:30 smithi160 ceph-mon[101879]: osdmap e1078: 8 total, 8 up, 8 in 2024-09-21T10:32:31.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:30 smithi070 ceph-mon[107385]: osdmap e1078: 8 total, 8 up, 8 in 2024-09-21T10:32:31.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:30 smithi070 ceph-mon[112139]: osdmap e1078: 8 total, 8 up, 8 in 2024-09-21T10:32:31.674 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:32:32.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:31 smithi160 ceph-mon[101879]: pgmap v1687: 105 pgs: 105 active+clean; 179 MiB data, 928 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 1.3 MiB/s wr, 17 op/s 2024-09-21T10:32:32.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:31 smithi160 ceph-mon[101879]: osdmap e1079: 8 total, 8 up, 8 in 2024-09-21T10:32:32.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:31 smithi070 ceph-mon[107385]: pgmap v1687: 105 pgs: 105 active+clean; 179 MiB data, 928 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 1.3 MiB/s wr, 17 op/s 2024-09-21T10:32:32.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:31 smithi070 ceph-mon[107385]: osdmap e1079: 8 total, 8 up, 8 in 2024-09-21T10:32:32.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:31 smithi070 ceph-mon[112139]: pgmap v1687: 105 pgs: 105 active+clean; 179 MiB data, 928 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 1.3 MiB/s wr, 17 op/s 2024-09-21T10:32:32.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:31 smithi070 ceph-mon[112139]: osdmap e1079: 8 total, 8 up, 8 in 2024-09-21T10:32:33.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:32 smithi160 ceph-mon[101879]: osdmap e1080: 8 total, 8 up, 8 in 2024-09-21T10:32:33.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:32 smithi070 ceph-mon[107385]: osdmap e1080: 8 total, 8 up, 8 in 2024-09-21T10:32:33.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:32 smithi070 ceph-mon[112139]: osdmap e1080: 8 total, 8 up, 8 in 2024-09-21T10:32:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:32:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:32:33] "GET /metrics HTTP/1.1" 200 36473 "" "Prometheus/2.51.0" 2024-09-21T10:32:34.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:33 smithi160 ceph-mon[101879]: pgmap v1690: 105 pgs: 105 active+clean; 184 MiB data, 960 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 2.7 MiB/s wr, 23 op/s 2024-09-21T10:32:34.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:33 smithi070 ceph-mon[112139]: pgmap v1690: 105 pgs: 105 active+clean; 184 MiB data, 960 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 2.7 MiB/s wr, 23 op/s 2024-09-21T10:32:34.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:33 smithi070 ceph-mon[107385]: pgmap v1690: 105 pgs: 105 active+clean; 184 MiB data, 960 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 2.7 MiB/s wr, 23 op/s 2024-09-21T10:32:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:35 smithi070 ceph-mon[107385]: pgmap v1691: 105 pgs: 105 active+clean; 184 MiB data, 960 MiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 2.7 MiB/s wr, 28 op/s 2024-09-21T10:32:35.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:35 smithi070 ceph-mon[112139]: pgmap v1691: 105 pgs: 105 active+clean; 184 MiB data, 960 MiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 2.7 MiB/s wr, 28 op/s 2024-09-21T10:32:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:35 smithi160 ceph-mon[101879]: pgmap v1691: 105 pgs: 105 active+clean; 184 MiB data, 960 MiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 2.7 MiB/s wr, 28 op/s 2024-09-21T10:32:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:38.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:37 smithi160 ceph-mon[101879]: pgmap v1692: 105 pgs: 105 active+clean; 185 MiB data, 961 MiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 1.2 MiB/s wr, 38 op/s 2024-09-21T10:32:38.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:37 smithi070 ceph-mon[107385]: pgmap v1692: 105 pgs: 105 active+clean; 185 MiB data, 961 MiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 1.2 MiB/s wr, 38 op/s 2024-09-21T10:32:38.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:37 smithi070 ceph-mon[112139]: pgmap v1692: 105 pgs: 105 active+clean; 185 MiB data, 961 MiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 1.2 MiB/s wr, 38 op/s 2024-09-21T10:32:40.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:39 smithi160 ceph-mon[101879]: pgmap v1693: 105 pgs: 105 active+clean; 185 MiB data, 961 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 517 KiB/s wr, 33 op/s 2024-09-21T10:32:40.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:39 smithi070 ceph-mon[107385]: pgmap v1693: 105 pgs: 105 active+clean; 185 MiB data, 961 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 517 KiB/s wr, 33 op/s 2024-09-21T10:32:40.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:39 smithi070 ceph-mon[112139]: pgmap v1693: 105 pgs: 105 active+clean; 185 MiB data, 961 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 517 KiB/s wr, 33 op/s 2024-09-21T10:32:41.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:40 smithi160 ceph-mon[101879]: osdmap e1081: 8 total, 8 up, 8 in 2024-09-21T10:32:41.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:40 smithi070 ceph-mon[107385]: osdmap e1081: 8 total, 8 up, 8 in 2024-09-21T10:32:41.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:40 smithi070 ceph-mon[112139]: osdmap e1081: 8 total, 8 up, 8 in 2024-09-21T10:32:41.704 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:32:42.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:41 smithi160 ceph-mon[101879]: pgmap v1695: 105 pgs: 105 active+clean; 185 MiB data, 965 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 2.9 KiB/s wr, 18 op/s 2024-09-21T10:32:42.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:41 smithi160 ceph-mon[101879]: osdmap e1082: 8 total, 8 up, 8 in 2024-09-21T10:32:42.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:41 smithi070 ceph-mon[107385]: pgmap v1695: 105 pgs: 105 active+clean; 185 MiB data, 965 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 2.9 KiB/s wr, 18 op/s 2024-09-21T10:32:42.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:41 smithi070 ceph-mon[107385]: osdmap e1082: 8 total, 8 up, 8 in 2024-09-21T10:32:42.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:41 smithi070 ceph-mon[112139]: pgmap v1695: 105 pgs: 105 active+clean; 185 MiB data, 965 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 2.9 KiB/s wr, 18 op/s 2024-09-21T10:32:42.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:41 smithi070 ceph-mon[112139]: osdmap e1082: 8 total, 8 up, 8 in 2024-09-21T10:32:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:42 smithi160 ceph-mon[101879]: osdmap e1083: 8 total, 8 up, 8 in 2024-09-21T10:32:43.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:42 smithi070 ceph-mon[107385]: osdmap e1083: 8 total, 8 up, 8 in 2024-09-21T10:32:43.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:42 smithi070 ceph-mon[112139]: osdmap e1083: 8 total, 8 up, 8 in 2024-09-21T10:32:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:32:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:32:43] "GET /metrics HTTP/1.1" 200 36480 "" "Prometheus/2.51.0" 2024-09-21T10:32:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:43 smithi160 ceph-mon[101879]: pgmap v1698: 105 pgs: 105 active+clean; 198 MiB data, 989 MiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 MiB/s wr, 7 op/s 2024-09-21T10:32:44.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:43 smithi070 ceph-mon[107385]: pgmap v1698: 105 pgs: 105 active+clean; 198 MiB data, 989 MiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 MiB/s wr, 7 op/s 2024-09-21T10:32:44.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:43 smithi070 ceph-mon[112139]: pgmap v1698: 105 pgs: 105 active+clean; 198 MiB data, 989 MiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 MiB/s wr, 7 op/s 2024-09-21T10:32:46.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:45 smithi160 ceph-mon[101879]: pgmap v1699: 105 pgs: 105 active+clean; 203 MiB data, 1009 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 2.7 MiB/s wr, 36 op/s 2024-09-21T10:32:46.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:45 smithi070 ceph-mon[107385]: pgmap v1699: 105 pgs: 105 active+clean; 203 MiB data, 1009 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 2.7 MiB/s wr, 36 op/s 2024-09-21T10:32:46.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:45 smithi070 ceph-mon[112139]: pgmap v1699: 105 pgs: 105 active+clean; 203 MiB data, 1009 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 2.7 MiB/s wr, 36 op/s 2024-09-21T10:32:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:47 smithi160 ceph-mon[101879]: pgmap v1700: 105 pgs: 105 active+clean; 203 MiB data, 1009 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 2.3 MiB/s wr, 33 op/s 2024-09-21T10:32:48.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:47 smithi070 ceph-mon[107385]: pgmap v1700: 105 pgs: 105 active+clean; 203 MiB data, 1009 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 2.3 MiB/s wr, 33 op/s 2024-09-21T10:32:48.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:47 smithi070 ceph-mon[112139]: pgmap v1700: 105 pgs: 105 active+clean; 203 MiB data, 1009 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 2.3 MiB/s wr, 33 op/s 2024-09-21T10:32:50.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:49 smithi160 ceph-mon[101879]: pgmap v1701: 105 pgs: 105 active+clean; 203 MiB data, 1009 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 2.0 MiB/s wr, 28 op/s 2024-09-21T10:32:50.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:49 smithi070 ceph-mon[112139]: pgmap v1701: 105 pgs: 105 active+clean; 203 MiB data, 1009 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 2.0 MiB/s wr, 28 op/s 2024-09-21T10:32:50.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:49 smithi070 ceph-mon[107385]: pgmap v1701: 105 pgs: 105 active+clean; 203 MiB data, 1009 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 2.0 MiB/s wr, 28 op/s 2024-09-21T10:32:50.993 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:50 smithi160 ceph-mon[101879]: osdmap e1084: 8 total, 8 up, 8 in 2024-09-21T10:32:50.993 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:51.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:50 smithi070 ceph-mon[112139]: osdmap e1084: 8 total, 8 up, 8 in 2024-09-21T10:32:51.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:51.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:50 smithi070 ceph-mon[107385]: osdmap e1084: 8 total, 8 up, 8 in 2024-09-21T10:32:51.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:51.737 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/0.DiffIterateDeterministicLUKS2 (50354 ms) 2024-09-21T10:32:51.737 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/0.DiffIterateDeterministicLUKS2PP 2024-09-21T10:32:51.738 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:32:52.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:51 smithi160 ceph-mon[101879]: pgmap v1703: 105 pgs: 105 active+clean; 203 MiB data, 1009 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 464 KiB/s wr, 25 op/s 2024-09-21T10:32:52.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:51 smithi160 ceph-mon[101879]: osdmap e1085: 8 total, 8 up, 8 in 2024-09-21T10:32:52.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:51 smithi070 ceph-mon[107385]: pgmap v1703: 105 pgs: 105 active+clean; 203 MiB data, 1009 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 464 KiB/s wr, 25 op/s 2024-09-21T10:32:52.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:51 smithi070 ceph-mon[107385]: osdmap e1085: 8 total, 8 up, 8 in 2024-09-21T10:32:52.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:51 smithi070 ceph-mon[112139]: pgmap v1703: 105 pgs: 105 active+clean; 203 MiB data, 1009 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 464 KiB/s wr, 25 op/s 2024-09-21T10:32:52.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:51 smithi070 ceph-mon[112139]: osdmap e1085: 8 total, 8 up, 8 in 2024-09-21T10:32:53.028 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:52 smithi070 ceph-mon[107385]: osdmap e1086: 8 total, 8 up, 8 in 2024-09-21T10:32:53.028 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:52 smithi070 ceph-mon[112139]: osdmap e1086: 8 total, 8 up, 8 in 2024-09-21T10:32:53.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:52 smithi160 ceph-mon[101879]: osdmap e1086: 8 total, 8 up, 8 in 2024-09-21T10:32:53.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:32:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:32:53] "GET /metrics HTTP/1.1" 200 36482 "" "Prometheus/2.51.0" 2024-09-21T10:32:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:53 smithi160 ceph-mon[101879]: pgmap v1706: 105 pgs: 105 active+clean; 211 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 686 KiB/s wr, 22 op/s 2024-09-21T10:32:54.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:53 smithi070 ceph-mon[107385]: pgmap v1706: 105 pgs: 105 active+clean; 211 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 686 KiB/s wr, 22 op/s 2024-09-21T10:32:54.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:53 smithi070 ceph-mon[112139]: pgmap v1706: 105 pgs: 105 active+clean; 211 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 686 KiB/s wr, 22 op/s 2024-09-21T10:32:56.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:55 smithi160 ceph-mon[101879]: pgmap v1707: 105 pgs: 105 active+clean; 223 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 25 op/s 2024-09-21T10:32:56.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:55 smithi070 ceph-mon[107385]: pgmap v1707: 105 pgs: 105 active+clean; 223 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 25 op/s 2024-09-21T10:32:56.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:55 smithi070 ceph-mon[112139]: pgmap v1707: 105 pgs: 105 active+clean; 223 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 25 op/s 2024-09-21T10:32:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:57 smithi160 ceph-mon[101879]: pgmap v1708: 105 pgs: 105 active+clean; 227 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.3 MiB/s wr, 39 op/s 2024-09-21T10:32:58.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:57 smithi070 ceph-mon[107385]: pgmap v1708: 105 pgs: 105 active+clean; 227 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.3 MiB/s wr, 39 op/s 2024-09-21T10:32:58.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:57 smithi070 ceph-mon[112139]: pgmap v1708: 105 pgs: 105 active+clean; 227 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.3 MiB/s wr, 39 op/s 2024-09-21T10:33:00.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:59 smithi160 ceph-mon[101879]: pgmap v1709: 105 pgs: 105 active+clean; 227 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 2.0 MiB/s wr, 33 op/s 2024-09-21T10:33:00.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:32:59 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:33:00.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:59 smithi070 ceph-mon[107385]: pgmap v1709: 105 pgs: 105 active+clean; 227 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 2.0 MiB/s wr, 33 op/s 2024-09-21T10:33:00.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:32:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:33:00.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:59 smithi070 ceph-mon[112139]: pgmap v1709: 105 pgs: 105 active+clean; 227 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 2.0 MiB/s wr, 33 op/s 2024-09-21T10:33:00.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:32:59 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:33:01.039 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:00 smithi160 ceph-mon[101879]: osdmap e1087: 8 total, 8 up, 8 in 2024-09-21T10:33:01.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:00 smithi070 ceph-mon[107385]: osdmap e1087: 8 total, 8 up, 8 in 2024-09-21T10:33:01.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:00 smithi070 ceph-mon[112139]: osdmap e1087: 8 total, 8 up, 8 in 2024-09-21T10:33:01.763 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:33:02.038 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:01 smithi160 ceph-mon[101879]: pgmap v1711: 105 pgs: 105 active+clean; 239 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.4 KiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-21T10:33:02.038 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:01 smithi160 ceph-mon[101879]: osdmap e1088: 8 total, 8 up, 8 in 2024-09-21T10:33:02.067 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:01 smithi070 ceph-mon[107385]: pgmap v1711: 105 pgs: 105 active+clean; 239 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.4 KiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-21T10:33:02.067 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:01 smithi070 ceph-mon[107385]: osdmap e1088: 8 total, 8 up, 8 in 2024-09-21T10:33:02.067 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:01 smithi070 ceph-mon[112139]: pgmap v1711: 105 pgs: 105 active+clean; 239 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.4 KiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-21T10:33:02.067 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:01 smithi070 ceph-mon[112139]: osdmap e1088: 8 total, 8 up, 8 in 2024-09-21T10:33:02.863 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:02 smithi160 ceph-mon[101879]: osdmap e1089: 8 total, 8 up, 8 in 2024-09-21T10:33:03.055 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:02 smithi070 ceph-mon[107385]: osdmap e1089: 8 total, 8 up, 8 in 2024-09-21T10:33:03.055 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:02 smithi070 ceph-mon[112139]: osdmap e1089: 8 total, 8 up, 8 in 2024-09-21T10:33:03.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:33:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:33:03] "GET /metrics HTTP/1.1" 200 36482 "" "Prometheus/2.51.0" 2024-09-21T10:33:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:04 smithi160 ceph-mon[101879]: pgmap v1714: 105 pgs: 105 active+clean; 239 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.0 MiB/s wr, 13 op/s 2024-09-21T10:33:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:04 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:33:04.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:04 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:33:04.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:04 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:33:04.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:04 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:33:04.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:04 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:33:04.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:04 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:33:04.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:04 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:33:04.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:04 smithi070 ceph-mon[107385]: pgmap v1714: 105 pgs: 105 active+clean; 239 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.0 MiB/s wr, 13 op/s 2024-09-21T10:33:04.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:04 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:33:04.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:04 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:33:04.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:04 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:33:04.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:04 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:33:04.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:04 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:33:04.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:04 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:33:04.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:04 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:33:04.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:04 smithi070 ceph-mon[112139]: pgmap v1714: 105 pgs: 105 active+clean; 239 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.0 MiB/s wr, 13 op/s 2024-09-21T10:33:04.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:04 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:33:04.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:04 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:33:04.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:04 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:33:04.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:04 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:33:04.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:04 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:33:04.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:04 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:33:04.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:04 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:33:06.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:06 smithi160 ceph-mon[101879]: pgmap v1715: 105 pgs: 105 active+clean; 243 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 27 op/s 2024-09-21T10:33:06.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:06 smithi070 ceph-mon[107385]: pgmap v1715: 105 pgs: 105 active+clean; 243 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 27 op/s 2024-09-21T10:33:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:06.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:06 smithi070 ceph-mon[112139]: pgmap v1715: 105 pgs: 105 active+clean; 243 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 27 op/s 2024-09-21T10:33:06.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:08 smithi160 ceph-mon[101879]: pgmap v1716: 105 pgs: 105 active+clean; 243 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 605 KiB/s wr, 38 op/s 2024-09-21T10:33:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:08 smithi160 ceph-mon[101879]: osdmap e1090: 8 total, 8 up, 8 in 2024-09-21T10:33:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:08 smithi070 ceph-mon[107385]: pgmap v1716: 105 pgs: 105 active+clean; 243 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 605 KiB/s wr, 38 op/s 2024-09-21T10:33:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:08 smithi070 ceph-mon[107385]: osdmap e1090: 8 total, 8 up, 8 in 2024-09-21T10:33:08.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:08 smithi070 ceph-mon[112139]: pgmap v1716: 105 pgs: 105 active+clean; 243 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 605 KiB/s wr, 38 op/s 2024-09-21T10:33:08.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:08 smithi070 ceph-mon[112139]: osdmap e1090: 8 total, 8 up, 8 in 2024-09-21T10:33:09.360 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:33:09.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:09 smithi160 ceph-mon[101879]: osdmap e1091: 8 total, 8 up, 8 in 2024-09-21T10:33:09.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:09 smithi070 ceph-mon[112139]: osdmap e1091: 8 total, 8 up, 8 in 2024-09-21T10:33:09.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:09 smithi070 ceph-mon[107385]: osdmap e1091: 8 total, 8 up, 8 in 2024-09-21T10:33:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:10 smithi160 ceph-mon[101879]: pgmap v1719: 105 pgs: 105 active+clean; 243 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 605 KiB/s wr, 27 op/s 2024-09-21T10:33:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:10 smithi160 ceph-mon[101879]: osdmap e1092: 8 total, 8 up, 8 in 2024-09-21T10:33:10.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:10 smithi070 ceph-mon[107385]: pgmap v1719: 105 pgs: 105 active+clean; 243 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 605 KiB/s wr, 27 op/s 2024-09-21T10:33:10.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:10 smithi070 ceph-mon[107385]: osdmap e1092: 8 total, 8 up, 8 in 2024-09-21T10:33:10.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:10 smithi070 ceph-mon[112139]: pgmap v1719: 105 pgs: 105 active+clean; 243 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 605 KiB/s wr, 27 op/s 2024-09-21T10:33:10.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:10 smithi070 ceph-mon[112139]: osdmap e1092: 8 total, 8 up, 8 in 2024-09-21T10:33:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:12 smithi160 ceph-mon[101879]: pgmap v1721: 105 pgs: 105 active+clean; 251 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 1.3 MiB/s wr, 19 op/s 2024-09-21T10:33:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:12 smithi070 ceph-mon[107385]: pgmap v1721: 105 pgs: 105 active+clean; 251 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 1.3 MiB/s wr, 19 op/s 2024-09-21T10:33:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:12 smithi070 ceph-mon[112139]: pgmap v1721: 105 pgs: 105 active+clean; 251 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 1.3 MiB/s wr, 19 op/s 2024-09-21T10:33:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:33:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:33:13] "GET /metrics HTTP/1.1" 200 36482 "" "Prometheus/2.51.0" 2024-09-21T10:33:14.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:14 smithi160 ceph-mon[101879]: pgmap v1722: 105 pgs: 105 active+clean; 265 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.7 MiB/s wr, 39 op/s 2024-09-21T10:33:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:14 smithi070 ceph-mon[107385]: pgmap v1722: 105 pgs: 105 active+clean; 265 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.7 MiB/s wr, 39 op/s 2024-09-21T10:33:14.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:14 smithi070 ceph-mon[112139]: pgmap v1722: 105 pgs: 105 active+clean; 265 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.7 MiB/s wr, 39 op/s 2024-09-21T10:33:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:16 smithi160 ceph-mon[101879]: pgmap v1723: 105 pgs: 105 active+clean; 265 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 2.2 MiB/s wr, 37 op/s 2024-09-21T10:33:16.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:16 smithi070 ceph-mon[107385]: pgmap v1723: 105 pgs: 105 active+clean; 265 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 2.2 MiB/s wr, 37 op/s 2024-09-21T10:33:16.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:16 smithi070 ceph-mon[112139]: pgmap v1723: 105 pgs: 105 active+clean; 265 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 2.2 MiB/s wr, 37 op/s 2024-09-21T10:33:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:18 smithi070 ceph-mon[107385]: pgmap v1724: 105 pgs: 105 active+clean; 265 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 1.9 MiB/s wr, 32 op/s 2024-09-21T10:33:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:18 smithi070 ceph-mon[107385]: osdmap e1093: 8 total, 8 up, 8 in 2024-09-21T10:33:18.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:18 smithi070 ceph-mon[112139]: pgmap v1724: 105 pgs: 105 active+clean; 265 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 1.9 MiB/s wr, 32 op/s 2024-09-21T10:33:18.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:18 smithi070 ceph-mon[112139]: osdmap e1093: 8 total, 8 up, 8 in 2024-09-21T10:33:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:18 smithi160 ceph-mon[101879]: pgmap v1724: 105 pgs: 105 active+clean; 265 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 1.9 MiB/s wr, 32 op/s 2024-09-21T10:33:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:18 smithi160 ceph-mon[101879]: osdmap e1093: 8 total, 8 up, 8 in 2024-09-21T10:33:19.380 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:33:19.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:19 smithi070 ceph-mon[112139]: osdmap e1094: 8 total, 8 up, 8 in 2024-09-21T10:33:19.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:19 smithi070 ceph-mon[107385]: osdmap e1094: 8 total, 8 up, 8 in 2024-09-21T10:33:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:19 smithi160 ceph-mon[101879]: osdmap e1094: 8 total, 8 up, 8 in 2024-09-21T10:33:20.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:20 smithi070 ceph-mon[112139]: pgmap v1727: 105 pgs: 105 active+clean; 265 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 1.0 MiB/s wr, 32 op/s 2024-09-21T10:33:20.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:20 smithi070 ceph-mon[112139]: osdmap e1095: 8 total, 8 up, 8 in 2024-09-21T10:33:20.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:20.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:20 smithi070 ceph-mon[107385]: pgmap v1727: 105 pgs: 105 active+clean; 265 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 1.0 MiB/s wr, 32 op/s 2024-09-21T10:33:20.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:20 smithi070 ceph-mon[107385]: osdmap e1095: 8 total, 8 up, 8 in 2024-09-21T10:33:20.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:20.743 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:20 smithi160 ceph-mon[101879]: pgmap v1727: 105 pgs: 105 active+clean; 265 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 1.0 MiB/s wr, 32 op/s 2024-09-21T10:33:20.743 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:20 smithi160 ceph-mon[101879]: osdmap e1095: 8 total, 8 up, 8 in 2024-09-21T10:33:20.743 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:22 smithi070 ceph-mon[107385]: pgmap v1729: 105 pgs: 105 active+clean; 269 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 684 KiB/s wr, 14 op/s 2024-09-21T10:33:22.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:22 smithi070 ceph-mon[112139]: pgmap v1729: 105 pgs: 105 active+clean; 269 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 684 KiB/s wr, 14 op/s 2024-09-21T10:33:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:22 smithi160 ceph-mon[101879]: pgmap v1729: 105 pgs: 105 active+clean; 269 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 684 KiB/s wr, 14 op/s 2024-09-21T10:33:23.529 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:33:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:33:23] "GET /metrics HTTP/1.1" 200 36475 "" "Prometheus/2.51.0" 2024-09-21T10:33:24.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:24 smithi070 ceph-mon[107385]: pgmap v1730: 105 pgs: 105 active+clean; 287 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 2.7 MiB/s wr, 36 op/s 2024-09-21T10:33:24.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:24 smithi070 ceph-mon[112139]: pgmap v1730: 105 pgs: 105 active+clean; 287 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 2.7 MiB/s wr, 36 op/s 2024-09-21T10:33:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:24 smithi160 ceph-mon[101879]: pgmap v1730: 105 pgs: 105 active+clean; 287 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 2.7 MiB/s wr, 36 op/s 2024-09-21T10:33:26.365 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:26 smithi070 ceph-mon[112139]: pgmap v1731: 105 pgs: 105 active+clean; 287 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 2.2 MiB/s wr, 37 op/s 2024-09-21T10:33:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:26 smithi070 ceph-mon[107385]: pgmap v1731: 105 pgs: 105 active+clean; 287 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 2.2 MiB/s wr, 37 op/s 2024-09-21T10:33:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:26 smithi160 ceph-mon[101879]: pgmap v1731: 105 pgs: 105 active+clean; 287 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 2.2 MiB/s wr, 37 op/s 2024-09-21T10:33:28.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:28 smithi070 ceph-mon[107385]: pgmap v1732: 105 pgs: 105 active+clean; 287 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 2.0 MiB/s wr, 32 op/s 2024-09-21T10:33:28.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:28 smithi070 ceph-mon[107385]: osdmap e1096: 8 total, 8 up, 8 in 2024-09-21T10:33:28.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:28 smithi070 ceph-mon[112139]: pgmap v1732: 105 pgs: 105 active+clean; 287 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 2.0 MiB/s wr, 32 op/s 2024-09-21T10:33:28.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:28 smithi070 ceph-mon[112139]: osdmap e1096: 8 total, 8 up, 8 in 2024-09-21T10:33:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:28 smithi160 ceph-mon[101879]: pgmap v1732: 105 pgs: 105 active+clean; 287 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 2.0 MiB/s wr, 32 op/s 2024-09-21T10:33:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:28 smithi160 ceph-mon[101879]: osdmap e1096: 8 total, 8 up, 8 in 2024-09-21T10:33:29.409 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:33:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:29 smithi160 ceph-mon[101879]: osdmap e1097: 8 total, 8 up, 8 in 2024-09-21T10:33:29.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:29 smithi070 ceph-mon[107385]: osdmap e1097: 8 total, 8 up, 8 in 2024-09-21T10:33:29.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:29 smithi070 ceph-mon[112139]: osdmap e1097: 8 total, 8 up, 8 in 2024-09-21T10:33:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:30 smithi160 ceph-mon[101879]: pgmap v1735: 105 pgs: 105 active+clean; 287 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 1.5 MiB/s wr, 22 op/s 2024-09-21T10:33:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:30 smithi160 ceph-mon[101879]: osdmap e1098: 8 total, 8 up, 8 in 2024-09-21T10:33:30.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:30 smithi070 ceph-mon[107385]: pgmap v1735: 105 pgs: 105 active+clean; 287 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 1.5 MiB/s wr, 22 op/s 2024-09-21T10:33:30.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:30 smithi070 ceph-mon[107385]: osdmap e1098: 8 total, 8 up, 8 in 2024-09-21T10:33:30.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:30 smithi070 ceph-mon[112139]: pgmap v1735: 105 pgs: 105 active+clean; 287 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 1.5 MiB/s wr, 22 op/s 2024-09-21T10:33:30.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:30 smithi070 ceph-mon[112139]: osdmap e1098: 8 total, 8 up, 8 in 2024-09-21T10:33:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:32 smithi160 ceph-mon[101879]: pgmap v1737: 105 pgs: 105 active+clean; 302 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.0 KiB/s rd, 2.0 MiB/s wr, 15 op/s 2024-09-21T10:33:32.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:32 smithi070 ceph-mon[107385]: pgmap v1737: 105 pgs: 105 active+clean; 302 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.0 KiB/s rd, 2.0 MiB/s wr, 15 op/s 2024-09-21T10:33:32.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:32 smithi070 ceph-mon[112139]: pgmap v1737: 105 pgs: 105 active+clean; 302 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.0 KiB/s rd, 2.0 MiB/s wr, 15 op/s 2024-09-21T10:33:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:33:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:33:33] "GET /metrics HTTP/1.1" 200 36475 "" "Prometheus/2.51.0" 2024-09-21T10:33:33.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:33 smithi070 ceph-mon[107385]: pgmap v1738: 105 pgs: 105 active+clean; 309 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 MiB/s wr, 44 op/s 2024-09-21T10:33:33.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:33 smithi070 ceph-mon[112139]: pgmap v1738: 105 pgs: 105 active+clean; 309 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 MiB/s wr, 44 op/s 2024-09-21T10:33:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:33 smithi160 ceph-mon[101879]: pgmap v1738: 105 pgs: 105 active+clean; 309 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 MiB/s wr, 44 op/s 2024-09-21T10:33:35.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:35 smithi070 ceph-mon[107385]: pgmap v1739: 105 pgs: 105 active+clean; 309 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 2.2 MiB/s wr, 37 op/s 2024-09-21T10:33:35.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:35.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:35 smithi070 ceph-mon[112139]: pgmap v1739: 105 pgs: 105 active+clean; 309 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 2.2 MiB/s wr, 37 op/s 2024-09-21T10:33:35.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:35 smithi160 ceph-mon[101879]: pgmap v1739: 105 pgs: 105 active+clean; 309 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 2.2 MiB/s wr, 37 op/s 2024-09-21T10:33:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:37.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:37 smithi070 ceph-mon[107385]: pgmap v1740: 105 pgs: 105 active+clean; 309 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 2.0 MiB/s wr, 32 op/s 2024-09-21T10:33:37.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:37 smithi070 ceph-mon[112139]: pgmap v1740: 105 pgs: 105 active+clean; 309 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 2.0 MiB/s wr, 32 op/s 2024-09-21T10:33:38.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:37 smithi160 ceph-mon[101879]: pgmap v1740: 105 pgs: 105 active+clean; 309 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 2.0 MiB/s wr, 32 op/s 2024-09-21T10:33:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:38 smithi160 ceph-mon[101879]: osdmap e1099: 8 total, 8 up, 8 in 2024-09-21T10:33:39.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:38 smithi070 ceph-mon[107385]: osdmap e1099: 8 total, 8 up, 8 in 2024-09-21T10:33:39.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:38 smithi070 ceph-mon[112139]: osdmap e1099: 8 total, 8 up, 8 in 2024-09-21T10:33:39.665 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/0.DiffIterateDeterministicLUKS2PP (47928 ms) 2024-09-21T10:33:39.665 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/0.DiffIterateDiscard 2024-09-21T10:33:39.666 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:33:40.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:39 smithi160 ceph-mon[101879]: pgmap v1742: 105 pgs: 105 active+clean; 309 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 450 KiB/s wr, 28 op/s 2024-09-21T10:33:40.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:39 smithi160 ceph-mon[101879]: osdmap e1100: 8 total, 8 up, 8 in 2024-09-21T10:33:40.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:39 smithi070 ceph-mon[107385]: pgmap v1742: 105 pgs: 105 active+clean; 309 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 450 KiB/s wr, 28 op/s 2024-09-21T10:33:40.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:39 smithi070 ceph-mon[107385]: osdmap e1100: 8 total, 8 up, 8 in 2024-09-21T10:33:40.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:39 smithi070 ceph-mon[112139]: pgmap v1742: 105 pgs: 105 active+clean; 309 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 450 KiB/s wr, 28 op/s 2024-09-21T10:33:40.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:39 smithi070 ceph-mon[112139]: osdmap e1100: 8 total, 8 up, 8 in 2024-09-21T10:33:41.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:40 smithi160 ceph-mon[101879]: osdmap e1101: 8 total, 8 up, 8 in 2024-09-21T10:33:41.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:40 smithi070 ceph-mon[107385]: osdmap e1101: 8 total, 8 up, 8 in 2024-09-21T10:33:41.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:40 smithi070 ceph-mon[112139]: osdmap e1101: 8 total, 8 up, 8 in 2024-09-21T10:33:42.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:41 smithi160 ceph-mon[101879]: pgmap v1745: 105 pgs: 105 active+clean; 325 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.0 KiB/s rd, 2.0 MiB/s wr, 13 op/s 2024-09-21T10:33:42.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:41 smithi160 ceph-mon[101879]: osdmap e1102: 8 total, 8 up, 8 in 2024-09-21T10:33:42.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:41 smithi070 ceph-mon[107385]: pgmap v1745: 105 pgs: 105 active+clean; 325 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.0 KiB/s rd, 2.0 MiB/s wr, 13 op/s 2024-09-21T10:33:42.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:41 smithi070 ceph-mon[107385]: osdmap e1102: 8 total, 8 up, 8 in 2024-09-21T10:33:42.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:41 smithi070 ceph-mon[112139]: pgmap v1745: 105 pgs: 105 active+clean; 325 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.0 KiB/s rd, 2.0 MiB/s wr, 13 op/s 2024-09-21T10:33:42.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:41 smithi070 ceph-mon[112139]: osdmap e1102: 8 total, 8 up, 8 in 2024-09-21T10:33:42.680 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/0.DiffIterateDiscard (3015 ms) 2024-09-21T10:33:42.680 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/0.DiffIterateTruncate 2024-09-21T10:33:42.681 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:33:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:42 smithi160 ceph-mon[101879]: osdmap e1103: 8 total, 8 up, 8 in 2024-09-21T10:33:43.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:42 smithi070 ceph-mon[107385]: osdmap e1103: 8 total, 8 up, 8 in 2024-09-21T10:33:43.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:42 smithi070 ceph-mon[112139]: osdmap e1103: 8 total, 8 up, 8 in 2024-09-21T10:33:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:33:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:33:43] "GET /metrics HTTP/1.1" 200 36471 "" "Prometheus/2.51.0" 2024-09-21T10:33:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:43 smithi160 ceph-mon[101879]: pgmap v1748: 105 pgs: 105 active+clean; 333 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 47 KiB/s rd, 4.0 MiB/s wr, 76 op/s 2024-09-21T10:33:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:43 smithi160 ceph-mon[101879]: osdmap e1104: 8 total, 8 up, 8 in 2024-09-21T10:33:44.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:43 smithi070 ceph-mon[107385]: pgmap v1748: 105 pgs: 105 active+clean; 333 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 47 KiB/s rd, 4.0 MiB/s wr, 76 op/s 2024-09-21T10:33:44.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:43 smithi070 ceph-mon[107385]: osdmap e1104: 8 total, 8 up, 8 in 2024-09-21T10:33:44.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:43 smithi070 ceph-mon[112139]: pgmap v1748: 105 pgs: 105 active+clean; 333 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 47 KiB/s rd, 4.0 MiB/s wr, 76 op/s 2024-09-21T10:33:44.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:43 smithi070 ceph-mon[112139]: osdmap e1104: 8 total, 8 up, 8 in 2024-09-21T10:33:45.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:44 smithi160 ceph-mon[101879]: osdmap e1105: 8 total, 8 up, 8 in 2024-09-21T10:33:45.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:44 smithi070 ceph-mon[107385]: osdmap e1105: 8 total, 8 up, 8 in 2024-09-21T10:33:45.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:44 smithi070 ceph-mon[112139]: osdmap e1105: 8 total, 8 up, 8 in 2024-09-21T10:33:45.716 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/0.DiffIterateTruncate (3036 ms) 2024-09-21T10:33:45.716 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/0.DiffIterateWriteAndTruncate 2024-09-21T10:33:45.716 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:33:46.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:45 smithi160 ceph-mon[101879]: pgmap v1751: 105 pgs: 105 active+clean; 333 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 1.1 MiB/s wr, 63 op/s 2024-09-21T10:33:46.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:45 smithi160 ceph-mon[101879]: osdmap e1106: 8 total, 8 up, 8 in 2024-09-21T10:33:46.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:45 smithi070 ceph-mon[112139]: pgmap v1751: 105 pgs: 105 active+clean; 333 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 1.1 MiB/s wr, 63 op/s 2024-09-21T10:33:46.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:45 smithi070 ceph-mon[112139]: osdmap e1106: 8 total, 8 up, 8 in 2024-09-21T10:33:46.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:45 smithi070 ceph-mon[107385]: pgmap v1751: 105 pgs: 105 active+clean; 333 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 1.1 MiB/s wr, 63 op/s 2024-09-21T10:33:46.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:45 smithi070 ceph-mon[107385]: osdmap e1106: 8 total, 8 up, 8 in 2024-09-21T10:33:47.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:46 smithi160 ceph-mon[101879]: osdmap e1107: 8 total, 8 up, 8 in 2024-09-21T10:33:47.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:46 smithi070 ceph-mon[107385]: osdmap e1107: 8 total, 8 up, 8 in 2024-09-21T10:33:47.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:46 smithi070 ceph-mon[112139]: osdmap e1107: 8 total, 8 up, 8 in 2024-09-21T10:33:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:47 smithi160 ceph-mon[101879]: pgmap v1754: 105 pgs: 105 active+clean; 333 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 65 KiB/s rd, 136 KiB/s wr, 100 op/s 2024-09-21T10:33:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:47 smithi160 ceph-mon[101879]: osdmap e1108: 8 total, 8 up, 8 in 2024-09-21T10:33:48.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:47 smithi070 ceph-mon[112139]: pgmap v1754: 105 pgs: 105 active+clean; 333 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 65 KiB/s rd, 136 KiB/s wr, 100 op/s 2024-09-21T10:33:48.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:47 smithi070 ceph-mon[112139]: osdmap e1108: 8 total, 8 up, 8 in 2024-09-21T10:33:48.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:47 smithi070 ceph-mon[107385]: pgmap v1754: 105 pgs: 105 active+clean; 333 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 65 KiB/s rd, 136 KiB/s wr, 100 op/s 2024-09-21T10:33:48.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:47 smithi070 ceph-mon[107385]: osdmap e1108: 8 total, 8 up, 8 in 2024-09-21T10:33:48.728 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/0.DiffIterateWriteAndTruncate (3012 ms) 2024-09-21T10:33:48.728 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/0.DiffIterateStress 2024-09-21T10:33:48.728 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:33:48.808 INFO:tasks.workunit.client.0.smithi070.stdout: i=0 exists [36667536~4209336,57228549~303119,79026249~7399686,119268089~4418966,152571700~77698,307178110~2334570,309571487~12024809,362062612~8053411] wrote [36667536~4209336,57228549~303119,79026249~7399686,119268089~4418966,152571700~77698,307178110~2334570,309571487~12024809,362062612~8053411] 2024-09-21T10:33:49.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:48 smithi160 ceph-mon[101879]: osdmap e1109: 8 total, 8 up, 8 in 2024-09-21T10:33:49.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:48 smithi070 ceph-mon[107385]: osdmap e1109: 8 total, 8 up, 8 in 2024-09-21T10:33:49.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:48 smithi070 ceph-mon[112139]: osdmap e1109: 8 total, 8 up, 8 in 2024-09-21T10:33:49.754 INFO:tasks.workunit.client.0.smithi070.stdout: i=1 exists [36667536~4209336,57228549~303119,72446799~256446,79026249~7399686,119268089~4418966,139085476~6479527,152571700~77698,168542535~3547892,191243759~5891731,307178110~14418186,323189366~3910707,362062612~11816902,376447308~19278,388773220~4929446,410761440~3528643] wrote [72446799~256446,139085476~6479527,168542535~3547892,191243759~5891731,308097160~5425190,323189366~3910707,367586957~6292557,376447308~19278,388773220~4929446,410761440~3528643] 2024-09-21T10:33:50.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:49 smithi160 ceph-mon[101879]: pgmap v1757: 105 pgs: 105 active+clean; 333 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 63 KiB/s rd, 7.0 KiB/s wr, 92 op/s 2024-09-21T10:33:50.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:49 smithi160 ceph-mon[101879]: osdmap e1110: 8 total, 8 up, 8 in 2024-09-21T10:33:50.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:49 smithi070 ceph-mon[107385]: pgmap v1757: 105 pgs: 105 active+clean; 333 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 63 KiB/s rd, 7.0 KiB/s wr, 92 op/s 2024-09-21T10:33:50.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:49 smithi070 ceph-mon[107385]: osdmap e1110: 8 total, 8 up, 8 in 2024-09-21T10:33:50.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:49 smithi070 ceph-mon[112139]: pgmap v1757: 105 pgs: 105 active+clean; 333 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 63 KiB/s rd, 7.0 KiB/s wr, 92 op/s 2024-09-21T10:33:50.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:49 smithi070 ceph-mon[112139]: osdmap e1110: 8 total, 8 up, 8 in 2024-09-21T10:33:50.779 INFO:tasks.workunit.client.0.smithi070.stdout: i=2 exists [1848814~6541440,30271399~11594100,47017959~6564547,57228549~303119,72446799~256446,79026249~7399686,119268089~4418966,132602666~5246362,139085476~6479527,152571700~77698,158059215~1020680,168542535~3547892,191243759~5891731,242881143~2665712,259674737~2646128,307178110~15777254,323189366~3910707,362062612~11816902,376447308~19278,388773220~4929446,410761440~3528643] wrote [1848814~6541440,30271399~11594100,47017959~6564547,132602666~5246362,158059215~1020680,169402859~1955236,242881143~2665712,259674737~2646128,317001032~5954332] 2024-09-21T10:33:50.992 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:50 smithi160 ceph-mon[101879]: osdmap e1111: 8 total, 8 up, 8 in 2024-09-21T10:33:50.992 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:51.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:50 smithi070 ceph-mon[107385]: osdmap e1111: 8 total, 8 up, 8 in 2024-09-21T10:33:51.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:51.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:50 smithi070 ceph-mon[112139]: osdmap e1111: 8 total, 8 up, 8 in 2024-09-21T10:33:51.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:51.773 INFO:tasks.workunit.client.0.smithi070.stdout: i=3 exists [1848814~6541440,30271399~11594100,47017959~6564547,57228549~303119,72446799~256446,75540009~10885926,119268089~4418966,132602666~5246362,139085476~6479527,152571700~77698,158059215~1020680,168542535~3547892,191243759~5891731,198347760~7715923,211221032~4081388,220782438~3218309,242881143~2665712,258320585~4000280,267194315~7846772,307178110~15777254,323189366~3910707,362062612~11816902,376447308~19278,388773220~4929446,410761440~3528643] wrote [75540009~4520923,198347760~7715923,211221032~4081388,220782438~3218309,258320585~2404839,267194315~7846772,314468828~3622333,319022865~2209955,362533785~5973519] 2024-09-21T10:33:52.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:51 smithi160 ceph-mon[101879]: pgmap v1760: 105 pgs: 105 active+clean; 348 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 2.5 MiB/s wr, 36 op/s 2024-09-21T10:33:52.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:51 smithi160 ceph-mon[101879]: osdmap e1112: 8 total, 8 up, 8 in 2024-09-21T10:33:52.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:51 smithi070 ceph-mon[107385]: pgmap v1760: 105 pgs: 105 active+clean; 348 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 2.5 MiB/s wr, 36 op/s 2024-09-21T10:33:52.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:51 smithi070 ceph-mon[107385]: osdmap e1112: 8 total, 8 up, 8 in 2024-09-21T10:33:52.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:51 smithi070 ceph-mon[112139]: pgmap v1760: 105 pgs: 105 active+clean; 348 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 2.5 MiB/s wr, 36 op/s 2024-09-21T10:33:52.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:51 smithi070 ceph-mon[112139]: osdmap e1112: 8 total, 8 up, 8 in 2024-09-21T10:33:52.781 INFO:tasks.workunit.client.0.smithi070.stdout: i=4 exists [1848814~8910670,30271399~11594100,47017959~6564547,57228549~303119,71935619~14490316,106701993~2513206,119268089~4418966,132602666~5246362,139085476~6479527,152571700~77698,152870944~280627,158059215~1020680,168542535~3547892,181304442~350166,186731431~3510386,191243759~5891731,198347760~7715923,211221032~4081388,220782438~3218309,242881143~2665712,255735910~6584955,267194315~8751145,307178110~15777254,323189366~3910707,349416768~304856,362062612~11816902,376447308~19278,388773220~4929446,410761440~3528643] wrote [7577098~3182386,71935619~6785475,106701993~2513206,152870944~280627,181304442~350166,186731431~3510386,255735910~5447384,268907245~7038215,349416768~304856] 2024-09-21T10:33:53.028 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:52 smithi070 ceph-mon[107385]: osdmap e1113: 8 total, 8 up, 8 in 2024-09-21T10:33:53.029 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:52 smithi070 ceph-mon[112139]: osdmap e1113: 8 total, 8 up, 8 in 2024-09-21T10:33:53.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:52 smithi160 ceph-mon[101879]: osdmap e1113: 8 total, 8 up, 8 in 2024-09-21T10:33:53.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:33:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:33:53] "GET /metrics HTTP/1.1" 200 36480 "" "Prometheus/2.51.0" 2024-09-21T10:33:53.787 INFO:tasks.workunit.client.0.smithi070.stdout: i=5 exists [1848814~8910670,30271399~11594100,45048342~8534164,57228549~303119,71935619~14490316,93118269~4798812,101882815~4086712,106701993~2513206,119268089~4418966,127714892~10134136,138318126~7246877,152571700~77698,152870944~280627,158059215~1020680,168542535~3547892,180722510~2055247,186731431~3510386,191243759~14819924,211221032~4081388,220782438~3218309,238706507~3881063,242881143~2665712,255735910~6584955,267194315~8751145,289443310~747069,307178110~15777254,323189366~3910707,349416768~304856,362062612~11816902,376447308~19278,388773220~4929446,410761440~3528643] wrote [45048342~8105318,93118269~4798812,101882815~4086712,127714892~7812122,138318126~6701638,180722510~2055247,195156550~3672575,238706507~3881063,289443310~747069] 2024-09-21T10:33:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:53 smithi160 ceph-mon[101879]: pgmap v1763: 105 pgs: 105 active+clean; 432 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 131 KiB/s rd, 20 MiB/s wr, 217 op/s 2024-09-21T10:33:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:53 smithi160 ceph-mon[101879]: osdmap e1114: 8 total, 8 up, 8 in 2024-09-21T10:33:54.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:53 smithi070 ceph-mon[112139]: pgmap v1763: 105 pgs: 105 active+clean; 432 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 131 KiB/s rd, 20 MiB/s wr, 217 op/s 2024-09-21T10:33:54.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:53 smithi070 ceph-mon[112139]: osdmap e1114: 8 total, 8 up, 8 in 2024-09-21T10:33:54.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:53 smithi070 ceph-mon[107385]: pgmap v1763: 105 pgs: 105 active+clean; 432 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 131 KiB/s rd, 20 MiB/s wr, 217 op/s 2024-09-21T10:33:54.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:53 smithi070 ceph-mon[107385]: osdmap e1114: 8 total, 8 up, 8 in 2024-09-21T10:33:54.783 INFO:tasks.workunit.client.0.smithi070.stdout: i=6 exists [1848814~8910670,28558403~593635,30271399~11594100,45048342~8534164,57228549~303119,57756520~1469204,71935619~14490316,93118269~4798812,101882815~4086712,106701993~2513206,119268089~4418966,127714892~10134136,138318126~7246877,146239075~3597571,152571700~77698,152870944~280627,158059215~1020680,167991057~5531964,180722510~2055247,186731431~3510386,191243759~14819924,211221032~4081388,220782438~3218309,238706507~3881063,242881143~2665712,255735910~6584955,267194315~8751145,289443310~747069,294637427~5715128,303093367~19861997,323189366~3910707,349416768~304856,361489784~12389730,376447308~19278,388773220~4929446,410761440~3528643] wrote [28558403~593635,35578395~2688472,57756520~1469204,146239075~3597571,167991057~5531964,294637427~5715128,303093367~4354075,319887642~2175374,361489784~7914531] 2024-09-21T10:33:55.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:54 smithi160 ceph-mon[101879]: osdmap e1115: 8 total, 8 up, 8 in 2024-09-21T10:33:55.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:54 smithi070 ceph-mon[107385]: osdmap e1115: 8 total, 8 up, 8 in 2024-09-21T10:33:55.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:54 smithi070 ceph-mon[112139]: osdmap e1115: 8 total, 8 up, 8 in 2024-09-21T10:33:55.793 INFO:tasks.workunit.client.0.smithi070.stdout: i=7 exists [1848814~8910670,28558403~593635,30271399~11594100,45048342~8534164,57228549~303119,57756520~1469204,71935619~14490316,93118269~4798812,101882815~4086712,106701993~2513206,119268089~4418966,127714892~10134136,138318126~7793832,146239075~3597571,152571700~77698,152870944~280627,158059215~1020680,167991057~6561399,180722510~2055247,186731431~3510386,191243759~14819924,211221032~4081388,220782438~3218309,224340984~6971051,238706507~3881063,242881143~2665712,255735910~6584955,267194315~8751145,289443310~747069,294637427~5715128,303093367~25576710,349416768~304856,361489784~15792220,388773220~4929446,400476606~3423466,410761440~3528643] wrote [7016728~1370550,39931377~204155,140524653~5587305,172681089~1871367,224340984~6971051,322140464~6529613,365837885~120768,369976939~7305065,400476606~3423466] 2024-09-21T10:33:56.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:55 smithi160 ceph-mon[101879]: pgmap v1766: 105 pgs: 105 active+clean; 518 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 115 KiB/s rd, 34 MiB/s wr, 189 op/s 2024-09-21T10:33:56.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:55 smithi160 ceph-mon[101879]: osdmap e1116: 8 total, 8 up, 8 in 2024-09-21T10:33:56.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:55 smithi070 ceph-mon[107385]: pgmap v1766: 105 pgs: 105 active+clean; 518 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 115 KiB/s rd, 34 MiB/s wr, 189 op/s 2024-09-21T10:33:56.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:55 smithi070 ceph-mon[107385]: osdmap e1116: 8 total, 8 up, 8 in 2024-09-21T10:33:56.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:55 smithi070 ceph-mon[112139]: pgmap v1766: 105 pgs: 105 active+clean; 518 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 115 KiB/s rd, 34 MiB/s wr, 189 op/s 2024-09-21T10:33:56.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:55 smithi070 ceph-mon[112139]: osdmap e1116: 8 total, 8 up, 8 in 2024-09-21T10:33:56.832 INFO:tasks.workunit.client.0.smithi070.stdout: i=8 exists [1848814~8910670,16642670~3711712,28558403~593635,30271399~11594100,45048342~8534164,57228549~303119,57756520~1469204,71935619~14490316,93118269~4798812,101882815~4086712,106701993~2513206,119268089~4418966,127714892~10134136,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,180722510~2055247,186731431~3510386,191243759~14819924,211221032~4081388,220782438~3218309,224340984~6971051,238706507~3881063,242881143~2665712,254252427~9347451,267194315~8751145,289443310~747069,294637427~5715128,303093367~25576710,349416768~304856,361489784~17718026,388773220~4929446,400476606~3423466,410761440~3528643] wrote [2875569~6387626,16642670~3711712,152039608~7712103,165324500~859426,167641738~2360828,254252427~9347451,373355742~5852068] 2024-09-21T10:33:57.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:56 smithi160 ceph-mon[101879]: osdmap e1117: 8 total, 8 up, 8 in 2024-09-21T10:33:57.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:56 smithi070 ceph-mon[112139]: osdmap e1117: 8 total, 8 up, 8 in 2024-09-21T10:33:57.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:56 smithi070 ceph-mon[107385]: osdmap e1117: 8 total, 8 up, 8 in 2024-09-21T10:33:57.798 INFO:tasks.workunit.client.0.smithi070.stdout: i=9 exists [1848814~8910670,16642670~3711712,28558403~593635,30271399~11594100,45048342~8534164,57228549~303119,57756520~1469204,71935619~14490316,93118269~4798812,101882815~4086712,106701993~2513206,115995156~2434204,119268089~4418966,127714892~10134136,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,186731431~3510386,191243759~14819924,211221032~4081388,220782438~3218309,224340984~6971051,238706507~8545657,254252427~21693033,289443310~747069,294637427~5715128,303093367~25576710,343124038~1645752,349416768~304856,361489784~17718026,381111195~1866659,384603168~9099498,400476606~3423466,410761440~3528643] wrote [115995156~2434204,152797416~596734,175387176~3839850,240673574~6578590,259644006~8285489,317667912~4641221,343124038~1645752,381111195~1866659,384603168~8037415] 2024-09-21T10:33:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:57 smithi160 ceph-mon[101879]: pgmap v1769: 105 pgs: 105 active+clean; 668 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 169 KiB/s rd, 49 MiB/s wr, 279 op/s 2024-09-21T10:33:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:57 smithi160 ceph-mon[101879]: osdmap e1118: 8 total, 8 up, 8 in 2024-09-21T10:33:58.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:57 smithi070 ceph-mon[112139]: pgmap v1769: 105 pgs: 105 active+clean; 668 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 169 KiB/s rd, 49 MiB/s wr, 279 op/s 2024-09-21T10:33:58.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:57 smithi070 ceph-mon[112139]: osdmap e1118: 8 total, 8 up, 8 in 2024-09-21T10:33:58.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:57 smithi070 ceph-mon[107385]: pgmap v1769: 105 pgs: 105 active+clean; 668 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 169 KiB/s rd, 49 MiB/s wr, 279 op/s 2024-09-21T10:33:58.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:57 smithi070 ceph-mon[107385]: osdmap e1118: 8 total, 8 up, 8 in 2024-09-21T10:33:58.807 INFO:tasks.workunit.client.0.smithi070.stdout: i=10 exists [1848814~8910670,16642670~3711712,28558403~593635,30271399~11594100,45048342~8534164,57228549~303119,57756520~1469204,71935619~14490316,93118269~4798812,98906260~7063267,106701993~2513206,114683657~9003398,127714892~10134136,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,186731431~3510386,191243759~14819924,207781564~1395640,211221032~4081388,220782438~3218309,224340984~6971051,238706507~8545657,254252427~26661514,289443310~747069,294637427~5715128,303093367~25576710,339701748~5068042,349416768~304856,361489784~17718026,381111195~1866659,384603168~9099498,400476606~3423466,410761440~3528643] wrote [98906260~4781139,114683657~4879263,207781564~1395640,269577554~11336387,305173155~4590075,311582767~3753918,339701748~4136464,367688090~3556220,376984253~2159000] 2024-09-21T10:33:59.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:58 smithi160 ceph-mon[101879]: osdmap e1119: 8 total, 8 up, 8 in 2024-09-21T10:33:59.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:58 smithi070 ceph-mon[107385]: osdmap e1119: 8 total, 8 up, 8 in 2024-09-21T10:33:59.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:58 smithi070 ceph-mon[112139]: osdmap e1119: 8 total, 8 up, 8 in 2024-09-21T10:33:59.817 INFO:tasks.workunit.client.0.smithi070.stdout: i=11 exists [1848814~8910670,16642670~3711712,28558403~593635,30271399~11594100,45048342~10301049,57228549~303119,57756520~1469204,62918670~4568250,71935619~14490316,93118269~4798812,98906260~7063267,106701993~2513206,114683657~9003398,127714892~10134136,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,186731431~3510386,191243759~14819924,207781564~1395640,211221032~4081388,220782438~3218309,224340984~6971051,236839701~10412463,254252427~26661514,289443310~747069,294637427~34032650,339701748~5068042,349416768~304856,361489784~17718026,381111195~1866659,384603168~9099498,400476606~3423466,410761440~3528643] wrote [31536075~6482204,49169063~6180328,62918670~4568250,82908058~1860108,101539283~3768910,157423190~3438017,236839701~5403448,298351917~8033557,314950007~3894333] 2024-09-21T10:34:00.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:59 smithi160 ceph-mon[101879]: pgmap v1772: 105 pgs: 105 active+clean; 668 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 169 KiB/s rd, 32 MiB/s wr, 270 op/s 2024-09-21T10:34:00.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:33:59 smithi160 ceph-mon[101879]: osdmap e1120: 8 total, 8 up, 8 in 2024-09-21T10:34:00.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:59 smithi070 ceph-mon[112139]: pgmap v1772: 105 pgs: 105 active+clean; 668 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 169 KiB/s rd, 32 MiB/s wr, 270 op/s 2024-09-21T10:34:00.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:33:59 smithi070 ceph-mon[112139]: osdmap e1120: 8 total, 8 up, 8 in 2024-09-21T10:34:00.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:59 smithi070 ceph-mon[107385]: pgmap v1772: 105 pgs: 105 active+clean; 668 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 169 KiB/s rd, 32 MiB/s wr, 270 op/s 2024-09-21T10:34:00.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:33:59 smithi070 ceph-mon[107385]: osdmap e1120: 8 total, 8 up, 8 in 2024-09-21T10:34:00.839 INFO:tasks.workunit.client.0.smithi070.stdout: i=12 exists [1848814~8910670,16642670~3711712,27991252~13874247,45048342~10301049,57228549~303119,57756520~1469204,62918670~4568250,71935619~14490316,93118269~4798812,98906260~7063267,106701993~2513206,114683657~9003398,127714892~10134136,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,167641738~6910718,175387176~3839850,180722510~9519307,191243759~14819924,207781564~1395640,211221032~4081388,220782438~3218309,224340984~8144255,236839701~10412463,254252427~26661514,289443310~747069,294637427~34032650,339701748~5068042,349416768~304856,361489784~17718026,381111195~1866659,384603168~9099498,400476606~3423466,410761440~3528643] wrote [27991252~3630235,72069759~2339334,116074398~2736787,182407418~5708844,191528500~10608093,224814781~7670458,306516710~9213977] 2024-09-21T10:34:01.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:00 smithi160 ceph-mon[101879]: osdmap e1121: 8 total, 8 up, 8 in 2024-09-21T10:34:01.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:00 smithi070 ceph-mon[107385]: osdmap e1121: 8 total, 8 up, 8 in 2024-09-21T10:34:01.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:00 smithi070 ceph-mon[112139]: osdmap e1121: 8 total, 8 up, 8 in 2024-09-21T10:34:01.831 INFO:tasks.workunit.client.0.smithi070.stdout: i=13 exists [1848814~8910670,16642670~3711712,27991252~13874247,45048342~12483326,57756520~1469204,62918670~4568250,71935619~14490316,93118269~4798812,98906260~7063267,106701993~2513206,114683657~9003398,127714892~10134136,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~1395640,211221032~4081388,220782438~3218309,224340984~8144255,236839701~14643462,254252427~26661514,289443310~747069,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,361489784~17718026,381111195~1866659,384603168~9099498,394221093~9678979,410761440~3528643] wrote [53616434~3873502,166204955~7344626,175217960~2272728,244912682~6570481,301105270~570107,315123129~3037979,318527970~2909978,343689268~606553,347289030~490774,394221093~8158476] 2024-09-21T10:34:02.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:01 smithi160 ceph-mon[101879]: pgmap v1775: 105 pgs: 105 active+clean; 765 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 22 MiB/s wr, 12 op/s 2024-09-21T10:34:02.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:01 smithi160 ceph-mon[101879]: osdmap e1122: 8 total, 8 up, 8 in 2024-09-21T10:34:02.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:01 smithi070 ceph-mon[112139]: pgmap v1775: 105 pgs: 105 active+clean; 765 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 22 MiB/s wr, 12 op/s 2024-09-21T10:34:02.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:01 smithi070 ceph-mon[112139]: osdmap e1122: 8 total, 8 up, 8 in 2024-09-21T10:34:02.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:01 smithi070 ceph-mon[107385]: pgmap v1775: 105 pgs: 105 active+clean; 765 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 22 MiB/s wr, 12 op/s 2024-09-21T10:34:02.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:01 smithi070 ceph-mon[107385]: osdmap e1122: 8 total, 8 up, 8 in 2024-09-21T10:34:02.810 INFO:tasks.workunit.client.0.smithi070.stdout: i=14 exists [1848814~8910670,16642670~4162967,27991252~13874247,45048342~12483326,57756520~1469204,62918670~4568250,71935619~14490316,93118269~4798812,98906260~7063267,106701993~2513206,114683657~9118904,127714892~10134136,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~1395640,211221032~4081388,220782438~3218309,224340984~8144255,236839701~14643462,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,360853018~18354792,381111195~1866659,384603168~9099498,394221093~9678979,410761440~3528643] wrote [20106162~699475,32450955~218199,116263063~292497,119655166~4147395,135872708~1754391,181624406~3276255,241096252~6979125,286147115~4491537,306016051~3154216,360853018~1275580] 2024-09-21T10:34:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:02 smithi160 ceph-mon[101879]: osdmap e1123: 8 total, 8 up, 8 in 2024-09-21T10:34:03.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:02 smithi070 ceph-mon[107385]: osdmap e1123: 8 total, 8 up, 8 in 2024-09-21T10:34:03.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:02 smithi070 ceph-mon[112139]: osdmap e1123: 8 total, 8 up, 8 in 2024-09-21T10:34:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:34:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:34:03] "GET /metrics HTTP/1.1" 200 36480 "" "Prometheus/2.51.0" 2024-09-21T10:34:03.836 INFO:tasks.workunit.client.0.smithi070.stdout: i=15 exists [1848814~8910670,16642670~4162967,27991252~13874247,45048342~12483326,57756520~2066114,62918670~4568250,71935619~14490316,93118269~4798812,98906260~7063267,106701993~2513206,114683657~9118904,127714892~10134136,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~1395640,210226943~5075477,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~4338875,360853018~18354792,381111195~1866659,384505041~9197625,394221093~9678979,410761440~3528643] wrote [58639820~1182814,144777497~6623225,210226943~2559490,230058264~11640976,301283769~5609120,351227316~4338875,363514600~4814024,384505041~360305,397020628~4740508] 2024-09-21T10:34:04.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:03 smithi160 ceph-mon[101879]: pgmap v1778: 105 pgs: 105 active+clean; 865 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 204 KiB/s rd, 45 MiB/s wr, 327 op/s 2024-09-21T10:34:04.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:03 smithi160 ceph-mon[101879]: osdmap e1124: 8 total, 8 up, 8 in 2024-09-21T10:34:04.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:03 smithi070 ceph-mon[107385]: pgmap v1778: 105 pgs: 105 active+clean; 865 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 204 KiB/s rd, 45 MiB/s wr, 327 op/s 2024-09-21T10:34:04.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:03 smithi070 ceph-mon[107385]: osdmap e1124: 8 total, 8 up, 8 in 2024-09-21T10:34:04.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:03 smithi070 ceph-mon[112139]: pgmap v1778: 105 pgs: 105 active+clean; 865 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 204 KiB/s rd, 45 MiB/s wr, 327 op/s 2024-09-21T10:34:04.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:03 smithi070 ceph-mon[112139]: osdmap e1124: 8 total, 8 up, 8 in 2024-09-21T10:34:04.854 INFO:tasks.workunit.client.0.smithi070.stdout: i=16 exists [1848814~8910670,16642670~4956869,25706100~16159399,45048342~14774292,62918670~4568250,71935619~14490316,93118269~4798812,98522774~7446753,106701993~2513206,114683657~9118904,127714892~10134136,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~1395640,210226943~5075477,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~4338875,360853018~18354792,380931932~31586,381111195~1866659,384505041~9197625,394221093~9678979,409717800~4572283] wrote [16883580~4715959,25706100~3951165,53413917~5605327,98522774~6314663,225041157~7372785,380931932~31586,395383365~5743695,409717800~2000564] 2024-09-21T10:34:05.037 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:04 smithi070 ceph-mon[107385]: osdmap e1125: 8 total, 8 up, 8 in 2024-09-21T10:34:05.037 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:04 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:34:05.038 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:04 smithi070 ceph-mon[112139]: osdmap e1125: 8 total, 8 up, 8 in 2024-09-21T10:34:05.038 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:04 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:34:05.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:04 smithi160 ceph-mon[101879]: osdmap e1125: 8 total, 8 up, 8 in 2024-09-21T10:34:05.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:04 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:34:05.846 INFO:tasks.workunit.client.0.smithi070.stdout: i=17 exists [1848814~8910670,15258652~7024612,25706100~16159399,45048342~14774292,62918670~23507265,93118269~16096930,114683657~9118904,127714892~10134136,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~1395640,209930960~5371460,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~4338875,360853018~18354792,380931932~31586,381111195~1866659,384505041~9197625,394221093~9678979,409717800~4572283] wrote [15258652~7024612,66238036~7394424,95398996~13719063,115435950~5729260,168967775~4510479,209930960~3307546,269871960~1086953,364615164~692640] 2024-09-21T10:34:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:05 smithi160 ceph-mon[101879]: pgmap v1781: 105 pgs: 105 active+clean; 933 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 203 KiB/s rd, 41 MiB/s wr, 324 op/s 2024-09-21T10:34:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:05 smithi160 ceph-mon[101879]: osdmap e1126: 8 total, 8 up, 8 in 2024-09-21T10:34:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:06.103 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:05 smithi070 ceph-mon[107385]: pgmap v1781: 105 pgs: 105 active+clean; 933 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 203 KiB/s rd, 41 MiB/s wr, 324 op/s 2024-09-21T10:34:06.103 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:05 smithi070 ceph-mon[107385]: osdmap e1126: 8 total, 8 up, 8 in 2024-09-21T10:34:06.103 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:06.103 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:05 smithi070 ceph-mon[112139]: pgmap v1781: 105 pgs: 105 active+clean; 933 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 203 KiB/s rd, 41 MiB/s wr, 324 op/s 2024-09-21T10:34:06.103 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:05 smithi070 ceph-mon[112139]: osdmap e1126: 8 total, 8 up, 8 in 2024-09-21T10:34:06.103 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:06.851 INFO:tasks.workunit.client.0.smithi070.stdout: i=18 exists [1848814~8910670,15258652~7024612,25706100~16159399,45048342~14774292,62918670~23507265,93118269~16096930,114683657~9118904,124591324~13257704,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~9355151,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,291766799~42181,292083899~1585781,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] wrote [96121327~6900963,124591324~5118882,208968224~8168491,265149402~4866558,291766799~42181,292083899~1585781,310892222~4271109,353876173~3908848,389651616~4254468,399140449~1040101] 2024-09-21T10:34:07.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:06 smithi160 ceph-mon[101879]: osdmap e1127: 8 total, 8 up, 8 in 2024-09-21T10:34:07.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:06 smithi070 ceph-mon[107385]: osdmap e1127: 8 total, 8 up, 8 in 2024-09-21T10:34:07.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:06 smithi070 ceph-mon[112139]: osdmap e1127: 8 total, 8 up, 8 in 2024-09-21T10:34:07.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:07 smithi070 ceph-mon[107385]: pgmap v1784: 105 pgs: 105 active+clean; 1.0 GiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 236 KiB/s rd, 52 MiB/s wr, 373 op/s 2024-09-21T10:34:07.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:07 smithi070 ceph-mon[107385]: osdmap e1128: 8 total, 8 up, 8 in 2024-09-21T10:34:07.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:07 smithi070 ceph-mon[112139]: pgmap v1784: 105 pgs: 105 active+clean; 1.0 GiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 236 KiB/s rd, 52 MiB/s wr, 373 op/s 2024-09-21T10:34:07.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:07 smithi070 ceph-mon[112139]: osdmap e1128: 8 total, 8 up, 8 in 2024-09-21T10:34:07.888 INFO:tasks.workunit.client.0.smithi070.stdout: i=19 exists [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~24371839,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] wrote [19737427~3953085,47127613~6678875,82127504~5163005,118871052~5826351,146673284~5474980,216496383~1716249,282279101~6816084,333198080~5481608] 2024-09-21T10:34:08.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:07 smithi160 ceph-mon[101879]: pgmap v1784: 105 pgs: 105 active+clean; 1.0 GiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 236 KiB/s rd, 52 MiB/s wr, 373 op/s 2024-09-21T10:34:08.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:07 smithi160 ceph-mon[101879]: osdmap e1128: 8 total, 8 up, 8 in 2024-09-21T10:34:08.813 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 1 diff [72446799~256446,139085476~6479527,168542535~3547892,191243759~5891731,308097160~5425190,323189366~3910707,367586957~6292557,376447308~19278,388773220~4929446,410761440~3528643] 2024-09-21T10:34:08.813 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [72446799~256446,139085476~6479527,168542535~3547892,191243759~5891731,308097160~5425190,323189366~3910707,367586957~6292557,376447308~19278,388773220~4929446,410761440~3528643] 2024-09-21T10:34:08.831 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [71303168~1400077,138412032~7152971,167772160~4318267,188743680~8391810,308097160~5425190,322961408~4138665,367586957~8879629,385875968~7826698,406847488~7442595] 2024-09-21T10:34:08.831 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 2 diff [1848814~6541440,30271399~11594100,47017959~6564547,72446799~256446,132602666~5246362,139085476~6479527,158059215~1020680,168542535~3547892,191243759~5891731,242881143~2665712,259674737~2646128,308097160~5425190,317001032~5954332,323189366~3910707,367586957~6292557,376447308~19278,388773220~4929446,410761440~3528643] 2024-09-21T10:34:08.831 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~6541440,30271399~11594100,47017959~6564547,72446799~256446,132602666~5246362,139085476~6479527,158059215~1020680,168542535~3547892,191243759~5891731,242881143~2665712,259674737~2646128,308097160~5425190,317001032~5954332,323189366~3910707,367586957~6292557,376447308~19278,388773220~4929446,410761440~3528643] 2024-09-21T10:34:08.837 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~8390254,29360128~12505371,46137344~7445162,71303168~1400077,130023424~7825604,138412032~7152971,155189248~3890647,167772160~4318267,188743680~8391810,239075328~6471527,255852544~6468321,308097160~5425190,317001032~5954332,322961408~4138665,367586957~8879629,385875968~7826698,406847488~7442595] 2024-09-21T10:34:08.837 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 3 diff [1848814~6541440,30271399~11594100,47017959~6564547,72446799~256446,75540009~4520923,132602666~5246362,139085476~6479527,158059215~1020680,168542535~3547892,191243759~5891731,198347760~7715923,211221032~4081388,220782438~3218309,242881143~2665712,258320585~4000280,267194315~7846772,308097160~5425190,314468828~8486536,323189366~3910707,362533785~11345729,376447308~19278,388773220~4929446,410761440~3528643] 2024-09-21T10:34:08.837 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~6541440,30271399~11594100,47017959~6564547,72446799~256446,75540009~4520923,132602666~5246362,139085476~6479527,158059215~1020680,168542535~3547892,191243759~5891731,198347760~7715923,211221032~4081388,220782438~3218309,242881143~2665712,258320585~4000280,267194315~7846772,308097160~5425190,314468828~8486536,323189366~3910707,362533785~11345729,376447308~19278,388773220~4929446,410761440~3528643] 2024-09-21T10:34:08.843 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~8390254,29360128~12505371,46137344~7445162,71303168~1400077,75540009~4520923,130023424~7825604,138412032~7152971,155189248~3890647,167772160~4318267,188743680~17320003,209715200~5587220,218103808~5896939,239075328~6471527,255852544~6468321,264241152~10799935,308097160~5425190,314468828~8486536,322961408~4138665,362533785~13932801,385875968~7826698,406847488~7442595] 2024-09-21T10:34:08.843 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 4 diff [1848814~8910670,30271399~11594100,47017959~6564547,71935619~8125313,106701993~2513206,132602666~5246362,139085476~6479527,152870944~280627,158059215~1020680,168542535~3547892,181304442~350166,186731431~3510386,191243759~5891731,198347760~7715923,211221032~4081388,220782438~3218309,242881143~2665712,255735910~6584955,267194315~8751145,308097160~5425190,314468828~8486536,323189366~3910707,349416768~304856,362533785~11345729,376447308~19278,388773220~4929446,410761440~3528643] 2024-09-21T10:34:08.843 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,30271399~11594100,47017959~6564547,71935619~8125313,106701993~2513206,132602666~5246362,139085476~6479527,152870944~280627,158059215~1020680,168542535~3547892,181304442~350166,186731431~3510386,191243759~5891731,198347760~7715923,211221032~4081388,220782438~3218309,242881143~2665712,255735910~6584955,267194315~8751145,308097160~5425190,314468828~8486536,323189366~3910707,349416768~304856,362533785~11345729,376447308~19278,388773220~4929446,410761440~3528643] 2024-09-21T10:34:08.848 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,29360128~12505371,46137344~7445162,71303168~8757764,104857600~4357599,130023424~7825604,138412032~7152971,152649398~502173,155189248~3890647,167772160~4318267,180355072~1299536,184549376~21514307,209715200~5587220,218103808~5896939,239075328~6471527,251658240~10662625,264241152~11704308,308097160~5425190,314468828~8486536,322961408~4138665,348127232~1594392,362533785~13932801,385875968~7826698,406847488~7442595] 2024-09-21T10:34:08.849 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 5 diff [1848814~8910670,30271399~11594100,45048342~8534164,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,127714892~10134136,138318126~7246877,152870944~280627,158059215~1020680,168542535~3547892,180722510~2055247,186731431~3510386,191243759~14819924,211221032~4081388,220782438~3218309,238706507~3881063,242881143~2665712,255735910~6584955,267194315~8751145,289443310~747069,308097160~5425190,314468828~8486536,323189366~3910707,349416768~304856,362533785~11345729,376447308~19278,388773220~4929446,410761440~3528643] 2024-09-21T10:34:08.849 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,30271399~11594100,45048342~8534164,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,127714892~10134136,138318126~7246877,152870944~280627,158059215~1020680,168542535~3547892,180722510~2055247,186731431~3510386,191243759~14819924,211221032~4081388,220782438~3218309,238706507~3881063,242881143~2665712,255735910~6584955,267194315~8751145,289443310~747069,308097160~5425190,314468828~8486536,323189366~3910707,349416768~304856,362533785~11345729,376447308~19278,388773220~4929446,410761440~3528643] 2024-09-21T10:34:08.854 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,29360128~12505371,41943040~11639466,71303168~8757764,92274688~5642393,100663296~8551903,125829120~19735883,152649398~502173,155189248~3890647,167772160~4318267,180355072~2422685,184549376~21514307,209715200~5587220,218103808~5896939,234881024~10665831,251658240~10662625,264241152~11704308,289406976~783403,308097160~5425190,314468828~8486536,322961408~4138665,348127232~1594392,362533785~13932801,385875968~7826698,406847488~7442595] 2024-09-21T10:34:08.854 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 6 diff [1848814~8910670,28558403~593635,30271399~11594100,45048342~8534164,57756520~1469204,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,127714892~10134136,138318126~7246877,146239075~3597571,152870944~280627,158059215~1020680,167991057~5531964,180722510~2055247,186731431~3510386,191243759~14819924,211221032~4081388,220782438~3218309,238706507~3881063,242881143~2665712,255735910~6584955,267194315~8751145,289443310~747069,294637427~5715128,303093367~4354075,308097160~5425190,314468828~8486536,323189366~3910707,349416768~304856,361489784~12389730,376447308~19278,388773220~4929446,410761440~3528643] 2024-09-21T10:34:08.854 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,28558403~593635,30271399~11594100,45048342~8534164,57756520~1469204,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,127714892~10134136,138318126~7246877,146239075~3597571,152870944~280627,158059215~1020680,167991057~5531964,180722510~2055247,186731431~3510386,191243759~14819924,211221032~4081388,220782438~3218309,238706507~3881063,242881143~2665712,255735910~6584955,267194315~8751145,289443310~747069,294637427~5715128,303093367~4354075,308097160~5425190,314468828~8486536,323189366~3910707,349416768~304856,361489784~12389730,376447308~19278,388773220~4929446,410761440~3528643] 2024-09-21T10:34:08.860 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,25165824~3986214,29360128~12505371,41943040~11639466,57531668~1694056,71303168~8757764,92274688~5642393,100663296~8551903,125829120~24007526,152649398~502173,155189248~3890647,167772160~5750861,180355072~2422685,184549376~21514307,209715200~5587220,218103808~5896939,234881024~10665831,251658240~10662625,264241152~11704308,289406976~783403,293601280~6751275,301989888~5457554,308097160~5425190,314468828~8486536,322961408~4138665,348127232~1594392,361489784~14976802,385875968~7826698,406847488~7442595] 2024-09-21T10:34:08.860 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 7 diff [1848814~8910670,28558403~593635,30271399~11594100,45048342~8534164,57756520~1469204,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,127714892~10134136,138318126~7793832,146239075~3597571,152870944~280627,158059215~1020680,167991057~6561399,180722510~2055247,186731431~3510386,191243759~14819924,211221032~4081388,220782438~3218309,224340984~6971051,238706507~3881063,242881143~2665712,255735910~6584955,267194315~8751145,289443310~747069,294637427~5715128,303093367~4354075,308097160~5425190,314468828~14201249,349416768~304856,361489784~15792220,388773220~4929446,400476606~3423466,410761440~3528643] 2024-09-21T10:34:08.860 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,28558403~593635,30271399~11594100,45048342~8534164,57756520~1469204,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,127714892~10134136,138318126~7793832,146239075~3597571,152870944~280627,158059215~1020680,167991057~6561399,180722510~2055247,186731431~3510386,191243759~14819924,211221032~4081388,220782438~3218309,224340984~6971051,238706507~3881063,242881143~2665712,255735910~6584955,267194315~8751145,289443310~747069,294637427~5715128,303093367~4354075,308097160~5425190,314468828~14201249,349416768~304856,361489784~15792220,388773220~4929446,400476606~3423466,410761440~3528643] 2024-09-21T10:34:08.866 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,25165824~3986214,29360128~12505371,41943040~11639466,57531668~1694056,71303168~8757764,92274688~5642393,100663296~8551903,125829120~24007526,152649398~502173,155189248~3890647,167772160~6780296,180355072~2422685,184549376~21514307,209715200~5587220,218103808~13208227,234881024~10665831,251658240~10662625,264241152~11704308,289406976~783403,293601280~6751275,301989888~5457554,308097160~5425190,314468828~14201249,348127232~1594392,361489784~15792220,385875968~7826698,398458880~5441192,406847488~7442595] 2024-09-21T10:34:08.866 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 8 diff [1848814~8910670,16642670~3711712,28558403~593635,30271399~11594100,45048342~8534164,57756520~1469204,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,127714892~10134136,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,180722510~2055247,186731431~3510386,191243759~14819924,211221032~4081388,220782438~3218309,224340984~6971051,238706507~3881063,242881143~2665712,254252427~9347451,267194315~8751145,289443310~747069,294637427~5715128,303093367~4354075,308097160~5425190,314468828~14201249,349416768~304856,361489784~17718026,388773220~4929446,400476606~3423466,410761440~3528643] 2024-09-21T10:34:08.866 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,16642670~3711712,28558403~593635,30271399~11594100,45048342~8534164,57756520~1469204,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,127714892~10134136,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,180722510~2055247,186731431~3510386,191243759~14819924,211221032~4081388,220782438~3218309,224340984~6971051,238706507~3881063,242881143~2665712,254252427~9347451,267194315~8751145,289443310~747069,294637427~5715128,303093367~4354075,308097160~5425190,314468828~14201249,349416768~304856,361489784~17718026,388773220~4929446,400476606~3423466,410761440~3528643] 2024-09-21T10:34:08.872 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~7771470,25165824~3986214,29360128~12505371,41943040~11639466,57531668~1694056,71303168~8757764,92274688~5642393,100663296~8551903,125829120~24007526,152039608~7712103,163577856~10974600,180355072~2422685,184549376~21514307,209715200~5587220,218103808~13208227,234881024~10665831,251658240~11941638,264241152~11704308,289406976~783403,293601280~6751275,301989888~5457554,308097160~5425190,314468828~14201249,348127232~1594392,361489784~17718026,385875968~7826698,398458880~5441192,406847488~7442595] 2024-09-21T10:34:08.872 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 9 diff [1848814~8910670,16642670~3711712,28558403~593635,30271399~11594100,45048342~8534164,57756520~1469204,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,115995156~2434204,127714892~10134136,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,186731431~3510386,191243759~14819924,211221032~4081388,220782438~3218309,224340984~6971051,238706507~8545657,254252427~21693033,289443310~747069,294637427~5715128,303093367~4354075,308097160~5425190,314468828~14201249,343124038~1645752,349416768~304856,361489784~17718026,381111195~1866659,384603168~9099498,400476606~3423466,410761440~3528643] 2024-09-21T10:34:08.872 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,16642670~3711712,28558403~593635,30271399~11594100,45048342~8534164,57756520~1469204,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,115995156~2434204,127714892~10134136,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,186731431~3510386,191243759~14819924,211221032~4081388,220782438~3218309,224340984~6971051,238706507~8545657,254252427~21693033,289443310~747069,294637427~5715128,303093367~4354075,308097160~5425190,314468828~14201249,343124038~1645752,349416768~304856,361489784~17718026,381111195~1866659,384603168~9099498,400476606~3423466,410761440~3528643] 2024-09-21T10:34:08.878 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~7771470,25165824~3986214,29360128~12505371,41943040~11639466,57531668~1694056,71303168~8757764,92274688~5642393,100663296~8551903,113246208~5183152,125829120~24007526,152039608~7712103,163577856~15649170,180355072~2422685,184549376~21514307,209715200~5587220,218103808~13208227,234881024~12371140,251658240~24287220,289406976~783403,293601280~6751275,301989888~5457554,308097160~5425190,314468828~14201249,339738624~5031166,348127232~1594392,361489784~32212882,398458880~5441192,406847488~7442595] 2024-09-21T10:34:08.878 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 10 diff [1848814~8910670,16642670~3711712,28558403~593635,30271399~11594100,45048342~8534164,57756520~1469204,71935619~8125313,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~10134136,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,186731431~3510386,191243759~14819924,207781564~1395640,211221032~4081388,220782438~3218309,224340984~6971051,238706507~8545657,254252427~26661514,289443310~747069,294637427~5715128,303093367~25576710,339701748~5068042,349416768~304856,361489784~17718026,381111195~1866659,384603168~9099498,400476606~3423466,410761440~3528643] 2024-09-21T10:34:08.878 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,16642670~3711712,28558403~593635,30271399~11594100,45048342~8534164,57756520~1469204,71935619~8125313,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~10134136,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,186731431~3510386,191243759~14819924,207781564~1395640,211221032~4081388,220782438~3218309,224340984~6971051,238706507~8545657,254252427~26661514,289443310~747069,294637427~5715128,303093367~25576710,339701748~5068042,349416768~304856,361489784~17718026,381111195~1866659,384603168~9099498,400476606~3423466,410761440~3528643] 2024-09-21T10:34:08.883 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~7771470,25165824~3986214,29360128~12505371,41943040~11639466,57531668~1694056,71303168~8757764,92274688~16940511,113246208~6316712,125829120~24007526,152039608~7712103,163577856~15649170,180355072~2422685,184549376~24627828,209715200~5587220,218103808~13208227,234881024~12371140,251658240~29255701,289406976~783403,293601280~6751275,301989888~26680189,335544320~9225470,348127232~1594392,361489784~32212882,398458880~5441192,406847488~7442595] 2024-09-21T10:34:08.883 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 11 diff [1848814~8910670,16642670~3711712,28558403~593635,30271399~11594100,45048342~10301049,57756520~1469204,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~10134136,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,186731431~3510386,191243759~14819924,207781564~1395640,211221032~4081388,220782438~3218309,224340984~6971051,236839701~10412463,254252427~26661514,289443310~747069,294637427~34032650,339701748~5068042,349416768~304856,361489784~17718026,381111195~1866659,384603168~9099498,400476606~3423466,410761440~3528643] 2024-09-21T10:34:08.884 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,16642670~3711712,28558403~593635,30271399~11594100,45048342~10301049,57756520~1469204,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~10134136,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,186731431~3510386,191243759~14819924,207781564~1395640,211221032~4081388,220782438~3218309,224340984~6971051,236839701~10412463,254252427~26661514,289443310~747069,294637427~34032650,339701748~5068042,349416768~304856,361489784~17718026,381111195~1866659,384603168~9099498,400476606~3423466,410761440~3528643] 2024-09-21T10:34:08.889 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~7771470,25165824~3986214,29360128~12505371,41943040~13406351,57531668~1694056,62914560~4572360,71303168~8757764,82908058~1860108,92274688~16940511,113246208~6316712,125829120~24007526,152039608~8821599,163577856~15649170,180355072~2422685,184549376~24627828,209715200~5587220,218103808~13208227,234881024~12371140,251658240~29255701,289406976~783403,293601280~35068797,335544320~9225470,348127232~1594392,361489784~32212882,398458880~5441192,406847488~7442595] 2024-09-21T10:34:08.889 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 12 diff [1848814~8910670,16642670~3711712,27991252~13874247,45048342~10301049,57756520~1469204,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~10134136,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,167641738~6910718,175387176~3839850,180722510~9519307,191243759~14819924,207781564~1395640,211221032~4081388,220782438~3218309,224340984~8144255,236839701~10412463,254252427~26661514,289443310~747069,294637427~34032650,339701748~5068042,349416768~304856,361489784~17718026,381111195~1866659,384603168~9099498,400476606~3423466,410761440~3528643] 2024-09-21T10:34:08.889 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,16642670~3711712,27991252~13874247,45048342~10301049,57756520~1469204,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~10134136,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,167641738~6910718,175387176~3839850,180722510~9519307,191243759~14819924,207781564~1395640,211221032~4081388,220782438~3218309,224340984~8144255,236839701~10412463,254252427~26661514,289443310~747069,294637427~34032650,339701748~5068042,349416768~304856,361489784~17718026,381111195~1866659,384603168~9099498,400476606~3423466,410761440~3528643] 2024-09-21T10:34:08.898 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~7771470,25165824~16699675,41943040~13406351,57531668~1694056,62914560~4572360,71303168~8757764,82908058~1860108,92274688~16940511,113246208~6316712,125829120~24007526,152039608~8821599,163577856~15649170,180355072~28822132,209715200~5587220,218103808~14381431,234881024~12371140,251658240~29255701,289406976~783403,293601280~35068797,335544320~9225470,348127232~1594392,361489784~32212882,398458880~5441192,406847488~7442595] 2024-09-21T10:34:08.898 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 13 diff [1848814~8910670,16642670~3711712,27991252~13874247,45048342~12441594,57756520~1469204,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~10134136,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~1395640,211221032~4081388,220782438~3218309,224340984~8144255,236839701~14643462,254252427~26661514,289443310~747069,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,361489784~17718026,381111195~1866659,384603168~9099498,394221093~9678979,410761440~3528643] 2024-09-21T10:34:08.898 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,16642670~3711712,27991252~13874247,45048342~12441594,57756520~1469204,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~10134136,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~1395640,211221032~4081388,220782438~3218309,224340984~8144255,236839701~14643462,254252427~26661514,289443310~747069,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,361489784~17718026,381111195~1866659,384603168~9099498,394221093~9678979,410761440~3528643] 2024-09-21T10:34:08.901 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~7771470,25165824~16699675,41943040~15546896,57531668~1694056,62914560~4572360,71303168~8757764,82908058~1860108,92274688~16940511,113246208~6316712,125829120~24007526,152039608~8821599,163577856~15649170,180355072~28822132,209715200~5587220,218103808~14381431,234881024~16602139,251658240~29255701,289406976~783403,293601280~35068797,335544320~12235484,348127232~1594392,361489784~42410288,406847488~7442595] 2024-09-21T10:34:08.901 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 14 diff [1848814~8910670,16642670~4162967,27991252~13874247,45048342~12441594,57756520~1469204,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,119655166~4147395,127714892~10134136,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~1395640,211221032~4081388,220782438~3218309,224340984~8144255,236839701~14643462,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,360853018~18354792,381111195~1866659,384603168~9099498,394221093~9678979,410761440~3528643] 2024-09-21T10:34:08.901 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,16642670~4162967,27991252~13874247,45048342~12441594,57756520~1469204,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,119655166~4147395,127714892~10134136,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~1395640,211221032~4081388,220782438~3218309,224340984~8144255,236839701~14643462,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,360853018~18354792,381111195~1866659,384603168~9099498,394221093~9678979,410761440~3528643] 2024-09-21T10:34:08.907 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~8222725,25165824~16699675,41943040~15546896,57531668~1694056,62914560~4572360,71303168~8757764,82908058~1860108,92274688~16940511,113246208~6316712,119655166~4147395,125829120~24007526,152039608~8821599,163577856~15649170,180355072~28822132,209715200~5587220,218103808~14381431,234881024~16602139,251658240~29255701,285212672~5425980,293601280~35068797,335544320~12235484,348127232~1594392,360853018~43047054,406847488~7442595] 2024-09-21T10:34:08.907 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 15 diff [1848814~8910670,16642670~4162967,27991252~13874247,45048342~12441594,57756520~2066114,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,119655166~4147395,127714892~10134136,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~1395640,210226943~5075477,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~4338875,360853018~18354792,381111195~1866659,384505041~9197625,394221093~9678979,410761440~3528643] 2024-09-21T10:34:08.907 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,16642670~4162967,27991252~13874247,45048342~12441594,57756520~2066114,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,119655166~4147395,127714892~10134136,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~1395640,210226943~5075477,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~4338875,360853018~18354792,381111195~1866659,384505041~9197625,394221093~9678979,410761440~3528643] 2024-09-21T10:34:08.912 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~8222725,25165824~16699675,41943040~15546896,57531668~2290966,62914560~4572360,71303168~8757764,82908058~1860108,92274688~16940511,113246208~6316712,119655166~4147395,125829120~25571602,152039608~8821599,163577856~15649170,180355072~28822132,209715200~5587220,218103808~33379355,251658240~29255701,285212672~5425980,293601280~35068797,335544320~12235484,348127232~7438959,360853018~43047054,406847488~7442595] 2024-09-21T10:34:08.913 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 16 diff [1848814~8910670,16642670~4956869,25706100~16159399,45048342~14774292,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98522774~7446753,106701993~2513206,114683657~4879263,119655166~4147395,127714892~10134136,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~1395640,210226943~5075477,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~4338875,360853018~18354792,380931932~31586,381111195~1866659,384505041~9197625,394221093~9678979,409717800~4572283] 2024-09-21T10:34:08.913 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,16642670~4956869,25706100~16159399,45048342~14774292,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98522774~7446753,106701993~2513206,114683657~4879263,119655166~4147395,127714892~10134136,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~1395640,210226943~5075477,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~4338875,360853018~18354792,380931932~31586,381111195~1866659,384505041~9197625,394221093~9678979,409717800~4572283] 2024-09-21T10:34:08.918 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~9016627,25165824~16699675,41943040~17879594,62914560~4572360,71303168~8757764,82908058~1860108,92274688~16940511,113246208~6316712,119655166~4147395,125829120~25571602,152039608~8821599,163577856~15649170,180355072~28822132,209715200~5587220,218103808~33379355,251658240~29255701,285212672~5425980,293601280~35068797,335544320~12235484,348127232~7438959,360853018~43047054,406847488~7442595] 2024-09-21T10:34:08.918 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 17 diff [1848814~8910670,15258652~7024612,25706100~16159399,45048342~14774292,62918670~17142262,82908058~1860108,93118269~16096930,114683657~9118904,127714892~10134136,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~1395640,209930960~5371460,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~4338875,360853018~18354792,380931932~31586,381111195~1866659,384505041~9197625,394221093~9678979,409717800~4572283] 2024-09-21T10:34:08.919 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~7024612,25706100~16159399,45048342~14774292,62918670~17142262,82908058~1860108,93118269~16096930,114683657~9118904,127714892~10134136,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~1395640,209930960~5371460,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~4338875,360853018~18354792,380931932~31586,381111195~1866659,384505041~9197625,394221093~9678979,409717800~4572283] 2024-09-21T10:34:08.924 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~9700352,25165824~16699675,41943040~17879594,62914560~17146372,82908058~1860108,92274688~16940511,113246208~10556353,125829120~25571602,152039608~8821599,163577856~15649170,180355072~28822132,209715200~5587220,218103808~33379355,251658240~29255701,285212672~5425980,293601280~35068797,335544320~12235484,348127232~7438959,360853018~43047054,406847488~7442595] 2024-09-21T10:34:08.925 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 18 diff [1848814~8910670,15258652~7024612,25706100~16159399,45048342~14774292,62918670~17142262,82908058~1860108,93118269~16096930,114683657~9118904,124591324~13257704,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~9355151,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,291766799~42181,292083899~1585781,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:08.925 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~7024612,25706100~16159399,45048342~14774292,62918670~17142262,82908058~1860108,93118269~16096930,114683657~9118904,124591324~13257704,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~9355151,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,291766799~42181,292083899~1585781,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:08.930 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~9700352,25165824~16699675,41943040~17879594,62914560~17146372,82908058~1860108,92274688~16940511,113246208~38154514,152039608~8821599,163577856~15649170,180355072~36781643,218103808~33379355,251658240~29255701,285212672~43457405,335544320~12235484,348127232~9657789,360853018~43047054,406847488~7442595] 2024-09-21T10:34:08.930 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 19 diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:08.931 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:08.936 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~71128091,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,406847488~7442595] 2024-09-21T10:34:08.936 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 2 diff [1848814~6541440,30271399~11594100,47017959~6564547,132602666~5246362,158059215~1020680,169402859~1955236,242881143~2665712,259674737~2646128,317001032~5954332] 2024-09-21T10:34:08.936 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~6541440,30271399~11594100,47017959~6564547,132602666~5246362,158059215~1020680,169402859~1955236,242881143~2665712,259674737~2646128,317001032~5954332] 2024-09-21T10:34:08.943 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~8390254,29360128~12505371,46137344~7445162,130023424~7825604,155189248~3890647,169402859~1955236,239075328~6471527,255852544~6468321,317001032~5954332] 2024-09-21T10:34:08.943 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 3 diff [1848814~6541440,30271399~11594100,47017959~6564547,75540009~4520923,132602666~5246362,158059215~1020680,169402859~1955236,198347760~7715923,211221032~4081388,220782438~3218309,242881143~2665712,258320585~4000280,267194315~7846772,314468828~8486536,362533785~5973519] 2024-09-21T10:34:08.943 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~6541440,30271399~11594100,47017959~6564547,75540009~4520923,132602666~5246362,158059215~1020680,169402859~1955236,198347760~7715923,211221032~4081388,220782438~3218309,242881143~2665712,258320585~4000280,267194315~7846772,314468828~8486536,362533785~5973519] 2024-09-21T10:34:08.949 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~8390254,29360128~12505371,46137344~7445162,75540009~4520923,130023424~7825604,155189248~3890647,169402859~1955236,197135490~8928193,209715200~5587220,218103808~5896939,239075328~6471527,255852544~6468321,264241152~10799935,314468828~8486536,362533785~5973519] 2024-09-21T10:34:08.949 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 4 diff [1848814~8910670,30271399~11594100,47017959~6564547,71935619~8125313,106701993~2513206,132602666~5246362,152870944~280627,158059215~1020680,169402859~1955236,181304442~350166,186731431~3510386,198347760~7715923,211221032~4081388,220782438~3218309,242881143~2665712,255735910~6584955,267194315~8751145,314468828~8486536,349416768~304856,362533785~5973519] 2024-09-21T10:34:08.949 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,30271399~11594100,47017959~6564547,71935619~8125313,106701993~2513206,132602666~5246362,152870944~280627,158059215~1020680,169402859~1955236,181304442~350166,186731431~3510386,198347760~7715923,211221032~4081388,220782438~3218309,242881143~2665712,255735910~6584955,267194315~8751145,314468828~8486536,349416768~304856,362533785~5973519] 2024-09-21T10:34:08.955 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,29360128~12505371,46137344~7445162,71935619~8125313,104857600~4357599,130023424~7825604,152649398~502173,155189248~3890647,169402859~1955236,180355072~1299536,184549376~5692441,197135490~8928193,209715200~5587220,218103808~5896939,239075328~6471527,251658240~10662625,264241152~11704308,314468828~8486536,348127232~1594392,362533785~5973519] 2024-09-21T10:34:08.955 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 5 diff [1848814~8910670,30271399~11594100,45048342~8534164,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,127714892~10134136,138318126~6701638,152870944~280627,158059215~1020680,169402859~1955236,180722510~2055247,186731431~3510386,195156550~10907133,211221032~4081388,220782438~3218309,238706507~3881063,242881143~2665712,255735910~6584955,267194315~8751145,289443310~747069,314468828~8486536,349416768~304856,362533785~5973519] 2024-09-21T10:34:08.955 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,30271399~11594100,45048342~8534164,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,127714892~10134136,138318126~6701638,152870944~280627,158059215~1020680,169402859~1955236,180722510~2055247,186731431~3510386,195156550~10907133,211221032~4081388,220782438~3218309,238706507~3881063,242881143~2665712,255735910~6584955,267194315~8751145,289443310~747069,314468828~8486536,349416768~304856,362533785~5973519] 2024-09-21T10:34:08.960 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,29360128~12505371,41943040~11639466,71935619~8125313,92274688~5642393,100663296~8551903,125829120~19190644,152649398~502173,155189248~3890647,169402859~1955236,180355072~2422685,184549376~5692441,195156550~10907133,209715200~5587220,218103808~5896939,234881024~10665831,251658240~10662625,264241152~11704308,289406976~783403,314468828~8486536,348127232~1594392,362533785~5973519] 2024-09-21T10:34:08.960 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 6 diff [1848814~8910670,28558403~593635,30271399~11594100,45048342~8534164,57756520~1469204,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,127714892~10134136,138318126~6701638,146239075~3597571,152870944~280627,158059215~1020680,167991057~5531964,180722510~2055247,186731431~3510386,195156550~10907133,211221032~4081388,220782438~3218309,238706507~3881063,242881143~2665712,255735910~6584955,267194315~8751145,289443310~747069,294637427~5715128,303093367~4354075,314468828~8486536,349416768~304856,361489784~7914531] 2024-09-21T10:34:08.960 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,28558403~593635,30271399~11594100,45048342~8534164,57756520~1469204,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,127714892~10134136,138318126~6701638,146239075~3597571,152870944~280627,158059215~1020680,167991057~5531964,180722510~2055247,186731431~3510386,195156550~10907133,211221032~4081388,220782438~3218309,238706507~3881063,242881143~2665712,255735910~6584955,267194315~8751145,289443310~747069,294637427~5715128,303093367~4354075,314468828~8486536,349416768~304856,361489784~7914531] 2024-09-21T10:34:08.966 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,25165824~3986214,29360128~12505371,41943040~11639466,57531668~1694056,71935619~8125313,92274688~5642393,100663296~8551903,125829120~19190644,145565003~4271643,152649398~502173,155189248~3890647,167991057~5531964,180355072~2422685,184549376~5692441,195156550~10907133,209715200~5587220,218103808~5896939,234881024~10665831,251658240~10662625,264241152~11704308,289406976~783403,293601280~6751275,301989888~5457554,314468828~8486536,348127232~1594392,361489784~7914531] 2024-09-21T10:34:08.966 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 7 diff [1848814~8910670,28558403~593635,30271399~11594100,45048342~8534164,57756520~1469204,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,127714892~10134136,138318126~7793832,146239075~3597571,152870944~280627,158059215~1020680,167991057~6561399,180722510~2055247,186731431~3510386,195156550~10907133,211221032~4081388,220782438~3218309,224340984~6971051,238706507~3881063,242881143~2665712,255735910~6584955,267194315~8751145,289443310~747069,294637427~5715128,303093367~4354075,314468828~14201249,349416768~304856,361489784~7914531,369976939~7305065,400476606~3423466] 2024-09-21T10:34:08.966 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,28558403~593635,30271399~11594100,45048342~8534164,57756520~1469204,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,127714892~10134136,138318126~7793832,146239075~3597571,152870944~280627,158059215~1020680,167991057~6561399,180722510~2055247,186731431~3510386,195156550~10907133,211221032~4081388,220782438~3218309,224340984~6971051,238706507~3881063,242881143~2665712,255735910~6584955,267194315~8751145,289443310~747069,294637427~5715128,303093367~4354075,314468828~14201249,349416768~304856,361489784~7914531,369976939~7305065,400476606~3423466] 2024-09-21T10:34:08.972 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,25165824~3986214,29360128~12505371,41943040~11639466,57531668~1694056,71935619~8125313,92274688~5642393,100663296~8551903,125829120~24007526,152649398~502173,155189248~3890647,167991057~6561399,180355072~2422685,184549376~5692441,195156550~10907133,209715200~5587220,218103808~13208227,234881024~10665831,251658240~10662625,264241152~11704308,289406976~783403,293601280~6751275,301989888~5457554,314468828~14201249,348127232~1594392,361489784~7914531,369976939~7305065,398458880~5441192] 2024-09-21T10:34:08.972 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 8 diff [1848814~8910670,16642670~3711712,28558403~593635,30271399~11594100,45048342~8534164,57756520~1469204,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,127714892~10134136,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,180722510~2055247,186731431~3510386,195156550~10907133,211221032~4081388,220782438~3218309,224340984~6971051,238706507~3881063,242881143~2665712,254252427~9347451,267194315~8751145,289443310~747069,294637427~5715128,303093367~4354075,314468828~14201249,349416768~304856,361489784~7914531,369976939~9230871,400476606~3423466] 2024-09-21T10:34:08.972 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,16642670~3711712,28558403~593635,30271399~11594100,45048342~8534164,57756520~1469204,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,127714892~10134136,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,180722510~2055247,186731431~3510386,195156550~10907133,211221032~4081388,220782438~3218309,224340984~6971051,238706507~3881063,242881143~2665712,254252427~9347451,267194315~8751145,289443310~747069,294637427~5715128,303093367~4354075,314468828~14201249,349416768~304856,361489784~7914531,369976939~9230871,400476606~3423466] 2024-09-21T10:34:08.977 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~7771470,25165824~3986214,29360128~12505371,41943040~11639466,57531668~1694056,71935619~8125313,92274688~5642393,100663296~8551903,125829120~24007526,152039608~7712103,163577856~10974600,180355072~2422685,184549376~5692441,195156550~10907133,209715200~5587220,218103808~13208227,234881024~10665831,251658240~11941638,264241152~11704308,289406976~783403,293601280~6751275,301989888~5457554,314468828~14201249,348127232~1594392,361489784~7914531,369976939~9230871,398458880~5441192] 2024-09-21T10:34:08.978 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 9 diff [1848814~8910670,16642670~3711712,28558403~593635,30271399~11594100,45048342~8534164,57756520~1469204,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,115995156~2434204,127714892~10134136,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,186731431~3510386,195156550~10907133,211221032~4081388,220782438~3218309,224340984~6971051,238706507~8545657,254252427~21693033,289443310~747069,294637427~5715128,303093367~4354075,314468828~14201249,343124038~1645752,349416768~304856,361489784~7914531,369976939~9230871,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:08.978 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,16642670~3711712,28558403~593635,30271399~11594100,45048342~8534164,57756520~1469204,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,115995156~2434204,127714892~10134136,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,186731431~3510386,195156550~10907133,211221032~4081388,220782438~3218309,224340984~6971051,238706507~8545657,254252427~21693033,289443310~747069,294637427~5715128,303093367~4354075,314468828~14201249,343124038~1645752,349416768~304856,361489784~7914531,369976939~9230871,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:08.983 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~7771470,25165824~3986214,29360128~12505371,41943040~11639466,57531668~1694056,71935619~8125313,92274688~5642393,100663296~8551903,113246208~5183152,125829120~24007526,152039608~7712103,163577856~15649170,180355072~2422685,184549376~5692441,195156550~10907133,209715200~5587220,218103808~13208227,234881024~12371140,251658240~24287220,289406976~783403,293601280~6751275,301989888~5457554,314468828~14201249,339738624~5031166,348127232~1594392,361489784~7914531,369976939~22663644,398458880~5441192] 2024-09-21T10:34:08.983 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 10 diff [1848814~8910670,16642670~3711712,28558403~593635,30271399~11594100,45048342~8534164,57756520~1469204,71935619~8125313,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~10134136,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,186731431~3510386,195156550~10907133,207781564~1395640,211221032~4081388,220782438~3218309,224340984~6971051,238706507~8545657,254252427~26661514,289443310~747069,294637427~5715128,303093367~6669863,311582767~17087310,339701748~5068042,349416768~304856,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:08.983 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,16642670~3711712,28558403~593635,30271399~11594100,45048342~8534164,57756520~1469204,71935619~8125313,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~10134136,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,186731431~3510386,195156550~10907133,207781564~1395640,211221032~4081388,220782438~3218309,224340984~6971051,238706507~8545657,254252427~26661514,289443310~747069,294637427~5715128,303093367~6669863,311582767~17087310,339701748~5068042,349416768~304856,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:08.989 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~7771470,25165824~3986214,29360128~12505371,41943040~11639466,57531668~1694056,71935619~8125313,92274688~16940511,113246208~6316712,125829120~24007526,152039608~7712103,163577856~15649170,180355072~2422685,184549376~5692441,195156550~14020654,209715200~5587220,218103808~13208227,234881024~12371140,251658240~29255701,289406976~783403,293601280~6751275,301989888~7773342,311582767~17087310,335544320~9225470,348127232~1594392,361489784~31150799,398458880~5441192] 2024-09-21T10:34:08.989 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 11 diff [1848814~8910670,16642670~3711712,28558403~593635,30271399~11594100,45048342~10301049,57756520~1469204,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~10134136,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,186731431~3510386,195156550~10907133,207781564~1395640,211221032~4081388,220782438~3218309,224340984~6971051,236839701~10412463,254252427~26661514,289443310~747069,294637427~15125803,311582767~17087310,339701748~5068042,349416768~304856,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:08.990 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,16642670~3711712,28558403~593635,30271399~11594100,45048342~10301049,57756520~1469204,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~10134136,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,186731431~3510386,195156550~10907133,207781564~1395640,211221032~4081388,220782438~3218309,224340984~6971051,236839701~10412463,254252427~26661514,289443310~747069,294637427~15125803,311582767~17087310,339701748~5068042,349416768~304856,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:08.995 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~7771470,25165824~3986214,29360128~12505371,41943040~13406351,57531668~1694056,62914560~4572360,71935619~8125313,82908058~1860108,92274688~16940511,113246208~6316712,125829120~24007526,152039608~8821599,163577856~15649170,180355072~2422685,184549376~5692441,195156550~14020654,209715200~5587220,218103808~13208227,234881024~12371140,251658240~29255701,289406976~783403,293601280~16161950,311582767~17087310,335544320~9225470,348127232~1594392,361489784~31150799,398458880~5441192] 2024-09-21T10:34:08.995 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 12 diff [1848814~8910670,16642670~3711712,27991252~13874247,45048342~10301049,57756520~1469204,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~10134136,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,167641738~6910718,175387176~3839850,180722510~9519307,191528500~14535183,207781564~1395640,211221032~4081388,220782438~3218309,224340984~8144255,236839701~10412463,254252427~26661514,289443310~747069,294637427~34032650,339701748~5068042,349416768~304856,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:08.995 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,16642670~3711712,27991252~13874247,45048342~10301049,57756520~1469204,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~10134136,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,167641738~6910718,175387176~3839850,180722510~9519307,191528500~14535183,207781564~1395640,211221032~4081388,220782438~3218309,224340984~8144255,236839701~10412463,254252427~26661514,289443310~747069,294637427~34032650,339701748~5068042,349416768~304856,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.001 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~7771470,25165824~16699675,41943040~13406351,57531668~1694056,62914560~4572360,71935619~8125313,82908058~1860108,92274688~16940511,113246208~6316712,125829120~24007526,152039608~8821599,163577856~15649170,180355072~9886745,191528500~17648704,209715200~5587220,218103808~14381431,234881024~12371140,251658240~29255701,289406976~783403,293601280~35068797,335544320~9225470,348127232~1594392,361489784~31150799,398458880~5441192] 2024-09-21T10:34:09.001 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 13 diff [1848814~8910670,16642670~3711712,27991252~13874247,45048342~12441594,57756520~1469204,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~10134136,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~1395640,211221032~4081388,220782438~3218309,224340984~8144255,236839701~14643462,254252427~26661514,289443310~747069,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,361489784~17718026,381111195~1866659,384603168~8037415,394221093~9678979] 2024-09-21T10:34:09.001 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,16642670~3711712,27991252~13874247,45048342~12441594,57756520~1469204,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~10134136,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~1395640,211221032~4081388,220782438~3218309,224340984~8144255,236839701~14643462,254252427~26661514,289443310~747069,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,361489784~17718026,381111195~1866659,384603168~8037415,394221093~9678979] 2024-09-21T10:34:09.007 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~7771470,25165824~16699675,41943040~15546896,57531668~1694056,62914560~4572360,71935619~8125313,82908058~1860108,92274688~16940511,113246208~6316712,125829120~24007526,152039608~8821599,163577856~15649170,180355072~9886745,191528500~17648704,209715200~5587220,218103808~14381431,234881024~16602139,251658240~29255701,289406976~783403,293601280~35068797,335544320~12235484,348127232~1594392,361489784~31150799,393702666~10197406] 2024-09-21T10:34:09.007 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 14 diff [1848814~8910670,16642670~4162967,27991252~13874247,45048342~12441594,57756520~1469204,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,119655166~4147395,127714892~10134136,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~1395640,211221032~4081388,220782438~3218309,224340984~8144255,236839701~14643462,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,360853018~18354792,381111195~1866659,384603168~8037415,394221093~9678979] 2024-09-21T10:34:09.007 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,16642670~4162967,27991252~13874247,45048342~12441594,57756520~1469204,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,119655166~4147395,127714892~10134136,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~1395640,211221032~4081388,220782438~3218309,224340984~8144255,236839701~14643462,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,360853018~18354792,381111195~1866659,384603168~8037415,394221093~9678979] 2024-09-21T10:34:09.013 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~8222725,25165824~16699675,41943040~15546896,57531668~1694056,62914560~4572360,71935619~8125313,82908058~1860108,92274688~16940511,113246208~6316712,119655166~4147395,125829120~24007526,152039608~8821599,163577856~15649170,180355072~9886745,191528500~17648704,209715200~5587220,218103808~14381431,234881024~16602139,251658240~29255701,285212672~5425980,293601280~35068797,335544320~12235484,348127232~1594392,360853018~31787565,393702666~10197406] 2024-09-21T10:34:09.013 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 15 diff [1848814~8910670,16642670~4162967,27991252~13874247,45048342~12441594,57756520~2066114,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,119655166~4147395,127714892~10134136,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~1395640,210226943~5075477,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~4338875,360853018~18354792,381111195~1866659,384505041~8135542,394221093~9678979] 2024-09-21T10:34:09.013 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,16642670~4162967,27991252~13874247,45048342~12441594,57756520~2066114,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,119655166~4147395,127714892~10134136,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~1395640,210226943~5075477,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~4338875,360853018~18354792,381111195~1866659,384505041~8135542,394221093~9678979] 2024-09-21T10:34:09.020 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~8222725,25165824~16699675,41943040~15546896,57531668~2290966,62914560~4572360,71935619~8125313,82908058~1860108,92274688~16940511,113246208~6316712,119655166~4147395,125829120~25571602,152039608~8821599,163577856~15649170,180355072~9886745,191528500~17648704,209715200~5587220,218103808~33379355,251658240~29255701,285212672~5425980,293601280~35068797,335544320~12235484,348127232~7438959,360853018~31787565,393702666~10197406] 2024-09-21T10:34:09.020 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 16 diff [1848814~8910670,16642670~4956869,25706100~16159399,45048342~14774292,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98522774~7446753,106701993~2513206,114683657~4879263,119655166~4147395,127714892~10134136,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~1395640,210226943~5075477,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~4338875,360853018~18354792,380931932~31586,381111195~1866659,384505041~8135542,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.020 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,16642670~4956869,25706100~16159399,45048342~14774292,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98522774~7446753,106701993~2513206,114683657~4879263,119655166~4147395,127714892~10134136,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~1395640,210226943~5075477,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~4338875,360853018~18354792,380931932~31586,381111195~1866659,384505041~8135542,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.026 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~9016627,25165824~16699675,41943040~17879594,62914560~4572360,71935619~8125313,82908058~1860108,92274688~16940511,113246208~6316712,119655166~4147395,125829120~25571602,152039608~8821599,163577856~15649170,180355072~9886745,191528500~17648704,209715200~5587220,218103808~33379355,251658240~29255701,285212672~5425980,293601280~35068797,335544320~12235484,348127232~7438959,360853018~31787565,393702666~10197406,409717800~2000564] 2024-09-21T10:34:09.027 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 17 diff [1848814~8910670,15258652~7024612,25706100~16159399,45048342~14774292,62918670~17142262,82908058~1860108,93118269~16096930,114683657~9118904,127714892~10134136,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~1395640,209930960~5371460,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~4338875,360853018~18354792,380931932~31586,381111195~1866659,384505041~8135542,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.027 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~7024612,25706100~16159399,45048342~14774292,62918670~17142262,82908058~1860108,93118269~16096930,114683657~9118904,127714892~10134136,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~1395640,209930960~5371460,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~4338875,360853018~18354792,380931932~31586,381111195~1866659,384505041~8135542,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.032 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~9700352,25165824~16699675,41943040~17879594,62914560~17146372,82908058~1860108,92274688~16940511,113246208~10556353,125829120~25571602,152039608~8821599,163577856~15649170,180355072~9886745,191528500~17648704,209715200~5587220,218103808~33379355,251658240~29255701,285212672~5425980,293601280~35068797,335544320~12235484,348127232~7438959,360853018~31787565,393702666~10197406,409717800~2000564] 2024-09-21T10:34:09.032 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 18 diff [1848814~8910670,15258652~7024612,25706100~16159399,45048342~14774292,62918670~17142262,82908058~1860108,93118269~16096930,114683657~9118904,124591324~13257704,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~9355151,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,291766799~42181,292083899~1585781,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.032 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~7024612,25706100~16159399,45048342~14774292,62918670~17142262,82908058~1860108,93118269~16096930,114683657~9118904,124591324~13257704,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~9355151,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,291766799~42181,292083899~1585781,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.038 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~9700352,25165824~16699675,41943040~17879594,62914560~17146372,82908058~1860108,92274688~16940511,113246208~38154514,152039608~8821599,163577856~15649170,180355072~9886745,191528500~25608215,218103808~33379355,251658240~29255701,285212672~43457405,335544320~12235484,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:09.038 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 19 diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.039 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.044 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:09.045 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 3 diff [75540009~4520923,198347760~7715923,211221032~4081388,220782438~3218309,258320585~2404839,267194315~7846772,314468828~3622333,319022865~2209955,362533785~5973519] 2024-09-21T10:34:09.045 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [75540009~4520923,198347760~7715923,211221032~4081388,220782438~3218309,258320585~2404839,267194315~7846772,314468828~3622333,319022865~2209955,362533785~5973519] 2024-09-21T10:34:09.050 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [75540009~4520923,197135490~8928193,209715200~5587220,218103808~5896939,258320585~2404839,264241152~10799935,314468828~3622333,319022865~2209955,362533785~5973519] 2024-09-21T10:34:09.051 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 4 diff [7577098~3182386,71935619~8125313,106701993~2513206,152870944~280627,181304442~350166,186731431~3510386,198347760~7715923,211221032~4081388,220782438~3218309,255735910~5447384,267194315~8751145,314468828~3622333,319022865~2209955,349416768~304856,362533785~5973519] 2024-09-21T10:34:09.051 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [7577098~3182386,71935619~8125313,106701993~2513206,152870944~280627,181304442~350166,186731431~3510386,198347760~7715923,211221032~4081388,220782438~3218309,255735910~5447384,267194315~8751145,314468828~3622333,319022865~2209955,349416768~304856,362533785~5973519] 2024-09-21T10:34:09.056 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [7577098~3182386,71935619~8125313,104857600~4357599,152649398~502173,180355072~1299536,184549376~5692441,197135490~8928193,209715200~5587220,218103808~5896939,251658240~9525054,264241152~11704308,314468828~3622333,319022865~2209955,348127232~1594392,362533785~5973519] 2024-09-21T10:34:09.056 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 5 diff [7577098~3182386,45048342~8105318,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,127714892~7812122,138318126~6701638,152870944~280627,180722510~2055247,186731431~3510386,195156550~10907133,211221032~4081388,220782438~3218309,238706507~3881063,255735910~5447384,267194315~8751145,289443310~747069,314468828~3622333,319022865~2209955,349416768~304856,362533785~5973519] 2024-09-21T10:34:09.056 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [7577098~3182386,45048342~8105318,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,127714892~7812122,138318126~6701638,152870944~280627,180722510~2055247,186731431~3510386,195156550~10907133,211221032~4081388,220782438~3218309,238706507~3881063,255735910~5447384,267194315~8751145,289443310~747069,314468828~3622333,319022865~2209955,349416768~304856,362533785~5973519] 2024-09-21T10:34:09.062 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [7577098~3182386,41943040~11210620,71935619~8125313,92274688~5642393,100663296~8551903,125829120~9697894,137849028~7170736,152649398~502173,180355072~2422685,184549376~5692441,195156550~10907133,209715200~5587220,218103808~5896939,234881024~7706546,251658240~9525054,264241152~11704308,289406976~783403,314468828~3622333,319022865~2209955,348127232~1594392,362533785~5973519] 2024-09-21T10:34:09.062 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 6 diff [7577098~3182386,28558403~593635,35578395~2688472,45048342~8105318,57756520~1469204,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,127714892~7812122,138318126~6701638,146239075~3597571,152870944~280627,167991057~5531964,180722510~2055247,186731431~3510386,195156550~10907133,211221032~4081388,220782438~3218309,238706507~3881063,255735910~5447384,267194315~8751145,289443310~747069,294637427~5715128,303093367~4354075,314468828~3622333,319022865~3040151,349416768~304856,361489784~7914531] 2024-09-21T10:34:09.062 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [7577098~3182386,28558403~593635,35578395~2688472,45048342~8105318,57756520~1469204,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,127714892~7812122,138318126~6701638,146239075~3597571,152870944~280627,167991057~5531964,180722510~2055247,186731431~3510386,195156550~10907133,211221032~4081388,220782438~3218309,238706507~3881063,255735910~5447384,267194315~8751145,289443310~747069,294637427~5715128,303093367~4354075,314468828~3622333,319022865~3040151,349416768~304856,361489784~7914531] 2024-09-21T10:34:09.068 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [7577098~3182386,25165824~3986214,35578395~2688472,41943040~11210620,57531668~1694056,71935619~8125313,92274688~5642393,100663296~8551903,125829120~9697894,137849028~7170736,145565003~4271643,152649398~502173,167991057~5531964,180355072~2422685,184549376~5692441,195156550~10907133,209715200~5587220,218103808~5896939,234881024~7706546,251658240~9525054,264241152~11704308,289406976~783403,293601280~6751275,301989888~5457554,314468828~3622333,319022865~3040151,348127232~1594392,361489784~7914531] 2024-09-21T10:34:09.068 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 7 diff [7016728~3742756,28558403~593635,35578395~2688472,39931377~204155,45048342~8105318,57756520~1469204,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,127714892~7812122,138318126~7793832,146239075~3597571,152870944~280627,167991057~6561399,180722510~2055247,186731431~3510386,195156550~10907133,211221032~4081388,220782438~3218309,224340984~6971051,238706507~3881063,255735910~5447384,267194315~8751145,289443310~747069,294637427~5715128,303093367~4354075,314468828~3622333,319022865~3040151,322140464~6529613,349416768~304856,361489784~7914531,369976939~7305065,400476606~3423466] 2024-09-21T10:34:09.068 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [7016728~3742756,28558403~593635,35578395~2688472,39931377~204155,45048342~8105318,57756520~1469204,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,127714892~7812122,138318126~7793832,146239075~3597571,152870944~280627,167991057~6561399,180722510~2055247,186731431~3510386,195156550~10907133,211221032~4081388,220782438~3218309,224340984~6971051,238706507~3881063,255735910~5447384,267194315~8751145,289443310~747069,294637427~5715128,303093367~4354075,314468828~3622333,319022865~3040151,322140464~6529613,349416768~304856,361489784~7914531,369976939~7305065,400476606~3423466] 2024-09-21T10:34:09.074 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [7016728~3742756,25165824~3986214,35578395~2688472,39931377~204155,41943040~11210620,57531668~1694056,71935619~8125313,92274688~5642393,100663296~8551903,125829120~9697894,137849028~11987618,152649398~502173,167991057~6561399,180355072~2422685,184549376~5692441,195156550~10907133,209715200~5587220,218103808~13208227,234881024~7706546,251658240~9525054,264241152~11704308,289406976~783403,293601280~6751275,301989888~5457554,314468828~3622333,319022865~3040151,322140464~6529613,348127232~1594392,361489784~7914531,369976939~7305065,398458880~5441192] 2024-09-21T10:34:09.074 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 8 diff [2875569~7883915,16642670~3711712,28558403~593635,35578395~2688472,39931377~204155,45048342~8105318,57756520~1469204,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,127714892~7812122,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,180722510~2055247,186731431~3510386,195156550~10907133,211221032~4081388,220782438~3218309,224340984~6971051,238706507~3881063,254252427~9347451,267194315~8751145,289443310~747069,294637427~5715128,303093367~4354075,314468828~3622333,319022865~3040151,322140464~6529613,349416768~304856,361489784~7914531,369976939~9230871,400476606~3423466] 2024-09-21T10:34:09.074 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,16642670~3711712,28558403~593635,35578395~2688472,39931377~204155,45048342~8105318,57756520~1469204,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,127714892~7812122,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,180722510~2055247,186731431~3510386,195156550~10907133,211221032~4081388,220782438~3218309,224340984~6971051,238706507~3881063,254252427~9347451,267194315~8751145,289443310~747069,294637427~5715128,303093367~4354075,314468828~3622333,319022865~3040151,322140464~6529613,349416768~304856,361489784~7914531,369976939~9230871,400476606~3423466] 2024-09-21T10:34:09.082 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~7771470,25165824~3986214,35578395~2688472,39931377~204155,41943040~11210620,57531668~1694056,71935619~8125313,92274688~5642393,100663296~8551903,125829120~9697894,137849028~11987618,152039608~7712103,163577856~10974600,180355072~2422685,184549376~5692441,195156550~10907133,209715200~5587220,218103808~13208227,234881024~7706546,251658240~11941638,264241152~11704308,289406976~783403,293601280~6751275,301989888~5457554,314468828~3622333,319022865~3040151,322140464~6529613,348127232~1594392,361489784~7914531,369976939~9230871,398458880~5441192] 2024-09-21T10:34:09.082 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 9 diff [2875569~7883915,16642670~3711712,28558403~593635,35578395~2688472,39931377~204155,45048342~8105318,57756520~1469204,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,115995156~2434204,127714892~7812122,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,186731431~3510386,195156550~10907133,211221032~4081388,220782438~3218309,224340984~6971051,238706507~8545657,254252427~21693033,289443310~747069,294637427~5715128,303093367~4354075,314468828~14201249,343124038~1645752,349416768~304856,361489784~7914531,369976939~9230871,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.082 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,16642670~3711712,28558403~593635,35578395~2688472,39931377~204155,45048342~8105318,57756520~1469204,71935619~8125313,93118269~4798812,101882815~4086712,106701993~2513206,115995156~2434204,127714892~7812122,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,186731431~3510386,195156550~10907133,211221032~4081388,220782438~3218309,224340984~6971051,238706507~8545657,254252427~21693033,289443310~747069,294637427~5715128,303093367~4354075,314468828~14201249,343124038~1645752,349416768~304856,361489784~7914531,369976939~9230871,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.088 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~7771470,25165824~3986214,35578395~2688472,39931377~204155,41943040~11210620,57531668~1694056,71935619~8125313,92274688~5642393,100663296~8551903,113246208~5183152,125829120~9697894,137849028~11987618,152039608~7712103,163577856~15649170,180355072~2422685,184549376~5692441,195156550~10907133,209715200~5587220,218103808~13208227,234881024~12371140,251658240~24287220,289406976~783403,293601280~6751275,301989888~5457554,314468828~14201249,339738624~5031166,348127232~1594392,361489784~7914531,369976939~22663644,398458880~5441192] 2024-09-21T10:34:09.089 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 10 diff [2875569~7883915,16642670~3711712,28558403~593635,35578395~2688472,39931377~204155,45048342~8105318,57756520~1469204,71935619~8125313,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~7812122,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,186731431~3510386,195156550~10907133,207781564~1395640,211221032~4081388,220782438~3218309,224340984~6971051,238706507~8545657,254252427~26661514,289443310~747069,294637427~5715128,303093367~6669863,311582767~17087310,339701748~5068042,349416768~304856,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.089 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,16642670~3711712,28558403~593635,35578395~2688472,39931377~204155,45048342~8105318,57756520~1469204,71935619~8125313,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~7812122,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,186731431~3510386,195156550~10907133,207781564~1395640,211221032~4081388,220782438~3218309,224340984~6971051,238706507~8545657,254252427~26661514,289443310~747069,294637427~5715128,303093367~6669863,311582767~17087310,339701748~5068042,349416768~304856,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:08 smithi160 ceph-mon[101879]: osdmap e1129: 8 total, 8 up, 8 in 2024-09-21T10:34:09.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:08 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:34:09.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:08 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:34:09.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:08 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:34:09.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:08 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:34:09.094 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~7771470,25165824~3986214,35578395~2688472,39931377~204155,41943040~11210620,57531668~1694056,71935619~8125313,92274688~16940511,113246208~6316712,125829120~9697894,137849028~11987618,152039608~7712103,163577856~15649170,180355072~2422685,184549376~5692441,195156550~14020654,209715200~5587220,218103808~13208227,234881024~12371140,251658240~29255701,289406976~783403,293601280~6751275,301989888~7773342,311582767~17087310,335544320~9225470,348127232~1594392,361489784~31150799,398458880~5441192] 2024-09-21T10:34:09.094 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 11 diff [2875569~7883915,16642670~3711712,28558403~593635,31536075~6730792,39931377~204155,45048342~10301049,57756520~1469204,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~7812122,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,186731431~3510386,195156550~10907133,207781564~1395640,211221032~4081388,220782438~3218309,224340984~6971051,236839701~10412463,254252427~26661514,289443310~747069,294637427~15125803,311582767~17087310,339701748~5068042,349416768~304856,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.095 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,16642670~3711712,28558403~593635,31536075~6730792,39931377~204155,45048342~10301049,57756520~1469204,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~7812122,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,186731431~3510386,195156550~10907133,207781564~1395640,211221032~4081388,220782438~3218309,224340984~6971051,236839701~10412463,254252427~26661514,289443310~747069,294637427~15125803,311582767~17087310,339701748~5068042,349416768~304856,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.102 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~7771470,25165824~3986214,31536075~6730792,39931377~204155,41943040~13406351,57531668~1694056,62914560~4572360,71935619~8125313,82908058~1860108,92274688~16940511,113246208~6316712,125829120~9697894,137849028~11987618,152039608~8821599,163577856~15649170,180355072~2422685,184549376~5692441,195156550~14020654,209715200~5587220,218103808~13208227,234881024~12371140,251658240~29255701,289406976~783403,293601280~16161950,311582767~17087310,335544320~9225470,348127232~1594392,361489784~31150799,398458880~5441192] 2024-09-21T10:34:09.102 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 12 diff [2875569~7883915,16642670~3711712,27991252~10275615,39931377~204155,45048342~10301049,57756520~1469204,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~7812122,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,167641738~6910718,175387176~3839850,180722510~9519307,191528500~14535183,207781564~1395640,211221032~4081388,220782438~3218309,224340984~8144255,236839701~10412463,254252427~26661514,289443310~747069,294637427~34032650,339701748~5068042,349416768~304856,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.102 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,16642670~3711712,27991252~10275615,39931377~204155,45048342~10301049,57756520~1469204,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~7812122,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,167641738~6910718,175387176~3839850,180722510~9519307,191528500~14535183,207781564~1395640,211221032~4081388,220782438~3218309,224340984~8144255,236839701~10412463,254252427~26661514,289443310~747069,294637427~34032650,339701748~5068042,349416768~304856,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.107 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~7771470,25165824~13101043,39931377~204155,41943040~13406351,57531668~1694056,62914560~4572360,71935619~8125313,82908058~1860108,92274688~16940511,113246208~6316712,125829120~9697894,137849028~11987618,152039608~8821599,163577856~15649170,180355072~9886745,191528500~17648704,209715200~5587220,218103808~14381431,234881024~12371140,251658240~29255701,289406976~783403,293601280~35068797,335544320~9225470,348127232~1594392,361489784~31150799,398458880~5441192] 2024-09-21T10:34:09.107 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 13 diff [2875569~7883915,16642670~3711712,27991252~10275615,39931377~204155,45048342~12441594,57756520~1469204,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~7812122,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~1395640,211221032~4081388,220782438~3218309,224340984~8144255,236839701~14643462,254252427~26661514,289443310~747069,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,361489784~17718026,381111195~1866659,384603168~8037415,394221093~9678979] 2024-09-21T10:34:09.107 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,16642670~3711712,27991252~10275615,39931377~204155,45048342~12441594,57756520~1469204,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~7812122,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~1395640,211221032~4081388,220782438~3218309,224340984~8144255,236839701~14643462,254252427~26661514,289443310~747069,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,361489784~17718026,381111195~1866659,384603168~8037415,394221093~9678979] 2024-09-21T10:34:09.113 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~7771470,25165824~13101043,39931377~204155,41943040~15546896,57531668~1694056,62914560~4572360,71935619~8125313,82908058~1860108,92274688~16940511,113246208~6316712,125829120~9697894,137849028~11987618,152039608~8821599,163577856~15649170,180355072~9886745,191528500~17648704,209715200~5587220,218103808~14381431,234881024~16602139,251658240~29255701,289406976~783403,293601280~35068797,335544320~12235484,348127232~1594392,361489784~31150799,393702666~10197406] 2024-09-21T10:34:09.113 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 14 diff [2875569~7883915,16642670~4162967,27991252~10275615,39931377~204155,45048342~12441594,57756520~1469204,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,119655166~4147395,127714892~7812122,135872708~1754391,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~1395640,211221032~4081388,220782438~3218309,224340984~8144255,236839701~14643462,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,360853018~18354792,381111195~1866659,384603168~8037415,394221093~9678979] 2024-09-21T10:34:09.113 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,16642670~4162967,27991252~10275615,39931377~204155,45048342~12441594,57756520~1469204,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,119655166~4147395,127714892~7812122,135872708~1754391,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~1395640,211221032~4081388,220782438~3218309,224340984~8144255,236839701~14643462,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,360853018~18354792,381111195~1866659,384603168~8037415,394221093~9678979] 2024-09-21T10:34:09.119 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~8222725,25165824~13101043,39931377~204155,41943040~15546896,57531668~1694056,62914560~4572360,71935619~8125313,82908058~1860108,92274688~16940511,113246208~6316712,119655166~4147395,125829120~9697894,135872708~1754391,137849028~11987618,152039608~8821599,163577856~15649170,180355072~9886745,191528500~17648704,209715200~5587220,218103808~14381431,234881024~16602139,251658240~29255701,285212672~5425980,293601280~35068797,335544320~12235484,348127232~1594392,360853018~31787565,393702666~10197406] 2024-09-21T10:34:09.119 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 15 diff [2875569~7883915,16642670~4162967,27991252~10275615,39931377~204155,45048342~12441594,57756520~2066114,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,119655166~4147395,127714892~7812122,135872708~1754391,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~1395640,210226943~5075477,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~4338875,360853018~18354792,381111195~1866659,384505041~8135542,394221093~9678979] 2024-09-21T10:34:09.119 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,16642670~4162967,27991252~10275615,39931377~204155,45048342~12441594,57756520~2066114,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,119655166~4147395,127714892~7812122,135872708~1754391,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~1395640,210226943~5075477,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~4338875,360853018~18354792,381111195~1866659,384505041~8135542,394221093~9678979] 2024-09-21T10:34:09.125 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~8222725,25165824~13101043,39931377~204155,41943040~15546896,57531668~2290966,62914560~4572360,71935619~8125313,82908058~1860108,92274688~16940511,113246208~6316712,119655166~4147395,125829120~9697894,135872708~1754391,137849028~13551694,152039608~8821599,163577856~15649170,180355072~9886745,191528500~17648704,209715200~5587220,218103808~33379355,251658240~29255701,285212672~5425980,293601280~35068797,335544320~12235484,348127232~7438959,360853018~31787565,393702666~10197406] 2024-09-21T10:34:09.126 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 16 diff [2875569~7883915,16642670~4956869,25706100~12560767,39931377~204155,45048342~14774292,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98522774~7446753,106701993~2513206,114683657~4879263,119655166~4147395,127714892~7812122,135872708~1754391,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~1395640,210226943~5075477,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~4338875,360853018~18354792,380931932~31586,381111195~1866659,384505041~8135542,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.126 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,16642670~4956869,25706100~12560767,39931377~204155,45048342~14774292,62918670~4568250,71935619~8125313,82908058~1860108,93118269~4798812,98522774~7446753,106701993~2513206,114683657~4879263,119655166~4147395,127714892~7812122,135872708~1754391,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~1395640,210226943~5075477,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~4338875,360853018~18354792,380931932~31586,381111195~1866659,384505041~8135542,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:08 smithi070 ceph-mon[107385]: osdmap e1129: 8 total, 8 up, 8 in 2024-09-21T10:34:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:08 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:34:09.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:08 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:34:09.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:08 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:34:09.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:08 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:34:09.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:08 smithi070 ceph-mon[112139]: osdmap e1129: 8 total, 8 up, 8 in 2024-09-21T10:34:09.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:08 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:34:09.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:08 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:34:09.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:08 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:34:09.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:08 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:34:09.131 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~9016627,25165824~13101043,39931377~204155,41943040~17879594,62914560~4572360,71935619~8125313,82908058~1860108,92274688~16940511,113246208~6316712,119655166~4147395,125829120~9697894,135872708~1754391,137849028~13551694,152039608~8821599,163577856~15649170,180355072~9886745,191528500~17648704,209715200~5587220,218103808~33379355,251658240~29255701,285212672~5425980,293601280~35068797,335544320~12235484,348127232~7438959,360853018~31787565,393702666~10197406,409717800~2000564] 2024-09-21T10:34:09.131 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 17 diff [2875569~7883915,15258652~7024612,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82908058~1860108,93118269~16096930,114683657~9118904,127714892~7812122,135872708~1754391,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~1395640,209930960~5371460,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~4338875,360853018~18354792,380931932~31586,381111195~1866659,384505041~8135542,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.132 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~7024612,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82908058~1860108,93118269~16096930,114683657~9118904,127714892~7812122,135872708~1754391,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~1395640,209930960~5371460,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~4338875,360853018~18354792,380931932~31586,381111195~1866659,384505041~8135542,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.137 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~9700352,25165824~13101043,39931377~204155,41943040~17879594,62914560~17146372,82908058~1860108,92274688~16940511,113246208~10556353,125829120~9697894,135872708~1754391,137849028~13551694,152039608~8821599,163577856~15649170,180355072~9886745,191528500~17648704,209715200~5587220,218103808~33379355,251658240~29255701,285212672~5425980,293601280~35068797,335544320~12235484,348127232~7438959,360853018~31787565,393702666~10197406,409717800~2000564] 2024-09-21T10:34:09.137 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 18 diff [2875569~7883915,15258652~7024612,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82908058~1860108,93118269~16096930,114683657~9118904,124591324~10935690,135872708~1754391,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~9355151,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,291766799~42181,292083899~1585781,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.137 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~7024612,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82908058~1860108,93118269~16096930,114683657~9118904,124591324~10935690,135872708~1754391,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~9355151,220782438~3218309,224340984~27142179,254252427~26661514,286147115~4491537,291766799~42181,292083899~1585781,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.143 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~9700352,25165824~13101043,39931377~204155,41943040~17879594,62914560~17146372,82908058~1860108,92274688~16940511,113246208~22280806,135872708~1754391,137849028~13551694,152039608~8821599,163577856~15649170,180355072~9886745,191528500~25608215,218103808~33379355,251658240~29255701,285212672~43457405,335544320~12235484,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:09.143 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 19 diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.143 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.148 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:09.149 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 4 diff [7577098~3182386,71935619~6785475,106701993~2513206,152870944~280627,181304442~350166,186731431~3510386,255735910~5447384,268907245~7038215,349416768~304856] 2024-09-21T10:34:09.149 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [7577098~3182386,71935619~6785475,106701993~2513206,152870944~280627,181304442~350166,186731431~3510386,255735910~5447384,268907245~7038215,349416768~304856] 2024-09-21T10:34:09.154 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [7577098~3182386,71935619~6785475,104857600~4357599,152649398~502173,180355072~1299536,184549376~5692441,251658240~9525054,268907245~7038215,348127232~1594392] 2024-09-21T10:34:09.154 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 5 diff [7577098~3182386,45048342~8105318,71935619~6785475,93118269~4798812,101882815~4086712,106701993~2513206,127714892~7812122,138318126~6701638,152870944~280627,180722510~2055247,186731431~3510386,195156550~3672575,238706507~3881063,255735910~5447384,268907245~7038215,289443310~747069,349416768~304856] 2024-09-21T10:34:09.154 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [7577098~3182386,45048342~8105318,71935619~6785475,93118269~4798812,101882815~4086712,106701993~2513206,127714892~7812122,138318126~6701638,152870944~280627,180722510~2055247,186731431~3510386,195156550~3672575,238706507~3881063,255735910~5447384,268907245~7038215,289443310~747069,349416768~304856] 2024-09-21T10:34:09.160 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [7577098~3182386,41943040~11210620,71935619~6785475,92274688~5642393,100663296~8551903,125829120~9697894,137849028~7170736,152649398~502173,180355072~2422685,184549376~5692441,195156550~3672575,234881024~7706546,251658240~9525054,268907245~7038215,289406976~783403,348127232~1594392] 2024-09-21T10:34:09.161 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 6 diff [7577098~3182386,28558403~593635,35578395~2688472,45048342~8105318,57756520~1469204,71935619~6785475,93118269~4798812,101882815~4086712,106701993~2513206,127714892~7812122,138318126~6701638,146239075~3597571,152870944~280627,167991057~5531964,180722510~2055247,186731431~3510386,195156550~3672575,238706507~3881063,255735910~5447384,268907245~7038215,289443310~747069,294637427~5715128,303093367~4354075,319887642~2175374,349416768~304856,361489784~7914531] 2024-09-21T10:34:09.161 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [7577098~3182386,28558403~593635,35578395~2688472,45048342~8105318,57756520~1469204,71935619~6785475,93118269~4798812,101882815~4086712,106701993~2513206,127714892~7812122,138318126~6701638,146239075~3597571,152870944~280627,167991057~5531964,180722510~2055247,186731431~3510386,195156550~3672575,238706507~3881063,255735910~5447384,268907245~7038215,289443310~747069,294637427~5715128,303093367~4354075,319887642~2175374,349416768~304856,361489784~7914531] 2024-09-21T10:34:09.166 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [7577098~3182386,25165824~3986214,35578395~2688472,41943040~11210620,57531668~1694056,71935619~6785475,92274688~5642393,100663296~8551903,125829120~9697894,137849028~7170736,145565003~4271643,152649398~502173,167991057~5531964,180355072~2422685,184549376~5692441,195156550~3672575,234881024~7706546,251658240~9525054,268907245~7038215,289406976~783403,293601280~6751275,301989888~5457554,319887642~2175374,348127232~1594392,361489784~7914531] 2024-09-21T10:34:09.166 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 7 diff [7016728~3742756,28558403~593635,35578395~2688472,39931377~204155,45048342~8105318,57756520~1469204,71935619~6785475,93118269~4798812,101882815~4086712,106701993~2513206,127714892~7812122,138318126~7793832,146239075~3597571,152870944~280627,167991057~6561399,180722510~2055247,186731431~3510386,195156550~3672575,224340984~6971051,238706507~3881063,255735910~5447384,268907245~7038215,289443310~747069,294637427~5715128,303093367~4354075,319887642~2175374,322140464~6529613,349416768~304856,361489784~7914531,369976939~7305065,400476606~3423466] 2024-09-21T10:34:09.167 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [7016728~3742756,28558403~593635,35578395~2688472,39931377~204155,45048342~8105318,57756520~1469204,71935619~6785475,93118269~4798812,101882815~4086712,106701993~2513206,127714892~7812122,138318126~7793832,146239075~3597571,152870944~280627,167991057~6561399,180722510~2055247,186731431~3510386,195156550~3672575,224340984~6971051,238706507~3881063,255735910~5447384,268907245~7038215,289443310~747069,294637427~5715128,303093367~4354075,319887642~2175374,322140464~6529613,349416768~304856,361489784~7914531,369976939~7305065,400476606~3423466] 2024-09-21T10:34:09.172 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [7016728~3742756,25165824~3986214,35578395~2688472,39931377~204155,41943040~11210620,57531668~1694056,71935619~6785475,92274688~5642393,100663296~8551903,125829120~9697894,137849028~11987618,152649398~502173,167991057~6561399,180355072~2422685,184549376~5692441,195156550~3672575,224000747~7311288,234881024~7706546,251658240~9525054,268907245~7038215,289406976~783403,293601280~6751275,301989888~5457554,319887642~2175374,322140464~6529613,348127232~1594392,361489784~7914531,369976939~7305065,398458880~5441192] 2024-09-21T10:34:09.173 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 8 diff [2875569~7883915,16642670~3711712,28558403~593635,35578395~2688472,39931377~204155,45048342~8105318,57756520~1469204,71935619~6785475,93118269~4798812,101882815~4086712,106701993~2513206,127714892~7812122,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,180722510~2055247,186731431~3510386,195156550~3672575,224340984~6971051,238706507~3881063,254252427~9347451,268907245~7038215,289443310~747069,294637427~5715128,303093367~4354075,319887642~2175374,322140464~6529613,349416768~304856,361489784~7914531,369976939~9230871,400476606~3423466] 2024-09-21T10:34:09.173 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,16642670~3711712,28558403~593635,35578395~2688472,39931377~204155,45048342~8105318,57756520~1469204,71935619~6785475,93118269~4798812,101882815~4086712,106701993~2513206,127714892~7812122,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,180722510~2055247,186731431~3510386,195156550~3672575,224340984~6971051,238706507~3881063,254252427~9347451,268907245~7038215,289443310~747069,294637427~5715128,303093367~4354075,319887642~2175374,322140464~6529613,349416768~304856,361489784~7914531,369976939~9230871,400476606~3423466] 2024-09-21T10:34:09.178 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~7771470,25165824~3986214,35578395~2688472,39931377~204155,41943040~11210620,57531668~1694056,71935619~6785475,92274688~5642393,100663296~8551903,125829120~9697894,137849028~11987618,152039608~7712103,163577856~10974600,180355072~2422685,184549376~5692441,195156550~3672575,224000747~7311288,234881024~7706546,251658240~11941638,268907245~7038215,289406976~783403,293601280~6751275,301989888~5457554,319887642~2175374,322140464~6529613,348127232~1594392,361489784~7914531,369976939~9230871,398458880~5441192] 2024-09-21T10:34:09.178 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 9 diff [2875569~7883915,16642670~3711712,28558403~593635,35578395~2688472,39931377~204155,45048342~8105318,57756520~1469204,71935619~6785475,93118269~4798812,101882815~4086712,106701993~2513206,115995156~2434204,127714892~7812122,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,186731431~3510386,195156550~3672575,224340984~6971051,238706507~8545657,254252427~13677068,268907245~7038215,289443310~747069,294637427~5715128,303093367~4354075,317667912~11002165,343124038~1645752,349416768~304856,361489784~7914531,369976939~9230871,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.178 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,16642670~3711712,28558403~593635,35578395~2688472,39931377~204155,45048342~8105318,57756520~1469204,71935619~6785475,93118269~4798812,101882815~4086712,106701993~2513206,115995156~2434204,127714892~7812122,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,186731431~3510386,195156550~3672575,224340984~6971051,238706507~8545657,254252427~13677068,268907245~7038215,289443310~747069,294637427~5715128,303093367~4354075,317667912~11002165,343124038~1645752,349416768~304856,361489784~7914531,369976939~9230871,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.184 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~7771470,25165824~3986214,35578395~2688472,39931377~204155,41943040~11210620,57531668~1694056,71935619~6785475,92274688~5642393,100663296~8551903,113246208~5183152,125829120~9697894,137849028~11987618,152039608~7712103,163577856~15649170,180355072~2422685,184549376~5692441,195156550~3672575,224000747~7311288,234881024~12371140,251658240~16271255,268907245~7038215,289406976~783403,293601280~6751275,301989888~5457554,317667912~11002165,339738624~5031166,348127232~1594392,361489784~7914531,369976939~22663644,398458880~5441192] 2024-09-21T10:34:09.184 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 10 diff [2875569~7883915,16642670~3711712,28558403~593635,35578395~2688472,39931377~204155,45048342~8105318,57756520~1469204,71935619~6785475,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~7812122,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,186731431~3510386,195156550~3672575,207781564~1395640,224340984~6971051,238706507~8545657,254252427~13677068,268907245~12006696,289443310~747069,294637427~5715128,303093367~6669863,311582767~3753918,317667912~11002165,339701748~5068042,349416768~304856,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.184 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,16642670~3711712,28558403~593635,35578395~2688472,39931377~204155,45048342~8105318,57756520~1469204,71935619~6785475,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~7812122,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,186731431~3510386,195156550~3672575,207781564~1395640,224340984~6971051,238706507~8545657,254252427~13677068,268907245~12006696,289443310~747069,294637427~5715128,303093367~6669863,311582767~3753918,317667912~11002165,339701748~5068042,349416768~304856,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.190 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~7771470,25165824~3986214,35578395~2688472,39931377~204155,41943040~11210620,57531668~1694056,71935619~6785475,92274688~16940511,113246208~6316712,125829120~9697894,137849028~11987618,152039608~7712103,163577856~15649170,180355072~2422685,184549376~5692441,195156550~3672575,206063683~3113521,224000747~7311288,234881024~12371140,251658240~16271255,268907245~12006696,289406976~783403,293601280~6751275,301989888~7773342,311582767~3753918,317667912~11002165,335544320~9225470,348127232~1594392,361489784~31150799,398458880~5441192] 2024-09-21T10:34:09.190 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 11 diff [2875569~7883915,16642670~3711712,28558403~593635,31536075~6730792,39931377~204155,45048342~10301049,57756520~1469204,62918670~4568250,71935619~6785475,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~7812122,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,186731431~3510386,195156550~3672575,207781564~1395640,224340984~6971051,236839701~10412463,254252427~13677068,268907245~12006696,289443310~747069,294637427~15125803,311582767~17087310,339701748~5068042,349416768~304856,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.190 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,16642670~3711712,28558403~593635,31536075~6730792,39931377~204155,45048342~10301049,57756520~1469204,62918670~4568250,71935619~6785475,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~7812122,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,186731431~3510386,195156550~3672575,207781564~1395640,224340984~6971051,236839701~10412463,254252427~13677068,268907245~12006696,289443310~747069,294637427~15125803,311582767~17087310,339701748~5068042,349416768~304856,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.195 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~7771470,25165824~3986214,31536075~6730792,39931377~204155,41943040~13406351,57531668~1694056,62914560~4572360,71935619~6785475,82908058~1860108,92274688~16940511,113246208~6316712,125829120~9697894,137849028~11987618,152039608~8821599,163577856~15649170,180355072~2422685,184549376~5692441,195156550~3672575,206063683~3113521,224000747~7311288,234881024~12371140,251658240~16271255,268907245~12006696,289406976~783403,293601280~16161950,311582767~17087310,335544320~9225470,348127232~1594392,361489784~31150799,398458880~5441192] 2024-09-21T10:34:09.195 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 12 diff [2875569~7883915,16642670~3711712,27991252~10275615,39931377~204155,45048342~10301049,57756520~1469204,62918670~4568250,71935619~6785475,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~7812122,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,167641738~6910718,175387176~3839850,180722510~9519307,191528500~10608093,207781564~1395640,224340984~8144255,236839701~10412463,254252427~13677068,268907245~12006696,289443310~747069,294637427~34032650,339701748~5068042,349416768~304856,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.196 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,16642670~3711712,27991252~10275615,39931377~204155,45048342~10301049,57756520~1469204,62918670~4568250,71935619~6785475,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~7812122,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,167641738~6910718,175387176~3839850,180722510~9519307,191528500~10608093,207781564~1395640,224340984~8144255,236839701~10412463,254252427~13677068,268907245~12006696,289443310~747069,294637427~34032650,339701748~5068042,349416768~304856,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.201 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~7771470,25165824~13101043,39931377~204155,41943040~13406351,57531668~1694056,62914560~4572360,71935619~6785475,82908058~1860108,92274688~16940511,113246208~6316712,125829120~9697894,137849028~11987618,152039608~8821599,163577856~15649170,180355072~9886745,191528500~10608093,206063683~3113521,224000747~8484492,234881024~12371140,251658240~16271255,268907245~12006696,289406976~783403,293601280~35068797,335544320~9225470,348127232~1594392,361489784~31150799,398458880~5441192] 2024-09-21T10:34:09.201 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 13 diff [2875569~7883915,16642670~3711712,27991252~10275615,39931377~204155,45048342~12441594,57756520~1469204,62918670~4568250,71935619~6785475,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~7812122,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~1395640,224340984~8144255,236839701~14643462,254252427~13677068,268907245~12006696,289443310~747069,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,361489784~17718026,381111195~1866659,384603168~8037415,394221093~9678979] 2024-09-21T10:34:09.201 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,16642670~3711712,27991252~10275615,39931377~204155,45048342~12441594,57756520~1469204,62918670~4568250,71935619~6785475,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,127714892~7812122,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~1395640,224340984~8144255,236839701~14643462,254252427~13677068,268907245~12006696,289443310~747069,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,361489784~17718026,381111195~1866659,384603168~8037415,394221093~9678979] 2024-09-21T10:34:09.207 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~7771470,25165824~13101043,39931377~204155,41943040~15546896,57531668~1694056,62914560~4572360,71935619~6785475,82908058~1860108,92274688~16940511,113246208~6316712,125829120~9697894,137849028~11987618,152039608~8821599,163577856~15649170,180355072~9886745,191528500~10608093,206063683~3113521,224000747~8484492,234881024~16602139,251658240~16271255,268907245~12006696,289406976~783403,293601280~35068797,335544320~12235484,348127232~1594392,361489784~31150799,393702666~10197406] 2024-09-21T10:34:09.207 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 14 diff [2875569~7883915,16642670~4162967,27991252~10275615,39931377~204155,45048342~12441594,57756520~1469204,62918670~4568250,71935619~6785475,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,119655166~4147395,127714892~7812122,135872708~1754391,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~1395640,224340984~8144255,236839701~14643462,254252427~13677068,268907245~12006696,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,360853018~18354792,381111195~1866659,384603168~8037415,394221093~9678979] 2024-09-21T10:34:09.207 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,16642670~4162967,27991252~10275615,39931377~204155,45048342~12441594,57756520~1469204,62918670~4568250,71935619~6785475,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,119655166~4147395,127714892~7812122,135872708~1754391,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~1395640,224340984~8144255,236839701~14643462,254252427~13677068,268907245~12006696,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,360853018~18354792,381111195~1866659,384603168~8037415,394221093~9678979] 2024-09-21T10:34:09.213 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~8222725,25165824~13101043,39931377~204155,41943040~15546896,57531668~1694056,62914560~4572360,71935619~6785475,82908058~1860108,92274688~16940511,113246208~6316712,119655166~4147395,125829120~9697894,135872708~1754391,137849028~11987618,152039608~8821599,163577856~15649170,180355072~9886745,191528500~10608093,206063683~3113521,224000747~8484492,234881024~16602139,251658240~16271255,268907245~12006696,285212672~5425980,293601280~35068797,335544320~12235484,348127232~1594392,360853018~31787565,393702666~10197406] 2024-09-21T10:34:09.213 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 15 diff [2875569~7883915,16642670~4162967,27991252~10275615,39931377~204155,45048342~12441594,57756520~2066114,62918670~4568250,71935619~6785475,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,119655166~4147395,127714892~7812122,135872708~1754391,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~1395640,210226943~2559490,224340984~27142179,254252427~13677068,268907245~12006696,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~4338875,360853018~18354792,381111195~1866659,384505041~8135542,394221093~9678979] 2024-09-21T10:34:09.213 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,16642670~4162967,27991252~10275615,39931377~204155,45048342~12441594,57756520~2066114,62918670~4568250,71935619~6785475,82908058~1860108,93118269~4798812,98906260~7063267,106701993~2513206,114683657~4879263,119655166~4147395,127714892~7812122,135872708~1754391,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~1395640,210226943~2559490,224340984~27142179,254252427~13677068,268907245~12006696,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~4338875,360853018~18354792,381111195~1866659,384505041~8135542,394221093~9678979] 2024-09-21T10:34:09.219 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~8222725,25165824~13101043,39931377~204155,41943040~15546896,57531668~2290966,62914560~4572360,71935619~6785475,82908058~1860108,92274688~16940511,113246208~6316712,119655166~4147395,125829120~9697894,135872708~1754391,137849028~13551694,152039608~8821599,163577856~15649170,180355072~9886745,191528500~10608093,206063683~3113521,210226943~2559490,224000747~27482416,251658240~16271255,268907245~12006696,285212672~5425980,293601280~35068797,335544320~12235484,348127232~7438959,360853018~31787565,393702666~10197406] 2024-09-21T10:34:09.219 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 16 diff [2875569~7883915,16642670~4956869,25706100~12560767,39931377~204155,45048342~14774292,62918670~4568250,71935619~6785475,82908058~1860108,93118269~4798812,98522774~7446753,106701993~2513206,114683657~4879263,119655166~4147395,127714892~7812122,135872708~1754391,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~1395640,210226943~2559490,224340984~27142179,254252427~13677068,268907245~12006696,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~4338875,360853018~18354792,380931932~31586,381111195~1866659,384505041~8135542,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.219 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,16642670~4956869,25706100~12560767,39931377~204155,45048342~14774292,62918670~4568250,71935619~6785475,82908058~1860108,93118269~4798812,98522774~7446753,106701993~2513206,114683657~4879263,119655166~4147395,127714892~7812122,135872708~1754391,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~1395640,210226943~2559490,224340984~27142179,254252427~13677068,268907245~12006696,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~4338875,360853018~18354792,380931932~31586,381111195~1866659,384505041~8135542,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.224 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~9016627,25165824~13101043,39931377~204155,41943040~17879594,62914560~4572360,71935619~6785475,82908058~1860108,92274688~16940511,113246208~6316712,119655166~4147395,125829120~9697894,135872708~1754391,137849028~13551694,152039608~8821599,163577856~15649170,180355072~9886745,191528500~10608093,206063683~3113521,210226943~2559490,224000747~27482416,251658240~16271255,268907245~12006696,285212672~5425980,293601280~35068797,335544320~12235484,348127232~7438959,360853018~31787565,393702666~10197406,409717800~2000564] 2024-09-21T10:34:09.225 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 17 diff [2875569~7883915,15258652~7024612,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82908058~1860108,93118269~16096930,114683657~9118904,127714892~7812122,135872708~1754391,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~1395640,209930960~3307546,224340984~27142179,254252427~13677068,268907245~12006696,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~4338875,360853018~18354792,380931932~31586,381111195~1866659,384505041~8135542,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.225 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~7024612,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82908058~1860108,93118269~16096930,114683657~9118904,127714892~7812122,135872708~1754391,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~1395640,209930960~3307546,224340984~27142179,254252427~13677068,268907245~12006696,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~4338875,360853018~18354792,380931932~31586,381111195~1866659,384505041~8135542,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.231 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~9700352,25165824~13101043,39931377~204155,41943040~17879594,62914560~15806534,82908058~1860108,92274688~16940511,113246208~10556353,125829120~9697894,135872708~1754391,137849028~13551694,152039608~8821599,163577856~15649170,180355072~9886745,191528500~10608093,206063683~3113521,209930960~3307546,224000747~27482416,251658240~16271255,268907245~12006696,285212672~5425980,293601280~35068797,335544320~12235484,348127232~7438959,360853018~31787565,393702666~10197406,409717800~2000564] 2024-09-21T10:34:09.231 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 18 diff [2875569~7883915,15258652~7024612,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82908058~1860108,93118269~16096930,114683657~9118904,124591324~10935690,135872708~1754391,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~9355151,224340984~27142179,254252427~26661514,286147115~4491537,291766799~42181,292083899~1585781,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.231 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~7024612,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82908058~1860108,93118269~16096930,114683657~9118904,124591324~10935690,135872708~1754391,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~9355151,224340984~27142179,254252427~26661514,286147115~4491537,291766799~42181,292083899~1585781,294637427~34032650,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.236 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~9700352,25165824~13101043,39931377~204155,41943040~17879594,62914560~15806534,82908058~1860108,92274688~16940511,113246208~22280806,135872708~1754391,137849028~13551694,152039608~8821599,163577856~15649170,180355072~9886745,191528500~10608093,206063683~11073032,224000747~27482416,251658240~29255701,285212672~43457405,335544320~12235484,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:09.236 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 19 diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.236 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.242 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~15806534,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~10608093,206063683~12148949,224000747~27482416,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:09.242 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 5 diff [45048342~8105318,93118269~4798812,101882815~4086712,127714892~7812122,138318126~6701638,180722510~2055247,195156550~3672575,238706507~3881063,289443310~747069] 2024-09-21T10:34:09.242 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [45048342~8105318,93118269~4798812,101882815~4086712,127714892~7812122,138318126~6701638,180722510~2055247,195156550~3672575,238706507~3881063,289443310~747069] 2024-09-21T10:34:09.247 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [41943040~11210620,92274688~5642393,100663296~5306231,125829120~9697894,137849028~7170736,180722510~2055247,195156550~3672575,234881024~7706546,289406976~783403] 2024-09-21T10:34:09.247 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 6 diff [28558403~593635,35578395~2688472,45048342~8105318,57756520~1469204,93118269~4798812,101882815~4086712,127714892~7812122,138318126~6701638,146239075~3597571,167991057~5531964,180722510~2055247,195156550~3672575,238706507~3881063,289443310~747069,294637427~5715128,303093367~4354075,319887642~2175374,361489784~7914531] 2024-09-21T10:34:09.248 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [28558403~593635,35578395~2688472,45048342~8105318,57756520~1469204,93118269~4798812,101882815~4086712,127714892~7812122,138318126~6701638,146239075~3597571,167991057~5531964,180722510~2055247,195156550~3672575,238706507~3881063,289443310~747069,294637427~5715128,303093367~4354075,319887642~2175374,361489784~7914531] 2024-09-21T10:34:09.253 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [25165824~3986214,35578395~2688472,41943040~11210620,57531668~1694056,92274688~5642393,100663296~5306231,125829120~9697894,137849028~7170736,145565003~4271643,167991057~5531964,180722510~2055247,195156550~3672575,234881024~7706546,289406976~783403,293601280~6751275,301989888~5457554,319887642~2175374,361489784~7914531] 2024-09-21T10:34:09.253 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 7 diff [7016728~1370550,28558403~593635,35578395~2688472,39931377~204155,45048342~8105318,57756520~1469204,93118269~4798812,101882815~4086712,127714892~7812122,138318126~7793832,146239075~3597571,167991057~6561399,180722510~2055247,195156550~3672575,224340984~6971051,238706507~3881063,289443310~747069,294637427~5715128,303093367~4354075,319887642~2175374,322140464~6529613,361489784~7914531,369976939~7305065,400476606~3423466] 2024-09-21T10:34:09.253 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [7016728~1370550,28558403~593635,35578395~2688472,39931377~204155,45048342~8105318,57756520~1469204,93118269~4798812,101882815~4086712,127714892~7812122,138318126~7793832,146239075~3597571,167991057~6561399,180722510~2055247,195156550~3672575,224340984~6971051,238706507~3881063,289443310~747069,294637427~5715128,303093367~4354075,319887642~2175374,322140464~6529613,361489784~7914531,369976939~7305065,400476606~3423466] 2024-09-21T10:34:09.260 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [7016728~1370550,25165824~3986214,35578395~2688472,39931377~204155,41943040~11210620,57531668~1694056,92274688~5642393,100663296~5306231,125829120~9697894,137849028~11987618,167991057~6561399,180722510~2055247,195156550~3672575,224000747~7311288,234881024~7706546,289406976~783403,293601280~6751275,301989888~5457554,319887642~2175374,322140464~6529613,361489784~7914531,369976939~7305065,398458880~5441192] 2024-09-21T10:34:09.260 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 8 diff [2875569~6387626,16642670~3711712,28558403~593635,35578395~2688472,39931377~204155,45048342~8105318,57756520~1469204,93118269~4798812,101882815~4086712,127714892~7812122,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,180722510~2055247,195156550~3672575,224340984~6971051,238706507~3881063,254252427~9347451,289443310~747069,294637427~5715128,303093367~4354075,319887642~2175374,322140464~6529613,361489784~7914531,369976939~9230871,400476606~3423466] 2024-09-21T10:34:09.260 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~3711712,28558403~593635,35578395~2688472,39931377~204155,45048342~8105318,57756520~1469204,93118269~4798812,101882815~4086712,127714892~7812122,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,180722510~2055247,195156550~3672575,224340984~6971051,238706507~3881063,254252427~9347451,289443310~747069,294637427~5715128,303093367~4354075,319887642~2175374,322140464~6529613,361489784~7914531,369976939~9230871,400476606~3423466] 2024-09-21T10:34:09.265 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~7771470,25165824~3986214,35578395~2688472,39931377~204155,41943040~11210620,57531668~1694056,92274688~5642393,100663296~5306231,125829120~9697894,137849028~11987618,152039608~7712103,163577856~10974600,180722510~2055247,195156550~3672575,224000747~7311288,234881024~7706546,254252427~9347451,289406976~783403,293601280~6751275,301989888~5457554,319887642~2175374,322140464~6529613,361489784~7914531,369976939~9230871,398458880~5441192] 2024-09-21T10:34:09.265 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 9 diff [2875569~6387626,16642670~3711712,28558403~593635,35578395~2688472,39931377~204155,45048342~8105318,57756520~1469204,93118269~4798812,101882815~4086712,115995156~2434204,127714892~7812122,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,195156550~3672575,224340984~6971051,238706507~8545657,254252427~13677068,289443310~747069,294637427~5715128,303093367~4354075,317667912~11002165,343124038~1645752,361489784~7914531,369976939~9230871,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.265 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~3711712,28558403~593635,35578395~2688472,39931377~204155,45048342~8105318,57756520~1469204,93118269~4798812,101882815~4086712,115995156~2434204,127714892~7812122,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,195156550~3672575,224340984~6971051,238706507~8545657,254252427~13677068,289443310~747069,294637427~5715128,303093367~4354075,317667912~11002165,343124038~1645752,361489784~7914531,369976939~9230871,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.271 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~7771470,25165824~3986214,35578395~2688472,39931377~204155,41943040~11210620,57531668~1694056,92274688~5642393,100663296~5306231,113246208~5183152,125829120~9697894,137849028~11987618,152039608~7712103,163577856~15649170,180722510~2055247,195156550~3672575,224000747~7311288,234881024~12371140,254252427~13677068,289406976~783403,293601280~6751275,301989888~5457554,317667912~11002165,339738624~5031166,361489784~7914531,369976939~22663644,398458880~5441192] 2024-09-21T10:34:09.271 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 10 diff [2875569~6387626,16642670~3711712,28558403~593635,35578395~2688472,39931377~204155,45048342~8105318,57756520~1469204,93118269~4798812,98906260~7063267,114683657~4879263,127714892~7812122,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,195156550~3672575,207781564~1395640,224340984~6971051,238706507~8545657,254252427~13677068,269577554~11336387,289443310~747069,294637427~5715128,303093367~6669863,311582767~3753918,317667912~11002165,339701748~5068042,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.271 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~3711712,28558403~593635,35578395~2688472,39931377~204155,45048342~8105318,57756520~1469204,93118269~4798812,98906260~7063267,114683657~4879263,127714892~7812122,138318126~7793832,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,195156550~3672575,207781564~1395640,224340984~6971051,238706507~8545657,254252427~13677068,269577554~11336387,289443310~747069,294637427~5715128,303093367~6669863,311582767~3753918,317667912~11002165,339701748~5068042,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.277 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~7771470,25165824~3986214,35578395~2688472,39931377~204155,41943040~11210620,57531668~1694056,92274688~13694839,113246208~6316712,125829120~9697894,137849028~11987618,152039608~7712103,163577856~15649170,180722510~2055247,195156550~3672575,206063683~3113521,224000747~7311288,234881024~12371140,254252427~13677068,269577554~11336387,289406976~783403,293601280~6751275,301989888~7773342,311582767~3753918,317667912~11002165,335544320~9225470,361489784~31150799,398458880~5441192] 2024-09-21T10:34:09.277 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 11 diff [2875569~6387626,16642670~3711712,28558403~593635,31536075~6730792,39931377~204155,45048342~10301049,57756520~1469204,62918670~4568250,82908058~1860108,93118269~4798812,98906260~7063267,114683657~4879263,127714892~7812122,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,195156550~3672575,207781564~1395640,224340984~6971051,236839701~10412463,254252427~13677068,269577554~11336387,289443310~747069,294637427~15125803,311582767~17087310,339701748~5068042,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.277 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~3711712,28558403~593635,31536075~6730792,39931377~204155,45048342~10301049,57756520~1469204,62918670~4568250,82908058~1860108,93118269~4798812,98906260~7063267,114683657~4879263,127714892~7812122,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,167641738~6910718,175387176~3839850,180722510~2055247,195156550~3672575,207781564~1395640,224340984~6971051,236839701~10412463,254252427~13677068,269577554~11336387,289443310~747069,294637427~15125803,311582767~17087310,339701748~5068042,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.282 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~7771470,25165824~3986214,31536075~6730792,39931377~204155,41943040~13406351,57531668~1694056,62914560~4572360,82908058~1860108,92274688~13694839,113246208~6316712,125829120~9697894,137849028~11987618,152039608~8821599,163577856~15649170,180722510~2055247,195156550~3672575,206063683~3113521,224000747~7311288,234881024~12371140,254252427~13677068,269577554~11336387,289406976~783403,293601280~16161950,311582767~17087310,335544320~9225470,361489784~31150799,398458880~5441192] 2024-09-21T10:34:09.283 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 12 diff [2875569~6387626,16642670~3711712,27991252~10275615,39931377~204155,45048342~10301049,57756520~1469204,62918670~4568250,72069759~2339334,82908058~1860108,93118269~4798812,98906260~7063267,114683657~4879263,127714892~7812122,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,167641738~6910718,175387176~3839850,180722510~7393752,191528500~10608093,207781564~1395640,224340984~8144255,236839701~10412463,254252427~13677068,269577554~11336387,289443310~747069,294637427~34032650,339701748~5068042,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.283 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~3711712,27991252~10275615,39931377~204155,45048342~10301049,57756520~1469204,62918670~4568250,72069759~2339334,82908058~1860108,93118269~4798812,98906260~7063267,114683657~4879263,127714892~7812122,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,167641738~6910718,175387176~3839850,180722510~7393752,191528500~10608093,207781564~1395640,224340984~8144255,236839701~10412463,254252427~13677068,269577554~11336387,289443310~747069,294637427~34032650,339701748~5068042,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.288 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~7771470,25165824~13101043,39931377~204155,41943040~13406351,57531668~1694056,62914560~4572360,72069759~2339334,82908058~1860108,92274688~13694839,113246208~6316712,125829120~9697894,137849028~11987618,152039608~8821599,163577856~15649170,180722510~7393752,191528500~10608093,206063683~3113521,224000747~8484492,234881024~12371140,254252427~13677068,269577554~11336387,289406976~783403,293601280~35068797,335544320~9225470,361489784~31150799,398458880~5441192] 2024-09-21T10:34:09.288 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 13 diff [2875569~6387626,16642670~3711712,27991252~10275615,39931377~204155,45048342~12441594,57756520~1469204,62918670~4568250,72069759~2339334,82908058~1860108,93118269~4798812,98906260~7063267,114683657~4879263,127714892~7812122,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~1395640,224340984~8144255,236839701~14643462,254252427~13677068,269577554~11336387,289443310~747069,294637427~34032650,339701748~5068042,347289030~490774,361489784~17718026,381111195~1866659,384603168~8037415,394221093~9678979] 2024-09-21T10:34:09.288 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~3711712,27991252~10275615,39931377~204155,45048342~12441594,57756520~1469204,62918670~4568250,72069759~2339334,82908058~1860108,93118269~4798812,98906260~7063267,114683657~4879263,127714892~7812122,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~1395640,224340984~8144255,236839701~14643462,254252427~13677068,269577554~11336387,289443310~747069,294637427~34032650,339701748~5068042,347289030~490774,361489784~17718026,381111195~1866659,384603168~8037415,394221093~9678979] 2024-09-21T10:34:09.294 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~7771470,25165824~13101043,39931377~204155,41943040~15546896,57531668~1694056,62914560~4572360,72069759~2339334,82908058~1860108,92274688~13694839,113246208~6316712,125829120~9697894,137849028~11987618,152039608~8821599,163577856~15649170,180722510~7393752,191528500~10608093,206063683~3113521,224000747~8484492,234881024~16602139,254252427~13677068,269577554~11336387,289406976~783403,293601280~35068797,335544320~12235484,361489784~31150799,393702666~10197406] 2024-09-21T10:34:09.294 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 14 diff [2875569~6387626,16642670~4162967,27991252~10275615,39931377~204155,45048342~12441594,57756520~1469204,62918670~4568250,72069759~2339334,82908058~1860108,93118269~4798812,98906260~7063267,114683657~4879263,119655166~4147395,127714892~7812122,135872708~1754391,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~1395640,224340984~8144255,236839701~14643462,254252427~13677068,269577554~11336387,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,360853018~18354792,381111195~1866659,384603168~8037415,394221093~9678979] 2024-09-21T10:34:09.294 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~4162967,27991252~10275615,39931377~204155,45048342~12441594,57756520~1469204,62918670~4568250,72069759~2339334,82908058~1860108,93118269~4798812,98906260~7063267,114683657~4879263,119655166~4147395,127714892~7812122,135872708~1754391,138318126~7793832,146239075~3597571,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~1395640,224340984~8144255,236839701~14643462,254252427~13677068,269577554~11336387,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,360853018~18354792,381111195~1866659,384603168~8037415,394221093~9678979] 2024-09-21T10:34:09.300 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~8222725,25165824~13101043,39931377~204155,41943040~15546896,57531668~1694056,62914560~4572360,72069759~2339334,82908058~1860108,92274688~13694839,113246208~6316712,119655166~4147395,125829120~9697894,135872708~1754391,137849028~11987618,152039608~8821599,163577856~15649170,180722510~7393752,191528500~10608093,206063683~3113521,224000747~8484492,234881024~16602139,254252427~13677068,269577554~11336387,285212672~5425980,293601280~35068797,335544320~12235484,360853018~31787565,393702666~10197406] 2024-09-21T10:34:09.300 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 15 diff [2875569~6387626,16642670~4162967,27991252~10275615,39931377~204155,45048342~12441594,57756520~2066114,62918670~4568250,72069759~2339334,82908058~1860108,93118269~4798812,98906260~7063267,114683657~4879263,119655166~4147395,127714892~7812122,135872708~1754391,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~1395640,210226943~2559490,224340984~27142179,254252427~13677068,269577554~11336387,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,351227316~4338875,360853018~18354792,381111195~1866659,384505041~8135542,394221093~9678979] 2024-09-21T10:34:09.300 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~4162967,27991252~10275615,39931377~204155,45048342~12441594,57756520~2066114,62918670~4568250,72069759~2339334,82908058~1860108,93118269~4798812,98906260~7063267,114683657~4879263,119655166~4147395,127714892~7812122,135872708~1754391,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~1395640,210226943~2559490,224340984~27142179,254252427~13677068,269577554~11336387,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,351227316~4338875,360853018~18354792,381111195~1866659,384505041~8135542,394221093~9678979] 2024-09-21T10:34:09.306 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~8222725,25165824~13101043,39931377~204155,41943040~15546896,57531668~2290966,62914560~4572360,72069759~2339334,82908058~1860108,92274688~13694839,113246208~6316712,119655166~4147395,125829120~9697894,135872708~1754391,137849028~13551694,152039608~8821599,163577856~15649170,180722510~7393752,191528500~10608093,206063683~3113521,210226943~2559490,224000747~27482416,254252427~13677068,269577554~11336387,285212672~5425980,293601280~35068797,335544320~12235484,349721624~5844567,360853018~31787565,393702666~10197406] 2024-09-21T10:34:09.306 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 16 diff [2875569~6387626,16642670~4956869,25706100~12560767,39931377~204155,45048342~14774292,62918670~4568250,72069759~2339334,82908058~1860108,93118269~4798812,98522774~7446753,114683657~4879263,119655166~4147395,127714892~7812122,135872708~1754391,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~1395640,210226943~2559490,224340984~27142179,254252427~13677068,269577554~11336387,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,351227316~4338875,360853018~18354792,380931932~31586,381111195~1866659,384505041~8135542,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.306 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~4956869,25706100~12560767,39931377~204155,45048342~14774292,62918670~4568250,72069759~2339334,82908058~1860108,93118269~4798812,98522774~7446753,114683657~4879263,119655166~4147395,127714892~7812122,135872708~1754391,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~1395640,210226943~2559490,224340984~27142179,254252427~13677068,269577554~11336387,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,351227316~4338875,360853018~18354792,380931932~31586,381111195~1866659,384505041~8135542,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.311 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~9016627,25165824~13101043,39931377~204155,41943040~17879594,62914560~4572360,72069759~2339334,82908058~1860108,92274688~13694839,113246208~6316712,119655166~4147395,125829120~9697894,135872708~1754391,137849028~13551694,152039608~8821599,163577856~15649170,180722510~7393752,191528500~10608093,206063683~3113521,210226943~2559490,224000747~27482416,254252427~13677068,269577554~11336387,285212672~5425980,293601280~35068797,335544320~12235484,349721624~5844567,360853018~31787565,393702666~10197406,409717800~2000564] 2024-09-21T10:34:09.311 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 17 diff [2875569~6387626,15258652~7024612,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82908058~1860108,93118269~15999790,114683657~9118904,127714892~7812122,135872708~1754391,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~1395640,209930960~3307546,224340984~27142179,254252427~13677068,269577554~11336387,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,351227316~4338875,360853018~18354792,380931932~31586,381111195~1866659,384505041~8135542,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.311 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~7024612,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82908058~1860108,93118269~15999790,114683657~9118904,127714892~7812122,135872708~1754391,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~1395640,209930960~3307546,224340984~27142179,254252427~13677068,269577554~11336387,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,351227316~4338875,360853018~18354792,380931932~31586,381111195~1866659,384505041~8135542,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.317 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~9700352,25165824~13101043,39931377~204155,41943040~17879594,62914560~11494533,82908058~1860108,92274688~16843371,113246208~10556353,125829120~9697894,135872708~1754391,137849028~13551694,152039608~8821599,163577856~15649170,180722510~7393752,191528500~10608093,206063683~3113521,209930960~3307546,224000747~27482416,254252427~13677068,269577554~11336387,285212672~5425980,293601280~35068797,335544320~12235484,349721624~5844567,360853018~31787565,393702666~10197406,409717800~2000564] 2024-09-21T10:34:09.317 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 18 diff [2875569~6387626,15258652~7024612,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82908058~1860108,93118269~15999790,114683657~9118904,124591324~10935690,135872708~1754391,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~9355151,224340984~27142179,254252427~26661514,286147115~4491537,291766799~42181,292083899~1585781,294637427~34032650,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.317 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~7024612,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82908058~1860108,93118269~15999790,114683657~9118904,124591324~10935690,135872708~1754391,138318126~13082596,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~9355151,224340984~27142179,254252427~26661514,286147115~4491537,291766799~42181,292083899~1585781,294637427~34032650,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.323 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~9700352,25165824~13101043,39931377~204155,41943040~17879594,62914560~11494533,82908058~1860108,92274688~16843371,113246208~22280806,135872708~1754391,137849028~13551694,152039608~8821599,163577856~15649170,180722510~7393752,191528500~10608093,206063683~11073032,224000747~27482416,254252427~26661514,285212672~43457405,335544320~12235484,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:09.323 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 19 diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.323 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.329 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~11494533,82127504~5163005,92274688~16843371,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180722510~7393752,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:09.329 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 6 diff [28558403~593635,35578395~2688472,57756520~1469204,146239075~3597571,167991057~5531964,294637427~5715128,303093367~4354075,319887642~2175374,361489784~7914531] 2024-09-21T10:34:09.329 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [28558403~593635,35578395~2688472,57756520~1469204,146239075~3597571,167991057~5531964,294637427~5715128,303093367~4354075,319887642~2175374,361489784~7914531] 2024-09-21T10:34:09.335 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [25165824~3986214,35578395~2688472,57531668~1694056,145565003~4271643,167991057~5531964,293601280~6751275,301989888~5457554,319887642~2175374,361489784~7914531] 2024-09-21T10:34:09.335 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 7 diff [7016728~1370550,28558403~593635,35578395~2688472,39931377~204155,57756520~1469204,140524653~5587305,146239075~3597571,167991057~6561399,224340984~6971051,294637427~5715128,303093367~4354075,319887642~2175374,322140464~6529613,361489784~7914531,369976939~7305065,400476606~3423466] 2024-09-21T10:34:09.335 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [7016728~1370550,28558403~593635,35578395~2688472,39931377~204155,57756520~1469204,140524653~5587305,146239075~3597571,167991057~6561399,224340984~6971051,294637427~5715128,303093367~4354075,319887642~2175374,322140464~6529613,361489784~7914531,369976939~7305065,400476606~3423466] 2024-09-21T10:34:09.341 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [7016728~1370550,25165824~3986214,35578395~2688472,39931377~204155,57531668~1694056,140524653~9311993,167991057~6561399,224000747~7311288,293601280~6751275,301989888~5457554,319887642~2175374,322140464~6529613,361489784~7914531,369976939~7305065,398458880~5441192] 2024-09-21T10:34:09.342 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 8 diff [2875569~6387626,16642670~3711712,28558403~593635,35578395~2688472,39931377~204155,57756520~1469204,140524653~5587305,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,224340984~6971051,254252427~9347451,294637427~5715128,303093367~4354075,319887642~2175374,322140464~6529613,361489784~7914531,369976939~9230871,400476606~3423466] 2024-09-21T10:34:09.342 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~3711712,28558403~593635,35578395~2688472,39931377~204155,57756520~1469204,140524653~5587305,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,224340984~6971051,254252427~9347451,294637427~5715128,303093367~4354075,319887642~2175374,322140464~6529613,361489784~7914531,369976939~9230871,400476606~3423466] 2024-09-21T10:34:09.348 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~7771470,25165824~3986214,35578395~2688472,39931377~204155,57531668~1694056,140524653~9311993,152039608~7712103,163577856~10974600,224000747~7311288,254252427~9347451,293601280~6751275,301989888~5457554,319887642~2175374,322140464~6529613,361489784~7914531,369976939~9230871,398458880~5441192] 2024-09-21T10:34:09.348 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 9 diff [2875569~6387626,16642670~3711712,28558403~593635,35578395~2688472,39931377~204155,57756520~1469204,115995156~2434204,140524653~5587305,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,175387176~3839850,224340984~6971051,240673574~6578590,254252427~13677068,294637427~5715128,303093367~4354075,317667912~11002165,343124038~1645752,361489784~7914531,369976939~9230871,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.348 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~3711712,28558403~593635,35578395~2688472,39931377~204155,57756520~1469204,115995156~2434204,140524653~5587305,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,175387176~3839850,224340984~6971051,240673574~6578590,254252427~13677068,294637427~5715128,303093367~4354075,317667912~11002165,343124038~1645752,361489784~7914531,369976939~9230871,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.353 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~7771470,25165824~3986214,35578395~2688472,39931377~204155,57531668~1694056,113246208~5183152,140524653~9311993,152039608~7712103,163577856~15649170,224000747~7311288,240673574~6578590,254252427~13677068,293601280~6751275,301989888~5457554,317667912~11002165,339738624~5031166,361489784~7914531,369976939~22663644,398458880~5441192] 2024-09-21T10:34:09.353 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 10 diff [2875569~6387626,16642670~3711712,28558403~593635,35578395~2688472,39931377~204155,57756520~1469204,98906260~4781139,114683657~4879263,140524653~5587305,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,175387176~3839850,207781564~1395640,224340984~6971051,240673574~6578590,254252427~13677068,269577554~11336387,294637427~5715128,303093367~6669863,311582767~3753918,317667912~11002165,339701748~5068042,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.354 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~3711712,28558403~593635,35578395~2688472,39931377~204155,57756520~1469204,98906260~4781139,114683657~4879263,140524653~5587305,146239075~3597571,152039608~7712103,165324500~859426,167641738~6910718,175387176~3839850,207781564~1395640,224340984~6971051,240673574~6578590,254252427~13677068,269577554~11336387,294637427~5715128,303093367~6669863,311582767~3753918,317667912~11002165,339701748~5068042,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.359 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~7771470,25165824~3986214,35578395~2688472,39931377~204155,57531668~1694056,97917081~5770318,113246208~6316712,140524653~9311993,152039608~7712103,163577856~15649170,206063683~3113521,224000747~7311288,240673574~6578590,254252427~13677068,269577554~11336387,293601280~6751275,301989888~7773342,311582767~3753918,317667912~11002165,335544320~9225470,361489784~31150799,398458880~5441192] 2024-09-21T10:34:09.359 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 11 diff [2875569~6387626,16642670~3711712,28558403~593635,31536075~6730792,39931377~204155,49169063~6180328,57756520~1469204,62918670~4568250,82908058~1860108,98906260~6401933,114683657~4879263,140524653~5587305,146239075~3597571,152039608~8821599,165324500~859426,167641738~6910718,175387176~3839850,207781564~1395640,224340984~6971051,236839701~10412463,254252427~13677068,269577554~11336387,294637427~15125803,311582767~17087310,339701748~5068042,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.359 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~3711712,28558403~593635,31536075~6730792,39931377~204155,49169063~6180328,57756520~1469204,62918670~4568250,82908058~1860108,98906260~6401933,114683657~4879263,140524653~5587305,146239075~3597571,152039608~8821599,165324500~859426,167641738~6910718,175387176~3839850,207781564~1395640,224340984~6971051,236839701~10412463,254252427~13677068,269577554~11336387,294637427~15125803,311582767~17087310,339701748~5068042,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.365 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~7771470,25165824~3986214,31536075~6730792,39931377~204155,49169063~6180328,57531668~1694056,62914560~4572360,82908058~1860108,97917081~7391112,113246208~6316712,140524653~9311993,152039608~8821599,163577856~15649170,206063683~3113521,224000747~7311288,236839701~10412463,254252427~13677068,269577554~11336387,293601280~16161950,311582767~17087310,335544320~9225470,361489784~31150799,398458880~5441192] 2024-09-21T10:34:09.366 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 12 diff [2875569~6387626,16642670~3711712,27991252~10275615,39931377~204155,49169063~6180328,57756520~1469204,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,140524653~5587305,146239075~3597571,152039608~8821599,165324500~859426,167641738~6910718,175387176~3839850,182407418~5708844,191528500~10608093,207781564~1395640,224340984~8144255,236839701~10412463,254252427~13677068,269577554~11336387,294637427~34032650,339701748~5068042,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.366 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~3711712,27991252~10275615,39931377~204155,49169063~6180328,57756520~1469204,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,140524653~5587305,146239075~3597571,152039608~8821599,165324500~859426,167641738~6910718,175387176~3839850,182407418~5708844,191528500~10608093,207781564~1395640,224340984~8144255,236839701~10412463,254252427~13677068,269577554~11336387,294637427~34032650,339701748~5068042,361489784~17718026,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.372 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~7771470,25165824~13101043,39931377~204155,49169063~6180328,57531668~1694056,62914560~4572360,72069759~2339334,82908058~1860108,97917081~7391112,113246208~6316712,140524653~9311993,152039608~8821599,163577856~15649170,182407418~5708844,191528500~10608093,206063683~3113521,224000747~8484492,236839701~10412463,254252427~13677068,269577554~11336387,293601280~35068797,335544320~9225470,361489784~31150799,398458880~5441192] 2024-09-21T10:34:09.372 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 13 diff [2875569~6387626,16642670~3711712,27991252~10275615,39931377~204155,49169063~8320873,57756520~1469204,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,140524653~5587305,146239075~3597571,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,182407418~5708844,191528500~10608093,207781564~1395640,224340984~8144255,236839701~14643462,254252427~13677068,269577554~11336387,294637427~34032650,339701748~5068042,347289030~490774,361489784~17718026,381111195~1866659,384603168~8037415,394221093~9678979] 2024-09-21T10:34:09.372 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~3711712,27991252~10275615,39931377~204155,49169063~8320873,57756520~1469204,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,140524653~5587305,146239075~3597571,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,182407418~5708844,191528500~10608093,207781564~1395640,224340984~8144255,236839701~14643462,254252427~13677068,269577554~11336387,294637427~34032650,339701748~5068042,347289030~490774,361489784~17718026,381111195~1866659,384603168~8037415,394221093~9678979] 2024-09-21T10:34:09.378 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~7771470,25165824~13101043,39931377~204155,49169063~8320873,57531668~1694056,62914560~4572360,72069759~2339334,82908058~1860108,97917081~7391112,113246208~6316712,140524653~9311993,152039608~8821599,163577856~15649170,182407418~5708844,191528500~10608093,206063683~3113521,224000747~8484492,236839701~14643462,254252427~13677068,269577554~11336387,293601280~35068797,335544320~12235484,361489784~31150799,393702666~10197406] 2024-09-21T10:34:09.378 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 14 diff [2875569~6387626,16642670~4162967,27991252~10275615,39931377~204155,49169063~8320873,57756520~1469204,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,119655166~4147395,135872708~1754391,140524653~5587305,146239075~3597571,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,224340984~8144255,236839701~14643462,254252427~13677068,269577554~11336387,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,360853018~18354792,381111195~1866659,384603168~8037415,394221093~9678979] 2024-09-21T10:34:09.378 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~4162967,27991252~10275615,39931377~204155,49169063~8320873,57756520~1469204,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,119655166~4147395,135872708~1754391,140524653~5587305,146239075~3597571,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,224340984~8144255,236839701~14643462,254252427~13677068,269577554~11336387,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,360853018~18354792,381111195~1866659,384603168~8037415,394221093~9678979] 2024-09-21T10:34:09.383 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~8222725,25165824~13101043,39931377~204155,49169063~8320873,57531668~1694056,62914560~4572360,72069759~2339334,82908058~1860108,97917081~7391112,113246208~6316712,119655166~4147395,135872708~1754391,140524653~9311993,152039608~8821599,163577856~15649170,181624406~6491856,191528500~10608093,206063683~3113521,224000747~8484492,236839701~14643462,254252427~13677068,269577554~11336387,285212672~5425980,293601280~35068797,335544320~12235484,360853018~31787565,393702666~10197406] 2024-09-21T10:34:09.383 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 15 diff [2875569~6387626,16642670~4162967,27991252~10275615,39931377~204155,49169063~8320873,57756520~2066114,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,119655166~4147395,135872708~1754391,140524653~10876069,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,210226943~2559490,224340984~27142179,254252427~13677068,269577554~11336387,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,351227316~4338875,360853018~18354792,381111195~1866659,384505041~8135542,394221093~9678979] 2024-09-21T10:34:09.383 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~4162967,27991252~10275615,39931377~204155,49169063~8320873,57756520~2066114,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,119655166~4147395,135872708~1754391,140524653~10876069,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,210226943~2559490,224340984~27142179,254252427~13677068,269577554~11336387,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,351227316~4338875,360853018~18354792,381111195~1866659,384505041~8135542,394221093~9678979] 2024-09-21T10:34:09.389 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~8222725,25165824~13101043,39931377~204155,49169063~8320873,57531668~2290966,62914560~4572360,72069759~2339334,82908058~1860108,97917081~7391112,113246208~6316712,119655166~4147395,135872708~1754391,140524653~10876069,152039608~8821599,163577856~15649170,181624406~6491856,191528500~10608093,206063683~3113521,210226943~2559490,224000747~27482416,254252427~13677068,269577554~11336387,285212672~5425980,293601280~35068797,335544320~12235484,349721624~5844567,360853018~31787565,393702666~10197406] 2024-09-21T10:34:09.389 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 16 diff [2875569~6387626,16642670~4956869,25706100~12560767,39931377~204155,49169063~10653571,62918670~4568250,72069759~2339334,82908058~1860108,98522774~6785419,114683657~4879263,119655166~4147395,135872708~1754391,140524653~10876069,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,210226943~2559490,224340984~27142179,254252427~13677068,269577554~11336387,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,351227316~4338875,360853018~18354792,380931932~31586,381111195~1866659,384505041~8135542,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.390 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~4956869,25706100~12560767,39931377~204155,49169063~10653571,62918670~4568250,72069759~2339334,82908058~1860108,98522774~6785419,114683657~4879263,119655166~4147395,135872708~1754391,140524653~10876069,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,210226943~2559490,224340984~27142179,254252427~13677068,269577554~11336387,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,351227316~4338875,360853018~18354792,380931932~31586,381111195~1866659,384505041~8135542,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.396 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~9016627,25165824~13101043,39931377~204155,49169063~10653571,62914560~4572360,72069759~2339334,82908058~1860108,97917081~7391112,113246208~6316712,119655166~4147395,135872708~1754391,140524653~10876069,152039608~8821599,163577856~15649170,181624406~6491856,191528500~10608093,206063683~3113521,210226943~2559490,224000747~27482416,254252427~13677068,269577554~11336387,285212672~5425980,293601280~35068797,335544320~12235484,349721624~5844567,360853018~31787565,393702666~10197406,409717800~2000564] 2024-09-21T10:34:09.396 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 17 diff [2875569~6387626,15258652~7024612,25706100~12560767,39931377~204155,49169063~10653571,62918670~11490423,82908058~1860108,95398996~13719063,114683657~9118904,135872708~1754391,140524653~10876069,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,209930960~3307546,224340984~27142179,254252427~13677068,269577554~11336387,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,351227316~4338875,360853018~18354792,380931932~31586,381111195~1866659,384505041~8135542,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.396 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~7024612,25706100~12560767,39931377~204155,49169063~10653571,62918670~11490423,82908058~1860108,95398996~13719063,114683657~9118904,135872708~1754391,140524653~10876069,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,209930960~3307546,224340984~27142179,254252427~13677068,269577554~11336387,286147115~4491537,294637427~34032650,339701748~5068042,347289030~490774,351227316~4338875,360853018~18354792,380931932~31586,381111195~1866659,384505041~8135542,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.402 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~9700352,25165824~13101043,39931377~204155,49169063~10653571,62914560~11494533,82908058~1860108,95398996~13719063,113246208~10556353,135872708~1754391,140524653~10876069,152039608~8821599,163577856~15649170,181624406~6491856,191528500~10608093,206063683~3113521,209930960~3307546,224000747~27482416,254252427~13677068,269577554~11336387,285212672~5425980,293601280~35068797,335544320~12235484,349721624~5844567,360853018~31787565,393702666~10197406,409717800~2000564] 2024-09-21T10:34:09.402 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 18 diff [2875569~6387626,15258652~7024612,25706100~12560767,39931377~204155,49169063~10653571,62918670~11490423,82908058~1860108,95398996~13719063,114683657~9118904,124591324~5118882,135872708~1754391,140524653~10876069,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~9355151,224340984~27142179,254252427~26661514,286147115~4491537,291766799~42181,292083899~1585781,294637427~34032650,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.402 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~7024612,25706100~12560767,39931377~204155,49169063~10653571,62918670~11490423,82908058~1860108,95398996~13719063,114683657~9118904,124591324~5118882,135872708~1754391,140524653~10876069,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~9355151,224340984~27142179,254252427~26661514,286147115~4491537,291766799~42181,292083899~1585781,294637427~34032650,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.408 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~9700352,25165824~13101043,39931377~204155,49169063~10653571,62914560~11494533,82908058~1860108,95398996~13719063,113246208~16463998,135872708~1754391,140524653~10876069,152039608~8821599,163577856~15649170,181624406~6491856,191528500~10608093,206063683~11073032,224000747~27482416,254252427~26661514,285212672~43457405,335544320~12235484,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:09.408 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 19 diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.408 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.414 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,140524653~20336554,163577856~15649170,181624406~6491856,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:09.415 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to 7 diff [7016728~1370550,39931377~204155,140524653~5587305,172681089~1871367,224340984~6971051,322140464~6529613,365837885~120768,369976939~7305065,400476606~3423466] 2024-09-21T10:34:09.415 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [7016728~1370550,39931377~204155,140524653~5587305,172681089~1871367,224340984~6971051,322140464~6529613,365837885~120768,369976939~7305065,400476606~3423466] 2024-09-21T10:34:09.420 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [7016728~1370550,39931377~204155,140524653~5587305,172681089~1871367,224000747~7311288,322140464~6529613,365837885~120768,369976939~7305065,398458880~5441192] 2024-09-21T10:34:09.420 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to 8 diff [2875569~6387626,16642670~3711712,39931377~204155,140524653~5587305,152039608~7712103,165324500~859426,167641738~2360828,172681089~1871367,224340984~6971051,254252427~9347451,322140464~6529613,365837885~120768,369976939~9230871,400476606~3423466] 2024-09-21T10:34:09.420 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~3711712,39931377~204155,140524653~5587305,152039608~7712103,165324500~859426,167641738~2360828,172681089~1871367,224340984~6971051,254252427~9347451,322140464~6529613,365837885~120768,369976939~9230871,400476606~3423466] 2024-09-21T10:34:09.426 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~7771470,39931377~204155,140524653~5587305,152039608~7712103,163577856~6424710,172681089~1871367,224000747~7311288,254252427~9347451,322140464~6529613,365837885~120768,369976939~9230871,398458880~5441192] 2024-09-21T10:34:09.426 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to 9 diff [2875569~6387626,16642670~3711712,39931377~204155,115995156~2434204,140524653~5587305,152039608~7712103,165324500~859426,167641738~2360828,172681089~1871367,175387176~3839850,224340984~6971051,240673574~6578590,254252427~13677068,317667912~11002165,343124038~1645752,365837885~120768,369976939~9230871,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.427 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~3711712,39931377~204155,115995156~2434204,140524653~5587305,152039608~7712103,165324500~859426,167641738~2360828,172681089~1871367,175387176~3839850,224340984~6971051,240673574~6578590,254252427~13677068,317667912~11002165,343124038~1645752,365837885~120768,369976939~9230871,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.432 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~7771470,39931377~204155,113246208~5183152,140524653~5587305,152039608~7712103,163577856~6424710,172681089~6545937,224000747~7311288,240673574~6578590,254252427~13677068,317667912~11002165,339738624~5031166,365837885~120768,369976939~22663644,398458880~5441192] 2024-09-21T10:34:09.432 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to 10 diff [2875569~6387626,16642670~3711712,39931377~204155,98906260~4781139,114683657~4879263,140524653~5587305,152039608~7712103,165324500~859426,167641738~2360828,172681089~1871367,175387176~3839850,207781564~1395640,224340984~6971051,240673574~6578590,254252427~13677068,269577554~11336387,305173155~4590075,311582767~3753918,317667912~11002165,339701748~5068042,365837885~120768,367688090~11519720,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.432 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~3711712,39931377~204155,98906260~4781139,114683657~4879263,140524653~5587305,152039608~7712103,165324500~859426,167641738~2360828,172681089~1871367,175387176~3839850,207781564~1395640,224340984~6971051,240673574~6578590,254252427~13677068,269577554~11336387,305173155~4590075,311582767~3753918,317667912~11002165,339701748~5068042,365837885~120768,367688090~11519720,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.438 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~7771470,39931377~204155,97917081~5770318,113246208~6316712,140524653~5587305,152039608~7712103,163577856~6424710,172681089~6545937,206063683~3113521,224000747~7311288,240673574~6578590,254252427~13677068,269577554~11336387,305173155~4590075,311582767~3753918,317667912~11002165,335544320~9225470,365837885~120768,367688090~24952493,398458880~5441192] 2024-09-21T10:34:09.438 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to 11 diff [2875569~6387626,16642670~3711712,31536075~6482204,39931377~204155,49169063~6180328,62918670~4568250,82908058~1860108,98906260~6401933,114683657~4879263,140524653~5587305,152039608~8821599,165324500~859426,167641738~2360828,172681089~1871367,175387176~3839850,207781564~1395640,224340984~6971051,236839701~10412463,254252427~13677068,269577554~11336387,298351917~11411313,311582767~17087310,339701748~5068042,365837885~120768,367688090~11519720,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.438 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~3711712,31536075~6482204,39931377~204155,49169063~6180328,62918670~4568250,82908058~1860108,98906260~6401933,114683657~4879263,140524653~5587305,152039608~8821599,165324500~859426,167641738~2360828,172681089~1871367,175387176~3839850,207781564~1395640,224340984~6971051,236839701~10412463,254252427~13677068,269577554~11336387,298351917~11411313,311582767~17087310,339701748~5068042,365837885~120768,367688090~11519720,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.444 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~7771470,31536075~6482204,39931377~204155,49169063~6180328,62914560~4572360,82908058~1860108,97917081~7391112,113246208~6316712,140524653~5587305,152039608~8821599,163577856~6424710,172681089~6545937,206063683~3113521,224000747~7311288,236839701~10412463,254252427~13677068,269577554~11336387,298351917~11411313,311582767~17087310,335544320~9225470,365837885~120768,367688090~24952493,398458880~5441192] 2024-09-21T10:34:09.444 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to 12 diff [2875569~6387626,16642670~3711712,27991252~10027027,39931377~204155,49169063~6180328,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,140524653~5587305,152039608~8821599,165324500~859426,167641738~2360828,172681089~1871367,175387176~3839850,182407418~5708844,191528500~10608093,207781564~1395640,224340984~8144255,236839701~10412463,254252427~13677068,269577554~11336387,298351917~30318160,339701748~5068042,365837885~120768,367688090~11519720,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.444 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~3711712,27991252~10027027,39931377~204155,49169063~6180328,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,140524653~5587305,152039608~8821599,165324500~859426,167641738~2360828,172681089~1871367,175387176~3839850,182407418~5708844,191528500~10608093,207781564~1395640,224340984~8144255,236839701~10412463,254252427~13677068,269577554~11336387,298351917~30318160,339701748~5068042,365837885~120768,367688090~11519720,381111195~1866659,384603168~8037415,400476606~3423466] 2024-09-21T10:34:09.450 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~7771470,27991252~10027027,39931377~204155,49169063~6180328,62914560~4572360,72069759~2339334,82908058~1860108,97917081~7391112,113246208~6316712,140524653~5587305,152039608~8821599,163577856~6424710,172681089~6545937,182407418~5708844,191528500~10608093,206063683~3113521,224000747~8484492,236839701~10412463,254252427~13677068,269577554~11336387,298351917~30318160,335544320~9225470,365837885~120768,367688090~24952493,398458880~5441192] 2024-09-21T10:34:09.450 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to 13 diff [2875569~6387626,16642670~3711712,27991252~10027027,39931377~204155,49169063~8320873,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,140524653~5587305,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,182407418~5708844,191528500~10608093,207781564~1395640,224340984~8144255,236839701~14643462,254252427~13677068,269577554~11336387,298351917~30318160,339701748~5068042,347289030~490774,365837885~120768,367688090~11519720,381111195~1866659,384603168~8037415,394221093~9678979] 2024-09-21T10:34:09.450 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~3711712,27991252~10027027,39931377~204155,49169063~8320873,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,140524653~5587305,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,182407418~5708844,191528500~10608093,207781564~1395640,224340984~8144255,236839701~14643462,254252427~13677068,269577554~11336387,298351917~30318160,339701748~5068042,347289030~490774,365837885~120768,367688090~11519720,381111195~1866659,384603168~8037415,394221093~9678979] 2024-09-21T10:34:09.456 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~7771470,27991252~10027027,39931377~204155,49169063~8320873,62914560~4572360,72069759~2339334,82908058~1860108,97917081~7391112,113246208~6316712,140524653~5587305,152039608~8821599,163577856~15649170,182407418~5708844,191528500~10608093,206063683~3113521,224000747~8484492,236839701~14643462,254252427~13677068,269577554~11336387,298351917~30318160,335544320~12235484,365837885~120768,367688090~24952493,393702666~10197406] 2024-09-21T10:34:09.456 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to 14 diff [2875569~6387626,16642670~4162967,27991252~10027027,39931377~204155,49169063~8320873,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,119655166~4147395,135872708~1754391,140524653~5587305,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,224340984~8144255,236839701~14643462,254252427~13677068,269577554~11336387,286147115~4491537,298351917~30318160,339701748~5068042,347289030~490774,360853018~1275580,365837885~120768,367688090~11519720,381111195~1866659,384603168~8037415,394221093~9678979] 2024-09-21T10:34:09.456 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~4162967,27991252~10027027,39931377~204155,49169063~8320873,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,119655166~4147395,135872708~1754391,140524653~5587305,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,224340984~8144255,236839701~14643462,254252427~13677068,269577554~11336387,286147115~4491537,298351917~30318160,339701748~5068042,347289030~490774,360853018~1275580,365837885~120768,367688090~11519720,381111195~1866659,384603168~8037415,394221093~9678979] 2024-09-21T10:34:09.463 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~8222725,27991252~10027027,39931377~204155,49169063~8320873,62914560~4572360,72069759~2339334,82908058~1860108,97917081~7391112,113246208~6316712,119655166~4147395,135872708~1754391,140524653~5587305,152039608~8821599,163577856~15649170,181624406~6491856,191528500~10608093,206063683~3113521,224000747~8484492,236839701~14643462,254252427~13677068,269577554~11336387,285212672~5425980,298351917~30318160,335544320~12235484,360853018~1275580,365837885~120768,367688090~24952493,393702666~10197406] 2024-09-21T10:34:09.463 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to 15 diff [2875569~6387626,16642670~4162967,27991252~10027027,39931377~204155,49169063~8320873,58639820~1182814,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,119655166~4147395,135872708~1754391,140524653~10876069,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,210226943~2559490,224340984~27142179,254252427~13677068,269577554~11336387,286147115~4491537,298351917~30318160,339701748~5068042,347289030~490774,351227316~4338875,360853018~1275580,363514600~15693210,381111195~1866659,384505041~8135542,394221093~9678979] 2024-09-21T10:34:09.463 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~4162967,27991252~10027027,39931377~204155,49169063~8320873,58639820~1182814,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,119655166~4147395,135872708~1754391,140524653~10876069,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,210226943~2559490,224340984~27142179,254252427~13677068,269577554~11336387,286147115~4491537,298351917~30318160,339701748~5068042,347289030~490774,351227316~4338875,360853018~1275580,363514600~15693210,381111195~1866659,384505041~8135542,394221093~9678979] 2024-09-21T10:34:09.469 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~8222725,27991252~10027027,39931377~204155,49169063~8320873,58639820~1182814,62914560~4572360,72069759~2339334,82908058~1860108,97917081~7391112,113246208~6316712,119655166~4147395,135872708~1754391,140524653~10876069,152039608~8821599,163577856~15649170,181624406~6491856,191528500~10608093,206063683~3113521,210226943~2559490,224000747~27482416,254252427~13677068,269577554~11336387,285212672~5425980,298351917~30318160,335544320~12235484,349721624~5844567,360853018~1275580,363514600~29125983,393702666~10197406] 2024-09-21T10:34:09.469 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to 16 diff [2875569~6387626,16642670~4956869,25706100~12312179,39931377~204155,49169063~10653571,62918670~4568250,72069759~2339334,82908058~1860108,98522774~6785419,114683657~4879263,119655166~4147395,135872708~1754391,140524653~10876069,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,210226943~2559490,224340984~27142179,254252427~13677068,269577554~11336387,286147115~4491537,298351917~30318160,339701748~5068042,347289030~490774,351227316~4338875,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~8135542,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.469 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~4956869,25706100~12312179,39931377~204155,49169063~10653571,62918670~4568250,72069759~2339334,82908058~1860108,98522774~6785419,114683657~4879263,119655166~4147395,135872708~1754391,140524653~10876069,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,210226943~2559490,224340984~27142179,254252427~13677068,269577554~11336387,286147115~4491537,298351917~30318160,339701748~5068042,347289030~490774,351227316~4338875,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~8135542,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.476 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~9016627,25706100~12312179,39931377~204155,49169063~10653571,62914560~4572360,72069759~2339334,82908058~1860108,97917081~7391112,113246208~6316712,119655166~4147395,135872708~1754391,140524653~10876069,152039608~8821599,163577856~15649170,181624406~6491856,191528500~10608093,206063683~3113521,210226943~2559490,224000747~27482416,254252427~13677068,269577554~11336387,285212672~5425980,298351917~30318160,335544320~12235484,349721624~5844567,360853018~1275580,363514600~29125983,393702666~10197406,409717800~2000564] 2024-09-21T10:34:09.476 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to 17 diff [2875569~6387626,15258652~7024612,25706100~12312179,39931377~204155,49169063~10653571,62918670~11490423,82908058~1860108,95398996~13719063,114683657~9118904,135872708~1754391,140524653~10876069,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,209930960~3307546,224340984~27142179,254252427~13677068,269577554~11336387,286147115~4491537,298351917~30318160,339701748~5068042,347289030~490774,351227316~4338875,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~8135542,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.476 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~7024612,25706100~12312179,39931377~204155,49169063~10653571,62918670~11490423,82908058~1860108,95398996~13719063,114683657~9118904,135872708~1754391,140524653~10876069,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,209930960~3307546,224340984~27142179,254252427~13677068,269577554~11336387,286147115~4491537,298351917~30318160,339701748~5068042,347289030~490774,351227316~4338875,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~8135542,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.482 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~9700352,25706100~12312179,39931377~204155,49169063~10653571,62914560~11494533,82908058~1860108,95398996~13719063,113246208~10556353,135872708~1754391,140524653~10876069,152039608~8821599,163577856~15649170,181624406~6491856,191528500~10608093,206063683~3113521,209930960~3307546,224000747~27482416,254252427~13677068,269577554~11336387,285212672~5425980,298351917~30318160,335544320~12235484,349721624~5844567,360853018~1275580,363514600~29125983,393702666~10197406,409717800~2000564] 2024-09-21T10:34:09.482 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to 18 diff [2875569~6387626,15258652~7024612,25706100~12312179,39931377~204155,49169063~10653571,62918670~11490423,82908058~1860108,95398996~13719063,114683657~9118904,124591324~5118882,135872708~1754391,140524653~10876069,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~9355151,224340984~27142179,254252427~26661514,286147115~4491537,291766799~42181,292083899~1585781,298351917~30318160,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.482 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~7024612,25706100~12312179,39931377~204155,49169063~10653571,62918670~11490423,82908058~1860108,95398996~13719063,114683657~9118904,124591324~5118882,135872708~1754391,140524653~10876069,152039608~8821599,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~9355151,224340984~27142179,254252427~26661514,286147115~4491537,291766799~42181,292083899~1585781,298351917~30318160,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.487 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~9700352,25706100~12312179,39931377~204155,49169063~10653571,62914560~11494533,82908058~1860108,95398996~13719063,113246208~16463998,135872708~1754391,140524653~10876069,152039608~8821599,163577856~15649170,181624406~6491856,191528500~10608093,206063683~11073032,224000747~27482416,254252427~26661514,285212672~8457008,298351917~30318160,335544320~12235484,349721624~8063397,360853018~1275580,363514600~40385472,409717800~2000564] 2024-09-21T10:34:09.487 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to 19 diff [2875569~6387626,15258652~8431860,25706100~12312179,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~30318160,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.487 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12312179,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~30318160,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:09.493 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25706100~12312179,39931377~204155,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,140524653~20336554,163577856~15649170,181624406~6491856,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~12651312,298351917~30318160,331350016~16429788,349721624~8063397,360853018~1275580,363514600~40385472,409717800~2000564] 2024-09-21T10:34:09.493 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to 8 diff [2875569~6387626,16642670~3711712,152039608~7712103,165324500~859426,167641738~2360828,254252427~9347451,373355742~5852068] 2024-09-21T10:34:09.493 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~3711712,152039608~7712103,165324500~859426,167641738~2360828,254252427~9347451,373355742~5852068] 2024-09-21T10:34:09.498 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~7771470,152039608~7712103,163577856~6424710,254252427~9347451,373355742~5852068] 2024-09-21T10:34:09.498 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to 9 diff [2875569~6387626,16642670~3711712,115995156~2434204,152039608~7712103,165324500~859426,167641738~2360828,175387176~3839850,240673574~6578590,254252427~13677068,317667912~4641221,343124038~1645752,373355742~5852068,381111195~1866659,384603168~8037415] 2024-09-21T10:34:09.498 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~3711712,115995156~2434204,152039608~7712103,165324500~859426,167641738~2360828,175387176~3839850,240673574~6578590,254252427~13677068,317667912~4641221,343124038~1645752,373355742~5852068,381111195~1866659,384603168~8037415] 2024-09-21T10:34:09.504 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~7771470,113246208~5183152,152039608~7712103,163577856~6424710,174552456~4674570,240673574~6578590,254252427~13677068,317667912~4641221,339738624~5031166,373355742~19284841] 2024-09-21T10:34:09.504 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to 10 diff [2875569~6387626,16642670~3711712,98906260~4781139,114683657~4879263,152039608~7712103,165324500~859426,167641738~2360828,175387176~3839850,207781564~1395640,240673574~6578590,254252427~13677068,269577554~11336387,305173155~4590075,311582767~3753918,317667912~4641221,339701748~5068042,367688090~3556220,373355742~5852068,381111195~1866659,384603168~8037415] 2024-09-21T10:34:09.504 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~3711712,98906260~4781139,114683657~4879263,152039608~7712103,165324500~859426,167641738~2360828,175387176~3839850,207781564~1395640,240673574~6578590,254252427~13677068,269577554~11336387,305173155~4590075,311582767~3753918,317667912~4641221,339701748~5068042,367688090~3556220,373355742~5852068,381111195~1866659,384603168~8037415] 2024-09-21T10:34:09.510 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~7771470,97917081~5770318,113246208~6316712,152039608~7712103,163577856~6424710,174552456~4674570,206063683~3113521,240673574~6578590,254252427~13677068,269577554~11336387,305173155~4590075,311582767~3753918,317667912~4641221,335544320~9225470,367688090~3556220,373355742~19284841] 2024-09-21T10:34:09.510 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to 11 diff [2875569~6387626,16642670~3711712,31536075~6482204,49169063~6180328,62918670~4568250,82908058~1860108,98906260~6401933,114683657~4879263,152039608~8821599,165324500~859426,167641738~2360828,175387176~3839850,207781564~1395640,236839701~10412463,254252427~13677068,269577554~11336387,298351917~11411313,311582767~10726366,339701748~5068042,367688090~3556220,373355742~5852068,381111195~1866659,384603168~8037415] 2024-09-21T10:34:09.510 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~3711712,31536075~6482204,49169063~6180328,62918670~4568250,82908058~1860108,98906260~6401933,114683657~4879263,152039608~8821599,165324500~859426,167641738~2360828,175387176~3839850,207781564~1395640,236839701~10412463,254252427~13677068,269577554~11336387,298351917~11411313,311582767~10726366,339701748~5068042,367688090~3556220,373355742~5852068,381111195~1866659,384603168~8037415] 2024-09-21T10:34:09.515 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~7771470,31536075~6482204,49169063~6180328,62914560~4572360,82908058~1860108,97917081~7391112,113246208~6316712,152039608~8821599,163577856~6424710,174552456~4674570,206063683~3113521,236839701~10412463,254252427~13677068,269577554~11336387,298351917~11411313,311582767~10726366,335544320~9225470,367688090~3556220,373355742~19284841] 2024-09-21T10:34:09.515 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to 12 diff [2875569~6387626,16642670~3711712,27991252~10027027,49169063~6180328,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,152039608~8821599,165324500~859426,167641738~2360828,175387176~3839850,182407418~5708844,191528500~10608093,207781564~1395640,224814781~7670458,236839701~10412463,254252427~13677068,269577554~11336387,298351917~23957216,339701748~5068042,367688090~3556220,373355742~5852068,381111195~1866659,384603168~8037415] 2024-09-21T10:34:09.515 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~3711712,27991252~10027027,49169063~6180328,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,152039608~8821599,165324500~859426,167641738~2360828,175387176~3839850,182407418~5708844,191528500~10608093,207781564~1395640,224814781~7670458,236839701~10412463,254252427~13677068,269577554~11336387,298351917~23957216,339701748~5068042,367688090~3556220,373355742~5852068,381111195~1866659,384603168~8037415] 2024-09-21T10:34:09.520 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~7771470,27991252~10027027,49169063~6180328,62914560~4572360,72069759~2339334,82908058~1860108,97917081~7391112,113246208~6316712,152039608~8821599,163577856~6424710,174552456~4674570,182407418~5708844,191528500~10608093,206063683~3113521,224814781~7670458,236839701~10412463,254252427~13677068,269577554~11336387,298351917~23957216,335544320~9225470,367688090~3556220,373355742~19284841] 2024-09-21T10:34:09.521 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to 13 diff [2875569~6387626,16642670~3711712,27991252~10027027,49169063~8320873,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,152039608~8821599,165324500~859426,166204955~7344626,175217960~4009066,182407418~5708844,191528500~10608093,207781564~1395640,224814781~7670458,236839701~14643462,254252427~13677068,269577554~11336387,298351917~23957216,339701748~5068042,347289030~490774,367688090~3556220,373355742~5852068,381111195~1866659,384603168~8037415,394221093~8158476] 2024-09-21T10:34:09.521 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~3711712,27991252~10027027,49169063~8320873,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,152039608~8821599,165324500~859426,166204955~7344626,175217960~4009066,182407418~5708844,191528500~10608093,207781564~1395640,224814781~7670458,236839701~14643462,254252427~13677068,269577554~11336387,298351917~23957216,339701748~5068042,347289030~490774,367688090~3556220,373355742~5852068,381111195~1866659,384603168~8037415,394221093~8158476] 2024-09-21T10:34:09.526 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~7771470,27991252~10027027,49169063~8320873,62914560~4572360,72069759~2339334,82908058~1860108,97917081~7391112,113246208~6316712,152039608~8821599,163577856~9971725,174552456~4674570,182407418~5708844,191528500~10608093,206063683~3113521,224814781~7670458,236839701~14643462,254252427~13677068,269577554~11336387,298351917~23957216,335544320~12235484,367688090~3556220,373355742~19284841,393702666~8676903] 2024-09-21T10:34:09.526 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to 14 diff [2875569~6387626,16642670~4162967,27991252~10027027,49169063~8320873,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,119655166~4147395,135872708~1754391,152039608~8821599,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,224814781~7670458,236839701~14643462,254252427~13677068,269577554~11336387,286147115~4491537,298351917~23957216,339701748~5068042,347289030~490774,360853018~1275580,367688090~3556220,373355742~5852068,381111195~1866659,384603168~8037415,394221093~8158476] 2024-09-21T10:34:09.526 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~4162967,27991252~10027027,49169063~8320873,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,119655166~4147395,135872708~1754391,152039608~8821599,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,224814781~7670458,236839701~14643462,254252427~13677068,269577554~11336387,286147115~4491537,298351917~23957216,339701748~5068042,347289030~490774,360853018~1275580,367688090~3556220,373355742~5852068,381111195~1866659,384603168~8037415,394221093~8158476] 2024-09-21T10:34:09.532 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~8222725,27991252~10027027,49169063~8320873,62914560~4572360,72069759~2339334,82908058~1860108,97917081~7391112,113246208~6316712,119655166~4147395,135872708~1754391,152039608~8821599,163577856~9971725,174552456~4674570,181624406~6491856,191528500~10608093,206063683~3113521,224814781~7670458,236839701~14643462,254252427~13677068,269577554~11336387,285212672~5425980,298351917~23957216,335544320~12235484,360853018~1275580,367688090~3556220,373355742~19284841,393702666~8676903] 2024-09-21T10:34:09.532 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to 15 diff [2875569~6387626,16642670~4162967,27991252~10027027,49169063~8320873,58639820~1182814,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,119655166~4147395,135872708~1754391,144777497~6623225,152039608~8821599,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,210226943~2559490,224814781~26668382,254252427~13677068,269577554~11336387,286147115~4491537,298351917~23957216,339701748~5068042,347289030~490774,351227316~4338875,360853018~1275580,363514600~7729710,373355742~5852068,381111195~1866659,384505041~8135542,394221093~8158476] 2024-09-21T10:34:09.532 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~4162967,27991252~10027027,49169063~8320873,58639820~1182814,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,119655166~4147395,135872708~1754391,144777497~6623225,152039608~8821599,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,210226943~2559490,224814781~26668382,254252427~13677068,269577554~11336387,286147115~4491537,298351917~23957216,339701748~5068042,347289030~490774,351227316~4338875,360853018~1275580,363514600~7729710,373355742~5852068,381111195~1866659,384505041~8135542,394221093~8158476] 2024-09-21T10:34:09.537 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~8222725,27991252~10027027,49169063~8320873,58639820~1182814,62914560~4572360,72069759~2339334,82908058~1860108,97917081~7391112,113246208~6316712,119655166~4147395,135872708~1754391,144777497~6623225,152039608~8821599,163577856~9971725,174552456~4674570,181624406~6491856,191528500~10608093,206063683~3113521,210226943~2559490,224814781~26668382,254252427~13677068,269577554~11336387,285212672~5425980,298351917~23957216,335544320~12235484,349721624~5844567,360853018~1275580,363514600~7729710,373355742~19284841,393702666~8676903] 2024-09-21T10:34:09.537 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to 16 diff [2875569~6387626,16642670~4956869,25706100~12312179,49169063~10653571,62918670~4568250,72069759~2339334,82908058~1860108,98522774~6785419,114683657~4879263,119655166~4147395,135872708~1754391,144777497~6623225,152039608~8821599,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,210226943~2559490,224814781~26668382,254252427~13677068,269577554~11336387,286147115~4491537,298351917~23957216,339701748~5068042,347289030~490774,351227316~4338875,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~8135542,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.537 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,16642670~4956869,25706100~12312179,49169063~10653571,62918670~4568250,72069759~2339334,82908058~1860108,98522774~6785419,114683657~4879263,119655166~4147395,135872708~1754391,144777497~6623225,152039608~8821599,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,210226943~2559490,224814781~26668382,254252427~13677068,269577554~11336387,286147115~4491537,298351917~23957216,339701748~5068042,347289030~490774,351227316~4338875,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~8135542,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.543 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~9016627,25706100~12312179,49169063~10653571,62914560~4572360,72069759~2339334,82908058~1860108,97917081~7391112,113246208~6316712,119655166~4147395,135872708~1754391,144777497~6623225,152039608~8821599,163577856~9971725,174552456~4674570,181624406~6491856,191528500~10608093,206063683~3113521,210226943~2559490,224814781~26668382,254252427~13677068,269577554~11336387,285212672~5425980,298351917~23957216,335544320~12235484,349721624~5844567,360853018~1275580,363514600~7729710,373355742~19284841,393702666~8676903,409717800~2000564] 2024-09-21T10:34:09.543 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to 17 diff [2875569~6387626,15258652~7024612,25706100~12312179,49169063~10653571,62918670~11490423,82908058~1860108,95398996~13719063,114683657~9118904,135872708~1754391,144777497~6623225,152039608~8821599,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,209930960~3307546,224814781~26668382,254252427~13677068,269577554~11336387,286147115~4491537,298351917~23957216,339701748~5068042,347289030~490774,351227316~4338875,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~8135542,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.543 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~7024612,25706100~12312179,49169063~10653571,62918670~11490423,82908058~1860108,95398996~13719063,114683657~9118904,135872708~1754391,144777497~6623225,152039608~8821599,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,209930960~3307546,224814781~26668382,254252427~13677068,269577554~11336387,286147115~4491537,298351917~23957216,339701748~5068042,347289030~490774,351227316~4338875,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~8135542,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.548 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~9700352,25706100~12312179,49169063~10653571,62914560~11494533,82908058~1860108,95398996~13719063,113246208~10556353,135872708~1754391,144777497~6623225,152039608~8821599,163577856~9971725,174552456~4674570,181624406~6491856,191528500~10608093,206063683~3113521,209930960~3307546,224814781~26668382,254252427~13677068,269577554~11336387,285212672~5425980,298351917~23957216,335544320~12235484,349721624~5844567,360853018~1275580,363514600~7729710,373355742~19284841,393702666~8676903,409717800~2000564] 2024-09-21T10:34:09.548 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to 18 diff [2875569~6387626,15258652~7024612,25706100~12312179,49169063~10653571,62918670~11490423,82908058~1860108,95398996~13719063,114683657~9118904,124591324~5118882,135872708~1754391,144777497~6623225,152039608~8821599,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~9355151,224814781~26668382,254252427~26661514,286147115~4491537,291766799~42181,292083899~1585781,298351917~23957216,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.549 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~7024612,25706100~12312179,49169063~10653571,62918670~11490423,82908058~1860108,95398996~13719063,114683657~9118904,124591324~5118882,135872708~1754391,144777497~6623225,152039608~8821599,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~9355151,224814781~26668382,254252427~26661514,286147115~4491537,291766799~42181,292083899~1585781,298351917~23957216,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.554 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~9700352,25706100~12312179,49169063~10653571,62914560~11494533,82908058~1860108,95398996~13719063,113246208~16463998,135872708~1754391,144777497~6623225,152039608~8821599,163577856~9971725,174552456~4674570,181624406~6491856,191528500~10608093,206063683~11073032,224814781~26668382,254252427~26661514,285212672~8457008,298351917~23957216,335544320~12235484,349721624~8063397,360853018~1275580,363514600~7729710,373355742~29023827,409717800~2000564] 2024-09-21T10:34:09.554 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to 19 diff [2875569~6387626,15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~16083710,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.554 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~16083710,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.559 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,144777497~16083710,163577856~9971725,174552456~4674570,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,254252427~26661514,281018368~12651312,298351917~23957216,331350016~16429788,349721624~8063397,360853018~1275580,363514600~7729710,373355742~29023827,409717800~2000564] 2024-09-21T10:34:09.560 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to 9 diff [115995156~2434204,152797416~596734,175387176~3839850,240673574~6578590,259644006~8285489,317667912~4641221,343124038~1645752,381111195~1866659,384603168~8037415] 2024-09-21T10:34:09.560 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [115995156~2434204,152797416~596734,175387176~3839850,240673574~6578590,259644006~8285489,317667912~4641221,343124038~1645752,381111195~1866659,384603168~8037415] 2024-09-21T10:34:09.565 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [113246208~5183152,152797416~596734,174552456~4674570,240673574~6578590,259644006~8285489,317667912~4641221,339738624~5031166,379207810~13432773] 2024-09-21T10:34:09.565 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to 10 diff [98906260~4781139,114683657~4879263,152797416~596734,175387176~3839850,207781564~1395640,240673574~6578590,259644006~8285489,269577554~11336387,305173155~4590075,311582767~3753918,317667912~4641221,339701748~5068042,367688090~3556220,376984253~2159000,381111195~1866659,384603168~8037415] 2024-09-21T10:34:09.565 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [98906260~4781139,114683657~4879263,152797416~596734,175387176~3839850,207781564~1395640,240673574~6578590,259644006~8285489,269577554~11336387,305173155~4590075,311582767~3753918,317667912~4641221,339701748~5068042,367688090~3556220,376984253~2159000,381111195~1866659,384603168~8037415] 2024-09-21T10:34:09.570 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [97917081~5770318,113246208~6316712,152797416~596734,174552456~4674570,206063683~3113521,240673574~6578590,259644006~8285489,269577554~11336387,305173155~4590075,311582767~3753918,317667912~4641221,335544320~9225470,367688090~3556220,376984253~2159000,379207810~13432773] 2024-09-21T10:34:09.571 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to 11 diff [31536075~6482204,49169063~6180328,62918670~4568250,82908058~1860108,98906260~6401933,114683657~4879263,152797416~596734,157423190~3438017,175387176~3839850,207781564~1395640,236839701~10412463,259644006~8285489,269577554~11336387,298351917~11411313,311582767~10726366,339701748~5068042,367688090~3556220,376984253~2159000,381111195~1866659,384603168~8037415] 2024-09-21T10:34:09.571 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [31536075~6482204,49169063~6180328,62918670~4568250,82908058~1860108,98906260~6401933,114683657~4879263,152797416~596734,157423190~3438017,175387176~3839850,207781564~1395640,236839701~10412463,259644006~8285489,269577554~11336387,298351917~11411313,311582767~10726366,339701748~5068042,367688090~3556220,376984253~2159000,381111195~1866659,384603168~8037415] 2024-09-21T10:34:09.576 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [31536075~6482204,49169063~6180328,62914560~4572360,82908058~1860108,97917081~7391112,113246208~6316712,152797416~596734,157423190~3438017,174552456~4674570,206063683~3113521,236839701~10412463,259644006~8285489,269577554~11336387,298351917~11411313,311582767~10726366,335544320~9225470,367688090~3556220,376984253~2159000,379207810~13432773] 2024-09-21T10:34:09.577 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to 12 diff [27991252~10027027,49169063~6180328,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,152797416~596734,157423190~3438017,175387176~3839850,182407418~5708844,191528500~10608093,207781564~1395640,224814781~7670458,236839701~10412463,259644006~8285489,269577554~11336387,298351917~23957216,339701748~5068042,367688090~3556220,376984253~2159000,381111195~1866659,384603168~8037415] 2024-09-21T10:34:09.577 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [27991252~10027027,49169063~6180328,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,152797416~596734,157423190~3438017,175387176~3839850,182407418~5708844,191528500~10608093,207781564~1395640,224814781~7670458,236839701~10412463,259644006~8285489,269577554~11336387,298351917~23957216,339701748~5068042,367688090~3556220,376984253~2159000,381111195~1866659,384603168~8037415] 2024-09-21T10:34:09.582 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [27991252~10027027,49169063~6180328,62914560~4572360,72069759~2339334,82908058~1860108,97917081~7391112,113246208~6316712,152797416~596734,157423190~3438017,174552456~4674570,182407418~5708844,191528500~10608093,206063683~3113521,224814781~7670458,236839701~10412463,259644006~8285489,269577554~11336387,298351917~23957216,335544320~9225470,367688090~3556220,376984253~2159000,379207810~13432773] 2024-09-21T10:34:09.582 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to 13 diff [27991252~10027027,49169063~8320873,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,182407418~5708844,191528500~10608093,207781564~1395640,224814781~7670458,236839701~14643462,259644006~8285489,269577554~11336387,298351917~23957216,339701748~5068042,347289030~490774,367688090~3556220,376984253~2159000,381111195~1866659,384603168~8037415,394221093~8158476] 2024-09-21T10:34:09.582 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [27991252~10027027,49169063~8320873,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,182407418~5708844,191528500~10608093,207781564~1395640,224814781~7670458,236839701~14643462,259644006~8285489,269577554~11336387,298351917~23957216,339701748~5068042,347289030~490774,367688090~3556220,376984253~2159000,381111195~1866659,384603168~8037415,394221093~8158476] 2024-09-21T10:34:09.588 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [27991252~10027027,49169063~8320873,62914560~4572360,72069759~2339334,82908058~1860108,97917081~7391112,113246208~6316712,152797416~596734,157423190~3438017,166204955~7344626,174552456~4674570,182407418~5708844,191528500~10608093,206063683~3113521,224814781~7670458,236839701~14643462,259644006~8285489,269577554~11336387,298351917~23957216,335544320~12235484,367688090~3556220,376984253~2159000,379207810~13432773,393702666~8676903] 2024-09-21T10:34:09.588 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to 14 diff [20106162~699475,27991252~10027027,49169063~8320873,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,119655166~4147395,135872708~1754391,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,224814781~7670458,236839701~14643462,259644006~8285489,269577554~11336387,286147115~4491537,298351917~23957216,339701748~5068042,347289030~490774,360853018~1275580,367688090~3556220,376984253~2159000,381111195~1866659,384603168~8037415,394221093~8158476] 2024-09-21T10:34:09.588 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [20106162~699475,27991252~10027027,49169063~8320873,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,119655166~4147395,135872708~1754391,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,224814781~7670458,236839701~14643462,259644006~8285489,269577554~11336387,286147115~4491537,298351917~23957216,339701748~5068042,347289030~490774,360853018~1275580,367688090~3556220,376984253~2159000,381111195~1866659,384603168~8037415,394221093~8158476] 2024-09-21T10:34:09.593 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [20106162~699475,27991252~10027027,49169063~8320873,62914560~4572360,72069759~2339334,82908058~1860108,97917081~7391112,113246208~6316712,119655166~4147395,135872708~1754391,152797416~596734,157423190~3438017,166204955~7344626,174552456~4674570,181624406~6491856,191528500~10608093,206063683~3113521,224814781~7670458,236839701~14643462,259644006~8285489,269577554~11336387,285212672~5425980,298351917~23957216,335544320~12235484,360853018~1275580,367688090~3556220,376984253~2159000,379207810~13432773,393702666~8676903] 2024-09-21T10:34:09.593 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to 15 diff [20106162~699475,27991252~10027027,49169063~8320873,58639820~1182814,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,119655166~4147395,135872708~1754391,144777497~6623225,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,210226943~2559490,224814781~26668382,259644006~8285489,269577554~11336387,286147115~4491537,298351917~23957216,339701748~5068042,347289030~490774,351227316~4338875,360853018~1275580,363514600~7729710,376984253~2159000,381111195~1866659,384505041~8135542,394221093~8158476] 2024-09-21T10:34:09.593 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [20106162~699475,27991252~10027027,49169063~8320873,58639820~1182814,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,119655166~4147395,135872708~1754391,144777497~6623225,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,210226943~2559490,224814781~26668382,259644006~8285489,269577554~11336387,286147115~4491537,298351917~23957216,339701748~5068042,347289030~490774,351227316~4338875,360853018~1275580,363514600~7729710,376984253~2159000,381111195~1866659,384505041~8135542,394221093~8158476] 2024-09-21T10:34:09.598 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [20106162~699475,27991252~10027027,49169063~8320873,58639820~1182814,62914560~4572360,72069759~2339334,82908058~1860108,97917081~7391112,113246208~6316712,119655166~4147395,135872708~1754391,144777497~6623225,152797416~596734,157423190~3438017,166204955~7344626,174552456~4674570,181624406~6491856,191528500~10608093,206063683~3113521,210226943~2559490,224814781~26668382,259644006~8285489,269577554~11336387,285212672~5425980,298351917~23957216,335544320~12235484,349721624~5844567,360853018~1275580,363514600~7729710,376984253~2159000,379207810~13432773,393702666~8676903] 2024-09-21T10:34:09.599 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to 16 diff [16883580~4715959,25706100~12312179,49169063~10653571,62918670~4568250,72069759~2339334,82908058~1860108,98522774~6785419,114683657~4879263,119655166~4147395,135872708~1754391,144777497~6623225,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,210226943~2559490,224814781~26668382,259644006~8285489,269577554~11336387,286147115~4491537,298351917~23957216,339701748~5068042,347289030~490774,351227316~4338875,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~8135542,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.599 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [16883580~4715959,25706100~12312179,49169063~10653571,62918670~4568250,72069759~2339334,82908058~1860108,98522774~6785419,114683657~4879263,119655166~4147395,135872708~1754391,144777497~6623225,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,210226943~2559490,224814781~26668382,259644006~8285489,269577554~11336387,286147115~4491537,298351917~23957216,339701748~5068042,347289030~490774,351227316~4338875,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~8135542,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.604 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [16883580~4715959,25706100~12312179,49169063~10653571,62914560~4572360,72069759~2339334,82908058~1860108,97917081~7391112,113246208~6316712,119655166~4147395,135872708~1754391,144777497~6623225,152797416~596734,157423190~3438017,166204955~7344626,174552456~4674570,181624406~6491856,191528500~10608093,206063683~3113521,210226943~2559490,224814781~26668382,259644006~8285489,269577554~11336387,285212672~5425980,298351917~23957216,335544320~12235484,349721624~5844567,360853018~1275580,363514600~7729710,376984253~2159000,379207810~13432773,393702666~8676903,409717800~2000564] 2024-09-21T10:34:09.604 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to 17 diff [15258652~7024612,25706100~12312179,49169063~10653571,62918670~11490423,82908058~1860108,95398996~13719063,114683657~9118904,135872708~1754391,144777497~6623225,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,209930960~3307546,224814781~26668382,259644006~8285489,269577554~11336387,286147115~4491537,298351917~23957216,339701748~5068042,347289030~490774,351227316~4338875,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~8135542,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.604 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~7024612,25706100~12312179,49169063~10653571,62918670~11490423,82908058~1860108,95398996~13719063,114683657~9118904,135872708~1754391,144777497~6623225,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~1395640,209930960~3307546,224814781~26668382,259644006~8285489,269577554~11336387,286147115~4491537,298351917~23957216,339701748~5068042,347289030~490774,351227316~4338875,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~8135542,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.609 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~7024612,25706100~12312179,49169063~10653571,62914560~11494533,82908058~1860108,95398996~13719063,113246208~10556353,135872708~1754391,144777497~6623225,152797416~596734,157423190~3438017,166204955~7344626,174552456~4674570,181624406~6491856,191528500~10608093,206063683~3113521,209930960~3307546,224814781~26668382,259644006~8285489,269577554~11336387,285212672~5425980,298351917~23957216,335544320~12235484,349721624~5844567,360853018~1275580,363514600~7729710,376984253~2159000,379207810~13432773,393702666~8676903,409717800~2000564] 2024-09-21T10:34:09.610 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to 18 diff [15258652~7024612,25706100~12312179,49169063~10653571,62918670~11490423,82908058~1860108,95398996~13719063,114683657~9118904,124591324~5118882,135872708~1754391,144777497~6623225,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~9355151,224814781~26668382,259644006~21269935,286147115~4491537,291766799~42181,292083899~1585781,298351917~23957216,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.610 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~7024612,25706100~12312179,49169063~10653571,62918670~11490423,82908058~1860108,95398996~13719063,114683657~9118904,124591324~5118882,135872708~1754391,144777497~6623225,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~9355151,224814781~26668382,259644006~21269935,286147115~4491537,291766799~42181,292083899~1585781,298351917~23957216,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.615 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~7024612,25706100~12312179,49169063~10653571,62914560~11494533,82908058~1860108,95398996~13719063,113246208~16463998,135872708~1754391,144777497~6623225,152797416~596734,157423190~3438017,166204955~7344626,174552456~4674570,181624406~6491856,191528500~10608093,206063683~11073032,224814781~26668382,259644006~21269935,285212672~8457008,298351917~23957216,335544320~12235484,349721624~8063397,360853018~1275580,363514600~7729710,376984253~2159000,379207810~23171759,409717800~2000564] 2024-09-21T10:34:09.615 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to 19 diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,259644006~21269935,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.615 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,259644006~21269935,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.622 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,174552456~4674570,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,259644006~21269935,281018368~12651312,298351917~23957216,331350016~16429788,349721624~8063397,360853018~1275580,363514600~7729710,376984253~2159000,379207810~23171759,409717800~2000564] 2024-09-21T10:34:09.622 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to 10 diff [98906260~4781139,114683657~4879263,207781564~1395640,269577554~11336387,305173155~4590075,311582767~3753918,339701748~4136464,367688090~3556220,376984253~2159000] 2024-09-21T10:34:09.622 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [98906260~4781139,114683657~4879263,207781564~1395640,269577554~11336387,305173155~4590075,311582767~3753918,339701748~4136464,367688090~3556220,376984253~2159000] 2024-09-21T10:34:09.628 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [97917081~5770318,114683657~4879263,206063683~3113521,269577554~11336387,305173155~4590075,311582767~3753918,335544320~8293892,367688090~3556220,376984253~2159000] 2024-09-21T10:34:09.628 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to 11 diff [31536075~6482204,49169063~6180328,62918670~4568250,82908058~1860108,98906260~6401933,114683657~4879263,157423190~3438017,207781564~1395640,236839701~5403448,269577554~11336387,298351917~11411313,311582767~7261573,339701748~4136464,367688090~3556220,376984253~2159000] 2024-09-21T10:34:09.628 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [31536075~6482204,49169063~6180328,62918670~4568250,82908058~1860108,98906260~6401933,114683657~4879263,157423190~3438017,207781564~1395640,236839701~5403448,269577554~11336387,298351917~11411313,311582767~7261573,339701748~4136464,367688090~3556220,376984253~2159000] 2024-09-21T10:34:09.633 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [31536075~6482204,49169063~6180328,62914560~4572360,82908058~1860108,97917081~7391112,114683657~4879263,157423190~3438017,206063683~3113521,236839701~5403448,269577554~11336387,298351917~11411313,311582767~7261573,335544320~8293892,367688090~3556220,376984253~2159000] 2024-09-21T10:34:09.633 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to 12 diff [27991252~10027027,49169063~6180328,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,157423190~3438017,182407418~5708844,191528500~10608093,207781564~1395640,224814781~7670458,236839701~5403448,269577554~11336387,298351917~20492423,339701748~4136464,367688090~3556220,376984253~2159000] 2024-09-21T10:34:09.633 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [27991252~10027027,49169063~6180328,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,157423190~3438017,182407418~5708844,191528500~10608093,207781564~1395640,224814781~7670458,236839701~5403448,269577554~11336387,298351917~20492423,339701748~4136464,367688090~3556220,376984253~2159000] 2024-09-21T10:34:09.639 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [27991252~10027027,49169063~6180328,62914560~4572360,72069759~2339334,82908058~1860108,97917081~7391112,114683657~4879263,157423190~3438017,182407418~5708844,191528500~10608093,206063683~3113521,224814781~7670458,236839701~5403448,269577554~11336387,298351917~20492423,335544320~8293892,367688090~3556220,376984253~2159000] 2024-09-21T10:34:09.639 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to 13 diff [27991252~10027027,49169063~8320873,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,157423190~3438017,166204955~7344626,175217960~2272728,182407418~5708844,191528500~10608093,207781564~1395640,224814781~7670458,236839701~5403448,244912682~6570481,269577554~11336387,298351917~23086031,339701748~4594073,347289030~490774,367688090~3556220,376984253~2159000,394221093~8158476] 2024-09-21T10:34:09.639 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [27991252~10027027,49169063~8320873,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,157423190~3438017,166204955~7344626,175217960~2272728,182407418~5708844,191528500~10608093,207781564~1395640,224814781~7670458,236839701~5403448,244912682~6570481,269577554~11336387,298351917~23086031,339701748~4594073,347289030~490774,367688090~3556220,376984253~2159000,394221093~8158476] 2024-09-21T10:34:09.644 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [27991252~10027027,49169063~8320873,62914560~4572360,72069759~2339334,82908058~1860108,97917081~7391112,114683657~4879263,157423190~3438017,166204955~7344626,175217960~2272728,182407418~5708844,191528500~10608093,206063683~3113521,224814781~7670458,236839701~5403448,244912682~6570481,269577554~11336387,298351917~23086031,335544320~8751501,344769790~3010014,367688090~3556220,376984253~2159000,393702666~8676903] 2024-09-21T10:34:09.644 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to 14 diff [20106162~699475,27991252~10027027,49169063~8320873,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,119655166~4147395,135872708~1754391,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~1395640,224814781~7670458,236839701~14643462,269577554~11336387,286147115~4491537,298351917~23086031,339701748~4594073,347289030~490774,360853018~1275580,367688090~3556220,376984253~2159000,394221093~8158476] 2024-09-21T10:34:09.644 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [20106162~699475,27991252~10027027,49169063~8320873,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,119655166~4147395,135872708~1754391,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~1395640,224814781~7670458,236839701~14643462,269577554~11336387,286147115~4491537,298351917~23086031,339701748~4594073,347289030~490774,360853018~1275580,367688090~3556220,376984253~2159000,394221093~8158476] 2024-09-21T10:34:09.649 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [20106162~699475,27991252~10027027,49169063~8320873,62914560~4572360,72069759~2339334,82908058~1860108,97917081~7391112,114683657~4879263,119655166~4147395,135872708~1754391,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,206063683~3113521,224814781~7670458,236839701~14643462,269577554~11336387,285212672~5425980,298351917~23086031,335544320~8751501,344769790~3010014,360853018~1275580,367688090~3556220,376984253~2159000,393702666~8676903] 2024-09-21T10:34:09.649 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to 15 diff [20106162~699475,27991252~10027027,49169063~8320873,58639820~1182814,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,119655166~4147395,135872708~1754391,144777497~6623225,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~1395640,210226943~2559490,224814781~26668382,269577554~11336387,286147115~4491537,298351917~23086031,339701748~4594073,347289030~490774,351227316~4338875,360853018~1275580,363514600~7729710,376984253~2159000,384505041~360305,394221093~8158476] 2024-09-21T10:34:09.649 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [20106162~699475,27991252~10027027,49169063~8320873,58639820~1182814,62918670~4568250,72069759~2339334,82908058~1860108,98906260~6401933,114683657~4879263,119655166~4147395,135872708~1754391,144777497~6623225,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~1395640,210226943~2559490,224814781~26668382,269577554~11336387,286147115~4491537,298351917~23086031,339701748~4594073,347289030~490774,351227316~4338875,360853018~1275580,363514600~7729710,376984253~2159000,384505041~360305,394221093~8158476] 2024-09-21T10:34:09.655 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [20106162~699475,27991252~10027027,49169063~8320873,58639820~1182814,62914560~4572360,72069759~2339334,82908058~1860108,97917081~7391112,114683657~4879263,119655166~4147395,135872708~1754391,144777497~6623225,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,206063683~3113521,210226943~2559490,224814781~26668382,269577554~11336387,285212672~5425980,298351917~23086031,335544320~8751501,344769790~3010014,349721624~5844567,360853018~1275580,363514600~7729710,376984253~2159000,384505041~360305,393702666~8676903] 2024-09-21T10:34:09.655 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to 16 diff [16883580~4715959,25706100~12312179,49169063~10653571,62918670~4568250,72069759~2339334,82908058~1860108,98522774~6785419,114683657~4879263,119655166~4147395,135872708~1754391,144777497~6623225,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~1395640,210226943~2559490,224814781~26668382,269577554~11336387,286147115~4491537,298351917~23086031,339701748~4594073,347289030~490774,351227316~4338875,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.655 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [16883580~4715959,25706100~12312179,49169063~10653571,62918670~4568250,72069759~2339334,82908058~1860108,98522774~6785419,114683657~4879263,119655166~4147395,135872708~1754391,144777497~6623225,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~1395640,210226943~2559490,224814781~26668382,269577554~11336387,286147115~4491537,298351917~23086031,339701748~4594073,347289030~490774,351227316~4338875,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.661 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [16883580~4715959,25706100~12312179,49169063~10653571,62914560~4572360,72069759~2339334,82908058~1860108,97917081~7391112,114683657~4879263,119655166~4147395,135872708~1754391,144777497~6623225,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,206063683~3113521,210226943~2559490,224814781~26668382,269577554~11336387,285212672~5425980,298351917~23086031,335544320~8751501,344769790~3010014,349721624~5844567,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,393702666~8676903,409717800~2000564] 2024-09-21T10:34:09.661 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to 17 diff [15258652~7024612,25706100~12312179,49169063~10653571,62918670~11490423,82908058~1860108,95398996~13719063,114683657~9118904,135872708~1754391,144777497~6623225,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~1395640,209930960~3307546,224814781~26668382,269577554~11336387,286147115~4491537,298351917~23086031,339701748~4594073,347289030~490774,351227316~4338875,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.661 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~7024612,25706100~12312179,49169063~10653571,62918670~11490423,82908058~1860108,95398996~13719063,114683657~9118904,135872708~1754391,144777497~6623225,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~1395640,209930960~3307546,224814781~26668382,269577554~11336387,286147115~4491537,298351917~23086031,339701748~4594073,347289030~490774,351227316~4338875,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.666 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~7024612,25706100~12312179,49169063~10653571,62914560~11494533,82908058~1860108,95398996~13719063,114683657~9118904,135872708~1754391,144777497~6623225,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,206063683~3113521,209930960~3307546,224814781~26668382,269577554~11336387,285212672~5425980,298351917~23086031,335544320~8751501,344769790~3010014,349721624~5844567,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,393702666~8676903,409717800~2000564] 2024-09-21T10:34:09.667 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to 18 diff [15258652~7024612,25706100~12312179,49169063~10653571,62918670~11490423,82908058~1860108,95398996~13719063,114683657~9118904,124591324~5118882,135872708~1754391,144777497~6623225,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~9355151,224814781~26668382,265149402~15764539,286147115~4491537,291766799~42181,292083899~1585781,298351917~23086031,339701748~4594073,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.667 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~7024612,25706100~12312179,49169063~10653571,62918670~11490423,82908058~1860108,95398996~13719063,114683657~9118904,124591324~5118882,135872708~1754391,144777497~6623225,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~9355151,224814781~26668382,265149402~15764539,286147115~4491537,291766799~42181,292083899~1585781,298351917~23086031,339701748~4594073,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.673 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~7024612,25706100~12312179,49169063~10653571,62914560~11494533,82908058~1860108,95398996~13719063,114683657~15026549,135872708~1754391,144777497~6623225,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,206063683~11073032,224814781~26668382,265149402~15764539,285212672~8457008,298351917~23086031,335544320~8751501,344769790~3010014,349721624~8063397,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:09.673 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to 19 diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,265149402~15764539,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,339701748~4594073,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.673 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,265149402~15764539,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,339701748~4594073,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.678 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,265149402~15764539,281018368~12651312,298351917~23086031,331350016~12945805,344769790~3010014,349721624~8063397,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:09.678 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to 11 diff [31536075~6482204,49169063~6180328,62918670~4568250,82908058~1860108,101539283~3768910,157423190~3438017,236839701~5403448,298351917~8033557,314950007~3894333] 2024-09-21T10:34:09.678 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [31536075~6482204,49169063~6180328,62918670~4568250,82908058~1860108,101539283~3768910,157423190~3438017,236839701~5403448,298351917~8033557,314950007~3894333] 2024-09-21T10:34:09.683 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [31536075~6482204,49169063~6180328,62914560~4572360,82908058~1860108,101539283~3768910,157423190~3438017,236839701~5403448,298351917~8033557,314950007~3894333] 2024-09-21T10:34:09.684 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to 12 diff [27991252~10027027,49169063~6180328,62918670~4568250,72069759~2339334,82908058~1860108,101539283~3768910,116074398~2736787,157423190~3438017,182407418~5708844,191528500~10608093,224814781~7670458,236839701~5403448,298351917~8033557,306516710~12327630] 2024-09-21T10:34:09.684 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [27991252~10027027,49169063~6180328,62918670~4568250,72069759~2339334,82908058~1860108,101539283~3768910,116074398~2736787,157423190~3438017,182407418~5708844,191528500~10608093,224814781~7670458,236839701~5403448,298351917~8033557,306516710~12327630] 2024-09-21T10:34:09.689 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [27991252~10027027,49169063~6180328,62914560~4572360,72069759~2339334,82908058~1860108,101539283~3768910,116074398~2736787,157423190~3438017,182407418~5708844,191528500~10608093,224814781~7670458,236839701~5403448,298351917~8033557,306516710~12327630] 2024-09-21T10:34:09.689 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to 13 diff [27991252~10027027,49169063~8320873,62918670~4568250,72069759~2339334,82908058~1860108,101539283~3768910,116074398~2736787,157423190~3438017,166204955~7344626,175217960~2272728,182407418~5708844,191528500~10608093,224814781~7670458,236839701~5403448,244912682~6570481,298351917~8033557,306516710~14921238,343689268~606553,347289030~490774,394221093~8158476] 2024-09-21T10:34:09.689 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [27991252~10027027,49169063~8320873,62918670~4568250,72069759~2339334,82908058~1860108,101539283~3768910,116074398~2736787,157423190~3438017,166204955~7344626,175217960~2272728,182407418~5708844,191528500~10608093,224814781~7670458,236839701~5403448,244912682~6570481,298351917~8033557,306516710~14921238,343689268~606553,347289030~490774,394221093~8158476] 2024-09-21T10:34:09.695 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [27991252~10027027,49169063~8320873,62914560~4572360,72069759~2339334,82908058~1860108,101539283~3768910,116074398~2736787,157423190~3438017,166204955~7344626,175217960~2272728,182407418~5708844,191528500~10608093,224814781~7670458,236839701~5403448,244912682~6570481,298351917~8033557,306516710~14921238,343689268~606553,344769790~3010014,393702666~8676903] 2024-09-21T10:34:09.695 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to 14 diff [20106162~699475,27991252~10027027,49169063~8320873,62918670~4568250,72069759~2339334,82908058~1860108,101539283~3768910,116074398~2736787,119655166~4147395,135872708~1754391,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,224814781~7670458,236839701~14643462,286147115~4491537,298351917~23086031,343689268~606553,347289030~490774,360853018~1275580,394221093~8158476] 2024-09-21T10:34:09.696 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [20106162~699475,27991252~10027027,49169063~8320873,62918670~4568250,72069759~2339334,82908058~1860108,101539283~3768910,116074398~2736787,119655166~4147395,135872708~1754391,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,224814781~7670458,236839701~14643462,286147115~4491537,298351917~23086031,343689268~606553,347289030~490774,360853018~1275580,394221093~8158476] 2024-09-21T10:34:09.700 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [20106162~699475,27991252~10027027,49169063~8320873,62914560~4572360,72069759~2339334,82908058~1860108,101539283~3768910,116074398~2736787,119655166~4147395,135872708~1754391,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,224814781~7670458,236839701~14643462,285212672~5425980,298351917~23086031,343689268~606553,344769790~3010014,360853018~1275580,393702666~8676903] 2024-09-21T10:34:09.700 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to 15 diff [20106162~699475,27991252~10027027,49169063~8320873,58639820~1182814,62918670~4568250,72069759~2339334,82908058~1860108,101539283~3768910,116074398~2736787,119655166~4147395,135872708~1754391,144777497~6623225,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,210226943~2559490,224814781~26668382,286147115~4491537,298351917~23086031,343689268~606553,347289030~490774,351227316~4338875,360853018~1275580,363514600~4814024,384505041~360305,394221093~8158476] 2024-09-21T10:34:09.701 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [20106162~699475,27991252~10027027,49169063~8320873,58639820~1182814,62918670~4568250,72069759~2339334,82908058~1860108,101539283~3768910,116074398~2736787,119655166~4147395,135872708~1754391,144777497~6623225,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,210226943~2559490,224814781~26668382,286147115~4491537,298351917~23086031,343689268~606553,347289030~490774,351227316~4338875,360853018~1275580,363514600~4814024,384505041~360305,394221093~8158476] 2024-09-21T10:34:09.706 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [20106162~699475,27991252~10027027,49169063~8320873,58639820~1182814,62914560~4572360,72069759~2339334,82908058~1860108,101539283~3768910,116074398~2736787,119655166~4147395,135872708~1754391,144777497~6623225,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,210226943~2559490,224814781~26668382,285212672~5425980,298351917~23086031,343689268~606553,344769790~3010014,349721624~5844567,360853018~1275580,363514600~4814024,384505041~360305,393702666~8676903] 2024-09-21T10:34:09.706 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to 16 diff [16883580~4715959,25706100~12312179,49169063~10653571,62918670~4568250,72069759~2339334,82908058~1860108,98522774~6785419,116074398~2736787,119655166~4147395,135872708~1754391,144777497~6623225,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,210226943~2559490,224814781~26668382,286147115~4491537,298351917~23086031,343689268~606553,347289030~490774,351227316~4338875,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.706 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [16883580~4715959,25706100~12312179,49169063~10653571,62918670~4568250,72069759~2339334,82908058~1860108,98522774~6785419,116074398~2736787,119655166~4147395,135872708~1754391,144777497~6623225,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,210226943~2559490,224814781~26668382,286147115~4491537,298351917~23086031,343689268~606553,347289030~490774,351227316~4338875,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.711 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [16883580~4715959,25706100~12312179,49169063~10653571,62914560~4572360,72069759~2339334,82908058~1860108,98522774~6785419,116074398~2736787,119655166~4147395,135872708~1754391,144777497~6623225,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,210226943~2559490,224814781~26668382,285212672~5425980,298351917~23086031,343689268~606553,344769790~3010014,349721624~5844567,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,393702666~8676903,409717800~2000564] 2024-09-21T10:34:09.711 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to 17 diff [15258652~7024612,25706100~12312179,49169063~10653571,62918670~11490423,82908058~1860108,95398996~13719063,115435950~8366611,135872708~1754391,144777497~6623225,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,209930960~3307546,224814781~26668382,269871960~1086953,286147115~4491537,298351917~23086031,343689268~606553,347289030~490774,351227316~4338875,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.712 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~7024612,25706100~12312179,49169063~10653571,62918670~11490423,82908058~1860108,95398996~13719063,115435950~8366611,135872708~1754391,144777497~6623225,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,209930960~3307546,224814781~26668382,269871960~1086953,286147115~4491537,298351917~23086031,343689268~606553,347289030~490774,351227316~4338875,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.717 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~7024612,25706100~12312179,49169063~10653571,62914560~11494533,82908058~1860108,95398996~13719063,115435950~8366611,135872708~1754391,144777497~6623225,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,209930960~3307546,224814781~26668382,269871960~1086953,285212672~5425980,298351917~23086031,343689268~606553,344769790~3010014,349721624~5844567,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,393702666~8676903,409717800~2000564] 2024-09-21T10:34:09.717 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to 18 diff [15258652~7024612,25706100~12312179,49169063~10653571,62918670~11490423,82908058~1860108,95398996~13719063,115435950~8366611,124591324~5118882,135872708~1754391,144777497~6623225,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~8168491,224814781~26668382,265149402~5809511,286147115~4491537,291766799~42181,292083899~1585781,298351917~23086031,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.717 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~7024612,25706100~12312179,49169063~10653571,62918670~11490423,82908058~1860108,95398996~13719063,115435950~8366611,124591324~5118882,135872708~1754391,144777497~6623225,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~8168491,224814781~26668382,265149402~5809511,286147115~4491537,291766799~42181,292083899~1585781,298351917~23086031,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.724 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~7024612,25706100~12312179,49169063~10653571,62914560~11494533,82908058~1860108,95398996~13719063,115435950~14274256,135872708~1754391,144777497~6623225,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~8168491,224814781~26668382,265149402~5809511,285212672~8457008,298351917~23086031,343689268~606553,344769790~3010014,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:09.724 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to 19 diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.724 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.729 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,281018368~12651312,298351917~23086031,331350016~7329672,343689268~606553,344769790~3010014,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:09.730 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to 12 diff [27991252~3630235,72069759~2339334,116074398~2736787,182407418~5708844,191528500~10608093,224814781~7670458,306516710~9213977] 2024-09-21T10:34:09.730 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [27991252~3630235,72069759~2339334,116074398~2736787,182407418~5708844,191528500~10608093,224814781~7670458,306516710~9213977] 2024-09-21T10:34:09.735 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [27991252~3630235,72069759~2339334,116074398~2736787,182407418~5708844,191528500~10608093,224814781~7670458,306516710~9213977] 2024-09-21T10:34:09.735 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to 13 diff [27991252~3630235,53616434~3873502,72069759~2339334,116074398~2736787,166204955~7344626,175217960~2272728,182407418~5708844,191528500~10608093,224814781~7670458,244912682~6570481,301105270~570107,306516710~11644398,318527970~2909978,343689268~606553,347289030~490774,394221093~8158476] 2024-09-21T10:34:09.735 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [27991252~3630235,53616434~3873502,72069759~2339334,116074398~2736787,166204955~7344626,175217960~2272728,182407418~5708844,191528500~10608093,224814781~7670458,244912682~6570481,301105270~570107,306516710~11644398,318527970~2909978,343689268~606553,347289030~490774,394221093~8158476] 2024-09-21T10:34:09.741 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [27991252~3630235,53616434~3873502,72069759~2339334,116074398~2736787,166204955~7344626,175217960~2272728,182407418~5708844,191528500~10608093,224814781~7670458,244912682~6570481,301105270~570107,306516710~11644398,318527970~2909978,343689268~606553,344769790~3010014,393702666~8676903] 2024-09-21T10:34:09.741 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to 14 diff [20106162~699475,27991252~3630235,32450955~218199,53616434~3873502,72069759~2339334,116074398~2736787,119655166~4147395,135872708~1754391,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,224814781~7670458,241096252~10386911,286147115~4491537,301105270~570107,306016051~12145057,318527970~2909978,343689268~606553,347289030~490774,360853018~1275580,394221093~8158476] 2024-09-21T10:34:09.741 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [20106162~699475,27991252~3630235,32450955~218199,53616434~3873502,72069759~2339334,116074398~2736787,119655166~4147395,135872708~1754391,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,224814781~7670458,241096252~10386911,286147115~4491537,301105270~570107,306016051~12145057,318527970~2909978,343689268~606553,347289030~490774,360853018~1275580,394221093~8158476] 2024-09-21T10:34:09.747 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [20106162~699475,27991252~3630235,32450955~218199,53616434~3873502,72069759~2339334,116074398~2736787,119655166~4147395,135872708~1754391,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,224814781~7670458,241096252~10386911,285212672~5425980,301105270~570107,306016051~12145057,318527970~2909978,343689268~606553,344769790~3010014,360853018~1275580,393702666~8676903] 2024-09-21T10:34:09.747 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to 15 diff [20106162~699475,27991252~3630235,32450955~218199,53616434~3873502,58639820~1182814,72069759~2339334,116074398~2736787,119655166~4147395,135872708~1754391,144777497~6623225,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,210226943~2559490,224814781~26668382,286147115~4491537,301105270~17055838,318527970~2909978,343689268~606553,347289030~490774,351227316~4338875,360853018~1275580,363514600~4814024,384505041~360305,394221093~8158476] 2024-09-21T10:34:09.747 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [20106162~699475,27991252~3630235,32450955~218199,53616434~3873502,58639820~1182814,72069759~2339334,116074398~2736787,119655166~4147395,135872708~1754391,144777497~6623225,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,210226943~2559490,224814781~26668382,286147115~4491537,301105270~17055838,318527970~2909978,343689268~606553,347289030~490774,351227316~4338875,360853018~1275580,363514600~4814024,384505041~360305,394221093~8158476] 2024-09-21T10:34:09.754 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [20106162~699475,27991252~3630235,32450955~218199,53616434~3873502,58639820~1182814,72069759~2339334,116074398~2736787,119655166~4147395,135872708~1754391,144777497~6623225,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,210226943~2559490,224814781~26668382,285212672~5425980,301105270~17055838,318527970~2909978,343689268~606553,344769790~3010014,349721624~5844567,360853018~1275580,363514600~4814024,384505041~360305,393702666~8676903] 2024-09-21T10:34:09.754 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to 16 diff [16883580~4715959,25706100~5915387,32450955~218199,53413917~6408717,72069759~2339334,98522774~6314663,116074398~2736787,119655166~4147395,135872708~1754391,144777497~6623225,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,210226943~2559490,224814781~26668382,286147115~4491537,301105270~17055838,318527970~2909978,343689268~606553,347289030~490774,351227316~4338875,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.754 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [16883580~4715959,25706100~5915387,32450955~218199,53413917~6408717,72069759~2339334,98522774~6314663,116074398~2736787,119655166~4147395,135872708~1754391,144777497~6623225,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,210226943~2559490,224814781~26668382,286147115~4491537,301105270~17055838,318527970~2909978,343689268~606553,347289030~490774,351227316~4338875,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.759 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [16883580~4715959,25706100~5915387,32450955~218199,53413917~6408717,72069759~2339334,98522774~6314663,116074398~2736787,119655166~4147395,135872708~1754391,144777497~6623225,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,210226943~2559490,224814781~26668382,285212672~5425980,301105270~17055838,318527970~2909978,343689268~606553,344769790~3010014,349721624~5844567,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,393702666~8676903,409717800~2000564] 2024-09-21T10:34:09.760 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to 17 diff [15258652~7024612,25706100~5915387,32450955~218199,53413917~6408717,66238036~8171057,95398996~13719063,115435950~8366611,135872708~1754391,144777497~6623225,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,209930960~3307546,224814781~26668382,269871960~1086953,286147115~4491537,301105270~17055838,318527970~2909978,343689268~606553,347289030~490774,351227316~4338875,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.760 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~7024612,25706100~5915387,32450955~218199,53413917~6408717,66238036~8171057,95398996~13719063,115435950~8366611,135872708~1754391,144777497~6623225,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,209930960~3307546,224814781~26668382,269871960~1086953,286147115~4491537,301105270~17055838,318527970~2909978,343689268~606553,347289030~490774,351227316~4338875,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.765 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~7024612,25706100~5915387,32450955~218199,53413917~6408717,66238036~8171057,95398996~13719063,115435950~8366611,135872708~1754391,144777497~6623225,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,209930960~3307546,224814781~26668382,269871960~1086953,285212672~5425980,301105270~17055838,318527970~2909978,343689268~606553,344769790~3010014,349721624~5844567,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,393702666~8676903,409717800~2000564] 2024-09-21T10:34:09.765 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to 18 diff [15258652~7024612,25706100~5915387,32450955~218199,53413917~6408717,66238036~8171057,95398996~13719063,115435950~8366611,124591324~5118882,135872708~1754391,144777497~6623225,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~8168491,224814781~26668382,265149402~5809511,286147115~4491537,291766799~42181,292083899~1585781,301105270~17055838,318527970~2909978,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.765 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~7024612,25706100~5915387,32450955~218199,53413917~6408717,66238036~8171057,95398996~13719063,115435950~8366611,124591324~5118882,135872708~1754391,144777497~6623225,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~8168491,224814781~26668382,265149402~5809511,286147115~4491537,291766799~42181,292083899~1585781,301105270~17055838,318527970~2909978,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.771 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~7024612,25706100~5915387,32450955~218199,53413917~6408717,66238036~8171057,95398996~13719063,115435950~14274256,135872708~1754391,144777497~6623225,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~8168491,224814781~26668382,265149402~5809511,285212672~8457008,301105270~17055838,318527970~2909978,343689268~606553,344769790~3010014,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:09.771 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to 19 diff [15258652~8431860,25706100~5915387,32450955~218199,47127613~12695021,66238036~8171057,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~17055838,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.771 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~5915387,32450955~218199,47127613~12695021,66238036~8171057,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~17055838,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.777 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~5915387,32450955~218199,47127613~12695021,66238036~8171057,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,281018368~12651312,301105270~17055838,318527970~2909978,331350016~7329672,343689268~606553,344769790~3010014,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:09.777 INFO:tasks.workunit.client.0.smithi070.stdout:from 12 to 13 diff [53616434~3873502,166204955~7344626,175217960~2272728,244912682~6570481,301105270~570107,315123129~3037979,318527970~2909978,343689268~606553,347289030~490774,394221093~8158476] 2024-09-21T10:34:09.777 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [53616434~3873502,166204955~7344626,175217960~2272728,244912682~6570481,301105270~570107,315123129~3037979,318527970~2909978,343689268~606553,347289030~490774,394221093~8158476] 2024-09-21T10:34:09.783 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [53616434~3873502,166204955~7344626,175217960~2272728,244912682~6570481,301105270~570107,315123129~3037979,318527970~2909978,343689268~606553,344769790~3010014,393702666~8676903] 2024-09-21T10:34:09.783 INFO:tasks.workunit.client.0.smithi070.stdout:from 12 to 14 diff [20106162~699475,32450955~218199,53616434~3873502,116263063~292497,119655166~4147395,135872708~1754391,166204955~7344626,175217960~2272728,181624406~3276255,241096252~10386911,286147115~4491537,301105270~570107,306016051~3154216,315123129~3037979,318527970~2909978,343689268~606553,347289030~490774,360853018~1275580,394221093~8158476] 2024-09-21T10:34:09.783 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [20106162~699475,32450955~218199,53616434~3873502,116263063~292497,119655166~4147395,135872708~1754391,166204955~7344626,175217960~2272728,181624406~3276255,241096252~10386911,286147115~4491537,301105270~570107,306016051~3154216,315123129~3037979,318527970~2909978,343689268~606553,347289030~490774,360853018~1275580,394221093~8158476] 2024-09-21T10:34:09.789 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [20106162~699475,32450955~218199,53616434~3873502,116263063~292497,119655166~4147395,135872708~1754391,166204955~7344626,175217960~2272728,181624406~3276255,241096252~10386911,285212672~5425980,301105270~570107,306016051~3154216,315123129~3037979,318527970~2909978,343689268~606553,344769790~3010014,360853018~1275580,393702666~8676903] 2024-09-21T10:34:09.789 INFO:tasks.workunit.client.0.smithi070.stdout:from 12 to 15 diff [20106162~699475,32450955~218199,53616434~3873502,58639820~1182814,116263063~292497,119655166~4147395,135872708~1754391,144777497~6623225,166204955~7344626,175217960~2272728,181624406~3276255,210226943~2559490,230058264~21424899,286147115~4491537,301105270~8064997,315123129~3037979,318527970~2909978,343689268~606553,347289030~490774,351227316~4338875,360853018~1275580,363514600~4814024,384505041~360305,394221093~8158476] 2024-09-21T10:34:09.789 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [20106162~699475,32450955~218199,53616434~3873502,58639820~1182814,116263063~292497,119655166~4147395,135872708~1754391,144777497~6623225,166204955~7344626,175217960~2272728,181624406~3276255,210226943~2559490,230058264~21424899,286147115~4491537,301105270~8064997,315123129~3037979,318527970~2909978,343689268~606553,347289030~490774,351227316~4338875,360853018~1275580,363514600~4814024,384505041~360305,394221093~8158476] 2024-09-21T10:34:09.795 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [20106162~699475,32450955~218199,53616434~3873502,58639820~1182814,116263063~292497,119655166~4147395,135872708~1754391,144777497~6623225,166204955~7344626,175217960~2272728,181624406~3276255,210226943~2559490,230058264~21424899,285212672~5425980,301105270~8064997,315123129~3037979,318527970~2909978,343689268~606553,344769790~3010014,349721624~5844567,360853018~1275580,363514600~4814024,384505041~360305,393702666~8676903] 2024-09-21T10:34:09.795 INFO:tasks.workunit.client.0.smithi070.stdout:from 12 to 16 diff [16883580~4715959,25706100~3951165,32450955~218199,53413917~6408717,98522774~6314663,116263063~292497,119655166~4147395,135872708~1754391,144777497~6623225,166204955~7344626,175217960~2272728,181624406~3276255,210226943~2559490,225041157~26442006,286147115~4491537,301105270~8064997,315123129~3037979,318527970~2909978,343689268~606553,347289030~490774,351227316~4338875,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.795 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [16883580~4715959,25706100~3951165,32450955~218199,53413917~6408717,98522774~6314663,116263063~292497,119655166~4147395,135872708~1754391,144777497~6623225,166204955~7344626,175217960~2272728,181624406~3276255,210226943~2559490,225041157~26442006,286147115~4491537,301105270~8064997,315123129~3037979,318527970~2909978,343689268~606553,347289030~490774,351227316~4338875,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.801 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [16883580~4715959,25706100~3951165,32450955~218199,53413917~6408717,98522774~6314663,116263063~292497,119655166~4147395,135872708~1754391,144777497~6623225,166204955~7344626,175217960~2272728,181624406~3276255,210226943~2559490,225041157~26442006,285212672~5425980,301105270~8064997,315123129~3037979,318527970~2909978,343689268~606553,344769790~3010014,349721624~5844567,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,393702666~8676903,409717800~2000564] 2024-09-21T10:34:09.801 INFO:tasks.workunit.client.0.smithi070.stdout:from 12 to 17 diff [15258652~7024612,25706100~3951165,32450955~218199,53413917~6408717,66238036~7394424,95398996~13719063,115435950~8366611,135872708~1754391,144777497~6623225,166204955~7344626,175217960~2272728,181624406~3276255,209930960~3307546,225041157~26442006,269871960~1086953,286147115~4491537,301105270~8064997,315123129~3037979,318527970~2909978,343689268~606553,347289030~490774,351227316~4338875,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.801 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~7024612,25706100~3951165,32450955~218199,53413917~6408717,66238036~7394424,95398996~13719063,115435950~8366611,135872708~1754391,144777497~6623225,166204955~7344626,175217960~2272728,181624406~3276255,209930960~3307546,225041157~26442006,269871960~1086953,286147115~4491537,301105270~8064997,315123129~3037979,318527970~2909978,343689268~606553,347289030~490774,351227316~4338875,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.808 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~7024612,25706100~3951165,32450955~218199,53413917~6408717,66238036~7394424,95398996~13719063,115435950~8366611,135872708~1754391,144777497~6623225,166204955~7344626,175217960~2272728,181624406~3276255,209930960~3307546,225041157~26442006,269871960~1086953,285212672~5425980,301105270~8064997,315123129~3037979,318527970~2909978,343689268~606553,344769790~3010014,349721624~5844567,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,393702666~8676903,409717800~2000564] 2024-09-21T10:34:09.808 INFO:tasks.workunit.client.0.smithi070.stdout:from 12 to 18 diff [15258652~7024612,25706100~3951165,32450955~218199,53413917~6408717,66238036~7394424,95398996~13719063,115435950~8366611,124591324~5118882,135872708~1754391,144777497~6623225,166204955~7344626,175217960~2272728,181624406~3276255,208968224~8168491,225041157~26442006,265149402~5809511,286147115~4491537,291766799~42181,292083899~1585781,301105270~8064997,310892222~7268886,318527970~2909978,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.808 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~7024612,25706100~3951165,32450955~218199,53413917~6408717,66238036~7394424,95398996~13719063,115435950~8366611,124591324~5118882,135872708~1754391,144777497~6623225,166204955~7344626,175217960~2272728,181624406~3276255,208968224~8168491,225041157~26442006,265149402~5809511,286147115~4491537,291766799~42181,292083899~1585781,301105270~8064997,310892222~7268886,318527970~2909978,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.813 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~7024612,25706100~3951165,32450955~218199,53413917~6408717,66238036~7394424,95398996~13719063,115435950~14274256,135872708~1754391,144777497~6623225,166204955~7344626,175217960~2272728,181624406~3276255,208968224~8168491,225041157~26442006,265149402~5809511,285212672~8457008,301105270~8064997,310892222~7268886,318527970~2909978,343689268~606553,344769790~3010014,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:09.813 INFO:tasks.workunit.client.0.smithi070.stdout:from 12 to 19 diff [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~3276255,208968224~9244408,225041157~26442006,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~8064997,310892222~7268886,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.813 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~3276255,208968224~9244408,225041157~26442006,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~8064997,310892222~7268886,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:09.819 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~3276255,208968224~9244408,225041157~26442006,265149402~5809511,281018368~12651312,301105270~8064997,310892222~7268886,318527970~2909978,331350016~7329672,343689268~606553,344769790~3010014,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:09.819 INFO:tasks.workunit.client.0.smithi070.stdout:from 13 to 14 diff [20106162~699475,32450955~218199,116263063~292497,119655166~4147395,135872708~1754391,181624406~3276255,241096252~6979125,286147115~4491537,306016051~3154216,360853018~1275580] 2024-09-21T10:34:09.819 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [20106162~699475,32450955~218199,116263063~292497,119655166~4147395,135872708~1754391,181624406~3276255,241096252~6979125,286147115~4491537,306016051~3154216,360853018~1275580] 2024-09-21T10:34:09.824 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [20106162~699475,32450955~218199,116263063~292497,119655166~4147395,135872708~1754391,181624406~3276255,241096252~6979125,285212672~5425980,306016051~3154216,360853018~1275580] 2024-09-21T10:34:09.824 INFO:tasks.workunit.client.0.smithi070.stdout:from 13 to 15 diff [20106162~699475,32450955~218199,58639820~1182814,116263063~292497,119655166~4147395,135872708~1754391,144777497~6623225,181624406~3276255,210226943~2559490,230058264~18017113,286147115~4491537,301283769~7886498,351227316~4338875,360853018~1275580,363514600~4814024,384505041~360305,397020628~4740508] 2024-09-21T10:34:09.826 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [20106162~699475,32450955~218199,58639820~1182814,116263063~292497,119655166~4147395,135872708~1754391,144777497~6623225,181624406~3276255,210226943~2559490,230058264~18017113,286147115~4491537,301283769~7886498,351227316~4338875,360853018~1275580,363514600~4814024,384505041~360305,397020628~4740508] 2024-09-21T10:34:09.830 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [20106162~699475,32450955~218199,58639820~1182814,116263063~292497,119655166~4147395,135872708~1754391,144777497~6623225,181624406~3276255,210226943~2559490,230058264~18017113,285212672~5425980,301283769~7886498,349721624~5844567,360853018~1275580,363514600~4814024,384505041~360305,397020628~4740508] 2024-09-21T10:34:09.830 INFO:tasks.workunit.client.0.smithi070.stdout:from 13 to 16 diff [16883580~4715959,25706100~3951165,32450955~218199,53413917~6408717,98522774~6314663,116263063~292497,119655166~4147395,135872708~1754391,144777497~6623225,181624406~3276255,210226943~2559490,225041157~23034220,286147115~4491537,301283769~7886498,351227316~4338875,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,395383365~6377771,409717800~2000564] 2024-09-21T10:34:09.830 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [16883580~4715959,25706100~3951165,32450955~218199,53413917~6408717,98522774~6314663,116263063~292497,119655166~4147395,135872708~1754391,144777497~6623225,181624406~3276255,210226943~2559490,225041157~23034220,286147115~4491537,301283769~7886498,351227316~4338875,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,395383365~6377771,409717800~2000564] 2024-09-21T10:34:09.835 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [16883580~4715959,25706100~3951165,32450955~218199,53413917~6408717,98522774~6314663,116263063~292497,119655166~4147395,135872708~1754391,144777497~6623225,181624406~3276255,210226943~2559490,225041157~23034220,285212672~5425980,301283769~7886498,349721624~5844567,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,395383365~6377771,409717800~2000564] 2024-09-21T10:34:09.835 INFO:tasks.workunit.client.0.smithi070.stdout:from 13 to 17 diff [15258652~7024612,25706100~3951165,32450955~218199,53413917~6408717,66238036~7394424,95398996~13719063,115435950~8366611,135872708~1754391,144777497~6623225,168967775~4510479,181624406~3276255,209930960~3307546,225041157~23034220,269871960~1086953,286147115~4491537,301283769~7886498,351227316~4338875,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,395383365~6377771,409717800~2000564] 2024-09-21T10:34:09.836 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~7024612,25706100~3951165,32450955~218199,53413917~6408717,66238036~7394424,95398996~13719063,115435950~8366611,135872708~1754391,144777497~6623225,168967775~4510479,181624406~3276255,209930960~3307546,225041157~23034220,269871960~1086953,286147115~4491537,301283769~7886498,351227316~4338875,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,395383365~6377771,409717800~2000564] 2024-09-21T10:34:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:09 smithi160 ceph-mon[101879]: pgmap v1787: 105 pgs: 105 active+clean; 1.0 GiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 236 KiB/s rd, 35 MiB/s wr, 364 op/s 2024-09-21T10:34:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:09 smithi160 ceph-mon[101879]: osdmap e1130: 8 total, 8 up, 8 in 2024-09-21T10:34:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:09 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:34:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:09 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:34:09.842 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~7024612,25706100~3951165,32450955~218199,53413917~6408717,66238036~7394424,95398996~13719063,115435950~8366611,135872708~1754391,144777497~6623225,168967775~4510479,181624406~3276255,209930960~3307546,225041157~23034220,269871960~1086953,285212672~5425980,301283769~7886498,349721624~5844567,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,395383365~6377771,409717800~2000564] 2024-09-21T10:34:09.842 INFO:tasks.workunit.client.0.smithi070.stdout:from 13 to 18 diff [15258652~7024612,25706100~3951165,32450955~218199,53413917~6408717,66238036~7394424,95398996~13719063,115435950~8366611,124591324~5118882,135872708~1754391,144777497~6623225,168967775~4510479,181624406~3276255,208968224~8168491,225041157~23034220,265149402~5809511,286147115~4491537,291766799~42181,292083899~1585781,301283769~7886498,310892222~4271109,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:09.842 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~7024612,25706100~3951165,32450955~218199,53413917~6408717,66238036~7394424,95398996~13719063,115435950~8366611,124591324~5118882,135872708~1754391,144777497~6623225,168967775~4510479,181624406~3276255,208968224~8168491,225041157~23034220,265149402~5809511,286147115~4491537,291766799~42181,292083899~1585781,301283769~7886498,310892222~4271109,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:09.847 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~7024612,25706100~3951165,32450955~218199,53413917~6408717,66238036~7394424,95398996~13719063,115435950~14274256,135872708~1754391,144777497~6623225,168967775~4510479,181624406~3276255,208968224~8168491,225041157~23034220,265149402~5809511,285212672~8457008,301283769~7886498,310892222~4271109,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:09.847 INFO:tasks.workunit.client.0.smithi070.stdout:from 13 to 19 diff [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,168967775~4510479,181624406~3276255,208968224~9244408,225041157~23034220,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301283769~7886498,310892222~4271109,333198080~5481608,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:09.847 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,168967775~4510479,181624406~3276255,208968224~9244408,225041157~23034220,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301283769~7886498,310892222~4271109,333198080~5481608,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:09.853 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,168967775~4510479,181624406~3276255,208968224~9244408,225041157~23034220,265149402~5809511,281018368~12651312,301283769~7886498,310892222~4271109,331350016~7329672,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:09.853 INFO:tasks.workunit.client.0.smithi070.stdout:from 14 to 15 diff [58639820~1182814,144777497~6623225,210226943~2559490,230058264~11640976,301283769~5609120,351227316~4338875,363514600~4814024,384505041~360305,397020628~4740508] 2024-09-21T10:34:09.853 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [58639820~1182814,144777497~6623225,210226943~2559490,230058264~11640976,301283769~5609120,351227316~4338875,363514600~4814024,384505041~360305,397020628~4740508] 2024-09-21T10:34:09.858 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [58639820~1182814,144777497~6623225,210226943~2559490,230058264~11640976,301283769~5609120,349721624~5844567,363514600~4814024,384505041~360305,397020628~4740508] 2024-09-21T10:34:09.858 INFO:tasks.workunit.client.0.smithi070.stdout:from 14 to 16 diff [16883580~4715959,25706100~3951165,53413917~6408717,98522774~6314663,144777497~6623225,210226943~2559490,225041157~16658083,301283769~5609120,351227316~4338875,363514600~4814024,380931932~31586,384505041~360305,395383365~6377771,409717800~2000564] 2024-09-21T10:34:09.859 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [16883580~4715959,25706100~3951165,53413917~6408717,98522774~6314663,144777497~6623225,210226943~2559490,225041157~16658083,301283769~5609120,351227316~4338875,363514600~4814024,380931932~31586,384505041~360305,395383365~6377771,409717800~2000564] 2024-09-21T10:34:09.864 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [16883580~4715959,25706100~3951165,53413917~6408717,98522774~6314663,144777497~6623225,210226943~2559490,225041157~16658083,301283769~5609120,349721624~5844567,363514600~4814024,380931932~31586,384505041~360305,395383365~6377771,409717800~2000564] 2024-09-21T10:34:09.864 INFO:tasks.workunit.client.0.smithi070.stdout:from 14 to 17 diff [15258652~7024612,25706100~3951165,53413917~6408717,66238036~7394424,95398996~13719063,115435950~5729260,144777497~6623225,168967775~4510479,209930960~3307546,225041157~16658083,269871960~1086953,301283769~5609120,351227316~4338875,363514600~4814024,380931932~31586,384505041~360305,395383365~6377771,409717800~2000564] 2024-09-21T10:34:09.864 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~7024612,25706100~3951165,53413917~6408717,66238036~7394424,95398996~13719063,115435950~5729260,144777497~6623225,168967775~4510479,209930960~3307546,225041157~16658083,269871960~1086953,301283769~5609120,351227316~4338875,363514600~4814024,380931932~31586,384505041~360305,395383365~6377771,409717800~2000564] 2024-09-21T10:34:09.869 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~7024612,25706100~3951165,53413917~6408717,66238036~7394424,95398996~13719063,115435950~5729260,144777497~6623225,168967775~4510479,209930960~3307546,225041157~16658083,269871960~1086953,301283769~5609120,349721624~5844567,363514600~4814024,380931932~31586,384505041~360305,395383365~6377771,409717800~2000564] 2024-09-21T10:34:09.870 INFO:tasks.workunit.client.0.smithi070.stdout:from 14 to 18 diff [15258652~7024612,25706100~3951165,53413917~6408717,66238036~7394424,95398996~13719063,115435950~5729260,124591324~5118882,144777497~6623225,168967775~4510479,208968224~8168491,225041157~16658083,265149402~5809511,291766799~42181,292083899~1585781,301283769~5609120,310892222~4271109,351227316~6557705,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:09.870 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~7024612,25706100~3951165,53413917~6408717,66238036~7394424,95398996~13719063,115435950~5729260,124591324~5118882,144777497~6623225,168967775~4510479,208968224~8168491,225041157~16658083,265149402~5809511,291766799~42181,292083899~1585781,301283769~5609120,310892222~4271109,351227316~6557705,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:09.876 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~7024612,25706100~3951165,53413917~6408717,66238036~7394424,95398996~13719063,115435950~5729260,123802561~5907645,144777497~6623225,168967775~4510479,208968224~8168491,225041157~16658083,265149402~5809511,290638652~3031028,301283769~5609120,310892222~4271109,349721624~8063397,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:09.876 INFO:tasks.workunit.client.0.smithi070.stdout:from 14 to 19 diff [15258652~8431860,25706100~3951165,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,144777497~7370767,168967775~4510479,208968224~9244408,225041157~16658083,265149402~5809511,282279101~6816084,291766799~42181,292083899~1585781,301283769~5609120,310892222~4271109,333198080~5481608,351227316~6557705,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:09.877 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~3951165,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,144777497~7370767,168967775~4510479,208968224~9244408,225041157~16658083,265149402~5809511,282279101~6816084,291766799~42181,292083899~1585781,301283769~5609120,310892222~4271109,333198080~5481608,351227316~6557705,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:09.882 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~3951165,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,144777497~7370767,168967775~4510479,208968224~9244408,225041157~16658083,265149402~5809511,281018368~8076817,290638652~3031028,301283769~5609120,310892222~4271109,331350016~7329672,349721624~8063397,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:09.882 INFO:tasks.workunit.client.0.smithi070.stdout:from 15 to 16 diff [16883580~4715959,25706100~3951165,53413917~5605327,98522774~6314663,225041157~7372785,380931932~31586,395383365~5743695,409717800~2000564] 2024-09-21T10:34:09.882 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [16883580~4715959,25706100~3951165,53413917~5605327,98522774~6314663,225041157~7372785,380931932~31586,395383365~5743695,409717800~2000564] 2024-09-21T10:34:09.888 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [16883580~4715959,25706100~3951165,53413917~5605327,98522774~6314663,225041157~7372785,380931932~31586,395383365~5743695,409717800~2000564] 2024-09-21T10:34:09.888 INFO:tasks.workunit.client.0.smithi070.stdout:from 15 to 17 diff [15258652~7024612,25706100~3951165,53413917~5605327,66238036~7394424,95398996~13719063,115435950~5729260,168967775~4510479,209930960~3307546,225041157~7372785,269871960~1086953,364615164~692640,380931932~31586,395383365~5743695,409717800~2000564] 2024-09-21T10:34:09.888 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~7024612,25706100~3951165,53413917~5605327,66238036~7394424,95398996~13719063,115435950~5729260,168967775~4510479,209930960~3307546,225041157~7372785,269871960~1086953,364615164~692640,380931932~31586,395383365~5743695,409717800~2000564] 2024-09-21T10:34:09.893 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~7024612,25706100~3951165,53413917~5605327,66238036~7394424,95398996~13719063,115435950~5729260,168967775~4510479,209930960~3307546,225041157~7372785,269871960~1086953,364615164~692640,380931932~31586,395383365~5743695,409717800~2000564] 2024-09-21T10:34:09.894 INFO:tasks.workunit.client.0.smithi070.stdout:from 15 to 18 diff [15258652~7024612,25706100~3951165,53413917~5605327,66238036~7394424,95398996~13719063,115435950~5729260,124591324~5118882,168967775~4510479,208968224~8168491,225041157~7372785,265149402~5809511,291766799~42181,292083899~1585781,310892222~4271109,353876173~3908848,364615164~692640,380931932~31586,389651616~4254468,395383365~5743695,409717800~2000564] 2024-09-21T10:34:09.894 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~7024612,25706100~3951165,53413917~5605327,66238036~7394424,95398996~13719063,115435950~5729260,124591324~5118882,168967775~4510479,208968224~8168491,225041157~7372785,265149402~5809511,291766799~42181,292083899~1585781,310892222~4271109,353876173~3908848,364615164~692640,380931932~31586,389651616~4254468,395383365~5743695,409717800~2000564] 2024-09-21T10:34:09.899 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~7024612,25706100~3951165,53413917~5605327,66238036~7394424,95398996~13719063,115435950~5729260,123802561~5907645,168967775~4510479,208968224~8168491,225041157~7372785,265149402~5809511,290638652~3031028,310892222~4271109,353876173~3908848,364615164~692640,380931932~31586,389651616~4254468,395383365~5743695,409717800~2000564] 2024-09-21T10:34:09.899 INFO:tasks.workunit.client.0.smithi070.stdout:from 15 to 19 diff [15258652~8431860,25706100~3951165,47127613~11891631,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,146673284~5474980,168967775~4510479,208968224~9244408,225041157~7372785,265149402~5809511,282279101~6816084,291766799~42181,292083899~1585781,310892222~4271109,333198080~5481608,353876173~3908848,364615164~692640,380931932~31586,389651616~4254468,395383365~5743695,409717800~2000564] 2024-09-21T10:34:09.900 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~3951165,47127613~11891631,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,146673284~5474980,168967775~4510479,208968224~9244408,225041157~7372785,265149402~5809511,282279101~6816084,291766799~42181,292083899~1585781,310892222~4271109,333198080~5481608,353876173~3908848,364615164~692640,380931932~31586,389651616~4254468,395383365~5743695,409717800~2000564] 2024-09-21T10:34:09.905 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~3951165,47127613~11891631,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,146673284~5474980,168967775~4510479,208968224~9244408,225041157~7372785,265149402~5809511,281018368~8076817,290638652~3031028,310892222~4271109,331350016~7329672,353876173~3908848,364615164~692640,380931932~31586,389651616~4254468,395383365~5743695,409717800~2000564] 2024-09-21T10:34:09.906 INFO:tasks.workunit.client.0.smithi070.stdout:from 16 to 17 diff [15258652~7024612,66238036~7394424,95398996~13719063,115435950~5729260,168967775~4510479,209930960~3307546,269871960~1086953,364615164~692640] 2024-09-21T10:34:09.906 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~7024612,66238036~7394424,95398996~13719063,115435950~5729260,168967775~4510479,209930960~3307546,269871960~1086953,364615164~692640] 2024-09-21T10:34:09.912 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~7024612,66238036~7394424,95398996~13719063,115435950~5729260,168967775~4510479,209930960~3307546,269871960~1086953,364615164~692640] 2024-09-21T10:34:09.912 INFO:tasks.workunit.client.0.smithi070.stdout:from 16 to 18 diff [15258652~7024612,66238036~7394424,95398996~13719063,115435950~5729260,124591324~5118882,168967775~4510479,208968224~8168491,265149402~5809511,291766799~42181,292083899~1585781,310892222~4271109,353876173~3908848,364615164~692640,389651616~4254468,399140449~1040101] 2024-09-21T10:34:09.912 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~7024612,66238036~7394424,95398996~13719063,115435950~5729260,124591324~5118882,168967775~4510479,208968224~8168491,265149402~5809511,291766799~42181,292083899~1585781,310892222~4271109,353876173~3908848,364615164~692640,389651616~4254468,399140449~1040101] 2024-09-21T10:34:09.918 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~7024612,66238036~7394424,95398996~13719063,115435950~5729260,123802561~5907645,168967775~4510479,208968224~8168491,265149402~5809511,290638652~3031028,310892222~4271109,353876173~3908848,364615164~692640,389651616~4254468,399140449~1040101] 2024-09-21T10:34:09.918 INFO:tasks.workunit.client.0.smithi070.stdout:from 16 to 19 diff [15258652~8431860,47127613~6678875,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,146673284~5474980,168967775~4510479,208968224~9244408,265149402~5809511,282279101~6816084,291766799~42181,292083899~1585781,310892222~4271109,333198080~5481608,353876173~3908848,364615164~692640,389651616~4254468,399140449~1040101] 2024-09-21T10:34:09.918 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,47127613~6678875,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,146673284~5474980,168967775~4510479,208968224~9244408,265149402~5809511,282279101~6816084,291766799~42181,292083899~1585781,310892222~4271109,333198080~5481608,353876173~3908848,364615164~692640,389651616~4254468,399140449~1040101] 2024-09-21T10:34:09.926 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,47127613~6678875,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,146673284~5474980,168967775~4510479,208968224~9244408,265149402~5809511,281018368~8076817,290638652~3031028,310892222~4271109,331350016~7329672,353876173~3908848,364615164~692640,389651616~4254468,399140449~1040101] 2024-09-21T10:34:09.926 INFO:tasks.workunit.client.0.smithi070.stdout:from 17 to 18 diff [96121327~6900963,124591324~5118882,208968224~8168491,265149402~4866558,291766799~42181,292083899~1585781,310892222~4271109,353876173~3908848,389651616~4254468,399140449~1040101] 2024-09-21T10:34:09.926 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [96121327~6900963,124591324~5118882,208968224~8168491,265149402~4866558,291766799~42181,292083899~1585781,310892222~4271109,353876173~3908848,389651616~4254468,399140449~1040101] 2024-09-21T10:34:09.930 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [96121327~6900963,123802561~5907645,208968224~8168491,265149402~4866558,290638652~3031028,310892222~4271109,353876173~3908848,389651616~4254468,399140449~1040101] 2024-09-21T10:34:09.930 INFO:tasks.workunit.client.0.smithi070.stdout:from 17 to 19 diff [19737427~3953085,47127613~6678875,82127504~5163005,96121327~6900963,118871052~10839154,146673284~5474980,208968224~9244408,265149402~4866558,282279101~6816084,291766799~42181,292083899~1585781,310892222~4271109,333198080~5481608,353876173~3908848,389651616~4254468,399140449~1040101] 2024-09-21T10:34:09.930 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [19737427~3953085,47127613~6678875,82127504~5163005,96121327~6900963,118871052~10839154,146673284~5474980,208968224~9244408,265149402~4866558,282279101~6816084,291766799~42181,292083899~1585781,310892222~4271109,333198080~5481608,353876173~3908848,389651616~4254468,399140449~1040101] 2024-09-21T10:34:09.936 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [19737427~3953085,47127613~6678875,82127504~5163005,96121327~6900963,118871052~10839154,146673284~5474980,208968224~9244408,265149402~4866558,281018368~8076817,290638652~3031028,310892222~4271109,331350016~7329672,353876173~3908848,389651616~4254468,399140449~1040101] 2024-09-21T10:34:09.936 INFO:tasks.workunit.client.0.smithi070.stdout:from 18 to 19 diff [19737427~3953085,47127613~6678875,82127504~5163005,118871052~5826351,146673284~5474980,216496383~1716249,282279101~6816084,333198080~5481608] 2024-09-21T10:34:09.936 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [19737427~3953085,47127613~6678875,82127504~5163005,118871052~5826351,146673284~5474980,216496383~1716249,282279101~6816084,333198080~5481608] 2024-09-21T10:34:09.941 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [19737427~3953085,47127613~6678875,82127504~5163005,118871052~5826351,146673284~5474980,216496383~1716249,281018368~8076817,331350016~7329672] 2024-09-21T10:34:10.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:09 smithi070 ceph-mon[112139]: pgmap v1787: 105 pgs: 105 active+clean; 1.0 GiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 236 KiB/s rd, 35 MiB/s wr, 364 op/s 2024-09-21T10:34:10.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:09 smithi070 ceph-mon[112139]: osdmap e1130: 8 total, 8 up, 8 in 2024-09-21T10:34:10.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:09 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:34:10.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:09 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:34:10.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:09 smithi070 ceph-mon[107385]: pgmap v1787: 105 pgs: 105 active+clean; 1.0 GiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 236 KiB/s rd, 35 MiB/s wr, 364 op/s 2024-09-21T10:34:10.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:09 smithi070 ceph-mon[107385]: osdmap e1130: 8 total, 8 up, 8 in 2024-09-21T10:34:10.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:09 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:34:10.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:09 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:34:10.815 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:10.815 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:10.833 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~71128091,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,406847488~7442595] 2024-09-21T10:34:10.833 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:10.834 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:10.840 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:10.840 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:10.840 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:10.845 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:10.845 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:10.845 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:10.851 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~15806534,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~10608093,206063683~12148949,224000747~27482416,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:10.851 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:10.851 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:10.857 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~11494533,82127504~5163005,92274688~16843371,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180722510~7393752,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:10.857 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:10.857 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:10.863 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,140524653~20336554,163577856~15649170,181624406~6491856,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:10.863 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12312179,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~30318160,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:10.863 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12312179,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~30318160,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:10.869 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25706100~12312179,39931377~204155,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,140524653~20336554,163577856~15649170,181624406~6491856,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~12651312,298351917~30318160,331350016~16429788,349721624~8063397,360853018~1275580,363514600~40385472,409717800~2000564] 2024-09-21T10:34:10.869 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~16083710,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:10.870 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~16083710,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:10.875 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,144777497~16083710,163577856~9971725,174552456~4674570,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,254252427~26661514,281018368~12651312,298351917~23957216,331350016~16429788,349721624~8063397,360853018~1275580,363514600~7729710,373355742~29023827,409717800~2000564] 2024-09-21T10:34:10.875 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to HEAD diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,259644006~21269935,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:10.875 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,259644006~21269935,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:10.881 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,174552456~4674570,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,259644006~21269935,281018368~12651312,298351917~23957216,331350016~16429788,349721624~8063397,360853018~1275580,363514600~7729710,376984253~2159000,379207810~23171759,409717800~2000564] 2024-09-21T10:34:10.881 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to HEAD diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,265149402~15764539,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,339701748~4594073,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:10.881 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,265149402~15764539,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,339701748~4594073,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:10.886 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,265149402~15764539,281018368~12651312,298351917~23086031,331350016~12945805,344769790~3010014,349721624~8063397,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:10.886 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to HEAD diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:10.887 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:10.892 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,281018368~12651312,298351917~23086031,331350016~7329672,343689268~606553,344769790~3010014,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:10.892 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to HEAD diff [15258652~8431860,25706100~5915387,32450955~218199,47127613~12695021,66238036~8171057,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~17055838,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:10.893 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~5915387,32450955~218199,47127613~12695021,66238036~8171057,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~17055838,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:10.898 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~5915387,32450955~218199,47127613~12695021,66238036~8171057,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,281018368~12651312,301105270~17055838,318527970~2909978,331350016~7329672,343689268~606553,344769790~3010014,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:10.898 INFO:tasks.workunit.client.0.smithi070.stdout:from 12 to HEAD diff [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~3276255,208968224~9244408,225041157~26442006,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~8064997,310892222~7268886,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:10.898 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~3276255,208968224~9244408,225041157~26442006,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~8064997,310892222~7268886,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:10.903 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~3276255,208968224~9244408,225041157~26442006,265149402~5809511,281018368~12651312,301105270~8064997,310892222~7268886,318527970~2909978,331350016~7329672,343689268~606553,344769790~3010014,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:10.904 INFO:tasks.workunit.client.0.smithi070.stdout:from 13 to HEAD diff [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,168967775~4510479,181624406~3276255,208968224~9244408,225041157~23034220,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301283769~7886498,310892222~4271109,333198080~5481608,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:10.904 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,168967775~4510479,181624406~3276255,208968224~9244408,225041157~23034220,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301283769~7886498,310892222~4271109,333198080~5481608,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:10.909 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,168967775~4510479,181624406~3276255,208968224~9244408,225041157~23034220,265149402~5809511,281018368~12651312,301283769~7886498,310892222~4271109,331350016~7329672,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:10.909 INFO:tasks.workunit.client.0.smithi070.stdout:from 14 to HEAD diff [15258652~8431860,25706100~3951165,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,144777497~7370767,168967775~4510479,208968224~9244408,225041157~16658083,265149402~5809511,282279101~6816084,291766799~42181,292083899~1585781,301283769~5609120,310892222~4271109,333198080~5481608,351227316~6557705,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:10.910 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~3951165,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,144777497~7370767,168967775~4510479,208968224~9244408,225041157~16658083,265149402~5809511,282279101~6816084,291766799~42181,292083899~1585781,301283769~5609120,310892222~4271109,333198080~5481608,351227316~6557705,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:10.915 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~3951165,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,144777497~7370767,168967775~4510479,208968224~9244408,225041157~16658083,265149402~5809511,281018368~8076817,290638652~3031028,301283769~5609120,310892222~4271109,331350016~7329672,349721624~8063397,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:10.915 INFO:tasks.workunit.client.0.smithi070.stdout:from 15 to HEAD diff [15258652~8431860,25706100~3951165,47127613~11891631,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,146673284~5474980,168967775~4510479,208968224~9244408,225041157~7372785,265149402~5809511,282279101~6816084,291766799~42181,292083899~1585781,310892222~4271109,333198080~5481608,353876173~3908848,364615164~692640,380931932~31586,389651616~4254468,395383365~5743695,409717800~2000564] 2024-09-21T10:34:10.915 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~3951165,47127613~11891631,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,146673284~5474980,168967775~4510479,208968224~9244408,225041157~7372785,265149402~5809511,282279101~6816084,291766799~42181,292083899~1585781,310892222~4271109,333198080~5481608,353876173~3908848,364615164~692640,380931932~31586,389651616~4254468,395383365~5743695,409717800~2000564] 2024-09-21T10:34:10.920 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~3951165,47127613~11891631,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,146673284~5474980,168967775~4510479,208968224~9244408,225041157~7372785,265149402~5809511,281018368~8076817,290638652~3031028,310892222~4271109,331350016~7329672,353876173~3908848,364615164~692640,380931932~31586,389651616~4254468,395383365~5743695,409717800~2000564] 2024-09-21T10:34:10.921 INFO:tasks.workunit.client.0.smithi070.stdout:from 16 to HEAD diff [15258652~8431860,47127613~6678875,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,146673284~5474980,168967775~4510479,208968224~9244408,265149402~5809511,282279101~6816084,291766799~42181,292083899~1585781,310892222~4271109,333198080~5481608,353876173~3908848,364615164~692640,389651616~4254468,399140449~1040101] 2024-09-21T10:34:10.921 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,47127613~6678875,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,146673284~5474980,168967775~4510479,208968224~9244408,265149402~5809511,282279101~6816084,291766799~42181,292083899~1585781,310892222~4271109,333198080~5481608,353876173~3908848,364615164~692640,389651616~4254468,399140449~1040101] 2024-09-21T10:34:10.926 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,47127613~6678875,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,146673284~5474980,168967775~4510479,208968224~9244408,265149402~5809511,281018368~8076817,290638652~3031028,310892222~4271109,331350016~7329672,353876173~3908848,364615164~692640,389651616~4254468,399140449~1040101] 2024-09-21T10:34:10.926 INFO:tasks.workunit.client.0.smithi070.stdout:from 17 to HEAD diff [19737427~3953085,47127613~6678875,82127504~5163005,96121327~6900963,118871052~10839154,146673284~5474980,208968224~9244408,265149402~4866558,282279101~6816084,291766799~42181,292083899~1585781,310892222~4271109,333198080~5481608,353876173~3908848,389651616~4254468,399140449~1040101] 2024-09-21T10:34:10.926 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [19737427~3953085,47127613~6678875,82127504~5163005,96121327~6900963,118871052~10839154,146673284~5474980,208968224~9244408,265149402~4866558,282279101~6816084,291766799~42181,292083899~1585781,310892222~4271109,333198080~5481608,353876173~3908848,389651616~4254468,399140449~1040101] 2024-09-21T10:34:10.931 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [19737427~3953085,47127613~6678875,82127504~5163005,96121327~6900963,118871052~10839154,146673284~5474980,208968224~9244408,265149402~4866558,281018368~8076817,290638652~3031028,310892222~4271109,331350016~7329672,353876173~3908848,389651616~4254468,399140449~1040101] 2024-09-21T10:34:11.819 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:11.819 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:11.833 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~71128091,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,406847488~7442595] 2024-09-21T10:34:11.833 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:11.834 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:11.839 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:11.839 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:11.840 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:11.845 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:11.846 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:11.846 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:11.852 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~15806534,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~10608093,206063683~12148949,224000747~27482416,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:11.852 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:11.852 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:11.857 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~11494533,82127504~5163005,92274688~16843371,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180722510~7393752,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:11.858 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:11.858 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:11.863 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,140524653~20336554,163577856~15649170,181624406~6491856,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:11.863 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12312179,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~30318160,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:11.863 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12312179,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~30318160,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:11.868 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25706100~12312179,39931377~204155,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,140524653~20336554,163577856~15649170,181624406~6491856,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~12651312,298351917~30318160,331350016~16429788,349721624~8063397,360853018~1275580,363514600~40385472,409717800~2000564] 2024-09-21T10:34:11.869 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~16083710,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:11.869 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~16083710,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:11.874 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,144777497~16083710,163577856~9971725,174552456~4674570,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,254252427~26661514,281018368~12651312,298351917~23957216,331350016~16429788,349721624~8063397,360853018~1275580,363514600~7729710,373355742~29023827,409717800~2000564] 2024-09-21T10:34:11.874 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to HEAD diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,259644006~21269935,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:11.875 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,259644006~21269935,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:11.880 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,174552456~4674570,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,259644006~21269935,281018368~12651312,298351917~23957216,331350016~16429788,349721624~8063397,360853018~1275580,363514600~7729710,376984253~2159000,379207810~23171759,409717800~2000564] 2024-09-21T10:34:11.880 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to HEAD diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,265149402~15764539,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,339701748~4594073,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:11.880 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,265149402~15764539,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,339701748~4594073,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:11.886 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,265149402~15764539,281018368~12651312,298351917~23086031,331350016~12945805,344769790~3010014,349721624~8063397,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:11.886 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to HEAD diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:11.886 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:11.891 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,281018368~12651312,298351917~23086031,331350016~7329672,343689268~606553,344769790~3010014,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:11.891 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to HEAD diff [15258652~8431860,25706100~5915387,32450955~218199,47127613~12695021,66238036~8171057,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~17055838,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:11.891 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~5915387,32450955~218199,47127613~12695021,66238036~8171057,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~17055838,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:11.897 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~5915387,32450955~218199,47127613~12695021,66238036~8171057,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,281018368~12651312,301105270~17055838,318527970~2909978,331350016~7329672,343689268~606553,344769790~3010014,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:11.897 INFO:tasks.workunit.client.0.smithi070.stdout:from 12 to HEAD diff [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~3276255,208968224~9244408,225041157~26442006,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~8064997,310892222~7268886,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:11.897 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~3276255,208968224~9244408,225041157~26442006,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~8064997,310892222~7268886,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:11.903 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~3276255,208968224~9244408,225041157~26442006,265149402~5809511,281018368~12651312,301105270~8064997,310892222~7268886,318527970~2909978,331350016~7329672,343689268~606553,344769790~3010014,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:11.903 INFO:tasks.workunit.client.0.smithi070.stdout:from 13 to HEAD diff [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,168967775~4510479,181624406~3276255,208968224~9244408,225041157~23034220,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301283769~7886498,310892222~4271109,333198080~5481608,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:11.903 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,168967775~4510479,181624406~3276255,208968224~9244408,225041157~23034220,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301283769~7886498,310892222~4271109,333198080~5481608,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:11.908 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,168967775~4510479,181624406~3276255,208968224~9244408,225041157~23034220,265149402~5809511,281018368~12651312,301283769~7886498,310892222~4271109,331350016~7329672,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:11.908 INFO:tasks.workunit.client.0.smithi070.stdout:from 14 to HEAD diff [15258652~8431860,25706100~3951165,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,144777497~7370767,168967775~4510479,208968224~9244408,225041157~16658083,265149402~5809511,282279101~6816084,291766799~42181,292083899~1585781,301283769~5609120,310892222~4271109,333198080~5481608,351227316~6557705,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:11.908 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~3951165,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,144777497~7370767,168967775~4510479,208968224~9244408,225041157~16658083,265149402~5809511,282279101~6816084,291766799~42181,292083899~1585781,301283769~5609120,310892222~4271109,333198080~5481608,351227316~6557705,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:11.913 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~3951165,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,144777497~7370767,168967775~4510479,208968224~9244408,225041157~16658083,265149402~5809511,281018368~8076817,290638652~3031028,301283769~5609120,310892222~4271109,331350016~7329672,349721624~8063397,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:11.913 INFO:tasks.workunit.client.0.smithi070.stdout:from 15 to HEAD diff [15258652~8431860,25706100~3951165,47127613~11891631,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,146673284~5474980,168967775~4510479,208968224~9244408,225041157~7372785,265149402~5809511,282279101~6816084,291766799~42181,292083899~1585781,310892222~4271109,333198080~5481608,353876173~3908848,364615164~692640,380931932~31586,389651616~4254468,395383365~5743695,409717800~2000564] 2024-09-21T10:34:11.913 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~3951165,47127613~11891631,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,146673284~5474980,168967775~4510479,208968224~9244408,225041157~7372785,265149402~5809511,282279101~6816084,291766799~42181,292083899~1585781,310892222~4271109,333198080~5481608,353876173~3908848,364615164~692640,380931932~31586,389651616~4254468,395383365~5743695,409717800~2000564] 2024-09-21T10:34:11.919 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~3951165,47127613~11891631,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,146673284~5474980,168967775~4510479,208968224~9244408,225041157~7372785,265149402~5809511,281018368~8076817,290638652~3031028,310892222~4271109,331350016~7329672,353876173~3908848,364615164~692640,380931932~31586,389651616~4254468,395383365~5743695,409717800~2000564] 2024-09-21T10:34:11.919 INFO:tasks.workunit.client.0.smithi070.stdout:from 16 to HEAD diff [15258652~8431860,47127613~6678875,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,146673284~5474980,168967775~4510479,208968224~9244408,265149402~5809511,282279101~6816084,291766799~42181,292083899~1585781,310892222~4271109,333198080~5481608,353876173~3908848,364615164~692640,389651616~4254468,399140449~1040101] 2024-09-21T10:34:11.919 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,47127613~6678875,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,146673284~5474980,168967775~4510479,208968224~9244408,265149402~5809511,282279101~6816084,291766799~42181,292083899~1585781,310892222~4271109,333198080~5481608,353876173~3908848,364615164~692640,389651616~4254468,399140449~1040101] 2024-09-21T10:34:11.924 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,47127613~6678875,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,146673284~5474980,168967775~4510479,208968224~9244408,265149402~5809511,281018368~8076817,290638652~3031028,310892222~4271109,331350016~7329672,353876173~3908848,364615164~692640,389651616~4254468,399140449~1040101] 2024-09-21T10:34:12.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:11 smithi160 ceph-mon[101879]: pgmap v1789: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 206 KiB/s rd, 35 MiB/s wr, 1.02k op/s 2024-09-21T10:34:12.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:11 smithi160 ceph-mon[101879]: osdmap e1131: 8 total, 8 up, 8 in 2024-09-21T10:34:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:11 smithi070 ceph-mon[107385]: pgmap v1789: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 206 KiB/s rd, 35 MiB/s wr, 1.02k op/s 2024-09-21T10:34:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:11 smithi070 ceph-mon[107385]: osdmap e1131: 8 total, 8 up, 8 in 2024-09-21T10:34:12.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:11 smithi070 ceph-mon[112139]: pgmap v1789: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 206 KiB/s rd, 35 MiB/s wr, 1.02k op/s 2024-09-21T10:34:12.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:11 smithi070 ceph-mon[112139]: osdmap e1131: 8 total, 8 up, 8 in 2024-09-21T10:34:12.829 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:12.829 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:12.844 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~71128091,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,406847488~7442595] 2024-09-21T10:34:12.844 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:12.844 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:12.850 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:12.850 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:12.850 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:12.855 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:12.855 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:12.855 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:12.861 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~15806534,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~10608093,206063683~12148949,224000747~27482416,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:12.861 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:12.861 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:12.866 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~11494533,82127504~5163005,92274688~16843371,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180722510~7393752,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:12.866 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:12.866 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:12.871 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,140524653~20336554,163577856~15649170,181624406~6491856,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:12.871 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12312179,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~30318160,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:12.872 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12312179,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~30318160,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:12.877 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25706100~12312179,39931377~204155,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,140524653~20336554,163577856~15649170,181624406~6491856,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~12651312,298351917~30318160,331350016~16429788,349721624~8063397,360853018~1275580,363514600~40385472,409717800~2000564] 2024-09-21T10:34:12.877 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~16083710,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:12.877 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~16083710,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:12.883 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,144777497~16083710,163577856~9971725,174552456~4674570,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,254252427~26661514,281018368~12651312,298351917~23957216,331350016~16429788,349721624~8063397,360853018~1275580,363514600~7729710,373355742~29023827,409717800~2000564] 2024-09-21T10:34:12.883 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to HEAD diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,259644006~21269935,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:12.883 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,259644006~21269935,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:12.888 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,174552456~4674570,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,259644006~21269935,281018368~12651312,298351917~23957216,331350016~16429788,349721624~8063397,360853018~1275580,363514600~7729710,376984253~2159000,379207810~23171759,409717800~2000564] 2024-09-21T10:34:12.888 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to HEAD diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,265149402~15764539,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,339701748~4594073,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:12.888 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,265149402~15764539,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,339701748~4594073,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:12.893 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,265149402~15764539,281018368~12651312,298351917~23086031,331350016~12945805,344769790~3010014,349721624~8063397,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:12.894 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to HEAD diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:12.894 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:12.899 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,281018368~12651312,298351917~23086031,331350016~7329672,343689268~606553,344769790~3010014,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:12.899 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to HEAD diff [15258652~8431860,25706100~5915387,32450955~218199,47127613~12695021,66238036~8171057,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~17055838,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:12.899 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~5915387,32450955~218199,47127613~12695021,66238036~8171057,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~17055838,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:12.904 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~5915387,32450955~218199,47127613~12695021,66238036~8171057,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,281018368~12651312,301105270~17055838,318527970~2909978,331350016~7329672,343689268~606553,344769790~3010014,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:12.904 INFO:tasks.workunit.client.0.smithi070.stdout:from 12 to HEAD diff [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~3276255,208968224~9244408,225041157~26442006,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~8064997,310892222~7268886,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:12.904 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~3276255,208968224~9244408,225041157~26442006,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~8064997,310892222~7268886,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:12.909 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~3276255,208968224~9244408,225041157~26442006,265149402~5809511,281018368~12651312,301105270~8064997,310892222~7268886,318527970~2909978,331350016~7329672,343689268~606553,344769790~3010014,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:12.909 INFO:tasks.workunit.client.0.smithi070.stdout:from 13 to HEAD diff [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,168967775~4510479,181624406~3276255,208968224~9244408,225041157~23034220,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301283769~7886498,310892222~4271109,333198080~5481608,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:12.909 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,168967775~4510479,181624406~3276255,208968224~9244408,225041157~23034220,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301283769~7886498,310892222~4271109,333198080~5481608,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:12.914 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,168967775~4510479,181624406~3276255,208968224~9244408,225041157~23034220,265149402~5809511,281018368~12651312,301283769~7886498,310892222~4271109,331350016~7329672,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:12.914 INFO:tasks.workunit.client.0.smithi070.stdout:from 14 to HEAD diff [15258652~8431860,25706100~3951165,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,144777497~7370767,168967775~4510479,208968224~9244408,225041157~16658083,265149402~5809511,282279101~6816084,291766799~42181,292083899~1585781,301283769~5609120,310892222~4271109,333198080~5481608,351227316~6557705,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:12.914 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~3951165,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,144777497~7370767,168967775~4510479,208968224~9244408,225041157~16658083,265149402~5809511,282279101~6816084,291766799~42181,292083899~1585781,301283769~5609120,310892222~4271109,333198080~5481608,351227316~6557705,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:12.920 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~3951165,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,144777497~7370767,168967775~4510479,208968224~9244408,225041157~16658083,265149402~5809511,281018368~8076817,290638652~3031028,301283769~5609120,310892222~4271109,331350016~7329672,349721624~8063397,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:12.920 INFO:tasks.workunit.client.0.smithi070.stdout:from 15 to HEAD diff [15258652~8431860,25706100~3951165,47127613~11891631,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,146673284~5474980,168967775~4510479,208968224~9244408,225041157~7372785,265149402~5809511,282279101~6816084,291766799~42181,292083899~1585781,310892222~4271109,333198080~5481608,353876173~3908848,364615164~692640,380931932~31586,389651616~4254468,395383365~5743695,409717800~2000564] 2024-09-21T10:34:12.920 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~3951165,47127613~11891631,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,146673284~5474980,168967775~4510479,208968224~9244408,225041157~7372785,265149402~5809511,282279101~6816084,291766799~42181,292083899~1585781,310892222~4271109,333198080~5481608,353876173~3908848,364615164~692640,380931932~31586,389651616~4254468,395383365~5743695,409717800~2000564] 2024-09-21T10:34:12.925 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~3951165,47127613~11891631,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,146673284~5474980,168967775~4510479,208968224~9244408,225041157~7372785,265149402~5809511,281018368~8076817,290638652~3031028,310892222~4271109,331350016~7329672,353876173~3908848,364615164~692640,380931932~31586,389651616~4254468,395383365~5743695,409717800~2000564] 2024-09-21T10:34:13.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:12 smithi160 ceph-mon[101879]: osdmap e1132: 8 total, 8 up, 8 in 2024-09-21T10:34:13.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:12 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:34:13.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:12 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:34:13.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:12 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:34:13.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:12 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:34:13.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:12 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:34:13.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:12 smithi070 ceph-mon[107385]: osdmap e1132: 8 total, 8 up, 8 in 2024-09-21T10:34:13.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:12 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:34:13.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:12 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:34:13.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:12 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:34:13.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:12 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:34:13.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:12 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:34:13.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:12 smithi070 ceph-mon[112139]: osdmap e1132: 8 total, 8 up, 8 in 2024-09-21T10:34:13.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:12 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:34:13.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:12 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:34:13.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:12 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:34:13.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:12 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:34:13.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:12 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:34:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:34:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:34:13] "GET /metrics HTTP/1.1" 200 36483 "" "Prometheus/2.51.0" 2024-09-21T10:34:13.849 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:13.850 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:13.867 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~71128091,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,406847488~7442595] 2024-09-21T10:34:13.867 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:13.867 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:13.875 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:13.875 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:13.875 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:13.881 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:13.882 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:13.882 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:13.888 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~15806534,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~10608093,206063683~12148949,224000747~27482416,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:13.888 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:13.888 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:13.894 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~11494533,82127504~5163005,92274688~16843371,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180722510~7393752,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:13.895 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:13.895 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:13.900 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,140524653~20336554,163577856~15649170,181624406~6491856,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:13.900 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12312179,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~30318160,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:13.900 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12312179,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~30318160,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:13.907 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25706100~12312179,39931377~204155,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,140524653~20336554,163577856~15649170,181624406~6491856,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~12651312,298351917~30318160,331350016~16429788,349721624~8063397,360853018~1275580,363514600~40385472,409717800~2000564] 2024-09-21T10:34:13.907 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~16083710,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:13.907 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~16083710,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:13.913 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,144777497~16083710,163577856~9971725,174552456~4674570,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,254252427~26661514,281018368~12651312,298351917~23957216,331350016~16429788,349721624~8063397,360853018~1275580,363514600~7729710,373355742~29023827,409717800~2000564] 2024-09-21T10:34:13.913 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to HEAD diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,259644006~21269935,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:13.913 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,259644006~21269935,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:13.918 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,174552456~4674570,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,259644006~21269935,281018368~12651312,298351917~23957216,331350016~16429788,349721624~8063397,360853018~1275580,363514600~7729710,376984253~2159000,379207810~23171759,409717800~2000564] 2024-09-21T10:34:13.918 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to HEAD diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,265149402~15764539,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,339701748~4594073,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:13.918 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,265149402~15764539,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,339701748~4594073,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:13.924 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,265149402~15764539,281018368~12651312,298351917~23086031,331350016~12945805,344769790~3010014,349721624~8063397,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:13.924 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to HEAD diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:13.924 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:13.929 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,281018368~12651312,298351917~23086031,331350016~7329672,343689268~606553,344769790~3010014,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:13.929 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to HEAD diff [15258652~8431860,25706100~5915387,32450955~218199,47127613~12695021,66238036~8171057,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~17055838,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:13.929 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~5915387,32450955~218199,47127613~12695021,66238036~8171057,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~17055838,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:13.934 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~5915387,32450955~218199,47127613~12695021,66238036~8171057,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,281018368~12651312,301105270~17055838,318527970~2909978,331350016~7329672,343689268~606553,344769790~3010014,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:13.935 INFO:tasks.workunit.client.0.smithi070.stdout:from 12 to HEAD diff [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~3276255,208968224~9244408,225041157~26442006,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~8064997,310892222~7268886,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:13.935 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~3276255,208968224~9244408,225041157~26442006,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~8064997,310892222~7268886,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:13.940 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~3276255,208968224~9244408,225041157~26442006,265149402~5809511,281018368~12651312,301105270~8064997,310892222~7268886,318527970~2909978,331350016~7329672,343689268~606553,344769790~3010014,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:13.940 INFO:tasks.workunit.client.0.smithi070.stdout:from 13 to HEAD diff [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,168967775~4510479,181624406~3276255,208968224~9244408,225041157~23034220,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301283769~7886498,310892222~4271109,333198080~5481608,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:13.940 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,168967775~4510479,181624406~3276255,208968224~9244408,225041157~23034220,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301283769~7886498,310892222~4271109,333198080~5481608,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:13.945 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,168967775~4510479,181624406~3276255,208968224~9244408,225041157~23034220,265149402~5809511,281018368~12651312,301283769~7886498,310892222~4271109,331350016~7329672,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:13.945 INFO:tasks.workunit.client.0.smithi070.stdout:from 14 to HEAD diff [15258652~8431860,25706100~3951165,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,144777497~7370767,168967775~4510479,208968224~9244408,225041157~16658083,265149402~5809511,282279101~6816084,291766799~42181,292083899~1585781,301283769~5609120,310892222~4271109,333198080~5481608,351227316~6557705,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:13.945 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~3951165,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,144777497~7370767,168967775~4510479,208968224~9244408,225041157~16658083,265149402~5809511,282279101~6816084,291766799~42181,292083899~1585781,301283769~5609120,310892222~4271109,333198080~5481608,351227316~6557705,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:13.950 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~3951165,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,144777497~7370767,168967775~4510479,208968224~9244408,225041157~16658083,265149402~5809511,281018368~8076817,290638652~3031028,301283769~5609120,310892222~4271109,331350016~7329672,349721624~8063397,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:14.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:13 smithi160 ceph-mon[101879]: pgmap v1792: 105 pgs: 105 active+clean; 1.2 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 174 KiB/s rd, 24 MiB/s wr, 3.15k op/s 2024-09-21T10:34:14.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:13 smithi160 ceph-mon[101879]: osdmap e1133: 8 total, 8 up, 8 in 2024-09-21T10:34:14.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:13 smithi070 ceph-mon[107385]: pgmap v1792: 105 pgs: 105 active+clean; 1.2 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 174 KiB/s rd, 24 MiB/s wr, 3.15k op/s 2024-09-21T10:34:14.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:13 smithi070 ceph-mon[107385]: osdmap e1133: 8 total, 8 up, 8 in 2024-09-21T10:34:14.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:13 smithi070 ceph-mon[112139]: pgmap v1792: 105 pgs: 105 active+clean; 1.2 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 174 KiB/s rd, 24 MiB/s wr, 3.15k op/s 2024-09-21T10:34:14.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:13 smithi070 ceph-mon[112139]: osdmap e1133: 8 total, 8 up, 8 in 2024-09-21T10:34:14.853 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:14.853 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:14.872 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~71128091,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,406847488~7442595] 2024-09-21T10:34:14.872 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:14.872 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:14.877 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:14.878 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:14.878 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:14.883 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:14.883 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:14.883 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:14.889 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~15806534,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~10608093,206063683~12148949,224000747~27482416,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:14.889 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:14.889 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:14.894 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~11494533,82127504~5163005,92274688~16843371,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180722510~7393752,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:14.894 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:14.894 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:14.900 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,140524653~20336554,163577856~15649170,181624406~6491856,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:14.900 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12312179,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~30318160,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:14.900 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12312179,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~30318160,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:14.905 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25706100~12312179,39931377~204155,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,140524653~20336554,163577856~15649170,181624406~6491856,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~12651312,298351917~30318160,331350016~16429788,349721624~8063397,360853018~1275580,363514600~40385472,409717800~2000564] 2024-09-21T10:34:14.905 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~16083710,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:14.905 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~16083710,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:14.910 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,144777497~16083710,163577856~9971725,174552456~4674570,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,254252427~26661514,281018368~12651312,298351917~23957216,331350016~16429788,349721624~8063397,360853018~1275580,363514600~7729710,373355742~29023827,409717800~2000564] 2024-09-21T10:34:14.910 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to HEAD diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,259644006~21269935,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:14.911 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,259644006~21269935,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:14.915 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,174552456~4674570,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,259644006~21269935,281018368~12651312,298351917~23957216,331350016~16429788,349721624~8063397,360853018~1275580,363514600~7729710,376984253~2159000,379207810~23171759,409717800~2000564] 2024-09-21T10:34:14.915 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to HEAD diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,265149402~15764539,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,339701748~4594073,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:14.915 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,265149402~15764539,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,339701748~4594073,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:14.921 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,265149402~15764539,281018368~12651312,298351917~23086031,331350016~12945805,344769790~3010014,349721624~8063397,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:14.921 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to HEAD diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:14.921 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:14.926 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,281018368~12651312,298351917~23086031,331350016~7329672,343689268~606553,344769790~3010014,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:14.926 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to HEAD diff [15258652~8431860,25706100~5915387,32450955~218199,47127613~12695021,66238036~8171057,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~17055838,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:14.926 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~5915387,32450955~218199,47127613~12695021,66238036~8171057,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~17055838,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:14.931 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~5915387,32450955~218199,47127613~12695021,66238036~8171057,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,281018368~12651312,301105270~17055838,318527970~2909978,331350016~7329672,343689268~606553,344769790~3010014,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:14.931 INFO:tasks.workunit.client.0.smithi070.stdout:from 12 to HEAD diff [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~3276255,208968224~9244408,225041157~26442006,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~8064997,310892222~7268886,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:14.932 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~3276255,208968224~9244408,225041157~26442006,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~8064997,310892222~7268886,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:14.937 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~3276255,208968224~9244408,225041157~26442006,265149402~5809511,281018368~12651312,301105270~8064997,310892222~7268886,318527970~2909978,331350016~7329672,343689268~606553,344769790~3010014,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:14.937 INFO:tasks.workunit.client.0.smithi070.stdout:from 13 to HEAD diff [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,168967775~4510479,181624406~3276255,208968224~9244408,225041157~23034220,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301283769~7886498,310892222~4271109,333198080~5481608,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:14.937 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,168967775~4510479,181624406~3276255,208968224~9244408,225041157~23034220,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301283769~7886498,310892222~4271109,333198080~5481608,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:14.942 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,168967775~4510479,181624406~3276255,208968224~9244408,225041157~23034220,265149402~5809511,281018368~12651312,301283769~7886498,310892222~4271109,331350016~7329672,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,395383365~6377771,409717800~2000564] 2024-09-21T10:34:15.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:14 smithi160 ceph-mon[101879]: osdmap e1134: 8 total, 8 up, 8 in 2024-09-21T10:34:15.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:14 smithi070 ceph-mon[107385]: osdmap e1134: 8 total, 8 up, 8 in 2024-09-21T10:34:15.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:14 smithi070 ceph-mon[112139]: osdmap e1134: 8 total, 8 up, 8 in 2024-09-21T10:34:15.860 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:15.860 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:15.879 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~71128091,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,406847488~7442595] 2024-09-21T10:34:15.879 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:15.879 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:15.885 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:15.885 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:15.885 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:15.890 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:15.891 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:15.891 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:15.896 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~15806534,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~10608093,206063683~12148949,224000747~27482416,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:15.896 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:15.897 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:15.902 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~11494533,82127504~5163005,92274688~16843371,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180722510~7393752,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:15.902 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:15.902 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:15.908 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,140524653~20336554,163577856~15649170,181624406~6491856,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:15.908 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12312179,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~30318160,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:15.908 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12312179,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~30318160,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:15.914 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25706100~12312179,39931377~204155,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,140524653~20336554,163577856~15649170,181624406~6491856,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~12651312,298351917~30318160,331350016~16429788,349721624~8063397,360853018~1275580,363514600~40385472,409717800~2000564] 2024-09-21T10:34:15.914 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~16083710,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:15.914 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~16083710,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:15.919 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,144777497~16083710,163577856~9971725,174552456~4674570,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,254252427~26661514,281018368~12651312,298351917~23957216,331350016~16429788,349721624~8063397,360853018~1275580,363514600~7729710,373355742~29023827,409717800~2000564] 2024-09-21T10:34:15.919 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to HEAD diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,259644006~21269935,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:15.919 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,259644006~21269935,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:15.925 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,174552456~4674570,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,259644006~21269935,281018368~12651312,298351917~23957216,331350016~16429788,349721624~8063397,360853018~1275580,363514600~7729710,376984253~2159000,379207810~23171759,409717800~2000564] 2024-09-21T10:34:15.925 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to HEAD diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,265149402~15764539,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,339701748~4594073,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:15.925 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,265149402~15764539,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,339701748~4594073,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:15.931 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,265149402~15764539,281018368~12651312,298351917~23086031,331350016~12945805,344769790~3010014,349721624~8063397,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:15.931 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to HEAD diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:15.931 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:15.937 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,281018368~12651312,298351917~23086031,331350016~7329672,343689268~606553,344769790~3010014,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:15.937 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to HEAD diff [15258652~8431860,25706100~5915387,32450955~218199,47127613~12695021,66238036~8171057,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~17055838,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:15.937 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~5915387,32450955~218199,47127613~12695021,66238036~8171057,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~17055838,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:15.942 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~5915387,32450955~218199,47127613~12695021,66238036~8171057,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,281018368~12651312,301105270~17055838,318527970~2909978,331350016~7329672,343689268~606553,344769790~3010014,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:15.942 INFO:tasks.workunit.client.0.smithi070.stdout:from 12 to HEAD diff [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~3276255,208968224~9244408,225041157~26442006,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~8064997,310892222~7268886,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:15.943 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~3276255,208968224~9244408,225041157~26442006,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~8064997,310892222~7268886,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:15.948 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~3951165,32450955~218199,47127613~12695021,66238036~7394424,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~3276255,208968224~9244408,225041157~26442006,265149402~5809511,281018368~12651312,301105270~8064997,310892222~7268886,318527970~2909978,331350016~7329672,343689268~606553,344769790~3010014,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:16.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:15 smithi160 ceph-mon[101879]: pgmap v1795: 105 pgs: 105 active+clean; 1.1 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 205 KiB/s rd, 11 MiB/s wr, 4.81k op/s 2024-09-21T10:34:16.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:15 smithi160 ceph-mon[101879]: osdmap e1135: 8 total, 8 up, 8 in 2024-09-21T10:34:16.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:15 smithi070 ceph-mon[107385]: pgmap v1795: 105 pgs: 105 active+clean; 1.1 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 205 KiB/s rd, 11 MiB/s wr, 4.81k op/s 2024-09-21T10:34:16.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:15 smithi070 ceph-mon[107385]: osdmap e1135: 8 total, 8 up, 8 in 2024-09-21T10:34:16.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:15 smithi070 ceph-mon[112139]: pgmap v1795: 105 pgs: 105 active+clean; 1.1 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 205 KiB/s rd, 11 MiB/s wr, 4.81k op/s 2024-09-21T10:34:16.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:15 smithi070 ceph-mon[112139]: osdmap e1135: 8 total, 8 up, 8 in 2024-09-21T10:34:16.864 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:16.864 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:16.887 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~71128091,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,406847488~7442595] 2024-09-21T10:34:16.887 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:16.887 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:16.893 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:16.893 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:16.893 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:16.899 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:16.899 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:16.899 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:16.905 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~15806534,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~10608093,206063683~12148949,224000747~27482416,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:16.905 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:16.905 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:16.911 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~11494533,82127504~5163005,92274688~16843371,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180722510~7393752,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:16.911 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:16.911 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:16.917 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,140524653~20336554,163577856~15649170,181624406~6491856,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:16.917 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12312179,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~30318160,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:16.917 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12312179,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~30318160,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:16.923 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25706100~12312179,39931377~204155,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,140524653~20336554,163577856~15649170,181624406~6491856,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~12651312,298351917~30318160,331350016~16429788,349721624~8063397,360853018~1275580,363514600~40385472,409717800~2000564] 2024-09-21T10:34:16.923 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~16083710,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:16.923 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~16083710,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:16.928 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,144777497~16083710,163577856~9971725,174552456~4674570,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,254252427~26661514,281018368~12651312,298351917~23957216,331350016~16429788,349721624~8063397,360853018~1275580,363514600~7729710,373355742~29023827,409717800~2000564] 2024-09-21T10:34:16.929 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to HEAD diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,259644006~21269935,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:16.929 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,259644006~21269935,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:16.934 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,174552456~4674570,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,259644006~21269935,281018368~12651312,298351917~23957216,331350016~16429788,349721624~8063397,360853018~1275580,363514600~7729710,376984253~2159000,379207810~23171759,409717800~2000564] 2024-09-21T10:34:16.934 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to HEAD diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,265149402~15764539,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,339701748~4594073,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:16.934 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,265149402~15764539,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,339701748~4594073,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:16.940 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,265149402~15764539,281018368~12651312,298351917~23086031,331350016~12945805,344769790~3010014,349721624~8063397,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:16.940 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to HEAD diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:16.940 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:16.945 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,281018368~12651312,298351917~23086031,331350016~7329672,343689268~606553,344769790~3010014,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:16.945 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to HEAD diff [15258652~8431860,25706100~5915387,32450955~218199,47127613~12695021,66238036~8171057,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~17055838,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:16.946 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~5915387,32450955~218199,47127613~12695021,66238036~8171057,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,301105270~17055838,318527970~2909978,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:16.951 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~5915387,32450955~218199,47127613~12695021,66238036~8171057,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,281018368~12651312,301105270~17055838,318527970~2909978,331350016~7329672,343689268~606553,344769790~3010014,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:17.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:16 smithi070 ceph-mon[107385]: osdmap e1136: 8 total, 8 up, 8 in 2024-09-21T10:34:17.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:16 smithi070 ceph-mon[112139]: osdmap e1136: 8 total, 8 up, 8 in 2024-09-21T10:34:17.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:16 smithi160 ceph-mon[101879]: osdmap e1136: 8 total, 8 up, 8 in 2024-09-21T10:34:17.872 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:17.872 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:17.894 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~71128091,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,406847488~7442595] 2024-09-21T10:34:17.894 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:17.894 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:17.900 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:17.900 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:17.900 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:17.906 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:17.906 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:17.907 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:17.912 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~15806534,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~10608093,206063683~12148949,224000747~27482416,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:17.912 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:17.912 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:17.918 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~11494533,82127504~5163005,92274688~16843371,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180722510~7393752,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:17.918 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:17.918 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:17.924 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,140524653~20336554,163577856~15649170,181624406~6491856,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:17.924 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12312179,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~30318160,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:17.924 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12312179,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~30318160,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:17.930 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25706100~12312179,39931377~204155,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,140524653~20336554,163577856~15649170,181624406~6491856,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~12651312,298351917~30318160,331350016~16429788,349721624~8063397,360853018~1275580,363514600~40385472,409717800~2000564] 2024-09-21T10:34:17.930 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~16083710,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:17.930 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~16083710,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:17.935 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,144777497~16083710,163577856~9971725,174552456~4674570,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,254252427~26661514,281018368~12651312,298351917~23957216,331350016~16429788,349721624~8063397,360853018~1275580,363514600~7729710,373355742~29023827,409717800~2000564] 2024-09-21T10:34:17.935 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to HEAD diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,259644006~21269935,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:17.935 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,259644006~21269935,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:17.941 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,174552456~4674570,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,259644006~21269935,281018368~12651312,298351917~23957216,331350016~16429788,349721624~8063397,360853018~1275580,363514600~7729710,376984253~2159000,379207810~23171759,409717800~2000564] 2024-09-21T10:34:17.942 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to HEAD diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,265149402~15764539,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,339701748~4594073,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:17.942 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,265149402~15764539,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,339701748~4594073,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:17.947 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,265149402~15764539,281018368~12651312,298351917~23086031,331350016~12945805,344769790~3010014,349721624~8063397,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:17.947 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to HEAD diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:17.947 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,343689268~606553,347289030~490774,351227316~6557705,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:17.952 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,115435950~14274256,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,208968224~9244408,224814781~26668382,265149402~5809511,281018368~12651312,298351917~23086031,331350016~7329672,343689268~606553,344769790~3010014,349721624~8063397,360853018~1275580,363514600~4814024,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:18.058 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:18.058 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:18.079 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~71128091,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,406847488~7442595] 2024-09-21T10:34:18.079 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:18.079 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:18.085 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:18.085 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:18.085 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:18.090 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:18.090 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:18.090 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:18.096 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~15806534,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~10608093,206063683~12148949,224000747~27482416,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:18.096 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:18.096 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:18.101 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~11494533,82127504~5163005,92274688~16843371,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180722510~7393752,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:18.101 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:18.102 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:18.107 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,140524653~20336554,163577856~15649170,181624406~6491856,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:18.107 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12312179,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~30318160,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:18.107 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12312179,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~30318160,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:18.112 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25706100~12312179,39931377~204155,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,140524653~20336554,163577856~15649170,181624406~6491856,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~12651312,298351917~30318160,331350016~16429788,349721624~8063397,360853018~1275580,363514600~40385472,409717800~2000564] 2024-09-21T10:34:18.112 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~16083710,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:18.112 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~16083710,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:18.117 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,144777497~16083710,163577856~9971725,174552456~4674570,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,254252427~26661514,281018368~12651312,298351917~23957216,331350016~16429788,349721624~8063397,360853018~1275580,363514600~7729710,373355742~29023827,409717800~2000564] 2024-09-21T10:34:18.117 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to HEAD diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,259644006~21269935,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:18.117 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,259644006~21269935,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:18.122 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,174552456~4674570,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,259644006~21269935,281018368~12651312,298351917~23957216,331350016~16429788,349721624~8063397,360853018~1275580,363514600~7729710,376984253~2159000,379207810~23171759,409717800~2000564] 2024-09-21T10:34:18.123 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to HEAD diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,265149402~15764539,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,339701748~4594073,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:18.123 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,265149402~15764539,282279101~8359551,291766799~42181,292083899~1585781,298351917~23086031,333198080~5481608,339701748~4594073,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~4254468,394221093~8158476,409717800~2000564] 2024-09-21T10:34:18.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:17 smithi070 ceph-mon[107385]: pgmap v1798: 105 pgs: 105 active+clean; 1.1 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 144 KiB/s rd, 5.2 KiB/s wr, 3.33k op/s 2024-09-21T10:34:18.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:17 smithi070 ceph-mon[107385]: osdmap e1137: 8 total, 8 up, 8 in 2024-09-21T10:34:18.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:17 smithi070 ceph-mon[112139]: pgmap v1798: 105 pgs: 105 active+clean; 1.1 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 144 KiB/s rd, 5.2 KiB/s wr, 3.33k op/s 2024-09-21T10:34:18.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:17 smithi070 ceph-mon[112139]: osdmap e1137: 8 total, 8 up, 8 in 2024-09-21T10:34:18.128 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,157423190~3438017,166204955~7344626,175217960~2272728,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,265149402~15764539,281018368~12651312,298351917~23086031,331350016~12945805,344769790~3010014,349721624~8063397,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,384505041~360305,389651616~12727953,409717800~2000564] 2024-09-21T10:34:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:17 smithi160 ceph-mon[101879]: pgmap v1798: 105 pgs: 105 active+clean; 1.1 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 144 KiB/s rd, 5.2 KiB/s wr, 3.33k op/s 2024-09-21T10:34:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:17 smithi160 ceph-mon[101879]: osdmap e1137: 8 total, 8 up, 8 in 2024-09-21T10:34:19.047 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:19.047 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:19.068 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~71128091,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,406847488~7442595] 2024-09-21T10:34:19.068 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:19.068 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:19.074 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:19.074 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:19.074 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:19.080 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:19.080 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:19.080 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:19.085 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~15806534,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~10608093,206063683~12148949,224000747~27482416,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:19.085 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:19.086 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:19.091 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~11494533,82127504~5163005,92274688~16843371,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180722510~7393752,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:19.091 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:19.091 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:19.096 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,140524653~20336554,163577856~15649170,181624406~6491856,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:19.096 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12312179,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~30318160,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:19.096 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12312179,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~30318160,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:19.102 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25706100~12312179,39931377~204155,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,140524653~20336554,163577856~15649170,181624406~6491856,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~12651312,298351917~30318160,331350016~16429788,349721624~8063397,360853018~1275580,363514600~40385472,409717800~2000564] 2024-09-21T10:34:19.102 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~16083710,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:19.102 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~16083710,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:19.107 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,144777497~16083710,163577856~9971725,174552456~4674570,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,254252427~26661514,281018368~12651312,298351917~23957216,331350016~16429788,349721624~8063397,360853018~1275580,363514600~7729710,373355742~29023827,409717800~2000564] 2024-09-21T10:34:19.107 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to HEAD diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,259644006~21269935,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:19.107 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,259644006~21269935,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,376984253~2159000,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:19.112 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [15258652~8431860,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,144777497~7370767,152797416~596734,157423190~3438017,166204955~7344626,174552456~4674570,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,259644006~21269935,281018368~12651312,298351917~23957216,331350016~16429788,349721624~8063397,360853018~1275580,363514600~7729710,376984253~2159000,379207810~23171759,409717800~2000564] 2024-09-21T10:34:19.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:18 smithi070 ceph-mon[107385]: osdmap e1138: 8 total, 8 up, 8 in 2024-09-21T10:34:19.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:18 smithi070 ceph-mon[107385]: osdmap e1139: 8 total, 8 up, 8 in 2024-09-21T10:34:19.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:18 smithi070 ceph-mon[112139]: osdmap e1138: 8 total, 8 up, 8 in 2024-09-21T10:34:19.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:18 smithi070 ceph-mon[112139]: osdmap e1139: 8 total, 8 up, 8 in 2024-09-21T10:34:19.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:18 smithi160 ceph-mon[101879]: osdmap e1138: 8 total, 8 up, 8 in 2024-09-21T10:34:19.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:18 smithi160 ceph-mon[101879]: osdmap e1139: 8 total, 8 up, 8 in 2024-09-21T10:34:20.051 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:20.051 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:20.077 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~71128091,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,406847488~7442595] 2024-09-21T10:34:20.077 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:20.077 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:20.084 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:20.084 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:20.084 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:20.090 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:20.090 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:20.090 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:20.096 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~15806534,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~10608093,206063683~12148949,224000747~27482416,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:20.096 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:20.096 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:20.102 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~11494533,82127504~5163005,92274688~16843371,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180722510~7393752,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:20.102 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:20.102 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:20.108 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,140524653~20336554,163577856~15649170,181624406~6491856,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:20.108 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12312179,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~30318160,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:20.108 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12312179,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~30318160,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:20.113 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25706100~12312179,39931377~204155,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,140524653~20336554,163577856~15649170,181624406~6491856,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~12651312,298351917~30318160,331350016~16429788,349721624~8063397,360853018~1275580,363514600~40385472,409717800~2000564] 2024-09-21T10:34:20.113 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~16083710,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:20.113 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12312179,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,144777497~16083710,165324500~859426,166204955~7344626,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224814781~26668382,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~23957216,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~7729710,373355742~5852068,380931932~31586,381111195~1866659,384505041~9401043,394221093~8158476,409717800~2000564] 2024-09-21T10:34:20.118 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25706100~12312179,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,144777497~16083710,163577856~9971725,174552456~4674570,181624406~6491856,191528500~10608093,206063683~12148949,224814781~26668382,254252427~26661514,281018368~12651312,298351917~23957216,331350016~16429788,349721624~8063397,360853018~1275580,363514600~7729710,373355742~29023827,409717800~2000564] 2024-09-21T10:34:20.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:20 smithi160 ceph-mon[101879]: pgmap v1802: 105 pgs: 105 active+clean; 1.1 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 142 KiB/s rd, 5.6 KiB/s wr, 1.20k op/s 2024-09-21T10:34:20.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:20 smithi160 ceph-mon[101879]: osdmap e1140: 8 total, 8 up, 8 in 2024-09-21T10:34:20.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:20 smithi070 ceph-mon[107385]: pgmap v1802: 105 pgs: 105 active+clean; 1.1 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 142 KiB/s rd, 5.6 KiB/s wr, 1.20k op/s 2024-09-21T10:34:20.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:20 smithi070 ceph-mon[107385]: osdmap e1140: 8 total, 8 up, 8 in 2024-09-21T10:34:20.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:20 smithi070 ceph-mon[112139]: pgmap v1802: 105 pgs: 105 active+clean; 1.1 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 142 KiB/s rd, 5.6 KiB/s wr, 1.20k op/s 2024-09-21T10:34:20.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:20 smithi070 ceph-mon[112139]: osdmap e1140: 8 total, 8 up, 8 in 2024-09-21T10:34:21.056 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:21.057 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:21.080 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~71128091,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,406847488~7442595] 2024-09-21T10:34:21.080 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:21.080 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:21.087 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:21.087 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:21.087 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:21.093 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:21.093 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:21.093 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:21.099 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~15806534,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~10608093,206063683~12148949,224000747~27482416,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:21.099 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:21.099 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:21.104 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~11494533,82127504~5163005,92274688~16843371,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180722510~7393752,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:21.105 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:21.105 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:21.110 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,140524653~20336554,163577856~15649170,181624406~6491856,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:21.110 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12312179,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~30318160,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:21.111 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12312179,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,298351917~30318160,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~1275580,363514600~15693210,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:21.115 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25706100~12312179,39931377~204155,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,140524653~20336554,163577856~15649170,181624406~6491856,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~12651312,298351917~30318160,331350016~16429788,349721624~8063397,360853018~1275580,363514600~40385472,409717800~2000564] 2024-09-21T10:34:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:21 smithi160 ceph-mon[101879]: osdmap e1141: 8 total, 8 up, 8 in 2024-09-21T10:34:21.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:21.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:21 smithi070 ceph-mon[112139]: osdmap e1141: 8 total, 8 up, 8 in 2024-09-21T10:34:21.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:21.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:21 smithi070 ceph-mon[107385]: osdmap e1141: 8 total, 8 up, 8 in 2024-09-21T10:34:21.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:22.062 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:22.062 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:22.083 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~71128091,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,406847488~7442595] 2024-09-21T10:34:22.083 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:22.083 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:22.088 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:22.088 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:22.088 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:22.094 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:22.094 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:22.094 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:22.100 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~15806534,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~10608093,206063683~12148949,224000747~27482416,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:22.100 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:22.100 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:22.105 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~11494533,82127504~5163005,92274688~16843371,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180722510~7393752,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:22.105 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:22.105 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,47127613~12695021,62918670~11490423,82127504~5163005,95398996~13719063,114683657~15026549,135872708~1754391,140524653~20336554,165324500~859426,166204955~8347501,175217960~4009066,181624406~6491856,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:22.110 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,47127613~12695021,62914560~11494533,82127504~5163005,95398996~13719063,113246208~16463998,135872708~1754391,140524653~20336554,163577856~15649170,181624406~6491856,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:22.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:22 smithi160 ceph-mon[101879]: pgmap v1805: 105 pgs: 105 active+clean; 1.1 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 914 op/s 2024-09-21T10:34:22.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:22 smithi160 ceph-mon[101879]: osdmap e1142: 8 total, 8 up, 8 in 2024-09-21T10:34:22.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:22 smithi070 ceph-mon[112139]: pgmap v1805: 105 pgs: 105 active+clean; 1.1 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 914 op/s 2024-09-21T10:34:22.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:22 smithi070 ceph-mon[112139]: osdmap e1142: 8 total, 8 up, 8 in 2024-09-21T10:34:22.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:22 smithi070 ceph-mon[107385]: pgmap v1805: 105 pgs: 105 active+clean; 1.1 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 914 op/s 2024-09-21T10:34:22.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:22 smithi070 ceph-mon[107385]: osdmap e1142: 8 total, 8 up, 8 in 2024-09-21T10:34:23.045 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:23.046 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:23.077 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~71128091,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,406847488~7442595] 2024-09-21T10:34:23.077 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:23.077 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:23.083 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:23.083 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:23.083 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:23.088 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:23.089 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:23.089 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:23.094 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~15806534,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~10608093,206063683~12148949,224000747~27482416,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:23.094 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:23.094 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~6387626,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~11490423,82127504~5163005,93118269~15999790,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~7393752,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:23.100 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~6387626,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~11494533,82127504~5163005,92274688~16843371,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180722510~7393752,191528500~10608093,206063683~12148949,224000747~27482416,254252427~26661514,281018368~47651709,331350016~16429788,349721624~8063397,360853018~43047054,409717800~2000564] 2024-09-21T10:34:23.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:23 smithi160 ceph-mon[101879]: osdmap e1143: 8 total, 8 up, 8 in 2024-09-21T10:34:23.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:23 smithi160 ceph-mon[101879]: osdmap e1144: 8 total, 8 up, 8 in 2024-09-21T10:34:23.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:34:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:34:23] "GET /metrics HTTP/1.1" 200 36485 "" "Prometheus/2.51.0" 2024-09-21T10:34:23.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:23 smithi070 ceph-mon[107385]: osdmap e1143: 8 total, 8 up, 8 in 2024-09-21T10:34:23.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:23 smithi070 ceph-mon[107385]: osdmap e1144: 8 total, 8 up, 8 in 2024-09-21T10:34:23.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:23 smithi070 ceph-mon[112139]: osdmap e1143: 8 total, 8 up, 8 in 2024-09-21T10:34:23.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:23 smithi070 ceph-mon[112139]: osdmap e1144: 8 total, 8 up, 8 in 2024-09-21T10:34:24.083 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:24.084 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:24.096 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~71128091,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,406847488~7442595] 2024-09-21T10:34:24.096 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:24.096 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:24.102 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:24.102 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:24.102 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:24.108 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:24.108 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:24.108 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~15802424,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~10608093,207781564~10431068,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:24.114 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~15806534,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~10608093,206063683~12148949,224000747~27482416,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:24.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:24 smithi160 ceph-mon[101879]: pgmap v1808: 105 pgs: 1 active+clean+snaptrim, 104 active+clean; 1.0 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 106 KiB/s rd, 7.2 KiB/s wr, 1.62k op/s 2024-09-21T10:34:24.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:24 smithi070 ceph-mon[107385]: pgmap v1808: 105 pgs: 1 active+clean+snaptrim, 104 active+clean; 1.0 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 106 KiB/s rd, 7.2 KiB/s wr, 1.62k op/s 2024-09-21T10:34:24.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:24 smithi070 ceph-mon[112139]: pgmap v1808: 105 pgs: 1 active+clean+snaptrim, 104 active+clean; 1.0 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 106 KiB/s rd, 7.2 KiB/s wr, 1.62k op/s 2024-09-21T10:34:25.082 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:25.082 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:25.104 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~71128091,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,406847488~7442595] 2024-09-21T10:34:25.104 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:25.104 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:25.110 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:25.110 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:25.110 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [2875569~7883915,15258652~8431860,25706100~12560767,39931377~204155,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~20843357,135872708~1754391,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:25.115 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [2875569~7883915,12582912~11107600,25165824~13101043,39931377~204155,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~22280806,135872708~1754391,137849028~23012179,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:25.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:25 smithi160 ceph-mon[101879]: osdmap e1145: 8 total, 8 up, 8 in 2024-09-21T10:34:25.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:25 smithi070 ceph-mon[107385]: osdmap e1145: 8 total, 8 up, 8 in 2024-09-21T10:34:25.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:25 smithi070 ceph-mon[112139]: osdmap e1145: 8 total, 8 up, 8 in 2024-09-21T10:34:26.089 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:26.089 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:26.107 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~71128091,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,406847488~7442595] 2024-09-21T10:34:26.107 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:26.108 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191528500~14535183,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~2000564] 2024-09-21T10:34:26.113 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~9886745,191528500~59954663,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,409717800~2000564] 2024-09-21T10:34:26.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:26 smithi160 ceph-mon[101879]: pgmap v1811: 105 pgs: 1 active+clean+snaptrim, 104 active+clean; 988 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 104 KiB/s rd, 7.2 KiB/s wr, 1.07k op/s 2024-09-21T10:34:26.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:26 smithi160 ceph-mon[101879]: osdmap e1146: 8 total, 8 up, 8 in 2024-09-21T10:34:26.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:26 smithi070 ceph-mon[107385]: pgmap v1811: 105 pgs: 1 active+clean+snaptrim, 104 active+clean; 988 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 104 KiB/s rd, 7.2 KiB/s wr, 1.07k op/s 2024-09-21T10:34:26.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:26 smithi070 ceph-mon[107385]: osdmap e1146: 8 total, 8 up, 8 in 2024-09-21T10:34:26.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:26 smithi070 ceph-mon[112139]: pgmap v1811: 105 pgs: 1 active+clean+snaptrim, 104 active+clean; 988 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 104 KiB/s rd, 7.2 KiB/s wr, 1.07k op/s 2024-09-21T10:34:26.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:26 smithi070 ceph-mon[112139]: osdmap e1146: 8 total, 8 up, 8 in 2024-09-21T10:34:27.102 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:27.102 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [1848814~8910670,15258652~8431860,25706100~16159399,45048342~14774292,62918670~17142262,82127504~5163005,93118269~16096930,114683657~23165371,138318126~22543081,165324500~859426,166204955~8347501,175217960~4009066,180722510~9519307,191243759~14819924,207781564~10431068,220782438~3218309,224340984~27142179,254252427~26661514,282279101~8359551,291766799~42181,292083899~1585781,294637427~34032650,333198080~5481608,339701748~5068042,347289030~490774,349416768~304856,351227316~6557705,360853018~18354792,380931932~31586,381111195~1866659,384505041~9401043,394221093~9678979,409717800~4572283] 2024-09-21T10:34:27.113 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [0~10759484,12582912~11107600,25165824~16699675,41943040~17879594,62914560~17146372,82127504~5163005,92274688~16940511,113246208~47614999,163577856~15649170,180355072~71128091,251658240~29255701,281018368~47651709,331350016~16429788,348127232~9657789,360853018~43047054,406847488~7442595] 2024-09-21T10:34:27.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:27 smithi160 ceph-mon[101879]: osdmap e1147: 8 total, 8 up, 8 in 2024-09-21T10:34:27.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:27 smithi070 ceph-mon[107385]: osdmap e1147: 8 total, 8 up, 8 in 2024-09-21T10:34:27.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:27 smithi070 ceph-mon[112139]: osdmap e1147: 8 total, 8 up, 8 in 2024-09-21T10:34:28.059 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/0.DiffIterateStress (39331 ms) 2024-09-21T10:34:28.059 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/0.DiffIterateParent 2024-09-21T10:34:28.060 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:34:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:28 smithi160 ceph-mon[101879]: pgmap v1814: 105 pgs: 105 active+clean; 909 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 71 KiB/s rd, 5.7 KiB/s wr, 790 op/s 2024-09-21T10:34:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:28 smithi160 ceph-mon[101879]: osdmap e1148: 8 total, 8 up, 8 in 2024-09-21T10:34:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:28 smithi160 ceph-mon[101879]: osdmap e1149: 8 total, 8 up, 8 in 2024-09-21T10:34:28.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:28 smithi070 ceph-mon[107385]: pgmap v1814: 105 pgs: 105 active+clean; 909 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 71 KiB/s rd, 5.7 KiB/s wr, 790 op/s 2024-09-21T10:34:28.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:28 smithi070 ceph-mon[107385]: osdmap e1148: 8 total, 8 up, 8 in 2024-09-21T10:34:28.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:28 smithi070 ceph-mon[107385]: osdmap e1149: 8 total, 8 up, 8 in 2024-09-21T10:34:28.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:28 smithi070 ceph-mon[112139]: pgmap v1814: 105 pgs: 105 active+clean; 909 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 71 KiB/s rd, 5.7 KiB/s wr, 790 op/s 2024-09-21T10:34:28.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:28 smithi070 ceph-mon[112139]: osdmap e1148: 8 total, 8 up, 8 in 2024-09-21T10:34:28.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:28 smithi070 ceph-mon[112139]: osdmap e1149: 8 total, 8 up, 8 in 2024-09-21T10:34:29.201 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/0.DiffIterateParent (1142 ms) 2024-09-21T10:34:29.201 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/0.DiffIterateIgnoreParent 2024-09-21T10:34:29.201 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:34:30.209 INFO:tasks.workunit.client.0.smithi070.stdout: wrote [925619~1166,2619318~90668,5963371~28353,6506929~55280,6727931~35550,9605173~38417,16073363~9415,16231267~58552,17483286~70649,19737988~7106] to clone 2024-09-21T10:34:30.209 INFO:tasks.workunit.client.0.smithi070.stdout: diff was [0~2709986,4194304~2569177,8388608~1254982,12582912~3706907,16777216~2967878] 2024-09-21T10:34:30.213 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/0.DiffIterateIgnoreParent (1011 ms) 2024-09-21T10:34:30.213 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/0.DiffIterateCallbackError 2024-09-21T10:34:30.213 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:34:30.232 INFO:tasks.workunit.client.0.smithi070.stdout: wrote [200923~34931,4851726~33343,8529183~89032,9687726~21363,12002259~44428,12379065~69399,12886095~40718,13790931~73584,18127624~4282,18871330~19012] 2024-09-21T10:34:30.242 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/0.DiffIterateCallbackError (29 ms) 2024-09-21T10:34:30.242 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/0.DiffIterateParentDiscard 2024-09-21T10:34:30.242 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:34:30.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:30 smithi070 ceph-mon[107385]: pgmap v1817: 105 pgs: 105 active+clean; 909 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 71 KiB/s rd, 5.7 KiB/s wr, 495 op/s 2024-09-21T10:34:30.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:30 smithi070 ceph-mon[107385]: osdmap e1150: 8 total, 8 up, 8 in 2024-09-21T10:34:30.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:30 smithi070 ceph-mon[112139]: pgmap v1817: 105 pgs: 105 active+clean; 909 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 71 KiB/s rd, 5.7 KiB/s wr, 495 op/s 2024-09-21T10:34:30.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:30 smithi070 ceph-mon[112139]: osdmap e1150: 8 total, 8 up, 8 in 2024-09-21T10:34:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:30 smithi160 ceph-mon[101879]: pgmap v1817: 105 pgs: 105 active+clean; 909 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 71 KiB/s rd, 5.7 KiB/s wr, 495 op/s 2024-09-21T10:34:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:30 smithi160 ceph-mon[101879]: osdmap e1150: 8 total, 8 up, 8 in 2024-09-21T10:34:31.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:31 smithi070 ceph-mon[107385]: osdmap e1151: 8 total, 8 up, 8 in 2024-09-21T10:34:31.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:31 smithi070 ceph-mon[112139]: osdmap e1151: 8 total, 8 up, 8 in 2024-09-21T10:34:31.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:31 smithi160 ceph-mon[101879]: osdmap e1151: 8 total, 8 up, 8 in 2024-09-21T10:34:32.208 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/0.DiffIterateParentDiscard (1965 ms) 2024-09-21T10:34:32.208 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/0.DiffIterateUnalignedSmall 2024-09-21T10:34:32.208 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:34:32.260 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/0.DiffIterateUnalignedSmall (53 ms) 2024-09-21T10:34:32.260 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/0.DiffIterateUnaligned 2024-09-21T10:34:32.260 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:34:32.335 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/0.DiffIterateUnaligned (75 ms) 2024-09-21T10:34:32.335 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/0.DiffIterateTryAcquireLock 2024-09-21T10:34:32.335 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:34:32.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:32 smithi160 ceph-mon[101879]: pgmap v1820: 105 pgs: 105 active+clean; 841 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 38 KiB/s rd, 6.0 MiB/s wr, 142 op/s 2024-09-21T10:34:32.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:32 smithi160 ceph-mon[101879]: osdmap e1152: 8 total, 8 up, 8 in 2024-09-21T10:34:32.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:32 smithi070 ceph-mon[107385]: pgmap v1820: 105 pgs: 105 active+clean; 841 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 38 KiB/s rd, 6.0 MiB/s wr, 142 op/s 2024-09-21T10:34:32.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:32 smithi070 ceph-mon[107385]: osdmap e1152: 8 total, 8 up, 8 in 2024-09-21T10:34:32.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:32 smithi070 ceph-mon[112139]: pgmap v1820: 105 pgs: 105 active+clean; 841 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 38 KiB/s rd, 6.0 MiB/s wr, 142 op/s 2024-09-21T10:34:32.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:32 smithi070 ceph-mon[112139]: osdmap e1152: 8 total, 8 up, 8 in 2024-09-21T10:34:33.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:33 smithi070 ceph-mon[107385]: osdmap e1153: 8 total, 8 up, 8 in 2024-09-21T10:34:33.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:33 smithi070 ceph-mon[107385]: osdmap e1154: 8 total, 8 up, 8 in 2024-09-21T10:34:33.378 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:34:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:34:33] "GET /metrics HTTP/1.1" 200 36485 "" "Prometheus/2.51.0" 2024-09-21T10:34:33.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:33 smithi070 ceph-mon[112139]: osdmap e1153: 8 total, 8 up, 8 in 2024-09-21T10:34:33.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:33 smithi070 ceph-mon[112139]: osdmap e1154: 8 total, 8 up, 8 in 2024-09-21T10:34:33.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:33 smithi160 ceph-mon[101879]: osdmap e1153: 8 total, 8 up, 8 in 2024-09-21T10:34:33.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:33 smithi160 ceph-mon[101879]: osdmap e1154: 8 total, 8 up, 8 in 2024-09-21T10:34:34.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:34 smithi070 ceph-mon[107385]: pgmap v1823: 105 pgs: 105 active+clean; 809 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 9.7 MiB/s wr, 314 op/s 2024-09-21T10:34:34.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:34 smithi070 ceph-mon[112139]: pgmap v1823: 105 pgs: 105 active+clean; 809 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 9.7 MiB/s wr, 314 op/s 2024-09-21T10:34:34.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:34 smithi160 ceph-mon[101879]: pgmap v1823: 105 pgs: 105 active+clean; 809 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 9.7 MiB/s wr, 314 op/s 2024-09-21T10:34:36.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:36 smithi070 ceph-mon[107385]: pgmap v1825: 105 pgs: 105 active+clean; 843 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.8 MiB/s rd, 7.0 MiB/s wr, 306 op/s 2024-09-21T10:34:36.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:36.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:36 smithi070 ceph-mon[112139]: pgmap v1825: 105 pgs: 105 active+clean; 843 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.8 MiB/s rd, 7.0 MiB/s wr, 306 op/s 2024-09-21T10:34:36.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:36 smithi160 ceph-mon[101879]: pgmap v1825: 105 pgs: 105 active+clean; 843 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.8 MiB/s rd, 7.0 MiB/s wr, 306 op/s 2024-09-21T10:34:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:37.371 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/0.DiffIterateTryAcquireLock (5036 ms) 2024-09-21T10:34:37.372 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/0.DiffIterateStriping 2024-09-21T10:34:37.372 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/test_librbd.cc:8958: Skipped 2024-09-21T10:34:37.372 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T10:34:37.372 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] DiffIterateTest/0.DiffIterateStriping (0 ms) 2024-09-21T10:34:38.040 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 19 tests from DiffIterateTest/0 (251363 ms total) 2024-09-21T10:34:38.040 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:34:38.040 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 19 tests from DiffIterateTest/1, where TypeParam = DiffIterateParams 2024-09-21T10:34:38.076 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/1.DiffIterate 2024-09-21T10:34:38.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:38 smithi070 ceph-mon[107385]: pgmap v1826: 105 pgs: 105 active+clean; 874 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 8.0 MiB/s wr, 287 op/s 2024-09-21T10:34:38.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:38 smithi070 ceph-mon[107385]: osdmap e1155: 8 total, 8 up, 8 in 2024-09-21T10:34:38.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:38 smithi070 ceph-mon[112139]: pgmap v1826: 105 pgs: 105 active+clean; 874 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 8.0 MiB/s wr, 287 op/s 2024-09-21T10:34:38.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:38 smithi070 ceph-mon[112139]: osdmap e1155: 8 total, 8 up, 8 in 2024-09-21T10:34:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:38 smithi160 ceph-mon[101879]: pgmap v1826: 105 pgs: 105 active+clean; 874 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 8.0 MiB/s wr, 287 op/s 2024-09-21T10:34:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:38 smithi160 ceph-mon[101879]: osdmap e1155: 8 total, 8 up, 8 in 2024-09-21T10:34:40.049 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:34:40.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:40 smithi160 ceph-mon[101879]: pgmap v1828: 73 pgs: 73 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:34:40.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:40 smithi160 ceph-mon[101879]: osdmap e1156: 8 total, 8 up, 8 in 2024-09-21T10:34:40.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:40 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1102044455' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-48","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:34:40.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:40 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-48","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:34:40.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:40 smithi070 ceph-mon[107385]: pgmap v1828: 73 pgs: 73 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:34:40.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:40 smithi070 ceph-mon[107385]: osdmap e1156: 8 total, 8 up, 8 in 2024-09-21T10:34:40.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:40 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1102044455' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-48","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:34:40.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:40 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-48","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:34:40.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:40 smithi070 ceph-mon[112139]: pgmap v1828: 73 pgs: 73 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:34:40.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:40 smithi070 ceph-mon[112139]: osdmap e1156: 8 total, 8 up, 8 in 2024-09-21T10:34:40.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:40 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1102044455' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-48","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:34:40.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:40 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-48","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:34:41.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:41 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-48","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:34:41.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:41 smithi160 ceph-mon[101879]: osdmap e1157: 8 total, 8 up, 8 in 2024-09-21T10:34:41.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:41 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-48","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:34:41.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:41 smithi070 ceph-mon[107385]: osdmap e1157: 8 total, 8 up, 8 in 2024-09-21T10:34:41.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:41 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi070-86016-48","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:34:41.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:41 smithi070 ceph-mon[112139]: osdmap e1157: 8 total, 8 up, 8 in 2024-09-21T10:34:42.088 INFO:tasks.workunit.client.0.smithi070.stdout: wrote [2288320~42351,4000398~92669,5356607~57225,5502353~595,8125852~38252,12398736~98021,13471798~52461,15597956~9809,16211716~51274,19449851~97768] 2024-09-21T10:34:42.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:42 smithi160 ceph-mon[101879]: pgmap v1831: 105 pgs: 10 creating+peering, 22 unknown, 73 active+clean; 582 KiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:34:42.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:42 smithi160 ceph-mon[101879]: osdmap e1158: 8 total, 8 up, 8 in 2024-09-21T10:34:42.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:42 smithi070 ceph-mon[107385]: pgmap v1831: 105 pgs: 10 creating+peering, 22 unknown, 73 active+clean; 582 KiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:34:42.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:42 smithi070 ceph-mon[107385]: osdmap e1158: 8 total, 8 up, 8 in 2024-09-21T10:34:42.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:42 smithi070 ceph-mon[112139]: pgmap v1831: 105 pgs: 10 creating+peering, 22 unknown, 73 active+clean; 582 KiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:34:42.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:42 smithi070 ceph-mon[112139]: osdmap e1158: 8 total, 8 up, 8 in 2024-09-21T10:34:43.086 INFO:tasks.workunit.client.0.smithi070.stdout: wrote [0~8388608,12582912~8388608] 2024-09-21T10:34:43.090 INFO:tasks.workunit.client.0.smithi070.stdout: diff was [0~8388608,12582912~8388608] 2024-09-21T10:34:43.094 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/1.DiffIterate (5019 ms) 2024-09-21T10:34:43.095 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/1.DiffIterateDeterministic 2024-09-21T10:34:43.095 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:34:43.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:43 smithi160 ceph-mon[101879]: osdmap e1159: 8 total, 8 up, 8 in 2024-09-21T10:34:43.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:43 smithi070 ceph-mon[107385]: osdmap e1159: 8 total, 8 up, 8 in 2024-09-21T10:34:43.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:34:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:34:43] "GET /metrics HTTP/1.1" 200 36483 "" "Prometheus/2.51.0" 2024-09-21T10:34:43.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:43 smithi070 ceph-mon[112139]: osdmap e1159: 8 total, 8 up, 8 in 2024-09-21T10:34:44.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:44 smithi160 ceph-mon[101879]: pgmap v1834: 105 pgs: 10 creating+peering, 6 unknown, 89 active+clean; 582 KiB data, 914 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:34:44.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:44 smithi160 ceph-mon[101879]: osdmap e1160: 8 total, 8 up, 8 in 2024-09-21T10:34:44.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:44 smithi070 ceph-mon[107385]: pgmap v1834: 105 pgs: 10 creating+peering, 6 unknown, 89 active+clean; 582 KiB data, 914 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:34:44.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:44 smithi070 ceph-mon[107385]: osdmap e1160: 8 total, 8 up, 8 in 2024-09-21T10:34:44.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:44 smithi070 ceph-mon[112139]: pgmap v1834: 105 pgs: 10 creating+peering, 6 unknown, 89 active+clean; 582 KiB data, 914 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:34:44.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:44 smithi070 ceph-mon[112139]: osdmap e1160: 8 total, 8 up, 8 in 2024-09-21T10:34:45.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:45 smithi160 ceph-mon[101879]: osdmap e1161: 8 total, 8 up, 8 in 2024-09-21T10:34:45.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:45 smithi070 ceph-mon[107385]: osdmap e1161: 8 total, 8 up, 8 in 2024-09-21T10:34:45.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:45 smithi070 ceph-mon[112139]: osdmap e1161: 8 total, 8 up, 8 in 2024-09-21T10:34:46.113 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:34:46.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:46 smithi160 ceph-mon[101879]: pgmap v1837: 105 pgs: 3 creating+peering, 102 active+clean; 11 MiB data, 687 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 207 KiB/s wr, 8 op/s 2024-09-21T10:34:46.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:46 smithi160 ceph-mon[101879]: osdmap e1162: 8 total, 8 up, 8 in 2024-09-21T10:34:46.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:46 smithi070 ceph-mon[107385]: pgmap v1837: 105 pgs: 3 creating+peering, 102 active+clean; 11 MiB data, 687 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 207 KiB/s wr, 8 op/s 2024-09-21T10:34:46.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:46 smithi070 ceph-mon[107385]: osdmap e1162: 8 total, 8 up, 8 in 2024-09-21T10:34:46.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:46 smithi070 ceph-mon[112139]: pgmap v1837: 105 pgs: 3 creating+peering, 102 active+clean; 11 MiB data, 687 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 207 KiB/s wr, 8 op/s 2024-09-21T10:34:46.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:46 smithi070 ceph-mon[112139]: osdmap e1162: 8 total, 8 up, 8 in 2024-09-21T10:34:47.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:47 smithi160 ceph-mon[101879]: osdmap e1163: 8 total, 8 up, 8 in 2024-09-21T10:34:47.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:47 smithi070 ceph-mon[107385]: osdmap e1163: 8 total, 8 up, 8 in 2024-09-21T10:34:47.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:47 smithi070 ceph-mon[112139]: osdmap e1163: 8 total, 8 up, 8 in 2024-09-21T10:34:48.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:48 smithi160 ceph-mon[101879]: pgmap v1840: 105 pgs: 105 active+clean; 19 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 298 KiB/s wr, 72 op/s 2024-09-21T10:34:48.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:48 smithi160 ceph-mon[101879]: osdmap e1164: 8 total, 8 up, 8 in 2024-09-21T10:34:48.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:48 smithi160 ceph-mon[101879]: osdmap e1165: 8 total, 8 up, 8 in 2024-09-21T10:34:48.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:48 smithi070 ceph-mon[107385]: pgmap v1840: 105 pgs: 105 active+clean; 19 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 298 KiB/s wr, 72 op/s 2024-09-21T10:34:48.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:48 smithi070 ceph-mon[107385]: osdmap e1164: 8 total, 8 up, 8 in 2024-09-21T10:34:48.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:48 smithi070 ceph-mon[107385]: osdmap e1165: 8 total, 8 up, 8 in 2024-09-21T10:34:48.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:48 smithi070 ceph-mon[112139]: pgmap v1840: 105 pgs: 105 active+clean; 19 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 298 KiB/s wr, 72 op/s 2024-09-21T10:34:48.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:48 smithi070 ceph-mon[112139]: osdmap e1164: 8 total, 8 up, 8 in 2024-09-21T10:34:48.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:48 smithi070 ceph-mon[112139]: osdmap e1165: 8 total, 8 up, 8 in 2024-09-21T10:34:49.066 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:34:50.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:50 smithi160 ceph-mon[101879]: pgmap v1843: 105 pgs: 105 active+clean; 19 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 90 KiB/s wr, 64 op/s 2024-09-21T10:34:50.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:50 smithi160 ceph-mon[101879]: osdmap e1166: 8 total, 8 up, 8 in 2024-09-21T10:34:50.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:50 smithi070 ceph-mon[107385]: pgmap v1843: 105 pgs: 105 active+clean; 19 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 90 KiB/s wr, 64 op/s 2024-09-21T10:34:50.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:50 smithi070 ceph-mon[107385]: osdmap e1166: 8 total, 8 up, 8 in 2024-09-21T10:34:50.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:50 smithi070 ceph-mon[112139]: pgmap v1843: 105 pgs: 105 active+clean; 19 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 90 KiB/s wr, 64 op/s 2024-09-21T10:34:50.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:50 smithi070 ceph-mon[112139]: osdmap e1166: 8 total, 8 up, 8 in 2024-09-21T10:34:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:51 smithi160 ceph-mon[101879]: osdmap e1167: 8 total, 8 up, 8 in 2024-09-21T10:34:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:51.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:51 smithi070 ceph-mon[107385]: osdmap e1167: 8 total, 8 up, 8 in 2024-09-21T10:34:51.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:51.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:51 smithi070 ceph-mon[112139]: osdmap e1167: 8 total, 8 up, 8 in 2024-09-21T10:34:51.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:52.083 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:34:52.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:52 smithi160 ceph-mon[101879]: pgmap v1846: 105 pgs: 105 active+clean; 20 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 KiB/s wr, 39 op/s 2024-09-21T10:34:52.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:52 smithi160 ceph-mon[101879]: osdmap e1168: 8 total, 8 up, 8 in 2024-09-21T10:34:52.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:52 smithi070 ceph-mon[107385]: pgmap v1846: 105 pgs: 105 active+clean; 20 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 KiB/s wr, 39 op/s 2024-09-21T10:34:52.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:52 smithi070 ceph-mon[107385]: osdmap e1168: 8 total, 8 up, 8 in 2024-09-21T10:34:52.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:52 smithi070 ceph-mon[112139]: pgmap v1846: 105 pgs: 105 active+clean; 20 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 KiB/s wr, 39 op/s 2024-09-21T10:34:52.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:52 smithi070 ceph-mon[112139]: osdmap e1168: 8 total, 8 up, 8 in 2024-09-21T10:34:53.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:53 smithi160 ceph-mon[101879]: osdmap e1169: 8 total, 8 up, 8 in 2024-09-21T10:34:53.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:53 smithi070 ceph-mon[107385]: osdmap e1169: 8 total, 8 up, 8 in 2024-09-21T10:34:53.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:34:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:34:53] "GET /metrics HTTP/1.1" 200 36460 "" "Prometheus/2.51.0" 2024-09-21T10:34:53.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:53 smithi070 ceph-mon[112139]: osdmap e1169: 8 total, 8 up, 8 in 2024-09-21T10:34:54.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:54 smithi160 ceph-mon[101879]: pgmap v1849: 105 pgs: 105 active+clean; 22 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 8.5 KiB/s wr, 104 op/s 2024-09-21T10:34:54.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:54 smithi160 ceph-mon[101879]: osdmap e1170: 8 total, 8 up, 8 in 2024-09-21T10:34:54.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:54 smithi070 ceph-mon[107385]: pgmap v1849: 105 pgs: 105 active+clean; 22 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 8.5 KiB/s wr, 104 op/s 2024-09-21T10:34:54.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:54 smithi070 ceph-mon[107385]: osdmap e1170: 8 total, 8 up, 8 in 2024-09-21T10:34:54.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:54 smithi070 ceph-mon[112139]: pgmap v1849: 105 pgs: 105 active+clean; 22 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 8.5 KiB/s wr, 104 op/s 2024-09-21T10:34:54.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:54 smithi070 ceph-mon[112139]: osdmap e1170: 8 total, 8 up, 8 in 2024-09-21T10:34:55.102 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:34:55.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:55 smithi160 ceph-mon[101879]: osdmap e1171: 8 total, 8 up, 8 in 2024-09-21T10:34:55.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:55 smithi070 ceph-mon[112139]: osdmap e1171: 8 total, 8 up, 8 in 2024-09-21T10:34:55.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:55 smithi070 ceph-mon[107385]: osdmap e1171: 8 total, 8 up, 8 in 2024-09-21T10:34:56.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:56 smithi160 ceph-mon[101879]: pgmap v1852: 105 pgs: 105 active+clean; 24 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s rd, 11 KiB/s wr, 134 op/s 2024-09-21T10:34:56.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:56 smithi160 ceph-mon[101879]: osdmap e1172: 8 total, 8 up, 8 in 2024-09-21T10:34:56.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:56 smithi070 ceph-mon[107385]: pgmap v1852: 105 pgs: 105 active+clean; 24 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s rd, 11 KiB/s wr, 134 op/s 2024-09-21T10:34:56.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:56 smithi070 ceph-mon[107385]: osdmap e1172: 8 total, 8 up, 8 in 2024-09-21T10:34:56.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:56 smithi070 ceph-mon[112139]: pgmap v1852: 105 pgs: 105 active+clean; 24 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s rd, 11 KiB/s wr, 134 op/s 2024-09-21T10:34:56.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:56 smithi070 ceph-mon[112139]: osdmap e1172: 8 total, 8 up, 8 in 2024-09-21T10:34:57.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:57 smithi160 ceph-mon[101879]: osdmap e1173: 8 total, 8 up, 8 in 2024-09-21T10:34:57.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:57 smithi070 ceph-mon[107385]: osdmap e1173: 8 total, 8 up, 8 in 2024-09-21T10:34:57.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:57 smithi070 ceph-mon[112139]: osdmap e1173: 8 total, 8 up, 8 in 2024-09-21T10:34:58.121 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/1.DiffIterateDeterministic (15027 ms) 2024-09-21T10:34:58.121 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/1.DiffIterateDeterministicPP 2024-09-21T10:34:58.122 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:34:58.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:58 smithi160 ceph-mon[101879]: pgmap v1855: 105 pgs: 105 active+clean; 25 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 8.0 KiB/s wr, 88 op/s 2024-09-21T10:34:58.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:58 smithi160 ceph-mon[101879]: osdmap e1174: 8 total, 8 up, 8 in 2024-09-21T10:34:58.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:58 smithi070 ceph-mon[107385]: pgmap v1855: 105 pgs: 105 active+clean; 25 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 8.0 KiB/s wr, 88 op/s 2024-09-21T10:34:58.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:58 smithi070 ceph-mon[107385]: osdmap e1174: 8 total, 8 up, 8 in 2024-09-21T10:34:58.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:58 smithi070 ceph-mon[112139]: pgmap v1855: 105 pgs: 105 active+clean; 25 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 8.0 KiB/s wr, 88 op/s 2024-09-21T10:34:58.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:58 smithi070 ceph-mon[112139]: osdmap e1174: 8 total, 8 up, 8 in 2024-09-21T10:34:59.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:34:59 smithi070 ceph-mon[107385]: osdmap e1175: 8 total, 8 up, 8 in 2024-09-21T10:34:59.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:34:59 smithi070 ceph-mon[112139]: osdmap e1175: 8 total, 8 up, 8 in 2024-09-21T10:34:59.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:34:59 smithi160 ceph-mon[101879]: osdmap e1175: 8 total, 8 up, 8 in 2024-09-21T10:35:00.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:00 smithi070 ceph-mon[107385]: pgmap v1858: 105 pgs: 105 active+clean; 25 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 2.2 KiB/s wr, 19 op/s 2024-09-21T10:35:00.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:00 smithi070 ceph-mon[107385]: osdmap e1176: 8 total, 8 up, 8 in 2024-09-21T10:35:00.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:00 smithi070 ceph-mon[112139]: pgmap v1858: 105 pgs: 105 active+clean; 25 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 2.2 KiB/s wr, 19 op/s 2024-09-21T10:35:00.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:00 smithi070 ceph-mon[112139]: osdmap e1176: 8 total, 8 up, 8 in 2024-09-21T10:35:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:00 smithi160 ceph-mon[101879]: pgmap v1858: 105 pgs: 105 active+clean; 25 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 2.2 KiB/s wr, 19 op/s 2024-09-21T10:35:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:00 smithi160 ceph-mon[101879]: osdmap e1176: 8 total, 8 up, 8 in 2024-09-21T10:35:01.140 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:35:01.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:01 smithi070 ceph-mon[107385]: osdmap e1177: 8 total, 8 up, 8 in 2024-09-21T10:35:01.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:01 smithi070 ceph-mon[112139]: osdmap e1177: 8 total, 8 up, 8 in 2024-09-21T10:35:01.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:01 smithi160 ceph-mon[101879]: osdmap e1177: 8 total, 8 up, 8 in 2024-09-21T10:35:02.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:02 smithi070 ceph-mon[107385]: pgmap v1861: 105 pgs: 105 active+clean; 25 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 6.2 KiB/s wr, 77 op/s 2024-09-21T10:35:02.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:02 smithi070 ceph-mon[107385]: osdmap e1178: 8 total, 8 up, 8 in 2024-09-21T10:35:02.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:02 smithi070 ceph-mon[112139]: pgmap v1861: 105 pgs: 105 active+clean; 25 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 6.2 KiB/s wr, 77 op/s 2024-09-21T10:35:02.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:02 smithi070 ceph-mon[112139]: osdmap e1178: 8 total, 8 up, 8 in 2024-09-21T10:35:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:02 smithi160 ceph-mon[101879]: pgmap v1861: 105 pgs: 105 active+clean; 25 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 6.2 KiB/s wr, 77 op/s 2024-09-21T10:35:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:02 smithi160 ceph-mon[101879]: osdmap e1178: 8 total, 8 up, 8 in 2024-09-21T10:35:03.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:03 smithi070 ceph-mon[107385]: osdmap e1179: 8 total, 8 up, 8 in 2024-09-21T10:35:03.378 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:35:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:35:03] "GET /metrics HTTP/1.1" 200 36460 "" "Prometheus/2.51.0" 2024-09-21T10:35:03.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:03 smithi070 ceph-mon[112139]: osdmap e1179: 8 total, 8 up, 8 in 2024-09-21T10:35:03.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:03 smithi160 ceph-mon[101879]: osdmap e1179: 8 total, 8 up, 8 in 2024-09-21T10:35:04.155 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:35:04.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:04 smithi070 ceph-mon[107385]: pgmap v1864: 105 pgs: 105 active+clean; 33 MiB data, 688 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 9.5 KiB/s wr, 113 op/s 2024-09-21T10:35:04.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:04 smithi070 ceph-mon[107385]: osdmap e1180: 8 total, 8 up, 8 in 2024-09-21T10:35:04.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:04 smithi070 ceph-mon[112139]: pgmap v1864: 105 pgs: 105 active+clean; 33 MiB data, 688 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 9.5 KiB/s wr, 113 op/s 2024-09-21T10:35:04.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:04 smithi070 ceph-mon[112139]: osdmap e1180: 8 total, 8 up, 8 in 2024-09-21T10:35:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:04 smithi160 ceph-mon[101879]: pgmap v1864: 105 pgs: 105 active+clean; 33 MiB data, 688 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 9.5 KiB/s wr, 113 op/s 2024-09-21T10:35:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:04 smithi160 ceph-mon[101879]: osdmap e1180: 8 total, 8 up, 8 in 2024-09-21T10:35:05.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:05 smithi160 ceph-mon[101879]: osdmap e1181: 8 total, 8 up, 8 in 2024-09-21T10:35:05.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:05 smithi070 ceph-mon[107385]: osdmap e1181: 8 total, 8 up, 8 in 2024-09-21T10:35:05.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:05 smithi070 ceph-mon[112139]: osdmap e1181: 8 total, 8 up, 8 in 2024-09-21T10:35:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:06 smithi160 ceph-mon[101879]: pgmap v1867: 105 pgs: 105 active+clean; 33 MiB data, 688 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 7.5 KiB/s wr, 67 op/s 2024-09-21T10:35:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:06 smithi160 ceph-mon[101879]: osdmap e1182: 8 total, 8 up, 8 in 2024-09-21T10:35:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:35:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:06 smithi070 ceph-mon[107385]: pgmap v1867: 105 pgs: 105 active+clean; 33 MiB data, 688 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 7.5 KiB/s wr, 67 op/s 2024-09-21T10:35:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:06 smithi070 ceph-mon[107385]: osdmap e1182: 8 total, 8 up, 8 in 2024-09-21T10:35:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:35:06.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:06 smithi070 ceph-mon[112139]: pgmap v1867: 105 pgs: 105 active+clean; 33 MiB data, 688 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 7.5 KiB/s wr, 67 op/s 2024-09-21T10:35:06.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:06 smithi070 ceph-mon[112139]: osdmap e1182: 8 total, 8 up, 8 in 2024-09-21T10:35:06.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:35:07.177 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:35:07.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:07 smithi160 ceph-mon[101879]: osdmap e1183: 8 total, 8 up, 8 in 2024-09-21T10:35:07.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:07 smithi070 ceph-mon[107385]: osdmap e1183: 8 total, 8 up, 8 in 2024-09-21T10:35:07.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:07 smithi070 ceph-mon[112139]: osdmap e1183: 8 total, 8 up, 8 in 2024-09-21T10:35:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:08 smithi160 ceph-mon[101879]: pgmap v1870: 105 pgs: 105 active+clean; 42 MiB data, 688 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 9.0 KiB/s wr, 103 op/s 2024-09-21T10:35:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:08 smithi160 ceph-mon[101879]: osdmap e1184: 8 total, 8 up, 8 in 2024-09-21T10:35:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:08 smithi070 ceph-mon[107385]: pgmap v1870: 105 pgs: 105 active+clean; 42 MiB data, 688 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 9.0 KiB/s wr, 103 op/s 2024-09-21T10:35:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:08 smithi070 ceph-mon[107385]: osdmap e1184: 8 total, 8 up, 8 in 2024-09-21T10:35:08.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:08 smithi070 ceph-mon[112139]: pgmap v1870: 105 pgs: 105 active+clean; 42 MiB data, 688 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 9.0 KiB/s wr, 103 op/s 2024-09-21T10:35:08.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:08 smithi070 ceph-mon[112139]: osdmap e1184: 8 total, 8 up, 8 in 2024-09-21T10:35:09.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:09 smithi160 ceph-mon[101879]: osdmap e1185: 8 total, 8 up, 8 in 2024-09-21T10:35:09.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:09 smithi070 ceph-mon[107385]: osdmap e1185: 8 total, 8 up, 8 in 2024-09-21T10:35:09.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:09 smithi070 ceph-mon[112139]: osdmap e1185: 8 total, 8 up, 8 in 2024-09-21T10:35:10.194 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:35:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:10 smithi160 ceph-mon[101879]: pgmap v1873: 105 pgs: 105 active+clean; 42 MiB data, 688 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 4.7 KiB/s wr, 71 op/s 2024-09-21T10:35:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:10 smithi160 ceph-mon[101879]: osdmap e1186: 8 total, 8 up, 8 in 2024-09-21T10:35:10.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:10 smithi070 ceph-mon[112139]: pgmap v1873: 105 pgs: 105 active+clean; 42 MiB data, 688 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 4.7 KiB/s wr, 71 op/s 2024-09-21T10:35:10.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:10 smithi070 ceph-mon[112139]: osdmap e1186: 8 total, 8 up, 8 in 2024-09-21T10:35:10.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:10 smithi070 ceph-mon[107385]: pgmap v1873: 105 pgs: 105 active+clean; 42 MiB data, 688 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 4.7 KiB/s wr, 71 op/s 2024-09-21T10:35:10.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:10 smithi070 ceph-mon[107385]: osdmap e1186: 8 total, 8 up, 8 in 2024-09-21T10:35:11.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:11 smithi160 ceph-mon[101879]: osdmap e1187: 8 total, 8 up, 8 in 2024-09-21T10:35:11.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:11 smithi070 ceph-mon[107385]: osdmap e1187: 8 total, 8 up, 8 in 2024-09-21T10:35:11.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:11 smithi070 ceph-mon[112139]: osdmap e1187: 8 total, 8 up, 8 in 2024-09-21T10:35:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:12 smithi160 ceph-mon[101879]: pgmap v1876: 105 pgs: 105 active+clean; 45 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 4.5 KiB/s wr, 28 op/s 2024-09-21T10:35:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:12 smithi160 ceph-mon[101879]: osdmap e1188: 8 total, 8 up, 8 in 2024-09-21T10:35:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:12 smithi070 ceph-mon[107385]: pgmap v1876: 105 pgs: 105 active+clean; 45 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 4.5 KiB/s wr, 28 op/s 2024-09-21T10:35:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:12 smithi070 ceph-mon[107385]: osdmap e1188: 8 total, 8 up, 8 in 2024-09-21T10:35:12.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:12 smithi070 ceph-mon[112139]: pgmap v1876: 105 pgs: 105 active+clean; 45 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 4.5 KiB/s wr, 28 op/s 2024-09-21T10:35:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:12 smithi070 ceph-mon[112139]: osdmap e1188: 8 total, 8 up, 8 in 2024-09-21T10:35:13.217 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/1.DiffIterateDeterministicPP (15096 ms) 2024-09-21T10:35:13.218 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/1.DiffIterateDeterministicLUKS1 2024-09-21T10:35:13.218 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:35:13.455 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:35:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:35:13] "GET /metrics HTTP/1.1" 200 36465 "" "Prometheus/2.51.0" 2024-09-21T10:35:13.455 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:13 smithi070 ceph-mon[107385]: osdmap e1189: 8 total, 8 up, 8 in 2024-09-21T10:35:13.455 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:13 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:35:13.456 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:13 smithi070 ceph-mon[112139]: osdmap e1189: 8 total, 8 up, 8 in 2024-09-21T10:35:13.456 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:13 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:35:13.488 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:13 smithi160 ceph-mon[101879]: osdmap e1189: 8 total, 8 up, 8 in 2024-09-21T10:35:13.489 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:13 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:35:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:14 smithi160 ceph-mon[101879]: pgmap v1879: 105 pgs: 105 active+clean; 51 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 8.5 KiB/s wr, 95 op/s 2024-09-21T10:35:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:14 smithi160 ceph-mon[101879]: osdmap e1190: 8 total, 8 up, 8 in 2024-09-21T10:35:14.475 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:14 smithi070 ceph-mon[112139]: pgmap v1879: 105 pgs: 105 active+clean; 51 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 8.5 KiB/s wr, 95 op/s 2024-09-21T10:35:14.475 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:14 smithi070 ceph-mon[112139]: osdmap e1190: 8 total, 8 up, 8 in 2024-09-21T10:35:14.475 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:14 smithi070 ceph-mon[107385]: pgmap v1879: 105 pgs: 105 active+clean; 51 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 8.5 KiB/s wr, 95 op/s 2024-09-21T10:35:14.475 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:14 smithi070 ceph-mon[107385]: osdmap e1190: 8 total, 8 up, 8 in 2024-09-21T10:35:16.486 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:16 smithi160 ceph-mon[101879]: pgmap v1881: 105 pgs: 105 active+clean; 51 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 6.8 KiB/s wr, 111 op/s 2024-09-21T10:35:16.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:16 smithi070 ceph-mon[112139]: pgmap v1881: 105 pgs: 105 active+clean; 51 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 6.8 KiB/s wr, 111 op/s 2024-09-21T10:35:16.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:16 smithi070 ceph-mon[107385]: pgmap v1881: 105 pgs: 105 active+clean; 51 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 6.8 KiB/s wr, 111 op/s 2024-09-21T10:35:17.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:17 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:35:17.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:17 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:35:17.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:17 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:35:17.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:17 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:35:17.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:17 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:35:17.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:17 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:35:17.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:17 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:35:17.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:35:17.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:35:17.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:35:17.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:35:17.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:35:17.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:35:17.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:35:17.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:35:17.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:35:17.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:35:17.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:35:17.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:35:17.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:35:17.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:35:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:18 smithi160 ceph-mon[101879]: pgmap v1882: 105 pgs: 105 active+clean; 59 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 7.0 KiB/s wr, 96 op/s 2024-09-21T10:35:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:18 smithi160 ceph-mon[101879]: osdmap e1191: 8 total, 8 up, 8 in 2024-09-21T10:35:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:18 smithi070 ceph-mon[107385]: pgmap v1882: 105 pgs: 105 active+clean; 59 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 7.0 KiB/s wr, 96 op/s 2024-09-21T10:35:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:18 smithi070 ceph-mon[107385]: osdmap e1191: 8 total, 8 up, 8 in 2024-09-21T10:35:18.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:18 smithi070 ceph-mon[112139]: pgmap v1882: 105 pgs: 105 active+clean; 59 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 7.0 KiB/s wr, 96 op/s 2024-09-21T10:35:18.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:18 smithi070 ceph-mon[112139]: osdmap e1191: 8 total, 8 up, 8 in 2024-09-21T10:35:19.291 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:35:19.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:19 smithi160 ceph-mon[101879]: osdmap e1192: 8 total, 8 up, 8 in 2024-09-21T10:35:19.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:19 smithi070 ceph-mon[112139]: osdmap e1192: 8 total, 8 up, 8 in 2024-09-21T10:35:19.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:19 smithi070 ceph-mon[107385]: osdmap e1192: 8 total, 8 up, 8 in 2024-09-21T10:35:20.512 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:20 smithi160 ceph-mon[101879]: pgmap v1885: 105 pgs: 105 active+clean; 59 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 4.3 KiB/s wr, 52 op/s 2024-09-21T10:35:20.512 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:20 smithi160 ceph-mon[101879]: osdmap e1193: 8 total, 8 up, 8 in 2024-09-21T10:35:20.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:20 smithi070 ceph-mon[112139]: pgmap v1885: 105 pgs: 105 active+clean; 59 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 4.3 KiB/s wr, 52 op/s 2024-09-21T10:35:20.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:20 smithi070 ceph-mon[112139]: osdmap e1193: 8 total, 8 up, 8 in 2024-09-21T10:35:20.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:20 smithi070 ceph-mon[107385]: pgmap v1885: 105 pgs: 105 active+clean; 59 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 4.3 KiB/s wr, 52 op/s 2024-09-21T10:35:20.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:20 smithi070 ceph-mon[107385]: osdmap e1193: 8 total, 8 up, 8 in 2024-09-21T10:35:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:35:21.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:35:21.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:35:22.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:22 smithi160 ceph-mon[101879]: pgmap v1887: 105 pgs: 105 active+clean; 59 MiB data, 702 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 3.3 KiB/s wr, 35 op/s 2024-09-21T10:35:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:22 smithi070 ceph-mon[107385]: pgmap v1887: 105 pgs: 105 active+clean; 59 MiB data, 702 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 3.3 KiB/s wr, 35 op/s 2024-09-21T10:35:22.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:22 smithi070 ceph-mon[112139]: pgmap v1887: 105 pgs: 105 active+clean; 59 MiB data, 702 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 3.3 KiB/s wr, 35 op/s 2024-09-21T10:35:23.528 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:35:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:35:23] "GET /metrics HTTP/1.1" 200 36466 "" "Prometheus/2.51.0" 2024-09-21T10:35:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:24 smithi160 ceph-mon[101879]: pgmap v1888: 105 pgs: 105 active+clean; 63 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 686 KiB/s wr, 36 op/s 2024-09-21T10:35:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:24 smithi160 ceph-mon[101879]: osdmap e1194: 8 total, 8 up, 8 in 2024-09-21T10:35:24.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:24 smithi070 ceph-mon[112139]: pgmap v1888: 105 pgs: 105 active+clean; 63 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 686 KiB/s wr, 36 op/s 2024-09-21T10:35:24.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:24 smithi070 ceph-mon[112139]: osdmap e1194: 8 total, 8 up, 8 in 2024-09-21T10:35:24.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:24 smithi070 ceph-mon[107385]: pgmap v1888: 105 pgs: 105 active+clean; 63 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 686 KiB/s wr, 36 op/s 2024-09-21T10:35:24.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:24 smithi070 ceph-mon[107385]: osdmap e1194: 8 total, 8 up, 8 in 2024-09-21T10:35:25.312 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:35:25.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:25 smithi160 ceph-mon[101879]: osdmap e1195: 8 total, 8 up, 8 in 2024-09-21T10:35:25.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:25 smithi070 ceph-mon[107385]: osdmap e1195: 8 total, 8 up, 8 in 2024-09-21T10:35:25.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:25 smithi070 ceph-mon[112139]: osdmap e1195: 8 total, 8 up, 8 in 2024-09-21T10:35:26.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:26 smithi160 ceph-mon[101879]: pgmap v1891: 105 pgs: 105 active+clean; 63 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 41 op/s 2024-09-21T10:35:26.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:26 smithi160 ceph-mon[101879]: osdmap e1196: 8 total, 8 up, 8 in 2024-09-21T10:35:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:26 smithi070 ceph-mon[107385]: pgmap v1891: 105 pgs: 105 active+clean; 63 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 41 op/s 2024-09-21T10:35:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:26 smithi070 ceph-mon[107385]: osdmap e1196: 8 total, 8 up, 8 in 2024-09-21T10:35:26.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:26 smithi070 ceph-mon[112139]: pgmap v1891: 105 pgs: 105 active+clean; 63 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 41 op/s 2024-09-21T10:35:26.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:26 smithi070 ceph-mon[112139]: osdmap e1196: 8 total, 8 up, 8 in 2024-09-21T10:35:28.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:28 smithi160 ceph-mon[101879]: pgmap v1893: 105 pgs: 105 active+clean; 68 MiB data, 719 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 1.3 MiB/s wr, 56 op/s 2024-09-21T10:35:28.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:28 smithi070 ceph-mon[107385]: pgmap v1893: 105 pgs: 105 active+clean; 68 MiB data, 719 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 1.3 MiB/s wr, 56 op/s 2024-09-21T10:35:28.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:28 smithi070 ceph-mon[112139]: pgmap v1893: 105 pgs: 105 active+clean; 68 MiB data, 719 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 1.3 MiB/s wr, 56 op/s 2024-09-21T10:35:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:30 smithi160 ceph-mon[101879]: pgmap v1894: 105 pgs: 105 active+clean; 68 MiB data, 719 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 686 KiB/s wr, 41 op/s 2024-09-21T10:35:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:30 smithi160 ceph-mon[101879]: osdmap e1197: 8 total, 8 up, 8 in 2024-09-21T10:35:30.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:30 smithi070 ceph-mon[107385]: pgmap v1894: 105 pgs: 105 active+clean; 68 MiB data, 719 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 686 KiB/s wr, 41 op/s 2024-09-21T10:35:30.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:30 smithi070 ceph-mon[107385]: osdmap e1197: 8 total, 8 up, 8 in 2024-09-21T10:35:30.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:30 smithi070 ceph-mon[112139]: pgmap v1894: 105 pgs: 105 active+clean; 68 MiB data, 719 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 686 KiB/s wr, 41 op/s 2024-09-21T10:35:30.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:30 smithi070 ceph-mon[112139]: osdmap e1197: 8 total, 8 up, 8 in 2024-09-21T10:35:31.331 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:35:31.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:31 smithi160 ceph-mon[101879]: osdmap e1198: 8 total, 8 up, 8 in 2024-09-21T10:35:31.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:31 smithi070 ceph-mon[107385]: osdmap e1198: 8 total, 8 up, 8 in 2024-09-21T10:35:31.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:31 smithi070 ceph-mon[112139]: osdmap e1198: 8 total, 8 up, 8 in 2024-09-21T10:35:32.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:32 smithi160 ceph-mon[101879]: pgmap v1897: 105 pgs: 105 active+clean; 73 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 1.3 MiB/s wr, 44 op/s 2024-09-21T10:35:32.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:32 smithi160 ceph-mon[101879]: osdmap e1199: 8 total, 8 up, 8 in 2024-09-21T10:35:32.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:32 smithi070 ceph-mon[112139]: pgmap v1897: 105 pgs: 105 active+clean; 73 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 1.3 MiB/s wr, 44 op/s 2024-09-21T10:35:32.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:32 smithi070 ceph-mon[112139]: osdmap e1199: 8 total, 8 up, 8 in 2024-09-21T10:35:32.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:32 smithi070 ceph-mon[107385]: pgmap v1897: 105 pgs: 105 active+clean; 73 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 1.3 MiB/s wr, 44 op/s 2024-09-21T10:35:32.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:32 smithi070 ceph-mon[107385]: osdmap e1199: 8 total, 8 up, 8 in 2024-09-21T10:35:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:35:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:35:33] "GET /metrics HTTP/1.1" 200 36466 "" "Prometheus/2.51.0" 2024-09-21T10:35:34.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:34 smithi160 ceph-mon[101879]: pgmap v1899: 105 pgs: 105 active+clean; 74 MiB data, 727 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 684 KiB/s wr, 9 op/s 2024-09-21T10:35:34.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:34 smithi070 ceph-mon[107385]: pgmap v1899: 105 pgs: 105 active+clean; 74 MiB data, 727 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 684 KiB/s wr, 9 op/s 2024-09-21T10:35:34.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:34 smithi070 ceph-mon[112139]: pgmap v1899: 105 pgs: 105 active+clean; 74 MiB data, 727 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 684 KiB/s wr, 9 op/s 2024-09-21T10:35:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:36 smithi160 ceph-mon[101879]: pgmap v1900: 105 pgs: 105 active+clean; 75 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 687 KiB/s wr, 45 op/s 2024-09-21T10:35:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:36 smithi160 ceph-mon[101879]: osdmap e1200: 8 total, 8 up, 8 in 2024-09-21T10:35:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:35:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:36 smithi070 ceph-mon[107385]: pgmap v1900: 105 pgs: 105 active+clean; 75 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 687 KiB/s wr, 45 op/s 2024-09-21T10:35:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:36 smithi070 ceph-mon[107385]: osdmap e1200: 8 total, 8 up, 8 in 2024-09-21T10:35:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:35:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:36 smithi070 ceph-mon[112139]: pgmap v1900: 105 pgs: 105 active+clean; 75 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 687 KiB/s wr, 45 op/s 2024-09-21T10:35:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:36 smithi070 ceph-mon[112139]: osdmap e1200: 8 total, 8 up, 8 in 2024-09-21T10:35:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:35:37.354 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:35:37.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:37 smithi160 ceph-mon[101879]: osdmap e1201: 8 total, 8 up, 8 in 2024-09-21T10:35:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:37 smithi070 ceph-mon[107385]: osdmap e1201: 8 total, 8 up, 8 in 2024-09-21T10:35:37.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:37 smithi070 ceph-mon[112139]: osdmap e1201: 8 total, 8 up, 8 in 2024-09-21T10:35:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:38 smithi160 ceph-mon[101879]: pgmap v1903: 105 pgs: 105 active+clean; 80 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 687 KiB/s wr, 43 op/s 2024-09-21T10:35:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:38 smithi160 ceph-mon[101879]: osdmap e1202: 8 total, 8 up, 8 in 2024-09-21T10:35:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:38 smithi070 ceph-mon[107385]: pgmap v1903: 105 pgs: 105 active+clean; 80 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 687 KiB/s wr, 43 op/s 2024-09-21T10:35:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:38 smithi070 ceph-mon[107385]: osdmap e1202: 8 total, 8 up, 8 in 2024-09-21T10:35:38.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:38 smithi070 ceph-mon[112139]: pgmap v1903: 105 pgs: 105 active+clean; 80 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 687 KiB/s wr, 43 op/s 2024-09-21T10:35:38.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:38 smithi070 ceph-mon[112139]: osdmap e1202: 8 total, 8 up, 8 in 2024-09-21T10:35:40.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:40 smithi160 ceph-mon[101879]: pgmap v1905: 105 pgs: 105 active+clean; 80 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 687 KiB/s wr, 43 op/s 2024-09-21T10:35:40.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:40 smithi070 ceph-mon[107385]: pgmap v1905: 105 pgs: 105 active+clean; 80 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 687 KiB/s wr, 43 op/s 2024-09-21T10:35:40.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:40 smithi070 ceph-mon[112139]: pgmap v1905: 105 pgs: 105 active+clean; 80 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 687 KiB/s wr, 43 op/s 2024-09-21T10:35:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:42 smithi160 ceph-mon[101879]: pgmap v1906: 105 pgs: 105 active+clean; 80 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 686 KiB/s wr, 35 op/s 2024-09-21T10:35:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:42 smithi160 ceph-mon[101879]: osdmap e1203: 8 total, 8 up, 8 in 2024-09-21T10:35:42.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:42 smithi070 ceph-mon[107385]: pgmap v1906: 105 pgs: 105 active+clean; 80 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 686 KiB/s wr, 35 op/s 2024-09-21T10:35:42.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:42 smithi070 ceph-mon[107385]: osdmap e1203: 8 total, 8 up, 8 in 2024-09-21T10:35:42.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:42 smithi070 ceph-mon[112139]: pgmap v1906: 105 pgs: 105 active+clean; 80 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 686 KiB/s wr, 35 op/s 2024-09-21T10:35:42.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:42 smithi070 ceph-mon[112139]: osdmap e1203: 8 total, 8 up, 8 in 2024-09-21T10:35:43.370 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/1.DiffIterateDeterministicLUKS1 (30153 ms) 2024-09-21T10:35:43.371 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/1.DiffIterateDeterministicLUKS1PP 2024-09-21T10:35:43.371 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:35:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:35:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:35:43] "GET /metrics HTTP/1.1" 200 36474 "" "Prometheus/2.51.0" 2024-09-21T10:35:43.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:43 smithi070 ceph-mon[107385]: osdmap e1204: 8 total, 8 up, 8 in 2024-09-21T10:35:43.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:43 smithi070 ceph-mon[112139]: osdmap e1204: 8 total, 8 up, 8 in 2024-09-21T10:35:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:43 smithi160 ceph-mon[101879]: osdmap e1204: 8 total, 8 up, 8 in 2024-09-21T10:35:44.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:44 smithi070 ceph-mon[107385]: pgmap v1909: 105 pgs: 105 active+clean; 81 MiB data, 748 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 3.0 KiB/s wr, 37 op/s 2024-09-21T10:35:44.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:44 smithi070 ceph-mon[107385]: osdmap e1205: 8 total, 8 up, 8 in 2024-09-21T10:35:44.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:44 smithi070 ceph-mon[112139]: pgmap v1909: 105 pgs: 105 active+clean; 81 MiB data, 748 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 3.0 KiB/s wr, 37 op/s 2024-09-21T10:35:44.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:44 smithi070 ceph-mon[112139]: osdmap e1205: 8 total, 8 up, 8 in 2024-09-21T10:35:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:44 smithi160 ceph-mon[101879]: pgmap v1909: 105 pgs: 105 active+clean; 81 MiB data, 748 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 3.0 KiB/s wr, 37 op/s 2024-09-21T10:35:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:44 smithi160 ceph-mon[101879]: osdmap e1205: 8 total, 8 up, 8 in 2024-09-21T10:35:46.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:46 smithi070 ceph-mon[107385]: pgmap v1911: 105 pgs: 105 active+clean; 81 MiB data, 748 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 5.3 KiB/s wr, 56 op/s 2024-09-21T10:35:46.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:46 smithi070 ceph-mon[112139]: pgmap v1911: 105 pgs: 105 active+clean; 81 MiB data, 748 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 5.3 KiB/s wr, 56 op/s 2024-09-21T10:35:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:46 smithi160 ceph-mon[101879]: pgmap v1911: 105 pgs: 105 active+clean; 81 MiB data, 748 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 5.3 KiB/s wr, 56 op/s 2024-09-21T10:35:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:48 smithi070 ceph-mon[107385]: pgmap v1912: 105 pgs: 105 active+clean; 93 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 34 KiB/s rd, 687 KiB/s wr, 50 op/s 2024-09-21T10:35:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:48 smithi070 ceph-mon[107385]: osdmap e1206: 8 total, 8 up, 8 in 2024-09-21T10:35:48.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:48 smithi070 ceph-mon[112139]: pgmap v1912: 105 pgs: 105 active+clean; 93 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 34 KiB/s rd, 687 KiB/s wr, 50 op/s 2024-09-21T10:35:48.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:48 smithi070 ceph-mon[112139]: osdmap e1206: 8 total, 8 up, 8 in 2024-09-21T10:35:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:48 smithi160 ceph-mon[101879]: pgmap v1912: 105 pgs: 105 active+clean; 93 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 34 KiB/s rd, 687 KiB/s wr, 50 op/s 2024-09-21T10:35:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:48 smithi160 ceph-mon[101879]: osdmap e1206: 8 total, 8 up, 8 in 2024-09-21T10:35:49.392 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:35:49.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:49 smithi070 ceph-mon[107385]: osdmap e1207: 8 total, 8 up, 8 in 2024-09-21T10:35:49.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:49 smithi070 ceph-mon[112139]: osdmap e1207: 8 total, 8 up, 8 in 2024-09-21T10:35:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:49 smithi160 ceph-mon[101879]: osdmap e1207: 8 total, 8 up, 8 in 2024-09-21T10:35:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:50 smithi070 ceph-mon[107385]: pgmap v1915: 105 pgs: 105 active+clean; 93 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 686 KiB/s wr, 41 op/s 2024-09-21T10:35:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:50 smithi070 ceph-mon[107385]: osdmap e1208: 8 total, 8 up, 8 in 2024-09-21T10:35:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:35:50.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:50 smithi070 ceph-mon[112139]: pgmap v1915: 105 pgs: 105 active+clean; 93 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 686 KiB/s wr, 41 op/s 2024-09-21T10:35:50.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:50 smithi070 ceph-mon[112139]: osdmap e1208: 8 total, 8 up, 8 in 2024-09-21T10:35:50.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:35:50.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:50 smithi160 ceph-mon[101879]: pgmap v1915: 105 pgs: 105 active+clean; 93 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 686 KiB/s wr, 41 op/s 2024-09-21T10:35:50.743 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:50 smithi160 ceph-mon[101879]: osdmap e1208: 8 total, 8 up, 8 in 2024-09-21T10:35:50.743 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:35:52.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:52 smithi070 ceph-mon[107385]: pgmap v1917: 105 pgs: 105 active+clean; 97 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 1.3 MiB/s wr, 35 op/s 2024-09-21T10:35:52.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:52 smithi070 ceph-mon[112139]: pgmap v1917: 105 pgs: 105 active+clean; 97 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 1.3 MiB/s wr, 35 op/s 2024-09-21T10:35:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:52 smithi160 ceph-mon[101879]: pgmap v1917: 105 pgs: 105 active+clean; 97 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 1.3 MiB/s wr, 35 op/s 2024-09-21T10:35:53.528 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:35:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:35:53] "GET /metrics HTTP/1.1" 200 36469 "" "Prometheus/2.51.0" 2024-09-21T10:35:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:54 smithi160 ceph-mon[101879]: pgmap v1918: 105 pgs: 105 active+clean; 97 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 686 KiB/s wr, 30 op/s 2024-09-21T10:35:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:54 smithi160 ceph-mon[101879]: osdmap e1209: 8 total, 8 up, 8 in 2024-09-21T10:35:54.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:54 smithi070 ceph-mon[107385]: pgmap v1918: 105 pgs: 105 active+clean; 97 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 686 KiB/s wr, 30 op/s 2024-09-21T10:35:54.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:54 smithi070 ceph-mon[107385]: osdmap e1209: 8 total, 8 up, 8 in 2024-09-21T10:35:54.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:54 smithi070 ceph-mon[112139]: pgmap v1918: 105 pgs: 105 active+clean; 97 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 686 KiB/s wr, 30 op/s 2024-09-21T10:35:54.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:54 smithi070 ceph-mon[112139]: osdmap e1209: 8 total, 8 up, 8 in 2024-09-21T10:35:55.414 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:35:55.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:55 smithi160 ceph-mon[101879]: osdmap e1210: 8 total, 8 up, 8 in 2024-09-21T10:35:55.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:55 smithi070 ceph-mon[107385]: osdmap e1210: 8 total, 8 up, 8 in 2024-09-21T10:35:55.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:55 smithi070 ceph-mon[112139]: osdmap e1210: 8 total, 8 up, 8 in 2024-09-21T10:35:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:56 smithi160 ceph-mon[101879]: pgmap v1921: 105 pgs: 105 active+clean; 100 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 42 op/s 2024-09-21T10:35:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:56 smithi160 ceph-mon[101879]: osdmap e1211: 8 total, 8 up, 8 in 2024-09-21T10:35:56.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:56 smithi070 ceph-mon[107385]: pgmap v1921: 105 pgs: 105 active+clean; 100 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 42 op/s 2024-09-21T10:35:56.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:56 smithi070 ceph-mon[107385]: osdmap e1211: 8 total, 8 up, 8 in 2024-09-21T10:35:56.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:56 smithi070 ceph-mon[112139]: pgmap v1921: 105 pgs: 105 active+clean; 100 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 42 op/s 2024-09-21T10:35:56.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:56 smithi070 ceph-mon[112139]: osdmap e1211: 8 total, 8 up, 8 in 2024-09-21T10:35:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:35:58 smithi160 ceph-mon[101879]: pgmap v1923: 105 pgs: 105 active+clean; 104 MiB data, 781 MiB used, 714 GiB / 715 GiB avail; 42 KiB/s rd, 689 KiB/s wr, 61 op/s 2024-09-21T10:35:58.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:35:58 smithi070 ceph-mon[107385]: pgmap v1923: 105 pgs: 105 active+clean; 104 MiB data, 781 MiB used, 714 GiB / 715 GiB avail; 42 KiB/s rd, 689 KiB/s wr, 61 op/s 2024-09-21T10:35:58.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:35:58 smithi070 ceph-mon[112139]: pgmap v1923: 105 pgs: 105 active+clean; 104 MiB data, 781 MiB used, 714 GiB / 715 GiB avail; 42 KiB/s rd, 689 KiB/s wr, 61 op/s 2024-09-21T10:36:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:00 smithi160 ceph-mon[101879]: pgmap v1924: 105 pgs: 105 active+clean; 104 MiB data, 781 MiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 686 KiB/s wr, 43 op/s 2024-09-21T10:36:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:00 smithi160 ceph-mon[101879]: osdmap e1212: 8 total, 8 up, 8 in 2024-09-21T10:36:00.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:00 smithi070 ceph-mon[107385]: pgmap v1924: 105 pgs: 105 active+clean; 104 MiB data, 781 MiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 686 KiB/s wr, 43 op/s 2024-09-21T10:36:00.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:00 smithi070 ceph-mon[107385]: osdmap e1212: 8 total, 8 up, 8 in 2024-09-21T10:36:00.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:00 smithi070 ceph-mon[112139]: pgmap v1924: 105 pgs: 105 active+clean; 104 MiB data, 781 MiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 686 KiB/s wr, 43 op/s 2024-09-21T10:36:00.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:00 smithi070 ceph-mon[112139]: osdmap e1212: 8 total, 8 up, 8 in 2024-09-21T10:36:01.434 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:36:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:01 smithi160 ceph-mon[101879]: osdmap e1213: 8 total, 8 up, 8 in 2024-09-21T10:36:01.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:01 smithi070 ceph-mon[107385]: osdmap e1213: 8 total, 8 up, 8 in 2024-09-21T10:36:01.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:01 smithi070 ceph-mon[112139]: osdmap e1213: 8 total, 8 up, 8 in 2024-09-21T10:36:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:02 smithi160 ceph-mon[101879]: pgmap v1927: 105 pgs: 105 active+clean; 107 MiB data, 781 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 686 KiB/s wr, 41 op/s 2024-09-21T10:36:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:02 smithi160 ceph-mon[101879]: osdmap e1214: 8 total, 8 up, 8 in 2024-09-21T10:36:02.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:02 smithi070 ceph-mon[107385]: pgmap v1927: 105 pgs: 105 active+clean; 107 MiB data, 781 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 686 KiB/s wr, 41 op/s 2024-09-21T10:36:02.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:02 smithi070 ceph-mon[107385]: osdmap e1214: 8 total, 8 up, 8 in 2024-09-21T10:36:02.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:02 smithi070 ceph-mon[112139]: pgmap v1927: 105 pgs: 105 active+clean; 107 MiB data, 781 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 686 KiB/s wr, 41 op/s 2024-09-21T10:36:02.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:02 smithi070 ceph-mon[112139]: osdmap e1214: 8 total, 8 up, 8 in 2024-09-21T10:36:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:36:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:36:03] "GET /metrics HTTP/1.1" 200 36469 "" "Prometheus/2.51.0" 2024-09-21T10:36:04.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:04 smithi160 ceph-mon[101879]: pgmap v1929: 105 pgs: 105 active+clean; 107 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 1.8 KiB/s wr, 21 op/s 2024-09-21T10:36:04.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:04 smithi070 ceph-mon[107385]: pgmap v1929: 105 pgs: 105 active+clean; 107 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 1.8 KiB/s wr, 21 op/s 2024-09-21T10:36:04.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:04 smithi070 ceph-mon[112139]: pgmap v1929: 105 pgs: 105 active+clean; 107 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 1.8 KiB/s wr, 21 op/s 2024-09-21T10:36:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:05 smithi160 ceph-mon[101879]: pgmap v1930: 105 pgs: 105 active+clean; 117 MiB data, 793 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 39 op/s 2024-09-21T10:36:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:36:05.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:05 smithi070 ceph-mon[112139]: pgmap v1930: 105 pgs: 105 active+clean; 117 MiB data, 793 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 39 op/s 2024-09-21T10:36:05.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:36:05.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:05 smithi070 ceph-mon[107385]: pgmap v1930: 105 pgs: 105 active+clean; 117 MiB data, 793 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 39 op/s 2024-09-21T10:36:05.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:36:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:06 smithi160 ceph-mon[101879]: osdmap e1215: 8 total, 8 up, 8 in 2024-09-21T10:36:06.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:06 smithi070 ceph-mon[107385]: osdmap e1215: 8 total, 8 up, 8 in 2024-09-21T10:36:06.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:06 smithi070 ceph-mon[112139]: osdmap e1215: 8 total, 8 up, 8 in 2024-09-21T10:36:07.452 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:36:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:07 smithi160 ceph-mon[101879]: osdmap e1216: 8 total, 8 up, 8 in 2024-09-21T10:36:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:07 smithi160 ceph-mon[101879]: pgmap v1933: 105 pgs: 105 active+clean; 120 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 41 op/s 2024-09-21T10:36:07.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:07 smithi070 ceph-mon[107385]: osdmap e1216: 8 total, 8 up, 8 in 2024-09-21T10:36:07.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:07 smithi070 ceph-mon[107385]: pgmap v1933: 105 pgs: 105 active+clean; 120 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 41 op/s 2024-09-21T10:36:07.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:07 smithi070 ceph-mon[112139]: osdmap e1216: 8 total, 8 up, 8 in 2024-09-21T10:36:07.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:07 smithi070 ceph-mon[112139]: pgmap v1933: 105 pgs: 105 active+clean; 120 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 41 op/s 2024-09-21T10:36:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:08 smithi160 ceph-mon[101879]: osdmap e1217: 8 total, 8 up, 8 in 2024-09-21T10:36:08.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:08 smithi070 ceph-mon[112139]: osdmap e1217: 8 total, 8 up, 8 in 2024-09-21T10:36:08.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:08 smithi070 ceph-mon[107385]: osdmap e1217: 8 total, 8 up, 8 in 2024-09-21T10:36:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:09 smithi160 ceph-mon[101879]: pgmap v1935: 105 pgs: 105 active+clean; 120 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 686 KiB/s wr, 29 op/s 2024-09-21T10:36:09.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:09 smithi070 ceph-mon[107385]: pgmap v1935: 105 pgs: 105 active+clean; 120 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 686 KiB/s wr, 29 op/s 2024-09-21T10:36:09.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:09 smithi070 ceph-mon[112139]: pgmap v1935: 105 pgs: 105 active+clean; 120 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 686 KiB/s wr, 29 op/s 2024-09-21T10:36:12.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:11 smithi160 ceph-mon[101879]: pgmap v1936: 105 pgs: 105 active+clean; 124 MiB data, 806 MiB used, 714 GiB / 715 GiB avail; 34 KiB/s rd, 687 KiB/s wr, 50 op/s 2024-09-21T10:36:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:11 smithi070 ceph-mon[107385]: pgmap v1936: 105 pgs: 105 active+clean; 124 MiB data, 806 MiB used, 714 GiB / 715 GiB avail; 34 KiB/s rd, 687 KiB/s wr, 50 op/s 2024-09-21T10:36:12.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:11 smithi070 ceph-mon[112139]: pgmap v1936: 105 pgs: 105 active+clean; 124 MiB data, 806 MiB used, 714 GiB / 715 GiB avail; 34 KiB/s rd, 687 KiB/s wr, 50 op/s 2024-09-21T10:36:13.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:12 smithi160 ceph-mon[101879]: osdmap e1218: 8 total, 8 up, 8 in 2024-09-21T10:36:13.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:12 smithi070 ceph-mon[112139]: osdmap e1218: 8 total, 8 up, 8 in 2024-09-21T10:36:13.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:12 smithi070 ceph-mon[107385]: osdmap e1218: 8 total, 8 up, 8 in 2024-09-21T10:36:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:36:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:36:13] "GET /metrics HTTP/1.1" 200 36472 "" "Prometheus/2.51.0" 2024-09-21T10:36:13.700 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/1.DiffIterateDeterministicLUKS1PP (30330 ms) 2024-09-21T10:36:13.700 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/1.DiffIterateDeterministicLUKS2 2024-09-21T10:36:13.700 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:36:14.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:13 smithi160 ceph-mon[101879]: pgmap v1938: 105 pgs: 105 active+clean; 127 MiB data, 806 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 667 KiB/s wr, 42 op/s 2024-09-21T10:36:14.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:13 smithi160 ceph-mon[101879]: osdmap e1219: 8 total, 8 up, 8 in 2024-09-21T10:36:14.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:13 smithi070 ceph-mon[107385]: pgmap v1938: 105 pgs: 105 active+clean; 127 MiB data, 806 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 667 KiB/s wr, 42 op/s 2024-09-21T10:36:14.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:13 smithi070 ceph-mon[107385]: osdmap e1219: 8 total, 8 up, 8 in 2024-09-21T10:36:14.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:13 smithi070 ceph-mon[112139]: pgmap v1938: 105 pgs: 105 active+clean; 127 MiB data, 806 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 667 KiB/s wr, 42 op/s 2024-09-21T10:36:14.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:13 smithi070 ceph-mon[112139]: osdmap e1219: 8 total, 8 up, 8 in 2024-09-21T10:36:15.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:14 smithi160 ceph-mon[101879]: osdmap e1220: 8 total, 8 up, 8 in 2024-09-21T10:36:15.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:14 smithi070 ceph-mon[107385]: osdmap e1220: 8 total, 8 up, 8 in 2024-09-21T10:36:15.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:14 smithi070 ceph-mon[112139]: osdmap e1220: 8 total, 8 up, 8 in 2024-09-21T10:36:16.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:15 smithi160 ceph-mon[101879]: pgmap v1941: 105 pgs: 105 active+clean; 135 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 1.3 MiB/s wr, 47 op/s 2024-09-21T10:36:16.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:15 smithi070 ceph-mon[107385]: pgmap v1941: 105 pgs: 105 active+clean; 135 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 1.3 MiB/s wr, 47 op/s 2024-09-21T10:36:16.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:15 smithi070 ceph-mon[112139]: pgmap v1941: 105 pgs: 105 active+clean; 135 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 1.3 MiB/s wr, 47 op/s 2024-09-21T10:36:17.936 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:17 smithi160 ceph-mon[101879]: pgmap v1942: 105 pgs: 105 active+clean; 139 MiB data, 818 MiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 687 KiB/s wr, 44 op/s 2024-09-21T10:36:17.936 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:17 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:36:18.019 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:17 smithi070 ceph-mon[107385]: pgmap v1942: 105 pgs: 105 active+clean; 139 MiB data, 818 MiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 687 KiB/s wr, 44 op/s 2024-09-21T10:36:18.020 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:17 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:36:18.020 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:17 smithi070 ceph-mon[112139]: pgmap v1942: 105 pgs: 105 active+clean; 139 MiB data, 818 MiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 687 KiB/s wr, 44 op/s 2024-09-21T10:36:18.020 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:17 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:36:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:19 smithi160 ceph-mon[101879]: pgmap v1943: 105 pgs: 105 active+clean; 139 MiB data, 818 MiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 593 KiB/s wr, 34 op/s 2024-09-21T10:36:20.066 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:19 smithi070 ceph-mon[107385]: pgmap v1943: 105 pgs: 105 active+clean; 139 MiB data, 818 MiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 593 KiB/s wr, 34 op/s 2024-09-21T10:36:20.067 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:19 smithi070 ceph-mon[112139]: pgmap v1943: 105 pgs: 105 active+clean; 139 MiB data, 818 MiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 593 KiB/s wr, 34 op/s 2024-09-21T10:36:20.945 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:36:20.945 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:36:21.025 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:36:21.778 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:21 smithi160 ceph-mon[101879]: pgmap v1944: 105 pgs: 105 active+clean; 139 MiB data, 818 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 515 KiB/s wr, 29 op/s 2024-09-21T10:36:21.778 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:36:21.778 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:36:21.778 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:36:21.778 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:36:22.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:21 smithi070 ceph-mon[112139]: pgmap v1944: 105 pgs: 105 active+clean; 139 MiB data, 818 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 515 KiB/s wr, 29 op/s 2024-09-21T10:36:22.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:36:22.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:36:22.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:36:22.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:36:22.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:21 smithi070 ceph-mon[107385]: pgmap v1944: 105 pgs: 105 active+clean; 139 MiB data, 818 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 515 KiB/s wr, 29 op/s 2024-09-21T10:36:22.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:36:22.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:36:22.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:36:22.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:36:23.029 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:22 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:36:23.029 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:22 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:36:23.029 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:22 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:36:23.030 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:22 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:36:23.030 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:22 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:36:23.030 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:22 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:36:23.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:22 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:36:23.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:22 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:36:23.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:22 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:36:23.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:36:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:36:23] "GET /metrics HTTP/1.1" 200 36476 "" "Prometheus/2.51.0" 2024-09-21T10:36:24.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:24 smithi070 ceph-mon[107385]: pgmap v1945: 105 pgs: 105 active+clean; 139 MiB data, 818 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 414 KiB/s wr, 24 op/s 2024-09-21T10:36:24.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:24 smithi070 ceph-mon[112139]: pgmap v1945: 105 pgs: 105 active+clean; 139 MiB data, 818 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 414 KiB/s wr, 24 op/s 2024-09-21T10:36:24.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:24 smithi160 ceph-mon[101879]: pgmap v1945: 105 pgs: 105 active+clean; 139 MiB data, 818 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 414 KiB/s wr, 24 op/s 2024-09-21T10:36:25.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:25 smithi160 ceph-mon[101879]: osdmap e1221: 8 total, 8 up, 8 in 2024-09-21T10:36:25.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:25 smithi070 ceph-mon[107385]: osdmap e1221: 8 total, 8 up, 8 in 2024-09-21T10:36:25.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:25 smithi070 ceph-mon[112139]: osdmap e1221: 8 total, 8 up, 8 in 2024-09-21T10:36:26.105 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:36:26.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:26 smithi160 ceph-mon[101879]: pgmap v1947: 105 pgs: 105 active+clean; 147 MiB data, 826 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 821 KiB/s wr, 21 op/s 2024-09-21T10:36:26.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:26 smithi160 ceph-mon[101879]: osdmap e1222: 8 total, 8 up, 8 in 2024-09-21T10:36:26.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:26 smithi070 ceph-mon[107385]: pgmap v1947: 105 pgs: 105 active+clean; 147 MiB data, 826 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 821 KiB/s wr, 21 op/s 2024-09-21T10:36:26.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:26 smithi070 ceph-mon[107385]: osdmap e1222: 8 total, 8 up, 8 in 2024-09-21T10:36:26.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:26 smithi070 ceph-mon[112139]: pgmap v1947: 105 pgs: 105 active+clean; 147 MiB data, 826 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 821 KiB/s wr, 21 op/s 2024-09-21T10:36:26.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:26 smithi070 ceph-mon[112139]: osdmap e1222: 8 total, 8 up, 8 in 2024-09-21T10:36:27.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:27 smithi160 ceph-mon[101879]: osdmap e1223: 8 total, 8 up, 8 in 2024-09-21T10:36:27.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:27 smithi070 ceph-mon[107385]: osdmap e1223: 8 total, 8 up, 8 in 2024-09-21T10:36:27.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:27 smithi070 ceph-mon[112139]: osdmap e1223: 8 total, 8 up, 8 in 2024-09-21T10:36:28.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:28 smithi160 ceph-mon[101879]: pgmap v1950: 105 pgs: 105 active+clean; 155 MiB data, 863 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 2.7 MiB/s wr, 23 op/s 2024-09-21T10:36:28.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:28 smithi070 ceph-mon[107385]: pgmap v1950: 105 pgs: 105 active+clean; 155 MiB data, 863 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 2.7 MiB/s wr, 23 op/s 2024-09-21T10:36:28.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:28 smithi070 ceph-mon[112139]: pgmap v1950: 105 pgs: 105 active+clean; 155 MiB data, 863 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 2.7 MiB/s wr, 23 op/s 2024-09-21T10:36:30.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:30 smithi160 ceph-mon[101879]: pgmap v1951: 105 pgs: 105 active+clean; 155 MiB data, 863 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 2.7 MiB/s wr, 23 op/s 2024-09-21T10:36:30.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:30 smithi070 ceph-mon[107385]: pgmap v1951: 105 pgs: 105 active+clean; 155 MiB data, 863 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 2.7 MiB/s wr, 23 op/s 2024-09-21T10:36:30.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:30 smithi070 ceph-mon[112139]: pgmap v1951: 105 pgs: 105 active+clean; 155 MiB data, 863 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 2.7 MiB/s wr, 23 op/s 2024-09-21T10:36:32.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:32 smithi160 ceph-mon[101879]: pgmap v1952: 105 pgs: 105 active+clean; 155 MiB data, 867 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 1.8 MiB/s wr, 26 op/s 2024-09-21T10:36:32.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:32 smithi070 ceph-mon[107385]: pgmap v1952: 105 pgs: 105 active+clean; 155 MiB data, 867 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 1.8 MiB/s wr, 26 op/s 2024-09-21T10:36:32.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:32 smithi070 ceph-mon[112139]: pgmap v1952: 105 pgs: 105 active+clean; 155 MiB data, 867 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 1.8 MiB/s wr, 26 op/s 2024-09-21T10:36:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:36:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:36:33] "GET /metrics HTTP/1.1" 200 36476 "" "Prometheus/2.51.0" 2024-09-21T10:36:34.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:34 smithi070 ceph-mon[107385]: pgmap v1953: 105 pgs: 105 active+clean; 155 MiB data, 867 MiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 1.0 MiB/s wr, 34 op/s 2024-09-21T10:36:34.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:34 smithi070 ceph-mon[112139]: pgmap v1953: 105 pgs: 105 active+clean; 155 MiB data, 867 MiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 1.0 MiB/s wr, 34 op/s 2024-09-21T10:36:34.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:34 smithi160 ceph-mon[101879]: pgmap v1953: 105 pgs: 105 active+clean; 155 MiB data, 867 MiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 1.0 MiB/s wr, 34 op/s 2024-09-21T10:36:35.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:35 smithi070 ceph-mon[107385]: osdmap e1224: 8 total, 8 up, 8 in 2024-09-21T10:36:35.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:35 smithi070 ceph-mon[112139]: osdmap e1224: 8 total, 8 up, 8 in 2024-09-21T10:36:35.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:35 smithi160 ceph-mon[101879]: osdmap e1224: 8 total, 8 up, 8 in 2024-09-21T10:36:36.139 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:36:36.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:36 smithi070 ceph-mon[112139]: pgmap v1955: 105 pgs: 105 active+clean; 160 MiB data, 879 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 483 KiB/s wr, 16 op/s 2024-09-21T10:36:36.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:36 smithi070 ceph-mon[112139]: osdmap e1225: 8 total, 8 up, 8 in 2024-09-21T10:36:36.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:36:36.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:36 smithi070 ceph-mon[107385]: pgmap v1955: 105 pgs: 105 active+clean; 160 MiB data, 879 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 483 KiB/s wr, 16 op/s 2024-09-21T10:36:36.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:36 smithi070 ceph-mon[107385]: osdmap e1225: 8 total, 8 up, 8 in 2024-09-21T10:36:36.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:36:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:36 smithi160 ceph-mon[101879]: pgmap v1955: 105 pgs: 105 active+clean; 160 MiB data, 879 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 483 KiB/s wr, 16 op/s 2024-09-21T10:36:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:36 smithi160 ceph-mon[101879]: osdmap e1225: 8 total, 8 up, 8 in 2024-09-21T10:36:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:36:37.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:37 smithi070 ceph-mon[107385]: osdmap e1226: 8 total, 8 up, 8 in 2024-09-21T10:36:37.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:37 smithi070 ceph-mon[112139]: osdmap e1226: 8 total, 8 up, 8 in 2024-09-21T10:36:37.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:37 smithi160 ceph-mon[101879]: osdmap e1226: 8 total, 8 up, 8 in 2024-09-21T10:36:38.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:38 smithi070 ceph-mon[107385]: pgmap v1958: 105 pgs: 105 active+clean; 173 MiB data, 915 MiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 2.7 MiB/s wr, 41 op/s 2024-09-21T10:36:38.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:38 smithi070 ceph-mon[112139]: pgmap v1958: 105 pgs: 105 active+clean; 173 MiB data, 915 MiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 2.7 MiB/s wr, 41 op/s 2024-09-21T10:36:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:38 smithi160 ceph-mon[101879]: pgmap v1958: 105 pgs: 105 active+clean; 173 MiB data, 915 MiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 2.7 MiB/s wr, 41 op/s 2024-09-21T10:36:40.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:40 smithi070 ceph-mon[107385]: pgmap v1959: 105 pgs: 105 active+clean; 173 MiB data, 915 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 2.7 MiB/s wr, 23 op/s 2024-09-21T10:36:40.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:40 smithi070 ceph-mon[112139]: pgmap v1959: 105 pgs: 105 active+clean; 173 MiB data, 915 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 2.7 MiB/s wr, 23 op/s 2024-09-21T10:36:40.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:40 smithi160 ceph-mon[101879]: pgmap v1959: 105 pgs: 105 active+clean; 173 MiB data, 915 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 2.7 MiB/s wr, 23 op/s 2024-09-21T10:36:42.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:42 smithi070 ceph-mon[107385]: pgmap v1960: 105 pgs: 105 active+clean; 173 MiB data, 915 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.5 MiB/s wr, 24 op/s 2024-09-21T10:36:42.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:42 smithi070 ceph-mon[112139]: pgmap v1960: 105 pgs: 105 active+clean; 173 MiB data, 915 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.5 MiB/s wr, 24 op/s 2024-09-21T10:36:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:42 smithi160 ceph-mon[101879]: pgmap v1960: 105 pgs: 105 active+clean; 173 MiB data, 915 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.5 MiB/s wr, 24 op/s 2024-09-21T10:36:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:36:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:36:43] "GET /metrics HTTP/1.1" 200 36472 "" "Prometheus/2.51.0" 2024-09-21T10:36:44.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:44 smithi070 ceph-mon[107385]: pgmap v1961: 105 pgs: 105 active+clean; 173 MiB data, 915 MiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 1.5 MiB/s wr, 34 op/s 2024-09-21T10:36:44.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:44 smithi070 ceph-mon[112139]: pgmap v1961: 105 pgs: 105 active+clean; 173 MiB data, 915 MiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 1.5 MiB/s wr, 34 op/s 2024-09-21T10:36:44.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:44 smithi160 ceph-mon[101879]: pgmap v1961: 105 pgs: 105 active+clean; 173 MiB data, 915 MiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 1.5 MiB/s wr, 34 op/s 2024-09-21T10:36:45.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:45 smithi160 ceph-mon[101879]: osdmap e1227: 8 total, 8 up, 8 in 2024-09-21T10:36:45.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:45 smithi070 ceph-mon[107385]: osdmap e1227: 8 total, 8 up, 8 in 2024-09-21T10:36:45.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:45 smithi070 ceph-mon[112139]: osdmap e1227: 8 total, 8 up, 8 in 2024-09-21T10:36:46.164 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:36:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:46 smithi160 ceph-mon[101879]: pgmap v1963: 105 pgs: 105 active+clean; 181 MiB data, 923 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 964 KiB/s wr, 16 op/s 2024-09-21T10:36:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:46 smithi160 ceph-mon[101879]: osdmap e1228: 8 total, 8 up, 8 in 2024-09-21T10:36:46.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:46 smithi070 ceph-mon[107385]: pgmap v1963: 105 pgs: 105 active+clean; 181 MiB data, 923 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 964 KiB/s wr, 16 op/s 2024-09-21T10:36:46.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:46 smithi070 ceph-mon[107385]: osdmap e1228: 8 total, 8 up, 8 in 2024-09-21T10:36:46.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:46 smithi070 ceph-mon[112139]: pgmap v1963: 105 pgs: 105 active+clean; 181 MiB data, 923 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 964 KiB/s wr, 16 op/s 2024-09-21T10:36:46.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:46 smithi070 ceph-mon[112139]: osdmap e1228: 8 total, 8 up, 8 in 2024-09-21T10:36:47.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:47 smithi160 ceph-mon[101879]: osdmap e1229: 8 total, 8 up, 8 in 2024-09-21T10:36:47.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:47 smithi070 ceph-mon[107385]: osdmap e1229: 8 total, 8 up, 8 in 2024-09-21T10:36:47.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:47 smithi070 ceph-mon[112139]: osdmap e1229: 8 total, 8 up, 8 in 2024-09-21T10:36:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:48 smithi160 ceph-mon[101879]: pgmap v1966: 105 pgs: 105 active+clean; 191 MiB data, 952 MiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 2.7 MiB/s wr, 43 op/s 2024-09-21T10:36:48.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:48 smithi070 ceph-mon[112139]: pgmap v1966: 105 pgs: 105 active+clean; 191 MiB data, 952 MiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 2.7 MiB/s wr, 43 op/s 2024-09-21T10:36:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:48 smithi070 ceph-mon[107385]: pgmap v1966: 105 pgs: 105 active+clean; 191 MiB data, 952 MiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 2.7 MiB/s wr, 43 op/s 2024-09-21T10:36:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:50 smithi160 ceph-mon[101879]: pgmap v1967: 105 pgs: 105 active+clean; 191 MiB data, 952 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 2.7 MiB/s wr, 23 op/s 2024-09-21T10:36:50.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:50 smithi070 ceph-mon[112139]: pgmap v1967: 105 pgs: 105 active+clean; 191 MiB data, 952 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 2.7 MiB/s wr, 23 op/s 2024-09-21T10:36:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:50 smithi070 ceph-mon[107385]: pgmap v1967: 105 pgs: 105 active+clean; 191 MiB data, 952 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 2.7 MiB/s wr, 23 op/s 2024-09-21T10:36:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:36:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:36:51.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:36:52.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:52 smithi160 ceph-mon[101879]: pgmap v1968: 105 pgs: 105 active+clean; 191 MiB data, 964 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 2.5 MiB/s wr, 30 op/s 2024-09-21T10:36:52.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:52 smithi070 ceph-mon[107385]: pgmap v1968: 105 pgs: 105 active+clean; 191 MiB data, 964 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 2.5 MiB/s wr, 30 op/s 2024-09-21T10:36:52.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:52 smithi070 ceph-mon[112139]: pgmap v1968: 105 pgs: 105 active+clean; 191 MiB data, 964 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 2.5 MiB/s wr, 30 op/s 2024-09-21T10:36:53.527 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:36:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:36:53] "GET /metrics HTTP/1.1" 200 36476 "" "Prometheus/2.51.0" 2024-09-21T10:36:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:54 smithi160 ceph-mon[101879]: pgmap v1969: 105 pgs: 105 active+clean; 191 MiB data, 964 MiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 1.0 MiB/s wr, 34 op/s 2024-09-21T10:36:54.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:54 smithi070 ceph-mon[107385]: pgmap v1969: 105 pgs: 105 active+clean; 191 MiB data, 964 MiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 1.0 MiB/s wr, 34 op/s 2024-09-21T10:36:54.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:54 smithi070 ceph-mon[112139]: pgmap v1969: 105 pgs: 105 active+clean; 191 MiB data, 964 MiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 1.0 MiB/s wr, 34 op/s 2024-09-21T10:36:55.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:55 smithi160 ceph-mon[101879]: osdmap e1230: 8 total, 8 up, 8 in 2024-09-21T10:36:55.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:55 smithi070 ceph-mon[112139]: osdmap e1230: 8 total, 8 up, 8 in 2024-09-21T10:36:55.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:55 smithi070 ceph-mon[107385]: osdmap e1230: 8 total, 8 up, 8 in 2024-09-21T10:36:56.200 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:36:56.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:56 smithi160 ceph-mon[101879]: pgmap v1971: 105 pgs: 105 active+clean; 203 MiB data, 980 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 1.4 MiB/s wr, 24 op/s 2024-09-21T10:36:56.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:56 smithi160 ceph-mon[101879]: osdmap e1231: 8 total, 8 up, 8 in 2024-09-21T10:36:56.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:56 smithi070 ceph-mon[107385]: pgmap v1971: 105 pgs: 105 active+clean; 203 MiB data, 980 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 1.4 MiB/s wr, 24 op/s 2024-09-21T10:36:56.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:56 smithi070 ceph-mon[107385]: osdmap e1231: 8 total, 8 up, 8 in 2024-09-21T10:36:56.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:56 smithi070 ceph-mon[112139]: pgmap v1971: 105 pgs: 105 active+clean; 203 MiB data, 980 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 1.4 MiB/s wr, 24 op/s 2024-09-21T10:36:56.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:56 smithi070 ceph-mon[112139]: osdmap e1231: 8 total, 8 up, 8 in 2024-09-21T10:36:57.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:57 smithi160 ceph-mon[101879]: osdmap e1232: 8 total, 8 up, 8 in 2024-09-21T10:36:57.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:57 smithi070 ceph-mon[112139]: osdmap e1232: 8 total, 8 up, 8 in 2024-09-21T10:36:57.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:57 smithi070 ceph-mon[107385]: osdmap e1232: 8 total, 8 up, 8 in 2024-09-21T10:36:58.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:36:58 smithi160 ceph-mon[101879]: pgmap v1974: 105 pgs: 105 active+clean; 208 MiB data, 1008 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 2.7 MiB/s wr, 26 op/s 2024-09-21T10:36:58.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:36:58 smithi070 ceph-mon[107385]: pgmap v1974: 105 pgs: 105 active+clean; 208 MiB data, 1008 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 2.7 MiB/s wr, 26 op/s 2024-09-21T10:36:58.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:36:58 smithi070 ceph-mon[112139]: pgmap v1974: 105 pgs: 105 active+clean; 208 MiB data, 1008 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 2.7 MiB/s wr, 26 op/s 2024-09-21T10:37:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:00 smithi160 ceph-mon[101879]: pgmap v1975: 105 pgs: 105 active+clean; 208 MiB data, 1008 MiB used, 714 GiB / 715 GiB avail; 7.8 KiB/s rd, 2.7 MiB/s wr, 12 op/s 2024-09-21T10:37:00.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:00 smithi070 ceph-mon[107385]: pgmap v1975: 105 pgs: 105 active+clean; 208 MiB data, 1008 MiB used, 714 GiB / 715 GiB avail; 7.8 KiB/s rd, 2.7 MiB/s wr, 12 op/s 2024-09-21T10:37:00.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:00 smithi070 ceph-mon[112139]: pgmap v1975: 105 pgs: 105 active+clean; 208 MiB data, 1008 MiB used, 714 GiB / 715 GiB avail; 7.8 KiB/s rd, 2.7 MiB/s wr, 12 op/s 2024-09-21T10:37:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:02 smithi160 ceph-mon[101879]: pgmap v1976: 105 pgs: 105 active+clean; 209 MiB data, 1012 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.5 MiB/s wr, 37 op/s 2024-09-21T10:37:02.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:02 smithi070 ceph-mon[112139]: pgmap v1976: 105 pgs: 105 active+clean; 209 MiB data, 1012 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.5 MiB/s wr, 37 op/s 2024-09-21T10:37:02.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:02 smithi070 ceph-mon[107385]: pgmap v1976: 105 pgs: 105 active+clean; 209 MiB data, 1012 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.5 MiB/s wr, 37 op/s 2024-09-21T10:37:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:37:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:37:03] "GET /metrics HTTP/1.1" 200 36476 "" "Prometheus/2.51.0" 2024-09-21T10:37:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:04 smithi160 ceph-mon[101879]: pgmap v1977: 105 pgs: 105 active+clean; 209 MiB data, 1012 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 515 KiB/s wr, 25 op/s 2024-09-21T10:37:04.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:04 smithi070 ceph-mon[107385]: pgmap v1977: 105 pgs: 105 active+clean; 209 MiB data, 1012 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 515 KiB/s wr, 25 op/s 2024-09-21T10:37:04.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:04 smithi070 ceph-mon[112139]: pgmap v1977: 105 pgs: 105 active+clean; 209 MiB data, 1012 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 515 KiB/s wr, 25 op/s 2024-09-21T10:37:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:06 smithi160 ceph-mon[101879]: pgmap v1978: 105 pgs: 105 active+clean; 213 MiB data, 1012 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 870 KiB/s wr, 21 op/s 2024-09-21T10:37:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:06 smithi160 ceph-mon[101879]: osdmap e1233: 8 total, 8 up, 8 in 2024-09-21T10:37:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:37:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:06 smithi070 ceph-mon[107385]: pgmap v1978: 105 pgs: 105 active+clean; 213 MiB data, 1012 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 870 KiB/s wr, 21 op/s 2024-09-21T10:37:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:06 smithi070 ceph-mon[107385]: osdmap e1233: 8 total, 8 up, 8 in 2024-09-21T10:37:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:37:06.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:06 smithi070 ceph-mon[112139]: pgmap v1978: 105 pgs: 105 active+clean; 213 MiB data, 1012 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 870 KiB/s wr, 21 op/s 2024-09-21T10:37:06.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:06 smithi070 ceph-mon[112139]: osdmap e1233: 8 total, 8 up, 8 in 2024-09-21T10:37:06.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:37:07.230 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/1.DiffIterateDeterministicLUKS2 (53530 ms) 2024-09-21T10:37:07.230 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/1.DiffIterateDeterministicLUKS2PP 2024-09-21T10:37:07.230 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:37:07.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:07 smithi160 ceph-mon[101879]: osdmap e1234: 8 total, 8 up, 8 in 2024-09-21T10:37:07.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:07 smithi070 ceph-mon[107385]: osdmap e1234: 8 total, 8 up, 8 in 2024-09-21T10:37:07.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:07 smithi070 ceph-mon[112139]: osdmap e1234: 8 total, 8 up, 8 in 2024-09-21T10:37:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:08 smithi160 ceph-mon[101879]: pgmap v1981: 105 pgs: 105 active+clean; 229 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 2.0 MiB/s wr, 32 op/s 2024-09-21T10:37:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:08 smithi160 ceph-mon[101879]: osdmap e1235: 8 total, 8 up, 8 in 2024-09-21T10:37:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:08 smithi070 ceph-mon[107385]: pgmap v1981: 105 pgs: 105 active+clean; 229 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 2.0 MiB/s wr, 32 op/s 2024-09-21T10:37:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:08 smithi070 ceph-mon[107385]: osdmap e1235: 8 total, 8 up, 8 in 2024-09-21T10:37:08.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:08 smithi070 ceph-mon[112139]: pgmap v1981: 105 pgs: 105 active+clean; 229 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 2.0 MiB/s wr, 32 op/s 2024-09-21T10:37:08.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:08 smithi070 ceph-mon[112139]: osdmap e1235: 8 total, 8 up, 8 in 2024-09-21T10:37:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:10 smithi160 ceph-mon[101879]: pgmap v1983: 105 pgs: 105 active+clean; 229 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.7 MiB/s wr, 9 op/s 2024-09-21T10:37:10.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:10 smithi070 ceph-mon[107385]: pgmap v1983: 105 pgs: 105 active+clean; 229 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.7 MiB/s wr, 9 op/s 2024-09-21T10:37:10.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:10 smithi070 ceph-mon[112139]: pgmap v1983: 105 pgs: 105 active+clean; 229 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.7 MiB/s wr, 9 op/s 2024-09-21T10:37:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:12 smithi160 ceph-mon[101879]: pgmap v1984: 105 pgs: 105 active+clean; 233 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 2.0 MiB/s wr, 38 op/s 2024-09-21T10:37:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:12 smithi070 ceph-mon[107385]: pgmap v1984: 105 pgs: 105 active+clean; 233 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 2.0 MiB/s wr, 38 op/s 2024-09-21T10:37:12.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:12 smithi070 ceph-mon[112139]: pgmap v1984: 105 pgs: 105 active+clean; 233 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 2.0 MiB/s wr, 38 op/s 2024-09-21T10:37:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:37:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:37:13] "GET /metrics HTTP/1.1" 200 36477 "" "Prometheus/2.51.0" 2024-09-21T10:37:14.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:14 smithi160 ceph-mon[101879]: pgmap v1985: 105 pgs: 105 active+clean; 233 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 1.6 MiB/s wr, 36 op/s 2024-09-21T10:37:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:14 smithi070 ceph-mon[107385]: pgmap v1985: 105 pgs: 105 active+clean; 233 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 1.6 MiB/s wr, 36 op/s 2024-09-21T10:37:14.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:14 smithi070 ceph-mon[112139]: pgmap v1985: 105 pgs: 105 active+clean; 233 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 1.6 MiB/s wr, 36 op/s 2024-09-21T10:37:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:16 smithi160 ceph-mon[101879]: pgmap v1986: 105 pgs: 105 active+clean; 233 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 2.5 KiB/s wr, 26 op/s 2024-09-21T10:37:16.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:16 smithi070 ceph-mon[107385]: pgmap v1986: 105 pgs: 105 active+clean; 233 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 2.5 KiB/s wr, 26 op/s 2024-09-21T10:37:16.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:16 smithi070 ceph-mon[112139]: pgmap v1986: 105 pgs: 105 active+clean; 233 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 2.5 KiB/s wr, 26 op/s 2024-09-21T10:37:17.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:17 smithi160 ceph-mon[101879]: osdmap e1236: 8 total, 8 up, 8 in 2024-09-21T10:37:17.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:17 smithi070 ceph-mon[107385]: osdmap e1236: 8 total, 8 up, 8 in 2024-09-21T10:37:17.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:17 smithi070 ceph-mon[112139]: osdmap e1236: 8 total, 8 up, 8 in 2024-09-21T10:37:18.267 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:37:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:18 smithi160 ceph-mon[101879]: pgmap v1988: 105 pgs: 105 active+clean; 237 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 437 KiB/s wr, 23 op/s 2024-09-21T10:37:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:18 smithi160 ceph-mon[101879]: osdmap e1237: 8 total, 8 up, 8 in 2024-09-21T10:37:18.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:18 smithi070 ceph-mon[112139]: pgmap v1988: 105 pgs: 105 active+clean; 237 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 437 KiB/s wr, 23 op/s 2024-09-21T10:37:18.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:18 smithi070 ceph-mon[112139]: osdmap e1237: 8 total, 8 up, 8 in 2024-09-21T10:37:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:18 smithi070 ceph-mon[107385]: pgmap v1988: 105 pgs: 105 active+clean; 237 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 437 KiB/s wr, 23 op/s 2024-09-21T10:37:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:18 smithi070 ceph-mon[107385]: osdmap e1237: 8 total, 8 up, 8 in 2024-09-21T10:37:19.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:19 smithi160 ceph-mon[101879]: osdmap e1238: 8 total, 8 up, 8 in 2024-09-21T10:37:19.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:19 smithi070 ceph-mon[107385]: osdmap e1238: 8 total, 8 up, 8 in 2024-09-21T10:37:19.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:19 smithi070 ceph-mon[112139]: osdmap e1238: 8 total, 8 up, 8 in 2024-09-21T10:37:20.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:20 smithi160 ceph-mon[101879]: pgmap v1991: 105 pgs: 105 active+clean; 237 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 683 KiB/s wr, 0 op/s 2024-09-21T10:37:20.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:20 smithi070 ceph-mon[107385]: pgmap v1991: 105 pgs: 105 active+clean; 237 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 683 KiB/s wr, 0 op/s 2024-09-21T10:37:20.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:20 smithi070 ceph-mon[112139]: pgmap v1991: 105 pgs: 105 active+clean; 237 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 683 KiB/s wr, 0 op/s 2024-09-21T10:37:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:37:21.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:37:21.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:37:22.566 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:22 smithi160 ceph-mon[101879]: pgmap v1992: 105 pgs: 105 active+clean; 249 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.7 MiB/s wr, 39 op/s 2024-09-21T10:37:22.566 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:22 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:37:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:22 smithi070 ceph-mon[107385]: pgmap v1992: 105 pgs: 105 active+clean; 249 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.7 MiB/s wr, 39 op/s 2024-09-21T10:37:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:22 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:37:22.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:22 smithi070 ceph-mon[112139]: pgmap v1992: 105 pgs: 105 active+clean; 249 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.7 MiB/s wr, 39 op/s 2024-09-21T10:37:22.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:22 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:37:23.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:37:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:37:23] "GET /metrics HTTP/1.1" 200 36477 "" "Prometheus/2.51.0" 2024-09-21T10:37:24.259 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:24 smithi070 ceph-mon[107385]: pgmap v1993: 105 pgs: 105 active+clean; 249 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 1.9 MiB/s wr, 43 op/s 2024-09-21T10:37:24.259 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:24 smithi070 ceph-mon[112139]: pgmap v1993: 105 pgs: 105 active+clean; 249 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 1.9 MiB/s wr, 43 op/s 2024-09-21T10:37:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:24 smithi160 ceph-mon[101879]: pgmap v1993: 105 pgs: 105 active+clean; 249 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 1.9 MiB/s wr, 43 op/s 2024-09-21T10:37:26.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:26 smithi160 ceph-mon[101879]: pgmap v1994: 105 pgs: 105 active+clean; 249 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 1.5 MiB/s wr, 34 op/s 2024-09-21T10:37:26.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:26 smithi160 ceph-mon[101879]: osdmap e1239: 8 total, 8 up, 8 in 2024-09-21T10:37:26.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:26 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:37:26.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:26 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:37:26.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:26 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:37:26.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:26 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:37:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:26 smithi070 ceph-mon[107385]: pgmap v1994: 105 pgs: 105 active+clean; 249 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 1.5 MiB/s wr, 34 op/s 2024-09-21T10:37:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:26 smithi070 ceph-mon[107385]: osdmap e1239: 8 total, 8 up, 8 in 2024-09-21T10:37:26.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:26 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:37:26.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:26 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:37:26.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:26 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:37:26.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:26 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:37:26.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:26 smithi070 ceph-mon[112139]: pgmap v1994: 105 pgs: 105 active+clean; 249 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 1.5 MiB/s wr, 34 op/s 2024-09-21T10:37:26.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:26 smithi070 ceph-mon[112139]: osdmap e1239: 8 total, 8 up, 8 in 2024-09-21T10:37:26.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:26 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:37:26.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:26 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:37:26.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:26 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:37:26.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:26 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:37:27.302 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:37:27.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:27 smithi160 ceph-mon[101879]: osdmap e1240: 8 total, 8 up, 8 in 2024-09-21T10:37:27.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:27 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:37:27.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:27 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:37:27.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:27 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:37:27.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:27 smithi070 ceph-mon[107385]: osdmap e1240: 8 total, 8 up, 8 in 2024-09-21T10:37:27.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:27 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:37:27.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:27 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:37:27.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:27 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:37:27.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:27 smithi070 ceph-mon[112139]: osdmap e1240: 8 total, 8 up, 8 in 2024-09-21T10:37:27.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:27 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:37:27.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:27 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:37:27.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:27 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:37:28.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:28 smithi160 ceph-mon[101879]: pgmap v1997: 105 pgs: 105 active+clean; 268 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 3.5 MiB/s wr, 45 op/s 2024-09-21T10:37:28.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:28 smithi160 ceph-mon[101879]: osdmap e1241: 8 total, 8 up, 8 in 2024-09-21T10:37:28.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:28 smithi070 ceph-mon[107385]: pgmap v1997: 105 pgs: 105 active+clean; 268 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 3.5 MiB/s wr, 45 op/s 2024-09-21T10:37:28.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:28 smithi070 ceph-mon[107385]: osdmap e1241: 8 total, 8 up, 8 in 2024-09-21T10:37:28.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:28 smithi070 ceph-mon[112139]: pgmap v1997: 105 pgs: 105 active+clean; 268 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 3.5 MiB/s wr, 45 op/s 2024-09-21T10:37:28.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:28 smithi070 ceph-mon[112139]: osdmap e1241: 8 total, 8 up, 8 in 2024-09-21T10:37:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:30 smithi160 ceph-mon[101879]: pgmap v1999: 105 pgs: 105 active+clean; 268 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.0 KiB/s rd, 2.7 MiB/s wr, 14 op/s 2024-09-21T10:37:30.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:30 smithi070 ceph-mon[107385]: pgmap v1999: 105 pgs: 105 active+clean; 268 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.0 KiB/s rd, 2.7 MiB/s wr, 14 op/s 2024-09-21T10:37:30.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:30 smithi070 ceph-mon[112139]: pgmap v1999: 105 pgs: 105 active+clean; 268 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.0 KiB/s rd, 2.7 MiB/s wr, 14 op/s 2024-09-21T10:37:32.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:32 smithi160 ceph-mon[101879]: pgmap v2000: 105 pgs: 105 active+clean; 268 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 25 op/s 2024-09-21T10:37:32.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:32 smithi070 ceph-mon[107385]: pgmap v2000: 105 pgs: 105 active+clean; 268 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 25 op/s 2024-09-21T10:37:32.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:32 smithi070 ceph-mon[112139]: pgmap v2000: 105 pgs: 105 active+clean; 268 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 25 op/s 2024-09-21T10:37:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:37:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:37:33] "GET /metrics HTTP/1.1" 200 36477 "" "Prometheus/2.51.0" 2024-09-21T10:37:34.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:34 smithi160 ceph-mon[101879]: pgmap v2001: 105 pgs: 105 active+clean; 271 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.2 MiB/s wr, 37 op/s 2024-09-21T10:37:34.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:34 smithi070 ceph-mon[107385]: pgmap v2001: 105 pgs: 105 active+clean; 271 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.2 MiB/s wr, 37 op/s 2024-09-21T10:37:34.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:34 smithi070 ceph-mon[112139]: pgmap v2001: 105 pgs: 105 active+clean; 271 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.2 MiB/s wr, 37 op/s 2024-09-21T10:37:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:36 smithi160 ceph-mon[101879]: pgmap v2002: 105 pgs: 105 active+clean; 271 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 2.3 KiB/s wr, 22 op/s 2024-09-21T10:37:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:36 smithi160 ceph-mon[101879]: osdmap e1242: 8 total, 8 up, 8 in 2024-09-21T10:37:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:37:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:36 smithi070 ceph-mon[107385]: pgmap v2002: 105 pgs: 105 active+clean; 271 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 2.3 KiB/s wr, 22 op/s 2024-09-21T10:37:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:36 smithi070 ceph-mon[107385]: osdmap e1242: 8 total, 8 up, 8 in 2024-09-21T10:37:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:37:36.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:36 smithi070 ceph-mon[112139]: pgmap v2002: 105 pgs: 105 active+clean; 271 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 2.3 KiB/s wr, 22 op/s 2024-09-21T10:37:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:36 smithi070 ceph-mon[112139]: osdmap e1242: 8 total, 8 up, 8 in 2024-09-21T10:37:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:37:37.333 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:37:37.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:37 smithi160 ceph-mon[101879]: osdmap e1243: 8 total, 8 up, 8 in 2024-09-21T10:37:37.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:37 smithi070 ceph-mon[112139]: osdmap e1243: 8 total, 8 up, 8 in 2024-09-21T10:37:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:37 smithi070 ceph-mon[107385]: osdmap e1243: 8 total, 8 up, 8 in 2024-09-21T10:37:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:38 smithi160 ceph-mon[101879]: pgmap v2005: 105 pgs: 105 active+clean; 278 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 517 KiB/s wr, 30 op/s 2024-09-21T10:37:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:38 smithi160 ceph-mon[101879]: osdmap e1244: 8 total, 8 up, 8 in 2024-09-21T10:37:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:38 smithi070 ceph-mon[107385]: pgmap v2005: 105 pgs: 105 active+clean; 278 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 517 KiB/s wr, 30 op/s 2024-09-21T10:37:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:38 smithi070 ceph-mon[107385]: osdmap e1244: 8 total, 8 up, 8 in 2024-09-21T10:37:38.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:38 smithi070 ceph-mon[112139]: pgmap v2005: 105 pgs: 105 active+clean; 278 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 517 KiB/s wr, 30 op/s 2024-09-21T10:37:38.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:38 smithi070 ceph-mon[112139]: osdmap e1244: 8 total, 8 up, 8 in 2024-09-21T10:37:40.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:40 smithi160 ceph-mon[101879]: pgmap v2007: 105 pgs: 105 active+clean; 278 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 686 KiB/s wr, 8 op/s 2024-09-21T10:37:40.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:40 smithi070 ceph-mon[107385]: pgmap v2007: 105 pgs: 105 active+clean; 278 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 686 KiB/s wr, 8 op/s 2024-09-21T10:37:40.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:40 smithi070 ceph-mon[112139]: pgmap v2007: 105 pgs: 105 active+clean; 278 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 686 KiB/s wr, 8 op/s 2024-09-21T10:37:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:42 smithi160 ceph-mon[101879]: pgmap v2008: 105 pgs: 105 active+clean; 293 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.7 MiB/s wr, 35 op/s 2024-09-21T10:37:42.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:42 smithi070 ceph-mon[107385]: pgmap v2008: 105 pgs: 105 active+clean; 293 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.7 MiB/s wr, 35 op/s 2024-09-21T10:37:42.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:42 smithi070 ceph-mon[112139]: pgmap v2008: 105 pgs: 105 active+clean; 293 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.7 MiB/s wr, 35 op/s 2024-09-21T10:37:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:37:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:37:43] "GET /metrics HTTP/1.1" 200 36480 "" "Prometheus/2.51.0" 2024-09-21T10:37:44.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:44 smithi160 ceph-mon[101879]: pgmap v2009: 105 pgs: 105 active+clean; 293 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.2 MiB/s wr, 37 op/s 2024-09-21T10:37:44.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:44 smithi070 ceph-mon[107385]: pgmap v2009: 105 pgs: 105 active+clean; 293 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.2 MiB/s wr, 37 op/s 2024-09-21T10:37:44.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:44 smithi070 ceph-mon[112139]: pgmap v2009: 105 pgs: 105 active+clean; 293 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.2 MiB/s wr, 37 op/s 2024-09-21T10:37:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:46 smithi160 ceph-mon[101879]: pgmap v2010: 105 pgs: 105 active+clean; 293 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 1.4 MiB/s wr, 27 op/s 2024-09-21T10:37:46.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:46 smithi070 ceph-mon[107385]: pgmap v2010: 105 pgs: 105 active+clean; 293 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 1.4 MiB/s wr, 27 op/s 2024-09-21T10:37:46.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:46 smithi070 ceph-mon[112139]: pgmap v2010: 105 pgs: 105 active+clean; 293 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 1.4 MiB/s wr, 27 op/s 2024-09-21T10:37:47.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:47 smithi160 ceph-mon[101879]: osdmap e1245: 8 total, 8 up, 8 in 2024-09-21T10:37:47.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:47 smithi070 ceph-mon[107385]: osdmap e1245: 8 total, 8 up, 8 in 2024-09-21T10:37:47.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:47 smithi070 ceph-mon[112139]: osdmap e1245: 8 total, 8 up, 8 in 2024-09-21T10:37:48.360 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:37:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:48 smithi160 ceph-mon[101879]: pgmap v2012: 105 pgs: 105 active+clean; 301 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.1 MiB/s wr, 24 op/s 2024-09-21T10:37:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:48 smithi160 ceph-mon[101879]: osdmap e1246: 8 total, 8 up, 8 in 2024-09-21T10:37:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:48 smithi070 ceph-mon[107385]: pgmap v2012: 105 pgs: 105 active+clean; 301 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.1 MiB/s wr, 24 op/s 2024-09-21T10:37:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:48 smithi070 ceph-mon[107385]: osdmap e1246: 8 total, 8 up, 8 in 2024-09-21T10:37:48.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:48 smithi070 ceph-mon[112139]: pgmap v2012: 105 pgs: 105 active+clean; 301 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.1 MiB/s wr, 24 op/s 2024-09-21T10:37:48.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:48 smithi070 ceph-mon[112139]: osdmap e1246: 8 total, 8 up, 8 in 2024-09-21T10:37:49.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:49 smithi160 ceph-mon[101879]: osdmap e1247: 8 total, 8 up, 8 in 2024-09-21T10:37:49.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:49 smithi070 ceph-mon[107385]: osdmap e1247: 8 total, 8 up, 8 in 2024-09-21T10:37:49.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:49 smithi070 ceph-mon[112139]: osdmap e1247: 8 total, 8 up, 8 in 2024-09-21T10:37:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:50 smithi160 ceph-mon[101879]: pgmap v2015: 105 pgs: 105 active+clean; 301 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s wr, 0 op/s 2024-09-21T10:37:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:50 smithi070 ceph-mon[107385]: pgmap v2015: 105 pgs: 105 active+clean; 301 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s wr, 0 op/s 2024-09-21T10:37:50.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:50 smithi070 ceph-mon[112139]: pgmap v2015: 105 pgs: 105 active+clean; 301 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s wr, 0 op/s 2024-09-21T10:37:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:37:51.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:37:51.705 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:37:52.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:52 smithi070 ceph-mon[107385]: pgmap v2016: 105 pgs: 105 active+clean; 308 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 2.0 MiB/s wr, 27 op/s 2024-09-21T10:37:52.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:52 smithi070 ceph-mon[112139]: pgmap v2016: 105 pgs: 105 active+clean; 308 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 2.0 MiB/s wr, 27 op/s 2024-09-21T10:37:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:52 smithi160 ceph-mon[101879]: pgmap v2016: 105 pgs: 105 active+clean; 308 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 2.0 MiB/s wr, 27 op/s 2024-09-21T10:37:53.529 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:37:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:37:53] "GET /metrics HTTP/1.1" 200 36482 "" "Prometheus/2.51.0" 2024-09-21T10:37:54.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:54 smithi070 ceph-mon[107385]: pgmap v2017: 105 pgs: 105 active+clean; 315 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 1.3 MiB/s wr, 43 op/s 2024-09-21T10:37:54.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:54 smithi070 ceph-mon[112139]: pgmap v2017: 105 pgs: 105 active+clean; 315 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 1.3 MiB/s wr, 43 op/s 2024-09-21T10:37:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:54 smithi160 ceph-mon[101879]: pgmap v2017: 105 pgs: 105 active+clean; 315 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 1.3 MiB/s wr, 43 op/s 2024-09-21T10:37:56.349 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:56 smithi070 ceph-mon[107385]: pgmap v2018: 105 pgs: 105 active+clean; 315 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 1.0 MiB/s wr, 34 op/s 2024-09-21T10:37:56.349 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:56 smithi070 ceph-mon[112139]: pgmap v2018: 105 pgs: 105 active+clean; 315 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 1.0 MiB/s wr, 34 op/s 2024-09-21T10:37:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:56 smithi160 ceph-mon[101879]: pgmap v2018: 105 pgs: 105 active+clean; 315 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 1.0 MiB/s wr, 34 op/s 2024-09-21T10:37:58.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:58 smithi070 ceph-mon[107385]: pgmap v2019: 105 pgs: 105 active+clean; 315 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 885 KiB/s wr, 29 op/s 2024-09-21T10:37:58.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:58 smithi070 ceph-mon[107385]: osdmap e1248: 8 total, 8 up, 8 in 2024-09-21T10:37:58.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:58 smithi070 ceph-mon[112139]: pgmap v2019: 105 pgs: 105 active+clean; 315 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 885 KiB/s wr, 29 op/s 2024-09-21T10:37:58.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:58 smithi070 ceph-mon[112139]: osdmap e1248: 8 total, 8 up, 8 in 2024-09-21T10:37:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:58 smithi160 ceph-mon[101879]: pgmap v2019: 105 pgs: 105 active+clean; 315 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 885 KiB/s wr, 29 op/s 2024-09-21T10:37:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:58 smithi160 ceph-mon[101879]: osdmap e1248: 8 total, 8 up, 8 in 2024-09-21T10:37:59.392 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/1.DiffIterateDeterministicLUKS2PP (52162 ms) 2024-09-21T10:37:59.393 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/1.DiffIterateDiscard 2024-09-21T10:37:59.393 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:37:59.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:37:59 smithi070 ceph-mon[112139]: osdmap e1249: 8 total, 8 up, 8 in 2024-09-21T10:37:59.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:37:59 smithi070 ceph-mon[107385]: osdmap e1249: 8 total, 8 up, 8 in 2024-09-21T10:37:59.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:37:59 smithi160 ceph-mon[101879]: osdmap e1249: 8 total, 8 up, 8 in 2024-09-21T10:38:00.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:00 smithi070 ceph-mon[107385]: pgmap v2022: 105 pgs: 105 active+clean; 315 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.5 KiB/s rd, 515 KiB/s wr, 13 op/s 2024-09-21T10:38:00.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:00 smithi070 ceph-mon[107385]: osdmap e1250: 8 total, 8 up, 8 in 2024-09-21T10:38:00.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:00 smithi070 ceph-mon[112139]: pgmap v2022: 105 pgs: 105 active+clean; 315 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.5 KiB/s rd, 515 KiB/s wr, 13 op/s 2024-09-21T10:38:00.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:00 smithi070 ceph-mon[112139]: osdmap e1250: 8 total, 8 up, 8 in 2024-09-21T10:38:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:00 smithi160 ceph-mon[101879]: pgmap v2022: 105 pgs: 105 active+clean; 315 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.5 KiB/s rd, 515 KiB/s wr, 13 op/s 2024-09-21T10:38:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:00 smithi160 ceph-mon[101879]: osdmap e1250: 8 total, 8 up, 8 in 2024-09-21T10:38:01.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:01 smithi160 ceph-mon[101879]: osdmap e1251: 8 total, 8 up, 8 in 2024-09-21T10:38:01.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:01 smithi070 ceph-mon[107385]: osdmap e1251: 8 total, 8 up, 8 in 2024-09-21T10:38:01.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:01 smithi070 ceph-mon[112139]: osdmap e1251: 8 total, 8 up, 8 in 2024-09-21T10:38:02.415 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/1.DiffIterateDiscard (3023 ms) 2024-09-21T10:38:02.415 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/1.DiffIterateTruncate 2024-09-21T10:38:02.415 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:38:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:02 smithi160 ceph-mon[101879]: pgmap v2025: 105 pgs: 105 active+clean; 323 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 1.0 MiB/s wr, 34 op/s 2024-09-21T10:38:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:02 smithi160 ceph-mon[101879]: osdmap e1252: 8 total, 8 up, 8 in 2024-09-21T10:38:02.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:02 smithi070 ceph-mon[107385]: pgmap v2025: 105 pgs: 105 active+clean; 323 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 1.0 MiB/s wr, 34 op/s 2024-09-21T10:38:02.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:02 smithi070 ceph-mon[107385]: osdmap e1252: 8 total, 8 up, 8 in 2024-09-21T10:38:02.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:02 smithi070 ceph-mon[112139]: pgmap v2025: 105 pgs: 105 active+clean; 323 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 1.0 MiB/s wr, 34 op/s 2024-09-21T10:38:02.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:02 smithi070 ceph-mon[112139]: osdmap e1252: 8 total, 8 up, 8 in 2024-09-21T10:38:03.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:03 smithi070 ceph-mon[107385]: osdmap e1253: 8 total, 8 up, 8 in 2024-09-21T10:38:03.628 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:38:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:38:03] "GET /metrics HTTP/1.1" 200 36482 "" "Prometheus/2.51.0" 2024-09-21T10:38:03.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:03 smithi070 ceph-mon[112139]: osdmap e1253: 8 total, 8 up, 8 in 2024-09-21T10:38:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:03 smithi160 ceph-mon[101879]: osdmap e1253: 8 total, 8 up, 8 in 2024-09-21T10:38:04.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:04 smithi160 ceph-mon[101879]: pgmap v2028: 105 pgs: 105 active+clean; 339 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 48 KiB/s rd, 4.0 MiB/s wr, 74 op/s 2024-09-21T10:38:04.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:04 smithi160 ceph-mon[101879]: osdmap e1254: 8 total, 8 up, 8 in 2024-09-21T10:38:04.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:04 smithi070 ceph-mon[107385]: pgmap v2028: 105 pgs: 105 active+clean; 339 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 48 KiB/s rd, 4.0 MiB/s wr, 74 op/s 2024-09-21T10:38:04.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:04 smithi070 ceph-mon[107385]: osdmap e1254: 8 total, 8 up, 8 in 2024-09-21T10:38:04.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:04 smithi070 ceph-mon[112139]: pgmap v2028: 105 pgs: 105 active+clean; 339 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 48 KiB/s rd, 4.0 MiB/s wr, 74 op/s 2024-09-21T10:38:04.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:04 smithi070 ceph-mon[112139]: osdmap e1254: 8 total, 8 up, 8 in 2024-09-21T10:38:05.449 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/1.DiffIterateTruncate (3034 ms) 2024-09-21T10:38:05.449 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/1.DiffIterateWriteAndTruncate 2024-09-21T10:38:05.449 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:38:05.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:05 smithi070 ceph-mon[107385]: osdmap e1255: 8 total, 8 up, 8 in 2024-09-21T10:38:05.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:38:05.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:05 smithi070 ceph-mon[112139]: osdmap e1255: 8 total, 8 up, 8 in 2024-09-21T10:38:05.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:38:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:05 smithi160 ceph-mon[101879]: osdmap e1255: 8 total, 8 up, 8 in 2024-09-21T10:38:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:38:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:06 smithi160 ceph-mon[101879]: pgmap v2031: 105 pgs: 105 active+clean; 340 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 56 KiB/s rd, 3.3 MiB/s wr, 89 op/s 2024-09-21T10:38:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:06 smithi160 ceph-mon[101879]: osdmap e1256: 8 total, 8 up, 8 in 2024-09-21T10:38:06.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:06 smithi070 ceph-mon[107385]: pgmap v2031: 105 pgs: 105 active+clean; 340 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 56 KiB/s rd, 3.3 MiB/s wr, 89 op/s 2024-09-21T10:38:06.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:06 smithi070 ceph-mon[107385]: osdmap e1256: 8 total, 8 up, 8 in 2024-09-21T10:38:06.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:06 smithi070 ceph-mon[112139]: pgmap v2031: 105 pgs: 105 active+clean; 340 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 56 KiB/s rd, 3.3 MiB/s wr, 89 op/s 2024-09-21T10:38:06.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:06 smithi070 ceph-mon[112139]: osdmap e1256: 8 total, 8 up, 8 in 2024-09-21T10:38:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:07 smithi160 ceph-mon[101879]: osdmap e1257: 8 total, 8 up, 8 in 2024-09-21T10:38:07.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:07 smithi070 ceph-mon[107385]: osdmap e1257: 8 total, 8 up, 8 in 2024-09-21T10:38:07.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:07 smithi070 ceph-mon[112139]: osdmap e1257: 8 total, 8 up, 8 in 2024-09-21T10:38:08.464 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/1.DiffIterateWriteAndTruncate (3015 ms) 2024-09-21T10:38:08.465 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/1.DiffIterateStress 2024-09-21T10:38:08.465 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:38:08.537 INFO:tasks.workunit.client.0.smithi070.stdout: i=0 exists [2408023~5121055,24530340~236635,38122723~5793195,129773183~1454843,225561011~441686,280643393~7699141,299883854~7775347,353891773~5731003,369435542~7204265] wrote [2408023~5121055,24530340~236635,38122723~5793195,129773183~1454843,225561011~441686,280643393~7699141,299883854~7775347,353891773~5731003,369435542~7204265] 2024-09-21T10:38:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:08 smithi160 ceph-mon[101879]: pgmap v2034: 105 pgs: 105 active+clean; 340 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 71 KiB/s rd, 265 KiB/s wr, 114 op/s 2024-09-21T10:38:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:08 smithi160 ceph-mon[101879]: osdmap e1258: 8 total, 8 up, 8 in 2024-09-21T10:38:08.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:08 smithi070 ceph-mon[107385]: pgmap v2034: 105 pgs: 105 active+clean; 340 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 71 KiB/s rd, 265 KiB/s wr, 114 op/s 2024-09-21T10:38:08.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:08 smithi070 ceph-mon[107385]: osdmap e1258: 8 total, 8 up, 8 in 2024-09-21T10:38:08.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:08 smithi070 ceph-mon[112139]: pgmap v2034: 105 pgs: 105 active+clean; 340 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 71 KiB/s rd, 265 KiB/s wr, 114 op/s 2024-09-21T10:38:08.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:08 smithi070 ceph-mon[112139]: osdmap e1258: 8 total, 8 up, 8 in 2024-09-21T10:38:09.511 INFO:tasks.workunit.client.0.smithi070.stdout: i=1 exists [2408023~5121055,24530340~236635,28001474~8185098,38122723~8678310,82563757~1962773,129773183~7839537,147581806~1593566,225561011~441686,280643393~7699141,299883854~7775347,321941080~4910884,353891773~5731003,369435542~7204265,395687556~5801263] wrote [28001474~8185098,38292456~8508577,82563757~1962773,130543966~7068754,147581806~1593566,282175451~3842134,321941080~4910884,395687556~5801263] 2024-09-21T10:38:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:09 smithi160 ceph-mon[101879]: osdmap e1259: 8 total, 8 up, 8 in 2024-09-21T10:38:09.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:09 smithi070 ceph-mon[107385]: osdmap e1259: 8 total, 8 up, 8 in 2024-09-21T10:38:09.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:09 smithi070 ceph-mon[112139]: osdmap e1259: 8 total, 8 up, 8 in 2024-09-21T10:38:10.475 INFO:tasks.workunit.client.0.smithi070.stdout: i=2 exists [2408023~5121055,15303978~7699002,24530340~236635,28001474~8185098,38122723~8678310,73230681~2245816,82563757~1962773,91211084~1056857,129773183~7839537,147581806~1593566,178125002~3470218,181613708~87247,225561011~441686,270768041~2406894,280643393~7699141,294570124~13089077,321941080~4910884,353891773~5731003,369435542~7204265,377066254~1330633,391700263~9788556] wrote [15303978~7699002,33329005~2734858,73230681~2245816,91211084~1056857,178125002~3470218,181613708~87247,270768041~2406894,294570124~6138077,377066254~1330633,391700263~4446807] 2024-09-21T10:38:10.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:10 smithi160 ceph-mon[101879]: pgmap v2037: 105 pgs: 105 active+clean; 340 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 39 KiB/s rd, 5.2 KiB/s wr, 64 op/s 2024-09-21T10:38:10.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:10 smithi160 ceph-mon[101879]: osdmap e1260: 8 total, 8 up, 8 in 2024-09-21T10:38:10.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:10 smithi070 ceph-mon[107385]: pgmap v2037: 105 pgs: 105 active+clean; 340 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 39 KiB/s rd, 5.2 KiB/s wr, 64 op/s 2024-09-21T10:38:10.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:10 smithi070 ceph-mon[107385]: osdmap e1260: 8 total, 8 up, 8 in 2024-09-21T10:38:10.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:10 smithi070 ceph-mon[112139]: pgmap v2037: 105 pgs: 105 active+clean; 340 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 39 KiB/s rd, 5.2 KiB/s wr, 64 op/s 2024-09-21T10:38:10.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:10 smithi070 ceph-mon[112139]: osdmap e1260: 8 total, 8 up, 8 in 2024-09-21T10:38:11.529 INFO:tasks.workunit.client.0.smithi070.stdout: i=3 exists [2408023~5121055,15303978~7699002,24530340~236635,28001474~8185098,38122723~9123568,55550535~7865303,63573204~5859224,73230681~2245816,82563757~1962773,89334432~6338369,129773183~7839537,147581806~1593566,178125002~3470218,181613708~87247,205009188~989636,212083757~5151815,223933731~2068966,241799571~4960924,270768041~2406894,280643393~7699141,288920979~5495918,294570124~13089077,321941080~4910884,353891773~5731003,369435542~7204265,377066254~1330633,391700263~9788556] wrote [40788377~6457914,55550535~7865303,63573204~5859224,89334432~6338369,205009188~989636,212083757~5151815,223933731~1854493,241799571~4960924,288920979~5495918] 2024-09-21T10:38:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:11 smithi160 ceph-mon[101879]: osdmap e1261: 8 total, 8 up, 8 in 2024-09-21T10:38:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:11 smithi160 ceph-mon[101879]: pgmap v2040: 105 pgs: 105 active+clean; 363 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 67 KiB/s rd, 4.1 MiB/s wr, 99 op/s 2024-09-21T10:38:11.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:11 smithi070 ceph-mon[112139]: osdmap e1261: 8 total, 8 up, 8 in 2024-09-21T10:38:11.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:11 smithi070 ceph-mon[112139]: pgmap v2040: 105 pgs: 105 active+clean; 363 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 67 KiB/s rd, 4.1 MiB/s wr, 99 op/s 2024-09-21T10:38:11.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:11 smithi070 ceph-mon[107385]: osdmap e1261: 8 total, 8 up, 8 in 2024-09-21T10:38:11.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:11 smithi070 ceph-mon[107385]: pgmap v2040: 105 pgs: 105 active+clean; 363 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 67 KiB/s rd, 4.1 MiB/s wr, 99 op/s 2024-09-21T10:38:12.523 INFO:tasks.workunit.client.0.smithi070.stdout: i=4 exists [2408023~5121055,15303978~7699002,23965623~2896641,28001474~19244817,55550535~7865303,63573204~5859224,73230681~2245816,82563757~1962773,89334432~6338369,118983611~7771636,129773183~7839537,147581806~1593566,178125002~3470218,181613708~87247,205009188~989636,212083757~5151815,219259767~812374,223933731~2068966,241799571~4960924,258501767~6500306,270768041~2406894,275819167~12523367,288920979~5495918,294570124~13089077,321941080~4910884,330719651~5717922,353891773~5731003,369435542~7204265,377066254~1330633,391700263~9788556,406074456~2512586] wrote [23965623~2896641,34546517~7867609,59222541~350187,118983611~7771636,219259767~812374,258501767~6500306,275819167~5122371,304058544~1228003,330719651~5717922,406074456~2512586] 2024-09-21T10:38:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:12 smithi160 ceph-mon[101879]: osdmap e1262: 8 total, 8 up, 8 in 2024-09-21T10:38:12.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:12 smithi070 ceph-mon[107385]: osdmap e1262: 8 total, 8 up, 8 in 2024-09-21T10:38:12.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:12 smithi070 ceph-mon[112139]: osdmap e1262: 8 total, 8 up, 8 in 2024-09-21T10:38:13.450 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:38:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:38:13] "GET /metrics HTTP/1.1" 200 36475 "" "Prometheus/2.51.0" 2024-09-21T10:38:13.511 INFO:tasks.workunit.client.0.smithi070.stdout: i=5 exists [2408023~5121055,15303978~7699002,23965623~2896641,28001474~19244817,55550535~7865303,63573204~11903293,82563757~1962773,89334432~10613611,109035701~2342031,118983611~7771636,129773183~7839537,147581806~1593566,178125002~3470218,181613708~87247,205009188~989636,212083757~5151815,219259767~812374,223933731~2068966,241799571~4960924,258501767~6500306,270768041~2406894,275819167~12523367,288920979~5495918,294570124~13089077,316447783~2245348,321941080~4910884,330719651~5717922,353891773~5731003,364747124~391806,369435542~7204265,377066254~1330633,379123067~7170468,391700263~9788556,406074456~2512586] wrote [65754878~8334636,93149394~6798649,109035701~2342031,316447783~2245348,323787413~451816,364747124~391806,379123067~7170468,394921809~5534057] 2024-09-21T10:38:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:13 smithi160 ceph-mon[101879]: osdmap e1263: 8 total, 8 up, 8 in 2024-09-21T10:38:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:13 smithi160 ceph-mon[101879]: pgmap v2043: 105 pgs: 105 active+clean; 469 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 138 KiB/s rd, 25 MiB/s wr, 226 op/s 2024-09-21T10:38:13.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:13 smithi070 ceph-mon[107385]: osdmap e1263: 8 total, 8 up, 8 in 2024-09-21T10:38:13.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:13 smithi070 ceph-mon[107385]: pgmap v2043: 105 pgs: 105 active+clean; 469 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 138 KiB/s rd, 25 MiB/s wr, 226 op/s 2024-09-21T10:38:13.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:13 smithi070 ceph-mon[112139]: osdmap e1263: 8 total, 8 up, 8 in 2024-09-21T10:38:13.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:13 smithi070 ceph-mon[112139]: pgmap v2043: 105 pgs: 105 active+clean; 469 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 138 KiB/s rd, 25 MiB/s wr, 226 op/s 2024-09-21T10:38:14.543 INFO:tasks.workunit.client.0.smithi070.stdout: i=6 exists [2408023~11061772,15303978~31942313,55550535~7865303,63573204~11903293,76929863~1260403,82563757~1962773,89334432~10613611,109035701~2342031,117368573~9386674,129773183~7839537,147581806~1593566,178125002~3470218,181613708~87247,205009188~989636,212083757~5151815,219259767~812374,223933731~2068966,241799571~4960924,253337416~11664657,270768041~2406894,275234754~13107780,288920979~5495918,294570124~13089077,316447783~2245348,321941080~4910884,330719651~5717922,353891773~5731003,364747124~391806,369435542~8961345,379123067~7170468,391700263~9788556,406074456~2512586] wrote [5765543~7704252,21632180~6842620,65319176~542994,76929863~1260403,117368573~8165961,253337416~6988831,275234754~2440392,372039425~6312058,392252233~7413265] 2024-09-21T10:38:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:14 smithi160 ceph-mon[101879]: osdmap e1264: 8 total, 8 up, 8 in 2024-09-21T10:38:14.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:14 smithi070 ceph-mon[107385]: osdmap e1264: 8 total, 8 up, 8 in 2024-09-21T10:38:14.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:14 smithi070 ceph-mon[112139]: osdmap e1264: 8 total, 8 up, 8 in 2024-09-21T10:38:15.565 INFO:tasks.workunit.client.0.smithi070.stdout: i=7 exists [2408023~11061772,15303978~31942313,55550535~7865303,63573204~14617062,82563757~1962773,89334432~10613611,109035701~2342031,117368573~20244147,147581806~1593566,178125002~3470218,181613708~87247,205009188~989636,212083757~5151815,219259767~812374,223933731~2068966,241799571~4960924,253337416~11664657,270768041~2406894,275234754~13107780,288920979~18738222,313525224~5226548,321941080~7567317,330719651~5717922,339261431~780997,353891773~5731003,364747124~391806,369435542~8961345,379123067~7170468,391700263~9788556,406074456~6364921] wrote [72505901~4760038,122972465~9335473,256129850~1334982,291158704~6555106,313525224~5226548,326661958~2846439,339261431~780997,406661885~5777492] 2024-09-21T10:38:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:15 smithi160 ceph-mon[101879]: osdmap e1265: 8 total, 8 up, 8 in 2024-09-21T10:38:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:15 smithi160 ceph-mon[101879]: pgmap v2046: 105 pgs: 105 active+clean; 570 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 116 KiB/s rd, 40 MiB/s wr, 195 op/s 2024-09-21T10:38:15.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:15 smithi070 ceph-mon[112139]: osdmap e1265: 8 total, 8 up, 8 in 2024-09-21T10:38:15.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:15 smithi070 ceph-mon[112139]: pgmap v2046: 105 pgs: 105 active+clean; 570 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 116 KiB/s rd, 40 MiB/s wr, 195 op/s 2024-09-21T10:38:15.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:15 smithi070 ceph-mon[107385]: osdmap e1265: 8 total, 8 up, 8 in 2024-09-21T10:38:15.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:15 smithi070 ceph-mon[107385]: pgmap v2046: 105 pgs: 105 active+clean; 570 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 116 KiB/s rd, 40 MiB/s wr, 195 op/s 2024-09-21T10:38:16.536 INFO:tasks.workunit.client.0.smithi070.stdout: i=8 exists [2408023~11061772,15303978~31942313,50208073~595751,55550535~7865303,63573204~14617062,82563757~1962773,86169760~13778283,109035701~2342031,117368573~20244147,147581806~1593566,157422876~1619984,173726917~7868303,181613708~87247,193537460~6930103,205009188~989636,212083757~5151815,219259767~812374,223933731~2068966,241038308~8191010,253337416~11664657,270768041~2406894,275234754~13107780,288920979~18738222,307974740~5201175,313525224~5226548,321941080~7567317,330719651~5717922,339261431~780997,353891773~5731003,364747124~391806,369435542~8961345,379123067~7170468,391700263~9788556,406074456~6364921] wrote [11737379~1339650,50208073~595751,86169760~7558723,157422876~1619984,173726917~5027955,193537460~6930103,241038308~8191010,294712224~5494778,307974740~5201175,322182405~216528] 2024-09-21T10:38:16.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:16 smithi160 ceph-mon[101879]: osdmap e1266: 8 total, 8 up, 8 in 2024-09-21T10:38:16.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:16 smithi070 ceph-mon[112139]: osdmap e1266: 8 total, 8 up, 8 in 2024-09-21T10:38:16.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:16 smithi070 ceph-mon[107385]: osdmap e1266: 8 total, 8 up, 8 in 2024-09-21T10:38:17.494 INFO:tasks.workunit.client.0.smithi070.stdout: i=9 exists [2408023~11061772,15303978~31942313,49457587~629165,50208073~595751,55550535~7865303,63573204~14617062,82563757~1962773,86169760~13778283,109035701~2342031,113678845~496722,117368573~20244147,147581806~1593566,157422876~1619984,173726917~7868303,181613708~87247,193537460~8760270,205009188~989636,212083757~5151815,219259767~812374,223933731~2068966,241038308~8191010,253337416~11664657,270768041~2406894,275234754~13107780,288920979~18738222,307974740~5201175,313525224~5226548,321941080~7567317,330719651~5717922,339261431~780997,353891773~5731003,364747124~391806,369435542~8961345,379123067~7170468,391700263~9788556,406074456~6364921] wrote [6455423~1257825,9392669~2733955,35765410~6452398,49457587~629165,95620007~1108737,113678845~496722,197177282~5120448,281125474~2459263,370064564~1426687,382549641~1394181] 2024-09-21T10:38:17.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:17 smithi160 ceph-mon[101879]: osdmap e1267: 8 total, 8 up, 8 in 2024-09-21T10:38:17.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:17 smithi160 ceph-mon[101879]: pgmap v2049: 105 pgs: 105 active+clean; 684 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 166 KiB/s rd, 44 MiB/s wr, 276 op/s 2024-09-21T10:38:17.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:17 smithi070 ceph-mon[112139]: osdmap e1267: 8 total, 8 up, 8 in 2024-09-21T10:38:17.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:17 smithi070 ceph-mon[112139]: pgmap v2049: 105 pgs: 105 active+clean; 684 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 166 KiB/s rd, 44 MiB/s wr, 276 op/s 2024-09-21T10:38:17.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:17 smithi070 ceph-mon[107385]: osdmap e1267: 8 total, 8 up, 8 in 2024-09-21T10:38:17.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:17 smithi070 ceph-mon[107385]: pgmap v2049: 105 pgs: 105 active+clean; 684 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 166 KiB/s rd, 44 MiB/s wr, 276 op/s 2024-09-21T10:38:18.584 INFO:tasks.workunit.client.0.smithi070.stdout: i=10 exists [2408023~11061772,15303978~31942313,49457587~629165,50208073~595751,55126164~8289674,63573204~14617062,82563757~1962773,86169760~13778283,109035701~2342031,111911706~25701014,147581806~1593566,157422876~1619984,173726917~7868303,181613708~87247,193537460~8760270,205009188~989636,212083757~16702696,241038308~8191010,253337416~11664657,270768041~2406894,275234754~13107780,288920979~18738222,307974740~5201175,313525224~5226548,321941080~7567317,329622376~6815197,339261431~780997,353891773~5731003,364747124~391806,369435542~8961345,379123067~7170468,391700263~9788556,406074456~6364921] wrote [55126164~4403805,92248156~3457139,111911706~7441477,119518112~4493542,215672615~13113838,258290470~3892644,295589057~6219086,329622376~4119242] 2024-09-21T10:38:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:18 smithi160 ceph-mon[101879]: osdmap e1268: 8 total, 8 up, 8 in 2024-09-21T10:38:18.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:18 smithi070 ceph-mon[107385]: osdmap e1268: 8 total, 8 up, 8 in 2024-09-21T10:38:18.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:18 smithi070 ceph-mon[112139]: osdmap e1268: 8 total, 8 up, 8 in 2024-09-21T10:38:19.583 INFO:tasks.workunit.client.0.smithi070.stdout: i=11 exists [2408023~44838268,49457587~629165,50208073~13207765,63573204~14617062,82563757~1962773,86169760~13778283,109035701~2342031,111911706~25701014,147581806~1593566,157422876~1619984,173726917~7868303,181613708~87247,193537460~8760270,205009188~989636,212083757~16702696,228847847~2621665,241038308~8191010,253337416~11664657,270768041~2406894,275234754~13107780,288920979~18738222,307974740~5201175,313525224~5226548,321941080~7567317,329622376~6815197,339261431~780997,342053810~6480216,353891773~5731003,364747124~391806,369435542~8961345,379123067~7170468,391700263~9788556,406074456~6364921] wrote [9063767~8095106,27234755~4425879,40388856~6411324,50512463~6987640,177595770~2663564,228847847~2621665,342053810~6480216,371692059~3060653] 2024-09-21T10:38:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:19 smithi160 ceph-mon[101879]: osdmap e1269: 8 total, 8 up, 8 in 2024-09-21T10:38:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:19 smithi160 ceph-mon[101879]: pgmap v2052: 105 pgs: 105 active+clean; 684 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 121 KiB/s rd, 24 MiB/s wr, 206 op/s 2024-09-21T10:38:19.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:19 smithi070 ceph-mon[107385]: osdmap e1269: 8 total, 8 up, 8 in 2024-09-21T10:38:19.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:19 smithi070 ceph-mon[107385]: pgmap v2052: 105 pgs: 105 active+clean; 684 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 121 KiB/s rd, 24 MiB/s wr, 206 op/s 2024-09-21T10:38:19.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:19 smithi070 ceph-mon[112139]: osdmap e1269: 8 total, 8 up, 8 in 2024-09-21T10:38:19.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:19 smithi070 ceph-mon[112139]: pgmap v2052: 105 pgs: 105 active+clean; 684 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 121 KiB/s rd, 24 MiB/s wr, 206 op/s 2024-09-21T10:38:20.564 INFO:tasks.workunit.client.0.smithi070.stdout: i=12 exists [2408023~44838268,49457587~629165,50208073~13207765,63573204~14617062,81519016~3007514,86169760~13778283,109035701~2342031,111911706~25701014,147581806~1593566,157422876~1619984,171843894~9751326,181613708~87247,193537460~8760270,205009188~989636,212083757~23808121,241038308~10684737,253337416~11664657,270768041~2406894,275234754~13107780,288920979~18738222,307974740~5201175,313525224~5226548,321941080~7567317,329622376~6815197,339261431~780997,342053810~6480216,353891773~5731003,364747124~391806,369435542~8961345,379123067~7170468,389394986~12093833,406074456~6364921] wrote [17747577~1172108,27890857~7699535,61012696~834455,81519016~1685566,171843894~4605002,177176603~470790,228549015~7342863,247272371~4450674,385085859~410226,389394986~3091969] 2024-09-21T10:38:20.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:20 smithi160 ceph-mon[101879]: osdmap e1270: 8 total, 8 up, 8 in 2024-09-21T10:38:20.743 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:38:20.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:20 smithi070 ceph-mon[107385]: osdmap e1270: 8 total, 8 up, 8 in 2024-09-21T10:38:20.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:38:20.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:20 smithi070 ceph-mon[112139]: osdmap e1270: 8 total, 8 up, 8 in 2024-09-21T10:38:20.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:38:21.565 INFO:tasks.workunit.client.0.smithi070.stdout: i=13 exists [2408023~44838268,48531221~14884617,63573204~14617062,81519016~3007514,86169760~13778283,109035701~2342031,111911706~25701014,147581806~1593566,154488324~4554536,171843894~9751326,181613708~87247,193537460~8760270,205009188~989636,212083757~23808121,241038308~10684737,253337416~11664657,270768041~2406894,275234754~13107780,288920979~18738222,307974740~5201175,313525224~5226548,321941080~7567317,329622376~6815197,339261431~780997,342053810~6480216,353891773~5731003,364747124~391806,369435542~8961345,379123067~7170468,386876242~14612577,406074456~6364921] wrote [48531221~13894348,64055006~925904,154488324~3048841,292633402~5650381,298617887~3383739,371099037~1312026,386876242~5288953,393559182~2209768] 2024-09-21T10:38:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:21 smithi160 ceph-mon[101879]: osdmap e1271: 8 total, 8 up, 8 in 2024-09-21T10:38:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:21 smithi160 ceph-mon[101879]: pgmap v2055: 105 pgs: 105 active+clean; 791 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 76 KiB/s rd, 24 MiB/s wr, 113 op/s 2024-09-21T10:38:21.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:21 smithi070 ceph-mon[107385]: osdmap e1271: 8 total, 8 up, 8 in 2024-09-21T10:38:21.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:21 smithi070 ceph-mon[107385]: pgmap v2055: 105 pgs: 105 active+clean; 791 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 76 KiB/s rd, 24 MiB/s wr, 113 op/s 2024-09-21T10:38:21.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:21 smithi070 ceph-mon[112139]: osdmap e1271: 8 total, 8 up, 8 in 2024-09-21T10:38:21.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:21 smithi070 ceph-mon[112139]: pgmap v2055: 105 pgs: 105 active+clean; 791 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 76 KiB/s rd, 24 MiB/s wr, 113 op/s 2024-09-21T10:38:22.581 INFO:tasks.workunit.client.0.smithi070.stdout: i=14 exists [2408023~61007815,63573204~14617062,81519016~3007514,86169760~13778283,105282580~1184967,109035701~2342031,111911706~25701014,145495214~3680158,154488324~4554536,171843894~9751326,181613708~87247,193537460~8760270,205009188~989636,212083757~23808121,241038308~10684737,253337416~11664657,270768041~2406894,275234754~32424447,307974740~5201175,313525224~5226548,321941080~15784903,339261431~780997,342053810~6480216,353891773~5731003,364747124~391806,369435542~8961345,379123067~7170468,386876242~14612577,406074456~6364921] wrote [45270588~5664760,105282580~1184967,145495214~2751541,214783758~3015375,230746335~5014009,288040922~2270485,327564964~6754889,334709706~3016277,345191192~2448500,370247764~7123969] 2024-09-21T10:38:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:22 smithi160 ceph-mon[101879]: osdmap e1272: 8 total, 8 up, 8 in 2024-09-21T10:38:22.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:22 smithi070 ceph-mon[107385]: osdmap e1272: 8 total, 8 up, 8 in 2024-09-21T10:38:22.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:22 smithi070 ceph-mon[112139]: osdmap e1272: 8 total, 8 up, 8 in 2024-09-21T10:38:23.510 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:38:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:38:23] "GET /metrics HTTP/1.1" 200 36481 "" "Prometheus/2.51.0" 2024-09-21T10:38:23.560 INFO:tasks.workunit.client.0.smithi070.stdout: i=15 exists [2408023~61007815,63573204~17215500,81519016~3007514,86169760~13778283,105282580~1184967,109035701~2342031,111911706~25701014,145495214~3680158,154488324~4554536,171843894~9751326,181613708~87247,186573944~5173098,193537460~8760270,205009188~989636,212083757~23808121,241038308~10684737,253337416~11664657,270768041~2406894,275234754~32424447,307974740~5201175,313525224~5226548,321941080~15784903,339261431~780997,342053810~6480216,353891773~5731003,363284173~5041974,369435542~8961345,379123067~7170468,386876242~14612577,406074456~6364921] wrote [29916197~4694861,76950534~3838170,112170798~1236519,186573944~5173098,228054223~5006552,243100104~4054801,254844796~880598,310109221~794550,363284173~5041974,369633077~399172] 2024-09-21T10:38:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:23 smithi160 ceph-mon[101879]: osdmap e1273: 8 total, 8 up, 8 in 2024-09-21T10:38:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:23 smithi160 ceph-mon[101879]: pgmap v2058: 105 pgs: 105 active+clean; 899 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 190 KiB/s rd, 52 MiB/s wr, 310 op/s 2024-09-21T10:38:23.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:23 smithi070 ceph-mon[107385]: osdmap e1273: 8 total, 8 up, 8 in 2024-09-21T10:38:23.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:23 smithi070 ceph-mon[107385]: pgmap v2058: 105 pgs: 105 active+clean; 899 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 190 KiB/s rd, 52 MiB/s wr, 310 op/s 2024-09-21T10:38:23.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:23 smithi070 ceph-mon[112139]: osdmap e1273: 8 total, 8 up, 8 in 2024-09-21T10:38:23.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:23 smithi070 ceph-mon[112139]: pgmap v2058: 105 pgs: 105 active+clean; 899 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 190 KiB/s rd, 52 MiB/s wr, 310 op/s 2024-09-21T10:38:24.580 INFO:tasks.workunit.client.0.smithi070.stdout: i=16 exists [2408023~61007815,63573204~20953326,86169760~13778283,105282580~1184967,109035701~2342031,111911706~25701014,145495214~3680158,151866151~7176709,170295837~491535,171843894~9751326,181613708~87247,186573944~5173098,193537460~8760270,205009188~989636,212083757~23808121,241038308~10684737,253337416~11664657,270768041~2406894,275234754~43517018,321941080~15784903,339261431~780997,342053810~6480216,353891773~5731003,362594999~5731148,369435542~8961345,379123067~7170468,386876242~29173969] wrote [53369918~6476674,74778598~7679104,119664645~309039,151866151~4659172,170295837~491535,291790878~936030,306125800~8152045,362594999~1089715,400542689~5874449,410288064~5762147] 2024-09-21T10:38:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:24 smithi160 ceph-mon[101879]: osdmap e1274: 8 total, 8 up, 8 in 2024-09-21T10:38:24.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:24 smithi070 ceph-mon[107385]: osdmap e1274: 8 total, 8 up, 8 in 2024-09-21T10:38:24.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:24 smithi070 ceph-mon[112139]: osdmap e1274: 8 total, 8 up, 8 in 2024-09-21T10:38:25.624 INFO:tasks.workunit.client.0.smithi070.stdout: i=17 exists [2408023~61007815,63573204~20953326,86169760~13778283,105282580~1184967,109035701~2342031,111911706~25701014,145495214~13547646,170295837~491535,171843894~9751326,181613708~87247,186573944~5173098,193537460~8760270,205009188~989636,212083757~23808121,236906089~763706,238029844~13693201,253337416~11664657,265426699~7748236,275234754~43517018,321941080~15784903,339261431~780997,342053810~6480216,353891773~5731003,362594999~5731148,369435542~8961345,379123067~7170468,386876242~29173969] wrote [69100558~8100141,148758918~4857947,236906089~763706,238029844~7291680,260774167~689644,265426699~7581768,304623697~6997041,329463865~7526305,364936029~1256630,370637611~6260801] 2024-09-21T10:38:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:25 smithi160 ceph-mon[101879]: osdmap e1275: 8 total, 8 up, 8 in 2024-09-21T10:38:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:25 smithi160 ceph-mon[101879]: pgmap v2061: 105 pgs: 105 active+clean; 991 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 223 KiB/s rd, 48 MiB/s wr, 353 op/s 2024-09-21T10:38:25.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:25 smithi070 ceph-mon[107385]: osdmap e1275: 8 total, 8 up, 8 in 2024-09-21T10:38:25.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:25 smithi070 ceph-mon[107385]: pgmap v2061: 105 pgs: 105 active+clean; 991 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 223 KiB/s rd, 48 MiB/s wr, 353 op/s 2024-09-21T10:38:25.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:25 smithi070 ceph-mon[112139]: osdmap e1275: 8 total, 8 up, 8 in 2024-09-21T10:38:25.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:25 smithi070 ceph-mon[112139]: pgmap v2061: 105 pgs: 105 active+clean; 991 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 223 KiB/s rd, 48 MiB/s wr, 353 op/s 2024-09-21T10:38:26.595 INFO:tasks.workunit.client.0.smithi070.stdout: i=18 exists [2408023~61007815,63573204~20953326,86169760~13778283,105282580~1184967,109035701~2342031,111911706~25701014,137625655~1377313,145495214~13547646,170295837~491535,171843894~9751326,181613708~87247,186573944~5173098,193537460~8760270,205009188~989636,212083757~23808121,236906089~763706,238029844~13693201,253337416~11664657,265426699~7748236,275234754~43517018,321941080~15784903,339261431~780997,342053810~6480216,351555187~9885945,362594999~5731148,369435542~8961345,379123067~7170468,386876242~29173969] wrote [67930240~293651,74739449~4853195,137625655~1377313,241138352~508655,283438955~5252246,292334310~269421,351555187~9885945,398132049~4011233] 2024-09-21T10:38:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:26 smithi160 ceph-mon[101879]: osdmap e1276: 8 total, 8 up, 8 in 2024-09-21T10:38:26.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:26 smithi070 ceph-mon[107385]: osdmap e1276: 8 total, 8 up, 8 in 2024-09-21T10:38:26.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:26 smithi070 ceph-mon[112139]: osdmap e1276: 8 total, 8 up, 8 in 2024-09-21T10:38:27.598 INFO:tasks.workunit.client.0.smithi070.stdout: i=19 exists [2408023~61007815,63573204~20953326,86169760~13778283,99951399~172627,105282580~1184967,109035701~2342031,111911706~25701014,137625655~1377313,145495214~13547646,163932516~17662704,181613708~87247,186573944~5173098,192034314~92573,193537460~8760270,205009188~989636,212083757~23808121,236906089~763706,238029844~26972229,265426699~7748236,275234754~43517018,321941080~15784903,339261431~780997,342053810~6480216,351555187~9885945,362594999~5731148,369435542~8961345,379123067~7170468,386876242~29173969] wrote [99951399~172627,118082737~5447391,163932516~8017314,192034314~92573,246559573~7620731,307583422~5550271,322560782~1653937,390540424~2490005,399661859~5373194] 2024-09-21T10:38:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:27 smithi160 ceph-mon[101879]: osdmap e1277: 8 total, 8 up, 8 in 2024-09-21T10:38:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:27 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:38:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:27 smithi160 ceph-mon[101879]: pgmap v2064: 105 pgs: 105 active+clean; 1.1 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 227 KiB/s rd, 44 MiB/s wr, 356 op/s 2024-09-21T10:38:27.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:27 smithi070 ceph-mon[107385]: osdmap e1277: 8 total, 8 up, 8 in 2024-09-21T10:38:27.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:27 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:38:27.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:27 smithi070 ceph-mon[107385]: pgmap v2064: 105 pgs: 105 active+clean; 1.1 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 227 KiB/s rd, 44 MiB/s wr, 356 op/s 2024-09-21T10:38:27.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:27 smithi070 ceph-mon[112139]: osdmap e1277: 8 total, 8 up, 8 in 2024-09-21T10:38:27.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:27 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:38:27.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:27 smithi070 ceph-mon[112139]: pgmap v2064: 105 pgs: 105 active+clean; 1.1 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 227 KiB/s rd, 44 MiB/s wr, 356 op/s 2024-09-21T10:38:28.551 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 1 diff [25165824~8388608,37748736~12582912,79691776~4194304,130023424~8388608,146800640~4194304,281018368~4194304,318767104~8388608,394264576~8388608] 2024-09-21T10:38:28.551 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [25165824~8388608,37748736~12582912,79691776~4194304,130023424~8388608,146800640~4194304,281018368~4194304,318767104~8388608,394264576~8388608] 2024-09-21T10:38:28.559 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [25165824~25165824,79691776~8388608,130023424~8388608,146800640~4194304,281018368~8388608,318767104~8388608,394264576~8388608] 2024-09-21T10:38:28.560 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 2 diff [12582912~8388608,25165824~8388608,37748736~12582912,71303168~4194304,79691776~4194304,88080384~4194304,130023424~8388608,146800640~4194304,176160768~8388608,268435456~4194304,281018368~4194304,293601280~8388608,318767104~8388608,373293056~4194304,390070272~12582912] 2024-09-21T10:38:28.560 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [12582912~8388608,25165824~8388608,37748736~12582912,71303168~4194304,79691776~4194304,88080384~4194304,130023424~8388608,146800640~4194304,176160768~8388608,268435456~4194304,281018368~4194304,293601280~8388608,318767104~8388608,373293056~4194304,390070272~12582912] 2024-09-21T10:38:28.561 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [12582912~37748736,71303168~4194304,79691776~12582912,130023424~8388608,146800640~4194304,176160768~8388608,268435456~8388608,281018368~8388608,293601280~8388608,318767104~8388608,373293056~8388608,390070272~12582912] 2024-09-21T10:38:28.561 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 3 diff [12582912~8388608,25165824~8388608,37748736~12582912,54525952~20971520,79691776~4194304,88080384~8388608,130023424~8388608,146800640~4194304,176160768~8388608,201326592~4194304,209715200~8388608,222298112~4194304,239075328~8388608,268435456~4194304,281018368~20971520,318767104~8388608,373293056~4194304,390070272~12582912] 2024-09-21T10:38:28.561 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [12582912~8388608,25165824~8388608,37748736~12582912,54525952~20971520,79691776~4194304,88080384~8388608,130023424~8388608,146800640~4194304,176160768~8388608,201326592~4194304,209715200~8388608,222298112~4194304,239075328~8388608,268435456~4194304,281018368~20971520,318767104~8388608,373293056~4194304,390070272~12582912] 2024-09-21T10:38:28.565 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [12582912~37748736,54525952~20971520,79691776~16777216,130023424~8388608,146800640~4194304,176160768~8388608,201326592~16777216,222298112~4194304,239075328~8388608,268435456~8388608,281018368~20971520,318767104~8388608,373293056~8388608,390070272~12582912] 2024-09-21T10:38:28.565 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 4 diff [12582912~33554432,54525952~20971520,79691776~4194304,88080384~8388608,117440512~8388608,130023424~8388608,146800640~4194304,176160768~8388608,201326592~4194304,209715200~16777216,239075328~8388608,255852544~8388608,268435456~37748736,318767104~16777216,373293056~4194304,390070272~16777216] 2024-09-21T10:38:28.565 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [12582912~33554432,54525952~20971520,79691776~4194304,88080384~8388608,117440512~8388608,130023424~8388608,146800640~4194304,176160768~8388608,201326592~4194304,209715200~16777216,239075328~8388608,255852544~8388608,268435456~37748736,318767104~16777216,373293056~4194304,390070272~16777216] 2024-09-21T10:38:28.567 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [12582912~37748736,54525952~20971520,79691776~16777216,117440512~20971520,146800640~4194304,176160768~8388608,201326592~25165824,239075328~8388608,255852544~50331648,318767104~20971520,373293056~8388608,390070272~20971520] 2024-09-21T10:38:28.567 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 5 diff [12582912~33554432,54525952~20971520,79691776~4194304,88080384~12582912,104857600~4194304,117440512~8388608,130023424~8388608,146800640~4194304,176160768~8388608,201326592~4194304,209715200~16777216,239075328~8388608,255852544~8388608,268435456~37748736,314572800~20971520,360710144~4194304,373293056~12582912,390070272~16777216] 2024-09-21T10:38:28.567 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [12582912~33554432,54525952~20971520,79691776~4194304,88080384~12582912,104857600~4194304,117440512~8388608,130023424~8388608,146800640~4194304,176160768~8388608,201326592~4194304,209715200~16777216,239075328~8388608,255852544~8388608,268435456~37748736,314572800~20971520,360710144~4194304,373293056~12582912,390070272~16777216] 2024-09-21T10:38:28.570 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [12582912~37748736,54525952~20971520,79691776~20971520,104857600~8388608,117440512~20971520,146800640~4194304,176160768~8388608,201326592~25165824,239075328~8388608,255852544~50331648,314572800~25165824,360710144~8388608,373293056~37748736] 2024-09-21T10:38:28.570 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 6 diff [4194304~41943040,54525952~29360128,88080384~12582912,104857600~4194304,113246208~12582912,130023424~8388608,146800640~4194304,176160768~8388608,201326592~4194304,209715200~16777216,239075328~8388608,251658240~12582912,268435456~37748736,314572800~20971520,360710144~4194304,369098752~16777216,390070272~16777216] 2024-09-21T10:38:28.570 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~41943040,54525952~29360128,88080384~12582912,104857600~4194304,113246208~12582912,130023424~8388608,146800640~4194304,176160768~8388608,201326592~4194304,209715200~16777216,239075328~8388608,251658240~12582912,268435456~37748736,314572800~20971520,360710144~4194304,369098752~16777216,390070272~16777216] 2024-09-21T10:38:28.573 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~46137344,54525952~46137344,104857600~33554432,146800640~4194304,176160768~8388608,201326592~25165824,239075328~8388608,251658240~54525952,314572800~25165824,360710144~50331648] 2024-09-21T10:38:28.573 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 7 diff [4194304~41943040,54525952~29360128,88080384~12582912,104857600~4194304,113246208~20971520,146800640~4194304,176160768~8388608,201326592~4194304,209715200~16777216,239075328~8388608,251658240~12582912,268435456~29360128,301989888~4194304,310378496~29360128,360710144~4194304,369098752~16777216,390070272~20971520] 2024-09-21T10:38:28.573 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~41943040,54525952~29360128,88080384~12582912,104857600~4194304,113246208~20971520,146800640~4194304,176160768~8388608,201326592~4194304,209715200~16777216,239075328~8388608,251658240~12582912,268435456~29360128,301989888~4194304,310378496~29360128,360710144~4194304,369098752~16777216,390070272~20971520] 2024-09-21T10:38:28.577 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~46137344,54525952~46137344,104857600~33554432,146800640~4194304,176160768~8388608,201326592~25165824,239075328~8388608,251658240~54525952,310378496~33554432,360710144~54525952] 2024-09-21T10:38:28.577 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 8 diff [4194304~46137344,54525952~46137344,104857600~4194304,113246208~20971520,146800640~4194304,155189248~4194304,171966464~12582912,192937984~12582912,209715200~16777216,239075328~8388608,251658240~12582912,268435456~29360128,301989888~37748736,360710144~4194304,369098752~16777216,390070272~20971520] 2024-09-21T10:38:28.577 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~46137344,54525952~46137344,104857600~4194304,113246208~20971520,146800640~4194304,155189248~4194304,171966464~12582912,192937984~12582912,209715200~16777216,239075328~8388608,251658240~12582912,268435456~29360128,301989888~37748736,360710144~4194304,369098752~16777216,390070272~20971520] 2024-09-21T10:38:28.581 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~33554432,146800640~4194304,155189248~4194304,171966464~12582912,192937984~33554432,239075328~104857600,360710144~54525952] 2024-09-21T10:38:28.581 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 9 diff [4194304~46137344,54525952~46137344,104857600~4194304,113246208~20971520,146800640~4194304,155189248~4194304,171966464~12582912,192937984~12582912,209715200~16777216,239075328~8388608,251658240~12582912,268435456~29360128,301989888~37748736,360710144~4194304,369098752~16777216,390070272~20971520] 2024-09-21T10:38:28.581 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~46137344,54525952~46137344,104857600~4194304,113246208~20971520,146800640~4194304,155189248~4194304,171966464~12582912,192937984~12582912,209715200~16777216,239075328~8388608,251658240~12582912,268435456~29360128,301989888~37748736,360710144~4194304,369098752~16777216,390070272~20971520] 2024-09-21T10:38:28.585 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~33554432,146800640~4194304,155189248~4194304,171966464~12582912,192937984~33554432,239075328~104857600,360710144~54525952] 2024-09-21T10:38:28.585 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 10 diff [4194304~46137344,54525952~46137344,104857600~33554432,146800640~4194304,155189248~4194304,171966464~12582912,192937984~12582912,209715200~16777216,239075328~8388608,251658240~12582912,268435456~71303168,360710144~4194304,369098752~16777216,390070272~20971520] 2024-09-21T10:38:28.585 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~46137344,54525952~46137344,104857600~33554432,146800640~4194304,155189248~4194304,171966464~12582912,192937984~12582912,209715200~16777216,239075328~8388608,251658240~12582912,268435456~71303168,360710144~4194304,369098752~16777216,390070272~20971520] 2024-09-21T10:38:28.588 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~33554432,146800640~4194304,155189248~4194304,171966464~12582912,192937984~37748736,239075328~104857600,360710144~54525952] 2024-09-21T10:38:28.588 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 11 diff [4194304~96468992,104857600~33554432,146800640~4194304,155189248~4194304,171966464~12582912,192937984~12582912,209715200~20971520,239075328~8388608,251658240~12582912,268435456~79691776,360710144~4194304,369098752~16777216,390070272~20971520] 2024-09-21T10:38:28.588 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~33554432,146800640~4194304,155189248~4194304,171966464~12582912,192937984~12582912,209715200~20971520,239075328~8388608,251658240~12582912,268435456~79691776,360710144~4194304,369098752~16777216,390070272~20971520] 2024-09-21T10:38:28.594 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~33554432,146800640~4194304,155189248~4194304,171966464~12582912,192937984~41943040,239075328~113246208,360710144~54525952] 2024-09-21T10:38:28.594 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 12 diff [4194304~96468992,104857600~33554432,146800640~4194304,155189248~4194304,167772160~16777216,192937984~12582912,209715200~25165824,239075328~25165824,268435456~79691776,360710144~4194304,369098752~29360128,402653184~8388608] 2024-09-21T10:38:28.594 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~33554432,146800640~4194304,155189248~4194304,167772160~16777216,192937984~12582912,209715200~25165824,239075328~25165824,268435456~79691776,360710144~4194304,369098752~29360128,402653184~8388608] 2024-09-21T10:38:28.599 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~33554432,146800640~4194304,155189248~4194304,167772160~16777216,192937984~159383552,360710144~54525952] 2024-09-21T10:38:28.599 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 13 diff [4194304~96468992,104857600~33554432,146800640~12582912,167772160~16777216,192937984~12582912,209715200~25165824,239075328~25165824,268435456~79691776,360710144~4194304,369098752~41943040] 2024-09-21T10:38:28.599 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~33554432,146800640~12582912,167772160~16777216,192937984~12582912,209715200~25165824,239075328~25165824,268435456~79691776,360710144~4194304,369098752~41943040] 2024-09-21T10:38:28.604 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~33554432,146800640~12582912,167772160~16777216,192937984~159383552,360710144~54525952] 2024-09-21T10:38:28.604 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 14 diff [4194304~96468992,104857600~33554432,142606336~4194304,150994944~8388608,167772160~16777216,192937984~12582912,209715200~25165824,239075328~25165824,268435456~79691776,360710144~4194304,369098752~41943040] 2024-09-21T10:38:28.604 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~33554432,142606336~4194304,150994944~8388608,167772160~16777216,192937984~12582912,209715200~25165824,239075328~25165824,268435456~79691776,360710144~4194304,369098752~41943040] 2024-09-21T10:38:28.610 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~33554432,142606336~16777216,167772160~16777216,192937984~159383552,360710144~54525952] 2024-09-21T10:38:28.610 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 15 diff [4194304~96468992,104857600~33554432,142606336~4194304,150994944~8388608,167772160~37748736,209715200~25165824,239075328~25165824,268435456~79691776,360710144~50331648] 2024-09-21T10:38:28.610 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~33554432,142606336~4194304,150994944~8388608,167772160~37748736,209715200~25165824,239075328~25165824,268435456~79691776,360710144~50331648] 2024-09-21T10:38:28.616 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~33554432,142606336~16777216,167772160~184549376,360710144~54525952] 2024-09-21T10:38:28.616 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 16 diff [4194304~96468992,104857600~33554432,142606336~4194304,150994944~8388608,167772160~37748736,209715200~25165824,239075328~25165824,268435456~79691776,360710144~54525952] 2024-09-21T10:38:28.616 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~33554432,142606336~4194304,150994944~8388608,167772160~37748736,209715200~25165824,239075328~25165824,268435456~79691776,360710144~54525952] 2024-09-21T10:38:28.622 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~33554432,142606336~16777216,167772160~184549376,360710144~58720256] 2024-09-21T10:38:28.622 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 17 diff [4194304~96468992,104857600~33554432,142606336~16777216,167772160~37748736,209715200~138412032,360710144~54525952] 2024-09-21T10:38:28.622 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~33554432,142606336~16777216,167772160~37748736,209715200~138412032,360710144~54525952] 2024-09-21T10:38:28.629 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~33554432,142606336~16777216,167772160~184549376,360710144~58720256] 2024-09-21T10:38:28.629 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 18 diff [4194304~96468992,104857600~33554432,142606336~16777216,167772160~37748736,209715200~205520896] 2024-09-21T10:38:28.629 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~33554432,142606336~16777216,167772160~37748736,209715200~205520896] 2024-09-21T10:38:28.636 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,167772160~251658240] 2024-09-21T10:38:28.636 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to 19 diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:28.636 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:28.644 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:28.644 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 2 diff [12582912~8388608,29360128~4194304,71303168~4194304,88080384~4194304,176160768~8388608,268435456~4194304,293601280~8388608,373293056~4194304,390070272~8388608] 2024-09-21T10:38:28.645 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [12582912~8388608,29360128~4194304,71303168~4194304,88080384~4194304,176160768~8388608,268435456~4194304,293601280~8388608,373293056~4194304,390070272~8388608] 2024-09-21T10:38:28.646 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [12582912~12582912,29360128~8388608,71303168~4194304,88080384~4194304,176160768~8388608,268435456~8388608,293601280~8388608,373293056~8388608,390070272~8388608] 2024-09-21T10:38:28.646 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 3 diff [12582912~8388608,29360128~4194304,37748736~8388608,54525952~20971520,88080384~8388608,176160768~8388608,201326592~4194304,209715200~8388608,222298112~4194304,239075328~8388608,268435456~4194304,285212672~16777216,373293056~4194304,390070272~8388608] 2024-09-21T10:38:28.646 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [12582912~8388608,29360128~4194304,37748736~8388608,54525952~20971520,88080384~8388608,176160768~8388608,201326592~4194304,209715200~8388608,222298112~4194304,239075328~8388608,268435456~4194304,285212672~16777216,373293056~4194304,390070272~8388608] 2024-09-21T10:38:28.647 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [12582912~12582912,29360128~20971520,54525952~20971520,88080384~8388608,176160768~8388608,201326592~16777216,222298112~4194304,239075328~8388608,268435456~8388608,285212672~16777216,373293056~8388608,390070272~8388608] 2024-09-21T10:38:28.647 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 4 diff [12582912~12582912,29360128~16777216,54525952~20971520,88080384~8388608,117440512~8388608,176160768~8388608,201326592~4194304,209715200~16777216,239075328~8388608,255852544~8388608,268435456~12582912,285212672~20971520,327155712~8388608,373293056~4194304,390070272~8388608,402653184~4194304] 2024-09-21T10:38:28.647 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [12582912~12582912,29360128~16777216,54525952~20971520,88080384~8388608,117440512~8388608,176160768~8388608,201326592~4194304,209715200~16777216,239075328~8388608,255852544~8388608,268435456~12582912,285212672~20971520,327155712~8388608,373293056~4194304,390070272~8388608,402653184~4194304] 2024-09-21T10:38:28.649 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [12582912~37748736,54525952~20971520,88080384~8388608,117440512~12582912,176160768~8388608,201326592~25165824,239075328~8388608,255852544~25165824,285212672~20971520,327155712~12582912,373293056~8388608,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.649 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 5 diff [12582912~12582912,29360128~16777216,54525952~20971520,88080384~12582912,104857600~4194304,117440512~8388608,176160768~8388608,201326592~4194304,209715200~16777216,239075328~8388608,255852544~8388608,268435456~12582912,285212672~20971520,314572800~4194304,322961408~12582912,360710144~4194304,373293056~12582912,390070272~16777216] 2024-09-21T10:38:28.649 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [12582912~12582912,29360128~16777216,54525952~20971520,88080384~12582912,104857600~4194304,117440512~8388608,176160768~8388608,201326592~4194304,209715200~16777216,239075328~8388608,255852544~8388608,268435456~12582912,285212672~20971520,314572800~4194304,322961408~12582912,360710144~4194304,373293056~12582912,390070272~16777216] 2024-09-21T10:38:28.651 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [12582912~37748736,54525952~20971520,88080384~12582912,104857600~8388608,117440512~12582912,176160768~8388608,201326592~25165824,239075328~8388608,255852544~25165824,285212672~20971520,314572800~4194304,322961408~16777216,360710144~8388608,373293056~37748736] 2024-09-21T10:38:28.651 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 6 diff [4194304~41943040,54525952~25165824,88080384~12582912,104857600~4194304,113246208~12582912,176160768~8388608,201326592~4194304,209715200~16777216,239075328~8388608,251658240~12582912,268435456~12582912,285212672~20971520,314572800~4194304,322961408~12582912,360710144~4194304,369098752~16777216,390070272~16777216] 2024-09-21T10:38:28.651 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~41943040,54525952~25165824,88080384~12582912,104857600~4194304,113246208~12582912,176160768~8388608,201326592~4194304,209715200~16777216,239075328~8388608,251658240~12582912,268435456~12582912,285212672~20971520,314572800~4194304,322961408~12582912,360710144~4194304,369098752~16777216,390070272~16777216] 2024-09-21T10:38:28.653 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~46137344,54525952~25165824,88080384~12582912,104857600~25165824,176160768~8388608,201326592~25165824,239075328~8388608,251658240~29360128,285212672~20971520,314572800~4194304,322961408~16777216,360710144~50331648] 2024-09-21T10:38:28.653 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 7 diff [4194304~41943040,54525952~25165824,88080384~12582912,104857600~4194304,113246208~16777216,176160768~8388608,201326592~4194304,209715200~16777216,239075328~8388608,251658240~12582912,268435456~12582912,285212672~12582912,301989888~4194304,310378496~8388608,322961408~16777216,360710144~4194304,369098752~16777216,390070272~20971520] 2024-09-21T10:38:28.654 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~41943040,54525952~25165824,88080384~12582912,104857600~4194304,113246208~16777216,176160768~8388608,201326592~4194304,209715200~16777216,239075328~8388608,251658240~12582912,268435456~12582912,285212672~12582912,301989888~4194304,310378496~8388608,322961408~16777216,360710144~4194304,369098752~16777216,390070272~20971520] 2024-09-21T10:38:28.656 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~46137344,54525952~25165824,88080384~12582912,104857600~29360128,176160768~8388608,201326592~25165824,239075328~8388608,251658240~29360128,285212672~20971520,310378496~8388608,322961408~20971520,360710144~54525952] 2024-09-21T10:38:28.656 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 8 diff [4194304~46137344,54525952~25165824,83886080~16777216,104857600~4194304,113246208~16777216,155189248~4194304,171966464~12582912,192937984~12582912,209715200~16777216,239075328~8388608,251658240~12582912,268435456~12582912,285212672~12582912,301989888~37748736,360710144~4194304,369098752~16777216,390070272~20971520] 2024-09-21T10:38:28.656 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~46137344,54525952~25165824,83886080~16777216,104857600~4194304,113246208~16777216,155189248~4194304,171966464~12582912,192937984~12582912,209715200~16777216,239075328~8388608,251658240~12582912,268435456~12582912,285212672~12582912,301989888~37748736,360710144~4194304,369098752~16777216,390070272~20971520] 2024-09-21T10:38:28.659 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~75497472,83886080~16777216,104857600~29360128,155189248~4194304,171966464~12582912,192937984~33554432,239075328~41943040,285212672~58720256,360710144~54525952] 2024-09-21T10:38:28.659 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 9 diff [4194304~46137344,54525952~25165824,83886080~16777216,104857600~4194304,113246208~16777216,155189248~4194304,171966464~12582912,192937984~12582912,209715200~16777216,239075328~8388608,251658240~12582912,268435456~29360128,301989888~37748736,360710144~4194304,369098752~16777216,390070272~20971520] 2024-09-21T10:38:28.659 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~46137344,54525952~25165824,83886080~16777216,104857600~4194304,113246208~16777216,155189248~4194304,171966464~12582912,192937984~12582912,209715200~16777216,239075328~8388608,251658240~12582912,268435456~29360128,301989888~37748736,360710144~4194304,369098752~16777216,390070272~20971520] 2024-09-21T10:38:28.662 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~75497472,83886080~16777216,104857600~29360128,155189248~4194304,171966464~12582912,192937984~33554432,239075328~104857600,360710144~54525952] 2024-09-21T10:38:28.662 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 10 diff [4194304~46137344,54525952~25165824,83886080~16777216,104857600~25165824,155189248~4194304,171966464~12582912,192937984~12582912,209715200~16777216,239075328~8388608,251658240~12582912,268435456~71303168,360710144~4194304,369098752~16777216,390070272~20971520] 2024-09-21T10:38:28.662 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~46137344,54525952~25165824,83886080~16777216,104857600~25165824,155189248~4194304,171966464~12582912,192937984~12582912,209715200~16777216,239075328~8388608,251658240~12582912,268435456~71303168,360710144~4194304,369098752~16777216,390070272~20971520] 2024-09-21T10:38:28.665 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~75497472,83886080~16777216,104857600~29360128,155189248~4194304,171966464~12582912,192937984~37748736,239075328~104857600,360710144~54525952] 2024-09-21T10:38:28.666 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 11 diff [4194304~75497472,83886080~16777216,104857600~25165824,155189248~4194304,171966464~12582912,192937984~12582912,209715200~20971520,239075328~8388608,251658240~12582912,268435456~79691776,360710144~4194304,369098752~16777216,390070272~20971520] 2024-09-21T10:38:28.666 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~75497472,83886080~16777216,104857600~25165824,155189248~4194304,171966464~12582912,192937984~12582912,209715200~20971520,239075328~8388608,251658240~12582912,268435456~79691776,360710144~4194304,369098752~16777216,390070272~20971520] 2024-09-21T10:38:28.670 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~75497472,83886080~16777216,104857600~29360128,155189248~4194304,171966464~12582912,192937984~41943040,239075328~113246208,360710144~54525952] 2024-09-21T10:38:28.670 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 12 diff [4194304~96468992,104857600~25165824,155189248~4194304,167772160~16777216,192937984~12582912,209715200~25165824,239075328~25165824,268435456~79691776,360710144~4194304,369098752~29360128,402653184~8388608] 2024-09-21T10:38:28.670 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,155189248~4194304,167772160~16777216,192937984~12582912,209715200~25165824,239075328~25165824,268435456~79691776,360710144~4194304,369098752~29360128,402653184~8388608] 2024-09-21T10:38:28.674 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~29360128,155189248~4194304,167772160~16777216,192937984~159383552,360710144~54525952] 2024-09-21T10:38:28.674 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 13 diff [4194304~96468992,104857600~25165824,150994944~8388608,167772160~16777216,192937984~12582912,209715200~25165824,239075328~25165824,268435456~79691776,360710144~4194304,369098752~41943040] 2024-09-21T10:38:28.674 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,150994944~8388608,167772160~16777216,192937984~12582912,209715200~25165824,239075328~25165824,268435456~79691776,360710144~4194304,369098752~41943040] 2024-09-21T10:38:28.679 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~29360128,150994944~8388608,167772160~16777216,192937984~159383552,360710144~54525952] 2024-09-21T10:38:28.679 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 14 diff [4194304~96468992,104857600~25165824,142606336~4194304,150994944~8388608,167772160~16777216,192937984~12582912,209715200~25165824,239075328~25165824,268435456~79691776,360710144~4194304,369098752~41943040] 2024-09-21T10:38:28.679 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,142606336~4194304,150994944~8388608,167772160~16777216,192937984~12582912,209715200~25165824,239075328~25165824,268435456~79691776,360710144~4194304,369098752~41943040] 2024-09-21T10:38:28.684 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~29360128,142606336~16777216,167772160~16777216,192937984~159383552,360710144~54525952] 2024-09-21T10:38:28.684 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 15 diff [4194304~96468992,104857600~25165824,142606336~4194304,150994944~8388608,167772160~37748736,209715200~25165824,239075328~25165824,268435456~79691776,360710144~50331648] 2024-09-21T10:38:28.684 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,142606336~4194304,150994944~8388608,167772160~37748736,209715200~25165824,239075328~25165824,268435456~79691776,360710144~50331648] 2024-09-21T10:38:28.690 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~29360128,142606336~16777216,167772160~184549376,360710144~54525952] 2024-09-21T10:38:28.690 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 16 diff [4194304~96468992,104857600~25165824,142606336~4194304,150994944~8388608,167772160~37748736,209715200~25165824,239075328~25165824,268435456~79691776,360710144~54525952] 2024-09-21T10:38:28.690 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,142606336~4194304,150994944~8388608,167772160~37748736,209715200~25165824,239075328~25165824,268435456~79691776,360710144~54525952] 2024-09-21T10:38:28.696 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~29360128,142606336~16777216,167772160~184549376,360710144~58720256] 2024-09-21T10:38:28.696 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 17 diff [4194304~96468992,104857600~25165824,142606336~16777216,167772160~37748736,209715200~138412032,360710144~54525952] 2024-09-21T10:38:28.696 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,142606336~16777216,167772160~37748736,209715200~138412032,360710144~54525952] 2024-09-21T10:38:28.702 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~29360128,142606336~16777216,167772160~184549376,360710144~58720256] 2024-09-21T10:38:28.702 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 18 diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,167772160~37748736,209715200~205520896] 2024-09-21T10:38:28.702 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,167772160~37748736,209715200~205520896] 2024-09-21T10:38:28.709 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,167772160~251658240] 2024-09-21T10:38:28.709 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to 19 diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:28.709 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:28.716 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:28.716 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 3 diff [37748736~8388608,54525952~16777216,88080384~8388608,201326592~4194304,209715200~8388608,222298112~4194304,239075328~8388608,285212672~8388608] 2024-09-21T10:38:28.716 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [37748736~8388608,54525952~16777216,88080384~8388608,201326592~4194304,209715200~8388608,222298112~4194304,239075328~8388608,285212672~8388608] 2024-09-21T10:38:28.717 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [37748736~12582912,54525952~16777216,88080384~8388608,201326592~16777216,222298112~4194304,239075328~8388608,285212672~12582912] 2024-09-21T10:38:28.717 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 4 diff [20971520~4194304,33554432~16777216,54525952~16777216,88080384~8388608,117440512~8388608,201326592~4194304,209715200~16777216,239075328~8388608,255852544~8388608,272629760~8388608,285212672~8388608,301989888~4194304,327155712~8388608,402653184~4194304] 2024-09-21T10:38:28.717 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [20971520~4194304,33554432~16777216,54525952~16777216,88080384~8388608,117440512~8388608,201326592~4194304,209715200~16777216,239075328~8388608,255852544~8388608,272629760~8388608,285212672~8388608,301989888~4194304,327155712~8388608,402653184~4194304] 2024-09-21T10:38:28.718 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [20971520~8388608,33554432~16777216,54525952~16777216,88080384~8388608,117440512~12582912,201326592~25165824,239075328~8388608,255852544~12582912,272629760~8388608,285212672~12582912,301989888~4194304,327155712~12582912,402653184~8388608] 2024-09-21T10:38:28.719 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 5 diff [20971520~4194304,33554432~16777216,54525952~20971520,88080384~12582912,104857600~4194304,117440512~8388608,201326592~4194304,209715200~16777216,239075328~8388608,255852544~8388608,272629760~8388608,285212672~8388608,301989888~4194304,314572800~4194304,322961408~12582912,360710144~4194304,377487360~8388608,394264576~12582912] 2024-09-21T10:38:28.719 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [20971520~4194304,33554432~16777216,54525952~20971520,88080384~12582912,104857600~4194304,117440512~8388608,201326592~4194304,209715200~16777216,239075328~8388608,255852544~8388608,272629760~8388608,285212672~8388608,301989888~4194304,314572800~4194304,322961408~12582912,360710144~4194304,377487360~8388608,394264576~12582912] 2024-09-21T10:38:28.720 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [20971520~8388608,33554432~16777216,54525952~20971520,88080384~12582912,104857600~8388608,117440512~12582912,201326592~25165824,239075328~8388608,255852544~12582912,272629760~8388608,285212672~12582912,301989888~4194304,314572800~4194304,322961408~16777216,360710144~8388608,377487360~12582912,394264576~16777216] 2024-09-21T10:38:28.720 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 6 diff [4194304~8388608,20971520~8388608,33554432~16777216,54525952~25165824,88080384~12582912,104857600~4194304,113246208~12582912,201326592~4194304,209715200~16777216,239075328~8388608,251658240~12582912,272629760~8388608,285212672~8388608,301989888~4194304,314572800~4194304,322961408~12582912,360710144~4194304,369098752~16777216,390070272~8388608,402653184~4194304] 2024-09-21T10:38:28.720 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~8388608,20971520~8388608,33554432~16777216,54525952~25165824,88080384~12582912,104857600~4194304,113246208~12582912,201326592~4194304,209715200~16777216,239075328~8388608,251658240~12582912,272629760~8388608,285212672~8388608,301989888~4194304,314572800~4194304,322961408~12582912,360710144~4194304,369098752~16777216,390070272~8388608,402653184~4194304] 2024-09-21T10:38:28.722 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~12582912,20971520~8388608,33554432~16777216,54525952~25165824,88080384~12582912,104857600~25165824,201326592~25165824,239075328~8388608,251658240~16777216,272629760~8388608,285212672~12582912,301989888~4194304,314572800~4194304,322961408~16777216,360710144~50331648] 2024-09-21T10:38:28.722 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 7 diff [4194304~8388608,20971520~8388608,33554432~16777216,54525952~25165824,88080384~12582912,104857600~4194304,113246208~16777216,201326592~4194304,209715200~16777216,239075328~8388608,251658240~12582912,272629760~8388608,285212672~12582912,301989888~4194304,310378496~8388608,322961408~16777216,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.722 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~8388608,20971520~8388608,33554432~16777216,54525952~25165824,88080384~12582912,104857600~4194304,113246208~16777216,201326592~4194304,209715200~16777216,239075328~8388608,251658240~12582912,272629760~8388608,285212672~12582912,301989888~4194304,310378496~8388608,322961408~16777216,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.724 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~12582912,20971520~8388608,33554432~16777216,54525952~25165824,88080384~12582912,104857600~29360128,201326592~25165824,239075328~8388608,251658240~16777216,272629760~8388608,285212672~12582912,301989888~4194304,310378496~8388608,322961408~20971520,360710144~54525952] 2024-09-21T10:38:28.724 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 8 diff [4194304~8388608,20971520~8388608,33554432~16777216,54525952~25165824,83886080~16777216,104857600~4194304,113246208~16777216,155189248~4194304,171966464~8388608,192937984~12582912,209715200~16777216,239075328~8388608,251658240~12582912,272629760~8388608,285212672~12582912,301989888~37748736,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.724 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~8388608,20971520~8388608,33554432~16777216,54525952~25165824,83886080~16777216,104857600~4194304,113246208~16777216,155189248~4194304,171966464~8388608,192937984~12582912,209715200~16777216,239075328~8388608,251658240~12582912,272629760~8388608,285212672~12582912,301989888~37748736,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.727 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~12582912,20971520~8388608,33554432~46137344,83886080~16777216,104857600~29360128,155189248~4194304,171966464~8388608,192937984~33554432,239075328~29360128,272629760~8388608,285212672~58720256,360710144~54525952] 2024-09-21T10:38:28.727 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 9 diff [4194304~8388608,20971520~8388608,33554432~16777216,54525952~25165824,83886080~16777216,104857600~4194304,113246208~16777216,155189248~4194304,171966464~8388608,192937984~12582912,209715200~16777216,239075328~8388608,251658240~12582912,272629760~25165824,301989888~37748736,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.727 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~8388608,20971520~8388608,33554432~16777216,54525952~25165824,83886080~16777216,104857600~4194304,113246208~16777216,155189248~4194304,171966464~8388608,192937984~12582912,209715200~16777216,239075328~8388608,251658240~12582912,272629760~25165824,301989888~37748736,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.729 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~12582912,20971520~8388608,33554432~46137344,83886080~16777216,104857600~29360128,155189248~4194304,171966464~8388608,192937984~33554432,239075328~29360128,272629760~71303168,360710144~54525952] 2024-09-21T10:38:28.730 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 10 diff [4194304~8388608,20971520~8388608,33554432~16777216,54525952~25165824,83886080~16777216,104857600~25165824,155189248~4194304,171966464~8388608,192937984~12582912,209715200~16777216,239075328~8388608,251658240~12582912,272629760~67108864,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.730 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~8388608,20971520~8388608,33554432~16777216,54525952~25165824,83886080~16777216,104857600~25165824,155189248~4194304,171966464~8388608,192937984~12582912,209715200~16777216,239075328~8388608,251658240~12582912,272629760~67108864,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.732 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~12582912,20971520~8388608,33554432~46137344,83886080~16777216,104857600~29360128,155189248~4194304,171966464~8388608,192937984~37748736,239075328~29360128,272629760~71303168,360710144~54525952] 2024-09-21T10:38:28.732 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 11 diff [4194304~12582912,20971520~58720256,83886080~16777216,104857600~25165824,155189248~4194304,171966464~8388608,192937984~12582912,209715200~20971520,239075328~8388608,251658240~12582912,272629760~75497472,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.733 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~12582912,20971520~58720256,83886080~16777216,104857600~25165824,155189248~4194304,171966464~8388608,192937984~12582912,209715200~20971520,239075328~8388608,251658240~12582912,272629760~75497472,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.736 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~75497472,83886080~16777216,104857600~29360128,155189248~4194304,171966464~8388608,192937984~41943040,239075328~29360128,272629760~79691776,360710144~54525952] 2024-09-21T10:38:28.736 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 12 diff [4194304~96468992,104857600~25165824,155189248~4194304,167772160~12582912,192937984~12582912,209715200~25165824,239075328~25165824,272629760~75497472,360710144~4194304,369098752~29360128,402653184~8388608] 2024-09-21T10:38:28.736 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,155189248~4194304,167772160~12582912,192937984~12582912,209715200~25165824,239075328~25165824,272629760~75497472,360710144~4194304,369098752~29360128,402653184~8388608] 2024-09-21T10:38:28.740 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~29360128,155189248~4194304,167772160~12582912,192937984~75497472,272629760~79691776,360710144~54525952] 2024-09-21T10:38:28.741 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 13 diff [4194304~96468992,104857600~25165824,150994944~8388608,167772160~12582912,192937984~12582912,209715200~25165824,239075328~25165824,272629760~75497472,360710144~4194304,369098752~41943040] 2024-09-21T10:38:28.741 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,150994944~8388608,167772160~12582912,192937984~12582912,209715200~25165824,239075328~25165824,272629760~75497472,360710144~4194304,369098752~41943040] 2024-09-21T10:38:28.745 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~29360128,150994944~8388608,167772160~12582912,192937984~75497472,272629760~79691776,360710144~54525952] 2024-09-21T10:38:28.745 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 14 diff [4194304~96468992,104857600~25165824,142606336~4194304,150994944~8388608,167772160~12582912,192937984~12582912,209715200~25165824,239075328~25165824,272629760~75497472,360710144~4194304,369098752~41943040] 2024-09-21T10:38:28.745 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,142606336~4194304,150994944~8388608,167772160~12582912,192937984~12582912,209715200~25165824,239075328~25165824,272629760~75497472,360710144~4194304,369098752~41943040] 2024-09-21T10:38:28.749 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~29360128,142606336~16777216,167772160~12582912,192937984~75497472,272629760~79691776,360710144~54525952] 2024-09-21T10:38:28.749 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 15 diff [4194304~96468992,104857600~25165824,142606336~4194304,150994944~8388608,167772160~12582912,184549376~20971520,209715200~25165824,239075328~25165824,272629760~75497472,360710144~50331648] 2024-09-21T10:38:28.749 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,142606336~4194304,150994944~8388608,167772160~12582912,184549376~20971520,209715200~25165824,239075328~25165824,272629760~75497472,360710144~50331648] 2024-09-21T10:38:28.754 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~29360128,142606336~16777216,167772160~12582912,184549376~83886080,272629760~79691776,360710144~54525952] 2024-09-21T10:38:28.754 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 16 diff [4194304~96468992,104857600~25165824,142606336~4194304,150994944~8388608,167772160~12582912,184549376~20971520,209715200~25165824,239075328~25165824,272629760~75497472,360710144~54525952] 2024-09-21T10:38:28.754 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,142606336~4194304,150994944~8388608,167772160~12582912,184549376~20971520,209715200~25165824,239075328~25165824,272629760~75497472,360710144~54525952] 2024-09-21T10:38:28.759 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~29360128,142606336~16777216,167772160~12582912,184549376~83886080,272629760~79691776,360710144~58720256] 2024-09-21T10:38:28.759 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 17 diff [4194304~96468992,104857600~25165824,142606336~16777216,167772160~12582912,184549376~20971520,209715200~138412032,360710144~54525952] 2024-09-21T10:38:28.759 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,142606336~16777216,167772160~12582912,184549376~20971520,209715200~138412032,360710144~54525952] 2024-09-21T10:38:28.764 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~29360128,142606336~16777216,167772160~12582912,184549376~167772160,360710144~58720256] 2024-09-21T10:38:28.764 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 18 diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,167772160~12582912,184549376~20971520,209715200~205520896] 2024-09-21T10:38:28.765 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,167772160~12582912,184549376~20971520,209715200~205520896] 2024-09-21T10:38:28.770 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,167772160~12582912,184549376~234881024] 2024-09-21T10:38:28.770 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to 19 diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:28.770 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:28.776 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~234881024] 2024-09-21T10:38:28.776 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 4 diff [20971520~4194304,33554432~8388608,58720256~4194304,117440512~8388608,218103808~4194304,255852544~8388608,272629760~8388608,301989888~4194304,327155712~8388608,402653184~4194304] 2024-09-21T10:38:28.776 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [20971520~4194304,33554432~8388608,58720256~4194304,117440512~8388608,218103808~4194304,255852544~8388608,272629760~8388608,301989888~4194304,327155712~8388608,402653184~4194304] 2024-09-21T10:38:28.777 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [20971520~8388608,33554432~12582912,58720256~4194304,117440512~12582912,218103808~4194304,255852544~12582912,272629760~8388608,301989888~4194304,327155712~12582912,402653184~8388608] 2024-09-21T10:38:28.777 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 5 diff [20971520~4194304,33554432~8388608,58720256~12582912,92274688~8388608,104857600~4194304,117440512~8388608,218103808~4194304,255852544~8388608,272629760~8388608,301989888~4194304,314572800~4194304,322961408~12582912,360710144~4194304,377487360~8388608,394264576~12582912] 2024-09-21T10:38:28.777 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [20971520~4194304,33554432~8388608,58720256~12582912,92274688~8388608,104857600~4194304,117440512~8388608,218103808~4194304,255852544~8388608,272629760~8388608,301989888~4194304,314572800~4194304,322961408~12582912,360710144~4194304,377487360~8388608,394264576~12582912] 2024-09-21T10:38:28.778 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [20971520~8388608,33554432~12582912,58720256~16777216,92274688~8388608,104857600~8388608,117440512~12582912,218103808~4194304,255852544~12582912,272629760~8388608,301989888~4194304,314572800~4194304,322961408~16777216,360710144~8388608,377487360~12582912,394264576~16777216] 2024-09-21T10:38:28.778 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 6 diff [4194304~8388608,20971520~8388608,33554432~8388608,58720256~20971520,92274688~8388608,104857600~4194304,113246208~12582912,218103808~4194304,251658240~12582912,272629760~8388608,301989888~4194304,314572800~4194304,322961408~12582912,360710144~4194304,369098752~16777216,390070272~8388608,402653184~4194304] 2024-09-21T10:38:28.779 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~8388608,20971520~8388608,33554432~8388608,58720256~20971520,92274688~8388608,104857600~4194304,113246208~12582912,218103808~4194304,251658240~12582912,272629760~8388608,301989888~4194304,314572800~4194304,322961408~12582912,360710144~4194304,369098752~16777216,390070272~8388608,402653184~4194304] 2024-09-21T10:38:28.780 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~12582912,20971520~8388608,33554432~12582912,58720256~20971520,92274688~8388608,104857600~25165824,218103808~4194304,251658240~16777216,272629760~8388608,301989888~4194304,314572800~4194304,322961408~16777216,360710144~50331648] 2024-09-21T10:38:28.780 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 7 diff [4194304~8388608,20971520~8388608,33554432~8388608,58720256~20971520,92274688~8388608,104857600~4194304,113246208~16777216,218103808~4194304,251658240~12582912,272629760~8388608,289406976~8388608,301989888~4194304,310378496~8388608,322961408~16777216,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.780 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~8388608,20971520~8388608,33554432~8388608,58720256~20971520,92274688~8388608,104857600~4194304,113246208~16777216,218103808~4194304,251658240~12582912,272629760~8388608,289406976~8388608,301989888~4194304,310378496~8388608,322961408~16777216,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.782 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~12582912,20971520~8388608,33554432~12582912,58720256~20971520,92274688~8388608,104857600~29360128,218103808~4194304,251658240~16777216,272629760~8388608,289406976~8388608,301989888~4194304,310378496~8388608,322961408~20971520,360710144~54525952] 2024-09-21T10:38:28.782 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 8 diff [4194304~8388608,20971520~8388608,33554432~8388608,46137344~4194304,58720256~20971520,83886080~16777216,104857600~4194304,113246208~16777216,155189248~4194304,171966464~8388608,192937984~8388608,218103808~4194304,239075328~8388608,251658240~12582912,272629760~8388608,289406976~50331648,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.782 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~8388608,20971520~8388608,33554432~8388608,46137344~4194304,58720256~20971520,83886080~16777216,104857600~4194304,113246208~16777216,155189248~4194304,171966464~8388608,192937984~8388608,218103808~4194304,239075328~8388608,251658240~12582912,272629760~8388608,289406976~50331648,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.784 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~12582912,20971520~8388608,33554432~20971520,58720256~20971520,83886080~16777216,104857600~29360128,155189248~4194304,171966464~8388608,192937984~8388608,218103808~4194304,239075328~29360128,272629760~8388608,289406976~54525952,360710144~54525952] 2024-09-21T10:38:28.784 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 9 diff [4194304~8388608,20971520~8388608,33554432~8388608,46137344~4194304,58720256~20971520,83886080~16777216,104857600~4194304,113246208~16777216,155189248~4194304,171966464~8388608,192937984~12582912,218103808~4194304,239075328~8388608,251658240~12582912,272629760~12582912,289406976~50331648,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.785 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~8388608,20971520~8388608,33554432~8388608,46137344~4194304,58720256~20971520,83886080~16777216,104857600~4194304,113246208~16777216,155189248~4194304,171966464~8388608,192937984~12582912,218103808~4194304,239075328~8388608,251658240~12582912,272629760~12582912,289406976~50331648,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.787 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~12582912,20971520~8388608,33554432~20971520,58720256~20971520,83886080~16777216,104857600~29360128,155189248~4194304,171966464~8388608,192937984~12582912,218103808~4194304,239075328~29360128,272629760~12582912,289406976~54525952,360710144~54525952] 2024-09-21T10:38:28.787 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 10 diff [4194304~8388608,20971520~8388608,33554432~8388608,46137344~4194304,54525952~25165824,83886080~16777216,104857600~25165824,155189248~4194304,171966464~8388608,192937984~12582912,213909504~16777216,239075328~8388608,251658240~12582912,272629760~12582912,289406976~50331648,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.787 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~8388608,20971520~8388608,33554432~8388608,46137344~4194304,54525952~25165824,83886080~16777216,104857600~25165824,155189248~4194304,171966464~8388608,192937984~12582912,213909504~16777216,239075328~8388608,251658240~12582912,272629760~12582912,289406976~50331648,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.791 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~12582912,20971520~8388608,33554432~46137344,83886080~16777216,104857600~29360128,155189248~4194304,171966464~8388608,192937984~12582912,213909504~16777216,239075328~29360128,272629760~12582912,289406976~54525952,360710144~54525952] 2024-09-21T10:38:28.791 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 11 diff [4194304~12582912,20971520~37748736,62914560~16777216,83886080~16777216,104857600~25165824,155189248~4194304,171966464~8388608,192937984~12582912,213909504~16777216,239075328~8388608,251658240~12582912,272629760~12582912,289406976~58720256,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.791 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~12582912,20971520~37748736,62914560~16777216,83886080~16777216,104857600~25165824,155189248~4194304,171966464~8388608,192937984~12582912,213909504~16777216,239075328~8388608,251658240~12582912,272629760~12582912,289406976~58720256,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.794 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~75497472,83886080~16777216,104857600~29360128,155189248~4194304,171966464~8388608,192937984~12582912,213909504~20971520,239075328~29360128,272629760~12582912,289406976~62914560,360710144~54525952] 2024-09-21T10:38:28.794 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 12 diff [4194304~96468992,104857600~25165824,155189248~4194304,167772160~12582912,192937984~12582912,213909504~20971520,239075328~25165824,272629760~12582912,289406976~58720256,360710144~4194304,369098752~29360128,402653184~8388608] 2024-09-21T10:38:28.794 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,155189248~4194304,167772160~12582912,192937984~12582912,213909504~20971520,239075328~25165824,272629760~12582912,289406976~58720256,360710144~4194304,369098752~29360128,402653184~8388608] 2024-09-21T10:38:28.797 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~29360128,155189248~4194304,167772160~12582912,192937984~12582912,213909504~54525952,272629760~12582912,289406976~62914560,360710144~54525952] 2024-09-21T10:38:28.797 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 13 diff [4194304~96468992,104857600~25165824,150994944~8388608,167772160~12582912,192937984~12582912,213909504~20971520,239075328~25165824,272629760~12582912,289406976~58720256,360710144~4194304,369098752~41943040] 2024-09-21T10:38:28.797 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,150994944~8388608,167772160~12582912,192937984~12582912,213909504~20971520,239075328~25165824,272629760~12582912,289406976~58720256,360710144~4194304,369098752~41943040] 2024-09-21T10:38:28.801 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~29360128,150994944~8388608,167772160~12582912,192937984~12582912,213909504~54525952,272629760~12582912,289406976~62914560,360710144~54525952] 2024-09-21T10:38:28.801 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 14 diff [4194304~96468992,104857600~25165824,142606336~4194304,150994944~8388608,167772160~12582912,192937984~12582912,213909504~50331648,272629760~75497472,360710144~4194304,369098752~41943040] 2024-09-21T10:38:28.801 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,142606336~4194304,150994944~8388608,167772160~12582912,192937984~12582912,213909504~50331648,272629760~75497472,360710144~4194304,369098752~41943040] 2024-09-21T10:38:28.804 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:28 smithi160 ceph-mon[101879]: osdmap e1278: 8 total, 8 up, 8 in 2024-09-21T10:38:28.805 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~29360128,142606336~16777216,167772160~12582912,192937984~12582912,213909504~54525952,272629760~79691776,360710144~54525952] 2024-09-21T10:38:28.805 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 15 diff [4194304~96468992,104857600~25165824,142606336~4194304,150994944~8388608,167772160~12582912,184549376~20971520,213909504~50331648,272629760~75497472,360710144~50331648] 2024-09-21T10:38:28.805 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,142606336~4194304,150994944~8388608,167772160~12582912,184549376~20971520,213909504~50331648,272629760~75497472,360710144~50331648] 2024-09-21T10:38:28.810 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~29360128,142606336~16777216,167772160~12582912,184549376~20971520,213909504~54525952,272629760~79691776,360710144~54525952] 2024-09-21T10:38:28.810 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 16 diff [4194304~96468992,104857600~25165824,142606336~4194304,150994944~8388608,167772160~12582912,184549376~20971520,213909504~50331648,272629760~75497472,360710144~54525952] 2024-09-21T10:38:28.810 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,142606336~4194304,150994944~8388608,167772160~12582912,184549376~20971520,213909504~50331648,272629760~75497472,360710144~54525952] 2024-09-21T10:38:28.815 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~29360128,142606336~16777216,167772160~12582912,184549376~20971520,213909504~54525952,272629760~79691776,360710144~58720256] 2024-09-21T10:38:28.815 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 17 diff [4194304~96468992,104857600~25165824,142606336~16777216,167772160~12582912,184549376~20971520,213909504~134217728,360710144~54525952] 2024-09-21T10:38:28.815 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,142606336~16777216,167772160~12582912,184549376~20971520,213909504~134217728,360710144~54525952] 2024-09-21T10:38:28.820 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~29360128,142606336~16777216,167772160~12582912,184549376~20971520,213909504~138412032,360710144~58720256] 2024-09-21T10:38:28.820 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 18 diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,167772160~12582912,184549376~20971520,213909504~201326592] 2024-09-21T10:38:28.820 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,167772160~12582912,184549376~20971520,213909504~201326592] 2024-09-21T10:38:28.826 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,167772160~12582912,184549376~20971520,213909504~205520896] 2024-09-21T10:38:28.826 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to 19 diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:28.826 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:28.832 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:28.832 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 5 diff [62914560~8388608,92274688~8388608,104857600~4194304,314572800~4194304,322961408~4194304,360710144~4194304,377487360~8388608,394264576~8388608] 2024-09-21T10:38:28.832 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [62914560~8388608,92274688~8388608,104857600~4194304,314572800~4194304,322961408~4194304,360710144~4194304,377487360~8388608,394264576~8388608] 2024-09-21T10:38:28.833 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [62914560~12582912,92274688~8388608,104857600~8388608,314572800~4194304,322961408~4194304,360710144~8388608,377487360~12582912,394264576~8388608] 2024-09-21T10:38:28.833 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 6 diff [4194304~8388608,20971520~8388608,62914560~16777216,92274688~8388608,104857600~4194304,113246208~8388608,251658240~8388608,272629760~4194304,314572800~4194304,322961408~4194304,360710144~4194304,369098752~16777216,390070272~8388608] 2024-09-21T10:38:28.833 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~8388608,20971520~8388608,62914560~16777216,92274688~8388608,104857600~4194304,113246208~8388608,251658240~8388608,272629760~4194304,314572800~4194304,322961408~4194304,360710144~4194304,369098752~16777216,390070272~8388608] 2024-09-21T10:38:28.834 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~12582912,20971520~8388608,62914560~16777216,92274688~8388608,104857600~20971520,251658240~12582912,272629760~8388608,314572800~4194304,322961408~4194304,360710144~41943040] 2024-09-21T10:38:28.834 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 7 diff [4194304~8388608,20971520~8388608,62914560~16777216,92274688~8388608,104857600~4194304,113246208~16777216,251658240~8388608,272629760~4194304,289406976~8388608,310378496~8388608,322961408~4194304,335544320~4194304,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.834 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~8388608,20971520~8388608,62914560~16777216,92274688~8388608,104857600~4194304,113246208~16777216,251658240~8388608,272629760~4194304,289406976~8388608,310378496~8388608,322961408~4194304,335544320~4194304,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.835 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~12582912,20971520~8388608,62914560~16777216,92274688~8388608,104857600~29360128,251658240~12582912,272629760~8388608,289406976~8388608,310378496~8388608,322961408~8388608,335544320~8388608,360710144~54525952] 2024-09-21T10:38:28.836 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 8 diff [4194304~8388608,20971520~8388608,46137344~4194304,62914560~16777216,83886080~16777216,104857600~4194304,113246208~16777216,155189248~4194304,171966464~8388608,192937984~8388608,239075328~8388608,251658240~8388608,272629760~4194304,289406976~12582912,306184192~20971520,335544320~4194304,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.836 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~8388608,20971520~8388608,46137344~4194304,62914560~16777216,83886080~16777216,104857600~4194304,113246208~16777216,155189248~4194304,171966464~8388608,192937984~8388608,239075328~8388608,251658240~8388608,272629760~4194304,289406976~12582912,306184192~20971520,335544320~4194304,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.837 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~12582912,20971520~8388608,46137344~8388608,62914560~16777216,83886080~16777216,104857600~29360128,155189248~4194304,171966464~8388608,192937984~8388608,239075328~25165824,272629760~8388608,289406976~12582912,306184192~25165824,335544320~8388608,360710144~54525952] 2024-09-21T10:38:28.837 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 9 diff [4194304~8388608,20971520~8388608,33554432~8388608,46137344~4194304,62914560~16777216,83886080~16777216,104857600~4194304,113246208~16777216,155189248~4194304,171966464~8388608,192937984~12582912,239075328~8388608,251658240~8388608,272629760~4194304,281018368~4194304,289406976~12582912,306184192~20971520,335544320~4194304,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.837 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~8388608,20971520~8388608,33554432~8388608,46137344~4194304,62914560~16777216,83886080~16777216,104857600~4194304,113246208~16777216,155189248~4194304,171966464~8388608,192937984~12582912,239075328~8388608,251658240~8388608,272629760~4194304,281018368~4194304,289406976~12582912,306184192~20971520,335544320~4194304,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.839 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~12582912,20971520~8388608,33554432~20971520,62914560~16777216,83886080~16777216,104857600~29360128,155189248~4194304,171966464~8388608,192937984~12582912,239075328~25165824,272629760~12582912,289406976~12582912,306184192~25165824,335544320~8388608,360710144~54525952] 2024-09-21T10:38:28.839 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 10 diff [4194304~8388608,20971520~8388608,33554432~8388608,46137344~4194304,54525952~25165824,83886080~16777216,104857600~25165824,155189248~4194304,171966464~8388608,192937984~12582912,213909504~16777216,239075328~8388608,251658240~12582912,272629760~4194304,281018368~4194304,289406976~12582912,306184192~25165824,335544320~4194304,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.839 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~8388608,20971520~8388608,33554432~8388608,46137344~4194304,54525952~25165824,83886080~16777216,104857600~25165824,155189248~4194304,171966464~8388608,192937984~12582912,213909504~16777216,239075328~8388608,251658240~12582912,272629760~4194304,281018368~4194304,289406976~12582912,306184192~25165824,335544320~4194304,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.841 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~12582912,20971520~8388608,33554432~46137344,83886080~16777216,104857600~29360128,155189248~4194304,171966464~8388608,192937984~12582912,213909504~16777216,239075328~25165824,272629760~12582912,289406976~12582912,306184192~37748736,360710144~54525952] 2024-09-21T10:38:28.841 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 11 diff [4194304~12582912,20971520~37748736,62914560~16777216,83886080~16777216,104857600~25165824,155189248~4194304,171966464~8388608,192937984~12582912,213909504~16777216,239075328~8388608,251658240~12582912,272629760~4194304,281018368~4194304,289406976~12582912,306184192~25165824,335544320~12582912,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.842 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~12582912,20971520~37748736,62914560~16777216,83886080~16777216,104857600~25165824,155189248~4194304,171966464~8388608,192937984~12582912,213909504~16777216,239075328~8388608,251658240~12582912,272629760~4194304,281018368~4194304,289406976~12582912,306184192~25165824,335544320~12582912,360710144~4194304,369098752~16777216,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.843 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:28 smithi070 ceph-mon[107385]: osdmap e1278: 8 total, 8 up, 8 in 2024-09-21T10:38:28.843 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:28 smithi070 ceph-mon[112139]: osdmap e1278: 8 total, 8 up, 8 in 2024-09-21T10:38:28.844 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~75497472,83886080~16777216,104857600~29360128,155189248~4194304,171966464~8388608,192937984~12582912,213909504~20971520,239075328~25165824,272629760~12582912,289406976~12582912,306184192~46137344,360710144~54525952] 2024-09-21T10:38:28.844 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 12 diff [4194304~96468992,104857600~25165824,155189248~4194304,167772160~12582912,192937984~12582912,213909504~20971520,239075328~25165824,272629760~4194304,281018368~4194304,289406976~12582912,306184192~25165824,335544320~12582912,360710144~4194304,369098752~29360128,402653184~8388608] 2024-09-21T10:38:28.844 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,155189248~4194304,167772160~12582912,192937984~12582912,213909504~20971520,239075328~25165824,272629760~4194304,281018368~4194304,289406976~12582912,306184192~25165824,335544320~12582912,360710144~4194304,369098752~29360128,402653184~8388608] 2024-09-21T10:38:28.848 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~29360128,155189248~4194304,167772160~12582912,192937984~12582912,213909504~50331648,272629760~12582912,289406976~12582912,306184192~46137344,360710144~54525952] 2024-09-21T10:38:28.848 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 13 diff [4194304~96468992,104857600~25165824,150994944~8388608,167772160~12582912,192937984~12582912,213909504~20971520,239075328~25165824,272629760~4194304,281018368~4194304,289406976~12582912,306184192~25165824,335544320~12582912,360710144~4194304,369098752~41943040] 2024-09-21T10:38:28.848 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,150994944~8388608,167772160~12582912,192937984~12582912,213909504~20971520,239075328~25165824,272629760~4194304,281018368~4194304,289406976~12582912,306184192~25165824,335544320~12582912,360710144~4194304,369098752~41943040] 2024-09-21T10:38:28.852 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~29360128,150994944~8388608,167772160~12582912,192937984~12582912,213909504~50331648,272629760~12582912,289406976~62914560,360710144~54525952] 2024-09-21T10:38:28.852 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 14 diff [4194304~96468992,104857600~25165824,142606336~4194304,150994944~8388608,167772160~12582912,192937984~12582912,213909504~50331648,272629760~4194304,281018368~20971520,306184192~41943040,360710144~4194304,369098752~41943040] 2024-09-21T10:38:28.852 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,142606336~4194304,150994944~8388608,167772160~12582912,192937984~12582912,213909504~50331648,272629760~4194304,281018368~20971520,306184192~41943040,360710144~4194304,369098752~41943040] 2024-09-21T10:38:28.856 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~29360128,142606336~16777216,167772160~12582912,192937984~12582912,213909504~50331648,272629760~79691776,360710144~54525952] 2024-09-21T10:38:28.856 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 15 diff [4194304~96468992,104857600~25165824,142606336~4194304,150994944~8388608,167772160~12582912,184549376~20971520,213909504~50331648,272629760~4194304,281018368~20971520,306184192~41943040,360710144~50331648] 2024-09-21T10:38:28.856 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,142606336~4194304,150994944~8388608,167772160~12582912,184549376~20971520,213909504~50331648,272629760~4194304,281018368~20971520,306184192~41943040,360710144~50331648] 2024-09-21T10:38:28.860 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~29360128,142606336~16777216,167772160~12582912,184549376~20971520,213909504~50331648,272629760~79691776,360710144~54525952] 2024-09-21T10:38:28.861 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 16 diff [4194304~96468992,104857600~25165824,142606336~4194304,150994944~8388608,167772160~12582912,184549376~20971520,213909504~50331648,272629760~4194304,281018368~67108864,360710144~54525952] 2024-09-21T10:38:28.861 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,142606336~4194304,150994944~8388608,167772160~12582912,184549376~20971520,213909504~50331648,272629760~4194304,281018368~67108864,360710144~54525952] 2024-09-21T10:38:28.866 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~29360128,142606336~16777216,167772160~12582912,184549376~20971520,213909504~50331648,272629760~79691776,360710144~58720256] 2024-09-21T10:38:28.866 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 17 diff [4194304~96468992,104857600~25165824,142606336~16777216,167772160~12582912,184549376~20971520,213909504~62914560,281018368~67108864,360710144~54525952] 2024-09-21T10:38:28.866 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,142606336~16777216,167772160~12582912,184549376~20971520,213909504~62914560,281018368~67108864,360710144~54525952] 2024-09-21T10:38:28.871 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~29360128,142606336~16777216,167772160~12582912,184549376~20971520,213909504~138412032,360710144~58720256] 2024-09-21T10:38:28.871 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 18 diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,167772160~12582912,184549376~20971520,213909504~62914560,281018368~134217728] 2024-09-21T10:38:28.871 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,167772160~12582912,184549376~20971520,213909504~62914560,281018368~134217728] 2024-09-21T10:38:28.876 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,167772160~12582912,184549376~20971520,213909504~205520896] 2024-09-21T10:38:28.876 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to 19 diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:28.876 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:28.881 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:28.881 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 6 diff [4194304~8388608,20971520~8388608,62914560~4194304,75497472~4194304,113246208~8388608,251658240~8388608,272629760~4194304,369098752~8388608,390070272~8388608] 2024-09-21T10:38:28.881 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~8388608,20971520~8388608,62914560~4194304,75497472~4194304,113246208~8388608,251658240~8388608,272629760~4194304,369098752~8388608,390070272~8388608] 2024-09-21T10:38:28.882 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~12582912,20971520~8388608,62914560~4194304,75497472~4194304,113246208~12582912,251658240~12582912,272629760~8388608,369098752~12582912,390070272~12582912] 2024-09-21T10:38:28.882 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 7 diff [4194304~8388608,20971520~8388608,62914560~4194304,71303168~8388608,113246208~16777216,251658240~8388608,272629760~4194304,289406976~8388608,310378496~8388608,322961408~4194304,335544320~4194304,369098752~8388608,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.882 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~8388608,20971520~8388608,62914560~4194304,71303168~8388608,113246208~16777216,251658240~8388608,272629760~4194304,289406976~8388608,310378496~8388608,322961408~4194304,335544320~4194304,369098752~8388608,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.883 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~12582912,20971520~8388608,62914560~4194304,71303168~8388608,113246208~20971520,251658240~12582912,272629760~8388608,289406976~8388608,310378496~8388608,322961408~8388608,335544320~8388608,369098752~12582912,390070272~25165824] 2024-09-21T10:38:28.883 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 8 diff [4194304~8388608,20971520~8388608,46137344~4194304,62914560~4194304,71303168~8388608,83886080~8388608,113246208~16777216,155189248~4194304,171966464~8388608,192937984~8388608,239075328~8388608,251658240~8388608,272629760~4194304,289406976~12582912,306184192~20971520,335544320~4194304,369098752~8388608,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.884 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~8388608,20971520~8388608,46137344~4194304,62914560~4194304,71303168~8388608,83886080~8388608,113246208~16777216,155189248~4194304,171966464~8388608,192937984~8388608,239075328~8388608,251658240~8388608,272629760~4194304,289406976~12582912,306184192~20971520,335544320~4194304,369098752~8388608,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.885 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~12582912,20971520~8388608,46137344~8388608,62914560~4194304,71303168~8388608,83886080~12582912,113246208~20971520,155189248~4194304,171966464~8388608,192937984~8388608,239075328~25165824,272629760~8388608,289406976~12582912,306184192~25165824,335544320~8388608,369098752~12582912,390070272~25165824] 2024-09-21T10:38:28.885 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 9 diff [4194304~8388608,20971520~8388608,33554432~8388608,46137344~4194304,62914560~4194304,71303168~8388608,83886080~12582912,113246208~16777216,155189248~4194304,171966464~8388608,192937984~12582912,239075328~8388608,251658240~8388608,272629760~4194304,281018368~4194304,289406976~12582912,306184192~20971520,335544320~4194304,369098752~8388608,381681664~4194304,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.885 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~8388608,20971520~8388608,33554432~8388608,46137344~4194304,62914560~4194304,71303168~8388608,83886080~12582912,113246208~16777216,155189248~4194304,171966464~8388608,192937984~12582912,239075328~8388608,251658240~8388608,272629760~4194304,281018368~4194304,289406976~12582912,306184192~20971520,335544320~4194304,369098752~8388608,381681664~4194304,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.886 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~12582912,20971520~8388608,33554432~20971520,62914560~4194304,71303168~8388608,83886080~16777216,113246208~20971520,155189248~4194304,171966464~8388608,192937984~12582912,239075328~25165824,272629760~12582912,289406976~12582912,306184192~25165824,335544320~8388608,369098752~16777216,390070272~25165824] 2024-09-21T10:38:28.887 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 10 diff [4194304~8388608,20971520~8388608,33554432~8388608,46137344~4194304,54525952~12582912,71303168~8388608,83886080~12582912,109051904~20971520,155189248~4194304,171966464~8388608,192937984~12582912,213909504~16777216,239075328~8388608,251658240~12582912,272629760~4194304,281018368~4194304,289406976~12582912,306184192~25165824,335544320~4194304,369098752~8388608,381681664~4194304,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.887 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~8388608,20971520~8388608,33554432~8388608,46137344~4194304,54525952~12582912,71303168~8388608,83886080~12582912,109051904~20971520,155189248~4194304,171966464~8388608,192937984~12582912,213909504~16777216,239075328~8388608,251658240~12582912,272629760~4194304,281018368~4194304,289406976~12582912,306184192~25165824,335544320~4194304,369098752~8388608,381681664~4194304,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.888 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~12582912,20971520~8388608,33554432~33554432,71303168~8388608,83886080~16777216,109051904~25165824,155189248~4194304,171966464~8388608,192937984~12582912,213909504~16777216,239075328~25165824,272629760~12582912,289406976~12582912,306184192~37748736,369098752~16777216,390070272~25165824] 2024-09-21T10:38:28.888 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 11 diff [4194304~12582912,20971520~37748736,62914560~4194304,71303168~8388608,83886080~12582912,109051904~20971520,155189248~4194304,171966464~8388608,192937984~12582912,213909504~16777216,239075328~8388608,251658240~12582912,272629760~4194304,281018368~4194304,289406976~12582912,306184192~25165824,335544320~12582912,369098752~8388608,381681664~4194304,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.889 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~12582912,20971520~37748736,62914560~4194304,71303168~8388608,83886080~12582912,109051904~20971520,155189248~4194304,171966464~8388608,192937984~12582912,213909504~16777216,239075328~8388608,251658240~12582912,272629760~4194304,281018368~4194304,289406976~12582912,306184192~25165824,335544320~12582912,369098752~8388608,381681664~4194304,390070272~8388608,402653184~8388608] 2024-09-21T10:38:28.891 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~62914560,71303168~8388608,83886080~16777216,109051904~25165824,155189248~4194304,171966464~8388608,192937984~12582912,213909504~20971520,239075328~25165824,272629760~12582912,289406976~12582912,306184192~46137344,369098752~16777216,390070272~25165824] 2024-09-21T10:38:28.891 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 12 diff [4194304~62914560,71303168~25165824,109051904~20971520,155189248~4194304,167772160~12582912,192937984~12582912,213909504~20971520,239075328~25165824,272629760~4194304,281018368~4194304,289406976~12582912,306184192~25165824,335544320~12582912,369098752~8388608,381681664~16777216,402653184~8388608] 2024-09-21T10:38:28.891 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~62914560,71303168~25165824,109051904~20971520,155189248~4194304,167772160~12582912,192937984~12582912,213909504~20971520,239075328~25165824,272629760~4194304,281018368~4194304,289406976~12582912,306184192~25165824,335544320~12582912,369098752~8388608,381681664~16777216,402653184~8388608] 2024-09-21T10:38:28.894 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~62914560,71303168~29360128,109051904~25165824,155189248~4194304,167772160~12582912,192937984~12582912,213909504~50331648,272629760~12582912,289406976~12582912,306184192~46137344,369098752~46137344] 2024-09-21T10:38:28.894 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 13 diff [4194304~62914560,71303168~25165824,109051904~20971520,150994944~8388608,167772160~12582912,192937984~12582912,213909504~20971520,239075328~25165824,272629760~4194304,281018368~4194304,289406976~12582912,306184192~25165824,335544320~12582912,369098752~8388608,381681664~29360128] 2024-09-21T10:38:28.894 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~62914560,71303168~25165824,109051904~20971520,150994944~8388608,167772160~12582912,192937984~12582912,213909504~20971520,239075328~25165824,272629760~4194304,281018368~4194304,289406976~12582912,306184192~25165824,335544320~12582912,369098752~8388608,381681664~29360128] 2024-09-21T10:38:28.897 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~62914560,71303168~29360128,109051904~25165824,150994944~8388608,167772160~12582912,192937984~12582912,213909504~50331648,272629760~12582912,289406976~62914560,369098752~46137344] 2024-09-21T10:38:28.897 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 14 diff [4194304~62914560,71303168~25165824,104857600~25165824,142606336~4194304,150994944~8388608,167772160~12582912,192937984~12582912,213909504~50331648,272629760~4194304,281018368~20971520,306184192~41943040,369098752~8388608,381681664~29360128] 2024-09-21T10:38:28.897 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~62914560,71303168~25165824,104857600~25165824,142606336~4194304,150994944~8388608,167772160~12582912,192937984~12582912,213909504~50331648,272629760~4194304,281018368~20971520,306184192~41943040,369098752~8388608,381681664~29360128] 2024-09-21T10:38:28.901 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~62914560,71303168~29360128,104857600~29360128,142606336~16777216,167772160~12582912,192937984~12582912,213909504~50331648,272629760~79691776,369098752~46137344] 2024-09-21T10:38:28.901 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 15 diff [4194304~62914560,71303168~25165824,104857600~25165824,142606336~4194304,150994944~8388608,167772160~12582912,184549376~20971520,213909504~50331648,272629760~4194304,281018368~20971520,306184192~41943040,360710144~16777216,381681664~29360128] 2024-09-21T10:38:28.901 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~62914560,71303168~25165824,104857600~25165824,142606336~4194304,150994944~8388608,167772160~12582912,184549376~20971520,213909504~50331648,272629760~4194304,281018368~20971520,306184192~41943040,360710144~16777216,381681664~29360128] 2024-09-21T10:38:28.905 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~62914560,71303168~29360128,104857600~29360128,142606336~16777216,167772160~12582912,184549376~20971520,213909504~50331648,272629760~79691776,360710144~54525952] 2024-09-21T10:38:28.905 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 16 diff [4194304~62914560,71303168~25165824,104857600~25165824,142606336~4194304,150994944~8388608,167772160~12582912,184549376~20971520,213909504~50331648,272629760~4194304,281018368~67108864,360710144~16777216,381681664~33554432] 2024-09-21T10:38:28.905 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~62914560,71303168~25165824,104857600~25165824,142606336~4194304,150994944~8388608,167772160~12582912,184549376~20971520,213909504~50331648,272629760~4194304,281018368~67108864,360710144~16777216,381681664~33554432] 2024-09-21T10:38:28.910 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~62914560,71303168~29360128,104857600~29360128,142606336~16777216,167772160~12582912,184549376~20971520,213909504~50331648,272629760~79691776,360710144~58720256] 2024-09-21T10:38:28.910 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 17 diff [4194304~92274688,104857600~25165824,142606336~16777216,167772160~12582912,184549376~20971520,213909504~62914560,281018368~67108864,360710144~16777216,381681664~33554432] 2024-09-21T10:38:28.910 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~92274688,104857600~25165824,142606336~16777216,167772160~12582912,184549376~20971520,213909504~62914560,281018368~67108864,360710144~16777216,381681664~33554432] 2024-09-21T10:38:28.915 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~29360128,142606336~16777216,167772160~12582912,184549376~20971520,213909504~138412032,360710144~58720256] 2024-09-21T10:38:28.915 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 18 diff [4194304~92274688,104857600~25165824,134217728~4194304,142606336~16777216,167772160~12582912,184549376~20971520,213909504~62914560,281018368~96468992,381681664~33554432] 2024-09-21T10:38:28.915 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~92274688,104857600~25165824,134217728~4194304,142606336~16777216,167772160~12582912,184549376~20971520,213909504~62914560,281018368~96468992,381681664~33554432] 2024-09-21T10:38:28.920 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,167772160~12582912,184549376~20971520,213909504~205520896] 2024-09-21T10:38:28.920 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to 19 diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:28.920 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:28.927 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:28.927 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to 7 diff [71303168~8388608,121634816~12582912,255852544~4194304,289406976~8388608,310378496~8388608,322961408~4194304,335544320~4194304,402653184~8388608] 2024-09-21T10:38:28.927 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [71303168~8388608,121634816~12582912,255852544~4194304,289406976~8388608,310378496~8388608,322961408~4194304,335544320~4194304,402653184~8388608] 2024-09-21T10:38:28.927 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [71303168~8388608,121634816~12582912,255852544~4194304,289406976~8388608,310378496~8388608,322961408~8388608,335544320~8388608,402653184~12582912] 2024-09-21T10:38:28.927 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to 8 diff [8388608~4194304,46137344~4194304,71303168~8388608,83886080~8388608,121634816~12582912,155189248~4194304,171966464~8388608,192937984~8388608,239075328~8388608,255852544~4194304,289406976~12582912,306184192~20971520,335544320~4194304,402653184~8388608] 2024-09-21T10:38:28.927 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~4194304,46137344~4194304,71303168~8388608,83886080~8388608,121634816~12582912,155189248~4194304,171966464~8388608,192937984~8388608,239075328~8388608,255852544~4194304,289406976~12582912,306184192~20971520,335544320~4194304,402653184~8388608] 2024-09-21T10:38:28.928 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~8388608,46137344~8388608,71303168~8388608,83886080~12582912,121634816~12582912,155189248~4194304,171966464~8388608,192937984~8388608,239075328~12582912,255852544~4194304,289406976~12582912,306184192~25165824,335544320~8388608,402653184~12582912] 2024-09-21T10:38:28.928 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to 9 diff [4194304~8388608,33554432~8388608,46137344~4194304,71303168~8388608,83886080~12582912,113246208~4194304,121634816~12582912,155189248~4194304,171966464~8388608,192937984~12582912,239075328~8388608,255852544~4194304,281018368~4194304,289406976~12582912,306184192~20971520,335544320~4194304,369098752~4194304,381681664~4194304,402653184~8388608] 2024-09-21T10:38:28.928 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~8388608,33554432~8388608,46137344~4194304,71303168~8388608,83886080~12582912,113246208~4194304,121634816~12582912,155189248~4194304,171966464~8388608,192937984~12582912,239075328~8388608,255852544~4194304,281018368~4194304,289406976~12582912,306184192~20971520,335544320~4194304,369098752~4194304,381681664~4194304,402653184~8388608] 2024-09-21T10:38:28.929 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~12582912,33554432~20971520,71303168~8388608,83886080~16777216,113246208~4194304,121634816~12582912,155189248~4194304,171966464~8388608,192937984~12582912,239075328~12582912,255852544~4194304,281018368~4194304,289406976~12582912,306184192~25165824,335544320~8388608,369098752~4194304,381681664~4194304,402653184~12582912] 2024-09-21T10:38:28.929 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to 10 diff [4194304~8388608,33554432~8388608,46137344~4194304,54525952~8388608,71303168~8388608,83886080~12582912,109051904~25165824,155189248~4194304,171966464~8388608,192937984~12582912,213909504~16777216,239075328~8388608,255852544~4194304,281018368~4194304,289406976~12582912,306184192~25165824,335544320~4194304,369098752~4194304,381681664~4194304,402653184~8388608] 2024-09-21T10:38:28.929 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~8388608,33554432~8388608,46137344~4194304,54525952~8388608,71303168~8388608,83886080~12582912,109051904~25165824,155189248~4194304,171966464~8388608,192937984~12582912,213909504~16777216,239075328~8388608,255852544~4194304,281018368~4194304,289406976~12582912,306184192~25165824,335544320~4194304,369098752~4194304,381681664~4194304,402653184~8388608] 2024-09-21T10:38:28.931 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~12582912,33554432~29360128,71303168~8388608,83886080~16777216,109051904~25165824,155189248~4194304,171966464~8388608,192937984~12582912,213909504~16777216,239075328~12582912,255852544~8388608,281018368~4194304,289406976~12582912,306184192~37748736,369098752~4194304,381681664~4194304,402653184~12582912] 2024-09-21T10:38:28.931 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to 11 diff [4194304~12582912,25165824~33554432,71303168~8388608,83886080~12582912,109051904~25165824,155189248~4194304,171966464~8388608,192937984~12582912,213909504~16777216,239075328~8388608,255852544~4194304,281018368~4194304,289406976~12582912,306184192~25165824,335544320~12582912,369098752~4194304,381681664~4194304,402653184~8388608] 2024-09-21T10:38:28.931 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~12582912,25165824~33554432,71303168~8388608,83886080~12582912,109051904~25165824,155189248~4194304,171966464~8388608,192937984~12582912,213909504~16777216,239075328~8388608,255852544~4194304,281018368~4194304,289406976~12582912,306184192~25165824,335544320~12582912,369098752~4194304,381681664~4194304,402653184~8388608] 2024-09-21T10:38:28.933 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~37748736,71303168~8388608,83886080~16777216,109051904~25165824,155189248~4194304,171966464~8388608,192937984~12582912,213909504~20971520,239075328~12582912,255852544~8388608,281018368~4194304,289406976~12582912,306184192~46137344,369098752~8388608,381681664~4194304,402653184~12582912] 2024-09-21T10:38:28.933 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to 12 diff [4194304~16777216,25165824~37748736,71303168~25165824,109051904~25165824,155189248~4194304,167772160~12582912,192937984~12582912,213909504~20971520,239075328~12582912,255852544~4194304,281018368~4194304,289406976~12582912,306184192~25165824,335544320~12582912,369098752~4194304,381681664~8388608,402653184~8388608] 2024-09-21T10:38:28.933 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~37748736,71303168~25165824,109051904~25165824,155189248~4194304,167772160~12582912,192937984~12582912,213909504~20971520,239075328~12582912,255852544~4194304,281018368~4194304,289406976~12582912,306184192~25165824,335544320~12582912,369098752~4194304,381681664~8388608,402653184~8388608] 2024-09-21T10:38:28.935 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~37748736,71303168~29360128,109051904~25165824,155189248~4194304,167772160~12582912,192937984~12582912,213909504~50331648,281018368~4194304,289406976~12582912,306184192~46137344,369098752~8388608,381681664~12582912,402653184~12582912] 2024-09-21T10:38:28.936 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to 13 diff [4194304~16777216,25165824~41943040,71303168~25165824,109051904~25165824,150994944~8388608,167772160~12582912,192937984~12582912,213909504~20971520,239075328~12582912,255852544~4194304,281018368~4194304,289406976~12582912,306184192~25165824,335544320~12582912,369098752~8388608,381681664~12582912,402653184~8388608] 2024-09-21T10:38:28.936 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~41943040,71303168~25165824,109051904~25165824,150994944~8388608,167772160~12582912,192937984~12582912,213909504~20971520,239075328~12582912,255852544~4194304,281018368~4194304,289406976~12582912,306184192~25165824,335544320~12582912,369098752~8388608,381681664~12582912,402653184~8388608] 2024-09-21T10:38:28.939 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~41943040,71303168~29360128,109051904~25165824,150994944~8388608,167772160~12582912,192937984~12582912,213909504~50331648,281018368~4194304,289406976~62914560,369098752~8388608,381681664~16777216,402653184~12582912] 2024-09-21T10:38:28.939 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to 14 diff [4194304~16777216,25165824~41943040,71303168~25165824,104857600~29360128,142606336~4194304,150994944~8388608,167772160~12582912,192937984~12582912,213909504~37748736,255852544~4194304,281018368~20971520,306184192~41943040,369098752~8388608,381681664~12582912,402653184~8388608] 2024-09-21T10:38:28.939 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~41943040,71303168~25165824,104857600~29360128,142606336~4194304,150994944~8388608,167772160~12582912,192937984~12582912,213909504~37748736,255852544~4194304,281018368~20971520,306184192~41943040,369098752~8388608,381681664~12582912,402653184~8388608] 2024-09-21T10:38:28.942 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~41943040,71303168~29360128,104857600~29360128,142606336~16777216,167772160~12582912,192937984~12582912,213909504~50331648,281018368~71303168,369098752~8388608,381681664~16777216,402653184~12582912] 2024-09-21T10:38:28.942 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to 15 diff [4194304~16777216,25165824~41943040,71303168~25165824,104857600~29360128,142606336~4194304,150994944~8388608,167772160~12582912,184549376~20971520,213909504~46137344,281018368~20971520,306184192~41943040,360710144~16777216,381681664~12582912,402653184~8388608] 2024-09-21T10:38:28.942 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~41943040,71303168~25165824,104857600~29360128,142606336~4194304,150994944~8388608,167772160~12582912,184549376~20971520,213909504~46137344,281018368~20971520,306184192~41943040,360710144~16777216,381681664~12582912,402653184~8388608] 2024-09-21T10:38:28.945 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~41943040,71303168~29360128,104857600~29360128,142606336~16777216,167772160~12582912,184549376~20971520,213909504~50331648,281018368~71303168,360710144~16777216,381681664~16777216,402653184~12582912] 2024-09-21T10:38:28.945 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to 16 diff [4194304~16777216,25165824~41943040,71303168~25165824,104857600~29360128,142606336~4194304,150994944~8388608,167772160~12582912,184549376~20971520,213909504~46137344,281018368~67108864,360710144~16777216,381681664~12582912,398458880~16777216] 2024-09-21T10:38:28.946 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~41943040,71303168~25165824,104857600~29360128,142606336~4194304,150994944~8388608,167772160~12582912,184549376~20971520,213909504~46137344,281018368~67108864,360710144~16777216,381681664~12582912,398458880~16777216] 2024-09-21T10:38:28.949 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~41943040,71303168~29360128,104857600~29360128,142606336~16777216,167772160~12582912,184549376~20971520,213909504~50331648,281018368~71303168,360710144~16777216,381681664~37748736] 2024-09-21T10:38:28.949 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to 17 diff [4194304~16777216,25165824~71303168,104857600~29360128,142606336~16777216,167772160~12582912,184549376~20971520,213909504~46137344,264241152~8388608,281018368~67108864,360710144~16777216,381681664~12582912,398458880~16777216] 2024-09-21T10:38:28.949 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~71303168,104857600~29360128,142606336~16777216,167772160~12582912,184549376~20971520,213909504~46137344,264241152~8388608,281018368~67108864,360710144~16777216,381681664~12582912,398458880~16777216] 2024-09-21T10:38:28.953 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~29360128,142606336~16777216,167772160~12582912,184549376~20971520,213909504~62914560,281018368~71303168,360710144~16777216,381681664~37748736] 2024-09-21T10:38:28.953 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to 18 diff [4194304~16777216,25165824~71303168,104857600~33554432,142606336~16777216,167772160~12582912,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:28.953 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~71303168,104857600~33554432,142606336~16777216,167772160~12582912,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:28.957 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~54525952,167772160~12582912,184549376~20971520,213909504~62914560,281018368~96468992,381681664~37748736] 2024-09-21T10:38:28.958 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to 19 diff [4194304~16777216,25165824~75497472,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:28.958 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~75497472,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:28.963 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~54525952,163577856~16777216,184549376~20971520,213909504~62914560,281018368~96468992,381681664~37748736] 2024-09-21T10:38:28.963 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to 8 diff [8388608~4194304,46137344~4194304,83886080~8388608,155189248~4194304,171966464~8388608,192937984~8388608,239075328~8388608,293601280~8388608,306184192~8388608,318767104~4194304] 2024-09-21T10:38:28.963 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~4194304,46137344~4194304,83886080~8388608,155189248~4194304,171966464~8388608,192937984~8388608,239075328~8388608,293601280~8388608,306184192~8388608,318767104~4194304] 2024-09-21T10:38:28.963 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~8388608,46137344~8388608,83886080~12582912,155189248~4194304,171966464~8388608,192937984~8388608,239075328~12582912,293601280~8388608,306184192~8388608,318767104~4194304] 2024-09-21T10:38:28.964 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to 9 diff [4194304~8388608,33554432~8388608,46137344~4194304,83886080~12582912,113246208~4194304,155189248~4194304,171966464~8388608,192937984~12582912,239075328~8388608,281018368~4194304,293601280~8388608,306184192~8388608,318767104~4194304,369098752~4194304,381681664~4194304] 2024-09-21T10:38:28.964 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~8388608,33554432~8388608,46137344~4194304,83886080~12582912,113246208~4194304,155189248~4194304,171966464~8388608,192937984~12582912,239075328~8388608,281018368~4194304,293601280~8388608,306184192~8388608,318767104~4194304,369098752~4194304,381681664~4194304] 2024-09-21T10:38:28.965 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~12582912,33554432~20971520,83886080~16777216,113246208~4194304,155189248~4194304,171966464~8388608,192937984~12582912,239075328~12582912,281018368~4194304,293601280~8388608,306184192~8388608,318767104~4194304,369098752~4194304,381681664~4194304] 2024-09-21T10:38:28.965 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to 10 diff [4194304~8388608,33554432~8388608,46137344~4194304,54525952~8388608,83886080~12582912,109051904~16777216,155189248~4194304,171966464~8388608,192937984~12582912,213909504~16777216,239075328~8388608,255852544~4194304,281018368~4194304,293601280~8388608,306184192~8388608,318767104~4194304,327155712~4194304,369098752~4194304,381681664~4194304] 2024-09-21T10:38:28.965 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~8388608,33554432~8388608,46137344~4194304,54525952~8388608,83886080~12582912,109051904~16777216,155189248~4194304,171966464~8388608,192937984~12582912,213909504~16777216,239075328~8388608,255852544~4194304,281018368~4194304,293601280~8388608,306184192~8388608,318767104~4194304,327155712~4194304,369098752~4194304,381681664~4194304] 2024-09-21T10:38:28.966 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~12582912,33554432~29360128,83886080~16777216,109051904~16777216,155189248~4194304,171966464~8388608,192937984~12582912,213909504~16777216,239075328~12582912,255852544~8388608,281018368~4194304,293601280~8388608,306184192~8388608,318767104~4194304,327155712~8388608,369098752~4194304,381681664~4194304] 2024-09-21T10:38:28.966 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to 11 diff [4194304~12582912,25165824~33554432,83886080~12582912,109051904~16777216,155189248~4194304,171966464~8388608,192937984~12582912,213909504~16777216,239075328~8388608,255852544~4194304,281018368~4194304,293601280~8388608,306184192~8388608,318767104~4194304,327155712~4194304,339738624~8388608,369098752~4194304,381681664~4194304] 2024-09-21T10:38:28.966 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~12582912,25165824~33554432,83886080~12582912,109051904~16777216,155189248~4194304,171966464~8388608,192937984~12582912,213909504~16777216,239075328~8388608,255852544~4194304,281018368~4194304,293601280~8388608,306184192~8388608,318767104~4194304,327155712~4194304,339738624~8388608,369098752~4194304,381681664~4194304] 2024-09-21T10:38:28.968 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~37748736,83886080~16777216,109051904~16777216,155189248~4194304,171966464~8388608,192937984~12582912,213909504~20971520,239075328~12582912,255852544~8388608,281018368~4194304,293601280~8388608,306184192~8388608,318767104~4194304,327155712~8388608,339738624~12582912,369098752~8388608,381681664~4194304] 2024-09-21T10:38:28.968 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to 12 diff [4194304~16777216,25165824~37748736,79691776~16777216,109051904~16777216,155189248~4194304,167772160~12582912,192937984~12582912,213909504~20971520,239075328~12582912,255852544~4194304,281018368~4194304,293601280~8388608,306184192~8388608,318767104~4194304,327155712~4194304,339738624~8388608,369098752~4194304,381681664~8388608] 2024-09-21T10:38:28.968 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~37748736,79691776~16777216,109051904~16777216,155189248~4194304,167772160~12582912,192937984~12582912,213909504~20971520,239075328~12582912,255852544~4194304,281018368~4194304,293601280~8388608,306184192~8388608,318767104~4194304,327155712~4194304,339738624~8388608,369098752~4194304,381681664~8388608] 2024-09-21T10:38:28.970 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~37748736,79691776~20971520,109051904~16777216,155189248~4194304,167772160~12582912,192937984~12582912,213909504~50331648,281018368~4194304,293601280~8388608,306184192~8388608,318767104~4194304,327155712~8388608,339738624~12582912,369098752~8388608,381681664~12582912] 2024-09-21T10:38:28.970 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to 13 diff [4194304~16777216,25165824~41943040,79691776~16777216,109051904~16777216,150994944~8388608,167772160~12582912,192937984~12582912,213909504~20971520,239075328~12582912,255852544~4194304,281018368~4194304,289406976~12582912,306184192~8388608,318767104~4194304,327155712~4194304,339738624~8388608,369098752~8388608,381681664~12582912] 2024-09-21T10:38:28.970 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~41943040,79691776~16777216,109051904~16777216,150994944~8388608,167772160~12582912,192937984~12582912,213909504~20971520,239075328~12582912,255852544~4194304,281018368~4194304,289406976~12582912,306184192~8388608,318767104~4194304,327155712~4194304,339738624~8388608,369098752~8388608,381681664~12582912] 2024-09-21T10:38:28.973 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~41943040,79691776~20971520,109051904~16777216,150994944~8388608,167772160~12582912,192937984~12582912,213909504~50331648,281018368~4194304,289406976~25165824,318767104~4194304,327155712~8388608,339738624~12582912,369098752~8388608,381681664~16777216] 2024-09-21T10:38:28.973 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to 14 diff [4194304~16777216,25165824~41943040,79691776~16777216,104857600~20971520,142606336~4194304,150994944~8388608,167772160~12582912,192937984~12582912,213909504~37748736,255852544~4194304,281018368~20971520,306184192~8388608,318767104~4194304,327155712~8388608,339738624~8388608,369098752~8388608,381681664~12582912] 2024-09-21T10:38:28.973 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~41943040,79691776~16777216,104857600~20971520,142606336~4194304,150994944~8388608,167772160~12582912,192937984~12582912,213909504~37748736,255852544~4194304,281018368~20971520,306184192~8388608,318767104~4194304,327155712~8388608,339738624~8388608,369098752~8388608,381681664~12582912] 2024-09-21T10:38:28.975 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~41943040,79691776~20971520,104857600~20971520,142606336~16777216,167772160~12582912,192937984~12582912,213909504~50331648,281018368~33554432,318767104~4194304,327155712~25165824,369098752~8388608,381681664~16777216] 2024-09-21T10:38:28.976 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to 15 diff [4194304~16777216,25165824~41943040,75497472~20971520,104857600~20971520,142606336~4194304,150994944~8388608,167772160~12582912,184549376~20971520,213909504~46137344,281018368~20971520,306184192~8388608,318767104~4194304,327155712~8388608,339738624~8388608,360710144~16777216,381681664~12582912] 2024-09-21T10:38:28.976 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~41943040,75497472~20971520,104857600~20971520,142606336~4194304,150994944~8388608,167772160~12582912,184549376~20971520,213909504~46137344,281018368~20971520,306184192~8388608,318767104~4194304,327155712~8388608,339738624~8388608,360710144~16777216,381681664~12582912] 2024-09-21T10:38:28.979 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~41943040,75497472~25165824,104857600~20971520,142606336~16777216,167772160~12582912,184549376~20971520,213909504~50331648,281018368~33554432,318767104~4194304,327155712~25165824,360710144~16777216,381681664~16777216] 2024-09-21T10:38:28.979 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to 16 diff [4194304~16777216,25165824~41943040,71303168~25165824,104857600~20971520,142606336~4194304,150994944~8388608,167772160~12582912,184549376~20971520,213909504~46137344,281018368~29360128,318767104~4194304,327155712~8388608,339738624~8388608,360710144~16777216,381681664~12582912,398458880~16777216] 2024-09-21T10:38:28.979 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~41943040,71303168~25165824,104857600~20971520,142606336~4194304,150994944~8388608,167772160~12582912,184549376~20971520,213909504~46137344,281018368~29360128,318767104~4194304,327155712~8388608,339738624~8388608,360710144~16777216,381681664~12582912,398458880~16777216] 2024-09-21T10:38:28.983 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~41943040,71303168~29360128,104857600~20971520,142606336~16777216,167772160~12582912,184549376~20971520,213909504~50331648,281018368~33554432,318767104~4194304,327155712~25165824,360710144~16777216,381681664~37748736] 2024-09-21T10:38:28.983 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to 17 diff [4194304~16777216,25165824~71303168,104857600~20971520,142606336~16777216,167772160~12582912,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~20971520,360710144~16777216,381681664~12582912,398458880~16777216] 2024-09-21T10:38:28.983 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~71303168,104857600~20971520,142606336~16777216,167772160~12582912,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~20971520,360710144~16777216,381681664~12582912,398458880~16777216] 2024-09-21T10:38:28.987 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~20971520,142606336~16777216,167772160~12582912,184549376~20971520,213909504~62914560,281018368~33554432,318767104~4194304,327155712~25165824,360710144~16777216,381681664~37748736] 2024-09-21T10:38:28.987 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to 18 diff [4194304~16777216,25165824~71303168,104857600~20971520,134217728~4194304,142606336~16777216,167772160~12582912,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:28.987 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~71303168,104857600~20971520,134217728~4194304,142606336~16777216,167772160~12582912,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:28.991 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~20971520,134217728~25165824,167772160~12582912,184549376~20971520,213909504~62914560,281018368~33554432,318767104~4194304,327155712~50331648,381681664~37748736] 2024-09-21T10:38:28.991 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to 19 diff [4194304~16777216,25165824~75497472,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:28.991 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~75497472,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:28.996 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~20971520,134217728~25165824,163577856~16777216,184549376~20971520,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:28.996 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to 9 diff [4194304~8388608,33554432~8388608,46137344~4194304,92274688~4194304,113246208~4194304,197132288~8388608,281018368~4194304,369098752~4194304,381681664~4194304] 2024-09-21T10:38:28.996 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~8388608,33554432~8388608,46137344~4194304,92274688~4194304,113246208~4194304,197132288~8388608,281018368~4194304,369098752~4194304,381681664~4194304] 2024-09-21T10:38:28.997 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~8388608,33554432~16777216,92274688~8388608,113246208~4194304,197132288~8388608,281018368~4194304,369098752~4194304,381681664~4194304] 2024-09-21T10:38:28.997 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to 10 diff [4194304~8388608,33554432~8388608,46137344~4194304,54525952~8388608,88080384~8388608,109051904~16777216,197132288~8388608,213909504~16777216,255852544~4194304,281018368~4194304,293601280~8388608,327155712~4194304,369098752~4194304,381681664~4194304] 2024-09-21T10:38:28.997 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~8388608,33554432~8388608,46137344~4194304,54525952~8388608,88080384~8388608,109051904~16777216,197132288~8388608,213909504~16777216,255852544~4194304,281018368~4194304,293601280~8388608,327155712~4194304,369098752~4194304,381681664~4194304] 2024-09-21T10:38:28.998 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~8388608,33554432~16777216,54525952~8388608,88080384~12582912,109051904~16777216,197132288~8388608,213909504~16777216,255852544~8388608,281018368~4194304,293601280~8388608,327155712~8388608,369098752~4194304,381681664~4194304] 2024-09-21T10:38:28.998 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to 11 diff [4194304~12582912,25165824~37748736,88080384~8388608,109051904~16777216,176160768~4194304,197132288~8388608,213909504~16777216,255852544~4194304,281018368~4194304,293601280~8388608,327155712~4194304,339738624~8388608,369098752~4194304,381681664~4194304] 2024-09-21T10:38:28.998 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~12582912,25165824~37748736,88080384~8388608,109051904~16777216,176160768~4194304,197132288~8388608,213909504~16777216,255852544~4194304,281018368~4194304,293601280~8388608,327155712~4194304,339738624~8388608,369098752~4194304,381681664~4194304] 2024-09-21T10:38:28.999 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~37748736,88080384~12582912,109051904~16777216,176160768~4194304,197132288~8388608,213909504~20971520,255852544~8388608,281018368~4194304,293601280~8388608,327155712~8388608,339738624~12582912,369098752~8388608,381681664~4194304] 2024-09-21T10:38:29.000 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to 12 diff [4194304~16777216,25165824~37748736,79691776~4194304,88080384~8388608,109051904~16777216,167772160~12582912,197132288~8388608,213909504~20971520,243269632~8388608,255852544~4194304,281018368~4194304,293601280~8388608,327155712~4194304,339738624~8388608,369098752~4194304,381681664~8388608] 2024-09-21T10:38:29.000 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~37748736,79691776~4194304,88080384~8388608,109051904~16777216,167772160~12582912,197132288~8388608,213909504~20971520,243269632~8388608,255852544~4194304,281018368~4194304,293601280~8388608,327155712~4194304,339738624~8388608,369098752~4194304,381681664~8388608] 2024-09-21T10:38:29.002 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~37748736,79691776~4194304,88080384~12582912,109051904~16777216,167772160~12582912,197132288~8388608,213909504~25165824,243269632~20971520,281018368~4194304,293601280~8388608,327155712~8388608,339738624~12582912,369098752~8388608,381681664~12582912] 2024-09-21T10:38:29.002 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to 13 diff [4194304~16777216,25165824~41943040,79691776~4194304,88080384~8388608,109051904~16777216,150994944~4194304,167772160~12582912,197132288~8388608,213909504~20971520,243269632~8388608,255852544~4194304,281018368~4194304,289406976~12582912,327155712~4194304,339738624~8388608,369098752~8388608,381681664~12582912] 2024-09-21T10:38:29.002 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~41943040,79691776~4194304,88080384~8388608,109051904~16777216,150994944~4194304,167772160~12582912,197132288~8388608,213909504~20971520,243269632~8388608,255852544~4194304,281018368~4194304,289406976~12582912,327155712~4194304,339738624~8388608,369098752~8388608,381681664~12582912] 2024-09-21T10:38:29.004 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~41943040,79691776~4194304,88080384~12582912,109051904~16777216,150994944~8388608,167772160~12582912,197132288~8388608,213909504~25165824,243269632~20971520,281018368~4194304,289406976~16777216,327155712~8388608,339738624~12582912,369098752~8388608,381681664~16777216] 2024-09-21T10:38:29.004 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to 14 diff [4194304~16777216,25165824~41943040,79691776~4194304,88080384~8388608,104857600~20971520,142606336~4194304,150994944~4194304,167772160~12582912,197132288~8388608,213909504~25165824,243269632~8388608,255852544~4194304,281018368~20971520,327155712~8388608,339738624~8388608,369098752~8388608,381681664~12582912] 2024-09-21T10:38:29.004 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~41943040,79691776~4194304,88080384~8388608,104857600~20971520,142606336~4194304,150994944~4194304,167772160~12582912,197132288~8388608,213909504~25165824,243269632~8388608,255852544~4194304,281018368~20971520,327155712~8388608,339738624~8388608,369098752~8388608,381681664~12582912] 2024-09-21T10:38:29.007 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~41943040,79691776~4194304,88080384~12582912,104857600~20971520,142606336~16777216,167772160~12582912,197132288~8388608,213909504~25165824,243269632~20971520,281018368~25165824,327155712~25165824,369098752~8388608,381681664~16777216] 2024-09-21T10:38:29.007 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to 15 diff [4194304~16777216,25165824~41943040,75497472~8388608,88080384~8388608,104857600~20971520,142606336~4194304,150994944~4194304,167772160~12582912,184549376~8388608,197132288~8388608,213909504~46137344,281018368~20971520,306184192~4194304,327155712~8388608,339738624~8388608,360710144~16777216,381681664~12582912] 2024-09-21T10:38:29.007 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~41943040,75497472~8388608,88080384~8388608,104857600~20971520,142606336~4194304,150994944~4194304,167772160~12582912,184549376~8388608,197132288~8388608,213909504~46137344,281018368~20971520,306184192~4194304,327155712~8388608,339738624~8388608,360710144~16777216,381681664~12582912] 2024-09-21T10:38:29.011 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~41943040,75497472~8388608,88080384~12582912,104857600~20971520,142606336~16777216,167772160~12582912,184549376~8388608,197132288~8388608,213909504~50331648,281018368~33554432,327155712~25165824,360710144~16777216,381681664~16777216] 2024-09-21T10:38:29.011 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to 16 diff [4194304~16777216,25165824~41943040,71303168~12582912,88080384~8388608,104857600~20971520,142606336~4194304,150994944~8388608,167772160~12582912,184549376~8388608,197132288~8388608,213909504~46137344,281018368~29360128,327155712~8388608,339738624~8388608,360710144~16777216,381681664~12582912,398458880~16777216] 2024-09-21T10:38:29.011 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~41943040,71303168~12582912,88080384~8388608,104857600~20971520,142606336~4194304,150994944~8388608,167772160~12582912,184549376~8388608,197132288~8388608,213909504~46137344,281018368~29360128,327155712~8388608,339738624~8388608,360710144~16777216,381681664~12582912,398458880~16777216] 2024-09-21T10:38:29.014 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~41943040,71303168~12582912,88080384~12582912,104857600~20971520,142606336~16777216,167772160~12582912,184549376~8388608,197132288~8388608,213909504~50331648,281018368~33554432,327155712~25165824,360710144~16777216,381681664~37748736] 2024-09-21T10:38:29.014 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to 17 diff [4194304~16777216,25165824~58720256,88080384~8388608,104857600~20971520,142606336~16777216,167772160~12582912,184549376~8388608,197132288~8388608,213909504~46137344,264241152~8388608,281018368~33554432,327155712~20971520,360710144~16777216,381681664~12582912,398458880~16777216] 2024-09-21T10:38:29.014 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~58720256,88080384~8388608,104857600~20971520,142606336~16777216,167772160~12582912,184549376~8388608,197132288~8388608,213909504~46137344,264241152~8388608,281018368~33554432,327155712~20971520,360710144~16777216,381681664~12582912,398458880~16777216] 2024-09-21T10:38:29.018 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~58720256,88080384~12582912,104857600~20971520,142606336~16777216,167772160~12582912,184549376~8388608,197132288~8388608,213909504~62914560,281018368~33554432,327155712~25165824,360710144~16777216,381681664~37748736] 2024-09-21T10:38:29.018 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to 18 diff [4194304~16777216,25165824~58720256,88080384~8388608,104857600~20971520,134217728~4194304,142606336~16777216,167772160~12582912,184549376~8388608,197132288~8388608,213909504~46137344,264241152~8388608,281018368~33554432,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:29.018 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~58720256,88080384~8388608,104857600~20971520,134217728~4194304,142606336~16777216,167772160~12582912,184549376~8388608,197132288~8388608,213909504~46137344,264241152~8388608,281018368~33554432,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:29.022 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~58720256,88080384~12582912,104857600~20971520,134217728~25165824,167772160~12582912,184549376~8388608,197132288~8388608,213909504~62914560,281018368~33554432,327155712~50331648,381681664~37748736] 2024-09-21T10:38:29.022 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to 19 diff [4194304~16777216,25165824~58720256,88080384~12582912,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,197132288~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:29.022 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~58720256,88080384~12582912,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,197132288~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:29.027 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~58720256,88080384~12582912,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,197132288~8388608,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:29.027 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to 10 diff [54525952~8388608,88080384~4194304,109051904~16777216,213909504~16777216,255852544~4194304,293601280~8388608,327155712~4194304] 2024-09-21T10:38:29.028 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [54525952~8388608,88080384~4194304,109051904~16777216,213909504~16777216,255852544~4194304,293601280~8388608,327155712~4194304] 2024-09-21T10:38:29.028 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [54525952~8388608,88080384~8388608,109051904~16777216,213909504~16777216,255852544~8388608,293601280~8388608,327155712~8388608] 2024-09-21T10:38:29.028 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to 11 diff [8388608~8388608,25165824~8388608,37748736~8388608,50331648~12582912,88080384~4194304,109051904~16777216,176160768~4194304,213909504~16777216,255852544~4194304,293601280~8388608,327155712~4194304,339738624~8388608,369098752~4194304] 2024-09-21T10:38:29.029 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~8388608,25165824~8388608,37748736~8388608,50331648~12582912,88080384~4194304,109051904~16777216,176160768~4194304,213909504~16777216,255852544~4194304,293601280~8388608,327155712~4194304,339738624~8388608,369098752~4194304] 2024-09-21T10:38:29.029 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~8388608,37748736~25165824,88080384~8388608,109051904~16777216,176160768~4194304,213909504~20971520,255852544~8388608,293601280~8388608,327155712~8388608,339738624~12582912,369098752~8388608] 2024-09-21T10:38:29.030 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to 12 diff [8388608~12582912,25165824~8388608,37748736~8388608,50331648~12582912,79691776~4194304,88080384~4194304,109051904~16777216,167772160~12582912,213909504~20971520,243269632~8388608,255852544~4194304,293601280~8388608,327155712~4194304,339738624~8388608,369098752~4194304,381681664~8388608] 2024-09-21T10:38:29.030 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~8388608,50331648~12582912,79691776~4194304,88080384~4194304,109051904~16777216,167772160~12582912,213909504~20971520,243269632~8388608,255852544~4194304,293601280~8388608,327155712~4194304,339738624~8388608,369098752~4194304,381681664~8388608] 2024-09-21T10:38:29.031 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~37748736,79691776~4194304,88080384~8388608,109051904~16777216,167772160~12582912,213909504~25165824,243269632~20971520,293601280~8388608,327155712~8388608,339738624~12582912,369098752~8388608,381681664~12582912] 2024-09-21T10:38:29.031 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to 13 diff [8388608~12582912,25165824~8388608,37748736~29360128,79691776~4194304,88080384~4194304,109051904~16777216,150994944~4194304,167772160~12582912,213909504~20971520,243269632~8388608,255852544~4194304,289406976~12582912,327155712~4194304,339738624~8388608,369098752~4194304,381681664~12582912] 2024-09-21T10:38:29.032 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~29360128,79691776~4194304,88080384~4194304,109051904~16777216,150994944~4194304,167772160~12582912,213909504~20971520,243269632~8388608,255852544~4194304,289406976~12582912,327155712~4194304,339738624~8388608,369098752~4194304,381681664~12582912] 2024-09-21T10:38:29.033 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~41943040,79691776~4194304,88080384~8388608,109051904~16777216,150994944~8388608,167772160~12582912,213909504~25165824,243269632~20971520,289406976~16777216,327155712~8388608,339738624~12582912,369098752~8388608,381681664~16777216] 2024-09-21T10:38:29.033 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to 14 diff [8388608~12582912,25165824~8388608,37748736~29360128,79691776~4194304,88080384~4194304,104857600~20971520,142606336~4194304,150994944~4194304,167772160~12582912,213909504~25165824,243269632~8388608,255852544~4194304,285212672~16777216,327155712~8388608,339738624~8388608,369098752~8388608,381681664~12582912] 2024-09-21T10:38:29.033 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~29360128,79691776~4194304,88080384~4194304,104857600~20971520,142606336~4194304,150994944~4194304,167772160~12582912,213909504~25165824,243269632~8388608,255852544~4194304,285212672~16777216,327155712~8388608,339738624~8388608,369098752~8388608,381681664~12582912] 2024-09-21T10:38:29.035 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~41943040,79691776~4194304,88080384~8388608,104857600~20971520,142606336~16777216,167772160~12582912,213909504~25165824,243269632~20971520,285212672~20971520,327155712~25165824,369098752~8388608,381681664~16777216] 2024-09-21T10:38:29.035 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to 15 diff [8388608~12582912,25165824~8388608,37748736~29360128,75497472~8388608,88080384~4194304,104857600~20971520,142606336~4194304,150994944~4194304,167772160~12582912,184549376~8388608,213909504~46137344,285212672~16777216,306184192~4194304,327155712~8388608,339738624~8388608,360710144~16777216,381681664~12582912] 2024-09-21T10:38:29.035 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~29360128,75497472~8388608,88080384~4194304,104857600~20971520,142606336~4194304,150994944~4194304,167772160~12582912,184549376~8388608,213909504~46137344,285212672~16777216,306184192~4194304,327155712~8388608,339738624~8388608,360710144~16777216,381681664~12582912] 2024-09-21T10:38:29.038 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~41943040,75497472~8388608,88080384~8388608,104857600~20971520,142606336~16777216,167772160~12582912,184549376~8388608,213909504~50331648,285212672~29360128,327155712~25165824,360710144~16777216,381681664~16777216] 2024-09-21T10:38:29.038 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to 16 diff [8388608~12582912,25165824~8388608,37748736~29360128,71303168~12582912,88080384~4194304,104857600~20971520,142606336~4194304,150994944~8388608,167772160~12582912,184549376~8388608,213909504~46137344,285212672~25165824,327155712~8388608,339738624~8388608,360710144~16777216,381681664~12582912,398458880~16777216] 2024-09-21T10:38:29.038 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~29360128,71303168~12582912,88080384~4194304,104857600~20971520,142606336~4194304,150994944~8388608,167772160~12582912,184549376~8388608,213909504~46137344,285212672~25165824,327155712~8388608,339738624~8388608,360710144~16777216,381681664~12582912,398458880~16777216] 2024-09-21T10:38:29.041 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~41943040,71303168~12582912,88080384~8388608,104857600~20971520,142606336~16777216,167772160~12582912,184549376~8388608,213909504~50331648,285212672~29360128,327155712~25165824,360710144~16777216,381681664~37748736] 2024-09-21T10:38:29.041 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to 17 diff [8388608~12582912,25165824~8388608,37748736~46137344,88080384~4194304,104857600~20971520,142606336~16777216,167772160~12582912,184549376~8388608,213909504~46137344,264241152~8388608,285212672~29360128,327155712~20971520,360710144~16777216,381681664~12582912,398458880~16777216] 2024-09-21T10:38:29.041 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~46137344,88080384~4194304,104857600~20971520,142606336~16777216,167772160~12582912,184549376~8388608,213909504~46137344,264241152~8388608,285212672~29360128,327155712~20971520,360710144~16777216,381681664~12582912,398458880~16777216] 2024-09-21T10:38:29.044 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~58720256,88080384~8388608,104857600~20971520,142606336~16777216,167772160~12582912,184549376~8388608,213909504~62914560,285212672~29360128,327155712~25165824,360710144~16777216,381681664~37748736] 2024-09-21T10:38:29.045 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to 18 diff [8388608~12582912,25165824~8388608,37748736~46137344,88080384~4194304,104857600~20971520,134217728~4194304,142606336~16777216,167772160~12582912,184549376~8388608,213909504~46137344,264241152~8388608,281018368~33554432,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:29.045 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~46137344,88080384~4194304,104857600~20971520,134217728~4194304,142606336~16777216,167772160~12582912,184549376~8388608,213909504~46137344,264241152~8388608,281018368~33554432,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:29.049 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~58720256,88080384~8388608,104857600~20971520,134217728~25165824,167772160~12582912,184549376~8388608,213909504~62914560,281018368~33554432,327155712~50331648,381681664~37748736] 2024-09-21T10:38:29.049 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to 19 diff [8388608~12582912,25165824~8388608,37748736~46137344,88080384~4194304,96468992~4194304,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:29.049 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~46137344,88080384~4194304,96468992~4194304,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:29.053 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~58720256,88080384~12582912,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:29.053 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to 11 diff [8388608~8388608,25165824~8388608,37748736~8388608,50331648~8388608,176160768~4194304,226492416~4194304,339738624~8388608,369098752~4194304] 2024-09-21T10:38:29.053 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~8388608,25165824~8388608,37748736~8388608,50331648~8388608,176160768~4194304,226492416~4194304,339738624~8388608,369098752~4194304] 2024-09-21T10:38:29.054 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~8388608,37748736~20971520,176160768~4194304,226492416~8388608,339738624~12582912,369098752~8388608] 2024-09-21T10:38:29.054 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to 12 diff [8388608~12582912,25165824~8388608,37748736~8388608,50331648~12582912,79691776~4194304,167772160~12582912,226492416~8388608,243269632~8388608,339738624~8388608,369098752~4194304,381681664~8388608] 2024-09-21T10:38:29.055 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~8388608,50331648~12582912,79691776~4194304,167772160~12582912,226492416~8388608,243269632~8388608,339738624~8388608,369098752~4194304,381681664~8388608] 2024-09-21T10:38:29.056 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~37748736,79691776~4194304,167772160~12582912,226492416~12582912,243269632~12582912,339738624~12582912,369098752~8388608,381681664~12582912] 2024-09-21T10:38:29.056 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to 13 diff [8388608~12582912,25165824~8388608,37748736~29360128,79691776~4194304,150994944~4194304,167772160~12582912,226492416~8388608,243269632~8388608,289406976~12582912,339738624~8388608,369098752~4194304,381681664~12582912] 2024-09-21T10:38:29.056 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~29360128,79691776~4194304,150994944~4194304,167772160~12582912,226492416~8388608,243269632~8388608,289406976~12582912,339738624~8388608,369098752~4194304,381681664~12582912] 2024-09-21T10:38:29.058 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~41943040,79691776~4194304,150994944~8388608,167772160~12582912,226492416~12582912,243269632~12582912,289406976~16777216,339738624~12582912,369098752~8388608,381681664~16777216] 2024-09-21T10:38:29.058 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to 14 diff [8388608~12582912,25165824~8388608,37748736~29360128,79691776~4194304,104857600~4194304,142606336~4194304,150994944~4194304,167772160~12582912,213909504~4194304,226492416~8388608,243269632~8388608,285212672~16777216,327155712~8388608,339738624~8388608,369098752~8388608,381681664~12582912] 2024-09-21T10:38:29.058 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~29360128,79691776~4194304,104857600~4194304,142606336~4194304,150994944~4194304,167772160~12582912,213909504~4194304,226492416~8388608,243269632~8388608,285212672~16777216,327155712~8388608,339738624~8388608,369098752~8388608,381681664~12582912] 2024-09-21T10:38:29.060 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~41943040,79691776~4194304,104857600~4194304,142606336~16777216,167772160~12582912,213909504~4194304,226492416~12582912,243269632~12582912,285212672~20971520,327155712~25165824,369098752~8388608,381681664~16777216] 2024-09-21T10:38:29.060 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to 15 diff [8388608~12582912,25165824~8388608,37748736~29360128,75497472~8388608,104857600~8388608,142606336~4194304,150994944~4194304,167772160~12582912,184549376~8388608,213909504~4194304,226492416~8388608,239075328~16777216,285212672~16777216,306184192~4194304,327155712~8388608,339738624~8388608,360710144~16777216,381681664~12582912] 2024-09-21T10:38:29.060 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~29360128,75497472~8388608,104857600~8388608,142606336~4194304,150994944~4194304,167772160~12582912,184549376~8388608,213909504~4194304,226492416~8388608,239075328~16777216,285212672~16777216,306184192~4194304,327155712~8388608,339738624~8388608,360710144~16777216,381681664~12582912] 2024-09-21T10:38:29.063 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~41943040,75497472~8388608,104857600~12582912,142606336~16777216,167772160~12582912,184549376~8388608,213909504~4194304,226492416~29360128,285212672~29360128,327155712~25165824,360710144~16777216,381681664~16777216] 2024-09-21T10:38:29.063 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to 16 diff [8388608~12582912,25165824~8388608,37748736~29360128,71303168~12582912,104857600~8388608,117440512~4194304,142606336~4194304,150994944~8388608,167772160~12582912,184549376~8388608,213909504~4194304,226492416~8388608,239075328~16777216,285212672~25165824,327155712~8388608,339738624~8388608,360710144~16777216,381681664~12582912,398458880~16777216] 2024-09-21T10:38:29.063 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~29360128,71303168~12582912,104857600~8388608,117440512~4194304,142606336~4194304,150994944~8388608,167772160~12582912,184549376~8388608,213909504~4194304,226492416~8388608,239075328~16777216,285212672~25165824,327155712~8388608,339738624~8388608,360710144~16777216,381681664~12582912,398458880~16777216] 2024-09-21T10:38:29.066 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~41943040,71303168~12582912,104857600~16777216,142606336~16777216,167772160~12582912,184549376~8388608,213909504~4194304,226492416~29360128,285212672~29360128,327155712~25165824,360710144~16777216,381681664~37748736] 2024-09-21T10:38:29.066 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to 17 diff [8388608~12582912,25165824~8388608,37748736~46137344,104857600~8388608,117440512~4194304,142606336~16777216,167772160~12582912,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,285212672~29360128,327155712~20971520,360710144~16777216,381681664~12582912,398458880~16777216] 2024-09-21T10:38:29.066 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~46137344,104857600~8388608,117440512~4194304,142606336~16777216,167772160~12582912,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,285212672~29360128,327155712~20971520,360710144~16777216,381681664~12582912,398458880~16777216] 2024-09-21T10:38:29.070 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~58720256,104857600~16777216,142606336~16777216,167772160~12582912,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,285212672~29360128,327155712~25165824,360710144~16777216,381681664~37748736] 2024-09-21T10:38:29.070 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to 18 diff [8388608~12582912,25165824~8388608,37748736~46137344,104857600~8388608,117440512~4194304,134217728~4194304,142606336~16777216,167772160~12582912,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:29.071 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~46137344,104857600~8388608,117440512~4194304,134217728~4194304,142606336~16777216,167772160~12582912,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:29.074 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~58720256,104857600~16777216,134217728~25165824,167772160~12582912,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~33554432,327155712~50331648,381681664~37748736] 2024-09-21T10:38:29.074 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to 19 diff [8388608~12582912,25165824~8388608,37748736~46137344,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:29.074 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~46137344,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:29.077 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~58720256,96468992~4194304,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:29.077 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to 12 diff [16777216~4194304,25165824~8388608,58720256~4194304,79691776~4194304,167772160~12582912,226492416~8388608,243269632~8388608,381681664~8388608] 2024-09-21T10:38:29.077 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [16777216~4194304,25165824~8388608,58720256~4194304,79691776~4194304,167772160~12582912,226492416~8388608,243269632~8388608,381681664~8388608] 2024-09-21T10:38:29.078 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [16777216~4194304,25165824~12582912,58720256~4194304,79691776~4194304,167772160~12582912,226492416~12582912,243269632~12582912,381681664~12582912] 2024-09-21T10:38:29.078 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to 13 diff [16777216~4194304,25165824~8388608,46137344~20971520,79691776~4194304,150994944~4194304,167772160~12582912,226492416~8388608,243269632~8388608,289406976~12582912,369098752~4194304,381681664~12582912] 2024-09-21T10:38:29.078 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [16777216~4194304,25165824~8388608,46137344~20971520,79691776~4194304,150994944~4194304,167772160~12582912,226492416~8388608,243269632~8388608,289406976~12582912,369098752~4194304,381681664~12582912] 2024-09-21T10:38:29.079 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [16777216~4194304,25165824~12582912,46137344~20971520,79691776~4194304,150994944~8388608,167772160~12582912,226492416~12582912,243269632~12582912,289406976~16777216,369098752~4194304,381681664~16777216] 2024-09-21T10:38:29.079 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to 14 diff [16777216~4194304,25165824~8388608,41943040~25165824,79691776~4194304,104857600~4194304,142606336~4194304,150994944~4194304,167772160~12582912,213909504~4194304,226492416~8388608,243269632~8388608,285212672~16777216,327155712~8388608,343932928~4194304,369098752~8388608,381681664~12582912] 2024-09-21T10:38:29.079 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [16777216~4194304,25165824~8388608,41943040~25165824,79691776~4194304,104857600~4194304,142606336~4194304,150994944~4194304,167772160~12582912,213909504~4194304,226492416~8388608,243269632~8388608,285212672~16777216,327155712~8388608,343932928~4194304,369098752~8388608,381681664~12582912] 2024-09-21T10:38:29.081 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [16777216~4194304,25165824~12582912,41943040~25165824,79691776~4194304,104857600~4194304,142606336~16777216,167772160~12582912,213909504~4194304,226492416~12582912,243269632~12582912,285212672~20971520,327155712~12582912,343932928~4194304,369098752~8388608,381681664~16777216] 2024-09-21T10:38:29.081 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to 15 diff [16777216~4194304,25165824~8388608,41943040~25165824,75497472~8388608,104857600~8388608,142606336~4194304,150994944~4194304,167772160~12582912,184549376~8388608,213909504~4194304,226492416~8388608,239075328~16777216,285212672~16777216,306184192~4194304,327155712~8388608,343932928~4194304,360710144~16777216,381681664~12582912] 2024-09-21T10:38:29.081 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [16777216~4194304,25165824~8388608,41943040~25165824,75497472~8388608,104857600~8388608,142606336~4194304,150994944~4194304,167772160~12582912,184549376~8388608,213909504~4194304,226492416~8388608,239075328~16777216,285212672~16777216,306184192~4194304,327155712~8388608,343932928~4194304,360710144~16777216,381681664~12582912] 2024-09-21T10:38:29.083 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [16777216~4194304,25165824~12582912,41943040~25165824,75497472~8388608,104857600~12582912,142606336~16777216,167772160~12582912,184549376~8388608,213909504~4194304,226492416~29360128,285212672~29360128,327155712~12582912,343932928~4194304,360710144~16777216,381681664~16777216] 2024-09-21T10:38:29.083 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to 16 diff [16777216~4194304,25165824~8388608,41943040~25165824,71303168~12582912,104857600~8388608,117440512~4194304,142606336~4194304,150994944~8388608,167772160~12582912,184549376~8388608,213909504~4194304,226492416~8388608,239075328~16777216,285212672~25165824,327155712~8388608,343932928~4194304,360710144~16777216,381681664~12582912,398458880~16777216] 2024-09-21T10:38:29.083 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [16777216~4194304,25165824~8388608,41943040~25165824,71303168~12582912,104857600~8388608,117440512~4194304,142606336~4194304,150994944~8388608,167772160~12582912,184549376~8388608,213909504~4194304,226492416~8388608,239075328~16777216,285212672~25165824,327155712~8388608,343932928~4194304,360710144~16777216,381681664~12582912,398458880~16777216] 2024-09-21T10:38:29.085 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [16777216~4194304,25165824~12582912,41943040~25165824,71303168~12582912,104857600~16777216,142606336~16777216,167772160~12582912,184549376~8388608,213909504~4194304,226492416~29360128,285212672~29360128,327155712~12582912,343932928~4194304,360710144~16777216,381681664~37748736] 2024-09-21T10:38:29.085 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to 17 diff [16777216~4194304,25165824~8388608,41943040~41943040,104857600~8388608,117440512~4194304,142606336~16777216,167772160~12582912,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,285212672~29360128,327155712~12582912,343932928~4194304,360710144~16777216,381681664~12582912,398458880~16777216] 2024-09-21T10:38:29.085 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [16777216~4194304,25165824~8388608,41943040~41943040,104857600~8388608,117440512~4194304,142606336~16777216,167772160~12582912,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,285212672~29360128,327155712~12582912,343932928~4194304,360710144~16777216,381681664~12582912,398458880~16777216] 2024-09-21T10:38:29.088 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [16777216~4194304,25165824~12582912,41943040~41943040,104857600~16777216,142606336~16777216,167772160~12582912,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,285212672~29360128,327155712~12582912,343932928~4194304,360710144~16777216,381681664~37748736] 2024-09-21T10:38:29.089 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to 18 diff [16777216~4194304,25165824~8388608,41943040~41943040,104857600~8388608,117440512~4194304,134217728~4194304,142606336~16777216,167772160~12582912,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,327155712~12582912,343932928~33554432,381681664~20971520,406847488~8388608] 2024-09-21T10:38:29.089 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [16777216~4194304,25165824~8388608,41943040~41943040,104857600~8388608,117440512~4194304,134217728~4194304,142606336~16777216,167772160~12582912,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,327155712~12582912,343932928~33554432,381681664~20971520,406847488~8388608] 2024-09-21T10:38:29.091 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [16777216~4194304,25165824~12582912,41943040~41943040,104857600~16777216,134217728~25165824,167772160~12582912,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~33554432,327155712~12582912,343932928~33554432,381681664~37748736] 2024-09-21T10:38:29.092 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to 19 diff [16777216~4194304,25165824~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,381681664~20971520,406847488~8388608] 2024-09-21T10:38:29.092 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [16777216~4194304,25165824~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,381681664~20971520,406847488~8388608] 2024-09-21T10:38:29.095 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [16777216~4194304,25165824~12582912,41943040~41943040,96468992~4194304,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~33554432,318767104~20971520,343932928~33554432,381681664~37748736] 2024-09-21T10:38:29.095 INFO:tasks.workunit.client.0.smithi070.stdout:from 12 to 13 diff [46137344~20971520,150994944~4194304,289406976~12582912,369098752~4194304,385875968~8388608] 2024-09-21T10:38:29.095 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [46137344~20971520,150994944~4194304,289406976~12582912,369098752~4194304,385875968~8388608] 2024-09-21T10:38:29.096 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [46137344~20971520,150994944~8388608,289406976~16777216,369098752~4194304,385875968~12582912] 2024-09-21T10:38:29.096 INFO:tasks.workunit.client.0.smithi070.stdout:from 12 to 14 diff [41943040~25165824,104857600~4194304,142606336~4194304,150994944~4194304,213909504~4194304,230686720~8388608,285212672~16777216,327155712~8388608,343932928~4194304,369098752~8388608,385875968~8388608] 2024-09-21T10:38:29.096 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [41943040~25165824,104857600~4194304,142606336~4194304,150994944~4194304,213909504~4194304,230686720~8388608,285212672~16777216,327155712~8388608,343932928~4194304,369098752~8388608,385875968~8388608] 2024-09-21T10:38:29.097 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [41943040~25165824,104857600~4194304,142606336~16777216,213909504~4194304,230686720~8388608,285212672~20971520,327155712~12582912,343932928~4194304,369098752~8388608,385875968~12582912] 2024-09-21T10:38:29.097 INFO:tasks.workunit.client.0.smithi070.stdout:from 12 to 15 diff [29360128~8388608,41943040~25165824,75497472~4194304,104857600~8388608,142606336~4194304,150994944~4194304,184549376~8388608,213909504~4194304,226492416~8388608,239075328~4194304,251658240~4194304,285212672~16777216,306184192~4194304,327155712~8388608,343932928~4194304,360710144~16777216,385875968~8388608] 2024-09-21T10:38:29.097 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,41943040~25165824,75497472~4194304,104857600~8388608,142606336~4194304,150994944~4194304,184549376~8388608,213909504~4194304,226492416~8388608,239075328~4194304,251658240~4194304,285212672~16777216,306184192~4194304,327155712~8388608,343932928~4194304,360710144~16777216,385875968~8388608] 2024-09-21T10:38:29.098 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,41943040~25165824,75497472~8388608,104857600~12582912,142606336~16777216,184549376~8388608,213909504~4194304,226492416~20971520,251658240~4194304,285212672~29360128,327155712~12582912,343932928~4194304,360710144~16777216,385875968~12582912] 2024-09-21T10:38:29.098 INFO:tasks.workunit.client.0.smithi070.stdout:from 12 to 16 diff [29360128~8388608,41943040~25165824,71303168~8388608,104857600~8388608,117440512~4194304,142606336~4194304,150994944~8388608,167772160~4194304,184549376~8388608,213909504~4194304,226492416~8388608,239075328~4194304,251658240~4194304,285212672~25165824,327155712~8388608,343932928~4194304,360710144~16777216,385875968~8388608,398458880~16777216] 2024-09-21T10:38:29.099 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,41943040~25165824,71303168~8388608,104857600~8388608,117440512~4194304,142606336~4194304,150994944~8388608,167772160~4194304,184549376~8388608,213909504~4194304,226492416~8388608,239075328~4194304,251658240~4194304,285212672~25165824,327155712~8388608,343932928~4194304,360710144~16777216,385875968~8388608,398458880~16777216] 2024-09-21T10:38:29.100 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,41943040~25165824,71303168~12582912,104857600~16777216,142606336~16777216,167772160~4194304,184549376~8388608,213909504~4194304,226492416~20971520,251658240~4194304,285212672~29360128,327155712~12582912,343932928~4194304,360710144~16777216,385875968~33554432] 2024-09-21T10:38:29.101 INFO:tasks.workunit.client.0.smithi070.stdout:from 12 to 17 diff [29360128~8388608,41943040~41943040,104857600~8388608,117440512~4194304,142606336~16777216,167772160~4194304,184549376~8388608,213909504~4194304,226492416~20971520,251658240~4194304,260046848~12582912,285212672~29360128,327155712~12582912,343932928~4194304,360710144~16777216,385875968~8388608,398458880~16777216] 2024-09-21T10:38:29.101 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,41943040~41943040,104857600~8388608,117440512~4194304,142606336~16777216,167772160~4194304,184549376~8388608,213909504~4194304,226492416~20971520,251658240~4194304,260046848~12582912,285212672~29360128,327155712~12582912,343932928~4194304,360710144~16777216,385875968~8388608,398458880~16777216] 2024-09-21T10:38:29.103 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,41943040~41943040,104857600~16777216,142606336~16777216,167772160~4194304,184549376~8388608,213909504~4194304,226492416~20971520,251658240~4194304,260046848~16777216,285212672~29360128,327155712~12582912,343932928~4194304,360710144~16777216,385875968~33554432] 2024-09-21T10:38:29.103 INFO:tasks.workunit.client.0.smithi070.stdout:from 12 to 18 diff [29360128~8388608,41943040~41943040,104857600~8388608,117440512~4194304,134217728~4194304,142606336~16777216,167772160~4194304,184549376~8388608,213909504~4194304,226492416~20971520,251658240~4194304,260046848~12582912,281018368~33554432,327155712~12582912,343932928~33554432,385875968~16777216,406847488~8388608] 2024-09-21T10:38:29.103 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,41943040~41943040,104857600~8388608,117440512~4194304,134217728~4194304,142606336~16777216,167772160~4194304,184549376~8388608,213909504~4194304,226492416~20971520,251658240~4194304,260046848~12582912,281018368~33554432,327155712~12582912,343932928~33554432,385875968~16777216,406847488~8388608] 2024-09-21T10:38:29.106 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,41943040~41943040,104857600~16777216,134217728~25165824,167772160~4194304,184549376~8388608,213909504~4194304,226492416~20971520,251658240~4194304,260046848~16777216,281018368~33554432,327155712~12582912,343932928~33554432,385875968~33554432] 2024-09-21T10:38:29.106 INFO:tasks.workunit.client.0.smithi070.stdout:from 12 to 19 diff [29360128~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,385875968~16777216,406847488~8388608] 2024-09-21T10:38:29.106 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,385875968~16777216,406847488~8388608] 2024-09-21T10:38:29.109 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,41943040~41943040,96468992~4194304,104857600~20971520,134217728~25165824,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~33554432,318767104~20971520,343932928~33554432,385875968~33554432] 2024-09-21T10:38:29.109 INFO:tasks.workunit.client.0.smithi070.stdout:from 13 to 14 diff [41943040~8388608,104857600~4194304,142606336~4194304,213909504~4194304,230686720~8388608,285212672~4194304,327155712~8388608,343932928~4194304,369098752~8388608] 2024-09-21T10:38:29.109 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [41943040~8388608,104857600~4194304,142606336~4194304,213909504~4194304,230686720~8388608,285212672~4194304,327155712~8388608,343932928~4194304,369098752~8388608] 2024-09-21T10:38:29.110 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [41943040~12582912,104857600~4194304,142606336~8388608,213909504~4194304,230686720~8388608,285212672~8388608,327155712~12582912,343932928~4194304,369098752~8388608] 2024-09-21T10:38:29.110 INFO:tasks.workunit.client.0.smithi070.stdout:from 13 to 15 diff [29360128~8388608,41943040~8388608,75497472~4194304,104857600~8388608,142606336~4194304,184549376~8388608,213909504~4194304,226492416~8388608,239075328~4194304,251658240~4194304,285212672~4194304,306184192~4194304,327155712~8388608,343932928~4194304,360710144~16777216] 2024-09-21T10:38:29.110 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,41943040~8388608,75497472~4194304,104857600~8388608,142606336~4194304,184549376~8388608,213909504~4194304,226492416~8388608,239075328~4194304,251658240~4194304,285212672~4194304,306184192~4194304,327155712~8388608,343932928~4194304,360710144~16777216] 2024-09-21T10:38:29.112 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,41943040~12582912,75497472~8388608,104857600~12582912,142606336~8388608,184549376~8388608,213909504~4194304,226492416~20971520,251658240~4194304,285212672~8388608,306184192~8388608,327155712~12582912,343932928~4194304,360710144~16777216] 2024-09-21T10:38:29.112 INFO:tasks.workunit.client.0.smithi070.stdout:from 13 to 16 diff [29360128~8388608,41943040~16777216,71303168~8388608,104857600~8388608,117440512~4194304,142606336~4194304,150994944~8388608,167772160~4194304,184549376~8388608,213909504~4194304,226492416~8388608,239075328~4194304,251658240~4194304,285212672~8388608,301989888~8388608,327155712~8388608,343932928~4194304,360710144~16777216,398458880~16777216] 2024-09-21T10:38:29.112 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,41943040~16777216,71303168~8388608,104857600~8388608,117440512~4194304,142606336~4194304,150994944~8388608,167772160~4194304,184549376~8388608,213909504~4194304,226492416~8388608,239075328~4194304,251658240~4194304,285212672~8388608,301989888~8388608,327155712~8388608,343932928~4194304,360710144~16777216,398458880~16777216] 2024-09-21T10:38:29.113 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,41943040~20971520,71303168~12582912,104857600~16777216,142606336~16777216,167772160~4194304,184549376~8388608,213909504~4194304,226492416~20971520,251658240~4194304,285212672~8388608,301989888~12582912,327155712~12582912,343932928~4194304,360710144~16777216,398458880~20971520] 2024-09-21T10:38:29.113 INFO:tasks.workunit.client.0.smithi070.stdout:from 13 to 17 diff [29360128~8388608,41943040~16777216,67108864~16777216,104857600~8388608,117440512~4194304,142606336~12582912,167772160~4194304,184549376~8388608,213909504~4194304,226492416~20971520,251658240~4194304,260046848~12582912,285212672~8388608,301989888~12582912,327155712~12582912,343932928~4194304,360710144~16777216,398458880~16777216] 2024-09-21T10:38:29.113 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,41943040~16777216,67108864~16777216,104857600~8388608,117440512~4194304,142606336~12582912,167772160~4194304,184549376~8388608,213909504~4194304,226492416~20971520,251658240~4194304,260046848~12582912,285212672~8388608,301989888~12582912,327155712~12582912,343932928~4194304,360710144~16777216,398458880~16777216] 2024-09-21T10:38:29.115 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,41943040~20971520,67108864~16777216,104857600~16777216,142606336~16777216,167772160~4194304,184549376~8388608,213909504~4194304,226492416~20971520,251658240~4194304,260046848~16777216,285212672~8388608,301989888~12582912,327155712~12582912,343932928~4194304,360710144~16777216,398458880~20971520] 2024-09-21T10:38:29.115 INFO:tasks.workunit.client.0.smithi070.stdout:from 13 to 18 diff [29360128~8388608,41943040~16777216,67108864~16777216,104857600~8388608,117440512~4194304,134217728~4194304,142606336~12582912,167772160~4194304,184549376~8388608,213909504~4194304,226492416~20971520,251658240~4194304,260046848~12582912,281018368~12582912,301989888~12582912,327155712~12582912,343932928~33554432,394264576~8388608,406847488~8388608] 2024-09-21T10:38:29.115 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,41943040~16777216,67108864~16777216,104857600~8388608,117440512~4194304,134217728~4194304,142606336~12582912,167772160~4194304,184549376~8388608,213909504~4194304,226492416~20971520,251658240~4194304,260046848~12582912,281018368~12582912,301989888~12582912,327155712~12582912,343932928~33554432,394264576~8388608,406847488~8388608] 2024-09-21T10:38:29.118 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,41943040~20971520,67108864~16777216,104857600~16777216,134217728~25165824,167772160~4194304,184549376~8388608,213909504~4194304,226492416~20971520,251658240~4194304,260046848~16777216,281018368~12582912,301989888~12582912,327155712~12582912,343932928~33554432,394264576~25165824] 2024-09-21T10:38:29.118 INFO:tasks.workunit.client.0.smithi070.stdout:from 13 to 19 diff [29360128~8388608,41943040~16777216,67108864~16777216,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~12582912,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~12582912,343932928~33554432,390070272~12582912,406847488~8388608] 2024-09-21T10:38:29.118 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,41943040~16777216,67108864~16777216,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~12582912,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~12582912,343932928~33554432,390070272~12582912,406847488~8388608] 2024-09-21T10:38:29.121 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,41943040~20971520,67108864~16777216,96468992~4194304,104857600~20971520,134217728~25165824,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~12582912,301989888~12582912,318767104~20971520,343932928~33554432,390070272~29360128] 2024-09-21T10:38:29.121 INFO:tasks.workunit.client.0.smithi070.stdout:from 14 to 15 diff [29360128~8388608,75497472~4194304,109051904~4194304,184549376~8388608,226492416~8388608,239075328~4194304,251658240~4194304,306184192~4194304,360710144~12582912] 2024-09-21T10:38:29.121 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,75497472~4194304,109051904~4194304,184549376~8388608,226492416~8388608,239075328~4194304,251658240~4194304,306184192~4194304,360710144~12582912] 2024-09-21T10:38:29.122 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,75497472~8388608,109051904~8388608,184549376~8388608,226492416~8388608,239075328~8388608,251658240~4194304,306184192~8388608,360710144~12582912] 2024-09-21T10:38:29.122 INFO:tasks.workunit.client.0.smithi070.stdout:from 14 to 16 diff [29360128~8388608,50331648~8388608,71303168~8388608,109051904~4194304,117440512~4194304,150994944~8388608,167772160~4194304,184549376~8388608,226492416~8388608,239075328~4194304,251658240~4194304,289406976~4194304,301989888~8388608,360710144~12582912,398458880~16777216] 2024-09-21T10:38:29.122 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,50331648~8388608,71303168~8388608,109051904~4194304,117440512~4194304,150994944~8388608,167772160~4194304,184549376~8388608,226492416~8388608,239075328~4194304,251658240~4194304,289406976~4194304,301989888~8388608,360710144~12582912,398458880~16777216] 2024-09-21T10:38:29.123 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,50331648~12582912,71303168~12582912,109051904~12582912,150994944~8388608,167772160~4194304,184549376~8388608,226492416~8388608,239075328~8388608,251658240~4194304,289406976~4194304,301989888~12582912,360710144~12582912,398458880~20971520] 2024-09-21T10:38:29.123 INFO:tasks.workunit.client.0.smithi070.stdout:from 14 to 17 diff [29360128~8388608,50331648~8388608,67108864~16777216,109051904~4194304,117440512~4194304,146800640~8388608,167772160~4194304,184549376~8388608,226492416~20971520,251658240~4194304,260046848~12582912,289406976~4194304,301989888~12582912,327155712~8388608,360710144~16777216,398458880~16777216] 2024-09-21T10:38:29.123 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,50331648~8388608,67108864~16777216,109051904~4194304,117440512~4194304,146800640~8388608,167772160~4194304,184549376~8388608,226492416~20971520,251658240~4194304,260046848~12582912,289406976~4194304,301989888~12582912,327155712~8388608,360710144~16777216,398458880~16777216] 2024-09-21T10:38:29.125 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,50331648~12582912,67108864~16777216,109051904~12582912,146800640~12582912,167772160~4194304,184549376~8388608,226492416~20971520,251658240~4194304,260046848~16777216,289406976~4194304,301989888~12582912,327155712~12582912,360710144~16777216,398458880~20971520] 2024-09-21T10:38:29.125 INFO:tasks.workunit.client.0.smithi070.stdout:from 14 to 18 diff [29360128~8388608,50331648~8388608,67108864~16777216,109051904~4194304,117440512~4194304,134217728~4194304,146800640~8388608,167772160~4194304,184549376~8388608,226492416~20971520,251658240~4194304,260046848~12582912,281018368~12582912,301989888~12582912,327155712~8388608,348127232~29360128,394264576~8388608,406847488~8388608] 2024-09-21T10:38:29.125 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,50331648~8388608,67108864~16777216,109051904~4194304,117440512~4194304,134217728~4194304,146800640~8388608,167772160~4194304,184549376~8388608,226492416~20971520,251658240~4194304,260046848~12582912,281018368~12582912,301989888~12582912,327155712~8388608,348127232~29360128,394264576~8388608,406847488~8388608] 2024-09-21T10:38:29.126 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,50331648~12582912,67108864~16777216,109051904~12582912,134217728~8388608,146800640~12582912,167772160~4194304,184549376~8388608,226492416~20971520,251658240~4194304,260046848~16777216,281018368~12582912,301989888~12582912,327155712~12582912,348127232~29360128,394264576~25165824] 2024-09-21T10:38:29.127 INFO:tasks.workunit.client.0.smithi070.stdout:from 14 to 19 diff [29360128~8388608,50331648~8388608,67108864~16777216,96468992~4194304,109051904~4194304,117440512~8388608,134217728~4194304,146800640~8388608,163577856~8388608,184549376~8388608,226492416~29360128,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~8388608,348127232~29360128,390070272~12582912,406847488~8388608] 2024-09-21T10:38:29.127 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,50331648~8388608,67108864~16777216,96468992~4194304,109051904~4194304,117440512~8388608,134217728~4194304,146800640~8388608,163577856~8388608,184549376~8388608,226492416~29360128,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~8388608,348127232~29360128,390070272~12582912,406847488~8388608] 2024-09-21T10:38:29.129 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,50331648~12582912,67108864~16777216,96468992~4194304,109051904~16777216,134217728~8388608,146800640~12582912,163577856~8388608,184549376~8388608,226492416~29360128,260046848~16777216,281018368~12582912,301989888~12582912,318767104~20971520,348127232~29360128,390070272~29360128] 2024-09-21T10:38:29.129 INFO:tasks.workunit.client.0.smithi070.stdout:from 15 to 16 diff [50331648~8388608,71303168~8388608,117440512~4194304,150994944~8388608,167772160~4194304,289406976~4194304,301989888~8388608,360710144~4194304,398458880~16777216] 2024-09-21T10:38:29.130 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [50331648~8388608,71303168~8388608,117440512~4194304,150994944~8388608,167772160~4194304,289406976~4194304,301989888~8388608,360710144~4194304,398458880~16777216] 2024-09-21T10:38:29.130 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [50331648~12582912,71303168~12582912,117440512~4194304,150994944~8388608,167772160~4194304,289406976~4194304,301989888~12582912,360710144~4194304,398458880~20971520] 2024-09-21T10:38:29.130 INFO:tasks.workunit.client.0.smithi070.stdout:from 15 to 17 diff [50331648~8388608,67108864~16777216,117440512~4194304,146800640~8388608,167772160~4194304,234881024~8388608,260046848~12582912,289406976~4194304,301989888~12582912,327155712~8388608,360710144~16777216,398458880~16777216] 2024-09-21T10:38:29.130 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [50331648~8388608,67108864~16777216,117440512~4194304,146800640~8388608,167772160~4194304,234881024~8388608,260046848~12582912,289406976~4194304,301989888~12582912,327155712~8388608,360710144~16777216,398458880~16777216] 2024-09-21T10:38:29.131 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [50331648~12582912,67108864~16777216,117440512~4194304,146800640~12582912,167772160~4194304,234881024~12582912,260046848~16777216,289406976~4194304,301989888~12582912,327155712~12582912,360710144~16777216,398458880~20971520] 2024-09-21T10:38:29.132 INFO:tasks.workunit.client.0.smithi070.stdout:from 15 to 18 diff [50331648~8388608,67108864~16777216,117440512~4194304,134217728~4194304,146800640~8388608,167772160~4194304,234881024~8388608,260046848~12582912,281018368~12582912,301989888~12582912,327155712~8388608,348127232~29360128,394264576~8388608,406847488~8388608] 2024-09-21T10:38:29.132 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [50331648~8388608,67108864~16777216,117440512~4194304,134217728~4194304,146800640~8388608,167772160~4194304,234881024~8388608,260046848~12582912,281018368~12582912,301989888~12582912,327155712~8388608,348127232~29360128,394264576~8388608,406847488~8388608] 2024-09-21T10:38:29.133 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [50331648~12582912,67108864~16777216,117440512~4194304,134217728~8388608,146800640~12582912,167772160~4194304,234881024~12582912,260046848~16777216,281018368~12582912,301989888~12582912,327155712~12582912,348127232~29360128,394264576~25165824] 2024-09-21T10:38:29.133 INFO:tasks.workunit.client.0.smithi070.stdout:from 15 to 19 diff [50331648~8388608,67108864~16777216,96468992~4194304,117440512~8388608,134217728~4194304,146800640~8388608,163577856~8388608,188743680~4194304,234881024~16777216,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~8388608,348127232~29360128,390070272~12582912,406847488~8388608] 2024-09-21T10:38:29.133 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [50331648~8388608,67108864~16777216,96468992~4194304,117440512~8388608,134217728~4194304,146800640~8388608,163577856~8388608,188743680~4194304,234881024~16777216,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~8388608,348127232~29360128,390070272~12582912,406847488~8388608] 2024-09-21T10:38:29.135 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [50331648~12582912,67108864~16777216,96468992~4194304,117440512~8388608,134217728~8388608,146800640~12582912,163577856~8388608,188743680~4194304,234881024~20971520,260046848~16777216,281018368~12582912,301989888~12582912,318767104~20971520,348127232~29360128,390070272~29360128] 2024-09-21T10:38:29.135 INFO:tasks.workunit.client.0.smithi070.stdout:from 16 to 17 diff [67108864~8388608,146800640~8388608,234881024~8388608,260046848~12582912,301989888~8388608,327155712~8388608,364904448~12582912] 2024-09-21T10:38:29.135 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [67108864~8388608,146800640~8388608,234881024~8388608,260046848~12582912,301989888~8388608,327155712~8388608,364904448~12582912] 2024-09-21T10:38:29.136 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [67108864~12582912,146800640~8388608,234881024~12582912,260046848~16777216,301989888~12582912,327155712~12582912,364904448~12582912] 2024-09-21T10:38:29.136 INFO:tasks.workunit.client.0.smithi070.stdout:from 16 to 18 diff [67108864~12582912,134217728~4194304,146800640~8388608,234881024~8388608,260046848~12582912,281018368~12582912,301989888~8388608,327155712~8388608,348127232~12582912,364904448~12582912,394264576~4194304] 2024-09-21T10:38:29.136 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [67108864~12582912,134217728~4194304,146800640~8388608,234881024~8388608,260046848~12582912,281018368~12582912,301989888~8388608,327155712~8388608,348127232~12582912,364904448~12582912,394264576~4194304] 2024-09-21T10:38:29.137 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [67108864~12582912,134217728~8388608,146800640~8388608,234881024~12582912,260046848~16777216,281018368~12582912,301989888~12582912,327155712~12582912,348127232~29360128,394264576~8388608] 2024-09-21T10:38:29.137 INFO:tasks.workunit.client.0.smithi070.stdout:from 16 to 19 diff [67108864~12582912,96468992~4194304,117440512~8388608,134217728~4194304,146800640~8388608,163577856~8388608,188743680~4194304,234881024~16777216,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~8388608,348127232~12582912,364904448~12582912,390070272~12582912] 2024-09-21T10:38:29.137 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [67108864~12582912,96468992~4194304,117440512~8388608,134217728~4194304,146800640~8388608,163577856~8388608,188743680~4194304,234881024~16777216,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~8388608,348127232~12582912,364904448~12582912,390070272~12582912] 2024-09-21T10:38:29.139 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [67108864~12582912,96468992~4194304,117440512~8388608,134217728~8388608,146800640~8388608,163577856~8388608,188743680~4194304,234881024~20971520,260046848~16777216,281018368~12582912,301989888~12582912,318767104~20971520,348127232~29360128,390070272~16777216] 2024-09-21T10:38:29.139 INFO:tasks.workunit.client.0.smithi070.stdout:from 17 to 18 diff [67108864~12582912,134217728~4194304,239075328~4194304,281018368~12582912,348127232~12582912,394264576~4194304] 2024-09-21T10:38:29.139 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [67108864~12582912,134217728~4194304,239075328~4194304,281018368~12582912,348127232~12582912,394264576~4194304] 2024-09-21T10:38:29.140 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [67108864~12582912,134217728~8388608,239075328~4194304,281018368~12582912,348127232~16777216,394264576~8388608] 2024-09-21T10:38:29.140 INFO:tasks.workunit.client.0.smithi070.stdout:from 17 to 19 diff [67108864~12582912,96468992~4194304,117440512~8388608,134217728~4194304,163577856~8388608,188743680~4194304,239075328~12582912,281018368~12582912,306184192~8388608,318767104~4194304,348127232~12582912,390070272~12582912] 2024-09-21T10:38:29.140 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [67108864~12582912,96468992~4194304,117440512~8388608,134217728~4194304,163577856~8388608,188743680~4194304,239075328~12582912,281018368~12582912,306184192~8388608,318767104~4194304,348127232~12582912,390070272~12582912] 2024-09-21T10:38:29.141 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [67108864~12582912,96468992~4194304,117440512~8388608,134217728~8388608,163577856~8388608,188743680~4194304,239075328~16777216,281018368~12582912,306184192~8388608,318767104~8388608,348127232~16777216,390070272~16777216] 2024-09-21T10:38:29.141 INFO:tasks.workunit.client.0.smithi070.stdout:from 18 to 19 diff [96468992~4194304,117440512~8388608,163577856~8388608,188743680~4194304,243269632~8388608,306184192~8388608,318767104~4194304,390070272~4194304,398458880~8388608] 2024-09-21T10:38:29.141 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [96468992~4194304,117440512~8388608,163577856~8388608,188743680~4194304,243269632~8388608,306184192~8388608,318767104~4194304,390070272~4194304,398458880~8388608] 2024-09-21T10:38:29.141 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [96468992~4194304,117440512~8388608,163577856~8388608,188743680~4194304,243269632~12582912,306184192~8388608,318767104~8388608,390070272~4194304,398458880~8388608] 2024-09-21T10:38:29.551 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:29.551 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:29.569 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:29.569 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:29.569 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:29.576 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:29.576 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:29.577 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:29.582 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~234881024] 2024-09-21T10:38:29.582 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:29.583 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:29.589 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:29.589 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:29.589 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:29.595 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:29.595 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:29.595 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:29.600 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:29.600 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to HEAD diff [4194304~16777216,25165824~75497472,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:29.600 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~75497472,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:29.605 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~54525952,163577856~16777216,184549376~20971520,213909504~62914560,281018368~96468992,381681664~37748736] 2024-09-21T10:38:29.605 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to HEAD diff [4194304~16777216,25165824~75497472,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:29.605 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~75497472,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:29.610 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~20971520,134217728~25165824,163577856~16777216,184549376~20971520,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:29.610 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to HEAD diff [4194304~16777216,25165824~58720256,88080384~12582912,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,197132288~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:29.610 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~58720256,88080384~12582912,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,197132288~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:29.615 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~58720256,88080384~12582912,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,197132288~8388608,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:29.615 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to HEAD diff [8388608~12582912,25165824~8388608,37748736~46137344,88080384~4194304,96468992~4194304,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:29.615 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~46137344,88080384~4194304,96468992~4194304,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:29.619 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~58720256,88080384~12582912,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:29.619 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to HEAD diff [8388608~12582912,25165824~8388608,37748736~46137344,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:29.619 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~46137344,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:29.622 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~58720256,96468992~4194304,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:29.622 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to HEAD diff [16777216~4194304,25165824~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,381681664~20971520,406847488~8388608] 2024-09-21T10:38:29.622 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [16777216~4194304,25165824~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,381681664~20971520,406847488~8388608] 2024-09-21T10:38:29.625 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [16777216~4194304,25165824~12582912,41943040~41943040,96468992~4194304,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~33554432,318767104~20971520,343932928~33554432,381681664~37748736] 2024-09-21T10:38:29.625 INFO:tasks.workunit.client.0.smithi070.stdout:from 12 to HEAD diff [29360128~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,385875968~16777216,406847488~8388608] 2024-09-21T10:38:29.626 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,385875968~16777216,406847488~8388608] 2024-09-21T10:38:29.629 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,41943040~41943040,96468992~4194304,104857600~20971520,134217728~25165824,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~33554432,318767104~20971520,343932928~33554432,385875968~33554432] 2024-09-21T10:38:29.629 INFO:tasks.workunit.client.0.smithi070.stdout:from 13 to HEAD diff [29360128~8388608,41943040~16777216,67108864~16777216,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~12582912,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~12582912,343932928~33554432,390070272~12582912,406847488~8388608] 2024-09-21T10:38:29.629 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,41943040~16777216,67108864~16777216,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~12582912,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~12582912,343932928~33554432,390070272~12582912,406847488~8388608] 2024-09-21T10:38:29.631 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,41943040~20971520,67108864~16777216,96468992~4194304,104857600~20971520,134217728~25165824,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~12582912,301989888~12582912,318767104~20971520,343932928~33554432,390070272~29360128] 2024-09-21T10:38:29.632 INFO:tasks.workunit.client.0.smithi070.stdout:from 14 to HEAD diff [29360128~8388608,50331648~8388608,67108864~16777216,96468992~4194304,109051904~4194304,117440512~8388608,134217728~4194304,146800640~8388608,163577856~8388608,184549376~8388608,226492416~29360128,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~8388608,348127232~29360128,390070272~12582912,406847488~8388608] 2024-09-21T10:38:29.632 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,50331648~8388608,67108864~16777216,96468992~4194304,109051904~4194304,117440512~8388608,134217728~4194304,146800640~8388608,163577856~8388608,184549376~8388608,226492416~29360128,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~8388608,348127232~29360128,390070272~12582912,406847488~8388608] 2024-09-21T10:38:29.634 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,50331648~12582912,67108864~16777216,96468992~4194304,109051904~16777216,134217728~8388608,146800640~12582912,163577856~8388608,184549376~8388608,226492416~29360128,260046848~16777216,281018368~12582912,301989888~12582912,318767104~20971520,348127232~29360128,390070272~29360128] 2024-09-21T10:38:29.634 INFO:tasks.workunit.client.0.smithi070.stdout:from 15 to HEAD diff [50331648~8388608,67108864~16777216,96468992~4194304,117440512~8388608,134217728~4194304,146800640~8388608,163577856~8388608,188743680~4194304,234881024~16777216,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~8388608,348127232~29360128,390070272~12582912,406847488~8388608] 2024-09-21T10:38:29.634 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [50331648~8388608,67108864~16777216,96468992~4194304,117440512~8388608,134217728~4194304,146800640~8388608,163577856~8388608,188743680~4194304,234881024~16777216,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~8388608,348127232~29360128,390070272~12582912,406847488~8388608] 2024-09-21T10:38:29.635 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [50331648~12582912,67108864~16777216,96468992~4194304,117440512~8388608,134217728~8388608,146800640~12582912,163577856~8388608,188743680~4194304,234881024~20971520,260046848~16777216,281018368~12582912,301989888~12582912,318767104~20971520,348127232~29360128,390070272~29360128] 2024-09-21T10:38:29.636 INFO:tasks.workunit.client.0.smithi070.stdout:from 16 to HEAD diff [67108864~12582912,96468992~4194304,117440512~8388608,134217728~4194304,146800640~8388608,163577856~8388608,188743680~4194304,234881024~16777216,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~8388608,348127232~12582912,364904448~12582912,390070272~12582912] 2024-09-21T10:38:29.636 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [67108864~12582912,96468992~4194304,117440512~8388608,134217728~4194304,146800640~8388608,163577856~8388608,188743680~4194304,234881024~16777216,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~8388608,348127232~12582912,364904448~12582912,390070272~12582912] 2024-09-21T10:38:29.637 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [67108864~12582912,96468992~4194304,117440512~8388608,134217728~8388608,146800640~8388608,163577856~8388608,188743680~4194304,234881024~20971520,260046848~16777216,281018368~12582912,301989888~12582912,318767104~20971520,348127232~29360128,390070272~16777216] 2024-09-21T10:38:29.637 INFO:tasks.workunit.client.0.smithi070.stdout:from 17 to HEAD diff [67108864~12582912,96468992~4194304,117440512~8388608,134217728~4194304,163577856~8388608,188743680~4194304,239075328~12582912,281018368~12582912,306184192~8388608,318767104~4194304,348127232~12582912,390070272~12582912] 2024-09-21T10:38:29.637 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [67108864~12582912,96468992~4194304,117440512~8388608,134217728~4194304,163577856~8388608,188743680~4194304,239075328~12582912,281018368~12582912,306184192~8388608,318767104~4194304,348127232~12582912,390070272~12582912] 2024-09-21T10:38:29.638 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [67108864~12582912,96468992~4194304,117440512~8388608,134217728~8388608,163577856~8388608,188743680~4194304,239075328~16777216,281018368~12582912,306184192~8388608,318767104~8388608,348127232~16777216,390070272~16777216] 2024-09-21T10:38:29.798 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:29 smithi070 ceph-mon[107385]: osdmap e1279: 8 total, 8 up, 8 in 2024-09-21T10:38:29.798 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:29 smithi070 ceph-mon[107385]: pgmap v2067: 105 pgs: 105 active+clean; 1.1 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 117 KiB/s rd, 24 MiB/s wr, 200 op/s 2024-09-21T10:38:29.798 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:29 smithi070 ceph-mon[112139]: osdmap e1279: 8 total, 8 up, 8 in 2024-09-21T10:38:29.798 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:29 smithi070 ceph-mon[112139]: pgmap v2067: 105 pgs: 105 active+clean; 1.1 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 117 KiB/s rd, 24 MiB/s wr, 200 op/s 2024-09-21T10:38:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:29 smithi160 ceph-mon[101879]: osdmap e1279: 8 total, 8 up, 8 in 2024-09-21T10:38:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:29 smithi160 ceph-mon[101879]: pgmap v2067: 105 pgs: 105 active+clean; 1.1 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 117 KiB/s rd, 24 MiB/s wr, 200 op/s 2024-09-21T10:38:30.558 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:30.558 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:30.575 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:30.576 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:30.576 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:30.581 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:30.581 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:30.582 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:30.588 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~234881024] 2024-09-21T10:38:30.588 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:30.588 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:30.594 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:30.594 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:30.594 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:30.599 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:30.599 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:30.599 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:30.603 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:30.603 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to HEAD diff [4194304~16777216,25165824~75497472,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:30.603 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~75497472,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:30.607 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~54525952,163577856~16777216,184549376~20971520,213909504~62914560,281018368~96468992,381681664~37748736] 2024-09-21T10:38:30.608 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to HEAD diff [4194304~16777216,25165824~75497472,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:30.608 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~75497472,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:30.611 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~20971520,134217728~25165824,163577856~16777216,184549376~20971520,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:30.611 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to HEAD diff [4194304~16777216,25165824~58720256,88080384~12582912,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,197132288~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:30.612 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~58720256,88080384~12582912,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,197132288~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:30.615 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~58720256,88080384~12582912,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,197132288~8388608,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:30.615 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to HEAD diff [8388608~12582912,25165824~8388608,37748736~46137344,88080384~4194304,96468992~4194304,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:30.615 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~46137344,88080384~4194304,96468992~4194304,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:30.619 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~58720256,88080384~12582912,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:30.619 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to HEAD diff [8388608~12582912,25165824~8388608,37748736~46137344,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:30.619 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~46137344,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:30.622 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~58720256,96468992~4194304,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:30.622 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to HEAD diff [16777216~4194304,25165824~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,381681664~20971520,406847488~8388608] 2024-09-21T10:38:30.622 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [16777216~4194304,25165824~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,381681664~20971520,406847488~8388608] 2024-09-21T10:38:30.626 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [16777216~4194304,25165824~12582912,41943040~41943040,96468992~4194304,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~33554432,318767104~20971520,343932928~33554432,381681664~37748736] 2024-09-21T10:38:30.626 INFO:tasks.workunit.client.0.smithi070.stdout:from 12 to HEAD diff [29360128~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,385875968~16777216,406847488~8388608] 2024-09-21T10:38:30.626 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,385875968~16777216,406847488~8388608] 2024-09-21T10:38:30.628 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,41943040~41943040,96468992~4194304,104857600~20971520,134217728~25165824,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~33554432,318767104~20971520,343932928~33554432,385875968~33554432] 2024-09-21T10:38:30.628 INFO:tasks.workunit.client.0.smithi070.stdout:from 13 to HEAD diff [29360128~8388608,41943040~16777216,67108864~16777216,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~12582912,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~12582912,343932928~33554432,390070272~12582912,406847488~8388608] 2024-09-21T10:38:30.628 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,41943040~16777216,67108864~16777216,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~12582912,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~12582912,343932928~33554432,390070272~12582912,406847488~8388608] 2024-09-21T10:38:30.631 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,41943040~20971520,67108864~16777216,96468992~4194304,104857600~20971520,134217728~25165824,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~12582912,301989888~12582912,318767104~20971520,343932928~33554432,390070272~29360128] 2024-09-21T10:38:30.631 INFO:tasks.workunit.client.0.smithi070.stdout:from 14 to HEAD diff [29360128~8388608,50331648~8388608,67108864~16777216,96468992~4194304,109051904~4194304,117440512~8388608,134217728~4194304,146800640~8388608,163577856~8388608,184549376~8388608,226492416~29360128,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~8388608,348127232~29360128,390070272~12582912,406847488~8388608] 2024-09-21T10:38:30.631 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,50331648~8388608,67108864~16777216,96468992~4194304,109051904~4194304,117440512~8388608,134217728~4194304,146800640~8388608,163577856~8388608,184549376~8388608,226492416~29360128,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~8388608,348127232~29360128,390070272~12582912,406847488~8388608] 2024-09-21T10:38:30.633 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,50331648~12582912,67108864~16777216,96468992~4194304,109051904~16777216,134217728~8388608,146800640~12582912,163577856~8388608,184549376~8388608,226492416~29360128,260046848~16777216,281018368~12582912,301989888~12582912,318767104~20971520,348127232~29360128,390070272~29360128] 2024-09-21T10:38:30.633 INFO:tasks.workunit.client.0.smithi070.stdout:from 15 to HEAD diff [50331648~8388608,67108864~16777216,96468992~4194304,117440512~8388608,134217728~4194304,146800640~8388608,163577856~8388608,188743680~4194304,234881024~16777216,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~8388608,348127232~29360128,390070272~12582912,406847488~8388608] 2024-09-21T10:38:30.633 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [50331648~8388608,67108864~16777216,96468992~4194304,117440512~8388608,134217728~4194304,146800640~8388608,163577856~8388608,188743680~4194304,234881024~16777216,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~8388608,348127232~29360128,390070272~12582912,406847488~8388608] 2024-09-21T10:38:30.634 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [50331648~12582912,67108864~16777216,96468992~4194304,117440512~8388608,134217728~8388608,146800640~12582912,163577856~8388608,188743680~4194304,234881024~20971520,260046848~16777216,281018368~12582912,301989888~12582912,318767104~20971520,348127232~29360128,390070272~29360128] 2024-09-21T10:38:30.634 INFO:tasks.workunit.client.0.smithi070.stdout:from 16 to HEAD diff [67108864~12582912,96468992~4194304,117440512~8388608,134217728~4194304,146800640~8388608,163577856~8388608,188743680~4194304,234881024~16777216,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~8388608,348127232~12582912,364904448~12582912,390070272~12582912] 2024-09-21T10:38:30.634 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [67108864~12582912,96468992~4194304,117440512~8388608,134217728~4194304,146800640~8388608,163577856~8388608,188743680~4194304,234881024~16777216,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~8388608,348127232~12582912,364904448~12582912,390070272~12582912] 2024-09-21T10:38:30.635 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [67108864~12582912,96468992~4194304,117440512~8388608,134217728~8388608,146800640~8388608,163577856~8388608,188743680~4194304,234881024~20971520,260046848~16777216,281018368~12582912,301989888~12582912,318767104~20971520,348127232~29360128,390070272~16777216] 2024-09-21T10:38:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:30 smithi160 ceph-mon[101879]: osdmap e1280: 8 total, 8 up, 8 in 2024-09-21T10:38:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:30 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:38:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:30 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:38:30.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:30 smithi070 ceph-mon[107385]: osdmap e1280: 8 total, 8 up, 8 in 2024-09-21T10:38:30.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:30 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:38:30.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:30 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:38:30.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:30 smithi070 ceph-mon[112139]: osdmap e1280: 8 total, 8 up, 8 in 2024-09-21T10:38:30.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:30 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:38:30.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:30 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:38:31.567 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:31.568 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:31.581 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:31.581 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:31.581 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:31.587 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:31.587 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:31.587 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:31.591 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~234881024] 2024-09-21T10:38:31.592 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:31.592 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:31.597 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:31.597 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:31.597 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:31.603 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:31.603 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:31.603 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:31.606 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:31.607 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to HEAD diff [4194304~16777216,25165824~75497472,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:31.607 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~75497472,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:31.611 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~54525952,163577856~16777216,184549376~20971520,213909504~62914560,281018368~96468992,381681664~37748736] 2024-09-21T10:38:31.611 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to HEAD diff [4194304~16777216,25165824~75497472,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:31.611 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~75497472,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:31.615 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~20971520,134217728~25165824,163577856~16777216,184549376~20971520,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:31.615 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to HEAD diff [4194304~16777216,25165824~58720256,88080384~12582912,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,197132288~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:31.615 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~58720256,88080384~12582912,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,197132288~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:31.618 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~58720256,88080384~12582912,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,197132288~8388608,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:31.619 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to HEAD diff [8388608~12582912,25165824~8388608,37748736~46137344,88080384~4194304,96468992~4194304,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:31.619 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~46137344,88080384~4194304,96468992~4194304,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:31.622 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~58720256,88080384~12582912,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:31.622 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to HEAD diff [8388608~12582912,25165824~8388608,37748736~46137344,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:31.622 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~46137344,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:31.625 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~58720256,96468992~4194304,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:31.625 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to HEAD diff [16777216~4194304,25165824~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,381681664~20971520,406847488~8388608] 2024-09-21T10:38:31.625 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [16777216~4194304,25165824~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,381681664~20971520,406847488~8388608] 2024-09-21T10:38:31.629 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [16777216~4194304,25165824~12582912,41943040~41943040,96468992~4194304,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~33554432,318767104~20971520,343932928~33554432,381681664~37748736] 2024-09-21T10:38:31.629 INFO:tasks.workunit.client.0.smithi070.stdout:from 12 to HEAD diff [29360128~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,385875968~16777216,406847488~8388608] 2024-09-21T10:38:31.629 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,385875968~16777216,406847488~8388608] 2024-09-21T10:38:31.631 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,41943040~41943040,96468992~4194304,104857600~20971520,134217728~25165824,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~33554432,318767104~20971520,343932928~33554432,385875968~33554432] 2024-09-21T10:38:31.632 INFO:tasks.workunit.client.0.smithi070.stdout:from 13 to HEAD diff [29360128~8388608,41943040~16777216,67108864~16777216,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~12582912,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~12582912,343932928~33554432,390070272~12582912,406847488~8388608] 2024-09-21T10:38:31.632 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,41943040~16777216,67108864~16777216,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~12582912,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~12582912,343932928~33554432,390070272~12582912,406847488~8388608] 2024-09-21T10:38:31.633 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,41943040~20971520,67108864~16777216,96468992~4194304,104857600~20971520,134217728~25165824,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~12582912,301989888~12582912,318767104~20971520,343932928~33554432,390070272~29360128] 2024-09-21T10:38:31.634 INFO:tasks.workunit.client.0.smithi070.stdout:from 14 to HEAD diff [29360128~8388608,50331648~8388608,67108864~16777216,96468992~4194304,109051904~4194304,117440512~8388608,134217728~4194304,146800640~8388608,163577856~8388608,184549376~8388608,226492416~29360128,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~8388608,348127232~29360128,390070272~12582912,406847488~8388608] 2024-09-21T10:38:31.634 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,50331648~8388608,67108864~16777216,96468992~4194304,109051904~4194304,117440512~8388608,134217728~4194304,146800640~8388608,163577856~8388608,184549376~8388608,226492416~29360128,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~8388608,348127232~29360128,390070272~12582912,406847488~8388608] 2024-09-21T10:38:31.636 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,50331648~12582912,67108864~16777216,96468992~4194304,109051904~16777216,134217728~8388608,146800640~12582912,163577856~8388608,184549376~8388608,226492416~29360128,260046848~16777216,281018368~12582912,301989888~12582912,318767104~20971520,348127232~29360128,390070272~29360128] 2024-09-21T10:38:31.636 INFO:tasks.workunit.client.0.smithi070.stdout:from 15 to HEAD diff [50331648~8388608,67108864~16777216,96468992~4194304,117440512~8388608,134217728~4194304,146800640~8388608,163577856~8388608,188743680~4194304,234881024~16777216,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~8388608,348127232~29360128,390070272~12582912,406847488~8388608] 2024-09-21T10:38:31.636 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [50331648~8388608,67108864~16777216,96468992~4194304,117440512~8388608,134217728~4194304,146800640~8388608,163577856~8388608,188743680~4194304,234881024~16777216,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~8388608,348127232~29360128,390070272~12582912,406847488~8388608] 2024-09-21T10:38:31.636 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [50331648~12582912,67108864~16777216,96468992~4194304,117440512~8388608,134217728~8388608,146800640~12582912,163577856~8388608,188743680~4194304,234881024~20971520,260046848~16777216,281018368~12582912,301989888~12582912,318767104~20971520,348127232~29360128,390070272~29360128] 2024-09-21T10:38:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:31 smithi160 ceph-mon[101879]: osdmap e1281: 8 total, 8 up, 8 in 2024-09-21T10:38:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:31 smithi160 ceph-mon[101879]: pgmap v2070: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 385 KiB/s rd, 19 MiB/s wr, 687 op/s 2024-09-21T10:38:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:38:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:38:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:38:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:38:31.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:38:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:31 smithi070 ceph-mon[107385]: osdmap e1281: 8 total, 8 up, 8 in 2024-09-21T10:38:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:31 smithi070 ceph-mon[107385]: pgmap v2070: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 385 KiB/s rd, 19 MiB/s wr, 687 op/s 2024-09-21T10:38:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:38:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:38:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:38:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:38:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:38:31.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:31 smithi070 ceph-mon[112139]: osdmap e1281: 8 total, 8 up, 8 in 2024-09-21T10:38:31.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:31 smithi070 ceph-mon[112139]: pgmap v2070: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 385 KiB/s rd, 19 MiB/s wr, 687 op/s 2024-09-21T10:38:31.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:38:31.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:38:31.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:38:31.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:38:31.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:38:32.576 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:32.576 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:32.594 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:32.594 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:32.594 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:32.600 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:32.600 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:32.600 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:32.606 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~234881024] 2024-09-21T10:38:32.607 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:32.607 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:32.611 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:32.612 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:32.612 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:32.617 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:32.617 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:32.617 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:32.621 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:32.621 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to HEAD diff [4194304~16777216,25165824~75497472,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:32.621 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~75497472,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:32.625 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~54525952,163577856~16777216,184549376~20971520,213909504~62914560,281018368~96468992,381681664~37748736] 2024-09-21T10:38:32.625 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to HEAD diff [4194304~16777216,25165824~75497472,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:32.625 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~75497472,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:32.630 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~20971520,134217728~25165824,163577856~16777216,184549376~20971520,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:32.630 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to HEAD diff [4194304~16777216,25165824~58720256,88080384~12582912,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,197132288~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:32.630 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~58720256,88080384~12582912,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,197132288~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:32.634 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~58720256,88080384~12582912,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,197132288~8388608,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:32.634 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to HEAD diff [8388608~12582912,25165824~8388608,37748736~46137344,88080384~4194304,96468992~4194304,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:32.634 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~46137344,88080384~4194304,96468992~4194304,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:32.636 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~58720256,88080384~12582912,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:32.636 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to HEAD diff [8388608~12582912,25165824~8388608,37748736~46137344,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:32.636 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~46137344,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:32.639 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~58720256,96468992~4194304,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:32.639 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to HEAD diff [16777216~4194304,25165824~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,381681664~20971520,406847488~8388608] 2024-09-21T10:38:32.639 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [16777216~4194304,25165824~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,381681664~20971520,406847488~8388608] 2024-09-21T10:38:32.642 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [16777216~4194304,25165824~12582912,41943040~41943040,96468992~4194304,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~33554432,318767104~20971520,343932928~33554432,381681664~37748736] 2024-09-21T10:38:32.642 INFO:tasks.workunit.client.0.smithi070.stdout:from 12 to HEAD diff [29360128~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,385875968~16777216,406847488~8388608] 2024-09-21T10:38:32.642 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,385875968~16777216,406847488~8388608] 2024-09-21T10:38:32.644 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,41943040~41943040,96468992~4194304,104857600~20971520,134217728~25165824,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~33554432,318767104~20971520,343932928~33554432,385875968~33554432] 2024-09-21T10:38:32.644 INFO:tasks.workunit.client.0.smithi070.stdout:from 13 to HEAD diff [29360128~8388608,41943040~16777216,67108864~16777216,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~12582912,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~12582912,343932928~33554432,390070272~12582912,406847488~8388608] 2024-09-21T10:38:32.644 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,41943040~16777216,67108864~16777216,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~12582912,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~12582912,343932928~33554432,390070272~12582912,406847488~8388608] 2024-09-21T10:38:32.646 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,41943040~20971520,67108864~16777216,96468992~4194304,104857600~20971520,134217728~25165824,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~12582912,301989888~12582912,318767104~20971520,343932928~33554432,390070272~29360128] 2024-09-21T10:38:32.646 INFO:tasks.workunit.client.0.smithi070.stdout:from 14 to HEAD diff [29360128~8388608,50331648~8388608,67108864~16777216,96468992~4194304,109051904~4194304,117440512~8388608,134217728~4194304,146800640~8388608,163577856~8388608,184549376~8388608,226492416~29360128,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~8388608,348127232~29360128,390070272~12582912,406847488~8388608] 2024-09-21T10:38:32.646 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,50331648~8388608,67108864~16777216,96468992~4194304,109051904~4194304,117440512~8388608,134217728~4194304,146800640~8388608,163577856~8388608,184549376~8388608,226492416~29360128,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~8388608,348127232~29360128,390070272~12582912,406847488~8388608] 2024-09-21T10:38:32.647 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,50331648~12582912,67108864~16777216,96468992~4194304,109051904~16777216,134217728~8388608,146800640~12582912,163577856~8388608,184549376~8388608,226492416~29360128,260046848~16777216,281018368~12582912,301989888~12582912,318767104~20971520,348127232~29360128,390070272~29360128] 2024-09-21T10:38:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:32 smithi160 ceph-mon[101879]: osdmap e1282: 8 total, 8 up, 8 in 2024-09-21T10:38:32.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:32 smithi070 ceph-mon[107385]: osdmap e1282: 8 total, 8 up, 8 in 2024-09-21T10:38:32.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:32 smithi070 ceph-mon[112139]: osdmap e1282: 8 total, 8 up, 8 in 2024-09-21T10:38:33.566 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:38:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:38:33] "GET /metrics HTTP/1.1" 200 36481 "" "Prometheus/2.51.0" 2024-09-21T10:38:33.579 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:33.579 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:33.601 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:33.601 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:33.601 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:33.606 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:33.606 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:33.606 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:33.612 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~234881024] 2024-09-21T10:38:33.612 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:33.612 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:33.617 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:33.617 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:33.617 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:33.622 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:33.622 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:33.622 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:33.626 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:33.626 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to HEAD diff [4194304~16777216,25165824~75497472,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:33.626 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~75497472,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:33.630 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~54525952,163577856~16777216,184549376~20971520,213909504~62914560,281018368~96468992,381681664~37748736] 2024-09-21T10:38:33.630 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to HEAD diff [4194304~16777216,25165824~75497472,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:33.630 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~75497472,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:33.633 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~20971520,134217728~25165824,163577856~16777216,184549376~20971520,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:33.633 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to HEAD diff [4194304~16777216,25165824~58720256,88080384~12582912,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,197132288~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:33.633 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~58720256,88080384~12582912,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,197132288~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:33.636 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~58720256,88080384~12582912,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,197132288~8388608,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:33.636 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to HEAD diff [8388608~12582912,25165824~8388608,37748736~46137344,88080384~4194304,96468992~4194304,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:33.636 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~46137344,88080384~4194304,96468992~4194304,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:33.639 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~58720256,88080384~12582912,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:33.639 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to HEAD diff [8388608~12582912,25165824~8388608,37748736~46137344,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:33.639 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~46137344,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:33.641 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~58720256,96468992~4194304,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:33.641 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to HEAD diff [16777216~4194304,25165824~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,381681664~20971520,406847488~8388608] 2024-09-21T10:38:33.641 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [16777216~4194304,25165824~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,381681664~20971520,406847488~8388608] 2024-09-21T10:38:33.643 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [16777216~4194304,25165824~12582912,41943040~41943040,96468992~4194304,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~33554432,318767104~20971520,343932928~33554432,381681664~37748736] 2024-09-21T10:38:33.644 INFO:tasks.workunit.client.0.smithi070.stdout:from 12 to HEAD diff [29360128~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,385875968~16777216,406847488~8388608] 2024-09-21T10:38:33.644 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,385875968~16777216,406847488~8388608] 2024-09-21T10:38:33.646 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,41943040~41943040,96468992~4194304,104857600~20971520,134217728~25165824,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~33554432,318767104~20971520,343932928~33554432,385875968~33554432] 2024-09-21T10:38:33.646 INFO:tasks.workunit.client.0.smithi070.stdout:from 13 to HEAD diff [29360128~8388608,41943040~16777216,67108864~16777216,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~12582912,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~12582912,343932928~33554432,390070272~12582912,406847488~8388608] 2024-09-21T10:38:33.646 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,41943040~16777216,67108864~16777216,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~12582912,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~12582912,301989888~12582912,318767104~4194304,327155712~12582912,343932928~33554432,390070272~12582912,406847488~8388608] 2024-09-21T10:38:33.647 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,41943040~20971520,67108864~16777216,96468992~4194304,104857600~20971520,134217728~25165824,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~12582912,301989888~12582912,318767104~20971520,343932928~33554432,390070272~29360128] 2024-09-21T10:38:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:33 smithi160 ceph-mon[101879]: osdmap e1283: 8 total, 8 up, 8 in 2024-09-21T10:38:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:33 smithi160 ceph-mon[101879]: pgmap v2073: 105 pgs: 105 active+clean; 1.2 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 605 KiB/s rd, 26 MiB/s wr, 1.11k op/s 2024-09-21T10:38:33.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:33 smithi070 ceph-mon[107385]: osdmap e1283: 8 total, 8 up, 8 in 2024-09-21T10:38:33.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:33 smithi070 ceph-mon[107385]: pgmap v2073: 105 pgs: 105 active+clean; 1.2 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 605 KiB/s rd, 26 MiB/s wr, 1.11k op/s 2024-09-21T10:38:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:33 smithi070 ceph-mon[112139]: osdmap e1283: 8 total, 8 up, 8 in 2024-09-21T10:38:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:33 smithi070 ceph-mon[112139]: pgmap v2073: 105 pgs: 105 active+clean; 1.2 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 605 KiB/s rd, 26 MiB/s wr, 1.11k op/s 2024-09-21T10:38:34.587 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:34.587 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:34.603 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:34.603 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:34.603 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:34.610 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:34.610 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:34.610 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:34.615 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~234881024] 2024-09-21T10:38:34.615 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:34.615 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:34.619 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:34.619 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:34.619 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:34.623 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:34.623 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:34.624 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:34.627 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:34.627 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to HEAD diff [4194304~16777216,25165824~75497472,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:34.627 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~75497472,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:34.630 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~54525952,163577856~16777216,184549376~20971520,213909504~62914560,281018368~96468992,381681664~37748736] 2024-09-21T10:38:34.630 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to HEAD diff [4194304~16777216,25165824~75497472,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:34.630 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~75497472,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:34.633 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~20971520,134217728~25165824,163577856~16777216,184549376~20971520,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:34.633 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to HEAD diff [4194304~16777216,25165824~58720256,88080384~12582912,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,197132288~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:34.633 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~58720256,88080384~12582912,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,197132288~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:34.636 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~58720256,88080384~12582912,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,197132288~8388608,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:34.636 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to HEAD diff [8388608~12582912,25165824~8388608,37748736~46137344,88080384~4194304,96468992~4194304,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:34.636 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~46137344,88080384~4194304,96468992~4194304,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:34.638 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~58720256,88080384~12582912,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:34.638 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to HEAD diff [8388608~12582912,25165824~8388608,37748736~46137344,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:34.638 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~46137344,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:34.640 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~58720256,96468992~4194304,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:34.640 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to HEAD diff [16777216~4194304,25165824~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,381681664~20971520,406847488~8388608] 2024-09-21T10:38:34.640 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [16777216~4194304,25165824~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,381681664~20971520,406847488~8388608] 2024-09-21T10:38:34.642 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [16777216~4194304,25165824~12582912,41943040~41943040,96468992~4194304,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~33554432,318767104~20971520,343932928~33554432,381681664~37748736] 2024-09-21T10:38:34.642 INFO:tasks.workunit.client.0.smithi070.stdout:from 12 to HEAD diff [29360128~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,385875968~16777216,406847488~8388608] 2024-09-21T10:38:34.642 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [29360128~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,385875968~16777216,406847488~8388608] 2024-09-21T10:38:34.643 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [29360128~8388608,41943040~41943040,96468992~4194304,104857600~20971520,134217728~25165824,163577856~8388608,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~33554432,318767104~20971520,343932928~33554432,385875968~33554432] 2024-09-21T10:38:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:34 smithi160 ceph-mon[101879]: osdmap e1284: 8 total, 8 up, 8 in 2024-09-21T10:38:34.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:34 smithi070 ceph-mon[107385]: osdmap e1284: 8 total, 8 up, 8 in 2024-09-21T10:38:34.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:34 smithi070 ceph-mon[112139]: osdmap e1284: 8 total, 8 up, 8 in 2024-09-21T10:38:35.591 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:35.591 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:35.612 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:35.612 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:35.612 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:35.618 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:35.618 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:35.618 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:35.623 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~234881024] 2024-09-21T10:38:35.623 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:35.623 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:35.627 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:35.627 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:35.627 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:35.630 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:35.631 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:35.631 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:35.634 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:35.634 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to HEAD diff [4194304~16777216,25165824~75497472,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:35.634 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~75497472,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:35.638 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~54525952,163577856~16777216,184549376~20971520,213909504~62914560,281018368~96468992,381681664~37748736] 2024-09-21T10:38:35.638 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to HEAD diff [4194304~16777216,25165824~75497472,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:35.638 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~75497472,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:35.640 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~20971520,134217728~25165824,163577856~16777216,184549376~20971520,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:35.641 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to HEAD diff [4194304~16777216,25165824~58720256,88080384~12582912,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,197132288~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:35.641 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~58720256,88080384~12582912,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,197132288~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:35.643 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~58720256,88080384~12582912,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,197132288~8388608,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:35.643 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to HEAD diff [8388608~12582912,25165824~8388608,37748736~46137344,88080384~4194304,96468992~4194304,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:35.643 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~46137344,88080384~4194304,96468992~4194304,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:35.645 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~58720256,88080384~12582912,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:35.645 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to HEAD diff [8388608~12582912,25165824~8388608,37748736~46137344,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:35.645 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~46137344,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:35.647 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~58720256,96468992~4194304,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:35.647 INFO:tasks.workunit.client.0.smithi070.stdout:from 11 to HEAD diff [16777216~4194304,25165824~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,381681664~20971520,406847488~8388608] 2024-09-21T10:38:35.647 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [16777216~4194304,25165824~8388608,41943040~41943040,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~12582912,343932928~33554432,381681664~20971520,406847488~8388608] 2024-09-21T10:38:35.648 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [16777216~4194304,25165824~12582912,41943040~41943040,96468992~4194304,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~33554432,318767104~20971520,343932928~33554432,381681664~37748736] 2024-09-21T10:38:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:35 smithi160 ceph-mon[101879]: osdmap e1285: 8 total, 8 up, 8 in 2024-09-21T10:38:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:35 smithi160 ceph-mon[101879]: pgmap v2076: 105 pgs: 105 active+clean; 1.2 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 535 KiB/s rd, 8.0 MiB/s wr, 971 op/s 2024-09-21T10:38:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:38:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:35 smithi070 ceph-mon[107385]: osdmap e1285: 8 total, 8 up, 8 in 2024-09-21T10:38:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:35 smithi070 ceph-mon[107385]: pgmap v2076: 105 pgs: 105 active+clean; 1.2 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 535 KiB/s rd, 8.0 MiB/s wr, 971 op/s 2024-09-21T10:38:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:38:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:35 smithi070 ceph-mon[112139]: osdmap e1285: 8 total, 8 up, 8 in 2024-09-21T10:38:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:35 smithi070 ceph-mon[112139]: pgmap v2076: 105 pgs: 105 active+clean; 1.2 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 535 KiB/s rd, 8.0 MiB/s wr, 971 op/s 2024-09-21T10:38:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:38:36.601 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:36.601 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:36.620 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:36.620 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:36.620 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:36.625 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:36.625 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:36.625 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:36.629 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~234881024] 2024-09-21T10:38:36.629 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:36.629 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:36.633 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:36.633 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:36.633 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:36.636 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:36.637 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:36.637 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:36.639 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:36.639 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to HEAD diff [4194304~16777216,25165824~75497472,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:36.639 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~75497472,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:36.642 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~54525952,163577856~16777216,184549376~20971520,213909504~62914560,281018368~96468992,381681664~37748736] 2024-09-21T10:38:36.642 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to HEAD diff [4194304~16777216,25165824~75497472,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:36.642 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~75497472,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:36.644 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~20971520,134217728~25165824,163577856~16777216,184549376~20971520,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:36.644 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to HEAD diff [4194304~16777216,25165824~58720256,88080384~12582912,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,197132288~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:36.644 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~58720256,88080384~12582912,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,197132288~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:36.646 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~58720256,88080384~12582912,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,197132288~8388608,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:36.646 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to HEAD diff [8388608~12582912,25165824~8388608,37748736~46137344,88080384~4194304,96468992~4194304,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:36.646 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~46137344,88080384~4194304,96468992~4194304,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:36.648 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~58720256,88080384~12582912,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:36.648 INFO:tasks.workunit.client.0.smithi070.stdout:from 10 to HEAD diff [8388608~12582912,25165824~8388608,37748736~46137344,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:36.648 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~46137344,96468992~4194304,104857600~8388608,117440512~8388608,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~12582912,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:36.649 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~58720256,96468992~4194304,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,213909504~4194304,226492416~29360128,260046848~16777216,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:36 smithi160 ceph-mon[101879]: osdmap e1286: 8 total, 8 up, 8 in 2024-09-21T10:38:36.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:36 smithi070 ceph-mon[107385]: osdmap e1286: 8 total, 8 up, 8 in 2024-09-21T10:38:36.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:36 smithi070 ceph-mon[112139]: osdmap e1286: 8 total, 8 up, 8 in 2024-09-21T10:38:37.604 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:37.604 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:37.625 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:37.626 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:37.626 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:37.630 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:37.630 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:37.630 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:37.633 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~234881024] 2024-09-21T10:38:37.633 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:37.633 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:37.637 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:37.637 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:37.637 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:37.640 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:37.640 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:37.640 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:37.643 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:37.643 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to HEAD diff [4194304~16777216,25165824~75497472,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:37.643 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~75497472,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:37.645 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~54525952,163577856~16777216,184549376~20971520,213909504~62914560,281018368~96468992,381681664~37748736] 2024-09-21T10:38:37.645 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to HEAD diff [4194304~16777216,25165824~75497472,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:37.645 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~75497472,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:37.647 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~20971520,134217728~25165824,163577856~16777216,184549376~20971520,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:37.647 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to HEAD diff [4194304~16777216,25165824~58720256,88080384~12582912,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,197132288~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:37.647 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~58720256,88080384~12582912,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,197132288~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:37.648 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~58720256,88080384~12582912,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,197132288~8388608,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:37.648 INFO:tasks.workunit.client.0.smithi070.stdout:from 9 to HEAD diff [8388608~12582912,25165824~8388608,37748736~46137344,88080384~4194304,96468992~4194304,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:37.649 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [8388608~12582912,25165824~8388608,37748736~46137344,88080384~4194304,96468992~4194304,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:37.649 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [8388608~12582912,25165824~58720256,88080384~12582912,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:37.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:37 smithi070 ceph-mon[107385]: osdmap e1287: 8 total, 8 up, 8 in 2024-09-21T10:38:37.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:37 smithi070 ceph-mon[107385]: pgmap v2079: 105 pgs: 105 active+clean; 1.1 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 382 KiB/s rd, 7.0 KiB/s wr, 675 op/s 2024-09-21T10:38:37.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:37 smithi070 ceph-mon[112139]: osdmap e1287: 8 total, 8 up, 8 in 2024-09-21T10:38:37.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:37 smithi070 ceph-mon[112139]: pgmap v2079: 105 pgs: 105 active+clean; 1.1 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 382 KiB/s rd, 7.0 KiB/s wr, 675 op/s 2024-09-21T10:38:38.089 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:38.089 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:38.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:37 smithi160 ceph-mon[101879]: osdmap e1287: 8 total, 8 up, 8 in 2024-09-21T10:38:38.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:37 smithi160 ceph-mon[101879]: pgmap v2079: 105 pgs: 105 active+clean; 1.1 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 382 KiB/s rd, 7.0 KiB/s wr, 675 op/s 2024-09-21T10:38:38.107 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:38.107 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:38.107 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:38.113 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:38.113 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:38.113 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:38.117 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~234881024] 2024-09-21T10:38:38.117 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:38.117 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:38.120 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:38.120 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:38.120 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:38.122 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:38.123 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:38.123 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:38.125 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:38.125 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to HEAD diff [4194304~16777216,25165824~75497472,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:38.125 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~75497472,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:38.127 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~54525952,163577856~16777216,184549376~20971520,213909504~62914560,281018368~96468992,381681664~37748736] 2024-09-21T10:38:38.127 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to HEAD diff [4194304~16777216,25165824~75497472,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:38.127 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~75497472,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:38.128 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~20971520,134217728~25165824,163577856~16777216,184549376~20971520,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:38.128 INFO:tasks.workunit.client.0.smithi070.stdout:from 8 to HEAD diff [4194304~16777216,25165824~58720256,88080384~12582912,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,197132288~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:38.129 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~58720256,88080384~12582912,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~8388608,197132288~8388608,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:38.130 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~58720256,88080384~12582912,104857600~20971520,134217728~25165824,163577856~16777216,184549376~8388608,197132288~8388608,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:38.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:38 smithi070 ceph-mon[107385]: osdmap e1288: 8 total, 8 up, 8 in 2024-09-21T10:38:38.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:38 smithi070 ceph-mon[107385]: osdmap e1289: 8 total, 8 up, 8 in 2024-09-21T10:38:38.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:38 smithi070 ceph-mon[112139]: osdmap e1288: 8 total, 8 up, 8 in 2024-09-21T10:38:38.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:38 smithi070 ceph-mon[112139]: osdmap e1289: 8 total, 8 up, 8 in 2024-09-21T10:38:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:38 smithi160 ceph-mon[101879]: osdmap e1288: 8 total, 8 up, 8 in 2024-09-21T10:38:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:38 smithi160 ceph-mon[101879]: osdmap e1289: 8 total, 8 up, 8 in 2024-09-21T10:38:39.093 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:39.094 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:39.115 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:39.115 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:39.115 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:39.121 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:39.121 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:39.121 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:39.123 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~234881024] 2024-09-21T10:38:39.124 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:39.124 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:39.127 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:39.127 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:39.127 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:39.129 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:39.129 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:39.129 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:39.131 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:39.132 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to HEAD diff [4194304~16777216,25165824~75497472,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:39.132 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~75497472,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:39.134 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~54525952,163577856~16777216,184549376~20971520,213909504~62914560,281018368~96468992,381681664~37748736] 2024-09-21T10:38:39.134 INFO:tasks.workunit.client.0.smithi070.stdout:from 7 to HEAD diff [4194304~16777216,25165824~75497472,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:39.134 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~75497472,104857600~16777216,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~33554432,318767104~4194304,327155712~50331648,381681664~20971520,406847488~8388608] 2024-09-21T10:38:39.135 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~20971520,134217728~25165824,163577856~16777216,184549376~20971520,213909504~62914560,281018368~33554432,318767104~58720256,381681664~37748736] 2024-09-21T10:38:40.095 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:40.095 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:40.119 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:40.119 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:40.119 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:40.123 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:40.123 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:40.123 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:40.128 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~234881024] 2024-09-21T10:38:40.128 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:40.128 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:40.130 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:40.131 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:40.131 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:40.134 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:40.134 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:40.134 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:40.134 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:40.135 INFO:tasks.workunit.client.0.smithi070.stdout:from 6 to HEAD diff [4194304~16777216,25165824~75497472,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:40.135 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~16777216,25165824~75497472,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~8388608,281018368~96468992,381681664~33554432] 2024-09-21T10:38:40.135 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~16777216,25165824~75497472,104857600~54525952,163577856~16777216,184549376~20971520,213909504~62914560,281018368~96468992,381681664~37748736] 2024-09-21T10:38:40.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:40 smithi160 ceph-mon[101879]: pgmap v2082: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 67 KiB/s rd, 1.2 KiB/s wr, 135 op/s 2024-09-21T10:38:40.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:40 smithi160 ceph-mon[101879]: osdmap e1290: 8 total, 8 up, 8 in 2024-09-21T10:38:40.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:40 smithi070 ceph-mon[112139]: pgmap v2082: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 67 KiB/s rd, 1.2 KiB/s wr, 135 op/s 2024-09-21T10:38:40.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:40 smithi070 ceph-mon[112139]: osdmap e1290: 8 total, 8 up, 8 in 2024-09-21T10:38:40.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:40 smithi070 ceph-mon[107385]: pgmap v2082: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 67 KiB/s rd, 1.2 KiB/s wr, 135 op/s 2024-09-21T10:38:40.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:40 smithi070 ceph-mon[107385]: osdmap e1290: 8 total, 8 up, 8 in 2024-09-21T10:38:41.107 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:41.107 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:41.124 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:41.124 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:41.124 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:41.127 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:41.127 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:41.127 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:41.130 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~234881024] 2024-09-21T10:38:41.130 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:41.130 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:41.132 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:41.132 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:41.132 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:41.134 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:41.134 INFO:tasks.workunit.client.0.smithi070.stdout:from 5 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:41.134 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~96468992,381681664~33554432] 2024-09-21T10:38:41.135 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:41.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:41 smithi070 ceph-mon[107385]: osdmap e1291: 8 total, 8 up, 8 in 2024-09-21T10:38:41.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:41 smithi070 ceph-mon[112139]: osdmap e1291: 8 total, 8 up, 8 in 2024-09-21T10:38:41.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:41 smithi160 ceph-mon[101879]: osdmap e1291: 8 total, 8 up, 8 in 2024-09-21T10:38:42.119 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:42.119 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:42.130 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:42.130 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:42.130 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:42.134 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:42.134 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:42.135 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:42.137 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~234881024] 2024-09-21T10:38:42.137 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:42.137 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:42.139 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:42.139 INFO:tasks.workunit.client.0.smithi070.stdout:from 4 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:42.140 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~46137344,264241152~12582912,281018368~134217728] 2024-09-21T10:38:42.141 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:42.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:42 smithi070 ceph-mon[107385]: pgmap v2085: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 197 KiB/s rd, 5.0 KiB/s wr, 324 op/s 2024-09-21T10:38:42.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:42 smithi070 ceph-mon[107385]: osdmap e1292: 8 total, 8 up, 8 in 2024-09-21T10:38:42.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:42 smithi070 ceph-mon[112139]: pgmap v2085: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 197 KiB/s rd, 5.0 KiB/s wr, 324 op/s 2024-09-21T10:38:42.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:42 smithi070 ceph-mon[112139]: osdmap e1292: 8 total, 8 up, 8 in 2024-09-21T10:38:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:42 smithi160 ceph-mon[101879]: pgmap v2085: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 197 KiB/s rd, 5.0 KiB/s wr, 324 op/s 2024-09-21T10:38:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:42 smithi160 ceph-mon[101879]: osdmap e1292: 8 total, 8 up, 8 in 2024-09-21T10:38:43.095 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:43.096 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:43.108 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:43.108 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:43.108 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:43.112 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:43.113 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:43.113 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:43.115 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~234881024] 2024-09-21T10:38:43.115 INFO:tasks.workunit.client.0.smithi070.stdout:from 3 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:43.116 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,213909504~201326592] 2024-09-21T10:38:43.117 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~20971520,213909504~205520896] 2024-09-21T10:38:43.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:38:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:38:43] "GET /metrics HTTP/1.1" 200 36487 "" "Prometheus/2.51.0" 2024-09-21T10:38:43.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:43 smithi070 ceph-mon[107385]: osdmap e1293: 8 total, 8 up, 8 in 2024-09-21T10:38:43.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:43 smithi070 ceph-mon[107385]: osdmap e1294: 8 total, 8 up, 8 in 2024-09-21T10:38:43.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:43 smithi070 ceph-mon[112139]: osdmap e1293: 8 total, 8 up, 8 in 2024-09-21T10:38:43.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:43 smithi070 ceph-mon[112139]: osdmap e1294: 8 total, 8 up, 8 in 2024-09-21T10:38:43.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:43 smithi160 ceph-mon[101879]: osdmap e1293: 8 total, 8 up, 8 in 2024-09-21T10:38:43.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:43 smithi160 ceph-mon[101879]: osdmap e1294: 8 total, 8 up, 8 in 2024-09-21T10:38:44.128 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:44.128 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:44.140 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:44.140 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:44.140 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:44.144 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:44.145 INFO:tasks.workunit.client.0.smithi070.stdout:from 2 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:44.145 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~16777216,184549376~20971520,209715200~205520896] 2024-09-21T10:38:44.146 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~16777216,184549376~234881024] 2024-09-21T10:38:44.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:44 smithi070 ceph-mon[107385]: pgmap v2088: 105 pgs: 105 active+clean; 1.0 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 216 KiB/s rd, 6.2 KiB/s wr, 364 op/s 2024-09-21T10:38:44.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:44 smithi070 ceph-mon[112139]: pgmap v2088: 105 pgs: 105 active+clean; 1.0 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 216 KiB/s rd, 6.2 KiB/s wr, 364 op/s 2024-09-21T10:38:44.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:44 smithi160 ceph-mon[101879]: pgmap v2088: 105 pgs: 105 active+clean; 1.0 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 216 KiB/s rd, 6.2 KiB/s wr, 364 op/s 2024-09-21T10:38:45.138 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:45.139 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:45.147 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:45.147 INFO:tasks.workunit.client.0.smithi070.stdout:from 1 to HEAD diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:45.147 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~25165824,134217728~4194304,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:45.149 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:45.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:45 smithi070 ceph-mon[107385]: osdmap e1295: 8 total, 8 up, 8 in 2024-09-21T10:38:45.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:45 smithi070 ceph-mon[112139]: osdmap e1295: 8 total, 8 up, 8 in 2024-09-21T10:38:45.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:45 smithi160 ceph-mon[101879]: osdmap e1295: 8 total, 8 up, 8 in 2024-09-21T10:38:46.137 INFO:tasks.workunit.client.0.smithi070.stdout:from 0 to HEAD diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:46.137 INFO:tasks.workunit.client.0.smithi070.stdout: limited diff [4194304~96468992,104857600~33554432,142606336~16777216,163577856~41943040,209715200~205520896] 2024-09-21T10:38:46.153 INFO:tasks.workunit.client.0.smithi070.stdout: actual was [4194304~96468992,104857600~54525952,163577856~255852544] 2024-09-21T10:38:46.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:46 smithi070 ceph-mon[107385]: pgmap v2091: 105 pgs: 105 active+clean; 959 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 117 KiB/s rd, 6.2 KiB/s wr, 188 op/s 2024-09-21T10:38:46.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:46 smithi070 ceph-mon[107385]: osdmap e1296: 8 total, 8 up, 8 in 2024-09-21T10:38:46.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:46 smithi070 ceph-mon[112139]: pgmap v2091: 105 pgs: 105 active+clean; 959 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 117 KiB/s rd, 6.2 KiB/s wr, 188 op/s 2024-09-21T10:38:46.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:46 smithi070 ceph-mon[112139]: osdmap e1296: 8 total, 8 up, 8 in 2024-09-21T10:38:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:46 smithi160 ceph-mon[101879]: pgmap v2091: 105 pgs: 105 active+clean; 959 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 117 KiB/s rd, 6.2 KiB/s wr, 188 op/s 2024-09-21T10:38:46.630 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:46 smithi160 ceph-mon[101879]: osdmap e1296: 8 total, 8 up, 8 in 2024-09-21T10:38:47.163 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/1.DiffIterateStress (38699 ms) 2024-09-21T10:38:47.163 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/1.DiffIterateParent 2024-09-21T10:38:47.163 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:38:47.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:47 smithi160 ceph-mon[101879]: osdmap e1297: 8 total, 8 up, 8 in 2024-09-21T10:38:47.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:47 smithi070 ceph-mon[112139]: osdmap e1297: 8 total, 8 up, 8 in 2024-09-21T10:38:47.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:47 smithi070 ceph-mon[107385]: osdmap e1297: 8 total, 8 up, 8 in 2024-09-21T10:38:48.197 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/1.DiffIterateParent (1034 ms) 2024-09-21T10:38:48.197 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/1.DiffIterateIgnoreParent 2024-09-21T10:38:48.197 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:38:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:48 smithi160 ceph-mon[101879]: pgmap v2094: 105 pgs: 105 active+clean; 884 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 109 KiB/s rd, 5.7 KiB/s wr, 173 op/s 2024-09-21T10:38:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:48 smithi160 ceph-mon[101879]: osdmap e1298: 8 total, 8 up, 8 in 2024-09-21T10:38:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:48 smithi160 ceph-mon[101879]: osdmap e1299: 8 total, 8 up, 8 in 2024-09-21T10:38:48.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:48 smithi070 ceph-mon[112139]: pgmap v2094: 105 pgs: 105 active+clean; 884 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 109 KiB/s rd, 5.7 KiB/s wr, 173 op/s 2024-09-21T10:38:48.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:48 smithi070 ceph-mon[112139]: osdmap e1298: 8 total, 8 up, 8 in 2024-09-21T10:38:48.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:48 smithi070 ceph-mon[112139]: osdmap e1299: 8 total, 8 up, 8 in 2024-09-21T10:38:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:48 smithi070 ceph-mon[107385]: pgmap v2094: 105 pgs: 105 active+clean; 884 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 109 KiB/s rd, 5.7 KiB/s wr, 173 op/s 2024-09-21T10:38:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:48 smithi070 ceph-mon[107385]: osdmap e1298: 8 total, 8 up, 8 in 2024-09-21T10:38:48.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:48 smithi070 ceph-mon[107385]: osdmap e1299: 8 total, 8 up, 8 in 2024-09-21T10:38:49.270 INFO:tasks.workunit.client.0.smithi070.stdout: wrote [0~20971520] to clone 2024-09-21T10:38:49.270 INFO:tasks.workunit.client.0.smithi070.stdout: diff was [0~20971520] 2024-09-21T10:38:49.274 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/1.DiffIterateIgnoreParent (1077 ms) 2024-09-21T10:38:49.274 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/1.DiffIterateCallbackError 2024-09-21T10:38:49.274 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:38:49.294 INFO:tasks.workunit.client.0.smithi070.stdout: wrote [3080679~79594,7332043~53156,10031654~32750,10214008~19838,10368725~17318,11529317~32877,12647529~21412,13224359~84952,18477119~26451,19509117~5477] 2024-09-21T10:38:49.305 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/1.DiffIterateCallbackError (31 ms) 2024-09-21T10:38:49.305 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/1.DiffIterateParentDiscard 2024-09-21T10:38:49.305 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:38:50.394 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:50 smithi160 ceph-mon[101879]: pgmap v2097: 105 pgs: 105 active+clean; 884 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 21 KiB/s rd, 1.0 MiB/s wr, 40 op/s 2024-09-21T10:38:50.394 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:50 smithi160 ceph-mon[101879]: osdmap e1300: 8 total, 8 up, 8 in 2024-09-21T10:38:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:50 smithi070 ceph-mon[107385]: pgmap v2097: 105 pgs: 105 active+clean; 884 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 21 KiB/s rd, 1.0 MiB/s wr, 40 op/s 2024-09-21T10:38:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:50 smithi070 ceph-mon[107385]: osdmap e1300: 8 total, 8 up, 8 in 2024-09-21T10:38:50.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:50 smithi070 ceph-mon[112139]: pgmap v2097: 105 pgs: 105 active+clean; 884 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 21 KiB/s rd, 1.0 MiB/s wr, 40 op/s 2024-09-21T10:38:50.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:50 smithi070 ceph-mon[112139]: osdmap e1300: 8 total, 8 up, 8 in 2024-09-21T10:38:51.245 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/1.DiffIterateParentDiscard (1940 ms) 2024-09-21T10:38:51.245 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/1.DiffIterateUnalignedSmall 2024-09-21T10:38:51.245 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:38:51.300 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/1.DiffIterateUnalignedSmall (55 ms) 2024-09-21T10:38:51.300 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/1.DiffIterateUnaligned 2024-09-21T10:38:51.300 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:38:51.360 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/1.DiffIterateUnaligned (60 ms) 2024-09-21T10:38:51.360 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/1.DiffIterateTryAcquireLock 2024-09-21T10:38:51.360 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-21T10:38:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:51 smithi160 ceph-mon[101879]: osdmap e1301: 8 total, 8 up, 8 in 2024-09-21T10:38:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:38:51.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:51 smithi070 ceph-mon[112139]: osdmap e1301: 8 total, 8 up, 8 in 2024-09-21T10:38:51.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:38:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:51 smithi070 ceph-mon[107385]: osdmap e1301: 8 total, 8 up, 8 in 2024-09-21T10:38:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:38:52.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:52 smithi160 ceph-mon[101879]: pgmap v2100: 105 pgs: 105 active+clean; 861 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 5.2 MiB/s wr, 110 op/s 2024-09-21T10:38:52.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:52 smithi160 ceph-mon[101879]: osdmap e1302: 8 total, 8 up, 8 in 2024-09-21T10:38:52.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:52 smithi070 ceph-mon[112139]: pgmap v2100: 105 pgs: 105 active+clean; 861 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 5.2 MiB/s wr, 110 op/s 2024-09-21T10:38:52.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:52 smithi070 ceph-mon[112139]: osdmap e1302: 8 total, 8 up, 8 in 2024-09-21T10:38:52.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:52 smithi070 ceph-mon[107385]: pgmap v2100: 105 pgs: 105 active+clean; 861 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 5.2 MiB/s wr, 110 op/s 2024-09-21T10:38:52.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:52 smithi070 ceph-mon[107385]: osdmap e1302: 8 total, 8 up, 8 in 2024-09-21T10:38:53.528 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:38:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:38:53] "GET /metrics HTTP/1.1" 200 36485 "" "Prometheus/2.51.0" 2024-09-21T10:38:54.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:54 smithi160 ceph-mon[101879]: pgmap v2102: 105 pgs: 105 active+clean; 849 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.6 MiB/s rd, 9.5 MiB/s wr, 246 op/s 2024-09-21T10:38:54.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:54 smithi160 ceph-mon[101879]: osdmap e1303: 8 total, 8 up, 8 in 2024-09-21T10:38:54.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:54 smithi070 ceph-mon[107385]: pgmap v2102: 105 pgs: 105 active+clean; 849 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.6 MiB/s rd, 9.5 MiB/s wr, 246 op/s 2024-09-21T10:38:54.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:54 smithi070 ceph-mon[107385]: osdmap e1303: 8 total, 8 up, 8 in 2024-09-21T10:38:54.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:54 smithi070 ceph-mon[112139]: pgmap v2102: 105 pgs: 105 active+clean; 849 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.6 MiB/s rd, 9.5 MiB/s wr, 246 op/s 2024-09-21T10:38:54.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:54 smithi070 ceph-mon[112139]: osdmap e1303: 8 total, 8 up, 8 in 2024-09-21T10:38:56.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:56 smithi160 ceph-mon[101879]: pgmap v2104: 105 pgs: 105 active+clean; 871 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 4.5 MiB/s rd, 8.6 MiB/s wr, 239 op/s 2024-09-21T10:38:56.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:56 smithi070 ceph-mon[107385]: pgmap v2104: 105 pgs: 105 active+clean; 871 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 4.5 MiB/s rd, 8.6 MiB/s wr, 239 op/s 2024-09-21T10:38:56.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:56 smithi070 ceph-mon[112139]: pgmap v2104: 105 pgs: 105 active+clean; 871 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 4.5 MiB/s rd, 8.6 MiB/s wr, 239 op/s 2024-09-21T10:38:56.400 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] DiffIterateTest/1.DiffIterateTryAcquireLock (5040 ms) 2024-09-21T10:38:56.400 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] DiffIterateTest/1.DiffIterateStriping 2024-09-21T10:38:56.400 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/librbd/test_librbd.cc:8958: Skipped 2024-09-21T10:38:56.400 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-21T10:38:56.400 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] DiffIterateTest/1.DiffIterateStriping (0 ms) 2024-09-21T10:38:57.096 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 19 tests from DiffIterateTest/1 (258325 ms total) 2024-09-21T10:38:57.097 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:38:57.097 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] Global test environment tear-down 2024-09-21T10:38:57.097 INFO:tasks.workunit.client.0.smithi070.stdout:[==========] 490 tests from 24 test suites ran. (2571085 ms total) 2024-09-21T10:38:57.097 INFO:tasks.workunit.client.0.smithi070.stdout:[ PASSED ] 444 tests. 2024-09-21T10:38:57.097 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] 46 tests, listed below: 2024-09-21T10:38:57.097 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalStress.DiscardWithPruneWriteOverlap 2024-09-21T10:38:57.097 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.AioDiscardEvent 2024-09-21T10:38:57.098 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.AioWriteEvent 2024-09-21T10:38:57.098 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.AioFlushEvent 2024-09-21T10:38:57.098 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.SnapCreate 2024-09-21T10:38:57.098 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.SnapProtect 2024-09-21T10:38:57.098 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.SnapUnprotect 2024-09-21T10:38:57.098 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.SnapRename 2024-09-21T10:38:57.098 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.SnapRollback 2024-09-21T10:38:57.098 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.SnapRemove 2024-09-21T10:38:57.098 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.Rename 2024-09-21T10:38:57.098 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.Resize 2024-09-21T10:38:57.099 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.Flatten 2024-09-21T10:38:57.099 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.UpdateFeatures 2024-09-21T10:38:57.099 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.MetadataSet 2024-09-21T10:38:57.099 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.MetadataRemove 2024-09-21T10:38:57.099 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalReplay.ObjectPosition 2024-09-21T10:38:57.099 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalEntries.AioWrite 2024-09-21T10:38:57.099 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalEntries.AioDiscard 2024-09-21T10:38:57.099 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalEntries.AioDiscardWithPrune 2024-09-21T10:38:57.099 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestJournalEntries.AioFlush 2024-09-21T10:38:57.099 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestOperations.DisableJournalingCorrupt 2024-09-21T10:38:57.099 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestMigration.MirroringSamePool 2024-09-21T10:38:57.100 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestMigration.MirroringAbort 2024-09-21T10:38:57.100 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestMigration.MirroringOtherPoolDisabled 2024-09-21T10:38:57.100 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestMigration.MirroringOtherPoolEnabled 2024-09-21T10:38:57.100 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestMigration.MirroringPool 2024-09-21T10:38:57.101 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestDeepCopy.NoSnaps_StrippingLargerDstObjSize 2024-09-21T10:38:57.101 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestDeepCopy.Snaps_StrippingLargerDstObjSize 2024-09-21T10:38:57.101 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestDeepCopy.Clone_StrippingLargerDstObjSize 2024-09-21T10:38:57.102 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestDeepCopy.CloneFlatten_StrippingLargerDstObjSize 2024-09-21T10:38:57.102 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestDeepCopy.Stress_StrippingLargerDstObjSize 2024-09-21T10:38:57.102 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestDeepCopy.NoSnaps_StrippingSmallerDstObjSize 2024-09-21T10:38:57.102 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestDeepCopy.Snaps_StrippingSmallerDstObjSize 2024-09-21T10:38:57.102 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestDeepCopy.Clone_StrippingSmallerDstObjSize 2024-09-21T10:38:57.102 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestDeepCopy.CloneFlatten_StrippingSmallerDstObjSize 2024-09-21T10:38:57.102 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestDeepCopy.Stress_StrippingSmallerDstObjSize 2024-09-21T10:38:57.102 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestMirroring.DisableImageMirrorWithPeer 2024-09-21T10:38:57.103 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestMirroring.DisableJournalingWithPeer 2024-09-21T10:38:57.103 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestInternal.DiscardCopyup 2024-09-21T10:38:57.103 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestInternal.TestCoR 2024-09-21T10:38:57.103 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadBadStripePattern 2024-09-21T10:38:57.103 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestLibRBD.EnableJournalingViaLockOwner 2024-09-21T10:38:57.103 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLockReadTransition 2024-09-21T10:38:57.103 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] DiffIterateTest/0.DiffIterateStriping 2024-09-21T10:38:57.104 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] DiffIterateTest/1.DiffIterateStriping 2024-09-21T10:38:57.104 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:38:57.104 INFO:tasks.workunit.client.0.smithi070.stdout: YOU HAVE 2 DISABLED TESTS 2024-09-21T10:38:57.104 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T10:38:57.106 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-21T10:38:57.106 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-21T10:38:57.176 INFO:tasks.workunit:Stopping ['rbd/test_librbd.sh'] on client.0... 2024-09-21T10:38:57.177 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-21T10:38:57.432 DEBUG:teuthology.parallel:result is None 2024-09-21T10:38:57.432 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T10:38:57.457 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T10:38:57.457 DEBUG:teuthology.orchestra.run.smithi070:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-21T10:38:57.529 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T10:38:57.529 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-21T10:38:57.529 INFO:teuthology.task.print:**** done rbd/test_librbd.sh 7-workload 2024-09-21T10:38:57.530 DEBUG:teuthology.parallel:result is None 2024-09-21T10:38:57.530 INFO:teuthology.run_tasks:Running task print... 2024-09-21T10:38:57.545 INFO:teuthology.task.print:**** done upgrading first half of cluster **** 2024-09-21T10:38:57.545 INFO:teuthology.run_tasks:Running task ceph.healthy... 2024-09-21T10:38:57.577 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T10:38:57.577 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-21T10:38:57.577 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph mgr dump --format=json 2024-09-21T10:38:57.771 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:38:58.175 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:58 smithi070 ceph-mon[112139]: pgmap v2105: 105 pgs: 105 active+clean; 893 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 7.9 MiB/s wr, 213 op/s 2024-09-21T10:38:58.176 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:38:58 smithi070 ceph-mon[112139]: osdmap e1304: 8 total, 8 up, 8 in 2024-09-21T10:38:58.176 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:58 smithi070 ceph-mon[107385]: pgmap v2105: 105 pgs: 105 active+clean; 893 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 7.9 MiB/s wr, 213 op/s 2024-09-21T10:38:58.176 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:38:58 smithi070 ceph-mon[107385]: osdmap e1304: 8 total, 8 up, 8 in 2024-09-21T10:38:58.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:58 smithi160 ceph-mon[101879]: pgmap v2105: 105 pgs: 105 active+clean; 893 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 7.9 MiB/s wr, 213 op/s 2024-09-21T10:38:58.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:38:58 smithi160 ceph-mon[101879]: osdmap e1304: 8 total, 8 up, 8 in 2024-09-21T10:38:59.841 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:39:00.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:00 smithi070 ceph-mon[107385]: pgmap v2107: 73 pgs: 73 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:39:00.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:00 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/519209932' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T10:39:00.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:00 smithi070 ceph-mon[112139]: pgmap v2107: 73 pgs: 73 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:39:00.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:00 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/519209932' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T10:39:00.423 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":40,"flags":0,"active_gid":34107,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":608499648},{"type":"v1","addr":"172.21.15.70:6801","nonce":608499648}]},"active_addr":"172.21.15.70:6801/608499648","active_change":"2024-09-21T10:00:49.897544+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":25163,"name":"x","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.70:8443/","prometheus":"http://172.21.15.70:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":344,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":3074500387}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":501225570}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":2007327347}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":2575397231}]}]} 2024-09-21T10:39:00.427 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-21T10:39:00.427 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T10:39:00.427 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T10:39:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:00 smithi160 ceph-mon[101879]: pgmap v2107: 73 pgs: 73 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:39:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:00 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/519209932' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T10:39:00.623 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:02.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:02 smithi070 ceph-mon[107385]: pgmap v2108: 73 pgs: 73 active+clean; 582 KiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:39:02.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:02 smithi070 ceph-mon[112139]: pgmap v2108: 73 pgs: 73 active+clean; 582 KiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:39:02.540 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:39:02.540 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":1304,"fsid":"7c68bfbc-77fe-11ef-baf6-efdc52797490","created":"2024-09-21T09:47:59.216934+0000","modified":"2024-09-21T10:38:57.083270+0000","last_up_change":"2024-09-21T10:09:05.480343+0000","last_in_change":"2024-09-21T09:52:44.345943+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":52,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:51:05.514472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:53:41.830233+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-21T09:54:52.551642+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-21T09:54:52.861791+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6870ff8-14eb-4110-957a-00c73645b181","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":660,"up_from":666,"up_thru":1156,"down_at":660,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":452014366},{"type":"v1","addr":"172.21.15.70:6803","nonce":452014366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":452014366},{"type":"v1","addr":"172.21.15.70:6805","nonce":452014366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":452014366},{"type":"v1","addr":"172.21.15.70:6809","nonce":452014366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":452014366},{"type":"v1","addr":"172.21.15.70:6807","nonce":452014366}]},"public_addr":"172.21.15.70:6803/452014366","cluster_addr":"172.21.15.70:6805/452014366","heartbeat_back_addr":"172.21.15.70:6809/452014366","heartbeat_front_addr":"172.21.15.70:6807/452014366","state":["exists","up"]},{"osd":1,"uuid":"b1349e31-7493-489f-823a-55edb866d25f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":669,"up_from":674,"up_thru":1156,"down_at":669,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3938187175},{"type":"v1","addr":"172.21.15.70:6811","nonce":3938187175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3938187175},{"type":"v1","addr":"172.21.15.70:6813","nonce":3938187175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":3938187175},{"type":"v1","addr":"172.21.15.70:6817","nonce":3938187175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3938187175},{"type":"v1","addr":"172.21.15.70:6815","nonce":3938187175}]},"public_addr":"172.21.15.70:6811/3938187175","cluster_addr":"172.21.15.70:6813/3938187175","heartbeat_back_addr":"172.21.15.70:6817/3938187175","heartbeat_front_addr":"172.21.15.70:6815/3938187175","state":["exists","up"]},{"osd":2,"uuid":"ac4496ba-a860-4b09-bb38-7f433e081541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":677,"up_from":680,"up_thru":1156,"down_at":677,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":890146607},{"type":"v1","addr":"172.21.15.70:6819","nonce":890146607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":890146607},{"type":"v1","addr":"172.21.15.70:6821","nonce":890146607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":890146607},{"type":"v1","addr":"172.21.15.70:6825","nonce":890146607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":890146607},{"type":"v1","addr":"172.21.15.70:6823","nonce":890146607}]},"public_addr":"172.21.15.70:6819/890146607","cluster_addr":"172.21.15.70:6821/890146607","heartbeat_back_addr":"172.21.15.70:6825/890146607","heartbeat_front_addr":"172.21.15.70:6823/890146607","state":["exists","up"]},{"osd":3,"uuid":"1eedf4ce-057c-4576-ae5b-51d3595df195","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":689,"up_from":692,"up_thru":1156,"down_at":689,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3858266770},{"type":"v1","addr":"172.21.15.70:6827","nonce":3858266770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3858266770},{"type":"v1","addr":"172.21.15.70:6829","nonce":3858266770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3858266770},{"type":"v1","addr":"172.21.15.70:6833","nonce":3858266770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3858266770},{"type":"v1","addr":"172.21.15.70:6831","nonce":3858266770}]},"public_addr":"172.21.15.70:6827/3858266770","cluster_addr":"172.21.15.70:6829/3858266770","heartbeat_back_addr":"172.21.15.70:6833/3858266770","heartbeat_front_addr":"172.21.15.70:6831/3858266770","state":["exists","up"]},{"osd":4,"uuid":"a4b728b8-97a4-478f-8a11-ba590c0213e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":1156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6801","nonce":3634505869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6803","nonce":3634505869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6807","nonce":3634505869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6805","nonce":3634505869}]},"public_addr":"172.21.15.160:6801/3634505869","cluster_addr":"172.21.15.160:6803/3634505869","heartbeat_back_addr":"172.21.15.160:6807/3634505869","heartbeat_front_addr":"172.21.15.160:6805/3634505869","state":["exists","up"]},{"osd":5,"uuid":"56915347-9254-4e94-b202-1f16d09e7946","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":1156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6809","nonce":2006621188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6811","nonce":2006621188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6815","nonce":2006621188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6813","nonce":2006621188}]},"public_addr":"172.21.15.160:6809/2006621188","cluster_addr":"172.21.15.160:6811/2006621188","heartbeat_back_addr":"172.21.15.160:6815/2006621188","heartbeat_front_addr":"172.21.15.160:6813/2006621188","state":["exists","up"]},{"osd":6,"uuid":"79e6de8c-b78b-40e2-9229-0275a873d40e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":1156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6817","nonce":1949075400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6819","nonce":1949075400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6823","nonce":1949075400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6821","nonce":1949075400}]},"public_addr":"172.21.15.160:6817/1949075400","cluster_addr":"172.21.15.160:6819/1949075400","heartbeat_back_addr":"172.21.15.160:6823/1949075400","heartbeat_front_addr":"172.21.15.160:6821/1949075400","state":["exists","up"]},{"osd":7,"uuid":"72245c8a-2fdb-4445-b378-16569f337fcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":1156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6825","nonce":601673919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6827","nonce":601673919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6831","nonce":601673919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6829","nonce":601673919}]},"public_addr":"172.21.15.160:6825/601673919","cluster_addr":"172.21.15.160:6827/601673919","heartbeat_back_addr":"172.21.15.160:6831/601673919","heartbeat_front_addr":"172.21.15.160:6829/601673919","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:15.134515+0000","dead_epoch":660},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:39.191791+0000","dead_epoch":669},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:51:02.934581+0000","dead_epoch":676},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:51:26.514393+0000","dead_epoch":689},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:51:48.329249+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:12.268576+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:36.370859+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:59.888857+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/4293543511":"2024-09-21T11:04:19.859325+0000","172.21.15.70:0/2422341096":"2024-09-22T10:00:49.897297+0000","172.21.15.160:0/1742426098":"2024-09-22T09:58:10.497359+0000","172.21.15.70:0/1905911777":"2024-09-22T09:53:31.134530+0000","172.21.15.160:6834/2915531836":"2024-09-22T09:58:10.497359+0000","172.21.15.70:0/4150778424":"2024-09-22T09:48:32.256720+0000","172.21.15.70:0/2308708361":"2024-09-22T10:00:49.897297+0000","172.21.15.70:6800/177993484":"2024-09-22T09:48:32.256720+0000","172.21.15.70:6800/1520336634":"2024-09-22T09:53:31.134530+0000","172.21.15.160:0/1014888319":"2024-09-22T09:58:10.497359+0000","172.21.15.70:6800/2142025682":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/1469701388":"2024-09-22T09:53:31.134530+0000","172.21.15.70:6801/603216094":"2024-09-22T10:00:49.897297+0000","172.21.15.160:0/19285887":"2024-09-22T09:58:10.497359+0000","172.21.15.70:6801/177993484":"2024-09-22T09:48:32.256720+0000","172.21.15.160:6835/2915531836":"2024-09-22T09:58:10.497359+0000","172.21.15.70:0/2521911561":"2024-09-22T09:48:32.256720+0000","172.21.15.70:0/2918782890":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/1012316198":"2024-09-22T09:48:32.256720+0000","172.21.15.70:0/1102418977":"2024-09-21T11:13:38.558014+0000","172.21.15.70:0/749465179":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/595094674":"2024-09-22T09:57:34.443100+0000","172.21.15.70:0/2386870039":"2024-09-22T09:57:34.443100+0000","172.21.15.70:0/3159615643":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/1066794139":"2024-09-22T09:53:31.134530+0000","172.21.15.70:0/2110605013":"2024-09-22T09:57:34.443100+0000","172.21.15.70:6801/1520336634":"2024-09-22T09:53:31.134530+0000","172.21.15.160:0/2751391809":"2024-09-22T09:58:10.497359+0000","172.21.15.160:0/3012862160":"2024-09-22T09:58:10.497359+0000","172.21.15.70:0/752126770":"2024-09-22T10:00:49.897297+0000","172.21.15.70:6801/2142025682":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/1635833554":"2024-09-22T10:00:49.897297+0000","172.21.15.70:0/694065139":"2024-09-21T11:04:18.868322+0000","172.21.15.70:6800/3639085022":"2024-09-22T09:57:34.443100+0000","172.21.15.70:0/2141342771":"2024-09-22T09:53:31.134530+0000","172.21.15.70:6801/3639085022":"2024-09-22T09:57:34.443100+0000","172.21.15.70:0/47418078":"2024-09-22T09:57:34.443100+0000","172.21.15.160:0/2880106921":"2024-09-22T09:58:10.497359+0000","172.21.15.70:6800/603216094":"2024-09-22T10:00:49.897297+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":1}]},{"pool":6,"snaps":[{"begin":2,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":1}]},{"pool":8,"snaps":[{"begin":2,"length":1}]},{"pool":9,"snaps":[{"begin":2,"length":1}]},{"pool":10,"snaps":[{"begin":2,"length":1}]},{"pool":11,"snaps":[{"begin":2,"length":1}]},{"pool":14,"snaps":[{"begin":7,"length":2}]},{"pool":16,"snaps":[{"begin":35,"length":3},{"begin":45,"length":3}]},{"pool":17,"snaps":[{"begin":22,"length":1}]},{"pool":18,"snaps":[{"begin":2,"length":1}]},{"pool":52,"snaps":[{"begin":104,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T10:39:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:02 smithi160 ceph-mon[101879]: pgmap v2108: 73 pgs: 73 active+clean; 582 KiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:39:03.155 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:03 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2857603701' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:39:03.155 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:03 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2857603701' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:39:03.286 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T10:39:03.287 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T10:39:03.481 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:03.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:03 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2857603701' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:39:03.601 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:39:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:39:03] "GET /metrics HTTP/1.1" 200 36485 "" "Prometheus/2.51.0" 2024-09-21T10:39:04.306 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:04 smithi070 ceph-mon[107385]: pgmap v2109: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:04.306 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:04 smithi070 ceph-mon[112139]: pgmap v2109: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:04 smithi160 ceph-mon[101879]: pgmap v2109: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:05.265 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:39:05.265 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":1304,"fsid":"7c68bfbc-77fe-11ef-baf6-efdc52797490","created":"2024-09-21T09:47:59.216934+0000","modified":"2024-09-21T10:38:57.083270+0000","last_up_change":"2024-09-21T10:09:05.480343+0000","last_in_change":"2024-09-21T09:52:44.345943+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":52,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:51:05.514472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:53:41.830233+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-21T09:54:52.551642+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-21T09:54:52.861791+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6870ff8-14eb-4110-957a-00c73645b181","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":660,"up_from":666,"up_thru":1156,"down_at":660,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":452014366},{"type":"v1","addr":"172.21.15.70:6803","nonce":452014366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":452014366},{"type":"v1","addr":"172.21.15.70:6805","nonce":452014366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":452014366},{"type":"v1","addr":"172.21.15.70:6809","nonce":452014366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":452014366},{"type":"v1","addr":"172.21.15.70:6807","nonce":452014366}]},"public_addr":"172.21.15.70:6803/452014366","cluster_addr":"172.21.15.70:6805/452014366","heartbeat_back_addr":"172.21.15.70:6809/452014366","heartbeat_front_addr":"172.21.15.70:6807/452014366","state":["exists","up"]},{"osd":1,"uuid":"b1349e31-7493-489f-823a-55edb866d25f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":669,"up_from":674,"up_thru":1156,"down_at":669,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3938187175},{"type":"v1","addr":"172.21.15.70:6811","nonce":3938187175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3938187175},{"type":"v1","addr":"172.21.15.70:6813","nonce":3938187175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":3938187175},{"type":"v1","addr":"172.21.15.70:6817","nonce":3938187175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3938187175},{"type":"v1","addr":"172.21.15.70:6815","nonce":3938187175}]},"public_addr":"172.21.15.70:6811/3938187175","cluster_addr":"172.21.15.70:6813/3938187175","heartbeat_back_addr":"172.21.15.70:6817/3938187175","heartbeat_front_addr":"172.21.15.70:6815/3938187175","state":["exists","up"]},{"osd":2,"uuid":"ac4496ba-a860-4b09-bb38-7f433e081541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":677,"up_from":680,"up_thru":1156,"down_at":677,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":890146607},{"type":"v1","addr":"172.21.15.70:6819","nonce":890146607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":890146607},{"type":"v1","addr":"172.21.15.70:6821","nonce":890146607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":890146607},{"type":"v1","addr":"172.21.15.70:6825","nonce":890146607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":890146607},{"type":"v1","addr":"172.21.15.70:6823","nonce":890146607}]},"public_addr":"172.21.15.70:6819/890146607","cluster_addr":"172.21.15.70:6821/890146607","heartbeat_back_addr":"172.21.15.70:6825/890146607","heartbeat_front_addr":"172.21.15.70:6823/890146607","state":["exists","up"]},{"osd":3,"uuid":"1eedf4ce-057c-4576-ae5b-51d3595df195","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":689,"up_from":692,"up_thru":1156,"down_at":689,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3858266770},{"type":"v1","addr":"172.21.15.70:6827","nonce":3858266770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3858266770},{"type":"v1","addr":"172.21.15.70:6829","nonce":3858266770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3858266770},{"type":"v1","addr":"172.21.15.70:6833","nonce":3858266770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3858266770},{"type":"v1","addr":"172.21.15.70:6831","nonce":3858266770}]},"public_addr":"172.21.15.70:6827/3858266770","cluster_addr":"172.21.15.70:6829/3858266770","heartbeat_back_addr":"172.21.15.70:6833/3858266770","heartbeat_front_addr":"172.21.15.70:6831/3858266770","state":["exists","up"]},{"osd":4,"uuid":"a4b728b8-97a4-478f-8a11-ba590c0213e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":1156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6801","nonce":3634505869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6803","nonce":3634505869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6807","nonce":3634505869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6805","nonce":3634505869}]},"public_addr":"172.21.15.160:6801/3634505869","cluster_addr":"172.21.15.160:6803/3634505869","heartbeat_back_addr":"172.21.15.160:6807/3634505869","heartbeat_front_addr":"172.21.15.160:6805/3634505869","state":["exists","up"]},{"osd":5,"uuid":"56915347-9254-4e94-b202-1f16d09e7946","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":1156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6809","nonce":2006621188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6811","nonce":2006621188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6815","nonce":2006621188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":2006621188},{"type":"v1","addr":"172.21.15.160:6813","nonce":2006621188}]},"public_addr":"172.21.15.160:6809/2006621188","cluster_addr":"172.21.15.160:6811/2006621188","heartbeat_back_addr":"172.21.15.160:6815/2006621188","heartbeat_front_addr":"172.21.15.160:6813/2006621188","state":["exists","up"]},{"osd":6,"uuid":"79e6de8c-b78b-40e2-9229-0275a873d40e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":1156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6817","nonce":1949075400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6819","nonce":1949075400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6823","nonce":1949075400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":1949075400},{"type":"v1","addr":"172.21.15.160:6821","nonce":1949075400}]},"public_addr":"172.21.15.160:6817/1949075400","cluster_addr":"172.21.15.160:6819/1949075400","heartbeat_back_addr":"172.21.15.160:6823/1949075400","heartbeat_front_addr":"172.21.15.160:6821/1949075400","state":["exists","up"]},{"osd":7,"uuid":"72245c8a-2fdb-4445-b378-16569f337fcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":1156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6825","nonce":601673919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6827","nonce":601673919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6831","nonce":601673919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6829","nonce":601673919}]},"public_addr":"172.21.15.160:6825/601673919","cluster_addr":"172.21.15.160:6827/601673919","heartbeat_back_addr":"172.21.15.160:6831/601673919","heartbeat_front_addr":"172.21.15.160:6829/601673919","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:15.134515+0000","dead_epoch":660},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:39.191791+0000","dead_epoch":669},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:51:02.934581+0000","dead_epoch":676},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:51:26.514393+0000","dead_epoch":689},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:51:48.329249+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:12.268576+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:36.370859+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:59.888857+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/4293543511":"2024-09-21T11:04:19.859325+0000","172.21.15.70:0/2422341096":"2024-09-22T10:00:49.897297+0000","172.21.15.160:0/1742426098":"2024-09-22T09:58:10.497359+0000","172.21.15.70:0/1905911777":"2024-09-22T09:53:31.134530+0000","172.21.15.160:6834/2915531836":"2024-09-22T09:58:10.497359+0000","172.21.15.70:0/4150778424":"2024-09-22T09:48:32.256720+0000","172.21.15.70:0/2308708361":"2024-09-22T10:00:49.897297+0000","172.21.15.70:6800/177993484":"2024-09-22T09:48:32.256720+0000","172.21.15.70:6800/1520336634":"2024-09-22T09:53:31.134530+0000","172.21.15.160:0/1014888319":"2024-09-22T09:58:10.497359+0000","172.21.15.70:6800/2142025682":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/1469701388":"2024-09-22T09:53:31.134530+0000","172.21.15.70:6801/603216094":"2024-09-22T10:00:49.897297+0000","172.21.15.160:0/19285887":"2024-09-22T09:58:10.497359+0000","172.21.15.70:6801/177993484":"2024-09-22T09:48:32.256720+0000","172.21.15.160:6835/2915531836":"2024-09-22T09:58:10.497359+0000","172.21.15.70:0/2521911561":"2024-09-22T09:48:32.256720+0000","172.21.15.70:0/2918782890":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/1012316198":"2024-09-22T09:48:32.256720+0000","172.21.15.70:0/1102418977":"2024-09-21T11:13:38.558014+0000","172.21.15.70:0/749465179":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/595094674":"2024-09-22T09:57:34.443100+0000","172.21.15.70:0/2386870039":"2024-09-22T09:57:34.443100+0000","172.21.15.70:0/3159615643":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/1066794139":"2024-09-22T09:53:31.134530+0000","172.21.15.70:0/2110605013":"2024-09-22T09:57:34.443100+0000","172.21.15.70:6801/1520336634":"2024-09-22T09:53:31.134530+0000","172.21.15.160:0/2751391809":"2024-09-22T09:58:10.497359+0000","172.21.15.160:0/3012862160":"2024-09-22T09:58:10.497359+0000","172.21.15.70:0/752126770":"2024-09-22T10:00:49.897297+0000","172.21.15.70:6801/2142025682":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/1635833554":"2024-09-22T10:00:49.897297+0000","172.21.15.70:0/694065139":"2024-09-21T11:04:18.868322+0000","172.21.15.70:6800/3639085022":"2024-09-22T09:57:34.443100+0000","172.21.15.70:0/2141342771":"2024-09-22T09:53:31.134530+0000","172.21.15.70:6801/3639085022":"2024-09-22T09:57:34.443100+0000","172.21.15.70:0/47418078":"2024-09-22T09:57:34.443100+0000","172.21.15.160:0/2880106921":"2024-09-22T09:58:10.497359+0000","172.21.15.70:6800/603216094":"2024-09-22T10:00:49.897297+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":1}]},{"pool":6,"snaps":[{"begin":2,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":1}]},{"pool":8,"snaps":[{"begin":2,"length":1}]},{"pool":9,"snaps":[{"begin":2,"length":1}]},{"pool":10,"snaps":[{"begin":2,"length":1}]},{"pool":11,"snaps":[{"begin":2,"length":1}]},{"pool":14,"snaps":[{"begin":7,"length":2}]},{"pool":16,"snaps":[{"begin":35,"length":3},{"begin":45,"length":3}]},{"pool":17,"snaps":[{"begin":22,"length":1}]},{"pool":18,"snaps":[{"begin":2,"length":1}]},{"pool":52,"snaps":[{"begin":104,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T10:39:05.899 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T10:39:05.900 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T10:39:05.900 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T10:39:05.901 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T10:39:05.901 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T10:39:05.901 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T10:39:05.902 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T10:39:05.902 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T10:39:06.339 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:06.345 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:06.345 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:06.347 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:06.347 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:06.351 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:06.355 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:06.359 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:06.380 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:06 smithi070 ceph-mon[107385]: pgmap v2110: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:06.380 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:06 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2742187447' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:39:06.380 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:39:06.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:06 smithi070 ceph-mon[112139]: pgmap v2110: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:06.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:06 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2742187447' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:39:06.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:39:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:06 smithi160 ceph-mon[101879]: pgmap v2110: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:06 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2742187447' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:39:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:39:08.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:08 smithi070 ceph-mon[112139]: pgmap v2111: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:08.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:08 smithi070 ceph-mon[107385]: pgmap v2111: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:08 smithi160 ceph-mon[101879]: pgmap v2111: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:10.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:10 smithi070 ceph-mon[107385]: pgmap v2112: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:10.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:10 smithi070 ceph-mon[112139]: pgmap v2112: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:10 smithi160 ceph-mon[101879]: pgmap v2112: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:12.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:12 smithi070 ceph-mon[107385]: pgmap v2113: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:12.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:12 smithi070 ceph-mon[112139]: pgmap v2113: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:12.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:12 smithi160 ceph-mon[101879]: pgmap v2113: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:13.378 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:39:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:39:13] "GET /metrics HTTP/1.1" 200 34046 "" "Prometheus/2.51.0" 2024-09-21T10:39:14.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:14 smithi070 ceph-mon[107385]: pgmap v2114: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:14.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:14 smithi070 ceph-mon[112139]: pgmap v2114: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:14.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:14 smithi160 ceph-mon[101879]: pgmap v2114: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:14.920 INFO:teuthology.orchestra.run.smithi070.stdout:197568496179 2024-09-21T10:39:14.921 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T10:39:15.858 INFO:teuthology.orchestra.run.smithi070.stdout:163208757816 2024-09-21T10:39:15.858 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T10:39:16.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:16 smithi070 ceph-mon[107385]: pgmap v2115: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:16.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:16 smithi070 ceph-mon[112139]: pgmap v2115: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:16 smithi160 ceph-mon[101879]: pgmap v2115: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:18.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:18 smithi070 ceph-mon[112139]: pgmap v2116: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:18.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:18 smithi070 ceph-mon[107385]: pgmap v2116: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:18 smithi160 ceph-mon[101879]: pgmap v2116: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:19.642 INFO:teuthology.orchestra.run.smithi070.stdout:231928234542 2024-09-21T10:39:19.643 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T10:39:20.312 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:20 smithi070 ceph-mon[107385]: pgmap v2117: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:20.312 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:20 smithi070 ceph-mon[112139]: pgmap v2117: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:20.333 INFO:teuthology.orchestra.run.smithi070.stdout:128849019453 2024-09-21T10:39:20.333 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T10:39:20.393 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:20 smithi160 ceph-mon[101879]: pgmap v2117: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:21.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:39:21.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:39:21.444 INFO:teuthology.orchestra.run.smithi070.stdout:2860448219522 2024-09-21T10:39:21.444 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T10:39:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:39:22.407 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:22 smithi070 ceph-mon[107385]: pgmap v2118: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:22.407 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:22 smithi070 ceph-mon[112139]: pgmap v2118: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:22.428 INFO:teuthology.orchestra.run.smithi070.stdout:2894807957883 2024-09-21T10:39:22.428 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T10:39:22.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:22 smithi160 ceph-mon[101879]: pgmap v2118: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:23.388 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:39:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:39:23] "GET /metrics HTTP/1.1" 200 34048 "" "Prometheus/2.51.0" 2024-09-21T10:39:23.390 INFO:teuthology.orchestra.run.smithi070.stdout:2972117369196 2024-09-21T10:39:23.390 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T10:39:23.444 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:23.444 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:23.812 INFO:teuthology.orchestra.run.smithi070.stdout:2920577761652 2024-09-21T10:39:23.812 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T10:39:23.829 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:23.836 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:23.838 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:24.307 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:24 smithi070 ceph-mon[107385]: pgmap v2119: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:24.307 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:24 smithi070 ceph-mon[112139]: pgmap v2119: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:24 smithi160 ceph-mon[101879]: pgmap v2119: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:26.272 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:26.454 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:26 smithi070 ceph-mon[107385]: pgmap v2120: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:26.455 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:26 smithi070 ceph-mon[112139]: pgmap v2120: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:26.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:26 smithi160 ceph-mon[101879]: pgmap v2120: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:28.224 INFO:teuthology.orchestra.run.smithi070.stdout:163208757819 2024-09-21T10:39:28.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:28 smithi070 ceph-mon[107385]: pgmap v2121: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:28.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:28 smithi070 ceph-mon[112139]: pgmap v2121: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:28.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:28 smithi160 ceph-mon[101879]: pgmap v2121: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:29.372 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:29 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1530316379' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T10:39:29.372 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:29 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1530316379' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T10:39:29.411 INFO:teuthology.orchestra.run.smithi070.stdout:128849019455 2024-09-21T10:39:29.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:29 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1530316379' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T10:39:30.100 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849019453 got 128849019455 for osd.4 2024-09-21T10:39:30.100 DEBUG:teuthology.parallel:result is None 2024-09-21T10:39:30.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:30 smithi070 ceph-mon[107385]: pgmap v2122: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:30.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:30 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2207672917' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T10:39:30.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:30 smithi070 ceph-mon[112139]: pgmap v2122: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:30.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:30 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2207672917' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T10:39:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:30 smithi160 ceph-mon[101879]: pgmap v2122: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:30 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2207672917' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T10:39:31.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:39:31.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:39:31.413 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757816 got 163208757819 for osd.5 2024-09-21T10:39:31.413 DEBUG:teuthology.parallel:result is None 2024-09-21T10:39:31.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:39:32.000 INFO:teuthology.orchestra.run.smithi070.stdout:2860448219524 2024-09-21T10:39:32.108 INFO:teuthology.orchestra.run.smithi070.stdout:231928234545 2024-09-21T10:39:32.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:32 smithi070 ceph-mon[107385]: pgmap v2123: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:32.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:32 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3666265118' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T10:39:32.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:32 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/240890084' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T10:39:32.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:32 smithi070 ceph-mon[112139]: pgmap v2123: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:32.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:32 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3666265118' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T10:39:32.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:32 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/240890084' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T10:39:32.475 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:32 smithi160 ceph-mon[101879]: pgmap v2123: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:32.475 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:32 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3666265118' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T10:39:32.475 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:32 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/240890084' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T10:39:32.857 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:33.265 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:39:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:39:33] "GET /metrics HTTP/1.1" 200 34048 "" "Prometheus/2.51.0" 2024-09-21T10:39:33.286 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2860448219522 got 2860448219524 for osd.0 2024-09-21T10:39:33.287 DEBUG:teuthology.parallel:result is None 2024-09-21T10:39:33.690 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234542 got 231928234545 for osd.7 2024-09-21T10:39:33.690 DEBUG:teuthology.parallel:result is None 2024-09-21T10:39:34.222 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:34 smithi070 ceph-mon[107385]: pgmap v2124: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:34.222 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:34 smithi070 ceph-mon[112139]: pgmap v2124: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:34.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:34 smithi160 ceph-mon[101879]: pgmap v2124: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:34.748 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:35.975 INFO:teuthology.orchestra.run.smithi070.stdout:197568496183 2024-09-21T10:39:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:36 smithi160 ceph-mon[101879]: pgmap v2125: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:39:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:39:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:39:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:36 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2595974409' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T10:39:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:36 smithi070 ceph-mon[107385]: pgmap v2125: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:39:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:39:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:39:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:36 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2595974409' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T10:39:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:36 smithi070 ceph-mon[112139]: pgmap v2125: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:39:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:39:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:39:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:36 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2595974409' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T10:39:36.676 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568496179 got 197568496183 for osd.6 2024-09-21T10:39:36.676 DEBUG:teuthology.parallel:result is None 2024-09-21T10:39:38.223 INFO:teuthology.orchestra.run.smithi070.stdout:2972117369200 2024-09-21T10:39:38.490 INFO:teuthology.orchestra.run.smithi070.stdout:2894807957886 2024-09-21T10:39:38.491 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:38 smithi070 ceph-mon[107385]: pgmap v2126: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:38.491 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:38 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2561746064' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T10:39:38.491 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:38 smithi070 ceph-mon[112139]: pgmap v2126: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:38.491 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:38 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2561746064' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T10:39:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:38 smithi160 ceph-mon[101879]: pgmap v2126: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:38 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2561746064' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T10:39:39.108 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2894807957883 got 2894807957886 for osd.1 2024-09-21T10:39:39.109 DEBUG:teuthology.parallel:result is None 2024-09-21T10:39:39.375 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:39 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3640381368' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T10:39:39.375 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:39 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3640381368' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T10:39:39.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:39 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3640381368' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T10:39:40.060 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2972117369196 got 2972117369200 for osd.3 2024-09-21T10:39:40.060 DEBUG:teuthology.parallel:result is None 2024-09-21T10:39:40.589 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:40 smithi070 ceph-mon[107385]: pgmap v2127: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:40.589 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:40 smithi070 ceph-mon[112139]: pgmap v2127: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:40.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:40 smithi160 ceph-mon[101879]: pgmap v2127: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:41.126 INFO:teuthology.orchestra.run.smithi070.stdout:2920577761656 2024-09-21T10:39:41.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:41 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1008758343' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T10:39:41.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:41 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1008758343' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T10:39:41.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:41 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1008758343' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T10:39:41.740 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2920577761652 got 2920577761656 for osd.2 2024-09-21T10:39:41.741 DEBUG:teuthology.parallel:result is None 2024-09-21T10:39:41.741 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T10:39:41.741 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:39:42.495 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:42 smithi160 ceph-mon[101879]: pgmap v2128: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:42.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:42 smithi070 ceph-mon[112139]: pgmap v2128: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:42.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:42 smithi070 ceph-mon[107385]: pgmap v2128: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:43.486 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:39:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:39:43] "GET /metrics HTTP/1.1" 200 34046 "" "Prometheus/2.51.0" 2024-09-21T10:39:44.478 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:44 smithi070 ceph-mon[107385]: pgmap v2129: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:44.478 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:44 smithi070 ceph-mon[112139]: pgmap v2129: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:44.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:44 smithi160 ceph-mon[101879]: pgmap v2129: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:45.144 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:39:45.145 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T10:39:46.002 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":2130,"stamp":"2024-09-21T10:39:44.647115+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595941,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":279,"num_write_kb":2449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":835156,"kb_used_data":116728,"kb_used_omap":296,"kb_used_meta":718039,"kb_avail":749126060,"statfs":{"total":767960285184,"available":767105085440,"internally_reserved":0,"allocated":119529472,"data_stored":75688591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":303807,"internal_metadata":735272257},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002810"},"pg_stats":[{"pgid":"3.1f","version":"71'4","reported_seq":1904,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.116537+0000","last_change":"2024-09-21T10:08:30.578838+0000","last_active":"2024-09-21T10:38:57.116537+0000","last_peered":"2024-09-21T10:38:57.116537+0000","last_clean":"2024-09-21T10:38:57.116537+0000","last_became_active":"2024-09-21T10:08:30.577773+0000","last_became_peered":"2024-09-21T10:08:30.577773+0000","last_unstale":"2024-09-21T10:38:57.116537+0000","last_undegraded":"2024-09-21T10:38:57.116537+0000","last_fullsized":"2024-09-21T10:38:57.116537+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:46:51.619094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1270,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.701486+0000","last_change":"2024-09-21T10:07:55.999392+0000","last_active":"2024-09-21T10:38:53.701486+0000","last_peered":"2024-09-21T10:38:53.701486+0000","last_clean":"2024-09-21T10:38:53.701486+0000","last_became_active":"2024-09-21T10:07:55.999103+0000","last_became_peered":"2024-09-21T10:07:55.999103+0000","last_unstale":"2024-09-21T10:38:53.701486+0000","last_undegraded":"2024-09-21T10:38:53.701486+0000","last_fullsized":"2024-09-21T10:38:53.701486+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:48.757242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":1891,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.116274+0000","last_change":"2024-09-21T10:09:06.592799+0000","last_active":"2024-09-21T10:38:57.116274+0000","last_peered":"2024-09-21T10:38:57.116274+0000","last_clean":"2024-09-21T10:38:57.116274+0000","last_became_active":"2024-09-21T10:09:06.592468+0000","last_became_peered":"2024-09-21T10:09:06.592468+0000","last_unstale":"2024-09-21T10:38:57.116274+0000","last_undegraded":"2024-09-21T10:38:57.116274+0000","last_fullsized":"2024-09-21T10:38:57.116274+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:45:04.534377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1908,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.119459+0000","last_change":"2024-09-21T10:09:06.584044+0000","last_active":"2024-09-21T10:38:57.119459+0000","last_peered":"2024-09-21T10:38:57.119459+0000","last_clean":"2024-09-21T10:38:57.119459+0000","last_became_active":"2024-09-21T10:09:06.583761+0000","last_became_peered":"2024-09-21T10:09:06.583761+0000","last_unstale":"2024-09-21T10:38:57.119459+0000","last_undegraded":"2024-09-21T10:38:57.119459+0000","last_fullsized":"2024-09-21T10:38:57.119459+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:53.839091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":1257,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:54.019511+0000","last_change":"2024-09-21T09:54:53.634508+0000","last_active":"2024-09-21T10:38:54.019511+0000","last_peered":"2024-09-21T10:38:54.019511+0000","last_clean":"2024-09-21T10:38:54.019511+0000","last_became_active":"2024-09-21T09:54:53.634254+0000","last_became_peered":"2024-09-21T09:54:53.634254+0000","last_unstale":"2024-09-21T10:38:54.019511+0000","last_undegraded":"2024-09-21T10:38:54.019511+0000","last_fullsized":"2024-09-21T10:38:54.019511+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:18.122107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1284,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.701607+0000","last_change":"2024-09-21T10:09:06.574933+0000","last_active":"2024-09-21T10:38:53.701607+0000","last_peered":"2024-09-21T10:38:53.701607+0000","last_clean":"2024-09-21T10:38:53.701607+0000","last_became_active":"2024-09-21T10:09:06.574467+0000","last_became_peered":"2024-09-21T10:09:06.574467+0000","last_unstale":"2024-09-21T10:38:53.701607+0000","last_undegraded":"2024-09-21T10:38:53.701607+0000","last_fullsized":"2024-09-21T10:38:53.701607+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:06.142269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":1272,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:54.020704+0000","last_change":"2024-09-21T10:07:55.991189+0000","last_active":"2024-09-21T10:38:54.020704+0000","last_peered":"2024-09-21T10:38:54.020704+0000","last_clean":"2024-09-21T10:38:54.020704+0000","last_became_active":"2024-09-21T10:07:55.990784+0000","last_became_peered":"2024-09-21T10:07:55.990784+0000","last_unstale":"2024-09-21T10:38:54.020704+0000","last_undegraded":"2024-09-21T10:38:54.020704+0000","last_fullsized":"2024-09-21T10:38:54.020704+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:00.580471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1284,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.702162+0000","last_change":"2024-09-21T10:09:06.560175+0000","last_active":"2024-09-21T10:38:53.702162+0000","last_peered":"2024-09-21T10:38:53.702162+0000","last_clean":"2024-09-21T10:38:53.702162+0000","last_became_active":"2024-09-21T10:09:06.558584+0000","last_became_peered":"2024-09-21T10:09:06.558584+0000","last_unstale":"2024-09-21T10:38:53.702162+0000","last_undegraded":"2024-09-21T10:38:53.702162+0000","last_fullsized":"2024-09-21T10:38:53.702162+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:02.918339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":1886,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114958+0000","last_change":"2024-09-21T10:07:21.532238+0000","last_active":"2024-09-21T10:38:57.114958+0000","last_peered":"2024-09-21T10:38:57.114958+0000","last_clean":"2024-09-21T10:38:57.114958+0000","last_became_active":"2024-09-21T10:07:21.522339+0000","last_became_peered":"2024-09-21T10:07:21.522339+0000","last_unstale":"2024-09-21T10:38:57.114958+0000","last_undegraded":"2024-09-21T10:38:57.114958+0000","last_fullsized":"2024-09-21T10:38:57.114958+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:34:09.116737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1908,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114132+0000","last_change":"2024-09-21T10:09:06.542669+0000","last_active":"2024-09-21T10:38:57.114132+0000","last_peered":"2024-09-21T10:38:57.114132+0000","last_clean":"2024-09-21T10:38:57.114132+0000","last_became_active":"2024-09-21T10:09:06.542384+0000","last_became_peered":"2024-09-21T10:09:06.542384+0000","last_unstale":"2024-09-21T10:38:57.114132+0000","last_undegraded":"2024-09-21T10:38:57.114132+0000","last_fullsized":"2024-09-21T10:38:57.114132+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:53.692798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":1286,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.701746+0000","last_change":"2024-09-21T10:08:30.598724+0000","last_active":"2024-09-21T10:38:53.701746+0000","last_peered":"2024-09-21T10:38:53.701746+0000","last_clean":"2024-09-21T10:38:53.701746+0000","last_became_active":"2024-09-21T10:08:30.598229+0000","last_became_peered":"2024-09-21T10:08:30.598229+0000","last_unstale":"2024-09-21T10:38:53.701746+0000","last_undegraded":"2024-09-21T10:38:53.701746+0000","last_fullsized":"2024-09-21T10:38:53.701746+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:57:57.056712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1902,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.118795+0000","last_change":"2024-09-21T10:09:06.590661+0000","last_active":"2024-09-21T10:38:57.118795+0000","last_peered":"2024-09-21T10:38:57.118795+0000","last_clean":"2024-09-21T10:38:57.118795+0000","last_became_active":"2024-09-21T10:09:06.588941+0000","last_became_peered":"2024-09-21T10:09:06.588941+0000","last_unstale":"2024-09-21T10:38:57.118795+0000","last_undegraded":"2024-09-21T10:38:57.118795+0000","last_fullsized":"2024-09-21T10:38:57.118795+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:18:09.719872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1881,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.118623+0000","last_change":"2024-09-21T10:09:06.530394+0000","last_active":"2024-09-21T10:38:57.118623+0000","last_peered":"2024-09-21T10:38:57.118623+0000","last_clean":"2024-09-21T10:38:57.118623+0000","last_became_active":"2024-09-21T10:09:06.529307+0000","last_became_peered":"2024-09-21T10:09:06.529307+0000","last_unstale":"2024-09-21T10:38:57.118623+0000","last_undegraded":"2024-09-21T10:38:57.118623+0000","last_fullsized":"2024-09-21T10:38:57.118623+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:54.287397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1899,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115865+0000","last_change":"2024-09-21T10:09:06.558433+0000","last_active":"2024-09-21T10:38:57.115865+0000","last_peered":"2024-09-21T10:38:57.115865+0000","last_clean":"2024-09-21T10:38:57.115865+0000","last_became_active":"2024-09-21T10:09:06.557907+0000","last_became_peered":"2024-09-21T10:09:06.557907+0000","last_unstale":"2024-09-21T10:38:57.115865+0000","last_undegraded":"2024-09-21T10:38:57.115865+0000","last_fullsized":"2024-09-21T10:38:57.115865+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:05.109492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1895,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114707+0000","last_change":"2024-09-21T10:09:06.552324+0000","last_active":"2024-09-21T10:38:57.114707+0000","last_peered":"2024-09-21T10:38:57.114707+0000","last_clean":"2024-09-21T10:38:57.114707+0000","last_became_active":"2024-09-21T10:09:06.540545+0000","last_became_peered":"2024-09-21T10:09:06.540545+0000","last_unstale":"2024-09-21T10:38:57.114707+0000","last_undegraded":"2024-09-21T10:38:57.114707+0000","last_fullsized":"2024-09-21T10:38:57.114707+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:08.506149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1274,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.322114+0000","last_change":"2024-09-21T10:07:56.003005+0000","last_active":"2024-09-21T10:38:53.322114+0000","last_peered":"2024-09-21T10:38:53.322114+0000","last_clean":"2024-09-21T10:38:53.322114+0000","last_became_active":"2024-09-21T10:07:56.001550+0000","last_became_peered":"2024-09-21T10:07:56.001550+0000","last_unstale":"2024-09-21T10:38:53.322114+0000","last_undegraded":"2024-09-21T10:38:53.322114+0000","last_fullsized":"2024-09-21T10:38:53.322114+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:54.553261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1888,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114926+0000","last_change":"2024-09-21T10:09:06.559017+0000","last_active":"2024-09-21T10:38:57.114926+0000","last_peered":"2024-09-21T10:38:57.114926+0000","last_clean":"2024-09-21T10:38:57.114926+0000","last_became_active":"2024-09-21T10:09:06.555256+0000","last_became_peered":"2024-09-21T10:09:06.555256+0000","last_unstale":"2024-09-21T10:38:57.114926+0000","last_undegraded":"2024-09-21T10:38:57.114926+0000","last_fullsized":"2024-09-21T10:38:57.114926+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:33:45.145138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1893,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.116730+0000","last_change":"2024-09-21T10:09:06.562504+0000","last_active":"2024-09-21T10:38:57.116730+0000","last_peered":"2024-09-21T10:38:57.116730+0000","last_clean":"2024-09-21T10:38:57.116730+0000","last_became_active":"2024-09-21T10:09:06.561747+0000","last_became_peered":"2024-09-21T10:09:06.561747+0000","last_unstale":"2024-09-21T10:38:57.116730+0000","last_undegraded":"2024-09-21T10:38:57.116730+0000","last_fullsized":"2024-09-21T10:38:57.116730+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:03:58.012406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1879,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.120517+0000","last_change":"2024-09-21T10:09:06.566050+0000","last_active":"2024-09-21T10:38:57.120517+0000","last_peered":"2024-09-21T10:38:57.120517+0000","last_clean":"2024-09-21T10:38:57.120517+0000","last_became_active":"2024-09-21T10:09:06.565791+0000","last_became_peered":"2024-09-21T10:09:06.565791+0000","last_unstale":"2024-09-21T10:38:57.120517+0000","last_undegraded":"2024-09-21T10:38:57.120517+0000","last_fullsized":"2024-09-21T10:38:57.120517+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:59:20.542888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1271,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:54.020837+0000","last_change":"2024-09-21T10:07:56.023722+0000","last_active":"2024-09-21T10:38:54.020837+0000","last_peered":"2024-09-21T10:38:54.020837+0000","last_clean":"2024-09-21T10:38:54.020837+0000","last_became_active":"2024-09-21T10:07:56.023085+0000","last_became_peered":"2024-09-21T10:07:56.023085+0000","last_unstale":"2024-09-21T10:38:54.020837+0000","last_undegraded":"2024-09-21T10:38:54.020837+0000","last_fullsized":"2024-09-21T10:38:54.020837+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:34:07.168469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":1276,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.081521+0000","last_change":"2024-09-21T10:09:06.572971+0000","last_active":"2024-09-21T10:38:53.081521+0000","last_peered":"2024-09-21T10:38:53.081521+0000","last_clean":"2024-09-21T10:38:53.081521+0000","last_became_active":"2024-09-21T10:09:06.571680+0000","last_became_peered":"2024-09-21T10:09:06.571680+0000","last_unstale":"2024-09-21T10:38:53.081521+0000","last_undegraded":"2024-09-21T10:38:53.081521+0000","last_fullsized":"2024-09-21T10:38:53.081521+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:21:45.051132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1880,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.117421+0000","last_change":"2024-09-21T10:08:30.589733+0000","last_active":"2024-09-21T10:38:57.117421+0000","last_peered":"2024-09-21T10:38:57.117421+0000","last_clean":"2024-09-21T10:38:57.117421+0000","last_became_active":"2024-09-21T10:08:30.589456+0000","last_became_peered":"2024-09-21T10:08:30.589456+0000","last_unstale":"2024-09-21T10:38:57.117421+0000","last_undegraded":"2024-09-21T10:38:57.117421+0000","last_fullsized":"2024-09-21T10:38:57.117421+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:09.654350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":1280,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.092674+0000","last_change":"2024-09-21T09:54:53.624887+0000","last_active":"2024-09-21T10:38:57.092674+0000","last_peered":"2024-09-21T10:38:57.092674+0000","last_clean":"2024-09-21T10:38:57.092674+0000","last_became_active":"2024-09-21T09:54:53.624525+0000","last_became_peered":"2024-09-21T09:54:53.624525+0000","last_unstale":"2024-09-21T10:38:57.092674+0000","last_undegraded":"2024-09-21T10:38:57.092674+0000","last_fullsized":"2024-09-21T10:38:57.092674+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:05:59.605316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1271,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.094087+0000","last_change":"2024-09-21T10:07:56.001756+0000","last_active":"2024-09-21T10:38:57.094087+0000","last_peered":"2024-09-21T10:38:57.094087+0000","last_clean":"2024-09-21T10:38:57.094087+0000","last_became_active":"2024-09-21T10:07:56.000789+0000","last_became_peered":"2024-09-21T10:07:56.000789+0000","last_unstale":"2024-09-21T10:38:57.094087+0000","last_undegraded":"2024-09-21T10:38:57.094087+0000","last_fullsized":"2024-09-21T10:38:57.094087+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:56:04.302902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1258,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.701544+0000","last_change":"2024-09-21T09:54:53.636902+0000","last_active":"2024-09-21T10:38:53.701544+0000","last_peered":"2024-09-21T10:38:53.701544+0000","last_clean":"2024-09-21T10:38:53.701544+0000","last_became_active":"2024-09-21T09:54:53.636616+0000","last_became_peered":"2024-09-21T09:54:53.636616+0000","last_unstale":"2024-09-21T10:38:53.701544+0000","last_undegraded":"2024-09-21T10:38:53.701544+0000","last_fullsized":"2024-09-21T10:38:53.701544+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:46:47.694495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1287,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.116875+0000","last_change":"2024-09-21T10:08:30.587337+0000","last_active":"2024-09-21T10:38:57.116875+0000","last_peered":"2024-09-21T10:38:57.116875+0000","last_clean":"2024-09-21T10:38:57.116875+0000","last_became_active":"2024-09-21T10:08:30.585404+0000","last_became_peered":"2024-09-21T10:08:30.585404+0000","last_unstale":"2024-09-21T10:38:57.116875+0000","last_undegraded":"2024-09-21T10:38:57.116875+0000","last_fullsized":"2024-09-21T10:38:57.116875+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:24:41.559947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":1900,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115709+0000","last_change":"2024-09-21T10:07:56.023462+0000","last_active":"2024-09-21T10:38:57.115709+0000","last_peered":"2024-09-21T10:38:57.115709+0000","last_clean":"2024-09-21T10:38:57.115709+0000","last_became_active":"2024-09-21T10:07:56.021859+0000","last_became_peered":"2024-09-21T10:07:56.021859+0000","last_unstale":"2024-09-21T10:38:57.115709+0000","last_undegraded":"2024-09-21T10:38:57.115709+0000","last_fullsized":"2024-09-21T10:38:57.115709+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:42:03.134729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1875,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114840+0000","last_change":"2024-09-21T10:09:06.580853+0000","last_active":"2024-09-21T10:38:57.114840+0000","last_peered":"2024-09-21T10:38:57.114840+0000","last_clean":"2024-09-21T10:38:57.114840+0000","last_became_active":"2024-09-21T10:09:06.568439+0000","last_became_peered":"2024-09-21T10:09:06.568439+0000","last_unstale":"2024-09-21T10:38:57.114840+0000","last_undegraded":"2024-09-21T10:38:57.114840+0000","last_fullsized":"2024-09-21T10:38:57.114840+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:17.633827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1881,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115718+0000","last_change":"2024-09-21T10:07:56.006236+0000","last_active":"2024-09-21T10:38:57.115718+0000","last_peered":"2024-09-21T10:38:57.115718+0000","last_clean":"2024-09-21T10:38:57.115718+0000","last_became_active":"2024-09-21T10:07:56.004439+0000","last_became_peered":"2024-09-21T10:07:56.004439+0000","last_unstale":"2024-09-21T10:38:57.115718+0000","last_undegraded":"2024-09-21T10:38:57.115718+0000","last_fullsized":"2024-09-21T10:38:57.115718+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:39.422467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":1876,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.121117+0000","last_change":"2024-09-21T10:09:06.578081+0000","last_active":"2024-09-21T10:38:57.121117+0000","last_peered":"2024-09-21T10:38:57.121117+0000","last_clean":"2024-09-21T10:38:57.121117+0000","last_became_active":"2024-09-21T10:09:06.575129+0000","last_became_peered":"2024-09-21T10:09:06.575129+0000","last_unstale":"2024-09-21T10:38:57.121117+0000","last_undegraded":"2024-09-21T10:38:57.121117+0000","last_fullsized":"2024-09-21T10:38:57.121117+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:47:26.651862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1887,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115125+0000","last_change":"2024-09-21T10:07:21.531644+0000","last_active":"2024-09-21T10:38:57.115125+0000","last_peered":"2024-09-21T10:38:57.115125+0000","last_clean":"2024-09-21T10:38:57.115125+0000","last_became_active":"2024-09-21T10:07:21.520764+0000","last_became_peered":"2024-09-21T10:07:21.520764+0000","last_unstale":"2024-09-21T10:38:57.115125+0000","last_undegraded":"2024-09-21T10:38:57.115125+0000","last_fullsized":"2024-09-21T10:38:57.115125+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:11:32.147647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1288,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.081786+0000","last_change":"2024-09-21T10:07:21.499183+0000","last_active":"2024-09-21T10:38:53.081786+0000","last_peered":"2024-09-21T10:38:53.081786+0000","last_clean":"2024-09-21T10:38:53.081786+0000","last_became_active":"2024-09-21T10:07:21.498610+0000","last_became_peered":"2024-09-21T10:07:21.498610+0000","last_unstale":"2024-09-21T10:38:53.081786+0000","last_undegraded":"2024-09-21T10:38:53.081786+0000","last_fullsized":"2024-09-21T10:38:53.081786+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:20:17.212088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1910,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.118834+0000","last_change":"2024-09-21T10:08:30.588282+0000","last_active":"2024-09-21T10:38:57.118834+0000","last_peered":"2024-09-21T10:38:57.118834+0000","last_clean":"2024-09-21T10:38:57.118834+0000","last_became_active":"2024-09-21T10:08:30.587736+0000","last_became_peered":"2024-09-21T10:08:30.587736+0000","last_unstale":"2024-09-21T10:38:57.118834+0000","last_undegraded":"2024-09-21T10:38:57.118834+0000","last_fullsized":"2024-09-21T10:38:57.118834+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:22:18.890670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1872,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.117294+0000","last_change":"2024-09-21T10:07:56.067772+0000","last_active":"2024-09-21T10:38:57.117294+0000","last_peered":"2024-09-21T10:38:57.117294+0000","last_clean":"2024-09-21T10:38:57.117294+0000","last_became_active":"2024-09-21T10:07:56.052602+0000","last_became_peered":"2024-09-21T10:07:56.052602+0000","last_unstale":"2024-09-21T10:38:57.117294+0000","last_undegraded":"2024-09-21T10:38:57.117294+0000","last_fullsized":"2024-09-21T10:38:57.117294+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:07:59.899214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1926,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.118605+0000","last_change":"2024-09-21T10:07:56.017999+0000","last_active":"2024-09-21T10:38:57.118605+0000","last_peered":"2024-09-21T10:38:57.118605+0000","last_clean":"2024-09-21T10:38:57.118605+0000","last_became_active":"2024-09-21T10:07:56.017099+0000","last_became_peered":"2024-09-21T10:07:56.017099+0000","last_unstale":"2024-09-21T10:38:57.118605+0000","last_undegraded":"2024-09-21T10:38:57.118605+0000","last_fullsized":"2024-09-21T10:38:57.118605+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:58.288984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1289,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:55.981301+0000","last_change":"2024-09-21T10:09:06.558796+0000","last_active":"2024-09-21T10:38:55.981301+0000","last_peered":"2024-09-21T10:38:55.981301+0000","last_clean":"2024-09-21T10:38:55.981301+0000","last_became_active":"2024-09-21T10:09:06.557834+0000","last_became_peered":"2024-09-21T10:09:06.557834+0000","last_unstale":"2024-09-21T10:38:55.981301+0000","last_undegraded":"2024-09-21T10:38:55.981301+0000","last_fullsized":"2024-09-21T10:38:55.981301+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:02:46.366855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1920,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.118992+0000","last_change":"2024-09-21T10:09:06.534003+0000","last_active":"2024-09-21T10:38:57.118992+0000","last_peered":"2024-09-21T10:38:57.118992+0000","last_clean":"2024-09-21T10:38:57.118992+0000","last_became_active":"2024-09-21T10:09:06.533714+0000","last_became_peered":"2024-09-21T10:09:06.533714+0000","last_unstale":"2024-09-21T10:38:57.118992+0000","last_undegraded":"2024-09-21T10:38:57.118992+0000","last_fullsized":"2024-09-21T10:38:57.118992+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:15.622055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1274,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.702267+0000","last_change":"2024-09-21T10:07:21.524082+0000","last_active":"2024-09-21T10:38:53.702267+0000","last_peered":"2024-09-21T10:38:53.702267+0000","last_clean":"2024-09-21T10:38:53.702267+0000","last_became_active":"2024-09-21T10:07:21.523779+0000","last_became_peered":"2024-09-21T10:07:21.523779+0000","last_unstale":"2024-09-21T10:38:53.702267+0000","last_undegraded":"2024-09-21T10:38:53.702267+0000","last_fullsized":"2024-09-21T10:38:53.702267+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:13.813832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":1289,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114172+0000","last_change":"2024-09-21T10:07:56.024239+0000","last_active":"2024-09-21T10:38:57.114172+0000","last_peered":"2024-09-21T10:38:57.114172+0000","last_clean":"2024-09-21T10:38:57.114172+0000","last_became_active":"2024-09-21T10:07:56.023914+0000","last_became_peered":"2024-09-21T10:07:56.023914+0000","last_unstale":"2024-09-21T10:38:57.114172+0000","last_undegraded":"2024-09-21T10:38:57.114172+0000","last_fullsized":"2024-09-21T10:38:57.114172+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:23.361736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084477399999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1909,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115265+0000","last_change":"2024-09-21T10:07:56.056338+0000","last_active":"2024-09-21T10:38:57.115265+0000","last_peered":"2024-09-21T10:38:57.115265+0000","last_clean":"2024-09-21T10:38:57.115265+0000","last_became_active":"2024-09-21T10:07:56.052888+0000","last_became_peered":"2024-09-21T10:07:56.052888+0000","last_unstale":"2024-09-21T10:38:57.115265+0000","last_undegraded":"2024-09-21T10:38:57.115265+0000","last_fullsized":"2024-09-21T10:38:57.115265+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:34:48.662436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1883,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114052+0000","last_change":"2024-09-21T10:08:30.599270+0000","last_active":"2024-09-21T10:38:57.114052+0000","last_peered":"2024-09-21T10:38:57.114052+0000","last_clean":"2024-09-21T10:38:57.114052+0000","last_became_active":"2024-09-21T10:08:30.598873+0000","last_became_peered":"2024-09-21T10:08:30.598873+0000","last_unstale":"2024-09-21T10:38:57.114052+0000","last_undegraded":"2024-09-21T10:38:57.114052+0000","last_fullsized":"2024-09-21T10:38:57.114052+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:15:19.305854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1916,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.116026+0000","last_change":"2024-09-21T10:09:06.542888+0000","last_active":"2024-09-21T10:38:57.116026+0000","last_peered":"2024-09-21T10:38:57.116026+0000","last_clean":"2024-09-21T10:38:57.116026+0000","last_became_active":"2024-09-21T10:09:06.542593+0000","last_became_peered":"2024-09-21T10:09:06.542593+0000","last_unstale":"2024-09-21T10:38:57.116026+0000","last_undegraded":"2024-09-21T10:38:57.116026+0000","last_fullsized":"2024-09-21T10:38:57.116026+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:18:24.657282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1908,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115920+0000","last_change":"2024-09-21T10:08:30.589052+0000","last_active":"2024-09-21T10:38:57.115920+0000","last_peered":"2024-09-21T10:38:57.115920+0000","last_clean":"2024-09-21T10:38:57.115920+0000","last_became_active":"2024-09-21T10:08:30.588532+0000","last_became_peered":"2024-09-21T10:08:30.588532+0000","last_unstale":"2024-09-21T10:38:57.115920+0000","last_undegraded":"2024-09-21T10:38:57.115920+0000","last_fullsized":"2024-09-21T10:38:57.115920+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:55:48.460046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1896,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.116246+0000","last_change":"2024-09-21T10:09:06.560130+0000","last_active":"2024-09-21T10:38:57.116246+0000","last_peered":"2024-09-21T10:38:57.116246+0000","last_clean":"2024-09-21T10:38:57.116246+0000","last_became_active":"2024-09-21T10:09:06.559827+0000","last_became_peered":"2024-09-21T10:09:06.559827+0000","last_unstale":"2024-09-21T10:38:57.116246+0000","last_undegraded":"2024-09-21T10:38:57.116246+0000","last_fullsized":"2024-09-21T10:38:57.116246+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:05:17.166734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1304,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.081095+0000","last_change":"2024-09-21T10:07:55.995501+0000","last_active":"2024-09-21T10:38:53.081095+0000","last_peered":"2024-09-21T10:38:53.081095+0000","last_clean":"2024-09-21T10:38:53.081095+0000","last_became_active":"2024-09-21T10:07:55.995188+0000","last_became_peered":"2024-09-21T10:07:55.995188+0000","last_unstale":"2024-09-21T10:38:53.081095+0000","last_undegraded":"2024-09-21T10:38:53.081095+0000","last_fullsized":"2024-09-21T10:38:53.081095+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:49:30.127774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1259,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:56.085563+0000","last_change":"2024-09-21T09:54:53.722213+0000","last_active":"2024-09-21T10:38:56.085563+0000","last_peered":"2024-09-21T10:38:56.085563+0000","last_clean":"2024-09-21T10:38:56.085563+0000","last_became_active":"2024-09-21T09:54:53.721437+0000","last_became_peered":"2024-09-21T09:54:53.721437+0000","last_unstale":"2024-09-21T10:38:56.085563+0000","last_undegraded":"2024-09-21T10:38:56.085563+0000","last_fullsized":"2024-09-21T10:38:56.085563+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:01:41.456731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1877,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115726+0000","last_change":"2024-09-21T10:09:06.560589+0000","last_active":"2024-09-21T10:38:57.115726+0000","last_peered":"2024-09-21T10:38:57.115726+0000","last_clean":"2024-09-21T10:38:57.115726+0000","last_became_active":"2024-09-21T10:09:06.557552+0000","last_became_peered":"2024-09-21T10:09:06.557552+0000","last_unstale":"2024-09-21T10:38:57.115726+0000","last_undegraded":"2024-09-21T10:38:57.115726+0000","last_fullsized":"2024-09-21T10:38:57.115726+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:09:24.866759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":1288,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115185+0000","last_change":"2024-09-21T10:08:30.576357+0000","last_active":"2024-09-21T10:38:57.115185+0000","last_peered":"2024-09-21T10:38:57.115185+0000","last_clean":"2024-09-21T10:38:57.115185+0000","last_became_active":"2024-09-21T10:08:30.576033+0000","last_became_peered":"2024-09-21T10:08:30.576033+0000","last_unstale":"2024-09-21T10:38:57.115185+0000","last_undegraded":"2024-09-21T10:38:57.115185+0000","last_fullsized":"2024-09-21T10:38:57.115185+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:21:45.522197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":1606,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.091312+0000","last_change":"2024-09-21T10:07:21.499838+0000","last_active":"2024-09-21T10:38:57.091312+0000","last_peered":"2024-09-21T10:38:57.091312+0000","last_clean":"2024-09-21T10:38:57.091312+0000","last_became_active":"2024-09-21T10:07:21.499449+0000","last_became_peered":"2024-09-21T10:07:21.499449+0000","last_unstale":"2024-09-21T10:38:57.091312+0000","last_undegraded":"2024-09-21T10:38:57.091312+0000","last_fullsized":"2024-09-21T10:38:57.091312+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:59:19.260791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1882,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115215+0000","last_change":"2024-09-21T10:09:06.538940+0000","last_active":"2024-09-21T10:38:57.115215+0000","last_peered":"2024-09-21T10:38:57.115215+0000","last_clean":"2024-09-21T10:38:57.115215+0000","last_became_active":"2024-09-21T10:09:06.538668+0000","last_became_peered":"2024-09-21T10:09:06.538668+0000","last_unstale":"2024-09-21T10:38:57.115215+0000","last_undegraded":"2024-09-21T10:38:57.115215+0000","last_fullsized":"2024-09-21T10:38:57.115215+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:42:44.299488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1883,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114655+0000","last_change":"2024-09-21T10:09:06.582426+0000","last_active":"2024-09-21T10:38:57.114655+0000","last_peered":"2024-09-21T10:38:57.114655+0000","last_clean":"2024-09-21T10:38:57.114655+0000","last_became_active":"2024-09-21T10:09:06.580117+0000","last_became_peered":"2024-09-21T10:09:06.580117+0000","last_unstale":"2024-09-21T10:38:57.114655+0000","last_undegraded":"2024-09-21T10:38:57.114655+0000","last_fullsized":"2024-09-21T10:38:57.114655+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:56:45.278650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1271,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.702325+0000","last_change":"2024-09-21T10:07:55.998799+0000","last_active":"2024-09-21T10:38:53.702325+0000","last_peered":"2024-09-21T10:38:53.702325+0000","last_clean":"2024-09-21T10:38:53.702325+0000","last_became_active":"2024-09-21T10:07:55.998510+0000","last_became_peered":"2024-09-21T10:07:55.998510+0000","last_unstale":"2024-09-21T10:38:53.702325+0000","last_undegraded":"2024-09-21T10:38:53.702325+0000","last_fullsized":"2024-09-21T10:38:53.702325+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:10.474008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1273,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.701779+0000","last_change":"2024-09-21T10:08:30.602686+0000","last_active":"2024-09-21T10:38:53.701779+0000","last_peered":"2024-09-21T10:38:53.701779+0000","last_clean":"2024-09-21T10:38:53.701779+0000","last_became_active":"2024-09-21T10:08:30.602363+0000","last_became_peered":"2024-09-21T10:08:30.602363+0000","last_unstale":"2024-09-21T10:38:53.701779+0000","last_undegraded":"2024-09-21T10:38:53.701779+0000","last_fullsized":"2024-09-21T10:38:53.701779+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:10.278387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1286,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.702194+0000","last_change":"2024-09-21T10:08:30.601654+0000","last_active":"2024-09-21T10:38:53.702194+0000","last_peered":"2024-09-21T10:38:53.702194+0000","last_clean":"2024-09-21T10:38:53.702194+0000","last_became_active":"2024-09-21T10:08:30.601354+0000","last_became_peered":"2024-09-21T10:08:30.601354+0000","last_unstale":"2024-09-21T10:38:53.702194+0000","last_undegraded":"2024-09-21T10:38:53.702194+0000","last_fullsized":"2024-09-21T10:38:53.702194+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:42:25.902106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1274,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.321824+0000","last_change":"2024-09-21T10:07:56.020791+0000","last_active":"2024-09-21T10:38:53.321824+0000","last_peered":"2024-09-21T10:38:53.321824+0000","last_clean":"2024-09-21T10:38:53.321824+0000","last_became_active":"2024-09-21T10:07:56.020205+0000","last_became_peered":"2024-09-21T10:07:56.020205+0000","last_unstale":"2024-09-21T10:38:53.321824+0000","last_undegraded":"2024-09-21T10:38:53.321824+0000","last_fullsized":"2024-09-21T10:38:53.321824+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:09.813166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1271,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.701976+0000","last_change":"2024-09-21T10:09:06.533271+0000","last_active":"2024-09-21T10:38:53.701976+0000","last_peered":"2024-09-21T10:38:53.701976+0000","last_clean":"2024-09-21T10:38:53.701976+0000","last_became_active":"2024-09-21T10:09:06.532041+0000","last_became_peered":"2024-09-21T10:09:06.532041+0000","last_unstale":"2024-09-21T10:38:53.701976+0000","last_undegraded":"2024-09-21T10:38:53.701976+0000","last_fullsized":"2024-09-21T10:38:53.701976+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:04.236375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":1898,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.119850+0000","last_change":"2024-09-21T10:09:06.543422+0000","last_active":"2024-09-21T10:38:57.119850+0000","last_peered":"2024-09-21T10:38:57.119850+0000","last_clean":"2024-09-21T10:38:57.119850+0000","last_became_active":"2024-09-21T10:09:06.539647+0000","last_became_peered":"2024-09-21T10:09:06.539647+0000","last_unstale":"2024-09-21T10:38:57.119850+0000","last_undegraded":"2024-09-21T10:38:57.119850+0000","last_fullsized":"2024-09-21T10:38:57.119850+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:04.990085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1897,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114294+0000","last_change":"2024-09-21T10:07:56.018560+0000","last_active":"2024-09-21T10:38:57.114294+0000","last_peered":"2024-09-21T10:38:57.114294+0000","last_clean":"2024-09-21T10:38:57.114294+0000","last_became_active":"2024-09-21T10:07:56.018261+0000","last_became_peered":"2024-09-21T10:07:56.018261+0000","last_unstale":"2024-09-21T10:38:57.114294+0000","last_undegraded":"2024-09-21T10:38:57.114294+0000","last_fullsized":"2024-09-21T10:38:57.114294+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:04:54.395456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1272,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:54.020787+0000","last_change":"2024-09-21T10:09:06.531935+0000","last_active":"2024-09-21T10:38:54.020787+0000","last_peered":"2024-09-21T10:38:54.020787+0000","last_clean":"2024-09-21T10:38:54.020787+0000","last_became_active":"2024-09-21T10:09:06.531610+0000","last_became_peered":"2024-09-21T10:09:06.531610+0000","last_unstale":"2024-09-21T10:38:54.020787+0000","last_undegraded":"2024-09-21T10:38:54.020787+0000","last_fullsized":"2024-09-21T10:38:54.020787+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:33.874405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1282,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.321716+0000","last_change":"2024-09-21T10:08:30.586447+0000","last_active":"2024-09-21T10:38:53.321716+0000","last_peered":"2024-09-21T10:38:53.321716+0000","last_clean":"2024-09-21T10:38:53.321716+0000","last_became_active":"2024-09-21T10:08:30.585434+0000","last_became_peered":"2024-09-21T10:08:30.585434+0000","last_unstale":"2024-09-21T10:38:53.321716+0000","last_undegraded":"2024-09-21T10:38:53.321716+0000","last_fullsized":"2024-09-21T10:38:53.321716+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:51:30.174217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":1260,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.081691+0000","last_change":"2024-09-21T09:54:53.628923+0000","last_active":"2024-09-21T10:38:53.081691+0000","last_peered":"2024-09-21T10:38:53.081691+0000","last_clean":"2024-09-21T10:38:53.081691+0000","last_became_active":"2024-09-21T09:54:53.627939+0000","last_became_peered":"2024-09-21T09:54:53.627939+0000","last_unstale":"2024-09-21T10:38:53.081691+0000","last_undegraded":"2024-09-21T10:38:53.081691+0000","last_fullsized":"2024-09-21T10:38:53.081691+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:07:02.820689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1287,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.701674+0000","last_change":"2024-09-21T10:09:06.559132+0000","last_active":"2024-09-21T10:38:53.701674+0000","last_peered":"2024-09-21T10:38:53.701674+0000","last_clean":"2024-09-21T10:38:53.701674+0000","last_became_active":"2024-09-21T10:09:06.558395+0000","last_became_peered":"2024-09-21T10:09:06.558395+0000","last_unstale":"2024-09-21T10:38:53.701674+0000","last_undegraded":"2024-09-21T10:38:53.701674+0000","last_fullsized":"2024-09-21T10:38:53.701674+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:27:00.073554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":1276,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.081928+0000","last_change":"2024-09-21T10:07:56.029192+0000","last_active":"2024-09-21T10:38:53.081928+0000","last_peered":"2024-09-21T10:38:53.081928+0000","last_clean":"2024-09-21T10:38:53.081928+0000","last_became_active":"2024-09-21T10:07:56.028747+0000","last_became_peered":"2024-09-21T10:07:56.028747+0000","last_unstale":"2024-09-21T10:38:53.081928+0000","last_undegraded":"2024-09-21T10:38:53.081928+0000","last_fullsized":"2024-09-21T10:38:53.081928+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:08.918337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1257,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:54.019865+0000","last_change":"2024-09-21T09:54:54.677572+0000","last_active":"2024-09-21T10:38:54.019865+0000","last_peered":"2024-09-21T10:38:54.019865+0000","last_clean":"2024-09-21T10:38:54.019865+0000","last_became_active":"2024-09-21T09:54:54.677253+0000","last_became_peered":"2024-09-21T09:54:54.677253+0000","last_unstale":"2024-09-21T10:38:54.019865+0000","last_undegraded":"2024-09-21T10:38:54.019865+0000","last_fullsized":"2024-09-21T10:38:54.019865+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:31:40.629299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":1275,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.081671+0000","last_change":"2024-09-21T10:07:21.499297+0000","last_active":"2024-09-21T10:38:53.081671+0000","last_peered":"2024-09-21T10:38:53.081671+0000","last_clean":"2024-09-21T10:38:53.081671+0000","last_became_active":"2024-09-21T10:07:21.498961+0000","last_became_peered":"2024-09-21T10:07:21.498961+0000","last_unstale":"2024-09-21T10:38:53.081671+0000","last_undegraded":"2024-09-21T10:38:53.081671+0000","last_fullsized":"2024-09-21T10:38:53.081671+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:07.281950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1281,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114824+0000","last_change":"2024-09-21T10:07:21.497877+0000","last_active":"2024-09-21T10:38:57.114824+0000","last_peered":"2024-09-21T10:38:57.114824+0000","last_clean":"2024-09-21T10:38:57.114824+0000","last_became_active":"2024-09-21T10:07:21.496317+0000","last_became_peered":"2024-09-21T10:07:21.496317+0000","last_unstale":"2024-09-21T10:38:57.114824+0000","last_undegraded":"2024-09-21T10:38:57.114824+0000","last_fullsized":"2024-09-21T10:38:57.114824+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:39:03.848183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1879,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.121039+0000","last_change":"2024-09-21T10:09:06.575204+0000","last_active":"2024-09-21T10:38:57.121039+0000","last_peered":"2024-09-21T10:38:57.121039+0000","last_clean":"2024-09-21T10:38:57.121039+0000","last_became_active":"2024-09-21T10:09:06.571218+0000","last_became_peered":"2024-09-21T10:09:06.571218+0000","last_unstale":"2024-09-21T10:38:57.121039+0000","last_undegraded":"2024-09-21T10:38:57.121039+0000","last_fullsized":"2024-09-21T10:38:57.121039+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:27.754524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1259,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.092165+0000","last_change":"2024-09-21T09:54:53.625016+0000","last_active":"2024-09-21T10:38:57.092165+0000","last_peered":"2024-09-21T10:38:57.092165+0000","last_clean":"2024-09-21T10:38:57.092165+0000","last_became_active":"2024-09-21T09:54:53.624750+0000","last_became_peered":"2024-09-21T09:54:53.624750+0000","last_unstale":"2024-09-21T10:38:57.092165+0000","last_undegraded":"2024-09-21T10:38:57.092165+0000","last_fullsized":"2024-09-21T10:38:57.092165+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:34:26.823544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1910,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115602+0000","last_change":"2024-09-21T10:09:06.558492+0000","last_active":"2024-09-21T10:38:57.115602+0000","last_peered":"2024-09-21T10:38:57.115602+0000","last_clean":"2024-09-21T10:38:57.115602+0000","last_became_active":"2024-09-21T10:09:06.558146+0000","last_became_peered":"2024-09-21T10:09:06.558146+0000","last_unstale":"2024-09-21T10:38:57.115602+0000","last_undegraded":"2024-09-21T10:38:57.115602+0000","last_fullsized":"2024-09-21T10:38:57.115602+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:28:50.638241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1893,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.117388+0000","last_change":"2024-09-21T10:09:06.579005+0000","last_active":"2024-09-21T10:38:57.117388+0000","last_peered":"2024-09-21T10:38:57.117388+0000","last_clean":"2024-09-21T10:38:57.117388+0000","last_became_active":"2024-09-21T10:09:06.578119+0000","last_became_peered":"2024-09-21T10:09:06.578119+0000","last_unstale":"2024-09-21T10:38:57.117388+0000","last_undegraded":"2024-09-21T10:38:57.117388+0000","last_fullsized":"2024-09-21T10:38:57.117388+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:20.615838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1270,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:54.020627+0000","last_change":"2024-09-21T10:09:06.557453+0000","last_active":"2024-09-21T10:38:54.020627+0000","last_peered":"2024-09-21T10:38:54.020627+0000","last_clean":"2024-09-21T10:38:54.020627+0000","last_became_active":"2024-09-21T10:09:06.557124+0000","last_became_peered":"2024-09-21T10:09:06.557124+0000","last_unstale":"2024-09-21T10:38:54.020627+0000","last_undegraded":"2024-09-21T10:38:54.020627+0000","last_fullsized":"2024-09-21T10:38:54.020627+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:03:27.731832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":1276,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.091450+0000","last_change":"2024-09-21T10:08:30.574245+0000","last_active":"2024-09-21T10:38:57.091450+0000","last_peered":"2024-09-21T10:38:57.091450+0000","last_clean":"2024-09-21T10:38:57.091450+0000","last_became_active":"2024-09-21T10:08:30.573909+0000","last_became_peered":"2024-09-21T10:08:30.573909+0000","last_unstale":"2024-09-21T10:38:57.091450+0000","last_undegraded":"2024-09-21T10:38:57.091450+0000","last_fullsized":"2024-09-21T10:38:57.091450+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:18:03.911157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1880,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.120094+0000","last_change":"2024-09-21T10:09:06.578773+0000","last_active":"2024-09-21T10:38:57.120094+0000","last_peered":"2024-09-21T10:38:57.120094+0000","last_clean":"2024-09-21T10:38:57.120094+0000","last_became_active":"2024-09-21T10:09:06.576688+0000","last_became_peered":"2024-09-21T10:09:06.576688+0000","last_unstale":"2024-09-21T10:38:57.120094+0000","last_undegraded":"2024-09-21T10:38:57.120094+0000","last_fullsized":"2024-09-21T10:38:57.120094+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:37.699650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234548,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":140356,"kb_used_data":14968,"kb_used_omap":1,"kb_used_meta":125374,"kb_avail":93604796,"statfs":{"total":95995035648,"available":95851311104,"internally_reserved":0,"allocated":15327232,"data_stored":9832930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":128383435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496185,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":148540,"kb_used_data":14960,"kb_used_omap":1,"kb_used_meta":133566,"kb_avail":93596612,"statfs":{"total":95995035648,"available":95842930688,"internally_reserved":0,"allocated":15319040,"data_stored":9832803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":136772042},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757822,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":107824,"kb_used_data":14372,"kb_used_omap":1,"kb_used_meta":93438,"kb_avail":93637328,"statfs":{"total":95995035648,"available":95884623872,"internally_reserved":0,"allocated":14716928,"data_stored":9238261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95680966},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019458,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":117624,"kb_used_data":14384,"kb_used_omap":1,"kb_used_meta":103230,"kb_avail":93627528,"statfs":{"total":95995035648,"available":95874588672,"internally_reserved":0,"allocated":14729216,"data_stored":9242492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105707974},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369202,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78440,"kb_used_data":14364,"kb_used_omap":69,"kb_used_meta":63994,"kb_avail":93666712,"statfs":{"total":95995035648,"available":95914713088,"internally_reserved":0,"allocated":14708736,"data_stored":9237692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":65529986},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":680,"seq":2920577761657,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":68716,"kb_used_data":14368,"kb_used_omap":76,"kb_used_meta":54259,"kb_avail":93676436,"statfs":{"total":95995035648,"available":95924670464,"internally_reserved":0,"allocated":14712832,"data_stored":9238218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78585,"internal_metadata":55561479},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807957888,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86632,"kb_used_data":14364,"kb_used_omap":66,"kb_used_meta":72189,"kb_avail":93658520,"statfs":{"total":95995035648,"available":95906324480,"internally_reserved":0,"allocated":14708736,"data_stored":9237621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68589,"internal_metadata":73921555},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":666,"seq":2860448219527,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87024,"kb_used_data":14948,"kb_used_omap":76,"kb_used_meta":71987,"kb_avail":93658128,"statfs":{"total":95995035648,"available":95905923072,"internally_reserved":0,"allocated":15306752,"data_stored":9828574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78716,"internal_metadata":73714820},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:39:46.006 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:39:46.214 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:46.626 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:46 smithi070 ceph-mon[107385]: pgmap v2130: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:46.626 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:46 smithi070 ceph-mon[107385]: from='client.44939 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:39:46.626 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:46 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:39:46.626 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:46 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:39:46.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:46 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:39:46.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:46 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:39:46.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:46 smithi070 ceph-mon[112139]: pgmap v2130: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:46.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:46 smithi070 ceph-mon[112139]: from='client.44939 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:39:46.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:46 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:39:46.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:46 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:39:46.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:46 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:39:46.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:46 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:39:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:46 smithi160 ceph-mon[101879]: pgmap v2130: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:46 smithi160 ceph-mon[101879]: from='client.44939 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:39:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:46 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:39:46.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:46 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:39:46.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:46 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:39:46.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:46 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:39:48.274 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:39:48.275 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T10:39:48.615 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:48 smithi070 ceph-mon[112139]: pgmap v2131: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:48.615 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:48 smithi070 ceph-mon[107385]: pgmap v2131: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:48 smithi160 ceph-mon[101879]: pgmap v2131: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:49.334 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":2131,"stamp":"2024-09-21T10:39:46.647524+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595941,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":279,"num_write_kb":2449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":835156,"kb_used_data":116728,"kb_used_omap":296,"kb_used_meta":718039,"kb_avail":749126060,"statfs":{"total":767960285184,"available":767105085440,"internally_reserved":0,"allocated":119529472,"data_stored":75688591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":303807,"internal_metadata":735272257},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002776"},"pg_stats":[{"pgid":"3.1f","version":"71'4","reported_seq":1904,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.116537+0000","last_change":"2024-09-21T10:08:30.578838+0000","last_active":"2024-09-21T10:38:57.116537+0000","last_peered":"2024-09-21T10:38:57.116537+0000","last_clean":"2024-09-21T10:38:57.116537+0000","last_became_active":"2024-09-21T10:08:30.577773+0000","last_became_peered":"2024-09-21T10:08:30.577773+0000","last_unstale":"2024-09-21T10:38:57.116537+0000","last_undegraded":"2024-09-21T10:38:57.116537+0000","last_fullsized":"2024-09-21T10:38:57.116537+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:46:51.619094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1270,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.701486+0000","last_change":"2024-09-21T10:07:55.999392+0000","last_active":"2024-09-21T10:38:53.701486+0000","last_peered":"2024-09-21T10:38:53.701486+0000","last_clean":"2024-09-21T10:38:53.701486+0000","last_became_active":"2024-09-21T10:07:55.999103+0000","last_became_peered":"2024-09-21T10:07:55.999103+0000","last_unstale":"2024-09-21T10:38:53.701486+0000","last_undegraded":"2024-09-21T10:38:53.701486+0000","last_fullsized":"2024-09-21T10:38:53.701486+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:48.757242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":1891,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.116274+0000","last_change":"2024-09-21T10:09:06.592799+0000","last_active":"2024-09-21T10:38:57.116274+0000","last_peered":"2024-09-21T10:38:57.116274+0000","last_clean":"2024-09-21T10:38:57.116274+0000","last_became_active":"2024-09-21T10:09:06.592468+0000","last_became_peered":"2024-09-21T10:09:06.592468+0000","last_unstale":"2024-09-21T10:38:57.116274+0000","last_undegraded":"2024-09-21T10:38:57.116274+0000","last_fullsized":"2024-09-21T10:38:57.116274+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:45:04.534377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1908,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.119459+0000","last_change":"2024-09-21T10:09:06.584044+0000","last_active":"2024-09-21T10:38:57.119459+0000","last_peered":"2024-09-21T10:38:57.119459+0000","last_clean":"2024-09-21T10:38:57.119459+0000","last_became_active":"2024-09-21T10:09:06.583761+0000","last_became_peered":"2024-09-21T10:09:06.583761+0000","last_unstale":"2024-09-21T10:38:57.119459+0000","last_undegraded":"2024-09-21T10:38:57.119459+0000","last_fullsized":"2024-09-21T10:38:57.119459+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:53.839091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":1257,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:54.019511+0000","last_change":"2024-09-21T09:54:53.634508+0000","last_active":"2024-09-21T10:38:54.019511+0000","last_peered":"2024-09-21T10:38:54.019511+0000","last_clean":"2024-09-21T10:38:54.019511+0000","last_became_active":"2024-09-21T09:54:53.634254+0000","last_became_peered":"2024-09-21T09:54:53.634254+0000","last_unstale":"2024-09-21T10:38:54.019511+0000","last_undegraded":"2024-09-21T10:38:54.019511+0000","last_fullsized":"2024-09-21T10:38:54.019511+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:18.122107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1284,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.701607+0000","last_change":"2024-09-21T10:09:06.574933+0000","last_active":"2024-09-21T10:38:53.701607+0000","last_peered":"2024-09-21T10:38:53.701607+0000","last_clean":"2024-09-21T10:38:53.701607+0000","last_became_active":"2024-09-21T10:09:06.574467+0000","last_became_peered":"2024-09-21T10:09:06.574467+0000","last_unstale":"2024-09-21T10:38:53.701607+0000","last_undegraded":"2024-09-21T10:38:53.701607+0000","last_fullsized":"2024-09-21T10:38:53.701607+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:06.142269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":1272,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:54.020704+0000","last_change":"2024-09-21T10:07:55.991189+0000","last_active":"2024-09-21T10:38:54.020704+0000","last_peered":"2024-09-21T10:38:54.020704+0000","last_clean":"2024-09-21T10:38:54.020704+0000","last_became_active":"2024-09-21T10:07:55.990784+0000","last_became_peered":"2024-09-21T10:07:55.990784+0000","last_unstale":"2024-09-21T10:38:54.020704+0000","last_undegraded":"2024-09-21T10:38:54.020704+0000","last_fullsized":"2024-09-21T10:38:54.020704+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:00.580471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1284,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.702162+0000","last_change":"2024-09-21T10:09:06.560175+0000","last_active":"2024-09-21T10:38:53.702162+0000","last_peered":"2024-09-21T10:38:53.702162+0000","last_clean":"2024-09-21T10:38:53.702162+0000","last_became_active":"2024-09-21T10:09:06.558584+0000","last_became_peered":"2024-09-21T10:09:06.558584+0000","last_unstale":"2024-09-21T10:38:53.702162+0000","last_undegraded":"2024-09-21T10:38:53.702162+0000","last_fullsized":"2024-09-21T10:38:53.702162+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:02.918339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":1886,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114958+0000","last_change":"2024-09-21T10:07:21.532238+0000","last_active":"2024-09-21T10:38:57.114958+0000","last_peered":"2024-09-21T10:38:57.114958+0000","last_clean":"2024-09-21T10:38:57.114958+0000","last_became_active":"2024-09-21T10:07:21.522339+0000","last_became_peered":"2024-09-21T10:07:21.522339+0000","last_unstale":"2024-09-21T10:38:57.114958+0000","last_undegraded":"2024-09-21T10:38:57.114958+0000","last_fullsized":"2024-09-21T10:38:57.114958+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:34:09.116737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1908,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114132+0000","last_change":"2024-09-21T10:09:06.542669+0000","last_active":"2024-09-21T10:38:57.114132+0000","last_peered":"2024-09-21T10:38:57.114132+0000","last_clean":"2024-09-21T10:38:57.114132+0000","last_became_active":"2024-09-21T10:09:06.542384+0000","last_became_peered":"2024-09-21T10:09:06.542384+0000","last_unstale":"2024-09-21T10:38:57.114132+0000","last_undegraded":"2024-09-21T10:38:57.114132+0000","last_fullsized":"2024-09-21T10:38:57.114132+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:53.692798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":1286,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.701746+0000","last_change":"2024-09-21T10:08:30.598724+0000","last_active":"2024-09-21T10:38:53.701746+0000","last_peered":"2024-09-21T10:38:53.701746+0000","last_clean":"2024-09-21T10:38:53.701746+0000","last_became_active":"2024-09-21T10:08:30.598229+0000","last_became_peered":"2024-09-21T10:08:30.598229+0000","last_unstale":"2024-09-21T10:38:53.701746+0000","last_undegraded":"2024-09-21T10:38:53.701746+0000","last_fullsized":"2024-09-21T10:38:53.701746+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:57:57.056712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1902,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.118795+0000","last_change":"2024-09-21T10:09:06.590661+0000","last_active":"2024-09-21T10:38:57.118795+0000","last_peered":"2024-09-21T10:38:57.118795+0000","last_clean":"2024-09-21T10:38:57.118795+0000","last_became_active":"2024-09-21T10:09:06.588941+0000","last_became_peered":"2024-09-21T10:09:06.588941+0000","last_unstale":"2024-09-21T10:38:57.118795+0000","last_undegraded":"2024-09-21T10:38:57.118795+0000","last_fullsized":"2024-09-21T10:38:57.118795+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:18:09.719872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1881,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.118623+0000","last_change":"2024-09-21T10:09:06.530394+0000","last_active":"2024-09-21T10:38:57.118623+0000","last_peered":"2024-09-21T10:38:57.118623+0000","last_clean":"2024-09-21T10:38:57.118623+0000","last_became_active":"2024-09-21T10:09:06.529307+0000","last_became_peered":"2024-09-21T10:09:06.529307+0000","last_unstale":"2024-09-21T10:38:57.118623+0000","last_undegraded":"2024-09-21T10:38:57.118623+0000","last_fullsized":"2024-09-21T10:38:57.118623+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:54.287397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1899,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115865+0000","last_change":"2024-09-21T10:09:06.558433+0000","last_active":"2024-09-21T10:38:57.115865+0000","last_peered":"2024-09-21T10:38:57.115865+0000","last_clean":"2024-09-21T10:38:57.115865+0000","last_became_active":"2024-09-21T10:09:06.557907+0000","last_became_peered":"2024-09-21T10:09:06.557907+0000","last_unstale":"2024-09-21T10:38:57.115865+0000","last_undegraded":"2024-09-21T10:38:57.115865+0000","last_fullsized":"2024-09-21T10:38:57.115865+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:05.109492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1895,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114707+0000","last_change":"2024-09-21T10:09:06.552324+0000","last_active":"2024-09-21T10:38:57.114707+0000","last_peered":"2024-09-21T10:38:57.114707+0000","last_clean":"2024-09-21T10:38:57.114707+0000","last_became_active":"2024-09-21T10:09:06.540545+0000","last_became_peered":"2024-09-21T10:09:06.540545+0000","last_unstale":"2024-09-21T10:38:57.114707+0000","last_undegraded":"2024-09-21T10:38:57.114707+0000","last_fullsized":"2024-09-21T10:38:57.114707+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:08.506149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1274,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.322114+0000","last_change":"2024-09-21T10:07:56.003005+0000","last_active":"2024-09-21T10:38:53.322114+0000","last_peered":"2024-09-21T10:38:53.322114+0000","last_clean":"2024-09-21T10:38:53.322114+0000","last_became_active":"2024-09-21T10:07:56.001550+0000","last_became_peered":"2024-09-21T10:07:56.001550+0000","last_unstale":"2024-09-21T10:38:53.322114+0000","last_undegraded":"2024-09-21T10:38:53.322114+0000","last_fullsized":"2024-09-21T10:38:53.322114+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:54.553261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1888,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114926+0000","last_change":"2024-09-21T10:09:06.559017+0000","last_active":"2024-09-21T10:38:57.114926+0000","last_peered":"2024-09-21T10:38:57.114926+0000","last_clean":"2024-09-21T10:38:57.114926+0000","last_became_active":"2024-09-21T10:09:06.555256+0000","last_became_peered":"2024-09-21T10:09:06.555256+0000","last_unstale":"2024-09-21T10:38:57.114926+0000","last_undegraded":"2024-09-21T10:38:57.114926+0000","last_fullsized":"2024-09-21T10:38:57.114926+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:33:45.145138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1893,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.116730+0000","last_change":"2024-09-21T10:09:06.562504+0000","last_active":"2024-09-21T10:38:57.116730+0000","last_peered":"2024-09-21T10:38:57.116730+0000","last_clean":"2024-09-21T10:38:57.116730+0000","last_became_active":"2024-09-21T10:09:06.561747+0000","last_became_peered":"2024-09-21T10:09:06.561747+0000","last_unstale":"2024-09-21T10:38:57.116730+0000","last_undegraded":"2024-09-21T10:38:57.116730+0000","last_fullsized":"2024-09-21T10:38:57.116730+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:03:58.012406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1879,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.120517+0000","last_change":"2024-09-21T10:09:06.566050+0000","last_active":"2024-09-21T10:38:57.120517+0000","last_peered":"2024-09-21T10:38:57.120517+0000","last_clean":"2024-09-21T10:38:57.120517+0000","last_became_active":"2024-09-21T10:09:06.565791+0000","last_became_peered":"2024-09-21T10:09:06.565791+0000","last_unstale":"2024-09-21T10:38:57.120517+0000","last_undegraded":"2024-09-21T10:38:57.120517+0000","last_fullsized":"2024-09-21T10:38:57.120517+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:59:20.542888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1271,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:54.020837+0000","last_change":"2024-09-21T10:07:56.023722+0000","last_active":"2024-09-21T10:38:54.020837+0000","last_peered":"2024-09-21T10:38:54.020837+0000","last_clean":"2024-09-21T10:38:54.020837+0000","last_became_active":"2024-09-21T10:07:56.023085+0000","last_became_peered":"2024-09-21T10:07:56.023085+0000","last_unstale":"2024-09-21T10:38:54.020837+0000","last_undegraded":"2024-09-21T10:38:54.020837+0000","last_fullsized":"2024-09-21T10:38:54.020837+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:34:07.168469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":1276,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.081521+0000","last_change":"2024-09-21T10:09:06.572971+0000","last_active":"2024-09-21T10:38:53.081521+0000","last_peered":"2024-09-21T10:38:53.081521+0000","last_clean":"2024-09-21T10:38:53.081521+0000","last_became_active":"2024-09-21T10:09:06.571680+0000","last_became_peered":"2024-09-21T10:09:06.571680+0000","last_unstale":"2024-09-21T10:38:53.081521+0000","last_undegraded":"2024-09-21T10:38:53.081521+0000","last_fullsized":"2024-09-21T10:38:53.081521+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:21:45.051132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1880,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.117421+0000","last_change":"2024-09-21T10:08:30.589733+0000","last_active":"2024-09-21T10:38:57.117421+0000","last_peered":"2024-09-21T10:38:57.117421+0000","last_clean":"2024-09-21T10:38:57.117421+0000","last_became_active":"2024-09-21T10:08:30.589456+0000","last_became_peered":"2024-09-21T10:08:30.589456+0000","last_unstale":"2024-09-21T10:38:57.117421+0000","last_undegraded":"2024-09-21T10:38:57.117421+0000","last_fullsized":"2024-09-21T10:38:57.117421+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:09.654350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":1280,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.092674+0000","last_change":"2024-09-21T09:54:53.624887+0000","last_active":"2024-09-21T10:38:57.092674+0000","last_peered":"2024-09-21T10:38:57.092674+0000","last_clean":"2024-09-21T10:38:57.092674+0000","last_became_active":"2024-09-21T09:54:53.624525+0000","last_became_peered":"2024-09-21T09:54:53.624525+0000","last_unstale":"2024-09-21T10:38:57.092674+0000","last_undegraded":"2024-09-21T10:38:57.092674+0000","last_fullsized":"2024-09-21T10:38:57.092674+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:05:59.605316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1271,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.094087+0000","last_change":"2024-09-21T10:07:56.001756+0000","last_active":"2024-09-21T10:38:57.094087+0000","last_peered":"2024-09-21T10:38:57.094087+0000","last_clean":"2024-09-21T10:38:57.094087+0000","last_became_active":"2024-09-21T10:07:56.000789+0000","last_became_peered":"2024-09-21T10:07:56.000789+0000","last_unstale":"2024-09-21T10:38:57.094087+0000","last_undegraded":"2024-09-21T10:38:57.094087+0000","last_fullsized":"2024-09-21T10:38:57.094087+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:56:04.302902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1258,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.701544+0000","last_change":"2024-09-21T09:54:53.636902+0000","last_active":"2024-09-21T10:38:53.701544+0000","last_peered":"2024-09-21T10:38:53.701544+0000","last_clean":"2024-09-21T10:38:53.701544+0000","last_became_active":"2024-09-21T09:54:53.636616+0000","last_became_peered":"2024-09-21T09:54:53.636616+0000","last_unstale":"2024-09-21T10:38:53.701544+0000","last_undegraded":"2024-09-21T10:38:53.701544+0000","last_fullsized":"2024-09-21T10:38:53.701544+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:46:47.694495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1287,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.116875+0000","last_change":"2024-09-21T10:08:30.587337+0000","last_active":"2024-09-21T10:38:57.116875+0000","last_peered":"2024-09-21T10:38:57.116875+0000","last_clean":"2024-09-21T10:38:57.116875+0000","last_became_active":"2024-09-21T10:08:30.585404+0000","last_became_peered":"2024-09-21T10:08:30.585404+0000","last_unstale":"2024-09-21T10:38:57.116875+0000","last_undegraded":"2024-09-21T10:38:57.116875+0000","last_fullsized":"2024-09-21T10:38:57.116875+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:24:41.559947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":1900,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115709+0000","last_change":"2024-09-21T10:07:56.023462+0000","last_active":"2024-09-21T10:38:57.115709+0000","last_peered":"2024-09-21T10:38:57.115709+0000","last_clean":"2024-09-21T10:38:57.115709+0000","last_became_active":"2024-09-21T10:07:56.021859+0000","last_became_peered":"2024-09-21T10:07:56.021859+0000","last_unstale":"2024-09-21T10:38:57.115709+0000","last_undegraded":"2024-09-21T10:38:57.115709+0000","last_fullsized":"2024-09-21T10:38:57.115709+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:42:03.134729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1875,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114840+0000","last_change":"2024-09-21T10:09:06.580853+0000","last_active":"2024-09-21T10:38:57.114840+0000","last_peered":"2024-09-21T10:38:57.114840+0000","last_clean":"2024-09-21T10:38:57.114840+0000","last_became_active":"2024-09-21T10:09:06.568439+0000","last_became_peered":"2024-09-21T10:09:06.568439+0000","last_unstale":"2024-09-21T10:38:57.114840+0000","last_undegraded":"2024-09-21T10:38:57.114840+0000","last_fullsized":"2024-09-21T10:38:57.114840+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:17.633827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1881,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115718+0000","last_change":"2024-09-21T10:07:56.006236+0000","last_active":"2024-09-21T10:38:57.115718+0000","last_peered":"2024-09-21T10:38:57.115718+0000","last_clean":"2024-09-21T10:38:57.115718+0000","last_became_active":"2024-09-21T10:07:56.004439+0000","last_became_peered":"2024-09-21T10:07:56.004439+0000","last_unstale":"2024-09-21T10:38:57.115718+0000","last_undegraded":"2024-09-21T10:38:57.115718+0000","last_fullsized":"2024-09-21T10:38:57.115718+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:39.422467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":1876,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.121117+0000","last_change":"2024-09-21T10:09:06.578081+0000","last_active":"2024-09-21T10:38:57.121117+0000","last_peered":"2024-09-21T10:38:57.121117+0000","last_clean":"2024-09-21T10:38:57.121117+0000","last_became_active":"2024-09-21T10:09:06.575129+0000","last_became_peered":"2024-09-21T10:09:06.575129+0000","last_unstale":"2024-09-21T10:38:57.121117+0000","last_undegraded":"2024-09-21T10:38:57.121117+0000","last_fullsized":"2024-09-21T10:38:57.121117+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:47:26.651862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1887,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115125+0000","last_change":"2024-09-21T10:07:21.531644+0000","last_active":"2024-09-21T10:38:57.115125+0000","last_peered":"2024-09-21T10:38:57.115125+0000","last_clean":"2024-09-21T10:38:57.115125+0000","last_became_active":"2024-09-21T10:07:21.520764+0000","last_became_peered":"2024-09-21T10:07:21.520764+0000","last_unstale":"2024-09-21T10:38:57.115125+0000","last_undegraded":"2024-09-21T10:38:57.115125+0000","last_fullsized":"2024-09-21T10:38:57.115125+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:11:32.147647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1288,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.081786+0000","last_change":"2024-09-21T10:07:21.499183+0000","last_active":"2024-09-21T10:38:53.081786+0000","last_peered":"2024-09-21T10:38:53.081786+0000","last_clean":"2024-09-21T10:38:53.081786+0000","last_became_active":"2024-09-21T10:07:21.498610+0000","last_became_peered":"2024-09-21T10:07:21.498610+0000","last_unstale":"2024-09-21T10:38:53.081786+0000","last_undegraded":"2024-09-21T10:38:53.081786+0000","last_fullsized":"2024-09-21T10:38:53.081786+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:20:17.212088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1910,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.118834+0000","last_change":"2024-09-21T10:08:30.588282+0000","last_active":"2024-09-21T10:38:57.118834+0000","last_peered":"2024-09-21T10:38:57.118834+0000","last_clean":"2024-09-21T10:38:57.118834+0000","last_became_active":"2024-09-21T10:08:30.587736+0000","last_became_peered":"2024-09-21T10:08:30.587736+0000","last_unstale":"2024-09-21T10:38:57.118834+0000","last_undegraded":"2024-09-21T10:38:57.118834+0000","last_fullsized":"2024-09-21T10:38:57.118834+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:22:18.890670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1872,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.117294+0000","last_change":"2024-09-21T10:07:56.067772+0000","last_active":"2024-09-21T10:38:57.117294+0000","last_peered":"2024-09-21T10:38:57.117294+0000","last_clean":"2024-09-21T10:38:57.117294+0000","last_became_active":"2024-09-21T10:07:56.052602+0000","last_became_peered":"2024-09-21T10:07:56.052602+0000","last_unstale":"2024-09-21T10:38:57.117294+0000","last_undegraded":"2024-09-21T10:38:57.117294+0000","last_fullsized":"2024-09-21T10:38:57.117294+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:07:59.899214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1926,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.118605+0000","last_change":"2024-09-21T10:07:56.017999+0000","last_active":"2024-09-21T10:38:57.118605+0000","last_peered":"2024-09-21T10:38:57.118605+0000","last_clean":"2024-09-21T10:38:57.118605+0000","last_became_active":"2024-09-21T10:07:56.017099+0000","last_became_peered":"2024-09-21T10:07:56.017099+0000","last_unstale":"2024-09-21T10:38:57.118605+0000","last_undegraded":"2024-09-21T10:38:57.118605+0000","last_fullsized":"2024-09-21T10:38:57.118605+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:58.288984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1289,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:55.981301+0000","last_change":"2024-09-21T10:09:06.558796+0000","last_active":"2024-09-21T10:38:55.981301+0000","last_peered":"2024-09-21T10:38:55.981301+0000","last_clean":"2024-09-21T10:38:55.981301+0000","last_became_active":"2024-09-21T10:09:06.557834+0000","last_became_peered":"2024-09-21T10:09:06.557834+0000","last_unstale":"2024-09-21T10:38:55.981301+0000","last_undegraded":"2024-09-21T10:38:55.981301+0000","last_fullsized":"2024-09-21T10:38:55.981301+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:02:46.366855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1920,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.118992+0000","last_change":"2024-09-21T10:09:06.534003+0000","last_active":"2024-09-21T10:38:57.118992+0000","last_peered":"2024-09-21T10:38:57.118992+0000","last_clean":"2024-09-21T10:38:57.118992+0000","last_became_active":"2024-09-21T10:09:06.533714+0000","last_became_peered":"2024-09-21T10:09:06.533714+0000","last_unstale":"2024-09-21T10:38:57.118992+0000","last_undegraded":"2024-09-21T10:38:57.118992+0000","last_fullsized":"2024-09-21T10:38:57.118992+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:15.622055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1274,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.702267+0000","last_change":"2024-09-21T10:07:21.524082+0000","last_active":"2024-09-21T10:38:53.702267+0000","last_peered":"2024-09-21T10:38:53.702267+0000","last_clean":"2024-09-21T10:38:53.702267+0000","last_became_active":"2024-09-21T10:07:21.523779+0000","last_became_peered":"2024-09-21T10:07:21.523779+0000","last_unstale":"2024-09-21T10:38:53.702267+0000","last_undegraded":"2024-09-21T10:38:53.702267+0000","last_fullsized":"2024-09-21T10:38:53.702267+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:13.813832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":1289,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114172+0000","last_change":"2024-09-21T10:07:56.024239+0000","last_active":"2024-09-21T10:38:57.114172+0000","last_peered":"2024-09-21T10:38:57.114172+0000","last_clean":"2024-09-21T10:38:57.114172+0000","last_became_active":"2024-09-21T10:07:56.023914+0000","last_became_peered":"2024-09-21T10:07:56.023914+0000","last_unstale":"2024-09-21T10:38:57.114172+0000","last_undegraded":"2024-09-21T10:38:57.114172+0000","last_fullsized":"2024-09-21T10:38:57.114172+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:23.361736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084477399999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1909,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115265+0000","last_change":"2024-09-21T10:07:56.056338+0000","last_active":"2024-09-21T10:38:57.115265+0000","last_peered":"2024-09-21T10:38:57.115265+0000","last_clean":"2024-09-21T10:38:57.115265+0000","last_became_active":"2024-09-21T10:07:56.052888+0000","last_became_peered":"2024-09-21T10:07:56.052888+0000","last_unstale":"2024-09-21T10:38:57.115265+0000","last_undegraded":"2024-09-21T10:38:57.115265+0000","last_fullsized":"2024-09-21T10:38:57.115265+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:34:48.662436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1883,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114052+0000","last_change":"2024-09-21T10:08:30.599270+0000","last_active":"2024-09-21T10:38:57.114052+0000","last_peered":"2024-09-21T10:38:57.114052+0000","last_clean":"2024-09-21T10:38:57.114052+0000","last_became_active":"2024-09-21T10:08:30.598873+0000","last_became_peered":"2024-09-21T10:08:30.598873+0000","last_unstale":"2024-09-21T10:38:57.114052+0000","last_undegraded":"2024-09-21T10:38:57.114052+0000","last_fullsized":"2024-09-21T10:38:57.114052+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:15:19.305854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1916,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.116026+0000","last_change":"2024-09-21T10:09:06.542888+0000","last_active":"2024-09-21T10:38:57.116026+0000","last_peered":"2024-09-21T10:38:57.116026+0000","last_clean":"2024-09-21T10:38:57.116026+0000","last_became_active":"2024-09-21T10:09:06.542593+0000","last_became_peered":"2024-09-21T10:09:06.542593+0000","last_unstale":"2024-09-21T10:38:57.116026+0000","last_undegraded":"2024-09-21T10:38:57.116026+0000","last_fullsized":"2024-09-21T10:38:57.116026+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:18:24.657282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1908,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115920+0000","last_change":"2024-09-21T10:08:30.589052+0000","last_active":"2024-09-21T10:38:57.115920+0000","last_peered":"2024-09-21T10:38:57.115920+0000","last_clean":"2024-09-21T10:38:57.115920+0000","last_became_active":"2024-09-21T10:08:30.588532+0000","last_became_peered":"2024-09-21T10:08:30.588532+0000","last_unstale":"2024-09-21T10:38:57.115920+0000","last_undegraded":"2024-09-21T10:38:57.115920+0000","last_fullsized":"2024-09-21T10:38:57.115920+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:55:48.460046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1896,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.116246+0000","last_change":"2024-09-21T10:09:06.560130+0000","last_active":"2024-09-21T10:38:57.116246+0000","last_peered":"2024-09-21T10:38:57.116246+0000","last_clean":"2024-09-21T10:38:57.116246+0000","last_became_active":"2024-09-21T10:09:06.559827+0000","last_became_peered":"2024-09-21T10:09:06.559827+0000","last_unstale":"2024-09-21T10:38:57.116246+0000","last_undegraded":"2024-09-21T10:38:57.116246+0000","last_fullsized":"2024-09-21T10:38:57.116246+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:05:17.166734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1304,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.081095+0000","last_change":"2024-09-21T10:07:55.995501+0000","last_active":"2024-09-21T10:38:53.081095+0000","last_peered":"2024-09-21T10:38:53.081095+0000","last_clean":"2024-09-21T10:38:53.081095+0000","last_became_active":"2024-09-21T10:07:55.995188+0000","last_became_peered":"2024-09-21T10:07:55.995188+0000","last_unstale":"2024-09-21T10:38:53.081095+0000","last_undegraded":"2024-09-21T10:38:53.081095+0000","last_fullsized":"2024-09-21T10:38:53.081095+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:49:30.127774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1259,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:56.085563+0000","last_change":"2024-09-21T09:54:53.722213+0000","last_active":"2024-09-21T10:38:56.085563+0000","last_peered":"2024-09-21T10:38:56.085563+0000","last_clean":"2024-09-21T10:38:56.085563+0000","last_became_active":"2024-09-21T09:54:53.721437+0000","last_became_peered":"2024-09-21T09:54:53.721437+0000","last_unstale":"2024-09-21T10:38:56.085563+0000","last_undegraded":"2024-09-21T10:38:56.085563+0000","last_fullsized":"2024-09-21T10:38:56.085563+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:01:41.456731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1877,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115726+0000","last_change":"2024-09-21T10:09:06.560589+0000","last_active":"2024-09-21T10:38:57.115726+0000","last_peered":"2024-09-21T10:38:57.115726+0000","last_clean":"2024-09-21T10:38:57.115726+0000","last_became_active":"2024-09-21T10:09:06.557552+0000","last_became_peered":"2024-09-21T10:09:06.557552+0000","last_unstale":"2024-09-21T10:38:57.115726+0000","last_undegraded":"2024-09-21T10:38:57.115726+0000","last_fullsized":"2024-09-21T10:38:57.115726+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:09:24.866759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":1288,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115185+0000","last_change":"2024-09-21T10:08:30.576357+0000","last_active":"2024-09-21T10:38:57.115185+0000","last_peered":"2024-09-21T10:38:57.115185+0000","last_clean":"2024-09-21T10:38:57.115185+0000","last_became_active":"2024-09-21T10:08:30.576033+0000","last_became_peered":"2024-09-21T10:08:30.576033+0000","last_unstale":"2024-09-21T10:38:57.115185+0000","last_undegraded":"2024-09-21T10:38:57.115185+0000","last_fullsized":"2024-09-21T10:38:57.115185+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:21:45.522197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":1606,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.091312+0000","last_change":"2024-09-21T10:07:21.499838+0000","last_active":"2024-09-21T10:38:57.091312+0000","last_peered":"2024-09-21T10:38:57.091312+0000","last_clean":"2024-09-21T10:38:57.091312+0000","last_became_active":"2024-09-21T10:07:21.499449+0000","last_became_peered":"2024-09-21T10:07:21.499449+0000","last_unstale":"2024-09-21T10:38:57.091312+0000","last_undegraded":"2024-09-21T10:38:57.091312+0000","last_fullsized":"2024-09-21T10:38:57.091312+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:59:19.260791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1882,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115215+0000","last_change":"2024-09-21T10:09:06.538940+0000","last_active":"2024-09-21T10:38:57.115215+0000","last_peered":"2024-09-21T10:38:57.115215+0000","last_clean":"2024-09-21T10:38:57.115215+0000","last_became_active":"2024-09-21T10:09:06.538668+0000","last_became_peered":"2024-09-21T10:09:06.538668+0000","last_unstale":"2024-09-21T10:38:57.115215+0000","last_undegraded":"2024-09-21T10:38:57.115215+0000","last_fullsized":"2024-09-21T10:38:57.115215+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:42:44.299488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1883,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114655+0000","last_change":"2024-09-21T10:09:06.582426+0000","last_active":"2024-09-21T10:38:57.114655+0000","last_peered":"2024-09-21T10:38:57.114655+0000","last_clean":"2024-09-21T10:38:57.114655+0000","last_became_active":"2024-09-21T10:09:06.580117+0000","last_became_peered":"2024-09-21T10:09:06.580117+0000","last_unstale":"2024-09-21T10:38:57.114655+0000","last_undegraded":"2024-09-21T10:38:57.114655+0000","last_fullsized":"2024-09-21T10:38:57.114655+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:56:45.278650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1271,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.702325+0000","last_change":"2024-09-21T10:07:55.998799+0000","last_active":"2024-09-21T10:38:53.702325+0000","last_peered":"2024-09-21T10:38:53.702325+0000","last_clean":"2024-09-21T10:38:53.702325+0000","last_became_active":"2024-09-21T10:07:55.998510+0000","last_became_peered":"2024-09-21T10:07:55.998510+0000","last_unstale":"2024-09-21T10:38:53.702325+0000","last_undegraded":"2024-09-21T10:38:53.702325+0000","last_fullsized":"2024-09-21T10:38:53.702325+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:10.474008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1273,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.701779+0000","last_change":"2024-09-21T10:08:30.602686+0000","last_active":"2024-09-21T10:38:53.701779+0000","last_peered":"2024-09-21T10:38:53.701779+0000","last_clean":"2024-09-21T10:38:53.701779+0000","last_became_active":"2024-09-21T10:08:30.602363+0000","last_became_peered":"2024-09-21T10:08:30.602363+0000","last_unstale":"2024-09-21T10:38:53.701779+0000","last_undegraded":"2024-09-21T10:38:53.701779+0000","last_fullsized":"2024-09-21T10:38:53.701779+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:10.278387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1286,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.702194+0000","last_change":"2024-09-21T10:08:30.601654+0000","last_active":"2024-09-21T10:38:53.702194+0000","last_peered":"2024-09-21T10:38:53.702194+0000","last_clean":"2024-09-21T10:38:53.702194+0000","last_became_active":"2024-09-21T10:08:30.601354+0000","last_became_peered":"2024-09-21T10:08:30.601354+0000","last_unstale":"2024-09-21T10:38:53.702194+0000","last_undegraded":"2024-09-21T10:38:53.702194+0000","last_fullsized":"2024-09-21T10:38:53.702194+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:42:25.902106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1274,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.321824+0000","last_change":"2024-09-21T10:07:56.020791+0000","last_active":"2024-09-21T10:38:53.321824+0000","last_peered":"2024-09-21T10:38:53.321824+0000","last_clean":"2024-09-21T10:38:53.321824+0000","last_became_active":"2024-09-21T10:07:56.020205+0000","last_became_peered":"2024-09-21T10:07:56.020205+0000","last_unstale":"2024-09-21T10:38:53.321824+0000","last_undegraded":"2024-09-21T10:38:53.321824+0000","last_fullsized":"2024-09-21T10:38:53.321824+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:09.813166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1271,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.701976+0000","last_change":"2024-09-21T10:09:06.533271+0000","last_active":"2024-09-21T10:38:53.701976+0000","last_peered":"2024-09-21T10:38:53.701976+0000","last_clean":"2024-09-21T10:38:53.701976+0000","last_became_active":"2024-09-21T10:09:06.532041+0000","last_became_peered":"2024-09-21T10:09:06.532041+0000","last_unstale":"2024-09-21T10:38:53.701976+0000","last_undegraded":"2024-09-21T10:38:53.701976+0000","last_fullsized":"2024-09-21T10:38:53.701976+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:04.236375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":1898,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.119850+0000","last_change":"2024-09-21T10:09:06.543422+0000","last_active":"2024-09-21T10:38:57.119850+0000","last_peered":"2024-09-21T10:38:57.119850+0000","last_clean":"2024-09-21T10:38:57.119850+0000","last_became_active":"2024-09-21T10:09:06.539647+0000","last_became_peered":"2024-09-21T10:09:06.539647+0000","last_unstale":"2024-09-21T10:38:57.119850+0000","last_undegraded":"2024-09-21T10:38:57.119850+0000","last_fullsized":"2024-09-21T10:38:57.119850+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:04.990085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1897,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114294+0000","last_change":"2024-09-21T10:07:56.018560+0000","last_active":"2024-09-21T10:38:57.114294+0000","last_peered":"2024-09-21T10:38:57.114294+0000","last_clean":"2024-09-21T10:38:57.114294+0000","last_became_active":"2024-09-21T10:07:56.018261+0000","last_became_peered":"2024-09-21T10:07:56.018261+0000","last_unstale":"2024-09-21T10:38:57.114294+0000","last_undegraded":"2024-09-21T10:38:57.114294+0000","last_fullsized":"2024-09-21T10:38:57.114294+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:04:54.395456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1272,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:54.020787+0000","last_change":"2024-09-21T10:09:06.531935+0000","last_active":"2024-09-21T10:38:54.020787+0000","last_peered":"2024-09-21T10:38:54.020787+0000","last_clean":"2024-09-21T10:38:54.020787+0000","last_became_active":"2024-09-21T10:09:06.531610+0000","last_became_peered":"2024-09-21T10:09:06.531610+0000","last_unstale":"2024-09-21T10:38:54.020787+0000","last_undegraded":"2024-09-21T10:38:54.020787+0000","last_fullsized":"2024-09-21T10:38:54.020787+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:33.874405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1282,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.321716+0000","last_change":"2024-09-21T10:08:30.586447+0000","last_active":"2024-09-21T10:38:53.321716+0000","last_peered":"2024-09-21T10:38:53.321716+0000","last_clean":"2024-09-21T10:38:53.321716+0000","last_became_active":"2024-09-21T10:08:30.585434+0000","last_became_peered":"2024-09-21T10:08:30.585434+0000","last_unstale":"2024-09-21T10:38:53.321716+0000","last_undegraded":"2024-09-21T10:38:53.321716+0000","last_fullsized":"2024-09-21T10:38:53.321716+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:51:30.174217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":1260,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.081691+0000","last_change":"2024-09-21T09:54:53.628923+0000","last_active":"2024-09-21T10:38:53.081691+0000","last_peered":"2024-09-21T10:38:53.081691+0000","last_clean":"2024-09-21T10:38:53.081691+0000","last_became_active":"2024-09-21T09:54:53.627939+0000","last_became_peered":"2024-09-21T09:54:53.627939+0000","last_unstale":"2024-09-21T10:38:53.081691+0000","last_undegraded":"2024-09-21T10:38:53.081691+0000","last_fullsized":"2024-09-21T10:38:53.081691+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:07:02.820689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1287,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.701674+0000","last_change":"2024-09-21T10:09:06.559132+0000","last_active":"2024-09-21T10:38:53.701674+0000","last_peered":"2024-09-21T10:38:53.701674+0000","last_clean":"2024-09-21T10:38:53.701674+0000","last_became_active":"2024-09-21T10:09:06.558395+0000","last_became_peered":"2024-09-21T10:09:06.558395+0000","last_unstale":"2024-09-21T10:38:53.701674+0000","last_undegraded":"2024-09-21T10:38:53.701674+0000","last_fullsized":"2024-09-21T10:38:53.701674+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:27:00.073554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":1276,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.081928+0000","last_change":"2024-09-21T10:07:56.029192+0000","last_active":"2024-09-21T10:38:53.081928+0000","last_peered":"2024-09-21T10:38:53.081928+0000","last_clean":"2024-09-21T10:38:53.081928+0000","last_became_active":"2024-09-21T10:07:56.028747+0000","last_became_peered":"2024-09-21T10:07:56.028747+0000","last_unstale":"2024-09-21T10:38:53.081928+0000","last_undegraded":"2024-09-21T10:38:53.081928+0000","last_fullsized":"2024-09-21T10:38:53.081928+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:08.918337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1257,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:54.019865+0000","last_change":"2024-09-21T09:54:54.677572+0000","last_active":"2024-09-21T10:38:54.019865+0000","last_peered":"2024-09-21T10:38:54.019865+0000","last_clean":"2024-09-21T10:38:54.019865+0000","last_became_active":"2024-09-21T09:54:54.677253+0000","last_became_peered":"2024-09-21T09:54:54.677253+0000","last_unstale":"2024-09-21T10:38:54.019865+0000","last_undegraded":"2024-09-21T10:38:54.019865+0000","last_fullsized":"2024-09-21T10:38:54.019865+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:31:40.629299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":1275,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.081671+0000","last_change":"2024-09-21T10:07:21.499297+0000","last_active":"2024-09-21T10:38:53.081671+0000","last_peered":"2024-09-21T10:38:53.081671+0000","last_clean":"2024-09-21T10:38:53.081671+0000","last_became_active":"2024-09-21T10:07:21.498961+0000","last_became_peered":"2024-09-21T10:07:21.498961+0000","last_unstale":"2024-09-21T10:38:53.081671+0000","last_undegraded":"2024-09-21T10:38:53.081671+0000","last_fullsized":"2024-09-21T10:38:53.081671+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:07.281950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1281,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114824+0000","last_change":"2024-09-21T10:07:21.497877+0000","last_active":"2024-09-21T10:38:57.114824+0000","last_peered":"2024-09-21T10:38:57.114824+0000","last_clean":"2024-09-21T10:38:57.114824+0000","last_became_active":"2024-09-21T10:07:21.496317+0000","last_became_peered":"2024-09-21T10:07:21.496317+0000","last_unstale":"2024-09-21T10:38:57.114824+0000","last_undegraded":"2024-09-21T10:38:57.114824+0000","last_fullsized":"2024-09-21T10:38:57.114824+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:39:03.848183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1879,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.121039+0000","last_change":"2024-09-21T10:09:06.575204+0000","last_active":"2024-09-21T10:38:57.121039+0000","last_peered":"2024-09-21T10:38:57.121039+0000","last_clean":"2024-09-21T10:38:57.121039+0000","last_became_active":"2024-09-21T10:09:06.571218+0000","last_became_peered":"2024-09-21T10:09:06.571218+0000","last_unstale":"2024-09-21T10:38:57.121039+0000","last_undegraded":"2024-09-21T10:38:57.121039+0000","last_fullsized":"2024-09-21T10:38:57.121039+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:27.754524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1259,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.092165+0000","last_change":"2024-09-21T09:54:53.625016+0000","last_active":"2024-09-21T10:38:57.092165+0000","last_peered":"2024-09-21T10:38:57.092165+0000","last_clean":"2024-09-21T10:38:57.092165+0000","last_became_active":"2024-09-21T09:54:53.624750+0000","last_became_peered":"2024-09-21T09:54:53.624750+0000","last_unstale":"2024-09-21T10:38:57.092165+0000","last_undegraded":"2024-09-21T10:38:57.092165+0000","last_fullsized":"2024-09-21T10:38:57.092165+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:34:26.823544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1910,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115602+0000","last_change":"2024-09-21T10:09:06.558492+0000","last_active":"2024-09-21T10:38:57.115602+0000","last_peered":"2024-09-21T10:38:57.115602+0000","last_clean":"2024-09-21T10:38:57.115602+0000","last_became_active":"2024-09-21T10:09:06.558146+0000","last_became_peered":"2024-09-21T10:09:06.558146+0000","last_unstale":"2024-09-21T10:38:57.115602+0000","last_undegraded":"2024-09-21T10:38:57.115602+0000","last_fullsized":"2024-09-21T10:38:57.115602+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:28:50.638241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1893,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.117388+0000","last_change":"2024-09-21T10:09:06.579005+0000","last_active":"2024-09-21T10:38:57.117388+0000","last_peered":"2024-09-21T10:38:57.117388+0000","last_clean":"2024-09-21T10:38:57.117388+0000","last_became_active":"2024-09-21T10:09:06.578119+0000","last_became_peered":"2024-09-21T10:09:06.578119+0000","last_unstale":"2024-09-21T10:38:57.117388+0000","last_undegraded":"2024-09-21T10:38:57.117388+0000","last_fullsized":"2024-09-21T10:38:57.117388+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:20.615838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1270,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:54.020627+0000","last_change":"2024-09-21T10:09:06.557453+0000","last_active":"2024-09-21T10:38:54.020627+0000","last_peered":"2024-09-21T10:38:54.020627+0000","last_clean":"2024-09-21T10:38:54.020627+0000","last_became_active":"2024-09-21T10:09:06.557124+0000","last_became_peered":"2024-09-21T10:09:06.557124+0000","last_unstale":"2024-09-21T10:38:54.020627+0000","last_undegraded":"2024-09-21T10:38:54.020627+0000","last_fullsized":"2024-09-21T10:38:54.020627+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:03:27.731832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":1276,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.091450+0000","last_change":"2024-09-21T10:08:30.574245+0000","last_active":"2024-09-21T10:38:57.091450+0000","last_peered":"2024-09-21T10:38:57.091450+0000","last_clean":"2024-09-21T10:38:57.091450+0000","last_became_active":"2024-09-21T10:08:30.573909+0000","last_became_peered":"2024-09-21T10:08:30.573909+0000","last_unstale":"2024-09-21T10:38:57.091450+0000","last_undegraded":"2024-09-21T10:38:57.091450+0000","last_fullsized":"2024-09-21T10:38:57.091450+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:18:03.911157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1880,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.120094+0000","last_change":"2024-09-21T10:09:06.578773+0000","last_active":"2024-09-21T10:38:57.120094+0000","last_peered":"2024-09-21T10:38:57.120094+0000","last_clean":"2024-09-21T10:38:57.120094+0000","last_became_active":"2024-09-21T10:09:06.576688+0000","last_became_peered":"2024-09-21T10:09:06.576688+0000","last_unstale":"2024-09-21T10:38:57.120094+0000","last_undegraded":"2024-09-21T10:38:57.120094+0000","last_fullsized":"2024-09-21T10:38:57.120094+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:37.699650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234549,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":140356,"kb_used_data":14968,"kb_used_omap":1,"kb_used_meta":125374,"kb_avail":93604796,"statfs":{"total":95995035648,"available":95851311104,"internally_reserved":0,"allocated":15327232,"data_stored":9832930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":128383435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496186,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":148540,"kb_used_data":14960,"kb_used_omap":1,"kb_used_meta":133566,"kb_avail":93596612,"statfs":{"total":95995035648,"available":95842930688,"internally_reserved":0,"allocated":15319040,"data_stored":9832803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":136772042},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757823,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":107824,"kb_used_data":14372,"kb_used_omap":1,"kb_used_meta":93438,"kb_avail":93637328,"statfs":{"total":95995035648,"available":95884623872,"internally_reserved":0,"allocated":14716928,"data_stored":9238261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95680966},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019459,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":117624,"kb_used_data":14384,"kb_used_omap":1,"kb_used_meta":103230,"kb_avail":93627528,"statfs":{"total":95995035648,"available":95874588672,"internally_reserved":0,"allocated":14729216,"data_stored":9242492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105707974},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369202,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78440,"kb_used_data":14364,"kb_used_omap":69,"kb_used_meta":63994,"kb_avail":93666712,"statfs":{"total":95995035648,"available":95914713088,"internally_reserved":0,"allocated":14708736,"data_stored":9237692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":65529986},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":680,"seq":2920577761657,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":68716,"kb_used_data":14368,"kb_used_omap":76,"kb_used_meta":54259,"kb_avail":93676436,"statfs":{"total":95995035648,"available":95924670464,"internally_reserved":0,"allocated":14712832,"data_stored":9238218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78585,"internal_metadata":55561479},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807957888,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86632,"kb_used_data":14364,"kb_used_omap":66,"kb_used_meta":72189,"kb_avail":93658520,"statfs":{"total":95995035648,"available":95906324480,"internally_reserved":0,"allocated":14708736,"data_stored":9237621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68589,"internal_metadata":73921555},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":666,"seq":2860448219527,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87024,"kb_used_data":14948,"kb_used_omap":76,"kb_used_meta":71987,"kb_avail":93658128,"statfs":{"total":95995035648,"available":95905923072,"internally_reserved":0,"allocated":15306752,"data_stored":9828574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78716,"internal_metadata":73714820},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:39:49.337 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T10:39:49.337 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T10:39:49.337 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-21T10:39:49.337 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph health --format=json 2024-09-21T10:39:49.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:49 smithi070 ceph-mon[107385]: from='client.44945 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:39:49.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:49 smithi070 ceph-mon[112139]: from='client.44945 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:39:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:49 smithi160 ceph-mon[101879]: from='client.44945 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:39:50.331 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:50 smithi070 ceph-mon[107385]: pgmap v2132: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:39:50.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:50 smithi070 ceph-mon[112139]: pgmap v2132: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:50.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:39:50.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:50 smithi160 ceph-mon[101879]: pgmap v2132: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:50.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:39:52.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:52 smithi070 ceph-mon[112139]: pgmap v2133: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:52.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:52 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:39:52.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:52 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:39:52.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:52 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:39:52.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:52 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:39:52.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:52 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:39:52.573 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:52 smithi070 ceph-mon[107385]: pgmap v2133: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:52.573 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:52 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:39:52.573 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:52 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:39:52.573 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:52 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:39:52.574 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:52 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:39:52.574 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:52 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:39:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:52 smithi160 ceph-mon[101879]: pgmap v2133: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:52 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:39:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:52 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:39:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:52 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:39:52.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:52 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:39:52.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:52 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:39:52.996 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:39:52.996 INFO:teuthology.orchestra.run.smithi070.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T10:39:53.292 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:39:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:39:53] "GET /metrics HTTP/1.1" 200 34047 "" "Prometheus/2.51.0" 2024-09-21T10:39:53.626 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:53 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2352229709' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T10:39:53.626 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:53 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2352229709' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T10:39:53.648 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-21T10:39:53.648 INFO:teuthology.run_tasks:Running task print... 2024-09-21T10:39:53.657 INFO:teuthology.task.print:**** applying stress + thrashing to mixed-version cluster **** 2024-09-21T10:39:53.657 INFO:teuthology.run_tasks:Running task parallel... 2024-09-21T10:39:53.713 INFO:teuthology.task.parallel:starting parallel... 2024-09-21T10:39:53.714 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-21T10:39:53.714 INFO:teuthology.task.sequential:In sequential, running task thrashosds... 2024-09-21T10:39:53.718 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_force_recovery': 0, 'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_thrash_pg_upmap_items': 0, 'disable_objectstore_tool_tests': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-21T10:39:53.718 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'create_rbd_pool': True, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'mons down', 'mon down', 'MON_DOWN', 'out of quorum', 'PG_AVAILABILITY'], 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'main', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}} 2024-09-21T10:39:53.719 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_force_recovery': 0, 'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_thrash_pg_upmap_items': 0, 'disable_objectstore_tool_tests': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-21T10:39:53.719 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-21T10:39:53.719 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T10:39:53.719 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:39:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:53 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2352229709' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T10:39:53.916 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:54.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:54 smithi070 ceph-mon[107385]: pgmap v2134: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:54.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:54 smithi070 ceph-mon[112139]: pgmap v2134: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:54 smithi160 ceph-mon[101879]: pgmap v2134: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:55.591 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 10:39:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=cleanup t=2024-09-21T10:39:55.129385325Z level=info msg="Completed cleanup jobs" duration=12.369375ms 2024-09-21T10:39:55.591 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 10:39:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=plugins.update.checker t=2024-09-21T10:39:55.270969013Z level=info msg="Update check succeeded" duration=65.186438ms 2024-09-21T10:39:55.661 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:39:55.661 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T10:39:56.361 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":2135,"stamp":"2024-09-21T10:39:54.649259+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595941,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":279,"num_write_kb":2449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":835156,"kb_used_data":116728,"kb_used_omap":296,"kb_used_meta":718039,"kb_avail":749126060,"statfs":{"total":767960285184,"available":767105085440,"internally_reserved":0,"allocated":119529472,"data_stored":75688591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":303807,"internal_metadata":735272257},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002574"},"pg_stats":[{"pgid":"3.1f","version":"71'4","reported_seq":1904,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.116537+0000","last_change":"2024-09-21T10:08:30.578838+0000","last_active":"2024-09-21T10:38:57.116537+0000","last_peered":"2024-09-21T10:38:57.116537+0000","last_clean":"2024-09-21T10:38:57.116537+0000","last_became_active":"2024-09-21T10:08:30.577773+0000","last_became_peered":"2024-09-21T10:08:30.577773+0000","last_unstale":"2024-09-21T10:38:57.116537+0000","last_undegraded":"2024-09-21T10:38:57.116537+0000","last_fullsized":"2024-09-21T10:38:57.116537+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:46:51.619094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1270,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.701486+0000","last_change":"2024-09-21T10:07:55.999392+0000","last_active":"2024-09-21T10:38:53.701486+0000","last_peered":"2024-09-21T10:38:53.701486+0000","last_clean":"2024-09-21T10:38:53.701486+0000","last_became_active":"2024-09-21T10:07:55.999103+0000","last_became_peered":"2024-09-21T10:07:55.999103+0000","last_unstale":"2024-09-21T10:38:53.701486+0000","last_undegraded":"2024-09-21T10:38:53.701486+0000","last_fullsized":"2024-09-21T10:38:53.701486+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:48.757242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":1891,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.116274+0000","last_change":"2024-09-21T10:09:06.592799+0000","last_active":"2024-09-21T10:38:57.116274+0000","last_peered":"2024-09-21T10:38:57.116274+0000","last_clean":"2024-09-21T10:38:57.116274+0000","last_became_active":"2024-09-21T10:09:06.592468+0000","last_became_peered":"2024-09-21T10:09:06.592468+0000","last_unstale":"2024-09-21T10:38:57.116274+0000","last_undegraded":"2024-09-21T10:38:57.116274+0000","last_fullsized":"2024-09-21T10:38:57.116274+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:45:04.534377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1908,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.119459+0000","last_change":"2024-09-21T10:09:06.584044+0000","last_active":"2024-09-21T10:38:57.119459+0000","last_peered":"2024-09-21T10:38:57.119459+0000","last_clean":"2024-09-21T10:38:57.119459+0000","last_became_active":"2024-09-21T10:09:06.583761+0000","last_became_peered":"2024-09-21T10:09:06.583761+0000","last_unstale":"2024-09-21T10:38:57.119459+0000","last_undegraded":"2024-09-21T10:38:57.119459+0000","last_fullsized":"2024-09-21T10:38:57.119459+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:53.839091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":1257,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:54.019511+0000","last_change":"2024-09-21T09:54:53.634508+0000","last_active":"2024-09-21T10:38:54.019511+0000","last_peered":"2024-09-21T10:38:54.019511+0000","last_clean":"2024-09-21T10:38:54.019511+0000","last_became_active":"2024-09-21T09:54:53.634254+0000","last_became_peered":"2024-09-21T09:54:53.634254+0000","last_unstale":"2024-09-21T10:38:54.019511+0000","last_undegraded":"2024-09-21T10:38:54.019511+0000","last_fullsized":"2024-09-21T10:38:54.019511+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:18.122107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1284,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.701607+0000","last_change":"2024-09-21T10:09:06.574933+0000","last_active":"2024-09-21T10:38:53.701607+0000","last_peered":"2024-09-21T10:38:53.701607+0000","last_clean":"2024-09-21T10:38:53.701607+0000","last_became_active":"2024-09-21T10:09:06.574467+0000","last_became_peered":"2024-09-21T10:09:06.574467+0000","last_unstale":"2024-09-21T10:38:53.701607+0000","last_undegraded":"2024-09-21T10:38:53.701607+0000","last_fullsized":"2024-09-21T10:38:53.701607+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:06.142269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":1272,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:54.020704+0000","last_change":"2024-09-21T10:07:55.991189+0000","last_active":"2024-09-21T10:38:54.020704+0000","last_peered":"2024-09-21T10:38:54.020704+0000","last_clean":"2024-09-21T10:38:54.020704+0000","last_became_active":"2024-09-21T10:07:55.990784+0000","last_became_peered":"2024-09-21T10:07:55.990784+0000","last_unstale":"2024-09-21T10:38:54.020704+0000","last_undegraded":"2024-09-21T10:38:54.020704+0000","last_fullsized":"2024-09-21T10:38:54.020704+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:00.580471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1284,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.702162+0000","last_change":"2024-09-21T10:09:06.560175+0000","last_active":"2024-09-21T10:38:53.702162+0000","last_peered":"2024-09-21T10:38:53.702162+0000","last_clean":"2024-09-21T10:38:53.702162+0000","last_became_active":"2024-09-21T10:09:06.558584+0000","last_became_peered":"2024-09-21T10:09:06.558584+0000","last_unstale":"2024-09-21T10:38:53.702162+0000","last_undegraded":"2024-09-21T10:38:53.702162+0000","last_fullsized":"2024-09-21T10:38:53.702162+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:02.918339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":1887,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115016+0000","last_change":"2024-09-21T10:07:21.532238+0000","last_active":"2024-09-21T10:38:57.115016+0000","last_peered":"2024-09-21T10:38:57.115016+0000","last_clean":"2024-09-21T10:38:57.115016+0000","last_became_active":"2024-09-21T10:07:21.522339+0000","last_became_peered":"2024-09-21T10:07:21.522339+0000","last_unstale":"2024-09-21T10:38:57.115016+0000","last_undegraded":"2024-09-21T10:38:57.115016+0000","last_fullsized":"2024-09-21T10:38:57.115016+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:34:09.116737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1908,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114132+0000","last_change":"2024-09-21T10:09:06.542669+0000","last_active":"2024-09-21T10:38:57.114132+0000","last_peered":"2024-09-21T10:38:57.114132+0000","last_clean":"2024-09-21T10:38:57.114132+0000","last_became_active":"2024-09-21T10:09:06.542384+0000","last_became_peered":"2024-09-21T10:09:06.542384+0000","last_unstale":"2024-09-21T10:38:57.114132+0000","last_undegraded":"2024-09-21T10:38:57.114132+0000","last_fullsized":"2024-09-21T10:38:57.114132+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:53.692798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":1286,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.701746+0000","last_change":"2024-09-21T10:08:30.598724+0000","last_active":"2024-09-21T10:38:53.701746+0000","last_peered":"2024-09-21T10:38:53.701746+0000","last_clean":"2024-09-21T10:38:53.701746+0000","last_became_active":"2024-09-21T10:08:30.598229+0000","last_became_peered":"2024-09-21T10:08:30.598229+0000","last_unstale":"2024-09-21T10:38:53.701746+0000","last_undegraded":"2024-09-21T10:38:53.701746+0000","last_fullsized":"2024-09-21T10:38:53.701746+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:57:57.056712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1902,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.118795+0000","last_change":"2024-09-21T10:09:06.590661+0000","last_active":"2024-09-21T10:38:57.118795+0000","last_peered":"2024-09-21T10:38:57.118795+0000","last_clean":"2024-09-21T10:38:57.118795+0000","last_became_active":"2024-09-21T10:09:06.588941+0000","last_became_peered":"2024-09-21T10:09:06.588941+0000","last_unstale":"2024-09-21T10:38:57.118795+0000","last_undegraded":"2024-09-21T10:38:57.118795+0000","last_fullsized":"2024-09-21T10:38:57.118795+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:18:09.719872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1881,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.118623+0000","last_change":"2024-09-21T10:09:06.530394+0000","last_active":"2024-09-21T10:38:57.118623+0000","last_peered":"2024-09-21T10:38:57.118623+0000","last_clean":"2024-09-21T10:38:57.118623+0000","last_became_active":"2024-09-21T10:09:06.529307+0000","last_became_peered":"2024-09-21T10:09:06.529307+0000","last_unstale":"2024-09-21T10:38:57.118623+0000","last_undegraded":"2024-09-21T10:38:57.118623+0000","last_fullsized":"2024-09-21T10:38:57.118623+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:54.287397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1899,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115865+0000","last_change":"2024-09-21T10:09:06.558433+0000","last_active":"2024-09-21T10:38:57.115865+0000","last_peered":"2024-09-21T10:38:57.115865+0000","last_clean":"2024-09-21T10:38:57.115865+0000","last_became_active":"2024-09-21T10:09:06.557907+0000","last_became_peered":"2024-09-21T10:09:06.557907+0000","last_unstale":"2024-09-21T10:38:57.115865+0000","last_undegraded":"2024-09-21T10:38:57.115865+0000","last_fullsized":"2024-09-21T10:38:57.115865+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:05.109492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1895,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114707+0000","last_change":"2024-09-21T10:09:06.552324+0000","last_active":"2024-09-21T10:38:57.114707+0000","last_peered":"2024-09-21T10:38:57.114707+0000","last_clean":"2024-09-21T10:38:57.114707+0000","last_became_active":"2024-09-21T10:09:06.540545+0000","last_became_peered":"2024-09-21T10:09:06.540545+0000","last_unstale":"2024-09-21T10:38:57.114707+0000","last_undegraded":"2024-09-21T10:38:57.114707+0000","last_fullsized":"2024-09-21T10:38:57.114707+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:08.506149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1274,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.322114+0000","last_change":"2024-09-21T10:07:56.003005+0000","last_active":"2024-09-21T10:38:53.322114+0000","last_peered":"2024-09-21T10:38:53.322114+0000","last_clean":"2024-09-21T10:38:53.322114+0000","last_became_active":"2024-09-21T10:07:56.001550+0000","last_became_peered":"2024-09-21T10:07:56.001550+0000","last_unstale":"2024-09-21T10:38:53.322114+0000","last_undegraded":"2024-09-21T10:38:53.322114+0000","last_fullsized":"2024-09-21T10:38:53.322114+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:54.553261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1888,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114926+0000","last_change":"2024-09-21T10:09:06.559017+0000","last_active":"2024-09-21T10:38:57.114926+0000","last_peered":"2024-09-21T10:38:57.114926+0000","last_clean":"2024-09-21T10:38:57.114926+0000","last_became_active":"2024-09-21T10:09:06.555256+0000","last_became_peered":"2024-09-21T10:09:06.555256+0000","last_unstale":"2024-09-21T10:38:57.114926+0000","last_undegraded":"2024-09-21T10:38:57.114926+0000","last_fullsized":"2024-09-21T10:38:57.114926+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:33:45.145138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1893,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.116730+0000","last_change":"2024-09-21T10:09:06.562504+0000","last_active":"2024-09-21T10:38:57.116730+0000","last_peered":"2024-09-21T10:38:57.116730+0000","last_clean":"2024-09-21T10:38:57.116730+0000","last_became_active":"2024-09-21T10:09:06.561747+0000","last_became_peered":"2024-09-21T10:09:06.561747+0000","last_unstale":"2024-09-21T10:38:57.116730+0000","last_undegraded":"2024-09-21T10:38:57.116730+0000","last_fullsized":"2024-09-21T10:38:57.116730+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:03:58.012406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1879,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.120517+0000","last_change":"2024-09-21T10:09:06.566050+0000","last_active":"2024-09-21T10:38:57.120517+0000","last_peered":"2024-09-21T10:38:57.120517+0000","last_clean":"2024-09-21T10:38:57.120517+0000","last_became_active":"2024-09-21T10:09:06.565791+0000","last_became_peered":"2024-09-21T10:09:06.565791+0000","last_unstale":"2024-09-21T10:38:57.120517+0000","last_undegraded":"2024-09-21T10:38:57.120517+0000","last_fullsized":"2024-09-21T10:38:57.120517+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:59:20.542888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1271,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:54.020837+0000","last_change":"2024-09-21T10:07:56.023722+0000","last_active":"2024-09-21T10:38:54.020837+0000","last_peered":"2024-09-21T10:38:54.020837+0000","last_clean":"2024-09-21T10:38:54.020837+0000","last_became_active":"2024-09-21T10:07:56.023085+0000","last_became_peered":"2024-09-21T10:07:56.023085+0000","last_unstale":"2024-09-21T10:38:54.020837+0000","last_undegraded":"2024-09-21T10:38:54.020837+0000","last_fullsized":"2024-09-21T10:38:54.020837+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:34:07.168469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":1276,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.081521+0000","last_change":"2024-09-21T10:09:06.572971+0000","last_active":"2024-09-21T10:38:53.081521+0000","last_peered":"2024-09-21T10:38:53.081521+0000","last_clean":"2024-09-21T10:38:53.081521+0000","last_became_active":"2024-09-21T10:09:06.571680+0000","last_became_peered":"2024-09-21T10:09:06.571680+0000","last_unstale":"2024-09-21T10:38:53.081521+0000","last_undegraded":"2024-09-21T10:38:53.081521+0000","last_fullsized":"2024-09-21T10:38:53.081521+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:21:45.051132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1880,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.117421+0000","last_change":"2024-09-21T10:08:30.589733+0000","last_active":"2024-09-21T10:38:57.117421+0000","last_peered":"2024-09-21T10:38:57.117421+0000","last_clean":"2024-09-21T10:38:57.117421+0000","last_became_active":"2024-09-21T10:08:30.589456+0000","last_became_peered":"2024-09-21T10:08:30.589456+0000","last_unstale":"2024-09-21T10:38:57.117421+0000","last_undegraded":"2024-09-21T10:38:57.117421+0000","last_fullsized":"2024-09-21T10:38:57.117421+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:09.654350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":1280,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.092674+0000","last_change":"2024-09-21T09:54:53.624887+0000","last_active":"2024-09-21T10:38:57.092674+0000","last_peered":"2024-09-21T10:38:57.092674+0000","last_clean":"2024-09-21T10:38:57.092674+0000","last_became_active":"2024-09-21T09:54:53.624525+0000","last_became_peered":"2024-09-21T09:54:53.624525+0000","last_unstale":"2024-09-21T10:38:57.092674+0000","last_undegraded":"2024-09-21T10:38:57.092674+0000","last_fullsized":"2024-09-21T10:38:57.092674+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:05:59.605316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1271,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.094087+0000","last_change":"2024-09-21T10:07:56.001756+0000","last_active":"2024-09-21T10:38:57.094087+0000","last_peered":"2024-09-21T10:38:57.094087+0000","last_clean":"2024-09-21T10:38:57.094087+0000","last_became_active":"2024-09-21T10:07:56.000789+0000","last_became_peered":"2024-09-21T10:07:56.000789+0000","last_unstale":"2024-09-21T10:38:57.094087+0000","last_undegraded":"2024-09-21T10:38:57.094087+0000","last_fullsized":"2024-09-21T10:38:57.094087+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:56:04.302902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1258,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.701544+0000","last_change":"2024-09-21T09:54:53.636902+0000","last_active":"2024-09-21T10:38:53.701544+0000","last_peered":"2024-09-21T10:38:53.701544+0000","last_clean":"2024-09-21T10:38:53.701544+0000","last_became_active":"2024-09-21T09:54:53.636616+0000","last_became_peered":"2024-09-21T09:54:53.636616+0000","last_unstale":"2024-09-21T10:38:53.701544+0000","last_undegraded":"2024-09-21T10:38:53.701544+0000","last_fullsized":"2024-09-21T10:38:53.701544+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:46:47.694495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1287,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.116875+0000","last_change":"2024-09-21T10:08:30.587337+0000","last_active":"2024-09-21T10:38:57.116875+0000","last_peered":"2024-09-21T10:38:57.116875+0000","last_clean":"2024-09-21T10:38:57.116875+0000","last_became_active":"2024-09-21T10:08:30.585404+0000","last_became_peered":"2024-09-21T10:08:30.585404+0000","last_unstale":"2024-09-21T10:38:57.116875+0000","last_undegraded":"2024-09-21T10:38:57.116875+0000","last_fullsized":"2024-09-21T10:38:57.116875+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:24:41.559947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":1901,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115763+0000","last_change":"2024-09-21T10:07:56.023462+0000","last_active":"2024-09-21T10:38:57.115763+0000","last_peered":"2024-09-21T10:38:57.115763+0000","last_clean":"2024-09-21T10:38:57.115763+0000","last_became_active":"2024-09-21T10:07:56.021859+0000","last_became_peered":"2024-09-21T10:07:56.021859+0000","last_unstale":"2024-09-21T10:38:57.115763+0000","last_undegraded":"2024-09-21T10:38:57.115763+0000","last_fullsized":"2024-09-21T10:38:57.115763+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:42:03.134729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1875,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114840+0000","last_change":"2024-09-21T10:09:06.580853+0000","last_active":"2024-09-21T10:38:57.114840+0000","last_peered":"2024-09-21T10:38:57.114840+0000","last_clean":"2024-09-21T10:38:57.114840+0000","last_became_active":"2024-09-21T10:09:06.568439+0000","last_became_peered":"2024-09-21T10:09:06.568439+0000","last_unstale":"2024-09-21T10:38:57.114840+0000","last_undegraded":"2024-09-21T10:38:57.114840+0000","last_fullsized":"2024-09-21T10:38:57.114840+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:17.633827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1881,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115718+0000","last_change":"2024-09-21T10:07:56.006236+0000","last_active":"2024-09-21T10:38:57.115718+0000","last_peered":"2024-09-21T10:38:57.115718+0000","last_clean":"2024-09-21T10:38:57.115718+0000","last_became_active":"2024-09-21T10:07:56.004439+0000","last_became_peered":"2024-09-21T10:07:56.004439+0000","last_unstale":"2024-09-21T10:38:57.115718+0000","last_undegraded":"2024-09-21T10:38:57.115718+0000","last_fullsized":"2024-09-21T10:38:57.115718+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:39.422467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":1876,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.121117+0000","last_change":"2024-09-21T10:09:06.578081+0000","last_active":"2024-09-21T10:38:57.121117+0000","last_peered":"2024-09-21T10:38:57.121117+0000","last_clean":"2024-09-21T10:38:57.121117+0000","last_became_active":"2024-09-21T10:09:06.575129+0000","last_became_peered":"2024-09-21T10:09:06.575129+0000","last_unstale":"2024-09-21T10:38:57.121117+0000","last_undegraded":"2024-09-21T10:38:57.121117+0000","last_fullsized":"2024-09-21T10:38:57.121117+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:47:26.651862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1887,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115125+0000","last_change":"2024-09-21T10:07:21.531644+0000","last_active":"2024-09-21T10:38:57.115125+0000","last_peered":"2024-09-21T10:38:57.115125+0000","last_clean":"2024-09-21T10:38:57.115125+0000","last_became_active":"2024-09-21T10:07:21.520764+0000","last_became_peered":"2024-09-21T10:07:21.520764+0000","last_unstale":"2024-09-21T10:38:57.115125+0000","last_undegraded":"2024-09-21T10:38:57.115125+0000","last_fullsized":"2024-09-21T10:38:57.115125+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:11:32.147647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1289,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.094214+0000","last_change":"2024-09-21T10:07:21.499183+0000","last_active":"2024-09-21T10:38:57.094214+0000","last_peered":"2024-09-21T10:38:57.094214+0000","last_clean":"2024-09-21T10:38:57.094214+0000","last_became_active":"2024-09-21T10:07:21.498610+0000","last_became_peered":"2024-09-21T10:07:21.498610+0000","last_unstale":"2024-09-21T10:38:57.094214+0000","last_undegraded":"2024-09-21T10:38:57.094214+0000","last_fullsized":"2024-09-21T10:38:57.094214+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:20:17.212088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1910,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.118834+0000","last_change":"2024-09-21T10:08:30.588282+0000","last_active":"2024-09-21T10:38:57.118834+0000","last_peered":"2024-09-21T10:38:57.118834+0000","last_clean":"2024-09-21T10:38:57.118834+0000","last_became_active":"2024-09-21T10:08:30.587736+0000","last_became_peered":"2024-09-21T10:08:30.587736+0000","last_unstale":"2024-09-21T10:38:57.118834+0000","last_undegraded":"2024-09-21T10:38:57.118834+0000","last_fullsized":"2024-09-21T10:38:57.118834+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:22:18.890670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1872,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.117294+0000","last_change":"2024-09-21T10:07:56.067772+0000","last_active":"2024-09-21T10:38:57.117294+0000","last_peered":"2024-09-21T10:38:57.117294+0000","last_clean":"2024-09-21T10:38:57.117294+0000","last_became_active":"2024-09-21T10:07:56.052602+0000","last_became_peered":"2024-09-21T10:07:56.052602+0000","last_unstale":"2024-09-21T10:38:57.117294+0000","last_undegraded":"2024-09-21T10:38:57.117294+0000","last_fullsized":"2024-09-21T10:38:57.117294+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:07:59.899214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1926,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.118605+0000","last_change":"2024-09-21T10:07:56.017999+0000","last_active":"2024-09-21T10:38:57.118605+0000","last_peered":"2024-09-21T10:38:57.118605+0000","last_clean":"2024-09-21T10:38:57.118605+0000","last_became_active":"2024-09-21T10:07:56.017099+0000","last_became_peered":"2024-09-21T10:07:56.017099+0000","last_unstale":"2024-09-21T10:38:57.118605+0000","last_undegraded":"2024-09-21T10:38:57.118605+0000","last_fullsized":"2024-09-21T10:38:57.118605+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:58.288984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1289,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:55.981301+0000","last_change":"2024-09-21T10:09:06.558796+0000","last_active":"2024-09-21T10:38:55.981301+0000","last_peered":"2024-09-21T10:38:55.981301+0000","last_clean":"2024-09-21T10:38:55.981301+0000","last_became_active":"2024-09-21T10:09:06.557834+0000","last_became_peered":"2024-09-21T10:09:06.557834+0000","last_unstale":"2024-09-21T10:38:55.981301+0000","last_undegraded":"2024-09-21T10:38:55.981301+0000","last_fullsized":"2024-09-21T10:38:55.981301+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:02:46.366855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1920,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.118992+0000","last_change":"2024-09-21T10:09:06.534003+0000","last_active":"2024-09-21T10:38:57.118992+0000","last_peered":"2024-09-21T10:38:57.118992+0000","last_clean":"2024-09-21T10:38:57.118992+0000","last_became_active":"2024-09-21T10:09:06.533714+0000","last_became_peered":"2024-09-21T10:09:06.533714+0000","last_unstale":"2024-09-21T10:38:57.118992+0000","last_undegraded":"2024-09-21T10:38:57.118992+0000","last_fullsized":"2024-09-21T10:38:57.118992+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:15.622055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1274,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.702267+0000","last_change":"2024-09-21T10:07:21.524082+0000","last_active":"2024-09-21T10:38:53.702267+0000","last_peered":"2024-09-21T10:38:53.702267+0000","last_clean":"2024-09-21T10:38:53.702267+0000","last_became_active":"2024-09-21T10:07:21.523779+0000","last_became_peered":"2024-09-21T10:07:21.523779+0000","last_unstale":"2024-09-21T10:38:53.702267+0000","last_undegraded":"2024-09-21T10:38:53.702267+0000","last_fullsized":"2024-09-21T10:38:53.702267+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:13.813832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":1289,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114172+0000","last_change":"2024-09-21T10:07:56.024239+0000","last_active":"2024-09-21T10:38:57.114172+0000","last_peered":"2024-09-21T10:38:57.114172+0000","last_clean":"2024-09-21T10:38:57.114172+0000","last_became_active":"2024-09-21T10:07:56.023914+0000","last_became_peered":"2024-09-21T10:07:56.023914+0000","last_unstale":"2024-09-21T10:38:57.114172+0000","last_undegraded":"2024-09-21T10:38:57.114172+0000","last_fullsized":"2024-09-21T10:38:57.114172+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:23.361736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084477399999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1909,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115265+0000","last_change":"2024-09-21T10:07:56.056338+0000","last_active":"2024-09-21T10:38:57.115265+0000","last_peered":"2024-09-21T10:38:57.115265+0000","last_clean":"2024-09-21T10:38:57.115265+0000","last_became_active":"2024-09-21T10:07:56.052888+0000","last_became_peered":"2024-09-21T10:07:56.052888+0000","last_unstale":"2024-09-21T10:38:57.115265+0000","last_undegraded":"2024-09-21T10:38:57.115265+0000","last_fullsized":"2024-09-21T10:38:57.115265+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:34:48.662436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1883,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114052+0000","last_change":"2024-09-21T10:08:30.599270+0000","last_active":"2024-09-21T10:38:57.114052+0000","last_peered":"2024-09-21T10:38:57.114052+0000","last_clean":"2024-09-21T10:38:57.114052+0000","last_became_active":"2024-09-21T10:08:30.598873+0000","last_became_peered":"2024-09-21T10:08:30.598873+0000","last_unstale":"2024-09-21T10:38:57.114052+0000","last_undegraded":"2024-09-21T10:38:57.114052+0000","last_fullsized":"2024-09-21T10:38:57.114052+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:15:19.305854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1917,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.116069+0000","last_change":"2024-09-21T10:09:06.542888+0000","last_active":"2024-09-21T10:38:57.116069+0000","last_peered":"2024-09-21T10:38:57.116069+0000","last_clean":"2024-09-21T10:38:57.116069+0000","last_became_active":"2024-09-21T10:09:06.542593+0000","last_became_peered":"2024-09-21T10:09:06.542593+0000","last_unstale":"2024-09-21T10:38:57.116069+0000","last_undegraded":"2024-09-21T10:38:57.116069+0000","last_fullsized":"2024-09-21T10:38:57.116069+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:18:24.657282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1909,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115959+0000","last_change":"2024-09-21T10:08:30.589052+0000","last_active":"2024-09-21T10:38:57.115959+0000","last_peered":"2024-09-21T10:38:57.115959+0000","last_clean":"2024-09-21T10:38:57.115959+0000","last_became_active":"2024-09-21T10:08:30.588532+0000","last_became_peered":"2024-09-21T10:08:30.588532+0000","last_unstale":"2024-09-21T10:38:57.115959+0000","last_undegraded":"2024-09-21T10:38:57.115959+0000","last_fullsized":"2024-09-21T10:38:57.115959+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:55:48.460046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1896,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.116246+0000","last_change":"2024-09-21T10:09:06.560130+0000","last_active":"2024-09-21T10:38:57.116246+0000","last_peered":"2024-09-21T10:38:57.116246+0000","last_clean":"2024-09-21T10:38:57.116246+0000","last_became_active":"2024-09-21T10:09:06.559827+0000","last_became_peered":"2024-09-21T10:09:06.559827+0000","last_unstale":"2024-09-21T10:38:57.116246+0000","last_undegraded":"2024-09-21T10:38:57.116246+0000","last_fullsized":"2024-09-21T10:38:57.116246+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:05:17.166734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1304,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.081095+0000","last_change":"2024-09-21T10:07:55.995501+0000","last_active":"2024-09-21T10:38:53.081095+0000","last_peered":"2024-09-21T10:38:53.081095+0000","last_clean":"2024-09-21T10:38:53.081095+0000","last_became_active":"2024-09-21T10:07:55.995188+0000","last_became_peered":"2024-09-21T10:07:55.995188+0000","last_unstale":"2024-09-21T10:38:53.081095+0000","last_undegraded":"2024-09-21T10:38:53.081095+0000","last_fullsized":"2024-09-21T10:38:53.081095+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:49:30.127774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1259,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:56.085563+0000","last_change":"2024-09-21T09:54:53.722213+0000","last_active":"2024-09-21T10:38:56.085563+0000","last_peered":"2024-09-21T10:38:56.085563+0000","last_clean":"2024-09-21T10:38:56.085563+0000","last_became_active":"2024-09-21T09:54:53.721437+0000","last_became_peered":"2024-09-21T09:54:53.721437+0000","last_unstale":"2024-09-21T10:38:56.085563+0000","last_undegraded":"2024-09-21T10:38:56.085563+0000","last_fullsized":"2024-09-21T10:38:56.085563+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:01:41.456731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1877,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115726+0000","last_change":"2024-09-21T10:09:06.560589+0000","last_active":"2024-09-21T10:38:57.115726+0000","last_peered":"2024-09-21T10:38:57.115726+0000","last_clean":"2024-09-21T10:38:57.115726+0000","last_became_active":"2024-09-21T10:09:06.557552+0000","last_became_peered":"2024-09-21T10:09:06.557552+0000","last_unstale":"2024-09-21T10:38:57.115726+0000","last_undegraded":"2024-09-21T10:38:57.115726+0000","last_fullsized":"2024-09-21T10:38:57.115726+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:09:24.866759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":1288,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115185+0000","last_change":"2024-09-21T10:08:30.576357+0000","last_active":"2024-09-21T10:38:57.115185+0000","last_peered":"2024-09-21T10:38:57.115185+0000","last_clean":"2024-09-21T10:38:57.115185+0000","last_became_active":"2024-09-21T10:08:30.576033+0000","last_became_peered":"2024-09-21T10:08:30.576033+0000","last_unstale":"2024-09-21T10:38:57.115185+0000","last_undegraded":"2024-09-21T10:38:57.115185+0000","last_fullsized":"2024-09-21T10:38:57.115185+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:21:45.522197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":1606,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.091312+0000","last_change":"2024-09-21T10:07:21.499838+0000","last_active":"2024-09-21T10:38:57.091312+0000","last_peered":"2024-09-21T10:38:57.091312+0000","last_clean":"2024-09-21T10:38:57.091312+0000","last_became_active":"2024-09-21T10:07:21.499449+0000","last_became_peered":"2024-09-21T10:07:21.499449+0000","last_unstale":"2024-09-21T10:38:57.091312+0000","last_undegraded":"2024-09-21T10:38:57.091312+0000","last_fullsized":"2024-09-21T10:38:57.091312+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:59:19.260791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1882,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115215+0000","last_change":"2024-09-21T10:09:06.538940+0000","last_active":"2024-09-21T10:38:57.115215+0000","last_peered":"2024-09-21T10:38:57.115215+0000","last_clean":"2024-09-21T10:38:57.115215+0000","last_became_active":"2024-09-21T10:09:06.538668+0000","last_became_peered":"2024-09-21T10:09:06.538668+0000","last_unstale":"2024-09-21T10:38:57.115215+0000","last_undegraded":"2024-09-21T10:38:57.115215+0000","last_fullsized":"2024-09-21T10:38:57.115215+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:42:44.299488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1883,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114655+0000","last_change":"2024-09-21T10:09:06.582426+0000","last_active":"2024-09-21T10:38:57.114655+0000","last_peered":"2024-09-21T10:38:57.114655+0000","last_clean":"2024-09-21T10:38:57.114655+0000","last_became_active":"2024-09-21T10:09:06.580117+0000","last_became_peered":"2024-09-21T10:09:06.580117+0000","last_unstale":"2024-09-21T10:38:57.114655+0000","last_undegraded":"2024-09-21T10:38:57.114655+0000","last_fullsized":"2024-09-21T10:38:57.114655+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:56:45.278650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1271,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.702325+0000","last_change":"2024-09-21T10:07:55.998799+0000","last_active":"2024-09-21T10:38:53.702325+0000","last_peered":"2024-09-21T10:38:53.702325+0000","last_clean":"2024-09-21T10:38:53.702325+0000","last_became_active":"2024-09-21T10:07:55.998510+0000","last_became_peered":"2024-09-21T10:07:55.998510+0000","last_unstale":"2024-09-21T10:38:53.702325+0000","last_undegraded":"2024-09-21T10:38:53.702325+0000","last_fullsized":"2024-09-21T10:38:53.702325+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:10.474008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1273,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.701779+0000","last_change":"2024-09-21T10:08:30.602686+0000","last_active":"2024-09-21T10:38:53.701779+0000","last_peered":"2024-09-21T10:38:53.701779+0000","last_clean":"2024-09-21T10:38:53.701779+0000","last_became_active":"2024-09-21T10:08:30.602363+0000","last_became_peered":"2024-09-21T10:08:30.602363+0000","last_unstale":"2024-09-21T10:38:53.701779+0000","last_undegraded":"2024-09-21T10:38:53.701779+0000","last_fullsized":"2024-09-21T10:38:53.701779+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:10.278387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1286,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.702194+0000","last_change":"2024-09-21T10:08:30.601654+0000","last_active":"2024-09-21T10:38:53.702194+0000","last_peered":"2024-09-21T10:38:53.702194+0000","last_clean":"2024-09-21T10:38:53.702194+0000","last_became_active":"2024-09-21T10:08:30.601354+0000","last_became_peered":"2024-09-21T10:08:30.601354+0000","last_unstale":"2024-09-21T10:38:53.702194+0000","last_undegraded":"2024-09-21T10:38:53.702194+0000","last_fullsized":"2024-09-21T10:38:53.702194+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:42:25.902106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1274,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.321824+0000","last_change":"2024-09-21T10:07:56.020791+0000","last_active":"2024-09-21T10:38:53.321824+0000","last_peered":"2024-09-21T10:38:53.321824+0000","last_clean":"2024-09-21T10:38:53.321824+0000","last_became_active":"2024-09-21T10:07:56.020205+0000","last_became_peered":"2024-09-21T10:07:56.020205+0000","last_unstale":"2024-09-21T10:38:53.321824+0000","last_undegraded":"2024-09-21T10:38:53.321824+0000","last_fullsized":"2024-09-21T10:38:53.321824+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:09.813166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1271,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.701976+0000","last_change":"2024-09-21T10:09:06.533271+0000","last_active":"2024-09-21T10:38:53.701976+0000","last_peered":"2024-09-21T10:38:53.701976+0000","last_clean":"2024-09-21T10:38:53.701976+0000","last_became_active":"2024-09-21T10:09:06.532041+0000","last_became_peered":"2024-09-21T10:09:06.532041+0000","last_unstale":"2024-09-21T10:38:53.701976+0000","last_undegraded":"2024-09-21T10:38:53.701976+0000","last_fullsized":"2024-09-21T10:38:53.701976+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:04.236375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":1898,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.119850+0000","last_change":"2024-09-21T10:09:06.543422+0000","last_active":"2024-09-21T10:38:57.119850+0000","last_peered":"2024-09-21T10:38:57.119850+0000","last_clean":"2024-09-21T10:38:57.119850+0000","last_became_active":"2024-09-21T10:09:06.539647+0000","last_became_peered":"2024-09-21T10:09:06.539647+0000","last_unstale":"2024-09-21T10:38:57.119850+0000","last_undegraded":"2024-09-21T10:38:57.119850+0000","last_fullsized":"2024-09-21T10:38:57.119850+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:04.990085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1897,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114294+0000","last_change":"2024-09-21T10:07:56.018560+0000","last_active":"2024-09-21T10:38:57.114294+0000","last_peered":"2024-09-21T10:38:57.114294+0000","last_clean":"2024-09-21T10:38:57.114294+0000","last_became_active":"2024-09-21T10:07:56.018261+0000","last_became_peered":"2024-09-21T10:07:56.018261+0000","last_unstale":"2024-09-21T10:38:57.114294+0000","last_undegraded":"2024-09-21T10:38:57.114294+0000","last_fullsized":"2024-09-21T10:38:57.114294+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:04:54.395456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1272,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:54.020787+0000","last_change":"2024-09-21T10:09:06.531935+0000","last_active":"2024-09-21T10:38:54.020787+0000","last_peered":"2024-09-21T10:38:54.020787+0000","last_clean":"2024-09-21T10:38:54.020787+0000","last_became_active":"2024-09-21T10:09:06.531610+0000","last_became_peered":"2024-09-21T10:09:06.531610+0000","last_unstale":"2024-09-21T10:38:54.020787+0000","last_undegraded":"2024-09-21T10:38:54.020787+0000","last_fullsized":"2024-09-21T10:38:54.020787+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:33.874405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1282,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.321716+0000","last_change":"2024-09-21T10:08:30.586447+0000","last_active":"2024-09-21T10:38:53.321716+0000","last_peered":"2024-09-21T10:38:53.321716+0000","last_clean":"2024-09-21T10:38:53.321716+0000","last_became_active":"2024-09-21T10:08:30.585434+0000","last_became_peered":"2024-09-21T10:08:30.585434+0000","last_unstale":"2024-09-21T10:38:53.321716+0000","last_undegraded":"2024-09-21T10:38:53.321716+0000","last_fullsized":"2024-09-21T10:38:53.321716+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:51:30.174217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":1260,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.081691+0000","last_change":"2024-09-21T09:54:53.628923+0000","last_active":"2024-09-21T10:38:53.081691+0000","last_peered":"2024-09-21T10:38:53.081691+0000","last_clean":"2024-09-21T10:38:53.081691+0000","last_became_active":"2024-09-21T09:54:53.627939+0000","last_became_peered":"2024-09-21T09:54:53.627939+0000","last_unstale":"2024-09-21T10:38:53.081691+0000","last_undegraded":"2024-09-21T10:38:53.081691+0000","last_fullsized":"2024-09-21T10:38:53.081691+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:07:02.820689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1287,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.701674+0000","last_change":"2024-09-21T10:09:06.559132+0000","last_active":"2024-09-21T10:38:53.701674+0000","last_peered":"2024-09-21T10:38:53.701674+0000","last_clean":"2024-09-21T10:38:53.701674+0000","last_became_active":"2024-09-21T10:09:06.558395+0000","last_became_peered":"2024-09-21T10:09:06.558395+0000","last_unstale":"2024-09-21T10:38:53.701674+0000","last_undegraded":"2024-09-21T10:38:53.701674+0000","last_fullsized":"2024-09-21T10:38:53.701674+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:27:00.073554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":1276,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.081928+0000","last_change":"2024-09-21T10:07:56.029192+0000","last_active":"2024-09-21T10:38:53.081928+0000","last_peered":"2024-09-21T10:38:53.081928+0000","last_clean":"2024-09-21T10:38:53.081928+0000","last_became_active":"2024-09-21T10:07:56.028747+0000","last_became_peered":"2024-09-21T10:07:56.028747+0000","last_unstale":"2024-09-21T10:38:53.081928+0000","last_undegraded":"2024-09-21T10:38:53.081928+0000","last_fullsized":"2024-09-21T10:38:53.081928+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:08.918337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1257,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:54.019865+0000","last_change":"2024-09-21T09:54:54.677572+0000","last_active":"2024-09-21T10:38:54.019865+0000","last_peered":"2024-09-21T10:38:54.019865+0000","last_clean":"2024-09-21T10:38:54.019865+0000","last_became_active":"2024-09-21T09:54:54.677253+0000","last_became_peered":"2024-09-21T09:54:54.677253+0000","last_unstale":"2024-09-21T10:38:54.019865+0000","last_undegraded":"2024-09-21T10:38:54.019865+0000","last_fullsized":"2024-09-21T10:38:54.019865+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:31:40.629299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":1275,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.081671+0000","last_change":"2024-09-21T10:07:21.499297+0000","last_active":"2024-09-21T10:38:53.081671+0000","last_peered":"2024-09-21T10:38:53.081671+0000","last_clean":"2024-09-21T10:38:53.081671+0000","last_became_active":"2024-09-21T10:07:21.498961+0000","last_became_peered":"2024-09-21T10:07:21.498961+0000","last_unstale":"2024-09-21T10:38:53.081671+0000","last_undegraded":"2024-09-21T10:38:53.081671+0000","last_fullsized":"2024-09-21T10:38:53.081671+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:07.281950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1281,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114824+0000","last_change":"2024-09-21T10:07:21.497877+0000","last_active":"2024-09-21T10:38:57.114824+0000","last_peered":"2024-09-21T10:38:57.114824+0000","last_clean":"2024-09-21T10:38:57.114824+0000","last_became_active":"2024-09-21T10:07:21.496317+0000","last_became_peered":"2024-09-21T10:07:21.496317+0000","last_unstale":"2024-09-21T10:38:57.114824+0000","last_undegraded":"2024-09-21T10:38:57.114824+0000","last_fullsized":"2024-09-21T10:38:57.114824+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:39:03.848183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1879,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.121039+0000","last_change":"2024-09-21T10:09:06.575204+0000","last_active":"2024-09-21T10:38:57.121039+0000","last_peered":"2024-09-21T10:38:57.121039+0000","last_clean":"2024-09-21T10:38:57.121039+0000","last_became_active":"2024-09-21T10:09:06.571218+0000","last_became_peered":"2024-09-21T10:09:06.571218+0000","last_unstale":"2024-09-21T10:38:57.121039+0000","last_undegraded":"2024-09-21T10:38:57.121039+0000","last_fullsized":"2024-09-21T10:38:57.121039+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:27.754524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1259,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.092165+0000","last_change":"2024-09-21T09:54:53.625016+0000","last_active":"2024-09-21T10:38:57.092165+0000","last_peered":"2024-09-21T10:38:57.092165+0000","last_clean":"2024-09-21T10:38:57.092165+0000","last_became_active":"2024-09-21T09:54:53.624750+0000","last_became_peered":"2024-09-21T09:54:53.624750+0000","last_unstale":"2024-09-21T10:38:57.092165+0000","last_undegraded":"2024-09-21T10:38:57.092165+0000","last_fullsized":"2024-09-21T10:38:57.092165+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:34:26.823544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1910,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115602+0000","last_change":"2024-09-21T10:09:06.558492+0000","last_active":"2024-09-21T10:38:57.115602+0000","last_peered":"2024-09-21T10:38:57.115602+0000","last_clean":"2024-09-21T10:38:57.115602+0000","last_became_active":"2024-09-21T10:09:06.558146+0000","last_became_peered":"2024-09-21T10:09:06.558146+0000","last_unstale":"2024-09-21T10:38:57.115602+0000","last_undegraded":"2024-09-21T10:38:57.115602+0000","last_fullsized":"2024-09-21T10:38:57.115602+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:28:50.638241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1893,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.117388+0000","last_change":"2024-09-21T10:09:06.579005+0000","last_active":"2024-09-21T10:38:57.117388+0000","last_peered":"2024-09-21T10:38:57.117388+0000","last_clean":"2024-09-21T10:38:57.117388+0000","last_became_active":"2024-09-21T10:09:06.578119+0000","last_became_peered":"2024-09-21T10:09:06.578119+0000","last_unstale":"2024-09-21T10:38:57.117388+0000","last_undegraded":"2024-09-21T10:38:57.117388+0000","last_fullsized":"2024-09-21T10:38:57.117388+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:20.615838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1270,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:54.020627+0000","last_change":"2024-09-21T10:09:06.557453+0000","last_active":"2024-09-21T10:38:54.020627+0000","last_peered":"2024-09-21T10:38:54.020627+0000","last_clean":"2024-09-21T10:38:54.020627+0000","last_became_active":"2024-09-21T10:09:06.557124+0000","last_became_peered":"2024-09-21T10:09:06.557124+0000","last_unstale":"2024-09-21T10:38:54.020627+0000","last_undegraded":"2024-09-21T10:38:54.020627+0000","last_fullsized":"2024-09-21T10:38:54.020627+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:03:27.731832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":1276,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.091450+0000","last_change":"2024-09-21T10:08:30.574245+0000","last_active":"2024-09-21T10:38:57.091450+0000","last_peered":"2024-09-21T10:38:57.091450+0000","last_clean":"2024-09-21T10:38:57.091450+0000","last_became_active":"2024-09-21T10:08:30.573909+0000","last_became_peered":"2024-09-21T10:08:30.573909+0000","last_unstale":"2024-09-21T10:38:57.091450+0000","last_undegraded":"2024-09-21T10:38:57.091450+0000","last_fullsized":"2024-09-21T10:38:57.091450+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:18:03.911157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1880,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.120094+0000","last_change":"2024-09-21T10:09:06.578773+0000","last_active":"2024-09-21T10:38:57.120094+0000","last_peered":"2024-09-21T10:38:57.120094+0000","last_clean":"2024-09-21T10:38:57.120094+0000","last_became_active":"2024-09-21T10:09:06.576688+0000","last_became_peered":"2024-09-21T10:09:06.576688+0000","last_unstale":"2024-09-21T10:38:57.120094+0000","last_undegraded":"2024-09-21T10:38:57.120094+0000","last_fullsized":"2024-09-21T10:38:57.120094+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:37.699650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234550,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":140356,"kb_used_data":14968,"kb_used_omap":1,"kb_used_meta":125374,"kb_avail":93604796,"statfs":{"total":95995035648,"available":95851311104,"internally_reserved":0,"allocated":15327232,"data_stored":9832930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":128383435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496187,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":148540,"kb_used_data":14960,"kb_used_omap":1,"kb_used_meta":133566,"kb_avail":93596612,"statfs":{"total":95995035648,"available":95842930688,"internally_reserved":0,"allocated":15319040,"data_stored":9832803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":136772042},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757824,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":107824,"kb_used_data":14372,"kb_used_omap":1,"kb_used_meta":93438,"kb_avail":93637328,"statfs":{"total":95995035648,"available":95884623872,"internally_reserved":0,"allocated":14716928,"data_stored":9238261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95680966},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019460,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":117624,"kb_used_data":14384,"kb_used_omap":1,"kb_used_meta":103230,"kb_avail":93627528,"statfs":{"total":95995035648,"available":95874588672,"internally_reserved":0,"allocated":14729216,"data_stored":9242492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105707974},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369204,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78440,"kb_used_data":14364,"kb_used_omap":69,"kb_used_meta":63994,"kb_avail":93666712,"statfs":{"total":95995035648,"available":95914713088,"internally_reserved":0,"allocated":14708736,"data_stored":9237692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":65529986},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":680,"seq":2920577761659,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":68716,"kb_used_data":14368,"kb_used_omap":76,"kb_used_meta":54259,"kb_avail":93676436,"statfs":{"total":95995035648,"available":95924670464,"internally_reserved":0,"allocated":14712832,"data_stored":9238218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78585,"internal_metadata":55561479},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807957890,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86632,"kb_used_data":14364,"kb_used_omap":66,"kb_used_meta":72189,"kb_avail":93658520,"statfs":{"total":95995035648,"available":95906324480,"internally_reserved":0,"allocated":14708736,"data_stored":9237621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68589,"internal_metadata":73921555},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":666,"seq":2860448219529,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87024,"kb_used_data":14948,"kb_used_omap":76,"kb_used_meta":71987,"kb_avail":93658128,"statfs":{"total":95995035648,"available":95905923072,"internally_reserved":0,"allocated":15306752,"data_stored":9828574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78716,"internal_metadata":73714820},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:39:56.364 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:39:56.575 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:56.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:56 smithi070 ceph-mon[107385]: pgmap v2135: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:56.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:56 smithi070 ceph-mon[112139]: pgmap v2135: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:56 smithi160 ceph-mon[101879]: pgmap v2135: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:57.722 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:57 smithi070 ceph-mon[107385]: from='client.44957 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:39:57.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:57 smithi070 ceph-mon[112139]: from='client.44957 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:39:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:57 smithi160 ceph-mon[101879]: from='client.44957 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:39:58.386 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:39:58.386 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T10:39:58.679 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:58 smithi070 ceph-mon[107385]: pgmap v2136: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:58.680 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:58 smithi070 ceph-mon[112139]: pgmap v2136: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:58 smithi160 ceph-mon[101879]: pgmap v2136: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:39:59.011 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":2136,"stamp":"2024-09-21T10:39:56.649726+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595941,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":279,"num_write_kb":2449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":835156,"kb_used_data":116728,"kb_used_omap":296,"kb_used_meta":718039,"kb_avail":749126060,"statfs":{"total":767960285184,"available":767105085440,"internally_reserved":0,"allocated":119529472,"data_stored":75688591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":303807,"internal_metadata":735272257},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002610"},"pg_stats":[{"pgid":"3.1f","version":"71'4","reported_seq":1904,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.116537+0000","last_change":"2024-09-21T10:08:30.578838+0000","last_active":"2024-09-21T10:38:57.116537+0000","last_peered":"2024-09-21T10:38:57.116537+0000","last_clean":"2024-09-21T10:38:57.116537+0000","last_became_active":"2024-09-21T10:08:30.577773+0000","last_became_peered":"2024-09-21T10:08:30.577773+0000","last_unstale":"2024-09-21T10:38:57.116537+0000","last_undegraded":"2024-09-21T10:38:57.116537+0000","last_fullsized":"2024-09-21T10:38:57.116537+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:46:51.619094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1270,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.701486+0000","last_change":"2024-09-21T10:07:55.999392+0000","last_active":"2024-09-21T10:38:53.701486+0000","last_peered":"2024-09-21T10:38:53.701486+0000","last_clean":"2024-09-21T10:38:53.701486+0000","last_became_active":"2024-09-21T10:07:55.999103+0000","last_became_peered":"2024-09-21T10:07:55.999103+0000","last_unstale":"2024-09-21T10:38:53.701486+0000","last_undegraded":"2024-09-21T10:38:53.701486+0000","last_fullsized":"2024-09-21T10:38:53.701486+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:48.757242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":1891,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.116274+0000","last_change":"2024-09-21T10:09:06.592799+0000","last_active":"2024-09-21T10:38:57.116274+0000","last_peered":"2024-09-21T10:38:57.116274+0000","last_clean":"2024-09-21T10:38:57.116274+0000","last_became_active":"2024-09-21T10:09:06.592468+0000","last_became_peered":"2024-09-21T10:09:06.592468+0000","last_unstale":"2024-09-21T10:38:57.116274+0000","last_undegraded":"2024-09-21T10:38:57.116274+0000","last_fullsized":"2024-09-21T10:38:57.116274+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:45:04.534377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1908,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.119459+0000","last_change":"2024-09-21T10:09:06.584044+0000","last_active":"2024-09-21T10:38:57.119459+0000","last_peered":"2024-09-21T10:38:57.119459+0000","last_clean":"2024-09-21T10:38:57.119459+0000","last_became_active":"2024-09-21T10:09:06.583761+0000","last_became_peered":"2024-09-21T10:09:06.583761+0000","last_unstale":"2024-09-21T10:38:57.119459+0000","last_undegraded":"2024-09-21T10:38:57.119459+0000","last_fullsized":"2024-09-21T10:38:57.119459+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:53.839091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":1258,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.113486+0000","last_change":"2024-09-21T09:54:53.634508+0000","last_active":"2024-09-21T10:38:57.113486+0000","last_peered":"2024-09-21T10:38:57.113486+0000","last_clean":"2024-09-21T10:38:57.113486+0000","last_became_active":"2024-09-21T09:54:53.634254+0000","last_became_peered":"2024-09-21T09:54:53.634254+0000","last_unstale":"2024-09-21T10:38:57.113486+0000","last_undegraded":"2024-09-21T10:38:57.113486+0000","last_fullsized":"2024-09-21T10:38:57.113486+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:18.122107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1285,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.091098+0000","last_change":"2024-09-21T10:09:06.574933+0000","last_active":"2024-09-21T10:38:57.091098+0000","last_peered":"2024-09-21T10:38:57.091098+0000","last_clean":"2024-09-21T10:38:57.091098+0000","last_became_active":"2024-09-21T10:09:06.574467+0000","last_became_peered":"2024-09-21T10:09:06.574467+0000","last_unstale":"2024-09-21T10:38:57.091098+0000","last_undegraded":"2024-09-21T10:38:57.091098+0000","last_fullsized":"2024-09-21T10:38:57.091098+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:06.142269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":1273,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.113581+0000","last_change":"2024-09-21T10:07:55.991189+0000","last_active":"2024-09-21T10:38:57.113581+0000","last_peered":"2024-09-21T10:38:57.113581+0000","last_clean":"2024-09-21T10:38:57.113581+0000","last_became_active":"2024-09-21T10:07:55.990784+0000","last_became_peered":"2024-09-21T10:07:55.990784+0000","last_unstale":"2024-09-21T10:38:57.113581+0000","last_undegraded":"2024-09-21T10:38:57.113581+0000","last_fullsized":"2024-09-21T10:38:57.113581+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:00.580471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1285,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.094019+0000","last_change":"2024-09-21T10:09:06.560175+0000","last_active":"2024-09-21T10:38:57.094019+0000","last_peered":"2024-09-21T10:38:57.094019+0000","last_clean":"2024-09-21T10:38:57.094019+0000","last_became_active":"2024-09-21T10:09:06.558584+0000","last_became_peered":"2024-09-21T10:09:06.558584+0000","last_unstale":"2024-09-21T10:38:57.094019+0000","last_undegraded":"2024-09-21T10:38:57.094019+0000","last_fullsized":"2024-09-21T10:38:57.094019+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:02.918339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":1887,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115016+0000","last_change":"2024-09-21T10:07:21.532238+0000","last_active":"2024-09-21T10:38:57.115016+0000","last_peered":"2024-09-21T10:38:57.115016+0000","last_clean":"2024-09-21T10:38:57.115016+0000","last_became_active":"2024-09-21T10:07:21.522339+0000","last_became_peered":"2024-09-21T10:07:21.522339+0000","last_unstale":"2024-09-21T10:38:57.115016+0000","last_undegraded":"2024-09-21T10:38:57.115016+0000","last_fullsized":"2024-09-21T10:38:57.115016+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:34:09.116737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1908,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114132+0000","last_change":"2024-09-21T10:09:06.542669+0000","last_active":"2024-09-21T10:38:57.114132+0000","last_peered":"2024-09-21T10:38:57.114132+0000","last_clean":"2024-09-21T10:38:57.114132+0000","last_became_active":"2024-09-21T10:09:06.542384+0000","last_became_peered":"2024-09-21T10:09:06.542384+0000","last_unstale":"2024-09-21T10:38:57.114132+0000","last_undegraded":"2024-09-21T10:38:57.114132+0000","last_fullsized":"2024-09-21T10:38:57.114132+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:53.692798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":1287,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.091417+0000","last_change":"2024-09-21T10:08:30.598724+0000","last_active":"2024-09-21T10:38:57.091417+0000","last_peered":"2024-09-21T10:38:57.091417+0000","last_clean":"2024-09-21T10:38:57.091417+0000","last_became_active":"2024-09-21T10:08:30.598229+0000","last_became_peered":"2024-09-21T10:08:30.598229+0000","last_unstale":"2024-09-21T10:38:57.091417+0000","last_undegraded":"2024-09-21T10:38:57.091417+0000","last_fullsized":"2024-09-21T10:38:57.091417+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:57:57.056712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1902,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.118795+0000","last_change":"2024-09-21T10:09:06.590661+0000","last_active":"2024-09-21T10:38:57.118795+0000","last_peered":"2024-09-21T10:38:57.118795+0000","last_clean":"2024-09-21T10:38:57.118795+0000","last_became_active":"2024-09-21T10:09:06.588941+0000","last_became_peered":"2024-09-21T10:09:06.588941+0000","last_unstale":"2024-09-21T10:38:57.118795+0000","last_undegraded":"2024-09-21T10:38:57.118795+0000","last_fullsized":"2024-09-21T10:38:57.118795+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:18:09.719872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1881,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.118623+0000","last_change":"2024-09-21T10:09:06.530394+0000","last_active":"2024-09-21T10:38:57.118623+0000","last_peered":"2024-09-21T10:38:57.118623+0000","last_clean":"2024-09-21T10:38:57.118623+0000","last_became_active":"2024-09-21T10:09:06.529307+0000","last_became_peered":"2024-09-21T10:09:06.529307+0000","last_unstale":"2024-09-21T10:38:57.118623+0000","last_undegraded":"2024-09-21T10:38:57.118623+0000","last_fullsized":"2024-09-21T10:38:57.118623+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:54.287397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1899,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115865+0000","last_change":"2024-09-21T10:09:06.558433+0000","last_active":"2024-09-21T10:38:57.115865+0000","last_peered":"2024-09-21T10:38:57.115865+0000","last_clean":"2024-09-21T10:38:57.115865+0000","last_became_active":"2024-09-21T10:09:06.557907+0000","last_became_peered":"2024-09-21T10:09:06.557907+0000","last_unstale":"2024-09-21T10:38:57.115865+0000","last_undegraded":"2024-09-21T10:38:57.115865+0000","last_fullsized":"2024-09-21T10:38:57.115865+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:05.109492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1895,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114707+0000","last_change":"2024-09-21T10:09:06.552324+0000","last_active":"2024-09-21T10:38:57.114707+0000","last_peered":"2024-09-21T10:38:57.114707+0000","last_clean":"2024-09-21T10:38:57.114707+0000","last_became_active":"2024-09-21T10:09:06.540545+0000","last_became_peered":"2024-09-21T10:09:06.540545+0000","last_unstale":"2024-09-21T10:38:57.114707+0000","last_undegraded":"2024-09-21T10:38:57.114707+0000","last_fullsized":"2024-09-21T10:38:57.114707+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:08.506149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1274,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.322114+0000","last_change":"2024-09-21T10:07:56.003005+0000","last_active":"2024-09-21T10:38:53.322114+0000","last_peered":"2024-09-21T10:38:53.322114+0000","last_clean":"2024-09-21T10:38:53.322114+0000","last_became_active":"2024-09-21T10:07:56.001550+0000","last_became_peered":"2024-09-21T10:07:56.001550+0000","last_unstale":"2024-09-21T10:38:53.322114+0000","last_undegraded":"2024-09-21T10:38:53.322114+0000","last_fullsized":"2024-09-21T10:38:53.322114+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:54.553261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1888,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114926+0000","last_change":"2024-09-21T10:09:06.559017+0000","last_active":"2024-09-21T10:38:57.114926+0000","last_peered":"2024-09-21T10:38:57.114926+0000","last_clean":"2024-09-21T10:38:57.114926+0000","last_became_active":"2024-09-21T10:09:06.555256+0000","last_became_peered":"2024-09-21T10:09:06.555256+0000","last_unstale":"2024-09-21T10:38:57.114926+0000","last_undegraded":"2024-09-21T10:38:57.114926+0000","last_fullsized":"2024-09-21T10:38:57.114926+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:33:45.145138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1893,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.116730+0000","last_change":"2024-09-21T10:09:06.562504+0000","last_active":"2024-09-21T10:38:57.116730+0000","last_peered":"2024-09-21T10:38:57.116730+0000","last_clean":"2024-09-21T10:38:57.116730+0000","last_became_active":"2024-09-21T10:09:06.561747+0000","last_became_peered":"2024-09-21T10:09:06.561747+0000","last_unstale":"2024-09-21T10:38:57.116730+0000","last_undegraded":"2024-09-21T10:38:57.116730+0000","last_fullsized":"2024-09-21T10:38:57.116730+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:03:58.012406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1879,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.120517+0000","last_change":"2024-09-21T10:09:06.566050+0000","last_active":"2024-09-21T10:38:57.120517+0000","last_peered":"2024-09-21T10:38:57.120517+0000","last_clean":"2024-09-21T10:38:57.120517+0000","last_became_active":"2024-09-21T10:09:06.565791+0000","last_became_peered":"2024-09-21T10:09:06.565791+0000","last_unstale":"2024-09-21T10:38:57.120517+0000","last_undegraded":"2024-09-21T10:38:57.120517+0000","last_fullsized":"2024-09-21T10:38:57.120517+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:59:20.542888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1271,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:54.020837+0000","last_change":"2024-09-21T10:07:56.023722+0000","last_active":"2024-09-21T10:38:54.020837+0000","last_peered":"2024-09-21T10:38:54.020837+0000","last_clean":"2024-09-21T10:38:54.020837+0000","last_became_active":"2024-09-21T10:07:56.023085+0000","last_became_peered":"2024-09-21T10:07:56.023085+0000","last_unstale":"2024-09-21T10:38:54.020837+0000","last_undegraded":"2024-09-21T10:38:54.020837+0000","last_fullsized":"2024-09-21T10:38:54.020837+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:34:07.168469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":1276,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.081521+0000","last_change":"2024-09-21T10:09:06.572971+0000","last_active":"2024-09-21T10:38:53.081521+0000","last_peered":"2024-09-21T10:38:53.081521+0000","last_clean":"2024-09-21T10:38:53.081521+0000","last_became_active":"2024-09-21T10:09:06.571680+0000","last_became_peered":"2024-09-21T10:09:06.571680+0000","last_unstale":"2024-09-21T10:38:53.081521+0000","last_undegraded":"2024-09-21T10:38:53.081521+0000","last_fullsized":"2024-09-21T10:38:53.081521+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:21:45.051132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1880,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.117421+0000","last_change":"2024-09-21T10:08:30.589733+0000","last_active":"2024-09-21T10:38:57.117421+0000","last_peered":"2024-09-21T10:38:57.117421+0000","last_clean":"2024-09-21T10:38:57.117421+0000","last_became_active":"2024-09-21T10:08:30.589456+0000","last_became_peered":"2024-09-21T10:08:30.589456+0000","last_unstale":"2024-09-21T10:38:57.117421+0000","last_undegraded":"2024-09-21T10:38:57.117421+0000","last_fullsized":"2024-09-21T10:38:57.117421+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:09.654350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":1280,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.092674+0000","last_change":"2024-09-21T09:54:53.624887+0000","last_active":"2024-09-21T10:38:57.092674+0000","last_peered":"2024-09-21T10:38:57.092674+0000","last_clean":"2024-09-21T10:38:57.092674+0000","last_became_active":"2024-09-21T09:54:53.624525+0000","last_became_peered":"2024-09-21T09:54:53.624525+0000","last_unstale":"2024-09-21T10:38:57.092674+0000","last_undegraded":"2024-09-21T10:38:57.092674+0000","last_fullsized":"2024-09-21T10:38:57.092674+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:05:59.605316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1271,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.094087+0000","last_change":"2024-09-21T10:07:56.001756+0000","last_active":"2024-09-21T10:38:57.094087+0000","last_peered":"2024-09-21T10:38:57.094087+0000","last_clean":"2024-09-21T10:38:57.094087+0000","last_became_active":"2024-09-21T10:07:56.000789+0000","last_became_peered":"2024-09-21T10:07:56.000789+0000","last_unstale":"2024-09-21T10:38:57.094087+0000","last_undegraded":"2024-09-21T10:38:57.094087+0000","last_fullsized":"2024-09-21T10:38:57.094087+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:56:04.302902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1258,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.701544+0000","last_change":"2024-09-21T09:54:53.636902+0000","last_active":"2024-09-21T10:38:53.701544+0000","last_peered":"2024-09-21T10:38:53.701544+0000","last_clean":"2024-09-21T10:38:53.701544+0000","last_became_active":"2024-09-21T09:54:53.636616+0000","last_became_peered":"2024-09-21T09:54:53.636616+0000","last_unstale":"2024-09-21T10:38:53.701544+0000","last_undegraded":"2024-09-21T10:38:53.701544+0000","last_fullsized":"2024-09-21T10:38:53.701544+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:46:47.694495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1287,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.116875+0000","last_change":"2024-09-21T10:08:30.587337+0000","last_active":"2024-09-21T10:38:57.116875+0000","last_peered":"2024-09-21T10:38:57.116875+0000","last_clean":"2024-09-21T10:38:57.116875+0000","last_became_active":"2024-09-21T10:08:30.585404+0000","last_became_peered":"2024-09-21T10:08:30.585404+0000","last_unstale":"2024-09-21T10:38:57.116875+0000","last_undegraded":"2024-09-21T10:38:57.116875+0000","last_fullsized":"2024-09-21T10:38:57.116875+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:24:41.559947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":1901,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115763+0000","last_change":"2024-09-21T10:07:56.023462+0000","last_active":"2024-09-21T10:38:57.115763+0000","last_peered":"2024-09-21T10:38:57.115763+0000","last_clean":"2024-09-21T10:38:57.115763+0000","last_became_active":"2024-09-21T10:07:56.021859+0000","last_became_peered":"2024-09-21T10:07:56.021859+0000","last_unstale":"2024-09-21T10:38:57.115763+0000","last_undegraded":"2024-09-21T10:38:57.115763+0000","last_fullsized":"2024-09-21T10:38:57.115763+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:42:03.134729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1875,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114840+0000","last_change":"2024-09-21T10:09:06.580853+0000","last_active":"2024-09-21T10:38:57.114840+0000","last_peered":"2024-09-21T10:38:57.114840+0000","last_clean":"2024-09-21T10:38:57.114840+0000","last_became_active":"2024-09-21T10:09:06.568439+0000","last_became_peered":"2024-09-21T10:09:06.568439+0000","last_unstale":"2024-09-21T10:38:57.114840+0000","last_undegraded":"2024-09-21T10:38:57.114840+0000","last_fullsized":"2024-09-21T10:38:57.114840+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:17.633827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1881,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115718+0000","last_change":"2024-09-21T10:07:56.006236+0000","last_active":"2024-09-21T10:38:57.115718+0000","last_peered":"2024-09-21T10:38:57.115718+0000","last_clean":"2024-09-21T10:38:57.115718+0000","last_became_active":"2024-09-21T10:07:56.004439+0000","last_became_peered":"2024-09-21T10:07:56.004439+0000","last_unstale":"2024-09-21T10:38:57.115718+0000","last_undegraded":"2024-09-21T10:38:57.115718+0000","last_fullsized":"2024-09-21T10:38:57.115718+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:39.422467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":1876,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.121117+0000","last_change":"2024-09-21T10:09:06.578081+0000","last_active":"2024-09-21T10:38:57.121117+0000","last_peered":"2024-09-21T10:38:57.121117+0000","last_clean":"2024-09-21T10:38:57.121117+0000","last_became_active":"2024-09-21T10:09:06.575129+0000","last_became_peered":"2024-09-21T10:09:06.575129+0000","last_unstale":"2024-09-21T10:38:57.121117+0000","last_undegraded":"2024-09-21T10:38:57.121117+0000","last_fullsized":"2024-09-21T10:38:57.121117+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:47:26.651862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1887,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115125+0000","last_change":"2024-09-21T10:07:21.531644+0000","last_active":"2024-09-21T10:38:57.115125+0000","last_peered":"2024-09-21T10:38:57.115125+0000","last_clean":"2024-09-21T10:38:57.115125+0000","last_became_active":"2024-09-21T10:07:21.520764+0000","last_became_peered":"2024-09-21T10:07:21.520764+0000","last_unstale":"2024-09-21T10:38:57.115125+0000","last_undegraded":"2024-09-21T10:38:57.115125+0000","last_fullsized":"2024-09-21T10:38:57.115125+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:11:32.147647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1289,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.094214+0000","last_change":"2024-09-21T10:07:21.499183+0000","last_active":"2024-09-21T10:38:57.094214+0000","last_peered":"2024-09-21T10:38:57.094214+0000","last_clean":"2024-09-21T10:38:57.094214+0000","last_became_active":"2024-09-21T10:07:21.498610+0000","last_became_peered":"2024-09-21T10:07:21.498610+0000","last_unstale":"2024-09-21T10:38:57.094214+0000","last_undegraded":"2024-09-21T10:38:57.094214+0000","last_fullsized":"2024-09-21T10:38:57.094214+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:20:17.212088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1910,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.118834+0000","last_change":"2024-09-21T10:08:30.588282+0000","last_active":"2024-09-21T10:38:57.118834+0000","last_peered":"2024-09-21T10:38:57.118834+0000","last_clean":"2024-09-21T10:38:57.118834+0000","last_became_active":"2024-09-21T10:08:30.587736+0000","last_became_peered":"2024-09-21T10:08:30.587736+0000","last_unstale":"2024-09-21T10:38:57.118834+0000","last_undegraded":"2024-09-21T10:38:57.118834+0000","last_fullsized":"2024-09-21T10:38:57.118834+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:22:18.890670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1872,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.117294+0000","last_change":"2024-09-21T10:07:56.067772+0000","last_active":"2024-09-21T10:38:57.117294+0000","last_peered":"2024-09-21T10:38:57.117294+0000","last_clean":"2024-09-21T10:38:57.117294+0000","last_became_active":"2024-09-21T10:07:56.052602+0000","last_became_peered":"2024-09-21T10:07:56.052602+0000","last_unstale":"2024-09-21T10:38:57.117294+0000","last_undegraded":"2024-09-21T10:38:57.117294+0000","last_fullsized":"2024-09-21T10:38:57.117294+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:07:59.899214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1926,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.118605+0000","last_change":"2024-09-21T10:07:56.017999+0000","last_active":"2024-09-21T10:38:57.118605+0000","last_peered":"2024-09-21T10:38:57.118605+0000","last_clean":"2024-09-21T10:38:57.118605+0000","last_became_active":"2024-09-21T10:07:56.017099+0000","last_became_peered":"2024-09-21T10:07:56.017099+0000","last_unstale":"2024-09-21T10:38:57.118605+0000","last_undegraded":"2024-09-21T10:38:57.118605+0000","last_fullsized":"2024-09-21T10:38:57.118605+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:58.288984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1289,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:55.981301+0000","last_change":"2024-09-21T10:09:06.558796+0000","last_active":"2024-09-21T10:38:55.981301+0000","last_peered":"2024-09-21T10:38:55.981301+0000","last_clean":"2024-09-21T10:38:55.981301+0000","last_became_active":"2024-09-21T10:09:06.557834+0000","last_became_peered":"2024-09-21T10:09:06.557834+0000","last_unstale":"2024-09-21T10:38:55.981301+0000","last_undegraded":"2024-09-21T10:38:55.981301+0000","last_fullsized":"2024-09-21T10:38:55.981301+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:02:46.366855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1920,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.118992+0000","last_change":"2024-09-21T10:09:06.534003+0000","last_active":"2024-09-21T10:38:57.118992+0000","last_peered":"2024-09-21T10:38:57.118992+0000","last_clean":"2024-09-21T10:38:57.118992+0000","last_became_active":"2024-09-21T10:09:06.533714+0000","last_became_peered":"2024-09-21T10:09:06.533714+0000","last_unstale":"2024-09-21T10:38:57.118992+0000","last_undegraded":"2024-09-21T10:38:57.118992+0000","last_fullsized":"2024-09-21T10:38:57.118992+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:15.622055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1274,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.702267+0000","last_change":"2024-09-21T10:07:21.524082+0000","last_active":"2024-09-21T10:38:53.702267+0000","last_peered":"2024-09-21T10:38:53.702267+0000","last_clean":"2024-09-21T10:38:53.702267+0000","last_became_active":"2024-09-21T10:07:21.523779+0000","last_became_peered":"2024-09-21T10:07:21.523779+0000","last_unstale":"2024-09-21T10:38:53.702267+0000","last_undegraded":"2024-09-21T10:38:53.702267+0000","last_fullsized":"2024-09-21T10:38:53.702267+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:13.813832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":1289,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114172+0000","last_change":"2024-09-21T10:07:56.024239+0000","last_active":"2024-09-21T10:38:57.114172+0000","last_peered":"2024-09-21T10:38:57.114172+0000","last_clean":"2024-09-21T10:38:57.114172+0000","last_became_active":"2024-09-21T10:07:56.023914+0000","last_became_peered":"2024-09-21T10:07:56.023914+0000","last_unstale":"2024-09-21T10:38:57.114172+0000","last_undegraded":"2024-09-21T10:38:57.114172+0000","last_fullsized":"2024-09-21T10:38:57.114172+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:23.361736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084477399999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1909,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115265+0000","last_change":"2024-09-21T10:07:56.056338+0000","last_active":"2024-09-21T10:38:57.115265+0000","last_peered":"2024-09-21T10:38:57.115265+0000","last_clean":"2024-09-21T10:38:57.115265+0000","last_became_active":"2024-09-21T10:07:56.052888+0000","last_became_peered":"2024-09-21T10:07:56.052888+0000","last_unstale":"2024-09-21T10:38:57.115265+0000","last_undegraded":"2024-09-21T10:38:57.115265+0000","last_fullsized":"2024-09-21T10:38:57.115265+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:34:48.662436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1883,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114052+0000","last_change":"2024-09-21T10:08:30.599270+0000","last_active":"2024-09-21T10:38:57.114052+0000","last_peered":"2024-09-21T10:38:57.114052+0000","last_clean":"2024-09-21T10:38:57.114052+0000","last_became_active":"2024-09-21T10:08:30.598873+0000","last_became_peered":"2024-09-21T10:08:30.598873+0000","last_unstale":"2024-09-21T10:38:57.114052+0000","last_undegraded":"2024-09-21T10:38:57.114052+0000","last_fullsized":"2024-09-21T10:38:57.114052+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:15:19.305854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1917,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.116069+0000","last_change":"2024-09-21T10:09:06.542888+0000","last_active":"2024-09-21T10:38:57.116069+0000","last_peered":"2024-09-21T10:38:57.116069+0000","last_clean":"2024-09-21T10:38:57.116069+0000","last_became_active":"2024-09-21T10:09:06.542593+0000","last_became_peered":"2024-09-21T10:09:06.542593+0000","last_unstale":"2024-09-21T10:38:57.116069+0000","last_undegraded":"2024-09-21T10:38:57.116069+0000","last_fullsized":"2024-09-21T10:38:57.116069+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:18:24.657282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1909,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115959+0000","last_change":"2024-09-21T10:08:30.589052+0000","last_active":"2024-09-21T10:38:57.115959+0000","last_peered":"2024-09-21T10:38:57.115959+0000","last_clean":"2024-09-21T10:38:57.115959+0000","last_became_active":"2024-09-21T10:08:30.588532+0000","last_became_peered":"2024-09-21T10:08:30.588532+0000","last_unstale":"2024-09-21T10:38:57.115959+0000","last_undegraded":"2024-09-21T10:38:57.115959+0000","last_fullsized":"2024-09-21T10:38:57.115959+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:55:48.460046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1896,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.116246+0000","last_change":"2024-09-21T10:09:06.560130+0000","last_active":"2024-09-21T10:38:57.116246+0000","last_peered":"2024-09-21T10:38:57.116246+0000","last_clean":"2024-09-21T10:38:57.116246+0000","last_became_active":"2024-09-21T10:09:06.559827+0000","last_became_peered":"2024-09-21T10:09:06.559827+0000","last_unstale":"2024-09-21T10:38:57.116246+0000","last_undegraded":"2024-09-21T10:38:57.116246+0000","last_fullsized":"2024-09-21T10:38:57.116246+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:05:17.166734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1304,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.081095+0000","last_change":"2024-09-21T10:07:55.995501+0000","last_active":"2024-09-21T10:38:53.081095+0000","last_peered":"2024-09-21T10:38:53.081095+0000","last_clean":"2024-09-21T10:38:53.081095+0000","last_became_active":"2024-09-21T10:07:55.995188+0000","last_became_peered":"2024-09-21T10:07:55.995188+0000","last_unstale":"2024-09-21T10:38:53.081095+0000","last_undegraded":"2024-09-21T10:38:53.081095+0000","last_fullsized":"2024-09-21T10:38:53.081095+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:49:30.127774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1259,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:56.085563+0000","last_change":"2024-09-21T09:54:53.722213+0000","last_active":"2024-09-21T10:38:56.085563+0000","last_peered":"2024-09-21T10:38:56.085563+0000","last_clean":"2024-09-21T10:38:56.085563+0000","last_became_active":"2024-09-21T09:54:53.721437+0000","last_became_peered":"2024-09-21T09:54:53.721437+0000","last_unstale":"2024-09-21T10:38:56.085563+0000","last_undegraded":"2024-09-21T10:38:56.085563+0000","last_fullsized":"2024-09-21T10:38:56.085563+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:01:41.456731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1877,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115726+0000","last_change":"2024-09-21T10:09:06.560589+0000","last_active":"2024-09-21T10:38:57.115726+0000","last_peered":"2024-09-21T10:38:57.115726+0000","last_clean":"2024-09-21T10:38:57.115726+0000","last_became_active":"2024-09-21T10:09:06.557552+0000","last_became_peered":"2024-09-21T10:09:06.557552+0000","last_unstale":"2024-09-21T10:38:57.115726+0000","last_undegraded":"2024-09-21T10:38:57.115726+0000","last_fullsized":"2024-09-21T10:38:57.115726+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:09:24.866759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":1288,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115185+0000","last_change":"2024-09-21T10:08:30.576357+0000","last_active":"2024-09-21T10:38:57.115185+0000","last_peered":"2024-09-21T10:38:57.115185+0000","last_clean":"2024-09-21T10:38:57.115185+0000","last_became_active":"2024-09-21T10:08:30.576033+0000","last_became_peered":"2024-09-21T10:08:30.576033+0000","last_unstale":"2024-09-21T10:38:57.115185+0000","last_undegraded":"2024-09-21T10:38:57.115185+0000","last_fullsized":"2024-09-21T10:38:57.115185+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:21:45.522197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":1606,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.091312+0000","last_change":"2024-09-21T10:07:21.499838+0000","last_active":"2024-09-21T10:38:57.091312+0000","last_peered":"2024-09-21T10:38:57.091312+0000","last_clean":"2024-09-21T10:38:57.091312+0000","last_became_active":"2024-09-21T10:07:21.499449+0000","last_became_peered":"2024-09-21T10:07:21.499449+0000","last_unstale":"2024-09-21T10:38:57.091312+0000","last_undegraded":"2024-09-21T10:38:57.091312+0000","last_fullsized":"2024-09-21T10:38:57.091312+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:59:19.260791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1882,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115215+0000","last_change":"2024-09-21T10:09:06.538940+0000","last_active":"2024-09-21T10:38:57.115215+0000","last_peered":"2024-09-21T10:38:57.115215+0000","last_clean":"2024-09-21T10:38:57.115215+0000","last_became_active":"2024-09-21T10:09:06.538668+0000","last_became_peered":"2024-09-21T10:09:06.538668+0000","last_unstale":"2024-09-21T10:38:57.115215+0000","last_undegraded":"2024-09-21T10:38:57.115215+0000","last_fullsized":"2024-09-21T10:38:57.115215+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:42:44.299488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1883,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114655+0000","last_change":"2024-09-21T10:09:06.582426+0000","last_active":"2024-09-21T10:38:57.114655+0000","last_peered":"2024-09-21T10:38:57.114655+0000","last_clean":"2024-09-21T10:38:57.114655+0000","last_became_active":"2024-09-21T10:09:06.580117+0000","last_became_peered":"2024-09-21T10:09:06.580117+0000","last_unstale":"2024-09-21T10:38:57.114655+0000","last_undegraded":"2024-09-21T10:38:57.114655+0000","last_fullsized":"2024-09-21T10:38:57.114655+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:56:45.278650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1272,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.094593+0000","last_change":"2024-09-21T10:07:55.998799+0000","last_active":"2024-09-21T10:38:57.094593+0000","last_peered":"2024-09-21T10:38:57.094593+0000","last_clean":"2024-09-21T10:38:57.094593+0000","last_became_active":"2024-09-21T10:07:55.998510+0000","last_became_peered":"2024-09-21T10:07:55.998510+0000","last_unstale":"2024-09-21T10:38:57.094593+0000","last_undegraded":"2024-09-21T10:38:57.094593+0000","last_fullsized":"2024-09-21T10:38:57.094593+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:10.474008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1273,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.701779+0000","last_change":"2024-09-21T10:08:30.602686+0000","last_active":"2024-09-21T10:38:53.701779+0000","last_peered":"2024-09-21T10:38:53.701779+0000","last_clean":"2024-09-21T10:38:53.701779+0000","last_became_active":"2024-09-21T10:08:30.602363+0000","last_became_peered":"2024-09-21T10:08:30.602363+0000","last_unstale":"2024-09-21T10:38:53.701779+0000","last_undegraded":"2024-09-21T10:38:53.701779+0000","last_fullsized":"2024-09-21T10:38:53.701779+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:10.278387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1287,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.094381+0000","last_change":"2024-09-21T10:08:30.601654+0000","last_active":"2024-09-21T10:38:57.094381+0000","last_peered":"2024-09-21T10:38:57.094381+0000","last_clean":"2024-09-21T10:38:57.094381+0000","last_became_active":"2024-09-21T10:08:30.601354+0000","last_became_peered":"2024-09-21T10:08:30.601354+0000","last_unstale":"2024-09-21T10:38:57.094381+0000","last_undegraded":"2024-09-21T10:38:57.094381+0000","last_fullsized":"2024-09-21T10:38:57.094381+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:42:25.902106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1274,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.321824+0000","last_change":"2024-09-21T10:07:56.020791+0000","last_active":"2024-09-21T10:38:53.321824+0000","last_peered":"2024-09-21T10:38:53.321824+0000","last_clean":"2024-09-21T10:38:53.321824+0000","last_became_active":"2024-09-21T10:07:56.020205+0000","last_became_peered":"2024-09-21T10:07:56.020205+0000","last_unstale":"2024-09-21T10:38:53.321824+0000","last_undegraded":"2024-09-21T10:38:53.321824+0000","last_fullsized":"2024-09-21T10:38:53.321824+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:09.813166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1272,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.092576+0000","last_change":"2024-09-21T10:09:06.533271+0000","last_active":"2024-09-21T10:38:57.092576+0000","last_peered":"2024-09-21T10:38:57.092576+0000","last_clean":"2024-09-21T10:38:57.092576+0000","last_became_active":"2024-09-21T10:09:06.532041+0000","last_became_peered":"2024-09-21T10:09:06.532041+0000","last_unstale":"2024-09-21T10:38:57.092576+0000","last_undegraded":"2024-09-21T10:38:57.092576+0000","last_fullsized":"2024-09-21T10:38:57.092576+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:04.236375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":1898,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.119850+0000","last_change":"2024-09-21T10:09:06.543422+0000","last_active":"2024-09-21T10:38:57.119850+0000","last_peered":"2024-09-21T10:38:57.119850+0000","last_clean":"2024-09-21T10:38:57.119850+0000","last_became_active":"2024-09-21T10:09:06.539647+0000","last_became_peered":"2024-09-21T10:09:06.539647+0000","last_unstale":"2024-09-21T10:38:57.119850+0000","last_undegraded":"2024-09-21T10:38:57.119850+0000","last_fullsized":"2024-09-21T10:38:57.119850+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:04.990085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1897,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114294+0000","last_change":"2024-09-21T10:07:56.018560+0000","last_active":"2024-09-21T10:38:57.114294+0000","last_peered":"2024-09-21T10:38:57.114294+0000","last_clean":"2024-09-21T10:38:57.114294+0000","last_became_active":"2024-09-21T10:07:56.018261+0000","last_became_peered":"2024-09-21T10:07:56.018261+0000","last_unstale":"2024-09-21T10:38:57.114294+0000","last_undegraded":"2024-09-21T10:38:57.114294+0000","last_fullsized":"2024-09-21T10:38:57.114294+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:04:54.395456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1273,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.113708+0000","last_change":"2024-09-21T10:09:06.531935+0000","last_active":"2024-09-21T10:38:57.113708+0000","last_peered":"2024-09-21T10:38:57.113708+0000","last_clean":"2024-09-21T10:38:57.113708+0000","last_became_active":"2024-09-21T10:09:06.531610+0000","last_became_peered":"2024-09-21T10:09:06.531610+0000","last_unstale":"2024-09-21T10:38:57.113708+0000","last_undegraded":"2024-09-21T10:38:57.113708+0000","last_fullsized":"2024-09-21T10:38:57.113708+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:33.874405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1282,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.321716+0000","last_change":"2024-09-21T10:08:30.586447+0000","last_active":"2024-09-21T10:38:53.321716+0000","last_peered":"2024-09-21T10:38:53.321716+0000","last_clean":"2024-09-21T10:38:53.321716+0000","last_became_active":"2024-09-21T10:08:30.585434+0000","last_became_peered":"2024-09-21T10:08:30.585434+0000","last_unstale":"2024-09-21T10:38:53.321716+0000","last_undegraded":"2024-09-21T10:38:53.321716+0000","last_fullsized":"2024-09-21T10:38:53.321716+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:51:30.174217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":1261,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.094221+0000","last_change":"2024-09-21T09:54:53.628923+0000","last_active":"2024-09-21T10:38:57.094221+0000","last_peered":"2024-09-21T10:38:57.094221+0000","last_clean":"2024-09-21T10:38:57.094221+0000","last_became_active":"2024-09-21T09:54:53.627939+0000","last_became_peered":"2024-09-21T09:54:53.627939+0000","last_unstale":"2024-09-21T10:38:57.094221+0000","last_undegraded":"2024-09-21T10:38:57.094221+0000","last_fullsized":"2024-09-21T10:38:57.094221+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:07:02.820689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1287,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.701674+0000","last_change":"2024-09-21T10:09:06.559132+0000","last_active":"2024-09-21T10:38:53.701674+0000","last_peered":"2024-09-21T10:38:53.701674+0000","last_clean":"2024-09-21T10:38:53.701674+0000","last_became_active":"2024-09-21T10:09:06.558395+0000","last_became_peered":"2024-09-21T10:09:06.558395+0000","last_unstale":"2024-09-21T10:38:53.701674+0000","last_undegraded":"2024-09-21T10:38:53.701674+0000","last_fullsized":"2024-09-21T10:38:53.701674+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:27:00.073554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":1277,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.093638+0000","last_change":"2024-09-21T10:07:56.029192+0000","last_active":"2024-09-21T10:38:57.093638+0000","last_peered":"2024-09-21T10:38:57.093638+0000","last_clean":"2024-09-21T10:38:57.093638+0000","last_became_active":"2024-09-21T10:07:56.028747+0000","last_became_peered":"2024-09-21T10:07:56.028747+0000","last_unstale":"2024-09-21T10:38:57.093638+0000","last_undegraded":"2024-09-21T10:38:57.093638+0000","last_fullsized":"2024-09-21T10:38:57.093638+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:08.918337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1257,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:54.019865+0000","last_change":"2024-09-21T09:54:54.677572+0000","last_active":"2024-09-21T10:38:54.019865+0000","last_peered":"2024-09-21T10:38:54.019865+0000","last_clean":"2024-09-21T10:38:54.019865+0000","last_became_active":"2024-09-21T09:54:54.677253+0000","last_became_peered":"2024-09-21T09:54:54.677253+0000","last_unstale":"2024-09-21T10:38:54.019865+0000","last_undegraded":"2024-09-21T10:38:54.019865+0000","last_fullsized":"2024-09-21T10:38:54.019865+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:31:40.629299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":1275,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:53.081671+0000","last_change":"2024-09-21T10:07:21.499297+0000","last_active":"2024-09-21T10:38:53.081671+0000","last_peered":"2024-09-21T10:38:53.081671+0000","last_clean":"2024-09-21T10:38:53.081671+0000","last_became_active":"2024-09-21T10:07:21.498961+0000","last_became_peered":"2024-09-21T10:07:21.498961+0000","last_unstale":"2024-09-21T10:38:53.081671+0000","last_undegraded":"2024-09-21T10:38:53.081671+0000","last_fullsized":"2024-09-21T10:38:53.081671+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:07.281950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1281,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.114824+0000","last_change":"2024-09-21T10:07:21.497877+0000","last_active":"2024-09-21T10:38:57.114824+0000","last_peered":"2024-09-21T10:38:57.114824+0000","last_clean":"2024-09-21T10:38:57.114824+0000","last_became_active":"2024-09-21T10:07:21.496317+0000","last_became_peered":"2024-09-21T10:07:21.496317+0000","last_unstale":"2024-09-21T10:38:57.114824+0000","last_undegraded":"2024-09-21T10:38:57.114824+0000","last_fullsized":"2024-09-21T10:38:57.114824+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:39:03.848183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1879,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.121039+0000","last_change":"2024-09-21T10:09:06.575204+0000","last_active":"2024-09-21T10:38:57.121039+0000","last_peered":"2024-09-21T10:38:57.121039+0000","last_clean":"2024-09-21T10:38:57.121039+0000","last_became_active":"2024-09-21T10:09:06.571218+0000","last_became_peered":"2024-09-21T10:09:06.571218+0000","last_unstale":"2024-09-21T10:38:57.121039+0000","last_undegraded":"2024-09-21T10:38:57.121039+0000","last_fullsized":"2024-09-21T10:38:57.121039+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:27.754524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1259,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.092165+0000","last_change":"2024-09-21T09:54:53.625016+0000","last_active":"2024-09-21T10:38:57.092165+0000","last_peered":"2024-09-21T10:38:57.092165+0000","last_clean":"2024-09-21T10:38:57.092165+0000","last_became_active":"2024-09-21T09:54:53.624750+0000","last_became_peered":"2024-09-21T09:54:53.624750+0000","last_unstale":"2024-09-21T10:38:57.092165+0000","last_undegraded":"2024-09-21T10:38:57.092165+0000","last_fullsized":"2024-09-21T10:38:57.092165+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:34:26.823544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1910,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.115602+0000","last_change":"2024-09-21T10:09:06.558492+0000","last_active":"2024-09-21T10:38:57.115602+0000","last_peered":"2024-09-21T10:38:57.115602+0000","last_clean":"2024-09-21T10:38:57.115602+0000","last_became_active":"2024-09-21T10:09:06.558146+0000","last_became_peered":"2024-09-21T10:09:06.558146+0000","last_unstale":"2024-09-21T10:38:57.115602+0000","last_undegraded":"2024-09-21T10:38:57.115602+0000","last_fullsized":"2024-09-21T10:38:57.115602+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:28:50.638241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1893,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.117388+0000","last_change":"2024-09-21T10:09:06.579005+0000","last_active":"2024-09-21T10:38:57.117388+0000","last_peered":"2024-09-21T10:38:57.117388+0000","last_clean":"2024-09-21T10:38:57.117388+0000","last_became_active":"2024-09-21T10:09:06.578119+0000","last_became_peered":"2024-09-21T10:09:06.578119+0000","last_unstale":"2024-09-21T10:38:57.117388+0000","last_undegraded":"2024-09-21T10:38:57.117388+0000","last_fullsized":"2024-09-21T10:38:57.117388+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:20.615838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1270,"reported_epoch":1303,"state":"active+clean","last_fresh":"2024-09-21T10:38:54.020627+0000","last_change":"2024-09-21T10:09:06.557453+0000","last_active":"2024-09-21T10:38:54.020627+0000","last_peered":"2024-09-21T10:38:54.020627+0000","last_clean":"2024-09-21T10:38:54.020627+0000","last_became_active":"2024-09-21T10:09:06.557124+0000","last_became_peered":"2024-09-21T10:09:06.557124+0000","last_unstale":"2024-09-21T10:38:54.020627+0000","last_undegraded":"2024-09-21T10:38:54.020627+0000","last_fullsized":"2024-09-21T10:38:54.020627+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:03:27.731832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":1276,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.091450+0000","last_change":"2024-09-21T10:08:30.574245+0000","last_active":"2024-09-21T10:38:57.091450+0000","last_peered":"2024-09-21T10:38:57.091450+0000","last_clean":"2024-09-21T10:38:57.091450+0000","last_became_active":"2024-09-21T10:08:30.573909+0000","last_became_peered":"2024-09-21T10:08:30.573909+0000","last_unstale":"2024-09-21T10:38:57.091450+0000","last_undegraded":"2024-09-21T10:38:57.091450+0000","last_fullsized":"2024-09-21T10:38:57.091450+0000","mapping_epoch":680,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:18:03.911157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1880,"reported_epoch":1304,"state":"active+clean","last_fresh":"2024-09-21T10:38:57.120094+0000","last_change":"2024-09-21T10:09:06.578773+0000","last_active":"2024-09-21T10:38:57.120094+0000","last_peered":"2024-09-21T10:38:57.120094+0000","last_clean":"2024-09-21T10:38:57.120094+0000","last_became_active":"2024-09-21T10:09:06.576688+0000","last_became_peered":"2024-09-21T10:09:06.576688+0000","last_unstale":"2024-09-21T10:38:57.120094+0000","last_undegraded":"2024-09-21T10:38:57.120094+0000","last_fullsized":"2024-09-21T10:38:57.120094+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:37.699650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234551,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":140356,"kb_used_data":14968,"kb_used_omap":1,"kb_used_meta":125374,"kb_avail":93604796,"statfs":{"total":95995035648,"available":95851311104,"internally_reserved":0,"allocated":15327232,"data_stored":9832930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":128383435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496188,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":148540,"kb_used_data":14960,"kb_used_omap":1,"kb_used_meta":133566,"kb_avail":93596612,"statfs":{"total":95995035648,"available":95842930688,"internally_reserved":0,"allocated":15319040,"data_stored":9832803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":136772042},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757825,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":107824,"kb_used_data":14372,"kb_used_omap":1,"kb_used_meta":93438,"kb_avail":93637328,"statfs":{"total":95995035648,"available":95884623872,"internally_reserved":0,"allocated":14716928,"data_stored":9238261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95680966},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019461,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":117624,"kb_used_data":14384,"kb_used_omap":1,"kb_used_meta":103230,"kb_avail":93627528,"statfs":{"total":95995035648,"available":95874588672,"internally_reserved":0,"allocated":14729216,"data_stored":9242492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105707974},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369204,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78440,"kb_used_data":14364,"kb_used_omap":69,"kb_used_meta":63994,"kb_avail":93666712,"statfs":{"total":95995035648,"available":95914713088,"internally_reserved":0,"allocated":14708736,"data_stored":9237692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":65529986},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":680,"seq":2920577761659,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":68716,"kb_used_data":14368,"kb_used_omap":76,"kb_used_meta":54259,"kb_avail":93676436,"statfs":{"total":95995035648,"available":95924670464,"internally_reserved":0,"allocated":14712832,"data_stored":9238218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78585,"internal_metadata":55561479},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807957890,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86632,"kb_used_data":14364,"kb_used_omap":66,"kb_used_meta":72189,"kb_avail":93658520,"statfs":{"total":95995035648,"available":95906324480,"internally_reserved":0,"allocated":14708736,"data_stored":9237621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68589,"internal_metadata":73921555},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":666,"seq":2860448219529,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87024,"kb_used_data":14948,"kb_used_omap":76,"kb_used_meta":71987,"kb_avail":93658128,"statfs":{"total":95995035648,"available":95905923072,"internally_reserved":0,"allocated":15306752,"data_stored":9828574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78716,"internal_metadata":73714820},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:39:59.014 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T10:39:59.014 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd dump 2024-09-21T10:39:59.210 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:59.581 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:39:59 smithi070 ceph-mon[107385]: from='client.44963 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:39:59.581 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:39:59 smithi070 ceph-mon[112139]: from='client.44963 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:39:59.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:39:59 smithi160 ceph-mon[101879]: from='client.44963 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:40:00.599 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:00 smithi070 ceph-mon[107385]: pgmap v2137: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:40:00.599 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:00 smithi070 ceph-mon[107385]: overall HEALTH_OK 2024-09-21T10:40:00.599 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:00 smithi070 ceph-mon[112139]: pgmap v2137: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:40:00.599 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:00 smithi070 ceph-mon[112139]: overall HEALTH_OK 2024-09-21T10:40:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:00 smithi160 ceph-mon[101879]: pgmap v2137: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:40:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:00 smithi160 ceph-mon[101879]: overall HEALTH_OK 2024-09-21T10:40:00.978 INFO:teuthology.orchestra.run.smithi070.stdout:epoch 1304 2024-09-21T10:40:00.978 INFO:teuthology.orchestra.run.smithi070.stdout:fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T10:40:00.978 INFO:teuthology.orchestra.run.smithi070.stdout:created 2024-09-21T09:47:59.216934+0000 2024-09-21T10:40:00.978 INFO:teuthology.orchestra.run.smithi070.stdout:modified 2024-09-21T10:38:57.083270+0000 2024-09-21T10:40:00.978 INFO:teuthology.orchestra.run.smithi070.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-21T10:40:00.978 INFO:teuthology.orchestra.run.smithi070.stdout:crush_version 22 2024-09-21T10:40:00.978 INFO:teuthology.orchestra.run.smithi070.stdout:full_ratio 0.95 2024-09-21T10:40:00.978 INFO:teuthology.orchestra.run.smithi070.stdout:backfillfull_ratio 0.9 2024-09-21T10:40:00.978 INFO:teuthology.orchestra.run.smithi070.stdout:nearfull_ratio 0.85 2024-09-21T10:40:00.979 INFO:teuthology.orchestra.run.smithi070.stdout:require_min_compat_client luminous 2024-09-21T10:40:00.979 INFO:teuthology.orchestra.run.smithi070.stdout:min_compat_client luminous 2024-09-21T10:40:00.979 INFO:teuthology.orchestra.run.smithi070.stdout:require_osd_release reef 2024-09-21T10:40:00.979 INFO:teuthology.orchestra.run.smithi070.stdout:stretch_mode_enabled false 2024-09-21T10:40:00.979 INFO:teuthology.orchestra.run.smithi070.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-09-21T10:40:00.979 INFO:teuthology.orchestra.run.smithi070.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 65 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.00 2024-09-21T10:40:00.979 INFO:teuthology.orchestra.run.smithi070.stdout:pool 3 'cephfs.foo.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 70 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.50 2024-09-21T10:40:00.979 INFO:teuthology.orchestra.run.smithi070.stdout:pool 4 'cephfs.foo.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 71 flags hashpspool,bulk stripe_width 0 application cephfs read_balance_score 2.25 2024-09-21T10:40:00.979 INFO:teuthology.orchestra.run.smithi070.stdout:max_osd 8 2024-09-21T10:40:00.979 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 up in weight 1 up_from 666 up_thru 1156 down_at 660 last_clean_interval [8,660) [v2:172.21.15.70:6802/452014366,v1:172.21.15.70:6803/452014366] [v2:172.21.15.70:6804/452014366,v1:172.21.15.70:6805/452014366] exists,up f6870ff8-14eb-4110-957a-00c73645b181 2024-09-21T10:40:00.979 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 up in weight 1 up_from 674 up_thru 1156 down_at 669 last_clean_interval [13,669) [v2:172.21.15.70:6810/3938187175,v1:172.21.15.70:6811/3938187175] [v2:172.21.15.70:6812/3938187175,v1:172.21.15.70:6813/3938187175] exists,up b1349e31-7493-489f-823a-55edb866d25f 2024-09-21T10:40:00.979 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 up in weight 1 up_from 680 up_thru 1156 down_at 677 last_clean_interval [18,677) [v2:172.21.15.70:6818/890146607,v1:172.21.15.70:6819/890146607] [v2:172.21.15.70:6820/890146607,v1:172.21.15.70:6821/890146607] exists,up ac4496ba-a860-4b09-bb38-7f433e081541 2024-09-21T10:40:00.979 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 up in weight 1 up_from 692 up_thru 1156 down_at 689 last_clean_interval [25,689) [v2:172.21.15.70:6826/3858266770,v1:172.21.15.70:6827/3858266770] [v2:172.21.15.70:6828/3858266770,v1:172.21.15.70:6829/3858266770] exists,up 1eedf4ce-057c-4576-ae5b-51d3595df195 2024-09-21T10:40:00.980 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 up in weight 1 up_from 30 up_thru 1156 down_at 0 last_clean_interval [0,0) [v2:172.21.15.160:6800/3634505869,v1:172.21.15.160:6801/3634505869] [v2:172.21.15.160:6802/3634505869,v1:172.21.15.160:6803/3634505869] exists,up a4b728b8-97a4-478f-8a11-ba590c0213e9 2024-09-21T10:40:00.980 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 up in weight 1 up_from 38 up_thru 1156 down_at 0 last_clean_interval [0,0) [v2:172.21.15.160:6808/2006621188,v1:172.21.15.160:6809/2006621188] [v2:172.21.15.160:6810/2006621188,v1:172.21.15.160:6811/2006621188] exists,up 56915347-9254-4e94-b202-1f16d09e7946 2024-09-21T10:40:00.980 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 up in weight 1 up_from 46 up_thru 1156 down_at 0 last_clean_interval [0,0) [v2:172.21.15.160:6816/1949075400,v1:172.21.15.160:6817/1949075400] [v2:172.21.15.160:6818/1949075400,v1:172.21.15.160:6819/1949075400] exists,up 79e6de8c-b78b-40e2-9229-0275a873d40e 2024-09-21T10:40:00.980 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 up in weight 1 up_from 54 up_thru 1156 down_at 0 last_clean_interval [0,0) [v2:172.21.15.160:6824/601673919,v1:172.21.15.160:6825/601673919] [v2:172.21.15.160:6826/601673919,v1:172.21.15.160:6827/601673919] exists,up 72245c8a-2fdb-4445-b378-16569f337fcd 2024-09-21T10:40:00.980 INFO:teuthology.orchestra.run.smithi070.stdout:pg_upmap_items 4.a [1,2] 2024-09-21T10:40:00.980 INFO:teuthology.orchestra.run.smithi070.stdout:pg_upmap_items 4.10 [1,5] 2024-09-21T10:40:00.980 INFO:teuthology.orchestra.run.smithi070.stdout:pg_upmap_items 4.1d [1,2] 2024-09-21T10:40:00.980 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/4293543511 expires 2024-09-21T11:04:19.859325+0000 2024-09-21T10:40:00.980 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/2422341096 expires 2024-09-22T10:00:49.897297+0000 2024-09-21T10:40:00.980 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.160:0/1742426098 expires 2024-09-22T09:58:10.497359+0000 2024-09-21T10:40:00.980 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/1905911777 expires 2024-09-22T09:53:31.134530+0000 2024-09-21T10:40:00.980 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.160:6834/2915531836 expires 2024-09-22T09:58:10.497359+0000 2024-09-21T10:40:00.980 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/4150778424 expires 2024-09-22T09:48:32.256720+0000 2024-09-21T10:40:00.981 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/2308708361 expires 2024-09-22T10:00:49.897297+0000 2024-09-21T10:40:00.981 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:6800/177993484 expires 2024-09-22T09:48:32.256720+0000 2024-09-21T10:40:00.981 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:6800/1520336634 expires 2024-09-22T09:53:31.134530+0000 2024-09-21T10:40:00.981 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.160:0/1014888319 expires 2024-09-22T09:58:10.497359+0000 2024-09-21T10:40:00.981 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:6800/2142025682 expires 2024-09-22T09:48:55.219103+0000 2024-09-21T10:40:00.981 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/1469701388 expires 2024-09-22T09:53:31.134530+0000 2024-09-21T10:40:00.981 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:6801/603216094 expires 2024-09-22T10:00:49.897297+0000 2024-09-21T10:40:00.981 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.160:0/19285887 expires 2024-09-22T09:58:10.497359+0000 2024-09-21T10:40:00.981 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:6801/177993484 expires 2024-09-22T09:48:32.256720+0000 2024-09-21T10:40:00.981 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.160:6835/2915531836 expires 2024-09-22T09:58:10.497359+0000 2024-09-21T10:40:00.981 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/2521911561 expires 2024-09-22T09:48:32.256720+0000 2024-09-21T10:40:00.981 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/2918782890 expires 2024-09-22T09:48:55.219103+0000 2024-09-21T10:40:00.981 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/1012316198 expires 2024-09-22T09:48:32.256720+0000 2024-09-21T10:40:00.981 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/1102418977 expires 2024-09-21T11:13:38.558014+0000 2024-09-21T10:40:00.982 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/749465179 expires 2024-09-22T09:48:55.219103+0000 2024-09-21T10:40:00.982 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/595094674 expires 2024-09-22T09:57:34.443100+0000 2024-09-21T10:40:00.982 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/2386870039 expires 2024-09-22T09:57:34.443100+0000 2024-09-21T10:40:00.982 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/3159615643 expires 2024-09-22T09:48:55.219103+0000 2024-09-21T10:40:00.982 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/1066794139 expires 2024-09-22T09:53:31.134530+0000 2024-09-21T10:40:00.982 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/2110605013 expires 2024-09-22T09:57:34.443100+0000 2024-09-21T10:40:00.982 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:6801/1520336634 expires 2024-09-22T09:53:31.134530+0000 2024-09-21T10:40:00.982 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.160:0/2751391809 expires 2024-09-22T09:58:10.497359+0000 2024-09-21T10:40:00.982 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.160:0/3012862160 expires 2024-09-22T09:58:10.497359+0000 2024-09-21T10:40:00.982 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/752126770 expires 2024-09-22T10:00:49.897297+0000 2024-09-21T10:40:00.982 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:6801/2142025682 expires 2024-09-22T09:48:55.219103+0000 2024-09-21T10:40:00.982 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/1635833554 expires 2024-09-22T10:00:49.897297+0000 2024-09-21T10:40:00.982 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/694065139 expires 2024-09-21T11:04:18.868322+0000 2024-09-21T10:40:00.982 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:6800/3639085022 expires 2024-09-22T09:57:34.443100+0000 2024-09-21T10:40:00.983 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/2141342771 expires 2024-09-22T09:53:31.134530+0000 2024-09-21T10:40:00.983 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:6801/3639085022 expires 2024-09-22T09:57:34.443100+0000 2024-09-21T10:40:00.983 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/47418078 expires 2024-09-22T09:57:34.443100+0000 2024-09-21T10:40:00.983 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.160:0/2880106921 expires 2024-09-22T09:58:10.497359+0000 2024-09-21T10:40:00.983 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:6800/603216094 expires 2024-09-22T10:00:49.897297+0000 2024-09-21T10:40:01.573 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:01 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1495571006' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T10:40:01.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:01 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1495571006' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T10:40:01.595 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 666 up_thru 1156 down_at 660 last_clean_interval [8,660) [v2:172.21.15.70:6802/452014366,v1:172.21.15.70:6803/452014366] [v2:172.21.15.70:6804/452014366,v1:172.21.15.70:6805/452014366] exists,up f6870ff8-14eb-4110-957a-00c73645b181', 'osd.1 up in weight 1 up_from 674 up_thru 1156 down_at 669 last_clean_interval [13,669) [v2:172.21.15.70:6810/3938187175,v1:172.21.15.70:6811/3938187175] [v2:172.21.15.70:6812/3938187175,v1:172.21.15.70:6813/3938187175] exists,up b1349e31-7493-489f-823a-55edb866d25f', 'osd.2 up in weight 1 up_from 680 up_thru 1156 down_at 677 last_clean_interval [18,677) [v2:172.21.15.70:6818/890146607,v1:172.21.15.70:6819/890146607] [v2:172.21.15.70:6820/890146607,v1:172.21.15.70:6821/890146607] exists,up ac4496ba-a860-4b09-bb38-7f433e081541', 'osd.3 up in weight 1 up_from 692 up_thru 1156 down_at 689 last_clean_interval [25,689) [v2:172.21.15.70:6826/3858266770,v1:172.21.15.70:6827/3858266770] [v2:172.21.15.70:6828/3858266770,v1:172.21.15.70:6829/3858266770] exists,up 1eedf4ce-057c-4576-ae5b-51d3595df195', 'osd.4 up in weight 1 up_from 30 up_thru 1156 down_at 0 last_clean_interval [0,0) [v2:172.21.15.160:6800/3634505869,v1:172.21.15.160:6801/3634505869] [v2:172.21.15.160:6802/3634505869,v1:172.21.15.160:6803/3634505869] exists,up a4b728b8-97a4-478f-8a11-ba590c0213e9', 'osd.5 up in weight 1 up_from 38 up_thru 1156 down_at 0 last_clean_interval [0,0) [v2:172.21.15.160:6808/2006621188,v1:172.21.15.160:6809/2006621188] [v2:172.21.15.160:6810/2006621188,v1:172.21.15.160:6811/2006621188] exists,up 56915347-9254-4e94-b202-1f16d09e7946', 'osd.6 up in weight 1 up_from 46 up_thru 1156 down_at 0 last_clean_interval [0,0) [v2:172.21.15.160:6816/1949075400,v1:172.21.15.160:6817/1949075400] [v2:172.21.15.160:6818/1949075400,v1:172.21.15.160:6819/1949075400] exists,up 79e6de8c-b78b-40e2-9229-0275a873d40e', 'osd.7 up in weight 1 up_from 54 up_thru 1156 down_at 0 last_clean_interval [0,0) [v2:172.21.15.160:6824/601673919,v1:172.21.15.160:6825/601673919] [v2:172.21.15.160:6826/601673919,v1:172.21.15.160:6827/601673919] exists,up 72245c8a-2fdb-4445-b378-16569f337fcd'] 2024-09-21T10:40:01.595 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon mon.a config show 2024-09-21T10:40:01.792 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:01 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1495571006' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T10:40:02.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:02 smithi070 ceph-mon[107385]: pgmap v2138: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:40:02.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:02 smithi070 ceph-mon[112139]: pgmap v2138: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:40:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:02 smithi160 ceph-mon[101879]: pgmap v2138: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:40:03.226 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:40:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:40:03] "GET /metrics HTTP/1.1" 200 34047 "" "Prometheus/2.51.0" 2024-09-21T10:40:03.443 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:40:03.443 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mon.a", 2024-09-21T10:40:03.443 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster": "ceph", 2024-09-21T10:40:03.443 INFO:teuthology.orchestra.run.smithi070.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-21T10:40:03.443 INFO:teuthology.orchestra.run.smithi070.stdout: "admin_socket_mode": "", 2024-09-21T10:40:03.443 INFO:teuthology.orchestra.run.smithi070.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-09-21T10:40:03.443 INFO:teuthology.orchestra.run.smithi070.stdout: "auth_client_required": "cephx, none", 2024-09-21T10:40:03.443 INFO:teuthology.orchestra.run.smithi070.stdout: "auth_cluster_required": "cephx", 2024-09-21T10:40:03.443 INFO:teuthology.orchestra.run.smithi070.stdout: "auth_debug": "true", 2024-09-21T10:40:03.443 INFO:teuthology.orchestra.run.smithi070.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-21T10:40:03.444 INFO:teuthology.orchestra.run.smithi070.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-21T10:40:03.444 INFO:teuthology.orchestra.run.smithi070.stdout: "auth_service_required": "cephx", 2024-09-21T10:40:03.444 INFO:teuthology.orchestra.run.smithi070.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-21T10:40:03.444 INFO:teuthology.orchestra.run.smithi070.stdout: "auth_supported": "", 2024-09-21T10:40:03.444 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_aio": "true", 2024-09-21T10:40:03.444 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-21T10:40:03.444 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_aio_poll_ms": "250", 2024-09-21T10:40:03.444 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_aio_reap_max": "16", 2024-09-21T10:40:03.444 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_async_discard_threads": "0", 2024-09-21T10:40:03.444 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_block_size": "4096", 2024-09-21T10:40:03.444 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_debug_aio": "false", 2024-09-21T10:40:03.444 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-21T10:40:03.444 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-21T10:40:03.445 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_debug_inflight_ios": "false", 2024-09-21T10:40:03.445 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_enable_discard": "false", 2024-09-21T10:40:03.445 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_flock_retry": "3", 2024-09-21T10:40:03.445 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-21T10:40:03.445 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_inject_crash": "0", 2024-09-21T10:40:03.445 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-21T10:40:03.445 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_ioring": "false", 2024-09-21T10:40:03.445 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_ioring_hipri": "false", 2024-09-21T10:40:03.445 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-21T10:40:03.445 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-21T10:40:03.445 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-21T10:40:03.445 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-21T10:40:03.445 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-21T10:40:03.445 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-21T10:40:03.446 INFO:teuthology.orchestra.run.smithi070.stdout: "bdev_type": "", 2024-09-21T10:40:03.446 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_alloc_size": "1048576", 2024-09-21T10:40:03.446 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_allocator": "hybrid", 2024-09-21T10:40:03.446 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_buffered_io": "true", 2024-09-21T10:40:03.446 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_check_for_zeros": "false", 2024-09-21T10:40:03.446 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-21T10:40:03.446 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-21T10:40:03.446 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_compact_log_sync": "false", 2024-09-21T10:40:03.446 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-21T10:40:03.446 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-21T10:40:03.446 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-21T10:40:03.446 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-21T10:40:03.446 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_max_log_runway": "4194304", 2024-09-21T10:40:03.447 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_max_prefetch": "1048576", 2024-09-21T10:40:03.447 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_min_flush_size": "524288", 2024-09-21T10:40:03.447 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_min_log_runway": "1048576", 2024-09-21T10:40:03.447 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_replay_recovery": "false", 2024-09-21T10:40:03.447 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-21T10:40:03.447 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-21T10:40:03.447 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_sync_write": "false", 2024-09-21T10:40:03.447 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-21T10:40:03.447 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-21T10:40:03.447 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-21T10:40:03.447 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_allocation_from_file": "true", 2024-09-21T10:40:03.447 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_allocator": "hybrid", 2024-09-21T10:40:03.447 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_async_db_compaction": "true", 2024-09-21T10:40:03.448 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-21T10:40:03.448 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-21T10:40:03.448 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-21T10:40:03.448 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-21T10:40:03.448 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_bdev_label_multi": "true", 2024-09-21T10:40:03.448 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-21T10:40:03.448 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-21T10:40:03.448 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-21T10:40:03.448 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-21T10:40:03.448 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-21T10:40:03.448 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_block_create": "true", 2024-09-21T10:40:03.448 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_block_db_create": "false", 2024-09-21T10:40:03.448 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_block_db_path": "", 2024-09-21T10:40:03.448 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_block_db_size": "0", 2024-09-21T10:40:03.449 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_block_path": "", 2024-09-21T10:40:03.449 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_block_preallocate_file": "false", 2024-09-21T10:40:03.449 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_block_size": "107374182400", 2024-09-21T10:40:03.449 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_block_wal_create": "false", 2024-09-21T10:40:03.449 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_block_wal_path": "", 2024-09-21T10:40:03.449 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_block_wal_size": "100663296", 2024-09-21T10:40:03.449 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_bluefs": "true", 2024-09-21T10:40:03.449 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-21T10:40:03.449 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-21T10:40:03.449 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-21T10:40:03.449 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-21T10:40:03.449 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-21T10:40:03.449 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T10:40:03.449 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T10:40:03.450 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-21T10:40:03.450 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T10:40:03.450 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_autotune": "true", 2024-09-21T10:40:03.450 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-21T10:40:03.450 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-21T10:40:03.450 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-21T10:40:03.450 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-21T10:40:03.450 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_size": "0", 2024-09-21T10:40:03.450 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-21T10:40:03.450 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-21T10:40:03.450 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-21T10:40:03.450 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-21T10:40:03.451 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_type": "2q", 2024-09-21T10:40:03.451 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-21T10:40:03.451 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_clone_cow": "true", 2024-09-21T10:40:03.451 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-21T10:40:03.451 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-21T10:40:03.451 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-21T10:40:03.451 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-21T10:40:03.451 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-21T10:40:03.451 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-21T10:40:03.451 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-21T10:40:03.451 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_compression_mode": "none", 2024-09-21T10:40:03.451 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-21T10:40:03.451 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_csum_type": "crc32c", 2024-09-21T10:40:03.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-21T10:40:03.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-21T10:40:03.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_freelist": "false", 2024-09-21T10:40:03.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-21T10:40:03.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-21T10:40:03.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-21T10:40:03.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-21T10:40:03.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-21T10:40:03.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-21T10:40:03.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-21T10:40:03.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-21T10:40:03.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-21T10:40:03.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-21T10:40:03.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-21T10:40:03.453 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-21T10:40:03.453 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_small_allocations": "0", 2024-09-21T10:40:03.453 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-21T10:40:03.453 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_default_buffered_read": "true", 2024-09-21T10:40:03.453 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_default_buffered_write": "false", 2024-09-21T10:40:03.453 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-21T10:40:03.453 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-21T10:40:03.453 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-21T10:40:03.453 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-21T10:40:03.453 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-21T10:40:03.453 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-21T10:40:03.453 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-21T10:40:03.453 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-21T10:40:03.454 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-21T10:40:03.454 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fail_eio": "false", 2024-09-21T10:40:03.454 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-21T10:40:03.454 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-21T10:40:03.454 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-21T10:40:03.454 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-21T10:40:03.454 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-21T10:40:03.454 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-21T10:40:03.454 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_on_mount": "false", 2024-09-21T10:40:03.454 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-21T10:40:03.454 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_on_umount": "false", 2024-09-21T10:40:03.454 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-21T10:40:03.454 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-21T10:40:03.454 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-21T10:40:03.455 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-21T10:40:03.455 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-21T10:40:03.455 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-21T10:40:03.455 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-21T10:40:03.455 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-21T10:40:03.455 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_ignore_data_csum": "false", 2024-09-21T10:40:03.455 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-21T10:40:03.455 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-21T10:40:03.455 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-21T10:40:03.455 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-21T10:40:03.455 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_log_op_age": "5.000000", 2024-09-21T10:40:03.455 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_max_alloc_size": "0", 2024-09-21T10:40:03.455 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_max_blob_size": "0", 2024-09-21T10:40:03.456 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-21T10:40:03.456 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-21T10:40:03.456 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-21T10:40:03.456 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_max_deferred_txc": "32", 2024-09-21T10:40:03.456 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_min_alloc_size": "0", 2024-09-21T10:40:03.456 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-21T10:40:03.456 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-21T10:40:03.456 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_nid_prealloc": "1024", 2024-09-21T10:40:03.456 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-21T10:40:03.456 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-21T10:40:03.456 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-21T10:40:03.456 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-21T10:40:03.457 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_retry_disk_reads": "3", 2024-09-21T10:40:03.457 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_rocksdb_cf": "true", 2024-09-21T10:40:03.457 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-21T10:40:03.457 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-21T10:40:03.457 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-21T10:40:03.457 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-21T10:40:03.457 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-21T10:40:03.458 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-21T10:40:03.458 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-21T10:40:03.458 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-21T10:40:03.458 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_spdk_mem": "512", 2024-09-21T10:40:03.458 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-21T10:40:03.458 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-21T10:40:03.458 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-21T10:40:03.458 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-21T10:40:03.458 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-21T10:40:03.458 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-21T10:40:03.458 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-21T10:40:03.458 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_tracing": "false", 2024-09-21T10:40:03.458 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-21T10:40:03.459 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-21T10:40:03.459 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-21T10:40:03.459 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-21T10:40:03.459 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-21T10:40:03.459 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-21T10:40:03.459 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-21T10:40:03.459 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-21T10:40:03.459 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-21T10:40:03.459 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_write_v2": "false", 2024-09-21T10:40:03.459 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_write_v2_random": "false", 2024-09-21T10:40:03.459 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_zero_block_detection": "false", 2024-09-21T10:40:03.459 INFO:teuthology.orchestra.run.smithi070.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-21T10:40:03.459 INFO:teuthology.orchestra.run.smithi070.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-21T10:40:03.459 INFO:teuthology.orchestra.run.smithi070.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-21T10:40:03.459 INFO:teuthology.orchestra.run.smithi070.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-21T10:40:03.460 INFO:teuthology.orchestra.run.smithi070.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-21T10:40:03.460 INFO:teuthology.orchestra.run.smithi070.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-21T10:40:03.460 INFO:teuthology.orchestra.run.smithi070.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-21T10:40:03.460 INFO:teuthology.orchestra.run.smithi070.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-21T10:40:03.460 INFO:teuthology.orchestra.run.smithi070.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-21T10:40:03.460 INFO:teuthology.orchestra.run.smithi070.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-21T10:40:03.460 INFO:teuthology.orchestra.run.smithi070.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-21T10:40:03.460 INFO:teuthology.orchestra.run.smithi070.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-21T10:40:03.460 INFO:teuthology.orchestra.run.smithi070.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-21T10:40:03.460 INFO:teuthology.orchestra.run.smithi070.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-21T10:40:03.460 INFO:teuthology.orchestra.run.smithi070.stdout: "cephx_cluster_require_signatures": "false", 2024-09-21T10:40:03.460 INFO:teuthology.orchestra.run.smithi070.stdout: "cephx_cluster_require_version": "2", 2024-09-21T10:40:03.460 INFO:teuthology.orchestra.run.smithi070.stdout: "cephx_require_signatures": "false", 2024-09-21T10:40:03.460 INFO:teuthology.orchestra.run.smithi070.stdout: "cephx_require_version": "2", 2024-09-21T10:40:03.461 INFO:teuthology.orchestra.run.smithi070.stdout: "cephx_service_require_signatures": "false", 2024-09-21T10:40:03.461 INFO:teuthology.orchestra.run.smithi070.stdout: "cephx_service_require_version": "2", 2024-09-21T10:40:03.461 INFO:teuthology.orchestra.run.smithi070.stdout: "cephx_sign_messages": "true", 2024-09-21T10:40:03.461 INFO:teuthology.orchestra.run.smithi070.stdout: "chdir": "", 2024-09-21T10:40:03.461 INFO:teuthology.orchestra.run.smithi070.stdout: "client_acl_type": "", 2024-09-21T10:40:03.461 INFO:teuthology.orchestra.run.smithi070.stdout: "client_asio_thread_count": "2", 2024-09-21T10:40:03.461 INFO:teuthology.orchestra.run.smithi070.stdout: "client_cache_mid": "0.750000", 2024-09-21T10:40:03.461 INFO:teuthology.orchestra.run.smithi070.stdout: "client_cache_size": "16384", 2024-09-21T10:40:03.461 INFO:teuthology.orchestra.run.smithi070.stdout: "client_caps_release_delay": "5", 2024-09-21T10:40:03.461 INFO:teuthology.orchestra.run.smithi070.stdout: "client_check_pool_perm": "true", 2024-09-21T10:40:03.461 INFO:teuthology.orchestra.run.smithi070.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-21T10:40:03.461 INFO:teuthology.orchestra.run.smithi070.stdout: "client_debug_force_sync_read": "false", 2024-09-21T10:40:03.461 INFO:teuthology.orchestra.run.smithi070.stdout: "client_debug_getattr_caps": "false", 2024-09-21T10:40:03.462 INFO:teuthology.orchestra.run.smithi070.stdout: "client_debug_inject_features": "", 2024-09-21T10:40:03.462 INFO:teuthology.orchestra.run.smithi070.stdout: "client_debug_inject_tick_delay": "0", 2024-09-21T10:40:03.462 INFO:teuthology.orchestra.run.smithi070.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-21T10:40:03.462 INFO:teuthology.orchestra.run.smithi070.stdout: "client_die_on_failed_remount": "false", 2024-09-21T10:40:03.462 INFO:teuthology.orchestra.run.smithi070.stdout: "client_dirsize_rbytes": "true", 2024-09-21T10:40:03.462 INFO:teuthology.orchestra.run.smithi070.stdout: "client_force_lazyio": "false", 2024-09-21T10:40:03.462 INFO:teuthology.orchestra.run.smithi070.stdout: "client_fs": "", 2024-09-21T10:40:03.462 INFO:teuthology.orchestra.run.smithi070.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-21T10:40:03.462 INFO:teuthology.orchestra.run.smithi070.stdout: "client_inject_release_failure": "false", 2024-09-21T10:40:03.462 INFO:teuthology.orchestra.run.smithi070.stdout: "client_max_inline_size": "4096", 2024-09-21T10:40:03.462 INFO:teuthology.orchestra.run.smithi070.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-21T10:40:03.462 INFO:teuthology.orchestra.run.smithi070.stdout: "client_mds_namespace": "", 2024-09-21T10:40:03.462 INFO:teuthology.orchestra.run.smithi070.stdout: "client_metadata": "", 2024-09-21T10:40:03.462 INFO:teuthology.orchestra.run.smithi070.stdout: "client_mount_gid": "-1", 2024-09-21T10:40:03.463 INFO:teuthology.orchestra.run.smithi070.stdout: "client_mount_timeout": "300", 2024-09-21T10:40:03.463 INFO:teuthology.orchestra.run.smithi070.stdout: "client_mount_uid": "-1", 2024-09-21T10:40:03.463 INFO:teuthology.orchestra.run.smithi070.stdout: "client_mountpoint": "/", 2024-09-21T10:40:03.463 INFO:teuthology.orchestra.run.smithi070.stdout: "client_notify_timeout": "10", 2024-09-21T10:40:03.463 INFO:teuthology.orchestra.run.smithi070.stdout: "client_oc": "true", 2024-09-21T10:40:03.463 INFO:teuthology.orchestra.run.smithi070.stdout: "client_oc_max_dirty": "104857600", 2024-09-21T10:40:03.463 INFO:teuthology.orchestra.run.smithi070.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-21T10:40:03.463 INFO:teuthology.orchestra.run.smithi070.stdout: "client_oc_max_objects": "1000", 2024-09-21T10:40:03.463 INFO:teuthology.orchestra.run.smithi070.stdout: "client_oc_size": "209715200", 2024-09-21T10:40:03.463 INFO:teuthology.orchestra.run.smithi070.stdout: "client_oc_target_dirty": "8388608", 2024-09-21T10:40:03.463 INFO:teuthology.orchestra.run.smithi070.stdout: "client_permissions": "true", 2024-09-21T10:40:03.463 INFO:teuthology.orchestra.run.smithi070.stdout: "client_quota": "true", 2024-09-21T10:40:03.463 INFO:teuthology.orchestra.run.smithi070.stdout: "client_quota_df": "true", 2024-09-21T10:40:03.463 INFO:teuthology.orchestra.run.smithi070.stdout: "client_readahead_max_bytes": "0", 2024-09-21T10:40:03.464 INFO:teuthology.orchestra.run.smithi070.stdout: "client_readahead_max_periods": "4", 2024-09-21T10:40:03.464 INFO:teuthology.orchestra.run.smithi070.stdout: "client_readahead_min": "131072", 2024-09-21T10:40:03.464 INFO:teuthology.orchestra.run.smithi070.stdout: "client_reconnect_stale": "false", 2024-09-21T10:40:03.464 INFO:teuthology.orchestra.run.smithi070.stdout: "client_shutdown_timeout": "30", 2024-09-21T10:40:03.464 INFO:teuthology.orchestra.run.smithi070.stdout: "client_snapdir": ".snap", 2024-09-21T10:40:03.464 INFO:teuthology.orchestra.run.smithi070.stdout: "client_tick_interval": "1", 2024-09-21T10:40:03.464 INFO:teuthology.orchestra.run.smithi070.stdout: "client_trace": "", 2024-09-21T10:40:03.464 INFO:teuthology.orchestra.run.smithi070.stdout: "client_try_dentry_invalidate": "false", 2024-09-21T10:40:03.464 INFO:teuthology.orchestra.run.smithi070.stdout: "client_use_faked_inos": "false", 2024-09-21T10:40:03.464 INFO:teuthology.orchestra.run.smithi070.stdout: "client_use_random_mds": "false", 2024-09-21T10:40:03.464 INFO:teuthology.orchestra.run.smithi070.stdout: "clog_to_graylog": "false", 2024-09-21T10:40:03.464 INFO:teuthology.orchestra.run.smithi070.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-21T10:40:03.464 INFO:teuthology.orchestra.run.smithi070.stdout: "clog_to_graylog_port": "12201", 2024-09-21T10:40:03.464 INFO:teuthology.orchestra.run.smithi070.stdout: "clog_to_monitors": "default=true", 2024-09-21T10:40:03.465 INFO:teuthology.orchestra.run.smithi070.stdout: "clog_to_syslog": "false", 2024-09-21T10:40:03.465 INFO:teuthology.orchestra.run.smithi070.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-21T10:40:03.465 INFO:teuthology.orchestra.run.smithi070.stdout: "clog_to_syslog_level": "info", 2024-09-21T10:40:03.465 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_addr": "-", 2024-09-21T10:40:03.465 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_network": "", 2024-09-21T10:40:03.465 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_network_interface": "", 2024-09-21T10:40:03.465 INFO:teuthology.orchestra.run.smithi070.stdout: "compressor_zlib_isal": "false", 2024-09-21T10:40:03.465 INFO:teuthology.orchestra.run.smithi070.stdout: "compressor_zlib_level": "5", 2024-09-21T10:40:03.465 INFO:teuthology.orchestra.run.smithi070.stdout: "compressor_zlib_winsize": "-15", 2024-09-21T10:40:03.465 INFO:teuthology.orchestra.run.smithi070.stdout: "compressor_zstd_level": "1", 2024-09-21T10:40:03.465 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image": "quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47", 2024-09-21T10:40:03.465 INFO:teuthology.orchestra.run.smithi070.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-21T10:40:03.465 INFO:teuthology.orchestra.run.smithi070.stdout: "crimson_alien_op_num_threads": "6", 2024-09-21T10:40:03.465 INFO:teuthology.orchestra.run.smithi070.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-21T10:40:03.466 INFO:teuthology.orchestra.run.smithi070.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-21T10:40:03.466 INFO:teuthology.orchestra.run.smithi070.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-21T10:40:03.466 INFO:teuthology.orchestra.run.smithi070.stdout: "crimson_osd_stat_interval": "0", 2024-09-21T10:40:03.466 INFO:teuthology.orchestra.run.smithi070.stdout: "crimson_seastar_cpu_cores": "", 2024-09-21T10:40:03.466 INFO:teuthology.orchestra.run.smithi070.stdout: "crimson_seastar_num_threads": "0", 2024-09-21T10:40:03.466 INFO:teuthology.orchestra.run.smithi070.stdout: "crush_location": "", 2024-09-21T10:40:03.466 INFO:teuthology.orchestra.run.smithi070.stdout: "crush_location_hook": "", 2024-09-21T10:40:03.466 INFO:teuthology.orchestra.run.smithi070.stdout: "crush_location_hook_timeout": "10", 2024-09-21T10:40:03.466 INFO:teuthology.orchestra.run.smithi070.stdout: "daemonize": "false", 2024-09-21T10:40:03.466 INFO:teuthology.orchestra.run.smithi070.stdout: "daos_pool": "tank", 2024-09-21T10:40:03.466 INFO:teuthology.orchestra.run.smithi070.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-21T10:40:03.466 INFO:teuthology.orchestra.run.smithi070.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-21T10:40:03.466 INFO:teuthology.orchestra.run.smithi070.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-21T10:40:03.467 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_alienstore": "0/5", 2024-09-21T10:40:03.467 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_allow_any_pool_priority": "false", 2024-09-21T10:40:03.467 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_asok": "1/5", 2024-09-21T10:40:03.467 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_asok_assert_abort": "false", 2024-09-21T10:40:03.467 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_asserts_on_shutdown": "true", 2024-09-21T10:40:03.467 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_auth": "1/5", 2024-09-21T10:40:03.467 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_bdev": "1/3", 2024-09-21T10:40:03.467 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_bluefs": "1/5", 2024-09-21T10:40:03.467 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_bluestore": "1/5", 2024-09-21T10:40:03.467 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_buffer": "0/1", 2024-09-21T10:40:03.467 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_ceph_dedup": "0/5", 2024-09-21T10:40:03.467 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_ceph_exporter": "1/5", 2024-09-21T10:40:03.467 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_cephfs_mirror": "0/5", 2024-09-21T10:40:03.467 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_cephsqlite": "0/5", 2024-09-21T10:40:03.467 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_client": "0/5", 2024-09-21T10:40:03.468 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_compressor": "1/5", 2024-09-21T10:40:03.468 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_context": "0/1", 2024-09-21T10:40:03.468 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_crimson_interrupt": "0/5", 2024-09-21T10:40:03.468 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_crush": "1/1", 2024-09-21T10:40:03.468 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_crypto": "1/5", 2024-09-21T10:40:03.468 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_cyanstore": "0/5", 2024-09-21T10:40:03.468 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_deliberately_leak_memory": "false", 2024-09-21T10:40:03.468 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_disable_randomized_ping": "false", 2024-09-21T10:40:03.468 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_dpdk": "1/5", 2024-09-21T10:40:03.468 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_eventtrace": "1/5", 2024-09-21T10:40:03.468 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_filer": "0/1", 2024-09-21T10:40:03.468 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_filestore": "1/3", 2024-09-21T10:40:03.468 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_finisher": "1/1", 2024-09-21T10:40:03.468 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_fuse": "1/5", 2024-09-21T10:40:03.468 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_heartbeat_testing_span": "0", 2024-09-21T10:40:03.469 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_heartbeatmap": "1/5", 2024-09-21T10:40:03.469 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-21T10:40:03.469 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_javaclient": "1/5", 2024-09-21T10:40:03.469 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_journal": "1/3", 2024-09-21T10:40:03.469 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_journaler": "0/5", 2024-09-21T10:40:03.469 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_kstore": "1/5", 2024-09-21T10:40:03.469 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_lockdep": "0/1", 2024-09-21T10:40:03.469 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_mclock": "1/5", 2024-09-21T10:40:03.469 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_mds": "1/5", 2024-09-21T10:40:03.469 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_mds_balancer": "1/5", 2024-09-21T10:40:03.469 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_mds_locker": "1/5", 2024-09-21T10:40:03.469 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_mds_log": "1/5", 2024-09-21T10:40:03.469 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_mds_log_expire": "1/5", 2024-09-21T10:40:03.469 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_mds_migrator": "1/5", 2024-09-21T10:40:03.470 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_mds_quiesce": "3/5", 2024-09-21T10:40:03.470 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_memstore": "1/5", 2024-09-21T10:40:03.470 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_mgr": "2/5", 2024-09-21T10:40:03.470 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_mgrc": "1/5", 2024-09-21T10:40:03.470 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_mon": "20/20", 2024-09-21T10:40:03.470 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_monc": "0/10", 2024-09-21T10:40:03.470 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_ms": "1/1", 2024-09-21T10:40:03.470 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_none": "0/5", 2024-09-21T10:40:03.470 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_objclass": "0/5", 2024-09-21T10:40:03.470 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_objectcacher": "0/5", 2024-09-21T10:40:03.470 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_objecter": "0/1", 2024-09-21T10:40:03.470 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_optracker": "0/5", 2024-09-21T10:40:03.470 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_osd": "1/5", 2024-09-21T10:40:03.470 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_paxos": "20/20", 2024-09-21T10:40:03.471 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_perfcounter": "1/5", 2024-09-21T10:40:03.471 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_prioritycache": "1/5", 2024-09-21T10:40:03.471 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rados": "0/5", 2024-09-21T10:40:03.471 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rbd": "0/5", 2024-09-21T10:40:03.471 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rbd_mirror": "0/5", 2024-09-21T10:40:03.471 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rbd_pwl": "0/5", 2024-09-21T10:40:03.471 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rbd_replay": "0/5", 2024-09-21T10:40:03.471 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_refs": "0/0", 2024-09-21T10:40:03.471 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_reserver": "1/1", 2024-09-21T10:40:03.471 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rgw": "1/5", 2024-09-21T10:40:03.471 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rgw_access": "1/5", 2024-09-21T10:40:03.471 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rgw_datacache": "1/5", 2024-09-21T10:40:03.471 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rgw_dbstore": "1/5", 2024-09-21T10:40:03.471 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rgw_flight": "1/5", 2024-09-21T10:40:03.472 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-21T10:40:03.472 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rgw_notification": "1/5", 2024-09-21T10:40:03.472 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rgw_sync": "1/5", 2024-09-21T10:40:03.472 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_rocksdb": "4/5", 2024-09-21T10:40:03.472 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore": "0/5", 2024-09-21T10:40:03.472 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore_backref": "0/5", 2024-09-21T10:40:03.472 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore_cache": "0/5", 2024-09-21T10:40:03.472 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore_cleaner": "0/5", 2024-09-21T10:40:03.472 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore_device": "0/5", 2024-09-21T10:40:03.472 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore_epm": "0/5", 2024-09-21T10:40:03.472 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-21T10:40:03.472 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore_journal": "0/5", 2024-09-21T10:40:03.472 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore_lba": "0/5", 2024-09-21T10:40:03.473 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore_odata": "0/5", 2024-09-21T10:40:03.473 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore_omap": "0/5", 2024-09-21T10:40:03.473 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore_onode": "0/5", 2024-09-21T10:40:03.473 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore_t": "0/5", 2024-09-21T10:40:03.473 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_seastore_tm": "0/5", 2024-09-21T10:40:03.473 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_striper": "0/1", 2024-09-21T10:40:03.473 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_test": "0/5", 2024-09-21T10:40:03.473 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_throttle": "1/1", 2024-09-21T10:40:03.473 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_timer": "0/1", 2024-09-21T10:40:03.473 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_tp": "0/5", 2024-09-21T10:40:03.473 INFO:teuthology.orchestra.run.smithi070.stdout: "debug_trace": "1/5", 2024-09-21T10:40:03.473 INFO:teuthology.orchestra.run.smithi070.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-21T10:40:03.473 INFO:teuthology.orchestra.run.smithi070.stdout: "device_failure_prediction_mode": "none", 2024-09-21T10:40:03.473 INFO:teuthology.orchestra.run.smithi070.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-21T10:40:03.474 INFO:teuthology.orchestra.run.smithi070.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-21T10:40:03.474 INFO:teuthology.orchestra.run.smithi070.stdout: "err_to_graylog": "false", 2024-09-21T10:40:03.474 INFO:teuthology.orchestra.run.smithi070.stdout: "err_to_journald": "false", 2024-09-21T10:40:03.474 INFO:teuthology.orchestra.run.smithi070.stdout: "err_to_stderr": "true", 2024-09-21T10:40:03.474 INFO:teuthology.orchestra.run.smithi070.stdout: "err_to_syslog": "false", 2024-09-21T10:40:03.474 INFO:teuthology.orchestra.run.smithi070.stdout: "event_tracing": "false", 2024-09-21T10:40:03.474 INFO:teuthology.orchestra.run.smithi070.stdout: "exporter_addr": "0.0.0.0", 2024-09-21T10:40:03.474 INFO:teuthology.orchestra.run.smithi070.stdout: "exporter_cert_file": "", 2024-09-21T10:40:03.474 INFO:teuthology.orchestra.run.smithi070.stdout: "exporter_http_port": "9926", 2024-09-21T10:40:03.474 INFO:teuthology.orchestra.run.smithi070.stdout: "exporter_key_file": "", 2024-09-21T10:40:03.474 INFO:teuthology.orchestra.run.smithi070.stdout: "exporter_prio_limit": "5", 2024-09-21T10:40:03.474 INFO:teuthology.orchestra.run.smithi070.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-21T10:40:03.474 INFO:teuthology.orchestra.run.smithi070.stdout: "exporter_sort_metrics": "true", 2024-09-21T10:40:03.475 INFO:teuthology.orchestra.run.smithi070.stdout: "exporter_stats_period": "5", 2024-09-21T10:40:03.475 INFO:teuthology.orchestra.run.smithi070.stdout: "fake_statfs_for_testing": "0", 2024-09-21T10:40:03.475 INFO:teuthology.orchestra.run.smithi070.stdout: "fatal_signal_handlers": "true", 2024-09-21T10:40:03.475 INFO:teuthology.orchestra.run.smithi070.stdout: "filer_max_purge_ops": "10", 2024-09-21T10:40:03.475 INFO:teuthology.orchestra.run.smithi070.stdout: "filer_max_truncate_ops": "128", 2024-09-21T10:40:03.475 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_apply_finisher_threads": "1", 2024-09-21T10:40:03.475 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_blackhole": "false", 2024-09-21T10:40:03.475 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_btrfs_clone_range": "true", 2024-09-21T10:40:03.475 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_btrfs_snap": "true", 2024-09-21T10:40:03.475 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_caller_concurrency": "10", 2024-09-21T10:40:03.475 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_collect_device_partition_information": "true", 2024-09-21T10:40:03.475 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_commit_timeout": "600.000000", 2024-09-21T10:40:03.475 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_debug_inject_read_err": "false", 2024-09-21T10:40:03.475 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_debug_omap_check": "false", 2024-09-21T10:40:03.476 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_debug_verify_split": "false", 2024-09-21T10:40:03.476 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_dump_file": "", 2024-09-21T10:40:03.476 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-21T10:40:03.476 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-21T10:40:03.476 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_fadvise": "true", 2024-09-21T10:40:03.476 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_fail_eio": "true", 2024-09-21T10:40:03.476 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_fd_cache_shards": "16", 2024-09-21T10:40:03.476 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_fd_cache_size": "128", 2024-09-21T10:40:03.476 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_fiemap": "false", 2024-09-21T10:40:03.476 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_fiemap_threshold": "4096", 2024-09-21T10:40:03.476 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-21T10:40:03.476 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-21T10:40:03.476 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_inject_stall": "0", 2024-09-21T10:40:03.477 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_journal_parallel": "false", 2024-09-21T10:40:03.477 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_journal_trailing": "false", 2024-09-21T10:40:03.477 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_journal_writeahead": "false", 2024-09-21T10:40:03.477 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_kill_at": "0", 2024-09-21T10:40:03.477 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-21T10:40:03.477 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-21T10:40:03.477 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-21T10:40:03.477 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-21T10:40:03.477 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-21T10:40:03.477 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_inline_xattrs": "0", 2024-09-21T10:40:03.477 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-21T10:40:03.477 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-21T10:40:03.477 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-21T10:40:03.477 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-21T10:40:03.478 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_xattr_value_size": "0", 2024-09-21T10:40:03.478 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-21T10:40:03.478 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-21T10:40:03.478 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-21T10:40:03.478 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_merge_threshold": "-10", 2024-09-21T10:40:03.478 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-21T10:40:03.478 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_odsync_write": "false", 2024-09-21T10:40:03.478 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_omap_backend": "rocksdb", 2024-09-21T10:40:03.478 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_omap_backend_path": "", 2024-09-21T10:40:03.478 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-21T10:40:03.478 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-21T10:40:03.478 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-21T10:40:03.478 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_op_thread_timeout": "60", 2024-09-21T10:40:03.479 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_op_threads": "2", 2024-09-21T10:40:03.479 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_punch_hole": "false", 2024-09-21T10:40:03.479 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-21T10:40:03.479 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-21T10:40:03.479 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-21T10:40:03.479 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-21T10:40:03.479 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-21T10:40:03.479 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-21T10:40:03.479 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-21T10:40:03.479 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-21T10:40:03.479 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-21T10:40:03.479 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_queue_max_ops": "50", 2024-09-21T10:40:03.479 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-21T10:40:03.480 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_seek_data_hole": "false", 2024-09-21T10:40:03.480 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_sloppy_crc": "false", 2024-09-21T10:40:03.480 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-21T10:40:03.480 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_splice": "false", 2024-09-21T10:40:03.480 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_split_multiple": "2", 2024-09-21T10:40:03.480 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_split_rand_factor": "20", 2024-09-21T10:40:03.480 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_update_to": "1000", 2024-09-21T10:40:03.480 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-21T10:40:03.480 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-21T10:40:03.480 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-21T10:40:03.480 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-21T10:40:03.480 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-21T10:40:03.480 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-21T10:40:03.480 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_wbthrottle_enable": "true", 2024-09-21T10:40:03.481 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-21T10:40:03.481 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-21T10:40:03.481 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-21T10:40:03.481 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-21T10:40:03.481 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-21T10:40:03.481 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-21T10:40:03.481 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_xfs_extsize": "false", 2024-09-21T10:40:03.481 INFO:teuthology.orchestra.run.smithi070.stdout: "filestore_zfs_snap": "false", 2024-09-21T10:40:03.481 INFO:teuthology.orchestra.run.smithi070.stdout: "fio_dir": "/tmp/fio", 2024-09-21T10:40:03.481 INFO:teuthology.orchestra.run.smithi070.stdout: "fsid": "7c68bfbc-77fe-11ef-baf6-efdc52797490", 2024-09-21T10:40:03.481 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_allow_other": "true", 2024-09-21T10:40:03.481 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_atomic_o_trunc": "true", 2024-09-21T10:40:03.481 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_big_writes": "true", 2024-09-21T10:40:03.482 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_debug": "false", 2024-09-21T10:40:03.482 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_default_permissions": "false", 2024-09-21T10:40:03.482 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_disable_pagecache": "false", 2024-09-21T10:40:03.482 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_max_write": "0", 2024-09-21T10:40:03.482 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_multithreaded": "true", 2024-09-21T10:40:03.482 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_require_active_mds": "true", 2024-09-21T10:40:03.482 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_set_user_groups": "true", 2024-09-21T10:40:03.482 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_splice_move": "true", 2024-09-21T10:40:03.482 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_splice_read": "true", 2024-09-21T10:40:03.482 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_splice_write": "true", 2024-09-21T10:40:03.482 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-21T10:40:03.482 INFO:teuthology.orchestra.run.smithi070.stdout: "fuse_use_invalidate_cb": "true", 2024-09-21T10:40:03.482 INFO:teuthology.orchestra.run.smithi070.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-21T10:40:03.482 INFO:teuthology.orchestra.run.smithi070.stdout: "gss_target_name": "ceph", 2024-09-21T10:40:03.483 INFO:teuthology.orchestra.run.smithi070.stdout: "heartbeat_file": "", 2024-09-21T10:40:03.483 INFO:teuthology.orchestra.run.smithi070.stdout: "heartbeat_inject_failure": "0", 2024-09-21T10:40:03.483 INFO:teuthology.orchestra.run.smithi070.stdout: "heartbeat_interval": "5", 2024-09-21T10:40:03.483 INFO:teuthology.orchestra.run.smithi070.stdout: "host": "smithi070", 2024-09-21T10:40:03.483 INFO:teuthology.orchestra.run.smithi070.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-21T10:40:03.483 INFO:teuthology.orchestra.run.smithi070.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-21T10:40:03.483 INFO:teuthology.orchestra.run.smithi070.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-21T10:40:03.483 INFO:teuthology.orchestra.run.smithi070.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-21T10:40:03.483 INFO:teuthology.orchestra.run.smithi070.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-21T10:40:03.483 INFO:teuthology.orchestra.run.smithi070.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-21T10:40:03.483 INFO:teuthology.orchestra.run.smithi070.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-21T10:40:03.483 INFO:teuthology.orchestra.run.smithi070.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-21T10:40:03.483 INFO:teuthology.orchestra.run.smithi070.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-21T10:40:03.483 INFO:teuthology.orchestra.run.smithi070.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-21T10:40:03.484 INFO:teuthology.orchestra.run.smithi070.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-21T10:40:03.484 INFO:teuthology.orchestra.run.smithi070.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-21T10:40:03.484 INFO:teuthology.orchestra.run.smithi070.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-21T10:40:03.484 INFO:teuthology.orchestra.run.smithi070.stdout: "inject_early_sigterm": "false", 2024-09-21T10:40:03.484 INFO:teuthology.orchestra.run.smithi070.stdout: "jaeger_agent_port": "6799", 2024-09-21T10:40:03.484 INFO:teuthology.orchestra.run.smithi070.stdout: "jaeger_tracing_enable": "false", 2024-09-21T10:40:03.484 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_aio": "true", 2024-09-21T10:40:03.484 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_align_min_size": "65536", 2024-09-21T10:40:03.484 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_block_align": "true", 2024-09-21T10:40:03.484 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_block_size": "4096", 2024-09-21T10:40:03.484 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_dio": "true", 2024-09-21T10:40:03.484 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_discard": "false", 2024-09-21T10:40:03.484 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_force_aio": "false", 2024-09-21T10:40:03.484 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_ignore_corruption": "false", 2024-09-21T10:40:03.485 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_max_write_bytes": "10485760", 2024-09-21T10:40:03.485 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_max_write_entries": "100", 2024-09-21T10:40:03.485 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_replay_from": "0", 2024-09-21T10:40:03.485 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-21T10:40:03.485 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-21T10:40:03.485 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-21T10:40:03.485 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-21T10:40:03.485 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_write_header_frequency": "0", 2024-09-21T10:40:03.485 INFO:teuthology.orchestra.run.smithi070.stdout: "journal_zero_on_create": "false", 2024-09-21T10:40:03.485 INFO:teuthology.orchestra.run.smithi070.stdout: "journaler_prefetch_periods": "10", 2024-09-21T10:40:03.485 INFO:teuthology.orchestra.run.smithi070.stdout: "journaler_prezero_periods": "5", 2024-09-21T10:40:03.485 INFO:teuthology.orchestra.run.smithi070.stdout: "journaler_write_head_interval": "15", 2024-09-21T10:40:03.485 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "", 2024-09-21T10:40:03.486 INFO:teuthology.orchestra.run.smithi070.stdout: "keyfile": "", 2024-09-21T10:40:03.486 INFO:teuthology.orchestra.run.smithi070.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-21T10:40:03.486 INFO:teuthology.orchestra.run.smithi070.stdout: "kstore_backend": "rocksdb", 2024-09-21T10:40:03.486 INFO:teuthology.orchestra.run.smithi070.stdout: "kstore_default_stripe_size": "65536", 2024-09-21T10:40:03.486 INFO:teuthology.orchestra.run.smithi070.stdout: "kstore_fsck_on_mount": "false", 2024-09-21T10:40:03.486 INFO:teuthology.orchestra.run.smithi070.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-21T10:40:03.486 INFO:teuthology.orchestra.run.smithi070.stdout: "kstore_max_bytes": "67108864", 2024-09-21T10:40:03.486 INFO:teuthology.orchestra.run.smithi070.stdout: "kstore_max_ops": "512", 2024-09-21T10:40:03.486 INFO:teuthology.orchestra.run.smithi070.stdout: "kstore_nid_prealloc": "1024", 2024-09-21T10:40:03.486 INFO:teuthology.orchestra.run.smithi070.stdout: "kstore_onode_map_size": "1024", 2024-09-21T10:40:03.486 INFO:teuthology.orchestra.run.smithi070.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-21T10:40:03.486 INFO:teuthology.orchestra.run.smithi070.stdout: "kstore_sync_submit_transaction": "false", 2024-09-21T10:40:03.486 INFO:teuthology.orchestra.run.smithi070.stdout: "kstore_sync_transaction": "false", 2024-09-21T10:40:03.486 INFO:teuthology.orchestra.run.smithi070.stdout: "librados_thread_count": "2", 2024-09-21T10:40:03.487 INFO:teuthology.orchestra.run.smithi070.stdout: "lockdep": "false", 2024-09-21T10:40:03.487 INFO:teuthology.orchestra.run.smithi070.stdout: "lockdep_force_backtrace": "false", 2024-09-21T10:40:03.487 INFO:teuthology.orchestra.run.smithi070.stdout: "log_coarse_timestamps": "true", 2024-09-21T10:40:03.487 INFO:teuthology.orchestra.run.smithi070.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-21T10:40:03.487 INFO:teuthology.orchestra.run.smithi070.stdout: "log_flush_on_exit": "false", 2024-09-21T10:40:03.487 INFO:teuthology.orchestra.run.smithi070.stdout: "log_graylog_host": "127.0.0.1", 2024-09-21T10:40:03.487 INFO:teuthology.orchestra.run.smithi070.stdout: "log_graylog_port": "12201", 2024-09-21T10:40:03.487 INFO:teuthology.orchestra.run.smithi070.stdout: "log_max_new": "1000", 2024-09-21T10:40:03.487 INFO:teuthology.orchestra.run.smithi070.stdout: "log_max_recent": "10000", 2024-09-21T10:40:03.487 INFO:teuthology.orchestra.run.smithi070.stdout: "log_stderr_prefix": "", 2024-09-21T10:40:03.487 INFO:teuthology.orchestra.run.smithi070.stdout: "log_stop_at_utilization": "0.970000", 2024-09-21T10:40:03.487 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": "true", 2024-09-21T10:40:03.487 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_graylog": "false", 2024-09-21T10:40:03.487 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_journald": "false", 2024-09-21T10:40:03.488 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_stderr": "false", 2024-09-21T10:40:03.488 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_syslog": "false", 2024-09-21T10:40:03.488 INFO:teuthology.orchestra.run.smithi070.stdout: "max_rotating_auth_attempts": "10", 2024-09-21T10:40:03.488 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-21T10:40:03.488 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_action_on_write_error": "1", 2024-09-21T10:40:03.488 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_alternate_name_max": "8192", 2024-09-21T10:40:03.488 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_asio_thread_count": "2", 2024-09-21T10:40:03.488 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_export_pin": "true", 2024-09-21T10:40:03.488 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_fragment_dirs": "true", 2024-09-21T10:40:03.488 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-21T10:40:03.488 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_fragment_interval": "5", 2024-09-21T10:40:03.488 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-21T10:40:03.488 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-21T10:40:03.488 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_interval": "10", 2024-09-21T10:40:03.489 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_max": "-1", 2024-09-21T10:40:03.489 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_max_until": "-1", 2024-09-21T10:40:03.489 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_merge_size": "50", 2024-09-21T10:40:03.489 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_midchunk": "0.300000", 2024-09-21T10:40:03.489 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-21T10:40:03.489 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_min_start": "0.200000", 2024-09-21T10:40:03.489 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_minchunk": "0.001000", 2024-09-21T10:40:03.489 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_mode": "0", 2024-09-21T10:40:03.489 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_need_max": "1.200000", 2024-09-21T10:40:03.489 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_need_min": "0.800000", 2024-09-21T10:40:03.489 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_overload_epochs": "2", 2024-09-21T10:40:03.489 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-21T10:40:03.489 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-21T10:40:03.489 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_split_bits": "3", 2024-09-21T10:40:03.490 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-21T10:40:03.490 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_split_size": "10000", 2024-09-21T10:40:03.490 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-21T10:40:03.490 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_target_decay": "10.000000", 2024-09-21T10:40:03.490 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-21T10:40:03.490 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_beacon_grace": "15.000000", 2024-09-21T10:40:03.490 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_beacon_interval": "4.000000", 2024-09-21T10:40:03.490 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-21T10:40:03.490 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-21T10:40:03.490 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache_mid": "0.700000", 2024-09-21T10:40:03.490 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-21T10:40:03.490 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache_quiesce_delay": "0", 2024-09-21T10:40:03.491 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-21T10:40:03.491 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-21T10:40:03.491 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-21T10:40:03.491 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache_release_free_interval": "10", 2024-09-21T10:40:03.491 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache_reservation": "0.050000", 2024-09-21T10:40:03.491 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-21T10:40:03.491 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache_trim_interval": "1", 2024-09-21T10:40:03.491 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache_trim_threshold": "262144", 2024-09-21T10:40:03.491 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-21T10:40:03.491 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-21T10:40:03.491 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-21T10:40:03.491 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_client_prealloc_inos": "1000", 2024-09-21T10:40:03.491 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-21T10:40:03.492 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_connect_bootstrapping": "false", 2024-09-21T10:40:03.492 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_damage_table_max_entries": "10000", 2024-09-21T10:40:03.492 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-21T10:40:03.492 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_debug_auth_pins": "false", 2024-09-21T10:40:03.492 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_debug_frag": "false", 2024-09-21T10:40:03.492 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_debug_scatterstat": "false", 2024-09-21T10:40:03.492 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_debug_subtrees": "false", 2024-09-21T10:40:03.492 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_decay_halflife": "5.000000", 2024-09-21T10:40:03.492 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_default_dir_hash": "2", 2024-09-21T10:40:03.492 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_defer_session_stale": "true", 2024-09-21T10:40:03.492 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_deny_all_reconnect": "false", 2024-09-21T10:40:03.492 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_dir_keys_per_op": "16384", 2024-09-21T10:40:03.492 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_dir_max_commit_size": "10", 2024-09-21T10:40:03.492 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_dir_max_entries": "0", 2024-09-21T10:40:03.492 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_dir_prefetch": "true", 2024-09-21T10:40:03.493 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-21T10:40:03.493 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-21T10:40:03.493 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_dump_cache_on_map": "false", 2024-09-21T10:40:03.493 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-21T10:40:03.493 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-21T10:40:03.493 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_early_reply": "true", 2024-09-21T10:40:03.493 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_enable_op_tracker": "true", 2024-09-21T10:40:03.493 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_enforce_unique_name": "true", 2024-09-21T10:40:03.493 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-21T10:40:03.493 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-21T10:40:03.493 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_export_ephemeral_random": "true", 2024-09-21T10:40:03.493 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-21T10:40:03.493 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-21T10:40:03.493 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-21T10:40:03.493 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-21T10:40:03.494 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-21T10:40:03.494 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-21T10:40:03.494 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-21T10:40:03.494 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-21T10:40:03.494 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_health_summarize_threshold": "10", 2024-09-21T10:40:03.494 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-21T10:40:03.494 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-21T10:40:03.494 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_inject_health_dummy": "false", 2024-09-21T10:40:03.494 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-21T10:40:03.494 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_inject_migrator_session_race": "false", 2024-09-21T10:40:03.494 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-21T10:40:03.494 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-21T10:40:03.495 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-21T10:40:03.495 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_join_fs": "", 2024-09-21T10:40:03.495 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_journal_format": "1", 2024-09-21T10:40:03.495 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-21T10:40:03.495 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_kill_create_at": "0", 2024-09-21T10:40:03.495 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_kill_dirfrag_at": "0", 2024-09-21T10:40:03.495 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_kill_export_at": "0", 2024-09-21T10:40:03.495 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_kill_import_at": "0", 2024-09-21T10:40:03.495 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_kill_journal_at": "0", 2024-09-21T10:40:03.495 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_kill_journal_expire_at": "0", 2024-09-21T10:40:03.495 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_kill_journal_replay_at": "0", 2024-09-21T10:40:03.495 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_kill_link_at": "0", 2024-09-21T10:40:03.495 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_kill_mdstable_at": "0", 2024-09-21T10:40:03.496 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_kill_openc_at": "0", 2024-09-21T10:40:03.496 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_kill_rename_at": "0", 2024-09-21T10:40:03.496 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_kill_shutdown_at": "0", 2024-09-21T10:40:03.496 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log_event_large_threshold": "524288", 2024-09-21T10:40:03.496 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log_events_per_segment": "1024", 2024-09-21T10:40:03.496 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-21T10:40:03.496 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log_max_events": "-1", 2024-09-21T10:40:03.496 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log_max_segments": "128", 2024-09-21T10:40:03.496 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log_pause": "false", 2024-09-21T10:40:03.496 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log_segment_size": "0", 2024-09-21T10:40:03.496 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-21T10:40:03.496 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-21T10:40:03.496 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-21T10:40:03.496 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log_trim_threshold": "128", 2024-09-21T10:40:03.497 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-21T10:40:03.497 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log_warn_factor": "2.000000", 2024-09-21T10:40:03.497 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_max_caps_per_client": "1048576", 2024-09-21T10:40:03.497 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_max_completed_flushes": "100000", 2024-09-21T10:40:03.497 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_max_completed_requests": "100000", 2024-09-21T10:40:03.497 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_max_export_size": "20971520", 2024-09-21T10:40:03.497 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_max_file_recover": "32", 2024-09-21T10:40:03.497 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_max_purge_files": "64", 2024-09-21T10:40:03.497 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_max_purge_ops": "8192", 2024-09-21T10:40:03.497 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-21T10:40:03.497 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-21T10:40:03.497 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_max_snaps_per_dir": "100", 2024-09-21T10:40:03.497 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_metrics_update_interval": "2", 2024-09-21T10:40:03.497 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_min_caps_per_client": "100", 2024-09-21T10:40:03.498 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_min_caps_working_set": "10000", 2024-09-21T10:40:03.498 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-21T10:40:03.498 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_numa_node": "-1", 2024-09-21T10:40:03.498 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-21T10:40:03.498 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_op_complaint_time": "30.000000", 2024-09-21T10:40:03.498 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_op_history_duration": "600", 2024-09-21T10:40:03.498 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_op_history_size": "20", 2024-09-21T10:40:03.498 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_op_history_slow_op_size": "20", 2024-09-21T10:40:03.498 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-21T10:40:03.498 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_op_log_threshold": "5", 2024-09-21T10:40:03.498 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_ping_grace": "15", 2024-09-21T10:40:03.498 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_ping_interval": "5", 2024-09-21T10:40:03.498 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-21T10:40:03.499 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-21T10:40:03.499 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_recall_max_caps": "30000", 2024-09-21T10:40:03.499 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-21T10:40:03.499 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-21T10:40:03.499 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-21T10:40:03.499 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_recall_warning_threshold": "262144", 2024-09-21T10:40:03.499 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-21T10:40:03.499 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_replay_interval": "1.000000", 2024-09-21T10:40:03.499 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-21T10:40:03.499 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-21T10:40:03.499 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_root_ino_gid": "0", 2024-09-21T10:40:03.499 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_root_ino_uid": "0", 2024-09-21T10:40:03.499 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-21T10:40:03.500 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_scrub_stats_review_period": "1", 2024-09-21T10:40:03.500 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-21T10:40:03.500 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-21T10:40:03.500 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-21T10:40:03.500 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-21T10:40:03.500 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-21T10:40:03.500 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-21T10:40:03.500 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-21T10:40:03.500 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-21T10:40:03.500 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-21T10:40:03.500 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-21T10:40:03.500 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-21T10:40:03.500 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_shutdown_check": "0", 2024-09-21T10:40:03.500 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_skip_ino": "0", 2024-09-21T10:40:03.501 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-21T10:40:03.501 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_snap_max_uid": "4294967294", 2024-09-21T10:40:03.501 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_snap_min_uid": "0", 2024-09-21T10:40:03.501 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_snap_rstat": "false", 2024-09-21T10:40:03.502 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_standby_replay_damaged": "false", 2024-09-21T10:40:03.502 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_symlink_recovery": "true", 2024-09-21T10:40:03.502 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-21T10:40:03.502 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_thrash_exports": "0", 2024-09-21T10:40:03.502 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_thrash_fragments": "0", 2024-09-21T10:40:03.502 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_tick_interval": "5.000000", 2024-09-21T10:40:03.502 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_valgrind_exit": "false", 2024-09-21T10:40:03.503 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_verify_backtrace": "1", 2024-09-21T10:40:03.503 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_verify_scatter": "false", 2024-09-21T10:40:03.503 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-21T10:40:03.503 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_wipe_sessions": "false", 2024-09-21T10:40:03.503 INFO:teuthology.orchestra.run.smithi070.stdout: "mempool_debug": "false", 2024-09-21T10:40:03.503 INFO:teuthology.orchestra.run.smithi070.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-21T10:40:03.503 INFO:teuthology.orchestra.run.smithi070.stdout: "memstore_device_bytes": "1073741824", 2024-09-21T10:40:03.503 INFO:teuthology.orchestra.run.smithi070.stdout: "memstore_page_set": "false", 2024-09-21T10:40:03.503 INFO:teuthology.orchestra.run.smithi070.stdout: "memstore_page_size": "65536", 2024-09-21T10:40:03.503 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_client_bytes": "134217728", 2024-09-21T10:40:03.503 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_client_messages": "512", 2024-09-21T10:40:03.503 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-21T10:40:03.503 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-21T10:40:03.503 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-21T10:40:03.504 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-21T10:40:03.504 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_disabled_modules": "", 2024-09-21T10:40:03.504 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_enable_op_tracker": "true", 2024-09-21T10:40:03.504 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-21T10:40:03.504 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_max_pg_creating": "1024", 2024-09-21T10:40:03.504 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_max_pg_num_change": "128", 2024-09-21T10:40:03.504 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_mds_bytes": "134217728", 2024-09-21T10:40:03.504 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_mds_messages": "128", 2024-09-21T10:40:03.504 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-21T10:40:03.504 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_mon_bytes": "134217728", 2024-09-21T10:40:03.504 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_mon_messages": "128", 2024-09-21T10:40:03.504 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-21T10:40:03.504 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-21T10:40:03.504 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_op_history_duration": "600", 2024-09-21T10:40:03.505 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_op_history_size": "20", 2024-09-21T10:40:03.505 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-21T10:40:03.505 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-21T10:40:03.505 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_op_log_threshold": "5", 2024-09-21T10:40:03.505 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_osd_bytes": "536870912", 2024-09-21T10:40:03.505 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_osd_messages": "8192", 2024-09-21T10:40:03.505 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_pool": "true", 2024-09-21T10:40:03.505 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-21T10:40:03.505 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_standby_modules": "true", 2024-09-21T10:40:03.505 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_stats_period": "5", 2024-09-21T10:40:03.505 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_stats_threshold": "5", 2024-09-21T10:40:03.505 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_tick_period": "2", 2024-09-21T10:40:03.506 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-21T10:40:03.506 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-21T10:40:03.506 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_allow_pool_delete": "true", 2024-09-21T10:40:03.506 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_allow_pool_size_one": "false", 2024-09-21T10:40:03.506 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_auth_validate_all_caps": "true", 2024-09-21T10:40:03.506 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-21T10:40:03.506 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-21T10:40:03.506 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_client_bytes": "104857600", 2024-09-21T10:40:03.506 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_client_directed_command_retry": "2", 2024-09-21T10:40:03.506 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-21T10:40:03.506 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-21T10:40:03.506 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-21T10:40:03.506 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-21T10:40:03.507 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_client_hunt_parallel": "3", 2024-09-21T10:40:03.507 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_client_log_interval": "1.000000", 2024-09-21T10:40:03.507 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-21T10:40:03.507 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_client_ping_interval": "10.000000", 2024-09-21T10:40:03.507 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-21T10:40:03.507 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_client_target_rank": "-1", 2024-09-21T10:40:03.507 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-21T10:40:03.507 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-21T10:40:03.507 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-21T10:40:03.507 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_cluster_log_level": "debug", 2024-09-21T10:40:03.507 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_cluster_log_to_file": "true", 2024-09-21T10:40:03.507 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-21T10:40:03.507 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-21T10:40:03.508 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-21T10:40:03.508 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_cluster_log_to_journald": "true", 2024-09-21T10:40:03.508 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-21T10:40:03.508 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-21T10:40:03.508 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-21T10:40:03.508 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_compact_on_bootstrap": "false", 2024-09-21T10:40:03.508 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_compact_on_start": "false", 2024-09-21T10:40:03.508 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_compact_on_trim": "true", 2024-09-21T10:40:03.508 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-21T10:40:03.508 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-21T10:40:03.508 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-21T10:40:03.508 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_cpu_threads": "4", 2024-09-21T10:40:03.508 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_crush_min_required_version": "hammer", 2024-09-21T10:40:03.508 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_daemon_bytes": "419430400", 2024-09-21T10:40:03.509 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-21T10:40:03.509 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_data_avail_crit": "5", 2024-09-21T10:40:03.509 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_data_avail_warn": "5", 2024-09-21T10:40:03.509 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_data_size_warn": "16106127360", 2024-09-21T10:40:03.509 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-21T10:40:03.509 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-21T10:40:03.509 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_debug_dump_json": "false", 2024-09-21T10:40:03.509 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-21T10:40:03.509 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_debug_dump_transactions": "false", 2024-09-21T10:40:03.509 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_debug_extra_checks": "false", 2024-09-21T10:40:03.509 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-21T10:40:03.509 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-21T10:40:03.509 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_debug_no_require_reef": "false", 2024-09-21T10:40:03.509 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_debug_no_require_squid": "false", 2024-09-21T10:40:03.509 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-21T10:40:03.510 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-21T10:40:03.510 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-21T10:40:03.510 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_down_mkfs_grace": "60", 2024-09-21T10:40:03.510 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_down_uptime_grace": "60", 2024-09-21T10:40:03.510 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_election_default_strategy": "1", 2024-09-21T10:40:03.510 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_election_timeout": "5.000000", 2024-09-21T10:40:03.510 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-21T10:40:03.510 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_elector_ping_divisor": "2", 2024-09-21T10:40:03.510 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-21T10:40:03.510 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_enable_op_tracker": "true", 2024-09-21T10:40:03.510 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_fake_pool_delete": "false", 2024-09-21T10:40:03.510 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_force_quorum_join": "false", 2024-09-21T10:40:03.510 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-21T10:40:03.511 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_globalid_prealloc": "10000", 2024-09-21T10:40:03.511 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_health_detail_to_clog": "true", 2024-09-21T10:40:03.511 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_health_log_update_period": "5", 2024-09-21T10:40:03.511 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_health_max_detail": "50", 2024-09-21T10:40:03.511 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_health_to_clog": "true", 2024-09-21T10:40:03.511 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_health_to_clog_interval": "600", 2024-09-21T10:40:03.511 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-21T10:40:03.511 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_host": "[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0]", 2024-09-21T10:40:03.511 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_host_override": "", 2024-09-21T10:40:03.511 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_initial_members": "", 2024-09-21T10:40:03.511 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-21T10:40:03.511 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-21T10:40:03.511 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-21T10:40:03.511 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-21T10:40:03.512 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-21T10:40:03.512 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_lease": "5.000000", 2024-09-21T10:40:03.512 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-21T10:40:03.512 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-21T10:40:03.512 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_log_full_interval": "50", 2024-09-21T10:40:03.512 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_log_max": "10000", 2024-09-21T10:40:03.512 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_log_max_summary": "50", 2024-09-21T10:40:03.512 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-21T10:40:03.512 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_max_log_epochs": "500", 2024-09-21T10:40:03.512 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-21T10:40:03.512 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-21T10:40:03.512 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-21T10:40:03.512 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_max_osd": "10000", 2024-09-21T10:40:03.512 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_max_pg_per_osd": "10000", 2024-09-21T10:40:03.513 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_max_pool_pg_num": "65536", 2024-09-21T10:40:03.513 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-21T10:40:03.513 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-21T10:40:03.513 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_mds_force_trim_to": "0", 2024-09-21T10:40:03.513 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_mds_skip_sanity": "true", 2024-09-21T10:40:03.513 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_memory_autotune": "true", 2024-09-21T10:40:03.513 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_memory_target": "2147483648", 2024-09-21T10:40:03.513 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_mgr_beacon_grace": "30", 2024-09-21T10:40:03.513 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-21T10:40:03.513 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_mgr_digest_period": "5", 2024-09-21T10:40:03.513 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_mgr_inactive_grace": "60", 2024-09-21T10:40:03.513 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-21T10:40:03.513 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-21T10:40:03.514 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_min_osdmap_epochs": "500", 2024-09-21T10:40:03.514 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-21T10:40:03.514 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-21T10:40:03.514 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_op_complaint_time": "30", 2024-09-21T10:40:03.514 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_op_history_duration": "600", 2024-09-21T10:40:03.514 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_op_history_size": "20", 2024-09-21T10:40:03.514 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_op_history_slow_op_size": "20", 2024-09-21T10:40:03.514 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-21T10:40:03.514 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_op_log_threshold": "5", 2024-09-21T10:40:03.514 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-21T10:40:03.514 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-21T10:40:03.514 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-21T10:40:03.514 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_auto_mark_in": "false", 2024-09-21T10:40:03.514 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-21T10:40:03.515 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-21T10:40:03.515 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-21T10:40:03.515 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_cache_size": "500", 2024-09-21T10:40:03.515 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-21T10:40:03.515 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-21T10:40:03.515 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-21T10:40:03.515 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_down_out_interval": "600", 2024-09-21T10:40:03.515 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-21T10:40:03.515 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-21T10:40:03.515 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_force_trim_to": "0", 2024-09-21T10:40:03.515 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-21T10:40:03.515 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-21T10:40:03.515 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-21T10:40:03.515 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-21T10:40:03.515 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-21T10:40:03.516 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-21T10:40:03.516 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-21T10:40:03.516 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_min_down_reporters": "2", 2024-09-21T10:40:03.516 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-21T10:40:03.516 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-21T10:40:03.516 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-21T10:40:03.516 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-21T10:40:03.516 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-21T10:40:03.516 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-21T10:40:03.516 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_report_timeout": "900", 2024-09-21T10:40:03.516 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-21T10:40:03.516 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-21T10:40:03.516 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-21T10:40:03.516 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-21T10:40:03.517 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-21T10:40:03.517 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-21T10:40:03.517 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-21T10:40:03.517 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-21T10:40:03.517 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-21T10:40:03.517 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_pg_stuck_threshold": "60", 2024-09-21T10:40:03.517 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-21T10:40:03.517 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-21T10:40:03.517 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-21T10:40:03.517 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-21T10:40:03.517 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-21T10:40:03.517 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-21T10:40:03.517 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_probe_timeout": "2.000000", 2024-09-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_reweight_max_change": "0.050000", 2024-09-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_reweight_max_osds": "4", 2024-09-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_scrub_interval": "86400", 2024-09-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_scrub_max_keys": "100", 2024-09-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_scrub_timeout": "300", 2024-09-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_session_timeout": "300", 2024-09-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_smart_report_timeout": "5", 2024-09-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_stat_smooth_intervals": "6", 2024-09-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-21T10:40:03.519 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_stretch_pool_min_size": "2", 2024-09-21T10:40:03.519 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_stretch_pool_size": "4", 2024-09-21T10:40:03.519 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-21T10:40:03.519 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-21T10:40:03.519 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_sync_debug": "false", 2024-09-21T10:40:03.519 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-21T10:40:03.519 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-21T10:40:03.519 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_sync_provider_kill_at": "0", 2024-09-21T10:40:03.519 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_sync_requester_kill_at": "0", 2024-09-21T10:40:03.519 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_sync_timeout": "60.000000", 2024-09-21T10:40:03.519 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_target_pg_per_osd": "100", 2024-09-21T10:40:03.519 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_tick_interval": "5", 2024-09-21T10:40:03.519 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_timecheck_interval": "300.000000", 2024-09-21T10:40:03.520 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-21T10:40:03.520 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_use_min_delay_socket": "false", 2024-09-21T10:40:03.520 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_older_version_delay": "604800", 2024-09-21T10:40:03.520 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-21T10:40:03.520 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-21T10:40:03.520 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-21T10:40:03.520 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-21T10:40:03.520 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-21T10:40:03.521 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-21T10:40:03.522 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-21T10:40:03.522 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_misplaced": "false", 2024-09-21T10:40:03.522 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-21T10:40:03.522 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_older_version": "true", 2024-09-21T10:40:03.522 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-21T10:40:03.522 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-21T10:40:03.522 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-21T10:40:03.522 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-21T10:40:03.522 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-21T10:40:03.522 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-21T10:40:03.522 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-21T10:40:03.522 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-21T10:40:03.522 INFO:teuthology.orchestra.run.smithi070.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-21T10:40:03.523 INFO:teuthology.orchestra.run.smithi070.stdout: "monmap": "", 2024-09-21T10:40:03.523 INFO:teuthology.orchestra.run.smithi070.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-21T10:40:03.523 INFO:teuthology.orchestra.run.smithi070.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-21T10:40:03.523 INFO:teuthology.orchestra.run.smithi070.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-21T10:40:03.523 INFO:teuthology.orchestra.run.smithi070.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-21T10:40:03.523 INFO:teuthology.orchestra.run.smithi070.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-21T10:40:03.523 INFO:teuthology.orchestra.run.smithi070.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-21T10:40:03.523 INFO:teuthology.orchestra.run.smithi070.stdout: "motr_tracing_enabled": "false", 2024-09-21T10:40:03.523 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_op_threads": "3", 2024-09-21T10:40:03.523 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-21T10:40:03.523 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_cm": "false", 2024-09-21T10:40:03.523 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_device_name": "", 2024-09-21T10:40:03.523 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_dscp": "96", 2024-09-21T10:40:03.524 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-21T10:40:03.524 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-21T10:40:03.524 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_local_gid": "", 2024-09-21T10:40:03.524 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-21T10:40:03.524 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_port_num": "1", 2024-09-21T10:40:03.524 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-21T10:40:03.524 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-21T10:40:03.524 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-21T10:40:03.524 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-21T10:40:03.524 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_sl": "3", 2024-09-21T10:40:03.524 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_support_srq": "true", 2024-09-21T10:40:03.524 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_rdma_type": "ib", 2024-09-21T10:40:03.524 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_async_reap_threshold": "5", 2024-09-21T10:40:03.524 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_bind_before_connect": "false", 2024-09-21T10:40:03.525 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_bind_ipv4": "true", 2024-09-21T10:40:03.525 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_bind_ipv6": "false", 2024-09-21T10:40:03.525 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_bind_msgr1": "true", 2024-09-21T10:40:03.525 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_bind_msgr2": "true", 2024-09-21T10:40:03.525 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_bind_port_max": "7568", 2024-09-21T10:40:03.525 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_bind_port_min": "6800", 2024-09-21T10:40:03.525 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-21T10:40:03.525 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_bind_retry_count": "3", 2024-09-21T10:40:03.525 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_bind_retry_delay": "5", 2024-09-21T10:40:03.525 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_blackhole_client": "false", 2024-09-21T10:40:03.525 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_blackhole_mds": "false", 2024-09-21T10:40:03.525 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_blackhole_mgr": "false", 2024-09-21T10:40:03.525 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_blackhole_mon": "false", 2024-09-21T10:40:03.525 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_blackhole_osd": "false", 2024-09-21T10:40:03.526 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_client_mode": "crc secure", 2024-09-21T10:40:03.526 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-21T10:40:03.526 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_cluster_mode": "crc secure", 2024-09-21T10:40:03.526 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_cluster_type": "", 2024-09-21T10:40:03.526 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_compress_secure": "false", 2024-09-21T10:40:03.526 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_connection_idle_timeout": "900", 2024-09-21T10:40:03.526 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_connection_ready_timeout": "10", 2024-09-21T10:40:03.526 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_crc_data": "true", 2024-09-21T10:40:03.526 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_crc_header": "true", 2024-09-21T10:40:03.526 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_die_on_bad_msg": "false", 2024-09-21T10:40:03.526 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_die_on_bug": "true", 2024-09-21T10:40:03.526 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_die_on_old_message": "true", 2024-09-21T10:40:03.526 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_die_on_skipped_message": "false", 2024-09-21T10:40:03.527 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-21T10:40:03.527 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-21T10:40:03.527 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_coremask": "0xF", 2024-09-21T10:40:03.527 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-21T10:40:03.527 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-21T10:40:03.527 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_enable_tso": "true", 2024-09-21T10:40:03.527 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-21T10:40:03.527 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-21T10:40:03.527 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_hugepages": "", 2024-09-21T10:40:03.527 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-21T10:40:03.527 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-21T10:40:03.527 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_lro": "true", 2024-09-21T10:40:03.528 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_memory_channel": "4", 2024-09-21T10:40:03.528 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-21T10:40:03.528 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_pmd": "", 2024-09-21T10:40:03.528 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_port_id": "0", 2024-09-21T10:40:03.528 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-21T10:40:03.528 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-21T10:40:03.528 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_dump_on_send": "false", 2024-09-21T10:40:03.528 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_initial_backoff": "0.200000", 2024-09-21T10:40:03.528 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_inject_delay_max": "1.000000", 2024-09-21T10:40:03.528 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-21T10:40:03.528 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_inject_delay_type": "", 2024-09-21T10:40:03.528 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-21T10:40:03.528 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_inject_network_congestion": "0", 2024-09-21T10:40:03.528 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_inject_socket_failures": "0", 2024-09-21T10:40:03.529 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_learn_addr_from_peer": "true", 2024-09-21T10:40:03.529 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_max_accept_failures": "4", 2024-09-21T10:40:03.529 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_max_backoff": "15.000000", 2024-09-21T10:40:03.529 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_mon_client_mode": "secure crc", 2024-09-21T10:40:03.529 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-21T10:40:03.529 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_mon_service_mode": "secure crc", 2024-09-21T10:40:03.529 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_osd_compress_min_size": "1024", 2024-09-21T10:40:03.529 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_osd_compress_mode": "none", 2024-09-21T10:40:03.529 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-21T10:40:03.529 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-21T10:40:03.529 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_pq_min_cost": "65536", 2024-09-21T10:40:03.529 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_public_type": "", 2024-09-21T10:40:03.529 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_service_mode": "crc secure", 2024-09-21T10:40:03.529 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_tcp_listen_backlog": "512", 2024-09-21T10:40:03.530 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_tcp_nodelay": "true", 2024-09-21T10:40:03.530 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-21T10:40:03.530 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_tcp_rcvbuf": "0", 2024-09-21T10:40:03.530 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-21T10:40:03.530 INFO:teuthology.orchestra.run.smithi070.stdout: "ms_type": "async+posix", 2024-09-21T10:40:03.530 INFO:teuthology.orchestra.run.smithi070.stdout: "no_config_file": "false", 2024-09-21T10:40:03.530 INFO:teuthology.orchestra.run.smithi070.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-21T10:40:03.530 INFO:teuthology.orchestra.run.smithi070.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-21T10:40:03.530 INFO:teuthology.orchestra.run.smithi070.stdout: "objecter_completion_locks_per_session": "32", 2024-09-21T10:40:03.530 INFO:teuthology.orchestra.run.smithi070.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-21T10:40:03.530 INFO:teuthology.orchestra.run.smithi070.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-21T10:40:03.530 INFO:teuthology.orchestra.run.smithi070.stdout: "objecter_inflight_ops": "1024", 2024-09-21T10:40:03.530 INFO:teuthology.orchestra.run.smithi070.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-21T10:40:03.531 INFO:teuthology.orchestra.run.smithi070.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-21T10:40:03.531 INFO:teuthology.orchestra.run.smithi070.stdout: "objecter_tick_interval": "5.000000", 2024-09-21T10:40:03.531 INFO:teuthology.orchestra.run.smithi070.stdout: "objecter_timeout": "10.000000", 2024-09-21T10:40:03.531 INFO:teuthology.orchestra.run.smithi070.stdout: "objectstore_blackhole": "false", 2024-09-21T10:40:03.531 INFO:teuthology.orchestra.run.smithi070.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-21T10:40:03.531 INFO:teuthology.orchestra.run.smithi070.stdout: "openssl_engine_opts": "", 2024-09-21T10:40:03.531 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_agent_delay_time": "5.000000", 2024-09-21T10:40:03.531 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_agent_hist_halflife": "1000", 2024-09-21T10:40:03.531 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_agent_max_low_ops": "2", 2024-09-21T10:40:03.531 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_agent_max_ops": "4", 2024-09-21T10:40:03.531 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-21T10:40:03.531 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-21T10:40:03.531 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_agent_slop": "0.020000", 2024-09-21T10:40:03.531 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-21T10:40:03.532 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-21T10:40:03.532 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_asio_thread_count": "2", 2024-09-21T10:40:03.532 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_async_recovery_min_cost": "100", 2024-09-21T10:40:03.532 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-21T10:40:03.532 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-21T10:40:03.532 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_backfill_scan_max": "512", 2024-09-21T10:40:03.532 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_backfill_scan_min": "64", 2024-09-21T10:40:03.532 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_backoff_on_degraded": "false", 2024-09-21T10:40:03.532 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_backoff_on_peering": "false", 2024-09-21T10:40:03.532 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_backoff_on_unfound": "true", 2024-09-21T10:40:03.532 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_beacon_report_interval": "300", 2024-09-21T10:40:03.532 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_bench_duration": "30", 2024-09-21T10:40:03.532 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-21T10:40:03.532 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_bench_max_block_size": "67108864", 2024-09-21T10:40:03.532 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-21T10:40:03.533 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_blkin_trace_all": "false", 2024-09-21T10:40:03.533 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-21T10:40:03.533 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-21T10:40:03.533 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-21T10:40:03.533 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-21T10:40:03.533 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_check_for_log_corruption": "false", 2024-09-21T10:40:03.533 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-21T10:40:03.533 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-21T10:40:03.533 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-21T10:40:03.533 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-21T10:40:03.533 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_class_update_on_start": "true", 2024-09-21T10:40:03.533 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_client_message_cap": "256", 2024-09-21T10:40:03.534 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_client_message_size_cap": "524288000", 2024-09-21T10:40:03.534 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_client_op_priority": "63", 2024-09-21T10:40:03.534 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_client_watch_timeout": "30", 2024-09-21T10:40:03.534 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_command_max_records": "256", 2024-09-21T10:40:03.534 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_compact_on_start": "false", 2024-09-21T10:40:03.534 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-21T10:40:03.534 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-21T10:40:03.534 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-21T10:40:03.534 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_crush_update_on_start": "true", 2024-09-21T10:40:03.534 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_crush_update_weight_set": "true", 2024-09-21T10:40:03.534 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-21T10:40:03.534 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-21T10:40:03.534 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-21T10:40:03.534 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-21T10:40:03.535 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-21T10:40:03.535 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_feed_pullee": "-1", 2024-09-21T10:40:03.535 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-21T10:40:03.535 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-21T10:40:03.535 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-21T10:40:03.535 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_misdirected_ops": "false", 2024-09-21T10:40:03.535 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_no_acting_change": "false", 2024-09-21T10:40:03.535 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_no_purge_strays": "false", 2024-09-21T10:40:03.535 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_op_order": "false", 2024-09-21T10:40:03.535 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-21T10:40:03.535 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-21T10:40:03.535 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-21T10:40:03.535 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-21T10:40:03.536 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_shutdown": "false", 2024-09-21T10:40:03.536 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-21T10:40:03.536 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-21T10:40:03.536 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_trim_objects": "false", 2024-09-21T10:40:03.536 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-21T10:40:03.536 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-21T10:40:03.536 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_verify_snaps": "false", 2024-09-21T10:40:03.536 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-21T10:40:03.536 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-21T10:40:03.536 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_deep_scrub_keys": "1024", 2024-09-21T10:40:03.536 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-21T10:40:03.536 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-21T10:40:03.536 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-21T10:40:03.537 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_deep_scrub_stride": "524288", 2024-09-21T10:40:03.537 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-21T10:40:03.537 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-21T10:40:03.537 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_default_notify_timeout": "30", 2024-09-21T10:40:03.537 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_delete_sleep": "0.000000", 2024-09-21T10:40:03.537 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-21T10:40:03.537 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-21T10:40:03.537 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-21T10:40:03.537 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_discard_disconnected_ops": "true", 2024-09-21T10:40:03.537 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_ec_partial_reads": "true", 2024-09-21T10:40:03.537 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_enable_op_tracker": "true", 2024-09-21T10:40:03.537 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-21T10:40:03.537 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-21T10:40:03.537 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-21T10:40:03.538 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-21T10:40:03.538 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_fast_info": "true", 2024-09-21T10:40:03.538 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_fast_shutdown": "true", 2024-09-21T10:40:03.538 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-21T10:40:03.538 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-21T10:40:03.538 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-21T10:40:03.538 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-21T10:40:03.538 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-21T10:40:03.538 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_function_tracing": "false", 2024-09-21T10:40:03.538 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_heartbeat_grace": "20", 2024-09-21T10:40:03.538 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_heartbeat_interval": "6", 2024-09-21T10:40:03.538 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-21T10:40:03.539 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_heartbeat_min_peers": "10", 2024-09-21T10:40:03.539 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_heartbeat_min_size": "2000", 2024-09-21T10:40:03.539 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_heartbeat_stale": "600", 2024-09-21T10:40:03.539 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-21T10:40:03.539 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_hit_set_max_size": "100000", 2024-09-21T10:40:03.539 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_hit_set_min_size": "1000", 2024-09-21T10:40:03.539 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-21T10:40:03.539 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-21T10:40:03.539 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-21T10:40:03.539 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-21T10:40:03.539 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-21T10:40:03.539 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-21T10:40:03.539 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_journal_size": "5120", 2024-09-21T10:40:03.539 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_kill_backfill_at": "0", 2024-09-21T10:40:03.540 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-21T10:40:03.540 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_map_cache_size": "50", 2024-09-21T10:40:03.540 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_map_dedup": "true", 2024-09-21T10:40:03.540 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_map_message_max": "40", 2024-09-21T10:40:03.540 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-21T10:40:03.540 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_map_share_max_epochs": "40", 2024-09-21T10:40:03.540 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_attr_name_len": "100", 2024-09-21T10:40:03.540 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_attr_size": "0", 2024-09-21T10:40:03.540 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_backfills": "1", 2024-09-21T10:40:03.540 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_markdown_count": "5", 2024-09-21T10:40:03.540 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_markdown_period": "600", 2024-09-21T10:40:03.540 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_object_name_len": "2048", 2024-09-21T10:40:03.540 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_object_namespace_len": "256", 2024-09-21T10:40:03.540 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_object_size": "134217728", 2024-09-21T10:40:03.541 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-21T10:40:03.541 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-21T10:40:03.541 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_pg_blocked_by": "16", 2024-09-21T10:40:03.541 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_pg_log_entries": "10000", 2024-09-21T10:40:03.541 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-21T10:40:03.541 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_pgls": "1024", 2024-09-21T10:40:03.541 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_push_cost": "8388608", 2024-09-21T10:40:03.541 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_push_objects": "10", 2024-09-21T10:40:03.541 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_scrubs": "3", 2024-09-21T10:40:03.541 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-21T10:40:03.541 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_trimming_pgs": "2", 2024-09-21T10:40:03.541 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_write_op_reply_len": "64", 2024-09-21T10:40:03.541 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_max_write_size": "90", 2024-09-21T10:40:03.541 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-21T10:40:03.542 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-21T10:40:03.542 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-21T10:40:03.542 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-21T10:40:03.542 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-21T10:40:03.542 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-21T10:40:03.542 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-21T10:40:03.542 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-21T10:40:03.542 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_profile": "balanced", 2024-09-21T10:40:03.542 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-21T10:40:03.542 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-21T10:40:03.542 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-21T10:40:03.542 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-21T10:40:03.542 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-21T10:40:03.543 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-21T10:40:03.543 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-21T10:40:03.543 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-21T10:40:03.543 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-21T10:40:03.543 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-21T10:40:03.543 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-21T10:40:03.543 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_memory_base": "805306368", 2024-09-21T10:40:03.543 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_memory_cache_min": "134217728", 2024-09-21T10:40:03.543 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-21T10:40:03.543 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-21T10:40:03.543 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_memory_target": "4294967296", 2024-09-21T10:40:03.543 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_memory_target_autotune": "false", 2024-09-21T10:40:03.543 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-21T10:40:03.544 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_min_pg_log_entries": "250", 2024-09-21T10:40:03.544 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_min_recovery_priority": "0", 2024-09-21T10:40:03.544 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-21T10:40:03.544 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-21T10:40:03.544 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mon_report_interval": "5", 2024-09-21T10:40:03.544 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-21T10:40:03.544 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-21T10:40:03.544 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_num_cache_shards": "32", 2024-09-21T10:40:03.544 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_num_op_tracker_shard": "32", 2024-09-21T10:40:03.544 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_numa_auto_affinity": "true", 2024-09-21T10:40:03.544 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_numa_node": "-1", 2024-09-21T10:40:03.544 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_numa_prefer_iface": "true", 2024-09-21T10:40:03.544 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-21T10:40:03.544 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_objecter_finishers": "1", 2024-09-21T10:40:03.545 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_objectstore": "bluestore", 2024-09-21T10:40:03.545 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_objectstore_fuse": "false", 2024-09-21T10:40:03.545 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-21T10:40:03.545 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_objectstore_tracing": "false", 2024-09-21T10:40:03.545 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_complaint_time": "30.000000", 2024-09-21T10:40:03.545 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_history_duration": "600", 2024-09-21T10:40:03.545 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_history_size": "20", 2024-09-21T10:40:03.545 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_history_slow_op_size": "20", 2024-09-21T10:40:03.545 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-21T10:40:03.545 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_log_threshold": "5", 2024-09-21T10:40:03.545 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_num_shards": "0", 2024-09-21T10:40:03.545 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_num_shards_hdd": "5", 2024-09-21T10:40:03.545 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_num_shards_ssd": "8", 2024-09-21T10:40:03.545 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-21T10:40:03.546 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-21T10:40:03.546 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-21T10:40:03.546 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-21T10:40:03.546 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_pq_min_cost": "65536", 2024-09-21T10:40:03.546 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-21T10:40:03.546 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_queue_cut_off": "high", 2024-09-21T10:40:03.546 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-21T10:40:03.546 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_op_thread_timeout": "15", 2024-09-21T10:40:03.546 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_open_classes_on_start": "true", 2024-09-21T10:40:03.546 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_os_flags": "0", 2024-09-21T10:40:03.546 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_peering_op_priority": "255", 2024-09-21T10:40:03.546 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pg_delete_cost": "1048576", 2024-09-21T10:40:03.546 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pg_delete_priority": "5", 2024-09-21T10:40:03.547 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-21T10:40:03.547 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-21T10:40:03.547 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-21T10:40:03.547 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pg_log_trim_max": "10000", 2024-09-21T10:40:03.547 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pg_log_trim_min": "100", 2024-09-21T10:40:03.547 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-21T10:40:03.547 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-21T10:40:03.547 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-21T10:40:03.547 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-21T10:40:03.547 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-21T10:40:03.547 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-21T10:40:03.547 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-21T10:40:03.547 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-21T10:40:03.548 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-21T10:40:03.548 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-21T10:40:03.548 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_crimson": "false", 2024-09-21T10:40:03.548 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-21T10:40:03.548 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-21T10:40:03.548 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-21T10:40:03.548 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-21T10:40:03.548 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-21T10:40:03.548 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-21T10:40:03.548 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-21T10:40:03.548 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-21T10:40:03.548 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_flags": "0", 2024-09-21T10:40:03.548 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-21T10:40:03.548 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_min_size": "0", 2024-09-21T10:40:03.549 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-21T10:40:03.549 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_pg_num": "32", 2024-09-21T10:40:03.549 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_pgp_num": "0", 2024-09-21T10:40:03.549 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-21T10:40:03.549 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_read_ratio": "70", 2024-09-21T10:40:03.549 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_size": "3", 2024-09-21T10:40:03.549 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_default_type": "replicated", 2024-09-21T10:40:03.549 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-21T10:40:03.549 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-21T10:40:03.549 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_push_per_object_cost": "1000", 2024-09-21T10:40:03.549 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-21T10:40:03.549 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recover_clone_overlap": "true", 2024-09-21T10:40:03.549 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-21T10:40:03.549 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_cost": "20971520", 2024-09-21T10:40:03.550 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-21T10:40:03.550 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_max_active": "0", 2024-09-21T10:40:03.550 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-21T10:40:03.550 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-21T10:40:03.550 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-21T10:40:03.550 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-21T10:40:03.550 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_max_single_start": "1", 2024-09-21T10:40:03.550 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_op_priority": "3", 2024-09-21T10:40:03.550 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-21T10:40:03.550 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_priority": "5", 2024-09-21T10:40:03.550 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-21T10:40:03.550 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_sleep": "0.000000", 2024-09-21T10:40:03.550 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-21T10:40:03.551 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-21T10:40:03.551 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-21T10:40:03.551 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_repair_during_recovery": "false", 2024-09-21T10:40:03.551 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_requested_scrub_priority": "5", 2024-09-21T10:40:03.551 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-21T10:40:03.551 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-21T10:40:03.551 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_auto_repair": "false", 2024-09-21T10:40:03.551 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-21T10:40:03.551 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-21T10:40:03.551 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_begin_hour": "0", 2024-09-21T10:40:03.551 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_begin_week_day": "0", 2024-09-21T10:40:03.551 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_chunk_max": "15", 2024-09-21T10:40:03.551 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_chunk_min": "5", 2024-09-21T10:40:03.551 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_cost": "52428800", 2024-09-21T10:40:03.552 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-21T10:40:03.552 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_during_recovery": "false", 2024-09-21T10:40:03.552 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_end_hour": "0", 2024-09-21T10:40:03.552 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_end_week_day": "0", 2024-09-21T10:40:03.552 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_event_cost": "4096", 2024-09-21T10:40:03.552 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-21T10:40:03.552 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-21T10:40:03.552 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_invalid_stats": "true", 2024-09-21T10:40:03.552 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-21T10:40:03.552 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-21T10:40:03.552 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_max_preemptions": "5", 2024-09-21T10:40:03.552 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-21T10:40:03.552 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_priority": "5", 2024-09-21T10:40:03.552 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_retry_after_noscrub": "60", 2024-09-21T10:40:03.553 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_retry_delay": "30", 2024-09-21T10:40:03.553 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_retry_new_interval": "10", 2024-09-21T10:40:03.553 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_retry_pg_state": "60", 2024-09-21T10:40:03.553 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_retry_trimming": "10", 2024-09-21T10:40:03.553 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_scrub_sleep": "0.000000", 2024-09-21T10:40:03.553 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-21T10:40:03.553 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-21T10:40:03.553 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-21T10:40:03.553 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-21T10:40:03.553 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_skip_data_digest": "false", 2024-09-21T10:40:03.553 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_smart_report_timeout": "5", 2024-09-21T10:40:03.553 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_snap_trim_cost": "1048576", 2024-09-21T10:40:03.553 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_snap_trim_priority": "5", 2024-09-21T10:40:03.554 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-21T10:40:03.554 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-21T10:40:03.554 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-21T10:40:03.554 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-21T10:40:03.554 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-21T10:40:03.554 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-21T10:40:03.554 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-21T10:40:03.554 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_target_transaction_size": "30", 2024-09-21T10:40:03.554 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-21T10:40:03.554 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-21T10:40:03.554 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-21T10:40:03.554 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-21T10:40:03.554 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-21T10:40:03.555 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-21T10:40:03.555 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-21T10:40:03.555 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-21T10:40:03.555 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-21T10:40:03.555 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-21T10:40:03.555 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_tracing": "false", 2024-09-21T10:40:03.555 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_use_stale_snap": "false", 2024-09-21T10:40:03.555 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-21T10:40:03.555 INFO:teuthology.orchestra.run.smithi070.stdout: "osdc_blkin_trace_all": "false", 2024-09-21T10:40:03.555 INFO:teuthology.orchestra.run.smithi070.stdout: "paxos_kill_at": "0", 2024-09-21T10:40:03.555 INFO:teuthology.orchestra.run.smithi070.stdout: "paxos_max_join_drift": "10", 2024-09-21T10:40:03.555 INFO:teuthology.orchestra.run.smithi070.stdout: "paxos_min": "500", 2024-09-21T10:40:03.555 INFO:teuthology.orchestra.run.smithi070.stdout: "paxos_min_wait": "0.050000", 2024-09-21T10:40:03.555 INFO:teuthology.orchestra.run.smithi070.stdout: "paxos_propose_interval": "1.000000", 2024-09-21T10:40:03.556 INFO:teuthology.orchestra.run.smithi070.stdout: "paxos_service_trim_max": "500", 2024-09-21T10:40:03.556 INFO:teuthology.orchestra.run.smithi070.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-21T10:40:03.556 INFO:teuthology.orchestra.run.smithi070.stdout: "paxos_service_trim_min": "250", 2024-09-21T10:40:03.556 INFO:teuthology.orchestra.run.smithi070.stdout: "paxos_stash_full_interval": "25", 2024-09-21T10:40:03.556 INFO:teuthology.orchestra.run.smithi070.stdout: "paxos_trim_max": "500", 2024-09-21T10:40:03.556 INFO:teuthology.orchestra.run.smithi070.stdout: "paxos_trim_min": "250", 2024-09-21T10:40:03.556 INFO:teuthology.orchestra.run.smithi070.stdout: "perf": "true", 2024-09-21T10:40:03.556 INFO:teuthology.orchestra.run.smithi070.stdout: "pid_file": "", 2024-09-21T10:40:03.556 INFO:teuthology.orchestra.run.smithi070.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-21T10:40:03.556 INFO:teuthology.orchestra.run.smithi070.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-21T10:40:03.556 INFO:teuthology.orchestra.run.smithi070.stdout: "public_addr": "-", 2024-09-21T10:40:03.556 INFO:teuthology.orchestra.run.smithi070.stdout: "public_addrv": "", 2024-09-21T10:40:03.556 INFO:teuthology.orchestra.run.smithi070.stdout: "public_bind_addr": "-", 2024-09-21T10:40:03.556 INFO:teuthology.orchestra.run.smithi070.stdout: "public_network": "172.21.0.0/20", 2024-09-21T10:40:03.557 INFO:teuthology.orchestra.run.smithi070.stdout: "public_network_interface": "", 2024-09-21T10:40:03.557 INFO:teuthology.orchestra.run.smithi070.stdout: "qat_compressor_busy_polling": "false", 2024-09-21T10:40:03.557 INFO:teuthology.orchestra.run.smithi070.stdout: "qat_compressor_enabled": "false", 2024-09-21T10:40:03.557 INFO:teuthology.orchestra.run.smithi070.stdout: "qat_compressor_session_max_number": "256", 2024-09-21T10:40:03.557 INFO:teuthology.orchestra.run.smithi070.stdout: "rados_mon_op_timeout": "0", 2024-09-21T10:40:03.557 INFO:teuthology.orchestra.run.smithi070.stdout: "rados_osd_op_timeout": "0", 2024-09-21T10:40:03.557 INFO:teuthology.orchestra.run.smithi070.stdout: "rados_tracing": "false", 2024-09-21T10:40:03.557 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_atime_update_interval": "60", 2024-09-21T10:40:03.557 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-21T10:40:03.557 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_balance_parent_reads": "false", 2024-09-21T10:40:03.557 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_balance_snap_reads": "false", 2024-09-21T10:40:03.557 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_blkin_trace_all": "false", 2024-09-21T10:40:03.557 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-21T10:40:03.557 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-21T10:40:03.558 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_cache": "true", 2024-09-21T10:40:03.558 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-21T10:40:03.558 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-21T10:40:03.558 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-21T10:40:03.558 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-21T10:40:03.558 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_cache_policy": "writearound", 2024-09-21T10:40:03.558 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_cache_size": "33554432", 2024-09-21T10:40:03.558 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-21T10:40:03.558 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-21T10:40:03.558 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_clone_copy_on_read": "false", 2024-09-21T10:40:03.558 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_compression_hint": "none", 2024-09-21T10:40:03.558 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_concurrent_management_ops": "10", 2024-09-21T10:40:03.558 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_config_pool_override_update_timestamp": "1726913621", 2024-09-21T10:40:03.559 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_default_clone_format": "auto", 2024-09-21T10:40:03.559 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_default_data_pool": "", 2024-09-21T10:40:03.559 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_default_features": "61", 2024-09-21T10:40:03.559 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_default_format": "2", 2024-09-21T10:40:03.559 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_default_map_options": "", 2024-09-21T10:40:03.559 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_default_order": "22", 2024-09-21T10:40:03.559 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_default_pool": "rbd", 2024-09-21T10:40:03.559 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-21T10:40:03.559 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_default_stripe_count": "0", 2024-09-21T10:40:03.559 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_default_stripe_unit": "0", 2024-09-21T10:40:03.559 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-21T10:40:03.559 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-21T10:40:03.559 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-21T10:40:03.559 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_enable_alloc_hint": "true", 2024-09-21T10:40:03.560 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-21T10:40:03.560 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_io_scheduler": "simple", 2024-09-21T10:40:03.560 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-21T10:40:03.560 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-21T10:40:03.560 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-21T10:40:03.560 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-21T10:40:03.560 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-21T10:40:03.560 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-21T10:40:03.560 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-21T10:40:03.560 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-21T10:40:03.560 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-21T10:40:03.560 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_journal_order": "24", 2024-09-21T10:40:03.560 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_journal_pool": "", 2024-09-21T10:40:03.560 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_journal_splay_width": "4", 2024-09-21T10:40:03.560 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_localize_parent_reads": "false", 2024-09-21T10:40:03.561 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_localize_snap_reads": "false", 2024-09-21T10:40:03.561 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-21T10:40:03.561 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-21T10:40:03.561 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-21T10:40:03.561 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-21T10:40:03.561 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-21T10:40:03.561 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-21T10:40:03.561 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-21T10:40:03.561 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-21T10:40:03.561 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-21T10:40:03.561 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-21T10:40:03.561 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-21T10:40:03.561 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-21T10:40:03.561 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-21T10:40:03.562 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-21T10:40:03.562 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-21T10:40:03.562 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-21T10:40:03.562 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-21T10:40:03.562 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-21T10:40:03.562 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-21T10:40:03.562 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-21T10:40:03.562 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-21T10:40:03.562 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-21T10:40:03.562 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-21T10:40:03.562 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-21T10:40:03.562 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-21T10:40:03.563 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-21T10:40:03.563 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-21T10:40:03.563 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-21T10:40:03.563 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-21T10:40:03.563 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-21T10:40:03.563 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-21T10:40:03.563 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_mtime_update_interval": "60", 2024-09-21T10:40:03.563 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_non_blocking_aio": "true", 2024-09-21T10:40:03.563 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_op_thread_timeout": "60", 2024-09-21T10:40:03.563 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_op_threads": "1", 2024-09-21T10:40:03.563 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_parent_cache_enabled": "false", 2024-09-21T10:40:03.563 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-21T10:40:03.563 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-21T10:40:03.563 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-21T10:40:03.564 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_plugins": "", 2024-09-21T10:40:03.564 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_bps_burst": "0", 2024-09-21T10:40:03.564 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-21T10:40:03.564 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_bps_limit": "0", 2024-09-21T10:40:03.564 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_exclude_ops": "0", 2024-09-21T10:40:03.564 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_iops_burst": "0", 2024-09-21T10:40:03.564 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-21T10:40:03.564 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_iops_limit": "0", 2024-09-21T10:40:03.564 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-21T10:40:03.564 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-21T10:40:03.564 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-21T10:40:03.564 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-21T10:40:03.564 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-21T10:40:03.565 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-21T10:40:03.565 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-21T10:40:03.565 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-21T10:40:03.565 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-21T10:40:03.565 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-21T10:40:03.565 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-21T10:40:03.565 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-21T10:40:03.565 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-21T10:40:03.565 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-21T10:40:03.565 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_read_from_replica_policy": "default", 2024-09-21T10:40:03.565 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-21T10:40:03.565 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-21T10:40:03.565 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-21T10:40:03.566 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-21T10:40:03.566 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_skip_partial_discard": "true", 2024-09-21T10:40:03.566 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-21T10:40:03.566 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_tracing": "false", 2024-09-21T10:40:03.566 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_validate_names": "true", 2024-09-21T10:40:03.566 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_validate_pool": "true", 2024-09-21T10:40:03.566 INFO:teuthology.orchestra.run.smithi070.stdout: "restapi_base_url": "", 2024-09-21T10:40:03.566 INFO:teuthology.orchestra.run.smithi070.stdout: "restapi_log_level": "", 2024-09-21T10:40:03.566 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-21T10:40:03.566 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-21T10:40:03.566 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_acl_grants_max_num": "100", 2024-09-21T10:40:03.566 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_admin_entry": "admin", 2024-09-21T10:40:03.567 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-21T10:40:03.567 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_asio_assert_yielding": "false", 2024-09-21T10:40:03.567 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_backend_store": "rados", 2024-09-21T10:40:03.567 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_barbican_url": "", 2024-09-21T10:40:03.567 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_beast_enable_async": "true", 2024-09-21T10:40:03.567 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_bucket_counters_cache": "false", 2024-09-21T10:40:03.567 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-21T10:40:03.567 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-21T10:40:03.567 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-21T10:40:03.567 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-21T10:40:03.567 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-21T10:40:03.567 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-21T10:40:03.567 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-21T10:40:03.568 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-21T10:40:03.568 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_cache_enabled": "true", 2024-09-21T10:40:03.568 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_cache_expiry_interval": "900", 2024-09-21T10:40:03.568 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_cache_lru_size": "25000", 2024-09-21T10:40:03.568 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_config_store": "rados", 2024-09-21T10:40:03.568 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_content_length_compat": "false", 2024-09-21T10:40:03.568 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_copy_obj_progress": "true", 2024-09-21T10:40:03.568 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-21T10:40:03.568 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_cors_rules_max_num": "100", 2024-09-21T10:40:03.568 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_cross_domain_policy": "", 2024-09-21T10:40:03.568 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-21T10:40:03.568 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_kmip_addr": "", 2024-09-21T10:40:03.568 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-21T10:40:03.568 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-21T10:40:03.568 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-21T10:40:03.569 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-21T10:40:03.569 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_kmip_password": "", 2024-09-21T10:40:03.569 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-21T10:40:03.569 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_kmip_username": "", 2024-09-21T10:40:03.569 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_require_ssl": "true", 2024-09-21T10:40:03.569 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-21T10:40:03.569 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-21T10:40:03.569 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-21T10:40:03.569 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-21T10:40:03.569 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-21T10:40:03.569 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-21T10:40:03.569 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-21T10:40:03.569 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-21T10:40:03.570 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-21T10:40:03.570 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-21T10:40:03.570 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-21T10:40:03.570 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-21T10:40:03.570 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-21T10:40:03.570 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-21T10:40:03.570 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-21T10:40:03.570 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_vault_addr": "", 2024-09-21T10:40:03.570 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_vault_auth": "token", 2024-09-21T10:40:03.570 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_vault_namespace": "", 2024-09-21T10:40:03.570 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_vault_prefix": "", 2024-09-21T10:40:03.570 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-21T10:40:03.570 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-21T10:40:03.571 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-21T10:40:03.571 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-21T10:40:03.571 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_vault_token_file": "", 2024-09-21T10:40:03.571 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-21T10:40:03.571 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_curl_buffersize": "524288", 2024-09-21T10:40:03.571 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-21T10:40:03.571 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_curl_low_speed_time": "300", 2024-09-21T10:40:03.571 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-21T10:40:03.571 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-21T10:40:03.571 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-21T10:40:03.571 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-21T10:40:03.571 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-21T10:40:03.571 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-21T10:40:03.572 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-21T10:40:03.572 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-21T10:40:03.572 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-21T10:40:03.572 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-21T10:40:03.572 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-21T10:40:03.572 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-21T10:40:03.572 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-21T10:40:03.572 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-21T10:40:03.572 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-21T10:40:03.572 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-21T10:40:03.572 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-21T10:40:03.572 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-21T10:40:03.572 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-21T10:40:03.572 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_data_log_changes_size": "1000", 2024-09-21T10:40:03.572 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_data_log_num_shards": "128", 2024-09-21T10:40:03.573 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-21T10:40:03.573 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_data_log_window": "30", 2024-09-21T10:40:03.573 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-21T10:40:03.573 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-21T10:40:03.573 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-21T10:40:03.573 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-21T10:40:03.573 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-21T10:40:03.573 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-21T10:40:03.573 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-21T10:40:03.573 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-21T10:40:03.573 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-21T10:40:03.573 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-21T10:40:03.573 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-21T10:40:03.573 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-21T10:40:03.574 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-21T10:40:03.574 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_disable_s3select": "false", 2024-09-21T10:40:03.574 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-21T10:40:03.574 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-21T10:40:03.574 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-21T10:40:03.574 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-21T10:40:03.574 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-21T10:40:03.574 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-21T10:40:03.574 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-21T10:40:03.574 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-21T10:40:03.574 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-21T10:40:03.574 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-21T10:40:03.574 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-21T10:40:03.574 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-21T10:40:03.575 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dns_name": "", 2024-09-21T10:40:03.575 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dns_s3website_name": "", 2024-09-21T10:40:03.575 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dynamic_resharding": "true", 2024-09-21T10:40:03.575 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-21T10:40:03.575 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-21T10:40:03.575 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-21T10:40:03.575 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_enable_gc_threads": "true", 2024-09-21T10:40:03.575 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_enable_lc_threads": "true", 2024-09-21T10:40:03.575 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_enable_mdsearch": "true", 2024-09-21T10:40:03.575 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_enable_ops_log": "false", 2024-09-21T10:40:03.575 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_enable_quota_threads": "true", 2024-09-21T10:40:03.575 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_enable_static_website": "false", 2024-09-21T10:40:03.576 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_enable_usage_log": "false", 2024-09-21T10:40:03.576 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_enforce_swift_acls": "true", 2024-09-21T10:40:03.576 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_exit_timeout_secs": "120", 2024-09-21T10:40:03.576 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_expose_bucket": "false", 2024-09-21T10:40:03.576 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_extended_http_attrs": "", 2024-09-21T10:40:03.576 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_filter": "none", 2024-09-21T10:40:03.576 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-21T10:40:03.576 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_frontends": "beast port=7480", 2024-09-21T10:40:03.576 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-21T10:40:03.576 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_gc_max_deferred": "50", 2024-09-21T10:40:03.576 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-21T10:40:03.576 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_gc_max_objs": "32", 2024-09-21T10:40:03.576 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-21T10:40:03.576 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-21T10:40:03.577 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-21T10:40:03.577 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-21T10:40:03.577 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_gc_processor_period": "3600", 2024-09-21T10:40:03.577 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-21T10:40:03.577 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-21T10:40:03.577 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-21T10:40:03.577 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-21T10:40:03.577 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_init_timeout": "300", 2024-09-21T10:40:03.577 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-21T10:40:03.577 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-21T10:40:03.577 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_kafka_connection_idle": "300", 2024-09-21T10:40:03.577 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-21T10:40:03.577 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-21T10:40:03.577 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-21T10:40:03.578 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-21T10:40:03.578 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-21T10:40:03.578 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_admin_domain": "", 2024-09-21T10:40:03.578 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_admin_password": "", 2024-09-21T10:40:03.578 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_admin_password_path": "", 2024-09-21T10:40:03.578 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_admin_project": "", 2024-09-21T10:40:03.578 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_admin_tenant": "", 2024-09-21T10:40:03.578 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_admin_token": "", 2024-09-21T10:40:03.578 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_admin_token_path": "", 2024-09-21T10:40:03.578 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_admin_user": "", 2024-09-21T10:40:03.578 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_barbican_domain": "", 2024-09-21T10:40:03.578 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_barbican_password": "", 2024-09-21T10:40:03.578 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_barbican_project": "", 2024-09-21T10:40:03.579 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-21T10:40:03.579 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_barbican_user": "", 2024-09-21T10:40:03.579 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-21T10:40:03.579 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-21T10:40:03.579 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-21T10:40:03.579 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-21T10:40:03.579 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-21T10:40:03.579 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_url": "", 2024-09-21T10:40:03.579 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-21T10:40:03.579 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_lc_debug_interval": "-1", 2024-09-21T10:40:03.579 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_lc_lock_max_time": "90", 2024-09-21T10:40:03.579 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_lc_max_objs": "32", 2024-09-21T10:40:03.579 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_lc_max_rules": "1000", 2024-09-21T10:40:03.579 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_lc_max_worker": "3", 2024-09-21T10:40:03.580 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-21T10:40:03.580 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_lc_thread_delay": "0", 2024-09-21T10:40:03.580 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-21T10:40:03.580 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_ldap_dnattr": "uid", 2024-09-21T10:40:03.580 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-21T10:40:03.580 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_ldap_searchfilter": "", 2024-09-21T10:40:03.580 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-21T10:40:03.580 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_ldap_uri": "", 2024-09-21T10:40:03.580 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-21T10:40:03.580 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-21T10:40:03.580 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-21T10:40:03.580 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-21T10:40:03.580 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_log_http_headers": "", 2024-09-21T10:40:03.580 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-21T10:40:03.581 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-21T10:40:03.581 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_log_object_name_utc": "false", 2024-09-21T10:40:03.581 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-21T10:40:03.581 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-21T10:40:03.581 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_max_attr_name_len": "0", 2024-09-21T10:40:03.581 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_max_attr_size": "0", 2024-09-21T10:40:03.581 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-21T10:40:03.581 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_max_chunk_size": "4194304", 2024-09-21T10:40:03.581 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-21T10:40:03.581 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-21T10:40:03.581 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-21T10:40:03.581 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_max_listing_results": "5000", 2024-09-21T10:40:03.581 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_max_notify_retries": "10", 2024-09-21T10:40:03.582 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-21T10:40:03.582 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_max_put_param_size": "1048576", 2024-09-21T10:40:03.582 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_max_put_size": "5368709120", 2024-09-21T10:40:03.582 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_max_slo_entries": "1000", 2024-09-21T10:40:03.582 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_md_log_max_shards": "64", 2024-09-21T10:40:03.582 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-21T10:40:03.582 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-21T10:40:03.582 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-21T10:40:03.582 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-21T10:40:03.582 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_mp_lock_max_time": "600", 2024-09-21T10:40:03.582 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-21T10:40:03.582 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-21T10:40:03.582 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-21T10:40:03.583 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-21T10:40:03.583 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-21T10:40:03.583 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-21T10:40:03.583 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-21T10:40:03.583 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-21T10:40:03.583 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_nfs_max_gc": "300", 2024-09-21T10:40:03.583 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-21T10:40:03.583 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-21T10:40:03.583 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-21T10:40:03.583 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-21T10:40:03.583 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-21T10:40:03.583 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-21T10:40:03.583 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-21T10:40:03.583 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_num_async_rados_threads": "32", 2024-09-21T10:40:03.584 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_num_control_oids": "8", 2024-09-21T10:40:03.584 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_numa_node": "-1", 2024-09-21T10:40:03.584 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-21T10:40:03.584 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-21T10:40:03.584 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_objexp_chunk_size": "100", 2024-09-21T10:40:03.584 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_objexp_gc_interval": "600", 2024-09-21T10:40:03.584 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-21T10:40:03.584 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-21T10:40:03.584 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-21T10:40:03.584 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_op_thread_timeout": "600", 2024-09-21T10:40:03.584 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_op_tracing": "false", 2024-09-21T10:40:03.584 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_opa_token": "", 2024-09-21T10:40:03.584 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_opa_url": "", 2024-09-21T10:40:03.584 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_opa_verify_ssl": "true", 2024-09-21T10:40:03.585 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-21T10:40:03.585 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-21T10:40:03.585 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_ops_log_rados": "false", 2024-09-21T10:40:03.585 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_ops_log_socket_path": "", 2024-09-21T10:40:03.585 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-21T10:40:03.585 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-21T10:40:03.585 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-21T10:40:03.585 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_period_push_interval": "2.000000", 2024-09-21T10:40:03.585 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-21T10:40:03.585 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-21T10:40:03.585 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-21T10:40:03.585 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-21T10:40:03.585 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_posix_cache_lanes": "3", 2024-09-21T10:40:03.585 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-21T10:40:03.586 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-21T10:40:03.586 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_posix_cache_partitions": "3", 2024-09-21T10:40:03.586 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-21T10:40:03.586 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_print_continue": "true", 2024-09-21T10:40:03.586 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-21T10:40:03.586 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-21T10:40:03.586 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-21T10:40:03.586 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-21T10:40:03.586 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-21T10:40:03.586 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_rados_tracing": "false", 2024-09-21T10:40:03.586 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_realm": "", 2024-09-21T10:40:03.586 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_realm_id": "", 2024-09-21T10:40:03.586 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-21T10:40:03.587 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_region": "", 2024-09-21T10:40:03.587 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-21T10:40:03.587 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-21T10:40:03.587 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-21T10:40:03.587 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-21T10:40:03.587 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_request_uri": "", 2024-09-21T10:40:03.587 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_reshard_batch_size": "64", 2024-09-21T10:40:03.587 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-21T10:40:03.587 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-21T10:40:03.587 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_reshard_max_aio": "128", 2024-09-21T10:40:03.587 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_reshard_num_logs": "16", 2024-09-21T10:40:03.587 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_reshard_progress_judge_interval": "120", 2024-09-21T10:40:03.587 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_reshard_progress_judge_ratio": "0.500000", 2024-09-21T10:40:03.588 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_reshard_thread_interval": "600", 2024-09-21T10:40:03.588 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_reshardlog_threshold": "30000", 2024-09-21T10:40:03.588 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_resolve_cname": "false", 2024-09-21T10:40:03.588 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-21T10:40:03.588 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_run_sync_thread": "true", 2024-09-21T10:40:03.588 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-21T10:40:03.588 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-21T10:40:03.588 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-21T10:40:03.588 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-21T10:40:03.588 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-21T10:40:03.588 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-21T10:40:03.588 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-21T10:40:03.588 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-21T10:40:03.588 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-21T10:40:03.588 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_scheduler_type": "throttler", 2024-09-21T10:40:03.589 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_script_uri": "", 2024-09-21T10:40:03.589 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_service_provider_name": "", 2024-09-21T10:40:03.589 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-21T10:40:03.589 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sts_client_id": "", 2024-09-21T10:40:03.589 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sts_client_secret": "", 2024-09-21T10:40:03.589 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sts_entry": "sts", 2024-09-21T10:40:03.589 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sts_key": "sts", 2024-09-21T10:40:03.589 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-21T10:40:03.589 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sts_min_session_duration": "900", 2024-09-21T10:40:03.589 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sts_token_introspection_url": "", 2024-09-21T10:40:03.589 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_swift_account_in_url": "false", 2024-09-21T10:40:03.589 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_swift_auth_entry": "auth", 2024-09-21T10:40:03.589 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_swift_auth_url": "", 2024-09-21T10:40:03.590 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_swift_custom_header": "", 2024-09-21T10:40:03.590 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-21T10:40:03.590 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_swift_need_stats": "true", 2024-09-21T10:40:03.590 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_swift_tenant_name": "", 2024-09-21T10:40:03.590 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_swift_token_expiration": "86400", 2024-09-21T10:40:03.590 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_swift_url": "", 2024-09-21T10:40:03.590 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_swift_url_prefix": "swift", 2024-09-21T10:40:03.590 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-21T10:40:03.590 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-21T10:40:03.590 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-21T10:40:03.590 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sync_lease_period": "120", 2024-09-21T10:40:03.590 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-21T10:40:03.590 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-21T10:40:03.591 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-21T10:40:03.591 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-21T10:40:03.591 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-21T10:40:03.591 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-21T10:40:03.591 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-21T10:40:03.591 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-21T10:40:03.591 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-21T10:40:03.591 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_thread_pool_size": "512", 2024-09-21T10:40:03.591 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-21T10:40:03.591 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-21T10:40:03.591 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-21T10:40:03.591 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-21T10:40:03.591 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_torrent_comment": "", 2024-09-21T10:40:03.592 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_torrent_createby": "", 2024-09-21T10:40:03.592 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_torrent_encoding": "", 2024-09-21T10:40:03.592 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_torrent_flag": "false", 2024-09-21T10:40:03.592 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-21T10:40:03.592 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_torrent_origin": "", 2024-09-21T10:40:03.592 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-21T10:40:03.592 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_torrent_tracker": "", 2024-09-21T10:40:03.592 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_trust_forwarded_https": "false", 2024-09-21T10:40:03.592 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-21T10:40:03.592 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-21T10:40:03.592 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_usage_max_shards": "32", 2024-09-21T10:40:03.592 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_usage_max_user_shards": "1", 2024-09-21T10:40:03.592 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_use_opa_authz": "false", 2024-09-21T10:40:03.592 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_user_counters_cache": "false", 2024-09-21T10:40:03.593 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-21T10:40:03.593 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-21T10:40:03.593 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-21T10:40:03.593 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_user_max_buckets": "1000", 2024-09-21T10:40:03.593 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_user_policies_max_num": "100", 2024-09-21T10:40:03.593 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-21T10:40:03.593 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-21T10:40:03.593 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-21T10:40:03.593 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-21T10:40:03.593 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_user_unique_email": "true", 2024-09-21T10:40:03.593 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_verify_ssl": "true", 2024-09-21T10:40:03.593 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-21T10:40:03.593 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_zone": "", 2024-09-21T10:40:03.593 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_zone_id": "", 2024-09-21T10:40:03.594 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-21T10:40:03.594 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_zonegroup": "", 2024-09-21T10:40:03.594 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_zonegroup_id": "", 2024-09-21T10:40:03.594 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-21T10:40:03.594 INFO:teuthology.orchestra.run.smithi070.stdout: "rgwlc_auto_session_clear": "true", 2024-09-21T10:40:03.594 INFO:teuthology.orchestra.run.smithi070.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-21T10:40:03.594 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_block_size": "4096", 2024-09-21T10:40:03.594 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-21T10:40:03.594 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-21T10:40:03.594 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-21T10:40:03.594 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-21T10:40:03.594 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-21T10:40:03.594 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_cache_size": "536870912", 2024-09-21T10:40:03.595 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-21T10:40:03.595 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-21T10:40:03.595 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-21T10:40:03.595 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-21T10:40:03.595 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-21T10:40:03.595 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-21T10:40:03.595 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-21T10:40:03.595 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-21T10:40:03.595 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_index_type": "binary_search", 2024-09-21T10:40:03.595 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-21T10:40:03.595 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-21T10:40:03.595 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_partition_filters": "false", 2024-09-21T10:40:03.595 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_perf": "false", 2024-09-21T10:40:03.595 INFO:teuthology.orchestra.run.smithi070.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-21T10:40:03.595 INFO:teuthology.orchestra.run.smithi070.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-21T10:40:03.596 INFO:teuthology.orchestra.run.smithi070.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-21T10:40:03.596 INFO:teuthology.orchestra.run.smithi070.stdout: "run_dir": "/var/run/ceph", 2024-09-21T10:40:03.596 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_block_create": "true", 2024-09-21T10:40:03.596 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_cache_lru_size": "67108864", 2024-09-21T10:40:03.596 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-21T10:40:03.596 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-21T10:40:03.596 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_default_max_object_size": "16777216", 2024-09-21T10:40:03.596 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-21T10:40:03.596 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_device_size": "53687091200", 2024-09-21T10:40:03.596 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-21T10:40:03.596 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_full_integrity_check": "false", 2024-09-21T10:40:03.596 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_journal_batch_capacity": "16", 2024-09-21T10:40:03.596 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-21T10:40:03.596 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-21T10:40:03.596 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-21T10:40:03.597 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_main_device_type": "SSD", 2024-09-21T10:40:03.597 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-21T10:40:03.597 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-21T10:40:03.597 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-21T10:40:03.597 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-21T10:40:03.597 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-21T10:40:03.597 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-21T10:40:03.597 INFO:teuthology.orchestra.run.smithi070.stdout: "seastore_segment_size": "67108864", 2024-09-21T10:40:03.597 INFO:teuthology.orchestra.run.smithi070.stdout: "set_keepcaps": "false", 2024-09-21T10:40:03.597 INFO:teuthology.orchestra.run.smithi070.stdout: "setgroup": "ceph", 2024-09-21T10:40:03.597 INFO:teuthology.orchestra.run.smithi070.stdout: "setuser": "ceph", 2024-09-21T10:40:03.597 INFO:teuthology.orchestra.run.smithi070.stdout: "setuser_match_path": "", 2024-09-21T10:40:03.597 INFO:teuthology.orchestra.run.smithi070.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-21T10:40:03.597 INFO:teuthology.orchestra.run.smithi070.stdout: "thp": "false", 2024-09-21T10:40:03.598 INFO:teuthology.orchestra.run.smithi070.stdout: "threadpool_default_timeout": "60", 2024-09-21T10:40:03.598 INFO:teuthology.orchestra.run.smithi070.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-21T10:40:03.598 INFO:teuthology.orchestra.run.smithi070.stdout: "throttler_perf_counter": "true", 2024-09-21T10:40:03.598 INFO:teuthology.orchestra.run.smithi070.stdout: "tmp_dir": "/tmp", 2024-09-21T10:40:03.598 INFO:teuthology.orchestra.run.smithi070.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-21T10:40:03.598 INFO:teuthology.orchestra.run.smithi070.stdout: "uadk_compressor_enabled": "false", 2024-09-21T10:40:03.598 INFO:teuthology.orchestra.run.smithi070.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-21T10:40:03.598 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:40:04.536 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-21T10:40:04.732 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:04.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:04 smithi160 ceph-mon[101879]: pgmap v2139: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:40:04.847 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:04 smithi070 ceph-mon[107385]: pgmap v2139: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:40:04.847 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:04 smithi070 ceph-mon[112139]: pgmap v2139: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:40:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:40:05.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:40:05.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:40:06.431 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a: {} 2024-09-21T10:40:06.431 INFO:teuthology.orchestra.run.smithi070.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T10:40:06.440 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b: {} 2024-09-21T10:40:06.440 INFO:teuthology.orchestra.run.smithi070.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T10:40:06.448 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c: {} 2024-09-21T10:40:06.448 INFO:teuthology.orchestra.run.smithi070.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T10:40:06.596 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:06 smithi070 ceph-mon[107385]: pgmap v2140: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:40:06.596 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:06 smithi070 ceph-mon[112139]: pgmap v2140: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:40:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:06 smithi160 ceph-mon[101879]: pgmap v2140: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:40:07.074 INFO:tasks.thrashosds.thrasher:Unable to test ceph-objectstore-tool, not available on all OSD nodes 2024-09-21T10:40:07.074 INFO:teuthology.task.sequential:In sequential, running task full_sequential... 2024-09-21T10:40:07.075 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T10:40:07.076 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T10:40:07.076 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-21T10:40:07.076 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_0 16 2024-09-21T10:40:07.077 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-21T10:40:07.077 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:40:07.077 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:40:07.077 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:40:07.077 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd out 2 2024-09-21T10:40:07.078 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-21T10:40:07.078 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:07.078 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-21T10:40:07.078 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:40:07.079 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-21T10:40:07.079 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:40:07.079 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-21T10:40:07.079 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:40:07.108 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:07.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:07.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:07.210 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:07.243 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:07.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:07.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:07.317 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:07.321 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:07.325 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:07.345 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:07.368 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:07.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:07.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:07.470 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:07.494 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:07.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:07.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:07.595 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:07.617 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:07.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:07.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:07.705 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:07 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3780025760' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T10:40:07.705 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:07 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3780025760' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T10:40:07.719 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:07.744 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:07.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:07.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:07 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3780025760' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T10:40:07.846 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:07.868 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:07.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:07.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:07.969 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:07.995 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:07.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:07.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:08.097 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:08.121 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:08.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:08.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:08.223 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:08.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:08.244 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:08.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:08.345 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:08.371 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:08.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:08.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:08.472 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:08.498 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:08.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:08.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:08.578 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:08 smithi070 ceph-mon[107385]: pgmap v2141: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:40:08.578 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:08 smithi070 ceph-mon[112139]: pgmap v2141: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:40:08.599 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:08.624 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:08.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:08.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:08.727 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:08.750 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:08.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:08.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:08 smithi160 ceph-mon[101879]: pgmap v2141: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:40:08.852 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:08.875 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:08.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:08.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:08.977 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:09.002 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:09.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:09.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:09.104 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:09.127 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:09.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:09.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:09.228 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:09.251 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:09.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:09.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:09.353 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:09.379 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:09.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:09.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:09.480 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:09.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:09.503 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:09.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:09.603 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:09.624 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:09.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:09.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:09.726 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:09.751 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:09.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:09.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:09.754 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:09.754 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:09.853 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:09.874 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:09.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:09.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:09.975 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:09.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:09.997 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:09.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:10.098 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:10.123 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:10.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:10.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:10.225 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:10.246 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:10.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:10.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:10.347 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:10.373 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:10.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:10.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:10.475 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:10.500 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:10.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:10.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:10.602 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:10.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:10.624 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:10.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:10.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:10 smithi070 ceph-mon[107385]: pgmap v2142: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:40:10.724 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:10 smithi070 ceph-mon[112139]: pgmap v2142: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:40:10.724 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:10.748 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:10.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:10.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:10.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:10 smithi160 ceph-mon[101879]: pgmap v2142: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:40:10.850 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:10.873 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:10.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:10.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:10.975 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:11.001 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:11.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:11.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:11.103 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:11.129 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:11.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:11.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:11.230 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:11.252 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:11.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:11.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:11.353 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:11.375 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:11.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:11.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:11.476 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:11.502 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:11.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:11.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:11.603 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:11.628 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:11.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:11.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:11.730 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:11.756 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:11.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:11.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:11.858 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:11.883 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:11.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:11.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:11.957 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:11 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a[107362]: 2024-09-21T10:40:11.665+0000 7fad4a465640 -1 mon.a@0(leader).osd e1304 definitely_dead 0 2024-09-21T10:40:11.985 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:12.007 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:12.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:12.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:12.108 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:12.134 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:12.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:12.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:12.236 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:12.261 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:12.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:12.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:12.363 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:12.388 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:12.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:12.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:12.420 INFO:teuthology.orchestra.run.smithi070.stderr:marked out osd.2. 2024-09-21T10:40:12.423 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T10:40:12.489 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:12.514 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:12.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:12.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:12.537 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:12 smithi070 ceph-mon[112139]: pgmap v2143: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:40:12.537 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:12 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2233434386' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:40:12.537 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:12 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:40:12.538 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:12 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2829262639' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-21T10:40:12.538 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:12 smithi070 ceph-mon[112139]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-21T10:40:12.538 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:12 smithi070 ceph-mon[107385]: pgmap v2143: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:40:12.538 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:12 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2233434386' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:40:12.538 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:12 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:40:12.538 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:12 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2829262639' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-21T10:40:12.538 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:12 smithi070 ceph-mon[107385]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-21T10:40:12.616 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:12.637 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:12.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:12.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:12.739 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:12.760 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:12.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:12.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:12 smithi160 ceph-mon[101879]: pgmap v2143: 73 pgs: 73 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-09-21T10:40:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:12 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2233434386' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:40:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:12 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:40:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:12 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2829262639' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-21T10:40:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:12 smithi160 ceph-mon[101879]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-21T10:40:12.862 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:12.887 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:12.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:12.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:12.989 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:13.010 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:13.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:13.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:13.112 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:13.137 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:13.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:13.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:13.239 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:13.263 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:13.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:13.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:13.365 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:13.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:40:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:40:13] "GET /metrics HTTP/1.1" 200 34046 "" "Prometheus/2.51.0" 2024-09-21T10:40:13.387 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:13.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:13.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:13.488 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:13.509 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:13.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:13.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:13.611 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:13.633 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:13.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:13.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:13.734 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:13.755 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:13.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:13.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:13 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:40:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:13 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:40:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:13 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2829262639' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-21T10:40:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:13 smithi160 ceph-mon[101879]: osdmap e1305: 8 total, 8 up, 7 in 2024-09-21T10:40:13.857 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:13.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:13 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:40:13.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:13 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:40:13.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:13 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2829262639' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-21T10:40:13.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:13 smithi070 ceph-mon[107385]: osdmap e1305: 8 total, 8 up, 7 in 2024-09-21T10:40:13.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:13 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:40:13.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:13 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:40:13.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:13 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2829262639' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-21T10:40:13.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:13 smithi070 ceph-mon[112139]: osdmap e1305: 8 total, 8 up, 7 in 2024-09-21T10:40:13.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:13.879 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:13.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:13.980 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:14.005 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:14.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:14.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:14.107 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:14.129 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:14.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:14.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:14.230 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:14.254 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:14.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:14.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:14.356 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:14.381 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:14.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:14.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:14.483 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:14.507 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:14.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:14.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:14.534 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:14 smithi070 ceph-mon[107385]: pgmap v2145: 73 pgs: 73 active+clean; 582 KiB data, 748 MiB used, 625 GiB / 626 GiB avail 2024-09-21T10:40:14.534 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:14 smithi070 ceph-mon[107385]: osdmap e1306: 8 total, 8 up, 7 in 2024-09-21T10:40:14.535 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:14 smithi070 ceph-mon[112139]: pgmap v2145: 73 pgs: 73 active+clean; 582 KiB data, 748 MiB used, 625 GiB / 626 GiB avail 2024-09-21T10:40:14.535 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:14 smithi070 ceph-mon[112139]: osdmap e1306: 8 total, 8 up, 7 in 2024-09-21T10:40:14.608 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:14.630 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:14.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:14.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:14.732 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:14.754 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:14.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:14.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:14 smithi160 ceph-mon[101879]: pgmap v2145: 73 pgs: 73 active+clean; 582 KiB data, 748 MiB used, 625 GiB / 626 GiB avail 2024-09-21T10:40:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:14 smithi160 ceph-mon[101879]: osdmap e1306: 8 total, 8 up, 7 in 2024-09-21T10:40:14.855 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:14.877 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:14.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:14.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:14.945 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:40:14.945 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T10:40:14.945 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T10:40:14.945 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:40:14.979 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:15.004 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:15.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:15.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:15.106 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:15.131 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:15.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:15.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:15.233 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:15.258 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:15.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:15.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:15.360 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:15.381 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:15.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:15.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:15.432 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'unique_pool_0' created 2024-09-21T10:40:15.483 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:15.504 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:15.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:15.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:15.606 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:15.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:15 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/384578134' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-21T10:40:15.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:15 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/384578134' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-21T10:40:15.632 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:15.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:15.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:15.734 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:15.759 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:15.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:15.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:15.804 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:40:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:15 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/384578134' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-21T10:40:15.860 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:15.886 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:15.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:15.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:15.989 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:16.015 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:16.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:16.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:16.117 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:16.141 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:16.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:16.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:16.243 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:16.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:16.266 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:16.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:16.367 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:16.390 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:16.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:16.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:16.492 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:16.514 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:16.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:16.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:16.616 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:16.638 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:16.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:16.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:16.739 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:16.760 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:16.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:16.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:16.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:16 smithi160 ceph-mon[101879]: pgmap v2147: 73 pgs: 73 active+clean; 582 KiB data, 748 MiB used, 625 GiB / 626 GiB avail; 72 B/s, 0 objects/s recovering 2024-09-21T10:40:16.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:16 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/384578134' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-21T10:40:16.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:16 smithi160 ceph-mon[101879]: osdmap e1307: 8 total, 8 up, 7 in 2024-09-21T10:40:16.842 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:16 smithi070 ceph-mon[107385]: pgmap v2147: 73 pgs: 73 active+clean; 582 KiB data, 748 MiB used, 625 GiB / 626 GiB avail; 72 B/s, 0 objects/s recovering 2024-09-21T10:40:16.843 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:16 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/384578134' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-21T10:40:16.843 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:16 smithi070 ceph-mon[107385]: osdmap e1307: 8 total, 8 up, 7 in 2024-09-21T10:40:16.843 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:16 smithi070 ceph-mon[112139]: pgmap v2147: 73 pgs: 73 active+clean; 582 KiB data, 748 MiB used, 625 GiB / 626 GiB avail; 72 B/s, 0 objects/s recovering 2024-09-21T10:40:16.843 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:16 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/384578134' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-21T10:40:16.843 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:16 smithi070 ceph-mon[112139]: osdmap e1307: 8 total, 8 up, 7 in 2024-09-21T10:40:16.861 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:16.885 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:16.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:16.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:16.986 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:17.009 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:17.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:17.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:17.111 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:17.132 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:17.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:17.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:17.232 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-21T10:40:17.234 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:17.257 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:17.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:17.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:17.358 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:17.381 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:17.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:17.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:17.479 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:17 smithi070 ceph-mon[107385]: osdmap e1308: 8 total, 8 up, 7 in 2024-09-21T10:40:17.479 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:17 smithi070 ceph-mon[112139]: osdmap e1308: 8 total, 8 up, 7 in 2024-09-21T10:40:17.482 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:17.506 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:17.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:17.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:17.578 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:40:17.578 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:40:17.596 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:40:17.596 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:40:17.608 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:17.612 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:40:17.612 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:40:17.630 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:17.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:17.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:17.635 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:40:17.635 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:40:17.636 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:17.652 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:40:17.652 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:40:17.672 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:40:17.673 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:40:17.673 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:17.690 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:40:17.690 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:40:17.707 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:40:17.707 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:40:17.732 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:17.754 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:17.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:17.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:17.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:17 smithi160 ceph-mon[101879]: osdmap e1308: 8 total, 8 up, 7 in 2024-09-21T10:40:17.856 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:17.878 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:17.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:17.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:17.980 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:18.001 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:18.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:18.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:18.103 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:18.124 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:18.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:18.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:18.226 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:18.247 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:18.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:18.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:18.349 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:18.370 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:18.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:18.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:18.472 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:18.494 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:18.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:18.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:18.595 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:18.617 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:18.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:18.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:18.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:18 smithi070 ceph-mon[112139]: pgmap v2150: 89 pgs: 8 creating+peering, 8 unknown, 73 active+clean; 582 KiB data, 746 MiB used, 625 GiB / 626 GiB avail; 145 B/s, 2 keys/s, 1 objects/s recovering 2024-09-21T10:40:18.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:18 smithi070 ceph-mon[112139]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:40:18.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:18 smithi070 ceph-mon[112139]: osdmap e1309: 8 total, 8 up, 7 in 2024-09-21T10:40:18.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:18 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/413769084' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:40:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:18 smithi070 ceph-mon[107385]: pgmap v2150: 89 pgs: 8 creating+peering, 8 unknown, 73 active+clean; 582 KiB data, 746 MiB used, 625 GiB / 626 GiB avail; 145 B/s, 2 keys/s, 1 objects/s recovering 2024-09-21T10:40:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:18 smithi070 ceph-mon[107385]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:40:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:18 smithi070 ceph-mon[107385]: osdmap e1309: 8 total, 8 up, 7 in 2024-09-21T10:40:18.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:18 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/413769084' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:40:18.719 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:18.740 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:18.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:18.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:18 smithi160 ceph-mon[101879]: pgmap v2150: 89 pgs: 8 creating+peering, 8 unknown, 73 active+clean; 582 KiB data, 746 MiB used, 625 GiB / 626 GiB avail; 145 B/s, 2 keys/s, 1 objects/s recovering 2024-09-21T10:40:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:18 smithi160 ceph-mon[101879]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:40:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:18 smithi160 ceph-mon[101879]: osdmap e1309: 8 total, 8 up, 7 in 2024-09-21T10:40:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:18 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/413769084' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:40:18.842 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:18.862 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:18.863 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:40:18.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:18.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:18.965 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:18.989 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:18.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:18.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:19.091 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:19.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:19.113 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:19.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:19.214 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:19.236 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:19.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:19.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:19.337 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:19.359 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:19.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:19.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:19.460 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:19.466 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:19 smithi070 ceph-mon[107385]: pgmap v2152: 89 pgs: 8 creating+peering, 8 unknown, 73 active+clean; 582 KiB data, 746 MiB used, 625 GiB / 626 GiB avail; 117 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:40:19.466 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:19 smithi070 ceph-mon[112139]: pgmap v2152: 89 pgs: 8 creating+peering, 8 unknown, 73 active+clean; 582 KiB data, 746 MiB used, 625 GiB / 626 GiB avail; 117 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:40:19.485 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:19.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:19.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:19.587 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:19.612 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:19.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:19.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:19.677 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:19.713 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:19.738 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:19.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:19.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:19.839 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:19 smithi160 ceph-mon[101879]: pgmap v2152: 89 pgs: 8 creating+peering, 8 unknown, 73 active+clean; 582 KiB data, 746 MiB used, 625 GiB / 626 GiB avail; 117 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:40:19.860 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:19.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:19.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:19.962 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:19.988 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:19.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:19.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:20.089 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:20.113 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:20.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:20.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:20.215 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:20.239 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:20.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:20.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:20.340 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:20.364 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:20.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:20.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:20.465 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:20.486 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:20.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:20.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:20.588 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:20.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:20.609 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:20.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:20.710 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:20.731 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:20.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:20.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:20.832 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:20.858 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:20.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:20.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:20.959 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:20.981 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:20.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:20.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:21.082 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:21.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:21.104 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:21.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:21.171 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:40:21.171 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:40:21.171 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 3, 4, 5, 6, 7] 2024-09-21T10:40:21.171 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd out 7 2024-09-21T10:40:21.205 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:21.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:21.231 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:21.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:21.332 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:21.357 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:21.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:21.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:21.459 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:21.466 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:40:21.466 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:40:21.466 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:40:21.466 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:40:21.484 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:21.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:21.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:21.586 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:21.608 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:21.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:21.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:21.643 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:40:21.644 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:40:21.710 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:21.735 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:21.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:21.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:21.837 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:21.862 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:21.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:21.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:21.964 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:21.989 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:21.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:21.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:22.091 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:22.112 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:22.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:22.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:22.125 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:40:22.214 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:22.240 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:22.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:22.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:22.342 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:22.363 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:22.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:22.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:22.370 INFO:teuthology.orchestra.run.smithi070.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-21T10:40:22.438 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:40:22.438 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:40:22.464 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:22.485 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:22.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:22.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:22.586 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:22.608 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:22.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:22.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:22 smithi070 ceph-mon[107385]: pgmap v2153: 89 pgs: 8 creating+peering, 81 active+clean; 582 KiB data, 748 MiB used, 625 GiB / 626 GiB avail; 5 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:40:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:22 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2502453021' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:40:22.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:22 smithi070 ceph-mon[112139]: pgmap v2153: 89 pgs: 8 creating+peering, 81 active+clean; 582 KiB data, 748 MiB used, 625 GiB / 626 GiB avail; 5 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:40:22.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:22 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2502453021' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:40:22.710 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:22.736 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:22.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:22.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:22.837 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:22 smithi160 ceph-mon[101879]: pgmap v2153: 89 pgs: 8 creating+peering, 81 active+clean; 582 KiB data, 748 MiB used, 625 GiB / 626 GiB avail; 5 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:40:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:22 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2502453021' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:40:22.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:22.863 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:22.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:22.964 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:22.989 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:22.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:22.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:23.090 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:23.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:23.117 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:23.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:23.218 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:23.239 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:23.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:23.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:23.337 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:40:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:40:23] "GET /metrics HTTP/1.1" 200 36462 "" "Prometheus/2.51.0" 2024-09-21T10:40:23.340 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:23.360 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:23.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:23.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:23.462 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:23.488 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:23.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:23.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:23.589 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:23.615 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:23.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:23.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:23.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:23 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2502453021' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:40:23.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:23 smithi070 ceph-mon[107385]: osdmap e1310: 8 total, 8 up, 7 in 2024-09-21T10:40:23.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:23 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2502453021' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:40:23.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:23 smithi070 ceph-mon[112139]: osdmap e1310: 8 total, 8 up, 7 in 2024-09-21T10:40:23.717 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:23.742 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:23.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:23.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:23.803 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:40:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:23 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2502453021' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:40:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:23 smithi160 ceph-mon[101879]: osdmap e1310: 8 total, 8 up, 7 in 2024-09-21T10:40:23.844 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:23.866 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:23.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:23.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:23.968 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:23.993 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:23.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:23.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:24.055 DEBUG:teuthology.orchestra.run.smithi070:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-09-21T10:40:24.058 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T10:40:24.058 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T10:40:24.094 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:24.109 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:hints = 1 2024-09-21T10:40:24.109 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T10:40:24.109 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Object prefix: benchmark_data_smithi070_228967 2024-09-21T10:40:24.109 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:40:24.109 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 0 0 0 0 0 0 - 0 2024-09-21T10:40:24.122 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:24.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:24.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:24.223 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:24.263 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:24.266 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:24.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:24.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:24.368 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:24.398 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:24 smithi070 ceph-mon[107385]: pgmap v2155: 89 pgs: 89 active+clean; 582 KiB data, 745 MiB used, 625 GiB / 626 GiB avail 2024-09-21T10:40:24.398 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:24 smithi070 ceph-mon[107385]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:40:24.398 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:24 smithi070 ceph-mon[112139]: pgmap v2155: 89 pgs: 89 active+clean; 582 KiB data, 745 MiB used, 625 GiB / 626 GiB avail 2024-09-21T10:40:24.398 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:24 smithi070 ceph-mon[112139]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:40:24.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:24.404 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:24.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:24.505 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:24.545 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:24.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:24.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:24.646 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:24.660 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:24.661 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:24.675 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:24.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:24.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:24.776 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:24.817 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:24.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:24.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:24 smithi160 ceph-mon[101879]: pgmap v2155: 89 pgs: 89 active+clean; 582 KiB data, 745 MiB used, 625 GiB / 626 GiB avail 2024-09-21T10:40:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:24 smithi160 ceph-mon[101879]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:40:24.918 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:24.956 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:24.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:24.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:25.058 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:25.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:25.095 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:25.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:25.110 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 1 16 3477 3461 216.299 216.312 0.00482259 0.00461125 2024-09-21T10:40:25.197 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:25.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:25.233 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:25.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:25.334 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:25.368 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:25.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:25.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:25.392 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T10:40:25.470 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:25.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:25.506 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:25.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:25.608 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:25.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:25 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/472057316' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:40:25.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:25 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:40:25.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:25 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/472057316' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:40:25.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:25 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:40:25.648 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:25.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:25.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:25.750 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:25.786 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:25.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:25.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:25 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/472057316' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:40:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:25 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:40:25.888 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:25.920 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:25.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:25.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:26.023 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:26.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:26.056 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:26.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:26.110 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 2 16 7062 7046 220.171 224.062 0.00499883 0.0045325 2024-09-21T10:40:26.156 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:26.201 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:26.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:26.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:26.304 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:26.343 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:26.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:26.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:26.445 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:26.481 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:26.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:26.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:26.515 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:26 smithi070 ceph-mon[112139]: pgmap v2156: 89 pgs: 89 active+clean; 15 MiB data, 744 MiB used, 625 GiB / 626 GiB avail; 1.8 MiB/s wr, 28 op/s 2024-09-21T10:40:26.515 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:26 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:40:26.515 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:26 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:40:26.515 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:26 smithi070 ceph-mon[112139]: osdmap e1311: 8 total, 8 up, 7 in 2024-09-21T10:40:26.515 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:26 smithi070 ceph-mon[107385]: pgmap v2156: 89 pgs: 89 active+clean; 15 MiB data, 744 MiB used, 625 GiB / 626 GiB avail; 1.8 MiB/s wr, 28 op/s 2024-09-21T10:40:26.515 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:26 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:40:26.515 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:26 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:40:26.515 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:26 smithi070 ceph-mon[107385]: osdmap e1311: 8 total, 8 up, 7 in 2024-09-21T10:40:26.583 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:26.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:26.618 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:26.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:26.719 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:26.756 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:26.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:26.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:26 smithi160 ceph-mon[101879]: pgmap v2156: 89 pgs: 89 active+clean; 15 MiB data, 744 MiB used, 625 GiB / 626 GiB avail; 1.8 MiB/s wr, 28 op/s 2024-09-21T10:40:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:26 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:40:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:26 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:40:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:26 smithi160 ceph-mon[101879]: osdmap e1311: 8 total, 8 up, 7 in 2024-09-21T10:40:26.858 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:26.890 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:26.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:26.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:26.992 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:27.027 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:27.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:27.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:27.110 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 3 16 10682 10666 222.191 226.25 0.00371315 0.00449442 2024-09-21T10:40:27.129 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:27.163 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:27.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:27.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:27.265 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:27.300 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:27.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:27.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:27.402 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:27.442 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:27.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:27.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:27.544 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:27.577 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:27.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:27.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:27.679 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:27.713 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:27.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:27.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:27.815 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:27.847 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:27.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:27.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:27.949 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:27.983 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:27.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:27.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:28.085 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:28.110 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 4 16 14214 14198 221.826 220.75 0.00425885 0.00450176 2024-09-21T10:40:28.119 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:28.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:28.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:28.221 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:28.255 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:28.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:28.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:28.357 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:28.390 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:28.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:28.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:28.430 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:28 smithi070 ceph-mon[107385]: pgmap v2158: 89 pgs: 89 active+clean; 226 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 28 MiB/s wr, 450 op/s 2024-09-21T10:40:28.430 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:28 smithi070 ceph-mon[112139]: pgmap v2158: 89 pgs: 89 active+clean; 226 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 28 MiB/s wr, 450 op/s 2024-09-21T10:40:28.492 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:28.525 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:28.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:28.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:28.627 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:28.660 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:28.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:28.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:28.762 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:28.795 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:28.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:28.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:28 smithi160 ceph-mon[101879]: pgmap v2158: 89 pgs: 89 active+clean; 226 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 28 MiB/s wr, 450 op/s 2024-09-21T10:40:28.897 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:28.938 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:28.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:28.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:29.040 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:29.069 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:29.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:29.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:29.110 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 5 16 17322 17306 216.307 194.25 0.00467762 0.00461709 2024-09-21T10:40:29.171 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:29.207 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:29.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:29.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:29.309 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:29.341 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:29.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:29.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:29.393 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:29 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a[107362]: 2024-09-21T10:40:29.355+0000 7fad4a465640 -1 mon.a@0(leader).osd e1311 definitely_dead 0 2024-09-21T10:40:29.403 INFO:teuthology.orchestra.run.smithi070.stderr:marked out osd.7. 2024-09-21T10:40:29.442 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:29.454 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:40:29.471 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:29.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:29.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:29.572 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:29.593 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:29.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:29.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:29.694 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:29.715 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:29.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:29.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:29.731 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:29 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2807960897' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-21T10:40:29.731 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:29 smithi070 ceph-mon[107385]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-21T10:40:29.731 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:29 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2807960897' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-21T10:40:29.731 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:29 smithi070 ceph-mon[112139]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-21T10:40:29.817 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:29.838 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:29.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:29.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:29 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2807960897' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-21T10:40:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:29 smithi160 ceph-mon[101879]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-21T10:40:29.939 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:29.961 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:29.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:29.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:30.063 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:30.084 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:30.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:30.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:30.110 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 6 16 18508 18492 192.608 74.125 0.0022296 0.00465921 2024-09-21T10:40:30.185 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:30.207 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:30.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:30.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:30.308 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:30.329 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:30.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:30.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:30.431 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:30.452 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:30.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:30.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:30.553 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:30.573 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:30.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:30.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:30.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:30 smithi070 ceph-mon[107385]: pgmap v2159: 89 pgs: 89 active+clean; 226 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 28 MiB/s wr, 450 op/s 2024-09-21T10:40:30.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:30 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2807960897' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-21T10:40:30.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:30 smithi070 ceph-mon[107385]: osdmap e1312: 8 total, 8 up, 6 in 2024-09-21T10:40:30.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:30 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3206951892' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:40:30.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:30 smithi070 ceph-mon[112139]: pgmap v2159: 89 pgs: 89 active+clean; 226 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 28 MiB/s wr, 450 op/s 2024-09-21T10:40:30.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:30 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2807960897' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-21T10:40:30.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:30 smithi070 ceph-mon[112139]: osdmap e1312: 8 total, 8 up, 6 in 2024-09-21T10:40:30.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:30 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3206951892' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:40:30.675 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:30.696 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:30.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:30.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:30.797 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:30.819 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:30.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:30.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:30 smithi160 ceph-mon[101879]: pgmap v2159: 89 pgs: 89 active+clean; 226 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 28 MiB/s wr, 450 op/s 2024-09-21T10:40:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:30 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2807960897' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-21T10:40:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:30 smithi160 ceph-mon[101879]: osdmap e1312: 8 total, 8 up, 6 in 2024-09-21T10:40:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:30 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3206951892' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:40:30.920 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:30.943 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:30.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:30.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:31.045 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:31.070 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:31.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:31.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:31.092 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:40:31.092 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:40:31.092 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:40:31.092 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:40:31.092 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:40:31.092 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:40:31.092 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:40:31.092 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:40:31.093 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:40:31.093 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:40:31.093 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:40:31.093 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:40:31.093 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:40:31.093 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:40:31.093 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:40:31.093 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:40:31.110 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 7 16 18508 18492 165.091 0 - 0.00465921 2024-09-21T10:40:31.172 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:31.197 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:31.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:31.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:31.298 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:31.324 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:31.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:31.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:31.426 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:31.451 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:31.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:31.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:31.552 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:31.590 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:31.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:31.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:31.692 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:31.725 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:31.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:31.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:31.827 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:31 smithi160 ceph-mon[101879]: osdmap e1313: 8 total, 8 up, 6 in 2024-09-21T10:40:31.854 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:31.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:31.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:31 smithi070 ceph-mon[107385]: osdmap e1313: 8 total, 8 up, 6 in 2024-09-21T10:40:31.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:31 smithi070 ceph-mon[112139]: osdmap e1313: 8 total, 8 up, 6 in 2024-09-21T10:40:31.956 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:31.991 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:31.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:31.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:32.093 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:32.111 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 8 16 20056 20040 156.526 48.375 0.00701332 0.006382 2024-09-21T10:40:32.132 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:32.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:32.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:32.233 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:32.268 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:32.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:32.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:32.370 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:32.407 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:32.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:32.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:32.509 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:32.542 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:32.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:32.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:32.644 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:32.680 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:32.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:32.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:32.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:32 smithi070 ceph-mon[107385]: pgmap v2162: 89 pgs: 7 peering, 1 unknown, 81 active+clean; 745 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-21T10:40:32.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:32 smithi070 ceph-mon[107385]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:40:32.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:32 smithi070 ceph-mon[107385]: osdmap e1314: 8 total, 8 up, 6 in 2024-09-21T10:40:32.740 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:32 smithi070 ceph-mon[112139]: pgmap v2162: 89 pgs: 7 peering, 1 unknown, 81 active+clean; 745 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-21T10:40:32.740 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:32 smithi070 ceph-mon[112139]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:40:32.740 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:32 smithi070 ceph-mon[112139]: osdmap e1314: 8 total, 8 up, 6 in 2024-09-21T10:40:32.782 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:32.816 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:32.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:32.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:32 smithi160 ceph-mon[101879]: pgmap v2162: 89 pgs: 7 peering, 1 unknown, 81 active+clean; 745 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-21T10:40:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:32 smithi160 ceph-mon[101879]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:40:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:32 smithi160 ceph-mon[101879]: osdmap e1314: 8 total, 8 up, 6 in 2024-09-21T10:40:32.918 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:32.952 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:32.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:32.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:33.053 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:33.068 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:40:33.068 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T10:40:33.068 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T10:40:33.068 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T10:40:33.068 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.068 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.34860.0:1592 51.1a 51:589aa541:::rbd_data.882c2f9b5d6.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1071)", 2024-09-21T10:40:33.068 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:32:08.622391+0000", 2024-09-21T10:40:33.068 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 504.43942229700002, 2024-09-21T10:40:33.068 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.040921918000000002, 2024-09-21T10:40:33.069 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:40:33.069 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:40:33.069 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:33.069 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:40:33.069 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.34860", 2024-09-21T10:40:33.069 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/4090022452", 2024-09-21T10:40:33.069 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 1592 2024-09-21T10:40:33.069 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.069 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:40:33.069 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.070 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:40:33.070 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:08.622391+0000", 2024-09-21T10:40:33.070 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.070 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.070 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.070 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:40:33.070 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:08.622391+0000", 2024-09-21T10:40:33.070 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.070 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.070 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.070 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:40:33.070 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:08.622393+0000", 2024-09-21T10:40:33.071 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.556e-06 2024-09-21T10:40:33.071 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.071 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.071 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:40:33.071 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:08.625948+0000", 2024-09-21T10:40:33.071 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035545849999999999 2024-09-21T10:40:33.071 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.071 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.071 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:40:33.071 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:08.625969+0000", 2024-09-21T10:40:33.071 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1005e-05 2024-09-21T10:40:33.071 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.071 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.071 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:40:33.071 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:08.625978+0000", 2024-09-21T10:40:33.072 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1549999999999996e-06 2024-09-21T10:40:33.072 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.072 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.072 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:40:33.072 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:08.626033+0000", 2024-09-21T10:40:33.072 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5414999999999997e-05 2024-09-21T10:40:33.072 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.072 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.072 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:40:33.072 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:08.626141+0000", 2024-09-21T10:40:33.072 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000107475 2024-09-21T10:40:33.072 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.072 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.072 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:40:33.073 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:08.626281+0000", 2024-09-21T10:40:33.073 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014034200000000001 2024-09-21T10:40:33.073 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.073 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.073 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:40:33.073 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:08.633255+0000", 2024-09-21T10:40:33.073 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0069743920000000003 2024-09-21T10:40:33.073 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.073 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.073 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.073 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:08.663150+0000", 2024-09-21T10:40:33.073 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.029894548999999999 2024-09-21T10:40:33.073 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.073 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.073 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.073 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:08.663234+0000", 2024-09-21T10:40:33.074 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.4550999999999997e-05 2024-09-21T10:40:33.074 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.074 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.074 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:40:33.074 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:08.663271+0000", 2024-09-21T10:40:33.074 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6368999999999997e-05 2024-09-21T10:40:33.074 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.074 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.074 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:40:33.074 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:08.663313+0000", 2024-09-21T10:40:33.074 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2524000000000001e-05 2024-09-21T10:40:33.074 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.074 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:40:33.074 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.075 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.075 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.075 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.34860.0:1921 51.1a 51:58003ff3:::rbd_data.882c2fe5d61b.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1077)", 2024-09-21T10:40:33.075 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:32:28.576286+0000", 2024-09-21T10:40:33.075 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 484.48552764800002, 2024-09-21T10:40:33.075 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.052974457000000003, 2024-09-21T10:40:33.075 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:40:33.075 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:40:33.075 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:33.075 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:40:33.075 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.34860", 2024-09-21T10:40:33.076 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/4090022452", 2024-09-21T10:40:33.076 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 1921 2024-09-21T10:40:33.076 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.076 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:40:33.076 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.076 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:40:33.076 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:28.576286+0000", 2024-09-21T10:40:33.076 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.076 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.076 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.076 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:40:33.076 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:28.576286+0000", 2024-09-21T10:40:33.076 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.077 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.077 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.077 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:40:33.077 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:28.576290+0000", 2024-09-21T10:40:33.077 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7510000000000002e-06 2024-09-21T10:40:33.077 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.077 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.077 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:40:33.077 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:28.579537+0000", 2024-09-21T10:40:33.077 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0032471079999999999 2024-09-21T10:40:33.077 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.077 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.077 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:40:33.077 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:28.579563+0000", 2024-09-21T10:40:33.077 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6446999999999999e-05 2024-09-21T10:40:33.078 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.078 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.078 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:40:33.078 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:28.579573+0000", 2024-09-21T10:40:33.078 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9490000000000008e-06 2024-09-21T10:40:33.078 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.078 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.078 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:40:33.078 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:28.579650+0000", 2024-09-21T10:40:33.078 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6301e-05 2024-09-21T10:40:33.078 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.078 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.078 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:40:33.078 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:28.579802+0000", 2024-09-21T10:40:33.078 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000152578 2024-09-21T10:40:33.079 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.079 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.079 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:40:33.079 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:28.579962+0000", 2024-09-21T10:40:33.079 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000159603 2024-09-21T10:40:33.079 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.079 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.079 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:40:33.079 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:28.588612+0000", 2024-09-21T10:40:33.079 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0086496679999999992 2024-09-21T10:40:33.079 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.079 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.079 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.079 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:28.602471+0000", 2024-09-21T10:40:33.079 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.013859674000000001 2024-09-21T10:40:33.080 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.080 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.080 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.080 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:28.629153+0000", 2024-09-21T10:40:33.080 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.026681852999999998 2024-09-21T10:40:33.080 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.080 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.080 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:40:33.080 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:28.629206+0000", 2024-09-21T10:40:33.080 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3018e-05 2024-09-21T10:40:33.080 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.080 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.080 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:40:33.080 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:32:28.629261+0000", 2024-09-21T10:40:33.080 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4506999999999998e-05 2024-09-21T10:40:33.081 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.081 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:40:33.081 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.081 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.081 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.081 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.44795.0:2337 51.1e 51:7cf33e09:::rbd_data.aefbbc05505b.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1098)", 2024-09-21T10:40:33.081 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:33:36.635117+0000", 2024-09-21T10:40:33.081 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 416.42669722900001, 2024-09-21T10:40:33.081 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.040840485000000003, 2024-09-21T10:40:33.081 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:40:33.081 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:40:33.081 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:33.081 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:40:33.082 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.44795", 2024-09-21T10:40:33.082 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/189219855", 2024-09-21T10:40:33.082 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 2337 2024-09-21T10:40:33.082 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.082 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:40:33.082 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.082 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:40:33.082 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:36.635117+0000", 2024-09-21T10:40:33.082 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.082 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.082 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.082 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:40:33.083 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:36.635117+0000", 2024-09-21T10:40:33.083 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.083 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.083 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.083 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:40:33.083 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:36.635120+0000", 2024-09-21T10:40:33.083 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6940000000000001e-06 2024-09-21T10:40:33.083 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.083 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.083 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:40:33.083 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:36.638433+0000", 2024-09-21T10:40:33.083 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0033128889999999999 2024-09-21T10:40:33.083 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.083 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.083 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:40:33.084 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:36.638465+0000", 2024-09-21T10:40:33.084 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2305999999999999e-05 2024-09-21T10:40:33.084 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.084 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.084 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:40:33.084 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:36.638489+0000", 2024-09-21T10:40:33.084 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3434e-05 2024-09-21T10:40:33.084 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.084 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.084 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:40:33.084 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:36.638572+0000", 2024-09-21T10:40:33.084 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2805000000000004e-05 2024-09-21T10:40:33.084 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.084 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.084 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:40:33.084 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:36.638735+0000", 2024-09-21T10:40:33.085 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000163497 2024-09-21T10:40:33.085 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.085 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.085 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:40:33.085 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:36.638903+0000", 2024-09-21T10:40:33.085 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000167813 2024-09-21T10:40:33.085 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.085 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.085 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:40:33.085 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:36.649282+0000", 2024-09-21T10:40:33.085 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.010378528999999999 2024-09-21T10:40:33.085 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.085 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.085 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.085 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:36.649720+0000", 2024-09-21T10:40:33.086 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00043815 2024-09-21T10:40:33.086 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.086 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.086 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.086 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:36.675892+0000", 2024-09-21T10:40:33.086 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.026172244000000001 2024-09-21T10:40:33.086 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.086 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.086 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:40:33.086 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:36.675929+0000", 2024-09-21T10:40:33.086 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7347e-05 2024-09-21T10:40:33.086 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.086 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.087 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:40:33.087 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:36.675957+0000", 2024-09-21T10:40:33.087 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7776999999999998e-05 2024-09-21T10:40:33.087 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.087 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:40:33.087 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.087 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.087 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.087 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.44795.0:3102 51.1a 51:5aeb1405:::rbd_data.aefbe2c9e6d6.000000000000004d:head [set-alloc-hint object_size 4194304 write_size 4194304,write 227958~3910707] snapc 67=[67] ondisk+write+known_if_redirected+supports_pool_eio e1111)", 2024-09-21T10:40:33.087 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:33:49.714869+0000", 2024-09-21T10:40:33.087 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 403.34694460999998, 2024-09-21T10:40:33.087 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.038357652999999998, 2024-09-21T10:40:33.087 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:40:33.087 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:40:33.088 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:33.088 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:40:33.088 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.44795", 2024-09-21T10:40:33.088 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/189219855", 2024-09-21T10:40:33.088 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 3102 2024-09-21T10:40:33.088 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.088 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:40:33.088 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.088 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:40:33.088 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:49.714869+0000", 2024-09-21T10:40:33.088 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.088 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.088 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.088 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:40:33.089 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:49.714869+0000", 2024-09-21T10:40:33.089 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.089 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.089 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.089 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:40:33.089 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:49.714872+0000", 2024-09-21T10:40:33.089 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.039e-06 2024-09-21T10:40:33.089 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.089 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.089 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:40:33.089 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:49.721456+0000", 2024-09-21T10:40:33.089 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0065834070000000003 2024-09-21T10:40:33.089 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.090 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.090 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:40:33.090 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:49.721508+0000", 2024-09-21T10:40:33.090 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1997999999999999e-05 2024-09-21T10:40:33.090 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.090 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.090 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:40:33.090 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:49.721535+0000", 2024-09-21T10:40:33.090 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7690000000000001e-05 2024-09-21T10:40:33.090 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.090 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.090 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:40:33.090 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:49.721809+0000", 2024-09-21T10:40:33.090 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00027415700000000002 2024-09-21T10:40:33.091 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.091 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.091 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:40:33.091 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:49.722027+0000", 2024-09-21T10:40:33.091 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00021723200000000001 2024-09-21T10:40:33.091 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.091 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.091 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:40:33.091 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:49.722266+0000", 2024-09-21T10:40:33.091 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00023902299999999999 2024-09-21T10:40:33.091 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.091 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.091 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:40:33.091 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:49.736826+0000", 2024-09-21T10:40:33.092 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.014560066 2024-09-21T10:40:33.092 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.092 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.092 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.092 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:49.752943+0000", 2024-09-21T10:40:33.092 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0161175 2024-09-21T10:40:33.092 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.092 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.092 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.092 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:49.753133+0000", 2024-09-21T10:40:33.092 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018940600000000001 2024-09-21T10:40:33.092 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.092 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.092 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:40:33.092 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:49.753176+0000", 2024-09-21T10:40:33.093 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3380000000000001e-05 2024-09-21T10:40:33.093 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.093 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.093 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:40:33.093 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:49.753227+0000", 2024-09-21T10:40:33.093 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0754999999999998e-05 2024-09-21T10:40:33.093 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.093 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:40:33.093 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.093 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.093 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.093 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.44795.0:3217 51.b 51:d6a6563d:::rbd_data.aefbe2c9e6d6.0000000000000040:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 69=[69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1113)", 2024-09-21T10:40:33.093 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:33:51.735589+0000", 2024-09-21T10:40:33.094 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 401.326225131, 2024-09-21T10:40:33.094 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.036701536, 2024-09-21T10:40:33.094 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:40:33.094 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:40:33.094 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:33.094 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:40:33.094 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.44795", 2024-09-21T10:40:33.094 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/189219855", 2024-09-21T10:40:33.094 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 3217 2024-09-21T10:40:33.094 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.094 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:40:33.094 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.094 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:40:33.094 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:51.735589+0000", 2024-09-21T10:40:33.094 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.095 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.095 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.095 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:40:33.095 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:51.735589+0000", 2024-09-21T10:40:33.095 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.095 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.095 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.095 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:40:33.095 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:51.735592+0000", 2024-09-21T10:40:33.096 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8399999999999997e-06 2024-09-21T10:40:33.096 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.096 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.096 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:40:33.096 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:51.741524+0000", 2024-09-21T10:40:33.096 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0059313539999999998 2024-09-21T10:40:33.096 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.096 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.096 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:40:33.096 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:51.741550+0000", 2024-09-21T10:40:33.096 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6081999999999999e-05 2024-09-21T10:40:33.096 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.096 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.096 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:40:33.097 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:51.741563+0000", 2024-09-21T10:40:33.097 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2755000000000001e-05 2024-09-21T10:40:33.097 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.097 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.097 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:40:33.097 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:51.741668+0000", 2024-09-21T10:40:33.097 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010556999999999999 2024-09-21T10:40:33.097 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.097 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.097 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:40:33.097 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:51.741849+0000", 2024-09-21T10:40:33.097 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018116700000000001 2024-09-21T10:40:33.097 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.097 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.097 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:40:33.098 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:51.742018+0000", 2024-09-21T10:40:33.098 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0001683 2024-09-21T10:40:33.098 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.098 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.098 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:40:33.098 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:51.755958+0000", 2024-09-21T10:40:33.098 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.013940406000000001 2024-09-21T10:40:33.098 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.098 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.098 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.098 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:51.770230+0000", 2024-09-21T10:40:33.098 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.014271781000000001 2024-09-21T10:40:33.098 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.099 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.099 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.099 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:51.772197+0000", 2024-09-21T10:40:33.099 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0019669879999999998 2024-09-21T10:40:33.099 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.099 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.099 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:40:33.099 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:51.772255+0000", 2024-09-21T10:40:33.099 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8554999999999998e-05 2024-09-21T10:40:33.099 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.099 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.099 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:40:33.099 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:51.772290+0000", 2024-09-21T10:40:33.099 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4737999999999998e-05 2024-09-21T10:40:33.100 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.100 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:40:33.100 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.100 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.100 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.100 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.44795.0:3269 51.13 51:cdd49107:::rbd_data.aefbe2c9e6d6.0000000000000011:head [write 632451~3561853] snapc 6a=[6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1114)", 2024-09-21T10:40:33.100 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:33:52.734734+0000", 2024-09-21T10:40:33.100 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 400.32708004199998, 2024-09-21T10:40:33.100 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.045165453000000001, 2024-09-21T10:40:33.100 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:40:33.100 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:40:33.101 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:33.101 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:40:33.101 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.44795", 2024-09-21T10:40:33.101 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/189219855", 2024-09-21T10:40:33.101 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 3269 2024-09-21T10:40:33.101 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.101 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:40:33.101 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.101 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:40:33.101 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.734734+0000", 2024-09-21T10:40:33.101 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.101 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.101 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.101 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:40:33.101 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.734734+0000", 2024-09-21T10:40:33.102 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.102 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.102 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.102 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:40:33.102 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.734737+0000", 2024-09-21T10:40:33.102 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9179999999999998e-06 2024-09-21T10:40:33.102 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.102 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.102 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:40:33.102 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.738343+0000", 2024-09-21T10:40:33.102 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0036063940000000002 2024-09-21T10:40:33.102 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.102 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.103 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:40:33.103 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.738385+0000", 2024-09-21T10:40:33.103 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.1794e-05 2024-09-21T10:40:33.103 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.103 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.103 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:40:33.103 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.738398+0000", 2024-09-21T10:40:33.103 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3495e-05 2024-09-21T10:40:33.103 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.103 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.103 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:40:33.103 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.738551+0000", 2024-09-21T10:40:33.103 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015263399999999999 2024-09-21T10:40:33.103 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.103 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.103 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:40:33.104 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.738684+0000", 2024-09-21T10:40:33.104 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000133366 2024-09-21T10:40:33.104 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.104 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.104 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:40:33.104 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.738952+0000", 2024-09-21T10:40:33.104 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00026774399999999999 2024-09-21T10:40:33.104 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.104 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.104 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:40:33.104 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.752539+0000", 2024-09-21T10:40:33.104 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.013587299000000001 2024-09-21T10:40:33.104 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.104 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.104 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.105 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.752801+0000", 2024-09-21T10:40:33.105 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00026154599999999998 2024-09-21T10:40:33.105 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.105 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.105 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.105 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.779744+0000", 2024-09-21T10:40:33.105 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.026943478 2024-09-21T10:40:33.105 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.105 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.105 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:40:33.105 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.779833+0000", 2024-09-21T10:40:33.105 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9017000000000005e-05 2024-09-21T10:40:33.105 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.105 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.105 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:40:33.105 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.779899+0000", 2024-09-21T10:40:33.106 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5767999999999998e-05 2024-09-21T10:40:33.106 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.106 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:40:33.106 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.106 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.106 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.106 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.44795.0:3271 51.b 51:d6a6563d:::rbd_data.aefbe2c9e6d6.0000000000000040:head [write 471789~1902889] snapc 6a=[6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1114)", 2024-09-21T10:40:33.106 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:33:52.738431+0000", 2024-09-21T10:40:33.106 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 400.32338258200002, 2024-09-21T10:40:33.106 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.041151710000000001, 2024-09-21T10:40:33.106 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:40:33.106 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:40:33.106 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:33.106 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:40:33.106 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.44795", 2024-09-21T10:40:33.107 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/189219855", 2024-09-21T10:40:33.107 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 3271 2024-09-21T10:40:33.107 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.107 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:40:33.107 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.107 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:40:33.107 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.738431+0000", 2024-09-21T10:40:33.107 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.107 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.107 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.107 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:40:33.107 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.738431+0000", 2024-09-21T10:40:33.108 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.108 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.108 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.108 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:40:33.108 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.738434+0000", 2024-09-21T10:40:33.108 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3580000000000001e-06 2024-09-21T10:40:33.108 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.108 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.108 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:40:33.108 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.740290+0000", 2024-09-21T10:40:33.108 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0018565089999999999 2024-09-21T10:40:33.108 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.108 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.108 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:40:33.108 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.740316+0000", 2024-09-21T10:40:33.109 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5786e-05 2024-09-21T10:40:33.109 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.109 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.109 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:40:33.109 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.740366+0000", 2024-09-21T10:40:33.109 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9897e-05 2024-09-21T10:40:33.109 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.109 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.109 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:40:33.109 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.740460+0000", 2024-09-21T10:40:33.109 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4282000000000003e-05 2024-09-21T10:40:33.109 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.109 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.109 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:40:33.110 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.740583+0000", 2024-09-21T10:40:33.110 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012258700000000001 2024-09-21T10:40:33.110 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.110 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.110 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:40:33.110 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.740822+0000", 2024-09-21T10:40:33.110 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00023939100000000001 2024-09-21T10:40:33.110 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.110 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.110 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:40:33.110 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.749179+0000", 2024-09-21T10:40:33.110 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0083573769999999992 2024-09-21T10:40:33.110 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.111 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.111 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.111 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.773913+0000", 2024-09-21T10:40:33.111 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.024734011 2024-09-21T10:40:33.111 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.111 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.111 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.111 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.779508+0000", 2024-09-21T10:40:33.111 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.005594679 2024-09-21T10:40:33.111 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.111 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.111 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:40:33.111 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.779550+0000", 2024-09-21T10:40:33.112 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2188999999999999e-05 2024-09-21T10:40:33.112 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.112 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.112 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:40:33.112 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:52.779583+0000", 2024-09-21T10:40:33.112 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2644000000000001e-05 2024-09-21T10:40:33.112 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.112 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:40:33.112 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.112 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.112 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.112 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.44795.0:3592 51.b 51:d6a6563d:::rbd_data.aefbe2c9e6d6.0000000000000040:head [write 1142098~3052206] snapc 70=[70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1120)", 2024-09-21T10:40:33.112 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:33:58.782081+0000", 2024-09-21T10:40:33.112 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 394.27973303099998, 2024-09-21T10:40:33.112 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.064595633999999999, 2024-09-21T10:40:33.112 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:40:33.113 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:40:33.113 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:33.113 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:40:33.113 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.44795", 2024-09-21T10:40:33.113 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/189219855", 2024-09-21T10:40:33.113 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 3592 2024-09-21T10:40:33.113 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.113 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:40:33.113 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.113 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:40:33.113 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:58.782081+0000", 2024-09-21T10:40:33.113 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.113 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.113 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.113 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:40:33.114 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:58.782081+0000", 2024-09-21T10:40:33.114 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.114 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.114 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.114 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:40:33.114 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:58.782085+0000", 2024-09-21T10:40:33.114 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5390000000000001e-06 2024-09-21T10:40:33.114 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.114 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.114 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:40:33.114 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:58.784729+0000", 2024-09-21T10:40:33.114 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002644197 2024-09-21T10:40:33.114 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.115 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.115 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:40:33.115 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:58.784748+0000", 2024-09-21T10:40:33.115 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8810000000000001e-05 2024-09-21T10:40:33.115 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.115 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.115 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:40:33.115 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:58.784758+0000", 2024-09-21T10:40:33.115 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.6930000000000002e-06 2024-09-21T10:40:33.115 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.115 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.115 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:40:33.115 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:58.784836+0000", 2024-09-21T10:40:33.115 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7645999999999995e-05 2024-09-21T10:40:33.115 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.116 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.116 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:40:33.116 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:58.784966+0000", 2024-09-21T10:40:33.116 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013045400000000001 2024-09-21T10:40:33.116 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.116 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.116 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:40:33.116 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:58.785202+0000", 2024-09-21T10:40:33.116 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00023629799999999999 2024-09-21T10:40:33.116 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.116 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.116 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:40:33.116 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:58.793933+0000", 2024-09-21T10:40:33.116 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0087305060000000007 2024-09-21T10:40:33.116 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.117 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.117 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.117 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:58.803631+0000", 2024-09-21T10:40:33.117 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0096983060000000003 2024-09-21T10:40:33.117 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.117 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.117 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.117 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:58.846592+0000", 2024-09-21T10:40:33.117 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.042961199999999998 2024-09-21T10:40:33.117 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.117 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.117 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:40:33.117 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:58.846635+0000", 2024-09-21T10:40:33.118 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2500000000000003e-05 2024-09-21T10:40:33.118 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.118 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.118 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:40:33.118 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:58.846676+0000", 2024-09-21T10:40:33.118 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.1485e-05 2024-09-21T10:40:33.118 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.118 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:40:33.118 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.118 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.118 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.118 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.44795.0:3660 51.13 51:cf7d7388:::rbd_data.aefbe2c9e6d6.0000000000000008:head [writefull 0~4194304] snapc 71=[71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1121)", 2024-09-21T10:40:33.118 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:33:59.813686+0000", 2024-09-21T10:40:33.119 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 393.248128141, 2024-09-21T10:40:33.119 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.039902307999999997, 2024-09-21T10:40:33.119 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:40:33.119 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:40:33.119 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:33.119 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:40:33.119 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.44795", 2024-09-21T10:40:33.119 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/189219855", 2024-09-21T10:40:33.119 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 3660 2024-09-21T10:40:33.119 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.119 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:40:33.119 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.119 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:40:33.119 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:59.813686+0000", 2024-09-21T10:40:33.119 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.120 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.120 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.120 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:40:33.120 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:59.813686+0000", 2024-09-21T10:40:33.120 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.120 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.120 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.120 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:40:33.120 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:59.813688+0000", 2024-09-21T10:40:33.120 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3590000000000002e-06 2024-09-21T10:40:33.120 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.120 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.120 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:40:33.120 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:59.816178+0000", 2024-09-21T10:40:33.120 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0024904430000000002 2024-09-21T10:40:33.120 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.121 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.121 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:40:33.121 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:59.816202+0000", 2024-09-21T10:40:33.121 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3845999999999998e-05 2024-09-21T10:40:33.121 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.121 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.121 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:40:33.121 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:59.816213+0000", 2024-09-21T10:40:33.121 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0922e-05 2024-09-21T10:40:33.121 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.121 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.121 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:40:33.121 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:59.816309+0000", 2024-09-21T10:40:33.122 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.6161000000000003e-05 2024-09-21T10:40:33.122 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.122 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.122 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:40:33.122 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:59.816446+0000", 2024-09-21T10:40:33.122 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013662200000000001 2024-09-21T10:40:33.122 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.122 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.122 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:40:33.122 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:59.816663+0000", 2024-09-21T10:40:33.123 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00021656400000000001 2024-09-21T10:40:33.123 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.123 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.123 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.123 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:59.828791+0000", 2024-09-21T10:40:33.123 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012128328000000001 2024-09-21T10:40:33.123 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.123 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.123 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:40:33.123 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:59.832080+0000", 2024-09-21T10:40:33.123 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0032889600000000001 2024-09-21T10:40:33.123 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.123 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.123 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.124 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:59.853492+0000", 2024-09-21T10:40:33.124 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.021412352999999999 2024-09-21T10:40:33.124 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.124 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.124 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:40:33.124 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:59.853549+0000", 2024-09-21T10:40:33.124 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6468000000000002e-05 2024-09-21T10:40:33.124 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.124 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.124 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:40:33.124 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:33:59.853588+0000", 2024-09-21T10:40:33.124 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9282000000000001e-05 2024-09-21T10:40:33.124 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.124 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:40:33.124 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.125 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.125 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.125 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.44795.0:3769 51.13 51:cdde7d83:::rbd_data.aefbe2c9e6d6.000000000000004b:head [write 550329~3037979] snapc 73=[73,72,71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-09-21T10:40:33.125 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:34:01.803428+0000", 2024-09-21T10:40:33.125 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 391.25838566499999, 2024-09-21T10:40:33.125 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.037020803999999997, 2024-09-21T10:40:33.125 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:40:33.125 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:40:33.125 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:33.125 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:40:33.125 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.44795", 2024-09-21T10:40:33.125 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/189219855", 2024-09-21T10:40:33.125 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 3769 2024-09-21T10:40:33.125 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.126 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:40:33.126 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.126 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:40:33.126 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:01.803428+0000", 2024-09-21T10:40:33.126 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.126 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.126 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.126 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:40:33.126 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:01.803428+0000", 2024-09-21T10:40:33.126 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.126 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.126 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.126 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:40:33.126 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:01.803430+0000", 2024-09-21T10:40:33.126 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9709999999999998e-06 2024-09-21T10:40:33.127 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.127 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.127 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:40:33.127 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:01.805097+0000", 2024-09-21T10:40:33.127 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001667373 2024-09-21T10:40:33.127 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.127 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.127 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:40:33.127 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:01.805113+0000", 2024-09-21T10:40:33.127 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5837000000000001e-05 2024-09-21T10:40:33.127 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.127 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.127 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:40:33.127 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:01.805125+0000", 2024-09-21T10:40:33.128 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.172e-05 2024-09-21T10:40:33.128 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.128 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.128 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:40:33.128 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:01.805206+0000", 2024-09-21T10:40:33.128 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0767999999999996e-05 2024-09-21T10:40:33.128 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.128 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.128 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:40:33.128 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:01.805339+0000", 2024-09-21T10:40:33.128 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000133621 2024-09-21T10:40:33.128 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.128 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.129 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:40:33.129 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:01.805545+0000", 2024-09-21T10:40:33.129 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00020552299999999999 2024-09-21T10:40:33.129 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.129 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.129 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:40:33.129 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:01.811501+0000", 2024-09-21T10:40:33.129 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0059564029999999999 2024-09-21T10:40:33.129 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.129 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.129 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.129 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:01.812079+0000", 2024-09-21T10:40:33.129 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00057757100000000003 2024-09-21T10:40:33.130 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.130 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.130 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.130 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:01.840310+0000", 2024-09-21T10:40:33.130 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.028231216 2024-09-21T10:40:33.130 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.130 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.130 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:40:33.130 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:01.840389+0000", 2024-09-21T10:40:33.130 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8591999999999996e-05 2024-09-21T10:40:33.130 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.130 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.130 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:40:33.131 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:01.840449+0000", 2024-09-21T10:40:33.131 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0208999999999999e-05 2024-09-21T10:40:33.131 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.131 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:40:33.131 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.131 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.131 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.131 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.44795.0:40751 51.4 51:24def725:::rbd_data.aefb6da59c7e.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1149)", 2024-09-21T10:40:33.131 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:34:28.090468+0000", 2024-09-21T10:40:33.131 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 364.97134552300002, 2024-09-21T10:40:33.131 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.040917850999999998, 2024-09-21T10:40:33.131 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:40:33.131 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:40:33.132 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:33.132 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:40:33.132 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.44795", 2024-09-21T10:40:33.132 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/189219855", 2024-09-21T10:40:33.132 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 40751 2024-09-21T10:40:33.132 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.132 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:40:33.132 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.132 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:40:33.132 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:28.090468+0000", 2024-09-21T10:40:33.132 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.132 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.132 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.133 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:40:33.133 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:28.090468+0000", 2024-09-21T10:40:33.133 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.133 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.133 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.133 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:40:33.133 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:28.090470+0000", 2024-09-21T10:40:33.133 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7770000000000001e-06 2024-09-21T10:40:33.133 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.133 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.133 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:40:33.133 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:28.092707+0000", 2024-09-21T10:40:33.133 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0022373900000000001 2024-09-21T10:40:33.133 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.133 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.134 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:40:33.134 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:28.092726+0000", 2024-09-21T10:40:33.134 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8754e-05 2024-09-21T10:40:33.134 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.134 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.134 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:40:33.134 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:28.092735+0000", 2024-09-21T10:40:33.134 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2620000000000004e-06 2024-09-21T10:40:33.134 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.134 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.134 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:40:33.134 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:28.092800+0000", 2024-09-21T10:40:33.134 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4871999999999998e-05 2024-09-21T10:40:33.134 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.135 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.135 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:40:33.135 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:28.092927+0000", 2024-09-21T10:40:33.135 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012647 2024-09-21T10:40:33.135 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.135 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.135 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 2,4", 2024-09-21T10:40:33.135 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:28.093073+0000", 2024-09-21T10:40:33.135 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000145919 2024-09-21T10:40:33.135 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.135 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.135 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:40:33.135 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:28.102118+0000", 2024-09-21T10:40:33.135 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0090451980000000008 2024-09-21T10:40:33.135 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.135 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.136 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.136 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:28.113670+0000", 2024-09-21T10:40:33.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:33.136 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:33.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:33.137 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.011552312 2024-09-21T10:40:33.137 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.137 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.137 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.137 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:28.131241+0000", 2024-09-21T10:40:33.137 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.01757125 2024-09-21T10:40:33.137 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.137 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.137 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:40:33.137 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:28.131331+0000", 2024-09-21T10:40:33.138 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9166000000000007e-05 2024-09-21T10:40:33.138 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.138 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.138 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:40:33.138 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:28.131386+0000", 2024-09-21T10:40:33.138 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5481e-05 2024-09-21T10:40:33.138 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.138 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:40:33.138 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.138 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.138 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.138 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.44795.0:40937 51.1a 51:5e94397d:::rbd_data.aefbd082a7e9.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1150)", 2024-09-21T10:40:33.139 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:34:29.222116+0000", 2024-09-21T10:40:33.139 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 363.83969803500003, 2024-09-21T10:40:33.139 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.055392023999999998, 2024-09-21T10:40:33.139 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:40:33.139 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:40:33.139 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:33.139 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:40:33.139 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.44795", 2024-09-21T10:40:33.139 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/189219855", 2024-09-21T10:40:33.139 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 40937 2024-09-21T10:40:33.139 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.139 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:40:33.139 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.139 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:40:33.140 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.222116+0000", 2024-09-21T10:40:33.140 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.140 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.140 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.140 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:40:33.140 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.222116+0000", 2024-09-21T10:40:33.140 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.140 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.140 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.140 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:40:33.140 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.222118+0000", 2024-09-21T10:40:33.140 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2010000000000002e-06 2024-09-21T10:40:33.140 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.140 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.141 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:40:33.141 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.224943+0000", 2024-09-21T10:40:33.141 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0028250430000000002 2024-09-21T10:40:33.141 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.141 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.141 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:40:33.141 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.224992+0000", 2024-09-21T10:40:33.141 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8578999999999997e-05 2024-09-21T10:40:33.141 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.141 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.141 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:40:33.141 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.225013+0000", 2024-09-21T10:40:33.141 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1563e-05 2024-09-21T10:40:33.142 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.142 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.142 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:40:33.142 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.225123+0000", 2024-09-21T10:40:33.142 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000110363 2024-09-21T10:40:33.142 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.142 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.142 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:40:33.142 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.225249+0000", 2024-09-21T10:40:33.142 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012504700000000001 2024-09-21T10:40:33.142 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.142 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.142 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:40:33.143 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.225378+0000", 2024-09-21T10:40:33.143 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000129014 2024-09-21T10:40:33.143 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.143 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.143 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:40:33.143 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.235915+0000", 2024-09-21T10:40:33.143 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.010537202000000001 2024-09-21T10:40:33.143 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.143 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.143 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.143 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.276055+0000", 2024-09-21T10:40:33.143 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.040140215 2024-09-21T10:40:33.143 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.144 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.144 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.144 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.277357+0000", 2024-09-21T10:40:33.144 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0013018140000000001 2024-09-21T10:40:33.144 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.144 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.144 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:40:33.144 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.277461+0000", 2024-09-21T10:40:33.144 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000103915 2024-09-21T10:40:33.144 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.144 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.144 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:40:33.144 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.277508+0000", 2024-09-21T10:40:33.144 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7067999999999997e-05 2024-09-21T10:40:33.144 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.144 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:40:33.145 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.145 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.145 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.145 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.44795.0:40940 51.4 51:22640140:::rbd_data.aefbd082a7e9.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1150)", 2024-09-21T10:40:33.145 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:34:29.225065+0000", 2024-09-21T10:40:33.145 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 363.83674883700002, 2024-09-21T10:40:33.145 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.051087992999999998, 2024-09-21T10:40:33.145 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:40:33.145 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:40:33.145 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:33.145 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:40:33.145 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.44795", 2024-09-21T10:40:33.145 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/189219855", 2024-09-21T10:40:33.145 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 40940 2024-09-21T10:40:33.146 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.146 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:40:33.146 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.146 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:40:33.146 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.225065+0000", 2024-09-21T10:40:33.146 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.146 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.146 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.146 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:40:33.146 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.225065+0000", 2024-09-21T10:40:33.146 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.146 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.146 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.146 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:40:33.147 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.225067+0000", 2024-09-21T10:40:33.147 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5160000000000001e-06 2024-09-21T10:40:33.147 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.147 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.147 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:40:33.147 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.228006+0000", 2024-09-21T10:40:33.147 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0029381839999999999 2024-09-21T10:40:33.147 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.147 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.147 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:40:33.147 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.228041+0000", 2024-09-21T10:40:33.147 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4879000000000003e-05 2024-09-21T10:40:33.147 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.147 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.148 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:40:33.148 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.228054+0000", 2024-09-21T10:40:33.148 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3189e-05 2024-09-21T10:40:33.148 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.148 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.148 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:40:33.148 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.228145+0000", 2024-09-21T10:40:33.148 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0860999999999996e-05 2024-09-21T10:40:33.148 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.148 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.148 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:40:33.148 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.228337+0000", 2024-09-21T10:40:33.148 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00019225399999999999 2024-09-21T10:40:33.148 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.148 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.149 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 2,4", 2024-09-21T10:40:33.149 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.228521+0000", 2024-09-21T10:40:33.149 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018437400000000001 2024-09-21T10:40:33.149 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.149 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.149 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:40:33.149 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.237689+0000", 2024-09-21T10:40:33.149 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0091682280000000005 2024-09-21T10:40:33.149 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.149 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.149 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.149 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.238038+0000", 2024-09-21T10:40:33.149 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00034852800000000002 2024-09-21T10:40:33.149 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.149 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.150 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.150 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.276055+0000", 2024-09-21T10:40:33.150 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.038016933000000003 2024-09-21T10:40:33.150 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.150 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.150 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:40:33.150 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.276119+0000", 2024-09-21T10:40:33.150 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3899000000000006e-05 2024-09-21T10:40:33.150 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.150 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.150 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:40:33.151 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 9 16 22741 22725 157.778 167.812 0.00215025 0.00633107 2024-09-21T10:40:33.151 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:34:29.276153+0000", 2024-09-21T10:40:33.151 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4147999999999999e-05 2024-09-21T10:40:33.151 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.151 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:40:33.151 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.151 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.151 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.152 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.34884.0:2314 52.8 52:12908c8e:::rbd_data.88447b362a13.0000000000000045:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 69=[69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1262)", 2024-09-21T10:40:33.152 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:38:11.458075+0000", 2024-09-21T10:40:33.152 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 141.603738905, 2024-09-21T10:40:33.152 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.068254980000000007, 2024-09-21T10:40:33.152 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:40:33.152 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:40:33.152 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:33.152 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:40:33.152 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.34884", 2024-09-21T10:40:33.152 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1225916197", 2024-09-21T10:40:33.152 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 2314 2024-09-21T10:40:33.152 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.152 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:40:33.152 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.153 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:40:33.153 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:11.458075+0000", 2024-09-21T10:40:33.153 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.153 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.153 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.153 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:40:33.153 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:11.458075+0000", 2024-09-21T10:40:33.153 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.153 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.153 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.153 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:40:33.153 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:11.458076+0000", 2024-09-21T10:40:33.153 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.218e-06 2024-09-21T10:40:33.153 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.154 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.154 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:40:33.154 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:11.461312+0000", 2024-09-21T10:40:33.154 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0032355690000000002 2024-09-21T10:40:33.154 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.154 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.155 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:40:33.155 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:11.461332+0000", 2024-09-21T10:40:33.155 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9877e-05 2024-09-21T10:40:33.155 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.155 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.155 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:40:33.155 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:11.461339+0000", 2024-09-21T10:40:33.155 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6590000000000007e-06 2024-09-21T10:40:33.155 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.156 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.156 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:40:33.156 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:11.461477+0000", 2024-09-21T10:40:33.156 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000137403 2024-09-21T10:40:33.156 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.156 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.156 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:40:33.157 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:11.461628+0000", 2024-09-21T10:40:33.157 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015128299999999999 2024-09-21T10:40:33.157 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.157 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.157 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:40:33.157 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:11.461807+0000", 2024-09-21T10:40:33.157 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000179187 2024-09-21T10:40:33.157 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.158 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.158 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:40:33.158 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:11.469454+0000", 2024-09-21T10:40:33.158 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0076467729999999999 2024-09-21T10:40:33.158 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.158 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.158 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.158 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:11.506887+0000", 2024-09-21T10:40:33.158 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.037433084999999998 2024-09-21T10:40:33.158 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.158 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.158 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.158 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:11.526229+0000", 2024-09-21T10:40:33.159 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.019341957 2024-09-21T10:40:33.159 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.159 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.159 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:40:33.159 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:11.526290+0000", 2024-09-21T10:40:33.159 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1066000000000001e-05 2024-09-21T10:40:33.159 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.159 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.159 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:40:33.159 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:11.526330+0000", 2024-09-21T10:40:33.159 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9903000000000001e-05 2024-09-21T10:40:33.159 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.159 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:40:33.159 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.160 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.160 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.160 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.34884.0:2555 52.8 52:12908c8e:::rbd_data.88447b362a13.0000000000000045:head [write 1751728~2442576] snapc 6d=[6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1266)", 2024-09-21T10:40:33.160 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:38:15.524284+0000", 2024-09-21T10:40:33.160 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 137.53752977600001, 2024-09-21T10:40:33.160 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.040462181999999999, 2024-09-21T10:40:33.160 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:40:33.160 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:40:33.160 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:33.160 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:40:33.160 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.34884", 2024-09-21T10:40:33.160 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1225916197", 2024-09-21T10:40:33.160 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 2555 2024-09-21T10:40:33.160 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.160 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:40:33.161 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.161 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:40:33.161 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:15.524284+0000", 2024-09-21T10:40:33.161 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.161 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.161 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.161 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:40:33.161 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:15.524284+0000", 2024-09-21T10:40:33.161 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.161 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.162 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.162 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:40:33.162 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:15.524287+0000", 2024-09-21T10:40:33.162 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6010000000000002e-06 2024-09-21T10:40:33.162 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.162 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.162 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:40:33.162 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:15.526114+0000", 2024-09-21T10:40:33.162 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001827586 2024-09-21T10:40:33.162 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.162 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.163 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:40:33.163 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:15.526147+0000", 2024-09-21T10:40:33.163 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2724e-05 2024-09-21T10:40:33.163 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.163 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.163 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:40:33.163 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:15.526164+0000", 2024-09-21T10:40:33.163 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7036000000000001e-05 2024-09-21T10:40:33.163 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.163 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.163 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:40:33.163 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:15.526240+0000", 2024-09-21T10:40:33.164 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6428999999999996e-05 2024-09-21T10:40:33.164 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.164 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.164 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:40:33.164 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:15.526338+0000", 2024-09-21T10:40:33.164 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7873000000000003e-05 2024-09-21T10:40:33.164 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.164 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.164 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:40:33.164 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:15.526507+0000", 2024-09-21T10:40:33.164 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000169042 2024-09-21T10:40:33.164 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.164 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.164 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:40:33.164 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:15.534096+0000", 2024-09-21T10:40:33.165 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0075887200000000002 2024-09-21T10:40:33.165 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.165 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.165 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.165 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:15.538715+0000", 2024-09-21T10:40:33.165 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0046194419999999996 2024-09-21T10:40:33.165 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.165 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.165 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.165 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:15.564662+0000", 2024-09-21T10:40:33.165 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.025946477999999999 2024-09-21T10:40:33.165 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.165 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.165 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:40:33.165 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:15.564709+0000", 2024-09-21T10:40:33.165 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6721999999999998e-05 2024-09-21T10:40:33.166 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.166 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.166 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:40:33.166 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:15.564746+0000", 2024-09-21T10:40:33.166 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7528999999999999e-05 2024-09-21T10:40:33.166 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.166 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:40:33.166 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.166 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.166 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.166 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.34884.0:2915 52.1d 52:bfa7cec5:::rbd_data.88447b362a13.0000000000000052:head [write 1258264~2448500] snapc 74=[74,73,72,71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1273)", 2024-09-21T10:40:33.166 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:38:22.527310+0000", 2024-09-21T10:40:33.166 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 130.53450399100001, 2024-09-21T10:40:33.166 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.029950455000000001, 2024-09-21T10:40:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:40:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:40:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:40:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.34884", 2024-09-21T10:40:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1225916197", 2024-09-21T10:40:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 2915 2024-09-21T10:40:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:40:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:40:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.527310+0000", 2024-09-21T10:40:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.167 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.168 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:40:33.168 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.527310+0000", 2024-09-21T10:40:33.168 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.168 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.168 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.168 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:40:33.168 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.527312+0000", 2024-09-21T10:40:33.168 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4059999999999999e-06 2024-09-21T10:40:33.168 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.168 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.168 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:40:33.168 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.529507+0000", 2024-09-21T10:40:33.168 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0021945929999999999 2024-09-21T10:40:33.169 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.169 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.169 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:40:33.169 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.529528+0000", 2024-09-21T10:40:33.169 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1597e-05 2024-09-21T10:40:33.169 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.169 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.169 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:40:33.169 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.529540+0000", 2024-09-21T10:40:33.169 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2096000000000001e-05 2024-09-21T10:40:33.169 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.169 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.169 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:40:33.170 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.529613+0000", 2024-09-21T10:40:33.170 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3008000000000003e-05 2024-09-21T10:40:33.170 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.170 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.170 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:40:33.170 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.529774+0000", 2024-09-21T10:40:33.170 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016072999999999999 2024-09-21T10:40:33.170 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.170 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.170 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:40:33.170 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.529981+0000", 2024-09-21T10:40:33.170 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00020677299999999999 2024-09-21T10:40:33.170 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.171 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.171 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:40:33.171 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.539452+0000", 2024-09-21T10:40:33.171 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0094709760000000007 2024-09-21T10:40:33.171 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.171 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.171 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.171 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.554974+0000", 2024-09-21T10:40:33.171 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015522029999999999 2024-09-21T10:40:33.171 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.171 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.171 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.171 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.557171+0000", 2024-09-21T10:40:33.171 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0021971600000000001 2024-09-21T10:40:33.171 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.171 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.172 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:40:33.172 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.557215+0000", 2024-09-21T10:40:33.172 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.4150000000000003e-05 2024-09-21T10:40:33.172 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.172 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.172 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:40:33.172 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.557260+0000", 2024-09-21T10:40:33.172 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.4935999999999998e-05 2024-09-21T10:40:33.172 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.172 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:40:33.172 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.172 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.172 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.173 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.34884.0:2938 52.1d 52:bf520f90:::rbd_data.88447b362a13.0000000000000044:head [write 2828250~1366054] snapc 74=[74,73,72,71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1273)", 2024-09-21T10:40:33.173 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:38:22.565384+0000", 2024-09-21T10:40:33.173 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 130.49642999400001, 2024-09-21T10:40:33.173 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.031849553000000003, 2024-09-21T10:40:33.173 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:40:33.173 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:40:33.173 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:33.173 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:40:33.173 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.34884", 2024-09-21T10:40:33.173 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1225916197", 2024-09-21T10:40:33.173 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 2938 2024-09-21T10:40:33.173 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.173 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:40:33.173 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.174 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:40:33.174 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.565384+0000", 2024-09-21T10:40:33.174 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.174 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.174 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.174 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:40:33.174 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.565384+0000", 2024-09-21T10:40:33.174 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.174 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.174 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.174 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:40:33.174 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.565386+0000", 2024-09-21T10:40:33.174 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.537e-06 2024-09-21T10:40:33.174 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.174 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.175 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:40:33.175 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.565728+0000", 2024-09-21T10:40:33.175 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00034197999999999997 2024-09-21T10:40:33.175 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.175 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.175 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:40:33.175 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.565753+0000", 2024-09-21T10:40:33.175 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4740999999999999e-05 2024-09-21T10:40:33.175 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.175 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.175 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:40:33.175 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.565763+0000", 2024-09-21T10:40:33.175 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0283e-05 2024-09-21T10:40:33.175 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.175 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.176 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:40:33.176 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.565851+0000", 2024-09-21T10:40:33.176 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7516999999999996e-05 2024-09-21T10:40:33.176 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.176 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.176 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:40:33.176 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.565942+0000", 2024-09-21T10:40:33.176 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0864000000000003e-05 2024-09-21T10:40:33.176 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.176 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.176 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:40:33.176 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.566084+0000", 2024-09-21T10:40:33.176 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000142118 2024-09-21T10:40:33.176 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.176 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.177 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:40:33.177 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.568057+0000", 2024-09-21T10:40:33.177 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0019734819999999999 2024-09-21T10:40:33.177 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.177 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.177 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.177 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.597137+0000", 2024-09-21T10:40:33.177 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.029079852999999999 2024-09-21T10:40:33.177 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.177 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.177 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.177 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.597173+0000", 2024-09-21T10:40:33.177 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6155000000000002e-05 2024-09-21T10:40:33.177 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.177 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.178 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:40:33.178 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.597207+0000", 2024-09-21T10:40:33.178 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3612000000000003e-05 2024-09-21T10:40:33.178 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.178 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.178 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:40:33.178 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.597233+0000", 2024-09-21T10:40:33.178 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6410999999999998e-05 2024-09-21T10:40:33.178 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.178 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:40:33.178 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.178 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.178 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.178 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.34884.0:3150 52.1d 52:bf520f90:::rbd_data.88447b362a13.0000000000000044:head [write 0~3478529] snapc 78=[78,77,76,75,74,73,72,71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1277)", 2024-09-21T10:40:33.178 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:38:26.558946+0000", 2024-09-21T10:40:33.179 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 126.50286778, 2024-09-21T10:40:33.179 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.034713731999999997, 2024-09-21T10:40:33.179 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:40:33.179 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:40:33.179 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:33.179 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:40:33.179 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.34884", 2024-09-21T10:40:33.179 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1225916197", 2024-09-21T10:40:33.179 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 3150 2024-09-21T10:40:33.179 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.179 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:40:33.179 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.179 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:40:33.179 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:26.558946+0000", 2024-09-21T10:40:33.179 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.179 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.180 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.180 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:40:33.180 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:26.558946+0000", 2024-09-21T10:40:33.180 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.180 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.180 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.180 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:40:33.180 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:26.558950+0000", 2024-09-21T10:40:33.180 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0559999999999998e-06 2024-09-21T10:40:33.180 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.180 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.180 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:40:33.180 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:26.562961+0000", 2024-09-21T10:40:33.180 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0040110170000000004 2024-09-21T10:40:33.180 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.181 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.181 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:40:33.181 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:26.562988+0000", 2024-09-21T10:40:33.181 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7379999999999999e-05 2024-09-21T10:40:33.181 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.181 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.181 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:40:33.181 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:26.563000+0000", 2024-09-21T10:40:33.181 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1364e-05 2024-09-21T10:40:33.182 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.182 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.182 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:40:33.182 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:26.563256+0000", 2024-09-21T10:40:33.182 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00025601299999999998 2024-09-21T10:40:33.182 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.182 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.182 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:40:33.182 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:26.563381+0000", 2024-09-21T10:40:33.182 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012504500000000001 2024-09-21T10:40:33.182 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.182 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.182 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:40:33.183 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:26.563609+0000", 2024-09-21T10:40:33.183 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00022790199999999999 2024-09-21T10:40:33.183 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.183 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.183 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:40:33.183 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:26.574842+0000", 2024-09-21T10:40:33.183 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.011233391000000001 2024-09-21T10:40:33.183 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.183 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.183 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.183 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:26.593384+0000", 2024-09-21T10:40:33.183 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.018541953999999999 2024-09-21T10:40:33.183 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.183 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.183 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.184 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:26.593524+0000", 2024-09-21T10:40:33.184 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014028399999999999 2024-09-21T10:40:33.184 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.184 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.184 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:40:33.184 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:26.593599+0000", 2024-09-21T10:40:33.184 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4585000000000005e-05 2024-09-21T10:40:33.184 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.184 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.184 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:40:33.184 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:26.593660+0000", 2024-09-21T10:40:33.184 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0741e-05 2024-09-21T10:40:33.184 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.184 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:40:33.184 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.185 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.185 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.185 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:19394 53.9 53:9f91e2ba:::benchmark_data_smithi070_228967_object19393:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1314)", 2024-09-21T10:40:33.185 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:40:31.831590+0000", 2024-09-21T10:40:33.185 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 1.230223748, 2024-09-21T10:40:33.185 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.055071779000000001, 2024-09-21T10:40:33.185 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:40:33.185 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:40:33.185 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:33.185 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:40:33.185 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:40:33.185 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:40:33.185 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 19394 2024-09-21T10:40:33.185 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.185 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:40:33.186 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.186 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:40:33.186 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.831590+0000", 2024-09-21T10:40:33.186 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.186 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.186 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.186 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:40:33.186 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.831590+0000", 2024-09-21T10:40:33.186 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.186 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.186 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.186 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:40:33.186 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.831591+0000", 2024-09-21T10:40:33.187 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9900000000000009e-07 2024-09-21T10:40:33.187 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.187 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.187 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:40:33.187 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.831618+0000", 2024-09-21T10:40:33.187 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6859000000000002e-05 2024-09-21T10:40:33.187 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.187 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.187 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:40:33.187 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.831627+0000", 2024-09-21T10:40:33.187 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7979999999999995e-06 2024-09-21T10:40:33.187 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.187 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.187 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:40:33.187 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.831633+0000", 2024-09-21T10:40:33.188 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.516e-06 2024-09-21T10:40:33.188 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.188 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.188 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:40:33.188 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.832525+0000", 2024-09-21T10:40:33.188 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00089167999999999995 2024-09-21T10:40:33.188 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.188 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.188 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:40:33.188 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.832640+0000", 2024-09-21T10:40:33.188 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000115474 2024-09-21T10:40:33.188 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.188 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.188 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:40:33.189 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.832771+0000", 2024-09-21T10:40:33.189 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013029299999999999 2024-09-21T10:40:33.189 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.189 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.189 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:40:33.189 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.834030+0000", 2024-09-21T10:40:33.189 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00125968 2024-09-21T10:40:33.189 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.189 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.189 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.189 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.834141+0000", 2024-09-21T10:40:33.189 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000110995 2024-09-21T10:40:33.189 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.189 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.189 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.190 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.886595+0000", 2024-09-21T10:40:33.190 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.052453631000000001 2024-09-21T10:40:33.190 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.190 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.190 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:40:33.190 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.886633+0000", 2024-09-21T10:40:33.190 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7948999999999999e-05 2024-09-21T10:40:33.190 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.190 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.190 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:40:33.190 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.886662+0000", 2024-09-21T10:40:33.190 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8904999999999999e-05 2024-09-21T10:40:33.190 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.190 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:40:33.190 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.191 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.191 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.191 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:19413 53.9 53:9fd8d645:::benchmark_data_smithi070_228967_object19412:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1314)", 2024-09-21T10:40:33.191 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:40:31.840699+0000", 2024-09-21T10:40:33.191 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 1.221115215, 2024-09-21T10:40:33.191 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.048707662999999998, 2024-09-21T10:40:33.191 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:40:33.191 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:40:33.191 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:33.191 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:40:33.191 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:40:33.191 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:40:33.191 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 19413 2024-09-21T10:40:33.191 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.192 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:40:33.192 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.192 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:40:33.192 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.840699+0000", 2024-09-21T10:40:33.192 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.192 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.192 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.192 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:40:33.192 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.840699+0000", 2024-09-21T10:40:33.192 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:40:33.192 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.192 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.192 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:40:33.192 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.840700+0000", 2024-09-21T10:40:33.192 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1319999999999999e-06 2024-09-21T10:40:33.192 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.193 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.193 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:40:33.193 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.840722+0000", 2024-09-21T10:40:33.193 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1863000000000001e-05 2024-09-21T10:40:33.193 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.193 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.193 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:40:33.193 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.840732+0000", 2024-09-21T10:40:33.193 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0543999999999999e-05 2024-09-21T10:40:33.193 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.194 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.194 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:40:33.194 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.840738+0000", 2024-09-21T10:40:33.194 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4199999999999998e-06 2024-09-21T10:40:33.194 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.194 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.194 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:40:33.194 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.840789+0000", 2024-09-21T10:40:33.194 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1839000000000003e-05 2024-09-21T10:40:33.194 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.194 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.194 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:40:33.194 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.840942+0000", 2024-09-21T10:40:33.194 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015300399999999999 2024-09-21T10:40:33.195 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.195 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.195 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:40:33.195 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.841088+0000", 2024-09-21T10:40:33.195 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000145993 2024-09-21T10:40:33.195 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.195 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.195 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:40:33.195 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.841866+0000", 2024-09-21T10:40:33.195 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00077790400000000005 2024-09-21T10:40:33.195 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.195 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.195 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.195 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.842377+0000", 2024-09-21T10:40:33.195 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000510831 2024-09-21T10:40:33.195 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.196 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.196 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:33.196 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.889318+0000", 2024-09-21T10:40:33.196 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.046941301999999997 2024-09-21T10:40:33.196 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.196 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.196 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:40:33.196 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.889357+0000", 2024-09-21T10:40:33.196 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8247000000000002e-05 2024-09-21T10:40:33.196 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:40:33.196 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:40:33.196 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:40:33.196 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:31.889406+0000", 2024-09-21T10:40:33.196 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9583999999999998e-05 2024-09-21T10:40:33.196 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.197 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:40:33.197 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.197 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:40:33.197 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:40:33.197 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:40:33.237 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:33.271 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:33.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:33.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:33.373 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:33.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:40:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:40:33] "GET /metrics HTTP/1.1" 200 36462 "" "Prometheus/2.51.0" 2024-09-21T10:40:33.408 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:33.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:33.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:33.510 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:33.541 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:33.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:33.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:33.642 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:33.674 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:33.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:33.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:33.765 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:33 smithi070 ceph-mon[107385]: osdmap e1315: 8 total, 8 up, 6 in 2024-09-21T10:40:33.765 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:33 smithi070 ceph-mon[112139]: osdmap e1315: 8 total, 8 up, 6 in 2024-09-21T10:40:33.776 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:33.811 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:33.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:33.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:33 smithi160 ceph-mon[101879]: osdmap e1315: 8 total, 8 up, 6 in 2024-09-21T10:40:33.913 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:33.948 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:33.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:33.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:34.050 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:34.084 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:34.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:34.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:34.111 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 10 16 25345 25329 158.274 162.75 0.00949519 0.00631235 2024-09-21T10:40:34.186 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:34.222 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:34.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:34.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:34.324 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:34.363 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:34.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:34.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:34.464 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:34.493 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:40:34.501 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:34.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:34.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:34.602 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:34.631 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:34.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:34.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:34.733 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:34.773 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:34.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:34.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:34.824 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:34 smithi070 ceph-mon[107385]: pgmap v2165: 89 pgs: 7 peering, 1 remapped+peering, 1 unknown, 80 active+clean; 1.1 GiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 221 MiB/s wr, 3.54k op/s; 1.2 KiB/s, 1 objects/s recovering 2024-09-21T10:40:34.825 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:34 smithi070 ceph-mon[107385]: osdmap e1316: 8 total, 8 up, 6 in 2024-09-21T10:40:34.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:34 smithi070 ceph-mon[112139]: pgmap v2165: 89 pgs: 7 peering, 1 remapped+peering, 1 unknown, 80 active+clean; 1.1 GiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 221 MiB/s wr, 3.54k op/s; 1.2 KiB/s, 1 objects/s recovering 2024-09-21T10:40:34.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:34 smithi070 ceph-mon[112139]: osdmap e1316: 8 total, 8 up, 6 in 2024-09-21T10:40:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:34 smithi160 ceph-mon[101879]: pgmap v2165: 89 pgs: 7 peering, 1 remapped+peering, 1 unknown, 80 active+clean; 1.1 GiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 221 MiB/s wr, 3.54k op/s; 1.2 KiB/s, 1 objects/s recovering 2024-09-21T10:40:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:34 smithi160 ceph-mon[101879]: osdmap e1316: 8 total, 8 up, 6 in 2024-09-21T10:40:34.870 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:40:34.875 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:34.904 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:34.955 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:34.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:34.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:35.018 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6] out_osds: [2, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:40:35.018 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:40:35.019 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-21T10:40:35.019 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:40:35.057 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:35.088 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:35.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:35.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:35.112 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 11 15 27599 27584 156.691 140.938 0.00191372 0.00637603 2024-09-21T10:40:35.189 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:35.223 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:35.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:35.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:35.325 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:35.346 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:35.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:35.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:35.447 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:35.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:35.481 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:35.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:35.582 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:35.618 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:35.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:35.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:35.663 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:35.720 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:35.761 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:35.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:35.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:35.862 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:35.882 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:35.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:35.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:35.984 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:36.017 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:36.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:36.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:36.112 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 12 16 29565 29549 153.867 122.812 0.00699981 0.00649439 2024-09-21T10:40:36.118 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:36.155 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:36.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:36.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:36.256 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:36.288 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:36.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:36.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:36.390 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:36.411 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:36.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:36.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:36.513 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:36.550 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:36.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:36.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:36 smithi070 ceph-mon[107385]: pgmap v2167: 89 pgs: 1 remapped+peering, 88 active+clean; 1.3 GiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 145 MiB/s wr, 2.32k op/s; 137 KiB/s, 3 objects/s recovering 2024-09-21T10:40:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:40:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:40:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:36 smithi070 ceph-mon[107385]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:40:36.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:36 smithi070 ceph-mon[112139]: pgmap v2167: 89 pgs: 1 remapped+peering, 88 active+clean; 1.3 GiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 145 MiB/s wr, 2.32k op/s; 137 KiB/s, 3 objects/s recovering 2024-09-21T10:40:36.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:40:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:40:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:36 smithi070 ceph-mon[112139]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:40:36.652 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:36.687 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:36.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:36.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:36.788 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:36.824 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:36.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:36.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:36 smithi160 ceph-mon[101879]: pgmap v2167: 89 pgs: 1 remapped+peering, 88 active+clean; 1.3 GiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 145 MiB/s wr, 2.32k op/s; 137 KiB/s, 3 objects/s recovering 2024-09-21T10:40:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:40:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:40:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:36 smithi160 ceph-mon[101879]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:40:36.925 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:36.953 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:36.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:36.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:37.054 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:37.089 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:37.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:37.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:37.112 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 13 16 31767 31751 152.617 137.625 0.00628385 0.00654759 2024-09-21T10:40:37.190 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:37.226 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:37.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:37.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:37.328 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:37.361 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:37.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:37.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:37.464 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:37.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:37.496 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:37.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:37.597 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:37.636 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:37.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:37.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:37.737 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:37.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:37.778 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:37.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:37.879 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:37.901 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:37.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:37.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:38.002 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:38.035 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:38.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:38.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:38.113 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 14 16 33783 33767 150.706 126 0.0078862 0.00662955 2024-09-21T10:40:38.118 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:38.136 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:38.182 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:38.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:38.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:38.283 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:38.318 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:38.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:38.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:38.420 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:38.444 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:38.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:38.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:38.546 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:38.582 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:38.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:38.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:38.625 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:38 smithi070 ceph-mon[112139]: pgmap v2168: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.7 GiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 159 MiB/s wr, 2.55k op/s; 718/81723 objects misplaced (0.879%); 5.5 MiB/s, 88 objects/s recovering 2024-09-21T10:40:38.625 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:38 smithi070 ceph-mon[107385]: pgmap v2168: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.7 GiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 159 MiB/s wr, 2.55k op/s; 718/81723 objects misplaced (0.879%); 5.5 MiB/s, 88 objects/s recovering 2024-09-21T10:40:38.684 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:38.718 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:38.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:38.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:38.820 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:38 smithi160 ceph-mon[101879]: pgmap v2168: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.7 GiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 159 MiB/s wr, 2.55k op/s; 718/81723 objects misplaced (0.879%); 5.5 MiB/s, 88 objects/s recovering 2024-09-21T10:40:38.851 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:38.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:38.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:38.953 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:38.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:38.994 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:38.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:39.095 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:39.114 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 15 16 35840 35824 149.218 128.562 0.00493859 0.00669709 2024-09-21T10:40:39.129 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:39.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:39.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:39.231 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:39.267 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:39.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:39.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:39.369 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:39.402 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:39.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:39.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:39.504 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:39.537 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:39.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:39.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:39.639 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:39.676 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:39.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:39.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:39.778 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:39.814 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:39.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:39.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:39.915 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:39.953 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:39.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:39.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:40.055 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:40.089 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:40.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:40.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:40.114 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 16 16 38014 37998 148.383 135.875 0.00672833 0.00673399 2024-09-21T10:40:40.191 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:40.217 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:40.227 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:40.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:40.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:40.330 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:40.370 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:40.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:40.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:40.471 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:40.509 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:40.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:40.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:40.611 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:40.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:40 smithi070 ceph-mon[107385]: pgmap v2169: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.7 GiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 82 MiB/s wr, 1.30k op/s; 718/81723 objects misplaced (0.879%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:40:40.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:40 smithi070 ceph-mon[112139]: pgmap v2169: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.7 GiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 82 MiB/s wr, 1.30k op/s; 718/81723 objects misplaced (0.879%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:40:40.648 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:40.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:40.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:40.749 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:40.779 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:40.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:40.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:40 smithi160 ceph-mon[101879]: pgmap v2169: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.7 GiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 82 MiB/s wr, 1.30k op/s; 718/81723 objects misplaced (0.879%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:40:40.881 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:40.908 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:40.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:40.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:41.009 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:41.043 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:41.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:41.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:41.115 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 17 16 40043 40027 147.114 126.812 0.00486301 0.0067923 2024-09-21T10:40:41.145 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:41.182 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:41.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:41.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:41.284 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:41.320 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:41.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:41.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:41.388 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:41 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2907537709' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:40:41.388 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:41 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:40:41.390 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T10:40:41.391 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:41 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2907537709' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:40:41.391 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:41 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:40:41.422 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:41.456 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:41.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:41.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:41.557 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:41.580 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:41.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:41.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:41.682 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:41.703 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:41.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:41.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:41.804 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:41.825 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:41.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:41.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:41.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:41 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2907537709' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:40:41.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:41 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:40:41.928 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:41.948 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:41.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:41.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:42.050 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:42.071 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:42.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:42.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:42.115 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 18 16 40912 40896 141.959 54.3125 0.00217854 0.00679105 2024-09-21T10:40:42.172 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:42.194 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:42.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:42.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:42.295 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:42.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:42.318 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:42.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:42.419 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:42.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:42.457 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:42.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:42.557 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:42.590 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:42 smithi070 ceph-mon[107385]: pgmap v2170: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 2.0 GiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 112 MiB/s wr, 1.79k op/s; 718/97611 objects misplaced (0.736%); 4.0 MiB/s, 63 objects/s recovering 2024-09-21T10:40:42.590 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:42 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:40:42.590 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:42 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:40:42.590 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:42 smithi070 ceph-mon[107385]: osdmap e1317: 8 total, 8 up, 6 in 2024-09-21T10:40:42.590 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:42 smithi070 ceph-mon[112139]: pgmap v2170: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 2.0 GiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 112 MiB/s wr, 1.79k op/s; 718/97611 objects misplaced (0.736%); 4.0 MiB/s, 63 objects/s recovering 2024-09-21T10:40:42.591 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:42 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:40:42.591 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:42 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:40:42.591 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:42 smithi070 ceph-mon[112139]: osdmap e1317: 8 total, 8 up, 6 in 2024-09-21T10:40:42.592 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:42.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:42.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:42.694 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:42.726 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:42.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:42.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:42.827 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:42 smithi160 ceph-mon[101879]: pgmap v2170: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 2.0 GiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 112 MiB/s wr, 1.79k op/s; 718/97611 objects misplaced (0.736%); 4.0 MiB/s, 63 objects/s recovering 2024-09-21T10:40:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:42 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:40:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:42 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:40:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:42 smithi160 ceph-mon[101879]: osdmap e1317: 8 total, 8 up, 6 in 2024-09-21T10:40:42.860 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:42.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:42.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:42.961 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:42.999 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:43.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:43.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:43.101 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:43.115 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 19 16 42394 42378 139.362 92.625 0.0127086 0.00717057 2024-09-21T10:40:43.133 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:43.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:43.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:43.159 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:40:43.159 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:40:43.195 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:40:43.195 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:40:43.227 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:40:43.228 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:40:43.235 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:43.269 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:40:43.270 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:40:43.277 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:43.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:43.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:43.305 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:40:43.305 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:40:43.338 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:40:43.338 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:40:43.367 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:40:43.367 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:40:43.380 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:43.400 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:40:43.401 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:40:43.412 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:43.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:43.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:43.419 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:40:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:40:43] "GET /metrics HTTP/1.1" 200 36611 "" "Prometheus/2.51.0" 2024-09-21T10:40:43.420 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:43 smithi070 ceph-mon[107385]: osdmap e1318: 8 total, 8 up, 6 in 2024-09-21T10:40:43.420 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:43 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1475082161' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:40:43.420 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:43 smithi070 ceph-mon[112139]: osdmap e1318: 8 total, 8 up, 6 in 2024-09-21T10:40:43.420 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:43 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1475082161' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:40:43.513 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:43.553 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:43.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:43.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:43.654 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:43.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:43.691 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:43.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:43.791 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:43.827 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:43.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:43.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:43 smithi160 ceph-mon[101879]: osdmap e1318: 8 total, 8 up, 6 in 2024-09-21T10:40:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:43 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1475082161' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:40:43.928 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:43.959 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:43.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:43.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:44.061 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:44.098 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:44.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:44.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:44.115 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:2024-09-21T10:40:44.114273+0000 min lat: 0.00114886 max lat: 2.0939 avg lat: 0.00720128 2024-09-21T10:40:44.115 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:40:44.115 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 20 16 44435 44419 138.772 127.562 0.00469624 0.00720128 2024-09-21T10:40:44.200 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:44.231 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:44.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:44.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:44.333 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:44.350 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:40:44.350 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:40:44.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:44.375 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:44.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:44.476 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:44.507 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:44.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:44.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:44.609 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:44.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:44 smithi070 ceph-mon[107385]: pgmap v2173: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 2.3 GiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 125 MiB/s wr, 2.00k op/s; 36/113367 objects misplaced (0.032%); 9.4 MiB/s, 149 objects/s recovering 2024-09-21T10:40:44.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:44 smithi070 ceph-mon[112139]: pgmap v2173: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 2.3 GiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 125 MiB/s wr, 2.00k op/s; 36/113367 objects misplaced (0.032%); 9.4 MiB/s, 149 objects/s recovering 2024-09-21T10:40:44.628 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:40:44.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:44.643 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:44.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:44.744 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:44.773 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:44.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:44.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:44 smithi160 ceph-mon[101879]: pgmap v2173: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 2.3 GiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 125 MiB/s wr, 2.00k op/s; 36/113367 objects misplaced (0.032%); 9.4 MiB/s, 149 objects/s recovering 2024-09-21T10:40:44.875 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:44.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:44.915 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:44.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:45.016 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:45.056 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:45.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:45.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:45.115 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 21 16 46398 46382 138.005 122.688 0.00526346 0.00724213 2024-09-21T10:40:45.158 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:45.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:45.200 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:45.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:45.301 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:45.334 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:45.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:45.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:45.436 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:45.471 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:45.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:45.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:45.572 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:45.605 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:45.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:45.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:45.707 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:45.741 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:45.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:45.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:45.843 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:45.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:45.883 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:45.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:45.984 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:46.020 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:46.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:46.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:46.115 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 22 16 48356 48340 137.295 122.375 0.0062657 0.00727871 2024-09-21T10:40:46.122 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:46.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:46.158 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:46.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:46.258 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:46.294 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:46.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:46.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:46.396 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:46.432 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:46.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:46.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:46.534 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:46.567 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:46.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:46.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:46.668 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:46.701 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:46.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:46.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:46.776 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:40:46.803 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:46.813 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:46 smithi070 ceph-mon[107385]: pgmap v2174: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 2.6 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 116 MiB/s wr, 1.86k op/s; 36/126462 objects misplaced (0.028%); 5.3 MiB/s, 85 objects/s recovering 2024-09-21T10:40:46.814 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:46 smithi070 ceph-mon[112139]: pgmap v2174: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 2.6 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 116 MiB/s wr, 1.86k op/s; 36/126462 objects misplaced (0.028%); 5.3 MiB/s, 85 objects/s recovering 2024-09-21T10:40:46.831 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:46.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:46.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:46 smithi160 ceph-mon[101879]: pgmap v2174: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 2.6 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 116 MiB/s wr, 1.86k op/s; 36/126462 objects misplaced (0.028%); 5.3 MiB/s, 85 objects/s recovering 2024-09-21T10:40:46.933 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:46.965 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:46.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:46.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:47.066 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:47.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:47.094 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:47.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:47.116 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 23 16 49998 49982 135.782 102.625 0.00609083 0.0073597 2024-09-21T10:40:47.195 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:47.226 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:47.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:47.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:47.328 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:47.337 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:40:47.353 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:47.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:47.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:47.386 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:47.393 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:47.455 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:47.482 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:47.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:47.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:47.583 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:40:47.584 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T10:40:47.585 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:47.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:47.615 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:47.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:47.716 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:47.744 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:47.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:47.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:47.845 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:47.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:47.881 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:47.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:47.982 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:48.020 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:48.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:48.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:48.116 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 24 16 51669 51653 134.476 104.438 0.020082 0.00742978 2024-09-21T10:40:48.121 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:48.154 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:48.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:48.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:48.255 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:48.291 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:48.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:48.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:48.393 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:48.428 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:48.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:48.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:48.530 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:48.567 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:48.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:48.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:48.669 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:48.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:48.705 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:48.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:48.806 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:48 smithi160 ceph-mon[101879]: pgmap v2175: 89 pgs: 89 active+clean; 2.8 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 149 MiB/s wr, 2.38k op/s; 5.6 MiB/s, 89 objects/s recovering 2024-09-21T10:40:48.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:48.842 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:48.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:48.873 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:48 smithi070 ceph-mon[107385]: pgmap v2175: 89 pgs: 89 active+clean; 2.8 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 149 MiB/s wr, 2.38k op/s; 5.6 MiB/s, 89 objects/s recovering 2024-09-21T10:40:48.875 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:48 smithi070 ceph-mon[112139]: pgmap v2175: 89 pgs: 89 active+clean; 2.8 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 149 MiB/s wr, 2.38k op/s; 5.6 MiB/s, 89 objects/s recovering 2024-09-21T10:40:48.914 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":2175,"stamp":"2024-09-21T10:40:46.662427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3030652901,"num_objects":46261,"num_object_clones":0,"num_object_copies":138783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46261,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":46514,"num_write_kb":2961489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1259,"num_bytes_recovered":83303986,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46443,"ondisk_log_size":46443,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":9393196,"kb_used_data":8673236,"kb_used_omap":218,"kb_used_meta":719077,"kb_avail":553077716,"statfs":{"total":575970213888,"available":566351581184,"internally_reserved":0,"allocated":8881393664,"data_stored":8857060778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":223633,"internal_metadata":736335471},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,7,9,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1246494720,"num_objects":19020,"num_object_clones":0,"num_object_copies":57060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-718,"num_objects_unfound":0,"num_objects_dirty":19020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19020,"num_write_kb":1217280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":718,"num_bytes_recovered":47054848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002042"},"pg_stats":[{"pgid":"53.f","version":"1318'2910","reported_seq":2927,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:46.104755+0000","last_change":"2024-09-21T10:40:16.447365+0000","last_active":"2024-09-21T10:40:46.104755+0000","last_peered":"2024-09-21T10:40:46.104755+0000","last_clean":"2024-09-21T10:40:46.104755+0000","last_became_active":"2024-09-21T10:40:16.447155+0000","last_became_peered":"2024-09-21T10:40:16.447155+0000","last_unstale":"2024-09-21T10:40:46.104755+0000","last_undegraded":"2024-09-21T10:40:46.104755+0000","last_fullsized":"2024-09-21T10:40:46.104755+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":2910,"log_dups_size":0,"ondisk_log_size":2910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:39:36.972987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190709760,"num_objects":2910,"num_object_clones":0,"num_object_copies":8730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2910,"num_write_kb":186240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.e","version":"1318'2970","reported_seq":2990,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:46.083864+0000","last_change":"2024-09-21T10:40:16.450893+0000","last_active":"2024-09-21T10:40:46.083864+0000","last_peered":"2024-09-21T10:40:46.083864+0000","last_clean":"2024-09-21T10:40:46.083864+0000","last_became_active":"2024-09-21T10:40:16.450584+0000","last_became_peered":"2024-09-21T10:40:16.450584+0000","last_unstale":"2024-09-21T10:40:46.083864+0000","last_undegraded":"2024-09-21T10:40:46.083864+0000","last_fullsized":"2024-09-21T10:40:46.083864+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":2970,"log_dups_size":0,"ondisk_log_size":2970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:38:24.901001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194641920,"num_objects":2970,"num_object_clones":0,"num_object_copies":8910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2970,"num_write_kb":190080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.d","version":"1318'2636","reported_seq":2669,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:43.682039+0000","last_change":"2024-09-21T10:40:16.449312+0000","last_active":"2024-09-21T10:40:43.682039+0000","last_peered":"2024-09-21T10:40:43.682039+0000","last_clean":"2024-09-21T10:40:43.682039+0000","last_became_active":"2024-09-21T10:40:16.448919+0000","last_became_peered":"2024-09-21T10:40:16.448919+0000","last_unstale":"2024-09-21T10:40:43.682039+0000","last_undegraded":"2024-09-21T10:40:43.682039+0000","last_fullsized":"2024-09-21T10:40:43.682039+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":2636,"log_dups_size":0,"ondisk_log_size":2636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:35:51.952080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172752896,"num_objects":2636,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2636,"num_write_kb":168704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.c","version":"1318'3125","reported_seq":5633,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:45.974107+0000","last_change":"2024-09-21T10:40:42.390568+0000","last_active":"2024-09-21T10:40:45.974107+0000","last_peered":"2024-09-21T10:40:45.974107+0000","last_clean":"2024-09-21T10:40:45.974107+0000","last_became_active":"2024-09-21T10:40:42.388608+0000","last_became_peered":"2024-09-21T10:40:42.388608+0000","last_unstale":"2024-09-21T10:40:45.974107+0000","last_undegraded":"2024-09-21T10:40:45.974107+0000","last_fullsized":"2024-09-21T10:40:45.974107+0000","mapping_epoch":1317,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":3125,"log_dups_size":0,"ondisk_log_size":3125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:06:04.278392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204800000,"num_objects":3125,"num_object_clones":0,"num_object_copies":9375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3125,"num_write_kb":200000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1235,"num_bytes_recovered":80936960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"53.b","version":"1318'3011","reported_seq":3030,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:45.990011+0000","last_change":"2024-09-21T10:40:16.445075+0000","last_active":"2024-09-21T10:40:45.990011+0000","last_peered":"2024-09-21T10:40:45.990011+0000","last_clean":"2024-09-21T10:40:45.990011+0000","last_became_active":"2024-09-21T10:40:16.444802+0000","last_became_peered":"2024-09-21T10:40:16.444802+0000","last_unstale":"2024-09-21T10:40:45.990011+0000","last_undegraded":"2024-09-21T10:40:45.990011+0000","last_fullsized":"2024-09-21T10:40:45.990011+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":3011,"log_dups_size":0,"ondisk_log_size":3011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:36:32.765238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197328896,"num_objects":3011,"num_object_clones":0,"num_object_copies":9033,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3011,"num_write_kb":192704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"53.a","version":"1318'2725","reported_seq":2758,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:44.298747+0000","last_change":"2024-09-21T10:40:16.451613+0000","last_active":"2024-09-21T10:40:44.298747+0000","last_peered":"2024-09-21T10:40:44.298747+0000","last_clean":"2024-09-21T10:40:44.298747+0000","last_became_active":"2024-09-21T10:40:16.451348+0000","last_became_peered":"2024-09-21T10:40:16.451348+0000","last_unstale":"2024-09-21T10:40:44.298747+0000","last_undegraded":"2024-09-21T10:40:44.298747+0000","last_fullsized":"2024-09-21T10:40:44.298747+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":2725,"log_dups_size":0,"ondisk_log_size":2725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:09:36.405529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178585600,"num_objects":2725,"num_object_clones":0,"num_object_copies":8175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2725,"num_write_kb":174400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"53.9","version":"1318'2870","reported_seq":2903,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:44.457247+0000","last_change":"2024-09-21T10:40:16.450122+0000","last_active":"2024-09-21T10:40:44.457247+0000","last_peered":"2024-09-21T10:40:44.457247+0000","last_clean":"2024-09-21T10:40:44.457247+0000","last_became_active":"2024-09-21T10:40:16.449499+0000","last_became_peered":"2024-09-21T10:40:16.449499+0000","last_unstale":"2024-09-21T10:40:44.457247+0000","last_undegraded":"2024-09-21T10:40:44.457247+0000","last_fullsized":"2024-09-21T10:40:44.457247+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":2870,"log_dups_size":0,"ondisk_log_size":2870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:19.536823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188088320,"num_objects":2870,"num_object_clones":0,"num_object_copies":8610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2870,"num_write_kb":183680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.8","version":"1318'2945","reported_seq":2964,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:45.964991+0000","last_change":"2024-09-21T10:40:16.452473+0000","last_active":"2024-09-21T10:40:45.964991+0000","last_peered":"2024-09-21T10:40:45.964991+0000","last_clean":"2024-09-21T10:40:45.964991+0000","last_became_active":"2024-09-21T10:40:16.452167+0000","last_became_peered":"2024-09-21T10:40:16.452167+0000","last_unstale":"2024-09-21T10:40:45.964991+0000","last_undegraded":"2024-09-21T10:40:45.964991+0000","last_fullsized":"2024-09-21T10:40:45.964991+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":2945,"log_dups_size":0,"ondisk_log_size":2945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:13.100070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193003520,"num_objects":2945,"num_object_clones":0,"num_object_copies":8835,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2945,"num_write_kb":188480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"53.7","version":"1318'2769","reported_seq":2802,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:43.690567+0000","last_change":"2024-09-21T10:40:16.452600+0000","last_active":"2024-09-21T10:40:43.690567+0000","last_peered":"2024-09-21T10:40:43.690567+0000","last_clean":"2024-09-21T10:40:43.690567+0000","last_became_active":"2024-09-21T10:40:16.452230+0000","last_became_peered":"2024-09-21T10:40:16.452230+0000","last_unstale":"2024-09-21T10:40:43.690567+0000","last_undegraded":"2024-09-21T10:40:43.690567+0000","last_fullsized":"2024-09-21T10:40:43.690567+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":2769,"log_dups_size":0,"ondisk_log_size":2769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:49:23.263674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181469184,"num_objects":2769,"num_object_clones":0,"num_object_copies":8307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2769,"num_write_kb":177216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.6","version":"1318'2818","reported_seq":2851,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:44.284190+0000","last_change":"2024-09-21T10:40:16.451231+0000","last_active":"2024-09-21T10:40:44.284190+0000","last_peered":"2024-09-21T10:40:44.284190+0000","last_clean":"2024-09-21T10:40:44.284190+0000","last_became_active":"2024-09-21T10:40:16.450727+0000","last_became_peered":"2024-09-21T10:40:16.450727+0000","last_unstale":"2024-09-21T10:40:44.284190+0000","last_undegraded":"2024-09-21T10:40:44.284190+0000","last_fullsized":"2024-09-21T10:40:44.284190+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":2818,"log_dups_size":0,"ondisk_log_size":2818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:24:23.161484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184680448,"num_objects":2818,"num_object_clones":0,"num_object_copies":8454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2818,"num_write_kb":180352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"53.5","version":"1318'2864","reported_seq":2897,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:44.468509+0000","last_change":"2024-09-21T10:40:16.453661+0000","last_active":"2024-09-21T10:40:44.468509+0000","last_peered":"2024-09-21T10:40:44.468509+0000","last_clean":"2024-09-21T10:40:44.468509+0000","last_became_active":"2024-09-21T10:40:16.453348+0000","last_became_peered":"2024-09-21T10:40:16.453348+0000","last_unstale":"2024-09-21T10:40:44.468509+0000","last_undegraded":"2024-09-21T10:40:44.468509+0000","last_fullsized":"2024-09-21T10:40:44.468509+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":2864,"log_dups_size":0,"ondisk_log_size":2864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:25:03.624945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187695104,"num_objects":2864,"num_object_clones":0,"num_object_copies":8592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2864,"num_write_kb":183296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.4","version":"1318'2788","reported_seq":2821,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:43.664626+0000","last_change":"2024-09-21T10:40:16.447393+0000","last_active":"2024-09-21T10:40:43.664626+0000","last_peered":"2024-09-21T10:40:43.664626+0000","last_clean":"2024-09-21T10:40:43.664626+0000","last_became_active":"2024-09-21T10:40:16.447106+0000","last_became_peered":"2024-09-21T10:40:16.447106+0000","last_unstale":"2024-09-21T10:40:43.664626+0000","last_undegraded":"2024-09-21T10:40:43.664626+0000","last_fullsized":"2024-09-21T10:40:43.664626+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":2788,"log_dups_size":0,"ondisk_log_size":2788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:27:57.732120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182714368,"num_objects":2788,"num_object_clones":0,"num_object_copies":8364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2788,"num_write_kb":178432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.3","version":"1318'3110","reported_seq":3130,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:46.092307+0000","last_change":"2024-09-21T10:40:16.444971+0000","last_active":"2024-09-21T10:40:46.092307+0000","last_peered":"2024-09-21T10:40:46.092307+0000","last_clean":"2024-09-21T10:40:46.092307+0000","last_became_active":"2024-09-21T10:40:16.444685+0000","last_became_peered":"2024-09-21T10:40:16.444685+0000","last_unstale":"2024-09-21T10:40:46.092307+0000","last_undegraded":"2024-09-21T10:40:46.092307+0000","last_fullsized":"2024-09-21T10:40:46.092307+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":3110,"log_dups_size":0,"ondisk_log_size":3110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:14:05.945707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203816960,"num_objects":3110,"num_object_clones":0,"num_object_copies":9330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3110,"num_write_kb":199040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.2","version":"1318'2922","reported_seq":2942,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:45.986879+0000","last_change":"2024-09-21T10:40:16.448723+0000","last_active":"2024-09-21T10:40:45.986879+0000","last_peered":"2024-09-21T10:40:45.986879+0000","last_clean":"2024-09-21T10:40:45.986879+0000","last_became_active":"2024-09-21T10:40:16.448497+0000","last_became_peered":"2024-09-21T10:40:16.448497+0000","last_unstale":"2024-09-21T10:40:45.986879+0000","last_undegraded":"2024-09-21T10:40:45.986879+0000","last_fullsized":"2024-09-21T10:40:45.986879+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":2922,"log_dups_size":0,"ondisk_log_size":2922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:01:57.091359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191496192,"num_objects":2922,"num_object_clones":0,"num_object_copies":8766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2922,"num_write_kb":187008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"53.1","version":"1318'2765","reported_seq":2797,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:43.675963+0000","last_change":"2024-09-21T10:40:16.452246+0000","last_active":"2024-09-21T10:40:43.675963+0000","last_peered":"2024-09-21T10:40:43.675963+0000","last_clean":"2024-09-21T10:40:43.675963+0000","last_became_active":"2024-09-21T10:40:16.451960+0000","last_became_peered":"2024-09-21T10:40:16.451960+0000","last_unstale":"2024-09-21T10:40:43.675963+0000","last_undegraded":"2024-09-21T10:40:43.675963+0000","last_fullsized":"2024-09-21T10:40:43.675963+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":2765,"log_dups_size":0,"ondisk_log_size":2765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:55:54.581768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181207040,"num_objects":2765,"num_object_clones":0,"num_object_copies":8295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2765,"num_write_kb":176960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.0","version":"1318'3007","reported_seq":3027,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:45.994244+0000","last_change":"2024-09-21T10:40:16.452286+0000","last_active":"2024-09-21T10:40:45.994244+0000","last_peered":"2024-09-21T10:40:45.994244+0000","last_clean":"2024-09-21T10:40:45.994244+0000","last_became_active":"2024-09-21T10:40:16.451985+0000","last_became_peered":"2024-09-21T10:40:16.451985+0000","last_unstale":"2024-09-21T10:40:45.994244+0000","last_undegraded":"2024-09-21T10:40:45.994244+0000","last_fullsized":"2024-09-21T10:40:45.994244+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":3007,"log_dups_size":0,"ondisk_log_size":3007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:17:56.363808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197066752,"num_objects":3007,"num_object_clones":0,"num_object_copies":9021,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3007,"num_write_kb":192448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":1946,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.393786+0000","last_change":"2024-09-21T10:40:13.438023+0000","last_active":"2024-09-21T10:40:42.393786+0000","last_peered":"2024-09-21T10:40:42.393786+0000","last_clean":"2024-09-21T10:40:42.393786+0000","last_became_active":"2024-09-21T10:40:13.431103+0000","last_became_peered":"2024-09-21T10:40:13.431103+0000","last_unstale":"2024-09-21T10:40:42.393786+0000","last_undegraded":"2024-09-21T10:40:42.393786+0000","last_fullsized":"2024-09-21T10:40:42.393786+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:00:24.887661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1284,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.396758+0000","last_change":"2024-09-21T10:07:55.999392+0000","last_active":"2024-09-21T10:40:41.396758+0000","last_peered":"2024-09-21T10:40:41.396758+0000","last_clean":"2024-09-21T10:40:41.396758+0000","last_became_active":"2024-09-21T10:07:55.999103+0000","last_became_peered":"2024-09-21T10:07:55.999103+0000","last_unstale":"2024-09-21T10:40:41.396758+0000","last_undegraded":"2024-09-21T10:40:41.396758+0000","last_fullsized":"2024-09-21T10:40:41.396758+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:48.757242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":1926,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.386674+0000","last_change":"2024-09-21T10:40:13.432818+0000","last_active":"2024-09-21T10:40:42.386674+0000","last_peered":"2024-09-21T10:40:42.386674+0000","last_clean":"2024-09-21T10:40:42.386674+0000","last_became_active":"2024-09-21T10:40:13.432448+0000","last_became_peered":"2024-09-21T10:40:13.432448+0000","last_unstale":"2024-09-21T10:40:42.386674+0000","last_undegraded":"2024-09-21T10:40:42.386674+0000","last_fullsized":"2024-09-21T10:40:42.386674+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:20.448242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1944,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.385795+0000","last_change":"2024-09-21T10:40:13.431824+0000","last_active":"2024-09-21T10:40:42.385795+0000","last_peered":"2024-09-21T10:40:42.385795+0000","last_clean":"2024-09-21T10:40:42.385795+0000","last_became_active":"2024-09-21T10:40:13.431516+0000","last_became_peered":"2024-09-21T10:40:13.431516+0000","last_unstale":"2024-09-21T10:40:42.385795+0000","last_undegraded":"2024-09-21T10:40:42.385795+0000","last_fullsized":"2024-09-21T10:40:42.385795+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:30:40.226452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":1272,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.392404+0000","last_change":"2024-09-21T09:54:53.634508+0000","last_active":"2024-09-21T10:40:41.392404+0000","last_peered":"2024-09-21T10:40:41.392404+0000","last_clean":"2024-09-21T10:40:41.392404+0000","last_became_active":"2024-09-21T09:54:53.634254+0000","last_became_peered":"2024-09-21T09:54:53.634254+0000","last_unstale":"2024-09-21T10:40:41.392404+0000","last_undegraded":"2024-09-21T10:40:41.392404+0000","last_fullsized":"2024-09-21T10:40:41.392404+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:18.122107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1299,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.396791+0000","last_change":"2024-09-21T10:09:06.574933+0000","last_active":"2024-09-21T10:40:41.396791+0000","last_peered":"2024-09-21T10:40:41.396791+0000","last_clean":"2024-09-21T10:40:41.396791+0000","last_became_active":"2024-09-21T10:09:06.574467+0000","last_became_peered":"2024-09-21T10:09:06.574467+0000","last_unstale":"2024-09-21T10:40:41.396791+0000","last_undegraded":"2024-09-21T10:40:41.396791+0000","last_fullsized":"2024-09-21T10:40:41.396791+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:06.142269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":1287,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.394080+0000","last_change":"2024-09-21T10:07:55.991189+0000","last_active":"2024-09-21T10:40:41.394080+0000","last_peered":"2024-09-21T10:40:41.394080+0000","last_clean":"2024-09-21T10:40:41.394080+0000","last_became_active":"2024-09-21T10:07:55.990784+0000","last_became_peered":"2024-09-21T10:07:55.990784+0000","last_unstale":"2024-09-21T10:40:41.394080+0000","last_undegraded":"2024-09-21T10:40:41.394080+0000","last_fullsized":"2024-09-21T10:40:41.394080+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:00.580471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1299,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.396984+0000","last_change":"2024-09-21T10:09:06.560175+0000","last_active":"2024-09-21T10:40:41.396984+0000","last_peered":"2024-09-21T10:40:41.396984+0000","last_clean":"2024-09-21T10:40:41.396984+0000","last_became_active":"2024-09-21T10:09:06.558584+0000","last_became_peered":"2024-09-21T10:09:06.558584+0000","last_unstale":"2024-09-21T10:40:41.396984+0000","last_undegraded":"2024-09-21T10:40:41.396984+0000","last_fullsized":"2024-09-21T10:40:41.396984+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:02.918339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":1922,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.393166+0000","last_change":"2024-09-21T10:40:30.424630+0000","last_active":"2024-09-21T10:40:42.393166+0000","last_peered":"2024-09-21T10:40:42.393166+0000","last_clean":"2024-09-21T10:40:42.393166+0000","last_became_active":"2024-09-21T10:40:30.424295+0000","last_became_peered":"2024-09-21T10:40:30.424295+0000","last_unstale":"2024-09-21T10:40:42.393166+0000","last_undegraded":"2024-09-21T10:40:42.393166+0000","last_fullsized":"2024-09-21T10:40:42.393166+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:59:50.861993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":706,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.387778+0000","last_change":"2024-09-21T10:40:13.436927+0000","last_active":"2024-09-21T10:40:42.387778+0000","last_peered":"2024-09-21T10:40:42.387778+0000","last_clean":"2024-09-21T10:40:42.387778+0000","last_became_active":"2024-09-21T10:40:13.436329+0000","last_became_peered":"2024-09-21T10:40:13.436329+0000","last_unstale":"2024-09-21T10:40:42.387778+0000","last_undegraded":"2024-09-21T10:40:42.387778+0000","last_fullsized":"2024-09-21T10:40:42.387778+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:05.668898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":1310,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.396978+0000","last_change":"2024-09-21T10:40:13.433278+0000","last_active":"2024-09-21T10:40:41.396978+0000","last_peered":"2024-09-21T10:40:41.396978+0000","last_clean":"2024-09-21T10:40:41.396978+0000","last_became_active":"2024-09-21T10:40:13.432814+0000","last_became_peered":"2024-09-21T10:40:13.432814+0000","last_unstale":"2024-09-21T10:40:41.396978+0000","last_undegraded":"2024-09-21T10:40:41.396978+0000","last_fullsized":"2024-09-21T10:40:41.396978+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:55.909872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1937,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.386571+0000","last_change":"2024-09-21T10:40:13.432531+0000","last_active":"2024-09-21T10:40:42.386571+0000","last_peered":"2024-09-21T10:40:42.386571+0000","last_clean":"2024-09-21T10:40:42.386571+0000","last_became_active":"2024-09-21T10:40:13.432210+0000","last_became_peered":"2024-09-21T10:40:13.432210+0000","last_unstale":"2024-09-21T10:40:42.386571+0000","last_undegraded":"2024-09-21T10:40:42.386571+0000","last_fullsized":"2024-09-21T10:40:42.386571+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:20:10.998607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1909,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.387036+0000","last_change":"2024-09-21T10:09:06.530394+0000","last_active":"2024-09-21T10:40:42.387036+0000","last_peered":"2024-09-21T10:40:42.387036+0000","last_clean":"2024-09-21T10:40:42.387036+0000","last_became_active":"2024-09-21T10:09:06.529307+0000","last_became_peered":"2024-09-21T10:09:06.529307+0000","last_unstale":"2024-09-21T10:40:42.387036+0000","last_undegraded":"2024-09-21T10:40:42.387036+0000","last_fullsized":"2024-09-21T10:40:42.387036+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:54.287397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1934,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.393688+0000","last_change":"2024-09-21T10:40:30.420276+0000","last_active":"2024-09-21T10:40:42.393688+0000","last_peered":"2024-09-21T10:40:42.393688+0000","last_clean":"2024-09-21T10:40:42.393688+0000","last_became_active":"2024-09-21T10:40:30.419723+0000","last_became_peered":"2024-09-21T10:40:30.419723+0000","last_unstale":"2024-09-21T10:40:42.393688+0000","last_undegraded":"2024-09-21T10:40:42.393688+0000","last_fullsized":"2024-09-21T10:40:42.393688+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:52:04.181647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1924,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.385772+0000","last_change":"2024-09-21T10:09:06.552324+0000","last_active":"2024-09-21T10:40:42.385772+0000","last_peered":"2024-09-21T10:40:42.385772+0000","last_clean":"2024-09-21T10:40:42.385772+0000","last_became_active":"2024-09-21T10:09:06.540545+0000","last_became_peered":"2024-09-21T10:09:06.540545+0000","last_unstale":"2024-09-21T10:40:42.385772+0000","last_undegraded":"2024-09-21T10:40:42.385772+0000","last_fullsized":"2024-09-21T10:40:42.385772+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:08.506149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1289,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.400649+0000","last_change":"2024-09-21T10:07:56.003005+0000","last_active":"2024-09-21T10:40:42.400649+0000","last_peered":"2024-09-21T10:40:42.400649+0000","last_clean":"2024-09-21T10:40:42.400649+0000","last_became_active":"2024-09-21T10:07:56.001550+0000","last_became_peered":"2024-09-21T10:07:56.001550+0000","last_unstale":"2024-09-21T10:40:42.400649+0000","last_undegraded":"2024-09-21T10:40:42.400649+0000","last_fullsized":"2024-09-21T10:40:42.400649+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:54.553261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1916,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.385887+0000","last_change":"2024-09-21T10:09:06.559017+0000","last_active":"2024-09-21T10:40:42.385887+0000","last_peered":"2024-09-21T10:40:42.385887+0000","last_clean":"2024-09-21T10:40:42.385887+0000","last_became_active":"2024-09-21T10:09:06.555256+0000","last_became_peered":"2024-09-21T10:09:06.555256+0000","last_unstale":"2024-09-21T10:40:42.385887+0000","last_undegraded":"2024-09-21T10:40:42.385887+0000","last_fullsized":"2024-09-21T10:40:42.385887+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:33:45.145138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1921,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.393916+0000","last_change":"2024-09-21T10:09:06.562504+0000","last_active":"2024-09-21T10:40:42.393916+0000","last_peered":"2024-09-21T10:40:42.393916+0000","last_clean":"2024-09-21T10:40:42.393916+0000","last_became_active":"2024-09-21T10:09:06.561747+0000","last_became_peered":"2024-09-21T10:09:06.561747+0000","last_unstale":"2024-09-21T10:40:42.393916+0000","last_undegraded":"2024-09-21T10:40:42.393916+0000","last_fullsized":"2024-09-21T10:40:42.393916+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:03:58.012406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1914,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.385991+0000","last_change":"2024-09-21T10:40:30.417426+0000","last_active":"2024-09-21T10:40:42.385991+0000","last_peered":"2024-09-21T10:40:42.385991+0000","last_clean":"2024-09-21T10:40:42.385991+0000","last_became_active":"2024-09-21T10:40:30.417075+0000","last_became_peered":"2024-09-21T10:40:30.417075+0000","last_unstale":"2024-09-21T10:40:42.385991+0000","last_undegraded":"2024-09-21T10:40:42.385991+0000","last_fullsized":"2024-09-21T10:40:42.385991+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:58.077873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1294,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.392441+0000","last_change":"2024-09-21T10:40:30.421413+0000","last_active":"2024-09-21T10:40:41.392441+0000","last_peered":"2024-09-21T10:40:41.392441+0000","last_clean":"2024-09-21T10:40:41.392441+0000","last_became_active":"2024-09-21T10:40:30.420689+0000","last_became_peered":"2024-09-21T10:40:30.420689+0000","last_unstale":"2024-09-21T10:40:41.392441+0000","last_undegraded":"2024-09-21T10:40:41.392441+0000","last_fullsized":"2024-09-21T10:40:41.392441+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:53.860610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":41,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.386644+0000","last_change":"2024-09-21T10:40:30.431008+0000","last_active":"2024-09-21T10:40:42.386644+0000","last_peered":"2024-09-21T10:40:42.386644+0000","last_clean":"2024-09-21T10:40:42.386644+0000","last_became_active":"2024-09-21T10:40:30.422769+0000","last_became_peered":"2024-09-21T10:40:30.422769+0000","last_unstale":"2024-09-21T10:40:42.386644+0000","last_undegraded":"2024-09-21T10:40:42.386644+0000","last_fullsized":"2024-09-21T10:40:42.386644+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:42:21.520242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1916,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.387181+0000","last_change":"2024-09-21T10:40:13.437741+0000","last_active":"2024-09-21T10:40:42.387181+0000","last_peered":"2024-09-21T10:40:42.387181+0000","last_clean":"2024-09-21T10:40:42.387181+0000","last_became_active":"2024-09-21T10:40:13.437447+0000","last_became_peered":"2024-09-21T10:40:13.437447+0000","last_unstale":"2024-09-21T10:40:42.387181+0000","last_undegraded":"2024-09-21T10:40:42.387181+0000","last_fullsized":"2024-09-21T10:40:42.387181+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:40:47.072907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":1313,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.397032+0000","last_change":"2024-09-21T10:40:30.437433+0000","last_active":"2024-09-21T10:40:41.397032+0000","last_peered":"2024-09-21T10:40:41.397032+0000","last_clean":"2024-09-21T10:40:41.397032+0000","last_became_active":"2024-09-21T10:40:30.426024+0000","last_became_peered":"2024-09-21T10:40:30.426024+0000","last_unstale":"2024-09-21T10:40:41.397032+0000","last_undegraded":"2024-09-21T10:40:41.397032+0000","last_fullsized":"2024-09-21T10:40:41.397032+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:13:47.175034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4756,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1285,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.397116+0000","last_change":"2024-09-21T10:07:56.001756+0000","last_active":"2024-09-21T10:40:41.397116+0000","last_peered":"2024-09-21T10:40:41.397116+0000","last_clean":"2024-09-21T10:40:41.397116+0000","last_became_active":"2024-09-21T10:07:56.000789+0000","last_became_peered":"2024-09-21T10:07:56.000789+0000","last_unstale":"2024-09-21T10:40:41.397116+0000","last_undegraded":"2024-09-21T10:40:41.397116+0000","last_fullsized":"2024-09-21T10:40:41.397116+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:56:04.302902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1273,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.396359+0000","last_change":"2024-09-21T09:54:53.636902+0000","last_active":"2024-09-21T10:40:42.396359+0000","last_peered":"2024-09-21T10:40:42.396359+0000","last_clean":"2024-09-21T10:40:42.396359+0000","last_became_active":"2024-09-21T09:54:53.636616+0000","last_became_peered":"2024-09-21T09:54:53.636616+0000","last_unstale":"2024-09-21T10:40:42.396359+0000","last_undegraded":"2024-09-21T10:40:42.396359+0000","last_fullsized":"2024-09-21T10:40:42.396359+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:46:47.694495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1319,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.387321+0000","last_change":"2024-09-21T10:40:30.423568+0000","last_active":"2024-09-21T10:40:41.387321+0000","last_peered":"2024-09-21T10:40:41.387321+0000","last_clean":"2024-09-21T10:40:41.387321+0000","last_became_active":"2024-09-21T10:40:30.423208+0000","last_became_peered":"2024-09-21T10:40:30.423208+0000","last_unstale":"2024-09-21T10:40:41.387321+0000","last_undegraded":"2024-09-21T10:40:41.387321+0000","last_fullsized":"2024-09-21T10:40:41.387321+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:22:29.996089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":1929,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.393657+0000","last_change":"2024-09-21T10:07:56.023462+0000","last_active":"2024-09-21T10:40:42.393657+0000","last_peered":"2024-09-21T10:40:42.393657+0000","last_clean":"2024-09-21T10:40:42.393657+0000","last_became_active":"2024-09-21T10:07:56.021859+0000","last_became_peered":"2024-09-21T10:07:56.021859+0000","last_unstale":"2024-09-21T10:40:42.393657+0000","last_undegraded":"2024-09-21T10:40:42.393657+0000","last_fullsized":"2024-09-21T10:40:42.393657+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:42:03.134729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1910,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.385873+0000","last_change":"2024-09-21T10:40:30.420880+0000","last_active":"2024-09-21T10:40:42.385873+0000","last_peered":"2024-09-21T10:40:42.385873+0000","last_clean":"2024-09-21T10:40:42.385873+0000","last_became_active":"2024-09-21T10:40:30.420555+0000","last_became_peered":"2024-09-21T10:40:30.420555+0000","last_unstale":"2024-09-21T10:40:42.385873+0000","last_undegraded":"2024-09-21T10:40:42.385873+0000","last_fullsized":"2024-09-21T10:40:42.385873+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:15:30.473928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1910,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.387918+0000","last_change":"2024-09-21T10:07:56.006236+0000","last_active":"2024-09-21T10:40:42.387918+0000","last_peered":"2024-09-21T10:40:42.387918+0000","last_clean":"2024-09-21T10:40:42.387918+0000","last_became_active":"2024-09-21T10:07:56.004439+0000","last_became_peered":"2024-09-21T10:07:56.004439+0000","last_unstale":"2024-09-21T10:40:42.387918+0000","last_undegraded":"2024-09-21T10:40:42.387918+0000","last_fullsized":"2024-09-21T10:40:42.387918+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:39.422467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":1915,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.386568+0000","last_change":"2024-09-21T10:40:30.425998+0000","last_active":"2024-09-21T10:40:42.386568+0000","last_peered":"2024-09-21T10:40:42.386568+0000","last_clean":"2024-09-21T10:40:42.386568+0000","last_became_active":"2024-09-21T10:40:30.421240+0000","last_became_peered":"2024-09-21T10:40:30.421240+0000","last_unstale":"2024-09-21T10:40:42.386568+0000","last_undegraded":"2024-09-21T10:40:42.386568+0000","last_fullsized":"2024-09-21T10:40:42.386568+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:43:44.457431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1923,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.393399+0000","last_change":"2024-09-21T10:40:30.417473+0000","last_active":"2024-09-21T10:40:42.393399+0000","last_peered":"2024-09-21T10:40:42.393399+0000","last_clean":"2024-09-21T10:40:42.393399+0000","last_became_active":"2024-09-21T10:40:30.416952+0000","last_became_peered":"2024-09-21T10:40:30.416952+0000","last_unstale":"2024-09-21T10:40:42.393399+0000","last_undegraded":"2024-09-21T10:40:42.393399+0000","last_fullsized":"2024-09-21T10:40:42.393399+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:07:33.486362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":652,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.393535+0000","last_change":"2024-09-21T10:40:30.425276+0000","last_active":"2024-09-21T10:40:42.393535+0000","last_peered":"2024-09-21T10:40:42.393535+0000","last_clean":"2024-09-21T10:40:42.393535+0000","last_became_active":"2024-09-21T10:40:30.425046+0000","last_became_peered":"2024-09-21T10:40:30.425046+0000","last_unstale":"2024-09-21T10:40:42.393535+0000","last_undegraded":"2024-09-21T10:40:42.393535+0000","last_fullsized":"2024-09-21T10:40:42.393535+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:09:59.127115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1945,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.387672+0000","last_change":"2024-09-21T10:40:13.437609+0000","last_active":"2024-09-21T10:40:42.387672+0000","last_peered":"2024-09-21T10:40:42.387672+0000","last_clean":"2024-09-21T10:40:42.387672+0000","last_became_active":"2024-09-21T10:40:13.437296+0000","last_became_peered":"2024-09-21T10:40:13.437296+0000","last_unstale":"2024-09-21T10:40:42.387672+0000","last_undegraded":"2024-09-21T10:40:42.387672+0000","last_fullsized":"2024-09-21T10:40:42.387672+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:41:58.963386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1900,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.387067+0000","last_change":"2024-09-21T10:07:56.067772+0000","last_active":"2024-09-21T10:40:42.387067+0000","last_peered":"2024-09-21T10:40:42.387067+0000","last_clean":"2024-09-21T10:40:42.387067+0000","last_became_active":"2024-09-21T10:07:56.052602+0000","last_became_peered":"2024-09-21T10:07:56.052602+0000","last_unstale":"2024-09-21T10:40:42.387067+0000","last_undegraded":"2024-09-21T10:40:42.387067+0000","last_fullsized":"2024-09-21T10:40:42.387067+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:07:59.899214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1961,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.387485+0000","last_change":"2024-09-21T10:40:30.418199+0000","last_active":"2024-09-21T10:40:42.387485+0000","last_peered":"2024-09-21T10:40:42.387485+0000","last_clean":"2024-09-21T10:40:42.387485+0000","last_became_active":"2024-09-21T10:40:30.417414+0000","last_became_peered":"2024-09-21T10:40:30.417414+0000","last_unstale":"2024-09-21T10:40:42.387485+0000","last_undegraded":"2024-09-21T10:40:42.387485+0000","last_fullsized":"2024-09-21T10:40:42.387485+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:56:54.232106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1312,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.392555+0000","last_change":"2024-09-21T10:40:13.435375+0000","last_active":"2024-09-21T10:40:41.392555+0000","last_peered":"2024-09-21T10:40:41.392555+0000","last_clean":"2024-09-21T10:40:41.392555+0000","last_became_active":"2024-09-21T10:40:13.435050+0000","last_became_peered":"2024-09-21T10:40:13.435050+0000","last_unstale":"2024-09-21T10:40:41.392555+0000","last_undegraded":"2024-09-21T10:40:41.392555+0000","last_fullsized":"2024-09-21T10:40:41.392555+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:57.889610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1955,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.387942+0000","last_change":"2024-09-21T10:40:13.434189+0000","last_active":"2024-09-21T10:40:42.387942+0000","last_peered":"2024-09-21T10:40:42.387942+0000","last_clean":"2024-09-21T10:40:42.387942+0000","last_became_active":"2024-09-21T10:40:13.433892+0000","last_became_peered":"2024-09-21T10:40:13.433892+0000","last_unstale":"2024-09-21T10:40:42.387942+0000","last_undegraded":"2024-09-21T10:40:42.387942+0000","last_fullsized":"2024-09-21T10:40:42.387942+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:49:17.296220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1289,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.396491+0000","last_change":"2024-09-21T10:07:21.524082+0000","last_active":"2024-09-21T10:40:42.396491+0000","last_peered":"2024-09-21T10:40:42.396491+0000","last_clean":"2024-09-21T10:40:42.396491+0000","last_became_active":"2024-09-21T10:07:21.523779+0000","last_became_peered":"2024-09-21T10:07:21.523779+0000","last_unstale":"2024-09-21T10:40:42.396491+0000","last_undegraded":"2024-09-21T10:40:42.396491+0000","last_fullsized":"2024-09-21T10:40:42.396491+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:13.813832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":1303,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.392584+0000","last_change":"2024-09-21T10:07:56.024239+0000","last_active":"2024-09-21T10:40:41.392584+0000","last_peered":"2024-09-21T10:40:41.392584+0000","last_clean":"2024-09-21T10:40:41.392584+0000","last_became_active":"2024-09-21T10:07:56.023914+0000","last_became_peered":"2024-09-21T10:07:56.023914+0000","last_unstale":"2024-09-21T10:40:41.392584+0000","last_undegraded":"2024-09-21T10:40:41.392584+0000","last_fullsized":"2024-09-21T10:40:41.392584+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:23.361736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084477399999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1938,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.387905+0000","last_change":"2024-09-21T10:07:56.056338+0000","last_active":"2024-09-21T10:40:42.387905+0000","last_peered":"2024-09-21T10:40:42.387905+0000","last_clean":"2024-09-21T10:40:42.387905+0000","last_became_active":"2024-09-21T10:07:56.052888+0000","last_became_peered":"2024-09-21T10:07:56.052888+0000","last_unstale":"2024-09-21T10:40:42.387905+0000","last_undegraded":"2024-09-21T10:40:42.387905+0000","last_fullsized":"2024-09-21T10:40:42.387905+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:34:48.662436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1919,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.386193+0000","last_change":"2024-09-21T10:40:13.436234+0000","last_active":"2024-09-21T10:40:42.386193+0000","last_peered":"2024-09-21T10:40:42.386193+0000","last_clean":"2024-09-21T10:40:42.386193+0000","last_became_active":"2024-09-21T10:40:13.435946+0000","last_became_peered":"2024-09-21T10:40:13.435946+0000","last_unstale":"2024-09-21T10:40:42.386193+0000","last_undegraded":"2024-09-21T10:40:42.386193+0000","last_fullsized":"2024-09-21T10:40:42.386193+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:08:08.670739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":712,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.385766+0000","last_change":"2024-09-21T10:40:13.431693+0000","last_active":"2024-09-21T10:40:42.385766+0000","last_peered":"2024-09-21T10:40:42.385766+0000","last_clean":"2024-09-21T10:40:42.385766+0000","last_became_active":"2024-09-21T10:40:13.431389+0000","last_became_peered":"2024-09-21T10:40:13.431389+0000","last_unstale":"2024-09-21T10:40:42.385766+0000","last_undegraded":"2024-09-21T10:40:42.385766+0000","last_fullsized":"2024-09-21T10:40:42.385766+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:22.227336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1951,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.393768+0000","last_change":"2024-09-21T10:40:30.420482+0000","last_active":"2024-09-21T10:40:42.393768+0000","last_peered":"2024-09-21T10:40:42.393768+0000","last_clean":"2024-09-21T10:40:42.393768+0000","last_became_active":"2024-09-21T10:40:30.419902+0000","last_became_peered":"2024-09-21T10:40:30.419902+0000","last_unstale":"2024-09-21T10:40:42.393768+0000","last_undegraded":"2024-09-21T10:40:42.393768+0000","last_fullsized":"2024-09-21T10:40:42.393768+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:37:40.880948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1925,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.392685+0000","last_change":"2024-09-21T10:09:06.560130+0000","last_active":"2024-09-21T10:40:42.392685+0000","last_peered":"2024-09-21T10:40:42.392685+0000","last_clean":"2024-09-21T10:40:42.392685+0000","last_became_active":"2024-09-21T10:09:06.559827+0000","last_became_peered":"2024-09-21T10:09:06.559827+0000","last_unstale":"2024-09-21T10:40:42.392685+0000","last_undegraded":"2024-09-21T10:40:42.392685+0000","last_fullsized":"2024-09-21T10:40:42.392685+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:05:17.166734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":675,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.386855+0000","last_change":"2024-09-21T10:40:30.417793+0000","last_active":"2024-09-21T10:40:42.386855+0000","last_peered":"2024-09-21T10:40:42.386855+0000","last_clean":"2024-09-21T10:40:42.386855+0000","last_became_active":"2024-09-21T10:40:30.417232+0000","last_became_peered":"2024-09-21T10:40:30.417232+0000","last_unstale":"2024-09-21T10:40:42.386855+0000","last_undegraded":"2024-09-21T10:40:42.386855+0000","last_fullsized":"2024-09-21T10:40:42.386855+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:09:58.959904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1282,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.386985+0000","last_change":"2024-09-21T10:40:30.426430+0000","last_active":"2024-09-21T10:40:41.386985+0000","last_peered":"2024-09-21T10:40:41.386985+0000","last_clean":"2024-09-21T10:40:41.386985+0000","last_became_active":"2024-09-21T10:40:30.426120+0000","last_became_peered":"2024-09-21T10:40:30.426120+0000","last_unstale":"2024-09-21T10:40:41.386985+0000","last_undegraded":"2024-09-21T10:40:41.386985+0000","last_fullsized":"2024-09-21T10:40:41.386985+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:23:15.280912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1906,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.386095+0000","last_change":"2024-09-21T10:09:06.560589+0000","last_active":"2024-09-21T10:40:42.386095+0000","last_peered":"2024-09-21T10:40:42.386095+0000","last_clean":"2024-09-21T10:40:42.386095+0000","last_became_active":"2024-09-21T10:09:06.557552+0000","last_became_peered":"2024-09-21T10:09:06.557552+0000","last_unstale":"2024-09-21T10:40:42.386095+0000","last_undegraded":"2024-09-21T10:40:42.386095+0000","last_fullsized":"2024-09-21T10:40:42.386095+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:09:24.866759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":1328,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.392371+0000","last_change":"2024-09-21T10:40:30.431163+0000","last_active":"2024-09-21T10:40:41.392371+0000","last_peered":"2024-09-21T10:40:41.392371+0000","last_clean":"2024-09-21T10:40:41.392371+0000","last_became_active":"2024-09-21T10:40:30.424471+0000","last_became_peered":"2024-09-21T10:40:30.424471+0000","last_unstale":"2024-09-21T10:40:41.392371+0000","last_undegraded":"2024-09-21T10:40:41.392371+0000","last_fullsized":"2024-09-21T10:40:41.392371+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:02:45.899239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":648,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.392989+0000","last_change":"2024-09-21T10:40:33.444446+0000","last_active":"2024-09-21T10:40:42.392989+0000","last_peered":"2024-09-21T10:40:42.392989+0000","last_clean":"2024-09-21T10:40:42.392989+0000","last_became_active":"2024-09-21T10:40:33.443958+0000","last_became_peered":"2024-09-21T10:40:33.443958+0000","last_unstale":"2024-09-21T10:40:42.392989+0000","last_undegraded":"2024-09-21T10:40:42.392989+0000","last_fullsized":"2024-09-21T10:40:42.392989+0000","mapping_epoch":1315,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:15:12.818119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1910,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.387825+0000","last_change":"2024-09-21T10:09:06.538940+0000","last_active":"2024-09-21T10:40:42.387825+0000","last_peered":"2024-09-21T10:40:42.387825+0000","last_clean":"2024-09-21T10:40:42.387825+0000","last_became_active":"2024-09-21T10:09:06.538668+0000","last_became_peered":"2024-09-21T10:09:06.538668+0000","last_unstale":"2024-09-21T10:40:42.387825+0000","last_undegraded":"2024-09-21T10:40:42.387825+0000","last_fullsized":"2024-09-21T10:40:42.387825+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:42:44.299488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1918,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.385739+0000","last_change":"2024-09-21T10:40:30.419468+0000","last_active":"2024-09-21T10:40:42.385739+0000","last_peered":"2024-09-21T10:40:42.385739+0000","last_clean":"2024-09-21T10:40:42.385739+0000","last_became_active":"2024-09-21T10:40:30.419140+0000","last_became_peered":"2024-09-21T10:40:30.419140+0000","last_unstale":"2024-09-21T10:40:42.385739+0000","last_undegraded":"2024-09-21T10:40:42.385739+0000","last_fullsized":"2024-09-21T10:40:42.385739+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:37.424900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1286,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.396704+0000","last_change":"2024-09-21T10:07:55.998799+0000","last_active":"2024-09-21T10:40:41.396704+0000","last_peered":"2024-09-21T10:40:41.396704+0000","last_clean":"2024-09-21T10:40:41.396704+0000","last_became_active":"2024-09-21T10:07:55.998510+0000","last_became_peered":"2024-09-21T10:07:55.998510+0000","last_unstale":"2024-09-21T10:40:41.396704+0000","last_undegraded":"2024-09-21T10:40:41.396704+0000","last_fullsized":"2024-09-21T10:40:41.396704+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:10.474008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1306,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.396203+0000","last_change":"2024-09-21T10:40:30.419793+0000","last_active":"2024-09-21T10:40:41.396203+0000","last_peered":"2024-09-21T10:40:41.396203+0000","last_clean":"2024-09-21T10:40:41.396203+0000","last_became_active":"2024-09-21T10:40:30.419149+0000","last_became_peered":"2024-09-21T10:40:30.419149+0000","last_unstale":"2024-09-21T10:40:41.396203+0000","last_undegraded":"2024-09-21T10:40:41.396203+0000","last_fullsized":"2024-09-21T10:40:41.396203+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:33.741419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1310,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.396477+0000","last_change":"2024-09-21T10:40:13.432387+0000","last_active":"2024-09-21T10:40:41.396477+0000","last_peered":"2024-09-21T10:40:41.396477+0000","last_clean":"2024-09-21T10:40:41.396477+0000","last_became_active":"2024-09-21T10:40:13.432053+0000","last_became_peered":"2024-09-21T10:40:13.432053+0000","last_unstale":"2024-09-21T10:40:41.396477+0000","last_undegraded":"2024-09-21T10:40:41.396477+0000","last_fullsized":"2024-09-21T10:40:41.396477+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:37.486101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1289,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.386199+0000","last_change":"2024-09-21T10:07:56.020791+0000","last_active":"2024-09-21T10:40:41.386199+0000","last_peered":"2024-09-21T10:40:41.386199+0000","last_clean":"2024-09-21T10:40:41.386199+0000","last_became_active":"2024-09-21T10:07:56.020205+0000","last_became_peered":"2024-09-21T10:07:56.020205+0000","last_unstale":"2024-09-21T10:40:41.386199+0000","last_undegraded":"2024-09-21T10:40:41.386199+0000","last_fullsized":"2024-09-21T10:40:41.386199+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:09.813166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1286,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.396947+0000","last_change":"2024-09-21T10:09:06.533271+0000","last_active":"2024-09-21T10:40:41.396947+0000","last_peered":"2024-09-21T10:40:41.396947+0000","last_clean":"2024-09-21T10:40:41.396947+0000","last_became_active":"2024-09-21T10:09:06.532041+0000","last_became_peered":"2024-09-21T10:09:06.532041+0000","last_unstale":"2024-09-21T10:40:41.396947+0000","last_undegraded":"2024-09-21T10:40:41.396947+0000","last_fullsized":"2024-09-21T10:40:41.396947+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:04.236375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":1927,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.386238+0000","last_change":"2024-09-21T10:09:06.543422+0000","last_active":"2024-09-21T10:40:42.386238+0000","last_peered":"2024-09-21T10:40:42.386238+0000","last_clean":"2024-09-21T10:40:42.386238+0000","last_became_active":"2024-09-21T10:09:06.539647+0000","last_became_peered":"2024-09-21T10:09:06.539647+0000","last_unstale":"2024-09-21T10:40:42.386238+0000","last_undegraded":"2024-09-21T10:40:42.386238+0000","last_fullsized":"2024-09-21T10:40:42.386238+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:04.990085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1925,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.393303+0000","last_change":"2024-09-21T10:07:56.018560+0000","last_active":"2024-09-21T10:40:42.393303+0000","last_peered":"2024-09-21T10:40:42.393303+0000","last_clean":"2024-09-21T10:40:42.393303+0000","last_became_active":"2024-09-21T10:07:56.018261+0000","last_became_peered":"2024-09-21T10:07:56.018261+0000","last_unstale":"2024-09-21T10:40:42.393303+0000","last_undegraded":"2024-09-21T10:40:42.393303+0000","last_fullsized":"2024-09-21T10:40:42.393303+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:04:54.395456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1287,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.394160+0000","last_change":"2024-09-21T10:09:06.531935+0000","last_active":"2024-09-21T10:40:41.394160+0000","last_peered":"2024-09-21T10:40:41.394160+0000","last_clean":"2024-09-21T10:40:41.394160+0000","last_became_active":"2024-09-21T10:09:06.531610+0000","last_became_peered":"2024-09-21T10:09:06.531610+0000","last_unstale":"2024-09-21T10:40:41.394160+0000","last_undegraded":"2024-09-21T10:40:41.394160+0000","last_fullsized":"2024-09-21T10:40:41.394160+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:33.874405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1315,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.386076+0000","last_change":"2024-09-21T10:40:30.423107+0000","last_active":"2024-09-21T10:40:41.386076+0000","last_peered":"2024-09-21T10:40:41.386076+0000","last_clean":"2024-09-21T10:40:41.386076+0000","last_became_active":"2024-09-21T10:40:30.422770+0000","last_became_peered":"2024-09-21T10:40:30.422770+0000","last_unstale":"2024-09-21T10:40:41.386076+0000","last_undegraded":"2024-09-21T10:40:41.386076+0000","last_fullsized":"2024-09-21T10:40:41.386076+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:35:23.752720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":33,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.392336+0000","last_change":"2024-09-21T10:40:30.428793+0000","last_active":"2024-09-21T10:40:41.392336+0000","last_peered":"2024-09-21T10:40:41.392336+0000","last_clean":"2024-09-21T10:40:41.392336+0000","last_became_active":"2024-09-21T10:40:30.421116+0000","last_became_peered":"2024-09-21T10:40:30.421116+0000","last_unstale":"2024-09-21T10:40:41.392336+0000","last_undegraded":"2024-09-21T10:40:41.392336+0000","last_fullsized":"2024-09-21T10:40:41.392336+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:24:49.342671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1302,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.396073+0000","last_change":"2024-09-21T10:09:06.559132+0000","last_active":"2024-09-21T10:40:41.396073+0000","last_peered":"2024-09-21T10:40:41.396073+0000","last_clean":"2024-09-21T10:40:41.396073+0000","last_became_active":"2024-09-21T10:09:06.558395+0000","last_became_peered":"2024-09-21T10:09:06.558395+0000","last_unstale":"2024-09-21T10:40:41.396073+0000","last_undegraded":"2024-09-21T10:40:41.396073+0000","last_fullsized":"2024-09-21T10:40:41.396073+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:27:00.073554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":33,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.396621+0000","last_change":"2024-09-21T10:40:30.428625+0000","last_active":"2024-09-21T10:40:41.396621+0000","last_peered":"2024-09-21T10:40:41.396621+0000","last_clean":"2024-09-21T10:40:41.396621+0000","last_became_active":"2024-09-21T10:40:30.421565+0000","last_became_peered":"2024-09-21T10:40:30.421565+0000","last_unstale":"2024-09-21T10:40:41.396621+0000","last_undegraded":"2024-09-21T10:40:41.396621+0000","last_fullsized":"2024-09-21T10:40:41.396621+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:23.191395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1281,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.392233+0000","last_change":"2024-09-21T10:40:30.421601+0000","last_active":"2024-09-21T10:40:41.392233+0000","last_peered":"2024-09-21T10:40:41.392233+0000","last_clean":"2024-09-21T10:40:41.392233+0000","last_became_active":"2024-09-21T10:40:30.420430+0000","last_became_peered":"2024-09-21T10:40:30.420430+0000","last_unstale":"2024-09-21T10:40:41.392233+0000","last_undegraded":"2024-09-21T10:40:41.392233+0000","last_fullsized":"2024-09-21T10:40:41.392233+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:15.416335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":33,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.397775+0000","last_change":"2024-09-21T10:40:30.433050+0000","last_active":"2024-09-21T10:40:41.397775+0000","last_peered":"2024-09-21T10:40:41.397775+0000","last_clean":"2024-09-21T10:40:41.397775+0000","last_became_active":"2024-09-21T10:40:30.421211+0000","last_became_peered":"2024-09-21T10:40:30.421211+0000","last_unstale":"2024-09-21T10:40:41.397775+0000","last_undegraded":"2024-09-21T10:40:41.397775+0000","last_fullsized":"2024-09-21T10:40:41.397775+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:01:33.357050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1295,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.385788+0000","last_change":"2024-09-21T10:07:21.497877+0000","last_active":"2024-09-21T10:40:41.385788+0000","last_peered":"2024-09-21T10:40:41.385788+0000","last_clean":"2024-09-21T10:40:41.385788+0000","last_became_active":"2024-09-21T10:07:21.496317+0000","last_became_peered":"2024-09-21T10:07:21.496317+0000","last_unstale":"2024-09-21T10:40:41.385788+0000","last_undegraded":"2024-09-21T10:40:41.385788+0000","last_fullsized":"2024-09-21T10:40:41.385788+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:39:03.848183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1915,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.386559+0000","last_change":"2024-09-21T10:40:30.418410+0000","last_active":"2024-09-21T10:40:42.386559+0000","last_peered":"2024-09-21T10:40:42.386559+0000","last_clean":"2024-09-21T10:40:42.386559+0000","last_became_active":"2024-09-21T10:40:30.417846+0000","last_became_peered":"2024-09-21T10:40:30.417846+0000","last_unstale":"2024-09-21T10:40:42.386559+0000","last_undegraded":"2024-09-21T10:40:42.386559+0000","last_fullsized":"2024-09-21T10:40:42.386559+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:36:02.636480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.395935+0000","last_change":"2024-09-21T10:40:30.423721+0000","last_active":"2024-09-21T10:40:41.395935+0000","last_peered":"2024-09-21T10:40:41.395935+0000","last_clean":"2024-09-21T10:40:41.395935+0000","last_became_active":"2024-09-21T10:40:30.423415+0000","last_became_peered":"2024-09-21T10:40:30.423415+0000","last_unstale":"2024-09-21T10:40:41.395935+0000","last_undegraded":"2024-09-21T10:40:41.395935+0000","last_fullsized":"2024-09-21T10:40:41.395935+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:57.212463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1945,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.393594+0000","last_change":"2024-09-21T10:40:30.418240+0000","last_active":"2024-09-21T10:40:42.393594+0000","last_peered":"2024-09-21T10:40:42.393594+0000","last_clean":"2024-09-21T10:40:42.393594+0000","last_became_active":"2024-09-21T10:40:30.417843+0000","last_became_peered":"2024-09-21T10:40:30.417843+0000","last_unstale":"2024-09-21T10:40:42.393594+0000","last_undegraded":"2024-09-21T10:40:42.393594+0000","last_fullsized":"2024-09-21T10:40:42.393594+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:50.860905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1929,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.392561+0000","last_change":"2024-09-21T10:40:30.419513+0000","last_active":"2024-09-21T10:40:42.392561+0000","last_peered":"2024-09-21T10:40:42.392561+0000","last_clean":"2024-09-21T10:40:42.392561+0000","last_became_active":"2024-09-21T10:40:30.419182+0000","last_became_peered":"2024-09-21T10:40:30.419182+0000","last_unstale":"2024-09-21T10:40:42.392561+0000","last_undegraded":"2024-09-21T10:40:42.392561+0000","last_fullsized":"2024-09-21T10:40:42.392561+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:38:19.185388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1293,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.392473+0000","last_change":"2024-09-21T10:40:30.422480+0000","last_active":"2024-09-21T10:40:41.392473+0000","last_peered":"2024-09-21T10:40:41.392473+0000","last_clean":"2024-09-21T10:40:41.392473+0000","last_became_active":"2024-09-21T10:40:30.421818+0000","last_became_peered":"2024-09-21T10:40:30.421818+0000","last_unstale":"2024-09-21T10:40:41.392473+0000","last_undegraded":"2024-09-21T10:40:41.392473+0000","last_fullsized":"2024-09-21T10:40:41.392473+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:51:14.217028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":35,"reported_epoch":1317,"state":"active+clean","last_fresh":"2024-09-21T10:40:41.396815+0000","last_change":"2024-09-21T10:40:30.440162+0000","last_active":"2024-09-21T10:40:41.396815+0000","last_peered":"2024-09-21T10:40:41.396815+0000","last_clean":"2024-09-21T10:40:41.396815+0000","last_became_active":"2024-09-21T10:40:30.427218+0000","last_became_peered":"2024-09-21T10:40:30.427218+0000","last_unstale":"2024-09-21T10:40:41.396815+0000","last_undegraded":"2024-09-21T10:40:41.396815+0000","last_fullsized":"2024-09-21T10:40:41.396815+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:25:22.251409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1915,"reported_epoch":1318,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.386423+0000","last_change":"2024-09-21T10:40:30.418759+0000","last_active":"2024-09-21T10:40:42.386423+0000","last_peered":"2024-09-21T10:40:42.386423+0000","last_clean":"2024-09-21T10:40:42.386423+0000","last_became_active":"2024-09-21T10:40:30.418325+0000","last_became_peered":"2024-09-21T10:40:30.418325+0000","last_unstale":"2024-09-21T10:40:42.386423+0000","last_undegraded":"2024-09-21T10:40:42.386423+0000","last_fullsized":"2024-09-21T10:40:42.386423+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:57.793473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":53,"num_pg":16,"stat_sum":{"num_bytes":3030056960,"num_objects":46235,"num_object_clones":0,"num_object_copies":138705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46235,"num_write_kb":2959040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1235,"num_bytes_recovered":80936960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9173073920,"data_stored":9173073920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46235,"ondisk_log_size":46235,"up":48,"acting":48,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":5554,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":21634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234561,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496198,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1104672,"kb_used_data":965336,"kb_used_omap":1,"kb_used_meta":139326,"kb_avail":92640480,"statfs":{"total":95995035648,"available":94863851520,"internally_reserved":0,"allocated":988504064,"data_stored":984642476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":142670282},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757835,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1461068,"kb_used_data":1339716,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":92284084,"statfs":{"total":95995035648,"available":94498902016,"internally_reserved":0,"allocated":1371869184,"data_stored":1368022742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124254662},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019471,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2397720,"kb_used_data":2232144,"kb_used_omap":1,"kb_used_meta":165566,"kb_avail":91347432,"statfs":{"total":95995035648,"available":93539770368,"internally_reserved":0,"allocated":2285715456,"data_stored":2281860957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":169540038},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369214,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1145356,"kb_used_data":1054916,"kb_used_omap":69,"kb_used_meta":90362,"kb_avail":92599796,"statfs":{"total":95995035648,"available":94822191104,"internally_reserved":0,"allocated":1080233984,"data_stored":1075984658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":92530818},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2920577761669,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807957900,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1961592,"kb_used_data":1848456,"kb_used_omap":66,"kb_used_meta":112445,"kb_avail":91783560,"statfs":{"total":95995035648,"available":93986365440,"internally_reserved":0,"allocated":1892818944,"data_stored":1888565947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68589,"internal_metadata":115143699},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,3,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":666,"seq":2860448219539,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1322788,"kb_used_data":1232668,"kb_used_omap":76,"kb_used_meta":90035,"kb_avail":92422364,"statfs":{"total":95995035648,"available":94640500736,"internally_reserved":0,"allocated":1262252032,"data_stored":1257983998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78716,"internal_metadata":92195972},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1297285120,"data_stored":1297285120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1968177152,"data_stored":1968177152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1108279296,"data_stored":1108279296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2353856512,"data_stored":2353856512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1373962240,"data_stored":1373962240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":990576640,"data_stored":990576640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":80936960,"data_stored":80936960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:40:48.918 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T10:40:48.918 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_0 pg_num 26 2024-09-21T10:40:48.944 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:49.001 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:49.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:49.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:49.103 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:49.116 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 25 16 53533 53517 133.757 116.5 0.007364 0.00747113 2024-09-21T10:40:49.142 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:49.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:49.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:49.244 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:49.274 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:49.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:49.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:49.376 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:49.413 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:49.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:49.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:49.516 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:49.531 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:49 smithi070 ceph-mon[112139]: from='client.54709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:40:49.531 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:49 smithi070 ceph-mon[107385]: from='client.54709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:40:49.548 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:49.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:49.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:49.650 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:49.682 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:49.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:49.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:49.784 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:49.821 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:49.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:49.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:49 smithi160 ceph-mon[101879]: from='client.54709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:40:49.922 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:49.987 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:49.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:49.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:50.089 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:50.116 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 26 16 55334 55318 132.941 112.562 0.0104582 0.00751697 2024-09-21T10:40:50.128 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:50.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:50.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:50.230 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:50.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:50.265 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:50.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:50.366 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:50.402 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:50.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:50.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:50.503 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:50.554 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:50.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:50.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:50.656 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:50.695 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:50.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:50.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:50.796 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:50.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:50.830 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:50.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:50 smithi160 ceph-mon[101879]: pgmap v2176: 89 pgs: 89 active+clean; 2.8 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 107 MiB/s wr, 1.72k op/s; 5.6 MiB/s, 89 objects/s recovering 2024-09-21T10:40:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:40:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:40:50.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:50 smithi070 ceph-mon[107385]: pgmap v2176: 89 pgs: 89 active+clean; 2.8 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 107 MiB/s wr, 1.72k op/s; 5.6 MiB/s, 89 objects/s recovering 2024-09-21T10:40:50.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:40:50.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:40:50.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:50 smithi070 ceph-mon[112139]: pgmap v2176: 89 pgs: 89 active+clean; 2.8 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 107 MiB/s wr, 1.72k op/s; 5.6 MiB/s, 89 objects/s recovering 2024-09-21T10:40:50.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:40:50.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:40:50.931 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:50.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:50.967 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:50.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:51.067 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:51.102 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:51.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:51.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:51.116 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 27 16 57099 57083 132.103 110.312 0.00500379 0.00756479 2024-09-21T10:40:51.207 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:51.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:51.240 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:51.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:51.341 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:51.370 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:51.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:51.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:51.472 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:51.502 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:51.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:51.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:51.604 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:51.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:51.636 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:51.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:51.737 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:51.766 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:51.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:51.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:51.861 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:51.867 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:51.897 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:51.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:51.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:51.999 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:52.047 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:52.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:52.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:52.116 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 28 16 58681 58665 130.916 98.875 0.00822557 0.00763316 2024-09-21T10:40:52.149 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:52.199 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:52.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:52.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:52.302 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:52.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:52.352 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:52.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:52.431 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T10:40:52.453 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:52.509 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:52.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:52.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:52.611 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:52.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:52.656 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:52.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:52.757 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:52.760 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:52 smithi070 ceph-mon[107385]: pgmap v2177: 89 pgs: 89 active+clean; 3.1 GiB data, 9.9 GiB used, 526 GiB / 536 GiB avail; 88 MiB/s wr, 1.41k op/s; 248 KiB/s, 3 objects/s recovering 2024-09-21T10:40:52.760 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:52 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:40:52.760 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:52 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1704956054' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:40:52.760 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:52 smithi070 ceph-mon[112139]: pgmap v2177: 89 pgs: 89 active+clean; 3.1 GiB data, 9.9 GiB used, 526 GiB / 536 GiB avail; 88 MiB/s wr, 1.41k op/s; 248 KiB/s, 3 objects/s recovering 2024-09-21T10:40:52.760 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:52 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:40:52.760 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:52 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1704956054' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:40:52.799 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:52.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:52.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:52.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:52 smithi160 ceph-mon[101879]: pgmap v2177: 89 pgs: 89 active+clean; 3.1 GiB data, 9.9 GiB used, 526 GiB / 536 GiB avail; 88 MiB/s wr, 1.41k op/s; 248 KiB/s, 3 objects/s recovering 2024-09-21T10:40:52.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:52 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:40:52.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:52 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1704956054' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:40:52.901 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:52.941 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:52.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:52.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:53.042 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:53.073 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:53.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:53.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:53.117 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 29 16 60187 60171 129.647 94.125 0.00455424 0.00770782 2024-09-21T10:40:53.175 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:53.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:53.222 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:53.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:53.323 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:53.372 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:53.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:53.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:53.453 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:40:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:40:53] "GET /metrics HTTP/1.1" 200 36612 "" "Prometheus/2.51.0" 2024-09-21T10:40:53.474 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:53.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:53.512 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:53.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:53.613 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:53.649 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:53.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:53.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:53.711 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:53 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:40:53.711 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:53 smithi070 ceph-mon[112139]: Cluster is now healthy 2024-09-21T10:40:53.711 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:53 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1704956054' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:40:53.711 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:53 smithi070 ceph-mon[112139]: osdmap e1319: 8 total, 8 up, 6 in 2024-09-21T10:40:53.711 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:53 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:40:53.711 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:53 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:40:53.711 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:53 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:40:53.711 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:53 smithi070 ceph-mon[107385]: Cluster is now healthy 2024-09-21T10:40:53.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:53 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1704956054' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:40:53.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:53 smithi070 ceph-mon[107385]: osdmap e1319: 8 total, 8 up, 6 in 2024-09-21T10:40:53.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:53 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:40:53.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:53 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:40:53.751 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:53.785 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:53.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:53.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:53 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:40:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:53 smithi160 ceph-mon[101879]: Cluster is now healthy 2024-09-21T10:40:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:53 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1704956054' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:40:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:53 smithi160 ceph-mon[101879]: osdmap e1319: 8 total, 8 up, 6 in 2024-09-21T10:40:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:53 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:40:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:53 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:40:53.887 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:53.920 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:53.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:53.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:54.021 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:54.060 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:54.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:54.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:54.115 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:40:54.115 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:40:54.117 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 30 16 61525 61509 128.113 83.625 0.0105487 0.0078006 2024-09-21T10:40:54.161 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:54.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:54.198 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:54.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:54.299 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:54.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:54.338 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:54.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:54.438 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:54.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:54.473 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:54.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:54.574 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:54.625 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:54.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:54.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:54.727 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:54.759 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:54.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:54.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:54.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:54 smithi070 ceph-mon[107385]: pgmap v2179: 89 pgs: 89 active+clean; 3.4 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 109 MiB/s wr, 1.74k op/s; 230 KiB/s, 3 objects/s recovering 2024-09-21T10:40:54.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:54 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 4]} : dispatch 2024-09-21T10:40:54.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:54 smithi070 ceph-mon[112139]: pgmap v2179: 89 pgs: 89 active+clean; 3.4 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 109 MiB/s wr, 1.74k op/s; 230 KiB/s, 3 objects/s recovering 2024-09-21T10:40:54.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:54 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 4]} : dispatch 2024-09-21T10:40:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:54 smithi160 ceph-mon[101879]: pgmap v2179: 89 pgs: 89 active+clean; 3.4 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 109 MiB/s wr, 1.74k op/s; 230 KiB/s, 3 objects/s recovering 2024-09-21T10:40:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:54 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 4]} : dispatch 2024-09-21T10:40:54.861 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:54.896 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:54.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:54.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:54.997 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:55.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:55.044 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:55.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:55.117 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 31 16 62923 62907 126.799 87.375 0.00839776 0.00788146 2024-09-21T10:40:55.145 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:55.175 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:55.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:55.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:55.276 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:55.310 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:55.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:55.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:55.411 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:55.453 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:55.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:55.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:55.555 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:55.601 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:55.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:55.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:55.639 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:40:55.639 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:40:55.639 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:40:55.639 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:40:55.639 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:40:55.639 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:40:55.639 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:40:55.639 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:40:55.639 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:40:55.639 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:40:55.640 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:40:55.640 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:40:55.640 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:40:55.640 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:40:55.640 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:40:55.640 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:40:55.704 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:55.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:55.745 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:55.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:55.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:55 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 4]}]': finished 2024-09-21T10:40:55.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:55 smithi160 ceph-mon[101879]: osdmap e1320: 8 total, 8 up, 6 in 2024-09-21T10:40:55.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:55 smithi160 ceph-mon[101879]: pgmap v2181: 89 pgs: 89 active+clean; 3.6 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-09-21T10:40:55.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:55 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/113705529' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:40:55.845 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:55.874 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:55.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:55.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:55.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:55 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 4]}]': finished 2024-09-21T10:40:55.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:55 smithi070 ceph-mon[107385]: osdmap e1320: 8 total, 8 up, 6 in 2024-09-21T10:40:55.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:55 smithi070 ceph-mon[107385]: pgmap v2181: 89 pgs: 89 active+clean; 3.6 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-09-21T10:40:55.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:55 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/113705529' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:40:55.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:55 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 4]}]': finished 2024-09-21T10:40:55.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:55 smithi070 ceph-mon[112139]: osdmap e1320: 8 total, 8 up, 6 in 2024-09-21T10:40:55.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:55 smithi070 ceph-mon[112139]: pgmap v2181: 89 pgs: 89 active+clean; 3.6 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-09-21T10:40:55.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:55 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/113705529' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:40:55.975 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:56.020 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:56.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:56.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:56.119 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 32 16 64337 64321 125.59 88.375 0.0125911 0.00795705 2024-09-21T10:40:56.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:56.152 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:56.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:56.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:56.254 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:56.300 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:56.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:56.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:56.402 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:56.446 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:56.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:56.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:56.548 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:56.588 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:56.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:56.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:56.690 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:56.729 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:56.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:56.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:56.831 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:56 smithi160 ceph-mon[101879]: osdmap e1321: 8 total, 8 up, 6 in 2024-09-21T10:40:56.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:56 smithi070 ceph-mon[107385]: osdmap e1321: 8 total, 8 up, 6 in 2024-09-21T10:40:56.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:56 smithi070 ceph-mon[112139]: osdmap e1321: 8 total, 8 up, 6 in 2024-09-21T10:40:56.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:56.881 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:56.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:56.982 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:57.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:57.016 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:57.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:57.117 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:57.119 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 33 16 65853 65837 124.655 94.75 0.00449301 0.00801695 2024-09-21T10:40:57.150 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:40:57.153 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:57.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:57.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:57.255 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:57.286 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:57.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:57.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:57.350 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:40:57.388 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:57.436 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:57.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:57.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:57.487 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:57.538 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:57.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:57.576 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:57.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:57.612 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:57 smithi070 ceph-mon[112139]: pgmap v2183: 89 pgs: 89 active+clean; 3.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-09-21T10:40:57.613 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:57 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2823543210' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:40:57.613 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:57 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:40:57.613 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:57 smithi070 ceph-mon[107385]: pgmap v2183: 89 pgs: 89 active+clean; 3.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-09-21T10:40:57.613 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:57 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2823543210' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:40:57.613 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:57 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:40:57.618 INFO:teuthology.orchestra.run.smithi070.stderr:set pool 53 pg_num to 26 2024-09-21T10:40:57.677 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:57.705 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:57.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:57.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:57.807 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:57.828 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:57.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:57.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:57.930 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:57.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:57.971 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:57.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:58.071 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:57 smithi160 ceph-mon[101879]: pgmap v2183: 89 pgs: 89 active+clean; 3.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-09-21T10:40:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:57 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2823543210' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:40:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:57 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:40:58.106 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:58.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:58.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:58.119 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 34 15 67135 67120 123.348 80.1875 0.0109185 0.00810243 2024-09-21T10:40:58.208 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:58.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:58.245 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:58.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:58.346 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:58.378 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:58.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:58.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:58.480 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:58.514 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:58.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:58.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:58.616 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:58.650 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:40:58.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:58.651 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:58.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:58.752 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:58.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:58.780 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:58.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:58.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:58 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-21T10:40:58.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:58 smithi070 ceph-mon[107385]: osdmap e1322: 8 total, 8 up, 6 in 2024-09-21T10:40:58.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:58 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-21T10:40:58.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:58 smithi070 ceph-mon[112139]: osdmap e1322: 8 total, 8 up, 6 in 2024-09-21T10:40:58.881 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:58.917 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:58.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:58.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:59.018 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:59.052 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:59.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:59.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:59.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:58 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-21T10:40:59.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:58 smithi160 ceph-mon[101879]: osdmap e1322: 8 total, 8 up, 6 in 2024-09-21T10:40:59.119 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 35 16 68533 68517 122.318 87.3125 0.00988987 0.00817055 2024-09-21T10:40:59.153 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:59.190 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:59.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:59.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:59.292 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:59.315 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:59.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:59.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:59.417 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:59.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:59.459 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:40:59.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:59.561 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:59.581 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:59.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:59.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:59.683 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:59.716 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:59.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:59.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:59.759 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:59 smithi070 ceph-mon[107385]: pgmap v2185: 89 pgs: 89 active+clean; 3.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-21T10:40:59.759 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:40:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T10:40:59.759 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:59 smithi070 ceph-mon[112139]: pgmap v2185: 89 pgs: 89 active+clean; 3.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-21T10:40:59.759 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:40:59 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T10:40:59.818 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:59.839 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:59.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:59.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:59.941 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:59.962 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:40:59.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:59.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:00.063 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:00.084 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:00.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:00.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:00.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:59 smithi160 ceph-mon[101879]: pgmap v2185: 89 pgs: 89 active+clean; 3.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-21T10:41:00.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:40:59 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T10:41:00.119 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 36 16 69157 69141 120.004 39 0.00307153 0.00821639 2024-09-21T10:41:00.186 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:00.212 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:00.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:00.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:00.313 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:00.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:00.335 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:00.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:00.436 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:00.457 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:00.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:00.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:00.559 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:00.584 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:00.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:00.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:00.686 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:00.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:00.723 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:00.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:00.824 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:00.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:00 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T10:41:00.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:00 smithi160 ceph-mon[101879]: osdmap e1323: 8 total, 8 up, 6 in 2024-09-21T10:41:00.865 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:00.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:00.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:00.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:00 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T10:41:00.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:00 smithi070 ceph-mon[112139]: osdmap e1323: 8 total, 8 up, 6 in 2024-09-21T10:41:00.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:00 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T10:41:00.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:00 smithi070 ceph-mon[107385]: osdmap e1323: 8 total, 8 up, 6 in 2024-09-21T10:41:00.967 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:01.003 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:01.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:01.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:01.106 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:01.120 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 37 16 69895 69879 118.007 46.125 0.00442065 0.00846421 2024-09-21T10:41:01.128 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:01.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:01.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:01.230 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:01.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:01.279 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:01.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:01.380 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:01.408 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:01.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:01.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:01.510 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:01.544 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:01.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:01.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:01.646 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:01.683 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:01.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:01.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:01.784 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:01.818 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:01.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:01.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:01.920 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:01.957 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:01.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:01.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:02.058 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:02.090 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:02.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:02.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:02.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:01 smithi160 ceph-mon[101879]: osdmap e1324: 8 total, 8 up, 6 in 2024-09-21T10:41:02.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:01 smithi160 ceph-mon[101879]: pgmap v2188: 99 pgs: 10 unknown, 89 active+clean; 4.0 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-21T10:41:02.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:01 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:41:02.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:01 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:41:02.119 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 38 16 71423 71407 117.415 95.5 0.0121694 0.00850426 2024-09-21T10:41:02.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:01 smithi070 ceph-mon[107385]: osdmap e1324: 8 total, 8 up, 6 in 2024-09-21T10:41:02.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:01 smithi070 ceph-mon[107385]: pgmap v2188: 99 pgs: 10 unknown, 89 active+clean; 4.0 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-21T10:41:02.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:01 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:41:02.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:01 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:41:02.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:01 smithi070 ceph-mon[112139]: osdmap e1324: 8 total, 8 up, 6 in 2024-09-21T10:41:02.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:01 smithi070 ceph-mon[112139]: pgmap v2188: 99 pgs: 10 unknown, 89 active+clean; 4.0 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-21T10:41:02.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:01 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:41:02.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:01 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:41:02.192 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:02.228 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:02.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:02.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:02.330 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:02.365 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:02.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:02.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:02.467 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:02.498 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:02.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:02.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:02.600 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:02.621 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:02.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:02.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:02.722 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:02.754 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:02.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:02.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:02.855 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:02.893 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:02.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:02.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:02.995 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:03.027 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:03.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:03.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:02 smithi160 ceph-mon[101879]: 53.7 scrub starts 2024-09-21T10:41:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:02 smithi160 ceph-mon[101879]: 53.3 scrub starts 2024-09-21T10:41:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:02 smithi160 ceph-mon[101879]: 53.8 scrub starts 2024-09-21T10:41:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:02 smithi160 ceph-mon[101879]: 53.7 scrub ok 2024-09-21T10:41:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:02 smithi160 ceph-mon[101879]: 53.3 scrub ok 2024-09-21T10:41:03.119 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 39 16 73077 73061 117.055 103.375 0.00560264 0.00853852 2024-09-21T10:41:03.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:02 smithi070 ceph-mon[107385]: 53.7 scrub starts 2024-09-21T10:41:03.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:02 smithi070 ceph-mon[107385]: 53.3 scrub starts 2024-09-21T10:41:03.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:02 smithi070 ceph-mon[107385]: 53.8 scrub starts 2024-09-21T10:41:03.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:02 smithi070 ceph-mon[107385]: 53.7 scrub ok 2024-09-21T10:41:03.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:02 smithi070 ceph-mon[107385]: 53.3 scrub ok 2024-09-21T10:41:03.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:02 smithi070 ceph-mon[112139]: 53.7 scrub starts 2024-09-21T10:41:03.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:02 smithi070 ceph-mon[112139]: 53.3 scrub starts 2024-09-21T10:41:03.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:02 smithi070 ceph-mon[112139]: 53.8 scrub starts 2024-09-21T10:41:03.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:02 smithi070 ceph-mon[112139]: 53.7 scrub ok 2024-09-21T10:41:03.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:02 smithi070 ceph-mon[112139]: 53.3 scrub ok 2024-09-21T10:41:03.130 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:03.157 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:03.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:03.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:03.259 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:03.293 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:03.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:03.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:03.395 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:03.428 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:03.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:03.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:03.529 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:03.558 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:03.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:03.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:41:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:41:03] "GET /metrics HTTP/1.1" 200 36612 "" "Prometheus/2.51.0" 2024-09-21T10:41:03.660 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:03.697 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:03.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:03.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:03.799 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:03.822 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:03.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:03.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:03.923 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:03.957 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:03.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:03.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:04.059 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:04.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:03 smithi160 ceph-mon[101879]: 53.1 scrub starts 2024-09-21T10:41:04.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:03 smithi160 ceph-mon[101879]: 53.8 scrub ok 2024-09-21T10:41:04.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:03 smithi160 ceph-mon[101879]: 53.0 scrub starts 2024-09-21T10:41:04.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:03 smithi160 ceph-mon[101879]: 53.1 scrub ok 2024-09-21T10:41:04.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:03 smithi160 ceph-mon[101879]: pgmap v2189: 99 pgs: 1 active+clean+scrubbing, 6 unknown, 92 active+clean; 4.2 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 65 MiB/s wr, 1.04k op/s; 11 KiB/s, 0 objects/s recovering 2024-09-21T10:41:04.094 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:04.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:04.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:04.120 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:2024-09-21T10:41:04.119058+0000 min lat: 0.00114886 max lat: 2.0939 avg lat: 0.00855618 2024-09-21T10:41:04.120 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:41:04.120 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 40 16 74794 74778 116.811 107.312 0.00331462 0.00855618 2024-09-21T10:41:04.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:03 smithi070 ceph-mon[107385]: 53.1 scrub starts 2024-09-21T10:41:04.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:03 smithi070 ceph-mon[107385]: 53.8 scrub ok 2024-09-21T10:41:04.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:03 smithi070 ceph-mon[107385]: 53.0 scrub starts 2024-09-21T10:41:04.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:03 smithi070 ceph-mon[107385]: 53.1 scrub ok 2024-09-21T10:41:04.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:03 smithi070 ceph-mon[107385]: pgmap v2189: 99 pgs: 1 active+clean+scrubbing, 6 unknown, 92 active+clean; 4.2 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 65 MiB/s wr, 1.04k op/s; 11 KiB/s, 0 objects/s recovering 2024-09-21T10:41:04.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:03 smithi070 ceph-mon[112139]: 53.1 scrub starts 2024-09-21T10:41:04.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:03 smithi070 ceph-mon[112139]: 53.8 scrub ok 2024-09-21T10:41:04.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:03 smithi070 ceph-mon[112139]: 53.0 scrub starts 2024-09-21T10:41:04.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:03 smithi070 ceph-mon[112139]: 53.1 scrub ok 2024-09-21T10:41:04.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:03 smithi070 ceph-mon[112139]: pgmap v2189: 99 pgs: 1 active+clean+scrubbing, 6 unknown, 92 active+clean; 4.2 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 65 MiB/s wr, 1.04k op/s; 11 KiB/s, 0 objects/s recovering 2024-09-21T10:41:04.198 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:04.237 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:04.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:04.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:04.339 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:04.375 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:04.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:04.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:04.476 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:04.512 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:04.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:04.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:04.614 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:04.648 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:04.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:04.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:04.750 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:04.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:04.788 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:04.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:04.889 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:04.931 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:04.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:04.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:05.020 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:05.033 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:05.087 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:05.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:05.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:05.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:04 smithi160 ceph-mon[101879]: 53.0 scrub ok 2024-09-21T10:41:05.120 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 41 16 76481 76465 116.533 105.438 0.00818221 0.00857673 2024-09-21T10:41:05.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:04 smithi070 ceph-mon[107385]: 53.0 scrub ok 2024-09-21T10:41:05.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:04 smithi070 ceph-mon[112139]: 53.0 scrub ok 2024-09-21T10:41:05.189 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:05.220 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:05.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:05.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:05.322 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:05.363 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:05.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:05.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:05.465 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:05.501 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:05.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:05.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:05.508 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:05.603 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:05.639 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:05.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:05.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:05.741 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:05.749 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:05 smithi070 ceph-mon[112139]: 53.4 scrub starts 2024-09-21T10:41:05.749 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:05 smithi070 ceph-mon[112139]: 53.13 scrub starts 2024-09-21T10:41:05.749 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:05 smithi070 ceph-mon[112139]: 53.4 scrub ok 2024-09-21T10:41:05.749 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:05 smithi070 ceph-mon[112139]: 53.13 scrub ok 2024-09-21T10:41:05.750 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:05 smithi070 ceph-mon[112139]: pgmap v2190: 99 pgs: 2 active+clean+scrubbing, 97 active+clean; 4.4 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 44 MiB/s wr, 700 op/s; 18 KiB/s, 0 objects/s recovering 2024-09-21T10:41:05.750 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-21T10:41:05.750 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:41:05.750 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:41:05.750 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:05 smithi070 ceph-mon[107385]: 53.4 scrub starts 2024-09-21T10:41:05.750 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:05 smithi070 ceph-mon[107385]: 53.13 scrub starts 2024-09-21T10:41:05.750 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:05 smithi070 ceph-mon[107385]: 53.4 scrub ok 2024-09-21T10:41:05.750 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:05 smithi070 ceph-mon[107385]: 53.13 scrub ok 2024-09-21T10:41:05.751 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:05 smithi070 ceph-mon[107385]: pgmap v2190: 99 pgs: 2 active+clean+scrubbing, 97 active+clean; 4.4 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 44 MiB/s wr, 700 op/s; 18 KiB/s, 0 objects/s recovering 2024-09-21T10:41:05.751 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-21T10:41:05.751 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:41:05.751 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:41:05.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:05.790 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:05.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:05.891 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:05.915 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:05.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:05.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:06.017 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:06.038 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:06.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:06.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:05 smithi160 ceph-mon[101879]: 53.4 scrub starts 2024-09-21T10:41:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:05 smithi160 ceph-mon[101879]: 53.13 scrub starts 2024-09-21T10:41:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:05 smithi160 ceph-mon[101879]: 53.4 scrub ok 2024-09-21T10:41:06.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:05 smithi160 ceph-mon[101879]: 53.13 scrub ok 2024-09-21T10:41:06.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:05 smithi160 ceph-mon[101879]: pgmap v2190: 99 pgs: 2 active+clean+scrubbing, 97 active+clean; 4.4 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 44 MiB/s wr, 700 op/s; 18 KiB/s, 0 objects/s recovering 2024-09-21T10:41:06.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-21T10:41:06.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:41:06.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:41:06.120 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 42 16 77683 77667 115.547 75.125 0.00198305 0.00858295 2024-09-21T10:41:06.140 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:06.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:06.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:06.263 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:06.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:06.285 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:06.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:06.386 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:06.407 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:06.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:06.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:06.509 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:06.530 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:06.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:06.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:06.631 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:06.657 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:06.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:06.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:06.759 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:06.781 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:06.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:06.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:06.883 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:06.905 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:06.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:06.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:07.007 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:07.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:07.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:07.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:06 smithi160 ceph-mon[101879]: 53.14 scrub starts 2024-09-21T10:41:07.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:06 smithi160 ceph-mon[101879]: 53.2 scrub starts 2024-09-21T10:41:07.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:06 smithi160 ceph-mon[101879]: 53.14 scrub ok 2024-09-21T10:41:07.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:06 smithi160 ceph-mon[101879]: 53.6 scrub starts 2024-09-21T10:41:07.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:06 smithi160 ceph-mon[101879]: 53.2 scrub ok 2024-09-21T10:41:07.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-21T10:41:07.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:06 smithi160 ceph-mon[101879]: osdmap e1325: 8 total, 8 up, 6 in 2024-09-21T10:41:07.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:06 smithi160 ceph-mon[101879]: 53.6 scrub ok 2024-09-21T10:41:07.120 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 43 16 77683 77667 112.86 0 - 0.00858295 2024-09-21T10:41:07.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:06 smithi070 ceph-mon[107385]: 53.14 scrub starts 2024-09-21T10:41:07.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:06 smithi070 ceph-mon[107385]: 53.2 scrub starts 2024-09-21T10:41:07.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:06 smithi070 ceph-mon[107385]: 53.14 scrub ok 2024-09-21T10:41:07.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:06 smithi070 ceph-mon[107385]: 53.6 scrub starts 2024-09-21T10:41:07.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:06 smithi070 ceph-mon[107385]: 53.2 scrub ok 2024-09-21T10:41:07.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-21T10:41:07.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:06 smithi070 ceph-mon[107385]: osdmap e1325: 8 total, 8 up, 6 in 2024-09-21T10:41:07.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:06 smithi070 ceph-mon[107385]: 53.6 scrub ok 2024-09-21T10:41:07.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:06 smithi070 ceph-mon[112139]: 53.14 scrub starts 2024-09-21T10:41:07.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:06 smithi070 ceph-mon[112139]: 53.2 scrub starts 2024-09-21T10:41:07.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:06 smithi070 ceph-mon[112139]: 53.14 scrub ok 2024-09-21T10:41:07.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:06 smithi070 ceph-mon[112139]: 53.6 scrub starts 2024-09-21T10:41:07.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:06 smithi070 ceph-mon[112139]: 53.2 scrub ok 2024-09-21T10:41:07.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-21T10:41:07.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:06 smithi070 ceph-mon[112139]: osdmap e1325: 8 total, 8 up, 6 in 2024-09-21T10:41:07.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:06 smithi070 ceph-mon[112139]: 53.6 scrub ok 2024-09-21T10:41:07.129 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:07.155 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:07.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:07.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:07.257 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:07.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:07.278 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:07.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:07.379 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:07.399 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:07.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:07.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:07.501 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:07.527 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:07.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:07.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:07.629 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:07.654 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:07.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:07.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:07.756 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:07.768 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:07 smithi070 ceph-mon[112139]: 53.12 scrub starts 2024-09-21T10:41:07.768 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:07 smithi070 ceph-mon[112139]: 53.12 scrub ok 2024-09-21T10:41:07.768 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:07 smithi070 ceph-mon[112139]: osdmap e1326: 8 total, 8 up, 6 in 2024-09-21T10:41:07.768 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:07 smithi070 ceph-mon[112139]: pgmap v2193: 99 pgs: 1 active+clean+scrubbing, 98 active+clean; 4.6 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 11 KiB/s, 0 objects/s recovering 2024-09-21T10:41:07.768 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-21T10:41:07.768 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:07 smithi070 ceph-mon[112139]: 53.15 scrub starts 2024-09-21T10:41:07.768 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:07 smithi070 ceph-mon[112139]: 53.15 scrub ok 2024-09-21T10:41:07.769 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-21T10:41:07.769 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:07 smithi070 ceph-mon[112139]: osdmap e1327: 8 total, 8 up, 6 in 2024-09-21T10:41:07.769 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:07 smithi070 ceph-mon[107385]: 53.12 scrub starts 2024-09-21T10:41:07.769 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:07 smithi070 ceph-mon[107385]: 53.12 scrub ok 2024-09-21T10:41:07.769 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:07 smithi070 ceph-mon[107385]: osdmap e1326: 8 total, 8 up, 6 in 2024-09-21T10:41:07.769 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:07 smithi070 ceph-mon[107385]: pgmap v2193: 99 pgs: 1 active+clean+scrubbing, 98 active+clean; 4.6 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 11 KiB/s, 0 objects/s recovering 2024-09-21T10:41:07.769 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-21T10:41:07.769 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:07 smithi070 ceph-mon[107385]: 53.15 scrub starts 2024-09-21T10:41:07.769 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:07 smithi070 ceph-mon[107385]: 53.15 scrub ok 2024-09-21T10:41:07.769 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-21T10:41:07.769 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:07 smithi070 ceph-mon[107385]: osdmap e1327: 8 total, 8 up, 6 in 2024-09-21T10:41:07.780 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:07.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:07.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:07.881 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:07.912 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:07.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:07.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:08.014 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:08.051 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:08.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:08.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:08.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:07 smithi160 ceph-mon[101879]: 53.12 scrub starts 2024-09-21T10:41:08.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:07 smithi160 ceph-mon[101879]: 53.12 scrub ok 2024-09-21T10:41:08.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:07 smithi160 ceph-mon[101879]: osdmap e1326: 8 total, 8 up, 6 in 2024-09-21T10:41:08.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:07 smithi160 ceph-mon[101879]: pgmap v2193: 99 pgs: 1 active+clean+scrubbing, 98 active+clean; 4.6 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 11 KiB/s, 0 objects/s recovering 2024-09-21T10:41:08.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-21T10:41:08.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:07 smithi160 ceph-mon[101879]: 53.15 scrub starts 2024-09-21T10:41:08.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:07 smithi160 ceph-mon[101879]: 53.15 scrub ok 2024-09-21T10:41:08.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-21T10:41:08.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:07 smithi160 ceph-mon[101879]: osdmap e1327: 8 total, 8 up, 6 in 2024-09-21T10:41:08.121 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 44 16 78073 78057 110.849 12.1875 0.00398919 0.00898156 2024-09-21T10:41:08.152 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:08.176 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:08.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:08.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:08.278 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:08.299 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:08.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:08.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:08.401 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:08.422 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:08.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:08.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:08.524 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:08.545 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:08.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:08.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:08.646 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:08.662 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T10:41:08.668 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:08.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:08.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:08.769 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:08.800 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:08.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:08.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:08.902 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:08.935 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:08.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:08.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:09.036 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:09.070 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:09.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:09.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:09.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:08 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3617293737' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:41:09.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:08 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:41:09.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:08 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3617293737' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:41:09.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:08 smithi160 ceph-mon[101879]: osdmap e1328: 8 total, 8 up, 6 in 2024-09-21T10:41:09.121 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 45 16 78884 78868 109.512 50.6875 0.00413415 0.00912682 2024-09-21T10:41:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:08 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3617293737' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:41:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:08 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:41:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:08 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3617293737' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:41:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:08 smithi070 ceph-mon[107385]: osdmap e1328: 8 total, 8 up, 6 in 2024-09-21T10:41:09.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:08 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3617293737' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:41:09.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:08 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:41:09.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:08 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3617293737' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:41:09.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:08 smithi070 ceph-mon[112139]: osdmap e1328: 8 total, 8 up, 6 in 2024-09-21T10:41:09.173 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:09.206 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:09.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:09.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:09.308 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:09.344 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:09.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:09.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:09.362 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6] out_osds: [2, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:41:09.362 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:41:09.362 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-21T10:41:09.362 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:41:09.446 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:09.481 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:09.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:09.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:09.582 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:09.624 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:09.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:09.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:09.725 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:09.749 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:41:09.749 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T10:41:09.749 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T10:41:09.749 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T10:41:09.749 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.749 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:69135 53.4 53:24e51e3f:::benchmark_data_smithi070_228967_object69134:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1322)", 2024-09-21T10:41:09.749 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:40:59.556013+0000", 2024-09-21T10:41:09.749 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 9.2308843389999993, 2024-09-21T10:41:09.750 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.24871251599999999, 2024-09-21T10:41:09.750 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:41:09.750 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:41:09.750 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:41:09.750 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:41:09.750 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:41:09.750 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:41:09.750 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 69135 2024-09-21T10:41:09.750 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.750 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:41:09.750 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.750 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:41:09.750 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.556013+0000", 2024-09-21T10:41:09.751 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.751 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.751 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.751 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:41:09.751 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.556013+0000", 2024-09-21T10:41:09.751 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.751 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.751 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.751 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:41:09.751 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.556014+0000", 2024-09-21T10:41:09.751 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.451e-06 2024-09-21T10:41:09.751 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.751 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.752 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:41:09.752 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.556035+0000", 2024-09-21T10:41:09.752 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0995000000000001e-05 2024-09-21T10:41:09.752 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.752 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.752 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:41:09.752 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.556045+0000", 2024-09-21T10:41:09.752 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9680000000000001e-06 2024-09-21T10:41:09.752 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.752 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.752 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:41:09.752 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.556052+0000", 2024-09-21T10:41:09.752 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6839999999999999e-06 2024-09-21T10:41:09.753 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.753 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.753 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.753 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.556104+0000", 2024-09-21T10:41:09.753 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2185999999999998e-05 2024-09-21T10:41:09.753 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.753 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.753 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:41:09.753 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.556240+0000", 2024-09-21T10:41:09.753 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013657799999999999 2024-09-21T10:41:09.753 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.753 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.753 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:41:09.754 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.556386+0000", 2024-09-21T10:41:09.754 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014505799999999999 2024-09-21T10:41:09.754 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.754 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.754 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:09.754 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.559000+0000", 2024-09-21T10:41:09.754 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0026141519999999998 2024-09-21T10:41:09.754 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.754 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.754 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:41:09.754 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.559039+0000", 2024-09-21T10:41:09.754 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9057000000000002e-05 2024-09-21T10:41:09.754 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.754 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.755 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.755 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.804713+0000", 2024-09-21T10:41:09.755 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.24567454 2024-09-21T10:41:09.755 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.755 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.755 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:41:09.755 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.804725+0000", 2024-09-21T10:41:09.755 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1847e-05 2024-09-21T10:41:09.755 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.755 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:41:09.755 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.755 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.755 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.755 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:69140 53.1 53:8c59eef6:::benchmark_data_smithi070_228967_object69139:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1322)", 2024-09-21T10:41:09.756 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:40:59.558924+0000", 2024-09-21T10:41:09.756 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 9.2279730149999999, 2024-09-21T10:41:09.756 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20915397299999999, 2024-09-21T10:41:09.756 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:41:09.756 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:41:09.756 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:41:09.756 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:41:09.756 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:41:09.756 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:41:09.756 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 69140 2024-09-21T10:41:09.756 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.756 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:41:09.756 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.756 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:41:09.757 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.558924+0000", 2024-09-21T10:41:09.757 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.757 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.757 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.757 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:41:09.757 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.558924+0000", 2024-09-21T10:41:09.757 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.757 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.757 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.757 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:41:09.757 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.558925+0000", 2024-09-21T10:41:09.757 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7000000000000003e-07 2024-09-21T10:41:09.757 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.757 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.758 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:41:09.758 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.558944+0000", 2024-09-21T10:41:09.758 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9539999999999999e-05 2024-09-21T10:41:09.758 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.758 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.758 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:41:09.758 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.558973+0000", 2024-09-21T10:41:09.758 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8257999999999999e-05 2024-09-21T10:41:09.758 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.758 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.758 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:41:09.758 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.558981+0000", 2024-09-21T10:41:09.758 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5639999999999997e-06 2024-09-21T10:41:09.759 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.759 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.759 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.759 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.559047+0000", 2024-09-21T10:41:09.759 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5724999999999998e-05 2024-09-21T10:41:09.759 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.759 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.759 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:41:09.759 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.559180+0000", 2024-09-21T10:41:09.759 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013296799999999999 2024-09-21T10:41:09.759 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.759 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.759 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:41:09.759 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.559331+0000", 2024-09-21T10:41:09.761 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000151368 2024-09-21T10:41:09.761 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.761 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.761 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:41:09.761 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.561724+0000", 2024-09-21T10:41:09.761 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002393086 2024-09-21T10:41:09.762 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.762 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.762 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:09.762 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.563758+0000", 2024-09-21T10:41:09.762 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0020331419999999999 2024-09-21T10:41:09.762 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.762 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.762 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.762 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.768064+0000", 2024-09-21T10:41:09.762 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20430605700000001 2024-09-21T10:41:09.762 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.762 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.762 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:41:09.763 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.768078+0000", 2024-09-21T10:41:09.763 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4295e-05 2024-09-21T10:41:09.763 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.763 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:41:09.763 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.763 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.763 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.763 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.45020.0:69143 53.c e1322/1317 53:3f048da0:::benchmark_data_smithi070_228967_object69142:head v 1322'4484, mlcod=1322'4482)", 2024-09-21T10:41:09.763 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:40:59.624172+0000", 2024-09-21T10:41:09.763 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 9.1627246390000003, 2024-09-21T10:41:09.763 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.18133756700000001, 2024-09-21T10:41:09.763 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:41:09.764 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:41:09.764 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:41:09.764 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:41:09.764 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.764 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:41:09.764 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.624172+0000", 2024-09-21T10:41:09.764 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.764 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.764 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.764 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:41:09.764 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.624172+0000", 2024-09-21T10:41:09.764 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.764 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.765 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.765 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:41:09.765 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.624174+0000", 2024-09-21T10:41:09.765 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5889999999999999e-06 2024-09-21T10:41:09.765 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.765 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.765 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:41:09.765 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.624260+0000", 2024-09-21T10:41:09.765 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5915000000000007e-05 2024-09-21T10:41:09.765 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.765 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.765 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:41:09.765 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.624271+0000", 2024-09-21T10:41:09.766 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1212999999999999e-05 2024-09-21T10:41:09.766 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.766 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.766 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:41:09.766 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.624282+0000", 2024-09-21T10:41:09.766 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1211e-05 2024-09-21T10:41:09.766 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.766 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.766 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.766 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.633535+0000", 2024-09-21T10:41:09.766 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0092530879999999996 2024-09-21T10:41:09.766 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.766 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.766 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:41:09.767 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.633563+0000", 2024-09-21T10:41:09.767 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7328999999999999e-05 2024-09-21T10:41:09.767 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.767 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.767 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:41:09.767 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.805467+0000", 2024-09-21T10:41:09.767 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.171904801 2024-09-21T10:41:09.767 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.767 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.767 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:41:09.767 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.805510+0000", 2024-09-21T10:41:09.767 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2420999999999999e-05 2024-09-21T10:41:09.767 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.768 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:41:09.768 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.768 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.768 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.768 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:69144 53.4 53.90737214 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1322)", 2024-09-21T10:41:09.768 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:40:59.625196+0000", 2024-09-21T10:41:09.768 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 9.1617006280000002, 2024-09-21T10:41:09.768 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.17956636600000001, 2024-09-21T10:41:09.768 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:41:09.768 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:41:09.768 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:41:09.768 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:41:09.768 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:41:09.768 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:41:09.769 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 69144 2024-09-21T10:41:09.769 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.769 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:41:09.769 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.769 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:41:09.769 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.625196+0000", 2024-09-21T10:41:09.769 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.769 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.769 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.769 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:41:09.769 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.625196+0000", 2024-09-21T10:41:09.769 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.769 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.769 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.770 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:41:09.770 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.625198+0000", 2024-09-21T10:41:09.770 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.136e-06 2024-09-21T10:41:09.770 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.770 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.770 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:41:09.770 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.625229+0000", 2024-09-21T10:41:09.770 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0372999999999999e-05 2024-09-21T10:41:09.770 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.770 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.770 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:41:09.770 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.625243+0000", 2024-09-21T10:41:09.770 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4647000000000001e-05 2024-09-21T10:41:09.770 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.770 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.771 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:41:09.771 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.625252+0000", 2024-09-21T10:41:09.771 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.157e-06 2024-09-21T10:41:09.771 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.771 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.771 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.771 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.804754+0000", 2024-09-21T10:41:09.771 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.17950281600000001 2024-09-21T10:41:09.771 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.771 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.771 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:41:09.771 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.804763+0000", 2024-09-21T10:41:09.771 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2369999999999992e-06 2024-09-21T10:41:09.771 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.772 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:41:09.772 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.772 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.772 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.772 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:69145 53.4 53.850dd1c4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1322)", 2024-09-21T10:41:09.772 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:40:59.625270+0000", 2024-09-21T10:41:09.772 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 9.1616270110000002, 2024-09-21T10:41:09.772 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.179528465, 2024-09-21T10:41:09.772 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:41:09.772 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:41:09.772 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:41:09.772 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:41:09.772 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:41:09.772 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:41:09.772 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 69145 2024-09-21T10:41:09.773 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.773 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:41:09.773 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.773 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:41:09.773 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.625270+0000", 2024-09-21T10:41:09.773 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.773 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.773 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.773 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:41:09.773 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.625270+0000", 2024-09-21T10:41:09.773 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.773 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.773 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.773 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:41:09.773 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.625271+0000", 2024-09-21T10:41:09.773 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8599999999999998e-07 2024-09-21T10:41:09.774 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.774 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.774 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:41:09.774 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.625290+0000", 2024-09-21T10:41:09.774 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9619999999999998e-05 2024-09-21T10:41:09.774 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.774 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.774 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:41:09.774 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.625298+0000", 2024-09-21T10:41:09.774 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8730000000000007e-06 2024-09-21T10:41:09.774 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.774 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.774 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:41:09.775 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.625304+0000", 2024-09-21T10:41:09.775 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7409999999999998e-06 2024-09-21T10:41:09.775 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.775 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.775 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.775 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.804790+0000", 2024-09-21T10:41:09.775 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.17948651600000001 2024-09-21T10:41:09.775 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.775 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.775 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:41:09.775 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.804798+0000", 2024-09-21T10:41:09.775 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1289999999999996e-06 2024-09-21T10:41:09.775 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.775 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:41:09.776 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.776 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.776 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.776 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:69146 53.4 53.77bf594 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1322)", 2024-09-21T10:41:09.776 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:40:59.625500+0000", 2024-09-21T10:41:09.776 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 9.1613967869999993, 2024-09-21T10:41:09.776 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.17933437599999999, 2024-09-21T10:41:09.776 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:41:09.776 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:41:09.776 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:41:09.776 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:41:09.776 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:41:09.776 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:41:09.776 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 69146 2024-09-21T10:41:09.777 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.777 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:41:09.777 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.777 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:41:09.777 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.625500+0000", 2024-09-21T10:41:09.777 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.777 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.777 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.777 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:41:09.777 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.625500+0000", 2024-09-21T10:41:09.777 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.777 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.777 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.777 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:41:09.778 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.625501+0000", 2024-09-21T10:41:09.778 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4799999999999997e-07 2024-09-21T10:41:09.778 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.778 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.778 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:41:09.778 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.625522+0000", 2024-09-21T10:41:09.778 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0752999999999999e-05 2024-09-21T10:41:09.778 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.778 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.778 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:41:09.778 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.625532+0000", 2024-09-21T10:41:09.778 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7650000000000005e-06 2024-09-21T10:41:09.778 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.779 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.779 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:41:09.779 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.625538+0000", 2024-09-21T10:41:09.779 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6540000000000002e-06 2024-09-21T10:41:09.779 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.779 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.779 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.779 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.804826+0000", 2024-09-21T10:41:09.779 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.17928818199999999 2024-09-21T10:41:09.779 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.779 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.779 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:41:09.779 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.804835+0000", 2024-09-21T10:41:09.779 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0739999999999993e-06 2024-09-21T10:41:09.779 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.780 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:41:09.780 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.780 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.780 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.780 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:69147 53.1 53.7df06da1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1322)", 2024-09-21T10:41:09.780 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:40:59.625550+0000", 2024-09-21T10:41:09.780 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 9.1613469779999992, 2024-09-21T10:41:09.780 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.142683437, 2024-09-21T10:41:09.780 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:41:09.780 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:41:09.780 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:41:09.780 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:41:09.780 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:41:09.780 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:41:09.780 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 69147 2024-09-21T10:41:09.780 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.781 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:41:09.781 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.781 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:41:09.781 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.625550+0000", 2024-09-21T10:41:09.781 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.781 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.781 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.781 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:41:09.781 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.625550+0000", 2024-09-21T10:41:09.781 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.781 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.781 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.781 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:41:09.781 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.625550+0000", 2024-09-21T10:41:09.782 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1799999999999995e-07 2024-09-21T10:41:09.782 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.782 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.782 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:41:09.782 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.625570+0000", 2024-09-21T10:41:09.782 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9632000000000001e-05 2024-09-21T10:41:09.782 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.782 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.782 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:41:09.782 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.625577+0000", 2024-09-21T10:41:09.782 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3699999999999997e-06 2024-09-21T10:41:09.782 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.782 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.782 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:41:09.782 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.625583+0000", 2024-09-21T10:41:09.783 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7640000000000002e-06 2024-09-21T10:41:09.783 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.783 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.783 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.783 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.768227+0000", 2024-09-21T10:41:09.783 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.14264369499999999 2024-09-21T10:41:09.783 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.783 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.783 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:41:09.783 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.768233+0000", 2024-09-21T10:41:09.783 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4579999999999998e-06 2024-09-21T10:41:09.783 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.783 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:41:09.783 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.783 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.784 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.784 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:69149 53.1 53.67516ef1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1322)", 2024-09-21T10:41:09.784 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:40:59.628449+0000", 2024-09-21T10:41:09.784 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 9.1584482129999998, 2024-09-21T10:41:09.784 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13980906600000001, 2024-09-21T10:41:09.784 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:41:09.784 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:41:09.784 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:41:09.784 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:41:09.784 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:41:09.784 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:41:09.784 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 69149 2024-09-21T10:41:09.784 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.784 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:41:09.785 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.785 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:41:09.785 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.628449+0000", 2024-09-21T10:41:09.785 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.785 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.785 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.785 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:41:09.785 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.628449+0000", 2024-09-21T10:41:09.785 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.785 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.785 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.785 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:41:09.785 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.628452+0000", 2024-09-21T10:41:09.785 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.802e-06 2024-09-21T10:41:09.785 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.786 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.786 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:41:09.786 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.628488+0000", 2024-09-21T10:41:09.786 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6408000000000001e-05 2024-09-21T10:41:09.786 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.786 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.786 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:41:09.786 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.628509+0000", 2024-09-21T10:41:09.786 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1160999999999999e-05 2024-09-21T10:41:09.786 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.786 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.786 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:41:09.786 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.628523+0000", 2024-09-21T10:41:09.786 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3804e-05 2024-09-21T10:41:09.786 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.787 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.787 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.787 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.768252+0000", 2024-09-21T10:41:09.787 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13972874499999999 2024-09-21T10:41:09.787 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.787 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.787 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:41:09.787 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.768258+0000", 2024-09-21T10:41:09.787 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1460000000000001e-06 2024-09-21T10:41:09.787 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.787 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:41:09.787 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.787 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.787 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.788 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop_reply(client.45020.0:69135 53.4 e1322/1307)", 2024-09-21T10:41:09.788 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:40:59.629498+0000", 2024-09-21T10:41:09.788 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 9.1573992830000002, 2024-09-21T10:41:09.788 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.174692077, 2024-09-21T10:41:09.788 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:41:09.788 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:41:09.788 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:41:09.788 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:41:09.788 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.788 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:41:09.788 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.629498+0000", 2024-09-21T10:41:09.788 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.788 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.788 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.788 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:41:09.789 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.629498+0000", 2024-09-21T10:41:09.789 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.789 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.789 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.789 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:41:09.789 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.629500+0000", 2024-09-21T10:41:09.789 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9130000000000001e-06 2024-09-21T10:41:09.789 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.789 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.789 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:41:09.789 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.629505+0000", 2024-09-21T10:41:09.789 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4639999999999997e-06 2024-09-21T10:41:09.789 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.789 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.789 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:41:09.790 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.632657+0000", 2024-09-21T10:41:09.790 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0031514709999999999 2024-09-21T10:41:09.790 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.790 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.790 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:41:09.790 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.632666+0000", 2024-09-21T10:41:09.790 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9070000000000002e-06 2024-09-21T10:41:09.790 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.790 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.790 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.790 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.804171+0000", 2024-09-21T10:41:09.790 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.17150420299999999 2024-09-21T10:41:09.790 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.790 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.790 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:41:09.791 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.804190+0000", 2024-09-21T10:41:09.791 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9119000000000001e-05 2024-09-21T10:41:09.791 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.791 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:41:09.791 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.791 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.791 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.791 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop_reply(client.45020.0:69140 53.1 e1322/1307)", 2024-09-21T10:41:09.791 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:40:59.632719+0000", 2024-09-21T10:41:09.791 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 9.1541779969999997, 2024-09-21T10:41:09.791 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.135395761, 2024-09-21T10:41:09.791 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:41:09.791 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:41:09.791 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:41:09.792 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:41:09.792 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.792 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:41:09.792 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.632719+0000", 2024-09-21T10:41:09.792 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.792 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.792 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.792 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:41:09.792 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.632719+0000", 2024-09-21T10:41:09.792 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.792 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.792 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.792 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:41:09.792 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.632719+0000", 2024-09-21T10:41:09.792 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8299999999999998e-07 2024-09-21T10:41:09.793 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.793 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.793 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:41:09.793 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.632722+0000", 2024-09-21T10:41:09.793 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8210000000000002e-06 2024-09-21T10:41:09.793 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.793 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.793 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:41:09.793 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.632728+0000", 2024-09-21T10:41:09.793 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7610000000000004e-06 2024-09-21T10:41:09.793 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.793 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.793 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:41:09.793 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.632732+0000", 2024-09-21T10:41:09.794 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.177e-06 2024-09-21T10:41:09.794 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.794 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.794 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.794 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.768102+0000", 2024-09-21T10:41:09.794 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13537044300000001 2024-09-21T10:41:09.794 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.794 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.794 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:41:09.794 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.768115+0000", 2024-09-21T10:41:09.795 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2276000000000001e-05 2024-09-21T10:41:09.795 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.795 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:41:09.795 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.795 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.795 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.795 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.45020.0:69148 53.3 e1322/1307)", 2024-09-21T10:41:09.795 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:40:59.632757+0000", 2024-09-21T10:41:09.795 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 9.1541395029999997, 2024-09-21T10:41:09.795 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.16579619400000001, 2024-09-21T10:41:09.795 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:41:09.795 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:41:09.795 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:41:09.796 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:41:09.796 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.796 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:41:09.796 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.632757+0000", 2024-09-21T10:41:09.796 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.796 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.796 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.796 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:41:09.796 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.632757+0000", 2024-09-21T10:41:09.796 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.796 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.796 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.796 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:41:09.796 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.632758+0000", 2024-09-21T10:41:09.797 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5499999999999998e-07 2024-09-21T10:41:09.797 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.797 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.797 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:41:09.797 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.632800+0000", 2024-09-21T10:41:09.797 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2250999999999999e-05 2024-09-21T10:41:09.797 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.797 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.797 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:41:09.797 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.632807+0000", 2024-09-21T10:41:09.797 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2219999999999996e-06 2024-09-21T10:41:09.797 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.798 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.798 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:41:09.798 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.632813+0000", 2024-09-21T10:41:09.798 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5960000000000003e-06 2024-09-21T10:41:09.798 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.798 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.798 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.798 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.798536+0000", 2024-09-21T10:41:09.798 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.165722912 2024-09-21T10:41:09.798 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.798 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.798 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:41:09.798 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.798554+0000", 2024-09-21T10:41:09.798 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7757999999999998e-05 2024-09-21T10:41:09.798 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.798 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:41:09.799 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.799 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.799 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.799 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.45020.0:69142 53.5 e1322/1307)", 2024-09-21T10:41:09.799 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:40:59.632866+0000", 2024-09-21T10:41:09.799 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 9.1540310960000006, 2024-09-21T10:41:09.799 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.15432385100000001, 2024-09-21T10:41:09.799 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:41:09.799 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:41:09.799 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:41:09.799 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:41:09.799 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.799 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:41:09.799 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.632866+0000", 2024-09-21T10:41:09.800 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.800 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.800 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.800 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:41:09.800 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.632866+0000", 2024-09-21T10:41:09.800 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.800 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.800 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.800 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:41:09.800 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.632866+0000", 2024-09-21T10:41:09.800 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3499999999999996e-07 2024-09-21T10:41:09.800 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.800 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.801 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:41:09.801 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.632890+0000", 2024-09-21T10:41:09.801 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3730000000000001e-05 2024-09-21T10:41:09.801 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.801 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.801 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:41:09.801 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.632897+0000", 2024-09-21T10:41:09.801 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1890000000000001e-06 2024-09-21T10:41:09.801 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.801 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.801 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:41:09.801 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.632902+0000", 2024-09-21T10:41:09.801 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7140000000000001e-06 2024-09-21T10:41:09.801 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.801 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.801 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.802 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.787171+0000", 2024-09-21T10:41:09.802 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.15426863599999999 2024-09-21T10:41:09.802 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.802 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.802 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:41:09.802 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.787190+0000", 2024-09-21T10:41:09.802 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9046999999999999e-05 2024-09-21T10:41:09.802 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.802 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:41:09.802 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.802 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.802 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.803 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:69135 53.4 53:24e51e3f:::benchmark_data_smithi070_228967_object69134:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1323)", 2024-09-21T10:41:09.803 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:40:59.634297+0000", 2024-09-21T10:41:09.803 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 9.1526000150000009, 2024-09-21T10:41:09.803 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0216346810000001, 2024-09-21T10:41:09.803 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:41:09.803 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:41:09.803 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:41:09.803 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:41:09.803 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:41:09.803 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:41:09.803 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 69135 2024-09-21T10:41:09.803 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.803 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:41:09.803 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.803 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:41:09.804 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634297+0000", 2024-09-21T10:41:09.804 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.804 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.804 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.804 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:41:09.804 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634297+0000", 2024-09-21T10:41:09.804 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.804 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.804 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.804 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:41:09.804 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634299+0000", 2024-09-21T10:41:09.804 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1160000000000002e-06 2024-09-21T10:41:09.804 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.804 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.805 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:41:09.805 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634329+0000", 2024-09-21T10:41:09.805 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9680999999999999e-05 2024-09-21T10:41:09.805 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.805 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.805 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:41:09.805 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634343+0000", 2024-09-21T10:41:09.805 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4387e-05 2024-09-21T10:41:09.805 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.805 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.805 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:41:09.805 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634351+0000", 2024-09-21T10:41:09.805 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2709999999999992e-06 2024-09-21T10:41:09.805 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.806 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.806 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.806 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.804880+0000", 2024-09-21T10:41:09.806 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.17052815599999999 2024-09-21T10:41:09.806 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.806 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.806 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:41:09.806 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.804889+0000", 2024-09-21T10:41:09.806 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5580000000000006e-06 2024-09-21T10:41:09.806 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.806 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.806 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.806 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.655817+0000", 2024-09-21T10:41:09.807 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.85092809300000005 2024-09-21T10:41:09.807 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.807 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.807 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:41:09.807 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.655932+0000", 2024-09-21T10:41:09.807 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000114419 2024-09-21T10:41:09.807 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.807 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:41:09.807 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.807 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.807 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.807 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:69140 53.11 53:8c59eef6:::benchmark_data_smithi070_228967_object69139:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1323)", 2024-09-21T10:41:09.807 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:40:59.634375+0000", 2024-09-21T10:41:09.808 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 9.1525217899999998, 2024-09-21T10:41:09.808 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0624963780000001, 2024-09-21T10:41:09.808 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:41:09.808 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:41:09.808 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:41:09.808 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:41:09.808 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:41:09.808 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:41:09.808 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 69140 2024-09-21T10:41:09.808 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.808 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:41:09.808 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.808 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:41:09.808 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634375+0000", 2024-09-21T10:41:09.809 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.809 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.809 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.809 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:41:09.809 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634375+0000", 2024-09-21T10:41:09.809 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.809 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.809 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.809 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:41:09.809 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634376+0000", 2024-09-21T10:41:09.809 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3099999999999998e-07 2024-09-21T10:41:09.809 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.810 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.810 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:41:09.810 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634402+0000", 2024-09-21T10:41:09.810 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6662e-05 2024-09-21T10:41:09.810 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.810 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.810 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:41:09.810 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634409+0000", 2024-09-21T10:41:09.810 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5869999999999998e-06 2024-09-21T10:41:09.810 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.810 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.810 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:41:09.810 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634414+0000", 2024-09-21T10:41:09.811 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0259999999999998e-06 2024-09-21T10:41:09.811 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.811 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.811 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.811 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.769018+0000", 2024-09-21T10:41:09.811 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.134604431 2024-09-21T10:41:09.811 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.811 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.811 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:41:09.811 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.769026+0000", 2024-09-21T10:41:09.811 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1140000000000006e-06 2024-09-21T10:41:09.811 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.811 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.811 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.811 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.696793+0000", 2024-09-21T10:41:09.812 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.92776693899999996 2024-09-21T10:41:09.812 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.812 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.812 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:41:09.812 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.696872+0000", 2024-09-21T10:41:09.812 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8188000000000004e-05 2024-09-21T10:41:09.812 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.812 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:41:09.812 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.812 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.812 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.812 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:69144 53.14 53:284ece09:::benchmark_data_smithi070_228967_object69143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1323)", 2024-09-21T10:41:09.812 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:40:59.634423+0000", 2024-09-21T10:41:09.813 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 9.1524741800000005, 2024-09-21T10:41:09.813 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0310207650000001, 2024-09-21T10:41:09.813 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:41:09.813 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:41:09.813 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:41:09.813 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:41:09.813 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:41:09.813 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:41:09.813 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 69144 2024-09-21T10:41:09.813 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.813 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:41:09.813 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.813 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:41:09.813 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634423+0000", 2024-09-21T10:41:09.813 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.814 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.814 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.814 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:41:09.814 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634423+0000", 2024-09-21T10:41:09.814 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.814 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.814 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.814 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:41:09.814 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634423+0000", 2024-09-21T10:41:09.814 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7e-07 2024-09-21T10:41:09.814 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.814 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.814 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:41:09.814 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634448+0000", 2024-09-21T10:41:09.814 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5009000000000001e-05 2024-09-21T10:41:09.815 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.815 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.815 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:41:09.815 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634454+0000", 2024-09-21T10:41:09.815 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7420000000000003e-06 2024-09-21T10:41:09.815 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.815 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.815 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:41:09.815 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634459+0000", 2024-09-21T10:41:09.815 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7960000000000003e-06 2024-09-21T10:41:09.815 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.815 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.815 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.815 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.805652+0000", 2024-09-21T10:41:09.816 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.17119356299999999 2024-09-21T10:41:09.816 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.816 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.816 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:41:09.816 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.805658+0000", 2024-09-21T10:41:09.816 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.874e-06 2024-09-21T10:41:09.816 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.816 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.816 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.816 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.660900+0000", 2024-09-21T10:41:09.816 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.85524192799999998 2024-09-21T10:41:09.816 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.816 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.816 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:41:09.816 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.661040+0000", 2024-09-21T10:41:09.816 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000140433 2024-09-21T10:41:09.817 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.817 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.817 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:41:09.817 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.661231+0000", 2024-09-21T10:41:09.817 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00019054299999999999 2024-09-21T10:41:09.817 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.817 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.817 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:41:09.817 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.664833+0000", 2024-09-21T10:41:09.817 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.003602007 2024-09-21T10:41:09.817 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.817 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.817 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:09.817 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.664900+0000", 2024-09-21T10:41:09.818 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6889000000000003e-05 2024-09-21T10:41:09.818 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.818 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.818 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:09.818 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.665192+0000", 2024-09-21T10:41:09.818 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00029191800000000001 2024-09-21T10:41:09.818 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.818 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.818 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:41:09.818 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.665326+0000", 2024-09-21T10:41:09.818 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013376800000000001 2024-09-21T10:41:09.818 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.818 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.818 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:41:09.819 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.665444+0000", 2024-09-21T10:41:09.819 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011792499999999999 2024-09-21T10:41:09.819 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.819 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:41:09.819 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.819 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.819 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.819 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:69145 53.4 53:238bb0a1:::benchmark_data_smithi070_228967_object69144:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1323)", 2024-09-21T10:41:09.819 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:40:59.634466+0000", 2024-09-21T10:41:09.819 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 9.1524309820000003, 2024-09-21T10:41:09.819 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.028562446, 2024-09-21T10:41:09.819 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:41:09.819 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:41:09.819 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:41:09.820 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:41:09.820 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:41:09.820 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:41:09.820 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 69145 2024-09-21T10:41:09.820 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.820 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:41:09.820 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.820 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:41:09.820 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634466+0000", 2024-09-21T10:41:09.820 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.820 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.820 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.820 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:41:09.821 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634466+0000", 2024-09-21T10:41:09.821 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.821 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.821 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.821 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:41:09.821 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634466+0000", 2024-09-21T10:41:09.821 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8299999999999998e-07 2024-09-21T10:41:09.821 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.821 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.821 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:41:09.821 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634492+0000", 2024-09-21T10:41:09.822 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5224000000000001e-05 2024-09-21T10:41:09.822 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.822 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.822 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:41:09.822 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634497+0000", 2024-09-21T10:41:09.822 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8420000000000001e-06 2024-09-21T10:41:09.822 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.822 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.822 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:41:09.822 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634502+0000", 2024-09-21T10:41:09.822 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.4460000000000003e-06 2024-09-21T10:41:09.822 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.822 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.822 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.822 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.805453+0000", 2024-09-21T10:41:09.823 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.170950725 2024-09-21T10:41:09.823 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.823 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.823 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:41:09.823 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.805461+0000", 2024-09-21T10:41:09.823 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6859999999999995e-06 2024-09-21T10:41:09.823 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.823 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.823 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.823 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.655991+0000", 2024-09-21T10:41:09.823 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.85052968399999995 2024-09-21T10:41:09.823 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.823 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.824 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:41:09.824 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.656127+0000", 2024-09-21T10:41:09.824 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013582899999999999 2024-09-21T10:41:09.824 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.824 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.824 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:41:09.824 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.656288+0000", 2024-09-21T10:41:09.824 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016150800000000001 2024-09-21T10:41:09.824 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.824 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.824 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:41:09.824 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.658451+0000", 2024-09-21T10:41:09.824 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00216264 2024-09-21T10:41:09.824 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.824 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.824 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:09.825 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.662291+0000", 2024-09-21T10:41:09.825 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0038400700000000001 2024-09-21T10:41:09.825 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.825 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.825 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:09.825 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.662696+0000", 2024-09-21T10:41:09.825 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00040515800000000003 2024-09-21T10:41:09.825 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.825 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.825 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:41:09.825 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.662984+0000", 2024-09-21T10:41:09.825 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000288285 2024-09-21T10:41:09.825 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.825 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.826 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:41:09.826 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.663028+0000", 2024-09-21T10:41:09.826 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3965999999999999e-05 2024-09-21T10:41:09.826 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.826 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:41:09.826 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.826 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.826 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.826 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:69146 53.14 53:29afdee0:::benchmark_data_smithi070_228967_object69145:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1323)", 2024-09-21T10:41:09.826 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:40:59.634508+0000", 2024-09-21T10:41:09.826 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 9.1523891099999997, 2024-09-21T10:41:09.826 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0315896470000001, 2024-09-21T10:41:09.826 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:41:09.827 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:41:09.827 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:41:09.827 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:41:09.827 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:41:09.827 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:41:09.827 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 69146 2024-09-21T10:41:09.827 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.827 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:41:09.827 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.827 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:41:09.827 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634508+0000", 2024-09-21T10:41:09.827 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.827 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.827 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.828 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:41:09.828 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634508+0000", 2024-09-21T10:41:09.828 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.828 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.828 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.828 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:41:09.828 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634508+0000", 2024-09-21T10:41:09.828 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0499999999999999e-07 2024-09-21T10:41:09.828 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.828 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.828 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:41:09.828 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634603+0000", 2024-09-21T10:41:09.828 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5068999999999994e-05 2024-09-21T10:41:09.828 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.828 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.829 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:41:09.829 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634610+0000", 2024-09-21T10:41:09.829 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5379999999999999e-06 2024-09-21T10:41:09.829 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.829 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.829 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:41:09.829 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634614+0000", 2024-09-21T10:41:09.829 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5709999999999999e-06 2024-09-21T10:41:09.829 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.829 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.829 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.829 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.805676+0000", 2024-09-21T10:41:09.829 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.17106163899999999 2024-09-21T10:41:09.829 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.830 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.830 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:41:09.830 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.805681+0000", 2024-09-21T10:41:09.830 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7700000000000001e-06 2024-09-21T10:41:09.830 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.830 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.830 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.830 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.663062+0000", 2024-09-21T10:41:09.830 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.85738122000000005 2024-09-21T10:41:09.830 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.830 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.830 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:41:09.830 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.663180+0000", 2024-09-21T10:41:09.831 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011789 2024-09-21T10:41:09.831 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.831 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.831 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:41:09.831 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.663381+0000", 2024-09-21T10:41:09.831 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00020124699999999999 2024-09-21T10:41:09.831 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.831 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.831 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:41:09.831 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.664869+0000", 2024-09-21T10:41:09.831 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001487621 2024-09-21T10:41:09.831 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.831 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.831 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:09.832 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.665966+0000", 2024-09-21T10:41:09.832 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0010969599999999999 2024-09-21T10:41:09.832 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.832 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.832 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:09.832 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.666007+0000", 2024-09-21T10:41:09.832 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.1409000000000003e-05 2024-09-21T10:41:09.832 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.832 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.832 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:41:09.832 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.666059+0000", 2024-09-21T10:41:09.832 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1447999999999999e-05 2024-09-21T10:41:09.832 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.832 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.833 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:41:09.833 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.666097+0000", 2024-09-21T10:41:09.833 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8859999999999997e-05 2024-09-21T10:41:09.833 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.833 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:41:09.833 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.833 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.833 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.833 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:69147 53.1 53:85b60fbe:::benchmark_data_smithi070_228967_object69146:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1323)", 2024-09-21T10:41:09.833 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:40:59.634620+0000", 2024-09-21T10:41:09.833 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 9.1522765390000007, 2024-09-21T10:41:09.833 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.046164943, 2024-09-21T10:41:09.833 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:41:09.833 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:41:09.833 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:41:09.834 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:41:09.834 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:41:09.834 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:41:09.834 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 69147 2024-09-21T10:41:09.834 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.834 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:41:09.834 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.834 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:41:09.834 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634620+0000", 2024-09-21T10:41:09.834 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.834 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.834 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.834 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:41:09.835 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634620+0000", 2024-09-21T10:41:09.835 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.835 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.835 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.835 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:41:09.835 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634621+0000", 2024-09-21T10:41:09.835 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6199999999999999e-07 2024-09-21T10:41:09.835 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.835 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.835 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:41:09.835 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634644+0000", 2024-09-21T10:41:09.835 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3343e-05 2024-09-21T10:41:09.835 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.836 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.836 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:41:09.836 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634650+0000", 2024-09-21T10:41:09.836 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7579999999999998e-06 2024-09-21T10:41:09.836 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.836 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.836 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:41:09.836 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634654+0000", 2024-09-21T10:41:09.836 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2459999999999997e-06 2024-09-21T10:41:09.836 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.836 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.836 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.836 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.768290+0000", 2024-09-21T10:41:09.836 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13363550900000001 2024-09-21T10:41:09.836 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.837 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.837 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:41:09.837 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.768297+0000", 2024-09-21T10:41:09.837 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.729e-06 2024-09-21T10:41:09.837 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.837 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.837 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.837 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.648308+0000", 2024-09-21T10:41:09.837 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.88001059100000001 2024-09-21T10:41:09.837 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.837 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.837 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:41:09.837 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.648452+0000", 2024-09-21T10:41:09.837 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014430000000000001 2024-09-21T10:41:09.838 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.838 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.838 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:41:09.838 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.648595+0000", 2024-09-21T10:41:09.838 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000142366 2024-09-21T10:41:09.838 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.838 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.838 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:41:09.838 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.651430+0000", 2024-09-21T10:41:09.838 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0028349690000000001 2024-09-21T10:41:09.838 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.838 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.839 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:09.839 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.662756+0000", 2024-09-21T10:41:09.839 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.011326065999999999 2024-09-21T10:41:09.839 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.839 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.839 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:09.839 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.680668+0000", 2024-09-21T10:41:09.839 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.017912806999999999 2024-09-21T10:41:09.839 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.839 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.839 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:41:09.839 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.680739+0000", 2024-09-21T10:41:09.839 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0376999999999994e-05 2024-09-21T10:41:09.839 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.840 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.840 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:41:09.840 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.680785+0000", 2024-09-21T10:41:09.840 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6520000000000002e-05 2024-09-21T10:41:09.840 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.840 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:41:09.840 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.840 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.840 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.840 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:69149 53.11 53:8f768ae6:::benchmark_data_smithi070_228967_object69148:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1323)", 2024-09-21T10:41:09.840 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:40:59.634660+0000", 2024-09-21T10:41:09.840 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 9.1522370259999999, 2024-09-21T10:41:09.840 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0721572539999999, 2024-09-21T10:41:09.840 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:41:09.841 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:41:09.841 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:41:09.841 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:41:09.841 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:41:09.841 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:41:09.841 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 69149 2024-09-21T10:41:09.841 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.841 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:41:09.841 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.841 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:41:09.841 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634660+0000", 2024-09-21T10:41:09.841 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.841 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.842 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.842 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:41:09.842 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634660+0000", 2024-09-21T10:41:09.842 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.842 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.842 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.842 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:41:09.842 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634660+0000", 2024-09-21T10:41:09.842 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5100000000000001e-07 2024-09-21T10:41:09.842 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.842 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.842 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:41:09.842 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634684+0000", 2024-09-21T10:41:09.842 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3703999999999999e-05 2024-09-21T10:41:09.843 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.843 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.843 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:41:09.843 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634690+0000", 2024-09-21T10:41:09.843 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5300000000000004e-06 2024-09-21T10:41:09.843 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.843 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.843 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:41:09.843 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634694+0000", 2024-09-21T10:41:09.843 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.4290000000000003e-06 2024-09-21T10:41:09.843 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.843 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.843 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.844 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.769046+0000", 2024-09-21T10:41:09.844 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13435238199999999 2024-09-21T10:41:09.844 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.844 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.844 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:41:09.844 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.769052+0000", 2024-09-21T10:41:09.844 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3770000000000004e-06 2024-09-21T10:41:09.844 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.844 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.844 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.844 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.696899+0000", 2024-09-21T10:41:09.844 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.92784694000000001 2024-09-21T10:41:09.844 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.844 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.844 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:41:09.844 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.697002+0000", 2024-09-21T10:41:09.845 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010339400000000001 2024-09-21T10:41:09.845 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.845 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.845 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:41:09.845 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.697127+0000", 2024-09-21T10:41:09.845 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012478899999999999 2024-09-21T10:41:09.845 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.845 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.845 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:41:09.845 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.700680+0000", 2024-09-21T10:41:09.845 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035536389999999999 2024-09-21T10:41:09.845 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.845 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.846 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:09.846 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.704795+0000", 2024-09-21T10:41:09.846 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0041147240000000002 2024-09-21T10:41:09.846 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.846 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.846 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:09.846 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.706726+0000", 2024-09-21T10:41:09.846 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0019308319999999999 2024-09-21T10:41:09.846 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.846 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.846 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:41:09.846 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.706781+0000", 2024-09-21T10:41:09.846 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4951000000000003e-05 2024-09-21T10:41:09.846 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.847 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.847 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:41:09.847 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.706817+0000", 2024-09-21T10:41:09.847 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6211999999999998e-05 2024-09-21T10:41:09.847 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.847 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:41:09.847 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.847 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.847 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.847 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:69157 53.11 53:8bc0c75f:::benchmark_data_smithi070_228967_object69156:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1323)", 2024-09-21T10:41:09.847 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:40:59.835769+0000", 2024-09-21T10:41:09.847 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 8.9511277299999996, 2024-09-21T10:41:09.847 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.87115852400000005, 2024-09-21T10:41:09.848 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:41:09.848 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:41:09.848 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:41:09.848 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:41:09.848 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:41:09.848 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:41:09.848 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 69157 2024-09-21T10:41:09.848 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.848 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:41:09.848 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.848 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:41:09.848 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.835769+0000", 2024-09-21T10:41:09.848 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.848 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.848 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.849 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:41:09.849 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.835769+0000", 2024-09-21T10:41:09.849 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:41:09.849 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.849 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.849 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:41:09.849 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.835770+0000", 2024-09-21T10:41:09.849 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2799999999999995e-07 2024-09-21T10:41:09.849 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.849 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.849 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:41:09.849 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.835787+0000", 2024-09-21T10:41:09.849 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7019000000000001e-05 2024-09-21T10:41:09.850 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.850 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.850 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:41:09.850 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.835796+0000", 2024-09-21T10:41:09.850 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6400000000000003e-06 2024-09-21T10:41:09.850 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.850 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.850 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:41:09.850 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.835801+0000", 2024-09-21T10:41:09.850 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5219999999999998e-06 2024-09-21T10:41:09.850 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.850 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.850 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.850 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.835843+0000", 2024-09-21T10:41:09.851 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.1875000000000001e-05 2024-09-21T10:41:09.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.851 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.851 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:41:09.851 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.835849+0000", 2024-09-21T10:41:09.851 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3219999999999998e-06 2024-09-21T10:41:09.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.851 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.851 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:41:09.851 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.698269+0000", 2024-09-21T10:41:09.851 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.86242006500000001 2024-09-21T10:41:09.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.851 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.851 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:41:09.851 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.698384+0000", 2024-09-21T10:41:09.852 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011491 2024-09-21T10:41:09.852 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.852 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.852 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:41:09.852 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.698496+0000", 2024-09-21T10:41:09.852 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011205900000000001 2024-09-21T10:41:09.852 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.852 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.852 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:09.852 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.704841+0000", 2024-09-21T10:41:09.852 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0063445300000000001 2024-09-21T10:41:09.852 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.852 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.852 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:41:09.853 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.704896+0000", 2024-09-21T10:41:09.853 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5373e-05 2024-09-21T10:41:09.853 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.853 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.853 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:09.853 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.706853+0000", 2024-09-21T10:41:09.853 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0019569710000000001 2024-09-21T10:41:09.853 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.853 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.853 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:41:09.853 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.706894+0000", 2024-09-21T10:41:09.854 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0886000000000001e-05 2024-09-21T10:41:09.854 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:41:09.854 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:41:09.854 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:41:09.854 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.706928+0000", 2024-09-21T10:41:09.854 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3624000000000002e-05 2024-09-21T10:41:09.854 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.854 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:41:09.854 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.854 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:41:09.854 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:41:09.854 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:41:09.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:09.856 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:09.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:09.956 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:09.984 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:09.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:09.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:10.086 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:10.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:09 smithi160 ceph-mon[101879]: pgmap v2196: 99 pgs: 1 active+clean+scrubbing, 98 active+clean; 4.6 GiB data, 15 GiB used, 522 GiB / 536 GiB avail 2024-09-21T10:41:10.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:09 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-09-21T10:41:10.108 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:10.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:10.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:10.120 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 46 16 80208 80192 108.93 82.75 0.00170194 0.00913033 2024-09-21T10:41:10.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:09 smithi070 ceph-mon[107385]: pgmap v2196: 99 pgs: 1 active+clean+scrubbing, 98 active+clean; 4.6 GiB data, 15 GiB used, 522 GiB / 536 GiB avail 2024-09-21T10:41:10.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:09 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-09-21T10:41:10.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:09 smithi070 ceph-mon[112139]: pgmap v2196: 99 pgs: 1 active+clean+scrubbing, 98 active+clean; 4.6 GiB data, 15 GiB used, 522 GiB / 536 GiB avail 2024-09-21T10:41:10.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:09 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-09-21T10:41:10.210 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:10.231 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:10.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:10.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:10.333 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:10.355 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:10.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:10.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:10.457 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:10.479 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:10.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:10.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:10.581 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:10.602 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:10.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:10.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:10.703 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:10.724 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:10.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:10.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:10.826 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:10.847 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:10.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:10.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:10.949 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:10.970 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:10.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:10.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:11.072 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:11.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:10 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "19"}]': finished 2024-09-21T10:41:11.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:10 smithi160 ceph-mon[101879]: osdmap e1329: 8 total, 8 up, 6 in 2024-09-21T10:41:11.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:10 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-09-21T10:41:11.093 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:11.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:11.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:11.121 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 47 16 80214 80198 106.621 0.375 0.00201184 0.00912981 2024-09-21T10:41:11.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:10 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "19"}]': finished 2024-09-21T10:41:11.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:10 smithi070 ceph-mon[107385]: osdmap e1329: 8 total, 8 up, 6 in 2024-09-21T10:41:11.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:10 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-09-21T10:41:11.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:10 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "19"}]': finished 2024-09-21T10:41:11.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:10 smithi070 ceph-mon[112139]: osdmap e1329: 8 total, 8 up, 6 in 2024-09-21T10:41:11.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:10 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-09-21T10:41:11.195 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:11.216 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:11.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:11.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:11.318 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:11.338 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:11.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:11.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:11.440 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:11.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:11.461 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:11.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:11.562 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:11.583 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:11.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:11.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:11.684 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:11.707 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:11.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:11.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:11.809 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:11.829 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:11.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:11.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:11.931 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:11.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:11.954 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:11.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:12.055 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:12.084 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:12.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:12.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:12.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:11 smithi160 ceph-mon[101879]: pgmap v2198: 99 pgs: 1 active+recovering+undersized+remapped, 98 active+clean; 4.8 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 2224/235560 objects misplaced (0.944%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:41:12.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:11 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "20"}]': finished 2024-09-21T10:41:12.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:11 smithi160 ceph-mon[101879]: osdmap e1330: 8 total, 8 up, 6 in 2024-09-21T10:41:12.121 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 48 16 80392 80376 104.632 11.125 0.00837837 0.00954833 2024-09-21T10:41:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:11 smithi070 ceph-mon[107385]: pgmap v2198: 99 pgs: 1 active+recovering+undersized+remapped, 98 active+clean; 4.8 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 2224/235560 objects misplaced (0.944%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:41:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:11 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "20"}]': finished 2024-09-21T10:41:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:11 smithi070 ceph-mon[107385]: osdmap e1330: 8 total, 8 up, 6 in 2024-09-21T10:41:12.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:11 smithi070 ceph-mon[112139]: pgmap v2198: 99 pgs: 1 active+recovering+undersized+remapped, 98 active+clean; 4.8 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 2224/235560 objects misplaced (0.944%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:41:12.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:11 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "20"}]': finished 2024-09-21T10:41:12.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:11 smithi070 ceph-mon[112139]: osdmap e1330: 8 total, 8 up, 6 in 2024-09-21T10:41:12.186 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:12.206 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:12.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:12.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:12.308 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:12.334 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:12.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:12.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:12.436 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:12.468 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:12.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:12.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:12.570 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:12.600 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:12.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:12.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:12.701 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:12.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:12.723 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:12.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:12.824 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:12.847 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:12.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:12.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:12.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:12 smithi070 ceph-mon[107385]: osdmap e1331: 8 total, 8 up, 6 in 2024-09-21T10:41:12.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:12 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T10:41:12.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:12 smithi070 ceph-mon[112139]: osdmap e1331: 8 total, 8 up, 6 in 2024-09-21T10:41:12.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:12 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T10:41:12.949 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:12.978 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:12.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:12.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:13.080 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:13.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:12 smithi160 ceph-mon[101879]: osdmap e1331: 8 total, 8 up, 6 in 2024-09-21T10:41:13.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:12 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T10:41:13.116 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:13.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:13.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:13.121 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 49 16 81160 81144 103.476 48 0.00961448 0.00964952 2024-09-21T10:41:13.218 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:13.251 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:13.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:13.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:13.352 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:13.374 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:13.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:13.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:13.475 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:13.496 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:13.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:13.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:13.597 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:13.618 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:13.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:13.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:41:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:41:13] "GET /metrics HTTP/1.1" 200 36610 "" "Prometheus/2.51.0" 2024-09-21T10:41:13.720 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:13.741 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:13.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:13.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:13.843 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:13.864 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:13.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:13.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:13.965 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:13.986 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:13.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:13.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:14.088 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:14.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:13 smithi160 ceph-mon[101879]: pgmap v2201: 99 pgs: 1 active+recovering+undersized+remapped, 98 active+clean; 4.9 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 2224/238584 objects misplaced (0.932%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:41:14.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:13 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T10:41:14.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:13 smithi160 ceph-mon[101879]: osdmap e1332: 8 total, 8 up, 6 in 2024-09-21T10:41:14.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:13 smithi160 ceph-mon[101879]: osdmap e1333: 8 total, 8 up, 6 in 2024-09-21T10:41:14.109 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:14.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:14.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:14.122 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 50 16 81468 81452 101.789 19.25 0.002314 0.00963387 2024-09-21T10:41:14.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:13 smithi070 ceph-mon[107385]: pgmap v2201: 99 pgs: 1 active+recovering+undersized+remapped, 98 active+clean; 4.9 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 2224/238584 objects misplaced (0.932%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:41:14.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:13 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T10:41:14.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:13 smithi070 ceph-mon[107385]: osdmap e1332: 8 total, 8 up, 6 in 2024-09-21T10:41:14.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:13 smithi070 ceph-mon[107385]: osdmap e1333: 8 total, 8 up, 6 in 2024-09-21T10:41:14.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:13 smithi070 ceph-mon[112139]: pgmap v2201: 99 pgs: 1 active+recovering+undersized+remapped, 98 active+clean; 4.9 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 2224/238584 objects misplaced (0.932%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:41:14.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:13 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T10:41:14.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:13 smithi070 ceph-mon[112139]: osdmap e1332: 8 total, 8 up, 6 in 2024-09-21T10:41:14.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:13 smithi070 ceph-mon[112139]: osdmap e1333: 8 total, 8 up, 6 in 2024-09-21T10:41:14.211 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:14.246 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:14.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:14.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:14.348 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:14.386 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:14.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:14.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:14.488 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:14.523 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:14.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:14.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:14.624 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:14.651 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:14.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:14.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:14.752 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:14.786 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:14.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:14.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:14.888 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:14.921 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:14.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:14.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:15.023 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:15.054 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:15.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:15.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:15.123 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 51 16 82733 82717 101.343 79.0625 0.00692923 0.00986317 2024-09-21T10:41:15.156 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:15.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:15.193 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:15.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:15.293 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:15.329 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:15.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:15.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:15.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:15 smithi070 ceph-mon[107385]: osdmap e1334: 8 total, 8 up, 6 in 2024-09-21T10:41:15.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:15 smithi070 ceph-mon[112139]: osdmap e1334: 8 total, 8 up, 6 in 2024-09-21T10:41:15.431 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:15.468 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:15.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:15.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:15.570 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:15.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:15 smithi160 ceph-mon[101879]: osdmap e1334: 8 total, 8 up, 6 in 2024-09-21T10:41:15.614 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:15.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:15.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:15.716 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:15.760 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:15.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:15.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:15.861 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:15.888 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:15.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:15.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:15.990 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:16.029 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:16.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:16.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:16.123 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 52 16 84173 84157 101.125 90 0.0166051 0.00988358 2024-09-21T10:41:16.131 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:16.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:16.171 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:16.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:16.272 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:16.307 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:16.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:16.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:16.332 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:41:16.408 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:16.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:16.438 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:16.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:16.539 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:16 smithi160 ceph-mon[101879]: pgmap v2205: 99 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.0 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 4.2 MiB/s wr, 67 op/s; 9980/243585 objects degraded (4.097%); 6425/243585 objects misplaced (2.638%); 12 MiB/s, 198 objects/s recovering 2024-09-21T10:41:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:16 smithi160 ceph-mon[101879]: Health check failed: Degraded data redundancy: 9980/243585 objects degraded (4.097%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:41:16.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:16.592 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:16.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:16.606 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:16 smithi070 ceph-mon[112139]: pgmap v2205: 99 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.0 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 4.2 MiB/s wr, 67 op/s; 9980/243585 objects degraded (4.097%); 6425/243585 objects misplaced (2.638%); 12 MiB/s, 198 objects/s recovering 2024-09-21T10:41:16.606 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:16 smithi070 ceph-mon[112139]: Health check failed: Degraded data redundancy: 9980/243585 objects degraded (4.097%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:41:16.606 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:16 smithi070 ceph-mon[107385]: pgmap v2205: 99 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.0 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 4.2 MiB/s wr, 67 op/s; 9980/243585 objects degraded (4.097%); 6425/243585 objects misplaced (2.638%); 12 MiB/s, 198 objects/s recovering 2024-09-21T10:41:16.606 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:16 smithi070 ceph-mon[107385]: Health check failed: Degraded data redundancy: 9980/243585 objects degraded (4.097%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:41:16.693 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:16.726 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:16.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:16.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:16.827 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:16.872 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:16.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:16.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:16.917 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:41:16.917 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:41:16.954 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:41:16.954 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:41:16.974 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:16.987 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:41:16.988 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:41:17.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:17.022 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:17.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:17.025 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:41:17.025 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:41:17.060 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:41:17.061 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:41:17.093 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:41:17.094 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:41:17.123 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 53 16 85566 85550 100.859 87.0625 0.0175969 0.00991014 2024-09-21T10:41:17.123 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:17.131 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:41:17.131 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:41:17.155 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:17.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:17.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:17.166 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:41:17.166 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:41:17.183 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:17 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3646729038' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:41:17.183 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:17 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3646729038' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:41:17.257 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:17.290 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:17.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:17.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:17.392 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:17.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:17.442 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:17.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:17.542 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:17.575 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:41:17.580 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:17.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:17.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:17.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:17 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3646729038' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:41:17.682 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:17.709 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:17.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:17.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:17.810 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:17.841 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:17.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:17.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:17.943 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:17.983 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:17.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:17.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:18.085 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:18.114 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:18.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:18.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:18.123 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 54 16 86863 86847 100.493 81.0625 0.00315934 0.009944 2024-09-21T10:41:18.216 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:18.248 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:18.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:18.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:18.350 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:18.376 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:18.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:18.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:18.477 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:18.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:18.516 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:18.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:18 smithi160 ceph-mon[101879]: pgmap v2206: 99 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 10 MiB/s wr, 164 op/s; 9980/248262 objects degraded (4.020%); 6425/248262 objects misplaced (2.588%); 9.9 MiB/s, 158 objects/s recovering 2024-09-21T10:41:18.617 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:18 smithi070 ceph-mon[107385]: pgmap v2206: 99 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 10 MiB/s wr, 164 op/s; 9980/248262 objects degraded (4.020%); 6425/248262 objects misplaced (2.588%); 9.9 MiB/s, 158 objects/s recovering 2024-09-21T10:41:18.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:18 smithi070 ceph-mon[112139]: pgmap v2206: 99 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 10 MiB/s wr, 164 op/s; 9980/248262 objects degraded (4.020%); 6425/248262 objects misplaced (2.588%); 9.9 MiB/s, 158 objects/s recovering 2024-09-21T10:41:18.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:18.657 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:18.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:18.757 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:18.789 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:18.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:18.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:18.890 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:18.932 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:18.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:18.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:19.034 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:19.069 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:19.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:19.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:19.123 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 55 16 88256 88240 100.248 87.0625 0.00605201 0.00997042 2024-09-21T10:41:19.171 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:19.202 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:19.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:19.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:19.303 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:19.353 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:19.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:19.378 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:19.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:19.383 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:19.479 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:19.511 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:19.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:19.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:19.613 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:19.644 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:19.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:19.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:19.745 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:19.778 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:19.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:19.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:19.880 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:19.913 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:19.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:19.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:20.015 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:20.055 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:20.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:20.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:20.123 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 56 16 89642 89626 100.005 86.625 0.00357886 0.00999469 2024-09-21T10:41:20.156 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:20.185 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:20.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:20.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:20.225 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:20 smithi070 ceph-mon[107385]: pgmap v2207: 99 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 8.5 MiB/s wr, 136 op/s; 9980/248262 objects degraded (4.020%); 6425/248262 objects misplaced (2.588%); 8.2 MiB/s, 131 objects/s recovering 2024-09-21T10:41:20.226 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:20 smithi070 ceph-mon[112139]: pgmap v2207: 99 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 8.5 MiB/s wr, 136 op/s; 9980/248262 objects degraded (4.020%); 6425/248262 objects misplaced (2.588%); 8.2 MiB/s, 131 objects/s recovering 2024-09-21T10:41:20.287 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:20.324 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:20.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:20.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:20.425 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:20.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:20.466 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:20.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:20.567 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:20.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:20 smithi160 ceph-mon[101879]: pgmap v2207: 99 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 8.5 MiB/s wr, 136 op/s; 9980/248262 objects degraded (4.020%); 6425/248262 objects misplaced (2.588%); 8.2 MiB/s, 131 objects/s recovering 2024-09-21T10:41:20.603 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:20.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:20.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:20.705 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:20.740 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:20.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:20.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:20.842 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:20.879 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:20.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:20.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:20.981 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:21.035 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:21.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:21.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:21.123 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 57 16 91087 91071 99.8346 90.3125 0.00983367 0.0100117 2024-09-21T10:41:21.138 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:21.171 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:21.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:21.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:21.273 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:21.303 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:21.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:21.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:21.323 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:41:21.405 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:21.436 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:21.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:21.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:21.440 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:41:21.440 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:41:21.442 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:41:21.442 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:41:21.539 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:21.571 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:21.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:21.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:21.673 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:21.717 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:21.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:21.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:21.819 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:41:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:41:21.855 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:21.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:21.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:21.957 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:21.966 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:21.988 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:21.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:21.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:22.090 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:22.123 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:22.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:22.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:22.125 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 58 16 92504 92488 99.6401 88.5625 0.0123195 0.0100317 2024-09-21T10:41:22.225 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:22.260 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:22.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:22.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:22.362 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:22.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:22.391 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:22.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:22.492 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:22.541 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:22.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:22.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:22 smithi070 ceph-mon[107385]: pgmap v2208: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.3 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 38 MiB/s wr, 606 op/s; 12500/260304 objects degraded (4.802%); 5948/260304 objects misplaced (2.285%); 9.9 MiB/s, 158 objects/s recovering 2024-09-21T10:41:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:22 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 12500/260304 objects degraded (4.802%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:41:22.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:22 smithi070 ceph-mon[112139]: pgmap v2208: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.3 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 38 MiB/s wr, 606 op/s; 12500/260304 objects degraded (4.802%); 5948/260304 objects misplaced (2.285%); 9.9 MiB/s, 158 objects/s recovering 2024-09-21T10:41:22.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:22 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 12500/260304 objects degraded (4.802%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:41:22.643 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:22.675 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:22.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:22.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:22.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:22 smithi160 ceph-mon[101879]: pgmap v2208: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.3 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 38 MiB/s wr, 606 op/s; 12500/260304 objects degraded (4.802%); 5948/260304 objects misplaced (2.285%); 9.9 MiB/s, 158 objects/s recovering 2024-09-21T10:41:22.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:22 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 12500/260304 objects degraded (4.802%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:41:22.776 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:22.810 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:22.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:22.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:22.911 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:22.947 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:22.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:22.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:23.049 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:23.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:23.088 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:23.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:23.123 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 59 16 93895 93879 99.4247 86.9375 0.00546272 0.0100532 2024-09-21T10:41:23.189 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:23.224 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:23.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:23.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:23.326 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:23.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:23.361 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:23.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:23.462 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:23.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:23.497 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:23.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:23.598 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:23.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:41:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:41:23] "GET /metrics HTTP/1.1" 200 36625 "" "Prometheus/2.51.0" 2024-09-21T10:41:23.639 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:23.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:23.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:23.741 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:23.778 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:23.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:23.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:23.879 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:23.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:23.920 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:23.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:24.021 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:24.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:24.051 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:24.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:24.123 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:2024-09-21T10:41:24.122921+0000 min lat: 0.00114886 max lat: 2.28954 avg lat: 0.0100763 2024-09-21T10:41:24.124 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:41:24.124 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 60 16 95267 95251 99.1966 85.75 0.0142542 0.0100763 2024-09-21T10:41:24.151 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:24.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:24.209 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:24.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:24.310 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:24.341 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:24.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:24.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:24.443 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:24.490 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:24.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:24.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:24.529 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:24 smithi070 ceph-mon[107385]: pgmap v2209: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 49 MiB/s wr, 791 op/s; 12500/268497 objects degraded (4.656%); 5948/268497 objects misplaced (2.215%); 8.3 MiB/s, 132 objects/s recovering 2024-09-21T10:41:24.529 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:24 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:41:24.529 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:24 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:41:24.529 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:24 smithi070 ceph-mon[112139]: pgmap v2209: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 49 MiB/s wr, 791 op/s; 12500/268497 objects degraded (4.656%); 5948/268497 objects misplaced (2.215%); 8.3 MiB/s, 132 objects/s recovering 2024-09-21T10:41:24.530 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:24 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:41:24.530 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:24 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:41:24.591 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:24.639 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:24 smithi160 ceph-mon[101879]: pgmap v2209: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 49 MiB/s wr, 791 op/s; 12500/268497 objects degraded (4.656%); 5948/268497 objects misplaced (2.215%); 8.3 MiB/s, 132 objects/s recovering 2024-09-21T10:41:24.639 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:24 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:41:24.639 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:24 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:41:24.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:24.657 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:24.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:24.758 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:24.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:24.819 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:24.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:24.919 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:24.947 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:24.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:24.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:25.049 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:25.101 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:25.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:25.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:25.124 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 61 16 96262 96246 98.5898 62.1875 0.0245057 0.0101374 2024-09-21T10:41:25.202 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:25.245 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:25.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:25.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:25.255 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:41:25.256 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T10:41:25.347 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:25.384 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:25.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:25.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:25.486 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:25.515 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:25.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:25.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:25.616 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:25.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:25 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:41:25.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:25 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:41:25.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:25 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:41:25.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:25 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:41:25.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:25 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:41:25.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:25 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:41:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:25.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:25.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:25.758 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:25.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:25.812 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:25.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:25 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:41:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:25 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:41:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:25 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:41:25.913 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:25.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:25.958 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:25.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:26.059 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:26.087 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:26.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:26.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:26.124 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 62 16 97402 97386 98.1488 71.25 0.00533103 0.0101832 2024-09-21T10:41:26.188 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:26.238 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:26.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:26.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:26.340 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:26.376 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:26.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:26.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:26.388 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:26.407 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T10:41:26.428 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:26 smithi070 ceph-mon[107385]: pgmap v2210: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.7 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 65 MiB/s wr, 1.04k op/s; 12500/281142 objects degraded (4.446%); 5339/281142 objects misplaced (1.899%); 11 MiB/s, 177 objects/s recovering 2024-09-21T10:41:26.428 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:26 smithi070 ceph-mon[107385]: from='client.45080 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:41:26.428 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:26 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/593363409' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:41:26.428 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:26 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:41:26.428 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:26 smithi070 ceph-mon[112139]: pgmap v2210: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.7 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 65 MiB/s wr, 1.04k op/s; 12500/281142 objects degraded (4.446%); 5339/281142 objects misplaced (1.899%); 11 MiB/s, 177 objects/s recovering 2024-09-21T10:41:26.429 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:26 smithi070 ceph-mon[112139]: from='client.45080 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:41:26.429 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:26 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/593363409' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:41:26.429 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:26 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:41:26.477 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:26.510 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:26.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:26.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:26.611 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:26.652 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:26.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:26.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:26.754 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:26.792 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:26.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:26.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:26 smithi160 ceph-mon[101879]: pgmap v2210: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.7 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 65 MiB/s wr, 1.04k op/s; 12500/281142 objects degraded (4.446%); 5339/281142 objects misplaced (1.899%); 11 MiB/s, 177 objects/s recovering 2024-09-21T10:41:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:26 smithi160 ceph-mon[101879]: from='client.45080 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:41:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:26 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/593363409' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:41:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:26 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:41:26.865 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":2210,"stamp":"2024-09-21T10:41:24.672507+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6140467173,"num_objects":93714,"num_object_clones":0,"num_object_copies":281142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12500,"num_objects_misplaced":5339,"num_objects_unfound":0,"num_objects_dirty":93714,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":65899,"num_write_kb":4202065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3323,"num_bytes_recovered":218570290,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136049,"ondisk_log_size":136049,"up":297,"acting":293,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":305,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":19065528,"kb_used_data":18019976,"kb_used_omap":218,"kb_used_meta":1045285,"kb_avail":543405384,"statfs":{"total":575970213888,"available":556447113216,"internally_reserved":0,"allocated":18452455424,"data_stored":18428621621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":223633,"internal_metadata":1070372463},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":879329280,"num_objects":13417,"num_object_clones":0,"num_object_copies":40251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7460,"num_objects_misplaced":595,"num_objects_unfound":0,"num_objects_dirty":13417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11011,"num_write_kb":704736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1875,"num_bytes_recovered":122880000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.553596"},"pg_stats":[{"pgid":"53.19","version":"1334'5211","reported_seq":5290,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:24.464930+0000","last_change":"2024-09-21T10:41:00.685085+0000","last_active":"2024-09-21T10:41:24.464930+0000","last_peered":"2024-09-21T10:41:24.464930+0000","last_clean":"2024-09-21T10:41:24.464930+0000","last_became_active":"2024-09-21T10:41:00.679759+0000","last_became_peered":"2024-09-21T10:41:00.679759+0000","last_unstale":"2024-09-21T10:41:24.464930+0000","last_undegraded":"2024-09-21T10:41:24.464930+0000","last_fullsized":"2024-09-21T10:41:24.464930+0000","mapping_epoch":1323,"log_start":"0'0","ondisk_log_start":"0'0","created":1323,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":5211,"log_dups_size":0,"ondisk_log_size":5211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198246400,"num_objects":3025,"num_object_clones":0,"num_object_copies":9075,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3025,"num_write_kb":193600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.18","version":"1334'4957","reported_seq":5003,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:20.935303+0000","last_change":"2024-09-21T10:41:00.683072+0000","last_active":"2024-09-21T10:41:20.935303+0000","last_peered":"2024-09-21T10:41:20.935303+0000","last_clean":"2024-09-21T10:41:20.935303+0000","last_became_active":"2024-09-21T10:41:00.681104+0000","last_became_peered":"2024-09-21T10:41:00.681104+0000","last_unstale":"2024-09-21T10:41:20.935303+0000","last_undegraded":"2024-09-21T10:41:20.935303+0000","last_fullsized":"2024-09-21T10:41:20.935303+0000","mapping_epoch":1323,"log_start":"0'0","ondisk_log_start":"0'0","created":1323,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":4957,"log_dups_size":0,"ondisk_log_size":4957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185565184,"num_objects":2832,"num_object_clones":0,"num_object_copies":8496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2832,"num_write_kb":181216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"53.17","version":"1334'5165","reported_seq":5244,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:23.786416+0000","last_change":"2024-09-21T10:41:00.699239+0000","last_active":"2024-09-21T10:41:23.786416+0000","last_peered":"2024-09-21T10:41:23.786416+0000","last_clean":"2024-09-21T10:41:23.786416+0000","last_became_active":"2024-09-21T10:41:00.695046+0000","last_became_peered":"2024-09-21T10:41:00.695046+0000","last_unstale":"2024-09-21T10:41:23.786416+0000","last_undegraded":"2024-09-21T10:41:23.786416+0000","last_fullsized":"2024-09-21T10:41:23.786416+0000","mapping_epoch":1323,"log_start":"0'0","ondisk_log_start":"0'0","created":1323,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":5165,"log_dups_size":0,"ondisk_log_size":5165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196083712,"num_objects":2992,"num_object_clones":0,"num_object_copies":8976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2992,"num_write_kb":191488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.16","version":"1334'5155","reported_seq":5234,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:24.312607+0000","last_change":"2024-09-21T10:41:00.664226+0000","last_active":"2024-09-21T10:41:24.312607+0000","last_peered":"2024-09-21T10:41:24.312607+0000","last_clean":"2024-09-21T10:41:24.312607+0000","last_became_active":"2024-09-21T10:41:00.662248+0000","last_became_peered":"2024-09-21T10:41:00.662248+0000","last_unstale":"2024-09-21T10:41:24.312607+0000","last_undegraded":"2024-09-21T10:41:24.312607+0000","last_fullsized":"2024-09-21T10:41:24.312607+0000","mapping_epoch":1323,"log_start":"0'0","ondisk_log_start":"0'0","created":1323,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":5155,"log_dups_size":0,"ondisk_log_size":5155,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195690496,"num_objects":2986,"num_object_clones":0,"num_object_copies":8958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2986,"num_write_kb":191104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"53.15","version":"1334'5120","reported_seq":5230,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:24.505627+0000","last_change":"2024-09-21T10:41:07.032615+0000","last_active":"2024-09-21T10:41:24.505627+0000","last_peered":"2024-09-21T10:41:24.505627+0000","last_clean":"2024-09-21T10:41:24.505627+0000","last_became_active":"2024-09-21T10:41:00.700283+0000","last_became_peered":"2024-09-21T10:41:00.700283+0000","last_unstale":"2024-09-21T10:41:24.505627+0000","last_undegraded":"2024-09-21T10:41:24.505627+0000","last_fullsized":"2024-09-21T10:41:24.505627+0000","mapping_epoch":1323,"log_start":"0'0","ondisk_log_start":"0'0","created":1323,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":5,"last_scrub":"1325'4554","last_scrub_stamp":"2024-09-21T10:41:07.032531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:41:07.032531+0000","objects_scrubbed":2432,"log_size":5120,"log_dups_size":0,"ondisk_log_size":5120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:36:54.404966+0000","scrub_duration":237,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196476928,"num_objects":2998,"num_object_clones":0,"num_object_copies":8994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":566,"num_write_kb":36224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.14","version":"1334'5165","reported_seq":5276,"reported_epoch":1334,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:41:23.787290+0000","last_change":"2024-09-21T10:41:14.189185+0000","last_active":"2024-09-21T10:41:23.787290+0000","last_peered":"2024-09-21T10:41:23.787290+0000","last_clean":"2024-09-21T10:41:12.657195+0000","last_became_active":"2024-09-21T10:41:14.185480+0000","last_became_peered":"2024-09-21T10:41:14.185480+0000","last_unstale":"2024-09-21T10:41:23.787290+0000","last_undegraded":"2024-09-21T10:41:14.133427+0000","last_fullsized":"2024-09-21T10:41:14.121595+0000","mapping_epoch":1333,"log_start":"0'0","ondisk_log_start":"0'0","created":1323,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":5,"last_scrub":"1324'4633","last_scrub_stamp":"2024-09-21T10:41:05.405418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:41:05.405418+0000","objects_scrubbed":2382,"log_size":5165,"log_dups_size":0,"ondisk_log_size":5165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192217088,"num_objects":2933,"num_object_clones":0,"num_object_copies":8799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":551,"num_write_kb":35264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[1,4],"avail_no_missing":["1","4","6"],"object_location_counts":[{"shards":"1,4","objects":413},{"shards":"1,4,6","objects":2520}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.13","version":"1334'5282","reported_seq":5399,"reported_epoch":1334,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:41:23.754521+0000","last_change":"2024-09-21T10:41:12.819672+0000","last_active":"2024-09-21T10:41:23.754521+0000","last_peered":"2024-09-21T10:41:23.754521+0000","last_clean":"2024-09-21T10:41:10.039904+0000","last_became_active":"2024-09-21T10:41:12.815959+0000","last_became_peered":"2024-09-21T10:41:12.815959+0000","last_unstale":"2024-09-21T10:41:23.754521+0000","last_undegraded":"2024-09-21T10:41:12.813242+0000","last_fullsized":"2024-09-21T10:41:12.728072+0000","mapping_epoch":1331,"log_start":"0'0","ondisk_log_start":"0'0","created":1323,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":5,"last_scrub":"1324'4675","last_scrub_stamp":"2024-09-21T10:41:04.559688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:41:04.559688+0000","objects_scrubbed":2364,"log_size":5282,"log_dups_size":0,"ondisk_log_size":5282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":0.55553660900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195100672,"num_objects":2977,"num_object_clones":0,"num_object_copies":8931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5040,"num_objects_misplaced":2520,"num_objects_unfound":0,"num_objects_dirty":2977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":613,"num_write_kb":39232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[1,0],"avail_no_missing":["1","4","6"],"object_location_counts":[{"shards":"0,1","objects":457},{"shards":"1,4,6","objects":2520}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.12","version":"1334'5068","reported_seq":5183,"reported_epoch":1334,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:41:24.519797+0000","last_change":"2024-09-21T10:41:11.942354+0000","last_active":"2024-09-21T10:41:24.519797+0000","last_peered":"2024-09-21T10:41:24.519797+0000","last_clean":"2024-09-21T10:41:09.653271+0000","last_became_active":"2024-09-21T10:41:11.937366+0000","last_became_peered":"2024-09-21T10:41:11.937366+0000","last_unstale":"2024-09-21T10:41:24.519797+0000","last_undegraded":"2024-09-21T10:41:11.934586+0000","last_fullsized":"2024-09-21T10:41:11.697775+0000","mapping_epoch":1330,"log_start":"0'0","ondisk_log_start":"0'0","created":1323,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":5,"last_scrub":"1325'4505","last_scrub_stamp":"2024-09-21T10:41:06.325848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:41:06.325848+0000","objects_scrubbed":2406,"log_size":5068,"log_dups_size":0,"ondisk_log_size":5068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":0.23505495100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194576384,"num_objects":2969,"num_object_clones":0,"num_object_copies":8907,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4940,"num_objects_misplaced":2470,"num_objects_unfound":0,"num_objects_dirty":2969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":563,"num_write_kb":36032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[0,1],"avail_no_missing":["0","4","5"],"object_location_counts":[{"shards":"0,1","objects":499},{"shards":"0,4,5","objects":2470}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.11","version":"1334'5185","reported_seq":5267,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:24.493622+0000","last_change":"2024-09-21T10:41:08.675260+0000","last_active":"2024-09-21T10:41:24.493622+0000","last_peered":"2024-09-21T10:41:24.493622+0000","last_clean":"2024-09-21T10:41:24.493622+0000","last_became_active":"2024-09-21T10:41:08.669075+0000","last_became_peered":"2024-09-21T10:41:08.669075+0000","last_unstale":"2024-09-21T10:41:24.493622+0000","last_undegraded":"2024-09-21T10:41:24.493622+0000","last_fullsized":"2024-09-21T10:41:24.493622+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1323,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":5185,"log_dups_size":0,"ondisk_log_size":5185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197296128,"num_objects":3011,"num_object_clones":0,"num_object_copies":9033,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3011,"num_write_kb":192672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.10","version":"1334'5155","reported_seq":9340,"reported_epoch":1334,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-21T10:41:24.552109+0000","last_change":"2024-09-21T10:41:07.804137+0000","last_active":"2024-09-21T10:41:24.552109+0000","last_peered":"2024-09-21T10:41:24.552109+0000","last_clean":"2024-09-21T10:41:05.612849+0000","last_became_active":"2024-09-21T10:41:07.792427+0000","last_became_peered":"2024-09-21T10:41:07.792427+0000","last_unstale":"2024-09-21T10:41:24.552109+0000","last_undegraded":"2024-09-21T10:41:24.552109+0000","last_fullsized":"2024-09-21T10:41:07.662007+0000","mapping_epoch":1326,"log_start":"0'0","ondisk_log_start":"0'0","created":1323,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":5155,"log_dups_size":0,"ondisk_log_size":5155,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195854336,"num_objects":2989,"num_object_clones":0,"num_object_copies":8967,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":349,"num_objects_unfound":0,"num_objects_dirty":2989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2989,"num_write_kb":191264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2063,"num_bytes_recovered":135200768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5],"avail_no_missing":["0","4","5"],"object_location_counts":[{"shards":"0,4,5","objects":349},{"shards":"0,5","objects":2640}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.f","version":"1334'5557","reported_seq":5593,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:21.093954+0000","last_change":"2024-09-21T10:40:16.447365+0000","last_active":"2024-09-21T10:41:21.093954+0000","last_peered":"2024-09-21T10:41:21.093954+0000","last_clean":"2024-09-21T10:41:21.093954+0000","last_became_active":"2024-09-21T10:40:16.447155+0000","last_became_peered":"2024-09-21T10:40:16.447155+0000","last_unstale":"2024-09-21T10:41:21.093954+0000","last_undegraded":"2024-09-21T10:41:21.093954+0000","last_fullsized":"2024-09-21T10:41:21.093954+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":5557,"log_dups_size":0,"ondisk_log_size":5557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:39:36.972987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364183552,"num_objects":5557,"num_object_clones":0,"num_object_copies":16671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5557,"num_write_kb":355648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.e","version":"1334'5591","reported_seq":5627,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:21.115034+0000","last_change":"2024-09-21T10:40:16.450893+0000","last_active":"2024-09-21T10:41:21.115034+0000","last_peered":"2024-09-21T10:41:21.115034+0000","last_clean":"2024-09-21T10:41:21.115034+0000","last_became_active":"2024-09-21T10:40:16.450584+0000","last_became_peered":"2024-09-21T10:40:16.450584+0000","last_unstale":"2024-09-21T10:41:21.115034+0000","last_undegraded":"2024-09-21T10:41:21.115034+0000","last_fullsized":"2024-09-21T10:41:21.115034+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":5591,"log_dups_size":0,"ondisk_log_size":5591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:38:24.901001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366411776,"num_objects":5591,"num_object_clones":0,"num_object_copies":16773,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5591,"num_write_kb":357824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.d","version":"1334'5826","reported_seq":5898,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:23.799147+0000","last_change":"2024-09-21T10:40:16.449312+0000","last_active":"2024-09-21T10:41:23.799147+0000","last_peered":"2024-09-21T10:41:23.799147+0000","last_clean":"2024-09-21T10:41:23.799147+0000","last_became_active":"2024-09-21T10:40:16.448919+0000","last_became_peered":"2024-09-21T10:40:16.448919+0000","last_unstale":"2024-09-21T10:41:23.799147+0000","last_undegraded":"2024-09-21T10:41:23.799147+0000","last_fullsized":"2024-09-21T10:41:23.799147+0000","mapping_epoch":1307,"log_start":"1314'1114","ondisk_log_start":"1314'1114","created":1307,"last_epoch_clean":1308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":4712,"log_dups_size":0,"ondisk_log_size":4712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:30.319509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381812736,"num_objects":5826,"num_object_clones":0,"num_object_copies":17478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5826,"num_write_kb":372864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.c","version":"1334'5807","reported_seq":8331,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:20.979129+0000","last_change":"2024-09-21T10:40:42.390568+0000","last_active":"2024-09-21T10:41:20.979129+0000","last_peered":"2024-09-21T10:41:20.979129+0000","last_clean":"2024-09-21T10:41:20.979129+0000","last_became_active":"2024-09-21T10:40:42.388608+0000","last_became_peered":"2024-09-21T10:40:42.388608+0000","last_unstale":"2024-09-21T10:41:20.979129+0000","last_undegraded":"2024-09-21T10:41:20.979129+0000","last_fullsized":"2024-09-21T10:41:20.979129+0000","mapping_epoch":1317,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":5807,"log_dups_size":0,"ondisk_log_size":5807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:06:04.278392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380567552,"num_objects":5807,"num_object_clones":0,"num_object_copies":17421,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5807,"num_write_kb":371648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1235,"num_bytes_recovered":80936960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"53.b","version":"1334'5633","reported_seq":5669,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:20.985813+0000","last_change":"2024-09-21T10:40:16.445075+0000","last_active":"2024-09-21T10:41:20.985813+0000","last_peered":"2024-09-21T10:41:20.985813+0000","last_clean":"2024-09-21T10:41:20.985813+0000","last_became_active":"2024-09-21T10:40:16.444802+0000","last_became_peered":"2024-09-21T10:40:16.444802+0000","last_unstale":"2024-09-21T10:41:20.985813+0000","last_undegraded":"2024-09-21T10:41:20.985813+0000","last_fullsized":"2024-09-21T10:41:20.985813+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":5633,"log_dups_size":0,"ondisk_log_size":5633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:36:32.765238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369164288,"num_objects":5633,"num_object_clones":0,"num_object_copies":16899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5633,"num_write_kb":360512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"53.a","version":"1334'5967","reported_seq":6039,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:24.332833+0000","last_change":"2024-09-21T10:40:16.451613+0000","last_active":"2024-09-21T10:41:24.332833+0000","last_peered":"2024-09-21T10:41:24.332833+0000","last_clean":"2024-09-21T10:41:24.332833+0000","last_became_active":"2024-09-21T10:40:16.451348+0000","last_became_peered":"2024-09-21T10:40:16.451348+0000","last_unstale":"2024-09-21T10:41:24.332833+0000","last_undegraded":"2024-09-21T10:41:24.332833+0000","last_fullsized":"2024-09-21T10:41:24.332833+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":5967,"log_dups_size":0,"ondisk_log_size":5967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:05:08.562680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391053312,"num_objects":5967,"num_object_clones":0,"num_object_copies":17901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5967,"num_write_kb":381888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"53.9","version":"1334'5190","reported_seq":5269,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:24.360116+0000","last_change":"2024-09-21T10:41:00.659513+0000","last_active":"2024-09-21T10:41:24.360116+0000","last_peered":"2024-09-21T10:41:24.360116+0000","last_clean":"2024-09-21T10:41:24.360116+0000","last_became_active":"2024-09-21T10:41:00.656007+0000","last_became_peered":"2024-09-21T10:41:00.656007+0000","last_unstale":"2024-09-21T10:41:24.360116+0000","last_undegraded":"2024-09-21T10:41:24.360116+0000","last_fullsized":"2024-09-21T10:41:24.360116+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":5190,"log_dups_size":0,"ondisk_log_size":5190,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196870144,"num_objects":3004,"num_object_clones":0,"num_object_copies":9012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3004,"num_write_kb":192256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.8","version":"1334'4984","reported_seq":5090,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:20.953523+0000","last_change":"2024-09-21T10:41:01.831615+0000","last_active":"2024-09-21T10:41:20.953523+0000","last_peered":"2024-09-21T10:41:20.953523+0000","last_clean":"2024-09-21T10:41:20.953523+0000","last_became_active":"2024-09-21T10:41:00.642849+0000","last_became_peered":"2024-09-21T10:41:00.642849+0000","last_unstale":"2024-09-21T10:41:20.953523+0000","last_undegraded":"2024-09-21T10:41:20.953523+0000","last_fullsized":"2024-09-21T10:41:20.953523+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":0,"last_scrub":"1324'4314","last_scrub_stamp":"2024-09-21T10:41:01.831467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:41:01.831467+0000","objects_scrubbed":2229,"log_size":4984,"log_dups_size":0,"ondisk_log_size":4984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:24:13.824542+0000","scrub_duration":0.79565424399999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191627264,"num_objects":2924,"num_object_clones":0,"num_object_copies":8772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":695,"num_write_kb":44480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"53.7","version":"1334'5161","reported_seq":5269,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:23.766777+0000","last_change":"2024-09-21T10:41:01.305079+0000","last_active":"2024-09-21T10:41:23.766777+0000","last_peered":"2024-09-21T10:41:23.766777+0000","last_clean":"2024-09-21T10:41:23.766777+0000","last_became_active":"2024-09-21T10:41:00.643851+0000","last_became_peered":"2024-09-21T10:41:00.643851+0000","last_unstale":"2024-09-21T10:41:23.766777+0000","last_undegraded":"2024-09-21T10:41:23.766777+0000","last_fullsized":"2024-09-21T10:41:23.766777+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":0,"last_scrub":"1324'4378","last_scrub_stamp":"2024-09-21T10:41:01.304972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:41:01.304972+0000","objects_scrubbed":2133,"log_size":5161,"log_dups_size":0,"ondisk_log_size":5161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:55:41.695678+0000","scrub_duration":601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192282624,"num_objects":2934,"num_object_clones":0,"num_object_copies":8802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":801,"num_write_kb":51264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.6","version":"1334'5174","reported_seq":5285,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:24.274877+0000","last_change":"2024-09-21T10:41:05.966135+0000","last_active":"2024-09-21T10:41:24.274877+0000","last_peered":"2024-09-21T10:41:24.274877+0000","last_clean":"2024-09-21T10:41:24.274877+0000","last_became_active":"2024-09-21T10:41:00.643784+0000","last_became_peered":"2024-09-21T10:41:00.643784+0000","last_unstale":"2024-09-21T10:41:24.274877+0000","last_undegraded":"2024-09-21T10:41:24.274877+0000","last_fullsized":"2024-09-21T10:41:24.274877+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":0,"last_scrub":"1325'4633","last_scrub_stamp":"2024-09-21T10:41:05.966042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:41:05.966042+0000","objects_scrubbed":2477,"log_size":5174,"log_dups_size":0,"ondisk_log_size":5174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:25.235633+0000","scrub_duration":485,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198508544,"num_objects":3029,"num_object_clones":0,"num_object_copies":9087,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":552,"num_write_kb":35328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"53.5","version":"1334'5153","reported_seq":5236,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:24.512751+0000","last_change":"2024-09-21T10:41:00.694221+0000","last_active":"2024-09-21T10:41:24.512751+0000","last_peered":"2024-09-21T10:41:24.512751+0000","last_clean":"2024-09-21T10:41:24.512751+0000","last_became_active":"2024-09-21T10:41:00.677969+0000","last_became_peered":"2024-09-21T10:41:00.677969+0000","last_unstale":"2024-09-21T10:41:24.512751+0000","last_undegraded":"2024-09-21T10:41:24.512751+0000","last_fullsized":"2024-09-21T10:41:24.512751+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":5153,"log_dups_size":0,"ondisk_log_size":5153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197033984,"num_objects":3006,"num_object_clones":0,"num_object_copies":9018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3006,"num_write_kb":192416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.4","version":"1334'5210","reported_seq":5321,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:23.749131+0000","last_change":"2024-09-21T10:41:04.470741+0000","last_active":"2024-09-21T10:41:23.749131+0000","last_peered":"2024-09-21T10:41:23.749131+0000","last_clean":"2024-09-21T10:41:23.749131+0000","last_became_active":"2024-09-21T10:41:00.652865+0000","last_became_peered":"2024-09-21T10:41:00.652865+0000","last_unstale":"2024-09-21T10:41:23.749131+0000","last_undegraded":"2024-09-21T10:41:23.749131+0000","last_fullsized":"2024-09-21T10:41:23.749131+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":0,"last_scrub":"1324'4563","last_scrub_stamp":"2024-09-21T10:41:04.470538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:41:04.470538+0000","objects_scrubbed":2413,"log_size":5210,"log_dups_size":0,"ondisk_log_size":5210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:00.635682+0000","scrub_duration":748,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201457664,"num_objects":3074,"num_object_clones":0,"num_object_copies":9222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":661,"num_write_kb":42304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.3","version":"1334'5183","reported_seq":5291,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:21.101211+0000","last_change":"2024-09-21T10:41:01.640980+0000","last_active":"2024-09-21T10:41:21.101211+0000","last_peered":"2024-09-21T10:41:21.101211+0000","last_clean":"2024-09-21T10:41:21.101211+0000","last_became_active":"2024-09-21T10:41:00.656721+0000","last_became_peered":"2024-09-21T10:41:00.656721+0000","last_unstale":"2024-09-21T10:41:21.101211+0000","last_undegraded":"2024-09-21T10:41:21.101211+0000","last_fullsized":"2024-09-21T10:41:21.101211+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":0,"last_scrub":"1324'4534","last_scrub_stamp":"2024-09-21T10:41:01.640809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:41:01.640809+0000","objects_scrubbed":2329,"log_size":5183,"log_dups_size":0,"ondisk_log_size":5183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:46.711073+0000","scrub_duration":0.68720467699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195887104,"num_objects":2989,"num_object_clones":0,"num_object_copies":8967,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":660,"num_write_kb":42240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.2","version":"1334'4918","reported_seq":5024,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:20.987200+0000","last_change":"2024-09-21T10:41:05.605174+0000","last_active":"2024-09-21T10:41:20.987200+0000","last_peered":"2024-09-21T10:41:20.987200+0000","last_clean":"2024-09-21T10:41:20.987200+0000","last_became_active":"2024-09-21T10:41:00.653765+0000","last_became_peered":"2024-09-21T10:41:00.653765+0000","last_unstale":"2024-09-21T10:41:20.987200+0000","last_undegraded":"2024-09-21T10:41:20.987200+0000","last_fullsized":"2024-09-21T10:41:20.987200+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":0,"last_scrub":"1324'4518","last_scrub_stamp":"2024-09-21T10:41:05.604498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:41:05.604498+0000","objects_scrubbed":2371,"log_size":4918,"log_dups_size":0,"ondisk_log_size":4918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:02:29.176750+0000","scrub_duration":0.52519471500000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182714368,"num_objects":2788,"num_object_clones":0,"num_object_copies":8364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":417,"num_write_kb":26688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"53.1","version":"1334'5152","reported_seq":5260,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:23.781401+0000","last_change":"2024-09-21T10:41:02.404669+0000","last_active":"2024-09-21T10:41:23.781401+0000","last_peered":"2024-09-21T10:41:23.781401+0000","last_clean":"2024-09-21T10:41:23.781401+0000","last_became_active":"2024-09-21T10:41:00.644585+0000","last_became_peered":"2024-09-21T10:41:00.644585+0000","last_unstale":"2024-09-21T10:41:23.781401+0000","last_undegraded":"2024-09-21T10:41:23.781401+0000","last_fullsized":"2024-09-21T10:41:23.781401+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":0,"last_scrub":"1324'4434","last_scrub_stamp":"2024-09-21T10:41:02.404449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:41:02.404449+0000","objects_scrubbed":2298,"log_size":5152,"log_dups_size":0,"ondisk_log_size":5152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:23:47.765903+0000","scrub_duration":693,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199098368,"num_objects":3038,"num_object_clones":0,"num_object_copies":9114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":740,"num_write_kb":47360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.0","version":"1334'4986","reported_seq":5088,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:20.957310+0000","last_change":"2024-09-21T10:41:02.770416+0000","last_active":"2024-09-21T10:41:20.957310+0000","last_peered":"2024-09-21T10:41:20.957310+0000","last_clean":"2024-09-21T10:41:20.957310+0000","last_became_active":"2024-09-21T10:41:00.642952+0000","last_became_peered":"2024-09-21T10:41:00.642952+0000","last_unstale":"2024-09-21T10:41:20.957310+0000","last_undegraded":"2024-09-21T10:41:20.957310+0000","last_fullsized":"2024-09-21T10:41:20.957310+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":0,"last_scrub":"1324'4424","last_scrub_stamp":"2024-09-21T10:41:02.770194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:41:02.770194+0000","objects_scrubbed":2234,"log_size":4986,"log_dups_size":0,"ondisk_log_size":4986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:17:14.808618+0000","scrub_duration":0.69884458100000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184090624,"num_objects":2809,"num_object_clones":0,"num_object_copies":8427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":575,"num_write_kb":36800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":1978,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.128930+0000","last_change":"2024-09-21T10:40:13.438023+0000","last_active":"2024-09-21T10:41:14.128930+0000","last_peered":"2024-09-21T10:41:14.128930+0000","last_clean":"2024-09-21T10:41:14.128930+0000","last_became_active":"2024-09-21T10:40:13.431103+0000","last_became_peered":"2024-09-21T10:40:13.431103+0000","last_unstale":"2024-09-21T10:41:14.128930+0000","last_undegraded":"2024-09-21T10:41:14.128930+0000","last_fullsized":"2024-09-21T10:41:14.128930+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:00:24.887661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1301,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.121561+0000","last_change":"2024-09-21T10:07:55.999392+0000","last_active":"2024-09-21T10:41:13.121561+0000","last_peered":"2024-09-21T10:41:13.121561+0000","last_clean":"2024-09-21T10:41:13.121561+0000","last_became_active":"2024-09-21T10:07:55.999103+0000","last_became_peered":"2024-09-21T10:07:55.999103+0000","last_unstale":"2024-09-21T10:41:13.121561+0000","last_undegraded":"2024-09-21T10:41:13.121561+0000","last_fullsized":"2024-09-21T10:41:13.121561+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:48.757242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":1959,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.120944+0000","last_change":"2024-09-21T10:40:13.432818+0000","last_active":"2024-09-21T10:41:14.120944+0000","last_peered":"2024-09-21T10:41:14.120944+0000","last_clean":"2024-09-21T10:41:14.120944+0000","last_became_active":"2024-09-21T10:40:13.432448+0000","last_became_peered":"2024-09-21T10:40:13.432448+0000","last_unstale":"2024-09-21T10:41:14.120944+0000","last_undegraded":"2024-09-21T10:41:14.120944+0000","last_fullsized":"2024-09-21T10:41:14.120944+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:20.448242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1976,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.120077+0000","last_change":"2024-09-21T10:40:13.431824+0000","last_active":"2024-09-21T10:41:14.120077+0000","last_peered":"2024-09-21T10:41:14.120077+0000","last_clean":"2024-09-21T10:41:14.120077+0000","last_became_active":"2024-09-21T10:40:13.431516+0000","last_became_peered":"2024-09-21T10:40:13.431516+0000","last_unstale":"2024-09-21T10:41:14.120077+0000","last_undegraded":"2024-09-21T10:41:14.120077+0000","last_fullsized":"2024-09-21T10:41:14.120077+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:30:40.226452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":1288,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.123362+0000","last_change":"2024-09-21T09:54:53.634508+0000","last_active":"2024-09-21T10:41:13.123362+0000","last_peered":"2024-09-21T10:41:13.123362+0000","last_clean":"2024-09-21T10:41:13.123362+0000","last_became_active":"2024-09-21T09:54:53.634254+0000","last_became_peered":"2024-09-21T09:54:53.634254+0000","last_unstale":"2024-09-21T10:41:13.123362+0000","last_undegraded":"2024-09-21T10:41:13.123362+0000","last_fullsized":"2024-09-21T10:41:13.123362+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:18.122107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1315,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.122605+0000","last_change":"2024-09-21T10:09:06.574933+0000","last_active":"2024-09-21T10:41:13.122605+0000","last_peered":"2024-09-21T10:41:13.122605+0000","last_clean":"2024-09-21T10:41:13.122605+0000","last_became_active":"2024-09-21T10:09:06.574467+0000","last_became_peered":"2024-09-21T10:09:06.574467+0000","last_unstale":"2024-09-21T10:41:13.122605+0000","last_undegraded":"2024-09-21T10:41:13.122605+0000","last_fullsized":"2024-09-21T10:41:13.122605+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:06.142269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":1303,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.123509+0000","last_change":"2024-09-21T10:07:55.991189+0000","last_active":"2024-09-21T10:41:13.123509+0000","last_peered":"2024-09-21T10:41:13.123509+0000","last_clean":"2024-09-21T10:41:13.123509+0000","last_became_active":"2024-09-21T10:07:55.990784+0000","last_became_peered":"2024-09-21T10:07:55.990784+0000","last_unstale":"2024-09-21T10:41:13.123509+0000","last_undegraded":"2024-09-21T10:41:13.123509+0000","last_fullsized":"2024-09-21T10:41:13.123509+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:00.580471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1315,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.122004+0000","last_change":"2024-09-21T10:09:06.560175+0000","last_active":"2024-09-21T10:41:13.122004+0000","last_peered":"2024-09-21T10:41:13.122004+0000","last_clean":"2024-09-21T10:41:13.122004+0000","last_became_active":"2024-09-21T10:09:06.558584+0000","last_became_peered":"2024-09-21T10:09:06.558584+0000","last_unstale":"2024-09-21T10:41:13.122004+0000","last_undegraded":"2024-09-21T10:41:13.122004+0000","last_fullsized":"2024-09-21T10:41:13.122004+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:02.918339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":1954,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.128397+0000","last_change":"2024-09-21T10:40:30.424630+0000","last_active":"2024-09-21T10:41:14.128397+0000","last_peered":"2024-09-21T10:41:14.128397+0000","last_clean":"2024-09-21T10:41:14.128397+0000","last_became_active":"2024-09-21T10:40:30.424295+0000","last_became_peered":"2024-09-21T10:40:30.424295+0000","last_unstale":"2024-09-21T10:41:14.128397+0000","last_undegraded":"2024-09-21T10:41:14.128397+0000","last_fullsized":"2024-09-21T10:41:14.128397+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:59:50.861993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":738,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.134470+0000","last_change":"2024-09-21T10:40:13.436927+0000","last_active":"2024-09-21T10:41:14.134470+0000","last_peered":"2024-09-21T10:41:14.134470+0000","last_clean":"2024-09-21T10:41:14.134470+0000","last_became_active":"2024-09-21T10:40:13.436329+0000","last_became_peered":"2024-09-21T10:40:13.436329+0000","last_unstale":"2024-09-21T10:41:14.134470+0000","last_undegraded":"2024-09-21T10:41:14.134470+0000","last_fullsized":"2024-09-21T10:41:14.134470+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:05.668898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":1326,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.122843+0000","last_change":"2024-09-21T10:40:13.433278+0000","last_active":"2024-09-21T10:41:13.122843+0000","last_peered":"2024-09-21T10:41:13.122843+0000","last_clean":"2024-09-21T10:41:13.122843+0000","last_became_active":"2024-09-21T10:40:13.432814+0000","last_became_peered":"2024-09-21T10:40:13.432814+0000","last_unstale":"2024-09-21T10:41:13.122843+0000","last_undegraded":"2024-09-21T10:41:13.122843+0000","last_fullsized":"2024-09-21T10:41:13.122843+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:55.909872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1969,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.121216+0000","last_change":"2024-09-21T10:40:13.432531+0000","last_active":"2024-09-21T10:41:14.121216+0000","last_peered":"2024-09-21T10:41:14.121216+0000","last_clean":"2024-09-21T10:41:14.121216+0000","last_became_active":"2024-09-21T10:40:13.432210+0000","last_became_peered":"2024-09-21T10:40:13.432210+0000","last_unstale":"2024-09-21T10:41:14.121216+0000","last_undegraded":"2024-09-21T10:41:14.121216+0000","last_fullsized":"2024-09-21T10:41:14.121216+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:20:10.998607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1941,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.121878+0000","last_change":"2024-09-21T10:09:06.530394+0000","last_active":"2024-09-21T10:41:14.121878+0000","last_peered":"2024-09-21T10:41:14.121878+0000","last_clean":"2024-09-21T10:41:14.121878+0000","last_became_active":"2024-09-21T10:09:06.529307+0000","last_became_peered":"2024-09-21T10:09:06.529307+0000","last_unstale":"2024-09-21T10:41:14.121878+0000","last_undegraded":"2024-09-21T10:41:14.121878+0000","last_fullsized":"2024-09-21T10:41:14.121878+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:54.287397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1966,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.128708+0000","last_change":"2024-09-21T10:40:30.420276+0000","last_active":"2024-09-21T10:41:14.128708+0000","last_peered":"2024-09-21T10:41:14.128708+0000","last_clean":"2024-09-21T10:41:14.128708+0000","last_became_active":"2024-09-21T10:40:30.419723+0000","last_became_peered":"2024-09-21T10:40:30.419723+0000","last_unstale":"2024-09-21T10:41:14.128708+0000","last_undegraded":"2024-09-21T10:41:14.128708+0000","last_fullsized":"2024-09-21T10:41:14.128708+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:52:04.181647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1956,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.120037+0000","last_change":"2024-09-21T10:09:06.552324+0000","last_active":"2024-09-21T10:41:14.120037+0000","last_peered":"2024-09-21T10:41:14.120037+0000","last_clean":"2024-09-21T10:41:14.120037+0000","last_became_active":"2024-09-21T10:09:06.540545+0000","last_became_peered":"2024-09-21T10:09:06.540545+0000","last_unstale":"2024-09-21T10:41:14.120037+0000","last_undegraded":"2024-09-21T10:41:14.120037+0000","last_fullsized":"2024-09-21T10:41:14.120037+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:08.506149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1305,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.132925+0000","last_change":"2024-09-21T10:07:56.003005+0000","last_active":"2024-09-21T10:41:13.132925+0000","last_peered":"2024-09-21T10:41:13.132925+0000","last_clean":"2024-09-21T10:41:13.132925+0000","last_became_active":"2024-09-21T10:07:56.001550+0000","last_became_peered":"2024-09-21T10:07:56.001550+0000","last_unstale":"2024-09-21T10:41:13.132925+0000","last_undegraded":"2024-09-21T10:41:13.132925+0000","last_fullsized":"2024-09-21T10:41:13.132925+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:54.553261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1948,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.120137+0000","last_change":"2024-09-21T10:09:06.559017+0000","last_active":"2024-09-21T10:41:14.120137+0000","last_peered":"2024-09-21T10:41:14.120137+0000","last_clean":"2024-09-21T10:41:14.120137+0000","last_became_active":"2024-09-21T10:09:06.555256+0000","last_became_peered":"2024-09-21T10:09:06.555256+0000","last_unstale":"2024-09-21T10:41:14.120137+0000","last_undegraded":"2024-09-21T10:41:14.120137+0000","last_fullsized":"2024-09-21T10:41:14.120137+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:33:45.145138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1954,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.129055+0000","last_change":"2024-09-21T10:09:06.562504+0000","last_active":"2024-09-21T10:41:14.129055+0000","last_peered":"2024-09-21T10:41:14.129055+0000","last_clean":"2024-09-21T10:41:14.129055+0000","last_became_active":"2024-09-21T10:09:06.561747+0000","last_became_peered":"2024-09-21T10:09:06.561747+0000","last_unstale":"2024-09-21T10:41:14.129055+0000","last_undegraded":"2024-09-21T10:41:14.129055+0000","last_fullsized":"2024-09-21T10:41:14.129055+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:03:58.012406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1946,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.120192+0000","last_change":"2024-09-21T10:40:30.417426+0000","last_active":"2024-09-21T10:41:14.120192+0000","last_peered":"2024-09-21T10:41:14.120192+0000","last_clean":"2024-09-21T10:41:14.120192+0000","last_became_active":"2024-09-21T10:40:30.417075+0000","last_became_peered":"2024-09-21T10:40:30.417075+0000","last_unstale":"2024-09-21T10:41:14.120192+0000","last_undegraded":"2024-09-21T10:41:14.120192+0000","last_fullsized":"2024-09-21T10:41:14.120192+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:58.077873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1311,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.122879+0000","last_change":"2024-09-21T10:40:30.421413+0000","last_active":"2024-09-21T10:41:13.122879+0000","last_peered":"2024-09-21T10:41:13.122879+0000","last_clean":"2024-09-21T10:41:13.122879+0000","last_became_active":"2024-09-21T10:40:30.420689+0000","last_became_peered":"2024-09-21T10:40:30.420689+0000","last_unstale":"2024-09-21T10:41:13.122879+0000","last_undegraded":"2024-09-21T10:41:13.122879+0000","last_fullsized":"2024-09-21T10:41:13.122879+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:53.860610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":73,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.121272+0000","last_change":"2024-09-21T10:40:30.431008+0000","last_active":"2024-09-21T10:41:14.121272+0000","last_peered":"2024-09-21T10:41:14.121272+0000","last_clean":"2024-09-21T10:41:14.121272+0000","last_became_active":"2024-09-21T10:40:30.422769+0000","last_became_peered":"2024-09-21T10:40:30.422769+0000","last_unstale":"2024-09-21T10:41:14.121272+0000","last_undegraded":"2024-09-21T10:41:14.121272+0000","last_fullsized":"2024-09-21T10:41:14.121272+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:42:21.520242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1948,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.122599+0000","last_change":"2024-09-21T10:40:13.437741+0000","last_active":"2024-09-21T10:41:14.122599+0000","last_peered":"2024-09-21T10:41:14.122599+0000","last_clean":"2024-09-21T10:41:14.122599+0000","last_became_active":"2024-09-21T10:40:13.437447+0000","last_became_peered":"2024-09-21T10:40:13.437447+0000","last_unstale":"2024-09-21T10:41:14.122599+0000","last_undegraded":"2024-09-21T10:41:14.122599+0000","last_fullsized":"2024-09-21T10:41:14.122599+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:40:47.072907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":1329,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.122573+0000","last_change":"2024-09-21T10:40:30.437433+0000","last_active":"2024-09-21T10:41:13.122573+0000","last_peered":"2024-09-21T10:41:13.122573+0000","last_clean":"2024-09-21T10:41:13.122573+0000","last_became_active":"2024-09-21T10:40:30.426024+0000","last_became_peered":"2024-09-21T10:40:30.426024+0000","last_unstale":"2024-09-21T10:41:13.122573+0000","last_undegraded":"2024-09-21T10:41:13.122573+0000","last_fullsized":"2024-09-21T10:41:13.122573+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:13:47.175034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4756,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1301,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.122006+0000","last_change":"2024-09-21T10:07:56.001756+0000","last_active":"2024-09-21T10:41:13.122006+0000","last_peered":"2024-09-21T10:41:13.122006+0000","last_clean":"2024-09-21T10:41:13.122006+0000","last_became_active":"2024-09-21T10:07:56.000789+0000","last_became_peered":"2024-09-21T10:07:56.000789+0000","last_unstale":"2024-09-21T10:41:13.122006+0000","last_undegraded":"2024-09-21T10:41:13.122006+0000","last_fullsized":"2024-09-21T10:41:13.122006+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:56:04.302902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1289,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.121878+0000","last_change":"2024-09-21T09:54:53.636902+0000","last_active":"2024-09-21T10:41:13.121878+0000","last_peered":"2024-09-21T10:41:13.121878+0000","last_clean":"2024-09-21T10:41:13.121878+0000","last_became_active":"2024-09-21T09:54:53.636616+0000","last_became_peered":"2024-09-21T09:54:53.636616+0000","last_unstale":"2024-09-21T10:41:13.121878+0000","last_undegraded":"2024-09-21T10:41:13.121878+0000","last_fullsized":"2024-09-21T10:41:13.121878+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:46:47.694495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1335,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.133159+0000","last_change":"2024-09-21T10:40:30.423568+0000","last_active":"2024-09-21T10:41:13.133159+0000","last_peered":"2024-09-21T10:41:13.133159+0000","last_clean":"2024-09-21T10:41:13.133159+0000","last_became_active":"2024-09-21T10:40:30.423208+0000","last_became_peered":"2024-09-21T10:40:30.423208+0000","last_unstale":"2024-09-21T10:41:13.133159+0000","last_undegraded":"2024-09-21T10:41:13.133159+0000","last_fullsized":"2024-09-21T10:41:13.133159+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:22:29.996089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":1961,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.128612+0000","last_change":"2024-09-21T10:07:56.023462+0000","last_active":"2024-09-21T10:41:14.128612+0000","last_peered":"2024-09-21T10:41:14.128612+0000","last_clean":"2024-09-21T10:41:14.128612+0000","last_became_active":"2024-09-21T10:07:56.021859+0000","last_became_peered":"2024-09-21T10:07:56.021859+0000","last_unstale":"2024-09-21T10:41:14.128612+0000","last_undegraded":"2024-09-21T10:41:14.128612+0000","last_fullsized":"2024-09-21T10:41:14.128612+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:42:03.134729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1943,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.120128+0000","last_change":"2024-09-21T10:40:30.420880+0000","last_active":"2024-09-21T10:41:14.120128+0000","last_peered":"2024-09-21T10:41:14.120128+0000","last_clean":"2024-09-21T10:41:14.120128+0000","last_became_active":"2024-09-21T10:40:30.420555+0000","last_became_peered":"2024-09-21T10:40:30.420555+0000","last_unstale":"2024-09-21T10:41:14.120128+0000","last_undegraded":"2024-09-21T10:41:14.120128+0000","last_fullsized":"2024-09-21T10:41:14.120128+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:15:30.473928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1942,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.122844+0000","last_change":"2024-09-21T10:07:56.006236+0000","last_active":"2024-09-21T10:41:14.122844+0000","last_peered":"2024-09-21T10:41:14.122844+0000","last_clean":"2024-09-21T10:41:14.122844+0000","last_became_active":"2024-09-21T10:07:56.004439+0000","last_became_peered":"2024-09-21T10:07:56.004439+0000","last_unstale":"2024-09-21T10:41:14.122844+0000","last_undegraded":"2024-09-21T10:41:14.122844+0000","last_fullsized":"2024-09-21T10:41:14.122844+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:39.422467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":1947,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.121095+0000","last_change":"2024-09-21T10:40:30.425998+0000","last_active":"2024-09-21T10:41:14.121095+0000","last_peered":"2024-09-21T10:41:14.121095+0000","last_clean":"2024-09-21T10:41:14.121095+0000","last_became_active":"2024-09-21T10:40:30.421240+0000","last_became_peered":"2024-09-21T10:40:30.421240+0000","last_unstale":"2024-09-21T10:41:14.121095+0000","last_undegraded":"2024-09-21T10:41:14.121095+0000","last_fullsized":"2024-09-21T10:41:14.121095+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:43:44.457431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1955,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.128555+0000","last_change":"2024-09-21T10:40:30.417473+0000","last_active":"2024-09-21T10:41:14.128555+0000","last_peered":"2024-09-21T10:41:14.128555+0000","last_clean":"2024-09-21T10:41:14.128555+0000","last_became_active":"2024-09-21T10:40:30.416952+0000","last_became_peered":"2024-09-21T10:40:30.416952+0000","last_unstale":"2024-09-21T10:41:14.128555+0000","last_undegraded":"2024-09-21T10:41:14.128555+0000","last_fullsized":"2024-09-21T10:41:14.128555+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:07:33.486362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":684,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.128802+0000","last_change":"2024-09-21T10:40:30.425276+0000","last_active":"2024-09-21T10:41:14.128802+0000","last_peered":"2024-09-21T10:41:14.128802+0000","last_clean":"2024-09-21T10:41:14.128802+0000","last_became_active":"2024-09-21T10:40:30.425046+0000","last_became_peered":"2024-09-21T10:40:30.425046+0000","last_unstale":"2024-09-21T10:41:14.128802+0000","last_undegraded":"2024-09-21T10:41:14.128802+0000","last_fullsized":"2024-09-21T10:41:14.128802+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:09:59.127115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1978,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.134520+0000","last_change":"2024-09-21T10:40:13.437609+0000","last_active":"2024-09-21T10:41:14.134520+0000","last_peered":"2024-09-21T10:41:14.134520+0000","last_clean":"2024-09-21T10:41:14.134520+0000","last_became_active":"2024-09-21T10:40:13.437296+0000","last_became_peered":"2024-09-21T10:40:13.437296+0000","last_unstale":"2024-09-21T10:41:14.134520+0000","last_undegraded":"2024-09-21T10:41:14.134520+0000","last_fullsized":"2024-09-21T10:41:14.134520+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:41:58.963386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1933,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.122490+0000","last_change":"2024-09-21T10:07:56.067772+0000","last_active":"2024-09-21T10:41:14.122490+0000","last_peered":"2024-09-21T10:41:14.122490+0000","last_clean":"2024-09-21T10:41:14.122490+0000","last_became_active":"2024-09-21T10:07:56.052602+0000","last_became_peered":"2024-09-21T10:07:56.052602+0000","last_unstale":"2024-09-21T10:41:14.122490+0000","last_undegraded":"2024-09-21T10:41:14.122490+0000","last_fullsized":"2024-09-21T10:41:14.122490+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:07:59.899214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1994,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.134366+0000","last_change":"2024-09-21T10:40:30.418199+0000","last_active":"2024-09-21T10:41:14.134366+0000","last_peered":"2024-09-21T10:41:14.134366+0000","last_clean":"2024-09-21T10:41:14.134366+0000","last_became_active":"2024-09-21T10:40:30.417414+0000","last_became_peered":"2024-09-21T10:40:30.417414+0000","last_unstale":"2024-09-21T10:41:14.134366+0000","last_undegraded":"2024-09-21T10:41:14.134366+0000","last_fullsized":"2024-09-21T10:41:14.134366+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:56:54.232106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1329,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.123590+0000","last_change":"2024-09-21T10:40:13.435375+0000","last_active":"2024-09-21T10:41:13.123590+0000","last_peered":"2024-09-21T10:41:13.123590+0000","last_clean":"2024-09-21T10:41:13.123590+0000","last_became_active":"2024-09-21T10:40:13.435050+0000","last_became_peered":"2024-09-21T10:40:13.435050+0000","last_unstale":"2024-09-21T10:41:13.123590+0000","last_undegraded":"2024-09-21T10:41:13.123590+0000","last_fullsized":"2024-09-21T10:41:13.123590+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:57.889610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1988,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.134548+0000","last_change":"2024-09-21T10:40:13.434189+0000","last_active":"2024-09-21T10:41:14.134548+0000","last_peered":"2024-09-21T10:41:14.134548+0000","last_clean":"2024-09-21T10:41:14.134548+0000","last_became_active":"2024-09-21T10:40:13.433892+0000","last_became_peered":"2024-09-21T10:40:13.433892+0000","last_unstale":"2024-09-21T10:41:14.134548+0000","last_undegraded":"2024-09-21T10:41:14.134548+0000","last_fullsized":"2024-09-21T10:41:14.134548+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:49:17.296220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1305,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.122082+0000","last_change":"2024-09-21T10:07:21.524082+0000","last_active":"2024-09-21T10:41:13.122082+0000","last_peered":"2024-09-21T10:41:13.122082+0000","last_clean":"2024-09-21T10:41:13.122082+0000","last_became_active":"2024-09-21T10:07:21.523779+0000","last_became_peered":"2024-09-21T10:07:21.523779+0000","last_unstale":"2024-09-21T10:41:13.122082+0000","last_undegraded":"2024-09-21T10:41:13.122082+0000","last_fullsized":"2024-09-21T10:41:13.122082+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:13.813832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":1319,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.123676+0000","last_change":"2024-09-21T10:07:56.024239+0000","last_active":"2024-09-21T10:41:13.123676+0000","last_peered":"2024-09-21T10:41:13.123676+0000","last_clean":"2024-09-21T10:41:13.123676+0000","last_became_active":"2024-09-21T10:07:56.023914+0000","last_became_peered":"2024-09-21T10:07:56.023914+0000","last_unstale":"2024-09-21T10:41:13.123676+0000","last_undegraded":"2024-09-21T10:41:13.123676+0000","last_fullsized":"2024-09-21T10:41:13.123676+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:23.361736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084477399999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":24,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.122660+0000","last_change":"2024-09-21T10:40:55.616051+0000","last_active":"2024-09-21T10:41:13.122660+0000","last_peered":"2024-09-21T10:41:13.122660+0000","last_clean":"2024-09-21T10:41:13.122660+0000","last_became_active":"2024-09-21T10:40:55.615741+0000","last_became_peered":"2024-09-21T10:40:55.615741+0000","last_unstale":"2024-09-21T10:41:13.122660+0000","last_undegraded":"2024-09-21T10:41:13.122660+0000","last_fullsized":"2024-09-21T10:41:13.122660+0000","mapping_epoch":1320,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:07.673527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1951,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.121793+0000","last_change":"2024-09-21T10:40:13.436234+0000","last_active":"2024-09-21T10:41:14.121793+0000","last_peered":"2024-09-21T10:41:14.121793+0000","last_clean":"2024-09-21T10:41:14.121793+0000","last_became_active":"2024-09-21T10:40:13.435946+0000","last_became_peered":"2024-09-21T10:40:13.435946+0000","last_unstale":"2024-09-21T10:41:14.121793+0000","last_undegraded":"2024-09-21T10:41:14.121793+0000","last_fullsized":"2024-09-21T10:41:14.121793+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:08:08.670739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":744,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.120011+0000","last_change":"2024-09-21T10:40:13.431693+0000","last_active":"2024-09-21T10:41:14.120011+0000","last_peered":"2024-09-21T10:41:14.120011+0000","last_clean":"2024-09-21T10:41:14.120011+0000","last_became_active":"2024-09-21T10:40:13.431389+0000","last_became_peered":"2024-09-21T10:40:13.431389+0000","last_unstale":"2024-09-21T10:41:14.120011+0000","last_undegraded":"2024-09-21T10:41:14.120011+0000","last_fullsized":"2024-09-21T10:41:14.120011+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:22.227336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1983,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.128800+0000","last_change":"2024-09-21T10:40:30.420482+0000","last_active":"2024-09-21T10:41:14.128800+0000","last_peered":"2024-09-21T10:41:14.128800+0000","last_clean":"2024-09-21T10:41:14.128800+0000","last_became_active":"2024-09-21T10:40:30.419902+0000","last_became_peered":"2024-09-21T10:40:30.419902+0000","last_unstale":"2024-09-21T10:41:14.128800+0000","last_undegraded":"2024-09-21T10:41:14.128800+0000","last_fullsized":"2024-09-21T10:41:14.128800+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:37:40.880948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1957,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.128811+0000","last_change":"2024-09-21T10:09:06.560130+0000","last_active":"2024-09-21T10:41:14.128811+0000","last_peered":"2024-09-21T10:41:14.128811+0000","last_clean":"2024-09-21T10:41:14.128811+0000","last_became_active":"2024-09-21T10:09:06.559827+0000","last_became_peered":"2024-09-21T10:09:06.559827+0000","last_unstale":"2024-09-21T10:41:14.128811+0000","last_undegraded":"2024-09-21T10:41:14.128811+0000","last_fullsized":"2024-09-21T10:41:14.128811+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:05:17.166734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":707,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.122096+0000","last_change":"2024-09-21T10:40:30.417793+0000","last_active":"2024-09-21T10:41:14.122096+0000","last_peered":"2024-09-21T10:41:14.122096+0000","last_clean":"2024-09-21T10:41:14.122096+0000","last_became_active":"2024-09-21T10:40:30.417232+0000","last_became_peered":"2024-09-21T10:40:30.417232+0000","last_unstale":"2024-09-21T10:41:14.122096+0000","last_undegraded":"2024-09-21T10:41:14.122096+0000","last_fullsized":"2024-09-21T10:41:14.122096+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:09:58.959904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1299,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.132703+0000","last_change":"2024-09-21T10:40:30.426430+0000","last_active":"2024-09-21T10:41:13.132703+0000","last_peered":"2024-09-21T10:41:13.132703+0000","last_clean":"2024-09-21T10:41:13.132703+0000","last_became_active":"2024-09-21T10:40:30.426120+0000","last_became_peered":"2024-09-21T10:40:30.426120+0000","last_unstale":"2024-09-21T10:41:13.132703+0000","last_undegraded":"2024-09-21T10:41:13.132703+0000","last_fullsized":"2024-09-21T10:41:13.132703+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:23:15.280912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1938,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.120241+0000","last_change":"2024-09-21T10:09:06.560589+0000","last_active":"2024-09-21T10:41:14.120241+0000","last_peered":"2024-09-21T10:41:14.120241+0000","last_clean":"2024-09-21T10:41:14.120241+0000","last_became_active":"2024-09-21T10:09:06.557552+0000","last_became_peered":"2024-09-21T10:09:06.557552+0000","last_unstale":"2024-09-21T10:41:14.120241+0000","last_undegraded":"2024-09-21T10:41:14.120241+0000","last_fullsized":"2024-09-21T10:41:14.120241+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:09:24.866759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":1344,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.123083+0000","last_change":"2024-09-21T10:40:30.431163+0000","last_active":"2024-09-21T10:41:13.123083+0000","last_peered":"2024-09-21T10:41:13.123083+0000","last_clean":"2024-09-21T10:41:13.123083+0000","last_became_active":"2024-09-21T10:40:30.424471+0000","last_became_peered":"2024-09-21T10:40:30.424471+0000","last_unstale":"2024-09-21T10:41:13.123083+0000","last_undegraded":"2024-09-21T10:41:13.123083+0000","last_fullsized":"2024-09-21T10:41:13.123083+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:02:45.899239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":680,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.128276+0000","last_change":"2024-09-21T10:40:33.444446+0000","last_active":"2024-09-21T10:41:14.128276+0000","last_peered":"2024-09-21T10:41:14.128276+0000","last_clean":"2024-09-21T10:41:14.128276+0000","last_became_active":"2024-09-21T10:40:33.443958+0000","last_became_peered":"2024-09-21T10:40:33.443958+0000","last_unstale":"2024-09-21T10:41:14.128276+0000","last_undegraded":"2024-09-21T10:41:14.128276+0000","last_fullsized":"2024-09-21T10:41:14.128276+0000","mapping_epoch":1315,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:15:12.818119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1942,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.122651+0000","last_change":"2024-09-21T10:09:06.538940+0000","last_active":"2024-09-21T10:41:14.122651+0000","last_peered":"2024-09-21T10:41:14.122651+0000","last_clean":"2024-09-21T10:41:14.122651+0000","last_became_active":"2024-09-21T10:09:06.538668+0000","last_became_peered":"2024-09-21T10:09:06.538668+0000","last_unstale":"2024-09-21T10:41:14.122651+0000","last_undegraded":"2024-09-21T10:41:14.122651+0000","last_fullsized":"2024-09-21T10:41:14.122651+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:42:44.299488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1950,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.119978+0000","last_change":"2024-09-21T10:40:30.419468+0000","last_active":"2024-09-21T10:41:14.119978+0000","last_peered":"2024-09-21T10:41:14.119978+0000","last_clean":"2024-09-21T10:41:14.119978+0000","last_became_active":"2024-09-21T10:40:30.419140+0000","last_became_peered":"2024-09-21T10:40:30.419140+0000","last_unstale":"2024-09-21T10:41:14.119978+0000","last_undegraded":"2024-09-21T10:41:14.119978+0000","last_fullsized":"2024-09-21T10:41:14.119978+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:37.424900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1302,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.122711+0000","last_change":"2024-09-21T10:07:55.998799+0000","last_active":"2024-09-21T10:41:13.122711+0000","last_peered":"2024-09-21T10:41:13.122711+0000","last_clean":"2024-09-21T10:41:13.122711+0000","last_became_active":"2024-09-21T10:07:55.998510+0000","last_became_peered":"2024-09-21T10:07:55.998510+0000","last_unstale":"2024-09-21T10:41:13.122711+0000","last_undegraded":"2024-09-21T10:41:13.122711+0000","last_fullsized":"2024-09-21T10:41:13.122711+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:10.474008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1322,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.122099+0000","last_change":"2024-09-21T10:40:30.419793+0000","last_active":"2024-09-21T10:41:13.122099+0000","last_peered":"2024-09-21T10:41:13.122099+0000","last_clean":"2024-09-21T10:41:13.122099+0000","last_became_active":"2024-09-21T10:40:30.419149+0000","last_became_peered":"2024-09-21T10:40:30.419149+0000","last_unstale":"2024-09-21T10:41:13.122099+0000","last_undegraded":"2024-09-21T10:41:13.122099+0000","last_fullsized":"2024-09-21T10:41:13.122099+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:33.741419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1326,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.122978+0000","last_change":"2024-09-21T10:40:13.432387+0000","last_active":"2024-09-21T10:41:13.122978+0000","last_peered":"2024-09-21T10:41:13.122978+0000","last_clean":"2024-09-21T10:41:13.122978+0000","last_became_active":"2024-09-21T10:40:13.432053+0000","last_became_peered":"2024-09-21T10:40:13.432053+0000","last_unstale":"2024-09-21T10:41:13.122978+0000","last_undegraded":"2024-09-21T10:41:13.122978+0000","last_fullsized":"2024-09-21T10:41:13.122978+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:37.486101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1305,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.132758+0000","last_change":"2024-09-21T10:07:56.020791+0000","last_active":"2024-09-21T10:41:13.132758+0000","last_peered":"2024-09-21T10:41:13.132758+0000","last_clean":"2024-09-21T10:41:13.132758+0000","last_became_active":"2024-09-21T10:07:56.020205+0000","last_became_peered":"2024-09-21T10:07:56.020205+0000","last_unstale":"2024-09-21T10:41:13.132758+0000","last_undegraded":"2024-09-21T10:41:13.132758+0000","last_fullsized":"2024-09-21T10:41:13.132758+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:09.813166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1302,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.122478+0000","last_change":"2024-09-21T10:09:06.533271+0000","last_active":"2024-09-21T10:41:13.122478+0000","last_peered":"2024-09-21T10:41:13.122478+0000","last_clean":"2024-09-21T10:41:13.122478+0000","last_became_active":"2024-09-21T10:09:06.532041+0000","last_became_peered":"2024-09-21T10:09:06.532041+0000","last_unstale":"2024-09-21T10:41:13.122478+0000","last_undegraded":"2024-09-21T10:41:13.122478+0000","last_fullsized":"2024-09-21T10:41:13.122478+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:04.236375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":1959,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.120398+0000","last_change":"2024-09-21T10:09:06.543422+0000","last_active":"2024-09-21T10:41:14.120398+0000","last_peered":"2024-09-21T10:41:14.120398+0000","last_clean":"2024-09-21T10:41:14.120398+0000","last_became_active":"2024-09-21T10:09:06.539647+0000","last_became_peered":"2024-09-21T10:09:06.539647+0000","last_unstale":"2024-09-21T10:41:14.120398+0000","last_undegraded":"2024-09-21T10:41:14.120398+0000","last_fullsized":"2024-09-21T10:41:14.120398+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:04.990085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1958,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.128905+0000","last_change":"2024-09-21T10:07:56.018560+0000","last_active":"2024-09-21T10:41:14.128905+0000","last_peered":"2024-09-21T10:41:14.128905+0000","last_clean":"2024-09-21T10:41:14.128905+0000","last_became_active":"2024-09-21T10:07:56.018261+0000","last_became_peered":"2024-09-21T10:07:56.018261+0000","last_unstale":"2024-09-21T10:41:14.128905+0000","last_undegraded":"2024-09-21T10:41:14.128905+0000","last_fullsized":"2024-09-21T10:41:14.128905+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:04:54.395456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1303,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.123615+0000","last_change":"2024-09-21T10:09:06.531935+0000","last_active":"2024-09-21T10:41:13.123615+0000","last_peered":"2024-09-21T10:41:13.123615+0000","last_clean":"2024-09-21T10:41:13.123615+0000","last_became_active":"2024-09-21T10:09:06.531610+0000","last_became_peered":"2024-09-21T10:09:06.531610+0000","last_unstale":"2024-09-21T10:41:13.123615+0000","last_undegraded":"2024-09-21T10:41:13.123615+0000","last_fullsized":"2024-09-21T10:41:13.123615+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:33.874405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1331,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.132565+0000","last_change":"2024-09-21T10:40:30.423107+0000","last_active":"2024-09-21T10:41:13.132565+0000","last_peered":"2024-09-21T10:41:13.132565+0000","last_clean":"2024-09-21T10:41:13.132565+0000","last_became_active":"2024-09-21T10:40:30.422770+0000","last_became_peered":"2024-09-21T10:40:30.422770+0000","last_unstale":"2024-09-21T10:41:13.132565+0000","last_undegraded":"2024-09-21T10:41:13.132565+0000","last_fullsized":"2024-09-21T10:41:13.132565+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:35:23.752720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":49,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.123354+0000","last_change":"2024-09-21T10:40:30.428793+0000","last_active":"2024-09-21T10:41:13.123354+0000","last_peered":"2024-09-21T10:41:13.123354+0000","last_clean":"2024-09-21T10:41:13.123354+0000","last_became_active":"2024-09-21T10:40:30.421116+0000","last_became_peered":"2024-09-21T10:40:30.421116+0000","last_unstale":"2024-09-21T10:41:13.123354+0000","last_undegraded":"2024-09-21T10:41:13.123354+0000","last_fullsized":"2024-09-21T10:41:13.123354+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:24:49.342671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1318,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.121958+0000","last_change":"2024-09-21T10:09:06.559132+0000","last_active":"2024-09-21T10:41:13.121958+0000","last_peered":"2024-09-21T10:41:13.121958+0000","last_clean":"2024-09-21T10:41:13.121958+0000","last_became_active":"2024-09-21T10:09:06.558395+0000","last_became_peered":"2024-09-21T10:09:06.558395+0000","last_unstale":"2024-09-21T10:41:13.121958+0000","last_undegraded":"2024-09-21T10:41:13.121958+0000","last_fullsized":"2024-09-21T10:41:13.121958+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:27:00.073554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":49,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.122616+0000","last_change":"2024-09-21T10:40:30.428625+0000","last_active":"2024-09-21T10:41:13.122616+0000","last_peered":"2024-09-21T10:41:13.122616+0000","last_clean":"2024-09-21T10:41:13.122616+0000","last_became_active":"2024-09-21T10:40:30.421565+0000","last_became_peered":"2024-09-21T10:40:30.421565+0000","last_unstale":"2024-09-21T10:41:13.122616+0000","last_undegraded":"2024-09-21T10:41:13.122616+0000","last_fullsized":"2024-09-21T10:41:13.122616+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:23.191395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1297,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.122542+0000","last_change":"2024-09-21T10:40:30.421601+0000","last_active":"2024-09-21T10:41:13.122542+0000","last_peered":"2024-09-21T10:41:13.122542+0000","last_clean":"2024-09-21T10:41:13.122542+0000","last_became_active":"2024-09-21T10:40:30.420430+0000","last_became_peered":"2024-09-21T10:40:30.420430+0000","last_unstale":"2024-09-21T10:41:13.122542+0000","last_undegraded":"2024-09-21T10:41:13.122542+0000","last_fullsized":"2024-09-21T10:41:13.122542+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:15.416335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":49,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.122819+0000","last_change":"2024-09-21T10:40:30.433050+0000","last_active":"2024-09-21T10:41:13.122819+0000","last_peered":"2024-09-21T10:41:13.122819+0000","last_clean":"2024-09-21T10:41:13.122819+0000","last_became_active":"2024-09-21T10:40:30.421211+0000","last_became_peered":"2024-09-21T10:40:30.421211+0000","last_unstale":"2024-09-21T10:41:13.122819+0000","last_undegraded":"2024-09-21T10:41:13.122819+0000","last_fullsized":"2024-09-21T10:41:13.122819+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:01:33.357050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1311,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.132439+0000","last_change":"2024-09-21T10:07:21.497877+0000","last_active":"2024-09-21T10:41:13.132439+0000","last_peered":"2024-09-21T10:41:13.132439+0000","last_clean":"2024-09-21T10:41:13.132439+0000","last_became_active":"2024-09-21T10:07:21.496317+0000","last_became_peered":"2024-09-21T10:07:21.496317+0000","last_unstale":"2024-09-21T10:41:13.132439+0000","last_undegraded":"2024-09-21T10:41:13.132439+0000","last_fullsized":"2024-09-21T10:41:13.132439+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:39:03.848183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1947,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.121006+0000","last_change":"2024-09-21T10:40:30.418410+0000","last_active":"2024-09-21T10:41:14.121006+0000","last_peered":"2024-09-21T10:41:14.121006+0000","last_clean":"2024-09-21T10:41:14.121006+0000","last_became_active":"2024-09-21T10:40:30.417846+0000","last_became_peered":"2024-09-21T10:40:30.417846+0000","last_unstale":"2024-09-21T10:41:14.121006+0000","last_undegraded":"2024-09-21T10:41:14.121006+0000","last_fullsized":"2024-09-21T10:41:14.121006+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:36:02.636480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.121775+0000","last_change":"2024-09-21T10:40:30.423721+0000","last_active":"2024-09-21T10:41:13.121775+0000","last_peered":"2024-09-21T10:41:13.121775+0000","last_clean":"2024-09-21T10:41:13.121775+0000","last_became_active":"2024-09-21T10:40:30.423415+0000","last_became_peered":"2024-09-21T10:40:30.423415+0000","last_unstale":"2024-09-21T10:41:13.121775+0000","last_undegraded":"2024-09-21T10:41:13.121775+0000","last_fullsized":"2024-09-21T10:41:13.121775+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:57.212463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1978,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.128560+0000","last_change":"2024-09-21T10:40:30.418240+0000","last_active":"2024-09-21T10:41:14.128560+0000","last_peered":"2024-09-21T10:41:14.128560+0000","last_clean":"2024-09-21T10:41:14.128560+0000","last_became_active":"2024-09-21T10:40:30.417843+0000","last_became_peered":"2024-09-21T10:40:30.417843+0000","last_unstale":"2024-09-21T10:41:14.128560+0000","last_undegraded":"2024-09-21T10:41:14.128560+0000","last_fullsized":"2024-09-21T10:41:14.128560+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:50.860905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1961,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.128469+0000","last_change":"2024-09-21T10:40:30.419513+0000","last_active":"2024-09-21T10:41:14.128469+0000","last_peered":"2024-09-21T10:41:14.128469+0000","last_clean":"2024-09-21T10:41:14.128469+0000","last_became_active":"2024-09-21T10:40:30.419182+0000","last_became_peered":"2024-09-21T10:40:30.419182+0000","last_unstale":"2024-09-21T10:41:14.128469+0000","last_undegraded":"2024-09-21T10:41:14.128469+0000","last_fullsized":"2024-09-21T10:41:14.128469+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:38:19.185388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1310,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.123438+0000","last_change":"2024-09-21T10:40:30.422480+0000","last_active":"2024-09-21T10:41:13.123438+0000","last_peered":"2024-09-21T10:41:13.123438+0000","last_clean":"2024-09-21T10:41:13.123438+0000","last_became_active":"2024-09-21T10:40:30.421818+0000","last_became_peered":"2024-09-21T10:40:30.421818+0000","last_unstale":"2024-09-21T10:41:13.123438+0000","last_undegraded":"2024-09-21T10:41:13.123438+0000","last_fullsized":"2024-09-21T10:41:13.123438+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:51:14.217028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":51,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.121905+0000","last_change":"2024-09-21T10:40:30.440162+0000","last_active":"2024-09-21T10:41:13.121905+0000","last_peered":"2024-09-21T10:41:13.121905+0000","last_clean":"2024-09-21T10:41:13.121905+0000","last_became_active":"2024-09-21T10:40:30.427218+0000","last_became_peered":"2024-09-21T10:40:30.427218+0000","last_unstale":"2024-09-21T10:41:13.121905+0000","last_undegraded":"2024-09-21T10:41:13.121905+0000","last_fullsized":"2024-09-21T10:41:13.121905+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:25:22.251409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1947,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.120688+0000","last_change":"2024-09-21T10:40:30.418759+0000","last_active":"2024-09-21T10:41:14.120688+0000","last_peered":"2024-09-21T10:41:14.120688+0000","last_clean":"2024-09-21T10:41:14.120688+0000","last_became_active":"2024-09-21T10:40:30.418325+0000","last_became_peered":"2024-09-21T10:40:30.418325+0000","last_unstale":"2024-09-21T10:41:14.120688+0000","last_undegraded":"2024-09-21T10:41:14.120688+0000","last_fullsized":"2024-09-21T10:41:14.120688+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:57.793473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":53,"num_pg":26,"stat_sum":{"num_bytes":6139871232,"num_objects":93688,"num_object_clones":0,"num_object_copies":281064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12500,"num_objects_misplaced":5339,"num_objects_unfound":0,"num_objects_dirty":93688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65620,"num_write_kb":4199616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3299,"num_bytes_recovered":216203264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18589417472,"data_stored":18589417472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135841,"ondisk_log_size":135841,"up":78,"acting":74,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":5554,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":21634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234568,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496205,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2124128,"kb_used_data":1940504,"kb_used_omap":1,"kb_used_meta":183614,"kb_avail":91621024,"statfs":{"total":95995035648,"available":93819928576,"internally_reserved":0,"allocated":1987076096,"data_stored":1983094448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":188021194},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757842,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2706188,"kb_used_data":2544644,"kb_used_omap":1,"kb_used_meta":161534,"kb_avail":91038964,"statfs":{"total":95995035648,"available":93223899136,"internally_reserved":0,"allocated":2605715456,"data_stored":2601748954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":165411270},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019478,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4499608,"kb_used_data":4251984,"kb_used_omap":1,"kb_used_meta":247614,"kb_avail":89245544,"statfs":{"total":95995035648,"available":91387437056,"internally_reserved":0,"allocated":4354031616,"data_stored":4350057057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":253557190},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369222,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2485512,"kb_used_data":2367296,"kb_used_omap":69,"kb_used_meta":118138,"kb_avail":91259640,"statfs":{"total":95995035648,"available":93449871360,"internally_reserved":0,"allocated":2424111104,"data_stored":2420148199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":120973442},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":2920577761677,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807957908,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4380492,"kb_used_data":4208260,"kb_used_omap":66,"kb_used_meta":172157,"kb_avail":89364660,"statfs":{"total":95995035648,"available":91509411840,"internally_reserved":0,"allocated":4309258240,"data_stored":4305291664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68589,"internal_metadata":176288787},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":666,"seq":2860448219547,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2869600,"kb_used_data":2707288,"kb_used_omap":76,"kb_used_meta":162227,"kb_avail":90875552,"statfs":{"total":95995035648,"available":93056565248,"internally_reserved":0,"allocated":2772262912,"data_stored":2768281299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78716,"internal_metadata":166120580},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2780037120,"data_stored":2780037120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4311285760,"data_stored":4311285760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2443509760,"data_stored":2443509760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4392288256,"data_stored":4392288256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2600992768,"data_stored":2600992768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1980366848,"data_stored":1980366848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":80936960,"data_stored":80936960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:41:26.870 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pgp_num 1 2024-09-21T10:41:26.894 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:26.921 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:26.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:26.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:27.022 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:27.051 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:27.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:27.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:27.124 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 63 16 98717 98701 97.895 82.1875 0.00935269 0.0102095 2024-09-21T10:41:27.153 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:27.185 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:27.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:27.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:27.287 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:27.318 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:27.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:27.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:27.419 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:27.453 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:27.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:27.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:27.540 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:27 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:41:27.540 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:27 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:41:27.540 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:27 smithi070 ceph-mon[107385]: osdmap e1335: 8 total, 8 up, 6 in 2024-09-21T10:41:27.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:27 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:41:27.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:27 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:41:27.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:27 smithi070 ceph-mon[112139]: osdmap e1335: 8 total, 8 up, 6 in 2024-09-21T10:41:27.555 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:27.587 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:27.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:27.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:27.689 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:27.724 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:27.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:27.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:27.826 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:27 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:41:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:27 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:41:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:27 smithi160 ceph-mon[101879]: osdmap e1335: 8 total, 8 up, 6 in 2024-09-21T10:41:27.850 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:27.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:27.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:27.951 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:27.973 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:27.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:27.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:28.072 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:41:28.073 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:41:28.075 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:28.096 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:28.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:28.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:28.124 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 64 16 99675 99659 97.301 59.875 0.0035744 0.0102082 2024-09-21T10:41:28.199 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:28.222 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:28.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:28.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:28.324 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:28.347 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:28.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:28.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:28.449 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:28.480 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:28.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:28.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:28.581 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:28.614 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:28.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:28.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:28.711 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:28 smithi070 ceph-mon[107385]: pgmap v2212: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 85 MiB/s wr, 1.35k op/s; 12500/288885 objects degraded (4.327%); 5339/288885 objects misplaced (1.848%); 6.8 MiB/s, 108 objects/s recovering 2024-09-21T10:41:28.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:28 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 12500/288885 objects degraded (4.327%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:41:28.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:28 smithi070 ceph-mon[107385]: osdmap e1336: 8 total, 8 up, 6 in 2024-09-21T10:41:28.713 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:28 smithi070 ceph-mon[112139]: pgmap v2212: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 85 MiB/s wr, 1.35k op/s; 12500/288885 objects degraded (4.327%); 5339/288885 objects misplaced (1.848%); 6.8 MiB/s, 108 objects/s recovering 2024-09-21T10:41:28.713 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:28 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 12500/288885 objects degraded (4.327%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:41:28.713 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:28 smithi070 ceph-mon[112139]: osdmap e1336: 8 total, 8 up, 6 in 2024-09-21T10:41:28.716 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:28.741 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:28.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:28.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:28.754 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:41:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:28 smithi160 ceph-mon[101879]: pgmap v2212: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 85 MiB/s wr, 1.35k op/s; 12500/288885 objects degraded (4.327%); 5339/288885 objects misplaced (1.848%); 6.8 MiB/s, 108 objects/s recovering 2024-09-21T10:41:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:28 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 12500/288885 objects degraded (4.327%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:41:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:28 smithi160 ceph-mon[101879]: osdmap e1336: 8 total, 8 up, 6 in 2024-09-21T10:41:28.842 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:28.891 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:28.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:28.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:28.995 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:29.025 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:29.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:29.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:29.124 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 65 16 100638 100622 96.73 60.1875 0.0123633 0.0103325 2024-09-21T10:41:29.127 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:29.155 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:29.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:29.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:29.257 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:29.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:29.297 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:29.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:29.398 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:29.437 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:29.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:29.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:29.538 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:29.575 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:29.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:29.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:29.604 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:29.677 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:29.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:29.704 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:29.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:29.747 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:29 smithi070 ceph-mon[112139]: osdmap e1337: 8 total, 8 up, 6 in 2024-09-21T10:41:29.747 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:29 smithi070 ceph-mon[107385]: osdmap e1337: 8 total, 8 up, 6 in 2024-09-21T10:41:29.805 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:29 smithi160 ceph-mon[101879]: osdmap e1337: 8 total, 8 up, 6 in 2024-09-21T10:41:29.849 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:29.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:29.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:29.951 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:29.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:29.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:30.099 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:30.125 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 66 16 101879 101863 96.4383 77.5625 0.0070753 0.0103642 2024-09-21T10:41:30.139 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:30.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:30.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:30.240 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:30.281 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:30.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:30.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:30.383 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:30.416 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:30.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:30.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:30.518 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:30.545 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:30.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:30.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:30.595 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:30 smithi070 ceph-mon[112139]: pgmap v2215: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 71 MiB/s wr, 1.13k op/s; 12500/288885 objects degraded (4.327%); 5339/288885 objects misplaced (1.848%); 6.3 MiB/s, 101 objects/s recovering 2024-09-21T10:41:30.597 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:30 smithi070 ceph-mon[107385]: pgmap v2215: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 71 MiB/s wr, 1.13k op/s; 12500/288885 objects degraded (4.327%); 5339/288885 objects misplaced (1.848%); 6.3 MiB/s, 101 objects/s recovering 2024-09-21T10:41:30.646 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:30.672 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:30.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:30.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:30.774 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:30.782 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:41:30.782 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:41:30.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:30.822 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:30.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:30.836 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:41:30.836 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:41:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:30 smithi160 ceph-mon[101879]: pgmap v2215: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 71 MiB/s wr, 1.13k op/s; 12500/288885 objects degraded (4.327%); 5339/288885 objects misplaced (1.848%); 6.3 MiB/s, 101 objects/s recovering 2024-09-21T10:41:30.867 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:41:30.868 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:41:30.906 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:41:30.906 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:41:30.923 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:30.935 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:41:30.935 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:41:30.949 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:30.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:30.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:30.976 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:41:30.976 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:41:31.008 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:41:31.009 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:41:31.042 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:41:31.043 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:41:31.051 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:31.095 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:31.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:31.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:31.125 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 67 16 103116 103100 96.1528 77.3125 0.0254577 0.010395 2024-09-21T10:41:31.183 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:41:31.197 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:31.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:31.273 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:31.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:31.295 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:31.374 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:31.405 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:31.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:31.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:31.507 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:31.553 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:31.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:31.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:31.613 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:31 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/269947777' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:41:31.613 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:31 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/269947777' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:41:31.654 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:31.692 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:31.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:31.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:31.794 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:31.830 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:31.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:31.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:31 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/269947777' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:41:31.931 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:31.967 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:31.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:31.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:32.069 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:32.103 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:32.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:32.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:32.125 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 68 16 104376 104360 95.8968 78.75 0.015102 0.0104231 2024-09-21T10:41:32.206 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:32.246 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:32.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:32.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:32.348 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:32.382 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:32.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:32.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:32.484 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:32.513 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:32.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:32.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:32.615 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:32.647 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:32.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:32.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:32.720 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:32 smithi070 ceph-mon[107385]: pgmap v2216: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.1 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 60 MiB/s wr, 963 op/s; 7460/298491 objects degraded (2.499%); 9786/298491 objects misplaced (3.278%); 4.5 MiB/s, 71 objects/s recovering 2024-09-21T10:41:32.720 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:32 smithi070 ceph-mon[112139]: pgmap v2216: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.1 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 60 MiB/s wr, 963 op/s; 7460/298491 objects degraded (2.499%); 9786/298491 objects misplaced (3.278%); 4.5 MiB/s, 71 objects/s recovering 2024-09-21T10:41:32.749 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:32.800 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:32.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:32.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:32 smithi160 ceph-mon[101879]: pgmap v2216: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.1 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 60 MiB/s wr, 963 op/s; 7460/298491 objects degraded (2.499%); 9786/298491 objects misplaced (3.278%); 4.5 MiB/s, 71 objects/s recovering 2024-09-21T10:41:32.902 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:32.943 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:32.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:32.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:33.045 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:33.090 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:33.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:33.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:33.126 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 69 16 105632 105616 95.6443 78.5 0.00803299 0.0104506 2024-09-21T10:41:33.193 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:33.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:33.234 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:33.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:33.335 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:33.339 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:41:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:41:33] "GET /metrics HTTP/1.1" 200 36625 "" "Prometheus/2.51.0" 2024-09-21T10:41:33.388 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:33.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:33.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:33.490 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:33.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:33.534 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:33.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:33.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:33 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 7460/298491 objects degraded (2.499%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:41:33.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:33 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 7460/298491 objects degraded (2.499%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:41:33.635 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:33.660 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:33.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:33.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:33.762 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:33.790 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:33.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:33.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:33 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 7460/298491 objects degraded (2.499%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:41:33.892 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:33.932 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:33.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:33.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:34.010 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:41:34.035 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:34.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:34.119 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:34.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:34.127 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 70 16 106748 106732 95.273 69.75 0.00854678 0.0104909 2024-09-21T10:41:34.220 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:34.266 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:34.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:34.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:34.367 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:34.393 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:34.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:34.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:34.471 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:34 smithi070 ceph-mon[107385]: pgmap v2217: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.2 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 55 MiB/s wr, 878 op/s; 7460/305424 objects degraded (2.443%); 9786/305424 objects misplaced (3.204%); 4.3 MiB/s, 68 objects/s recovering 2024-09-21T10:41:34.471 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:34 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3001270889' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-21T10:41:34.472 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:34 smithi070 ceph-mon[112139]: pgmap v2217: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.2 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 55 MiB/s wr, 878 op/s; 7460/305424 objects degraded (2.443%); 9786/305424 objects misplaced (3.204%); 4.3 MiB/s, 68 objects/s recovering 2024-09-21T10:41:34.472 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:34 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3001270889' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-21T10:41:34.472 INFO:teuthology.orchestra.run.smithi070.stderr:set pool 1 pgp_num to 1 2024-09-21T10:41:34.495 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:34.523 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:34.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:34.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:34.625 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:34.666 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:34.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:34.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:34.737 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:34 smithi160 ceph-mon[101879]: pgmap v2217: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.2 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 55 MiB/s wr, 878 op/s; 7460/305424 objects degraded (2.443%); 9786/305424 objects misplaced (3.204%); 4.3 MiB/s, 68 objects/s recovering 2024-09-21T10:41:34.737 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:34 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3001270889' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-21T10:41:34.768 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:34.790 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:34.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:34.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:34.891 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:34.915 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:41:34.915 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T10:41:34.915 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T10:41:34.915 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T10:41:34.915 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:41:34.933 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:34.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:34.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:35.035 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:35.067 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:35.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:35.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:35.127 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 71 16 107813 107797 94.8686 66.5625 0.0210434 0.0105358 2024-09-21T10:41:35.169 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:35.213 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:35.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:35.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:35.315 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:35.346 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:35.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:35.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:35.448 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:35.495 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:35.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:35.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:35.597 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:35.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:35 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3001270889' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-21T10:41:35.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:35 smithi070 ceph-mon[107385]: osdmap e1338: 8 total, 8 up, 6 in 2024-09-21T10:41:35.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:35 smithi070 ceph-mon[107385]: pgmap v2219: 99 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 79 MiB/s wr, 1.26k op/s; 7460/316386 objects degraded (2.358%); 9355/316386 objects misplaced (2.957%); 5.0 MiB/s, 79 objects/s recovering 2024-09-21T10:41:35.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:41:35.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:35 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3001270889' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-21T10:41:35.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:35 smithi070 ceph-mon[112139]: osdmap e1338: 8 total, 8 up, 6 in 2024-09-21T10:41:35.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:35 smithi070 ceph-mon[112139]: pgmap v2219: 99 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 79 MiB/s wr, 1.26k op/s; 7460/316386 objects degraded (2.358%); 9355/316386 objects misplaced (2.957%); 5.0 MiB/s, 79 objects/s recovering 2024-09-21T10:41:35.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:41:35.637 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:35.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:35.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:35.688 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:41:35.740 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:35.766 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:35.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:35.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:35 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3001270889' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-21T10:41:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:35 smithi160 ceph-mon[101879]: osdmap e1338: 8 total, 8 up, 6 in 2024-09-21T10:41:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:35 smithi160 ceph-mon[101879]: pgmap v2219: 99 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 79 MiB/s wr, 1.26k op/s; 7460/316386 objects degraded (2.358%); 9355/316386 objects misplaced (2.957%); 5.0 MiB/s, 79 objects/s recovering 2024-09-21T10:41:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:41:35.869 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:35.890 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:35.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:35.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:35.992 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:36.020 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:36.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:36.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:36.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:36.127 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 72 16 108954 108938 94.5414 71.3125 0.016216 0.0105725 2024-09-21T10:41:36.152 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:36.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:36.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:36.254 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:36.284 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:36.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:36.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:36.386 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:36.414 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:36.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:36.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:36.516 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:36.559 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:36.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:36.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:36.661 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:36.700 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:36.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:36.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:36.802 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:36.833 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:36.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:36.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:36.934 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:36.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:36.970 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:36.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:37.071 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:37.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:37.109 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:37.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:37.127 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 73 16 110074 110058 94.2052 70 0.0118292 0.0106094 2024-09-21T10:41:37.209 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:37.245 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:37.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:37.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:37.347 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:37.378 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:37.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:37.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:37.479 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:37.515 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:37.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:37.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:37.616 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:37.648 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:37.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:37.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:37.750 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:37.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:37.787 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:37.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:37.888 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:37.909 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:37.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:37.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:38.010 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:38.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:38.043 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:38.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:38.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:37 smithi160 ceph-mon[101879]: pgmap v2220: 99 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 87 MiB/s wr, 1.39k op/s; 7460/323433 objects degraded (2.307%); 9355/323433 objects misplaced (2.892%); 4.3 MiB/s, 69 objects/s recovering 2024-09-21T10:41:38.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:37 smithi070 ceph-mon[112139]: pgmap v2220: 99 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 87 MiB/s wr, 1.39k op/s; 7460/323433 objects degraded (2.307%); 9355/323433 objects misplaced (2.892%); 4.3 MiB/s, 69 objects/s recovering 2024-09-21T10:41:38.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:37 smithi070 ceph-mon[107385]: pgmap v2220: 99 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 87 MiB/s wr, 1.39k op/s; 7460/323433 objects degraded (2.307%); 9355/323433 objects misplaced (2.892%); 4.3 MiB/s, 69 objects/s recovering 2024-09-21T10:41:38.127 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 74 16 111294 111278 93.9626 76.25 0.00839684 0.0106376 2024-09-21T10:41:38.144 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:38.177 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:38.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:38.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:38.278 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:38.319 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:38.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:38.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:38.421 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:38.449 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:38.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:38.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:38.551 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:38.583 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:38.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:38.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:38.684 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:38.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:38.715 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:38.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:38.816 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:38.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:38.853 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:38.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:38.954 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:38.976 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:38.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:38.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:39.078 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:38 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 7460/323433 objects degraded (2.307%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:41:39.112 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:39.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:39.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:39.117 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:38 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 7460/323433 objects degraded (2.307%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:41:39.119 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:38 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 7460/323433 objects degraded (2.307%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:41:39.124 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:39.127 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 75 16 112532 112516 93.7414 77.375 0.0140624 0.0106624 2024-09-21T10:41:39.213 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:39.243 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:39.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:39.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:39.345 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:39.381 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:39.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:39.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:39.443 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:39.482 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:39.509 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:39.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:39.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:39.611 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:39.642 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:39.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:39.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:39.743 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:39.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:39.787 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:39.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:39.888 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:39.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:39.929 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:39.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:40.030 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:40.057 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:39 smithi070 ceph-mon[107385]: pgmap v2221: 99 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 72 MiB/s wr, 1.15k op/s; 7460/323433 objects degraded (2.307%); 9355/323433 objects misplaced (2.892%); 3.6 MiB/s, 57 objects/s recovering 2024-09-21T10:41:40.057 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:39 smithi070 ceph-mon[112139]: pgmap v2221: 99 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 72 MiB/s wr, 1.15k op/s; 7460/323433 objects degraded (2.307%); 9355/323433 objects misplaced (2.892%); 3.6 MiB/s, 57 objects/s recovering 2024-09-21T10:41:40.057 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:40.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:40.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:40.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:39 smithi160 ceph-mon[101879]: pgmap v2221: 99 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 72 MiB/s wr, 1.15k op/s; 7460/323433 objects degraded (2.307%); 9355/323433 objects misplaced (2.892%); 3.6 MiB/s, 57 objects/s recovering 2024-09-21T10:41:40.127 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 76 16 113743 113727 93.5037 75.6875 0.0306085 0.0106896 2024-09-21T10:41:40.160 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:40.199 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:40.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:40.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:40.301 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:40.336 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:40.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:40.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:40.437 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:40.477 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:40.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:40.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:40.579 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:40.614 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:40.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:40.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:40.716 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:40.743 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:40.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:40.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:40.845 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:40.876 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:40.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:40.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:40.978 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:41.021 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:41.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:41.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:41.123 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:41.127 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 77 16 114901 114885 93.2292 72.375 0.0370491 0.0107216 2024-09-21T10:41:41.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:41.164 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:41.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:41.265 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:41.306 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:41.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:41.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:41.408 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:41.442 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:41.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:41.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:41.544 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:41.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:41.589 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:41.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:41.690 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:41.726 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:41.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:41.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:41.828 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:41.854 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:41.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:41.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:41.956 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:41.987 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:41.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:41.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:42.089 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:42.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:41 smithi160 ceph-mon[101879]: pgmap v2222: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.8 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 74 MiB/s wr, 1.18k op/s; 7460/333873 objects degraded (2.234%); 8938/333873 objects misplaced (2.677%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:41:42.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:41 smithi070 ceph-mon[107385]: pgmap v2222: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.8 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 74 MiB/s wr, 1.18k op/s; 7460/333873 objects degraded (2.234%); 8938/333873 objects misplaced (2.677%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:41:42.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:41 smithi070 ceph-mon[112139]: pgmap v2222: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.8 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 74 MiB/s wr, 1.18k op/s; 7460/333873 objects degraded (2.234%); 8938/333873 objects misplaced (2.677%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:41:42.128 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:42.128 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 78 16 116065 116049 92.9666 72.75 0.00795775 0.0107513 2024-09-21T10:41:42.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:42.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:42.230 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:42.263 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:42.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:42.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:42.364 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:42.397 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:42.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:42.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:42.499 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:42.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:42.537 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:42.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:42.638 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:42.677 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:42.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:42.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:42.779 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:42.822 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:42.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:42.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:42.924 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:42.948 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:42.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:42.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:43.049 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:43.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:43.083 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:43.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:43.127 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 79 16 117130 117114 92.6323 66.5625 0.0197791 0.0107907 2024-09-21T10:41:43.184 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:43.212 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:43.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:43.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:43.313 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:43.342 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:43.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:43.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:43.444 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:43.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:43.485 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:43.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:43.585 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:43.622 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:43.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:43.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:41:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:41:43] "GET /metrics HTTP/1.1" 200 36620 "" "Prometheus/2.51.0" 2024-09-21T10:41:43.723 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:43.747 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:43.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:43.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:43.849 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:43.887 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:43.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:43.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:43.988 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:44.030 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:44.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:44.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:43 smithi160 ceph-mon[101879]: pgmap v2223: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 74 MiB/s wr, 1.19k op/s; 7460/341103 objects degraded (2.187%); 8938/341103 objects misplaced (2.620%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:41:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:43 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 7460/341103 objects degraded (2.187%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:41:44.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:43 smithi070 ceph-mon[107385]: pgmap v2223: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 74 MiB/s wr, 1.19k op/s; 7460/341103 objects degraded (2.187%); 8938/341103 objects misplaced (2.620%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:41:44.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:43 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 7460/341103 objects degraded (2.187%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:41:44.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:43 smithi070 ceph-mon[112139]: pgmap v2223: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 74 MiB/s wr, 1.19k op/s; 7460/341103 objects degraded (2.187%); 8938/341103 objects misplaced (2.620%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:41:44.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:43 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 7460/341103 objects degraded (2.187%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:41:44.128 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:2024-09-21T10:41:44.127006+0000 min lat: 0.00114886 max lat: 2.28954 avg lat: 0.0108227 2024-09-21T10:41:44.128 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:41:44.128 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 80 16 118254 118238 92.3525 70.25 0.00565667 0.0108227 2024-09-21T10:41:44.131 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:44.166 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6] out_osds: [2, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:41:44.166 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:41:44.166 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-09-21T10:41:44.167 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:41:44.171 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:44.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:44.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:44.273 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:44.298 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:44.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:44.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:44.400 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:44.433 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:44.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:44.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:44.534 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:44.568 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:44.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:44.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:44.669 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:44.707 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:44.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:44.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:44.809 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:44.844 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:44.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:44.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:44.946 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:44.975 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:44.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:44.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:45.077 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:45.109 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:45.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:45.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:45.128 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 81 16 119378 119362 92.0796 70.25 0.00585276 0.0108553 2024-09-21T10:41:45.211 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:45.251 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:45.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:45.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:45.353 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:45.392 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:45.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:45.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:45.493 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:45.514 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:45.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:45.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:45.616 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:45.655 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:45.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:45.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:45.677 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:45.758 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:45.782 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:45.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:45.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:45.884 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:45.920 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:45.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:45.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:46.022 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:46.049 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:46.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:46.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:46.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:45 smithi160 ceph-mon[101879]: pgmap v2224: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.1 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 80 MiB/s wr, 1.27k op/s; 7460/351378 objects degraded (2.123%); 8497/351378 objects misplaced (2.418%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:41:46.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:45 smithi070 ceph-mon[112139]: pgmap v2224: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.1 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 80 MiB/s wr, 1.27k op/s; 7460/351378 objects degraded (2.123%); 8497/351378 objects misplaced (2.418%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:41:46.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:45 smithi070 ceph-mon[107385]: pgmap v2224: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.1 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 80 MiB/s wr, 1.27k op/s; 7460/351378 objects degraded (2.123%); 8497/351378 objects misplaced (2.418%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:41:46.128 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 82 16 120339 120323 91.6891 60.0625 0.00247178 0.0108932 2024-09-21T10:41:46.151 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:46.181 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:46.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:46.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:46.282 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:46.316 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:46.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:46.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:46.418 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:46.456 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:46.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:46.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:46.557 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:46.593 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:46.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:46.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:46.695 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:46.734 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:46.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:46.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:46.836 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:46.858 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:46.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:46.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:46.960 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:46.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:46.991 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:46.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:47.092 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:47.119 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:47.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:47.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:47.128 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 83 16 121402 121386 91.3849 66.4375 0.0158747 0.0109376 2024-09-21T10:41:47.220 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:47.241 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:47.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:47.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:47.343 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:47.368 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:47.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:47.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:47.469 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:47.509 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:47.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:47.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:47.611 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:47.649 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:47.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:47.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:47.751 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:47.786 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:47.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:47.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:47.888 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:47.916 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:47.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:47.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:48.018 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:48.049 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:48.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:48.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:47 smithi160 ceph-mon[101879]: pgmap v2225: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.3 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 72 MiB/s wr, 1.15k op/s; 7460/357942 objects degraded (2.084%); 8497/357942 objects misplaced (2.374%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:41:48.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:47 smithi070 ceph-mon[107385]: pgmap v2225: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.3 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 72 MiB/s wr, 1.15k op/s; 7460/357942 objects degraded (2.084%); 8497/357942 objects misplaced (2.374%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:41:48.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:47 smithi070 ceph-mon[112139]: pgmap v2225: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.3 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 72 MiB/s wr, 1.15k op/s; 7460/357942 objects degraded (2.084%); 8497/357942 objects misplaced (2.374%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:41:48.128 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 84 16 122425 122409 91.0581 63.9375 0.00522901 0.0109701 2024-09-21T10:41:48.150 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:48.181 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:48.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:48.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:48.283 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:48.317 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:48.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:48.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:48.419 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:48.444 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:48.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:48.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:48.545 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:48.567 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:48.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:48.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:48.668 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:48.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:48.698 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:48.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:48.799 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:48.829 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:48.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:48.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:48.930 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:48.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:48.971 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:48.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:49.071 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:49.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:48 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 7460/357942 objects degraded (2.084%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:41:49.115 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:49.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:49.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:49.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:48 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 7460/357942 objects degraded (2.084%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:41:49.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:48 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 7460/357942 objects degraded (2.084%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:41:49.128 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 85 16 123447 123431 90.7382 63.875 0.00844889 0.0110155 2024-09-21T10:41:49.218 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:49.244 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:49.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:49.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:49.346 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:49.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:49.391 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:49.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:49.492 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:49.536 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:49.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:49.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:49.637 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:49.674 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:49.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:49.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:49.776 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:49.828 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:49.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:49.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:49.930 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:49.951 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:49 smithi070 ceph-mon[107385]: pgmap v2226: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.3 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 60 MiB/s wr, 958 op/s; 7460/357942 objects degraded (2.084%); 8497/357942 objects misplaced (2.374%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:41:49.951 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:49 smithi070 ceph-mon[112139]: pgmap v2226: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.3 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 60 MiB/s wr, 958 op/s; 7460/357942 objects degraded (2.084%); 8497/357942 objects misplaced (2.374%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:41:49.959 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:49.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:49.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:50.061 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:50.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:49 smithi160 ceph-mon[101879]: pgmap v2226: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.3 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 60 MiB/s wr, 958 op/s; 7460/357942 objects degraded (2.084%); 8497/357942 objects misplaced (2.374%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:41:50.107 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:50.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:50.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:50.128 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 86 16 124571 124555 90.5 70.25 0.0061234 0.0110446 2024-09-21T10:41:50.209 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:50.238 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:50.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:50.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:50.340 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:50.376 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:50.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:50.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:50.477 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:50.520 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:50.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:50.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:50.622 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:50.650 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:50.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:50.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:50.752 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:50.785 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:50.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:50.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:50.887 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:50.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:50.938 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:50.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:51.013 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:41:51.014 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:41:51.039 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:51.057 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:41:51.057 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:41:51.057 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:41:51.057 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:41:51.076 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:51.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:51.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:51.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:41:51.096 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:41:51.097 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:41:51.128 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 87 16 125685 125669 90.26 69.625 0.00865329 0.0110742 2024-09-21T10:41:51.133 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:41:51.133 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:41:51.168 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:41:51.168 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:41:51.178 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:51.201 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:41:51.201 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:41:51.214 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:51.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:51.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:51.236 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:41:51.236 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:41:51.272 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:41:51.272 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:41:51.316 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:51.351 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:51.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:51.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:51.464 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:51.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:51.506 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:51.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:51.607 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:51.647 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:51.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:51.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:51.748 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:51.787 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:51.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:51.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:51.889 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:51.920 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:51.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:51.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:52.022 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:52.047 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:52.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:52.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:52.060 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:51 smithi070 ceph-mon[107385]: pgmap v2227: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.5 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 76 MiB/s wr, 1.22k op/s; 7460/367389 objects degraded (2.031%); 8083/367389 objects misplaced (2.200%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:41:52.060 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:51 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2970964919' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:41:52.060 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:51 smithi070 ceph-mon[112139]: pgmap v2227: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.5 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 76 MiB/s wr, 1.22k op/s; 7460/367389 objects degraded (2.031%); 8083/367389 objects misplaced (2.200%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:41:52.061 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:51 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2970964919' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:41:52.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:51 smithi160 ceph-mon[101879]: pgmap v2227: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.5 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 76 MiB/s wr, 1.22k op/s; 7460/367389 objects degraded (2.031%); 8083/367389 objects misplaced (2.200%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:41:52.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:51 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2970964919' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:41:52.128 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 88 16 126791 126775 90.0198 69.125 0.0236185 0.0111038 2024-09-21T10:41:52.149 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:52.178 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:52.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:52.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:52.280 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:52.303 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:52.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:52.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:52.405 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:52.449 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:52.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:52.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:52.551 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:52.575 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:52.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:52.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:52.677 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:52.710 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:52.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:52.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:52.811 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T10:41:52.812 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:52.836 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:52 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1745001270' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:41:52.836 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:52 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1745001270' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:41:52.837 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:52.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:52.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:52.939 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:52.969 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:52.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:52.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:53.071 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:53.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:52 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1745001270' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:41:53.106 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:53.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:53.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:53.128 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 89 16 127871 127855 89.7667 67.5 0.0113139 0.0111353 2024-09-21T10:41:53.207 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:53.237 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:53.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:53.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:53.339 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:53.375 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:53.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:53.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:53.477 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:53.512 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:53.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:53.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:53.597 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:41:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:41:53] "GET /metrics HTTP/1.1" 200 36621 "" "Prometheus/2.51.0" 2024-09-21T10:41:53.615 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:53.657 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:53.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:53.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:53.759 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:53.788 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:53.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:53.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:53.848 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:53 smithi070 ceph-mon[107385]: pgmap v2228: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.6 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 69 MiB/s wr, 1.10k op/s; 7460/373644 objects degraded (1.997%); 8083/373644 objects misplaced (2.163%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:41:53.848 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:53 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:41:53.848 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:53 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1745001270' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:41:53.849 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:53 smithi070 ceph-mon[107385]: osdmap e1339: 8 total, 8 up, 6 in 2024-09-21T10:41:53.849 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:53 smithi070 ceph-mon[107385]: 53.18 scrub starts 2024-09-21T10:41:53.849 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:53 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 7460/373644 objects degraded (1.997%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:41:53.849 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:53 smithi070 ceph-mon[112139]: pgmap v2228: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.6 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 69 MiB/s wr, 1.10k op/s; 7460/373644 objects degraded (1.997%); 8083/373644 objects misplaced (2.163%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:41:53.849 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:53 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:41:53.849 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:53 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1745001270' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:41:53.849 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:53 smithi070 ceph-mon[112139]: osdmap e1339: 8 total, 8 up, 6 in 2024-09-21T10:41:53.849 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:53 smithi070 ceph-mon[112139]: 53.18 scrub starts 2024-09-21T10:41:53.849 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:53 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 7460/373644 objects degraded (1.997%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:41:53.890 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:53.905 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:41:53.905 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:41:53.915 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:53.921 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:53.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:53.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:54.023 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:54.051 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:54.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:54.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:53 smithi160 ceph-mon[101879]: pgmap v2228: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.6 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 69 MiB/s wr, 1.10k op/s; 7460/373644 objects degraded (1.997%); 8083/373644 objects misplaced (2.163%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:41:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:53 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:41:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:53 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1745001270' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:41:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:53 smithi160 ceph-mon[101879]: osdmap e1339: 8 total, 8 up, 6 in 2024-09-21T10:41:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:53 smithi160 ceph-mon[101879]: 53.18 scrub starts 2024-09-21T10:41:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:53 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 7460/373644 objects degraded (1.997%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:41:54.128 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 90 16 128917 128901 89.4956 65.375 0.0025459 0.0111646 2024-09-21T10:41:54.153 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:54.181 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:54.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:54.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:54.282 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:54.303 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:54.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:54.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:54.405 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:54.426 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:54.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:54.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:54.527 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:54.550 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:54.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:54.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:54.652 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:54.673 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:54.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:54.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:54.774 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:54.795 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:54.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:54.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:54.897 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:54.920 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:54.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:54.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:54.948 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:41:54.952 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:41:55.014 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:54 smithi070 ceph-mon[107385]: 53.18 scrub ok 2024-09-21T10:41:55.015 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:54 smithi070 ceph-mon[112139]: 53.18 scrub ok 2024-09-21T10:41:55.021 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:55.080 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:55.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:55.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:55.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:54 smithi160 ceph-mon[101879]: 53.18 scrub ok 2024-09-21T10:41:55.129 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Total time run: 90.0636 2024-09-21T10:41:55.129 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Total writes made: 128917 2024-09-21T10:41:55.129 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Write size: 65536 2024-09-21T10:41:55.129 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Object size: 65536 2024-09-21T10:41:55.129 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Bandwidth (MB/sec): 89.4625 2024-09-21T10:41:55.129 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Stddev Bandwidth: 44.529 2024-09-21T10:41:55.129 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Max bandwidth (MB/sec): 226.25 2024-09-21T10:41:55.129 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Min bandwidth (MB/sec): 0 2024-09-21T10:41:55.129 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Average IOPS: 1431 2024-09-21T10:41:55.129 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Stddev IOPS: 712.465 2024-09-21T10:41:55.129 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Max IOPS: 3620 2024-09-21T10:41:55.130 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Min IOPS: 0 2024-09-21T10:41:55.130 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Average Latency(s): 0.0111732 2024-09-21T10:41:55.130 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Stddev Latency(s): 0.0469247 2024-09-21T10:41:55.130 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Max latency(s): 2.28954 2024-09-21T10:41:55.130 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Min latency(s): 0.00114886 2024-09-21T10:41:55.133 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T10:41:55.182 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:55.207 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:55.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:55.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:55.308 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:55.333 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:55.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:55.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:55.434 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:55.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:55.476 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:55.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:55.576 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:55.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:55.616 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:55.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:55.636 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:41:55.716 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:55.800 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:55.801 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:55.803 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:55.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:55.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:55.905 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:55.937 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:55.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:55.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:55.992 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:55 smithi070 ceph-mon[107385]: pgmap v2230: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.8 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 66 MiB/s wr, 1.06k op/s; 7460/383175 objects degraded (1.947%); 7612/383175 objects misplaced (1.987%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T10:41:55.992 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:55 smithi070 ceph-mon[112139]: pgmap v2230: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.8 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 66 MiB/s wr, 1.06k op/s; 7460/383175 objects degraded (1.947%); 7612/383175 objects misplaced (1.987%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T10:41:56.039 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:56.075 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:56.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:56.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:56.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:55 smithi160 ceph-mon[101879]: pgmap v2230: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.8 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 66 MiB/s wr, 1.06k op/s; 7460/383175 objects degraded (1.947%); 7612/383175 objects misplaced (1.987%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T10:41:56.177 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:56.204 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:56.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:56.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:56.306 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:56.334 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:56.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:56.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:56.436 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:56.463 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:56.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:56.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:56.566 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:56.604 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:56.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:56.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:56.706 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:56.739 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:56.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:56.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:56.841 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:56.875 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:56.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:56.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:56.977 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:57.005 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:57.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:57.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:57.107 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:57.136 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:57.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:57.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:57.237 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:57.280 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:57.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:57.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:57.381 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:57.427 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:57.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:57.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:57.529 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:57.571 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:57.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:57.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:57.673 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:57.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:57.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:57.810 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:57.836 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:57.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:57.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:57.938 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:57.967 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:57.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:57.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:57.987 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:57 smithi070 ceph-mon[107385]: pgmap v2231: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.8 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 35 MiB/s wr, 602 op/s; 7460/385812 objects degraded (1.934%); 7612/385812 objects misplaced (1.973%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T10:41:57.987 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:57 smithi070 ceph-mon[112139]: pgmap v2231: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.8 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 35 MiB/s wr, 602 op/s; 7460/385812 objects degraded (1.934%); 7612/385812 objects misplaced (1.973%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T10:41:58.069 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:57 smithi160 ceph-mon[101879]: pgmap v2231: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.8 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 35 MiB/s wr, 602 op/s; 7460/385812 objects degraded (1.934%); 7612/385812 objects misplaced (1.973%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T10:41:58.099 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:58.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:58.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:58.201 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:58.225 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:58.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:58.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:58.327 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:58.353 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:58.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:58.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:58.455 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:58.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:58.500 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:58.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:58.601 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:58.626 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:58.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:58.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:58.728 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:58.758 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:58.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:58.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:58.860 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:58.900 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:58.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:58.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:58.917 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:59.002 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:59.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:59.043 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:59.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:59.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:58 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 7460/385812 objects degraded (1.934%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:41:59.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:58 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 7460/385812 objects degraded (1.934%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:41:59.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:58 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 7460/385812 objects degraded (1.934%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:41:59.144 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:59.170 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:59.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:59.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:59.272 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:59.296 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:59.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:59.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:59.398 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:59.437 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:41:59.439 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T10:41:59.441 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:59.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:59.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:59.543 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:59.581 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:59.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:59.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:59.683 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:59.711 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:59.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:59.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:59.813 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:59.845 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:41:59.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:59.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:59.947 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:59.972 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:41:59.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:59.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:00.074 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:00.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:59 smithi160 ceph-mon[101879]: pgmap v2232: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.8 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 35 MiB/s wr, 602 op/s; 7460/385812 objects degraded (1.934%); 7612/385812 objects misplaced (1.973%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T10:42:00.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:41:59 smithi160 ceph-mon[101879]: from='client.45104 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:42:00.099 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:00.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:00.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:00.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:59 smithi070 ceph-mon[107385]: pgmap v2232: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.8 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 35 MiB/s wr, 602 op/s; 7460/385812 objects degraded (1.934%); 7612/385812 objects misplaced (1.973%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T10:42:00.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:41:59 smithi070 ceph-mon[107385]: from='client.45104 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:42:00.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:59 smithi070 ceph-mon[112139]: pgmap v2232: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.8 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 35 MiB/s wr, 602 op/s; 7460/385812 objects degraded (1.934%); 7612/385812 objects misplaced (1.973%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T10:42:00.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:41:59 smithi070 ceph-mon[112139]: from='client.45104 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:42:00.201 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:00.206 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:42:00.206 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:42:00.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:00.241 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:00.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:00.342 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:00.367 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:00.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:00.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:00.469 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:00.494 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:00.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:00.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:00.595 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:00.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:00.635 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:00.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:00.736 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:00.772 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:00.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:00.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:00.874 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:00.901 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:00.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:00.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:01.003 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:01.040 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:01.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:01.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:01.142 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:01.183 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:01.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:01.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:01.285 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:01.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:01.327 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:01.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:01.428 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:01.459 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:01.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:01.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:01.561 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:01.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:01.601 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:01.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:01.702 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:01.740 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:01.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:01.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:01.842 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:01.865 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:01.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:01.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:01.969 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:01.970 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:01 smithi070 ceph-mon[112139]: pgmap v2233: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.7 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 102 B/s rd, 16 MiB/s wr, 556 op/s; 7196/377862 objects degraded (1.904%); 6781/377862 objects misplaced (1.795%); 3.3 MiB/s, 52 objects/s recovering 2024-09-21T10:42:01.971 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:01 smithi070 ceph-mon[107385]: pgmap v2233: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.7 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 102 B/s rd, 16 MiB/s wr, 556 op/s; 7196/377862 objects degraded (1.904%); 6781/377862 objects misplaced (1.795%); 3.3 MiB/s, 52 objects/s recovering 2024-09-21T10:42:01.997 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:01.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:01.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:02.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:01 smithi160 ceph-mon[101879]: pgmap v2233: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.7 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 102 B/s rd, 16 MiB/s wr, 556 op/s; 7196/377862 objects degraded (1.904%); 6781/377862 objects misplaced (1.795%); 3.3 MiB/s, 52 objects/s recovering 2024-09-21T10:42:02.098 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:02.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:02.151 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:02.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:02.252 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:02.296 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:02.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:02.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:02.398 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:02.434 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:02.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:02.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:02.536 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:02.573 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:02.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:02.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:02.675 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:02.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:02.702 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:02.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:02.735 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:42:02.803 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:02.837 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:02.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:02.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:02.939 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:03.000 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:03.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:03.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:03.101 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:03.151 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:03.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:03.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:03.253 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:03.285 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:03.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:03.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:03.293 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:42:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:42:03] "GET /metrics HTTP/1.1" 200 36621 "" "Prometheus/2.51.0" 2024-09-21T10:42:03.376 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:42:03.377 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:42:03.387 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:03.419 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:42:03.419 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:42:03.428 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:03.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:03.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:03.453 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:42:03.453 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:42:03.485 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:42:03.485 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:42:03.518 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:42:03.518 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:42:03.531 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:03.554 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:42:03.555 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:42:03.576 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:03.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:03.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:03.591 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:42:03.591 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:42:03.623 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:42:03.623 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:42:03.677 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:03.708 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:03.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:03.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:03.810 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:03.852 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:03.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:03.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:03.953 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:03.997 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:03.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:03.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:04.098 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:04.126 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:04.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:04.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:04.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:03 smithi070 ceph-mon[107385]: pgmap v2234: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.6 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 102 B/s rd, 2.8 MiB/s wr, 569 op/s; 7196/371235 objects degraded (1.938%); 6781/371235 objects misplaced (1.827%); 3.3 MiB/s, 52 objects/s recovering 2024-09-21T10:42:04.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:03 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 7196/371235 objects degraded (1.938%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:42:04.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:03 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2589685675' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:42:04.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:03 smithi070 ceph-mon[112139]: pgmap v2234: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.6 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 102 B/s rd, 2.8 MiB/s wr, 569 op/s; 7196/371235 objects degraded (1.938%); 6781/371235 objects misplaced (1.827%); 3.3 MiB/s, 52 objects/s recovering 2024-09-21T10:42:04.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:03 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 7196/371235 objects degraded (1.938%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:42:04.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:03 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2589685675' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:42:04.227 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:04.263 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:04.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:04.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:04.294 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":2232,"stamp":"2024-09-21T10:41:58.688093+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8427050981,"num_objects":128604,"num_object_clones":0,"num_object_copies":385812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7460,"num_objects_misplaced":7612,"num_objects_unfound":0,"num_objects_dirty":128604,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":97529,"num_write_kb":6203057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4479,"num_bytes_recovered":294329906,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":164535,"ondisk_log_size":164535,"up":297,"acting":294,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":305,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":26548000,"kb_used_data":25278828,"kb_used_omap":218,"kb_used_meta":1268837,"kb_avail":535922912,"statfs":{"total":575970213888,"available":548785061888,"internally_reserved":0,"allocated":25885519872,"data_stored":25861456103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":223633,"internal_metadata":1299289711},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":608862208,"num_objects":9290,"num_object_clones":0,"num_object_copies":27870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-885,"num_objects_unfound":0,"num_objects_dirty":9290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6030,"num_write_kb":362592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":295,"num_bytes_recovered":19333120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003731"},"pg_stats":[{"pgid":"53.19","version":"1339'6320","reported_seq":6409,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:54.086956+0000","last_change":"2024-09-21T10:41:00.685085+0000","last_active":"2024-09-21T10:41:54.086956+0000","last_peered":"2024-09-21T10:41:54.086956+0000","last_clean":"2024-09-21T10:41:54.086956+0000","last_became_active":"2024-09-21T10:41:00.679759+0000","last_became_peered":"2024-09-21T10:41:00.679759+0000","last_unstale":"2024-09-21T10:41:54.086956+0000","last_undegraded":"2024-09-21T10:41:54.086956+0000","last_fullsized":"2024-09-21T10:41:54.086956+0000","mapping_epoch":1323,"log_start":"0'0","ondisk_log_start":"0'0","created":1323,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":6320,"log_dups_size":0,"ondisk_log_size":6320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270925824,"num_objects":4134,"num_object_clones":0,"num_object_copies":12402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4134,"num_write_kb":264576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.18","version":"1339'6232","reported_seq":6377,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:55.971719+0000","last_change":"2024-09-21T10:41:54.045448+0000","last_active":"2024-09-21T10:41:55.971719+0000","last_peered":"2024-09-21T10:41:55.971719+0000","last_clean":"2024-09-21T10:41:55.971719+0000","last_became_active":"2024-09-21T10:41:00.681104+0000","last_became_peered":"2024-09-21T10:41:00.681104+0000","last_unstale":"2024-09-21T10:41:55.971719+0000","last_undegraded":"2024-09-21T10:41:55.971719+0000","last_fullsized":"2024-09-21T10:41:55.971719+0000","mapping_epoch":1323,"log_start":"0'0","ondisk_log_start":"0'0","created":1323,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":5,"last_scrub":"1339'6199","last_scrub_stamp":"2024-09-21T10:41:54.045119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:41:54.045119+0000","objects_scrubbed":3990,"log_size":6232,"log_dups_size":0,"ondisk_log_size":6232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:23:01.505483+0000","scrub_duration":0.94622197900000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260767744,"num_objects":3979,"num_object_clones":0,"num_object_copies":11937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"53.17","version":"1339'6237","reported_seq":6326,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:54.005997+0000","last_change":"2024-09-21T10:41:00.699239+0000","last_active":"2024-09-21T10:41:54.005997+0000","last_peered":"2024-09-21T10:41:54.005997+0000","last_clean":"2024-09-21T10:41:54.005997+0000","last_became_active":"2024-09-21T10:41:00.695046+0000","last_became_peered":"2024-09-21T10:41:00.695046+0000","last_unstale":"2024-09-21T10:41:54.005997+0000","last_undegraded":"2024-09-21T10:41:54.005997+0000","last_fullsized":"2024-09-21T10:41:54.005997+0000","mapping_epoch":1323,"log_start":"0'0","ondisk_log_start":"0'0","created":1323,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":6237,"log_dups_size":0,"ondisk_log_size":6237,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266338304,"num_objects":4064,"num_object_clones":0,"num_object_copies":12192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4064,"num_write_kb":260096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.16","version":"1339'6206","reported_seq":6295,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:54.161134+0000","last_change":"2024-09-21T10:41:00.664226+0000","last_active":"2024-09-21T10:41:54.161134+0000","last_peered":"2024-09-21T10:41:54.161134+0000","last_clean":"2024-09-21T10:41:54.161134+0000","last_became_active":"2024-09-21T10:41:00.662248+0000","last_became_peered":"2024-09-21T10:41:00.662248+0000","last_unstale":"2024-09-21T10:41:54.161134+0000","last_undegraded":"2024-09-21T10:41:54.161134+0000","last_fullsized":"2024-09-21T10:41:54.161134+0000","mapping_epoch":1323,"log_start":"0'0","ondisk_log_start":"0'0","created":1323,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":6206,"log_dups_size":0,"ondisk_log_size":6206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264568832,"num_objects":4037,"num_object_clones":0,"num_object_copies":12111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":258368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"53.15","version":"1339'6188","reported_seq":6308,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:54.030352+0000","last_change":"2024-09-21T10:41:07.032615+0000","last_active":"2024-09-21T10:41:54.030352+0000","last_peered":"2024-09-21T10:41:54.030352+0000","last_clean":"2024-09-21T10:41:54.030352+0000","last_became_active":"2024-09-21T10:41:00.700283+0000","last_became_peered":"2024-09-21T10:41:00.700283+0000","last_unstale":"2024-09-21T10:41:54.030352+0000","last_undegraded":"2024-09-21T10:41:54.030352+0000","last_fullsized":"2024-09-21T10:41:54.030352+0000","mapping_epoch":1323,"log_start":"0'0","ondisk_log_start":"0'0","created":1323,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":5,"last_scrub":"1325'4554","last_scrub_stamp":"2024-09-21T10:41:07.032531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:41:07.032531+0000","objects_scrubbed":2432,"log_size":6188,"log_dups_size":0,"ondisk_log_size":6188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:36:54.404966+0000","scrub_duration":237,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266469376,"num_objects":4066,"num_object_clones":0,"num_object_copies":12198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1634,"num_write_kb":104576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.14","version":"1339'6212","reported_seq":6333,"reported_epoch":1339,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:41:54.022663+0000","last_change":"2024-09-21T10:41:14.189185+0000","last_active":"2024-09-21T10:41:54.022663+0000","last_peered":"2024-09-21T10:41:54.022663+0000","last_clean":"2024-09-21T10:41:12.657195+0000","last_became_active":"2024-09-21T10:41:14.185480+0000","last_became_peered":"2024-09-21T10:41:14.185480+0000","last_unstale":"2024-09-21T10:41:54.022663+0000","last_undegraded":"2024-09-21T10:41:14.133427+0000","last_fullsized":"2024-09-21T10:41:14.121595+0000","mapping_epoch":1333,"log_start":"0'0","ondisk_log_start":"0'0","created":1323,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":5,"last_scrub":"1324'4633","last_scrub_stamp":"2024-09-21T10:41:05.405418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:41:05.405418+0000","objects_scrubbed":2382,"log_size":6212,"log_dups_size":0,"ondisk_log_size":6212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260833280,"num_objects":3980,"num_object_clones":0,"num_object_copies":11940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1598,"num_write_kb":102272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[1,4],"avail_no_missing":["1","4","6"],"object_location_counts":[{"shards":"1,4","objects":1460},{"shards":"1,4,6","objects":2520}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.13","version":"1339'6376","reported_seq":9728,"reported_epoch":1339,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-21T10:41:54.059880+0000","last_change":"2024-09-21T10:41:26.897594+0000","last_active":"2024-09-21T10:41:54.059880+0000","last_peered":"2024-09-21T10:41:54.059880+0000","last_clean":"2024-09-21T10:41:10.039904+0000","last_became_active":"2024-09-21T10:41:12.815959+0000","last_became_peered":"2024-09-21T10:41:12.815959+0000","last_unstale":"2024-09-21T10:41:54.059880+0000","last_undegraded":"2024-09-21T10:41:54.059880+0000","last_fullsized":"2024-09-21T10:41:12.728072+0000","mapping_epoch":1331,"log_start":"0'0","ondisk_log_start":"0'0","created":1323,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":5,"last_scrub":"1324'4675","last_scrub_stamp":"2024-09-21T10:41:04.559688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:41:04.559688+0000","objects_scrubbed":2364,"log_size":6376,"log_dups_size":0,"ondisk_log_size":6376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":0.55553660900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266797056,"num_objects":4071,"num_object_clones":0,"num_object_copies":12213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5142,"num_objects_unfound":0,"num_objects_dirty":4071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1707,"num_write_kb":109248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":806,"num_bytes_recovered":52822016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[1,0],"avail_no_missing":["1","4","6"],"object_location_counts":[{"shards":"0,1","objects":2357},{"shards":"1,4,6","objects":1714}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.12","version":"1339'6082","reported_seq":6207,"reported_epoch":1339,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:41:54.004286+0000","last_change":"2024-09-21T10:41:11.942354+0000","last_active":"2024-09-21T10:41:54.004286+0000","last_peered":"2024-09-21T10:41:54.004286+0000","last_clean":"2024-09-21T10:41:09.653271+0000","last_became_active":"2024-09-21T10:41:11.937366+0000","last_became_peered":"2024-09-21T10:41:11.937366+0000","last_unstale":"2024-09-21T10:41:54.004286+0000","last_undegraded":"2024-09-21T10:41:11.934586+0000","last_fullsized":"2024-09-21T10:41:11.697775+0000","mapping_epoch":1330,"log_start":"0'0","ondisk_log_start":"0'0","created":1323,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":5,"last_scrub":"1325'4505","last_scrub_stamp":"2024-09-21T10:41:06.325848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:41:06.325848+0000","objects_scrubbed":2406,"log_size":6082,"log_dups_size":0,"ondisk_log_size":6082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":0.23505495100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261029888,"num_objects":3983,"num_object_clones":0,"num_object_copies":11949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4940,"num_objects_misplaced":2470,"num_objects_unfound":0,"num_objects_dirty":3983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1577,"num_write_kb":100928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[0,1],"avail_no_missing":["0","4","5"],"object_location_counts":[{"shards":"0,1","objects":1513},{"shards":"0,4,5","objects":2470}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.11","version":"1339'6205","reported_seq":6297,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:54.078560+0000","last_change":"2024-09-21T10:41:08.675260+0000","last_active":"2024-09-21T10:41:54.078560+0000","last_peered":"2024-09-21T10:41:54.078560+0000","last_clean":"2024-09-21T10:41:54.078560+0000","last_became_active":"2024-09-21T10:41:08.669075+0000","last_became_peered":"2024-09-21T10:41:08.669075+0000","last_unstale":"2024-09-21T10:41:54.078560+0000","last_undegraded":"2024-09-21T10:41:54.078560+0000","last_fullsized":"2024-09-21T10:41:54.078560+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1323,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":6205,"log_dups_size":0,"ondisk_log_size":6205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264142848,"num_objects":4031,"num_object_clones":0,"num_object_copies":12093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4031,"num_write_kb":257952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.10","version":"1339'6180","reported_seq":11081,"reported_epoch":1339,"state":"active+recovery_wait","last_fresh":"2024-09-21T10:41:54.080289+0000","last_change":"2024-09-21T10:41:28.439087+0000","last_active":"2024-09-21T10:41:54.080289+0000","last_peered":"2024-09-21T10:41:54.080289+0000","last_clean":"2024-09-21T10:41:05.612849+0000","last_became_active":"2024-09-21T10:41:28.426739+0000","last_became_peered":"2024-09-21T10:41:28.426739+0000","last_unstale":"2024-09-21T10:41:54.080289+0000","last_undegraded":"2024-09-21T10:41:54.080289+0000","last_fullsized":"2024-09-21T10:41:54.080289+0000","mapping_epoch":1336,"log_start":"0'0","ondisk_log_start":"0'0","created":1323,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":6180,"log_dups_size":0,"ondisk_log_size":6180,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263028736,"num_objects":4014,"num_object_clones":0,"num_object_copies":12042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4014,"num_write_kb":256864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2413,"num_bytes_recovered":158138368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":["0","4","5","6"],"object_location_counts":[{"shards":"0,5,6","objects":4014}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.f","version":"1339'7901","reported_seq":7942,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:56.124020+0000","last_change":"2024-09-21T10:40:16.447365+0000","last_active":"2024-09-21T10:41:56.124020+0000","last_peered":"2024-09-21T10:41:56.124020+0000","last_clean":"2024-09-21T10:41:56.124020+0000","last_became_active":"2024-09-21T10:40:16.447155+0000","last_became_peered":"2024-09-21T10:40:16.447155+0000","last_unstale":"2024-09-21T10:41:56.124020+0000","last_undegraded":"2024-09-21T10:41:56.124020+0000","last_fullsized":"2024-09-21T10:41:56.124020+0000","mapping_epoch":1307,"log_start":"1314'1200","ondisk_log_start":"1314'1200","created":1307,"last_epoch_clean":1308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":6701,"log_dups_size":0,"ondisk_log_size":6701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:39:36.972987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":510066688,"num_objects":7783,"num_object_clones":0,"num_object_copies":23349,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7901,"num_write_kb":501888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.e","version":"1339'8033","reported_seq":8073,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:56.095054+0000","last_change":"2024-09-21T10:40:16.450893+0000","last_active":"2024-09-21T10:41:56.095054+0000","last_peered":"2024-09-21T10:41:56.095054+0000","last_clean":"2024-09-21T10:41:56.095054+0000","last_became_active":"2024-09-21T10:40:16.450584+0000","last_became_peered":"2024-09-21T10:40:16.450584+0000","last_unstale":"2024-09-21T10:41:56.095054+0000","last_undegraded":"2024-09-21T10:41:56.095054+0000","last_fullsized":"2024-09-21T10:41:56.095054+0000","mapping_epoch":1307,"log_start":"1315'1300","ondisk_log_start":"1315'1300","created":1307,"last_epoch_clean":1308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":6733,"log_dups_size":0,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:38:24.901001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":518193152,"num_objects":7907,"num_object_clones":0,"num_object_copies":23721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8033,"num_write_kb":510080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.d","version":"1339'7947","reported_seq":8029,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:54.037296+0000","last_change":"2024-09-21T10:40:16.449312+0000","last_active":"2024-09-21T10:41:54.037296+0000","last_peered":"2024-09-21T10:41:54.037296+0000","last_clean":"2024-09-21T10:41:54.037296+0000","last_became_active":"2024-09-21T10:40:16.448919+0000","last_became_peered":"2024-09-21T10:40:16.448919+0000","last_unstale":"2024-09-21T10:41:54.037296+0000","last_undegraded":"2024-09-21T10:41:54.037296+0000","last_fullsized":"2024-09-21T10:41:54.037296+0000","mapping_epoch":1307,"log_start":"1318'3214","ondisk_log_start":"1318'3214","created":1307,"last_epoch_clean":1308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":4733,"log_dups_size":0,"ondisk_log_size":4733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:30.319509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520814592,"num_objects":7947,"num_object_clones":0,"num_object_copies":23841,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7947,"num_write_kb":508608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.c","version":"1339'8194","reported_seq":10722,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:55.964290+0000","last_change":"2024-09-21T10:40:42.390568+0000","last_active":"2024-09-21T10:41:55.964290+0000","last_peered":"2024-09-21T10:41:55.964290+0000","last_clean":"2024-09-21T10:41:55.964290+0000","last_became_active":"2024-09-21T10:40:42.388608+0000","last_became_peered":"2024-09-21T10:40:42.388608+0000","last_unstale":"2024-09-21T10:41:55.964290+0000","last_undegraded":"2024-09-21T10:41:55.964290+0000","last_fullsized":"2024-09-21T10:41:55.964290+0000","mapping_epoch":1317,"log_start":"1311'500","ondisk_log_start":"1311'500","created":1307,"last_epoch_clean":1318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":7694,"log_dups_size":0,"ondisk_log_size":7694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:06:04.278392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":529268736,"num_objects":8076,"num_object_clones":0,"num_object_copies":24228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8194,"num_write_kb":520640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1235,"num_bytes_recovered":80936960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"53.b","version":"1339'7961","reported_seq":8002,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:55.966818+0000","last_change":"2024-09-21T10:40:16.445075+0000","last_active":"2024-09-21T10:41:55.966818+0000","last_peered":"2024-09-21T10:41:55.966818+0000","last_clean":"2024-09-21T10:41:55.966818+0000","last_became_active":"2024-09-21T10:40:16.444802+0000","last_became_peered":"2024-09-21T10:40:16.444802+0000","last_unstale":"2024-09-21T10:41:55.966818+0000","last_undegraded":"2024-09-21T10:41:55.966818+0000","last_fullsized":"2024-09-21T10:41:55.966818+0000","mapping_epoch":1307,"log_start":"1310'200","ondisk_log_start":"1310'200","created":1307,"last_epoch_clean":1308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":7761,"log_dups_size":0,"ondisk_log_size":7761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:36:32.765238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":515964928,"num_objects":7873,"num_object_clones":0,"num_object_copies":23619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7961,"num_write_kb":506688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"53.a","version":"1339'8062","reported_seq":8144,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:54.064245+0000","last_change":"2024-09-21T10:40:16.451613+0000","last_active":"2024-09-21T10:41:54.064245+0000","last_peered":"2024-09-21T10:41:54.064245+0000","last_clean":"2024-09-21T10:41:54.064245+0000","last_became_active":"2024-09-21T10:40:16.451348+0000","last_became_peered":"2024-09-21T10:40:16.451348+0000","last_unstale":"2024-09-21T10:41:54.064245+0000","last_undegraded":"2024-09-21T10:41:54.064245+0000","last_fullsized":"2024-09-21T10:41:54.064245+0000","mapping_epoch":1307,"log_start":"1316'1900","ondisk_log_start":"1316'1900","created":1307,"last_epoch_clean":1308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":6162,"log_dups_size":0,"ondisk_log_size":6162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:05:08.562680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528351232,"num_objects":8062,"num_object_clones":0,"num_object_copies":24186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8062,"num_write_kb":515968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"53.9","version":"1339'6207","reported_seq":6296,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:53.934708+0000","last_change":"2024-09-21T10:41:00.659513+0000","last_active":"2024-09-21T10:41:53.934708+0000","last_peered":"2024-09-21T10:41:53.934708+0000","last_clean":"2024-09-21T10:41:53.934708+0000","last_became_active":"2024-09-21T10:41:00.656007+0000","last_became_peered":"2024-09-21T10:41:00.656007+0000","last_unstale":"2024-09-21T10:41:53.934708+0000","last_undegraded":"2024-09-21T10:41:53.934708+0000","last_fullsized":"2024-09-21T10:41:53.934708+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":6207,"log_dups_size":0,"ondisk_log_size":6207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263520256,"num_objects":4021,"num_object_clones":0,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":257344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.8","version":"1339'6197","reported_seq":6307,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:55.994921+0000","last_change":"2024-09-21T10:41:01.831615+0000","last_active":"2024-09-21T10:41:55.994921+0000","last_peered":"2024-09-21T10:41:55.994921+0000","last_clean":"2024-09-21T10:41:55.994921+0000","last_became_active":"2024-09-21T10:41:00.642849+0000","last_became_peered":"2024-09-21T10:41:00.642849+0000","last_unstale":"2024-09-21T10:41:55.994921+0000","last_undegraded":"2024-09-21T10:41:55.994921+0000","last_fullsized":"2024-09-21T10:41:55.994921+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":0,"last_scrub":"1324'4314","last_scrub_stamp":"2024-09-21T10:41:01.831467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:41:01.831467+0000","objects_scrubbed":2229,"log_size":6197,"log_dups_size":0,"ondisk_log_size":6197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:24:13.824542+0000","scrub_duration":0.79565424399999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267452416,"num_objects":4081,"num_object_clones":0,"num_object_copies":12243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1908,"num_write_kb":120320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"53.7","version":"1339'6301","reported_seq":6419,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:54.048738+0000","last_change":"2024-09-21T10:41:01.305079+0000","last_active":"2024-09-21T10:41:54.048738+0000","last_peered":"2024-09-21T10:41:54.048738+0000","last_clean":"2024-09-21T10:41:54.048738+0000","last_became_active":"2024-09-21T10:41:00.643851+0000","last_became_peered":"2024-09-21T10:41:00.643851+0000","last_unstale":"2024-09-21T10:41:54.048738+0000","last_undegraded":"2024-09-21T10:41:54.048738+0000","last_fullsized":"2024-09-21T10:41:54.048738+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":0,"last_scrub":"1324'4378","last_scrub_stamp":"2024-09-21T10:41:01.304972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:41:01.304972+0000","objects_scrubbed":2133,"log_size":6301,"log_dups_size":0,"ondisk_log_size":6301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:55:41.695678+0000","scrub_duration":601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266993664,"num_objects":4074,"num_object_clones":0,"num_object_copies":12222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1941,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.6","version":"1339'6218","reported_seq":6339,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:54.057405+0000","last_change":"2024-09-21T10:41:05.966135+0000","last_active":"2024-09-21T10:41:54.057405+0000","last_peered":"2024-09-21T10:41:54.057405+0000","last_clean":"2024-09-21T10:41:54.057405+0000","last_became_active":"2024-09-21T10:41:00.643784+0000","last_became_peered":"2024-09-21T10:41:00.643784+0000","last_unstale":"2024-09-21T10:41:54.057405+0000","last_undegraded":"2024-09-21T10:41:54.057405+0000","last_fullsized":"2024-09-21T10:41:54.057405+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":0,"last_scrub":"1325'4633","last_scrub_stamp":"2024-09-21T10:41:05.966042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:41:05.966042+0000","objects_scrubbed":2477,"log_size":6218,"log_dups_size":0,"ondisk_log_size":6218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:25.235633+0000","scrub_duration":485,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266928128,"num_objects":4073,"num_object_clones":0,"num_object_copies":12219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1596,"num_write_kb":102144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"53.5","version":"1339'6216","reported_seq":6309,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:54.020858+0000","last_change":"2024-09-21T10:41:00.694221+0000","last_active":"2024-09-21T10:41:54.020858+0000","last_peered":"2024-09-21T10:41:54.020858+0000","last_clean":"2024-09-21T10:41:54.020858+0000","last_became_active":"2024-09-21T10:41:00.677969+0000","last_became_peered":"2024-09-21T10:41:00.677969+0000","last_unstale":"2024-09-21T10:41:54.020858+0000","last_undegraded":"2024-09-21T10:41:54.020858+0000","last_fullsized":"2024-09-21T10:41:54.020858+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:40:15.426072+0000","objects_scrubbed":0,"log_size":6216,"log_dups_size":0,"ondisk_log_size":6216,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266698752,"num_objects":4069,"num_object_clones":0,"num_object_copies":12207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4069,"num_write_kb":260448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.4","version":"1339'6268","reported_seq":6389,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:54.054775+0000","last_change":"2024-09-21T10:41:04.470741+0000","last_active":"2024-09-21T10:41:54.054775+0000","last_peered":"2024-09-21T10:41:54.054775+0000","last_clean":"2024-09-21T10:41:54.054775+0000","last_became_active":"2024-09-21T10:41:00.652865+0000","last_became_peered":"2024-09-21T10:41:00.652865+0000","last_unstale":"2024-09-21T10:41:54.054775+0000","last_undegraded":"2024-09-21T10:41:54.054775+0000","last_fullsized":"2024-09-21T10:41:54.054775+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":0,"last_scrub":"1324'4563","last_scrub_stamp":"2024-09-21T10:41:04.470538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:41:04.470538+0000","objects_scrubbed":2413,"log_size":6268,"log_dups_size":0,"ondisk_log_size":6268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:00.635682+0000","scrub_duration":748,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270794752,"num_objects":4132,"num_object_clones":0,"num_object_copies":12396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1719,"num_write_kb":110016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.3","version":"1339'6457","reported_seq":6570,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:55.945369+0000","last_change":"2024-09-21T10:41:01.640980+0000","last_active":"2024-09-21T10:41:55.945369+0000","last_peered":"2024-09-21T10:41:55.945369+0000","last_clean":"2024-09-21T10:41:55.945369+0000","last_became_active":"2024-09-21T10:41:00.656721+0000","last_became_peered":"2024-09-21T10:41:00.656721+0000","last_unstale":"2024-09-21T10:41:55.945369+0000","last_undegraded":"2024-09-21T10:41:55.945369+0000","last_fullsized":"2024-09-21T10:41:55.945369+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":0,"last_scrub":"1324'4534","last_scrub_stamp":"2024-09-21T10:41:01.640809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:41:01.640809+0000","objects_scrubbed":2329,"log_size":6457,"log_dups_size":0,"ondisk_log_size":6457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:46.711073+0000","scrub_duration":0.68720467699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276365312,"num_objects":4217,"num_object_clones":0,"num_object_copies":12651,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1934,"num_write_kb":122304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.2","version":"1339'6089","reported_seq":6200,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:55.973771+0000","last_change":"2024-09-21T10:41:05.605174+0000","last_active":"2024-09-21T10:41:55.973771+0000","last_peered":"2024-09-21T10:41:55.973771+0000","last_clean":"2024-09-21T10:41:55.973771+0000","last_became_active":"2024-09-21T10:41:00.653765+0000","last_became_peered":"2024-09-21T10:41:00.653765+0000","last_unstale":"2024-09-21T10:41:55.973771+0000","last_undegraded":"2024-09-21T10:41:55.973771+0000","last_fullsized":"2024-09-21T10:41:55.973771+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":0,"last_scrub":"1324'4518","last_scrub_stamp":"2024-09-21T10:41:05.604498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:41:05.604498+0000","objects_scrubbed":2371,"log_size":6089,"log_dups_size":0,"ondisk_log_size":6089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:02:29.176750+0000","scrub_duration":0.52519471500000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255524864,"num_objects":3899,"num_object_clones":0,"num_object_copies":11697,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1588,"num_write_kb":99712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"53.1","version":"1339'6201","reported_seq":6319,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:53.900246+0000","last_change":"2024-09-21T10:41:02.404669+0000","last_active":"2024-09-21T10:41:53.900246+0000","last_peered":"2024-09-21T10:41:53.900246+0000","last_clean":"2024-09-21T10:41:53.900246+0000","last_became_active":"2024-09-21T10:41:00.644585+0000","last_became_peered":"2024-09-21T10:41:00.644585+0000","last_unstale":"2024-09-21T10:41:53.900246+0000","last_undegraded":"2024-09-21T10:41:53.900246+0000","last_fullsized":"2024-09-21T10:41:53.900246+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":0,"last_scrub":"1324'4434","last_scrub_stamp":"2024-09-21T10:41:02.404449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:41:02.404449+0000","objects_scrubbed":2298,"log_size":6201,"log_dups_size":0,"ondisk_log_size":6201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:23:47.765903+0000","scrub_duration":693,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267845632,"num_objects":4087,"num_object_clones":0,"num_object_copies":12261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1789,"num_write_kb":114496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.0","version":"1339'6151","reported_seq":6258,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:56.021835+0000","last_change":"2024-09-21T10:41:02.770416+0000","last_active":"2024-09-21T10:41:56.021835+0000","last_peered":"2024-09-21T10:41:56.021835+0000","last_clean":"2024-09-21T10:41:56.021835+0000","last_became_active":"2024-09-21T10:41:00.642952+0000","last_became_peered":"2024-09-21T10:41:00.642952+0000","last_unstale":"2024-09-21T10:41:56.021835+0000","last_undegraded":"2024-09-21T10:41:56.021835+0000","last_fullsized":"2024-09-21T10:41:56.021835+0000","mapping_epoch":1307,"log_start":"0'0","ondisk_log_start":"0'0","created":1307,"last_epoch_clean":1324,"parent":"0.0","parent_split_bits":0,"last_scrub":"1324'4424","last_scrub_stamp":"2024-09-21T10:41:02.770194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:40:15.426072+0000","last_clean_scrub_stamp":"2024-09-21T10:41:02.770194+0000","objects_scrubbed":2234,"log_size":6151,"log_dups_size":0,"ondisk_log_size":6151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:17:14.808618+0000","scrub_duration":0.69884458100000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256770048,"num_objects":3918,"num_object_clones":0,"num_object_copies":11754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1740,"num_write_kb":109568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":1988,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.824739+0000","last_change":"2024-09-21T10:40:13.438023+0000","last_active":"2024-09-21T10:41:52.824739+0000","last_peered":"2024-09-21T10:41:52.824739+0000","last_clean":"2024-09-21T10:41:52.824739+0000","last_became_active":"2024-09-21T10:40:13.431103+0000","last_became_peered":"2024-09-21T10:40:13.431103+0000","last_unstale":"2024-09-21T10:41:52.824739+0000","last_undegraded":"2024-09-21T10:41:52.824739+0000","last_fullsized":"2024-09-21T10:41:52.824739+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:00:24.887661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1306,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.474271+0000","last_change":"2024-09-21T10:07:55.999392+0000","last_active":"2024-09-21T10:41:34.474271+0000","last_peered":"2024-09-21T10:41:34.474271+0000","last_clean":"2024-09-21T10:41:34.474271+0000","last_became_active":"2024-09-21T10:07:55.999103+0000","last_became_peered":"2024-09-21T10:07:55.999103+0000","last_unstale":"2024-09-21T10:41:34.474271+0000","last_undegraded":"2024-09-21T10:41:34.474271+0000","last_fullsized":"2024-09-21T10:41:34.474271+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:48.757242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":1969,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.834209+0000","last_change":"2024-09-21T10:40:13.432818+0000","last_active":"2024-09-21T10:41:52.834209+0000","last_peered":"2024-09-21T10:41:52.834209+0000","last_clean":"2024-09-21T10:41:52.834209+0000","last_became_active":"2024-09-21T10:40:13.432448+0000","last_became_peered":"2024-09-21T10:40:13.432448+0000","last_unstale":"2024-09-21T10:41:52.834209+0000","last_undegraded":"2024-09-21T10:41:52.834209+0000","last_fullsized":"2024-09-21T10:41:52.834209+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:20.448242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1986,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.833102+0000","last_change":"2024-09-21T10:40:13.431824+0000","last_active":"2024-09-21T10:41:52.833102+0000","last_peered":"2024-09-21T10:41:52.833102+0000","last_clean":"2024-09-21T10:41:52.833102+0000","last_became_active":"2024-09-21T10:40:13.431516+0000","last_became_peered":"2024-09-21T10:40:13.431516+0000","last_unstale":"2024-09-21T10:41:52.833102+0000","last_undegraded":"2024-09-21T10:41:52.833102+0000","last_fullsized":"2024-09-21T10:41:52.833102+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:30:40.226452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":1293,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.475937+0000","last_change":"2024-09-21T09:54:53.634508+0000","last_active":"2024-09-21T10:41:34.475937+0000","last_peered":"2024-09-21T10:41:34.475937+0000","last_clean":"2024-09-21T10:41:34.475937+0000","last_became_active":"2024-09-21T09:54:53.634254+0000","last_became_peered":"2024-09-21T09:54:53.634254+0000","last_unstale":"2024-09-21T10:41:34.475937+0000","last_undegraded":"2024-09-21T10:41:34.475937+0000","last_fullsized":"2024-09-21T10:41:34.475937+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:18.122107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1320,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.474440+0000","last_change":"2024-09-21T10:09:06.574933+0000","last_active":"2024-09-21T10:41:34.474440+0000","last_peered":"2024-09-21T10:41:34.474440+0000","last_clean":"2024-09-21T10:41:34.474440+0000","last_became_active":"2024-09-21T10:09:06.574467+0000","last_became_peered":"2024-09-21T10:09:06.574467+0000","last_unstale":"2024-09-21T10:41:34.474440+0000","last_undegraded":"2024-09-21T10:41:34.474440+0000","last_fullsized":"2024-09-21T10:41:34.474440+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:06.142269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":1308,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.475864+0000","last_change":"2024-09-21T10:07:55.991189+0000","last_active":"2024-09-21T10:41:34.475864+0000","last_peered":"2024-09-21T10:41:34.475864+0000","last_clean":"2024-09-21T10:41:34.475864+0000","last_became_active":"2024-09-21T10:07:55.990784+0000","last_became_peered":"2024-09-21T10:07:55.990784+0000","last_unstale":"2024-09-21T10:41:34.475864+0000","last_undegraded":"2024-09-21T10:41:34.475864+0000","last_fullsized":"2024-09-21T10:41:34.475864+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:00.580471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1320,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.475562+0000","last_change":"2024-09-21T10:09:06.560175+0000","last_active":"2024-09-21T10:41:34.475562+0000","last_peered":"2024-09-21T10:41:34.475562+0000","last_clean":"2024-09-21T10:41:34.475562+0000","last_became_active":"2024-09-21T10:09:06.558584+0000","last_became_peered":"2024-09-21T10:09:06.558584+0000","last_unstale":"2024-09-21T10:41:34.475562+0000","last_undegraded":"2024-09-21T10:41:34.475562+0000","last_fullsized":"2024-09-21T10:41:34.475562+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:02.918339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":1964,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.825110+0000","last_change":"2024-09-21T10:40:30.424630+0000","last_active":"2024-09-21T10:41:52.825110+0000","last_peered":"2024-09-21T10:41:52.825110+0000","last_clean":"2024-09-21T10:41:52.825110+0000","last_became_active":"2024-09-21T10:40:30.424295+0000","last_became_peered":"2024-09-21T10:40:30.424295+0000","last_unstale":"2024-09-21T10:41:52.825110+0000","last_undegraded":"2024-09-21T10:41:52.825110+0000","last_fullsized":"2024-09-21T10:41:52.825110+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:59:50.861993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":748,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.837272+0000","last_change":"2024-09-21T10:40:13.436927+0000","last_active":"2024-09-21T10:41:52.837272+0000","last_peered":"2024-09-21T10:41:52.837272+0000","last_clean":"2024-09-21T10:41:52.837272+0000","last_became_active":"2024-09-21T10:40:13.436329+0000","last_became_peered":"2024-09-21T10:40:13.436329+0000","last_unstale":"2024-09-21T10:41:52.837272+0000","last_undegraded":"2024-09-21T10:41:52.837272+0000","last_fullsized":"2024-09-21T10:41:52.837272+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:05.668898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":1331,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.475266+0000","last_change":"2024-09-21T10:40:13.433278+0000","last_active":"2024-09-21T10:41:34.475266+0000","last_peered":"2024-09-21T10:41:34.475266+0000","last_clean":"2024-09-21T10:41:34.475266+0000","last_became_active":"2024-09-21T10:40:13.432814+0000","last_became_peered":"2024-09-21T10:40:13.432814+0000","last_unstale":"2024-09-21T10:41:34.475266+0000","last_undegraded":"2024-09-21T10:41:34.475266+0000","last_fullsized":"2024-09-21T10:41:34.475266+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:55.909872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1980,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.833421+0000","last_change":"2024-09-21T10:40:13.432531+0000","last_active":"2024-09-21T10:41:52.833421+0000","last_peered":"2024-09-21T10:41:52.833421+0000","last_clean":"2024-09-21T10:41:52.833421+0000","last_became_active":"2024-09-21T10:40:13.432210+0000","last_became_peered":"2024-09-21T10:40:13.432210+0000","last_unstale":"2024-09-21T10:41:52.833421+0000","last_undegraded":"2024-09-21T10:41:52.833421+0000","last_fullsized":"2024-09-21T10:41:52.833421+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:20:10.998607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1951,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.829208+0000","last_change":"2024-09-21T10:09:06.530394+0000","last_active":"2024-09-21T10:41:52.829208+0000","last_peered":"2024-09-21T10:41:52.829208+0000","last_clean":"2024-09-21T10:41:52.829208+0000","last_became_active":"2024-09-21T10:09:06.529307+0000","last_became_peered":"2024-09-21T10:09:06.529307+0000","last_unstale":"2024-09-21T10:41:52.829208+0000","last_undegraded":"2024-09-21T10:41:52.829208+0000","last_fullsized":"2024-09-21T10:41:52.829208+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:54.287397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1977,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.824346+0000","last_change":"2024-09-21T10:40:30.420276+0000","last_active":"2024-09-21T10:41:52.824346+0000","last_peered":"2024-09-21T10:41:52.824346+0000","last_clean":"2024-09-21T10:41:52.824346+0000","last_became_active":"2024-09-21T10:40:30.419723+0000","last_became_peered":"2024-09-21T10:40:30.419723+0000","last_unstale":"2024-09-21T10:41:52.824346+0000","last_undegraded":"2024-09-21T10:41:52.824346+0000","last_fullsized":"2024-09-21T10:41:52.824346+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:52:04.181647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1966,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.832978+0000","last_change":"2024-09-21T10:09:06.552324+0000","last_active":"2024-09-21T10:41:52.832978+0000","last_peered":"2024-09-21T10:41:52.832978+0000","last_clean":"2024-09-21T10:41:52.832978+0000","last_became_active":"2024-09-21T10:09:06.540545+0000","last_became_peered":"2024-09-21T10:09:06.540545+0000","last_unstale":"2024-09-21T10:41:52.832978+0000","last_undegraded":"2024-09-21T10:41:52.832978+0000","last_fullsized":"2024-09-21T10:41:52.832978+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:08.506149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1310,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.468740+0000","last_change":"2024-09-21T10:07:56.003005+0000","last_active":"2024-09-21T10:41:34.468740+0000","last_peered":"2024-09-21T10:41:34.468740+0000","last_clean":"2024-09-21T10:41:34.468740+0000","last_became_active":"2024-09-21T10:07:56.001550+0000","last_became_peered":"2024-09-21T10:07:56.001550+0000","last_unstale":"2024-09-21T10:41:34.468740+0000","last_undegraded":"2024-09-21T10:41:34.468740+0000","last_fullsized":"2024-09-21T10:41:34.468740+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:54.553261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1959,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.833132+0000","last_change":"2024-09-21T10:09:06.559017+0000","last_active":"2024-09-21T10:41:52.833132+0000","last_peered":"2024-09-21T10:41:52.833132+0000","last_clean":"2024-09-21T10:41:52.833132+0000","last_became_active":"2024-09-21T10:09:06.555256+0000","last_became_peered":"2024-09-21T10:09:06.555256+0000","last_unstale":"2024-09-21T10:41:52.833132+0000","last_undegraded":"2024-09-21T10:41:52.833132+0000","last_fullsized":"2024-09-21T10:41:52.833132+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:33:45.145138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1964,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.824897+0000","last_change":"2024-09-21T10:09:06.562504+0000","last_active":"2024-09-21T10:41:52.824897+0000","last_peered":"2024-09-21T10:41:52.824897+0000","last_clean":"2024-09-21T10:41:52.824897+0000","last_became_active":"2024-09-21T10:09:06.561747+0000","last_became_peered":"2024-09-21T10:09:06.561747+0000","last_unstale":"2024-09-21T10:41:52.824897+0000","last_undegraded":"2024-09-21T10:41:52.824897+0000","last_fullsized":"2024-09-21T10:41:52.824897+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:03:58.012406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1957,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.833285+0000","last_change":"2024-09-21T10:40:30.417426+0000","last_active":"2024-09-21T10:41:52.833285+0000","last_peered":"2024-09-21T10:41:52.833285+0000","last_clean":"2024-09-21T10:41:52.833285+0000","last_became_active":"2024-09-21T10:40:30.417075+0000","last_became_peered":"2024-09-21T10:40:30.417075+0000","last_unstale":"2024-09-21T10:41:52.833285+0000","last_undegraded":"2024-09-21T10:41:52.833285+0000","last_fullsized":"2024-09-21T10:41:52.833285+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:58.077873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1316,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.476365+0000","last_change":"2024-09-21T10:40:30.421413+0000","last_active":"2024-09-21T10:41:34.476365+0000","last_peered":"2024-09-21T10:41:34.476365+0000","last_clean":"2024-09-21T10:41:34.476365+0000","last_became_active":"2024-09-21T10:40:30.420689+0000","last_became_peered":"2024-09-21T10:40:30.420689+0000","last_unstale":"2024-09-21T10:41:34.476365+0000","last_undegraded":"2024-09-21T10:41:34.476365+0000","last_fullsized":"2024-09-21T10:41:34.476365+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:53.860610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":83,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.833580+0000","last_change":"2024-09-21T10:40:30.431008+0000","last_active":"2024-09-21T10:41:52.833580+0000","last_peered":"2024-09-21T10:41:52.833580+0000","last_clean":"2024-09-21T10:41:52.833580+0000","last_became_active":"2024-09-21T10:40:30.422769+0000","last_became_peered":"2024-09-21T10:40:30.422769+0000","last_unstale":"2024-09-21T10:41:52.833580+0000","last_undegraded":"2024-09-21T10:41:52.833580+0000","last_fullsized":"2024-09-21T10:41:52.833580+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:42:21.520242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1958,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.829354+0000","last_change":"2024-09-21T10:40:13.437741+0000","last_active":"2024-09-21T10:41:52.829354+0000","last_peered":"2024-09-21T10:41:52.829354+0000","last_clean":"2024-09-21T10:41:52.829354+0000","last_became_active":"2024-09-21T10:40:13.437447+0000","last_became_peered":"2024-09-21T10:40:13.437447+0000","last_unstale":"2024-09-21T10:41:52.829354+0000","last_undegraded":"2024-09-21T10:41:52.829354+0000","last_fullsized":"2024-09-21T10:41:52.829354+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:40:47.072907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":1335,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:51.032368+0000","last_change":"2024-09-21T10:40:30.437433+0000","last_active":"2024-09-21T10:41:51.032368+0000","last_peered":"2024-09-21T10:41:51.032368+0000","last_clean":"2024-09-21T10:41:51.032368+0000","last_became_active":"2024-09-21T10:40:30.426024+0000","last_became_peered":"2024-09-21T10:40:30.426024+0000","last_unstale":"2024-09-21T10:41:51.032368+0000","last_undegraded":"2024-09-21T10:41:51.032368+0000","last_fullsized":"2024-09-21T10:41:51.032368+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:13:47.175034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4756,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1307,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:36.026152+0000","last_change":"2024-09-21T10:07:56.001756+0000","last_active":"2024-09-21T10:41:36.026152+0000","last_peered":"2024-09-21T10:41:36.026152+0000","last_clean":"2024-09-21T10:41:36.026152+0000","last_became_active":"2024-09-21T10:07:56.000789+0000","last_became_peered":"2024-09-21T10:07:56.000789+0000","last_unstale":"2024-09-21T10:41:36.026152+0000","last_undegraded":"2024-09-21T10:41:36.026152+0000","last_fullsized":"2024-09-21T10:41:36.026152+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:56:04.302902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1294,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.475043+0000","last_change":"2024-09-21T09:54:53.636902+0000","last_active":"2024-09-21T10:41:34.475043+0000","last_peered":"2024-09-21T10:41:34.475043+0000","last_clean":"2024-09-21T10:41:34.475043+0000","last_became_active":"2024-09-21T09:54:53.636616+0000","last_became_peered":"2024-09-21T09:54:53.636616+0000","last_unstale":"2024-09-21T10:41:34.475043+0000","last_undegraded":"2024-09-21T10:41:34.475043+0000","last_fullsized":"2024-09-21T10:41:34.475043+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:46:47.694495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1341,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.468728+0000","last_change":"2024-09-21T10:40:30.423568+0000","last_active":"2024-09-21T10:41:34.468728+0000","last_peered":"2024-09-21T10:41:34.468728+0000","last_clean":"2024-09-21T10:41:34.468728+0000","last_became_active":"2024-09-21T10:40:30.423208+0000","last_became_peered":"2024-09-21T10:40:30.423208+0000","last_unstale":"2024-09-21T10:41:34.468728+0000","last_undegraded":"2024-09-21T10:41:34.468728+0000","last_fullsized":"2024-09-21T10:41:34.468728+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:22:29.996089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":1971,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.824258+0000","last_change":"2024-09-21T10:07:56.023462+0000","last_active":"2024-09-21T10:41:52.824258+0000","last_peered":"2024-09-21T10:41:52.824258+0000","last_clean":"2024-09-21T10:41:52.824258+0000","last_became_active":"2024-09-21T10:07:56.021859+0000","last_became_peered":"2024-09-21T10:07:56.021859+0000","last_unstale":"2024-09-21T10:41:52.824258+0000","last_undegraded":"2024-09-21T10:41:52.824258+0000","last_fullsized":"2024-09-21T10:41:52.824258+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:42:03.134729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1953,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.833056+0000","last_change":"2024-09-21T10:40:30.420880+0000","last_active":"2024-09-21T10:41:52.833056+0000","last_peered":"2024-09-21T10:41:52.833056+0000","last_clean":"2024-09-21T10:41:52.833056+0000","last_became_active":"2024-09-21T10:40:30.420555+0000","last_became_peered":"2024-09-21T10:40:30.420555+0000","last_unstale":"2024-09-21T10:41:52.833056+0000","last_undegraded":"2024-09-21T10:41:52.833056+0000","last_fullsized":"2024-09-21T10:41:52.833056+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:15:30.473928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1952,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.830833+0000","last_change":"2024-09-21T10:07:56.006236+0000","last_active":"2024-09-21T10:41:52.830833+0000","last_peered":"2024-09-21T10:41:52.830833+0000","last_clean":"2024-09-21T10:41:52.830833+0000","last_became_active":"2024-09-21T10:07:56.004439+0000","last_became_peered":"2024-09-21T10:07:56.004439+0000","last_unstale":"2024-09-21T10:41:52.830833+0000","last_undegraded":"2024-09-21T10:41:52.830833+0000","last_fullsized":"2024-09-21T10:41:52.830833+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:39.422467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":1958,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.836302+0000","last_change":"2024-09-21T10:40:30.425998+0000","last_active":"2024-09-21T10:41:52.836302+0000","last_peered":"2024-09-21T10:41:52.836302+0000","last_clean":"2024-09-21T10:41:52.836302+0000","last_became_active":"2024-09-21T10:40:30.421240+0000","last_became_peered":"2024-09-21T10:40:30.421240+0000","last_unstale":"2024-09-21T10:41:52.836302+0000","last_undegraded":"2024-09-21T10:41:52.836302+0000","last_fullsized":"2024-09-21T10:41:52.836302+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:43:44.457431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1965,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.825260+0000","last_change":"2024-09-21T10:40:30.417473+0000","last_active":"2024-09-21T10:41:52.825260+0000","last_peered":"2024-09-21T10:41:52.825260+0000","last_clean":"2024-09-21T10:41:52.825260+0000","last_became_active":"2024-09-21T10:40:30.416952+0000","last_became_peered":"2024-09-21T10:40:30.416952+0000","last_unstale":"2024-09-21T10:41:52.825260+0000","last_undegraded":"2024-09-21T10:41:52.825260+0000","last_fullsized":"2024-09-21T10:41:52.825260+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:07:33.486362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":694,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.824308+0000","last_change":"2024-09-21T10:40:30.425276+0000","last_active":"2024-09-21T10:41:52.824308+0000","last_peered":"2024-09-21T10:41:52.824308+0000","last_clean":"2024-09-21T10:41:52.824308+0000","last_became_active":"2024-09-21T10:40:30.425046+0000","last_became_peered":"2024-09-21T10:40:30.425046+0000","last_unstale":"2024-09-21T10:41:52.824308+0000","last_undegraded":"2024-09-21T10:41:52.824308+0000","last_fullsized":"2024-09-21T10:41:52.824308+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:09:59.127115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1988,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.837174+0000","last_change":"2024-09-21T10:40:13.437609+0000","last_active":"2024-09-21T10:41:52.837174+0000","last_peered":"2024-09-21T10:41:52.837174+0000","last_clean":"2024-09-21T10:41:52.837174+0000","last_became_active":"2024-09-21T10:40:13.437296+0000","last_became_peered":"2024-09-21T10:40:13.437296+0000","last_unstale":"2024-09-21T10:41:52.837174+0000","last_undegraded":"2024-09-21T10:41:52.837174+0000","last_fullsized":"2024-09-21T10:41:52.837174+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:41:58.963386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1943,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.829280+0000","last_change":"2024-09-21T10:07:56.067772+0000","last_active":"2024-09-21T10:41:52.829280+0000","last_peered":"2024-09-21T10:41:52.829280+0000","last_clean":"2024-09-21T10:41:52.829280+0000","last_became_active":"2024-09-21T10:07:56.052602+0000","last_became_peered":"2024-09-21T10:07:56.052602+0000","last_unstale":"2024-09-21T10:41:52.829280+0000","last_undegraded":"2024-09-21T10:41:52.829280+0000","last_fullsized":"2024-09-21T10:41:52.829280+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:07:59.899214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2004,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.836977+0000","last_change":"2024-09-21T10:40:30.418199+0000","last_active":"2024-09-21T10:41:52.836977+0000","last_peered":"2024-09-21T10:41:52.836977+0000","last_clean":"2024-09-21T10:41:52.836977+0000","last_became_active":"2024-09-21T10:40:30.417414+0000","last_became_peered":"2024-09-21T10:40:30.417414+0000","last_unstale":"2024-09-21T10:41:52.836977+0000","last_undegraded":"2024-09-21T10:41:52.836977+0000","last_fullsized":"2024-09-21T10:41:52.836977+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:56:54.232106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1334,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.476085+0000","last_change":"2024-09-21T10:40:13.435375+0000","last_active":"2024-09-21T10:41:34.476085+0000","last_peered":"2024-09-21T10:41:34.476085+0000","last_clean":"2024-09-21T10:41:34.476085+0000","last_became_active":"2024-09-21T10:40:13.435050+0000","last_became_peered":"2024-09-21T10:40:13.435050+0000","last_unstale":"2024-09-21T10:41:34.476085+0000","last_undegraded":"2024-09-21T10:41:34.476085+0000","last_fullsized":"2024-09-21T10:41:34.476085+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:57.889610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1998,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.837340+0000","last_change":"2024-09-21T10:40:13.434189+0000","last_active":"2024-09-21T10:41:52.837340+0000","last_peered":"2024-09-21T10:41:52.837340+0000","last_clean":"2024-09-21T10:41:52.837340+0000","last_became_active":"2024-09-21T10:40:13.433892+0000","last_became_peered":"2024-09-21T10:40:13.433892+0000","last_unstale":"2024-09-21T10:41:52.837340+0000","last_undegraded":"2024-09-21T10:41:52.837340+0000","last_fullsized":"2024-09-21T10:41:52.837340+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:49:17.296220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1310,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.475710+0000","last_change":"2024-09-21T10:07:21.524082+0000","last_active":"2024-09-21T10:41:34.475710+0000","last_peered":"2024-09-21T10:41:34.475710+0000","last_clean":"2024-09-21T10:41:34.475710+0000","last_became_active":"2024-09-21T10:07:21.523779+0000","last_became_peered":"2024-09-21T10:07:21.523779+0000","last_unstale":"2024-09-21T10:41:34.475710+0000","last_undegraded":"2024-09-21T10:41:34.475710+0000","last_fullsized":"2024-09-21T10:41:34.475710+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:13.813832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":1325,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:41.021774+0000","last_change":"2024-09-21T10:07:56.024239+0000","last_active":"2024-09-21T10:41:41.021774+0000","last_peered":"2024-09-21T10:41:41.021774+0000","last_clean":"2024-09-21T10:41:41.021774+0000","last_became_active":"2024-09-21T10:07:56.023914+0000","last_became_peered":"2024-09-21T10:07:56.023914+0000","last_unstale":"2024-09-21T10:41:41.021774+0000","last_undegraded":"2024-09-21T10:41:41.021774+0000","last_fullsized":"2024-09-21T10:41:41.021774+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:23.361736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084477399999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":29,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.474600+0000","last_change":"2024-09-21T10:40:55.616051+0000","last_active":"2024-09-21T10:41:34.474600+0000","last_peered":"2024-09-21T10:41:34.474600+0000","last_clean":"2024-09-21T10:41:34.474600+0000","last_became_active":"2024-09-21T10:40:55.615741+0000","last_became_peered":"2024-09-21T10:40:55.615741+0000","last_unstale":"2024-09-21T10:41:34.474600+0000","last_undegraded":"2024-09-21T10:41:34.474600+0000","last_fullsized":"2024-09-21T10:41:34.474600+0000","mapping_epoch":1320,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:07.673527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1961,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.828661+0000","last_change":"2024-09-21T10:40:13.436234+0000","last_active":"2024-09-21T10:41:52.828661+0000","last_peered":"2024-09-21T10:41:52.828661+0000","last_clean":"2024-09-21T10:41:52.828661+0000","last_became_active":"2024-09-21T10:40:13.435946+0000","last_became_peered":"2024-09-21T10:40:13.435946+0000","last_unstale":"2024-09-21T10:41:52.828661+0000","last_undegraded":"2024-09-21T10:41:52.828661+0000","last_fullsized":"2024-09-21T10:41:52.828661+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:08:08.670739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":754,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.833005+0000","last_change":"2024-09-21T10:40:13.431693+0000","last_active":"2024-09-21T10:41:52.833005+0000","last_peered":"2024-09-21T10:41:52.833005+0000","last_clean":"2024-09-21T10:41:52.833005+0000","last_became_active":"2024-09-21T10:40:13.431389+0000","last_became_peered":"2024-09-21T10:40:13.431389+0000","last_unstale":"2024-09-21T10:41:52.833005+0000","last_undegraded":"2024-09-21T10:41:52.833005+0000","last_fullsized":"2024-09-21T10:41:52.833005+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:22.227336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1993,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.824440+0000","last_change":"2024-09-21T10:40:30.420482+0000","last_active":"2024-09-21T10:41:52.824440+0000","last_peered":"2024-09-21T10:41:52.824440+0000","last_clean":"2024-09-21T10:41:52.824440+0000","last_became_active":"2024-09-21T10:40:30.419902+0000","last_became_peered":"2024-09-21T10:40:30.419902+0000","last_unstale":"2024-09-21T10:41:52.824440+0000","last_undegraded":"2024-09-21T10:41:52.824440+0000","last_fullsized":"2024-09-21T10:41:52.824440+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:37:40.880948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1967,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.824781+0000","last_change":"2024-09-21T10:09:06.560130+0000","last_active":"2024-09-21T10:41:52.824781+0000","last_peered":"2024-09-21T10:41:52.824781+0000","last_clean":"2024-09-21T10:41:52.824781+0000","last_became_active":"2024-09-21T10:09:06.559827+0000","last_became_peered":"2024-09-21T10:09:06.559827+0000","last_unstale":"2024-09-21T10:41:52.824781+0000","last_undegraded":"2024-09-21T10:41:52.824781+0000","last_fullsized":"2024-09-21T10:41:52.824781+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:05:17.166734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":717,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.828903+0000","last_change":"2024-09-21T10:40:30.417793+0000","last_active":"2024-09-21T10:41:52.828903+0000","last_peered":"2024-09-21T10:41:52.828903+0000","last_clean":"2024-09-21T10:41:52.828903+0000","last_became_active":"2024-09-21T10:40:30.417232+0000","last_became_peered":"2024-09-21T10:40:30.417232+0000","last_unstale":"2024-09-21T10:41:52.828903+0000","last_undegraded":"2024-09-21T10:41:52.828903+0000","last_fullsized":"2024-09-21T10:41:52.828903+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:09:58.959904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1304,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.468579+0000","last_change":"2024-09-21T10:40:30.426430+0000","last_active":"2024-09-21T10:41:34.468579+0000","last_peered":"2024-09-21T10:41:34.468579+0000","last_clean":"2024-09-21T10:41:34.468579+0000","last_became_active":"2024-09-21T10:40:30.426120+0000","last_became_peered":"2024-09-21T10:40:30.426120+0000","last_unstale":"2024-09-21T10:41:34.468579+0000","last_undegraded":"2024-09-21T10:41:34.468579+0000","last_fullsized":"2024-09-21T10:41:34.468579+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:23:15.280912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1948,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.833643+0000","last_change":"2024-09-21T10:09:06.560589+0000","last_active":"2024-09-21T10:41:52.833643+0000","last_peered":"2024-09-21T10:41:52.833643+0000","last_clean":"2024-09-21T10:41:52.833643+0000","last_became_active":"2024-09-21T10:09:06.557552+0000","last_became_peered":"2024-09-21T10:09:06.557552+0000","last_unstale":"2024-09-21T10:41:52.833643+0000","last_undegraded":"2024-09-21T10:41:52.833643+0000","last_fullsized":"2024-09-21T10:41:52.833643+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:09:24.866759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":1350,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:41.021809+0000","last_change":"2024-09-21T10:40:30.431163+0000","last_active":"2024-09-21T10:41:41.021809+0000","last_peered":"2024-09-21T10:41:41.021809+0000","last_clean":"2024-09-21T10:41:41.021809+0000","last_became_active":"2024-09-21T10:40:30.424471+0000","last_became_peered":"2024-09-21T10:40:30.424471+0000","last_unstale":"2024-09-21T10:41:41.021809+0000","last_undegraded":"2024-09-21T10:41:41.021809+0000","last_fullsized":"2024-09-21T10:41:41.021809+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:02:45.899239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":691,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.824764+0000","last_change":"2024-09-21T10:40:33.444446+0000","last_active":"2024-09-21T10:41:52.824764+0000","last_peered":"2024-09-21T10:41:52.824764+0000","last_clean":"2024-09-21T10:41:52.824764+0000","last_became_active":"2024-09-21T10:40:33.443958+0000","last_became_peered":"2024-09-21T10:40:33.443958+0000","last_unstale":"2024-09-21T10:41:52.824764+0000","last_undegraded":"2024-09-21T10:41:52.824764+0000","last_fullsized":"2024-09-21T10:41:52.824764+0000","mapping_epoch":1315,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:01:05.119015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1953,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.836137+0000","last_change":"2024-09-21T10:09:06.538940+0000","last_active":"2024-09-21T10:41:52.836137+0000","last_peered":"2024-09-21T10:41:52.836137+0000","last_clean":"2024-09-21T10:41:52.836137+0000","last_became_active":"2024-09-21T10:09:06.538668+0000","last_became_peered":"2024-09-21T10:09:06.538668+0000","last_unstale":"2024-09-21T10:41:52.836137+0000","last_undegraded":"2024-09-21T10:41:52.836137+0000","last_fullsized":"2024-09-21T10:41:52.836137+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:42:44.299488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1961,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.832881+0000","last_change":"2024-09-21T10:40:30.419468+0000","last_active":"2024-09-21T10:41:52.832881+0000","last_peered":"2024-09-21T10:41:52.832881+0000","last_clean":"2024-09-21T10:41:52.832881+0000","last_became_active":"2024-09-21T10:40:30.419140+0000","last_became_peered":"2024-09-21T10:40:30.419140+0000","last_unstale":"2024-09-21T10:41:52.832881+0000","last_undegraded":"2024-09-21T10:41:52.832881+0000","last_fullsized":"2024-09-21T10:41:52.832881+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:37.424900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1307,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.474805+0000","last_change":"2024-09-21T10:07:55.998799+0000","last_active":"2024-09-21T10:41:34.474805+0000","last_peered":"2024-09-21T10:41:34.474805+0000","last_clean":"2024-09-21T10:41:34.474805+0000","last_became_active":"2024-09-21T10:07:55.998510+0000","last_became_peered":"2024-09-21T10:07:55.998510+0000","last_unstale":"2024-09-21T10:41:34.474805+0000","last_undegraded":"2024-09-21T10:41:34.474805+0000","last_fullsized":"2024-09-21T10:41:34.474805+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:10.474008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1327,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.475233+0000","last_change":"2024-09-21T10:40:30.419793+0000","last_active":"2024-09-21T10:41:34.475233+0000","last_peered":"2024-09-21T10:41:34.475233+0000","last_clean":"2024-09-21T10:41:34.475233+0000","last_became_active":"2024-09-21T10:40:30.419149+0000","last_became_peered":"2024-09-21T10:40:30.419149+0000","last_unstale":"2024-09-21T10:41:34.475233+0000","last_undegraded":"2024-09-21T10:41:34.475233+0000","last_fullsized":"2024-09-21T10:41:34.475233+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:33.741419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1331,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.474435+0000","last_change":"2024-09-21T10:40:13.432387+0000","last_active":"2024-09-21T10:41:34.474435+0000","last_peered":"2024-09-21T10:41:34.474435+0000","last_clean":"2024-09-21T10:41:34.474435+0000","last_became_active":"2024-09-21T10:40:13.432053+0000","last_became_peered":"2024-09-21T10:40:13.432053+0000","last_unstale":"2024-09-21T10:41:34.474435+0000","last_undegraded":"2024-09-21T10:41:34.474435+0000","last_fullsized":"2024-09-21T10:41:34.474435+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:37.486101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1310,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.468455+0000","last_change":"2024-09-21T10:07:56.020791+0000","last_active":"2024-09-21T10:41:34.468455+0000","last_peered":"2024-09-21T10:41:34.468455+0000","last_clean":"2024-09-21T10:41:34.468455+0000","last_became_active":"2024-09-21T10:07:56.020205+0000","last_became_peered":"2024-09-21T10:07:56.020205+0000","last_unstale":"2024-09-21T10:41:34.468455+0000","last_undegraded":"2024-09-21T10:41:34.468455+0000","last_fullsized":"2024-09-21T10:41:34.468455+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:09.813166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1307,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.474378+0000","last_change":"2024-09-21T10:09:06.533271+0000","last_active":"2024-09-21T10:41:34.474378+0000","last_peered":"2024-09-21T10:41:34.474378+0000","last_clean":"2024-09-21T10:41:34.474378+0000","last_became_active":"2024-09-21T10:09:06.532041+0000","last_became_peered":"2024-09-21T10:09:06.532041+0000","last_unstale":"2024-09-21T10:41:34.474378+0000","last_undegraded":"2024-09-21T10:41:34.474378+0000","last_fullsized":"2024-09-21T10:41:34.474378+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:04.236375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":1969,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.833418+0000","last_change":"2024-09-21T10:09:06.543422+0000","last_active":"2024-09-21T10:41:52.833418+0000","last_peered":"2024-09-21T10:41:52.833418+0000","last_clean":"2024-09-21T10:41:52.833418+0000","last_became_active":"2024-09-21T10:09:06.539647+0000","last_became_peered":"2024-09-21T10:09:06.539647+0000","last_unstale":"2024-09-21T10:41:52.833418+0000","last_undegraded":"2024-09-21T10:41:52.833418+0000","last_fullsized":"2024-09-21T10:41:52.833418+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:04.990085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1968,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.825025+0000","last_change":"2024-09-21T10:07:56.018560+0000","last_active":"2024-09-21T10:41:52.825025+0000","last_peered":"2024-09-21T10:41:52.825025+0000","last_clean":"2024-09-21T10:41:52.825025+0000","last_became_active":"2024-09-21T10:07:56.018261+0000","last_became_peered":"2024-09-21T10:07:56.018261+0000","last_unstale":"2024-09-21T10:41:52.825025+0000","last_undegraded":"2024-09-21T10:41:52.825025+0000","last_fullsized":"2024-09-21T10:41:52.825025+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:04:54.395456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1308,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.475944+0000","last_change":"2024-09-21T10:09:06.531935+0000","last_active":"2024-09-21T10:41:34.475944+0000","last_peered":"2024-09-21T10:41:34.475944+0000","last_clean":"2024-09-21T10:41:34.475944+0000","last_became_active":"2024-09-21T10:09:06.531610+0000","last_became_peered":"2024-09-21T10:09:06.531610+0000","last_unstale":"2024-09-21T10:41:34.475944+0000","last_undegraded":"2024-09-21T10:41:34.475944+0000","last_fullsized":"2024-09-21T10:41:34.475944+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:33.874405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1336,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.468288+0000","last_change":"2024-09-21T10:40:30.423107+0000","last_active":"2024-09-21T10:41:34.468288+0000","last_peered":"2024-09-21T10:41:34.468288+0000","last_clean":"2024-09-21T10:41:34.468288+0000","last_became_active":"2024-09-21T10:40:30.422770+0000","last_became_peered":"2024-09-21T10:40:30.422770+0000","last_unstale":"2024-09-21T10:41:34.468288+0000","last_undegraded":"2024-09-21T10:41:34.468288+0000","last_fullsized":"2024-09-21T10:41:34.468288+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:35:23.752720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":54,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.475857+0000","last_change":"2024-09-21T10:40:30.428793+0000","last_active":"2024-09-21T10:41:34.475857+0000","last_peered":"2024-09-21T10:41:34.475857+0000","last_clean":"2024-09-21T10:41:34.475857+0000","last_became_active":"2024-09-21T10:40:30.421116+0000","last_became_peered":"2024-09-21T10:40:30.421116+0000","last_unstale":"2024-09-21T10:41:34.475857+0000","last_undegraded":"2024-09-21T10:41:34.475857+0000","last_fullsized":"2024-09-21T10:41:34.475857+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:24:49.342671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1323,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.475110+0000","last_change":"2024-09-21T10:09:06.559132+0000","last_active":"2024-09-21T10:41:34.475110+0000","last_peered":"2024-09-21T10:41:34.475110+0000","last_clean":"2024-09-21T10:41:34.475110+0000","last_became_active":"2024-09-21T10:09:06.558395+0000","last_became_peered":"2024-09-21T10:09:06.558395+0000","last_unstale":"2024-09-21T10:41:34.475110+0000","last_undegraded":"2024-09-21T10:41:34.475110+0000","last_fullsized":"2024-09-21T10:41:34.475110+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:27:00.073554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":54,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.474777+0000","last_change":"2024-09-21T10:40:30.428625+0000","last_active":"2024-09-21T10:41:34.474777+0000","last_peered":"2024-09-21T10:41:34.474777+0000","last_clean":"2024-09-21T10:41:34.474777+0000","last_became_active":"2024-09-21T10:40:30.421565+0000","last_became_peered":"2024-09-21T10:40:30.421565+0000","last_unstale":"2024-09-21T10:41:34.474777+0000","last_undegraded":"2024-09-21T10:41:34.474777+0000","last_fullsized":"2024-09-21T10:41:34.474777+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:23.191395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1302,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.475784+0000","last_change":"2024-09-21T10:40:30.421601+0000","last_active":"2024-09-21T10:41:34.475784+0000","last_peered":"2024-09-21T10:41:34.475784+0000","last_clean":"2024-09-21T10:41:34.475784+0000","last_became_active":"2024-09-21T10:40:30.420430+0000","last_became_peered":"2024-09-21T10:40:30.420430+0000","last_unstale":"2024-09-21T10:41:34.475784+0000","last_undegraded":"2024-09-21T10:41:34.475784+0000","last_fullsized":"2024-09-21T10:41:34.475784+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:15.416335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":54,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.476390+0000","last_change":"2024-09-21T10:40:30.433050+0000","last_active":"2024-09-21T10:41:34.476390+0000","last_peered":"2024-09-21T10:41:34.476390+0000","last_clean":"2024-09-21T10:41:34.476390+0000","last_became_active":"2024-09-21T10:40:30.421211+0000","last_became_peered":"2024-09-21T10:40:30.421211+0000","last_unstale":"2024-09-21T10:41:34.476390+0000","last_undegraded":"2024-09-21T10:41:34.476390+0000","last_fullsized":"2024-09-21T10:41:34.476390+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:01:33.357050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1317,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.468234+0000","last_change":"2024-09-21T10:07:21.497877+0000","last_active":"2024-09-21T10:41:34.468234+0000","last_peered":"2024-09-21T10:41:34.468234+0000","last_clean":"2024-09-21T10:41:34.468234+0000","last_became_active":"2024-09-21T10:07:21.496317+0000","last_became_peered":"2024-09-21T10:07:21.496317+0000","last_unstale":"2024-09-21T10:41:34.468234+0000","last_undegraded":"2024-09-21T10:41:34.468234+0000","last_fullsized":"2024-09-21T10:41:34.468234+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:39:03.848183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1957,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.836246+0000","last_change":"2024-09-21T10:40:30.418410+0000","last_active":"2024-09-21T10:41:52.836246+0000","last_peered":"2024-09-21T10:41:52.836246+0000","last_clean":"2024-09-21T10:41:52.836246+0000","last_became_active":"2024-09-21T10:40:30.417846+0000","last_became_peered":"2024-09-21T10:40:30.417846+0000","last_unstale":"2024-09-21T10:41:52.836246+0000","last_undegraded":"2024-09-21T10:41:52.836246+0000","last_fullsized":"2024-09-21T10:41:52.836246+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:36:02.636480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":38,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.474976+0000","last_change":"2024-09-21T10:40:30.423721+0000","last_active":"2024-09-21T10:41:34.474976+0000","last_peered":"2024-09-21T10:41:34.474976+0000","last_clean":"2024-09-21T10:41:34.474976+0000","last_became_active":"2024-09-21T10:40:30.423415+0000","last_became_peered":"2024-09-21T10:40:30.423415+0000","last_unstale":"2024-09-21T10:41:34.474976+0000","last_undegraded":"2024-09-21T10:41:34.474976+0000","last_fullsized":"2024-09-21T10:41:34.474976+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:57.212463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1988,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.824172+0000","last_change":"2024-09-21T10:40:30.418240+0000","last_active":"2024-09-21T10:41:52.824172+0000","last_peered":"2024-09-21T10:41:52.824172+0000","last_clean":"2024-09-21T10:41:52.824172+0000","last_became_active":"2024-09-21T10:40:30.417843+0000","last_became_peered":"2024-09-21T10:40:30.417843+0000","last_unstale":"2024-09-21T10:41:52.824172+0000","last_undegraded":"2024-09-21T10:41:52.824172+0000","last_fullsized":"2024-09-21T10:41:52.824172+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:50.860905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1971,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.827942+0000","last_change":"2024-09-21T10:40:30.419513+0000","last_active":"2024-09-21T10:41:52.827942+0000","last_peered":"2024-09-21T10:41:52.827942+0000","last_clean":"2024-09-21T10:41:52.827942+0000","last_became_active":"2024-09-21T10:40:30.419182+0000","last_became_peered":"2024-09-21T10:40:30.419182+0000","last_unstale":"2024-09-21T10:41:52.827942+0000","last_undegraded":"2024-09-21T10:41:52.827942+0000","last_fullsized":"2024-09-21T10:41:52.827942+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:38:19.185388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1315,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.476013+0000","last_change":"2024-09-21T10:40:30.422480+0000","last_active":"2024-09-21T10:41:34.476013+0000","last_peered":"2024-09-21T10:41:34.476013+0000","last_clean":"2024-09-21T10:41:34.476013+0000","last_became_active":"2024-09-21T10:40:30.421818+0000","last_became_peered":"2024-09-21T10:40:30.421818+0000","last_unstale":"2024-09-21T10:41:34.476013+0000","last_undegraded":"2024-09-21T10:41:34.476013+0000","last_fullsized":"2024-09-21T10:41:34.476013+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:51:14.217028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":56,"reported_epoch":1338,"state":"active+clean","last_fresh":"2024-09-21T10:41:34.475790+0000","last_change":"2024-09-21T10:40:30.440162+0000","last_active":"2024-09-21T10:41:34.475790+0000","last_peered":"2024-09-21T10:41:34.475790+0000","last_clean":"2024-09-21T10:41:34.475790+0000","last_became_active":"2024-09-21T10:40:30.427218+0000","last_became_peered":"2024-09-21T10:40:30.427218+0000","last_unstale":"2024-09-21T10:41:34.475790+0000","last_undegraded":"2024-09-21T10:41:34.475790+0000","last_fullsized":"2024-09-21T10:41:34.475790+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:25:22.251409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1958,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-09-21T10:41:52.833590+0000","last_change":"2024-09-21T10:40:30.418759+0000","last_active":"2024-09-21T10:41:52.833590+0000","last_peered":"2024-09-21T10:41:52.833590+0000","last_clean":"2024-09-21T10:41:52.833590+0000","last_became_active":"2024-09-21T10:40:30.418325+0000","last_became_peered":"2024-09-21T10:40:30.418325+0000","last_unstale":"2024-09-21T10:41:52.833590+0000","last_undegraded":"2024-09-21T10:41:52.833590+0000","last_fullsized":"2024-09-21T10:41:52.833590+0000","mapping_epoch":1312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:57.793473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":53,"num_pg":26,"stat_sum":{"num_bytes":8426455040,"num_objects":128578,"num_object_clones":0,"num_object_copies":385734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7460,"num_objects_misplaced":7612,"num_objects_unfound":0,"num_objects_dirty":128578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97250,"num_write_kb":6200608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4455,"num_bytes_recovered":291962880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":25877487616,"data_stored":25877479472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":164327,"ondisk_log_size":164327,"up":78,"acting":75,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":5554,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":21634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234575,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496212,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2923760,"kb_used_data":2740136,"kb_used_omap":1,"kb_used_meta":183614,"kb_avail":90821392,"statfs":{"total":95995035648,"available":93001105408,"internally_reserved":0,"allocated":2805899264,"data_stored":2801879967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":188021194},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757849,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3839072,"kb_used_data":3659416,"kb_used_omap":1,"kb_used_meta":179582,"kb_avail":89906080,"statfs":{"total":95995035648,"available":92063825920,"internally_reserved":0,"allocated":3747241984,"data_stored":3743233761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":183892422},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019485,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6165612,"kb_used_data":5885984,"kb_used_omap":1,"kb_used_meta":279614,"kb_avail":87579540,"statfs":{"total":95995035648,"available":89681448960,"internally_reserved":0,"allocated":6027247616,"data_stored":6023235408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":286325190},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369228,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3599384,"kb_used_data":3436752,"kb_used_omap":69,"kb_used_meta":162554,"kb_avail":90145768,"statfs":{"total":95995035648,"available":92309266432,"internally_reserved":0,"allocated":3519234048,"data_stored":3515233494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":166455426},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":2920577761684,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807957914,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6051804,"kb_used_data":5776852,"kb_used_omap":66,"kb_used_meta":274877,"kb_avail":87693348,"statfs":{"total":95995035648,"available":89797988352,"internally_reserved":0,"allocated":5915496448,"data_stored":5911492223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68589,"internal_metadata":281474067},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":666,"seq":2860448219553,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3968368,"kb_used_data":3779688,"kb_used_omap":76,"kb_used_meta":188595,"kb_avail":89776784,"statfs":{"total":95995035648,"available":91931426816,"internally_reserved":0,"allocated":3870400512,"data_stored":3866381250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78716,"internal_metadata":193121412},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3861577728,"data_stored":3861577728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5923471360,"data_stored":5923471360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3533570048,"data_stored":3533570048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5981736960,"data_stored":5981732888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3716026368,"data_stored":3716022296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2780168192,"data_stored":2780168192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":80936960,"data_stored":80936960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:42:04.299 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_0 pgp_num 26 2024-09-21T10:42:04.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:03 smithi160 ceph-mon[101879]: pgmap v2234: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.6 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 102 B/s rd, 2.8 MiB/s wr, 569 op/s; 7196/371235 objects degraded (1.938%); 6781/371235 objects misplaced (1.827%); 3.3 MiB/s, 52 objects/s recovering 2024-09-21T10:42:04.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:03 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 7196/371235 objects degraded (1.938%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:42:04.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:03 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2589685675' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:42:04.365 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:04.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:04.391 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:04.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:04.492 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:04.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:04.536 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:04.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:04.637 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:04.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:04.663 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:04.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:04.764 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:04.803 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:04.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:04.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:04.905 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:04.928 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:04.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:04.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:05.029 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:05.061 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:05.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:05.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:05.162 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:05.188 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:05.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:05.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:05.290 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:05.324 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:05.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:05.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:05.426 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:05.475 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:05.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:05.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:05.576 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:05.599 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:05.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:05.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:05.700 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:05.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:05.747 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:05.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:05.848 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:05.875 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:05.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:05.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:05.976 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:06.012 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:06.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:06.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:06.114 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:06.119 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:05 smithi070 ceph-mon[107385]: pgmap v2235: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.4 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 86 B/s rd, 2.4 MiB/s wr, 754 op/s; 6915/361425 objects degraded (1.913%); 6116/361425 objects misplaced (1.692%); 4.4 MiB/s, 70 objects/s recovering 2024-09-21T10:42:06.120 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:42:06.120 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:05 smithi070 ceph-mon[112139]: pgmap v2235: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.4 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 86 B/s rd, 2.4 MiB/s wr, 754 op/s; 6915/361425 objects degraded (1.913%); 6116/361425 objects misplaced (1.692%); 4.4 MiB/s, 70 objects/s recovering 2024-09-21T10:42:06.120 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:42:06.177 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:06.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:06.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:06.279 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:06.303 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:06.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:06.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:05 smithi160 ceph-mon[101879]: pgmap v2235: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.4 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 86 B/s rd, 2.4 MiB/s wr, 754 op/s; 6915/361425 objects degraded (1.913%); 6116/361425 objects misplaced (1.692%); 4.4 MiB/s, 70 objects/s recovering 2024-09-21T10:42:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:42:06.404 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:06.412 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:06.428 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:06.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:06.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:06.529 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:06.572 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:06.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:06.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:06.674 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:06.736 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:06.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:06.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:06.839 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:06.866 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:06.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:06.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:06.968 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:07.017 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:07.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:07.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:07.119 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:07.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:07.158 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:07.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:07.260 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:07.300 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:07.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:07.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:07.402 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:07.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:07.432 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:07.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:07.491 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:07.534 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:07.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:07.580 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:07.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:07.681 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:07.707 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:07.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:07.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:07.809 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:07.855 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:07.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:07.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:07.861 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T10:42:07.886 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:07 smithi070 ceph-mon[107385]: pgmap v2236: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.2 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 85 B/s rd, 0 B/s wr, 662 op/s; 6915/354915 objects degraded (1.948%); 6116/354915 objects misplaced (1.723%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:42:07.886 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:07 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3149779524' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:42:07.886 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:07 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:42:07.887 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:07 smithi070 ceph-mon[112139]: pgmap v2236: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.2 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 85 B/s rd, 0 B/s wr, 662 op/s; 6915/354915 objects degraded (1.948%); 6116/354915 objects misplaced (1.723%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:42:07.887 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:07 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3149779524' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:42:07.887 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:07 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:42:07.957 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:07.986 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:07.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:07.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:08.088 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:08.120 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:08.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:08.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:08.164 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:42:08.222 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:08.329 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:08.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:08.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:08.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:07 smithi160 ceph-mon[101879]: pgmap v2236: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.2 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 85 B/s rd, 0 B/s wr, 662 op/s; 6915/354915 objects degraded (1.948%); 6116/354915 objects misplaced (1.723%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:42:08.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:07 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3149779524' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:42:08.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:07 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:42:08.432 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:08.477 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:08.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:08.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:08.579 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:08.609 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:08.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:08.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:08.710 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:08.748 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:08.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:08.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:08.849 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:08.876 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:08.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:08.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:08.977 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:09.021 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:09.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:09.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:09.123 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:08 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:42:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:08 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:42:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:08 smithi070 ceph-mon[107385]: osdmap e1340: 8 total, 8 up, 6 in 2024-09-21T10:42:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:08 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 6915/354915 objects degraded (1.948%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:42:09.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:08 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:42:09.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:08 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:42:09.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:08 smithi070 ceph-mon[112139]: osdmap e1340: 8 total, 8 up, 6 in 2024-09-21T10:42:09.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:08 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 6915/354915 objects degraded (1.948%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:42:09.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:09.165 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:09.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:09.266 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:09.294 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:09.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:09.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:09.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:08 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:42:09.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:08 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:42:09.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:08 smithi160 ceph-mon[101879]: osdmap e1340: 8 total, 8 up, 6 in 2024-09-21T10:42:09.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:08 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 6915/354915 objects degraded (1.948%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:42:09.396 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:09.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:09.428 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:09.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:09.529 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:09.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:09.558 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:09.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:09.659 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:09.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:09.693 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:09.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:09.794 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:09.829 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:09.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:09.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:09.930 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:09.962 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:09.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:09.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:10.064 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:10.101 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:10.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:10.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:10.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:09 smithi070 ceph-mon[107385]: pgmap v2238: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.2 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 102 B/s rd, 122 KiB/s wr, 1.03k op/s; 6915/354915 objects degraded (1.948%); 6116/354915 objects misplaced (1.723%); 4.2 MiB/s, 67 objects/s recovering 2024-09-21T10:42:10.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:09 smithi070 ceph-mon[112139]: pgmap v2238: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.2 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 102 B/s rd, 122 KiB/s wr, 1.03k op/s; 6915/354915 objects degraded (1.948%); 6116/354915 objects misplaced (1.723%); 4.2 MiB/s, 67 objects/s recovering 2024-09-21T10:42:10.203 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:10.232 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:10.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:10.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:10.333 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:10.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:09 smithi160 ceph-mon[101879]: pgmap v2238: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.2 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 102 B/s rd, 122 KiB/s wr, 1.03k op/s; 6915/354915 objects degraded (1.948%); 6116/354915 objects misplaced (1.723%); 4.2 MiB/s, 67 objects/s recovering 2024-09-21T10:42:10.354 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:10.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:10.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:10.456 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:10.499 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:10.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:10.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:10.600 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:10.639 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:10.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:10.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:10.740 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:10.778 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:10.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:10.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:10.880 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:10.915 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:10.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:10.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:11.016 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:11.058 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:11.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:11.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:11.160 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:11.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:11.197 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:11.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:11.298 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:11.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:11.327 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:11.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:11.428 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:11.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:11.449 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:11.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:11.550 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:11.578 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:11.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:11.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:11.608 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:42:11.679 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:11.700 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:11.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:11.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:11.802 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:11.838 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:11.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:11.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:11.945 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:11.984 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:11.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:11.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:12.086 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:12.117 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:12.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:12.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:11 smithi070 ceph-mon[107385]: pgmap v2239: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.0 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 6574/345654 objects degraded (1.902%); 5339/345654 objects misplaced (1.545%); 4.3 MiB/s, 68 objects/s recovering 2024-09-21T10:42:12.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:11 smithi070 ceph-mon[112139]: pgmap v2239: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.0 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 6574/345654 objects degraded (1.902%); 5339/345654 objects misplaced (1.545%); 4.3 MiB/s, 68 objects/s recovering 2024-09-21T10:42:12.218 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:12.252 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:12.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:12.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:12.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:11 smithi160 ceph-mon[101879]: pgmap v2239: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.0 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 6574/345654 objects degraded (1.902%); 5339/345654 objects misplaced (1.545%); 4.3 MiB/s, 68 objects/s recovering 2024-09-21T10:42:12.353 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:12.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:12.419 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:12.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:12.520 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:12.557 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:12.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:12.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:12.659 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:12.687 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:12.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:12.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:12.789 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:12.814 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:12.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:12.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:12.916 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:12.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:12.941 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:12.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:13.042 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:13.074 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:13.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:13.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:13.176 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:13.213 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:13.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:13.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:13.315 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:13.362 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:13.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:13.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:13.427 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:42:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:42:13] "GET /metrics HTTP/1.1" 200 36630 "" "Prometheus/2.51.0" 2024-09-21T10:42:13.464 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:13.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:13.493 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:13.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:13.593 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:13.614 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:13.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:13.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:13.716 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:13.752 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:13.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:13.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:13.853 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:13.893 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:13.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:13.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:13.995 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:14.021 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:14.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:14.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:14.123 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:14.155 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:14.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:14.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:14.257 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:14.280 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:14.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:14.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:13 smithi160 ceph-mon[101879]: pgmap v2240: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.9 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 6574/339147 objects degraded (1.938%); 5339/339147 objects misplaced (1.574%); 4.3 MiB/s, 68 objects/s recovering 2024-09-21T10:42:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:13 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 6574/339147 objects degraded (1.938%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:42:14.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:13 smithi070 ceph-mon[107385]: pgmap v2240: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.9 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 6574/339147 objects degraded (1.938%); 5339/339147 objects misplaced (1.574%); 4.3 MiB/s, 68 objects/s recovering 2024-09-21T10:42:14.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:13 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 6574/339147 objects degraded (1.938%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:42:14.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:13 smithi070 ceph-mon[112139]: pgmap v2240: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.9 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 6574/339147 objects degraded (1.938%); 5339/339147 objects misplaced (1.574%); 4.3 MiB/s, 68 objects/s recovering 2024-09-21T10:42:14.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:13 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 6574/339147 objects degraded (1.938%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:42:14.382 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:14.414 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:14.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:14.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:14.427 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:42:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T10:42:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T10:42:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T10:42:14.428 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:42:14.516 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:14.546 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:14.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:14.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:14.648 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:14.674 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:14.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:14.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:14.776 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:14.801 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:14.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:14.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:14.903 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:14.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:14.931 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:14.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:15.031 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:15.065 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:15.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:15.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:15.168 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:15.193 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:15.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:15.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:15.295 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:15.332 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:15.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:15.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:15.434 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:15.466 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:15.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:15.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:15.568 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:15.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:15.589 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:15.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:15.690 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:15.732 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:15.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:15.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:15.833 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:15.855 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:15.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:15.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:15.957 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:15.985 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:15.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:15.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:16.087 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:16.115 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:16.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:16.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:16.216 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:16.241 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:16.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:16.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:16.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:15 smithi160 ceph-mon[101879]: pgmap v2241: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.7 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 1.04k op/s; 6260/330264 objects degraded (1.895%); 4600/330264 objects misplaced (1.393%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:42:16.342 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:16.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:15 smithi070 ceph-mon[107385]: pgmap v2241: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.7 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 1.04k op/s; 6260/330264 objects degraded (1.895%); 4600/330264 objects misplaced (1.393%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:42:16.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:15 smithi070 ceph-mon[112139]: pgmap v2241: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.7 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 1.04k op/s; 6260/330264 objects degraded (1.895%); 4600/330264 objects misplaced (1.393%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:42:16.383 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:16.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:16.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:16.485 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:16.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:16.522 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:16.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:16.623 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:16.663 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:16.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:16.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:16.765 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:16.807 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:16.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:16.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:16.909 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:16.957 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:16.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:16.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:17.059 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:17.084 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:17.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:17.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:17.186 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:17.207 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:17.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:17.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:17.308 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:17.347 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:17.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:17.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:17.448 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:17.490 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:17.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:17.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:17.591 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:17.627 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:17.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:17.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:17.729 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:17.764 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:17.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:17.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:17.866 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:17.889 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:17.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:17.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:17.991 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:18.016 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:18.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:18.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:18.061 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:18.117 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:18.141 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:18.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:18.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:18.207 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:17 smithi070 ceph-mon[107385]: pgmap v2242: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 1.02k op/s; 6260/324399 objects degraded (1.930%); 4600/324399 objects misplaced (1.418%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:42:18.208 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:17 smithi070 ceph-mon[112139]: pgmap v2242: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 1.02k op/s; 6260/324399 objects degraded (1.930%); 4600/324399 objects misplaced (1.418%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:42:18.243 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:18.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:18.290 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:18.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:17 smithi160 ceph-mon[101879]: pgmap v2242: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 1.02k op/s; 6260/324399 objects degraded (1.930%); 4600/324399 objects misplaced (1.418%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:42:18.391 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:18.415 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:18.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:18.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:18.516 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:18.542 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:18.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:18.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:18.644 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:18.679 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:18.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:18.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:18.780 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:18.807 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:18.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:18.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:18.875 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:42:18.903 INFO:teuthology.orchestra.run.smithi070.stderr:set pool 53 pgp_num to 26 2024-09-21T10:42:18.909 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:18.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:18.937 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:18.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:19.038 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:19.062 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:19.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:19.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:19.131 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:18 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 6260/324399 objects degraded (1.930%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:42:19.131 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:18 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2107991465' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "26"} : dispatch 2024-09-21T10:42:19.131 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:18 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "26"} : dispatch 2024-09-21T10:42:19.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:18 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 6260/324399 objects degraded (1.930%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:42:19.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:18 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2107991465' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "26"} : dispatch 2024-09-21T10:42:19.131 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:18 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "26"} : dispatch 2024-09-21T10:42:19.164 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:19.210 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:19.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:19.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:19.312 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:19.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:18 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 6260/324399 objects degraded (1.930%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:42:19.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:18 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2107991465' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "26"} : dispatch 2024-09-21T10:42:19.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:18 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "26"} : dispatch 2024-09-21T10:42:19.348 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:19.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:19.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:19.449 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:19.492 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:19.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:19.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:19.594 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:19.642 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:19.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:19.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:19.744 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:19.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:19.779 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:19.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:19.880 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:19.906 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:19.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:19.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:20.008 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:20.055 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:20.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:20.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:20.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:19 smithi070 ceph-mon[107385]: pgmap v2243: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 937 op/s; 6260/324399 objects degraded (1.930%); 4600/324399 objects misplaced (1.418%); 4.2 MiB/s, 66 objects/s recovering 2024-09-21T10:42:20.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:19 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "26"}]': finished 2024-09-21T10:42:20.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:19 smithi070 ceph-mon[107385]: osdmap e1341: 8 total, 8 up, 6 in 2024-09-21T10:42:20.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:19 smithi070 ceph-mon[112139]: pgmap v2243: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 937 op/s; 6260/324399 objects degraded (1.930%); 4600/324399 objects misplaced (1.418%); 4.2 MiB/s, 66 objects/s recovering 2024-09-21T10:42:20.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:19 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "26"}]': finished 2024-09-21T10:42:20.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:19 smithi070 ceph-mon[112139]: osdmap e1341: 8 total, 8 up, 6 in 2024-09-21T10:42:20.157 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:20.188 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:20.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:20.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:20.289 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:20.330 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:20.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:20.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:20.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:19 smithi160 ceph-mon[101879]: pgmap v2243: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 937 op/s; 6260/324399 objects degraded (1.930%); 4600/324399 objects misplaced (1.418%); 4.2 MiB/s, 66 objects/s recovering 2024-09-21T10:42:20.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:19 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "26"}]': finished 2024-09-21T10:42:20.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:19 smithi160 ceph-mon[101879]: osdmap e1341: 8 total, 8 up, 6 in 2024-09-21T10:42:20.432 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:20.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:20.470 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:20.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:20.476 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:20.571 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:20.598 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:20.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:20.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:20.700 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:20.730 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:20.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:20.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:20.832 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:20.859 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:20.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:20.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:20.960 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:21.001 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:21.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:21.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:21.103 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:21.140 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:21.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:21.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:21.201 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:42:21.201 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:42:21.237 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:42:21.237 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:42:21.242 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:21.268 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:21.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:21.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:21.289 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:42:21.290 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:42:21.345 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:42:21.346 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:42:21.370 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:21.379 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:42:21.379 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:42:21.397 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:21.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:21.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:21.424 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:42:21.424 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:42:21.464 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:42:21.464 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:42:21.498 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:21.501 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:42:21.501 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:42:21.521 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:42:21.521 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:42:21.521 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:21 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1933751086' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:42:21.522 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:42:21.522 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:42:21.522 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:21 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1933751086' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:42:21.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:21.539 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:21.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:21.640 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:21.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:21.671 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:21.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:21.772 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:21.800 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:21.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:21.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:42:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:42:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:21 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1933751086' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:42:21.902 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:21.933 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:21.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:21.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:22.035 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:22.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:22.078 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:22.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:22.179 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:22.216 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:22.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:22.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:22.318 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:22.341 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:22.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:22.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:22.443 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:22.466 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:22.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:22.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:22.567 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:22.592 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:22.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:22.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:22 smithi070 ceph-mon[107385]: pgmap v2245: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.4 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 1.01k op/s; 5944/315432 objects degraded (1.884%); 3797/315432 objects misplaced (1.204%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:42:22.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:22 smithi070 ceph-mon[112139]: pgmap v2245: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.4 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 1.01k op/s; 5944/315432 objects degraded (1.884%); 3797/315432 objects misplaced (1.204%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:42:22.694 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:22.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:22.729 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:22.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:22.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:22 smithi160 ceph-mon[101879]: pgmap v2245: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.4 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 1.01k op/s; 5944/315432 objects degraded (1.884%); 3797/315432 objects misplaced (1.204%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:42:22.830 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:22.859 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:22.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:22.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:22.961 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:22.999 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:23.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:23.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:23.101 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:23.128 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:23.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:23.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:23.229 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:23.254 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:23.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:23.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:23.355 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:23.397 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:23.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:23.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:23.499 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:23.528 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:42:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:42:23] "GET /metrics HTTP/1.1" 200 36628 "" "Prometheus/2.51.0" 2024-09-21T10:42:23.528 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:23 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 5944/315432 objects degraded (1.884%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:42:23.528 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:23 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 5944/315432 objects degraded (1.884%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:42:23.538 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:23.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:23.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:23.640 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:23.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:23.667 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:23.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:23.768 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:23.802 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:23.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:23.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:23 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 5944/315432 objects degraded (1.884%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:42:23.904 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:23.933 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:23.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:23.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:24.034 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:24.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:24.072 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:24.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:24.173 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:24.201 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:24.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:24.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:24.302 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:24.354 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:24.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:24.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:24.455 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:24.477 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:24.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:24.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:24.579 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:24.610 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:24.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:24.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:24.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:24 smithi070 ceph-mon[107385]: pgmap v2246: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 1.00k op/s; 5944/309096 objects degraded (1.923%); 3797/309096 objects misplaced (1.228%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:42:24.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:24 smithi070 ceph-mon[112139]: pgmap v2246: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 1.00k op/s; 5944/309096 objects degraded (1.923%); 3797/309096 objects misplaced (1.228%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:42:24.712 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:24.732 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:24.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:24.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:24.834 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:24 smithi160 ceph-mon[101879]: pgmap v2246: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 1.00k op/s; 5944/309096 objects degraded (1.923%); 3797/309096 objects misplaced (1.228%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:42:24.873 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:24.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:24.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:24.974 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:24.995 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:24.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:24.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:25.097 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:25.154 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:25.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:25.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:25.256 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:25.290 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:25.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:25.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:25.391 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:25.412 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:25.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:25.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:25.514 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:25.518 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6] out_osds: [2, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:42:25.518 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:42:25.519 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-21T10:42:25.519 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.0 'trim stale osdmaps' 2024-09-21T10:42:25.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:25.550 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:25.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:25.623 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:25 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:42:25.623 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:25 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:42:25.624 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:25 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:42:25.624 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:25 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:42:25.624 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:25 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:42:25.624 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:25 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:42:25.651 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:25.687 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:25.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:25.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:25.788 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:25 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:42:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:25 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:42:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:25 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:42:25.845 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:25.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:25.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:25.947 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:25.973 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:25.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:25.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:26.075 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:26.098 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:26.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:26.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:26.200 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:26.240 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:26.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:26.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:26.342 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:26.378 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:26.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:26.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:26.480 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:26.517 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:26.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:26.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:26.618 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:26.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:26.656 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:26.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:26.757 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:26.793 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:26.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:26.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:26 smithi160 ceph-mon[101879]: pgmap v2247: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 995 op/s; 5636/300399 objects degraded (1.876%); 2908/300399 objects misplaced (0.968%); 4.8 MiB/s, 77 objects/s recovering 2024-09-21T10:42:26.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:26 smithi070 ceph-mon[107385]: pgmap v2247: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 995 op/s; 5636/300399 objects degraded (1.876%); 2908/300399 objects misplaced (0.968%); 4.8 MiB/s, 77 objects/s recovering 2024-09-21T10:42:26.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:26 smithi070 ceph-mon[112139]: pgmap v2247: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 995 op/s; 5636/300399 objects degraded (1.876%); 2908/300399 objects misplaced (0.968%); 4.8 MiB/s, 77 objects/s recovering 2024-09-21T10:42:26.894 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:26.928 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:26.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:26.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:27.029 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:27.068 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:27.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:27.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:27.170 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:27.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:27.239 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:27.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:27.340 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:27.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:27.372 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:27.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:27.472 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:27.508 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:27.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:27.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:27.609 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:27.644 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:27.644 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:27.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:27.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:27.746 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:27.794 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:27.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:27.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:27.895 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:27.943 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:27.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:27.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:28.045 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:28.072 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:28.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:28.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:28.174 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:28.200 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:28.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:28.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:28.302 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:28.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:28.340 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:28.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:28.441 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:28.482 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:28.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:28.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:28.584 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:28.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:28 smithi160 ceph-mon[101879]: pgmap v2248: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 976 op/s; 5636/295092 objects degraded (1.910%); 2908/295092 objects misplaced (0.985%); 4.8 MiB/s, 77 objects/s recovering 2024-09-21T10:42:28.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:28 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 5636/295092 objects degraded (1.910%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:42:28.614 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:28.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:28.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:28.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:28 smithi070 ceph-mon[107385]: pgmap v2248: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 976 op/s; 5636/295092 objects degraded (1.910%); 2908/295092 objects misplaced (0.985%); 4.8 MiB/s, 77 objects/s recovering 2024-09-21T10:42:28.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:28 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 5636/295092 objects degraded (1.910%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:42:28.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:28 smithi070 ceph-mon[112139]: pgmap v2248: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 976 op/s; 5636/295092 objects degraded (1.910%); 2908/295092 objects misplaced (0.985%); 4.8 MiB/s, 77 objects/s recovering 2024-09-21T10:42:28.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:28 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 5636/295092 objects degraded (1.910%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:42:28.716 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:28.740 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:28.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:28.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:28.843 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:28.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:28.877 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:28.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:28.978 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:29.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:29.017 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:29.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:29.057 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:29.113 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:42:29.117 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:29.207 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:29.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:29.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:29.309 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:29.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:29.363 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:29.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:29.406 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T10:42:29.426 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:29 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3584565383' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:42:29.426 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:29 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3584565383' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:42:29.465 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:29.495 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:29.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:29.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:29.583 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:29 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3584565383' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:42:29.597 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:29.670 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:29.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:29.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:29.772 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:29.825 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:29.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:29.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:29.927 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:29.961 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:29.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:29.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:30.062 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:30.095 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:30.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:30.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:30.196 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:30.237 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:30.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:30.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:30.338 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:30.383 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:30.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:30.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:30.486 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:30.513 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:30 smithi070 ceph-mon[107385]: pgmap v2249: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 976 op/s; 5636/295092 objects degraded (1.910%); 2908/295092 objects misplaced (0.985%); 4.8 MiB/s, 77 objects/s recovering 2024-09-21T10:42:30.513 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:30 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:42:30.513 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:30 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3584565383' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:42:30.514 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:30 smithi070 ceph-mon[107385]: osdmap e1342: 8 total, 8 up, 6 in 2024-09-21T10:42:30.514 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:30 smithi070 ceph-mon[112139]: pgmap v2249: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 976 op/s; 5636/295092 objects degraded (1.910%); 2908/295092 objects misplaced (0.985%); 4.8 MiB/s, 77 objects/s recovering 2024-09-21T10:42:30.514 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:30 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:42:30.514 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:30 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3584565383' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:42:30.514 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:30 smithi070 ceph-mon[112139]: osdmap e1342: 8 total, 8 up, 6 in 2024-09-21T10:42:30.521 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:30.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:30.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:30 smithi160 ceph-mon[101879]: pgmap v2249: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 976 op/s; 5636/295092 objects degraded (1.910%); 2908/295092 objects misplaced (0.985%); 4.8 MiB/s, 77 objects/s recovering 2024-09-21T10:42:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:30 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:42:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:30 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3584565383' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:42:30.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:30 smithi160 ceph-mon[101879]: osdmap e1342: 8 total, 8 up, 6 in 2024-09-21T10:42:30.624 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:30.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:30.677 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:30.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:30.777 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:30.814 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:30.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:30.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:30.915 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:30.955 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:30.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:30.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:31.058 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:31.097 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:31.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:31.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:31.198 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:31.227 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:31.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:31.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:31.328 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:31.359 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:31.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:31.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:31.461 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:31.490 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:31.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:31.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:31.592 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:31.613 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:31.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:31.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:31.714 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:31.749 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:31.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:31.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:31.851 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:31.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:31.874 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:31.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:31.975 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:32.009 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:32.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:32.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:32.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:31 smithi160 ceph-mon[101879]: pgmap v2251: 99 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 957 op/s; 5321/286707 objects degraded (1.856%); 1980/286707 objects misplaced (0.691%); 5.1 MiB/s, 81 objects/s recovering 2024-09-21T10:42:32.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:42:32.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:42:32.111 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:32.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:31 smithi070 ceph-mon[107385]: pgmap v2251: 99 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 957 op/s; 5321/286707 objects degraded (1.856%); 1980/286707 objects misplaced (0.691%); 5.1 MiB/s, 81 objects/s recovering 2024-09-21T10:42:32.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:42:32.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:42:32.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:31 smithi070 ceph-mon[112139]: pgmap v2251: 99 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 957 op/s; 5321/286707 objects degraded (1.856%); 1980/286707 objects misplaced (0.691%); 5.1 MiB/s, 81 objects/s recovering 2024-09-21T10:42:32.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:42:32.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:42:32.141 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:32.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:32.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:32.242 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:32.251 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:42:32.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:32.278 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:32.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:32.380 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:32.408 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:32.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:32.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:32.510 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:32.551 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:32.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:32.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:32.563 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:42:32.563 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:42:32.653 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:32.684 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:32.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:32.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:32.785 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:32.827 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:32.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:32.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:32.928 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:32.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:32.964 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:32.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:33.064 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:33.090 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:33.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:33.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:33.192 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:33.222 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:33.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:33.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:33.324 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:33.356 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:33.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:33.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:33.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:42:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:42:33] "GET /metrics HTTP/1.1" 200 36628 "" "Prometheus/2.51.0" 2024-09-21T10:42:33.458 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:33.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:33.480 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:33.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:33.581 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:33.603 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:33.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:33.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:33.704 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:33.725 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:33.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:33.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:33.827 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:33.853 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:33.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:33.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:33.955 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:33.976 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:33.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:33.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:34.078 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:34.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:33 smithi160 ceph-mon[101879]: pgmap v2252: 99 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 958 op/s; 5321/280347 objects degraded (1.898%); 1980/280347 objects misplaced (0.706%); 5.1 MiB/s, 81 objects/s recovering 2024-09-21T10:42:34.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:33 smithi160 ceph-mon[101879]: osdmap e1343: 8 total, 8 up, 6 in 2024-09-21T10:42:34.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:33 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 5321/280347 objects degraded (1.898%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:42:34.100 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:34.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:34.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:34.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:33 smithi070 ceph-mon[107385]: pgmap v2252: 99 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 958 op/s; 5321/280347 objects degraded (1.898%); 1980/280347 objects misplaced (0.706%); 5.1 MiB/s, 81 objects/s recovering 2024-09-21T10:42:34.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:33 smithi070 ceph-mon[107385]: osdmap e1343: 8 total, 8 up, 6 in 2024-09-21T10:42:34.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:33 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 5321/280347 objects degraded (1.898%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:42:34.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:33 smithi070 ceph-mon[112139]: pgmap v2252: 99 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 958 op/s; 5321/280347 objects degraded (1.898%); 1980/280347 objects misplaced (0.706%); 5.1 MiB/s, 81 objects/s recovering 2024-09-21T10:42:34.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:33 smithi070 ceph-mon[112139]: osdmap e1343: 8 total, 8 up, 6 in 2024-09-21T10:42:34.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:33 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 5321/280347 objects degraded (1.898%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:42:34.201 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:34.224 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:34.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:34.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:34.326 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:34.350 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:34.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:34.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:34.353 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:42:34.407 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:34.415 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:34.452 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:34.514 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:34.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:34.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:34.615 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:34.643 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:42:34.658 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:34.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:34.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:34.760 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:34.796 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:34.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:34.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:34.898 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:34.957 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:34.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:34.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:34.986 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:34 smithi070 ceph-mon[107385]: osdmap e1344: 8 total, 8 up, 6 in 2024-09-21T10:42:34.986 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:34 smithi070 ceph-mon[112139]: osdmap e1344: 8 total, 8 up, 6 in 2024-09-21T10:42:35.059 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:35.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:34 smithi160 ceph-mon[101879]: osdmap e1344: 8 total, 8 up, 6 in 2024-09-21T10:42:35.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:35.098 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:35.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:35.199 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:35.234 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:35.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:35.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:35.336 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:35.385 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:35.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:35.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:35.487 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:35.515 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:35.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:35.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:35.617 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:35.644 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:35.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:35.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:35.746 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:35.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:35.790 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:35.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:35.891 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:35.923 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:35.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:35.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:36.025 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:36.067 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:36.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:36.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:36.168 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:36.199 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:36.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:36.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:36.301 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:36.335 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:36.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:36.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:36.437 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:36.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:36.482 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:36.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:36.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:36 smithi070 ceph-mon[107385]: pgmap v2255: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+remapped, 95 active+clean; 5.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 1.22k op/s; 2520/273096 objects degraded (0.923%); 3242/273096 objects misplaced (1.187%); 9.7 MiB/s, 154 objects/s recovering 2024-09-21T10:42:36.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:42:36.499 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:42:36.499 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:36 smithi070 ceph-mon[112139]: pgmap v2255: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+remapped, 95 active+clean; 5.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 1.22k op/s; 2520/273096 objects degraded (0.923%); 3242/273096 objects misplaced (1.187%); 9.7 MiB/s, 154 objects/s recovering 2024-09-21T10:42:36.499 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:42:36.499 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:42:36.584 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:36.613 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:36.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:36.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:36.684 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:36 smithi160 ceph-mon[101879]: pgmap v2255: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+remapped, 95 active+clean; 5.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 1.22k op/s; 2520/273096 objects degraded (0.923%); 3242/273096 objects misplaced (1.187%); 9.7 MiB/s, 154 objects/s recovering 2024-09-21T10:42:36.684 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:42:36.684 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:42:36.715 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:36.748 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:36.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:36.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:36.850 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:36.882 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:36.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:36.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:36.984 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:36.990 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:42:36.990 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:42:37.022 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:37.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:37.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:37.125 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:37.158 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:37.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:37.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:37.259 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:37.285 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:37.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:37.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:37.387 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:37.425 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:37.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:37.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:37.571 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:37.610 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:37.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:37.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:37.654 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:42:37.699 INFO:teuthology.orchestra.run.smithi070.stderr:no valid command found; 10 closest matches: 2024-09-21T10:42:37.699 INFO:teuthology.orchestra.run.smithi070.stderr:0 2024-09-21T10:42:37.699 INFO:teuthology.orchestra.run.smithi070.stderr:1 2024-09-21T10:42:37.699 INFO:teuthology.orchestra.run.smithi070.stderr:2 2024-09-21T10:42:37.699 INFO:teuthology.orchestra.run.smithi070.stderr:abort 2024-09-21T10:42:37.700 INFO:teuthology.orchestra.run.smithi070.stderr:assert 2024-09-21T10:42:37.700 INFO:teuthology.orchestra.run.smithi070.stderr:bench [] [] [] [] 2024-09-21T10:42:37.700 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:42:37.700 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs files list 2024-09-21T10:42:37.700 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs stats 2024-09-21T10:42:37.700 INFO:teuthology.orchestra.run.smithi070.stderr:bluestore allocator dump block 2024-09-21T10:42:37.700 INFO:teuthology.orchestra.run.smithi070.stderr:Error EINVAL: invalid command 2024-09-21T10:42:37.712 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:37.749 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:37.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:37.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:37.851 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:37.887 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:37.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:37.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:37.964 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:42:37.989 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:38.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:38.029 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:38.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:38.130 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:38.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:38.160 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:38.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:38.261 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:38.297 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:38.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:38.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:38.357 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:42:38.357 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:42:38.398 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:42:38.398 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:42:38.399 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:38.434 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:38.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:38.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:38.435 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:42:38.436 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:42:38.436 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:38 smithi070 ceph-mon[107385]: pgmap v2256: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+remapped, 95 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 1.06k op/s; 2520/268227 objects degraded (0.940%); 3242/268227 objects misplaced (1.209%); 5.7 MiB/s, 91 objects/s recovering 2024-09-21T10:42:38.436 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:38 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 2520/268227 objects degraded (0.940%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:42:38.436 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:38 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3481678593' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:42:38.436 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:38 smithi070 ceph-mon[112139]: pgmap v2256: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+remapped, 95 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 1.06k op/s; 2520/268227 objects degraded (0.940%); 3242/268227 objects misplaced (1.209%); 5.7 MiB/s, 91 objects/s recovering 2024-09-21T10:42:38.436 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:38 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 2520/268227 objects degraded (0.940%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:42:38.436 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:38 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3481678593' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:42:38.473 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:42:38.473 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:42:38.506 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:42:38.506 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:42:38.536 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:38.539 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:42:38.539 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:42:38.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:38.573 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:38.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:38.576 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:42:38.576 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:42:38.608 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:42:38.608 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:42:38.674 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:38.715 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:38.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:38.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:38.737 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:42:38.737 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:42:38.737 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-21T10:42:38.816 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:38 smithi160 ceph-mon[101879]: pgmap v2256: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+remapped, 95 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 1.06k op/s; 2520/268227 objects degraded (0.940%); 3242/268227 objects misplaced (1.209%); 5.7 MiB/s, 91 objects/s recovering 2024-09-21T10:42:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:38 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 2520/268227 objects degraded (0.940%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:42:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:38 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3481678593' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:42:38.870 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:38.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:38.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:38.972 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:39.001 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:39.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:39.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:39.103 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:39.144 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:39.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:39.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:39.245 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:39.281 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:39.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:39.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:39.383 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:39.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:39.408 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:39.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:39.509 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:39.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:39.559 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:39.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:39.660 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:39.697 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:39.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:39.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:39.799 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:39.830 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:39.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:39.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:39.932 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:39.968 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:39.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:39.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:39.977 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:42:39.977 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T10:42:39.977 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T10:42:39.977 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T10:42:39.977 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:42:40.069 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:40.102 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:40.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:40.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:40.204 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:40.246 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:40.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:40.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:40.348 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:40.385 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:40.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:40.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:40.487 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:40.513 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:40.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:40.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:40.536 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:40 smithi070 ceph-mon[107385]: pgmap v2257: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+remapped, 95 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 769 op/s; 2520/268227 objects degraded (0.940%); 3242/268227 objects misplaced (1.209%); 4.0 MiB/s, 64 objects/s recovering 2024-09-21T10:42:40.536 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:40 smithi070 ceph-mon[112139]: pgmap v2257: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+remapped, 95 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 769 op/s; 2520/268227 objects degraded (0.940%); 3242/268227 objects misplaced (1.209%); 4.0 MiB/s, 64 objects/s recovering 2024-09-21T10:42:40.614 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:40.631 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:40.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:40.683 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:40.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:40.784 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:40.811 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:40.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:40.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:40 smithi160 ceph-mon[101879]: pgmap v2257: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait, 1 active+recovering+undersized+remapped, 95 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 769 op/s; 2520/268227 objects degraded (0.940%); 3242/268227 objects misplaced (1.209%); 4.0 MiB/s, 64 objects/s recovering 2024-09-21T10:42:40.876 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:42:40.913 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:40.943 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:40.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:40.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:41.045 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:41.085 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:41.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:41.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:41.187 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:41.211 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:42:41.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:41.230 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:41.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:41.331 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:41.364 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:41.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:41.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:41.392 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T10:42:41.465 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:41.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:41.518 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:41.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:41.619 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:41.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:41 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2476573186' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:42:41.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:41 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:42:41.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:41 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2476573186' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:42:41.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:41 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:42:41.652 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:41.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:41.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:41.654 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:41 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2476573186' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:42:41.654 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:41 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:42:41.754 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:41.786 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:41.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:41.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:41.885 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:42:41.887 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:41.973 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:41.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:41.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:42.075 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:42.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:42.131 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:42.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:42.231 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:42.262 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:42.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:42.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:42.364 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:42.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:42.403 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:42.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:42.475 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:42 smithi070 ceph-mon[107385]: pgmap v2258: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 5.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 916 op/s; 2520/258345 objects degraded (0.975%); 2634/258345 objects misplaced (1.020%); 5.6 MiB/s, 89 objects/s recovering 2024-09-21T10:42:42.475 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:42 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:42:42.476 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:42 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:42:42.476 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:42 smithi070 ceph-mon[107385]: osdmap e1345: 8 total, 8 up, 6 in 2024-09-21T10:42:42.476 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:42 smithi070 ceph-mon[112139]: pgmap v2258: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 5.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 916 op/s; 2520/258345 objects degraded (0.975%); 2634/258345 objects misplaced (1.020%); 5.6 MiB/s, 89 objects/s recovering 2024-09-21T10:42:42.476 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:42 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:42:42.476 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:42 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:42:42.476 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:42 smithi070 ceph-mon[112139]: osdmap e1345: 8 total, 8 up, 6 in 2024-09-21T10:42:42.505 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:42.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:42.549 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:42.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:42.650 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:42.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:42 smithi160 ceph-mon[101879]: pgmap v2258: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 5.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 916 op/s; 2520/258345 objects degraded (0.975%); 2634/258345 objects misplaced (1.020%); 5.6 MiB/s, 89 objects/s recovering 2024-09-21T10:42:42.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:42 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:42:42.662 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:42 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:42:42.662 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:42 smithi160 ceph-mon[101879]: osdmap e1345: 8 total, 8 up, 6 in 2024-09-21T10:42:42.686 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:42.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:42.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:42.787 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:42.809 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:42.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:42.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:42.911 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:42.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:42.944 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:42.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:43.045 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:43.061 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:42:43.062 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T10:42:43.062 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T10:42:43.062 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T10:42:43.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.062 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:27980 53.0 53:0c162dc8:::benchmark_data_smithi070_228967_object27979:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1316)", 2024-09-21T10:42:43.062 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:40:35.297206+0000", 2024-09-21T10:42:43.062 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 127.759466027, 2024-09-21T10:42:43.062 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.087637404000000002, 2024-09-21T10:42:43.062 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:42:43.062 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:42:43.062 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:42:43.062 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:42:43.063 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:42:43.063 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 27980 2024-09-21T10:42:43.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.063 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:42:43.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:42:43.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.297206+0000", 2024-09-21T10:42:43.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:42:43.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.297206+0000", 2024-09-21T10:42:43.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.064 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:42:43.064 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.297206+0000", 2024-09-21T10:42:43.064 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T10:42:43.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.064 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.064 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:42:43.064 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.297231+0000", 2024-09-21T10:42:43.064 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4854999999999998e-05 2024-09-21T10:42:43.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.064 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.064 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:42:43.064 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.297238+0000", 2024-09-21T10:42:43.064 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0870000000000001e-06 2024-09-21T10:42:43.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.064 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.065 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:42:43.065 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.297243+0000", 2024-09-21T10:42:43.065 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0370000000000002e-06 2024-09-21T10:42:43.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.065 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:42:43.065 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.298589+0000", 2024-09-21T10:42:43.065 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001345958 2024-09-21T10:42:43.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.065 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:42:43.065 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.298740+0000", 2024-09-21T10:42:43.065 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015125800000000001 2024-09-21T10:42:43.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:42:43.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.298899+0000", 2024-09-21T10:42:43.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015911000000000001 2024-09-21T10:42:43.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:42:43.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.302799+0000", 2024-09-21T10:42:43.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0038999170000000001 2024-09-21T10:42:43.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.302891+0000", 2024-09-21T10:42:43.067 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2040999999999995e-05 2024-09-21T10:42:43.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.384723+0000", 2024-09-21T10:42:43.067 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.081832026000000002 2024-09-21T10:42:43.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:42:43.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.384777+0000", 2024-09-21T10:42:43.067 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4403000000000001e-05 2024-09-21T10:42:43.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.068 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:42:43.068 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.384843+0000", 2024-09-21T10:42:43.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6154000000000004e-05 2024-09-21T10:42:43.068 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.068 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:42:43.068 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.068 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:27985 53.0 53:06b44fde:::benchmark_data_smithi070_228967_object27984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1316)", 2024-09-21T10:42:43.068 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:40:35.298393+0000", 2024-09-21T10:42:43.068 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 127.75827949400001, 2024-09-21T10:42:43.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.086603391000000002, 2024-09-21T10:42:43.068 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:42:43.069 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:42:43.069 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:42:43.069 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:42:43.069 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:42:43.069 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 27985 2024-09-21T10:42:43.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.069 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:42:43.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.069 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:42:43.069 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.298393+0000", 2024-09-21T10:42:43.069 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.069 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:42:43.069 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.298393+0000", 2024-09-21T10:42:43.070 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.070 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.070 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:42:43.070 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.298391+0000", 2024-09-21T10:42:43.070 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:42:43.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.070 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.070 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:42:43.070 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.298454+0000", 2024-09-21T10:42:43.070 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3310000000000002e-05 2024-09-21T10:42:43.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.070 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:42:43.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.298476+0000", 2024-09-21T10:42:43.071 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1803000000000001e-05 2024-09-21T10:42:43.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:42:43.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.298483+0000", 2024-09-21T10:42:43.071 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7510000000000002e-06 2024-09-21T10:42:43.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:42:43.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.300003+0000", 2024-09-21T10:42:43.071 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0015197139999999999 2024-09-21T10:42:43.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.072 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:42:43.072 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.300112+0000", 2024-09-21T10:42:43.072 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000109687 2024-09-21T10:42:43.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.072 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:42:43.072 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.300237+0000", 2024-09-21T10:42:43.072 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000124463 2024-09-21T10:42:43.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.072 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:42:43.073 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.302907+0000", 2024-09-21T10:42:43.073 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0026698080000000001 2024-09-21T10:42:43.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.073 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.073 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.303044+0000", 2024-09-21T10:42:43.073 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000137348 2024-09-21T10:42:43.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.073 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.073 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.384889+0000", 2024-09-21T10:42:43.073 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.081844633 2024-09-21T10:42:43.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.074 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:42:43.074 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.384929+0000", 2024-09-21T10:42:43.074 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0111999999999998e-05 2024-09-21T10:42:43.074 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.074 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.074 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:42:43.074 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.384996+0000", 2024-09-21T10:42:43.074 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7236000000000005e-05 2024-09-21T10:42:43.074 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.074 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:42:43.074 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.074 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.074 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.075 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:27988 53.0 53:0a943efb:::benchmark_data_smithi070_228967_object27987:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1316)", 2024-09-21T10:42:43.075 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:40:35.299984+0000", 2024-09-21T10:42:43.075 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 127.75668856599999, 2024-09-21T10:42:43.075 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.085195193000000002, 2024-09-21T10:42:43.075 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:42:43.075 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:42:43.075 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:42:43.075 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:42:43.076 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:42:43.076 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 27988 2024-09-21T10:42:43.076 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.076 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:42:43.077 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.077 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:42:43.077 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.299984+0000", 2024-09-21T10:42:43.077 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.077 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.077 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:42:43.077 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.299984+0000", 2024-09-21T10:42:43.077 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.077 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.077 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:42:43.077 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.299982+0000", 2024-09-21T10:42:43.077 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:42:43.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.078 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.078 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:42:43.078 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.300010+0000", 2024-09-21T10:42:43.078 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8079e-05 2024-09-21T10:42:43.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.078 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.078 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:42:43.078 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.300020+0000", 2024-09-21T10:42:43.078 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6090000000000007e-06 2024-09-21T10:42:43.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.078 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.078 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:42:43.078 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.300026+0000", 2024-09-21T10:42:43.079 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3609999999999997e-06 2024-09-21T10:42:43.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.079 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.079 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:42:43.079 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.301137+0000", 2024-09-21T10:42:43.079 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0011109189999999999 2024-09-21T10:42:43.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.079 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.079 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:42:43.079 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.301820+0000", 2024-09-21T10:42:43.079 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00068358100000000005 2024-09-21T10:42:43.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.079 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.079 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:42:43.079 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.301952+0000", 2024-09-21T10:42:43.080 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013160400000000001 2024-09-21T10:42:43.080 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.080 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.080 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.080 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.303292+0000", 2024-09-21T10:42:43.080 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00134015 2024-09-21T10:42:43.080 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.080 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.080 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:42:43.080 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.304462+0000", 2024-09-21T10:42:43.080 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0011701789999999999 2024-09-21T10:42:43.080 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.080 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.080 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.081 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.385081+0000", 2024-09-21T10:42:43.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.080618598 2024-09-21T10:42:43.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.081 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:42:43.081 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.385128+0000", 2024-09-21T10:42:43.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6573000000000003e-05 2024-09-21T10:42:43.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.081 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:42:43.081 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:35.385179+0000", 2024-09-21T10:42:43.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1199999999999998e-05 2024-09-21T10:42:43.081 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.081 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:42:43.082 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.082 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.082 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.082 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:69116 53.0 53:0424d225:::benchmark_data_smithi070_228967_object69115:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1322)", 2024-09-21T10:42:43.082 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:40:59.539315+0000", 2024-09-21T10:42:43.082 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 103.517357137, 2024-09-21T10:42:43.082 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.085500283999999996, 2024-09-21T10:42:43.082 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:42:43.082 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:42:43.082 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:42:43.082 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:42:43.082 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:42:43.082 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 69116 2024-09-21T10:42:43.082 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.082 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:42:43.083 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.083 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:42:43.083 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.539315+0000", 2024-09-21T10:42:43.083 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.083 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.083 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:42:43.083 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.539315+0000", 2024-09-21T10:42:43.083 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.083 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.083 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:42:43.083 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.539314+0000", 2024-09-21T10:42:43.084 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:42:43.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.084 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:42:43.084 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.539377+0000", 2024-09-21T10:42:43.084 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3759999999999999e-05 2024-09-21T10:42:43.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.084 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:42:43.084 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.539391+0000", 2024-09-21T10:42:43.084 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3315e-05 2024-09-21T10:42:43.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.084 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:42:43.084 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.539398+0000", 2024-09-21T10:42:43.085 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1949999999999997e-06 2024-09-21T10:42:43.085 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.085 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.085 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:42:43.085 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.545541+0000", 2024-09-21T10:42:43.085 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0061431400000000001 2024-09-21T10:42:43.085 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.085 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.085 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:42:43.085 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.545660+0000", 2024-09-21T10:42:43.085 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119392 2024-09-21T10:42:43.085 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.085 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.085 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:42:43.085 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.545800+0000", 2024-09-21T10:42:43.086 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013975500000000001 2024-09-21T10:42:43.086 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.086 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.086 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.086 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.547670+0000", 2024-09-21T10:42:43.086 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001870053 2024-09-21T10:42:43.086 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.086 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.086 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:42:43.086 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.547676+0000", 2024-09-21T10:42:43.086 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1140000000000002e-06 2024-09-21T10:42:43.086 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.086 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.086 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.086 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.624725+0000", 2024-09-21T10:42:43.086 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.077048338999999993 2024-09-21T10:42:43.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.087 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.087 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:42:43.087 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.624774+0000", 2024-09-21T10:42:43.087 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9670999999999999e-05 2024-09-21T10:42:43.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.087 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.087 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:42:43.087 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.624815+0000", 2024-09-21T10:42:43.087 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0975e-05 2024-09-21T10:42:43.087 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.087 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:42:43.087 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.087 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.087 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.45020.0:69150 53.a e1323/1307 53:5469a6c9:::benchmark_data_smithi070_228967_object69149:head v 1323'4304, mlcod=1322'4302)", 2024-09-21T10:42:43.088 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:40:59.633579+0000", 2024-09-21T10:42:43.088 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 103.423093309, 2024-09-21T10:42:43.088 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098647451999999997, 2024-09-21T10:42:43.088 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:42:43.088 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:42:43.088 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:42:43.088 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.088 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:42:43.088 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.633579+0000", 2024-09-21T10:42:43.088 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.088 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.088 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:42:43.088 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.633579+0000", 2024-09-21T10:42:43.088 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.089 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.089 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:42:43.089 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.633575+0000", 2024-09-21T10:42:43.089 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999967 2024-09-21T10:42:43.089 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.089 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.089 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:42:43.089 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.633640+0000", 2024-09-21T10:42:43.089 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5078000000000003e-05 2024-09-21T10:42:43.089 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.089 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.089 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:42:43.089 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.633650+0000", 2024-09-21T10:42:43.089 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2310000000000002e-06 2024-09-21T10:42:43.089 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.090 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.090 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:42:43.090 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.633661+0000", 2024-09-21T10:42:43.090 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1141000000000001e-05 2024-09-21T10:42:43.090 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.090 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.090 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:42:43.090 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.633725+0000", 2024-09-21T10:42:43.090 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4228999999999998e-05 2024-09-21T10:42:43.090 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.090 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.090 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:42:43.091 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.633750+0000", 2024-09-21T10:42:43.091 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5282e-05 2024-09-21T10:42:43.091 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.091 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.091 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:42:43.091 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.729657+0000", 2024-09-21T10:42:43.091 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.095906405 2024-09-21T10:42:43.091 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.091 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.091 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:42:43.091 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.732226+0000", 2024-09-21T10:42:43.091 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0025695290000000001 2024-09-21T10:42:43.091 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.091 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:42:43.091 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.092 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.092 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.092 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:72211 53.8 53:10f6294e:::benchmark_data_smithi070_228967_object72210:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1324)", 2024-09-21T10:42:43.092 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:02.573709+0000", 2024-09-21T10:42:43.092 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 100.482963599, 2024-09-21T10:42:43.092 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.125090165, 2024-09-21T10:42:43.092 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:42:43.092 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:42:43.092 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:42:43.092 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:42:43.092 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:42:43.092 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 72211 2024-09-21T10:42:43.092 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.092 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:42:43.092 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.093 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:42:43.093 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.573709+0000", 2024-09-21T10:42:43.093 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.093 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.093 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.093 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:42:43.093 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.573709+0000", 2024-09-21T10:42:43.093 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.093 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.093 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.093 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:42:43.093 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.573708+0000", 2024-09-21T10:42:43.093 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:42:43.093 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.093 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.094 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:42:43.094 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.573736+0000", 2024-09-21T10:42:43.094 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8065e-05 2024-09-21T10:42:43.094 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.094 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.094 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:42:43.094 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.573754+0000", 2024-09-21T10:42:43.094 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8134999999999998e-05 2024-09-21T10:42:43.094 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.094 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.094 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:42:43.094 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.573761+0000", 2024-09-21T10:42:43.094 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5000000000000002e-06 2024-09-21T10:42:43.094 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.094 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.095 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:42:43.095 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.573956+0000", 2024-09-21T10:42:43.095 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000194814 2024-09-21T10:42:43.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.095 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.095 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:42:43.095 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.574103+0000", 2024-09-21T10:42:43.095 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000146408 2024-09-21T10:42:43.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.095 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.095 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:42:43.095 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.574221+0000", 2024-09-21T10:42:43.095 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000118412 2024-09-21T10:42:43.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.096 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.096 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.096 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.583179+0000", 2024-09-21T10:42:43.096 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0089579670000000007 2024-09-21T10:42:43.096 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.096 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.096 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:42:43.096 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.583183+0000", 2024-09-21T10:42:43.096 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3800000000000004e-06 2024-09-21T10:42:43.096 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.096 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.096 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.096 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.698714+0000", 2024-09-21T10:42:43.096 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.115530562 2024-09-21T10:42:43.097 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.097 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.097 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:42:43.097 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.698764+0000", 2024-09-21T10:42:43.097 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9527999999999997e-05 2024-09-21T10:42:43.097 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.097 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.097 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:42:43.097 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.698799+0000", 2024-09-21T10:42:43.097 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5172000000000001e-05 2024-09-21T10:42:43.097 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.097 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:42:43.097 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.097 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.097 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.097 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:82094 53.8 53:172d25c6:::benchmark_data_smithi070_228967_object82093:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1334)", 2024-09-21T10:42:43.098 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:14.628422+0000", 2024-09-21T10:42:43.098 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 88.428250418000005, 2024-09-21T10:42:43.098 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10206596699999999, 2024-09-21T10:42:43.098 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:42:43.098 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:42:43.098 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:42:43.098 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:42:43.098 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:42:43.098 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 82094 2024-09-21T10:42:43.098 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.098 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:42:43.098 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.098 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:42:43.098 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.628422+0000", 2024-09-21T10:42:43.099 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.099 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.099 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.099 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:42:43.099 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.628422+0000", 2024-09-21T10:42:43.099 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.099 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.099 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.099 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:42:43.099 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.628419+0000", 2024-09-21T10:42:43.099 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 2024-09-21T10:42:43.099 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.099 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.099 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:42:43.099 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.628455+0000", 2024-09-21T10:42:43.100 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6025000000000002e-05 2024-09-21T10:42:43.100 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.100 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.100 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:42:43.100 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.628469+0000", 2024-09-21T10:42:43.100 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3253e-05 2024-09-21T10:42:43.100 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.100 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.100 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:42:43.100 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.628476+0000", 2024-09-21T10:42:43.100 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4889999999999998e-06 2024-09-21T10:42:43.100 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.100 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.101 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:42:43.101 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.628513+0000", 2024-09-21T10:42:43.101 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6776999999999997e-05 2024-09-21T10:42:43.101 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.101 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.101 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:42:43.101 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.628617+0000", 2024-09-21T10:42:43.101 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000104369 2024-09-21T10:42:43.101 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.101 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.101 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:42:43.101 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.628710+0000", 2024-09-21T10:42:43.101 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2288999999999997e-05 2024-09-21T10:42:43.101 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.101 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.102 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:42:43.102 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.630530+0000", 2024-09-21T10:42:43.102 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001819942 2024-09-21T10:42:43.102 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.102 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.102 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.102 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.631864+0000", 2024-09-21T10:42:43.102 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0013344769999999999 2024-09-21T10:42:43.102 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.102 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.102 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.102 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.730412+0000", 2024-09-21T10:42:43.102 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098548018000000001 2024-09-21T10:42:43.103 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.103 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.103 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:42:43.103 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.730452+0000", 2024-09-21T10:42:43.103 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9804e-05 2024-09-21T10:42:43.103 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.103 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.103 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:42:43.103 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.730488+0000", 2024-09-21T10:42:43.103 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5768e-05 2024-09-21T10:42:43.103 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.103 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:42:43.103 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.103 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.104 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.104 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:82102 53.8 53:16f516de:::benchmark_data_smithi070_228967_object82101:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1334)", 2024-09-21T10:42:43.104 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:14.632737+0000", 2024-09-21T10:42:43.104 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 88.423935342999997, 2024-09-21T10:42:43.104 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.103853668, 2024-09-21T10:42:43.104 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:42:43.104 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:42:43.104 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:42:43.104 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:42:43.104 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:42:43.104 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 82102 2024-09-21T10:42:43.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.104 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:42:43.104 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.104 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:42:43.104 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.632737+0000", 2024-09-21T10:42:43.105 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.105 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.105 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.105 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:42:43.105 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.632737+0000", 2024-09-21T10:42:43.105 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.105 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.105 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.105 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:42:43.105 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.632735+0000", 2024-09-21T10:42:43.105 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:42:43.105 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.105 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.105 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:42:43.105 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.632789+0000", 2024-09-21T10:42:43.105 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3702000000000002e-05 2024-09-21T10:42:43.105 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.106 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.106 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:42:43.106 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.632804+0000", 2024-09-21T10:42:43.106 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5557e-05 2024-09-21T10:42:43.106 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.106 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.106 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:42:43.106 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.632812+0000", 2024-09-21T10:42:43.106 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2490000000000004e-06 2024-09-21T10:42:43.106 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.106 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.106 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:42:43.106 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.632855+0000", 2024-09-21T10:42:43.106 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3569000000000002e-05 2024-09-21T10:42:43.107 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.107 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.107 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:42:43.107 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.633008+0000", 2024-09-21T10:42:43.107 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000152845 2024-09-21T10:42:43.107 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.107 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.107 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:42:43.107 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.633191+0000", 2024-09-21T10:42:43.107 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000183427 2024-09-21T10:42:43.107 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.107 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.107 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:42:43.107 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.634595+0000", 2024-09-21T10:42:43.108 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0014039650000000001 2024-09-21T10:42:43.108 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.108 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.108 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.108 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.635290+0000", 2024-09-21T10:42:43.108 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000695063 2024-09-21T10:42:43.108 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.108 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.108 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.108 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.736510+0000", 2024-09-21T10:42:43.108 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.101219369 2024-09-21T10:42:43.108 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.108 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.108 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:42:43.108 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.736556+0000", 2024-09-21T10:42:43.109 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6711000000000001e-05 2024-09-21T10:42:43.109 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.109 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.109 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:42:43.109 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.736590+0000", 2024-09-21T10:42:43.109 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3955000000000003e-05 2024-09-21T10:42:43.109 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.109 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:42:43.109 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.109 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.109 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.109 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:82118 53.0 53:000ac806:::benchmark_data_smithi070_228967_object82117:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1334)", 2024-09-21T10:42:43.109 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:14.650340+0000", 2024-09-21T10:42:43.109 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 88.406332031000005, 2024-09-21T10:42:43.109 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.086881378999999995, 2024-09-21T10:42:43.110 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:42:43.110 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:42:43.110 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:42:43.110 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:42:43.110 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:42:43.110 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 82118 2024-09-21T10:42:43.110 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.110 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:42:43.110 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.110 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:42:43.110 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.650340+0000", 2024-09-21T10:42:43.110 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.110 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.111 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.111 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:42:43.111 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.650340+0000", 2024-09-21T10:42:43.111 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.111 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.111 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.111 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:42:43.111 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.650339+0000", 2024-09-21T10:42:43.111 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:42:43.111 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.111 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.111 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:42:43.111 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.650393+0000", 2024-09-21T10:42:43.111 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3718999999999998e-05 2024-09-21T10:42:43.111 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.112 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.112 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:42:43.112 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.650402+0000", 2024-09-21T10:42:43.112 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8659999999999995e-06 2024-09-21T10:42:43.112 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.112 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.112 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:42:43.112 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.650406+0000", 2024-09-21T10:42:43.112 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8899999999999998e-06 2024-09-21T10:42:43.112 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.112 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.112 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:42:43.112 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.650442+0000", 2024-09-21T10:42:43.112 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5309999999999999e-05 2024-09-21T10:42:43.112 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.112 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.113 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:42:43.113 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.650528+0000", 2024-09-21T10:42:43.113 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6017e-05 2024-09-21T10:42:43.113 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.113 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.113 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:42:43.113 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.650613+0000", 2024-09-21T10:42:43.113 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5051999999999998e-05 2024-09-21T10:42:43.113 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.113 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.113 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:42:43.113 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.651778+0000", 2024-09-21T10:42:43.113 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0011655840000000001 2024-09-21T10:42:43.113 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.113 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.113 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.114 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.652097+0000", 2024-09-21T10:42:43.114 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00031853799999999999 2024-09-21T10:42:43.114 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.114 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.114 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.114 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.737152+0000", 2024-09-21T10:42:43.114 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.085054578000000006 2024-09-21T10:42:43.114 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.114 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.114 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:42:43.114 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.737189+0000", 2024-09-21T10:42:43.114 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7902999999999999e-05 2024-09-21T10:42:43.114 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.114 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.115 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:42:43.115 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.737221+0000", 2024-09-21T10:42:43.115 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2048000000000002e-05 2024-09-21T10:42:43.115 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.115 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:42:43.115 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.115 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.115 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:43.116 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:43.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:43.116 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:107445 53.18 53:1cc3ffc3:::benchmark_data_smithi070_228967_object107444:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1338)", 2024-09-21T10:42:43.116 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:34.749308+0000", 2024-09-21T10:42:43.117 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 68.30736392, 2024-09-21T10:42:43.117 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.105378707, 2024-09-21T10:42:43.117 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:42:43.117 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:42:43.117 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:42:43.117 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:42:43.117 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:42:43.117 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 107445 2024-09-21T10:42:43.117 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.117 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:42:43.117 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.117 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:42:43.117 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.749308+0000", 2024-09-21T10:42:43.117 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.118 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.118 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.118 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:42:43.118 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.749308+0000", 2024-09-21T10:42:43.118 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.118 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.118 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.118 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:42:43.118 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.749306+0000", 2024-09-21T10:42:43.118 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:42:43.118 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.118 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.118 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:42:43.119 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.749363+0000", 2024-09-21T10:42:43.119 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.677e-05 2024-09-21T10:42:43.119 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.119 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.119 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:42:43.119 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.749375+0000", 2024-09-21T10:42:43.119 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2138e-05 2024-09-21T10:42:43.119 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.119 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.119 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:42:43.119 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.749382+0000", 2024-09-21T10:42:43.119 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0119999999999999e-06 2024-09-21T10:42:43.119 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.119 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.119 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:42:43.120 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.749427+0000", 2024-09-21T10:42:43.120 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4496999999999998e-05 2024-09-21T10:42:43.120 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.120 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.120 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:42:43.120 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.749569+0000", 2024-09-21T10:42:43.120 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000141915 2024-09-21T10:42:43.120 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.120 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.120 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:42:43.120 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.749917+0000", 2024-09-21T10:42:43.120 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034883599999999999 2024-09-21T10:42:43.120 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.120 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.121 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.121 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.753701+0000", 2024-09-21T10:42:43.121 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0037838910000000002 2024-09-21T10:42:43.121 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.121 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.121 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:42:43.121 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.753828+0000", 2024-09-21T10:42:43.121 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012615 2024-09-21T10:42:43.121 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.121 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.121 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.121 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.854616+0000", 2024-09-21T10:42:43.121 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100788135 2024-09-21T10:42:43.121 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.122 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.122 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:42:43.122 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.854650+0000", 2024-09-21T10:42:43.122 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3943999999999999e-05 2024-09-21T10:42:43.122 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.122 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.122 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:42:43.122 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.854687+0000", 2024-09-21T10:42:43.122 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7320000000000002e-05 2024-09-21T10:42:43.122 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.122 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:42:43.122 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.122 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.122 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.122 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:120321 53.18 53:1969d74f:::benchmark_data_smithi070_228967_object120320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1338)", 2024-09-21T10:42:43.122 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:46.031613+0000", 2024-09-21T10:42:43.123 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 57.025059059, 2024-09-21T10:42:43.123 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.088143306000000005, 2024-09-21T10:42:43.123 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:42:43.123 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:42:43.123 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:42:43.123 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:42:43.123 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:42:43.123 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 120321 2024-09-21T10:42:43.123 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.123 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:42:43.123 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.123 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:42:43.123 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:46.031613+0000", 2024-09-21T10:42:43.123 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.123 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.124 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.124 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:42:43.124 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:46.031613+0000", 2024-09-21T10:42:43.124 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.124 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.124 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.124 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:42:43.124 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:46.031612+0000", 2024-09-21T10:42:43.124 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:42:43.124 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.124 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.124 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:42:43.124 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:46.031656+0000", 2024-09-21T10:42:43.124 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4391999999999998e-05 2024-09-21T10:42:43.124 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.124 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.125 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:42:43.125 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:46.031667+0000", 2024-09-21T10:42:43.125 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0992000000000001e-05 2024-09-21T10:42:43.125 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.125 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.125 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:42:43.125 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:46.031672+0000", 2024-09-21T10:42:43.125 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2360000000000004e-06 2024-09-21T10:42:43.125 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.125 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.125 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:42:43.125 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:46.031720+0000", 2024-09-21T10:42:43.125 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7246000000000001e-05 2024-09-21T10:42:43.125 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.126 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.126 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:42:43.126 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:46.031811+0000", 2024-09-21T10:42:43.126 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0971999999999996e-05 2024-09-21T10:42:43.126 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.126 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.126 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:42:43.126 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:46.032163+0000", 2024-09-21T10:42:43.126 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000352097 2024-09-21T10:42:43.126 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.126 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.126 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:42:43.126 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:46.033434+0000", 2024-09-21T10:42:43.127 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0012709679999999999 2024-09-21T10:42:43.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.127 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.127 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.127 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:46.035028+0000", 2024-09-21T10:42:43.127 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001594294 2024-09-21T10:42:43.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.127 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.127 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.127 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:46.119671+0000", 2024-09-21T10:42:43.127 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.084643353000000005 2024-09-21T10:42:43.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.127 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.128 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:42:43.128 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:46.119718+0000", 2024-09-21T10:42:43.128 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6768999999999999e-05 2024-09-21T10:42:43.128 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.128 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.128 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:42:43.128 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:46.119756+0000", 2024-09-21T10:42:43.128 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8232000000000003e-05 2024-09-21T10:42:43.128 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.128 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:42:43.128 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.128 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.128 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.128 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:139369 53.8 53:111f1ba3:::benchmark_data_smithi070_228967_object10449:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1339)", 2024-09-21T10:42:43.129 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:42:04.849122+0000", 2024-09-21T10:42:43.129 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 38.207550552000001, 2024-09-21T10:42:43.129 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.088413628999999994, 2024-09-21T10:42:43.129 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:42:43.129 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:42:43.129 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:42:43.129 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:42:43.129 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:42:43.129 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 139369 2024-09-21T10:42:43.129 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.129 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:42:43.129 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.129 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:42:43.130 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:04.849122+0000", 2024-09-21T10:42:43.130 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.130 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.130 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.130 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:42:43.130 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:04.849122+0000", 2024-09-21T10:42:43.130 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.130 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.130 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.130 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:42:43.130 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:04.849120+0000", 2024-09-21T10:42:43.130 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:42:43.130 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.131 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.131 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:42:43.131 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:04.849127+0000", 2024-09-21T10:42:43.131 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1550000000000001e-06 2024-09-21T10:42:43.131 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.131 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.131 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:42:43.131 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:04.849136+0000", 2024-09-21T10:42:43.131 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1190000000000003e-06 2024-09-21T10:42:43.131 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.131 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.131 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:42:43.131 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:04.849142+0000", 2024-09-21T10:42:43.131 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6169999999999998e-06 2024-09-21T10:42:43.132 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.132 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.132 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:42:43.132 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:04.849175+0000", 2024-09-21T10:42:43.132 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2934e-05 2024-09-21T10:42:43.132 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.132 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.132 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:42:43.132 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:04.849254+0000", 2024-09-21T10:42:43.132 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8808999999999997e-05 2024-09-21T10:42:43.132 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.132 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.132 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:42:43.132 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:04.849603+0000", 2024-09-21T10:42:43.133 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034925400000000001 2024-09-21T10:42:43.133 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.133 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.133 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:42:43.133 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:04.851222+0000", 2024-09-21T10:42:43.133 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0016195999999999999 2024-09-21T10:42:43.133 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.133 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.133 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.133 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:04.908767+0000", 2024-09-21T10:42:43.133 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.057545016999999997 2024-09-21T10:42:43.133 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.133 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.133 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.134 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:04.937444+0000", 2024-09-21T10:42:43.134 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.028676602999999998 2024-09-21T10:42:43.134 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.134 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.134 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:42:43.134 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:04.937493+0000", 2024-09-21T10:42:43.134 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8741e-05 2024-09-21T10:42:43.134 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.134 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.134 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:42:43.134 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:04.937535+0000", 2024-09-21T10:42:43.134 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2352999999999999e-05 2024-09-21T10:42:43.134 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.135 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:42:43.135 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.135 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.135 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.135 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:148407 53.8 53:16194cd2:::benchmark_data_smithi070_228967_object19487:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:42:43.135 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:42:13.548705+0000", 2024-09-21T10:42:43.135 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 29.507967283999999, 2024-09-21T10:42:43.135 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.088726602000000002, 2024-09-21T10:42:43.135 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:42:43.135 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:42:43.135 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:42:43.135 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:42:43.135 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:42:43.135 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 148407 2024-09-21T10:42:43.136 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.136 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:42:43.136 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.136 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:42:43.136 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:13.548705+0000", 2024-09-21T10:42:43.136 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.136 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.136 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.136 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:42:43.136 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:13.548705+0000", 2024-09-21T10:42:43.136 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.136 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.136 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.136 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:42:43.137 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:13.548703+0000", 2024-09-21T10:42:43.137 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:42:43.137 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.137 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.137 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:42:43.137 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:13.548709+0000", 2024-09-21T10:42:43.137 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4289999999999998e-06 2024-09-21T10:42:43.137 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.137 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.137 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:42:43.137 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:13.548725+0000", 2024-09-21T10:42:43.137 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5489e-05 2024-09-21T10:42:43.137 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.137 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.137 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:42:43.137 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:13.548732+0000", 2024-09-21T10:42:43.138 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9450000000000004e-06 2024-09-21T10:42:43.138 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.138 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.138 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:42:43.138 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:13.548773+0000", 2024-09-21T10:42:43.138 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0784000000000001e-05 2024-09-21T10:42:43.138 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.138 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.138 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:42:43.138 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:13.548886+0000", 2024-09-21T10:42:43.138 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000113403 2024-09-21T10:42:43.138 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.138 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.138 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:42:43.138 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:13.549290+0000", 2024-09-21T10:42:43.139 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00040441099999999997 2024-09-21T10:42:43.139 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.139 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.139 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:42:43.139 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:13.552635+0000", 2024-09-21T10:42:43.139 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0033441679999999998 2024-09-21T10:42:43.139 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.139 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.139 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.139 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:13.555312+0000", 2024-09-21T10:42:43.139 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002677497 2024-09-21T10:42:43.139 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.139 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.139 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.140 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:13.637335+0000", 2024-09-21T10:42:43.140 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.082022784000000001 2024-09-21T10:42:43.140 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.140 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.140 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:42:43.140 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:13.637383+0000", 2024-09-21T10:42:43.140 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8571e-05 2024-09-21T10:42:43.140 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.140 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.140 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:42:43.140 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:13.637431+0000", 2024-09-21T10:42:43.140 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8019999999999998e-05 2024-09-21T10:42:43.140 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.140 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:42:43.141 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.141 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.141 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.141 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:149478 53.8 53:10e8e2f8:::benchmark_data_smithi070_228967_object20558:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:42:43.141 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:42:14.689470+0000", 2024-09-21T10:42:43.141 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 28.367202473999999, 2024-09-21T10:42:43.141 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100990149, 2024-09-21T10:42:43.141 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:42:43.141 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:42:43.141 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:42:43.141 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:42:43.141 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:42:43.141 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 149478 2024-09-21T10:42:43.141 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.141 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:42:43.141 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.142 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:42:43.142 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.689470+0000", 2024-09-21T10:42:43.142 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.142 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.142 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.142 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:42:43.142 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.689470+0000", 2024-09-21T10:42:43.142 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.142 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.142 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.142 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:42:43.142 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.689468+0000", 2024-09-21T10:42:43.142 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:42:43.142 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.143 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.143 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:42:43.143 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.689474+0000", 2024-09-21T10:42:43.143 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9560000000000002e-06 2024-09-21T10:42:43.143 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.143 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.143 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:42:43.143 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.689487+0000", 2024-09-21T10:42:43.143 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2499e-05 2024-09-21T10:42:43.143 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.143 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.143 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:42:43.143 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.689497+0000", 2024-09-21T10:42:43.143 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.023e-05 2024-09-21T10:42:43.143 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.144 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.144 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:42:43.144 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.689544+0000", 2024-09-21T10:42:43.144 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6703999999999999e-05 2024-09-21T10:42:43.144 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.144 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.144 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:42:43.144 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.689663+0000", 2024-09-21T10:42:43.144 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119355 2024-09-21T10:42:43.144 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.144 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.144 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:42:43.144 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.690075+0000", 2024-09-21T10:42:43.144 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000411467 2024-09-21T10:42:43.144 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.144 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.145 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:42:43.145 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.695058+0000", 2024-09-21T10:42:43.145 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0049839389999999997 2024-09-21T10:42:43.145 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.145 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.145 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.145 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.780094+0000", 2024-09-21T10:42:43.145 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.085035225000000006 2024-09-21T10:42:43.145 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.145 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.145 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.145 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.790371+0000", 2024-09-21T10:42:43.145 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.010277435999999999 2024-09-21T10:42:43.145 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.146 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.146 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:42:43.146 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.790423+0000", 2024-09-21T10:42:43.146 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1966999999999999e-05 2024-09-21T10:42:43.146 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.146 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.146 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:42:43.146 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.790460+0000", 2024-09-21T10:42:43.146 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6711000000000001e-05 2024-09-21T10:42:43.146 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.146 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:42:43.146 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.146 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.146 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.147 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:149481 53.18 53:1dddc79d:::benchmark_data_smithi070_228967_object20561:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:42:43.147 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:42:14.692316+0000", 2024-09-21T10:42:43.147 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 28.364356393000001, 2024-09-21T10:42:43.147 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.095082052, 2024-09-21T10:42:43.147 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:42:43.147 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:42:43.147 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:42:43.147 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:42:43.147 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:42:43.147 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 149481 2024-09-21T10:42:43.147 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.147 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:42:43.147 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.147 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:42:43.147 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.692316+0000", 2024-09-21T10:42:43.148 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.148 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.148 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.148 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:42:43.148 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.692316+0000", 2024-09-21T10:42:43.148 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.148 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.148 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.148 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:42:43.148 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.692314+0000", 2024-09-21T10:42:43.148 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:42:43.148 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.148 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.148 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:42:43.148 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.692320+0000", 2024-09-21T10:42:43.149 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.319e-06 2024-09-21T10:42:43.149 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.149 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.149 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:42:43.149 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.692334+0000", 2024-09-21T10:42:43.149 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4011e-05 2024-09-21T10:42:43.149 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.149 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.149 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:42:43.149 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.692342+0000", 2024-09-21T10:42:43.149 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5009999999999998e-06 2024-09-21T10:42:43.149 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.149 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.149 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:42:43.149 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.692453+0000", 2024-09-21T10:42:43.150 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011135499999999999 2024-09-21T10:42:43.150 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.150 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.150 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:42:43.150 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.692558+0000", 2024-09-21T10:42:43.150 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000105214 2024-09-21T10:42:43.150 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.150 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.150 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:42:43.150 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.692994+0000", 2024-09-21T10:42:43.150 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00043552400000000001 2024-09-21T10:42:43.150 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.151 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.151 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:42:43.151 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.695191+0000", 2024-09-21T10:42:43.151 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021967739999999999 2024-09-21T10:42:43.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.151 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.151 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.151 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.785049+0000", 2024-09-21T10:42:43.151 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.089858510000000003 2024-09-21T10:42:43.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.151 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.151 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.151 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.787279+0000", 2024-09-21T10:42:43.151 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022303010000000001 2024-09-21T10:42:43.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.152 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.152 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:42:43.152 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.787359+0000", 2024-09-21T10:42:43.152 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9586999999999995e-05 2024-09-21T10:42:43.152 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.152 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.152 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:42:43.152 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.787398+0000", 2024-09-21T10:42:43.152 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8770000000000003e-05 2024-09-21T10:42:43.152 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.152 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:42:43.152 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.152 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.152 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.152 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:150502 53.8 53:16f8466a:::benchmark_data_smithi070_228967_object21582:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:42:43.153 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:42:15.781123+0000", 2024-09-21T10:42:43.153 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 27.27554868, 2024-09-21T10:42:43.153 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.107127848, 2024-09-21T10:42:43.153 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:42:43.153 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:42:43.153 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:42:43.153 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:42:43.153 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:42:43.153 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 150502 2024-09-21T10:42:43.153 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.153 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:42:43.153 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.153 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:42:43.153 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.781123+0000", 2024-09-21T10:42:43.153 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.153 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.154 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.154 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:42:43.154 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.781123+0000", 2024-09-21T10:42:43.154 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.154 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.154 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.154 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:42:43.154 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.781122+0000", 2024-09-21T10:42:43.154 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:42:43.154 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.154 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.154 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:42:43.154 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.781129+0000", 2024-09-21T10:42:43.154 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6980000000000001e-06 2024-09-21T10:42:43.154 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.155 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.155 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:42:43.155 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.781140+0000", 2024-09-21T10:42:43.155 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0832000000000001e-05 2024-09-21T10:42:43.155 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.155 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.155 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:42:43.155 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.781145+0000", 2024-09-21T10:42:43.155 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3310000000000003e-06 2024-09-21T10:42:43.155 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.155 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.155 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:42:43.155 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.781201+0000", 2024-09-21T10:42:43.155 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6326000000000002e-05 2024-09-21T10:42:43.155 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.156 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.156 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:42:43.156 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.781316+0000", 2024-09-21T10:42:43.156 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000115028 2024-09-21T10:42:43.156 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.156 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.156 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:42:43.156 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.781745+0000", 2024-09-21T10:42:43.157 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00042860799999999997 2024-09-21T10:42:43.157 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.157 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.157 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:42:43.157 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.783714+0000", 2024-09-21T10:42:43.157 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0019695670000000002 2024-09-21T10:42:43.157 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.157 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.157 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.157 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.785918+0000", 2024-09-21T10:42:43.157 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022033420000000001 2024-09-21T10:42:43.157 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.157 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.157 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.158 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.888167+0000", 2024-09-21T10:42:43.158 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.1022491 2024-09-21T10:42:43.158 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.158 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.158 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:42:43.158 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.888213+0000", 2024-09-21T10:42:43.158 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5768e-05 2024-09-21T10:42:43.158 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.158 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.158 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:42:43.158 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.888251+0000", 2024-09-21T10:42:43.158 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8658000000000002e-05 2024-09-21T10:42:43.158 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.158 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:42:43.159 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.159 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.159 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:151891 53.8 53:11033a1c:::benchmark_data_smithi070_228967_object22971:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:42:43.159 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:42:17.152780+0000", 2024-09-21T10:42:43.159 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 25.903891618999999, 2024-09-21T10:42:43.159 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.085709881000000002, 2024-09-21T10:42:43.159 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:42:43.159 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:42:43.159 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:42:43.159 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:42:43.159 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:42:43.159 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 151891 2024-09-21T10:42:43.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.160 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:42:43.160 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.160 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:42:43.160 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.152780+0000", 2024-09-21T10:42:43.160 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.160 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.160 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.160 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:42:43.160 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.152780+0000", 2024-09-21T10:42:43.160 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.160 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.160 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.160 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:42:43.161 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.152779+0000", 2024-09-21T10:42:43.161 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:42:43.161 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.161 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.161 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:42:43.161 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.152786+0000", 2024-09-21T10:42:43.161 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7429999999999996e-06 2024-09-21T10:42:43.161 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.161 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.161 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:42:43.161 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.152795+0000", 2024-09-21T10:42:43.161 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6830000000000003e-06 2024-09-21T10:42:43.161 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.161 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.162 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:42:43.162 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.152802+0000", 2024-09-21T10:42:43.162 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.956e-06 2024-09-21T10:42:43.162 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.162 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.162 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:42:43.162 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.152842+0000", 2024-09-21T10:42:43.162 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9511999999999997e-05 2024-09-21T10:42:43.162 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.162 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.162 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:42:43.162 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.152981+0000", 2024-09-21T10:42:43.162 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000138928 2024-09-21T10:42:43.162 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.163 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.163 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:42:43.163 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.153407+0000", 2024-09-21T10:42:43.163 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00042597000000000002 2024-09-21T10:42:43.163 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.163 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.163 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:42:43.163 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.155905+0000", 2024-09-21T10:42:43.163 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0024984769999999998 2024-09-21T10:42:43.163 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.163 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.163 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.163 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.158833+0000", 2024-09-21T10:42:43.163 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029274869999999999 2024-09-21T10:42:43.164 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.164 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.164 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.164 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.238224+0000", 2024-09-21T10:42:43.164 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.079391626000000007 2024-09-21T10:42:43.164 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.164 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.164 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:42:43.164 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.238453+0000", 2024-09-21T10:42:43.164 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00022906399999999999 2024-09-21T10:42:43.164 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.164 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.164 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:42:43.164 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.238490+0000", 2024-09-21T10:42:43.165 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7075e-05 2024-09-21T10:42:43.165 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.165 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:42:43.165 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.165 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.165 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.165 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:151895 53.0 53:0504c56e:::benchmark_data_smithi070_228967_object22975:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:42:43.165 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:42:17.156006+0000", 2024-09-21T10:42:43.165 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 25.900666577999999, 2024-09-21T10:42:43.165 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.091976057, 2024-09-21T10:42:43.165 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:42:43.165 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:42:43.165 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:42:43.165 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:42:43.166 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:42:43.166 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 151895 2024-09-21T10:42:43.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.166 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:42:43.166 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.166 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:42:43.166 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.156006+0000", 2024-09-21T10:42:43.166 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.166 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.166 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:42:43.166 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.156006+0000", 2024-09-21T10:42:43.166 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.167 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.167 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.167 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:42:43.167 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.156004+0000", 2024-09-21T10:42:43.167 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:42:43.167 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.167 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.167 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:42:43.167 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.156010+0000", 2024-09-21T10:42:43.167 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0630000000000002e-06 2024-09-21T10:42:43.167 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.167 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.167 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:42:43.168 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.156020+0000", 2024-09-21T10:42:43.168 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3479999999999993e-06 2024-09-21T10:42:43.168 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.168 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.168 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:42:43.168 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.156026+0000", 2024-09-21T10:42:43.168 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0299999999999999e-06 2024-09-21T10:42:43.168 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.168 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.168 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:42:43.168 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.156056+0000", 2024-09-21T10:42:43.168 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0229000000000001e-05 2024-09-21T10:42:43.169 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.169 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.169 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:42:43.169 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.156136+0000", 2024-09-21T10:42:43.169 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0018000000000005e-05 2024-09-21T10:42:43.169 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.169 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.169 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:42:43.169 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.156494+0000", 2024-09-21T10:42:43.169 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00035786699999999999 2024-09-21T10:42:43.169 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.169 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.169 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:42:43.169 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.158749+0000", 2024-09-21T10:42:43.170 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022554290000000002 2024-09-21T10:42:43.170 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.170 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.170 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.170 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.159189+0000", 2024-09-21T10:42:43.170 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000440178 2024-09-21T10:42:43.170 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.170 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.170 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.170 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.247897+0000", 2024-09-21T10:42:43.170 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.088707649 2024-09-21T10:42:43.170 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.170 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.171 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:42:43.171 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.247946+0000", 2024-09-21T10:42:43.171 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9002999999999998e-05 2024-09-21T10:42:43.171 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.171 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.171 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:42:43.171 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.247982+0000", 2024-09-21T10:42:43.171 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5448999999999999e-05 2024-09-21T10:42:43.171 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.171 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:42:43.171 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.171 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.171 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.171 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:151898 53.0 53:07ad7f83:::benchmark_data_smithi070_228967_object22978:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:42:43.172 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:42:17.160670+0000", 2024-09-21T10:42:43.172 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 25.896002067000001, 2024-09-21T10:42:43.172 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.088851665999999996, 2024-09-21T10:42:43.172 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:42:43.172 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:42:43.172 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:42:43.172 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:42:43.172 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:42:43.172 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 151898 2024-09-21T10:42:43.172 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.172 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:42:43.172 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.172 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:42:43.172 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.160670+0000", 2024-09-21T10:42:43.173 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.173 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.173 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.173 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:42:43.173 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.160670+0000", 2024-09-21T10:42:43.173 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.173 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.173 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.173 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:42:43.173 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.160668+0000", 2024-09-21T10:42:43.173 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:42:43.173 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.173 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.173 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:42:43.173 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.160676+0000", 2024-09-21T10:42:43.174 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.075e-06 2024-09-21T10:42:43.174 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.174 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.174 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:42:43.174 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.160687+0000", 2024-09-21T10:42:43.174 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1807000000000001e-05 2024-09-21T10:42:43.174 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.174 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.174 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:42:43.174 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.160702+0000", 2024-09-21T10:42:43.174 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4168999999999999e-05 2024-09-21T10:42:43.174 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.174 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.175 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:42:43.175 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.160739+0000", 2024-09-21T10:42:43.175 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7589000000000002e-05 2024-09-21T10:42:43.175 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.175 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.175 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:42:43.175 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.160873+0000", 2024-09-21T10:42:43.175 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000133891 2024-09-21T10:42:43.175 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.175 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.175 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:42:43.175 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.161227+0000", 2024-09-21T10:42:43.175 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00035380699999999998 2024-09-21T10:42:43.175 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.176 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.176 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:42:43.176 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.164006+0000", 2024-09-21T10:42:43.176 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0027791909999999999 2024-09-21T10:42:43.176 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.176 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.176 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.176 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.164234+0000", 2024-09-21T10:42:43.176 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00022799499999999999 2024-09-21T10:42:43.176 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.176 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.176 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.176 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.249451+0000", 2024-09-21T10:42:43.177 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.085217434999999994 2024-09-21T10:42:43.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.177 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.177 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:42:43.177 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.249488+0000", 2024-09-21T10:42:43.177 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6557999999999998e-05 2024-09-21T10:42:43.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.177 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.177 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:42:43.177 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.249522+0000", 2024-09-21T10:42:43.177 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3720000000000002e-05 2024-09-21T10:42:43.177 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.177 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:42:43.177 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.178 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.178 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:151903 53.8 53:1620c14c:::benchmark_data_smithi070_228967_object22983:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:42:43.178 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:42:17.161324+0000", 2024-09-21T10:42:43.178 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 25.895347655999998, 2024-09-21T10:42:43.178 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.088191720000000001, 2024-09-21T10:42:43.178 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:42:43.178 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:42:43.178 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:42:43.178 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:42:43.178 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:42:43.178 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 151903 2024-09-21T10:42:43.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.179 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:42:43.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.179 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:42:43.179 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.161324+0000", 2024-09-21T10:42:43.179 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.179 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.179 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:42:43.179 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.161324+0000", 2024-09-21T10:42:43.179 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:43.179 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.179 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:42:43.179 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.161324+0000", 2024-09-21T10:42:43.180 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T10:42:43.180 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.180 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.180 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:42:43.180 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.161328+0000", 2024-09-21T10:42:43.180 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6090000000000001e-06 2024-09-21T10:42:43.180 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.180 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.180 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:42:43.180 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.161338+0000", 2024-09-21T10:42:43.180 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0219e-05 2024-09-21T10:42:43.180 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.180 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.180 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:42:43.181 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.161343+0000", 2024-09-21T10:42:43.181 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0640000000000001e-06 2024-09-21T10:42:43.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.181 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.181 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:42:43.181 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.161390+0000", 2024-09-21T10:42:43.181 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7191000000000003e-05 2024-09-21T10:42:43.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.181 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.181 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:42:43.181 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.161463+0000", 2024-09-21T10:42:43.181 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3378000000000001e-05 2024-09-21T10:42:43.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.181 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.182 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:42:43.182 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.161832+0000", 2024-09-21T10:42:43.182 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00036868099999999999 2024-09-21T10:42:43.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.182 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.182 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:42:43.182 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.164241+0000", 2024-09-21T10:42:43.182 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0024092300000000001 2024-09-21T10:42:43.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.182 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.182 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.182 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.164528+0000", 2024-09-21T10:42:43.182 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000286268 2024-09-21T10:42:43.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.183 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.183 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:43.183 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.249439+0000", 2024-09-21T10:42:43.183 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.084910952999999997 2024-09-21T10:42:43.183 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.183 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.183 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:42:43.183 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.249482+0000", 2024-09-21T10:42:43.183 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3491000000000001e-05 2024-09-21T10:42:43.183 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:43.183 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:43.183 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:42:43.183 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.249516+0000", 2024-09-21T10:42:43.183 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4184999999999999e-05 2024-09-21T10:42:43.184 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.184 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:42:43.184 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.184 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:43.184 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:42:43.184 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:42:43.216 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:43.250 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:43.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:43.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:43.351 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:43.386 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:43.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:43.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:43.408 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:42:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:42:43] "GET /metrics HTTP/1.1" 200 36630 "" "Prometheus/2.51.0" 2024-09-21T10:42:43.488 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:43.514 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:43.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:43.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:43.616 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:43.651 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:43.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:43.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:43.753 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:43.786 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:43.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:43.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:43 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 2520/258345 objects degraded (0.975%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:42:43.858 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:43 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 2520/258345 objects degraded (0.975%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:42:43.858 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:43 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 2520/258345 objects degraded (0.975%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:42:43.887 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:43.933 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:43.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:43.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:44.016 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:42:44.035 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:44.094 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:44.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:44.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:44.196 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:44.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:44.248 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:44.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:44.333 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:42:44.348 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:44.380 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:44.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:44.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:44.483 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:44.512 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:42:44.519 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:44.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:44.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:44.621 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:44.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:44.660 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:44.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:44.663 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:44 smithi160 ceph-mon[101879]: pgmap v2260: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 5.1 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 1.06k op/s; 2520/252012 objects degraded (1.000%); 2634/252012 objects misplaced (1.045%); 5.0 MiB/s, 80 objects/s recovering 2024-09-21T10:42:44.730 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:44 smithi070 ceph-mon[107385]: pgmap v2260: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 5.1 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 1.06k op/s; 2520/252012 objects degraded (1.000%); 2634/252012 objects misplaced (1.045%); 5.0 MiB/s, 80 objects/s recovering 2024-09-21T10:42:44.731 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:44 smithi070 ceph-mon[112139]: pgmap v2260: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 5.1 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 1.06k op/s; 2520/252012 objects degraded (1.000%); 2634/252012 objects misplaced (1.045%); 5.0 MiB/s, 80 objects/s recovering 2024-09-21T10:42:44.761 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:44.796 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:44.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:44.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:44.898 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:44.933 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:44.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:44.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:45.035 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:45.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:45.057 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:45.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:45.157 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:45.185 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:45.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:45.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:45.286 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:45.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:45.313 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:45.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:45.414 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:45.442 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:45.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:45.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:45.544 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:45.583 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:45.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:45.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:45.684 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:45.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:45.710 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:45.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:45.811 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:45.849 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:45.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:45.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:45.950 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:45.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:45.977 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:45.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:46.078 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:46.111 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:46.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:46.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:46.213 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:46.244 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:46.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:46.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:46.346 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:46.378 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:46.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:46.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:46.479 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:46.538 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:46.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:46.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:46.567 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:46 smithi160 ceph-mon[101879]: pgmap v2261: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 4.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 1.01k op/s; 2520/242619 objects degraded (1.039%); 1913/242619 objects misplaced (0.788%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T10:42:46.603 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:42:46.604 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T10:42:46.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:46.604 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:180596 53.c 53:35076b9b:::benchmark_data_smithi070_228967_object51676:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1345)", 2024-09-21T10:42:46.604 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:42:46.600820+0000", 2024-09-21T10:42:46.604 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.00088500299999999998, 2024-09-21T10:42:46.604 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000915722, 2024-09-21T10:42:46.604 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:42:46.604 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:42:46.604 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:42:46.604 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:42:46.604 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:42:46.604 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 180596 2024-09-21T10:42:46.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:46.604 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:42:46.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:46.605 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:42:46.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:46.600820+0000", 2024-09-21T10:42:46.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:46.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:46.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:46.605 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:42:46.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:46.600820+0000", 2024-09-21T10:42:46.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:46.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:46.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:46.605 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:42:46.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:46.600818+0000", 2024-09-21T10:42:46.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:42:46.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:46.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:46.606 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:42:46.606 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:46.600825+0000", 2024-09-21T10:42:46.606 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8900000000000001e-06 2024-09-21T10:42:46.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:46.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:46.606 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:42:46.606 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:46.600838+0000", 2024-09-21T10:42:46.606 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3497e-05 2024-09-21T10:42:46.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:46.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:46.606 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:42:46.606 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:46.600845+0000", 2024-09-21T10:42:46.606 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.489e-06 2024-09-21T10:42:46.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:46.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:46.607 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:42:46.607 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:46.600896+0000", 2024-09-21T10:42:46.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1511000000000002e-05 2024-09-21T10:42:46.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:46.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:46.607 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:42:46.607 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:46.601023+0000", 2024-09-21T10:42:46.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012686899999999999 2024-09-21T10:42:46.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:46.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:46.607 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:42:46.607 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:46.601159+0000", 2024-09-21T10:42:46.608 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013616600000000001 2024-09-21T10:42:46.608 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:46.608 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:42:46.608 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:46.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:46.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:46.608 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:180602 53.b 53:da959962:::benchmark_data_smithi070_228967_object51682:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1345)", 2024-09-21T10:42:46.608 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:42:46.601534+0000", 2024-09-21T10:42:46.608 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.00017125800000000001, 2024-09-21T10:42:46.608 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00029027499999999999, 2024-09-21T10:42:46.608 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:42:46.608 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:42:46.608 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:42:46.608 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:42:46.609 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:42:46.609 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 180602 2024-09-21T10:42:46.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:46.609 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:42:46.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:46.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:42:46.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:46.601534+0000", 2024-09-21T10:42:46.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:46.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:46.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:46.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:42:46.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:46.601534+0000", 2024-09-21T10:42:46.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:42:46.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:46.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:46.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:42:46.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:46.601533+0000", 2024-09-21T10:42:46.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:42:46.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:46.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:46.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:42:46.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:46.601538+0000", 2024-09-21T10:42:46.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0139999999999998e-06 2024-09-21T10:42:46.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:46.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:46.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:42:46.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:46.601549+0000", 2024-09-21T10:42:46.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1071e-05 2024-09-21T10:42:46.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:46.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:46.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:42:46.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:46.601555+0000", 2024-09-21T10:42:46.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6479999999999997e-06 2024-09-21T10:42:46.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:46.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:46.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:42:46.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:46.601599+0000", 2024-09-21T10:42:46.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3599e-05 2024-09-21T10:42:46.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:46.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:46.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:42:46.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:46.601708+0000", 2024-09-21T10:42:46.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000108597 2024-09-21T10:42:46.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:42:46.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:42:46.612 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:42:46.612 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:46.601808+0000", 2024-09-21T10:42:46.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000100546 2024-09-21T10:42:46.612 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:46.612 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:42:46.612 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:46.612 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:42:46.612 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T10:42:46.612 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 2 2024-09-21T10:42:46.612 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:42:46.640 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:46.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:46.678 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:46.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:46.696 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:46 smithi070 ceph-mon[107385]: pgmap v2261: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 4.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 1.01k op/s; 2520/242619 objects degraded (1.039%); 1913/242619 objects misplaced (0.788%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T10:42:46.696 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:46 smithi070 ceph-mon[112139]: pgmap v2261: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 4.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 1.01k op/s; 2520/242619 objects degraded (1.039%); 1913/242619 objects misplaced (0.788%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T10:42:46.779 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:46.801 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:46.846 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:46.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:46.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:46.861 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:46.948 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:46.977 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:46.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:46.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:47.078 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:47.119 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:47.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:47.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:47.221 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:47.263 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:47.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:47.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:47.317 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:42:47.365 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:47.403 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:47.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:47.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:47.505 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:47.539 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:47.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:47.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:47.619 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:42:47.640 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:47.689 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:47.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:47.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:47.791 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:47.816 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:47.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:47.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:47.918 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:47.945 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:47.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:47.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:48.047 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:48.070 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:48.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:48.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:48.171 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:48.213 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:48.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:48.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:48.314 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:48.341 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:48.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:48.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:48.442 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:48.480 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:48.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:48.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:48.581 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:48.614 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:48.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:48.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:48.716 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:48.754 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:48.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:48.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:48 smithi160 ceph-mon[101879]: pgmap v2262: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 4.8 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 1.06k op/s; 2520/236361 objects degraded (1.066%); 1913/236361 objects misplaced (0.809%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T10:42:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:48 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 2520/236361 objects degraded (1.066%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:42:48.855 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:48.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:48 smithi070 ceph-mon[107385]: pgmap v2262: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 4.8 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 1.06k op/s; 2520/236361 objects degraded (1.066%); 1913/236361 objects misplaced (0.809%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T10:42:48.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:48 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 2520/236361 objects degraded (1.066%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:42:48.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:48 smithi070 ceph-mon[112139]: pgmap v2262: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 4.8 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 1.06k op/s; 2520/236361 objects degraded (1.066%); 1913/236361 objects misplaced (0.809%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T10:42:48.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:48 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 2520/236361 objects degraded (1.066%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:42:48.880 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:48.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:48.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:48.982 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:49.024 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:49.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:49.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:49.125 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:49.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:49.169 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:49.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:49.270 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:49.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:49.300 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:49.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:49.400 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:49.443 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:49.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:49.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:49.544 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:49.577 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:49.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:49.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:49.607 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:42:49.607 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T10:42:49.607 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T10:42:49.607 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T10:42:49.607 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:42:49.678 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:49.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:49.743 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:49.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:49.843 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:49.878 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:49.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:49.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:49.979 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:50.013 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:50.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:50.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:50.114 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:50.144 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:50.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:50.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:50.246 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:50.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:50.279 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:50.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:50.301 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:42:50.381 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:50.418 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:50.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:50.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:50.522 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:50.561 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:50.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:50.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:50 smithi160 ceph-mon[101879]: pgmap v2263: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 4.8 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 1.06k op/s; 2520/236361 objects degraded (1.066%); 1913/236361 objects misplaced (0.809%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T10:42:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:42:50.642 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:42:50.662 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:50 smithi070 ceph-mon[107385]: pgmap v2263: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 4.8 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 1.06k op/s; 2520/236361 objects degraded (1.066%); 1913/236361 objects misplaced (0.809%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T10:42:50.662 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:42:50.662 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:50 smithi070 ceph-mon[112139]: pgmap v2263: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 4.8 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 1.06k op/s; 2520/236361 objects degraded (1.066%); 1913/236361 objects misplaced (0.809%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T10:42:50.662 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:42:50.663 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:50.701 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:50.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:50.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:50.803 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:50.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:50.828 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:50.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:50.929 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:50.967 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:50.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:50.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:50.973 INFO:teuthology.orchestra.run.smithi070.stderr:no valid command found; 10 closest matches: 2024-09-21T10:42:50.973 INFO:teuthology.orchestra.run.smithi070.stderr:0 2024-09-21T10:42:50.973 INFO:teuthology.orchestra.run.smithi070.stderr:1 2024-09-21T10:42:50.973 INFO:teuthology.orchestra.run.smithi070.stderr:2 2024-09-21T10:42:50.973 INFO:teuthology.orchestra.run.smithi070.stderr:abort 2024-09-21T10:42:50.973 INFO:teuthology.orchestra.run.smithi070.stderr:assert 2024-09-21T10:42:50.973 INFO:teuthology.orchestra.run.smithi070.stderr:bench [] [] [] [] 2024-09-21T10:42:50.973 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:42:50.973 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs files list 2024-09-21T10:42:50.973 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs stats 2024-09-21T10:42:50.974 INFO:teuthology.orchestra.run.smithi070.stderr:bluestore allocator dump block 2024-09-21T10:42:50.974 INFO:teuthology.orchestra.run.smithi070.stderr:Error EINVAL: invalid command 2024-09-21T10:42:51.068 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:51.103 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:51.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:51.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:51.204 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:51.251 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:51.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:51.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:51.353 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:51.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:51.390 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:51.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:51.491 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:51.526 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:51.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:51.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:51.628 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:51.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:51.687 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:51.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:51.788 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:51.817 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:51.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:51.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:51.919 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:51.958 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:51.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:51.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:52.060 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:52.093 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:52.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:52.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:52.194 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:52.203 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:42:52.203 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:42:52.203 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-21T10:42:52.235 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:52.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:52.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:52.336 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:52.391 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:52.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:52.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:52.492 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:52.562 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:52.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:52.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:52.664 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:52.685 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:52 smithi160 ceph-mon[101879]: pgmap v2264: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 4.6 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 1.04k op/s; 2520/226986 objects degraded (1.110%); 1158/226986 objects misplaced (0.510%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T10:42:52.686 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:52 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4277579461' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:42:52.699 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:52.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:52.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:52.801 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:52.834 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:52.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:52.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:52.858 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:52 smithi070 ceph-mon[112139]: pgmap v2264: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 4.6 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 1.04k op/s; 2520/226986 objects degraded (1.110%); 1158/226986 objects misplaced (0.510%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T10:42:52.859 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:52 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4277579461' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:42:52.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:52 smithi070 ceph-mon[107385]: pgmap v2264: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 4.6 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 1.04k op/s; 2520/226986 objects degraded (1.110%); 1158/226986 objects misplaced (0.510%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T10:42:52.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:52 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4277579461' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:42:52.935 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:52.984 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:52.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:52.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:52.995 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:42:52.995 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:42:52.995 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:42:52.995 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:42:52.996 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:42:52.996 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:42:52.996 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:42:52.996 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:42:52.996 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:42:52.996 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:42:52.996 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:42:52.996 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:42:52.996 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:42:52.996 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:42:52.996 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:42:52.997 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:42:53.086 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:53.128 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:53.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:53.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:53.230 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:53.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:53.267 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:53.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:53.341 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:42:53.341 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:42:53.368 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:53.411 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:53.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:53.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:53.437 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:42:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:42:53] "GET /metrics HTTP/1.1" 200 36628 "" "Prometheus/2.51.0" 2024-09-21T10:42:53.512 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:53.548 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:53.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:53.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:53.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:53 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 2520/226986 objects degraded (1.110%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:42:53.649 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:53.686 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:53.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:53.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:53.788 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:53.823 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:53.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:53.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:53.861 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:53 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 2520/226986 objects degraded (1.110%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:42:53.861 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:53 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 2520/226986 objects degraded (1.110%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:42:53.924 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:53.953 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:53.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:53.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:54.054 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:42:54.055 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:54.100 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:54.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:54.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:54.202 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:54.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:54.253 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:54.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:54.345 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:42:54.354 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:54.379 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:54.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:54.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:54.481 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:54.517 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:54.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:54.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:54.619 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:54.659 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:54.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:54.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:54.760 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:54.789 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:54 smithi160 ceph-mon[101879]: pgmap v2265: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 925 op/s; 2520/220542 objects degraded (1.143%); 1158/220542 objects misplaced (0.525%); 2.7 MiB/s, 43 objects/s recovering 2024-09-21T10:42:54.793 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:54.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:54.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:54.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:54 smithi070 ceph-mon[107385]: pgmap v2265: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 925 op/s; 2520/220542 objects degraded (1.143%); 1158/220542 objects misplaced (0.525%); 2.7 MiB/s, 43 objects/s recovering 2024-09-21T10:42:54.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:54 smithi070 ceph-mon[112139]: pgmap v2265: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 925 op/s; 2520/220542 objects degraded (1.143%); 1158/220542 objects misplaced (0.525%); 2.7 MiB/s, 43 objects/s recovering 2024-09-21T10:42:54.895 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:54.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:54.930 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:54.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:55.031 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:55.061 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:55.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:55.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:55.162 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:55.185 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:55.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:55.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:55.287 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:55.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:55.324 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:55.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:55.425 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:55.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:55.461 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:55.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:55.562 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:55.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:55.592 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:55.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:55.693 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:55.720 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:55.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:55.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:55.821 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:55.845 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:55.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:55.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:55.947 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:55.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:55.979 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:55.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:56.080 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:56.115 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:56.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:56.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:56.216 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:56.253 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:56.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:56.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:56.354 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:56.391 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:56.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:56.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:56.493 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:56.519 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:56.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:56.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:56.621 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:56.659 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:56.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:56.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:56.665 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:56 smithi070 ceph-mon[107385]: pgmap v2266: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 4.3 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 1.13k op/s; 2520/211449 objects degraded (1.192%); 360/211449 objects misplaced (0.170%); 3.9 MiB/s, 63 objects/s recovering 2024-09-21T10:42:56.665 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:56 smithi070 ceph-mon[112139]: pgmap v2266: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 4.3 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 1.13k op/s; 2520/211449 objects degraded (1.192%); 360/211449 objects misplaced (0.170%); 3.9 MiB/s, 63 objects/s recovering 2024-09-21T10:42:56.696 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:56 smithi160 ceph-mon[101879]: pgmap v2266: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 4.3 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 1.13k op/s; 2520/211449 objects degraded (1.192%); 360/211449 objects misplaced (0.170%); 3.9 MiB/s, 63 objects/s recovering 2024-09-21T10:42:56.761 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:56.791 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:56.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:56.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:56.892 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:56.920 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:56.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:56.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:57.006 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:42:57.006 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:42:57.022 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:57.058 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:57.119 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:57.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:57.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:57.224 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:57.256 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:57.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:57.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:57.358 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:57.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:57.389 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:57.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:57.461 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T10:42:57.487 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:57 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4009798359' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:42:57.487 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:57 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:42:57.488 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:57 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:42:57.488 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:57 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4009798359' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:42:57.488 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:57 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:42:57.488 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:57 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:42:57.489 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:57.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:57.558 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:57.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:57.658 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:57.663 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:57 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4009798359' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:42:57.663 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:57 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:42:57.663 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:57 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:42:57.694 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:57.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:57.696 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:42:57.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:57.797 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:57.824 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:57.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:57.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:57.925 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:57.955 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:42:57.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:57.956 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:57.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:58.057 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:58.079 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:58.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:58.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:58.180 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:58.202 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:58.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:58.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:58.303 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:58.325 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:58.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:58.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:58.426 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:58.448 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:58.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:58.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:58.550 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:58.577 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:58.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:58.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:58.591 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:58 smithi070 ceph-mon[107385]: pgmap v2267: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 1.03k op/s; 2520/205662 objects degraded (1.225%); 360/205662 objects misplaced (0.175%); 2.7 MiB/s, 43 objects/s recovering 2024-09-21T10:42:58.591 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:58 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:42:58.591 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:58 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4009798359' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:42:58.591 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:58 smithi070 ceph-mon[107385]: osdmap e1346: 8 total, 8 up, 6 in 2024-09-21T10:42:58.591 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:58 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:42:58.591 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:58 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:42:58.591 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:58 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:42:58.591 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:58 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 2520/205662 objects degraded (1.225%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:42:58.591 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:58 smithi070 ceph-mon[112139]: pgmap v2267: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 1.03k op/s; 2520/205662 objects degraded (1.225%); 360/205662 objects misplaced (0.175%); 2.7 MiB/s, 43 objects/s recovering 2024-09-21T10:42:58.592 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:58 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:42:58.592 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:58 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4009798359' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:42:58.592 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:58 smithi070 ceph-mon[112139]: osdmap e1346: 8 total, 8 up, 6 in 2024-09-21T10:42:58.592 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:58 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:42:58.592 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:58 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:42:58.592 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:58 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:42:58.592 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:58 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 2520/205662 objects degraded (1.225%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:42:58.679 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:58.714 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:58.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:58.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:58.816 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:58.838 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:58 smithi160 ceph-mon[101879]: pgmap v2267: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 1.03k op/s; 2520/205662 objects degraded (1.225%); 360/205662 objects misplaced (0.175%); 2.7 MiB/s, 43 objects/s recovering 2024-09-21T10:42:58.838 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:58 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:42:58.838 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:58 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4009798359' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:42:58.838 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:58 smithi160 ceph-mon[101879]: osdmap e1346: 8 total, 8 up, 6 in 2024-09-21T10:42:58.839 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:58 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:42:58.839 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:58 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:42:58.839 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:58 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:42:58.839 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:58 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 2520/205662 objects degraded (1.225%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:42:58.854 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:58.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:58.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:58.956 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:58.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:58.989 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:58.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:59.021 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:42:59.090 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:59.119 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:59.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:59.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:59.220 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:59.251 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:59.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:59.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:59.353 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:59.401 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:59.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:59.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:59.503 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:59.543 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:59.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:59.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:59.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:42:59 smithi070 ceph-mon[107385]: osdmap e1347: 8 total, 8 up, 6 in 2024-09-21T10:42:59.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:42:59 smithi070 ceph-mon[112139]: osdmap e1347: 8 total, 8 up, 6 in 2024-09-21T10:42:59.645 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:59.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:59.679 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:42:59.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:59.722 INFO:teuthology.orchestra.run.smithi070.stderr:no valid command found; 10 closest matches: 2024-09-21T10:42:59.722 INFO:teuthology.orchestra.run.smithi070.stderr:0 2024-09-21T10:42:59.722 INFO:teuthology.orchestra.run.smithi070.stderr:1 2024-09-21T10:42:59.722 INFO:teuthology.orchestra.run.smithi070.stderr:2 2024-09-21T10:42:59.722 INFO:teuthology.orchestra.run.smithi070.stderr:abort 2024-09-21T10:42:59.722 INFO:teuthology.orchestra.run.smithi070.stderr:assert 2024-09-21T10:42:59.722 INFO:teuthology.orchestra.run.smithi070.stderr:bench [] [] [] [] 2024-09-21T10:42:59.722 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:42:59.722 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs files list 2024-09-21T10:42:59.722 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs stats 2024-09-21T10:42:59.722 INFO:teuthology.orchestra.run.smithi070.stderr:bluestore allocator dump block 2024-09-21T10:42:59.722 INFO:teuthology.orchestra.run.smithi070.stderr:Error EINVAL: invalid command 2024-09-21T10:42:59.780 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:59.783 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:42:59 smithi160 ceph-mon[101879]: osdmap e1347: 8 total, 8 up, 6 in 2024-09-21T10:42:59.815 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:42:59.815 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:42:59.829 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:59.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:59.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:59.930 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:59.964 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:59.968 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:42:59.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:59.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:00.069 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:00.107 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:00.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:00.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:00.209 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:00.235 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:00.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:00.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:00.337 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:00.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:00.380 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:00.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:00.481 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:00.513 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:43:00.513 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:43:00.513 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.4 'trim stale osdmaps' 2024-09-21T10:43:00.517 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:00.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:00.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:00.519 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:43:00.619 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:00.656 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:00.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:00.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:00.758 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:00.801 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:00.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:00.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:00.819 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:00 smithi070 ceph-mon[107385]: pgmap v2270: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 888 op/s; 2520/205662 objects degraded (1.225%); 360/205662 objects misplaced (0.175%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T10:43:00.820 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:00 smithi070 ceph-mon[112139]: pgmap v2270: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 888 op/s; 2520/205662 objects degraded (1.225%); 360/205662 objects misplaced (0.175%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T10:43:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:00 smithi160 ceph-mon[101879]: pgmap v2270: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 95 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 888 op/s; 2520/205662 objects degraded (1.225%); 360/205662 objects misplaced (0.175%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T10:43:00.903 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:00.953 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:00.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:00.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:01.055 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:01.089 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:43:01.091 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:01.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:01.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:01.166 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:43:01.193 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:01.257 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:01.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:01.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:01.359 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:01.389 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:01.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:01.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:01.490 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:01.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:01.538 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:01.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:01.638 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:01.672 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:01.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:01.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:01.774 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:01.812 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:01.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:01.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:01.914 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:01.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:01.944 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:01.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:02.045 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:02.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:02.079 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:02.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:02.126 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:02.180 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:02.234 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:02.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:02.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:02.335 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:02.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:02.362 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:02.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:02.462 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:02.488 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:02.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:02.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:02.590 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:02.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:02.627 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:02.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:02.720 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:02 smithi070 ceph-mon[107385]: pgmap v2271: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 96 active+clean; 4.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 924 op/s; 1928/198336 objects degraded (0.972%); 746/198336 objects misplaced (0.376%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T10:43:02.720 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:02 smithi070 ceph-mon[112139]: pgmap v2271: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 96 active+clean; 4.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 924 op/s; 1928/198336 objects degraded (0.972%); 746/198336 objects misplaced (0.376%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T10:43:02.728 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:02.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:02.755 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:02.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:02.823 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:02 smithi160 ceph-mon[101879]: pgmap v2271: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 96 active+clean; 4.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 924 op/s; 1928/198336 objects degraded (0.972%); 746/198336 objects misplaced (0.376%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T10:43:02.855 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:02.892 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:02.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:02.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:02.993 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:03.029 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:03.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:03.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:03.131 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:03.142 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:43:03.142 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:43:03.174 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:03.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:03.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:03.180 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:43:03.180 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:43:03.212 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:43:03.212 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:43:03.253 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:43:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:43:03] "GET /metrics HTTP/1.1" 200 36628 "" "Prometheus/2.51.0" 2024-09-21T10:43:03.253 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:43:03.253 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:43:03.276 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:03.287 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:43:03.287 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:43:03.317 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:03.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:03.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:03.319 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:43:03.319 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:43:03.353 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:43:03.353 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:43:03.388 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:43:03.388 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:43:03.418 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:03.444 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:03.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:03.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:03.545 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:03.576 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:03.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:03.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:03.610 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:03.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:03 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3285604743' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:43:03.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:03 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 1928/198336 objects degraded (0.972%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:43:03.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:03 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3285604743' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:43:03.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:03 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 1928/198336 objects degraded (0.972%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:43:03.678 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:03.708 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:03.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:03.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:03.809 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:03.815 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:03 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3285604743' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:43:03.815 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:03 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 1928/198336 objects degraded (0.972%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:43:03.844 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:03.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:03.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:03.945 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:03.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:03.982 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:03.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:04.083 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:04.118 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:04.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:04.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:04.220 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:04.256 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:04.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:04.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:04.358 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:04.391 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:04.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:04.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:04.476 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:43:04.476 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T10:43:04.476 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T10:43:04.476 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T10:43:04.476 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.476 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:69148 53.3 53:c834d3e0:::benchmark_data_smithi070_228967_object69147:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1322)", 2024-09-21T10:43:04.476 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:40:59.628000+0000", 2024-09-21T10:43:04.476 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 124.843618487, 2024-09-21T10:43:04.476 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16083451500000001, 2024-09-21T10:43:04.476 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:43:04.476 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:43:04.477 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:43:04.477 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:43:04.477 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:04.477 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 69148 2024-09-21T10:43:04.477 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.477 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:43:04.477 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.477 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:43:04.477 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.628000+0000", 2024-09-21T10:43:04.477 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.477 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.477 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.477 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:43:04.477 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.628000+0000", 2024-09-21T10:43:04.477 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.477 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.478 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.478 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:43:04.478 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.627999+0000", 2024-09-21T10:43:04.478 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:43:04.478 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.478 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.478 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:43:04.478 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.628041+0000", 2024-09-21T10:43:04.478 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1968999999999997e-05 2024-09-21T10:43:04.478 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.478 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.478 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:43:04.478 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.628049+0000", 2024-09-21T10:43:04.479 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8729999999999996e-06 2024-09-21T10:43:04.479 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.479 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.479 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:43:04.479 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.628055+0000", 2024-09-21T10:43:04.479 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5829999999999997e-06 2024-09-21T10:43:04.479 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.479 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.479 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.479 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.628104+0000", 2024-09-21T10:43:04.479 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8816999999999997e-05 2024-09-21T10:43:04.479 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.479 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.480 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:43:04.480 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.628250+0000", 2024-09-21T10:43:04.480 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014617800000000001 2024-09-21T10:43:04.480 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.480 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.480 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:43:04.480 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.628412+0000", 2024-09-21T10:43:04.480 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000162187 2024-09-21T10:43:04.480 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.480 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.480 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:43:04.480 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.630470+0000", 2024-09-21T10:43:04.480 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0020574439999999999 2024-09-21T10:43:04.481 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.481 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.481 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:04.481 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.630758+0000", 2024-09-21T10:43:04.481 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000288254 2024-09-21T10:43:04.481 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.481 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.481 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.481 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.788815+0000", 2024-09-21T10:43:04.481 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.15805709200000001 2024-09-21T10:43:04.481 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.481 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.482 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:43:04.482 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.788835+0000", 2024-09-21T10:43:04.482 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9894e-05 2024-09-21T10:43:04.482 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.482 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:43:04.482 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.482 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.482 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.482 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:69148 53.13 53:c834d3e0:::benchmark_data_smithi070_228967_object69147:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1323)", 2024-09-21T10:43:04.482 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:40:59.631114+0000", 2024-09-21T10:43:04.482 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 124.840504866, 2024-09-21T10:43:04.482 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0407627319999999, 2024-09-21T10:43:04.482 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:43:04.483 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:43:04.483 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:43:04.483 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:43:04.483 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:04.483 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 69148 2024-09-21T10:43:04.483 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.483 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:43:04.483 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.483 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:43:04.483 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.631114+0000", 2024-09-21T10:43:04.483 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.483 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.483 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.484 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:43:04.484 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.631114+0000", 2024-09-21T10:43:04.484 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.484 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.484 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.484 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:43:04.484 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.631113+0000", 2024-09-21T10:43:04.484 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:43:04.484 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.484 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.484 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:43:04.484 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.631157+0000", 2024-09-21T10:43:04.484 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4168999999999997e-05 2024-09-21T10:43:04.484 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.484 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.485 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:43:04.485 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.631174+0000", 2024-09-21T10:43:04.485 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7478000000000001e-05 2024-09-21T10:43:04.485 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.485 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.485 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:43:04.485 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.631182+0000", 2024-09-21T10:43:04.485 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0010000000000001e-06 2024-09-21T10:43:04.485 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.485 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.485 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.485 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.789648+0000", 2024-09-21T10:43:04.485 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.15846608200000001 2024-09-21T10:43:04.485 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.485 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.486 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:43:04.486 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.789655+0000", 2024-09-21T10:43:04.486 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2809999999999997e-06 2024-09-21T10:43:04.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.486 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.486 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.486 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:00.669766+0000", 2024-09-21T10:43:04.486 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88011092800000001 2024-09-21T10:43:04.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.486 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.486 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:43:04.486 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:00.670343+0000", 2024-09-21T10:43:04.486 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00057693899999999995 2024-09-21T10:43:04.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.486 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.487 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.487 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:00.671828+0000", 2024-09-21T10:43:04.487 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0014857819999999999 2024-09-21T10:43:04.487 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.487 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.487 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:43:04.487 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:00.671877+0000", 2024-09-21T10:43:04.487 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8415000000000003e-05 2024-09-21T10:43:04.487 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.487 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:43:04.487 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.487 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.487 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.487 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:72193 53.f 53:fc063432:::benchmark_data_smithi070_228967_object72192:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1324)", 2024-09-21T10:43:04.487 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:02.563364+0000", 2024-09-21T10:43:04.488 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 121.90825527, 2024-09-21T10:43:04.488 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13326911799999999, 2024-09-21T10:43:04.488 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:43:04.488 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:04.488 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:43:04.488 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:43:04.488 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:04.488 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 72193 2024-09-21T10:43:04.488 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.488 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:43:04.488 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.488 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:43:04.488 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.563364+0000", 2024-09-21T10:43:04.488 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.488 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.489 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.489 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:43:04.489 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.563364+0000", 2024-09-21T10:43:04.489 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.489 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.489 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.489 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:43:04.489 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.563363+0000", 2024-09-21T10:43:04.489 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T10:43:04.490 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.490 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.490 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:43:04.490 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.563418+0000", 2024-09-21T10:43:04.490 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5495000000000003e-05 2024-09-21T10:43:04.490 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.490 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.490 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:43:04.490 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.563432+0000", 2024-09-21T10:43:04.490 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3713000000000001e-05 2024-09-21T10:43:04.491 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.491 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.491 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:43:04.491 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.563439+0000", 2024-09-21T10:43:04.491 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7379999999999997e-06 2024-09-21T10:43:04.491 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.491 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.491 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.491 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.566489+0000", 2024-09-21T10:43:04.491 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0030501270000000001 2024-09-21T10:43:04.491 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.491 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.491 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:43:04.492 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.566634+0000", 2024-09-21T10:43:04.492 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000144692 2024-09-21T10:43:04.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.492 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.492 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:43:04.492 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.566781+0000", 2024-09-21T10:43:04.492 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014745599999999999 2024-09-21T10:43:04.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.492 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.492 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:43:04.492 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.572208+0000", 2024-09-21T10:43:04.492 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0054267279999999996 2024-09-21T10:43:04.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.492 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.492 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:04.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.573385+0000", 2024-09-21T10:43:04.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001176617 2024-09-21T10:43:04.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:04.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.696541+0000", 2024-09-21T10:43:04.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.123156062 2024-09-21T10:43:04.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:43:04.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.696593+0000", 2024-09-21T10:43:04.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2070999999999997e-05 2024-09-21T10:43:04.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:43:04.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.696633+0000", 2024-09-21T10:43:04.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0045e-05 2024-09-21T10:43:04.494 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.494 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:43:04.494 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.494 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.494 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:72214 53.f 53:fcbb5044:::benchmark_data_smithi070_228967_object72213:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1324)", 2024-09-21T10:43:04.494 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:02.573981+0000", 2024-09-21T10:43:04.494 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 121.897638157, 2024-09-21T10:43:04.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13525400200000001, 2024-09-21T10:43:04.494 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:43:04.494 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:04.494 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:43:04.494 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:43:04.494 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:04.495 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 72214 2024-09-21T10:43:04.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.495 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:43:04.495 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.495 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:43:04.495 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.573981+0000", 2024-09-21T10:43:04.495 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.495 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.495 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:43:04.495 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.573981+0000", 2024-09-21T10:43:04.495 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.496 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:43:04.496 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.573980+0000", 2024-09-21T10:43:04.496 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:43:04.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.496 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:43:04.496 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.574050+0000", 2024-09-21T10:43:04.496 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0363000000000004e-05 2024-09-21T10:43:04.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.496 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:43:04.496 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.574061+0000", 2024-09-21T10:43:04.496 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0607e-05 2024-09-21T10:43:04.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.497 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.497 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:43:04.497 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.574068+0000", 2024-09-21T10:43:04.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9299999999999997e-06 2024-09-21T10:43:04.497 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.497 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.497 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.497 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.574114+0000", 2024-09-21T10:43:04.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6526000000000001e-05 2024-09-21T10:43:04.497 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.497 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.497 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:43:04.497 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.574236+0000", 2024-09-21T10:43:04.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000121866 2024-09-21T10:43:04.497 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.497 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.498 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:43:04.498 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.574397+0000", 2024-09-21T10:43:04.498 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016128099999999999 2024-09-21T10:43:04.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.498 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:43:04.498 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.576840+0000", 2024-09-21T10:43:04.498 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0024422530000000001 2024-09-21T10:43:04.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.498 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:04.498 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.577166+0000", 2024-09-21T10:43:04.498 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000325858 2024-09-21T10:43:04.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.499 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:04.499 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.709140+0000", 2024-09-21T10:43:04.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13197424399999999 2024-09-21T10:43:04.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.499 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:43:04.499 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.709192+0000", 2024-09-21T10:43:04.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1758999999999997e-05 2024-09-21T10:43:04.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.499 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:43:04.499 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.709235+0000", 2024-09-21T10:43:04.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3149999999999999e-05 2024-09-21T10:43:04.500 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.500 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:43:04.500 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.500 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.500 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.500 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:79863 53.12 53.de667272 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1329)", 2024-09-21T10:43:04.500 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:09.710219+0000", 2024-09-21T10:43:04.500 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 114.761399743, 2024-09-21T10:43:04.500 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98437714899999995, 2024-09-21T10:43:04.500 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:43:04.500 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:43:04.500 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:43:04.500 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:43:04.501 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:04.501 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 79863 2024-09-21T10:43:04.501 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.501 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:43:04.501 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.501 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:43:04.501 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.710219+0000", 2024-09-21T10:43:04.501 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.501 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.501 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.501 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:43:04.501 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.710219+0000", 2024-09-21T10:43:04.501 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.501 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.502 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.502 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:43:04.502 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.710216+0000", 2024-09-21T10:43:04.502 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999967 2024-09-21T10:43:04.502 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.502 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.502 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:43:04.502 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.710289+0000", 2024-09-21T10:43:04.502 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2967999999999997e-05 2024-09-21T10:43:04.502 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.502 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.502 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:43:04.502 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.710297+0000", 2024-09-21T10:43:04.502 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.875e-06 2024-09-21T10:43:04.502 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.502 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.503 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:43:04.503 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.710302+0000", 2024-09-21T10:43:04.503 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3449999999999996e-06 2024-09-21T10:43:04.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.503 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.503 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.503 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.710363+0000", 2024-09-21T10:43:04.503 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1414999999999994e-05 2024-09-21T10:43:04.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.503 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.503 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:43:04.503 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.710374+0000", 2024-09-21T10:43:04.503 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0164e-05 2024-09-21T10:43:04.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.503 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.503 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.504 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694583+0000", 2024-09-21T10:43:04.504 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98420943299999997 2024-09-21T10:43:04.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.504 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:43:04.504 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694596+0000", 2024-09-21T10:43:04.504 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3342e-05 2024-09-21T10:43:04.504 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.504 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:43:04.504 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.504 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:79867 53.12 53.8ded8132 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1329)", 2024-09-21T10:43:04.504 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:09.710968+0000", 2024-09-21T10:43:04.504 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 114.760650465, 2024-09-21T10:43:04.504 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98366795799999995, 2024-09-21T10:43:04.505 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:43:04.505 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:43:04.505 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:43:04.505 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:43:04.505 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:04.505 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 79867 2024-09-21T10:43:04.505 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.505 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:43:04.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.505 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:43:04.505 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.710968+0000", 2024-09-21T10:43:04.505 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.505 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.505 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:43:04.506 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.710968+0000", 2024-09-21T10:43:04.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.506 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:43:04.506 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.710967+0000", 2024-09-21T10:43:04.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:43:04.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.506 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:43:04.506 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.711004+0000", 2024-09-21T10:43:04.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6563999999999998e-05 2024-09-21T10:43:04.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.506 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:43:04.507 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.711017+0000", 2024-09-21T10:43:04.507 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3481e-05 2024-09-21T10:43:04.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.507 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.507 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:43:04.507 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.711025+0000", 2024-09-21T10:43:04.507 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8199999999999997e-06 2024-09-21T10:43:04.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.507 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.507 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.507 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.711065+0000", 2024-09-21T10:43:04.507 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9629000000000003e-05 2024-09-21T10:43:04.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.507 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.507 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:43:04.508 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.711072+0000", 2024-09-21T10:43:04.508 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6719999999999998e-06 2024-09-21T10:43:04.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.508 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.508 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.508 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694626+0000", 2024-09-21T10:43:04.508 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98355480200000001 2024-09-21T10:43:04.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.508 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.508 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:43:04.508 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694636+0000", 2024-09-21T10:43:04.508 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0057e-05 2024-09-21T10:43:04.508 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.508 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:43:04.508 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.509 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:79921 53.12 53.e42a4eb2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1329)", 2024-09-21T10:43:04.509 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:09.743427+0000", 2024-09-21T10:43:04.509 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 114.728191975, 2024-09-21T10:43:04.509 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.95123868700000003, 2024-09-21T10:43:04.509 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:43:04.509 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:43:04.509 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:43:04.509 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:43:04.509 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:04.509 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 79921 2024-09-21T10:43:04.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.509 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:43:04.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.509 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:43:04.510 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.743427+0000", 2024-09-21T10:43:04.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.510 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:43:04.510 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.743427+0000", 2024-09-21T10:43:04.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.510 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:43:04.510 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.743426+0000", 2024-09-21T10:43:04.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:43:04.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.511 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:43:04.511 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.743496+0000", 2024-09-21T10:43:04.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0148999999999995e-05 2024-09-21T10:43:04.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.511 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:43:04.511 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.743507+0000", 2024-09-21T10:43:04.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1277000000000001e-05 2024-09-21T10:43:04.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.511 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:43:04.511 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.743514+0000", 2024-09-21T10:43:04.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4259999999999997e-06 2024-09-21T10:43:04.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.511 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.743579+0000", 2024-09-21T10:43:04.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4304e-05 2024-09-21T10:43:04.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:43:04.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.743587+0000", 2024-09-21T10:43:04.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4109999999999996e-06 2024-09-21T10:43:04.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694656+0000", 2024-09-21T10:43:04.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.95106875000000002 2024-09-21T10:43:04.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:43:04.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694666+0000", 2024-09-21T10:43:04.513 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7079999999999992e-06 2024-09-21T10:43:04.513 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.513 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:43:04.513 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.513 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:79939 53.12 53.4ae2b0d2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1329)", 2024-09-21T10:43:04.513 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:09.754107+0000", 2024-09-21T10:43:04.513 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 114.717511801, 2024-09-21T10:43:04.513 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.94060030500000003, 2024-09-21T10:43:04.513 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:43:04.513 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:43:04.513 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:43:04.514 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:43:04.514 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:04.514 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 79939 2024-09-21T10:43:04.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.514 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:43:04.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.514 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:43:04.514 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.754107+0000", 2024-09-21T10:43:04.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.514 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:43:04.514 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.754107+0000", 2024-09-21T10:43:04.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.515 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:43:04.515 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.754106+0000", 2024-09-21T10:43:04.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:43:04.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.515 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:43:04.515 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.754154+0000", 2024-09-21T10:43:04.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8631999999999998e-05 2024-09-21T10:43:04.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.515 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:43:04.515 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.754168+0000", 2024-09-21T10:43:04.516 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3719e-05 2024-09-21T10:43:04.516 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.516 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.516 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:43:04.516 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.754177+0000", 2024-09-21T10:43:04.516 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0089999999999996e-06 2024-09-21T10:43:04.516 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.516 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.516 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.516 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:04.517 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.754224+0000", 2024-09-21T10:43:04.517 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6765999999999999e-05 2024-09-21T10:43:04.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:43:04.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.754234+0000", 2024-09-21T10:43:04.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8800000000000003e-06 2024-09-21T10:43:04.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694691+0000", 2024-09-21T10:43:04.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.94045756899999999 2024-09-21T10:43:04.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:43:04.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694707+0000", 2024-09-21T10:43:04.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6049000000000002e-05 2024-09-21T10:43:04.519 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.519 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:43:04.519 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.519 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:79977 53.12 53.a7d3712 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1329)", 2024-09-21T10:43:04.519 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:09.777895+0000", 2024-09-21T10:43:04.519 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 114.693724319, 2024-09-21T10:43:04.519 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91684465900000001, 2024-09-21T10:43:04.519 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:43:04.519 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:43:04.519 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:43:04.519 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:43:04.520 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:04.520 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 79977 2024-09-21T10:43:04.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.520 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:43:04.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:43:04.520 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.777895+0000", 2024-09-21T10:43:04.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:43:04.520 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.777895+0000", 2024-09-21T10:43:04.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:43:04.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.777893+0000", 2024-09-21T10:43:04.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:43:04.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:43:04.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.777997+0000", 2024-09-21T10:43:04.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000103127 2024-09-21T10:43:04.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:43:04.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.778006+0000", 2024-09-21T10:43:04.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6500000000000008e-06 2024-09-21T10:43:04.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.522 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:43:04.522 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.778014+0000", 2024-09-21T10:43:04.522 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8169999999999999e-06 2024-09-21T10:43:04.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.522 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.522 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.522 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.778049+0000", 2024-09-21T10:43:04.522 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4878e-05 2024-09-21T10:43:04.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.522 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.522 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:43:04.522 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.778054+0000", 2024-09-21T10:43:04.522 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1070000000000004e-06 2024-09-21T10:43:04.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.522 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.523 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.523 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694730+0000", 2024-09-21T10:43:04.523 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91667635199999997 2024-09-21T10:43:04.523 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.523 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.523 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:43:04.523 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694739+0000", 2024-09-21T10:43:04.523 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8859999999999993e-06 2024-09-21T10:43:04.523 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.523 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:43:04.523 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.523 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.523 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.524 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:80028 53.12 53.4f822472 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1329)", 2024-09-21T10:43:04.524 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:09.890679+0000", 2024-09-21T10:43:04.524 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 114.58093954, 2024-09-21T10:43:04.524 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.80408908300000004, 2024-09-21T10:43:04.524 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:43:04.524 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:43:04.524 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:43:04.524 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:43:04.524 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:04.524 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 80028 2024-09-21T10:43:04.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.524 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:43:04.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.524 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:43:04.524 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.890679+0000", 2024-09-21T10:43:04.524 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.525 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.525 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:43:04.525 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.890679+0000", 2024-09-21T10:43:04.525 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.525 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.525 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:43:04.525 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.890677+0000", 2024-09-21T10:43:04.525 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:43:04.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.525 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.525 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:43:04.525 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.890728+0000", 2024-09-21T10:43:04.526 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0900999999999999e-05 2024-09-21T10:43:04.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.526 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.526 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:43:04.526 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.890745+0000", 2024-09-21T10:43:04.526 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7186000000000001e-05 2024-09-21T10:43:04.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.526 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.526 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:43:04.526 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.890754+0000", 2024-09-21T10:43:04.526 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6750000000000008e-06 2024-09-21T10:43:04.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.526 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.526 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.526 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.890806+0000", 2024-09-21T10:43:04.527 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2346000000000003e-05 2024-09-21T10:43:04.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.527 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.527 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:43:04.527 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.890816+0000", 2024-09-21T10:43:04.527 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0042e-05 2024-09-21T10:43:04.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.527 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.527 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.527 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694761+0000", 2024-09-21T10:43:04.527 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.80394469700000004 2024-09-21T10:43:04.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.527 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.527 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:43:04.527 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694768+0000", 2024-09-21T10:43:04.527 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.323e-06 2024-09-21T10:43:04.528 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.528 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:43:04.528 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.528 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.528 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.528 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:80063 53.12 53.1f239a32 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1329)", 2024-09-21T10:43:04.528 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:09.910109+0000", 2024-09-21T10:43:04.528 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 114.56150969700001, 2024-09-21T10:43:04.528 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.78468356900000003, 2024-09-21T10:43:04.528 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:43:04.528 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:43:04.528 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:43:04.528 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:43:04.528 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:04.528 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 80063 2024-09-21T10:43:04.528 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.529 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:43:04.529 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.529 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:43:04.529 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.910109+0000", 2024-09-21T10:43:04.529 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.529 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.529 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:43:04.529 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.910109+0000", 2024-09-21T10:43:04.529 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.529 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.529 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:43:04.529 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.910108+0000", 2024-09-21T10:43:04.529 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:43:04.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.530 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.530 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:43:04.530 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.910156+0000", 2024-09-21T10:43:04.530 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8130999999999997e-05 2024-09-21T10:43:04.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.530 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.530 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:43:04.530 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.910168+0000", 2024-09-21T10:43:04.530 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1868000000000001e-05 2024-09-21T10:43:04.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.530 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.530 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:43:04.530 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.910175+0000", 2024-09-21T10:43:04.530 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6209999999999998e-06 2024-09-21T10:43:04.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.531 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.531 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.531 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.910228+0000", 2024-09-21T10:43:04.531 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3012e-05 2024-09-21T10:43:04.531 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.531 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.531 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:43:04.531 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.910236+0000", 2024-09-21T10:43:04.531 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4339999999999992e-06 2024-09-21T10:43:04.531 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.531 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.531 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.531 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694784+0000", 2024-09-21T10:43:04.531 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.78454796299999996 2024-09-21T10:43:04.532 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.532 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.532 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:43:04.532 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694793+0000", 2024-09-21T10:43:04.532 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8000000000000004e-06 2024-09-21T10:43:04.532 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.532 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:43:04.532 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.532 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.532 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.532 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:80071 53.12 53.9031e672 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1329)", 2024-09-21T10:43:04.532 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:09.915187+0000", 2024-09-21T10:43:04.532 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 114.556431765, 2024-09-21T10:43:04.532 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.77964120699999995, 2024-09-21T10:43:04.532 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:43:04.533 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:43:04.533 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:43:04.533 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:43:04.533 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:04.533 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 80071 2024-09-21T10:43:04.533 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.533 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:43:04.533 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.533 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:43:04.533 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.915187+0000", 2024-09-21T10:43:04.533 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.533 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.533 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.533 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:43:04.533 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.915187+0000", 2024-09-21T10:43:04.533 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.534 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.534 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:43:04.534 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.915186+0000", 2024-09-21T10:43:04.534 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:43:04.534 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.534 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:43:04.534 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.915296+0000", 2024-09-21T10:43:04.534 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000109795 2024-09-21T10:43:04.534 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.534 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:43:04.534 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.915307+0000", 2024-09-21T10:43:04.534 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0930999999999999e-05 2024-09-21T10:43:04.534 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.535 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.535 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:43:04.535 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.915312+0000", 2024-09-21T10:43:04.535 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5319999999999997e-06 2024-09-21T10:43:04.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.535 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.535 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.535 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.915358+0000", 2024-09-21T10:43:04.535 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5896000000000002e-05 2024-09-21T10:43:04.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.535 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.535 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:43:04.535 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.915366+0000", 2024-09-21T10:43:04.535 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3719999999999999e-06 2024-09-21T10:43:04.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.536 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.536 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.536 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694819+0000", 2024-09-21T10:43:04.536 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.77945362900000004 2024-09-21T10:43:04.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.536 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.536 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:43:04.536 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694828+0000", 2024-09-21T10:43:04.536 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1500000000000005e-06 2024-09-21T10:43:04.536 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.536 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:43:04.536 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.536 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.536 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:80090 53.12 53.a4fb4b92 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1329)", 2024-09-21T10:43:04.537 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:09.929351+0000", 2024-09-21T10:43:04.537 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 114.542267974, 2024-09-21T10:43:04.537 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.76552332300000003, 2024-09-21T10:43:04.537 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:43:04.537 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:43:04.537 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:43:04.537 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:43:04.537 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:04.537 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 80090 2024-09-21T10:43:04.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.537 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:43:04.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.537 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:43:04.537 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.929351+0000", 2024-09-21T10:43:04.537 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.538 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.538 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:43:04.538 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.929351+0000", 2024-09-21T10:43:04.538 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.538 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.538 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:43:04.538 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.929350+0000", 2024-09-21T10:43:04.538 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:43:04.538 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.538 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:43:04.538 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.929379+0000", 2024-09-21T10:43:04.538 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9161000000000001e-05 2024-09-21T10:43:04.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.539 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.539 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:43:04.539 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.929387+0000", 2024-09-21T10:43:04.539 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.362e-06 2024-09-21T10:43:04.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.539 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.539 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:43:04.539 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.929391+0000", 2024-09-21T10:43:04.539 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8969999999999999e-06 2024-09-21T10:43:04.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.539 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.539 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.540 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.929433+0000", 2024-09-21T10:43:04.540 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1794e-05 2024-09-21T10:43:04.540 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.540 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:43:04.540 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.929442+0000", 2024-09-21T10:43:04.540 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6989999999999992e-06 2024-09-21T10:43:04.540 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.540 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.540 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694863+0000", 2024-09-21T10:43:04.540 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.76542121100000005 2024-09-21T10:43:04.540 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.540 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:43:04.541 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694874+0000", 2024-09-21T10:43:04.541 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1048e-05 2024-09-21T10:43:04.541 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.541 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:43:04.541 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.541 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.541 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.541 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:80091 53.12 53.3618a532 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1329)", 2024-09-21T10:43:04.541 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:09.931720+0000", 2024-09-21T10:43:04.541 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 114.53989934499999, 2024-09-21T10:43:04.541 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.76317143399999998, 2024-09-21T10:43:04.541 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:43:04.541 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:43:04.541 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:43:04.541 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:43:04.541 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:04.542 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 80091 2024-09-21T10:43:04.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.542 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:43:04.542 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.542 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:43:04.542 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.931720+0000", 2024-09-21T10:43:04.542 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.542 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.542 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:43:04.542 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.931720+0000", 2024-09-21T10:43:04.542 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.542 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:43:04.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.931718+0000", 2024-09-21T10:43:04.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:43:04.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:43:04.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.931756+0000", 2024-09-21T10:43:04.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7830000000000002e-05 2024-09-21T10:43:04.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:43:04.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.931772+0000", 2024-09-21T10:43:04.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6115000000000001e-05 2024-09-21T10:43:04.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:43:04.544 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.931779+0000", 2024-09-21T10:43:04.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1500000000000002e-06 2024-09-21T10:43:04.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.544 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.931824+0000", 2024-09-21T10:43:04.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5215e-05 2024-09-21T10:43:04.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:43:04.544 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.931833+0000", 2024-09-21T10:43:04.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6920000000000008e-06 2024-09-21T10:43:04.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.546 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.546 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694886+0000", 2024-09-21T10:43:04.546 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.76305307600000005 2024-09-21T10:43:04.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.546 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:43:04.546 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694891+0000", 2024-09-21T10:43:04.546 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1189999999999996e-06 2024-09-21T10:43:04.546 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.546 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:43:04.546 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.547 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:80093 53.12 53.7d169af2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1329)", 2024-09-21T10:43:04.547 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:09.932541+0000", 2024-09-21T10:43:04.547 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 114.539077757, 2024-09-21T10:43:04.547 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.76237195899999999, 2024-09-21T10:43:04.547 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:43:04.547 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:43:04.547 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:43:04.547 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:43:04.547 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:04.547 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 80093 2024-09-21T10:43:04.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.547 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:43:04.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.548 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:43:04.548 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.932541+0000", 2024-09-21T10:43:04.548 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.548 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.548 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.548 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:43:04.548 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.932541+0000", 2024-09-21T10:43:04.548 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.548 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.548 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.548 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:43:04.548 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.932540+0000", 2024-09-21T10:43:04.548 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:43:04.548 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.549 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.549 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:43:04.549 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.932593+0000", 2024-09-21T10:43:04.549 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2790000000000001e-05 2024-09-21T10:43:04.549 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.549 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.549 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:43:04.549 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.932604+0000", 2024-09-21T10:43:04.549 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1042e-05 2024-09-21T10:43:04.549 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.549 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.549 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:43:04.549 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.932610+0000", 2024-09-21T10:43:04.550 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2659999999999998e-06 2024-09-21T10:43:04.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.550 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.550 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.932638+0000", 2024-09-21T10:43:04.550 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7761999999999999e-05 2024-09-21T10:43:04.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.550 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:43:04.550 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.932642+0000", 2024-09-21T10:43:04.550 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2889999999999999e-06 2024-09-21T10:43:04.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.550 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.551 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694905+0000", 2024-09-21T10:43:04.551 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.76226226500000005 2024-09-21T10:43:04.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.551 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.551 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:43:04.551 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694913+0000", 2024-09-21T10:43:04.551 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4230000000000005e-06 2024-09-21T10:43:04.551 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.551 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:43:04.551 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.551 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.551 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:80099 53.12 53.2b1c9a72 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1329)", 2024-09-21T10:43:04.552 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:09.936718+0000", 2024-09-21T10:43:04.552 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 114.534900556, 2024-09-21T10:43:04.552 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.75820982299999995, 2024-09-21T10:43:04.552 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:43:04.552 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:43:04.552 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:43:04.552 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:43:04.552 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:04.552 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 80099 2024-09-21T10:43:04.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.552 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:43:04.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.552 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:43:04.552 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.936718+0000", 2024-09-21T10:43:04.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.553 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:43:04.553 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.936718+0000", 2024-09-21T10:43:04.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.553 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:43:04.553 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.936717+0000", 2024-09-21T10:43:04.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:43:04.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.553 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:43:04.553 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.936809+0000", 2024-09-21T10:43:04.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1834000000000002e-05 2024-09-21T10:43:04.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:43:04.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.936826+0000", 2024-09-21T10:43:04.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7323e-05 2024-09-21T10:43:04.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:43:04.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.936833+0000", 2024-09-21T10:43:04.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4209999999999998e-06 2024-09-21T10:43:04.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.555 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.936885+0000", 2024-09-21T10:43:04.555 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1446000000000002e-05 2024-09-21T10:43:04.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.555 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.555 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:43:04.555 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.936893+0000", 2024-09-21T10:43:04.555 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8269999999999998e-06 2024-09-21T10:43:04.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.555 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.555 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.555 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694923+0000", 2024-09-21T10:43:04.555 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.75803034199999997 2024-09-21T10:43:04.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.555 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.556 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:43:04.556 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694928+0000", 2024-09-21T10:43:04.556 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2039999999999996e-06 2024-09-21T10:43:04.556 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.556 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:43:04.556 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.556 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.556 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.556 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:80122 53.12 53.d10cf0d2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1329)", 2024-09-21T10:43:04.556 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:09.956535+0000", 2024-09-21T10:43:04.556 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 114.515084007, 2024-09-21T10:43:04.556 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.73841511500000001, 2024-09-21T10:43:04.556 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:43:04.556 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:43:04.557 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:43:04.557 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:43:04.557 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:04.557 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 80122 2024-09-21T10:43:04.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.557 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:43:04.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.557 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:43:04.557 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.956535+0000", 2024-09-21T10:43:04.557 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.557 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:43:04.557 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.956535+0000", 2024-09-21T10:43:04.557 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.558 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:43:04.558 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.956533+0000", 2024-09-21T10:43:04.558 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:43:04.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.558 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:43:04.558 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.956583+0000", 2024-09-21T10:43:04.558 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9929999999999998e-05 2024-09-21T10:43:04.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.558 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:43:04.558 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.956596+0000", 2024-09-21T10:43:04.559 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2573e-05 2024-09-21T10:43:04.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.559 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.559 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:43:04.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.956603+0000", 2024-09-21T10:43:04.560 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6930000000000001e-06 2024-09-21T10:43:04.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.560 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.956655+0000", 2024-09-21T10:43:04.560 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2172000000000001e-05 2024-09-21T10:43:04.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.560 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:43:04.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.956664+0000", 2024-09-21T10:43:04.560 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3130000000000004e-06 2024-09-21T10:43:04.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.560 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.561 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694941+0000", 2024-09-21T10:43:04.561 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.73827710700000004 2024-09-21T10:43:04.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.561 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:43:04.561 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694950+0000", 2024-09-21T10:43:04.561 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7280000000000001e-06 2024-09-21T10:43:04.561 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.561 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:43:04.561 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.561 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:80155 53.12 53.71610e32 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1329)", 2024-09-21T10:43:04.561 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:09.991724+0000", 2024-09-21T10:43:04.562 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 114.479895361, 2024-09-21T10:43:04.562 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.70324484799999998, 2024-09-21T10:43:04.562 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:43:04.562 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:43:04.562 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:43:04.562 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:43:04.562 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:04.562 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 80155 2024-09-21T10:43:04.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.562 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:43:04.562 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.562 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:43:04.562 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.991724+0000", 2024-09-21T10:43:04.562 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.563 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.563 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:43:04.563 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.991724+0000", 2024-09-21T10:43:04.563 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.563 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.563 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.563 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:43:04.563 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.991723+0000", 2024-09-21T10:43:04.563 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T10:43:04.563 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.563 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.563 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:43:04.563 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.991752+0000", 2024-09-21T10:43:04.563 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8887999999999999e-05 2024-09-21T10:43:04.563 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.564 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.564 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:43:04.564 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.991759+0000", 2024-09-21T10:43:04.564 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4630000000000004e-06 2024-09-21T10:43:04.564 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.564 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.564 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:43:04.564 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.991763+0000", 2024-09-21T10:43:04.564 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8789999999999996e-06 2024-09-21T10:43:04.564 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.564 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.564 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.564 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.991802+0000", 2024-09-21T10:43:04.564 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8553000000000002e-05 2024-09-21T10:43:04.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.565 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.565 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:43:04.565 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.991808+0000", 2024-09-21T10:43:04.565 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.66e-06 2024-09-21T10:43:04.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.565 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.565 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.565 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694964+0000", 2024-09-21T10:43:04.565 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.70315639900000004 2024-09-21T10:43:04.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.565 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.565 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:43:04.565 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694968+0000", 2024-09-21T10:43:04.566 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.42e-06 2024-09-21T10:43:04.566 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.566 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:43:04.566 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.566 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.566 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.566 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:80160 53.12 53.b7431a92 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1329)", 2024-09-21T10:43:04.566 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:09.998347+0000", 2024-09-21T10:43:04.566 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 114.473272345, 2024-09-21T10:43:04.566 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.69664577999999999, 2024-09-21T10:43:04.566 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:43:04.566 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:43:04.566 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:43:04.566 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:43:04.567 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:04.567 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 80160 2024-09-21T10:43:04.567 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.567 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:43:04.567 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.567 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:43:04.567 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.998347+0000", 2024-09-21T10:43:04.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.567 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.567 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.567 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:43:04.567 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.998347+0000", 2024-09-21T10:43:04.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.567 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.568 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.568 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:43:04.568 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.998345+0000", 2024-09-21T10:43:04.568 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:43:04.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.568 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.568 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:43:04.568 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.998412+0000", 2024-09-21T10:43:04.568 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6142000000000005e-05 2024-09-21T10:43:04.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.568 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.568 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:43:04.568 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.998422+0000", 2024-09-21T10:43:04.568 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0902e-05 2024-09-21T10:43:04.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.568 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.569 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:43:04.569 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.998428+0000", 2024-09-21T10:43:04.569 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5979999999999996e-06 2024-09-21T10:43:04.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.569 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.569 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.998467+0000", 2024-09-21T10:43:04.569 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8856000000000002e-05 2024-09-21T10:43:04.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.569 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:43:04.569 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.998475+0000", 2024-09-21T10:43:04.569 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5690000000000005e-06 2024-09-21T10:43:04.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.569 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.569 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694984+0000", 2024-09-21T10:43:04.570 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.69650833499999998 2024-09-21T10:43:04.570 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.570 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.570 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:43:04.570 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694992+0000", 2024-09-21T10:43:04.570 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5150000000000006e-06 2024-09-21T10:43:04.570 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.570 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:43:04.570 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.570 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.570 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.570 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:80214 53.12 53.3a53c112 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1329)", 2024-09-21T10:43:04.570 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:10.130986+0000", 2024-09-21T10:43:04.571 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 114.340632794, 2024-09-21T10:43:04.571 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.56402021599999996, 2024-09-21T10:43:04.571 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:43:04.571 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:43:04.571 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:43:04.571 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:43:04.571 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:04.571 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 80214 2024-09-21T10:43:04.571 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.571 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:43:04.571 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.571 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:43:04.571 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.130986+0000", 2024-09-21T10:43:04.571 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.571 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.572 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.572 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:43:04.572 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.130986+0000", 2024-09-21T10:43:04.572 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:43:04.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.572 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.572 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:43:04.572 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.130985+0000", 2024-09-21T10:43:04.572 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:43:04.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.572 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.572 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:43:04.572 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.131031+0000", 2024-09-21T10:43:04.572 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5708999999999999e-05 2024-09-21T10:43:04.573 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.573 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.573 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:43:04.573 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.131041+0000", 2024-09-21T10:43:04.573 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0875e-05 2024-09-21T10:43:04.573 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.573 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.573 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:43:04.573 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.131046+0000", 2024-09-21T10:43:04.573 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9910000000000002e-06 2024-09-21T10:43:04.573 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.573 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.573 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.573 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.131086+0000", 2024-09-21T10:43:04.573 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9369999999999997e-05 2024-09-21T10:43:04.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.574 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.574 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:43:04.574 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.131092+0000", 2024-09-21T10:43:04.574 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8819999999999998e-06 2024-09-21T10:43:04.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.574 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.574 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:43:04.574 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.695002+0000", 2024-09-21T10:43:04.574 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.56391046899999997 2024-09-21T10:43:04.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:43:04.574 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:43:04.574 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:43:04.574 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.695006+0000", 2024-09-21T10:43:04.574 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1420000000000003e-06 2024-09-21T10:43:04.574 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.575 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:43:04.575 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.575 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:43:04.575 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:43:04.575 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:43:04.593 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:04.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:04.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:04.677 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:04 smithi070 ceph-mon[107385]: pgmap v2272: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 3.9 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 820 op/s; 1928/191751 objects degraded (1.005%); 386/191751 objects misplaced (0.201%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:43:04.677 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:04 smithi070 ceph-mon[112139]: pgmap v2272: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 3.9 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 820 op/s; 1928/191751 objects degraded (1.005%); 386/191751 objects misplaced (0.201%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:43:04.695 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:04.722 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:04.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:04.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:04.823 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:04.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:04 smithi160 ceph-mon[101879]: pgmap v2272: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 3.9 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 820 op/s; 1928/191751 objects degraded (1.005%); 386/191751 objects misplaced (0.201%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:43:04.853 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:04.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:04.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:04.955 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:04.985 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:04.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:04.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:05.086 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:05.127 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:05.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:05.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:05.228 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:05.265 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:05.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:05.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:05.363 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:43:05.367 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:05.405 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:05.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:05.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:05.507 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:05.563 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:05.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:05.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:05.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:05 smithi160 ceph-mon[101879]: pgmap v2273: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 3.8 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 816 op/s; 1928/186054 objects degraded (1.036%); 145/186054 objects misplaced (0.078%); 4.4 MiB/s, 70 objects/s recovering 2024-09-21T10:43:05.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:43:05.664 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:05.669 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:43:05.704 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:05.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:05.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:05.806 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:05.820 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:05 smithi070 ceph-mon[107385]: pgmap v2273: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 3.8 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 816 op/s; 1928/186054 objects degraded (1.036%); 145/186054 objects misplaced (0.078%); 4.4 MiB/s, 70 objects/s recovering 2024-09-21T10:43:05.820 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:43:05.820 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:05 smithi070 ceph-mon[112139]: pgmap v2273: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 3.8 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 816 op/s; 1928/186054 objects degraded (1.036%); 145/186054 objects misplaced (0.078%); 4.4 MiB/s, 70 objects/s recovering 2024-09-21T10:43:05.820 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:43:05.845 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:05.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:05.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:05.947 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:05.975 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:05.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:05.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:06.076 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:06.096 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:43:06.104 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:06.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:06.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:06.205 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:06.231 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:06.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:06.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:06.332 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:06.378 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:06.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:06.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:06.479 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:06.512 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:06.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:06.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:06.614 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:06.647 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:06.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:06.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:06.749 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:06.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:06.788 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:06.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:06.889 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:06.914 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:06.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:06.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:06.955 INFO:teuthology.orchestra.run.smithi070.stderr:no valid command found; 10 closest matches: 2024-09-21T10:43:06.955 INFO:teuthology.orchestra.run.smithi070.stderr:0 2024-09-21T10:43:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:1 2024-09-21T10:43:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:2 2024-09-21T10:43:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:abort 2024-09-21T10:43:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:assert 2024-09-21T10:43:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:bench [] [] [] [] 2024-09-21T10:43:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:43:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs files list 2024-09-21T10:43:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs stats 2024-09-21T10:43:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:bluestore allocator dump block 2024-09-21T10:43:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:Error EINVAL: invalid command 2024-09-21T10:43:07.015 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:07.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:07.049 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:07.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:07.150 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:07.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:07.184 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:07.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:07.285 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:07.313 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:07.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:07.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:07.414 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:07.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:07.450 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:07.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:07.551 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:07.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:07.579 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:07.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:07.680 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:07.705 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:07.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:07.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:07.796 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T10:43:07.806 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:07.845 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:07.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:07.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:07.947 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:07.979 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:07.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:07.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:08.033 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:07 smithi160 ceph-mon[101879]: pgmap v2274: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 3.6 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 1.10k op/s; 1928/175050 objects degraded (1.101%); 145/175050 objects misplaced (0.083%); 3.8 MiB/s, 61 objects/s recovering 2024-09-21T10:43:08.033 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:07 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3985587100' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:08.033 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:07 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:08.080 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:08.089 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:43:08.089 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T10:43:08.089 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T10:43:08.089 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:43:08.111 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:08.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:08.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:08.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:07 smithi070 ceph-mon[107385]: pgmap v2274: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 3.6 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 1.10k op/s; 1928/175050 objects degraded (1.101%); 145/175050 objects misplaced (0.083%); 3.8 MiB/s, 61 objects/s recovering 2024-09-21T10:43:08.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:07 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3985587100' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:08.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:07 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:08.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:07 smithi070 ceph-mon[112139]: pgmap v2274: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 3.6 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 1.10k op/s; 1928/175050 objects degraded (1.101%); 145/175050 objects misplaced (0.083%); 3.8 MiB/s, 61 objects/s recovering 2024-09-21T10:43:08.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:07 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3985587100' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:08.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:07 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:08.213 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:08.243 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:08.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:08.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:08.345 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:08.379 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:08.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:08.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:08.481 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:08.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:08.516 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:08.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:08.616 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:08.641 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:08.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:08.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:08.743 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:08.780 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:08.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:08.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:08.813 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:43:08.882 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:08.922 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:08.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:08.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:09.024 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:09.063 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:09.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:09.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:09.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:08 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:43:09.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:08 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:43:09.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:08 smithi160 ceph-mon[101879]: osdmap e1348: 8 total, 8 up, 6 in 2024-09-21T10:43:09.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:08 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 1928/175050 objects degraded (1.101%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:43:09.114 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:43:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:08 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:43:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:08 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:43:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:08 smithi070 ceph-mon[107385]: osdmap e1348: 8 total, 8 up, 6 in 2024-09-21T10:43:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:08 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 1928/175050 objects degraded (1.101%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:43:09.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:08 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:43:09.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:08 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:43:09.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:08 smithi070 ceph-mon[112139]: osdmap e1348: 8 total, 8 up, 6 in 2024-09-21T10:43:09.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:08 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 1928/175050 objects degraded (1.101%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:43:09.165 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:09.199 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:09.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:09.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:09.300 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:09.342 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:09.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:09.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:09.443 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:09.475 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:09.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:09.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:09.576 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:09.605 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:09.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:09.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:09.707 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:09.736 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:09.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:09.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:09.753 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:09.761 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:43:09.761 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:43:09.762 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.5 'trim stale osdmaps' 2024-09-21T10:43:09.838 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:09.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:09.883 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:09.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:09.984 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:10.013 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:10.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:10.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:10.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:09 smithi160 ceph-mon[101879]: pgmap v2276: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 3.6 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 1.02k op/s; 1928/175050 objects degraded (1.101%); 145/175050 objects misplaced (0.083%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:43:10.115 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:10.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:09 smithi070 ceph-mon[107385]: pgmap v2276: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 3.6 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 1.02k op/s; 1928/175050 objects degraded (1.101%); 145/175050 objects misplaced (0.083%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:43:10.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:09 smithi070 ceph-mon[112139]: pgmap v2276: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 3.6 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 1.02k op/s; 1928/175050 objects degraded (1.101%); 145/175050 objects misplaced (0.083%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:43:10.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:10.164 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:10.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:10.265 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:10.293 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:10.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:10.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:10.394 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:10.427 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:10.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:10.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:10.529 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:10.568 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:10.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:10.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:10.670 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:10.706 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:10.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:10.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:10.808 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:10.840 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:10.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:10.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:10.941 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:10.975 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:10.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:10.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:11.077 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:11.104 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:43:11.104 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T10:43:11.104 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T10:43:11.104 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T10:43:11.104 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:43:11.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:11.114 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:11.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:11.215 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:11.242 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:11.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:11.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:11.344 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:11.376 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:11.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:11.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:11.477 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:11.515 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:11.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:11.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:11.616 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:11.648 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:11.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:11.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:11.750 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:11.782 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:11.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:11.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:11.801 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:43:11.884 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:11.918 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:11.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:11.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:12.021 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:12.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:12.055 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:12.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:12.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:11 smithi160 ceph-mon[101879]: pgmap v2277: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 3.4 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 1806/166101 objects degraded (1.087%); 3.2 MiB/s, 62 objects/s recovering 2024-09-21T10:43:12.113 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:43:12.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:11 smithi070 ceph-mon[112139]: pgmap v2277: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 3.4 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 1806/166101 objects degraded (1.087%); 3.2 MiB/s, 62 objects/s recovering 2024-09-21T10:43:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:11 smithi070 ceph-mon[107385]: pgmap v2277: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 3.4 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 1806/166101 objects degraded (1.087%); 3.2 MiB/s, 62 objects/s recovering 2024-09-21T10:43:12.156 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:12.189 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:12.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:12.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:12.283 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:43:12.290 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:12.316 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:12.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:12.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:12.418 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:12.447 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:12.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:12.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:12.464 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:12.549 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:12.598 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:12.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:12.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:12.699 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:12.740 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:12.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:12.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:12.842 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:12.881 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:12.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:12.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:12.983 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:13.011 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:13.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:13.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:13.113 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:13.147 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:13.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:13.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:13.249 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:13.270 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:43:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:43:13] "GET /metrics HTTP/1.1" 200 36625 "" "Prometheus/2.51.0" 2024-09-21T10:43:13.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:13.350 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:13.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:13.353 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:43:13.353 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:43:13.392 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:43:13.392 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:43:13.428 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:43:13.429 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:43:13.450 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:13.467 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:43:13.467 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:43:13.486 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:13.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:13.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:13.504 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:43:13.505 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:43:13.538 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:43:13.538 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:43:13.572 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:43:13.572 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:43:13.588 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:13.602 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:43:13.603 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:43:13.620 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:13.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:13.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:13.722 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:13.760 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:13.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:13.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:13.861 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:13.893 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:13.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:13.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:13.929 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:13.995 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:14.031 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:14.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:14.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:14.062 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:13 smithi070 ceph-mon[107385]: pgmap v2278: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 3.2 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 1.10k op/s; 1806/158754 objects degraded (1.138%); 2.4 MiB/s, 50 objects/s recovering 2024-09-21T10:43:14.062 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:13 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 1806/158754 objects degraded (1.138%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:43:14.063 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:13 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1032591038' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:43:14.063 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:13 smithi070 ceph-mon[112139]: pgmap v2278: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 3.2 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 1.10k op/s; 1806/158754 objects degraded (1.138%); 2.4 MiB/s, 50 objects/s recovering 2024-09-21T10:43:14.063 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:13 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 1806/158754 objects degraded (1.138%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:43:14.063 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:13 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1032591038' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:43:14.083 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:13 smithi160 ceph-mon[101879]: pgmap v2278: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 3.2 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 1.10k op/s; 1806/158754 objects degraded (1.138%); 2.4 MiB/s, 50 objects/s recovering 2024-09-21T10:43:14.083 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:13 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 1806/158754 objects degraded (1.138%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:43:14.083 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:13 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1032591038' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:43:14.133 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:14.170 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:14.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:14.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:14.271 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:14.311 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:14.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:14.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:14.412 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:14.447 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:14.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:14.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:14.549 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:14.590 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:14.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:14.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:14.692 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:14.728 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:14.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:14.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:14.759 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:43:14.759 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:43:14.829 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:14.863 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:14.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:14.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:14.965 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:15.002 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:15.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:15.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:15.104 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:15.132 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:15.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:15.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:15.233 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:15.269 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:15.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:15.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:15.371 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:15.404 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:15.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:15.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:15.479 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:43:15.506 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:15.530 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:15.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:15.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:15.632 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:15.683 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:15.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:15.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:15.701 INFO:teuthology.orchestra.run.smithi070.stderr:no valid command found; 10 closest matches: 2024-09-21T10:43:15.701 INFO:teuthology.orchestra.run.smithi070.stderr:0 2024-09-21T10:43:15.701 INFO:teuthology.orchestra.run.smithi070.stderr:1 2024-09-21T10:43:15.701 INFO:teuthology.orchestra.run.smithi070.stderr:2 2024-09-21T10:43:15.701 INFO:teuthology.orchestra.run.smithi070.stderr:abort 2024-09-21T10:43:15.701 INFO:teuthology.orchestra.run.smithi070.stderr:assert 2024-09-21T10:43:15.701 INFO:teuthology.orchestra.run.smithi070.stderr:bench [] [] [] [] 2024-09-21T10:43:15.701 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:43:15.701 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs files list 2024-09-21T10:43:15.701 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs stats 2024-09-21T10:43:15.701 INFO:teuthology.orchestra.run.smithi070.stderr:bluestore allocator dump block 2024-09-21T10:43:15.701 INFO:teuthology.orchestra.run.smithi070.stderr:Error EINVAL: invalid command 2024-09-21T10:43:15.785 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:15.822 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:15.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:15.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:15.924 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:15.953 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:15.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:15.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:16.055 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:16.083 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:16.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:16.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:16.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:15 smithi160 ceph-mon[101879]: pgmap v2279: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 3.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 1.09k op/s; 1532/153270 objects degraded (1.000%); 928 KiB/s, 54 objects/s recovering 2024-09-21T10:43:16.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:15 smithi070 ceph-mon[107385]: pgmap v2279: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 3.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 1.09k op/s; 1532/153270 objects degraded (1.000%); 928 KiB/s, 54 objects/s recovering 2024-09-21T10:43:16.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:15 smithi070 ceph-mon[112139]: pgmap v2279: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 3.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 1.09k op/s; 1532/153270 objects degraded (1.000%); 928 KiB/s, 54 objects/s recovering 2024-09-21T10:43:16.185 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:16.216 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:16.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:16.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:16.317 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:16.342 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:16.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:16.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:16.444 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:16.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:16.485 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:16.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:16.586 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:16.615 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:16.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:16.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:16.717 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:16.743 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:16.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:16.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:16.844 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:16.869 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:16.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:16.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:16.970 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:17.008 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:17.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:17.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:17.110 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:17.135 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:17.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:17.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:17.237 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:17.266 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:17.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:17.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:17.368 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:17.405 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:17.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:17.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:17.507 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:17.546 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:17.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:17.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:17.647 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:17.690 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:17.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:17.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:17.792 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:17.824 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:17.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:17.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:17.926 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:17.956 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:17.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:17.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:18.058 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:18.078 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:18.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:18.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:18.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:17 smithi160 ceph-mon[101879]: pgmap v2280: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 2.9 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 1532/142764 objects degraded (1.073%); 928 KiB/s, 54 objects/s recovering 2024-09-21T10:43:18.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:17 smithi070 ceph-mon[107385]: pgmap v2280: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 2.9 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 1532/142764 objects degraded (1.073%); 928 KiB/s, 54 objects/s recovering 2024-09-21T10:43:18.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:17 smithi070 ceph-mon[112139]: pgmap v2280: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 2.9 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 1532/142764 objects degraded (1.073%); 928 KiB/s, 54 objects/s recovering 2024-09-21T10:43:18.181 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:18.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:18.219 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:18.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:18.320 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:18.350 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:18.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:18.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:18.451 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:18.478 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:18.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:18.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:18.580 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:18.608 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:18.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:18.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:18.709 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:18.730 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:18.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:18.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:18.831 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:18.858 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:18.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:18.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:18.961 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:18.981 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:18.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:18.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:19.083 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:19.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:18 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 1532/142764 objects degraded (1.073%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:43:19.111 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:19.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:19.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:19.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:18 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 1532/142764 objects degraded (1.073%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:43:19.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:18 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 1532/142764 objects degraded (1.073%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:43:19.213 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:19.233 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:19.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:19.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:19.334 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:19.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:19.370 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:19.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:19.471 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:19.503 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:19.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:19.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:19.604 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:19.650 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:19.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:19.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:19.751 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:19.787 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:19.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:19.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:19.888 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:19.920 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:19.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:19.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:20.022 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:20.060 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:20.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:20.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:20.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:19 smithi160 ceph-mon[101879]: pgmap v2281: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 2.9 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 983 op/s; 1532/142764 objects degraded (1.073%); 848 KiB/s, 49 objects/s recovering 2024-09-21T10:43:20.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:19 smithi070 ceph-mon[112139]: pgmap v2281: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 2.9 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 983 op/s; 1532/142764 objects degraded (1.073%); 848 KiB/s, 49 objects/s recovering 2024-09-21T10:43:20.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:19 smithi070 ceph-mon[107385]: pgmap v2281: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 2.9 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 983 op/s; 1532/142764 objects degraded (1.073%); 848 KiB/s, 49 objects/s recovering 2024-09-21T10:43:20.163 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:20.184 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:20.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:20.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:20.286 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:20.320 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:20.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:20.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:20.421 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:20.447 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:20.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:20.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:20.548 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:20.579 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:20.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:20.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:20.681 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:20.707 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:20.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:20.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:20.809 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:20.836 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:20.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:20.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:20.938 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:20.971 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:20.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:20.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:21.073 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:21.114 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:21.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:21.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:21.216 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:21.246 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:21.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:21.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:21.347 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:21.373 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:21.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:21.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:21.475 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:21.509 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:21.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:21.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:21.610 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:21.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:43:21.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:43:21.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:43:21.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:43:21.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:21.648 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:21.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:21.748 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:21.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:21.781 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:21.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:43:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:43:21.882 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:21.918 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:21.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:21.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:22.020 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:22.059 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:22.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:22.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:22.161 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:22.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:22.200 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:22.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:22.301 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:22.331 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:22.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:22.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:22.433 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:22.461 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:22.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:22.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:22.563 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:22.596 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:22.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:22.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:22 smithi070 ceph-mon[107385]: pgmap v2282: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 2.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 1.15k op/s; 1256/133671 objects degraded (0.940%); 773 KiB/s, 68 objects/s recovering 2024-09-21T10:43:22.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:22 smithi070 ceph-mon[112139]: pgmap v2282: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 2.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 1.15k op/s; 1256/133671 objects degraded (0.940%); 773 KiB/s, 68 objects/s recovering 2024-09-21T10:43:22.697 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:22.733 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:22.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:22.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:22.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:22 smithi160 ceph-mon[101879]: pgmap v2282: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 2.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 1.15k op/s; 1256/133671 objects degraded (0.940%); 773 KiB/s, 68 objects/s recovering 2024-09-21T10:43:22.834 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:22.854 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:43:22.855 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:43:22.855 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-21T10:43:22.860 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:22.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:22.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:22.918 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:22.961 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:23.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:23.025 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:23.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:23.126 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:23.168 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:23.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:23.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:23.269 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:23.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:23.298 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:23.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:23.389 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:43:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:43:23] "GET /metrics HTTP/1.1" 200 36624 "" "Prometheus/2.51.0" 2024-09-21T10:43:23.389 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:23 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 1256/133671 objects degraded (0.940%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:43:23.399 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:23.433 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:23.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:23.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:23.487 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:23 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 1256/133671 objects degraded (0.940%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:43:23.535 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:23.569 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:23.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:23.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:23.671 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:23.699 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:23 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 1256/133671 objects degraded (0.940%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:43:23.710 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:23.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:23.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:23.811 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:23.856 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:23.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:23.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:23.957 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:23.986 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:23.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:23.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:24.088 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:24.128 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:24.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:24.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:24.229 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:24.269 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:24.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:24.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:24.371 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:43:24.371 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:24.413 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:24.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:24.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:24.420 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T10:43:24.517 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:24.565 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:24 smithi070 ceph-mon[107385]: pgmap v2283: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 2.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 1.09k op/s; 1256/126834 objects degraded (0.990%); 0 B/s, 45 objects/s recovering 2024-09-21T10:43:24.565 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:24 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/305617738' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:43:24.566 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:24 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:43:24.566 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:24 smithi070 ceph-mon[112139]: pgmap v2283: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 2.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 1.09k op/s; 1256/126834 objects degraded (0.990%); 0 B/s, 45 objects/s recovering 2024-09-21T10:43:24.566 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:24 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/305617738' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:43:24.566 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:24 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:43:24.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:24.573 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:24.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:24.674 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:24.714 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:24.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:24.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:24.816 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:24 smithi160 ceph-mon[101879]: pgmap v2283: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 2.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 1.09k op/s; 1256/126834 objects degraded (0.990%); 0 B/s, 45 objects/s recovering 2024-09-21T10:43:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:24 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/305617738' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:43:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:24 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:43:24.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:24.848 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:24.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:24.949 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:24.983 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:24.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:24.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:25.085 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:25.113 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:25.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:25.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:25.183 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:25.215 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:25.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:25.270 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:25.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:25.371 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:25.406 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:25.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:25.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:25.507 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:25.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:25.543 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:25.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:25.644 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:25.693 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:25 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:43:25.693 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:25 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:43:25.693 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:25 smithi070 ceph-mon[107385]: osdmap e1349: 8 total, 8 up, 6 in 2024-09-21T10:43:25.694 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:25 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:43:25.694 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:25 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:43:25.694 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:25 smithi070 ceph-mon[112139]: osdmap e1349: 8 total, 8 up, 6 in 2024-09-21T10:43:25.706 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:25.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:25.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:25.807 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:25 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:43:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:25 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:43:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:25 smithi160 ceph-mon[101879]: osdmap e1349: 8 total, 8 up, 6 in 2024-09-21T10:43:25.847 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:25.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:25.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:25.948 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:25.988 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:25.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:25.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:26.090 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:26.117 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:26.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:26.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:26.219 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:26.256 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:26.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:26.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:26.358 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:26.393 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:26.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:26.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:26.402 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:26.444 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:43:26.444 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:43:26.495 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:26.539 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:26.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:26.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:26.593 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:26 smithi070 ceph-mon[107385]: pgmap v2285: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 2.5 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 1.06k op/s; 969/121287 objects degraded (0.799%); 0 B/s, 56 objects/s recovering 2024-09-21T10:43:26.593 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:26 smithi070 ceph-mon[112139]: pgmap v2285: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 2.5 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 1.06k op/s; 969/121287 objects degraded (0.799%); 0 B/s, 56 objects/s recovering 2024-09-21T10:43:26.641 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:26.675 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:26.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:26.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:26.776 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:26.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:26.810 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:26.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:26 smithi160 ceph-mon[101879]: pgmap v2285: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 2.5 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 1.06k op/s; 969/121287 objects degraded (0.799%); 0 B/s, 56 objects/s recovering 2024-09-21T10:43:26.911 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:26.950 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:26.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:26.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:27.051 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:27.086 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:27.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:27.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:27.189 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:27.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:27.222 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:27.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:27.323 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:27.361 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:27.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:27.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:27.463 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:27.490 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:27.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:27.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:27.592 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:27.636 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:27.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:27.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:27.695 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:43:27.738 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:27.783 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:27.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:27.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:27.885 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:27.912 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:27.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:27.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:28.014 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:28.043 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:28.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:28.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:28.144 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:28.187 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:28.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:28.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:28.198 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:43:28.289 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:28.317 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:28.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:28.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:28.418 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:28.440 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:28.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:28.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:28.542 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:28.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:28.576 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:28.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:28.677 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:28.723 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:28 smithi070 ceph-mon[107385]: pgmap v2286: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 2.3 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 1.05k op/s; 969/111183 objects degraded (0.872%); 0 B/s, 56 objects/s recovering 2024-09-21T10:43:28.723 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:28 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 969/111183 objects degraded (0.872%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:43:28.724 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:28 smithi070 ceph-mon[112139]: pgmap v2286: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 2.3 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 1.05k op/s; 969/111183 objects degraded (0.872%); 0 B/s, 56 objects/s recovering 2024-09-21T10:43:28.724 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:28 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 969/111183 objects degraded (0.872%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:43:28.728 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:28.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:28.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:28.829 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:28 smithi160 ceph-mon[101879]: pgmap v2286: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 2.3 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 1.05k op/s; 969/111183 objects degraded (0.872%); 0 B/s, 56 objects/s recovering 2024-09-21T10:43:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:28 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 969/111183 objects degraded (0.872%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:43:28.852 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:28.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:28.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:28.953 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:28.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:28.998 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:28.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:29.099 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:29.142 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:29.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:29.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:29.229 INFO:teuthology.orchestra.run.smithi070.stderr:no valid command found; 10 closest matches: 2024-09-21T10:43:29.229 INFO:teuthology.orchestra.run.smithi070.stderr:0 2024-09-21T10:43:29.229 INFO:teuthology.orchestra.run.smithi070.stderr:1 2024-09-21T10:43:29.229 INFO:teuthology.orchestra.run.smithi070.stderr:2 2024-09-21T10:43:29.229 INFO:teuthology.orchestra.run.smithi070.stderr:abort 2024-09-21T10:43:29.229 INFO:teuthology.orchestra.run.smithi070.stderr:assert 2024-09-21T10:43:29.229 INFO:teuthology.orchestra.run.smithi070.stderr:bench [] [] [] [] 2024-09-21T10:43:29.229 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:43:29.229 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs files list 2024-09-21T10:43:29.229 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs stats 2024-09-21T10:43:29.229 INFO:teuthology.orchestra.run.smithi070.stderr:bluestore allocator dump block 2024-09-21T10:43:29.230 INFO:teuthology.orchestra.run.smithi070.stderr:Error EINVAL: invalid command 2024-09-21T10:43:29.244 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:29.269 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:29.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:29.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:29.370 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:29.408 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:29.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:29.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:29.509 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:29.548 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:29.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:29.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:29.649 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:29.680 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:29.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:29.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:29.781 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:29.817 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:29.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:29.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:29.919 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:29.953 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:29.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:29.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:29.989 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:29.999 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:30.055 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:30.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:30.087 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:30.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:30.188 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:30.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:30.229 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:30.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:30.330 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:30.367 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:30.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:30.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:30.468 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:30.504 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:30.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:30.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:30.605 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:30.639 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:30.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:30.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:30.740 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:30.779 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:30.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:30.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:30 smithi160 ceph-mon[101879]: pgmap v2287: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 2.3 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 1.05k op/s; 969/111183 objects degraded (0.872%); 0 B/s, 56 objects/s recovering 2024-09-21T10:43:30.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:30 smithi070 ceph-mon[107385]: pgmap v2287: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 2.3 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 1.05k op/s; 969/111183 objects degraded (0.872%); 0 B/s, 56 objects/s recovering 2024-09-21T10:43:30.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:30 smithi070 ceph-mon[112139]: pgmap v2287: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 2.3 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 1.05k op/s; 969/111183 objects degraded (0.872%); 0 B/s, 56 objects/s recovering 2024-09-21T10:43:30.880 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:30.917 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:30.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:30.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:31.019 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:31.052 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:31.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:31.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:31.153 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:31.191 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:31.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:31.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:31.293 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:31.332 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:31.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:31.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:31.433 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:31.464 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:31.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:31.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:31.566 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:31.601 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:31.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:31.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:31.663 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:43:31.663 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:43:31.703 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:31.731 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:31.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:31.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:31.833 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:31.864 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:31.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:31.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:31.966 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:32.003 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:32.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:32.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:32.104 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:32.147 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:32.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:32.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:32.249 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:32.281 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:32.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:32.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:32.382 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:32.410 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:32.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:32.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:32.512 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:32.547 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:32.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:32.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:32.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:32 smithi070 ceph-mon[107385]: pgmap v2288: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 2.1 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 1.06k op/s; 645/101817 objects degraded (0.633%); 0 B/s, 61 objects/s recovering 2024-09-21T10:43:32.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:32 smithi070 ceph-mon[112139]: pgmap v2288: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 2.1 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 1.06k op/s; 645/101817 objects degraded (0.633%); 0 B/s, 61 objects/s recovering 2024-09-21T10:43:32.651 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:32.684 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:32.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:32.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:32.786 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:32.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:32.819 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:32.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:32 smithi160 ceph-mon[101879]: pgmap v2288: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 2.1 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 1.06k op/s; 645/101817 objects degraded (0.633%); 0 B/s, 61 objects/s recovering 2024-09-21T10:43:32.919 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:32.961 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:32.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:32.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:33.063 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:33.106 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:33.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:33.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:33.207 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:33.223 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:43:33.223 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:43:33.235 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:33.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:33.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:33.255 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:43:33.255 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:43:33.290 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:43:33.290 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:43:33.332 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:43:33.332 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:43:33.334 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:43:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:43:33] "GET /metrics HTTP/1.1" 200 36624 "" "Prometheus/2.51.0" 2024-09-21T10:43:33.336 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:33.363 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:33.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:33.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:33.370 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:43:33.370 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:43:33.406 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:43:33.407 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:43:33.442 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:43:33.443 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:43:33.465 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:33.473 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:43:33.473 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:43:33.491 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:33.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:33.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:33.593 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:33.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:33 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 645/101817 objects degraded (0.633%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:43:33.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:33 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3105616245' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:43:33.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:33 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 645/101817 objects degraded (0.633%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:43:33.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:33 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3105616245' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:43:33.629 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:33.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:33.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:33.730 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:33.760 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:33.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:33.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:33 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 645/101817 objects degraded (0.633%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:43:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:33 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3105616245' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:43:33.862 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:33.897 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:33.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:33.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:33.998 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:34.028 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:34.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:34.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:34.130 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:34.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:34.179 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:34.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:34.280 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:34.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:34.309 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:34.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:34.410 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:34.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:34.457 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:34.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:34.558 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:34.603 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:34 smithi070 ceph-mon[107385]: pgmap v2289: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 1.9 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.08k op/s; 645/94503 objects degraded (0.683%); 0 B/s, 61 objects/s recovering 2024-09-21T10:43:34.603 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:34 smithi070 ceph-mon[112139]: pgmap v2289: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 1.9 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.08k op/s; 645/94503 objects degraded (0.683%); 0 B/s, 61 objects/s recovering 2024-09-21T10:43:34.604 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:34.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:34.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:34.705 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:34.742 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:34.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:34.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:34 smithi160 ceph-mon[101879]: pgmap v2289: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 1.9 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.08k op/s; 645/94503 objects degraded (0.683%); 0 B/s, 61 objects/s recovering 2024-09-21T10:43:34.844 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:34.879 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:34.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:34.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:34.981 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:35.015 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:35.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:35.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:35.116 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:35.150 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:35.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:35.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:35.252 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:35.295 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:35.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:35.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:35.397 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:35.428 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:35.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:35.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:35.529 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:35.569 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:35.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:35.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:35.671 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:35.712 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:35.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:35.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:35.814 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:35.845 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:35.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:35.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:35.947 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:35.974 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:35.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:35.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:36.075 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:36.101 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:36.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:36.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:36.203 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:36.240 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:36.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:36.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:36.342 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:36.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:36.394 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:36.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:36.409 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:36 smithi070 ceph-mon[112139]: pgmap v2290: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 1.8 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.12k op/s; 276/88584 objects degraded (0.312%); 0 B/s, 67 objects/s recovering 2024-09-21T10:43:36.409 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:43:36.409 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:43:36.409 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:36 smithi070 ceph-mon[107385]: pgmap v2290: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 1.8 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.12k op/s; 276/88584 objects degraded (0.312%); 0 B/s, 67 objects/s recovering 2024-09-21T10:43:36.409 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:43:36.409 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:43:36.495 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:36.524 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:36.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:36.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:36.560 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:43:36.561 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T10:43:36.561 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T10:43:36.561 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T10:43:36.561 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:43:36.625 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:36.657 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:36.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:36.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:36.664 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6] out_osds: [2, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:43:36.664 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:43:36.665 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-21T10:43:36.665 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:43:36.665 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:43:36.665 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-21T10:43:36.733 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:36 smithi160 ceph-mon[101879]: pgmap v2290: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 1.8 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.12k op/s; 276/88584 objects degraded (0.312%); 0 B/s, 67 objects/s recovering 2024-09-21T10:43:36.733 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:43:36.733 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:43:36.758 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:36.794 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:36.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:36.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:36.896 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:36.939 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:36.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:36.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:37.036 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:43:37.041 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:37.077 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:37.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:37.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:37.179 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:37.213 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:37.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:37.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:37.222 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:43:37.258 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:43:37.315 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:37.401 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:37.401 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T10:43:37.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:37.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:37.503 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:37.511 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:37 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2309389174' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:37.511 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:37 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:37.511 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:37 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:43:37.547 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:37.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:37.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:37 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2309389174' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:37 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:37 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:43:37.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:37 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2309389174' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:37.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:37 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:37 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:43:37.649 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:37.685 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:37.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:37.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:37.787 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:37.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:37.817 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:37.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:37.917 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:37.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:37.955 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:37.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:38.056 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:38.077 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:38.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:38.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:38.179 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:38.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:38.213 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:38.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:38.314 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:38.340 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:38.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:38.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:38.442 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:38.471 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:38.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:38.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:38.507 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:38.535 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:38.572 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:38 smithi160 ceph-mon[101879]: pgmap v2291: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 1.6 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 1.22k op/s; 276/77352 objects degraded (0.357%); 0 B/s, 57 objects/s recovering 2024-09-21T10:43:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:38 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:43:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:38 smithi160 ceph-mon[101879]: osdmap e1350: 8 total, 8 up, 6 in 2024-09-21T10:43:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:38 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 276/77352 objects degraded (0.357%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:43:38.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:38 smithi160 ceph-mon[101879]: osdmap e1351: 8 total, 8 up, 6 in 2024-09-21T10:43:38.596 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:38.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:38.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:38.637 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:38 smithi070 ceph-mon[107385]: pgmap v2291: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 1.6 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 1.22k op/s; 276/77352 objects degraded (0.357%); 0 B/s, 57 objects/s recovering 2024-09-21T10:43:38.637 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:38 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:43:38.638 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:38 smithi070 ceph-mon[107385]: osdmap e1350: 8 total, 8 up, 6 in 2024-09-21T10:43:38.638 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:38 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 276/77352 objects degraded (0.357%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:43:38.638 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:38 smithi070 ceph-mon[107385]: osdmap e1351: 8 total, 8 up, 6 in 2024-09-21T10:43:38.638 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:38 smithi070 ceph-mon[112139]: pgmap v2291: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 1.6 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 1.22k op/s; 276/77352 objects degraded (0.357%); 0 B/s, 57 objects/s recovering 2024-09-21T10:43:38.638 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:38 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:43:38.638 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:38 smithi070 ceph-mon[112139]: osdmap e1350: 8 total, 8 up, 6 in 2024-09-21T10:43:38.638 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:38 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 276/77352 objects degraded (0.357%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:43:38.638 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:38 smithi070 ceph-mon[112139]: osdmap e1351: 8 total, 8 up, 6 in 2024-09-21T10:43:38.698 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:38.722 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:38.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:38.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:38.823 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:38.846 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:38.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:38.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:38.914 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:43:38.914 INFO:teuthology.orchestra.run.smithi160.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T10:43:38.914 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:43:38.948 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:38.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:38.970 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:38.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:39.070 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:39.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:39.093 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:39.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:39.194 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:39.224 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:39.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:39.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:39.326 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:39.365 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:39.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:39.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:39.467 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:39.504 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:39.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:39.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:39.606 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:39.645 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:39.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:39.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:39.746 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:39.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:39.784 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:39.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:39.884 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:39.911 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:39.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:39.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:40.013 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:40.038 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:40.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:40.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:40.140 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:40.162 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:40 smithi070 ceph-mon[107385]: pgmap v2294: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 1.6 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 1.02k op/s; 276/77352 objects degraded (0.357%); 0 B/s, 46 objects/s recovering 2024-09-21T10:43:40.178 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:40.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:40.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:40.279 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:40.313 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:40.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:40.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:40.415 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:40.441 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:43:40.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:40.454 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:40.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:40.500 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:40 smithi070 ceph-mon[112139]: pgmap v2294: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 1.6 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 1.02k op/s; 276/77352 objects degraded (0.357%); 0 B/s, 46 objects/s recovering 2024-09-21T10:43:40.501 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:40 smithi070 ceph-mon[112139]: osdmap e1352: 8 total, 8 up, 6 in 2024-09-21T10:43:40.502 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:40 smithi070 ceph-mon[107385]: osdmap e1352: 8 total, 8 up, 6 in 2024-09-21T10:43:40.555 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:40.580 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:40.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:40.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:40.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:40 smithi160 ceph-mon[101879]: pgmap v2294: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait, 96 active+clean; 1.6 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 1.02k op/s; 276/77352 objects degraded (0.357%); 0 B/s, 46 objects/s recovering 2024-09-21T10:43:40.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:40 smithi160 ceph-mon[101879]: osdmap e1352: 8 total, 8 up, 6 in 2024-09-21T10:43:40.682 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:40.731 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:40.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:40.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:40.832 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:40.857 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:40.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:40.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:40.959 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:40.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:40.990 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:40.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:41.090 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:41.123 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:41.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:41.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:41.224 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:41.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:41.259 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:41.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:41.360 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:41.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:41 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "22"} : dispatch 2024-09-21T10:43:41.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:41 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "22"} : dispatch 2024-09-21T10:43:41.382 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:41.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:41.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:41.484 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:41.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:41.505 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:41.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:41.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:41 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "22"} : dispatch 2024-09-21T10:43:41.607 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:41.628 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:41.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:41.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:41.704 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:43:41.704 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T10:43:41.704 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T10:43:41.704 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T10:43:41.704 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.704 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77348 53.10 53:0a4ac980:::benchmark_data_smithi070_228967_object77347:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:43:41.704 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.666247+0000", 2024-09-21T10:43:41.705 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 155.94596471, 2024-09-21T10:43:41.705 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1333923189999999, 2024-09-21T10:43:41.705 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:43:41.705 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:43:41.705 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:41.705 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:43:41.705 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:43:41.705 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:41.705 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77348 2024-09-21T10:43:41.705 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.705 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:43:41.705 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.705 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:43:41.705 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.666247+0000", 2024-09-21T10:43:41.706 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.706 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.706 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.706 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:43:41.706 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.666247+0000", 2024-09-21T10:43:41.706 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.706 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.706 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.706 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:43:41.706 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.666249+0000", 2024-09-21T10:43:41.706 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.432e-06 2024-09-21T10:43:41.706 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.706 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.707 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:43:41.707 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.666267+0000", 2024-09-21T10:43:41.707 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7736000000000001e-05 2024-09-21T10:43:41.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.707 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.707 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:43:41.707 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.666278+0000", 2024-09-21T10:43:41.707 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1112e-05 2024-09-21T10:43:41.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.707 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.707 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:43:41.707 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.666286+0000", 2024-09-21T10:43:41.707 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9820000000000008e-06 2024-09-21T10:43:41.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.708 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.708 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.708 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.666343+0000", 2024-09-21T10:43:41.708 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7099000000000003e-05 2024-09-21T10:43:41.708 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.708 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.708 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.708 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.666353+0000", 2024-09-21T10:43:41.708 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.6590000000000002e-06 2024-09-21T10:43:41.708 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.708 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.708 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.708 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659584+0000", 2024-09-21T10:43:41.709 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99323149799999999 2024-09-21T10:43:41.709 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.709 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.709 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.709 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659596+0000", 2024-09-21T10:43:41.709 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2294e-05 2024-09-21T10:43:41.709 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.709 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.709 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.709 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.795107+0000", 2024-09-21T10:43:41.709 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1355107019999999 2024-09-21T10:43:41.709 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.709 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.709 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:43:41.710 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.795298+0000", 2024-09-21T10:43:41.710 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000190893 2024-09-21T10:43:41.710 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.710 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.710 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:43:41.710 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.795500+0000", 2024-09-21T10:43:41.710 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000202084 2024-09-21T10:43:41.710 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.710 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.710 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.710 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.798730+0000", 2024-09-21T10:43:41.710 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00323025 2024-09-21T10:43:41.710 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.710 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.711 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:43:41.711 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.798743+0000", 2024-09-21T10:43:41.711 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3009e-05 2024-09-21T10:43:41.711 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.711 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.711 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.711 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.798773+0000", 2024-09-21T10:43:41.711 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9974000000000001e-05 2024-09-21T10:43:41.711 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.711 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.711 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:43:41.711 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.798843+0000", 2024-09-21T10:43:41.712 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9523999999999994e-05 2024-09-21T10:43:41.712 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.712 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.712 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:43:41.712 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.799640+0000", 2024-09-21T10:43:41.712 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000797071 2024-09-21T10:43:41.712 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.712 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:43:41.712 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.712 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.712 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.712 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77354 53.10 53:09eb2e30:::benchmark_data_smithi070_228967_object77353:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:43:41.712 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.669089+0000", 2024-09-21T10:43:41.712 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 155.94312317800001, 2024-09-21T10:43:41.712 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1315911619999999, 2024-09-21T10:43:41.713 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:43:41.713 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:43:41.713 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:41.713 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:43:41.713 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:43:41.713 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:41.713 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77354 2024-09-21T10:43:41.713 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.713 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:43:41.713 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.713 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:43:41.713 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.669089+0000", 2024-09-21T10:43:41.713 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.714 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.714 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:43:41.714 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.669089+0000", 2024-09-21T10:43:41.714 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.714 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.714 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:43:41.714 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.669091+0000", 2024-09-21T10:43:41.714 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7099999999999999e-06 2024-09-21T10:43:41.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.714 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.715 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:43:41.715 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.669118+0000", 2024-09-21T10:43:41.715 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7500000000000001e-05 2024-09-21T10:43:41.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.715 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.715 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:43:41.715 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.669137+0000", 2024-09-21T10:43:41.715 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8529000000000002e-05 2024-09-21T10:43:41.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.715 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.715 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:43:41.715 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.669145+0000", 2024-09-21T10:43:41.715 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6580000000000008e-06 2024-09-21T10:43:41.716 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.716 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.716 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.716 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.669213+0000", 2024-09-21T10:43:41.716 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7706000000000005e-05 2024-09-21T10:43:41.716 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.716 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.716 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.716 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.669223+0000", 2024-09-21T10:43:41.716 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0098999999999999e-05 2024-09-21T10:43:41.716 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.716 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.716 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.716 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659628+0000", 2024-09-21T10:43:41.716 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99040486100000003 2024-09-21T10:43:41.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.717 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.717 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.717 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659636+0000", 2024-09-21T10:43:41.717 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4449999999999999e-06 2024-09-21T10:43:41.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.717 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.717 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.717 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.797849+0000", 2024-09-21T10:43:41.717 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.138213956 2024-09-21T10:43:41.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.717 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.717 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:43:41.718 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.797958+0000", 2024-09-21T10:43:41.718 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010826400000000001 2024-09-21T10:43:41.718 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.718 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.718 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:43:41.718 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.798068+0000", 2024-09-21T10:43:41.718 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011027399999999999 2024-09-21T10:43:41.718 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.718 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.718 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.718 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.799784+0000", 2024-09-21T10:43:41.718 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001716239 2024-09-21T10:43:41.718 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.719 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.719 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.719 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.799842+0000", 2024-09-21T10:43:41.719 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7738999999999997e-05 2024-09-21T10:43:41.719 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.719 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.719 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:43:41.719 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.799858+0000", 2024-09-21T10:43:41.719 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5605000000000001e-05 2024-09-21T10:43:41.719 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.719 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.719 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:43:41.719 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.799915+0000", 2024-09-21T10:43:41.720 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7553999999999998e-05 2024-09-21T10:43:41.720 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.720 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.720 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:43:41.720 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.800680+0000", 2024-09-21T10:43:41.720 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00076502299999999998 2024-09-21T10:43:41.720 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.720 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:43:41.720 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.720 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.720 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.720 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77415 53.10 53:0eba5a74:::benchmark_data_smithi070_228967_object77414:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:43:41.720 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.700483+0000", 2024-09-21T10:43:41.721 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 155.911729278, 2024-09-21T10:43:41.721 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1042608860000001, 2024-09-21T10:43:41.721 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:43:41.721 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:43:41.721 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:41.721 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:43:41.721 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:43:41.721 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:41.721 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77415 2024-09-21T10:43:41.721 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.721 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:43:41.721 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.721 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:43:41.721 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700483+0000", 2024-09-21T10:43:41.722 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.722 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.722 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:43:41.722 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700483+0000", 2024-09-21T10:43:41.722 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.722 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.722 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:43:41.722 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700485+0000", 2024-09-21T10:43:41.722 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5880000000000001e-06 2024-09-21T10:43:41.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.722 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.723 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:43:41.723 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700506+0000", 2024-09-21T10:43:41.723 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1049000000000001e-05 2024-09-21T10:43:41.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.723 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.723 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:43:41.723 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700517+0000", 2024-09-21T10:43:41.723 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1603000000000001e-05 2024-09-21T10:43:41.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.723 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.723 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:43:41.723 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700524+0000", 2024-09-21T10:43:41.723 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1670000000000002e-06 2024-09-21T10:43:41.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.724 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.724 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.724 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700751+0000", 2024-09-21T10:43:41.724 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00022716300000000001 2024-09-21T10:43:41.724 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.724 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.724 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.724 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700762+0000", 2024-09-21T10:43:41.724 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0550999999999999e-05 2024-09-21T10:43:41.724 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.724 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.724 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.724 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659665+0000", 2024-09-21T10:43:41.724 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.95890305200000003 2024-09-21T10:43:41.725 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.725 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.725 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.725 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659673+0000", 2024-09-21T10:43:41.725 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4950000000000002e-06 2024-09-21T10:43:41.725 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.725 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.725 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.725 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.800708+0000", 2024-09-21T10:43:41.725 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.141035383 2024-09-21T10:43:41.725 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.725 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.725 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:43:41.725 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.800824+0000", 2024-09-21T10:43:41.726 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000116431 2024-09-21T10:43:41.726 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.726 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.726 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:43:41.726 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.800934+0000", 2024-09-21T10:43:41.726 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010966799999999999 2024-09-21T10:43:41.726 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.726 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.726 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.726 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.803154+0000", 2024-09-21T10:43:41.726 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0022197940000000002 2024-09-21T10:43:41.726 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.727 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.727 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.727 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.804093+0000", 2024-09-21T10:43:41.727 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00093908499999999998 2024-09-21T10:43:41.727 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.727 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.727 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:43:41.727 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.804102+0000", 2024-09-21T10:43:41.727 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5979999999999997e-06 2024-09-21T10:43:41.727 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.727 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.727 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:43:41.727 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.804136+0000", 2024-09-21T10:43:41.727 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4536999999999998e-05 2024-09-21T10:43:41.728 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.728 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.728 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:43:41.728 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.804744+0000", 2024-09-21T10:43:41.728 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00060772199999999999 2024-09-21T10:43:41.728 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.728 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:43:41.728 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.728 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.728 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.728 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77416 53.10 53:0b890937:::benchmark_data_smithi070_228967_object77415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:43:41.728 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.700546+0000", 2024-09-21T10:43:41.728 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 155.91166617900001, 2024-09-21T10:43:41.728 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1047850320000001, 2024-09-21T10:43:41.729 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:43:41.729 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:43:41.729 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:41.729 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:43:41.729 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:43:41.729 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:41.729 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77416 2024-09-21T10:43:41.729 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.729 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:43:41.729 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.729 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:43:41.729 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700546+0000", 2024-09-21T10:43:41.729 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.729 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.730 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.730 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:43:41.730 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700546+0000", 2024-09-21T10:43:41.730 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.730 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.730 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.730 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:43:41.730 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700547+0000", 2024-09-21T10:43:41.730 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0999999999999999e-07 2024-09-21T10:43:41.730 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.730 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.731 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:43:41.731 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700564+0000", 2024-09-21T10:43:41.731 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7252e-05 2024-09-21T10:43:41.731 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.731 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.731 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:43:41.731 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700572+0000", 2024-09-21T10:43:41.731 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9039999999999992e-06 2024-09-21T10:43:41.731 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.731 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.731 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:43:41.731 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700577+0000", 2024-09-21T10:43:41.731 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.699e-06 2024-09-21T10:43:41.732 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.732 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.732 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.732 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700788+0000", 2024-09-21T10:43:41.732 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00021084000000000001 2024-09-21T10:43:41.732 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.732 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.732 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.732 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700796+0000", 2024-09-21T10:43:41.732 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6090000000000003e-06 2024-09-21T10:43:41.732 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.732 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.732 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.732 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659707+0000", 2024-09-21T10:43:41.733 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.95891092 2024-09-21T10:43:41.733 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.733 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.733 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.733 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659711+0000", 2024-09-21T10:43:41.733 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3789999999999999e-06 2024-09-21T10:43:41.733 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.733 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.733 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.733 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.802199+0000", 2024-09-21T10:43:41.734 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.142488341 2024-09-21T10:43:41.734 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.734 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.734 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:43:41.734 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.802320+0000", 2024-09-21T10:43:41.734 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000120209 2024-09-21T10:43:41.734 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.734 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.734 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:43:41.734 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.802494+0000", 2024-09-21T10:43:41.734 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000174771 2024-09-21T10:43:41.734 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.735 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.735 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.735 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.804796+0000", 2024-09-21T10:43:41.735 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002301726 2024-09-21T10:43:41.735 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.735 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.735 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.735 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.804844+0000", 2024-09-21T10:43:41.735 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8266000000000002e-05 2024-09-21T10:43:41.735 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.735 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.735 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:43:41.735 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.804854+0000", 2024-09-21T10:43:41.736 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0018e-05 2024-09-21T10:43:41.736 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.736 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.736 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:43:41.736 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.804893+0000", 2024-09-21T10:43:41.736 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8430999999999998e-05 2024-09-21T10:43:41.736 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.736 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.736 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:43:41.736 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.805331+0000", 2024-09-21T10:43:41.736 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000438157 2024-09-21T10:43:41.736 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.736 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:43:41.737 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.737 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.737 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.737 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77417 53.10 53:0e921901:::benchmark_data_smithi070_228967_object77416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:43:41.737 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.700585+0000", 2024-09-21T10:43:41.737 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 155.911627012, 2024-09-21T10:43:41.737 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.118979521, 2024-09-21T10:43:41.737 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:43:41.737 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:43:41.737 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:41.737 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:43:41.737 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:43:41.737 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:41.738 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77417 2024-09-21T10:43:41.738 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.738 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:43:41.738 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.738 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:43:41.738 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700585+0000", 2024-09-21T10:43:41.738 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.738 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.738 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.738 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:43:41.738 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700585+0000", 2024-09-21T10:43:41.738 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.738 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.739 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.739 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:43:41.739 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700586+0000", 2024-09-21T10:43:41.739 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0400000000000002e-07 2024-09-21T10:43:41.739 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.739 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.739 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:43:41.739 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700605+0000", 2024-09-21T10:43:41.739 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9463999999999999e-05 2024-09-21T10:43:41.739 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.739 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.739 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:43:41.739 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700612+0000", 2024-09-21T10:43:41.739 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5019999999999998e-06 2024-09-21T10:43:41.740 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.740 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.740 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:43:41.740 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700617+0000", 2024-09-21T10:43:41.740 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9520000000000002e-06 2024-09-21T10:43:41.740 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.740 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.740 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.740 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700902+0000", 2024-09-21T10:43:41.740 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00028501699999999999 2024-09-21T10:43:41.740 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.740 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.740 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.741 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700910+0000", 2024-09-21T10:43:41.741 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1999999999999994e-06 2024-09-21T10:43:41.741 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.741 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.741 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.741 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659744+0000", 2024-09-21T10:43:41.741 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.95883443300000004 2024-09-21T10:43:41.741 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.741 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.741 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.741 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659753+0000", 2024-09-21T10:43:41.741 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.4379999999999995e-06 2024-09-21T10:43:41.741 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.741 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.742 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.742 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.809941+0000", 2024-09-21T10:43:41.742 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1501882539999999 2024-09-21T10:43:41.742 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.742 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.742 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:43:41.742 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.810050+0000", 2024-09-21T10:43:41.742 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010925999999999999 2024-09-21T10:43:41.742 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.742 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.742 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:43:41.742 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.810156+0000", 2024-09-21T10:43:41.742 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010574 2024-09-21T10:43:41.743 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.743 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.743 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:43:41.743 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.813660+0000", 2024-09-21T10:43:41.743 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035046370000000001 2024-09-21T10:43:41.743 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.743 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.743 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.743 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.813692+0000", 2024-09-21T10:43:41.743 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1769e-05 2024-09-21T10:43:41.743 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.743 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.743 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.743 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.818533+0000", 2024-09-21T10:43:41.744 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0048406539999999998 2024-09-21T10:43:41.744 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.744 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.744 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:43:41.744 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.818577+0000", 2024-09-21T10:43:41.744 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3915000000000002e-05 2024-09-21T10:43:41.744 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.744 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.744 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:43:41.744 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.819565+0000", 2024-09-21T10:43:41.744 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00098788200000000004 2024-09-21T10:43:41.744 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.744 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:43:41.744 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.745 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.745 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.745 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77499 53.10 53:0c45bed1:::benchmark_data_smithi070_228967_object77498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:43:41.745 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.763242+0000", 2024-09-21T10:43:41.745 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 155.84896974099999, 2024-09-21T10:43:41.745 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0610106340000001, 2024-09-21T10:43:41.745 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:43:41.745 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:43:41.745 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:41.745 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:43:41.745 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:43:41.745 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:41.745 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77499 2024-09-21T10:43:41.746 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.746 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:43:41.746 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.746 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:43:41.746 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.763242+0000", 2024-09-21T10:43:41.746 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.746 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.746 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.746 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:43:41.746 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.763242+0000", 2024-09-21T10:43:41.746 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.746 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.746 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.746 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:43:41.747 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.763244+0000", 2024-09-21T10:43:41.747 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.646e-06 2024-09-21T10:43:41.747 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.747 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.747 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:43:41.747 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.763268+0000", 2024-09-21T10:43:41.747 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4303000000000001e-05 2024-09-21T10:43:41.747 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.747 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.747 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:43:41.747 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.763282+0000", 2024-09-21T10:43:41.747 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4018e-05 2024-09-21T10:43:41.748 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.748 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.748 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:43:41.748 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.763290+0000", 2024-09-21T10:43:41.748 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1570000000000003e-06 2024-09-21T10:43:41.748 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.748 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.748 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.748 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.763365+0000", 2024-09-21T10:43:41.748 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5511000000000003e-05 2024-09-21T10:43:41.748 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.748 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.749 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.749 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.763387+0000", 2024-09-21T10:43:41.749 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1639000000000001e-05 2024-09-21T10:43:41.749 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.749 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.749 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.749 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659776+0000", 2024-09-21T10:43:41.749 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.89638960999999995 2024-09-21T10:43:41.749 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.749 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.749 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.749 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659780+0000", 2024-09-21T10:43:41.749 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8890000000000004e-06 2024-09-21T10:43:41.750 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.750 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.750 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.750 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.810900+0000", 2024-09-21T10:43:41.750 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1511196960000001 2024-09-21T10:43:41.750 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.750 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.750 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:43:41.750 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.810983+0000", 2024-09-21T10:43:41.750 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2664999999999995e-05 2024-09-21T10:43:41.750 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.750 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.750 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:43:41.751 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.811076+0000", 2024-09-21T10:43:41.751 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3386000000000003e-05 2024-09-21T10:43:41.751 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.751 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.751 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.751 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.813728+0000", 2024-09-21T10:43:41.751 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0026517609999999999 2024-09-21T10:43:41.751 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.751 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.751 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:43:41.751 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.813804+0000", 2024-09-21T10:43:41.751 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6404999999999998e-05 2024-09-21T10:43:41.751 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.751 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.752 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.752 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.823202+0000", 2024-09-21T10:43:41.752 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0093975829999999993 2024-09-21T10:43:41.752 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.752 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.752 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:43:41.752 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.823265+0000", 2024-09-21T10:43:41.752 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3585999999999997e-05 2024-09-21T10:43:41.752 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.752 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.752 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:43:41.752 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.824253+0000", 2024-09-21T10:43:41.752 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00098777900000000004 2024-09-21T10:43:41.753 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.753 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:43:41.753 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.753 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.753 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.753 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77507 53.10 53:0c3b2271:::benchmark_data_smithi070_228967_object77506:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:43:41.753 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.767593+0000", 2024-09-21T10:43:41.753 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 155.84461922400001, 2024-09-21T10:43:41.753 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0575902039999998, 2024-09-21T10:43:41.753 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:43:41.753 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:43:41.753 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:41.753 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:43:41.753 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:43:41.754 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:41.754 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77507 2024-09-21T10:43:41.754 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.754 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:43:41.754 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.754 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:43:41.754 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.767593+0000", 2024-09-21T10:43:41.754 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.754 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.754 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.754 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:43:41.754 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.767593+0000", 2024-09-21T10:43:41.754 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.755 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.755 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.755 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:43:41.755 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.767595+0000", 2024-09-21T10:43:41.755 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5880000000000001e-06 2024-09-21T10:43:41.755 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.755 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.755 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:43:41.755 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.767643+0000", 2024-09-21T10:43:41.755 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8563000000000002e-05 2024-09-21T10:43:41.755 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.755 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.755 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:43:41.755 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.767659+0000", 2024-09-21T10:43:41.756 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6186000000000001e-05 2024-09-21T10:43:41.756 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.756 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.756 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:43:41.756 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.767667+0000", 2024-09-21T10:43:41.756 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2200000000000003e-06 2024-09-21T10:43:41.756 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.756 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.756 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.756 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.767727+0000", 2024-09-21T10:43:41.756 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0702999999999998e-05 2024-09-21T10:43:41.756 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.756 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.757 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.757 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.767737+0000", 2024-09-21T10:43:41.757 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.6050000000000004e-06 2024-09-21T10:43:41.757 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.757 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.757 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.757 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659809+0000", 2024-09-21T10:43:41.757 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.89207242600000003 2024-09-21T10:43:41.757 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.757 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.757 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.757 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659817+0000", 2024-09-21T10:43:41.757 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5440000000000001e-06 2024-09-21T10:43:41.758 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.758 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.758 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.758 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.811691+0000", 2024-09-21T10:43:41.758 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1518743419999999 2024-09-21T10:43:41.758 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.758 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.758 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:43:41.758 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.811779+0000", 2024-09-21T10:43:41.758 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.8119000000000001e-05 2024-09-21T10:43:41.758 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.758 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.758 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:43:41.758 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.811886+0000", 2024-09-21T10:43:41.759 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000106805 2024-09-21T10:43:41.759 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.759 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.759 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.759 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.813784+0000", 2024-09-21T10:43:41.759 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00189806 2024-09-21T10:43:41.759 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.759 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.759 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:43:41.759 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.813814+0000", 2024-09-21T10:43:41.759 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0090000000000002e-05 2024-09-21T10:43:41.759 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.759 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.760 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.760 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.824353+0000", 2024-09-21T10:43:41.760 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.010538403 2024-09-21T10:43:41.760 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.760 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.760 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:43:41.760 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.824440+0000", 2024-09-21T10:43:41.760 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7063000000000003e-05 2024-09-21T10:43:41.760 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.760 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.760 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:43:41.760 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.825183+0000", 2024-09-21T10:43:41.760 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00074348700000000003 2024-09-21T10:43:41.761 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.761 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:43:41.761 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.761 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.761 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.761 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77522 53.10 53:0d13612e:::benchmark_data_smithi070_228967_object77521:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:43:41.761 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.778011+0000", 2024-09-21T10:43:41.761 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 155.83420123299999, 2024-09-21T10:43:41.761 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0483339169999999, 2024-09-21T10:43:41.761 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:43:41.761 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:43:41.761 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:41.761 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:43:41.761 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:43:41.762 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:41.762 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77522 2024-09-21T10:43:41.762 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.762 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:43:41.762 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.762 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:43:41.762 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.778011+0000", 2024-09-21T10:43:41.762 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.762 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.762 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.762 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:43:41.762 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.778011+0000", 2024-09-21T10:43:41.762 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.763 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.763 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.763 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:43:41.763 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.778013+0000", 2024-09-21T10:43:41.763 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9970000000000001e-06 2024-09-21T10:43:41.763 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.763 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.763 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:43:41.763 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.778037+0000", 2024-09-21T10:43:41.763 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4521000000000001e-05 2024-09-21T10:43:41.763 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.763 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.763 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:43:41.763 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.778050+0000", 2024-09-21T10:43:41.764 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2578999999999999e-05 2024-09-21T10:43:41.764 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.764 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.764 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:43:41.764 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.778058+0000", 2024-09-21T10:43:41.764 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2320000000000001e-06 2024-09-21T10:43:41.764 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.764 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.764 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.764 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.778124+0000", 2024-09-21T10:43:41.764 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5396999999999997e-05 2024-09-21T10:43:41.764 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.764 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.764 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.765 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.778133+0000", 2024-09-21T10:43:41.765 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2730000000000008e-06 2024-09-21T10:43:41.765 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.765 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.765 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.765 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659837+0000", 2024-09-21T10:43:41.765 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.88170432399999998 2024-09-21T10:43:41.765 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.765 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.765 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.765 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659841+0000", 2024-09-21T10:43:41.765 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7290000000000002e-06 2024-09-21T10:43:41.765 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.766 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.766 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.766 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.812569+0000", 2024-09-21T10:43:41.766 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.152727949 2024-09-21T10:43:41.766 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.766 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.766 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:43:41.766 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.812704+0000", 2024-09-21T10:43:41.766 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000135019 2024-09-21T10:43:41.766 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.766 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.766 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:43:41.766 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.812863+0000", 2024-09-21T10:43:41.767 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015892200000000001 2024-09-21T10:43:41.767 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.767 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.767 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.767 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.815571+0000", 2024-09-21T10:43:41.767 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0027076460000000002 2024-09-21T10:43:41.767 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.767 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.767 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:43:41.767 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.815618+0000", 2024-09-21T10:43:41.767 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7732000000000003e-05 2024-09-21T10:43:41.767 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.767 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.767 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.768 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.825288+0000", 2024-09-21T10:43:41.768 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0096699300000000002 2024-09-21T10:43:41.768 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.768 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.768 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:43:41.768 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.825349+0000", 2024-09-21T10:43:41.768 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0720999999999997e-05 2024-09-21T10:43:41.768 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.768 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.768 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:43:41.768 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.826345+0000", 2024-09-21T10:43:41.768 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00099594599999999994 2024-09-21T10:43:41.768 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.768 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:43:41.769 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.769 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.769 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.769 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77550 53.10 53:0c033939:::benchmark_data_smithi070_228967_object77549:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:43:41.769 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.799776+0000", 2024-09-21T10:43:41.769 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 155.81243667800001, 2024-09-21T10:43:41.769 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0288999130000001, 2024-09-21T10:43:41.769 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:43:41.769 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:43:41.769 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:41.769 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:43:41.769 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:43:41.769 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:41.770 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77550 2024-09-21T10:43:41.770 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.770 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:43:41.770 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.770 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:43:41.770 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.799776+0000", 2024-09-21T10:43:41.770 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.770 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.770 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.770 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:43:41.770 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.799776+0000", 2024-09-21T10:43:41.770 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.770 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.770 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.771 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:43:41.771 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.799776+0000", 2024-09-21T10:43:41.771 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7599999999999997e-07 2024-09-21T10:43:41.771 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.771 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.771 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:43:41.771 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.799824+0000", 2024-09-21T10:43:41.771 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7649999999999999e-05 2024-09-21T10:43:41.771 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.771 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.771 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:43:41.771 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.799831+0000", 2024-09-21T10:43:41.771 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1330000000000001e-06 2024-09-21T10:43:41.772 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.772 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.772 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:43:41.772 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.799838+0000", 2024-09-21T10:43:41.772 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6270000000000002e-06 2024-09-21T10:43:41.772 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.772 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.772 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.772 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.799898+0000", 2024-09-21T10:43:41.772 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0849e-05 2024-09-21T10:43:41.772 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.772 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.772 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.773 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.799907+0000", 2024-09-21T10:43:41.773 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1500000000000005e-06 2024-09-21T10:43:41.773 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.773 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.773 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.773 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659869+0000", 2024-09-21T10:43:41.773 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.85996196300000005 2024-09-21T10:43:41.773 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.773 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.773 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.773 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659877+0000", 2024-09-21T10:43:41.773 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.255e-06 2024-09-21T10:43:41.773 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.774 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.774 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.774 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.813839+0000", 2024-09-21T10:43:41.774 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.153962089 2024-09-21T10:43:41.774 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.774 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.774 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:43:41.774 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.813940+0000", 2024-09-21T10:43:41.774 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000100719 2024-09-21T10:43:41.774 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.774 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.774 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:43:41.774 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.814072+0000", 2024-09-21T10:43:41.774 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000132065 2024-09-21T10:43:41.775 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.775 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.775 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.775 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.815610+0000", 2024-09-21T10:43:41.775 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0015380389999999999 2024-09-21T10:43:41.775 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.775 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.775 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:43:41.775 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.815622+0000", 2024-09-21T10:43:41.775 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2714000000000001e-05 2024-09-21T10:43:41.775 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.775 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.775 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.776 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.828115+0000", 2024-09-21T10:43:41.776 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012492207999999999 2024-09-21T10:43:41.776 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.776 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.776 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:43:41.776 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.828184+0000", 2024-09-21T10:43:41.776 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9218e-05 2024-09-21T10:43:41.776 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.776 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.776 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:43:41.776 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.828675+0000", 2024-09-21T10:43:41.776 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00049165799999999996 2024-09-21T10:43:41.776 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.777 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:43:41.777 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.777 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.777 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.777 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77552 53.10 53:092395bc:::benchmark_data_smithi070_228967_object77551:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:43:41.777 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.801039+0000", 2024-09-21T10:43:41.777 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 155.81117319200001, 2024-09-21T10:43:41.777 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0284018509999999, 2024-09-21T10:43:41.777 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:43:41.777 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:43:41.777 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:41.777 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:43:41.777 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:43:41.777 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:41.777 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77552 2024-09-21T10:43:41.778 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.778 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:43:41.778 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.778 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:43:41.778 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.801039+0000", 2024-09-21T10:43:41.778 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.778 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.778 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.778 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:43:41.778 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.801039+0000", 2024-09-21T10:43:41.778 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.778 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.778 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.779 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:43:41.779 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.801040+0000", 2024-09-21T10:43:41.779 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.184e-06 2024-09-21T10:43:41.779 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.779 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.779 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:43:41.779 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.801089+0000", 2024-09-21T10:43:41.779 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9313e-05 2024-09-21T10:43:41.779 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.779 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.779 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:43:41.779 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.801103+0000", 2024-09-21T10:43:41.780 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3816e-05 2024-09-21T10:43:41.780 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.780 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.780 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:43:41.780 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.801111+0000", 2024-09-21T10:43:41.780 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5820000000000004e-06 2024-09-21T10:43:41.780 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.780 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.780 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.780 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.801164+0000", 2024-09-21T10:43:41.780 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3362000000000002e-05 2024-09-21T10:43:41.780 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.780 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.780 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.781 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.801173+0000", 2024-09-21T10:43:41.781 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6649999999999992e-06 2024-09-21T10:43:41.781 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.781 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.781 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.781 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659897+0000", 2024-09-21T10:43:41.781 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.85872434499999994 2024-09-21T10:43:41.781 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.781 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.781 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.781 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659901+0000", 2024-09-21T10:43:41.781 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4479999999999999e-06 2024-09-21T10:43:41.781 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.781 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.782 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.782 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.814871+0000", 2024-09-21T10:43:41.782 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.154970026 2024-09-21T10:43:41.782 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.782 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.782 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:43:41.782 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.814966+0000", 2024-09-21T10:43:41.782 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5260999999999995e-05 2024-09-21T10:43:41.782 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.782 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.782 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:43:41.782 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.815058+0000", 2024-09-21T10:43:41.782 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1976000000000002e-05 2024-09-21T10:43:41.782 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.783 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.783 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.783 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.818386+0000", 2024-09-21T10:43:41.783 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0033278079999999998 2024-09-21T10:43:41.783 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.783 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.783 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:43:41.783 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.818396+0000", 2024-09-21T10:43:41.783 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.046e-05 2024-09-21T10:43:41.783 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.783 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.783 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.783 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.828708+0000", 2024-09-21T10:43:41.784 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.010312089999999999 2024-09-21T10:43:41.784 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.784 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.784 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:43:41.784 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.828766+0000", 2024-09-21T10:43:41.784 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7964000000000003e-05 2024-09-21T10:43:41.784 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.784 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.784 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:43:41.784 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.829441+0000", 2024-09-21T10:43:41.784 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00067455100000000001 2024-09-21T10:43:41.784 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.785 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:43:41.785 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.785 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.785 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.785 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77583 53.10 53:0c625067:::benchmark_data_smithi070_228967_object77582:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:43:41.785 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.823905+0000", 2024-09-21T10:43:41.785 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 155.788307342, 2024-09-21T10:43:41.785 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0063239049999999, 2024-09-21T10:43:41.785 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:43:41.785 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:43:41.785 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:41.785 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:43:41.785 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:43:41.785 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:41.786 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77583 2024-09-21T10:43:41.786 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.786 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:43:41.786 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.786 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:43:41.786 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.823905+0000", 2024-09-21T10:43:41.786 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.786 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.786 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.786 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:43:41.786 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.823905+0000", 2024-09-21T10:43:41.786 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.786 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.786 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.787 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:43:41.787 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.823906+0000", 2024-09-21T10:43:41.787 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6190000000000001e-06 2024-09-21T10:43:41.787 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.787 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.787 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:43:41.787 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.823936+0000", 2024-09-21T10:43:41.787 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9220000000000001e-05 2024-09-21T10:43:41.787 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.787 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.787 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:43:41.787 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.823947+0000", 2024-09-21T10:43:41.787 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1337e-05 2024-09-21T10:43:41.788 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.788 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.788 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:43:41.788 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.823954+0000", 2024-09-21T10:43:41.788 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8719999999999996e-06 2024-09-21T10:43:41.788 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.788 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.788 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.788 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.824052+0000", 2024-09-21T10:43:41.788 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.8405000000000004e-05 2024-09-21T10:43:41.788 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.788 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.788 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.788 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.824059+0000", 2024-09-21T10:43:41.789 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4110000000000001e-06 2024-09-21T10:43:41.789 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.789 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.789 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.789 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659929+0000", 2024-09-21T10:43:41.789 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.83586998099999998 2024-09-21T10:43:41.789 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.789 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.789 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.789 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659936+0000", 2024-09-21T10:43:41.789 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1180000000000003e-06 2024-09-21T10:43:41.789 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.789 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.790 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.790 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.815647+0000", 2024-09-21T10:43:41.790 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.155710826 2024-09-21T10:43:41.790 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.790 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.790 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:43:41.790 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.815726+0000", 2024-09-21T10:43:41.790 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9102000000000003e-05 2024-09-21T10:43:41.790 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.790 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.790 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:43:41.790 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.815820+0000", 2024-09-21T10:43:41.790 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4326000000000005e-05 2024-09-21T10:43:41.791 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.791 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.791 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:43:41.791 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.818401+0000", 2024-09-21T10:43:41.791 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025805509999999999 2024-09-21T10:43:41.791 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.791 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.791 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.791 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.818441+0000", 2024-09-21T10:43:41.791 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0170999999999999e-05 2024-09-21T10:43:41.791 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.791 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.791 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.791 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.829499+0000", 2024-09-21T10:43:41.792 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.011058182 2024-09-21T10:43:41.792 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.792 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.792 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:43:41.792 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.829557+0000", 2024-09-21T10:43:41.792 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8409999999999998e-05 2024-09-21T10:43:41.792 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.792 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.792 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:43:41.792 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.830229+0000", 2024-09-21T10:43:41.792 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00067137399999999997 2024-09-21T10:43:41.792 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.792 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:43:41.792 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.793 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.793 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.793 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77587 53.10 53:098fcb2a:::benchmark_data_smithi070_228967_object77586:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:43:41.793 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.825548+0000", 2024-09-21T10:43:41.793 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 155.78666373600001, 2024-09-21T10:43:41.793 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0105805079999999, 2024-09-21T10:43:41.793 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:43:41.793 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:43:41.793 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:41.793 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:43:41.793 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:43:41.793 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:41.793 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77587 2024-09-21T10:43:41.793 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.793 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:43:41.794 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.794 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:43:41.794 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.825548+0000", 2024-09-21T10:43:41.794 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.794 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.794 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.794 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:43:41.794 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.825548+0000", 2024-09-21T10:43:41.794 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.794 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.794 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.794 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:43:41.794 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.825550+0000", 2024-09-21T10:43:41.794 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2759999999999999e-06 2024-09-21T10:43:41.795 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.795 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.795 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:43:41.795 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.825602+0000", 2024-09-21T10:43:41.795 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2707000000000002e-05 2024-09-21T10:43:41.795 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.795 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.795 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:43:41.795 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.825617+0000", 2024-09-21T10:43:41.795 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4871e-05 2024-09-21T10:43:41.795 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.795 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.795 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:43:41.796 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.825625+0000", 2024-09-21T10:43:41.796 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0809999999999994e-06 2024-09-21T10:43:41.796 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.796 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.796 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.796 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.825677+0000", 2024-09-21T10:43:41.796 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1218000000000003e-05 2024-09-21T10:43:41.796 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.796 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.796 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.796 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.825685+0000", 2024-09-21T10:43:41.796 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3280000000000006e-06 2024-09-21T10:43:41.796 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.796 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.797 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.797 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659956+0000", 2024-09-21T10:43:41.797 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.83427153799999998 2024-09-21T10:43:41.797 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.797 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.797 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.797 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659961+0000", 2024-09-21T10:43:41.797 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9540000000000003e-06 2024-09-21T10:43:41.797 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.797 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.797 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.797 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.816365+0000", 2024-09-21T10:43:41.797 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1564039589999999 2024-09-21T10:43:41.798 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.798 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.798 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:43:41.798 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.816477+0000", 2024-09-21T10:43:41.798 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000111575 2024-09-21T10:43:41.798 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.798 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.798 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:43:41.798 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.816577+0000", 2024-09-21T10:43:41.798 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000100078 2024-09-21T10:43:41.798 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.798 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.798 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.799 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.818499+0000", 2024-09-21T10:43:41.799 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0019215129999999999 2024-09-21T10:43:41.799 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.799 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.799 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:43:41.799 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.818509+0000", 2024-09-21T10:43:41.799 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0200999999999999e-05 2024-09-21T10:43:41.799 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.799 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.799 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.799 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.834949+0000", 2024-09-21T10:43:41.799 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.016440735000000001 2024-09-21T10:43:41.799 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.799 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.800 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:43:41.800 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.835043+0000", 2024-09-21T10:43:41.800 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3177999999999995e-05 2024-09-21T10:43:41.800 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.800 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.800 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:43:41.800 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.836129+0000", 2024-09-21T10:43:41.800 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001086296 2024-09-21T10:43:41.800 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.800 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:43:41.800 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.800 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.800 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.801 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77621 53.10 53:08d8a996:::benchmark_data_smithi070_228967_object77620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:43:41.801 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.858882+0000", 2024-09-21T10:43:41.801 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 155.75332993200001, 2024-09-21T10:43:41.801 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9785834369999999, 2024-09-21T10:43:41.801 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:43:41.801 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:43:41.801 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:41.801 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:43:41.801 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:43:41.801 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:41.801 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77621 2024-09-21T10:43:41.801 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.801 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:43:41.802 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.802 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:43:41.802 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.858882+0000", 2024-09-21T10:43:41.802 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.802 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.802 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.802 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:43:41.802 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.858882+0000", 2024-09-21T10:43:41.802 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.802 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.802 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.802 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:43:41.802 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.858883+0000", 2024-09-21T10:43:41.802 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5099999999999998e-07 2024-09-21T10:43:41.803 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.803 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.803 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:43:41.803 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.858933+0000", 2024-09-21T10:43:41.803 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9700000000000002e-05 2024-09-21T10:43:41.803 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.803 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.803 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:43:41.803 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.858943+0000", 2024-09-21T10:43:41.803 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7939999999999997e-06 2024-09-21T10:43:41.803 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.803 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.803 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:43:41.803 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.858949+0000", 2024-09-21T10:43:41.804 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7769999999999997e-06 2024-09-21T10:43:41.804 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.804 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.804 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.804 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.859007+0000", 2024-09-21T10:43:41.804 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7456e-05 2024-09-21T10:43:41.804 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.804 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.804 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.804 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.859017+0000", 2024-09-21T10:43:41.804 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0071999999999999e-05 2024-09-21T10:43:41.804 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.805 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.805 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.805 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659993+0000", 2024-09-21T10:43:41.805 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.80097583500000002 2024-09-21T10:43:41.805 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.805 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.805 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.805 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659998+0000", 2024-09-21T10:43:41.805 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4929999999999998e-06 2024-09-21T10:43:41.805 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.805 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.805 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.805 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.817285+0000", 2024-09-21T10:43:41.806 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1572862640000001 2024-09-21T10:43:41.806 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.806 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.806 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:43:41.806 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.817418+0000", 2024-09-21T10:43:41.806 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013367300000000001 2024-09-21T10:43:41.806 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.806 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.806 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:43:41.806 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.817576+0000", 2024-09-21T10:43:41.806 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015830299999999999 2024-09-21T10:43:41.806 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.806 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.806 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.807 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.819630+0000", 2024-09-21T10:43:41.807 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0020534120000000001 2024-09-21T10:43:41.807 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.807 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.807 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:43:41.807 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.820470+0000", 2024-09-21T10:43:41.807 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00083965699999999995 2024-09-21T10:43:41.807 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.807 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.807 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.807 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.836186+0000", 2024-09-21T10:43:41.807 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015716218000000001 2024-09-21T10:43:41.807 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.808 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.808 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:43:41.808 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.836282+0000", 2024-09-21T10:43:41.808 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5839000000000002e-05 2024-09-21T10:43:41.808 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.808 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.808 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:43:41.808 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.837466+0000", 2024-09-21T10:43:41.808 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001184093 2024-09-21T10:43:41.808 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.808 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:43:41.808 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.808 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.809 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.809 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77656 53.10 53:08530a06:::benchmark_data_smithi070_228967_object77655:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:43:41.809 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.901366+0000", 2024-09-21T10:43:41.809 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 155.71084578700001, 2024-09-21T10:43:41.809 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9373663639999998, 2024-09-21T10:43:41.809 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:43:41.809 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:43:41.809 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:41.809 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:43:41.809 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:43:41.809 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:41.809 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77656 2024-09-21T10:43:41.809 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.809 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:43:41.810 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.810 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:43:41.810 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.901366+0000", 2024-09-21T10:43:41.810 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.810 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.810 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.810 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:43:41.810 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.901366+0000", 2024-09-21T10:43:41.810 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.810 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.810 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.810 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:43:41.810 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.901377+0000", 2024-09-21T10:43:41.811 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0122999999999999e-05 2024-09-21T10:43:41.811 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.811 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.811 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:43:41.811 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.901395+0000", 2024-09-21T10:43:41.811 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8896e-05 2024-09-21T10:43:41.811 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.811 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.811 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:43:41.811 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.901407+0000", 2024-09-21T10:43:41.811 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1409999999999999e-05 2024-09-21T10:43:41.811 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.812 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.812 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:43:41.812 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.901413+0000", 2024-09-21T10:43:41.812 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9399999999999999e-06 2024-09-21T10:43:41.812 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.812 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.812 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.812 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.901471+0000", 2024-09-21T10:43:41.812 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7853000000000003e-05 2024-09-21T10:43:41.812 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.812 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.812 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.812 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.901480+0000", 2024-09-21T10:43:41.812 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1959999999999997e-06 2024-09-21T10:43:41.813 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.813 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.813 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.813 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.660022+0000", 2024-09-21T10:43:41.813 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.75854222100000002 2024-09-21T10:43:41.813 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.813 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.813 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.813 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.660026+0000", 2024-09-21T10:43:41.813 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7840000000000001e-06 2024-09-21T10:43:41.813 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.813 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.813 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.814 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.819670+0000", 2024-09-21T10:43:41.814 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1596445310000001 2024-09-21T10:43:41.814 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.814 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.814 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:43:41.814 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.819755+0000", 2024-09-21T10:43:41.814 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5062000000000007e-05 2024-09-21T10:43:41.814 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.814 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.814 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:43:41.814 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.819853+0000", 2024-09-21T10:43:41.814 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7116999999999999e-05 2024-09-21T10:43:41.814 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.814 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.815 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.815 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.823074+0000", 2024-09-21T10:43:41.815 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.003221776 2024-09-21T10:43:41.815 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.815 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.815 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:43:41.815 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.823085+0000", 2024-09-21T10:43:41.815 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0924000000000001e-05 2024-09-21T10:43:41.815 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.815 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.815 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.815 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.837697+0000", 2024-09-21T10:43:41.816 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.014611455000000001 2024-09-21T10:43:41.816 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.816 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.816 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:43:41.816 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.837931+0000", 2024-09-21T10:43:41.816 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00023461000000000001 2024-09-21T10:43:41.816 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.816 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.816 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:43:41.816 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.838733+0000", 2024-09-21T10:43:41.816 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00080146600000000003 2024-09-21T10:43:41.816 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.816 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:43:41.817 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.817 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.817 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.817 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77672 53.10 53:0f020cb9:::benchmark_data_smithi070_228967_object77671:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:43:41.817 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.923481+0000", 2024-09-21T10:43:41.817 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 155.68873076700001, 2024-09-21T10:43:41.817 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9159655249999998, 2024-09-21T10:43:41.817 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:43:41.817 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:43:41.817 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:41.817 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:43:41.817 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:43:41.818 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:41.818 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77672 2024-09-21T10:43:41.818 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.818 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:43:41.818 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.818 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:43:41.818 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.923481+0000", 2024-09-21T10:43:41.818 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.818 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.818 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.818 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:43:41.819 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.923481+0000", 2024-09-21T10:43:41.819 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.819 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.819 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.819 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:43:41.819 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.923483+0000", 2024-09-21T10:43:41.819 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.19e-06 2024-09-21T10:43:41.819 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.819 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.819 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:43:41.819 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.923502+0000", 2024-09-21T10:43:41.819 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9454e-05 2024-09-21T10:43:41.819 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.820 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.820 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:43:41.820 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.923512+0000", 2024-09-21T10:43:41.820 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0390999999999999e-05 2024-09-21T10:43:41.820 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.820 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.820 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:43:41.820 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.923518+0000", 2024-09-21T10:43:41.820 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2599999999999996e-06 2024-09-21T10:43:41.820 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.820 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.820 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.821 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.923561+0000", 2024-09-21T10:43:41.821 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3621999999999997e-05 2024-09-21T10:43:41.821 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.821 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.821 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.821 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.923568+0000", 2024-09-21T10:43:41.821 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1689999999999997e-06 2024-09-21T10:43:41.821 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.822 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.822 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.822 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.660046+0000", 2024-09-21T10:43:41.822 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.73647844299999998 2024-09-21T10:43:41.822 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.822 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.822 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.822 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.660050+0000", 2024-09-21T10:43:41.823 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8070000000000001e-06 2024-09-21T10:43:41.823 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.823 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.823 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.823 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.820493+0000", 2024-09-21T10:43:41.823 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.160443109 2024-09-21T10:43:41.823 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.823 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.823 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:43:41.824 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.820584+0000", 2024-09-21T10:43:41.824 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0705999999999995e-05 2024-09-21T10:43:41.824 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.824 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.824 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:43:41.824 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.820680+0000", 2024-09-21T10:43:41.824 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.6836999999999995e-05 2024-09-21T10:43:41.824 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.824 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.825 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:43:41.825 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.823091+0000", 2024-09-21T10:43:41.825 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0024103309999999999 2024-09-21T10:43:41.825 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.825 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.825 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.825 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.823139+0000", 2024-09-21T10:43:41.825 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7985999999999998e-05 2024-09-21T10:43:41.825 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.825 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.825 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.825 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.838802+0000", 2024-09-21T10:43:41.825 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015663567999999999 2024-09-21T10:43:41.825 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.826 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.826 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:43:41.826 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.838880+0000", 2024-09-21T10:43:41.826 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7302e-05 2024-09-21T10:43:41.826 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.826 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.826 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:43:41.826 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.839447+0000", 2024-09-21T10:43:41.826 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00056734999999999999 2024-09-21T10:43:41.826 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.826 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:43:41.826 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.826 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.827 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.827 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77683 53.10 53:0c041dae:::benchmark_data_smithi070_228967_object77682:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:43:41.827 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.950914+0000", 2024-09-21T10:43:41.827 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 155.66129839800001, 2024-09-21T10:43:41.827 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.889211167, 2024-09-21T10:43:41.827 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:43:41.827 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:43:41.827 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:41.827 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:43:41.827 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:43:41.827 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:41.827 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77683 2024-09-21T10:43:41.827 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.828 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:43:41.828 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.828 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:43:41.828 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.950914+0000", 2024-09-21T10:43:41.828 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.828 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.828 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.828 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:43:41.828 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.950914+0000", 2024-09-21T10:43:41.828 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.828 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.828 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.828 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:43:41.828 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.950915+0000", 2024-09-21T10:43:41.829 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.508e-06 2024-09-21T10:43:41.829 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.829 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.829 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:43:41.829 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.950958+0000", 2024-09-21T10:43:41.829 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3133999999999997e-05 2024-09-21T10:43:41.829 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.829 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.829 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:43:41.829 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.950974+0000", 2024-09-21T10:43:41.829 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5828000000000002e-05 2024-09-21T10:43:41.829 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.829 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.829 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:43:41.830 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.950981+0000", 2024-09-21T10:43:41.830 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1670000000000002e-06 2024-09-21T10:43:41.830 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.830 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.830 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.830 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.951050+0000", 2024-09-21T10:43:41.830 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8666999999999998e-05 2024-09-21T10:43:41.830 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.830 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.830 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.830 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.951062+0000", 2024-09-21T10:43:41.830 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2064e-05 2024-09-21T10:43:41.830 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.831 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.831 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.831 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.660067+0000", 2024-09-21T10:43:41.831 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.70900450999999998 2024-09-21T10:43:41.831 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.831 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.831 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.831 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.660071+0000", 2024-09-21T10:43:41.831 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8580000000000002e-06 2024-09-21T10:43:41.831 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.831 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.831 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.831 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.821230+0000", 2024-09-21T10:43:41.832 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1611593650000001 2024-09-21T10:43:41.832 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.832 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.832 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:43:41.832 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.821312+0000", 2024-09-21T10:43:41.832 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2452000000000003e-05 2024-09-21T10:43:41.832 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.832 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.832 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:43:41.832 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.821416+0000", 2024-09-21T10:43:41.832 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010381400000000001 2024-09-21T10:43:41.832 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.832 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.833 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:43:41.833 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.824269+0000", 2024-09-21T10:43:41.833 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0028524179999999998 2024-09-21T10:43:41.833 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.833 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.833 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.833 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.824301+0000", 2024-09-21T10:43:41.833 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2481999999999998e-05 2024-09-21T10:43:41.833 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.833 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.833 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.833 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.839511+0000", 2024-09-21T10:43:41.833 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015209926 2024-09-21T10:43:41.833 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.834 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.834 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:43:41.834 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.839581+0000", 2024-09-21T10:43:41.834 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9913000000000006e-05 2024-09-21T10:43:41.834 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.834 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.834 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:43:41.834 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.840125+0000", 2024-09-21T10:43:41.834 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00054406099999999998 2024-09-21T10:43:41.834 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.834 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:43:41.834 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.835 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.835 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.835 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:80122 53.12 53:4b0f308b:::benchmark_data_smithi070_228967_object80121:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1330)", 2024-09-21T10:43:41.835 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:10.695719+0000", 2024-09-21T10:43:41.835 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 150.91649347500001, 2024-09-21T10:43:41.835 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3313222339999999, 2024-09-21T10:43:41.835 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:43:41.835 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:43:41.835 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:41.835 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:43:41.835 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:43:41.835 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:41.835 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 80122 2024-09-21T10:43:41.835 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.836 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:43:41.836 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.836 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:43:41.836 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695719+0000", 2024-09-21T10:43:41.836 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.836 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.836 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.836 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:43:41.836 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695719+0000", 2024-09-21T10:43:41.836 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.836 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.836 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.836 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:43:41.837 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695719+0000", 2024-09-21T10:43:41.837 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.1100000000000001e-07 2024-09-21T10:43:41.837 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.837 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.837 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:43:41.837 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695737+0000", 2024-09-21T10:43:41.837 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7699000000000001e-05 2024-09-21T10:43:41.837 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.837 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.837 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:43:41.837 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695743+0000", 2024-09-21T10:43:41.837 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.596e-06 2024-09-21T10:43:41.838 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.838 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.838 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:43:41.838 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695748+0000", 2024-09-21T10:43:41.838 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9119999999999997e-06 2024-09-21T10:43:41.838 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.838 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.838 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.838 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.697233+0000", 2024-09-21T10:43:41.838 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001484948 2024-09-21T10:43:41.838 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.838 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.838 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.839 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.697240+0000", 2024-09-21T10:43:41.839 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9619999999999996e-06 2024-09-21T10:43:41.839 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.839 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.839 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.839 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.012737+0000", 2024-09-21T10:43:41.839 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3154967870000001 2024-09-21T10:43:41.839 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.839 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.839 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:43:41.839 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.012879+0000", 2024-09-21T10:43:41.839 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014228600000000001 2024-09-21T10:43:41.839 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.839 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.840 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,3,6", 2024-09-21T10:43:41.840 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.013130+0000", 2024-09-21T10:43:41.840 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000250237 2024-09-21T10:43:41.840 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.840 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.840 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.840 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.018827+0000", 2024-09-21T10:43:41.840 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0056977180000000001 2024-09-21T10:43:41.840 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.840 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.840 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:43:41.840 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.020553+0000", 2024-09-21T10:43:41.840 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0017259529999999999 2024-09-21T10:43:41.841 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.841 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.841 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.841 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.026098+0000", 2024-09-21T10:43:41.841 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0055445950000000003 2024-09-21T10:43:41.841 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.841 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.841 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.841 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.026157+0000", 2024-09-21T10:43:41.841 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9213999999999999e-05 2024-09-21T10:43:41.841 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.841 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.842 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:43:41.842 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.026212+0000", 2024-09-21T10:43:41.842 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5345e-05 2024-09-21T10:43:41.842 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.842 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.842 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:43:41.842 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.027041+0000", 2024-09-21T10:43:41.842 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00082857099999999995 2024-09-21T10:43:41.842 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.842 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:43:41.842 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.842 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.842 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.843 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:80155 53.12 53:4c70868e:::benchmark_data_smithi070_228967_object80154:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1330)", 2024-09-21T10:43:41.843 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:10.695776+0000", 2024-09-21T10:43:41.843 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 150.91643605100001, 2024-09-21T10:43:41.843 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3340382100000001, 2024-09-21T10:43:41.843 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:43:41.843 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:43:41.843 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:41.843 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:43:41.843 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:43:41.843 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:41.843 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 80155 2024-09-21T10:43:41.843 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.843 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:43:41.843 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.844 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:43:41.844 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695776+0000", 2024-09-21T10:43:41.844 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.844 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.844 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.844 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:43:41.844 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695776+0000", 2024-09-21T10:43:41.844 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.844 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.844 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.844 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:43:41.844 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695777+0000", 2024-09-21T10:43:41.844 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0900000000000002e-07 2024-09-21T10:43:41.844 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.845 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.845 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:43:41.845 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695795+0000", 2024-09-21T10:43:41.845 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8150000000000001e-05 2024-09-21T10:43:41.845 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.845 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.845 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:43:41.845 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695802+0000", 2024-09-21T10:43:41.845 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9750000000000001e-06 2024-09-21T10:43:41.845 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.845 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.845 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:43:41.845 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695807+0000", 2024-09-21T10:43:41.845 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1869999999999996e-06 2024-09-21T10:43:41.845 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.846 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.846 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.846 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.697268+0000", 2024-09-21T10:43:41.846 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001461237 2024-09-21T10:43:41.846 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.846 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.846 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.846 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.697275+0000", 2024-09-21T10:43:41.846 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0600000000000002e-06 2024-09-21T10:43:41.846 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.846 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.846 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.846 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.014876+0000", 2024-09-21T10:43:41.847 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3176006689999999 2024-09-21T10:43:41.847 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.847 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.847 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:43:41.847 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.015180+0000", 2024-09-21T10:43:41.847 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00030391000000000001 2024-09-21T10:43:41.847 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.847 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.847 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,3,6", 2024-09-21T10:43:41.847 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.015361+0000", 2024-09-21T10:43:41.847 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018097299999999999 2024-09-21T10:43:41.847 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.847 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.848 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.848 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.018976+0000", 2024-09-21T10:43:41.848 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0036151109999999998 2024-09-21T10:43:41.848 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.848 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.848 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:43:41.848 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.020561+0000", 2024-09-21T10:43:41.848 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0015851369999999999 2024-09-21T10:43:41.848 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.848 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.848 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.848 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.027092+0000", 2024-09-21T10:43:41.849 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0065314250000000004 2024-09-21T10:43:41.849 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.849 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.849 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.849 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.028432+0000", 2024-09-21T10:43:41.849 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00134005 2024-09-21T10:43:41.849 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.849 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.849 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:43:41.849 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.028542+0000", 2024-09-21T10:43:41.849 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000109574 2024-09-21T10:43:41.849 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.849 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.849 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:43:41.850 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.029814+0000", 2024-09-21T10:43:41.850 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001272343 2024-09-21T10:43:41.850 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.850 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:43:41.850 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.850 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.850 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.850 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:80160 53.12 53:4958c2ed:::benchmark_data_smithi070_228967_object80159:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1330)", 2024-09-21T10:43:41.850 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:10.695813+0000", 2024-09-21T10:43:41.850 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 150.916398867, 2024-09-21T10:43:41.850 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3537495580000001, 2024-09-21T10:43:41.850 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:43:41.851 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:43:41.851 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:41.851 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:43:41.851 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:43:41.851 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:41.851 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 80160 2024-09-21T10:43:41.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.851 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:43:41.851 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.851 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:43:41.851 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695813+0000", 2024-09-21T10:43:41.851 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.852 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.852 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:43:41.852 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695813+0000", 2024-09-21T10:43:41.852 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.852 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.852 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.852 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:43:41.852 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695814+0000", 2024-09-21T10:43:41.852 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7800000000000002e-07 2024-09-21T10:43:41.852 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.852 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.852 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:43:41.852 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695831+0000", 2024-09-21T10:43:41.853 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7311000000000001e-05 2024-09-21T10:43:41.853 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.853 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.853 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:43:41.853 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695846+0000", 2024-09-21T10:43:41.853 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.47e-05 2024-09-21T10:43:41.853 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.853 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.853 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:43:41.853 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695852+0000", 2024-09-21T10:43:41.853 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2190000000000001e-06 2024-09-21T10:43:41.853 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.853 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.854 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.854 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.697303+0000", 2024-09-21T10:43:41.854 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0014514840000000001 2024-09-21T10:43:41.854 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.854 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.854 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.854 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.697310+0000", 2024-09-21T10:43:41.854 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5019999999999998e-06 2024-09-21T10:43:41.854 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.854 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.854 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.854 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.029868+0000", 2024-09-21T10:43:41.855 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3325584930000001 2024-09-21T10:43:41.855 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.855 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.855 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:43:41.855 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.030023+0000", 2024-09-21T10:43:41.855 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015419899999999999 2024-09-21T10:43:41.855 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.855 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.855 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,3,6", 2024-09-21T10:43:41.855 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.030189+0000", 2024-09-21T10:43:41.855 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016664199999999999 2024-09-21T10:43:41.855 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.855 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.856 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.856 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.038428+0000", 2024-09-21T10:43:41.856 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0082384810000000006 2024-09-21T10:43:41.856 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.856 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.856 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.856 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.041162+0000", 2024-09-21T10:43:41.856 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0027339759999999999 2024-09-21T10:43:41.856 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.856 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.856 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:43:41.856 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.041175+0000", 2024-09-21T10:43:41.856 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3271000000000001e-05 2024-09-21T10:43:41.856 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.857 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.857 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.857 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.047349+0000", 2024-09-21T10:43:41.857 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0061744950000000003 2024-09-21T10:43:41.857 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.857 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.857 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:43:41.857 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.047430+0000", 2024-09-21T10:43:41.857 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0260999999999996e-05 2024-09-21T10:43:41.857 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.857 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.857 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:43:41.857 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.049563+0000", 2024-09-21T10:43:41.857 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0021331459999999998 2024-09-21T10:43:41.858 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.858 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:43:41.858 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.858 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.858 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.858 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:80214 53.12 53:4883ca5c:::benchmark_data_smithi070_228967_object80213:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1330)", 2024-09-21T10:43:41.858 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:10.695859+0000", 2024-09-21T10:43:41.858 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 150.916353657, 2024-09-21T10:43:41.858 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.355014194, 2024-09-21T10:43:41.858 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:43:41.858 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:43:41.858 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:41.859 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:43:41.859 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:43:41.859 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:43:41.859 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 80214 2024-09-21T10:43:41.859 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.859 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:43:41.859 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.859 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:43:41.859 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695859+0000", 2024-09-21T10:43:41.859 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.859 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.859 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.859 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:43:41.859 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695859+0000", 2024-09-21T10:43:41.860 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:43:41.860 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.860 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.860 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:43:41.860 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695859+0000", 2024-09-21T10:43:41.860 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3599999999999999e-07 2024-09-21T10:43:41.860 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.860 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.860 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:43:41.860 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695880+0000", 2024-09-21T10:43:41.860 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1478e-05 2024-09-21T10:43:41.860 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.860 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.861 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:43:41.861 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695921+0000", 2024-09-21T10:43:41.861 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0139999999999999e-05 2024-09-21T10:43:41.861 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.861 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.861 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:43:41.861 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695926+0000", 2024-09-21T10:43:41.861 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9039999999999997e-06 2024-09-21T10:43:41.861 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.861 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.861 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.861 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.698208+0000", 2024-09-21T10:43:41.861 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00228123 2024-09-21T10:43:41.861 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.862 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.862 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:43:41.862 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.698216+0000", 2024-09-21T10:43:41.862 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1529999999999997e-06 2024-09-21T10:43:41.862 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.862 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.862 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:43:41.862 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.031500+0000", 2024-09-21T10:43:41.862 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3332841609999999 2024-09-21T10:43:41.862 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.862 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.862 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:43:41.863 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.031666+0000", 2024-09-21T10:43:41.863 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000165515 2024-09-21T10:43:41.863 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.863 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.863 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,3,6", 2024-09-21T10:43:41.863 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.031885+0000", 2024-09-21T10:43:41.863 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00021924600000000001 2024-09-21T10:43:41.863 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.863 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.863 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.863 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.038468+0000", 2024-09-21T10:43:41.863 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0065835700000000004 2024-09-21T10:43:41.863 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.863 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.864 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:43:41.864 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.041181+0000", 2024-09-21T10:43:41.864 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0027126630000000001 2024-09-21T10:43:41.864 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.864 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.864 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.864 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.041240+0000", 2024-09-21T10:43:41.864 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8641999999999999e-05 2024-09-21T10:43:41.864 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.864 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.864 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:41.864 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.049667+0000", 2024-09-21T10:43:41.865 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0084272570000000005 2024-09-21T10:43:41.865 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.865 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.865 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:43:41.865 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.049759+0000", 2024-09-21T10:43:41.865 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2334000000000001e-05 2024-09-21T10:43:41.865 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:43:41.865 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:43:41.865 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:43:41.865 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.050873+0000", 2024-09-21T10:43:41.865 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0011134649999999999 2024-09-21T10:43:41.865 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.865 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:43:41.865 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.866 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:43:41.866 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:43:41.866 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:43:41.866 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:41.887 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:41.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:41.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:41.989 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:42.014 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:42.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:42.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:42.116 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:42.141 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:42.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:42.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:42.242 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:42.265 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:42.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:42.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:42.367 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:42.391 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:42.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:42.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:42.492 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:42.516 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:42.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:42.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:42 smithi160 ceph-mon[101879]: pgmap v2296: 99 pgs: 1 active+recovery_wait, 98 active+clean; 1.4 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 1.09k op/s; 0 B/s, 45 objects/s recovering 2024-09-21T10:43:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:42 smithi160 ceph-mon[101879]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 276/77352 objects degraded (0.357%), 1 pg degraded, 1 pg undersized) 2024-09-21T10:43:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:42 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "22"}]': finished 2024-09-21T10:43:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:42 smithi160 ceph-mon[101879]: osdmap e1353: 8 total, 8 up, 6 in 2024-09-21T10:43:42.617 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:42.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:42 smithi070 ceph-mon[107385]: pgmap v2296: 99 pgs: 1 active+recovery_wait, 98 active+clean; 1.4 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 1.09k op/s; 0 B/s, 45 objects/s recovering 2024-09-21T10:43:42.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:42 smithi070 ceph-mon[107385]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 276/77352 objects degraded (0.357%), 1 pg degraded, 1 pg undersized) 2024-09-21T10:43:42.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:42 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "22"}]': finished 2024-09-21T10:43:42.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:42 smithi070 ceph-mon[107385]: osdmap e1353: 8 total, 8 up, 6 in 2024-09-21T10:43:42.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:42 smithi070 ceph-mon[112139]: pgmap v2296: 99 pgs: 1 active+recovery_wait, 98 active+clean; 1.4 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 1.09k op/s; 0 B/s, 45 objects/s recovering 2024-09-21T10:43:42.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:42 smithi070 ceph-mon[112139]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 276/77352 objects degraded (0.357%), 1 pg degraded, 1 pg undersized) 2024-09-21T10:43:42.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:42 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "22"}]': finished 2024-09-21T10:43:42.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:42 smithi070 ceph-mon[112139]: osdmap e1353: 8 total, 8 up, 6 in 2024-09-21T10:43:42.640 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:42.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:42.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:42.742 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:42.764 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:42.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:42.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:42.865 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:42.891 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:42.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:42.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:42.993 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:43.014 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:43.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:43.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:43.116 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:43.137 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:43.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:43.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:43.238 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:43.261 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:43.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:43.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:43.362 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:43.384 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:43.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:43.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:43.485 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:43.523 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:43.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:43.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:43.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:43 smithi160 ceph-mon[101879]: osdmap e1354: 8 total, 8 up, 6 in 2024-09-21T10:43:43.625 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:43.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:43 smithi070 ceph-mon[107385]: osdmap e1354: 8 total, 8 up, 6 in 2024-09-21T10:43:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:43:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:43:43] "GET /metrics HTTP/1.1" 200 36601 "" "Prometheus/2.51.0" 2024-09-21T10:43:43.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:43 smithi070 ceph-mon[112139]: osdmap e1354: 8 total, 8 up, 6 in 2024-09-21T10:43:43.652 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:43.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:43.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:43.754 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:43.774 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:43.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:43.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:43.876 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:43.909 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:43.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:43.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:43.959 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:43:44.011 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:44.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:44.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:44.181 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:44 smithi070 ceph-mon[107385]: pgmap v2299: 99 pgs: 1 unknown, 98 active+clean; 1.2 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 0 B/s, 60 objects/s recovering 2024-09-21T10:43:44.181 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:44 smithi070 ceph-mon[107385]: osdmap e1355: 8 total, 8 up, 6 in 2024-09-21T10:43:44.181 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:44 smithi070 ceph-mon[107385]: 53.15 starting backfill to osd.4 from (0'0,0'0] MAX to 1352'9611 2024-09-21T10:43:44.181 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:44 smithi070 ceph-mon[107385]: 53.15 starting backfill to osd.6 from (0'0,0'0] MAX to 1352'9611 2024-09-21T10:43:44.191 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:44.237 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:44.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:44.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:44.339 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:44.372 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:44.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:44.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:44.474 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:44.508 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:44.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:44.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:44.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:44 smithi160 ceph-mon[101879]: pgmap v2299: 99 pgs: 1 unknown, 98 active+clean; 1.2 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 0 B/s, 60 objects/s recovering 2024-09-21T10:43:44.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:44 smithi160 ceph-mon[101879]: osdmap e1355: 8 total, 8 up, 6 in 2024-09-21T10:43:44.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:44 smithi160 ceph-mon[101879]: 53.15 starting backfill to osd.4 from (0'0,0'0] MAX to 1352'9611 2024-09-21T10:43:44.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:44 smithi160 ceph-mon[101879]: 53.15 starting backfill to osd.6 from (0'0,0'0] MAX to 1352'9611 2024-09-21T10:43:44.610 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:44.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:44 smithi070 ceph-mon[112139]: pgmap v2299: 99 pgs: 1 unknown, 98 active+clean; 1.2 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 0 B/s, 60 objects/s recovering 2024-09-21T10:43:44.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:44 smithi070 ceph-mon[112139]: osdmap e1355: 8 total, 8 up, 6 in 2024-09-21T10:43:44.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:44 smithi070 ceph-mon[112139]: 53.15 starting backfill to osd.4 from (0'0,0'0] MAX to 1352'9611 2024-09-21T10:43:44.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:44 smithi070 ceph-mon[112139]: 53.15 starting backfill to osd.6 from (0'0,0'0] MAX to 1352'9611 2024-09-21T10:43:44.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:44.648 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:44.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:44.749 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:44.782 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:44.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:44.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:44.884 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:44.912 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:44.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:44.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:44.939 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6] out_osds: [2, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:43:44.939 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:43:44.939 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T10:43:44.939 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 1.000000 2024-09-21T10:43:44.940 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 5 1 2024-09-21T10:43:45.013 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:45.049 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:45.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:45.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:45.150 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:45.181 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:45.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:45.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:45.271 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:45 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2252965429' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:43:45.272 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:45 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2252965429' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:43:45.283 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:45.314 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:45.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:45.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:45.415 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:45.453 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:45.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:45.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:45.555 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:45.585 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:45.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:45.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:45.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:45 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2252965429' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:43:45.687 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:45.723 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:45.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:45.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:45.825 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:45.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:45.864 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:45.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:45.965 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:45.967 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:43:45.967 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:43:45.968 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:43:45.968 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:43:45.968 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:43:45.968 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:43:45.968 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:43:45.968 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:43:45.968 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:43:45.968 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:43:45.968 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:43:45.968 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:43:45.968 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:43:45.968 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:43:45.969 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:43:45.969 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:43:45.994 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:45.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:45.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:46.095 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:46.125 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:46.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:46.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:46.227 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:46.266 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:46.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:46.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:46.367 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:46.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:46 smithi070 ceph-mon[107385]: pgmap v2301: 99 pgs: 1 unknown, 98 active+clean; 1.2 GiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 196 op/s; 0 B/s, 49 objects/s recovering 2024-09-21T10:43:46.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:46 smithi070 ceph-mon[112139]: pgmap v2301: 99 pgs: 1 unknown, 98 active+clean; 1.2 GiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 196 op/s; 0 B/s, 49 objects/s recovering 2024-09-21T10:43:46.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:46.405 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:46.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:46.506 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:46.557 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:46.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:46.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:46.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:46 smithi160 ceph-mon[101879]: pgmap v2301: 99 pgs: 1 unknown, 98 active+clean; 1.2 GiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 196 op/s; 0 B/s, 49 objects/s recovering 2024-09-21T10:43:46.658 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:46.698 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:46.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:46.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:46.800 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:46.831 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:46.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:46.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:46.932 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:46.957 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:46.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:46.984 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:46.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:47.085 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:47.133 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:47.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:47.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:47.190 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:47 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "23"} : dispatch 2024-09-21T10:43:47.235 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:47.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:47.272 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:47.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:47.373 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:47.397 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:47.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:47.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:47.499 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:47.521 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:47.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:47.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:47.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:47 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "23"} : dispatch 2024-09-21T10:43:47.623 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:47.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:47 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "23"} : dispatch 2024-09-21T10:43:47.646 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:47.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:47.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:47.747 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:47.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:47.769 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:47.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:47.870 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:47.891 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:47.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:47.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:47.967 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:47.993 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:47.999 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:48.014 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:48.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:48.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:48.116 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:48.136 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:48.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:48.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:48.238 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:48.260 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:48.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:48.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:48.349 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:48 smithi070 ceph-mon[107385]: pgmap v2302: 99 pgs: 1 active+remapped+backfilling, 98 active+clean; 1.1 GiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 895 op/s; 876/52818 objects misplaced (1.659%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:43:48.349 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:48 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "23"}]': finished 2024-09-21T10:43:48.349 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:48 smithi070 ceph-mon[107385]: osdmap e1356: 8 total, 8 up, 6 in 2024-09-21T10:43:48.349 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:48 smithi070 ceph-mon[107385]: osdmap e1357: 8 total, 8 up, 6 in 2024-09-21T10:43:48.349 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:48 smithi070 ceph-mon[112139]: pgmap v2302: 99 pgs: 1 active+remapped+backfilling, 98 active+clean; 1.1 GiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 895 op/s; 876/52818 objects misplaced (1.659%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:43:48.350 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:48 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "23"}]': finished 2024-09-21T10:43:48.350 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:48 smithi070 ceph-mon[112139]: osdmap e1356: 8 total, 8 up, 6 in 2024-09-21T10:43:48.350 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:48 smithi070 ceph-mon[112139]: osdmap e1357: 8 total, 8 up, 6 in 2024-09-21T10:43:48.362 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:48.383 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:48.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:48.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:48.485 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:48.506 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:48.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:48.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:48 smithi160 ceph-mon[101879]: pgmap v2302: 99 pgs: 1 active+remapped+backfilling, 98 active+clean; 1.1 GiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 895 op/s; 876/52818 objects misplaced (1.659%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:43:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:48 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "23"}]': finished 2024-09-21T10:43:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:48 smithi160 ceph-mon[101879]: osdmap e1356: 8 total, 8 up, 6 in 2024-09-21T10:43:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:48 smithi160 ceph-mon[101879]: osdmap e1357: 8 total, 8 up, 6 in 2024-09-21T10:43:48.607 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:48.628 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:48.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:48.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:48.730 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:48.750 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:48.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:48.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:48.852 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:48.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:48.874 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:48.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:48.961 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:43:48.975 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:48.996 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:48.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:48.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:49.097 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:49.118 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:49.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:49.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:49.220 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:49.243 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:49.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:49.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:49.345 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:49.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:49 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "24"} : dispatch 2024-09-21T10:43:49.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:49 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "24"}]': finished 2024-09-21T10:43:49.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:49 smithi070 ceph-mon[112139]: osdmap e1358: 8 total, 8 up, 6 in 2024-09-21T10:43:49.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:49 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "24"} : dispatch 2024-09-21T10:43:49.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:49 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "24"}]': finished 2024-09-21T10:43:49.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:49 smithi070 ceph-mon[107385]: osdmap e1358: 8 total, 8 up, 6 in 2024-09-21T10:43:49.382 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:49.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:49.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:49.483 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:49.513 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:49.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:49.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:49.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:49 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "24"} : dispatch 2024-09-21T10:43:49.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:49 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "24"}]': finished 2024-09-21T10:43:49.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:49 smithi160 ceph-mon[101879]: osdmap e1358: 8 total, 8 up, 6 in 2024-09-21T10:43:49.614 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:49.647 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:49.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:49.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:49.749 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:49.773 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:49.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:49.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:49.875 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:49.897 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:49.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:49.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:49.999 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:50.020 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:50.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:50.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:50.121 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:50.144 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:50.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:50.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:50.245 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:50.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:50.267 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:50.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:50.368 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:50.388 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:50.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:50.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:50.490 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:50.511 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:50.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:50.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:50 smithi160 ceph-mon[101879]: pgmap v2305: 99 pgs: 1 active+remapped+backfilling, 98 active+clean; 1.1 GiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 17 MiB/s wr, 1.36k op/s; 876/52818 objects misplaced (1.659%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:43:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:50 smithi160 ceph-mon[101879]: 53.16 starting backfill to osd.6 from (0'0,0'0] MAX to 1355'9716 2024-09-21T10:43:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:50 smithi160 ceph-mon[101879]: osdmap e1359: 8 total, 8 up, 6 in 2024-09-21T10:43:50.612 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:50 smithi070 ceph-mon[107385]: pgmap v2305: 99 pgs: 1 active+remapped+backfilling, 98 active+clean; 1.1 GiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 17 MiB/s wr, 1.36k op/s; 876/52818 objects misplaced (1.659%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:43:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:50 smithi070 ceph-mon[107385]: 53.16 starting backfill to osd.6 from (0'0,0'0] MAX to 1355'9716 2024-09-21T10:43:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:50 smithi070 ceph-mon[107385]: osdmap e1359: 8 total, 8 up, 6 in 2024-09-21T10:43:50.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:50 smithi070 ceph-mon[112139]: pgmap v2305: 99 pgs: 1 active+remapped+backfilling, 98 active+clean; 1.1 GiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 17 MiB/s wr, 1.36k op/s; 876/52818 objects misplaced (1.659%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:43:50.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:50 smithi070 ceph-mon[112139]: 53.16 starting backfill to osd.6 from (0'0,0'0] MAX to 1355'9716 2024-09-21T10:43:50.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:50 smithi070 ceph-mon[112139]: osdmap e1359: 8 total, 8 up, 6 in 2024-09-21T10:43:50.633 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:50.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:50.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:50.735 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:50.756 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:50.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:50.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:50.858 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:50.880 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:50.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:50.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:50.981 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:51.002 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:51.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:51.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:51.103 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:51.124 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:51.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:51.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:51.166 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T10:43:51.226 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:51.250 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:51.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:51.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:51.352 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:51.389 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:51.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:51.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:51.491 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:51.516 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:51.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:51.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:51.618 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:43:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:43:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "25"} : dispatch 2024-09-21T10:43:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:51 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3742436944' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:43:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:51 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:43:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:51 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:43:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "25"}]': finished 2024-09-21T10:43:51.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:51 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:43:51.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:51 smithi070 ceph-mon[107385]: osdmap e1360: 8 total, 8 up, 6 in 2024-09-21T10:43:51.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:51 smithi070 ceph-mon[107385]: 53.17 starting backfill to osd.4 from (0'0,0'0] MAX to 1356'9925 2024-09-21T10:43:51.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:51 smithi070 ceph-mon[107385]: 53.17 starting backfill to osd.5 from (0'0,0'0] MAX to 1356'9925 2024-09-21T10:43:51.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:51 smithi070 ceph-mon[107385]: 53.17 starting backfill to osd.6 from (0'0,0'0] MAX to 1356'9925 2024-09-21T10:43:51.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:43:51.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:43:51.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "25"} : dispatch 2024-09-21T10:43:51.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:51 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3742436944' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:43:51.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:51 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:43:51.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:51 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:43:51.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "25"}]': finished 2024-09-21T10:43:51.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:51 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:43:51.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:51 smithi070 ceph-mon[112139]: osdmap e1360: 8 total, 8 up, 6 in 2024-09-21T10:43:51.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:51 smithi070 ceph-mon[112139]: 53.17 starting backfill to osd.4 from (0'0,0'0] MAX to 1356'9925 2024-09-21T10:43:51.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:51 smithi070 ceph-mon[112139]: 53.17 starting backfill to osd.5 from (0'0,0'0] MAX to 1356'9925 2024-09-21T10:43:51.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:51 smithi070 ceph-mon[112139]: 53.17 starting backfill to osd.6 from (0'0,0'0] MAX to 1356'9925 2024-09-21T10:43:51.641 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:51.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:51.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:51.742 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:51.766 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:51.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:51.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:43:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:43:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "25"} : dispatch 2024-09-21T10:43:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:51 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3742436944' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:43:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:51 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:43:51.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:51 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:43:51.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "25"}]': finished 2024-09-21T10:43:51.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:51 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:43:51.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:51 smithi160 ceph-mon[101879]: osdmap e1360: 8 total, 8 up, 6 in 2024-09-21T10:43:51.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:51 smithi160 ceph-mon[101879]: 53.17 starting backfill to osd.4 from (0'0,0'0] MAX to 1356'9925 2024-09-21T10:43:51.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:51 smithi160 ceph-mon[101879]: 53.17 starting backfill to osd.5 from (0'0,0'0] MAX to 1356'9925 2024-09-21T10:43:51.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:51 smithi160 ceph-mon[101879]: 53.17 starting backfill to osd.6 from (0'0,0'0] MAX to 1356'9925 2024-09-21T10:43:51.867 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:51.891 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:51.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:51.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:51.993 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:52.015 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:52.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:52.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:52.117 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:52.141 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:52.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:52.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:52.243 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:52.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:52.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:52.374 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:52.395 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:52.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:52.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:52.497 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:52.501 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:52 smithi070 ceph-mon[112139]: pgmap v2308: 99 pgs: 2 active+remapped+backfilling, 97 active+clean; 986 MiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 451 op/s; 1380/47397 objects misplaced (2.912%); 80 KiB/s, 1 objects/s recovering 2024-09-21T10:43:52.501 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:52 smithi070 ceph-mon[112139]: osdmap e1361: 8 total, 8 up, 6 in 2024-09-21T10:43:52.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:52 smithi070 ceph-mon[107385]: pgmap v2308: 99 pgs: 2 active+remapped+backfilling, 97 active+clean; 986 MiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 451 op/s; 1380/47397 objects misplaced (2.912%); 80 KiB/s, 1 objects/s recovering 2024-09-21T10:43:52.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:52 smithi070 ceph-mon[107385]: osdmap e1361: 8 total, 8 up, 6 in 2024-09-21T10:43:52.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:52.518 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:52.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:52.619 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:52.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:52.641 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:52.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:52.643 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:43:52.643 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:43:52.742 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:52.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:52 smithi160 ceph-mon[101879]: pgmap v2308: 99 pgs: 2 active+remapped+backfilling, 97 active+clean; 986 MiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 451 op/s; 1380/47397 objects misplaced (2.912%); 80 KiB/s, 1 objects/s recovering 2024-09-21T10:43:52.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:52 smithi160 ceph-mon[101879]: osdmap e1361: 8 total, 8 up, 6 in 2024-09-21T10:43:52.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:52.764 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:52.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:52.864 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:52.885 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:52.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:52.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:52.987 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:53.008 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:53.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:53.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:53.110 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:53.131 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:53.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:53.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:53.233 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:53.254 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:53.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:53.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:53.356 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:53.389 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:53.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:53.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:53.490 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:53.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:53.526 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:53.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:53.627 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:53.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:43:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:43:53] "GET /metrics HTTP/1.1" 200 36617 "" "Prometheus/2.51.0" 2024-09-21T10:43:53.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:53 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-21T10:43:53.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:53 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-21T10:43:53.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:53 smithi070 ceph-mon[107385]: osdmap e1362: 8 total, 8 up, 6 in 2024-09-21T10:43:53.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:53 smithi070 ceph-mon[107385]: 53.18 starting backfill to osd.3 from (0'0,0'0] MAX to 1358'9733 2024-09-21T10:43:53.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:53 smithi070 ceph-mon[107385]: 53.18 starting backfill to osd.5 from (0'0,0'0] MAX to 1358'9733 2024-09-21T10:43:53.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:53 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-21T10:43:53.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:53 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-21T10:43:53.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:53 smithi070 ceph-mon[112139]: osdmap e1362: 8 total, 8 up, 6 in 2024-09-21T10:43:53.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:53 smithi070 ceph-mon[112139]: 53.18 starting backfill to osd.3 from (0'0,0'0] MAX to 1358'9733 2024-09-21T10:43:53.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:53 smithi070 ceph-mon[112139]: 53.18 starting backfill to osd.5 from (0'0,0'0] MAX to 1358'9733 2024-09-21T10:43:53.656 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:53.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:53.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:53.758 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:53.784 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:53.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:53.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:53 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-21T10:43:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:53 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-21T10:43:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:53 smithi160 ceph-mon[101879]: osdmap e1362: 8 total, 8 up, 6 in 2024-09-21T10:43:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:53 smithi160 ceph-mon[101879]: 53.18 starting backfill to osd.3 from (0'0,0'0] MAX to 1358'9733 2024-09-21T10:43:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:53 smithi160 ceph-mon[101879]: 53.18 starting backfill to osd.5 from (0'0,0'0] MAX to 1358'9733 2024-09-21T10:43:53.885 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:53.906 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:53.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:53.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:54.008 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:54.030 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:54.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:54.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:54.132 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:54.161 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:54.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:54.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:54.262 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:54.289 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:54.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:54.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:54.391 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:54.405 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:54 smithi070 ceph-mon[107385]: pgmap v2311: 99 pgs: 1 active+remapped+backfilling, 98 active+clean; 925 MiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 697 op/s; 504/44451 objects misplaced (1.134%); 6.4 MiB/s, 102 objects/s recovering 2024-09-21T10:43:54.406 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:54 smithi070 ceph-mon[112139]: pgmap v2311: 99 pgs: 1 active+remapped+backfilling, 98 active+clean; 925 MiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 697 op/s; 504/44451 objects misplaced (1.134%); 6.4 MiB/s, 102 objects/s recovering 2024-09-21T10:43:54.416 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:54.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:54.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:54.518 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:54.522 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:43:54.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:54.540 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:54.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:54 smithi160 ceph-mon[101879]: pgmap v2311: 99 pgs: 1 active+remapped+backfilling, 98 active+clean; 925 MiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 697 op/s; 504/44451 objects misplaced (1.134%); 6.4 MiB/s, 102 objects/s recovering 2024-09-21T10:43:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:54 smithi160 ceph-mon[101879]: osdmap e1363: 8 total, 8 up, 6 in 2024-09-21T10:43:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:54 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/180749239' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:43:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:54 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:43:54.641 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:54.662 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:54.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:54.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:54.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:54 smithi070 ceph-mon[112139]: osdmap e1363: 8 total, 8 up, 6 in 2024-09-21T10:43:54.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:54 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/180749239' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:43:54.705 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:54 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:43:54.705 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:54 smithi070 ceph-mon[107385]: osdmap e1363: 8 total, 8 up, 6 in 2024-09-21T10:43:54.705 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:54 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/180749239' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:43:54.705 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:54 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:43:54.763 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:54.785 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:54.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:54.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:54.807 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:54.887 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:54.908 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:54.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:54.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:55.010 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:55.032 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:55.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:55.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:55.134 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:55.158 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:55.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:55.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:55.165 INFO:teuthology.orchestra.run.smithi070.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-21T10:43:55.224 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:43:55.259 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:55.281 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:55.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:55.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:55.382 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:55.410 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:55.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:55.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:55.512 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:55.546 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:55.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:55.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:55.650 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:55.676 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:55.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:55.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:55.778 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:55.804 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:55.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:55.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:55.901 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:55.906 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:55.927 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:55.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:55.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:56.029 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:56.051 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:56.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:56.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:56.152 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:56.167 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:56 smithi070 ceph-mon[107385]: pgmap v2314: 99 pgs: 99 active+clean; 902 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 335 op/s; 14 MiB/s, 216 objects/s recovering 2024-09-21T10:43:56.167 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:56 smithi070 ceph-mon[112139]: pgmap v2314: 99 pgs: 99 active+clean; 902 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 335 op/s; 14 MiB/s, 216 objects/s recovering 2024-09-21T10:43:56.167 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:56 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:43:56.167 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:56 smithi070 ceph-mon[112139]: 53.19 starting backfill to osd.1 from (0'0,0'0] MAX to 1361'10027 2024-09-21T10:43:56.168 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:56 smithi070 ceph-mon[112139]: osdmap e1364: 8 total, 8 up, 6 in 2024-09-21T10:43:56.168 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:56 smithi070 ceph-mon[112139]: 53.17 scrub starts 2024-09-21T10:43:56.168 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:56 smithi070 ceph-mon[112139]: 53.17 scrub ok 2024-09-21T10:43:56.181 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:56.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:56.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:56.282 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:56.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:56.318 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:56.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:56.419 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:56.433 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:56 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:43:56.433 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:56 smithi070 ceph-mon[107385]: 53.19 starting backfill to osd.1 from (0'0,0'0] MAX to 1361'10027 2024-09-21T10:43:56.433 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:56 smithi070 ceph-mon[107385]: osdmap e1364: 8 total, 8 up, 6 in 2024-09-21T10:43:56.433 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:56 smithi070 ceph-mon[107385]: 53.17 scrub starts 2024-09-21T10:43:56.434 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:56 smithi070 ceph-mon[107385]: 53.17 scrub ok 2024-09-21T10:43:56.447 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:56.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:56.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:56.549 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:56.583 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:56.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:56.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:56.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:56 smithi160 ceph-mon[101879]: pgmap v2314: 99 pgs: 99 active+clean; 902 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 335 op/s; 14 MiB/s, 216 objects/s recovering 2024-09-21T10:43:56.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:56 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:43:56.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:56 smithi160 ceph-mon[101879]: 53.19 starting backfill to osd.1 from (0'0,0'0] MAX to 1361'10027 2024-09-21T10:43:56.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:56 smithi160 ceph-mon[101879]: osdmap e1364: 8 total, 8 up, 6 in 2024-09-21T10:43:56.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:56 smithi160 ceph-mon[101879]: 53.17 scrub starts 2024-09-21T10:43:56.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:56 smithi160 ceph-mon[101879]: 53.17 scrub ok 2024-09-21T10:43:56.635 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:56.685 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:56.718 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:56.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:56.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:56.819 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:56.856 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:56.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:56.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:56.958 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:56.994 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:56.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:56.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:57.095 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:57.128 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:57.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:57.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:57.230 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:57.262 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:57.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:57.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:57.364 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:57.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:57 smithi070 ceph-mon[107385]: 53.5 scrub starts 2024-09-21T10:43:57.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:57 smithi070 ceph-mon[107385]: 53.5 scrub ok 2024-09-21T10:43:57.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:57 smithi070 ceph-mon[107385]: osdmap e1365: 8 total, 8 up, 6 in 2024-09-21T10:43:57.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:57 smithi070 ceph-mon[112139]: 53.5 scrub starts 2024-09-21T10:43:57.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:57 smithi070 ceph-mon[112139]: 53.5 scrub ok 2024-09-21T10:43:57.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:57 smithi070 ceph-mon[112139]: osdmap e1365: 8 total, 8 up, 6 in 2024-09-21T10:43:57.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:57.394 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:57.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:57.495 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:57.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:57.523 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:57.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:57.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:57 smithi160 ceph-mon[101879]: 53.5 scrub starts 2024-09-21T10:43:57.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:57 smithi160 ceph-mon[101879]: 53.5 scrub ok 2024-09-21T10:43:57.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:57 smithi160 ceph-mon[101879]: osdmap e1365: 8 total, 8 up, 6 in 2024-09-21T10:43:57.624 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:57.646 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:57.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:57.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:57.747 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:57.768 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:57.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:57.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:57.869 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:57.891 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:57.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:57.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:57.992 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:58.014 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:58.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:58.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:58.116 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:58.138 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:58.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:58.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:58.239 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:58.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:58.277 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:58.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:58.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:58 smithi070 ceph-mon[107385]: pgmap v2317: 99 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 97 active+clean; 839 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 4/40341 objects misplaced (0.010%); 20 MiB/s, 325 objects/s recovering 2024-09-21T10:43:58.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:58 smithi070 ceph-mon[107385]: osdmap e1366: 8 total, 8 up, 6 in 2024-09-21T10:43:58.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:58 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:43:58.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:58 smithi070 ceph-mon[112139]: pgmap v2317: 99 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 97 active+clean; 839 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 4/40341 objects misplaced (0.010%); 20 MiB/s, 325 objects/s recovering 2024-09-21T10:43:58.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:58 smithi070 ceph-mon[112139]: osdmap e1366: 8 total, 8 up, 6 in 2024-09-21T10:43:58.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:58 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:43:58.378 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:58.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:58.425 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:58.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:58.452 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:58 smithi160 ceph-mon[101879]: pgmap v2317: 99 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 97 active+clean; 839 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 4/40341 objects misplaced (0.010%); 20 MiB/s, 325 objects/s recovering 2024-09-21T10:43:58.452 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:58 smithi160 ceph-mon[101879]: osdmap e1366: 8 total, 8 up, 6 in 2024-09-21T10:43:58.452 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:58 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:43:58.526 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:58.566 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:58.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:58.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:58.668 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:58.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:58.699 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:58.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:58.800 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:58.846 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:58.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:58.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:58.948 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:58.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:58.985 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:58.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:59.086 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:59.126 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:59.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:59.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:59.227 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:59.264 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:59.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:59.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:59.320 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:43:59.320 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:43:59.321 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:59 smithi070 ceph-mon[107385]: osdmap e1367: 8 total, 8 up, 6 in 2024-09-21T10:43:59.321 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:59 smithi070 ceph-mon[107385]: 53.19 scrub starts 2024-09-21T10:43:59.321 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:43:59 smithi070 ceph-mon[107385]: 53.19 scrub ok 2024-09-21T10:43:59.321 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:59 smithi070 ceph-mon[112139]: osdmap e1367: 8 total, 8 up, 6 in 2024-09-21T10:43:59.321 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:59 smithi070 ceph-mon[112139]: 53.19 scrub starts 2024-09-21T10:43:59.321 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:43:59 smithi070 ceph-mon[112139]: 53.19 scrub ok 2024-09-21T10:43:59.356 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:43:59.399 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:43:59.400 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:43:59.400 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:43:59.400 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:59.414 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:43:59.415 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:43:59.432 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:59.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:59.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:59.447 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:43:59.447 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:43:59.477 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:43:59.477 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:43:59.504 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:59 smithi160 ceph-mon[101879]: osdmap e1367: 8 total, 8 up, 6 in 2024-09-21T10:43:59.505 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:59 smithi160 ceph-mon[101879]: 53.19 scrub starts 2024-09-21T10:43:59.505 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:43:59 smithi160 ceph-mon[101879]: 53.19 scrub ok 2024-09-21T10:43:59.515 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:43:59.515 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:43:59.534 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:59.550 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:43:59.551 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:43:59.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:59.583 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:43:59.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:59.684 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:59.725 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:59.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:59.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:59.827 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:59.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:59.873 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:43:59.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:59.974 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:00.007 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:00.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:00.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:00.108 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:00.137 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:00.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:00.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:00.239 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:00.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:00.271 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:00.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:00.372 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:00.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:00.405 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:00.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:00.461 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:00 smithi070 ceph-mon[107385]: pgmap v2320: 99 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 97 active+clean; 839 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 4/40341 objects misplaced (0.010%); 13 MiB/s, 210 objects/s recovering 2024-09-21T10:44:00.461 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:00 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1221473712' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:00.461 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:00 smithi070 ceph-mon[107385]: 53.16 scrub starts 2024-09-21T10:44:00.461 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:00 smithi070 ceph-mon[107385]: 53.16 scrub ok 2024-09-21T10:44:00.461 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:00 smithi070 ceph-mon[112139]: pgmap v2320: 99 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 97 active+clean; 839 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 4/40341 objects misplaced (0.010%); 13 MiB/s, 210 objects/s recovering 2024-09-21T10:44:00.461 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:00 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1221473712' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:00.461 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:00 smithi070 ceph-mon[112139]: 53.16 scrub starts 2024-09-21T10:44:00.461 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:00 smithi070 ceph-mon[112139]: 53.16 scrub ok 2024-09-21T10:44:00.506 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:00.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:00.545 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:00.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:00.551 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:00 smithi160 ceph-mon[101879]: pgmap v2320: 99 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 97 active+clean; 839 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 4/40341 objects misplaced (0.010%); 13 MiB/s, 210 objects/s recovering 2024-09-21T10:44:00.551 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:00 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1221473712' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:00.551 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:00 smithi160 ceph-mon[101879]: 53.16 scrub starts 2024-09-21T10:44:00.551 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:00 smithi160 ceph-mon[101879]: 53.16 scrub ok 2024-09-21T10:44:00.646 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:00.682 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:00.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:00.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:00.784 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:00.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:00.823 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:00.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:00.903 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6] out_osds: [2, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:44:00.903 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:44:00.904 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-09-21T10:44:00.904 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd in 7 2024-09-21T10:44:00.924 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:00.988 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:00.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:00.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:01.089 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:01.113 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:01.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:01.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:01.214 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:01.247 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:01.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:01.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:01.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:01 smithi070 ceph-mon[107385]: 53.11 scrub starts 2024-09-21T10:44:01.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:01 smithi070 ceph-mon[107385]: 53.11 scrub ok 2024-09-21T10:44:01.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:01 smithi070 ceph-mon[112139]: 53.11 scrub starts 2024-09-21T10:44:01.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:01 smithi070 ceph-mon[112139]: 53.11 scrub ok 2024-09-21T10:44:01.349 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:01.377 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:01.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:01.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:01.440 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:44:01.440 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T10:44:01.441 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T10:44:01.441 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T10:44:01.441 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:44:01.479 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:01.512 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:01.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:01.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:01.571 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:01 smithi160 ceph-mon[101879]: 53.11 scrub starts 2024-09-21T10:44:01.571 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:01 smithi160 ceph-mon[101879]: 53.11 scrub ok 2024-09-21T10:44:01.613 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:01.650 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:01.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:01.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:01.752 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:01.787 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:01.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:01.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:01.888 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:01.927 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:01.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:01.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:02.028 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:02.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:02.065 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:02.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:02.166 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:02.191 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:02.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:02.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:02.208 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T10:44:02.293 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:02.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:02.331 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:02.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:02.432 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:02.478 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:02.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:02.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:02.576 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:02 smithi070 ceph-mon[107385]: 53.9 scrub starts 2024-09-21T10:44:02.576 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:02 smithi070 ceph-mon[107385]: 53.9 scrub ok 2024-09-21T10:44:02.576 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:02 smithi070 ceph-mon[107385]: pgmap v2321: 99 pgs: 1 active+remapped+backfilling, 98 active+clean; 716 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 4/34422 objects misplaced (0.012%); 4.4 MiB/s, 70 objects/s recovering 2024-09-21T10:44:02.576 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:02 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/150662674' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:44:02.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:02 smithi070 ceph-mon[112139]: 53.9 scrub starts 2024-09-21T10:44:02.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:02 smithi070 ceph-mon[112139]: 53.9 scrub ok 2024-09-21T10:44:02.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:02 smithi070 ceph-mon[112139]: pgmap v2321: 99 pgs: 1 active+remapped+backfilling, 98 active+clean; 716 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 4/34422 objects misplaced (0.012%); 4.4 MiB/s, 70 objects/s recovering 2024-09-21T10:44:02.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:02 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/150662674' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:44:02.580 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:02 smithi160 ceph-mon[101879]: 53.9 scrub starts 2024-09-21T10:44:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:02 smithi160 ceph-mon[101879]: 53.9 scrub ok 2024-09-21T10:44:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:02 smithi160 ceph-mon[101879]: pgmap v2321: 99 pgs: 1 active+remapped+backfilling, 98 active+clean; 716 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 4/34422 objects misplaced (0.012%); 4.4 MiB/s, 70 objects/s recovering 2024-09-21T10:44:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:02 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/150662674' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:44:02.613 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:02.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:02.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:02.714 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:02.750 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:02.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:02.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:02.852 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:02.885 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:02.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:02.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:02.986 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:03.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:03.014 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:03.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:03.115 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:03.135 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:03.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:03.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:03.237 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:03.272 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:03.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:03.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:03.338 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:44:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:44:03] "GET /metrics HTTP/1.1" 200 36617 "" "Prometheus/2.51.0" 2024-09-21T10:44:03.338 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:03 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:44:03.338 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:03 smithi070 ceph-mon[107385]: Cluster is now healthy 2024-09-21T10:44:03.338 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:03 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/150662674' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:44:03.339 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:03 smithi070 ceph-mon[107385]: osdmap e1368: 8 total, 8 up, 6 in 2024-09-21T10:44:03.339 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:03 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:44:03.339 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:03 smithi070 ceph-mon[112139]: Cluster is now healthy 2024-09-21T10:44:03.339 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:03 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/150662674' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:44:03.339 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:03 smithi070 ceph-mon[112139]: osdmap e1368: 8 total, 8 up, 6 in 2024-09-21T10:44:03.374 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:03.407 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:03.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:03.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:03.508 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:03.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:03.542 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:03.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:03.575 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:03 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:44:03.575 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:03 smithi160 ceph-mon[101879]: Cluster is now healthy 2024-09-21T10:44:03.575 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:03 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/150662674' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:44:03.575 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:03 smithi160 ceph-mon[101879]: osdmap e1368: 8 total, 8 up, 6 in 2024-09-21T10:44:03.644 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:03.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:03.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:03.776 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:03.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:03.807 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:03.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:03.907 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:03.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:03.945 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:03.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:04.046 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:04.067 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:44:04.098 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:04.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:04.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:04.168 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:04.186 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:44:04.200 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:04.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:04.286 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:04.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:04.319 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:04 smithi070 ceph-mon[107385]: 53.10 scrub starts 2024-09-21T10:44:04.320 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:04 smithi070 ceph-mon[107385]: 53.10 scrub ok 2024-09-21T10:44:04.320 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:04 smithi070 ceph-mon[107385]: pgmap v2323: 99 pgs: 99 active+clean; 564 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:44:04.320 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:04 smithi070 ceph-mon[112139]: 53.10 scrub starts 2024-09-21T10:44:04.320 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:04 smithi070 ceph-mon[112139]: 53.10 scrub ok 2024-09-21T10:44:04.320 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:04 smithi070 ceph-mon[112139]: pgmap v2323: 99 pgs: 99 active+clean; 564 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:44:04.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:04 smithi160 ceph-mon[101879]: 53.10 scrub starts 2024-09-21T10:44:04.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:04 smithi160 ceph-mon[101879]: 53.10 scrub ok 2024-09-21T10:44:04.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:04 smithi160 ceph-mon[101879]: pgmap v2323: 99 pgs: 99 active+clean; 564 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:44:04.387 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:04.419 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:04.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:04.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:04.521 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:04.589 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:04.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:04.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:04.690 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:04.728 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:04.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:04.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:04.830 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:04.866 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:04.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:04.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:04.968 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:04.995 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:04.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:04.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:05.056 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:44:05.097 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:05.124 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:05.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:05.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:05.226 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:05.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:05.255 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:05.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:05.356 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:05.400 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:05.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:05.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:05.502 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:05.534 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:05.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:05.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:05.636 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:05.669 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:05.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:05.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:05.771 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:05.808 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:05.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:05.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:05.909 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:05.964 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:05.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:05.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:06.066 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:06.107 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:06.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:06.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:06.134 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:06.139 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:06.208 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:06.237 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:06.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:06.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:06.339 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:06.383 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:06.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:06.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:06.485 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:06.522 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:06.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:06.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:06.624 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:06 smithi070 ceph-mon[107385]: pgmap v2324: 99 pgs: 99 active+clean; 451 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail 2024-09-21T10:44:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:44:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:44:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:44:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:44:06.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:06 smithi070 ceph-mon[112139]: pgmap v2324: 99 pgs: 99 active+clean; 451 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail 2024-09-21T10:44:06.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:44:06.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:44:06.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:44:06.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:44:06.660 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:06.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:06.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:06.762 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:06.795 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:06.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:06.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:06 smithi160 ceph-mon[101879]: pgmap v2324: 99 pgs: 99 active+clean; 451 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail 2024-09-21T10:44:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:44:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:44:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:44:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:44:06.897 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:06.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:06.932 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:06.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:07.033 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:07.081 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:07.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:07.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:07.182 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:07.219 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:07.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:07.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:07.321 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:07.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:07.361 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:07.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:07.462 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:07.502 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:07.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:07.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:07.604 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:07.635 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:07.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:07.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:07.736 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:07.768 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:07.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:07.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:07.870 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:07.904 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:07.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:07.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:08.005 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:08.047 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:08.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:08.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:08.148 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:08.184 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:08.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:08.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:08.193 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Removed 128917 objects 2024-09-21T10:44:08.194 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Clean up completed and total clean up time :133.06 2024-09-21T10:44:08.286 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:08.307 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:08.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:08.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:08.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:08 smithi070 ceph-mon[107385]: pgmap v2325: 99 pgs: 99 active+clean; 214 MiB data, 3.0 GiB used, 533 GiB / 536 GiB avail 2024-09-21T10:44:08.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:08 smithi070 ceph-mon[112139]: pgmap v2325: 99 pgs: 99 active+clean; 214 MiB data, 3.0 GiB used, 533 GiB / 536 GiB avail 2024-09-21T10:44:08.409 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:08.431 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:08.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:08.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:08.533 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:08.555 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:08.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:08.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:08.657 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:08.680 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:08.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:08.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:08.781 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:08.803 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:08.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:08.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:08 smithi160 ceph-mon[101879]: pgmap v2325: 99 pgs: 99 active+clean; 214 MiB data, 3.0 GiB used, 533 GiB / 536 GiB avail 2024-09-21T10:44:08.905 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:08.927 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:08.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:08.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:08.979 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:09.029 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:09.050 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:09.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:09.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:09.152 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:09.174 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:09.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:09.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:09.276 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:09.299 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:09.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:09.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:09.401 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:09.422 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:09.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:09.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:09.523 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:09.544 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:09.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:09.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:09.645 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:09.667 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:09.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:09.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:09.768 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:09.790 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:09.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:09.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:09.804 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-21T10:44:09.805 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-21T10:44:09.892 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:09.913 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:09.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:09.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:10.015 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:10.036 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:10.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:10.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:10.137 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:10.159 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:10.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:10.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:10.260 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:10.286 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:10.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:10.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:10.388 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:10.413 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:10.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:10.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:10.515 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:10.540 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:10.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:10.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:10.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:10 smithi070 ceph-mon[107385]: pgmap v2326: 99 pgs: 99 active+clean; 214 MiB data, 3.0 GiB used, 533 GiB / 536 GiB avail 2024-09-21T10:44:10.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:10 smithi070 ceph-mon[112139]: pgmap v2326: 99 pgs: 99 active+clean; 214 MiB data, 3.0 GiB used, 533 GiB / 536 GiB avail 2024-09-21T10:44:10.642 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:10.667 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:10.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:10.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:10.769 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:10.790 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:10.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:10.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:10.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:10 smithi160 ceph-mon[101879]: pgmap v2326: 99 pgs: 99 active+clean; 214 MiB data, 3.0 GiB used, 533 GiB / 536 GiB avail 2024-09-21T10:44:10.892 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:10.913 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:10.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:10.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:11.014 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:11.035 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:11.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:11.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:11.136 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:11.162 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:11.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:11.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:11.264 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:11.289 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:11.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:11.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:11.391 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:11.413 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:11.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:11.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:11.514 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:11.540 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:11.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:11.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:11.642 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:11.667 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:11.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:11.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:11.769 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:11.794 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:11.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:11.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:11.896 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:11.921 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:11.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:11.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:12.023 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:12.048 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:12.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:12.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:12.149 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:12.172 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:12.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:12.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:12.274 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:12.294 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:12.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:12.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:12.396 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:12.416 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:12.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:12.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:12.518 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:12.544 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:12.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:12.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:12.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:12 smithi070 ceph-mon[112139]: pgmap v2327: 99 pgs: 99 active+clean; 77 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail 2024-09-21T10:44:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:12 smithi070 ceph-mon[107385]: pgmap v2327: 99 pgs: 99 active+clean; 77 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail 2024-09-21T10:44:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a[107362]: 2024-09-21T10:44:12.378+0000 7fad4a465640 -1 mon.a@0(leader).osd e1368 definitely_dead 0 2024-09-21T10:44:12.646 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:12.669 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:12.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:12.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:12.771 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:12.796 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:12.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:12.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:12 smithi160 ceph-mon[101879]: pgmap v2327: 99 pgs: 99 active+clean; 77 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail 2024-09-21T10:44:12.898 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:12.924 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:12.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:12.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:13.025 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:13.051 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:13.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:13.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:13.153 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:13.178 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:13.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:13.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:13.279 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:13.305 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:13.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:13.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:13.393 INFO:teuthology.orchestra.run.smithi070.stderr:marked in osd.7. 2024-09-21T10:44:13.406 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:13.415 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:13 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1791126642' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-21T10:44:13.415 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:13 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-21T10:44:13.416 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:44:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:44:13] "GET /metrics HTTP/1.1" 200 36596 "" "Prometheus/2.51.0" 2024-09-21T10:44:13.416 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:13 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1791126642' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-21T10:44:13.416 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:13 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-21T10:44:13.430 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:13.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:13.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:13.532 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:13.555 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:13.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:13.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:13.656 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:13.679 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:13.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:13.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:13.781 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:13.803 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:13.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:13.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:13 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1791126642' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-21T10:44:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:13 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-21T10:44:13.904 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:13.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:13.927 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:13.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:14.027 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:14.052 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:14.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:14.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:14.154 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:14.175 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:14.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:14.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:14.276 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:14.302 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:14.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:14.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:14.389 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T10:44:14.393 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:14 smithi070 ceph-mon[112139]: pgmap v2328: 99 pgs: 99 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-09-21T10:44:14.393 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:14 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-21T10:44:14.393 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:14 smithi070 ceph-mon[112139]: osdmap e1369: 8 total, 8 up, 7 in 2024-09-21T10:44:14.394 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:14 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2810660760' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:44:14.394 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:14 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:44:14.403 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:14.428 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:14.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:14.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:14.530 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:14.552 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:14.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:14.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:14.654 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:14.679 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:14.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:14.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:14.681 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:14 smithi070 ceph-mon[107385]: pgmap v2328: 99 pgs: 99 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-09-21T10:44:14.681 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:14 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-21T10:44:14.681 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:14 smithi070 ceph-mon[107385]: osdmap e1369: 8 total, 8 up, 7 in 2024-09-21T10:44:14.681 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:14 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2810660760' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:44:14.681 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:14 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:44:14.781 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:14.805 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:14.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:14.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:14 smithi160 ceph-mon[101879]: pgmap v2328: 99 pgs: 99 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-09-21T10:44:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:14 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-21T10:44:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:14 smithi160 ceph-mon[101879]: osdmap e1369: 8 total, 8 up, 7 in 2024-09-21T10:44:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:14 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2810660760' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:44:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:14 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:44:14.864 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:44:14.864 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:44:14.864 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:44:14.864 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:44:14.864 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:44:14.865 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:44:14.865 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:44:14.865 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:44:14.865 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:44:14.865 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:44:14.865 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:44:14.865 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:44:14.879 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:44:14.879 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:44:14.896 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:44:14.896 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:44:14.907 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:14.928 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:14.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:14.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:15.030 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:15.055 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:15.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:15.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:15.157 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:15.182 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:15.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:15.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:15.284 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:15.309 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:15.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:15.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:15.314 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-09-21T10:44:15.411 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:15.442 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:15.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:15.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:15.543 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:15.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:15 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:44:15.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:15 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:44:15.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:15 smithi070 ceph-mon[107385]: osdmap e1370: 8 total, 8 up, 7 in 2024-09-21T10:44:15.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:15 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3119630424' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:15.545 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:15 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:44:15.545 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:15 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:44:15.545 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:15 smithi070 ceph-mon[112139]: osdmap e1370: 8 total, 8 up, 7 in 2024-09-21T10:44:15.545 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:15 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3119630424' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:15.565 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:15.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:15.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:15.667 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:15.689 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:15.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:15.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:15.791 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:15.812 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:15.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:15.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:15 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:44:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:15 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:44:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:15 smithi160 ceph-mon[101879]: osdmap e1370: 8 total, 8 up, 7 in 2024-09-21T10:44:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:15 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3119630424' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:15.914 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:15.935 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:15.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:15.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:16.022 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:44:16.022 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:44:16.037 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:16.059 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:16.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:16.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:16.160 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:16.181 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:16.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:16.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:16.282 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:16.305 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:16.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:16.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:16.407 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:16.430 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:16.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:16.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:16.531 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:16.552 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:16.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:16.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:16.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:16 smithi070 ceph-mon[107385]: pgmap v2331: 99 pgs: 2 remapped+peering, 97 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 425 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:44:16.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:16 smithi070 ceph-mon[107385]: 53.14 starting backfill to osd.7 from (0'0,0'0] MAX to 1368'10194 2024-09-21T10:44:16.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:16 smithi070 ceph-mon[107385]: 53.c starting backfill to osd.7 from (0'0,0'0] MAX to 1368'16270 2024-09-21T10:44:16.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:16 smithi070 ceph-mon[107385]: osdmap e1371: 8 total, 8 up, 7 in 2024-09-21T10:44:16.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:16 smithi070 ceph-mon[107385]: 53.17 starting backfill to osd.7 from (0'0,0'0] MAX to 1368'10365 2024-09-21T10:44:16.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:16 smithi070 ceph-mon[107385]: 53.16 starting backfill to osd.7 from (0'0,0'0] MAX to 1368'10219 2024-09-21T10:44:16.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:16 smithi070 ceph-mon[107385]: 53.11 starting backfill to osd.7 from (0'0,0'0] MAX to 1368'10175 2024-09-21T10:44:16.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:16 smithi070 ceph-mon[107385]: 53.13 starting backfill to osd.7 from (0'0,0'0] MAX to 1368'10455 2024-09-21T10:44:16.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:16 smithi070 ceph-mon[112139]: pgmap v2331: 99 pgs: 2 remapped+peering, 97 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 425 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:44:16.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:16 smithi070 ceph-mon[112139]: 53.14 starting backfill to osd.7 from (0'0,0'0] MAX to 1368'10194 2024-09-21T10:44:16.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:16 smithi070 ceph-mon[112139]: 53.c starting backfill to osd.7 from (0'0,0'0] MAX to 1368'16270 2024-09-21T10:44:16.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:16 smithi070 ceph-mon[112139]: osdmap e1371: 8 total, 8 up, 7 in 2024-09-21T10:44:16.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:16 smithi070 ceph-mon[112139]: 53.17 starting backfill to osd.7 from (0'0,0'0] MAX to 1368'10365 2024-09-21T10:44:16.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:16 smithi070 ceph-mon[112139]: 53.16 starting backfill to osd.7 from (0'0,0'0] MAX to 1368'10219 2024-09-21T10:44:16.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:16 smithi070 ceph-mon[112139]: 53.11 starting backfill to osd.7 from (0'0,0'0] MAX to 1368'10175 2024-09-21T10:44:16.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:16 smithi070 ceph-mon[112139]: 53.13 starting backfill to osd.7 from (0'0,0'0] MAX to 1368'10455 2024-09-21T10:44:16.654 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:16.675 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:16.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:16.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:16.777 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:16.800 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:16.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:16.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:16.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:16 smithi160 ceph-mon[101879]: pgmap v2331: 99 pgs: 2 remapped+peering, 97 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 425 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:44:16.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:16 smithi160 ceph-mon[101879]: 53.14 starting backfill to osd.7 from (0'0,0'0] MAX to 1368'10194 2024-09-21T10:44:16.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:16 smithi160 ceph-mon[101879]: 53.c starting backfill to osd.7 from (0'0,0'0] MAX to 1368'16270 2024-09-21T10:44:16.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:16 smithi160 ceph-mon[101879]: osdmap e1371: 8 total, 8 up, 7 in 2024-09-21T10:44:16.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:16 smithi160 ceph-mon[101879]: 53.17 starting backfill to osd.7 from (0'0,0'0] MAX to 1368'10365 2024-09-21T10:44:16.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:16 smithi160 ceph-mon[101879]: 53.16 starting backfill to osd.7 from (0'0,0'0] MAX to 1368'10219 2024-09-21T10:44:16.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:16 smithi160 ceph-mon[101879]: 53.11 starting backfill to osd.7 from (0'0,0'0] MAX to 1368'10175 2024-09-21T10:44:16.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:16 smithi160 ceph-mon[101879]: 53.13 starting backfill to osd.7 from (0'0,0'0] MAX to 1368'10455 2024-09-21T10:44:16.903 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:16.925 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:16.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:16.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:17.027 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:17.031 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:44:17.048 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:17.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:17.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:17.150 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:17.172 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:17.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:17.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:17.274 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:17.295 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:17.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:17.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:17.397 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:17.429 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:17.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:17.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:17.531 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:17.552 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:17.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:17.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:17.653 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:17.677 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:17.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:17.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:17.773 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:17 smithi070 ceph-mon[107385]: osdmap e1372: 8 total, 8 up, 7 in 2024-09-21T10:44:17.774 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:17 smithi070 ceph-mon[112139]: osdmap e1372: 8 total, 8 up, 7 in 2024-09-21T10:44:17.778 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:17.799 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:17.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:17.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:17.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:17 smithi160 ceph-mon[101879]: osdmap e1372: 8 total, 8 up, 7 in 2024-09-21T10:44:17.900 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:17.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:17.921 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:17.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:18.022 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:18.043 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:18.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:18.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:18.144 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:18.170 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:18.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:18.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:18.235 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:18.236 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:18.272 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:18.292 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:18.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:18.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:18.394 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:18.419 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:18.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:18.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:18.521 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:18.546 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:18.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:18.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:18.578 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:18 smithi070 ceph-mon[112139]: pgmap v2334: 99 pgs: 2 active+clean+remapped, 5 remapped+peering, 92 active+clean; 582 KiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s, 1 objects/s recovering 2024-09-21T10:44:18.578 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:18 smithi070 ceph-mon[112139]: osdmap e1373: 8 total, 8 up, 7 in 2024-09-21T10:44:18.579 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:18 smithi070 ceph-mon[107385]: pgmap v2334: 99 pgs: 2 active+clean+remapped, 5 remapped+peering, 92 active+clean; 582 KiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s, 1 objects/s recovering 2024-09-21T10:44:18.579 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:18 smithi070 ceph-mon[107385]: osdmap e1373: 8 total, 8 up, 7 in 2024-09-21T10:44:18.648 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:18.668 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:18.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:18.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:18.770 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:18.791 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:18.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:18.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:18 smithi160 ceph-mon[101879]: pgmap v2334: 99 pgs: 2 active+clean+remapped, 5 remapped+peering, 92 active+clean; 582 KiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s, 1 objects/s recovering 2024-09-21T10:44:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:18 smithi160 ceph-mon[101879]: osdmap e1373: 8 total, 8 up, 7 in 2024-09-21T10:44:18.893 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:18.915 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:18.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:18.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:19.016 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:19.042 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:19.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:19.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:19.144 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:19.165 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:19.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:19.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:19.267 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:19.292 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:19.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:19.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:19.394 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:19.416 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:19.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:19.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:19.517 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:19.543 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:19.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:19.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:19.645 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:19.666 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:19.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:19.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:19.768 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:19.793 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:19.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:19.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:19.794 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:44:19.895 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:19.920 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:19.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:19.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:20.022 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:20.047 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:20.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:20.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:20.148 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:20.172 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:20.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:20.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:20.220 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:44:20.274 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:20.298 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:20.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:20.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:20.314 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:44:20.314 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:44:20.400 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:20.423 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:20.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:20.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:20.497 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:20 smithi070 ceph-mon[112139]: pgmap v2336: 99 pgs: 2 active+clean+remapped, 5 remapped+peering, 92 active+clean; 582 KiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s, 1 objects/s recovering 2024-09-21T10:44:20.497 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:44:20.497 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:44:20.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:20 smithi070 ceph-mon[107385]: pgmap v2336: 99 pgs: 2 active+clean+remapped, 5 remapped+peering, 92 active+clean; 582 KiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s, 1 objects/s recovering 2024-09-21T10:44:20.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:44:20.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:44:20.524 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:20.549 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:20.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:20.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:20.651 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:20.675 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:20.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:20.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:20.777 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:20.798 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:20.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:20.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:20 smithi160 ceph-mon[101879]: pgmap v2336: 99 pgs: 2 active+clean+remapped, 5 remapped+peering, 92 active+clean; 582 KiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s, 1 objects/s recovering 2024-09-21T10:44:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:44:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:44:20.900 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:20.925 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:20.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:20.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:21.027 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:21.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:21.053 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:21.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:21.154 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:21.179 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:21.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:21.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:21.280 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:21.304 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:21.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:21.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:21.406 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:21.428 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:21.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:21.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:21.530 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:21.551 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:21.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:21.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:21.653 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:21.674 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:21.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:21.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:21.776 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:21.797 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:21.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:21.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:21.898 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:21.925 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:21.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:21.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:21.983 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:44:21.983 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:44:22.027 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:22.052 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:22.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:22.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:22.154 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:22.175 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:22.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:22.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:22.277 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:22.302 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:22.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:22.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:22.404 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:22.429 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:22.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:22.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:22.530 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:22.552 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:22.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:22.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:22 smithi070 ceph-mon[107385]: pgmap v2337: 99 pgs: 2 active+clean+remapped, 1 remapped+peering, 96 active+clean; 582 KiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 792 B/s, 0 objects/s recovering 2024-09-21T10:44:22.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:22 smithi070 ceph-mon[112139]: pgmap v2337: 99 pgs: 2 active+clean+remapped, 1 remapped+peering, 96 active+clean; 582 KiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 792 B/s, 0 objects/s recovering 2024-09-21T10:44:22.653 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:22.671 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:22 smithi160 ceph-mon[101879]: pgmap v2337: 99 pgs: 2 active+clean+remapped, 1 remapped+peering, 96 active+clean; 582 KiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 792 B/s, 0 objects/s recovering 2024-09-21T10:44:22.679 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:22.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:22.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:22.705 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:44:22.781 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:22.802 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:22.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:22.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:22.903 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:22.925 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:22.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:22.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:23.026 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:23.047 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:23.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:23.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:23.149 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:23.170 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:23.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:23.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:23.265 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:23.266 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:23.271 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:23.292 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:23.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:23.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:23.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:44:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:44:23] "GET /metrics HTTP/1.1" 200 36600 "" "Prometheus/2.51.0" 2024-09-21T10:44:23.394 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:23.416 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:23.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:23.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:23.518 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:23.538 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:23.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:23.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:23.640 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:23.660 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:23.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:23.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:23.762 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:23.783 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:23.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:23.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:23.884 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:23.906 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:23.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:23.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:24.008 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:24.031 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:24.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:24.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:24.133 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:24.156 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:24.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:24.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:24.257 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:24.278 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:24.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:24.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:24.380 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:24.401 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:24.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:24.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:24.469 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:24.503 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:24.525 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:24.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:24.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:24.563 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:24 smithi070 ceph-mon[107385]: pgmap v2338: 99 pgs: 99 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-21T10:44:24.563 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:24 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:44:24.563 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:24 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:44:24.563 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:24 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:44:24.563 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:24 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:44:24.563 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:24 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:44:24.563 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:24 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2346339298' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:44:24.563 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:24 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:44:24.563 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:24 smithi070 ceph-mon[112139]: pgmap v2338: 99 pgs: 99 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-21T10:44:24.563 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:24 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:44:24.563 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:24 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:44:24.564 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:24 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:44:24.564 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:24 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:44:24.564 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:24 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:44:24.564 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:24 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2346339298' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:44:24.564 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:24 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:44:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:24 smithi160 ceph-mon[101879]: pgmap v2338: 99 pgs: 99 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-21T10:44:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:24 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:44:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:24 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:44:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:24 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:44:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:24 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:44:24.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:24 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:44:24.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:24 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2346339298' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:44:24.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:24 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:44:24.626 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:24.647 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:24.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:24.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:24.681 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'unique_pool_0' removed 2024-09-21T10:44:24.749 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:24.770 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:24.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:24.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:24.872 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:24.893 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:24.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:24.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:24.995 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:25.016 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:25.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:25.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:25.117 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:25.139 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:25.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:25.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:25.241 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:25.262 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:25.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:25.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:25.363 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:25.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:25.387 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:25.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:25.487 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:25.510 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:25.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:25.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:25.611 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:25.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:25.640 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:25.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:25.741 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:25.762 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:25.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:25.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:25.804 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:25 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:44:25.804 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:25 smithi070 ceph-mon[107385]: osdmap e1374: 8 total, 8 up, 7 in 2024-09-21T10:44:25.804 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:25 smithi070 ceph-mon[107385]: pgmap v2340: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:25.804 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:25 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:44:25.805 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:25 smithi070 ceph-mon[112139]: osdmap e1374: 8 total, 8 up, 7 in 2024-09-21T10:44:25.805 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:25 smithi070 ceph-mon[112139]: pgmap v2340: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:25.864 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:25.885 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:25.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:25.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:25.987 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:26.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:26.009 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:26.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:26.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:25 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:44:26.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:25 smithi160 ceph-mon[101879]: osdmap e1374: 8 total, 8 up, 7 in 2024-09-21T10:44:26.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:25 smithi160 ceph-mon[101879]: pgmap v2340: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:26.109 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:26.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:26.132 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:26.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:26.233 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:26.256 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:26.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:26.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:26.358 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:26.379 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:26.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:26.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:26.480 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:26.501 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:26.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:26.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:26.602 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:26.625 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:26.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:26.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:26.726 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:26.747 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:26.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:26.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:26.849 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:26.871 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:26.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:26.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:26.972 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:26.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:26.995 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:26.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:27.096 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:27.117 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:27.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:27.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:27.218 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:27.239 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:27.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:27.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:27.340 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:27.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:27.362 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:27.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:27.463 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:27.489 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:27.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:27.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:27.590 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:27.614 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:27.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:27.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:27.716 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:27.737 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:27.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:27.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:27.767 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T10:44:27.768 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T10:44:27.768 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-09-21T10:44:27.768 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_1 16 2024-09-21T10:44:27.839 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:27.865 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:27.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:27.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:27.966 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:27.990 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:27.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:27.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:28.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:27 smithi160 ceph-mon[101879]: pgmap v2341: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:28.091 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:28.113 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:28.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:28.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:28.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:27 smithi070 ceph-mon[107385]: pgmap v2341: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:28.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:27 smithi070 ceph-mon[112139]: pgmap v2341: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:28.214 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:28.238 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:28.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:28.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:28.340 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:28.364 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:28.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:28.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:28.466 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:28.489 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:28.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:28.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:28.591 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:28.612 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:28.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:28.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:28.713 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:28.734 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:28.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:28.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:28.808 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T10:44:28.836 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:28.860 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:28.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:28.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:28.961 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:28.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:28.985 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:28.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:29.011 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:44:29.012 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:44:29.028 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:44:29.029 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:44:29.029 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:28 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2070847791' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:44:29.029 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:28 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2070847791' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:44:29.045 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:44:29.045 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:44:29.061 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:44:29.061 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:44:29.077 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:44:29.077 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:44:29.086 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:29.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:28 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2070847791' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:44:29.093 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:44:29.094 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:44:29.109 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:29.110 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:44:29.110 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:44:29.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:29.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:29.126 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:44:29.126 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:44:29.211 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:29.234 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:29.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:29.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:29.336 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:29.358 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:29.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:29.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:29.426 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:29.460 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:29.481 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:29.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:29.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:29.582 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:29.603 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:29.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:29.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:29.705 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:29.730 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:29.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:29.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:29.832 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:29.854 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:29.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:29.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:29.955 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:29.981 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:29.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:29.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:30.083 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:30.103 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:29 smithi160 ceph-mon[101879]: pgmap v2342: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:30.103 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:29 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:44:30.103 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:29 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2070847791' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:44:30.104 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:29 smithi160 ceph-mon[101879]: osdmap e1375: 8 total, 8 up, 7 in 2024-09-21T10:44:30.104 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:29 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2374259402' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:30.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:29 smithi070 ceph-mon[107385]: pgmap v2342: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:30.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:29 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:44:30.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:29 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2070847791' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:44:30.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:29 smithi070 ceph-mon[107385]: osdmap e1375: 8 total, 8 up, 7 in 2024-09-21T10:44:30.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:29 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2374259402' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:30.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:29 smithi070 ceph-mon[112139]: pgmap v2342: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:30.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:29 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:44:30.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:29 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2070847791' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:44:30.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:29 smithi070 ceph-mon[112139]: osdmap e1375: 8 total, 8 up, 7 in 2024-09-21T10:44:30.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:29 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2374259402' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:30.128 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:30.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:30.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:30.230 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:30.255 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:30.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:30.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:30.356 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:30.380 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:30.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:30.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:30.481 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:30.502 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:30.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:30.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:30.604 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:30.629 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:30.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:30.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:30.730 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:30.751 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:30.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:30.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:30.852 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:30.879 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:30.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:30.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:30.980 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:31.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:31.006 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:31.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:31.107 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:31.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:31.129 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:31.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:31.229 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:31.251 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:31.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:31.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:31.352 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:31.378 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:31.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:31.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:31.479 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:31.505 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:31.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:31.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:31.606 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:31.631 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:31.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:31.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:31.733 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:31.756 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:31.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:31.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:31.808 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'unique_pool_1' created 2024-09-21T10:44:31.858 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:31 smithi070 ceph-mon[107385]: pgmap v2344: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:31 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3100201437' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T10:44:31.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:31 smithi070 ceph-mon[112139]: pgmap v2344: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:31.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:31 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3100201437' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T10:44:31.879 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:31.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:31.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:31.981 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:32.006 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:32.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:32.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:32.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:31 smithi160 ceph-mon[101879]: pgmap v2344: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:32.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:31 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3100201437' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T10:44:32.106 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:44:32.106 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T10:44:32.106 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T10:44:32.107 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T10:44:32.107 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:44:32.108 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:32.133 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:32.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:32.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:32.234 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:32.260 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:32.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:32.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:32.361 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:32.386 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:32.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:32.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:32.487 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:32.509 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:32.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:32.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:32.610 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:32.632 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:32.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:32.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:32.733 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:32.755 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:32.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:32.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:32.857 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:32.878 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:32.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:32.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:32.980 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:33.005 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:33.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:33.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:33.032 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:44:33.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:32 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3100201437' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-21T10:44:33.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:32 smithi070 ceph-mon[112139]: osdmap e1376: 8 total, 8 up, 7 in 2024-09-21T10:44:33.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:32 smithi070 ceph-mon[112139]: osdmap e1377: 8 total, 8 up, 7 in 2024-09-21T10:44:33.054 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:32 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3100201437' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-21T10:44:33.054 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:32 smithi070 ceph-mon[107385]: osdmap e1376: 8 total, 8 up, 7 in 2024-09-21T10:44:33.054 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:32 smithi070 ceph-mon[107385]: osdmap e1377: 8 total, 8 up, 7 in 2024-09-21T10:44:33.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:32 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3100201437' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-21T10:44:33.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:32 smithi160 ceph-mon[101879]: osdmap e1376: 8 total, 8 up, 7 in 2024-09-21T10:44:33.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:32 smithi160 ceph-mon[101879]: osdmap e1377: 8 total, 8 up, 7 in 2024-09-21T10:44:33.098 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:44:33.106 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:33.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:33.133 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:33.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:33.234 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:33.260 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:33.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:33.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:33.361 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:33.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:44:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:44:33] "GET /metrics HTTP/1.1" 200 36600 "" "Prometheus/2.51.0" 2024-09-21T10:44:33.383 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:33.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:33.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:33.484 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:33.509 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:33.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:33.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:33.610 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:33.632 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:33.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:33.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:33.733 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:33.758 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:33.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:33.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:33.860 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:33.884 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:33.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:33.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:33.986 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:33.997 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-09-21T10:44:34.007 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:34.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:34.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:34.083 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:34.089 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:34.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:33 smithi160 ceph-mon[101879]: pgmap v2346: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:34.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:33 smithi160 ceph-mon[101879]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:44:34.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:33 smithi160 ceph-mon[101879]: osdmap e1378: 8 total, 8 up, 7 in 2024-09-21T10:44:34.109 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:34.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:33 smithi070 ceph-mon[107385]: pgmap v2346: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:34.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:33 smithi070 ceph-mon[107385]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:44:34.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:33 smithi070 ceph-mon[107385]: osdmap e1378: 8 total, 8 up, 7 in 2024-09-21T10:44:34.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:33 smithi070 ceph-mon[112139]: pgmap v2346: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:34.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:33 smithi070 ceph-mon[112139]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:44:34.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:33 smithi070 ceph-mon[112139]: osdmap e1378: 8 total, 8 up, 7 in 2024-09-21T10:44:34.130 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:34.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:34.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:34.232 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:34.253 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:34.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:34.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:34.354 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:34.378 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:34.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:34.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:34.480 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:34.501 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:34.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:34.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:34.603 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:34.627 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:34.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:34.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:34.729 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:34.752 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:34.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:34.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:34.854 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:34.877 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:34.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:34.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:34.978 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:35.000 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:35.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:35.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:35.102 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:35.123 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:35.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:35.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:35.224 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:35.250 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:35.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:35.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:35.352 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:35.377 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:35.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:35.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:35.478 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:35.503 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:35.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:35.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:35.527 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:44:35.605 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:35.629 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:35.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:35.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:35.731 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:35.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:35.753 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:35.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:35.853 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:35.879 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:35.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:35.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:35.980 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:36.002 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:36.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:36.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:36.103 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:36.126 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:36.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:36.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:36.140 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:44:36.140 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:44:36.157 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:44:36.157 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:44:36.173 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:44:36.173 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:44:36.189 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:44:36.190 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:44:36.206 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:44:36.206 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:44:36.223 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:44:36.223 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:44:36.227 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:36.239 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:44:36.239 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:44:36.253 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:36.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:36.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:36.255 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:44:36.255 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:44:36.355 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:36.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:36.378 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:36.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:36.479 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:36.504 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:36.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:36.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:36.605 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:36.616 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:36 smithi070 ceph-mon[112139]: pgmap v2349: 89 pgs: 9 unknown, 80 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:36.616 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:44:36.616 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:44:36.616 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:36 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/640364018' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:36.616 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:36 smithi070 ceph-mon[107385]: pgmap v2349: 89 pgs: 9 unknown, 80 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:36.616 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:44:36.616 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:44:36.616 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:36 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/640364018' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:36.627 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:36.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:36.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:36.728 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:36.754 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:36.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:36.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:36 smithi160 ceph-mon[101879]: pgmap v2349: 89 pgs: 9 unknown, 80 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:44:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:44:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:36 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/640364018' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:36.856 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:36.877 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:36.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:36.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:36.978 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:37.004 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:37.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:37.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:37.105 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:37.130 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:37.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:37.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:37.232 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:37.256 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:37.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:37.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:37.357 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:37.383 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:37.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:37.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:37.485 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:37.508 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:37.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:37.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:37.610 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:37.633 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:37.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:37.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:37.637 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:37.734 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:37.755 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:37.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:37.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:37.857 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:37.879 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:37.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:37.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:37.980 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:38.002 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:38.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:38.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:38.103 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:38.124 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:38.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:38.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:38.226 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:38.248 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:38.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:38.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:38.349 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:38.369 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:44:38.369 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:44:38.375 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:38.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:38.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:38.477 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:38.502 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:38.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:38.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:38.567 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:38.603 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:38 smithi070 ceph-mon[107385]: pgmap v2350: 89 pgs: 89 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:38.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:38 smithi070 ceph-mon[112139]: pgmap v2350: 89 pgs: 89 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:38.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:38.628 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:38.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:38.729 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:38.753 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:38.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:38.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:38 smithi160 ceph-mon[101879]: pgmap v2350: 89 pgs: 89 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:38.854 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:38.876 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:38.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:38.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:38.977 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:39.003 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:39.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:39.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:39.105 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:39.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:39.127 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:39.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:39.228 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:39.249 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:39.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:39.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:39.350 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:39.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:39.377 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:39.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:39.478 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:39.499 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:39.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:39.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:39.600 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:39.621 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:39.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:39.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:39.664 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:44:39.723 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:39.743 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:39.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:39.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:39.845 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:39.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:39.866 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:39.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:39.967 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:39.993 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:39.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:39.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:40.095 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:40.120 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:40.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:40.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:40.221 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:40.242 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:40.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:40.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:40.344 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:40.365 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:40.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:40.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:40.466 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:40.467 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:44:40.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:40.530 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:40.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:40.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:40 smithi070 ceph-mon[107385]: pgmap v2351: 89 pgs: 89 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:40.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:40 smithi070 ceph-mon[112139]: pgmap v2351: 89 pgs: 89 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:40.631 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:40.657 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:40.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:40.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:40.758 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:40.783 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:40.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:40.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:40 smithi160 ceph-mon[101879]: pgmap v2351: 89 pgs: 89 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:40.884 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:40.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:40.909 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:40.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:40.924 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:41.010 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:41.034 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:41.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:41.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:41.136 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:41.159 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:41.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:41.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:41.261 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:41.284 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:41.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:41.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:41.385 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:41.406 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:41.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:41.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:41.508 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:41.529 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:41.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:41.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:41.631 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:41.653 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:41.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:41.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:41.755 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:41.777 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:41.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:41.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:41.878 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:41.899 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:41.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:41.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:42.000 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:42.021 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:42.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:42.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:42.123 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:42.149 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:42.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:42.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:42.195 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:42.250 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:42.271 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:42.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:42.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:42.372 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:42.394 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:42.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:42.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:42.409 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:42 smithi070 ceph-mon[112139]: pgmap v2352: 89 pgs: 89 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:42.410 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:42 smithi070 ceph-mon[107385]: pgmap v2352: 89 pgs: 89 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:42.496 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:42.518 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:42.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:42.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:42.620 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:42.646 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:42.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:42.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:42.748 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:42.772 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:42.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:42.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:42 smithi160 ceph-mon[101879]: pgmap v2352: 89 pgs: 89 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:42.874 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:42.896 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:42.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:42.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:42.998 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:43.023 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:43.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:43.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:43.125 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:43.147 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:43.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:43.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:43.247 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:44:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:44:43] "GET /metrics HTTP/1.1" 200 36532 "" "Prometheus/2.51.0" 2024-09-21T10:44:43.249 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:43.271 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:43.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:43.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:43.372 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:43.396 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:43.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:43.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:43.467 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:44:43.467 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:44:43.467 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:44:43.468 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:44:43.468 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:44:43.468 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:44:43.469 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:44:43.469 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:44:43.469 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:44:43.469 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:44:43.469 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:44:43.469 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:44:43.470 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:44:43.470 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:44:43.470 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:44:43.470 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:44:43.498 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:43.519 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:43.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:43.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:43.621 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:43.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:43 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/652958052' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:43.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:43 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/652958052' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:43.642 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:43.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:43.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:43.744 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:43.770 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:43.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:43.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:43 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/652958052' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:43.872 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:43.893 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:43.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:43.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:43.994 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:44.015 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:44.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:44.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:44.117 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:44.138 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:44.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:44.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:44.240 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:44.260 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:44.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:44.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:44.362 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:44.388 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:44.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:44.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:44.489 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:44.514 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:44.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:44.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:44.616 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:44.640 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:44.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:44.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:44.742 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:44.763 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:44.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:44.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:44.773 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:44 smithi070 ceph-mon[112139]: pgmap v2353: 89 pgs: 89 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:44.774 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:44 smithi070 ceph-mon[107385]: pgmap v2353: 89 pgs: 89 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:44 smithi160 ceph-mon[101879]: pgmap v2353: 89 pgs: 89 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:44.864 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:44.890 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:44.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:44.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:44.992 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:45.016 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:45.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:45.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:45.118 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:45.139 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:45.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:45.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:45.240 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:45.261 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:45.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:45.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:45.362 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:45.388 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T10:44:45.388 INFO:teuthology.orchestra.run.smithi070.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-09-21T10:44:45.390 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:45.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:45.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:45.401 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:45 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2758722740' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:44:45.401 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:45 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:44:45.401 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:45 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1777019990' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:44:45.402 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:45 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:44:45.402 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:45 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2758722740' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:44:45.402 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:45 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:44:45.402 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:45 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1777019990' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:44:45.402 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:45 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:44:45.492 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:45.513 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:45.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:45.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:45.615 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:45.640 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:45.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:45.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:45.742 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:45.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:45.765 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:45.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:45.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:45 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2758722740' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:44:45.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:45 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:44:45.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:45 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1777019990' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:44:45.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:45 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:44:45.866 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:45.886 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:45.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:45.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:45.988 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:46.009 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:46.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:46.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:46.110 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:46.136 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:46.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:46.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:46.238 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:46.259 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:46.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:46.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:46.361 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:46.386 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:46.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:46.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:46.487 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:46.508 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:46.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:46.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:46.610 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:46.636 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:46.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:46.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:46.738 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:46.762 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:46.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:46.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:46.794 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:44:46.816 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:46 smithi070 ceph-mon[107385]: pgmap v2354: 89 pgs: 89 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:46.817 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:46 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:44:46.817 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:46 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:44:46.817 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:46 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:44:46.817 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:46 smithi070 ceph-mon[107385]: osdmap e1379: 8 total, 8 up, 7 in 2024-09-21T10:44:46.817 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:46 smithi070 ceph-mon[112139]: pgmap v2354: 89 pgs: 89 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:46.817 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:46 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:44:46.817 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:46 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:44:46.817 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:46 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:44:46.817 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:46 smithi070 ceph-mon[112139]: osdmap e1379: 8 total, 8 up, 7 in 2024-09-21T10:44:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:46 smithi160 ceph-mon[101879]: pgmap v2354: 89 pgs: 89 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:46 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:44:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:46 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:44:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:46 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:44:46.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:46 smithi160 ceph-mon[101879]: osdmap e1379: 8 total, 8 up, 7 in 2024-09-21T10:44:46.864 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:46.888 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:46.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:46.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:46.989 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:46.990 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:47.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:47.013 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:47.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:47.114 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:47.139 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:47.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:47.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:47.241 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:47.266 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:47.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:47.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:47.368 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:47.390 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:47.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:47.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:47.492 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:47.517 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:47.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:47.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:47.619 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:47.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:47 smithi070 ceph-mon[112139]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:44:47.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:47 smithi070 ceph-mon[107385]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:44:47.644 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:47.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:47.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:47.732 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:44:47.732 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T10:44:47.732 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T10:44:47.732 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:44:47.746 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:47.768 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:47.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:47.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:47.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:47 smithi160 ceph-mon[101879]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:44:47.871 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:47.892 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:47.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:47.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:47.934 DEBUG:teuthology.orchestra.run.smithi070:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-09-21T10:44:47.934 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_1 2024-09-21T10:44:47.935 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:44:47.937 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T10:44:47.937 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T10:44:47.987 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:hints = 1 2024-09-21T10:44:47.987 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T10:44:47.987 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Object prefix: benchmark_data_smithi070_261699 2024-09-21T10:44:47.988 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:44:47.988 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 0 0 0 0 0 0 - 0 2024-09-21T10:44:47.993 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:48.034 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:48.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:48.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:48.136 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:48.177 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:48.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:48.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:48.279 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:48.313 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:48.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:48.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:48.415 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:48.436 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:44:48.450 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:48.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:48.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:48.486 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:48 smithi070 ceph-mon[107385]: pgmap v2356: 89 pgs: 89 active+clean; 582 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:48.486 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:48 smithi070 ceph-mon[112139]: pgmap v2356: 89 pgs: 89 active+clean; 582 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:48.552 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:48.586 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:48.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:48.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:48.688 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:48.721 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:48.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:48.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:48.823 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:48 smithi160 ceph-mon[101879]: pgmap v2356: 89 pgs: 89 active+clean; 582 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:48.856 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:48.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:48.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:48.957 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:48.988 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 1 16 4014 3998 249.861 249.875 0.00519824 0.00398853 2024-09-21T10:44:48.991 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:48.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:48.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:49.093 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:49.128 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:49.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:49.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:49.230 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:49.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:49.266 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:49.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:49.367 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:49.402 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:49.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:49.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:49.460 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:44:49.504 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:49.539 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:49.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:49.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:49.641 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:49.673 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:49.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:49.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:49.775 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:49.803 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:49.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:49.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:49.905 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:49.939 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:49.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:49.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:49.988 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 2 16 7591 7575 236.701 223.562 0.00453993 0.00421593 2024-09-21T10:44:50.041 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:50.067 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:50.067 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:50.071 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:50.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:50.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:50.133 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:50.155 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:44:50.155 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:44:50.172 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:50.187 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:44:50.188 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:44:50.206 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:50.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:50.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:50.220 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:44:50.220 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:44:50.253 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:44:50.253 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:44:50.287 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:44:50.287 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:44:50.308 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:50.323 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:44:50.323 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:44:50.342 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:50.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:50.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:50.358 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:44:50.358 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:44:50.392 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:44:50.392 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:44:50.444 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:50.477 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:50.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:50.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:50.579 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:50.617 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:50.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:50.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:50.651 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:50 smithi070 ceph-mon[112139]: pgmap v2357: 89 pgs: 89 active+clean; 582 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:50.651 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:50 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3853560660' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:50.651 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:44:50.651 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:44:50.652 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:50 smithi070 ceph-mon[107385]: pgmap v2357: 89 pgs: 89 active+clean; 582 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:50.652 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:50 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3853560660' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:50.652 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:44:50.652 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:44:50.752 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:50.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:50.787 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:50.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:50 smithi160 ceph-mon[101879]: pgmap v2357: 89 pgs: 89 active+clean; 582 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:44:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:50 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3853560660' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:44:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:44:50.888 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:50.923 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:50.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:50.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:50.988 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 3 16 10973 10957 228.252 211.375 0.00453812 0.00437428 2024-09-21T10:44:51.024 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:51.059 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:51.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:51.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:51.161 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:51.198 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:51.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:51.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:51.301 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:51.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:51.341 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:51.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:51.441 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:51.476 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:51.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:51.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:51.577 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:51.612 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:51.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:51.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:51.713 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:51.747 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:51.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:51.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:51.848 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:51.878 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:51.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:51.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:51.980 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:51.988 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 4 16 14276 14260 222.794 206.438 0.00423439 0.00448065 2024-09-21T10:44:52.017 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:52.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:52.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:52.119 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:52.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:52.155 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:52.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:52.255 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:52.289 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:52.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:52.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:52.390 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:52.423 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:52.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:52.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:52.525 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:52.557 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:52.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:52.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:52.659 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:52.697 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:52.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:52.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:52.799 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:52.835 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:52.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:52.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:52 smithi160 ceph-mon[101879]: pgmap v2358: 89 pgs: 89 active+clean; 195 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 19 MiB/s wr, 311 op/s 2024-09-21T10:44:52.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:52 smithi070 ceph-mon[107385]: pgmap v2358: 89 pgs: 89 active+clean; 195 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 19 MiB/s wr, 311 op/s 2024-09-21T10:44:52.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:52 smithi070 ceph-mon[112139]: pgmap v2358: 89 pgs: 89 active+clean; 195 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 19 MiB/s wr, 311 op/s 2024-09-21T10:44:52.937 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:52.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:52.972 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:52.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:52.988 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 5 16 17356 17340 216.733 192.5 0.00685729 0.00460822 2024-09-21T10:44:53.073 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:53.109 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:53.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:53.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:53.211 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:53.246 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:53.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:53.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:53.348 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:53.380 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:53.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:53.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:53.482 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:53.517 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:53.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:53.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:53.619 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:53.626 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:44:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:44:53] "GET /metrics HTTP/1.1" 200 36533 "" "Prometheus/2.51.0" 2024-09-21T10:44:53.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:53.653 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:53.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:53.754 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:53.789 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:53.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:53.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:53.891 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:53.928 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:53.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:53.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:53.988 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 6 16 20349 20333 211.785 187.062 0.00422317 0.0047162 2024-09-21T10:44:54.030 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:54.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:54.065 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:54.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:54.166 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:54.199 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:54.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:54.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:54.301 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:54.335 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:54.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:54.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:54.437 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:54.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:54.475 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:54.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:54.576 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:54.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:54.614 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:54.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:54.715 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:54.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:54 smithi070 ceph-mon[107385]: pgmap v2359: 89 pgs: 89 active+clean; 564 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 56 MiB/s wr, 900 op/s 2024-09-21T10:44:54.755 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:54 smithi070 ceph-mon[112139]: pgmap v2359: 89 pgs: 89 active+clean; 564 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 56 MiB/s wr, 900 op/s 2024-09-21T10:44:54.755 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:54.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:54.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:54 smithi160 ceph-mon[101879]: pgmap v2359: 89 pgs: 89 active+clean; 564 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 56 MiB/s wr, 900 op/s 2024-09-21T10:44:54.860 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:54.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:54.889 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:54.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:54.988 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 7 16 22848 22832 203.841 156.188 0.00399474 0.00489895 2024-09-21T10:44:54.990 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:55.026 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:55.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:55.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:55.128 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:55.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:55.167 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:55.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:55.227 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:44:55.268 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:55.302 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:55.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:55.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:55.404 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:55.444 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:55.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:55.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:55.545 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:55.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:55.585 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:55.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:55.686 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:55.722 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:55.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:55.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:55.825 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:55.862 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:55.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:55.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:55.964 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:55.988 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 8 16 25235 25219 197.007 149.188 0.00562336 0.00507133 2024-09-21T10:44:55.998 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:55.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:55.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:56.100 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:56.125 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:44:56.129 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T10:44:56.139 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:56.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:56.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:56.241 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:56.276 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:56.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:56.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:56.378 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:56.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:56.415 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:56.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:56.451 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T10:44:56.516 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:56.550 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:56.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:56.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:56.652 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:56.686 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:56.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:56.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:56.788 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:56.824 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:56.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:56 smithi070 ceph-mon[112139]: pgmap v2360: 89 pgs: 89 active+clean; 990 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-21T10:44:56.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:56 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1809921189' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:44:56.825 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:56 smithi070 ceph-mon[107385]: pgmap v2360: 89 pgs: 89 active+clean; 990 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-21T10:44:56.825 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:56 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1809921189' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:44:56.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:56.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:56 smithi160 ceph-mon[101879]: pgmap v2360: 89 pgs: 89 active+clean; 990 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-21T10:44:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:56 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1809921189' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:44:56.927 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:56.959 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:56.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:56.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:56.988 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 9 16 27711 27695 192.31 154.75 0.00637294 0.00519499 2024-09-21T10:44:57.061 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:57.094 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:57.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:57.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:57.196 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:57.217 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:57.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:57.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:57.318 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:57.352 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:57.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:57.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:57.454 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:57.477 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:57.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:57.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:57.579 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:57.614 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:57.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:57.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:57.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:57 smithi070 ceph-mon[107385]: from='client.45320 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:44:57.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:57 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:44:57.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:57 smithi070 ceph-mon[107385]: Cluster is now healthy 2024-09-21T10:44:57.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:57 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1809921189' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:44:57.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:57 smithi070 ceph-mon[107385]: osdmap e1380: 8 total, 8 up, 7 in 2024-09-21T10:44:57.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:57 smithi070 ceph-mon[112139]: from='client.45320 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:44:57.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:57 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:44:57.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:57 smithi070 ceph-mon[112139]: Cluster is now healthy 2024-09-21T10:44:57.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:57 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1809921189' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:44:57.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:57 smithi070 ceph-mon[112139]: osdmap e1380: 8 total, 8 up, 7 in 2024-09-21T10:44:57.716 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:57.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:57.751 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:57.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:57.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:57 smithi160 ceph-mon[101879]: from='client.45320 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:44:57.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:57 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:44:57.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:57 smithi160 ceph-mon[101879]: Cluster is now healthy 2024-09-21T10:44:57.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:57 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1809921189' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:44:57.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:57 smithi160 ceph-mon[101879]: osdmap e1380: 8 total, 8 up, 7 in 2024-09-21T10:44:57.853 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:57.891 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:57.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:57.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:57.989 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 10 16 29829 29813 186.316 132.375 0.00381501 0.00536128 2024-09-21T10:44:57.992 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:58.023 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:58.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:58.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:58.126 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:58.157 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:58.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:58.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:58.259 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:58.292 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:58.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:58.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:58.394 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:58.427 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:58.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:58.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:58.530 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:58.562 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:58.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:58.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:58.664 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:58.684 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:58.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:58.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:58.786 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:58.822 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:58.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:58.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:44:58 smithi160 ceph-mon[101879]: pgmap v2362: 89 pgs: 89 active+clean; 1.5 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-09-21T10:44:58.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:44:58 smithi070 ceph-mon[107385]: pgmap v2362: 89 pgs: 89 active+clean; 1.5 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-09-21T10:44:58.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:44:58 smithi070 ceph-mon[112139]: pgmap v2362: 89 pgs: 89 active+clean; 1.5 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-09-21T10:44:58.924 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:58.961 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:58.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:58.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:58.988 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 11 16 31957 31941 181.468 133 0.008467 0.0055055 2024-09-21T10:44:59.063 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:59.084 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:59.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:59.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:59.185 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:59.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:59.212 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:59.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:59.312 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:59.351 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:59.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:59.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:59.402 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:44:59.453 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:59.474 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:59.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:59.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:59.575 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:59.612 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:44:59.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:59.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:59.714 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:59.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:59.741 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:59.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:59.841 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:59.877 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:44:59.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:59.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:59.978 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:59.989 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 12 16 33663 33647 175.23 106.625 0.00625856 0.00570225 2024-09-21T10:45:00.010 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:00.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:00.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:00.111 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:00.148 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:00.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:00.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:00.250 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:00.281 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:00.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:00.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:00.382 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:00.416 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:00.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:00.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:00.518 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:00.539 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:00.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:00.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:00.643 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:00.676 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:00.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:00.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:00.734 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:00 smithi070 ceph-mon[107385]: pgmap v2363: 89 pgs: 89 active+clean; 1.5 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-09-21T10:45:00.735 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:00 smithi070 ceph-mon[112139]: pgmap v2363: 89 pgs: 89 active+clean; 1.5 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-09-21T10:45:00.778 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:00.803 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:00.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:00.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:00 smithi160 ceph-mon[101879]: pgmap v2363: 89 pgs: 89 active+clean; 1.5 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-09-21T10:45:00.904 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:00.941 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:00.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:00.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:00.989 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 13 16 35579 35563 170.961 119.75 0.00546627 0.00584431 2024-09-21T10:45:01.042 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:01.069 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:01.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:01.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:01.171 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:01.204 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:01.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:01.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:01.306 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:01.329 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:01.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:01.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:01.430 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:01.463 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:01.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:01.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:01.564 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:01.598 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:01.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:01.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:01.700 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:01.731 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:01.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:01.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:01.832 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:01.868 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:01.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:01.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:01.970 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:01.989 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 14 16 37266 37250 166.28 105.438 0.00223731 0.00596073 2024-09-21T10:45:01.994 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:01.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:01.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:02.096 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:02.133 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:02.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:02.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:02.235 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:02.256 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:02.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:02.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:02.357 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:02.388 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:02.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:02.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:02.490 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:02.526 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:02.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:02.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:02.628 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:02.660 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:02.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:02.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:02.762 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:02.795 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:02.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:02.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:02 smithi160 ceph-mon[101879]: pgmap v2364: 89 pgs: 89 active+clean; 1.8 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 164 MiB/s wr, 2.63k op/s 2024-09-21T10:45:02.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:02 smithi070 ceph-mon[107385]: pgmap v2364: 89 pgs: 89 active+clean; 1.8 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 164 MiB/s wr, 2.63k op/s 2024-09-21T10:45:02.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:02 smithi070 ceph-mon[112139]: pgmap v2364: 89 pgs: 89 active+clean; 1.8 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 164 MiB/s wr, 2.63k op/s 2024-09-21T10:45:02.896 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:02.928 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:02.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:02.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:02.989 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 15 16 38946 38930 162.194 105 0.00353019 0.00615953 2024-09-21T10:45:03.029 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:03.062 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:03.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:03.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:03.164 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:03.188 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:03.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:03.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:03.289 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:03.324 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:03.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:03.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:03.426 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:03.446 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:03.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:03.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:03.548 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:03.569 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:03.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:03.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:45:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:45:03] "GET /metrics HTTP/1.1" 200 36533 "" "Prometheus/2.51.0" 2024-09-21T10:45:03.672 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:03.708 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:03.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:03.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:03.809 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:03.839 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":2360,"stamp":"2024-09-21T10:44:54.754171+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1038424037,"num_objects":15862,"num_object_clones":0,"num_object_copies":47586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15862,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":16115,"num_write_kb":1015953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2962150,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16044,"ondisk_log_size":16044,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":270,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":4676684,"kb_used_data":2544916,"kb_used_omap":219,"kb_used_meta":2131428,"kb_avail":651539380,"statfs":{"total":671965249536,"available":667176325120,"internally_reserved":0,"allocated":2605993984,"data_stored":2575860247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":225222,"internal_metadata":2182582330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,4],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1037828096,"num_objects":15836,"num_object_clones":0,"num_object_copies":47508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15836,"num_write_kb":1013504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002516"},"pg_stats":[{"pgid":"54.f","version":"1379'1420","reported_seq":1437,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:54.622140+0000","last_change":"2024-09-21T10:44:32.818675+0000","last_active":"2024-09-21T10:44:54.622140+0000","last_peered":"2024-09-21T10:44:54.622140+0000","last_clean":"2024-09-21T10:44:54.622140+0000","last_became_active":"2024-09-21T10:44:32.818171+0000","last_became_peered":"2024-09-21T10:44:32.818171+0000","last_unstale":"2024-09-21T10:44:54.622140+0000","last_undegraded":"2024-09-21T10:44:54.622140+0000","last_fullsized":"2024-09-21T10:44:54.622140+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":1420,"log_dups_size":0,"ondisk_log_size":1420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:49:54.135275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93061120,"num_objects":1420,"num_object_clones":0,"num_object_copies":4260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1420,"num_write_kb":90880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.e","version":"1379'737","reported_seq":749,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:51.255133+0000","last_change":"2024-09-21T10:44:32.821023+0000","last_active":"2024-09-21T10:44:51.255133+0000","last_peered":"2024-09-21T10:44:51.255133+0000","last_clean":"2024-09-21T10:44:51.255133+0000","last_became_active":"2024-09-21T10:44:32.820651+0000","last_became_peered":"2024-09-21T10:44:32.820651+0000","last_unstale":"2024-09-21T10:44:51.255133+0000","last_undegraded":"2024-09-21T10:44:51.255133+0000","last_fullsized":"2024-09-21T10:44:51.255133+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":737,"log_dups_size":0,"ondisk_log_size":737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:39:13.493921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48300032,"num_objects":737,"num_object_clones":0,"num_object_copies":2211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":737,"num_write_kb":47168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"54.d","version":"1379'758","reported_seq":769,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:51.266008+0000","last_change":"2024-09-21T10:44:32.816573+0000","last_active":"2024-09-21T10:44:51.266008+0000","last_peered":"2024-09-21T10:44:51.266008+0000","last_clean":"2024-09-21T10:44:51.266008+0000","last_became_active":"2024-09-21T10:44:32.816260+0000","last_became_peered":"2024-09-21T10:44:32.816260+0000","last_unstale":"2024-09-21T10:44:51.266008+0000","last_undegraded":"2024-09-21T10:44:51.266008+0000","last_fullsized":"2024-09-21T10:44:51.266008+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":758,"log_dups_size":0,"ondisk_log_size":758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:20:47.764042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49676288,"num_objects":758,"num_object_clones":0,"num_object_copies":2274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":758,"num_write_kb":48512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"54.c","version":"1379'1325","reported_seq":1342,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:54.621142+0000","last_change":"2024-09-21T10:44:32.819873+0000","last_active":"2024-09-21T10:44:54.621142+0000","last_peered":"2024-09-21T10:44:54.621142+0000","last_clean":"2024-09-21T10:44:54.621142+0000","last_became_active":"2024-09-21T10:44:32.819256+0000","last_became_peered":"2024-09-21T10:44:32.819256+0000","last_unstale":"2024-09-21T10:44:54.621142+0000","last_undegraded":"2024-09-21T10:44:54.621142+0000","last_fullsized":"2024-09-21T10:44:54.621142+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":1325,"log_dups_size":0,"ondisk_log_size":1325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:53:13.023451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86835200,"num_objects":1325,"num_object_clones":0,"num_object_copies":3975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1325,"num_write_kb":84800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.b","version":"1379'1381","reported_seq":1398,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:54.619613+0000","last_change":"2024-09-21T10:44:32.819321+0000","last_active":"2024-09-21T10:44:54.619613+0000","last_peered":"2024-09-21T10:44:54.619613+0000","last_clean":"2024-09-21T10:44:54.619613+0000","last_became_active":"2024-09-21T10:44:32.819084+0000","last_became_peered":"2024-09-21T10:44:32.819084+0000","last_unstale":"2024-09-21T10:44:54.619613+0000","last_undegraded":"2024-09-21T10:44:54.619613+0000","last_fullsized":"2024-09-21T10:44:54.619613+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":1381,"log_dups_size":0,"ondisk_log_size":1381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:42.595258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90505216,"num_objects":1381,"num_object_clones":0,"num_object_copies":4143,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1381,"num_write_kb":88384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.a","version":"1379'786","reported_seq":797,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:51.263633+0000","last_change":"2024-09-21T10:44:32.818041+0000","last_active":"2024-09-21T10:44:51.263633+0000","last_peered":"2024-09-21T10:44:51.263633+0000","last_clean":"2024-09-21T10:44:51.263633+0000","last_became_active":"2024-09-21T10:44:32.817725+0000","last_became_peered":"2024-09-21T10:44:32.817725+0000","last_unstale":"2024-09-21T10:44:51.263633+0000","last_undegraded":"2024-09-21T10:44:51.263633+0000","last_fullsized":"2024-09-21T10:44:51.263633+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":786,"log_dups_size":0,"ondisk_log_size":786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:22:24.084717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51511296,"num_objects":786,"num_object_clones":0,"num_object_copies":2358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":786,"num_write_kb":50304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"54.9","version":"1379'1317","reported_seq":1334,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:54.386020+0000","last_change":"2024-09-21T10:44:32.816734+0000","last_active":"2024-09-21T10:44:54.386020+0000","last_peered":"2024-09-21T10:44:54.386020+0000","last_clean":"2024-09-21T10:44:54.386020+0000","last_became_active":"2024-09-21T10:44:32.816261+0000","last_became_peered":"2024-09-21T10:44:32.816261+0000","last_unstale":"2024-09-21T10:44:54.386020+0000","last_undegraded":"2024-09-21T10:44:54.386020+0000","last_fullsized":"2024-09-21T10:44:54.386020+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":1317,"log_dups_size":0,"ondisk_log_size":1317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:32:36.453794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86310912,"num_objects":1317,"num_object_clones":0,"num_object_copies":3951,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1317,"num_write_kb":84288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"54.8","version":"1379'733","reported_seq":745,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:51.254084+0000","last_change":"2024-09-21T10:44:32.818221+0000","last_active":"2024-09-21T10:44:51.254084+0000","last_peered":"2024-09-21T10:44:51.254084+0000","last_clean":"2024-09-21T10:44:51.254084+0000","last_became_active":"2024-09-21T10:44:32.817941+0000","last_became_peered":"2024-09-21T10:44:32.817941+0000","last_unstale":"2024-09-21T10:44:51.254084+0000","last_undegraded":"2024-09-21T10:44:51.254084+0000","last_fullsized":"2024-09-21T10:44:51.254084+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":733,"log_dups_size":0,"ondisk_log_size":733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:11:40.024092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48037888,"num_objects":733,"num_object_clones":0,"num_object_copies":2199,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":733,"num_write_kb":46912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"54.7","version":"1379'1391","reported_seq":1408,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:54.600573+0000","last_change":"2024-09-21T10:44:32.818432+0000","last_active":"2024-09-21T10:44:54.600573+0000","last_peered":"2024-09-21T10:44:54.600573+0000","last_clean":"2024-09-21T10:44:54.600573+0000","last_became_active":"2024-09-21T10:44:32.818094+0000","last_became_peered":"2024-09-21T10:44:32.818094+0000","last_unstale":"2024-09-21T10:44:54.600573+0000","last_undegraded":"2024-09-21T10:44:54.600573+0000","last_fullsized":"2024-09-21T10:44:54.600573+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":1391,"log_dups_size":0,"ondisk_log_size":1391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:41:17.258046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91160576,"num_objects":1391,"num_object_clones":0,"num_object_copies":4173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1391,"num_write_kb":89024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.6","version":"1379'706","reported_seq":718,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:51.008844+0000","last_change":"2024-09-21T10:44:32.822589+0000","last_active":"2024-09-21T10:44:51.008844+0000","last_peered":"2024-09-21T10:44:51.008844+0000","last_clean":"2024-09-21T10:44:51.008844+0000","last_became_active":"2024-09-21T10:44:32.822265+0000","last_became_peered":"2024-09-21T10:44:32.822265+0000","last_unstale":"2024-09-21T10:44:51.008844+0000","last_undegraded":"2024-09-21T10:44:51.008844+0000","last_fullsized":"2024-09-21T10:44:51.008844+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:40:44.356501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46268416,"num_objects":706,"num_object_clones":0,"num_object_copies":2118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":706,"num_write_kb":45184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"54.5","version":"1379'728","reported_seq":740,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:51.255452+0000","last_change":"2024-09-21T10:44:32.815686+0000","last_active":"2024-09-21T10:44:51.255452+0000","last_peered":"2024-09-21T10:44:51.255452+0000","last_clean":"2024-09-21T10:44:51.255452+0000","last_became_active":"2024-09-21T10:44:32.815266+0000","last_became_peered":"2024-09-21T10:44:32.815266+0000","last_unstale":"2024-09-21T10:44:51.255452+0000","last_undegraded":"2024-09-21T10:44:51.255452+0000","last_fullsized":"2024-09-21T10:44:51.255452+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":728,"log_dups_size":0,"ondisk_log_size":728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:11:59.857675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47710208,"num_objects":728,"num_object_clones":0,"num_object_copies":2184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":728,"num_write_kb":46592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"54.4","version":"1379'1335","reported_seq":1350,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:54.617714+0000","last_change":"2024-09-21T10:44:32.821087+0000","last_active":"2024-09-21T10:44:54.617714+0000","last_peered":"2024-09-21T10:44:54.617714+0000","last_clean":"2024-09-21T10:44:54.617714+0000","last_became_active":"2024-09-21T10:44:32.820712+0000","last_became_peered":"2024-09-21T10:44:32.820712+0000","last_unstale":"2024-09-21T10:44:54.617714+0000","last_undegraded":"2024-09-21T10:44:54.617714+0000","last_fullsized":"2024-09-21T10:44:54.617714+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":1335,"log_dups_size":0,"ondisk_log_size":1335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:41:13.083038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87490560,"num_objects":1335,"num_object_clones":0,"num_object_copies":4005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1335,"num_write_kb":85440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.3","version":"1379'1319","reported_seq":1336,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:54.405998+0000","last_change":"2024-09-21T10:44:32.817547+0000","last_active":"2024-09-21T10:44:54.405998+0000","last_peered":"2024-09-21T10:44:54.405998+0000","last_clean":"2024-09-21T10:44:54.405998+0000","last_became_active":"2024-09-21T10:44:32.817238+0000","last_became_peered":"2024-09-21T10:44:32.817238+0000","last_unstale":"2024-09-21T10:44:54.405998+0000","last_undegraded":"2024-09-21T10:44:54.405998+0000","last_fullsized":"2024-09-21T10:44:54.405998+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":1319,"log_dups_size":0,"ondisk_log_size":1319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:51:18.419600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86441984,"num_objects":1319,"num_object_clones":0,"num_object_copies":3957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1319,"num_write_kb":84416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"54.2","version":"1379'449","reported_seq":466,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:49.815042+0000","last_change":"2024-09-21T10:44:32.820419+0000","last_active":"2024-09-21T10:44:49.815042+0000","last_peered":"2024-09-21T10:44:49.815042+0000","last_clean":"2024-09-21T10:44:49.815042+0000","last_became_active":"2024-09-21T10:44:32.820081+0000","last_became_peered":"2024-09-21T10:44:32.820081+0000","last_unstale":"2024-09-21T10:44:49.815042+0000","last_undegraded":"2024-09-21T10:44:49.815042+0000","last_fullsized":"2024-09-21T10:44:49.815042+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":449,"log_dups_size":0,"ondisk_log_size":449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:27.339291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29425664,"num_objects":449,"num_object_clones":0,"num_object_copies":1347,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":449,"num_write_kb":28736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"54.1","version":"1379'735","reported_seq":747,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:51.138620+0000","last_change":"2024-09-21T10:44:32.819407+0000","last_active":"2024-09-21T10:44:51.138620+0000","last_peered":"2024-09-21T10:44:51.138620+0000","last_clean":"2024-09-21T10:44:51.138620+0000","last_became_active":"2024-09-21T10:44:32.819056+0000","last_became_peered":"2024-09-21T10:44:32.819056+0000","last_unstale":"2024-09-21T10:44:51.138620+0000","last_undegraded":"2024-09-21T10:44:51.138620+0000","last_fullsized":"2024-09-21T10:44:51.138620+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":735,"log_dups_size":0,"ondisk_log_size":735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:39:58.641282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48168960,"num_objects":735,"num_object_clones":0,"num_object_copies":2205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":735,"num_write_kb":47040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"54.0","version":"1379'716","reported_seq":728,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:51.260126+0000","last_change":"2024-09-21T10:44:32.816008+0000","last_active":"2024-09-21T10:44:51.260126+0000","last_peered":"2024-09-21T10:44:51.260126+0000","last_clean":"2024-09-21T10:44:51.260126+0000","last_became_active":"2024-09-21T10:44:32.815627+0000","last_became_peered":"2024-09-21T10:44:32.815627+0000","last_unstale":"2024-09-21T10:44:51.260126+0000","last_undegraded":"2024-09-21T10:44:51.260126+0000","last_fullsized":"2024-09-21T10:44:51.260126+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":716,"log_dups_size":0,"ondisk_log_size":716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:48:54.596883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46923776,"num_objects":716,"num_object_clones":0,"num_object_copies":2148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":716,"num_write_kb":45824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":2069,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.536606+0000","last_change":"2024-09-21T10:40:13.438023+0000","last_active":"2024-09-21T10:44:45.536606+0000","last_peered":"2024-09-21T10:44:45.536606+0000","last_clean":"2024-09-21T10:44:45.536606+0000","last_became_active":"2024-09-21T10:40:13.431103+0000","last_became_peered":"2024-09-21T10:40:13.431103+0000","last_unstale":"2024-09-21T10:44:45.536606+0000","last_undegraded":"2024-09-21T10:44:45.536606+0000","last_fullsized":"2024-09-21T10:44:45.536606+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:00:24.887661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1347,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:34.315114+0000","last_change":"2024-09-21T10:07:55.999392+0000","last_active":"2024-09-21T10:44:34.315114+0000","last_peered":"2024-09-21T10:44:34.315114+0000","last_clean":"2024-09-21T10:44:34.315114+0000","last_became_active":"2024-09-21T10:07:55.999103+0000","last_became_peered":"2024-09-21T10:07:55.999103+0000","last_unstale":"2024-09-21T10:44:34.315114+0000","last_undegraded":"2024-09-21T10:44:34.315114+0000","last_fullsized":"2024-09-21T10:44:34.315114+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:48.757242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":2050,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.649383+0000","last_change":"2024-09-21T10:40:13.432818+0000","last_active":"2024-09-21T10:44:45.649383+0000","last_peered":"2024-09-21T10:44:45.649383+0000","last_clean":"2024-09-21T10:44:45.649383+0000","last_became_active":"2024-09-21T10:40:13.432448+0000","last_became_peered":"2024-09-21T10:40:13.432448+0000","last_unstale":"2024-09-21T10:44:45.649383+0000","last_undegraded":"2024-09-21T10:44:45.649383+0000","last_fullsized":"2024-09-21T10:44:45.649383+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:20.448242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2067,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.648869+0000","last_change":"2024-09-21T10:40:13.431824+0000","last_active":"2024-09-21T10:44:45.648869+0000","last_peered":"2024-09-21T10:44:45.648869+0000","last_clean":"2024-09-21T10:44:45.648869+0000","last_became_active":"2024-09-21T10:40:13.431516+0000","last_became_peered":"2024-09-21T10:40:13.431516+0000","last_unstale":"2024-09-21T10:44:45.648869+0000","last_undegraded":"2024-09-21T10:44:45.648869+0000","last_fullsized":"2024-09-21T10:44:45.648869+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:30:40.226452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":1334,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:34.315163+0000","last_change":"2024-09-21T09:54:53.634508+0000","last_active":"2024-09-21T10:44:34.315163+0000","last_peered":"2024-09-21T10:44:34.315163+0000","last_clean":"2024-09-21T10:44:34.315163+0000","last_became_active":"2024-09-21T09:54:53.634254+0000","last_became_peered":"2024-09-21T09:54:53.634254+0000","last_unstale":"2024-09-21T10:44:34.315163+0000","last_undegraded":"2024-09-21T10:44:34.315163+0000","last_fullsized":"2024-09-21T10:44:34.315163+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:18.122107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1361,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:34.314643+0000","last_change":"2024-09-21T10:09:06.574933+0000","last_active":"2024-09-21T10:44:34.314643+0000","last_peered":"2024-09-21T10:44:34.314643+0000","last_clean":"2024-09-21T10:44:34.314643+0000","last_became_active":"2024-09-21T10:09:06.574467+0000","last_became_peered":"2024-09-21T10:09:06.574467+0000","last_unstale":"2024-09-21T10:44:34.314643+0000","last_undegraded":"2024-09-21T10:44:34.314643+0000","last_fullsized":"2024-09-21T10:44:34.314643+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:06.142269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":1349,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:34.315376+0000","last_change":"2024-09-21T10:07:55.991189+0000","last_active":"2024-09-21T10:44:34.315376+0000","last_peered":"2024-09-21T10:44:34.315376+0000","last_clean":"2024-09-21T10:44:34.315376+0000","last_became_active":"2024-09-21T10:07:55.990784+0000","last_became_peered":"2024-09-21T10:07:55.990784+0000","last_unstale":"2024-09-21T10:44:34.315376+0000","last_undegraded":"2024-09-21T10:44:34.315376+0000","last_fullsized":"2024-09-21T10:44:34.315376+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:00.580471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1361,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:34.315150+0000","last_change":"2024-09-21T10:09:06.560175+0000","last_active":"2024-09-21T10:44:34.315150+0000","last_peered":"2024-09-21T10:44:34.315150+0000","last_clean":"2024-09-21T10:44:34.315150+0000","last_became_active":"2024-09-21T10:09:06.558584+0000","last_became_peered":"2024-09-21T10:09:06.558584+0000","last_unstale":"2024-09-21T10:44:34.315150+0000","last_undegraded":"2024-09-21T10:44:34.315150+0000","last_fullsized":"2024-09-21T10:44:34.315150+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:02.918339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":2052,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.536934+0000","last_change":"2024-09-21T10:44:14.409321+0000","last_active":"2024-09-21T10:44:45.536934+0000","last_peered":"2024-09-21T10:44:45.536934+0000","last_clean":"2024-09-21T10:44:45.536934+0000","last_became_active":"2024-09-21T10:44:14.408833+0000","last_became_peered":"2024-09-21T10:44:14.408833+0000","last_unstale":"2024-09-21T10:44:45.536934+0000","last_undegraded":"2024-09-21T10:44:45.536934+0000","last_fullsized":"2024-09-21T10:44:45.536934+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:10:00.997210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":829,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.568513+0000","last_change":"2024-09-21T10:40:13.436927+0000","last_active":"2024-09-21T10:44:45.568513+0000","last_peered":"2024-09-21T10:44:45.568513+0000","last_clean":"2024-09-21T10:44:45.568513+0000","last_became_active":"2024-09-21T10:40:13.436329+0000","last_became_peered":"2024-09-21T10:40:13.436329+0000","last_unstale":"2024-09-21T10:44:45.568513+0000","last_undegraded":"2024-09-21T10:44:45.568513+0000","last_fullsized":"2024-09-21T10:44:45.568513+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:05.668898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":1372,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:34.314872+0000","last_change":"2024-09-21T10:40:13.433278+0000","last_active":"2024-09-21T10:44:34.314872+0000","last_peered":"2024-09-21T10:44:34.314872+0000","last_clean":"2024-09-21T10:44:34.314872+0000","last_became_active":"2024-09-21T10:40:13.432814+0000","last_became_peered":"2024-09-21T10:40:13.432814+0000","last_unstale":"2024-09-21T10:44:34.314872+0000","last_undegraded":"2024-09-21T10:44:34.314872+0000","last_fullsized":"2024-09-21T10:44:34.314872+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:55.909872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2061,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.649405+0000","last_change":"2024-09-21T10:40:13.432531+0000","last_active":"2024-09-21T10:44:45.649405+0000","last_peered":"2024-09-21T10:44:45.649405+0000","last_clean":"2024-09-21T10:44:45.649405+0000","last_became_active":"2024-09-21T10:40:13.432210+0000","last_became_peered":"2024-09-21T10:40:13.432210+0000","last_unstale":"2024-09-21T10:44:45.649405+0000","last_undegraded":"2024-09-21T10:44:45.649405+0000","last_fullsized":"2024-09-21T10:44:45.649405+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:20:10.998607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2033,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.568299+0000","last_change":"2024-09-21T10:09:06.530394+0000","last_active":"2024-09-21T10:44:45.568299+0000","last_peered":"2024-09-21T10:44:45.568299+0000","last_clean":"2024-09-21T10:44:45.568299+0000","last_became_active":"2024-09-21T10:09:06.529307+0000","last_became_peered":"2024-09-21T10:09:06.529307+0000","last_unstale":"2024-09-21T10:44:45.568299+0000","last_undegraded":"2024-09-21T10:44:45.568299+0000","last_fullsized":"2024-09-21T10:44:45.568299+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:54.287397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2065,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.537125+0000","last_change":"2024-09-21T10:44:14.409422+0000","last_active":"2024-09-21T10:44:45.537125+0000","last_peered":"2024-09-21T10:44:45.537125+0000","last_clean":"2024-09-21T10:44:45.537125+0000","last_became_active":"2024-09-21T10:44:14.408726+0000","last_became_peered":"2024-09-21T10:44:14.408726+0000","last_unstale":"2024-09-21T10:44:45.537125+0000","last_undegraded":"2024-09-21T10:44:45.537125+0000","last_fullsized":"2024-09-21T10:44:45.537125+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:44:21.743112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2047,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.648997+0000","last_change":"2024-09-21T10:09:06.552324+0000","last_active":"2024-09-21T10:44:45.648997+0000","last_peered":"2024-09-21T10:44:45.648997+0000","last_clean":"2024-09-21T10:44:45.648997+0000","last_became_active":"2024-09-21T10:09:06.540545+0000","last_became_peered":"2024-09-21T10:09:06.540545+0000","last_unstale":"2024-09-21T10:44:45.648997+0000","last_undegraded":"2024-09-21T10:44:45.648997+0000","last_fullsized":"2024-09-21T10:44:45.648997+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:08.506149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1351,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:34.256074+0000","last_change":"2024-09-21T10:07:56.003005+0000","last_active":"2024-09-21T10:44:34.256074+0000","last_peered":"2024-09-21T10:44:34.256074+0000","last_clean":"2024-09-21T10:44:34.256074+0000","last_became_active":"2024-09-21T10:07:56.001550+0000","last_became_peered":"2024-09-21T10:07:56.001550+0000","last_unstale":"2024-09-21T10:44:34.256074+0000","last_undegraded":"2024-09-21T10:44:34.256074+0000","last_fullsized":"2024-09-21T10:44:34.256074+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:54.553261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2040,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.649082+0000","last_change":"2024-09-21T10:09:06.559017+0000","last_active":"2024-09-21T10:44:45.649082+0000","last_peered":"2024-09-21T10:44:45.649082+0000","last_clean":"2024-09-21T10:44:45.649082+0000","last_became_active":"2024-09-21T10:09:06.555256+0000","last_became_peered":"2024-09-21T10:09:06.555256+0000","last_unstale":"2024-09-21T10:44:45.649082+0000","last_undegraded":"2024-09-21T10:44:45.649082+0000","last_fullsized":"2024-09-21T10:44:45.649082+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:33:45.145138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2045,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.537160+0000","last_change":"2024-09-21T10:09:06.562504+0000","last_active":"2024-09-21T10:44:45.537160+0000","last_peered":"2024-09-21T10:44:45.537160+0000","last_clean":"2024-09-21T10:44:45.537160+0000","last_became_active":"2024-09-21T10:09:06.561747+0000","last_became_peered":"2024-09-21T10:09:06.561747+0000","last_unstale":"2024-09-21T10:44:45.537160+0000","last_undegraded":"2024-09-21T10:44:45.537160+0000","last_fullsized":"2024-09-21T10:44:45.537160+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:03:58.012406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2045,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.649216+0000","last_change":"2024-09-21T10:44:14.399948+0000","last_active":"2024-09-21T10:44:45.649216+0000","last_peered":"2024-09-21T10:44:45.649216+0000","last_clean":"2024-09-21T10:44:45.649216+0000","last_became_active":"2024-09-21T10:44:14.399337+0000","last_became_peered":"2024-09-21T10:44:14.399337+0000","last_unstale":"2024-09-21T10:44:45.649216+0000","last_undegraded":"2024-09-21T10:44:45.649216+0000","last_fullsized":"2024-09-21T10:44:45.649216+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:40.185020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1366,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:34.315194+0000","last_change":"2024-09-21T10:44:14.401716+0000","last_active":"2024-09-21T10:44:34.315194+0000","last_peered":"2024-09-21T10:44:34.315194+0000","last_clean":"2024-09-21T10:44:34.315194+0000","last_became_active":"2024-09-21T10:44:14.401254+0000","last_became_peered":"2024-09-21T10:44:14.401254+0000","last_unstale":"2024-09-21T10:44:34.315194+0000","last_undegraded":"2024-09-21T10:44:34.315194+0000","last_fullsized":"2024-09-21T10:44:34.315194+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:07:21.536272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":26,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:33.811764+0000","last_change":"2024-09-21T10:44:14.424249+0000","last_active":"2024-09-21T10:44:33.811764+0000","last_peered":"2024-09-21T10:44:33.811764+0000","last_clean":"2024-09-21T10:44:33.811764+0000","last_became_active":"2024-09-21T10:44:14.408884+0000","last_became_peered":"2024-09-21T10:44:14.408884+0000","last_unstale":"2024-09-21T10:44:33.811764+0000","last_undegraded":"2024-09-21T10:44:33.811764+0000","last_fullsized":"2024-09-21T10:44:33.811764+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:18:44.759435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2039,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.568328+0000","last_change":"2024-09-21T10:40:13.437741+0000","last_active":"2024-09-21T10:44:45.568328+0000","last_peered":"2024-09-21T10:44:45.568328+0000","last_clean":"2024-09-21T10:44:45.568328+0000","last_became_active":"2024-09-21T10:40:13.437447+0000","last_became_peered":"2024-09-21T10:40:13.437447+0000","last_unstale":"2024-09-21T10:44:45.568328+0000","last_undegraded":"2024-09-21T10:44:45.568328+0000","last_fullsized":"2024-09-21T10:44:45.568328+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:40:47.072907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":1395,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:34.314920+0000","last_change":"2024-09-21T10:44:14.415264+0000","last_active":"2024-09-21T10:44:34.314920+0000","last_peered":"2024-09-21T10:44:34.314920+0000","last_clean":"2024-09-21T10:44:34.314920+0000","last_became_active":"2024-09-21T10:44:14.407813+0000","last_became_peered":"2024-09-21T10:44:14.407813+0000","last_unstale":"2024-09-21T10:44:34.314920+0000","last_undegraded":"2024-09-21T10:44:34.314920+0000","last_fullsized":"2024-09-21T10:44:34.314920+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:55.933601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1348,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:34.315175+0000","last_change":"2024-09-21T10:07:56.001756+0000","last_active":"2024-09-21T10:44:34.315175+0000","last_peered":"2024-09-21T10:44:34.315175+0000","last_clean":"2024-09-21T10:44:34.315175+0000","last_became_active":"2024-09-21T10:07:56.000789+0000","last_became_peered":"2024-09-21T10:07:56.000789+0000","last_unstale":"2024-09-21T10:44:34.315175+0000","last_undegraded":"2024-09-21T10:44:34.315175+0000","last_fullsized":"2024-09-21T10:44:34.315175+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:56:04.302902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1335,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:34.314651+0000","last_change":"2024-09-21T09:54:53.636902+0000","last_active":"2024-09-21T10:44:34.314651+0000","last_peered":"2024-09-21T10:44:34.314651+0000","last_clean":"2024-09-21T10:44:34.314651+0000","last_became_active":"2024-09-21T09:54:53.636616+0000","last_became_peered":"2024-09-21T09:54:53.636616+0000","last_unstale":"2024-09-21T10:44:34.314651+0000","last_undegraded":"2024-09-21T10:44:34.314651+0000","last_fullsized":"2024-09-21T10:44:34.314651+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:46:47.694495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1391,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:34.256107+0000","last_change":"2024-09-21T10:44:14.402059+0000","last_active":"2024-09-21T10:44:34.256107+0000","last_peered":"2024-09-21T10:44:34.256107+0000","last_clean":"2024-09-21T10:44:34.256107+0000","last_became_active":"2024-09-21T10:44:14.401710+0000","last_became_peered":"2024-09-21T10:44:14.401710+0000","last_unstale":"2024-09-21T10:44:34.256107+0000","last_undegraded":"2024-09-21T10:44:34.256107+0000","last_fullsized":"2024-09-21T10:44:34.256107+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:17.314693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":2053,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.537037+0000","last_change":"2024-09-21T10:07:56.023462+0000","last_active":"2024-09-21T10:44:45.537037+0000","last_peered":"2024-09-21T10:44:45.537037+0000","last_clean":"2024-09-21T10:44:45.537037+0000","last_became_active":"2024-09-21T10:07:56.021859+0000","last_became_peered":"2024-09-21T10:07:56.021859+0000","last_unstale":"2024-09-21T10:44:45.537037+0000","last_undegraded":"2024-09-21T10:44:45.537037+0000","last_fullsized":"2024-09-21T10:44:45.537037+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:42:03.134729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2041,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.649053+0000","last_change":"2024-09-21T10:44:14.399028+0000","last_active":"2024-09-21T10:44:45.649053+0000","last_peered":"2024-09-21T10:44:45.649053+0000","last_clean":"2024-09-21T10:44:45.649053+0000","last_became_active":"2024-09-21T10:44:14.398115+0000","last_became_peered":"2024-09-21T10:44:14.398115+0000","last_unstale":"2024-09-21T10:44:45.649053+0000","last_undegraded":"2024-09-21T10:44:45.649053+0000","last_fullsized":"2024-09-21T10:44:45.649053+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:05.170376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2034,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.568634+0000","last_change":"2024-09-21T10:07:56.006236+0000","last_active":"2024-09-21T10:44:45.568634+0000","last_peered":"2024-09-21T10:44:45.568634+0000","last_clean":"2024-09-21T10:44:45.568634+0000","last_became_active":"2024-09-21T10:07:56.004439+0000","last_became_peered":"2024-09-21T10:07:56.004439+0000","last_unstale":"2024-09-21T10:44:45.568634+0000","last_undegraded":"2024-09-21T10:44:45.568634+0000","last_fullsized":"2024-09-21T10:44:45.568634+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:39.422467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":2050,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.649745+0000","last_change":"2024-09-21T10:44:14.410440+0000","last_active":"2024-09-21T10:44:45.649745+0000","last_peered":"2024-09-21T10:44:45.649745+0000","last_clean":"2024-09-21T10:44:45.649745+0000","last_became_active":"2024-09-21T10:44:14.400035+0000","last_became_peered":"2024-09-21T10:44:14.400035+0000","last_unstale":"2024-09-21T10:44:45.649745+0000","last_undegraded":"2024-09-21T10:44:45.649745+0000","last_fullsized":"2024-09-21T10:44:45.649745+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:49:39.536516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2053,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.537006+0000","last_change":"2024-09-21T10:44:14.409410+0000","last_active":"2024-09-21T10:44:45.537006+0000","last_peered":"2024-09-21T10:44:45.537006+0000","last_clean":"2024-09-21T10:44:45.537006+0000","last_became_active":"2024-09-21T10:44:14.409025+0000","last_became_peered":"2024-09-21T10:44:14.409025+0000","last_unstale":"2024-09-21T10:44:45.537006+0000","last_undegraded":"2024-09-21T10:44:45.537006+0000","last_fullsized":"2024-09-21T10:44:45.537006+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:25:09.197455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:33.811628+0000","last_change":"2024-09-21T10:44:14.405677+0000","last_active":"2024-09-21T10:44:33.811628+0000","last_peered":"2024-09-21T10:44:33.811628+0000","last_clean":"2024-09-21T10:44:33.811628+0000","last_became_active":"2024-09-21T10:44:14.405262+0000","last_became_peered":"2024-09-21T10:44:14.405262+0000","last_unstale":"2024-09-21T10:44:33.811628+0000","last_undegraded":"2024-09-21T10:44:33.811628+0000","last_fullsized":"2024-09-21T10:44:33.811628+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:53:12.127245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2069,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.567966+0000","last_change":"2024-09-21T10:40:13.437609+0000","last_active":"2024-09-21T10:44:45.567966+0000","last_peered":"2024-09-21T10:44:45.567966+0000","last_clean":"2024-09-21T10:44:45.567966+0000","last_became_active":"2024-09-21T10:40:13.437296+0000","last_became_peered":"2024-09-21T10:40:13.437296+0000","last_unstale":"2024-09-21T10:44:45.567966+0000","last_undegraded":"2024-09-21T10:44:45.567966+0000","last_fullsized":"2024-09-21T10:44:45.567966+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:41:58.963386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2024,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.568228+0000","last_change":"2024-09-21T10:07:56.067772+0000","last_active":"2024-09-21T10:44:45.568228+0000","last_peered":"2024-09-21T10:44:45.568228+0000","last_clean":"2024-09-21T10:44:45.568228+0000","last_became_active":"2024-09-21T10:07:56.052602+0000","last_became_peered":"2024-09-21T10:07:56.052602+0000","last_unstale":"2024-09-21T10:44:45.568228+0000","last_undegraded":"2024-09-21T10:44:45.568228+0000","last_fullsized":"2024-09-21T10:44:45.568228+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:07:59.899214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2092,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.568330+0000","last_change":"2024-09-21T10:44:14.405182+0000","last_active":"2024-09-21T10:44:45.568330+0000","last_peered":"2024-09-21T10:44:45.568330+0000","last_clean":"2024-09-21T10:44:45.568330+0000","last_became_active":"2024-09-21T10:44:14.404852+0000","last_became_peered":"2024-09-21T10:44:14.404852+0000","last_unstale":"2024-09-21T10:44:45.568330+0000","last_undegraded":"2024-09-21T10:44:45.568330+0000","last_fullsized":"2024-09-21T10:44:45.568330+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:51.687206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1375,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:34.315325+0000","last_change":"2024-09-21T10:40:13.435375+0000","last_active":"2024-09-21T10:44:34.315325+0000","last_peered":"2024-09-21T10:44:34.315325+0000","last_clean":"2024-09-21T10:44:34.315325+0000","last_became_active":"2024-09-21T10:40:13.435050+0000","last_became_peered":"2024-09-21T10:40:13.435050+0000","last_unstale":"2024-09-21T10:44:34.315325+0000","last_undegraded":"2024-09-21T10:44:34.315325+0000","last_fullsized":"2024-09-21T10:44:34.315325+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:57.889610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2079,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.568565+0000","last_change":"2024-09-21T10:40:13.434189+0000","last_active":"2024-09-21T10:44:45.568565+0000","last_peered":"2024-09-21T10:44:45.568565+0000","last_clean":"2024-09-21T10:44:45.568565+0000","last_became_active":"2024-09-21T10:40:13.433892+0000","last_became_peered":"2024-09-21T10:40:13.433892+0000","last_unstale":"2024-09-21T10:44:45.568565+0000","last_undegraded":"2024-09-21T10:44:45.568565+0000","last_fullsized":"2024-09-21T10:44:45.568565+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:49:17.296220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1351,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:34.315235+0000","last_change":"2024-09-21T10:07:21.524082+0000","last_active":"2024-09-21T10:44:34.315235+0000","last_peered":"2024-09-21T10:44:34.315235+0000","last_clean":"2024-09-21T10:44:34.315235+0000","last_became_active":"2024-09-21T10:07:21.523779+0000","last_became_peered":"2024-09-21T10:07:21.523779+0000","last_unstale":"2024-09-21T10:44:34.315235+0000","last_undegraded":"2024-09-21T10:44:34.315235+0000","last_fullsized":"2024-09-21T10:44:34.315235+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:13.813832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":1366,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:34.314589+0000","last_change":"2024-09-21T10:07:56.024239+0000","last_active":"2024-09-21T10:44:34.314589+0000","last_peered":"2024-09-21T10:44:34.314589+0000","last_clean":"2024-09-21T10:44:34.314589+0000","last_became_active":"2024-09-21T10:07:56.023914+0000","last_became_peered":"2024-09-21T10:07:56.023914+0000","last_unstale":"2024-09-21T10:44:34.314589+0000","last_undegraded":"2024-09-21T10:44:34.314589+0000","last_fullsized":"2024-09-21T10:44:34.314589+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:23.361736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084477399999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":70,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:34.315505+0000","last_change":"2024-09-21T10:40:55.616051+0000","last_active":"2024-09-21T10:44:34.315505+0000","last_peered":"2024-09-21T10:44:34.315505+0000","last_clean":"2024-09-21T10:44:34.315505+0000","last_became_active":"2024-09-21T10:40:55.615741+0000","last_became_peered":"2024-09-21T10:40:55.615741+0000","last_unstale":"2024-09-21T10:44:34.315505+0000","last_undegraded":"2024-09-21T10:44:34.315505+0000","last_fullsized":"2024-09-21T10:44:34.315505+0000","mapping_epoch":1320,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:07.673527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2043,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.568010+0000","last_change":"2024-09-21T10:40:13.436234+0000","last_active":"2024-09-21T10:44:45.568010+0000","last_peered":"2024-09-21T10:44:45.568010+0000","last_clean":"2024-09-21T10:44:45.568010+0000","last_became_active":"2024-09-21T10:40:13.435946+0000","last_became_peered":"2024-09-21T10:40:13.435946+0000","last_unstale":"2024-09-21T10:44:45.568010+0000","last_undegraded":"2024-09-21T10:44:45.568010+0000","last_fullsized":"2024-09-21T10:44:45.568010+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:08:08.670739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":836,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.649005+0000","last_change":"2024-09-21T10:40:13.431693+0000","last_active":"2024-09-21T10:44:45.649005+0000","last_peered":"2024-09-21T10:44:45.649005+0000","last_clean":"2024-09-21T10:44:45.649005+0000","last_became_active":"2024-09-21T10:40:13.431389+0000","last_became_peered":"2024-09-21T10:40:13.431389+0000","last_unstale":"2024-09-21T10:44:45.649005+0000","last_undegraded":"2024-09-21T10:44:45.649005+0000","last_fullsized":"2024-09-21T10:44:45.649005+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:22.227336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2082,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.537179+0000","last_change":"2024-09-21T10:44:14.409601+0000","last_active":"2024-09-21T10:44:45.537179+0000","last_peered":"2024-09-21T10:44:45.537179+0000","last_clean":"2024-09-21T10:44:45.537179+0000","last_became_active":"2024-09-21T10:44:14.409262+0000","last_became_peered":"2024-09-21T10:44:14.409262+0000","last_unstale":"2024-09-21T10:44:45.537179+0000","last_undegraded":"2024-09-21T10:44:45.537179+0000","last_fullsized":"2024-09-21T10:44:45.537179+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:53:01.809847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2048,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.536791+0000","last_change":"2024-09-21T10:09:06.560130+0000","last_active":"2024-09-21T10:44:45.536791+0000","last_peered":"2024-09-21T10:44:45.536791+0000","last_clean":"2024-09-21T10:44:45.536791+0000","last_became_active":"2024-09-21T10:09:06.559827+0000","last_became_peered":"2024-09-21T10:09:06.559827+0000","last_unstale":"2024-09-21T10:44:45.536791+0000","last_undegraded":"2024-09-21T10:44:45.536791+0000","last_fullsized":"2024-09-21T10:44:45.536791+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:05:17.166734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:33.811391+0000","last_change":"2024-09-21T10:44:14.400691+0000","last_active":"2024-09-21T10:44:33.811391+0000","last_peered":"2024-09-21T10:44:33.811391+0000","last_clean":"2024-09-21T10:44:33.811391+0000","last_became_active":"2024-09-21T10:44:14.400377+0000","last_became_peered":"2024-09-21T10:44:14.400377+0000","last_unstale":"2024-09-21T10:44:33.811391+0000","last_undegraded":"2024-09-21T10:44:33.811391+0000","last_fullsized":"2024-09-21T10:44:33.811391+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:03.555078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1354,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:34.256143+0000","last_change":"2024-09-21T10:44:14.402965+0000","last_active":"2024-09-21T10:44:34.256143+0000","last_peered":"2024-09-21T10:44:34.256143+0000","last_clean":"2024-09-21T10:44:34.256143+0000","last_became_active":"2024-09-21T10:44:14.402634+0000","last_became_peered":"2024-09-21T10:44:14.402634+0000","last_unstale":"2024-09-21T10:44:34.256143+0000","last_undegraded":"2024-09-21T10:44:34.256143+0000","last_fullsized":"2024-09-21T10:44:34.256143+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:40:16.323155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2030,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.649121+0000","last_change":"2024-09-21T10:09:06.560589+0000","last_active":"2024-09-21T10:44:45.649121+0000","last_peered":"2024-09-21T10:44:45.649121+0000","last_clean":"2024-09-21T10:44:45.649121+0000","last_became_active":"2024-09-21T10:09:06.557552+0000","last_became_peered":"2024-09-21T10:09:06.557552+0000","last_unstale":"2024-09-21T10:44:45.649121+0000","last_undegraded":"2024-09-21T10:44:45.649121+0000","last_fullsized":"2024-09-21T10:44:45.649121+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:09:24.866759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":1410,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:34.314965+0000","last_change":"2024-09-21T10:44:14.415910+0000","last_active":"2024-09-21T10:44:34.314965+0000","last_peered":"2024-09-21T10:44:34.314965+0000","last_clean":"2024-09-21T10:44:34.314965+0000","last_became_active":"2024-09-21T10:44:14.402926+0000","last_became_peered":"2024-09-21T10:44:14.402926+0000","last_unstale":"2024-09-21T10:44:34.314965+0000","last_undegraded":"2024-09-21T10:44:34.314965+0000","last_fullsized":"2024-09-21T10:44:34.314965+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:55:07.144611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":770,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:33.811302+0000","last_change":"2024-09-21T10:44:17.413348+0000","last_active":"2024-09-21T10:44:33.811302+0000","last_peered":"2024-09-21T10:44:33.811302+0000","last_clean":"2024-09-21T10:44:33.811302+0000","last_became_active":"2024-09-21T10:44:17.412916+0000","last_became_peered":"2024-09-21T10:44:17.412916+0000","last_unstale":"2024-09-21T10:44:33.811302+0000","last_undegraded":"2024-09-21T10:44:33.811302+0000","last_fullsized":"2024-09-21T10:44:33.811302+0000","mapping_epoch":1372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1373,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:18.592283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2034,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.568490+0000","last_change":"2024-09-21T10:09:06.538940+0000","last_active":"2024-09-21T10:44:45.568490+0000","last_peered":"2024-09-21T10:44:45.568490+0000","last_clean":"2024-09-21T10:44:45.568490+0000","last_became_active":"2024-09-21T10:09:06.538668+0000","last_became_peered":"2024-09-21T10:09:06.538668+0000","last_unstale":"2024-09-21T10:44:45.568490+0000","last_undegraded":"2024-09-21T10:44:45.568490+0000","last_fullsized":"2024-09-21T10:44:45.568490+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:42:44.299488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2049,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.648962+0000","last_change":"2024-09-21T10:44:14.398581+0000","last_active":"2024-09-21T10:44:45.648962+0000","last_peered":"2024-09-21T10:44:45.648962+0000","last_clean":"2024-09-21T10:44:45.648962+0000","last_became_active":"2024-09-21T10:44:14.398258+0000","last_became_peered":"2024-09-21T10:44:14.398258+0000","last_unstale":"2024-09-21T10:44:45.648962+0000","last_undegraded":"2024-09-21T10:44:45.648962+0000","last_fullsized":"2024-09-21T10:44:45.648962+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:23.960956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1349,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:34.315263+0000","last_change":"2024-09-21T10:07:55.998799+0000","last_active":"2024-09-21T10:44:34.315263+0000","last_peered":"2024-09-21T10:44:34.315263+0000","last_clean":"2024-09-21T10:44:34.315263+0000","last_became_active":"2024-09-21T10:07:55.998510+0000","last_became_peered":"2024-09-21T10:07:55.998510+0000","last_unstale":"2024-09-21T10:44:34.315263+0000","last_undegraded":"2024-09-21T10:44:34.315263+0000","last_fullsized":"2024-09-21T10:44:34.315263+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:10.474008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1378,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.649763+0000","last_change":"2024-09-21T10:44:14.399842+0000","last_active":"2024-09-21T10:44:45.649763+0000","last_peered":"2024-09-21T10:44:45.649763+0000","last_clean":"2024-09-21T10:44:45.649763+0000","last_became_active":"2024-09-21T10:44:14.399426+0000","last_became_peered":"2024-09-21T10:44:14.399426+0000","last_unstale":"2024-09-21T10:44:45.649763+0000","last_undegraded":"2024-09-21T10:44:45.649763+0000","last_fullsized":"2024-09-21T10:44:45.649763+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:26:45.362101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1373,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:34.315343+0000","last_change":"2024-09-21T10:40:13.432387+0000","last_active":"2024-09-21T10:44:34.315343+0000","last_peered":"2024-09-21T10:44:34.315343+0000","last_clean":"2024-09-21T10:44:34.315343+0000","last_became_active":"2024-09-21T10:40:13.432053+0000","last_became_peered":"2024-09-21T10:40:13.432053+0000","last_unstale":"2024-09-21T10:44:34.315343+0000","last_undegraded":"2024-09-21T10:44:34.315343+0000","last_fullsized":"2024-09-21T10:44:34.315343+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:37.486101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1351,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:34.255885+0000","last_change":"2024-09-21T10:07:56.020791+0000","last_active":"2024-09-21T10:44:34.255885+0000","last_peered":"2024-09-21T10:44:34.255885+0000","last_clean":"2024-09-21T10:44:34.255885+0000","last_became_active":"2024-09-21T10:07:56.020205+0000","last_became_peered":"2024-09-21T10:07:56.020205+0000","last_unstale":"2024-09-21T10:44:34.255885+0000","last_undegraded":"2024-09-21T10:44:34.255885+0000","last_fullsized":"2024-09-21T10:44:34.255885+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:09.813166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1349,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:34.314817+0000","last_change":"2024-09-21T10:09:06.533271+0000","last_active":"2024-09-21T10:44:34.314817+0000","last_peered":"2024-09-21T10:44:34.314817+0000","last_clean":"2024-09-21T10:44:34.314817+0000","last_became_active":"2024-09-21T10:09:06.532041+0000","last_became_peered":"2024-09-21T10:09:06.532041+0000","last_unstale":"2024-09-21T10:44:34.314817+0000","last_undegraded":"2024-09-21T10:44:34.314817+0000","last_fullsized":"2024-09-21T10:44:34.314817+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:04.236375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":2050,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.649307+0000","last_change":"2024-09-21T10:09:06.543422+0000","last_active":"2024-09-21T10:44:45.649307+0000","last_peered":"2024-09-21T10:44:45.649307+0000","last_clean":"2024-09-21T10:44:45.649307+0000","last_became_active":"2024-09-21T10:09:06.539647+0000","last_became_peered":"2024-09-21T10:09:06.539647+0000","last_unstale":"2024-09-21T10:44:45.649307+0000","last_undegraded":"2024-09-21T10:44:45.649307+0000","last_fullsized":"2024-09-21T10:44:45.649307+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:04.990085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2049,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.536869+0000","last_change":"2024-09-21T10:07:56.018560+0000","last_active":"2024-09-21T10:44:45.536869+0000","last_peered":"2024-09-21T10:44:45.536869+0000","last_clean":"2024-09-21T10:44:45.536869+0000","last_became_active":"2024-09-21T10:07:56.018261+0000","last_became_peered":"2024-09-21T10:07:56.018261+0000","last_unstale":"2024-09-21T10:44:45.536869+0000","last_undegraded":"2024-09-21T10:44:45.536869+0000","last_fullsized":"2024-09-21T10:44:45.536869+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:04:54.395456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1350,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:34.315102+0000","last_change":"2024-09-21T10:09:06.531935+0000","last_active":"2024-09-21T10:44:34.315102+0000","last_peered":"2024-09-21T10:44:34.315102+0000","last_clean":"2024-09-21T10:44:34.315102+0000","last_became_active":"2024-09-21T10:09:06.531610+0000","last_became_peered":"2024-09-21T10:09:06.531610+0000","last_unstale":"2024-09-21T10:44:34.315102+0000","last_undegraded":"2024-09-21T10:44:34.315102+0000","last_fullsized":"2024-09-21T10:44:34.315102+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:33.874405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1386,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:34.255817+0000","last_change":"2024-09-21T10:44:14.401913+0000","last_active":"2024-09-21T10:44:34.255817+0000","last_peered":"2024-09-21T10:44:34.255817+0000","last_clean":"2024-09-21T10:44:34.255817+0000","last_became_active":"2024-09-21T10:44:14.401496+0000","last_became_peered":"2024-09-21T10:44:14.401496+0000","last_unstale":"2024-09-21T10:44:34.255817+0000","last_undegraded":"2024-09-21T10:44:34.255817+0000","last_fullsized":"2024-09-21T10:44:34.255817+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:07:38.531723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":24,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:33.811627+0000","last_change":"2024-09-21T10:44:14.410444+0000","last_active":"2024-09-21T10:44:33.811627+0000","last_peered":"2024-09-21T10:44:33.811627+0000","last_clean":"2024-09-21T10:44:33.811627+0000","last_became_active":"2024-09-21T10:44:14.395465+0000","last_became_peered":"2024-09-21T10:44:14.395465+0000","last_unstale":"2024-09-21T10:44:33.811627+0000","last_undegraded":"2024-09-21T10:44:33.811627+0000","last_fullsized":"2024-09-21T10:44:33.811627+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:05:44.575878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1365,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.650187+0000","last_change":"2024-09-21T10:09:06.559132+0000","last_active":"2024-09-21T10:44:45.650187+0000","last_peered":"2024-09-21T10:44:45.650187+0000","last_clean":"2024-09-21T10:44:45.650187+0000","last_became_active":"2024-09-21T10:09:06.558395+0000","last_became_peered":"2024-09-21T10:09:06.558395+0000","last_unstale":"2024-09-21T10:44:45.650187+0000","last_undegraded":"2024-09-21T10:44:45.650187+0000","last_fullsized":"2024-09-21T10:44:45.650187+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:27:00.073554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":24,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:33.811832+0000","last_change":"2024-09-21T10:44:14.419937+0000","last_active":"2024-09-21T10:44:33.811832+0000","last_peered":"2024-09-21T10:44:33.811832+0000","last_clean":"2024-09-21T10:44:33.811832+0000","last_became_active":"2024-09-21T10:44:14.402761+0000","last_became_peered":"2024-09-21T10:44:14.402761+0000","last_unstale":"2024-09-21T10:44:33.811832+0000","last_undegraded":"2024-09-21T10:44:33.811832+0000","last_fullsized":"2024-09-21T10:44:33.811832+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:42.875310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1353,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.386256+0000","last_change":"2024-09-21T10:44:14.402411+0000","last_active":"2024-09-21T10:44:45.386256+0000","last_peered":"2024-09-21T10:44:45.386256+0000","last_clean":"2024-09-21T10:44:45.386256+0000","last_became_active":"2024-09-21T10:44:14.402045+0000","last_became_peered":"2024-09-21T10:44:14.402045+0000","last_unstale":"2024-09-21T10:44:45.386256+0000","last_undegraded":"2024-09-21T10:44:45.386256+0000","last_fullsized":"2024-09-21T10:44:45.386256+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:57:10.535776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":24,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:33.811868+0000","last_change":"2024-09-21T10:44:14.421612+0000","last_active":"2024-09-21T10:44:33.811868+0000","last_peered":"2024-09-21T10:44:33.811868+0000","last_clean":"2024-09-21T10:44:33.811868+0000","last_became_active":"2024-09-21T10:44:14.406095+0000","last_became_peered":"2024-09-21T10:44:14.406095+0000","last_unstale":"2024-09-21T10:44:33.811868+0000","last_undegraded":"2024-09-21T10:44:33.811868+0000","last_fullsized":"2024-09-21T10:44:33.811868+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:21:19.918156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1358,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:34.255805+0000","last_change":"2024-09-21T10:07:21.497877+0000","last_active":"2024-09-21T10:44:34.255805+0000","last_peered":"2024-09-21T10:44:34.255805+0000","last_clean":"2024-09-21T10:44:34.255805+0000","last_became_active":"2024-09-21T10:07:21.496317+0000","last_became_peered":"2024-09-21T10:07:21.496317+0000","last_unstale":"2024-09-21T10:44:34.255805+0000","last_undegraded":"2024-09-21T10:44:34.255805+0000","last_fullsized":"2024-09-21T10:44:34.255805+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:39:03.848183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2045,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.649708+0000","last_change":"2024-09-21T10:44:14.398487+0000","last_active":"2024-09-21T10:44:45.649708+0000","last_peered":"2024-09-21T10:44:45.649708+0000","last_clean":"2024-09-21T10:44:45.649708+0000","last_became_active":"2024-09-21T10:44:14.398142+0000","last_became_peered":"2024-09-21T10:44:14.398142+0000","last_unstale":"2024-09-21T10:44:45.649708+0000","last_undegraded":"2024-09-21T10:44:45.649708+0000","last_fullsized":"2024-09-21T10:44:45.649708+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:20:41.065055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:33.811337+0000","last_change":"2024-09-21T10:44:14.398878+0000","last_active":"2024-09-21T10:44:33.811337+0000","last_peered":"2024-09-21T10:44:33.811337+0000","last_clean":"2024-09-21T10:44:33.811337+0000","last_became_active":"2024-09-21T10:44:14.398547+0000","last_became_peered":"2024-09-21T10:44:14.398547+0000","last_unstale":"2024-09-21T10:44:33.811337+0000","last_undegraded":"2024-09-21T10:44:33.811337+0000","last_fullsized":"2024-09-21T10:44:33.811337+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:13.008134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2076,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.537014+0000","last_change":"2024-09-21T10:44:14.409051+0000","last_active":"2024-09-21T10:44:45.537014+0000","last_peered":"2024-09-21T10:44:45.537014+0000","last_clean":"2024-09-21T10:44:45.537014+0000","last_became_active":"2024-09-21T10:44:14.407892+0000","last_became_peered":"2024-09-21T10:44:14.407892+0000","last_unstale":"2024-09-21T10:44:45.537014+0000","last_undegraded":"2024-09-21T10:44:45.537014+0000","last_fullsized":"2024-09-21T10:44:45.537014+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:19.549648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":2060,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.536949+0000","last_change":"2024-09-21T10:44:14.408837+0000","last_active":"2024-09-21T10:44:45.536949+0000","last_peered":"2024-09-21T10:44:45.536949+0000","last_clean":"2024-09-21T10:44:45.536949+0000","last_became_active":"2024-09-21T10:44:14.408399+0000","last_became_peered":"2024-09-21T10:44:14.408399+0000","last_unstale":"2024-09-21T10:44:45.536949+0000","last_undegraded":"2024-09-21T10:44:45.536949+0000","last_fullsized":"2024-09-21T10:44:45.536949+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:41:38.729386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1365,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:34.315245+0000","last_change":"2024-09-21T10:44:14.401435+0000","last_active":"2024-09-21T10:44:34.315245+0000","last_peered":"2024-09-21T10:44:34.315245+0000","last_clean":"2024-09-21T10:44:34.315245+0000","last_became_active":"2024-09-21T10:44:14.400908+0000","last_became_peered":"2024-09-21T10:44:14.400908+0000","last_unstale":"2024-09-21T10:44:34.315245+0000","last_undegraded":"2024-09-21T10:44:34.315245+0000","last_fullsized":"2024-09-21T10:44:34.315245+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:46.748289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":26,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-09-21T10:44:33.811431+0000","last_change":"2024-09-21T10:44:14.418113+0000","last_active":"2024-09-21T10:44:33.811431+0000","last_peered":"2024-09-21T10:44:33.811431+0000","last_clean":"2024-09-21T10:44:33.811431+0000","last_became_active":"2024-09-21T10:44:14.398327+0000","last_became_peered":"2024-09-21T10:44:14.398327+0000","last_unstale":"2024-09-21T10:44:33.811431+0000","last_undegraded":"2024-09-21T10:44:33.811431+0000","last_fullsized":"2024-09-21T10:44:33.811431+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:08.810863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2046,"reported_epoch":1379,"state":"active+clean","last_fresh":"2024-09-21T10:44:45.649500+0000","last_change":"2024-09-21T10:44:14.398584+0000","last_active":"2024-09-21T10:44:45.649500+0000","last_peered":"2024-09-21T10:44:45.649500+0000","last_clean":"2024-09-21T10:44:45.649500+0000","last_became_active":"2024-09-21T10:44:14.398278+0000","last_became_peered":"2024-09-21T10:44:14.398278+0000","last_unstale":"2024-09-21T10:44:45.649500+0000","last_undegraded":"2024-09-21T10:44:45.649500+0000","last_fullsized":"2024-09-21T10:44:45.649500+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:19.681750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":54,"num_pg":16,"stat_sum":{"num_bytes":1037828096,"num_objects":15836,"num_object_clones":0,"num_object_copies":47508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15836,"num_write_kb":1013504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2800484352,"data_stored":2800484352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15836,"ondisk_log_size":15836,"up":48,"acting":48,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":10310,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234610,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":450284,"kb_used_data":302756,"kb_used_omap":1,"kb_used_meta":147518,"kb_avail":93294868,"statfs":{"total":95995035648,"available":95533944832,"internally_reserved":0,"allocated":310022144,"data_stored":305705000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":151058891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496247,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":653484,"kb_used_data":343460,"kb_used_omap":1,"kb_used_meta":309950,"kb_avail":93091668,"statfs":{"total":95995035648,"available":95325868032,"internally_reserved":0,"allocated":351703040,"data_stored":347386385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":317389258},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757884,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":584728,"kb_used_data":282896,"kb_used_omap":1,"kb_used_meta":301822,"kb_avail":93160424,"statfs":{"total":95995035648,"available":95396274176,"internally_reserved":0,"allocated":289685504,"data_stored":285383995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":309066182},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019520,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":578340,"kb_used_data":190428,"kb_used_omap":1,"kb_used_meta":387902,"kb_avail":93166812,"statfs":{"total":95995035648,"available":95402815488,"internally_reserved":0,"allocated":194998272,"data_stored":190688706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":397212102},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369264,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1286160,"kb_used_data":956936,"kb_used_omap":69,"kb_used_meta":329146,"kb_avail":92458992,"statfs":{"total":95995035648,"available":94678007808,"internally_reserved":0,"allocated":979902464,"data_stored":975608578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":337045634},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2920577761719,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807957950,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":691344,"kb_used_data":316424,"kb_used_omap":66,"kb_used_meta":374845,"kb_avail":93053808,"statfs":{"total":95995035648,"available":95287099392,"internally_reserved":0,"allocated":324018176,"data_stored":319724219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68589,"internal_metadata":383841299},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":666,"seq":2860448219588,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":432344,"kb_used_data":152016,"kb_used_omap":76,"kb_used_meta":280243,"kb_avail":93312808,"statfs":{"total":95995035648,"available":95552315392,"internally_reserved":0,"allocated":155664384,"data_stored":151363364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78716,"internal_metadata":286968964},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":172883968,"data_stored":172883968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":346357760,"data_stored":346357760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":992739328,"data_stored":992739328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":192282624,"data_stored":192282624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":338886656,"data_stored":338886656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":439877632,"data_stored":439877632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":317456384,"data_stored":317456384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:45:03.843 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_1 pgp_num 16 2024-09-21T10:45:03.851 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:03.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:03.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:03.953 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:03.986 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:03.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:03.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:03.989 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 16 16 40655 40639 158.732 106.812 0.0167266 0.00629477 2024-09-21T10:45:04.087 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:04.120 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:04.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:04.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:04.222 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:04.260 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:04.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:04.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:04.323 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:04.362 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:04.383 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:45:04.383 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:45:04.390 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:04.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:04.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:04.403 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:04.492 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:04.525 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:04.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:04.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:04.627 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:04.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:04 smithi070 ceph-mon[112139]: pgmap v2365: 89 pgs: 89 active+clean; 2.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-09-21T10:45:04.634 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:04 smithi070 ceph-mon[107385]: pgmap v2365: 89 pgs: 89 active+clean; 2.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-09-21T10:45:04.670 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:04.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:04.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:04.772 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:04.805 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:04.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:04.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:04.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:04 smithi160 ceph-mon[101879]: pgmap v2365: 89 pgs: 89 active+clean; 2.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-09-21T10:45:04.907 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:04.950 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:04.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:04.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:04.989 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 17 15 42616 42601 156.608 122.625 0.00637779 0.00638109 2024-09-21T10:45:05.053 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:05.088 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:05.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:05.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:05.189 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:05.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:05.227 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:05.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:05.328 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:05.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:05.364 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:05.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:05.384 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:45:05.465 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:05.513 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:05.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:05.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:05.615 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:05.653 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:05.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:05.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:05.755 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:05.790 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:05.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:05.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:05.891 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:05.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:05.927 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:05.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:05.989 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 18 16 44593 44577 154.768 123.5 0.00627767 0.00645675 2024-09-21T10:45:06.029 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:06.062 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:06.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:06.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:06.164 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:06.196 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:06.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:06.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:06.298 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:06.331 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:06.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:06.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:06.433 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:06.466 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:06.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:06.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:06.567 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:06.601 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:06.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:06.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:06 smithi070 ceph-mon[107385]: pgmap v2366: 89 pgs: 89 active+clean; 2.3 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 140 MiB/s wr, 2.25k op/s 2024-09-21T10:45:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:45:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:45:06.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:06 smithi070 ceph-mon[112139]: pgmap v2366: 89 pgs: 89 active+clean; 2.3 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 140 MiB/s wr, 2.25k op/s 2024-09-21T10:45:06.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:45:06.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:45:06.702 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:06.740 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:06.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:06.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:06 smithi160 ceph-mon[101879]: pgmap v2366: 89 pgs: 89 active+clean; 2.3 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 140 MiB/s wr, 2.25k op/s 2024-09-21T10:45:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:45:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:45:06.842 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:06.875 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:06.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:06.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:06.977 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:06.991 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 19 16 46558 46542 153.074 122.812 0.00678049 0.0065276 2024-09-21T10:45:07.014 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:07.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:07.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:07.116 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:07.149 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:07.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:07.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:07.251 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:07.290 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:07.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:07.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:07.392 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:07.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:07.427 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:07.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:07.528 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:07.561 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:07.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:07.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:07.664 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:07.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:07.708 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:07.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:07.810 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:07.849 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:07.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:07.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:07.951 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:07.984 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:07.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:07.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:07.991 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:2024-09-21T10:45:07.990179+0000 min lat: 0.00136946 max lat: 0.176296 avg lat: 0.00661081 2024-09-21T10:45:07.991 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:45:07.991 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 20 16 48392 48376 151.151 114.625 0.00742082 0.00661081 2024-09-21T10:45:08.086 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:08.117 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:08.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:08.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:08.219 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:08.255 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:08.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:08.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:08.330 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:45:08.330 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:45:08.330 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:45:08.330 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:45:08.357 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:08.361 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:45:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:45:08.391 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:08.391 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:45:08.392 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:45:08.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:08.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:08.420 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:45:08.420 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:45:08.453 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:45:08.453 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:45:08.488 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:45:08.489 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:45:08.493 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:08.524 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:45:08.525 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:45:08.526 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:08 smithi070 ceph-mon[107385]: pgmap v2367: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 118 MiB/s wr, 1.90k op/s 2024-09-21T10:45:08.526 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:08 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2110700952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:08.526 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:08 smithi070 ceph-mon[112139]: pgmap v2367: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 118 MiB/s wr, 1.90k op/s 2024-09-21T10:45:08.526 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:08 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2110700952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:08.533 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:08.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:08.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:08.635 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:08.668 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:08.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:08.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:08.770 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:08.806 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:08.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:08.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:08 smithi160 ceph-mon[101879]: pgmap v2367: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 118 MiB/s wr, 1.90k op/s 2024-09-21T10:45:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:08 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2110700952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:08.908 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:08.942 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:08.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:08.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:08.991 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 21 16 50176 50160 149.262 111.5 0.00523448 0.00669496 2024-09-21T10:45:09.043 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:09.081 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:09.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:09.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:09.183 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:09.222 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:09.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:09.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:09.324 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:09.339 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:09.369 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:09.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:09.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:09.385 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T10:45:09.471 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:09.507 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:09.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:09.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:09.608 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:09.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:09 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3620885697' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:45:09.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:09 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3620885697' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:45:09.646 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:09.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:09.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:09.748 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:09.782 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:09.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:09.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:09 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3620885697' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:45:09.883 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:09.918 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:09.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:09.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:09.986 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:09.991 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 22 16 51987 51971 147.621 113.188 0.0166835 0.00676865 2024-09-21T10:45:10.019 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:10.052 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:10.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:10.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:10.153 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:10.185 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:10.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:10.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:10.287 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:10.317 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:10.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:10.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:10.419 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:10.454 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:10.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:10.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:10.556 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:10.590 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:10.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:10.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:10.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:10 smithi070 ceph-mon[107385]: pgmap v2368: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-21T10:45:10.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:10 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:45:10.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:10 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3620885697' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:45:10.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:10 smithi070 ceph-mon[107385]: osdmap e1381: 8 total, 8 up, 7 in 2024-09-21T10:45:10.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:10 smithi070 ceph-mon[112139]: pgmap v2368: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-21T10:45:10.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:10 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:45:10.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:10 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3620885697' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:45:10.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:10 smithi070 ceph-mon[112139]: osdmap e1381: 8 total, 8 up, 7 in 2024-09-21T10:45:10.691 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:10.726 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:10.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:10.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:10.828 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:10.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:10 smithi160 ceph-mon[101879]: pgmap v2368: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-21T10:45:10.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:10 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:45:10.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:10 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3620885697' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:45:10.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:10 smithi160 ceph-mon[101879]: osdmap e1381: 8 total, 8 up, 7 in 2024-09-21T10:45:10.866 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:10.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:10.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:10.967 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:10.991 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 23 16 53769 53753 146.045 111.375 0.0143701 0.0068416 2024-09-21T10:45:11.005 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:11.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:11.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:11.107 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:11.139 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:11.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:11.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:11.241 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:11.275 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:11.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:11.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:11.371 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:45:11.377 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:11.413 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:11.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:11.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:11.515 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:11.552 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:11.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:11.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:11.654 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:11.689 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:11.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:11.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:11.791 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:11.830 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:11.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:11.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:11.932 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:11.954 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:11.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:11.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:11.991 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 24 16 55485 55469 144.428 107.25 0.0135148 0.00691847 2024-09-21T10:45:12.056 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:12.085 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:12.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:12.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:12.186 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:12.224 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:12.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:12.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:12.325 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:12.361 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:12.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:12.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:12.445 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:45:12.463 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:12.498 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:12.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:12.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:12.599 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:12.638 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:12 smithi070 ceph-mon[107385]: pgmap v2370: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 118 MiB/s wr, 1.88k op/s 2024-09-21T10:45:12.639 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:12 smithi070 ceph-mon[112139]: pgmap v2370: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 118 MiB/s wr, 1.88k op/s 2024-09-21T10:45:12.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:12.641 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:12.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:12.741 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:12.775 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:12.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:12.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:12 smithi160 ceph-mon[101879]: pgmap v2370: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 118 MiB/s wr, 1.88k op/s 2024-09-21T10:45:12.877 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:12.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:12.913 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:12.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:12.992 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 25 16 57157 57141 142.831 104.5 0.009673 0.00699662 2024-09-21T10:45:13.014 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:13.049 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:13.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:13.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:13.151 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:13.190 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:13.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:13.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:13.195 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:45:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:45:13] "GET /metrics HTTP/1.1" 200 36597 "" "Prometheus/2.51.0" 2024-09-21T10:45:13.294 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:13.330 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:13.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:13.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:13.431 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:13.467 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:13.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:13.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:13.517 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:13.559 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:45:13.559 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T10:45:13.559 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T10:45:13.559 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T10:45:13.559 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.559 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:69151 53.6 53:60bcece4:::benchmark_data_smithi070_228967_object69150:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1323)", 2024-09-21T10:45:13.559 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:40:59.631469+0000", 2024-09-21T10:45:13.559 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 253.908575523, 2024-09-21T10:45:13.559 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0285536900000001, 2024-09-21T10:45:13.560 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:45:13.560 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:45:13.560 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:13.560 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:45:13.560 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:45:13.560 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:45:13.560 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 69151 2024-09-21T10:45:13.560 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.560 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:45:13.560 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.560 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:45:13.560 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.631469+0000", 2024-09-21T10:45:13.560 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.560 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.561 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.561 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:45:13.561 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.631469+0000", 2024-09-21T10:45:13.561 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.561 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.561 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.561 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:45:13.561 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.631471+0000", 2024-09-21T10:45:13.561 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.421e-06 2024-09-21T10:45:13.561 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.561 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.561 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:45:13.562 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.631496+0000", 2024-09-21T10:45:13.562 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5309000000000001e-05 2024-09-21T10:45:13.562 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.562 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.562 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:45:13.562 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.631514+0000", 2024-09-21T10:45:13.562 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7476e-05 2024-09-21T10:45:13.562 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.562 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.562 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:45:13.562 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.631522+0000", 2024-09-21T10:45:13.562 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2840000000000006e-06 2024-09-21T10:45:13.562 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.562 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.563 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.563 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.728232+0000", 2024-09-21T10:45:13.563 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.096710214000000003 2024-09-21T10:45:13.563 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.563 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.563 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:45:13.563 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.728243+0000", 2024-09-21T10:45:13.563 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0655e-05 2024-09-21T10:45:13.563 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.563 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.563 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.563 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.648307+0000", 2024-09-21T10:45:13.563 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.92006405700000005 2024-09-21T10:45:13.564 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.564 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.564 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:45:13.564 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.648453+0000", 2024-09-21T10:45:13.564 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014592899999999999 2024-09-21T10:45:13.564 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.564 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.564 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:45:13.564 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.648595+0000", 2024-09-21T10:45:13.564 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014246800000000001 2024-09-21T10:45:13.564 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.564 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.564 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:45:13.564 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.654982+0000", 2024-09-21T10:45:13.564 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0063872460000000001 2024-09-21T10:45:13.564 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.565 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.565 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.565 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.657300+0000", 2024-09-21T10:45:13.565 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0023175679999999999 2024-09-21T10:45:13.565 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.565 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.565 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.565 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.659594+0000", 2024-09-21T10:45:13.565 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002293911 2024-09-21T10:45:13.565 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.565 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.565 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:45:13.565 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.659814+0000", 2024-09-21T10:45:13.565 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00022020399999999999 2024-09-21T10:45:13.566 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.566 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.566 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:45:13.566 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.660023+0000", 2024-09-21T10:45:13.566 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00020894800000000001 2024-09-21T10:45:13.566 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.566 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:45:13.566 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.566 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.566 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.566 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:80837 53.14 53.461937d4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1332)", 2024-09-21T10:45:13.566 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:12.901141+0000", 2024-09-21T10:45:13.566 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 240.63890373699999, 2024-09-21T10:45:13.566 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.229358014, 2024-09-21T10:45:13.567 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:45:13.567 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:45:13.567 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:45:13.567 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:45:13.567 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:45:13.567 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:45:13.567 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 80837 2024-09-21T10:45:13.567 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.567 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:45:13.567 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.567 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:45:13.567 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.901141+0000", 2024-09-21T10:45:13.567 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.567 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.567 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.568 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:45:13.568 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.901141+0000", 2024-09-21T10:45:13.568 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.568 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.568 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.568 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:45:13.568 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.901142+0000", 2024-09-21T10:45:13.568 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9399999999999993e-07 2024-09-21T10:45:13.568 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.568 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.568 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:45:13.568 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.901164+0000", 2024-09-21T10:45:13.568 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1988999999999999e-05 2024-09-21T10:45:13.568 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.569 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.569 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:45:13.569 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.901173+0000", 2024-09-21T10:45:13.569 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.4619999999999996e-06 2024-09-21T10:45:13.569 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.569 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.569 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:45:13.569 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.901180+0000", 2024-09-21T10:45:13.569 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0840000000000003e-06 2024-09-21T10:45:13.569 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.569 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.569 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.569 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.901241+0000", 2024-09-21T10:45:13.569 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1023000000000002e-05 2024-09-21T10:45:13.569 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.569 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.570 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:45:13.570 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.901248+0000", 2024-09-21T10:45:13.570 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1450000000000002e-06 2024-09-21T10:45:13.570 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.570 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.570 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.570 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.130491+0000", 2024-09-21T10:45:13.570 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.22924297699999999 2024-09-21T10:45:13.570 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.570 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.570 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:45:13.570 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.130499+0000", 2024-09-21T10:45:13.570 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3399999999999998e-06 2024-09-21T10:45:13.570 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.570 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:45:13.571 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.571 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.571 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.571 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:80865 53.14 53.12308bb4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1332)", 2024-09-21T10:45:13.571 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:12.921694+0000", 2024-09-21T10:45:13.571 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 240.61835083599999, 2024-09-21T10:45:13.571 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.208837631, 2024-09-21T10:45:13.571 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:45:13.571 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:45:13.571 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:45:13.571 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:45:13.571 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:45:13.571 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:45:13.572 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 80865 2024-09-21T10:45:13.572 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.572 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:45:13.572 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.572 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:45:13.572 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.921694+0000", 2024-09-21T10:45:13.572 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.572 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.572 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.572 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:45:13.572 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.921694+0000", 2024-09-21T10:45:13.572 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.572 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.572 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.572 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:45:13.573 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.921696+0000", 2024-09-21T10:45:13.573 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.761e-06 2024-09-21T10:45:13.573 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.573 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.573 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:45:13.573 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.921727+0000", 2024-09-21T10:45:13.573 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1430000000000002e-05 2024-09-21T10:45:13.573 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.573 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.573 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:45:13.573 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.921740+0000", 2024-09-21T10:45:13.573 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3067e-05 2024-09-21T10:45:13.573 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.573 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.573 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:45:13.574 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.921748+0000", 2024-09-21T10:45:13.574 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.498e-06 2024-09-21T10:45:13.574 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.574 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.574 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.574 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.921814+0000", 2024-09-21T10:45:13.574 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6113999999999998e-05 2024-09-21T10:45:13.574 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.574 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.574 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:45:13.574 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.921824+0000", 2024-09-21T10:45:13.574 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0428000000000001e-05 2024-09-21T10:45:13.574 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.575 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.575 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.575 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.130525+0000", 2024-09-21T10:45:13.575 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.208700311 2024-09-21T10:45:13.575 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.575 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.575 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:45:13.575 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.130532+0000", 2024-09-21T10:45:13.575 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0219999999999999e-06 2024-09-21T10:45:13.575 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.575 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:45:13.575 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.575 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.575 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.576 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:80885 53.14 53.40d44454 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1332)", 2024-09-21T10:45:13.576 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:12.933875+0000", 2024-09-21T10:45:13.576 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 240.606169411, 2024-09-21T10:45:13.576 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19668569399999999, 2024-09-21T10:45:13.576 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:45:13.576 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:45:13.576 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:45:13.576 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:45:13.576 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:45:13.576 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:45:13.576 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 80885 2024-09-21T10:45:13.576 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.576 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:45:13.576 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.577 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:45:13.577 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.933875+0000", 2024-09-21T10:45:13.577 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.577 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.577 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.577 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:45:13.577 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.933875+0000", 2024-09-21T10:45:13.577 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.577 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.577 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.577 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:45:13.577 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.933877+0000", 2024-09-21T10:45:13.577 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.81e-06 2024-09-21T10:45:13.577 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.578 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.578 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:45:13.578 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.933899+0000", 2024-09-21T10:45:13.578 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1379e-05 2024-09-21T10:45:13.578 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.578 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.578 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:45:13.578 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.933914+0000", 2024-09-21T10:45:13.578 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5648999999999999e-05 2024-09-21T10:45:13.578 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.578 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.578 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:45:13.578 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.933925+0000", 2024-09-21T10:45:13.579 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0913999999999999e-05 2024-09-21T10:45:13.579 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.579 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.579 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.579 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.937675+0000", 2024-09-21T10:45:13.579 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0037502910000000002 2024-09-21T10:45:13.579 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.579 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.579 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:45:13.579 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.937683+0000", 2024-09-21T10:45:13.579 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6669999999999996e-06 2024-09-21T10:45:13.579 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.579 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.579 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.580 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.130554+0000", 2024-09-21T10:45:13.580 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19287114499999999 2024-09-21T10:45:13.580 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.580 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.580 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:45:13.580 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.130561+0000", 2024-09-21T10:45:13.580 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8390000000000001e-06 2024-09-21T10:45:13.580 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.580 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:45:13.580 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.580 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.580 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.581 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:235308 53.14 53:2bd81261:::benchmark_data_smithi070_228967_object106388:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1351)", 2024-09-21T10:45:13.581 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:43:38.162496+0000", 2024-09-21T10:45:13.581 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 95.377548563000005, 2024-09-21T10:45:13.581 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.018336854, 2024-09-21T10:45:13.581 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:45:13.581 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:45:13.581 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:45:13.581 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:45:13.581 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:45:13.581 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:45:13.581 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235308 2024-09-21T10:45:13.581 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.581 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:45:13.582 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.582 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:45:13.582 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.162496+0000", 2024-09-21T10:45:13.582 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.582 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.582 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.582 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:45:13.582 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.162496+0000", 2024-09-21T10:45:13.582 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.582 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.582 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.582 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:45:13.582 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.162498+0000", 2024-09-21T10:45:13.582 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8220000000000001e-06 2024-09-21T10:45:13.582 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.583 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.583 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:45:13.583 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.162503+0000", 2024-09-21T10:45:13.583 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8459999999999999e-06 2024-09-21T10:45:13.583 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.583 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.583 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:45:13.583 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.162517+0000", 2024-09-21T10:45:13.583 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3642000000000001e-05 2024-09-21T10:45:13.583 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.583 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.583 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:45:13.583 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.162524+0000", 2024-09-21T10:45:13.583 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4320000000000002e-06 2024-09-21T10:45:13.583 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.584 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.584 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.584 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.171402+0000", 2024-09-21T10:45:13.584 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0088781989999999998 2024-09-21T10:45:13.584 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.584 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.584 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:45:13.584 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.171412+0000", 2024-09-21T10:45:13.584 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7880000000000001e-06 2024-09-21T10:45:13.584 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.584 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.584 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.584 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.180755+0000", 2024-09-21T10:45:13.584 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.009342961 2024-09-21T10:45:13.584 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.585 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.585 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:45:13.585 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.180833+0000", 2024-09-21T10:45:13.585 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8164000000000006e-05 2024-09-21T10:45:13.585 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.585 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:45:13.585 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.585 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.585 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.585 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:235336 53.14 53:28162202:::benchmark_data_smithi070_228967_object106416:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1351)", 2024-09-21T10:45:13.585 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:43:38.182875+0000", 2024-09-21T10:45:13.585 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 95.357169698000007, 2024-09-21T10:45:13.585 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0046595659999999, 2024-09-21T10:45:13.585 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:45:13.585 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:45:13.585 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:13.586 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:45:13.586 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:45:13.586 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:45:13.586 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235336 2024-09-21T10:45:13.586 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.586 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:45:13.586 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.586 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:45:13.586 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.182875+0000", 2024-09-21T10:45:13.586 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.586 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.586 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.586 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:45:13.586 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.182875+0000", 2024-09-21T10:45:13.586 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.587 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.587 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.587 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:45:13.587 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.182877+0000", 2024-09-21T10:45:13.587 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4810000000000001e-06 2024-09-21T10:45:13.587 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.587 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.587 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:45:13.587 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.182882+0000", 2024-09-21T10:45:13.587 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4299999999999997e-06 2024-09-21T10:45:13.587 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.587 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.587 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:45:13.587 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.182894+0000", 2024-09-21T10:45:13.587 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.187e-05 2024-09-21T10:45:13.588 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.588 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.588 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:45:13.588 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.182902+0000", 2024-09-21T10:45:13.588 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7700000000000001e-06 2024-09-21T10:45:13.588 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.588 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.588 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.588 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.183037+0000", 2024-09-21T10:45:13.588 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000135406 2024-09-21T10:45:13.588 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.588 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.588 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:45:13.588 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.183047+0000", 2024-09-21T10:45:13.588 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7850000000000003e-06 2024-09-21T10:45:13.589 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.589 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.589 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.589 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.181319+0000", 2024-09-21T10:45:13.589 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99827166199999995 2024-09-21T10:45:13.589 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.589 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.589 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:45:13.589 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.181428+0000", 2024-09-21T10:45:13.589 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000109176 2024-09-21T10:45:13.589 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.589 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.589 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:45:13.589 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.181519+0000", 2024-09-21T10:45:13.589 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1489000000000004e-05 2024-09-21T10:45:13.590 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.590 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.590 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.590 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.187366+0000", 2024-09-21T10:45:13.590 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0058467950000000001 2024-09-21T10:45:13.590 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.590 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.590 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:45:13.590 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.187396+0000", 2024-09-21T10:45:13.590 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9748e-05 2024-09-21T10:45:13.590 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.590 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.590 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.591 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.187441+0000", 2024-09-21T10:45:13.591 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5729000000000002e-05 2024-09-21T10:45:13.591 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.591 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.591 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:45:13.591 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.187488+0000", 2024-09-21T10:45:13.591 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6332999999999998e-05 2024-09-21T10:45:13.591 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.591 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.591 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:45:13.591 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.187535+0000", 2024-09-21T10:45:13.591 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6891999999999998e-05 2024-09-21T10:45:13.591 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.591 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:45:13.592 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.592 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.592 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.592 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:235339 53.14 53:28064d20:::benchmark_data_smithi070_228967_object106419:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1351)", 2024-09-21T10:45:13.592 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:43:38.187411+0000", 2024-09-21T10:45:13.592 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 95.352633573999995, 2024-09-21T10:45:13.592 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.004490621, 2024-09-21T10:45:13.592 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:45:13.592 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:45:13.592 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:13.592 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:45:13.592 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:45:13.592 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:45:13.592 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235339 2024-09-21T10:45:13.592 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.592 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:45:13.593 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.593 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:45:13.593 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.187411+0000", 2024-09-21T10:45:13.593 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.593 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.593 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.593 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:45:13.593 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.187411+0000", 2024-09-21T10:45:13.593 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.593 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.593 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.593 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:45:13.593 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.187413+0000", 2024-09-21T10:45:13.593 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.39e-06 2024-09-21T10:45:13.593 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.593 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.594 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:45:13.594 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.187417+0000", 2024-09-21T10:45:13.594 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7879999999999997e-06 2024-09-21T10:45:13.594 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.594 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.594 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:45:13.594 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.187432+0000", 2024-09-21T10:45:13.594 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4646e-05 2024-09-21T10:45:13.594 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.594 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.594 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:45:13.594 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.187441+0000", 2024-09-21T10:45:13.594 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9700000000000005e-06 2024-09-21T10:45:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.188682+0000", 2024-09-21T10:45:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0012409420000000001 2024-09-21T10:45:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:45:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.188692+0000", 2024-09-21T10:45:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9359999999999994e-06 2024-09-21T10:45:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.183727+0000", 2024-09-21T10:45:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99503534199999999 2024-09-21T10:45:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:45:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.183808+0000", 2024-09-21T10:45:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1067000000000002e-05 2024-09-21T10:45:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:45:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.183877+0000", 2024-09-21T10:45:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9130999999999999e-05 2024-09-21T10:45:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:45:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.187400+0000", 2024-09-21T10:45:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035228400000000002 2024-09-21T10:45:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.187599+0000", 2024-09-21T10:45:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00019847899999999999 2024-09-21T10:45:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.191803+0000", 2024-09-21T10:45:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0042046269999999998 2024-09-21T10:45:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:45:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.191859+0000", 2024-09-21T10:45:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5914e-05 2024-09-21T10:45:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:45:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.191902+0000", 2024-09-21T10:45:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2549000000000002e-05 2024-09-21T10:45:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:45:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:235353 53.14 53:293f8e5d:::benchmark_data_smithi070_228967_object106433:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1351)", 2024-09-21T10:45:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:43:38.196430+0000", 2024-09-21T10:45:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 95.343614410000001, 2024-09-21T10:45:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99561364299999999, 2024-09-21T10:45:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:45:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:45:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:45:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:45:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:45:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235353 2024-09-21T10:45:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:45:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:45:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.196430+0000", 2024-09-21T10:45:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:45:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.196430+0000", 2024-09-21T10:45:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:45:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.196431+0000", 2024-09-21T10:45:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5799999999999999e-07 2024-09-21T10:45:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:45:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.196433+0000", 2024-09-21T10:45:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3240000000000001e-06 2024-09-21T10:45:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:45:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.196440+0000", 2024-09-21T10:45:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9840000000000004e-06 2024-09-21T10:45:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:45:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.196446+0000", 2024-09-21T10:45:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3920000000000002e-06 2024-09-21T10:45:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.196519+0000", 2024-09-21T10:45:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3170999999999995e-05 2024-09-21T10:45:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:45:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.196530+0000", 2024-09-21T10:45:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0829000000000001e-05 2024-09-21T10:45:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.185458+0000", 2024-09-21T10:45:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.988928586 2024-09-21T10:45:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:45:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.185558+0000", 2024-09-21T10:45:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010016100000000001 2024-09-21T10:45:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:45:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.185658+0000", 2024-09-21T10:45:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9512999999999994e-05 2024-09-21T10:45:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.187668+0000", 2024-09-21T10:45:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002010501 2024-09-21T10:45:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:45:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.191914+0000", 2024-09-21T10:45:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0042453389999999999 2024-09-21T10:45:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.191953+0000", 2024-09-21T10:45:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9511999999999997e-05 2024-09-21T10:45:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:45:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.192001+0000", 2024-09-21T10:45:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7237999999999997e-05 2024-09-21T10:45:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.605 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:45:13.605 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.192044+0000", 2024-09-21T10:45:13.605 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3535000000000002e-05 2024-09-21T10:45:13.605 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.605 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:45:13.605 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.605 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.605 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.605 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:235376 53.14 53:2f7e5a46:::benchmark_data_smithi070_228967_object106456:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1351)", 2024-09-21T10:45:13.605 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:43:38.214572+0000", 2024-09-21T10:45:13.605 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 95.325473074000001, 2024-09-21T10:45:13.605 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.97766572699999998, 2024-09-21T10:45:13.605 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:45:13.606 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:45:13.606 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:13.606 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:45:13.606 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:45:13.606 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:45:13.606 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235376 2024-09-21T10:45:13.606 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.606 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:45:13.606 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.606 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:45:13.606 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.214572+0000", 2024-09-21T10:45:13.606 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.607 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.607 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.607 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:45:13.607 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.214572+0000", 2024-09-21T10:45:13.607 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.607 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.607 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.607 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:45:13.607 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.214573+0000", 2024-09-21T10:45:13.607 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6810000000000001e-06 2024-09-21T10:45:13.607 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.607 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.608 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:45:13.608 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.214579+0000", 2024-09-21T10:45:13.608 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8810000000000001e-06 2024-09-21T10:45:13.608 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.608 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.608 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:45:13.608 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.214597+0000", 2024-09-21T10:45:13.608 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7397e-05 2024-09-21T10:45:13.608 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.608 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.608 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:45:13.608 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.214608+0000", 2024-09-21T10:45:13.608 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.145e-05 2024-09-21T10:45:13.608 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.609 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.609 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.609 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.214886+0000", 2024-09-21T10:45:13.609 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00027823600000000001 2024-09-21T10:45:13.609 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.609 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.609 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:45:13.609 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.214897+0000", 2024-09-21T10:45:13.609 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0574000000000001e-05 2024-09-21T10:45:13.609 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.609 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.609 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.609 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.187730+0000", 2024-09-21T10:45:13.610 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.97283255099999999 2024-09-21T10:45:13.610 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.610 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.610 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:45:13.610 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.187839+0000", 2024-09-21T10:45:13.610 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000109654 2024-09-21T10:45:13.610 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.610 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.610 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:45:13.610 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.187935+0000", 2024-09-21T10:45:13.610 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5859000000000005e-05 2024-09-21T10:45:13.610 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.610 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.610 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:45:13.610 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.192055+0000", 2024-09-21T10:45:13.611 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0041197760000000003 2024-09-21T10:45:13.611 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.611 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.611 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.611 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.192113+0000", 2024-09-21T10:45:13.611 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8418000000000002e-05 2024-09-21T10:45:13.611 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.611 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.611 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.611 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.192151+0000", 2024-09-21T10:45:13.611 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7317999999999997e-05 2024-09-21T10:45:13.611 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.611 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.611 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:45:13.611 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.192202+0000", 2024-09-21T10:45:13.612 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1449000000000002e-05 2024-09-21T10:45:13.612 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.612 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.612 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:45:13.612 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.192238+0000", 2024-09-21T10:45:13.612 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5482999999999999e-05 2024-09-21T10:45:13.612 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.612 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:45:13.612 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.612 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.612 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.612 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:235447 53.14 53:2855794f:::benchmark_data_smithi070_228967_object106527:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1351)", 2024-09-21T10:45:13.612 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:43:38.269800+0000", 2024-09-21T10:45:13.612 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 95.270244704000007, 2024-09-21T10:45:13.612 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.92777093099999997, 2024-09-21T10:45:13.613 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:45:13.613 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:45:13.613 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:13.613 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:45:13.613 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:13.614 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:45:13.614 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:45:13.614 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235447 2024-09-21T10:45:13.614 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.614 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:45:13.614 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.614 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:45:13.614 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.269800+0000", 2024-09-21T10:45:13.614 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.615 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.615 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.615 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:45:13.615 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.269800+0000", 2024-09-21T10:45:13.615 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.615 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.615 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.615 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:45:13.615 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.269801+0000", 2024-09-21T10:45:13.615 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3300000000000001e-07 2024-09-21T10:45:13.615 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.615 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.615 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:45:13.615 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.269805+0000", 2024-09-21T10:45:13.615 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9179999999999996e-06 2024-09-21T10:45:13.616 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.616 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.616 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:45:13.616 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.269813+0000", 2024-09-21T10:45:13.616 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.354e-06 2024-09-21T10:45:13.616 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.616 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.616 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:45:13.616 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.269821+0000", 2024-09-21T10:45:13.616 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3900000000000004e-06 2024-09-21T10:45:13.616 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.616 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.616 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.616 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.269885+0000", 2024-09-21T10:45:13.616 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4636999999999998e-05 2024-09-21T10:45:13.616 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.617 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.617 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:45:13.617 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.269895+0000", 2024-09-21T10:45:13.617 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.6369999999999994e-06 2024-09-21T10:45:13.617 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.617 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.617 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.617 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.190074+0000", 2024-09-21T10:45:13.617 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.920179419 2024-09-21T10:45:13.617 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.617 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.617 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:45:13.617 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.190178+0000", 2024-09-21T10:45:13.617 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000103772 2024-09-21T10:45:13.617 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.617 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.618 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:45:13.618 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.190267+0000", 2024-09-21T10:45:13.618 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.8831000000000004e-05 2024-09-21T10:45:13.618 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.618 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.618 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:45:13.618 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.192066+0000", 2024-09-21T10:45:13.618 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0017991050000000001 2024-09-21T10:45:13.618 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.618 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.618 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.618 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.197402+0000", 2024-09-21T10:45:13.618 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0053359080000000003 2024-09-21T10:45:13.618 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.618 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.618 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.619 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.197472+0000", 2024-09-21T10:45:13.619 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9715000000000006e-05 2024-09-21T10:45:13.619 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.619 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.619 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:45:13.619 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.197528+0000", 2024-09-21T10:45:13.619 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6867999999999998e-05 2024-09-21T10:45:13.619 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.619 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.619 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:45:13.619 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.197571+0000", 2024-09-21T10:45:13.620 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2644e-05 2024-09-21T10:45:13.620 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.620 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:45:13.620 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.620 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.620 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.620 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:235481 53.14 53:2d6b314b:::benchmark_data_smithi070_228967_object106561:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1351)", 2024-09-21T10:45:13.620 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:43:38.298638+0000", 2024-09-21T10:45:13.620 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 95.241407295000002, 2024-09-21T10:45:13.620 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.90537712100000001, 2024-09-21T10:45:13.620 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:45:13.620 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:45:13.620 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:13.620 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:45:13.620 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:45:13.621 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:45:13.621 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235481 2024-09-21T10:45:13.621 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.621 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:45:13.621 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.621 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:45:13.621 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.298638+0000", 2024-09-21T10:45:13.621 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.621 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.621 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.621 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:45:13.621 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.298638+0000", 2024-09-21T10:45:13.621 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.622 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.622 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.622 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:45:13.622 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.298639+0000", 2024-09-21T10:45:13.622 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0589999999999999e-06 2024-09-21T10:45:13.622 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.622 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.622 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:45:13.622 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.298643+0000", 2024-09-21T10:45:13.622 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0260000000000001e-06 2024-09-21T10:45:13.622 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.622 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.622 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:45:13.623 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.298657+0000", 2024-09-21T10:45:13.623 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4032000000000001e-05 2024-09-21T10:45:13.623 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.623 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.623 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:45:13.623 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.298665+0000", 2024-09-21T10:45:13.623 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1319999999999994e-06 2024-09-21T10:45:13.623 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.623 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.623 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.623 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.298733+0000", 2024-09-21T10:45:13.623 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7698000000000001e-05 2024-09-21T10:45:13.623 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.623 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.624 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:45:13.624 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.298741+0000", 2024-09-21T10:45:13.624 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9779999999999994e-06 2024-09-21T10:45:13.624 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.624 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.624 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.624 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.192274+0000", 2024-09-21T10:45:13.624 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.89353281200000001 2024-09-21T10:45:13.624 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.624 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.624 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:45:13.624 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.192380+0000", 2024-09-21T10:45:13.624 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010577 2024-09-21T10:45:13.625 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.625 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.625 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:45:13.625 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.192480+0000", 2024-09-21T10:45:13.625 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9444000000000005e-05 2024-09-21T10:45:13.625 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.625 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.625 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:45:13.625 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.197584+0000", 2024-09-21T10:45:13.625 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0051049190000000003 2024-09-21T10:45:13.625 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.625 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.625 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.626 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.197636+0000", 2024-09-21T10:45:13.626 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1153000000000003e-05 2024-09-21T10:45:13.626 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.626 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.626 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.626 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.203927+0000", 2024-09-21T10:45:13.626 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0062917149999999998 2024-09-21T10:45:13.626 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.626 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.626 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:45:13.626 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.203977+0000", 2024-09-21T10:45:13.626 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9696e-05 2024-09-21T10:45:13.627 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.627 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.627 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:45:13.627 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.204015+0000", 2024-09-21T10:45:13.627 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7687e-05 2024-09-21T10:45:13.627 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.627 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:45:13.627 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.627 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.627 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.627 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:235516 53.14 53:28b3de9a:::benchmark_data_smithi070_228967_object106596:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1351)", 2024-09-21T10:45:13.627 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:43:38.331776+0000", 2024-09-21T10:45:13.627 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 95.208269290999993, 2024-09-21T10:45:13.627 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.87234518500000002, 2024-09-21T10:45:13.627 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:45:13.628 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:45:13.628 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:13.628 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:45:13.628 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:45:13.628 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:45:13.628 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235516 2024-09-21T10:45:13.628 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.628 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:45:13.628 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.628 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:45:13.628 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.331776+0000", 2024-09-21T10:45:13.628 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.628 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.629 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.629 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:45:13.629 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.331776+0000", 2024-09-21T10:45:13.629 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.629 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.629 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.629 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:45:13.629 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.331777+0000", 2024-09-21T10:45:13.629 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5009999999999999e-06 2024-09-21T10:45:13.629 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.629 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.629 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:45:13.629 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.331781+0000", 2024-09-21T10:45:13.629 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.574e-06 2024-09-21T10:45:13.629 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.629 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.630 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:45:13.630 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.331791+0000", 2024-09-21T10:45:13.630 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0567999999999999e-05 2024-09-21T10:45:13.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.630 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.630 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:45:13.630 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.331799+0000", 2024-09-21T10:45:13.630 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6190000000000002e-06 2024-09-21T10:45:13.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.630 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.630 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.630 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.331884+0000", 2024-09-21T10:45:13.630 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5290000000000005e-05 2024-09-21T10:45:13.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.630 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.631 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:45:13.631 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.331893+0000", 2024-09-21T10:45:13.631 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2240000000000001e-06 2024-09-21T10:45:13.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.631 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.631 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.631 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.195121+0000", 2024-09-21T10:45:13.631 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.86322791099999996 2024-09-21T10:45:13.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.631 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.631 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:45:13.631 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.195245+0000", 2024-09-21T10:45:13.631 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000123356 2024-09-21T10:45:13.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.631 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.631 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:45:13.632 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.195386+0000", 2024-09-21T10:45:13.632 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014090700000000001 2024-09-21T10:45:13.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.632 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:45:13.632 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.200081+0000", 2024-09-21T10:45:13.632 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0046955670000000003 2024-09-21T10:45:13.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.632 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.632 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.203891+0000", 2024-09-21T10:45:13.632 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0038095059999999998 2024-09-21T10:45:13.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.632 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.632 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.204051+0000", 2024-09-21T10:45:13.633 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016084699999999999 2024-09-21T10:45:13.633 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.633 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.633 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:45:13.633 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.204090+0000", 2024-09-21T10:45:13.633 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8473999999999998e-05 2024-09-21T10:45:13.633 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.633 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.633 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:45:13.633 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.204121+0000", 2024-09-21T10:45:13.633 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0840999999999998e-05 2024-09-21T10:45:13.633 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.633 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:45:13.633 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.633 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.633 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.634 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:235520 53.14 53:2b0936b9:::benchmark_data_smithi070_228967_object106600:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1351)", 2024-09-21T10:45:13.634 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:43:38.336693+0000", 2024-09-21T10:45:13.634 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 95.203351467000005, 2024-09-21T10:45:13.634 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.87020330899999998, 2024-09-21T10:45:13.634 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:45:13.634 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:45:13.634 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:13.634 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:45:13.634 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:45:13.634 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:45:13.634 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235520 2024-09-21T10:45:13.634 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.634 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:45:13.634 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.634 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:45:13.634 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.336693+0000", 2024-09-21T10:45:13.635 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.635 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.635 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:45:13.635 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.336693+0000", 2024-09-21T10:45:13.635 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.635 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.635 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:45:13.635 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.336695+0000", 2024-09-21T10:45:13.635 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3999999999999999e-06 2024-09-21T10:45:13.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.635 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.635 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:45:13.635 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.336700+0000", 2024-09-21T10:45:13.635 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8879999999999996e-06 2024-09-21T10:45:13.636 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.636 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:45:13.636 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.336712+0000", 2024-09-21T10:45:13.636 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1963999999999999e-05 2024-09-21T10:45:13.636 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.636 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:45:13.636 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.336717+0000", 2024-09-21T10:45:13.636 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8379999999999998e-06 2024-09-21T10:45:13.636 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.636 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.636 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.336765+0000", 2024-09-21T10:45:13.636 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7648000000000002e-05 2024-09-21T10:45:13.637 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.637 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.637 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:45:13.637 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.336772+0000", 2024-09-21T10:45:13.637 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.511e-06 2024-09-21T10:45:13.637 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.637 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.637 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.637 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.197696+0000", 2024-09-21T10:45:13.637 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.86092402499999998 2024-09-21T10:45:13.637 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.637 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.637 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:45:13.637 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.197808+0000", 2024-09-21T10:45:13.637 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011212199999999999 2024-09-21T10:45:13.638 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.638 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.638 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:45:13.638 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.197921+0000", 2024-09-21T10:45:13.638 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011299299999999999 2024-09-21T10:45:13.638 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.638 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.638 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:45:13.638 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.204023+0000", 2024-09-21T10:45:13.638 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0061023450000000003 2024-09-21T10:45:13.638 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.638 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.638 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.638 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.206766+0000", 2024-09-21T10:45:13.639 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0027428439999999999 2024-09-21T10:45:13.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.639 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.639 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.639 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.206799+0000", 2024-09-21T10:45:13.639 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2577000000000003e-05 2024-09-21T10:45:13.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.639 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.639 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:45:13.639 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.206856+0000", 2024-09-21T10:45:13.639 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7247000000000003e-05 2024-09-21T10:45:13.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.639 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.639 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:45:13.639 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.206897+0000", 2024-09-21T10:45:13.640 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0907e-05 2024-09-21T10:45:13.640 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.640 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:45:13.640 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.640 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.640 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.640 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:235646 53.14 53:2ee4b7ba:::benchmark_data_smithi070_228967_object106726:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1351)", 2024-09-21T10:45:13.640 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:43:38.460539+0000", 2024-09-21T10:45:13.640 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 95.079505472999998, 2024-09-21T10:45:13.640 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.74915749099999995, 2024-09-21T10:45:13.640 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:45:13.640 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:45:13.640 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:13.641 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:45:13.641 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:45:13.641 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:45:13.641 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235646 2024-09-21T10:45:13.641 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.641 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:45:13.641 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.641 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:45:13.641 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.460539+0000", 2024-09-21T10:45:13.641 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.641 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.641 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.641 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:45:13.641 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.460539+0000", 2024-09-21T10:45:13.642 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.642 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.642 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.642 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:45:13.642 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.460541+0000", 2024-09-21T10:45:13.642 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.269e-06 2024-09-21T10:45:13.642 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.642 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.642 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:45:13.642 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.460544+0000", 2024-09-21T10:45:13.642 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5920000000000001e-06 2024-09-21T10:45:13.642 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.643 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.643 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:45:13.643 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.460554+0000", 2024-09-21T10:45:13.643 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.8339999999999994e-06 2024-09-21T10:45:13.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.643 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.643 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:45:13.643 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.460561+0000", 2024-09-21T10:45:13.643 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5950000000000004e-06 2024-09-21T10:45:13.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.643 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.643 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.643 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.460617+0000", 2024-09-21T10:45:13.643 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6002999999999999e-05 2024-09-21T10:45:13.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.644 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.644 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:45:13.644 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.460625+0000", 2024-09-21T10:45:13.644 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2269999999999993e-06 2024-09-21T10:45:13.644 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.644 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.644 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.644 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.200127+0000", 2024-09-21T10:45:13.644 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.739502302 2024-09-21T10:45:13.644 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.644 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.644 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:45:13.644 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.200240+0000", 2024-09-21T10:45:13.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000113115 2024-09-21T10:45:13.645 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.645 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.645 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:45:13.645 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.200351+0000", 2024-09-21T10:45:13.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011068799999999999 2024-09-21T10:45:13.645 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.645 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.645 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.645 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.209540+0000", 2024-09-21T10:45:13.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0091893579999999999 2024-09-21T10:45:13.645 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.645 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.645 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:45:13.646 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.209551+0000", 2024-09-21T10:45:13.646 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0481e-05 2024-09-21T10:45:13.646 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.646 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.646 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.646 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.209595+0000", 2024-09-21T10:45:13.646 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.4258000000000002e-05 2024-09-21T10:45:13.646 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.646 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.646 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:45:13.646 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.209652+0000", 2024-09-21T10:45:13.646 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6740000000000002e-05 2024-09-21T10:45:13.646 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.647 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:45:13.647 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.209697+0000", 2024-09-21T10:45:13.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5028999999999999e-05 2024-09-21T10:45:13.647 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.647 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:45:13.647 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.647 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:235654 53.14 53:2a53ab46:::benchmark_data_smithi070_228967_object106734:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1351)", 2024-09-21T10:45:13.647 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:43:38.473308+0000", 2024-09-21T10:45:13.647 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 95.066736606999996, 2024-09-21T10:45:13.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.74604452899999996, 2024-09-21T10:45:13.647 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:45:13.647 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:45:13.648 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:13.648 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:45:13.648 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:45:13.648 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:45:13.648 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235654 2024-09-21T10:45:13.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.648 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:45:13.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.648 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:45:13.648 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.473308+0000", 2024-09-21T10:45:13.648 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:45:13.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.473308+0000", 2024-09-21T10:45:13.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:45:13.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.473310+0000", 2024-09-21T10:45:13.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5579999999999999e-06 2024-09-21T10:45:13.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:45:13.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.473316+0000", 2024-09-21T10:45:13.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6980000000000001e-06 2024-09-21T10:45:13.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:45:13.650 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.473330+0000", 2024-09-21T10:45:13.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3351e-05 2024-09-21T10:45:13.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.650 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:45:13.650 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.473337+0000", 2024-09-21T10:45:13.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5599999999999996e-06 2024-09-21T10:45:13.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.650 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.650 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.473410+0000", 2024-09-21T10:45:13.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2769999999999996e-05 2024-09-21T10:45:13.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.651 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:45:13.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.473417+0000", 2024-09-21T10:45:13.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7839999999999998e-06 2024-09-21T10:45:13.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.651 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.204159+0000", 2024-09-21T10:45:13.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.73074245900000001 2024-09-21T10:45:13.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.651 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:45:13.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.204255+0000", 2024-09-21T10:45:13.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.6018000000000001e-05 2024-09-21T10:45:13.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.652 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.652 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:45:13.652 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.204352+0000", 2024-09-21T10:45:13.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.6896000000000002e-05 2024-09-21T10:45:13.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.652 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.652 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:45:13.652 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.209556+0000", 2024-09-21T10:45:13.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0052032149999999998 2024-09-21T10:45:13.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.652 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.652 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.652 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.214484+0000", 2024-09-21T10:45:13.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0049286909999999998 2024-09-21T10:45:13.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.653 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.653 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.653 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.219285+0000", 2024-09-21T10:45:13.653 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0048006519999999999 2024-09-21T10:45:13.653 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.653 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.653 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:45:13.653 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.219323+0000", 2024-09-21T10:45:13.653 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8000000000000002e-05 2024-09-21T10:45:13.653 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.653 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.653 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:45:13.653 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.219353+0000", 2024-09-21T10:45:13.653 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9876999999999999e-05 2024-09-21T10:45:13.654 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:13.655 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.655 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:45:13.655 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.655 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:235687 53.14 53:2df42cca:::benchmark_data_smithi070_228967_object106767:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1351)", 2024-09-21T10:45:13.655 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:43:38.512892+0000", 2024-09-21T10:45:13.655 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 95.027152349000005, 2024-09-21T10:45:13.655 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.70667267600000006, 2024-09-21T10:45:13.655 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:45:13.655 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:45:13.655 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:13.655 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:45:13.656 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:45:13.656 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:45:13.656 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235687 2024-09-21T10:45:13.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.656 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:45:13.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.656 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:45:13.656 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.512892+0000", 2024-09-21T10:45:13.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.656 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:45:13.656 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.512892+0000", 2024-09-21T10:45:13.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.657 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.657 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:45:13.657 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.512894+0000", 2024-09-21T10:45:13.657 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8589999999999999e-06 2024-09-21T10:45:13.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.657 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.657 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:45:13.657 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.512901+0000", 2024-09-21T10:45:13.657 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9110000000000004e-06 2024-09-21T10:45:13.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.657 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.657 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:45:13.657 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.512921+0000", 2024-09-21T10:45:13.658 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.931e-05 2024-09-21T10:45:13.658 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.658 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.658 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:45:13.658 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.512930+0000", 2024-09-21T10:45:13.658 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3740000000000004e-06 2024-09-21T10:45:13.658 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.658 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.658 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.658 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.513001+0000", 2024-09-21T10:45:13.658 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0593999999999995e-05 2024-09-21T10:45:13.658 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.659 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:45:13.659 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.513013+0000", 2024-09-21T10:45:13.659 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2328e-05 2024-09-21T10:45:13.659 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.659 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.659 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.206965+0000", 2024-09-21T10:45:13.659 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.69395247400000004 2024-09-21T10:45:13.659 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.659 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:45:13.659 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.207070+0000", 2024-09-21T10:45:13.660 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000105066 2024-09-21T10:45:13.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.660 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:45:13.660 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.207169+0000", 2024-09-21T10:45:13.660 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.8220000000000005e-05 2024-09-21T10:45:13.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.660 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:45:13.660 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.212129+0000", 2024-09-21T10:45:13.660 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0049600240000000004 2024-09-21T10:45:13.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.661 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.661 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.219415+0000", 2024-09-21T10:45:13.661 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0072860549999999996 2024-09-21T10:45:13.661 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.661 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.661 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.661 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.219472+0000", 2024-09-21T10:45:13.661 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7034999999999999e-05 2024-09-21T10:45:13.661 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.661 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.661 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:45:13.661 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.219524+0000", 2024-09-21T10:45:13.661 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2301e-05 2024-09-21T10:45:13.661 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.662 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.662 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:45:13.662 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.219565+0000", 2024-09-21T10:45:13.662 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.1124999999999997e-05 2024-09-21T10:45:13.662 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.662 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:45:13.662 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.662 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.662 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.662 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:235702 53.14 53:2bb6ac2c:::benchmark_data_smithi070_228967_object106782:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1351)", 2024-09-21T10:45:13.662 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:43:38.531568+0000", 2024-09-21T10:45:13.662 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 95.008476806999994, 2024-09-21T10:45:13.662 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.69566826999999998, 2024-09-21T10:45:13.662 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:45:13.663 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:45:13.663 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:13.663 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:45:13.663 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:45:13.663 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:45:13.663 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235702 2024-09-21T10:45:13.663 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.663 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:45:13.663 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:45:13.663 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.531568+0000", 2024-09-21T10:45:13.663 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.663 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.663 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:45:13.664 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.531568+0000", 2024-09-21T10:45:13.664 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.664 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.664 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:45:13.664 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.531570+0000", 2024-09-21T10:45:13.664 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.604e-06 2024-09-21T10:45:13.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.664 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.664 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:45:13.664 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.531575+0000", 2024-09-21T10:45:13.664 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9559999999999996e-06 2024-09-21T10:45:13.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.664 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.664 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:45:13.665 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.531591+0000", 2024-09-21T10:45:13.665 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6205000000000002e-05 2024-09-21T10:45:13.665 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.665 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.665 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:45:13.665 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.531598+0000", 2024-09-21T10:45:13.665 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2479999999999999e-06 2024-09-21T10:45:13.665 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.665 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.665 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.665 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.531664+0000", 2024-09-21T10:45:13.665 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6370000000000003e-05 2024-09-21T10:45:13.665 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.665 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.665 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:45:13.665 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.531675+0000", 2024-09-21T10:45:13.666 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1039999999999999e-05 2024-09-21T10:45:13.666 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.666 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.666 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.666 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.209724+0000", 2024-09-21T10:45:13.666 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.67804870900000003 2024-09-21T10:45:13.666 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.666 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.666 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:45:13.666 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.209829+0000", 2024-09-21T10:45:13.666 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000104398 2024-09-21T10:45:13.666 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.666 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.666 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:45:13.667 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.209926+0000", 2024-09-21T10:45:13.667 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7454000000000006e-05 2024-09-21T10:45:13.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.667 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.667 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:45:13.667 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.216966+0000", 2024-09-21T10:45:13.667 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0070399440000000002 2024-09-21T10:45:13.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.667 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.667 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.667 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.219653+0000", 2024-09-21T10:45:13.667 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0026874080000000001 2024-09-21T10:45:13.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.667 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.667 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.668 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.227132+0000", 2024-09-21T10:45:13.668 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0074789219999999998 2024-09-21T10:45:13.668 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.668 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.668 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:45:13.668 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.227188+0000", 2024-09-21T10:45:13.668 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5492999999999999e-05 2024-09-21T10:45:13.668 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.668 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.668 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:45:13.668 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.227236+0000", 2024-09-21T10:45:13.668 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8519000000000001e-05 2024-09-21T10:45:13.668 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.668 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:45:13.668 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.669 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.669 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.669 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:235707 53.14 53:2fe874e0:::benchmark_data_smithi070_228967_object106787:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1351)", 2024-09-21T10:45:13.669 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:43:38.538988+0000", 2024-09-21T10:45:13.669 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 95.001056644000002, 2024-09-21T10:45:13.669 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.68840786499999995, 2024-09-21T10:45:13.669 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:45:13.669 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:45:13.669 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:13.669 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:45:13.669 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:45:13.669 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:45:13.669 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235707 2024-09-21T10:45:13.669 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.669 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:45:13.669 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.670 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:45:13.670 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.538988+0000", 2024-09-21T10:45:13.670 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.670 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.670 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.670 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:45:13.670 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.538988+0000", 2024-09-21T10:45:13.670 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.670 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.670 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.670 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:45:13.670 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.538989+0000", 2024-09-21T10:45:13.670 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7999999999999995e-07 2024-09-21T10:45:13.670 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.670 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.671 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:45:13.671 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.538992+0000", 2024-09-21T10:45:13.671 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.433e-06 2024-09-21T10:45:13.671 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.671 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.671 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:45:13.671 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.539002+0000", 2024-09-21T10:45:13.671 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.8400000000000007e-06 2024-09-21T10:45:13.671 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.671 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.671 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:45:13.671 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.539009+0000", 2024-09-21T10:45:13.671 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.204e-06 2024-09-21T10:45:13.671 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.671 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.672 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.672 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.539070+0000", 2024-09-21T10:45:13.672 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1081999999999996e-05 2024-09-21T10:45:13.672 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.672 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.672 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:45:13.672 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.539080+0000", 2024-09-21T10:45:13.672 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.6199999999999994e-06 2024-09-21T10:45:13.672 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.672 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.672 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.672 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.212157+0000", 2024-09-21T10:45:13.672 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.67307646899999996 2024-09-21T10:45:13.672 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.672 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.673 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:45:13.673 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.212252+0000", 2024-09-21T10:45:13.673 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5029000000000001e-05 2024-09-21T10:45:13.673 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.673 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.673 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:45:13.673 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.212344+0000", 2024-09-21T10:45:13.673 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2026000000000003e-05 2024-09-21T10:45:13.673 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.673 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.673 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:45:13.673 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.219366+0000", 2024-09-21T10:45:13.673 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0070228269999999997 2024-09-21T10:45:13.673 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.673 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.674 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.674 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.223447+0000", 2024-09-21T10:45:13.674 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0040807650000000001 2024-09-21T10:45:13.674 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.674 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.674 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.674 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.227286+0000", 2024-09-21T10:45:13.674 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0038384880000000001 2024-09-21T10:45:13.674 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.674 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.674 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:45:13.674 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.227352+0000", 2024-09-21T10:45:13.674 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6460999999999999e-05 2024-09-21T10:45:13.674 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.675 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.675 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:45:13.675 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.227396+0000", 2024-09-21T10:45:13.675 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3940999999999998e-05 2024-09-21T10:45:13.675 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.675 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:45:13.675 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.675 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.675 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.675 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:235810 53.14 53:29d44d5b:::benchmark_data_smithi070_228967_object106890:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1351)", 2024-09-21T10:45:13.675 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:43:38.732921+0000", 2024-09-21T10:45:13.675 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 94.807124266000002, 2024-09-21T10:45:13.675 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.49466439499999998, 2024-09-21T10:45:13.675 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:45:13.676 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:45:13.676 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:13.676 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:45:13.676 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:45:13.676 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:45:13.676 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235810 2024-09-21T10:45:13.676 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.676 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:45:13.676 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.676 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:45:13.676 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.732921+0000", 2024-09-21T10:45:13.676 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.676 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.676 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.677 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:45:13.677 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.732921+0000", 2024-09-21T10:45:13.677 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.677 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.677 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.677 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:45:13.677 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.732922+0000", 2024-09-21T10:45:13.677 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3430000000000001e-06 2024-09-21T10:45:13.677 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.677 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.677 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:45:13.677 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.732927+0000", 2024-09-21T10:45:13.677 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2009999999999998e-06 2024-09-21T10:45:13.677 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.677 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.678 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:45:13.678 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.732939+0000", 2024-09-21T10:45:13.678 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1841000000000001e-05 2024-09-21T10:45:13.678 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.678 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.678 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:45:13.678 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.732945+0000", 2024-09-21T10:45:13.678 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3470000000000004e-06 2024-09-21T10:45:13.678 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.678 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.678 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.678 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.732992+0000", 2024-09-21T10:45:13.678 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6779000000000001e-05 2024-09-21T10:45:13.678 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.679 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.679 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:45:13.679 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.732999+0000", 2024-09-21T10:45:13.679 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5520000000000001e-06 2024-09-21T10:45:13.679 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.679 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.679 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.679 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.214552+0000", 2024-09-21T10:45:13.679 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.48155361600000002 2024-09-21T10:45:13.679 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.679 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.679 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:45:13.679 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.214649+0000", 2024-09-21T10:45:13.679 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.6316000000000004e-05 2024-09-21T10:45:13.679 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.679 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.680 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:45:13.680 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.214761+0000", 2024-09-21T10:45:13.680 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000112428 2024-09-21T10:45:13.680 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.680 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.680 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:45:13.680 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.219601+0000", 2024-09-21T10:45:13.680 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0048397090000000002 2024-09-21T10:45:13.680 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.680 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.680 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.680 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.223488+0000", 2024-09-21T10:45:13.680 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0038875519999999998 2024-09-21T10:45:13.681 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.681 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.681 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.681 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.227481+0000", 2024-09-21T10:45:13.681 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0039930979999999996 2024-09-21T10:45:13.681 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.681 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.681 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:45:13.681 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.227546+0000", 2024-09-21T10:45:13.681 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4783999999999995e-05 2024-09-21T10:45:13.681 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.681 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.681 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:45:13.681 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.227585+0000", 2024-09-21T10:45:13.682 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8828999999999997e-05 2024-09-21T10:45:13.682 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.682 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:45:13.682 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.682 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.682 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.682 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:235846 53.14 53:2dba28b4:::benchmark_data_smithi070_228967_object106926:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1351)", 2024-09-21T10:45:13.682 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:43:38.865549+0000", 2024-09-21T10:45:13.682 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 94.674495457000006, 2024-09-21T10:45:13.682 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.36599646000000002, 2024-09-21T10:45:13.682 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:45:13.682 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:45:13.682 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:13.683 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:45:13.683 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:45:13.683 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:45:13.683 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235846 2024-09-21T10:45:13.683 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.683 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:45:13.683 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.683 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:45:13.683 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.865549+0000", 2024-09-21T10:45:13.683 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.683 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.683 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.684 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:45:13.684 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.865549+0000", 2024-09-21T10:45:13.684 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:45:13.684 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.684 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.684 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:45:13.684 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.865550+0000", 2024-09-21T10:45:13.684 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3099999999999998e-07 2024-09-21T10:45:13.684 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.684 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.684 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:45:13.684 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.865553+0000", 2024-09-21T10:45:13.684 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7920000000000001e-06 2024-09-21T10:45:13.684 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.685 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.685 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:45:13.685 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.865561+0000", 2024-09-21T10:45:13.685 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3070000000000003e-06 2024-09-21T10:45:13.685 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.685 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.685 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:45:13.685 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.865565+0000", 2024-09-21T10:45:13.685 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.4519999999999999e-06 2024-09-21T10:45:13.685 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.685 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.685 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.685 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.865606+0000", 2024-09-21T10:45:13.685 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0370000000000001e-05 2024-09-21T10:45:13.686 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.686 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.686 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:45:13.686 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:38.865611+0000", 2024-09-21T10:45:13.686 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2299999999999999e-06 2024-09-21T10:45:13.686 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.686 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.686 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:45:13.686 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.217030+0000", 2024-09-21T10:45:13.686 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.35141898599999999 2024-09-21T10:45:13.686 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.686 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.686 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:45:13.686 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.217149+0000", 2024-09-21T10:45:13.686 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011898800000000001 2024-09-21T10:45:13.687 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.687 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.687 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:45:13.687 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.217262+0000", 2024-09-21T10:45:13.687 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000113336 2024-09-21T10:45:13.687 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.687 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.687 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:45:13.687 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.219607+0000", 2024-09-21T10:45:13.687 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0023442350000000001 2024-09-21T10:45:13.687 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.687 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.687 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.687 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.223529+0000", 2024-09-21T10:45:13.687 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0039221270000000001 2024-09-21T10:45:13.688 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.688 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.688 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:13.688 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.231445+0000", 2024-09-21T10:45:13.688 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0079161079999999998 2024-09-21T10:45:13.688 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.688 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.688 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:45:13.688 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.231500+0000", 2024-09-21T10:45:13.688 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5634000000000003e-05 2024-09-21T10:45:13.688 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:45:13.688 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:45:13.688 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:45:13.688 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:39.231546+0000", 2024-09-21T10:45:13.688 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5364000000000001e-05 2024-09-21T10:45:13.688 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.689 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:45:13.689 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.689 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:45:13.689 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:45:13.689 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:45:13.691 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:13.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:13.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:13.793 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:13.829 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:13.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:13.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:13.931 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:13.964 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:13.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:13.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:13.991 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 26 16 58723 58707 141.102 97.875 0.00866804 0.00708179 2024-09-21T10:45:14.066 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:14.103 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:14.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:14.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:14.204 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:14.238 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:14.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:14.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:14.340 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:14.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:14.376 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:14.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:14.394 INFO:teuthology.orchestra.run.smithi070.stderr:set pool 54 pgp_num to 16 2024-09-21T10:45:14.477 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:14.515 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:14.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:14.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:14.616 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:14.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:14.650 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:14.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:14.751 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:14.785 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:14.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:14.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:14 smithi160 ceph-mon[101879]: pgmap v2371: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 114 MiB/s wr, 1.83k op/s 2024-09-21T10:45:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:14 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3073275784' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T10:45:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:14 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T10:45:14.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:14 smithi070 ceph-mon[107385]: pgmap v2371: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 114 MiB/s wr, 1.83k op/s 2024-09-21T10:45:14.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:14 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3073275784' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T10:45:14.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:14 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T10:45:14.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:14 smithi070 ceph-mon[112139]: pgmap v2371: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 114 MiB/s wr, 1.83k op/s 2024-09-21T10:45:14.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:14 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3073275784' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T10:45:14.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:14 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T10:45:14.887 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:14.924 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:14.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:14.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:14.992 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 27 16 60286 60270 139.494 97.6875 0.00895016 0.0071637 2024-09-21T10:45:15.026 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:15.060 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:15.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:15.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:15.161 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:15.197 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:15.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:15.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:15.298 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:15.332 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:15.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:15.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:15.434 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:15.464 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:15.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:15.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:15.565 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:15.602 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:15.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:15.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:15.704 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:15.737 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:15.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:15.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:15.839 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:15.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:15 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-09-21T10:45:15.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:15 smithi160 ceph-mon[101879]: osdmap e1382: 8 total, 8 up, 7 in 2024-09-21T10:45:15.875 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:15.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:15.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:15.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:15 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-09-21T10:45:15.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:15 smithi070 ceph-mon[107385]: osdmap e1382: 8 total, 8 up, 7 in 2024-09-21T10:45:15.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:15 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-09-21T10:45:15.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:15 smithi070 ceph-mon[112139]: osdmap e1382: 8 total, 8 up, 7 in 2024-09-21T10:45:15.978 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:15.992 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 28 16 61915 61899 138.148 101.812 0.00833758 0.00723422 2024-09-21T10:45:16.009 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:16.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:16.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:16.110 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:16.141 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:16.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:16.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:16.243 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:16.263 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:16.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:16.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:16.365 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:16.408 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:16.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:16.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:16.510 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:16.533 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:16.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:16.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:16.635 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:16.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:16.673 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:16.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:16.774 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:16.808 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:16.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:16.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:16.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:16 smithi160 ceph-mon[101879]: pgmap v2373: 89 pgs: 89 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-21T10:45:16.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:16 smithi070 ceph-mon[107385]: pgmap v2373: 89 pgs: 89 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-21T10:45:16.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:16 smithi070 ceph-mon[112139]: pgmap v2373: 89 pgs: 89 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-21T10:45:16.909 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:16.930 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:16.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:16.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:16.992 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 29 16 63028 63012 135.782 69.5625 0.00538912 0.00736016 2024-09-21T10:45:17.032 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:17.069 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:17.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:17.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:17.171 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:17.206 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:17.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:17.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:17.308 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:17.339 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:17.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:17.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:17.440 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:17.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:17.471 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:17.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:17.571 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:17.602 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:17.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:17.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:17.704 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:17.740 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:17.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:17.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:17.842 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:17.880 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:17.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:17.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:17.982 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:17.993 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 30 16 64425 64409 134.167 87.3125 0.0126564 0.00744837 2024-09-21T10:45:18.015 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:18.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:18.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:18.117 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:18.152 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:18.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:18.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:18.254 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:18.285 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:18.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:18.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:18.387 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:18.427 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:18.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:18.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:18.529 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:18.559 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:18.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:18.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:18 smithi070 ceph-mon[107385]: pgmap v2374: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-09-21T10:45:18.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:18 smithi070 ceph-mon[112139]: pgmap v2374: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-09-21T10:45:18.661 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:18.696 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:18.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:18.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:18.798 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:18.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:18.834 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:18.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:18 smithi160 ceph-mon[101879]: pgmap v2374: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-09-21T10:45:18.935 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:18.977 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:18.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:18.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:18.992 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 31 16 65967 65951 132.947 96.375 0.0126271 0.0075166 2024-09-21T10:45:18.998 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:45:19.079 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:19.129 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:19.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:19.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:19.230 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:19.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:19.253 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:19.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:19.316 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:45:19.354 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:19.385 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:19.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:19.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:19.487 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:19.528 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:19.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:19.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:19.629 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:19.660 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:19.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:19.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:19.762 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:19.794 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:19.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:19.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:19.896 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:19.933 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:19.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:19.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:19.992 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 32 16 67269 67253 131.336 81.375 0.0145083 0.00760894 2024-09-21T10:45:20.035 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:20.063 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:20.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:20.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:20.165 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:20.192 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:20.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:20.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:20.293 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:20.332 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:20.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:20.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:20.433 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:20.442 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:20 smithi070 ceph-mon[107385]: pgmap v2375: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-09-21T10:45:20.443 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:45:20.443 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:45:20.443 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:20 smithi070 ceph-mon[112139]: pgmap v2375: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-09-21T10:45:20.443 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:45:20.443 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:45:20.471 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:20.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:20.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:20.573 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:20.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:20 smithi160 ceph-mon[101879]: pgmap v2375: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-09-21T10:45:20.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:45:20.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:45:20.605 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:20.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:20.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:20.707 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:20.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:20.744 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:20.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:20.844 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:20.876 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:20.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:20.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:20.978 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:20.992 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 33 16 68676 68660 130.019 87.9375 0.00650509 0.0076861 2024-09-21T10:45:21.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:21.011 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:21.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:21.112 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:21.139 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:21.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:21.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:21.241 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:21.274 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:21.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:21.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:21.376 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:21.405 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:21.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:21.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:21.506 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:21.535 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:21.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:21.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:21.637 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:21.678 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:21.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:21.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:21.780 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:21.821 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:21.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:21.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:21.923 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:21.956 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:21.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:21.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:21.993 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 34 16 70099 70083 128.811 88.9375 0.0129574 0.00775827 2024-09-21T10:45:22.058 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:22.087 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:22.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:22.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:22.189 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:22.223 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:22.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:22.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:22.251 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:45:22.252 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T10:45:22.252 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:22.252 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:70440 54.8 54:14ff0068:::benchmark_data_smithi070_261699_object70439:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1382)", 2024-09-21T10:45:22.252 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:22.241447+0000", 2024-09-21T10:45:22.252 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0071369490000000001, 2024-09-21T10:45:22.252 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0071713749999999998, 2024-09-21T10:45:22.252 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:45:22.252 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:45:22.252 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:45:22.252 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:45:22.252 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:45:22.252 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 70440 2024-09-21T10:45:22.252 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:22.252 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:45:22.252 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:22.253 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:45:22.253 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:22.241447+0000", 2024-09-21T10:45:22.253 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:45:22.253 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:22.253 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:22.253 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:45:22.253 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:22.241447+0000", 2024-09-21T10:45:22.253 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:45:22.253 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:22.253 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:22.253 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:45:22.253 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:22.241445+0000", 2024-09-21T10:45:22.253 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:45:22.253 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:22.253 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:22.254 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:45:22.254 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:22.241541+0000", 2024-09-21T10:45:22.254 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5574000000000003e-05 2024-09-21T10:45:22.254 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:22.254 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:22.254 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:45:22.254 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:22.241555+0000", 2024-09-21T10:45:22.254 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382e-05 2024-09-21T10:45:22.254 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:22.254 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:22.254 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:45:22.254 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:22.241562+0000", 2024-09-21T10:45:22.254 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3130000000000001e-06 2024-09-21T10:45:22.254 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:22.254 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:22.255 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:45:22.255 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:22.241765+0000", 2024-09-21T10:45:22.255 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00020313899999999999 2024-09-21T10:45:22.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:22.255 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:22.255 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:45:22.255 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:22.242119+0000", 2024-09-21T10:45:22.255 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00035449100000000002 2024-09-21T10:45:22.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:22.255 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:22.255 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:45:22.255 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:22.242579+0000", 2024-09-21T10:45:22.255 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00045993199999999997 2024-09-21T10:45:22.255 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:45:22.255 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:45:22.256 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:45:22.256 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:22.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:22.256 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:70446 54.8 54:151e80c9:::benchmark_data_smithi070_261699_object70445:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1382)", 2024-09-21T10:45:22.256 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:22.243914+0000", 2024-09-21T10:45:22.256 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0046693430000000003, 2024-09-21T10:45:22.256 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0047858019999999996, 2024-09-21T10:45:22.256 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:45:22.256 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:45:22.256 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:45:22.256 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:45:22.256 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:45:22.256 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 70446 2024-09-21T10:45:22.256 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:22.256 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:45:22.257 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:22.257 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:45:22.257 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:22.243914+0000", 2024-09-21T10:45:22.257 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:45:22.257 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:22.257 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:22.257 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:45:22.257 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:22.243914+0000", 2024-09-21T10:45:22.257 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:45:22.257 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:22.257 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:22.257 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:45:22.257 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:22.243912+0000", 2024-09-21T10:45:22.257 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:45:22.257 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:22.257 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:22.258 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:45:22.258 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:22.243987+0000", 2024-09-21T10:45:22.258 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4438999999999997e-05 2024-09-21T10:45:22.258 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:22.258 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:22.258 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:45:22.258 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:22.244004+0000", 2024-09-21T10:45:22.258 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7382999999999999e-05 2024-09-21T10:45:22.258 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:22.258 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:22.258 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:45:22.258 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:22.244013+0000", 2024-09-21T10:45:22.258 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1449999999999997e-06 2024-09-21T10:45:22.258 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:22.258 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:22.259 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:45:22.259 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:22.246201+0000", 2024-09-21T10:45:22.259 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021880229999999999 2024-09-21T10:45:22.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:22.259 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:22.259 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:45:22.259 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:22.246540+0000", 2024-09-21T10:45:22.259 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00033828599999999998 2024-09-21T10:45:22.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:22.259 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:22.259 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:45:22.259 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:22.247002+0000", 2024-09-21T10:45:22.259 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00046274 2024-09-21T10:45:22.259 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:45:22.259 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:45:22.260 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:45:22.260 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:22.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:22.260 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.45308.0:70440 54.8 e1382/1376)", 2024-09-21T10:45:22.260 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:22.245194+0000", 2024-09-21T10:45:22.260 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0033897380000000002, 2024-09-21T10:45:22.260 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0035691350000000002, 2024-09-21T10:45:22.260 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:45:22.260 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T10:45:22.260 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:45:22.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:22.260 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:45:22.260 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:22.245194+0000", 2024-09-21T10:45:22.260 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:45:22.260 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:22.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:22.261 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:45:22.261 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:22.245194+0000", 2024-09-21T10:45:22.261 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:45:22.261 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:22.261 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:22.261 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:45:22.261 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:22.245193+0000", 2024-09-21T10:45:22.261 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:45:22.261 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:22.261 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:22.261 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:45:22.261 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:22.245199+0000", 2024-09-21T10:45:22.261 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7440000000000004e-06 2024-09-21T10:45:22.261 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:22.261 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:22.262 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:45:22.262 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:22.245215+0000", 2024-09-21T10:45:22.262 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6578e-05 2024-09-21T10:45:22.262 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:22.262 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:22.262 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:45:22.262 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:22.245222+0000", 2024-09-21T10:45:22.262 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8369999999999999e-06 2024-09-21T10:45:22.262 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:45:22.262 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:45:22.262 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:45:22.262 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:45:22.262 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T10:45:22.262 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 3 2024-09-21T10:45:22.262 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:45:22.325 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:22.357 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:22.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:22.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:22.459 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:22.499 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:22.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:22.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:22.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:22 smithi160 ceph-mon[101879]: pgmap v2376: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-21T10:45:22.601 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:22 smithi070 ceph-mon[107385]: pgmap v2376: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-21T10:45:22.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:22 smithi070 ceph-mon[112139]: pgmap v2376: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-21T10:45:22.638 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:22.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:22.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:22.740 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:22.779 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:22.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:22.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:22.880 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:22.912 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:22.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:22.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:22.992 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 35 16 71458 71442 127.557 84.9375 0.0166125 0.00783513 2024-09-21T10:45:23.014 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:23.051 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:23.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:23.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:23.153 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:23.192 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:23.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:23.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:23.294 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:23.328 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:23.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:23.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:23.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:45:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:45:23] "GET /metrics HTTP/1.1" 200 36600 "" "Prometheus/2.51.0" 2024-09-21T10:45:23.425 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:45:23.430 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:23.508 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:23.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:23.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:23.610 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:23.629 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:45:23.629 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:45:23.656 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:45:23.656 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:45:23.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:23.669 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:23.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:23.683 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:45:23.684 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:45:23.715 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:45:23.715 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:45:23.746 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:45:23.747 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:45:23.770 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:23.770 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:45:23.785 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:45:23.786 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:45:23.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:23.818 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:23.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:23.829 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:45:23.829 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:45:23.860 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:45:23.860 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:45:23.920 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:23.947 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:23.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:23.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:23.993 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 36 16 72648 72632 126.08 74.375 0.00963552 0.00792617 2024-09-21T10:45:24.048 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:24.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:24.110 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:24.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:24.211 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:24.249 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:24.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:24.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:24.351 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:24.384 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:24.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:24.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:24.440 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T10:45:24.486 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:24.519 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:24.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:24.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:24 smithi160 ceph-mon[101879]: pgmap v2377: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-21T10:45:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:24 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2243820659' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:24 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:45:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:24 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2922638666' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:45:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:24 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:45:24.621 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:24.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:24 smithi070 ceph-mon[107385]: pgmap v2377: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-21T10:45:24.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:24 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2243820659' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:24.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:24 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:45:24.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:24 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2922638666' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:45:24.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:24 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:45:24.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:24 smithi070 ceph-mon[112139]: pgmap v2377: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-21T10:45:24.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:24 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2243820659' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:24.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:24 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:45:24.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:24 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2922638666' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:45:24.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:24 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:45:24.655 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:24.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:24.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:24.757 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:24.797 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:24.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:24.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:24.899 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:24.935 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:24.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:24.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:24.993 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 37 16 73904 73888 124.794 78.5 0.00613277 0.00800806 2024-09-21T10:45:25.037 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:25.069 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:25.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:25.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:25.171 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:25.204 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:25.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:25.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:25.305 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:25.346 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:25.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:25.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:25.448 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:25.480 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:25.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:25.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:25.581 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:25.610 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:25.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:25.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:25.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:25 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:45:25.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:25 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:45:25.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:25 smithi070 ceph-mon[107385]: osdmap e1383: 8 total, 8 up, 7 in 2024-09-21T10:45:25.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:25 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:45:25.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:25 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:45:25.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:25 smithi070 ceph-mon[112139]: osdmap e1383: 8 total, 8 up, 7 in 2024-09-21T10:45:25.712 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:25.721 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:25 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:45:25.721 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:25 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:45:25.722 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:25 smithi160 ceph-mon[101879]: osdmap e1383: 8 total, 8 up, 7 in 2024-09-21T10:45:25.757 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:25.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:25.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:25.777 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:45:25.777 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:45:25.858 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:25.896 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:25.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:25.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:25.993 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 38 16 75092 75076 123.463 74.25 0.0123878 0.00809461 2024-09-21T10:45:25.998 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:26.027 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:26.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:26.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:26.128 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:26.170 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:26.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:26.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:26.271 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:26.310 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:26.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:26.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:26.411 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:26.449 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:26.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:26.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:26.550 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:26.588 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:26.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:26.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:26.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:26 smithi160 ceph-mon[101879]: pgmap v2379: 89 pgs: 89 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-21T10:45:26.630 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:45:26.630 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:45:26.631 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_1 2024-09-21T10:45:26.631 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:45:26.691 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:27.105 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:45:27.106 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:26 smithi070 ceph-mon[107385]: pgmap v2379: 89 pgs: 89 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-21T10:45:27.106 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:26 smithi070 ceph-mon[112139]: pgmap v2379: 89 pgs: 89 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-21T10:45:27.107 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:27.107 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 39 16 76402 76386 122.397 81.875 0.007674 0.00816558 2024-09-21T10:45:27.108 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:45:27.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:27.200 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:27.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:27.301 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:27.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:27.358 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:27.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:27.459 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:27.492 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:27.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:27.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:27.594 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:27.624 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:27.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:27.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:27.725 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:27.750 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:45:27.757 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:27.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:27.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:27.858 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:27.893 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:27.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:27.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:27.910 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:27.994 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:2024-09-21T10:45:27.992873+0000 min lat: 0.00136946 max lat: 0.178279 avg lat: 0.00824856 2024-09-21T10:45:27.994 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:45:27.994 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 40 16 77575 77559 121.168 73.3125 0.00978931 0.00824856 2024-09-21T10:45:27.995 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:28.029 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:28.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:28.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:28.102 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:45:28.132 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:28.190 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:28.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:28.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:28.294 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:28.338 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:28.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:28.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:28.439 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:28.489 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:28.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:28.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:28.508 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:28 smithi160 ceph-mon[101879]: pgmap v2380: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-21T10:45:28.591 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:28.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:28.627 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:28.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:28.728 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:28.757 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:28.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:28.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:28.859 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:28.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:28 smithi070 ceph-mon[112139]: pgmap v2380: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-21T10:45:28.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:28 smithi070 ceph-mon[107385]: pgmap v2380: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-21T10:45:28.892 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:28.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:28.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:28.994 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:28.994 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 41 16 78735 78719 119.981 72.5 0.00617707 0.00832986 2024-09-21T10:45:29.024 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:29.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:29.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:29.125 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:29.167 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:29.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:29.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:29.269 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:29.301 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:29.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:29.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:29.403 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:29.435 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:29.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:29.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:29.536 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:29.566 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:29.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:29.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:29.668 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:29.703 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:29.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:29.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:29.804 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:29.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:29.845 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:29.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:29.946 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:29.984 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:29.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:29.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:29.994 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 42 16 79862 79846 118.801 70.4375 0.016767 0.00841228 2024-09-21T10:45:30.085 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:30.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:30.115 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:30.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:30.216 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:30.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:30.287 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:30.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:30.387 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:30.422 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:30.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:30.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:30.524 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:30.556 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:30.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:30.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:30.609 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:30 smithi070 ceph-mon[112139]: pgmap v2381: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-21T10:45:30.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:30 smithi070 ceph-mon[107385]: pgmap v2381: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-21T10:45:30.657 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:30.685 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:30.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:30.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:30.786 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:30.823 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:30.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:30.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:30 smithi160 ceph-mon[101879]: pgmap v2381: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-21T10:45:30.925 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:30.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:30.963 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:30.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:30.994 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 43 16 81109 81093 117.851 77.9375 0.0140128 0.00848066 2024-09-21T10:45:31.064 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:31.107 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:31.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:31.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:31.209 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:31.260 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:31.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:31.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:31.288 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:45:31.288 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:45:31.362 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:31.397 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:31.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:31.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:31.499 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:31.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:31.536 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:31.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:31.576 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:45:31.577 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T10:45:31.637 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:31.673 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:31.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:31.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:31.775 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:31.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:31.810 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:31.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:31.911 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:31.959 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:31.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:31.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:31.994 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 44 16 82353 82337 116.939 77.75 0.0194109 0.0085469 2024-09-21T10:45:32.061 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:32.103 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:32.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:32.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:32.205 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:32.247 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:32.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:32.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:32.278 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:45:32.349 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:32.376 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:32.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:32.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:32.430 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":2382,"stamp":"2024-09-21T10:45:30.765927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5066921957,"num_objects":77332,"num_object_clones":0,"num_object_copies":231996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77332,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":77585,"num_write_kb":4950033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2962150,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77514,"ondisk_log_size":77514,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":17244752,"kb_used_data":14820180,"kb_used_omap":219,"kb_used_meta":2424292,"kb_avail":638971312,"statfs":{"total":671965249536,"available":654306623488,"internally_reserved":0,"allocated":15175864320,"data_stored":15145518147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":225222,"internal_metadata":2482475066},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":842858496,"num_objects":12861,"num_object_clones":0,"num_object_copies":38583,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12861,"num_write_kb":823104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003204"},"pg_stats":[{"pgid":"54.f","version":"1383'4987","reported_seq":5013,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:29.638479+0000","last_change":"2024-09-21T10:44:32.818675+0000","last_active":"2024-09-21T10:45:29.638479+0000","last_peered":"2024-09-21T10:45:29.638479+0000","last_clean":"2024-09-21T10:45:29.638479+0000","last_became_active":"2024-09-21T10:44:32.818171+0000","last_became_peered":"2024-09-21T10:44:32.818171+0000","last_unstale":"2024-09-21T10:45:29.638479+0000","last_undegraded":"2024-09-21T10:45:29.638479+0000","last_fullsized":"2024-09-21T10:45:29.638479+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":4987,"log_dups_size":0,"ondisk_log_size":4987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:43:37.724584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326828032,"num_objects":4987,"num_object_clones":0,"num_object_copies":14961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4987,"num_write_kb":319168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.e","version":"1383'4674","reported_seq":4690,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:26.276953+0000","last_change":"2024-09-21T10:44:32.821023+0000","last_active":"2024-09-21T10:45:26.276953+0000","last_peered":"2024-09-21T10:45:26.276953+0000","last_clean":"2024-09-21T10:45:26.276953+0000","last_became_active":"2024-09-21T10:44:32.820651+0000","last_became_peered":"2024-09-21T10:44:32.820651+0000","last_unstale":"2024-09-21T10:45:26.276953+0000","last_undegraded":"2024-09-21T10:45:26.276953+0000","last_fullsized":"2024-09-21T10:45:26.276953+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":4674,"log_dups_size":0,"ondisk_log_size":4674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:39:13.493921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306315264,"num_objects":4674,"num_object_clones":0,"num_object_copies":14022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4674,"num_write_kb":299136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"54.d","version":"1383'4690","reported_seq":4706,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:26.301021+0000","last_change":"2024-09-21T10:44:32.816573+0000","last_active":"2024-09-21T10:45:26.301021+0000","last_peered":"2024-09-21T10:45:26.301021+0000","last_clean":"2024-09-21T10:45:26.301021+0000","last_became_active":"2024-09-21T10:44:32.816260+0000","last_became_peered":"2024-09-21T10:44:32.816260+0000","last_unstale":"2024-09-21T10:45:26.301021+0000","last_undegraded":"2024-09-21T10:45:26.301021+0000","last_fullsized":"2024-09-21T10:45:26.301021+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:20:47.764042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307363840,"num_objects":4690,"num_object_clones":0,"num_object_copies":14070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4690,"num_write_kb":300160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"54.c","version":"1383'4715","reported_seq":4741,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:29.641976+0000","last_change":"2024-09-21T10:44:32.819873+0000","last_active":"2024-09-21T10:45:29.641976+0000","last_peered":"2024-09-21T10:45:29.641976+0000","last_clean":"2024-09-21T10:45:29.641976+0000","last_became_active":"2024-09-21T10:44:32.819256+0000","last_became_peered":"2024-09-21T10:44:32.819256+0000","last_unstale":"2024-09-21T10:45:29.641976+0000","last_undegraded":"2024-09-21T10:45:29.641976+0000","last_fullsized":"2024-09-21T10:45:29.641976+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":4715,"log_dups_size":0,"ondisk_log_size":4715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:05:05.243860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309002240,"num_objects":4715,"num_object_clones":0,"num_object_copies":14145,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4715,"num_write_kb":301760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.b","version":"1383'4913","reported_seq":4939,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:29.658171+0000","last_change":"2024-09-21T10:44:32.819321+0000","last_active":"2024-09-21T10:45:29.658171+0000","last_peered":"2024-09-21T10:45:29.658171+0000","last_clean":"2024-09-21T10:45:29.658171+0000","last_became_active":"2024-09-21T10:44:32.819084+0000","last_became_peered":"2024-09-21T10:44:32.819084+0000","last_unstale":"2024-09-21T10:45:29.658171+0000","last_undegraded":"2024-09-21T10:45:29.658171+0000","last_fullsized":"2024-09-21T10:45:29.658171+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":4913,"log_dups_size":0,"ondisk_log_size":4913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:45:43.371058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321978368,"num_objects":4913,"num_object_clones":0,"num_object_copies":14739,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4913,"num_write_kb":314432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.a","version":"1383'4734","reported_seq":4750,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:26.302302+0000","last_change":"2024-09-21T10:44:32.818041+0000","last_active":"2024-09-21T10:45:26.302302+0000","last_peered":"2024-09-21T10:45:26.302302+0000","last_clean":"2024-09-21T10:45:26.302302+0000","last_became_active":"2024-09-21T10:44:32.817725+0000","last_became_peered":"2024-09-21T10:44:32.817725+0000","last_unstale":"2024-09-21T10:45:26.302302+0000","last_undegraded":"2024-09-21T10:45:26.302302+0000","last_fullsized":"2024-09-21T10:45:26.302302+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":4734,"log_dups_size":0,"ondisk_log_size":4734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:22:24.084717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310247424,"num_objects":4734,"num_object_clones":0,"num_object_copies":14202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4734,"num_write_kb":302976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"54.9","version":"1383'4915","reported_seq":4941,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:29.405541+0000","last_change":"2024-09-21T10:44:32.816734+0000","last_active":"2024-09-21T10:45:29.405541+0000","last_peered":"2024-09-21T10:45:29.405541+0000","last_clean":"2024-09-21T10:45:29.405541+0000","last_became_active":"2024-09-21T10:44:32.816261+0000","last_became_peered":"2024-09-21T10:44:32.816261+0000","last_unstale":"2024-09-21T10:45:29.405541+0000","last_undegraded":"2024-09-21T10:45:29.405541+0000","last_fullsized":"2024-09-21T10:45:29.405541+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":4915,"log_dups_size":0,"ondisk_log_size":4915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:48:11.051953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322109440,"num_objects":4915,"num_object_clones":0,"num_object_copies":14745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4915,"num_write_kb":314560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"54.8","version":"1383'4856","reported_seq":4871,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:26.259793+0000","last_change":"2024-09-21T10:44:32.818221+0000","last_active":"2024-09-21T10:45:26.259793+0000","last_peered":"2024-09-21T10:45:26.259793+0000","last_clean":"2024-09-21T10:45:26.259793+0000","last_became_active":"2024-09-21T10:44:32.817941+0000","last_became_peered":"2024-09-21T10:44:32.817941+0000","last_unstale":"2024-09-21T10:45:26.259793+0000","last_undegraded":"2024-09-21T10:45:26.259793+0000","last_fullsized":"2024-09-21T10:45:26.259793+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":4856,"log_dups_size":0,"ondisk_log_size":4856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:11:40.024092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318242816,"num_objects":4856,"num_object_clones":0,"num_object_copies":14568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4856,"num_write_kb":310784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"54.7","version":"1383'5042","reported_seq":5067,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:29.670617+0000","last_change":"2024-09-21T10:44:32.818432+0000","last_active":"2024-09-21T10:45:29.670617+0000","last_peered":"2024-09-21T10:45:29.670617+0000","last_clean":"2024-09-21T10:45:29.670617+0000","last_became_active":"2024-09-21T10:44:32.818094+0000","last_became_peered":"2024-09-21T10:44:32.818094+0000","last_unstale":"2024-09-21T10:45:29.670617+0000","last_undegraded":"2024-09-21T10:45:29.670617+0000","last_fullsized":"2024-09-21T10:45:29.670617+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":5042,"log_dups_size":0,"ondisk_log_size":5042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:39.301581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330432512,"num_objects":5042,"num_object_clones":0,"num_object_copies":15126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5042,"num_write_kb":322688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.6","version":"1383'4744","reported_seq":4759,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:26.030666+0000","last_change":"2024-09-21T10:44:32.822589+0000","last_active":"2024-09-21T10:45:26.030666+0000","last_peered":"2024-09-21T10:45:26.030666+0000","last_clean":"2024-09-21T10:45:26.030666+0000","last_became_active":"2024-09-21T10:44:32.822265+0000","last_became_peered":"2024-09-21T10:44:32.822265+0000","last_unstale":"2024-09-21T10:45:26.030666+0000","last_undegraded":"2024-09-21T10:45:26.030666+0000","last_fullsized":"2024-09-21T10:45:26.030666+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":4744,"log_dups_size":0,"ondisk_log_size":4744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:40:44.356501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310902784,"num_objects":4744,"num_object_clones":0,"num_object_copies":14232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4744,"num_write_kb":303616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"54.5","version":"1383'4776","reported_seq":4791,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:26.302899+0000","last_change":"2024-09-21T10:44:32.815686+0000","last_active":"2024-09-21T10:45:26.302899+0000","last_peered":"2024-09-21T10:45:26.302899+0000","last_clean":"2024-09-21T10:45:26.302899+0000","last_became_active":"2024-09-21T10:44:32.815266+0000","last_became_peered":"2024-09-21T10:44:32.815266+0000","last_unstale":"2024-09-21T10:45:26.302899+0000","last_undegraded":"2024-09-21T10:45:26.302899+0000","last_fullsized":"2024-09-21T10:45:26.302899+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":4776,"log_dups_size":0,"ondisk_log_size":4776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:11:59.857675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312999936,"num_objects":4776,"num_object_clones":0,"num_object_copies":14328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4776,"num_write_kb":305664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"54.4","version":"1383'5007","reported_seq":5033,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:29.653756+0000","last_change":"2024-09-21T10:44:32.821087+0000","last_active":"2024-09-21T10:45:29.653756+0000","last_peered":"2024-09-21T10:45:29.653756+0000","last_clean":"2024-09-21T10:45:29.653756+0000","last_became_active":"2024-09-21T10:44:32.820712+0000","last_became_peered":"2024-09-21T10:44:32.820712+0000","last_unstale":"2024-09-21T10:45:29.653756+0000","last_undegraded":"2024-09-21T10:45:29.653756+0000","last_fullsized":"2024-09-21T10:45:29.653756+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":5007,"log_dups_size":0,"ondisk_log_size":5007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:06:30.540016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328138752,"num_objects":5007,"num_object_clones":0,"num_object_copies":15021,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5007,"num_write_kb":320448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.3","version":"1383'4939","reported_seq":4965,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:29.400334+0000","last_change":"2024-09-21T10:44:32.817547+0000","last_active":"2024-09-21T10:45:29.400334+0000","last_peered":"2024-09-21T10:45:29.400334+0000","last_clean":"2024-09-21T10:45:29.400334+0000","last_became_active":"2024-09-21T10:44:32.817238+0000","last_became_peered":"2024-09-21T10:44:32.817238+0000","last_unstale":"2024-09-21T10:45:29.400334+0000","last_undegraded":"2024-09-21T10:45:29.400334+0000","last_fullsized":"2024-09-21T10:45:29.400334+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":4939,"log_dups_size":0,"ondisk_log_size":4939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:31:41.893668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323682304,"num_objects":4939,"num_object_clones":0,"num_object_copies":14817,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4939,"num_write_kb":316096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"54.2","version":"1383'4918","reported_seq":4944,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:29.796431+0000","last_change":"2024-09-21T10:44:32.820419+0000","last_active":"2024-09-21T10:45:29.796431+0000","last_peered":"2024-09-21T10:45:29.796431+0000","last_clean":"2024-09-21T10:45:29.796431+0000","last_became_active":"2024-09-21T10:44:32.820081+0000","last_became_peered":"2024-09-21T10:44:32.820081+0000","last_unstale":"2024-09-21T10:45:29.796431+0000","last_undegraded":"2024-09-21T10:45:29.796431+0000","last_fullsized":"2024-09-21T10:45:29.796431+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":4918,"log_dups_size":0,"ondisk_log_size":4918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:16:04.392290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322306048,"num_objects":4918,"num_object_clones":0,"num_object_copies":14754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4918,"num_write_kb":314752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"54.1","version":"1383'4709","reported_seq":4725,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:26.155134+0000","last_change":"2024-09-21T10:44:32.819407+0000","last_active":"2024-09-21T10:45:26.155134+0000","last_peered":"2024-09-21T10:45:26.155134+0000","last_clean":"2024-09-21T10:45:26.155134+0000","last_became_active":"2024-09-21T10:44:32.819056+0000","last_became_peered":"2024-09-21T10:44:32.819056+0000","last_unstale":"2024-09-21T10:45:26.155134+0000","last_undegraded":"2024-09-21T10:45:26.155134+0000","last_fullsized":"2024-09-21T10:45:26.155134+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":4709,"log_dups_size":0,"ondisk_log_size":4709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:39:58.641282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308609024,"num_objects":4709,"num_object_clones":0,"num_object_copies":14127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4709,"num_write_kb":301376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"54.0","version":"1383'4687","reported_seq":4703,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:26.285924+0000","last_change":"2024-09-21T10:44:32.816008+0000","last_active":"2024-09-21T10:45:26.285924+0000","last_peered":"2024-09-21T10:45:26.285924+0000","last_clean":"2024-09-21T10:45:26.285924+0000","last_became_active":"2024-09-21T10:44:32.815627+0000","last_became_peered":"2024-09-21T10:44:32.815627+0000","last_unstale":"2024-09-21T10:45:26.285924+0000","last_undegraded":"2024-09-21T10:45:26.285924+0000","last_fullsized":"2024-09-21T10:45:26.285924+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":4687,"log_dups_size":0,"ondisk_log_size":4687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:48:54.596883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307167232,"num_objects":4687,"num_object_clones":0,"num_object_copies":14061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4687,"num_write_kb":299968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":2078,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.440529+0000","last_change":"2024-09-21T10:40:13.438023+0000","last_active":"2024-09-21T10:45:24.440529+0000","last_peered":"2024-09-21T10:45:24.440529+0000","last_clean":"2024-09-21T10:45:24.440529+0000","last_became_active":"2024-09-21T10:40:13.431103+0000","last_became_peered":"2024-09-21T10:40:13.431103+0000","last_unstale":"2024-09-21T10:45:24.440529+0000","last_undegraded":"2024-09-21T10:45:24.440529+0000","last_fullsized":"2024-09-21T10:45:24.440529+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:00:24.887661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1351,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.406568+0000","last_change":"2024-09-21T10:07:55.999392+0000","last_active":"2024-09-21T10:45:14.406568+0000","last_peered":"2024-09-21T10:45:14.406568+0000","last_clean":"2024-09-21T10:45:14.406568+0000","last_became_active":"2024-09-21T10:07:55.999103+0000","last_became_peered":"2024-09-21T10:07:55.999103+0000","last_unstale":"2024-09-21T10:45:14.406568+0000","last_undegraded":"2024-09-21T10:45:14.406568+0000","last_fullsized":"2024-09-21T10:45:14.406568+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:48.757242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":2059,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.436291+0000","last_change":"2024-09-21T10:40:13.432818+0000","last_active":"2024-09-21T10:45:24.436291+0000","last_peered":"2024-09-21T10:45:24.436291+0000","last_clean":"2024-09-21T10:45:24.436291+0000","last_became_active":"2024-09-21T10:40:13.432448+0000","last_became_peered":"2024-09-21T10:40:13.432448+0000","last_unstale":"2024-09-21T10:45:24.436291+0000","last_undegraded":"2024-09-21T10:45:24.436291+0000","last_fullsized":"2024-09-21T10:45:24.436291+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:20.448242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2076,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.436985+0000","last_change":"2024-09-21T10:40:13.431824+0000","last_active":"2024-09-21T10:45:24.436985+0000","last_peered":"2024-09-21T10:45:24.436985+0000","last_clean":"2024-09-21T10:45:24.436985+0000","last_became_active":"2024-09-21T10:40:13.431516+0000","last_became_peered":"2024-09-21T10:40:13.431516+0000","last_unstale":"2024-09-21T10:45:24.436985+0000","last_undegraded":"2024-09-21T10:45:24.436985+0000","last_fullsized":"2024-09-21T10:45:24.436985+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:30:40.226452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":1339,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.400896+0000","last_change":"2024-09-21T09:54:53.634508+0000","last_active":"2024-09-21T10:45:14.400896+0000","last_peered":"2024-09-21T10:45:14.400896+0000","last_clean":"2024-09-21T10:45:14.400896+0000","last_became_active":"2024-09-21T09:54:53.634254+0000","last_became_peered":"2024-09-21T09:54:53.634254+0000","last_unstale":"2024-09-21T10:45:14.400896+0000","last_undegraded":"2024-09-21T10:45:14.400896+0000","last_fullsized":"2024-09-21T10:45:14.400896+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:18.122107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1366,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.406363+0000","last_change":"2024-09-21T10:09:06.574933+0000","last_active":"2024-09-21T10:45:14.406363+0000","last_peered":"2024-09-21T10:45:14.406363+0000","last_clean":"2024-09-21T10:45:14.406363+0000","last_became_active":"2024-09-21T10:09:06.574467+0000","last_became_peered":"2024-09-21T10:09:06.574467+0000","last_unstale":"2024-09-21T10:45:14.406363+0000","last_undegraded":"2024-09-21T10:45:14.406363+0000","last_fullsized":"2024-09-21T10:45:14.406363+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:06.142269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":1354,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.401437+0000","last_change":"2024-09-21T10:07:55.991189+0000","last_active":"2024-09-21T10:45:14.401437+0000","last_peered":"2024-09-21T10:45:14.401437+0000","last_clean":"2024-09-21T10:45:14.401437+0000","last_became_active":"2024-09-21T10:07:55.990784+0000","last_became_peered":"2024-09-21T10:07:55.990784+0000","last_unstale":"2024-09-21T10:45:14.401437+0000","last_undegraded":"2024-09-21T10:45:14.401437+0000","last_fullsized":"2024-09-21T10:45:14.401437+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:00.580471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1366,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.406928+0000","last_change":"2024-09-21T10:09:06.560175+0000","last_active":"2024-09-21T10:45:14.406928+0000","last_peered":"2024-09-21T10:45:14.406928+0000","last_clean":"2024-09-21T10:45:14.406928+0000","last_became_active":"2024-09-21T10:09:06.558584+0000","last_became_peered":"2024-09-21T10:09:06.558584+0000","last_unstale":"2024-09-21T10:45:14.406928+0000","last_undegraded":"2024-09-21T10:45:14.406928+0000","last_fullsized":"2024-09-21T10:45:14.406928+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:02.918339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":2061,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.440957+0000","last_change":"2024-09-21T10:44:14.409321+0000","last_active":"2024-09-21T10:45:24.440957+0000","last_peered":"2024-09-21T10:45:24.440957+0000","last_clean":"2024-09-21T10:45:24.440957+0000","last_became_active":"2024-09-21T10:44:14.408833+0000","last_became_peered":"2024-09-21T10:44:14.408833+0000","last_unstale":"2024-09-21T10:45:24.440957+0000","last_undegraded":"2024-09-21T10:45:24.440957+0000","last_fullsized":"2024-09-21T10:45:24.440957+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:10:00.997210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":838,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.443212+0000","last_change":"2024-09-21T10:40:13.436927+0000","last_active":"2024-09-21T10:45:24.443212+0000","last_peered":"2024-09-21T10:45:24.443212+0000","last_clean":"2024-09-21T10:45:24.443212+0000","last_became_active":"2024-09-21T10:40:13.436329+0000","last_became_peered":"2024-09-21T10:40:13.436329+0000","last_unstale":"2024-09-21T10:45:24.443212+0000","last_undegraded":"2024-09-21T10:45:24.443212+0000","last_fullsized":"2024-09-21T10:45:24.443212+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:05.668898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":1377,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.406541+0000","last_change":"2024-09-21T10:40:13.433278+0000","last_active":"2024-09-21T10:45:14.406541+0000","last_peered":"2024-09-21T10:45:14.406541+0000","last_clean":"2024-09-21T10:45:14.406541+0000","last_became_active":"2024-09-21T10:40:13.432814+0000","last_became_peered":"2024-09-21T10:40:13.432814+0000","last_unstale":"2024-09-21T10:45:14.406541+0000","last_undegraded":"2024-09-21T10:45:14.406541+0000","last_fullsized":"2024-09-21T10:45:14.406541+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:55.909872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2069,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.436506+0000","last_change":"2024-09-21T10:40:13.432531+0000","last_active":"2024-09-21T10:45:24.436506+0000","last_peered":"2024-09-21T10:45:24.436506+0000","last_clean":"2024-09-21T10:45:24.436506+0000","last_became_active":"2024-09-21T10:40:13.432210+0000","last_became_peered":"2024-09-21T10:40:13.432210+0000","last_unstale":"2024-09-21T10:45:24.436506+0000","last_undegraded":"2024-09-21T10:45:24.436506+0000","last_fullsized":"2024-09-21T10:45:24.436506+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:20:10.998607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2041,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.441999+0000","last_change":"2024-09-21T10:09:06.530394+0000","last_active":"2024-09-21T10:45:24.441999+0000","last_peered":"2024-09-21T10:45:24.441999+0000","last_clean":"2024-09-21T10:45:24.441999+0000","last_became_active":"2024-09-21T10:09:06.529307+0000","last_became_peered":"2024-09-21T10:09:06.529307+0000","last_unstale":"2024-09-21T10:45:24.441999+0000","last_undegraded":"2024-09-21T10:45:24.441999+0000","last_fullsized":"2024-09-21T10:45:24.441999+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:54.287397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2074,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.441666+0000","last_change":"2024-09-21T10:44:14.409422+0000","last_active":"2024-09-21T10:45:24.441666+0000","last_peered":"2024-09-21T10:45:24.441666+0000","last_clean":"2024-09-21T10:45:24.441666+0000","last_became_active":"2024-09-21T10:44:14.408726+0000","last_became_peered":"2024-09-21T10:44:14.408726+0000","last_unstale":"2024-09-21T10:45:24.441666+0000","last_undegraded":"2024-09-21T10:45:24.441666+0000","last_fullsized":"2024-09-21T10:45:24.441666+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:44:21.743112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2056,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.442979+0000","last_change":"2024-09-21T10:09:06.552324+0000","last_active":"2024-09-21T10:45:24.442979+0000","last_peered":"2024-09-21T10:45:24.442979+0000","last_clean":"2024-09-21T10:45:24.442979+0000","last_became_active":"2024-09-21T10:09:06.540545+0000","last_became_peered":"2024-09-21T10:09:06.540545+0000","last_unstale":"2024-09-21T10:45:24.442979+0000","last_undegraded":"2024-09-21T10:45:24.442979+0000","last_fullsized":"2024-09-21T10:45:24.442979+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:08.506149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1356,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.402487+0000","last_change":"2024-09-21T10:07:56.003005+0000","last_active":"2024-09-21T10:45:14.402487+0000","last_peered":"2024-09-21T10:45:14.402487+0000","last_clean":"2024-09-21T10:45:14.402487+0000","last_became_active":"2024-09-21T10:07:56.001550+0000","last_became_peered":"2024-09-21T10:07:56.001550+0000","last_unstale":"2024-09-21T10:45:14.402487+0000","last_undegraded":"2024-09-21T10:45:14.402487+0000","last_fullsized":"2024-09-21T10:45:14.402487+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:54.553261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2048,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.443110+0000","last_change":"2024-09-21T10:09:06.559017+0000","last_active":"2024-09-21T10:45:24.443110+0000","last_peered":"2024-09-21T10:45:24.443110+0000","last_clean":"2024-09-21T10:45:24.443110+0000","last_became_active":"2024-09-21T10:09:06.555256+0000","last_became_peered":"2024-09-21T10:09:06.555256+0000","last_unstale":"2024-09-21T10:45:24.443110+0000","last_undegraded":"2024-09-21T10:45:24.443110+0000","last_fullsized":"2024-09-21T10:45:24.443110+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:33:45.145138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2053,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.440829+0000","last_change":"2024-09-21T10:09:06.562504+0000","last_active":"2024-09-21T10:45:24.440829+0000","last_peered":"2024-09-21T10:45:24.440829+0000","last_clean":"2024-09-21T10:45:24.440829+0000","last_became_active":"2024-09-21T10:09:06.561747+0000","last_became_peered":"2024-09-21T10:09:06.561747+0000","last_unstale":"2024-09-21T10:45:24.440829+0000","last_undegraded":"2024-09-21T10:45:24.440829+0000","last_fullsized":"2024-09-21T10:45:24.440829+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:03:58.012406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2053,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.437139+0000","last_change":"2024-09-21T10:44:14.399948+0000","last_active":"2024-09-21T10:45:24.437139+0000","last_peered":"2024-09-21T10:45:24.437139+0000","last_clean":"2024-09-21T10:45:24.437139+0000","last_became_active":"2024-09-21T10:44:14.399337+0000","last_became_peered":"2024-09-21T10:44:14.399337+0000","last_unstale":"2024-09-21T10:45:24.437139+0000","last_undegraded":"2024-09-21T10:45:24.437139+0000","last_fullsized":"2024-09-21T10:45:24.437139+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:40.185020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1370,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.401467+0000","last_change":"2024-09-21T10:44:14.401716+0000","last_active":"2024-09-21T10:45:14.401467+0000","last_peered":"2024-09-21T10:45:14.401467+0000","last_clean":"2024-09-21T10:45:14.401467+0000","last_became_active":"2024-09-21T10:44:14.401254+0000","last_became_peered":"2024-09-21T10:44:14.401254+0000","last_unstale":"2024-09-21T10:45:14.401467+0000","last_undegraded":"2024-09-21T10:45:14.401467+0000","last_fullsized":"2024-09-21T10:45:14.401467+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:07:21.536272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":30,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.398499+0000","last_change":"2024-09-21T10:44:14.424249+0000","last_active":"2024-09-21T10:45:14.398499+0000","last_peered":"2024-09-21T10:45:14.398499+0000","last_clean":"2024-09-21T10:45:14.398499+0000","last_became_active":"2024-09-21T10:44:14.408884+0000","last_became_peered":"2024-09-21T10:44:14.408884+0000","last_unstale":"2024-09-21T10:45:14.398499+0000","last_undegraded":"2024-09-21T10:45:14.398499+0000","last_fullsized":"2024-09-21T10:45:14.398499+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:18:44.759435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2048,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.442254+0000","last_change":"2024-09-21T10:40:13.437741+0000","last_active":"2024-09-21T10:45:24.442254+0000","last_peered":"2024-09-21T10:45:24.442254+0000","last_clean":"2024-09-21T10:45:24.442254+0000","last_became_active":"2024-09-21T10:40:13.437447+0000","last_became_peered":"2024-09-21T10:40:13.437447+0000","last_unstale":"2024-09-21T10:45:24.442254+0000","last_undegraded":"2024-09-21T10:45:24.442254+0000","last_fullsized":"2024-09-21T10:45:24.442254+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:40:47.072907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":1399,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.407957+0000","last_change":"2024-09-21T10:44:14.415264+0000","last_active":"2024-09-21T10:45:14.407957+0000","last_peered":"2024-09-21T10:45:14.407957+0000","last_clean":"2024-09-21T10:45:14.407957+0000","last_became_active":"2024-09-21T10:44:14.407813+0000","last_became_peered":"2024-09-21T10:44:14.407813+0000","last_unstale":"2024-09-21T10:45:14.407957+0000","last_undegraded":"2024-09-21T10:45:14.407957+0000","last_fullsized":"2024-09-21T10:45:14.407957+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:55.933601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1352,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.407000+0000","last_change":"2024-09-21T10:07:56.001756+0000","last_active":"2024-09-21T10:45:14.407000+0000","last_peered":"2024-09-21T10:45:14.407000+0000","last_clean":"2024-09-21T10:45:14.407000+0000","last_became_active":"2024-09-21T10:07:56.000789+0000","last_became_peered":"2024-09-21T10:07:56.000789+0000","last_unstale":"2024-09-21T10:45:14.407000+0000","last_undegraded":"2024-09-21T10:45:14.407000+0000","last_fullsized":"2024-09-21T10:45:14.407000+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:56:04.302902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1340,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.406080+0000","last_change":"2024-09-21T09:54:53.636902+0000","last_active":"2024-09-21T10:45:14.406080+0000","last_peered":"2024-09-21T10:45:14.406080+0000","last_clean":"2024-09-21T10:45:14.406080+0000","last_became_active":"2024-09-21T09:54:53.636616+0000","last_became_peered":"2024-09-21T09:54:53.636616+0000","last_unstale":"2024-09-21T10:45:14.406080+0000","last_undegraded":"2024-09-21T10:45:14.406080+0000","last_fullsized":"2024-09-21T10:45:14.406080+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:46:47.694495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1395,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.402565+0000","last_change":"2024-09-21T10:44:14.402059+0000","last_active":"2024-09-21T10:45:14.402565+0000","last_peered":"2024-09-21T10:45:14.402565+0000","last_clean":"2024-09-21T10:45:14.402565+0000","last_became_active":"2024-09-21T10:44:14.401710+0000","last_became_peered":"2024-09-21T10:44:14.401710+0000","last_unstale":"2024-09-21T10:45:14.402565+0000","last_undegraded":"2024-09-21T10:45:14.402565+0000","last_fullsized":"2024-09-21T10:45:14.402565+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:17.314693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":2061,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.441559+0000","last_change":"2024-09-21T10:07:56.023462+0000","last_active":"2024-09-21T10:45:24.441559+0000","last_peered":"2024-09-21T10:45:24.441559+0000","last_clean":"2024-09-21T10:45:24.441559+0000","last_became_active":"2024-09-21T10:07:56.021859+0000","last_became_peered":"2024-09-21T10:07:56.021859+0000","last_unstale":"2024-09-21T10:45:24.441559+0000","last_undegraded":"2024-09-21T10:45:24.441559+0000","last_fullsized":"2024-09-21T10:45:24.441559+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:42:03.134729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2050,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.443025+0000","last_change":"2024-09-21T10:44:14.399028+0000","last_active":"2024-09-21T10:45:24.443025+0000","last_peered":"2024-09-21T10:45:24.443025+0000","last_clean":"2024-09-21T10:45:24.443025+0000","last_became_active":"2024-09-21T10:44:14.398115+0000","last_became_peered":"2024-09-21T10:44:14.398115+0000","last_unstale":"2024-09-21T10:45:24.443025+0000","last_undegraded":"2024-09-21T10:45:24.443025+0000","last_fullsized":"2024-09-21T10:45:24.443025+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:05.170376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2042,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.446079+0000","last_change":"2024-09-21T10:07:56.006236+0000","last_active":"2024-09-21T10:45:24.446079+0000","last_peered":"2024-09-21T10:45:24.446079+0000","last_clean":"2024-09-21T10:45:24.446079+0000","last_became_active":"2024-09-21T10:07:56.004439+0000","last_became_peered":"2024-09-21T10:07:56.004439+0000","last_unstale":"2024-09-21T10:45:24.446079+0000","last_undegraded":"2024-09-21T10:45:24.446079+0000","last_fullsized":"2024-09-21T10:45:24.446079+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:39.422467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":2058,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.438731+0000","last_change":"2024-09-21T10:44:14.410440+0000","last_active":"2024-09-21T10:45:24.438731+0000","last_peered":"2024-09-21T10:45:24.438731+0000","last_clean":"2024-09-21T10:45:24.438731+0000","last_became_active":"2024-09-21T10:44:14.400035+0000","last_became_peered":"2024-09-21T10:44:14.400035+0000","last_unstale":"2024-09-21T10:45:24.438731+0000","last_undegraded":"2024-09-21T10:45:24.438731+0000","last_fullsized":"2024-09-21T10:45:24.438731+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:49:39.536516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2062,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.441097+0000","last_change":"2024-09-21T10:44:14.409410+0000","last_active":"2024-09-21T10:45:24.441097+0000","last_peered":"2024-09-21T10:45:24.441097+0000","last_clean":"2024-09-21T10:45:24.441097+0000","last_became_active":"2024-09-21T10:44:14.409025+0000","last_became_peered":"2024-09-21T10:44:14.409025+0000","last_unstale":"2024-09-21T10:45:24.441097+0000","last_undegraded":"2024-09-21T10:45:24.441097+0000","last_fullsized":"2024-09-21T10:45:24.441097+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:25:09.197455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.398418+0000","last_change":"2024-09-21T10:44:14.405677+0000","last_active":"2024-09-21T10:45:14.398418+0000","last_peered":"2024-09-21T10:45:14.398418+0000","last_clean":"2024-09-21T10:45:14.398418+0000","last_became_active":"2024-09-21T10:44:14.405262+0000","last_became_peered":"2024-09-21T10:44:14.405262+0000","last_unstale":"2024-09-21T10:45:14.398418+0000","last_undegraded":"2024-09-21T10:45:14.398418+0000","last_fullsized":"2024-09-21T10:45:14.398418+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:53:12.127245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2078,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.443163+0000","last_change":"2024-09-21T10:40:13.437609+0000","last_active":"2024-09-21T10:45:24.443163+0000","last_peered":"2024-09-21T10:45:24.443163+0000","last_clean":"2024-09-21T10:45:24.443163+0000","last_became_active":"2024-09-21T10:40:13.437296+0000","last_became_peered":"2024-09-21T10:40:13.437296+0000","last_unstale":"2024-09-21T10:45:24.443163+0000","last_undegraded":"2024-09-21T10:45:24.443163+0000","last_fullsized":"2024-09-21T10:45:24.443163+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:41:58.963386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2033,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.442154+0000","last_change":"2024-09-21T10:07:56.067772+0000","last_active":"2024-09-21T10:45:24.442154+0000","last_peered":"2024-09-21T10:45:24.442154+0000","last_clean":"2024-09-21T10:45:24.442154+0000","last_became_active":"2024-09-21T10:07:56.052602+0000","last_became_peered":"2024-09-21T10:07:56.052602+0000","last_unstale":"2024-09-21T10:45:24.442154+0000","last_undegraded":"2024-09-21T10:45:24.442154+0000","last_fullsized":"2024-09-21T10:45:24.442154+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:07:59.899214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2101,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.442682+0000","last_change":"2024-09-21T10:44:14.405182+0000","last_active":"2024-09-21T10:45:24.442682+0000","last_peered":"2024-09-21T10:45:24.442682+0000","last_clean":"2024-09-21T10:45:24.442682+0000","last_became_active":"2024-09-21T10:44:14.404852+0000","last_became_peered":"2024-09-21T10:44:14.404852+0000","last_unstale":"2024-09-21T10:45:24.442682+0000","last_undegraded":"2024-09-21T10:45:24.442682+0000","last_fullsized":"2024-09-21T10:45:24.442682+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:51.687206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1380,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.401047+0000","last_change":"2024-09-21T10:40:13.435375+0000","last_active":"2024-09-21T10:45:14.401047+0000","last_peered":"2024-09-21T10:45:14.401047+0000","last_clean":"2024-09-21T10:45:14.401047+0000","last_became_active":"2024-09-21T10:40:13.435050+0000","last_became_peered":"2024-09-21T10:40:13.435050+0000","last_unstale":"2024-09-21T10:45:14.401047+0000","last_undegraded":"2024-09-21T10:45:14.401047+0000","last_fullsized":"2024-09-21T10:45:14.401047+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:57.889610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2088,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.443338+0000","last_change":"2024-09-21T10:40:13.434189+0000","last_active":"2024-09-21T10:45:24.443338+0000","last_peered":"2024-09-21T10:45:24.443338+0000","last_clean":"2024-09-21T10:45:24.443338+0000","last_became_active":"2024-09-21T10:40:13.433892+0000","last_became_peered":"2024-09-21T10:40:13.433892+0000","last_unstale":"2024-09-21T10:45:24.443338+0000","last_undegraded":"2024-09-21T10:45:24.443338+0000","last_fullsized":"2024-09-21T10:45:24.443338+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:49:17.296220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1356,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.407074+0000","last_change":"2024-09-21T10:07:21.524082+0000","last_active":"2024-09-21T10:45:14.407074+0000","last_peered":"2024-09-21T10:45:14.407074+0000","last_clean":"2024-09-21T10:45:14.407074+0000","last_became_active":"2024-09-21T10:07:21.523779+0000","last_became_peered":"2024-09-21T10:07:21.523779+0000","last_unstale":"2024-09-21T10:45:14.407074+0000","last_undegraded":"2024-09-21T10:45:14.407074+0000","last_fullsized":"2024-09-21T10:45:14.407074+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:13.813832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":1370,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.401184+0000","last_change":"2024-09-21T10:07:56.024239+0000","last_active":"2024-09-21T10:45:14.401184+0000","last_peered":"2024-09-21T10:45:14.401184+0000","last_clean":"2024-09-21T10:45:14.401184+0000","last_became_active":"2024-09-21T10:07:56.023914+0000","last_became_peered":"2024-09-21T10:07:56.023914+0000","last_unstale":"2024-09-21T10:45:14.401184+0000","last_undegraded":"2024-09-21T10:45:14.401184+0000","last_fullsized":"2024-09-21T10:45:14.401184+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:23.361736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084477399999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":75,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:16.273365+0000","last_change":"2024-09-21T10:40:55.616051+0000","last_active":"2024-09-21T10:45:16.273365+0000","last_peered":"2024-09-21T10:45:16.273365+0000","last_clean":"2024-09-21T10:45:16.273365+0000","last_became_active":"2024-09-21T10:40:55.615741+0000","last_became_peered":"2024-09-21T10:40:55.615741+0000","last_unstale":"2024-09-21T10:45:16.273365+0000","last_undegraded":"2024-09-21T10:45:16.273365+0000","last_fullsized":"2024-09-21T10:45:16.273365+0000","mapping_epoch":1320,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:07.673527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2051,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.440333+0000","last_change":"2024-09-21T10:40:13.436234+0000","last_active":"2024-09-21T10:45:24.440333+0000","last_peered":"2024-09-21T10:45:24.440333+0000","last_clean":"2024-09-21T10:45:24.440333+0000","last_became_active":"2024-09-21T10:40:13.435946+0000","last_became_peered":"2024-09-21T10:40:13.435946+0000","last_unstale":"2024-09-21T10:45:24.440333+0000","last_undegraded":"2024-09-21T10:45:24.440333+0000","last_fullsized":"2024-09-21T10:45:24.440333+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:08:08.670739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":844,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.436825+0000","last_change":"2024-09-21T10:40:13.431693+0000","last_active":"2024-09-21T10:45:24.436825+0000","last_peered":"2024-09-21T10:45:24.436825+0000","last_clean":"2024-09-21T10:45:24.436825+0000","last_became_active":"2024-09-21T10:40:13.431389+0000","last_became_peered":"2024-09-21T10:40:13.431389+0000","last_unstale":"2024-09-21T10:45:24.436825+0000","last_undegraded":"2024-09-21T10:45:24.436825+0000","last_fullsized":"2024-09-21T10:45:24.436825+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:22.227336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2090,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.441706+0000","last_change":"2024-09-21T10:44:14.409601+0000","last_active":"2024-09-21T10:45:24.441706+0000","last_peered":"2024-09-21T10:45:24.441706+0000","last_clean":"2024-09-21T10:45:24.441706+0000","last_became_active":"2024-09-21T10:44:14.409262+0000","last_became_peered":"2024-09-21T10:44:14.409262+0000","last_unstale":"2024-09-21T10:45:24.441706+0000","last_undegraded":"2024-09-21T10:45:24.441706+0000","last_fullsized":"2024-09-21T10:45:24.441706+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:53:01.809847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2057,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.440429+0000","last_change":"2024-09-21T10:09:06.560130+0000","last_active":"2024-09-21T10:45:24.440429+0000","last_peered":"2024-09-21T10:45:24.440429+0000","last_clean":"2024-09-21T10:45:24.440429+0000","last_became_active":"2024-09-21T10:09:06.559827+0000","last_became_peered":"2024-09-21T10:09:06.559827+0000","last_unstale":"2024-09-21T10:45:24.440429+0000","last_undegraded":"2024-09-21T10:45:24.440429+0000","last_fullsized":"2024-09-21T10:45:24.440429+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:05:17.166734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.396876+0000","last_change":"2024-09-21T10:44:14.400691+0000","last_active":"2024-09-21T10:45:14.396876+0000","last_peered":"2024-09-21T10:45:14.396876+0000","last_clean":"2024-09-21T10:45:14.396876+0000","last_became_active":"2024-09-21T10:44:14.400377+0000","last_became_peered":"2024-09-21T10:44:14.400377+0000","last_unstale":"2024-09-21T10:45:14.396876+0000","last_undegraded":"2024-09-21T10:45:14.396876+0000","last_fullsized":"2024-09-21T10:45:14.396876+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:03.555078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1359,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.404752+0000","last_change":"2024-09-21T10:44:14.402965+0000","last_active":"2024-09-21T10:45:14.404752+0000","last_peered":"2024-09-21T10:45:14.404752+0000","last_clean":"2024-09-21T10:45:14.404752+0000","last_became_active":"2024-09-21T10:44:14.402634+0000","last_became_peered":"2024-09-21T10:44:14.402634+0000","last_unstale":"2024-09-21T10:45:14.404752+0000","last_undegraded":"2024-09-21T10:45:14.404752+0000","last_fullsized":"2024-09-21T10:45:14.404752+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:40:16.323155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2038,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.436151+0000","last_change":"2024-09-21T10:09:06.560589+0000","last_active":"2024-09-21T10:45:24.436151+0000","last_peered":"2024-09-21T10:45:24.436151+0000","last_clean":"2024-09-21T10:45:24.436151+0000","last_became_active":"2024-09-21T10:09:06.557552+0000","last_became_peered":"2024-09-21T10:09:06.557552+0000","last_unstale":"2024-09-21T10:45:24.436151+0000","last_undegraded":"2024-09-21T10:45:24.436151+0000","last_fullsized":"2024-09-21T10:45:24.436151+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:09:24.866759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":1414,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.400872+0000","last_change":"2024-09-21T10:44:14.415910+0000","last_active":"2024-09-21T10:45:14.400872+0000","last_peered":"2024-09-21T10:45:14.400872+0000","last_clean":"2024-09-21T10:45:14.400872+0000","last_became_active":"2024-09-21T10:44:14.402926+0000","last_became_peered":"2024-09-21T10:44:14.402926+0000","last_unstale":"2024-09-21T10:45:14.400872+0000","last_undegraded":"2024-09-21T10:45:14.400872+0000","last_fullsized":"2024-09-21T10:45:14.400872+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:55:07.144611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":774,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.396807+0000","last_change":"2024-09-21T10:44:17.413348+0000","last_active":"2024-09-21T10:45:14.396807+0000","last_peered":"2024-09-21T10:45:14.396807+0000","last_clean":"2024-09-21T10:45:14.396807+0000","last_became_active":"2024-09-21T10:44:17.412916+0000","last_became_peered":"2024-09-21T10:44:17.412916+0000","last_unstale":"2024-09-21T10:45:14.396807+0000","last_undegraded":"2024-09-21T10:45:14.396807+0000","last_fullsized":"2024-09-21T10:45:14.396807+0000","mapping_epoch":1372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1373,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:18.592283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2042,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.442737+0000","last_change":"2024-09-21T10:09:06.538940+0000","last_active":"2024-09-21T10:45:24.442737+0000","last_peered":"2024-09-21T10:45:24.442737+0000","last_clean":"2024-09-21T10:45:24.442737+0000","last_became_active":"2024-09-21T10:09:06.538668+0000","last_became_peered":"2024-09-21T10:09:06.538668+0000","last_unstale":"2024-09-21T10:45:24.442737+0000","last_undegraded":"2024-09-21T10:45:24.442737+0000","last_fullsized":"2024-09-21T10:45:24.442737+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:42:44.299488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2057,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.442833+0000","last_change":"2024-09-21T10:44:14.398581+0000","last_active":"2024-09-21T10:45:24.442833+0000","last_peered":"2024-09-21T10:45:24.442833+0000","last_clean":"2024-09-21T10:45:24.442833+0000","last_became_active":"2024-09-21T10:44:14.398258+0000","last_became_peered":"2024-09-21T10:44:14.398258+0000","last_unstale":"2024-09-21T10:45:24.442833+0000","last_undegraded":"2024-09-21T10:45:24.442833+0000","last_fullsized":"2024-09-21T10:45:24.442833+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:23.960956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1353,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.408384+0000","last_change":"2024-09-21T10:07:55.998799+0000","last_active":"2024-09-21T10:45:14.408384+0000","last_peered":"2024-09-21T10:45:14.408384+0000","last_clean":"2024-09-21T10:45:14.408384+0000","last_became_active":"2024-09-21T10:07:55.998510+0000","last_became_peered":"2024-09-21T10:07:55.998510+0000","last_unstale":"2024-09-21T10:45:14.408384+0000","last_undegraded":"2024-09-21T10:45:14.408384+0000","last_fullsized":"2024-09-21T10:45:14.408384+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:10.474008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1382,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.406270+0000","last_change":"2024-09-21T10:44:14.399842+0000","last_active":"2024-09-21T10:45:14.406270+0000","last_peered":"2024-09-21T10:45:14.406270+0000","last_clean":"2024-09-21T10:45:14.406270+0000","last_became_active":"2024-09-21T10:44:14.399426+0000","last_became_peered":"2024-09-21T10:44:14.399426+0000","last_unstale":"2024-09-21T10:45:14.406270+0000","last_undegraded":"2024-09-21T10:45:14.406270+0000","last_fullsized":"2024-09-21T10:45:14.406270+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:26:45.362101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1377,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.407319+0000","last_change":"2024-09-21T10:40:13.432387+0000","last_active":"2024-09-21T10:45:14.407319+0000","last_peered":"2024-09-21T10:45:14.407319+0000","last_clean":"2024-09-21T10:45:14.407319+0000","last_became_active":"2024-09-21T10:40:13.432053+0000","last_became_peered":"2024-09-21T10:40:13.432053+0000","last_unstale":"2024-09-21T10:45:14.407319+0000","last_undegraded":"2024-09-21T10:45:14.407319+0000","last_fullsized":"2024-09-21T10:45:14.407319+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:37.486101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1356,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.402782+0000","last_change":"2024-09-21T10:07:56.020791+0000","last_active":"2024-09-21T10:45:14.402782+0000","last_peered":"2024-09-21T10:45:14.402782+0000","last_clean":"2024-09-21T10:45:14.402782+0000","last_became_active":"2024-09-21T10:07:56.020205+0000","last_became_peered":"2024-09-21T10:07:56.020205+0000","last_unstale":"2024-09-21T10:45:14.402782+0000","last_undegraded":"2024-09-21T10:45:14.402782+0000","last_fullsized":"2024-09-21T10:45:14.402782+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:09.813166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1353,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.407877+0000","last_change":"2024-09-21T10:09:06.533271+0000","last_active":"2024-09-21T10:45:14.407877+0000","last_peered":"2024-09-21T10:45:14.407877+0000","last_clean":"2024-09-21T10:45:14.407877+0000","last_became_active":"2024-09-21T10:09:06.532041+0000","last_became_peered":"2024-09-21T10:09:06.532041+0000","last_unstale":"2024-09-21T10:45:14.407877+0000","last_undegraded":"2024-09-21T10:45:14.407877+0000","last_fullsized":"2024-09-21T10:45:14.407877+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:04.236375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":2059,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.437649+0000","last_change":"2024-09-21T10:09:06.543422+0000","last_active":"2024-09-21T10:45:24.437649+0000","last_peered":"2024-09-21T10:45:24.437649+0000","last_clean":"2024-09-21T10:45:24.437649+0000","last_became_active":"2024-09-21T10:09:06.539647+0000","last_became_peered":"2024-09-21T10:09:06.539647+0000","last_unstale":"2024-09-21T10:45:24.437649+0000","last_undegraded":"2024-09-21T10:45:24.437649+0000","last_fullsized":"2024-09-21T10:45:24.437649+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:04.990085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2057,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.440871+0000","last_change":"2024-09-21T10:07:56.018560+0000","last_active":"2024-09-21T10:45:24.440871+0000","last_peered":"2024-09-21T10:45:24.440871+0000","last_clean":"2024-09-21T10:45:24.440871+0000","last_became_active":"2024-09-21T10:07:56.018261+0000","last_became_peered":"2024-09-21T10:07:56.018261+0000","last_unstale":"2024-09-21T10:45:24.440871+0000","last_undegraded":"2024-09-21T10:45:24.440871+0000","last_fullsized":"2024-09-21T10:45:24.440871+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:04:54.395456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1354,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.401511+0000","last_change":"2024-09-21T10:09:06.531935+0000","last_active":"2024-09-21T10:45:14.401511+0000","last_peered":"2024-09-21T10:45:14.401511+0000","last_clean":"2024-09-21T10:45:14.401511+0000","last_became_active":"2024-09-21T10:09:06.531610+0000","last_became_peered":"2024-09-21T10:09:06.531610+0000","last_unstale":"2024-09-21T10:45:14.401511+0000","last_undegraded":"2024-09-21T10:45:14.401511+0000","last_fullsized":"2024-09-21T10:45:14.401511+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:33.874405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1391,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.402741+0000","last_change":"2024-09-21T10:44:14.401913+0000","last_active":"2024-09-21T10:45:14.402741+0000","last_peered":"2024-09-21T10:45:14.402741+0000","last_clean":"2024-09-21T10:45:14.402741+0000","last_became_active":"2024-09-21T10:44:14.401496+0000","last_became_peered":"2024-09-21T10:44:14.401496+0000","last_unstale":"2024-09-21T10:45:14.402741+0000","last_undegraded":"2024-09-21T10:45:14.402741+0000","last_fullsized":"2024-09-21T10:45:14.402741+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:07:38.531723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":28,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.398081+0000","last_change":"2024-09-21T10:44:14.410444+0000","last_active":"2024-09-21T10:45:14.398081+0000","last_peered":"2024-09-21T10:45:14.398081+0000","last_clean":"2024-09-21T10:45:14.398081+0000","last_became_active":"2024-09-21T10:44:14.395465+0000","last_became_peered":"2024-09-21T10:44:14.395465+0000","last_unstale":"2024-09-21T10:45:14.398081+0000","last_undegraded":"2024-09-21T10:45:14.398081+0000","last_fullsized":"2024-09-21T10:45:14.398081+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:05:44.575878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1369,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.406148+0000","last_change":"2024-09-21T10:09:06.559132+0000","last_active":"2024-09-21T10:45:14.406148+0000","last_peered":"2024-09-21T10:45:14.406148+0000","last_clean":"2024-09-21T10:45:14.406148+0000","last_became_active":"2024-09-21T10:09:06.558395+0000","last_became_peered":"2024-09-21T10:09:06.558395+0000","last_unstale":"2024-09-21T10:45:14.406148+0000","last_undegraded":"2024-09-21T10:45:14.406148+0000","last_fullsized":"2024-09-21T10:45:14.406148+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:27:00.073554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":28,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.398509+0000","last_change":"2024-09-21T10:44:14.419937+0000","last_active":"2024-09-21T10:45:14.398509+0000","last_peered":"2024-09-21T10:45:14.398509+0000","last_clean":"2024-09-21T10:45:14.398509+0000","last_became_active":"2024-09-21T10:44:14.402761+0000","last_became_peered":"2024-09-21T10:44:14.402761+0000","last_unstale":"2024-09-21T10:45:14.398509+0000","last_undegraded":"2024-09-21T10:45:14.398509+0000","last_fullsized":"2024-09-21T10:45:14.398509+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:42.875310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1357,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.400578+0000","last_change":"2024-09-21T10:44:14.402411+0000","last_active":"2024-09-21T10:45:14.400578+0000","last_peered":"2024-09-21T10:45:14.400578+0000","last_clean":"2024-09-21T10:45:14.400578+0000","last_became_active":"2024-09-21T10:44:14.402045+0000","last_became_peered":"2024-09-21T10:44:14.402045+0000","last_unstale":"2024-09-21T10:45:14.400578+0000","last_undegraded":"2024-09-21T10:45:14.400578+0000","last_fullsized":"2024-09-21T10:45:14.400578+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:57:10.535776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":28,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.399679+0000","last_change":"2024-09-21T10:44:14.421612+0000","last_active":"2024-09-21T10:45:14.399679+0000","last_peered":"2024-09-21T10:45:14.399679+0000","last_clean":"2024-09-21T10:45:14.399679+0000","last_became_active":"2024-09-21T10:44:14.406095+0000","last_became_peered":"2024-09-21T10:44:14.406095+0000","last_unstale":"2024-09-21T10:45:14.399679+0000","last_undegraded":"2024-09-21T10:45:14.399679+0000","last_fullsized":"2024-09-21T10:45:14.399679+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:21:19.918156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1362,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.402231+0000","last_change":"2024-09-21T10:07:21.497877+0000","last_active":"2024-09-21T10:45:14.402231+0000","last_peered":"2024-09-21T10:45:14.402231+0000","last_clean":"2024-09-21T10:45:14.402231+0000","last_became_active":"2024-09-21T10:07:21.496317+0000","last_became_peered":"2024-09-21T10:07:21.496317+0000","last_unstale":"2024-09-21T10:45:14.402231+0000","last_undegraded":"2024-09-21T10:45:14.402231+0000","last_fullsized":"2024-09-21T10:45:14.402231+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:39:03.848183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2054,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.438646+0000","last_change":"2024-09-21T10:44:14.398487+0000","last_active":"2024-09-21T10:45:24.438646+0000","last_peered":"2024-09-21T10:45:24.438646+0000","last_clean":"2024-09-21T10:45:24.438646+0000","last_became_active":"2024-09-21T10:44:14.398142+0000","last_became_peered":"2024-09-21T10:44:14.398142+0000","last_unstale":"2024-09-21T10:45:24.438646+0000","last_undegraded":"2024-09-21T10:45:24.438646+0000","last_fullsized":"2024-09-21T10:45:24.438646+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:20:41.065055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.396690+0000","last_change":"2024-09-21T10:44:14.398878+0000","last_active":"2024-09-21T10:45:14.396690+0000","last_peered":"2024-09-21T10:45:14.396690+0000","last_clean":"2024-09-21T10:45:14.396690+0000","last_became_active":"2024-09-21T10:44:14.398547+0000","last_became_peered":"2024-09-21T10:44:14.398547+0000","last_unstale":"2024-09-21T10:45:14.396690+0000","last_undegraded":"2024-09-21T10:45:14.396690+0000","last_fullsized":"2024-09-21T10:45:14.396690+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:13.008134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2084,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.441473+0000","last_change":"2024-09-21T10:44:14.409051+0000","last_active":"2024-09-21T10:45:24.441473+0000","last_peered":"2024-09-21T10:45:24.441473+0000","last_clean":"2024-09-21T10:45:24.441473+0000","last_became_active":"2024-09-21T10:44:14.407892+0000","last_became_peered":"2024-09-21T10:44:14.407892+0000","last_unstale":"2024-09-21T10:45:24.441473+0000","last_undegraded":"2024-09-21T10:45:24.441473+0000","last_fullsized":"2024-09-21T10:45:24.441473+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:19.549648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":2068,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.447615+0000","last_change":"2024-09-21T10:44:14.408837+0000","last_active":"2024-09-21T10:45:24.447615+0000","last_peered":"2024-09-21T10:45:24.447615+0000","last_clean":"2024-09-21T10:45:24.447615+0000","last_became_active":"2024-09-21T10:44:14.408399+0000","last_became_peered":"2024-09-21T10:44:14.408399+0000","last_unstale":"2024-09-21T10:45:24.447615+0000","last_undegraded":"2024-09-21T10:45:24.447615+0000","last_fullsized":"2024-09-21T10:45:24.447615+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:41:38.729386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1369,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.400967+0000","last_change":"2024-09-21T10:44:14.401435+0000","last_active":"2024-09-21T10:45:14.400967+0000","last_peered":"2024-09-21T10:45:14.400967+0000","last_clean":"2024-09-21T10:45:14.400967+0000","last_became_active":"2024-09-21T10:44:14.400908+0000","last_became_peered":"2024-09-21T10:44:14.400908+0000","last_unstale":"2024-09-21T10:45:14.400967+0000","last_undegraded":"2024-09-21T10:45:14.400967+0000","last_fullsized":"2024-09-21T10:45:14.400967+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:46.748289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":30,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:45:14.397104+0000","last_change":"2024-09-21T10:44:14.418113+0000","last_active":"2024-09-21T10:45:14.397104+0000","last_peered":"2024-09-21T10:45:14.397104+0000","last_clean":"2024-09-21T10:45:14.397104+0000","last_became_active":"2024-09-21T10:44:14.398327+0000","last_became_peered":"2024-09-21T10:44:14.398327+0000","last_unstale":"2024-09-21T10:45:14.397104+0000","last_undegraded":"2024-09-21T10:45:14.397104+0000","last_fullsized":"2024-09-21T10:45:14.397104+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:08.810863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2054,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:45:24.437934+0000","last_change":"2024-09-21T10:44:14.398584+0000","last_active":"2024-09-21T10:45:24.437934+0000","last_peered":"2024-09-21T10:45:24.437934+0000","last_clean":"2024-09-21T10:45:24.437934+0000","last_became_active":"2024-09-21T10:44:14.398278+0000","last_became_peered":"2024-09-21T10:44:14.398278+0000","last_unstale":"2024-09-21T10:45:24.437934+0000","last_undegraded":"2024-09-21T10:45:24.437934+0000","last_fullsized":"2024-09-21T10:45:24.437934+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:19.681750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":54,"num_pg":16,"stat_sum":{"num_bytes":5066326016,"num_objects":77306,"num_object_clones":0,"num_object_copies":231918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77306,"num_write_kb":4947584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15159066624,"data_stored":15159066624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77306,"ondisk_log_size":77306,"up":48,"acting":48,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":10310,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234617,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2315564,"kb_used_data":2131940,"kb_used_omap":1,"kb_used_meta":183614,"kb_avail":91429588,"statfs":{"total":95995035648,"available":93623898112,"internally_reserved":0,"allocated":2183106560,"data_stored":2178759068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":188021195},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496254,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3101356,"kb_used_data":2727396,"kb_used_omap":1,"kb_used_meta":373950,"kb_avail":90643796,"statfs":{"total":95995035648,"available":92819247104,"internally_reserved":0,"allocated":2792853504,"data_stored":2788506501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":382925258},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757891,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2440600,"kb_used_data":2102672,"kb_used_omap":1,"kb_used_meta":337918,"kb_avail":91304552,"statfs":{"total":95995035648,"available":93495861248,"internally_reserved":0,"allocated":2153136128,"data_stored":2148804271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":346028486},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019527,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1642216,"kb_used_data":1205916,"kb_used_omap":1,"kb_used_meta":436286,"kb_avail":92102936,"statfs":{"total":95995035648,"available":94313406464,"internally_reserved":0,"allocated":1234857984,"data_stored":1230518070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":446757318},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369271,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3839824,"kb_used_data":3482824,"kb_used_omap":69,"kb_used_meta":356922,"kb_avail":89905328,"statfs":{"total":95995035648,"available":92063055872,"internally_reserved":0,"allocated":3566411776,"data_stored":3562087542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":365488258},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2920577761726,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807957957,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1655504,"kb_used_data":1258376,"kb_used_omap":66,"kb_used_meta":397053,"kb_avail":92089648,"statfs":{"total":95995035648,"available":94299799552,"internally_reserved":0,"allocated":1288577024,"data_stored":1284252719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68589,"internal_metadata":406582291},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":666,"seq":2860448219596,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2249688,"kb_used_data":1911056,"kb_used_omap":76,"kb_used_meta":338547,"kb_avail":91495464,"statfs":{"total":95995035648,"available":93691355136,"internally_reserved":0,"allocated":1956921344,"data_stored":1952589976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78716,"internal_metadata":346672260},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1958346752,"data_stored":1958346752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1292500992,"data_stored":1292500992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3574464512,"data_stored":3574464512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1226047488,"data_stored":1226047488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2144272384,"data_stored":2144272384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2786918400,"data_stored":2786918400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2176516096,"data_stored":2176516096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:45:32.433 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_1 pgp_num 16 2024-09-21T10:45:32.478 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:32.510 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:32.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:32.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:32.594 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:32 smithi160 ceph-mon[101879]: pgmap v2382: 89 pgs: 89 active+clean; 4.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-21T10:45:32.611 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:32.659 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:32.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:32.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:32.688 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:32 smithi070 ceph-mon[112139]: pgmap v2382: 89 pgs: 89 active+clean; 4.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-21T10:45:32.689 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:32 smithi070 ceph-mon[107385]: pgmap v2382: 89 pgs: 89 active+clean; 4.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-21T10:45:32.761 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:32.804 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:32.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:32.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:32.906 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:32.959 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:32.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:32.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:32.994 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 45 16 83527 83511 115.971 73.375 0.00958335 0.00861776 2024-09-21T10:45:33.060 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:33.092 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:33.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:33.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:33.194 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:33.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:33.235 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:33.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:33.248 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:45:33.248 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:45:33.274 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:45:33.274 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:45:33.290 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:45:33.290 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:45:33.290 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:45:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:45:33] "GET /metrics HTTP/1.1" 200 36600 "" "Prometheus/2.51.0" 2024-09-21T10:45:33.329 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:45:33.330 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:45:33.335 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:33.362 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:45:33.363 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:45:33.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:33.378 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:33.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:33.396 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:45:33.396 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:45:33.428 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:45:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:45:33.461 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:45:33.461 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:45:33.479 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:33.507 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:33.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:33.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:33.609 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:33.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:33 smithi070 ceph-mon[107385]: from='client.35409 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:45:33.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:33 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3442395963' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:33.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:33 smithi070 ceph-mon[112139]: from='client.35409 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:45:33.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:33 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3442395963' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:33.650 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:45:33.660 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:33.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:33.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:33.763 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:33.791 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:33 smithi160 ceph-mon[101879]: from='client.35409 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:45:33.791 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:33 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3442395963' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:33.793 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:33.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:33.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:33.894 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:33.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:33.941 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:33.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:33.994 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 46 16 84640 84624 114.962 69.5625 0.010573 0.00869376 2024-09-21T10:45:34.042 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:34.078 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:34.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:34.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:34.095 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:34.180 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:34.207 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:34.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:34.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:34.308 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:34.334 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:34.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:34.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:34.436 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:34.468 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:34.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:34.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:34.482 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:34 smithi070 ceph-mon[112139]: pgmap v2383: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-21T10:45:34.482 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:34 smithi070 ceph-mon[107385]: pgmap v2383: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-21T10:45:34.570 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:34.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:34 smithi160 ceph-mon[101879]: pgmap v2383: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-21T10:45:34.607 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:34.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:34.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:34.709 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:34.742 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:34.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:34.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:34.844 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:34.884 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:34.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:34.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:34.986 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:34.994 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 47 16 85805 85789 114.065 72.8125 0.00498335 0.00876212 2024-09-21T10:45:35.018 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:35.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:35.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:35.120 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:35.149 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:35.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:35.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:35.250 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:35.280 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:35.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:35.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:35.333 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:35.382 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:35.417 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:35.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:35.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:35.518 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:35.547 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:35.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:35.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:35.591 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:45:35.591 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:45:35.649 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:35.678 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:35.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:35.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:35.780 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:35.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:45:35.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:35.829 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:35.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:35.929 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:35.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:35.984 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:35.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:35.994 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 48 16 86925 86909 113.147 70 0.0190485 0.00883306 2024-09-21T10:45:36.085 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:36.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:36.120 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:36.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:36.221 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:36.256 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:36.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:36.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:36.358 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:36.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:36.408 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:36.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:36.509 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:36.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:36.551 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:36.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:36.559 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:36 smithi070 ceph-mon[107385]: pgmap v2384: 89 pgs: 89 active+clean; 5.1 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-21T10:45:36.559 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:45:36.559 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:45:36.559 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:36 smithi070 ceph-mon[112139]: pgmap v2384: 89 pgs: 89 active+clean; 5.1 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-21T10:45:36.559 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:45:36.559 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:45:36.652 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:36.695 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:36.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:36.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:36.748 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:36 smithi160 ceph-mon[101879]: pgmap v2384: 89 pgs: 89 active+clean; 5.1 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-21T10:45:36.748 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:45:36.748 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:45:36.797 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:36.833 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:36.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:36.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:36.935 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:36.973 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:36.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:36.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:36.994 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 49 16 88003 87987 112.213 67.375 0.00601758 0.00890576 2024-09-21T10:45:37.075 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:37.117 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:37.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:37.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:37.218 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:37.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:37.261 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:37.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:37.362 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:37.366 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:45:37.397 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:37.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:37.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:37.499 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:37.548 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:37.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:37.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:37.650 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:37.687 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:37.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:37.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:37.751 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:45:37.751 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T10:45:37.751 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.751 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:87930 54.1 54:8d270642:::benchmark_data_smithi070_261699_object87929:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1383)", 2024-09-21T10:45:37.751 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:36.870887+0000", 2024-09-21T10:45:37.751 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.025703235000000001, 2024-09-21T10:45:37.751 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.025730706999999998, 2024-09-21T10:45:37.752 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:45:37.752 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:45:37.752 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:45:37.752 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:45:37.752 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:45:37.752 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 87930 2024-09-21T10:45:37.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.752 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:45:37.752 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.752 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:45:37.752 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.870887+0000", 2024-09-21T10:45:37.752 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:45:37.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.753 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.753 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:45:37.753 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.870887+0000", 2024-09-21T10:45:37.753 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:45:37.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.753 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.753 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:45:37.753 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.870886+0000", 2024-09-21T10:45:37.753 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:45:37.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.753 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.753 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:45:37.753 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.870928+0000", 2024-09-21T10:45:37.753 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2651999999999998e-05 2024-09-21T10:45:37.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.754 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.754 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:45:37.754 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.870945+0000", 2024-09-21T10:45:37.754 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6677e-05 2024-09-21T10:45:37.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.754 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.754 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:45:37.754 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.870965+0000", 2024-09-21T10:45:37.754 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9738999999999999e-05 2024-09-21T10:45:37.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.754 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.754 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:45:37.754 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.871013+0000", 2024-09-21T10:45:37.754 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8401e-05 2024-09-21T10:45:37.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.755 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.755 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:45:37.755 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.871178+0000", 2024-09-21T10:45:37.755 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016527100000000001 2024-09-21T10:45:37.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.755 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.755 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:45:37.755 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.871335+0000", 2024-09-21T10:45:37.755 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000156716 2024-09-21T10:45:37.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.755 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.755 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:45:37.755 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.878333+0000", 2024-09-21T10:45:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0069976439999999999 2024-09-21T10:45:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.881534+0000", 2024-09-21T10:45:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0032010889999999998 2024-09-21T10:45:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:45:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:45:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:45:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:87932 54.1 54:8cdeca0e:::benchmark_data_smithi070_261699_object87931:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1383)", 2024-09-21T10:45:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:36.870981+0000", 2024-09-21T10:45:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.025609852999999998, 2024-09-21T10:45:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.025691537, 2024-09-21T10:45:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:45:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:45:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:45:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:45:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:45:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 87932 2024-09-21T10:45:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:45:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:45:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.870981+0000", 2024-09-21T10:45:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:45:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:45:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.870981+0000", 2024-09-21T10:45:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:45:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:45:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.870980+0000", 2024-09-21T10:45:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T10:45:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:45:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.871043+0000", 2024-09-21T10:45:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2948999999999996e-05 2024-09-21T10:45:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:45:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.871051+0000", 2024-09-21T10:45:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7989999999999994e-06 2024-09-21T10:45:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:45:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.871056+0000", 2024-09-21T10:45:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4489999999999998e-06 2024-09-21T10:45:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:45:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.874619+0000", 2024-09-21T10:45:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0035631930000000001 2024-09-21T10:45:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:45:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.874771+0000", 2024-09-21T10:45:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015127900000000001 2024-09-21T10:45:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:45:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.874889+0000", 2024-09-21T10:45:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000118552 2024-09-21T10:45:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:45:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:45:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:45:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:87933 54.1 54:8a6b7fbd:::benchmark_data_smithi070_261699_object87932:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1383)", 2024-09-21T10:45:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:36.871062+0000", 2024-09-21T10:45:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.025528254, 2024-09-21T10:45:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.025656940999999999, 2024-09-21T10:45:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:45:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:45:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:45:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:45:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:45:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 87933 2024-09-21T10:45:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:45:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:45:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.871062+0000", 2024-09-21T10:45:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:45:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:45:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.871062+0000", 2024-09-21T10:45:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:45:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:45:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.871062+0000", 2024-09-21T10:45:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T10:45:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:45:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.871116+0000", 2024-09-21T10:45:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.41e-05 2024-09-21T10:45:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:45:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.871123+0000", 2024-09-21T10:45:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0659999999999998e-06 2024-09-21T10:45:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:45:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.871128+0000", 2024-09-21T10:45:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9119999999999997e-06 2024-09-21T10:45:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:45:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.878385+0000", 2024-09-21T10:45:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.007257538 2024-09-21T10:45:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:45:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.878525+0000", 2024-09-21T10:45:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013955799999999999 2024-09-21T10:45:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:45:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.878605+0000", 2024-09-21T10:45:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9789000000000005e-05 2024-09-21T10:45:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:45:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:45:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:45:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:87938 54.1 54:8448d7d7:::benchmark_data_smithi070_261699_object87937:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1383)", 2024-09-21T10:45:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:36.878233+0000", 2024-09-21T10:45:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.018357099000000002, 2024-09-21T10:45:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.018526579000000001, 2024-09-21T10:45:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:45:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:45:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:45:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:45:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:45:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 87938 2024-09-21T10:45:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:45:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:45:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.878233+0000", 2024-09-21T10:45:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:45:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:45:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.878233+0000", 2024-09-21T10:45:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:45:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:45:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.878231+0000", 2024-09-21T10:45:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:45:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:45:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.878306+0000", 2024-09-21T10:45:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4877999999999997e-05 2024-09-21T10:45:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:45:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.878320+0000", 2024-09-21T10:45:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3558e-05 2024-09-21T10:45:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:45:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.878328+0000", 2024-09-21T10:45:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1880000000000002e-06 2024-09-21T10:45:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:45:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.881574+0000", 2024-09-21T10:45:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0032455320000000002 2024-09-21T10:45:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:45:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.881718+0000", 2024-09-21T10:45:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014446500000000001 2024-09-21T10:45:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:45:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.881916+0000", 2024-09-21T10:45:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0001975 2024-09-21T10:45:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:45:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:45:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:45:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.45308.0:87932 54.1 e1383/1376)", 2024-09-21T10:45:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:36.881573+0000", 2024-09-21T10:45:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.015017506, 2024-09-21T10:45:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.015222394, 2024-09-21T10:45:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:45:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T10:45:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:45:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:45:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.881573+0000", 2024-09-21T10:45:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:45:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:45:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.881573+0000", 2024-09-21T10:45:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:45:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:45:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.881572+0000", 2024-09-21T10:45:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:45:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:45:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.881578+0000", 2024-09-21T10:45:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8900000000000004e-06 2024-09-21T10:45:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:45:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.881586+0000", 2024-09-21T10:45:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5099999999999998e-06 2024-09-21T10:45:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:45:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.881592+0000", 2024-09-21T10:45:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7590000000000003e-06 2024-09-21T10:45:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:45:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:45:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:45:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.45308.0:87933 54.1 e1383/1376)", 2024-09-21T10:45:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:36.881830+0000", 2024-09-21T10:45:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.014760242999999999, 2024-09-21T10:45:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.014987978000000001, 2024-09-21T10:45:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:45:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T10:45:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:45:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:45:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.881830+0000", 2024-09-21T10:45:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:45:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:45:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.881830+0000", 2024-09-21T10:45:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:45:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:45:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.881829+0000", 2024-09-21T10:45:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T10:45:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:45:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.881834+0000", 2024-09-21T10:45:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9509999999999997e-06 2024-09-21T10:45:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:45:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.881842+0000", 2024-09-21T10:45:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1160000000000001e-06 2024-09-21T10:45:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:45:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:45:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:45:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.881847+0000", 2024-09-21T10:45:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7180000000000001e-06 2024-09-21T10:45:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:45:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:45:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:45:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:45:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T10:45:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 6 2024-09-21T10:45:37.774 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:45:37.789 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:37.828 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:37.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:37.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:37.930 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:37.968 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:37.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:37.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:37.994 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 50 16 89133 89117 111.381 70.625 0.0247249 0.00897338 2024-09-21T10:45:38.070 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:38.115 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:38.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:38.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:38.217 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:38.274 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:38.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:38.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:38.376 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:38.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:38.409 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:38.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:38.495 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T10:45:38.511 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:38.551 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:38.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:38.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:38.653 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:38.687 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:38.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:38.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:38.747 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:45:38.782 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:38 smithi160 ceph-mon[101879]: pgmap v2385: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-21T10:45:38.782 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:38 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1031564174' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:45:38.782 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:38 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:45:38.782 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:38 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:45:38.782 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:38 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:45:38.782 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:38 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:45:38.782 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:38 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:45:38.789 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:38.803 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:38 smithi070 ceph-mon[107385]: pgmap v2385: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-21T10:45:38.803 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:38 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1031564174' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:45:38.803 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:38 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:45:38.803 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:38 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:45:38.803 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:38 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:45:38.803 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:38 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:45:38.803 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:38 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:45:38.803 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:38 smithi070 ceph-mon[112139]: pgmap v2385: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-21T10:45:38.803 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:38 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1031564174' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:45:38.803 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:38 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:45:38.803 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:38 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:45:38.804 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:38 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:45:38.804 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:38 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:45:38.804 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:38 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:45:38.832 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:38.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:38.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:38.933 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:38.977 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:38.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:38.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:38.995 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 51 16 90195 90179 110.499 66.375 0.00404178 0.00903923 2024-09-21T10:45:39.045 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:45:39.078 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:39.111 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:39.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:39.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:39.212 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:39.243 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:39.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:39.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:39.345 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:39.381 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:39.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:39.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:39.483 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:39.523 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:39.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:39.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:39.571 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:39.625 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:39.656 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:39.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:39.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:39.757 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:39.793 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:39.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:39.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:39 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:45:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:39 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1031564174' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:45:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:39 smithi160 ceph-mon[101879]: osdmap e1384: 8 total, 8 up, 7 in 2024-09-21T10:45:39.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:39 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:45:39.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:39 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1031564174' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:45:39.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:39 smithi070 ceph-mon[107385]: osdmap e1384: 8 total, 8 up, 7 in 2024-09-21T10:45:39.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:39 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:45:39.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:39 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1031564174' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:45:39.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:39 smithi070 ceph-mon[112139]: osdmap e1384: 8 total, 8 up, 7 in 2024-09-21T10:45:39.895 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:39.923 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:39.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:39.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:39.994 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 52 16 91297 91281 109.698 68.875 0.00840215 0.00911085 2024-09-21T10:45:40.025 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:40.055 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:40.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:40.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:40.157 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:40.197 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:40.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:40.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:40.299 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:40.328 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:40.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:40.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:40.430 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:40.469 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:40.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:40.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:40.571 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:40.613 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:40.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:40.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:40.685 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:40 smithi070 ceph-mon[107385]: pgmap v2387: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-21T10:45:40.685 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:40 smithi070 ceph-mon[112139]: pgmap v2387: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-21T10:45:40.714 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:40.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:40 smithi160 ceph-mon[101879]: pgmap v2387: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-21T10:45:40.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:40.753 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:40.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:40.853 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:40.892 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:40.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:40.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:40.994 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:40.995 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 53 16 92475 92459 109.017 73.625 0.0106891 0.00916785 2024-09-21T10:45:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:41.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:41.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:41.129 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:41.172 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:41.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:41.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:41.273 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:41.310 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:41.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:41.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:41.412 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:41.443 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:41.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:41.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:41.473 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:45:41.473 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T10:45:41.473 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T10:45:41.473 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T10:45:41.473 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:45:41.545 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:41.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:41.586 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:41.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:41.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:41 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/184747707' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T10:45:41.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:41 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T10:45:41.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:41 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/184747707' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T10:45:41.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:41 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T10:45:41.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:41 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/184747707' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T10:45:41.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:41 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T10:45:41.688 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:41.726 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:41.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:41.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:41.827 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:41.858 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:41.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:41.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:41.960 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:41.995 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 54 16 93625 93609 108.33 71.875 0.0220573 0.00922567 2024-09-21T10:45:41.997 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:41.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:41.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:42.098 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:42.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:42.131 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:42.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:42.148 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:45:42.210 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:45:42.211 INFO:teuthology.orchestra.run.smithi070.stderr:set pool 54 pgp_num to 16 2024-09-21T10:45:42.232 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:42.300 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:42.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:42.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:42.402 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:42.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:42.445 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:42.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:42.546 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:42.581 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:42.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:42.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:42.585 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:45:42.684 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:42.713 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:42.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:42.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:42.807 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:42 smithi070 ceph-mon[107385]: pgmap v2388: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-21T10:45:42.807 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:42 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-09-21T10:45:42.807 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:42 smithi070 ceph-mon[107385]: osdmap e1385: 8 total, 8 up, 7 in 2024-09-21T10:45:42.808 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:42 smithi070 ceph-mon[112139]: pgmap v2388: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-21T10:45:42.808 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:42 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-09-21T10:45:42.808 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:42 smithi070 ceph-mon[112139]: osdmap e1385: 8 total, 8 up, 7 in 2024-09-21T10:45:42.814 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:42 smithi160 ceph-mon[101879]: pgmap v2388: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-21T10:45:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:42 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-09-21T10:45:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:42 smithi160 ceph-mon[101879]: osdmap e1385: 8 total, 8 up, 7 in 2024-09-21T10:45:42.862 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:42.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:42.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:42.968 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:42.995 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 55 16 94727 94711 107.612 68.875 0.0158794 0.00928756 2024-09-21T10:45:43.001 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:43.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:43.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:43.103 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:43.135 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:43.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:43.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:43.237 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:43.276 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:43.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:43.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:43.351 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:45:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:45:43] "GET /metrics HTTP/1.1" 200 36601 "" "Prometheus/2.51.0" 2024-09-21T10:45:43.378 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:43.403 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:45:43.403 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:45:43.408 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:43.410 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:43.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:43.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:43.434 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:45:43.434 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:45:43.465 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:45:43.465 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:45:43.498 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:45:43.498 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:45:43.512 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:43.527 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:45:43.527 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:45:43.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:43.551 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:43.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:43.562 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:45:43.563 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:45:43.599 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:45:43.600 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:45:43.625 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:45:43.625 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:45:43.626 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:43 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2818248753' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:43.626 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:43 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2818248753' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:43.652 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:43.686 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:43.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:43.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:43.787 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:43.818 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:43.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:43.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:43 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2818248753' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:43.919 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:43.945 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:43.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:43.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:43.995 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 56 16 95798 95782 106.886 66.9375 0.0102164 0.00935062 2024-09-21T10:45:44.047 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:44.082 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:44.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:44.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:44.184 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:44.215 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:44.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:44.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:44.317 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:44.348 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:44.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:44.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:44.450 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:44.477 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:44.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:44.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:44.578 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:44.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:44.611 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:44.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:44.614 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:44 smithi160 ceph-mon[101879]: pgmap v2390: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-21T10:45:44.713 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:44.739 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:44.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:44.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:44.841 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:44.872 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:44.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:44.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:44.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:44 smithi070 ceph-mon[107385]: pgmap v2390: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-21T10:45:44.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:44 smithi070 ceph-mon[112139]: pgmap v2390: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-21T10:45:44.975 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:44.995 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 57 16 96919 96903 106.24 70.0625 0.0135781 0.00940785 2024-09-21T10:45:45.002 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:45.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:45.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:45.104 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:45.104 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:45:45.105 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:45:45.132 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:45.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:45.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:45.234 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:45.264 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:45.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:45.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:45.366 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:45.399 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:45.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:45.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:45.501 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:45.537 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:45.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:45.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:45.639 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:45.667 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:45.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:45.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:45.769 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:45.798 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:45.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:45.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:45.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:45 smithi160 ceph-mon[101879]: pgmap v2391: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-21T10:45:45.857 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:45 smithi070 ceph-mon[107385]: pgmap v2391: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-21T10:45:45.857 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:45 smithi070 ceph-mon[112139]: pgmap v2391: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-21T10:45:45.900 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:45.935 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:45.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:45.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:45.995 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 58 16 98064 98048 105.642 71.5625 0.0149817 0.00946125 2024-09-21T10:45:46.036 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:46.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:46.068 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:46.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:46.129 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:45:46.169 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:46.198 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:46.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:46.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:46.299 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:46.341 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:46.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:46.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:46.443 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:46.485 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:46.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:46.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:46.508 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:45:46.587 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:46.630 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:46.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:46.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:46.733 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:46.768 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:46.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:46.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:46.811 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T10:45:46.869 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:46.896 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:46.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:46.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:46.969 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:46 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/954470400' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:45:46.970 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:46 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:45:46.995 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 59 16 99110 99094 104.959 65.375 0.0142952 0.00952253 2024-09-21T10:45:46.998 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:47.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:47.029 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:47.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:47.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:46 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/954470400' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:45:47.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:46 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:45:47.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:46 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/954470400' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:45:47.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:46 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:45:47.131 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:47.161 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:47.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:47.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:47.263 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:47.298 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:47.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:47.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:47.399 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:47.442 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:47.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:47.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:47.543 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:47.579 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:47.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:47.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:47.680 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:47.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:47.725 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:47.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:47.825 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:47.843 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:45:47.843 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:45:47.844 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-21T10:45:47.844 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:45:47.844 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:45:47.844 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-21T10:45:47.858 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:45:47.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:47.870 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:47.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:47.972 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:47.995 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:2024-09-21T10:45:47.994556+0000 min lat: 0.00136946 max lat: 0.178279 avg lat: 0.009579 2024-09-21T10:45:47.995 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:45:47.995 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 60 16 100193 100177 104.338 67.6875 0.0154792 0.009579 2024-09-21T10:45:48.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:48.026 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:48.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:48.043 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:47 smithi160 ceph-mon[101879]: pgmap v2392: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-21T10:45:48.043 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:47 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:45:48.043 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:47 smithi160 ceph-mon[101879]: Cluster is now healthy 2024-09-21T10:45:48.044 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:47 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:45:48.044 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:47 smithi160 ceph-mon[101879]: osdmap e1386: 8 total, 8 up, 7 in 2024-09-21T10:45:48.126 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:48.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:47 smithi070 ceph-mon[107385]: pgmap v2392: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-21T10:45:48.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:47 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:45:48.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:47 smithi070 ceph-mon[107385]: Cluster is now healthy 2024-09-21T10:45:48.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:47 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:45:48.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:47 smithi070 ceph-mon[107385]: osdmap e1386: 8 total, 8 up, 7 in 2024-09-21T10:45:48.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:47 smithi070 ceph-mon[112139]: pgmap v2392: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-21T10:45:48.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:47 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:45:48.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:47 smithi070 ceph-mon[112139]: Cluster is now healthy 2024-09-21T10:45:48.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:47 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:45:48.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:47 smithi070 ceph-mon[112139]: osdmap e1386: 8 total, 8 up, 7 in 2024-09-21T10:45:48.150 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:48.157 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:48.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:48.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:48.259 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:48.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:48.291 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:48.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:48.392 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:48.425 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:48.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:48.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:48.527 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:48.561 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:48.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:48.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:48.663 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:48.697 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:48.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:48.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:48.798 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:48.827 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:48.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:48.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:48.929 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:48.956 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:48.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:48.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:48.995 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 61 16 101237 101221 103.697 65.25 0.0214334 0.00963849 2024-09-21T10:45:49.058 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:49.092 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:49.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:49.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:49.194 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:49.235 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:45:49.235 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:45:49.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:49.250 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:49.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:49.351 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:49.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:49.391 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:49.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:49.470 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:45:49.492 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:49.540 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:49.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:49.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:49.641 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:49.667 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:49.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:49.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:49.768 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:49.810 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:49.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:49.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:49.820 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:45:49.911 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:49.934 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:49.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:49.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:49.995 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 62 16 102247 102231 103.043 63.125 0.019111 0.00970018 2024-09-21T10:45:50.036 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:50.078 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:50.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:50.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:50.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:49 smithi160 ceph-mon[101879]: pgmap v2394: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-21T10:45:50.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:49 smithi070 ceph-mon[107385]: pgmap v2394: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-21T10:45:50.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:49 smithi070 ceph-mon[112139]: pgmap v2394: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-21T10:45:50.180 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:50.215 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:50.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:50.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:50.317 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:50.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:50.352 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:50.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:50.415 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:45:50.453 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:50.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:50.499 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:50.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:50.600 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:50.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:50.647 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:50.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:50.748 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:50.780 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:50.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:50.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:50.882 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:50.921 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:50.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:50.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:50.996 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 63 16 103257 103241 102.409 63.125 0.0402769 0.00975709 2024-09-21T10:45:51.023 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:51.059 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:51.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:51.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:51.161 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:51.204 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:51.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:51.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:51.306 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:51.343 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:51.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:51.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:51.444 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:51.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:51.480 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:51.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:51.508 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:45:51.508 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:45:51.581 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:51.615 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:51.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:51.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:51.716 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:51.742 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:51.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:51.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:51.844 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:51.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:45:51.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:45:51.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:45:51.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:45:51.882 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:51.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:51.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:51.983 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:51.996 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 64 16 104298 104282 101.825 65.0625 0.0199409 0.00981566 2024-09-21T10:45:52.022 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:52.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:52.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:52.123 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:52.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:52.165 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:52.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:52.231 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:45:52.267 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:52.288 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:52.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:52.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:52.389 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:52.438 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:52.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:52.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:52.452 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:45:52.452 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:45:52.452 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:45:52.453 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:45:52.453 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:45:52.453 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:45:52.453 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:45:52.453 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:45:52.453 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:45:52.453 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:45:52.453 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:45:52.453 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:45:52.453 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:45:52.453 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:45:52.469 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:45:52.469 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:45:52.511 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:52 smithi160 ceph-mon[101879]: pgmap v2395: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 56 MiB/s wr, 897 op/s 2024-09-21T10:45:52.511 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:52 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3522962228' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:52.540 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:52.568 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:52.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:52.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:52.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:52 smithi070 ceph-mon[112139]: pgmap v2395: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 56 MiB/s wr, 897 op/s 2024-09-21T10:45:52.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:52 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3522962228' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:52.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:52 smithi070 ceph-mon[107385]: pgmap v2395: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 56 MiB/s wr, 897 op/s 2024-09-21T10:45:52.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:52 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3522962228' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:52.670 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:52.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:52.707 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:52.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:52.808 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:52.839 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:52.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:52.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:52.940 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:52.978 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:52.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:52.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:52.996 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 65 16 105278 105262 101.201 61.25 0.00502937 0.00987673 2024-09-21T10:45:53.080 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:53.115 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:53.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:53.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:53.217 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:53.245 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:53.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:53.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:53.347 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:53.404 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:53.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:53.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:53.505 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:53.512 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:53.526 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:45:53.526 INFO:teuthology.orchestra.run.smithi160.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T10:45:53.526 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:45:53.539 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:53.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:53.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:53.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:45:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:45:53] "GET /metrics HTTP/1.1" 200 36601 "" "Prometheus/2.51.0" 2024-09-21T10:45:53.640 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:53.672 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:53.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:53.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:53.774 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:53.820 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:53.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:53.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:53.922 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:53.964 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:53.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:53.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:53.996 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 66 16 106234 106218 100.573 59.75 0.0160019 0.00993769 2024-09-21T10:45:54.066 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:54.104 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:54.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:54.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:54.206 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:54.246 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:54.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:54.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:54.348 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:54.370 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:54.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:54.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:54.471 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:54.513 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:54.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:54.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:54.614 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:54.645 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:54.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:54.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:54.689 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:54 smithi160 ceph-mon[101879]: pgmap v2396: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-21T10:45:54.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:54 smithi070 ceph-mon[107385]: pgmap v2396: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-21T10:45:54.735 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:54 smithi070 ceph-mon[112139]: pgmap v2396: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-21T10:45:54.746 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:54.796 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:54.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:54.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:54.897 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:54.937 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:54.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:54.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:54.996 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 67 16 107108 107092 99.887 54.625 0.0213037 0.0100062 2024-09-21T10:45:55.038 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:55.082 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:55.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:55.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:55.184 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:55.226 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:55.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:55.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:55.328 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:55.356 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:55.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:55.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:55.373 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:45:55.458 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:55.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:55.489 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:55.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:55.590 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:55.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:55.630 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:55.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:55.731 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:55.767 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:55.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:55.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:55.788 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:55.869 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:55.909 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:55.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:55.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:55.977 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:45:55.977 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T10:45:55.977 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T10:45:55.977 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T10:45:55.977 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:45:55.996 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 68 16 107986 107970 99.225 54.875 0.0127425 0.010074 2024-09-21T10:45:56.011 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:56.043 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:56.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:56.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:56.144 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:56.174 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:56.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:56.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:56.275 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:56.319 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:56.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:56.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:56.400 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T10:45:56.421 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:56.455 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:56.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:56.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:56.556 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:56.587 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:56.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:56.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:56.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:56 smithi160 ceph-mon[101879]: pgmap v2397: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-21T10:45:56.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:56 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3733387038' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:45:56.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:56 smithi070 ceph-mon[107385]: pgmap v2397: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-21T10:45:56.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:56 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3733387038' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:45:56.609 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:56 smithi070 ceph-mon[112139]: pgmap v2397: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-21T10:45:56.609 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:56 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3733387038' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:45:56.657 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:45:56.689 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:56.714 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:56.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:56.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:56.816 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:56.842 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:56.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:56.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:56.943 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:56.970 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:56.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:56.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:56.996 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 69 16 108972 108956 98.68 61.625 0.0147398 0.0101285 2024-09-21T10:45:57.022 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:45:57.072 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:57.120 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:57.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:57.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:57.222 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:57.259 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:57.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:57.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:57.360 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:57.420 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:57.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:57.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:57.483 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:57 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:45:57.483 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:57 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3733387038' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:45:57.483 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:57 smithi160 ceph-mon[101879]: osdmap e1387: 8 total, 8 up, 7 in 2024-09-21T10:45:57.521 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:57.557 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:57.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:57.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:57.659 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:57.684 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:57.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:57.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:57.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:57 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:45:57.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:57 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3733387038' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:45:57.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:57 smithi070 ceph-mon[107385]: osdmap e1387: 8 total, 8 up, 7 in 2024-09-21T10:45:57.754 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:57 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:45:57.754 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:57 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3733387038' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:45:57.754 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:57 smithi070 ceph-mon[112139]: osdmap e1387: 8 total, 8 up, 7 in 2024-09-21T10:45:57.786 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:57.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:57.831 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:57.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:57.932 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:57.981 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:57.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:57.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:57.996 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 70 16 109937 109921 98.1319 60.3125 0.0124982 0.0101857 2024-09-21T10:45:58.083 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:58.124 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:58.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:58.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:58.225 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:58.259 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:58.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:58.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:58.361 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:58.386 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:58.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:58.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:58.488 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:58.520 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:58.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:58.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:58.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:58 smithi160 ceph-mon[101879]: pgmap v2399: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-21T10:45:58.621 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:58.657 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:58.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:58.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:58.735 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:58 smithi070 ceph-mon[107385]: pgmap v2399: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-21T10:45:58.735 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:58 smithi070 ceph-mon[112139]: pgmap v2399: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-21T10:45:58.758 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:58.789 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:58.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:58.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:58.887 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:45:58.888 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:45:58.891 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:58.925 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:45:58.926 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:45:58.926 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:58.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:58.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:58.954 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:45:58.954 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:45:58.987 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:45:58.987 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:45:58.996 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 71 16 110936 110920 97.6291 62.4375 0.013443 0.0102382 2024-09-21T10:45:59.023 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:45:59.023 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:45:59.028 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:59.057 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:45:59.057 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:45:59.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:59.063 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:59.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:59.108 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:45:59.108 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:45:59.150 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:45:59.150 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:45:59.165 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:59.192 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:45:59.193 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:45:59.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:59.205 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:59.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:59.305 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:59.333 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:59.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:59.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:59.435 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:59.480 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:59.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:59.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:59.581 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:59.628 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:59.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:59.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:59.729 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:59.759 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:45:59.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:59.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:59.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:45:59 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1613760515' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:59.862 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:59.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:45:59 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1613760515' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:59.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:45:59 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1613760515' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:59.890 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:45:59.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:59.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:59.992 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:59.996 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 72 16 111950 111934 97.1531 63.375 0.0172651 0.010288 2024-09-21T10:46:00.025 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:00.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:00.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:00.127 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:00.156 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:00.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:00.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:00.257 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:00.261 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:46:00.261 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:46:00.261 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-21T10:46:00.262 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:46:00.262 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:46:00.262 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-21T10:46:00.273 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:46:00.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:00.287 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:00.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:00.303 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:46:00.387 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:00.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:00.442 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:00.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:00.543 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:00.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:00.586 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:00.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:00 smithi160 ceph-mon[101879]: pgmap v2400: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 62 MiB/s wr, 999 op/s 2024-09-21T10:46:00.615 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:00.622 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:00.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:00 smithi070 ceph-mon[112139]: pgmap v2400: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 62 MiB/s wr, 999 op/s 2024-09-21T10:46:00.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:00 smithi070 ceph-mon[107385]: pgmap v2400: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 62 MiB/s wr, 999 op/s 2024-09-21T10:46:00.667 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:46:00.687 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:00.731 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:00.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:00.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:00.833 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:00.863 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:00.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:00.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:00.965 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:00.995 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:00.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:00.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:00.997 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 73 16 112911 112895 96.645 60.0625 0.0275813 0.0103425 2024-09-21T10:46:01.097 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:01.126 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:01.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:01.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:01.227 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:01.256 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:01.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:01.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:01.358 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:01.399 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:01.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:01.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:01.500 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:01.543 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:01.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:01.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:01.644 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:01.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:01.685 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:01.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:01.786 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:01.819 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:01.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:01.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:01.921 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:01.922 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:46:01.954 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:01.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:01.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:01.997 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 74 16 113910 113894 96.1826 62.4375 0.0136976 0.010392 2024-09-21T10:46:02.056 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:02.082 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:02.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:02.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:02.184 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:02.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:02.215 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:02.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:02.316 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:02.353 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:02.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:02.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:02.454 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:02.494 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:02.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:02.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:02.547 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:02 smithi160 ceph-mon[101879]: pgmap v2401: 89 pgs: 89 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 60 MiB/s wr, 960 op/s 2024-09-21T10:46:02.596 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:02.629 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:02.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:02.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:02.679 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:02 smithi070 ceph-mon[107385]: pgmap v2401: 89 pgs: 89 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 60 MiB/s wr, 960 op/s 2024-09-21T10:46:02.679 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:02 smithi070 ceph-mon[112139]: pgmap v2401: 89 pgs: 89 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 60 MiB/s wr, 960 op/s 2024-09-21T10:46:02.707 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:46:02.707 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:46:02.730 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:02.771 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:02.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:02.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:02.872 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:02.900 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:02.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:02.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:02.998 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 75 16 114878 114862 95.7059 60.5 0.0259694 0.0104436 2024-09-21T10:46:03.002 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:03.047 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:03.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:03.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:03.149 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:03.192 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:03.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:03.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:03.247 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:46:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:46:03] "GET /metrics HTTP/1.1" 200 36601 "" "Prometheus/2.51.0" 2024-09-21T10:46:03.294 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:03.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:03.329 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:03.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:03.419 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:46:03.430 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:03.508 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:03.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:03.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:03.609 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:03.638 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:03.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:03.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:03.740 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:03.774 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:03.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:03.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:03.780 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:46:03.876 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:03.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:03.921 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:03.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:03.982 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:03.999 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 76 15 115808 115793 95.2102 58.1875 0.0154604 0.0104976 2024-09-21T10:46:04.022 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:04.025 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:46:04.025 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T10:46:04.025 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:46:04.087 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:04.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:04.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:04.189 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:04.225 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:04.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:04.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:04.327 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:04.350 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:04.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:04.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:04.428 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T10:46:04.452 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:04.493 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:04.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:04.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:04.554 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:04 smithi160 ceph-mon[101879]: pgmap v2402: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 60 MiB/s wr, 963 op/s 2024-09-21T10:46:04.554 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:04 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/337228810' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:04.554 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:04 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:04.595 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:04.631 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:04.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:04.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:04.732 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:04.772 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:04.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:04.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:04.874 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:04.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:04 smithi070 ceph-mon[107385]: pgmap v2402: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 60 MiB/s wr, 963 op/s 2024-09-21T10:46:04.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:04 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/337228810' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:04.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:04 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:04.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:04 smithi070 ceph-mon[112139]: pgmap v2402: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 60 MiB/s wr, 963 op/s 2024-09-21T10:46:04.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:04 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/337228810' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:04.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:04 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:04.907 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:04.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:04.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:04.999 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 77 16 116792 116776 94.7714 61.4375 0.00759428 0.0105469 2024-09-21T10:46:05.009 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:05.046 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:05.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:05.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:05.148 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:05.178 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:05.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:05.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:05.280 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:05.329 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:05.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:05.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:05.431 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:05.474 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:05.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:05.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:05.575 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:05.610 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:05.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:05.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:05.712 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:05.753 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:05.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:05.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:05.768 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:05 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:46:05.768 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:05 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:46:05.768 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:05 smithi160 ceph-mon[101879]: osdmap e1388: 8 total, 8 up, 7 in 2024-09-21T10:46:05.768 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:46:05.768 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:46:05.814 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:05 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:46:05.814 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:05 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:46:05.814 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:05 smithi070 ceph-mon[107385]: osdmap e1388: 8 total, 8 up, 7 in 2024-09-21T10:46:05.814 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:46:05.814 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:46:05.814 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:05 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:46:05.814 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:05 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:46:05.814 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:05 smithi070 ceph-mon[112139]: osdmap e1388: 8 total, 8 up, 7 in 2024-09-21T10:46:05.814 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:46:05.814 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:46:05.821 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:46:05.821 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:46:05.855 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:05.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:05.887 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:05.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:05.988 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:05.999 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 78 16 117726 117710 94.3045 58.375 0.022477 0.0105993 2024-09-21T10:46:06.023 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:06.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:06.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:06.126 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:06.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:06.163 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:06.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:06.264 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:06.302 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:06.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:06.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:06.404 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:06.438 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:06.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:06.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:06.507 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:06 smithi160 ceph-mon[101879]: pgmap v2404: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 52 MiB/s wr, 833 op/s 2024-09-21T10:46:06.540 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:06.544 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:46:06.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:06.568 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:06.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:06 smithi070 ceph-mon[107385]: pgmap v2404: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 52 MiB/s wr, 833 op/s 2024-09-21T10:46:06.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:06 smithi070 ceph-mon[112139]: pgmap v2404: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 52 MiB/s wr, 833 op/s 2024-09-21T10:46:06.669 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:06.715 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:06.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:06.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:06.817 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:06.846 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:06.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:06.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:06.908 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:46:06.949 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:06.997 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:06.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:06.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:07.000 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 79 16 118691 118675 93.8742 60.3125 0.0093239 0.0106476 2024-09-21T10:46:07.099 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:07.139 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:07.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:07.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:07.241 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:07.265 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:07.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:07.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:07.367 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:07.407 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:07.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:07.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:07.509 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:07.545 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:07.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:07.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:07.647 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:07.676 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:07.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:07.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:07.754 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:46:07.754 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:46:07.778 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:07.788 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:46:07.788 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:46:07.809 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:46:07.809 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:46:07.823 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:07.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:07.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:07.838 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:46:07.838 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:46:07.848 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:46:07.868 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:46:07.868 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:46:07.890 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:46:07.890 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:46:07.926 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:07.929 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:46:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:46:07.956 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:07.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:07.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:07.966 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:46:07.967 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:46:08.000 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:2024-09-21T10:46:07.999095+0000 min lat: 0.00136946 max lat: 0.178279 avg lat: 0.0106955 2024-09-21T10:46:08.000 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:46:08.000 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 80 16 119650 119634 93.4499 59.9375 0.00823373 0.0106955 2024-09-21T10:46:08.058 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:08.092 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:08.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:08.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:08.194 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:08.226 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:08.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:08.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:08.328 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:08.358 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:08.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:08.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:08.460 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:08.486 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:08 smithi160 ceph-mon[101879]: pgmap v2405: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 60 MiB/s wr, 966 op/s 2024-09-21T10:46:08.486 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:08 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2451103220' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:08.500 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:08.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:08.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:08.602 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:08.603 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:08 smithi070 ceph-mon[107385]: pgmap v2405: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 60 MiB/s wr, 966 op/s 2024-09-21T10:46:08.603 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:08 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2451103220' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:08.604 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:08 smithi070 ceph-mon[112139]: pgmap v2405: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 60 MiB/s wr, 966 op/s 2024-09-21T10:46:08.604 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:08 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2451103220' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:08.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:08.647 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:08.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:08.748 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:08.783 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:08.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:08.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:08.793 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:08.884 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:08.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:08.911 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:08.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:08.981 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:46:08.981 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T10:46:08.982 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T10:46:08.982 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T10:46:08.982 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.982 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.34878.0:1562 52.15 52:aa0c4d9e:::rbd_data.883ee71b525e.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1232)", 2024-09-21T10:46:08.982 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:37:04.139911+0000", 2024-09-21T10:46:08.982 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 544.83704703599994, 2024-09-21T10:46:08.982 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.046660791, 2024-09-21T10:46:08.982 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:46:08.982 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:08.982 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:46:08.982 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.34878", 2024-09-21T10:46:08.982 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/720470466", 2024-09-21T10:46:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1562 2024-09-21T10:46:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:46:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:46:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:37:04.139911+0000", 2024-09-21T10:46:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:46:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:37:04.139911+0000", 2024-09-21T10:46:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.984 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:46:08.984 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:37:04.139908+0000", 2024-09-21T10:46:08.984 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 2024-09-21T10:46:08.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.984 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:46:08.984 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:37:04.152158+0000", 2024-09-21T10:46:08.984 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012249467999999999 2024-09-21T10:46:08.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.984 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:46:08.984 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:37:04.152213+0000", 2024-09-21T10:46:08.984 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5149999999999999e-05 2024-09-21T10:46:08.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.985 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.985 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:46:08.985 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:37:04.152231+0000", 2024-09-21T10:46:08.985 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7802e-05 2024-09-21T10:46:08.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.985 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.985 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:46:08.985 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:37:04.152365+0000", 2024-09-21T10:46:08.985 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013420700000000001 2024-09-21T10:46:08.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.985 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.986 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:46:08.986 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:37:04.152614+0000", 2024-09-21T10:46:08.986 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024889699999999999 2024-09-21T10:46:08.986 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.986 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.986 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,4", 2024-09-21T10:46:08.986 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:37:04.152854+0000", 2024-09-21T10:46:08.986 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024058400000000001 2024-09-21T10:46:08.986 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.986 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:08.987 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:37:04.166177+0000", 2024-09-21T10:46:08.987 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.013322232 2024-09-21T10:46:08.987 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.987 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:46:08.987 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:37:04.185929+0000", 2024-09-21T10:46:08.987 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.019752136 2024-09-21T10:46:08.987 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.987 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:08.987 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:37:04.186469+0000", 2024-09-21T10:46:08.987 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00054036100000000005 2024-09-21T10:46:08.987 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:46:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:37:04.186527+0000", 2024-09-21T10:46:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7839e-05 2024-09-21T10:46:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:46:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:37:04.186571+0000", 2024-09-21T10:46:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4493999999999998e-05 2024-09-21T10:46:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:46:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.989 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.34884.0:2220 52.6 52:659a246a:::rbd_data.88447b362a13.0000000000000007:head [writefull 0~4194304] snapc 67=[67] ondisk+write+known_if_redirected+supports_pool_eio e1260)", 2024-09-21T10:46:08.989 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:38:09.476445+0000", 2024-09-21T10:46:08.989 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 479.50051304499999, 2024-09-21T10:46:08.989 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.047907204000000002, 2024-09-21T10:46:08.989 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:46:08.989 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:08.989 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:46:08.989 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.34884", 2024-09-21T10:46:08.989 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1225916197", 2024-09-21T10:46:08.989 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2220 2024-09-21T10:46:08.989 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.989 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:46:08.989 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.990 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:46:08.990 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:09.476445+0000", 2024-09-21T10:46:08.990 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:08.990 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.990 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.990 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:46:08.990 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:09.476445+0000", 2024-09-21T10:46:08.990 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:08.990 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.990 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.990 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:46:08.990 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:09.476442+0000", 2024-09-21T10:46:08.990 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 2024-09-21T10:46:08.991 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.991 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.991 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:46:08.991 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:09.483331+0000", 2024-09-21T10:46:08.991 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0068889340000000002 2024-09-21T10:46:08.991 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.991 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.991 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:46:08.991 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:09.483345+0000", 2024-09-21T10:46:08.991 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3832000000000001e-05 2024-09-21T10:46:08.991 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.991 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.991 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:46:08.991 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:09.483351+0000", 2024-09-21T10:46:08.992 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7610000000000004e-06 2024-09-21T10:46:08.992 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.992 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.992 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:46:08.992 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:09.483394+0000", 2024-09-21T10:46:08.992 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3752999999999999e-05 2024-09-21T10:46:08.992 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.992 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.992 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:46:08.992 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:09.483492+0000", 2024-09-21T10:46:08.992 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7274000000000004e-05 2024-09-21T10:46:08.992 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.992 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.993 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:08.993 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:09.483611+0000", 2024-09-21T10:46:08.993 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119296 2024-09-21T10:46:08.993 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.993 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.993 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:08.993 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:09.518642+0000", 2024-09-21T10:46:08.993 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.035030552999999999 2024-09-21T10:46:08.993 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.993 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.993 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:08.993 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:09.519022+0000", 2024-09-21T10:46:08.993 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000380112 2024-09-21T10:46:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:46:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:09.524159+0000", 2024-09-21T10:46:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0051373529999999999 2024-09-21T10:46:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:46:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:09.524258+0000", 2024-09-21T10:46:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9038999999999998e-05 2024-09-21T10:46:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:46:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:09.524352+0000", 2024-09-21T10:46:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3712e-05 2024-09-21T10:46:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:46:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.34884.0:2314 52.8 e1262/1156 52:12908c8e:::rbd_data.88447b362a13.0000000000000045:head v 1262'25, mlcod=1256'23)", 2024-09-21T10:46:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:38:11.465125+0000", 2024-09-21T10:46:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 477.51183236399999, 2024-09-21T10:46:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060772406000000001, 2024-09-21T10:46:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:46:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:46:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.996 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:46:08.996 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:11.465125+0000", 2024-09-21T10:46:08.996 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:08.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.996 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.997 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:46:08.997 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:11.465125+0000", 2024-09-21T10:46:08.997 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:08.997 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.997 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.997 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:46:08.997 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:11.465125+0000", 2024-09-21T10:46:08.997 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T10:46:08.997 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.997 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.997 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:46:08.997 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:11.492908+0000", 2024-09-21T10:46:08.997 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.027783018 2024-09-21T10:46:08.998 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.998 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.998 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:46:08.998 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:11.492925+0000", 2024-09-21T10:46:08.998 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7499e-05 2024-09-21T10:46:08.998 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.998 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.998 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:46:08.998 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:11.492932+0000", 2024-09-21T10:46:08.998 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.075e-06 2024-09-21T10:46:08.998 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:46:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:11.492989+0000", 2024-09-21T10:46:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.63e-05 2024-09-21T10:46:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:46:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:11.493021+0000", 2024-09-21T10:46:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2162000000000001e-05 2024-09-21T10:46:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:46:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:11.525853+0000", 2024-09-21T10:46:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.032832171 2024-09-21T10:46:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:46:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:11.525898+0000", 2024-09-21T10:46:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4855000000000003e-05 2024-09-21T10:46:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:46:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.34884.0:3107 52.6 52:617e9277:::rbd_data.88447b362a13.0000000000000023:head [write 1958278~2236026] snapc 77=[77,76,75,74,73,72,71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1276)", 2024-09-21T10:46:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:38:25.610222+0000", 2024-09-21T10:46:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 463.36673576099997, 2024-09-21T10:46:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.047366620999999998, 2024-09-21T10:46:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:46:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:46:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.34884", 2024-09-21T10:46:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1225916197", 2024-09-21T10:46:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3107 2024-09-21T10:46:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:46:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:46:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:25.610222+0000", 2024-09-21T10:46:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.002 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.002 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:46:09.002 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:25.610222+0000", 2024-09-21T10:46:09.002 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.002 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.002 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.002 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:46:09.002 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:25.610220+0000", 2024-09-21T10:46:09.002 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:46:09.002 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.002 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.002 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:46:09.002 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:25.622536+0000", 2024-09-21T10:46:09.003 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012315421 2024-09-21T10:46:09.003 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.003 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.003 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:46:09.003 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:25.622576+0000", 2024-09-21T10:46:09.003 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0528000000000002e-05 2024-09-21T10:46:09.003 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.003 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.003 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:46:09.003 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:25.622593+0000", 2024-09-21T10:46:09.003 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6155e-05 2024-09-21T10:46:09.003 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:46:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:25.622773+0000", 2024-09-21T10:46:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000180345 2024-09-21T10:46:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:46:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:25.622978+0000", 2024-09-21T10:46:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000204868 2024-09-21T10:46:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:25.623318+0000", 2024-09-21T10:46:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034039299999999998 2024-09-21T10:46:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:46:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:25.639622+0000", 2024-09-21T10:46:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.016303673000000001 2024-09-21T10:46:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:25.653578+0000", 2024-09-21T10:46:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.013955769999999999 2024-09-21T10:46:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.006 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:25.657480+0000", 2024-09-21T10:46:09.006 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0039020600000000002 2024-09-21T10:46:09.006 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.006 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.006 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:46:09.006 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:25.657537+0000", 2024-09-21T10:46:09.006 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6872e-05 2024-09-21T10:46:09.006 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.006 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.006 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:46:09.006 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:25.657589+0000", 2024-09-21T10:46:09.006 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2012999999999998e-05 2024-09-21T10:46:09.006 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.006 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:46:09.007 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.007 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.007 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.007 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.34884.0:6760 52.6 52:630220c4:::rbd_data.884493f380ee.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1298)", 2024-09-21T10:46:09.007 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:38:47.191627+0000", 2024-09-21T10:46:09.007 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 441.78533079099998, 2024-09-21T10:46:09.007 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.048079338999999999, 2024-09-21T10:46:09.007 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:46:09.007 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:09.007 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:46:09.007 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.34884", 2024-09-21T10:46:09.008 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1225916197", 2024-09-21T10:46:09.008 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 6760 2024-09-21T10:46:09.008 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.008 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:46:09.008 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.008 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:46:09.008 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:47.191627+0000", 2024-09-21T10:46:09.008 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:46:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:47.191627+0000", 2024-09-21T10:46:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:46:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:47.191623+0000", 2024-09-21T10:46:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999967 2024-09-21T10:46:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:46:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:47.203116+0000", 2024-09-21T10:46:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011492234 2024-09-21T10:46:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:46:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:47.203145+0000", 2024-09-21T10:46:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9583000000000001e-05 2024-09-21T10:46:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:46:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:47.203157+0000", 2024-09-21T10:46:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1778e-05 2024-09-21T10:46:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:46:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:47.203224+0000", 2024-09-21T10:46:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6777000000000001e-05 2024-09-21T10:46:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:46:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:47.203443+0000", 2024-09-21T10:46:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00021932700000000001 2024-09-21T10:46:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:47.203631+0000", 2024-09-21T10:46:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000187951 2024-09-21T10:46:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:46:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:47.238728+0000", 2024-09-21T10:46:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.035097418999999998 2024-09-21T10:46:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:47.239384+0000", 2024-09-21T10:46:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00065548500000000003 2024-09-21T10:46:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:47.239615+0000", 2024-09-21T10:46:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000230762 2024-09-21T10:46:09.013 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.013 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.013 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:46:09.013 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:47.239662+0000", 2024-09-21T10:46:09.013 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7091e-05 2024-09-21T10:46:09.013 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.013 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.013 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:46:09.013 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:47.239706+0000", 2024-09-21T10:46:09.013 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4437999999999997e-05 2024-09-21T10:46:09.013 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.013 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:46:09.013 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.013 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.014 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.014 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.34884.0:6931 52.d e1299/1156 52:b65311e1:::rbd_data.88444b16d5ef.0000000000000002:head v 1299'88, mlcod=1299'86)", 2024-09-21T10:46:09.014 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:38:48.223754+0000", 2024-09-21T10:46:09.014 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 440.75320330400001, 2024-09-21T10:46:09.014 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.046912175, 2024-09-21T10:46:09.014 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:46:09.014 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:09.014 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:46:09.014 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.014 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:46:09.014 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:48.223754+0000", 2024-09-21T10:46:09.014 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.015 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.015 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.015 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:46:09.015 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:48.223754+0000", 2024-09-21T10:46:09.015 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.015 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.015 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.015 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:46:09.015 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:48.223753+0000", 2024-09-21T10:46:09.015 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:46:09.015 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.015 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.015 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:46:09.015 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:48.246187+0000", 2024-09-21T10:46:09.016 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.022433578999999999 2024-09-21T10:46:09.016 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.016 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:46:09.016 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:48.246200+0000", 2024-09-21T10:46:09.016 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3370000000000001e-05 2024-09-21T10:46:09.016 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.016 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:46:09.016 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:48.246206+0000", 2024-09-21T10:46:09.016 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9309999999999996e-06 2024-09-21T10:46:09.016 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.017 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.017 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:46:09.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:48.246244+0000", 2024-09-21T10:46:09.017 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7317999999999997e-05 2024-09-21T10:46:09.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.017 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.017 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:46:09.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:48.246261+0000", 2024-09-21T10:46:09.017 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6932e-05 2024-09-21T10:46:09.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.017 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.017 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:46:09.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:48.270617+0000", 2024-09-21T10:46:09.017 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.024356203 2024-09-21T10:46:09.018 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.018 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.018 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:46:09.018 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:38:48.270667+0000", 2024-09-21T10:46:09.018 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9764e-05 2024-09-21T10:46:09.018 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.018 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:46:09.018 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.018 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.018 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.018 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:68028 54.6 54:62847781:::benchmark_data_smithi070_261699_object68027:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1382)", 2024-09-21T10:46:09.018 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:20.516743+0000", 2024-09-21T10:46:09.019 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 48.460214526999998, 2024-09-21T10:46:09.019 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.050147530000000003, 2024-09-21T10:46:09.019 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:46:09.019 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:09.019 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:46:09.019 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:46:09.019 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:46:09.019 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 68028 2024-09-21T10:46:09.019 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.019 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:46:09.019 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.019 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:46:09.019 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.516743+0000", 2024-09-21T10:46:09.020 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.020 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.020 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.020 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:46:09.020 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.516743+0000", 2024-09-21T10:46:09.020 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.020 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.020 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.020 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:46:09.020 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.516741+0000", 2024-09-21T10:46:09.020 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:46:09.020 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.020 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.020 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:46:09.021 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.516813+0000", 2024-09-21T10:46:09.021 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1520999999999995e-05 2024-09-21T10:46:09.021 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.021 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.021 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:46:09.021 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.516827+0000", 2024-09-21T10:46:09.021 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3706999999999999e-05 2024-09-21T10:46:09.021 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.021 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.021 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:46:09.021 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.516834+0000", 2024-09-21T10:46:09.021 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5560000000000002e-06 2024-09-21T10:46:09.021 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.021 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.022 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:46:09.022 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.516881+0000", 2024-09-21T10:46:09.022 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6854999999999998e-05 2024-09-21T10:46:09.022 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.022 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.022 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:46:09.022 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.517009+0000", 2024-09-21T10:46:09.022 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012794099999999999 2024-09-21T10:46:09.022 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.022 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.022 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:09.022 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.517152+0000", 2024-09-21T10:46:09.022 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014295000000000001 2024-09-21T10:46:09.023 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.023 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.023 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:46:09.023 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.522232+0000", 2024-09-21T10:46:09.023 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0050798969999999999 2024-09-21T10:46:09.023 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.023 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.023 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.023 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.522612+0000", 2024-09-21T10:46:09.023 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00038059200000000002 2024-09-21T10:46:09.023 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.023 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.023 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.023 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.566759+0000", 2024-09-21T10:46:09.024 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.044146060000000001 2024-09-21T10:46:09.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.024 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.024 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:46:09.024 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.566848+0000", 2024-09-21T10:46:09.024 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9289999999999994e-05 2024-09-21T10:46:09.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.024 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.024 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:46:09.024 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.566891+0000", 2024-09-21T10:46:09.024 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2862999999999999e-05 2024-09-21T10:46:09.024 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.024 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:46:09.024 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.025 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.025 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:68033 54.6 54:6fd62d46:::benchmark_data_smithi070_261699_object68032:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1382)", 2024-09-21T10:46:09.025 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:20.517120+0000", 2024-09-21T10:46:09.025 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 48.459837364999998, 2024-09-21T10:46:09.025 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.049912522000000001, 2024-09-21T10:46:09.025 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:46:09.025 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:09.025 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:46:09.025 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:46:09.025 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:46:09.025 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 68033 2024-09-21T10:46:09.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.025 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:46:09.026 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.026 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:46:09.026 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.517120+0000", 2024-09-21T10:46:09.026 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.026 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.026 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.026 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:46:09.026 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.517120+0000", 2024-09-21T10:46:09.026 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.026 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.026 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.026 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:46:09.027 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.517119+0000", 2024-09-21T10:46:09.027 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:46:09.027 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.027 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.027 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:46:09.027 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.517351+0000", 2024-09-21T10:46:09.027 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023168 2024-09-21T10:46:09.027 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.027 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.027 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:46:09.027 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.517362+0000", 2024-09-21T10:46:09.027 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1354e-05 2024-09-21T10:46:09.027 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.027 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.028 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:46:09.028 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.517370+0000", 2024-09-21T10:46:09.028 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3139999999999998e-06 2024-09-21T10:46:09.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.028 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.028 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:46:09.028 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.519533+0000", 2024-09-21T10:46:09.028 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021636820000000001 2024-09-21T10:46:09.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.028 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.028 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:46:09.028 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.519730+0000", 2024-09-21T10:46:09.028 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019654399999999999 2024-09-21T10:46:09.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.028 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.029 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:09.029 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.519868+0000", 2024-09-21T10:46:09.029 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000137935 2024-09-21T10:46:09.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.029 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.029 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:46:09.029 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.522377+0000", 2024-09-21T10:46:09.029 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002508999 2024-09-21T10:46:09.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.029 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.029 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.029 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.522754+0000", 2024-09-21T10:46:09.029 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00037706300000000002 2024-09-21T10:46:09.030 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.030 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.030 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.566926+0000", 2024-09-21T10:46:09.030 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.044172613999999999 2024-09-21T10:46:09.030 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.030 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:46:09.030 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.566998+0000", 2024-09-21T10:46:09.030 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1606000000000005e-05 2024-09-21T10:46:09.030 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.030 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:46:09.030 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:20.567033+0000", 2024-09-21T10:46:09.031 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4808000000000003e-05 2024-09-21T10:46:09.031 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.031 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:46:09.031 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.031 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.031 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:86458 54.6 54:6eae050f:::benchmark_data_smithi070_261699_object86457:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1383)", 2024-09-21T10:46:09.031 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:35.532865+0000", 2024-09-21T10:46:09.031 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 33.444093099, 2024-09-21T10:46:09.031 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.053273037000000002, 2024-09-21T10:46:09.031 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:46:09.031 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:09.031 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:46:09.031 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:46:09.032 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:46:09.032 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 86458 2024-09-21T10:46:09.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.032 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:46:09.032 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.032 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:46:09.032 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:35.532865+0000", 2024-09-21T10:46:09.032 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.032 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.032 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:46:09.032 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:35.532865+0000", 2024-09-21T10:46:09.032 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.033 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.033 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:46:09.033 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:35.532863+0000", 2024-09-21T10:46:09.033 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:46:09.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.033 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.033 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:46:09.033 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:35.532898+0000", 2024-09-21T10:46:09.033 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5639000000000002e-05 2024-09-21T10:46:09.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.033 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.033 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:46:09.033 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:35.532915+0000", 2024-09-21T10:46:09.034 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6538999999999999e-05 2024-09-21T10:46:09.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.034 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.034 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:46:09.034 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:35.532925+0000", 2024-09-21T10:46:09.034 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.039e-05 2024-09-21T10:46:09.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.034 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.034 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:46:09.034 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:35.552059+0000", 2024-09-21T10:46:09.034 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.019133653 2024-09-21T10:46:09.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.034 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:46:09.035 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:35.552191+0000", 2024-09-21T10:46:09.035 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013160299999999999 2024-09-21T10:46:09.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.035 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:09.035 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:35.552314+0000", 2024-09-21T10:46:09.035 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012378199999999999 2024-09-21T10:46:09.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.035 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.035 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:35.564465+0000", 2024-09-21T10:46:09.035 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012150325999999999 2024-09-21T10:46:09.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.036 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.036 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:46:09.036 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:35.564472+0000", 2024-09-21T10:46:09.036 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5789999999999998e-06 2024-09-21T10:46:09.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.036 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.036 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.036 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:35.586025+0000", 2024-09-21T10:46:09.036 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.021552696 2024-09-21T10:46:09.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.036 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.036 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:46:09.036 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:35.586103+0000", 2024-09-21T10:46:09.037 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7768000000000005e-05 2024-09-21T10:46:09.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.037 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:46:09.037 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:35.586138+0000", 2024-09-21T10:46:09.037 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4822e-05 2024-09-21T10:46:09.037 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.037 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:46:09.037 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.037 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:87097 54.6 54:6345172c:::benchmark_data_smithi070_261699_object87096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1383)", 2024-09-21T10:46:09.037 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:36.128615+0000", 2024-09-21T10:46:09.037 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 32.848342967999997, 2024-09-21T10:46:09.038 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.056888490999999999, 2024-09-21T10:46:09.038 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:46:09.038 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:09.038 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:46:09.038 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:46:09.038 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:46:09.038 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 87097 2024-09-21T10:46:09.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.038 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:46:09.038 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.038 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:46:09.038 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.128615+0000", 2024-09-21T10:46:09.038 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.039 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.039 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:46:09.039 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.128615+0000", 2024-09-21T10:46:09.039 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.039 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.039 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:46:09.039 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.128613+0000", 2024-09-21T10:46:09.039 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:46:09.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.039 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.040 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:09.041 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 81 16 120581 120565 93.0145 58.1875 0.0133705 0.0107449 2024-09-21T10:46:09.041 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:46:09.041 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.128643+0000", 2024-09-21T10:46:09.041 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0383e-05 2024-09-21T10:46:09.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.041 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:46:09.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.128657+0000", 2024-09-21T10:46:09.042 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3229e-05 2024-09-21T10:46:09.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.042 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:46:09.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.128664+0000", 2024-09-21T10:46:09.042 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0480000000000001e-06 2024-09-21T10:46:09.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.042 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:46:09.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.132067+0000", 2024-09-21T10:46:09.042 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0034030890000000002 2024-09-21T10:46:09.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.043 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.043 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:46:09.043 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.132181+0000", 2024-09-21T10:46:09.043 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011461800000000001 2024-09-21T10:46:09.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.043 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.043 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:09.043 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.132301+0000", 2024-09-21T10:46:09.043 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119888 2024-09-21T10:46:09.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.043 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.043 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:46:09.043 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.136142+0000", 2024-09-21T10:46:09.044 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003841185 2024-09-21T10:46:09.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.044 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.044 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.139666+0000", 2024-09-21T10:46:09.044 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0035232570000000001 2024-09-21T10:46:09.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.044 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.044 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.185409+0000", 2024-09-21T10:46:09.044 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.045743314 2024-09-21T10:46:09.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.044 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:46:09.045 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.185464+0000", 2024-09-21T10:46:09.045 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5192999999999998e-05 2024-09-21T10:46:09.045 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.045 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.045 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:46:09.045 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.185503+0000", 2024-09-21T10:46:09.045 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9023999999999998e-05 2024-09-21T10:46:09.045 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.045 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:46:09.045 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.045 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.045 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.045 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:87123 54.6 54:6ae74ca9:::benchmark_data_smithi070_261699_object87122:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1383)", 2024-09-21T10:46:09.045 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:36.152033+0000", 2024-09-21T10:46:09.046 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 32.824925004999997, 2024-09-21T10:46:09.046 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.045717333999999998, 2024-09-21T10:46:09.046 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:46:09.046 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:09.046 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:46:09.046 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:46:09.046 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:46:09.046 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 87123 2024-09-21T10:46:09.046 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.046 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:46:09.046 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.046 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:46:09.046 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.152033+0000", 2024-09-21T10:46:09.046 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.047 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.047 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.047 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:46:09.047 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.152033+0000", 2024-09-21T10:46:09.047 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.047 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.047 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.047 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:46:09.047 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.152031+0000", 2024-09-21T10:46:09.047 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:46:09.047 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.047 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.047 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:46:09.048 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.152098+0000", 2024-09-21T10:46:09.048 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6846000000000003e-05 2024-09-21T10:46:09.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.048 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.048 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:46:09.048 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.152108+0000", 2024-09-21T10:46:09.048 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4590000000000004e-06 2024-09-21T10:46:09.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.048 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.048 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:46:09.048 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.152113+0000", 2024-09-21T10:46:09.048 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8960000000000002e-06 2024-09-21T10:46:09.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.048 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.049 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:46:09.049 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.152151+0000", 2024-09-21T10:46:09.049 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8912999999999998e-05 2024-09-21T10:46:09.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.049 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.049 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:46:09.049 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.152270+0000", 2024-09-21T10:46:09.049 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000118785 2024-09-21T10:46:09.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.049 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.049 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:09.049 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.152361+0000", 2024-09-21T10:46:09.049 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0533999999999998e-05 2024-09-21T10:46:09.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.050 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:46:09.050 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.154681+0000", 2024-09-21T10:46:09.050 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0023205280000000001 2024-09-21T10:46:09.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.050 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.050 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.155274+0000", 2024-09-21T10:46:09.050 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00059288600000000004 2024-09-21T10:46:09.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.050 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.050 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.197658+0000", 2024-09-21T10:46:09.051 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.042383990000000003 2024-09-21T10:46:09.051 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.051 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.051 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:46:09.051 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.197716+0000", 2024-09-21T10:46:09.051 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7476999999999999e-05 2024-09-21T10:46:09.051 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.051 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.051 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:46:09.051 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.197750+0000", 2024-09-21T10:46:09.051 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4345999999999999e-05 2024-09-21T10:46:09.051 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.051 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:46:09.051 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.052 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.052 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.052 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:87943 54.6 54:64180b92:::benchmark_data_smithi070_261699_object87942:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1383)", 2024-09-21T10:46:09.052 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:36.876719+0000", 2024-09-21T10:46:09.052 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 32.100239035999998, 2024-09-21T10:46:09.052 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.067665215000000001, 2024-09-21T10:46:09.052 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:46:09.052 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:09.052 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:46:09.052 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:46:09.052 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:46:09.052 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 87943 2024-09-21T10:46:09.052 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.052 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:46:09.053 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.053 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:46:09.053 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.876719+0000", 2024-09-21T10:46:09.053 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.053 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.053 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.053 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:46:09.053 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.876719+0000", 2024-09-21T10:46:09.053 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.053 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.053 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.053 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:46:09.053 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.876717+0000", 2024-09-21T10:46:09.053 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:46:09.054 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.054 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.054 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:46:09.054 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.876753+0000", 2024-09-21T10:46:09.054 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6455999999999998e-05 2024-09-21T10:46:09.054 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.054 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.054 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:46:09.054 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.876767+0000", 2024-09-21T10:46:09.054 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3854e-05 2024-09-21T10:46:09.054 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.054 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.054 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:46:09.054 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.876775+0000", 2024-09-21T10:46:09.054 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4599999999999997e-06 2024-09-21T10:46:09.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.055 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:46:09.055 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.877864+0000", 2024-09-21T10:46:09.055 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0010891970000000001 2024-09-21T10:46:09.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.055 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:46:09.055 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.878029+0000", 2024-09-21T10:46:09.055 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016493399999999999 2024-09-21T10:46:09.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.055 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:09.056 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.878170+0000", 2024-09-21T10:46:09.056 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014153500000000001 2024-09-21T10:46:09.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.056 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.056 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:46:09.056 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.881381+0000", 2024-09-21T10:46:09.056 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0032107709999999999 2024-09-21T10:46:09.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.056 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.056 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.056 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.881966+0000", 2024-09-21T10:46:09.056 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00058458200000000005 2024-09-21T10:46:09.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.056 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.056 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.057 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.944299+0000", 2024-09-21T10:46:09.057 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.062333338000000002 2024-09-21T10:46:09.057 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.057 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.057 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:46:09.057 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.944348+0000", 2024-09-21T10:46:09.057 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8801999999999998e-05 2024-09-21T10:46:09.057 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.057 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.057 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:46:09.057 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:36.944384+0000", 2024-09-21T10:46:09.057 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6025999999999997e-05 2024-09-21T10:46:09.057 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.057 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:46:09.058 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.058 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.058 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:94731 54.6 54:6c993fe8:::benchmark_data_smithi070_261699_object94730:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1385)", 2024-09-21T10:46:09.058 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:42.995374+0000", 2024-09-21T10:46:09.058 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 25.981583806, 2024-09-21T10:46:09.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.052825614, 2024-09-21T10:46:09.058 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:46:09.058 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:09.058 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:46:09.058 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:46:09.058 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:46:09.058 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 94731 2024-09-21T10:46:09.059 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.059 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:46:09.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.059 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:46:09.059 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:42.995374+0000", 2024-09-21T10:46:09.059 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.059 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.059 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:46:09.059 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:42.995374+0000", 2024-09-21T10:46:09.059 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.059 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.059 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:46:09.060 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:42.995372+0000", 2024-09-21T10:46:09.060 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:46:09.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.060 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.060 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:46:09.060 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:42.995420+0000", 2024-09-21T10:46:09.060 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7729000000000003e-05 2024-09-21T10:46:09.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.060 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.060 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:46:09.060 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:42.995432+0000", 2024-09-21T10:46:09.060 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2371000000000001e-05 2024-09-21T10:46:09.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.060 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.061 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:46:09.061 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:42.995439+0000", 2024-09-21T10:46:09.061 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6030000000000001e-06 2024-09-21T10:46:09.061 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.061 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.061 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:46:09.061 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:42.995492+0000", 2024-09-21T10:46:09.061 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2978e-05 2024-09-21T10:46:09.061 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.061 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.061 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:46:09.061 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:42.995600+0000", 2024-09-21T10:46:09.061 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000108848 2024-09-21T10:46:09.061 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.061 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.062 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:09.062 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:42.995734+0000", 2024-09-21T10:46:09.062 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013346500000000001 2024-09-21T10:46:09.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.062 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.062 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:43.007043+0000", 2024-09-21T10:46:09.062 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011308907999999999 2024-09-21T10:46:09.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.062 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:46:09.062 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:43.007053+0000", 2024-09-21T10:46:09.062 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0710000000000001e-05 2024-09-21T10:46:09.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:43.048079+0000", 2024-09-21T10:46:09.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.041025158999999999 2024-09-21T10:46:09.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:46:09.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:43.048162+0000", 2024-09-21T10:46:09.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3007e-05 2024-09-21T10:46:09.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:46:09.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:43.048199+0000", 2024-09-21T10:46:09.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7815999999999998e-05 2024-09-21T10:46:09.064 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.064 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:46:09.064 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.064 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.064 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:94733 54.6 54:644e2055:::benchmark_data_smithi070_261699_object94732:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1385)", 2024-09-21T10:46:09.064 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:43.003393+0000", 2024-09-21T10:46:09.064 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 25.973564953, 2024-09-21T10:46:09.064 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.045097258000000001, 2024-09-21T10:46:09.064 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:46:09.064 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:09.064 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:46:09.064 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:46:09.064 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:46:09.065 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 94733 2024-09-21T10:46:09.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.065 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:46:09.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.065 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:46:09.065 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:43.003393+0000", 2024-09-21T10:46:09.065 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.065 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:46:09.065 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:43.003393+0000", 2024-09-21T10:46:09.065 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:46:09.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:43.003391+0000", 2024-09-21T10:46:09.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:46:09.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:46:09.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:43.003427+0000", 2024-09-21T10:46:09.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5988999999999997e-05 2024-09-21T10:46:09.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:46:09.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:43.003441+0000", 2024-09-21T10:46:09.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3946e-05 2024-09-21T10:46:09.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:46:09.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:43.003448+0000", 2024-09-21T10:46:09.067 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1539999999999996e-06 2024-09-21T10:46:09.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:46:09.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:43.004354+0000", 2024-09-21T10:46:09.067 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00090565700000000003 2024-09-21T10:46:09.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:46:09.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:43.004458+0000", 2024-09-21T10:46:09.067 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000103976 2024-09-21T10:46:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:43.004540+0000", 2024-09-21T10:46:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1959000000000006e-05 2024-09-21T10:46:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:46:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:43.007143+0000", 2024-09-21T10:46:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002602998 2024-09-21T10:46:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:43.007498+0000", 2024-09-21T10:46:09.069 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00035513199999999997 2024-09-21T10:46:09.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.069 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.069 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:43.048359+0000", 2024-09-21T10:46:09.069 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.040861434000000002 2024-09-21T10:46:09.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.069 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:46:09.069 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:43.048450+0000", 2024-09-21T10:46:09.069 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0685000000000003e-05 2024-09-21T10:46:09.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.069 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:46:09.070 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:43.048490+0000", 2024-09-21T10:46:09.070 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9867000000000003e-05 2024-09-21T10:46:09.070 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.070 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:46:09.070 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.070 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.070 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:106558 54.6 54:619aacf4:::benchmark_data_smithi070_261699_object106557:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-09-21T10:46:09.070 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:54.354371+0000", 2024-09-21T10:46:09.070 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 14.622586402, 2024-09-21T10:46:09.070 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.049799046999999999, 2024-09-21T10:46:09.070 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:46:09.070 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:09.071 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:46:09.071 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:46:09.071 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:46:09.071 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 106558 2024-09-21T10:46:09.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.071 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:46:09.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:46:09.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.354371+0000", 2024-09-21T10:46:09.071 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:46:09.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.354371+0000", 2024-09-21T10:46:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:46:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.354370+0000", 2024-09-21T10:46:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:46:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:46:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.354426+0000", 2024-09-21T10:46:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6078000000000001e-05 2024-09-21T10:46:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:46:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.354437+0000", 2024-09-21T10:46:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1251e-05 2024-09-21T10:46:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:46:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.354443+0000", 2024-09-21T10:46:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.694e-06 2024-09-21T10:46:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:46:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.354479+0000", 2024-09-21T10:46:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6068000000000001e-05 2024-09-21T10:46:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:46:09.074 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.354559+0000", 2024-09-21T10:46:09.074 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9757000000000002e-05 2024-09-21T10:46:09.074 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.074 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.074 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:09.074 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.354644+0000", 2024-09-21T10:46:09.074 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4740000000000005e-05 2024-09-21T10:46:09.074 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.074 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.074 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:46:09.074 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.357590+0000", 2024-09-21T10:46:09.074 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029463079999999999 2024-09-21T10:46:09.074 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.358003+0000", 2024-09-21T10:46:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000412707 2024-09-21T10:46:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.404086+0000", 2024-09-21T10:46:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.046083269000000003 2024-09-21T10:46:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:46:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.404135+0000", 2024-09-21T10:46:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9465999999999997e-05 2024-09-21T10:46:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.076 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.076 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:46:09.076 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.404170+0000", 2024-09-21T10:46:09.076 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5039000000000001e-05 2024-09-21T10:46:09.076 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.076 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:46:09.076 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.076 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.076 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.076 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.45308.0:106877 54.7 e1386/1376 54:e0299293:::benchmark_data_smithi070_261699_object106876:head v 1386'6772, mlcod=1386'6768)", 2024-09-21T10:46:09.076 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:54.729951+0000", 2024-09-21T10:46:09.076 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 14.247007079999999, 2024-09-21T10:46:09.076 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.052565878000000003, 2024-09-21T10:46:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:46:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:46:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:46:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.729951+0000", 2024-09-21T10:46:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:46:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.729951+0000", 2024-09-21T10:46:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:46:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.729950+0000", 2024-09-21T10:46:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:46:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:46:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.730000+0000", 2024-09-21T10:46:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0256999999999997e-05 2024-09-21T10:46:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:46:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.730011+0000", 2024-09-21T10:46:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1268e-05 2024-09-21T10:46:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:46:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.730018+0000", 2024-09-21T10:46:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.737e-06 2024-09-21T10:46:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:46:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.735067+0000", 2024-09-21T10:46:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0050489649999999999 2024-09-21T10:46:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:46:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.735113+0000", 2024-09-21T10:46:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5728e-05 2024-09-21T10:46:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:46:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.780052+0000", 2024-09-21T10:46:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.044938902000000003 2024-09-21T10:46:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:46:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.782516+0000", 2024-09-21T10:46:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002464777 2024-09-21T10:46:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:46:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.45308.0:106886 54.7 e1386/1376 54:ee6c263d:::benchmark_data_smithi070_261699_object106885:head v 1386'6773, mlcod=1386'6769)", 2024-09-21T10:46:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:54.737643+0000", 2024-09-21T10:46:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 14.239314702, 2024-09-21T10:46:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.044927751000000002, 2024-09-21T10:46:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:46:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:46:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:46:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.737643+0000", 2024-09-21T10:46:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:46:09.082 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.737643+0000", 2024-09-21T10:46:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:46:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.737642+0000", 2024-09-21T10:46:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:46:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:46:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.737703+0000", 2024-09-21T10:46:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0673e-05 2024-09-21T10:46:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:46:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.737717+0000", 2024-09-21T10:46:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4708999999999999e-05 2024-09-21T10:46:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:46:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.737725+0000", 2024-09-21T10:46:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1200000000000002e-06 2024-09-21T10:46:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:46:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.740251+0000", 2024-09-21T10:46:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0025254819999999999 2024-09-21T10:46:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:46:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.740293+0000", 2024-09-21T10:46:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2354000000000001e-05 2024-09-21T10:46:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:46:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.782521+0000", 2024-09-21T10:46:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.042227712000000001 2024-09-21T10:46:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:46:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:54.782571+0000", 2024-09-21T10:46:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9784999999999999e-05 2024-09-21T10:46:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:46:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:108657 54.6 54:635e2522:::benchmark_data_smithi070_261699_object108656:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1387)", 2024-09-21T10:46:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:56.652319+0000", 2024-09-21T10:46:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 12.324638994000001, 2024-09-21T10:46:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.050551049000000001, 2024-09-21T10:46:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:46:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:46:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:46:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:46:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 108657 2024-09-21T10:46:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:46:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:46:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.652319+0000", 2024-09-21T10:46:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:46:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.652319+0000", 2024-09-21T10:46:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:46:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.652317+0000", 2024-09-21T10:46:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:46:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:46:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.652347+0000", 2024-09-21T10:46:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0320000000000001e-05 2024-09-21T10:46:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:46:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.652360+0000", 2024-09-21T10:46:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2833999999999999e-05 2024-09-21T10:46:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:46:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.652367+0000", 2024-09-21T10:46:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3300000000000001e-06 2024-09-21T10:46:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:46:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.673546+0000", 2024-09-21T10:46:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.021178755 2024-09-21T10:46:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:46:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.673670+0000", 2024-09-21T10:46:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000123888 2024-09-21T10:46:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.673808+0000", 2024-09-21T10:46:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013861 2024-09-21T10:46:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:46:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.687115+0000", 2024-09-21T10:46:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.013306517 2024-09-21T10:46:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.687190+0000", 2024-09-21T10:46:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4863000000000005e-05 2024-09-21T10:46:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.702744+0000", 2024-09-21T10:46:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.015553803 2024-09-21T10:46:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:46:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.702839+0000", 2024-09-21T10:46:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5628e-05 2024-09-21T10:46:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:46:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.702870+0000", 2024-09-21T10:46:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0425000000000001e-05 2024-09-21T10:46:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:46:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:108663 54.6 54:67d0e1c3:::benchmark_data_smithi070_261699_object108662:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1387)", 2024-09-21T10:46:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:56.656845+0000", 2024-09-21T10:46:09.093 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 12.320112265000001, 2024-09-21T10:46:09.093 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.047213948999999998, 2024-09-21T10:46:09.093 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:46:09.093 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:09.093 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:46:09.093 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:46:09.093 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:46:09.093 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 108663 2024-09-21T10:46:09.093 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.093 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:46:09.093 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.093 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:46:09.093 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.656845+0000", 2024-09-21T10:46:09.094 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.094 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.094 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.094 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:46:09.094 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.656845+0000", 2024-09-21T10:46:09.094 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.094 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.094 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.094 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:46:09.094 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.656844+0000", 2024-09-21T10:46:09.094 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:46:09.094 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.094 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.094 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:46:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.656938+0000", 2024-09-21T10:46:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4245000000000003e-05 2024-09-21T10:46:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:46:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.656955+0000", 2024-09-21T10:46:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6735000000000002e-05 2024-09-21T10:46:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:46:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.656964+0000", 2024-09-21T10:46:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6340000000000007e-06 2024-09-21T10:46:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:46:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.683280+0000", 2024-09-21T10:46:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.026316665 2024-09-21T10:46:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:46:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.683431+0000", 2024-09-21T10:46:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015063899999999999 2024-09-21T10:46:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.683577+0000", 2024-09-21T10:46:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014623300000000001 2024-09-21T10:46:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:46:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.687367+0000", 2024-09-21T10:46:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0037899180000000002 2024-09-21T10:46:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.692572+0000", 2024-09-21T10:46:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0052048779999999996 2024-09-21T10:46:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.703685+0000", 2024-09-21T10:46:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011112683 2024-09-21T10:46:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:46:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.703838+0000", 2024-09-21T10:46:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000153318 2024-09-21T10:46:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:46:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:56.704059+0000", 2024-09-21T10:46:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000221377 2024-09-21T10:46:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:46:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:120056 54.6 54:60466b3f:::benchmark_data_smithi070_261699_object120055:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1388)", 2024-09-21T10:46:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:46:08.425587+0000", 2024-09-21T10:46:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.55137072799999998, 2024-09-21T10:46:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.044797519000000001, 2024-09-21T10:46:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:46:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:46:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:46:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:46:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 120056 2024-09-21T10:46:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:46:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:46:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:46:08.425587+0000", 2024-09-21T10:46:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:46:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:46:08.425587+0000", 2024-09-21T10:46:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:46:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:46:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:46:08.425585+0000", 2024-09-21T10:46:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:46:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:46:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:46:08.425628+0000", 2024-09-21T10:46:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3001999999999999e-05 2024-09-21T10:46:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:46:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:46:08.425647+0000", 2024-09-21T10:46:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8533999999999999e-05 2024-09-21T10:46:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:46:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:46:08.425655+0000", 2024-09-21T10:46:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5550000000000003e-06 2024-09-21T10:46:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:46:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:46:08.430381+0000", 2024-09-21T10:46:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0047252359999999998 2024-09-21T10:46:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:46:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:46:08.430586+0000", 2024-09-21T10:46:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00020491800000000001 2024-09-21T10:46:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:46:08.430736+0000", 2024-09-21T10:46:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015028799999999999 2024-09-21T10:46:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:46:08.438876+0000", 2024-09-21T10:46:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0081402520000000006 2024-09-21T10:46:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:46:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:46:08.451852+0000", 2024-09-21T10:46:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012975453 2024-09-21T10:46:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:46:08.470066+0000", 2024-09-21T10:46:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.018214944 2024-09-21T10:46:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:46:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:46:08.470248+0000", 2024-09-21T10:46:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000181473 2024-09-21T10:46:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:46:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:46:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:46:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:46:08.470384+0000", 2024-09-21T10:46:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013649699999999999 2024-09-21T10:46:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:46:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.105 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:46:09.105 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:46:09.105 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:46:09.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:09.105 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:09.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:09.206 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:09.227 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:09.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:09.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:09.329 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:09.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:09.363 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:09.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:09.463 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:09.489 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:09.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:09.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:09.591 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:09.626 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:09.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:09.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:09.728 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:09.758 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:09.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:09.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:09.860 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:09.900 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:09.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:09.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:10.000 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 82 16 121536 121520 92.6081 59.6875 0.0187142 0.0107938 2024-09-21T10:46:10.002 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:10.025 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:46:10.025 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:46:10.025 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-21T10:46:10.026 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:46:10.026 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:46:10.026 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-21T10:46:10.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:10.028 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:10.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:10.129 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:10.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:10.162 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:10.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:10.263 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:10.297 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:10.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:10.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:10.400 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:10.445 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:10.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:10.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:10.547 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:10.574 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:10.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:10.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:10 smithi160 ceph-mon[101879]: pgmap v2406: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 60 MiB/s wr, 966 op/s 2024-09-21T10:46:10.634 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:46:10.676 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:10.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:10.721 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:10.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:10.822 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:10.831 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:10 smithi070 ceph-mon[112139]: pgmap v2406: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 60 MiB/s wr, 966 op/s 2024-09-21T10:46:10.831 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:10 smithi070 ceph-mon[107385]: pgmap v2406: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 60 MiB/s wr, 966 op/s 2024-09-21T10:46:10.860 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:10.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:10.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:10.961 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:11.000 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 83 16 122464 122448 92.1911 58 0.0133648 0.0108424 2024-09-21T10:46:11.001 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:11.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:11.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:11.102 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:11.112 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:46:11.139 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:11.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:11.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:11.182 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:46:11.214 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:11.241 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:11.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:11.298 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:11.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:11.398 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:11.433 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:11.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:11.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:11.464 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T10:46:11.535 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:11.576 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:11.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:11.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:11.678 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:11.716 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:11.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:11.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:11.818 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:11.835 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:11 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2188607049' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:46:11.835 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:11 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:46:11.835 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:11 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2188607049' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:46:11.835 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:11 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:46:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:11 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2188607049' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:46:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:11 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:46:11.847 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:11.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:11.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:11.949 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:11.982 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:11.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:11.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:12.000 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 84 16 123378 123362 91.7736 57.125 0.0101376 0.0108919 2024-09-21T10:46:12.084 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:12.112 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:12.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:12.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:12.213 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:12.255 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:12.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:12.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:12.356 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:12.398 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:12.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:12.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:12.499 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:12.528 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:12.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:12.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:12 smithi070 ceph-mon[107385]: pgmap v2407: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 61 MiB/s wr, 973 op/s 2024-09-21T10:46:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:12 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:46:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:12 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:46:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:12 smithi070 ceph-mon[107385]: osdmap e1389: 8 total, 8 up, 7 in 2024-09-21T10:46:12.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:12 smithi070 ceph-mon[112139]: pgmap v2407: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 61 MiB/s wr, 973 op/s 2024-09-21T10:46:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:12 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:46:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:12 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:46:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:12 smithi070 ceph-mon[112139]: osdmap e1389: 8 total, 8 up, 7 in 2024-09-21T10:46:12.630 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:12.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:12.662 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:12.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:12.763 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:12.793 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:12 smithi160 ceph-mon[101879]: pgmap v2407: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 61 MiB/s wr, 973 op/s 2024-09-21T10:46:12.793 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:12 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:46:12.793 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:12 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:46:12.793 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:12 smithi160 ceph-mon[101879]: osdmap e1389: 8 total, 8 up, 7 in 2024-09-21T10:46:12.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:12.825 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:12.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:12.926 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:12.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:12.967 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:12.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:13.000 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 85 16 124343 124327 91.4034 60.3125 0.0235778 0.0109354 2024-09-21T10:46:13.068 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:13.102 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:13.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:13.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:13.203 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:13.241 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:13.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:13.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:13.322 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:13.343 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:13.381 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:13.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:13.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:13.426 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:46:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:46:13] "GET /metrics HTTP/1.1" 200 36605 "" "Prometheus/2.51.0" 2024-09-21T10:46:13.483 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:13.514 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:13.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:13.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:13.555 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:46:13.555 INFO:teuthology.orchestra.run.smithi160.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T10:46:13.555 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:46:13.615 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:13.649 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:13.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:13.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:13.751 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:13.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:13.787 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:13.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:13.888 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:13.923 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:13.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:13.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:14.000 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 86 16 125285 125269 91.0252 58.875 0.0244648 0.0109818 2024-09-21T10:46:14.021 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:46:14.021 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:46:14.025 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:14.056 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:46:14.056 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:46:14.060 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:14.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:14.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:14.091 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:46:14.092 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:46:14.119 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:46:14.119 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:46:14.156 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:46:14.156 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:46:14.161 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:14.189 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:46:14.190 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:46:14.191 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:14.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:14.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:14.227 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:46:14.227 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:46:14.258 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:46:14.258 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:46:14.293 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:14.332 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:14.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:14.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:14.434 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:14.471 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:14.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:14.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:14.572 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:14.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:14 smithi160 ceph-mon[101879]: pgmap v2409: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-21T10:46:14.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:14 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2189504499' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:14.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:14.616 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:14.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:14.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:14 smithi070 ceph-mon[112139]: pgmap v2409: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-21T10:46:14.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:14 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2189504499' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:14 smithi070 ceph-mon[107385]: pgmap v2409: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-21T10:46:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:14 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2189504499' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:14.700 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:46:14.718 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:14.780 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:14.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:14.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:14.882 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:14.922 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:14.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:14.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:15.001 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 87 16 126239 126223 90.6639 59.625 0.00753874 0.0110252 2024-09-21T10:46:15.024 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:15.048 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:15.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:15.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:15.150 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:15.187 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:15.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:15.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:15.289 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:15.321 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:15.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:15.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:15.423 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:15.458 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:15.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:15.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:15.560 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:15.592 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:15.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:15.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:15.693 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:15.721 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:15.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:15.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:15.823 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:15.874 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:15.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:15.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:15.917 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:15.977 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:16.003 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 88 16 127189 127173 90.3065 59.375 0.021565 0.0110684 2024-09-21T10:46:16.008 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:16.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:16.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:16.110 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:16.144 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:16.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:16.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:16.246 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:16.286 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:16.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:16.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:16.388 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:16.421 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:16.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:16.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:16.523 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:16.558 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:16.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:16.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:16.660 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:16.692 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:16.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:16.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:16.794 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:16.830 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:16.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:16.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:16.837 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:16 smithi070 ceph-mon[107385]: pgmap v2410: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 60 MiB/s wr, 954 op/s 2024-09-21T10:46:16.837 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:16 smithi070 ceph-mon[112139]: pgmap v2410: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 60 MiB/s wr, 954 op/s 2024-09-21T10:46:16.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:16 smithi160 ceph-mon[101879]: pgmap v2410: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 60 MiB/s wr, 954 op/s 2024-09-21T10:46:16.932 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:16.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:16.973 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:16.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:17.003 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 89 16 128154 128138 89.9694 60.3125 0.00973554 0.01111 2024-09-21T10:46:17.073 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:17.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:17.111 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:17.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:17.211 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:17.249 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:17.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:17.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:17.255 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:46:17.256 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:46:17.351 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:17.391 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:17.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:17.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:17.425 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:46:17.493 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:17.531 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:17.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:17.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:17.632 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:17.675 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:17.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:17.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:17.777 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:17.815 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:17.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:17.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:17.917 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:17.951 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:17.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:17.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:18.003 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Total time run: 90.0145 2024-09-21T10:46:18.003 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Total writes made: 129110 2024-09-21T10:46:18.003 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Write size: 65536 2024-09-21T10:46:18.003 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Object size: 65536 2024-09-21T10:46:18.003 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Bandwidth (MB/sec): 89.6453 2024-09-21T10:46:18.003 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Stddev Bandwidth: 41.3397 2024-09-21T10:46:18.003 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Max bandwidth (MB/sec): 249.875 2024-09-21T10:46:18.003 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Min bandwidth (MB/sec): 54.625 2024-09-21T10:46:18.003 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Average IOPS: 1434 2024-09-21T10:46:18.003 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Stddev IOPS: 661.435 2024-09-21T10:46:18.003 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Max IOPS: 3998 2024-09-21T10:46:18.004 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Min IOPS: 874 2024-09-21T10:46:18.004 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Average Latency(s): 0.0111506 2024-09-21T10:46:18.004 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Stddev Latency(s): 0.00842536 2024-09-21T10:46:18.004 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Max latency(s): 0.178279 2024-09-21T10:46:18.004 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Min latency(s): 0.00136946 2024-09-21T10:46:18.009 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T10:46:18.053 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:18.089 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:18.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:18.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:18.190 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:18.223 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:18.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:18.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:18.324 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:18.357 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:18.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:18.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:18.458 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:18.486 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:18.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:18.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:18.588 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:18.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:18.619 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:18.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:18.720 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:18.754 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:18.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:18.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:18 smithi160 ceph-mon[101879]: pgmap v2411: 89 pgs: 89 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 60 MiB/s wr, 955 op/s 2024-09-21T10:46:18.856 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:18.872 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:18 smithi070 ceph-mon[107385]: pgmap v2411: 89 pgs: 89 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 60 MiB/s wr, 955 op/s 2024-09-21T10:46:18.872 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:18 smithi070 ceph-mon[112139]: pgmap v2411: 89 pgs: 89 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 60 MiB/s wr, 955 op/s 2024-09-21T10:46:18.883 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:18.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:18.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:18.906 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:46:18.986 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:19.019 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:19.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:19.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:19.120 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:19.140 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:19.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:19.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:19.242 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:19.275 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:46:19.275 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:46:19.275 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-21T10:46:19.275 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:46:19.275 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:46:19.275 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-21T10:46:19.283 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:19.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:19.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:19.385 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:19.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:19.415 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:19.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:19.516 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:19.549 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:19.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:19.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:19.651 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:19.681 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:19.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:19.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:19.782 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:19.815 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:19.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:19.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:19.917 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:19.947 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:19.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:19.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:20.049 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:20.086 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:20.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:20.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:20.188 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:20.223 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:20.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:20.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:20.325 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:20.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:20.350 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:20.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:20.451 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:20.492 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:20.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:20.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:20.594 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:20.622 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:20.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:20.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:20.724 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:20.760 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:20.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:20.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:20 smithi160 ceph-mon[101879]: pgmap v2412: 89 pgs: 89 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 60 MiB/s wr, 955 op/s 2024-09-21T10:46:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:46:20.863 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:20.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:20 smithi070 ceph-mon[107385]: pgmap v2412: 89 pgs: 89 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 60 MiB/s wr, 955 op/s 2024-09-21T10:46:20.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:46:20.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:20 smithi070 ceph-mon[112139]: pgmap v2412: 89 pgs: 89 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 60 MiB/s wr, 955 op/s 2024-09-21T10:46:20.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:46:20.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:20.894 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:20.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:20.995 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:21.019 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:21.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:21.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:21.121 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:21.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:21.149 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:21.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:21.250 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:21.282 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:21.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:21.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:21.384 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:21.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:21.417 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:21.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:21.518 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:21.552 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:21.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:21.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:21.653 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:21.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:21.682 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:21.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:21.750 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:21.783 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:21.813 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:21.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:21.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:21.915 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:21.951 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:21.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:21.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:22.052 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:22.088 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:22.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:22.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:22.190 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:22.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:22.230 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:22.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:22.331 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:22.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:22.374 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:22.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:22.475 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:22.512 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:22.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:22.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:22.526 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:22.613 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:22.638 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:22.653 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:22.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:22.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:22.737 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:22 smithi070 ceph-mon[107385]: pgmap v2413: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 54 MiB/s wr, 954 op/s 2024-09-21T10:46:22.738 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:22 smithi070 ceph-mon[112139]: pgmap v2413: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 54 MiB/s wr, 954 op/s 2024-09-21T10:46:22.755 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:22.792 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:22.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:22.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:22.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:22 smithi160 ceph-mon[101879]: pgmap v2413: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 54 MiB/s wr, 954 op/s 2024-09-21T10:46:22.894 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:22.923 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:22.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:22.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:23.024 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:23.058 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:23.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:23.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:23.160 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:23.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:23.201 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:23.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:23.302 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:23.331 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:23.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:23.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:23.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:46:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:46:23] "GET /metrics HTTP/1.1" 200 36599 "" "Prometheus/2.51.0" 2024-09-21T10:46:23.433 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:23.469 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:23.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:23.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:23.501 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T10:46:23.570 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:23.602 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:23.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:23.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:23.703 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:23.747 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:23 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2090064667' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:23.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:23 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:23.748 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:23 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2090064667' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:23.748 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:23 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:23.749 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:23.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:23.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:23 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2090064667' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:23 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:23.851 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:23.876 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:23.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:23.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:23.978 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:24.001 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:24.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:24.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:24.103 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:24.140 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:24.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:24.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:24.242 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:24.287 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:24.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:24.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:24.388 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:24.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:24.422 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:24.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:24.523 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:24.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:24.558 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:24.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:24.659 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:24.691 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:24 smithi070 ceph-mon[107385]: pgmap v2414: 89 pgs: 89 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 90 B/s rd, 40 MiB/s wr, 854 op/s 2024-09-21T10:46:24.691 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:24 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:46:24.691 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:24 smithi070 ceph-mon[107385]: Cluster is now healthy 2024-09-21T10:46:24.691 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:24 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:46:24.691 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:24 smithi070 ceph-mon[107385]: osdmap e1390: 8 total, 8 up, 7 in 2024-09-21T10:46:24.691 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:24 smithi070 ceph-mon[112139]: pgmap v2414: 89 pgs: 89 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 90 B/s rd, 40 MiB/s wr, 854 op/s 2024-09-21T10:46:24.692 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:24 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:46:24.692 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:24 smithi070 ceph-mon[112139]: Cluster is now healthy 2024-09-21T10:46:24.692 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:24 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:46:24.692 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:24 smithi070 ceph-mon[112139]: osdmap e1390: 8 total, 8 up, 7 in 2024-09-21T10:46:24.697 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:24.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:24.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:24.798 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:24.829 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:24.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:24.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:24 smithi160 ceph-mon[101879]: pgmap v2414: 89 pgs: 89 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 90 B/s rd, 40 MiB/s wr, 854 op/s 2024-09-21T10:46:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:24 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:46:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:24 smithi160 ceph-mon[101879]: Cluster is now healthy 2024-09-21T10:46:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:24 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:46:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:24 smithi160 ceph-mon[101879]: osdmap e1390: 8 total, 8 up, 7 in 2024-09-21T10:46:24.931 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:24.973 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:24.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:24.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:25.075 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:25.107 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:25.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:25.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:25.210 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:25.251 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:25.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:25.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:25.353 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:25.388 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:25.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:25.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:25.489 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:25.530 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:25.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:25.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:25.632 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:25.664 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:25.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:25.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:25.766 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:25.800 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:25.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:25.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:25.901 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:25.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:25.937 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:25.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:26.038 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:26.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:26.075 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:26.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:26.176 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:26.207 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:26.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:26.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:26.309 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:26.350 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:26.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:26.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:26.451 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:26.480 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:26.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:26.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:26.581 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:26.617 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:26.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:26.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:26.719 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:26.745 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:26.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:26.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:26.776 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:26 smithi070 ceph-mon[107385]: pgmap v2416: 89 pgs: 89 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 102 B/s rd, 32 MiB/s wr, 812 op/s 2024-09-21T10:46:26.777 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:26 smithi070 ceph-mon[112139]: pgmap v2416: 89 pgs: 89 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 102 B/s rd, 32 MiB/s wr, 812 op/s 2024-09-21T10:46:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:26 smithi160 ceph-mon[101879]: pgmap v2416: 89 pgs: 89 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 102 B/s rd, 32 MiB/s wr, 812 op/s 2024-09-21T10:46:26.847 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:26.879 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:26.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:26.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:26.981 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:27.014 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:27.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:27.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:27.026 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:46:27.026 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "" 2024-09-21T10:46:27.027 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:46:27.116 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:27.147 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:27.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:27.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:27.178 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:46:27.249 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:27.313 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:27.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:27.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:27.414 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:27.445 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:27.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:27.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:27.546 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:27.582 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:27.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:27.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:27.683 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:27.717 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:27.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:27.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:27.761 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:46:27.761 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:46:27.819 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:27.854 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:27.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:27.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:27.956 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:27.988 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:27.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:27.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:28.090 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:28.122 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:28.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:28.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:28.223 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:28.258 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:28.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:28.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:28.359 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:28.394 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:28.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:28.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:28.496 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:28.533 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:28.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:28.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:28.635 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:28.667 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:28.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:28.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:28.768 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:28.798 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:28.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:28.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:28 smithi160 ceph-mon[101879]: pgmap v2417: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 15 MiB/s wr, 956 op/s 2024-09-21T10:46:28.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:28 smithi070 ceph-mon[107385]: pgmap v2417: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 15 MiB/s wr, 956 op/s 2024-09-21T10:46:28.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:28 smithi070 ceph-mon[112139]: pgmap v2417: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 15 MiB/s wr, 956 op/s 2024-09-21T10:46:28.900 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:28.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:28.937 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:28.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:29.038 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:29.045 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:46:29.073 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:29.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:29.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:29.175 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:29.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:29.210 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:29.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:29.311 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:29.354 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:29.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:29.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:29.416 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:29.456 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:29.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:29.497 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:29.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:29.598 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:29.638 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:29.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:29.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:29.740 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:29.772 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:29.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:29.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:29.874 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:29.903 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:29.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:29.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:29.905 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:30.004 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:30.052 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:30.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:30.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:30.153 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:30.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:30.189 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:30.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:30.290 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:30.318 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:30.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:30.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:30.420 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:30.459 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:30.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:30.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:30.560 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:30.598 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:30.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:30.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:30.700 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:30.744 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:30.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:30.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:30 smithi160 ceph-mon[101879]: pgmap v2418: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 15 MiB/s wr, 956 op/s 2024-09-21T10:46:30.845 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:30.876 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:30.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:30.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:30.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:30 smithi070 ceph-mon[107385]: pgmap v2418: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 15 MiB/s wr, 956 op/s 2024-09-21T10:46:30.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:30 smithi070 ceph-mon[112139]: pgmap v2418: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 15 MiB/s wr, 956 op/s 2024-09-21T10:46:30.978 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:31.018 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:31.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:31.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:31.119 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:31.157 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:31.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:31.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:31.259 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:31.292 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:31.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:31.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:31.394 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:31.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:31.433 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:31.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:31.534 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:31.571 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:31.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:31.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:31.672 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:31.708 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:31.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:31.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:31.810 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:31.847 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:31.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:31.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:31.880 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:46:31.880 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:46:31.911 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:46:31.912 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:46:31.942 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:46:31.943 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:46:31.948 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:31.973 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:46:31.973 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:46:31.992 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:31.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:31.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:32.004 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:46:32.005 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:46:32.034 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:46:32.034 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:46:32.069 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:46:32.070 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:46:32.094 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:32.104 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:46:32.105 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:46:32.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:32.136 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:32.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:32.237 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:32.279 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:32.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:32.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:32.381 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:32.415 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:32.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:32.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:32.516 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:32.555 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:32.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:32.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:32.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:32 smithi070 ceph-mon[107385]: pgmap v2419: 89 pgs: 89 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 5.8 MiB/s wr, 967 op/s 2024-09-21T10:46:32.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:32 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/16453741' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:32.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:32 smithi070 ceph-mon[112139]: pgmap v2419: 89 pgs: 89 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 5.8 MiB/s wr, 967 op/s 2024-09-21T10:46:32.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:32 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/16453741' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:32.657 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:32.684 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:32.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:32.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:32.786 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:32.818 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:32.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:32.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:32 smithi160 ceph-mon[101879]: pgmap v2419: 89 pgs: 89 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 5.8 MiB/s wr, 967 op/s 2024-09-21T10:46:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:32 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/16453741' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:32.920 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:32.948 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:32.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:32.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:33.050 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:33.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:33.093 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:33.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:33.194 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:33.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:33.311 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:33.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:33.311 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:46:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:46:33] "GET /metrics HTTP/1.1" 200 36599 "" "Prometheus/2.51.0" 2024-09-21T10:46:33.412 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:33.441 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:33.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:33.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:33.537 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:46:33.537 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T10:46:33.543 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:33.578 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:33.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:33.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:33.681 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:33.725 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:33.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:33.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:33.826 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:33.858 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:33.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:33.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:33.960 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:33.999 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:33.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:34.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:34.100 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:34.135 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:34.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:34.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:34.237 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:34.269 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:34.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:34.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:34.371 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:34.407 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:34.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:34.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:34.509 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:34.522 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T10:46:34.549 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:34 smithi070 ceph-mon[112139]: pgmap v2420: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 972 op/s 2024-09-21T10:46:34.549 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:34 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2824599537' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:46:34.550 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:34 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:46:34.550 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:34 smithi070 ceph-mon[107385]: pgmap v2420: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 972 op/s 2024-09-21T10:46:34.550 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:34 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2824599537' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:46:34.550 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:34 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:46:34.554 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:34.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:34.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:34.576 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:46:34.576 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T10:46:34.576 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T10:46:34.576 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T10:46:34.576 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.576 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77348 53.10 53:0a4ac980:::benchmark_data_smithi070_228967_object77347:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:46:34.576 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.666247+0000", 2024-09-21T10:46:34.576 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 328.903446906, 2024-09-21T10:46:34.576 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1333923189999999, 2024-09-21T10:46:34.577 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:46:34.577 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:46:34.577 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:34.577 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:46:34.577 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:46:34.577 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:46:34.577 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77348 2024-09-21T10:46:34.577 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.577 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:46:34.577 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.577 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:46:34.577 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.666247+0000", 2024-09-21T10:46:34.577 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.578 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.578 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.578 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:46:34.578 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.666247+0000", 2024-09-21T10:46:34.578 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.578 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.578 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.578 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:46:34.578 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.666249+0000", 2024-09-21T10:46:34.578 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.432e-06 2024-09-21T10:46:34.578 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.578 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.579 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:46:34.579 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.666267+0000", 2024-09-21T10:46:34.579 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7736000000000001e-05 2024-09-21T10:46:34.579 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.579 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.579 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:46:34.579 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.666278+0000", 2024-09-21T10:46:34.579 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1112e-05 2024-09-21T10:46:34.579 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.579 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.579 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:46:34.580 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.666286+0000", 2024-09-21T10:46:34.580 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9820000000000008e-06 2024-09-21T10:46:34.580 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.580 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.580 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.580 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.666343+0000", 2024-09-21T10:46:34.580 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7099000000000003e-05 2024-09-21T10:46:34.580 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.580 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.580 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.580 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.666353+0000", 2024-09-21T10:46:34.580 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.6590000000000002e-06 2024-09-21T10:46:34.580 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.581 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.581 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.581 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659584+0000", 2024-09-21T10:46:34.581 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99323149799999999 2024-09-21T10:46:34.581 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.581 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.581 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.581 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659596+0000", 2024-09-21T10:46:34.581 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2294e-05 2024-09-21T10:46:34.581 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.581 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.581 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.581 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.795107+0000", 2024-09-21T10:46:34.581 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1355107019999999 2024-09-21T10:46:34.581 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.581 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.582 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:46:34.582 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.795298+0000", 2024-09-21T10:46:34.582 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000190893 2024-09-21T10:46:34.582 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.582 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.582 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:34.582 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.795500+0000", 2024-09-21T10:46:34.582 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000202084 2024-09-21T10:46:34.582 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.582 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.582 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.582 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.798730+0000", 2024-09-21T10:46:34.583 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00323025 2024-09-21T10:46:34.583 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.583 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.583 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:46:34.583 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.798743+0000", 2024-09-21T10:46:34.583 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3009e-05 2024-09-21T10:46:34.583 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.583 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.583 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.583 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.798773+0000", 2024-09-21T10:46:34.583 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9974000000000001e-05 2024-09-21T10:46:34.583 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.583 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.584 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:46:34.584 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.798843+0000", 2024-09-21T10:46:34.584 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9523999999999994e-05 2024-09-21T10:46:34.584 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.584 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.584 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:46:34.584 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.799640+0000", 2024-09-21T10:46:34.584 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000797071 2024-09-21T10:46:34.584 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.584 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:46:34.584 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.584 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.584 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.584 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77354 53.10 53:09eb2e30:::benchmark_data_smithi070_228967_object77353:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:46:34.585 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.669089+0000", 2024-09-21T10:46:34.585 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 328.90060537400001, 2024-09-21T10:46:34.585 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1315911619999999, 2024-09-21T10:46:34.585 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:46:34.585 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:46:34.585 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:34.585 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:46:34.585 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:46:34.585 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:46:34.585 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77354 2024-09-21T10:46:34.585 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.586 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:46:34.586 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.586 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:46:34.586 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.669089+0000", 2024-09-21T10:46:34.586 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.586 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.586 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.586 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:46:34.586 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.669089+0000", 2024-09-21T10:46:34.586 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.586 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.586 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.587 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:46:34.587 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.669091+0000", 2024-09-21T10:46:34.587 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7099999999999999e-06 2024-09-21T10:46:34.587 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.587 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.587 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:46:34.587 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.669118+0000", 2024-09-21T10:46:34.587 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7500000000000001e-05 2024-09-21T10:46:34.587 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.587 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.587 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:46:34.587 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.669137+0000", 2024-09-21T10:46:34.587 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8529000000000002e-05 2024-09-21T10:46:34.587 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.588 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.588 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:46:34.588 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.669145+0000", 2024-09-21T10:46:34.588 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6580000000000008e-06 2024-09-21T10:46:34.588 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.588 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.588 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.588 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.669213+0000", 2024-09-21T10:46:34.588 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7706000000000005e-05 2024-09-21T10:46:34.588 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.588 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.588 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.588 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.669223+0000", 2024-09-21T10:46:34.588 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0098999999999999e-05 2024-09-21T10:46:34.588 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.589 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.589 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.589 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659628+0000", 2024-09-21T10:46:34.589 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99040486100000003 2024-09-21T10:46:34.589 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.589 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.589 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.589 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659636+0000", 2024-09-21T10:46:34.589 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4449999999999999e-06 2024-09-21T10:46:34.589 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.589 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.589 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.589 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.797849+0000", 2024-09-21T10:46:34.590 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.138213956 2024-09-21T10:46:34.590 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.590 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.590 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:46:34.590 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.797958+0000", 2024-09-21T10:46:34.590 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010826400000000001 2024-09-21T10:46:34.590 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.590 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.590 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:34.590 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.798068+0000", 2024-09-21T10:46:34.590 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011027399999999999 2024-09-21T10:46:34.590 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.590 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.590 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.591 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.799784+0000", 2024-09-21T10:46:34.591 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001716239 2024-09-21T10:46:34.591 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.591 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.591 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.591 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.799842+0000", 2024-09-21T10:46:34.591 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7738999999999997e-05 2024-09-21T10:46:34.591 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.591 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.591 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:46:34.591 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.799858+0000", 2024-09-21T10:46:34.591 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5605000000000001e-05 2024-09-21T10:46:34.591 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.592 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.592 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:46:34.592 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.799915+0000", 2024-09-21T10:46:34.592 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7553999999999998e-05 2024-09-21T10:46:34.592 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.592 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.592 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:46:34.592 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.800680+0000", 2024-09-21T10:46:34.592 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00076502299999999998 2024-09-21T10:46:34.592 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.592 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:46:34.592 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.592 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.592 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.592 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77415 53.10 53:0eba5a74:::benchmark_data_smithi070_228967_object77414:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:46:34.593 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.700483+0000", 2024-09-21T10:46:34.593 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 328.869211474, 2024-09-21T10:46:34.593 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1042608860000001, 2024-09-21T10:46:34.593 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:46:34.593 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:46:34.593 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:34.593 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:46:34.593 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:46:34.593 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:46:34.593 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77415 2024-09-21T10:46:34.593 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.593 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:46:34.593 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.593 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:46:34.594 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700483+0000", 2024-09-21T10:46:34.594 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.594 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.594 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.594 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:46:34.594 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700483+0000", 2024-09-21T10:46:34.594 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.594 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.594 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.594 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:46:34.594 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700485+0000", 2024-09-21T10:46:34.594 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5880000000000001e-06 2024-09-21T10:46:34.594 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.594 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.595 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:46:34.595 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700506+0000", 2024-09-21T10:46:34.595 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1049000000000001e-05 2024-09-21T10:46:34.595 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.595 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.595 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:46:34.595 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700517+0000", 2024-09-21T10:46:34.595 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1603000000000001e-05 2024-09-21T10:46:34.595 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.595 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.595 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:46:34.595 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700524+0000", 2024-09-21T10:46:34.595 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1670000000000002e-06 2024-09-21T10:46:34.595 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.595 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.596 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.596 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700751+0000", 2024-09-21T10:46:34.596 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00022716300000000001 2024-09-21T10:46:34.596 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.596 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.596 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.596 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700762+0000", 2024-09-21T10:46:34.596 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0550999999999999e-05 2024-09-21T10:46:34.596 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.596 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.596 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.596 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659665+0000", 2024-09-21T10:46:34.597 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.95890305200000003 2024-09-21T10:46:34.597 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.597 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.597 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.597 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659673+0000", 2024-09-21T10:46:34.597 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4950000000000002e-06 2024-09-21T10:46:34.597 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.597 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.597 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.597 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.800708+0000", 2024-09-21T10:46:34.597 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.141035383 2024-09-21T10:46:34.597 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.597 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.597 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:46:34.597 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.800824+0000", 2024-09-21T10:46:34.598 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000116431 2024-09-21T10:46:34.598 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.598 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.598 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:34.598 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.800934+0000", 2024-09-21T10:46:34.598 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010966799999999999 2024-09-21T10:46:34.598 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.598 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.598 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.598 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.803154+0000", 2024-09-21T10:46:34.598 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0022197940000000002 2024-09-21T10:46:34.598 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.598 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.599 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.599 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.804093+0000", 2024-09-21T10:46:34.599 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00093908499999999998 2024-09-21T10:46:34.599 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.599 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.599 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:46:34.599 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.804102+0000", 2024-09-21T10:46:34.599 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5979999999999997e-06 2024-09-21T10:46:34.599 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.599 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.599 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:46:34.599 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.804136+0000", 2024-09-21T10:46:34.599 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4536999999999998e-05 2024-09-21T10:46:34.600 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.600 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.600 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:46:34.600 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.804744+0000", 2024-09-21T10:46:34.600 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00060772199999999999 2024-09-21T10:46:34.600 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.600 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:46:34.600 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.600 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.600 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.600 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77416 53.10 53:0b890937:::benchmark_data_smithi070_228967_object77415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:46:34.600 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.700546+0000", 2024-09-21T10:46:34.600 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 328.86914837500001, 2024-09-21T10:46:34.601 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1047850320000001, 2024-09-21T10:46:34.601 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:46:34.601 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:46:34.601 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:34.601 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:46:34.601 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:46:34.601 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:46:34.601 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77416 2024-09-21T10:46:34.601 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.601 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:46:34.601 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.601 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:46:34.601 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700546+0000", 2024-09-21T10:46:34.602 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.602 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.602 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.602 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:46:34.602 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700546+0000", 2024-09-21T10:46:34.602 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.602 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.602 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.602 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:46:34.602 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700547+0000", 2024-09-21T10:46:34.602 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0999999999999999e-07 2024-09-21T10:46:34.602 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.602 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.603 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:46:34.603 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700564+0000", 2024-09-21T10:46:34.603 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7252e-05 2024-09-21T10:46:34.603 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.603 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.603 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:46:34.603 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700572+0000", 2024-09-21T10:46:34.603 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9039999999999992e-06 2024-09-21T10:46:34.603 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.603 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.603 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:46:34.603 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700577+0000", 2024-09-21T10:46:34.603 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.699e-06 2024-09-21T10:46:34.604 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.604 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.604 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.604 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700788+0000", 2024-09-21T10:46:34.604 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00021084000000000001 2024-09-21T10:46:34.604 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.604 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.604 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.605 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700796+0000", 2024-09-21T10:46:34.605 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6090000000000003e-06 2024-09-21T10:46:34.605 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.605 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.605 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.605 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659707+0000", 2024-09-21T10:46:34.605 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.95891092 2024-09-21T10:46:34.605 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.605 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.606 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.606 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659711+0000", 2024-09-21T10:46:34.606 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3789999999999999e-06 2024-09-21T10:46:34.606 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.606 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.606 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.606 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.802199+0000", 2024-09-21T10:46:34.606 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.142488341 2024-09-21T10:46:34.606 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.606 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.606 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:46:34.606 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.802320+0000", 2024-09-21T10:46:34.607 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000120209 2024-09-21T10:46:34.607 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.607 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.607 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:34.607 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.802494+0000", 2024-09-21T10:46:34.607 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000174771 2024-09-21T10:46:34.607 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.607 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.607 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.607 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.804796+0000", 2024-09-21T10:46:34.607 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002301726 2024-09-21T10:46:34.607 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.607 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.608 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.608 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.804844+0000", 2024-09-21T10:46:34.608 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8266000000000002e-05 2024-09-21T10:46:34.608 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.608 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.608 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:46:34.608 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.804854+0000", 2024-09-21T10:46:34.608 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0018e-05 2024-09-21T10:46:34.608 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.608 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.608 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:46:34.608 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.804893+0000", 2024-09-21T10:46:34.608 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8430999999999998e-05 2024-09-21T10:46:34.609 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.609 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.609 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:46:34.609 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.805331+0000", 2024-09-21T10:46:34.609 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000438157 2024-09-21T10:46:34.609 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.609 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:46:34.609 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.609 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.609 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.609 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77417 53.10 53:0e921901:::benchmark_data_smithi070_228967_object77416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:46:34.609 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.700585+0000", 2024-09-21T10:46:34.609 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 328.869109208, 2024-09-21T10:46:34.609 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.118979521, 2024-09-21T10:46:34.610 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:46:34.610 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:46:34.610 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:34.610 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:46:34.610 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:46:34.610 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:46:34.610 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77417 2024-09-21T10:46:34.610 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.610 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:46:34.610 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.610 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:46:34.611 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700585+0000", 2024-09-21T10:46:34.611 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.611 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.611 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.611 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:46:34.611 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700585+0000", 2024-09-21T10:46:34.611 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.611 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.611 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.611 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:46:34.611 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700586+0000", 2024-09-21T10:46:34.611 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0400000000000002e-07 2024-09-21T10:46:34.611 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.611 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.612 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:46:34.612 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700605+0000", 2024-09-21T10:46:34.612 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9463999999999999e-05 2024-09-21T10:46:34.612 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.612 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.612 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:46:34.612 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700612+0000", 2024-09-21T10:46:34.612 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5019999999999998e-06 2024-09-21T10:46:34.612 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.612 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.612 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:46:34.612 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700617+0000", 2024-09-21T10:46:34.612 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9520000000000002e-06 2024-09-21T10:46:34.613 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.613 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.613 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.613 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700902+0000", 2024-09-21T10:46:34.613 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00028501699999999999 2024-09-21T10:46:34.613 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.613 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.613 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.613 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700910+0000", 2024-09-21T10:46:34.613 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1999999999999994e-06 2024-09-21T10:46:34.613 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.613 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.613 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.613 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659744+0000", 2024-09-21T10:46:34.614 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.95883443300000004 2024-09-21T10:46:34.614 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.614 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.614 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.614 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659753+0000", 2024-09-21T10:46:34.614 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.4379999999999995e-06 2024-09-21T10:46:34.614 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.614 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.614 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.614 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.809941+0000", 2024-09-21T10:46:34.614 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1501882539999999 2024-09-21T10:46:34.614 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.614 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.614 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:46:34.615 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.810050+0000", 2024-09-21T10:46:34.615 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010925999999999999 2024-09-21T10:46:34.615 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.615 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.615 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:34.615 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.810156+0000", 2024-09-21T10:46:34.615 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010574 2024-09-21T10:46:34.615 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.615 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.615 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:46:34.615 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.813660+0000", 2024-09-21T10:46:34.615 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035046370000000001 2024-09-21T10:46:34.615 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.615 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.615 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.616 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.813692+0000", 2024-09-21T10:46:34.616 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1769e-05 2024-09-21T10:46:34.616 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.616 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.616 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.616 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.818533+0000", 2024-09-21T10:46:34.616 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0048406539999999998 2024-09-21T10:46:34.616 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.616 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.616 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:46:34.616 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.818577+0000", 2024-09-21T10:46:34.616 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3915000000000002e-05 2024-09-21T10:46:34.616 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.616 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.616 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:46:34.617 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.819565+0000", 2024-09-21T10:46:34.617 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00098788200000000004 2024-09-21T10:46:34.617 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.617 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:46:34.617 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.617 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.617 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.617 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77499 53.10 53:0c45bed1:::benchmark_data_smithi070_228967_object77498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:46:34.617 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.763242+0000", 2024-09-21T10:46:34.617 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 328.80645193700002, 2024-09-21T10:46:34.617 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0610106340000001, 2024-09-21T10:46:34.617 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:46:34.617 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:46:34.617 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:34.618 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:46:34.618 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:46:34.618 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:46:34.618 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77499 2024-09-21T10:46:34.618 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.618 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:46:34.618 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.618 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:46:34.618 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.763242+0000", 2024-09-21T10:46:34.618 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.618 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.618 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.618 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:46:34.619 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.763242+0000", 2024-09-21T10:46:34.619 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.619 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.619 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.619 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:46:34.619 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.763244+0000", 2024-09-21T10:46:34.619 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.646e-06 2024-09-21T10:46:34.619 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.619 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.619 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:46:34.619 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.763268+0000", 2024-09-21T10:46:34.619 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4303000000000001e-05 2024-09-21T10:46:34.619 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.620 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.620 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:46:34.620 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.763282+0000", 2024-09-21T10:46:34.620 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4018e-05 2024-09-21T10:46:34.620 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.620 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.620 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:46:34.620 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.763290+0000", 2024-09-21T10:46:34.620 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1570000000000003e-06 2024-09-21T10:46:34.620 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.620 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.620 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.620 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.763365+0000", 2024-09-21T10:46:34.620 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5511000000000003e-05 2024-09-21T10:46:34.621 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.621 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.621 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.621 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.763387+0000", 2024-09-21T10:46:34.621 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1639000000000001e-05 2024-09-21T10:46:34.621 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.621 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.621 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.621 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659776+0000", 2024-09-21T10:46:34.621 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.89638960999999995 2024-09-21T10:46:34.621 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.621 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.621 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.621 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659780+0000", 2024-09-21T10:46:34.621 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8890000000000004e-06 2024-09-21T10:46:34.622 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.622 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.622 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.622 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.810900+0000", 2024-09-21T10:46:34.622 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1511196960000001 2024-09-21T10:46:34.622 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.622 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.622 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:46:34.622 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.810983+0000", 2024-09-21T10:46:34.622 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2664999999999995e-05 2024-09-21T10:46:34.622 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.622 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.623 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:34.623 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.811076+0000", 2024-09-21T10:46:34.623 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3386000000000003e-05 2024-09-21T10:46:34.623 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.623 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.623 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.623 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.813728+0000", 2024-09-21T10:46:34.623 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0026517609999999999 2024-09-21T10:46:34.623 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.623 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.623 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:46:34.623 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.813804+0000", 2024-09-21T10:46:34.623 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6404999999999998e-05 2024-09-21T10:46:34.624 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.624 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.624 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.624 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.823202+0000", 2024-09-21T10:46:34.624 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0093975829999999993 2024-09-21T10:46:34.624 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.624 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.624 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:46:34.624 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.823265+0000", 2024-09-21T10:46:34.624 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3585999999999997e-05 2024-09-21T10:46:34.624 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.624 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.624 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:46:34.624 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.824253+0000", 2024-09-21T10:46:34.624 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00098777900000000004 2024-09-21T10:46:34.625 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.625 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:46:34.625 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.625 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.625 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.625 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77507 53.10 53:0c3b2271:::benchmark_data_smithi070_228967_object77506:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:46:34.625 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.767593+0000", 2024-09-21T10:46:34.625 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 328.80210141999999, 2024-09-21T10:46:34.625 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0575902039999998, 2024-09-21T10:46:34.625 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:46:34.625 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:46:34.625 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:34.625 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:46:34.625 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:46:34.625 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:46:34.625 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77507 2024-09-21T10:46:34.626 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.626 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:46:34.626 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.626 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:46:34.626 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.767593+0000", 2024-09-21T10:46:34.626 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.626 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.626 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.626 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:46:34.626 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.767593+0000", 2024-09-21T10:46:34.626 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.626 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.626 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.626 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:46:34.627 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.767595+0000", 2024-09-21T10:46:34.627 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5880000000000001e-06 2024-09-21T10:46:34.627 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.627 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.627 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:46:34.627 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.767643+0000", 2024-09-21T10:46:34.627 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8563000000000002e-05 2024-09-21T10:46:34.627 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.627 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.627 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:46:34.627 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.767659+0000", 2024-09-21T10:46:34.627 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6186000000000001e-05 2024-09-21T10:46:34.627 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.627 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:46:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.767667+0000", 2024-09-21T10:46:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2200000000000003e-06 2024-09-21T10:46:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.767727+0000", 2024-09-21T10:46:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0702999999999998e-05 2024-09-21T10:46:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.767737+0000", 2024-09-21T10:46:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.6050000000000004e-06 2024-09-21T10:46:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.629 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.629 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659809+0000", 2024-09-21T10:46:34.629 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.89207242600000003 2024-09-21T10:46:34.629 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.629 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.629 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.629 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659817+0000", 2024-09-21T10:46:34.629 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5440000000000001e-06 2024-09-21T10:46:34.629 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.629 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.629 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.629 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.811691+0000", 2024-09-21T10:46:34.630 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1518743419999999 2024-09-21T10:46:34.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.630 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.630 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:46:34.630 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.811779+0000", 2024-09-21T10:46:34.630 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.8119000000000001e-05 2024-09-21T10:46:34.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.630 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.630 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:34.630 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.811886+0000", 2024-09-21T10:46:34.630 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000106805 2024-09-21T10:46:34.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.630 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.630 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.813784+0000", 2024-09-21T10:46:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00189806 2024-09-21T10:46:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:46:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.813814+0000", 2024-09-21T10:46:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0090000000000002e-05 2024-09-21T10:46:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.824353+0000", 2024-09-21T10:46:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.010538403 2024-09-21T10:46:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.632 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:46:34.632 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.824440+0000", 2024-09-21T10:46:34.632 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7063000000000003e-05 2024-09-21T10:46:34.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.632 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:46:34.632 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.825183+0000", 2024-09-21T10:46:34.632 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00074348700000000003 2024-09-21T10:46:34.632 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.632 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:46:34.632 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.633 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.633 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.633 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77522 53.10 53:0d13612e:::benchmark_data_smithi070_228967_object77521:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:46:34.633 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.778011+0000", 2024-09-21T10:46:34.633 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 328.79168342899999, 2024-09-21T10:46:34.633 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0483339169999999, 2024-09-21T10:46:34.633 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:46:34.633 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:46:34.633 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:34.634 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:46:34.634 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:46:34.634 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:46:34.634 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77522 2024-09-21T10:46:34.634 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.634 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:46:34.634 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.634 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:46:34.634 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.778011+0000", 2024-09-21T10:46:34.635 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.635 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.635 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:46:34.635 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.778011+0000", 2024-09-21T10:46:34.635 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.635 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.635 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:46:34.635 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.778013+0000", 2024-09-21T10:46:34.635 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9970000000000001e-06 2024-09-21T10:46:34.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.636 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:46:34.636 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.778037+0000", 2024-09-21T10:46:34.636 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4521000000000001e-05 2024-09-21T10:46:34.636 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.636 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:46:34.636 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.778050+0000", 2024-09-21T10:46:34.636 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2578999999999999e-05 2024-09-21T10:46:34.636 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.636 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:46:34.636 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.778058+0000", 2024-09-21T10:46:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2320000000000001e-06 2024-09-21T10:46:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.778124+0000", 2024-09-21T10:46:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5396999999999997e-05 2024-09-21T10:46:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.778133+0000", 2024-09-21T10:46:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2730000000000008e-06 2024-09-21T10:46:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.638 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.638 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659837+0000", 2024-09-21T10:46:34.638 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.88170432399999998 2024-09-21T10:46:34.638 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.638 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.638 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.638 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659841+0000", 2024-09-21T10:46:34.638 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7290000000000002e-06 2024-09-21T10:46:34.638 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.638 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.638 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.638 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.812569+0000", 2024-09-21T10:46:34.639 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.152727949 2024-09-21T10:46:34.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.639 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.639 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:46:34.639 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.812704+0000", 2024-09-21T10:46:34.639 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000135019 2024-09-21T10:46:34.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.639 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.639 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:34.639 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.812863+0000", 2024-09-21T10:46:34.639 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015892200000000001 2024-09-21T10:46:34.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.639 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.639 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.639 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.815571+0000", 2024-09-21T10:46:34.640 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0027076460000000002 2024-09-21T10:46:34.640 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.640 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.640 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:46:34.640 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.815618+0000", 2024-09-21T10:46:34.640 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7732000000000003e-05 2024-09-21T10:46:34.640 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.640 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.640 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.640 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.825288+0000", 2024-09-21T10:46:34.640 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0096699300000000002 2024-09-21T10:46:34.640 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.640 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.640 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:46:34.640 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.825349+0000", 2024-09-21T10:46:34.641 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0720999999999997e-05 2024-09-21T10:46:34.641 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.641 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.641 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:46:34.641 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.826345+0000", 2024-09-21T10:46:34.641 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00099594599999999994 2024-09-21T10:46:34.641 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.641 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:46:34.641 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.641 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.641 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.641 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77550 53.10 53:0c033939:::benchmark_data_smithi070_228967_object77549:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:46:34.641 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.799776+0000", 2024-09-21T10:46:34.641 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 328.76991887399998, 2024-09-21T10:46:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0288999130000001, 2024-09-21T10:46:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:46:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:46:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:46:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:46:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:46:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77550 2024-09-21T10:46:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:46:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:46:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.799776+0000", 2024-09-21T10:46:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.643 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.643 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:46:34.643 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.799776+0000", 2024-09-21T10:46:34.643 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.643 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.643 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:46:34.643 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.799776+0000", 2024-09-21T10:46:34.643 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7599999999999997e-07 2024-09-21T10:46:34.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.643 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.643 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:46:34.644 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.799824+0000", 2024-09-21T10:46:34.644 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7649999999999999e-05 2024-09-21T10:46:34.644 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.644 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.644 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:46:34.644 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.799831+0000", 2024-09-21T10:46:34.644 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1330000000000001e-06 2024-09-21T10:46:34.644 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.644 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.644 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:46:34.644 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.799838+0000", 2024-09-21T10:46:34.644 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6270000000000002e-06 2024-09-21T10:46:34.644 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.799898+0000", 2024-09-21T10:46:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0849e-05 2024-09-21T10:46:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.799907+0000", 2024-09-21T10:46:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1500000000000005e-06 2024-09-21T10:46:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659869+0000", 2024-09-21T10:46:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.85996196300000005 2024-09-21T10:46:34.646 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.646 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.646 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.646 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659877+0000", 2024-09-21T10:46:34.646 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.255e-06 2024-09-21T10:46:34.646 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.646 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.646 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.646 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.813839+0000", 2024-09-21T10:46:34.646 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.153962089 2024-09-21T10:46:34.646 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.646 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.646 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:46:34.647 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.813940+0000", 2024-09-21T10:46:34.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000100719 2024-09-21T10:46:34.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.647 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:34.647 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.814072+0000", 2024-09-21T10:46:34.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000132065 2024-09-21T10:46:34.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.648 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.648 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.815610+0000", 2024-09-21T10:46:34.648 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0015380389999999999 2024-09-21T10:46:34.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.648 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:46:34.648 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.815622+0000", 2024-09-21T10:46:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2714000000000001e-05 2024-09-21T10:46:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.828115+0000", 2024-09-21T10:46:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012492207999999999 2024-09-21T10:46:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:46:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.828184+0000", 2024-09-21T10:46:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9218e-05 2024-09-21T10:46:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:46:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.828675+0000", 2024-09-21T10:46:34.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00049165799999999996 2024-09-21T10:46:34.650 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.650 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:46:34.650 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.650 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77552 53.10 53:092395bc:::benchmark_data_smithi070_228967_object77551:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:46:34.650 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.801039+0000", 2024-09-21T10:46:34.650 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 328.76865538800001, 2024-09-21T10:46:34.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0284018509999999, 2024-09-21T10:46:34.650 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:46:34.650 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:46:34.650 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:34.651 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:46:34.651 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:46:34.651 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:46:34.651 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77552 2024-09-21T10:46:34.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.651 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:46:34.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.651 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:46:34.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.801039+0000", 2024-09-21T10:46:34.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.651 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:46:34.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.801039+0000", 2024-09-21T10:46:34.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.652 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.652 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:46:34.652 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.801040+0000", 2024-09-21T10:46:34.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.184e-06 2024-09-21T10:46:34.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.652 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.652 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:46:34.652 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.801089+0000", 2024-09-21T10:46:34.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9313e-05 2024-09-21T10:46:34.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:46:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.801103+0000", 2024-09-21T10:46:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3816e-05 2024-09-21T10:46:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:46:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.801111+0000", 2024-09-21T10:46:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5820000000000004e-06 2024-09-21T10:46:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.801164+0000", 2024-09-21T10:46:34.654 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3362000000000002e-05 2024-09-21T10:46:34.654 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.654 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.654 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.654 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.801173+0000", 2024-09-21T10:46:34.654 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6649999999999992e-06 2024-09-21T10:46:34.654 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.654 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.654 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.654 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659897+0000", 2024-09-21T10:46:34.654 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.85872434499999994 2024-09-21T10:46:34.654 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.655 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.655 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659901+0000", 2024-09-21T10:46:34.655 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4479999999999999e-06 2024-09-21T10:46:34.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.655 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.655 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.814871+0000", 2024-09-21T10:46:34.655 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.154970026 2024-09-21T10:46:34.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.655 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:46:34.655 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.814966+0000", 2024-09-21T10:46:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5260999999999995e-05 2024-09-21T10:46:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.815058+0000", 2024-09-21T10:46:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1976000000000002e-05 2024-09-21T10:46:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.818386+0000", 2024-09-21T10:46:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0033278079999999998 2024-09-21T10:46:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:46:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.818396+0000", 2024-09-21T10:46:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.046e-05 2024-09-21T10:46:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.828708+0000", 2024-09-21T10:46:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.010312089999999999 2024-09-21T10:46:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:46:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.828766+0000", 2024-09-21T10:46:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7964000000000003e-05 2024-09-21T10:46:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.658 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:46:34.658 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.829441+0000", 2024-09-21T10:46:34.658 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00067455100000000001 2024-09-21T10:46:34.658 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.658 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:46:34.658 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.658 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.658 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.658 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77583 53.10 53:0c625067:::benchmark_data_smithi070_228967_object77582:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:46:34.658 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.823905+0000", 2024-09-21T10:46:34.658 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 328.745789538, 2024-09-21T10:46:34.658 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0063239049999999, 2024-09-21T10:46:34.658 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:46:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:46:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:46:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:46:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:46:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77583 2024-09-21T10:46:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:46:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:46:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.823905+0000", 2024-09-21T10:46:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:46:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.823905+0000", 2024-09-21T10:46:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:46:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.823906+0000", 2024-09-21T10:46:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6190000000000001e-06 2024-09-21T10:46:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:46:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.823936+0000", 2024-09-21T10:46:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9220000000000001e-05 2024-09-21T10:46:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:46:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.823947+0000", 2024-09-21T10:46:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1337e-05 2024-09-21T10:46:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:46:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.823954+0000", 2024-09-21T10:46:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8719999999999996e-06 2024-09-21T10:46:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.824052+0000", 2024-09-21T10:46:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.8405000000000004e-05 2024-09-21T10:46:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.824059+0000", 2024-09-21T10:46:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4110000000000001e-06 2024-09-21T10:46:34.662 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.662 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.662 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.662 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659929+0000", 2024-09-21T10:46:34.662 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.83586998099999998 2024-09-21T10:46:34.662 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.662 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.662 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.662 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659936+0000", 2024-09-21T10:46:34.662 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1180000000000003e-06 2024-09-21T10:46:34.662 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.662 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.663 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.815647+0000", 2024-09-21T10:46:34.663 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.155710826 2024-09-21T10:46:34.663 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.663 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:46:34.663 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.815726+0000", 2024-09-21T10:46:34.663 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9102000000000003e-05 2024-09-21T10:46:34.663 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.663 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:34.663 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.815820+0000", 2024-09-21T10:46:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4326000000000005e-05 2024-09-21T10:46:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:46:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.818401+0000", 2024-09-21T10:46:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025805509999999999 2024-09-21T10:46:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.818441+0000", 2024-09-21T10:46:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0170999999999999e-05 2024-09-21T10:46:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.829499+0000", 2024-09-21T10:46:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.011058182 2024-09-21T10:46:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:46:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.829557+0000", 2024-09-21T10:46:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8409999999999998e-05 2024-09-21T10:46:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:46:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.830229+0000", 2024-09-21T10:46:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00067137399999999997 2024-09-21T10:46:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:46:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.666 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.666 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77587 53.10 53:098fcb2a:::benchmark_data_smithi070_228967_object77586:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:46:34.666 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.825548+0000", 2024-09-21T10:46:34.666 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 328.74414593199998, 2024-09-21T10:46:34.666 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0105805079999999, 2024-09-21T10:46:34.666 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:46:34.666 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:46:34.666 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:34.666 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:46:34.666 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:46:34.666 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:46:34.666 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77587 2024-09-21T10:46:34.666 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:46:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:46:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.825548+0000", 2024-09-21T10:46:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:46:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.825548+0000", 2024-09-21T10:46:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:46:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.825550+0000", 2024-09-21T10:46:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2759999999999999e-06 2024-09-21T10:46:34.668 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.668 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.668 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:46:34.668 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.825602+0000", 2024-09-21T10:46:34.668 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2707000000000002e-05 2024-09-21T10:46:34.668 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.668 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.668 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:46:34.668 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.825617+0000", 2024-09-21T10:46:34.668 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4871e-05 2024-09-21T10:46:34.668 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.668 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.668 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:46:34.669 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.825625+0000", 2024-09-21T10:46:34.669 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0809999999999994e-06 2024-09-21T10:46:34.669 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.669 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.669 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.669 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.825677+0000", 2024-09-21T10:46:34.669 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1218000000000003e-05 2024-09-21T10:46:34.669 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.669 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.669 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.669 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.825685+0000", 2024-09-21T10:46:34.669 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3280000000000006e-06 2024-09-21T10:46:34.669 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659956+0000", 2024-09-21T10:46:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.83427153799999998 2024-09-21T10:46:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659961+0000", 2024-09-21T10:46:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9540000000000003e-06 2024-09-21T10:46:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.816365+0000", 2024-09-21T10:46:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1564039589999999 2024-09-21T10:46:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:46:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.816477+0000", 2024-09-21T10:46:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000111575 2024-09-21T10:46:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.816577+0000", 2024-09-21T10:46:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000100078 2024-09-21T10:46:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.818499+0000", 2024-09-21T10:46:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0019215129999999999 2024-09-21T10:46:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:46:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.818509+0000", 2024-09-21T10:46:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0200999999999999e-05 2024-09-21T10:46:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.834949+0000", 2024-09-21T10:46:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.016440735000000001 2024-09-21T10:46:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:46:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.835043+0000", 2024-09-21T10:46:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3177999999999995e-05 2024-09-21T10:46:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:46:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.836129+0000", 2024-09-21T10:46:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001086296 2024-09-21T10:46:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:46:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77621 53.10 53:08d8a996:::benchmark_data_smithi070_228967_object77620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:46:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.858882+0000", 2024-09-21T10:46:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 328.71081212799999, 2024-09-21T10:46:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9785834369999999, 2024-09-21T10:46:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:46:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:46:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:46:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:46:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:46:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77621 2024-09-21T10:46:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:46:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:46:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.858882+0000", 2024-09-21T10:46:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:46:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.858882+0000", 2024-09-21T10:46:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:46:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.858883+0000", 2024-09-21T10:46:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5099999999999998e-07 2024-09-21T10:46:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:46:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.858933+0000", 2024-09-21T10:46:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9700000000000002e-05 2024-09-21T10:46:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:46:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.858943+0000", 2024-09-21T10:46:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7939999999999997e-06 2024-09-21T10:46:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:46:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.858949+0000", 2024-09-21T10:46:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7769999999999997e-06 2024-09-21T10:46:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.859007+0000", 2024-09-21T10:46:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7456e-05 2024-09-21T10:46:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.859017+0000", 2024-09-21T10:46:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0071999999999999e-05 2024-09-21T10:46:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659993+0000", 2024-09-21T10:46:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.80097583500000002 2024-09-21T10:46:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659998+0000", 2024-09-21T10:46:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4929999999999998e-06 2024-09-21T10:46:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.817285+0000", 2024-09-21T10:46:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1572862640000001 2024-09-21T10:46:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:46:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.817418+0000", 2024-09-21T10:46:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013367300000000001 2024-09-21T10:46:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.817576+0000", 2024-09-21T10:46:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015830299999999999 2024-09-21T10:46:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.819630+0000", 2024-09-21T10:46:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0020534120000000001 2024-09-21T10:46:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:46:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.820470+0000", 2024-09-21T10:46:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00083965699999999995 2024-09-21T10:46:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.836186+0000", 2024-09-21T10:46:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015716218000000001 2024-09-21T10:46:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:46:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.836282+0000", 2024-09-21T10:46:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5839000000000002e-05 2024-09-21T10:46:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:46:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.837466+0000", 2024-09-21T10:46:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001184093 2024-09-21T10:46:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:46:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77656 53.10 53:08530a06:::benchmark_data_smithi070_228967_object77655:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:46:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.901366+0000", 2024-09-21T10:46:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 328.66832798299998, 2024-09-21T10:46:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9373663639999998, 2024-09-21T10:46:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:46:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:46:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:46:34.681 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:46:34.681 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:46:34.681 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77656 2024-09-21T10:46:34.681 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.681 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:46:34.681 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.681 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:46:34.681 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.901366+0000", 2024-09-21T10:46:34.681 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.681 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.681 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.681 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:46:34.681 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.901366+0000", 2024-09-21T10:46:34.681 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.681 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.681 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.682 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:46:34.682 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.901377+0000", 2024-09-21T10:46:34.682 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0122999999999999e-05 2024-09-21T10:46:34.682 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.682 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.682 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:46:34.682 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.901395+0000", 2024-09-21T10:46:34.682 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8896e-05 2024-09-21T10:46:34.682 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.682 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.682 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:46:34.682 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.901407+0000", 2024-09-21T10:46:34.682 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1409999999999999e-05 2024-09-21T10:46:34.682 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.682 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:46:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.901413+0000", 2024-09-21T10:46:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9399999999999999e-06 2024-09-21T10:46:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.901471+0000", 2024-09-21T10:46:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7853000000000003e-05 2024-09-21T10:46:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.901480+0000", 2024-09-21T10:46:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1959999999999997e-06 2024-09-21T10:46:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.660022+0000", 2024-09-21T10:46:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.75854222100000002 2024-09-21T10:46:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.660026+0000", 2024-09-21T10:46:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7840000000000001e-06 2024-09-21T10:46:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.819670+0000", 2024-09-21T10:46:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1596445310000001 2024-09-21T10:46:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:46:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.819755+0000", 2024-09-21T10:46:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5062000000000007e-05 2024-09-21T10:46:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.819853+0000", 2024-09-21T10:46:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7116999999999999e-05 2024-09-21T10:46:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.823074+0000", 2024-09-21T10:46:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.003221776 2024-09-21T10:46:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:46:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.823085+0000", 2024-09-21T10:46:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0924000000000001e-05 2024-09-21T10:46:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.837697+0000", 2024-09-21T10:46:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.014611455000000001 2024-09-21T10:46:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:46:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.837931+0000", 2024-09-21T10:46:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00023461000000000001 2024-09-21T10:46:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:46:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.838733+0000", 2024-09-21T10:46:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00080146600000000003 2024-09-21T10:46:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:46:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77672 53.10 53:0f020cb9:::benchmark_data_smithi070_228967_object77671:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:46:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.923481+0000", 2024-09-21T10:46:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 328.64621296299998, 2024-09-21T10:46:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9159655249999998, 2024-09-21T10:46:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:46:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:46:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:46:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:46:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:46:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77672 2024-09-21T10:46:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:46:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:46:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.923481+0000", 2024-09-21T10:46:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:46:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.923481+0000", 2024-09-21T10:46:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:46:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.923483+0000", 2024-09-21T10:46:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.19e-06 2024-09-21T10:46:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:46:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.923502+0000", 2024-09-21T10:46:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9454e-05 2024-09-21T10:46:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:46:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.923512+0000", 2024-09-21T10:46:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0390999999999999e-05 2024-09-21T10:46:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:46:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.923518+0000", 2024-09-21T10:46:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2599999999999996e-06 2024-09-21T10:46:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.923561+0000", 2024-09-21T10:46:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3621999999999997e-05 2024-09-21T10:46:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.923568+0000", 2024-09-21T10:46:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1689999999999997e-06 2024-09-21T10:46:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.660046+0000", 2024-09-21T10:46:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.73647844299999998 2024-09-21T10:46:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.660050+0000", 2024-09-21T10:46:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8070000000000001e-06 2024-09-21T10:46:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.820493+0000", 2024-09-21T10:46:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.160443109 2024-09-21T10:46:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:46:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.820584+0000", 2024-09-21T10:46:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0705999999999995e-05 2024-09-21T10:46:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.820680+0000", 2024-09-21T10:46:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.6836999999999995e-05 2024-09-21T10:46:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:46:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.823091+0000", 2024-09-21T10:46:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0024103309999999999 2024-09-21T10:46:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.823139+0000", 2024-09-21T10:46:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7985999999999998e-05 2024-09-21T10:46:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.838802+0000", 2024-09-21T10:46:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015663567999999999 2024-09-21T10:46:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:46:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.838880+0000", 2024-09-21T10:46:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7302e-05 2024-09-21T10:46:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:46:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.839447+0000", 2024-09-21T10:46:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00056734999999999999 2024-09-21T10:46:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:46:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77683 53.10 53:0c041dae:::benchmark_data_smithi070_228967_object77682:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:46:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.950914+0000", 2024-09-21T10:46:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 328.61878059399999, 2024-09-21T10:46:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.889211167, 2024-09-21T10:46:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:46:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:46:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:46:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:46:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:46:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77683 2024-09-21T10:46:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:46:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:46:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.950914+0000", 2024-09-21T10:46:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:46:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.950914+0000", 2024-09-21T10:46:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:46:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.950915+0000", 2024-09-21T10:46:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.508e-06 2024-09-21T10:46:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:46:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.950958+0000", 2024-09-21T10:46:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3133999999999997e-05 2024-09-21T10:46:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:46:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.950974+0000", 2024-09-21T10:46:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5828000000000002e-05 2024-09-21T10:46:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:46:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.950981+0000", 2024-09-21T10:46:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1670000000000002e-06 2024-09-21T10:46:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.951050+0000", 2024-09-21T10:46:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8666999999999998e-05 2024-09-21T10:46:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.951062+0000", 2024-09-21T10:46:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2064e-05 2024-09-21T10:46:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.660067+0000", 2024-09-21T10:46:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.70900450999999998 2024-09-21T10:46:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.660071+0000", 2024-09-21T10:46:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8580000000000002e-06 2024-09-21T10:46:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.699 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.821230+0000", 2024-09-21T10:46:34.699 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1611593650000001 2024-09-21T10:46:34.699 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.699 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.699 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:46:34.699 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.821312+0000", 2024-09-21T10:46:34.699 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2452000000000003e-05 2024-09-21T10:46:34.699 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.699 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.699 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:46:34.699 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.821416+0000", 2024-09-21T10:46:34.699 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010381400000000001 2024-09-21T10:46:34.699 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.701 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:34.702 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:46:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.824269+0000", 2024-09-21T10:46:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0028524179999999998 2024-09-21T10:46:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.824301+0000", 2024-09-21T10:46:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2481999999999998e-05 2024-09-21T10:46:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.839511+0000", 2024-09-21T10:46:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015209926 2024-09-21T10:46:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:46:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.839581+0000", 2024-09-21T10:46:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9913000000000006e-05 2024-09-21T10:46:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:46:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.840125+0000", 2024-09-21T10:46:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00054406099999999998 2024-09-21T10:46:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:46:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:80155 53.12 53:4c70868e:::benchmark_data_smithi070_228967_object80154:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1330)", 2024-09-21T10:46:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:10.695776+0000", 2024-09-21T10:46:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 323.87391824700001, 2024-09-21T10:46:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3340382100000001, 2024-09-21T10:46:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:46:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:46:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:46:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:46:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:46:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 80155 2024-09-21T10:46:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:46:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:46:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695776+0000", 2024-09-21T10:46:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:46:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695776+0000", 2024-09-21T10:46:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:46:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695777+0000", 2024-09-21T10:46:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0900000000000002e-07 2024-09-21T10:46:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:46:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695795+0000", 2024-09-21T10:46:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8150000000000001e-05 2024-09-21T10:46:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:46:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695802+0000", 2024-09-21T10:46:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9750000000000001e-06 2024-09-21T10:46:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:46:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695807+0000", 2024-09-21T10:46:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1869999999999996e-06 2024-09-21T10:46:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.697268+0000", 2024-09-21T10:46:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001461237 2024-09-21T10:46:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.697275+0000", 2024-09-21T10:46:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0600000000000002e-06 2024-09-21T10:46:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.014876+0000", 2024-09-21T10:46:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3176006689999999 2024-09-21T10:46:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:46:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.015180+0000", 2024-09-21T10:46:34.709 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00030391000000000001 2024-09-21T10:46:34.709 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.709 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.709 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,3,6", 2024-09-21T10:46:34.709 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.015361+0000", 2024-09-21T10:46:34.709 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018097299999999999 2024-09-21T10:46:34.709 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.709 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.709 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.709 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.018976+0000", 2024-09-21T10:46:34.709 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0036151109999999998 2024-09-21T10:46:34.709 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.709 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:46:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.020561+0000", 2024-09-21T10:46:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0015851369999999999 2024-09-21T10:46:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.027092+0000", 2024-09-21T10:46:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0065314250000000004 2024-09-21T10:46:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.028432+0000", 2024-09-21T10:46:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00134005 2024-09-21T10:46:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.711 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.711 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:46:34.711 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.028542+0000", 2024-09-21T10:46:34.711 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000109574 2024-09-21T10:46:34.711 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.711 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.711 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:46:34.711 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.029814+0000", 2024-09-21T10:46:34.711 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001272343 2024-09-21T10:46:34.711 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.711 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:46:34.711 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.711 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.712 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.712 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:80160 53.12 53:4958c2ed:::benchmark_data_smithi070_228967_object80159:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1330)", 2024-09-21T10:46:34.712 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:10.695813+0000", 2024-09-21T10:46:34.712 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 323.873881063, 2024-09-21T10:46:34.712 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3537495580000001, 2024-09-21T10:46:34.712 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:46:34.712 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:46:34.712 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:34.712 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:46:34.712 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:46:34.712 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:46:34.712 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 80160 2024-09-21T10:46:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:46:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:46:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695813+0000", 2024-09-21T10:46:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:46:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695813+0000", 2024-09-21T10:46:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:46:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695814+0000", 2024-09-21T10:46:34.714 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7800000000000002e-07 2024-09-21T10:46:34.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.714 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.714 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:46:34.714 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695831+0000", 2024-09-21T10:46:34.714 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7311000000000001e-05 2024-09-21T10:46:34.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.714 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.714 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:46:34.714 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695846+0000", 2024-09-21T10:46:34.714 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.47e-05 2024-09-21T10:46:34.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.714 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:46:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695852+0000", 2024-09-21T10:46:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2190000000000001e-06 2024-09-21T10:46:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.697303+0000", 2024-09-21T10:46:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0014514840000000001 2024-09-21T10:46:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.697310+0000", 2024-09-21T10:46:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5019999999999998e-06 2024-09-21T10:46:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.716 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.716 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.029868+0000", 2024-09-21T10:46:34.716 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3325584930000001 2024-09-21T10:46:34.716 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.716 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.716 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:46:34.716 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.030023+0000", 2024-09-21T10:46:34.716 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015419899999999999 2024-09-21T10:46:34.716 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.716 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.716 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,3,6", 2024-09-21T10:46:34.716 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.030189+0000", 2024-09-21T10:46:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016664199999999999 2024-09-21T10:46:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.038428+0000", 2024-09-21T10:46:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0082384810000000006 2024-09-21T10:46:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.041162+0000", 2024-09-21T10:46:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0027339759999999999 2024-09-21T10:46:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:46:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.041175+0000", 2024-09-21T10:46:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3271000000000001e-05 2024-09-21T10:46:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.047349+0000", 2024-09-21T10:46:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0061744950000000003 2024-09-21T10:46:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:46:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.047430+0000", 2024-09-21T10:46:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0260999999999996e-05 2024-09-21T10:46:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:46:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.049563+0000", 2024-09-21T10:46:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0021331459999999998 2024-09-21T10:46:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:46:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:80214 53.12 53:4883ca5c:::benchmark_data_smithi070_228967_object80213:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1330)", 2024-09-21T10:46:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:10.695859+0000", 2024-09-21T10:46:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 323.873835853, 2024-09-21T10:46:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.355014194, 2024-09-21T10:46:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:46:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:46:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:46:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:46:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:46:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 80214 2024-09-21T10:46:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:46:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:46:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695859+0000", 2024-09-21T10:46:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:46:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695859+0000", 2024-09-21T10:46:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:46:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695859+0000", 2024-09-21T10:46:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3599999999999999e-07 2024-09-21T10:46:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:46:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695880+0000", 2024-09-21T10:46:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1478e-05 2024-09-21T10:46:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:46:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695921+0000", 2024-09-21T10:46:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0139999999999999e-05 2024-09-21T10:46:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:46:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695926+0000", 2024-09-21T10:46:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9039999999999997e-06 2024-09-21T10:46:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.698208+0000", 2024-09-21T10:46:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00228123 2024-09-21T10:46:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.698216+0000", 2024-09-21T10:46:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1529999999999997e-06 2024-09-21T10:46:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.031500+0000", 2024-09-21T10:46:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3332841609999999 2024-09-21T10:46:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:46:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.031666+0000", 2024-09-21T10:46:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000165515 2024-09-21T10:46:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,3,6", 2024-09-21T10:46:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.031885+0000", 2024-09-21T10:46:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00021924600000000001 2024-09-21T10:46:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.038468+0000", 2024-09-21T10:46:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0065835700000000004 2024-09-21T10:46:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:46:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.041181+0000", 2024-09-21T10:46:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0027126630000000001 2024-09-21T10:46:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.041240+0000", 2024-09-21T10:46:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8641999999999999e-05 2024-09-21T10:46:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.049667+0000", 2024-09-21T10:46:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0084272570000000005 2024-09-21T10:46:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:46:34.726 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.049759+0000", 2024-09-21T10:46:34.726 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2334000000000001e-05 2024-09-21T10:46:34.726 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.726 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.726 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:46:34.726 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.050873+0000", 2024-09-21T10:46:34.726 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0011134649999999999 2024-09-21T10:46:34.726 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.726 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:46:34.726 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.726 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.726 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.726 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:238408 53.15 53:ae2b89db:::benchmark_data_smithi070_228967_object109488:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1354)", 2024-09-21T10:46:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:43:42.177749+0000", 2024-09-21T10:46:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 172.391945881, 2024-09-21T10:46:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3328045070000001, 2024-09-21T10:46:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:46:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:46:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:46:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:46:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:46:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 238408 2024-09-21T10:46:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:46:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:46:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:42.177749+0000", 2024-09-21T10:46:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:46:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:42.177749+0000", 2024-09-21T10:46:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:46:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:46:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:42.177749+0000", 2024-09-21T10:46:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1300000000000001e-07 2024-09-21T10:46:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:46:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:42.177751+0000", 2024-09-21T10:46:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.193e-06 2024-09-21T10:46:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:46:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:42.177758+0000", 2024-09-21T10:46:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7460000000000003e-06 2024-09-21T10:46:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:46:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:42.177763+0000", 2024-09-21T10:46:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2599999999999996e-06 2024-09-21T10:46:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:42.177984+0000", 2024-09-21T10:46:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000220898 2024-09-21T10:46:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting_for_map not empty", 2024-09-21T10:46:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:42.177989+0000", 2024-09-21T10:46:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6029999999999998e-06 2024-09-21T10:46:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:42.181212+0000", 2024-09-21T10:46:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0032235710000000002 2024-09-21T10:46:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:46:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:42.181223+0000", 2024-09-21T10:46:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0832999999999999e-05 2024-09-21T10:46:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:46:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:43.487360+0000", 2024-09-21T10:46:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3061371799999999 2024-09-21T10:46:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:46:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:43.487503+0000", 2024-09-21T10:46:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014267899999999999 2024-09-21T10:46:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,3,4,6", 2024-09-21T10:46:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:43.489004+0000", 2024-09-21T10:46:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001501631 2024-09-21T10:46:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:43.500172+0000", 2024-09-21T10:46:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.011167912 2024-09-21T10:46:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.733 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.733 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.733 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:43.500245+0000", 2024-09-21T10:46:34.733 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3146999999999996e-05 2024-09-21T10:46:34.733 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.733 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.733 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.733 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:43.500318+0000", 2024-09-21T10:46:34.733 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2065999999999997e-05 2024-09-21T10:46:34.733 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.733 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.733 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:46:34.733 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:43.501132+0000", 2024-09-21T10:46:34.734 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00081481200000000004 2024-09-21T10:46:34.734 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.734 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.734 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:34.734 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:43.510386+0000", 2024-09-21T10:46:34.734 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0092536349999999996 2024-09-21T10:46:34.734 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.734 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.734 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:46:34.734 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:43.510434+0000", 2024-09-21T10:46:34.734 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7960000000000002e-05 2024-09-21T10:46:34.734 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:46:34.734 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:46:34.734 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:46:34.735 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:43.510553+0000", 2024-09-21T10:46:34.735 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011906800000000001 2024-09-21T10:46:34.735 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.735 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:46:34.735 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.735 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:46:34.735 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:46:34.735 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:46:34.765 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:34.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:34.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:34 smithi160 ceph-mon[101879]: pgmap v2420: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 972 op/s 2024-09-21T10:46:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:34 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2824599537' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:46:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:34 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:46:34.867 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:34.897 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:34.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:34.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:34.999 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:35.034 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:35.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:35.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:35.136 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:35.170 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:35.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:35.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:35.272 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:35.312 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:35.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:35.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:35.413 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:35.449 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:35.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:35.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:35.551 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:35.582 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:35.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:35.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:35.683 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:35.712 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:35.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:35.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:35.814 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:35 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:46:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:35 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:46:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:35 smithi160 ceph-mon[101879]: osdmap e1391: 8 total, 8 up, 7 in 2024-09-21T10:46:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:46:35.850 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:35.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:35.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:35.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:35 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:46:35.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:35 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:46:35.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:35 smithi070 ceph-mon[112139]: osdmap e1391: 8 total, 8 up, 7 in 2024-09-21T10:46:35.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:46:35.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:35 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:46:35.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:35 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:46:35.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:35 smithi070 ceph-mon[107385]: osdmap e1391: 8 total, 8 up, 7 in 2024-09-21T10:46:35.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:46:35.951 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:35.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:35.985 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:35.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:36.058 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:46:36.086 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:36.173 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:36.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:36.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:36.275 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:36.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:36.312 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:36.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:36.412 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:36.447 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:36.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:36.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:36.548 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:36.595 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:46:36.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:36.604 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:36.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:36.631 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:36.704 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:36.734 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:36.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:36.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:36.808 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:36 smithi070 ceph-mon[107385]: pgmap v2422: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 975 op/s 2024-09-21T10:46:36.808 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:36 smithi070 ceph-mon[112139]: pgmap v2422: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 975 op/s 2024-09-21T10:46:36.835 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:36 smithi160 ceph-mon[101879]: pgmap v2422: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 975 op/s 2024-09-21T10:46:36.875 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:36.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:36.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:36.976 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:37.019 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:37.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:37.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:37.121 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:37.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:37.158 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:37.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:37.259 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:37.289 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:37.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:37.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:37.391 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:37.428 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:37.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:37.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:37.529 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:37.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:37.554 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:37.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:37.655 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:37.686 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:37.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:37.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:37.788 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:37.825 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:37.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:37.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:37.828 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:37.927 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:37.955 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:37.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:37.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:38.057 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:38.084 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:38.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:38.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:38.096 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:46:38.115 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:38.186 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:38.225 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:38.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:38.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:38.327 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:38.364 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:38.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:38.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:38.465 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:38.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:38.511 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:38.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:38.612 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:38.651 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:38.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:38.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:38.752 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:38.792 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:38.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:38.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:38 smithi160 ceph-mon[101879]: pgmap v2423: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s 2024-09-21T10:46:38.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:38 smithi070 ceph-mon[107385]: pgmap v2423: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s 2024-09-21T10:46:38.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:38 smithi070 ceph-mon[112139]: pgmap v2423: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s 2024-09-21T10:46:38.894 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:38.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:38.935 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:38.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:39.036 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:39.062 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:39.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:39.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:39.163 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:39.197 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:39.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:39.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:39.298 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:39.339 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:39.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:39.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:39.440 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:39.474 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:39.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:39.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:39.576 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:39.622 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:39.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:39.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:39 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:46:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:39 smithi070 ceph-mon[107385]: pgmap v2424: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s 2024-09-21T10:46:39.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:39 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:46:39.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:39 smithi070 ceph-mon[112139]: pgmap v2424: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s 2024-09-21T10:46:39.724 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:39.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:39.752 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:39.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:39 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:46:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:39 smithi160 ceph-mon[101879]: pgmap v2424: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s 2024-09-21T10:46:39.854 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:39.887 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:39.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:39.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:39.989 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:40.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:40.021 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:40.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:40.122 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:40.150 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:40.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:40.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:40.252 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:40.292 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:40.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:40.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:40.394 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:40.431 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:40.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:40.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:40.533 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:40.556 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T10:46:40.571 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:40.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:40.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:40.576 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:40 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1465682379' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:46:40.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:40 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1465682379' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:46:40.673 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:40.704 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:40.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:40.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:40.806 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:40 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1465682379' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:46:40.853 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:40.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:40.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:40.954 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:40.991 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:40.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:40.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:41.093 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:41.164 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:41.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:41.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:41.265 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:41.318 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:41.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:41.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:41.419 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:41.459 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:41.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:41.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:41.560 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:41.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:41.593 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:41.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:41.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:41 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1465682379' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:46:41.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:41 smithi070 ceph-mon[107385]: osdmap e1392: 8 total, 8 up, 7 in 2024-09-21T10:46:41.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:41 smithi070 ceph-mon[107385]: pgmap v2426: 89 pgs: 89 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 926 op/s 2024-09-21T10:46:41.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:41 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1465682379' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:46:41.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:41 smithi070 ceph-mon[112139]: osdmap e1392: 8 total, 8 up, 7 in 2024-09-21T10:46:41.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:41 smithi070 ceph-mon[112139]: pgmap v2426: 89 pgs: 89 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 926 op/s 2024-09-21T10:46:41.694 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:41.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:41.744 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:41.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:41.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:41 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1465682379' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:46:41.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:41 smithi160 ceph-mon[101879]: osdmap e1392: 8 total, 8 up, 7 in 2024-09-21T10:46:41.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:41 smithi160 ceph-mon[101879]: pgmap v2426: 89 pgs: 89 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 926 op/s 2024-09-21T10:46:41.846 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:41.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:41.889 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:41.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:41.990 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:42.032 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:42.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:42.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:42.134 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:42.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:42.163 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:42.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:42.264 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:42.292 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:42.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:42.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:42.324 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:42.351 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T10:46:42.394 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:42.467 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:42.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:42.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:42.569 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:42.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:42.603 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:42.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:42.704 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:42.744 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:42.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:42.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:42.845 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:42.872 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:42.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:42.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:42.974 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:43.007 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:43.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:43.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:43.109 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:43.147 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:43.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:43.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:43.248 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:43.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:43.312 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:43.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:43.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:46:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:46:43] "GET /metrics HTTP/1.1" 200 36606 "" "Prometheus/2.51.0" 2024-09-21T10:46:43.413 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:43.438 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:43.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:43.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:43.583 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:43.614 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:43.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:43.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:43.716 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:43.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:43.749 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:43.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:43.850 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:43.878 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:43.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:43.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:43.903 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:43 smithi160 ceph-mon[101879]: pgmap v2427: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 1.11k op/s 2024-09-21T10:46:43.980 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:44.012 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:44.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:44.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:44.113 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:44.115 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:43 smithi070 ceph-mon[112139]: pgmap v2427: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 1.11k op/s 2024-09-21T10:46:44.115 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:43 smithi070 ceph-mon[107385]: pgmap v2427: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 1.11k op/s 2024-09-21T10:46:44.136 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:46:44.137 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:46:44.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:44.146 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:44.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:44.152 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:46:44.153 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:46:44.193 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:46:44.193 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:46:44.232 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:46:44.232 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:46:44.247 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:44.248 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:46:44.248 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:46:44.261 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:46:44.262 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:46:44.275 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:44.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:44.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:44.291 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:46:44.291 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:46:44.315 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:46:44.315 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:46:44.355 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:46:44.355 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:46:44.377 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:44.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:44.427 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:44.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:44.528 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:44.557 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:44.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:44.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:44.659 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:44.722 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:44.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:44.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:44.823 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:44.850 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:44.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:44.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:44.952 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:44.980 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:44.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:44.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:45.082 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:45.111 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:45.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:45.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:45.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:44 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1715545635' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:45.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:44 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:46:45.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:44 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:46:45.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:44 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1715545635' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:45.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:44 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:46:45.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:44 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:46:45.213 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:45.239 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:45.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:45.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:45.341 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:45.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:44 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1715545635' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:45.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:44 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:46:45.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:44 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:46:45.373 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:45.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:45.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:45.474 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:45.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:45.505 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:45.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:45.606 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:45.640 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:45.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:45.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:45.742 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:45.769 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:45.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:45.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:45.870 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:45.916 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:45.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:45.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:46.018 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:46.051 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:46.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:46.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:46.153 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:46.183 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:46.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:46.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:46.285 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:46.321 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:46.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:46.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:46.423 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:46.460 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:46.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:46.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:46.462 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:46 smithi070 ceph-mon[107385]: pgmap v2428: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 978 op/s 2024-09-21T10:46:46.462 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:46 smithi070 ceph-mon[112139]: pgmap v2428: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 978 op/s 2024-09-21T10:46:46.562 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:46.588 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:46.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:46.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:46.690 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:46.716 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:46.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:46.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:46.817 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:46.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:46 smithi160 ceph-mon[101879]: pgmap v2428: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 978 op/s 2024-09-21T10:46:46.853 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:46.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:46.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:46.955 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:46.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:46.987 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:46.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:47.088 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:47.130 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:47.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:47.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:47.231 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:47.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:47.277 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:47.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:47.378 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:47.412 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:47.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:47.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:47.514 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:47.547 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:47.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:47.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:47.599 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:46:47.649 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:47.692 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:47.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:47.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:47.794 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:47.825 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:47.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:47.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:47.927 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:47.954 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:47.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:47.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:48.056 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:48.080 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:48.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:48.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:48.182 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:48.210 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:48.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:48.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:48.312 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:48.347 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:48.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:48.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:48.415 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:46:48.418 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:48.449 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:48.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:48.491 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:48.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:48.592 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:48.621 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:48.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:48.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:48.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:48 smithi070 ceph-mon[112139]: pgmap v2429: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 947 op/s 2024-09-21T10:46:48.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:48 smithi070 ceph-mon[107385]: pgmap v2429: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 947 op/s 2024-09-21T10:46:48.723 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:48.758 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:48.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:48.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:48 smithi160 ceph-mon[101879]: pgmap v2429: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 947 op/s 2024-09-21T10:46:48.860 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:48.894 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:48.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:48.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:48.996 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:49.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:49.035 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:49.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:49.136 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:49.169 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:49.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:49.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:49.271 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:49.312 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:49.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:49.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:49.414 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:49.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:49.450 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:49.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:49.551 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:49.579 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:49.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:49.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:49.681 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:49.717 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:49.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:49.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:49.819 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:49.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:49.842 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:49.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:49.943 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:49.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:49.984 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:49.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:50.085 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:50.106 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:50.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:50.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:50.208 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:50.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:50.248 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:50.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:50.349 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:50.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:50.388 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:50.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:50.489 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:50.490 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:50.527 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:50.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:50.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:50 smithi070 ceph-mon[107385]: pgmap v2430: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 947 op/s 2024-09-21T10:46:50.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:50 smithi070 ceph-mon[112139]: pgmap v2430: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 947 op/s 2024-09-21T10:46:50.629 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:50.670 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:50.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:50.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:50.772 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:50.800 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:50.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:50.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:50 smithi160 ceph-mon[101879]: pgmap v2430: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 947 op/s 2024-09-21T10:46:50.902 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:50.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:50.933 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:50.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:51.033 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:51.074 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:51.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:51.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:51.175 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:51.209 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:51.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:51.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:51.311 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:51.346 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:51.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:51.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:51.448 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:51.479 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:51.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:51.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:51.581 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:51.624 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:51.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:51.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:51.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:46:51.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:46:51.726 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:51.761 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:51.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:51.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:46:51.864 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:51.906 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:51.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:51.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:52.008 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:52.048 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:52.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:52.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:52.150 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:52.191 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:52.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:52.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:52.292 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:52.333 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:52.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:52.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:52.378 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T10:46:52.435 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:52.464 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:52.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:52.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:52.566 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:52.581 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:52 smithi070 ceph-mon[107385]: pgmap v2431: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 929 op/s 2024-09-21T10:46:52.581 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:52 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1746837803' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:52.584 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:52 smithi070 ceph-mon[112139]: pgmap v2431: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 929 op/s 2024-09-21T10:46:52.584 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:52 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1746837803' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:52.598 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:52.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:52.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:52.700 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:52.728 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:52.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:52.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:52.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:52 smithi160 ceph-mon[101879]: pgmap v2431: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 929 op/s 2024-09-21T10:46:52.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:52 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1746837803' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:52.830 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:52.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:52.866 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:52.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:52.967 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:52.992 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:52.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:52.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:53.094 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:53.130 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:53.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:53.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:53.231 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:53.265 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:53.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:53.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:53.366 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:53.400 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:53.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:53.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:53.444 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:46:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:46:53] "GET /metrics HTTP/1.1" 200 36606 "" "Prometheus/2.51.0" 2024-09-21T10:46:53.444 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:53 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:46:53.444 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:53 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1746837803' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:46:53.444 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:53 smithi070 ceph-mon[107385]: osdmap e1393: 8 total, 8 up, 7 in 2024-09-21T10:46:53.445 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:53 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:46:53.445 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:53 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1746837803' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:46:53.445 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:53 smithi070 ceph-mon[112139]: osdmap e1393: 8 total, 8 up, 7 in 2024-09-21T10:46:53.481 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:53 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:46:53.481 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:53 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1746837803' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:46:53.481 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:53 smithi160 ceph-mon[101879]: osdmap e1393: 8 total, 8 up, 7 in 2024-09-21T10:46:53.501 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:53.530 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:53.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:53.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:53.632 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:53.658 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:53.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:53.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:53.760 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:53.796 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:53.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:53.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:53.898 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:53.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:53.932 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:53.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:54.033 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:54.066 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:54.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:54.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:54.168 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:54.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:54.209 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:54.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:54.310 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:54.345 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:54.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:54.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:54.366 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:54.448 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:54.496 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:54.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:54.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:54.598 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:54.623 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:54 smithi070 ceph-mon[112139]: pgmap v2433: 89 pgs: 89 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 943 op/s 2024-09-21T10:46:54.623 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:54 smithi070 ceph-mon[107385]: pgmap v2433: 89 pgs: 89 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 943 op/s 2024-09-21T10:46:54.648 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:54.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:54.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:54.673 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:54 smithi160 ceph-mon[101879]: pgmap v2433: 89 pgs: 89 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 943 op/s 2024-09-21T10:46:54.751 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:54.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:54.788 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:54.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:54.889 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:54.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:54.927 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:54.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:55.027 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:55.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:55.065 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:55.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:55.166 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:55.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:55.199 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:55.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:55.300 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:55.333 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:46:55.337 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:55.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:55.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:55.439 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:55.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:55.486 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:55.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:55.587 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:55.621 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:55.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:55.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:55.723 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:55.754 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:55.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:55.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:55.856 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:55.902 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:55.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:55.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:56.003 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:56.033 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:56.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:56.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:56.135 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:56.169 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:56.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:56.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:56.271 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:56.309 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:56.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:56.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:56.410 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:56.417 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:56 smithi070 ceph-mon[107385]: pgmap v2434: 89 pgs: 89 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 942 op/s 2024-09-21T10:46:56.417 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:56 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2239676512' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:56.445 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:56.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:56.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:56.547 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:56.583 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:56.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:56.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:56.685 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:56.728 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:56.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:56.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:56.736 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:56 smithi070 ceph-mon[112139]: pgmap v2434: 89 pgs: 89 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 942 op/s 2024-09-21T10:46:56.736 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:56 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2239676512' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:56.830 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:56.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:56 smithi160 ceph-mon[101879]: pgmap v2434: 89 pgs: 89 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 942 op/s 2024-09-21T10:46:56.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:56 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2239676512' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:56.865 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:56.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:56.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:56.870 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:46:56.870 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:46:56.870 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:46:56.870 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:46:56.871 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:46:56.871 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:46:56.871 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:46:56.871 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:46:56.871 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:46:56.871 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:46:56.871 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:46:56.871 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:46:56.871 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:46:56.871 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:46:56.871 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:46:56.872 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:46:56.966 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:57.005 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:57.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:57.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:57.107 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:57.140 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:57.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:57.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:57.242 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:57.275 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:57.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:57.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:57.376 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:57.412 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:57.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:57.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:57.432 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T10:46:57.432 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:57 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2605297011' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:46:57.432 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:57 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:46:57.433 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:57 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2605297011' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:46:57.433 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:57 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:46:57.513 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:57.543 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:57.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:57.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:57.645 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:57.688 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:57.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:57.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:57.790 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:57.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:57.821 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:57.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:57 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2605297011' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:46:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:57 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:46:57.922 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:57.965 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:57.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:57.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:58.067 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:58.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:58.104 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:58.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:58.205 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:58.225 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T10:46:58.249 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:58.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:58.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:58.351 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:58.386 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:58.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:58.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:58.456 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:58 smithi070 ceph-mon[112139]: pgmap v2435: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 933 op/s 2024-09-21T10:46:58.456 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:58 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:46:58.456 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:46:58 smithi070 ceph-mon[112139]: osdmap e1394: 8 total, 8 up, 7 in 2024-09-21T10:46:58.456 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:58 smithi070 ceph-mon[107385]: pgmap v2435: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 933 op/s 2024-09-21T10:46:58.456 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:58 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:46:58.456 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:46:58 smithi070 ceph-mon[107385]: osdmap e1394: 8 total, 8 up, 7 in 2024-09-21T10:46:58.488 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:58.531 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:58.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:58.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:58.633 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:58.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:58.673 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:58.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:58.774 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:58.798 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:58.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:58.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:58 smithi160 ceph-mon[101879]: pgmap v2435: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 933 op/s 2024-09-21T10:46:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:58 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:46:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:46:58 smithi160 ceph-mon[101879]: osdmap e1394: 8 total, 8 up, 7 in 2024-09-21T10:46:58.921 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:58.954 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:58.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:58.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:59.056 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:59.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:59.102 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:59.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:59.150 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:59.203 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:59.242 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:59.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:59.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:59.344 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:59.385 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:59.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:59.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:59.486 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:59.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:59.522 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:59.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:59.561 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:46:59.562 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T10:46:59.562 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T10:46:59.562 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T10:46:59.562 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:46:59.599 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:46:59.622 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:59.714 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:46:59.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:59.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:59.816 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:59.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:59.854 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:59.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:59.926 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:59.956 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:59.985 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:46:59.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:59.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:00.087 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:00.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:00.122 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:00.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:00.223 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:00.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:00.259 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:00.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:00.360 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:00.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:00.400 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:00.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:00.501 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:00.533 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:00.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:00.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:00.635 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:00.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:00.669 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:00.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:00.769 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:00.798 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:00.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:00.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:00 smithi160 ceph-mon[101879]: pgmap v2437: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 862 op/s 2024-09-21T10:47:00.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:00 smithi070 ceph-mon[112139]: pgmap v2437: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 862 op/s 2024-09-21T10:47:00.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:00 smithi070 ceph-mon[107385]: pgmap v2437: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 862 op/s 2024-09-21T10:47:00.900 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:00.928 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:00.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:00.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:01.030 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:01.068 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:01.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:01.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:01.170 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:01.191 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:47:01.199 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:01.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:01.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:01.301 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:01.339 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:01.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:01.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:01.441 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:01.483 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:01.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:01.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:01.585 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:01.620 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:01.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:01.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:01.722 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:01.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:01.755 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:01.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:01.856 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:01.887 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:01.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:01.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:01.989 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:02.027 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:02.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:02.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:02.129 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:02.157 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:02.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:02.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:02.262 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:02.299 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:02.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:02.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:02.400 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:02.436 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:02.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:02.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:02.537 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:02.575 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:02.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:02.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:02.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:02 smithi070 ceph-mon[107385]: pgmap v2438: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 827 op/s 2024-09-21T10:47:02.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:02 smithi070 ceph-mon[112139]: pgmap v2438: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 827 op/s 2024-09-21T10:47:02.677 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:02.719 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:02.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:02.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:02.820 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:02 smithi160 ceph-mon[101879]: pgmap v2438: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 827 op/s 2024-09-21T10:47:02.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:02.858 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:02.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:02.959 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:02.993 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:02.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:02.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:03.095 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:03.136 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:03.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:03.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:03.237 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:03.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:03.274 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:03.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:03.375 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:03.416 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:03.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:03.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:03.518 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:03.552 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:03.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:03.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:03.555 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:47:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:47:03] "GET /metrics HTTP/1.1" 200 36606 "" "Prometheus/2.51.0" 2024-09-21T10:47:03.654 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:03.680 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:03.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:03.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:03.781 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:03.823 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:03.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:03.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:03.924 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:03.952 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:03.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:03.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:04.054 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:04.104 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:04.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:04.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:04.205 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:04.240 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:04.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:04.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:04.342 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:04.352 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:04.353 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:04.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:04.375 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:04.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:04.476 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:04.521 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:04.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:04.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:04.593 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:04 smithi070 ceph-mon[112139]: pgmap v2439: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 949 op/s 2024-09-21T10:47:04.594 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:04 smithi070 ceph-mon[107385]: pgmap v2439: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 949 op/s 2024-09-21T10:47:04.624 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:04.658 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:04.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:04.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:04.759 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:04.787 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:04 smithi160 ceph-mon[101879]: pgmap v2439: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 949 op/s 2024-09-21T10:47:04.795 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:04.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:04.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:04.896 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:04.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:04.942 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:04.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:05.044 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:05.081 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:05.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:05.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:05.182 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:05.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:05.209 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:05.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:05.310 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:05.350 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:05.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:05.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:05.444 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T10:47:05.449 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T10:47:05.452 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:05.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:05.486 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:05.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:05.587 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:05.631 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:05.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:05.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:05.732 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:05.766 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:05.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:05.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:05 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1016048993' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:47:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:05 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:47:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:05 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4041191399' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:47:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:47:05.868 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:05.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:05 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1016048993' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:47:05.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:05 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:47:05.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:05 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041191399' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:47:05.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:47:05.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:05 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1016048993' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:47:05.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:05 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:47:05.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:05 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041191399' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:47:05.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:47:05.902 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:05.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:05.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:06.003 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:06.037 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:06.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:06.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:06.139 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:06.175 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:06.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:06.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:06.276 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:06.305 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:06.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:06.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:06.407 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:06.428 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:06.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:06.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:06.530 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:06.572 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:06.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:06.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:06.674 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:06.705 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:06.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:06.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:06.807 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:06.836 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:06.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:06.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:06 smithi160 ceph-mon[101879]: pgmap v2440: 89 pgs: 89 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 956 op/s 2024-09-21T10:47:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:06 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:47:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:06 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:47:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:06 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4041191399' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:47:06.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:06 smithi160 ceph-mon[101879]: osdmap e1395: 8 total, 8 up, 7 in 2024-09-21T10:47:06.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:06 smithi070 ceph-mon[112139]: pgmap v2440: 89 pgs: 89 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 956 op/s 2024-09-21T10:47:06.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:06 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:47:06.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:06 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:47:06.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:06 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4041191399' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:47:06.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:06 smithi070 ceph-mon[112139]: osdmap e1395: 8 total, 8 up, 7 in 2024-09-21T10:47:06.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:06 smithi070 ceph-mon[107385]: pgmap v2440: 89 pgs: 89 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 956 op/s 2024-09-21T10:47:06.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:06 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:47:06.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:06 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:47:06.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:06 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4041191399' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:47:06.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:06 smithi070 ceph-mon[107385]: osdmap e1395: 8 total, 8 up, 7 in 2024-09-21T10:47:06.937 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:06.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:06.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:07.076 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:07.104 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:07.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:07.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:07.206 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:07.237 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:07.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:07.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:07.339 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:07.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:07.370 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:07.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:07.471 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:07.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:07.499 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:07.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:07.599 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:07.639 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:07.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:07.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:07.741 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:07.779 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:07.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:07.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:07.881 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:07.906 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:07.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:07.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:08.008 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:08.046 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:08.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:08.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:08.148 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:08.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:08.177 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:08.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:08.278 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:08.312 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:08.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:08.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:08.413 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:08.444 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:08.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:08.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:08.546 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:08.581 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:08.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:08.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:08 smithi070 ceph-mon[107385]: pgmap v2442: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-21T10:47:08.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:08 smithi070 ceph-mon[112139]: pgmap v2442: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-21T10:47:08.683 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:08.729 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:08.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:08.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:08.831 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:08.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:08 smithi160 ceph-mon[101879]: pgmap v2442: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-21T10:47:08.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:08.869 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:08.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:08.970 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:09.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:09.004 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:09.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:09.105 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:09.134 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:09.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:09.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:09.235 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:09.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:09.262 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:09.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:09.362 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:09.389 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:09.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:09.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:09.490 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:09.527 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:09.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:09.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:09.628 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:09.666 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:09.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:09.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:09.768 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:09.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:09.799 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:09.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:09.822 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 4 1 2024-09-21T10:47:09.900 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:09.940 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:09.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:09.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:10.042 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:10.072 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:10.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:10.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:10.173 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:10.209 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:10.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:10.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:10.310 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:10.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:10.357 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:10.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:10.458 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:10.499 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:10.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:10.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:10.601 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:10.604 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:10 smithi070 ceph-mon[107385]: pgmap v2443: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 982 op/s 2024-09-21T10:47:10.604 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:10 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:47:10.604 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:10 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:47:10.604 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:10 smithi070 ceph-mon[112139]: pgmap v2443: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 982 op/s 2024-09-21T10:47:10.604 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:10 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:47:10.604 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:10 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:47:10.633 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:10.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:10.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:10.735 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:10.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:10.770 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:10.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:10.812 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:47:10.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:10 smithi160 ceph-mon[101879]: pgmap v2443: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 982 op/s 2024-09-21T10:47:10.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:10 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:47:10.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:10 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:47:10.871 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:10.902 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:10.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:10.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:11.004 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:11.045 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:11.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:11.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:11.147 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:11.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:11.192 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:11.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:11.293 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:11.327 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:11.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:11.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:11.429 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:11.465 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:11.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:11.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:11.566 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:11.598 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:11.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:11.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:11.700 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:11.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:11.731 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:11.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:11.832 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:11 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:47:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:11 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:47:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:11 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:47:11.867 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:11.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:11.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:11.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:11 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:47:11.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:11 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:47:11.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:11 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:47:11.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:11 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:47:11.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:11 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:47:11.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:11 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:47:11.968 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:12.000 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:12.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:12.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:12.102 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:12.132 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:12.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:12.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:12.234 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:12.261 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:12.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:12.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:12.363 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:12.403 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:12.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:12.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:12.504 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:12.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:12.539 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:12.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:12.540 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:47:12.540 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:47:12.560 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:12.561 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:12.576 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:47:12.576 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:47:12.608 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:47:12.608 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:47:12.609 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:12 smithi070 ceph-mon[112139]: pgmap v2444: 89 pgs: 89 active+clean; 4.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 987 op/s 2024-09-21T10:47:12.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:12 smithi070 ceph-mon[107385]: pgmap v2444: 89 pgs: 89 active+clean; 4.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 987 op/s 2024-09-21T10:47:12.640 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:12.641 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:47:12.642 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:47:12.676 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:47:12.677 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:47:12.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:12.687 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:12.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:12.707 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:47:12.707 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:47:12.750 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:47:12.750 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:47:12.782 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:47:12.782 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:47:12.789 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:12.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:12.823 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:12.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:12 smithi160 ceph-mon[101879]: pgmap v2444: 89 pgs: 89 active+clean; 4.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 987 op/s 2024-09-21T10:47:12.924 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:12.963 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:12.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:12.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:13.064 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:13.094 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:13.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:13.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:13.195 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:13.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:13.228 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:13.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:13.267 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:47:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:47:13] "GET /metrics HTTP/1.1" 200 36606 "" "Prometheus/2.51.0" 2024-09-21T10:47:13.329 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:13.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:13.360 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:13.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:13.461 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:13.489 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:13.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:13.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:13.576 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:13 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2869031135' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:47:13.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:13 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2869031135' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:47:13.591 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:13.633 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:13.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:13.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:13.735 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:13.767 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:13.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:13.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:13 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2869031135' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:47:13.870 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:13.908 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:13.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:13.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:14.010 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:14.053 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:14.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:14.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:14.155 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:14.169 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:47:14.169 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:47:14.203 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:14.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:14.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:14.305 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:14.340 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:14.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:14.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:14.442 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:14.477 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:14.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:14.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:14.579 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:14.609 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:14.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:14.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:14.710 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:14.746 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:14.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:14.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:14.763 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:14 smithi070 ceph-mon[107385]: pgmap v2445: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 978 op/s 2024-09-21T10:47:14.763 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:14 smithi070 ceph-mon[112139]: pgmap v2445: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 978 op/s 2024-09-21T10:47:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:14 smithi160 ceph-mon[101879]: pgmap v2445: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 978 op/s 2024-09-21T10:47:14.848 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:14.885 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:14.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:14.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:14.986 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:15.016 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:15.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:15.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:15.117 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:15.152 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:15.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:15.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:15.224 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:47:15.254 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:15.285 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:15.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:15.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:15.386 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:15.428 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:15.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:15.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:15.530 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:15.578 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:15.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:15.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:15.614 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:47:15.680 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:15.718 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:15.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:15.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:15.821 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:15.865 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:15.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:15.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:15.967 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:15.996 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:15.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:15.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:16.099 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:16.144 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:16.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:16.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:16.246 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:16.274 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:16.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:16.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:16.376 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:16.414 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:16.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:16.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:16.515 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:16.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:16.559 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:16.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:16.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:16 smithi070 ceph-mon[112139]: pgmap v2446: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 980 op/s 2024-09-21T10:47:16.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:16 smithi070 ceph-mon[107385]: pgmap v2446: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 980 op/s 2024-09-21T10:47:16.661 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:16.692 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:16.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:16.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:16.794 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:16.826 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:16.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:16.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:16.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:16 smithi160 ceph-mon[101879]: pgmap v2446: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 980 op/s 2024-09-21T10:47:16.927 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:16.955 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:16.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:16.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:17.057 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:17.078 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:17.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:17.130 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:17.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:17.231 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:17.265 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:17.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:17.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:17.366 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:17.397 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:17.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:17.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:17.499 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:17.530 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:17.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:17.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:17.594 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:17.632 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:17.678 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:17.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:17.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:17.780 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:17.825 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:17.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:17.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:17.927 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:17.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:17.960 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:17.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:18.061 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:18.093 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:18.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:18.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:18.195 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:18.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:18.227 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:18.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:18.328 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:18.362 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:18.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:18.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:18.464 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:18.513 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:18.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:18.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:18.615 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:18.644 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:18.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:18.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:18.746 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:18.782 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:18.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:18.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:18.799 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:18 smithi070 ceph-mon[112139]: pgmap v2447: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-21T10:47:18.799 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:18 smithi070 ceph-mon[107385]: pgmap v2447: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-21T10:47:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:18 smithi160 ceph-mon[101879]: pgmap v2447: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-21T10:47:18.884 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:18.915 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:18.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:18.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:19.017 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:19.050 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:19.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:19.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:19.152 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:19.179 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:19.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:19.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:19.281 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:19.378 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:19.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:19.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:19.480 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T10:47:19.480 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:19.507 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:19.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:19.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:19.611 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:19.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:19 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2072118618' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:47:19.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:19 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:47:19.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:19 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2072118618' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:47:19.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:19 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:47:19.655 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:19.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:19.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:19.758 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:19.797 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:19.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:19.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:19 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2072118618' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:47:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:19 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:47:19.899 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:19.929 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:19.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:19.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:20.031 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:20.073 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:20.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:20.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:20.175 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:20.206 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:20.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:20.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:20.308 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:20.339 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:20.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:20.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:20.440 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:20.480 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:20.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:20.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:20.504 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T10:47:20.583 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:20.613 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:20 smithi070 ceph-mon[107385]: pgmap v2448: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 819 op/s 2024-09-21T10:47:20.613 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:20 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:47:20.613 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:20 smithi070 ceph-mon[107385]: osdmap e1396: 8 total, 8 up, 7 in 2024-09-21T10:47:20.614 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:20 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2754326351' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:47:20.614 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:20 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:47:20.614 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:47:20.614 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:20 smithi070 ceph-mon[112139]: pgmap v2448: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 819 op/s 2024-09-21T10:47:20.614 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:20 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:47:20.614 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:20 smithi070 ceph-mon[112139]: osdmap e1396: 8 total, 8 up, 7 in 2024-09-21T10:47:20.614 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:20 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2754326351' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:47:20.614 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:20 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:47:20.614 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:47:20.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:20.622 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:20.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:20.724 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:20.761 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:20.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:20.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:20 smithi160 ceph-mon[101879]: pgmap v2448: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 819 op/s 2024-09-21T10:47:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:20 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:47:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:20 smithi160 ceph-mon[101879]: osdmap e1396: 8 total, 8 up, 7 in 2024-09-21T10:47:20.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:20 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2754326351' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:47:20.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:20 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:47:20.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:47:20.863 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:20.895 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:20.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:20.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:20.996 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:21.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:21.030 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:21.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:21.130 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:21.163 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:21.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:21.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:21.264 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:21.295 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:21.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:21.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:21.396 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:21.422 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:21.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:21.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:21.524 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:21.560 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:21.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:21.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:21.662 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:21.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:21.699 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:21.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:21.773 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:21 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:47:21.774 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:21 smithi070 ceph-mon[107385]: Cluster is now healthy 2024-09-21T10:47:21.774 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:21 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:47:21.774 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:21 smithi070 ceph-mon[107385]: osdmap e1397: 8 total, 8 up, 7 in 2024-09-21T10:47:21.774 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:21 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:47:21.774 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:21 smithi070 ceph-mon[112139]: Cluster is now healthy 2024-09-21T10:47:21.774 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:21 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:47:21.774 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:21 smithi070 ceph-mon[112139]: osdmap e1397: 8 total, 8 up, 7 in 2024-09-21T10:47:21.801 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:21.816 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:47:21.816 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:47:21.839 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:21.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:21.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:21 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:47:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:21 smithi160 ceph-mon[101879]: Cluster is now healthy 2024-09-21T10:47:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:21 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:47:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:21 smithi160 ceph-mon[101879]: osdmap e1397: 8 total, 8 up, 7 in 2024-09-21T10:47:21.941 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:21.978 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:21.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:21.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:22.080 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:22.111 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:22.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:22.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:22.213 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:22.273 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:22.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:22.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:22.375 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:22.403 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:22.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:22.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:22.504 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:22.539 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:22.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:22.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:22.640 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:22.677 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:22.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:22.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:22.779 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:22.812 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:22.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:22.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:22 smithi160 ceph-mon[101879]: pgmap v2451: 89 pgs: 89 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 929 op/s 2024-09-21T10:47:22.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:22 smithi070 ceph-mon[107385]: pgmap v2451: 89 pgs: 89 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 929 op/s 2024-09-21T10:47:22.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:22 smithi070 ceph-mon[112139]: pgmap v2451: 89 pgs: 89 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 929 op/s 2024-09-21T10:47:22.914 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:22.957 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:22.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:22.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:23.059 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:23.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:23.091 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:23.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:23.192 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:23.217 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:23.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:23.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:23.319 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:23.357 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:23.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:23.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:23.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:47:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:47:23] "GET /metrics HTTP/1.1" 200 36601 "" "Prometheus/2.51.0" 2024-09-21T10:47:23.459 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:23.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:23.505 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:23.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:23.606 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:23.639 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:23.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:23.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:23.741 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:23.775 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:23.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:23.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:23.877 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:23.914 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:23.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:23.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:23.949 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:47:24.015 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:24.063 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:24.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:24.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:24.165 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:24.205 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:24.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:24.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:24.307 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:24.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:24.347 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:24.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:24.448 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:24.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:24.490 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:24.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:24.558 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:24 smithi070 ceph-mon[107385]: pgmap v2452: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.16k op/s 2024-09-21T10:47:24.559 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:24 smithi070 ceph-mon[112139]: pgmap v2452: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.16k op/s 2024-09-21T10:47:24.591 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:24.632 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:24.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:24.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:24.734 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:24.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:24.768 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:24.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:24 smithi160 ceph-mon[101879]: pgmap v2452: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.16k op/s 2024-09-21T10:47:24.869 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:24.906 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:24.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:24.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:25.007 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:25.028 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T10:47:25.049 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:25.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:25.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:25.152 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:25.188 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:25.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:25.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:25.289 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:25.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:25.337 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:25.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:25.354 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:25.362 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:47:25.362 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:47:25.402 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:47:25.402 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:47:25.437 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:47:25.437 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:47:25.439 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:25.471 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:47:25.471 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:47:25.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:25.477 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:25.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:25.506 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:47:25.506 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:47:25.535 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:47:25.536 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:47:25.536 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:25 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2032649539' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:47:25.536 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:25 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2032649539' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:47:25.562 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:47:25.562 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:47:25.578 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:25.598 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:47:25.598 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:47:25.609 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:25.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:25.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:25.711 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:25.747 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:25.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:25.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:25 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2032649539' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:47:25.849 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:25.883 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:25.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:25.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:25.985 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:26.013 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:26.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:26.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:26.115 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:26.146 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:26.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:26.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:26.248 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:26.279 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:26.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:26.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:26.317 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:47:26.381 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:26.423 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:26.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:26.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:26.524 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:26.552 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:26.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:26.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:26 smithi070 ceph-mon[107385]: pgmap v2453: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 703 op/s 2024-09-21T10:47:26.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:26 smithi070 ceph-mon[112139]: pgmap v2453: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 703 op/s 2024-09-21T10:47:26.656 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:26.678 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:26.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:26.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:26.780 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:26.804 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:26.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:26.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:26 smithi160 ceph-mon[101879]: pgmap v2453: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 703 op/s 2024-09-21T10:47:26.907 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:26.943 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:26.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:26.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:27.045 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:27.089 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:27.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:27.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:27.191 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:27.214 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:27.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:27.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:27.316 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:27.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:27.360 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:27.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:27.461 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:27.470 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:27.497 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:27.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:27.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:27.599 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:27.626 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:27.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:27.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:27.728 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:27.763 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:27.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:27.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:27.865 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:27.891 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:27.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:27.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:27.993 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:28.028 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:28.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:28.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:28.130 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:28.165 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:28.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:28.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:28.267 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:28.297 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:28.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:28.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:28.398 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:28.438 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:28.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:28.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:28.539 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:28.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:28.571 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:28.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:28.672 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:28.704 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:28.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:28.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:28.806 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:28 smithi160 ceph-mon[101879]: pgmap v2454: 89 pgs: 89 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.23k op/s 2024-09-21T10:47:28.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:28.845 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:28.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:28.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:28 smithi070 ceph-mon[107385]: pgmap v2454: 89 pgs: 89 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.23k op/s 2024-09-21T10:47:28.860 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:28 smithi070 ceph-mon[112139]: pgmap v2454: 89 pgs: 89 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.23k op/s 2024-09-21T10:47:28.946 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:28.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:28.980 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:28.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:29.081 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:29.115 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:29.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:29.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:29.217 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:29.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:29.269 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:29.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:29.369 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:29.406 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:29.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:29.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:29.446 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:47:29.508 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:29.544 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:29.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:29.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:29.646 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:29.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:29.687 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:29.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:29.746 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:47:29.746 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T10:47:29.746 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T10:47:29.746 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T10:47:29.747 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:47:29.788 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:29.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:29.821 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:29.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:29.922 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:29.959 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:29.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:29.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:30.060 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:30.081 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:30.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:30.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:30.183 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:30.205 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:30.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:30.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:30.307 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:30.328 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:30.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:30.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:30.430 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:30.453 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:30.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:30.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:30.555 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:30.585 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:30.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:30.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:30.687 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:30.708 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:30.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:30.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:30.810 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:30 smithi160 ceph-mon[101879]: pgmap v2455: 89 pgs: 89 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-21T10:47:30.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:30.845 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:30.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:30.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:30 smithi070 ceph-mon[107385]: pgmap v2455: 89 pgs: 89 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-21T10:47:30.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:30 smithi070 ceph-mon[112139]: pgmap v2455: 89 pgs: 89 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-21T10:47:30.946 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:30.977 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:30.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:30.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:31.079 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:31.106 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:31.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:31.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:31.208 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:31.243 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:31.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:31.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:31.345 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:31.374 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:31.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:31.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:31.476 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:31.505 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:31.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:31.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:31.606 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:31.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:31.643 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:31.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:31.744 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:31.787 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:31.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:31.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:31.889 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:31.913 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:31.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:31.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:32.015 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:32.044 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:32.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:32.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:32.146 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:32.183 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:32.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:32.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:32.285 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:32.321 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:32.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:32.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:32.423 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:32.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:32.470 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:32.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:32.571 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:32.609 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:32.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:32.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:32.711 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:32.738 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:32.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:32.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:32.813 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:32 smithi070 ceph-mon[107385]: pgmap v2456: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 939 op/s 2024-09-21T10:47:32.813 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:32 smithi070 ceph-mon[112139]: pgmap v2456: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 939 op/s 2024-09-21T10:47:32.814 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:47:32.841 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:32 smithi160 ceph-mon[101879]: pgmap v2456: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 939 op/s 2024-09-21T10:47:32.877 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:32.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:32.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:32.979 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:33.028 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:33.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:33.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:33.129 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:33.159 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:33.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:33.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:33.261 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:33.296 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:33.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:33.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:33.370 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:33.398 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:33.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:33.459 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:33.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:33.520 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:47:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:47:33] "GET /metrics HTTP/1.1" 200 36601 "" "Prometheus/2.51.0" 2024-09-21T10:47:33.560 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:33.587 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:33.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:33.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:33.689 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:33.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:33.728 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:33.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:33.829 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:33.862 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:33.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:33.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:33.964 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:33.998 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:33.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:33.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:34.100 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:34.126 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:34.136 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:34.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:34.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:34.238 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:34.269 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:34.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:34.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:34.370 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:34.402 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:34.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:34.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:34.504 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:34.514 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:34.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:34 smithi070 ceph-mon[107385]: pgmap v2457: 89 pgs: 89 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 999 op/s 2024-09-21T10:47:34.545 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:34.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:34.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:34.646 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:34.687 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:34.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:34.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:34.789 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:34.817 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:34 smithi070 ceph-mon[112139]: pgmap v2457: 89 pgs: 89 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 999 op/s 2024-09-21T10:47:34.828 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:34.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:34.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:34.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:34 smithi160 ceph-mon[101879]: pgmap v2457: 89 pgs: 89 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 999 op/s 2024-09-21T10:47:34.930 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:34.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:34.979 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:34.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:35.080 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:35.111 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:35.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:35.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:35.212 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:35.248 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:35.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:35.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:35.351 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:35.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:35.388 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:35.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:35.489 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:35.534 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:35.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:35.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:35.636 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:35.669 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:35.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:35.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:35.771 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:35.808 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:35.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:35.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:35.909 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:35.940 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:35.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:35.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:36.042 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:36.069 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:36.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:36.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:36.170 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:36.200 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:36.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:36.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:36.302 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:36.328 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:36.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:36.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:36.375 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T10:47:36.431 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:36.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:36.466 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:36.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:36.567 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:36.588 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:36.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:36.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:36 smithi070 ceph-mon[107385]: pgmap v2458: 89 pgs: 89 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 839 op/s 2024-09-21T10:47:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:47:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:47:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:36 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/265201810' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:47:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:36 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:47:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:36 smithi070 ceph-mon[112139]: pgmap v2458: 89 pgs: 89 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 839 op/s 2024-09-21T10:47:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:47:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:47:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:36 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/265201810' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:47:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:36 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:47:36.689 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:36.725 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:36.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:36.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:36.826 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:36 smithi160 ceph-mon[101879]: pgmap v2458: 89 pgs: 89 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 839 op/s 2024-09-21T10:47:36.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:47:36.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:47:36.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:36 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/265201810' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:47:36.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:36 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:47:36.860 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:36.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:36.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:36.962 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:36.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:36.994 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:36.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:37.095 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:37.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:37.142 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:37.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:37.243 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:37.279 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:37.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:37.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:37.380 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:37.411 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:37.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:37.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:37.512 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:37.539 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:37.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:37.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:37.641 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:37.672 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:37.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:37.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:37.774 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:37.811 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:37.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:37.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:37.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:37 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:47:37.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:37 smithi160 ceph-mon[101879]: osdmap e1398: 8 total, 8 up, 7 in 2024-09-21T10:47:37.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:37 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:47:37.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:37 smithi070 ceph-mon[107385]: osdmap e1398: 8 total, 8 up, 7 in 2024-09-21T10:47:37.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:37 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:47:37.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:37 smithi070 ceph-mon[112139]: osdmap e1398: 8 total, 8 up, 7 in 2024-09-21T10:47:37.908 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T10:47:37.913 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:37.985 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:37.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:37.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:38.087 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:38.130 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:38.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:38.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:38.232 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:38.266 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:38.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:38.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:38.368 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:38.409 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:38.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:38.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:38.485 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:38 smithi070 ceph-mon[107385]: pgmap v2460: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 939 op/s 2024-09-21T10:47:38.485 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:38 smithi070 ceph-mon[112139]: pgmap v2460: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 939 op/s 2024-09-21T10:47:38.511 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:38.548 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:38.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:38.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:38.649 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:38.681 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:38.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:38.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:38.783 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:38.819 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:38.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:38.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:38 smithi160 ceph-mon[101879]: pgmap v2460: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 939 op/s 2024-09-21T10:47:38.921 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:38.965 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:38.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:38.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:39.066 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:39.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:39.099 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:39.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:39.200 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:39.231 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:39.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:39.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:39.332 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:39.374 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:39.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:39.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:39.476 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:39.506 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:39.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:39.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:39.607 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:39.635 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:39.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:39.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:39.736 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:39.771 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:39.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:39.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:39.930 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:39.967 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:39.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:39.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:40.069 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:40.096 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:40.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:40.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:40.197 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:40.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:40.241 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:40.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:40.320 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:47:40.320 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T10:47:40.321 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T10:47:40.321 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T10:47:40.321 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.321 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.34884.0:2386 52.d 52:b3c43908:::rbd_data.88447b362a13.000000000000004f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 6a=[6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1263)", 2024-09-21T10:47:40.321 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:38:12.487036+0000", 2024-09-21T10:47:40.321 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 566.23469874800003, 2024-09-21T10:47:40.321 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.034639395000000003, 2024-09-21T10:47:40.321 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:47:40.321 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:47:40.321 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:40.321 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:47:40.321 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.34884", 2024-09-21T10:47:40.322 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1225916197", 2024-09-21T10:47:40.322 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 2386 2024-09-21T10:47:40.322 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.322 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:47:40.322 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.322 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:47:40.322 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:12.487036+0000", 2024-09-21T10:47:40.322 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:47:40.322 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.322 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.322 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:47:40.322 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:12.487036+0000", 2024-09-21T10:47:40.322 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:47:40.323 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.323 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.323 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:47:40.323 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:12.487040+0000", 2024-09-21T10:47:40.323 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8089999999999999e-06 2024-09-21T10:47:40.323 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.323 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.323 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:47:40.323 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:12.490498+0000", 2024-09-21T10:47:40.323 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0034577319999999998 2024-09-21T10:47:40.323 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.323 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.323 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:47:40.324 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:12.490528+0000", 2024-09-21T10:47:40.324 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.993e-05 2024-09-21T10:47:40.324 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.324 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.324 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:47:40.324 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:12.490543+0000", 2024-09-21T10:47:40.324 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5551999999999999e-05 2024-09-21T10:47:40.324 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.324 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.324 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:47:40.324 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:12.490651+0000", 2024-09-21T10:47:40.324 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000107507 2024-09-21T10:47:40.324 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.324 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.325 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:47:40.325 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:12.490862+0000", 2024-09-21T10:47:40.325 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00021107299999999999 2024-09-21T10:47:40.325 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.325 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.325 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:47:40.325 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:12.491054+0000", 2024-09-21T10:47:40.325 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00019217 2024-09-21T10:47:40.325 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.325 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.325 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:47:40.325 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:12.499904+0000", 2024-09-21T10:47:40.325 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0088499609999999999 2024-09-21T10:47:40.325 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.326 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.326 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:40.326 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:12.521444+0000", 2024-09-21T10:47:40.326 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.021540134999999998 2024-09-21T10:47:40.326 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.326 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.326 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:40.326 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:12.521598+0000", 2024-09-21T10:47:40.326 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000153604 2024-09-21T10:47:40.326 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.326 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.326 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:47:40.326 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:12.521640+0000", 2024-09-21T10:47:40.326 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2203000000000002e-05 2024-09-21T10:47:40.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.327 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:47:40.327 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:12.521676+0000", 2024-09-21T10:47:40.327 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5719000000000001e-05 2024-09-21T10:47:40.327 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:47:40.327 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:47:40.327 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:47:40.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.327 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.34884.0:2758 52.a e1270/1156 52:564f5839:::rbd_data.88447b362a13.0000000000000003:head v 1270'34, mlcod=1269'31)", 2024-09-21T10:47:40.327 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:38:19.547045+0000", 2024-09-21T10:47:40.327 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 559.17469016400003, 2024-09-21T10:47:40.327 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.026302763, 2024-09-21T10:47:40.328 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:47:40.328 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:47:40.328 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:40.328 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:47:40.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:47:40.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:19.547045+0000", 2024-09-21T10:47:40.328 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:47:40.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:47:40.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:19.547045+0000", 2024-09-21T10:47:40.328 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:47:40.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:47:40.329 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:19.547046+0000", 2024-09-21T10:47:40.329 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4729999999999999e-06 2024-09-21T10:47:40.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:47:40.329 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:19.567010+0000", 2024-09-21T10:47:40.329 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.019963758000000002 2024-09-21T10:47:40.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:47:40.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:19.567028+0000", 2024-09-21T10:47:40.330 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7632999999999999e-05 2024-09-21T10:47:40.330 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.330 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:47:40.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:19.567036+0000", 2024-09-21T10:47:40.330 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7940000000000003e-06 2024-09-21T10:47:40.330 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.330 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:47:40.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:19.567099+0000", 2024-09-21T10:47:40.330 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3173e-05 2024-09-21T10:47:40.330 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.331 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.331 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:47:40.331 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:19.567133+0000", 2024-09-21T10:47:40.331 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4628000000000001e-05 2024-09-21T10:47:40.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.331 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.331 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:47:40.331 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:19.573273+0000", 2024-09-21T10:47:40.331 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0061395390000000003 2024-09-21T10:47:40.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.331 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.331 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:47:40.331 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:19.573348+0000", 2024-09-21T10:47:40.332 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4765000000000007e-05 2024-09-21T10:47:40.332 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:47:40.332 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:47:40.332 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:47:40.332 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.332 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.332 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.34884.0:2824 52.d 52:b623673b:::rbd_data.88447b362a13.0000000000000037:head [writefull 0~4194304] snapc 72=[72,71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1271)", 2024-09-21T10:47:40.332 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:38:20.519861+0000", 2024-09-21T10:47:40.333 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 558.20187372500004, 2024-09-21T10:47:40.333 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.043736738999999997, 2024-09-21T10:47:40.333 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:47:40.333 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:47:40.333 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:40.333 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:47:40.333 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.34884", 2024-09-21T10:47:40.333 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1225916197", 2024-09-21T10:47:40.333 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 2824 2024-09-21T10:47:40.333 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.334 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:47:40.335 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.335 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:47:40.335 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:20.519861+0000", 2024-09-21T10:47:40.335 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:47:40.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.335 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.335 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:47:40.335 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:20.519861+0000", 2024-09-21T10:47:40.335 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:47:40.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.335 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.335 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:47:40.335 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:20.519867+0000", 2024-09-21T10:47:40.336 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5690000000000004e-06 2024-09-21T10:47:40.336 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.336 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.336 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:47:40.336 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:20.524126+0000", 2024-09-21T10:47:40.336 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0042590190000000002 2024-09-21T10:47:40.336 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.336 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.336 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:47:40.336 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:20.524162+0000", 2024-09-21T10:47:40.336 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5617000000000001e-05 2024-09-21T10:47:40.336 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.336 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.337 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:47:40.337 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:20.524174+0000", 2024-09-21T10:47:40.337 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2499e-05 2024-09-21T10:47:40.337 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.337 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.337 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:47:40.337 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:20.524269+0000", 2024-09-21T10:47:40.337 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4549999999999994e-05 2024-09-21T10:47:40.337 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.337 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.337 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:47:40.337 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:20.524384+0000", 2024-09-21T10:47:40.337 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000115004 2024-09-21T10:47:40.338 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.338 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.338 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:47:40.338 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:20.524596+0000", 2024-09-21T10:47:40.338 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000212444 2024-09-21T10:47:40.338 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.338 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.338 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:47:40.338 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:20.534719+0000", 2024-09-21T10:47:40.338 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.010123056 2024-09-21T10:47:40.338 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.338 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.338 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:40.339 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:20.563408+0000", 2024-09-21T10:47:40.339 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.028688966999999999 2024-09-21T10:47:40.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.339 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:40.339 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:20.563500+0000", 2024-09-21T10:47:40.339 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2336000000000005e-05 2024-09-21T10:47:40.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.339 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:47:40.339 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:20.563557+0000", 2024-09-21T10:47:40.339 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6257e-05 2024-09-21T10:47:40.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.340 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.340 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:47:40.340 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:20.563598+0000", 2024-09-21T10:47:40.340 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.1421000000000002e-05 2024-09-21T10:47:40.340 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:47:40.340 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:47:40.340 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:47:40.340 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.340 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.340 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.34884.0:2926 52.d 52:b623673b:::rbd_data.88447b362a13.0000000000000037:head [write 59615~4134689] snapc 74=[74,73,72,71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1273)", 2024-09-21T10:47:40.340 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:38:22.535246+0000", 2024-09-21T10:47:40.340 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 556.18648924000001, 2024-09-21T10:47:40.340 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.054657022, 2024-09-21T10:47:40.340 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:47:40.341 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:47:40.341 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:40.341 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:47:40.341 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.34884", 2024-09-21T10:47:40.341 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1225916197", 2024-09-21T10:47:40.341 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 2926 2024-09-21T10:47:40.341 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.341 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:47:40.341 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.341 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:47:40.341 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.535246+0000", 2024-09-21T10:47:40.341 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:47:40.341 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.342 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.342 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:47:40.342 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.535246+0000", 2024-09-21T10:47:40.342 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:47:40.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.342 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.342 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:47:40.342 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.535251+0000", 2024-09-21T10:47:40.342 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6959999999999996e-06 2024-09-21T10:47:40.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.342 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.342 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:47:40.343 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.539968+0000", 2024-09-21T10:47:40.343 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0047178569999999998 2024-09-21T10:47:40.343 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.343 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.343 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:47:40.343 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.539987+0000", 2024-09-21T10:47:40.343 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8411e-05 2024-09-21T10:47:40.343 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.343 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.343 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:47:40.343 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.539998+0000", 2024-09-21T10:47:40.343 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1036000000000001e-05 2024-09-21T10:47:40.343 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.343 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.344 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:47:40.344 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.542215+0000", 2024-09-21T10:47:40.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002217643 2024-09-21T10:47:40.344 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.344 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:47:40.344 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.542298+0000", 2024-09-21T10:47:40.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2072999999999998e-05 2024-09-21T10:47:40.344 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.344 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:47:40.344 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.542472+0000", 2024-09-21T10:47:40.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017407499999999999 2024-09-21T10:47:40.344 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.345 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:47:40.345 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.551704+0000", 2024-09-21T10:47:40.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0092322129999999995 2024-09-21T10:47:40.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.345 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:40.345 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.589424+0000", 2024-09-21T10:47:40.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.037720048999999999 2024-09-21T10:47:40.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.345 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:40.345 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.589775+0000", 2024-09-21T10:47:40.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00035111000000000002 2024-09-21T10:47:40.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:47:40.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.589845+0000", 2024-09-21T10:47:40.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9747999999999997e-05 2024-09-21T10:47:40.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.346 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:47:40.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.589903+0000", 2024-09-21T10:47:40.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8111e-05 2024-09-21T10:47:40.346 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:47:40.346 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:47:40.346 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:47:40.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.346 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.346 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.34884.0:2935 52.d 52:b3c43908:::rbd_data.88447b362a13.000000000000004f:head [write 0~2969837] snapc 74=[74,73,72,71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1273)", 2024-09-21T10:47:40.347 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:38:22.559101+0000", 2024-09-21T10:47:40.347 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 556.16263417499999, 2024-09-21T10:47:40.347 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.030923540999999999, 2024-09-21T10:47:40.347 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:47:40.347 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:47:40.347 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:40.347 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:47:40.347 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.34884", 2024-09-21T10:47:40.347 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1225916197", 2024-09-21T10:47:40.347 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 2935 2024-09-21T10:47:40.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.347 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:47:40.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.347 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:47:40.348 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.559101+0000", 2024-09-21T10:47:40.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:47:40.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.348 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:47:40.348 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.559101+0000", 2024-09-21T10:47:40.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:47:40.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.348 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:47:40.348 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.559102+0000", 2024-09-21T10:47:40.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5829999999999999e-06 2024-09-21T10:47:40.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.348 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:47:40.348 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.559896+0000", 2024-09-21T10:47:40.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00079337799999999997 2024-09-21T10:47:40.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.349 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:47:40.349 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.559912+0000", 2024-09-21T10:47:40.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6494000000000002e-05 2024-09-21T10:47:40.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.349 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:47:40.349 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.559920+0000", 2024-09-21T10:47:40.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8660000000000006e-06 2024-09-21T10:47:40.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.349 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:47:40.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.559975+0000", 2024-09-21T10:47:40.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4913000000000001e-05 2024-09-21T10:47:40.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:47:40.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.560052+0000", 2024-09-21T10:47:40.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6942999999999998e-05 2024-09-21T10:47:40.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:47:40.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.560174+0000", 2024-09-21T10:47:40.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000121965 2024-09-21T10:47:40.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.351 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:47:40.351 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.563761+0000", 2024-09-21T10:47:40.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.003586554 2024-09-21T10:47:40.351 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.351 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:40.351 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.589468+0000", 2024-09-21T10:47:40.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.025707047 2024-09-21T10:47:40.351 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.351 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:40.351 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.589945+0000", 2024-09-21T10:47:40.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000477696 2024-09-21T10:47:40.352 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.352 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.352 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:47:40.352 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.589983+0000", 2024-09-21T10:47:40.352 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7472000000000003e-05 2024-09-21T10:47:40.352 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.352 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.352 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:47:40.352 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:22.590024+0000", 2024-09-21T10:47:40.352 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.163e-05 2024-09-21T10:47:40.352 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:47:40.352 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:47:40.352 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:47:40.352 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.353 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.353 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.34884.0:2977 52.d 52:b623673b:::rbd_data.88447b362a13.0000000000000037:head [write 0~2374055] snapc 75=[75,74,73,72,71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1274)", 2024-09-21T10:47:40.353 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:38:23.538457+0000", 2024-09-21T10:47:40.353 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 555.18327767799997, 2024-09-21T10:47:40.353 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.028012727000000001, 2024-09-21T10:47:40.353 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:47:40.353 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:47:40.353 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:40.353 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:47:40.353 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.34884", 2024-09-21T10:47:40.353 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1225916197", 2024-09-21T10:47:40.353 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 2977 2024-09-21T10:47:40.353 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.354 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:47:40.354 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.354 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:47:40.354 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:23.538457+0000", 2024-09-21T10:47:40.354 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:47:40.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.354 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.354 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:47:40.354 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:23.538457+0000", 2024-09-21T10:47:40.354 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:47:40.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.354 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.354 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:47:40.355 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:23.538460+0000", 2024-09-21T10:47:40.355 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6740000000000001e-06 2024-09-21T10:47:40.355 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.355 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.355 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:47:40.355 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:23.540193+0000", 2024-09-21T10:47:40.355 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0017329610000000001 2024-09-21T10:47:40.355 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.355 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.355 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:47:40.355 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:23.540230+0000", 2024-09-21T10:47:40.355 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7100000000000001e-05 2024-09-21T10:47:40.355 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.355 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.355 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:47:40.356 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:23.540243+0000", 2024-09-21T10:47:40.356 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3010000000000001e-05 2024-09-21T10:47:40.356 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.356 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.356 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:47:40.356 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:23.540348+0000", 2024-09-21T10:47:40.356 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010535300000000001 2024-09-21T10:47:40.356 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.356 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.356 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:47:40.356 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:23.540566+0000", 2024-09-21T10:47:40.356 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00021741600000000001 2024-09-21T10:47:40.356 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.356 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.357 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:47:40.357 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:23.540859+0000", 2024-09-21T10:47:40.357 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000293357 2024-09-21T10:47:40.357 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.357 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.357 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:47:40.357 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:23.545890+0000", 2024-09-21T10:47:40.357 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0050309439999999999 2024-09-21T10:47:40.357 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.357 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:40.358 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:23.561865+0000", 2024-09-21T10:47:40.358 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015974545999999999 2024-09-21T10:47:40.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:40.358 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:23.566384+0000", 2024-09-21T10:47:40.358 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0045195050000000001 2024-09-21T10:47:40.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:47:40.358 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:23.566430+0000", 2024-09-21T10:47:40.358 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6198000000000001e-05 2024-09-21T10:47:40.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:47:40.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:23.566470+0000", 2024-09-21T10:47:40.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9663000000000003e-05 2024-09-21T10:47:40.359 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:47:40.359 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:47:40.359 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:47:40.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.359 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.34884.0:3111 52.d 52:b3c43908:::rbd_data.88447b362a13.000000000000004f:head [writefull 0~4194304] snapc 77=[77,76,75,74,73,72,71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1276)", 2024-09-21T10:47:40.359 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:38:25.619770+0000", 2024-09-21T10:47:40.359 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 553.101965135, 2024-09-21T10:47:40.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.038165676000000003, 2024-09-21T10:47:40.360 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:47:40.360 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:47:40.360 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:40.360 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:47:40.360 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.34884", 2024-09-21T10:47:40.360 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1225916197", 2024-09-21T10:47:40.360 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 3111 2024-09-21T10:47:40.360 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.360 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:47:40.360 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.360 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:47:40.360 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:25.619770+0000", 2024-09-21T10:47:40.360 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:47:40.360 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.361 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.361 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:47:40.361 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:25.619770+0000", 2024-09-21T10:47:40.361 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:47:40.361 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.361 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.361 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:47:40.361 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:25.619772+0000", 2024-09-21T10:47:40.361 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.686e-06 2024-09-21T10:47:40.361 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.361 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.361 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:47:40.361 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:25.621012+0000", 2024-09-21T10:47:40.361 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001240578 2024-09-21T10:47:40.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.362 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:47:40.362 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:25.621027+0000", 2024-09-21T10:47:40.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4827999999999999e-05 2024-09-21T10:47:40.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.362 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:47:40.362 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:25.621036+0000", 2024-09-21T10:47:40.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6989999999999992e-06 2024-09-21T10:47:40.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.362 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:47:40.363 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:25.621091+0000", 2024-09-21T10:47:40.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5127000000000003e-05 2024-09-21T10:47:40.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.363 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:47:40.363 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:25.621178+0000", 2024-09-21T10:47:40.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7278e-05 2024-09-21T10:47:40.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.363 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:47:40.363 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:25.621319+0000", 2024-09-21T10:47:40.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014060799999999999 2024-09-21T10:47:40.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.364 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:47:40.364 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:25.626912+0000", 2024-09-21T10:47:40.364 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0055928549999999999 2024-09-21T10:47:40.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.364 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:40.364 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:25.653889+0000", 2024-09-21T10:47:40.364 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.026977458999999999 2024-09-21T10:47:40.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.364 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:40.364 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:25.657842+0000", 2024-09-21T10:47:40.364 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0039532559999999996 2024-09-21T10:47:40.365 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.365 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.365 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:47:40.365 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:25.657886+0000", 2024-09-21T10:47:40.365 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3378999999999999e-05 2024-09-21T10:47:40.365 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.365 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.365 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:47:40.365 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:25.657936+0000", 2024-09-21T10:47:40.365 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9923000000000003e-05 2024-09-21T10:47:40.365 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:47:40.365 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:47:40.365 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:47:40.365 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.366 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.366 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.34884.0:3161 52.a e1277/1156 52:512cabc6:::rbd_data.88447b362a13.000000000000005f:head v 1277'40, mlcod=1274'36)", 2024-09-21T10:47:40.366 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:38:26.601307+0000", 2024-09-21T10:47:40.366 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 552.12042759300004, 2024-09-21T10:47:40.366 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.021680451, 2024-09-21T10:47:40.366 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:47:40.366 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:47:40.366 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:40.366 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:47:40.366 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.366 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:47:40.366 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:26.601307+0000", 2024-09-21T10:47:40.366 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:47:40.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.367 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.367 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:47:40.367 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:26.601307+0000", 2024-09-21T10:47:40.367 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:47:40.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.367 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.367 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:47:40.367 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:26.601308+0000", 2024-09-21T10:47:40.367 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5200000000000001e-07 2024-09-21T10:47:40.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.367 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.368 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:47:40.368 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:26.616010+0000", 2024-09-21T10:47:40.368 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.014702213 2024-09-21T10:47:40.368 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.368 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.368 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:47:40.368 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:26.616031+0000", 2024-09-21T10:47:40.368 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1364000000000001e-05 2024-09-21T10:47:40.368 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.368 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.368 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:47:40.368 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:26.616041+0000", 2024-09-21T10:47:40.368 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.859e-06 2024-09-21T10:47:40.368 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.369 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.369 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:47:40.369 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:26.616117+0000", 2024-09-21T10:47:40.369 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5496999999999999e-05 2024-09-21T10:47:40.369 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.369 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.369 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:47:40.369 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:26.616140+0000", 2024-09-21T10:47:40.369 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.304e-05 2024-09-21T10:47:40.369 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.369 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.369 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:47:40.369 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:26.622947+0000", 2024-09-21T10:47:40.369 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0068077659999999998 2024-09-21T10:47:40.370 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.370 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.370 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:47:40.370 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:26.622988+0000", 2024-09-21T10:47:40.370 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0559999999999998e-05 2024-09-21T10:47:40.370 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:47:40.370 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:47:40.370 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:47:40.370 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.370 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.370 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.34884.0:3194 52.a e1278/1156 52:512cabc6:::rbd_data.88447b362a13.000000000000005f:head v 1278'42, mlcod=1275'38)", 2024-09-21T10:47:40.370 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:38:27.572499+0000", 2024-09-21T10:47:40.370 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 551.14923615999999, 2024-09-21T10:47:40.371 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.023783367999999999, 2024-09-21T10:47:40.371 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:47:40.371 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:47:40.371 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:40.371 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:47:40.371 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.371 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:47:40.371 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:27.572499+0000", 2024-09-21T10:47:40.371 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:47:40.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.371 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.371 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:47:40.371 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:27.572499+0000", 2024-09-21T10:47:40.371 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:47:40.372 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.372 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.372 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:47:40.372 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:27.572499+0000", 2024-09-21T10:47:40.372 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4999999999999999e-07 2024-09-21T10:47:40.372 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.372 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.372 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:47:40.372 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:27.587126+0000", 2024-09-21T10:47:40.372 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.01462639 2024-09-21T10:47:40.372 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.372 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.372 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:47:40.372 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:27.587146+0000", 2024-09-21T10:47:40.373 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0333999999999999e-05 2024-09-21T10:47:40.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.373 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:47:40.373 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:27.587156+0000", 2024-09-21T10:47:40.373 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.6209999999999999e-06 2024-09-21T10:47:40.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.373 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:47:40.373 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:27.587232+0000", 2024-09-21T10:47:40.373 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6386999999999999e-05 2024-09-21T10:47:40.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.373 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:47:40.373 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:27.587257+0000", 2024-09-21T10:47:40.374 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4822999999999999e-05 2024-09-21T10:47:40.374 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.374 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.374 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:47:40.374 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:27.596199+0000", 2024-09-21T10:47:40.374 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0089422270000000005 2024-09-21T10:47:40.374 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.374 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.374 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:47:40.374 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:27.596282+0000", 2024-09-21T10:47:40.374 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3336000000000003e-05 2024-09-21T10:47:40.374 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:47:40.374 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:47:40.374 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:47:40.375 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.375 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.375 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.34884.0:6931 52.d 52:b65311e1:::rbd_data.88444b16d5ef.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1299)", 2024-09-21T10:47:40.375 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:38:48.220043+0000", 2024-09-21T10:47:40.375 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 530.50169172400001, 2024-09-21T10:47:40.375 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.050977930999999997, 2024-09-21T10:47:40.375 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:47:40.375 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:47:40.375 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:40.375 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:47:40.375 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.34884", 2024-09-21T10:47:40.375 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1225916197", 2024-09-21T10:47:40.375 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 6931 2024-09-21T10:47:40.375 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.376 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:47:40.376 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.376 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:47:40.376 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:48.220043+0000", 2024-09-21T10:47:40.376 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:47:40.376 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.376 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.376 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:47:40.376 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:48.220043+0000", 2024-09-21T10:47:40.376 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:47:40.376 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.376 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.376 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:47:40.376 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:48.220045+0000", 2024-09-21T10:47:40.376 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7269999999999999e-06 2024-09-21T10:47:40.377 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.377 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:47:40.377 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:48.222548+0000", 2024-09-21T10:47:40.377 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025032409999999998 2024-09-21T10:47:40.377 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.377 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:47:40.377 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:48.222570+0000", 2024-09-21T10:47:40.377 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1370999999999999e-05 2024-09-21T10:47:40.377 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.377 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:47:40.378 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:48.222581+0000", 2024-09-21T10:47:40.378 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1721e-05 2024-09-21T10:47:40.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.378 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.378 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:47:40.378 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:48.222687+0000", 2024-09-21T10:47:40.378 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000105171 2024-09-21T10:47:40.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.378 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.378 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:47:40.378 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:48.222988+0000", 2024-09-21T10:47:40.378 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00030140700000000001 2024-09-21T10:47:40.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.378 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.379 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:47:40.379 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:48.223237+0000", 2024-09-21T10:47:40.379 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00024855 2024-09-21T10:47:40.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.379 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.379 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:47:40.379 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:48.233813+0000", 2024-09-21T10:47:40.379 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.010576479999999999 2024-09-21T10:47:40.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.379 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.379 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:40.379 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:48.269812+0000", 2024-09-21T10:47:40.379 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.035998752000000002 2024-09-21T10:47:40.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.381 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.381 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:40.381 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:48.270916+0000", 2024-09-21T10:47:40.381 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001104627 2024-09-21T10:47:40.381 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.381 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.381 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:47:40.381 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:48.270966+0000", 2024-09-21T10:47:40.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9796999999999998e-05 2024-09-21T10:47:40.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.382 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.382 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:47:40.382 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:38:48.271021+0000", 2024-09-21T10:47:40.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5087000000000003e-05 2024-09-21T10:47:40.382 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:47:40.382 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:47:40.382 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:47:40.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.382 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.382 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.25178.0:119 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e1301)", 2024-09-21T10:47:40.382 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:39:50.855050+0000", 2024-09-21T10:47:40.382 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 467.86668545800001, 2024-09-21T10:47:40.383 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00023059900000000001, 2024-09-21T10:47:40.383 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:47:40.383 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:47:40.383 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:47:40.383 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:47:40.383 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.25178", 2024-09-21T10:47:40.383 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/2007327347", 2024-09-21T10:47:40.383 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 119 2024-09-21T10:47:40.383 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.383 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:47:40.383 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.383 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:47:40.383 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:39:50.855050+0000", 2024-09-21T10:47:40.383 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:47:40.384 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.384 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.384 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:47:40.384 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:39:50.855050+0000", 2024-09-21T10:47:40.384 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:47:40.384 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.384 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.384 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:47:40.384 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:39:50.855052+0000", 2024-09-21T10:47:40.384 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.99e-06 2024-09-21T10:47:40.384 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.384 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.384 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:47:40.384 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:39:50.855061+0000", 2024-09-21T10:47:40.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.8910000000000007e-06 2024-09-21T10:47:40.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:47:40.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:39:50.855082+0000", 2024-09-21T10:47:40.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0939000000000001e-05 2024-09-21T10:47:40.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:47:40.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:39:50.855094+0000", 2024-09-21T10:47:40.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2021999999999999e-05 2024-09-21T10:47:40.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:47:40.386 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:39:50.855201+0000", 2024-09-21T10:47:40.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000106426 2024-09-21T10:47:40.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:47:40.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:47:40.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:47:40.386 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:39:50.855280+0000", 2024-09-21T10:47:40.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9331000000000003e-05 2024-09-21T10:47:40.386 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:47:40.386 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:47:40.386 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:47:40.386 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:47:40.386 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:47:40.387 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:47:40.387 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:40.415 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:40.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:40.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:40.517 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:40.557 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:40.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:40.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:40.660 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:40.694 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:40.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:40.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:40.796 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:40.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:40.823 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:40.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:40 smithi160 ceph-mon[101879]: pgmap v2461: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 939 op/s 2024-09-21T10:47:40.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:40 smithi070 ceph-mon[107385]: pgmap v2461: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 939 op/s 2024-09-21T10:47:40.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:40 smithi070 ceph-mon[112139]: pgmap v2461: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 939 op/s 2024-09-21T10:47:40.924 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:40.950 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:40.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:40.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:40.992 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:47:41.008 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:41.052 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:41.103 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:41.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:41.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:41.205 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:41.240 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:41.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:41.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:41.342 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:41.386 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:41.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:41.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:41.487 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:41.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:41.535 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:41.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:41.636 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:41.678 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:41.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:41.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:41.780 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:41.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:41.812 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:41.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:41.913 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:41.949 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:41.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:41.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:42.051 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:42.084 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:42.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:42.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:42.186 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:42.208 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:42.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:42.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:42.309 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:42.341 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:42.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:42.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:42.399 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T10:47:42.443 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:42.474 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:42.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:42.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:42.576 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:42.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:42.611 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:42.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:42.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:42 smithi070 ceph-mon[107385]: pgmap v2462: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 938 op/s 2024-09-21T10:47:42.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:42 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1795596124' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:47:42.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:42 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:47:42.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:42 smithi070 ceph-mon[112139]: pgmap v2462: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 938 op/s 2024-09-21T10:47:42.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:42 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1795596124' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:47:42.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:42 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:47:42.713 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:42.746 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:42.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:42.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:42 smithi160 ceph-mon[101879]: pgmap v2462: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 938 op/s 2024-09-21T10:47:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:42 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1795596124' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:47:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:42 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:47:42.848 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:42.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:42.893 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:42.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:42.994 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:43.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:43.027 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:43.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:43.128 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:43.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:43.160 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:43.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:43.261 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:43.303 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:43.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:43.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:43.405 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:43.410 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:47:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:47:43] "GET /metrics HTTP/1.1" 200 36607 "" "Prometheus/2.51.0" 2024-09-21T10:47:43.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:43.443 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:43.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:43.446 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:47:43.446 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:47:43.475 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:47:43.475 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:47:43.502 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:47:43.502 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:47:43.534 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:47:43.535 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:47:43.544 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:43.563 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:47:43.563 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:47:43.582 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:43.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:43.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:43.595 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:47:43.595 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:47:43.629 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:47:43.629 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:47:43.663 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:47:43.663 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:47:43.664 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:43 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:47:43.664 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:43 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:47:43.664 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:43 smithi070 ceph-mon[107385]: osdmap e1399: 8 total, 8 up, 7 in 2024-09-21T10:47:43.664 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:43 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:47:43.664 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:43 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:47:43.664 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:43 smithi070 ceph-mon[112139]: osdmap e1399: 8 total, 8 up, 7 in 2024-09-21T10:47:43.684 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:43.731 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:43.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:43.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:43.833 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:43 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:47:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:43 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:47:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:43 smithi160 ceph-mon[101879]: osdmap e1399: 8 total, 8 up, 7 in 2024-09-21T10:47:43.865 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:43.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:43.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:43.967 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:43.999 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:44.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:44.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:44.101 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:44.129 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:44.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:44.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:44.231 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:44.258 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:44.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:44.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:44.360 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:44.399 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:44.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:44.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:44.501 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:44.534 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:44.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:44.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:44.636 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:44.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:44.672 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:44.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:44.764 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:44.773 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:44.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:44.805 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:44.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:44 smithi160 ceph-mon[101879]: pgmap v2464: 89 pgs: 89 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.12k op/s 2024-09-21T10:47:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:44 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1172161935' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:47:44.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:44 smithi070 ceph-mon[107385]: pgmap v2464: 89 pgs: 89 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.12k op/s 2024-09-21T10:47:44.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:44 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1172161935' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:47:44.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:44 smithi070 ceph-mon[112139]: pgmap v2464: 89 pgs: 89 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.12k op/s 2024-09-21T10:47:44.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:44 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1172161935' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:47:44.907 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:44.934 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:44.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:44.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:45.036 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:45.068 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:45.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:45.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:45.169 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:45.203 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:45.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:45.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:45.209 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:47:45.304 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:45.346 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:45.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:45.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:45.447 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:45.477 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:45.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:45.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:45.578 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:45.608 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:45.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:45.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:45.709 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:45.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:45.753 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:45.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:45.854 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:45.894 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:45.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:45.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:45.995 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:46.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:46.026 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:46.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:46.127 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:46.159 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:46.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:46.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:46.261 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:46.305 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:46.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:46.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:46.407 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:46.439 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:47:46.447 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:46.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:46.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:46.549 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:46.588 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:46.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:46.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:46.690 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:46.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:46.736 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:46.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:46.837 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:46 smithi160 ceph-mon[101879]: pgmap v2465: 89 pgs: 89 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 682 op/s 2024-09-21T10:47:46.869 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:46.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:46.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:46.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:46 smithi070 ceph-mon[107385]: pgmap v2465: 89 pgs: 89 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 682 op/s 2024-09-21T10:47:46.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:46 smithi070 ceph-mon[112139]: pgmap v2465: 89 pgs: 89 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 682 op/s 2024-09-21T10:47:46.971 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:47.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:47.001 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:47.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:47.102 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:47.139 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:47.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:47.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:47.242 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:47.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:47.277 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:47.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:47.377 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:47.412 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:47.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:47.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:47.514 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:47.547 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:47.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:47.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:47.648 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:47.669 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:47.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:47.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:47.767 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T10:47:47.767 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:47 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/735103285' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:47:47.767 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:47 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:47:47.767 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:47 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/735103285' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:47:47.767 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:47 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:47:47.771 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:47.793 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:47.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:47.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:47.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:47 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/735103285' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:47:47.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:47 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:47:47.896 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:47.929 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:47.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:47.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:48.031 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:48.058 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:48.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:48.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:48.160 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:48.187 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:48.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:48.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:48.289 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:48.319 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:48.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:48.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:48.352 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:48.421 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:48.453 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:48.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:48.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:48.555 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:48.591 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:48.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:48.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:48.692 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:48.730 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:48.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:48.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:48.759 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:48 smithi070 ceph-mon[107385]: pgmap v2466: 89 pgs: 89 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 970 op/s 2024-09-21T10:47:48.759 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:48 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:47:48.759 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:48 smithi070 ceph-mon[107385]: osdmap e1400: 8 total, 8 up, 7 in 2024-09-21T10:47:48.759 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:48 smithi070 ceph-mon[112139]: pgmap v2466: 89 pgs: 89 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 970 op/s 2024-09-21T10:47:48.759 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:48 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:47:48.759 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:48 smithi070 ceph-mon[112139]: osdmap e1400: 8 total, 8 up, 7 in 2024-09-21T10:47:48.832 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:48.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:48 smithi160 ceph-mon[101879]: pgmap v2466: 89 pgs: 89 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 970 op/s 2024-09-21T10:47:48.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:48 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:47:48.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:48 smithi160 ceph-mon[101879]: osdmap e1400: 8 total, 8 up, 7 in 2024-09-21T10:47:48.871 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:48.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:48.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:48.973 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:48.994 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:48.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:48.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:49.096 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:49.126 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:49.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:49.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:49.228 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:49.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:49.261 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:49.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:49.362 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:49.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:49.383 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:49.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:49.484 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:49.505 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:49.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:49.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:49.608 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:49.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:49.629 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:49.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:49.730 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:49.754 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:49.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:49.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:49.856 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:49.894 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:49.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:49.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:49.996 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:50.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:50.031 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:50.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:50.133 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:50.172 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:50.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:50.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:50.274 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:50.318 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:50.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:50.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:50.420 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:50.469 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:50.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:50.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:50.571 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:50.603 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:50.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:50.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:50.705 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:50.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:50.740 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:50.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:50.761 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:50 smithi070 ceph-mon[107385]: pgmap v2468: 89 pgs: 89 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 919 op/s 2024-09-21T10:47:50.761 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:47:50.761 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:47:50.762 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:50 smithi070 ceph-mon[112139]: pgmap v2468: 89 pgs: 89 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 919 op/s 2024-09-21T10:47:50.762 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:47:50.762 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:47:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:50 smithi160 ceph-mon[101879]: pgmap v2468: 89 pgs: 89 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 919 op/s 2024-09-21T10:47:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:47:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:47:50.842 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:50.892 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:50.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:50.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:50.994 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:51.025 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:51.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:51.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:51.127 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:51.161 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:51.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:51.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:51.248 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T10:47:51.263 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:51.327 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:51.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:51.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:51.429 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:51.462 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:51.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:51.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:51.564 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:51.596 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:51.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:51.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:51.697 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:51.728 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:51.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:51.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:51.738 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:51.830 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:51.864 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:51.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:51.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:51.966 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:51.999 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:51.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:52.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:52.100 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:52.137 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:52.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:52.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:52.239 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:52.271 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:52.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:52.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:52.372 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:52.420 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:52.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:52.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:52.445 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T10:47:52.525 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:52.561 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:52.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:52.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:52.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:52 smithi070 ceph-mon[107385]: pgmap v2469: 89 pgs: 89 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 827 op/s 2024-09-21T10:47:52.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:52 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/835534646' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:47:52.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:52 smithi070 ceph-mon[112139]: pgmap v2469: 89 pgs: 89 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 827 op/s 2024-09-21T10:47:52.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:52 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/835534646' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:47:52.663 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:52.698 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:52.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:52.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:52.799 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:52.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:52.838 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:52.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:52 smithi160 ceph-mon[101879]: pgmap v2469: 89 pgs: 89 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 827 op/s 2024-09-21T10:47:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:52 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/835534646' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:47:52.939 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:52.975 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:52.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:52.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:53.077 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:53.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:53.107 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:53.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:53.115 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:47:53.115 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:47:53.208 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:53.240 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:53.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:53.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:53.342 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:53.378 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:53.378 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:47:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:47:53] "GET /metrics HTTP/1.1" 200 36601 "" "Prometheus/2.51.0" 2024-09-21T10:47:53.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:53.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:53.480 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:53.516 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:53.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:53.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:53.618 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:53.650 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:53.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:53.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:53.752 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:53.788 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:53.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:53.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:53.838 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:53 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:47:53.838 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:53 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/835534646' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:47:53.838 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:53 smithi070 ceph-mon[107385]: osdmap e1401: 8 total, 8 up, 7 in 2024-09-21T10:47:53.838 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:53 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:47:53.838 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:53 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/835534646' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:47:53.838 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:53 smithi070 ceph-mon[112139]: osdmap e1401: 8 total, 8 up, 7 in 2024-09-21T10:47:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:53 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:47:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:53 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/835534646' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:47:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:53 smithi160 ceph-mon[101879]: osdmap e1401: 8 total, 8 up, 7 in 2024-09-21T10:47:53.869 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:47:53.890 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:53.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:53.928 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:53.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:54.029 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:54.062 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:54.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:54.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:54.163 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:54.201 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:54.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:54.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:54.304 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:54.350 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:54.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:54.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:54.452 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:54.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:54.488 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:54.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:54.589 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:54.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:54.620 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:54.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:54.718 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:54 smithi070 ceph-mon[107385]: pgmap v2471: 89 pgs: 89 active+clean; 2.5 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-21T10:47:54.719 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:54 smithi070 ceph-mon[112139]: pgmap v2471: 89 pgs: 89 active+clean; 2.5 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-21T10:47:54.721 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:54.753 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:54.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:54.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:54.802 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:54.833 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:54 smithi160 ceph-mon[101879]: pgmap v2471: 89 pgs: 89 active+clean; 2.5 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-21T10:47:54.855 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:54.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:54.887 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:54.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:54.988 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:55.025 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:55.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:55.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:55.128 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:55.164 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:55.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:55.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:55.266 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:55.310 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:55.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:55.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:55.412 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:55.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:55.454 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:55.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:55.555 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:55.585 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:55.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:55.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:55.688 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:55.731 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:55.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:55.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:55.833 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:55.875 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:55.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:55.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:55.978 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:55.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:56.000 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:56.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:56.101 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:56.132 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:56.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:56.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:56.234 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:56.237 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:47:56.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:56.269 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:56.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:56.370 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:56.383 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:47:56.383 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:47:56.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:56.412 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:56.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:56.422 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:47:56.423 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:47:56.458 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:47:56.458 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:47:56.493 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:47:56.493 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:47:56.513 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:56.522 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:47:56.522 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:47:56.547 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:56.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:56.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:56.554 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:47:56.554 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:47:56.593 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:47:56.593 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:47:56.594 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:56 smithi070 ceph-mon[112139]: pgmap v2472: 89 pgs: 89 active+clean; 2.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 601 op/s 2024-09-21T10:47:56.594 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:56 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3285393967' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:47:56.594 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:56 smithi070 ceph-mon[107385]: pgmap v2472: 89 pgs: 89 active+clean; 2.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 601 op/s 2024-09-21T10:47:56.594 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:56 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3285393967' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:47:56.628 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:47:56.628 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:47:56.649 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:56.698 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:56.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:56.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:56.799 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:56.828 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:56.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:56.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:56 smithi160 ceph-mon[101879]: pgmap v2472: 89 pgs: 89 active+clean; 2.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 601 op/s 2024-09-21T10:47:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:56 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3285393967' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:47:56.930 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:56.974 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:56.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:56.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:57.076 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:57.101 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:57.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:57.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:57.204 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:57.240 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:57.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:57.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:57.342 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:57.374 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:57.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:57.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:57.476 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:57.510 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:57.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:57.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:57.612 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:57.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:57.644 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:57.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:57.745 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:57.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:57.773 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:57.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:57.874 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:57.905 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:57.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:57.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:58.006 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:58.044 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:58.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:58.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:58.146 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:58.188 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:58.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:58.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:58.290 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:58.320 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:58.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:58.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:58.422 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:58.453 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:58.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:58.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:58.555 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:58.588 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:58.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:58.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:58.690 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:58.730 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:58.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:58.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:58.832 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:47:58 smithi160 ceph-mon[101879]: pgmap v2473: 89 pgs: 89 active+clean; 2.2 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 958 op/s 2024-09-21T10:47:58.860 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:58.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:58.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:58.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:47:58 smithi070 ceph-mon[107385]: pgmap v2473: 89 pgs: 89 active+clean; 2.2 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 958 op/s 2024-09-21T10:47:58.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:47:58 smithi070 ceph-mon[112139]: pgmap v2473: 89 pgs: 89 active+clean; 2.2 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 958 op/s 2024-09-21T10:47:58.962 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:58.992 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:58.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:58.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:59.094 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:59.135 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:59.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:59.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:59.237 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:59.273 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:59.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:59.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:59.374 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:59.410 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:59.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:59.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:59.512 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:59.542 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:59.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:59.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:59.644 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:59.674 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:47:59.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:59.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:59.776 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:59.811 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:59.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:59.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:59.913 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:59.921 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:59.954 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:47:59.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:59.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:00.056 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:00.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:00.097 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:00.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:00.198 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:00.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:00.238 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:00.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:00.339 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:00.374 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:00.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:00.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:00.476 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:00.514 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:00.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:00.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:00.616 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:00.659 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:00.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:00.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:00.729 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:00 smithi070 ceph-mon[107385]: pgmap v2474: 89 pgs: 89 active+clean; 2.2 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 898 op/s 2024-09-21T10:48:00.729 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:00 smithi070 ceph-mon[112139]: pgmap v2474: 89 pgs: 89 active+clean; 2.2 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 898 op/s 2024-09-21T10:48:00.761 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:00.772 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:48:00.772 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T10:48:00.772 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T10:48:00.772 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T10:48:00.772 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:48:00.802 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:00.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:00.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:00 smithi160 ceph-mon[101879]: pgmap v2474: 89 pgs: 89 active+clean; 2.2 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 898 op/s 2024-09-21T10:48:00.904 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:00.941 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:00.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:00.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:01.042 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:01.073 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:01.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:01.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:01.175 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:01.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:01.206 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:01.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:01.307 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:01.345 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:01.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:01.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:01.451 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:01.484 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:01.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:01.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:01.485 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:48:01.489 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T10:48:01.586 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:01.618 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:01.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:01.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:01.719 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:01.760 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:01.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:01.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:01 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1320426008' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:48:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:01 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:48:01.862 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:01.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:01 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1320426008' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:48:01.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:01 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:48:01.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:01 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1320426008' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:48:01.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:01 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:48:01.891 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:01.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:01.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:01.939 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:48:01.993 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:02.032 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:02.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:02.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:02.134 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:02.154 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:02.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:02.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:02.256 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:02.301 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:02.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:02.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:02.403 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:02.447 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:02.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:02.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:02.549 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:02.588 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:02.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:02.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:02.690 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:02.717 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:02.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:02.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:02.819 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:02 smithi160 ceph-mon[101879]: pgmap v2475: 89 pgs: 89 active+clean; 2.0 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 927 op/s 2024-09-21T10:48:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:02 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:48:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:02 smithi160 ceph-mon[101879]: osdmap e1402: 8 total, 8 up, 7 in 2024-09-21T10:48:02.857 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:02.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:02.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:02.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:02 smithi070 ceph-mon[107385]: pgmap v2475: 89 pgs: 89 active+clean; 2.0 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 927 op/s 2024-09-21T10:48:02.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:02 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:48:02.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:02 smithi070 ceph-mon[107385]: osdmap e1402: 8 total, 8 up, 7 in 2024-09-21T10:48:02.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:02 smithi070 ceph-mon[112139]: pgmap v2475: 89 pgs: 89 active+clean; 2.0 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 927 op/s 2024-09-21T10:48:02.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:02 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:48:02.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:02 smithi070 ceph-mon[112139]: osdmap e1402: 8 total, 8 up, 7 in 2024-09-21T10:48:02.959 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:02.994 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:02.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:02.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:03.096 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:03.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:03.126 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:03.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:03.227 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:03.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:03.274 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:03.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:03.375 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:03.398 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:03.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:03.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:03.500 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:03.539 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:03.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:03.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:48:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:48:03] "GET /metrics HTTP/1.1" 200 36601 "" "Prometheus/2.51.0" 2024-09-21T10:48:03.642 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:03.678 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:03.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:03.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:03.780 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:03.815 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:03.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:03.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:03.916 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:03.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:03.959 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:03.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:04.060 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:04.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:04.095 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:04.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:04.196 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:04.225 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:04.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:04.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:04.327 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:04.357 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:04.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:04.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:04.459 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:04.514 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:04.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:04.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:04.616 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:04.656 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:04.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:04.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:04.758 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:04.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:04.799 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:04.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:04.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:04 smithi160 ceph-mon[101879]: pgmap v2477: 89 pgs: 89 active+clean; 1.9 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 958 op/s 2024-09-21T10:48:04.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:04 smithi070 ceph-mon[107385]: pgmap v2477: 89 pgs: 89 active+clean; 1.9 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 958 op/s 2024-09-21T10:48:04.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:04 smithi070 ceph-mon[112139]: pgmap v2477: 89 pgs: 89 active+clean; 1.9 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 958 op/s 2024-09-21T10:48:04.900 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:04.943 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:04.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:04.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:05.047 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:05.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:05.076 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:05.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:05.177 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:05.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:05.212 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:05.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:05.228 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T10:48:05.228 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:48:05.313 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:05.386 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:05.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:05.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:05.488 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:05.541 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:05.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:05.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:05.644 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:05.684 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:05.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:05.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:05.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:48:05.748 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:48:05.786 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:05.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:05.833 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:05.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:48:05.856 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:05.856 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:05.859 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:05.934 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:05.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:05.971 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:05.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:06.072 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:06.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:06.112 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:06.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:06.213 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:06.245 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:06.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:06.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:06.347 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:06.380 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:06.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:06.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:06.482 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:06.521 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:06.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:06.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:06.526 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T10:48:06.623 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:06.659 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:06.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:06.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:06.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:06 smithi070 ceph-mon[107385]: pgmap v2478: 89 pgs: 89 active+clean; 1.8 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 962 op/s 2024-09-21T10:48:06.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:06 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/656366823' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:48:06.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:06 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:48:06.725 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:06 smithi070 ceph-mon[112139]: pgmap v2478: 89 pgs: 89 active+clean; 1.8 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 962 op/s 2024-09-21T10:48:06.725 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:06 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/656366823' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:48:06.725 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:06 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:48:06.761 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:06.792 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:06.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:06.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:06.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:06 smithi160 ceph-mon[101879]: pgmap v2478: 89 pgs: 89 active+clean; 1.8 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 962 op/s 2024-09-21T10:48:06.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:06 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/656366823' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:48:06.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:06 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:48:06.894 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:06.932 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:06.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:06.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:07.034 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:07.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:07.073 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:07.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:07.174 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:07.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:07.215 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:07.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:07.316 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:07.342 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:07.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:07.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:07.444 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:07.476 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:07.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:07.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:07.578 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:07.614 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:07.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:07.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:07.691 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:07 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:48:07.691 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:07 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:48:07.691 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:07 smithi070 ceph-mon[112139]: osdmap e1403: 8 total, 8 up, 7 in 2024-09-21T10:48:07.691 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:07 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:48:07.691 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:07 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:48:07.691 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:07 smithi070 ceph-mon[107385]: osdmap e1403: 8 total, 8 up, 7 in 2024-09-21T10:48:07.716 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:07.754 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:07.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:07.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:07 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:48:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:07 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:48:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:07 smithi160 ceph-mon[101879]: osdmap e1403: 8 total, 8 up, 7 in 2024-09-21T10:48:07.856 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:07.894 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:07.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:07.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:07.996 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:08.028 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:08.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:08.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:08.129 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:08.163 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:08.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:08.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:08.265 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:08.298 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:08.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:08.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:08.400 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:08.437 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:08.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:08.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:08.526 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:08 smithi070 ceph-mon[107385]: pgmap v2480: 89 pgs: 89 active+clean; 1.6 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 1.20k op/s 2024-09-21T10:48:08.527 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:08 smithi070 ceph-mon[112139]: pgmap v2480: 89 pgs: 89 active+clean; 1.6 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 1.20k op/s 2024-09-21T10:48:08.538 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:08.574 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:08.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:08.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:08.675 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:08.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:08.721 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:08.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:08.822 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:08 smithi160 ceph-mon[101879]: pgmap v2480: 89 pgs: 89 active+clean; 1.6 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 1.20k op/s 2024-09-21T10:48:08.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:08.855 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:08.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:08.956 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:08.996 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:08.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:08.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:09.098 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:09.126 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:09.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:09.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:09.228 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:09.254 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:09.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:09.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:09.356 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:09.386 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:09.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:09.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:09.462 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:48:09.488 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:09.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:09.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:09.658 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:09.704 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:09.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:09.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:09.806 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:09.833 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:09.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:09.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:09.935 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:09.969 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:09.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:09.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:10.070 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:10.102 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:10.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:10.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:10.203 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:10.231 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:10.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:10.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:10.333 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:10.363 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:10.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:10.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:10.465 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:10.502 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:10.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:10.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:10.701 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:10.729 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:10.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:10.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:10.831 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:10.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:10 smithi160 ceph-mon[101879]: pgmap v2481: 89 pgs: 89 active+clean; 1.6 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 898 op/s 2024-09-21T10:48:10.852 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:10 smithi070 ceph-mon[107385]: pgmap v2481: 89 pgs: 89 active+clean; 1.6 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 898 op/s 2024-09-21T10:48:10.852 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:10 smithi070 ceph-mon[112139]: pgmap v2481: 89 pgs: 89 active+clean; 1.6 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 898 op/s 2024-09-21T10:48:10.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:10.867 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:10.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:10.968 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:10.995 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:10.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:10.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:11.032 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:48:11.033 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:48:11.066 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:48:11.066 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:48:11.092 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:48:11.092 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:48:11.097 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:11.120 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:48:11.120 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:48:11.140 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:11.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:11.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:11.156 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:48:11.156 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:48:11.184 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:48:11.184 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:48:11.216 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:48:11.216 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:48:11.242 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:11.258 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:48:11.259 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:48:11.283 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:11.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:11.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:11.384 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:11.411 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:11.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:11.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:11.512 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:11.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:11.557 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:11.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:11.658 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:11.678 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:11.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:11.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:11.780 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:11.812 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:11.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:11.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:11 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4059607465' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:48:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:11 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:48:11.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:11 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4059607465' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:48:11.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:11 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:48:11.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:11 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4059607465' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:48:11.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:11 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:48:11.914 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:11.940 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:11.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:11.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:12.042 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:12.066 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:12.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:12.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:12.168 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:12.211 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:12.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:12.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:12.312 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:12.345 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:12.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:12.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:12.447 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:12.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:12.500 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:12.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:12.601 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:12.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:12 smithi070 ceph-mon[107385]: pgmap v2482: 89 pgs: 89 active+clean; 1.4 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 804 op/s 2024-09-21T10:48:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:12 smithi070 ceph-mon[112139]: pgmap v2482: 89 pgs: 89 active+clean; 1.4 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 804 op/s 2024-09-21T10:48:12.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:12.642 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:12.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:12.696 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:48:12.696 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:48:12.743 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:12.771 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:48:12.772 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T10:48:12.785 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:12.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:12.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:12 smithi160 ceph-mon[101879]: pgmap v2482: 89 pgs: 89 active+clean; 1.4 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 804 op/s 2024-09-21T10:48:12.888 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:12.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:12.915 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:12.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:13.016 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:13.046 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:13.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:13.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:13.148 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:13.175 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:13.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:13.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:13.277 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:13.311 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:13.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:13.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:13.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:48:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:48:13] "GET /metrics HTTP/1.1" 200 36599 "" "Prometheus/2.51.0" 2024-09-21T10:48:13.413 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:13.452 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:13.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:13.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:13.554 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:13.584 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:13.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:13.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:13.685 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:13.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:13.719 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:13.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:13.820 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:13.823 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":2482,"stamp":"2024-09-21T10:48:10.809402+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1546393597,"num_objects":23614,"num_object_clones":0,"num_object_copies":70842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23614,"num_whiteouts":0,"num_read":379,"num_read_kb":823,"num_write":234913,"num_write_kb":8265490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2962150,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121242,"ondisk_log_size":121242,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":7730264,"kb_used_data":4737440,"kb_used_omap":219,"kb_used_meta":2992100,"kb_avail":648485800,"statfs":{"total":671965249536,"available":664049459200,"internally_reserved":0,"allocated":4851138560,"data_stored":4819739551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":225222,"internal_metadata":3063910458},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,8,7,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-491782144,"num_objects":-7504,"num_object_clones":0,"num_object_copies":-22512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7504,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.328444"},"pg_stats":[{"pgid":"54.f","version":"1403'14877","reported_seq":14945,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:10.040982+0000","last_change":"2024-09-21T10:44:32.818675+0000","last_active":"2024-09-21T10:48:10.040982+0000","last_peered":"2024-09-21T10:48:10.040982+0000","last_clean":"2024-09-21T10:48:10.040982+0000","last_became_active":"2024-09-21T10:44:32.818171+0000","last_became_peered":"2024-09-21T10:44:32.818171+0000","last_unstale":"2024-09-21T10:48:10.040982+0000","last_undegraded":"2024-09-21T10:48:10.040982+0000","last_fullsized":"2024-09-21T10:48:10.040982+0000","mapping_epoch":1376,"log_start":"1389'8000","ondisk_log_start":"1389'8000","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":6877,"log_dups_size":0,"ondisk_log_size":6877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:59:35.398933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90898432,"num_objects":1387,"num_object_clones":0,"num_object_copies":4161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14877,"num_write_kb":520448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.e","version":"1402'14407","reported_seq":14443,"reported_epoch":1402,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.009711+0000","last_change":"2024-09-21T10:44:32.821023+0000","last_active":"2024-09-21T10:48:06.009711+0000","last_peered":"2024-09-21T10:48:06.009711+0000","last_clean":"2024-09-21T10:48:06.009711+0000","last_became_active":"2024-09-21T10:44:32.820651+0000","last_became_peered":"2024-09-21T10:44:32.820651+0000","last_unstale":"2024-09-21T10:48:06.009711+0000","last_undegraded":"2024-09-21T10:48:06.009711+0000","last_fullsized":"2024-09-21T10:48:06.009711+0000","mapping_epoch":1376,"log_start":"1386'6500","ondisk_log_start":"1386'6500","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":7907,"log_dups_size":0,"ondisk_log_size":7907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:39:13.493921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101122048,"num_objects":1543,"num_object_clones":0,"num_object_copies":4629,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14407,"num_write_kb":510400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"54.d","version":"1402'14578","reported_seq":14614,"reported_epoch":1402,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.378800+0000","last_change":"2024-09-21T10:44:32.816573+0000","last_active":"2024-09-21T10:48:06.378800+0000","last_peered":"2024-09-21T10:48:06.378800+0000","last_clean":"2024-09-21T10:48:06.378800+0000","last_became_active":"2024-09-21T10:44:32.816260+0000","last_became_peered":"2024-09-21T10:44:32.816260+0000","last_unstale":"2024-09-21T10:48:06.378800+0000","last_undegraded":"2024-09-21T10:48:06.378800+0000","last_fullsized":"2024-09-21T10:48:06.378800+0000","mapping_epoch":1376,"log_start":"1387'7200","ondisk_log_start":"1387'7200","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":7378,"log_dups_size":0,"ondisk_log_size":7378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:20:47.764042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106889240,"num_objects":1632,"num_object_clones":0,"num_object_copies":4896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1632,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14578,"num_write_kb":518657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"54.c","version":"1403'14264","reported_seq":14332,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:10.055030+0000","last_change":"2024-09-21T10:44:32.819873+0000","last_active":"2024-09-21T10:48:10.055030+0000","last_peered":"2024-09-21T10:48:10.055030+0000","last_clean":"2024-09-21T10:48:10.055030+0000","last_became_active":"2024-09-21T10:44:32.819256+0000","last_became_peered":"2024-09-21T10:44:32.819256+0000","last_unstale":"2024-09-21T10:48:10.055030+0000","last_undegraded":"2024-09-21T10:48:10.055030+0000","last_fullsized":"2024-09-21T10:48:10.055030+0000","mapping_epoch":1376,"log_start":"1388'7400","ondisk_log_start":"1388'7400","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":6864,"log_dups_size":0,"ondisk_log_size":6864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:26:37.931891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90963968,"num_objects":1388,"num_object_clones":0,"num_object_copies":4164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14264,"num_write_kb":500864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.b","version":"1403'14572","reported_seq":14640,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:10.014716+0000","last_change":"2024-09-21T10:44:32.819321+0000","last_active":"2024-09-21T10:48:10.014716+0000","last_peered":"2024-09-21T10:48:10.014716+0000","last_clean":"2024-09-21T10:48:10.014716+0000","last_became_active":"2024-09-21T10:44:32.819084+0000","last_became_peered":"2024-09-21T10:44:32.819084+0000","last_unstale":"2024-09-21T10:48:10.014716+0000","last_undegraded":"2024-09-21T10:48:10.014716+0000","last_fullsized":"2024-09-21T10:48:10.014716+0000","mapping_epoch":1376,"log_start":"1389'7700","ondisk_log_start":"1389'7700","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":6872,"log_dups_size":0,"ondisk_log_size":6872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:58:43.441802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88604672,"num_objects":1352,"num_object_clones":0,"num_object_copies":4056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14572,"num_write_kb":509568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.a","version":"1402'14632","reported_seq":14667,"reported_epoch":1402,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.405989+0000","last_change":"2024-09-21T10:44:32.818041+0000","last_active":"2024-09-21T10:48:06.405989+0000","last_peered":"2024-09-21T10:48:06.405989+0000","last_clean":"2024-09-21T10:48:06.405989+0000","last_became_active":"2024-09-21T10:44:32.817725+0000","last_became_peered":"2024-09-21T10:44:32.817725+0000","last_unstale":"2024-09-21T10:48:06.405989+0000","last_undegraded":"2024-09-21T10:48:06.405989+0000","last_fullsized":"2024-09-21T10:48:06.405989+0000","mapping_epoch":1376,"log_start":"1387'7300","ondisk_log_start":"1387'7300","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":7332,"log_dups_size":0,"ondisk_log_size":7332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:22:24.084717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104595456,"num_objects":1596,"num_object_clones":0,"num_object_copies":4788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14632,"num_write_kb":519296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"54.9","version":"1403'14739","reported_seq":14807,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:09.752048+0000","last_change":"2024-09-21T10:44:32.816734+0000","last_active":"2024-09-21T10:48:09.752048+0000","last_peered":"2024-09-21T10:48:09.752048+0000","last_clean":"2024-09-21T10:48:09.752048+0000","last_became_active":"2024-09-21T10:44:32.816261+0000","last_became_peered":"2024-09-21T10:44:32.816261+0000","last_unstale":"2024-09-21T10:48:09.752048+0000","last_undegraded":"2024-09-21T10:48:09.752048+0000","last_fullsized":"2024-09-21T10:48:09.752048+0000","mapping_epoch":1376,"log_start":"1387'7200","ondisk_log_start":"1387'7200","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":7539,"log_dups_size":0,"ondisk_log_size":7539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:38.615950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89980928,"num_objects":1373,"num_object_clones":0,"num_object_copies":4119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14739,"num_write_kb":515584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"54.8","version":"1402'14957","reported_seq":14993,"reported_epoch":1402,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.016799+0000","last_change":"2024-09-21T10:44:32.818221+0000","last_active":"2024-09-21T10:48:06.016799+0000","last_peered":"2024-09-21T10:48:06.016799+0000","last_clean":"2024-09-21T10:48:06.016799+0000","last_became_active":"2024-09-21T10:44:32.817941+0000","last_became_peered":"2024-09-21T10:44:32.817941+0000","last_unstale":"2024-09-21T10:48:06.016799+0000","last_undegraded":"2024-09-21T10:48:06.016799+0000","last_fullsized":"2024-09-21T10:48:06.016799+0000","mapping_epoch":1376,"log_start":"1387'7000","ondisk_log_start":"1387'7000","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":7957,"log_dups_size":0,"ondisk_log_size":7957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:11:40.024092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106627072,"num_objects":1627,"num_object_clones":0,"num_object_copies":4881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14957,"num_write_kb":530688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"54.7","version":"1403'15017","reported_seq":15085,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:09.999196+0000","last_change":"2024-09-21T10:44:32.818432+0000","last_active":"2024-09-21T10:48:09.999196+0000","last_peered":"2024-09-21T10:48:09.999196+0000","last_clean":"2024-09-21T10:48:09.999196+0000","last_became_active":"2024-09-21T10:44:32.818094+0000","last_became_peered":"2024-09-21T10:44:32.818094+0000","last_unstale":"2024-09-21T10:48:09.999196+0000","last_undegraded":"2024-09-21T10:48:09.999196+0000","last_fullsized":"2024-09-21T10:48:09.999196+0000","mapping_epoch":1376,"log_start":"1389'8100","ondisk_log_start":"1389'8100","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":6917,"log_dups_size":0,"ondisk_log_size":6917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:55:13.023948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91947008,"num_objects":1403,"num_object_clones":0,"num_object_copies":4209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15017,"num_write_kb":525440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.6","version":"1402'14545","reported_seq":14581,"reported_epoch":1402,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.288548+0000","last_change":"2024-09-21T10:44:32.822589+0000","last_active":"2024-09-21T10:48:06.288548+0000","last_peered":"2024-09-21T10:48:06.288548+0000","last_clean":"2024-09-21T10:48:06.288548+0000","last_became_active":"2024-09-21T10:44:32.822265+0000","last_became_peered":"2024-09-21T10:44:32.822265+0000","last_unstale":"2024-09-21T10:48:06.288548+0000","last_undegraded":"2024-09-21T10:48:06.288548+0000","last_fullsized":"2024-09-21T10:48:06.288548+0000","mapping_epoch":1376,"log_start":"1386'6600","ondisk_log_start":"1386'6600","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":7945,"log_dups_size":0,"ondisk_log_size":7945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:40:44.356501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104136704,"num_objects":1589,"num_object_clones":0,"num_object_copies":4767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14545,"num_write_kb":516288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"54.5","version":"1402'14815","reported_seq":14851,"reported_epoch":1402,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.358468+0000","last_change":"2024-09-21T10:44:32.815686+0000","last_active":"2024-09-21T10:48:06.358468+0000","last_peered":"2024-09-21T10:48:06.358468+0000","last_clean":"2024-09-21T10:48:06.358468+0000","last_became_active":"2024-09-21T10:44:32.815266+0000","last_became_peered":"2024-09-21T10:44:32.815266+0000","last_unstale":"2024-09-21T10:48:06.358468+0000","last_undegraded":"2024-09-21T10:48:06.358468+0000","last_fullsized":"2024-09-21T10:48:06.358468+0000","mapping_epoch":1376,"log_start":"1387'7400","ondisk_log_start":"1387'7400","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":7415,"log_dups_size":0,"ondisk_log_size":7415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:11:59.857675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104660992,"num_objects":1597,"num_object_clones":0,"num_object_copies":4791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14815,"num_write_kb":525184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"54.4","version":"1403'14904","reported_seq":14972,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:10.029787+0000","last_change":"2024-09-21T10:44:32.821087+0000","last_active":"2024-09-21T10:48:10.029787+0000","last_peered":"2024-09-21T10:48:10.029787+0000","last_clean":"2024-09-21T10:48:10.029787+0000","last_became_active":"2024-09-21T10:44:32.820712+0000","last_became_peered":"2024-09-21T10:44:32.820712+0000","last_unstale":"2024-09-21T10:48:10.029787+0000","last_undegraded":"2024-09-21T10:48:10.029787+0000","last_fullsized":"2024-09-21T10:48:10.029787+0000","mapping_epoch":1376,"log_start":"1389'8000","ondisk_log_start":"1389'8000","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":6904,"log_dups_size":0,"ondisk_log_size":6904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:51:13.302857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82051072,"num_objects":1252,"num_object_clones":0,"num_object_copies":3756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14904,"num_write_kb":516992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.3","version":"1403'14748","reported_seq":14816,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:09.694877+0000","last_change":"2024-09-21T10:44:32.817547+0000","last_active":"2024-09-21T10:48:09.694877+0000","last_peered":"2024-09-21T10:48:09.694877+0000","last_clean":"2024-09-21T10:48:09.694877+0000","last_became_active":"2024-09-21T10:44:32.817238+0000","last_became_peered":"2024-09-21T10:44:32.817238+0000","last_unstale":"2024-09-21T10:48:09.694877+0000","last_undegraded":"2024-09-21T10:48:09.694877+0000","last_fullsized":"2024-09-21T10:48:09.694877+0000","mapping_epoch":1376,"log_start":"1387'7200","ondisk_log_start":"1387'7200","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":7548,"log_dups_size":0,"ondisk_log_size":7548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:02:33.135641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90177536,"num_objects":1376,"num_object_clones":0,"num_object_copies":4128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14748,"num_write_kb":515968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"54.2","version":"1403'14570","reported_seq":14638,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:10.005943+0000","last_change":"2024-09-21T10:44:32.820419+0000","last_active":"2024-09-21T10:48:10.005943+0000","last_peered":"2024-09-21T10:48:10.005943+0000","last_clean":"2024-09-21T10:48:10.005943+0000","last_became_active":"2024-09-21T10:44:32.820081+0000","last_became_peered":"2024-09-21T10:44:32.820081+0000","last_unstale":"2024-09-21T10:48:10.005943+0000","last_undegraded":"2024-09-21T10:48:10.005943+0000","last_fullsized":"2024-09-21T10:48:10.005943+0000","mapping_epoch":1376,"log_start":"1386'6400","ondisk_log_start":"1386'6400","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":8170,"log_dups_size":0,"ondisk_log_size":8170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:30:15.785931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88211456,"num_objects":1346,"num_object_clones":0,"num_object_copies":4038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14570,"num_write_kb":509312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"54.1","version":"1402'14619","reported_seq":14654,"reported_epoch":1402,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.463759+0000","last_change":"2024-09-21T10:44:32.819407+0000","last_active":"2024-09-21T10:48:06.463759+0000","last_peered":"2024-09-21T10:48:06.463759+0000","last_clean":"2024-09-21T10:48:06.463759+0000","last_became_active":"2024-09-21T10:44:32.819056+0000","last_became_peered":"2024-09-21T10:44:32.819056+0000","last_unstale":"2024-09-21T10:48:06.463759+0000","last_undegraded":"2024-09-21T10:48:06.463759+0000","last_fullsized":"2024-09-21T10:48:06.463759+0000","mapping_epoch":1376,"log_start":"1383'4600","ondisk_log_start":"1383'4600","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:39:58.641282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102039552,"num_objects":1557,"num_object_clones":0,"num_object_copies":4671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14619,"num_write_kb":517632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"54.0","version":"1402'14390","reported_seq":14426,"reported_epoch":1402,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.391369+0000","last_change":"2024-09-21T10:44:32.816008+0000","last_active":"2024-09-21T10:48:06.391369+0000","last_peered":"2024-09-21T10:48:06.391369+0000","last_clean":"2024-09-21T10:48:06.391369+0000","last_became_active":"2024-09-21T10:44:32.815627+0000","last_became_peered":"2024-09-21T10:44:32.815627+0000","last_unstale":"2024-09-21T10:48:06.391369+0000","last_undegraded":"2024-09-21T10:48:06.391369+0000","last_fullsized":"2024-09-21T10:48:06.391369+0000","mapping_epoch":1376,"log_start":"1387'7000","ondisk_log_start":"1387'7000","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":7390,"log_dups_size":0,"ondisk_log_size":7390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:48:54.596883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102891520,"num_objects":1570,"num_object_clones":0,"num_object_copies":4710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14390,"num_write_kb":510720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":2119,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.530878+0000","last_change":"2024-09-21T10:40:13.438023+0000","last_active":"2024-09-21T10:48:06.530878+0000","last_peered":"2024-09-21T10:48:06.530878+0000","last_clean":"2024-09-21T10:48:06.530878+0000","last_became_active":"2024-09-21T10:40:13.431103+0000","last_became_peered":"2024-09-21T10:40:13.431103+0000","last_unstale":"2024-09-21T10:48:06.530878+0000","last_undegraded":"2024-09-21T10:48:06.530878+0000","last_fullsized":"2024-09-21T10:48:06.530878+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:00:24.887661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1372,"reported_epoch":1402,"state":"active+clean","last_fresh":"2024-09-21T10:48:01.492209+0000","last_change":"2024-09-21T10:07:55.999392+0000","last_active":"2024-09-21T10:48:01.492209+0000","last_peered":"2024-09-21T10:48:01.492209+0000","last_clean":"2024-09-21T10:48:01.492209+0000","last_became_active":"2024-09-21T10:07:55.999103+0000","last_became_peered":"2024-09-21T10:07:55.999103+0000","last_unstale":"2024-09-21T10:48:01.492209+0000","last_undegraded":"2024-09-21T10:48:01.492209+0000","last_fullsized":"2024-09-21T10:48:01.492209+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:48.757242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":2100,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.531613+0000","last_change":"2024-09-21T10:40:13.432818+0000","last_active":"2024-09-21T10:48:06.531613+0000","last_peered":"2024-09-21T10:48:06.531613+0000","last_clean":"2024-09-21T10:48:06.531613+0000","last_became_active":"2024-09-21T10:40:13.432448+0000","last_became_peered":"2024-09-21T10:40:13.432448+0000","last_unstale":"2024-09-21T10:48:06.531613+0000","last_undegraded":"2024-09-21T10:48:06.531613+0000","last_fullsized":"2024-09-21T10:48:06.531613+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:20.448242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2117,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.531935+0000","last_change":"2024-09-21T10:40:13.431824+0000","last_active":"2024-09-21T10:48:06.531935+0000","last_peered":"2024-09-21T10:48:06.531935+0000","last_clean":"2024-09-21T10:48:06.531935+0000","last_became_active":"2024-09-21T10:40:13.431516+0000","last_became_peered":"2024-09-21T10:40:13.431516+0000","last_unstale":"2024-09-21T10:48:06.531935+0000","last_undegraded":"2024-09-21T10:48:06.531935+0000","last_fullsized":"2024-09-21T10:48:06.531935+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:30:40.226452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":1359,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.457395+0000","last_change":"2024-09-21T09:54:53.634508+0000","last_active":"2024-09-21T10:47:52.457395+0000","last_peered":"2024-09-21T10:47:52.457395+0000","last_clean":"2024-09-21T10:47:52.457395+0000","last_became_active":"2024-09-21T09:54:53.634254+0000","last_became_peered":"2024-09-21T09:54:53.634254+0000","last_unstale":"2024-09-21T10:47:52.457395+0000","last_undegraded":"2024-09-21T10:47:52.457395+0000","last_fullsized":"2024-09-21T10:47:52.457395+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:18.122107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1386,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.455462+0000","last_change":"2024-09-21T10:09:06.574933+0000","last_active":"2024-09-21T10:47:52.455462+0000","last_peered":"2024-09-21T10:47:52.455462+0000","last_clean":"2024-09-21T10:47:52.455462+0000","last_became_active":"2024-09-21T10:09:06.574467+0000","last_became_peered":"2024-09-21T10:09:06.574467+0000","last_unstale":"2024-09-21T10:47:52.455462+0000","last_undegraded":"2024-09-21T10:47:52.455462+0000","last_fullsized":"2024-09-21T10:47:52.455462+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:06.142269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":1374,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.461916+0000","last_change":"2024-09-21T10:07:55.991189+0000","last_active":"2024-09-21T10:47:52.461916+0000","last_peered":"2024-09-21T10:47:52.461916+0000","last_clean":"2024-09-21T10:47:52.461916+0000","last_became_active":"2024-09-21T10:07:55.990784+0000","last_became_peered":"2024-09-21T10:07:55.990784+0000","last_unstale":"2024-09-21T10:47:52.461916+0000","last_undegraded":"2024-09-21T10:47:52.461916+0000","last_fullsized":"2024-09-21T10:47:52.461916+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:00.580471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1386,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.455830+0000","last_change":"2024-09-21T10:09:06.560175+0000","last_active":"2024-09-21T10:47:52.455830+0000","last_peered":"2024-09-21T10:47:52.455830+0000","last_clean":"2024-09-21T10:47:52.455830+0000","last_became_active":"2024-09-21T10:09:06.558584+0000","last_became_peered":"2024-09-21T10:09:06.558584+0000","last_unstale":"2024-09-21T10:47:52.455830+0000","last_undegraded":"2024-09-21T10:47:52.455830+0000","last_fullsized":"2024-09-21T10:47:52.455830+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:02.918339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":2102,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.532219+0000","last_change":"2024-09-21T10:44:14.409321+0000","last_active":"2024-09-21T10:48:06.532219+0000","last_peered":"2024-09-21T10:48:06.532219+0000","last_clean":"2024-09-21T10:48:06.532219+0000","last_became_active":"2024-09-21T10:44:14.408833+0000","last_became_peered":"2024-09-21T10:44:14.408833+0000","last_unstale":"2024-09-21T10:48:06.532219+0000","last_undegraded":"2024-09-21T10:48:06.532219+0000","last_fullsized":"2024-09-21T10:48:06.532219+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:10:00.997210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":879,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.522530+0000","last_change":"2024-09-21T10:40:13.436927+0000","last_active":"2024-09-21T10:48:06.522530+0000","last_peered":"2024-09-21T10:48:06.522530+0000","last_clean":"2024-09-21T10:48:06.522530+0000","last_became_active":"2024-09-21T10:40:13.436329+0000","last_became_peered":"2024-09-21T10:40:13.436329+0000","last_unstale":"2024-09-21T10:48:06.522530+0000","last_undegraded":"2024-09-21T10:48:06.522530+0000","last_fullsized":"2024-09-21T10:48:06.522530+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:05.668898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":1397,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.455663+0000","last_change":"2024-09-21T10:40:13.433278+0000","last_active":"2024-09-21T10:47:52.455663+0000","last_peered":"2024-09-21T10:47:52.455663+0000","last_clean":"2024-09-21T10:47:52.455663+0000","last_became_active":"2024-09-21T10:40:13.432814+0000","last_became_peered":"2024-09-21T10:40:13.432814+0000","last_unstale":"2024-09-21T10:47:52.455663+0000","last_undegraded":"2024-09-21T10:47:52.455663+0000","last_fullsized":"2024-09-21T10:47:52.455663+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:55.909872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2110,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.531962+0000","last_change":"2024-09-21T10:40:13.432531+0000","last_active":"2024-09-21T10:48:06.531962+0000","last_peered":"2024-09-21T10:48:06.531962+0000","last_clean":"2024-09-21T10:48:06.531962+0000","last_became_active":"2024-09-21T10:40:13.432210+0000","last_became_peered":"2024-09-21T10:40:13.432210+0000","last_unstale":"2024-09-21T10:48:06.531962+0000","last_undegraded":"2024-09-21T10:48:06.531962+0000","last_fullsized":"2024-09-21T10:48:06.531962+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:20:10.998607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2082,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.523497+0000","last_change":"2024-09-21T10:09:06.530394+0000","last_active":"2024-09-21T10:48:06.523497+0000","last_peered":"2024-09-21T10:48:06.523497+0000","last_clean":"2024-09-21T10:48:06.523497+0000","last_became_active":"2024-09-21T10:09:06.529307+0000","last_became_peered":"2024-09-21T10:09:06.529307+0000","last_unstale":"2024-09-21T10:48:06.523497+0000","last_undegraded":"2024-09-21T10:48:06.523497+0000","last_fullsized":"2024-09-21T10:48:06.523497+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:54.287397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2115,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.531059+0000","last_change":"2024-09-21T10:44:14.409422+0000","last_active":"2024-09-21T10:48:06.531059+0000","last_peered":"2024-09-21T10:48:06.531059+0000","last_clean":"2024-09-21T10:48:06.531059+0000","last_became_active":"2024-09-21T10:44:14.408726+0000","last_became_peered":"2024-09-21T10:44:14.408726+0000","last_unstale":"2024-09-21T10:48:06.531059+0000","last_undegraded":"2024-09-21T10:48:06.531059+0000","last_fullsized":"2024-09-21T10:48:06.531059+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:44:21.743112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2097,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.533016+0000","last_change":"2024-09-21T10:09:06.552324+0000","last_active":"2024-09-21T10:48:06.533016+0000","last_peered":"2024-09-21T10:48:06.533016+0000","last_clean":"2024-09-21T10:48:06.533016+0000","last_became_active":"2024-09-21T10:09:06.540545+0000","last_became_peered":"2024-09-21T10:09:06.540545+0000","last_unstale":"2024-09-21T10:48:06.533016+0000","last_undegraded":"2024-09-21T10:48:06.533016+0000","last_fullsized":"2024-09-21T10:48:06.533016+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:08.506149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1376,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.462498+0000","last_change":"2024-09-21T10:07:56.003005+0000","last_active":"2024-09-21T10:47:52.462498+0000","last_peered":"2024-09-21T10:47:52.462498+0000","last_clean":"2024-09-21T10:47:52.462498+0000","last_became_active":"2024-09-21T10:07:56.001550+0000","last_became_peered":"2024-09-21T10:07:56.001550+0000","last_unstale":"2024-09-21T10:47:52.462498+0000","last_undegraded":"2024-09-21T10:47:52.462498+0000","last_fullsized":"2024-09-21T10:47:52.462498+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:54.553261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2089,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.533099+0000","last_change":"2024-09-21T10:09:06.559017+0000","last_active":"2024-09-21T10:48:06.533099+0000","last_peered":"2024-09-21T10:48:06.533099+0000","last_clean":"2024-09-21T10:48:06.533099+0000","last_became_active":"2024-09-21T10:09:06.555256+0000","last_became_peered":"2024-09-21T10:09:06.555256+0000","last_unstale":"2024-09-21T10:48:06.533099+0000","last_undegraded":"2024-09-21T10:48:06.533099+0000","last_fullsized":"2024-09-21T10:48:06.533099+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:33:45.145138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2094,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.531119+0000","last_change":"2024-09-21T10:09:06.562504+0000","last_active":"2024-09-21T10:48:06.531119+0000","last_peered":"2024-09-21T10:48:06.531119+0000","last_clean":"2024-09-21T10:48:06.531119+0000","last_became_active":"2024-09-21T10:09:06.561747+0000","last_became_peered":"2024-09-21T10:09:06.561747+0000","last_unstale":"2024-09-21T10:48:06.531119+0000","last_undegraded":"2024-09-21T10:48:06.531119+0000","last_fullsized":"2024-09-21T10:48:06.531119+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:03:58.012406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2094,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.532122+0000","last_change":"2024-09-21T10:44:14.399948+0000","last_active":"2024-09-21T10:48:06.532122+0000","last_peered":"2024-09-21T10:48:06.532122+0000","last_clean":"2024-09-21T10:48:06.532122+0000","last_became_active":"2024-09-21T10:44:14.399337+0000","last_became_peered":"2024-09-21T10:44:14.399337+0000","last_unstale":"2024-09-21T10:48:06.532122+0000","last_undegraded":"2024-09-21T10:48:06.532122+0000","last_fullsized":"2024-09-21T10:48:06.532122+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:40.185020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1391,"reported_epoch":1402,"state":"active+clean","last_fresh":"2024-09-21T10:48:01.498687+0000","last_change":"2024-09-21T10:44:14.401716+0000","last_active":"2024-09-21T10:48:01.498687+0000","last_peered":"2024-09-21T10:48:01.498687+0000","last_clean":"2024-09-21T10:48:01.498687+0000","last_became_active":"2024-09-21T10:44:14.401254+0000","last_became_peered":"2024-09-21T10:44:14.401254+0000","last_unstale":"2024-09-21T10:48:01.498687+0000","last_undegraded":"2024-09-21T10:48:01.498687+0000","last_fullsized":"2024-09-21T10:48:01.498687+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:07:21.536272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":50,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.453745+0000","last_change":"2024-09-21T10:44:14.424249+0000","last_active":"2024-09-21T10:47:52.453745+0000","last_peered":"2024-09-21T10:47:52.453745+0000","last_clean":"2024-09-21T10:47:52.453745+0000","last_became_active":"2024-09-21T10:44:14.408884+0000","last_became_peered":"2024-09-21T10:44:14.408884+0000","last_unstale":"2024-09-21T10:47:52.453745+0000","last_undegraded":"2024-09-21T10:47:52.453745+0000","last_fullsized":"2024-09-21T10:47:52.453745+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:18:44.759435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2089,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.521897+0000","last_change":"2024-09-21T10:40:13.437741+0000","last_active":"2024-09-21T10:48:06.521897+0000","last_peered":"2024-09-21T10:48:06.521897+0000","last_clean":"2024-09-21T10:48:06.521897+0000","last_became_active":"2024-09-21T10:40:13.437447+0000","last_became_peered":"2024-09-21T10:40:13.437447+0000","last_unstale":"2024-09-21T10:48:06.521897+0000","last_undegraded":"2024-09-21T10:48:06.521897+0000","last_fullsized":"2024-09-21T10:48:06.521897+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:40:47.072907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":1419,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.458247+0000","last_change":"2024-09-21T10:44:14.415264+0000","last_active":"2024-09-21T10:47:52.458247+0000","last_peered":"2024-09-21T10:47:52.458247+0000","last_clean":"2024-09-21T10:47:52.458247+0000","last_became_active":"2024-09-21T10:44:14.407813+0000","last_became_peered":"2024-09-21T10:44:14.407813+0000","last_unstale":"2024-09-21T10:47:52.458247+0000","last_undegraded":"2024-09-21T10:47:52.458247+0000","last_fullsized":"2024-09-21T10:47:52.458247+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:55.933601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1372,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.455863+0000","last_change":"2024-09-21T10:07:56.001756+0000","last_active":"2024-09-21T10:47:52.455863+0000","last_peered":"2024-09-21T10:47:52.455863+0000","last_clean":"2024-09-21T10:47:52.455863+0000","last_became_active":"2024-09-21T10:07:56.000789+0000","last_became_peered":"2024-09-21T10:07:56.000789+0000","last_unstale":"2024-09-21T10:47:52.455863+0000","last_undegraded":"2024-09-21T10:47:52.455863+0000","last_fullsized":"2024-09-21T10:47:52.455863+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:56:04.302902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1360,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.455142+0000","last_change":"2024-09-21T09:54:53.636902+0000","last_active":"2024-09-21T10:47:52.455142+0000","last_peered":"2024-09-21T10:47:52.455142+0000","last_clean":"2024-09-21T10:47:52.455142+0000","last_became_active":"2024-09-21T09:54:53.636616+0000","last_became_peered":"2024-09-21T09:54:53.636616+0000","last_unstale":"2024-09-21T10:47:52.455142+0000","last_undegraded":"2024-09-21T10:47:52.455142+0000","last_fullsized":"2024-09-21T10:47:52.455142+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:46:47.694495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1416,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:56.368688+0000","last_change":"2024-09-21T10:44:14.402059+0000","last_active":"2024-09-21T10:47:56.368688+0000","last_peered":"2024-09-21T10:47:56.368688+0000","last_clean":"2024-09-21T10:47:56.368688+0000","last_became_active":"2024-09-21T10:44:14.401710+0000","last_became_peered":"2024-09-21T10:44:14.401710+0000","last_unstale":"2024-09-21T10:47:56.368688+0000","last_undegraded":"2024-09-21T10:47:56.368688+0000","last_fullsized":"2024-09-21T10:47:56.368688+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:17.314693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":2102,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.530696+0000","last_change":"2024-09-21T10:07:56.023462+0000","last_active":"2024-09-21T10:48:06.530696+0000","last_peered":"2024-09-21T10:48:06.530696+0000","last_clean":"2024-09-21T10:48:06.530696+0000","last_became_active":"2024-09-21T10:07:56.021859+0000","last_became_peered":"2024-09-21T10:07:56.021859+0000","last_unstale":"2024-09-21T10:48:06.530696+0000","last_undegraded":"2024-09-21T10:48:06.530696+0000","last_fullsized":"2024-09-21T10:48:06.530696+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:42:03.134729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2091,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.533020+0000","last_change":"2024-09-21T10:44:14.399028+0000","last_active":"2024-09-21T10:48:06.533020+0000","last_peered":"2024-09-21T10:48:06.533020+0000","last_clean":"2024-09-21T10:48:06.533020+0000","last_became_active":"2024-09-21T10:44:14.398115+0000","last_became_peered":"2024-09-21T10:44:14.398115+0000","last_unstale":"2024-09-21T10:48:06.533020+0000","last_undegraded":"2024-09-21T10:48:06.533020+0000","last_fullsized":"2024-09-21T10:48:06.533020+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:05.170376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2083,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.523546+0000","last_change":"2024-09-21T10:07:56.006236+0000","last_active":"2024-09-21T10:48:06.523546+0000","last_peered":"2024-09-21T10:48:06.523546+0000","last_clean":"2024-09-21T10:48:06.523546+0000","last_became_active":"2024-09-21T10:07:56.004439+0000","last_became_peered":"2024-09-21T10:07:56.004439+0000","last_unstale":"2024-09-21T10:48:06.523546+0000","last_undegraded":"2024-09-21T10:48:06.523546+0000","last_fullsized":"2024-09-21T10:48:06.523546+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:39.422467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":2100,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.533535+0000","last_change":"2024-09-21T10:44:14.410440+0000","last_active":"2024-09-21T10:48:06.533535+0000","last_peered":"2024-09-21T10:48:06.533535+0000","last_clean":"2024-09-21T10:48:06.533535+0000","last_became_active":"2024-09-21T10:44:14.400035+0000","last_became_peered":"2024-09-21T10:44:14.400035+0000","last_unstale":"2024-09-21T10:48:06.533535+0000","last_undegraded":"2024-09-21T10:48:06.533535+0000","last_fullsized":"2024-09-21T10:48:06.533535+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:49:39.536516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2103,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.532323+0000","last_change":"2024-09-21T10:44:14.409410+0000","last_active":"2024-09-21T10:48:06.532323+0000","last_peered":"2024-09-21T10:48:06.532323+0000","last_clean":"2024-09-21T10:48:06.532323+0000","last_became_active":"2024-09-21T10:44:14.409025+0000","last_became_peered":"2024-09-21T10:44:14.409025+0000","last_unstale":"2024-09-21T10:48:06.532323+0000","last_undegraded":"2024-09-21T10:48:06.532323+0000","last_fullsized":"2024-09-21T10:48:06.532323+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:25:09.197455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":44,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.453658+0000","last_change":"2024-09-21T10:44:14.405677+0000","last_active":"2024-09-21T10:47:52.453658+0000","last_peered":"2024-09-21T10:47:52.453658+0000","last_clean":"2024-09-21T10:47:52.453658+0000","last_became_active":"2024-09-21T10:44:14.405262+0000","last_became_peered":"2024-09-21T10:44:14.405262+0000","last_unstale":"2024-09-21T10:47:52.453658+0000","last_undegraded":"2024-09-21T10:47:52.453658+0000","last_fullsized":"2024-09-21T10:47:52.453658+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:53:12.127245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2119,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.522431+0000","last_change":"2024-09-21T10:40:13.437609+0000","last_active":"2024-09-21T10:48:06.522431+0000","last_peered":"2024-09-21T10:48:06.522431+0000","last_clean":"2024-09-21T10:48:06.522431+0000","last_became_active":"2024-09-21T10:40:13.437296+0000","last_became_peered":"2024-09-21T10:40:13.437296+0000","last_unstale":"2024-09-21T10:48:06.522431+0000","last_undegraded":"2024-09-21T10:48:06.522431+0000","last_fullsized":"2024-09-21T10:48:06.522431+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:41:58.963386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2074,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.521779+0000","last_change":"2024-09-21T10:07:56.067772+0000","last_active":"2024-09-21T10:48:06.521779+0000","last_peered":"2024-09-21T10:48:06.521779+0000","last_clean":"2024-09-21T10:48:06.521779+0000","last_became_active":"2024-09-21T10:07:56.052602+0000","last_became_peered":"2024-09-21T10:07:56.052602+0000","last_unstale":"2024-09-21T10:48:06.521779+0000","last_undegraded":"2024-09-21T10:48:06.521779+0000","last_fullsized":"2024-09-21T10:48:06.521779+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:07:59.899214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2142,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.522236+0000","last_change":"2024-09-21T10:44:14.405182+0000","last_active":"2024-09-21T10:48:06.522236+0000","last_peered":"2024-09-21T10:48:06.522236+0000","last_clean":"2024-09-21T10:48:06.522236+0000","last_became_active":"2024-09-21T10:44:14.404852+0000","last_became_peered":"2024-09-21T10:44:14.404852+0000","last_unstale":"2024-09-21T10:48:06.522236+0000","last_undegraded":"2024-09-21T10:48:06.522236+0000","last_fullsized":"2024-09-21T10:48:06.522236+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:51.687206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1400,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.457540+0000","last_change":"2024-09-21T10:40:13.435375+0000","last_active":"2024-09-21T10:47:52.457540+0000","last_peered":"2024-09-21T10:47:52.457540+0000","last_clean":"2024-09-21T10:47:52.457540+0000","last_became_active":"2024-09-21T10:40:13.435050+0000","last_became_peered":"2024-09-21T10:40:13.435050+0000","last_unstale":"2024-09-21T10:47:52.457540+0000","last_undegraded":"2024-09-21T10:47:52.457540+0000","last_fullsized":"2024-09-21T10:47:52.457540+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:57.889610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2129,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.522603+0000","last_change":"2024-09-21T10:40:13.434189+0000","last_active":"2024-09-21T10:48:06.522603+0000","last_peered":"2024-09-21T10:48:06.522603+0000","last_clean":"2024-09-21T10:48:06.522603+0000","last_became_active":"2024-09-21T10:40:13.433892+0000","last_became_peered":"2024-09-21T10:40:13.433892+0000","last_unstale":"2024-09-21T10:48:06.522603+0000","last_undegraded":"2024-09-21T10:48:06.522603+0000","last_fullsized":"2024-09-21T10:48:06.522603+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:49:17.296220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1376,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.455903+0000","last_change":"2024-09-21T10:07:21.524082+0000","last_active":"2024-09-21T10:47:52.455903+0000","last_peered":"2024-09-21T10:47:52.455903+0000","last_clean":"2024-09-21T10:47:52.455903+0000","last_became_active":"2024-09-21T10:07:21.523779+0000","last_became_peered":"2024-09-21T10:07:21.523779+0000","last_unstale":"2024-09-21T10:47:52.455903+0000","last_undegraded":"2024-09-21T10:47:52.455903+0000","last_fullsized":"2024-09-21T10:47:52.455903+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:13.813832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":1391,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:56.462530+0000","last_change":"2024-09-21T10:07:56.024239+0000","last_active":"2024-09-21T10:47:56.462530+0000","last_peered":"2024-09-21T10:47:56.462530+0000","last_clean":"2024-09-21T10:47:56.462530+0000","last_became_active":"2024-09-21T10:07:56.023914+0000","last_became_peered":"2024-09-21T10:07:56.023914+0000","last_unstale":"2024-09-21T10:47:56.462530+0000","last_undegraded":"2024-09-21T10:47:56.462530+0000","last_fullsized":"2024-09-21T10:47:56.462530+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:23.361736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084477399999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":95,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.456051+0000","last_change":"2024-09-21T10:40:55.616051+0000","last_active":"2024-09-21T10:47:52.456051+0000","last_peered":"2024-09-21T10:47:52.456051+0000","last_clean":"2024-09-21T10:47:52.456051+0000","last_became_active":"2024-09-21T10:40:55.615741+0000","last_became_peered":"2024-09-21T10:40:55.615741+0000","last_unstale":"2024-09-21T10:47:52.456051+0000","last_undegraded":"2024-09-21T10:47:52.456051+0000","last_fullsized":"2024-09-21T10:47:52.456051+0000","mapping_epoch":1320,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:07.673527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2093,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.521445+0000","last_change":"2024-09-21T10:40:13.436234+0000","last_active":"2024-09-21T10:48:06.521445+0000","last_peered":"2024-09-21T10:48:06.521445+0000","last_clean":"2024-09-21T10:48:06.521445+0000","last_became_active":"2024-09-21T10:40:13.435946+0000","last_became_peered":"2024-09-21T10:40:13.435946+0000","last_unstale":"2024-09-21T10:48:06.521445+0000","last_undegraded":"2024-09-21T10:48:06.521445+0000","last_fullsized":"2024-09-21T10:48:06.521445+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:08:08.670739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":885,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.531333+0000","last_change":"2024-09-21T10:40:13.431693+0000","last_active":"2024-09-21T10:48:06.531333+0000","last_peered":"2024-09-21T10:48:06.531333+0000","last_clean":"2024-09-21T10:48:06.531333+0000","last_became_active":"2024-09-21T10:40:13.431389+0000","last_became_peered":"2024-09-21T10:40:13.431389+0000","last_unstale":"2024-09-21T10:48:06.531333+0000","last_undegraded":"2024-09-21T10:48:06.531333+0000","last_fullsized":"2024-09-21T10:48:06.531333+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:22.227336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2131,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.530958+0000","last_change":"2024-09-21T10:44:14.409601+0000","last_active":"2024-09-21T10:48:06.530958+0000","last_peered":"2024-09-21T10:48:06.530958+0000","last_clean":"2024-09-21T10:48:06.530958+0000","last_became_active":"2024-09-21T10:44:14.409262+0000","last_became_peered":"2024-09-21T10:44:14.409262+0000","last_unstale":"2024-09-21T10:48:06.530958+0000","last_undegraded":"2024-09-21T10:48:06.530958+0000","last_fullsized":"2024-09-21T10:48:06.530958+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:53:01.809847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2098,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.530692+0000","last_change":"2024-09-21T10:09:06.560130+0000","last_active":"2024-09-21T10:48:06.530692+0000","last_peered":"2024-09-21T10:48:06.530692+0000","last_clean":"2024-09-21T10:48:06.530692+0000","last_became_active":"2024-09-21T10:09:06.559827+0000","last_became_peered":"2024-09-21T10:09:06.559827+0000","last_unstale":"2024-09-21T10:48:06.530692+0000","last_undegraded":"2024-09-21T10:48:06.530692+0000","last_fullsized":"2024-09-21T10:48:06.530692+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:05:17.166734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":44,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.457012+0000","last_change":"2024-09-21T10:44:14.400691+0000","last_active":"2024-09-21T10:47:52.457012+0000","last_peered":"2024-09-21T10:47:52.457012+0000","last_clean":"2024-09-21T10:47:52.457012+0000","last_became_active":"2024-09-21T10:44:14.400377+0000","last_became_peered":"2024-09-21T10:44:14.400377+0000","last_unstale":"2024-09-21T10:47:52.457012+0000","last_undegraded":"2024-09-21T10:47:52.457012+0000","last_fullsized":"2024-09-21T10:47:52.457012+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:03.555078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1379,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.462544+0000","last_change":"2024-09-21T10:44:14.402965+0000","last_active":"2024-09-21T10:47:52.462544+0000","last_peered":"2024-09-21T10:47:52.462544+0000","last_clean":"2024-09-21T10:47:52.462544+0000","last_became_active":"2024-09-21T10:44:14.402634+0000","last_became_peered":"2024-09-21T10:44:14.402634+0000","last_unstale":"2024-09-21T10:47:52.462544+0000","last_undegraded":"2024-09-21T10:47:52.462544+0000","last_fullsized":"2024-09-21T10:47:52.462544+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:40:16.323155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2079,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.531607+0000","last_change":"2024-09-21T10:09:06.560589+0000","last_active":"2024-09-21T10:48:06.531607+0000","last_peered":"2024-09-21T10:48:06.531607+0000","last_clean":"2024-09-21T10:48:06.531607+0000","last_became_active":"2024-09-21T10:09:06.557552+0000","last_became_peered":"2024-09-21T10:09:06.557552+0000","last_unstale":"2024-09-21T10:48:06.531607+0000","last_undegraded":"2024-09-21T10:48:06.531607+0000","last_fullsized":"2024-09-21T10:48:06.531607+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:09:24.866759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":1435,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:56.462549+0000","last_change":"2024-09-21T10:44:14.415910+0000","last_active":"2024-09-21T10:47:56.462549+0000","last_peered":"2024-09-21T10:47:56.462549+0000","last_clean":"2024-09-21T10:47:56.462549+0000","last_became_active":"2024-09-21T10:44:14.402926+0000","last_became_peered":"2024-09-21T10:44:14.402926+0000","last_unstale":"2024-09-21T10:47:56.462549+0000","last_undegraded":"2024-09-21T10:47:56.462549+0000","last_fullsized":"2024-09-21T10:47:56.462549+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:55:07.144611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":794,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.457089+0000","last_change":"2024-09-21T10:44:17.413348+0000","last_active":"2024-09-21T10:47:52.457089+0000","last_peered":"2024-09-21T10:47:52.457089+0000","last_clean":"2024-09-21T10:47:52.457089+0000","last_became_active":"2024-09-21T10:44:17.412916+0000","last_became_peered":"2024-09-21T10:44:17.412916+0000","last_unstale":"2024-09-21T10:47:52.457089+0000","last_undegraded":"2024-09-21T10:47:52.457089+0000","last_fullsized":"2024-09-21T10:47:52.457089+0000","mapping_epoch":1372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1373,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:18.592283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2083,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.522808+0000","last_change":"2024-09-21T10:09:06.538940+0000","last_active":"2024-09-21T10:48:06.522808+0000","last_peered":"2024-09-21T10:48:06.522808+0000","last_clean":"2024-09-21T10:48:06.522808+0000","last_became_active":"2024-09-21T10:09:06.538668+0000","last_became_peered":"2024-09-21T10:09:06.538668+0000","last_unstale":"2024-09-21T10:48:06.522808+0000","last_undegraded":"2024-09-21T10:48:06.522808+0000","last_fullsized":"2024-09-21T10:48:06.522808+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:42:44.299488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2099,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.532968+0000","last_change":"2024-09-21T10:44:14.398581+0000","last_active":"2024-09-21T10:48:06.532968+0000","last_peered":"2024-09-21T10:48:06.532968+0000","last_clean":"2024-09-21T10:48:06.532968+0000","last_became_active":"2024-09-21T10:44:14.398258+0000","last_became_peered":"2024-09-21T10:44:14.398258+0000","last_unstale":"2024-09-21T10:48:06.532968+0000","last_undegraded":"2024-09-21T10:48:06.532968+0000","last_fullsized":"2024-09-21T10:48:06.532968+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:23.960956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1373,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.456389+0000","last_change":"2024-09-21T10:07:55.998799+0000","last_active":"2024-09-21T10:47:52.456389+0000","last_peered":"2024-09-21T10:47:52.456389+0000","last_clean":"2024-09-21T10:47:52.456389+0000","last_became_active":"2024-09-21T10:07:55.998510+0000","last_became_peered":"2024-09-21T10:07:55.998510+0000","last_unstale":"2024-09-21T10:47:52.456389+0000","last_undegraded":"2024-09-21T10:47:52.456389+0000","last_fullsized":"2024-09-21T10:47:52.456389+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:10.474008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1402,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.455362+0000","last_change":"2024-09-21T10:44:14.399842+0000","last_active":"2024-09-21T10:47:52.455362+0000","last_peered":"2024-09-21T10:47:52.455362+0000","last_clean":"2024-09-21T10:47:52.455362+0000","last_became_active":"2024-09-21T10:44:14.399426+0000","last_became_peered":"2024-09-21T10:44:14.399426+0000","last_unstale":"2024-09-21T10:47:52.455362+0000","last_undegraded":"2024-09-21T10:47:52.455362+0000","last_fullsized":"2024-09-21T10:47:52.455362+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:26:45.362101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1397,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.456046+0000","last_change":"2024-09-21T10:40:13.432387+0000","last_active":"2024-09-21T10:47:52.456046+0000","last_peered":"2024-09-21T10:47:52.456046+0000","last_clean":"2024-09-21T10:47:52.456046+0000","last_became_active":"2024-09-21T10:40:13.432053+0000","last_became_peered":"2024-09-21T10:40:13.432053+0000","last_unstale":"2024-09-21T10:47:52.456046+0000","last_undegraded":"2024-09-21T10:47:52.456046+0000","last_fullsized":"2024-09-21T10:47:52.456046+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:37.486101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1376,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.462485+0000","last_change":"2024-09-21T10:07:56.020791+0000","last_active":"2024-09-21T10:47:52.462485+0000","last_peered":"2024-09-21T10:47:52.462485+0000","last_clean":"2024-09-21T10:47:52.462485+0000","last_became_active":"2024-09-21T10:07:56.020205+0000","last_became_peered":"2024-09-21T10:07:56.020205+0000","last_unstale":"2024-09-21T10:47:52.462485+0000","last_undegraded":"2024-09-21T10:47:52.462485+0000","last_fullsized":"2024-09-21T10:47:52.462485+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:09.813166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1373,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.458121+0000","last_change":"2024-09-21T10:09:06.533271+0000","last_active":"2024-09-21T10:47:52.458121+0000","last_peered":"2024-09-21T10:47:52.458121+0000","last_clean":"2024-09-21T10:47:52.458121+0000","last_became_active":"2024-09-21T10:09:06.532041+0000","last_became_peered":"2024-09-21T10:09:06.532041+0000","last_unstale":"2024-09-21T10:47:52.458121+0000","last_undegraded":"2024-09-21T10:47:52.458121+0000","last_fullsized":"2024-09-21T10:47:52.458121+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:04.236375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":2100,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.532049+0000","last_change":"2024-09-21T10:09:06.543422+0000","last_active":"2024-09-21T10:48:06.532049+0000","last_peered":"2024-09-21T10:48:06.532049+0000","last_clean":"2024-09-21T10:48:06.532049+0000","last_became_active":"2024-09-21T10:09:06.539647+0000","last_became_peered":"2024-09-21T10:09:06.539647+0000","last_unstale":"2024-09-21T10:48:06.532049+0000","last_undegraded":"2024-09-21T10:48:06.532049+0000","last_fullsized":"2024-09-21T10:48:06.532049+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:04.990085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2099,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.532138+0000","last_change":"2024-09-21T10:07:56.018560+0000","last_active":"2024-09-21T10:48:06.532138+0000","last_peered":"2024-09-21T10:48:06.532138+0000","last_clean":"2024-09-21T10:48:06.532138+0000","last_became_active":"2024-09-21T10:07:56.018261+0000","last_became_peered":"2024-09-21T10:07:56.018261+0000","last_unstale":"2024-09-21T10:48:06.532138+0000","last_undegraded":"2024-09-21T10:48:06.532138+0000","last_fullsized":"2024-09-21T10:48:06.532138+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:04:54.395456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1374,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.462027+0000","last_change":"2024-09-21T10:09:06.531935+0000","last_active":"2024-09-21T10:47:52.462027+0000","last_peered":"2024-09-21T10:47:52.462027+0000","last_clean":"2024-09-21T10:47:52.462027+0000","last_became_active":"2024-09-21T10:09:06.531610+0000","last_became_peered":"2024-09-21T10:09:06.531610+0000","last_unstale":"2024-09-21T10:47:52.462027+0000","last_undegraded":"2024-09-21T10:47:52.462027+0000","last_fullsized":"2024-09-21T10:47:52.462027+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:33.874405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1411,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.462273+0000","last_change":"2024-09-21T10:44:14.401913+0000","last_active":"2024-09-21T10:47:52.462273+0000","last_peered":"2024-09-21T10:47:52.462273+0000","last_clean":"2024-09-21T10:47:52.462273+0000","last_became_active":"2024-09-21T10:44:14.401496+0000","last_became_peered":"2024-09-21T10:44:14.401496+0000","last_unstale":"2024-09-21T10:47:52.462273+0000","last_undegraded":"2024-09-21T10:47:52.462273+0000","last_fullsized":"2024-09-21T10:47:52.462273+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:07:38.531723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":48,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.453584+0000","last_change":"2024-09-21T10:44:14.410444+0000","last_active":"2024-09-21T10:47:52.453584+0000","last_peered":"2024-09-21T10:47:52.453584+0000","last_clean":"2024-09-21T10:47:52.453584+0000","last_became_active":"2024-09-21T10:44:14.395465+0000","last_became_peered":"2024-09-21T10:44:14.395465+0000","last_unstale":"2024-09-21T10:47:52.453584+0000","last_undegraded":"2024-09-21T10:47:52.453584+0000","last_fullsized":"2024-09-21T10:47:52.453584+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:05:44.575878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1389,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.455216+0000","last_change":"2024-09-21T10:09:06.559132+0000","last_active":"2024-09-21T10:47:52.455216+0000","last_peered":"2024-09-21T10:47:52.455216+0000","last_clean":"2024-09-21T10:47:52.455216+0000","last_became_active":"2024-09-21T10:09:06.558395+0000","last_became_peered":"2024-09-21T10:09:06.558395+0000","last_unstale":"2024-09-21T10:47:52.455216+0000","last_undegraded":"2024-09-21T10:47:52.455216+0000","last_fullsized":"2024-09-21T10:47:52.455216+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:27:00.073554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":48,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.454384+0000","last_change":"2024-09-21T10:44:14.419937+0000","last_active":"2024-09-21T10:47:52.454384+0000","last_peered":"2024-09-21T10:47:52.454384+0000","last_clean":"2024-09-21T10:47:52.454384+0000","last_became_active":"2024-09-21T10:44:14.402761+0000","last_became_peered":"2024-09-21T10:44:14.402761+0000","last_unstale":"2024-09-21T10:47:52.454384+0000","last_undegraded":"2024-09-21T10:47:52.454384+0000","last_fullsized":"2024-09-21T10:47:52.454384+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:42.875310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1377,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.457231+0000","last_change":"2024-09-21T10:44:14.402411+0000","last_active":"2024-09-21T10:47:52.457231+0000","last_peered":"2024-09-21T10:47:52.457231+0000","last_clean":"2024-09-21T10:47:52.457231+0000","last_became_active":"2024-09-21T10:44:14.402045+0000","last_became_peered":"2024-09-21T10:44:14.402045+0000","last_unstale":"2024-09-21T10:47:52.457231+0000","last_undegraded":"2024-09-21T10:47:52.457231+0000","last_fullsized":"2024-09-21T10:47:52.457231+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:57:10.535776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":48,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.454086+0000","last_change":"2024-09-21T10:44:14.421612+0000","last_active":"2024-09-21T10:47:52.454086+0000","last_peered":"2024-09-21T10:47:52.454086+0000","last_clean":"2024-09-21T10:47:52.454086+0000","last_became_active":"2024-09-21T10:44:14.406095+0000","last_became_peered":"2024-09-21T10:44:14.406095+0000","last_unstale":"2024-09-21T10:47:52.454086+0000","last_undegraded":"2024-09-21T10:47:52.454086+0000","last_fullsized":"2024-09-21T10:47:52.454086+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:21:19.918156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1382,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.462035+0000","last_change":"2024-09-21T10:07:21.497877+0000","last_active":"2024-09-21T10:47:52.462035+0000","last_peered":"2024-09-21T10:47:52.462035+0000","last_clean":"2024-09-21T10:47:52.462035+0000","last_became_active":"2024-09-21T10:07:21.496317+0000","last_became_peered":"2024-09-21T10:07:21.496317+0000","last_unstale":"2024-09-21T10:47:52.462035+0000","last_undegraded":"2024-09-21T10:47:52.462035+0000","last_fullsized":"2024-09-21T10:47:52.462035+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:39:03.848183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2095,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.533765+0000","last_change":"2024-09-21T10:44:14.398487+0000","last_active":"2024-09-21T10:48:06.533765+0000","last_peered":"2024-09-21T10:48:06.533765+0000","last_clean":"2024-09-21T10:48:06.533765+0000","last_became_active":"2024-09-21T10:44:14.398142+0000","last_became_peered":"2024-09-21T10:44:14.398142+0000","last_unstale":"2024-09-21T10:48:06.533765+0000","last_undegraded":"2024-09-21T10:48:06.533765+0000","last_fullsized":"2024-09-21T10:48:06.533765+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:20:41.065055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":44,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.452942+0000","last_change":"2024-09-21T10:44:14.398878+0000","last_active":"2024-09-21T10:47:52.452942+0000","last_peered":"2024-09-21T10:47:52.452942+0000","last_clean":"2024-09-21T10:47:52.452942+0000","last_became_active":"2024-09-21T10:44:14.398547+0000","last_became_peered":"2024-09-21T10:44:14.398547+0000","last_unstale":"2024-09-21T10:47:52.452942+0000","last_undegraded":"2024-09-21T10:47:52.452942+0000","last_fullsized":"2024-09-21T10:47:52.452942+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:13.008134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2125,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.530468+0000","last_change":"2024-09-21T10:44:14.409051+0000","last_active":"2024-09-21T10:48:06.530468+0000","last_peered":"2024-09-21T10:48:06.530468+0000","last_clean":"2024-09-21T10:48:06.530468+0000","last_became_active":"2024-09-21T10:44:14.407892+0000","last_became_peered":"2024-09-21T10:44:14.407892+0000","last_unstale":"2024-09-21T10:48:06.530468+0000","last_undegraded":"2024-09-21T10:48:06.530468+0000","last_fullsized":"2024-09-21T10:48:06.530468+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:19.549648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":2109,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.535460+0000","last_change":"2024-09-21T10:44:14.408837+0000","last_active":"2024-09-21T10:48:06.535460+0000","last_peered":"2024-09-21T10:48:06.535460+0000","last_clean":"2024-09-21T10:48:06.535460+0000","last_became_active":"2024-09-21T10:44:14.408399+0000","last_became_peered":"2024-09-21T10:44:14.408399+0000","last_unstale":"2024-09-21T10:48:06.535460+0000","last_undegraded":"2024-09-21T10:48:06.535460+0000","last_fullsized":"2024-09-21T10:48:06.535460+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:41:38.729386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1390,"reported_epoch":1402,"state":"active+clean","last_fresh":"2024-09-21T10:48:01.498627+0000","last_change":"2024-09-21T10:44:14.401435+0000","last_active":"2024-09-21T10:48:01.498627+0000","last_peered":"2024-09-21T10:48:01.498627+0000","last_clean":"2024-09-21T10:48:01.498627+0000","last_became_active":"2024-09-21T10:44:14.400908+0000","last_became_peered":"2024-09-21T10:44:14.400908+0000","last_unstale":"2024-09-21T10:48:01.498627+0000","last_undegraded":"2024-09-21T10:48:01.498627+0000","last_fullsized":"2024-09-21T10:48:01.498627+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:46.748289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":50,"reported_epoch":1401,"state":"active+clean","last_fresh":"2024-09-21T10:47:52.453296+0000","last_change":"2024-09-21T10:44:14.418113+0000","last_active":"2024-09-21T10:47:52.453296+0000","last_peered":"2024-09-21T10:47:52.453296+0000","last_clean":"2024-09-21T10:47:52.453296+0000","last_became_active":"2024-09-21T10:44:14.398327+0000","last_became_peered":"2024-09-21T10:44:14.398327+0000","last_unstale":"2024-09-21T10:47:52.453296+0000","last_undegraded":"2024-09-21T10:47:52.453296+0000","last_fullsized":"2024-09-21T10:47:52.453296+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:08.810863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2095,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.532429+0000","last_change":"2024-09-21T10:44:14.398584+0000","last_active":"2024-09-21T10:48:06.532429+0000","last_peered":"2024-09-21T10:48:06.532429+0000","last_clean":"2024-09-21T10:48:06.532429+0000","last_became_active":"2024-09-21T10:44:14.398278+0000","last_became_peered":"2024-09-21T10:44:14.398278+0000","last_unstale":"2024-09-21T10:48:06.532429+0000","last_undegraded":"2024-09-21T10:48:06.532429+0000","last_fullsized":"2024-09-21T10:48:06.532429+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:19.681750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":54,"num_pg":16,"stat_sum":{"num_bytes":1545797656,"num_objects":23588,"num_object_clones":0,"num_object_copies":70764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23588,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":234634,"num_write_kb":8263041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4673253376,"data_stored":4673241160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121034,"ondisk_log_size":121034,"up":48,"acting":48,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":10310,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234649,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1021148,"kb_used_data":728980,"kb_used_omap":1,"kb_used_meta":291902,"kb_avail":92724004,"statfs":{"total":95995035648,"available":94949380096,"internally_reserved":0,"allocated":746475520,"data_stored":741982656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":298908107},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496286,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1407456,"kb_used_data":943192,"kb_used_omap":1,"kb_used_meta":464190,"kb_avail":92337696,"statfs":{"total":95995035648,"available":94553800704,"internally_reserved":0,"allocated":965828608,"data_stored":961332161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":475331018},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757923,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1144328,"kb_used_data":734208,"kb_used_omap":1,"kb_used_meta":410110,"kb_avail":92600824,"statfs":{"total":95995035648,"available":94823243776,"internally_reserved":0,"allocated":751828992,"data_stored":747351763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":419953094},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019559,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":896468,"kb_used_data":424076,"kb_used_omap":1,"kb_used_meta":472382,"kb_avail":92848684,"statfs":{"total":95995035648,"available":95077052416,"internally_reserved":0,"allocated":434253824,"data_stored":429768538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":483719622},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369303,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1490772,"kb_used_data":989260,"kb_used_omap":69,"kb_used_meta":501306,"kb_avail":92254380,"statfs":{"total":95995035648,"available":94468485120,"internally_reserved":0,"allocated":1013002240,"data_stored":1008520890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":513337474},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2920577761758,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807957989,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":827280,"kb_used_data":367688,"kb_used_omap":66,"kb_used_meta":459517,"kb_avail":92917872,"statfs":{"total":95995035648,"available":95147900928,"internally_reserved":0,"allocated":376512512,"data_stored":372035163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68589,"internal_metadata":470545427},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":666,"seq":2860448219628,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":942812,"kb_used_data":550036,"kb_used_omap":76,"kb_used_meta":392691,"kb_avail":92802340,"statfs":{"total":95995035648,"available":95029596160,"internally_reserved":0,"allocated":563236864,"data_stored":558748380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78716,"internal_metadata":402115716},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":538906624,"data_stored":538902552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":357498880,"data_stored":357498880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":981602304,"data_stored":981598232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":417792000,"data_stored":417792000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":723648512,"data_stored":723648512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":928583680,"data_stored":928579608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":725221376,"data_stored":725221376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:48:13.827 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:48:13.852 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:13.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:13.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:13.954 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:13.986 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:13.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:13.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:14.090 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:14.121 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:14.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:14.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:14.223 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:14.259 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:14.270 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:14.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:14.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:14.371 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:14.421 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:14.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:14.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:14.523 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:14.565 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:14.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:14.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:14.666 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:14.705 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:14.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:14.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:14.735 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:48:14.801 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:14 smithi160 ceph-mon[101879]: from='client.35586 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:48:14.801 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:14 smithi160 ceph-mon[101879]: pgmap v2483: 89 pgs: 89 active+clean; 1.3 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 962 op/s 2024-09-21T10:48:14.807 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:14.831 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:14 smithi070 ceph-mon[107385]: from='client.35586 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:48:14.831 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:14 smithi070 ceph-mon[107385]: pgmap v2483: 89 pgs: 89 active+clean; 1.3 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 962 op/s 2024-09-21T10:48:14.832 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:14 smithi070 ceph-mon[112139]: from='client.35586 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:48:14.832 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:14 smithi070 ceph-mon[112139]: pgmap v2483: 89 pgs: 89 active+clean; 1.3 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 962 op/s 2024-09-21T10:48:14.841 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:14.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:14.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:14.943 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:14.971 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:14.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:14.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:15.074 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:15.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:15.127 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:15.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:15.228 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:15.278 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:48:15.285 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:15.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:15.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:15.388 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:15.432 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:15.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:15.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:15.534 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:15.562 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:15.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:15.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:15.664 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:15.691 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:15.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:15.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:15.793 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:15.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:15.844 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:15.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:15.945 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:15.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:15.978 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:15.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:16.079 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:16.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:16.117 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:16.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:16.209 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:48:16.217 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:16.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:16.296 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:16.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:16.397 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:16.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:16.437 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:16.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:16.464 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:16.538 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:16.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:16.575 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:16.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:16.676 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:16.707 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:16.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:16.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:16.809 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:16.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:16 smithi160 ceph-mon[101879]: pgmap v2484: 89 pgs: 89 active+clean; 1.3 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 954 op/s 2024-09-21T10:48:16.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:16.847 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:16.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:16.854 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:16 smithi070 ceph-mon[107385]: pgmap v2484: 89 pgs: 89 active+clean; 1.3 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 954 op/s 2024-09-21T10:48:16.854 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:16 smithi070 ceph-mon[112139]: pgmap v2484: 89 pgs: 89 active+clean; 1.3 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 954 op/s 2024-09-21T10:48:16.948 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:16.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:16.985 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:16.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:17.086 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:17.112 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:17.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:17.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:17.214 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:17.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:17.246 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:17.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:17.347 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:17.375 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:17.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:17.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:17.477 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:17.501 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:17.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:17.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:17.603 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:17.636 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:17.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:17.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:17.737 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:17.769 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:17.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:17.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:17.870 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:17.900 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:17.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:17.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:18.002 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:18.038 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:48:18.038 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:48:18.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:18.043 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:18.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:18.144 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:18.174 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:18.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:18.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:18.275 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:18.309 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:18.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:18.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:18.410 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:18.459 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:18.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:18.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:18.560 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:18.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:18.600 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:18.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:18.701 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:18.703 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:18 smithi160 ceph-mon[101879]: pgmap v2485: 89 pgs: 89 active+clean; 1.0 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 902 op/s 2024-09-21T10:48:18.736 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:18.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:18.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:18.838 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:18.859 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:18.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:18.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:18.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:18 smithi070 ceph-mon[107385]: pgmap v2485: 89 pgs: 89 active+clean; 1.0 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 902 op/s 2024-09-21T10:48:18.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:18 smithi070 ceph-mon[112139]: pgmap v2485: 89 pgs: 89 active+clean; 1.0 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 902 op/s 2024-09-21T10:48:18.961 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:19.003 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:19.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:19.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:19.048 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:48:19.106 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:19.152 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:19.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:19.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:19.253 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:19.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:19.301 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:19.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:19.402 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:19.427 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:19.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:19.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:19.523 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:19.530 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:19.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:19.569 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:19.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:19.577 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T10:48:19.664 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:48:19.669 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:19.704 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:19 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3519485884' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:48:19.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:19.725 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:19.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:19.826 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:19.864 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:19.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:19.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:19.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:19 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3519485884' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:48:19.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:19 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3519485884' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:48:19.965 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:19.993 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:19.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:19.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:20.095 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:20.135 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:20.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:20.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:20.237 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:20.272 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:20.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:20.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:20.374 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:20.407 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:20.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:20.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:20.509 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:20.553 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:20.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:20.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:20.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:20 smithi070 ceph-mon[107385]: pgmap v2486: 89 pgs: 89 active+clean; 1.0 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 773 op/s 2024-09-21T10:48:20.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:20 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:48:20.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:20 smithi070 ceph-mon[107385]: Cluster is now healthy 2024-09-21T10:48:20.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:20 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3519485884' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:48:20.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:20 smithi070 ceph-mon[107385]: osdmap e1404: 8 total, 8 up, 7 in 2024-09-21T10:48:20.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:48:20.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:48:20.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:48:20.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:48:20.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:20 smithi070 ceph-mon[112139]: pgmap v2486: 89 pgs: 89 active+clean; 1.0 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 773 op/s 2024-09-21T10:48:20.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:20 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:48:20.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:20 smithi070 ceph-mon[112139]: Cluster is now healthy 2024-09-21T10:48:20.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:20 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3519485884' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:48:20.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:20 smithi070 ceph-mon[112139]: osdmap e1404: 8 total, 8 up, 7 in 2024-09-21T10:48:20.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:48:20.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:48:20.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:48:20.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:48:20.655 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:20.684 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:20.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:20.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:20.786 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:20.814 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:20.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:20.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:20 smithi160 ceph-mon[101879]: pgmap v2486: 89 pgs: 89 active+clean; 1.0 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 773 op/s 2024-09-21T10:48:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:20 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:48:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:20 smithi160 ceph-mon[101879]: Cluster is now healthy 2024-09-21T10:48:20.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:20 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3519485884' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:48:20.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:20 smithi160 ceph-mon[101879]: osdmap e1404: 8 total, 8 up, 7 in 2024-09-21T10:48:20.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:48:20.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:48:20.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:48:20.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:48:20.917 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:20.950 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:20.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:20.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:21.051 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:21.081 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:21.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:21.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:21.183 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:21.215 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:21.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:21.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:21.317 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:21.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:21.354 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:21.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:21.455 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:21.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:21.511 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:21.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:21.612 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:21.637 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:21.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:21.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:21.728 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:21 smithi160 ceph-mon[101879]: pgmap v2488: 89 pgs: 89 active+clean; 903 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 915 op/s 2024-09-21T10:48:21.739 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:21.777 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:21.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:21.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:21.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:21 smithi070 ceph-mon[107385]: pgmap v2488: 89 pgs: 89 active+clean; 903 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 915 op/s 2024-09-21T10:48:21.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:21 smithi070 ceph-mon[112139]: pgmap v2488: 89 pgs: 89 active+clean; 903 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 915 op/s 2024-09-21T10:48:21.879 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:21.914 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:21.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:21.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:22.016 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:22.042 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:22.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:22.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:22.048 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:48:22.048 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:48:22.144 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:22.182 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:22.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:22.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:22.283 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:22.316 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:22.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:22.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:22.417 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:22.463 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:22.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:22.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:22.565 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:22.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:22.597 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:22.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:22.698 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:22.714 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:48:22.741 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:22.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:22.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:22.824 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:48:22.842 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:22.872 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:22.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:22.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:22.974 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:23.030 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:23.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:23.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:23.132 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:23.155 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:23.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:23.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:23.209 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:48:23.257 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:23.265 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:48:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:48:23] "GET /metrics HTTP/1.1" 200 36601 "" "Prometheus/2.51.0" 2024-09-21T10:48:23.310 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:23.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:23.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:23.411 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:23.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:23.446 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:23.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:23.547 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:23.574 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:23.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:23.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:23.675 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:23.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:23.716 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:23.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:23.817 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:23.854 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:23.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:23.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:23.956 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:23.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:23.990 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:23.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:24.091 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:24.098 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:23 smithi070 ceph-mon[107385]: pgmap v2489: 89 pgs: 89 active+clean; 762 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 898 op/s 2024-09-21T10:48:24.099 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:23 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/395480103' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:48:24.099 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:23 smithi070 ceph-mon[112139]: pgmap v2489: 89 pgs: 89 active+clean; 762 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 898 op/s 2024-09-21T10:48:24.099 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:23 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/395480103' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:48:24.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:24.132 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:24.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:24.233 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:24.267 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:24.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:24.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:24.303 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:23 smithi160 ceph-mon[101879]: pgmap v2489: 89 pgs: 89 active+clean; 762 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 898 op/s 2024-09-21T10:48:24.303 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:23 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/395480103' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:48:24.369 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:24.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:24.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:24.507 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:24.541 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:24.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:24.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:24.643 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:24.644 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:48:24.644 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:48:24.644 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:48:24.644 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:48:24.644 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:48:24.644 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:48:24.644 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:48:24.645 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:48:24.645 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:48:24.645 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:48:24.645 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:48:24.645 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:48:24.645 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:48:24.645 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:48:24.645 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:48:24.645 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:48:24.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:24.692 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:24.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:24.794 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:24.825 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:24.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:24.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:24.928 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:24.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:24.961 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:24.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:25.062 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:25.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:25.106 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:25.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:25.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:24 smithi070 ceph-mon[107385]: from='client.45536 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:48:25.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:24 smithi070 ceph-mon[112139]: from='client.45536 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:48:25.192 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:24 smithi160 ceph-mon[101879]: from='client.45536 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:48:25.207 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:25.259 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:25.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:25.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:25.362 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:25.364 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:48:25.364 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T10:48:25.364 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T10:48:25.364 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T10:48:25.364 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.364 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.45020.0:69150 53.a e1323/1307 53:5469a6c9:::benchmark_data_smithi070_228967_object69149:head v 1323'4304, mlcod=1322'4302)", 2024-09-21T10:48:25.365 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:40:59.633579+0000", 2024-09-21T10:48:25.365 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 445.72424485900001, 2024-09-21T10:48:25.365 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098647451999999997, 2024-09-21T10:48:25.365 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:25.365 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:25.365 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:25.366 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.366 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:25.366 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.633579+0000", 2024-09-21T10:48:25.366 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.366 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.367 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.367 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:25.367 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.633579+0000", 2024-09-21T10:48:25.367 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.367 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.368 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.368 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:25.368 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.633575+0000", 2024-09-21T10:48:25.368 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999967 2024-09-21T10:48:25.368 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.369 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.369 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:25.369 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.633640+0000", 2024-09-21T10:48:25.369 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5078000000000003e-05 2024-09-21T10:48:25.369 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.370 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.370 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:25.370 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.633650+0000", 2024-09-21T10:48:25.370 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2310000000000002e-06 2024-09-21T10:48:25.370 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.371 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.371 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:25.371 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.633661+0000", 2024-09-21T10:48:25.371 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1141000000000001e-05 2024-09-21T10:48:25.371 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.372 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.372 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:25.372 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.633725+0000", 2024-09-21T10:48:25.372 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4228999999999998e-05 2024-09-21T10:48:25.372 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.373 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.373 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:48:25.373 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.633750+0000", 2024-09-21T10:48:25.373 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5282e-05 2024-09-21T10:48:25.373 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.374 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.374 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:48:25.374 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.729657+0000", 2024-09-21T10:48:25.374 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.095906405 2024-09-21T10:48:25.374 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.375 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.375 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:25.375 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.732226+0000", 2024-09-21T10:48:25.375 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0025695290000000001 2024-09-21T10:48:25.376 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.376 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:25.376 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.376 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.376 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.377 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:72211 53.8 53:10f6294e:::benchmark_data_smithi070_228967_object72210:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1324)", 2024-09-21T10:48:25.377 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:02.573709+0000", 2024-09-21T10:48:25.377 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 442.784115149, 2024-09-21T10:48:25.377 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.125090165, 2024-09-21T10:48:25.377 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:25.378 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:25.378 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:25.378 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:25.378 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:25.378 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 72211 2024-09-21T10:48:25.379 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.379 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:25.379 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.379 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:25.379 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.573709+0000", 2024-09-21T10:48:25.380 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.380 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.380 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.380 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:25.380 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.573709+0000", 2024-09-21T10:48:25.381 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.381 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.381 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.381 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:25.381 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.573708+0000", 2024-09-21T10:48:25.382 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:48:25.382 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.382 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.382 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:25.382 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.573736+0000", 2024-09-21T10:48:25.383 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8065e-05 2024-09-21T10:48:25.383 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.383 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.383 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:25.383 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.573754+0000", 2024-09-21T10:48:25.384 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8134999999999998e-05 2024-09-21T10:48:25.384 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.384 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.384 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:25.384 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.573761+0000", 2024-09-21T10:48:25.385 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5000000000000002e-06 2024-09-21T10:48:25.385 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.385 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.385 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:25.385 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.573956+0000", 2024-09-21T10:48:25.386 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000194814 2024-09-21T10:48:25.386 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.386 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.386 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:48:25.386 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.574103+0000", 2024-09-21T10:48:25.386 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000146408 2024-09-21T10:48:25.387 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.387 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.387 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:48:25.387 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.574221+0000", 2024-09-21T10:48:25.387 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000118412 2024-09-21T10:48:25.387 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.388 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.388 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.388 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.583179+0000", 2024-09-21T10:48:25.388 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0089579670000000007 2024-09-21T10:48:25.388 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.388 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.389 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:48:25.389 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.583183+0000", 2024-09-21T10:48:25.389 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3800000000000004e-06 2024-09-21T10:48:25.389 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.389 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.389 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.389 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.698714+0000", 2024-09-21T10:48:25.390 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.115530562 2024-09-21T10:48:25.390 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.390 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.390 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:48:25.390 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.698764+0000", 2024-09-21T10:48:25.390 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9527999999999997e-05 2024-09-21T10:48:25.391 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.391 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.391 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:25.391 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:02.698799+0000", 2024-09-21T10:48:25.391 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5172000000000001e-05 2024-09-21T10:48:25.391 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.392 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:25.392 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.392 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.392 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.392 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:82094 53.8 53:172d25c6:::benchmark_data_smithi070_228967_object82093:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1334)", 2024-09-21T10:48:25.392 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:14.628422+0000", 2024-09-21T10:48:25.392 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 430.72940196799999, 2024-09-21T10:48:25.392 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10206596699999999, 2024-09-21T10:48:25.393 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:25.393 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:25.393 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:25.393 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:25.393 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:25.393 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 82094 2024-09-21T10:48:25.394 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.394 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:25.394 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.394 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:25.394 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.628422+0000", 2024-09-21T10:48:25.394 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.394 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.395 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.395 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:25.395 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.628422+0000", 2024-09-21T10:48:25.395 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.395 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.395 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.395 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:25.396 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.628419+0000", 2024-09-21T10:48:25.396 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 2024-09-21T10:48:25.396 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.396 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.396 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:25.396 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.628455+0000", 2024-09-21T10:48:25.397 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6025000000000002e-05 2024-09-21T10:48:25.397 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.397 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.397 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:25.397 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.628469+0000", 2024-09-21T10:48:25.397 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3253e-05 2024-09-21T10:48:25.397 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.398 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.398 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:25.398 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.628476+0000", 2024-09-21T10:48:25.398 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4889999999999998e-06 2024-09-21T10:48:25.398 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.398 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.399 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:25.399 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.628513+0000", 2024-09-21T10:48:25.399 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6776999999999997e-05 2024-09-21T10:48:25.399 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.399 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.399 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:48:25.400 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.628617+0000", 2024-09-21T10:48:25.400 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000104369 2024-09-21T10:48:25.400 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.400 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.400 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:48:25.400 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.628710+0000", 2024-09-21T10:48:25.400 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2288999999999997e-05 2024-09-21T10:48:25.401 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.401 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.401 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:48:25.401 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.630530+0000", 2024-09-21T10:48:25.401 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001819942 2024-09-21T10:48:25.401 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.402 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.402 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.402 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.631864+0000", 2024-09-21T10:48:25.402 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0013344769999999999 2024-09-21T10:48:25.402 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.402 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.403 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.403 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.730412+0000", 2024-09-21T10:48:25.403 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098548018000000001 2024-09-21T10:48:25.403 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.403 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.403 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:48:25.403 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.730452+0000", 2024-09-21T10:48:25.404 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9804e-05 2024-09-21T10:48:25.404 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.404 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.404 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:25.404 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.730488+0000", 2024-09-21T10:48:25.404 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5768e-05 2024-09-21T10:48:25.404 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.405 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:25.405 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.405 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.405 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.405 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:82102 53.8 53:16f516de:::benchmark_data_smithi070_228967_object82101:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1334)", 2024-09-21T10:48:25.405 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:14.632737+0000", 2024-09-21T10:48:25.406 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 430.72508689300003, 2024-09-21T10:48:25.406 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.103853668, 2024-09-21T10:48:25.406 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:25.406 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:25.406 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:25.406 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:25.407 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:25.407 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 82102 2024-09-21T10:48:25.407 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.407 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:25.407 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.407 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:25.408 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.632737+0000", 2024-09-21T10:48:25.408 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.408 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.408 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.408 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:25.408 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.632737+0000", 2024-09-21T10:48:25.408 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.409 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.409 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.409 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:25.409 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.632735+0000", 2024-09-21T10:48:25.410 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:48:25.410 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.410 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.410 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:25.410 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.632789+0000", 2024-09-21T10:48:25.410 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3702000000000002e-05 2024-09-21T10:48:25.411 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.411 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.411 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:25.411 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.632804+0000", 2024-09-21T10:48:25.411 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5557e-05 2024-09-21T10:48:25.411 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.411 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.412 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:25.412 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.632812+0000", 2024-09-21T10:48:25.412 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2490000000000004e-06 2024-09-21T10:48:25.412 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.412 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.412 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:25.413 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.632855+0000", 2024-09-21T10:48:25.413 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3569000000000002e-05 2024-09-21T10:48:25.413 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.413 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.413 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:48:25.413 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.633008+0000", 2024-09-21T10:48:25.413 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000152845 2024-09-21T10:48:25.414 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.414 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.414 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:48:25.414 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.633191+0000", 2024-09-21T10:48:25.414 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000183427 2024-09-21T10:48:25.414 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.415 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.415 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:48:25.415 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.634595+0000", 2024-09-21T10:48:25.415 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0014039650000000001 2024-09-21T10:48:25.415 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.415 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.416 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.416 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.635290+0000", 2024-09-21T10:48:25.416 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000695063 2024-09-21T10:48:25.416 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.416 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.416 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.417 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.736510+0000", 2024-09-21T10:48:25.417 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.101219369 2024-09-21T10:48:25.417 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.417 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.417 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:48:25.417 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.736556+0000", 2024-09-21T10:48:25.417 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6711000000000001e-05 2024-09-21T10:48:25.418 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.418 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.418 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:25.418 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:14.736590+0000", 2024-09-21T10:48:25.418 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3955000000000003e-05 2024-09-21T10:48:25.418 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.419 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:25.419 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.419 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.419 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.419 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:107445 53.18 53:1cc3ffc3:::benchmark_data_smithi070_228967_object107444:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1338)", 2024-09-21T10:48:25.419 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:34.749308+0000", 2024-09-21T10:48:25.419 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 410.60851546999999, 2024-09-21T10:48:25.420 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.105378707, 2024-09-21T10:48:25.420 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:25.420 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:25.420 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:25.420 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:25.420 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:25.420 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 107445 2024-09-21T10:48:25.420 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.420 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:25.420 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.421 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:25.421 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.749308+0000", 2024-09-21T10:48:25.421 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.421 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.421 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.421 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:25.421 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.749308+0000", 2024-09-21T10:48:25.421 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.421 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.421 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.421 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:25.421 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.749306+0000", 2024-09-21T10:48:25.421 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:48:25.421 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.422 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.422 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:25.422 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.749363+0000", 2024-09-21T10:48:25.422 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.677e-05 2024-09-21T10:48:25.422 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.422 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.422 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:25.422 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.749375+0000", 2024-09-21T10:48:25.422 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2138e-05 2024-09-21T10:48:25.422 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.422 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.422 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:25.422 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.749382+0000", 2024-09-21T10:48:25.423 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0119999999999999e-06 2024-09-21T10:48:25.423 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.423 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.423 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:25.423 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.749427+0000", 2024-09-21T10:48:25.423 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4496999999999998e-05 2024-09-21T10:48:25.423 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.423 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.423 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:48:25.423 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.749569+0000", 2024-09-21T10:48:25.423 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000141915 2024-09-21T10:48:25.423 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.423 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.424 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:48:25.425 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.749917+0000", 2024-09-21T10:48:25.425 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034883599999999999 2024-09-21T10:48:25.425 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.425 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.425 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.425 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.753701+0000", 2024-09-21T10:48:25.426 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0037838910000000002 2024-09-21T10:48:25.426 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.426 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.426 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:48:25.426 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.753828+0000", 2024-09-21T10:48:25.426 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012615 2024-09-21T10:48:25.426 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.426 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.426 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.426 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.854616+0000", 2024-09-21T10:48:25.426 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100788135 2024-09-21T10:48:25.426 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.427 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.427 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:48:25.427 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.854650+0000", 2024-09-21T10:48:25.427 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3943999999999999e-05 2024-09-21T10:48:25.427 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.427 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.427 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:25.427 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:34.854687+0000", 2024-09-21T10:48:25.427 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7320000000000002e-05 2024-09-21T10:48:25.427 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.427 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:25.427 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.427 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.428 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.428 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:149478 53.8 53:10e8e2f8:::benchmark_data_smithi070_228967_object20558:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:48:25.428 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:42:14.689470+0000", 2024-09-21T10:48:25.428 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 370.668354024, 2024-09-21T10:48:25.428 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100990149, 2024-09-21T10:48:25.428 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:25.428 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:25.428 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:25.428 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:25.428 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:25.428 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 149478 2024-09-21T10:48:25.428 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.428 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:25.428 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.429 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:25.429 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.689470+0000", 2024-09-21T10:48:25.429 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.429 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.429 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.429 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:25.429 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.689470+0000", 2024-09-21T10:48:25.429 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.429 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.429 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.429 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:25.429 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.689468+0000", 2024-09-21T10:48:25.429 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:48:25.430 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.430 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.430 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:25.430 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.689474+0000", 2024-09-21T10:48:25.430 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9560000000000002e-06 2024-09-21T10:48:25.430 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.430 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.430 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:25.430 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.689487+0000", 2024-09-21T10:48:25.430 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2499e-05 2024-09-21T10:48:25.430 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.430 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.430 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:25.431 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.689497+0000", 2024-09-21T10:48:25.431 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.023e-05 2024-09-21T10:48:25.431 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.431 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.431 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:25.431 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.689544+0000", 2024-09-21T10:48:25.431 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6703999999999999e-05 2024-09-21T10:48:25.431 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.431 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.431 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:48:25.431 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.689663+0000", 2024-09-21T10:48:25.431 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119355 2024-09-21T10:48:25.431 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.431 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.432 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:48:25.432 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.690075+0000", 2024-09-21T10:48:25.432 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000411467 2024-09-21T10:48:25.432 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.432 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.432 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:48:25.432 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.695058+0000", 2024-09-21T10:48:25.432 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0049839389999999997 2024-09-21T10:48:25.432 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.432 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.432 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.432 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.780094+0000", 2024-09-21T10:48:25.432 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.085035225000000006 2024-09-21T10:48:25.432 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.433 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.433 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.433 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.790371+0000", 2024-09-21T10:48:25.433 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.010277435999999999 2024-09-21T10:48:25.433 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.433 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.433 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:48:25.433 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.790423+0000", 2024-09-21T10:48:25.433 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1966999999999999e-05 2024-09-21T10:48:25.433 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.433 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.433 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:25.433 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.790460+0000", 2024-09-21T10:48:25.434 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6711000000000001e-05 2024-09-21T10:48:25.434 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.434 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:25.434 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.434 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.434 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.434 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:149481 53.18 53:1dddc79d:::benchmark_data_smithi070_228967_object20561:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:48:25.434 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:42:14.692316+0000", 2024-09-21T10:48:25.434 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 370.66550794300002, 2024-09-21T10:48:25.434 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.095082052, 2024-09-21T10:48:25.434 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:25.434 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:25.434 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:25.434 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:25.435 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:25.435 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 149481 2024-09-21T10:48:25.435 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.435 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:25.435 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.435 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:25.435 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.692316+0000", 2024-09-21T10:48:25.435 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.435 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.435 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.435 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:25.435 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.692316+0000", 2024-09-21T10:48:25.435 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.435 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.436 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.436 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:25.436 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.692314+0000", 2024-09-21T10:48:25.436 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:48:25.436 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.436 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.436 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:25.436 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.692320+0000", 2024-09-21T10:48:25.436 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.319e-06 2024-09-21T10:48:25.436 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.436 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.436 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:25.436 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.692334+0000", 2024-09-21T10:48:25.436 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4011e-05 2024-09-21T10:48:25.437 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.437 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.437 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:25.437 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.692342+0000", 2024-09-21T10:48:25.437 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5009999999999998e-06 2024-09-21T10:48:25.437 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.437 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.437 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:25.437 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.692453+0000", 2024-09-21T10:48:25.437 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011135499999999999 2024-09-21T10:48:25.437 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.437 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.437 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:48:25.437 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.692558+0000", 2024-09-21T10:48:25.438 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000105214 2024-09-21T10:48:25.438 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.438 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.438 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:48:25.438 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.692994+0000", 2024-09-21T10:48:25.438 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00043552400000000001 2024-09-21T10:48:25.438 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.438 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.438 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:48:25.438 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.695191+0000", 2024-09-21T10:48:25.438 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021967739999999999 2024-09-21T10:48:25.438 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.438 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.438 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.439 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.785049+0000", 2024-09-21T10:48:25.439 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.089858510000000003 2024-09-21T10:48:25.439 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.439 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.439 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.439 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.787279+0000", 2024-09-21T10:48:25.439 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022303010000000001 2024-09-21T10:48:25.439 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.439 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.439 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:48:25.439 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.787359+0000", 2024-09-21T10:48:25.439 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9586999999999995e-05 2024-09-21T10:48:25.439 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.439 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.440 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:25.440 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:14.787398+0000", 2024-09-21T10:48:25.440 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8770000000000003e-05 2024-09-21T10:48:25.440 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.440 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:25.440 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.440 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.440 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.440 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:150502 53.8 53:16f8466a:::benchmark_data_smithi070_228967_object21582:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:48:25.440 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:42:15.781123+0000", 2024-09-21T10:48:25.440 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 369.57670022999997, 2024-09-21T10:48:25.440 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.107127848, 2024-09-21T10:48:25.440 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:25.440 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:25.441 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:25.441 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:25.441 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:25.441 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 150502 2024-09-21T10:48:25.441 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.441 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:25.441 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.441 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:25.441 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.781123+0000", 2024-09-21T10:48:25.441 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.441 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.441 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.441 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:25.442 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.781123+0000", 2024-09-21T10:48:25.442 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.442 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.442 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.442 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:25.442 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.781122+0000", 2024-09-21T10:48:25.442 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:48:25.442 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.442 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.442 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:25.442 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.781129+0000", 2024-09-21T10:48:25.442 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6980000000000001e-06 2024-09-21T10:48:25.442 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.442 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.443 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:25.443 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.781140+0000", 2024-09-21T10:48:25.443 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0832000000000001e-05 2024-09-21T10:48:25.443 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.443 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.443 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:25.443 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.781145+0000", 2024-09-21T10:48:25.443 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3310000000000003e-06 2024-09-21T10:48:25.443 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.443 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.443 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:25.443 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.781201+0000", 2024-09-21T10:48:25.443 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6326000000000002e-05 2024-09-21T10:48:25.443 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.444 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.444 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:48:25.444 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.781316+0000", 2024-09-21T10:48:25.444 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000115028 2024-09-21T10:48:25.444 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.444 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.444 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:48:25.444 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.781745+0000", 2024-09-21T10:48:25.444 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00042860799999999997 2024-09-21T10:48:25.444 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.444 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.444 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:48:25.444 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.783714+0000", 2024-09-21T10:48:25.444 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0019695670000000002 2024-09-21T10:48:25.445 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.445 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.445 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.445 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.785918+0000", 2024-09-21T10:48:25.445 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022033420000000001 2024-09-21T10:48:25.445 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.445 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.445 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.445 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.888167+0000", 2024-09-21T10:48:25.445 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.1022491 2024-09-21T10:48:25.445 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.445 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.445 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:48:25.445 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.888213+0000", 2024-09-21T10:48:25.446 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5768e-05 2024-09-21T10:48:25.446 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.446 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.446 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:25.446 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:15.888251+0000", 2024-09-21T10:48:25.446 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8658000000000002e-05 2024-09-21T10:48:25.446 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.446 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:25.446 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.446 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.446 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.446 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:151895 53.0 53:0504c56e:::benchmark_data_smithi070_228967_object22975:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:48:25.446 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:42:17.156006+0000", 2024-09-21T10:48:25.446 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 368.20181812800001, 2024-09-21T10:48:25.447 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.091976057, 2024-09-21T10:48:25.447 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:25.447 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:25.447 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:25.447 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:25.447 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:25.447 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 151895 2024-09-21T10:48:25.447 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.447 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:25.447 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.447 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:25.447 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.156006+0000", 2024-09-21T10:48:25.447 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.447 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.448 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.448 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:25.448 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.156006+0000", 2024-09-21T10:48:25.448 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.448 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.448 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.448 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:25.448 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.156004+0000", 2024-09-21T10:48:25.448 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:48:25.448 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.448 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.448 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:25.448 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.156010+0000", 2024-09-21T10:48:25.448 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0630000000000002e-06 2024-09-21T10:48:25.448 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.449 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.449 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:25.449 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.156020+0000", 2024-09-21T10:48:25.449 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3479999999999993e-06 2024-09-21T10:48:25.449 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.449 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.449 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:25.449 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.156026+0000", 2024-09-21T10:48:25.449 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0299999999999999e-06 2024-09-21T10:48:25.449 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.449 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.449 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:25.449 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.156056+0000", 2024-09-21T10:48:25.449 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0229000000000001e-05 2024-09-21T10:48:25.450 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.450 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.450 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:48:25.450 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.156136+0000", 2024-09-21T10:48:25.450 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0018000000000005e-05 2024-09-21T10:48:25.450 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.450 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.450 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:48:25.450 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.156494+0000", 2024-09-21T10:48:25.450 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00035786699999999999 2024-09-21T10:48:25.450 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.450 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.450 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:48:25.450 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.158749+0000", 2024-09-21T10:48:25.451 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022554290000000002 2024-09-21T10:48:25.451 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.451 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.451 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.451 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.159189+0000", 2024-09-21T10:48:25.451 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000440178 2024-09-21T10:48:25.451 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.451 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.451 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.451 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.247897+0000", 2024-09-21T10:48:25.451 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.088707649 2024-09-21T10:48:25.451 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.451 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.451 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:48:25.452 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.247946+0000", 2024-09-21T10:48:25.452 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9002999999999998e-05 2024-09-21T10:48:25.452 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.452 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.452 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:25.452 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:17.247982+0000", 2024-09-21T10:48:25.452 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5448999999999999e-05 2024-09-21T10:48:25.452 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.452 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:25.452 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.452 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.452 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.452 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:179070 53.0 53:06391b30:::benchmark_data_smithi070_228967_object50150:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1345)", 2024-09-21T10:48:25.452 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:42:45.010126+0000", 2024-09-21T10:48:25.453 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 340.34769743499999, 2024-09-21T10:48:25.453 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098288894000000002, 2024-09-21T10:48:25.453 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:25.453 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:25.453 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:25.453 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:25.453 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:25.453 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 179070 2024-09-21T10:48:25.453 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.453 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:25.453 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.453 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:25.453 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.010126+0000", 2024-09-21T10:48:25.453 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.454 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.454 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.454 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:25.454 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.010126+0000", 2024-09-21T10:48:25.454 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.454 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.454 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.454 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:25.454 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.010125+0000", 2024-09-21T10:48:25.454 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:48:25.454 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.454 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.454 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:25.455 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.010131+0000", 2024-09-21T10:48:25.455 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.776e-06 2024-09-21T10:48:25.455 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.455 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.455 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:25.455 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.010142+0000", 2024-09-21T10:48:25.455 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0584000000000001e-05 2024-09-21T10:48:25.455 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.455 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.455 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:25.455 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.010149+0000", 2024-09-21T10:48:25.455 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1509999999999998e-06 2024-09-21T10:48:25.455 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.455 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.456 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:25.456 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.010188+0000", 2024-09-21T10:48:25.456 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9255000000000003e-05 2024-09-21T10:48:25.456 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.456 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.456 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:48:25.456 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.010290+0000", 2024-09-21T10:48:25.456 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010159500000000001 2024-09-21T10:48:25.456 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.456 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.456 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:48:25.456 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.010737+0000", 2024-09-21T10:48:25.456 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00044662 2024-09-21T10:48:25.456 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.457 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.457 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:48:25.457 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.013852+0000", 2024-09-21T10:48:25.457 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0031150409999999998 2024-09-21T10:48:25.457 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.457 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.457 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.457 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.014261+0000", 2024-09-21T10:48:25.457 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00040910700000000003 2024-09-21T10:48:25.457 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.457 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.457 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.457 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.108298+0000", 2024-09-21T10:48:25.457 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.094037134999999994 2024-09-21T10:48:25.457 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.458 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.458 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:48:25.458 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.108388+0000", 2024-09-21T10:48:25.458 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0221000000000002e-05 2024-09-21T10:48:25.458 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.458 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.458 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:25.458 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.108415+0000", 2024-09-21T10:48:25.458 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6990000000000001e-05 2024-09-21T10:48:25.458 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.458 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:25.458 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.458 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.459 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.459 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:179075 53.0 53:0484e749:::benchmark_data_smithi070_228967_object50155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1345)", 2024-09-21T10:48:25.459 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:42:45.017022+0000", 2024-09-21T10:48:25.459 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 340.34080180000001, 2024-09-21T10:48:25.459 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.091482206999999996, 2024-09-21T10:48:25.459 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:25.459 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:25.459 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:25.459 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:25.459 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:25.459 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 179075 2024-09-21T10:48:25.459 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.459 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:25.459 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.460 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:25.460 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.017022+0000", 2024-09-21T10:48:25.460 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.460 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.460 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.460 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:25.460 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.017022+0000", 2024-09-21T10:48:25.460 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.460 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.460 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.460 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:25.460 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.017021+0000", 2024-09-21T10:48:25.460 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:48:25.460 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.461 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.461 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:25.461 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.017026+0000", 2024-09-21T10:48:25.461 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2530000000000004e-06 2024-09-21T10:48:25.461 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.461 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.461 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:25.461 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.017037+0000", 2024-09-21T10:48:25.461 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1136e-05 2024-09-21T10:48:25.461 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.461 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.461 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:25.461 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.017042+0000", 2024-09-21T10:48:25.461 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2399999999999998e-06 2024-09-21T10:48:25.462 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.462 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.462 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:25.462 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.017080+0000", 2024-09-21T10:48:25.462 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8093000000000003e-05 2024-09-21T10:48:25.462 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.462 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.462 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:48:25.462 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.017171+0000", 2024-09-21T10:48:25.462 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0397999999999997e-05 2024-09-21T10:48:25.462 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.462 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.462 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:48:25.463 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.017542+0000", 2024-09-21T10:48:25.463 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00037132400000000002 2024-09-21T10:48:25.463 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.463 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.463 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:48:25.463 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.019425+0000", 2024-09-21T10:48:25.463 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0018829280000000001 2024-09-21T10:48:25.463 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.463 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.463 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.463 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.019898+0000", 2024-09-21T10:48:25.463 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00047353100000000002 2024-09-21T10:48:25.463 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.463 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.463 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.464 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.108452+0000", 2024-09-21T10:48:25.464 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.088553617000000001 2024-09-21T10:48:25.464 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.464 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.464 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:48:25.464 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.108482+0000", 2024-09-21T10:48:25.464 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9458000000000002e-05 2024-09-21T10:48:25.464 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.464 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.464 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:25.464 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:45.108504+0000", 2024-09-21T10:48:25.464 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2501999999999999e-05 2024-09-21T10:48:25.464 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.464 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:25.465 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.465 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.465 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.465 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:182128 53.8 53:14cb5112:::benchmark_data_smithi070_228967_object53208:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1345)", 2024-09-21T10:48:25.465 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:42:48.035899+0000", 2024-09-21T10:48:25.465 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 337.321924905, 2024-09-21T10:48:25.465 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.093860108999999997, 2024-09-21T10:48:25.465 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:25.465 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:25.465 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:25.465 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:25.465 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:25.465 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 182128 2024-09-21T10:48:25.465 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.466 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:25.466 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.466 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:25.466 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:48.035899+0000", 2024-09-21T10:48:25.466 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.466 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.466 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.466 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:25.466 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:48.035899+0000", 2024-09-21T10:48:25.466 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.466 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.466 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.466 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:25.466 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:48.035897+0000", 2024-09-21T10:48:25.467 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:48:25.467 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.467 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.467 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:25.467 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:48.035904+0000", 2024-09-21T10:48:25.467 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.241e-06 2024-09-21T10:48:25.467 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.467 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.467 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:25.467 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:48.035915+0000", 2024-09-21T10:48:25.467 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1735e-05 2024-09-21T10:48:25.467 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.467 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.467 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:25.468 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:48.035922+0000", 2024-09-21T10:48:25.468 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3080000000000004e-06 2024-09-21T10:48:25.468 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.468 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.468 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:25.468 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:48.039418+0000", 2024-09-21T10:48:25.468 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0034962130000000002 2024-09-21T10:48:25.468 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.468 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.468 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:48:25.468 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:48.039525+0000", 2024-09-21T10:48:25.468 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000107452 2024-09-21T10:48:25.468 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.468 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.469 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:48:25.469 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:48.040037+0000", 2024-09-21T10:48:25.469 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00051125400000000005 2024-09-21T10:48:25.469 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.469 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.469 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:48:25.469 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:48.042472+0000", 2024-09-21T10:48:25.469 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0024356669999999999 2024-09-21T10:48:25.469 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.469 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.469 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.469 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:48.049318+0000", 2024-09-21T10:48:25.469 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0068460029999999998 2024-09-21T10:48:25.469 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.470 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.470 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.470 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:48.129705+0000", 2024-09-21T10:48:25.470 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.080386985999999994 2024-09-21T10:48:25.470 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.470 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.470 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:48:25.470 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:48.129735+0000", 2024-09-21T10:48:25.470 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9473999999999999e-05 2024-09-21T10:48:25.470 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.470 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.470 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:25.470 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:42:48.129759+0000", 2024-09-21T10:48:25.471 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4236e-05 2024-09-21T10:48:25.471 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.471 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:25.471 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.471 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.471 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.471 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:62979 54.8 54:124a0706:::benchmark_data_smithi070_261699_object62978:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1382)", 2024-09-21T10:48:25.471 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:16.804088+0000", 2024-09-21T10:48:25.471 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 188.55373567000001, 2024-09-21T10:48:25.471 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.17356136999999999, 2024-09-21T10:48:25.471 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:25.471 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:25.471 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:25.471 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:48:25.472 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:48:25.472 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 62979 2024-09-21T10:48:25.472 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.472 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:25.472 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.472 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:25.472 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.804088+0000", 2024-09-21T10:48:25.472 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.472 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.472 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.472 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:25.472 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.804088+0000", 2024-09-21T10:48:25.472 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.472 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.473 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.473 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:25.473 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.804086+0000", 2024-09-21T10:48:25.473 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:48:25.473 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.473 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.473 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:25.473 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.804123+0000", 2024-09-21T10:48:25.473 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6477000000000003e-05 2024-09-21T10:48:25.473 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.473 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.473 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:25.473 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.804139+0000", 2024-09-21T10:48:25.474 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5608999999999999e-05 2024-09-21T10:48:25.474 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.474 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.474 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:25.474 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.804146+0000", 2024-09-21T10:48:25.474 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5120000000000002e-06 2024-09-21T10:48:25.474 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.474 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.474 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:25.474 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.804266+0000", 2024-09-21T10:48:25.474 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000120048 2024-09-21T10:48:25.474 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.474 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.475 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:48:25.475 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.804382+0000", 2024-09-21T10:48:25.475 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000115902 2024-09-21T10:48:25.475 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.475 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.475 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:48:25.475 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.804499+0000", 2024-09-21T10:48:25.475 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000117192 2024-09-21T10:48:25.475 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.475 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.475 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:48:25.475 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.806645+0000", 2024-09-21T10:48:25.475 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021456660000000001 2024-09-21T10:48:25.475 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.476 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.476 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.476 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.812886+0000", 2024-09-21T10:48:25.476 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0062415250000000004 2024-09-21T10:48:25.476 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.476 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.476 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.476 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.977555+0000", 2024-09-21T10:48:25.476 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16466847500000001 2024-09-21T10:48:25.476 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.476 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.476 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:48:25.476 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.977610+0000", 2024-09-21T10:48:25.476 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5195999999999998e-05 2024-09-21T10:48:25.477 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.477 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.477 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:25.477 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.977649+0000", 2024-09-21T10:48:25.477 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9267999999999997e-05 2024-09-21T10:48:25.477 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.477 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:25.477 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.477 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.477 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.477 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:62983 54.8 54:1bc53d7e:::benchmark_data_smithi070_261699_object62982:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1382)", 2024-09-21T10:48:25.477 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:16.805440+0000", 2024-09-21T10:48:25.477 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 188.552383913, 2024-09-21T10:48:25.477 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.172337825, 2024-09-21T10:48:25.478 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:25.478 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:25.478 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:25.478 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:48:25.478 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:48:25.478 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 62983 2024-09-21T10:48:25.478 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.478 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:25.478 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.478 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:25.478 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.805440+0000", 2024-09-21T10:48:25.478 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.478 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.479 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.479 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:25.479 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.805440+0000", 2024-09-21T10:48:25.479 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.479 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.479 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.479 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:25.479 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.805438+0000", 2024-09-21T10:48:25.479 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:48:25.479 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.479 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.479 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:25.479 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.805467+0000", 2024-09-21T10:48:25.479 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8269e-05 2024-09-21T10:48:25.480 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.480 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.480 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:25.480 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.805482+0000", 2024-09-21T10:48:25.480 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5766000000000001e-05 2024-09-21T10:48:25.480 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.480 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.480 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:25.480 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.805489+0000", 2024-09-21T10:48:25.480 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0570000000000004e-06 2024-09-21T10:48:25.480 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.480 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.480 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:25.481 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.807208+0000", 2024-09-21T10:48:25.481 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0017185200000000001 2024-09-21T10:48:25.481 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.481 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.481 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:48:25.481 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.807328+0000", 2024-09-21T10:48:25.481 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119848 2024-09-21T10:48:25.481 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.481 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.481 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:48:25.481 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.807459+0000", 2024-09-21T10:48:25.481 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013086799999999999 2024-09-21T10:48:25.481 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.481 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.482 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.482 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.812947+0000", 2024-09-21T10:48:25.482 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0054882489999999997 2024-09-21T10:48:25.482 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.482 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.482 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:48:25.482 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.812955+0000", 2024-09-21T10:48:25.482 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1149999999999994e-06 2024-09-21T10:48:25.482 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.482 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.482 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.482 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.977705+0000", 2024-09-21T10:48:25.482 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16474944599999999 2024-09-21T10:48:25.482 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.483 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.483 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:48:25.483 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.977745+0000", 2024-09-21T10:48:25.483 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0788000000000003e-05 2024-09-21T10:48:25.483 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.483 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.483 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:25.483 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.977778+0000", 2024-09-21T10:48:25.483 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2249000000000002e-05 2024-09-21T10:48:25.483 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.483 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:25.483 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.483 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.484 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.484 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:62991 54.8 54:17e38a96:::benchmark_data_smithi070_261699_object62990:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1382)", 2024-09-21T10:48:25.484 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:16.809905+0000", 2024-09-21T10:48:25.484 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 188.54791873900001, 2024-09-21T10:48:25.484 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.17707988299999999, 2024-09-21T10:48:25.484 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:25.484 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:25.484 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:25.484 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:48:25.484 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:48:25.484 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 62991 2024-09-21T10:48:25.484 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.484 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:25.484 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.485 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:25.485 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.809905+0000", 2024-09-21T10:48:25.485 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.485 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.485 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.485 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:25.485 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.809905+0000", 2024-09-21T10:48:25.485 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.485 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.485 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.485 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:25.485 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.809904+0000", 2024-09-21T10:48:25.485 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:48:25.485 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.486 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.486 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:25.486 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.809935+0000", 2024-09-21T10:48:25.486 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1415000000000003e-05 2024-09-21T10:48:25.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.486 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.486 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:25.486 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.809945+0000", 2024-09-21T10:48:25.486 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7240000000000004e-06 2024-09-21T10:48:25.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.486 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.486 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:25.486 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.809952+0000", 2024-09-21T10:48:25.486 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8430000000000004e-06 2024-09-21T10:48:25.487 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.487 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.487 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:25.487 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.809997+0000", 2024-09-21T10:48:25.487 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5275999999999998e-05 2024-09-21T10:48:25.487 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.487 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.487 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:48:25.487 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.810123+0000", 2024-09-21T10:48:25.487 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012623800000000001 2024-09-21T10:48:25.487 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.487 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.487 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:48:25.487 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.810253+0000", 2024-09-21T10:48:25.488 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000129751 2024-09-21T10:48:25.488 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.488 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.488 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.488 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.814963+0000", 2024-09-21T10:48:25.488 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0047100110000000001 2024-09-21T10:48:25.488 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.488 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.488 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:48:25.488 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.816768+0000", 2024-09-21T10:48:25.488 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0018055390000000001 2024-09-21T10:48:25.488 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.488 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.488 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.489 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.986870+0000", 2024-09-21T10:48:25.489 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.170101531 2024-09-21T10:48:25.489 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.489 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.489 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:48:25.489 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.986949+0000", 2024-09-21T10:48:25.489 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9368000000000003e-05 2024-09-21T10:48:25.489 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.489 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.489 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:25.489 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.986985+0000", 2024-09-21T10:48:25.489 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5488000000000003e-05 2024-09-21T10:48:25.489 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.489 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:25.490 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.490 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.490 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.490 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:199065 54.e 54:7def6a72:::benchmark_data_smithi070_261699_object69952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1397)", 2024-09-21T10:48:25.490 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:47:30.437124+0000", 2024-09-21T10:48:25.490 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 54.920699829, 2024-09-21T10:48:25.490 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.132874254, 2024-09-21T10:48:25.490 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:25.490 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:25.490 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:25.490 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:48:25.490 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:48:25.490 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 199065 2024-09-21T10:48:25.490 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.491 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:25.491 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.491 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:25.491 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.437124+0000", 2024-09-21T10:48:25.491 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.491 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.491 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.491 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:25.491 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.437124+0000", 2024-09-21T10:48:25.491 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.491 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.491 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.491 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:25.491 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.437123+0000", 2024-09-21T10:48:25.492 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:48:25.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.492 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.492 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:25.492 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.437128+0000", 2024-09-21T10:48:25.492 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4049999999999999e-06 2024-09-21T10:48:25.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.492 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.492 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:25.492 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.437137+0000", 2024-09-21T10:48:25.492 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3859999999999996e-06 2024-09-21T10:48:25.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.492 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.492 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:25.492 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.437145+0000", 2024-09-21T10:48:25.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7239999999999992e-06 2024-09-21T10:48:25.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:25.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.437186+0000", 2024-09-21T10:48:25.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1019000000000001e-05 2024-09-21T10:48:25.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:48:25.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.437288+0000", 2024-09-21T10:48:25.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000101956 2024-09-21T10:48:25.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,7", 2024-09-21T10:48:25.494 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.437389+0000", 2024-09-21T10:48:25.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000100451 2024-09-21T10:48:25.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.494 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.494 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.494 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.442342+0000", 2024-09-21T10:48:25.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0049530049999999999 2024-09-21T10:48:25.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.494 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.494 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:48:25.494 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.446206+0000", 2024-09-21T10:48:25.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003864751 2024-09-21T10:48:25.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.495 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.495 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.495 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.569921+0000", 2024-09-21T10:48:25.495 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12371433799999999 2024-09-21T10:48:25.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.495 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.495 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:48:25.495 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.569956+0000", 2024-09-21T10:48:25.495 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5797999999999998e-05 2024-09-21T10:48:25.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.495 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.495 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:25.495 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.569998+0000", 2024-09-21T10:48:25.495 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1644000000000003e-05 2024-09-21T10:48:25.496 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.496 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:25.496 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.496 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:199075 54.e 54:7afd5386:::benchmark_data_smithi070_261699_object69962:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1397)", 2024-09-21T10:48:25.496 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:47:30.450625+0000", 2024-09-21T10:48:25.496 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 54.907198207, 2024-09-21T10:48:25.496 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.124872817, 2024-09-21T10:48:25.496 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:25.496 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:25.496 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:25.496 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:48:25.496 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:48:25.497 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 199075 2024-09-21T10:48:25.497 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.497 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:25.497 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.497 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:25.497 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.450625+0000", 2024-09-21T10:48:25.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.497 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.497 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.497 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:25.497 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.450625+0000", 2024-09-21T10:48:25.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.497 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.497 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.498 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:25.498 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.450624+0000", 2024-09-21T10:48:25.498 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:48:25.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.498 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:25.498 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.450631+0000", 2024-09-21T10:48:25.498 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.182e-06 2024-09-21T10:48:25.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.498 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:25.498 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.450643+0000", 2024-09-21T10:48:25.498 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2177999999999999e-05 2024-09-21T10:48:25.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.499 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:25.499 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.450651+0000", 2024-09-21T10:48:25.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3590000000000002e-06 2024-09-21T10:48:25.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.499 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:25.499 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.450705+0000", 2024-09-21T10:48:25.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4573000000000001e-05 2024-09-21T10:48:25.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.499 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:48:25.499 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.451226+0000", 2024-09-21T10:48:25.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00052119899999999997 2024-09-21T10:48:25.500 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.500 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.500 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,7", 2024-09-21T10:48:25.500 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.451562+0000", 2024-09-21T10:48:25.500 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00033518900000000001 2024-09-21T10:48:25.500 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.500 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.500 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:48:25.500 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.460326+0000", 2024-09-21T10:48:25.500 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0087646949999999994 2024-09-21T10:48:25.500 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.500 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.500 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.500 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.466576+0000", 2024-09-21T10:48:25.501 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0062499340000000004 2024-09-21T10:48:25.501 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.501 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.501 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.501 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.575369+0000", 2024-09-21T10:48:25.501 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10879272299999999 2024-09-21T10:48:25.501 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.501 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.501 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:48:25.501 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.575462+0000", 2024-09-21T10:48:25.501 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2658999999999995e-05 2024-09-21T10:48:25.501 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.501 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.501 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:25.502 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.575498+0000", 2024-09-21T10:48:25.502 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6655000000000001e-05 2024-09-21T10:48:25.502 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.502 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:25.502 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.502 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.502 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.502 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:199076 54.e 54:7f608d74:::benchmark_data_smithi070_261699_object69963:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1397)", 2024-09-21T10:48:25.502 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:47:30.450760+0000", 2024-09-21T10:48:25.502 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 54.907064128000002, 2024-09-21T10:48:25.502 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.124826426, 2024-09-21T10:48:25.502 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:25.502 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:25.502 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:25.503 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:48:25.503 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:48:25.503 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 199076 2024-09-21T10:48:25.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.503 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:25.503 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.503 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:25.503 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.450760+0000", 2024-09-21T10:48:25.503 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.503 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.503 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:25.503 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.450760+0000", 2024-09-21T10:48:25.503 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.504 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:25.504 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.450759+0000", 2024-09-21T10:48:25.504 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T10:48:25.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.504 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:25.504 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.450763+0000", 2024-09-21T10:48:25.504 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9169999999999999e-06 2024-09-21T10:48:25.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.504 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:25.504 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.450779+0000", 2024-09-21T10:48:25.504 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5781999999999999e-05 2024-09-21T10:48:25.505 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.505 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:25.505 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.450784+0000", 2024-09-21T10:48:25.505 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.604e-06 2024-09-21T10:48:25.505 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.505 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:25.505 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.456076+0000", 2024-09-21T10:48:25.505 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0052919009999999999 2024-09-21T10:48:25.505 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.505 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:48:25.505 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.456217+0000", 2024-09-21T10:48:25.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014047300000000001 2024-09-21T10:48:25.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.506 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,7", 2024-09-21T10:48:25.506 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.456327+0000", 2024-09-21T10:48:25.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000110911 2024-09-21T10:48:25.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.506 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:48:25.506 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.466585+0000", 2024-09-21T10:48:25.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.010257142 2024-09-21T10:48:25.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.506 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.507 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.469381+0000", 2024-09-21T10:48:25.507 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0027964790000000002 2024-09-21T10:48:25.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.507 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.507 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.507 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.575533+0000", 2024-09-21T10:48:25.507 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.106151591 2024-09-21T10:48:25.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.507 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.507 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:48:25.508 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.575563+0000", 2024-09-21T10:48:25.508 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0617000000000002e-05 2024-09-21T10:48:25.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.508 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.509 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:25.509 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.575586+0000", 2024-09-21T10:48:25.509 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2679000000000001e-05 2024-09-21T10:48:25.509 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.509 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:25.509 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.509 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:199078 54.e 54:78d45066:::benchmark_data_smithi070_261699_object69965:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1397)", 2024-09-21T10:48:25.509 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:47:30.451075+0000", 2024-09-21T10:48:25.509 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 54.906748913999998, 2024-09-21T10:48:25.509 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.130861053, 2024-09-21T10:48:25.509 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:25.510 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:25.510 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:25.510 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:48:25.510 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:48:25.510 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 199078 2024-09-21T10:48:25.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.510 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:25.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.510 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:25.510 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.451075+0000", 2024-09-21T10:48:25.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.511 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:25.511 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.451075+0000", 2024-09-21T10:48:25.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.511 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:25.511 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.451074+0000", 2024-09-21T10:48:25.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:48:25.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.511 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:25.511 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.451078+0000", 2024-09-21T10:48:25.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4689999999999999e-06 2024-09-21T10:48:25.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:25.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.451087+0000", 2024-09-21T10:48:25.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1359999999999997e-06 2024-09-21T10:48:25.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:25.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.451093+0000", 2024-09-21T10:48:25.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9839999999999998e-06 2024-09-21T10:48:25.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:25.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.460351+0000", 2024-09-21T10:48:25.513 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0092588099999999993 2024-09-21T10:48:25.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.513 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:48:25.513 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.460453+0000", 2024-09-21T10:48:25.513 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000101852 2024-09-21T10:48:25.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.513 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,7", 2024-09-21T10:48:25.513 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.460555+0000", 2024-09-21T10:48:25.513 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000101912 2024-09-21T10:48:25.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.513 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:48:25.514 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.469391+0000", 2024-09-21T10:48:25.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0088353700000000004 2024-09-21T10:48:25.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.514 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.514 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.471356+0000", 2024-09-21T10:48:25.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001965404 2024-09-21T10:48:25.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.514 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.514 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.581842+0000", 2024-09-21T10:48:25.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.110485682 2024-09-21T10:48:25.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.515 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:48:25.515 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.581893+0000", 2024-09-21T10:48:25.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1548999999999997e-05 2024-09-21T10:48:25.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.515 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:25.515 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.581936+0000", 2024-09-21T10:48:25.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2663999999999997e-05 2024-09-21T10:48:25.515 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.515 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:25.515 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.516 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:199096 54.e 54:73e4d42a:::benchmark_data_smithi070_261699_object69983:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1397)", 2024-09-21T10:48:25.516 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:47:30.473712+0000", 2024-09-21T10:48:25.516 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 54.884111973000003, 2024-09-21T10:48:25.516 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.108357011, 2024-09-21T10:48:25.516 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:25.516 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:25.516 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:25.516 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:48:25.516 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:48:25.516 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 199096 2024-09-21T10:48:25.516 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.516 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:25.516 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.516 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:25.517 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.473712+0000", 2024-09-21T10:48:25.517 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.517 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:25.517 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.473712+0000", 2024-09-21T10:48:25.517 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:25.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.517 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:25.517 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.473710+0000", 2024-09-21T10:48:25.517 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:48:25.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:25.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.473718+0000", 2024-09-21T10:48:25.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6000000000000001e-06 2024-09-21T10:48:25.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:25.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.473731+0000", 2024-09-21T10:48:25.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2836999999999999e-05 2024-09-21T10:48:25.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:25.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.473736+0000", 2024-09-21T10:48:25.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6869999999999999e-06 2024-09-21T10:48:25.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.519 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:25.519 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.473768+0000", 2024-09-21T10:48:25.519 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1924000000000001e-05 2024-09-21T10:48:25.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.519 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:48:25.519 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.473861+0000", 2024-09-21T10:48:25.519 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2441000000000005e-05 2024-09-21T10:48:25.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.519 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,7", 2024-09-21T10:48:25.519 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.473941+0000", 2024-09-21T10:48:25.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0105000000000006e-05 2024-09-21T10:48:25.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:48:25.520 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.477110+0000", 2024-09-21T10:48:25.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0031688300000000001 2024-09-21T10:48:25.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.520 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.477275+0000", 2024-09-21T10:48:25.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000165761 2024-09-21T10:48:25.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:25.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.581981+0000", 2024-09-21T10:48:25.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.104705912 2024-09-21T10:48:25.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:48:25.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.582032+0000", 2024-09-21T10:48:25.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0862000000000002e-05 2024-09-21T10:48:25.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:25.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:25.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:25.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:47:30.582069+0000", 2024-09-21T10:48:25.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6538999999999997e-05 2024-09-21T10:48:25.521 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.521 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:25.522 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.522 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:25.522 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:25.522 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:48:25.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:25.523 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:25.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:25.624 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:25.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:25.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:25.757 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:25.797 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:25.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:25.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:25.847 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:48:25.848 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T10:48:25.859 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:25.899 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:25.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:25.934 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:25.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:26.035 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:26.061 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:26.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:26.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:26.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:25 smithi070 ceph-mon[107385]: pgmap v2490: 89 pgs: 89 active+clean; 725 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 900 op/s 2024-09-21T10:48:26.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:25 smithi070 ceph-mon[112139]: pgmap v2490: 89 pgs: 89 active+clean; 725 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 900 op/s 2024-09-21T10:48:26.163 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:26.195 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:26.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:26.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:26.297 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:26.327 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:26.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:26.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:26.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:25 smithi160 ceph-mon[101879]: pgmap v2490: 89 pgs: 89 active+clean; 725 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 900 op/s 2024-09-21T10:48:26.428 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:26.468 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:26.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:26.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:26.570 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:26.600 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:26.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:26.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:26.702 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:26.734 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:26.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:26.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:26.799 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:48:26.836 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:26.871 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:26.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:26.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:26.973 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:27.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:27.019 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:27.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:27.120 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:27.150 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:27.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:27.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:27.173 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:48:27.252 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:27.290 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:27.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:27.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:27.392 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:27.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:27.429 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:27.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:27.530 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:27.556 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:27.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:27.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:27.658 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:27.696 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:48:27.699 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:27.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:27.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:27.801 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:27.838 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:27.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:27.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:27.940 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:27.976 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:27.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:27.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:28.078 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:28.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:27 smithi160 ceph-mon[101879]: pgmap v2491: 89 pgs: 89 active+clean; 487 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 893 op/s 2024-09-21T10:48:28.112 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:28.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:28.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:28.126 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:27 smithi070 ceph-mon[107385]: pgmap v2491: 89 pgs: 89 active+clean; 487 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 893 op/s 2024-09-21T10:48:28.126 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:27 smithi070 ceph-mon[112139]: pgmap v2491: 89 pgs: 89 active+clean; 487 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 893 op/s 2024-09-21T10:48:28.163 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":2489,"stamp":"2024-09-21T10:48:22.812516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":798562301,"num_objects":12203,"num_object_clones":0,"num_object_copies":36609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12203,"num_whiteouts":0,"num_read":379,"num_read_kb":823,"num_write":246324,"num_write_kb":8265490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2962150,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121153,"ondisk_log_size":121153,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":5484852,"kb_used_data":2476656,"kb_used_omap":219,"kb_used_meta":3007652,"kb_avail":650731212,"statfs":{"total":671965249536,"available":666348761088,"internally_reserved":0,"allocated":2536095744,"data_stored":2504627687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":225222,"internal_metadata":3079835706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-588906496,"num_objects":-8986,"num_object_clones":0,"num_object_copies":-26958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8986,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002589"},"pg_stats":[{"pgid":"54.f","version":"1404'15461","reported_seq":15531,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:20.005621+0000","last_change":"2024-09-21T10:44:32.818675+0000","last_active":"2024-09-21T10:48:20.005621+0000","last_peered":"2024-09-21T10:48:20.005621+0000","last_clean":"2024-09-21T10:48:20.005621+0000","last_became_active":"2024-09-21T10:44:32.818171+0000","last_became_peered":"2024-09-21T10:44:32.818171+0000","last_unstale":"2024-09-21T10:48:20.005621+0000","last_undegraded":"2024-09-21T10:48:20.005621+0000","last_fullsized":"2024-09-21T10:48:20.005621+0000","mapping_epoch":1376,"log_start":"1390'8600","ondisk_log_start":"1390'8600","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":6861,"log_dups_size":0,"ondisk_log_size":6861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:59:35.398933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52625408,"num_objects":803,"num_object_clones":0,"num_object_copies":2409,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15461,"num_write_kb":520448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.e","version":"1404'15200","reported_seq":15238,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:21.078919+0000","last_change":"2024-09-21T10:44:32.821023+0000","last_active":"2024-09-21T10:48:21.078919+0000","last_peered":"2024-09-21T10:48:21.078919+0000","last_clean":"2024-09-21T10:48:21.078919+0000","last_became_active":"2024-09-21T10:44:32.820651+0000","last_became_peered":"2024-09-21T10:44:32.820651+0000","last_unstale":"2024-09-21T10:48:21.078919+0000","last_undegraded":"2024-09-21T10:48:21.078919+0000","last_fullsized":"2024-09-21T10:48:21.078919+0000","mapping_epoch":1376,"log_start":"1388'7300","ondisk_log_start":"1388'7300","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":7900,"log_dups_size":0,"ondisk_log_size":7900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:39:13.493921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49152000,"num_objects":750,"num_object_clones":0,"num_object_copies":2250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15200,"num_write_kb":510400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"54.d","version":"1404'15473","reported_seq":15512,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:21.441281+0000","last_change":"2024-09-21T10:44:32.816573+0000","last_active":"2024-09-21T10:48:21.441281+0000","last_peered":"2024-09-21T10:48:21.441281+0000","last_clean":"2024-09-21T10:48:21.441281+0000","last_became_active":"2024-09-21T10:44:32.816260+0000","last_became_peered":"2024-09-21T10:44:32.816260+0000","last_unstale":"2024-09-21T10:48:21.441281+0000","last_undegraded":"2024-09-21T10:48:21.441281+0000","last_fullsized":"2024-09-21T10:48:21.441281+0000","mapping_epoch":1376,"log_start":"1389'8100","ondisk_log_start":"1389'8100","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":7373,"log_dups_size":0,"ondisk_log_size":7373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:20:47.764042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48234520,"num_objects":737,"num_object_clones":0,"num_object_copies":2211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":737,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15473,"num_write_kb":518657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"54.c","version":"1404'14820","reported_seq":14890,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:20.086777+0000","last_change":"2024-09-21T10:44:32.819873+0000","last_active":"2024-09-21T10:48:20.086777+0000","last_peered":"2024-09-21T10:48:20.086777+0000","last_clean":"2024-09-21T10:48:20.086777+0000","last_became_active":"2024-09-21T10:44:32.819256+0000","last_became_peered":"2024-09-21T10:44:32.819256+0000","last_unstale":"2024-09-21T10:48:20.086777+0000","last_undegraded":"2024-09-21T10:48:20.086777+0000","last_fullsized":"2024-09-21T10:48:20.086777+0000","mapping_epoch":1376,"log_start":"1389'8000","ondisk_log_start":"1389'8000","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":6820,"log_dups_size":0,"ondisk_log_size":6820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:26:37.931891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":832,"num_object_clones":0,"num_object_copies":2496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14820,"num_write_kb":500864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.b","version":"1404'15130","reported_seq":15199,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:20.105323+0000","last_change":"2024-09-21T10:44:32.819321+0000","last_active":"2024-09-21T10:48:20.105323+0000","last_peered":"2024-09-21T10:48:20.105323+0000","last_clean":"2024-09-21T10:48:20.105323+0000","last_became_active":"2024-09-21T10:44:32.819084+0000","last_became_peered":"2024-09-21T10:44:32.819084+0000","last_unstale":"2024-09-21T10:48:20.105323+0000","last_undegraded":"2024-09-21T10:48:20.105323+0000","last_fullsized":"2024-09-21T10:48:20.105323+0000","mapping_epoch":1376,"log_start":"1389'8300","ondisk_log_start":"1389'8300","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":6830,"log_dups_size":0,"ondisk_log_size":6830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:58:43.441802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52035584,"num_objects":794,"num_object_clones":0,"num_object_copies":2382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15130,"num_write_kb":509568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.a","version":"1404'15498","reported_seq":15535,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:21.433376+0000","last_change":"2024-09-21T10:44:32.818041+0000","last_active":"2024-09-21T10:48:21.433376+0000","last_peered":"2024-09-21T10:48:21.433376+0000","last_clean":"2024-09-21T10:48:21.433376+0000","last_became_active":"2024-09-21T10:44:32.817725+0000","last_became_peered":"2024-09-21T10:44:32.817725+0000","last_unstale":"2024-09-21T10:48:21.433376+0000","last_undegraded":"2024-09-21T10:48:21.433376+0000","last_fullsized":"2024-09-21T10:48:21.433376+0000","mapping_epoch":1376,"log_start":"1389'8100","ondisk_log_start":"1389'8100","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":7398,"log_dups_size":0,"ondisk_log_size":7398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:22:24.084717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47841280,"num_objects":730,"num_object_clones":0,"num_object_copies":2190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15498,"num_write_kb":519296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"54.9","version":"1404'15311","reported_seq":15381,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.684011+0000","last_change":"2024-09-21T10:44:32.816734+0000","last_active":"2024-09-21T10:48:19.684011+0000","last_peered":"2024-09-21T10:48:19.684011+0000","last_clean":"2024-09-21T10:48:19.684011+0000","last_became_active":"2024-09-21T10:44:32.816261+0000","last_became_peered":"2024-09-21T10:44:32.816261+0000","last_unstale":"2024-09-21T10:48:19.684011+0000","last_undegraded":"2024-09-21T10:48:19.684011+0000","last_fullsized":"2024-09-21T10:48:19.684011+0000","mapping_epoch":1376,"log_start":"1389'7800","ondisk_log_start":"1389'7800","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":7511,"log_dups_size":0,"ondisk_log_size":7511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:38.615950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52494336,"num_objects":801,"num_object_clones":0,"num_object_copies":2403,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15311,"num_write_kb":515584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"54.8","version":"1404'15831","reported_seq":15869,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:21.021573+0000","last_change":"2024-09-21T10:44:32.818221+0000","last_active":"2024-09-21T10:48:21.021573+0000","last_peered":"2024-09-21T10:48:21.021573+0000","last_clean":"2024-09-21T10:48:21.021573+0000","last_became_active":"2024-09-21T10:44:32.817941+0000","last_became_peered":"2024-09-21T10:44:32.817941+0000","last_unstale":"2024-09-21T10:48:21.021573+0000","last_undegraded":"2024-09-21T10:48:21.021573+0000","last_fullsized":"2024-09-21T10:48:21.021573+0000","mapping_epoch":1376,"log_start":"1389'7900","ondisk_log_start":"1389'7900","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":7931,"log_dups_size":0,"ondisk_log_size":7931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:11:40.024092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49348608,"num_objects":753,"num_object_clones":0,"num_object_copies":2259,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15831,"num_write_kb":530688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"54.7","version":"1404'15597","reported_seq":15667,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:20.079670+0000","last_change":"2024-09-21T10:44:32.818432+0000","last_active":"2024-09-21T10:48:20.079670+0000","last_peered":"2024-09-21T10:48:20.079670+0000","last_clean":"2024-09-21T10:48:20.079670+0000","last_became_active":"2024-09-21T10:44:32.818094+0000","last_became_peered":"2024-09-21T10:44:32.818094+0000","last_unstale":"2024-09-21T10:48:20.079670+0000","last_undegraded":"2024-09-21T10:48:20.079670+0000","last_fullsized":"2024-09-21T10:48:20.079670+0000","mapping_epoch":1376,"log_start":"1390'8700","ondisk_log_start":"1390'8700","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":6897,"log_dups_size":0,"ondisk_log_size":6897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:55:13.023948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53936128,"num_objects":823,"num_object_clones":0,"num_object_copies":2469,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15597,"num_write_kb":525440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.6","version":"1404'15410","reported_seq":15448,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:21.331941+0000","last_change":"2024-09-21T10:44:32.822589+0000","last_active":"2024-09-21T10:48:21.331941+0000","last_peered":"2024-09-21T10:48:21.331941+0000","last_clean":"2024-09-21T10:48:21.331941+0000","last_became_active":"2024-09-21T10:44:32.822265+0000","last_became_peered":"2024-09-21T10:44:32.822265+0000","last_unstale":"2024-09-21T10:48:21.331941+0000","last_undegraded":"2024-09-21T10:48:21.331941+0000","last_fullsized":"2024-09-21T10:48:21.331941+0000","mapping_epoch":1376,"log_start":"1388'7500","ondisk_log_start":"1388'7500","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":7910,"log_dups_size":0,"ondisk_log_size":7910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:40:44.356501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47448064,"num_objects":724,"num_object_clones":0,"num_object_copies":2172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15410,"num_write_kb":516288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"54.5","version":"1404'15694","reported_seq":15732,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:21.395762+0000","last_change":"2024-09-21T10:44:32.815686+0000","last_active":"2024-09-21T10:48:21.395762+0000","last_peered":"2024-09-21T10:48:21.395762+0000","last_clean":"2024-09-21T10:48:21.395762+0000","last_became_active":"2024-09-21T10:44:32.815266+0000","last_became_peered":"2024-09-21T10:44:32.815266+0000","last_unstale":"2024-09-21T10:48:21.395762+0000","last_undegraded":"2024-09-21T10:48:21.395762+0000","last_fullsized":"2024-09-21T10:48:21.395762+0000","mapping_epoch":1376,"log_start":"1389'8300","ondisk_log_start":"1389'8300","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":7394,"log_dups_size":0,"ondisk_log_size":7394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:11:59.857675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47054848,"num_objects":718,"num_object_clones":0,"num_object_copies":2154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15694,"num_write_kb":525184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"54.4","version":"1404'15413","reported_seq":15482,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:20.093009+0000","last_change":"2024-09-21T10:44:32.821087+0000","last_active":"2024-09-21T10:48:20.093009+0000","last_peered":"2024-09-21T10:48:20.093009+0000","last_clean":"2024-09-21T10:48:20.093009+0000","last_became_active":"2024-09-21T10:44:32.820712+0000","last_became_peered":"2024-09-21T10:44:32.820712+0000","last_unstale":"2024-09-21T10:48:20.093009+0000","last_undegraded":"2024-09-21T10:48:20.093009+0000","last_fullsized":"2024-09-21T10:48:20.093009+0000","mapping_epoch":1376,"log_start":"1390'8500","ondisk_log_start":"1390'8500","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":6913,"log_dups_size":0,"ondisk_log_size":6913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:51:13.302857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48693248,"num_objects":743,"num_object_clones":0,"num_object_copies":2229,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15413,"num_write_kb":516992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.3","version":"1404'15293","reported_seq":15363,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.643854+0000","last_change":"2024-09-21T10:44:32.817547+0000","last_active":"2024-09-21T10:48:19.643854+0000","last_peered":"2024-09-21T10:48:19.643854+0000","last_clean":"2024-09-21T10:48:19.643854+0000","last_became_active":"2024-09-21T10:44:32.817238+0000","last_became_peered":"2024-09-21T10:44:32.817238+0000","last_unstale":"2024-09-21T10:48:19.643854+0000","last_undegraded":"2024-09-21T10:48:19.643854+0000","last_fullsized":"2024-09-21T10:48:19.643854+0000","mapping_epoch":1376,"log_start":"1389'7700","ondisk_log_start":"1389'7700","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":7593,"log_dups_size":0,"ondisk_log_size":7593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:02:33.135641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54460416,"num_objects":831,"num_object_clones":0,"num_object_copies":2493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15293,"num_write_kb":515968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"54.2","version":"1404'15134","reported_seq":15204,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.999964+0000","last_change":"2024-09-21T10:44:32.820419+0000","last_active":"2024-09-21T10:48:19.999964+0000","last_peered":"2024-09-21T10:48:19.999964+0000","last_clean":"2024-09-21T10:48:19.999964+0000","last_became_active":"2024-09-21T10:44:32.820081+0000","last_became_peered":"2024-09-21T10:44:32.820081+0000","last_unstale":"2024-09-21T10:48:19.999964+0000","last_undegraded":"2024-09-21T10:48:19.999964+0000","last_fullsized":"2024-09-21T10:48:19.999964+0000","mapping_epoch":1376,"log_start":"1387'7000","ondisk_log_start":"1387'7000","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":8134,"log_dups_size":0,"ondisk_log_size":8134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:30:15.785931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51249152,"num_objects":782,"num_object_clones":0,"num_object_copies":2346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15134,"num_write_kb":509312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"54.1","version":"1404'15500","reported_seq":15538,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:21.521938+0000","last_change":"2024-09-21T10:44:32.819407+0000","last_active":"2024-09-21T10:48:21.521938+0000","last_peered":"2024-09-21T10:48:21.521938+0000","last_clean":"2024-09-21T10:48:21.521938+0000","last_became_active":"2024-09-21T10:44:32.819056+0000","last_became_peered":"2024-09-21T10:44:32.819056+0000","last_unstale":"2024-09-21T10:48:21.521938+0000","last_undegraded":"2024-09-21T10:48:21.521938+0000","last_fullsized":"2024-09-21T10:48:21.521938+0000","mapping_epoch":1376,"log_start":"1383'5400","ondisk_log_start":"1383'5400","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:39:58.641282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44302336,"num_objects":676,"num_object_clones":0,"num_object_copies":2028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15500,"num_write_kb":517632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"54.0","version":"1404'15280","reported_seq":15318,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:21.449416+0000","last_change":"2024-09-21T10:44:32.816008+0000","last_active":"2024-09-21T10:48:21.449416+0000","last_peered":"2024-09-21T10:48:21.449416+0000","last_clean":"2024-09-21T10:48:21.449416+0000","last_became_active":"2024-09-21T10:44:32.815627+0000","last_became_peered":"2024-09-21T10:44:32.815627+0000","last_unstale":"2024-09-21T10:48:21.449416+0000","last_undegraded":"2024-09-21T10:48:21.449416+0000","last_fullsized":"2024-09-21T10:48:21.449416+0000","mapping_epoch":1376,"log_start":"1389'7900","ondisk_log_start":"1389'7900","created":1376,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:44:31.804647+0000","last_clean_scrub_stamp":"2024-09-21T10:44:31.804647+0000","objects_scrubbed":0,"log_size":7380,"log_dups_size":0,"ondisk_log_size":7380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:48:54.596883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44564480,"num_objects":680,"num_object_clones":0,"num_object_copies":2040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15280,"num_write_kb":510720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":2121,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.594718+0000","last_change":"2024-09-21T10:40:13.438023+0000","last_active":"2024-09-21T10:48:19.594718+0000","last_peered":"2024-09-21T10:48:19.594718+0000","last_clean":"2024-09-21T10:48:19.594718+0000","last_became_active":"2024-09-21T10:40:13.431103+0000","last_became_peered":"2024-09-21T10:40:13.431103+0000","last_unstale":"2024-09-21T10:48:19.594718+0000","last_undegraded":"2024-09-21T10:48:19.594718+0000","last_fullsized":"2024-09-21T10:48:19.594718+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:00:24.887661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1374,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.534832+0000","last_change":"2024-09-21T10:07:55.999392+0000","last_active":"2024-09-21T10:48:06.534832+0000","last_peered":"2024-09-21T10:48:06.534832+0000","last_clean":"2024-09-21T10:48:06.534832+0000","last_became_active":"2024-09-21T10:07:55.999103+0000","last_became_peered":"2024-09-21T10:07:55.999103+0000","last_unstale":"2024-09-21T10:48:06.534832+0000","last_undegraded":"2024-09-21T10:48:06.534832+0000","last_fullsized":"2024-09-21T10:48:06.534832+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:48.757242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":2102,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.580018+0000","last_change":"2024-09-21T10:40:13.432818+0000","last_active":"2024-09-21T10:48:19.580018+0000","last_peered":"2024-09-21T10:48:19.580018+0000","last_clean":"2024-09-21T10:48:19.580018+0000","last_became_active":"2024-09-21T10:40:13.432448+0000","last_became_peered":"2024-09-21T10:40:13.432448+0000","last_unstale":"2024-09-21T10:48:19.580018+0000","last_undegraded":"2024-09-21T10:48:19.580018+0000","last_fullsized":"2024-09-21T10:48:19.580018+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:20.448242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2119,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.579786+0000","last_change":"2024-09-21T10:40:13.431824+0000","last_active":"2024-09-21T10:48:19.579786+0000","last_peered":"2024-09-21T10:48:19.579786+0000","last_clean":"2024-09-21T10:48:19.579786+0000","last_became_active":"2024-09-21T10:40:13.431516+0000","last_became_peered":"2024-09-21T10:40:13.431516+0000","last_unstale":"2024-09-21T10:48:19.579786+0000","last_undegraded":"2024-09-21T10:48:19.579786+0000","last_fullsized":"2024-09-21T10:48:19.579786+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:30:40.226452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":1361,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.535541+0000","last_change":"2024-09-21T09:54:53.634508+0000","last_active":"2024-09-21T10:48:06.535541+0000","last_peered":"2024-09-21T10:48:06.535541+0000","last_clean":"2024-09-21T10:48:06.535541+0000","last_became_active":"2024-09-21T09:54:53.634254+0000","last_became_peered":"2024-09-21T09:54:53.634254+0000","last_unstale":"2024-09-21T10:48:06.535541+0000","last_undegraded":"2024-09-21T10:48:06.535541+0000","last_fullsized":"2024-09-21T10:48:06.535541+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:18.122107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1388,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.534309+0000","last_change":"2024-09-21T10:09:06.574933+0000","last_active":"2024-09-21T10:48:06.534309+0000","last_peered":"2024-09-21T10:48:06.534309+0000","last_clean":"2024-09-21T10:48:06.534309+0000","last_became_active":"2024-09-21T10:09:06.574467+0000","last_became_peered":"2024-09-21T10:09:06.574467+0000","last_unstale":"2024-09-21T10:48:06.534309+0000","last_undegraded":"2024-09-21T10:48:06.534309+0000","last_fullsized":"2024-09-21T10:48:06.534309+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:06.142269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":1376,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.537294+0000","last_change":"2024-09-21T10:07:55.991189+0000","last_active":"2024-09-21T10:48:06.537294+0000","last_peered":"2024-09-21T10:48:06.537294+0000","last_clean":"2024-09-21T10:48:06.537294+0000","last_became_active":"2024-09-21T10:07:55.990784+0000","last_became_peered":"2024-09-21T10:07:55.990784+0000","last_unstale":"2024-09-21T10:48:06.537294+0000","last_undegraded":"2024-09-21T10:48:06.537294+0000","last_fullsized":"2024-09-21T10:48:06.537294+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:00.580471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1388,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.534600+0000","last_change":"2024-09-21T10:09:06.560175+0000","last_active":"2024-09-21T10:48:06.534600+0000","last_peered":"2024-09-21T10:48:06.534600+0000","last_clean":"2024-09-21T10:48:06.534600+0000","last_became_active":"2024-09-21T10:09:06.558584+0000","last_became_peered":"2024-09-21T10:09:06.558584+0000","last_unstale":"2024-09-21T10:48:06.534600+0000","last_undegraded":"2024-09-21T10:48:06.534600+0000","last_fullsized":"2024-09-21T10:48:06.534600+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:02.918339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":2104,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.593216+0000","last_change":"2024-09-21T10:44:14.409321+0000","last_active":"2024-09-21T10:48:19.593216+0000","last_peered":"2024-09-21T10:48:19.593216+0000","last_clean":"2024-09-21T10:48:19.593216+0000","last_became_active":"2024-09-21T10:44:14.408833+0000","last_became_peered":"2024-09-21T10:44:14.408833+0000","last_unstale":"2024-09-21T10:48:19.593216+0000","last_undegraded":"2024-09-21T10:48:19.593216+0000","last_fullsized":"2024-09-21T10:48:19.593216+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:10:00.997210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":881,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.582729+0000","last_change":"2024-09-21T10:40:13.436927+0000","last_active":"2024-09-21T10:48:19.582729+0000","last_peered":"2024-09-21T10:48:19.582729+0000","last_clean":"2024-09-21T10:48:19.582729+0000","last_became_active":"2024-09-21T10:40:13.436329+0000","last_became_peered":"2024-09-21T10:40:13.436329+0000","last_unstale":"2024-09-21T10:48:19.582729+0000","last_undegraded":"2024-09-21T10:48:19.582729+0000","last_fullsized":"2024-09-21T10:48:19.582729+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:05.668898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":1399,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.534498+0000","last_change":"2024-09-21T10:40:13.433278+0000","last_active":"2024-09-21T10:48:06.534498+0000","last_peered":"2024-09-21T10:48:06.534498+0000","last_clean":"2024-09-21T10:48:06.534498+0000","last_became_active":"2024-09-21T10:40:13.432814+0000","last_became_peered":"2024-09-21T10:40:13.432814+0000","last_unstale":"2024-09-21T10:48:06.534498+0000","last_undegraded":"2024-09-21T10:48:06.534498+0000","last_fullsized":"2024-09-21T10:48:06.534498+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:55.909872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2112,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.582025+0000","last_change":"2024-09-21T10:40:13.432531+0000","last_active":"2024-09-21T10:48:19.582025+0000","last_peered":"2024-09-21T10:48:19.582025+0000","last_clean":"2024-09-21T10:48:19.582025+0000","last_became_active":"2024-09-21T10:40:13.432210+0000","last_became_peered":"2024-09-21T10:40:13.432210+0000","last_unstale":"2024-09-21T10:48:19.582025+0000","last_undegraded":"2024-09-21T10:48:19.582025+0000","last_fullsized":"2024-09-21T10:48:19.582025+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:20:10.998607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2084,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.586315+0000","last_change":"2024-09-21T10:09:06.530394+0000","last_active":"2024-09-21T10:48:19.586315+0000","last_peered":"2024-09-21T10:48:19.586315+0000","last_clean":"2024-09-21T10:48:19.586315+0000","last_became_active":"2024-09-21T10:09:06.529307+0000","last_became_peered":"2024-09-21T10:09:06.529307+0000","last_unstale":"2024-09-21T10:48:19.586315+0000","last_undegraded":"2024-09-21T10:48:19.586315+0000","last_fullsized":"2024-09-21T10:48:19.586315+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:54.287397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2117,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.593028+0000","last_change":"2024-09-21T10:44:14.409422+0000","last_active":"2024-09-21T10:48:19.593028+0000","last_peered":"2024-09-21T10:48:19.593028+0000","last_clean":"2024-09-21T10:48:19.593028+0000","last_became_active":"2024-09-21T10:44:14.408726+0000","last_became_peered":"2024-09-21T10:44:14.408726+0000","last_unstale":"2024-09-21T10:48:19.593028+0000","last_undegraded":"2024-09-21T10:48:19.593028+0000","last_fullsized":"2024-09-21T10:48:19.593028+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:44:21.743112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2099,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.580222+0000","last_change":"2024-09-21T10:09:06.552324+0000","last_active":"2024-09-21T10:48:19.580222+0000","last_peered":"2024-09-21T10:48:19.580222+0000","last_clean":"2024-09-21T10:48:19.580222+0000","last_became_active":"2024-09-21T10:09:06.540545+0000","last_became_peered":"2024-09-21T10:09:06.540545+0000","last_unstale":"2024-09-21T10:48:19.580222+0000","last_undegraded":"2024-09-21T10:48:19.580222+0000","last_fullsized":"2024-09-21T10:48:19.580222+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:08.506149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1378,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.529910+0000","last_change":"2024-09-21T10:07:56.003005+0000","last_active":"2024-09-21T10:48:06.529910+0000","last_peered":"2024-09-21T10:48:06.529910+0000","last_clean":"2024-09-21T10:48:06.529910+0000","last_became_active":"2024-09-21T10:07:56.001550+0000","last_became_peered":"2024-09-21T10:07:56.001550+0000","last_unstale":"2024-09-21T10:48:06.529910+0000","last_undegraded":"2024-09-21T10:48:06.529910+0000","last_fullsized":"2024-09-21T10:48:06.529910+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:54.553261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2091,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.580319+0000","last_change":"2024-09-21T10:09:06.559017+0000","last_active":"2024-09-21T10:48:19.580319+0000","last_peered":"2024-09-21T10:48:19.580319+0000","last_clean":"2024-09-21T10:48:19.580319+0000","last_became_active":"2024-09-21T10:09:06.555256+0000","last_became_peered":"2024-09-21T10:09:06.555256+0000","last_unstale":"2024-09-21T10:48:19.580319+0000","last_undegraded":"2024-09-21T10:48:19.580319+0000","last_fullsized":"2024-09-21T10:48:19.580319+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:33:45.145138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2096,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.594816+0000","last_change":"2024-09-21T10:09:06.562504+0000","last_active":"2024-09-21T10:48:19.594816+0000","last_peered":"2024-09-21T10:48:19.594816+0000","last_clean":"2024-09-21T10:48:19.594816+0000","last_became_active":"2024-09-21T10:09:06.561747+0000","last_became_peered":"2024-09-21T10:09:06.561747+0000","last_unstale":"2024-09-21T10:48:19.594816+0000","last_undegraded":"2024-09-21T10:48:19.594816+0000","last_fullsized":"2024-09-21T10:48:19.594816+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:03:58.012406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2096,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.580000+0000","last_change":"2024-09-21T10:44:14.399948+0000","last_active":"2024-09-21T10:48:19.580000+0000","last_peered":"2024-09-21T10:48:19.580000+0000","last_clean":"2024-09-21T10:48:19.580000+0000","last_became_active":"2024-09-21T10:44:14.399337+0000","last_became_peered":"2024-09-21T10:44:14.399337+0000","last_unstale":"2024-09-21T10:48:19.580000+0000","last_undegraded":"2024-09-21T10:48:19.580000+0000","last_fullsized":"2024-09-21T10:48:19.580000+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:40.185020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1393,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.544162+0000","last_change":"2024-09-21T10:44:14.401716+0000","last_active":"2024-09-21T10:48:06.544162+0000","last_peered":"2024-09-21T10:48:06.544162+0000","last_clean":"2024-09-21T10:48:06.544162+0000","last_became_active":"2024-09-21T10:44:14.401254+0000","last_became_peered":"2024-09-21T10:44:14.401254+0000","last_unstale":"2024-09-21T10:48:06.544162+0000","last_undegraded":"2024-09-21T10:48:06.544162+0000","last_fullsized":"2024-09-21T10:48:06.544162+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:07:21.536272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":52,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.532294+0000","last_change":"2024-09-21T10:44:14.424249+0000","last_active":"2024-09-21T10:48:06.532294+0000","last_peered":"2024-09-21T10:48:06.532294+0000","last_clean":"2024-09-21T10:48:06.532294+0000","last_became_active":"2024-09-21T10:44:14.408884+0000","last_became_peered":"2024-09-21T10:44:14.408884+0000","last_unstale":"2024-09-21T10:48:06.532294+0000","last_undegraded":"2024-09-21T10:48:06.532294+0000","last_fullsized":"2024-09-21T10:48:06.532294+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:18:44.759435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2091,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.582413+0000","last_change":"2024-09-21T10:40:13.437741+0000","last_active":"2024-09-21T10:48:19.582413+0000","last_peered":"2024-09-21T10:48:19.582413+0000","last_clean":"2024-09-21T10:48:19.582413+0000","last_became_active":"2024-09-21T10:40:13.437447+0000","last_became_peered":"2024-09-21T10:40:13.437447+0000","last_unstale":"2024-09-21T10:48:19.582413+0000","last_undegraded":"2024-09-21T10:48:19.582413+0000","last_fullsized":"2024-09-21T10:48:19.582413+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:40:47.072907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":1421,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.534813+0000","last_change":"2024-09-21T10:44:14.415264+0000","last_active":"2024-09-21T10:48:06.534813+0000","last_peered":"2024-09-21T10:48:06.534813+0000","last_clean":"2024-09-21T10:48:06.534813+0000","last_became_active":"2024-09-21T10:44:14.407813+0000","last_became_peered":"2024-09-21T10:44:14.407813+0000","last_unstale":"2024-09-21T10:48:06.534813+0000","last_undegraded":"2024-09-21T10:48:06.534813+0000","last_fullsized":"2024-09-21T10:48:06.534813+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:55.933601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1374,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.534671+0000","last_change":"2024-09-21T10:07:56.001756+0000","last_active":"2024-09-21T10:48:06.534671+0000","last_peered":"2024-09-21T10:48:06.534671+0000","last_clean":"2024-09-21T10:48:06.534671+0000","last_became_active":"2024-09-21T10:07:56.000789+0000","last_became_peered":"2024-09-21T10:07:56.000789+0000","last_unstale":"2024-09-21T10:48:06.534671+0000","last_undegraded":"2024-09-21T10:48:06.534671+0000","last_fullsized":"2024-09-21T10:48:06.534671+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:56:04.302902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1362,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.536906+0000","last_change":"2024-09-21T09:54:53.636902+0000","last_active":"2024-09-21T10:48:06.536906+0000","last_peered":"2024-09-21T10:48:06.536906+0000","last_clean":"2024-09-21T10:48:06.536906+0000","last_became_active":"2024-09-21T09:54:53.636616+0000","last_became_peered":"2024-09-21T09:54:53.636616+0000","last_unstale":"2024-09-21T10:48:06.536906+0000","last_undegraded":"2024-09-21T10:48:06.536906+0000","last_fullsized":"2024-09-21T10:48:06.536906+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:46:47.694495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1418,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.529951+0000","last_change":"2024-09-21T10:44:14.402059+0000","last_active":"2024-09-21T10:48:06.529951+0000","last_peered":"2024-09-21T10:48:06.529951+0000","last_clean":"2024-09-21T10:48:06.529951+0000","last_became_active":"2024-09-21T10:44:14.401710+0000","last_became_peered":"2024-09-21T10:44:14.401710+0000","last_unstale":"2024-09-21T10:48:06.529951+0000","last_undegraded":"2024-09-21T10:48:06.529951+0000","last_fullsized":"2024-09-21T10:48:06.529951+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:17.314693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":2104,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.592940+0000","last_change":"2024-09-21T10:07:56.023462+0000","last_active":"2024-09-21T10:48:19.592940+0000","last_peered":"2024-09-21T10:48:19.592940+0000","last_clean":"2024-09-21T10:48:19.592940+0000","last_became_active":"2024-09-21T10:07:56.021859+0000","last_became_peered":"2024-09-21T10:07:56.021859+0000","last_unstale":"2024-09-21T10:48:19.592940+0000","last_undegraded":"2024-09-21T10:48:19.592940+0000","last_fullsized":"2024-09-21T10:48:19.592940+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:42:03.134729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2093,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.580282+0000","last_change":"2024-09-21T10:44:14.399028+0000","last_active":"2024-09-21T10:48:19.580282+0000","last_peered":"2024-09-21T10:48:19.580282+0000","last_clean":"2024-09-21T10:48:19.580282+0000","last_became_active":"2024-09-21T10:44:14.398115+0000","last_became_peered":"2024-09-21T10:44:14.398115+0000","last_unstale":"2024-09-21T10:48:19.580282+0000","last_undegraded":"2024-09-21T10:48:19.580282+0000","last_fullsized":"2024-09-21T10:48:19.580282+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:05.170376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2085,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.583253+0000","last_change":"2024-09-21T10:07:56.006236+0000","last_active":"2024-09-21T10:48:19.583253+0000","last_peered":"2024-09-21T10:48:19.583253+0000","last_clean":"2024-09-21T10:48:19.583253+0000","last_became_active":"2024-09-21T10:07:56.004439+0000","last_became_peered":"2024-09-21T10:07:56.004439+0000","last_unstale":"2024-09-21T10:48:19.583253+0000","last_undegraded":"2024-09-21T10:48:19.583253+0000","last_fullsized":"2024-09-21T10:48:19.583253+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:39.422467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":2102,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.580870+0000","last_change":"2024-09-21T10:44:14.410440+0000","last_active":"2024-09-21T10:48:19.580870+0000","last_peered":"2024-09-21T10:48:19.580870+0000","last_clean":"2024-09-21T10:48:19.580870+0000","last_became_active":"2024-09-21T10:44:14.400035+0000","last_became_peered":"2024-09-21T10:44:14.400035+0000","last_unstale":"2024-09-21T10:48:19.580870+0000","last_undegraded":"2024-09-21T10:48:19.580870+0000","last_fullsized":"2024-09-21T10:48:19.580870+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:49:39.536516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2105,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.593353+0000","last_change":"2024-09-21T10:44:14.409410+0000","last_active":"2024-09-21T10:48:19.593353+0000","last_peered":"2024-09-21T10:48:19.593353+0000","last_clean":"2024-09-21T10:48:19.593353+0000","last_became_active":"2024-09-21T10:44:14.409025+0000","last_became_peered":"2024-09-21T10:44:14.409025+0000","last_unstale":"2024-09-21T10:48:19.593353+0000","last_undegraded":"2024-09-21T10:48:19.593353+0000","last_fullsized":"2024-09-21T10:48:19.593353+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:25:09.197455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":46,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.532206+0000","last_change":"2024-09-21T10:44:14.405677+0000","last_active":"2024-09-21T10:48:06.532206+0000","last_peered":"2024-09-21T10:48:06.532206+0000","last_clean":"2024-09-21T10:48:06.532206+0000","last_became_active":"2024-09-21T10:44:14.405262+0000","last_became_peered":"2024-09-21T10:44:14.405262+0000","last_unstale":"2024-09-21T10:48:06.532206+0000","last_undegraded":"2024-09-21T10:48:06.532206+0000","last_fullsized":"2024-09-21T10:48:06.532206+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:53:12.127245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2121,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.582645+0000","last_change":"2024-09-21T10:40:13.437609+0000","last_active":"2024-09-21T10:48:19.582645+0000","last_peered":"2024-09-21T10:48:19.582645+0000","last_clean":"2024-09-21T10:48:19.582645+0000","last_became_active":"2024-09-21T10:40:13.437296+0000","last_became_peered":"2024-09-21T10:40:13.437296+0000","last_unstale":"2024-09-21T10:48:19.582645+0000","last_undegraded":"2024-09-21T10:48:19.582645+0000","last_fullsized":"2024-09-21T10:48:19.582645+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:41:58.963386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2076,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.582330+0000","last_change":"2024-09-21T10:07:56.067772+0000","last_active":"2024-09-21T10:48:19.582330+0000","last_peered":"2024-09-21T10:48:19.582330+0000","last_clean":"2024-09-21T10:48:19.582330+0000","last_became_active":"2024-09-21T10:07:56.052602+0000","last_became_peered":"2024-09-21T10:07:56.052602+0000","last_unstale":"2024-09-21T10:48:19.582330+0000","last_undegraded":"2024-09-21T10:48:19.582330+0000","last_fullsized":"2024-09-21T10:48:19.582330+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:07:59.899214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2144,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.582490+0000","last_change":"2024-09-21T10:44:14.405182+0000","last_active":"2024-09-21T10:48:19.582490+0000","last_peered":"2024-09-21T10:48:19.582490+0000","last_clean":"2024-09-21T10:48:19.582490+0000","last_became_active":"2024-09-21T10:44:14.404852+0000","last_became_peered":"2024-09-21T10:44:14.404852+0000","last_unstale":"2024-09-21T10:48:19.582490+0000","last_undegraded":"2024-09-21T10:48:19.582490+0000","last_fullsized":"2024-09-21T10:48:19.582490+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:51.687206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1402,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.535672+0000","last_change":"2024-09-21T10:40:13.435375+0000","last_active":"2024-09-21T10:48:06.535672+0000","last_peered":"2024-09-21T10:48:06.535672+0000","last_clean":"2024-09-21T10:48:06.535672+0000","last_became_active":"2024-09-21T10:40:13.435050+0000","last_became_peered":"2024-09-21T10:40:13.435050+0000","last_unstale":"2024-09-21T10:48:06.535672+0000","last_undegraded":"2024-09-21T10:48:06.535672+0000","last_fullsized":"2024-09-21T10:48:06.535672+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:57.889610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2131,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.582802+0000","last_change":"2024-09-21T10:40:13.434189+0000","last_active":"2024-09-21T10:48:19.582802+0000","last_peered":"2024-09-21T10:48:19.582802+0000","last_clean":"2024-09-21T10:48:19.582802+0000","last_became_active":"2024-09-21T10:40:13.433892+0000","last_became_peered":"2024-09-21T10:40:13.433892+0000","last_unstale":"2024-09-21T10:48:19.582802+0000","last_undegraded":"2024-09-21T10:48:19.582802+0000","last_fullsized":"2024-09-21T10:48:19.582802+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:49:17.296220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1378,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.534755+0000","last_change":"2024-09-21T10:07:21.524082+0000","last_active":"2024-09-21T10:48:06.534755+0000","last_peered":"2024-09-21T10:48:06.534755+0000","last_clean":"2024-09-21T10:48:06.534755+0000","last_became_active":"2024-09-21T10:07:21.523779+0000","last_became_peered":"2024-09-21T10:07:21.523779+0000","last_unstale":"2024-09-21T10:48:06.534755+0000","last_undegraded":"2024-09-21T10:48:06.534755+0000","last_fullsized":"2024-09-21T10:48:06.534755+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:13.813832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":1393,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.539600+0000","last_change":"2024-09-21T10:07:56.024239+0000","last_active":"2024-09-21T10:48:06.539600+0000","last_peered":"2024-09-21T10:48:06.539600+0000","last_clean":"2024-09-21T10:48:06.539600+0000","last_became_active":"2024-09-21T10:07:56.023914+0000","last_became_peered":"2024-09-21T10:07:56.023914+0000","last_unstale":"2024-09-21T10:48:06.539600+0000","last_undegraded":"2024-09-21T10:48:06.539600+0000","last_fullsized":"2024-09-21T10:48:06.539600+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:23.361736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084477399999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":97,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.534999+0000","last_change":"2024-09-21T10:40:55.616051+0000","last_active":"2024-09-21T10:48:06.534999+0000","last_peered":"2024-09-21T10:48:06.534999+0000","last_clean":"2024-09-21T10:48:06.534999+0000","last_became_active":"2024-09-21T10:40:55.615741+0000","last_became_peered":"2024-09-21T10:40:55.615741+0000","last_unstale":"2024-09-21T10:48:06.534999+0000","last_undegraded":"2024-09-21T10:48:06.534999+0000","last_fullsized":"2024-09-21T10:48:06.534999+0000","mapping_epoch":1320,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:07.673527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2095,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.582277+0000","last_change":"2024-09-21T10:40:13.436234+0000","last_active":"2024-09-21T10:48:19.582277+0000","last_peered":"2024-09-21T10:48:19.582277+0000","last_clean":"2024-09-21T10:48:19.582277+0000","last_became_active":"2024-09-21T10:40:13.435946+0000","last_became_peered":"2024-09-21T10:40:13.435946+0000","last_unstale":"2024-09-21T10:48:19.582277+0000","last_undegraded":"2024-09-21T10:48:19.582277+0000","last_fullsized":"2024-09-21T10:48:19.582277+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:08:08.670739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":887,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.579694+0000","last_change":"2024-09-21T10:40:13.431693+0000","last_active":"2024-09-21T10:48:19.579694+0000","last_peered":"2024-09-21T10:48:19.579694+0000","last_clean":"2024-09-21T10:48:19.579694+0000","last_became_active":"2024-09-21T10:40:13.431389+0000","last_became_peered":"2024-09-21T10:40:13.431389+0000","last_unstale":"2024-09-21T10:48:19.579694+0000","last_undegraded":"2024-09-21T10:48:19.579694+0000","last_fullsized":"2024-09-21T10:48:19.579694+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:22.227336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2133,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.593100+0000","last_change":"2024-09-21T10:44:14.409601+0000","last_active":"2024-09-21T10:48:19.593100+0000","last_peered":"2024-09-21T10:48:19.593100+0000","last_clean":"2024-09-21T10:48:19.593100+0000","last_became_active":"2024-09-21T10:44:14.409262+0000","last_became_peered":"2024-09-21T10:44:14.409262+0000","last_unstale":"2024-09-21T10:48:19.593100+0000","last_undegraded":"2024-09-21T10:48:19.593100+0000","last_fullsized":"2024-09-21T10:48:19.593100+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:53:01.809847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2100,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.592587+0000","last_change":"2024-09-21T10:09:06.560130+0000","last_active":"2024-09-21T10:48:19.592587+0000","last_peered":"2024-09-21T10:48:19.592587+0000","last_clean":"2024-09-21T10:48:19.592587+0000","last_became_active":"2024-09-21T10:09:06.559827+0000","last_became_peered":"2024-09-21T10:09:06.559827+0000","last_unstale":"2024-09-21T10:48:19.592587+0000","last_undegraded":"2024-09-21T10:48:19.592587+0000","last_fullsized":"2024-09-21T10:48:19.592587+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:05:17.166734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":47,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:16.337277+0000","last_change":"2024-09-21T10:44:14.400691+0000","last_active":"2024-09-21T10:48:16.337277+0000","last_peered":"2024-09-21T10:48:16.337277+0000","last_clean":"2024-09-21T10:48:16.337277+0000","last_became_active":"2024-09-21T10:44:14.400377+0000","last_became_peered":"2024-09-21T10:44:14.400377+0000","last_unstale":"2024-09-21T10:48:16.337277+0000","last_undegraded":"2024-09-21T10:48:16.337277+0000","last_fullsized":"2024-09-21T10:48:16.337277+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:03.555078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1381,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.530189+0000","last_change":"2024-09-21T10:44:14.402965+0000","last_active":"2024-09-21T10:48:06.530189+0000","last_peered":"2024-09-21T10:48:06.530189+0000","last_clean":"2024-09-21T10:48:06.530189+0000","last_became_active":"2024-09-21T10:44:14.402634+0000","last_became_peered":"2024-09-21T10:44:14.402634+0000","last_unstale":"2024-09-21T10:48:06.530189+0000","last_undegraded":"2024-09-21T10:48:06.530189+0000","last_fullsized":"2024-09-21T10:48:06.530189+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:40:16.323155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2081,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.579775+0000","last_change":"2024-09-21T10:09:06.560589+0000","last_active":"2024-09-21T10:48:19.579775+0000","last_peered":"2024-09-21T10:48:19.579775+0000","last_clean":"2024-09-21T10:48:19.579775+0000","last_became_active":"2024-09-21T10:09:06.557552+0000","last_became_peered":"2024-09-21T10:09:06.557552+0000","last_unstale":"2024-09-21T10:48:19.579775+0000","last_undegraded":"2024-09-21T10:48:19.579775+0000","last_fullsized":"2024-09-21T10:48:19.579775+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:09:24.866759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":1437,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.538051+0000","last_change":"2024-09-21T10:44:14.415910+0000","last_active":"2024-09-21T10:48:06.538051+0000","last_peered":"2024-09-21T10:48:06.538051+0000","last_clean":"2024-09-21T10:48:06.538051+0000","last_became_active":"2024-09-21T10:44:14.402926+0000","last_became_peered":"2024-09-21T10:44:14.402926+0000","last_unstale":"2024-09-21T10:48:06.538051+0000","last_undegraded":"2024-09-21T10:48:06.538051+0000","last_fullsized":"2024-09-21T10:48:06.538051+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:55:07.144611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":797,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.581193+0000","last_change":"2024-09-21T10:44:17.413348+0000","last_active":"2024-09-21T10:48:19.581193+0000","last_peered":"2024-09-21T10:48:19.581193+0000","last_clean":"2024-09-21T10:48:19.581193+0000","last_became_active":"2024-09-21T10:44:17.412916+0000","last_became_peered":"2024-09-21T10:44:17.412916+0000","last_unstale":"2024-09-21T10:48:19.581193+0000","last_undegraded":"2024-09-21T10:48:19.581193+0000","last_fullsized":"2024-09-21T10:48:19.581193+0000","mapping_epoch":1372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1373,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:18.592283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2085,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.582740+0000","last_change":"2024-09-21T10:09:06.538940+0000","last_active":"2024-09-21T10:48:19.582740+0000","last_peered":"2024-09-21T10:48:19.582740+0000","last_clean":"2024-09-21T10:48:19.582740+0000","last_became_active":"2024-09-21T10:09:06.538668+0000","last_became_peered":"2024-09-21T10:09:06.538668+0000","last_unstale":"2024-09-21T10:48:19.582740+0000","last_undegraded":"2024-09-21T10:48:19.582740+0000","last_fullsized":"2024-09-21T10:48:19.582740+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:42:44.299488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2101,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.580192+0000","last_change":"2024-09-21T10:44:14.398581+0000","last_active":"2024-09-21T10:48:19.580192+0000","last_peered":"2024-09-21T10:48:19.580192+0000","last_clean":"2024-09-21T10:48:19.580192+0000","last_became_active":"2024-09-21T10:44:14.398258+0000","last_became_peered":"2024-09-21T10:44:14.398258+0000","last_unstale":"2024-09-21T10:48:19.580192+0000","last_undegraded":"2024-09-21T10:48:19.580192+0000","last_fullsized":"2024-09-21T10:48:19.580192+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:23.960956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1375,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.535006+0000","last_change":"2024-09-21T10:07:55.998799+0000","last_active":"2024-09-21T10:48:06.535006+0000","last_peered":"2024-09-21T10:48:06.535006+0000","last_clean":"2024-09-21T10:48:06.535006+0000","last_became_active":"2024-09-21T10:07:55.998510+0000","last_became_peered":"2024-09-21T10:07:55.998510+0000","last_unstale":"2024-09-21T10:48:06.535006+0000","last_undegraded":"2024-09-21T10:48:06.535006+0000","last_fullsized":"2024-09-21T10:48:06.535006+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:10.474008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1404,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.537136+0000","last_change":"2024-09-21T10:44:14.399842+0000","last_active":"2024-09-21T10:48:06.537136+0000","last_peered":"2024-09-21T10:48:06.537136+0000","last_clean":"2024-09-21T10:48:06.537136+0000","last_became_active":"2024-09-21T10:44:14.399426+0000","last_became_peered":"2024-09-21T10:44:14.399426+0000","last_unstale":"2024-09-21T10:48:06.537136+0000","last_undegraded":"2024-09-21T10:48:06.537136+0000","last_fullsized":"2024-09-21T10:48:06.537136+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:26:45.362101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1399,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.534682+0000","last_change":"2024-09-21T10:40:13.432387+0000","last_active":"2024-09-21T10:48:06.534682+0000","last_peered":"2024-09-21T10:48:06.534682+0000","last_clean":"2024-09-21T10:48:06.534682+0000","last_became_active":"2024-09-21T10:40:13.432053+0000","last_became_peered":"2024-09-21T10:40:13.432053+0000","last_unstale":"2024-09-21T10:48:06.534682+0000","last_undegraded":"2024-09-21T10:48:06.534682+0000","last_fullsized":"2024-09-21T10:48:06.534682+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:37.486101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1378,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.532771+0000","last_change":"2024-09-21T10:07:56.020791+0000","last_active":"2024-09-21T10:48:06.532771+0000","last_peered":"2024-09-21T10:48:06.532771+0000","last_clean":"2024-09-21T10:48:06.532771+0000","last_became_active":"2024-09-21T10:07:56.020205+0000","last_became_peered":"2024-09-21T10:07:56.020205+0000","last_unstale":"2024-09-21T10:48:06.532771+0000","last_undegraded":"2024-09-21T10:48:06.532771+0000","last_fullsized":"2024-09-21T10:48:06.532771+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:09.813166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1375,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.534730+0000","last_change":"2024-09-21T10:09:06.533271+0000","last_active":"2024-09-21T10:48:06.534730+0000","last_peered":"2024-09-21T10:48:06.534730+0000","last_clean":"2024-09-21T10:48:06.534730+0000","last_became_active":"2024-09-21T10:09:06.532041+0000","last_became_peered":"2024-09-21T10:09:06.532041+0000","last_unstale":"2024-09-21T10:48:06.534730+0000","last_undegraded":"2024-09-21T10:48:06.534730+0000","last_fullsized":"2024-09-21T10:48:06.534730+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:04.236375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":2102,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.580708+0000","last_change":"2024-09-21T10:09:06.543422+0000","last_active":"2024-09-21T10:48:19.580708+0000","last_peered":"2024-09-21T10:48:19.580708+0000","last_clean":"2024-09-21T10:48:19.580708+0000","last_became_active":"2024-09-21T10:09:06.539647+0000","last_became_peered":"2024-09-21T10:09:06.539647+0000","last_unstale":"2024-09-21T10:48:19.580708+0000","last_undegraded":"2024-09-21T10:48:19.580708+0000","last_fullsized":"2024-09-21T10:48:19.580708+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:04.990085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2101,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.593135+0000","last_change":"2024-09-21T10:07:56.018560+0000","last_active":"2024-09-21T10:48:19.593135+0000","last_peered":"2024-09-21T10:48:19.593135+0000","last_clean":"2024-09-21T10:48:19.593135+0000","last_became_active":"2024-09-21T10:07:56.018261+0000","last_became_peered":"2024-09-21T10:07:56.018261+0000","last_unstale":"2024-09-21T10:48:19.593135+0000","last_undegraded":"2024-09-21T10:48:19.593135+0000","last_fullsized":"2024-09-21T10:48:19.593135+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:04:54.395456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1376,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.539346+0000","last_change":"2024-09-21T10:09:06.531935+0000","last_active":"2024-09-21T10:48:06.539346+0000","last_peered":"2024-09-21T10:48:06.539346+0000","last_clean":"2024-09-21T10:48:06.539346+0000","last_became_active":"2024-09-21T10:09:06.531610+0000","last_became_peered":"2024-09-21T10:09:06.531610+0000","last_unstale":"2024-09-21T10:48:06.539346+0000","last_undegraded":"2024-09-21T10:48:06.539346+0000","last_fullsized":"2024-09-21T10:48:06.539346+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:33.874405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1413,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.532608+0000","last_change":"2024-09-21T10:44:14.401913+0000","last_active":"2024-09-21T10:48:06.532608+0000","last_peered":"2024-09-21T10:48:06.532608+0000","last_clean":"2024-09-21T10:48:06.532608+0000","last_became_active":"2024-09-21T10:44:14.401496+0000","last_became_peered":"2024-09-21T10:44:14.401496+0000","last_unstale":"2024-09-21T10:48:06.532608+0000","last_undegraded":"2024-09-21T10:48:06.532608+0000","last_fullsized":"2024-09-21T10:48:06.532608+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:07:38.531723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":50,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.532202+0000","last_change":"2024-09-21T10:44:14.410444+0000","last_active":"2024-09-21T10:48:06.532202+0000","last_peered":"2024-09-21T10:48:06.532202+0000","last_clean":"2024-09-21T10:48:06.532202+0000","last_became_active":"2024-09-21T10:44:14.395465+0000","last_became_peered":"2024-09-21T10:44:14.395465+0000","last_unstale":"2024-09-21T10:48:06.532202+0000","last_undegraded":"2024-09-21T10:48:06.532202+0000","last_fullsized":"2024-09-21T10:48:06.532202+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:05:44.575878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1391,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.536981+0000","last_change":"2024-09-21T10:09:06.559132+0000","last_active":"2024-09-21T10:48:06.536981+0000","last_peered":"2024-09-21T10:48:06.536981+0000","last_clean":"2024-09-21T10:48:06.536981+0000","last_became_active":"2024-09-21T10:09:06.558395+0000","last_became_peered":"2024-09-21T10:09:06.558395+0000","last_unstale":"2024-09-21T10:48:06.536981+0000","last_undegraded":"2024-09-21T10:48:06.536981+0000","last_fullsized":"2024-09-21T10:48:06.536981+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:27:00.073554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":50,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.532457+0000","last_change":"2024-09-21T10:44:14.419937+0000","last_active":"2024-09-21T10:48:06.532457+0000","last_peered":"2024-09-21T10:48:06.532457+0000","last_clean":"2024-09-21T10:48:06.532457+0000","last_became_active":"2024-09-21T10:44:14.402761+0000","last_became_peered":"2024-09-21T10:44:14.402761+0000","last_unstale":"2024-09-21T10:48:06.532457+0000","last_undegraded":"2024-09-21T10:48:06.532457+0000","last_fullsized":"2024-09-21T10:48:06.532457+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:42.875310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1379,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.534211+0000","last_change":"2024-09-21T10:44:14.402411+0000","last_active":"2024-09-21T10:48:06.534211+0000","last_peered":"2024-09-21T10:48:06.534211+0000","last_clean":"2024-09-21T10:48:06.534211+0000","last_became_active":"2024-09-21T10:44:14.402045+0000","last_became_peered":"2024-09-21T10:44:14.402045+0000","last_unstale":"2024-09-21T10:48:06.534211+0000","last_undegraded":"2024-09-21T10:48:06.534211+0000","last_fullsized":"2024-09-21T10:48:06.534211+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:57:10.535776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":50,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.532454+0000","last_change":"2024-09-21T10:44:14.421612+0000","last_active":"2024-09-21T10:48:06.532454+0000","last_peered":"2024-09-21T10:48:06.532454+0000","last_clean":"2024-09-21T10:48:06.532454+0000","last_became_active":"2024-09-21T10:44:14.406095+0000","last_became_peered":"2024-09-21T10:44:14.406095+0000","last_unstale":"2024-09-21T10:48:06.532454+0000","last_undegraded":"2024-09-21T10:48:06.532454+0000","last_fullsized":"2024-09-21T10:48:06.532454+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:21:19.918156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1384,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.530065+0000","last_change":"2024-09-21T10:07:21.497877+0000","last_active":"2024-09-21T10:48:06.530065+0000","last_peered":"2024-09-21T10:48:06.530065+0000","last_clean":"2024-09-21T10:48:06.530065+0000","last_became_active":"2024-09-21T10:07:21.496317+0000","last_became_peered":"2024-09-21T10:07:21.496317+0000","last_unstale":"2024-09-21T10:48:06.530065+0000","last_undegraded":"2024-09-21T10:48:06.530065+0000","last_fullsized":"2024-09-21T10:48:06.530065+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:39:03.848183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2097,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.580788+0000","last_change":"2024-09-21T10:44:14.398487+0000","last_active":"2024-09-21T10:48:19.580788+0000","last_peered":"2024-09-21T10:48:19.580788+0000","last_clean":"2024-09-21T10:48:19.580788+0000","last_became_active":"2024-09-21T10:44:14.398142+0000","last_became_peered":"2024-09-21T10:44:14.398142+0000","last_unstale":"2024-09-21T10:48:19.580788+0000","last_undegraded":"2024-09-21T10:48:19.580788+0000","last_fullsized":"2024-09-21T10:48:19.580788+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:20:41.065055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":46,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.531901+0000","last_change":"2024-09-21T10:44:14.398878+0000","last_active":"2024-09-21T10:48:06.531901+0000","last_peered":"2024-09-21T10:48:06.531901+0000","last_clean":"2024-09-21T10:48:06.531901+0000","last_became_active":"2024-09-21T10:44:14.398547+0000","last_became_peered":"2024-09-21T10:44:14.398547+0000","last_unstale":"2024-09-21T10:48:06.531901+0000","last_undegraded":"2024-09-21T10:48:06.531901+0000","last_fullsized":"2024-09-21T10:48:06.531901+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:13.008134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2127,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.592863+0000","last_change":"2024-09-21T10:44:14.409051+0000","last_active":"2024-09-21T10:48:19.592863+0000","last_peered":"2024-09-21T10:48:19.592863+0000","last_clean":"2024-09-21T10:48:19.592863+0000","last_became_active":"2024-09-21T10:44:14.407892+0000","last_became_peered":"2024-09-21T10:44:14.407892+0000","last_unstale":"2024-09-21T10:48:19.592863+0000","last_undegraded":"2024-09-21T10:48:19.592863+0000","last_fullsized":"2024-09-21T10:48:19.592863+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:19.549648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":2111,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.593292+0000","last_change":"2024-09-21T10:44:14.408837+0000","last_active":"2024-09-21T10:48:19.593292+0000","last_peered":"2024-09-21T10:48:19.593292+0000","last_clean":"2024-09-21T10:48:19.593292+0000","last_became_active":"2024-09-21T10:44:14.408399+0000","last_became_peered":"2024-09-21T10:44:14.408399+0000","last_unstale":"2024-09-21T10:48:19.593292+0000","last_undegraded":"2024-09-21T10:48:19.593292+0000","last_fullsized":"2024-09-21T10:48:19.593292+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:41:38.729386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1392,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.535607+0000","last_change":"2024-09-21T10:44:14.401435+0000","last_active":"2024-09-21T10:48:06.535607+0000","last_peered":"2024-09-21T10:48:06.535607+0000","last_clean":"2024-09-21T10:48:06.535607+0000","last_became_active":"2024-09-21T10:44:14.400908+0000","last_became_peered":"2024-09-21T10:44:14.400908+0000","last_unstale":"2024-09-21T10:48:06.535607+0000","last_undegraded":"2024-09-21T10:48:06.535607+0000","last_fullsized":"2024-09-21T10:48:06.535607+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:46.748289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":52,"reported_epoch":1403,"state":"active+clean","last_fresh":"2024-09-21T10:48:06.532004+0000","last_change":"2024-09-21T10:44:14.418113+0000","last_active":"2024-09-21T10:48:06.532004+0000","last_peered":"2024-09-21T10:48:06.532004+0000","last_clean":"2024-09-21T10:48:06.532004+0000","last_became_active":"2024-09-21T10:44:14.398327+0000","last_became_peered":"2024-09-21T10:44:14.398327+0000","last_unstale":"2024-09-21T10:48:06.532004+0000","last_undegraded":"2024-09-21T10:48:06.532004+0000","last_fullsized":"2024-09-21T10:48:06.532004+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:08.810863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2097,"reported_epoch":1404,"state":"active+clean","last_fresh":"2024-09-21T10:48:19.581129+0000","last_change":"2024-09-21T10:44:14.398584+0000","last_active":"2024-09-21T10:48:19.581129+0000","last_peered":"2024-09-21T10:48:19.581129+0000","last_clean":"2024-09-21T10:48:19.581129+0000","last_became_active":"2024-09-21T10:44:14.398278+0000","last_became_peered":"2024-09-21T10:44:14.398278+0000","last_unstale":"2024-09-21T10:48:19.581129+0000","last_undegraded":"2024-09-21T10:48:19.581129+0000","last_fullsized":"2024-09-21T10:48:19.581129+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:19.681750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":54,"num_pg":16,"stat_sum":{"num_bytes":797966360,"num_objects":12177,"num_object_clones":0,"num_object_copies":36531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12177,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":246045,"num_write_kb":8263041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2370908160,"data_stored":2370895944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120945,"ondisk_log_size":120945,"up":48,"acting":48,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":10310,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234652,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":639872,"kb_used_data":342132,"kb_used_omap":1,"kb_used_meta":297662,"kb_avail":93105280,"statfs":{"total":95995035648,"available":95339806720,"internally_reserved":0,"allocated":350343168,"data_stored":345834960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":304806347},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496289,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":896132,"kb_used_data":435960,"kb_used_omap":1,"kb_used_meta":460094,"kb_avail":92849020,"statfs":{"total":95995035648,"available":95077396480,"internally_reserved":0,"allocated":446423040,"data_stored":441911249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":471136714},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757926,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":778792,"kb_used_data":346464,"kb_used_omap":1,"kb_used_meta":432254,"kb_avail":92966360,"statfs":{"total":95995035648,"available":95197552640,"internally_reserved":0,"allocated":354779136,"data_stored":350286563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":442628550},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019562,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":675956,"kb_used_data":207596,"kb_used_omap":1,"kb_used_meta":468286,"kb_avail":93069196,"statfs":{"total":95995035648,"available":95302856704,"internally_reserved":0,"allocated":212578304,"data_stored":208077674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":479525318},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369305,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1096744,"kb_used_data":587036,"kb_used_omap":69,"kb_used_meta":509626,"kb_avail":92648408,"statfs":{"total":95995035648,"available":94871969792,"internally_reserved":0,"allocated":601124864,"data_stored":596635842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":521857154},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2920577761760,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807957991,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":674512,"kb_used_data":223240,"kb_used_omap":66,"kb_used_meta":451197,"kb_avail":93070640,"statfs":{"total":95995035648,"available":95304335360,"internally_reserved":0,"allocated":228597760,"data_stored":224120411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68589,"internal_metadata":462025747},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":666,"seq":2860448219630,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":722844,"kb_used_data":334228,"kb_used_omap":76,"kb_used_meta":388531,"kb_avail":93022308,"statfs":{"total":95995035648,"available":95254843392,"internally_reserved":0,"allocated":342249472,"data_stored":337760988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78716,"internal_metadata":397855876},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":315559936,"data_stored":315555864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":211943424,"data_stored":211943424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":573181952,"data_stored":573177880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":196083712,"data_stored":196083712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":326631424,"data_stored":326631424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":414650368,"data_stored":414646296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":332857344,"data_stored":332857344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:48:28.167 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T10:48:28.168 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:48:28.168 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-21T10:48:28.168 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:48:28.168 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:48:28.168 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-21T10:48:28.214 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:28.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:28.263 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:28.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:28.364 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:28.412 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:28.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:28.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:28.514 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:28.563 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:28.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:28.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:28.630 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:48:28.665 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:28.713 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:28.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:28.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:28.783 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:28.815 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:28.865 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:28.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:28.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:28.966 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:29.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:29.011 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:29.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:29.112 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:29.143 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:29.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:29.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:29.244 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:29.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:29.279 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:29.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:29.380 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:29.410 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:29.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:29.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:29.512 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:29.545 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:29.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:29.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:29.646 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:29.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:29.685 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:29.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:29.785 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:29.831 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:29.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:29.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:29.894 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T10:48:29.932 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:29.953 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:48:29.953 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T10:48:29.953 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:29.953 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.45308.0:254302 54.c e1405/1376 54:3aef7a38:::benchmark_data_smithi070_261699_object125189:head v 1405'15379, mlcod=1405'15376)", 2024-09-21T10:48:29.953 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:48:29.949613+0000", 2024-09-21T10:48:29.954 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.00090808800000000004, 2024-09-21T10:48:29.954 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00093723400000000001, 2024-09-21T10:48:29.954 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:29.954 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T10:48:29.954 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:29.954 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:29.954 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:29.954 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:48:29.949613+0000", 2024-09-21T10:48:29.954 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:29.954 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:29.954 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:29.954 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:29.954 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:48:29.949613+0000", 2024-09-21T10:48:29.955 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:29.955 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:29.955 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:29.955 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:29.955 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:48:29.949612+0000", 2024-09-21T10:48:29.955 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:48:29.955 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:29.955 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:29.955 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:29.955 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:48:29.949619+0000", 2024-09-21T10:48:29.955 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.221e-06 2024-09-21T10:48:29.955 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:29.955 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:29.956 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:29.956 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:48:29.949633+0000", 2024-09-21T10:48:29.956 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4064e-05 2024-09-21T10:48:29.956 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:29.956 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:29.956 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:29.956 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:48:29.949640+0000", 2024-09-21T10:48:29.956 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8549999999999996e-06 2024-09-21T10:48:29.956 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:29.956 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:29.956 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:29.956 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:48:29.949680+0000", 2024-09-21T10:48:29.956 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0661999999999998e-05 2024-09-21T10:48:29.956 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:29.957 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:29.957 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:48:29.957 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:48:29.949709+0000", 2024-09-21T10:48:29.957 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8568000000000002e-05 2024-09-21T10:48:29.957 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:29.957 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:29.957 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:29.957 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:29.957 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T10:48:29.957 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 1 2024-09-21T10:48:29.957 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:48:29.965 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:29.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:29.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:30.067 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:30.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:29 smithi160 ceph-mon[101879]: pgmap v2492: 89 pgs: 89 active+clean; 487 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 893 op/s 2024-09-21T10:48:30.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:29 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/11263647' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:48:30.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:29 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:48:30.100 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:30.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:30.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:30.202 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:30.231 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:30.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:30.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:30.241 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:29 smithi070 ceph-mon[107385]: pgmap v2492: 89 pgs: 89 active+clean; 487 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 893 op/s 2024-09-21T10:48:30.241 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:29 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/11263647' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:48:30.241 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:29 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:48:30.242 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:29 smithi070 ceph-mon[112139]: pgmap v2492: 89 pgs: 89 active+clean; 487 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 893 op/s 2024-09-21T10:48:30.242 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:29 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/11263647' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:48:30.242 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:29 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:48:30.332 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:30.359 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:30.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:30.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:30.461 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:30.490 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:30.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:30.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:30.591 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:30.621 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:30.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:30.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:30.723 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:30.726 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:48:30.762 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:30.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:30.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:30.865 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:30.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:30.912 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:30.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:31.011 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:30 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:48:31.011 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:30 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:48:31.012 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:30 smithi160 ceph-mon[101879]: osdmap e1405: 8 total, 8 up, 7 in 2024-09-21T10:48:31.014 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:31.040 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:31.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:31.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:31.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:30 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:48:31.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:30 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:48:31.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:30 smithi070 ceph-mon[107385]: osdmap e1405: 8 total, 8 up, 7 in 2024-09-21T10:48:31.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:30 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:48:31.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:30 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:48:31.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:30 smithi070 ceph-mon[112139]: osdmap e1405: 8 total, 8 up, 7 in 2024-09-21T10:48:31.143 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:31.186 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:31.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:31.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:31.287 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:31.318 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:31.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:31.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:31.420 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:31.448 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:31.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:31.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:31.550 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:31.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:31.588 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:31.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:31.689 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:31.724 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:31.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:31.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:31.826 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:31.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:31.854 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:31.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:31.955 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:31.983 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:31 smithi070 ceph-mon[107385]: pgmap v2494: 89 pgs: 89 active+clean; 345 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 891 op/s 2024-09-21T10:48:31.983 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:31 smithi070 ceph-mon[112139]: pgmap v2494: 89 pgs: 89 active+clean; 345 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 891 op/s 2024-09-21T10:48:31.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:31.987 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:31.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:32.088 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:32.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:32.125 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:32.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:32.227 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:32.268 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:32.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:32.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:32.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:31 smithi160 ceph-mon[101879]: pgmap v2494: 89 pgs: 89 active+clean; 345 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 891 op/s 2024-09-21T10:48:32.358 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:48:32.370 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:32.453 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:32.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:32.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:32.555 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:32.597 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:32.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:32.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:32.699 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:32.734 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:32.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:32.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:32.836 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:32.869 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:32.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:32.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:32.971 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:33.014 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:33.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:33.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:33.115 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:33.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:33.144 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:33.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:33.191 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:48:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:48:33] "GET /metrics HTTP/1.1" 200 36601 "" "Prometheus/2.51.0" 2024-09-21T10:48:33.245 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:33.252 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:48:33.252 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:48:33.281 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:48:33.281 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:48:33.283 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:33.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:33.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:33.310 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:48:33.310 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:48:33.339 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:48:33.339 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:48:33.376 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:48:33.376 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:48:33.385 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:33.408 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:48:33.408 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:48:33.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:33.424 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:33.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:33.445 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:48:33.446 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:48:33.475 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:48:33.475 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:48:33.525 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:33.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:33.575 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:33.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:33.661 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:48:33.661 INFO:teuthology.orchestra.run.smithi160.stdout: "success": "" 2024-09-21T10:48:33.661 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:48:33.676 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:33.706 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:33.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:33.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:33.808 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:33.840 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:33.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:33.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:33.942 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:33.985 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:33.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:33.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:34.087 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:34.116 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:34.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:34.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:34.164 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:33 smithi160 ceph-mon[101879]: pgmap v2495: 89 pgs: 89 active+clean; 205 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 889 op/s 2024-09-21T10:48:34.164 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:33 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3636853738' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:48:34.218 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:34.247 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:34.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:34.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:34.302 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Removed 129110 objects 2024-09-21T10:48:34.302 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Clean up completed and total clean up time :136.292 2024-09-21T10:48:34.349 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:34.370 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:34.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:34.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:34.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:33 smithi070 ceph-mon[107385]: pgmap v2495: 89 pgs: 89 active+clean; 205 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 889 op/s 2024-09-21T10:48:34.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:33 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3636853738' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:48:34.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:33 smithi070 ceph-mon[112139]: pgmap v2495: 89 pgs: 89 active+clean; 205 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 889 op/s 2024-09-21T10:48:34.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:33 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3636853738' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:48:34.471 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:34.492 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:34.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:34.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:34.594 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:34.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:34.616 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:34.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:34.717 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:34.739 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:34.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:34.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:34.840 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:34.861 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:34.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:34.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:34.933 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:48:34.933 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:48:34.962 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:48:34.963 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:34.985 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:34.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:34.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:35.086 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:35.107 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:35.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:35.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:35.164 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:35.209 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:35.231 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:35.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:35.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:35.332 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:35.354 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:35.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:35.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:35.455 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:35.476 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:35.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:35.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:35.578 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:35.600 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:35.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:35.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:35.702 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:35.726 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:35.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:35.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:35.828 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:35.851 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:35.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:35.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:48:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:48:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:35 smithi070 ceph-mon[107385]: pgmap v2496: 89 pgs: 89 active+clean; 175 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 879 op/s 2024-09-21T10:48:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:48:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:48:35.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:48:35.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:48:35.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:48:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:48:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:48:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:35 smithi070 ceph-mon[112139]: pgmap v2496: 89 pgs: 89 active+clean; 175 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 879 op/s 2024-09-21T10:48:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:48:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:48:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:48:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:48:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:48:35.953 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:35.977 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:35.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:35.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:36.079 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:48:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:48:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:35 smithi160 ceph-mon[101879]: pgmap v2496: 89 pgs: 89 active+clean; 175 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 879 op/s 2024-09-21T10:48:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:48:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:48:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:48:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:48:36.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:48:36.103 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:36.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:36.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:36.205 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:36.227 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:36.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:36.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:36.329 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:36.351 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:36.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:36.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:36.426 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:48:36.453 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:36.490 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:36.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:36.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:36.592 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:36.617 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:36.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:36.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:36.719 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:36.744 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:36.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:36.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:36.845 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:36.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:36.868 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:36.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:36.969 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:36.990 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:36.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:36.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:37.092 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:37.118 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:37.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:37.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:37.219 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:37.244 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:37.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:37.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:37.346 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:37.370 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:37.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:37.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:37.471 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:37.496 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:37.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:37.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:37.597 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:37.622 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:37.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:37.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:37.723 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:37.749 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:37.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:37.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:37.851 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:37.876 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:37.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:37.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:37.977 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:38.002 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:37 smithi070 ceph-mon[112139]: pgmap v2497: 89 pgs: 89 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 777 op/s 2024-09-21T10:48:38.002 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:37 smithi070 ceph-mon[107385]: pgmap v2497: 89 pgs: 89 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 777 op/s 2024-09-21T10:48:38.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:38.003 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:38.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:38.052 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:37 smithi160 ceph-mon[101879]: pgmap v2497: 89 pgs: 89 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 777 op/s 2024-09-21T10:48:38.104 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:38.127 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:38.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:38.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:38.218 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:48:38.229 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:38.283 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:38.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:38.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:38.312 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:38.385 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:38.407 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:38.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:38.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:38.423 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:48:38.509 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:38.531 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:38.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:38.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:38.633 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:38.654 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:38.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:38.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:38.756 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:38.777 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:38.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:38.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:38.879 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:38.882 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T10:48:38.900 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:38.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:38.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:39.002 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:39.024 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:39.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:39.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:39.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:38 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4266411121' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:48:39.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:38 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:48:39.073 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:38 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4266411121' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:48:39.073 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:38 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:48:39.126 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:39.147 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:39.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:39.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:39.209 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:38 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4266411121' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:48:39.210 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:38 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:48:39.249 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:39.270 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:39.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:39.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:39.372 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:39.393 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:39.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:39.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:39.495 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:39.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:39.517 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:39.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:39.618 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:39.640 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:39.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:39.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:39.742 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:39.765 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:39.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:39.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:39.787 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-09-21T10:48:39.787 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-09-21T10:48:39.866 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:39.890 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:39.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:39.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:39.992 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:40.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:40.013 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:40.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:40.083 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:39 smithi070 ceph-mon[107385]: pgmap v2498: 89 pgs: 89 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 778 op/s 2024-09-21T10:48:40.084 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:39 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:48:40.084 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:39 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:48:40.084 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:39 smithi070 ceph-mon[107385]: osdmap e1406: 8 total, 8 up, 7 in 2024-09-21T10:48:40.084 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:39 smithi070 ceph-mon[112139]: pgmap v2498: 89 pgs: 89 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 778 op/s 2024-09-21T10:48:40.084 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:39 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:48:40.084 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:39 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:48:40.084 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:39 smithi070 ceph-mon[112139]: osdmap e1406: 8 total, 8 up, 7 in 2024-09-21T10:48:40.114 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:40.129 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:48:40.129 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:48:40.135 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:40.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:40.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:40.137 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:39 smithi160 ceph-mon[101879]: pgmap v2498: 89 pgs: 89 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 778 op/s 2024-09-21T10:48:40.137 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:39 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:48:40.137 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:39 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:48:40.137 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:39 smithi160 ceph-mon[101879]: osdmap e1406: 8 total, 8 up, 7 in 2024-09-21T10:48:40.238 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:40.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:40.260 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:40.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:40.361 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:40.386 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:40.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:40.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:40.488 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:40.510 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:40.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:40.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:40.604 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:48:40.604 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:48:40.611 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:40.621 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:48:40.621 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:48:40.634 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:40.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:40.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:40.638 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:48:40.639 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:48:40.666 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:48:40.666 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:48:40.673 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:40.684 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:48:40.684 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:48:40.701 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:48:40.701 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:48:40.718 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:48:40.718 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:48:40.734 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:48:40.735 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:48:40.736 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:40.757 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:40.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:40.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:40.795 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:48:40.859 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:40.900 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:40.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:40.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:41.002 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:41.025 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:41.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:41.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:41.037 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:48:41.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:40 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3328608915' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:48:41.127 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:41.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:41.148 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:41.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:41.249 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:41.271 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:41.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:41.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:41.372 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:41.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:40 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3328608915' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:48:41.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:40 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3328608915' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:48:41.393 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:41.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:41.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:41.495 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:41.540 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:41.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:41.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:41.642 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:41.665 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:41.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:41.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:41.766 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:41.788 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:41.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:41.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:41.890 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:41.911 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:41.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:41.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:42.013 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:42.036 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:42.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:42.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:42.106 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:48:42.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:41 smithi070 ceph-mon[107385]: pgmap v2500: 89 pgs: 89 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 550 op/s 2024-09-21T10:48:42.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:41 smithi070 ceph-mon[112139]: pgmap v2500: 89 pgs: 89 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 550 op/s 2024-09-21T10:48:42.138 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:42.159 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:42.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:42.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:42.182 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:41 smithi160 ceph-mon[101879]: pgmap v2500: 89 pgs: 89 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 550 op/s 2024-09-21T10:48:42.261 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:42.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:42.285 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:42.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:42.386 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:42.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:42.409 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:42.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:42.489 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:42.511 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:42.532 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:42.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:42.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:42.634 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:42.654 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:42.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:42.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:42.677 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:48:42.677 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T10:48:42.678 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T10:48:42.678 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:48:42.756 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:42.778 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:42.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:42.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:42.880 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:42.902 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:42.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:42.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:43.003 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:43.025 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:43.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:43.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:43.077 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:48:43.077 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:48:43.077 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T10:48:43.077 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.341766 2024-09-21T10:48:43.078 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 5 0.3417656632934005 2024-09-21T10:48:43.127 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:43.148 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:43.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:43.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:43.249 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:43.271 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:43.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:43.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:43.284 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:48:43.373 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:43.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:48:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:48:43] "GET /metrics HTTP/1.1" 200 36598 "" "Prometheus/2.51.0" 2024-09-21T10:48:43.394 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:43.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:43.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:43.483 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:48:43.496 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:43.518 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:43.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:43.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:43.533 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:48:43.620 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:43.642 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:43.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:43.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:43.744 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:43.765 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:43.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:43.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:43.786 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:43.867 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:43.888 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:43.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:43.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:43.890 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'unique_pool_1' removed 2024-09-21T10:48:43.990 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:44.012 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:44.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:44.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:44.114 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:44.136 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:44.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:44.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:44.237 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:44.260 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:44.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:44.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:44.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:43 smithi070 ceph-mon[107385]: pgmap v2501: 89 pgs: 89 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 327 op/s 2024-09-21T10:48:44.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:43 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2757580539' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:48:44.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:43 smithi070 ceph-mon[112139]: pgmap v2501: 89 pgs: 89 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 327 op/s 2024-09-21T10:48:44.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:43 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2757580539' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:48:44.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:43 smithi160 ceph-mon[101879]: pgmap v2501: 89 pgs: 89 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 327 op/s 2024-09-21T10:48:44.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:43 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2757580539' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:48:44.362 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:44.384 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:44.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:44.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:44.486 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:44.507 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:44.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:44.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:44.609 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:44.631 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:44.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:44.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:44.732 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:44.758 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:44.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:44.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:44.860 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:44.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:44.886 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:44.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:44.972 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:44 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2757580539' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:48:44.972 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:44 smithi160 ceph-mon[101879]: osdmap e1407: 8 total, 8 up, 7 in 2024-09-21T10:48:44.988 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:45.009 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:45.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:45.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:45.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:44 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2757580539' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:48:45.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:44 smithi070 ceph-mon[107385]: osdmap e1407: 8 total, 8 up, 7 in 2024-09-21T10:48:45.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:44 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2757580539' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:48:45.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:44 smithi070 ceph-mon[112139]: osdmap e1407: 8 total, 8 up, 7 in 2024-09-21T10:48:45.110 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:45.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:45.132 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:45.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:45.233 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:45.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:45.255 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:45.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:45.302 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:48:45.302 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T10:48:45.302 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T10:48:45.302 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T10:48:45.302 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:48:45.356 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:45.377 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:45.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:45.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:45.479 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:45.501 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:45.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:45.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:45.602 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:45.624 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:45.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:45.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:45.726 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:45.748 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:45.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:45.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:45.850 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:45.872 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:45.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:45.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:45.923 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:48:45.973 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:45.995 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:45.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:45.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:46.097 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:46.119 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:46.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:46.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:46.121 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:48:46.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:45 smithi070 ceph-mon[107385]: pgmap v2503: 73 pgs: 73 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:48:46.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:45 smithi070 ceph-mon[112139]: pgmap v2503: 73 pgs: 73 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:48:46.221 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:46.242 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:46.242 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:45 smithi160 ceph-mon[101879]: pgmap v2503: 73 pgs: 73 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:48:46.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:46.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:46.318 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T10:48:46.318 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T10:48:46.319 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-09-21T10:48:46.319 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_2 16 2024-09-21T10:48:46.344 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:46.382 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:46.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:46.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:46.485 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:46.506 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:46.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:46.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:46.609 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:46.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:46.642 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:46.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:46.743 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:46.766 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:46.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:46.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:46.846 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:46.867 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:46.881 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:46.892 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:46.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:46.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:46.906 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T10:48:46.991 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:46 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2105461824' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:48:46.992 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:46 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:48:46.992 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:46 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2105461824' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:48:46.992 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:46 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:48:46.993 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:47.015 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:47.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:47.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:47.116 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:47.137 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:47.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:47.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:47.219 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:46 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2105461824' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:48:47.219 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:46 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:48:47.238 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:47.259 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:47.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:47.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:47.360 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:47.382 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:47.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:47.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:47.484 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:47.506 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:47.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:47.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:47.608 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:47.629 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:47.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:47.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:47.723 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:48:47.723 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T10:48:47.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T10:48:47.723 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T10:48:47.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.724 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:69148 53.13 53:c834d3e0:::benchmark_data_smithi070_228967_object69147:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1323)", 2024-09-21T10:48:47.724 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:40:59.631114+0000", 2024-09-21T10:48:47.724 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 468.08896582400001, 2024-09-21T10:48:47.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0407627319999999, 2024-09-21T10:48:47.724 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:47.724 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:48:47.724 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:47.724 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:47.724 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:47.724 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 69148 2024-09-21T10:48:47.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.724 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:47.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.725 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:47.725 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.631114+0000", 2024-09-21T10:48:47.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.725 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:47.725 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.631114+0000", 2024-09-21T10:48:47.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.725 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:47.725 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.631113+0000", 2024-09-21T10:48:47.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:48:47.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.726 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:47.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.631157+0000", 2024-09-21T10:48:47.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4168999999999997e-05 2024-09-21T10:48:47.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.726 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:47.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.631174+0000", 2024-09-21T10:48:47.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7478000000000001e-05 2024-09-21T10:48:47.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.726 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:47.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.631182+0000", 2024-09-21T10:48:47.727 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0010000000000001e-06 2024-09-21T10:48:47.727 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.727 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.727 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.789648+0000", 2024-09-21T10:48:47.727 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.15846608200000001 2024-09-21T10:48:47.727 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.727 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:48:47.727 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:40:59.789655+0000", 2024-09-21T10:48:47.727 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2809999999999997e-06 2024-09-21T10:48:47.727 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.728 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.728 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:00.669766+0000", 2024-09-21T10:48:47.728 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88011092800000001 2024-09-21T10:48:47.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.728 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.728 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:48:47.728 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:00.670343+0000", 2024-09-21T10:48:47.728 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00057693899999999995 2024-09-21T10:48:47.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.728 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.728 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.728 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:00.671828+0000", 2024-09-21T10:48:47.728 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0014857819999999999 2024-09-21T10:48:47.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.729 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.729 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:47.729 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:00.671877+0000", 2024-09-21T10:48:47.729 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8415000000000003e-05 2024-09-21T10:48:47.729 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.729 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:47.729 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.729 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.729 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:79863 53.12 53.de667272 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1329)", 2024-09-21T10:48:47.729 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:09.710219+0000", 2024-09-21T10:48:47.729 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 458.00986070099998, 2024-09-21T10:48:47.729 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98437714899999995, 2024-09-21T10:48:47.730 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:47.730 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:48:47.730 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:47.730 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:47.730 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:47.730 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 79863 2024-09-21T10:48:47.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.730 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:47.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:47.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.710219+0000", 2024-09-21T10:48:47.730 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:47.731 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.710219+0000", 2024-09-21T10:48:47.731 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:47.731 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.710216+0000", 2024-09-21T10:48:47.731 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999967 2024-09-21T10:48:47.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:47.731 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.710289+0000", 2024-09-21T10:48:47.731 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2967999999999997e-05 2024-09-21T10:48:47.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.732 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.732 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:47.732 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.710297+0000", 2024-09-21T10:48:47.732 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.875e-06 2024-09-21T10:48:47.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.732 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.732 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:47.732 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.710302+0000", 2024-09-21T10:48:47.732 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3449999999999996e-06 2024-09-21T10:48:47.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.732 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.732 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.732 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.710363+0000", 2024-09-21T10:48:47.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1414999999999994e-05 2024-09-21T10:48:47.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.733 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:48:47.733 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.710374+0000", 2024-09-21T10:48:47.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0164e-05 2024-09-21T10:48:47.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.733 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.733 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694583+0000", 2024-09-21T10:48:47.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98420943299999997 2024-09-21T10:48:47.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.733 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:47.734 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694596+0000", 2024-09-21T10:48:47.734 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3342e-05 2024-09-21T10:48:47.734 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.734 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:47.734 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.734 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:79867 53.12 53.8ded8132 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1329)", 2024-09-21T10:48:47.734 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:09.710968+0000", 2024-09-21T10:48:47.734 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 458.00911142299998, 2024-09-21T10:48:47.734 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98366795799999995, 2024-09-21T10:48:47.734 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:47.734 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:48:47.735 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:47.735 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:47.735 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:47.735 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 79867 2024-09-21T10:48:47.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.735 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:47.735 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.735 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:47.735 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.710968+0000", 2024-09-21T10:48:47.735 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.735 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.735 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:47.735 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.710968+0000", 2024-09-21T10:48:47.735 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.736 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.736 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:47.736 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.710967+0000", 2024-09-21T10:48:47.736 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:48:47.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.736 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.736 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:47.736 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.711004+0000", 2024-09-21T10:48:47.736 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6563999999999998e-05 2024-09-21T10:48:47.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.736 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.736 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:47.736 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.711017+0000", 2024-09-21T10:48:47.736 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3481e-05 2024-09-21T10:48:47.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.737 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.737 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:47.737 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.711025+0000", 2024-09-21T10:48:47.737 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8199999999999997e-06 2024-09-21T10:48:47.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.737 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.737 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.737 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.711065+0000", 2024-09-21T10:48:47.737 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9629000000000003e-05 2024-09-21T10:48:47.737 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:47.739 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.739 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.739 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:48:47.739 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.711072+0000", 2024-09-21T10:48:47.739 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6719999999999998e-06 2024-09-21T10:48:47.739 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.739 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.739 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.740 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694626+0000", 2024-09-21T10:48:47.740 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98355480200000001 2024-09-21T10:48:47.740 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.740 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.740 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:47.740 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694636+0000", 2024-09-21T10:48:47.740 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0057e-05 2024-09-21T10:48:47.740 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.740 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:47.740 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.740 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.740 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.740 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:79921 53.12 53.e42a4eb2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1329)", 2024-09-21T10:48:47.741 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:09.743427+0000", 2024-09-21T10:48:47.741 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 457.97665293300003, 2024-09-21T10:48:47.741 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.95123868700000003, 2024-09-21T10:48:47.741 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:47.741 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:48:47.741 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:47.741 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:47.741 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:47.741 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 79921 2024-09-21T10:48:47.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.741 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:47.741 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.741 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:47.741 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.743427+0000", 2024-09-21T10:48:47.742 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.742 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.742 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:47.742 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.743427+0000", 2024-09-21T10:48:47.742 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.742 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.742 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:47.742 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.743426+0000", 2024-09-21T10:48:47.742 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:48:47.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.743 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.743 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:47.743 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.743496+0000", 2024-09-21T10:48:47.743 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0148999999999995e-05 2024-09-21T10:48:47.743 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.743 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.743 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:47.743 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.743507+0000", 2024-09-21T10:48:47.743 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1277000000000001e-05 2024-09-21T10:48:47.743 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.743 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.743 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:47.743 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.743514+0000", 2024-09-21T10:48:47.744 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4259999999999997e-06 2024-09-21T10:48:47.744 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.744 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.744 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.744 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.743579+0000", 2024-09-21T10:48:47.744 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4304e-05 2024-09-21T10:48:47.744 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.744 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.744 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:48:47.744 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.743587+0000", 2024-09-21T10:48:47.744 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4109999999999996e-06 2024-09-21T10:48:47.744 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.745 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.745 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.745 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694656+0000", 2024-09-21T10:48:47.745 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.95106875000000002 2024-09-21T10:48:47.745 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.745 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.745 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:47.745 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694666+0000", 2024-09-21T10:48:47.745 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7079999999999992e-06 2024-09-21T10:48:47.745 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.745 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:47.745 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.745 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.745 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.746 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:79939 53.12 53.4ae2b0d2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1329)", 2024-09-21T10:48:47.746 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:09.754107+0000", 2024-09-21T10:48:47.746 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 457.96597275900001, 2024-09-21T10:48:47.746 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.94060030500000003, 2024-09-21T10:48:47.746 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:47.746 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:48:47.746 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:47.746 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:47.746 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:47.746 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 79939 2024-09-21T10:48:47.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.746 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:47.746 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.747 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:47.747 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.754107+0000", 2024-09-21T10:48:47.747 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.747 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.747 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:47.747 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.754107+0000", 2024-09-21T10:48:47.747 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.747 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.747 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:47.747 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.754106+0000", 2024-09-21T10:48:47.747 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:48:47.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.748 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.748 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:47.748 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.754154+0000", 2024-09-21T10:48:47.748 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8631999999999998e-05 2024-09-21T10:48:47.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.748 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.748 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:47.748 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.754168+0000", 2024-09-21T10:48:47.748 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3719e-05 2024-09-21T10:48:47.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.748 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.748 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:47.748 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.754177+0000", 2024-09-21T10:48:47.748 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0089999999999996e-06 2024-09-21T10:48:47.749 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.749 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.749 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.749 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.754224+0000", 2024-09-21T10:48:47.749 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6765999999999999e-05 2024-09-21T10:48:47.749 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.749 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.749 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:48:47.749 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.754234+0000", 2024-09-21T10:48:47.749 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8800000000000003e-06 2024-09-21T10:48:47.749 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.749 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.749 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.749 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694691+0000", 2024-09-21T10:48:47.750 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.94045756899999999 2024-09-21T10:48:47.750 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.750 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.750 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:47.750 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694707+0000", 2024-09-21T10:48:47.750 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6049000000000002e-05 2024-09-21T10:48:47.750 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.750 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:47.750 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.750 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.750 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.750 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:79977 53.12 53.a7d3712 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1329)", 2024-09-21T10:48:47.750 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:41:09.777895+0000", 2024-09-21T10:48:47.750 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 457.94218527700002, 2024-09-21T10:48:47.751 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91684465900000001, 2024-09-21T10:48:47.751 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:47.751 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:48:47.751 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:47.751 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:47.751 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:47.751 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 79977 2024-09-21T10:48:47.751 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.751 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:47.751 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.751 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:47.751 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.777895+0000", 2024-09-21T10:48:47.751 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.752 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.752 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:47.752 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.777895+0000", 2024-09-21T10:48:47.752 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.752 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.752 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:47.752 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.777893+0000", 2024-09-21T10:48:47.752 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:48:47.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.752 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.752 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:47.753 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.777997+0000", 2024-09-21T10:48:47.753 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000103127 2024-09-21T10:48:47.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.753 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.753 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:47.753 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.778006+0000", 2024-09-21T10:48:47.753 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6500000000000008e-06 2024-09-21T10:48:47.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.753 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.753 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:47.753 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.778014+0000", 2024-09-21T10:48:47.753 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8169999999999999e-06 2024-09-21T10:48:47.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.753 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.754 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.754 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.778049+0000", 2024-09-21T10:48:47.754 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4878e-05 2024-09-21T10:48:47.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.754 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.754 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:48:47.754 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:09.778054+0000", 2024-09-21T10:48:47.754 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1070000000000004e-06 2024-09-21T10:48:47.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.754 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.754 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.754 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694730+0000", 2024-09-21T10:48:47.754 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91667635199999997 2024-09-21T10:48:47.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.755 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.755 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:47.755 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:41:10.694739+0000", 2024-09-21T10:48:47.755 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8859999999999993e-06 2024-09-21T10:48:47.755 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.755 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:47.755 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.755 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.755 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:238167 53.15 53.6bfa1675 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e1353)", 2024-09-21T10:48:47.755 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:43:41.186729+0000", 2024-09-21T10:48:47.755 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 306.53335076500002, 2024-09-21T10:48:47.755 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99012371300000002, 2024-09-21T10:48:47.756 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:47.756 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:48:47.756 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:47.756 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:47.756 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:47.756 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 238167 2024-09-21T10:48:47.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.756 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:47.756 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.756 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:47.756 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.186729+0000", 2024-09-21T10:48:47.756 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.756 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.757 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:47.757 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.186729+0000", 2024-09-21T10:48:47.757 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.757 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.757 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.757 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:47.757 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.186729+0000", 2024-09-21T10:48:47.757 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T10:48:47.757 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.757 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.757 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:47.757 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.186732+0000", 2024-09-21T10:48:47.757 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6610000000000002e-06 2024-09-21T10:48:47.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.758 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:47.758 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.186750+0000", 2024-09-21T10:48:47.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7771999999999999e-05 2024-09-21T10:48:47.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.758 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:47.758 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.186763+0000", 2024-09-21T10:48:47.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3217000000000001e-05 2024-09-21T10:48:47.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.758 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.186817+0000", 2024-09-21T10:48:47.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3813000000000001e-05 2024-09-21T10:48:47.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:48:47.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.186825+0000", 2024-09-21T10:48:47.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1359999999999997e-06 2024-09-21T10:48:47.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176845+0000", 2024-09-21T10:48:47.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.990019749 2024-09-21T10:48:47.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.760 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:47.760 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176853+0000", 2024-09-21T10:48:47.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.875e-06 2024-09-21T10:48:47.760 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.760 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:47.760 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.760 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:238221 53.15 53.dc332215 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1353)", 2024-09-21T10:48:47.760 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:43:41.214446+0000", 2024-09-21T10:48:47.760 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 306.50563407300001, 2024-09-21T10:48:47.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96244874199999997, 2024-09-21T10:48:47.760 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:47.760 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:48:47.761 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:47.761 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:47.761 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:47.761 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 238221 2024-09-21T10:48:47.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.761 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:47.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.761 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:47.761 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.214446+0000", 2024-09-21T10:48:47.761 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.761 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:47.761 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.214446+0000", 2024-09-21T10:48:47.762 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.762 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.762 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:47.762 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.214445+0000", 2024-09-21T10:48:47.762 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:48:47.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.762 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.762 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:47.762 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.214450+0000", 2024-09-21T10:48:47.762 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4410000000000001e-06 2024-09-21T10:48:47.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.762 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.762 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:47.763 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.214462+0000", 2024-09-21T10:48:47.763 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1854e-05 2024-09-21T10:48:47.763 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.763 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.763 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:47.763 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.214468+0000", 2024-09-21T10:48:47.763 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1990000000000003e-06 2024-09-21T10:48:47.763 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.763 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.763 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.763 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.214518+0000", 2024-09-21T10:48:47.763 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9877999999999999e-05 2024-09-21T10:48:47.763 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.763 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.764 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:48:47.764 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.214527+0000", 2024-09-21T10:48:47.764 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.439e-06 2024-09-21T10:48:47.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.764 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.764 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.764 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176884+0000", 2024-09-21T10:48:47.764 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96235778699999996 2024-09-21T10:48:47.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.764 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.764 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:47.764 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176895+0000", 2024-09-21T10:48:47.764 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0207000000000001e-05 2024-09-21T10:48:47.764 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.765 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:47.765 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.765 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.765 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:238230 53.15 53.8764f335 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1353)", 2024-09-21T10:48:47.765 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:43:41.221236+0000", 2024-09-21T10:48:47.765 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 306.49884359599997, 2024-09-21T10:48:47.765 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.955676635, 2024-09-21T10:48:47.765 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:47.765 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:48:47.765 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:47.765 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:47.765 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:47.765 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 238230 2024-09-21T10:48:47.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.766 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:47.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.766 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:47.766 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.221236+0000", 2024-09-21T10:48:47.766 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.766 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:47.766 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.221236+0000", 2024-09-21T10:48:47.766 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.766 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:47.767 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.221235+0000", 2024-09-21T10:48:47.767 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:48:47.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.767 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:47.767 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.221241+0000", 2024-09-21T10:48:47.767 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1929999999999998e-06 2024-09-21T10:48:47.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.767 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:47.767 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.221258+0000", 2024-09-21T10:48:47.767 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6765e-05 2024-09-21T10:48:47.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.768 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:47.768 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.221265+0000", 2024-09-21T10:48:47.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3560000000000004e-06 2024-09-21T10:48:47.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.768 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.768 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.221314+0000", 2024-09-21T10:48:47.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9002999999999998e-05 2024-09-21T10:48:47.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.768 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:48:47.768 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.221323+0000", 2024-09-21T10:48:47.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5399999999999996e-06 2024-09-21T10:48:47.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.769 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.769 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.769 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176908+0000", 2024-09-21T10:48:47.769 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.95558458300000004 2024-09-21T10:48:47.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.769 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.769 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:47.769 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176913+0000", 2024-09-21T10:48:47.769 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2920000000000003e-06 2024-09-21T10:48:47.769 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.769 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:47.769 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.769 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.770 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:238249 53.15 53.96c6175 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1353)", 2024-09-21T10:48:47.770 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:43:41.237158+0000", 2024-09-21T10:48:47.770 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 306.48292146599999, 2024-09-21T10:48:47.770 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.93976896499999996, 2024-09-21T10:48:47.770 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:47.770 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:48:47.770 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:47.770 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:47.770 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:47.770 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 238249 2024-09-21T10:48:47.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.770 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:47.770 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.770 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:47.771 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.237158+0000", 2024-09-21T10:48:47.771 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.771 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.771 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:47.771 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.237158+0000", 2024-09-21T10:48:47.771 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.771 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.771 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:47.771 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.237157+0000", 2024-09-21T10:48:47.771 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:48:47.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.771 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.772 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:47.772 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.237163+0000", 2024-09-21T10:48:47.772 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1199999999999999e-06 2024-09-21T10:48:47.772 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.772 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.772 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:47.772 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.237174+0000", 2024-09-21T10:48:47.772 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0957e-05 2024-09-21T10:48:47.772 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.772 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.772 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:47.772 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.237180+0000", 2024-09-21T10:48:47.772 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3330000000000002e-06 2024-09-21T10:48:47.772 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.773 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.773 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.237224+0000", 2024-09-21T10:48:47.773 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3835999999999998e-05 2024-09-21T10:48:47.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.773 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:48:47.773 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.237232+0000", 2024-09-21T10:48:47.773 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8599999999999993e-06 2024-09-21T10:48:47.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.773 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.773 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176922+0000", 2024-09-21T10:48:47.773 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.93969054600000002 2024-09-21T10:48:47.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.774 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.774 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:47.774 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176927+0000", 2024-09-21T10:48:47.774 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9400000000000001e-06 2024-09-21T10:48:47.774 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.774 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:47.774 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.774 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.774 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:238253 53.15 53.c6d9c815 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1353)", 2024-09-21T10:48:47.774 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:43:41.239774+0000", 2024-09-21T10:48:47.774 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 306.48030533899998, 2024-09-21T10:48:47.775 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.93717435100000002, 2024-09-21T10:48:47.775 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:47.775 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:48:47.775 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:47.775 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:47.775 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:47.775 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 238253 2024-09-21T10:48:47.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.775 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:47.775 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.775 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:47.775 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.239774+0000", 2024-09-21T10:48:47.775 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.776 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.776 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:47.776 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.239774+0000", 2024-09-21T10:48:47.776 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.776 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.776 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.776 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:47.776 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.239773+0000", 2024-09-21T10:48:47.776 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:48:47.776 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.776 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.776 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:47.776 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.239780+0000", 2024-09-21T10:48:47.776 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4749999999999998e-06 2024-09-21T10:48:47.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.777 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:47.777 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.239796+0000", 2024-09-21T10:48:47.777 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6013000000000001e-05 2024-09-21T10:48:47.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.777 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:47.777 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.239803+0000", 2024-09-21T10:48:47.777 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9700000000000002e-06 2024-09-21T10:48:47.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.777 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.777 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.239882+0000", 2024-09-21T10:48:47.778 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9036999999999996e-05 2024-09-21T10:48:47.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.778 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.778 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:48:47.778 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.239890+0000", 2024-09-21T10:48:47.778 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2169999999999994e-06 2024-09-21T10:48:47.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.778 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.778 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.778 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176944+0000", 2024-09-21T10:48:47.778 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.93705439499999998 2024-09-21T10:48:47.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.778 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.779 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:47.779 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176949+0000", 2024-09-21T10:48:47.779 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.673e-06 2024-09-21T10:48:47.779 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.779 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:47.779 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.779 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.779 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.779 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:238271 53.15 53.17959e15 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1353)", 2024-09-21T10:48:47.779 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:43:41.253997+0000", 2024-09-21T10:48:47.779 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 306.46608240699999, 2024-09-21T10:48:47.779 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.92298151900000003, 2024-09-21T10:48:47.779 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:47.780 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:48:47.780 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:47.780 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:47.780 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:47.780 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 238271 2024-09-21T10:48:47.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.780 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:47.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.780 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:47.780 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.253997+0000", 2024-09-21T10:48:47.780 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.780 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:47.781 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.253997+0000", 2024-09-21T10:48:47.781 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.781 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:47.781 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.253996+0000", 2024-09-21T10:48:47.781 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:48:47.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.781 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:47.781 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.254003+0000", 2024-09-21T10:48:47.781 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6989999999999997e-06 2024-09-21T10:48:47.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.782 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:47.782 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.254015+0000", 2024-09-21T10:48:47.782 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2148e-05 2024-09-21T10:48:47.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.782 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:47.782 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.254022+0000", 2024-09-21T10:48:47.782 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.374e-06 2024-09-21T10:48:47.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.782 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.782 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.254065+0000", 2024-09-21T10:48:47.782 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2601000000000001e-05 2024-09-21T10:48:47.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.783 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.783 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:48:47.783 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.254073+0000", 2024-09-21T10:48:47.783 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3990000000000004e-06 2024-09-21T10:48:47.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.783 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.783 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.783 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176971+0000", 2024-09-21T10:48:47.783 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.92289776899999998 2024-09-21T10:48:47.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.783 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.783 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:47.783 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176979+0000", 2024-09-21T10:48:47.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0590000000000003e-06 2024-09-21T10:48:47.784 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.784 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:47.784 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.784 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:238279 53.15 53.da97f315 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1353)", 2024-09-21T10:48:47.784 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:43:41.257274+0000", 2024-09-21T10:48:47.784 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 306.46280564199998, 2024-09-21T10:48:47.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91972421199999999, 2024-09-21T10:48:47.784 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:47.784 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:48:47.784 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:47.785 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:47.785 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:47.785 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 238279 2024-09-21T10:48:47.785 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.785 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:47.785 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.785 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:47.785 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.257274+0000", 2024-09-21T10:48:47.785 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.785 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.785 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.785 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:47.785 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.257274+0000", 2024-09-21T10:48:47.786 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.786 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.786 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.786 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:47.786 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.257273+0000", 2024-09-21T10:48:47.786 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:48:47.786 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.786 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.786 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:47.786 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.257281+0000", 2024-09-21T10:48:47.786 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8320000000000006e-06 2024-09-21T10:48:47.786 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.786 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.787 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:47.787 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.257293+0000", 2024-09-21T10:48:47.787 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2615e-05 2024-09-21T10:48:47.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.787 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.787 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:47.787 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.257301+0000", 2024-09-21T10:48:47.787 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7389999999999999e-06 2024-09-21T10:48:47.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.787 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.787 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.787 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.258358+0000", 2024-09-21T10:48:47.787 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001057379 2024-09-21T10:48:47.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.788 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:48:47.788 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.258367+0000", 2024-09-21T10:48:47.788 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6750000000000008e-06 2024-09-21T10:48:47.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.788 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.788 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176991+0000", 2024-09-21T10:48:47.788 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91862456299999995 2024-09-21T10:48:47.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.788 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:47.788 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176998+0000", 2024-09-21T10:48:47.789 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9280000000000004e-06 2024-09-21T10:48:47.789 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.789 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:47.789 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.789 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.789 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.789 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:238285 53.15 53.412cc935 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1353)", 2024-09-21T10:48:47.789 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:43:41.262119+0000", 2024-09-21T10:48:47.789 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 306.45796058100001, 2024-09-21T10:48:47.789 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91490445300000001, 2024-09-21T10:48:47.789 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:47.789 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:48:47.789 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:47.789 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:47.790 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:47.790 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 238285 2024-09-21T10:48:47.790 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.790 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:47.790 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.790 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:47.790 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.262119+0000", 2024-09-21T10:48:47.790 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.790 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.790 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.790 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:47.790 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.262119+0000", 2024-09-21T10:48:47.790 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.791 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.791 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.791 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:47.791 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.262118+0000", 2024-09-21T10:48:47.791 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:48:47.791 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.791 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.791 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:47.791 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.262124+0000", 2024-09-21T10:48:47.791 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6690000000000003e-06 2024-09-21T10:48:47.791 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.791 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.791 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:47.791 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.262139+0000", 2024-09-21T10:48:47.792 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5252999999999999e-05 2024-09-21T10:48:47.792 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.792 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.792 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:47.792 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.262145+0000", 2024-09-21T10:48:47.792 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5010000000000001e-06 2024-09-21T10:48:47.792 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.792 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.792 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.792 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.262187+0000", 2024-09-21T10:48:47.792 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1896e-05 2024-09-21T10:48:47.792 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.792 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.792 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:48:47.793 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.262195+0000", 2024-09-21T10:48:47.793 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3150000000000003e-06 2024-09-21T10:48:47.793 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.793 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.793 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.793 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.177019+0000", 2024-09-21T10:48:47.793 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91482411299999999 2024-09-21T10:48:47.793 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.793 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.793 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:47.793 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.177024+0000", 2024-09-21T10:48:47.793 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9969999999999998e-06 2024-09-21T10:48:47.793 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.793 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:47.794 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.794 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.794 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:238289 53.15 53.9d0cb415 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1353)", 2024-09-21T10:48:47.794 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:43:41.267953+0000", 2024-09-21T10:48:47.794 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 306.452126728, 2024-09-21T10:48:47.794 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.90909275499999997, 2024-09-21T10:48:47.794 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:47.794 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:48:47.794 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:47.794 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:47.794 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:47.794 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 238289 2024-09-21T10:48:47.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.795 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:47.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.795 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:47.795 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.267953+0000", 2024-09-21T10:48:47.795 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.795 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:47.795 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.267953+0000", 2024-09-21T10:48:47.795 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.795 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:47.795 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.267952+0000", 2024-09-21T10:48:47.796 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:48:47.796 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.796 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.796 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:47.796 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.267959+0000", 2024-09-21T10:48:47.796 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7560000000000002e-06 2024-09-21T10:48:47.796 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.796 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.796 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:47.796 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.267969+0000", 2024-09-21T10:48:47.796 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.978e-06 2024-09-21T10:48:47.796 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.796 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.796 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:47.797 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.267976+0000", 2024-09-21T10:48:47.798 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2119999999999997e-06 2024-09-21T10:48:47.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.798 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.798 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.798 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.268736+0000", 2024-09-21T10:48:47.798 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00076052400000000005 2024-09-21T10:48:47.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.798 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.798 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:48:47.799 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.268743+0000", 2024-09-21T10:48:47.799 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1409999999999999e-06 2024-09-21T10:48:47.799 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.799 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.799 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.799 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.177041+0000", 2024-09-21T10:48:47.799 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.90829757300000002 2024-09-21T10:48:47.799 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.799 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.799 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:47.799 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.177046+0000", 2024-09-21T10:48:47.799 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8550000000000001e-06 2024-09-21T10:48:47.799 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.799 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:47.800 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.800 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.800 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:238293 53.15 53.c8cb59d5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1353)", 2024-09-21T10:48:47.800 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:43:41.269950+0000", 2024-09-21T10:48:47.800 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 306.45013022900002, 2024-09-21T10:48:47.800 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.9071169, 2024-09-21T10:48:47.800 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:47.800 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:48:47.800 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:47.800 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:47.800 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:47.800 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 238293 2024-09-21T10:48:47.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.801 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:47.801 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.801 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:47.801 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.269950+0000", 2024-09-21T10:48:47.801 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.801 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.801 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.801 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:47.801 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.269950+0000", 2024-09-21T10:48:47.801 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.801 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.801 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.801 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:47.801 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.269949+0000", 2024-09-21T10:48:47.802 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:48:47.802 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.802 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.802 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:47.802 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.269954+0000", 2024-09-21T10:48:47.802 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1669999999999998e-06 2024-09-21T10:48:47.802 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.802 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.802 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:47.802 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.269966+0000", 2024-09-21T10:48:47.802 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2048999999999999e-05 2024-09-21T10:48:47.802 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.802 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.803 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:47.803 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.269972+0000", 2024-09-21T10:48:47.803 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9240000000000004e-06 2024-09-21T10:48:47.803 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.803 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.803 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.803 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.270017+0000", 2024-09-21T10:48:47.803 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5034999999999998e-05 2024-09-21T10:48:47.803 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.803 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.803 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:48:47.803 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.270026+0000", 2024-09-21T10:48:47.803 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8910000000000001e-06 2024-09-21T10:48:47.803 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.804 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.804 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.804 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.177062+0000", 2024-09-21T10:48:47.804 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.90703622100000003 2024-09-21T10:48:47.804 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.804 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.804 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:47.804 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.177067+0000", 2024-09-21T10:48:47.804 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6539999999999998e-06 2024-09-21T10:48:47.804 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.804 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:47.804 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.804 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.804 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.805 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:238312 53.15 53.85eedcf5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1353)", 2024-09-21T10:48:47.805 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:43:41.288595+0000", 2024-09-21T10:48:47.805 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 306.43148443600001, 2024-09-21T10:48:47.805 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88849270400000002, 2024-09-21T10:48:47.805 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:47.805 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:48:47.805 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:47.805 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:47.805 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:47.805 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 238312 2024-09-21T10:48:47.805 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.805 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:47.805 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.805 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:47.806 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.288595+0000", 2024-09-21T10:48:47.806 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.806 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.806 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.806 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:47.806 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.288595+0000", 2024-09-21T10:48:47.806 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.806 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.806 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.806 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:47.806 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.288594+0000", 2024-09-21T10:48:47.806 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:48:47.806 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.807 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.807 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:47.807 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.288602+0000", 2024-09-21T10:48:47.807 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1289999999999996e-06 2024-09-21T10:48:47.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.807 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.807 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:47.807 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.288620+0000", 2024-09-21T10:48:47.807 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8495000000000002e-05 2024-09-21T10:48:47.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.807 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.807 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:47.807 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.288628+0000", 2024-09-21T10:48:47.807 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8189999999999992e-06 2024-09-21T10:48:47.808 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.808 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.808 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.808 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.288681+0000", 2024-09-21T10:48:47.808 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3013999999999998e-05 2024-09-21T10:48:47.808 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.808 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.808 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:48:47.808 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.288692+0000", 2024-09-21T10:48:47.808 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0366999999999999e-05 2024-09-21T10:48:47.808 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.808 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.808 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.808 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.177082+0000", 2024-09-21T10:48:47.809 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88839072100000005 2024-09-21T10:48:47.809 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.809 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.809 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:47.809 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.177088+0000", 2024-09-21T10:48:47.809 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8030000000000002e-06 2024-09-21T10:48:47.809 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.809 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:47.809 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.809 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.809 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.809 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:238337 53.15 53.2cf18455 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1353)", 2024-09-21T10:48:47.809 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:43:41.312887+0000", 2024-09-21T10:48:47.809 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 306.40719295700001, 2024-09-21T10:48:47.809 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.86421528199999997, 2024-09-21T10:48:47.810 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:47.810 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:48:47.810 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:47.810 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:47.810 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:47.810 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 238337 2024-09-21T10:48:47.810 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.810 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:47.810 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.810 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:47.810 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.312887+0000", 2024-09-21T10:48:47.810 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.811 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.811 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.811 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:47.811 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.312887+0000", 2024-09-21T10:48:47.811 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.811 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.811 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.811 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:47.811 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.312886+0000", 2024-09-21T10:48:47.811 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:48:47.811 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.811 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.811 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:47.811 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.312891+0000", 2024-09-21T10:48:47.812 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2190000000000003e-06 2024-09-21T10:48:47.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.812 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.812 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:47.812 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.312902+0000", 2024-09-21T10:48:47.812 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1457000000000001e-05 2024-09-21T10:48:47.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.812 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.812 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:47.812 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.312910+0000", 2024-09-21T10:48:47.812 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4259999999999997e-06 2024-09-21T10:48:47.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.812 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.813 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.813 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.312954+0000", 2024-09-21T10:48:47.813 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4648999999999999e-05 2024-09-21T10:48:47.813 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.813 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.813 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:48:47.813 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.312964+0000", 2024-09-21T10:48:47.813 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2739999999999996e-06 2024-09-21T10:48:47.813 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.813 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.813 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.813 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.177098+0000", 2024-09-21T10:48:47.813 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.86413380399999995 2024-09-21T10:48:47.813 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.814 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.814 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:47.814 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.177102+0000", 2024-09-21T10:48:47.814 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6160000000000004e-06 2024-09-21T10:48:47.814 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.814 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:47.814 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.814 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.814 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.814 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:238347 53.15 53.bc16aa75 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1353)", 2024-09-21T10:48:47.814 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:43:41.322981+0000", 2024-09-21T10:48:47.814 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 306.39709924499999, 2024-09-21T10:48:47.814 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.85414257199999999, 2024-09-21T10:48:47.815 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:47.815 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:48:47.815 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:47.815 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:47.815 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:47.815 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 238347 2024-09-21T10:48:47.815 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.815 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:47.815 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.815 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:47.815 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.322981+0000", 2024-09-21T10:48:47.815 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.815 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.816 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.816 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:47.816 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.322981+0000", 2024-09-21T10:48:47.816 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.816 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.816 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.816 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:47.816 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.322980+0000", 2024-09-21T10:48:47.816 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:48:47.816 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.816 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.816 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:47.816 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.322984+0000", 2024-09-21T10:48:47.817 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7249999999999997e-06 2024-09-21T10:48:47.817 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.817 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.817 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:47.817 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.322997+0000", 2024-09-21T10:48:47.817 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2335e-05 2024-09-21T10:48:47.817 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.817 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.817 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:47.817 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.323005+0000", 2024-09-21T10:48:47.817 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8949999999999998e-06 2024-09-21T10:48:47.817 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.817 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.817 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.818 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.323042+0000", 2024-09-21T10:48:47.818 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7574999999999999e-05 2024-09-21T10:48:47.818 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.818 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.818 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:48:47.818 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.323047+0000", 2024-09-21T10:48:47.818 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0130000000000001e-06 2024-09-21T10:48:47.818 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.818 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.818 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.818 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.177118+0000", 2024-09-21T10:48:47.818 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.85407121799999997 2024-09-21T10:48:47.819 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.819 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.819 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:47.819 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.177123+0000", 2024-09-21T10:48:47.819 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7589999999999997e-06 2024-09-21T10:48:47.819 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.819 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:47.819 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.819 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.819 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.819 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:238372 53.15 53.85e31d35 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1353)", 2024-09-21T10:48:47.819 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:43:41.357557+0000", 2024-09-21T10:48:47.819 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 306.36252329299998, 2024-09-21T10:48:47.820 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.81958719899999999, 2024-09-21T10:48:47.820 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:48:47.820 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:48:47.820 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:48:47.820 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:48:47.820 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:48:47.820 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 238372 2024-09-21T10:48:47.820 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.820 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:48:47.820 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.820 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:48:47.820 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.357557+0000", 2024-09-21T10:48:47.820 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.820 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.821 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.821 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:48:47.821 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.357557+0000", 2024-09-21T10:48:47.821 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:48:47.821 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.821 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.821 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:48:47.821 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.357556+0000", 2024-09-21T10:48:47.821 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:48:47.821 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.821 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.821 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:48:47.821 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.357560+0000", 2024-09-21T10:48:47.822 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.493e-06 2024-09-21T10:48:47.822 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.822 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.822 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:48:47.822 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.357569+0000", 2024-09-21T10:48:47.822 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2299999999999997e-06 2024-09-21T10:48:47.822 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.822 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.822 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:48:47.822 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.357575+0000", 2024-09-21T10:48:47.822 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6389999999999997e-06 2024-09-21T10:48:47.822 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.822 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.823 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.823 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.357611+0000", 2024-09-21T10:48:47.823 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5800000000000003e-05 2024-09-21T10:48:47.823 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.823 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.823 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:48:47.823 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.357616+0000", 2024-09-21T10:48:47.823 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.366e-06 2024-09-21T10:48:47.823 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.823 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.823 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:48:47.823 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.177139+0000", 2024-09-21T10:48:47.823 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.81952290500000002 2024-09-21T10:48:47.823 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:48:47.824 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:48:47.824 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:48:47.824 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.177144+0000", 2024-09-21T10:48:47.824 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5349999999999998e-06 2024-09-21T10:48:47.824 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.824 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:47.824 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.824 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:48:47.824 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:48:47.824 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:48:47.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:47.825 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:47.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:47.906 INFO:teuthology.orchestra.run.smithi070.stderr:set osd.5 primary-affinity to 0.341766 (8223972) 2024-09-21T10:48:47.926 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:47.948 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:47.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:47.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:48.050 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:48.071 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:48.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:48.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:47 smithi160 ceph-mon[101879]: pgmap v2504: 73 pgs: 73 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:48:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:47 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:48:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:47 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:48:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:47 smithi160 ceph-mon[101879]: osdmap e1408: 8 total, 8 up, 7 in 2024-09-21T10:48:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:47 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1059569726' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.3417656632934005} : dispatch 2024-09-21T10:48:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:47 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.3417656632934005} : dispatch 2024-09-21T10:48:48.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:47 smithi070 ceph-mon[112139]: pgmap v2504: 73 pgs: 73 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:48:48.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:47 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:48:48.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:47 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:48:48.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:47 smithi070 ceph-mon[112139]: osdmap e1408: 8 total, 8 up, 7 in 2024-09-21T10:48:48.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:47 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1059569726' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.3417656632934005} : dispatch 2024-09-21T10:48:48.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:47 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.3417656632934005} : dispatch 2024-09-21T10:48:48.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:47 smithi070 ceph-mon[107385]: pgmap v2504: 73 pgs: 73 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:48:48.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:47 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:48:48.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:47 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:48:48.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:47 smithi070 ceph-mon[107385]: osdmap e1408: 8 total, 8 up, 7 in 2024-09-21T10:48:48.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:47 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1059569726' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.3417656632934005} : dispatch 2024-09-21T10:48:48.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:47 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.3417656632934005} : dispatch 2024-09-21T10:48:48.172 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:48.194 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:48.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:48.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:48.296 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:48.317 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:48.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:48.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:48.419 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:48.440 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:48.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:48.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:48.542 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:48.564 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:48.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:48.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:48.641 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:48:48.666 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:48.687 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:48.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:48.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:48.788 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:48.810 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:48.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:48.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:48.881 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:48:48.908 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:48 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.3417656632934005}]': finished 2024-09-21T10:48:48.908 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:48 smithi070 ceph-mon[107385]: osdmap e1409: 8 total, 8 up, 7 in 2024-09-21T10:48:48.909 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:48 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.3417656632934005}]': finished 2024-09-21T10:48:48.909 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:48 smithi070 ceph-mon[112139]: osdmap e1409: 8 total, 8 up, 7 in 2024-09-21T10:48:48.912 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:48.916 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:48 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.3417656632934005}]': finished 2024-09-21T10:48:48.916 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:48 smithi160 ceph-mon[101879]: osdmap e1409: 8 total, 8 up, 7 in 2024-09-21T10:48:48.934 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:48.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:48.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:49.036 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:49.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:49.058 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:49.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:49.159 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:49.181 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:49.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:49.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:49.282 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:49.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:49.305 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:49.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:49.406 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:49.429 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:49.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:49.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:49.531 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:49.553 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:49.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:49.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:49.654 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:49.677 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:49.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:49.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:49.779 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:49.800 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:49.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:49.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:49.902 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:49.923 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:49.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:49.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:50.025 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:50.042 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:49 smithi160 ceph-mon[101879]: pgmap v2507: 73 pgs: 73 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:48:50.043 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:49 smithi160 ceph-mon[101879]: osdmap e1410: 8 total, 8 up, 7 in 2024-09-21T10:48:50.047 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:50.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:50.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:50.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:49 smithi070 ceph-mon[112139]: pgmap v2507: 73 pgs: 73 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:48:50.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:49 smithi070 ceph-mon[112139]: osdmap e1410: 8 total, 8 up, 7 in 2024-09-21T10:48:50.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:49 smithi070 ceph-mon[107385]: pgmap v2507: 73 pgs: 73 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:48:50.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:49 smithi070 ceph-mon[107385]: osdmap e1410: 8 total, 8 up, 7 in 2024-09-21T10:48:50.148 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:50.174 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:50.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:50.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:50.276 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:50.299 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:50.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:50.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:50.400 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:50.422 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:50.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:50.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:50.524 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:50.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:50.546 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:50.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:50.647 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:50.663 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:48:50.663 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T10:48:50.663 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T10:48:50.663 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:48:50.669 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:50.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:50.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:50.770 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:50.792 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:50.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:50.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:50.893 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:50.919 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:50.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:50.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:50.937 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:48:50.937 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:48:51.020 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:51.042 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:51.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:51.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:51.143 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:48:51.144 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:51.167 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:51.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:51.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:51.233 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:48:51.253 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:48:51.268 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:51.318 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:51.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:51.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:51.419 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:51.433 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:48:51.434 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:48:51.434 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:48:51.434 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:48:51.439 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:48:51.439 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:48:51.440 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:51.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:51.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:51.451 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:48:51.456 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:48:51.456 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:48:51.472 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:48:51.472 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:48:51.488 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:48:51.488 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:48:51.504 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:48:51.504 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:48:51.520 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:48:51.520 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:48:51.542 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:51.564 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:51.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:51.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:51.666 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:51.687 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:51.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:51.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:51.789 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:51.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:51.815 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:51.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:51.916 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:51.921 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'unique_pool_2' created 2024-09-21T10:48:51.941 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:51.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:51.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:52.031 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:51 smithi070 ceph-mon[112139]: pgmap v2509: 73 pgs: 73 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:48:52.031 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:51 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/789265812' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:48:52.031 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:51 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2237824025' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-21T10:48:52.032 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:51 smithi070 ceph-mon[107385]: pgmap v2509: 73 pgs: 73 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:48:52.032 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:51 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/789265812' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:48:52.032 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:51 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2237824025' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-21T10:48:52.043 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:52.067 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:52.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:52.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:52.169 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:52.192 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:52.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:52.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:52.295 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:52.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:52.321 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:52.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:52.324 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:51 smithi160 ceph-mon[101879]: pgmap v2509: 73 pgs: 73 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:48:52.324 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:51 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/789265812' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:48:52.324 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:51 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2237824025' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-21T10:48:52.422 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:52.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:52.448 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:52.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:52.549 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:52.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:52.574 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:52.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:52.675 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:52.698 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:52.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:52.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:52.734 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:52.800 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:52.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:52.823 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:52.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:52.924 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:52.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:52.950 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:52.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:53.051 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:53.073 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:53.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:53.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:53.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:52 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2237824025' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-21T10:48:53.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:52 smithi070 ceph-mon[107385]: osdmap e1411: 8 total, 8 up, 7 in 2024-09-21T10:48:53.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:52 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2237824025' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-21T10:48:53.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:52 smithi070 ceph-mon[112139]: osdmap e1411: 8 total, 8 up, 7 in 2024-09-21T10:48:53.175 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:53.189 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:52 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2237824025' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-21T10:48:53.189 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:52 smithi160 ceph-mon[101879]: osdmap e1411: 8 total, 8 up, 7 in 2024-09-21T10:48:53.200 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:53.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:53.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:53.303 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:53.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:53.327 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:53.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:53.428 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:53.432 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:48:53.432 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:48:53.452 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:53.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:53.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:53.553 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:53.575 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:53.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:53.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:53.597 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:48:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:48:53] "GET /metrics HTTP/1.1" 200 34190 "" "Prometheus/2.51.0" 2024-09-21T10:48:53.663 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:48:53.663 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:48:53.677 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:53.697 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:53.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:53.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:53.799 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:53.820 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:53.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:53.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:53.922 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:53.942 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-09-21T10:48:53.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:53.947 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:53.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:54.049 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:54.070 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:54.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:54.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:54.076 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:53 smithi160 ceph-mon[101879]: pgmap v2511: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:48:54.076 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:53 smithi160 ceph-mon[101879]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:48:54.076 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:53 smithi160 ceph-mon[101879]: osdmap e1412: 8 total, 8 up, 7 in 2024-09-21T10:48:54.097 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:48:54.172 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:54.193 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:54.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:54.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:54.295 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:54.309 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:53 smithi070 ceph-mon[107385]: pgmap v2511: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:48:54.309 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:53 smithi070 ceph-mon[107385]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:48:54.309 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:53 smithi070 ceph-mon[107385]: osdmap e1412: 8 total, 8 up, 7 in 2024-09-21T10:48:54.309 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:53 smithi070 ceph-mon[112139]: pgmap v2511: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:48:54.309 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:53 smithi070 ceph-mon[112139]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:48:54.309 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:53 smithi070 ceph-mon[112139]: osdmap e1412: 8 total, 8 up, 7 in 2024-09-21T10:48:54.309 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:48:54.317 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:54.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:54.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:54.376 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:54.418 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:54.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:54.441 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:54.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:54.474 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:48:54.541 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:54.563 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:54.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:54.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:54.665 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:54.687 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:54.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:54.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:54.789 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:54.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:54.811 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:54.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:54.912 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:54.936 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:54.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:54.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:55.038 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:55.059 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:55.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:55.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:55.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:54 smithi160 ceph-mon[101879]: osdmap e1413: 8 total, 8 up, 7 in 2024-09-21T10:48:55.161 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:55.182 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:55.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:55.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:55.265 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:54 smithi070 ceph-mon[107385]: osdmap e1413: 8 total, 8 up, 7 in 2024-09-21T10:48:55.265 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:54 smithi070 ceph-mon[112139]: osdmap e1413: 8 total, 8 up, 7 in 2024-09-21T10:48:55.284 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:55.307 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:55.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:55.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:55.408 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:55.430 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:55.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:55.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:55.532 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:55.553 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:55.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:55.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:55.655 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:55.676 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:55.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:55.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:55.778 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:55.800 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:55.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:55.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:55.902 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:55.923 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:55.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:55.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:55.945 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T10:48:55.991 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:48:55.991 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:48:55.992 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:55 smithi160 ceph-mon[101879]: pgmap v2514: 89 pgs: 14 unknown, 75 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:48:55.992 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:55 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1155118879' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:48:55.992 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:55 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:48:56.025 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:56.046 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:56.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:56.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:56.148 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:56.170 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:56.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:56.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:56.272 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:56.293 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:56.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:56.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:56.310 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:55 smithi070 ceph-mon[107385]: pgmap v2514: 89 pgs: 14 unknown, 75 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:48:56.310 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:55 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1155118879' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:48:56.310 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:55 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:48:56.311 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:55 smithi070 ceph-mon[112139]: pgmap v2514: 89 pgs: 14 unknown, 75 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:48:56.311 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:55 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1155118879' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:48:56.311 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:55 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:48:56.394 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:56.417 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:56.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:56.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:56.518 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:56.541 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:56.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:56.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:56.642 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:56.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:56.664 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:56.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:56.736 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:48:56.765 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:56.800 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:56.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:56.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:56.902 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:56.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:56.925 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:56.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:57.026 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:57.047 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:57.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:57.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:57.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:56 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:48:57.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:56 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:48:57.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:56 smithi160 ceph-mon[101879]: osdmap e1414: 8 total, 8 up, 7 in 2024-09-21T10:48:57.150 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:57.171 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:57.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:57.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:57.273 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:57.294 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:57.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:57.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:57.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:56 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:48:57.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:56 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:48:57.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:56 smithi070 ceph-mon[107385]: osdmap e1414: 8 total, 8 up, 7 in 2024-09-21T10:48:57.360 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:56 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:48:57.361 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:56 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:48:57.361 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:56 smithi070 ceph-mon[112139]: osdmap e1414: 8 total, 8 up, 7 in 2024-09-21T10:48:57.396 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:57.419 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:57.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:57.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:57.520 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:57.542 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:57.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:57.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:57.556 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:57.593 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:57.644 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:57.664 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:57.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:57.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:57.767 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:57.791 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:57.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:57.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:57.892 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:57.913 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:57.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:57.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:57.947 INFO:teuthology.orchestra.run.smithi070.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-09-21T10:48:58.015 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:58.036 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:58.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:58.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:58.138 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:58.159 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:58.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:58.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:58.261 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:58.281 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:57 smithi070 ceph-mon[107385]: pgmap v2516: 89 pgs: 89 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:48:58.281 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:57 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/52254319' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:48:58.282 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:57 smithi070 ceph-mon[112139]: pgmap v2516: 89 pgs: 89 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:48:58.282 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:57 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/52254319' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:48:58.285 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:58.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:58.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:58.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:57 smithi160 ceph-mon[101879]: pgmap v2516: 89 pgs: 89 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:48:58.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:57 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/52254319' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:48:58.387 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:58.410 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:58.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:58.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:58.512 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:58.534 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:58.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:58.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:58.636 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:58.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:58.662 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:58.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:58.763 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:58.787 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:58.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:58.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:58.889 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:58.913 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:58.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:58.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:58.989 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:48:59.015 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:59.055 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:59.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:59.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:59.157 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:59.183 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:59.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:59.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:59.285 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:59.310 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:59.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:59.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:59.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:58 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/52254319' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:48:59.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:58 smithi070 ceph-mon[112139]: osdmap e1415: 8 total, 8 up, 7 in 2024-09-21T10:48:59.322 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:58 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/52254319' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:48:59.323 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:58 smithi070 ceph-mon[107385]: osdmap e1415: 8 total, 8 up, 7 in 2024-09-21T10:48:59.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:58 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/52254319' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:48:59.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:58 smithi160 ceph-mon[101879]: osdmap e1415: 8 total, 8 up, 7 in 2024-09-21T10:48:59.413 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:59.434 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:59.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:59.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:59.535 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:59.562 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:59.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:59.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:59.663 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:59.684 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:59.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:59.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:59.786 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:59.814 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:48:59.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:59.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:59.916 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:59.937 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:48:59.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:59.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:00.039 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:00.065 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:00.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:00.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:00.107 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:00.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:59 smithi070 ceph-mon[107385]: pgmap v2518: 89 pgs: 89 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:49:00.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:59 smithi070 ceph-mon[107385]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:49:00.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:48:59 smithi070 ceph-mon[107385]: Cluster is now healthy 2024-09-21T10:49:00.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:59 smithi070 ceph-mon[112139]: pgmap v2518: 89 pgs: 89 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:49:00.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:59 smithi070 ceph-mon[112139]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:49:00.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:48:59 smithi070 ceph-mon[112139]: Cluster is now healthy 2024-09-21T10:49:00.167 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:00.191 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:00.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:00.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:00.293 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:00.317 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:00.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:00.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:00.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:59 smithi160 ceph-mon[101879]: pgmap v2518: 89 pgs: 89 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:49:00.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:59 smithi160 ceph-mon[101879]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:49:00.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:48:59 smithi160 ceph-mon[101879]: Cluster is now healthy 2024-09-21T10:49:00.420 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:00.441 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:00.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:00.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:00.542 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:00.568 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:00.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:00.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:00.670 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:00.692 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:00.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:00.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:00.727 DEBUG:teuthology.orchestra.run.smithi070:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-09-21T10:49:00.728 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-21T10:49:00.728 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:49:00.736 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T10:49:00.736 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T10:49:00.794 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:00.799 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:hints = 1 2024-09-21T10:49:00.799 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T10:49:00.799 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Object prefix: benchmark_data_smithi070_294394 2024-09-21T10:49:00.799 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:49:00.800 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 0 0 0 0 0 0 - 0 2024-09-21T10:49:00.831 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:00.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:00.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:00.933 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:00.969 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:00.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:00.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:01.071 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:01.113 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:01.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:01.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:01.214 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:01.252 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:01.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:01.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:01.353 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:01.387 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:01.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:01.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:01.488 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:01.523 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:01.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:01.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:01.625 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:01.662 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:01.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:01.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:01.754 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:49:01.754 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:49:01.764 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:01.791 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:49:01.791 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:49:01.799 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 1 16 3866 3850 240.612 240.625 0.00486783 0.00414411 2024-09-21T10:49:01.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:01.801 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:01.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:01.821 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:49:01.821 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:49:01.854 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:49:01.855 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:49:01.883 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:49:01.883 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:49:01.903 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:01.914 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:49:01.914 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:49:01.938 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:01.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:01.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:01.950 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:49:01.950 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:49:01.989 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:49:01.989 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:49:02.040 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:02.075 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:02.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:02.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:02.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:01 smithi070 ceph-mon[107385]: pgmap v2519: 89 pgs: 89 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:49:02.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:01 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3281283946' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:49:02.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:01 smithi070 ceph-mon[112139]: pgmap v2519: 89 pgs: 89 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:49:02.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:01 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3281283946' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:49:02.177 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:02.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:02.216 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:02.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:02.317 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:01 smithi160 ceph-mon[101879]: pgmap v2519: 89 pgs: 89 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:49:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:01 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3281283946' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:49:02.344 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:49:02.344 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T10:49:02.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:02.344 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:5817 55.8 55:1947d9a5:::benchmark_data_smithi070_294394_object5816:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1415)", 2024-09-21T10:49:02.345 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:49:02.340224+0000", 2024-09-21T10:49:02.345 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.001712611, 2024-09-21T10:49:02.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001749273, 2024-09-21T10:49:02.345 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:49:02.345 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:49:02.345 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:49:02.345 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:49:02.345 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:49:02.345 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:49:02.345 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 5817 2024-09-21T10:49:02.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:02.345 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:49:02.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:02.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:49:02.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:02.340224+0000", 2024-09-21T10:49:02.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:02.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:02.346 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:02.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:49:02.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:02.340224+0000", 2024-09-21T10:49:02.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:02.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:02.346 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:02.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:49:02.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:02.340226+0000", 2024-09-21T10:49:02.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7150000000000001e-06 2024-09-21T10:49:02.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:02.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:02.347 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:49:02.347 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:02.340249+0000", 2024-09-21T10:49:02.347 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3771e-05 2024-09-21T10:49:02.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:02.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:02.347 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:49:02.347 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:02.340267+0000", 2024-09-21T10:49:02.347 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7623999999999999e-05 2024-09-21T10:49:02.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:02.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:02.347 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:49:02.348 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:02.340276+0000", 2024-09-21T10:49:02.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7169999999999997e-06 2024-09-21T10:49:02.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:02.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:02.348 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:02.348 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:02.340474+0000", 2024-09-21T10:49:02.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000198438 2024-09-21T10:49:02.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:02.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:02.348 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:49:02.348 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:02.340687+0000", 2024-09-21T10:49:02.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00021305799999999999 2024-09-21T10:49:02.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:02.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:02.349 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:49:02.349 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:02.341053+0000", 2024-09-21T10:49:02.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00036562500000000001 2024-09-21T10:49:02.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:02.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:02.349 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:49:02.349 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:02.341766+0000", 2024-09-21T10:49:02.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00071357500000000004 2024-09-21T10:49:02.349 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:02.349 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:49:02.349 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:02.349 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:02.349 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-09-21T10:49:02.349 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 1 2024-09-21T10:49:02.349 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:49:02.352 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:02.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:02.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:02.454 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:02.485 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:02.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:02.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:02.587 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:02.625 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:02.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:02.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:02.727 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:02.758 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:02.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:02.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:02.800 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 2 16 7249 7233 226.014 211.438 0.00171015 0.00433213 2024-09-21T10:49:02.860 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:02.892 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:02.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:02.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:02.994 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:03.027 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:03.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:03.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:03.129 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:03.161 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:03.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:03.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:03.263 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:03.297 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:03.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:03.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:03.399 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:03.432 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:03.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:03.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:03.534 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:03.571 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:03.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:03.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:49:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:49:03] "GET /metrics HTTP/1.1" 200 34190 "" "Prometheus/2.51.0" 2024-09-21T10:49:03.672 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:03.708 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:03.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:03.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:03.800 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 3 16 10613 10597 220.754 210.25 0.00595825 0.0045235 2024-09-21T10:49:03.809 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:03.846 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:03.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:03.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:03.948 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:03.987 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:03.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:03.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:04.089 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:04.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:04.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:04.223 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:04.255 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:04.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:04.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:04.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:03 smithi160 ceph-mon[101879]: pgmap v2520: 89 pgs: 89 active+clean; 74 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 9.2 MiB/s wr, 147 op/s 2024-09-21T10:49:04.356 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:03 smithi070 ceph-mon[107385]: pgmap v2520: 89 pgs: 89 active+clean; 74 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 9.2 MiB/s wr, 147 op/s 2024-09-21T10:49:04.356 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:03 smithi070 ceph-mon[112139]: pgmap v2520: 89 pgs: 89 active+clean; 74 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 9.2 MiB/s wr, 147 op/s 2024-09-21T10:49:04.358 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:04.392 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:04.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:04.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:04.494 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:04.531 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:04.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:04.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:04.633 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:04.666 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:04.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:04.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:04.768 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:04.800 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 4 16 13685 13669 213.561 192 0.00307955 0.00467491 2024-09-21T10:49:04.803 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:04.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:04.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:04.905 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:04.939 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:04.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:04.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:04.941 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:49:04.960 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:05.042 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:05.076 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:05.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:05.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:05.178 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:05.215 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:05.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:05.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:05.317 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:05.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:05.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:05.455 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:05.493 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:05.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:05.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:05.594 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:05.618 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:05.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:05.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:05.720 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:05.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:05.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:05.800 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 5 16 16495 16479 205.971 175.625 0.00608421 0.00484768 2024-09-21T10:49:05.853 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:05.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:05.882 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:05.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:05.962 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:49:05.983 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:06.017 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:06.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:06.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:06.119 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:06.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:06.155 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:06.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:06.256 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:06.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:06.286 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:06.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:06.387 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:06.424 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:06.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:06.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:06.526 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:06.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:06.548 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:06.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:06 smithi070 ceph-mon[107385]: pgmap v2521: 89 pgs: 89 active+clean; 183 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 20 MiB/s wr, 327 op/s 2024-09-21T10:49:06.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:49:06.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:49:06.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:06 smithi070 ceph-mon[112139]: pgmap v2521: 89 pgs: 89 active+clean; 183 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 20 MiB/s wr, 327 op/s 2024-09-21T10:49:06.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:49:06.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:49:06.650 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:06.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:06.684 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:06.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:06.785 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:06.800 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 6 16 18814 18798 195.796 144.938 0.00607166 0.00510095 2024-09-21T10:49:06.819 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:06.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:06.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:06 smithi160 ceph-mon[101879]: pgmap v2521: 89 pgs: 89 active+clean; 183 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 20 MiB/s wr, 327 op/s 2024-09-21T10:49:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:49:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:49:06.922 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:06.954 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:06.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:06.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:07.056 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:07.087 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:07.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:07.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:07.188 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:07.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:07.225 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:07.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:07.326 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:07.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:07.358 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:07.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:07.388 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T10:49:07.459 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:07.489 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:07.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:07.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:07.590 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:07.623 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:07.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:07.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:07.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:07 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1193654245' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:49:07.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:07 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:49:07.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:07 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1193654245' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:49:07.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:07 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:49:07.725 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:07.754 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:07.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:07.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:07.800 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 7 16 21368 21352 190.627 159.625 0.00977837 0.00524001 2024-09-21T10:49:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:07 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1193654245' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:49:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:07 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:49:07.856 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:07.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:07.891 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:07.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:07.992 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:08.016 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:08.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:08.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:08.118 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:08.153 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:08.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:08.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:08.255 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:08.276 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:08.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:08.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:08.377 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:08.408 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:08.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:08.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:08.509 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:08.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:08.549 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:08.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:08.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:08 smithi070 ceph-mon[112139]: pgmap v2522: 89 pgs: 89 active+clean; 1015 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-21T10:49:08.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:08 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:49:08.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:08 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:49:08.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:08 smithi070 ceph-mon[112139]: osdmap e1416: 8 total, 8 up, 7 in 2024-09-21T10:49:08.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:08 smithi070 ceph-mon[107385]: pgmap v2522: 89 pgs: 89 active+clean; 1015 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-21T10:49:08.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:08 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:49:08.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:08 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:49:08.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:08 smithi070 ceph-mon[107385]: osdmap e1416: 8 total, 8 up, 7 in 2024-09-21T10:49:08.649 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:08.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:08.671 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:08.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:08.772 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:08.793 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:08.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:08.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:08.800 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 8 16 23088 23072 180.234 107.5 0.00187776 0.00549129 2024-09-21T10:49:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:08 smithi160 ceph-mon[101879]: pgmap v2522: 89 pgs: 89 active+clean; 1015 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-21T10:49:08.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:08 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:49:08.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:08 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:49:08.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:08 smithi160 ceph-mon[101879]: osdmap e1416: 8 total, 8 up, 7 in 2024-09-21T10:49:08.895 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:08.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:08.922 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:08.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:09.022 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:09.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:09.043 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:09.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:09.144 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:09.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:09.166 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:09.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:09.268 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:09.299 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:09.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:09.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:09.401 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:09.431 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:09.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:09.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:09.533 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:09.565 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:09.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:09.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:09.666 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:09.696 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:09.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:09.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:09.798 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:09.801 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 9 16 24443 24427 169.617 84.6875 0.00489751 0.00588997 2024-09-21T10:49:09.830 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:09.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:09.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:09.932 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:09.968 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:09.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:09.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:10.070 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:10.108 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:10.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:10.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:10.210 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:10.242 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:10.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:10.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:10.345 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:10.373 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:10.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:10.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:10.475 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:10.515 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:10.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:10.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:10.617 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:10.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:10.654 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:10.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:10.755 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:10.787 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:10.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:10.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:10.800 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 10 16 26880 26864 167.885 152.312 0.00739755 0.0059496 2024-09-21T10:49:10.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:10 smithi160 ceph-mon[101879]: pgmap v2524: 89 pgs: 89 active+clean; 1015 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-21T10:49:10.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:10 smithi070 ceph-mon[107385]: pgmap v2524: 89 pgs: 89 active+clean; 1015 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-21T10:49:10.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:10 smithi070 ceph-mon[112139]: pgmap v2524: 89 pgs: 89 active+clean; 1015 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-21T10:49:10.889 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:10.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:10.927 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:10.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:11.028 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:11.060 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:11.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:11.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:11.162 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:11.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:11.196 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:11.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:11.243 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:11.297 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:11.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:11.329 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:11.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:11.430 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:11.450 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:11.471 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:11.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:11.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:11.573 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:11.614 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:11.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:11.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:11.716 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:11.745 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:11.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:11.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:11.790 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:49:11.791 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T10:49:11.800 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 11 16 29054 29038 164.974 135.875 0.00758953 0.00605632 2024-09-21T10:49:11.847 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:11.878 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:11.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:11.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:11.979 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:12.000 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:12.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:12.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:12.102 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:12.133 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:12.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:12.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:12.235 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:12.265 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:12.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:12.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:12.367 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:12.403 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:12.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:12.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:12.504 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:12.538 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:12.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:12.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:12.640 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:12.670 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:12.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:12.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:12.710 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:12 smithi070 ceph-mon[107385]: pgmap v2525: 89 pgs: 89 active+clean; 1.3 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 134 MiB/s wr, 2.15k op/s 2024-09-21T10:49:12.710 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:12 smithi070 ceph-mon[112139]: pgmap v2525: 89 pgs: 89 active+clean; 1.3 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 134 MiB/s wr, 2.15k op/s 2024-09-21T10:49:12.771 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:12.800 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 12 16 31063 31047 161.689 125.562 0.00454716 0.00617845 2024-09-21T10:49:12.801 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:12.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:12.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:12 smithi160 ceph-mon[101879]: pgmap v2525: 89 pgs: 89 active+clean; 1.3 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 134 MiB/s wr, 2.15k op/s 2024-09-21T10:49:12.902 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:12.937 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:12.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:12.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:12.981 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:49:13.038 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:13.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:13.074 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:13.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:13.175 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:13.207 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:13.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:13.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:13.280 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:49:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:49:13] "GET /metrics HTTP/1.1" 200 36589 "" "Prometheus/2.51.0" 2024-09-21T10:49:13.309 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:13.343 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:13.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:13.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:13.445 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:13.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:13.482 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:13.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:13.583 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:13.619 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:13.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:13.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:13.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:13 smithi070 ceph-mon[107385]: from='client.45629 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:49:13.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:13 smithi070 ceph-mon[112139]: from='client.45629 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:49:13.722 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:13.746 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:13.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:13.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:13.801 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 13 16 32977 32961 158.452 119.625 0.00521569 0.00630499 2024-09-21T10:49:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:13 smithi160 ceph-mon[101879]: from='client.45629 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:49:13.848 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:13.883 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:13.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:13.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:13.985 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:14.014 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:14.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:14.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:14.115 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:14.154 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:14.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:14.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:14.256 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:14.277 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:14.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:14.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:14.378 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:14.410 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:14.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:14.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:14.512 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:14.545 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:14.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:14.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:14 smithi070 ceph-mon[107385]: pgmap v2526: 89 pgs: 89 active+clean; 1.6 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-09-21T10:49:14.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:14 smithi070 ceph-mon[112139]: pgmap v2526: 89 pgs: 89 active+clean; 1.6 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-09-21T10:49:14.647 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:14.687 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:14.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:14.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:14.789 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:14.801 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 14 16 34970 34954 156.031 124.562 0.00669156 0.00640387 2024-09-21T10:49:14.824 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:14.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:14.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:14.833 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":2525,"stamp":"2024-09-21T10:49:10.828547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1407850469,"num_objects":21499,"num_object_clones":0,"num_object_copies":64497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21499,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":21752,"num_write_kb":1376721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":2962150,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21681,"ondisk_log_size":21681,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":6977000,"kb_used_data":3824752,"kb_used_omap":219,"kb_used_meta":3151972,"kb_avail":649239064,"statfs":{"total":671965249536,"available":664820801536,"internally_reserved":0,"allocated":3916546048,"data_stored":3884546979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":225222,"internal_metadata":3227619386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,1,0,0,0,0,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1407254528,"num_objects":21473,"num_object_clones":0,"num_object_copies":64419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21473,"num_write_kb":1374272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003424"},"pg_stats":[{"pgid":"55.f","version":"1415'1172","reported_seq":1185,"reported_epoch":1415,"state":"active+clean","last_fresh":"2024-09-21T10:49:06.484486+0000","last_change":"2024-09-21T10:48:52.971561+0000","last_active":"2024-09-21T10:49:06.484486+0000","last_peered":"2024-09-21T10:49:06.484486+0000","last_clean":"2024-09-21T10:49:06.484486+0000","last_became_active":"2024-09-21T10:48:52.971254+0000","last_became_peered":"2024-09-21T10:48:52.971254+0000","last_unstale":"2024-09-21T10:49:06.484486+0000","last_undegraded":"2024-09-21T10:49:06.484486+0000","last_fullsized":"2024-09-21T10:49:06.484486+0000","mapping_epoch":1411,"log_start":"0'0","ondisk_log_start":"0'0","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":1172,"log_dups_size":0,"ondisk_log_size":1172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:19:52.513156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76808192,"num_objects":1172,"num_object_clones":0,"num_object_copies":3516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1172,"num_write_kb":75008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.e","version":"1416'1513","reported_seq":1534,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:10.140891+0000","last_change":"2024-09-21T10:48:52.932591+0000","last_active":"2024-09-21T10:49:10.140891+0000","last_peered":"2024-09-21T10:49:10.140891+0000","last_clean":"2024-09-21T10:49:10.140891+0000","last_became_active":"2024-09-21T10:48:52.932263+0000","last_became_peered":"2024-09-21T10:48:52.932263+0000","last_unstale":"2024-09-21T10:49:10.140891+0000","last_undegraded":"2024-09-21T10:49:10.140891+0000","last_fullsized":"2024-09-21T10:49:10.140891+0000","mapping_epoch":1411,"log_start":"0'0","ondisk_log_start":"0'0","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":1513,"log_dups_size":0,"ondisk_log_size":1513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:22.903755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99155968,"num_objects":1513,"num_object_clones":0,"num_object_copies":4539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1513,"num_write_kb":96832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.d","version":"1415'1179","reported_seq":1192,"reported_epoch":1415,"state":"active+clean","last_fresh":"2024-09-21T10:49:06.354079+0000","last_change":"2024-09-21T10:48:53.487054+0000","last_active":"2024-09-21T10:49:06.354079+0000","last_peered":"2024-09-21T10:49:06.354079+0000","last_clean":"2024-09-21T10:49:06.354079+0000","last_became_active":"2024-09-21T10:48:53.486775+0000","last_became_peered":"2024-09-21T10:48:53.486775+0000","last_unstale":"2024-09-21T10:49:06.354079+0000","last_undegraded":"2024-09-21T10:49:06.354079+0000","last_fullsized":"2024-09-21T10:49:06.354079+0000","mapping_epoch":1411,"log_start":"0'0","ondisk_log_start":"0'0","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":1179,"log_dups_size":0,"ondisk_log_size":1179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:39:25.691292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77266944,"num_objects":1179,"num_object_clones":0,"num_object_copies":3537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1179,"num_write_kb":75456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"55.c","version":"1416'1571","reported_seq":1592,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:10.133165+0000","last_change":"2024-09-21T10:48:53.486624+0000","last_active":"2024-09-21T10:49:10.133165+0000","last_peered":"2024-09-21T10:49:10.133165+0000","last_clean":"2024-09-21T10:49:10.133165+0000","last_became_active":"2024-09-21T10:48:53.486301+0000","last_became_peered":"2024-09-21T10:48:53.486301+0000","last_unstale":"2024-09-21T10:49:10.133165+0000","last_undegraded":"2024-09-21T10:49:10.133165+0000","last_fullsized":"2024-09-21T10:49:10.133165+0000","mapping_epoch":1411,"log_start":"0'0","ondisk_log_start":"0'0","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":1571,"log_dups_size":0,"ondisk_log_size":1571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:56:20.441354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102957056,"num_objects":1571,"num_object_clones":0,"num_object_copies":4713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1571,"num_write_kb":100544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.b","version":"1415'1214","reported_seq":1227,"reported_epoch":1415,"state":"active+clean","last_fresh":"2024-09-21T10:49:06.494100+0000","last_change":"2024-09-21T10:48:52.934001+0000","last_active":"2024-09-21T10:49:06.494100+0000","last_peered":"2024-09-21T10:49:06.494100+0000","last_clean":"2024-09-21T10:49:06.494100+0000","last_became_active":"2024-09-21T10:48:52.933674+0000","last_became_peered":"2024-09-21T10:48:52.933674+0000","last_unstale":"2024-09-21T10:49:06.494100+0000","last_undegraded":"2024-09-21T10:49:06.494100+0000","last_fullsized":"2024-09-21T10:49:06.494100+0000","mapping_epoch":1411,"log_start":"0'0","ondisk_log_start":"0'0","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":1214,"log_dups_size":0,"ondisk_log_size":1214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:13.178680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79560704,"num_objects":1214,"num_object_clones":0,"num_object_copies":3642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1214,"num_write_kb":77696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.a","version":"1416'1629","reported_seq":1650,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:10.121648+0000","last_change":"2024-09-21T10:48:53.487029+0000","last_active":"2024-09-21T10:49:10.121648+0000","last_peered":"2024-09-21T10:49:10.121648+0000","last_clean":"2024-09-21T10:49:10.121648+0000","last_became_active":"2024-09-21T10:48:53.486684+0000","last_became_peered":"2024-09-21T10:48:53.486684+0000","last_unstale":"2024-09-21T10:49:10.121648+0000","last_undegraded":"2024-09-21T10:49:10.121648+0000","last_fullsized":"2024-09-21T10:49:10.121648+0000","mapping_epoch":1411,"log_start":"0'0","ondisk_log_start":"0'0","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":1629,"log_dups_size":0,"ondisk_log_size":1629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:46:36.304213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106758144,"num_objects":1629,"num_object_clones":0,"num_object_copies":4887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1629,"num_write_kb":104256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.9","version":"1416'1534","reported_seq":1555,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:10.126478+0000","last_change":"2024-09-21T10:48:53.486971+0000","last_active":"2024-09-21T10:49:10.126478+0000","last_peered":"2024-09-21T10:49:10.126478+0000","last_clean":"2024-09-21T10:49:10.126478+0000","last_became_active":"2024-09-21T10:48:53.486699+0000","last_became_peered":"2024-09-21T10:48:53.486699+0000","last_unstale":"2024-09-21T10:49:10.126478+0000","last_undegraded":"2024-09-21T10:49:10.126478+0000","last_fullsized":"2024-09-21T10:49:10.126478+0000","mapping_epoch":1411,"log_start":"0'0","ondisk_log_start":"0'0","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":1534,"log_dups_size":0,"ondisk_log_size":1534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:47:42.307959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100532224,"num_objects":1534,"num_object_clones":0,"num_object_copies":4602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1534,"num_write_kb":98176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.8","version":"1416'1541","reported_seq":1561,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:10.049822+0000","last_change":"2024-09-21T10:48:52.971850+0000","last_active":"2024-09-21T10:49:10.049822+0000","last_peered":"2024-09-21T10:49:10.049822+0000","last_clean":"2024-09-21T10:49:10.049822+0000","last_became_active":"2024-09-21T10:48:52.971267+0000","last_became_peered":"2024-09-21T10:48:52.971267+0000","last_unstale":"2024-09-21T10:49:10.049822+0000","last_undegraded":"2024-09-21T10:49:10.049822+0000","last_fullsized":"2024-09-21T10:49:10.049822+0000","mapping_epoch":1411,"log_start":"0'0","ondisk_log_start":"0'0","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":1541,"log_dups_size":0,"ondisk_log_size":1541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:29:20.867133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100990976,"num_objects":1541,"num_object_clones":0,"num_object_copies":4623,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1541,"num_write_kb":98624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"55.7","version":"1415'1056","reported_seq":1069,"reported_epoch":1415,"state":"active+clean","last_fresh":"2024-09-21T10:49:06.342141+0000","last_change":"2024-09-21T10:48:52.933230+0000","last_active":"2024-09-21T10:49:06.342141+0000","last_peered":"2024-09-21T10:49:06.342141+0000","last_clean":"2024-09-21T10:49:06.342141+0000","last_became_active":"2024-09-21T10:48:52.932944+0000","last_became_peered":"2024-09-21T10:48:52.932944+0000","last_unstale":"2024-09-21T10:49:06.342141+0000","last_undegraded":"2024-09-21T10:49:06.342141+0000","last_fullsized":"2024-09-21T10:49:06.342141+0000","mapping_epoch":1411,"log_start":"0'0","ondisk_log_start":"0'0","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":1056,"log_dups_size":0,"ondisk_log_size":1056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:16:43.711173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69206016,"num_objects":1056,"num_object_clones":0,"num_object_copies":3168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1056,"num_write_kb":67584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"55.6","version":"1415'1146","reported_seq":1159,"reported_epoch":1415,"state":"active+clean","last_fresh":"2024-09-21T10:49:06.351216+0000","last_change":"2024-09-21T10:48:52.971553+0000","last_active":"2024-09-21T10:49:06.351216+0000","last_peered":"2024-09-21T10:49:06.351216+0000","last_clean":"2024-09-21T10:49:06.351216+0000","last_became_active":"2024-09-21T10:48:52.971198+0000","last_became_peered":"2024-09-21T10:48:52.971198+0000","last_unstale":"2024-09-21T10:49:06.351216+0000","last_undegraded":"2024-09-21T10:49:06.351216+0000","last_fullsized":"2024-09-21T10:49:06.351216+0000","mapping_epoch":1411,"log_start":"0'0","ondisk_log_start":"0'0","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":1146,"log_dups_size":0,"ondisk_log_size":1146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:30:25.171859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75104256,"num_objects":1146,"num_object_clones":0,"num_object_copies":3438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1146,"num_write_kb":73344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"55.5","version":"1415'1149","reported_seq":1161,"reported_epoch":1415,"state":"active+clean","last_fresh":"2024-09-21T10:49:06.497843+0000","last_change":"2024-09-21T10:48:52.934778+0000","last_active":"2024-09-21T10:49:06.497843+0000","last_peered":"2024-09-21T10:49:06.497843+0000","last_clean":"2024-09-21T10:49:06.497843+0000","last_became_active":"2024-09-21T10:48:52.934459+0000","last_became_peered":"2024-09-21T10:48:52.934459+0000","last_unstale":"2024-09-21T10:49:06.497843+0000","last_undegraded":"2024-09-21T10:49:06.497843+0000","last_fullsized":"2024-09-21T10:49:06.497843+0000","mapping_epoch":1411,"log_start":"0'0","ondisk_log_start":"0'0","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":1149,"log_dups_size":0,"ondisk_log_size":1149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:51:21.572478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75300864,"num_objects":1149,"num_object_clones":0,"num_object_copies":3447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1149,"num_write_kb":73536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.4","version":"1415'1121","reported_seq":1133,"reported_epoch":1415,"state":"active+clean","last_fresh":"2024-09-21T10:49:06.351326+0000","last_change":"2024-09-21T10:48:52.934634+0000","last_active":"2024-09-21T10:49:06.351326+0000","last_peered":"2024-09-21T10:49:06.351326+0000","last_clean":"2024-09-21T10:49:06.351326+0000","last_became_active":"2024-09-21T10:48:52.934368+0000","last_became_peered":"2024-09-21T10:48:52.934368+0000","last_unstale":"2024-09-21T10:49:06.351326+0000","last_undegraded":"2024-09-21T10:49:06.351326+0000","last_fullsized":"2024-09-21T10:49:06.351326+0000","mapping_epoch":1411,"log_start":"0'0","ondisk_log_start":"0'0","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":1121,"log_dups_size":0,"ondisk_log_size":1121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:27:09.216667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73465856,"num_objects":1121,"num_object_clones":0,"num_object_copies":3363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1121,"num_write_kb":71744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"55.3","version":"1416'1445","reported_seq":1466,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:09.747337+0000","last_change":"2024-09-21T10:48:52.971712+0000","last_active":"2024-09-21T10:49:09.747337+0000","last_peered":"2024-09-21T10:49:09.747337+0000","last_clean":"2024-09-21T10:49:09.747337+0000","last_became_active":"2024-09-21T10:48:52.971310+0000","last_became_peered":"2024-09-21T10:48:52.971310+0000","last_unstale":"2024-09-21T10:49:09.747337+0000","last_undegraded":"2024-09-21T10:49:09.747337+0000","last_fullsized":"2024-09-21T10:49:09.747337+0000","mapping_epoch":1411,"log_start":"0'0","ondisk_log_start":"0'0","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":1445,"log_dups_size":0,"ondisk_log_size":1445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:32:41.740925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94699520,"num_objects":1445,"num_object_clones":0,"num_object_copies":4335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1445,"num_write_kb":92480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"55.2","version":"1416'1542","reported_seq":1563,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:09.764401+0000","last_change":"2024-09-21T10:48:52.933672+0000","last_active":"2024-09-21T10:49:09.764401+0000","last_peered":"2024-09-21T10:49:09.764401+0000","last_clean":"2024-09-21T10:49:09.764401+0000","last_became_active":"2024-09-21T10:48:52.933329+0000","last_became_peered":"2024-09-21T10:48:52.933329+0000","last_unstale":"2024-09-21T10:49:09.764401+0000","last_undegraded":"2024-09-21T10:49:09.764401+0000","last_fullsized":"2024-09-21T10:49:09.764401+0000","mapping_epoch":1411,"log_start":"0'0","ondisk_log_start":"0'0","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":1542,"log_dups_size":0,"ondisk_log_size":1542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:19:00.652265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101056512,"num_objects":1542,"num_object_clones":0,"num_object_copies":4626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1542,"num_write_kb":98688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"55.1","version":"1415'1099","reported_seq":1112,"reported_epoch":1415,"state":"active+clean","last_fresh":"2024-09-21T10:49:06.346074+0000","last_change":"2024-09-21T10:48:53.487050+0000","last_active":"2024-09-21T10:49:06.346074+0000","last_peered":"2024-09-21T10:49:06.346074+0000","last_clean":"2024-09-21T10:49:06.346074+0000","last_became_active":"2024-09-21T10:48:53.486764+0000","last_became_peered":"2024-09-21T10:48:53.486764+0000","last_unstale":"2024-09-21T10:49:06.346074+0000","last_undegraded":"2024-09-21T10:49:06.346074+0000","last_fullsized":"2024-09-21T10:49:06.346074+0000","mapping_epoch":1411,"log_start":"0'0","ondisk_log_start":"0'0","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":1099,"log_dups_size":0,"ondisk_log_size":1099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:18:59.329738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72024064,"num_objects":1099,"num_object_clones":0,"num_object_copies":3297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1099,"num_write_kb":70336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"55.0","version":"1416'1562","reported_seq":1583,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:10.135640+0000","last_change":"2024-09-21T10:48:52.933895+0000","last_active":"2024-09-21T10:49:10.135640+0000","last_peered":"2024-09-21T10:49:10.135640+0000","last_clean":"2024-09-21T10:49:10.135640+0000","last_became_active":"2024-09-21T10:48:52.933584+0000","last_became_peered":"2024-09-21T10:48:52.933584+0000","last_unstale":"2024-09-21T10:49:10.135640+0000","last_undegraded":"2024-09-21T10:49:10.135640+0000","last_fullsized":"2024-09-21T10:49:10.135640+0000","mapping_epoch":1411,"log_start":"0'0","ondisk_log_start":"0'0","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":1562,"log_dups_size":0,"ondisk_log_size":1562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:05:47.571385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102367232,"num_objects":1562,"num_object_clones":0,"num_object_copies":4686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1562,"num_write_kb":99968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":2146,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.394038+0000","last_change":"2024-09-21T10:40:13.438023+0000","last_active":"2024-09-21T10:49:07.394038+0000","last_peered":"2024-09-21T10:49:07.394038+0000","last_clean":"2024-09-21T10:49:07.394038+0000","last_became_active":"2024-09-21T10:40:13.431103+0000","last_became_peered":"2024-09-21T10:40:13.431103+0000","last_unstale":"2024-09-21T10:49:07.394038+0000","last_undegraded":"2024-09-21T10:49:07.394038+0000","last_fullsized":"2024-09-21T10:49:07.394038+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:00:24.887661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1385,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.059914+0000","last_change":"2024-09-21T10:07:55.999392+0000","last_active":"2024-09-21T10:48:56.059914+0000","last_peered":"2024-09-21T10:48:56.059914+0000","last_clean":"2024-09-21T10:48:56.059914+0000","last_became_active":"2024-09-21T10:07:55.999103+0000","last_became_peered":"2024-09-21T10:07:55.999103+0000","last_unstale":"2024-09-21T10:48:56.059914+0000","last_undegraded":"2024-09-21T10:48:56.059914+0000","last_fullsized":"2024-09-21T10:48:56.059914+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:48.757242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":2125,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.386266+0000","last_change":"2024-09-21T10:40:13.432818+0000","last_active":"2024-09-21T10:49:07.386266+0000","last_peered":"2024-09-21T10:49:07.386266+0000","last_clean":"2024-09-21T10:49:07.386266+0000","last_became_active":"2024-09-21T10:40:13.432448+0000","last_became_peered":"2024-09-21T10:40:13.432448+0000","last_unstale":"2024-09-21T10:49:07.386266+0000","last_undegraded":"2024-09-21T10:49:07.386266+0000","last_fullsized":"2024-09-21T10:49:07.386266+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:20.448242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2142,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.386384+0000","last_change":"2024-09-21T10:40:13.431824+0000","last_active":"2024-09-21T10:49:07.386384+0000","last_peered":"2024-09-21T10:49:07.386384+0000","last_clean":"2024-09-21T10:49:07.386384+0000","last_became_active":"2024-09-21T10:40:13.431516+0000","last_became_peered":"2024-09-21T10:40:13.431516+0000","last_unstale":"2024-09-21T10:49:07.386384+0000","last_undegraded":"2024-09-21T10:49:07.386384+0000","last_fullsized":"2024-09-21T10:49:07.386384+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:30:40.226452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":29,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.060513+0000","last_change":"2024-09-21T10:48:48.918126+0000","last_active":"2024-09-21T10:48:56.060513+0000","last_peered":"2024-09-21T10:48:56.060513+0000","last_clean":"2024-09-21T10:48:56.060513+0000","last_became_active":"2024-09-21T10:48:48.917829+0000","last_became_peered":"2024-09-21T10:48:48.917829+0000","last_unstale":"2024-09-21T10:48:56.060513+0000","last_undegraded":"2024-09-21T10:48:56.060513+0000","last_fullsized":"2024-09-21T10:48:56.060513+0000","mapping_epoch":1409,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1410,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:38:37.248480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1399,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.060262+0000","last_change":"2024-09-21T10:09:06.574933+0000","last_active":"2024-09-21T10:48:56.060262+0000","last_peered":"2024-09-21T10:48:56.060262+0000","last_clean":"2024-09-21T10:48:56.060262+0000","last_became_active":"2024-09-21T10:09:06.574467+0000","last_became_peered":"2024-09-21T10:09:06.574467+0000","last_unstale":"2024-09-21T10:48:56.060262+0000","last_undegraded":"2024-09-21T10:48:56.060262+0000","last_fullsized":"2024-09-21T10:48:56.060262+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:06.142269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":29,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.060598+0000","last_change":"2024-09-21T10:48:48.916688+0000","last_active":"2024-09-21T10:48:56.060598+0000","last_peered":"2024-09-21T10:48:56.060598+0000","last_clean":"2024-09-21T10:48:56.060598+0000","last_became_active":"2024-09-21T10:48:48.916278+0000","last_became_peered":"2024-09-21T10:48:48.916278+0000","last_unstale":"2024-09-21T10:48:56.060598+0000","last_undegraded":"2024-09-21T10:48:56.060598+0000","last_fullsized":"2024-09-21T10:48:56.060598+0000","mapping_epoch":1409,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1410,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:57:13.241836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1399,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.060107+0000","last_change":"2024-09-21T10:09:06.560175+0000","last_active":"2024-09-21T10:48:56.060107+0000","last_peered":"2024-09-21T10:48:56.060107+0000","last_clean":"2024-09-21T10:48:56.060107+0000","last_became_active":"2024-09-21T10:09:06.558584+0000","last_became_peered":"2024-09-21T10:09:06.558584+0000","last_unstale":"2024-09-21T10:48:56.060107+0000","last_undegraded":"2024-09-21T10:48:56.060107+0000","last_fullsized":"2024-09-21T10:48:56.060107+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:02.918339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":2128,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.393508+0000","last_change":"2024-09-21T10:44:14.409321+0000","last_active":"2024-09-21T10:49:07.393508+0000","last_peered":"2024-09-21T10:49:07.393508+0000","last_clean":"2024-09-21T10:49:07.393508+0000","last_became_active":"2024-09-21T10:44:14.408833+0000","last_became_peered":"2024-09-21T10:44:14.408833+0000","last_unstale":"2024-09-21T10:49:07.393508+0000","last_undegraded":"2024-09-21T10:49:07.393508+0000","last_fullsized":"2024-09-21T10:49:07.393508+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:10:00.997210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":904,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.391028+0000","last_change":"2024-09-21T10:40:13.436927+0000","last_active":"2024-09-21T10:49:07.391028+0000","last_peered":"2024-09-21T10:49:07.391028+0000","last_clean":"2024-09-21T10:49:07.391028+0000","last_became_active":"2024-09-21T10:40:13.436329+0000","last_became_peered":"2024-09-21T10:40:13.436329+0000","last_unstale":"2024-09-21T10:49:07.391028+0000","last_undegraded":"2024-09-21T10:49:07.391028+0000","last_fullsized":"2024-09-21T10:49:07.391028+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:05.668898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":1410,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.060462+0000","last_change":"2024-09-21T10:40:13.433278+0000","last_active":"2024-09-21T10:48:56.060462+0000","last_peered":"2024-09-21T10:48:56.060462+0000","last_clean":"2024-09-21T10:48:56.060462+0000","last_became_active":"2024-09-21T10:40:13.432814+0000","last_became_peered":"2024-09-21T10:40:13.432814+0000","last_unstale":"2024-09-21T10:48:56.060462+0000","last_undegraded":"2024-09-21T10:48:56.060462+0000","last_fullsized":"2024-09-21T10:48:56.060462+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:55.909872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2136,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.385593+0000","last_change":"2024-09-21T10:40:13.432531+0000","last_active":"2024-09-21T10:49:07.385593+0000","last_peered":"2024-09-21T10:49:07.385593+0000","last_clean":"2024-09-21T10:49:07.385593+0000","last_became_active":"2024-09-21T10:40:13.432210+0000","last_became_peered":"2024-09-21T10:40:13.432210+0000","last_unstale":"2024-09-21T10:49:07.385593+0000","last_undegraded":"2024-09-21T10:49:07.385593+0000","last_fullsized":"2024-09-21T10:49:07.385593+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:20:10.998607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2107,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.390683+0000","last_change":"2024-09-21T10:09:06.530394+0000","last_active":"2024-09-21T10:49:07.390683+0000","last_peered":"2024-09-21T10:49:07.390683+0000","last_clean":"2024-09-21T10:49:07.390683+0000","last_became_active":"2024-09-21T10:09:06.529307+0000","last_became_peered":"2024-09-21T10:09:06.529307+0000","last_unstale":"2024-09-21T10:49:07.390683+0000","last_undegraded":"2024-09-21T10:49:07.390683+0000","last_fullsized":"2024-09-21T10:49:07.390683+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:54.287397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2141,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.393869+0000","last_change":"2024-09-21T10:44:14.409422+0000","last_active":"2024-09-21T10:49:07.393869+0000","last_peered":"2024-09-21T10:49:07.393869+0000","last_clean":"2024-09-21T10:49:07.393869+0000","last_became_active":"2024-09-21T10:44:14.408726+0000","last_became_peered":"2024-09-21T10:44:14.408726+0000","last_unstale":"2024-09-21T10:49:07.393869+0000","last_undegraded":"2024-09-21T10:49:07.393869+0000","last_fullsized":"2024-09-21T10:49:07.393869+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:44:21.743112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2122,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.386408+0000","last_change":"2024-09-21T10:09:06.552324+0000","last_active":"2024-09-21T10:49:07.386408+0000","last_peered":"2024-09-21T10:49:07.386408+0000","last_clean":"2024-09-21T10:49:07.386408+0000","last_became_active":"2024-09-21T10:09:06.540545+0000","last_became_peered":"2024-09-21T10:09:06.540545+0000","last_unstale":"2024-09-21T10:49:07.386408+0000","last_undegraded":"2024-09-21T10:49:07.386408+0000","last_fullsized":"2024-09-21T10:49:07.386408+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:08.506149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1389,"reported_epoch":1415,"state":"active+clean","last_fresh":"2024-09-21T10:48:58.087004+0000","last_change":"2024-09-21T10:07:56.003005+0000","last_active":"2024-09-21T10:48:58.087004+0000","last_peered":"2024-09-21T10:48:58.087004+0000","last_clean":"2024-09-21T10:48:58.087004+0000","last_became_active":"2024-09-21T10:07:56.001550+0000","last_became_peered":"2024-09-21T10:07:56.001550+0000","last_unstale":"2024-09-21T10:48:58.087004+0000","last_undegraded":"2024-09-21T10:48:58.087004+0000","last_fullsized":"2024-09-21T10:48:58.087004+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:54.553261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2115,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.386639+0000","last_change":"2024-09-21T10:09:06.559017+0000","last_active":"2024-09-21T10:49:07.386639+0000","last_peered":"2024-09-21T10:49:07.386639+0000","last_clean":"2024-09-21T10:49:07.386639+0000","last_became_active":"2024-09-21T10:09:06.555256+0000","last_became_peered":"2024-09-21T10:09:06.555256+0000","last_unstale":"2024-09-21T10:49:07.386639+0000","last_undegraded":"2024-09-21T10:49:07.386639+0000","last_fullsized":"2024-09-21T10:49:07.386639+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:33:45.145138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2121,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.394207+0000","last_change":"2024-09-21T10:09:06.562504+0000","last_active":"2024-09-21T10:49:07.394207+0000","last_peered":"2024-09-21T10:49:07.394207+0000","last_clean":"2024-09-21T10:49:07.394207+0000","last_became_active":"2024-09-21T10:09:06.561747+0000","last_became_peered":"2024-09-21T10:09:06.561747+0000","last_unstale":"2024-09-21T10:49:07.394207+0000","last_undegraded":"2024-09-21T10:49:07.394207+0000","last_fullsized":"2024-09-21T10:49:07.394207+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:03:58.012406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2120,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.386517+0000","last_change":"2024-09-21T10:44:14.399948+0000","last_active":"2024-09-21T10:49:07.386517+0000","last_peered":"2024-09-21T10:49:07.386517+0000","last_clean":"2024-09-21T10:49:07.386517+0000","last_became_active":"2024-09-21T10:44:14.399337+0000","last_became_peered":"2024-09-21T10:44:14.399337+0000","last_unstale":"2024-09-21T10:49:07.386517+0000","last_undegraded":"2024-09-21T10:49:07.386517+0000","last_fullsized":"2024-09-21T10:49:07.386517+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:40.185020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1404,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.060018+0000","last_change":"2024-09-21T10:44:14.401716+0000","last_active":"2024-09-21T10:48:56.060018+0000","last_peered":"2024-09-21T10:48:56.060018+0000","last_clean":"2024-09-21T10:48:56.060018+0000","last_became_active":"2024-09-21T10:44:14.401254+0000","last_became_peered":"2024-09-21T10:44:14.401254+0000","last_unstale":"2024-09-21T10:48:56.060018+0000","last_undegraded":"2024-09-21T10:48:56.060018+0000","last_fullsized":"2024-09-21T10:48:56.060018+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:07:21.536272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":64,"reported_epoch":1415,"state":"active+clean","last_fresh":"2024-09-21T10:48:58.087592+0000","last_change":"2024-09-21T10:44:14.424249+0000","last_active":"2024-09-21T10:48:58.087592+0000","last_peered":"2024-09-21T10:48:58.087592+0000","last_clean":"2024-09-21T10:48:58.087592+0000","last_became_active":"2024-09-21T10:44:14.408884+0000","last_became_peered":"2024-09-21T10:44:14.408884+0000","last_unstale":"2024-09-21T10:48:58.087592+0000","last_undegraded":"2024-09-21T10:48:58.087592+0000","last_fullsized":"2024-09-21T10:48:58.087592+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:18:44.759435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2114,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.390924+0000","last_change":"2024-09-21T10:40:13.437741+0000","last_active":"2024-09-21T10:49:07.390924+0000","last_peered":"2024-09-21T10:49:07.390924+0000","last_clean":"2024-09-21T10:49:07.390924+0000","last_became_active":"2024-09-21T10:40:13.437447+0000","last_became_peered":"2024-09-21T10:40:13.437447+0000","last_unstale":"2024-09-21T10:49:07.390924+0000","last_undegraded":"2024-09-21T10:49:07.390924+0000","last_fullsized":"2024-09-21T10:49:07.390924+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:40:47.072907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":1433,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.108360+0000","last_change":"2024-09-21T10:44:14.415264+0000","last_active":"2024-09-21T10:48:56.108360+0000","last_peered":"2024-09-21T10:48:56.108360+0000","last_clean":"2024-09-21T10:48:56.108360+0000","last_became_active":"2024-09-21T10:44:14.407813+0000","last_became_peered":"2024-09-21T10:44:14.407813+0000","last_unstale":"2024-09-21T10:48:56.108360+0000","last_undegraded":"2024-09-21T10:48:56.108360+0000","last_fullsized":"2024-09-21T10:48:56.108360+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:55.933601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1386,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.060182+0000","last_change":"2024-09-21T10:07:56.001756+0000","last_active":"2024-09-21T10:48:56.060182+0000","last_peered":"2024-09-21T10:48:56.060182+0000","last_clean":"2024-09-21T10:48:56.060182+0000","last_became_active":"2024-09-21T10:07:56.000789+0000","last_became_peered":"2024-09-21T10:07:56.000789+0000","last_unstale":"2024-09-21T10:48:56.060182+0000","last_undegraded":"2024-09-21T10:48:56.060182+0000","last_fullsized":"2024-09-21T10:48:56.060182+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:56:04.302902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1374,"reported_epoch":1415,"state":"active+clean","last_fresh":"2024-09-21T10:48:58.051392+0000","last_change":"2024-09-21T09:54:53.636902+0000","last_active":"2024-09-21T10:48:58.051392+0000","last_peered":"2024-09-21T10:48:58.051392+0000","last_clean":"2024-09-21T10:48:58.051392+0000","last_became_active":"2024-09-21T09:54:53.636616+0000","last_became_peered":"2024-09-21T09:54:53.636616+0000","last_unstale":"2024-09-21T10:48:58.051392+0000","last_undegraded":"2024-09-21T10:48:58.051392+0000","last_fullsized":"2024-09-21T10:48:58.051392+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:46:47.694495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1428,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.060308+0000","last_change":"2024-09-21T10:44:14.402059+0000","last_active":"2024-09-21T10:48:56.060308+0000","last_peered":"2024-09-21T10:48:56.060308+0000","last_clean":"2024-09-21T10:48:56.060308+0000","last_became_active":"2024-09-21T10:44:14.401710+0000","last_became_peered":"2024-09-21T10:44:14.401710+0000","last_unstale":"2024-09-21T10:48:56.060308+0000","last_undegraded":"2024-09-21T10:48:56.060308+0000","last_fullsized":"2024-09-21T10:48:56.060308+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:17.314693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":2129,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.393772+0000","last_change":"2024-09-21T10:07:56.023462+0000","last_active":"2024-09-21T10:49:07.393772+0000","last_peered":"2024-09-21T10:49:07.393772+0000","last_clean":"2024-09-21T10:49:07.393772+0000","last_became_active":"2024-09-21T10:07:56.021859+0000","last_became_peered":"2024-09-21T10:07:56.021859+0000","last_unstale":"2024-09-21T10:49:07.393772+0000","last_undegraded":"2024-09-21T10:49:07.393772+0000","last_fullsized":"2024-09-21T10:49:07.393772+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:42:03.134729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2116,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.386511+0000","last_change":"2024-09-21T10:44:14.399028+0000","last_active":"2024-09-21T10:49:07.386511+0000","last_peered":"2024-09-21T10:49:07.386511+0000","last_clean":"2024-09-21T10:49:07.386511+0000","last_became_active":"2024-09-21T10:44:14.398115+0000","last_became_peered":"2024-09-21T10:44:14.398115+0000","last_unstale":"2024-09-21T10:49:07.386511+0000","last_undegraded":"2024-09-21T10:49:07.386511+0000","last_fullsized":"2024-09-21T10:49:07.386511+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:05.170376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2109,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.391309+0000","last_change":"2024-09-21T10:07:56.006236+0000","last_active":"2024-09-21T10:49:07.391309+0000","last_peered":"2024-09-21T10:49:07.391309+0000","last_clean":"2024-09-21T10:49:07.391309+0000","last_became_active":"2024-09-21T10:07:56.004439+0000","last_became_peered":"2024-09-21T10:07:56.004439+0000","last_unstale":"2024-09-21T10:49:07.391309+0000","last_undegraded":"2024-09-21T10:49:07.391309+0000","last_fullsized":"2024-09-21T10:49:07.391309+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:39.422467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":2125,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.386070+0000","last_change":"2024-09-21T10:44:14.410440+0000","last_active":"2024-09-21T10:49:07.386070+0000","last_peered":"2024-09-21T10:49:07.386070+0000","last_clean":"2024-09-21T10:49:07.386070+0000","last_became_active":"2024-09-21T10:44:14.400035+0000","last_became_peered":"2024-09-21T10:44:14.400035+0000","last_unstale":"2024-09-21T10:49:07.386070+0000","last_undegraded":"2024-09-21T10:49:07.386070+0000","last_fullsized":"2024-09-21T10:49:07.386070+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:49:39.536516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2130,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.393655+0000","last_change":"2024-09-21T10:44:14.409410+0000","last_active":"2024-09-21T10:49:07.393655+0000","last_peered":"2024-09-21T10:49:07.393655+0000","last_clean":"2024-09-21T10:49:07.393655+0000","last_became_active":"2024-09-21T10:44:14.409025+0000","last_became_peered":"2024-09-21T10:44:14.409025+0000","last_unstale":"2024-09-21T10:49:07.393655+0000","last_undegraded":"2024-09-21T10:49:07.393655+0000","last_fullsized":"2024-09-21T10:49:07.393655+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:25:09.197455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":58,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.061616+0000","last_change":"2024-09-21T10:44:14.405677+0000","last_active":"2024-09-21T10:48:56.061616+0000","last_peered":"2024-09-21T10:48:56.061616+0000","last_clean":"2024-09-21T10:48:56.061616+0000","last_became_active":"2024-09-21T10:44:14.405262+0000","last_became_peered":"2024-09-21T10:44:14.405262+0000","last_unstale":"2024-09-21T10:48:56.061616+0000","last_undegraded":"2024-09-21T10:48:56.061616+0000","last_fullsized":"2024-09-21T10:48:56.061616+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:53:12.127245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2144,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.391020+0000","last_change":"2024-09-21T10:40:13.437609+0000","last_active":"2024-09-21T10:49:07.391020+0000","last_peered":"2024-09-21T10:49:07.391020+0000","last_clean":"2024-09-21T10:49:07.391020+0000","last_became_active":"2024-09-21T10:40:13.437296+0000","last_became_peered":"2024-09-21T10:40:13.437296+0000","last_unstale":"2024-09-21T10:49:07.391020+0000","last_undegraded":"2024-09-21T10:49:07.391020+0000","last_fullsized":"2024-09-21T10:49:07.391020+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:41:58.963386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2099,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.390819+0000","last_change":"2024-09-21T10:07:56.067772+0000","last_active":"2024-09-21T10:49:07.390819+0000","last_peered":"2024-09-21T10:49:07.390819+0000","last_clean":"2024-09-21T10:49:07.390819+0000","last_became_active":"2024-09-21T10:07:56.052602+0000","last_became_peered":"2024-09-21T10:07:56.052602+0000","last_unstale":"2024-09-21T10:49:07.390819+0000","last_undegraded":"2024-09-21T10:49:07.390819+0000","last_fullsized":"2024-09-21T10:49:07.390819+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:07:59.899214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2167,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.390809+0000","last_change":"2024-09-21T10:44:14.405182+0000","last_active":"2024-09-21T10:49:07.390809+0000","last_peered":"2024-09-21T10:49:07.390809+0000","last_clean":"2024-09-21T10:49:07.390809+0000","last_became_active":"2024-09-21T10:44:14.404852+0000","last_became_peered":"2024-09-21T10:44:14.404852+0000","last_unstale":"2024-09-21T10:49:07.390809+0000","last_undegraded":"2024-09-21T10:49:07.390809+0000","last_fullsized":"2024-09-21T10:49:07.390809+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:51.687206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1316,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.385411+0000","last_change":"2024-09-21T10:48:48.917164+0000","last_active":"2024-09-21T10:49:07.385411+0000","last_peered":"2024-09-21T10:49:07.385411+0000","last_clean":"2024-09-21T10:49:07.385411+0000","last_became_active":"2024-09-21T10:48:48.916870+0000","last_became_peered":"2024-09-21T10:48:48.916870+0000","last_unstale":"2024-09-21T10:49:07.385411+0000","last_undegraded":"2024-09-21T10:49:07.385411+0000","last_fullsized":"2024-09-21T10:49:07.385411+0000","mapping_epoch":1409,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1410,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:03:14.797873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2154,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.391116+0000","last_change":"2024-09-21T10:40:13.434189+0000","last_active":"2024-09-21T10:49:07.391116+0000","last_peered":"2024-09-21T10:49:07.391116+0000","last_clean":"2024-09-21T10:49:07.391116+0000","last_became_active":"2024-09-21T10:40:13.433892+0000","last_became_peered":"2024-09-21T10:40:13.433892+0000","last_unstale":"2024-09-21T10:49:07.391116+0000","last_undegraded":"2024-09-21T10:49:07.391116+0000","last_fullsized":"2024-09-21T10:49:07.391116+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:49:17.296220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1390,"reported_epoch":1415,"state":"active+clean","last_fresh":"2024-09-21T10:48:58.051861+0000","last_change":"2024-09-21T10:07:21.524082+0000","last_active":"2024-09-21T10:48:58.051861+0000","last_peered":"2024-09-21T10:48:58.051861+0000","last_clean":"2024-09-21T10:48:58.051861+0000","last_became_active":"2024-09-21T10:07:21.523779+0000","last_became_peered":"2024-09-21T10:07:21.523779+0000","last_unstale":"2024-09-21T10:48:58.051861+0000","last_undegraded":"2024-09-21T10:48:58.051861+0000","last_fullsized":"2024-09-21T10:48:58.051861+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:13.813832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":40,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.390736+0000","last_change":"2024-09-21T10:48:48.916728+0000","last_active":"2024-09-21T10:49:07.390736+0000","last_peered":"2024-09-21T10:49:07.390736+0000","last_clean":"2024-09-21T10:49:07.390736+0000","last_became_active":"2024-09-21T10:48:48.916346+0000","last_became_peered":"2024-09-21T10:48:48.916346+0000","last_unstale":"2024-09-21T10:49:07.390736+0000","last_undegraded":"2024-09-21T10:49:07.390736+0000","last_fullsized":"2024-09-21T10:49:07.390736+0000","mapping_epoch":1409,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1410,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:16.527722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":108,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.060008+0000","last_change":"2024-09-21T10:40:55.616051+0000","last_active":"2024-09-21T10:48:56.060008+0000","last_peered":"2024-09-21T10:48:56.060008+0000","last_clean":"2024-09-21T10:48:56.060008+0000","last_became_active":"2024-09-21T10:40:55.615741+0000","last_became_peered":"2024-09-21T10:40:55.615741+0000","last_unstale":"2024-09-21T10:48:56.060008+0000","last_undegraded":"2024-09-21T10:48:56.060008+0000","last_fullsized":"2024-09-21T10:48:56.060008+0000","mapping_epoch":1320,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:07.673527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2118,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.390535+0000","last_change":"2024-09-21T10:40:13.436234+0000","last_active":"2024-09-21T10:49:07.390535+0000","last_peered":"2024-09-21T10:49:07.390535+0000","last_clean":"2024-09-21T10:49:07.390535+0000","last_became_active":"2024-09-21T10:40:13.435946+0000","last_became_peered":"2024-09-21T10:40:13.435946+0000","last_unstale":"2024-09-21T10:49:07.390535+0000","last_undegraded":"2024-09-21T10:49:07.390535+0000","last_fullsized":"2024-09-21T10:49:07.390535+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:08:08.670739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":911,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.386295+0000","last_change":"2024-09-21T10:40:13.431693+0000","last_active":"2024-09-21T10:49:07.386295+0000","last_peered":"2024-09-21T10:49:07.386295+0000","last_clean":"2024-09-21T10:49:07.386295+0000","last_became_active":"2024-09-21T10:40:13.431389+0000","last_became_peered":"2024-09-21T10:40:13.431389+0000","last_unstale":"2024-09-21T10:49:07.386295+0000","last_undegraded":"2024-09-21T10:49:07.386295+0000","last_fullsized":"2024-09-21T10:49:07.386295+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:22.227336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2158,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.393939+0000","last_change":"2024-09-21T10:44:14.409601+0000","last_active":"2024-09-21T10:49:07.393939+0000","last_peered":"2024-09-21T10:49:07.393939+0000","last_clean":"2024-09-21T10:49:07.393939+0000","last_became_active":"2024-09-21T10:44:14.409262+0000","last_became_peered":"2024-09-21T10:44:14.409262+0000","last_unstale":"2024-09-21T10:49:07.393939+0000","last_undegraded":"2024-09-21T10:49:07.393939+0000","last_fullsized":"2024-09-21T10:49:07.393939+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:53:01.809847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2125,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.393447+0000","last_change":"2024-09-21T10:09:06.560130+0000","last_active":"2024-09-21T10:49:07.393447+0000","last_peered":"2024-09-21T10:49:07.393447+0000","last_clean":"2024-09-21T10:49:07.393447+0000","last_became_active":"2024-09-21T10:09:06.559827+0000","last_became_peered":"2024-09-21T10:09:06.559827+0000","last_unstale":"2024-09-21T10:49:07.393447+0000","last_undegraded":"2024-09-21T10:49:07.393447+0000","last_fullsized":"2024-09-21T10:49:07.393447+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:05:17.166734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":58,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.059814+0000","last_change":"2024-09-21T10:44:14.400691+0000","last_active":"2024-09-21T10:48:56.059814+0000","last_peered":"2024-09-21T10:48:56.059814+0000","last_clean":"2024-09-21T10:48:56.059814+0000","last_became_active":"2024-09-21T10:44:14.400377+0000","last_became_peered":"2024-09-21T10:44:14.400377+0000","last_unstale":"2024-09-21T10:48:56.059814+0000","last_undegraded":"2024-09-21T10:48:56.059814+0000","last_fullsized":"2024-09-21T10:48:56.059814+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:03.555078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1391,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.061238+0000","last_change":"2024-09-21T10:44:14.402965+0000","last_active":"2024-09-21T10:48:56.061238+0000","last_peered":"2024-09-21T10:48:56.061238+0000","last_clean":"2024-09-21T10:48:56.061238+0000","last_became_active":"2024-09-21T10:44:14.402634+0000","last_became_peered":"2024-09-21T10:44:14.402634+0000","last_unstale":"2024-09-21T10:48:56.061238+0000","last_undegraded":"2024-09-21T10:48:56.061238+0000","last_fullsized":"2024-09-21T10:48:56.061238+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:40:16.323155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2105,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.386842+0000","last_change":"2024-09-21T10:09:06.560589+0000","last_active":"2024-09-21T10:49:07.386842+0000","last_peered":"2024-09-21T10:49:07.386842+0000","last_clean":"2024-09-21T10:49:07.386842+0000","last_became_active":"2024-09-21T10:09:06.557552+0000","last_became_peered":"2024-09-21T10:09:06.557552+0000","last_unstale":"2024-09-21T10:49:07.386842+0000","last_undegraded":"2024-09-21T10:49:07.386842+0000","last_fullsized":"2024-09-21T10:49:07.386842+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:09:24.866759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":1399,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.061372+0000","last_change":"2024-09-21T10:48:48.914925+0000","last_active":"2024-09-21T10:48:56.061372+0000","last_peered":"2024-09-21T10:48:56.061372+0000","last_clean":"2024-09-21T10:48:56.061372+0000","last_became_active":"2024-09-21T10:48:48.914617+0000","last_became_peered":"2024-09-21T10:48:48.914617+0000","last_unstale":"2024-09-21T10:48:56.061372+0000","last_undegraded":"2024-09-21T10:48:56.061372+0000","last_fullsized":"2024-09-21T10:48:56.061372+0000","mapping_epoch":1409,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1410,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:34:18.035121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":808,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.059793+0000","last_change":"2024-09-21T10:44:17.413348+0000","last_active":"2024-09-21T10:48:56.059793+0000","last_peered":"2024-09-21T10:48:56.059793+0000","last_clean":"2024-09-21T10:48:56.059793+0000","last_became_active":"2024-09-21T10:44:17.412916+0000","last_became_peered":"2024-09-21T10:44:17.412916+0000","last_unstale":"2024-09-21T10:48:56.059793+0000","last_undegraded":"2024-09-21T10:48:56.059793+0000","last_fullsized":"2024-09-21T10:48:56.059793+0000","mapping_epoch":1372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1373,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:18.592283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2109,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.391144+0000","last_change":"2024-09-21T10:09:06.538940+0000","last_active":"2024-09-21T10:49:07.391144+0000","last_peered":"2024-09-21T10:49:07.391144+0000","last_clean":"2024-09-21T10:49:07.391144+0000","last_became_active":"2024-09-21T10:09:06.538668+0000","last_became_peered":"2024-09-21T10:09:06.538668+0000","last_unstale":"2024-09-21T10:49:07.391144+0000","last_undegraded":"2024-09-21T10:49:07.391144+0000","last_fullsized":"2024-09-21T10:49:07.391144+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:42:44.299488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2124,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.386712+0000","last_change":"2024-09-21T10:44:14.398581+0000","last_active":"2024-09-21T10:49:07.386712+0000","last_peered":"2024-09-21T10:49:07.386712+0000","last_clean":"2024-09-21T10:49:07.386712+0000","last_became_active":"2024-09-21T10:44:14.398258+0000","last_became_peered":"2024-09-21T10:44:14.398258+0000","last_unstale":"2024-09-21T10:49:07.386712+0000","last_undegraded":"2024-09-21T10:49:07.386712+0000","last_fullsized":"2024-09-21T10:49:07.386712+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:23.960956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1387,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.108385+0000","last_change":"2024-09-21T10:07:55.998799+0000","last_active":"2024-09-21T10:48:56.108385+0000","last_peered":"2024-09-21T10:48:56.108385+0000","last_clean":"2024-09-21T10:48:56.108385+0000","last_became_active":"2024-09-21T10:07:55.998510+0000","last_became_peered":"2024-09-21T10:07:55.998510+0000","last_unstale":"2024-09-21T10:48:56.108385+0000","last_undegraded":"2024-09-21T10:48:56.108385+0000","last_fullsized":"2024-09-21T10:48:56.108385+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:10.474008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1416,"reported_epoch":1415,"state":"active+clean","last_fresh":"2024-09-21T10:48:58.051782+0000","last_change":"2024-09-21T10:44:14.399842+0000","last_active":"2024-09-21T10:48:58.051782+0000","last_peered":"2024-09-21T10:48:58.051782+0000","last_clean":"2024-09-21T10:48:58.051782+0000","last_became_active":"2024-09-21T10:44:14.399426+0000","last_became_peered":"2024-09-21T10:44:14.399426+0000","last_unstale":"2024-09-21T10:48:58.051782+0000","last_undegraded":"2024-09-21T10:48:58.051782+0000","last_fullsized":"2024-09-21T10:48:58.051782+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:26:45.362101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1411,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.108375+0000","last_change":"2024-09-21T10:40:13.432387+0000","last_active":"2024-09-21T10:48:56.108375+0000","last_peered":"2024-09-21T10:48:56.108375+0000","last_clean":"2024-09-21T10:48:56.108375+0000","last_became_active":"2024-09-21T10:40:13.432053+0000","last_became_peered":"2024-09-21T10:40:13.432053+0000","last_unstale":"2024-09-21T10:48:56.108375+0000","last_undegraded":"2024-09-21T10:48:56.108375+0000","last_fullsized":"2024-09-21T10:48:56.108375+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:37.486101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1388,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.060664+0000","last_change":"2024-09-21T10:07:56.020791+0000","last_active":"2024-09-21T10:48:56.060664+0000","last_peered":"2024-09-21T10:48:56.060664+0000","last_clean":"2024-09-21T10:48:56.060664+0000","last_became_active":"2024-09-21T10:07:56.020205+0000","last_became_peered":"2024-09-21T10:07:56.020205+0000","last_unstale":"2024-09-21T10:48:56.060664+0000","last_undegraded":"2024-09-21T10:48:56.060664+0000","last_fullsized":"2024-09-21T10:48:56.060664+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:09.813166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1387,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.108348+0000","last_change":"2024-09-21T10:09:06.533271+0000","last_active":"2024-09-21T10:48:56.108348+0000","last_peered":"2024-09-21T10:48:56.108348+0000","last_clean":"2024-09-21T10:48:56.108348+0000","last_became_active":"2024-09-21T10:09:06.532041+0000","last_became_peered":"2024-09-21T10:09:06.532041+0000","last_unstale":"2024-09-21T10:48:56.108348+0000","last_undegraded":"2024-09-21T10:48:56.108348+0000","last_fullsized":"2024-09-21T10:48:56.108348+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:04.236375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":2125,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.385330+0000","last_change":"2024-09-21T10:09:06.543422+0000","last_active":"2024-09-21T10:49:07.385330+0000","last_peered":"2024-09-21T10:49:07.385330+0000","last_clean":"2024-09-21T10:49:07.385330+0000","last_became_active":"2024-09-21T10:09:06.539647+0000","last_became_peered":"2024-09-21T10:09:06.539647+0000","last_unstale":"2024-09-21T10:49:07.385330+0000","last_undegraded":"2024-09-21T10:49:07.385330+0000","last_fullsized":"2024-09-21T10:49:07.385330+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:04.990085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2125,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.393351+0000","last_change":"2024-09-21T10:07:56.018560+0000","last_active":"2024-09-21T10:49:07.393351+0000","last_peered":"2024-09-21T10:49:07.393351+0000","last_clean":"2024-09-21T10:49:07.393351+0000","last_became_active":"2024-09-21T10:07:56.018261+0000","last_became_peered":"2024-09-21T10:07:56.018261+0000","last_unstale":"2024-09-21T10:49:07.393351+0000","last_undegraded":"2024-09-21T10:49:07.393351+0000","last_fullsized":"2024-09-21T10:49:07.393351+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:04:54.395456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":665,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.385909+0000","last_change":"2024-09-21T10:48:48.917105+0000","last_active":"2024-09-21T10:49:07.385909+0000","last_peered":"2024-09-21T10:49:07.385909+0000","last_clean":"2024-09-21T10:49:07.385909+0000","last_became_active":"2024-09-21T10:48:48.916647+0000","last_became_peered":"2024-09-21T10:48:48.916647+0000","last_unstale":"2024-09-21T10:49:07.385909+0000","last_undegraded":"2024-09-21T10:49:07.385909+0000","last_fullsized":"2024-09-21T10:49:07.385909+0000","mapping_epoch":1409,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1410,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:09:04.179209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1423,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.060543+0000","last_change":"2024-09-21T10:44:14.401913+0000","last_active":"2024-09-21T10:48:56.060543+0000","last_peered":"2024-09-21T10:48:56.060543+0000","last_clean":"2024-09-21T10:48:56.060543+0000","last_became_active":"2024-09-21T10:44:14.401496+0000","last_became_peered":"2024-09-21T10:44:14.401496+0000","last_unstale":"2024-09-21T10:48:56.060543+0000","last_undegraded":"2024-09-21T10:48:56.060543+0000","last_fullsized":"2024-09-21T10:48:56.060543+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:07:38.531723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":62,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.061534+0000","last_change":"2024-09-21T10:44:14.410444+0000","last_active":"2024-09-21T10:48:56.061534+0000","last_peered":"2024-09-21T10:48:56.061534+0000","last_clean":"2024-09-21T10:48:56.061534+0000","last_became_active":"2024-09-21T10:44:14.395465+0000","last_became_peered":"2024-09-21T10:44:14.395465+0000","last_unstale":"2024-09-21T10:48:56.061534+0000","last_undegraded":"2024-09-21T10:48:56.061534+0000","last_fullsized":"2024-09-21T10:48:56.061534+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:05:44.575878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1403,"reported_epoch":1415,"state":"active+clean","last_fresh":"2024-09-21T10:48:58.051462+0000","last_change":"2024-09-21T10:09:06.559132+0000","last_active":"2024-09-21T10:48:58.051462+0000","last_peered":"2024-09-21T10:48:58.051462+0000","last_clean":"2024-09-21T10:48:58.051462+0000","last_became_active":"2024-09-21T10:09:06.558395+0000","last_became_peered":"2024-09-21T10:09:06.558395+0000","last_unstale":"2024-09-21T10:48:58.051462+0000","last_undegraded":"2024-09-21T10:48:58.051462+0000","last_fullsized":"2024-09-21T10:48:58.051462+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:27:00.073554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":62,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.061634+0000","last_change":"2024-09-21T10:44:14.419937+0000","last_active":"2024-09-21T10:48:56.061634+0000","last_peered":"2024-09-21T10:48:56.061634+0000","last_clean":"2024-09-21T10:48:56.061634+0000","last_became_active":"2024-09-21T10:44:14.402761+0000","last_became_peered":"2024-09-21T10:44:14.402761+0000","last_unstale":"2024-09-21T10:48:56.061634+0000","last_undegraded":"2024-09-21T10:48:56.061634+0000","last_fullsized":"2024-09-21T10:48:56.061634+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:42.875310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1351,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.059614+0000","last_change":"2024-09-21T10:48:48.916726+0000","last_active":"2024-09-21T10:48:56.059614+0000","last_peered":"2024-09-21T10:48:56.059614+0000","last_clean":"2024-09-21T10:48:56.059614+0000","last_became_active":"2024-09-21T10:48:48.916433+0000","last_became_peered":"2024-09-21T10:48:48.916433+0000","last_unstale":"2024-09-21T10:48:56.059614+0000","last_undegraded":"2024-09-21T10:48:56.059614+0000","last_fullsized":"2024-09-21T10:48:56.059614+0000","mapping_epoch":1409,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1410,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:09:45.836913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":62,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.061769+0000","last_change":"2024-09-21T10:44:14.421612+0000","last_active":"2024-09-21T10:48:56.061769+0000","last_peered":"2024-09-21T10:48:56.061769+0000","last_clean":"2024-09-21T10:48:56.061769+0000","last_became_active":"2024-09-21T10:44:14.406095+0000","last_became_peered":"2024-09-21T10:44:14.406095+0000","last_unstale":"2024-09-21T10:48:56.061769+0000","last_undegraded":"2024-09-21T10:48:56.061769+0000","last_fullsized":"2024-09-21T10:48:56.061769+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:21:19.918156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1395,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.061119+0000","last_change":"2024-09-21T10:07:21.497877+0000","last_active":"2024-09-21T10:48:56.061119+0000","last_peered":"2024-09-21T10:48:56.061119+0000","last_clean":"2024-09-21T10:48:56.061119+0000","last_became_active":"2024-09-21T10:07:21.496317+0000","last_became_peered":"2024-09-21T10:07:21.496317+0000","last_unstale":"2024-09-21T10:48:56.061119+0000","last_undegraded":"2024-09-21T10:48:56.061119+0000","last_fullsized":"2024-09-21T10:48:56.061119+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:39:03.848183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2120,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.385981+0000","last_change":"2024-09-21T10:44:14.398487+0000","last_active":"2024-09-21T10:49:07.385981+0000","last_peered":"2024-09-21T10:49:07.385981+0000","last_clean":"2024-09-21T10:49:07.385981+0000","last_became_active":"2024-09-21T10:44:14.398142+0000","last_became_peered":"2024-09-21T10:44:14.398142+0000","last_unstale":"2024-09-21T10:49:07.385981+0000","last_undegraded":"2024-09-21T10:49:07.385981+0000","last_fullsized":"2024-09-21T10:49:07.385981+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:20:41.065055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":58,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.355468+0000","last_change":"2024-09-21T10:44:14.398878+0000","last_active":"2024-09-21T10:48:56.355468+0000","last_peered":"2024-09-21T10:48:56.355468+0000","last_clean":"2024-09-21T10:48:56.355468+0000","last_became_active":"2024-09-21T10:44:14.398547+0000","last_became_peered":"2024-09-21T10:44:14.398547+0000","last_unstale":"2024-09-21T10:48:56.355468+0000","last_undegraded":"2024-09-21T10:48:56.355468+0000","last_fullsized":"2024-09-21T10:48:56.355468+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:13.008134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2152,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.393686+0000","last_change":"2024-09-21T10:44:14.409051+0000","last_active":"2024-09-21T10:49:07.393686+0000","last_peered":"2024-09-21T10:49:07.393686+0000","last_clean":"2024-09-21T10:49:07.393686+0000","last_became_active":"2024-09-21T10:44:14.407892+0000","last_became_peered":"2024-09-21T10:44:14.407892+0000","last_unstale":"2024-09-21T10:49:07.393686+0000","last_undegraded":"2024-09-21T10:49:07.393686+0000","last_fullsized":"2024-09-21T10:49:07.393686+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:19.549648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":2135,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.393388+0000","last_change":"2024-09-21T10:44:14.408837+0000","last_active":"2024-09-21T10:49:07.393388+0000","last_peered":"2024-09-21T10:49:07.393388+0000","last_clean":"2024-09-21T10:49:07.393388+0000","last_became_active":"2024-09-21T10:44:14.408399+0000","last_became_peered":"2024-09-21T10:44:14.408399+0000","last_unstale":"2024-09-21T10:49:07.393388+0000","last_undegraded":"2024-09-21T10:49:07.393388+0000","last_fullsized":"2024-09-21T10:49:07.393388+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:41:38.729386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1364,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.061714+0000","last_change":"2024-09-21T10:48:48.913881+0000","last_active":"2024-09-21T10:48:56.061714+0000","last_peered":"2024-09-21T10:48:56.061714+0000","last_clean":"2024-09-21T10:48:56.061714+0000","last_became_active":"2024-09-21T10:48:48.913580+0000","last_became_peered":"2024-09-21T10:48:48.913580+0000","last_unstale":"2024-09-21T10:48:56.061714+0000","last_undegraded":"2024-09-21T10:48:56.061714+0000","last_fullsized":"2024-09-21T10:48:56.061714+0000","mapping_epoch":1409,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1410,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:12:17.910460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":64,"reported_epoch":1414,"state":"active+clean","last_fresh":"2024-09-21T10:48:56.355485+0000","last_change":"2024-09-21T10:44:14.418113+0000","last_active":"2024-09-21T10:48:56.355485+0000","last_peered":"2024-09-21T10:48:56.355485+0000","last_clean":"2024-09-21T10:48:56.355485+0000","last_became_active":"2024-09-21T10:44:14.398327+0000","last_became_peered":"2024-09-21T10:44:14.398327+0000","last_unstale":"2024-09-21T10:48:56.355485+0000","last_undegraded":"2024-09-21T10:48:56.355485+0000","last_fullsized":"2024-09-21T10:48:56.355485+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:08.810863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2121,"reported_epoch":1416,"state":"active+clean","last_fresh":"2024-09-21T10:49:07.386041+0000","last_change":"2024-09-21T10:44:14.398584+0000","last_active":"2024-09-21T10:49:07.386041+0000","last_peered":"2024-09-21T10:49:07.386041+0000","last_clean":"2024-09-21T10:49:07.386041+0000","last_became_active":"2024-09-21T10:44:14.398278+0000","last_became_peered":"2024-09-21T10:44:14.398278+0000","last_unstale":"2024-09-21T10:49:07.386041+0000","last_undegraded":"2024-09-21T10:49:07.386041+0000","last_fullsized":"2024-09-21T10:49:07.386041+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:19.681750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":55,"num_pg":16,"stat_sum":{"num_bytes":1407254528,"num_objects":21473,"num_object_clones":0,"num_object_copies":64419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21473,"num_write_kb":1374272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4022075392,"data_stored":4022075392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21473,"ondisk_log_size":21473,"up":48,"acting":48,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":10310,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234661,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1183200,"kb_used_data":867480,"kb_used_omap":1,"kb_used_meta":315710,"kb_avail":92561952,"statfs":{"total":95995035648,"available":94783438848,"internally_reserved":0,"allocated":888299520,"data_stored":883719188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":323287499},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496298,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1137184,"kb_used_data":622936,"kb_used_omap":1,"kb_used_meta":514238,"kb_avail":92607968,"statfs":{"total":95995035648,"available":94830559232,"internally_reserved":0,"allocated":637886464,"data_stored":633306621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":526580170},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757935,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":785676,"kb_used_data":349316,"kb_used_omap":1,"kb_used_meta":436350,"kb_avail":92959476,"statfs":{"total":95995035648,"available":95190503424,"internally_reserved":0,"allocated":357699584,"data_stored":353134887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":446822854},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019571,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":858392,"kb_used_data":354000,"kb_used_omap":1,"kb_used_meta":504382,"kb_avail":92886760,"statfs":{"total":95995035648,"available":95116042240,"internally_reserved":0,"allocated":362496000,"data_stored":357923246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":516487622},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369315,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1296980,"kb_used_data":773388,"kb_used_omap":69,"kb_used_meta":523514,"kb_avail":92448172,"statfs":{"total":95995035648,"available":94666928128,"internally_reserved":0,"allocated":791949312,"data_stored":787384566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":536078466},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2920577761770,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807958001,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1009428,"kb_used_data":562316,"kb_used_omap":66,"kb_used_meta":447037,"kb_avail":92735724,"statfs":{"total":95995035648,"available":94961381376,"internally_reserved":0,"allocated":575811584,"data_stored":571246767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68589,"internal_metadata":457765907},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":666,"seq":2860448219640,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":706140,"kb_used_data":295316,"kb_used_omap":76,"kb_used_meta":410739,"kb_avail":93039012,"statfs":{"total":95995035648,"available":95271948288,"internally_reserved":0,"allocated":302403584,"data_stored":297831704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78716,"internal_metadata":420596868},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":312213504,"data_stored":312213504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":592904192,"data_stored":592904192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":826015744,"data_stored":826015744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":355074048,"data_stored":355074048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":371982336,"data_stored":371982336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":675086336,"data_stored":675086336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":888799232,"data_stored":888799232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:49:14.837 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T10:49:14.837 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pg_num 11 2024-09-21T10:49:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:14 smithi160 ceph-mon[101879]: pgmap v2526: 89 pgs: 89 active+clean; 1.6 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-09-21T10:49:14.925 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:14.963 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:14.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:14.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:15.065 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:15.095 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:15.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:15.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:15.196 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:15.232 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:15.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:15.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:15.334 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:15.369 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:15.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:15.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:15.471 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:15.500 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:15.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:15.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:15.602 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:15.631 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:15.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:15.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:15.733 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:15.768 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:15.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:15.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:15.801 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 15 16 37002 36986 154.095 127 0.00419978 0.00648393 2024-09-21T10:49:15.869 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:15.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:15.911 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:15.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:16.012 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:16.048 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:16.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:16.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:16.150 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:16.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:16.189 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:16.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:16.290 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:16.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:16.330 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:16.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:16.344 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:16.431 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:16.463 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:16.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:16.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:16.564 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:16.597 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:16.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:16.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:16.699 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:16.720 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:16.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:16.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:16.801 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 16 16 38869 38853 151.757 116.688 0.00548934 0.00658339 2024-09-21T10:49:16.822 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:16.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:16 smithi160 ceph-mon[101879]: pgmap v2527: 89 pgs: 89 active+clean; 1.7 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 156 MiB/s wr, 2.50k op/s 2024-09-21T10:49:16.860 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:16.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:16.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:16.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:16 smithi070 ceph-mon[107385]: pgmap v2527: 89 pgs: 89 active+clean; 1.7 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 156 MiB/s wr, 2.50k op/s 2024-09-21T10:49:16.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:16 smithi070 ceph-mon[112139]: pgmap v2527: 89 pgs: 89 active+clean; 1.7 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 156 MiB/s wr, 2.50k op/s 2024-09-21T10:49:16.962 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:17.000 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:17.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:17.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:17.102 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:17.131 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:17.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:17.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:17.233 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:17.261 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:17.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:17.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:17.363 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:17.384 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:17.395 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:17.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:17.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:17.497 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:17.530 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:17.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:17.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:17.632 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:17.675 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:17.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:17.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:17.776 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:17.801 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 17 16 40694 40678 149.539 114.062 0.0192699 0.00668162 2024-09-21T10:49:17.817 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:17.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:17.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:17.919 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:17.958 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:17.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:17.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:18.059 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:18.092 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:18.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:18.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:18.175 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:49:18.176 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:49:18.194 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:18.203 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:49:18.203 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:49:18.227 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:18.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:18.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:18.232 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:49:18.233 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:49:18.259 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:49:18.259 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:49:18.288 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:49:18.288 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:49:18.316 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:49:18.316 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:49:18.330 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:18.346 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:49:18.346 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:49:18.367 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:18.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:18.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:18.375 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:49:18.375 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:49:18.428 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:18 smithi070 ceph-mon[107385]: pgmap v2528: 89 pgs: 89 active+clean; 2.2 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-09-21T10:49:18.428 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:18 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1395197784' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:49:18.428 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:18 smithi070 ceph-mon[112139]: pgmap v2528: 89 pgs: 89 active+clean; 2.2 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-09-21T10:49:18.428 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:18 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1395197784' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:49:18.470 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:18.503 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:18.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:18.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:18.605 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:18.637 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:18.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:18.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:18.738 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:18.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:18.773 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:18.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:18.801 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 18 16 42577 42561 147.769 117.688 0.00737031 0.00676165 2024-09-21T10:49:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:18 smithi160 ceph-mon[101879]: pgmap v2528: 89 pgs: 89 active+clean; 2.2 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-09-21T10:49:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:18 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1395197784' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:49:18.874 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:18.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:18.907 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:18.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:19.008 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:19.041 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:19.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:19.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:19.143 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:19.180 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:19.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:19.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:19.282 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:19.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:19.320 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:19.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:19.421 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:19.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:19.453 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:19.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:19.554 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:19.578 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:19.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:19.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:19.679 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:19.715 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:19.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:19.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:19.801 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 19 16 44327 44311 145.747 109.375 0.00808821 0.0068564 2024-09-21T10:49:19.817 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:19.847 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:19.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:19.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:19.948 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:19.978 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:19.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:19.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:20.080 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:20.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:20.111 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:20.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:20.213 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:20.248 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:20.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:20.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:20.350 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:20.380 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:20.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:20.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:20.481 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:20.511 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:20.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:20.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:20.612 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:20.641 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:20.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:20.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:20.742 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:20.772 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:20.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:20.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:20.801 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:2024-09-21T10:49:20.800463+0000 min lat: 0.00113626 max lat: 0.385227 avg lat: 0.00694317 2024-09-21T10:49:20.802 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:49:20.802 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 20 16 46077 46061 143.928 109.375 0.00965755 0.00694317 2024-09-21T10:49:20.827 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:49:20.827 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:49:20.828 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:20 smithi070 ceph-mon[112139]: pgmap v2529: 89 pgs: 89 active+clean; 2.2 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 112 MiB/s wr, 1.80k op/s 2024-09-21T10:49:20.828 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:49:20.828 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:49:20.828 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:20 smithi070 ceph-mon[107385]: pgmap v2529: 89 pgs: 89 active+clean; 2.2 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 112 MiB/s wr, 1.80k op/s 2024-09-21T10:49:20.828 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:49:20.828 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:49:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:20 smithi160 ceph-mon[101879]: pgmap v2529: 89 pgs: 89 active+clean; 2.2 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 112 MiB/s wr, 1.80k op/s 2024-09-21T10:49:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:49:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:49:20.875 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:20.905 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:20.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:20.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:21.006 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:21.043 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:21.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:21.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:21.145 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:21.172 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:21.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:21.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:21.274 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:21.305 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:21.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:21.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:21.407 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:21.443 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:21.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:21.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:21.545 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:21.582 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:21.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:21.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:21.683 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:21.721 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:21.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:21.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:21.802 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 21 16 47765 47749 142.098 105.5 0.0120689 0.00703294 2024-09-21T10:49:21.823 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:21.862 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:21.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:21.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:21.964 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:22.004 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:22.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:22.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:22.106 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:22.135 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:22.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:22.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:22.237 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:22.266 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:22.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:22.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:22.368 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:22.400 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:22.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:22.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:22.501 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:22.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:22.541 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:22.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:22 smithi070 ceph-mon[107385]: pgmap v2530: 89 pgs: 89 active+clean; 2.5 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 131 MiB/s wr, 2.10k op/s 2024-09-21T10:49:22.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:22 smithi070 ceph-mon[112139]: pgmap v2530: 89 pgs: 89 active+clean; 2.5 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 131 MiB/s wr, 2.10k op/s 2024-09-21T10:49:22.642 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:22.680 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:22.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:22.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:22.702 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:49:22.781 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:22.802 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 22 16 49428 49412 140.363 103.938 0.0102362 0.0071191 2024-09-21T10:49:22.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:22.818 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:22.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:22 smithi160 ceph-mon[101879]: pgmap v2530: 89 pgs: 89 active+clean; 2.5 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 131 MiB/s wr, 2.10k op/s 2024-09-21T10:49:22.919 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:22.947 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:22.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:22.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:23.049 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:23.095 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:23.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:23.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:23.197 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:23.236 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:23.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:23.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:23.338 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:23.377 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:23.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:49:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:49:23] "GET /metrics HTTP/1.1" 200 36596 "" "Prometheus/2.51.0" 2024-09-21T10:49:23.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:23.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:23.479 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:23.508 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:23.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:23.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:23.610 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:23.656 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:23.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:23.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:23.758 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:23.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:23.798 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:23.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:23.801 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 23 16 50880 50864 138.205 90.75 0.0147159 0.00723081 2024-09-21T10:49:23.822 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:23.899 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:23.926 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:23.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:23.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:24.028 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:24.058 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:24.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:24.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:24.059 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:49:24.160 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:24.202 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:24.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:24.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:24.304 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:24.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:24.343 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:24.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:24.444 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:24.482 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:24.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:24.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:24.584 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:24.624 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:24.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:24.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:24.693 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:24 smithi160 ceph-mon[101879]: pgmap v2531: 89 pgs: 89 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-09-21T10:49:24.726 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:24.728 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:24 smithi070 ceph-mon[112139]: pgmap v2531: 89 pgs: 89 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-09-21T10:49:24.728 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:24 smithi070 ceph-mon[107385]: pgmap v2531: 89 pgs: 89 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-09-21T10:49:24.747 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:24.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:24.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:24.802 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 24 16 52206 52190 135.9 82.875 0.0101594 0.00735323 2024-09-21T10:49:24.849 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:24.883 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:24.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:24.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:24.985 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:25.021 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:25.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:25.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:25.122 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:25.154 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:25.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:25.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:25.255 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:25.287 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:25.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:25.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:25.389 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:25.424 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:25.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:25.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:25.526 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:25.566 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:25.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:25.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:25.667 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:25.707 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:25.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:25.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:25.802 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 25 16 53621 53605 134.001 88.4375 0.00383121 0.00743165 2024-09-21T10:49:25.808 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:25.830 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:25.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:25.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:25.931 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:25.959 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:25.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:25.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:26.061 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:26.096 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:26.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:26.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:26.198 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:26.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:26.235 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:26.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:26.336 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:26.358 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:26.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:26.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:26.459 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:26.496 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:26.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:26.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:26.566 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:26.598 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:26.629 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:26.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:26.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:26.730 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:26.751 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:26 smithi070 ceph-mon[107385]: pgmap v2532: 89 pgs: 89 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-09-21T10:49:26.751 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:26 smithi070 ceph-mon[112139]: pgmap v2532: 89 pgs: 89 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-09-21T10:49:26.751 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:26.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:26.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:26.802 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 26 16 54697 54681 131.433 67.25 0.00681255 0.00760137 2024-09-21T10:49:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:26 smithi160 ceph-mon[101879]: pgmap v2532: 89 pgs: 89 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-09-21T10:49:26.853 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:26.884 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:26.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:26.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:26.986 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:27.016 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:27.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:27.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:27.118 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:27.157 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:27.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:27.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:27.259 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:27.288 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:27.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:27.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:27.390 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:27.420 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:27.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:27.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:27.429 INFO:teuthology.orchestra.run.smithi070.stderr:set pool 1 pg_num to 11 2024-09-21T10:49:27.522 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:27.559 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:27.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:27.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:27.661 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:27.697 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:27.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:27.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:27.799 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:27.802 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 27 16 56240 56224 130.137 96.4375 0.0128297 0.00767905 2024-09-21T10:49:27.832 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:27.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:27.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:27 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3073380401' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T10:49:27.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:27 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3073380401' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T10:49:27.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:27 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3073380401' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T10:49:27.934 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:27.969 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:27.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:27.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:28.071 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:28.100 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:28.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:28.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:28.202 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:28.242 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:28.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:28.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:28.344 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:28.381 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:28.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:28.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:28.447 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T10:49:28.466 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:28 smithi070 ceph-mon[107385]: pgmap v2533: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-09-21T10:49:28.467 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:28 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3073380401' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-21T10:49:28.467 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:28 smithi070 ceph-mon[107385]: osdmap e1417: 8 total, 8 up, 7 in 2024-09-21T10:49:28.467 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:28 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1355563753' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:49:28.467 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:28 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:49:28.467 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:28 smithi070 ceph-mon[112139]: pgmap v2533: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-09-21T10:49:28.467 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:28 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3073380401' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-21T10:49:28.467 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:28 smithi070 ceph-mon[112139]: osdmap e1417: 8 total, 8 up, 7 in 2024-09-21T10:49:28.467 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:28 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1355563753' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:49:28.467 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:28 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:49:28.483 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:28.513 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:28.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:28.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:28.615 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:28.657 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:28.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:28.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:28.758 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:28.796 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:28.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:28.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:28.802 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 28 16 57735 57719 128.826 93.4375 0.0160943 0.00775802 2024-09-21T10:49:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:28 smithi160 ceph-mon[101879]: pgmap v2533: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-09-21T10:49:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:28 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3073380401' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-21T10:49:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:28 smithi160 ceph-mon[101879]: osdmap e1417: 8 total, 8 up, 7 in 2024-09-21T10:49:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:28 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1355563753' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:49:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:28 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:49:28.898 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:28.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:28.937 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:28.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:29.038 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:29.066 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:29.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:29.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:29.168 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:29.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:29.198 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:29.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:29.298 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:29.327 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:29.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:29.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:29.430 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:29.472 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:29.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:29.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:29.574 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:29.614 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:29.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:29.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:29.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:29 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:49:29.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:29 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:49:29.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:29 smithi070 ceph-mon[112139]: osdmap e1418: 8 total, 8 up, 7 in 2024-09-21T10:49:29.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:29 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-21T10:49:29.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:29 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:49:29.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:29 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:49:29.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:29 smithi070 ceph-mon[107385]: osdmap e1418: 8 total, 8 up, 7 in 2024-09-21T10:49:29.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:29 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-21T10:49:29.715 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:29.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:29.750 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:29.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:29.802 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 29 16 59227 59211 127.599 93.25 0.00719273 0.00783245 2024-09-21T10:49:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:29 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:49:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:29 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:49:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:29 smithi160 ceph-mon[101879]: osdmap e1418: 8 total, 8 up, 7 in 2024-09-21T10:49:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:29 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-21T10:49:29.852 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:29.880 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:29.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:29.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:29.982 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:30.010 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:30.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:30.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:30.112 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:30.147 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:30.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:30.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:30.249 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:30.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:30.283 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:30.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:30.384 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:30.418 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:30.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:30.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:30.519 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:30.557 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:30.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:30.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:30.659 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:30.689 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:30.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:30.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:30.791 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:30.800 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:30 smithi070 ceph-mon[107385]: pgmap v2536: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-21T10:49:30.801 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:30 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-21T10:49:30.801 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:30 smithi070 ceph-mon[107385]: osdmap e1419: 8 total, 8 up, 7 in 2024-09-21T10:49:30.801 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:30 smithi070 ceph-mon[112139]: pgmap v2536: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-21T10:49:30.801 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:30 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-21T10:49:30.801 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:30 smithi070 ceph-mon[112139]: osdmap e1419: 8 total, 8 up, 7 in 2024-09-21T10:49:30.802 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 30 16 60717 60701 126.449 93.125 0.00653413 0.00790333 2024-09-21T10:49:30.820 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:30.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:30.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:30 smithi160 ceph-mon[101879]: pgmap v2536: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-21T10:49:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:30 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-21T10:49:30.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:30 smithi160 ceph-mon[101879]: osdmap e1419: 8 total, 8 up, 7 in 2024-09-21T10:49:30.922 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:30.958 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:30.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:30.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:31.060 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:31.104 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:31.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:31.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:31.206 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:31.247 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:31.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:31.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:31.341 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:49:31.349 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:31.387 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:31.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:31.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:31.488 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:31.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:31.527 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:31.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:31.612 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:31 smithi070 ceph-mon[107385]: osdmap e1420: 8 total, 8 up, 7 in 2024-09-21T10:49:31.612 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:31 smithi070 ceph-mon[112139]: osdmap e1420: 8 total, 8 up, 7 in 2024-09-21T10:49:31.628 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:31.673 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:31.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:31.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:31.774 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:31.802 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:31.803 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 31 16 62077 62061 125.112 85 0.0105251 0.00798799 2024-09-21T10:49:31.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:31.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:31 smithi160 ceph-mon[101879]: osdmap e1420: 8 total, 8 up, 7 in 2024-09-21T10:49:31.904 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:31.949 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:31.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:31.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:32.051 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:32.083 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:32.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:32.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:32.185 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:32.215 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:32.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:32.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:32.316 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:32.344 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:32.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:32.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:32.446 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:32.476 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:32.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:32.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:32.579 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:32.602 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:32.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:32.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:32.704 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:32.740 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:32 smithi070 ceph-mon[112139]: pgmap v2539: 99 pgs: 10 unknown, 89 active+clean; 3.5 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 53 MiB/s wr, 842 op/s 2024-09-21T10:49:32.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:32 smithi070 ceph-mon[107385]: pgmap v2539: 99 pgs: 10 unknown, 89 active+clean; 3.5 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 53 MiB/s wr, 842 op/s 2024-09-21T10:49:32.740 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:32.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:32.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:32.802 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 32 15 63356 63341 123.702 80 0.0159296 0.00807879 2024-09-21T10:49:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:32 smithi160 ceph-mon[101879]: pgmap v2539: 99 pgs: 10 unknown, 89 active+clean; 3.5 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 53 MiB/s wr, 842 op/s 2024-09-21T10:49:32.842 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:32.879 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:32.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:32.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:32.981 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:33.005 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:33.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:33.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:33.106 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:33.135 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:33.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:33.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:33.239 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:33.283 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:33.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:33.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:33.385 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:33.419 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:33.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:33.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:33.435 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:49:33.435 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:49:33.435 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:49:33.435 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:49:33.436 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:49:33.436 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:49:33.436 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:49:33.436 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:49:33.436 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:49:33.436 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:49:33.436 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:49:33.436 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:49:33.436 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:49:33.437 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:49:33.437 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:49:33.437 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:49:33.437 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:49:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:49:33] "GET /metrics HTTP/1.1" 200 36596 "" "Prometheus/2.51.0" 2024-09-21T10:49:33.521 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:33.541 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:33.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:33.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:33.643 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:33.670 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:33.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:33.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:33.772 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:33.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:33.799 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:33.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:33.802 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 33 16 64478 64462 122.077 70.0625 0.0165537 0.00818609 2024-09-21T10:49:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:33 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2823377196' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:49:33.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:33 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-21T10:49:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:33 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2823377196' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:49:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:33 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-21T10:49:33.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:33 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2823377196' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:49:33.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:33 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-21T10:49:33.900 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:33.927 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:33.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:33.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:34.029 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:34.061 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:34.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:34.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:34.162 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:34.190 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:34.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:34.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:34.292 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:34.327 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:34.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:34.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:34.429 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:34.467 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:34.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:34.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:34.569 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:34.604 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:34.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:34.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:34.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:34 smithi070 ceph-mon[107385]: pgmap v2540: 99 pgs: 99 active+clean; 3.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-21T10:49:34.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-21T10:49:34.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:34 smithi070 ceph-mon[107385]: osdmap e1421: 8 total, 8 up, 7 in 2024-09-21T10:49:34.608 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:34 smithi070 ceph-mon[112139]: pgmap v2540: 99 pgs: 99 active+clean; 3.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-21T10:49:34.608 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-21T10:49:34.608 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:34 smithi070 ceph-mon[112139]: osdmap e1421: 8 total, 8 up, 7 in 2024-09-21T10:49:34.626 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:49:34.626 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T10:49:34.626 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T10:49:34.626 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T10:49:34.626 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.626 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77348 53.10 53:0a4ac980:::benchmark_data_smithi070_228967_object77347:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:49:34.626 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.666247+0000", 2024-09-21T10:49:34.626 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 508.95627678099999, 2024-09-21T10:49:34.627 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1333923189999999, 2024-09-21T10:49:34.627 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:49:34.627 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:49:34.627 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:49:34.627 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:49:34.627 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:49:34.627 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:49:34.627 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77348 2024-09-21T10:49:34.627 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.627 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:49:34.627 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.627 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:49:34.627 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.666247+0000", 2024-09-21T10:49:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:49:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.666247+0000", 2024-09-21T10:49:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:49:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.666249+0000", 2024-09-21T10:49:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.432e-06 2024-09-21T10:49:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.628 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:49:34.629 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.666267+0000", 2024-09-21T10:49:34.629 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7736000000000001e-05 2024-09-21T10:49:34.629 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.629 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.629 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:49:34.629 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.666278+0000", 2024-09-21T10:49:34.629 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1112e-05 2024-09-21T10:49:34.629 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.629 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.629 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:49:34.629 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.666286+0000", 2024-09-21T10:49:34.629 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9820000000000008e-06 2024-09-21T10:49:34.629 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.629 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.630 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.630 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.666343+0000", 2024-09-21T10:49:34.630 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7099000000000003e-05 2024-09-21T10:49:34.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.630 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.630 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.630 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.666353+0000", 2024-09-21T10:49:34.630 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.6590000000000002e-06 2024-09-21T10:49:34.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.630 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.630 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.630 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659584+0000", 2024-09-21T10:49:34.630 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99323149799999999 2024-09-21T10:49:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659596+0000", 2024-09-21T10:49:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2294e-05 2024-09-21T10:49:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.795107+0000", 2024-09-21T10:49:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1355107019999999 2024-09-21T10:49:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:49:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.795298+0000", 2024-09-21T10:49:34.631 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000190893 2024-09-21T10:49:34.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.632 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:49:34.632 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.795500+0000", 2024-09-21T10:49:34.632 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000202084 2024-09-21T10:49:34.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.632 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.632 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.798730+0000", 2024-09-21T10:49:34.632 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00323025 2024-09-21T10:49:34.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.632 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:49:34.633 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.798743+0000", 2024-09-21T10:49:34.633 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3009e-05 2024-09-21T10:49:34.633 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.633 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.633 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.633 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.798773+0000", 2024-09-21T10:49:34.633 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9974000000000001e-05 2024-09-21T10:49:34.633 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.633 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.633 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:49:34.633 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.798843+0000", 2024-09-21T10:49:34.633 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9523999999999994e-05 2024-09-21T10:49:34.633 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.633 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.633 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:49:34.634 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.799640+0000", 2024-09-21T10:49:34.634 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000797071 2024-09-21T10:49:34.634 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.634 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:49:34.634 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.634 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.634 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.634 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77354 53.10 53:09eb2e30:::benchmark_data_smithi070_228967_object77353:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:49:34.634 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.669089+0000", 2024-09-21T10:49:34.634 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 508.95343524899999, 2024-09-21T10:49:34.634 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1315911619999999, 2024-09-21T10:49:34.634 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:49:34.634 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:49:34.634 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:49:34.635 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:49:34.635 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:49:34.635 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:49:34.635 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77354 2024-09-21T10:49:34.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.635 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:49:34.635 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.635 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:49:34.635 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.669089+0000", 2024-09-21T10:49:34.635 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.635 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.635 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:49:34.635 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.669089+0000", 2024-09-21T10:49:34.636 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.636 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.636 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:49:34.636 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.669091+0000", 2024-09-21T10:49:34.636 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7099999999999999e-06 2024-09-21T10:49:34.636 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.636 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:49:34.636 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.669118+0000", 2024-09-21T10:49:34.636 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7500000000000001e-05 2024-09-21T10:49:34.636 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.636 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:49:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.669137+0000", 2024-09-21T10:49:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8529000000000002e-05 2024-09-21T10:49:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:49:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.669145+0000", 2024-09-21T10:49:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6580000000000008e-06 2024-09-21T10:49:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.669213+0000", 2024-09-21T10:49:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7706000000000005e-05 2024-09-21T10:49:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.637 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.638 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.669223+0000", 2024-09-21T10:49:34.638 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0098999999999999e-05 2024-09-21T10:49:34.638 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.638 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.638 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.638 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659628+0000", 2024-09-21T10:49:34.638 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99040486100000003 2024-09-21T10:49:34.638 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.638 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.638 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.638 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659636+0000", 2024-09-21T10:49:34.638 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4449999999999999e-06 2024-09-21T10:49:34.638 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.638 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.639 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.639 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.797849+0000", 2024-09-21T10:49:34.639 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.138213956 2024-09-21T10:49:34.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.639 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.639 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:49:34.639 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.797958+0000", 2024-09-21T10:49:34.639 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010826400000000001 2024-09-21T10:49:34.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.639 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.639 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:49:34.639 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.798068+0000", 2024-09-21T10:49:34.640 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011027399999999999 2024-09-21T10:49:34.640 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.640 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.640 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.640 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.799784+0000", 2024-09-21T10:49:34.640 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001716239 2024-09-21T10:49:34.641 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.641 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.641 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.641 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.799842+0000", 2024-09-21T10:49:34.641 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7738999999999997e-05 2024-09-21T10:49:34.641 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.641 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.641 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:49:34.641 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.799858+0000", 2024-09-21T10:49:34.641 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5605000000000001e-05 2024-09-21T10:49:34.641 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:49:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.799915+0000", 2024-09-21T10:49:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7553999999999998e-05 2024-09-21T10:49:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:49:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.800680+0000", 2024-09-21T10:49:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00076502299999999998 2024-09-21T10:49:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:49:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.642 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.643 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77415 53.10 53:0eba5a74:::benchmark_data_smithi070_228967_object77414:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:49:34.643 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.700483+0000", 2024-09-21T10:49:34.643 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 508.92204134899998, 2024-09-21T10:49:34.643 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1042608860000001, 2024-09-21T10:49:34.643 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:49:34.643 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:49:34.643 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:49:34.643 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:49:34.643 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:49:34.643 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:49:34.643 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77415 2024-09-21T10:49:34.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.644 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:49:34.644 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.644 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:49:34.644 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700483+0000", 2024-09-21T10:49:34.644 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.644 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.644 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.644 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:49:34.644 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700483+0000", 2024-09-21T10:49:34.644 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.644 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.644 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.644 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:49:34.644 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700485+0000", 2024-09-21T10:49:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5880000000000001e-06 2024-09-21T10:49:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:49:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700506+0000", 2024-09-21T10:49:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1049000000000001e-05 2024-09-21T10:49:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:49:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700517+0000", 2024-09-21T10:49:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1603000000000001e-05 2024-09-21T10:49:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.645 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:49:34.646 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700524+0000", 2024-09-21T10:49:34.646 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1670000000000002e-06 2024-09-21T10:49:34.646 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.646 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.646 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.646 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700751+0000", 2024-09-21T10:49:34.646 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00022716300000000001 2024-09-21T10:49:34.646 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.646 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.646 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.646 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700762+0000", 2024-09-21T10:49:34.646 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0550999999999999e-05 2024-09-21T10:49:34.646 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.647 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.647 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659665+0000", 2024-09-21T10:49:34.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.95890305200000003 2024-09-21T10:49:34.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.647 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.647 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659673+0000", 2024-09-21T10:49:34.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4950000000000002e-06 2024-09-21T10:49:34.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.647 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.647 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.800708+0000", 2024-09-21T10:49:34.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.141035383 2024-09-21T10:49:34.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.648 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:49:34.648 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.800824+0000", 2024-09-21T10:49:34.648 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000116431 2024-09-21T10:49:34.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.648 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:49:34.648 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.800934+0000", 2024-09-21T10:49:34.648 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010966799999999999 2024-09-21T10:49:34.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.648 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.648 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.803154+0000", 2024-09-21T10:49:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0022197940000000002 2024-09-21T10:49:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.804093+0000", 2024-09-21T10:49:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00093908499999999998 2024-09-21T10:49:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:49:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.804102+0000", 2024-09-21T10:49:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5979999999999997e-06 2024-09-21T10:49:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:49:34.650 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.804136+0000", 2024-09-21T10:49:34.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4536999999999998e-05 2024-09-21T10:49:34.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.650 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:49:34.650 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.804744+0000", 2024-09-21T10:49:34.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00060772199999999999 2024-09-21T10:49:34.650 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.650 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:49:34.650 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.650 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77416 53.10 53:0b890937:::benchmark_data_smithi070_228967_object77415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:49:34.650 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.700546+0000", 2024-09-21T10:49:34.651 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 508.92197825, 2024-09-21T10:49:34.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1047850320000001, 2024-09-21T10:49:34.651 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:49:34.651 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:49:34.651 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:49:34.651 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:49:34.651 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:49:34.651 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:49:34.651 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77416 2024-09-21T10:49:34.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.651 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:49:34.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.651 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:49:34.652 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700546+0000", 2024-09-21T10:49:34.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.652 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.652 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:49:34.652 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700546+0000", 2024-09-21T10:49:34.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.652 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.652 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:49:34.652 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700547+0000", 2024-09-21T10:49:34.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0999999999999999e-07 2024-09-21T10:49:34.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.652 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:49:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700564+0000", 2024-09-21T10:49:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7252e-05 2024-09-21T10:49:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:49:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700572+0000", 2024-09-21T10:49:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9039999999999992e-06 2024-09-21T10:49:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:49:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700577+0000", 2024-09-21T10:49:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.699e-06 2024-09-21T10:49:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.653 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.654 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.654 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700788+0000", 2024-09-21T10:49:34.654 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00021084000000000001 2024-09-21T10:49:34.654 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.654 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.654 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.654 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700796+0000", 2024-09-21T10:49:34.654 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6090000000000003e-06 2024-09-21T10:49:34.654 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.654 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.654 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.654 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659707+0000", 2024-09-21T10:49:34.654 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.95891092 2024-09-21T10:49:34.654 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.655 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.655 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659711+0000", 2024-09-21T10:49:34.655 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3789999999999999e-06 2024-09-21T10:49:34.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.655 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.655 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.802199+0000", 2024-09-21T10:49:34.655 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.142488341 2024-09-21T10:49:34.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.655 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:49:34.655 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.802320+0000", 2024-09-21T10:49:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000120209 2024-09-21T10:49:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:49:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.802494+0000", 2024-09-21T10:49:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000174771 2024-09-21T10:49:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.804796+0000", 2024-09-21T10:49:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002301726 2024-09-21T10:49:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.656 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.804844+0000", 2024-09-21T10:49:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8266000000000002e-05 2024-09-21T10:49:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:49:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.804854+0000", 2024-09-21T10:49:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0018e-05 2024-09-21T10:49:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:49:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.804893+0000", 2024-09-21T10:49:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8430999999999998e-05 2024-09-21T10:49:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.657 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:49:34.658 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.805331+0000", 2024-09-21T10:49:34.658 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000438157 2024-09-21T10:49:34.658 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.658 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:49:34.658 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.658 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.658 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.658 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77417 53.10 53:0e921901:::benchmark_data_smithi070_228967_object77416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:49:34.658 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.700585+0000", 2024-09-21T10:49:34.658 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 508.92193908299998, 2024-09-21T10:49:34.658 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.118979521, 2024-09-21T10:49:34.658 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:49:34.658 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:49:34.658 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:49:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:49:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:49:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:49:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77417 2024-09-21T10:49:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:49:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:49:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700585+0000", 2024-09-21T10:49:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.659 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:49:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700585+0000", 2024-09-21T10:49:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:49:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700586+0000", 2024-09-21T10:49:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0400000000000002e-07 2024-09-21T10:49:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:49:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700605+0000", 2024-09-21T10:49:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9463999999999999e-05 2024-09-21T10:49:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:49:34.660 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700612+0000", 2024-09-21T10:49:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5019999999999998e-06 2024-09-21T10:49:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:49:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700617+0000", 2024-09-21T10:49:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9520000000000002e-06 2024-09-21T10:49:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700902+0000", 2024-09-21T10:49:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00028501699999999999 2024-09-21T10:49:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.661 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.662 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.662 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.700910+0000", 2024-09-21T10:49:34.662 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1999999999999994e-06 2024-09-21T10:49:34.662 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.662 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.662 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.662 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659744+0000", 2024-09-21T10:49:34.662 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.95883443300000004 2024-09-21T10:49:34.662 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.662 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.662 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.662 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659753+0000", 2024-09-21T10:49:34.662 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.4379999999999995e-06 2024-09-21T10:49:34.662 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.663 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.663 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.809941+0000", 2024-09-21T10:49:34.663 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1501882539999999 2024-09-21T10:49:34.663 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.663 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:49:34.663 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.810050+0000", 2024-09-21T10:49:34.663 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010925999999999999 2024-09-21T10:49:34.663 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.663 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:49:34.663 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.810156+0000", 2024-09-21T10:49:34.663 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010574 2024-09-21T10:49:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:49:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.813660+0000", 2024-09-21T10:49:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035046370000000001 2024-09-21T10:49:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.813692+0000", 2024-09-21T10:49:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1769e-05 2024-09-21T10:49:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.664 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.818533+0000", 2024-09-21T10:49:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0048406539999999998 2024-09-21T10:49:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:49:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.818577+0000", 2024-09-21T10:49:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3915000000000002e-05 2024-09-21T10:49:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:49:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.819565+0000", 2024-09-21T10:49:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00098788200000000004 2024-09-21T10:49:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:49:34.665 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.666 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.666 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.666 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77499 53.10 53:0c45bed1:::benchmark_data_smithi070_228967_object77498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:49:34.666 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.763242+0000", 2024-09-21T10:49:34.666 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 508.85928181200001, 2024-09-21T10:49:34.666 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0610106340000001, 2024-09-21T10:49:34.666 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:49:34.666 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:49:34.666 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:49:34.666 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:49:34.666 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:49:34.666 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:49:34.666 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77499 2024-09-21T10:49:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:49:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:49:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.763242+0000", 2024-09-21T10:49:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:49:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.763242+0000", 2024-09-21T10:49:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.667 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:49:34.668 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.763244+0000", 2024-09-21T10:49:34.668 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.646e-06 2024-09-21T10:49:34.668 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.668 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.668 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:49:34.668 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.763268+0000", 2024-09-21T10:49:34.668 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4303000000000001e-05 2024-09-21T10:49:34.668 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.668 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.668 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:49:34.668 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.763282+0000", 2024-09-21T10:49:34.668 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4018e-05 2024-09-21T10:49:34.668 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.669 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.669 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:49:34.669 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.763290+0000", 2024-09-21T10:49:34.669 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1570000000000003e-06 2024-09-21T10:49:34.669 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.669 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.669 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.669 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.763365+0000", 2024-09-21T10:49:34.669 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5511000000000003e-05 2024-09-21T10:49:34.669 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.669 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.669 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.669 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.763387+0000", 2024-09-21T10:49:34.669 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1639000000000001e-05 2024-09-21T10:49:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659776+0000", 2024-09-21T10:49:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.89638960999999995 2024-09-21T10:49:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659780+0000", 2024-09-21T10:49:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8890000000000004e-06 2024-09-21T10:49:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.670 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.810900+0000", 2024-09-21T10:49:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1511196960000001 2024-09-21T10:49:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:49:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.810983+0000", 2024-09-21T10:49:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2664999999999995e-05 2024-09-21T10:49:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:49:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.811076+0000", 2024-09-21T10:49:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3386000000000003e-05 2024-09-21T10:49:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.813728+0000", 2024-09-21T10:49:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0026517609999999999 2024-09-21T10:49:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:49:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.813804+0000", 2024-09-21T10:49:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6404999999999998e-05 2024-09-21T10:49:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.823202+0000", 2024-09-21T10:49:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0093975829999999993 2024-09-21T10:49:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:49:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.823265+0000", 2024-09-21T10:49:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3585999999999997e-05 2024-09-21T10:49:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:49:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.824253+0000", 2024-09-21T10:49:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00098777900000000004 2024-09-21T10:49:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:49:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.673 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77507 53.10 53:0c3b2271:::benchmark_data_smithi070_228967_object77506:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:49:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.767593+0000", 2024-09-21T10:49:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 508.85493129499997, 2024-09-21T10:49:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0575902039999998, 2024-09-21T10:49:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:49:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:49:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:49:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:49:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:49:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:49:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77507 2024-09-21T10:49:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:49:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.674 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:49:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.767593+0000", 2024-09-21T10:49:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:49:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.767593+0000", 2024-09-21T10:49:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:49:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.767595+0000", 2024-09-21T10:49:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5880000000000001e-06 2024-09-21T10:49:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:49:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.767643+0000", 2024-09-21T10:49:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8563000000000002e-05 2024-09-21T10:49:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:49:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.767659+0000", 2024-09-21T10:49:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6186000000000001e-05 2024-09-21T10:49:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:49:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.767667+0000", 2024-09-21T10:49:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2200000000000003e-06 2024-09-21T10:49:34.676 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.767727+0000", 2024-09-21T10:49:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0702999999999998e-05 2024-09-21T10:49:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.767737+0000", 2024-09-21T10:49:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.6050000000000004e-06 2024-09-21T10:49:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659809+0000", 2024-09-21T10:49:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.89207242600000003 2024-09-21T10:49:34.677 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659817+0000", 2024-09-21T10:49:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5440000000000001e-06 2024-09-21T10:49:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.811691+0000", 2024-09-21T10:49:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1518743419999999 2024-09-21T10:49:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:49:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.811779+0000", 2024-09-21T10:49:34.678 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.8119000000000001e-05 2024-09-21T10:49:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:49:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.811886+0000", 2024-09-21T10:49:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000106805 2024-09-21T10:49:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.813784+0000", 2024-09-21T10:49:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00189806 2024-09-21T10:49:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:49:34.679 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.813814+0000", 2024-09-21T10:49:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0090000000000002e-05 2024-09-21T10:49:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.824353+0000", 2024-09-21T10:49:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.010538403 2024-09-21T10:49:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:49:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.824440+0000", 2024-09-21T10:49:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7063000000000003e-05 2024-09-21T10:49:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:49:34.680 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.825183+0000", 2024-09-21T10:49:34.681 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00074348700000000003 2024-09-21T10:49:34.681 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.681 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:49:34.681 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.681 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.681 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.682 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77522 53.10 53:0d13612e:::benchmark_data_smithi070_228967_object77521:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:49:34.682 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.778011+0000", 2024-09-21T10:49:34.682 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 508.84451330399997, 2024-09-21T10:49:34.682 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0483339169999999, 2024-09-21T10:49:34.682 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:49:34.682 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:49:34.682 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:49:34.682 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:49:34.682 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:49:34.682 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:49:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77522 2024-09-21T10:49:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:49:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:49:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.778011+0000", 2024-09-21T10:49:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:49:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.778011+0000", 2024-09-21T10:49:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.683 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:49:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.778013+0000", 2024-09-21T10:49:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9970000000000001e-06 2024-09-21T10:49:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:49:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.778037+0000", 2024-09-21T10:49:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4521000000000001e-05 2024-09-21T10:49:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:49:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.778050+0000", 2024-09-21T10:49:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2578999999999999e-05 2024-09-21T10:49:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:49:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.778058+0000", 2024-09-21T10:49:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2320000000000001e-06 2024-09-21T10:49:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.778124+0000", 2024-09-21T10:49:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5396999999999997e-05 2024-09-21T10:49:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.778133+0000", 2024-09-21T10:49:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2730000000000008e-06 2024-09-21T10:49:34.685 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659837+0000", 2024-09-21T10:49:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.88170432399999998 2024-09-21T10:49:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659841+0000", 2024-09-21T10:49:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7290000000000002e-06 2024-09-21T10:49:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.812569+0000", 2024-09-21T10:49:34.686 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.152727949 2024-09-21T10:49:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:49:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.812704+0000", 2024-09-21T10:49:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000135019 2024-09-21T10:49:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:49:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.812863+0000", 2024-09-21T10:49:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015892200000000001 2024-09-21T10:49:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.687 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.815571+0000", 2024-09-21T10:49:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0027076460000000002 2024-09-21T10:49:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:49:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.815618+0000", 2024-09-21T10:49:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7732000000000003e-05 2024-09-21T10:49:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.825288+0000", 2024-09-21T10:49:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0096699300000000002 2024-09-21T10:49:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.688 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:49:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.825349+0000", 2024-09-21T10:49:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0720999999999997e-05 2024-09-21T10:49:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:49:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.826345+0000", 2024-09-21T10:49:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00099594599999999994 2024-09-21T10:49:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:49:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77550 53.10 53:0c033939:::benchmark_data_smithi070_228967_object77549:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:49:34.689 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.799776+0000", 2024-09-21T10:49:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 508.82274874900003, 2024-09-21T10:49:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0288999130000001, 2024-09-21T10:49:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:49:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:49:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:49:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:49:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:49:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:49:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77550 2024-09-21T10:49:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:49:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:49:34.690 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.799776+0000", 2024-09-21T10:49:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:49:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.799776+0000", 2024-09-21T10:49:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:49:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.799776+0000", 2024-09-21T10:49:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7599999999999997e-07 2024-09-21T10:49:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:49:34.691 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.799824+0000", 2024-09-21T10:49:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7649999999999999e-05 2024-09-21T10:49:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:49:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.799831+0000", 2024-09-21T10:49:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1330000000000001e-06 2024-09-21T10:49:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:49:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.799838+0000", 2024-09-21T10:49:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6270000000000002e-06 2024-09-21T10:49:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.692 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.799898+0000", 2024-09-21T10:49:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0849e-05 2024-09-21T10:49:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.799907+0000", 2024-09-21T10:49:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1500000000000005e-06 2024-09-21T10:49:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659869+0000", 2024-09-21T10:49:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.85996196300000005 2024-09-21T10:49:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.693 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659877+0000", 2024-09-21T10:49:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.255e-06 2024-09-21T10:49:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.813839+0000", 2024-09-21T10:49:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.153962089 2024-09-21T10:49:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:49:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.813940+0000", 2024-09-21T10:49:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000100719 2024-09-21T10:49:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.694 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:49:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.814072+0000", 2024-09-21T10:49:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000132065 2024-09-21T10:49:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.815610+0000", 2024-09-21T10:49:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0015380389999999999 2024-09-21T10:49:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:49:34.695 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.815622+0000", 2024-09-21T10:49:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2714000000000001e-05 2024-09-21T10:49:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.828115+0000", 2024-09-21T10:49:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012492207999999999 2024-09-21T10:49:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:49:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.828184+0000", 2024-09-21T10:49:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9218e-05 2024-09-21T10:49:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.696 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:49:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.828675+0000", 2024-09-21T10:49:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00049165799999999996 2024-09-21T10:49:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:49:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77552 53.10 53:092395bc:::benchmark_data_smithi070_228967_object77551:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:49:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.801039+0000", 2024-09-21T10:49:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 508.821485263, 2024-09-21T10:49:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0284018509999999, 2024-09-21T10:49:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:49:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:49:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:49:34.697 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:49:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:49:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:49:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77552 2024-09-21T10:49:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:49:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:49:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.801039+0000", 2024-09-21T10:49:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:49:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.801039+0000", 2024-09-21T10:49:34.698 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.699 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.699 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.699 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:49:34.699 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.801040+0000", 2024-09-21T10:49:34.699 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.184e-06 2024-09-21T10:49:34.699 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.699 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.699 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:49:34.699 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.801089+0000", 2024-09-21T10:49:34.699 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9313e-05 2024-09-21T10:49:34.699 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.699 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.699 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:49:34.699 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.801103+0000", 2024-09-21T10:49:34.700 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3816e-05 2024-09-21T10:49:34.700 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.700 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.700 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:49:34.700 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.801111+0000", 2024-09-21T10:49:34.700 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5820000000000004e-06 2024-09-21T10:49:34.700 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.700 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.700 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.700 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.801164+0000", 2024-09-21T10:49:34.700 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3362000000000002e-05 2024-09-21T10:49:34.700 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.700 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.700 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.700 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.801173+0000", 2024-09-21T10:49:34.701 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6649999999999992e-06 2024-09-21T10:49:34.701 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.701 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.701 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.701 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659897+0000", 2024-09-21T10:49:34.701 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.85872434499999994 2024-09-21T10:49:34.701 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.701 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.701 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.701 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659901+0000", 2024-09-21T10:49:34.701 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4479999999999999e-06 2024-09-21T10:49:34.701 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.701 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.701 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.702 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.814871+0000", 2024-09-21T10:49:34.702 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.154970026 2024-09-21T10:49:34.702 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.702 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.702 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:49:34.702 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.814966+0000", 2024-09-21T10:49:34.702 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5260999999999995e-05 2024-09-21T10:49:34.702 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.702 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.702 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:49:34.702 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.815058+0000", 2024-09-21T10:49:34.702 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1976000000000002e-05 2024-09-21T10:49:34.702 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.702 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.818386+0000", 2024-09-21T10:49:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0033278079999999998 2024-09-21T10:49:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:49:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.818396+0000", 2024-09-21T10:49:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.046e-05 2024-09-21T10:49:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.828708+0000", 2024-09-21T10:49:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.010312089999999999 2024-09-21T10:49:34.703 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:49:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.828766+0000", 2024-09-21T10:49:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7964000000000003e-05 2024-09-21T10:49:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:49:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.829441+0000", 2024-09-21T10:49:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00067455100000000001 2024-09-21T10:49:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:49:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.704 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77583 53.10 53:0c625067:::benchmark_data_smithi070_228967_object77582:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:49:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.823905+0000", 2024-09-21T10:49:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 508.79861941299998, 2024-09-21T10:49:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0063239049999999, 2024-09-21T10:49:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:49:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:49:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:49:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:49:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:49:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:49:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77583 2024-09-21T10:49:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:49:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.705 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:49:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.823905+0000", 2024-09-21T10:49:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:49:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.823905+0000", 2024-09-21T10:49:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:49:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.823906+0000", 2024-09-21T10:49:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6190000000000001e-06 2024-09-21T10:49:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.706 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:49:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.823936+0000", 2024-09-21T10:49:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9220000000000001e-05 2024-09-21T10:49:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:49:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.823947+0000", 2024-09-21T10:49:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1337e-05 2024-09-21T10:49:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:49:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.823954+0000", 2024-09-21T10:49:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8719999999999996e-06 2024-09-21T10:49:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.707 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.824052+0000", 2024-09-21T10:49:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.8405000000000004e-05 2024-09-21T10:49:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.824059+0000", 2024-09-21T10:49:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4110000000000001e-06 2024-09-21T10:49:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659929+0000", 2024-09-21T10:49:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.83586998099999998 2024-09-21T10:49:34.708 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.709 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.709 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.709 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659936+0000", 2024-09-21T10:49:34.709 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1180000000000003e-06 2024-09-21T10:49:34.709 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.709 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.709 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.709 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.815647+0000", 2024-09-21T10:49:34.709 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.155710826 2024-09-21T10:49:34.709 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.709 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.709 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:49:34.709 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.815726+0000", 2024-09-21T10:49:34.709 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9102000000000003e-05 2024-09-21T10:49:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:49:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.815820+0000", 2024-09-21T10:49:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4326000000000005e-05 2024-09-21T10:49:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:49:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.818401+0000", 2024-09-21T10:49:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025805509999999999 2024-09-21T10:49:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.818441+0000", 2024-09-21T10:49:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0170999999999999e-05 2024-09-21T10:49:34.710 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.711 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.711 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.711 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.829499+0000", 2024-09-21T10:49:34.711 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.011058182 2024-09-21T10:49:34.711 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.711 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.711 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:49:34.711 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.829557+0000", 2024-09-21T10:49:34.711 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8409999999999998e-05 2024-09-21T10:49:34.711 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.711 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.711 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:49:34.711 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.830229+0000", 2024-09-21T10:49:34.712 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00067137399999999997 2024-09-21T10:49:34.712 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.712 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:49:34.712 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.712 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.712 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.712 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77587 53.10 53:098fcb2a:::benchmark_data_smithi070_228967_object77586:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:49:34.712 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.825548+0000", 2024-09-21T10:49:34.712 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 508.79697580700002, 2024-09-21T10:49:34.712 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0105805079999999, 2024-09-21T10:49:34.712 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:49:34.712 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:49:34.712 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:49:34.712 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:49:34.712 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:49:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:49:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77587 2024-09-21T10:49:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:49:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:49:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.825548+0000", 2024-09-21T10:49:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:49:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.825548+0000", 2024-09-21T10:49:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.713 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.714 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.714 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:49:34.714 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.825550+0000", 2024-09-21T10:49:34.714 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2759999999999999e-06 2024-09-21T10:49:34.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.714 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.714 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:49:34.714 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.825602+0000", 2024-09-21T10:49:34.714 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2707000000000002e-05 2024-09-21T10:49:34.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.714 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.714 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:49:34.714 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.825617+0000", 2024-09-21T10:49:34.714 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4871e-05 2024-09-21T10:49:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:49:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.825625+0000", 2024-09-21T10:49:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0809999999999994e-06 2024-09-21T10:49:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.825677+0000", 2024-09-21T10:49:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1218000000000003e-05 2024-09-21T10:49:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.715 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.825685+0000", 2024-09-21T10:49:34.716 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3280000000000006e-06 2024-09-21T10:49:34.716 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.716 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.716 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.716 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659956+0000", 2024-09-21T10:49:34.716 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.83427153799999998 2024-09-21T10:49:34.716 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.716 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.716 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.716 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659961+0000", 2024-09-21T10:49:34.716 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9540000000000003e-06 2024-09-21T10:49:34.716 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.716 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.716 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.716 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.816365+0000", 2024-09-21T10:49:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1564039589999999 2024-09-21T10:49:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:49:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.816477+0000", 2024-09-21T10:49:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000111575 2024-09-21T10:49:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:49:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.816577+0000", 2024-09-21T10:49:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000100078 2024-09-21T10:49:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.717 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.818499+0000", 2024-09-21T10:49:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0019215129999999999 2024-09-21T10:49:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:49:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.818509+0000", 2024-09-21T10:49:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0200999999999999e-05 2024-09-21T10:49:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.834949+0000", 2024-09-21T10:49:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.016440735000000001 2024-09-21T10:49:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.718 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:49:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.835043+0000", 2024-09-21T10:49:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3177999999999995e-05 2024-09-21T10:49:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:49:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.836129+0000", 2024-09-21T10:49:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001086296 2024-09-21T10:49:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:49:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77621 53.10 53:08d8a996:::benchmark_data_smithi070_228967_object77620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:49:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.858882+0000", 2024-09-21T10:49:34.719 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 508.76364200299997, 2024-09-21T10:49:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9785834369999999, 2024-09-21T10:49:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:49:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:49:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:49:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:49:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:49:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:49:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77621 2024-09-21T10:49:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:49:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:49:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.858882+0000", 2024-09-21T10:49:34.720 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:49:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.858882+0000", 2024-09-21T10:49:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:49:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.858883+0000", 2024-09-21T10:49:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5099999999999998e-07 2024-09-21T10:49:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:49:34.721 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.858933+0000", 2024-09-21T10:49:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9700000000000002e-05 2024-09-21T10:49:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:49:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.858943+0000", 2024-09-21T10:49:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7939999999999997e-06 2024-09-21T10:49:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:49:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.858949+0000", 2024-09-21T10:49:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7769999999999997e-06 2024-09-21T10:49:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.722 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.859007+0000", 2024-09-21T10:49:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7456e-05 2024-09-21T10:49:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.859017+0000", 2024-09-21T10:49:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0071999999999999e-05 2024-09-21T10:49:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659993+0000", 2024-09-21T10:49:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.80097583500000002 2024-09-21T10:49:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.723 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.659998+0000", 2024-09-21T10:49:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4929999999999998e-06 2024-09-21T10:49:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.817285+0000", 2024-09-21T10:49:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1572862640000001 2024-09-21T10:49:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:49:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.817418+0000", 2024-09-21T10:49:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013367300000000001 2024-09-21T10:49:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.724 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:49:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.817576+0000", 2024-09-21T10:49:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015830299999999999 2024-09-21T10:49:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.819630+0000", 2024-09-21T10:49:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0020534120000000001 2024-09-21T10:49:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:49:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.820470+0000", 2024-09-21T10:49:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00083965699999999995 2024-09-21T10:49:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.725 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.726 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.726 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.836186+0000", 2024-09-21T10:49:34.726 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015716218000000001 2024-09-21T10:49:34.726 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.726 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.726 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:49:34.726 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.836282+0000", 2024-09-21T10:49:34.726 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5839000000000002e-05 2024-09-21T10:49:34.726 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.726 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.726 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:49:34.726 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.837466+0000", 2024-09-21T10:49:34.726 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001184093 2024-09-21T10:49:34.726 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:49:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77656 53.10 53:08530a06:::benchmark_data_smithi070_228967_object77655:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:49:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.901366+0000", 2024-09-21T10:49:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 508.72115785800003, 2024-09-21T10:49:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9373663639999998, 2024-09-21T10:49:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:49:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:49:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:49:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:49:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:49:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:49:34.727 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77656 2024-09-21T10:49:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:49:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:49:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.901366+0000", 2024-09-21T10:49:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:49:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.901366+0000", 2024-09-21T10:49:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:49:34.728 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.901377+0000", 2024-09-21T10:49:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0122999999999999e-05 2024-09-21T10:49:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:49:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.901395+0000", 2024-09-21T10:49:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8896e-05 2024-09-21T10:49:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:49:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.901407+0000", 2024-09-21T10:49:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1409999999999999e-05 2024-09-21T10:49:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:49:34.729 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.901413+0000", 2024-09-21T10:49:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9399999999999999e-06 2024-09-21T10:49:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.901471+0000", 2024-09-21T10:49:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7853000000000003e-05 2024-09-21T10:49:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.901480+0000", 2024-09-21T10:49:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1959999999999997e-06 2024-09-21T10:49:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.730 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.660022+0000", 2024-09-21T10:49:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.75854222100000002 2024-09-21T10:49:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.660026+0000", 2024-09-21T10:49:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7840000000000001e-06 2024-09-21T10:49:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.819670+0000", 2024-09-21T10:49:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1596445310000001 2024-09-21T10:49:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.731 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:49:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.819755+0000", 2024-09-21T10:49:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5062000000000007e-05 2024-09-21T10:49:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:49:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.819853+0000", 2024-09-21T10:49:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7116999999999999e-05 2024-09-21T10:49:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.823074+0000", 2024-09-21T10:49:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.003221776 2024-09-21T10:49:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.732 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.733 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:49:34.733 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.823085+0000", 2024-09-21T10:49:34.733 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0924000000000001e-05 2024-09-21T10:49:34.733 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.733 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.733 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.733 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.837697+0000", 2024-09-21T10:49:34.733 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.014611455000000001 2024-09-21T10:49:34.733 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.733 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.733 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:49:34.733 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.837931+0000", 2024-09-21T10:49:34.733 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00023461000000000001 2024-09-21T10:49:34.733 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.734 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.734 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:49:34.734 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.838733+0000", 2024-09-21T10:49:34.734 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00080146600000000003 2024-09-21T10:49:34.734 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.734 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:49:34.734 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.734 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.734 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.734 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:34.735 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77672 53.10 53:0f020cb9:::benchmark_data_smithi070_228967_object77671:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:49:34.736 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.923481+0000", 2024-09-21T10:49:34.736 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 508.69904283800003, 2024-09-21T10:49:34.736 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9159655249999998, 2024-09-21T10:49:34.736 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:49:34.736 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:49:34.736 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:49:34.736 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:49:34.736 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:49:34.736 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:49:34.736 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77672 2024-09-21T10:49:34.736 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.736 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:49:34.737 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.737 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:49:34.737 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.923481+0000", 2024-09-21T10:49:34.737 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.737 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.737 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.737 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:49:34.737 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.923481+0000", 2024-09-21T10:49:34.737 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.737 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.737 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.737 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:49:34.737 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.923483+0000", 2024-09-21T10:49:34.737 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.19e-06 2024-09-21T10:49:34.738 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.738 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.738 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:49:34.738 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.923502+0000", 2024-09-21T10:49:34.738 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9454e-05 2024-09-21T10:49:34.738 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.738 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.738 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:49:34.738 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.923512+0000", 2024-09-21T10:49:34.738 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0390999999999999e-05 2024-09-21T10:49:34.738 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.738 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.738 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:49:34.738 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.923518+0000", 2024-09-21T10:49:34.739 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2599999999999996e-06 2024-09-21T10:49:34.739 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.739 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.739 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.739 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.923561+0000", 2024-09-21T10:49:34.739 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3621999999999997e-05 2024-09-21T10:49:34.739 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.739 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.739 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.739 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.923568+0000", 2024-09-21T10:49:34.739 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1689999999999997e-06 2024-09-21T10:49:34.739 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.739 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.739 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.740 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.660046+0000", 2024-09-21T10:49:34.740 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.73647844299999998 2024-09-21T10:49:34.740 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.740 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.740 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.740 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.660050+0000", 2024-09-21T10:49:34.740 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8070000000000001e-06 2024-09-21T10:49:34.740 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.740 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.740 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.740 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.820493+0000", 2024-09-21T10:49:34.740 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.160443109 2024-09-21T10:49:34.740 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.740 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.741 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:49:34.741 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.820584+0000", 2024-09-21T10:49:34.741 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0705999999999995e-05 2024-09-21T10:49:34.741 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.741 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.741 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:49:34.741 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.820680+0000", 2024-09-21T10:49:34.741 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.6836999999999995e-05 2024-09-21T10:49:34.741 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.741 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.741 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:49:34.741 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.823091+0000", 2024-09-21T10:49:34.741 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0024103309999999999 2024-09-21T10:49:34.741 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.742 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.742 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.742 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.823139+0000", 2024-09-21T10:49:34.742 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7985999999999998e-05 2024-09-21T10:49:34.742 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.742 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.742 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.742 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.838802+0000", 2024-09-21T10:49:34.742 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015663567999999999 2024-09-21T10:49:34.742 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.742 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.742 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:49:34.742 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.838880+0000", 2024-09-21T10:49:34.742 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7302e-05 2024-09-21T10:49:34.743 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.743 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.743 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:49:34.743 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.839447+0000", 2024-09-21T10:49:34.743 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00056734999999999999 2024-09-21T10:49:34.743 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.743 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:49:34.743 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.743 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.743 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.743 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:77683 53.10 53:0c041dae:::benchmark_data_smithi070_228967_object77682:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:49:34.743 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:05.950914+0000", 2024-09-21T10:49:34.743 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 508.67161046899997, 2024-09-21T10:49:34.743 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.889211167, 2024-09-21T10:49:34.744 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:49:34.744 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:49:34.744 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:49:34.744 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:49:34.744 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:49:34.744 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:49:34.744 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 77683 2024-09-21T10:49:34.744 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.744 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:49:34.744 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.744 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:49:34.744 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.950914+0000", 2024-09-21T10:49:34.744 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.744 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.745 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.745 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:49:34.745 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.950914+0000", 2024-09-21T10:49:34.745 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.745 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.745 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.745 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:49:34.745 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.950915+0000", 2024-09-21T10:49:34.745 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.508e-06 2024-09-21T10:49:34.745 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.745 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.745 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:49:34.745 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.950958+0000", 2024-09-21T10:49:34.745 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3133999999999997e-05 2024-09-21T10:49:34.745 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.746 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.746 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:49:34.746 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.950974+0000", 2024-09-21T10:49:34.746 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5828000000000002e-05 2024-09-21T10:49:34.746 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.746 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.746 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:49:34.746 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.950981+0000", 2024-09-21T10:49:34.746 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1670000000000002e-06 2024-09-21T10:49:34.746 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.746 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.746 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.746 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.951050+0000", 2024-09-21T10:49:34.746 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8666999999999998e-05 2024-09-21T10:49:34.747 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.747 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.747 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.747 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:05.951062+0000", 2024-09-21T10:49:34.747 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2064e-05 2024-09-21T10:49:34.747 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.747 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.747 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.747 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.660067+0000", 2024-09-21T10:49:34.747 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.70900450999999998 2024-09-21T10:49:34.747 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.747 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.747 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.747 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:06.660071+0000", 2024-09-21T10:49:34.748 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8580000000000002e-06 2024-09-21T10:49:34.748 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.748 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.748 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.748 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.821230+0000", 2024-09-21T10:49:34.748 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1611593650000001 2024-09-21T10:49:34.748 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.748 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.748 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:49:34.748 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.821312+0000", 2024-09-21T10:49:34.748 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2452000000000003e-05 2024-09-21T10:49:34.748 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.748 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.748 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:49:34.748 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.821416+0000", 2024-09-21T10:49:34.749 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010381400000000001 2024-09-21T10:49:34.749 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.749 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.749 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:49:34.749 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.824269+0000", 2024-09-21T10:49:34.749 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0028524179999999998 2024-09-21T10:49:34.749 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.749 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.749 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.749 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.824301+0000", 2024-09-21T10:49:34.749 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2481999999999998e-05 2024-09-21T10:49:34.749 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.749 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.749 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.750 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.839511+0000", 2024-09-21T10:49:34.750 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015209926 2024-09-21T10:49:34.750 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.750 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.750 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:49:34.750 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.839581+0000", 2024-09-21T10:49:34.750 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9913000000000006e-05 2024-09-21T10:49:34.750 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.750 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.750 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:49:34.750 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:07.840125+0000", 2024-09-21T10:49:34.750 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00054406099999999998 2024-09-21T10:49:34.750 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.750 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:49:34.751 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.751 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.751 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.751 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:80155 53.12 53:4c70868e:::benchmark_data_smithi070_228967_object80154:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1330)", 2024-09-21T10:49:34.751 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:10.695776+0000", 2024-09-21T10:49:34.751 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 503.92674812199999, 2024-09-21T10:49:34.751 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3340382100000001, 2024-09-21T10:49:34.751 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:49:34.751 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:49:34.751 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:49:34.751 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:49:34.751 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:49:34.751 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:49:34.751 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 80155 2024-09-21T10:49:34.752 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.752 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:49:34.752 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.752 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:49:34.752 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695776+0000", 2024-09-21T10:49:34.752 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.752 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.752 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.752 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:49:34.752 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695776+0000", 2024-09-21T10:49:34.752 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.752 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.752 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.752 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:49:34.753 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695777+0000", 2024-09-21T10:49:34.753 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0900000000000002e-07 2024-09-21T10:49:34.753 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.753 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.753 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:49:34.753 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695795+0000", 2024-09-21T10:49:34.753 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8150000000000001e-05 2024-09-21T10:49:34.753 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.753 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.753 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:49:34.753 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695802+0000", 2024-09-21T10:49:34.753 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9750000000000001e-06 2024-09-21T10:49:34.753 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.753 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.753 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:49:34.754 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695807+0000", 2024-09-21T10:49:34.754 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1869999999999996e-06 2024-09-21T10:49:34.754 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.754 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.754 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.754 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.697268+0000", 2024-09-21T10:49:34.754 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001461237 2024-09-21T10:49:34.754 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.754 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.754 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.754 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.697275+0000", 2024-09-21T10:49:34.754 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0600000000000002e-06 2024-09-21T10:49:34.754 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.754 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.755 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.755 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.014876+0000", 2024-09-21T10:49:34.755 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3176006689999999 2024-09-21T10:49:34.755 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.755 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.755 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:49:34.755 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.015180+0000", 2024-09-21T10:49:34.755 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00030391000000000001 2024-09-21T10:49:34.755 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.755 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.755 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,3,6", 2024-09-21T10:49:34.755 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.015361+0000", 2024-09-21T10:49:34.755 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018097299999999999 2024-09-21T10:49:34.756 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.756 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.756 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.756 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.018976+0000", 2024-09-21T10:49:34.756 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0036151109999999998 2024-09-21T10:49:34.756 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.756 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.756 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:49:34.756 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.020561+0000", 2024-09-21T10:49:34.756 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0015851369999999999 2024-09-21T10:49:34.756 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.756 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.756 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.756 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.027092+0000", 2024-09-21T10:49:34.757 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0065314250000000004 2024-09-21T10:49:34.757 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.757 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.757 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.757 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.028432+0000", 2024-09-21T10:49:34.757 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00134005 2024-09-21T10:49:34.757 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.757 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.757 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:49:34.757 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.028542+0000", 2024-09-21T10:49:34.757 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000109574 2024-09-21T10:49:34.757 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.757 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.757 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:49:34.757 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.029814+0000", 2024-09-21T10:49:34.758 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001272343 2024-09-21T10:49:34.758 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.758 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:49:34.758 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.758 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.758 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.758 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:80160 53.12 53:4958c2ed:::benchmark_data_smithi070_228967_object80159:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1330)", 2024-09-21T10:49:34.758 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:10.695813+0000", 2024-09-21T10:49:34.758 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 503.92671093799999, 2024-09-21T10:49:34.758 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3537495580000001, 2024-09-21T10:49:34.758 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:49:34.758 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:49:34.758 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:49:34.758 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:49:34.759 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:49:34.759 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:49:34.759 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 80160 2024-09-21T10:49:34.759 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.759 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:49:34.759 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.759 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:49:34.759 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695813+0000", 2024-09-21T10:49:34.759 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.759 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.759 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.759 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:49:34.759 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695813+0000", 2024-09-21T10:49:34.759 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.760 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.760 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.760 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:49:34.760 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695814+0000", 2024-09-21T10:49:34.760 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7800000000000002e-07 2024-09-21T10:49:34.760 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.760 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.760 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:49:34.760 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695831+0000", 2024-09-21T10:49:34.760 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7311000000000001e-05 2024-09-21T10:49:34.760 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.760 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.760 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:49:34.760 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695846+0000", 2024-09-21T10:49:34.760 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.47e-05 2024-09-21T10:49:34.761 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.761 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.761 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:49:34.761 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695852+0000", 2024-09-21T10:49:34.761 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2190000000000001e-06 2024-09-21T10:49:34.761 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.761 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.761 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.761 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.697303+0000", 2024-09-21T10:49:34.761 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0014514840000000001 2024-09-21T10:49:34.761 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.761 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.761 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.761 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.697310+0000", 2024-09-21T10:49:34.762 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5019999999999998e-06 2024-09-21T10:49:34.762 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.762 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.762 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.762 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.029868+0000", 2024-09-21T10:49:34.762 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3325584930000001 2024-09-21T10:49:34.762 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.762 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.762 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:49:34.762 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.030023+0000", 2024-09-21T10:49:34.762 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015419899999999999 2024-09-21T10:49:34.762 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.762 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.762 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,3,6", 2024-09-21T10:49:34.763 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.030189+0000", 2024-09-21T10:49:34.763 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016664199999999999 2024-09-21T10:49:34.763 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.763 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.763 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.763 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.038428+0000", 2024-09-21T10:49:34.763 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0082384810000000006 2024-09-21T10:49:34.763 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.763 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.763 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.763 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.041162+0000", 2024-09-21T10:49:34.763 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0027339759999999999 2024-09-21T10:49:34.763 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.763 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.764 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:49:34.764 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.041175+0000", 2024-09-21T10:49:34.764 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3271000000000001e-05 2024-09-21T10:49:34.764 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.764 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.764 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.764 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.047349+0000", 2024-09-21T10:49:34.764 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0061744950000000003 2024-09-21T10:49:34.764 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.764 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.764 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:49:34.764 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.047430+0000", 2024-09-21T10:49:34.764 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0260999999999996e-05 2024-09-21T10:49:34.764 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.765 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.765 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:49:34.765 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.049563+0000", 2024-09-21T10:49:34.765 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0021331459999999998 2024-09-21T10:49:34.765 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.765 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:49:34.765 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.765 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.765 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.765 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:80214 53.12 53:4883ca5c:::benchmark_data_smithi070_228967_object80213:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1330)", 2024-09-21T10:49:34.765 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:10.695859+0000", 2024-09-21T10:49:34.765 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 503.92666572799999, 2024-09-21T10:49:34.765 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.355014194, 2024-09-21T10:49:34.765 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:49:34.766 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:49:34.766 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:49:34.766 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:49:34.766 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:49:34.766 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:49:34.766 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 80214 2024-09-21T10:49:34.766 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.766 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:49:34.766 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.766 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:49:34.766 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695859+0000", 2024-09-21T10:49:34.766 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.766 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.766 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.767 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:49:34.767 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695859+0000", 2024-09-21T10:49:34.767 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.767 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.767 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.767 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:49:34.767 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695859+0000", 2024-09-21T10:49:34.767 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3599999999999999e-07 2024-09-21T10:49:34.767 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.767 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.767 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:49:34.767 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695880+0000", 2024-09-21T10:49:34.767 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1478e-05 2024-09-21T10:49:34.767 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.768 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.768 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:49:34.768 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695921+0000", 2024-09-21T10:49:34.768 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0139999999999999e-05 2024-09-21T10:49:34.768 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.768 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.768 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:49:34.768 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.695926+0000", 2024-09-21T10:49:34.768 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9039999999999997e-06 2024-09-21T10:49:34.768 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.768 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.768 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.768 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.698208+0000", 2024-09-21T10:49:34.768 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00228123 2024-09-21T10:49:34.768 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.769 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.769 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.769 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:10.698216+0000", 2024-09-21T10:49:34.769 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1529999999999997e-06 2024-09-21T10:49:34.769 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.769 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.769 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.769 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.031500+0000", 2024-09-21T10:49:34.769 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3332841609999999 2024-09-21T10:49:34.769 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.769 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.769 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:49:34.769 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.031666+0000", 2024-09-21T10:49:34.769 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000165515 2024-09-21T10:49:34.769 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.769 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.770 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,3,6", 2024-09-21T10:49:34.770 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.031885+0000", 2024-09-21T10:49:34.770 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00021924600000000001 2024-09-21T10:49:34.770 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.770 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.770 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.770 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.038468+0000", 2024-09-21T10:49:34.770 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0065835700000000004 2024-09-21T10:49:34.770 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.770 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.770 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:49:34.770 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.041181+0000", 2024-09-21T10:49:34.770 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0027126630000000001 2024-09-21T10:49:34.771 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.771 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.771 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.771 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.041240+0000", 2024-09-21T10:49:34.771 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8641999999999999e-05 2024-09-21T10:49:34.771 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.771 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.771 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.771 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.049667+0000", 2024-09-21T10:49:34.771 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0084272570000000005 2024-09-21T10:49:34.771 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.771 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.771 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:49:34.771 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.049759+0000", 2024-09-21T10:49:34.772 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2334000000000001e-05 2024-09-21T10:49:34.772 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.772 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.772 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:49:34.772 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:12.050873+0000", 2024-09-21T10:49:34.772 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0011134649999999999 2024-09-21T10:49:34.772 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.772 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:49:34.772 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.772 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.772 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.772 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:238408 53.15 53:ae2b89db:::benchmark_data_smithi070_228967_object109488:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1354)", 2024-09-21T10:49:34.772 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:43:42.177749+0000", 2024-09-21T10:49:34.772 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 352.44477575600001, 2024-09-21T10:49:34.772 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3328045070000001, 2024-09-21T10:49:34.773 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:49:34.773 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:49:34.773 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:49:34.773 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:49:34.773 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:49:34.773 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:49:34.773 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 238408 2024-09-21T10:49:34.773 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.773 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:49:34.773 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.773 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:49:34.773 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:42.177749+0000", 2024-09-21T10:49:34.773 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.773 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.774 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.774 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:49:34.774 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:42.177749+0000", 2024-09-21T10:49:34.774 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:49:34.774 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.774 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.774 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:49:34.774 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:42.177749+0000", 2024-09-21T10:49:34.774 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1300000000000001e-07 2024-09-21T10:49:34.774 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.774 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.774 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:49:34.774 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:42.177751+0000", 2024-09-21T10:49:34.774 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.193e-06 2024-09-21T10:49:34.775 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.775 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.775 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:49:34.775 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:42.177758+0000", 2024-09-21T10:49:34.775 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7460000000000003e-06 2024-09-21T10:49:34.775 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.775 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.775 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:49:34.775 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:42.177763+0000", 2024-09-21T10:49:34.775 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2599999999999996e-06 2024-09-21T10:49:34.775 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.775 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.775 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.775 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:42.177984+0000", 2024-09-21T10:49:34.776 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000220898 2024-09-21T10:49:34.776 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.776 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.776 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting_for_map not empty", 2024-09-21T10:49:34.776 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:42.177989+0000", 2024-09-21T10:49:34.776 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6029999999999998e-06 2024-09-21T10:49:34.776 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.776 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.776 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.776 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:42.181212+0000", 2024-09-21T10:49:34.776 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0032235710000000002 2024-09-21T10:49:34.776 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.776 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.776 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:49:34.777 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:42.181223+0000", 2024-09-21T10:49:34.777 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0832999999999999e-05 2024-09-21T10:49:34.777 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.777 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.777 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:49:34.777 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:43.487360+0000", 2024-09-21T10:49:34.777 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3061371799999999 2024-09-21T10:49:34.777 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.777 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.777 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:49:34.777 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:43.487503+0000", 2024-09-21T10:49:34.777 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014267899999999999 2024-09-21T10:49:34.777 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.777 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.777 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,3,4,6", 2024-09-21T10:49:34.778 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:43.489004+0000", 2024-09-21T10:49:34.778 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001501631 2024-09-21T10:49:34.778 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.778 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.778 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.778 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:43.500172+0000", 2024-09-21T10:49:34.778 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.011167912 2024-09-21T10:49:34.778 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.778 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.778 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.778 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:43.500245+0000", 2024-09-21T10:49:34.778 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3146999999999996e-05 2024-09-21T10:49:34.778 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.778 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.779 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.779 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:43.500318+0000", 2024-09-21T10:49:34.779 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2065999999999997e-05 2024-09-21T10:49:34.779 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.779 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.779 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:49:34.779 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:43.501132+0000", 2024-09-21T10:49:34.779 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00081481200000000004 2024-09-21T10:49:34.779 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.779 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.779 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:49:34.779 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:43.510386+0000", 2024-09-21T10:49:34.779 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0092536349999999996 2024-09-21T10:49:34.779 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.780 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.780 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:49:34.780 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:43.510434+0000", 2024-09-21T10:49:34.780 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7960000000000002e-05 2024-09-21T10:49:34.780 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:49:34.781 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:49:34.781 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:49:34.781 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:43:43.510553+0000", 2024-09-21T10:49:34.781 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011906800000000001 2024-09-21T10:49:34.782 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.782 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:49:34.782 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.782 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:49:34.782 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:49:34.782 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:49:34.802 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 34 16 65840 65824 120.99 85.125 0.00879314 0.00825999 2024-09-21T10:49:34.811 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:34.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:34.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:34 smithi160 ceph-mon[101879]: pgmap v2540: 99 pgs: 99 active+clean; 3.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-21T10:49:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-21T10:49:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:34 smithi160 ceph-mon[101879]: osdmap e1421: 8 total, 8 up, 7 in 2024-09-21T10:49:34.913 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:34.947 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:34.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:34.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:35.049 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:35.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:35.090 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:35.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:35.191 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:35.220 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:35.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:35.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:35.321 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:35.360 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:35.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:35.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:35.462 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:35.499 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:35.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:35.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:35.585 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:49:35.601 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:35.651 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:49:35.651 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:49:35.651 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 3, 4, 5, 6, 7] 2024-09-21T10:49:35.651 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd out 5 2024-09-21T10:49:35.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:35.681 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:35.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:35.689 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:35.782 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:35.803 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 35 16 67064 67048 119.718 76.5 0.0125452 0.0083478 2024-09-21T10:49:35.834 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:35 smithi070 ceph-mon[112139]: osdmap e1422: 8 total, 8 up, 7 in 2024-09-21T10:49:35.835 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:49:35.835 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:49:35.835 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:35 smithi070 ceph-mon[107385]: osdmap e1422: 8 total, 8 up, 7 in 2024-09-21T10:49:35.835 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:49:35.835 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:49:35.838 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:35.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:35.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:35 smithi160 ceph-mon[101879]: osdmap e1422: 8 total, 8 up, 7 in 2024-09-21T10:49:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:49:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:49:35.940 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:35.977 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:35.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:35.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:36.079 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:36.106 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:36.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:36.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:36.208 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:36.260 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:36.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:36.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:36.362 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:36.382 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:49:36.401 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:36.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:36.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:36.503 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:36.541 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:36.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:36.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:36.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:36 smithi070 ceph-mon[112139]: pgmap v2543: 99 pgs: 99 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-21T10:49:36.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:36 smithi070 ceph-mon[112139]: osdmap e1423: 8 total, 8 up, 7 in 2024-09-21T10:49:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:36 smithi070 ceph-mon[107385]: pgmap v2543: 99 pgs: 99 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-21T10:49:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:36 smithi070 ceph-mon[107385]: osdmap e1423: 8 total, 8 up, 7 in 2024-09-21T10:49:36.643 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:36.671 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:36.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:36.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:36.773 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:36.803 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 36 16 68297 68281 118.533 77.0625 0.00666806 0.00843137 2024-09-21T10:49:36.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:36.804 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:36.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:36 smithi160 ceph-mon[101879]: pgmap v2543: 99 pgs: 99 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-21T10:49:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:36 smithi160 ceph-mon[101879]: osdmap e1423: 8 total, 8 up, 7 in 2024-09-21T10:49:36.905 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:36.930 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:36.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:36.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:37.032 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:37.059 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:37.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:37.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:37.161 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:37.189 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:37.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:37.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:37.291 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:37.333 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:37.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:37.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:37.434 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:37.488 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:37.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:37.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:37.590 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:37.622 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:37.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:37.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:37.724 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:37.752 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:37.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:37.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:37.803 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 37 16 69523 69507 117.4 76.625 0.00956697 0.00851266 2024-09-21T10:49:37.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:37 smithi160 ceph-mon[101879]: osdmap e1424: 8 total, 8 up, 7 in 2024-09-21T10:49:37.850 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:37.854 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:37.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:37 smithi070 ceph-mon[112139]: osdmap e1424: 8 total, 8 up, 7 in 2024-09-21T10:49:37.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:37 smithi070 ceph-mon[107385]: osdmap e1424: 8 total, 8 up, 7 in 2024-09-21T10:49:37.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:37.891 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:37.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:37.992 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:38.021 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:38.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:38.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:38.123 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:38.159 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:38.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:38.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:38.260 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:38.301 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:38.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:38.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:38.403 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:38.433 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:38.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:38.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:38.445 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:38.535 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:38.589 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:38.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:38.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:38.683 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:38.686 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:38 smithi070 ceph-mon[107385]: pgmap v2546: 99 pgs: 2 peering, 3 active+undersized+remapped, 5 remapped+peering, 89 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-09-21T10:49:38.686 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:38 smithi070 ceph-mon[107385]: Health check failed: Reduced data availability: 5 pgs peering (PG_AVAILABILITY) 2024-09-21T10:49:38.686 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:38 smithi070 ceph-mon[107385]: osdmap e1425: 8 total, 8 up, 7 in 2024-09-21T10:49:38.687 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:38 smithi070 ceph-mon[112139]: pgmap v2546: 99 pgs: 2 peering, 3 active+undersized+remapped, 5 remapped+peering, 89 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-09-21T10:49:38.687 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:38 smithi070 ceph-mon[112139]: Health check failed: Reduced data availability: 5 pgs peering (PG_AVAILABILITY) 2024-09-21T10:49:38.687 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:38 smithi070 ceph-mon[112139]: osdmap e1425: 8 total, 8 up, 7 in 2024-09-21T10:49:38.690 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:38.731 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:38.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:38.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:38.803 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 38 16 70705 70689 116.255 73.875 0.0123627 0.00859681 2024-09-21T10:49:38.832 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:38 smithi160 ceph-mon[101879]: pgmap v2546: 99 pgs: 2 peering, 3 active+undersized+remapped, 5 remapped+peering, 89 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-09-21T10:49:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:38 smithi160 ceph-mon[101879]: Health check failed: Reduced data availability: 5 pgs peering (PG_AVAILABILITY) 2024-09-21T10:49:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:38 smithi160 ceph-mon[101879]: osdmap e1425: 8 total, 8 up, 7 in 2024-09-21T10:49:38.875 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:38.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:38.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:38.976 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:39.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:39.010 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:39.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:39.111 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:39.147 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:39.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:39.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:39.249 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:39.290 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:39.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:39.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:39.392 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:39.432 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:39.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:39.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:39.534 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:39.569 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:39.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:39.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:39.672 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:39.700 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:39.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:39.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:39.802 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:39.803 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 39 16 71940 71924 115.253 77.1875 0.0113144 0.00867228 2024-09-21T10:49:39.840 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:39.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:39.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:39.943 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:39.969 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:39.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:39.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:40.071 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:40.092 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:40.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:40.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:40.193 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:40.216 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:40.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:40.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:40.318 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:40.341 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:40.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:40.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:40.443 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:40.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:40.465 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:40.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:40.566 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:40.588 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:40.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:40.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:40.605 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T10:49:40.690 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:40.725 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:40.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:40.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:40.803 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:2024-09-21T10:49:40.802244+0000 min lat: 0.00113626 max lat: 0.385227 avg lat: 0.00882801 2024-09-21T10:49:40.803 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:49:40.803 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 40 16 72483 72467 113.22 33.9375 0.0138932 0.00882801 2024-09-21T10:49:40.827 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:40 smithi160 ceph-mon[101879]: pgmap v2548: 99 pgs: 2 peering, 3 active+undersized+remapped, 5 remapped+peering, 89 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-21T10:49:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:40 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1530017152' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:49:40.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:40 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:49:40.853 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:40.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:40.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:40.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:40 smithi070 ceph-mon[107385]: pgmap v2548: 99 pgs: 2 peering, 3 active+undersized+remapped, 5 remapped+peering, 89 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-21T10:49:40.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:40 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1530017152' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:49:40.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:40 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:49:40.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:40 smithi070 ceph-mon[112139]: pgmap v2548: 99 pgs: 2 peering, 3 active+undersized+remapped, 5 remapped+peering, 89 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-21T10:49:40.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:40 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1530017152' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:49:40.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:40 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:49:40.954 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:40.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:40.977 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:40.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:41.078 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:41.106 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:41.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:41.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:41.208 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:41.229 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:41.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:41.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:41.331 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:41.370 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:41.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:41.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:41.472 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:41.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:41.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:41.602 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:41.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:41 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:49:41.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:41 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:49:41.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:41 smithi070 ceph-mon[107385]: osdmap e1426: 8 total, 8 up, 7 in 2024-09-21T10:49:41.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:41 smithi070 ceph-mon[107385]: 1.6 scrub starts 2024-09-21T10:49:41.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:41 smithi070 ceph-mon[107385]: 1.6 scrub ok 2024-09-21T10:49:41.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:41 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:49:41.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:41 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:49:41.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:41 smithi070 ceph-mon[112139]: osdmap e1426: 8 total, 8 up, 7 in 2024-09-21T10:49:41.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:41 smithi070 ceph-mon[112139]: 1.6 scrub starts 2024-09-21T10:49:41.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:41 smithi070 ceph-mon[112139]: 1.6 scrub ok 2024-09-21T10:49:41.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:41.631 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:41.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:41.732 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:41.771 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:41.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:41.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:41.803 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 41 16 73576 73560 112.124 68.3125 0.00476451 0.00891309 2024-09-21T10:49:41.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:41 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:49:41.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:41 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:49:41.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:41 smithi160 ceph-mon[101879]: osdmap e1426: 8 total, 8 up, 7 in 2024-09-21T10:49:41.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:41 smithi160 ceph-mon[101879]: 1.6 scrub starts 2024-09-21T10:49:41.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:41 smithi160 ceph-mon[101879]: 1.6 scrub ok 2024-09-21T10:49:41.873 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:41.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:41.917 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:41.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:42.018 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:42.049 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:42.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:42.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:42.151 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:42.195 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:42.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:42.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:42.297 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:42.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:42.339 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:42.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:42.440 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:42.473 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:42.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:42.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:42.575 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:42.601 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:42.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:42.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:42.699 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:42 smithi070 ceph-mon[107385]: pgmap v2550: 99 pgs: 2 peering, 3 active+undersized+remapped, 1 remapped+peering, 93 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s; 49 KiB/s, 0 objects/s recovering 2024-09-21T10:49:42.699 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:42 smithi070 ceph-mon[107385]: 1.a scrub starts 2024-09-21T10:49:42.699 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:42 smithi070 ceph-mon[107385]: 1.a scrub ok 2024-09-21T10:49:42.699 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:42 smithi070 ceph-mon[112139]: pgmap v2550: 99 pgs: 2 peering, 3 active+undersized+remapped, 1 remapped+peering, 93 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s; 49 KiB/s, 0 objects/s recovering 2024-09-21T10:49:42.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:42 smithi070 ceph-mon[112139]: 1.a scrub starts 2024-09-21T10:49:42.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:42 smithi070 ceph-mon[112139]: 1.a scrub ok 2024-09-21T10:49:42.703 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:42.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:42.770 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:42.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:42.803 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 42 16 74762 74746 111.22 74.125 0.00830673 0.00898617 2024-09-21T10:49:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:42 smithi160 ceph-mon[101879]: pgmap v2550: 99 pgs: 2 peering, 3 active+undersized+remapped, 1 remapped+peering, 93 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s; 49 KiB/s, 0 objects/s recovering 2024-09-21T10:49:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:42 smithi160 ceph-mon[101879]: 1.a scrub starts 2024-09-21T10:49:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:42 smithi160 ceph-mon[101879]: 1.a scrub ok 2024-09-21T10:49:42.871 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:42.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:42.921 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:42.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:43.022 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:43.054 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:43.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:43.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:43.155 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:43.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:43.216 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:43.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:43.293 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:49:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:49:43] "GET /metrics HTTP/1.1" 200 36602 "" "Prometheus/2.51.0" 2024-09-21T10:49:43.318 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:43.344 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:43.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:43.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:43.447 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:43.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:43.494 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:43.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:43.595 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:43.626 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:43.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:43.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:43.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a[107362]: 2024-09-21T10:49:43.290+0000 7fad4a465640 -1 mon.a@0(leader).osd e1426 definitely_dead 0 2024-09-21T10:49:43.729 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:43.759 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:43.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:43.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:43.803 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 43 16 75942 75926 110.348 73.75 0.017142 0.00905723 2024-09-21T10:49:43.846 INFO:teuthology.orchestra.run.smithi070.stderr:marked out osd.5. 2024-09-21T10:49:43.861 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:43.894 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:43.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:43.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:43.996 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:44.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:44.019 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:44.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:43 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:49:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:43 smithi160 ceph-mon[101879]: pgmap v2551: 99 pgs: 99 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 67 MiB/s wr, 1.07k op/s; 17 KiB/s, 0 objects/s recovering 2024-09-21T10:49:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:43 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:49:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:43 smithi160 ceph-mon[101879]: Health check update: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:49:44.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:43 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3854284107' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-21T10:49:44.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:43 smithi160 ceph-mon[101879]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-21T10:49:44.120 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:44.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:43 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:49:44.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:43 smithi070 ceph-mon[112139]: pgmap v2551: 99 pgs: 99 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 67 MiB/s wr, 1.07k op/s; 17 KiB/s, 0 objects/s recovering 2024-09-21T10:49:44.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:43 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:49:44.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:43 smithi070 ceph-mon[112139]: Health check update: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:49:44.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:43 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3854284107' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-21T10:49:44.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:43 smithi070 ceph-mon[112139]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-21T10:49:44.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:43 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:49:44.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:43 smithi070 ceph-mon[107385]: pgmap v2551: 99 pgs: 99 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 67 MiB/s wr, 1.07k op/s; 17 KiB/s, 0 objects/s recovering 2024-09-21T10:49:44.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:43 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:49:44.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:43 smithi070 ceph-mon[107385]: Health check update: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:49:44.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:43 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3854284107' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-21T10:49:44.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:43 smithi070 ceph-mon[107385]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-21T10:49:44.141 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:44.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:44.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:44.244 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:44.266 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:44.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:44.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:44.368 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:44.390 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:44.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:44.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:44.491 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:44.514 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:44.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:44.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:44.615 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:44.637 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:44.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:44.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:44.739 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:44.761 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:44.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:44.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:44.803 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 44 16 76035 76019 107.972 5.8125 0.00360747 0.00906261 2024-09-21T10:49:44.863 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:44.888 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:44.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:44.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:44.989 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:45.011 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:45.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:45.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:45.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:44 smithi160 ceph-mon[101879]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T10:49:45.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:44 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3854284107' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-21T10:49:45.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:44 smithi160 ceph-mon[101879]: osdmap e1427: 8 total, 8 up, 6 in 2024-09-21T10:49:45.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:44 smithi160 ceph-mon[101879]: 1.7 scrub starts 2024-09-21T10:49:45.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:44 smithi160 ceph-mon[101879]: 1.7 scrub ok 2024-09-21T10:49:45.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:44 smithi160 ceph-mon[101879]: 1.0 scrub starts 2024-09-21T10:49:45.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:44 smithi160 ceph-mon[101879]: 1.0 scrub ok 2024-09-21T10:49:45.113 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:45.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:44 smithi070 ceph-mon[107385]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T10:49:45.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:44 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3854284107' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-21T10:49:45.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:44 smithi070 ceph-mon[107385]: osdmap e1427: 8 total, 8 up, 6 in 2024-09-21T10:49:45.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:44 smithi070 ceph-mon[107385]: 1.7 scrub starts 2024-09-21T10:49:45.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:44 smithi070 ceph-mon[107385]: 1.7 scrub ok 2024-09-21T10:49:45.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:44 smithi070 ceph-mon[107385]: 1.0 scrub starts 2024-09-21T10:49:45.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:44 smithi070 ceph-mon[107385]: 1.0 scrub ok 2024-09-21T10:49:45.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:44 smithi070 ceph-mon[112139]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T10:49:45.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:44 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3854284107' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-21T10:49:45.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:44 smithi070 ceph-mon[112139]: osdmap e1427: 8 total, 8 up, 6 in 2024-09-21T10:49:45.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:44 smithi070 ceph-mon[112139]: 1.7 scrub starts 2024-09-21T10:49:45.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:44 smithi070 ceph-mon[112139]: 1.7 scrub ok 2024-09-21T10:49:45.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:44 smithi070 ceph-mon[112139]: 1.0 scrub starts 2024-09-21T10:49:45.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:44 smithi070 ceph-mon[112139]: 1.0 scrub ok 2024-09-21T10:49:45.134 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:45.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:45.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:45.238 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:45.259 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:45.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:45.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:45.361 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:45.382 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:45.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:45.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:45.484 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:45.506 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:45.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:45.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:45.607 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:45.629 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:45.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:45.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:45.731 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:45.752 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:45.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:45.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:45.804 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 45 16 76035 76019 105.573 0 - 0.00906261 2024-09-21T10:49:45.854 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:45.886 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:45.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:45.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:45.894 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:49:45.972 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:45 smithi070 ceph-mon[107385]: pgmap v2553: 99 pgs: 4 peering, 95 active+clean; 4.5 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 52 MiB/s wr, 836 op/s; 57 KiB/s, 0 objects/s recovering 2024-09-21T10:49:45.973 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:45 smithi070 ceph-mon[107385]: osdmap e1428: 8 total, 8 up, 6 in 2024-09-21T10:49:45.973 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:45 smithi070 ceph-mon[112139]: pgmap v2553: 99 pgs: 4 peering, 95 active+clean; 4.5 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 52 MiB/s wr, 836 op/s; 57 KiB/s, 0 objects/s recovering 2024-09-21T10:49:45.973 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:45 smithi070 ceph-mon[112139]: osdmap e1428: 8 total, 8 up, 6 in 2024-09-21T10:49:45.987 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:46.009 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:46.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:46.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:46.111 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:46.136 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:46.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:46.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:46.238 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:46.271 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:46.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:46.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:46.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:45 smithi160 ceph-mon[101879]: pgmap v2553: 99 pgs: 4 peering, 95 active+clean; 4.5 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 52 MiB/s wr, 836 op/s; 57 KiB/s, 0 objects/s recovering 2024-09-21T10:49:46.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:45 smithi160 ceph-mon[101879]: osdmap e1428: 8 total, 8 up, 6 in 2024-09-21T10:49:46.374 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:46.405 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:46.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:46.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:46.507 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:46.544 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:46.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:46.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:46.646 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:46.680 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:46.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:46.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:46.782 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:46.803 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 46 16 76700 76684 104.181 20.7812 0.0107793 0.00959183 2024-09-21T10:49:46.814 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:46.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:46.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:46.916 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:46.944 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:46.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:46.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:47.046 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:47.082 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:47.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:47.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:47.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:46 smithi070 ceph-mon[107385]: osdmap e1429: 8 total, 8 up, 6 in 2024-09-21T10:49:47.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:46 smithi070 ceph-mon[112139]: osdmap e1429: 8 total, 8 up, 6 in 2024-09-21T10:49:47.184 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:47.218 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:47.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:47.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:47.320 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:47.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:46 smithi160 ceph-mon[101879]: osdmap e1429: 8 total, 8 up, 6 in 2024-09-21T10:49:47.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:47.358 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:47.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:47.459 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:47.495 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:47.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:47.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:47.597 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:47.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:47.639 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:47.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:47.740 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:47.782 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:47.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:47.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:47.804 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 47 16 77600 77584 103.161 56.25 0.00686598 0.00968792 2024-09-21T10:49:47.884 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:47.933 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:47.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:47.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:48.014 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:49:48.015 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:49:48.015 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:47 smithi070 ceph-mon[107385]: pgmap v2556: 99 pgs: 2 active+recovering+undersized+remapped, 3 active+undersized+remapped, 5 remapped+peering, 4 peering, 85 active+clean; 4.7 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 65 MiB/s wr, 1.03k op/s; 9559/228744 objects misplaced (4.179%); 123 KiB/s, 5 objects/s recovering 2024-09-21T10:49:48.015 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:47 smithi070 ceph-mon[107385]: osdmap e1430: 8 total, 8 up, 6 in 2024-09-21T10:49:48.015 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:47 smithi070 ceph-mon[112139]: pgmap v2556: 99 pgs: 2 active+recovering+undersized+remapped, 3 active+undersized+remapped, 5 remapped+peering, 4 peering, 85 active+clean; 4.7 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 65 MiB/s wr, 1.03k op/s; 9559/228744 objects misplaced (4.179%); 123 KiB/s, 5 objects/s recovering 2024-09-21T10:49:48.015 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:47 smithi070 ceph-mon[112139]: osdmap e1430: 8 total, 8 up, 6 in 2024-09-21T10:49:48.035 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:48.048 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:49:48.048 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:49:48.066 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:48.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:48.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:48.094 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:49:48.094 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:49:48.136 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:49:48.136 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:49:48.163 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:49:48.163 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:49:48.167 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:48.198 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:49:48.198 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:49:48.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:48.199 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:48.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:48.236 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:49:48.236 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:49:48.271 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:49:48.271 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:49:48.299 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:48.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:47 smithi160 ceph-mon[101879]: pgmap v2556: 99 pgs: 2 active+recovering+undersized+remapped, 3 active+undersized+remapped, 5 remapped+peering, 4 peering, 85 active+clean; 4.7 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 65 MiB/s wr, 1.03k op/s; 9559/228744 objects misplaced (4.179%); 123 KiB/s, 5 objects/s recovering 2024-09-21T10:49:48.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:47 smithi160 ceph-mon[101879]: osdmap e1430: 8 total, 8 up, 6 in 2024-09-21T10:49:48.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:48.349 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:48.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:48.350 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:49:48.350 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:49:48.451 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:48.491 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:48.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:48.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:48.593 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:48.629 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:48.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:48.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:48.731 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:48.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:48.766 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:48.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:48.804 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 48 16 78508 78492 102.194 56.75 0.00663021 0.00978024 2024-09-21T10:49:48.867 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:48.900 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:48.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:48.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:49.002 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:49.033 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:49.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:49.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:49.064 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:48 smithi070 ceph-mon[112139]: osdmap e1431: 8 total, 8 up, 6 in 2024-09-21T10:49:49.064 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:48 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2339463848' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:49:49.064 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:48 smithi070 ceph-mon[107385]: osdmap e1431: 8 total, 8 up, 6 in 2024-09-21T10:49:49.064 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:48 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2339463848' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:49:49.135 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:49.169 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:49.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:49.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:49.271 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:49.306 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:49.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:49.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:49.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:48 smithi160 ceph-mon[101879]: osdmap e1431: 8 total, 8 up, 6 in 2024-09-21T10:49:49.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:48 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2339463848' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:49:49.409 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:49.440 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:49.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:49.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:49.542 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:49.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:49.571 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:49.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:49.672 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:49.705 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:49.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:49.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:49.762 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:49:49.778 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:49.805 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 49 16 79399 79383 101.245 55.6875 0.0314449 0.00987015 2024-09-21T10:49:49.807 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:49.852 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:49.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:49.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:49.953 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:50.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:50.003 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:50.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:50.104 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:50.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:49 smithi070 ceph-mon[107385]: pgmap v2559: 99 pgs: 2 active+recovering+undersized+remapped, 3 active+undersized+remapped, 5 remapped+peering, 4 peering, 85 active+clean; 4.7 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 48 MiB/s wr, 757 op/s; 9559/228744 objects misplaced (4.179%); 224 KiB/s, 8 objects/s recovering 2024-09-21T10:49:50.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:49 smithi070 ceph-mon[112139]: pgmap v2559: 99 pgs: 2 active+recovering+undersized+remapped, 3 active+undersized+remapped, 5 remapped+peering, 4 peering, 85 active+clean; 4.7 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 48 MiB/s wr, 757 op/s; 9559/228744 objects misplaced (4.179%); 224 KiB/s, 8 objects/s recovering 2024-09-21T10:49:50.139 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:50.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:50.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:50.240 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:50.284 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:50.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:50.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:50.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:49 smithi160 ceph-mon[101879]: pgmap v2559: 99 pgs: 2 active+recovering+undersized+remapped, 3 active+undersized+remapped, 5 remapped+peering, 4 peering, 85 active+clean; 4.7 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 48 MiB/s wr, 757 op/s; 9559/228744 objects misplaced (4.179%); 224 KiB/s, 8 objects/s recovering 2024-09-21T10:49:50.385 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:50.419 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:50.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:50.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:50.522 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:50.556 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:50.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:50.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:50.658 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:50.693 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:50.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:50.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:50.796 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:50.804 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 50 16 80348 80332 100.406 59.3125 0.0264167 0.00995312 2024-09-21T10:49:50.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:50.829 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:50.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:50.930 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:50.969 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:50.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:50.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:51.071 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:51.108 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:49:51.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:51.110 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:51.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:51.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:49:51.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:49:51.211 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:51.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:51.242 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:51.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:49:51.343 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:51.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:51.387 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:51.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:51.488 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:51.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:51.524 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:51.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:51.624 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:51.656 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:51.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:51.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:51.758 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:51.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:51.792 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:51.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:51.804 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 51 16 81303 81287 99.6075 59.6875 0.0490401 0.010033 2024-09-21T10:49:51.844 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:51.893 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:51.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:51.951 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:51.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:52.052 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:52.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:52.087 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:52.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:52.119 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:51 smithi070 ceph-mon[107385]: pgmap v2560: 99 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+undersized+remapped, 91 active+clean; 4.8 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 51 MiB/s wr, 808 op/s; 13930/234189 objects degraded (5.948%); 9559/234189 objects misplaced (4.082%); 267 KiB/s, 6 objects/s recovering 2024-09-21T10:49:52.119 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:51 smithi070 ceph-mon[107385]: Health check failed: Degraded data redundancy: 13930/234189 objects degraded (5.948%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:49:52.119 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:51 smithi070 ceph-mon[112139]: pgmap v2560: 99 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+undersized+remapped, 91 active+clean; 4.8 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 51 MiB/s wr, 808 op/s; 13930/234189 objects degraded (5.948%); 9559/234189 objects misplaced (4.082%); 267 KiB/s, 6 objects/s recovering 2024-09-21T10:49:52.119 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:51 smithi070 ceph-mon[112139]: Health check failed: Degraded data redundancy: 13930/234189 objects degraded (5.948%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:49:52.188 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:52.226 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:52.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:52.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:52.242 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:51 smithi160 ceph-mon[101879]: pgmap v2560: 99 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+undersized+remapped, 91 active+clean; 4.8 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 51 MiB/s wr, 808 op/s; 13930/234189 objects degraded (5.948%); 9559/234189 objects misplaced (4.082%); 267 KiB/s, 6 objects/s recovering 2024-09-21T10:49:52.242 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:51 smithi160 ceph-mon[101879]: Health check failed: Degraded data redundancy: 13930/234189 objects degraded (5.948%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:49:52.328 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:52.366 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:52.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:52.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:52.468 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:52.509 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:52.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:52.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:52.611 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:52.659 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:52.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:52.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:52.760 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:52.792 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:52.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:52.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:52.804 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 52 16 82209 82193 98.7809 56.625 0.00841181 0.0101174 2024-09-21T10:49:52.894 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:52.922 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:52.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:52.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:53.024 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:53.056 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:53.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:53.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:53.091 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 6, 7] out_osds: [2, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:49:53.092 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:49:53.092 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-21T10:49:53.092 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 110 3.0 2024-09-21T10:49:53.158 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:53.179 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:53.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:53.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:53.281 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:53.301 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:49:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:49:53] "GET /metrics HTTP/1.1" 200 36623 "" "Prometheus/2.51.0" 2024-09-21T10:49:53.318 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:53.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:53.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:53.420 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:53.454 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:53.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:53.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:53.556 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:53.604 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:53.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:53.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:53.706 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:53.743 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:53.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:53.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:53.804 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 53 16 83055 83039 97.9147 52.875 0.0323705 0.0102082 2024-09-21T10:49:53.845 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:53.883 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:53.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:53.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:53.906 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T10:49:53.985 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:54.018 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:54.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:54.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:54.120 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:54.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:54.161 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:54.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:54.261 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:54.275 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:53 smithi070 ceph-mon[107385]: pgmap v2561: 99 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 94 active+clean; 4.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 51 MiB/s wr, 805 op/s; 13930/240999 objects degraded (5.780%); 9311/240999 objects misplaced (3.864%); 2.4 MiB/s, 40 objects/s recovering 2024-09-21T10:49:54.275 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:53 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1689863881' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:49:54.275 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:53 smithi070 ceph-mon[112139]: pgmap v2561: 99 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 94 active+clean; 4.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 51 MiB/s wr, 805 op/s; 13930/240999 objects degraded (5.780%); 9311/240999 objects misplaced (3.864%); 2.4 MiB/s, 40 objects/s recovering 2024-09-21T10:49:54.275 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:53 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1689863881' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:49:54.296 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:54.297 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:53 smithi160 ceph-mon[101879]: pgmap v2561: 99 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 94 active+clean; 4.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 51 MiB/s wr, 805 op/s; 13930/240999 objects degraded (5.780%); 9311/240999 objects misplaced (3.864%); 2.4 MiB/s, 40 objects/s recovering 2024-09-21T10:49:54.297 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:53 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1689863881' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:49:54.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:54.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:54.398 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:54.441 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:54.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:54.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:54.543 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:54.579 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:54.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:54.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:54.681 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:54.723 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:54.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:54.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:54.804 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 54 16 83917 83901 97.099 53.875 0.0066743 0.0102929 2024-09-21T10:49:54.825 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:54.859 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:54.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:54.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:54.961 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:54.993 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:54.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:54.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:55.095 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:55.130 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:55.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:55.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:55.198 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:54 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:49:55.198 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:54 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1689863881' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:49:55.198 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:54 smithi070 ceph-mon[107385]: osdmap e1432: 8 total, 8 up, 6 in 2024-09-21T10:49:55.199 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:54 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:49:55.199 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:54 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1689863881' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:49:55.199 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:54 smithi070 ceph-mon[112139]: osdmap e1432: 8 total, 8 up, 6 in 2024-09-21T10:49:55.214 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:54 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:49:55.214 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:54 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1689863881' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:49:55.215 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:54 smithi160 ceph-mon[101879]: osdmap e1432: 8 total, 8 up, 6 in 2024-09-21T10:49:55.233 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:55.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:55.263 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:55.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:55.364 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:55.415 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:55.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:55.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:55.517 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:55.529 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:55.529 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:55.547 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:55.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:55.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:55.572 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:49:55.573 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:49:55.591 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 10:49:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=cleanup t=2024-09-21T10:49:55.212084033Z level=info msg="Completed cleanup jobs" duration=95.348017ms 2024-09-21T10:49:55.591 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 10:49:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=plugins.update.checker t=2024-09-21T10:49:55.330261022Z level=info msg="Update check succeeded" duration=124.331102ms 2024-09-21T10:49:55.649 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:55.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:55.695 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:55.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:55.796 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:55.807 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 55 16 84706 84690 96.2259 49.3125 0.0164257 0.0103868 2024-09-21T10:49:55.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:55.847 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:55.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:55.948 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:55.978 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:55.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:55.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:56.080 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:56.116 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:56.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:56.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:56.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:55 smithi070 ceph-mon[107385]: pgmap v2563: 99 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 94 active+clean; 4.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 37 MiB/s wr, 585 op/s; 13930/242748 objects degraded (5.738%); 9311/242748 objects misplaced (3.836%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T10:49:56.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:55 smithi070 ceph-mon[112139]: pgmap v2563: 99 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 94 active+clean; 4.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 37 MiB/s wr, 585 op/s; 13930/242748 objects degraded (5.738%); 9311/242748 objects misplaced (3.836%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T10:49:56.217 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:56.258 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:56.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:56.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:56.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:55 smithi160 ceph-mon[101879]: pgmap v2563: 99 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 94 active+clean; 4.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 37 MiB/s wr, 585 op/s; 13930/242748 objects degraded (5.738%); 9311/242748 objects misplaced (3.836%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T10:49:56.360 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:56.399 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:56.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:56.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:56.501 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:56.531 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:56.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:56.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:56.626 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:49:56.633 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:56.706 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:56.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:56.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:56.807 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 56 16 85591 85575 95.4952 55.3125 0.0260916 0.0104662 2024-09-21T10:49:56.808 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:56.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:56.836 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:56.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:56.937 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:56.969 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:56.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:56.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:57.072 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:57.104 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:57.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:57.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:57.206 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:57.235 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:57.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:57.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:57.335 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:57 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 13930/254274 objects degraded (5.478%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:49:57.335 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:57 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 13930/254274 objects degraded (5.478%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:49:57.337 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:57.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:57 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 13930/254274 objects degraded (5.478%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:49:57.375 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:57.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:57.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:57.477 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:57.511 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:57.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:57.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:57.612 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:57.646 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:57.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:57.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:57.748 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:57.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:57.783 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:57.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:57.807 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 57 16 86546 86530 94.8669 59.6875 0.0110613 0.0105344 2024-09-21T10:49:57.823 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:49:57.885 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:57.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:57.916 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:57.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:58.017 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:58.057 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:58.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:58.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:58.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:49:58 smithi070 ceph-mon[107385]: pgmap v2564: 99 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 94 active+clean; 5.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 59 MiB/s wr, 947 op/s; 13930/254274 objects degraded (5.478%); 9050/254274 objects misplaced (3.559%); 3.6 MiB/s, 57 objects/s recovering 2024-09-21T10:49:58.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:49:58 smithi070 ceph-mon[112139]: pgmap v2564: 99 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 94 active+clean; 5.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 59 MiB/s wr, 947 op/s; 13930/254274 objects degraded (5.478%); 9050/254274 objects misplaced (3.559%); 3.6 MiB/s, 57 objects/s recovering 2024-09-21T10:49:58.158 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:58.196 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:58.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:58.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:58.298 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:58.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:49:58 smithi160 ceph-mon[101879]: pgmap v2564: 99 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 94 active+clean; 5.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 59 MiB/s wr, 947 op/s; 13930/254274 objects degraded (5.478%); 9050/254274 objects misplaced (3.559%); 3.6 MiB/s, 57 objects/s recovering 2024-09-21T10:49:58.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:58.344 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:58.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:58.445 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:58.479 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:58.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:58.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:58.581 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:58.614 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:58.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:58.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:58.715 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:58.753 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:58.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:58.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:58.756 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:58.807 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 58 16 87434 87418 94.1881 55.5 0.00979555 0.0106117 2024-09-21T10:49:58.855 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:58.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:58.895 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:58.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:58.996 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:59.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:59.046 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:59.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:59.147 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:59.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:59.191 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:59.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:59.292 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:59.326 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:49:59.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:59.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:59.428 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:59.465 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:59.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:59.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:59.567 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:59.603 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:59.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:59.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:59.705 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:59.706 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:59.735 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:59.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:59.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:59.808 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 59 16 88349 88333 93.5602 57.1875 0.00550317 0.0106824 2024-09-21T10:49:59.836 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:59.869 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:49:59.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:59.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:59.970 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:00.039 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:00.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:00.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:00.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:00 smithi070 ceph-mon[107385]: pgmap v2565: 99 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 94 active+clean; 5.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 53 MiB/s wr, 850 op/s; 13930/254274 objects degraded (5.478%); 9050/254274 objects misplaced (3.559%); 3.2 MiB/s, 51 objects/s recovering 2024-09-21T10:50:00.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:00 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:50:00.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:00 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:50:00.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:00 smithi070 ceph-mon[107385]: Health detail: HEALTH_WARN Degraded data redundancy: 13930/254274 objects degraded (5.478%), 3 pgs degraded 2024-09-21T10:50:00.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:00 smithi070 ceph-mon[107385]: [WRN] PG_DEGRADED: Degraded data redundancy: 13930/254274 objects degraded (5.478%), 3 pgs degraded 2024-09-21T10:50:00.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:00 smithi070 ceph-mon[107385]: pg 55.3 is active+recovery_wait+undersized+degraded+remapped, acting [1,7] 2024-09-21T10:50:00.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:00 smithi070 ceph-mon[107385]: pg 55.8 is active+recovery_wait+undersized+degraded+remapped, acting [0,6] 2024-09-21T10:50:00.129 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:00 smithi070 ceph-mon[107385]: pg 55.9 is active+recovery_wait+undersized+degraded+remapped, acting [3,4] 2024-09-21T10:50:00.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:00 smithi070 ceph-mon[112139]: pgmap v2565: 99 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 94 active+clean; 5.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 53 MiB/s wr, 850 op/s; 13930/254274 objects degraded (5.478%); 9050/254274 objects misplaced (3.559%); 3.2 MiB/s, 51 objects/s recovering 2024-09-21T10:50:00.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:00 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:50:00.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:00 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:50:00.129 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:00 smithi070 ceph-mon[112139]: Health detail: HEALTH_WARN Degraded data redundancy: 13930/254274 objects degraded (5.478%), 3 pgs degraded 2024-09-21T10:50:00.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:00 smithi070 ceph-mon[112139]: [WRN] PG_DEGRADED: Degraded data redundancy: 13930/254274 objects degraded (5.478%), 3 pgs degraded 2024-09-21T10:50:00.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:00 smithi070 ceph-mon[112139]: pg 55.3 is active+recovery_wait+undersized+degraded+remapped, acting [1,7] 2024-09-21T10:50:00.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:00 smithi070 ceph-mon[112139]: pg 55.8 is active+recovery_wait+undersized+degraded+remapped, acting [0,6] 2024-09-21T10:50:00.130 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:00 smithi070 ceph-mon[112139]: pg 55.9 is active+recovery_wait+undersized+degraded+remapped, acting [3,4] 2024-09-21T10:50:00.141 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:00.171 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:00.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:00.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:00.272 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:00.312 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:00.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:00.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:00.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:00 smithi160 ceph-mon[101879]: pgmap v2565: 99 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 94 active+clean; 5.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 53 MiB/s wr, 850 op/s; 13930/254274 objects degraded (5.478%); 9050/254274 objects misplaced (3.559%); 3.2 MiB/s, 51 objects/s recovering 2024-09-21T10:50:00.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:00 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:50:00.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:00 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:50:00.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:00 smithi160 ceph-mon[101879]: Health detail: HEALTH_WARN Degraded data redundancy: 13930/254274 objects degraded (5.478%), 3 pgs degraded 2024-09-21T10:50:00.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:00 smithi160 ceph-mon[101879]: [WRN] PG_DEGRADED: Degraded data redundancy: 13930/254274 objects degraded (5.478%), 3 pgs degraded 2024-09-21T10:50:00.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:00 smithi160 ceph-mon[101879]: pg 55.3 is active+recovery_wait+undersized+degraded+remapped, acting [1,7] 2024-09-21T10:50:00.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:00 smithi160 ceph-mon[101879]: pg 55.8 is active+recovery_wait+undersized+degraded+remapped, acting [0,6] 2024-09-21T10:50:00.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:00 smithi160 ceph-mon[101879]: pg 55.9 is active+recovery_wait+undersized+degraded+remapped, acting [3,4] 2024-09-21T10:50:00.413 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:00.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:00.446 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:00.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:00.547 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:00.578 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:00.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:00.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:00.680 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:00.711 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:00.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:00.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:00.808 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:2024-09-21T10:50:00.807007+0000 min lat: 0.00113626 max lat: 2.33372 avg lat: 0.0107554 2024-09-21T10:50:00.808 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:50:00.808 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 60 16 89242 89226 92.931 55.8125 0.024753 0.0107554 2024-09-21T10:50:00.813 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:00.848 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:00.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:00.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:00.950 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:00.983 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:00.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:00.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:01.085 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:01.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:01.123 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:01.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:01.224 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:01.256 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:01.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:01.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:01.358 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:01.397 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:01.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:01.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:01.499 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:01.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:01.535 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:01.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:01.636 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:01.667 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:01.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:01.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:01.769 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:01.797 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:01.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:01.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:01.808 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 61 16 90188 90172 92.3767 59.125 0.00571677 0.0108183 2024-09-21T10:50:01.899 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:01.932 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:01.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:01.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:02.034 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:02.073 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:02.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:02.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:02.161 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:50:02.161 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:50:02.161 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:50:02.161 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:50:02.162 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:50:02.162 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:50:02.162 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:50:02.162 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:50:02.162 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:50:02.163 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:50:02.163 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:50:02.163 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:50:02.163 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:50:02.163 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:50:02.163 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:50:02.164 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:50:02.175 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:02.209 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:02.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:02.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:02.310 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:02 smithi160 ceph-mon[101879]: pgmap v2566: 99 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 94 active+clean; 5.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 56 MiB/s wr, 889 op/s; 13930/260892 objects degraded (5.339%); 9050/260892 objects misplaced (3.469%); 3.2 MiB/s, 50 objects/s recovering 2024-09-21T10:50:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:02 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/900778205' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:50:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:02 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:50:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:02 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:50:02.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:02 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:50:02.342 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:02.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:02.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:02.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:02 smithi070 ceph-mon[112139]: pgmap v2566: 99 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 94 active+clean; 5.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 56 MiB/s wr, 889 op/s; 13930/260892 objects degraded (5.339%); 9050/260892 objects misplaced (3.469%); 3.2 MiB/s, 50 objects/s recovering 2024-09-21T10:50:02.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:02 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/900778205' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:50:02.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:02 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:50:02.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:02 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:50:02.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:02 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:50:02.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:02 smithi070 ceph-mon[107385]: pgmap v2566: 99 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 94 active+clean; 5.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 56 MiB/s wr, 889 op/s; 13930/260892 objects degraded (5.339%); 9050/260892 objects misplaced (3.469%); 3.2 MiB/s, 50 objects/s recovering 2024-09-21T10:50:02.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:02 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/900778205' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:50:02.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:02 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:50:02.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:02 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:50:02.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:02 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:50:02.444 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:02.478 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:02.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:02.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:02.579 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:02.614 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:02.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:02.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:02.716 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:02.745 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:02.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:02.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:02.808 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 62 16 91059 91043 91.7647 54.4375 0.0195803 0.0108912 2024-09-21T10:50:02.847 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:02.886 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:02.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:02.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:02.987 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:03.021 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:03.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:03.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:03.038 INFO:teuthology.orchestra.run.smithi070.stdout:moved 34 / 297 (11.4478%) 2024-09-21T10:50:03.039 INFO:teuthology.orchestra.run.smithi070.stdout:avg 49.5 2024-09-21T10:50:03.039 INFO:teuthology.orchestra.run.smithi070.stdout:stddev 3.0957 -> 10.2754 (expected baseline 6.42262) 2024-09-21T10:50:03.039 INFO:teuthology.orchestra.run.smithi070.stdout:min osd.0 with 44 -> 57 pgs (0.888889 -> 1.15152 * mean) 2024-09-21T10:50:03.039 INFO:teuthology.orchestra.run.smithi070.stdout:max osd.6 with 54 -> 51 pgs (1.09091 -> 1.0303 * mean) 2024-09-21T10:50:03.039 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:50:03.039 INFO:teuthology.orchestra.run.smithi070.stdout:oload 110 2024-09-21T10:50:03.039 INFO:teuthology.orchestra.run.smithi070.stdout:max_change 3 2024-09-21T10:50:03.039 INFO:teuthology.orchestra.run.smithi070.stdout:max_change_osds 4 2024-09-21T10:50:03.039 INFO:teuthology.orchestra.run.smithi070.stdout:average_utilization 0.0334 2024-09-21T10:50:03.039 INFO:teuthology.orchestra.run.smithi070.stdout:overload_utilization 0.0367 2024-09-21T10:50:03.039 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 weight 1.0000 -> 0.6567 2024-09-21T10:50:03.039 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 weight 1.0000 -> 0.8224 2024-09-21T10:50:03.039 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 weight 1.0000 -> 0.8866 2024-09-21T10:50:03.123 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:03.147 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:03.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:03.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:03.250 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:03.278 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:03.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:03.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:03.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:03 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'3':'58107','6':'53897','7':'43038'}"} : dispatch 2024-09-21T10:50:03.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:03 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'3':'58107','6':'53897','7':'43038'}"} : dispatch 2024-09-21T10:50:03.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:50:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:50:03] "GET /metrics HTTP/1.1" 200 36623 "" "Prometheus/2.51.0" 2024-09-21T10:50:03.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:03 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'3':'58107','6':'53897','7':'43038'}"} : dispatch 2024-09-21T10:50:03.381 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:03.405 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:03.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:03.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:03.506 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:03.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:03.529 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:03.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:03.630 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:03.651 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:03.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:03.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:03.753 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:03.774 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:03.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:03.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:03.808 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 63 16 91309 91293 90.5562 15.625 0.00368635 0.0109157 2024-09-21T10:50:03.877 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:03.899 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:03.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:03.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:04.001 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:04.034 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:04.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:04.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:04.136 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:04.158 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:04.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:04.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:04.260 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:04.293 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:04.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:04.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:04.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:04 smithi160 ceph-mon[101879]: pgmap v2567: 99 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 94 active+clean; 5.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 56 MiB/s wr, 894 op/s; 13930/267831 objects degraded (5.201%); 8787/267831 objects misplaced (3.281%); 3.3 MiB/s, 52 objects/s recovering 2024-09-21T10:50:04.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:04 smithi160 ceph-mon[101879]: from='client.55165 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:50:04.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:04 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 13930/267831 objects degraded (5.201%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:50:04.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:04 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'58107','6':'53897','7':'43038'}"}]': finished 2024-09-21T10:50:04.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:04 smithi160 ceph-mon[101879]: osdmap e1433: 8 total, 8 up, 6 in 2024-09-21T10:50:04.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:04 smithi160 ceph-mon[101879]: osdmap e1434: 8 total, 8 up, 6 in 2024-09-21T10:50:04.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:04 smithi070 ceph-mon[107385]: pgmap v2567: 99 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 94 active+clean; 5.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 56 MiB/s wr, 894 op/s; 13930/267831 objects degraded (5.201%); 8787/267831 objects misplaced (3.281%); 3.3 MiB/s, 52 objects/s recovering 2024-09-21T10:50:04.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:04 smithi070 ceph-mon[107385]: from='client.55165 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:50:04.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:04 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 13930/267831 objects degraded (5.201%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:50:04.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:04 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'58107','6':'53897','7':'43038'}"}]': finished 2024-09-21T10:50:04.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:04 smithi070 ceph-mon[107385]: osdmap e1433: 8 total, 8 up, 6 in 2024-09-21T10:50:04.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:04 smithi070 ceph-mon[107385]: osdmap e1434: 8 total, 8 up, 6 in 2024-09-21T10:50:04.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:04 smithi070 ceph-mon[112139]: pgmap v2567: 99 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 94 active+clean; 5.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 56 MiB/s wr, 894 op/s; 13930/267831 objects degraded (5.201%); 8787/267831 objects misplaced (3.281%); 3.3 MiB/s, 52 objects/s recovering 2024-09-21T10:50:04.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:04 smithi070 ceph-mon[112139]: from='client.55165 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:50:04.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:04 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 13930/267831 objects degraded (5.201%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:50:04.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:04 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'58107','6':'53897','7':'43038'}"}]': finished 2024-09-21T10:50:04.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:04 smithi070 ceph-mon[112139]: osdmap e1433: 8 total, 8 up, 6 in 2024-09-21T10:50:04.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:04 smithi070 ceph-mon[112139]: osdmap e1434: 8 total, 8 up, 6 in 2024-09-21T10:50:04.395 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:04.419 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:04.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:04.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:04.520 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:04.542 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:04.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:04.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:04.643 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:04.665 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:04.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:04.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:04.767 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:04.792 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:04.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:04.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:04.808 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 64 16 91369 91353 89.1999 3.75 0.0264626 0.0111461 2024-09-21T10:50:04.894 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:04.921 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:04.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:04.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:05.023 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:05.050 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:05.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:05.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:05.152 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:05.176 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:05.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:05.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:05.278 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:05.301 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:05.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:05.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:05.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:05 smithi160 ceph-mon[101879]: osdmap e1435: 8 total, 8 up, 6 in 2024-09-21T10:50:05.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:05 smithi160 ceph-mon[101879]: 55.5 starting backfill to osd.3 from (0'0,0'0] MAX to 1432'5715 2024-09-21T10:50:05.403 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:05.428 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:05.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:05.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:05.530 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:05.558 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:05.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:05.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:05.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:05 smithi070 ceph-mon[107385]: osdmap e1435: 8 total, 8 up, 6 in 2024-09-21T10:50:05.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:05 smithi070 ceph-mon[107385]: 55.5 starting backfill to osd.3 from (0'0,0'0] MAX to 1432'5715 2024-09-21T10:50:05.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:05 smithi070 ceph-mon[112139]: osdmap e1435: 8 total, 8 up, 6 in 2024-09-21T10:50:05.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:05 smithi070 ceph-mon[112139]: 55.5 starting backfill to osd.3 from (0'0,0'0] MAX to 1432'5715 2024-09-21T10:50:05.660 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:05.686 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:05.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:05.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:05.787 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:05.808 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 65 16 92018 92002 88.4516 40.5625 0.0193736 0.0112989 2024-09-21T10:50:05.814 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:05.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:05.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:05.917 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:05.954 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:05.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:05.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:06.055 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:06.092 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:06.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:06.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:06.195 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:06.219 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:06.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:06.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:06.320 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:06.347 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:06.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:06.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:06.449 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:06.474 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:06.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:06.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:06.576 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:06 smithi160 ceph-mon[101879]: pgmap v2571: 99 pgs: 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 92 active+clean; 5.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 51 MiB/s wr, 809 op/s; 30761/268848 objects degraded (11.442%); 8787/268848 objects misplaced (3.268%); 2.7 MiB/s, 43 objects/s recovering 2024-09-21T10:50:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:06 smithi160 ceph-mon[101879]: osdmap e1436: 8 total, 8 up, 6 in 2024-09-21T10:50:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:50:06.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:50:06.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:06.615 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:06.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:06.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:06 smithi070 ceph-mon[112139]: pgmap v2571: 99 pgs: 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 92 active+clean; 5.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 51 MiB/s wr, 809 op/s; 30761/268848 objects degraded (11.442%); 8787/268848 objects misplaced (3.268%); 2.7 MiB/s, 43 objects/s recovering 2024-09-21T10:50:06.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:06 smithi070 ceph-mon[112139]: osdmap e1436: 8 total, 8 up, 6 in 2024-09-21T10:50:06.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:50:06.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:50:06.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:06 smithi070 ceph-mon[107385]: pgmap v2571: 99 pgs: 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 92 active+clean; 5.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 51 MiB/s wr, 809 op/s; 30761/268848 objects degraded (11.442%); 8787/268848 objects misplaced (3.268%); 2.7 MiB/s, 43 objects/s recovering 2024-09-21T10:50:06.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:06 smithi070 ceph-mon[107385]: osdmap e1436: 8 total, 8 up, 6 in 2024-09-21T10:50:06.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:50:06.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:50:06.717 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:06.743 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:06.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:06.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:06.808 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 66 16 92585 92569 87.6483 35.4375 0.00602113 0.0113951 2024-09-21T10:50:06.845 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:06.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:06.878 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:06.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:06.979 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:07.006 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:07.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:07.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:07.108 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:07.139 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:07.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:07.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:07.240 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:07.271 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:07.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:07.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:07.373 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:07.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:07 smithi070 ceph-mon[107385]: osdmap e1437: 8 total, 8 up, 6 in 2024-09-21T10:50:07.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:07 smithi070 ceph-mon[112139]: osdmap e1437: 8 total, 8 up, 6 in 2024-09-21T10:50:07.427 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:07.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:07.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:07.529 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:07.549 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:07.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:07.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:07.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:07 smithi160 ceph-mon[101879]: osdmap e1437: 8 total, 8 up, 6 in 2024-09-21T10:50:07.652 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:07.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:07.698 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:07.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:07.799 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:07.808 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 67 16 93214 93198 86.9268 39.3125 0.0228524 0.011498 2024-09-21T10:50:07.832 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:07.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:07.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:07.934 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:07.973 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:07.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:07.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:08.075 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:08.098 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:08.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:08.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:08.199 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:08.227 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:08.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:08.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:08.329 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:08.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:08.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:08.464 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:08.496 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:08.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:08.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:08 smithi160 ceph-mon[101879]: pgmap v2574: 99 pgs: 1 active+recovery_wait+degraded, 1 active+remapped+backfill_wait, 3 active+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.6 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 19 KiB/s rd, 45 MiB/s wr, 725 op/s; 47263/275970 objects degraded (17.126%); 14868/275970 objects misplaced (5.388%); 5.1 MiB/s, 80 objects/s recovering 2024-09-21T10:50:08.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:08 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 47263/275970 objects degraded (17.126%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:50:08.598 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:08.622 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:08.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:08 smithi070 ceph-mon[107385]: pgmap v2574: 99 pgs: 1 active+recovery_wait+degraded, 1 active+remapped+backfill_wait, 3 active+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.6 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 19 KiB/s rd, 45 MiB/s wr, 725 op/s; 47263/275970 objects degraded (17.126%); 14868/275970 objects misplaced (5.388%); 5.1 MiB/s, 80 objects/s recovering 2024-09-21T10:50:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:08 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 47263/275970 objects degraded (17.126%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:50:08.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:08 smithi070 ceph-mon[112139]: pgmap v2574: 99 pgs: 1 active+recovery_wait+degraded, 1 active+remapped+backfill_wait, 3 active+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.6 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 19 KiB/s rd, 45 MiB/s wr, 725 op/s; 47263/275970 objects degraded (17.126%); 14868/275970 objects misplaced (5.388%); 5.1 MiB/s, 80 objects/s recovering 2024-09-21T10:50:08.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:08 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 47263/275970 objects degraded (17.126%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:50:08.724 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:08.750 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:08.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:08.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:08.809 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 68 16 93855 93839 86.2376 40.0625 0.0238178 0.0115907 2024-09-21T10:50:08.852 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:08.900 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:08.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:08.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:09.002 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:09.038 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:09.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:09.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:09.140 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:09.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:09.187 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:09.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:09.288 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:09.313 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:09.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:09.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:09.414 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:09.461 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:09.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:09.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:09.563 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:09.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:09 smithi160 ceph-mon[101879]: 1.5 deep-scrub starts 2024-09-21T10:50:09.602 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:09.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:09.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:09.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:09 smithi070 ceph-mon[107385]: 1.5 deep-scrub starts 2024-09-21T10:50:09.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:09 smithi070 ceph-mon[112139]: 1.5 deep-scrub starts 2024-09-21T10:50:09.704 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:09.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:09.728 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:09.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:09.809 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 69 16 94425 94409 85.5041 35.625 0.0531855 0.0116886 2024-09-21T10:50:09.829 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:09.858 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:09.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:09.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:09.960 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:10.002 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:10.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:10.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:10.104 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:10.129 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:10.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:10.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:10.231 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:10.267 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:10.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:10.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:10.369 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:10.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:10.404 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:10.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:10.505 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:10.545 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:10.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:10.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:10 smithi160 ceph-mon[101879]: pgmap v2575: 99 pgs: 1 active+recovery_wait+degraded, 1 active+remapped+backfill_wait, 3 active+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.6 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 13 KiB/s rd, 30 MiB/s wr, 493 op/s; 47263/275970 objects degraded (17.126%); 14868/275970 objects misplaced (5.388%); 3.5 MiB/s, 54 objects/s recovering 2024-09-21T10:50:10.596 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:10 smithi070 ceph-mon[107385]: pgmap v2575: 99 pgs: 1 active+recovery_wait+degraded, 1 active+remapped+backfill_wait, 3 active+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.6 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 13 KiB/s rd, 30 MiB/s wr, 493 op/s; 47263/275970 objects degraded (17.126%); 14868/275970 objects misplaced (5.388%); 3.5 MiB/s, 54 objects/s recovering 2024-09-21T10:50:10.596 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:10 smithi070 ceph-mon[112139]: pgmap v2575: 99 pgs: 1 active+recovery_wait+degraded, 1 active+remapped+backfill_wait, 3 active+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.6 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 13 KiB/s rd, 30 MiB/s wr, 493 op/s; 47263/275970 objects degraded (17.126%); 14868/275970 objects misplaced (5.388%); 3.5 MiB/s, 54 objects/s recovering 2024-09-21T10:50:10.647 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:10.669 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:10.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:10.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:10.771 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:10.795 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:10.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:10.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:10.809 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 70 16 95054 95038 84.8442 39.3125 0.035021 0.0117793 2024-09-21T10:50:10.896 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:10.938 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:10.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:10.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:11.040 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:11.085 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:11.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:11.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:11.186 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:11.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:11.226 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:11.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:11.327 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:11.378 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:11.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:11.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:11.479 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:11.502 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:11.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:11.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:11.512 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:50:11.512 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T10:50:11.512 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T10:50:11.512 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T10:50:11.512 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.512 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:69140 53.11 53:8c59eef6:::benchmark_data_smithi070_228967_object69139:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1323)", 2024-09-21T10:50:11.512 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:40:59.634375+0000", 2024-09-21T10:50:11.512 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 551.87236806199996, 2024-09-21T10:50:11.512 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0624963780000001, 2024-09-21T10:50:11.512 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:50:11.512 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:50:11.512 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:50:11.513 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:50:11.513 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:50:11.513 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:50:11.513 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 69140 2024-09-21T10:50:11.513 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.513 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:50:11.513 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.513 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:50:11.513 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634375+0000", 2024-09-21T10:50:11.513 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.513 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.513 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.513 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:50:11.514 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634375+0000", 2024-09-21T10:50:11.514 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.514 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.514 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.514 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:50:11.514 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634376+0000", 2024-09-21T10:50:11.514 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3099999999999998e-07 2024-09-21T10:50:11.514 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.514 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.514 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:50:11.514 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634402+0000", 2024-09-21T10:50:11.514 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6662e-05 2024-09-21T10:50:11.514 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.514 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.515 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:50:11.515 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634409+0000", 2024-09-21T10:50:11.515 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5869999999999998e-06 2024-09-21T10:50:11.515 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.515 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.515 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:50:11.515 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634414+0000", 2024-09-21T10:50:11.515 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0259999999999998e-06 2024-09-21T10:50:11.515 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.515 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.515 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.515 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.769018+0000", 2024-09-21T10:50:11.515 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.134604431 2024-09-21T10:50:11.515 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.516 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.516 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.516 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.769026+0000", 2024-09-21T10:50:11.516 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1140000000000006e-06 2024-09-21T10:50:11.516 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.516 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.516 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.516 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.696793+0000", 2024-09-21T10:50:11.516 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.92776693899999996 2024-09-21T10:50:11.516 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.516 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.516 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:50:11.516 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.696872+0000", 2024-09-21T10:50:11.516 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8188000000000004e-05 2024-09-21T10:50:11.517 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.517 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:50:11.517 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.517 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.517 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.517 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:69149 53.11 53:8f768ae6:::benchmark_data_smithi070_228967_object69148:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1323)", 2024-09-21T10:50:11.517 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:40:59.634660+0000", 2024-09-21T10:50:11.517 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 551.87208329800001, 2024-09-21T10:50:11.517 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0721572539999999, 2024-09-21T10:50:11.517 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:50:11.517 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:50:11.517 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:11.517 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:50:11.518 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:50:11.518 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:50:11.518 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 69149 2024-09-21T10:50:11.518 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.518 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:50:11.518 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.518 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:50:11.518 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634660+0000", 2024-09-21T10:50:11.518 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.518 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.518 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.518 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:50:11.518 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634660+0000", 2024-09-21T10:50:11.518 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.518 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.519 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.519 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:50:11.519 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634660+0000", 2024-09-21T10:50:11.519 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5100000000000001e-07 2024-09-21T10:50:11.519 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.519 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.519 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:50:11.519 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634684+0000", 2024-09-21T10:50:11.519 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3703999999999999e-05 2024-09-21T10:50:11.519 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.519 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.519 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:50:11.519 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634690+0000", 2024-09-21T10:50:11.519 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5300000000000004e-06 2024-09-21T10:50:11.519 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.520 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.520 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:50:11.520 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.634694+0000", 2024-09-21T10:50:11.520 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.4290000000000003e-06 2024-09-21T10:50:11.520 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.520 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.520 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.520 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.769046+0000", 2024-09-21T10:50:11.520 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13435238199999999 2024-09-21T10:50:11.520 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.520 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.520 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.520 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:40:59.769052+0000", 2024-09-21T10:50:11.520 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3770000000000004e-06 2024-09-21T10:50:11.521 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.521 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.521 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.521 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.696899+0000", 2024-09-21T10:50:11.521 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.92784694000000001 2024-09-21T10:50:11.521 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.521 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.521 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:50:11.521 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.697002+0000", 2024-09-21T10:50:11.521 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010339400000000001 2024-09-21T10:50:11.521 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.521 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.521 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:50:11.521 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.697127+0000", 2024-09-21T10:50:11.522 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012478899999999999 2024-09-21T10:50:11.522 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.522 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.522 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:50:11.522 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.700680+0000", 2024-09-21T10:50:11.522 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035536389999999999 2024-09-21T10:50:11.522 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.522 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.522 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.522 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.704795+0000", 2024-09-21T10:50:11.522 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0041147240000000002 2024-09-21T10:50:11.522 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.522 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.522 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.522 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.706726+0000", 2024-09-21T10:50:11.523 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0019308319999999999 2024-09-21T10:50:11.523 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.523 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.523 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:50:11.523 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.706781+0000", 2024-09-21T10:50:11.523 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4951000000000003e-05 2024-09-21T10:50:11.523 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.523 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.523 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:50:11.523 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:00.706817+0000", 2024-09-21T10:50:11.523 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6211999999999998e-05 2024-09-21T10:50:11.523 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.523 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:50:11.523 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.523 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.524 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.524 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:80837 53.14 53:2bec9862:::benchmark_data_smithi070_228967_object80836:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1333)", 2024-09-21T10:50:11.524 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:13.124032+0000", 2024-09-21T10:50:11.524 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 538.38271146399995, 2024-09-21T10:50:11.524 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.068596307, 2024-09-21T10:50:11.524 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:50:11.524 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:50:11.524 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:11.524 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:50:11.524 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:50:11.524 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:50:11.524 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 80837 2024-09-21T10:50:11.524 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.524 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:50:11.524 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.525 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:50:11.525 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124032+0000", 2024-09-21T10:50:11.525 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.525 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.525 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.525 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:50:11.526 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124032+0000", 2024-09-21T10:50:11.526 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.526 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.526 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.526 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:50:11.526 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124033+0000", 2024-09-21T10:50:11.526 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.714e-06 2024-09-21T10:50:11.526 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.526 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.526 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:50:11.526 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124055+0000", 2024-09-21T10:50:11.526 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1759e-05 2024-09-21T10:50:11.526 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.526 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.526 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:50:11.527 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124068+0000", 2024-09-21T10:50:11.527 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2952999999999999e-05 2024-09-21T10:50:11.527 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.527 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.527 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:50:11.527 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124076+0000", 2024-09-21T10:50:11.527 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0020000000000006e-06 2024-09-21T10:50:11.527 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.527 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.527 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.527 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124343+0000", 2024-09-21T10:50:11.527 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00026635100000000001 2024-09-21T10:50:11.527 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.528 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.528 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op must wait for map", 2024-09-21T10:50:11.528 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124348+0000", 2024-09-21T10:50:11.528 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7459999999999997e-06 2024-09-21T10:50:11.528 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.528 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.528 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.528 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.128445+0000", 2024-09-21T10:50:11.528 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0040971749999999998 2024-09-21T10:50:11.528 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.528 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.528 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.528 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.128454+0000", 2024-09-21T10:50:11.528 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.4829999999999999e-06 2024-09-21T10:50:11.528 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.528 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.529 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.529 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.187927+0000", 2024-09-21T10:50:11.529 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.059473195 2024-09-21T10:50:11.529 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.529 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.529 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:50:11.529 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.188061+0000", 2024-09-21T10:50:11.529 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013360299999999999 2024-09-21T10:50:11.529 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.529 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.529 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:50:11.529 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.188198+0000", 2024-09-21T10:50:11.529 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013701699999999999 2024-09-21T10:50:11.529 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.529 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.530 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.530 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.191426+0000", 2024-09-21T10:50:11.530 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0032279169999999999 2024-09-21T10:50:11.530 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.530 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.530 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.530 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.191477+0000", 2024-09-21T10:50:11.530 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1193000000000003e-05 2024-09-21T10:50:11.530 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.530 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.530 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:50:11.530 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.191496+0000", 2024-09-21T10:50:11.530 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8936999999999998e-05 2024-09-21T10:50:11.531 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.531 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.531 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:50:11.531 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.191541+0000", 2024-09-21T10:50:11.531 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5461999999999999e-05 2024-09-21T10:50:11.531 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.531 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.531 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:50:11.531 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.192628+0000", 2024-09-21T10:50:11.531 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0010868 2024-09-21T10:50:11.531 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.531 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:50:11.531 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.531 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.532 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.532 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:80865 53.14 53:2dd10c48:::benchmark_data_smithi070_228967_object80864:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1333)", 2024-09-21T10:50:11.532 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:13.124099+0000", 2024-09-21T10:50:11.532 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 538.38264457900004, 2024-09-21T10:50:11.532 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0715279170000001, 2024-09-21T10:50:11.532 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:50:11.532 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:50:11.532 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:11.532 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:50:11.532 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:50:11.532 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:50:11.532 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 80865 2024-09-21T10:50:11.532 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.532 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:50:11.532 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.533 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:50:11.533 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124099+0000", 2024-09-21T10:50:11.533 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.533 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.533 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.533 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:50:11.533 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124099+0000", 2024-09-21T10:50:11.533 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.533 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.533 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.533 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:50:11.533 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124099+0000", 2024-09-21T10:50:11.533 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0999999999999998e-07 2024-09-21T10:50:11.533 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.533 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.534 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:50:11.534 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124120+0000", 2024-09-21T10:50:11.534 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0951e-05 2024-09-21T10:50:11.534 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.534 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.534 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:50:11.534 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124128+0000", 2024-09-21T10:50:11.534 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1759999999999993e-06 2024-09-21T10:50:11.534 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.534 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.534 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:50:11.534 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124140+0000", 2024-09-21T10:50:11.534 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.171e-05 2024-09-21T10:50:11.534 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.535 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.535 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.535 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124429+0000", 2024-09-21T10:50:11.535 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00028871600000000001 2024-09-21T10:50:11.535 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.535 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.535 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting_for_map not empty", 2024-09-21T10:50:11.535 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124434+0000", 2024-09-21T10:50:11.535 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7639999999999996e-06 2024-09-21T10:50:11.535 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.535 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.535 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.535 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.129316+0000", 2024-09-21T10:50:11.535 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0048827779999999999 2024-09-21T10:50:11.535 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.536 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.536 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.536 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.129324+0000", 2024-09-21T10:50:11.536 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0409999999999998e-06 2024-09-21T10:50:11.536 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.536 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.536 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.536 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.190062+0000", 2024-09-21T10:50:11.536 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.060737735 2024-09-21T10:50:11.536 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.536 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.536 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:50:11.536 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.190239+0000", 2024-09-21T10:50:11.536 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017677599999999999 2024-09-21T10:50:11.536 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.536 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.537 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:50:11.537 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.190433+0000", 2024-09-21T10:50:11.537 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000193652 2024-09-21T10:50:11.537 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.537 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.537 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.537 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.194192+0000", 2024-09-21T10:50:11.537 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0037595380000000002 2024-09-21T10:50:11.537 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.537 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.537 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:50:11.537 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.194211+0000", 2024-09-21T10:50:11.537 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8576e-05 2024-09-21T10:50:11.537 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.537 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.537 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.538 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.194263+0000", 2024-09-21T10:50:11.538 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2049999999999998e-05 2024-09-21T10:50:11.538 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.538 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.538 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:50:11.538 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.194325+0000", 2024-09-21T10:50:11.538 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1956000000000001e-05 2024-09-21T10:50:11.538 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.538 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.538 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:50:11.538 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.195627+0000", 2024-09-21T10:50:11.538 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001301888 2024-09-21T10:50:11.538 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.538 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:50:11.538 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.539 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.539 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.539 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:80885 53.14 53:2a222b02:::benchmark_data_smithi070_228967_object80884:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1333)", 2024-09-21T10:50:11.539 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:13.124165+0000", 2024-09-21T10:50:11.539 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 538.382578061, 2024-09-21T10:50:11.539 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0753624589999999, 2024-09-21T10:50:11.539 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:50:11.539 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:50:11.539 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:11.539 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:50:11.539 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:50:11.539 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:50:11.539 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 80885 2024-09-21T10:50:11.539 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.540 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:50:11.540 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.540 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:50:11.540 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124165+0000", 2024-09-21T10:50:11.540 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.540 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.540 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.540 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:50:11.540 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124165+0000", 2024-09-21T10:50:11.540 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.540 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.540 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.540 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:50:11.541 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124166+0000", 2024-09-21T10:50:11.541 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2699999999999999e-07 2024-09-21T10:50:11.541 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.541 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.541 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:50:11.541 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124188+0000", 2024-09-21T10:50:11.541 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2413999999999999e-05 2024-09-21T10:50:11.541 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.541 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.541 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:50:11.541 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124197+0000", 2024-09-21T10:50:11.541 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6079999999999996e-06 2024-09-21T10:50:11.541 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.541 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.542 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:50:11.542 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124204+0000", 2024-09-21T10:50:11.542 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7569999999999998e-06 2024-09-21T10:50:11.542 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.542 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.542 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.542 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124454+0000", 2024-09-21T10:50:11.542 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00025049600000000001 2024-09-21T10:50:11.542 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.542 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.542 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting_for_map not empty", 2024-09-21T10:50:11.542 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124458+0000", 2024-09-21T10:50:11.542 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0579999999999999e-06 2024-09-21T10:50:11.543 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.543 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.543 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.543 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.129347+0000", 2024-09-21T10:50:11.543 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0048889659999999998 2024-09-21T10:50:11.543 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.543 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.543 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.543 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.129353+0000", 2024-09-21T10:50:11.543 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.224e-06 2024-09-21T10:50:11.543 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.543 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.543 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.543 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.192683+0000", 2024-09-21T10:50:11.543 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.063329293 2024-09-21T10:50:11.543 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.544 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.544 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:50:11.544 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.192826+0000", 2024-09-21T10:50:11.544 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014342099999999999 2024-09-21T10:50:11.544 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.544 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.544 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:50:11.544 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.193015+0000", 2024-09-21T10:50:11.544 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000188395 2024-09-21T10:50:11.544 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.544 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.544 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.544 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.195678+0000", 2024-09-21T10:50:11.544 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0026633329999999999 2024-09-21T10:50:11.545 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.545 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.545 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:50:11.545 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.197964+0000", 2024-09-21T10:50:11.545 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002285718 2024-09-21T10:50:11.545 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.545 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.545 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.545 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.198005+0000", 2024-09-21T10:50:11.545 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.1423e-05 2024-09-21T10:50:11.545 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.545 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.545 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:50:11.545 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.198041+0000", 2024-09-21T10:50:11.546 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5608000000000002e-05 2024-09-21T10:50:11.546 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.546 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.546 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:50:11.546 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.199528+0000", 2024-09-21T10:50:11.546 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0014870180000000001 2024-09-21T10:50:11.546 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.546 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:50:11.546 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.546 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.546 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.546 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:80983 53.14 53:2e23b57c:::benchmark_data_smithi070_228967_object80982:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1333)", 2024-09-21T10:50:11.546 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:13.124214+0000", 2024-09-21T10:50:11.547 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 538.38252884500002, 2024-09-21T10:50:11.547 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0782936949999999, 2024-09-21T10:50:11.547 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:50:11.547 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:50:11.547 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:11.547 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:50:11.547 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:50:11.547 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:50:11.547 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 80983 2024-09-21T10:50:11.547 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.547 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:50:11.547 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.547 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:50:11.547 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124214+0000", 2024-09-21T10:50:11.547 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.548 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.548 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.548 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:50:11.548 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124214+0000", 2024-09-21T10:50:11.548 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.548 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.548 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.548 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:50:11.548 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124215+0000", 2024-09-21T10:50:11.548 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8000000000000001e-07 2024-09-21T10:50:11.548 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.548 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.548 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:50:11.548 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124233+0000", 2024-09-21T10:50:11.549 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8722000000000001e-05 2024-09-21T10:50:11.549 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.549 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.549 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:50:11.549 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124240+0000", 2024-09-21T10:50:11.549 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.117e-06 2024-09-21T10:50:11.549 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.549 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.549 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:50:11.549 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124244+0000", 2024-09-21T10:50:11.549 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8210000000000001e-06 2024-09-21T10:50:11.549 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.549 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.549 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.550 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124478+0000", 2024-09-21T10:50:11.550 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00023360399999999999 2024-09-21T10:50:11.550 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.550 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.550 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting_for_map not empty", 2024-09-21T10:50:11.550 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124482+0000", 2024-09-21T10:50:11.550 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8689999999999997e-06 2024-09-21T10:50:11.550 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.550 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.550 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.550 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.129384+0000", 2024-09-21T10:50:11.550 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0049018300000000002 2024-09-21T10:50:11.550 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.550 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.551 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.551 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.129390+0000", 2024-09-21T10:50:11.551 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0229999999999998e-06 2024-09-21T10:50:11.551 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.551 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.551 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.551 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.195779+0000", 2024-09-21T10:50:11.551 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0663895219999999 2024-09-21T10:50:11.551 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.551 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.551 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:50:11.551 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.195939+0000", 2024-09-21T10:50:11.551 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015971400000000001 2024-09-21T10:50:11.551 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:50:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.196114+0000", 2024-09-21T10:50:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017518899999999999 2024-09-21T10:50:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:50:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.201033+0000", 2024-09-21T10:50:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0049189500000000001 2024-09-21T10:50:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.201095+0000", 2024-09-21T10:50:11.552 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1567000000000002e-05 2024-09-21T10:50:11.553 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.553 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.553 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.553 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.201216+0000", 2024-09-21T10:50:11.553 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000121137 2024-09-21T10:50:11.553 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.553 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.553 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:50:11.553 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.201281+0000", 2024-09-21T10:50:11.553 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5548000000000003e-05 2024-09-21T10:50:11.553 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.553 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.553 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:50:11.553 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.202508+0000", 2024-09-21T10:50:11.554 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0012267020000000001 2024-09-21T10:50:11.554 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.554 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:50:11.554 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.554 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.554 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.554 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:81026 53.14 53:2c15cc7d:::benchmark_data_smithi070_228967_object81025:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1333)", 2024-09-21T10:50:11.554 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:13.124251+0000", 2024-09-21T10:50:11.554 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 538.38249207299998, 2024-09-21T10:50:11.554 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0790572220000001, 2024-09-21T10:50:11.554 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:50:11.554 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:50:11.554 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:11.554 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:50:11.555 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:50:11.555 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:50:11.555 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 81026 2024-09-21T10:50:11.555 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.555 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:50:11.555 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.555 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:50:11.555 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124251+0000", 2024-09-21T10:50:11.555 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.555 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.555 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.555 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:50:11.555 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124251+0000", 2024-09-21T10:50:11.555 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.555 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.556 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.556 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:50:11.556 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124252+0000", 2024-09-21T10:50:11.556 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.77e-07 2024-09-21T10:50:11.556 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.556 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.556 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:50:11.556 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124270+0000", 2024-09-21T10:50:11.556 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8782000000000001e-05 2024-09-21T10:50:11.556 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.556 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.556 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:50:11.556 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124276+0000", 2024-09-21T10:50:11.556 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9039999999999997e-06 2024-09-21T10:50:11.556 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.557 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.557 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:50:11.557 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124281+0000", 2024-09-21T10:50:11.557 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7469999999999996e-06 2024-09-21T10:50:11.557 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.557 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.557 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.557 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124501+0000", 2024-09-21T10:50:11.557 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00022036199999999999 2024-09-21T10:50:11.557 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.557 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.557 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting_for_map not empty", 2024-09-21T10:50:11.557 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.124505+0000", 2024-09-21T10:50:11.557 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9740000000000004e-06 2024-09-21T10:50:11.557 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.558 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.558 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.558 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.129411+0000", 2024-09-21T10:50:11.558 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0049058360000000002 2024-09-21T10:50:11.558 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.558 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.558 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.558 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.129418+0000", 2024-09-21T10:50:11.558 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5629999999999997e-06 2024-09-21T10:50:11.558 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.558 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.558 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.558 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.197085+0000", 2024-09-21T10:50:11.558 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0676677859999999 2024-09-21T10:50:11.559 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.559 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.559 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:50:11.559 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.197179+0000", 2024-09-21T10:50:11.559 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3066999999999995e-05 2024-09-21T10:50:11.559 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.559 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.559 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:50:11.559 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.197277+0000", 2024-09-21T10:50:11.559 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.8886000000000002e-05 2024-09-21T10:50:11.559 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.559 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.559 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.560 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.201160+0000", 2024-09-21T10:50:11.560 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0038826680000000001 2024-09-21T10:50:11.560 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.560 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.560 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:50:11.560 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.201169+0000", 2024-09-21T10:50:11.560 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.524e-06 2024-09-21T10:50:11.560 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.560 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.560 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.560 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.202563+0000", 2024-09-21T10:50:11.560 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001394235 2024-09-21T10:50:11.560 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.560 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.560 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:50:11.561 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.202604+0000", 2024-09-21T10:50:11.561 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0809999999999997e-05 2024-09-21T10:50:11.561 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.561 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.561 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:50:11.561 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.203308+0000", 2024-09-21T10:50:11.561 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00070470099999999996 2024-09-21T10:50:11.561 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.561 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:50:11.561 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.561 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.561 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.561 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:81178 53.14 53:2ca30304:::benchmark_data_smithi070_228967_object81177:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1333)", 2024-09-21T10:50:11.561 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:13.136116+0000", 2024-09-21T10:50:11.561 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 538.37062697800002, 2024-09-21T10:50:11.562 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.067947644, 2024-09-21T10:50:11.562 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:50:11.562 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:50:11.562 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:11.562 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:50:11.562 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:50:11.562 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:50:11.562 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 81178 2024-09-21T10:50:11.562 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.562 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:50:11.562 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.562 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:50:11.562 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.136116+0000", 2024-09-21T10:50:11.562 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.562 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.563 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.563 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:50:11.563 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.136116+0000", 2024-09-21T10:50:11.563 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.563 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.563 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.563 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:50:11.563 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.136118+0000", 2024-09-21T10:50:11.563 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5370000000000001e-06 2024-09-21T10:50:11.563 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.563 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.563 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:50:11.563 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.136141+0000", 2024-09-21T10:50:11.563 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2903000000000001e-05 2024-09-21T10:50:11.564 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.564 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.564 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:50:11.564 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.136152+0000", 2024-09-21T10:50:11.564 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1379000000000001e-05 2024-09-21T10:50:11.564 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.564 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.564 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:50:11.564 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.136159+0000", 2024-09-21T10:50:11.564 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7619999999999998e-06 2024-09-21T10:50:11.564 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.564 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.565 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.565 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.136246+0000", 2024-09-21T10:50:11.565 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7253e-05 2024-09-21T10:50:11.565 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.565 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.565 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.565 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.136255+0000", 2024-09-21T10:50:11.565 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9700000000000005e-06 2024-09-21T10:50:11.565 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.565 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.566 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.566 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.199587+0000", 2024-09-21T10:50:11.566 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.063332368 2024-09-21T10:50:11.566 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.566 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.566 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:50:11.566 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.199741+0000", 2024-09-21T10:50:11.566 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015315899999999999 2024-09-21T10:50:11.566 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.566 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.566 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:50:11.566 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.199922+0000", 2024-09-21T10:50:11.566 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018125800000000001 2024-09-21T10:50:11.566 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.566 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.567 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:50:11.567 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.203323+0000", 2024-09-21T10:50:11.567 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.003401247 2024-09-21T10:50:11.567 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.567 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.567 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.567 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.203361+0000", 2024-09-21T10:50:11.567 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8241999999999997e-05 2024-09-21T10:50:11.567 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.567 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.567 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.567 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.203402+0000", 2024-09-21T10:50:11.567 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0306999999999999e-05 2024-09-21T10:50:11.567 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.567 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.568 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:50:11.568 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.203434+0000", 2024-09-21T10:50:11.568 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2839000000000002e-05 2024-09-21T10:50:11.568 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.568 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.568 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:50:11.568 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.204064+0000", 2024-09-21T10:50:11.568 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062942 2024-09-21T10:50:11.568 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.568 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:50:11.568 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.568 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.568 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.569 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.45020.0:81199 53.14 53:2a691e06:::benchmark_data_smithi070_228967_object81198:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1333)", 2024-09-21T10:50:11.569 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:41:13.150263+0000", 2024-09-21T10:50:11.569 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 538.35648029000004, 2024-09-21T10:50:11.569 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0602721310000001, 2024-09-21T10:50:11.569 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:50:11.569 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:50:11.569 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:11.569 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:50:11.569 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.45020", 2024-09-21T10:50:11.569 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:50:11.569 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 81199 2024-09-21T10:50:11.569 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.569 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:50:11.569 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.569 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:50:11.570 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.150263+0000", 2024-09-21T10:50:11.570 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.570 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.570 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.570 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:50:11.570 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.150263+0000", 2024-09-21T10:50:11.570 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.570 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.570 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.570 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:50:11.570 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.150264+0000", 2024-09-21T10:50:11.570 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3689999999999999e-06 2024-09-21T10:50:11.570 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.570 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.570 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:50:11.571 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.150284+0000", 2024-09-21T10:50:11.571 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0151000000000001e-05 2024-09-21T10:50:11.571 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.571 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.571 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:50:11.571 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.150296+0000", 2024-09-21T10:50:11.571 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1728e-05 2024-09-21T10:50:11.571 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.571 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.571 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:50:11.571 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.150301+0000", 2024-09-21T10:50:11.571 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1830000000000002e-06 2024-09-21T10:50:11.571 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.571 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.571 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.571 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.150338+0000", 2024-09-21T10:50:11.572 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6650000000000003e-05 2024-09-21T10:50:11.572 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.572 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.572 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.572 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:13.150344+0000", 2024-09-21T10:50:11.572 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5389999999999998e-06 2024-09-21T10:50:11.572 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.572 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.572 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.572 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.204122+0000", 2024-09-21T10:50:11.572 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0537786490000001 2024-09-21T10:50:11.572 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.572 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.572 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:50:11.572 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.204277+0000", 2024-09-21T10:50:11.573 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015522100000000001 2024-09-21T10:50:11.573 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.573 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.573 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:50:11.573 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.204483+0000", 2024-09-21T10:50:11.573 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00020547799999999999 2024-09-21T10:50:11.573 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.573 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.573 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.573 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.209091+0000", 2024-09-21T10:50:11.573 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0046077109999999996 2024-09-21T10:50:11.573 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.573 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.573 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:50:11.573 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.209106+0000", 2024-09-21T10:50:11.574 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5285e-05 2024-09-21T10:50:11.574 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.574 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.574 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.574 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.209143+0000", 2024-09-21T10:50:11.574 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6690999999999998e-05 2024-09-21T10:50:11.574 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.574 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.574 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:50:11.574 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.209203+0000", 2024-09-21T10:50:11.574 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0228e-05 2024-09-21T10:50:11.574 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.574 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.574 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:50:11.574 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:41:14.210535+0000", 2024-09-21T10:50:11.574 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0013322480000000001 2024-09-21T10:50:11.575 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.575 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:50:11.575 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.575 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.575 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.575 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:76000 55.3 55:cd844426:::benchmark_data_smithi070_294394_object75999:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1426)", 2024-09-21T10:50:11.575 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:49:43.855902+0000", 2024-09-21T10:50:11.575 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 27.650841223, 2024-09-21T10:50:11.575 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2807186910000001, 2024-09-21T10:50:11.575 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:50:11.575 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:50:11.575 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:50:11.575 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:50:11.575 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:50:11.575 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:50:11.575 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 76000 2024-09-21T10:50:11.576 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.576 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:50:11.576 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.576 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:50:11.576 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855902+0000", 2024-09-21T10:50:11.576 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.576 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.576 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.576 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:50:11.576 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855902+0000", 2024-09-21T10:50:11.576 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.576 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.576 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.576 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:50:11.576 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855904+0000", 2024-09-21T10:50:11.576 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1840000000000002e-06 2024-09-21T10:50:11.577 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.577 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.577 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:50:11.577 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855927+0000", 2024-09-21T10:50:11.577 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2909999999999999e-05 2024-09-21T10:50:11.577 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.577 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.577 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:50:11.577 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855957+0000", 2024-09-21T10:50:11.577 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9437e-05 2024-09-21T10:50:11.577 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.577 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.577 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:50:11.577 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855965+0000", 2024-09-21T10:50:11.578 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.4349999999999997e-06 2024-09-21T10:50:11.578 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.578 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.578 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.578 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.856021+0000", 2024-09-21T10:50:11.578 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6232e-05 2024-09-21T10:50:11.578 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.578 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.578 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:50:11.578 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.856214+0000", 2024-09-21T10:50:11.578 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00019308699999999999 2024-09-21T10:50:11.578 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.578 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.578 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,7", 2024-09-21T10:50:11.578 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.856386+0000", 2024-09-21T10:50:11.579 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000171621 2024-09-21T10:50:11.579 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.579 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.579 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.579 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.862577+0000", 2024-09-21T10:50:11.579 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0061906019999999999 2024-09-21T10:50:11.579 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.579 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.579 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.579 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.862589+0000", 2024-09-21T10:50:11.579 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2367e-05 2024-09-21T10:50:11.579 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.579 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.579 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.579 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.853010+0000", 2024-09-21T10:50:11.580 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99042137600000002 2024-09-21T10:50:11.580 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.580 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.580 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.580 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.853028+0000", 2024-09-21T10:50:11.580 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7776999999999999e-05 2024-09-21T10:50:11.580 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.580 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.580 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.580 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.116392+0000", 2024-09-21T10:50:11.580 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.263364207 2024-09-21T10:50:11.580 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.580 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.580 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:50:11.580 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.116926+0000", 2024-09-21T10:50:11.580 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053343100000000001 2024-09-21T10:50:11.581 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.581 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.581 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.581 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.136415+0000", 2024-09-21T10:50:11.581 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.019489623000000001 2024-09-21T10:50:11.581 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.581 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.581 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:50:11.581 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.136621+0000", 2024-09-21T10:50:11.581 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00020540200000000001 2024-09-21T10:50:11.581 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.581 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:50:11.581 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.581 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.581 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.582 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:76010 55.3 55:ca4bf1d6:::benchmark_data_smithi070_294394_object76009:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1427)", 2024-09-21T10:50:11.582 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:49:43.866567+0000", 2024-09-21T10:50:11.582 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 27.640176319999998, 2024-09-21T10:50:11.582 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2823839870000002, 2024-09-21T10:50:11.582 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:50:11.582 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:50:11.582 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:11.582 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:50:11.582 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:50:11.582 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:50:11.582 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 76010 2024-09-21T10:50:11.582 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.582 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:50:11.582 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.583 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:50:11.583 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.866567+0000", 2024-09-21T10:50:11.583 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.583 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.583 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.583 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:50:11.583 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.866567+0000", 2024-09-21T10:50:11.583 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.583 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.583 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.583 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:50:11.583 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.866568+0000", 2024-09-21T10:50:11.583 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5629999999999999e-06 2024-09-21T10:50:11.583 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.583 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.584 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:50:11.584 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.866589+0000", 2024-09-21T10:50:11.584 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0886999999999998e-05 2024-09-21T10:50:11.584 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.584 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.584 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:50:11.584 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.866600+0000", 2024-09-21T10:50:11.584 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0621e-05 2024-09-21T10:50:11.584 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.584 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.584 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:50:11.584 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.866608+0000", 2024-09-21T10:50:11.584 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9279999999999993e-06 2024-09-21T10:50:11.584 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.584 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.585 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.585 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.866999+0000", 2024-09-21T10:50:11.585 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000390734 2024-09-21T10:50:11.585 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.585 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.585 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.585 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.867008+0000", 2024-09-21T10:50:11.585 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2779999999999999e-06 2024-09-21T10:50:11.585 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.585 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.585 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.585 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.853163+0000", 2024-09-21T10:50:11.585 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.98615497699999999 2024-09-21T10:50:11.585 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.585 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.585 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.586 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.853170+0000", 2024-09-21T10:50:11.586 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5530000000000004e-06 2024-09-21T10:50:11.586 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.586 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.586 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.586 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.120718+0000", 2024-09-21T10:50:11.586 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2675472079999999 2024-09-21T10:50:11.586 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.586 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.586 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:50:11.586 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.120831+0000", 2024-09-21T10:50:11.586 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000113841 2024-09-21T10:50:11.586 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.586 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.587 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:50:11.587 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.120947+0000", 2024-09-21T10:50:11.587 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000115327 2024-09-21T10:50:11.587 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.587 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.587 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.587 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.136561+0000", 2024-09-21T10:50:11.587 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015613986 2024-09-21T10:50:11.587 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.587 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.587 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.587 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.136585+0000", 2024-09-21T10:50:11.587 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.429e-05 2024-09-21T10:50:11.587 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.588 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.588 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:50:11.588 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.144528+0000", 2024-09-21T10:50:11.588 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0079428559999999999 2024-09-21T10:50:11.588 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.588 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.588 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:50:11.588 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.144577+0000", 2024-09-21T10:50:11.588 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8749999999999999e-05 2024-09-21T10:50:11.588 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.588 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.588 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:50:11.588 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.148951+0000", 2024-09-21T10:50:11.588 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0043741880000000002 2024-09-21T10:50:11.588 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.588 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:50:11.589 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.589 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.589 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.589 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:76030 55.3 55:c3f9098d:::benchmark_data_smithi070_294394_object76029:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1427)", 2024-09-21T10:50:11.589 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:49:43.887718+0000", 2024-09-21T10:50:11.589 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 27.619025143999998, 2024-09-21T10:50:11.589 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2693207429999998, 2024-09-21T10:50:11.589 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:50:11.589 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:50:11.589 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:11.589 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:50:11.589 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:50:11.589 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:50:11.589 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 76030 2024-09-21T10:50:11.589 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.589 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:50:11.590 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.590 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:50:11.590 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887718+0000", 2024-09-21T10:50:11.590 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.590 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.590 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.590 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:50:11.590 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887718+0000", 2024-09-21T10:50:11.590 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.590 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.590 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.590 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:50:11.590 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887720+0000", 2024-09-21T10:50:11.590 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5880000000000001e-06 2024-09-21T10:50:11.590 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.591 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.591 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:50:11.591 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887741+0000", 2024-09-21T10:50:11.591 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0874e-05 2024-09-21T10:50:11.591 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.591 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.591 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:50:11.591 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887754+0000", 2024-09-21T10:50:11.591 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3876999999999999e-05 2024-09-21T10:50:11.591 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.591 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.591 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:50:11.591 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887761+0000", 2024-09-21T10:50:11.591 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8290000000000002e-06 2024-09-21T10:50:11.591 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.592 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.592 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.592 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887819+0000", 2024-09-21T10:50:11.592 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7469999999999997e-05 2024-09-21T10:50:11.592 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.592 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.592 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.592 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887828+0000", 2024-09-21T10:50:11.592 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7559999999999994e-06 2024-09-21T10:50:11.592 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.592 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.592 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.592 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.853198+0000", 2024-09-21T10:50:11.592 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.96536982999999998 2024-09-21T10:50:11.592 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.593 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.593 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.593 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.853205+0000", 2024-09-21T10:50:11.593 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1559999999999998e-06 2024-09-21T10:50:11.593 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.593 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.593 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.593 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.136704+0000", 2024-09-21T10:50:11.593 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.283498252 2024-09-21T10:50:11.593 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.593 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.593 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:50:11.593 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.136820+0000", 2024-09-21T10:50:11.593 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011636700000000001 2024-09-21T10:50:11.593 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.593 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.594 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:50:11.594 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.136933+0000", 2024-09-21T10:50:11.594 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000112872 2024-09-21T10:50:11.594 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.594 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.594 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.594 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.149001+0000", 2024-09-21T10:50:11.594 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012068146 2024-09-21T10:50:11.594 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.594 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.594 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:50:11.594 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.153155+0000", 2024-09-21T10:50:11.594 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0041541190000000004 2024-09-21T10:50:11.595 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.595 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.595 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.595 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.153219+0000", 2024-09-21T10:50:11.595 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3492999999999997e-05 2024-09-21T10:50:11.595 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.595 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.595 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:50:11.595 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.153287+0000", 2024-09-21T10:50:11.595 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7943999999999999e-05 2024-09-21T10:50:11.595 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.595 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.595 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:50:11.595 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.157039+0000", 2024-09-21T10:50:11.596 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00375217 2024-09-21T10:50:11.596 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.596 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:50:11.596 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.596 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.596 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.596 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:91257 55.2 55:43dc9448:::benchmark_data_smithi070_294394_object91256:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1432)", 2024-09-21T10:50:11.596 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:50:03.040864+0000", 2024-09-21T10:50:11.596 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 8.4658787479999997, 2024-09-21T10:50:11.596 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7075791259999999, 2024-09-21T10:50:11.596 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:50:11.596 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:50:11.596 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:50:11.596 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:50:11.596 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:50:11.597 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:50:11.597 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 91257 2024-09-21T10:50:11.597 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.597 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:50:11.597 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.597 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:50:11.597 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.040864+0000", 2024-09-21T10:50:11.597 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.597 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.597 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.597 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:50:11.597 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.040864+0000", 2024-09-21T10:50:11.597 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.598 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.598 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.598 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:50:11.598 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.040866+0000", 2024-09-21T10:50:11.598 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5859999999999999e-06 2024-09-21T10:50:11.598 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.598 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.598 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:50:11.598 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.040893+0000", 2024-09-21T10:50:11.598 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7036999999999999e-05 2024-09-21T10:50:11.598 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.598 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.598 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:50:11.598 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.040904+0000", 2024-09-21T10:50:11.599 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0513000000000001e-05 2024-09-21T10:50:11.599 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.599 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.599 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:50:11.599 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.040910+0000", 2024-09-21T10:50:11.599 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4280000000000001e-06 2024-09-21T10:50:11.599 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.599 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.599 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.599 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.041137+0000", 2024-09-21T10:50:11.599 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000227297 2024-09-21T10:50:11.599 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.599 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.599 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:50:11.600 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.045948+0000", 2024-09-21T10:50:11.600 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0048108129999999997 2024-09-21T10:50:11.600 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.600 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.600 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:50:11.600 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.046101+0000", 2024-09-21T10:50:11.600 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015316599999999999 2024-09-21T10:50:11.600 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.600 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.600 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:50:11.600 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.050938+0000", 2024-09-21T10:50:11.600 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0048367210000000004 2024-09-21T10:50:11.600 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.600 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.601 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.601 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.065829+0000", 2024-09-21T10:50:11.601 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.014891051000000001 2024-09-21T10:50:11.601 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.601 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.601 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.601 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.065844+0000", 2024-09-21T10:50:11.601 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4579999999999999e-05 2024-09-21T10:50:11.601 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.601 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.601 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.601 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.257262+0000", 2024-09-21T10:50:11.601 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.191418597 2024-09-21T10:50:11.602 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.602 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.602 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.602 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.257272+0000", 2024-09-21T10:50:11.602 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5789999999999993e-06 2024-09-21T10:50:11.602 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.602 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.602 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.602 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.718417+0000", 2024-09-21T10:50:11.602 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.461144759 2024-09-21T10:50:11.602 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.602 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.602 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:50:11.602 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.723923+0000", 2024-09-21T10:50:11.603 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0055066669999999998 2024-09-21T10:50:11.603 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.603 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.603 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.603 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.748299+0000", 2024-09-21T10:50:11.603 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.024375267999999999 2024-09-21T10:50:11.603 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.603 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:11.604 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.604 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:50:11.604 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.748444+0000", 2024-09-21T10:50:11.605 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014506400000000001 2024-09-21T10:50:11.605 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.605 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:50:11.605 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.605 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.605 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.605 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:91271 55.2 55:4c7a1cbb:::benchmark_data_smithi070_294394_object91270:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1433)", 2024-09-21T10:50:11.605 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:50:03.087644+0000", 2024-09-21T10:50:11.605 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 8.4190994190000001, 2024-09-21T10:50:11.605 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.665248112, 2024-09-21T10:50:11.605 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:50:11.605 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:50:11.605 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:11.606 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:50:11.606 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:50:11.606 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:50:11.606 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 91271 2024-09-21T10:50:11.606 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.606 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:50:11.606 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.606 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:50:11.606 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.087644+0000", 2024-09-21T10:50:11.606 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.606 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.606 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.606 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:50:11.607 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.087644+0000", 2024-09-21T10:50:11.607 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.607 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.607 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.607 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:50:11.607 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.087646+0000", 2024-09-21T10:50:11.607 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.204e-06 2024-09-21T10:50:11.607 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.607 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.607 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:50:11.607 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.087676+0000", 2024-09-21T10:50:11.607 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0414000000000001e-05 2024-09-21T10:50:11.607 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.608 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.608 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:50:11.608 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.087699+0000", 2024-09-21T10:50:11.608 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2390000000000001e-05 2024-09-21T10:50:11.608 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.608 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.608 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:50:11.608 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.087711+0000", 2024-09-21T10:50:11.608 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1701e-05 2024-09-21T10:50:11.608 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.608 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.608 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.608 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.087965+0000", 2024-09-21T10:50:11.608 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00025414200000000001 2024-09-21T10:50:11.609 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.609 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.609 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.609 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.087981+0000", 2024-09-21T10:50:11.609 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6010000000000001e-05 2024-09-21T10:50:11.609 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.609 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.609 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.609 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.257295+0000", 2024-09-21T10:50:11.609 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.169314252 2024-09-21T10:50:11.609 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.609 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.609 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.609 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.257301+0000", 2024-09-21T10:50:11.610 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.207e-06 2024-09-21T10:50:11.610 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.610 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.610 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.610 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.728462+0000", 2024-09-21T10:50:11.610 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.471161285 2024-09-21T10:50:11.610 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.610 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.610 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:50:11.610 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.728637+0000", 2024-09-21T10:50:11.610 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000174742 2024-09-21T10:50:11.610 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.610 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.610 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:50:11.611 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.729182+0000", 2024-09-21T10:50:11.611 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00054522299999999995 2024-09-21T10:50:11.611 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.611 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.611 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:50:11.611 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.748442+0000", 2024-09-21T10:50:11.611 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.019259486999999999 2024-09-21T10:50:11.611 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.611 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.611 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.611 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.748493+0000", 2024-09-21T10:50:11.611 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1008999999999999e-05 2024-09-21T10:50:11.611 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.611 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.612 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.612 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.748540+0000", 2024-09-21T10:50:11.612 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7389000000000003e-05 2024-09-21T10:50:11.612 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.612 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.612 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:50:11.612 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.748644+0000", 2024-09-21T10:50:11.612 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000103699 2024-09-21T10:50:11.612 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.612 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.612 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:50:11.612 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.752892+0000", 2024-09-21T10:50:11.612 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0042479579999999996 2024-09-21T10:50:11.612 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.612 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:50:11.613 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.613 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.613 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.613 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:91282 55.2 55:40800590:::benchmark_data_smithi070_294394_object91281:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1433)", 2024-09-21T10:50:11.613 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:50:03.100410+0000", 2024-09-21T10:50:11.613 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 8.4063327930000007, 2024-09-21T10:50:11.613 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7104403500000001, 2024-09-21T10:50:11.613 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:50:11.613 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:50:11.613 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:11.613 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:50:11.613 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:50:11.613 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:50:11.613 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 91282 2024-09-21T10:50:11.613 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.614 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:50:11.614 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.614 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:50:11.614 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.100410+0000", 2024-09-21T10:50:11.614 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.614 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.614 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.614 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:50:11.614 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.100410+0000", 2024-09-21T10:50:11.614 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.614 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.614 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.614 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:50:11.615 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.100412+0000", 2024-09-21T10:50:11.615 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9549999999999999e-06 2024-09-21T10:50:11.615 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.615 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.615 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:50:11.615 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.100438+0000", 2024-09-21T10:50:11.615 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5133000000000001e-05 2024-09-21T10:50:11.615 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.615 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.615 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:50:11.615 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.100461+0000", 2024-09-21T10:50:11.615 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3764000000000002e-05 2024-09-21T10:50:11.615 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.615 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.615 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:50:11.616 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.100471+0000", 2024-09-21T10:50:11.616 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0200999999999999e-05 2024-09-21T10:50:11.616 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.616 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.616 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.616 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.100550+0000", 2024-09-21T10:50:11.616 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8767e-05 2024-09-21T10:50:11.616 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.616 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.616 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.616 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.100561+0000", 2024-09-21T10:50:11.616 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0545e-05 2024-09-21T10:50:11.616 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.616 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.617 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.617 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.257320+0000", 2024-09-21T10:50:11.617 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.15675908199999999 2024-09-21T10:50:11.617 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.617 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.617 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.617 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.257326+0000", 2024-09-21T10:50:11.617 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0209999999999996e-06 2024-09-21T10:50:11.617 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.617 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.617 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.617 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.752938+0000", 2024-09-21T10:50:11.617 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4956121549999999 2024-09-21T10:50:11.617 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.617 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.618 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:50:11.618 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.753091+0000", 2024-09-21T10:50:11.618 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015340300000000001 2024-09-21T10:50:11.618 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.618 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.618 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:50:11.618 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.753264+0000", 2024-09-21T10:50:11.618 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000172924 2024-09-21T10:50:11.618 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.618 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.618 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:50:11.618 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.766215+0000", 2024-09-21T10:50:11.618 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012951028 2024-09-21T10:50:11.619 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.619 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.619 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.619 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.781445+0000", 2024-09-21T10:50:11.619 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015229351 2024-09-21T10:50:11.619 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.619 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.619 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.619 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.803339+0000", 2024-09-21T10:50:11.619 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.021893968999999999 2024-09-21T10:50:11.619 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.619 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.619 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:50:11.620 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.803430+0000", 2024-09-21T10:50:11.620 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1318999999999998e-05 2024-09-21T10:50:11.620 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.620 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.620 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:50:11.620 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.810851+0000", 2024-09-21T10:50:11.620 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0074207329999999997 2024-09-21T10:50:11.620 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.620 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:50:11.620 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.620 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.620 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.620 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:91286 55.2 55:4def9a29:::benchmark_data_smithi070_294394_object91285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1433)", 2024-09-21T10:50:11.620 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:50:03.104675+0000", 2024-09-21T10:50:11.620 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 8.4020684330000002, 2024-09-21T10:50:11.621 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7145347900000001, 2024-09-21T10:50:11.621 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:50:11.621 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:50:11.621 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:11.621 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:50:11.621 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:50:11.621 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:50:11.621 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 91286 2024-09-21T10:50:11.621 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.621 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:50:11.621 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.621 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:50:11.621 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.104675+0000", 2024-09-21T10:50:11.621 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.622 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.622 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.622 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:50:11.622 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.104675+0000", 2024-09-21T10:50:11.622 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.622 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.622 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.622 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:50:11.622 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.104676+0000", 2024-09-21T10:50:11.622 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.576e-06 2024-09-21T10:50:11.622 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.622 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.622 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:50:11.623 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.104694+0000", 2024-09-21T10:50:11.623 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7638999999999998e-05 2024-09-21T10:50:11.623 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.623 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.623 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:50:11.623 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.104704+0000", 2024-09-21T10:50:11.623 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0358e-05 2024-09-21T10:50:11.623 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.623 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.623 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:50:11.623 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.104709+0000", 2024-09-21T10:50:11.623 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1159999999999998e-06 2024-09-21T10:50:11.623 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.623 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.624 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.624 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.104755+0000", 2024-09-21T10:50:11.624 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5584000000000003e-05 2024-09-21T10:50:11.624 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.624 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.624 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.624 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.104761+0000", 2024-09-21T10:50:11.624 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6590000000000004e-06 2024-09-21T10:50:11.624 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.624 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.624 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.624 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.257345+0000", 2024-09-21T10:50:11.624 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.15258385399999999 2024-09-21T10:50:11.624 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.624 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.625 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.625 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.257350+0000", 2024-09-21T10:50:11.625 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8109999999999999e-06 2024-09-21T10:50:11.625 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.625 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.625 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.625 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.755959+0000", 2024-09-21T10:50:11.625 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4986089069999999 2024-09-21T10:50:11.625 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.625 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.625 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:50:11.625 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.756079+0000", 2024-09-21T10:50:11.625 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000119536 2024-09-21T10:50:11.625 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.626 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.626 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:50:11.626 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.756274+0000", 2024-09-21T10:50:11.626 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00019505400000000001 2024-09-21T10:50:11.626 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.626 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.626 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:50:11.626 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.766227+0000", 2024-09-21T10:50:11.626 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0099535519999999992 2024-09-21T10:50:11.626 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.626 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.626 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.626 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.785592+0000", 2024-09-21T10:50:11.627 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.019365054 2024-09-21T10:50:11.627 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.627 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.627 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.627 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.810900+0000", 2024-09-21T10:50:11.627 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.025307168000000001 2024-09-21T10:50:11.627 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.627 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.627 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:50:11.627 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.811297+0000", 2024-09-21T10:50:11.627 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00039743999999999999 2024-09-21T10:50:11.627 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.627 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.627 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:50:11.627 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.819210+0000", 2024-09-21T10:50:11.628 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0079124820000000002 2024-09-21T10:50:11.628 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.628 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:50:11.628 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.628 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.628 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.628 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:91298 55.2 55:4da05bce:::benchmark_data_smithi070_294394_object91297:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1433)", 2024-09-21T10:50:11.628 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:50:03.123418+0000", 2024-09-21T10:50:11.628 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 8.3833250330000002, 2024-09-21T10:50:11.628 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7002024090000001, 2024-09-21T10:50:11.628 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:50:11.628 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:50:11.628 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:11.628 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:50:11.628 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:50:11.629 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:50:11.629 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 91298 2024-09-21T10:50:11.629 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.629 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:50:11.629 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.629 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:50:11.629 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123418+0000", 2024-09-21T10:50:11.629 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.629 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.629 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.629 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:50:11.629 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123418+0000", 2024-09-21T10:50:11.629 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.629 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.629 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.630 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:50:11.630 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123420+0000", 2024-09-21T10:50:11.630 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.556e-06 2024-09-21T10:50:11.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.630 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.630 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:50:11.630 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123438+0000", 2024-09-21T10:50:11.630 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8634000000000002e-05 2024-09-21T10:50:11.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.630 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.630 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:50:11.630 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123449+0000", 2024-09-21T10:50:11.630 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0781999999999999e-05 2024-09-21T10:50:11.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.631 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.631 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:50:11.631 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123454+0000", 2024-09-21T10:50:11.631 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3109999999999996e-06 2024-09-21T10:50:11.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.631 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.631 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.631 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123509+0000", 2024-09-21T10:50:11.631 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4549000000000002e-05 2024-09-21T10:50:11.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.631 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.631 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.631 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123520+0000", 2024-09-21T10:50:11.631 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1070000000000001e-05 2024-09-21T10:50:11.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.632 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.632 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.257549+0000", 2024-09-21T10:50:11.632 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.134029119 2024-09-21T10:50:11.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.632 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.632 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.257556+0000", 2024-09-21T10:50:11.632 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8040000000000004e-06 2024-09-21T10:50:11.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.632 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.632 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.759290+0000", 2024-09-21T10:50:11.632 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5017340020000001 2024-09-21T10:50:11.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.633 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.633 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:50:11.633 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.759477+0000", 2024-09-21T10:50:11.633 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000187097 2024-09-21T10:50:11.633 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.633 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.633 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:50:11.633 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.759660+0000", 2024-09-21T10:50:11.633 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018245499999999999 2024-09-21T10:50:11.633 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.633 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.633 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:50:11.633 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.775049+0000", 2024-09-21T10:50:11.633 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015389534999999999 2024-09-21T10:50:11.633 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.633 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.634 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.634 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.785632+0000", 2024-09-21T10:50:11.634 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.010582505000000001 2024-09-21T10:50:11.634 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.634 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.634 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.634 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.819272+0000", 2024-09-21T10:50:11.634 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.033639948000000003 2024-09-21T10:50:11.634 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.634 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.634 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:50:11.634 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.819359+0000", 2024-09-21T10:50:11.634 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7269999999999996e-05 2024-09-21T10:50:11.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.635 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.635 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:50:11.635 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.823621+0000", 2024-09-21T10:50:11.635 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0042617719999999996 2024-09-21T10:50:11.635 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.635 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:50:11.635 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.635 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.635 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:91299 55.3 55:c9f1c749:::benchmark_data_smithi070_294394_object91298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1433)", 2024-09-21T10:50:11.637 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:50:03.123772+0000", 2024-09-21T10:50:11.637 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 8.3829714289999995, 2024-09-21T10:50:11.637 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6459212750000001, 2024-09-21T10:50:11.637 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:50:11.637 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:50:11.637 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:11.637 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:50:11.638 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:50:11.638 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:50:11.638 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 91299 2024-09-21T10:50:11.638 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.638 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:50:11.638 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.638 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:50:11.638 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123772+0000", 2024-09-21T10:50:11.638 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.638 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.638 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.638 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:50:11.638 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123772+0000", 2024-09-21T10:50:11.638 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.639 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.639 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:50:11.639 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123773+0000", 2024-09-21T10:50:11.639 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7599999999999996e-07 2024-09-21T10:50:11.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.639 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.639 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:50:11.639 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123792+0000", 2024-09-21T10:50:11.639 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9372999999999999e-05 2024-09-21T10:50:11.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.639 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.639 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:50:11.639 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123801+0000", 2024-09-21T10:50:11.640 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2660000000000007e-06 2024-09-21T10:50:11.640 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.640 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.640 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:50:11.640 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123805+0000", 2024-09-21T10:50:11.640 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2930000000000002e-06 2024-09-21T10:50:11.640 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.640 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.640 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.640 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123852+0000", 2024-09-21T10:50:11.640 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6292999999999999e-05 2024-09-21T10:50:11.640 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.640 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.640 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.641 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123860+0000", 2024-09-21T10:50:11.641 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5790000000000004e-06 2024-09-21T10:50:11.641 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.641 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.641 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.641 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.254002+0000", 2024-09-21T10:50:11.641 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13014203199999999 2024-09-21T10:50:11.641 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.641 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.641 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.641 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.254014+0000", 2024-09-21T10:50:11.641 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1302e-05 2024-09-21T10:50:11.641 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.641 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.642 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.642 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.598674+0000", 2024-09-21T10:50:11.642 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3446601089999999 2024-09-21T10:50:11.642 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.642 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.642 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:50:11.642 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.598843+0000", 2024-09-21T10:50:11.642 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016916300000000001 2024-09-21T10:50:11.642 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.642 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.642 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:50:11.642 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.599473+0000", 2024-09-21T10:50:11.642 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062992099999999997 2024-09-21T10:50:11.642 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.642 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.643 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.643 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.761869+0000", 2024-09-21T10:50:11.643 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.16239562599999999 2024-09-21T10:50:11.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.643 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.643 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.643 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.761909+0000", 2024-09-21T10:50:11.643 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0163999999999997e-05 2024-09-21T10:50:11.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.643 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.643 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:50:11.643 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.764195+0000", 2024-09-21T10:50:11.643 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002286293 2024-09-21T10:50:11.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.644 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.644 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:50:11.644 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.764537+0000", 2024-09-21T10:50:11.644 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000341892 2024-09-21T10:50:11.644 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.644 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.644 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:50:11.644 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.769693+0000", 2024-09-21T10:50:11.644 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0051561929999999999 2024-09-21T10:50:11.644 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.644 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:50:11.644 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.644 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.644 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.644 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:91300 55.3 55:c662b782:::benchmark_data_smithi070_294394_object91299:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1433)", 2024-09-21T10:50:11.644 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:50:03.123953+0000", 2024-09-21T10:50:11.645 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 8.3827903450000001, 2024-09-21T10:50:11.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.67947549, 2024-09-21T10:50:11.645 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:50:11.645 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:50:11.645 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:11.645 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:50:11.645 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:50:11.645 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:50:11.645 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 91300 2024-09-21T10:50:11.645 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.645 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:50:11.645 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.645 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:50:11.645 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123953+0000", 2024-09-21T10:50:11.646 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.646 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.646 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.646 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:50:11.646 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123953+0000", 2024-09-21T10:50:11.646 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.646 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.646 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.646 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:50:11.646 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123954+0000", 2024-09-21T10:50:11.646 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8400000000000003e-07 2024-09-21T10:50:11.646 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.646 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.647 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:50:11.647 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123973+0000", 2024-09-21T10:50:11.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9658000000000001e-05 2024-09-21T10:50:11.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.647 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:50:11.647 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123981+0000", 2024-09-21T10:50:11.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8469999999999996e-06 2024-09-21T10:50:11.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.647 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:50:11.647 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123985+0000", 2024-09-21T10:50:11.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9899999999999999e-06 2024-09-21T10:50:11.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.648 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.648 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.124026+0000", 2024-09-21T10:50:11.648 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.1102000000000001e-05 2024-09-21T10:50:11.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.648 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.648 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.124035+0000", 2024-09-21T10:50:11.648 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1579999999999994e-06 2024-09-21T10:50:11.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.648 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.648 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.254041+0000", 2024-09-21T10:50:11.648 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13000513499999999 2024-09-21T10:50:11.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.254059+0000", 2024-09-21T10:50:11.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8929000000000001e-05 2024-09-21T10:50:11.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.769844+0000", 2024-09-21T10:50:11.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.51578426 2024-09-21T10:50:11.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:50:11.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.769972+0000", 2024-09-21T10:50:11.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000128233 2024-09-21T10:50:11.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.650 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:50:11.650 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.770116+0000", 2024-09-21T10:50:11.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014439499999999999 2024-09-21T10:50:11.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.650 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:50:11.650 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.783082+0000", 2024-09-21T10:50:11.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012965829 2024-09-21T10:50:11.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.650 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.797652+0000", 2024-09-21T10:50:11.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.014569446 2024-09-21T10:50:11.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.651 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.798576+0000", 2024-09-21T10:50:11.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00092482000000000003 2024-09-21T10:50:11.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.651 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:50:11.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.798648+0000", 2024-09-21T10:50:11.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1123000000000003e-05 2024-09-21T10:50:11.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.651 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:50:11.652 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.803428+0000", 2024-09-21T10:50:11.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0047807809999999996 2024-09-21T10:50:11.652 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.652 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:50:11.652 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.652 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.652 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:91308 55.2 55:432b9bad:::benchmark_data_smithi070_294394_object91307:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1433)", 2024-09-21T10:50:11.652 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:50:03.151172+0000", 2024-09-21T10:50:11.652 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 8.3555710419999993, 2024-09-21T10:50:11.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6770448820000001, 2024-09-21T10:50:11.652 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:50:11.652 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:50:11.652 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:11.652 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:50:11.653 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:50:11.653 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:50:11.653 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 91308 2024-09-21T10:50:11.653 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.653 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:50:11.653 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.653 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:50:11.653 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.151172+0000", 2024-09-21T10:50:11.653 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.653 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.653 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.653 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:50:11.653 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.151172+0000", 2024-09-21T10:50:11.653 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:50:11.654 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.654 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.654 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:50:11.654 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.151174+0000", 2024-09-21T10:50:11.654 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.46e-06 2024-09-21T10:50:11.654 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.654 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.654 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:50:11.654 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.151192+0000", 2024-09-21T10:50:11.654 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8454e-05 2024-09-21T10:50:11.654 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.654 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.654 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:50:11.654 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.151203+0000", 2024-09-21T10:50:11.655 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1241e-05 2024-09-21T10:50:11.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.655 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:50:11.655 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.151209+0000", 2024-09-21T10:50:11.655 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3480000000000003e-06 2024-09-21T10:50:11.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.655 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.655 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.151270+0000", 2024-09-21T10:50:11.655 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1665e-05 2024-09-21T10:50:11.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.655 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.656 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.151282+0000", 2024-09-21T10:50:11.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1701e-05 2024-09-21T10:50:11.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.656 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.656 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.257576+0000", 2024-09-21T10:50:11.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10629356 2024-09-21T10:50:11.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.656 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:50:11.656 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.257582+0000", 2024-09-21T10:50:11.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9349999999999999e-06 2024-09-21T10:50:11.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.656 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:50:11.657 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.763154+0000", 2024-09-21T10:50:11.657 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.505572508 2024-09-21T10:50:11.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.657 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.657 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:50:11.657 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.763319+0000", 2024-09-21T10:50:11.657 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016480800000000001 2024-09-21T10:50:11.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.657 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.657 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:50:11.657 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.763503+0000", 2024-09-21T10:50:11.657 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018448299999999999 2024-09-21T10:50:11.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.657 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.657 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:50:11.658 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.775057+0000", 2024-09-21T10:50:11.658 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.011553178000000001 2024-09-21T10:50:11.658 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.658 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.658 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.658 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.785667+0000", 2024-09-21T10:50:11.658 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.010610385 2024-09-21T10:50:11.658 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.658 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.658 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:11.658 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.823676+0000", 2024-09-21T10:50:11.658 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.038008844999999999 2024-09-21T10:50:11.658 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.658 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.658 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:50:11.659 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.823737+0000", 2024-09-21T10:50:11.659 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1110000000000003e-05 2024-09-21T10:50:11.659 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:50:11.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:50:11.659 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:50:11.659 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.828217+0000", 2024-09-21T10:50:11.659 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0044802009999999996 2024-09-21T10:50:11.659 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.659 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:50:11.659 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.659 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:50:11.659 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:50:11.659 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:50:11.695 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:11.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:11.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:11.796 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:11.810 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 71 16 95635 95619 84.1586 36.3125 0.0214131 0.0118762 2024-09-21T10:50:11.834 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:11.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:11.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:11.936 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:11.973 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:11.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:11.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:12.075 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:12.112 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:12.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:12.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:12.214 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:12.237 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:12.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:12.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:12.321 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T10:50:12.339 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:12.360 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:12.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:12.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:12.386 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-21T10:50:12.386 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 110 1.0 2024-09-21T10:50:12.461 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:12.468 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:50:12.487 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:12.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:12.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:12.589 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:12 smithi160 ceph-mon[101879]: pgmap v2576: 99 pgs: 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 5.7 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 11 KiB/s rd, 42 MiB/s wr, 677 op/s; 47263/281109 objects degraded (16.813%); 14868/281109 objects misplaced (5.289%); 4.7 MiB/s, 74 objects/s recovering 2024-09-21T10:50:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:12 smithi160 ceph-mon[101879]: 1.5 deep-scrub starts 2024-09-21T10:50:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:12 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2113367557' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:50:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:12 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:50:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:12 smithi070 ceph-mon[107385]: pgmap v2576: 99 pgs: 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 5.7 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 11 KiB/s rd, 42 MiB/s wr, 677 op/s; 47263/281109 objects degraded (16.813%); 14868/281109 objects misplaced (5.289%); 4.7 MiB/s, 74 objects/s recovering 2024-09-21T10:50:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:12 smithi070 ceph-mon[107385]: 1.5 deep-scrub starts 2024-09-21T10:50:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:12 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2113367557' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:50:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:12 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:50:12.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:12 smithi070 ceph-mon[112139]: pgmap v2576: 99 pgs: 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 5.7 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 11 KiB/s rd, 42 MiB/s wr, 677 op/s; 47263/281109 objects degraded (16.813%); 14868/281109 objects misplaced (5.289%); 4.7 MiB/s, 74 objects/s recovering 2024-09-21T10:50:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:12 smithi070 ceph-mon[112139]: 1.5 deep-scrub starts 2024-09-21T10:50:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:12 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2113367557' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:50:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:12 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:50:12.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:12.651 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:12.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:12.752 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:12.794 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:12.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:12.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:12.811 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 72 16 96218 96202 83.4947 36.4375 0.00552973 0.0119686 2024-09-21T10:50:12.896 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:12.924 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:12.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:12.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:13.026 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:13.061 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:13.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:13.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:13.163 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:13.188 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:13.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:13.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:13.290 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:13.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:13.340 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:13.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:13.441 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:13.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:13.471 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:13.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:13.572 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:13.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:13 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:50:13.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:13 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:50:13.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:13 smithi160 ceph-mon[101879]: osdmap e1438: 8 total, 8 up, 6 in 2024-09-21T10:50:13.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:13 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 47263/281109 objects degraded (16.813%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:50:13.614 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:13.615 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:50:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:50:13] "GET /metrics HTTP/1.1" 200 36679 "" "Prometheus/2.51.0" 2024-09-21T10:50:13.615 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:13 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:50:13.615 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:13 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:50:13.615 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:13 smithi070 ceph-mon[107385]: osdmap e1438: 8 total, 8 up, 6 in 2024-09-21T10:50:13.615 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:13 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 47263/281109 objects degraded (16.813%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:50:13.615 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:13 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:50:13.616 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:13 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:50:13.616 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:13 smithi070 ceph-mon[112139]: osdmap e1438: 8 total, 8 up, 6 in 2024-09-21T10:50:13.616 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:13 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 47263/281109 objects degraded (16.813%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:50:13.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:13.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:13.717 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:13.747 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:13.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:13.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:13.812 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 73 16 96858 96842 82.8989 40 0.0206979 0.0120567 2024-09-21T10:50:13.848 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:13.874 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:13.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:13.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:13.976 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:14.047 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:50:14.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:14.049 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:14.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:14.057 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:50:14.096 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:50:14.150 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:14.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:14.267 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:14.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:14.368 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:14.394 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:14.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:14.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:14.496 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:14.523 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:14.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:14.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:14.588 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:14 smithi070 ceph-mon[107385]: pgmap v2578: 99 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 43 MiB/s wr, 681 op/s; 47263/285228 objects degraded (16.570%); 14868/285228 objects misplaced (5.213%); 3.3 MiB/s, 52 objects/s recovering 2024-09-21T10:50:14.588 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:14 smithi070 ceph-mon[112139]: pgmap v2578: 99 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 43 MiB/s wr, 681 op/s; 47263/285228 objects degraded (16.570%); 14868/285228 objects misplaced (5.213%); 3.3 MiB/s, 52 objects/s recovering 2024-09-21T10:50:14.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:14 smithi160 ceph-mon[101879]: pgmap v2578: 99 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 43 MiB/s wr, 681 op/s; 47263/285228 objects degraded (16.570%); 14868/285228 objects misplaced (5.213%); 3.3 MiB/s, 52 objects/s recovering 2024-09-21T10:50:14.625 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:14.651 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:14.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:14.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:14.753 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:14.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:14.797 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:14.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:14.812 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 74 16 97441 97425 82.271 36.4375 0.00611513 0.0121474 2024-09-21T10:50:14.898 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:14.933 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:14.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:14.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:15.034 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:15.080 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:15.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:15.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:15.182 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:15.209 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:15.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:15.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:15.311 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:15.338 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:15.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:15.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:15.440 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:15.464 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:15.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:15.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:15.566 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:15.609 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:15.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:15.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:15.657 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:50:15.710 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:15.738 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:15.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:15.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:15.812 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 75 16 98051 98035 81.6824 38.125 0.0253698 0.0122361 2024-09-21T10:50:15.839 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:15.887 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:15.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:15.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:15.988 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:16.035 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:16.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:16.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:16.136 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:16.161 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:16.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:16.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:16.262 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:16.286 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:16.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:16.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:16.388 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:16.445 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:16.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:16.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:16.547 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:16.575 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:16.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:16.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:16 smithi160 ceph-mon[101879]: pgmap v2579: 99 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 30 MiB/s wr, 487 op/s; 47263/285228 objects degraded (16.570%); 14868/285228 objects misplaced (5.213%); 2.1 MiB/s, 34 objects/s recovering 2024-09-21T10:50:16.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:16 smithi070 ceph-mon[107385]: pgmap v2579: 99 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 30 MiB/s wr, 487 op/s; 47263/285228 objects degraded (16.570%); 14868/285228 objects misplaced (5.213%); 2.1 MiB/s, 34 objects/s recovering 2024-09-21T10:50:16.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:16 smithi070 ceph-mon[112139]: pgmap v2579: 99 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 30 MiB/s wr, 487 op/s; 47263/285228 objects degraded (16.570%); 14868/285228 objects misplaced (5.213%); 2.1 MiB/s, 34 objects/s recovering 2024-09-21T10:50:16.677 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:16.701 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:16.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:16.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:16.802 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:16.812 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 76 16 98665 98649 81.1122 38.375 0.0140217 0.0123205 2024-09-21T10:50:16.848 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:16.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:16.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:16.950 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:16.998 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:16.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:16.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:17.099 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:17.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:17.145 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:17.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:17.246 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:17.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:17.282 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:17.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:17.383 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:17.416 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:17.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:17.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:17.517 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:17.545 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:17.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:17.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:17.647 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:17.695 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:17.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:17.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:17.796 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:17.815 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 77 16 99322 99306 80.5896 41.0625 0.0178419 0.0124022 2024-09-21T10:50:17.839 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:17.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:17.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:17.940 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:17.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:17.993 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:17.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:18.094 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:18.120 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:18.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:18.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:18.222 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:18.257 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:18.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:18.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:18.360 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:18.384 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:18.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:18.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:18.485 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:18.542 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:18.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:18.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:18 smithi160 ceph-mon[101879]: pgmap v2580: 99 pgs: 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 38 MiB/s wr, 608 op/s; 47263/294243 objects degraded (16.063%); 14862/294243 objects misplaced (5.051%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T10:50:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:18 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 47263/294243 objects degraded (16.063%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:50:18.619 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:18 smithi070 ceph-mon[107385]: pgmap v2580: 99 pgs: 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 38 MiB/s wr, 608 op/s; 47263/294243 objects degraded (16.063%); 14862/294243 objects misplaced (5.051%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T10:50:18.619 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:18 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 47263/294243 objects degraded (16.063%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:50:18.619 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:18 smithi070 ceph-mon[112139]: pgmap v2580: 99 pgs: 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 38 MiB/s wr, 608 op/s; 47263/294243 objects degraded (16.063%); 14862/294243 objects misplaced (5.051%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T10:50:18.619 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:18 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 47263/294243 objects degraded (16.063%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:50:18.644 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:18.668 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:18.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:18.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:18.770 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:18.791 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:18.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:18.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:18.815 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 78 16 99928 99912 80.042 37.875 0.0651064 0.0124862 2024-09-21T10:50:18.893 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:18.920 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:18.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:18.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:19.022 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:19.062 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:19.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:19.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:19.164 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:19.208 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:19.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:19.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:19.310 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:19.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:19.344 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:19.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:19.445 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:19.495 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:19.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:19.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:19.515 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:50:19.515 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:50:19.596 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:19.621 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:19.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:19.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:19.722 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:19.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:19.803 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:19.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:19.816 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 79 16 100525 100509 79.4999 37.3125 0.039654 0.0125712 2024-09-21T10:50:19.904 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:19.945 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:19.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:19.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:20.047 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:20.110 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:20.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:20.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:20.212 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:20.215 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:50:20.215 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:50:20.235 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:20.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:20.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:20.264 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:50:20.264 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:50:20.300 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:50:20.300 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:50:20.329 INFO:teuthology.orchestra.run.smithi070.stdout:moved 24 / 297 (8.08081%) 2024-09-21T10:50:20.329 INFO:teuthology.orchestra.run.smithi070.stdout:avg 49.5 2024-09-21T10:50:20.329 INFO:teuthology.orchestra.run.smithi070.stdout:stddev 10.2754 -> 13.6229 (expected baseline 6.42262) 2024-09-21T10:50:20.329 INFO:teuthology.orchestra.run.smithi070.stdout:min osd.7 with 28 -> 23 pgs (0.565657 -> 0.464646 * mean) 2024-09-21T10:50:20.329 INFO:teuthology.orchestra.run.smithi070.stdout:max osd.1 with 60 -> 62 pgs (1.21212 -> 1.25253 * mean) 2024-09-21T10:50:20.329 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:50:20.330 INFO:teuthology.orchestra.run.smithi070.stdout:oload 110 2024-09-21T10:50:20.330 INFO:teuthology.orchestra.run.smithi070.stdout:max_change 1 2024-09-21T10:50:20.330 INFO:teuthology.orchestra.run.smithi070.stdout:max_change_osds 4 2024-09-21T10:50:20.330 INFO:teuthology.orchestra.run.smithi070.stdout:average_utilization 0.0367 2024-09-21T10:50:20.330 INFO:teuthology.orchestra.run.smithi070.stdout:overload_utilization 0.0403 2024-09-21T10:50:20.330 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 weight 0.6567 -> 0.4399 2024-09-21T10:50:20.330 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 weight 0.8224 -> 0.6819 2024-09-21T10:50:20.330 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 weight 0.8866 -> 0.7775 2024-09-21T10:50:20.336 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:20.338 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:50:20.339 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:50:20.373 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:20.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:20.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:20.389 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:50:20.389 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:50:20.441 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:50:20.441 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:50:20.472 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:50:20.472 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:50:20.475 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:20.506 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:50:20.507 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:50:20.508 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:20.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:20.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:20.523 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:20 smithi070 ceph-mon[107385]: pgmap v2581: 99 pgs: 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 38 MiB/s wr, 608 op/s; 47263/294243 objects degraded (16.063%); 14862/294243 objects misplaced (5.051%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T10:50:20.523 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'3':'50957','6':'44689','7':'28832'}"} : dispatch 2024-09-21T10:50:20.523 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:20 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1376119137' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:50:20.523 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:20 smithi070 ceph-mon[112139]: pgmap v2581: 99 pgs: 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 38 MiB/s wr, 608 op/s; 47263/294243 objects degraded (16.063%); 14862/294243 objects misplaced (5.051%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T10:50:20.523 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'3':'50957','6':'44689','7':'28832'}"} : dispatch 2024-09-21T10:50:20.523 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:20 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1376119137' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:50:20.610 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:20.632 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:20.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:20.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:20.734 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:20.757 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:20.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:20.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:20.816 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:2024-09-21T10:50:20.815480+0000 min lat: 0.00113626 max lat: 2.33372 avg lat: 0.0126333 2024-09-21T10:50:20.816 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:50:20.817 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 80 16 100873 100857 78.7781 21.75 0.00426764 0.0126333 2024-09-21T10:50:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:20 smithi160 ceph-mon[101879]: pgmap v2581: 99 pgs: 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 38 MiB/s wr, 608 op/s; 47263/294243 objects degraded (16.063%); 14862/294243 objects misplaced (5.051%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T10:50:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'3':'50957','6':'44689','7':'28832'}"} : dispatch 2024-09-21T10:50:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:20 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1376119137' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:50:20.859 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:20.880 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:20.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:20.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:20.982 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:21.003 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:21.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:21.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:21.105 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:21.126 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:21.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:21.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:21.228 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:21.249 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:21.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:21.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:21.352 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:21.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:21.387 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:21.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:21.488 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:21.511 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:21.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:21.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:21.613 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:21.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:21 smithi070 ceph-mon[107385]: from='client.55174 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:50:21.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'50957','6':'44689','7':'28832'}"}]': finished 2024-09-21T10:50:21.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:21 smithi070 ceph-mon[107385]: osdmap e1439: 8 total, 8 up, 6 in 2024-09-21T10:50:21.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:50:21.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:50:21.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:21 smithi070 ceph-mon[112139]: from='client.55174 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:50:21.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'50957','6':'44689','7':'28832'}"}]': finished 2024-09-21T10:50:21.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:21 smithi070 ceph-mon[112139]: osdmap e1439: 8 total, 8 up, 6 in 2024-09-21T10:50:21.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:50:21.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:50:21.635 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:21.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:21.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:21.737 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:21.758 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:21.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:21.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:21.816 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 81 16 100873 100857 77.8056 0 - 0.0126333 2024-09-21T10:50:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:21 smithi160 ceph-mon[101879]: from='client.55174 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:50:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'50957','6':'44689','7':'28832'}"}]': finished 2024-09-21T10:50:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:21 smithi160 ceph-mon[101879]: osdmap e1439: 8 total, 8 up, 6 in 2024-09-21T10:50:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:50:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:50:21.860 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:21.881 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:21.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:21.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:21.983 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:22.005 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:22.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:22.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:22.106 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:22.130 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:22.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:22.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:22.232 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:22.253 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:22.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:22.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:22.355 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:22.376 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:22.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:22.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:22.477 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:22.503 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:22.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:22.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:22.605 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:22 smithi070 ceph-mon[107385]: pgmap v2583: 99 pgs: 4 peering, 1 active+recovery_wait+degraded, 1 active+remapped+backfill_wait, 9 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 82 active+clean; 6.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 34 MiB/s wr, 546 op/s; 47263/299238 objects degraded (15.794%); 14831/299238 objects misplaced (4.956%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:50:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:22 smithi070 ceph-mon[107385]: osdmap e1440: 8 total, 8 up, 6 in 2024-09-21T10:50:22.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:22 smithi070 ceph-mon[112139]: pgmap v2583: 99 pgs: 4 peering, 1 active+recovery_wait+degraded, 1 active+remapped+backfill_wait, 9 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 82 active+clean; 6.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 34 MiB/s wr, 546 op/s; 47263/299238 objects degraded (15.794%); 14831/299238 objects misplaced (4.956%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:50:22.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:22 smithi070 ceph-mon[112139]: osdmap e1440: 8 total, 8 up, 6 in 2024-09-21T10:50:22.632 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:22.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:22.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:22.734 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:22.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:22 smithi160 ceph-mon[101879]: pgmap v2583: 99 pgs: 4 peering, 1 active+recovery_wait+degraded, 1 active+remapped+backfill_wait, 9 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 82 active+clean; 6.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 34 MiB/s wr, 546 op/s; 47263/299238 objects degraded (15.794%); 14831/299238 objects misplaced (4.956%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:50:22.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:22 smithi160 ceph-mon[101879]: osdmap e1440: 8 total, 8 up, 6 in 2024-09-21T10:50:22.765 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:22.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:22.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:22.816 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 82 16 100950 100934 76.9156 2.40625 0.0823238 0.0129938 2024-09-21T10:50:22.867 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:22.894 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:22.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:22.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:22.996 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:23.026 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:23.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:23.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:23.128 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:23.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:23.157 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:23.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:23.258 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:23.297 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:23.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:23.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:23.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:50:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:50:23] "GET /metrics HTTP/1.1" 200 36666 "" "Prometheus/2.51.0" 2024-09-21T10:50:23.398 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:23.435 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:23.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:23.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:23.536 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:23.568 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:23.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:23.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:23.669 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:23.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:23.708 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:23.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:23.780 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:23 smithi160 ceph-mon[101879]: osdmap e1441: 8 total, 8 up, 6 in 2024-09-21T10:50:23.780 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:23 smithi160 ceph-mon[101879]: 55.e starting backfill to osd.4 from (0'0,0'0] MAX to 1438'6218 2024-09-21T10:50:23.780 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:23 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 47263/299238 objects degraded (15.794%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:50:23.780 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:23 smithi160 ceph-mon[101879]: osdmap e1442: 8 total, 8 up, 6 in 2024-09-21T10:50:23.810 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:23.817 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 83 16 101520 101504 76.4176 35.625 0.0196567 0.0130791 2024-09-21T10:50:23.844 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:23.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:23.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:23.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:23 smithi070 ceph-mon[107385]: osdmap e1441: 8 total, 8 up, 6 in 2024-09-21T10:50:23.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:23 smithi070 ceph-mon[107385]: 55.e starting backfill to osd.4 from (0'0,0'0] MAX to 1438'6218 2024-09-21T10:50:23.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:23 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 47263/299238 objects degraded (15.794%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:50:23.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:23 smithi070 ceph-mon[107385]: osdmap e1442: 8 total, 8 up, 6 in 2024-09-21T10:50:23.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:23 smithi070 ceph-mon[112139]: osdmap e1441: 8 total, 8 up, 6 in 2024-09-21T10:50:23.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:23 smithi070 ceph-mon[112139]: 55.e starting backfill to osd.4 from (0'0,0'0] MAX to 1438'6218 2024-09-21T10:50:23.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:23 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 47263/299238 objects degraded (15.794%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:50:23.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:23 smithi070 ceph-mon[112139]: osdmap e1442: 8 total, 8 up, 6 in 2024-09-21T10:50:23.949 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:23.991 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:23.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:23.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:24.093 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:24.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:24.122 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:24.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:24.223 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:24.264 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:24.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:24.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:24.366 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:24.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:24.394 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:24.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:24.495 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:24.523 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:24.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:24.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:24.625 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:24.665 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:24.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:24.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:24.711 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:24 smithi070 ceph-mon[107385]: pgmap v2586: 99 pgs: 2 remapped+peering, 1 unknown, 12 peering, 1 active+recovery_wait+degraded, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 72 active+clean; 6.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 28 MiB/s wr, 454 op/s; 42867/302427 objects degraded (14.174%); 14831/302427 objects misplaced (4.904%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T10:50:24.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:24 smithi070 ceph-mon[107385]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:50:24.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:24 smithi070 ceph-mon[107385]: osdmap e1443: 8 total, 8 up, 6 in 2024-09-21T10:50:24.712 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:24 smithi070 ceph-mon[112139]: pgmap v2586: 99 pgs: 2 remapped+peering, 1 unknown, 12 peering, 1 active+recovery_wait+degraded, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 72 active+clean; 6.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 28 MiB/s wr, 454 op/s; 42867/302427 objects degraded (14.174%); 14831/302427 objects misplaced (4.904%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T10:50:24.712 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:24 smithi070 ceph-mon[112139]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:50:24.712 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:24 smithi070 ceph-mon[112139]: osdmap e1443: 8 total, 8 up, 6 in 2024-09-21T10:50:24.768 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:24.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:24.807 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:24.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:24.817 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 84 16 102077 102061 75.9224 34.8125 0.00503783 0.0131641 2024-09-21T10:50:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:24 smithi160 ceph-mon[101879]: pgmap v2586: 99 pgs: 2 remapped+peering, 1 unknown, 12 peering, 1 active+recovery_wait+degraded, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 72 active+clean; 6.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 28 MiB/s wr, 454 op/s; 42867/302427 objects degraded (14.174%); 14831/302427 objects misplaced (4.904%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T10:50:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:24 smithi160 ceph-mon[101879]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:50:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:24 smithi160 ceph-mon[101879]: osdmap e1443: 8 total, 8 up, 6 in 2024-09-21T10:50:24.908 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:24.943 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:24.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:24.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:25.045 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:25.072 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:25.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:25.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:25.174 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:25.210 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:25.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:25.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:25.312 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:25.359 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:25.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:25.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:25.461 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:25.486 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:25.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:25.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:25.588 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:25.626 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:25.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:25.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:25.728 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:25.752 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:25.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:25.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:25.817 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 85 16 102544 102528 75.3726 29.1875 0.146389 0.0132494 2024-09-21T10:50:25.855 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:25.900 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:25.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:25.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:26.001 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:26.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:26.027 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:26.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:26.128 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:26.169 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:26.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:26.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:26.271 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:26.297 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:26.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:26.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:26.398 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:26.424 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:26.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:26.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:26.525 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:26.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:26.573 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:26.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:26 smithi070 ceph-mon[107385]: pgmap v2589: 99 pgs: 2 remapped+peering, 1 unknown, 12 peering, 1 active+recovery_wait+degraded, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 72 active+clean; 6.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 17 MiB/s wr, 265 op/s; 42867/302427 objects degraded (14.174%); 14831/302427 objects misplaced (4.904%) 2024-09-21T10:50:26.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:26 smithi070 ceph-mon[112139]: pgmap v2589: 99 pgs: 2 remapped+peering, 1 unknown, 12 peering, 1 active+recovery_wait+degraded, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 72 active+clean; 6.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 17 MiB/s wr, 265 op/s; 42867/302427 objects degraded (14.174%); 14831/302427 objects misplaced (4.904%) 2024-09-21T10:50:26.674 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:26.701 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:26.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:26.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:26.802 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:26.817 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 86 16 103124 103108 74.9175 36.25 0.016425 0.0133408 2024-09-21T10:50:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:26 smithi160 ceph-mon[101879]: pgmap v2589: 99 pgs: 2 remapped+peering, 1 unknown, 12 peering, 1 active+recovery_wait+degraded, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 72 active+clean; 6.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 17 MiB/s wr, 265 op/s; 42867/302427 objects degraded (14.174%); 14831/302427 objects misplaced (4.904%) 2024-09-21T10:50:26.848 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:26.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:26.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:26.950 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:26.975 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:26.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:26.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:27.077 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:27.117 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:27.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:27.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:27.220 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:27.255 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:27.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:27.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:27.357 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:27.386 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:27.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:27.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:27.488 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:27.513 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:27.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:27.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:27.614 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:27.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:27.656 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:27.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:27.757 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:27.791 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:27.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:27.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:27.817 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 87 15 103715 103700 74.4817 37 0.0188099 0.01342 2024-09-21T10:50:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:27 smithi160 ceph-mon[101879]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-21T10:50:27.846 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:27 smithi070 ceph-mon[107385]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-21T10:50:27.846 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:27 smithi070 ceph-mon[112139]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-21T10:50:27.884 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-21T10:50:27.884 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd test-reweight-by-utilization 110 3.0 2024-09-21T10:50:27.894 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:27.968 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:27.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:27.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:28.070 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:28.094 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:28.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:28.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:28.196 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:28.225 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:28.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:28.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:28.327 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:28.354 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:28.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:28.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:28.456 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:28.475 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:28 smithi070 ceph-mon[107385]: pgmap v2590: 99 pgs: 2 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 31 MiB/s wr, 493 op/s; 66408/307902 objects degraded (21.568%); 20763/307902 objects misplaced (6.743%); 5.5 MiB/s, 88 objects/s recovering 2024-09-21T10:50:28.475 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:28 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 66408/307902 objects degraded (21.568%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:50:28.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:28.501 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:28.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:28.602 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:28.647 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:28.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:28.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:28.748 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:28.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:28.784 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:28.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:28.818 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 88 16 104283 104267 74.038 35.4375 0.0336007 0.0134981 2024-09-21T10:50:28.833 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:28 smithi070 ceph-mon[112139]: pgmap v2590: 99 pgs: 2 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 31 MiB/s wr, 493 op/s; 66408/307902 objects degraded (21.568%); 20763/307902 objects misplaced (6.743%); 5.5 MiB/s, 88 objects/s recovering 2024-09-21T10:50:28.833 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:28 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 66408/307902 objects degraded (21.568%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:50:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:28 smithi160 ceph-mon[101879]: pgmap v2590: 99 pgs: 2 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 31 MiB/s wr, 493 op/s; 66408/307902 objects degraded (21.568%); 20763/307902 objects misplaced (6.743%); 5.5 MiB/s, 88 objects/s recovering 2024-09-21T10:50:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:28 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 66408/307902 objects degraded (21.568%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:50:28.885 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:28.912 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:28.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:28.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:29.014 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:29.041 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:29.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:29.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:29.143 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:29.174 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:29.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:29.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:29.276 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:29.287 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:50:29.288 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:50:29.323 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:29.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:29.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:29.426 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:29.450 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:29.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:29.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:29.516 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:50:29.553 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:29.560 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:50:29.611 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:29.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:29.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:29.713 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:29.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:29.760 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:29.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:29.818 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 89 16 104809 104793 73.5756 32.875 0.0279846 0.0135846 2024-09-21T10:50:29.861 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:29.888 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:29.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:29.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:29.989 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:30.021 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:30.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:30.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:30.059 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:50:30.123 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:30.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:30.154 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:30.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:30.255 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:30.279 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:30.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:30.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:30.380 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:30.407 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:30.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:30.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:30.484 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T10:50:30.509 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:30.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:30.556 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:30.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:30.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:30 smithi070 ceph-mon[107385]: pgmap v2591: 99 pgs: 2 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 18 MiB/s wr, 282 op/s; 66408/307902 objects degraded (21.568%); 20763/307902 objects misplaced (6.743%); 4.7 MiB/s, 75 objects/s recovering 2024-09-21T10:50:30.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:30 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1308259958' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:50:30.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:30 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:50:30.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:30 smithi070 ceph-mon[112139]: pgmap v2591: 99 pgs: 2 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 18 MiB/s wr, 282 op/s; 66408/307902 objects degraded (21.568%); 20763/307902 objects misplaced (6.743%); 4.7 MiB/s, 75 objects/s recovering 2024-09-21T10:50:30.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:30 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1308259958' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:50:30.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:30 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:50:30.657 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:30.683 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:30.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:30.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:30.785 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:30.812 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:30.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:30.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:30.818 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Total time run: 90.0177 2024-09-21T10:50:30.818 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Total writes made: 105338 2024-09-21T10:50:30.818 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Write size: 65536 2024-09-21T10:50:30.818 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Object size: 65536 2024-09-21T10:50:30.818 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Bandwidth (MB/sec): 73.137 2024-09-21T10:50:30.818 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Stddev Bandwidth: 48.5252 2024-09-21T10:50:30.818 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Max bandwidth (MB/sec): 240.625 2024-09-21T10:50:30.818 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Min bandwidth (MB/sec): 0 2024-09-21T10:50:30.818 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Average IOPS: 1170 2024-09-21T10:50:30.818 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Stddev IOPS: 776.417 2024-09-21T10:50:30.819 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Max IOPS: 3850 2024-09-21T10:50:30.819 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Min IOPS: 0 2024-09-21T10:50:30.819 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Average Latency(s): 0.0136677 2024-09-21T10:50:30.819 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Stddev Latency(s): 0.0456532 2024-09-21T10:50:30.819 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Max latency(s): 2.34169 2024-09-21T10:50:30.819 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Min latency(s): 0.00113626 2024-09-21T10:50:30.830 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T10:50:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:30 smithi160 ceph-mon[101879]: pgmap v2591: 99 pgs: 2 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 18 MiB/s wr, 282 op/s; 66408/307902 objects degraded (21.568%); 20763/307902 objects misplaced (6.743%); 4.7 MiB/s, 75 objects/s recovering 2024-09-21T10:50:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:30 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1308259958' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:50:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:30 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:50:30.913 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:30.941 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:30.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:30.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:31.043 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:31.068 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:31.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:31.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:31.170 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:31.202 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:31.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:31.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:31.303 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:31.327 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:31.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:31.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:31.429 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:31.450 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:31.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:31.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:31.552 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:31.603 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:31.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:31.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:31.705 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:31.730 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:31.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:31.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:31.831 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:31 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:50:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:31 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:50:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:31 smithi160 ceph-mon[101879]: osdmap e1444: 8 total, 8 up, 6 in 2024-09-21T10:50:31.852 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:31.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:31.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:31 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:50:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:31 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:50:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:31 smithi070 ceph-mon[107385]: osdmap e1444: 8 total, 8 up, 6 in 2024-09-21T10:50:31.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:31 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:50:31.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:31 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:50:31.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:31 smithi070 ceph-mon[112139]: osdmap e1444: 8 total, 8 up, 6 in 2024-09-21T10:50:31.954 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:32.004 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:32.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:32.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:32.106 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:32.127 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:32.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:32.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:32.229 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:32.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:32.262 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:32.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:32.363 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:32.405 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:32.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:32.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:32.506 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:32.540 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:32.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:32.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:32.642 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:32.665 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:32.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:32.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:32.766 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:32.771 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:32 smithi070 ceph-mon[107385]: pgmap v2593: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 28 MiB/s wr, 443 op/s; 66408/312546 objects degraded (21.247%); 20885/312546 objects misplaced (6.682%); 5.9 MiB/s, 94 objects/s recovering 2024-09-21T10:50:32.771 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:32 smithi070 ceph-mon[112139]: pgmap v2593: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 28 MiB/s wr, 443 op/s; 66408/312546 objects degraded (21.247%); 20885/312546 objects misplaced (6.682%); 5.9 MiB/s, 94 objects/s recovering 2024-09-21T10:50:32.791 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:32.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:32.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:32 smithi160 ceph-mon[101879]: pgmap v2593: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 28 MiB/s wr, 443 op/s; 66408/312546 objects degraded (21.247%); 20885/312546 objects misplaced (6.682%); 5.9 MiB/s, 94 objects/s recovering 2024-09-21T10:50:32.893 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:32.923 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:32.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:32.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:33.025 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:33.049 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:33.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:33.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:33.150 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:33.173 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:33.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:33.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:33.275 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:33.304 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:33.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:33.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:33.406 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:33.444 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:33.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:33.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:33.477 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:50:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:50:33] "GET /metrics HTTP/1.1" 200 36666 "" "Prometheus/2.51.0" 2024-09-21T10:50:33.546 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:33.580 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:33.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:33.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:33.682 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:33.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:33.728 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:33.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:33.829 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:33 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 66408/312546 objects degraded (21.247%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:50:33.869 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:33.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:33.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:33 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 66408/312546 objects degraded (21.247%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:50:33.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:33 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 66408/312546 objects degraded (21.247%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:50:33.971 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:34.024 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:34.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:34.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:34.032 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:50:34.035 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:50:34.126 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:34.152 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:34.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:34.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:34.253 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:34.278 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:34.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:34.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:34.380 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:34.433 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:34.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:34.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:34.534 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:34.559 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:34.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:34.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:34.660 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:34.690 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:34 smithi070 ceph-mon[107385]: pgmap v2594: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 118 B/s rd, 32 MiB/s wr, 531 op/s; 66408/315198 objects degraded (21.069%); 20885/315198 objects misplaced (6.626%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T10:50:34.691 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:34 smithi070 ceph-mon[112139]: pgmap v2594: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 118 B/s rd, 32 MiB/s wr, 531 op/s; 66408/315198 objects degraded (21.069%); 20885/315198 objects misplaced (6.626%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T10:50:34.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:34.715 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:34.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:34.816 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:34.842 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:34.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:34 smithi160 ceph-mon[101879]: pgmap v2594: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 118 B/s rd, 32 MiB/s wr, 531 op/s; 66408/315198 objects degraded (21.069%); 20885/315198 objects misplaced (6.626%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T10:50:34.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:34.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:34.944 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:34.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:34.989 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:34.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:35.089 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:35.113 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:35.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:35.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:35.215 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:35.222 INFO:teuthology.orchestra.run.smithi070.stdout:moved 22 / 297 (7.40741%) 2024-09-21T10:50:35.222 INFO:teuthology.orchestra.run.smithi070.stdout:avg 49.5 2024-09-21T10:50:35.222 INFO:teuthology.orchestra.run.smithi070.stdout:stddev 13.6229 -> 20.1804 (expected baseline 6.42262) 2024-09-21T10:50:35.222 INFO:teuthology.orchestra.run.smithi070.stdout:min osd.7 with 23 -> 14 pgs (0.464646 -> 0.282828 * mean) 2024-09-21T10:50:35.222 INFO:teuthology.orchestra.run.smithi070.stdout:max osd.1 with 62 -> 66 pgs (1.25253 -> 1.33333 * mean) 2024-09-21T10:50:35.222 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:50:35.223 INFO:teuthology.orchestra.run.smithi070.stdout:oload 110 2024-09-21T10:50:35.223 INFO:teuthology.orchestra.run.smithi070.stdout:max_change 3 2024-09-21T10:50:35.223 INFO:teuthology.orchestra.run.smithi070.stdout:max_change_osds 4 2024-09-21T10:50:35.223 INFO:teuthology.orchestra.run.smithi070.stdout:average_utilization 0.0393 2024-09-21T10:50:35.223 INFO:teuthology.orchestra.run.smithi070.stdout:overload_utilization 0.0433 2024-09-21T10:50:35.223 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 weight 0.4399 -> 0.3021 2024-09-21T10:50:35.223 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 weight 0.6819 -> 0.5723 2024-09-21T10:50:35.223 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 weight 0.7775 -> 0.6722 2024-09-21T10:50:35.223 INFO:teuthology.orchestra.run.smithi070.stderr:no change 2024-09-21T10:50:35.259 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:35.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:35.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:35.350 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:50:35 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T10:50:35.214+0000 7fd088d1d640 -1 mgr.server reply reply (3) No such process no change 2024-09-21T10:50:35.361 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:35.391 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:35.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:35.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:35.493 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:35.530 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:35.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:35.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:35.606 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:50:35.606 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:50:35.632 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:35.655 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:35.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:35.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:35.757 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:35.783 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:35.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:35.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:50:35.885 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:35.928 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:35.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:35.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:36.030 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:36.068 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:36.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:36.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:36.088 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:50:36.169 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:36.228 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:36.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:36.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:36.319 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-21T10:50:36.319 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 1000 1.0 2024-09-21T10:50:36.329 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:36.355 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:36.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:36.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:36.456 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:36.478 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:36.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:36.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:36.580 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:36.610 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:36.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:36.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:36.712 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:36.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:36 smithi070 ceph-mon[107385]: pgmap v2595: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 102 B/s rd, 28 MiB/s wr, 457 op/s; 66408/315198 objects degraded (21.069%); 20885/315198 objects misplaced (6.626%); 4.5 MiB/s, 71 objects/s recovering 2024-09-21T10:50:36.740 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:36 smithi070 ceph-mon[107385]: from='client.35757 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:50:36.740 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:36 smithi070 ceph-mon[112139]: pgmap v2595: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 102 B/s rd, 28 MiB/s wr, 457 op/s; 66408/315198 objects degraded (21.069%); 20885/315198 objects misplaced (6.626%); 4.5 MiB/s, 71 objects/s recovering 2024-09-21T10:50:36.740 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:36 smithi070 ceph-mon[112139]: from='client.35757 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:50:36.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:36.752 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:36.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:36 smithi160 ceph-mon[101879]: pgmap v2595: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 102 B/s rd, 28 MiB/s wr, 457 op/s; 66408/315198 objects degraded (21.069%); 20885/315198 objects misplaced (6.626%); 4.5 MiB/s, 71 objects/s recovering 2024-09-21T10:50:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:36 smithi160 ceph-mon[101879]: from='client.35757 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:50:36.853 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:36.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:36.899 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:36.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:37.000 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:37.025 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:37.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:37.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:37.127 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:37.156 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:37.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:37.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:37.258 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:37.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:37.297 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:37.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:37.398 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:37.406 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:50:37.423 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:37.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:37.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:37.524 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:37.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:37.585 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:37.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:37.686 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:37.710 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:37.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:37.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:37.812 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:37.845 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:37.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:37.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:37.946 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:37.995 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:37.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:37.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:38.097 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:38.122 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:38.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:38.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:38.224 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:38.247 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:38.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:38.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:38.349 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:38.372 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:38.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:38.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:38.474 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:38.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:38.523 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:38.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:38.603 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:38 smithi070 ceph-mon[107385]: pgmap v2596: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 102 B/s rd, 17 MiB/s wr, 506 op/s; 65962/309102 objects degraded (21.340%); 20369/309102 objects misplaced (6.590%); 4.6 MiB/s, 74 objects/s recovering 2024-09-21T10:50:38.603 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:38 smithi070 ceph-mon[112139]: pgmap v2596: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 102 B/s rd, 17 MiB/s wr, 506 op/s; 65962/309102 objects degraded (21.340%); 20369/309102 objects misplaced (6.590%); 4.6 MiB/s, 74 objects/s recovering 2024-09-21T10:50:38.624 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:38.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:38.677 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:38.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:38.778 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:38.801 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:38.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:38.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:38 smithi160 ceph-mon[101879]: pgmap v2596: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 102 B/s rd, 17 MiB/s wr, 506 op/s; 65962/309102 objects degraded (21.340%); 20369/309102 objects misplaced (6.590%); 4.6 MiB/s, 74 objects/s recovering 2024-09-21T10:50:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:38 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 65962/309102 objects degraded (21.340%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:50:38.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:38 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 65962/309102 objects degraded (21.340%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:50:38.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:38 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 65962/309102 objects degraded (21.340%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:50:38.904 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:38.940 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:38.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:38.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:39.042 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:39.068 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:39.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:39.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:39.170 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:39.216 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:39.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:39.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:39.318 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:39.361 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:39.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:39.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:39.463 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:39.484 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:39.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:39.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:39.586 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:39.608 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:39.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:39.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:39.710 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:39.732 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:39.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:39.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:39.833 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:39.859 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:39.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:39.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:39.961 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:39.984 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:39.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:39.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:40.085 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:40.109 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:40.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:40.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:40.211 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:40.248 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:40.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:40.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:40.350 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:40.393 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:40.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:40.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:40.495 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:40.519 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:40.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:40.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:40.621 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:40.671 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:40.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:40.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:40.772 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:40.802 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:40.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:40.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:40.804 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:50:40.804 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:50:40.804 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:50:40.804 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:50:40.804 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:50:40.805 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:50:40.805 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:50:40.805 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:50:40.805 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:50:40.805 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:50:40.805 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:50:40.805 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:50:40.805 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:50:40.805 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:50:40.806 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:50:40.806 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:50:40.806 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:40 smithi070 ceph-mon[107385]: pgmap v2597: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 102 B/s rd, 17 MiB/s wr, 506 op/s; 65962/309102 objects degraded (21.340%); 20369/309102 objects misplaced (6.590%); 4.6 MiB/s, 74 objects/s recovering 2024-09-21T10:50:40.806 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:40 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/85824649' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:50:40.806 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:40 smithi070 ceph-mon[112139]: pgmap v2597: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 102 B/s rd, 17 MiB/s wr, 506 op/s; 65962/309102 objects degraded (21.340%); 20369/309102 objects misplaced (6.590%); 4.6 MiB/s, 74 objects/s recovering 2024-09-21T10:50:40.806 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:40 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/85824649' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:50:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:40 smithi160 ceph-mon[101879]: pgmap v2597: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 102 B/s rd, 17 MiB/s wr, 506 op/s; 65962/309102 objects degraded (21.340%); 20369/309102 objects misplaced (6.590%); 4.6 MiB/s, 74 objects/s recovering 2024-09-21T10:50:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:40 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/85824649' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:50:40.904 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:40.926 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:40.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:40.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:41.028 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:41.064 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:41.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:41.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:41.166 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:41.198 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:41.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:41.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:41.300 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:41.323 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:41.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:41.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:41.425 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:41.447 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:41.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:41.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:41.549 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:41.574 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:41.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:41.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:41.676 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:41.703 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:41.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:41.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:41.805 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:41.830 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:41.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:41.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:41.931 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:41.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:41.960 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:41.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:42.061 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:42.089 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:42.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:42.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:42.190 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:42.216 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:42.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:42.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:42.318 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:42.344 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:42.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:42.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:42.446 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:42.472 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:42.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:42.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:42.573 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:42.611 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:42.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:42.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:42.712 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:42.753 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:42.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:42.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:42 smithi160 ceph-mon[101879]: pgmap v2598: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.2 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 98 B/s rd, 12 MiB/s wr, 542 op/s; 65382/305337 objects degraded (21.413%); 19789/305337 objects misplaced (6.481%); 7.3 MiB/s, 116 objects/s recovering 2024-09-21T10:50:42.854 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:42.877 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:42.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:42 smithi070 ceph-mon[107385]: pgmap v2598: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.2 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 98 B/s rd, 12 MiB/s wr, 542 op/s; 65382/305337 objects degraded (21.413%); 19789/305337 objects misplaced (6.481%); 7.3 MiB/s, 116 objects/s recovering 2024-09-21T10:50:42.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:42 smithi070 ceph-mon[112139]: pgmap v2598: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.2 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 98 B/s rd, 12 MiB/s wr, 542 op/s; 65382/305337 objects degraded (21.413%); 19789/305337 objects misplaced (6.481%); 7.3 MiB/s, 116 objects/s recovering 2024-09-21T10:50:42.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:42.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:42.979 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:43.007 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:43.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:43.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:43.109 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:43.133 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:43.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:43.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:43.235 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:43.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:43.270 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:43.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:43.371 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:43.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:50:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:50:43] "GET /metrics HTTP/1.1" 200 36666 "" "Prometheus/2.51.0" 2024-09-21T10:50:43.395 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:43.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:43.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:43.497 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:43.535 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:43.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:43.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:43.637 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:43.662 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:43.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:43.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:43.764 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:43.809 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:43.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:43.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:43.829 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:43 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 65382/305337 objects degraded (21.413%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:50:43.829 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:43 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 65382/305337 objects degraded (21.413%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:50:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:43 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 65382/305337 objects degraded (21.413%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:50:43.910 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:43.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:43.960 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:43.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:44.061 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:44.089 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:44.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:44.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:44.191 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:44.221 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:44.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:44.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:44.323 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:44.383 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:44.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:44.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:44.485 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:44.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:44.522 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:44.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:44.623 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:44.645 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:44.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:44.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:44.703 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:50:44.747 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:44.774 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:44.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:44.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:44 smithi160 ceph-mon[101879]: pgmap v2599: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 85 B/s rd, 6.2 MiB/s wr, 480 op/s; 65382/302334 objects degraded (21.626%); 19789/302334 objects misplaced (6.545%); 5.7 MiB/s, 91 objects/s recovering 2024-09-21T10:50:44.843 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:44 smithi070 ceph-mon[107385]: pgmap v2599: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 85 B/s rd, 6.2 MiB/s wr, 480 op/s; 65382/302334 objects degraded (21.626%); 19789/302334 objects misplaced (6.545%); 5.7 MiB/s, 91 objects/s recovering 2024-09-21T10:50:44.843 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:44 smithi070 ceph-mon[112139]: pgmap v2599: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 85 B/s rd, 6.2 MiB/s wr, 480 op/s; 65382/302334 objects degraded (21.626%); 19789/302334 objects misplaced (6.545%); 5.7 MiB/s, 91 objects/s recovering 2024-09-21T10:50:44.876 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:44.900 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:44.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:44.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:45.001 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:45.029 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:45.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:45.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:45.131 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:45.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:45.175 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:45.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:45.276 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:45.318 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:45.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:45.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:45.420 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:45.456 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:45.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:45.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:45.558 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:45.581 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:45.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:45.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:45.603 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:50:45.683 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:45.723 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:45.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:45.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:45.824 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:45.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:45.847 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:45.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:45.948 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:46.001 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:46.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:46.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:46.103 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:46.125 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:46.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:46.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:46.157 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:50:46.157 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T10:50:46.157 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T10:50:46.157 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T10:50:46.157 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:50:46.227 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:46.255 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:46.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:46.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:46.356 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:46.405 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:46.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:46.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:46.507 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:46.530 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:46.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:46.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:46.632 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:46.675 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:46.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:46.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:46.777 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:46.801 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:46.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:46.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:46 smithi160 ceph-mon[101879]: pgmap v2600: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 752 KiB/s wr, 380 op/s; 65382/302334 objects degraded (21.626%); 19789/302334 objects misplaced (6.545%); 5.7 MiB/s, 91 objects/s recovering 2024-09-21T10:50:46.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:46 smithi070 ceph-mon[107385]: pgmap v2600: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 752 KiB/s wr, 380 op/s; 65382/302334 objects degraded (21.626%); 19789/302334 objects misplaced (6.545%); 5.7 MiB/s, 91 objects/s recovering 2024-09-21T10:50:46.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:46 smithi070 ceph-mon[112139]: pgmap v2600: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 752 KiB/s wr, 380 op/s; 65382/302334 objects degraded (21.626%); 19789/302334 objects misplaced (6.545%); 5.7 MiB/s, 91 objects/s recovering 2024-09-21T10:50:46.903 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:46.949 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:46.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:46.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:47.051 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:47.077 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:47.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:47.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:47.178 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:47.220 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:47.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:47.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:47.322 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:47.345 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:47.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:47.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:47.447 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:47.477 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:47.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:47.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:47.578 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:47.603 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:47.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:47.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:47.704 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:47.745 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:47.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:47.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:47.824 INFO:teuthology.orchestra.run.smithi070.stdout:moved 0 / 297 (0%) 2024-09-21T10:50:47.824 INFO:teuthology.orchestra.run.smithi070.stdout:avg 49.5 2024-09-21T10:50:47.824 INFO:teuthology.orchestra.run.smithi070.stdout:stddev 13.6229 -> 13.6229 (expected baseline 6.42262) 2024-09-21T10:50:47.824 INFO:teuthology.orchestra.run.smithi070.stdout:min osd.7 with 23 -> 23 pgs (0.464646 -> 0.464646 * mean) 2024-09-21T10:50:47.824 INFO:teuthology.orchestra.run.smithi070.stdout:max osd.1 with 62 -> 62 pgs (1.25253 -> 1.25253 * mean) 2024-09-21T10:50:47.824 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:50:47.824 INFO:teuthology.orchestra.run.smithi070.stdout:oload 1000 2024-09-21T10:50:47.824 INFO:teuthology.orchestra.run.smithi070.stdout:max_change 1 2024-09-21T10:50:47.824 INFO:teuthology.orchestra.run.smithi070.stdout:max_change_osds 4 2024-09-21T10:50:47.825 INFO:teuthology.orchestra.run.smithi070.stdout:average_utilization 0.0378 2024-09-21T10:50:47.825 INFO:teuthology.orchestra.run.smithi070.stdout:overload_utilization 0.3783 2024-09-21T10:50:47.825 INFO:teuthology.orchestra.run.smithi070.stderr:no change 2024-09-21T10:50:47.847 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:47.873 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:47.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:47.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:47.975 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:48.029 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:48.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:48.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:48.131 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:48.157 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:48.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:48.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:48.259 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:48.288 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:48.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:48.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:48.389 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:48.412 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:50:48.426 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:48.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:48.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:48.528 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:48.554 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:48.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:48.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:48.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:48 smithi070 ceph-mon[107385]: pgmap v2601: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 752 KiB/s wr, 574 op/s; 64717/295356 objects degraded (21.912%); 19201/295356 objects misplaced (6.501%); 9.1 MiB/s, 145 objects/s recovering 2024-09-21T10:50:48.633 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:48 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 64717/295356 objects degraded (21.912%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:50:48.633 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:48 smithi070 ceph-mon[112139]: pgmap v2601: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 752 KiB/s wr, 574 op/s; 64717/295356 objects degraded (21.912%); 19201/295356 objects misplaced (6.501%); 9.1 MiB/s, 145 objects/s recovering 2024-09-21T10:50:48.634 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:48 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 64717/295356 objects degraded (21.912%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:50:48.655 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:48.679 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:48.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:48.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:48.781 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:48.806 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:48.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:48.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:48 smithi160 ceph-mon[101879]: pgmap v2601: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 752 KiB/s wr, 574 op/s; 64717/295356 objects degraded (21.912%); 19201/295356 objects misplaced (6.501%); 9.1 MiB/s, 145 objects/s recovering 2024-09-21T10:50:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:48 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 64717/295356 objects degraded (21.912%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:50:48.909 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:48.933 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:48.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:48.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:49.034 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:49.059 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:49.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:49.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:49.102 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-21T10:50:49.102 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd test-reweight-by-utilization 110 3.0 2024-09-21T10:50:49.161 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:49.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:49.190 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:49.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:49.291 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:49.317 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:49.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:49.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:49.418 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:49.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:49.474 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:49.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:49.575 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:49.587 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:50:49.601 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:49 smithi070 ceph-mon[107385]: from='client.35766 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:50:49.604 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:49 smithi070 ceph-mon[112139]: from='client.35766 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:50:49.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:49.664 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:49.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:49.677 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:50:49.710 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:50:49.766 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:49.790 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:49.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:49.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:49.892 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:49.915 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:49.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:49.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:50.016 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:50.039 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:50.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:50.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:50.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:49 smithi160 ceph-mon[101879]: from='client.35766 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:50:50.143 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:50.172 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:50.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:50.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:50.274 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:50.319 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:50.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:50.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:50.420 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:50.446 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:50.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:50.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:50.548 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:50.570 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:50.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:50.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:50.607 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T10:50:50.672 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:50.693 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:50.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:50.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:50.795 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:50.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:50.834 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:50.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:50.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:50 smithi070 ceph-mon[112139]: pgmap v2602: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 381 op/s; 64717/295356 objects degraded (21.912%); 19201/295356 objects misplaced (6.501%); 6.4 MiB/s, 103 objects/s recovering 2024-09-21T10:50:50.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:50 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/948768337' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:50:50.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:50 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:50:50.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:50:50.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:50 smithi070 ceph-mon[107385]: pgmap v2602: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 381 op/s; 64717/295356 objects degraded (21.912%); 19201/295356 objects misplaced (6.501%); 6.4 MiB/s, 103 objects/s recovering 2024-09-21T10:50:50.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:50 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/948768337' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:50:50.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:50 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:50:50.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:50:50.934 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:50.961 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:50.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:50.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:51.063 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:51.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:50 smithi160 ceph-mon[101879]: pgmap v2602: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 6.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 381 op/s; 64717/295356 objects degraded (21.912%); 19201/295356 objects misplaced (6.501%); 6.4 MiB/s, 103 objects/s recovering 2024-09-21T10:50:51.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:50 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/948768337' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:50:51.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:50 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:50:51.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:50:51.104 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:51.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:51.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:51.206 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:51.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:51.232 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:51.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:51.333 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:51.357 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:51.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:51.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:51.459 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:51.482 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:51.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:51.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:51.584 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:51.613 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:51.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:51.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:51.714 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:51.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:51.759 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:51.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:51.860 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:51.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:51 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:50:51.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:51 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:50:51.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:51 smithi070 ceph-mon[112139]: osdmap e1445: 8 total, 8 up, 6 in 2024-09-21T10:50:51.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:51 smithi070 ceph-mon[112139]: pgmap v2604: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.9 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 455 op/s; 64128/291657 objects degraded (21.987%); 18608/291657 objects misplaced (6.380%); 7.9 MiB/s, 125 objects/s recovering 2024-09-21T10:50:51.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:51 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:50:51.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:51 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:50:51.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:51 smithi070 ceph-mon[107385]: osdmap e1445: 8 total, 8 up, 6 in 2024-09-21T10:50:51.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:51 smithi070 ceph-mon[107385]: pgmap v2604: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.9 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 455 op/s; 64128/291657 objects degraded (21.987%); 18608/291657 objects misplaced (6.380%); 7.9 MiB/s, 125 objects/s recovering 2024-09-21T10:50:51.894 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:51.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:51.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:51.996 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:52.020 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:52.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:52.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:52.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:51 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:50:52.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:51 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:50:52.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:51 smithi160 ceph-mon[101879]: osdmap e1445: 8 total, 8 up, 6 in 2024-09-21T10:50:52.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:51 smithi160 ceph-mon[101879]: pgmap v2604: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.9 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 455 op/s; 64128/291657 objects degraded (21.987%); 18608/291657 objects misplaced (6.380%); 7.9 MiB/s, 125 objects/s recovering 2024-09-21T10:50:52.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:52.150 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:52.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:52.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:52.252 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:52.288 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:52.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:52.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:52.390 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:52.422 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:52.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:52.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:52.524 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:52.550 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:52.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:52.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:52.651 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:52.700 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:52.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:52.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:52.802 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:52.853 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:52.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:52.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:52.955 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:52.992 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:52.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:52.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:53.094 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:53.123 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:53.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:53.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:53.225 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:53.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:53.273 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:53.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:53.374 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:53.410 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:50:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:50:53] "GET /metrics HTTP/1.1" 200 36671 "" "Prometheus/2.51.0" 2024-09-21T10:50:53.432 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:53.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:53.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:53.533 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:53.561 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:53.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:53.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:53.578 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:50:53.662 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:53.705 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:53.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:53.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:53.807 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:53.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:53.872 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:53.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:53.972 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:53.999 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:54.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:54.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:54.101 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:54.126 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:54.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:53 smithi070 ceph-mon[107385]: pgmap v2605: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.9 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 450 op/s; 63842/288822 objects degraded (22.104%); 18608/288822 objects misplaced (6.443%); 8.8 MiB/s, 140 objects/s recovering 2024-09-21T10:50:54.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:53 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 63842/288822 objects degraded (22.104%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:50:54.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:53 smithi070 ceph-mon[112139]: pgmap v2605: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.9 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 450 op/s; 63842/288822 objects degraded (22.104%); 18608/288822 objects misplaced (6.443%); 8.8 MiB/s, 140 objects/s recovering 2024-09-21T10:50:54.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:53 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 63842/288822 objects degraded (22.104%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:50:54.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:54.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:54.229 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:54.267 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:54.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:54.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:54.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:53 smithi160 ceph-mon[101879]: pgmap v2605: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.9 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 450 op/s; 63842/288822 objects degraded (22.104%); 18608/288822 objects misplaced (6.443%); 8.8 MiB/s, 140 objects/s recovering 2024-09-21T10:50:54.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:53 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 63842/288822 objects degraded (22.104%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:50:54.368 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:54.400 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:54.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:54.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:54.501 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:54.552 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:54.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:54.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:54.654 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:54.678 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:54.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:54.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:54.780 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:54.822 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:54.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:54.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:54.923 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:54.960 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:54.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:54.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:55.062 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:55.098 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:55.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:55.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:55.200 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:55.228 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:55.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:55.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:55.330 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:55.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:55.375 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:55.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:55.476 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:55.499 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:55.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:55.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:55.601 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:55.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:55.640 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:55.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:55.740 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:55.763 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:55.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:55.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:55.864 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:55.890 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:55.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:55.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:55.991 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:56.016 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:56.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:56.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:56.118 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:56.167 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:56.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:56.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:56.237 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:55 smithi070 ceph-mon[107385]: pgmap v2606: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.9 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 450 op/s; 63842/288822 objects degraded (22.104%); 18608/288822 objects misplaced (6.443%); 8.8 MiB/s, 140 objects/s recovering 2024-09-21T10:50:56.237 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:55 smithi070 ceph-mon[112139]: pgmap v2606: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.9 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 450 op/s; 63842/288822 objects degraded (22.104%); 18608/288822 objects misplaced (6.443%); 8.8 MiB/s, 140 objects/s recovering 2024-09-21T10:50:56.269 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:56.296 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:56.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:56.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:56.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:55 smithi160 ceph-mon[101879]: pgmap v2606: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.9 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 450 op/s; 63842/288822 objects degraded (22.104%); 18608/288822 objects misplaced (6.443%); 8.8 MiB/s, 140 objects/s recovering 2024-09-21T10:50:56.397 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:56.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:56.427 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:56.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:56.527 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:56.570 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:56.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:56.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:56.672 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:56.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:56.699 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:56.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:56.800 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:56.829 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:56.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:56.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:56.930 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:56.958 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:56.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:56.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:57.059 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:57.090 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:57.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:57.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:57.192 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:57.231 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:57.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:57.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:57.333 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:57.361 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:57.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:57.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:57.463 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:57.491 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:57.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:57.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:57.593 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:57.636 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:57.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:57.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:57.738 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:57.764 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:57.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:57.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:57.865 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:57.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:57.897 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:57.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:57.998 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:58.021 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:58.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:58.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:58.123 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:58.149 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:58.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:58.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:58.250 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:58.278 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:58.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:58.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:58.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:57 smithi160 ceph-mon[101879]: pgmap v2607: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 449 op/s; 62945/281862 objects degraded (22.332%); 17974/281862 objects misplaced (6.377%); 9.6 MiB/s, 154 objects/s recovering 2024-09-21T10:50:58.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:57 smithi070 ceph-mon[112139]: pgmap v2607: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 449 op/s; 62945/281862 objects degraded (22.332%); 17974/281862 objects misplaced (6.377%); 9.6 MiB/s, 154 objects/s recovering 2024-09-21T10:50:58.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:57 smithi070 ceph-mon[107385]: pgmap v2607: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 449 op/s; 62945/281862 objects degraded (22.332%); 17974/281862 objects misplaced (6.377%); 9.6 MiB/s, 154 objects/s recovering 2024-09-21T10:50:58.380 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:58.425 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:58.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:58.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:58.527 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:58.552 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:58.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:58.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:58.653 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:58.693 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:58.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:58.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:58.794 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:58.819 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:58.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:58.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:58.921 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:58.944 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:58.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:58.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:59.046 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:59.094 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:59.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:59.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:59.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:58 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 62945/281862 objects degraded (22.332%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:50:59.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:58 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 62945/281862 objects degraded (22.332%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:50:59.196 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:59.228 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:59.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:59.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:59.330 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:59.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:58 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 62945/281862 objects degraded (22.332%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:50:59.362 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:59.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:59.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:59.464 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:59.510 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:59.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:59.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:59.612 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:59.655 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:59.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:59.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:59.757 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:59.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:59.798 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:50:59.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:59.899 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:59.928 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:50:59.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:59.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:00.030 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:00.082 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:00.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:00.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:00.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:50:59 smithi070 ceph-mon[112139]: pgmap v2608: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 449 op/s; 62945/281862 objects degraded (22.332%); 17974/281862 objects misplaced (6.377%); 9.6 MiB/s, 154 objects/s recovering 2024-09-21T10:51:00.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:50:59 smithi070 ceph-mon[107385]: pgmap v2608: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 449 op/s; 62945/281862 objects degraded (22.332%); 17974/281862 objects misplaced (6.377%); 9.6 MiB/s, 154 objects/s recovering 2024-09-21T10:51:00.184 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:00.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:00.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:00.316 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:00.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:50:59 smithi160 ceph-mon[101879]: pgmap v2608: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 449 op/s; 62945/281862 objects degraded (22.332%); 17974/281862 objects misplaced (6.377%); 9.6 MiB/s, 154 objects/s recovering 2024-09-21T10:51:00.352 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:00.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:00.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:00.453 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:00.478 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:00.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:00.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:00.579 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:00.606 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:00.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:00.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:00.708 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:00.741 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:00.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:00.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:00.843 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:00.873 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:00.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:00.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:00.975 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:01.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:01.005 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:01.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:01.106 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:01.131 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:01.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:01.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:01.233 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:01.267 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:01.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:01.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:01.368 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:01.405 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:01.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:01.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:01.506 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:01.531 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:01.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:01.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:01.633 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:01.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:01.671 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:01.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:01.772 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:01.822 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:01.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:01.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:01.924 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:01.959 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:01.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:01.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:02.061 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:02.098 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:02.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:02.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:02.200 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:02.251 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:02.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:02.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:02.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:01 smithi160 ceph-mon[101879]: pgmap v2609: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 499 op/s; 62383/278199 objects degraded (22.424%); 17411/278199 objects misplaced (6.258%); 10 MiB/s, 166 objects/s recovering 2024-09-21T10:51:02.353 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:02.354 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:01 smithi070 ceph-mon[107385]: pgmap v2609: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 499 op/s; 62383/278199 objects degraded (22.424%); 17411/278199 objects misplaced (6.258%); 10 MiB/s, 166 objects/s recovering 2024-09-21T10:51:02.355 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:01 smithi070 ceph-mon[112139]: pgmap v2609: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 499 op/s; 62383/278199 objects degraded (22.424%); 17411/278199 objects misplaced (6.258%); 10 MiB/s, 166 objects/s recovering 2024-09-21T10:51:02.377 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:02.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:02.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:02.478 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:02.502 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:51:02.502 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:51:02.502 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:51:02.502 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:51:02.503 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:51:02.503 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:51:02.503 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:51:02.503 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:51:02.524 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:51:02.524 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:51:02.533 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:02.536 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:02.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:02.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:02.563 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:51:02.563 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:51:02.597 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:51:02.597 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:51:02.626 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:51:02.626 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:51:02.639 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:02.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:02.684 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:02.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:02.785 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:02.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:02.837 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:02.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:02.938 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:02.964 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:02.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:02.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:03.065 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:03.106 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:03.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:03.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:03.208 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:03.222 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:02 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:51:03.222 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:02 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/632691464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:51:03.222 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:02 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:51:03.222 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:02 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:51:03.222 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:51:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:51:03] "GET /metrics HTTP/1.1" 200 36671 "" "Prometheus/2.51.0" 2024-09-21T10:51:03.223 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:02 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:51:03.223 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:02 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/632691464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:51:03.223 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:02 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:51:03.223 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:02 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:51:03.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:03.272 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:03.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:03.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:02 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:51:03.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:02 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/632691464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:51:03.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:02 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:51:03.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:02 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:51:03.373 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:03.404 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:03.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:03.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:03.506 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:03.558 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:03.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:03.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:03.626 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:51:03.626 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:51:03.640 INFO:teuthology.orchestra.run.smithi070.stdout:moved 21 / 297 (7.07071%) 2024-09-21T10:51:03.640 INFO:teuthology.orchestra.run.smithi070.stdout:avg 49.5 2024-09-21T10:51:03.640 INFO:teuthology.orchestra.run.smithi070.stdout:stddev 13.6229 -> 19.9395 (expected baseline 6.42262) 2024-09-21T10:51:03.640 INFO:teuthology.orchestra.run.smithi070.stdout:min osd.7 with 23 -> 14 pgs (0.464646 -> 0.282828 * mean) 2024-09-21T10:51:03.640 INFO:teuthology.orchestra.run.smithi070.stdout:max osd.1 with 62 -> 66 pgs (1.25253 -> 1.33333 * mean) 2024-09-21T10:51:03.640 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:51:03.640 INFO:teuthology.orchestra.run.smithi070.stdout:oload 110 2024-09-21T10:51:03.640 INFO:teuthology.orchestra.run.smithi070.stdout:max_change 3 2024-09-21T10:51:03.640 INFO:teuthology.orchestra.run.smithi070.stdout:max_change_osds 4 2024-09-21T10:51:03.640 INFO:teuthology.orchestra.run.smithi070.stdout:average_utilization 0.0364 2024-09-21T10:51:03.640 INFO:teuthology.orchestra.run.smithi070.stdout:overload_utilization 0.0400 2024-09-21T10:51:03.641 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 weight 0.4399 -> 0.2946 2024-09-21T10:51:03.641 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 weight 0.6819 -> 0.5875 2024-09-21T10:51:03.641 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 weight 0.7775 -> 0.6659 2024-09-21T10:51:03.641 INFO:teuthology.orchestra.run.smithi070.stderr:no change 2024-09-21T10:51:03.660 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:03.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:03.702 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:03.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:03.803 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:03.826 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:03.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:03.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:03.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:51:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T10:51:03.633+0000 7fd088d1d640 -1 mgr.server reply reply (3) No such process no change 2024-09-21T10:51:03.927 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:03.972 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:03.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:03.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:04.074 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:04.107 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:04.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:04.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:04.209 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:04.270 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:04.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:04.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:04.277 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:03 smithi160 ceph-mon[101879]: pgmap v2610: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.6 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 457 op/s; 62105/275184 objects degraded (22.569%); 17411/275184 objects misplaced (6.327%); 8.7 MiB/s, 139 objects/s recovering 2024-09-21T10:51:04.277 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:03 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 62105/275184 objects degraded (22.569%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:04.371 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:04.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:03 smithi070 ceph-mon[107385]: pgmap v2610: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.6 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 457 op/s; 62105/275184 objects degraded (22.569%); 17411/275184 objects misplaced (6.327%); 8.7 MiB/s, 139 objects/s recovering 2024-09-21T10:51:04.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:03 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 62105/275184 objects degraded (22.569%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:04.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:03 smithi070 ceph-mon[112139]: pgmap v2610: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.6 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 457 op/s; 62105/275184 objects degraded (22.569%); 17411/275184 objects misplaced (6.327%); 8.7 MiB/s, 139 objects/s recovering 2024-09-21T10:51:04.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:03 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 62105/275184 objects degraded (22.569%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:04.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:04.407 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:04.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:04.508 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:04.535 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:04.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:04.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:04.636 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:04.678 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:04.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:04.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:04.783 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:04.838 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:04.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:04.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:04.939 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:04.992 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:04.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:04.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:05.093 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:05.122 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:05.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:05.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:05.224 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:05.258 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:05.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:05.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:05.297 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:04 smithi160 ceph-mon[101879]: from='client.35781 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:51:05.332 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:04 smithi070 ceph-mon[112139]: from='client.35781 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:51:05.332 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:04 smithi070 ceph-mon[107385]: from='client.35781 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:51:05.363 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:05.401 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:05.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:05.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:05.503 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:05.549 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:05.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:05.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:05.653 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:05.691 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:05.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:05.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:05.792 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:05.822 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:05.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:05.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:05.923 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:05.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:05.968 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:05.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:06.069 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:06.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:06.105 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:06.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:06.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:05 smithi070 ceph-mon[112139]: pgmap v2611: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.6 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 378 op/s; 62105/275184 objects degraded (22.569%); 17411/275184 objects misplaced (6.327%); 8.0 MiB/s, 127 objects/s recovering 2024-09-21T10:51:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:51:06.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:05 smithi070 ceph-mon[107385]: pgmap v2611: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.6 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 378 op/s; 62105/275184 objects degraded (22.569%); 17411/275184 objects misplaced (6.327%); 8.0 MiB/s, 127 objects/s recovering 2024-09-21T10:51:06.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:51:06.206 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:06.237 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:06.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:06.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:06.266 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:05 smithi160 ceph-mon[101879]: pgmap v2611: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.6 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 378 op/s; 62105/275184 objects degraded (22.569%); 17411/275184 objects misplaced (6.327%); 8.0 MiB/s, 127 objects/s recovering 2024-09-21T10:51:06.266 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:51:06.340 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:06.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:06.367 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:06.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:06.468 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:06.501 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:06.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:06.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:06.603 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:06.604 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:51:06.629 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:06.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:06.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:06.731 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:06.759 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:06.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:06.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:06.860 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:06.910 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:06.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:06.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:07.011 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:07.048 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:07.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:07.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:07.150 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:07.177 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:07.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:07.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:07.279 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:07.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:07.306 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:07.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:07.407 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:07.440 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:07.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:07.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:07.542 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:07.578 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:07.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:07.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:07.680 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:07.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:07.708 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:07.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:07.809 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:07.843 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:07.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:07.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:07.945 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:07.985 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:07.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:07.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:07.990 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:51:08.087 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:08.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:08.132 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:08.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:08.233 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:08.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:08.298 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:08.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:08.322 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:08.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:07 smithi160 ceph-mon[101879]: pgmap v2612: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 572 op/s; 61277/268200 objects degraded (22.848%); 16792/268200 objects misplaced (6.261%); 12 MiB/s, 189 objects/s recovering 2024-09-21T10:51:08.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:07 smithi070 ceph-mon[112139]: pgmap v2612: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 572 op/s; 61277/268200 objects degraded (22.848%); 16792/268200 objects misplaced (6.261%); 12 MiB/s, 189 objects/s recovering 2024-09-21T10:51:08.380 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:07 smithi070 ceph-mon[107385]: pgmap v2612: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 572 op/s; 61277/268200 objects degraded (22.848%); 16792/268200 objects misplaced (6.261%); 12 MiB/s, 189 objects/s recovering 2024-09-21T10:51:08.399 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:08.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:08.467 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:08.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:08.568 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:08.615 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:08.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:08.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:08.716 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:08.768 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:08.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:08.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:08.871 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:08.895 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:08.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:08.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:08.998 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:09.021 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:09.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:09.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:09.123 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:09.158 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:09.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:09.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:09.177 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:08 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 61277/268200 objects degraded (22.848%), 12 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:09.177 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:08 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:51:09.177 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:08 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:51:09.178 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:08 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2085760815' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:51:09.178 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:08 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:51:09.178 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:08 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 61277/268200 objects degraded (22.848%), 12 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:09.178 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:08 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:51:09.178 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:08 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:51:09.178 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:08 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2085760815' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:51:09.178 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:08 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:51:09.260 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:09.287 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:09.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:09.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:09.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:08 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 61277/268200 objects degraded (22.848%), 12 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:09.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:08 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:51:09.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:08 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:51:09.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:08 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2085760815' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:51:09.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:08 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:51:09.389 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:09.428 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:09.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:09.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:09.451 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T10:51:09.530 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:09.556 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:09.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:09.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:09.658 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:09.697 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:09.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:09.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:09.799 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:09.825 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:09.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:09.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:09.927 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:09.961 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:09.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:09.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:10.062 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:10.086 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:10.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:10.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:10.188 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:10.232 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:10.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:10.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:10.333 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:10.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:09 smithi160 ceph-mon[101879]: pgmap v2613: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 379 op/s; 61277/268200 objects degraded (22.848%); 16792/268200 objects misplaced (6.261%); 7.7 MiB/s, 123 objects/s recovering 2024-09-21T10:51:10.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:09 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:51:10.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:09 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:51:10.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:09 smithi160 ceph-mon[101879]: osdmap e1446: 8 total, 8 up, 6 in 2024-09-21T10:51:10.369 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:10.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:10.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:10.376 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:10.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:09 smithi070 ceph-mon[112139]: pgmap v2613: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 379 op/s; 61277/268200 objects degraded (22.848%); 16792/268200 objects misplaced (6.261%); 7.7 MiB/s, 123 objects/s recovering 2024-09-21T10:51:10.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:09 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:51:10.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:09 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:51:10.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:09 smithi070 ceph-mon[112139]: osdmap e1446: 8 total, 8 up, 6 in 2024-09-21T10:51:10.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:09 smithi070 ceph-mon[107385]: pgmap v2613: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 379 op/s; 61277/268200 objects degraded (22.848%); 16792/268200 objects misplaced (6.261%); 7.7 MiB/s, 123 objects/s recovering 2024-09-21T10:51:10.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:09 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:51:10.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:09 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:51:10.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:09 smithi070 ceph-mon[107385]: osdmap e1446: 8 total, 8 up, 6 in 2024-09-21T10:51:10.471 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:10.509 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:10.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:10.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:10.611 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:10.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:10.649 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:10.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:10.750 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:10.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:10.784 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:10.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:10.885 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:10.907 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:10.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:10.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:11.009 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:11.034 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:11.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:11.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:11.135 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:11.171 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:11.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:11.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:11.216 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 6, 7] out_osds: [2, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:51:11.217 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:51:11.217 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 3, 4, 6, 7] 2024-09-21T10:51:11.217 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd out 0 2024-09-21T10:51:11.273 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:11.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:11.346 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:11.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:11.446 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:11.469 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:11.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:11.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:11.570 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:11.613 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:11.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:11.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:11.715 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:11.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:11.768 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:11.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:11.868 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:11.893 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:11.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:11.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:11.995 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:12.023 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:12.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:12.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:12.122 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:11 smithi070 ceph-mon[107385]: pgmap v2615: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.4 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 468 op/s; 60718/264144 objects degraded (22.987%); 16129/264144 objects misplaced (6.106%); 9.5 MiB/s, 152 objects/s recovering 2024-09-21T10:51:12.122 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:11 smithi070 ceph-mon[112139]: pgmap v2615: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.4 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 468 op/s; 60718/264144 objects degraded (22.987%); 16129/264144 objects misplaced (6.106%); 9.5 MiB/s, 152 objects/s recovering 2024-09-21T10:51:12.125 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:12.150 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:12.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:12.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:12.251 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:12.311 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:12.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:12.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:12.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:11 smithi160 ceph-mon[101879]: pgmap v2615: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.4 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 468 op/s; 60718/264144 objects degraded (22.987%); 16129/264144 objects misplaced (6.106%); 9.5 MiB/s, 152 objects/s recovering 2024-09-21T10:51:12.413 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:12.443 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:12.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:12.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:12.544 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:12.588 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:12.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:12.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:12.690 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:12.726 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:12.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:12.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:12.748 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:51:12.749 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:51:12.749 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:51:12.749 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:51:12.749 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:51:12.749 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:51:12.749 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:51:12.749 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:51:12.749 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:51:12.749 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:51:12.749 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:51:12.749 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:51:12.750 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:51:12.750 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:51:12.750 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:51:12.750 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:51:12.828 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:12.855 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:12.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:12.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:12.884 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:51:12.957 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:12.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:12.999 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:12.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:13.100 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:13.133 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:13.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:13.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:13.235 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:13.263 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:13.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:13.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:13.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:12 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1225898492' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:51:13.365 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:13.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:12 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1225898492' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:51:13.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:12 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1225898492' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:51:13.378 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:51:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:51:13] "GET /metrics HTTP/1.1" 200 36671 "" "Prometheus/2.51.0" 2024-09-21T10:51:13.391 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:13.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:13.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:13.492 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:13.520 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:13.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:13.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:13.622 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:13.649 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:13.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:13.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:13.750 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:13.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:13.801 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:13.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:13.902 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:13.947 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:13.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:13.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:14.049 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:14.097 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:14.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:14.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:14.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:13 smithi070 ceph-mon[107385]: pgmap v2616: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.3 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 469 op/s; 60440/261084 objects degraded (23.150%); 16129/261084 objects misplaced (6.178%); 9.5 MiB/s, 152 objects/s recovering 2024-09-21T10:51:14.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:13 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 60440/261084 objects degraded (23.150%), 12 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:14.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:13 smithi070 ceph-mon[112139]: pgmap v2616: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.3 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 469 op/s; 60440/261084 objects degraded (23.150%); 16129/261084 objects misplaced (6.178%); 9.5 MiB/s, 152 objects/s recovering 2024-09-21T10:51:14.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:13 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 60440/261084 objects degraded (23.150%), 12 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:14.199 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:14.227 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:14.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:14.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:14.329 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:13 smithi160 ceph-mon[101879]: pgmap v2616: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.3 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 469 op/s; 60440/261084 objects degraded (23.150%); 16129/261084 objects misplaced (6.178%); 9.5 MiB/s, 152 objects/s recovering 2024-09-21T10:51:14.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:13 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 60440/261084 objects degraded (23.150%), 12 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:14.355 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:14.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:14.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:14.457 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:14.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:14.504 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:14.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:14.605 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:14.632 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:14.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:14.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:14.734 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:14.770 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:14.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:14.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:14.872 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:14.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:14.899 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:14.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:15.000 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:15.035 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:15.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:15.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:15.137 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:15.185 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:15.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:15.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:15.287 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:15.324 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:15.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:15.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:15.425 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:15.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:15.452 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:15.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:15.552 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:15.591 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:15.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:15.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:15.692 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:15.720 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:15.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:15.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:15.819 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:15.822 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:15.849 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:15.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:15.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:15.950 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:16.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:16.001 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:16.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:16.036 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:16 smithi070 ceph-mon[107385]: pgmap v2617: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.3 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 469 op/s; 60440/261084 objects degraded (23.150%); 16129/261084 objects misplaced (6.178%); 9.5 MiB/s, 152 objects/s recovering 2024-09-21T10:51:16.036 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:16 smithi070 ceph-mon[112139]: pgmap v2617: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.3 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 469 op/s; 60440/261084 objects degraded (23.150%); 16129/261084 objects misplaced (6.178%); 9.5 MiB/s, 152 objects/s recovering 2024-09-21T10:51:16.097 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:51:16.097 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T10:51:16.097 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.097 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:126157 55.a 55:51c69b65:::benchmark_data_smithi070_294394_object20816:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:51:16.097 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:16.067503+0000", 2024-09-21T10:51:16.097 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.025922251, 2024-09-21T10:51:16.097 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.025955112999999998, 2024-09-21T10:51:16.098 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:51:16.098 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:51:16.098 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:51:16.098 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:51:16.098 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:51:16.098 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:51:16.098 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 126157 2024-09-21T10:51:16.098 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.098 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:51:16.098 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.098 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:51:16.098 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.067503+0000", 2024-09-21T10:51:16.099 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:51:16.099 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.099 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.099 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:51:16.099 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.067503+0000", 2024-09-21T10:51:16.099 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:51:16.099 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.099 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.099 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:51:16.099 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.067506+0000", 2024-09-21T10:51:16.099 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.96e-06 2024-09-21T10:51:16.099 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.099 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.100 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:51:16.100 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.067517+0000", 2024-09-21T10:51:16.100 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1345999999999999e-05 2024-09-21T10:51:16.100 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.100 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.100 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:51:16.100 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.067534+0000", 2024-09-21T10:51:16.100 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6385e-05 2024-09-21T10:51:16.100 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.100 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.100 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:51:16.100 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.067545+0000", 2024-09-21T10:51:16.101 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1388e-05 2024-09-21T10:51:16.101 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.101 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.101 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:51:16.101 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.078510+0000", 2024-09-21T10:51:16.101 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.010964422 2024-09-21T10:51:16.101 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.101 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.101 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:51:16.101 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.078675+0000", 2024-09-21T10:51:16.101 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016524000000000001 2024-09-21T10:51:16.101 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.101 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.102 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:51:16.102 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.081162+0000", 2024-09-21T10:51:16.102 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0024873510000000001 2024-09-21T10:51:16.102 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.102 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.102 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:51:16.102 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.084160+0000", 2024-09-21T10:51:16.102 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0029974120000000001 2024-09-21T10:51:16.102 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:51:16.102 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:51:16.102 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:51:16.102 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.102 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.103 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:126155 55.d e1446/1440 55:b7f3a0e6:::benchmark_data_smithi070_294394_object20814:head v 1446'8060, mlcod=0'0)", 2024-09-21T10:51:16.103 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:16.068179+0000", 2024-09-21T10:51:16.103 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.025245917, 2024-09-21T10:51:16.103 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.025352398000000002, 2024-09-21T10:51:16.103 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:51:16.103 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:51:16.103 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "started", 2024-09-21T10:51:16.103 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:51:16.103 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.103 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:51:16.103 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.068179+0000", 2024-09-21T10:51:16.103 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:51:16.103 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.104 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.104 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:51:16.104 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.068179+0000", 2024-09-21T10:51:16.104 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:51:16.104 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.104 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.104 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:51:16.104 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.068181+0000", 2024-09-21T10:51:16.104 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.114e-06 2024-09-21T10:51:16.104 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.104 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.104 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:51:16.104 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.068189+0000", 2024-09-21T10:51:16.104 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.4279999999999996e-06 2024-09-21T10:51:16.105 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.105 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.105 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:51:16.105 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.068200+0000", 2024-09-21T10:51:16.105 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0974e-05 2024-09-21T10:51:16.105 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.105 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.105 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:51:16.105 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.068206+0000", 2024-09-21T10:51:16.105 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3799999999999999e-06 2024-09-21T10:51:16.105 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.105 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.105 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:51:16.105 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.078736+0000", 2024-09-21T10:51:16.106 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.010529192 2024-09-21T10:51:16.106 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.106 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.106 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:51:16.106 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.078758+0000", 2024-09-21T10:51:16.106 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2461999999999999e-05 2024-09-21T10:51:16.106 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:51:16.106 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:51:16.106 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:51:16.106 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.106 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.107 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:126158 55.9 55:974b4c35:::benchmark_data_smithi070_294394_object20817:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:51:16.107 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:16.076283+0000", 2024-09-21T10:51:16.107 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.017142503, 2024-09-21T10:51:16.107 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.017299802, 2024-09-21T10:51:16.107 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:51:16.107 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:51:16.107 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:51:16.107 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:51:16.107 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:51:16.107 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:51:16.107 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 126158 2024-09-21T10:51:16.107 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.107 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:51:16.108 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.108 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:51:16.108 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.076283+0000", 2024-09-21T10:51:16.108 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:51:16.108 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.108 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.108 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:51:16.108 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.076283+0000", 2024-09-21T10:51:16.108 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:51:16.108 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.108 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.108 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:51:16.108 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.076286+0000", 2024-09-21T10:51:16.108 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7949999999999999e-06 2024-09-21T10:51:16.109 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.109 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.109 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:51:16.109 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.076297+0000", 2024-09-21T10:51:16.109 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1705000000000001e-05 2024-09-21T10:51:16.109 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.109 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.109 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:51:16.109 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.076317+0000", 2024-09-21T10:51:16.109 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9681e-05 2024-09-21T10:51:16.109 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.109 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.110 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:51:16.110 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.076329+0000", 2024-09-21T10:51:16.110 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1977999999999999e-05 2024-09-21T10:51:16.110 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.110 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.110 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:51:16.110 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.076660+0000", 2024-09-21T10:51:16.110 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00033108900000000001 2024-09-21T10:51:16.110 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.110 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.110 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:51:16.110 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.076811+0000", 2024-09-21T10:51:16.111 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000150989 2024-09-21T10:51:16.111 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.111 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.111 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:51:16.111 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.076961+0000", 2024-09-21T10:51:16.111 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000149844 2024-09-21T10:51:16.111 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.111 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.111 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:51:16.111 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.086376+0000", 2024-09-21T10:51:16.111 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0094149390000000006 2024-09-21T10:51:16.111 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.111 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.111 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:51:16.112 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.086425+0000", 2024-09-21T10:51:16.112 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8853999999999997e-05 2024-09-21T10:51:16.112 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:51:16.112 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:51:16.112 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:51:16.112 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.112 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.112 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:16.113 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:126161 55.a 55:52e33297:::benchmark_data_smithi070_294394_object20820:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:51:16.113 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:16.085335+0000", 2024-09-21T10:51:16.113 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0080904219999999999, 2024-09-21T10:51:16.113 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0083177600000000004, 2024-09-21T10:51:16.114 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:51:16.114 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:51:16.114 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:51:16.114 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:51:16.114 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:51:16.114 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:51:16.114 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 126161 2024-09-21T10:51:16.114 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.114 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:51:16.114 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.114 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:51:16.114 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.085335+0000", 2024-09-21T10:51:16.114 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:51:16.115 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.115 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.115 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:51:16.115 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.085335+0000", 2024-09-21T10:51:16.115 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:51:16.115 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.115 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.115 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:51:16.115 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.085336+0000", 2024-09-21T10:51:16.115 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5090000000000001e-06 2024-09-21T10:51:16.115 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.115 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.115 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:51:16.116 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.085340+0000", 2024-09-21T10:51:16.116 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9190000000000001e-06 2024-09-21T10:51:16.116 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.116 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.116 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:51:16.116 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.085350+0000", 2024-09-21T10:51:16.116 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3780000000000007e-06 2024-09-21T10:51:16.116 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.116 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.116 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:51:16.116 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.085356+0000", 2024-09-21T10:51:16.116 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2360000000000001e-06 2024-09-21T10:51:16.116 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.116 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.117 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:51:16.117 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.085696+0000", 2024-09-21T10:51:16.117 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00033956999999999998 2024-09-21T10:51:16.117 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.117 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.117 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:51:16.117 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.086200+0000", 2024-09-21T10:51:16.117 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000503993 2024-09-21T10:51:16.117 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.117 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.117 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:51:16.117 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.086496+0000", 2024-09-21T10:51:16.117 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00029679499999999998 2024-09-21T10:51:16.117 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:51:16.118 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:51:16.118 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:51:16.118 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.118 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.118 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:126138 55.8 e1446/1428 55:164b4fac:::benchmark_data_smithi070_294394_object20797:head v 1446'7801, mlcod=1415'89)", 2024-09-21T10:51:16.118 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:16.035381+0000", 2024-09-21T10:51:16.118 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.058044336000000002, 2024-09-21T10:51:16.118 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.058329639000000003, 2024-09-21T10:51:16.118 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:51:16.118 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:51:16.118 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "started", 2024-09-21T10:51:16.118 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:51:16.118 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.119 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:51:16.119 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.035381+0000", 2024-09-21T10:51:16.119 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:51:16.119 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.119 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.119 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:51:16.119 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.035381+0000", 2024-09-21T10:51:16.119 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:51:16.119 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.119 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.119 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:51:16.119 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.035382+0000", 2024-09-21T10:51:16.119 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.47e-07 2024-09-21T10:51:16.120 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.120 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.120 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:51:16.120 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.035388+0000", 2024-09-21T10:51:16.120 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5890000000000002e-06 2024-09-21T10:51:16.120 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.120 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.120 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:51:16.120 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.035400+0000", 2024-09-21T10:51:16.120 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2806000000000001e-05 2024-09-21T10:51:16.120 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.120 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.120 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:51:16.121 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.035408+0000", 2024-09-21T10:51:16.121 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0150000000000003e-06 2024-09-21T10:51:16.121 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.121 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.121 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:51:16.121 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.035609+0000", 2024-09-21T10:51:16.121 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00020038200000000001 2024-09-21T10:51:16.121 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.121 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.121 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:51:16.121 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.035633+0000", 2024-09-21T10:51:16.121 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3864000000000001e-05 2024-09-21T10:51:16.121 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:51:16.121 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:51:16.122 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:51:16.122 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.122 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.122 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:126163 55.0 55:0b2b2937:::benchmark_data_smithi070_294394_object20822:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:51:16.122 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:16.086048+0000", 2024-09-21T10:51:16.122 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.007377761, 2024-09-21T10:51:16.122 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0077118179999999996, 2024-09-21T10:51:16.122 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:51:16.122 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:51:16.122 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:51:16.122 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:51:16.122 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:51:16.122 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:51:16.123 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 126163 2024-09-21T10:51:16.123 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.123 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:51:16.123 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.123 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:51:16.123 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.086048+0000", 2024-09-21T10:51:16.123 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:51:16.123 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.123 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.123 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:51:16.123 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.086048+0000", 2024-09-21T10:51:16.123 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:51:16.123 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.123 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.124 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:51:16.124 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.086049+0000", 2024-09-21T10:51:16.124 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.6200000000000006e-07 2024-09-21T10:51:16.124 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.124 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.124 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:51:16.124 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.086053+0000", 2024-09-21T10:51:16.124 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7759999999999997e-06 2024-09-21T10:51:16.124 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.124 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.124 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:51:16.124 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.086063+0000", 2024-09-21T10:51:16.124 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.8500000000000006e-06 2024-09-21T10:51:16.125 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.125 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.125 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:51:16.125 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.086070+0000", 2024-09-21T10:51:16.125 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.387e-06 2024-09-21T10:51:16.125 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.125 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.125 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:51:16.125 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.088483+0000", 2024-09-21T10:51:16.125 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0024132279999999999 2024-09-21T10:51:16.125 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.125 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.125 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:51:16.125 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.088663+0000", 2024-09-21T10:51:16.125 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018024100000000001 2024-09-21T10:51:16.126 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.126 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.126 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:51:16.126 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.088855+0000", 2024-09-21T10:51:16.126 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000191838 2024-09-21T10:51:16.126 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:51:16.126 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:51:16.126 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:51:16.126 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.126 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.126 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:126160 55.8 e1446/1428 55:19aa9eb6:::benchmark_data_smithi070_294394_object20819:head v 1446'7803, mlcod=1415'89)", 2024-09-21T10:51:16.126 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:16.086691+0000", 2024-09-21T10:51:16.127 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0067348699999999996, 2024-09-21T10:51:16.127 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.007125541, 2024-09-21T10:51:16.127 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:51:16.127 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:51:16.127 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "started", 2024-09-21T10:51:16.127 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:51:16.127 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.127 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:51:16.127 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.086691+0000", 2024-09-21T10:51:16.127 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:51:16.127 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.127 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.127 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:51:16.127 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.086691+0000", 2024-09-21T10:51:16.128 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:51:16.128 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.128 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.128 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:51:16.128 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.086692+0000", 2024-09-21T10:51:16.128 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.443e-06 2024-09-21T10:51:16.128 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.128 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.128 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:51:16.128 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.086699+0000", 2024-09-21T10:51:16.128 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7580000000000003e-06 2024-09-21T10:51:16.128 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.128 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.129 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:51:16.129 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.086709+0000", 2024-09-21T10:51:16.129 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0227999999999999e-05 2024-09-21T10:51:16.129 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.129 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.129 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:51:16.129 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.086716+0000", 2024-09-21T10:51:16.129 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1189999999999999e-06 2024-09-21T10:51:16.129 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.129 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.129 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:51:16.129 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.092211+0000", 2024-09-21T10:51:16.129 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0054947010000000003 2024-09-21T10:51:16.130 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.130 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.130 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:51:16.130 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.092235+0000", 2024-09-21T10:51:16.130 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4226000000000001e-05 2024-09-21T10:51:16.130 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:51:16.130 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:51:16.130 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:51:16.130 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.130 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.130 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "MOSDPGPushReply(55.9 1446/1428 [PushReplyOp(55:965c8777:::benchmark_data_smithi070_294394_object38547:head)])", 2024-09-21T10:51:16.130 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:16.093522+0000", 2024-09-21T10:51:16.130 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 4294967295.9999037, 2024-09-21T10:51:16.131 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00034359100000000002, 2024-09-21T10:51:16.131 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:51:16.131 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:51:16.131 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "queued for pg", 2024-09-21T10:51:16.131 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:51:16.131 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.131 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:51:16.131 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.093522+0000", 2024-09-21T10:51:16.131 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:51:16.131 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.131 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.131 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:51:16.131 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.093522+0000", 2024-09-21T10:51:16.132 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:51:16.132 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.132 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.132 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:51:16.132 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.093523+0000", 2024-09-21T10:51:16.132 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.023e-06 2024-09-21T10:51:16.132 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.132 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.132 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:51:16.132 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.093528+0000", 2024-09-21T10:51:16.132 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0470000000000001e-06 2024-09-21T10:51:16.132 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.132 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.133 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:51:16.133 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.093541+0000", 2024-09-21T10:51:16.133 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2841e-05 2024-09-21T10:51:16.133 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.133 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.133 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:51:16.133 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.093549+0000", 2024-09-21T10:51:16.133 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.4679999999999992e-06 2024-09-21T10:51:16.133 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:51:16.133 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:51:16.133 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:51:16.133 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.133 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.134 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:126164 55.0 55.27400a00 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:51:16.134 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:16.093587+0000", 2024-09-21T10:51:16.134 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 4294967295.9998384, 2024-09-21T10:51:16.134 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00032305300000000001, 2024-09-21T10:51:16.134 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:51:16.134 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:51:16.134 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "queued for pg", 2024-09-21T10:51:16.134 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:51:16.134 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:51:16.134 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:51:16.134 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 126164 2024-09-21T10:51:16.134 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.134 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:51:16.134 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.135 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:51:16.135 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.093587+0000", 2024-09-21T10:51:16.135 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:51:16.135 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.135 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.135 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:51:16.135 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.093587+0000", 2024-09-21T10:51:16.135 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:51:16.135 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.135 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.135 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:51:16.135 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.093588+0000", 2024-09-21T10:51:16.135 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.375e-06 2024-09-21T10:51:16.135 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.136 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.136 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:51:16.136 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.093592+0000", 2024-09-21T10:51:16.136 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2520000000000001e-06 2024-09-21T10:51:16.136 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.136 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.136 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:51:16.136 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.093602+0000", 2024-09-21T10:51:16.136 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.302e-06 2024-09-21T10:51:16.136 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.136 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.136 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:51:16.136 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.093607+0000", 2024-09-21T10:51:16.137 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3329999999999996e-06 2024-09-21T10:51:16.137 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:51:16.137 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:51:16.137 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:51:16.137 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.137 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.137 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:126165 55.c 55.793383ac (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:51:16.137 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:16.093617+0000", 2024-09-21T10:51:16.137 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 4294967295.9998083, 2024-09-21T10:51:16.137 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00033681599999999998, 2024-09-21T10:51:16.137 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:51:16.137 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:51:16.137 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "queued for pg", 2024-09-21T10:51:16.137 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:51:16.138 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:51:16.138 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:51:16.138 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 126165 2024-09-21T10:51:16.138 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.138 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:51:16.138 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.138 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:51:16.138 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.093617+0000", 2024-09-21T10:51:16.138 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:51:16.138 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.138 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.138 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:51:16.138 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.093617+0000", 2024-09-21T10:51:16.138 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:51:16.139 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.139 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.139 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:51:16.139 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.093618+0000", 2024-09-21T10:51:16.139 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8700000000000001e-07 2024-09-21T10:51:16.139 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.139 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.139 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:51:16.139 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.093619+0000", 2024-09-21T10:51:16.139 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7799999999999999e-06 2024-09-21T10:51:16.139 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.139 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.139 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:51:16.140 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.093625+0000", 2024-09-21T10:51:16.140 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7350000000000001e-06 2024-09-21T10:51:16.140 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.140 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.140 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:51:16.140 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.093629+0000", 2024-09-21T10:51:16.140 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2599999999999999e-06 2024-09-21T10:51:16.140 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:51:16.140 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:51:16.140 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:51:16.140 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.140 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.140 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop_reply(client.35670.0:126163 55.0 e1446/1434)", 2024-09-21T10:51:16.141 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:16.093664+0000", 2024-09-21T10:51:16.141 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 4294967295.9997616, 2024-09-21T10:51:16.141 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00032928100000000001, 2024-09-21T10:51:16.141 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:51:16.141 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:51:16.141 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "queued for pg", 2024-09-21T10:51:16.141 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:51:16.141 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.141 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:51:16.141 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.093664+0000", 2024-09-21T10:51:16.141 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:51:16.141 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.141 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.141 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:51:16.142 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.093664+0000", 2024-09-21T10:51:16.142 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:51:16.142 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.142 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.142 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:51:16.142 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.093664+0000", 2024-09-21T10:51:16.142 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1099999999999995e-07 2024-09-21T10:51:16.142 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.142 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.142 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:51:16.142 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.093667+0000", 2024-09-21T10:51:16.142 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.289e-06 2024-09-21T10:51:16.142 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.142 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.143 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:51:16.143 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.093673+0000", 2024-09-21T10:51:16.143 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6730000000000003e-06 2024-09-21T10:51:16.143 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.143 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.143 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:51:16.143 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.093678+0000", 2024-09-21T10:51:16.143 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5140000000000003e-06 2024-09-21T10:51:16.143 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:51:16.143 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:51:16.143 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:51:16.143 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.143 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.144 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "MOSDPGPush(55.8 1446/1428 [PushOp(55:12178092:::benchmark_data_smithi070_294394_object33642:head, version: 1416'2082, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(55:12178092:::benchmark_data_smithi070_294394_object33642:head@1416'2082, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T10:51:16.144 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:16.062585+0000", 2024-09-21T10:51:16.144 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.030840528999999998, 2024-09-21T10:51:16.144 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.031480413999999998, 2024-09-21T10:51:16.144 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:51:16.144 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:51:16.144 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "started", 2024-09-21T10:51:16.144 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:51:16.144 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.144 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:51:16.144 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.062585+0000", 2024-09-21T10:51:16.144 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:51:16.144 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.144 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.145 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:51:16.145 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.062585+0000", 2024-09-21T10:51:16.145 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:51:16.145 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.145 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.145 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:51:16.145 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.062586+0000", 2024-09-21T10:51:16.145 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5900000000000005e-07 2024-09-21T10:51:16.145 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.145 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.145 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:51:16.145 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.062617+0000", 2024-09-21T10:51:16.145 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1541000000000002e-05 2024-09-21T10:51:16.145 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.146 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.146 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:51:16.146 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.062638+0000", 2024-09-21T10:51:16.146 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0205000000000001e-05 2024-09-21T10:51:16.146 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.146 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.146 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:51:16.146 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.062653+0000", 2024-09-21T10:51:16.146 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5403000000000001e-05 2024-09-21T10:51:16.146 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.146 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.146 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:51:16.146 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.080450+0000", 2024-09-21T10:51:16.146 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.017796640999999998 2024-09-21T10:51:16.147 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.147 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.147 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:51:16.147 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.080464+0000", 2024-09-21T10:51:16.147 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4079e-05 2024-09-21T10:51:16.147 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:51:16.147 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:51:16.147 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:51:16.147 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.147 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.147 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:126152 55.8 e1446/1428 55:1ee05b85:::benchmark_data_smithi070_294394_object20811:head v 1446'7802, mlcod=1415'89)", 2024-09-21T10:51:16.147 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:16.064645+0000", 2024-09-21T10:51:16.147 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.028780047999999999, 2024-09-21T10:51:16.148 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.029466306000000001, 2024-09-21T10:51:16.148 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:51:16.148 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:51:16.148 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "started", 2024-09-21T10:51:16.148 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:51:16.148 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.148 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:51:16.148 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.064645+0000", 2024-09-21T10:51:16.148 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:51:16.148 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.148 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.148 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:51:16.148 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.064645+0000", 2024-09-21T10:51:16.148 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.064646+0000", 2024-09-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1799999999999995e-07 2024-09-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.064652+0000", 2024-09-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.789e-06 2024-09-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.064660+0000", 2024-09-21T10:51:16.150 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.4540000000000007e-06 2024-09-21T10:51:16.150 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.150 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.150 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:51:16.150 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.064665+0000", 2024-09-21T10:51:16.150 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3580000000000002e-06 2024-09-21T10:51:16.150 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.150 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.150 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:51:16.150 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.068615+0000", 2024-09-21T10:51:16.150 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0039497969999999997 2024-09-21T10:51:16.150 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:51:16.150 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:51:16.151 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:51:16.151 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:16.068638+0000", 2024-09-21T10:51:16.151 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2812000000000001e-05 2024-09-21T10:51:16.151 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:51:16.151 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:51:16.151 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:51:16.151 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:51:16.151 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-09-21T10:51:16.151 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 13 2024-09-21T10:51:16.151 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:51:16.177 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:16.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:16.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:16.279 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:16.305 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:16.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:16.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:16.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:16 smithi160 ceph-mon[101879]: pgmap v2617: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.3 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 469 op/s; 60440/261084 objects degraded (23.150%); 16129/261084 objects misplaced (6.178%); 9.5 MiB/s, 152 objects/s recovering 2024-09-21T10:51:16.407 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:16.434 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:16.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:16.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:16.536 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:16.573 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:16.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:16.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:16.675 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:16.709 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:16.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:16.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:16.811 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:16.838 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:16.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:16.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:16.940 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:16.985 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:16.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:16.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:17.087 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:17.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:17.127 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:17.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:17.228 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:17.254 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:17.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:17.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:17.356 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:17.383 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:17.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:17.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:17.484 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:17.489 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:51:17.523 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:17.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:17.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:17.624 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:17.648 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:17.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:17.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:17.749 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:17.776 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:17.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:17.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:17.878 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:17.901 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:17.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:17.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:18.003 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:18.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:18.048 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:18.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:18.149 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:18.191 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:18.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:18.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:18.295 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:18.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:18.334 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:18.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:18 smithi160 ceph-mon[101879]: pgmap v2618: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.2 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 485 op/s; 59486/253614 objects degraded (23.455%); 15447/253614 objects misplaced (6.091%); 10 MiB/s, 163 objects/s recovering 2024-09-21T10:51:18.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:18 smithi070 ceph-mon[107385]: pgmap v2618: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.2 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 485 op/s; 59486/253614 objects degraded (23.455%); 15447/253614 objects misplaced (6.091%); 10 MiB/s, 163 objects/s recovering 2024-09-21T10:51:18.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:18 smithi070 ceph-mon[112139]: pgmap v2618: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.2 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 485 op/s; 59486/253614 objects degraded (23.455%); 15447/253614 objects misplaced (6.091%); 10 MiB/s, 163 objects/s recovering 2024-09-21T10:51:18.435 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:18.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:18.463 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:18.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:18.564 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:18.600 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:18.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:18.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:18.702 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:18.741 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:18.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:18.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:18.842 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:18.879 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:18.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:18.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:18.981 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:19.007 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:19.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:19.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:19.109 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:19.152 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:19.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:19.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:19.254 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:19.277 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:19.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:19.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:19.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:19 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 59486/253614 objects degraded (23.455%), 12 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:19.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:19 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 59486/253614 objects degraded (23.455%), 12 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:19.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:19 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 59486/253614 objects degraded (23.455%), 12 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:19.378 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:19.406 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:19.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:19.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:19.508 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:19.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:19.548 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:19.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:19.649 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:19.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:19.690 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:19.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:19.791 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:19.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:19.813 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:19.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:19.914 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:19.942 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:19.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:19.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:20.044 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:20.071 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:20.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:20.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:20.172 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:20.198 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:20.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:20.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:20.302 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:20.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:20 smithi160 ceph-mon[101879]: pgmap v2619: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.2 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 485 op/s; 59486/253614 objects degraded (23.455%); 15447/253614 objects misplaced (6.091%); 10 MiB/s, 163 objects/s recovering 2024-09-21T10:51:20.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:20.343 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:20.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:20.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:20 smithi070 ceph-mon[107385]: pgmap v2619: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.2 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 485 op/s; 59486/253614 objects degraded (23.455%); 15447/253614 objects misplaced (6.091%); 10 MiB/s, 163 objects/s recovering 2024-09-21T10:51:20.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:20 smithi070 ceph-mon[112139]: pgmap v2619: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.2 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 485 op/s; 59486/253614 objects degraded (23.455%); 15447/253614 objects misplaced (6.091%); 10 MiB/s, 163 objects/s recovering 2024-09-21T10:51:20.444 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:20.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:20.473 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:20.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:20.574 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:20.600 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:20.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:20.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:20.702 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:20.737 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:20.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:20.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:20.839 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:20.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:20.873 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:20.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:20.974 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:21.000 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:21.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:21.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:21.102 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:21.134 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:21.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:21.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:21.236 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:21.270 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:21.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:21.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:21.371 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:21.396 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:21.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:21.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:21.497 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:21.524 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:21.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:21.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:21.626 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:21.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:51:21.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:51:21.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:51:21.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:51:21.655 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:21.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:21.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:21.756 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:21.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:21.794 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:21.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:51:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:51:21.894 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:21.939 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:21.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:21.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:22.041 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:22.068 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:22.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:22.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:22.170 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:22.204 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:22.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:22.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:22.305 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:22.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:22.340 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:22.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:22.441 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:22.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:22.479 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:22.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:22.579 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:22.601 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:22.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:22.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:22.702 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:22.728 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:22.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:22.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:22.830 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:22 smithi160 ceph-mon[101879]: pgmap v2620: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 475 op/s; 58912/251886 objects degraded (23.388%); 15128/251886 objects misplaced (6.006%); 11 MiB/s, 182 objects/s recovering 2024-09-21T10:51:22.851 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:22.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:22.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:22.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:22 smithi070 ceph-mon[107385]: pgmap v2620: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 475 op/s; 58912/251886 objects degraded (23.388%); 15128/251886 objects misplaced (6.006%); 11 MiB/s, 182 objects/s recovering 2024-09-21T10:51:22.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:22 smithi070 ceph-mon[112139]: pgmap v2620: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 475 op/s; 58912/251886 objects degraded (23.388%); 15128/251886 objects misplaced (6.006%); 11 MiB/s, 182 objects/s recovering 2024-09-21T10:51:22.952 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:22.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:22.994 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:22.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:23.095 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:23.131 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:23.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:23.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:23.233 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:23.267 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:23.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:23.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:23.369 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:23.406 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:23 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 58912/251886 objects degraded (23.388%), 12 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:23.406 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:51:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:51:23] "GET /metrics HTTP/1.1" 200 36673 "" "Prometheus/2.51.0" 2024-09-21T10:51:23.411 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:23.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:23.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:23.512 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:23.537 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:23.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:23.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:23.639 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:23.665 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:23.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:23.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:23.766 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:23.807 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:23.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:23.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:23 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 58912/251886 objects degraded (23.388%), 12 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:23.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:23 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 58912/251886 objects degraded (23.388%), 12 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:23.908 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:23.934 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:23.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:23.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:24.036 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:24.062 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:24.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:24.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:24.164 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:24.191 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:24.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:24.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:24.288 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:51:24.292 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:24.366 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:24.389 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:24.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:24.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:24.491 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:24.563 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:24.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:24.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:24.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:24 smithi070 ceph-mon[112139]: pgmap v2621: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 489 op/s; 58588/246528 objects degraded (23.765%); 14779/246528 objects misplaced (5.995%); 9.5 MiB/s, 151 objects/s recovering 2024-09-21T10:51:24.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:24 smithi070 ceph-mon[107385]: pgmap v2621: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 489 op/s; 58588/246528 objects degraded (23.765%); 14779/246528 objects misplaced (5.995%); 9.5 MiB/s, 151 objects/s recovering 2024-09-21T10:51:24.665 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:24.709 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:24.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:24.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:24.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:24 smithi160 ceph-mon[101879]: pgmap v2621: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 489 op/s; 58588/246528 objects degraded (23.765%); 14779/246528 objects misplaced (5.995%); 9.5 MiB/s, 151 objects/s recovering 2024-09-21T10:51:24.811 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:24.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:24.872 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:24.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:24.973 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:25.007 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:25.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:25.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:25.108 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:25.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:25.150 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:25.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:25.251 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:25.280 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:25.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:25.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:25.382 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:25.427 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:25.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:25.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:25.528 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:25.554 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:25.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:25.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:25.656 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:25.681 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:25.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:25.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:25.782 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:25.809 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:25.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:25.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:25.910 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:25.934 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:25.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:25.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:26.036 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:26.107 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:26.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:26.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:26.208 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:26.235 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:26.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:26.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:26.337 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:26.361 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:26.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:26.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:26.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:26 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a[107362]: 2024-09-21T10:51:26.113+0000 7fad4a465640 -1 mon.a@0(leader).osd e1446 definitely_dead 0 2024-09-21T10:51:26.418 INFO:teuthology.orchestra.run.smithi070.stderr:marked out osd.0. 2024-09-21T10:51:26.463 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:26.492 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:26.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:26.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:26.594 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:26.616 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:26.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:26.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:26.718 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:26.739 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:26.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:26.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:26.840 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:26 smithi160 ceph-mon[101879]: pgmap v2622: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 404 op/s; 58588/246528 objects degraded (23.765%); 14779/246528 objects misplaced (5.995%); 8.7 MiB/s, 139 objects/s recovering 2024-09-21T10:51:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:26 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3549153076' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-09-21T10:51:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:26 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-09-21T10:51:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:26 smithi160 ceph-mon[101879]: Client client.admin marked osd.0 out, while it was still marked up 2024-09-21T10:51:26.862 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:26.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:26.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:26.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:26 smithi070 ceph-mon[107385]: pgmap v2622: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 404 op/s; 58588/246528 objects degraded (23.765%); 14779/246528 objects misplaced (5.995%); 8.7 MiB/s, 139 objects/s recovering 2024-09-21T10:51:26.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:26 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3549153076' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-09-21T10:51:26.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:26 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-09-21T10:51:26.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:26 smithi070 ceph-mon[107385]: Client client.admin marked osd.0 out, while it was still marked up 2024-09-21T10:51:26.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:26 smithi070 ceph-mon[112139]: pgmap v2622: 99 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 83 active+clean; 5.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 404 op/s; 58588/246528 objects degraded (23.765%); 14779/246528 objects misplaced (5.995%); 8.7 MiB/s, 139 objects/s recovering 2024-09-21T10:51:26.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:26 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3549153076' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-09-21T10:51:26.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:26 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-09-21T10:51:26.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:26 smithi070 ceph-mon[112139]: Client client.admin marked osd.0 out, while it was still marked up 2024-09-21T10:51:26.964 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:26.986 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:26.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:26.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:27.088 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:27.109 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:27.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:27.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:27.211 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:27.234 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:27.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:27.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:27.336 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:27.359 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:27.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:27.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:27.460 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:27.482 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:27.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:27.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:27.584 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:27.605 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:27.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:27.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:27.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:27 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-09-21T10:51:27.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:27 smithi070 ceph-mon[107385]: osdmap e1447: 8 total, 8 up, 5 in 2024-09-21T10:51:27.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:27 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-09-21T10:51:27.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:27 smithi070 ceph-mon[112139]: osdmap e1447: 8 total, 8 up, 5 in 2024-09-21T10:51:27.707 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:27.728 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:27.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:27.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:27.830 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:27 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-09-21T10:51:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:27 smithi160 ceph-mon[101879]: osdmap e1447: 8 total, 8 up, 5 in 2024-09-21T10:51:27.851 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:27.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:27.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:27.953 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:27.974 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:27.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:27.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:28.075 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:28.097 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:28.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:28.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:28.198 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:28.220 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:28.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:28.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:28.321 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:28.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:28.344 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:28.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:28.445 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:28.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:28.476 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:28.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:28.577 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:28.606 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:28.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:28.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:28.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:28 smithi070 ceph-mon[107385]: pgmap v2624: 99 pgs: 8 peering, 2 remapped+peering, 1 unknown, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 74 active+clean; 4.9 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 452 op/s; 39270/240021 objects degraded (16.361%); 21979/240021 objects misplaced (9.157%); 10 MiB/s, 162 objects/s recovering 2024-09-21T10:51:28.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:28 smithi070 ceph-mon[107385]: osdmap e1448: 8 total, 8 up, 5 in 2024-09-21T10:51:28.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:28 smithi070 ceph-mon[107385]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:51:28.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:28 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 39270/240021 objects degraded (16.361%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:28.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:28 smithi070 ceph-mon[107385]: osdmap e1449: 8 total, 8 up, 5 in 2024-09-21T10:51:28.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:28 smithi070 ceph-mon[112139]: pgmap v2624: 99 pgs: 8 peering, 2 remapped+peering, 1 unknown, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 74 active+clean; 4.9 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 452 op/s; 39270/240021 objects degraded (16.361%); 21979/240021 objects misplaced (9.157%); 10 MiB/s, 162 objects/s recovering 2024-09-21T10:51:28.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:28 smithi070 ceph-mon[112139]: osdmap e1448: 8 total, 8 up, 5 in 2024-09-21T10:51:28.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:28 smithi070 ceph-mon[112139]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:51:28.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:28 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 39270/240021 objects degraded (16.361%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:28.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:28 smithi070 ceph-mon[112139]: osdmap e1449: 8 total, 8 up, 5 in 2024-09-21T10:51:28.708 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:28.743 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:28.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:28.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:28 smithi160 ceph-mon[101879]: pgmap v2624: 99 pgs: 8 peering, 2 remapped+peering, 1 unknown, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 74 active+clean; 4.9 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 452 op/s; 39270/240021 objects degraded (16.361%); 21979/240021 objects misplaced (9.157%); 10 MiB/s, 162 objects/s recovering 2024-09-21T10:51:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:28 smithi160 ceph-mon[101879]: osdmap e1448: 8 total, 8 up, 5 in 2024-09-21T10:51:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:28 smithi160 ceph-mon[101879]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:51:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:28 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 39270/240021 objects degraded (16.361%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:28 smithi160 ceph-mon[101879]: osdmap e1449: 8 total, 8 up, 5 in 2024-09-21T10:51:28.845 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:28.903 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:28.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:28.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:29.005 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:29.031 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:29.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:29.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:29.133 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:29.174 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:29.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:29.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:29.276 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:29.301 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:29.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:29.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:29.403 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:29.434 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:29.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:29.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:29.535 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:29.559 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:29.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:29.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:29.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:29 smithi070 ceph-mon[107385]: 55.8 starting backfill to osd.1 from (0'0,0'0] MAX to 1446'8092 2024-09-21T10:51:29.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:29 smithi070 ceph-mon[107385]: 55.5 continuing backfill to osd.3 from (1416'3175,1446'8164] MIN to 1446'8164 2024-09-21T10:51:29.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:29 smithi070 ceph-mon[107385]: 55.5 starting backfill to osd.4 from (0'0,0'0] MAX to 1446'8164 2024-09-21T10:51:29.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:29 smithi070 ceph-mon[107385]: 55.3 starting backfill to osd.6 from (0'0,0'0] MAX to 1446'7893 2024-09-21T10:51:29.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:29 smithi070 ceph-mon[107385]: 55.2 starting backfill to osd.3 from (0'0,0'0] MAX to 1446'8056 2024-09-21T10:51:29.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:29 smithi070 ceph-mon[107385]: 55.c starting backfill to osd.1 from (0'0,0'0] MAX to 1446'8153 2024-09-21T10:51:29.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:29 smithi070 ceph-mon[107385]: 55.f starting backfill to osd.1 from (0'0,0'0] MAX to 1446'8177 2024-09-21T10:51:29.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:29 smithi070 ceph-mon[107385]: 55.d starting backfill to osd.6 from (0'0,0'0] MAX to 1446'8372 2024-09-21T10:51:29.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:29 smithi070 ceph-mon[107385]: osdmap e1450: 8 total, 8 up, 5 in 2024-09-21T10:51:29.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:29 smithi070 ceph-mon[112139]: 55.8 starting backfill to osd.1 from (0'0,0'0] MAX to 1446'8092 2024-09-21T10:51:29.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:29 smithi070 ceph-mon[112139]: 55.5 continuing backfill to osd.3 from (1416'3175,1446'8164] MIN to 1446'8164 2024-09-21T10:51:29.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:29 smithi070 ceph-mon[112139]: 55.5 starting backfill to osd.4 from (0'0,0'0] MAX to 1446'8164 2024-09-21T10:51:29.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:29 smithi070 ceph-mon[112139]: 55.3 starting backfill to osd.6 from (0'0,0'0] MAX to 1446'7893 2024-09-21T10:51:29.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:29 smithi070 ceph-mon[112139]: 55.2 starting backfill to osd.3 from (0'0,0'0] MAX to 1446'8056 2024-09-21T10:51:29.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:29 smithi070 ceph-mon[112139]: 55.c starting backfill to osd.1 from (0'0,0'0] MAX to 1446'8153 2024-09-21T10:51:29.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:29 smithi070 ceph-mon[112139]: 55.f starting backfill to osd.1 from (0'0,0'0] MAX to 1446'8177 2024-09-21T10:51:29.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:29 smithi070 ceph-mon[112139]: 55.d starting backfill to osd.6 from (0'0,0'0] MAX to 1446'8372 2024-09-21T10:51:29.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:29 smithi070 ceph-mon[112139]: osdmap e1450: 8 total, 8 up, 5 in 2024-09-21T10:51:29.661 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:29.717 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:29.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:29.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:29.819 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:29 smithi160 ceph-mon[101879]: 55.8 starting backfill to osd.1 from (0'0,0'0] MAX to 1446'8092 2024-09-21T10:51:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:29 smithi160 ceph-mon[101879]: 55.5 continuing backfill to osd.3 from (1416'3175,1446'8164] MIN to 1446'8164 2024-09-21T10:51:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:29 smithi160 ceph-mon[101879]: 55.5 starting backfill to osd.4 from (0'0,0'0] MAX to 1446'8164 2024-09-21T10:51:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:29 smithi160 ceph-mon[101879]: 55.3 starting backfill to osd.6 from (0'0,0'0] MAX to 1446'7893 2024-09-21T10:51:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:29 smithi160 ceph-mon[101879]: 55.2 starting backfill to osd.3 from (0'0,0'0] MAX to 1446'8056 2024-09-21T10:51:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:29 smithi160 ceph-mon[101879]: 55.c starting backfill to osd.1 from (0'0,0'0] MAX to 1446'8153 2024-09-21T10:51:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:29 smithi160 ceph-mon[101879]: 55.f starting backfill to osd.1 from (0'0,0'0] MAX to 1446'8177 2024-09-21T10:51:29.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:29 smithi160 ceph-mon[101879]: 55.d starting backfill to osd.6 from (0'0,0'0] MAX to 1446'8372 2024-09-21T10:51:29.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:29 smithi160 ceph-mon[101879]: osdmap e1450: 8 total, 8 up, 5 in 2024-09-21T10:51:29.858 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:29.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:29.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:29.960 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:29.985 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:29.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:29.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:30.087 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:30.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:30.127 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:30.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:30.228 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:30.263 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:30.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:30.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:30.365 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:30.413 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:30.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:30.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:30.515 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:30.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:30.560 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:30.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:30.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:30 smithi070 ceph-mon[112139]: pgmap v2627: 99 pgs: 8 peering, 2 remapped+peering, 1 unknown, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 74 active+clean; 4.9 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 361 op/s; 39270/240021 objects degraded (16.361%); 21979/240021 objects misplaced (9.157%); 8.4 MiB/s, 133 objects/s recovering 2024-09-21T10:51:30.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:30 smithi070 ceph-mon[112139]: osdmap e1451: 8 total, 8 up, 5 in 2024-09-21T10:51:30.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:30 smithi070 ceph-mon[107385]: pgmap v2627: 99 pgs: 8 peering, 2 remapped+peering, 1 unknown, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 74 active+clean; 4.9 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 361 op/s; 39270/240021 objects degraded (16.361%); 21979/240021 objects misplaced (9.157%); 8.4 MiB/s, 133 objects/s recovering 2024-09-21T10:51:30.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:30 smithi070 ceph-mon[107385]: osdmap e1451: 8 total, 8 up, 5 in 2024-09-21T10:51:30.661 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:30.702 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:30.707 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:30.708 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:30.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:30.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:30.810 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:30.834 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:30.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:30.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:30 smithi160 ceph-mon[101879]: pgmap v2627: 99 pgs: 8 peering, 2 remapped+peering, 1 unknown, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 74 active+clean; 4.9 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 361 op/s; 39270/240021 objects degraded (16.361%); 21979/240021 objects misplaced (9.157%); 8.4 MiB/s, 133 objects/s recovering 2024-09-21T10:51:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:30 smithi160 ceph-mon[101879]: osdmap e1451: 8 total, 8 up, 5 in 2024-09-21T10:51:30.936 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:30.983 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:30.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:30.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:31.085 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:31.140 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:31.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:31.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:31.242 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:31.267 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:31.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:31.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:31.371 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:31.425 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:31.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:31.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:31.459 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T10:51:31.527 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:31.557 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:31.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:31.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:31.659 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:31.696 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:31.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:31.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:31.798 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:31.825 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:31.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:31.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:31.834 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:31 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/60223095' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:51:31.834 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:31 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:51:31.834 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:51:31.834 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:51:31.835 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:31 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/60223095' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:51:31.835 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:31 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:51:31.835 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:51:31.835 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:51:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:31 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/60223095' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:51:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:31 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:51:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:51:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:51:31.927 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:31.963 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:31.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:31.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:32.065 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:32.092 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:32.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:32.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:32.194 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:32.221 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:32.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:32.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:32.322 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:32.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:32.362 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:32.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:32.463 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:32.493 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:32.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:32.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:32.560 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:32 smithi070 ceph-mon[112139]: pgmap v2630: 99 pgs: 2 active+recovery_wait+degraded+remapped, 8 peering, 2 remapped+peering, 1 unknown, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 6 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 74 active+clean; 4.9 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 55 op/s; 38976/239349 objects degraded (16.284%); 28457/239349 objects misplaced (11.889%); 2.3 MiB/s, 37 objects/s recovering 2024-09-21T10:51:32.561 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:32 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:51:32.561 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:32 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:51:32.561 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:32 smithi070 ceph-mon[112139]: osdmap e1452: 8 total, 8 up, 5 in 2024-09-21T10:51:32.561 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:32 smithi070 ceph-mon[107385]: pgmap v2630: 99 pgs: 2 active+recovery_wait+degraded+remapped, 8 peering, 2 remapped+peering, 1 unknown, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 6 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 74 active+clean; 4.9 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 55 op/s; 38976/239349 objects degraded (16.284%); 28457/239349 objects misplaced (11.889%); 2.3 MiB/s, 37 objects/s recovering 2024-09-21T10:51:32.561 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:32 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:51:32.561 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:32 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:51:32.561 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:32 smithi070 ceph-mon[107385]: osdmap e1452: 8 total, 8 up, 5 in 2024-09-21T10:51:32.595 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:32.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:32.634 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:32.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:32.735 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:32.778 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:32.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:32.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:32 smithi160 ceph-mon[101879]: pgmap v2630: 99 pgs: 2 active+recovery_wait+degraded+remapped, 8 peering, 2 remapped+peering, 1 unknown, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 6 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 74 active+clean; 4.9 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 55 op/s; 38976/239349 objects degraded (16.284%); 28457/239349 objects misplaced (11.889%); 2.3 MiB/s, 37 objects/s recovering 2024-09-21T10:51:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:32 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:51:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:32 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:51:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:32 smithi160 ceph-mon[101879]: osdmap e1452: 8 total, 8 up, 5 in 2024-09-21T10:51:32.880 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:32.903 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:32.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:32.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:33.005 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:33.029 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:33.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:33.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:33.131 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:33.168 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:33.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:33.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:33.270 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:33.297 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:33.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:33.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:33.399 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:33.424 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:33.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:33.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:33.485 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:51:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:51:33] "GET /metrics HTTP/1.1" 200 36673 "" "Prometheus/2.51.0" 2024-09-21T10:51:33.526 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:33.559 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:33.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:33.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:33.661 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:33.697 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:33.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:33.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:33.799 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:33.827 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:33.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:33.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:33 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:51:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:33 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:51:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:33 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:51:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:33 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 38976/239349 objects degraded (16.284%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:33 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:51:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:33 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:51:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:33 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:51:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:33 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 38976/239349 objects degraded (16.284%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:33.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:33 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:51:33.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:33 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:51:33.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:33 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:51:33.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:33 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 38976/239349 objects degraded (16.284%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:33.929 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:33.955 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:33.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:33.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:34.056 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:34.099 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:34.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:34.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:34.201 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:34.229 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:34.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:34.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:34.318 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:51:34.330 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:34.386 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:34.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:34.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:34.488 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:34.533 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:34.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:34.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:34.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:34 smithi070 ceph-mon[107385]: pgmap v2632: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.8 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 305 op/s; 55030/235920 objects degraded (23.326%); 32312/235920 objects misplaced (13.696%); 11 MiB/s, 176 objects/s recovering 2024-09-21T10:51:34.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:34 smithi070 ceph-mon[107385]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-21T10:51:34.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:34 smithi070 ceph-mon[112139]: pgmap v2632: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.8 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 305 op/s; 55030/235920 objects degraded (23.326%); 32312/235920 objects misplaced (13.696%); 11 MiB/s, 176 objects/s recovering 2024-09-21T10:51:34.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:34 smithi070 ceph-mon[112139]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-21T10:51:34.635 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:34.666 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:34.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:34.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:34.768 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:34.801 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:34.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:34.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:34 smithi160 ceph-mon[101879]: pgmap v2632: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.8 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 305 op/s; 55030/235920 objects degraded (23.326%); 32312/235920 objects misplaced (13.696%); 11 MiB/s, 176 objects/s recovering 2024-09-21T10:51:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:34 smithi160 ceph-mon[101879]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-21T10:51:34.903 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:34.924 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:34.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:34.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:35.026 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:35.059 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:35.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:35.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:35.141 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 6, 7] out_osds: [2, 5, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:51:35.141 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T10:51:35.161 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:35.184 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:35.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:35.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:35.285 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:35.333 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:35.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:35.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:35.435 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:35.483 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:35.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:35.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:35.585 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:35.621 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:35.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:35.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:35.722 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:35.751 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:35.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:35.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:35.853 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:35.883 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:35.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:35.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:35.984 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:36.018 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:36.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:36.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:36.120 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:36.150 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:36.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:36.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:36.252 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:36.291 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:36.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:36.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:36.392 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:36.421 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:36.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:36.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:36.523 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:36.571 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:36.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:36.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:36.672 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:36.699 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:36.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:36.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:36.718 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:36 smithi070 ceph-mon[107385]: pgmap v2633: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.8 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 227 op/s; 55030/235920 objects degraded (23.326%); 32312/235920 objects misplaced (13.696%); 8.1 MiB/s, 132 objects/s recovering 2024-09-21T10:51:36.718 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:51:36.718 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:51:36.718 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:36 smithi070 ceph-mon[112139]: pgmap v2633: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.8 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 227 op/s; 55030/235920 objects degraded (23.326%); 32312/235920 objects misplaced (13.696%); 8.1 MiB/s, 132 objects/s recovering 2024-09-21T10:51:36.719 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:51:36.719 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:51:36.800 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:36.826 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:36.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:36.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:36 smithi160 ceph-mon[101879]: pgmap v2633: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.8 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 227 op/s; 55030/235920 objects degraded (23.326%); 32312/235920 objects misplaced (13.696%); 8.1 MiB/s, 132 objects/s recovering 2024-09-21T10:51:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:51:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:51:36.927 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:36.970 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:36.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:36.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:37.072 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:37.105 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:37.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:37.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:37.207 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:37.240 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:37.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:37.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:37.342 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:37.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:37.387 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:37.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:37.488 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:37.533 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:37.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:37.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:37.634 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:37.680 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:37.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:37.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:37.782 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:37.818 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:37.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:37.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:37.920 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:37.947 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:37.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:37.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:38.048 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:38.090 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:38.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:38.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:38.191 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:38.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:38.221 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:38.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:38.322 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:38.372 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:38.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:38.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:38.474 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:38.509 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:38.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:38.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:38.611 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:38.653 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:38.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:38.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:38.754 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:38.760 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:38 smithi070 ceph-mon[107385]: pgmap v2634: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.7 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 478 op/s; 53784/229299 objects degraded (23.456%); 31512/229299 objects misplaced (13.743%); 14 MiB/s, 232 objects/s recovering 2024-09-21T10:51:38.760 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:38 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 53784/229299 objects degraded (23.456%), 13 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:38.760 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:38 smithi070 ceph-mon[112139]: pgmap v2634: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.7 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 478 op/s; 53784/229299 objects degraded (23.456%); 31512/229299 objects misplaced (13.743%); 14 MiB/s, 232 objects/s recovering 2024-09-21T10:51:38.760 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:38 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 53784/229299 objects degraded (23.456%), 13 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:38.781 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:38.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:38.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:38 smithi160 ceph-mon[101879]: pgmap v2634: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.7 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 478 op/s; 53784/229299 objects degraded (23.456%); 31512/229299 objects misplaced (13.743%); 14 MiB/s, 232 objects/s recovering 2024-09-21T10:51:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:38 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 53784/229299 objects degraded (23.456%), 13 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:38.883 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:38.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:38.910 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:38.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:39.011 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:39.072 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:39.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:39.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:39.153 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:51:39.154 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T10:51:39.154 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T10:51:39.154 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T10:51:39.154 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:51:39.174 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:39.215 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:51:39.215 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:51:39.215 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:51:39.215 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:51:39.216 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:51:39.216 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:51:39.218 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:39.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:39.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:39.230 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:39.234 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:39.274 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:51:39.274 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:51:39.309 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:51:39.309 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:51:39.320 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:39.337 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:51:39.338 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:51:39.364 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:39.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:39.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:39.376 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:51:39.376 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:51:39.405 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:51:39.405 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:51:39.466 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:39.509 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:39.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:39.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:39.611 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:39 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1449619808' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:51:39.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:39 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1449619808' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:51:39.641 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:39.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:39.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:39.743 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:39.770 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:39.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:39.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:39 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1449619808' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:51:39.872 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:39.901 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:39.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:39.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:40.003 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:40.034 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:40.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:40.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:40.136 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:40.163 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:40.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:40.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:40.265 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:40.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:40.306 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:40.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:40.407 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:40.440 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:40.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:40.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:40.542 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:40.584 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:40.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:40.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:40.616 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:40 smithi070 ceph-mon[107385]: pgmap v2635: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.7 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 421 op/s; 53784/229299 objects degraded (23.456%); 31512/229299 objects misplaced (13.743%); 13 MiB/s, 205 objects/s recovering 2024-09-21T10:51:40.617 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:40 smithi070 ceph-mon[112139]: pgmap v2635: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.7 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 421 op/s; 53784/229299 objects degraded (23.456%); 31512/229299 objects misplaced (13.743%); 13 MiB/s, 205 objects/s recovering 2024-09-21T10:51:40.656 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:51:40.686 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:40.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:40.736 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:40.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:40.837 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:40 smithi160 ceph-mon[101879]: pgmap v2635: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.7 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 421 op/s; 53784/229299 objects degraded (23.456%); 31512/229299 objects misplaced (13.743%); 13 MiB/s, 205 objects/s recovering 2024-09-21T10:51:40.869 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:40.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:40.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:40.971 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:40.994 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:40.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:40.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:41.096 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:41.127 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:41.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:41.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:41.228 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:41.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:41.282 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:41.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:41.383 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:41.422 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:41.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:41.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:41.525 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:41.550 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:41.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:41.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:41.651 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:41.678 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:41.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:41.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:41.780 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:41.809 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:41.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:41.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:41.911 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:41.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:41.952 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:41.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:42.053 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:42.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:42.097 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:42.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:42.199 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:42.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:42.236 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:42.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:42.248 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:51:42.337 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:42.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:42.400 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:42.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:42.502 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:42.534 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:42.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:42.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:42.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:42 smithi070 ceph-mon[107385]: pgmap v2636: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 378 op/s; 53202/227985 objects degraded (23.336%); 31217/227985 objects misplaced (13.693%); 12 MiB/s, 188 objects/s recovering 2024-09-21T10:51:42.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:42 smithi070 ceph-mon[112139]: pgmap v2636: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 378 op/s; 53202/227985 objects degraded (23.336%); 31217/227985 objects misplaced (13.693%); 12 MiB/s, 188 objects/s recovering 2024-09-21T10:51:42.636 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:42.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:42.674 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:42.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:42.775 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:42.827 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:42.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:42.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:42.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:42 smithi160 ceph-mon[101879]: pgmap v2636: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 378 op/s; 53202/227985 objects degraded (23.336%); 31217/227985 objects misplaced (13.693%); 12 MiB/s, 188 objects/s recovering 2024-09-21T10:51:42.932 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:42.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:42.982 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:42.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:43.083 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:43.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:43.116 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:43.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:43.210 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:43.217 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:43.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:43.305 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:43.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:43.406 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:43.407 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:51:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:51:43] "GET /metrics HTTP/1.1" 200 36688 "" "Prometheus/2.51.0" 2024-09-21T10:51:43.431 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:43.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:43.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:43.533 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:43.561 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:43.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:43.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:43.663 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:43.702 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:43 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 53202/227985 objects degraded (23.336%), 13 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:43.702 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:43 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 53202/227985 objects degraded (23.336%), 13 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:43.705 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:43.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:43.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:43.807 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:43.836 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:43.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:43.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:43 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 53202/227985 objects degraded (23.336%), 13 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:43.938 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:43.962 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:43.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:43.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:44.064 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:44.089 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:44.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:44.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:44.191 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:44.217 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:44.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:44.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:44.319 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:44.352 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:44.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:44.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:44.454 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:44.479 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:44.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:44.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:44.581 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:44.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:44.619 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:44.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:44.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:44 smithi070 ceph-mon[107385]: pgmap v2637: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.5 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 461 op/s; 52429/222033 objects degraded (23.613%); 30623/222033 objects misplaced (13.792%); 12 MiB/s, 189 objects/s recovering 2024-09-21T10:51:44.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:44 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2710805511' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:51:44.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:44 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:51:44.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:44 smithi070 ceph-mon[112139]: pgmap v2637: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.5 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 461 op/s; 52429/222033 objects degraded (23.613%); 30623/222033 objects misplaced (13.792%); 12 MiB/s, 189 objects/s recovering 2024-09-21T10:51:44.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:44 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2710805511' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:51:44.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:44 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:51:44.720 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:44.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:44.765 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:44.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:44 smithi160 ceph-mon[101879]: pgmap v2637: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.5 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 461 op/s; 52429/222033 objects degraded (23.613%); 30623/222033 objects misplaced (13.792%); 12 MiB/s, 189 objects/s recovering 2024-09-21T10:51:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:44 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2710805511' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:51:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:44 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:51:44.866 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:44.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:44.912 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:44.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:44.942 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T10:51:45.013 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:45.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:45.055 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:45.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:45.155 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:45.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:45.195 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:45.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:45.296 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:45.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:45.326 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:45.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:45.426 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T10:51:45.426 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:45.469 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:45.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:45.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:45.570 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:45.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:45.610 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:45.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:45.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:45 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:51:45.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:45 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:51:45.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:45 smithi070 ceph-mon[107385]: osdmap e1453: 8 total, 8 up, 5 in 2024-09-21T10:51:45.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:45 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3426642924' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:51:45.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:45 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:51:45.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:45 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:51:45.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:45 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:51:45.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:45 smithi070 ceph-mon[112139]: osdmap e1453: 8 total, 8 up, 5 in 2024-09-21T10:51:45.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:45 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3426642924' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:51:45.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:45 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:51:45.711 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:45.745 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:45.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:45.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:45.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:45 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:51:45.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:45 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:51:45.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:45 smithi160 ceph-mon[101879]: osdmap e1453: 8 total, 8 up, 5 in 2024-09-21T10:51:45.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:45 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3426642924' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:51:45.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:45 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:51:45.848 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:45.874 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:45.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:45.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:45.976 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:46.011 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:46.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:46.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:46.113 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:46.140 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:46.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:46.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:46.242 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:46.265 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:46.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:46.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:46.367 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:46.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:46.409 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:46.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:46.510 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:46.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:46.545 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:46.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:46.646 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:46.687 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:46.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:46.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:46.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:46 smithi070 ceph-mon[107385]: pgmap v2639: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.5 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 462 op/s; 52429/222033 objects degraded (23.613%); 30623/222033 objects misplaced (13.792%); 12 MiB/s, 198 objects/s recovering 2024-09-21T10:51:46.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:46 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:51:46.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:46 smithi070 ceph-mon[107385]: osdmap e1454: 8 total, 8 up, 5 in 2024-09-21T10:51:46.739 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:46 smithi070 ceph-mon[112139]: pgmap v2639: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.5 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 462 op/s; 52429/222033 objects degraded (23.613%); 30623/222033 objects misplaced (13.792%); 12 MiB/s, 198 objects/s recovering 2024-09-21T10:51:46.739 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:46 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:51:46.740 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:46 smithi070 ceph-mon[112139]: osdmap e1454: 8 total, 8 up, 5 in 2024-09-21T10:51:46.790 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:46.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:46.818 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:46.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:46 smithi160 ceph-mon[101879]: pgmap v2639: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.5 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 462 op/s; 52429/222033 objects degraded (23.613%); 30623/222033 objects misplaced (13.792%); 12 MiB/s, 198 objects/s recovering 2024-09-21T10:51:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:46 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:51:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:46 smithi160 ceph-mon[101879]: osdmap e1454: 8 total, 8 up, 5 in 2024-09-21T10:51:46.920 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:46.947 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:46.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:46.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:47.049 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:47.078 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:47.079 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:47.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:47.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:47.095 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T10:51:47.181 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:47.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:47.224 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:47.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:47.325 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:47.388 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:47.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:47.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:47.490 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:47.519 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:47.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:47.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:47.620 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:47.652 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:47.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:47.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:47.754 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:47.779 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:47.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:47.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:47.826 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:51:47.827 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:51:47.881 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:47.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:47.919 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:47.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:48.020 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:48.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:48.056 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:48.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:48.157 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:48.179 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:48.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:48.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:48.281 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:48.310 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:48.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:48.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:48.412 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:48.442 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:48.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:48.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:48.444 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:48 smithi070 ceph-mon[112139]: pgmap v2641: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.4 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 606 op/s; 51083/214746 objects degraded (23.788%); 29721/214746 objects misplaced (13.840%); 16 MiB/s, 259 objects/s recovering 2024-09-21T10:51:48.444 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:48 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 51083/214746 objects degraded (23.788%), 13 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:48.444 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:48 smithi070 ceph-mon[107385]: pgmap v2641: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.4 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 606 op/s; 51083/214746 objects degraded (23.788%); 29721/214746 objects misplaced (13.840%); 16 MiB/s, 259 objects/s recovering 2024-09-21T10:51:48.444 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:48 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 51083/214746 objects degraded (23.788%), 13 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:48.539 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:51:48.544 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:48.634 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:48.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:48.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:48.639 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:51:48.736 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:48.778 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:48.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:48.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:48.831 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:51:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:48 smithi160 ceph-mon[101879]: pgmap v2641: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.4 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 606 op/s; 51083/214746 objects degraded (23.788%); 29721/214746 objects misplaced (13.840%); 16 MiB/s, 259 objects/s recovering 2024-09-21T10:51:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:48 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 51083/214746 objects degraded (23.788%), 13 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:48.879 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:48.911 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:48.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:48.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:49.013 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:49.061 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:49.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:49.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:49.163 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:49.196 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:49.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:49.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:49.298 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:49.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:49.346 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:49.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:49.447 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:49.483 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:49.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:49.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:49.585 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:49.630 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:49.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:49.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:49.689 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:49.732 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:49.733 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:49.801 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:49.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:49.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:49.903 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:49.932 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:49.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:49.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:50.034 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:50.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:50.094 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:50.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:50.195 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:50.229 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:50.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:50.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:50.331 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:50.365 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:50.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:50.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:50.468 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:50.499 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:50.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:50.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:50 smithi160 ceph-mon[101879]: pgmap v2642: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.4 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 551 op/s; 51083/214746 objects degraded (23.788%); 29721/214746 objects misplaced (13.840%); 14 MiB/s, 222 objects/s recovering 2024-09-21T10:51:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:51:50.603 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:50.634 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:50.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:50.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:50.736 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:50.775 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:50.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:50.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:50.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:50 smithi070 ceph-mon[112139]: pgmap v2642: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.4 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 551 op/s; 51083/214746 objects degraded (23.788%); 29721/214746 objects misplaced (13.840%); 14 MiB/s, 222 objects/s recovering 2024-09-21T10:51:50.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:51:50.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:50 smithi070 ceph-mon[107385]: pgmap v2642: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.4 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 551 op/s; 51083/214746 objects degraded (23.788%); 29721/214746 objects misplaced (13.840%); 14 MiB/s, 222 objects/s recovering 2024-09-21T10:51:50.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:51:50.877 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:50.911 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:51:50.912 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:51:50.914 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:50.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:50.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:51.016 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:51.043 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:51.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:51.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:51.145 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:51.182 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:51.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:51.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:51.284 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:51.311 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:51.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:51.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:51.413 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:51.445 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:51.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:51.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:51.546 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:51.574 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:51.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:51.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:51.676 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:51.700 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:51.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:51.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:51.802 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:51.834 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:51.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:51.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:51.935 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:51.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:51.968 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:51.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:52.007 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:51:52.069 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:52.129 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:52.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:52.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:52.231 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:52.268 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:52.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:52.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:52.339 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:51:52.370 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:52.399 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:52.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:52.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:52.501 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:52.543 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:52.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:52.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:52.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:52 smithi070 ceph-mon[107385]: pgmap v2643: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.3 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 361 op/s; 50480/213357 objects degraded (23.660%); 29398/213357 objects misplaced (13.779%); 10 MiB/s, 166 objects/s recovering 2024-09-21T10:51:52.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:52 smithi070 ceph-mon[112139]: pgmap v2643: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.3 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 361 op/s; 50480/213357 objects degraded (23.660%); 29398/213357 objects misplaced (13.779%); 10 MiB/s, 166 objects/s recovering 2024-09-21T10:51:52.645 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:52.680 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:52.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:52.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:52.781 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:52.808 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:52 smithi160 ceph-mon[101879]: pgmap v2643: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.3 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 361 op/s; 50480/213357 objects degraded (23.660%); 29398/213357 objects misplaced (13.779%); 10 MiB/s, 166 objects/s recovering 2024-09-21T10:51:52.809 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:52.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:52.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:52.911 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:52.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:52.946 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:52.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:53.047 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:53.085 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:53.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:53.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:53.186 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:53.214 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:53.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:53.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:53.316 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:53.346 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:53.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:53.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:53.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:51:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:51:53] "GET /metrics HTTP/1.1" 200 36671 "" "Prometheus/2.51.0" 2024-09-21T10:51:53.448 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:53.475 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:53.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:53.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:53.577 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:53.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:53 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/323455172' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:51:53.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:53 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 50480/213357 objects degraded (23.660%), 13 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:53.612 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:53.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:53.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:53.714 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:53.752 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:53.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:53.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:53.781 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:53 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/323455172' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:51:53.781 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:53 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 50480/213357 objects degraded (23.660%), 13 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:53.782 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:53 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/323455172' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:51:53.782 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:53 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 50480/213357 objects degraded (23.660%), 13 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:53.853 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:53.885 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:53.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:53.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:53.987 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:54.022 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:54.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:54.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:54.124 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:54.157 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:54.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:54.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:54.259 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:54.292 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:51:54.292 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T10:51:54.292 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T10:51:54.292 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T10:51:54.292 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:51:54.293 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:54.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:54.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:54.395 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:54.426 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:54.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:54.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:54.528 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:54.559 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:54.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:54.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:54.661 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:54.693 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:54.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:54.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:54.795 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:54.821 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:54.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:54.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:54 smithi160 ceph-mon[101879]: pgmap v2644: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.2 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 572 op/s; 49736/207435 objects degraded (23.977%); 28792/207435 objects misplaced (13.880%); 15 MiB/s, 244 objects/s recovering 2024-09-21T10:51:54.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:54 smithi070 ceph-mon[107385]: pgmap v2644: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.2 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 572 op/s; 49736/207435 objects degraded (23.977%); 28792/207435 objects misplaced (13.880%); 15 MiB/s, 244 objects/s recovering 2024-09-21T10:51:54.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:54 smithi070 ceph-mon[112139]: pgmap v2644: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.2 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 572 op/s; 49736/207435 objects degraded (23.977%); 28792/207435 objects misplaced (13.880%); 15 MiB/s, 244 objects/s recovering 2024-09-21T10:51:54.923 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:54.951 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:54.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:54.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:55.053 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:55.057 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:51:55.082 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:55.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:55.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:55.185 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:55.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:55.217 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:55.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:55.319 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:55.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:55.354 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:55.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:55.370 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:51:55.454 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:55.492 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:55.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:55.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:55.594 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:55.627 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:55.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:55.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:55.729 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:55.760 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:55.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:55.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:55.862 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:55.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:55.899 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:55.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:56.000 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:56.039 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:56.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:56.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:56.140 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:56.161 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:56.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:56.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:56.263 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:56.303 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:56.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:56.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:56.404 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:56.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:56.434 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:56.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:56.535 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:56.564 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:56.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:56.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:56.666 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:56.692 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:56.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:56.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:56.794 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:56.816 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:56.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:56.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:56 smithi160 ceph-mon[101879]: pgmap v2645: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.2 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 486 op/s; 49736/207435 objects degraded (23.977%); 28792/207435 objects misplaced (13.880%); 13 MiB/s, 207 objects/s recovering 2024-09-21T10:51:56.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:56 smithi070 ceph-mon[112139]: pgmap v2645: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.2 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 486 op/s; 49736/207435 objects degraded (23.977%); 28792/207435 objects misplaced (13.880%); 13 MiB/s, 207 objects/s recovering 2024-09-21T10:51:56.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:56 smithi070 ceph-mon[107385]: pgmap v2645: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.2 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 486 op/s; 49736/207435 objects degraded (23.977%); 28792/207435 objects misplaced (13.880%); 13 MiB/s, 207 objects/s recovering 2024-09-21T10:51:56.918 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:56.939 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:56.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:56.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:57.041 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:57.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:57.093 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:57.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:57.194 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:57.219 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:57.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:57.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:57.321 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:57.357 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:57.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:57.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:57.459 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:57.496 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:57.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:57.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:57.598 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:57.637 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:57.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:57.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:57.739 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:57.779 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:57.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:57.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:57.881 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:57.913 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:57.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:57.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:58.015 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:58.058 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:58.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:58.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:58.136 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:51:58.136 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:51:58.136 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:51:58.136 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:51:58.136 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:51:58.136 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:51:58.136 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:51:58.136 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:51:58.136 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:51:58.137 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:51:58.137 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:51:58.137 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:51:58.137 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:51:58.137 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:51:58.137 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:51:58.137 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:51:58.161 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:58.195 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:58.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:58.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:58.297 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:58.324 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:58.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:58.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:58.426 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:58.449 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:58.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:58.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:58.550 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:58.584 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:58.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:58.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:58.685 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:58.728 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:58.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:58.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:58.830 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:58 smithi160 ceph-mon[101879]: pgmap v2646: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 628 op/s; 48399/200400 objects degraded (24.151%); 27867/200400 objects misplaced (13.906%); 17 MiB/s, 271 objects/s recovering 2024-09-21T10:51:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:51:58 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 48399/200400 objects degraded (24.151%), 13 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:58.869 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:58.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:58.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:58.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:58 smithi070 ceph-mon[107385]: pgmap v2646: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 628 op/s; 48399/200400 objects degraded (24.151%); 27867/200400 objects misplaced (13.906%); 17 MiB/s, 271 objects/s recovering 2024-09-21T10:51:58.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:51:58 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 48399/200400 objects degraded (24.151%), 13 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:58.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:58 smithi070 ceph-mon[112139]: pgmap v2646: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 628 op/s; 48399/200400 objects degraded (24.151%); 27867/200400 objects misplaced (13.906%); 17 MiB/s, 271 objects/s recovering 2024-09-21T10:51:58.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:51:58 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 48399/200400 objects degraded (24.151%), 13 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:51:58.971 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:59.006 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:59.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:59.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:59.107 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:59.143 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:59.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:59.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:59.244 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:59.278 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:59.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:59.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:59.379 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:59.413 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:59.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:59.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:59.514 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:59.549 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:59.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:59.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:59.651 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:59.683 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:51:59.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:59.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:59.784 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:59.818 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:59.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:59.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:59.920 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:59.955 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:51:59.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:59.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:00.057 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:00.087 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:00.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:00.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:00.189 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:00.222 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:00.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:00.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:00.323 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:00.354 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:00.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:00.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:00.456 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:00.491 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:00.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:00.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:00.546 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:00 smithi160 ceph-mon[101879]: pgmap v2647: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 398 op/s; 48399/200400 objects degraded (24.151%); 27867/200400 objects misplaced (13.906%); 11 MiB/s, 173 objects/s recovering 2024-09-21T10:52:00.592 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:52:00.592 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T10:52:00.592 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T10:52:00.592 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T10:52:00.592 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.592 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:62979 54.8 54:124a0706:::benchmark_data_smithi070_261699_object62978:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1382)", 2024-09-21T10:52:00.592 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:16.804088+0000", 2024-09-21T10:52:00.592 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 403.78229764500003, 2024-09-21T10:52:00.592 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.17356136999999999, 2024-09-21T10:52:00.593 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:00.593 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:52:00.593 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:00.593 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:52:00.593 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:52:00.593 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 62979 2024-09-21T10:52:00.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.593 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:00.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.593 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:00.593 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.804088+0000", 2024-09-21T10:52:00.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:00.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.804088+0000", 2024-09-21T10:52:00.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:00.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.804086+0000", 2024-09-21T10:52:00.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:52:00.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:00.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.804123+0000", 2024-09-21T10:52:00.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6477000000000003e-05 2024-09-21T10:52:00.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.595 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:00.595 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.804139+0000", 2024-09-21T10:52:00.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5608999999999999e-05 2024-09-21T10:52:00.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.595 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:00.595 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.804146+0000", 2024-09-21T10:52:00.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5120000000000002e-06 2024-09-21T10:52:00.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.595 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.596 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.804266+0000", 2024-09-21T10:52:00.596 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000120048 2024-09-21T10:52:00.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.596 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:00.596 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.804382+0000", 2024-09-21T10:52:00.596 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000115902 2024-09-21T10:52:00.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.596 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:52:00.596 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.804499+0000", 2024-09-21T10:52:00.596 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000117192 2024-09-21T10:52:00.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.597 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:52:00.597 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.806645+0000", 2024-09-21T10:52:00.597 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021456660000000001 2024-09-21T10:52:00.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.597 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.597 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.812886+0000", 2024-09-21T10:52:00.597 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0062415250000000004 2024-09-21T10:52:00.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.597 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.597 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.977555+0000", 2024-09-21T10:52:00.597 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16466847500000001 2024-09-21T10:52:00.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.598 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:52:00.598 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.977610+0000", 2024-09-21T10:52:00.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5195999999999998e-05 2024-09-21T10:52:00.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.598 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:00.598 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.977649+0000", 2024-09-21T10:52:00.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9267999999999997e-05 2024-09-21T10:52:00.598 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.598 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:00.598 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.599 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:62983 54.8 54:1bc53d7e:::benchmark_data_smithi070_261699_object62982:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1382)", 2024-09-21T10:52:00.599 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:16.805440+0000", 2024-09-21T10:52:00.599 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 403.78094588800002, 2024-09-21T10:52:00.599 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.172337825, 2024-09-21T10:52:00.599 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:00.599 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:52:00.599 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:00.599 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:52:00.599 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:52:00.599 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 62983 2024-09-21T10:52:00.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.600 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:00.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.600 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:00.600 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.805440+0000", 2024-09-21T10:52:00.600 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.600 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:00.600 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.805440+0000", 2024-09-21T10:52:00.600 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.600 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:00.600 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.805438+0000", 2024-09-21T10:52:00.601 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:52:00.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.601 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:00.601 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.805467+0000", 2024-09-21T10:52:00.601 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8269e-05 2024-09-21T10:52:00.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.601 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:00.601 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.805482+0000", 2024-09-21T10:52:00.601 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5766000000000001e-05 2024-09-21T10:52:00.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.601 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:00.602 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.805489+0000", 2024-09-21T10:52:00.602 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0570000000000004e-06 2024-09-21T10:52:00.602 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.602 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.602 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.602 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.807208+0000", 2024-09-21T10:52:00.602 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0017185200000000001 2024-09-21T10:52:00.602 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.602 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.602 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:00.602 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.807328+0000", 2024-09-21T10:52:00.602 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119848 2024-09-21T10:52:00.602 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.602 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.603 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:52:00.603 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.807459+0000", 2024-09-21T10:52:00.603 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013086799999999999 2024-09-21T10:52:00.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.603 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.603 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.812947+0000", 2024-09-21T10:52:00.603 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0054882489999999997 2024-09-21T10:52:00.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.603 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:52:00.603 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.812955+0000", 2024-09-21T10:52:00.603 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1149999999999994e-06 2024-09-21T10:52:00.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.604 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.977705+0000", 2024-09-21T10:52:00.604 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16474944599999999 2024-09-21T10:52:00.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:52:00.604 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.977745+0000", 2024-09-21T10:52:00.604 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0788000000000003e-05 2024-09-21T10:52:00.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:00.604 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.977778+0000", 2024-09-21T10:52:00.604 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2249000000000002e-05 2024-09-21T10:52:00.605 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.605 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:00.605 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.605 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45308.0:62991 54.8 54:17e38a96:::benchmark_data_smithi070_261699_object62990:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1382)", 2024-09-21T10:52:00.605 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:45:16.809905+0000", 2024-09-21T10:52:00.605 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 403.776480714, 2024-09-21T10:52:00.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.17707988299999999, 2024-09-21T10:52:00.605 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:00.605 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:52:00.605 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:00.605 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45308", 2024-09-21T10:52:00.606 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:00.607 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/569008526", 2024-09-21T10:52:00.607 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 62991 2024-09-21T10:52:00.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.607 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:00.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.607 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:00.607 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.809905+0000", 2024-09-21T10:52:00.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.608 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:00.608 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.809905+0000", 2024-09-21T10:52:00.608 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.608 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:00.608 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.809904+0000", 2024-09-21T10:52:00.608 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:52:00.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.608 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:00.608 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.809935+0000", 2024-09-21T10:52:00.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1415000000000003e-05 2024-09-21T10:52:00.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:00.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.809945+0000", 2024-09-21T10:52:00.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7240000000000004e-06 2024-09-21T10:52:00.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:00.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.809952+0000", 2024-09-21T10:52:00.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8430000000000004e-06 2024-09-21T10:52:00.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.809997+0000", 2024-09-21T10:52:00.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5275999999999998e-05 2024-09-21T10:52:00.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:00.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.810123+0000", 2024-09-21T10:52:00.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012623800000000001 2024-09-21T10:52:00.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:52:00.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.810253+0000", 2024-09-21T10:52:00.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000129751 2024-09-21T10:52:00.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.814963+0000", 2024-09-21T10:52:00.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0047100110000000001 2024-09-21T10:52:00.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:52:00.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.816768+0000", 2024-09-21T10:52:00.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0018055390000000001 2024-09-21T10:52:00.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.612 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.986870+0000", 2024-09-21T10:52:00.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.170101531 2024-09-21T10:52:00.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.612 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:52:00.612 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.986949+0000", 2024-09-21T10:52:00.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9368000000000003e-05 2024-09-21T10:52:00.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.612 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:00.612 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:45:16.986985+0000", 2024-09-21T10:52:00.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5488000000000003e-05 2024-09-21T10:52:00.612 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.613 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:00.613 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.613 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.35670.0:72212 55.a e1425/1411 55:51f2ed65:::benchmark_data_smithi070_294394_object72211:head v 1425'4475, mlcod=1425'4473)", 2024-09-21T10:52:00.613 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:49:40.176758+0000", 2024-09-21T10:52:00.613 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 140.40962741999999, 2024-09-21T10:52:00.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.29508102600000002, 2024-09-21T10:52:00.613 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:00.613 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:52:00.613 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:00.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:00.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.176758+0000", 2024-09-21T10:52:00.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:00.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.176758+0000", 2024-09-21T10:52:00.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:00.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.176757+0000", 2024-09-21T10:52:00.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:52:00.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.615 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:00.615 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.176788+0000", 2024-09-21T10:52:00.615 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0559000000000003e-05 2024-09-21T10:52:00.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.615 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.615 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:00.615 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.176800+0000", 2024-09-21T10:52:00.615 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1918e-05 2024-09-21T10:52:00.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.615 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.615 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:00.615 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.176807+0000", 2024-09-21T10:52:00.615 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2640000000000002e-06 2024-09-21T10:52:00.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.616 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.616 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.176854+0000", 2024-09-21T10:52:00.616 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6476e-05 2024-09-21T10:52:00.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.616 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:00.616 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.176873+0000", 2024-09-21T10:52:00.616 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9624999999999999e-05 2024-09-21T10:52:00.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.616 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:52:00.616 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.471800+0000", 2024-09-21T10:52:00.616 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.29492674299999999 2024-09-21T10:52:00.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.617 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:00.617 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.471839+0000", 2024-09-21T10:52:00.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9310999999999997e-05 2024-09-21T10:52:00.617 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.617 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:00.617 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.617 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.35670.0:72213 55.a e1425/1411 55:551f1baf:::benchmark_data_smithi070_294394_object72212:head v 1425'4476, mlcod=1425'4473)", 2024-09-21T10:52:00.617 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:49:40.178762+0000", 2024-09-21T10:52:00.617 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 140.40762378599999, 2024-09-21T10:52:00.618 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.293101098, 2024-09-21T10:52:00.618 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:00.618 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:52:00.618 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:00.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:00.618 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.178762+0000", 2024-09-21T10:52:00.618 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:00.618 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.178762+0000", 2024-09-21T10:52:00.618 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.619 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:00.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.178761+0000", 2024-09-21T10:52:00.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:52:00.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.619 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:00.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.178869+0000", 2024-09-21T10:52:00.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000108661 2024-09-21T10:52:00.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.619 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:00.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.178883+0000", 2024-09-21T10:52:00.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3338e-05 2024-09-21T10:52:00.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.620 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:00.620 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.178890+0000", 2024-09-21T10:52:00.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6159999999999996e-06 2024-09-21T10:52:00.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.620 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.620 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.179248+0000", 2024-09-21T10:52:00.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00035803700000000001 2024-09-21T10:52:00.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.620 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:00.620 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.179267+0000", 2024-09-21T10:52:00.621 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9004999999999998e-05 2024-09-21T10:52:00.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.621 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:52:00.621 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.471843+0000", 2024-09-21T10:52:00.621 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.292575843 2024-09-21T10:52:00.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.621 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:00.621 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.471863+0000", 2024-09-21T10:52:00.621 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9646000000000001e-05 2024-09-21T10:52:00.621 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.621 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:00.621 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.622 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.35670.0:72214 55.1 e1425/1411 55:8b604f8f:::benchmark_data_smithi070_294394_object72213:head v 1425'4519, mlcod=1425'4516)", 2024-09-21T10:52:00.622 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:49:40.179964+0000", 2024-09-21T10:52:00.622 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 140.40642169899999, 2024-09-21T10:52:00.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30231281900000001, 2024-09-21T10:52:00.622 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:00.622 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:52:00.622 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:00.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.622 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:00.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.179964+0000", 2024-09-21T10:52:00.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:00.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.179964+0000", 2024-09-21T10:52:00.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:00.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.179963+0000", 2024-09-21T10:52:00.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:52:00.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:00.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180001+0000", 2024-09-21T10:52:00.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8398000000000001e-05 2024-09-21T10:52:00.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.624 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:00.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180012+0000", 2024-09-21T10:52:00.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0916e-05 2024-09-21T10:52:00.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.624 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:00.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180020+0000", 2024-09-21T10:52:00.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2030000000000003e-06 2024-09-21T10:52:00.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.624 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.625 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180093+0000", 2024-09-21T10:52:00.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3728999999999998e-05 2024-09-21T10:52:00.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.625 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:00.625 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180114+0000", 2024-09-21T10:52:00.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0322999999999998e-05 2024-09-21T10:52:00.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.625 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:52:00.625 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.482168+0000", 2024-09-21T10:52:00.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30205438699999998 2024-09-21T10:52:00.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.626 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.626 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:00.626 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.482277+0000", 2024-09-21T10:52:00.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000108762 2024-09-21T10:52:00.626 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.626 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:00.626 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.626 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.626 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.35670.0:72216 55.c e1425/1411 55:33c7533d:::benchmark_data_smithi070_294394_object72215:head v 1425'4484, mlcod=1425'4481)", 2024-09-21T10:52:00.626 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:49:40.180613+0000", 2024-09-21T10:52:00.626 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 140.40577235699999, 2024-09-21T10:52:00.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.29111427499999998, 2024-09-21T10:52:00.626 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:00.627 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:52:00.627 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:00.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:00.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180613+0000", 2024-09-21T10:52:00.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:00.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180613+0000", 2024-09-21T10:52:00.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.628 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:00.628 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180612+0000", 2024-09-21T10:52:00.628 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:52:00.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.628 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.628 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:00.628 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180726+0000", 2024-09-21T10:52:00.628 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000113441 2024-09-21T10:52:00.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.628 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.628 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:00.628 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180749+0000", 2024-09-21T10:52:00.628 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2691999999999999e-05 2024-09-21T10:52:00.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.629 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.629 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:00.629 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180755+0000", 2024-09-21T10:52:00.629 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2179999999999996e-06 2024-09-21T10:52:00.629 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.629 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.629 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.629 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180798+0000", 2024-09-21T10:52:00.629 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3201000000000002e-05 2024-09-21T10:52:00.629 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.629 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.629 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:00.629 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180817+0000", 2024-09-21T10:52:00.629 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9369999999999999e-05 2024-09-21T10:52:00.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.630 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:52:00.630 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.471681+0000", 2024-09-21T10:52:00.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.29086372100000002 2024-09-21T10:52:00.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.630 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:00.630 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.471728+0000", 2024-09-21T10:52:00.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6445e-05 2024-09-21T10:52:00.630 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.630 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:00.630 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.631 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.631 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.35670.0:72217 55.9 e1425/1411 55:977475fe:::benchmark_data_smithi070_294394_object72216:head v 1425'4465, mlcod=1425'4463)", 2024-09-21T10:52:00.631 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:49:40.180772+0000", 2024-09-21T10:52:00.631 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 140.40561370899999, 2024-09-21T10:52:00.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30159776900000002, 2024-09-21T10:52:00.631 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:00.631 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:52:00.631 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:00.631 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:00.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180772+0000", 2024-09-21T10:52:00.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.632 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:00.632 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180772+0000", 2024-09-21T10:52:00.632 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.632 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:00.632 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180772+0000", 2024-09-21T10:52:00.632 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T10:52:00.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.632 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:00.632 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180802+0000", 2024-09-21T10:52:00.632 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9989e-05 2024-09-21T10:52:00.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.633 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.633 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:00.633 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180808+0000", 2024-09-21T10:52:00.633 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6549999999999998e-06 2024-09-21T10:52:00.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.633 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.633 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:00.633 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180812+0000", 2024-09-21T10:52:00.633 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2950000000000004e-06 2024-09-21T10:52:00.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.633 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.633 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.634 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180862+0000", 2024-09-21T10:52:00.634 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9017000000000002e-05 2024-09-21T10:52:00.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.634 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:00.634 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180880+0000", 2024-09-21T10:52:00.634 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8374999999999999e-05 2024-09-21T10:52:00.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.634 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:52:00.634 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.482283+0000", 2024-09-21T10:52:00.634 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30140297199999999 2024-09-21T10:52:00.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.635 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:00.635 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.482370+0000", 2024-09-21T10:52:00.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6848000000000006e-05 2024-09-21T10:52:00.635 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.635 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:00.635 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.635 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.635 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.35670.0:72215 55.d e1425/1411 55:b697a537:::benchmark_data_smithi070_294394_object72214:head v 1425'4658, mlcod=1425'4656)", 2024-09-21T10:52:00.635 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:49:40.182534+0000", 2024-09-21T10:52:00.635 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 140.40385180600001, 2024-09-21T10:52:00.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.29417858400000002, 2024-09-21T10:52:00.635 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:00.635 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:52:00.636 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:00.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:00.636 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.182534+0000", 2024-09-21T10:52:00.636 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:00.636 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.182534+0000", 2024-09-21T10:52:00.636 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:00.636 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.182532+0000", 2024-09-21T10:52:00.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:52:00.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:00.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.182576+0000", 2024-09-21T10:52:00.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3553999999999997e-05 2024-09-21T10:52:00.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:00.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.471751+0000", 2024-09-21T10:52:00.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.28917540600000002 2024-09-21T10:52:00.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:00.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.471764+0000", 2024-09-21T10:52:00.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2503e-05 2024-09-21T10:52:00.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.471815+0000", 2024-09-21T10:52:00.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1384000000000001e-05 2024-09-21T10:52:00.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:00.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.471842+0000", 2024-09-21T10:52:00.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6721e-05 2024-09-21T10:52:00.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.639 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:52:00.639 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.476625+0000", 2024-09-21T10:52:00.639 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.004782573 2024-09-21T10:52:00.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.639 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:00.639 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.476712+0000", 2024-09-21T10:52:00.639 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7843000000000006e-05 2024-09-21T10:52:00.639 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.639 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:00.639 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.640 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:129299 55.d 55:b83dbe9e:::benchmark_data_smithi070_294394_object23958:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:52:00.640 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:22.715175+0000", 2024-09-21T10:52:00.640 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 37.871210851000001, 2024-09-21T10:52:00.640 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.25006946699999999, 2024-09-21T10:52:00.640 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:00.640 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:52:00.640 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:00.640 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:52:00.640 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:52:00.640 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 129299 2024-09-21T10:52:00.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.640 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:00.640 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.640 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:00.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.715175+0000", 2024-09-21T10:52:00.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.641 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:00.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.715175+0000", 2024-09-21T10:52:00.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.641 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:00.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.715173+0000", 2024-09-21T10:52:00.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:52:00.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.642 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:00.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.715180+0000", 2024-09-21T10:52:00.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0269999999999998e-06 2024-09-21T10:52:00.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.642 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.642 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:00.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.715193+0000", 2024-09-21T10:52:00.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3064000000000001e-05 2024-09-21T10:52:00.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.642 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.642 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:00.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.715200+0000", 2024-09-21T10:52:00.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9059999999999996e-06 2024-09-21T10:52:00.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.643 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.643 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.718977+0000", 2024-09-21T10:52:00.643 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0037772579999999999 2024-09-21T10:52:00.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.643 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:00.643 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.719152+0000", 2024-09-21T10:52:00.643 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017478 2024-09-21T10:52:00.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.643 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:52:00.644 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.719313+0000", 2024-09-21T10:52:00.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000161185 2024-09-21T10:52:00.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.644 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:52:00.644 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.722530+0000", 2024-09-21T10:52:00.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0032165689999999999 2024-09-21T10:52:00.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.644 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.644 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.726307+0000", 2024-09-21T10:52:00.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0037772439999999999 2024-09-21T10:52:00.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.644 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.645 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.960947+0000", 2024-09-21T10:52:00.645 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.23464001400000001 2024-09-21T10:52:00.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.645 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.645 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:52:00.645 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.960996+0000", 2024-09-21T10:52:00.645 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8819999999999997e-05 2024-09-21T10:52:00.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.645 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.645 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:00.645 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.965244+0000", 2024-09-21T10:52:00.645 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0042483529999999999 2024-09-21T10:52:00.645 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.645 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:00.646 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.646 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.646 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:129315 55.d 55:bf0d41f3:::benchmark_data_smithi070_294394_object23974:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:52:00.646 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:22.765937+0000", 2024-09-21T10:52:00.646 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 37.820448773000003, 2024-09-21T10:52:00.646 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20239199699999999, 2024-09-21T10:52:00.646 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:00.646 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:52:00.646 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:00.646 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:52:00.646 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:52:00.646 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 129315 2024-09-21T10:52:00.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.647 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:00.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:00.647 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.765937+0000", 2024-09-21T10:52:00.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:00.647 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.765937+0000", 2024-09-21T10:52:00.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:00.647 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.765935+0000", 2024-09-21T10:52:00.648 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:52:00.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:00.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.765945+0000", 2024-09-21T10:52:00.648 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8879999999999992e-06 2024-09-21T10:52:00.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:00.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.765959+0000", 2024-09-21T10:52:00.648 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3892e-05 2024-09-21T10:52:00.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:00.649 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.765967+0000", 2024-09-21T10:52:00.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4379999999999998e-06 2024-09-21T10:52:00.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.649 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.649 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.649 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.766010+0000", 2024-09-21T10:52:00.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3838999999999998e-05 2024-09-21T10:52:00.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.649 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.650 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:00.650 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.766151+0000", 2024-09-21T10:52:00.650 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014064700000000001 2024-09-21T10:52:00.650 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.650 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.650 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:52:00.650 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.766258+0000", 2024-09-21T10:52:00.651 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000107228 2024-09-21T10:52:00.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.651 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:52:00.651 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.768407+0000", 2024-09-21T10:52:00.651 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021486399999999998 2024-09-21T10:52:00.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.651 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.651 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.772605+0000", 2024-09-21T10:52:00.651 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0041983259999999996 2024-09-21T10:52:00.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.651 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.652 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.965314+0000", 2024-09-21T10:52:00.652 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.192708883 2024-09-21T10:52:00.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.652 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.652 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:52:00.652 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.965373+0000", 2024-09-21T10:52:00.652 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8919000000000003e-05 2024-09-21T10:52:00.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.652 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.652 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:00.652 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.968329+0000", 2024-09-21T10:52:00.652 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002955877 2024-09-21T10:52:00.652 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.653 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:00.653 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.653 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:130860 55.d 55:b4ee1c49:::benchmark_data_smithi070_294394_object25519:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:52:00.653 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:26.394761+0000", 2024-09-21T10:52:00.653 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 34.191625094000003, 2024-09-21T10:52:00.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3490294309999999, 2024-09-21T10:52:00.653 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:00.653 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:52:00.653 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:00.653 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:52:00.653 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:52:00.653 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 130860 2024-09-21T10:52:00.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.654 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:00.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.654 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:00.654 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.394761+0000", 2024-09-21T10:52:00.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.654 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:00.654 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.394761+0000", 2024-09-21T10:52:00.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.655 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:00.655 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.394758+0000", 2024-09-21T10:52:00.655 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:52:00.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.655 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.655 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:00.655 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.394768+0000", 2024-09-21T10:52:00.655 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2429999999999994e-06 2024-09-21T10:52:00.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.655 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.655 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:00.655 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.394788+0000", 2024-09-21T10:52:00.655 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9777000000000001e-05 2024-09-21T10:52:00.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.656 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:00.656 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.394796+0000", 2024-09-21T10:52:00.656 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8040000000000007e-06 2024-09-21T10:52:00.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.656 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.656 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.394852+0000", 2024-09-21T10:52:00.656 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5896999999999997e-05 2024-09-21T10:52:00.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.656 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:00.656 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.395006+0000", 2024-09-21T10:52:00.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015377699999999999 2024-09-21T10:52:00.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:52:00.657 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.395149+0000", 2024-09-21T10:52:00.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000143514 2024-09-21T10:52:00.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:52:00.657 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.397840+0000", 2024-09-21T10:52:00.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0026902039999999999 2024-09-21T10:52:00.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.658 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.414884+0000", 2024-09-21T10:52:00.658 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.017044549999999999 2024-09-21T10:52:00.658 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.658 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.658 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.658 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.429059+0000", 2024-09-21T10:52:00.658 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.014175128 2024-09-21T10:52:00.658 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.658 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.658 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:00.658 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.429074+0000", 2024-09-21T10:52:00.658 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4436e-05 2024-09-21T10:52:00.658 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.659 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.659 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.427445+0000", 2024-09-21T10:52:00.659 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99837087800000002 2024-09-21T10:52:00.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.659 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:00.659 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.427457+0000", 2024-09-21T10:52:00.659 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2643000000000001e-05 2024-09-21T10:52:00.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.659 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.659 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.691524+0000", 2024-09-21T10:52:00.659 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2640668900000001 2024-09-21T10:52:00.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.660 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for missing object", 2024-09-21T10:52:00.660 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.691640+0000", 2024-09-21T10:52:00.660 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000116205 2024-09-21T10:52:00.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.660 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.660 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.743637+0000", 2024-09-21T10:52:00.660 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.051996268999999998 2024-09-21T10:52:00.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.660 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:00.660 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.743790+0000", 2024-09-21T10:52:00.661 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015328100000000001 2024-09-21T10:52:00.661 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.661 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:00.661 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.661 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.661 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:130862 55.d 55:bd7eda03:::benchmark_data_smithi070_294394_object25521:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:52:00.661 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:26.398474+0000", 2024-09-21T10:52:00.661 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 34.187911649999997, 2024-09-21T10:52:00.661 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3783652179999999, 2024-09-21T10:52:00.661 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:00.661 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:52:00.661 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:00.661 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:52:00.661 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:52:00.662 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 130862 2024-09-21T10:52:00.662 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.662 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:00.662 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.662 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:00.662 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398474+0000", 2024-09-21T10:52:00.662 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.662 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.662 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.662 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:00.662 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398474+0000", 2024-09-21T10:52:00.662 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.662 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.663 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:00.663 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398472+0000", 2024-09-21T10:52:00.663 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:52:00.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.663 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:00.663 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398481+0000", 2024-09-21T10:52:00.663 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1589999999999993e-06 2024-09-21T10:52:00.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.663 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:00.663 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398495+0000", 2024-09-21T10:52:00.663 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4829e-05 2024-09-21T10:52:00.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.664 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.664 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:00.664 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398503+0000", 2024-09-21T10:52:00.664 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1499999999999999e-06 2024-09-21T10:52:00.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.664 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.664 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.664 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398549+0000", 2024-09-21T10:52:00.664 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5802e-05 2024-09-21T10:52:00.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.664 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.664 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:00.664 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398680+0000", 2024-09-21T10:52:00.665 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000130784 2024-09-21T10:52:00.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.665 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:52:00.665 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398797+0000", 2024-09-21T10:52:00.665 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000117408 2024-09-21T10:52:00.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.665 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:52:00.665 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.402741+0000", 2024-09-21T10:52:00.665 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0039439920000000003 2024-09-21T10:52:00.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.666 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.666 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.414917+0000", 2024-09-21T10:52:00.666 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012175314 2024-09-21T10:52:00.666 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.666 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.666 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.666 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.430736+0000", 2024-09-21T10:52:00.666 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.015818895999999999 2024-09-21T10:52:00.666 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.666 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.666 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:00.666 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.430747+0000", 2024-09-21T10:52:00.666 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1589e-05 2024-09-21T10:52:00.666 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.667 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.667 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.667 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.427498+0000", 2024-09-21T10:52:00.667 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99675095499999999 2024-09-21T10:52:00.667 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.667 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.667 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:00.667 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.427507+0000", 2024-09-21T10:52:00.667 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9830000000000002e-06 2024-09-21T10:52:00.667 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.667 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.667 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.667 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.750176+0000", 2024-09-21T10:52:00.667 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3226684440000001 2024-09-21T10:52:00.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.668 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for missing object", 2024-09-21T10:52:00.668 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.750312+0000", 2024-09-21T10:52:00.668 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013604399999999999 2024-09-21T10:52:00.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.668 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.668 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.776747+0000", 2024-09-21T10:52:00.668 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.026435199 2024-09-21T10:52:00.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.668 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:00.668 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.776839+0000", 2024-09-21T10:52:00.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2300999999999996e-05 2024-09-21T10:52:00.669 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.669 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:00.669 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.669 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:130864 55.d 55:bf76459a:::benchmark_data_smithi070_294394_object25523:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:52:00.669 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:26.398512+0000", 2024-09-21T10:52:00.669 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 34.187873152999998, 2024-09-21T10:52:00.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.415816907, 2024-09-21T10:52:00.669 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:00.669 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:52:00.669 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:00.669 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:52:00.670 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:52:00.670 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 130864 2024-09-21T10:52:00.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.670 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:00.670 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.670 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:00.670 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398512+0000", 2024-09-21T10:52:00.670 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.670 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.670 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:00.670 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398512+0000", 2024-09-21T10:52:00.670 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.671 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:00.671 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398512+0000", 2024-09-21T10:52:00.671 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T10:52:00.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.671 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:00.671 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398515+0000", 2024-09-21T10:52:00.671 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4679999999999999e-06 2024-09-21T10:52:00.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.671 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:00.671 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398530+0000", 2024-09-21T10:52:00.672 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5201000000000001e-05 2024-09-21T10:52:00.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.672 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.672 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:00.672 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398535+0000", 2024-09-21T10:52:00.672 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0250000000000002e-06 2024-09-21T10:52:00.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.672 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.672 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.672 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.400838+0000", 2024-09-21T10:52:00.672 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0023031710000000001 2024-09-21T10:52:00.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.672 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.672 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:00.673 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.400934+0000", 2024-09-21T10:52:00.673 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6107999999999995e-05 2024-09-21T10:52:00.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.673 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:52:00.673 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.401033+0000", 2024-09-21T10:52:00.673 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9264000000000004e-05 2024-09-21T10:52:00.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.673 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:52:00.673 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.402893+0000", 2024-09-21T10:52:00.673 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0018592560000000001 2024-09-21T10:52:00.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.674 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.674 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.430776+0000", 2024-09-21T10:52:00.674 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.027883080000000001 2024-09-21T10:52:00.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.674 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.674 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:00.674 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.430791+0000", 2024-09-21T10:52:00.674 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5741999999999999e-05 2024-09-21T10:52:00.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.674 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.674 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.674 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.427535+0000", 2024-09-21T10:52:00.674 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99674326099999999 2024-09-21T10:52:00.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.675 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.675 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:00.675 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.427543+0000", 2024-09-21T10:52:00.675 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5269999999999998e-06 2024-09-21T10:52:00.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.675 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.675 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.675 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.750345+0000", 2024-09-21T10:52:00.675 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.322801291 2024-09-21T10:52:00.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.675 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.675 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for missing object", 2024-09-21T10:52:00.675 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.750449+0000", 2024-09-21T10:52:00.676 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000104609 2024-09-21T10:52:00.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.676 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.676 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.814210+0000", 2024-09-21T10:52:00.676 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.063760777000000005 2024-09-21T10:52:00.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.676 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:00.676 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.814329+0000", 2024-09-21T10:52:00.676 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119499 2024-09-21T10:52:00.676 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.676 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:00.676 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.677 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:130888 55.d 55:b8d17701:::benchmark_data_smithi070_294394_object25547:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1449)", 2024-09-21T10:52:00.677 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:28.580660+0000", 2024-09-21T10:52:00.677 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 32.005725583999997, 2024-09-21T10:52:00.677 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.27647097700000001, 2024-09-21T10:52:00.677 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:00.677 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:52:00.677 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:00.677 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:52:00.677 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:52:00.677 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 130888 2024-09-21T10:52:00.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.677 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:00.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.678 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:00.678 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.580660+0000", 2024-09-21T10:52:00.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.678 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:00.678 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.580660+0000", 2024-09-21T10:52:00.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.678 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:00.679 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.580658+0000", 2024-09-21T10:52:00.679 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 2024-09-21T10:52:00.679 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.679 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.679 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:00.679 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.580669+0000", 2024-09-21T10:52:00.679 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1199000000000001e-05 2024-09-21T10:52:00.679 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.679 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.679 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:00.679 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.580685+0000", 2024-09-21T10:52:00.679 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5889999999999999e-05 2024-09-21T10:52:00.679 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:00.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.580703+0000", 2024-09-21T10:52:00.680 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8664e-05 2024-09-21T10:52:00.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.751302+0000", 2024-09-21T10:52:00.680 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.170598571 2024-09-21T10:52:00.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:52:00.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.751884+0000", 2024-09-21T10:52:00.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00058179499999999997 2024-09-21T10:52:00.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.681 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.681 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.781175+0000", 2024-09-21T10:52:00.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.029291467000000002 2024-09-21T10:52:00.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.681 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:00.681 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.781250+0000", 2024-09-21T10:52:00.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4361000000000002e-05 2024-09-21T10:52:00.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3,6", 2024-09-21T10:52:00.682 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.781630+0000", 2024-09-21T10:52:00.682 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00037988099999999999 2024-09-21T10:52:00.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:52:00.682 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.806861+0000", 2024-09-21T10:52:00.682 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.025231923 2024-09-21T10:52:00.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.682 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.814370+0000", 2024-09-21T10:52:00.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.007508163 2024-09-21T10:52:00.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.814421+0000", 2024-09-21T10:52:00.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1409000000000002e-05 2024-09-21T10:52:00.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.853936+0000", 2024-09-21T10:52:00.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.039515330000000001 2024-09-21T10:52:00.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:52:00.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.854028+0000", 2024-09-21T10:52:00.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1755000000000005e-05 2024-09-21T10:52:00.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:00.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.857131+0000", 2024-09-21T10:52:00.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0031028890000000002 2024-09-21T10:52:00.684 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.684 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:00.684 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.685 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:130901 55.d 55:be5bbebe:::benchmark_data_smithi070_294394_object25560:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1449)", 2024-09-21T10:52:00.685 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:28.672601+0000", 2024-09-21T10:52:00.685 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 31.913784854999999, 2024-09-21T10:52:00.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.188728971, 2024-09-21T10:52:00.685 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:00.685 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:52:00.685 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:00.685 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:52:00.685 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:52:00.685 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 130901 2024-09-21T10:52:00.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.686 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:00.686 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.686 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:00.686 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.672601+0000", 2024-09-21T10:52:00.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.686 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.686 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.686 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:00.686 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.672601+0000", 2024-09-21T10:52:00.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.686 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.686 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.687 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:00.687 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.672599+0000", 2024-09-21T10:52:00.687 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:52:00.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.689 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:00.690 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.672609+0000", 2024-09-21T10:52:00.690 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5559999999999996e-06 2024-09-21T10:52:00.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.690 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.690 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:00.690 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.672622+0000", 2024-09-21T10:52:00.690 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3148e-05 2024-09-21T10:52:00.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.690 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.690 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:00.690 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.672630+0000", 2024-09-21T10:52:00.690 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1720000000000007e-06 2024-09-21T10:52:00.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.691 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.691 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.751914+0000", 2024-09-21T10:52:00.691 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.079284350000000003 2024-09-21T10:52:00.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.691 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:52:00.691 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.752435+0000", 2024-09-21T10:52:00.691 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00052104899999999999 2024-09-21T10:52:00.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.692 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.692 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.795817+0000", 2024-09-21T10:52:00.692 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.043381757999999999 2024-09-21T10:52:00.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.692 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:00.692 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.795884+0000", 2024-09-21T10:52:00.692 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7234e-05 2024-09-21T10:52:00.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.692 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3,6", 2024-09-21T10:52:00.692 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.795982+0000", 2024-09-21T10:52:00.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8054999999999995e-05 2024-09-21T10:52:00.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.693 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.693 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:52:00.693 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.814327+0000", 2024-09-21T10:52:00.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.018344580999999999 2024-09-21T10:52:00.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.693 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.693 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.693 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.814478+0000", 2024-09-21T10:52:00.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015079700000000001 2024-09-21T10:52:00.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.693 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.693 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.694 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.814525+0000", 2024-09-21T10:52:00.694 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7110000000000001e-05 2024-09-21T10:52:00.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.694 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.694 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.694 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.858640+0000", 2024-09-21T10:52:00.694 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.044115190999999998 2024-09-21T10:52:00.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.694 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.694 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:52:00.694 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.858707+0000", 2024-09-21T10:52:00.694 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7298999999999994e-05 2024-09-21T10:52:00.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.695 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.695 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:00.695 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.861330+0000", 2024-09-21T10:52:00.695 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0026224769999999998 2024-09-21T10:52:00.695 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.695 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:00.695 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.695 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.695 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.695 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:134230 55.d 55:bfc51a9c:::benchmark_data_smithi070_294394_object28889:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1452)", 2024-09-21T10:52:00.696 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:36.126464+0000", 2024-09-21T10:52:00.697 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 24.459921639000001, 2024-09-21T10:52:00.697 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.17098775599999999, 2024-09-21T10:52:00.697 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:00.697 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:52:00.697 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:00.697 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:52:00.697 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:52:00.697 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 134230 2024-09-21T10:52:00.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.697 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:00.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.697 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:00.697 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:36.126464+0000", 2024-09-21T10:52:00.698 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.698 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:00.698 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:36.126464+0000", 2024-09-21T10:52:00.698 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.698 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:00.698 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:36.126462+0000", 2024-09-21T10:52:00.698 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:52:00.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:00.699 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:36.126470+0000", 2024-09-21T10:52:00.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4670000000000004e-06 2024-09-21T10:52:00.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.699 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:00.699 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:36.126484+0000", 2024-09-21T10:52:00.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3040999999999999e-05 2024-09-21T10:52:00.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.699 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:00.699 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:36.126491+0000", 2024-09-21T10:52:00.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6459999999999993e-06 2024-09-21T10:52:00.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:36.238842+0000", 2024-09-21T10:52:00.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.112350588 2024-09-21T10:52:00.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:52:00.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:36.239439+0000", 2024-09-21T10:52:00.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00059686799999999996 2024-09-21T10:52:00.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:36.260237+0000", 2024-09-21T10:52:00.701 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020798609999999999 2024-09-21T10:52:00.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.701 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:00.701 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:36.260340+0000", 2024-09-21T10:52:00.701 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000102256 2024-09-21T10:52:00.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.701 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3,6", 2024-09-21T10:52:00.701 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:36.260793+0000", 2024-09-21T10:52:00.701 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00045344500000000002 2024-09-21T10:52:00.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.701 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.702 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:36.275442+0000", 2024-09-21T10:52:00.702 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.014648800999999999 2024-09-21T10:52:00.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.702 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.702 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:52:00.702 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:36.275450+0000", 2024-09-21T10:52:00.702 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1499999999999999e-06 2024-09-21T10:52:00.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.702 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.702 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.702 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:36.275483+0000", 2024-09-21T10:52:00.702 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3105000000000002e-05 2024-09-21T10:52:00.703 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.703 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.703 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.703 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:36.293669+0000", 2024-09-21T10:52:00.703 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.018185791 2024-09-21T10:52:00.703 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.703 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.703 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:52:00.703 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:36.293770+0000", 2024-09-21T10:52:00.703 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010130500000000001 2024-09-21T10:52:00.703 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.703 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.703 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:00.704 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:36.297452+0000", 2024-09-21T10:52:00.704 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0036816599999999998 2024-09-21T10:52:00.704 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.704 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:00.704 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.704 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.704 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.704 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:143954 55.d 55:b6667833:::benchmark_data_smithi070_294394_object38613:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1454)", 2024-09-21T10:52:00.704 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:56.379118+0000", 2024-09-21T10:52:00.704 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 4.2072674729999999, 2024-09-21T10:52:00.704 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.17386157799999999, 2024-09-21T10:52:00.704 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:00.704 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:52:00.705 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:00.705 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:52:00.705 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:52:00.705 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 143954 2024-09-21T10:52:00.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.705 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:00.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.705 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:00.705 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.379118+0000", 2024-09-21T10:52:00.705 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.705 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:00.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.379118+0000", 2024-09-21T10:52:00.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:00.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.379117+0000", 2024-09-21T10:52:00.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:52:00.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:00.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.379123+0000", 2024-09-21T10:52:00.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0680000000000002e-06 2024-09-21T10:52:00.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.707 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:00.707 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.379135+0000", 2024-09-21T10:52:00.707 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2072000000000001e-05 2024-09-21T10:52:00.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.707 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:00.707 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.379140+0000", 2024-09-21T10:52:00.707 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.49e-06 2024-09-21T10:52:00.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.707 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.707 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.379189+0000", 2024-09-21T10:52:00.708 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8773000000000002e-05 2024-09-21T10:52:00.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.708 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:52:00.708 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.379790+0000", 2024-09-21T10:52:00.708 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00060095400000000001 2024-09-21T10:52:00.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.708 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.708 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.405596+0000", 2024-09-21T10:52:00.708 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.025805792000000001 2024-09-21T10:52:00.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.708 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:00.709 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.405679+0000", 2024-09-21T10:52:00.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2974999999999998e-05 2024-09-21T10:52:00.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.709 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3,6", 2024-09-21T10:52:00.709 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.405809+0000", 2024-09-21T10:52:00.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012978800000000001 2024-09-21T10:52:00.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.709 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:52:00.709 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.411959+0000", 2024-09-21T10:52:00.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0061498990000000003 2024-09-21T10:52:00.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.710 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.710 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.413049+0000", 2024-09-21T10:52:00.710 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001090513 2024-09-21T10:52:00.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.710 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.710 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.413125+0000", 2024-09-21T10:52:00.710 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5541000000000001e-05 2024-09-21T10:52:00.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.711 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.711 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.549268+0000", 2024-09-21T10:52:00.711 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.136143653 2024-09-21T10:52:00.711 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.711 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.711 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:52:00.711 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.549357+0000", 2024-09-21T10:52:00.712 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8344000000000007e-05 2024-09-21T10:52:00.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:00.712 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.552980+0000", 2024-09-21T10:52:00.712 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0036230559999999999 2024-09-21T10:52:00.712 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.712 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:00.712 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.713 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:144058 55.d 55:b880c8e7:::benchmark_data_smithi070_294394_object38717:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1454)", 2024-09-21T10:52:00.713 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:56.685961+0000", 2024-09-21T10:52:00.713 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 3.9004246199999999, 2024-09-21T10:52:00.713 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16440098, 2024-09-21T10:52:00.713 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:00.713 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:52:00.713 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:00.713 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:52:00.713 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:52:00.713 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 144058 2024-09-21T10:52:00.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.714 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:00.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:00.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.685961+0000", 2024-09-21T10:52:00.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:00.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.685961+0000", 2024-09-21T10:52:00.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:00.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.685959+0000", 2024-09-21T10:52:00.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:52:00.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:00.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.685966+0000", 2024-09-21T10:52:00.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6789999999999999e-06 2024-09-21T10:52:00.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:00.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.685978+0000", 2024-09-21T10:52:00.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2247e-05 2024-09-21T10:52:00.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.716 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:00.716 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.685987+0000", 2024-09-21T10:52:00.716 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3699999999999995e-06 2024-09-21T10:52:00.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.716 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.716 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.689311+0000", 2024-09-21T10:52:00.716 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003324427 2024-09-21T10:52:00.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.716 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:52:00.716 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.689791+0000", 2024-09-21T10:52:00.717 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00048010199999999999 2024-09-21T10:52:00.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.717 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.717 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.717 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.834451+0000", 2024-09-21T10:52:00.717 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.14466015700000001 2024-09-21T10:52:00.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.717 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.717 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:00.717 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.834573+0000", 2024-09-21T10:52:00.717 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012110199999999999 2024-09-21T10:52:00.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.717 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.717 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3,6", 2024-09-21T10:52:00.718 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.834883+0000", 2024-09-21T10:52:00.718 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00031017799999999998 2024-09-21T10:52:00.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.718 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:52:00.718 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.839899+0000", 2024-09-21T10:52:00.718 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0050163600000000001 2024-09-21T10:52:00.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.718 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.718 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.843519+0000", 2024-09-21T10:52:00.718 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0036194909999999999 2024-09-21T10:52:00.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.719 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.719 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.719 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.843551+0000", 2024-09-21T10:52:00.719 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2616000000000001e-05 2024-09-21T10:52:00.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.719 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.719 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.719 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.846493+0000", 2024-09-21T10:52:00.719 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029413709999999999 2024-09-21T10:52:00.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.719 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.719 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:52:00.719 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.846542+0000", 2024-09-21T10:52:00.720 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9599999999999999e-05 2024-09-21T10:52:00.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.720 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.720 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:00.720 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.850362+0000", 2024-09-21T10:52:00.720 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003819851 2024-09-21T10:52:00.720 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.720 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:00.720 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.720 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.720 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:144061 55.d 55:bc83d20b:::benchmark_data_smithi070_294394_object38720:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1454)", 2024-09-21T10:52:00.720 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:56.698731+0000", 2024-09-21T10:52:00.721 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 3.8876549110000003, 2024-09-21T10:52:00.721 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.17595287400000001, 2024-09-21T10:52:00.721 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:00.721 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:52:00.721 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:00.721 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:52:00.721 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:52:00.721 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 144061 2024-09-21T10:52:00.721 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.721 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:00.721 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.721 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:00.721 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.698731+0000", 2024-09-21T10:52:00.721 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.722 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.722 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.722 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:00.722 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.698731+0000", 2024-09-21T10:52:00.722 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:00.722 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.722 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.722 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:00.722 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.698729+0000", 2024-09-21T10:52:00.722 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:52:00.722 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.722 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.722 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:00.722 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.698735+0000", 2024-09-21T10:52:00.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3569999999999997e-06 2024-09-21T10:52:00.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.723 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:00.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.698743+0000", 2024-09-21T10:52:00.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1780000000000003e-06 2024-09-21T10:52:00.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.723 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:00.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.698748+0000", 2024-09-21T10:52:00.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1340000000000003e-06 2024-09-21T10:52:00.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.837256+0000", 2024-09-21T10:52:00.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13850782 2024-09-21T10:52:00.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:52:00.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.837859+0000", 2024-09-21T10:52:00.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00060289100000000004 2024-09-21T10:52:00.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:00.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.858121+0000", 2024-09-21T10:52:00.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020262199000000002 2024-09-21T10:52:00.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.725 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:00.725 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.858233+0000", 2024-09-21T10:52:00.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000111845 2024-09-21T10:52:00.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.725 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3,6", 2024-09-21T10:52:00.725 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.858373+0000", 2024-09-21T10:52:00.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000140097 2024-09-21T10:52:00.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.725 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.725 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.869655+0000", 2024-09-21T10:52:00.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011282163 2024-09-21T10:52:00.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.726 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:52:00.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.869670+0000", 2024-09-21T10:52:00.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4506e-05 2024-09-21T10:52:00.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.726 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.869735+0000", 2024-09-21T10:52:00.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5648999999999994e-05 2024-09-21T10:52:00.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.727 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:00.727 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.871475+0000", 2024-09-21T10:52:00.727 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001739481 2024-09-21T10:52:00.727 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.727 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:52:00.727 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.871527+0000", 2024-09-21T10:52:00.727 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2812999999999997e-05 2024-09-21T10:52:00.727 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:00.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:00.727 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:00.727 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:56.874684+0000", 2024-09-21T10:52:00.727 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0031561359999999999 2024-09-21T10:52:00.728 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.728 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:00.728 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.728 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:00.728 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:00.728 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:52:00.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:00.729 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:00.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:00.830 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:00.862 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:00.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:00.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:00.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:00 smithi070 ceph-mon[107385]: pgmap v2647: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 398 op/s; 48399/200400 objects degraded (24.151%); 27867/200400 objects misplaced (13.906%); 11 MiB/s, 173 objects/s recovering 2024-09-21T10:52:00.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:00 smithi070 ceph-mon[112139]: pgmap v2647: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 398 op/s; 48399/200400 objects degraded (24.151%); 27867/200400 objects misplaced (13.906%); 11 MiB/s, 173 objects/s recovering 2024-09-21T10:52:00.964 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:00.993 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:00.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:00.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:01.095 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:01.137 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:01.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:01.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:01.238 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:01.278 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:01.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:01.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:01.380 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:01.417 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:01.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:01.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:01.519 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:01.558 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:01.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:01.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:01.660 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:01.691 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:52:01.700 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:01.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:01.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:01.802 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:01.832 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:01.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:01.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:01.934 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:01.970 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:01.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:01.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:01.980 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:52:02.071 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:02.112 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:02.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:02.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:02.213 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:02.252 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:02.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:02.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:02.353 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:02.382 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:02.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:02.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:02.443 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:52:02.482 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T10:52:02.484 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:02.486 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T10:52:02.512 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:02.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:02.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:02.614 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:02.648 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:02.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:02.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:02.749 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:02.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:02 smithi070 ceph-mon[107385]: pgmap v2648: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 398 op/s; 48399/200400 objects degraded (24.151%); 27867/200400 objects misplaced (13.906%); 11 MiB/s, 173 objects/s recovering 2024-09-21T10:52:02.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:02 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2631299892' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:52:02.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:02 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:52:02.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:02 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2825991548' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:52:02.753 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:02 smithi070 ceph-mon[112139]: pgmap v2648: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 398 op/s; 48399/200400 objects degraded (24.151%); 27867/200400 objects misplaced (13.906%); 11 MiB/s, 173 objects/s recovering 2024-09-21T10:52:02.754 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:02 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2631299892' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:52:02.754 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:02 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:52:02.754 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:02 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2825991548' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:52:02.771 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:02.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:02.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:02 smithi160 ceph-mon[101879]: pgmap v2648: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 398 op/s; 48399/200400 objects degraded (24.151%); 27867/200400 objects misplaced (13.906%); 11 MiB/s, 173 objects/s recovering 2024-09-21T10:52:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:02 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2631299892' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:52:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:02 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:52:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:02 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2825991548' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:52:02.873 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:02.894 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:02.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:02.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:02.996 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:03.017 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:03.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:03.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:03.119 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:03.141 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:03.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:03.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:03.243 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:03.265 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:03.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:03.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:03.367 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:03.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:52:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:52:03] "GET /metrics HTTP/1.1" 200 36671 "" "Prometheus/2.51.0" 2024-09-21T10:52:03.389 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:03.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:03.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:03.491 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:03.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:03.524 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:03.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:03.545 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 4 1 2024-09-21T10:52:03.625 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:03.657 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:03.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:03.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:03.759 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:03.802 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:03.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:03.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:03 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:52:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:03 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:52:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:03 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2825991548' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:52:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:03 smithi160 ceph-mon[101879]: osdmap e1455: 8 total, 8 up, 5 in 2024-09-21T10:52:03.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:03 smithi160 ceph-mon[101879]: osdmap e1456: 8 total, 8 up, 5 in 2024-09-21T10:52:03.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:03 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:52:03.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:03 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:52:03.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:03 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2825991548' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:52:03.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:03 smithi070 ceph-mon[107385]: osdmap e1455: 8 total, 8 up, 5 in 2024-09-21T10:52:03.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:03 smithi070 ceph-mon[107385]: osdmap e1456: 8 total, 8 up, 5 in 2024-09-21T10:52:03.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:03 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:52:03.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:03 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:52:03.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:03 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2825991548' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:52:03.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:03 smithi070 ceph-mon[112139]: osdmap e1455: 8 total, 8 up, 5 in 2024-09-21T10:52:03.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:03 smithi070 ceph-mon[112139]: osdmap e1456: 8 total, 8 up, 5 in 2024-09-21T10:52:03.904 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:03.946 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:03.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:03.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:04.048 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:04.118 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:04.124 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:04.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:04.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:04.151 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:04.226 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:04.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:04.276 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:04.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:04.377 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:04.412 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:04.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:04.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:04.514 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:04.524 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:04 smithi070 ceph-mon[107385]: pgmap v2650: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.9 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 470 op/s; 46990/193311 objects degraded (24.308%); 26978/193311 objects misplaced (13.956%); 14 MiB/s, 219 objects/s recovering 2024-09-21T10:52:04.524 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:04 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 46990/193311 objects degraded (24.308%), 13 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:52:04.524 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:04 smithi070 ceph-mon[107385]: osdmap e1457: 8 total, 8 up, 5 in 2024-09-21T10:52:04.524 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:04 smithi070 ceph-mon[112139]: pgmap v2650: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.9 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 470 op/s; 46990/193311 objects degraded (24.308%); 26978/193311 objects misplaced (13.956%); 14 MiB/s, 219 objects/s recovering 2024-09-21T10:52:04.524 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:04 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 46990/193311 objects degraded (24.308%), 13 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:52:04.524 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:04 smithi070 ceph-mon[112139]: osdmap e1457: 8 total, 8 up, 5 in 2024-09-21T10:52:04.549 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:04.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:04.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:04.651 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:04.678 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:04.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:04.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:04.780 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:04.806 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:04 smithi160 ceph-mon[101879]: pgmap v2650: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.9 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 470 op/s; 46990/193311 objects degraded (24.308%); 26978/193311 objects misplaced (13.956%); 14 MiB/s, 219 objects/s recovering 2024-09-21T10:52:04.806 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:04 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 46990/193311 objects degraded (24.308%), 13 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:52:04.806 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:04 smithi160 ceph-mon[101879]: osdmap e1457: 8 total, 8 up, 5 in 2024-09-21T10:52:04.813 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:04.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:04.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:04.915 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:04.956 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:04.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:04.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:05.058 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:05.102 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:05.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:05.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:05.204 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:05.228 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:05.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:05.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:05.330 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:05.335 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:52:05.336 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T10:52:05.336 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T10:52:05.336 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:52:05.353 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:05.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:05.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:05.455 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:05.487 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:05.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:05.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:05.589 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:05.618 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:05.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:05.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:05.720 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:05.741 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:05.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:05.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:05.842 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:05.865 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:05.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:05.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:05.966 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:05.992 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:05.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:05.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:05.995 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:52:06.094 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:06.120 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:06.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:06.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:06.222 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:06.249 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:06.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:06.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:06 smithi160 ceph-mon[101879]: pgmap v2653: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.9 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 393 op/s; 46990/193311 objects degraded (24.308%); 26978/193311 objects misplaced (13.956%); 12 MiB/s, 192 objects/s recovering 2024-09-21T10:52:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:06 smithi160 ceph-mon[101879]: osdmap e1458: 8 total, 8 up, 5 in 2024-09-21T10:52:06.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:52:06.352 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:06.386 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:06.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:06.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:06.391 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:52:06.488 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:06.519 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:06.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:06.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:06.613 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:06 smithi070 ceph-mon[107385]: pgmap v2653: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.9 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 393 op/s; 46990/193311 objects degraded (24.308%); 26978/193311 objects misplaced (13.956%); 12 MiB/s, 192 objects/s recovering 2024-09-21T10:52:06.614 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:06 smithi070 ceph-mon[107385]: osdmap e1458: 8 total, 8 up, 5 in 2024-09-21T10:52:06.614 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:52:06.614 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:06 smithi070 ceph-mon[112139]: pgmap v2653: 99 pgs: 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.9 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 393 op/s; 46990/193311 objects degraded (24.308%); 26978/193311 objects misplaced (13.956%); 12 MiB/s, 192 objects/s recovering 2024-09-21T10:52:06.614 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:06 smithi070 ceph-mon[112139]: osdmap e1458: 8 total, 8 up, 5 in 2024-09-21T10:52:06.614 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:52:06.621 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:06.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:06.654 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:06.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:06.680 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:52:06.755 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:06.797 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:06.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:06.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:06.900 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:06.927 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:06.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:06.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:07.029 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:07.057 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:07.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:07.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:07.159 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:07.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:07.212 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:07.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:07.313 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:07.344 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:07.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:07.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:07.434 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:07 smithi070 ceph-mon[107385]: osdmap e1459: 8 total, 8 up, 5 in 2024-09-21T10:52:07.435 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:07 smithi070 ceph-mon[112139]: osdmap e1459: 8 total, 8 up, 5 in 2024-09-21T10:52:07.445 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:07.489 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:07.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:07.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:07.518 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:52:07.518 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:52:07.555 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:52:07.555 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:52:07.591 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:07.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:07 smithi160 ceph-mon[101879]: osdmap e1459: 8 total, 8 up, 5 in 2024-09-21T10:52:07.593 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:52:07.594 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:52:07.630 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:07.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:07.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:07.648 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:52:07.649 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:52:07.683 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:52:07.684 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:52:07.732 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:07.732 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:52:07.733 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:52:07.763 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:07.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:07.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:07.774 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:52:07.775 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:52:07.809 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:07.825 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:52:07.825 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:52:07.865 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:07.891 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:07.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:07.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:07.993 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:08.030 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:08.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:08.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:08.132 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:08.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:08.164 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:08.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:08.265 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:08.297 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:08.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:08.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:08.350 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T10:52:08.399 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:08.446 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:08.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:08.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:08.517 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:08 smithi070 ceph-mon[107385]: pgmap v2656: 99 pgs: 1 active+recovering+undersized+remapped, 1 peering, 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 432 op/s; 38119/188121 objects degraded (20.263%); 26361/188121 objects misplaced (14.013%); 18 MiB/s, 293 objects/s recovering 2024-09-21T10:52:08.517 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:08 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1428404595' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:52:08.517 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:08 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/686904683' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:52:08.517 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:08 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:52:08.517 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:08 smithi070 ceph-mon[112139]: pgmap v2656: 99 pgs: 1 active+recovering+undersized+remapped, 1 peering, 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 432 op/s; 38119/188121 objects degraded (20.263%); 26361/188121 objects misplaced (14.013%); 18 MiB/s, 293 objects/s recovering 2024-09-21T10:52:08.517 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:08 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1428404595' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:52:08.517 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:08 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/686904683' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:52:08.517 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:08 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:52:08.548 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:08.573 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:08 smithi160 ceph-mon[101879]: pgmap v2656: 99 pgs: 1 active+recovering+undersized+remapped, 1 peering, 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 432 op/s; 38119/188121 objects degraded (20.263%); 26361/188121 objects misplaced (14.013%); 18 MiB/s, 293 objects/s recovering 2024-09-21T10:52:08.574 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:08 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1428404595' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:52:08.574 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:08 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/686904683' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:52:08.574 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:08 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:52:08.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:08.591 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:08.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:08.692 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:08.731 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:08.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:08.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:08.832 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:08.872 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:08.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:08.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:08.961 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:52:08.962 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:52:08.974 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:09.006 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:09.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:09.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:09.108 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:09.135 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:09.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:09.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:09.237 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:09.277 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:09.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:09.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:09.379 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:09.412 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:09.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:09.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:09.514 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:09.547 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:09.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:09.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:09.600 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:09 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:52:09.600 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:09 smithi070 ceph-mon[107385]: osdmap e1460: 8 total, 8 up, 5 in 2024-09-21T10:52:09.600 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:09 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:52:09.600 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:09 smithi070 ceph-mon[112139]: osdmap e1460: 8 total, 8 up, 5 in 2024-09-21T10:52:09.649 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:09.694 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:09.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:09.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:09.795 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:09.822 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:09.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:09.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:09 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:52:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:09 smithi160 ceph-mon[101879]: osdmap e1460: 8 total, 8 up, 5 in 2024-09-21T10:52:09.924 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:09.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:09.967 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:09.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:10.021 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T10:52:10.068 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:10.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:10.117 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:10.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:10.218 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:10.243 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:10.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:10.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:10.345 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:10.360 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:52:10.375 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:10.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:10.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:10.476 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:10.505 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:10.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:10.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:10.549 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:10 smithi160 ceph-mon[101879]: pgmap v2658: 99 pgs: 1 active+recovering+undersized+remapped, 1 peering, 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 376 op/s; 38119/188121 objects degraded (20.263%); 26361/188121 objects misplaced (14.013%); 16 MiB/s, 255 objects/s recovering 2024-09-21T10:52:10.606 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:10.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:10 smithi070 ceph-mon[107385]: pgmap v2658: 99 pgs: 1 active+recovering+undersized+remapped, 1 peering, 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 376 op/s; 38119/188121 objects degraded (20.263%); 26361/188121 objects misplaced (14.013%); 16 MiB/s, 255 objects/s recovering 2024-09-21T10:52:10.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:10 smithi070 ceph-mon[112139]: pgmap v2658: 99 pgs: 1 active+recovering+undersized+remapped, 1 peering, 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 376 op/s; 38119/188121 objects degraded (20.263%); 26361/188121 objects misplaced (14.013%); 16 MiB/s, 255 objects/s recovering 2024-09-21T10:52:10.632 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:52:10.653 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:10.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:10.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:10.755 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:10.792 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:10.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:10.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:10.894 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:10.924 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:10.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:10.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:10.969 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:52:11.026 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:11.050 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:11.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:11.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:11.152 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:11.202 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:11.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:11.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:11.304 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:11.331 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:11.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:11.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:11.433 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:11.468 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:11.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:11.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:11.570 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:11.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:11.619 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:11.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:11.720 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:11.759 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:11.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:11.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:11.861 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:11.887 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:11.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:11.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:11.989 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:12.025 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:12.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:12.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:12.126 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:12.152 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:12.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:12.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:12.254 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:12.290 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:12.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:12.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:12.377 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T10:52:12.391 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:12.417 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:12.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:12.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:12.519 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:12.520 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:12 smithi160 ceph-mon[101879]: pgmap v2659: 99 pgs: 1 active+recovering+undersized+remapped, 1 peering, 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 288 op/s; 38119/188121 objects degraded (20.263%); 26361/188121 objects misplaced (14.013%); 12 MiB/s, 195 objects/s recovering 2024-09-21T10:52:12.520 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:12 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/604784119' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:52:12.545 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:52:12.545 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:52:12.548 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:12.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:12.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:12 smithi070 ceph-mon[107385]: pgmap v2659: 99 pgs: 1 active+recovering+undersized+remapped, 1 peering, 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 288 op/s; 38119/188121 objects degraded (20.263%); 26361/188121 objects misplaced (14.013%); 12 MiB/s, 195 objects/s recovering 2024-09-21T10:52:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:12 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/604784119' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:52:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:12 smithi070 ceph-mon[112139]: pgmap v2659: 99 pgs: 1 active+recovering+undersized+remapped, 1 peering, 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 288 op/s; 38119/188121 objects degraded (20.263%); 26361/188121 objects misplaced (14.013%); 12 MiB/s, 195 objects/s recovering 2024-09-21T10:52:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:12 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/604784119' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:52:12.650 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:12.694 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:12.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:12.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:12.765 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:12.795 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:12.826 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:12.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:12.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:12.928 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:12.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:12.970 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:12.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:13.071 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:13.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:13.127 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:13.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:13.230 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:13.240 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:52:13.274 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:13.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:13.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:13.334 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:52:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:52:13] "GET /metrics HTTP/1.1" 200 36659 "" "Prometheus/2.51.0" 2024-09-21T10:52:13.375 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:13.415 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:13.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:13.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:13.517 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:13.532 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:52:13.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:13.563 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:13.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:13.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:13 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:52:13.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:13 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/604784119' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:52:13.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:13 smithi160 ceph-mon[101879]: osdmap e1461: 8 total, 8 up, 5 in 2024-09-21T10:52:13.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:13 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 38119/188121 objects degraded (20.263%), 13 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:52:13.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:13 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:52:13.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:13 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/604784119' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:52:13.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:13 smithi070 ceph-mon[107385]: osdmap e1461: 8 total, 8 up, 5 in 2024-09-21T10:52:13.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:13 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 38119/188121 objects degraded (20.263%), 13 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:52:13.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:13 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:52:13.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:13 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/604784119' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:52:13.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:13 smithi070 ceph-mon[112139]: osdmap e1461: 8 total, 8 up, 5 in 2024-09-21T10:52:13.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:13 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 38119/188121 objects degraded (20.263%), 13 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:52:13.664 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:13.691 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:13.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:13.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:13.793 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:13.818 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:13.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:13.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:13.920 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:13.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:13.969 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:13.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:14.069 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:14.108 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:14.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:14.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:14.210 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:14.231 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:14.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:14.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:14.333 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:14.364 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:14.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:14.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:14.399 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T10:52:14.466 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:14.504 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:14.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:14.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:14.606 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:14.629 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:14.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:14.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:14.731 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:14.754 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:14.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:14.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:14 smithi160 ceph-mon[101879]: pgmap v2661: 99 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.7 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 488 op/s; 36806/180813 objects degraded (20.356%); 25380/180813 objects misplaced (14.037%); 17 MiB/s, 283 objects/s recovering 2024-09-21T10:52:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:14 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2273102564' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:52:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:14 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:52:14.856 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:14.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:14 smithi070 ceph-mon[107385]: pgmap v2661: 99 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.7 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 488 op/s; 36806/180813 objects degraded (20.356%); 25380/180813 objects misplaced (14.037%); 17 MiB/s, 283 objects/s recovering 2024-09-21T10:52:14.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:14 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2273102564' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:52:14.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:14 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:52:14.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:14 smithi070 ceph-mon[112139]: pgmap v2661: 99 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.7 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 488 op/s; 36806/180813 objects degraded (20.356%); 25380/180813 objects misplaced (14.037%); 17 MiB/s, 283 objects/s recovering 2024-09-21T10:52:14.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:14 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2273102564' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:52:14.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:14 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:52:14.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:14.878 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:14.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:14.979 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:15.002 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:15.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:15.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:15.103 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:15.126 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:15.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:15.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:15.227 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:15.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:15.249 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:15.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:15.350 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:15.372 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:15.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:15.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:15.474 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:15.496 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:15.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:15.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:15.540 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T10:52:15.598 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:15.619 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:15.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:15.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:15.680 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:15 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:52:15.680 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:15 smithi070 ceph-mon[107385]: osdmap e1462: 8 total, 8 up, 5 in 2024-09-21T10:52:15.680 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:15 smithi070 ceph-mon[107385]: osdmap e1463: 8 total, 8 up, 5 in 2024-09-21T10:52:15.680 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:15 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:52:15.680 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:15 smithi070 ceph-mon[112139]: osdmap e1462: 8 total, 8 up, 5 in 2024-09-21T10:52:15.680 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:15 smithi070 ceph-mon[112139]: osdmap e1463: 8 total, 8 up, 5 in 2024-09-21T10:52:15.721 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:15.743 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:15.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:15.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:15 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:52:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:15 smithi160 ceph-mon[101879]: osdmap e1462: 8 total, 8 up, 5 in 2024-09-21T10:52:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:15 smithi160 ceph-mon[101879]: osdmap e1463: 8 total, 8 up, 5 in 2024-09-21T10:52:15.845 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:15.865 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:15.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:15.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:15.967 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:15.988 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:15.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:15.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:16.090 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:16.111 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:16.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:16.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:16.213 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:16.234 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:16.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:16.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:16.336 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:16.357 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:16.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:16.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:16.459 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:16.488 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:16.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:16.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:16.520 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:52:16.550 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:16.557 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:16 smithi070 ceph-mon[107385]: pgmap v2663: 99 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.7 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 371 op/s; 36806/180813 objects degraded (20.356%); 25380/180813 objects misplaced (14.037%); 12 MiB/s, 203 objects/s recovering 2024-09-21T10:52:16.558 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:16 smithi070 ceph-mon[107385]: 1.1 scrub starts 2024-09-21T10:52:16.558 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:16 smithi070 ceph-mon[107385]: osdmap e1464: 8 total, 8 up, 5 in 2024-09-21T10:52:16.558 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:16 smithi070 ceph-mon[112139]: pgmap v2663: 99 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.7 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 371 op/s; 36806/180813 objects degraded (20.356%); 25380/180813 objects misplaced (14.037%); 12 MiB/s, 203 objects/s recovering 2024-09-21T10:52:16.558 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:16 smithi070 ceph-mon[112139]: 1.1 scrub starts 2024-09-21T10:52:16.558 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:16 smithi070 ceph-mon[112139]: osdmap e1464: 8 total, 8 up, 5 in 2024-09-21T10:52:16.590 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:16.611 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:52:16.611 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:52:16.615 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:16.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:16.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:16.642 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:52:16.642 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:52:16.669 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:52:16.669 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:52:16.697 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:52:16.697 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:52:16.719 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:16.720 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:16 smithi160 ceph-mon[101879]: pgmap v2663: 99 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.7 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 371 op/s; 36806/180813 objects degraded (20.356%); 25380/180813 objects misplaced (14.037%); 12 MiB/s, 203 objects/s recovering 2024-09-21T10:52:16.720 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:16 smithi160 ceph-mon[101879]: 1.1 scrub starts 2024-09-21T10:52:16.720 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:16 smithi160 ceph-mon[101879]: osdmap e1464: 8 total, 8 up, 5 in 2024-09-21T10:52:16.722 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:52:16.722 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:52:16.742 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:16.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:16.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:16.753 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:52:16.753 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:52:16.778 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:52:16.778 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:52:16.807 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:52:16.808 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:52:16.843 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:16.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:16.878 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:16.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:16.979 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:17.015 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:17.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:17.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:17.117 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:17.154 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:17.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:17.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:17.256 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:17.287 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:17.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:17.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:17.389 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:17.422 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:17.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:17.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:17.524 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:17.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:17.583 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:17.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:17.639 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:17 smithi160 ceph-mon[101879]: 55.5 continuing backfill to osd.3 from (1432'5614,1461'9464] MIN to 1461'9464 2024-09-21T10:52:17.639 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:17 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1364929260' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:52:17.669 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:52:17.669 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T10:52:17.670 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.670 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:152542 55.8 55:14ba591b:::benchmark_data_smithi070_294394_object47201:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1464)", 2024-09-21T10:52:17.670 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:17.659147+0000", 2024-09-21T10:52:17.670 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0078400659999999997, 2024-09-21T10:52:17.670 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0078710959999999993, 2024-09-21T10:52:17.670 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:17.670 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:52:17.670 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:17.670 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:52:17.670 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:52:17.670 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 152542 2024-09-21T10:52:17.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.670 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:17.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.671 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:17.671 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.659147+0000", 2024-09-21T10:52:17.671 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:17.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.671 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:17.671 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.659147+0000", 2024-09-21T10:52:17.671 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:17.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.672 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:17.672 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.659145+0000", 2024-09-21T10:52:17.672 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:52:17.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.672 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.672 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:17.672 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.659154+0000", 2024-09-21T10:52:17.672 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4489999999999999e-06 2024-09-21T10:52:17.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.672 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.672 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:17.672 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.659166+0000", 2024-09-21T10:52:17.672 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2454e-05 2024-09-21T10:52:17.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.673 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:17.673 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.659174+0000", 2024-09-21T10:52:17.673 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1419999999999993e-06 2024-09-21T10:52:17.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.673 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:17.673 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.661031+0000", 2024-09-21T10:52:17.673 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001856655 2024-09-21T10:52:17.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.673 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:17.673 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.661180+0000", 2024-09-21T10:52:17.673 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014893799999999999 2024-09-21T10:52:17.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.674 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.674 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1,3", 2024-09-21T10:52:17.674 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.661332+0000", 2024-09-21T10:52:17.674 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015233799999999999 2024-09-21T10:52:17.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.674 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.674 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:52:17.674 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.666282+0000", 2024-09-21T10:52:17.674 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0049494350000000003 2024-09-21T10:52:17.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.674 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.674 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:17.675 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.666474+0000", 2024-09-21T10:52:17.675 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000191967 2024-09-21T10:52:17.675 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:17.675 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:17.675 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:17.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.675 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.675 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.35670.0:152524 55.2 e1464/1463 55:4a3b51bb:::benchmark_data_smithi070_294394_object47183:head v 1464'9372, mlcod=0'0)", 2024-09-21T10:52:17.675 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:17.663679+0000", 2024-09-21T10:52:17.675 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0033083570000000001, 2024-09-21T10:52:17.675 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003429456, 2024-09-21T10:52:17.675 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:17.675 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T10:52:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.663679+0000", 2024-09-21T10:52:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.663679+0000", 2024-09-21T10:52:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.663678+0000", 2024-09-21T10:52:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:52:17.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:17.677 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.663687+0000", 2024-09-21T10:52:17.677 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4549999999999995e-06 2024-09-21T10:52:17.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:17.677 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.663713+0000", 2024-09-21T10:52:17.677 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6585999999999999e-05 2024-09-21T10:52:17.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:17.677 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.663720+0000", 2024-09-21T10:52:17.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2470000000000002e-06 2024-09-21T10:52:17.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.678 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:17.678 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.663786+0000", 2024-09-21T10:52:17.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6024000000000003e-05 2024-09-21T10:52:17.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.678 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:17.678 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.663813+0000", 2024-09-21T10:52:17.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6131000000000001e-05 2024-09-21T10:52:17.678 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:17.678 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:17.678 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:17.679 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.679 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.679 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:152547 55.b 55:d4ded6cc:::benchmark_data_smithi070_294394_object47206:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1464)", 2024-09-21T10:52:17.679 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:17.666644+0000", 2024-09-21T10:52:17.679 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.00034351500000000002, 2024-09-21T10:52:17.679 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00051929700000000005, 2024-09-21T10:52:17.679 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:17.679 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:52:17.679 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:17.679 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:52:17.679 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:52:17.679 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 152547 2024-09-21T10:52:17.679 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.680 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:17.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:17.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.666644+0000", 2024-09-21T10:52:17.680 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:17.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:17.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.666644+0000", 2024-09-21T10:52:17.680 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:17.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.681 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:17.681 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.666642+0000", 2024-09-21T10:52:17.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:52:17.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.681 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:17.681 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.666649+0000", 2024-09-21T10:52:17.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3459999999999999e-06 2024-09-21T10:52:17.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.681 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:17.681 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.666660+0000", 2024-09-21T10:52:17.682 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1082e-05 2024-09-21T10:52:17.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:17.682 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.666667+0000", 2024-09-21T10:52:17.682 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9079999999999998e-06 2024-09-21T10:52:17.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:17.682 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.666731+0000", 2024-09-21T10:52:17.682 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3967000000000006e-05 2024-09-21T10:52:17.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:17.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.666861+0000", 2024-09-21T10:52:17.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013068199999999999 2024-09-21T10:52:17.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:17.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:17.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,7", 2024-09-21T10:52:17.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:17.666983+0000", 2024-09-21T10:52:17.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000121289 2024-09-21T10:52:17.683 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:17.683 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:17.683 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:17.683 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:17.683 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T10:52:17.683 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 3 2024-09-21T10:52:17.684 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:52:17.684 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:17.725 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:17.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:17.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:17.827 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:17.854 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:17.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:17.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:17.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:17 smithi070 ceph-mon[107385]: 55.5 continuing backfill to osd.3 from (1432'5614,1461'9464] MIN to 1461'9464 2024-09-21T10:52:17.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:17 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1364929260' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:52:17.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:17 smithi070 ceph-mon[112139]: 55.5 continuing backfill to osd.3 from (1432'5614,1461'9464] MIN to 1461'9464 2024-09-21T10:52:17.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:17 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1364929260' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:52:17.956 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:17.992 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:17.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:17.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:18.094 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:18.123 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:18.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:18.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:18.225 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:18.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:18.258 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:18.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:18.359 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:18.402 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:18.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:18.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:18.443 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:52:18.503 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:18.537 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:18.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:18.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:18 smithi070 ceph-mon[107385]: pgmap v2666: 99 pgs: 1 active+recovering+remapped, 1 remapped+peering, 2 peering, 4 active+recovery_wait+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 1 active+clean+scrubbing, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 345 op/s; 33629/176109 objects degraded (19.096%); 13325/176109 objects misplaced (7.566%); 15 MiB/s, 282 objects/s recovering 2024-09-21T10:52:18.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:18 smithi070 ceph-mon[107385]: 1.2 scrub starts 2024-09-21T10:52:18.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:18 smithi070 ceph-mon[107385]: 1.2 scrub ok 2024-09-21T10:52:18.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:18 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 33629/176109 objects degraded (19.096%), 11 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:52:18.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:18 smithi070 ceph-mon[112139]: pgmap v2666: 99 pgs: 1 active+recovering+remapped, 1 remapped+peering, 2 peering, 4 active+recovery_wait+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 1 active+clean+scrubbing, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 345 op/s; 33629/176109 objects degraded (19.096%); 13325/176109 objects misplaced (7.566%); 15 MiB/s, 282 objects/s recovering 2024-09-21T10:52:18.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:18 smithi070 ceph-mon[112139]: 1.2 scrub starts 2024-09-21T10:52:18.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:18 smithi070 ceph-mon[112139]: 1.2 scrub ok 2024-09-21T10:52:18.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:18 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 33629/176109 objects degraded (19.096%), 11 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:52:18.640 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:18.678 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:18.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:18.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:18.737 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:52:18.780 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:18.810 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:18.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:18.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:18 smithi160 ceph-mon[101879]: pgmap v2666: 99 pgs: 1 active+recovering+remapped, 1 remapped+peering, 2 peering, 4 active+recovery_wait+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 1 active+clean+scrubbing, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 345 op/s; 33629/176109 objects degraded (19.096%); 13325/176109 objects misplaced (7.566%); 15 MiB/s, 282 objects/s recovering 2024-09-21T10:52:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:18 smithi160 ceph-mon[101879]: 1.2 scrub starts 2024-09-21T10:52:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:18 smithi160 ceph-mon[101879]: 1.2 scrub ok 2024-09-21T10:52:18.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:18 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 33629/176109 objects degraded (19.096%), 11 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:52:18.912 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:18.941 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:18.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:18.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:19.043 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:19.075 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:19.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:19.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:19.177 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:19.212 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:19.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:19.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:19.314 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:19.348 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:19.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:19.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:19.450 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:19.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:19.486 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:19.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:19.587 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:19.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:19 smithi160 ceph-mon[101879]: 1.5 scrub starts 2024-09-21T10:52:19.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:19.617 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:19.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:19.718 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:19.750 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:19.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:19.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:19.851 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:19.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:19 smithi070 ceph-mon[107385]: 1.5 scrub starts 2024-09-21T10:52:19.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:19 smithi070 ceph-mon[112139]: 1.5 scrub starts 2024-09-21T10:52:19.883 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:19.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:19.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:19.985 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:20.020 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:20.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:20.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:20.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:20.156 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:20.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:20.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:20.244 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:52:20.258 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:20.305 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:20.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:20.366 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:20.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:20.466 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:20.495 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:20.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:20.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:20.525 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:20 smithi070 ceph-mon[107385]: pgmap v2667: 99 pgs: 1 active+recovering+remapped, 1 remapped+peering, 2 peering, 4 active+recovery_wait+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 1 active+clean+scrubbing, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 261 op/s; 33629/176109 objects degraded (19.096%); 13325/176109 objects misplaced (7.566%); 11 MiB/s, 213 objects/s recovering 2024-09-21T10:52:20.525 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:52:20.525 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:52:20.525 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:20 smithi070 ceph-mon[112139]: pgmap v2667: 99 pgs: 1 active+recovering+remapped, 1 remapped+peering, 2 peering, 4 active+recovery_wait+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 1 active+clean+scrubbing, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 261 op/s; 33629/176109 objects degraded (19.096%); 13325/176109 objects misplaced (7.566%); 11 MiB/s, 213 objects/s recovering 2024-09-21T10:52:20.525 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:52:20.525 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:52:20.553 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:20 smithi160 ceph-mon[101879]: pgmap v2667: 99 pgs: 1 active+recovering+remapped, 1 remapped+peering, 2 peering, 4 active+recovery_wait+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 1 active+clean+scrubbing, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 261 op/s; 33629/176109 objects degraded (19.096%); 13325/176109 objects misplaced (7.566%); 11 MiB/s, 213 objects/s recovering 2024-09-21T10:52:20.554 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:52:20.554 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:52:20.597 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:20.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:20.646 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:20.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:20.699 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:52:20.699 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T10:52:20.699 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T10:52:20.699 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T10:52:20.699 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:52:20.747 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:20.772 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:20.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:20.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:20.873 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:20.911 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:20.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:20.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:21.013 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:21.040 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:21.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:21.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:21.142 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:21.173 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:21.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:21.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:21.274 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:21.301 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:21.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:21.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:21.403 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:21.411 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:52:21.427 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:21.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:21.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:21.529 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:21.551 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:21.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:21.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:21.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:21 smithi070 ceph-mon[112139]: 1.1 scrub starts 2024-09-21T10:52:21.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:21 smithi070 ceph-mon[107385]: 1.1 scrub starts 2024-09-21T10:52:21.652 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:21.673 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:52:21.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:21.698 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:21.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:21.799 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:21.804 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:21 smithi160 ceph-mon[101879]: 1.1 scrub starts 2024-09-21T10:52:21.839 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:21.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:21.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:21.941 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:21.975 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:21.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:21.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:22.077 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:22.115 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:22.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:22.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:22.217 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:22.249 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:22.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:22.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:22.351 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:22.372 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:22.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:22.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:22.474 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:22.501 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:22.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:22.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:22.603 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:22.634 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:22.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:22.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:22.736 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:22.770 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:22.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:22.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:22 smithi160 ceph-mon[101879]: pgmap v2668: 99 pgs: 1 active+recovering+remapped, 1 remapped+peering, 2 peering, 4 active+recovery_wait+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 1 active+clean+scrubbing, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 240 op/s; 33629/176109 objects degraded (19.096%); 13325/176109 objects misplaced (7.566%); 10 MiB/s, 196 objects/s recovering 2024-09-21T10:52:22.871 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:22.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:22 smithi070 ceph-mon[107385]: pgmap v2668: 99 pgs: 1 active+recovering+remapped, 1 remapped+peering, 2 peering, 4 active+recovery_wait+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 1 active+clean+scrubbing, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 240 op/s; 33629/176109 objects degraded (19.096%); 13325/176109 objects misplaced (7.566%); 10 MiB/s, 196 objects/s recovering 2024-09-21T10:52:22.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:22 smithi070 ceph-mon[112139]: pgmap v2668: 99 pgs: 1 active+recovering+remapped, 1 remapped+peering, 2 peering, 4 active+recovery_wait+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 1 active+clean+scrubbing, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 240 op/s; 33629/176109 objects degraded (19.096%); 13325/176109 objects misplaced (7.566%); 10 MiB/s, 196 objects/s recovering 2024-09-21T10:52:22.905 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:22.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:22.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:23.007 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:23.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:23.046 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:23.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:23.148 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:23.177 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:23.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:23.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:23.281 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:23.323 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:23.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:23.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:23.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:52:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:52:23] "GET /metrics HTTP/1.1" 200 36648 "" "Prometheus/2.51.0" 2024-09-21T10:52:23.425 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:23.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:23.462 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:23.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:23.563 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:23.592 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:23.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:23.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:23.695 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:23.724 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:23.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:23.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:23.826 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:23.833 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:23 smithi160 ceph-mon[101879]: 1.5 scrub starts 2024-09-21T10:52:23.855 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:23.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:23.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:23.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:23 smithi070 ceph-mon[107385]: 1.5 scrub starts 2024-09-21T10:52:23.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:23 smithi070 ceph-mon[112139]: 1.5 scrub starts 2024-09-21T10:52:23.957 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:23.991 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:23.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:23.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:24.094 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:24.122 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:24.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:24.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:24.224 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:24.254 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:24.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:24.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:24.356 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:24.383 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:24.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:24.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:24.465 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T10:52:24.485 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:24.510 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:24.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:24.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:24.612 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:24.636 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:24.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:24.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:24.691 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:24 smithi070 ceph-mon[107385]: pgmap v2669: 99 pgs: 1 active+recovering+remapped, 4 active+recovery_wait+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.4 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 529 op/s; 38938/168033 objects degraded (23.173%); 12324/168033 objects misplaced (7.334%); 19 MiB/s, 325 objects/s recovering 2024-09-21T10:52:24.691 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:24 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 38938/168033 objects degraded (23.173%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:52:24.691 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:24 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2522164775' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:52:24.691 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:24 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:52:24.691 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:24 smithi070 ceph-mon[107385]: 1.1 scrub starts 2024-09-21T10:52:24.692 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:24 smithi070 ceph-mon[112139]: pgmap v2669: 99 pgs: 1 active+recovering+remapped, 4 active+recovery_wait+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.4 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 529 op/s; 38938/168033 objects degraded (23.173%); 12324/168033 objects misplaced (7.334%); 19 MiB/s, 325 objects/s recovering 2024-09-21T10:52:24.692 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:24 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 38938/168033 objects degraded (23.173%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:52:24.692 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:24 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2522164775' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:52:24.692 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:24 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:52:24.692 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:24 smithi070 ceph-mon[112139]: 1.1 scrub starts 2024-09-21T10:52:24.738 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:24.761 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:24.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:24.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:24 smithi160 ceph-mon[101879]: pgmap v2669: 99 pgs: 1 active+recovering+remapped, 4 active+recovery_wait+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.4 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 529 op/s; 38938/168033 objects degraded (23.173%); 12324/168033 objects misplaced (7.334%); 19 MiB/s, 325 objects/s recovering 2024-09-21T10:52:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:24 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 38938/168033 objects degraded (23.173%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:52:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:24 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2522164775' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:52:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:24 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:52:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:24 smithi160 ceph-mon[101879]: 1.1 scrub starts 2024-09-21T10:52:24.862 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:24.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:24.886 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:24.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:24.987 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:25.010 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:25.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:25.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:25.111 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:25.134 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:25.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:25.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:25.236 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:25.248 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:52:25.248 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T10:52:25.248 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T10:52:25.248 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T10:52:25.248 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.248 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:238167 53.15 53.6bfa1675 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e1353)", 2024-09-21T10:52:25.248 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:43:41.186729+0000", 2024-09-21T10:52:25.248 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 524.05660932199999, 2024-09-21T10:52:25.249 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99012371300000002, 2024-09-21T10:52:25.249 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:25.249 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:52:25.249 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:25.249 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:52:25.249 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:52:25.249 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 238167 2024-09-21T10:52:25.249 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.249 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:25.249 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.249 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:25.249 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.186729+0000", 2024-09-21T10:52:25.250 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.250 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.250 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.250 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:25.250 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.186729+0000", 2024-09-21T10:52:25.250 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.250 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.250 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.250 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:25.250 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.186729+0000", 2024-09-21T10:52:25.250 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T10:52:25.250 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.250 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.251 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:25.251 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.186732+0000", 2024-09-21T10:52:25.251 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6610000000000002e-06 2024-09-21T10:52:25.251 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.251 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.251 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:25.251 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.186750+0000", 2024-09-21T10:52:25.251 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7771999999999999e-05 2024-09-21T10:52:25.251 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.251 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.251 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:25.251 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.186763+0000", 2024-09-21T10:52:25.251 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3217000000000001e-05 2024-09-21T10:52:25.252 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.252 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.252 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.252 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.186817+0000", 2024-09-21T10:52:25.252 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3813000000000001e-05 2024-09-21T10:52:25.252 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.252 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.252 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:25.252 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.186825+0000", 2024-09-21T10:52:25.252 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1359999999999997e-06 2024-09-21T10:52:25.252 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.252 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.252 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.253 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176845+0000", 2024-09-21T10:52:25.253 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.990019749 2024-09-21T10:52:25.253 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.253 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.253 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:25.253 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176853+0000", 2024-09-21T10:52:25.253 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.875e-06 2024-09-21T10:52:25.253 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.253 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:25.253 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.253 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.253 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.253 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:238221 53.15 53.dc332215 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1353)", 2024-09-21T10:52:25.253 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:43:41.214446+0000", 2024-09-21T10:52:25.254 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 524.02889262999997, 2024-09-21T10:52:25.254 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96244874199999997, 2024-09-21T10:52:25.254 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:25.254 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:52:25.254 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:25.254 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:52:25.254 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:52:25.254 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 238221 2024-09-21T10:52:25.254 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.254 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:25.254 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.254 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:25.254 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.214446+0000", 2024-09-21T10:52:25.255 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.255 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.255 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:25.255 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.214446+0000", 2024-09-21T10:52:25.255 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.255 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.255 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:25.255 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.214445+0000", 2024-09-21T10:52:25.255 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:52:25.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.255 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.255 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:25.256 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.214450+0000", 2024-09-21T10:52:25.256 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4410000000000001e-06 2024-09-21T10:52:25.256 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.256 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:25.256 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.214462+0000", 2024-09-21T10:52:25.256 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1854e-05 2024-09-21T10:52:25.256 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.256 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:25.256 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.214468+0000", 2024-09-21T10:52:25.256 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1990000000000003e-06 2024-09-21T10:52:25.256 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.257 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.257 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.257 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.214518+0000", 2024-09-21T10:52:25.257 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9877999999999999e-05 2024-09-21T10:52:25.257 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.257 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.257 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:25.257 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.214527+0000", 2024-09-21T10:52:25.257 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.439e-06 2024-09-21T10:52:25.257 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.257 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.257 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.257 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176884+0000", 2024-09-21T10:52:25.258 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96235778699999996 2024-09-21T10:52:25.258 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.258 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.258 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:25.258 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176895+0000", 2024-09-21T10:52:25.258 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0207000000000001e-05 2024-09-21T10:52:25.258 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.258 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:25.258 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.258 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.258 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.258 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:238230 53.15 53.8764f335 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1353)", 2024-09-21T10:52:25.259 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:43:41.221236+0000", 2024-09-21T10:52:25.259 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 524.02210215299999, 2024-09-21T10:52:25.259 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.955676635, 2024-09-21T10:52:25.259 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:25.259 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:52:25.259 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:25.259 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:52:25.259 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:52:25.259 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 238230 2024-09-21T10:52:25.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.259 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:25.259 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.259 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:25.259 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.221236+0000", 2024-09-21T10:52:25.260 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.260 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.260 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:25.260 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.221236+0000", 2024-09-21T10:52:25.260 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.260 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.260 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:25.260 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.221235+0000", 2024-09-21T10:52:25.260 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:52:25.260 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.261 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:25.261 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.221241+0000", 2024-09-21T10:52:25.261 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1929999999999998e-06 2024-09-21T10:52:25.261 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.261 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.261 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:25.261 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.221258+0000", 2024-09-21T10:52:25.261 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6765e-05 2024-09-21T10:52:25.261 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.261 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.261 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:25.261 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.221265+0000", 2024-09-21T10:52:25.261 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3560000000000004e-06 2024-09-21T10:52:25.261 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.262 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.262 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.262 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.221314+0000", 2024-09-21T10:52:25.262 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9002999999999998e-05 2024-09-21T10:52:25.262 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.262 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.262 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:25.262 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.221323+0000", 2024-09-21T10:52:25.262 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5399999999999996e-06 2024-09-21T10:52:25.262 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.262 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.262 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.263 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176908+0000", 2024-09-21T10:52:25.263 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.95558458300000004 2024-09-21T10:52:25.263 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.263 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.263 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:25.263 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176913+0000", 2024-09-21T10:52:25.263 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2920000000000003e-06 2024-09-21T10:52:25.263 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.263 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:25.263 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.263 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.263 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.263 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:238249 53.15 53.96c6175 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1353)", 2024-09-21T10:52:25.264 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:43:41.237158+0000", 2024-09-21T10:52:25.264 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 524.00618002299996, 2024-09-21T10:52:25.264 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.93976896499999996, 2024-09-21T10:52:25.264 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:25.264 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:52:25.264 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:25.264 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:52:25.264 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:52:25.264 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 238249 2024-09-21T10:52:25.264 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.264 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:25.264 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.264 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:25.265 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.237158+0000", 2024-09-21T10:52:25.265 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.265 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.265 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.265 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:25.265 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.237158+0000", 2024-09-21T10:52:25.265 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.265 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.265 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.265 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:25.265 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.237157+0000", 2024-09-21T10:52:25.265 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:52:25.265 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.266 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.266 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:25.266 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.237163+0000", 2024-09-21T10:52:25.266 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1199999999999999e-06 2024-09-21T10:52:25.266 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.266 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.266 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:25.266 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.237174+0000", 2024-09-21T10:52:25.266 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0957e-05 2024-09-21T10:52:25.266 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.266 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.266 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:25.266 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.237180+0000", 2024-09-21T10:52:25.267 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3330000000000002e-06 2024-09-21T10:52:25.267 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.267 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.267 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.267 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.237224+0000", 2024-09-21T10:52:25.267 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3835999999999998e-05 2024-09-21T10:52:25.267 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.267 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.267 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:25.267 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.237232+0000", 2024-09-21T10:52:25.267 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8599999999999993e-06 2024-09-21T10:52:25.267 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.267 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.267 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.268 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176922+0000", 2024-09-21T10:52:25.268 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.93969054600000002 2024-09-21T10:52:25.268 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.268 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.268 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:25.268 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176927+0000", 2024-09-21T10:52:25.268 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9400000000000001e-06 2024-09-21T10:52:25.268 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.268 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:25.268 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.268 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.268 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.268 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:238253 53.15 53.c6d9c815 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1353)", 2024-09-21T10:52:25.268 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:43:41.239774+0000", 2024-09-21T10:52:25.269 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 524.00356389599995, 2024-09-21T10:52:25.269 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.93717435100000002, 2024-09-21T10:52:25.269 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:25.269 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:52:25.269 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:25.269 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:52:25.269 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:52:25.269 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 238253 2024-09-21T10:52:25.269 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.269 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:25.269 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.269 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:25.269 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.239774+0000", 2024-09-21T10:52:25.270 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.270 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.270 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.270 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:25.270 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.239774+0000", 2024-09-21T10:52:25.270 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.270 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.270 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.270 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:25.270 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.239773+0000", 2024-09-21T10:52:25.270 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:52:25.270 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.270 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.271 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:25.271 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.239780+0000", 2024-09-21T10:52:25.271 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4749999999999998e-06 2024-09-21T10:52:25.271 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.271 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.271 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:25.271 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.239796+0000", 2024-09-21T10:52:25.271 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6013000000000001e-05 2024-09-21T10:52:25.271 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.271 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.271 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:25.271 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.239803+0000", 2024-09-21T10:52:25.271 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9700000000000002e-06 2024-09-21T10:52:25.271 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.272 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.272 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.272 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.239882+0000", 2024-09-21T10:52:25.272 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9036999999999996e-05 2024-09-21T10:52:25.272 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.272 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.272 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:25.272 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.239890+0000", 2024-09-21T10:52:25.272 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2169999999999994e-06 2024-09-21T10:52:25.272 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.272 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.272 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.272 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176944+0000", 2024-09-21T10:52:25.273 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.93705439499999998 2024-09-21T10:52:25.273 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.273 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.273 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:25.273 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176949+0000", 2024-09-21T10:52:25.273 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.673e-06 2024-09-21T10:52:25.273 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.273 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:25.273 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.273 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.273 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.273 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:238271 53.15 53.17959e15 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1353)", 2024-09-21T10:52:25.273 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:43:41.253997+0000", 2024-09-21T10:52:25.274 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 523.98934096400001, 2024-09-21T10:52:25.274 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.92298151900000003, 2024-09-21T10:52:25.274 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:25.274 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:52:25.274 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:25.274 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:52:25.274 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:52:25.274 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 238271 2024-09-21T10:52:25.274 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.274 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:25.274 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.274 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:25.274 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.253997+0000", 2024-09-21T10:52:25.274 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.275 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.275 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:25.275 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.253997+0000", 2024-09-21T10:52:25.275 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.275 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.275 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:25.275 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.253996+0000", 2024-09-21T10:52:25.275 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:52:25.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.275 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.275 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:25.276 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.254003+0000", 2024-09-21T10:52:25.276 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6989999999999997e-06 2024-09-21T10:52:25.276 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.276 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.276 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:25.276 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.254015+0000", 2024-09-21T10:52:25.276 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2148e-05 2024-09-21T10:52:25.276 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.276 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.276 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:25.276 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.254022+0000", 2024-09-21T10:52:25.276 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.374e-06 2024-09-21T10:52:25.276 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.277 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.277 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.277 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.254065+0000", 2024-09-21T10:52:25.277 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2601000000000001e-05 2024-09-21T10:52:25.277 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.277 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.277 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:25.277 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.254073+0000", 2024-09-21T10:52:25.277 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3990000000000004e-06 2024-09-21T10:52:25.277 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.277 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.277 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.277 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176971+0000", 2024-09-21T10:52:25.278 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.92289776899999998 2024-09-21T10:52:25.278 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.278 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:25.280 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:25.280 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:25.280 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176979+0000", 2024-09-21T10:52:25.280 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0590000000000003e-06 2024-09-21T10:52:25.280 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.280 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:25.280 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.280 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.281 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.281 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:238279 53.15 53.da97f315 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1353)", 2024-09-21T10:52:25.281 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:43:41.257274+0000", 2024-09-21T10:52:25.281 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 523.986064199, 2024-09-21T10:52:25.281 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91972421199999999, 2024-09-21T10:52:25.281 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:25.281 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:52:25.281 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:25.281 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:52:25.281 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:52:25.281 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 238279 2024-09-21T10:52:25.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.281 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:25.282 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.282 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:25.282 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.257274+0000", 2024-09-21T10:52:25.282 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.282 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.282 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.282 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:25.282 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.257274+0000", 2024-09-21T10:52:25.282 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.282 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.282 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.282 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:25.282 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.257273+0000", 2024-09-21T10:52:25.283 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:52:25.283 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.283 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.283 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:25.283 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.257281+0000", 2024-09-21T10:52:25.283 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8320000000000006e-06 2024-09-21T10:52:25.283 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.283 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.283 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:25.283 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.257293+0000", 2024-09-21T10:52:25.283 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2615e-05 2024-09-21T10:52:25.283 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.283 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.284 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:25.284 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.257301+0000", 2024-09-21T10:52:25.284 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7389999999999999e-06 2024-09-21T10:52:25.284 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.284 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.284 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.284 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.258358+0000", 2024-09-21T10:52:25.284 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001057379 2024-09-21T10:52:25.284 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.284 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.284 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:25.284 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.258367+0000", 2024-09-21T10:52:25.284 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6750000000000008e-06 2024-09-21T10:52:25.284 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.285 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.285 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.285 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176991+0000", 2024-09-21T10:52:25.285 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91862456299999995 2024-09-21T10:52:25.285 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.285 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.285 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:25.285 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.176998+0000", 2024-09-21T10:52:25.285 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9280000000000004e-06 2024-09-21T10:52:25.285 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.285 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:25.285 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.285 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.286 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.286 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:238285 53.15 53.412cc935 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1353)", 2024-09-21T10:52:25.286 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:43:41.262119+0000", 2024-09-21T10:52:25.286 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 523.98121913800003, 2024-09-21T10:52:25.286 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91490445300000001, 2024-09-21T10:52:25.286 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:25.286 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:52:25.286 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:25.286 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:52:25.286 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:52:25.286 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 238285 2024-09-21T10:52:25.286 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.286 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:25.286 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.287 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:25.287 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.262119+0000", 2024-09-21T10:52:25.287 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.287 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.287 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.287 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:25.287 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.262119+0000", 2024-09-21T10:52:25.287 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.287 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.287 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.287 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:25.287 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.262118+0000", 2024-09-21T10:52:25.287 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:52:25.288 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.288 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.288 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:25.288 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.262124+0000", 2024-09-21T10:52:25.288 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6690000000000003e-06 2024-09-21T10:52:25.288 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.288 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.288 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:25.288 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.262139+0000", 2024-09-21T10:52:25.288 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5252999999999999e-05 2024-09-21T10:52:25.288 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.288 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.288 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:25.288 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.262145+0000", 2024-09-21T10:52:25.289 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5010000000000001e-06 2024-09-21T10:52:25.289 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.289 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.289 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.289 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.262187+0000", 2024-09-21T10:52:25.289 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1896e-05 2024-09-21T10:52:25.289 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.289 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.289 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:25.289 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.262195+0000", 2024-09-21T10:52:25.289 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3150000000000003e-06 2024-09-21T10:52:25.289 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.289 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.289 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.290 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.177019+0000", 2024-09-21T10:52:25.290 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91482411299999999 2024-09-21T10:52:25.290 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.290 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.290 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:25.290 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.177024+0000", 2024-09-21T10:52:25.290 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9969999999999998e-06 2024-09-21T10:52:25.290 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.290 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:25.290 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.290 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.290 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.290 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:238289 53.15 53.9d0cb415 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1353)", 2024-09-21T10:52:25.291 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:43:41.267953+0000", 2024-09-21T10:52:25.291 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 523.97538528500002, 2024-09-21T10:52:25.291 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.90909275499999997, 2024-09-21T10:52:25.291 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:25.291 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:52:25.291 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:25.291 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:52:25.291 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:52:25.291 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 238289 2024-09-21T10:52:25.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.291 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:25.291 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.291 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:25.291 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.267953+0000", 2024-09-21T10:52:25.292 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.292 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.292 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:25.292 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.267953+0000", 2024-09-21T10:52:25.292 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.292 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.292 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:25.292 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.267952+0000", 2024-09-21T10:52:25.292 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:52:25.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.292 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.293 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:25.293 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.267959+0000", 2024-09-21T10:52:25.293 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7560000000000002e-06 2024-09-21T10:52:25.293 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.293 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.293 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:25.293 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.267969+0000", 2024-09-21T10:52:25.293 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.978e-06 2024-09-21T10:52:25.293 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.293 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.293 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:25.293 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.267976+0000", 2024-09-21T10:52:25.293 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2119999999999997e-06 2024-09-21T10:52:25.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.294 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.294 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.294 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.268736+0000", 2024-09-21T10:52:25.294 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00076052400000000005 2024-09-21T10:52:25.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.294 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.294 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:25.294 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.268743+0000", 2024-09-21T10:52:25.294 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1409999999999999e-06 2024-09-21T10:52:25.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.294 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.294 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.295 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.177041+0000", 2024-09-21T10:52:25.297 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.90829757300000002 2024-09-21T10:52:25.297 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.297 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.298 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:25.298 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.177046+0000", 2024-09-21T10:52:25.298 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8550000000000001e-06 2024-09-21T10:52:25.298 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.298 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:25.298 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.298 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.298 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.298 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:238293 53.15 53.c8cb59d5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1353)", 2024-09-21T10:52:25.298 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:43:41.269950+0000", 2024-09-21T10:52:25.298 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 523.97338878599999, 2024-09-21T10:52:25.298 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.9071169, 2024-09-21T10:52:25.298 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:25.299 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:52:25.299 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:25.299 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:52:25.299 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:52:25.299 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 238293 2024-09-21T10:52:25.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.299 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:25.299 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.299 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:25.299 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.269950+0000", 2024-09-21T10:52:25.299 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.300 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:25.300 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.269950+0000", 2024-09-21T10:52:25.300 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.300 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:25.300 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.269949+0000", 2024-09-21T10:52:25.300 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:52:25.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.300 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:25.301 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.269954+0000", 2024-09-21T10:52:25.301 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1669999999999998e-06 2024-09-21T10:52:25.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.301 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.301 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:25.301 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.269966+0000", 2024-09-21T10:52:25.301 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2048999999999999e-05 2024-09-21T10:52:25.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.301 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.301 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:25.301 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.269972+0000", 2024-09-21T10:52:25.301 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9240000000000004e-06 2024-09-21T10:52:25.302 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.302 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.302 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.302 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.270017+0000", 2024-09-21T10:52:25.302 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5034999999999998e-05 2024-09-21T10:52:25.302 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.302 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.302 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:25.302 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.270026+0000", 2024-09-21T10:52:25.302 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8910000000000001e-06 2024-09-21T10:52:25.302 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.302 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.302 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.303 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.177062+0000", 2024-09-21T10:52:25.303 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.90703622100000003 2024-09-21T10:52:25.303 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.303 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.303 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:25.303 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.177067+0000", 2024-09-21T10:52:25.303 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6539999999999998e-06 2024-09-21T10:52:25.303 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.303 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:25.303 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.303 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.303 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.303 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45020.0:238312 53.15 53.85eedcf5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1353)", 2024-09-21T10:52:25.303 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:43:41.288595+0000", 2024-09-21T10:52:25.304 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 523.95474299299997, 2024-09-21T10:52:25.304 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88849270400000002, 2024-09-21T10:52:25.304 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:25.304 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:52:25.304 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:25.304 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45020", 2024-09-21T10:52:25.304 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1217222769", 2024-09-21T10:52:25.304 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 238312 2024-09-21T10:52:25.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.304 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:25.304 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.304 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:25.304 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.288595+0000", 2024-09-21T10:52:25.304 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.305 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:25.305 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.288595+0000", 2024-09-21T10:52:25.305 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.305 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:25.305 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.288594+0000", 2024-09-21T10:52:25.305 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:52:25.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.305 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:25.305 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.288602+0000", 2024-09-21T10:52:25.305 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1289999999999996e-06 2024-09-21T10:52:25.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.306 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.306 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:25.306 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.288620+0000", 2024-09-21T10:52:25.306 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8495000000000002e-05 2024-09-21T10:52:25.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.306 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.306 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:25.306 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.288628+0000", 2024-09-21T10:52:25.306 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8189999999999992e-06 2024-09-21T10:52:25.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.306 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.306 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.307 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.288681+0000", 2024-09-21T10:52:25.307 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3013999999999998e-05 2024-09-21T10:52:25.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.307 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:25.307 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:41.288692+0000", 2024-09-21T10:52:25.307 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0366999999999999e-05 2024-09-21T10:52:25.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.307 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.307 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.177082+0000", 2024-09-21T10:52:25.307 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88839072100000005 2024-09-21T10:52:25.308 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.308 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.308 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:25.308 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:43:42.177088+0000", 2024-09-21T10:52:25.308 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8030000000000002e-06 2024-09-21T10:52:25.308 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.308 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:25.308 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.308 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.308 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.308 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:75995 55.f 55:f41bc5b0:::benchmark_data_smithi070_294394_object75994:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1426)", 2024-09-21T10:52:25.308 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:49:43.845182+0000", 2024-09-21T10:52:25.308 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 161.398155997, 2024-09-21T10:52:25.309 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1966486170000001, 2024-09-21T10:52:25.309 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:25.309 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:52:25.309 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:25.309 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:52:25.309 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:52:25.309 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 75995 2024-09-21T10:52:25.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.309 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:25.309 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.309 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:25.309 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.845182+0000", 2024-09-21T10:52:25.309 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.310 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.310 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:25.310 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.845182+0000", 2024-09-21T10:52:25.310 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.310 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.310 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.310 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:25.310 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.845178+0000", 2024-09-21T10:52:25.310 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999952 2024-09-21T10:52:25.310 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.310 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.310 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:25.310 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.845218+0000", 2024-09-21T10:52:25.310 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0726000000000003e-05 2024-09-21T10:52:25.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.311 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:25.311 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.845245+0000", 2024-09-21T10:52:25.311 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6508000000000001e-05 2024-09-21T10:52:25.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.311 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:25.311 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.845254+0000", 2024-09-21T10:52:25.311 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4860000000000003e-06 2024-09-21T10:52:25.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.311 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.312 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.845921+0000", 2024-09-21T10:52:25.312 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00066632099999999999 2024-09-21T10:52:25.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.312 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.312 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:25.312 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.846058+0000", 2024-09-21T10:52:25.312 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013705499999999999 2024-09-21T10:52:25.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.312 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.312 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5,7", 2024-09-21T10:52:25.312 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.846184+0000", 2024-09-21T10:52:25.312 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012604000000000001 2024-09-21T10:52:25.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.312 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.312 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:52:25.313 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.852309+0000", 2024-09-21T10:52:25.313 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.006124979 2024-09-21T10:52:25.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.313 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.313 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:25.313 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.853735+0000", 2024-09-21T10:52:25.313 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001425934 2024-09-21T10:52:25.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.313 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.313 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.313 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.870055+0000", 2024-09-21T10:52:25.313 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.016320741999999999 2024-09-21T10:52:25.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.314 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.314 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:25.314 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.870069+0000", 2024-09-21T10:52:25.314 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3900000000000001e-05 2024-09-21T10:52:25.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.314 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.314 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.314 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:44.859296+0000", 2024-09-21T10:52:25.314 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98922714099999998 2024-09-21T10:52:25.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.314 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.314 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:25.314 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:44.859311+0000", 2024-09-21T10:52:25.314 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5136e-05 2024-09-21T10:52:25.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.315 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.315 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.041745+0000", 2024-09-21T10:52:25.315 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.182433394 2024-09-21T10:52:25.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.315 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:25.315 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.041831+0000", 2024-09-21T10:52:25.315 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6133999999999999e-05 2024-09-21T10:52:25.315 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.315 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:25.315 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.316 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.316 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:76011 55.f 55:ff61ee79:::benchmark_data_smithi070_294394_object76010:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1427)", 2024-09-21T10:52:25.316 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:49:43.867485+0000", 2024-09-21T10:52:25.316 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 161.37585327100001, 2024-09-21T10:52:25.316 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.192170902, 2024-09-21T10:52:25.316 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:25.316 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:52:25.316 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:25.316 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:52:25.316 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:52:25.316 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 76011 2024-09-21T10:52:25.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.316 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:25.316 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:25.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.867485+0000", 2024-09-21T10:52:25.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:25.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.867485+0000", 2024-09-21T10:52:25.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:25.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.867483+0000", 2024-09-21T10:52:25.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:52:25.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.318 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:25.318 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.867532+0000", 2024-09-21T10:52:25.318 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8603000000000002e-05 2024-09-21T10:52:25.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.318 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.318 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:25.318 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.867545+0000", 2024-09-21T10:52:25.318 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2490999999999999e-05 2024-09-21T10:52:25.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.318 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.318 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:25.318 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.867552+0000", 2024-09-21T10:52:25.318 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4540000000000001e-06 2024-09-21T10:52:25.319 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.319 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.319 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.319 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.870102+0000", 2024-09-21T10:52:25.319 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0025497300000000001 2024-09-21T10:52:25.319 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.319 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.319 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:25.319 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.870109+0000", 2024-09-21T10:52:25.319 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5220000000000001e-06 2024-09-21T10:52:25.319 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.319 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.319 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.319 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:44.860861+0000", 2024-09-21T10:52:25.320 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99075127799999996 2024-09-21T10:52:25.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.320 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.320 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:25.320 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:44.860870+0000", 2024-09-21T10:52:25.320 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3850000000000008e-06 2024-09-21T10:52:25.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.320 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.320 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.320 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.043404+0000", 2024-09-21T10:52:25.320 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1825337339999999 2024-09-21T10:52:25.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.320 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.320 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:25.321 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.043507+0000", 2024-09-21T10:52:25.321 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000103288 2024-09-21T10:52:25.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.321 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:52:25.321 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.043615+0000", 2024-09-21T10:52:25.321 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000107644 2024-09-21T10:52:25.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.321 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:25.321 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.047861+0000", 2024-09-21T10:52:25.321 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0042466639999999998 2024-09-21T10:52:25.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.322 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.322 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:52:25.322 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.047892+0000", 2024-09-21T10:52:25.322 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1214000000000003e-05 2024-09-21T10:52:25.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.322 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.322 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:25.322 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.057061+0000", 2024-09-21T10:52:25.322 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0091686730000000004 2024-09-21T10:52:25.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.322 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.322 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:52:25.322 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.057099+0000", 2024-09-21T10:52:25.323 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7576000000000001e-05 2024-09-21T10:52:25.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.323 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:25.323 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.059656+0000", 2024-09-21T10:52:25.323 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0025573219999999999 2024-09-21T10:52:25.323 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.323 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:25.323 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.323 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:76024 55.f 55:f47c1d3d:::benchmark_data_smithi070_294394_object76023:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1427)", 2024-09-21T10:52:25.323 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:49:43.882340+0000", 2024-09-21T10:52:25.324 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 161.36099872299999, 2024-09-21T10:52:25.324 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1799865559999998, 2024-09-21T10:52:25.324 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:25.324 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:52:25.324 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:25.324 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:52:25.324 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:52:25.324 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 76024 2024-09-21T10:52:25.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.324 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:25.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.324 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:25.324 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.882340+0000", 2024-09-21T10:52:25.324 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.325 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.325 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:25.325 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.882340+0000", 2024-09-21T10:52:25.325 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.325 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.325 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:25.325 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.882338+0000", 2024-09-21T10:52:25.325 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:52:25.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.325 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.325 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:25.325 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.882383+0000", 2024-09-21T10:52:25.326 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4848999999999997e-05 2024-09-21T10:52:25.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.326 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:25.326 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.882397+0000", 2024-09-21T10:52:25.326 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3774000000000001e-05 2024-09-21T10:52:25.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.326 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:25.326 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.882404+0000", 2024-09-21T10:52:25.326 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.345e-06 2024-09-21T10:52:25.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.327 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.327 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.882476+0000", 2024-09-21T10:52:25.327 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1624000000000004e-05 2024-09-21T10:52:25.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.327 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.327 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:25.327 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.882486+0000", 2024-09-21T10:52:25.327 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0129000000000001e-05 2024-09-21T10:52:25.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.327 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.327 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.327 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:44.860894+0000", 2024-09-21T10:52:25.327 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.97840792499999996 2024-09-21T10:52:25.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.328 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:25.328 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:44.860901+0000", 2024-09-21T10:52:25.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3499999999999999e-06 2024-09-21T10:52:25.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.328 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.328 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.045997+0000", 2024-09-21T10:52:25.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.185096062 2024-09-21T10:52:25.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.328 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:25.328 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.046097+0000", 2024-09-21T10:52:25.329 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010022399999999999 2024-09-21T10:52:25.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.329 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.329 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:52:25.329 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.046198+0000", 2024-09-21T10:52:25.329 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000100243 2024-09-21T10:52:25.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.329 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.329 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:25.329 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.047887+0000", 2024-09-21T10:52:25.329 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0016894049999999999 2024-09-21T10:52:25.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.329 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.329 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:52:25.330 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.047898+0000", 2024-09-21T10:52:25.330 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0842000000000001e-05 2024-09-21T10:52:25.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.330 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:25.330 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.059713+0000", 2024-09-21T10:52:25.330 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011815155000000001 2024-09-21T10:52:25.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.330 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:52:25.330 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.059753+0000", 2024-09-21T10:52:25.330 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0309999999999999e-05 2024-09-21T10:52:25.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.331 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:25.331 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.062326+0000", 2024-09-21T10:52:25.331 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002572918 2024-09-21T10:52:25.331 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.331 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:25.331 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.331 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:130871 55.f 55:f1490c0a:::benchmark_data_smithi070_294394_object25530:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1447)", 2024-09-21T10:52:25.331 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:26.456048+0000", 2024-09-21T10:52:25.331 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 58.787290398000003, 2024-09-21T10:52:25.331 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1801730020000001, 2024-09-21T10:52:25.332 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:25.332 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:52:25.332 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:25.332 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:52:25.332 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:52:25.332 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 130871 2024-09-21T10:52:25.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.332 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:25.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.332 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:25.332 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.456048+0000", 2024-09-21T10:52:25.332 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.333 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:25.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.456048+0000", 2024-09-21T10:52:25.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.333 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:25.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.456047+0000", 2024-09-21T10:52:25.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:52:25.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.333 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:25.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.456052+0000", 2024-09-21T10:52:25.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5589999999999996e-06 2024-09-21T10:52:25.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.334 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:25.334 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.456064+0000", 2024-09-21T10:52:25.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1384e-05 2024-09-21T10:52:25.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.334 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.334 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:25.334 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.456070+0000", 2024-09-21T10:52:25.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0959999999999997e-06 2024-09-21T10:52:25.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.334 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.334 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.334 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.456107+0000", 2024-09-21T10:52:25.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7459000000000002e-05 2024-09-21T10:52:25.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.335 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.335 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:25.335 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.456115+0000", 2024-09-21T10:52:25.335 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7919999999999993e-06 2024-09-21T10:52:25.335 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.335 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.335 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.335 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.417963+0000", 2024-09-21T10:52:25.335 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96184808200000005 2024-09-21T10:52:25.335 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.335 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.335 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:25.335 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.417974+0000", 2024-09-21T10:52:25.335 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1369000000000001e-05 2024-09-21T10:52:25.335 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.336 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.336 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.569706+0000", 2024-09-21T10:52:25.336 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1517316150000001 2024-09-21T10:52:25.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.336 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:52:25.336 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.570244+0000", 2024-09-21T10:52:25.336 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00053811400000000002 2024-09-21T10:52:25.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.336 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.336 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.618940+0000", 2024-09-21T10:52:25.336 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.048695671000000003 2024-09-21T10:52:25.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.337 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:25.337 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.619051+0000", 2024-09-21T10:52:25.337 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000111006 2024-09-21T10:52:25.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.337 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1,7", 2024-09-21T10:52:25.337 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.619552+0000", 2024-09-21T10:52:25.337 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00050132200000000001 2024-09-21T10:52:25.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.337 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:25.337 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.624954+0000", 2024-09-21T10:52:25.337 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0054023200000000004 2024-09-21T10:52:25.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.338 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:52:25.338 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.626385+0000", 2024-09-21T10:52:25.338 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0014304319999999999 2024-09-21T10:52:25.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.338 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.338 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:25.338 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.627839+0000", 2024-09-21T10:52:25.338 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0014536760000000001 2024-09-21T10:52:25.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.338 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.338 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:25.338 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.633734+0000", 2024-09-21T10:52:25.338 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0058950210000000003 2024-09-21T10:52:25.339 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.339 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.339 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:52:25.339 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.633786+0000", 2024-09-21T10:52:25.339 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2645999999999997e-05 2024-09-21T10:52:25.339 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.339 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.339 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:25.339 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.636221+0000", 2024-09-21T10:52:25.339 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0024347510000000002 2024-09-21T10:52:25.339 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.339 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:25.339 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.339 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.339 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.340 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:130858 55.8 55:178fc653:::benchmark_data_smithi070_294394_object25517:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1448)", 2024-09-21T10:52:25.340 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:27.414411+0000", 2024-09-21T10:52:25.340 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 57.828927344999997, 2024-09-21T10:52:25.340 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.147124306, 2024-09-21T10:52:25.340 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:25.340 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:52:25.340 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:25.340 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:52:25.340 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:52:25.340 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 130858 2024-09-21T10:52:25.340 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.340 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:25.340 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.340 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:25.341 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.414411+0000", 2024-09-21T10:52:25.341 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.341 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.341 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:25.341 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.414411+0000", 2024-09-21T10:52:25.341 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.341 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.341 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:25.341 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.414409+0000", 2024-09-21T10:52:25.341 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:52:25.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.342 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.342 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:25.342 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.414416+0000", 2024-09-21T10:52:25.342 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5289999999999997e-06 2024-09-21T10:52:25.342 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.342 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.342 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:25.342 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.414427+0000", 2024-09-21T10:52:25.342 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1259e-05 2024-09-21T10:52:25.342 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.342 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.342 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:25.342 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.414437+0000", 2024-09-21T10:52:25.343 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0297e-05 2024-09-21T10:52:25.343 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.343 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.343 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.343 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.431057+0000", 2024-09-21T10:52:25.343 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.016619359 2024-09-21T10:52:25.343 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.343 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.343 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:25.343 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.431068+0000", 2024-09-21T10:52:25.343 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1294999999999999e-05 2024-09-21T10:52:25.343 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.343 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.343 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.344 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.559849+0000", 2024-09-21T10:52:25.344 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.128780871 2024-09-21T10:52:25.344 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.344 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.344 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:25.344 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.561535+0000", 2024-09-21T10:52:25.344 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001686346 2024-09-21T10:52:25.344 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.344 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:25.344 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.344 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.344 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.344 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:130866 55.8 55:1cb64a88:::benchmark_data_smithi070_294394_object25525:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1448)", 2024-09-21T10:52:25.344 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:27.414452+0000", 2024-09-21T10:52:25.344 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 57.828886300999997, 2024-09-21T10:52:25.345 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1631621219999999, 2024-09-21T10:52:25.345 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:25.345 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:52:25.345 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:25.345 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:52:25.345 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:52:25.345 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 130866 2024-09-21T10:52:25.345 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.345 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:25.345 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.345 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:25.345 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.414452+0000", 2024-09-21T10:52:25.345 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.345 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.345 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.346 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:25.346 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.414452+0000", 2024-09-21T10:52:25.346 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.346 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.346 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.346 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:25.346 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.414452+0000", 2024-09-21T10:52:25.346 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T10:52:25.346 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.346 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.346 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:25.346 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.414455+0000", 2024-09-21T10:52:25.346 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9699999999999999e-06 2024-09-21T10:52:25.346 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.347 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.347 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:25.347 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.414462+0000", 2024-09-21T10:52:25.347 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4989999999999997e-06 2024-09-21T10:52:25.347 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.347 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.347 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:25.347 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.414468+0000", 2024-09-21T10:52:25.347 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4319999999999998e-06 2024-09-21T10:52:25.347 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.347 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.347 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.347 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.431092+0000", 2024-09-21T10:52:25.348 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.016624505000000001 2024-09-21T10:52:25.348 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.348 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.348 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:25.348 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.431100+0000", 2024-09-21T10:52:25.348 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7740000000000004e-06 2024-09-21T10:52:25.348 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.348 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.348 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.348 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.563319+0000", 2024-09-21T10:52:25.348 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.132219163 2024-09-21T10:52:25.348 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.348 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.348 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:25.349 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.564062+0000", 2024-09-21T10:52:25.349 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0007427 2024-09-21T10:52:25.349 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.349 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.349 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1,3", 2024-09-21T10:52:25.349 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.564353+0000", 2024-09-21T10:52:25.349 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00029125400000000001 2024-09-21T10:52:25.349 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.349 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.349 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:25.349 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.572897+0000", 2024-09-21T10:52:25.349 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0085442829999999997 2024-09-21T10:52:25.349 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.350 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.350 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:52:25.350 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.572910+0000", 2024-09-21T10:52:25.350 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2945000000000001e-05 2024-09-21T10:52:25.350 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.350 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.350 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:25.350 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.573016+0000", 2024-09-21T10:52:25.350 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000105943 2024-09-21T10:52:25.350 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.350 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.350 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:25.350 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.575815+0000", 2024-09-21T10:52:25.351 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0027989 2024-09-21T10:52:25.351 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.351 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.351 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:52:25.351 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.575854+0000", 2024-09-21T10:52:25.351 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8507999999999998e-05 2024-09-21T10:52:25.351 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.351 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.351 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:25.351 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.577614+0000", 2024-09-21T10:52:25.351 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001760677 2024-09-21T10:52:25.351 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.351 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:25.351 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.352 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.352 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.352 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:130872 55.8 55:1f224461:::benchmark_data_smithi070_294394_object25531:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1448)", 2024-09-21T10:52:25.352 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:27.414510+0000", 2024-09-21T10:52:25.352 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 57.828828584, 2024-09-21T10:52:25.352 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1677864229999999, 2024-09-21T10:52:25.352 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:25.352 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:52:25.352 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:25.352 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:52:25.352 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:52:25.352 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 130872 2024-09-21T10:52:25.352 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.352 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:25.353 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.353 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:25.353 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.414510+0000", 2024-09-21T10:52:25.353 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.353 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.353 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.353 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:25.353 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.414510+0000", 2024-09-21T10:52:25.353 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.353 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.353 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.353 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:25.353 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.414509+0000", 2024-09-21T10:52:25.354 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T10:52:25.354 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.354 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.354 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:25.354 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.414512+0000", 2024-09-21T10:52:25.354 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0469999999999998e-06 2024-09-21T10:52:25.354 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.354 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.354 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:25.354 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.414520+0000", 2024-09-21T10:52:25.354 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7589999999999997e-06 2024-09-21T10:52:25.354 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.354 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.355 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:25.355 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.414526+0000", 2024-09-21T10:52:25.355 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7010000000000001e-06 2024-09-21T10:52:25.355 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.355 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.355 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.355 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.431123+0000", 2024-09-21T10:52:25.355 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.016596900000000001 2024-09-21T10:52:25.355 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.355 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.355 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:25.355 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.431130+0000", 2024-09-21T10:52:25.355 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5939999999999996e-06 2024-09-21T10:52:25.355 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.356 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.356 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.356 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.567234+0000", 2024-09-21T10:52:25.356 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.136103256 2024-09-21T10:52:25.356 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.356 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.356 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:25.356 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.567389+0000", 2024-09-21T10:52:25.356 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000154955 2024-09-21T10:52:25.356 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.356 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.356 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1,3", 2024-09-21T10:52:25.356 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.567559+0000", 2024-09-21T10:52:25.356 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000170416 2024-09-21T10:52:25.357 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.357 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.357 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:52:25.357 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.572921+0000", 2024-09-21T10:52:25.357 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0053619419999999998 2024-09-21T10:52:25.357 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.357 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.357 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:25.357 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.572967+0000", 2024-09-21T10:52:25.357 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5769000000000002e-05 2024-09-21T10:52:25.357 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.357 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.357 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:25.357 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.573059+0000", 2024-09-21T10:52:25.358 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1943999999999999e-05 2024-09-21T10:52:25.358 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.358 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.358 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:25.358 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.580364+0000", 2024-09-21T10:52:25.358 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0073050739999999999 2024-09-21T10:52:25.358 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.358 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.358 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:52:25.358 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.580475+0000", 2024-09-21T10:52:25.358 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011096599999999999 2024-09-21T10:52:25.358 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.358 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.358 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:25.359 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.582296+0000", 2024-09-21T10:52:25.359 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001821549 2024-09-21T10:52:25.359 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.359 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:25.359 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.359 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.359 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.359 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:130880 55.f 55:f2984d22:::benchmark_data_smithi070_294394_object25539:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1448)", 2024-09-21T10:52:25.359 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:27.712373+0000", 2024-09-21T10:52:25.359 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 57.530965389999999, 2024-09-21T10:52:25.359 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.943867817, 2024-09-21T10:52:25.359 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:25.359 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:52:25.360 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:25.360 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:52:25.360 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:52:25.360 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 130880 2024-09-21T10:52:25.360 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.360 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:25.360 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.360 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:25.360 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.712373+0000", 2024-09-21T10:52:25.360 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.360 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.360 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.360 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:25.360 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.712373+0000", 2024-09-21T10:52:25.361 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.361 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:25.361 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.712372+0000", 2024-09-21T10:52:25.361 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:52:25.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.361 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:25.361 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.712381+0000", 2024-09-21T10:52:25.361 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2490000000000007e-06 2024-09-21T10:52:25.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.361 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:25.362 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.712393+0000", 2024-09-21T10:52:25.362 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1114000000000001e-05 2024-09-21T10:52:25.362 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.362 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.362 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:25.362 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.712398+0000", 2024-09-21T10:52:25.362 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4040000000000003e-06 2024-09-21T10:52:25.362 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.362 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.362 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.362 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.712460+0000", 2024-09-21T10:52:25.362 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1730999999999996e-05 2024-09-21T10:52:25.362 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.363 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.363 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:25.363 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.712472+0000", 2024-09-21T10:52:25.363 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2009e-05 2024-09-21T10:52:25.363 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.363 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.363 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.363 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.624988+0000", 2024-09-21T10:52:25.363 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91251623699999995 2024-09-21T10:52:25.363 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.363 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.363 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:52:25.363 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.625718+0000", 2024-09-21T10:52:25.363 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00072979100000000003 2024-09-21T10:52:25.363 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.364 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.364 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.364 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.640404+0000", 2024-09-21T10:52:25.364 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.014685828999999999 2024-09-21T10:52:25.364 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.364 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.364 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:25.364 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.640480+0000", 2024-09-21T10:52:25.364 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6629000000000001e-05 2024-09-21T10:52:25.364 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.364 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.364 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1,7", 2024-09-21T10:52:25.364 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.640581+0000", 2024-09-21T10:52:25.364 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010040299999999999 2024-09-21T10:52:25.364 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.364 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.365 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:25.365 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.648783+0000", 2024-09-21T10:52:25.365 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0082023329999999992 2024-09-21T10:52:25.365 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.365 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.365 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:25.365 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.648823+0000", 2024-09-21T10:52:25.365 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0544999999999999e-05 2024-09-21T10:52:25.365 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.365 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.365 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:52:25.365 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.650763+0000", 2024-09-21T10:52:25.365 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00193998 2024-09-21T10:52:25.365 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.365 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.366 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:25.366 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.653885+0000", 2024-09-21T10:52:25.366 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0031219529999999998 2024-09-21T10:52:25.366 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.366 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.366 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:52:25.366 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.653939+0000", 2024-09-21T10:52:25.366 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4055999999999999e-05 2024-09-21T10:52:25.366 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.366 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.366 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:25.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:25.367 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.656241+0000", 2024-09-21T10:52:25.367 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0023013579999999999 2024-09-21T10:52:25.367 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.367 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:25.367 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.367 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.367 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.367 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:130885 55.f 55:f4c3f8d9:::benchmark_data_smithi070_294394_object25544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1448)", 2024-09-21T10:52:25.367 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:27.769035+0000", 2024-09-21T10:52:25.367 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 57.474303057999997, 2024-09-21T10:52:25.368 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88945628200000004, 2024-09-21T10:52:25.368 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:25.368 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:52:25.368 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:25.368 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:52:25.368 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:52:25.368 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 130885 2024-09-21T10:52:25.368 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.368 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:25.368 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.368 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:25.368 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.769035+0000", 2024-09-21T10:52:25.368 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.368 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.368 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.369 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:25.369 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.769035+0000", 2024-09-21T10:52:25.369 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:25.369 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.369 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.369 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:25.369 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.769034+0000", 2024-09-21T10:52:25.369 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:52:25.369 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.369 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.369 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:25.369 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.769039+0000", 2024-09-21T10:52:25.369 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4979999999999997e-06 2024-09-21T10:52:25.369 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.369 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.370 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:25.370 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.769050+0000", 2024-09-21T10:52:25.370 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0954e-05 2024-09-21T10:52:25.370 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.370 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.370 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:25.370 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.769055+0000", 2024-09-21T10:52:25.370 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9799999999999998e-06 2024-09-21T10:52:25.370 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.370 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.370 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.370 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.769116+0000", 2024-09-21T10:52:25.370 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1180999999999996e-05 2024-09-21T10:52:25.370 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.370 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.371 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:52:25.371 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.769129+0000", 2024-09-21T10:52:25.371 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3064999999999999e-05 2024-09-21T10:52:25.371 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.371 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.371 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.371 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.625769+0000", 2024-09-21T10:52:25.371 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.85663909100000002 2024-09-21T10:52:25.371 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.371 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.371 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:52:25.371 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.626365+0000", 2024-09-21T10:52:25.371 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00059668099999999995 2024-09-21T10:52:25.371 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.371 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.372 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:25.372 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.646373+0000", 2024-09-21T10:52:25.372 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020007546000000001 2024-09-21T10:52:25.372 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.372 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.372 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:25.372 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.646450+0000", 2024-09-21T10:52:25.372 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7267000000000004e-05 2024-09-21T10:52:25.372 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.372 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.372 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1,7", 2024-09-21T10:52:25.372 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.646553+0000", 2024-09-21T10:52:25.372 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010268 2024-09-21T10:52:25.372 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.372 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.373 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:25.373 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.650833+0000", 2024-09-21T10:52:25.373 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.004280626 2024-09-21T10:52:25.373 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.373 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.373 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:25.373 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.650873+0000", 2024-09-21T10:52:25.373 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9660000000000003e-05 2024-09-21T10:52:25.373 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.373 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.373 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:52:25.373 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.650882+0000", 2024-09-21T10:52:25.373 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5439999999999999e-06 2024-09-21T10:52:25.374 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.374 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.374 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:52:25.374 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.656302+0000", 2024-09-21T10:52:25.374 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0054206209999999996 2024-09-21T10:52:25.374 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.374 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.374 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:52:25.374 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.656368+0000", 2024-09-21T10:52:25.374 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5673999999999995e-05 2024-09-21T10:52:25.374 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:25.374 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:25.374 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:52:25.375 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.658492+0000", 2024-09-21T10:52:25.375 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021237360000000002 2024-09-21T10:52:25.375 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.375 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:25.375 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.375 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:25.375 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:25.375 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:52:25.468 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:25.501 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:25.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:25.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:25.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:25 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:52:25.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:25 smithi160 ceph-mon[101879]: osdmap e1465: 8 total, 8 up, 5 in 2024-09-21T10:52:25.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:25 smithi160 ceph-mon[101879]: osdmap e1466: 8 total, 8 up, 5 in 2024-09-21T10:52:25.604 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:25.625 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:25.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:25.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:25.727 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:25.748 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:25.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:25.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:25.815 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:25 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:52:25.815 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:25 smithi070 ceph-mon[107385]: osdmap e1465: 8 total, 8 up, 5 in 2024-09-21T10:52:25.815 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:25 smithi070 ceph-mon[107385]: osdmap e1466: 8 total, 8 up, 5 in 2024-09-21T10:52:25.816 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:25 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:52:25.816 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:25 smithi070 ceph-mon[112139]: osdmap e1465: 8 total, 8 up, 5 in 2024-09-21T10:52:25.816 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:25 smithi070 ceph-mon[112139]: osdmap e1466: 8 total, 8 up, 5 in 2024-09-21T10:52:25.850 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:25.871 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:25.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:25.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:25.973 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:25.995 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:25.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:25.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:26.096 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:26.118 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:26.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:26.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:26.220 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:26.242 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:26.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:26.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:26.343 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:26.365 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:26.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:26.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:26.466 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:26.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:26.499 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:26.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:26.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:26 smithi160 ceph-mon[101879]: pgmap v2671: 99 pgs: 1 active+recovering+remapped, 4 active+recovery_wait+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.4 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 399 op/s; 38938/168033 objects degraded (23.173%); 12324/168033 objects misplaced (7.334%); 13 MiB/s, 237 objects/s recovering 2024-09-21T10:52:26.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:26 smithi160 ceph-mon[101879]: 55.c starting backfill to osd.7 from (0'0,1432'5685] MAX to 1464'9744 2024-09-21T10:52:26.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:26 smithi160 ceph-mon[101879]: 55.5 starting backfill to osd.7 from (0'0,0'0] MAX to 1464'9740 2024-09-21T10:52:26.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:26 smithi160 ceph-mon[101879]: osdmap e1467: 8 total, 8 up, 5 in 2024-09-21T10:52:26.601 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:26.632 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:26.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:26.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:26.733 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:26.761 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:26.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:26.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:26.863 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:26.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:26 smithi070 ceph-mon[107385]: pgmap v2671: 99 pgs: 1 active+recovering+remapped, 4 active+recovery_wait+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.4 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 399 op/s; 38938/168033 objects degraded (23.173%); 12324/168033 objects misplaced (7.334%); 13 MiB/s, 237 objects/s recovering 2024-09-21T10:52:26.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:26 smithi070 ceph-mon[107385]: 55.c starting backfill to osd.7 from (0'0,1432'5685] MAX to 1464'9744 2024-09-21T10:52:26.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:26 smithi070 ceph-mon[107385]: 55.5 starting backfill to osd.7 from (0'0,0'0] MAX to 1464'9740 2024-09-21T10:52:26.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:26 smithi070 ceph-mon[107385]: osdmap e1467: 8 total, 8 up, 5 in 2024-09-21T10:52:26.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:26 smithi070 ceph-mon[112139]: pgmap v2671: 99 pgs: 1 active+recovering+remapped, 4 active+recovery_wait+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.4 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 399 op/s; 38938/168033 objects degraded (23.173%); 12324/168033 objects misplaced (7.334%); 13 MiB/s, 237 objects/s recovering 2024-09-21T10:52:26.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:26 smithi070 ceph-mon[112139]: 55.c starting backfill to osd.7 from (0'0,1432'5685] MAX to 1464'9744 2024-09-21T10:52:26.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:26 smithi070 ceph-mon[112139]: 55.5 starting backfill to osd.7 from (0'0,0'0] MAX to 1464'9740 2024-09-21T10:52:26.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:26 smithi070 ceph-mon[112139]: osdmap e1467: 8 total, 8 up, 5 in 2024-09-21T10:52:26.896 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:26.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:26.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:26.998 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:27.030 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:27.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:27.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:27.132 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:27.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:27.181 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:27.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:27.188 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:52:27.283 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:27.323 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:27.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:27.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:27.425 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:27.446 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:27.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:27.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:27.507 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:52:27.548 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:27.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:27.590 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:27.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:27.691 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:27.712 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:27.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:27.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:27.804 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:27 smithi070 ceph-mon[107385]: 55.2 starting backfill to osd.7 from (0'0,1432'5645] MAX to 1464'9627 2024-09-21T10:52:27.805 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:27 smithi070 ceph-mon[107385]: 55.0 starting backfill to osd.7 from (0'0,1432'5766] MAX to 1464'9805 2024-09-21T10:52:27.805 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:27 smithi070 ceph-mon[107385]: 55.a starting backfill to osd.7 from (0'0,1432'5703] MAX to 1464'9768 2024-09-21T10:52:27.805 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:27 smithi070 ceph-mon[107385]: 55.3 starting backfill to osd.7 from (0'0,1432'5541] MAX to 1464'9405 2024-09-21T10:52:27.805 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:27 smithi070 ceph-mon[107385]: osdmap e1468: 8 total, 8 up, 5 in 2024-09-21T10:52:27.805 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:27 smithi070 ceph-mon[112139]: 55.2 starting backfill to osd.7 from (0'0,1432'5645] MAX to 1464'9627 2024-09-21T10:52:27.805 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:27 smithi070 ceph-mon[112139]: 55.0 starting backfill to osd.7 from (0'0,1432'5766] MAX to 1464'9805 2024-09-21T10:52:27.805 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:27 smithi070 ceph-mon[112139]: 55.a starting backfill to osd.7 from (0'0,1432'5703] MAX to 1464'9768 2024-09-21T10:52:27.805 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:27 smithi070 ceph-mon[112139]: 55.3 starting backfill to osd.7 from (0'0,1432'5541] MAX to 1464'9405 2024-09-21T10:52:27.805 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:27 smithi070 ceph-mon[112139]: osdmap e1468: 8 total, 8 up, 5 in 2024-09-21T10:52:27.814 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:27 smithi160 ceph-mon[101879]: 55.2 starting backfill to osd.7 from (0'0,1432'5645] MAX to 1464'9627 2024-09-21T10:52:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:27 smithi160 ceph-mon[101879]: 55.0 starting backfill to osd.7 from (0'0,1432'5766] MAX to 1464'9805 2024-09-21T10:52:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:27 smithi160 ceph-mon[101879]: 55.a starting backfill to osd.7 from (0'0,1432'5703] MAX to 1464'9768 2024-09-21T10:52:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:27 smithi160 ceph-mon[101879]: 55.3 starting backfill to osd.7 from (0'0,1432'5541] MAX to 1464'9405 2024-09-21T10:52:27.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:27 smithi160 ceph-mon[101879]: osdmap e1468: 8 total, 8 up, 5 in 2024-09-21T10:52:27.851 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:27.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:27.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:27.953 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:27.987 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:27.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:27.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:28.088 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:28.110 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:28.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:28.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:28.211 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:28.246 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:28.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:28.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:28.348 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:28.385 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:28.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:28.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:28.487 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:28.510 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:28.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:28.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:28.612 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:28.644 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:28.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:28.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:28.745 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:28.783 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:28.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:28.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:28 smithi160 ceph-mon[101879]: pgmap v2674: 99 pgs: 1 active+recovery_wait+remapped, 8 remapped+peering, 1 active+undersized+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 3.3 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 659 op/s; 13969/164226 objects degraded (8.506%); 8646/164226 objects misplaced (5.265%); 26 MiB/s, 420 objects/s recovering 2024-09-21T10:52:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:28 smithi160 ceph-mon[101879]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:52:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:28 smithi160 ceph-mon[101879]: osdmap e1469: 8 total, 8 up, 5 in 2024-09-21T10:52:28.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:28 smithi070 ceph-mon[107385]: pgmap v2674: 99 pgs: 1 active+recovery_wait+remapped, 8 remapped+peering, 1 active+undersized+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 3.3 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 659 op/s; 13969/164226 objects degraded (8.506%); 8646/164226 objects misplaced (5.265%); 26 MiB/s, 420 objects/s recovering 2024-09-21T10:52:28.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:28 smithi070 ceph-mon[107385]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:52:28.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:28 smithi070 ceph-mon[107385]: osdmap e1469: 8 total, 8 up, 5 in 2024-09-21T10:52:28.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:28 smithi070 ceph-mon[112139]: pgmap v2674: 99 pgs: 1 active+recovery_wait+remapped, 8 remapped+peering, 1 active+undersized+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 3.3 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 659 op/s; 13969/164226 objects degraded (8.506%); 8646/164226 objects misplaced (5.265%); 26 MiB/s, 420 objects/s recovering 2024-09-21T10:52:28.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:28 smithi070 ceph-mon[112139]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:52:28.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:28 smithi070 ceph-mon[112139]: osdmap e1469: 8 total, 8 up, 5 in 2024-09-21T10:52:28.884 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:28.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:28.923 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:28.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:28.950 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T10:52:28.950 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:52:29.024 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:29.079 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:29.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:29.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:29.181 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:29.211 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:29.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:29.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:29.313 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:29.345 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:29.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:29.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:29.447 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:29.494 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:29.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:29.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:29.577 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:29.596 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:29.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:29.658 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:29.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:29.759 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:29.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:29.798 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:29.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:29 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 9350/163089 objects degraded (5.733%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:52:29.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:29 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 9350/163089 objects degraded (5.733%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:52:29.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:29 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 9350/163089 objects degraded (5.733%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:52:29.899 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:29.934 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:29.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:29.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:30.036 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:30.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:30.067 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:30.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:30.168 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:30.220 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:30.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:30.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:30.322 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:30.348 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:30.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:30.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:30.449 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:30.488 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:30.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:30.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:30.576 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:30.590 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:30.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:30.641 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:30.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:30.742 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:30.778 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:30.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:30.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:30.831 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:30 smithi070 ceph-mon[107385]: pgmap v2677: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 8 remapped+peering, 1 active+undersized+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.3 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 19 KiB/s rd, 55 KiB/s wr, 425 op/s; 9350/163089 objects degraded (5.733%); 10397/163089 objects misplaced (6.375%); 22 MiB/s, 348 objects/s recovering 2024-09-21T10:52:30.831 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:30 smithi070 ceph-mon[112139]: pgmap v2677: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 8 remapped+peering, 1 active+undersized+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.3 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 19 KiB/s rd, 55 KiB/s wr, 425 op/s; 9350/163089 objects degraded (5.733%); 10397/163089 objects misplaced (6.375%); 22 MiB/s, 348 objects/s recovering 2024-09-21T10:52:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:30 smithi160 ceph-mon[101879]: pgmap v2677: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 8 remapped+peering, 1 active+undersized+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.3 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 19 KiB/s rd, 55 KiB/s wr, 425 op/s; 9350/163089 objects degraded (5.733%); 10397/163089 objects misplaced (6.375%); 22 MiB/s, 348 objects/s recovering 2024-09-21T10:52:30.879 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:30.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:30.921 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:30.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:31.022 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:31.058 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:31.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:31.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:31.159 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:31.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:31.194 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:31.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:31.295 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:31.335 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:31.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:31.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:31.437 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:31.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:31.478 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:31.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:31.564 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T10:52:31.579 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:31.630 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:31.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:31.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:31.732 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:31.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:31.768 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:31.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:31 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/833595231' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:52:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:31 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:52:31.868 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:31 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/833595231' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:52:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:31 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:52:31.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:31 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/833595231' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:52:31.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:31 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:52:31.905 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:31.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:31.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:32.007 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:32.044 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:32.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:32.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:32.145 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:32.179 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:32.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:32.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:32.281 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:32.320 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:32.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:32.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:32.421 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:32.457 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:32.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:32.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:32.559 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:32.597 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:32.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:32.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:32.620 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:52:32.620 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T10:52:32.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.620 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.35670.0:159578 55.6 e1470/1455)", 2024-09-21T10:52:32.620 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:32.609478+0000", 2024-09-21T10:52:32.620 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0072289820000000001, 2024-09-21T10:52:32.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0072532760000000003, 2024-09-21T10:52:32.620 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:32.620 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T10:52:32.621 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:32.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.621 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:32.621 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.609478+0000", 2024-09-21T10:52:32.621 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:32.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.621 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:32.621 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.609478+0000", 2024-09-21T10:52:32.621 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:32.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.621 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:32.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.609477+0000", 2024-09-21T10:52:32.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:52:32.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.622 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:32.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.609487+0000", 2024-09-21T10:52:32.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.061e-05 2024-09-21T10:52:32.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.622 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:32.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.609499+0000", 2024-09-21T10:52:32.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1085e-05 2024-09-21T10:52:32.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:32.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.609506+0000", 2024-09-21T10:52:32.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1709999999999996e-06 2024-09-21T10:52:32.623 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:32.623 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:32.623 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:32.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.623 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.35670.0:159581 55.6 e1470/1455)", 2024-09-21T10:52:32.623 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:32.609575+0000", 2024-09-21T10:52:32.623 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0071320419999999999, 2024-09-21T10:52:32.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0072025509999999997, 2024-09-21T10:52:32.623 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:32.623 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T10:52:32.624 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:32.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.624 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:32.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.609575+0000", 2024-09-21T10:52:32.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:32.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.624 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:32.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.609575+0000", 2024-09-21T10:52:32.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:32.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.624 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:32.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.609575+0000", 2024-09-21T10:52:32.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T10:52:32.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.625 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:32.625 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.609578+0000", 2024-09-21T10:52:32.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1259999999999998e-06 2024-09-21T10:52:32.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.625 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:32.625 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.609584+0000", 2024-09-21T10:52:32.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8290000000000002e-06 2024-09-21T10:52:32.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.625 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:32.626 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.609589+0000", 2024-09-21T10:52:32.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4449999999999998e-06 2024-09-21T10:52:32.626 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:32.626 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:32.626 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:32.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.626 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.626 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.35670.0:159590 55.6 e1470/1455)", 2024-09-21T10:52:32.626 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:32.609712+0000", 2024-09-21T10:52:32.626 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0069947589999999997, 2024-09-21T10:52:32.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0071027030000000001, 2024-09-21T10:52:32.626 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:32.626 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T10:52:32.626 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:32.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:32.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.609712+0000", 2024-09-21T10:52:32.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:32.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:32.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.609712+0000", 2024-09-21T10:52:32.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:32.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:32.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.609712+0000", 2024-09-21T10:52:32.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T10:52:32.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.628 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.628 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:32.628 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.609715+0000", 2024-09-21T10:52:32.628 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2899999999999998e-06 2024-09-21T10:52:32.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.628 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.628 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:32.628 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.609722+0000", 2024-09-21T10:52:32.628 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1060000000000002e-06 2024-09-21T10:52:32.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.628 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.628 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:32.628 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.609727+0000", 2024-09-21T10:52:32.628 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.515e-06 2024-09-21T10:52:32.629 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:32.629 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:32.629 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:32.629 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.629 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.629 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.35670.0:159578 55.6 e1470/1455)", 2024-09-21T10:52:32.629 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:32.612298+0000", 2024-09-21T10:52:32.629 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0044094039999999996, 2024-09-21T10:52:32.629 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0045537720000000002, 2024-09-21T10:52:32.629 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:32.629 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T10:52:32.629 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:32.629 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.629 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:32.629 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.612298+0000", 2024-09-21T10:52:32.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:32.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.630 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:32.630 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.612298+0000", 2024-09-21T10:52:32.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:32.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.630 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:32.630 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.612297+0000", 2024-09-21T10:52:32.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:52:32.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:32.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.612302+0000", 2024-09-21T10:52:32.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1540000000000001e-06 2024-09-21T10:52:32.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.631 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:32.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.612311+0000", 2024-09-21T10:52:32.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3819999999999993e-06 2024-09-21T10:52:32.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.631 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:32.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.612318+0000", 2024-09-21T10:52:32.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8970000000000002e-06 2024-09-21T10:52:32.631 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:32.631 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:32.632 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:32.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.632 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:159578 55.6 55:680186d2:::benchmark_data_smithi070_294394_object54237:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1470)", 2024-09-21T10:52:32.632 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:32.570995+0000", 2024-09-21T10:52:32.632 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.045711749000000003, 2024-09-21T10:52:32.632 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.045908040999999997, 2024-09-21T10:52:32.632 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:32.632 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:52:32.632 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:32.632 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:52:32.632 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:52:32.632 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 159578 2024-09-21T10:52:32.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.633 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:32.633 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.633 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:32.633 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.570995+0000", 2024-09-21T10:52:32.633 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:32.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.633 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.633 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:32.633 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.570995+0000", 2024-09-21T10:52:32.633 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:32.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.634 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:32.634 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.570994+0000", 2024-09-21T10:52:32.634 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:52:32.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.634 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:32.634 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.571000+0000", 2024-09-21T10:52:32.634 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5339999999999996e-06 2024-09-21T10:52:32.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.634 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:32.635 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.571011+0000", 2024-09-21T10:52:32.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1154e-05 2024-09-21T10:52:32.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.635 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.635 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:32.635 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.571018+0000", 2024-09-21T10:52:32.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8059999999999997e-06 2024-09-21T10:52:32.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.635 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:32.636 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.594443+0000", 2024-09-21T10:52:32.636 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.023424969 2024-09-21T10:52:32.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:32.636 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.594593+0000", 2024-09-21T10:52:32.636 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014961900000000001 2024-09-21T10:52:32.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:52:32.636 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.594751+0000", 2024-09-21T10:52:32.636 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015784600000000001 2024-09-21T10:52:32.636 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:32.637 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:32.637 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:32.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.637 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.35670.0:159581 55.6 e1470/1455)", 2024-09-21T10:52:32.637 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:32.612326+0000", 2024-09-21T10:52:32.637 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0043812959999999998, 2024-09-21T10:52:32.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0046484259999999998, 2024-09-21T10:52:32.637 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:32.637 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T10:52:32.637 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:32.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:32.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.612326+0000", 2024-09-21T10:52:32.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:32.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:32.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.612326+0000", 2024-09-21T10:52:32.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:32.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:32.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.612325+0000", 2024-09-21T10:52:32.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967296 2024-09-21T10:52:32.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:32.639 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.612328+0000", 2024-09-21T10:52:32.639 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3750000000000001e-06 2024-09-21T10:52:32.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.639 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:32.639 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.612334+0000", 2024-09-21T10:52:32.639 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6560000000000003e-06 2024-09-21T10:52:32.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.639 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:32.639 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.612339+0000", 2024-09-21T10:52:32.639 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3429999999999998e-06 2024-09-21T10:52:32.639 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:32.640 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:32.640 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:32.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.640 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.640 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.35670.0:159590 55.6 e1470/1455)", 2024-09-21T10:52:32.640 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:32.612343+0000", 2024-09-21T10:52:32.640 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0043642289999999999, 2024-09-21T10:52:32.640 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0046681420000000001, 2024-09-21T10:52:32.640 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:32.640 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T10:52:32.640 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:32.640 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.640 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:32.640 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.612343+0000", 2024-09-21T10:52:32.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:32.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.641 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:32.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.612343+0000", 2024-09-21T10:52:32.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:32.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.641 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:32.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.612343+0000", 2024-09-21T10:52:32.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967296 2024-09-21T10:52:32.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.641 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:32.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.612345+0000", 2024-09-21T10:52:32.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.058e-06 2024-09-21T10:52:32.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.642 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.642 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:32.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.612350+0000", 2024-09-21T10:52:32.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5629999999999999e-06 2024-09-21T10:52:32.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.642 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.642 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:32.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.612354+0000", 2024-09-21T10:52:32.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9859999999999996e-06 2024-09-21T10:52:32.642 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:32.642 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:32.642 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:32.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.643 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDPGPush(55.9 1470/1428 [PushOp(55:9d01bb0d:::benchmark_data_smithi070_294394_object63218:head, version: 1420'3921, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(55:9d01bb0d:::benchmark_data_smithi070_294394_object63218:head@1420'3921, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T10:52:32.643 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:32.614908+0000", 2024-09-21T10:52:32.643 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.001798666, 2024-09-21T10:52:32.643 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021610459999999998, 2024-09-21T10:52:32.643 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:32.643 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T10:52:32.643 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:32.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.643 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:32.643 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.614908+0000", 2024-09-21T10:52:32.643 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:32.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.644 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:32.644 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.614908+0000", 2024-09-21T10:52:32.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:32.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.644 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:32.644 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.614908+0000", 2024-09-21T10:52:32.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T10:52:32.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.644 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:32.644 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.615390+0000", 2024-09-21T10:52:32.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00048257200000000001 2024-09-21T10:52:32.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.645 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.645 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:32.645 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.615409+0000", 2024-09-21T10:52:32.645 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9143e-05 2024-09-21T10:52:32.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.645 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.645 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:32.645 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.615427+0000", 2024-09-21T10:52:32.645 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7317999999999999e-05 2024-09-21T10:52:32.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.645 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.645 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:32.645 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.615655+0000", 2024-09-21T10:52:32.645 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00022807700000000001 2024-09-21T10:52:32.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.646 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.646 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:32.646 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.615667+0000", 2024-09-21T10:52:32.646 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.258e-05 2024-09-21T10:52:32.646 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:32.646 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:32.646 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:32.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.646 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.646 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.35670.0:159587 55.d e1470/1466)", 2024-09-21T10:52:32.646 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:32.616733+0000", 2024-09-21T10:52:32.646 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 4294967295.9999743, 2024-09-21T10:52:32.646 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00038188700000000003, 2024-09-21T10:52:32.646 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:32.647 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T10:52:32.647 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:32.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:32.647 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.616733+0000", 2024-09-21T10:52:32.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:32.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:32.647 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.616733+0000", 2024-09-21T10:52:32.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:32.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:32.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.616732+0000", 2024-09-21T10:52:32.648 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:52:32.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:32.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.616740+0000", 2024-09-21T10:52:32.648 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.481e-06 2024-09-21T10:52:32.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:32.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.616752+0000", 2024-09-21T10:52:32.648 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2169e-05 2024-09-21T10:52:32.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.649 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:32.649 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.616758+0000", 2024-09-21T10:52:32.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5200000000000003e-06 2024-09-21T10:52:32.649 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:32.649 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:32.649 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:32.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.649 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.649 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:159581 55.6 55:6dc57de5:::benchmark_data_smithi070_294394_object54240:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1470)", 2024-09-21T10:52:32.649 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:32.581252+0000", 2024-09-21T10:52:32.649 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.035454613000000003, 2024-09-21T10:52:32.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.035907181000000003, 2024-09-21T10:52:32.649 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:32.649 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:52:32.650 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:32.650 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:52:32.650 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:52:32.650 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 159581 2024-09-21T10:52:32.650 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.650 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:32.650 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.650 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:32.650 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.581252+0000", 2024-09-21T10:52:32.650 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:32.650 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.650 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.650 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:32.650 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.581252+0000", 2024-09-21T10:52:32.651 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:32.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.651 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:32.651 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.581251+0000", 2024-09-21T10:52:32.651 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:52:32.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.651 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:32.651 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.581258+0000", 2024-09-21T10:52:32.651 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4520000000000002e-06 2024-09-21T10:52:32.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.651 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:32.652 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.581277+0000", 2024-09-21T10:52:32.652 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9008000000000002e-05 2024-09-21T10:52:32.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.652 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.652 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:32.652 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.581285+0000", 2024-09-21T10:52:32.652 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2099999999999993e-06 2024-09-21T10:52:32.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.652 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.652 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:32.652 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.597727+0000", 2024-09-21T10:52:32.652 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.016442579999999998 2024-09-21T10:52:32.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.652 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.653 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:32.653 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.597859+0000", 2024-09-21T10:52:32.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013187600000000001 2024-09-21T10:52:32.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.653 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:52:32.653 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.599052+0000", 2024-09-21T10:52:32.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001193026 2024-09-21T10:52:32.653 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:32.653 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:32.653 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:32.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.653 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:159590 55.6 55:6f18303b:::benchmark_data_smithi070_294394_object54249:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1470)", 2024-09-21T10:52:32.653 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:32.589022+0000", 2024-09-21T10:52:32.654 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.027685187, 2024-09-21T10:52:32.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.028201087, 2024-09-21T10:52:32.654 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:52:32.654 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:52:32.654 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:52:32.654 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:52:32.654 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:52:32.654 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 159590 2024-09-21T10:52:32.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.654 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:52:32.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.654 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:52:32.654 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.589022+0000", 2024-09-21T10:52:32.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:32.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.655 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.655 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:52:32.655 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.589022+0000", 2024-09-21T10:52:32.655 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:52:32.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.655 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.655 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:52:32.655 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.589020+0000", 2024-09-21T10:52:32.655 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:52:32.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.655 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.655 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:52:32.655 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.589026+0000", 2024-09-21T10:52:32.656 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9869999999999996e-06 2024-09-21T10:52:32.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.656 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:52:32.656 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.589038+0000", 2024-09-21T10:52:32.656 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.17e-05 2024-09-21T10:52:32.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.656 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:52:32.656 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.589044+0000", 2024-09-21T10:52:32.656 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3980000000000004e-06 2024-09-21T10:52:32.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.656 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:52:32.656 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.602136+0000", 2024-09-21T10:52:32.656 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.013091821 2024-09-21T10:52:32.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:52:32.657 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.602294+0000", 2024-09-21T10:52:32.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015732800000000001 2024-09-21T10:52:32.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:52:32.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:52:32.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:52:32.657 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:32.602494+0000", 2024-09-21T10:52:32.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00020037399999999999 2024-09-21T10:52:32.657 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:32.657 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:52:32.657 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:32.657 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:52:32.658 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T10:52:32.658 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 11 2024-09-21T10:52:32.658 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:52:32.699 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:32.738 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:32.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:32.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:32.761 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:32 smithi070 ceph-mon[107385]: pgmap v2678: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 8 remapped+peering, 1 active+undersized+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.3 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 14 KiB/s rd, 40 KiB/s wr, 312 op/s; 9350/163089 objects degraded (5.733%); 10397/163089 objects misplaced (6.375%); 16 MiB/s, 255 objects/s recovering 2024-09-21T10:52:32.761 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:32 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:52:32.761 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:32 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:52:32.761 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:32 smithi070 ceph-mon[107385]: osdmap e1470: 8 total, 8 up, 5 in 2024-09-21T10:52:32.762 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:32 smithi070 ceph-mon[112139]: pgmap v2678: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 8 remapped+peering, 1 active+undersized+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.3 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 14 KiB/s rd, 40 KiB/s wr, 312 op/s; 9350/163089 objects degraded (5.733%); 10397/163089 objects misplaced (6.375%); 16 MiB/s, 255 objects/s recovering 2024-09-21T10:52:32.762 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:32 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:52:32.762 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:32 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:52:32.762 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:32 smithi070 ceph-mon[112139]: osdmap e1470: 8 total, 8 up, 5 in 2024-09-21T10:52:32.840 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:32 smithi160 ceph-mon[101879]: pgmap v2678: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 8 remapped+peering, 1 active+undersized+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.3 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 14 KiB/s rd, 40 KiB/s wr, 312 op/s; 9350/163089 objects degraded (5.733%); 10397/163089 objects misplaced (6.375%); 16 MiB/s, 255 objects/s recovering 2024-09-21T10:52:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:32 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:52:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:32 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:52:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:32 smithi160 ceph-mon[101879]: osdmap e1470: 8 total, 8 up, 5 in 2024-09-21T10:52:32.891 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:32.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:32.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:32.992 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:33.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:33.041 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:33.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:33.142 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:33.177 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:33.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:33.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:33.279 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:33.316 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:33.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:33.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:33.418 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:33.448 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:33.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:33.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:33.505 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:52:33.552 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:52:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:52:33] "GET /metrics HTTP/1.1" 200 36648 "" "Prometheus/2.51.0" 2024-09-21T10:52:33.556 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:33.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:33.595 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:33.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:33.696 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:33.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:33.764 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:33.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:33.817 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:33 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:52:33.817 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:33 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:52:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:33 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:52:33.865 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:33.888 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:52:33.905 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:33.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:33.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:34.007 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:34.026 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:52:34.026 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:52:34.026 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:52:34.026 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:52:34.026 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:52:34.026 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:52:34.026 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:52:34.026 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:52:34.026 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:52:34.027 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:52:34.027 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:52:34.027 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:52:34.027 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:52:34.027 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:52:34.027 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:52:34.027 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:52:34.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:34.045 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:34.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:34.146 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:34.178 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:34.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:34.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:34.280 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:34.326 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:34.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:34.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:34.428 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:34.458 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:34.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:34.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:34.560 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:34.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:34.594 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:34.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:34.695 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:34.718 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:34.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:34.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:34.820 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:34 smithi160 ceph-mon[101879]: pgmap v2680: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.2 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 494 op/s; 32979/155316 objects degraded (21.233%); 11715/155316 objects misplaced (7.543%); 19 MiB/s, 302 objects/s recovering 2024-09-21T10:52:34.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:34 smithi160 ceph-mon[101879]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-21T10:52:34.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:34 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2931532207' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:52:34.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:52:34.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:52:34.857 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:34.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:34.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:34.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:34 smithi070 ceph-mon[107385]: pgmap v2680: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.2 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 494 op/s; 32979/155316 objects degraded (21.233%); 11715/155316 objects misplaced (7.543%); 19 MiB/s, 302 objects/s recovering 2024-09-21T10:52:34.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:34 smithi070 ceph-mon[107385]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-21T10:52:34.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:34 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2931532207' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:52:34.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:52:34.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:52:34.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:34 smithi070 ceph-mon[112139]: pgmap v2680: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.2 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 494 op/s; 32979/155316 objects degraded (21.233%); 11715/155316 objects misplaced (7.543%); 19 MiB/s, 302 objects/s recovering 2024-09-21T10:52:34.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:34 smithi070 ceph-mon[112139]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-21T10:52:34.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:34 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2931532207' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:52:34.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:52:34.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:52:34.959 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:34.996 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:34.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:34.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:35.098 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:35.132 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:35.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:35.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:35.234 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:35.270 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:35.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:35.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:35.371 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:35.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:35.411 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:35.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:35.512 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:35.538 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:35.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:35.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:35.640 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:35.668 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:35.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:35.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:35.770 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:35.772 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:52:35.799 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:35.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:35.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:35.900 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:35.931 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:35.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:35.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:36.033 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:36.058 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:36.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:36.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:36.159 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:36.198 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:36.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:36.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:36.300 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:36.323 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:36.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:36.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:36.425 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:36.453 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:36.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:36.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:36.555 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:36.582 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:36.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:36.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:36 smithi160 ceph-mon[101879]: pgmap v2681: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.2 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 347 op/s; 32979/155316 objects degraded (21.233%); 11715/155316 objects misplaced (7.543%); 14 MiB/s, 217 objects/s recovering 2024-09-21T10:52:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:52:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:52:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:36 smithi070 ceph-mon[107385]: pgmap v2681: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.2 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 347 op/s; 32979/155316 objects degraded (21.233%); 11715/155316 objects misplaced (7.543%); 14 MiB/s, 217 objects/s recovering 2024-09-21T10:52:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:52:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:52:36.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:36 smithi070 ceph-mon[112139]: pgmap v2681: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.2 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 347 op/s; 32979/155316 objects degraded (21.233%); 11715/155316 objects misplaced (7.543%); 14 MiB/s, 217 objects/s recovering 2024-09-21T10:52:36.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:52:36.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:52:36.684 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:36.719 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:36.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:36.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:36.820 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:36.848 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:36.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:36.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:36.950 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:36.971 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:36.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:36.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:37.073 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:37.118 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:37.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:37.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:37.220 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:37.257 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:37.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:37.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:37.359 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:37.391 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:37.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:37.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:37.493 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:37.526 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:37.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:37.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:37.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:37 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 31384/148917 objects degraded (21.075%), 11 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:52:37.628 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:37.664 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:37.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:37.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:37.765 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:37.806 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:37.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:37.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:37.868 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:37 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 31384/148917 objects degraded (21.075%), 11 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:52:37.869 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:37 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 31384/148917 objects degraded (21.075%), 11 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:52:37.908 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:37.942 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:37.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:37.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:38.043 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:38.080 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:38.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:38.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:38.182 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:38.205 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:52:38.205 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:52:38.218 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:38.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:38.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:38.320 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:38.364 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:38.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:38.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:38.440 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:38 smithi070 ceph-mon[107385]: pgmap v2682: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 559 op/s; 31384/148917 objects degraded (21.075%); 11170/148917 objects misplaced (7.501%); 24 MiB/s, 381 objects/s recovering 2024-09-21T10:52:38.466 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:38.473 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:38.496 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:38.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:38.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:38.598 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:38.636 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:38.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:38.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:38.712 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:38 smithi070 ceph-mon[112139]: pgmap v2682: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 559 op/s; 31384/148917 objects degraded (21.075%); 11170/148917 objects misplaced (7.501%); 24 MiB/s, 381 objects/s recovering 2024-09-21T10:52:38.738 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:38.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:38.769 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:38.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:38.801 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:52:38.803 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T10:52:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:38 smithi160 ceph-mon[101879]: pgmap v2682: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 559 op/s; 31384/148917 objects degraded (21.075%); 11170/148917 objects misplaced (7.501%); 24 MiB/s, 381 objects/s recovering 2024-09-21T10:52:38.870 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:38.905 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:38.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:38.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:39.007 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:39.040 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:52:39.045 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:39.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:39.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:39.147 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:39.179 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:39.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:39.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:39.281 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:39.319 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:39.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:39.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:39.421 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:39.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:39.458 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:39.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:39.506 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:52:39.559 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:39.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:39.608 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:39.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:39.709 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:39.730 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:39.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:39.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:39.831 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:39.880 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:39.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:39.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:39.905 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:52:39.982 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:40.012 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:40.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:40.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:40.114 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:40.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:40.149 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:40.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:40.250 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:40.286 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:40.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:40.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:40.387 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:40.409 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:40.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:40.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:40.511 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:40.544 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:40.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:40.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:40.646 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:40.672 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:40.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:40.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:40.774 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:40.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:40.802 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:40.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:40 smithi160 ceph-mon[101879]: from='client.35874 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:52:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:40 smithi160 ceph-mon[101879]: pgmap v2683: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 549 op/s; 31384/146586 objects degraded (21.410%); 10977/146586 objects misplaced (7.488%); 21 MiB/s, 341 objects/s recovering 2024-09-21T10:52:40.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:40 smithi070 ceph-mon[107385]: from='client.35874 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:52:40.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:40 smithi070 ceph-mon[107385]: pgmap v2683: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 549 op/s; 31384/146586 objects degraded (21.410%); 10977/146586 objects misplaced (7.488%); 21 MiB/s, 341 objects/s recovering 2024-09-21T10:52:40.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:40 smithi070 ceph-mon[112139]: from='client.35874 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:52:40.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:40 smithi070 ceph-mon[112139]: pgmap v2683: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 549 op/s; 31384/146586 objects degraded (21.410%); 10977/146586 objects misplaced (7.488%); 21 MiB/s, 341 objects/s recovering 2024-09-21T10:52:40.903 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:40.940 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:40.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:40.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:41.042 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:41.081 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:41.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:41.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:41.182 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:41.203 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:41.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:41.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:41.305 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:41.336 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:41.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:41.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:41.438 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:41.477 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:41.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:41.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:41.579 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:41.601 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:41.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:41.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:41.702 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:41.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:41.732 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:41.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:41.833 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:41.873 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:41.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:41.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:41.975 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:42.006 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:42.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:42.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:42.108 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:42.147 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:42.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:42.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:42.249 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:42.284 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:42.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:42.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:42.377 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":2682,"stamp":"2024-09-21T10:52:36.913864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3251806988,"num_objects":49639,"num_object_clones":0,"num_object_copies":148917,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31384,"num_objects_misplaced":11170,"num_objects_unfound":0,"num_objects_dirty":49639,"num_whiteouts":0,"num_read":277,"num_read_kb":599,"num_write":161286,"num_write_kb":6743423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27282,"num_bytes_recovered":1767764658,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72722,"ondisk_log_size":72722,"up":297,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":328,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":13372736,"kb_used_data":10193044,"kb_used_omap":141,"kb_used_meta":3179506,"kb_avail":455353024,"statfs":{"total":479975178240,"available":466281496576,"internally_reserved":0,"allocated":10437677056,"data_stored":10412806528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":144912,"internal_metadata":3255814640},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,4,2,2,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-309592064,"num_objects":-4724,"num_object_clones":0,"num_object_copies":-14172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22034,"num_objects_misplaced":773,"num_objects_unfound":0,"num_objects_dirty":-4724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4724,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3227,"num_bytes_recovered":211222528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.448357"},"pg_stats":[{"pgid":"55.f","version":"1470'10116","reported_seq":15002,"reported_epoch":1470,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T10:52:36.705976+0000","last_change":"2024-09-21T10:52:25.539794+0000","last_active":"2024-09-21T10:52:36.705976+0000","last_peered":"2024-09-21T10:52:36.705976+0000","last_clean":"2024-09-21T10:49:43.822243+0000","last_became_active":"2024-09-21T10:51:28.555652+0000","last_became_peered":"2024-09-21T10:51:28.555652+0000","last_unstale":"2024-09-21T10:52:36.705976+0000","last_undegraded":"2024-09-21T10:52:25.537884+0000","last_fullsized":"2024-09-21T10:52:36.705976+0000","mapping_epoch":1448,"log_start":"1438'6040","ondisk_log_start":"1438'6040","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4076,"log_dups_size":0,"ondisk_log_size":4076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:30:34.790715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197656576,"num_objects":3016,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2348,"num_objects_misplaced":668,"num_objects_unfound":0,"num_objects_dirty":3016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10116,"num_write_kb":420224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2397,"num_bytes_recovered":157089792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,0],"avail_no_missing":["6","5","7"],"object_location_counts":[{"shards":"0,6,7","objects":1821},{"shards":"5,6,7","objects":1195}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.e","version":"1470'10008","reported_seq":10147,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:36.354595+0000","last_change":"2024-09-21T10:52:16.415783+0000","last_active":"2024-09-21T10:52:36.354595+0000","last_peered":"2024-09-21T10:52:36.354595+0000","last_clean":"2024-09-21T10:52:36.354595+0000","last_became_active":"2024-09-21T10:52:16.397921+0000","last_became_peered":"2024-09-21T10:52:16.397921+0000","last_unstale":"2024-09-21T10:52:36.354595+0000","last_undegraded":"2024-09-21T10:52:36.354595+0000","last_fullsized":"2024-09-21T10:52:36.354595+0000","mapping_epoch":1463,"log_start":"1433'5656","ondisk_log_start":"1433'5656","created":1411,"last_epoch_clean":1464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4352,"log_dups_size":0,"ondisk_log_size":4352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:13.753306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196870144,"num_objects":3004,"num_object_clones":0,"num_object_copies":9012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10008,"num_write_kb":416384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.d","version":"1470'10330","reported_seq":14344,"reported_epoch":1470,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:52:36.269671+0000","last_change":"2024-09-21T10:52:26.803434+0000","last_active":"2024-09-21T10:52:36.269671+0000","last_peered":"2024-09-21T10:52:36.269671+0000","last_clean":"2024-09-21T10:50:20.259057+0000","last_became_active":"2024-09-21T10:52:26.653674+0000","last_became_peered":"2024-09-21T10:52:26.653674+0000","last_unstale":"2024-09-21T10:52:36.269671+0000","last_undegraded":"2024-09-21T10:52:26.638016+0000","last_fullsized":"2024-09-21T10:52:26.462648+0000","mapping_epoch":1466,"log_start":"1436'5826","ondisk_log_start":"1436'5826","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4504,"log_dups_size":0,"ondisk_log_size":4504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:31:18.206348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199688216,"num_objects":3048,"num_object_clones":0,"num_object_copies":9144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8013,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3048,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10330,"num_write_kb":428033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1963,"num_bytes_recovered":128319488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[4,3],"avail_no_missing":["4","0","6"],"object_location_counts":[{"shards":"0,3,4","objects":268},{"shards":"0,4,7","objects":2780}],"blocked_by":[],"up_primary":7,"acting_primary":4,"purged_snaps":[]},{"pgid":"55.c","version":"1470'10062","reported_seq":14022,"reported_epoch":1470,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T10:52:36.354936+0000","last_change":"2024-09-21T10:52:25.708024+0000","last_active":"2024-09-21T10:52:36.354936+0000","last_peered":"2024-09-21T10:52:36.354936+0000","last_clean":"2024-09-21T10:50:03.007234+0000","last_became_active":"2024-09-21T10:52:25.621852+0000","last_became_peered":"2024-09-21T10:52:25.621852+0000","last_unstale":"2024-09-21T10:52:36.354936+0000","last_undegraded":"2024-09-21T10:52:25.613400+0000","last_fullsized":"2024-09-21T10:52:36.354936+0000","mapping_epoch":1465,"log_start":"1437'5786","ondisk_log_start":"1437'5786","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4276,"log_dups_size":0,"ondisk_log_size":4276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201195520,"num_objects":3070,"num_object_clones":0,"num_object_copies":9210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3070,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10062,"num_write_kb":420224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1909,"num_bytes_recovered":125042688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,0],"avail_no_missing":["3","1","4"],"object_location_counts":[{"shards":"0,3,4","objects":882},{"shards":"3,4","objects":2188}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.b","version":"1470'10314","reported_seq":10383,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:36.684575+0000","last_change":"2024-09-21T10:48:52.934001+0000","last_active":"2024-09-21T10:52:36.684575+0000","last_peered":"2024-09-21T10:52:36.684575+0000","last_clean":"2024-09-21T10:52:36.684575+0000","last_became_active":"2024-09-21T10:48:52.933674+0000","last_became_peered":"2024-09-21T10:48:52.933674+0000","last_unstale":"2024-09-21T10:52:36.684575+0000","last_undegraded":"2024-09-21T10:52:36.684575+0000","last_fullsized":"2024-09-21T10:52:36.684575+0000","mapping_epoch":1411,"log_start":"1438'6206","ondisk_log_start":"1438'6206","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4108,"log_dups_size":0,"ondisk_log_size":4108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:13.178680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203554816,"num_objects":3106,"num_object_clones":0,"num_object_copies":9318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10314,"num_write_kb":429440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.a","version":"1470'10072","reported_seq":10823,"reported_epoch":1470,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T10:52:36.313057+0000","last_change":"2024-09-21T10:52:26.731569+0000","last_active":"2024-09-21T10:52:36.313057+0000","last_peered":"2024-09-21T10:52:36.313057+0000","last_clean":"2024-09-21T10:50:03.021536+0000","last_became_active":"2024-09-21T10:52:26.699954+0000","last_became_peered":"2024-09-21T10:52:26.699954+0000","last_unstale":"2024-09-21T10:52:36.313057+0000","last_undegraded":"2024-09-21T10:52:26.664328+0000","last_fullsized":"2024-09-21T10:52:36.313057+0000","mapping_epoch":1466,"log_start":"1436'5751","ondisk_log_start":"1436'5751","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4321,"log_dups_size":0,"ondisk_log_size":4321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200671232,"num_objects":3062,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10072,"num_write_kb":420288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":19988480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,4,6],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":2198},{"shards":"3,4,6","objects":864}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.9","version":"1470'10112","reported_seq":18301,"reported_epoch":1470,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T10:52:36.354714+0000","last_change":"2024-09-21T10:52:25.571169+0000","last_active":"2024-09-21T10:52:36.354714+0000","last_peered":"2024-09-21T10:52:36.354714+0000","last_clean":"2024-09-21T10:49:43.843813+0000","last_became_active":"2024-09-21T10:49:46.143463+0000","last_became_peered":"2024-09-21T10:49:46.143463+0000","last_unstale":"2024-09-21T10:52:36.354714+0000","last_undegraded":"2024-09-21T10:51:27.558460+0000","last_fullsized":"2024-09-21T10:49:45.850005+0000","mapping_epoch":1428,"log_start":"1437'5769","ondisk_log_start":"1437'5769","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4343,"log_dups_size":0,"ondisk_log_size":4343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200671232,"num_objects":3062,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":161,"num_objects_misplaced":516,"num_objects_unfound":0,"num_objects_dirty":3062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10112,"num_write_kb":421568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4024,"num_bytes_recovered":263716864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4],"avail_no_missing":["3","4","5"],"object_location_counts":[{"shards":"3,4","objects":2385},{"shards":"3,4,5","objects":516}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.8","version":"1470'10016","reported_seq":17785,"reported_epoch":1470,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T10:52:36.700291+0000","last_change":"2024-09-21T10:52:25.496588+0000","last_active":"2024-09-21T10:52:36.700291+0000","last_peered":"2024-09-21T10:52:36.700291+0000","last_clean":"2024-09-21T10:49:43.813262+0000","last_became_active":"2024-09-21T10:51:28.546990+0000","last_became_peered":"2024-09-21T10:51:28.546990+0000","last_unstale":"2024-09-21T10:52:36.700291+0000","last_undegraded":"2024-09-21T10:52:25.490859+0000","last_fullsized":"2024-09-21T10:52:36.700291+0000","mapping_epoch":1448,"log_start":"1438'5934","ondisk_log_start":"1438'5934","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4082,"log_dups_size":0,"ondisk_log_size":4082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:51:45.867981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197787648,"num_objects":3018,"num_object_clones":0,"num_object_copies":9054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":813,"num_objects_misplaced":3018,"num_objects_unfound":0,"num_objects_dirty":3018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10016,"num_write_kb":417088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3825,"num_bytes_recovered":250675200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,0],"avail_no_missing":["6","0","5"],"object_location_counts":[{"shards":"0,3,6","objects":2205},{"shards":"0,5,6","objects":813}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.7","version":"1470'10022","reported_seq":10091,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.889492+0000","last_change":"2024-09-21T10:48:52.933230+0000","last_active":"2024-09-21T10:52:31.889492+0000","last_peered":"2024-09-21T10:52:31.889492+0000","last_clean":"2024-09-21T10:52:31.889492+0000","last_became_active":"2024-09-21T10:48:52.932944+0000","last_became_peered":"2024-09-21T10:48:52.932944+0000","last_unstale":"2024-09-21T10:52:31.889492+0000","last_undegraded":"2024-09-21T10:52:31.889492+0000","last_fullsized":"2024-09-21T10:52:31.889492+0000","mapping_epoch":1411,"log_start":"1431'4928","ondisk_log_start":"1431'4928","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":5094,"log_dups_size":0,"ondisk_log_size":5094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:16:43.711173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214040576,"num_objects":3266,"num_object_clones":0,"num_object_copies":9798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10022,"num_write_kb":425216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"55.6","version":"1470'10096","reported_seq":19556,"reported_epoch":1470,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-21T10:52:31.915274+0000","last_change":"2024-09-21T10:52:25.578778+0000","last_active":"2024-09-21T10:52:31.915274+0000","last_peered":"2024-09-21T10:52:31.915274+0000","last_clean":"2024-09-21T10:49:43.854914+0000","last_became_active":"2024-09-21T10:52:03.460583+0000","last_became_peered":"2024-09-21T10:52:03.460583+0000","last_unstale":"2024-09-21T10:52:31.915274+0000","last_undegraded":"2024-09-21T10:52:31.915274+0000","last_fullsized":"2024-09-21T10:52:03.308381+0000","mapping_epoch":1455,"log_start":"1431'5026","ondisk_log_start":"1431'5026","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":5070,"log_dups_size":0,"ondisk_log_size":5070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:40.566532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211550208,"num_objects":3228,"num_object_clones":0,"num_object_copies":9684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1556,"num_objects_unfound":0,"num_objects_dirty":3228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10096,"num_write_kb":426368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3560,"num_bytes_recovered":232849408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,1],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"1,4,7","objects":272},{"shards":"1,7","objects":1672},{"shards":"3,5,7","objects":1284}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"55.5","version":"1470'10085","reported_seq":10204,"reported_epoch":1470,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-21T10:52:36.629681+0000","last_change":"2024-09-21T10:52:25.760897+0000","last_active":"2024-09-21T10:52:36.629681+0000","last_peered":"2024-09-21T10:52:36.629681+0000","last_clean":"2024-09-21T10:50:02.995767+0000","last_became_active":"2024-09-21T10:52:25.564312+0000","last_became_peered":"2024-09-21T10:52:25.564312+0000","last_unstale":"2024-09-21T10:52:36.629681+0000","last_undegraded":"2024-09-21T10:52:36.629681+0000","last_fullsized":"2024-09-21T10:52:36.629681+0000","mapping_epoch":1465,"log_start":"1438'6014","ondisk_log_start":"1438'6014","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4071,"log_dups_size":0,"ondisk_log_size":4071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:26:53.597959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200081408,"num_objects":3053,"num_object_clones":0,"num_object_copies":9159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3053,"num_objects_unfound":0,"num_objects_dirty":3053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10085,"num_write_kb":420416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,0],"avail_no_missing":["6","0","1","3","4"],"object_location_counts":[{"shards":"0,1,6","objects":3053}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.4","version":"1470'10027","reported_seq":10096,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.841700+0000","last_change":"2024-09-21T10:48:52.934634+0000","last_active":"2024-09-21T10:52:31.841700+0000","last_peered":"2024-09-21T10:52:31.841700+0000","last_clean":"2024-09-21T10:52:31.841700+0000","last_became_active":"2024-09-21T10:48:52.934368+0000","last_became_peered":"2024-09-21T10:48:52.934368+0000","last_unstale":"2024-09-21T10:52:31.841700+0000","last_undegraded":"2024-09-21T10:52:31.841700+0000","last_fullsized":"2024-09-21T10:52:31.841700+0000","mapping_epoch":1411,"log_start":"1431'4945","ondisk_log_start":"1431'4945","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":5082,"log_dups_size":0,"ondisk_log_size":5082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:27:09.216667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215810048,"num_objects":3293,"num_object_clones":0,"num_object_copies":9879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10027,"num_write_kb":426240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"55.3","version":"1470'9730","reported_seq":19960,"reported_epoch":1470,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:52:35.992837+0000","last_change":"2024-09-21T10:52:26.628130+0000","last_active":"2024-09-21T10:52:35.992837+0000","last_peered":"2024-09-21T10:52:35.992837+0000","last_clean":"2024-09-21T10:49:43.839828+0000","last_became_active":"2024-09-21T10:52:26.596220+0000","last_became_peered":"2024-09-21T10:52:26.596220+0000","last_unstale":"2024-09-21T10:52:35.992837+0000","last_undegraded":"2024-09-21T10:52:26.536372+0000","last_fullsized":"2024-09-21T10:52:26.462998+0000","mapping_epoch":1466,"log_start":"1437'5643","ondisk_log_start":"1437'5643","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4087,"log_dups_size":0,"ondisk_log_size":4087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197394432,"num_objects":3012,"num_object_clones":0,"num_object_copies":9036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3413,"num_objects_misplaced":831,"num_objects_unfound":0,"num_objects_dirty":3012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9730,"num_write_kb":407744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3363,"num_bytes_recovered":220266496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,0],"avail_no_missing":["1","5","6"],"object_location_counts":[{"shards":"0,1","objects":831},{"shards":"1,3","objects":949},{"shards":"1,5","objects":1232}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"55.2","version":"1470'9939","reported_seq":19948,"reported_epoch":1470,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T10:52:36.001559+0000","last_change":"2024-09-21T10:52:26.812464+0000","last_active":"2024-09-21T10:52:36.001559+0000","last_peered":"2024-09-21T10:52:36.001559+0000","last_clean":"2024-09-21T10:50:02.974304+0000","last_became_active":"2024-09-21T10:52:26.694338+0000","last_became_peered":"2024-09-21T10:52:26.694338+0000","last_unstale":"2024-09-21T10:52:36.001559+0000","last_undegraded":"2024-09-21T10:52:26.678206+0000","last_fullsized":"2024-09-21T10:52:36.001559+0000","mapping_epoch":1466,"log_start":"1437'5806","ondisk_log_start":"1437'5806","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4133,"log_dups_size":0,"ondisk_log_size":4133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194183168,"num_objects":2963,"num_object_clones":0,"num_object_copies":8889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5120,"num_objects_misplaced":1528,"num_objects_unfound":0,"num_objects_dirty":2963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9939,"num_write_kb":412864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3491,"num_bytes_recovered":228655104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,4],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"0,1,4","objects":806},{"shards":"1,6","objects":2157}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"55.1","version":"1470'9986","reported_seq":11272,"reported_epoch":1470,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:52:31.885597+0000","last_change":"2024-09-21T10:52:15.413755+0000","last_active":"2024-09-21T10:52:31.885597+0000","last_peered":"2024-09-21T10:52:31.885597+0000","last_clean":"2024-09-21T10:50:20.316846+0000","last_became_active":"2024-09-21T10:52:03.442393+0000","last_became_peered":"2024-09-21T10:52:03.442393+0000","last_unstale":"2024-09-21T10:52:31.885597+0000","last_undegraded":"2024-09-21T10:52:15.413422+0000","last_fullsized":"2024-09-21T10:52:03.308195+0000","mapping_epoch":1455,"log_start":"1431'4912","ondisk_log_start":"1431'4912","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":5074,"log_dups_size":0,"ondisk_log_size":5074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:45:04.793364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217055232,"num_objects":3312,"num_object_clones":0,"num_object_copies":9936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9986,"num_write_kb":425536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":596,"num_bytes_recovered":20643840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":3312}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"55.0","version":"1470'10150","reported_seq":13857,"reported_epoch":1470,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T10:52:36.344324+0000","last_change":"2024-09-21T10:52:26.707855+0000","last_active":"2024-09-21T10:52:36.344324+0000","last_peered":"2024-09-21T10:52:36.344324+0000","last_clean":"2024-09-21T10:50:03.041207+0000","last_became_active":"2024-09-21T10:52:26.645887+0000","last_became_peered":"2024-09-21T10:52:26.645887+0000","last_unstale":"2024-09-21T10:52:36.344324+0000","last_undegraded":"2024-09-21T10:52:26.635700+0000","last_fullsized":"2024-09-21T10:52:36.344324+0000","mapping_epoch":1466,"log_start":"1436'5799","ondisk_log_start":"1436'5799","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4351,"log_dups_size":0,"ondisk_log_size":4351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203161600,"num_objects":3100,"num_object_clones":0,"num_object_copies":9300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10150,"num_write_kb":424000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1783,"num_bytes_recovered":116850688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,1],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"1,3,6","objects":860},{"shards":"3,6","objects":2240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":1960,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.588324+0000","last_change":"2024-09-21T10:52:03.323899+0000","last_active":"2024-09-21T10:52:28.588324+0000","last_peered":"2024-09-21T10:52:28.588324+0000","last_clean":"2024-09-21T10:52:28.588324+0000","last_became_active":"2024-09-21T10:52:03.314626+0000","last_became_peered":"2024-09-21T10:52:03.314626+0000","last_unstale":"2024-09-21T10:52:28.588324+0000","last_undegraded":"2024-09-21T10:52:28.588324+0000","last_fullsized":"2024-09-21T10:52:28.588324+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:13.939806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1459,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.473254+0000","last_change":"2024-09-21T10:52:15.396755+0000","last_active":"2024-09-21T10:52:28.473254+0000","last_peered":"2024-09-21T10:52:28.473254+0000","last_clean":"2024-09-21T10:52:28.473254+0000","last_became_active":"2024-09-21T10:52:15.396417+0000","last_became_peered":"2024-09-21T10:52:15.396417+0000","last_unstale":"2024-09-21T10:52:28.473254+0000","last_undegraded":"2024-09-21T10:52:28.473254+0000","last_fullsized":"2024-09-21T10:52:28.473254+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:25.005574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":2249,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.568997+0000","last_change":"2024-09-21T10:52:15.400227+0000","last_active":"2024-09-21T10:52:31.568997+0000","last_peered":"2024-09-21T10:52:31.568997+0000","last_clean":"2024-09-21T10:52:31.568997+0000","last_became_active":"2024-09-21T10:52:15.399888+0000","last_became_peered":"2024-09-21T10:52:15.399888+0000","last_unstale":"2024-09-21T10:52:31.568997+0000","last_undegraded":"2024-09-21T10:52:31.568997+0000","last_fullsized":"2024-09-21T10:52:31.568997+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:42:52.565184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2265,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.568238+0000","last_change":"2024-09-21T10:51:27.458216+0000","last_active":"2024-09-21T10:52:31.568238+0000","last_peered":"2024-09-21T10:52:31.568238+0000","last_clean":"2024-09-21T10:52:31.568238+0000","last_became_active":"2024-09-21T10:51:27.457948+0000","last_became_peered":"2024-09-21T10:51:27.457948+0000","last_unstale":"2024-09-21T10:52:31.568238+0000","last_undegraded":"2024-09-21T10:52:31.568238+0000","last_fullsized":"2024-09-21T10:52:31.568238+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:25:29.584079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":98,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.474405+0000","last_change":"2024-09-21T10:49:44.878242+0000","last_active":"2024-09-21T10:52:28.474405+0000","last_peered":"2024-09-21T10:52:28.474405+0000","last_clean":"2024-09-21T10:52:28.474405+0000","last_became_active":"2024-09-21T10:49:44.866578+0000","last_became_peered":"2024-09-21T10:49:44.866578+0000","last_unstale":"2024-09-21T10:52:28.474405+0000","last_undegraded":"2024-09-21T10:52:28.474405+0000","last_fullsized":"2024-09-21T10:52:28.474405+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:17:24.145551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1465,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.473872+0000","last_change":"2024-09-21T10:51:27.460539+0000","last_active":"2024-09-21T10:52:28.473872+0000","last_peered":"2024-09-21T10:52:28.473872+0000","last_clean":"2024-09-21T10:52:28.473872+0000","last_became_active":"2024-09-21T10:51:27.460211+0000","last_became_peered":"2024-09-21T10:51:27.460211+0000","last_unstale":"2024-09-21T10:52:28.473872+0000","last_undegraded":"2024-09-21T10:52:28.473872+0000","last_fullsized":"2024-09-21T10:52:28.473872+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:14:15.700998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":137,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.473977+0000","last_change":"2024-09-21T10:52:15.405886+0000","last_active":"2024-09-21T10:52:28.473977+0000","last_peered":"2024-09-21T10:52:28.473977+0000","last_clean":"2024-09-21T10:52:28.473977+0000","last_became_active":"2024-09-21T10:52:15.397317+0000","last_became_peered":"2024-09-21T10:52:15.397317+0000","last_unstale":"2024-09-21T10:52:28.473977+0000","last_undegraded":"2024-09-21T10:52:28.473977+0000","last_fullsized":"2024-09-21T10:52:28.473977+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:43:48.813861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1456,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.474087+0000","last_change":"2024-09-21T10:09:06.560175+0000","last_active":"2024-09-21T10:52:28.474087+0000","last_peered":"2024-09-21T10:52:28.474087+0000","last_clean":"2024-09-21T10:52:28.474087+0000","last_became_active":"2024-09-21T10:09:06.558584+0000","last_became_peered":"2024-09-21T10:09:06.558584+0000","last_unstale":"2024-09-21T10:52:28.474087+0000","last_undegraded":"2024-09-21T10:52:28.474087+0000","last_fullsized":"2024-09-21T10:52:28.474087+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:02.918339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":46,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.473921+0000","last_change":"2024-09-21T10:51:27.437665+0000","last_active":"2024-09-21T10:52:28.473921+0000","last_peered":"2024-09-21T10:52:28.473921+0000","last_clean":"2024-09-21T10:52:28.473921+0000","last_became_active":"2024-09-21T10:51:27.436481+0000","last_became_peered":"2024-09-21T10:51:27.436481+0000","last_unstale":"2024-09-21T10:52:28.473921+0000","last_undegraded":"2024-09-21T10:52:28.473921+0000","last_fullsized":"2024-09-21T10:52:28.473921+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:07:50.267297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1020,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.575335+0000","last_change":"2024-09-21T10:51:27.459033+0000","last_active":"2024-09-21T10:52:31.575335+0000","last_peered":"2024-09-21T10:52:31.575335+0000","last_clean":"2024-09-21T10:52:31.575335+0000","last_became_active":"2024-09-21T10:51:27.458728+0000","last_became_peered":"2024-09-21T10:51:27.458728+0000","last_unstale":"2024-09-21T10:52:31.575335+0000","last_undegraded":"2024-09-21T10:52:31.575335+0000","last_fullsized":"2024-09-21T10:52:31.575335+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:20.133032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":1485,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.474053+0000","last_change":"2024-09-21T10:52:25.473550+0000","last_active":"2024-09-21T10:52:28.474053+0000","last_peered":"2024-09-21T10:52:28.474053+0000","last_clean":"2024-09-21T10:52:28.474053+0000","last_became_active":"2024-09-21T10:52:25.473105+0000","last_became_peered":"2024-09-21T10:52:25.473105+0000","last_unstale":"2024-09-21T10:52:28.474053+0000","last_undegraded":"2024-09-21T10:52:28.474053+0000","last_fullsized":"2024-09-21T10:52:28.474053+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:15:45.352518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2245,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.568319+0000","last_change":"2024-09-21T10:40:13.432531+0000","last_active":"2024-09-21T10:52:31.568319+0000","last_peered":"2024-09-21T10:52:31.568319+0000","last_clean":"2024-09-21T10:52:31.568319+0000","last_became_active":"2024-09-21T10:40:13.432210+0000","last_became_peered":"2024-09-21T10:40:13.432210+0000","last_unstale":"2024-09-21T10:52:31.568319+0000","last_undegraded":"2024-09-21T10:52:31.568319+0000","last_fullsized":"2024-09-21T10:52:31.568319+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:20:10.998607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2217,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.575815+0000","last_change":"2024-09-21T10:09:06.530394+0000","last_active":"2024-09-21T10:52:31.575815+0000","last_peered":"2024-09-21T10:52:31.575815+0000","last_clean":"2024-09-21T10:52:31.575815+0000","last_became_active":"2024-09-21T10:09:06.529307+0000","last_became_peered":"2024-09-21T10:09:06.529307+0000","last_unstale":"2024-09-21T10:52:31.575815+0000","last_undegraded":"2024-09-21T10:52:31.575815+0000","last_fullsized":"2024-09-21T10:52:31.575815+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:54.287397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":15,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.471331+0000","last_change":"2024-09-21T10:52:25.489896+0000","last_active":"2024-09-21T10:52:28.471331+0000","last_peered":"2024-09-21T10:52:28.471331+0000","last_clean":"2024-09-21T10:52:28.471331+0000","last_became_active":"2024-09-21T10:52:25.489154+0000","last_became_peered":"2024-09-21T10:52:25.489154+0000","last_unstale":"2024-09-21T10:52:28.471331+0000","last_undegraded":"2024-09-21T10:52:28.471331+0000","last_fullsized":"2024-09-21T10:52:28.471331+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:36:02.076931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2245,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.571390+0000","last_change":"2024-09-21T10:52:25.493309+0000","last_active":"2024-09-21T10:52:31.571390+0000","last_peered":"2024-09-21T10:52:31.571390+0000","last_clean":"2024-09-21T10:52:31.571390+0000","last_became_active":"2024-09-21T10:52:25.492568+0000","last_became_peered":"2024-09-21T10:52:25.492568+0000","last_unstale":"2024-09-21T10:52:31.571390+0000","last_undegraded":"2024-09-21T10:52:31.571390+0000","last_fullsized":"2024-09-21T10:52:31.571390+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:19:19.393370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1472,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.588372+0000","last_change":"2024-09-21T10:52:03.317474+0000","last_active":"2024-09-21T10:52:28.588372+0000","last_peered":"2024-09-21T10:52:28.588372+0000","last_clean":"2024-09-21T10:52:28.588372+0000","last_became_active":"2024-09-21T10:52:03.317001+0000","last_became_peered":"2024-09-21T10:52:03.317001+0000","last_unstale":"2024-09-21T10:52:28.588372+0000","last_undegraded":"2024-09-21T10:52:28.588372+0000","last_fullsized":"2024-09-21T10:52:28.588372+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:06:07.039763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2245,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.571653+0000","last_change":"2024-09-21T10:52:15.412246+0000","last_active":"2024-09-21T10:52:31.571653+0000","last_peered":"2024-09-21T10:52:31.571653+0000","last_clean":"2024-09-21T10:52:31.571653+0000","last_became_active":"2024-09-21T10:52:15.406204+0000","last_became_peered":"2024-09-21T10:52:15.406204+0000","last_unstale":"2024-09-21T10:52:31.571653+0000","last_undegraded":"2024-09-21T10:52:31.571653+0000","last_fullsized":"2024-09-21T10:52:31.571653+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:03.524324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2225,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.568127+0000","last_change":"2024-09-21T10:52:15.402822+0000","last_active":"2024-09-21T10:52:31.568127+0000","last_peered":"2024-09-21T10:52:31.568127+0000","last_clean":"2024-09-21T10:52:31.568127+0000","last_became_active":"2024-09-21T10:52:15.402548+0000","last_became_peered":"2024-09-21T10:52:15.402548+0000","last_unstale":"2024-09-21T10:52:31.568127+0000","last_undegraded":"2024-09-21T10:52:31.568127+0000","last_fullsized":"2024-09-21T10:52:31.568127+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:04:04.667598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2250,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.568444+0000","last_change":"2024-09-21T10:52:25.499172+0000","last_active":"2024-09-21T10:52:31.568444+0000","last_peered":"2024-09-21T10:52:31.568444+0000","last_clean":"2024-09-21T10:52:31.568444+0000","last_became_active":"2024-09-21T10:52:25.498835+0000","last_became_peered":"2024-09-21T10:52:25.498835+0000","last_unstale":"2024-09-21T10:52:31.568444+0000","last_undegraded":"2024-09-21T10:52:31.568444+0000","last_fullsized":"2024-09-21T10:52:31.568444+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:10:12.434470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.471253+0000","last_change":"2024-09-21T10:52:25.489774+0000","last_active":"2024-09-21T10:52:28.471253+0000","last_peered":"2024-09-21T10:52:28.471253+0000","last_clean":"2024-09-21T10:52:28.471253+0000","last_became_active":"2024-09-21T10:52:25.487788+0000","last_became_peered":"2024-09-21T10:52:25.487788+0000","last_unstale":"2024-09-21T10:52:28.471253+0000","last_undegraded":"2024-09-21T10:52:28.471253+0000","last_fullsized":"2024-09-21T10:52:28.471253+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:48.842419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":21,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.471215+0000","last_change":"2024-09-21T10:52:25.565465+0000","last_active":"2024-09-21T10:52:28.471215+0000","last_peered":"2024-09-21T10:52:28.471215+0000","last_clean":"2024-09-21T10:52:28.471215+0000","last_became_active":"2024-09-21T10:52:25.510131+0000","last_became_peered":"2024-09-21T10:52:25.510131+0000","last_unstale":"2024-09-21T10:52:28.471215+0000","last_undegraded":"2024-09-21T10:52:28.471215+0000","last_fullsized":"2024-09-21T10:52:28.471215+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:41:13.870503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":450,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2252,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.602995+0000","last_change":"2024-09-21T10:52:25.504787+0000","last_active":"2024-09-21T10:52:31.602995+0000","last_peered":"2024-09-21T10:52:31.602995+0000","last_clean":"2024-09-21T10:52:31.602995+0000","last_became_active":"2024-09-21T10:52:25.504198+0000","last_became_peered":"2024-09-21T10:52:25.504198+0000","last_unstale":"2024-09-21T10:52:31.602995+0000","last_undegraded":"2024-09-21T10:52:31.602995+0000","last_fullsized":"2024-09-21T10:52:31.602995+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:38:25.666224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":1489,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.474196+0000","last_change":"2024-09-21T10:44:14.415264+0000","last_active":"2024-09-21T10:52:28.474196+0000","last_peered":"2024-09-21T10:52:28.474196+0000","last_clean":"2024-09-21T10:52:28.474196+0000","last_became_active":"2024-09-21T10:44:14.407813+0000","last_became_peered":"2024-09-21T10:44:14.407813+0000","last_unstale":"2024-09-21T10:52:28.474196+0000","last_undegraded":"2024-09-21T10:52:28.474196+0000","last_fullsized":"2024-09-21T10:52:28.474196+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:55.933601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1442,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.474156+0000","last_change":"2024-09-21T10:07:56.001756+0000","last_active":"2024-09-21T10:52:28.474156+0000","last_peered":"2024-09-21T10:52:28.474156+0000","last_clean":"2024-09-21T10:52:28.474156+0000","last_became_active":"2024-09-21T10:07:56.000789+0000","last_became_peered":"2024-09-21T10:07:56.000789+0000","last_unstale":"2024-09-21T10:52:28.474156+0000","last_undegraded":"2024-09-21T10:52:28.474156+0000","last_fullsized":"2024-09-21T10:52:28.474156+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:56:04.302902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"346'149","reported_seq":902,"reported_epoch":1470,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:52:31.571394+0000","last_change":"2024-09-21T10:52:26.478603+0000","last_active":"2024-09-21T10:52:31.571394+0000","last_peered":"2024-09-21T10:52:31.571394+0000","last_clean":"2024-09-21T10:52:16.392213+0000","last_became_active":"2024-09-21T10:52:26.478167+0000","last_became_peered":"2024-09-21T10:52:26.478167+0000","last_unstale":"2024-09-21T10:52:31.571394+0000","last_undegraded":"2024-09-21T10:52:26.463682+0000","last_fullsized":"2024-09-21T10:52:26.463251+0000","mapping_epoch":1466,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1449,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1251389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,6],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":1},{"shards":"3,6","objects":-1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1457,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.475103+0000","last_change":"2024-09-21T10:52:25.473127+0000","last_active":"2024-09-21T10:52:28.475103+0000","last_peered":"2024-09-21T10:52:28.475103+0000","last_clean":"2024-09-21T10:52:28.475103+0000","last_became_active":"2024-09-21T10:52:25.472805+0000","last_became_peered":"2024-09-21T10:52:25.472805+0000","last_unstale":"2024-09-21T10:52:28.475103+0000","last_undegraded":"2024-09-21T10:52:28.475103+0000","last_fullsized":"2024-09-21T10:52:28.475103+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:13.038797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1503,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.588411+0000","last_change":"2024-09-21T10:52:25.493263+0000","last_active":"2024-09-21T10:52:28.588411+0000","last_peered":"2024-09-21T10:52:28.588411+0000","last_clean":"2024-09-21T10:52:28.588411+0000","last_became_active":"2024-09-21T10:52:25.492970+0000","last_became_peered":"2024-09-21T10:52:25.492970+0000","last_unstale":"2024-09-21T10:52:28.588411+0000","last_undegraded":"2024-09-21T10:52:28.588411+0000","last_fullsized":"2024-09-21T10:52:28.588411+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:43:40.305783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":96,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.577064+0000","last_change":"2024-09-21T10:52:25.513834+0000","last_active":"2024-09-21T10:52:31.577064+0000","last_peered":"2024-09-21T10:52:31.577064+0000","last_clean":"2024-09-21T10:52:31.577064+0000","last_became_active":"2024-09-21T10:52:25.475423+0000","last_became_peered":"2024-09-21T10:52:25.475423+0000","last_unstale":"2024-09-21T10:52:31.577064+0000","last_undegraded":"2024-09-21T10:52:31.577064+0000","last_fullsized":"2024-09-21T10:52:31.577064+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:10.216527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"346'149","reported_seq":889,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.473882+0000","last_change":"2024-09-21T10:52:24.459450+0000","last_active":"2024-09-21T10:52:28.473882+0000","last_peered":"2024-09-21T10:52:28.473882+0000","last_clean":"2024-09-21T10:52:28.473882+0000","last_became_active":"2024-09-21T10:52:15.405544+0000","last_became_peered":"2024-09-21T10:52:15.405544+0000","last_unstale":"2024-09-21T10:52:28.473882+0000","last_undegraded":"2024-09-21T10:52:28.473882+0000","last_fullsized":"2024-09-21T10:52:28.473882+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2217,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.571563+0000","last_change":"2024-09-21T10:52:25.494316+0000","last_active":"2024-09-21T10:52:31.571563+0000","last_peered":"2024-09-21T10:52:31.571563+0000","last_clean":"2024-09-21T10:52:31.571563+0000","last_became_active":"2024-09-21T10:52:25.494016+0000","last_became_peered":"2024-09-21T10:52:25.494016+0000","last_unstale":"2024-09-21T10:52:31.571563+0000","last_undegraded":"2024-09-21T10:52:31.571563+0000","last_fullsized":"2024-09-21T10:52:31.571563+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:07.859584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2218,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.577580+0000","last_change":"2024-09-21T10:07:56.006236+0000","last_active":"2024-09-21T10:52:31.577580+0000","last_peered":"2024-09-21T10:52:31.577580+0000","last_clean":"2024-09-21T10:52:31.577580+0000","last_became_active":"2024-09-21T10:07:56.004439+0000","last_became_peered":"2024-09-21T10:07:56.004439+0000","last_unstale":"2024-09-21T10:52:31.577580+0000","last_undegraded":"2024-09-21T10:52:31.577580+0000","last_fullsized":"2024-09-21T10:52:31.577580+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:39.422467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":2268,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.568431+0000","last_change":"2024-09-21T10:52:25.536401+0000","last_active":"2024-09-21T10:52:31.568431+0000","last_peered":"2024-09-21T10:52:31.568431+0000","last_clean":"2024-09-21T10:52:31.568431+0000","last_became_active":"2024-09-21T10:52:25.496480+0000","last_became_peered":"2024-09-21T10:52:25.496480+0000","last_unstale":"2024-09-21T10:52:31.568431+0000","last_undegraded":"2024-09-21T10:52:31.568431+0000","last_fullsized":"2024-09-21T10:52:31.568431+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:50:34.820373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"346'149","reported_seq":714,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.587032+0000","last_change":"2024-09-21T10:51:30.450301+0000","last_active":"2024-09-21T10:52:28.587032+0000","last_peered":"2024-09-21T10:52:28.587032+0000","last_clean":"2024-09-21T10:52:28.587032+0000","last_became_active":"2024-09-21T10:51:30.449925+0000","last_became_peered":"2024-09-21T10:51:30.449925+0000","last_unstale":"2024-09-21T10:52:28.587032+0000","last_undegraded":"2024-09-21T10:52:28.587032+0000","last_fullsized":"2024-09-21T10:52:28.587032+0000","mapping_epoch":1450,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1451,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:41.011085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:41.011085+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:38:33.451261+0000","scrub_duration":0.0061250310000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2185,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.470706+0000","last_change":"2024-09-21T10:51:27.431416+0000","last_active":"2024-09-21T10:52:28.470706+0000","last_peered":"2024-09-21T10:52:28.470706+0000","last_clean":"2024-09-21T10:52:28.470706+0000","last_became_active":"2024-09-21T10:51:27.431127+0000","last_became_peered":"2024-09-21T10:51:27.431127+0000","last_unstale":"2024-09-21T10:52:28.470706+0000","last_undegraded":"2024-09-21T10:52:28.470706+0000","last_fullsized":"2024-09-21T10:52:28.470706+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:27.721665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.471021+0000","last_change":"2024-09-21T10:52:25.468849+0000","last_active":"2024-09-21T10:52:28.471021+0000","last_peered":"2024-09-21T10:52:28.471021+0000","last_clean":"2024-09-21T10:52:28.471021+0000","last_became_active":"2024-09-21T10:52:25.468552+0000","last_became_peered":"2024-09-21T10:52:25.468552+0000","last_unstale":"2024-09-21T10:52:28.471021+0000","last_undegraded":"2024-09-21T10:52:28.471021+0000","last_fullsized":"2024-09-21T10:52:28.471021+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:36:41.692028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2268,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.575262+0000","last_change":"2024-09-21T10:51:27.436608+0000","last_active":"2024-09-21T10:52:31.575262+0000","last_peered":"2024-09-21T10:52:31.575262+0000","last_clean":"2024-09-21T10:52:31.575262+0000","last_became_active":"2024-09-21T10:51:27.434949+0000","last_became_peered":"2024-09-21T10:51:27.434949+0000","last_unstale":"2024-09-21T10:52:31.575262+0000","last_undegraded":"2024-09-21T10:52:31.575262+0000","last_fullsized":"2024-09-21T10:52:31.575262+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:49:23.439425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"346'149","reported_seq":757,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.595270+0000","last_change":"2024-09-21T10:52:28.590793+0000","last_active":"2024-09-21T10:52:28.595270+0000","last_peered":"2024-09-21T10:52:28.595270+0000","last_clean":"2024-09-21T10:52:28.595270+0000","last_became_active":"2024-09-21T10:52:28.588662+0000","last_became_peered":"2024-09-21T10:52:28.588662+0000","last_unstale":"2024-09-21T10:52:28.595270+0000","last_undegraded":"2024-09-21T10:52:28.595270+0000","last_fullsized":"2024-09-21T10:52:28.595270+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:44.018645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:44.018645+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:46.849427+0000","scrub_duration":0.0012789399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2223,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.602859+0000","last_change":"2024-09-21T10:51:27.440425+0000","last_active":"2024-09-21T10:52:31.602859+0000","last_peered":"2024-09-21T10:52:31.602859+0000","last_clean":"2024-09-21T10:52:31.602859+0000","last_became_active":"2024-09-21T10:51:27.439666+0000","last_became_peered":"2024-09-21T10:51:27.439666+0000","last_unstale":"2024-09-21T10:52:31.602859+0000","last_undegraded":"2024-09-21T10:52:31.602859+0000","last_fullsized":"2024-09-21T10:52:31.602859+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:56:47.410516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2291,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.575092+0000","last_change":"2024-09-21T10:52:03.314926+0000","last_active":"2024-09-21T10:52:31.575092+0000","last_peered":"2024-09-21T10:52:31.575092+0000","last_clean":"2024-09-21T10:52:31.575092+0000","last_became_active":"2024-09-21T10:52:03.314602+0000","last_became_peered":"2024-09-21T10:52:03.314602+0000","last_unstale":"2024-09-21T10:52:31.575092+0000","last_undegraded":"2024-09-21T10:52:31.575092+0000","last_fullsized":"2024-09-21T10:52:31.575092+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:13.774521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1446,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.567974+0000","last_change":"2024-09-21T10:52:25.502830+0000","last_active":"2024-09-21T10:52:31.567974+0000","last_peered":"2024-09-21T10:52:31.567974+0000","last_clean":"2024-09-21T10:52:31.567974+0000","last_became_active":"2024-09-21T10:52:25.502505+0000","last_became_peered":"2024-09-21T10:52:25.502505+0000","last_unstale":"2024-09-21T10:52:31.567974+0000","last_undegraded":"2024-09-21T10:52:31.567974+0000","last_fullsized":"2024-09-21T10:52:31.567974+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:01:14.965694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"346'149","reported_seq":812,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.473213+0000","last_change":"2024-09-21T10:52:24.458657+0000","last_active":"2024-09-21T10:52:28.473213+0000","last_peered":"2024-09-21T10:52:28.473213+0000","last_clean":"2024-09-21T10:52:28.473213+0000","last_became_active":"2024-09-21T10:51:30.436983+0000","last_became_peered":"2024-09-21T10:51:30.436983+0000","last_unstale":"2024-09-21T10:52:28.473213+0000","last_undegraded":"2024-09-21T10:52:28.473213+0000","last_fullsized":"2024-09-21T10:52:28.473213+0000","mapping_epoch":1450,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1451,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2285,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.575514+0000","last_change":"2024-09-21T10:52:15.397180+0000","last_active":"2024-09-21T10:52:31.575514+0000","last_peered":"2024-09-21T10:52:31.575514+0000","last_clean":"2024-09-21T10:52:31.575514+0000","last_became_active":"2024-09-21T10:52:15.396879+0000","last_became_peered":"2024-09-21T10:52:15.396879+0000","last_unstale":"2024-09-21T10:52:31.575514+0000","last_undegraded":"2024-09-21T10:52:31.575514+0000","last_fullsized":"2024-09-21T10:52:31.575514+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:03:07.162178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1455,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.474226+0000","last_change":"2024-09-21T10:51:27.436590+0000","last_active":"2024-09-21T10:52:28.474226+0000","last_peered":"2024-09-21T10:52:28.474226+0000","last_clean":"2024-09-21T10:52:28.474226+0000","last_became_active":"2024-09-21T10:51:27.435332+0000","last_became_peered":"2024-09-21T10:51:27.435332+0000","last_unstale":"2024-09-21T10:52:28.474226+0000","last_undegraded":"2024-09-21T10:52:28.474226+0000","last_fullsized":"2024-09-21T10:52:28.474226+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:05:42.885506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":180,"reported_epoch":1470,"state":"active+recovery_wait","last_fresh":"2024-09-21T10:52:31.602648+0000","last_change":"2024-09-21T10:52:26.680832+0000","last_active":"2024-09-21T10:52:31.602648+0000","last_peered":"2024-09-21T10:52:31.602648+0000","last_clean":"2024-09-21T10:51:45.437560+0000","last_became_active":"2024-09-21T10:52:03.315398+0000","last_became_peered":"2024-09-21T10:52:03.315398+0000","last_unstale":"2024-09-21T10:52:31.602648+0000","last_undegraded":"2024-09-21T10:52:31.602648+0000","last_fullsized":"2024-09-21T10:52:31.602648+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":["1","3","4","6"],"object_location_counts":[{"shards":"1,3,6","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"346'149","reported_seq":783,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.563512+0000","last_change":"2024-09-21T10:52:28.495319+0000","last_active":"2024-09-21T10:52:31.563512+0000","last_peered":"2024-09-21T10:52:31.563512+0000","last_clean":"2024-09-21T10:52:31.563512+0000","last_became_active":"2024-09-21T10:52:28.494923+0000","last_became_peered":"2024-09-21T10:52:28.494923+0000","last_unstale":"2024-09-21T10:52:31.563512+0000","last_undegraded":"2024-09-21T10:52:31.563512+0000","last_fullsized":"2024-09-21T10:52:31.563512+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:52:17.894787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:52:17.894787+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":173,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.474311+0000","last_change":"2024-09-21T10:52:25.475200+0000","last_active":"2024-09-21T10:52:28.474311+0000","last_peered":"2024-09-21T10:52:28.474311+0000","last_clean":"2024-09-21T10:52:28.474311+0000","last_became_active":"2024-09-21T10:52:25.474872+0000","last_became_peered":"2024-09-21T10:52:25.474872+0000","last_unstale":"2024-09-21T10:52:28.474311+0000","last_undegraded":"2024-09-21T10:52:28.474311+0000","last_fullsized":"2024-09-21T10:52:28.474311+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:36:34.207844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2242,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.576031+0000","last_change":"2024-09-21T10:52:03.325889+0000","last_active":"2024-09-21T10:52:31.576031+0000","last_peered":"2024-09-21T10:52:31.576031+0000","last_clean":"2024-09-21T10:52:31.576031+0000","last_became_active":"2024-09-21T10:52:03.325500+0000","last_became_peered":"2024-09-21T10:52:03.325500+0000","last_unstale":"2024-09-21T10:52:31.576031+0000","last_undegraded":"2024-09-21T10:52:31.576031+0000","last_fullsized":"2024-09-21T10:52:31.576031+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:20.708252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1027,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.568145+0000","last_change":"2024-09-21T10:51:27.457894+0000","last_active":"2024-09-21T10:52:31.568145+0000","last_peered":"2024-09-21T10:52:31.568145+0000","last_clean":"2024-09-21T10:52:31.568145+0000","last_became_active":"2024-09-21T10:51:27.457431+0000","last_became_peered":"2024-09-21T10:51:27.457431+0000","last_unstale":"2024-09-21T10:52:31.568145+0000","last_undegraded":"2024-09-21T10:52:31.568145+0000","last_fullsized":"2024-09-21T10:52:31.568145+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:03:33.550759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"346'149","reported_seq":898,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.577838+0000","last_change":"2024-09-21T10:52:15.396042+0000","last_active":"2024-09-21T10:52:31.577838+0000","last_peered":"2024-09-21T10:52:31.577838+0000","last_clean":"2024-09-21T10:52:31.577838+0000","last_became_active":"2024-09-21T10:52:15.395717+0000","last_became_peered":"2024-09-21T10:52:15.395717+0000","last_unstale":"2024-09-21T10:52:31.577838+0000","last_undegraded":"2024-09-21T10:52:31.577838+0000","last_fullsized":"2024-09-21T10:52:31.577838+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2267,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.576999+0000","last_change":"2024-09-21T10:52:25.476572+0000","last_active":"2024-09-21T10:52:31.576999+0000","last_peered":"2024-09-21T10:52:31.576999+0000","last_clean":"2024-09-21T10:52:31.576999+0000","last_became_active":"2024-09-21T10:52:25.475803+0000","last_became_peered":"2024-09-21T10:52:25.475803+0000","last_unstale":"2024-09-21T10:52:31.576999+0000","last_undegraded":"2024-09-21T10:52:31.576999+0000","last_fullsized":"2024-09-21T10:52:31.576999+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:06:44.191486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2201,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.474994+0000","last_change":"2024-09-21T10:52:03.326753+0000","last_active":"2024-09-21T10:52:28.474994+0000","last_peered":"2024-09-21T10:52:28.474994+0000","last_clean":"2024-09-21T10:52:28.474994+0000","last_became_active":"2024-09-21T10:52:03.326348+0000","last_became_peered":"2024-09-21T10:52:03.326348+0000","last_unstale":"2024-09-21T10:52:28.474994+0000","last_undegraded":"2024-09-21T10:52:28.474994+0000","last_fullsized":"2024-09-21T10:52:28.474994+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:57:38.100799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":124,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.470592+0000","last_change":"2024-09-21T10:51:27.426006+0000","last_active":"2024-09-21T10:52:28.470592+0000","last_peered":"2024-09-21T10:52:28.470592+0000","last_clean":"2024-09-21T10:52:28.470592+0000","last_became_active":"2024-09-21T10:51:27.425698+0000","last_became_peered":"2024-09-21T10:51:27.425698+0000","last_unstale":"2024-09-21T10:52:28.470592+0000","last_undegraded":"2024-09-21T10:52:28.470592+0000","last_fullsized":"2024-09-21T10:52:28.470592+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:35:13.210330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":18,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.586341+0000","last_change":"2024-09-21T10:52:15.401737+0000","last_active":"2024-09-21T10:52:28.586341+0000","last_peered":"2024-09-21T10:52:28.586341+0000","last_clean":"2024-09-21T10:52:28.586341+0000","last_became_active":"2024-09-21T10:52:15.400859+0000","last_became_peered":"2024-09-21T10:52:15.400859+0000","last_unstale":"2024-09-21T10:52:28.586341+0000","last_undegraded":"2024-09-21T10:52:28.586341+0000","last_fullsized":"2024-09-21T10:52:28.586341+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:15.723031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2221,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.576677+0000","last_change":"2024-09-21T10:49:44.865830+0000","last_active":"2024-09-21T10:52:31.576677+0000","last_peered":"2024-09-21T10:52:31.576677+0000","last_clean":"2024-09-21T10:52:31.576677+0000","last_became_active":"2024-09-21T10:49:44.865499+0000","last_became_peered":"2024-09-21T10:49:44.865499+0000","last_unstale":"2024-09-21T10:52:31.576677+0000","last_undegraded":"2024-09-21T10:52:31.576677+0000","last_fullsized":"2024-09-21T10:52:31.576677+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:19:22.013154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":1449,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.470850+0000","last_change":"2024-09-21T10:52:25.489550+0000","last_active":"2024-09-21T10:52:28.470850+0000","last_peered":"2024-09-21T10:52:28.470850+0000","last_clean":"2024-09-21T10:52:28.470850+0000","last_became_active":"2024-09-21T10:52:25.489171+0000","last_became_peered":"2024-09-21T10:52:25.489171+0000","last_unstale":"2024-09-21T10:52:28.470850+0000","last_undegraded":"2024-09-21T10:52:28.470850+0000","last_fullsized":"2024-09-21T10:52:28.470850+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:21.105058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":783,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.562944+0000","last_change":"2024-09-21T10:52:28.601906+0000","last_active":"2024-09-21T10:52:31.562944+0000","last_peered":"2024-09-21T10:52:31.562944+0000","last_clean":"2024-09-21T10:52:31.562944+0000","last_became_active":"2024-09-21T10:52:28.587447+0000","last_became_peered":"2024-09-21T10:52:28.587447+0000","last_unstale":"2024-09-21T10:52:31.562944+0000","last_undegraded":"2024-09-21T10:52:31.562944+0000","last_fullsized":"2024-09-21T10:52:31.562944+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:44.223578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:44.223578+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":74,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"346'149","reported_seq":731,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.588387+0000","last_change":"2024-09-21T10:51:30.453979+0000","last_active":"2024-09-21T10:52:28.588387+0000","last_peered":"2024-09-21T10:52:28.588387+0000","last_clean":"2024-09-21T10:52:28.588387+0000","last_became_active":"2024-09-21T10:51:30.450893+0000","last_became_peered":"2024-09-21T10:51:30.450893+0000","last_unstale":"2024-09-21T10:52:28.588387+0000","last_undegraded":"2024-09-21T10:52:28.588387+0000","last_fullsized":"2024-09-21T10:52:28.588387+0000","mapping_epoch":1450,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1451,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:42.008235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:42.008235+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:21:48.570809+0000","scrub_duration":0.011129834999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2218,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.576913+0000","last_change":"2024-09-21T10:09:06.538940+0000","last_active":"2024-09-21T10:52:31.576913+0000","last_peered":"2024-09-21T10:52:31.576913+0000","last_clean":"2024-09-21T10:52:31.576913+0000","last_became_active":"2024-09-21T10:09:06.538668+0000","last_became_peered":"2024-09-21T10:09:06.538668+0000","last_unstale":"2024-09-21T10:52:31.576913+0000","last_undegraded":"2024-09-21T10:52:31.576913+0000","last_fullsized":"2024-09-21T10:52:31.576913+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:42:44.299488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2217,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.568608+0000","last_change":"2024-09-21T10:52:25.493032+0000","last_active":"2024-09-21T10:52:31.568608+0000","last_peered":"2024-09-21T10:52:31.568608+0000","last_clean":"2024-09-21T10:52:31.568608+0000","last_became_active":"2024-09-21T10:52:25.492182+0000","last_became_peered":"2024-09-21T10:52:25.492182+0000","last_unstale":"2024-09-21T10:52:31.568608+0000","last_undegraded":"2024-09-21T10:52:31.568608+0000","last_fullsized":"2024-09-21T10:52:31.568608+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:12:57.118420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1461,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.474365+0000","last_change":"2024-09-21T10:52:15.402809+0000","last_active":"2024-09-21T10:52:28.474365+0000","last_peered":"2024-09-21T10:52:28.474365+0000","last_clean":"2024-09-21T10:52:28.474365+0000","last_became_active":"2024-09-21T10:52:15.402497+0000","last_became_peered":"2024-09-21T10:52:15.402497+0000","last_unstale":"2024-09-21T10:52:28.474365+0000","last_undegraded":"2024-09-21T10:52:28.474365+0000","last_fullsized":"2024-09-21T10:52:28.474365+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:07:35.713534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1472,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.475188+0000","last_change":"2024-09-21T10:44:14.399842+0000","last_active":"2024-09-21T10:52:28.475188+0000","last_peered":"2024-09-21T10:52:28.475188+0000","last_clean":"2024-09-21T10:52:28.475188+0000","last_became_active":"2024-09-21T10:44:14.399426+0000","last_became_peered":"2024-09-21T10:44:14.399426+0000","last_unstale":"2024-09-21T10:52:28.475188+0000","last_undegraded":"2024-09-21T10:52:28.475188+0000","last_fullsized":"2024-09-21T10:52:28.475188+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:26:45.362101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"346'149","reported_seq":898,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.561754+0000","last_change":"2024-09-21T10:52:16.404597+0000","last_active":"2024-09-21T10:52:31.561754+0000","last_peered":"2024-09-21T10:52:31.561754+0000","last_clean":"2024-09-21T10:52:31.561754+0000","last_became_active":"2024-09-21T10:52:16.404194+0000","last_became_peered":"2024-09-21T10:52:16.404194+0000","last_unstale":"2024-09-21T10:52:31.561754+0000","last_undegraded":"2024-09-21T10:52:31.561754+0000","last_fullsized":"2024-09-21T10:52:31.561754+0000","mapping_epoch":1463,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1464,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":793725,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1467,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.474509+0000","last_change":"2024-09-21T10:40:13.432387+0000","last_active":"2024-09-21T10:52:28.474509+0000","last_peered":"2024-09-21T10:52:28.474509+0000","last_clean":"2024-09-21T10:52:28.474509+0000","last_became_active":"2024-09-21T10:40:13.432053+0000","last_became_peered":"2024-09-21T10:40:13.432053+0000","last_unstale":"2024-09-21T10:52:28.474509+0000","last_undegraded":"2024-09-21T10:52:28.474509+0000","last_fullsized":"2024-09-21T10:52:28.474509+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:37.486101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1445,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.588590+0000","last_change":"2024-09-21T10:07:56.020791+0000","last_active":"2024-09-21T10:52:28.588590+0000","last_peered":"2024-09-21T10:52:28.588590+0000","last_clean":"2024-09-21T10:52:28.588590+0000","last_became_active":"2024-09-21T10:07:56.020205+0000","last_became_peered":"2024-09-21T10:07:56.020205+0000","last_unstale":"2024-09-21T10:52:28.588590+0000","last_undegraded":"2024-09-21T10:52:28.588590+0000","last_fullsized":"2024-09-21T10:52:28.588590+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:09.813166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"346'149","reported_seq":908,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.470604+0000","last_change":"2024-09-21T10:49:47.904295+0000","last_active":"2024-09-21T10:52:28.470604+0000","last_peered":"2024-09-21T10:52:28.470604+0000","last_clean":"2024-09-21T10:52:28.470604+0000","last_became_active":"2024-09-21T10:49:47.903650+0000","last_became_peered":"2024-09-21T10:49:47.903650+0000","last_unstale":"2024-09-21T10:52:28.470604+0000","last_undegraded":"2024-09-21T10:52:28.470604+0000","last_fullsized":"2024-09-21T10:52:28.470604+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":74,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1443,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.474558+0000","last_change":"2024-09-21T10:09:06.533271+0000","last_active":"2024-09-21T10:52:28.474558+0000","last_peered":"2024-09-21T10:52:28.474558+0000","last_clean":"2024-09-21T10:52:28.474558+0000","last_became_active":"2024-09-21T10:09:06.532041+0000","last_became_peered":"2024-09-21T10:09:06.532041+0000","last_unstale":"2024-09-21T10:52:28.474558+0000","last_undegraded":"2024-09-21T10:52:28.474558+0000","last_fullsized":"2024-09-21T10:52:28.474558+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:04.236375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":2246,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.568216+0000","last_change":"2024-09-21T10:51:27.529295+0000","last_active":"2024-09-21T10:52:31.568216+0000","last_peered":"2024-09-21T10:52:31.568216+0000","last_clean":"2024-09-21T10:52:31.568216+0000","last_became_active":"2024-09-21T10:51:27.446359+0000","last_became_peered":"2024-09-21T10:51:27.446359+0000","last_unstale":"2024-09-21T10:52:31.568216+0000","last_undegraded":"2024-09-21T10:52:31.568216+0000","last_fullsized":"2024-09-21T10:52:31.568216+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:17:08.788558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":676,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.578060+0000","last_change":"2024-09-21T10:51:27.481103+0000","last_active":"2024-09-21T10:52:31.578060+0000","last_peered":"2024-09-21T10:52:31.578060+0000","last_clean":"2024-09-21T10:52:31.578060+0000","last_became_active":"2024-09-21T10:51:27.480763+0000","last_became_peered":"2024-09-21T10:51:27.480763+0000","last_unstale":"2024-09-21T10:52:31.578060+0000","last_undegraded":"2024-09-21T10:52:31.578060+0000","last_fullsized":"2024-09-21T10:52:31.578060+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:35.375303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":781,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.571759+0000","last_change":"2024-09-21T10:49:44.864819+0000","last_active":"2024-09-21T10:52:31.571759+0000","last_peered":"2024-09-21T10:52:31.571759+0000","last_clean":"2024-09-21T10:52:31.571759+0000","last_became_active":"2024-09-21T10:49:44.864061+0000","last_became_peered":"2024-09-21T10:49:44.864061+0000","last_unstale":"2024-09-21T10:52:31.571759+0000","last_undegraded":"2024-09-21T10:52:31.571759+0000","last_fullsized":"2024-09-21T10:52:31.571759+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:37.266967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":27,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.586745+0000","last_change":"2024-09-21T10:52:25.492570+0000","last_active":"2024-09-21T10:52:28.586745+0000","last_peered":"2024-09-21T10:52:28.586745+0000","last_clean":"2024-09-21T10:52:28.586745+0000","last_became_active":"2024-09-21T10:52:25.486382+0000","last_became_peered":"2024-09-21T10:52:25.486382+0000","last_unstale":"2024-09-21T10:52:28.586745+0000","last_undegraded":"2024-09-21T10:52:28.586745+0000","last_fullsized":"2024-09-21T10:52:28.586745+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:06.702707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":131,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.471308+0000","last_change":"2024-09-21T10:49:44.874522+0000","last_active":"2024-09-21T10:52:28.471308+0000","last_peered":"2024-09-21T10:52:28.471308+0000","last_clean":"2024-09-21T10:52:28.471308+0000","last_became_active":"2024-09-21T10:49:44.863242+0000","last_became_peered":"2024-09-21T10:49:44.863242+0000","last_unstale":"2024-09-21T10:52:28.471308+0000","last_undegraded":"2024-09-21T10:52:28.471308+0000","last_fullsized":"2024-09-21T10:52:28.471308+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:09:02.066837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1477,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.475084+0000","last_change":"2024-09-21T10:52:03.315267+0000","last_active":"2024-09-21T10:52:28.475084+0000","last_peered":"2024-09-21T10:52:28.475084+0000","last_clean":"2024-09-21T10:52:28.475084+0000","last_became_active":"2024-09-21T10:52:03.314474+0000","last_became_peered":"2024-09-21T10:52:03.314474+0000","last_unstale":"2024-09-21T10:52:28.475084+0000","last_undegraded":"2024-09-21T10:52:28.475084+0000","last_fullsized":"2024-09-21T10:52:28.475084+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:00:57.791341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":19,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.471170+0000","last_change":"2024-09-21T10:52:25.518721+0000","last_active":"2024-09-21T10:52:28.471170+0000","last_peered":"2024-09-21T10:52:28.471170+0000","last_clean":"2024-09-21T10:52:28.471170+0000","last_became_active":"2024-09-21T10:52:25.472114+0000","last_became_peered":"2024-09-21T10:52:25.472114+0000","last_unstale":"2024-09-21T10:52:28.471170+0000","last_undegraded":"2024-09-21T10:52:28.471170+0000","last_fullsized":"2024-09-21T10:52:28.471170+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:33:48.471147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.470430+0000","last_change":"2024-09-21T10:52:25.494490+0000","last_active":"2024-09-21T10:52:28.470430+0000","last_peered":"2024-09-21T10:52:28.470430+0000","last_clean":"2024-09-21T10:52:28.470430+0000","last_became_active":"2024-09-21T10:52:25.494200+0000","last_became_peered":"2024-09-21T10:52:25.494200+0000","last_unstale":"2024-09-21T10:52:28.470430+0000","last_undegraded":"2024-09-21T10:52:28.470430+0000","last_fullsized":"2024-09-21T10:52:28.470430+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:15:21.030031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":19,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.471621+0000","last_change":"2024-09-21T10:52:25.479229+0000","last_active":"2024-09-21T10:52:28.471621+0000","last_peered":"2024-09-21T10:52:28.471621+0000","last_clean":"2024-09-21T10:52:28.471621+0000","last_became_active":"2024-09-21T10:52:25.470551+0000","last_became_peered":"2024-09-21T10:52:25.470551+0000","last_unstale":"2024-09-21T10:52:28.471621+0000","last_undegraded":"2024-09-21T10:52:28.471621+0000","last_fullsized":"2024-09-21T10:52:28.471621+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:15.458231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1489,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.586223+0000","last_change":"2024-09-21T10:51:27.517234+0000","last_active":"2024-09-21T10:52:28.586223+0000","last_peered":"2024-09-21T10:52:28.586223+0000","last_clean":"2024-09-21T10:52:28.586223+0000","last_became_active":"2024-09-21T10:51:27.431253+0000","last_became_peered":"2024-09-21T10:51:27.431253+0000","last_unstale":"2024-09-21T10:52:28.586223+0000","last_undegraded":"2024-09-21T10:52:28.586223+0000","last_fullsized":"2024-09-21T10:52:28.586223+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:35:42.249305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2201,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.568220+0000","last_change":"2024-09-21T10:52:25.497298+0000","last_active":"2024-09-21T10:52:31.568220+0000","last_peered":"2024-09-21T10:52:31.568220+0000","last_clean":"2024-09-21T10:52:31.568220+0000","last_became_active":"2024-09-21T10:52:25.496876+0000","last_became_peered":"2024-09-21T10:52:25.496876+0000","last_unstale":"2024-09-21T10:52:31.568220+0000","last_undegraded":"2024-09-21T10:52:31.568220+0000","last_fullsized":"2024-09-21T10:52:31.568220+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:16.587897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":114,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.470882+0000","last_change":"2024-09-21T10:44:14.398878+0000","last_active":"2024-09-21T10:52:28.470882+0000","last_peered":"2024-09-21T10:52:28.470882+0000","last_clean":"2024-09-21T10:52:28.470882+0000","last_became_active":"2024-09-21T10:44:14.398547+0000","last_became_peered":"2024-09-21T10:44:14.398547+0000","last_unstale":"2024-09-21T10:52:28.470882+0000","last_undegraded":"2024-09-21T10:52:28.470882+0000","last_fullsized":"2024-09-21T10:52:28.470882+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:13.008134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2249,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.569120+0000","last_change":"2024-09-21T10:52:25.498554+0000","last_active":"2024-09-21T10:52:31.569120+0000","last_peered":"2024-09-21T10:52:31.569120+0000","last_clean":"2024-09-21T10:52:31.569120+0000","last_became_active":"2024-09-21T10:52:25.498194+0000","last_became_peered":"2024-09-21T10:52:25.498194+0000","last_unstale":"2024-09-21T10:52:31.569120+0000","last_undegraded":"2024-09-21T10:52:31.569120+0000","last_fullsized":"2024-09-21T10:52:31.569120+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:47.115271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.470912+0000","last_change":"2024-09-21T10:52:25.489063+0000","last_active":"2024-09-21T10:52:28.470912+0000","last_peered":"2024-09-21T10:52:28.470912+0000","last_clean":"2024-09-21T10:52:28.470912+0000","last_became_active":"2024-09-21T10:52:25.488129+0000","last_became_peered":"2024-09-21T10:52:25.488129+0000","last_unstale":"2024-09-21T10:52:28.470912+0000","last_undegraded":"2024-09-21T10:52:28.470912+0000","last_fullsized":"2024-09-21T10:52:28.470912+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:36.996829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1429,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.471225+0000","last_change":"2024-09-21T10:49:44.864898+0000","last_active":"2024-09-21T10:52:28.471225+0000","last_peered":"2024-09-21T10:52:28.471225+0000","last_clean":"2024-09-21T10:52:28.471225+0000","last_became_active":"2024-09-21T10:49:44.862691+0000","last_became_peered":"2024-09-21T10:49:44.862691+0000","last_unstale":"2024-09-21T10:52:28.471225+0000","last_undegraded":"2024-09-21T10:52:28.471225+0000","last_fullsized":"2024-09-21T10:52:28.471225+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:57.809113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":120,"reported_epoch":1469,"state":"active+clean","last_fresh":"2024-09-21T10:52:28.471042+0000","last_change":"2024-09-21T10:44:14.418113+0000","last_active":"2024-09-21T10:52:28.471042+0000","last_peered":"2024-09-21T10:52:28.471042+0000","last_clean":"2024-09-21T10:52:28.471042+0000","last_became_active":"2024-09-21T10:44:14.398327+0000","last_became_peered":"2024-09-21T10:44:14.398327+0000","last_unstale":"2024-09-21T10:52:28.471042+0000","last_undegraded":"2024-09-21T10:52:28.471042+0000","last_fullsized":"2024-09-21T10:52:28.471042+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:08.810863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2230,"reported_epoch":1470,"state":"active+clean","last_fresh":"2024-09-21T10:52:31.571906+0000","last_change":"2024-09-21T10:44:14.398584+0000","last_active":"2024-09-21T10:52:31.571906+0000","last_peered":"2024-09-21T10:52:31.571906+0000","last_clean":"2024-09-21T10:52:31.571906+0000","last_became_active":"2024-09-21T10:44:14.398278+0000","last_became_peered":"2024-09-21T10:44:14.398278+0000","last_unstale":"2024-09-21T10:52:31.571906+0000","last_undegraded":"2024-09-21T10:52:31.571906+0000","last_fullsized":"2024-09-21T10:52:31.571906+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:19.681750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":55,"num_pg":16,"stat_sum":{"num_bytes":3251372056,"num_objects":49613,"num_object_clones":0,"num_object_copies":148839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31383,"num_objects_misplaced":11170,"num_objects_unfound":0,"num_objects_dirty":49613,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161065,"num_write_kb":6741633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27217,"num_bytes_recovered":1764098048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12831174656,"data_stored":12831162440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71024,"ondisk_log_size":71024,"up":48,"acting":43,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":429359,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":276,"num_read_kb":598,"num_write":157,"num_write_kb":1757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3655209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":32,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":95,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":11344,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":18188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234702,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4130896,"kb_used_data":3630596,"kb_used_omap":1,"kb_used_meta":500286,"kb_avail":89614256,"statfs":{"total":95995035648,"available":91764998144,"internally_reserved":0,"allocated":3717730304,"data_stored":3712757446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":512293323},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496340,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2613844,"kb_used_data":1955212,"kb_used_omap":1,"kb_used_meta":658622,"kb_avail":91131308,"statfs":{"total":95995035648,"available":93318459392,"internally_reserved":0,"allocated":2002137088,"data_stored":1997157161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":674429386},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757977,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019613,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1694680,"kb_used_data":1068048,"kb_used_omap":1,"kb_used_meta":626622,"kb_avail":92050472,"statfs":{"total":95995035648,"available":94259683328,"internally_reserved":0,"allocated":1093681152,"data_stored":1088697105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":641661382},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369356,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2901256,"kb_used_data":2174976,"kb_used_omap":69,"kb_used_meta":726202,"kb_avail":90843896,"statfs":{"total":95995035648,"available":93024149504,"internally_reserved":0,"allocated":2227175424,"data_stored":2222203031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":743630978},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2920577761811,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807958042,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2032060,"kb_used_data":1364212,"kb_used_omap":66,"kb_used_meta":667773,"kb_avail":91713092,"statfs":{"total":95995035648,"available":93914206208,"internally_reserved":0,"allocated":1396953088,"data_stored":1391991785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68589,"internal_metadata":683799571},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":2860448219681,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":936185856,"data_stored":936181784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1383399424,"data_stored":1383399424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2199261184,"data_stored":2199257112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1079250944,"data_stored":1079246872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1540947968,"data_stored":1540947968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1986789376,"data_stored":1986789376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3705339904,"data_stored":3705339904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:52:42.382 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:52:42.387 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:42.473 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:42.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:42.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:42.575 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:42.609 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:42.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:42.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:42.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:42 smithi070 ceph-mon[107385]: pgmap v2684: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 549 op/s; 31384/146586 objects degraded (21.410%); 10977/146586 objects misplaced (7.488%); 21 MiB/s, 341 objects/s recovering 2024-09-21T10:52:42.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:42 smithi070 ceph-mon[112139]: pgmap v2684: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 549 op/s; 31384/146586 objects degraded (21.410%); 10977/146586 objects misplaced (7.488%); 21 MiB/s, 341 objects/s recovering 2024-09-21T10:52:42.675 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:42 smithi160 ceph-mon[101879]: pgmap v2684: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 549 op/s; 31384/146586 objects degraded (21.410%); 10977/146586 objects misplaced (7.488%); 21 MiB/s, 341 objects/s recovering 2024-09-21T10:52:42.712 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:42.748 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:42.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:42.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:42.850 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:42.890 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:42.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:42.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:42.992 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:43.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:43.027 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:43.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:43.115 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:52:43.115 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:52:43.127 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:43.155 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:43.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:43.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:43.257 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:43.258 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:43.258 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:43.286 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:43.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:43.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:43.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:52:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:52:43] "GET /metrics HTTP/1.1" 200 36637 "" "Prometheus/2.51.0" 2024-09-21T10:52:43.388 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:43.416 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:43.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:43.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:43.518 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:43.553 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:43.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:43.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:43.654 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:43.671 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:43 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 31384/146586 objects degraded (21.410%), 11 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:52:43.671 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:43 smithi070 ceph-mon[107385]: osdmap e1471: 8 total, 8 up, 5 in 2024-09-21T10:52:43.671 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:43 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 31384/146586 objects degraded (21.410%), 11 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:52:43.671 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:43 smithi070 ceph-mon[112139]: osdmap e1471: 8 total, 8 up, 5 in 2024-09-21T10:52:43.694 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:43.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:43.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:43.796 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:43.826 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:43 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 31384/146586 objects degraded (21.410%), 11 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:52:43.826 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:43 smithi160 ceph-mon[101879]: osdmap e1471: 8 total, 8 up, 5 in 2024-09-21T10:52:43.831 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:43.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:43.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:43.853 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:52:43.933 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:43.957 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:43.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:43.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:44.059 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:44.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:44.081 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:44.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:44.182 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:44.207 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:44.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:44.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:44.309 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:44.318 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T10:52:44.334 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:44.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:44.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:44.436 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:44.477 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:44.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:44.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:44.566 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:44 smithi070 ceph-mon[112139]: pgmap v2685: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.8 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 625 op/s; 29842/138147 objects degraded (21.602%); 10140/138147 objects misplaced (7.340%); 22 MiB/s, 357 objects/s recovering 2024-09-21T10:52:44.567 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:44 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2387743244' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:52:44.567 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:44 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:52:44.567 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:44 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2387743244' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:52:44.567 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:44 smithi070 ceph-mon[112139]: osdmap e1472: 8 total, 8 up, 5 in 2024-09-21T10:52:44.567 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:44 smithi070 ceph-mon[107385]: pgmap v2685: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.8 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 625 op/s; 29842/138147 objects degraded (21.602%); 10140/138147 objects misplaced (7.340%); 22 MiB/s, 357 objects/s recovering 2024-09-21T10:52:44.567 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:44 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2387743244' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:52:44.567 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:44 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:52:44.567 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:44 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2387743244' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:52:44.568 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:44 smithi070 ceph-mon[107385]: osdmap e1472: 8 total, 8 up, 5 in 2024-09-21T10:52:44.579 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:44.611 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:44.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:44.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:44.712 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:44.758 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:44.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:44.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:44 smithi160 ceph-mon[101879]: pgmap v2685: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.8 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 625 op/s; 29842/138147 objects degraded (21.602%); 10140/138147 objects misplaced (7.340%); 22 MiB/s, 357 objects/s recovering 2024-09-21T10:52:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:44 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2387743244' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:52:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:44 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:52:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:44 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2387743244' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:52:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:44 smithi160 ceph-mon[101879]: osdmap e1472: 8 total, 8 up, 5 in 2024-09-21T10:52:44.860 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:44.892 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:44.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:44.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:44.994 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:45.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:45.046 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:45.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:45.147 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:45.174 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:45.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:45.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:45.276 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:45.306 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:45.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:45.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:45.408 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:45.434 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:45.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:45.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:45.535 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:45.569 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:45.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:45.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:45.670 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:45.705 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:45.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:45.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:45.807 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:45.834 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:45.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:45.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:45.936 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:45.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:45.982 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:45.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:46.083 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:46.115 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:46.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:46.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:46.216 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:46.258 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:46.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:46.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:46.359 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:46.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:46.394 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:46.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:46.495 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:46.533 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:46.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:46.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:46.635 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:46.669 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:46.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:46.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:46.771 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:46.780 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:46 smithi070 ceph-mon[107385]: pgmap v2688: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.8 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 448 op/s; 29842/138147 objects degraded (21.602%); 10140/138147 objects misplaced (7.340%); 16 MiB/s, 260 objects/s recovering 2024-09-21T10:52:46.780 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:46 smithi070 ceph-mon[112139]: pgmap v2688: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.8 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 448 op/s; 29842/138147 objects degraded (21.602%); 10140/138147 objects misplaced (7.340%); 16 MiB/s, 260 objects/s recovering 2024-09-21T10:52:46.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:46.808 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:46.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:46.825 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:46 smithi160 ceph-mon[101879]: pgmap v2688: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.8 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 448 op/s; 29842/138147 objects degraded (21.602%); 10140/138147 objects misplaced (7.340%); 16 MiB/s, 260 objects/s recovering 2024-09-21T10:52:46.911 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:46.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:46.958 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:46.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:47.059 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:47.091 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:47.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:47.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:47.193 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:47.241 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:47.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:47.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:47.343 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:47.376 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:47.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:47.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:47.477 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:47.517 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:47.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:47.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:47.534 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:47.619 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:47.658 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:47.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:47.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:47.760 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:47.795 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:47.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:47.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:47.896 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:47.909 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:52:47.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:47.930 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:47.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:48.031 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:48.056 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:48.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:48.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:48.158 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:48.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:48.216 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:48.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:48.317 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:48.347 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:48.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:48.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:48.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:48 smithi070 ceph-mon[107385]: pgmap v2689: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 614 op/s; 28120/131835 objects degraded (21.330%); 9908/131835 objects misplaced (7.515%); 25 MiB/s, 414 objects/s recovering 2024-09-21T10:52:48.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:48 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:52:48.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:48 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:52:48.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:48 smithi070 ceph-mon[112139]: pgmap v2689: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 614 op/s; 28120/131835 objects degraded (21.330%); 9908/131835 objects misplaced (7.515%); 25 MiB/s, 414 objects/s recovering 2024-09-21T10:52:48.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:48 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:52:48.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:48 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:52:48.449 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:48.454 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:52:48.454 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:52:48.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:48.489 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:48.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:48.493 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:52:48.493 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:52:48.524 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:52:48.524 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:52:48.588 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:52:48.588 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:52:48.590 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:48 smithi160 ceph-mon[101879]: pgmap v2689: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 614 op/s; 28120/131835 objects degraded (21.330%); 9908/131835 objects misplaced (7.515%); 25 MiB/s, 414 objects/s recovering 2024-09-21T10:52:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:48 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:52:48.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:48 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:52:48.615 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:52:48.615 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:52:48.618 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:48.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:48.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:48.647 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:52:48.647 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:52:48.678 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:52:48.679 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:52:48.711 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:52:48.712 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:52:48.720 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:48.762 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:48.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:48.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:48.912 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:48.943 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:48.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:48.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:49.045 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:49.080 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:49.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:49.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:49.182 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:49.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:49.203 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:49.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:49.304 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:49.340 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:49.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:49.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:49.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:49 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 28120/131835 objects degraded (21.330%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:52:49.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:49 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/805299739' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:52:49.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:49 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 28120/131835 objects degraded (21.330%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:52:49.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:49 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/805299739' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:52:49.442 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:49.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:49.483 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:49.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:49.584 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:49.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:49 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 28120/131835 objects degraded (21.330%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:52:49.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:49 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/805299739' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:52:49.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:49.608 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:49.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:49.709 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:49.743 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:49.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:49.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:49.845 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:49.878 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:49.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:49.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:49.979 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:50.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:50.015 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:50.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:50.116 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:50.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:50.151 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:50.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:50.252 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:50.283 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:50.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:50.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:50.354 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:52:50.357 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T10:52:50.384 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:50.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:50.413 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:50.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:50.514 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:50.535 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:50.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:50.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:50 smithi160 ceph-mon[101879]: pgmap v2690: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 704 op/s; 28120/129669 objects degraded (21.686%); 9649/129669 objects misplaced (7.441%); 27 MiB/s, 446 objects/s recovering 2024-09-21T10:52:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:50 smithi160 ceph-mon[101879]: from='client.45845 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:52:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:50 smithi070 ceph-mon[107385]: pgmap v2690: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 704 op/s; 28120/129669 objects degraded (21.686%); 9649/129669 objects misplaced (7.441%); 27 MiB/s, 446 objects/s recovering 2024-09-21T10:52:50.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:50 smithi070 ceph-mon[107385]: from='client.45845 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:52:50.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:50 smithi070 ceph-mon[112139]: pgmap v2690: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 704 op/s; 28120/129669 objects degraded (21.686%); 9649/129669 objects misplaced (7.441%); 27 MiB/s, 446 objects/s recovering 2024-09-21T10:52:50.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:50 smithi070 ceph-mon[112139]: from='client.45845 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:52:50.637 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:50.665 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:50.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:50.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:50.767 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:50.798 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:50.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:50.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:50.900 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:50.934 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:50.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:50.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:51.035 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:51.069 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:51.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:51.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:51.171 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:51.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:51.216 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:51.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:51.317 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:51.354 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:51.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:51.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:51.456 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:51.458 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:52:51.458 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:52:51.490 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:51.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:51.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:51.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:52:51.592 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:51.627 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:51.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:51.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:51.729 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:51.773 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:51.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:51.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:51.875 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:51.913 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:51.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:51.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:52.015 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:52.054 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:52.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:52.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:52.156 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:52.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:52.192 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:52.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:52.293 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:52.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:52.334 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:52.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:52.435 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:52.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:52.470 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:52.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:52.571 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:52.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:52 smithi160 ceph-mon[101879]: pgmap v2691: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 353 op/s; 28120/129669 objects degraded (21.686%); 9649/129669 objects misplaced (7.441%); 12 MiB/s, 209 objects/s recovering 2024-09-21T10:52:52.599 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:52.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:52.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:52.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:52 smithi070 ceph-mon[107385]: pgmap v2691: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 353 op/s; 28120/129669 objects degraded (21.686%); 9649/129669 objects misplaced (7.441%); 12 MiB/s, 209 objects/s recovering 2024-09-21T10:52:52.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:52 smithi070 ceph-mon[112139]: pgmap v2691: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 353 op/s; 28120/129669 objects degraded (21.686%); 9649/129669 objects misplaced (7.441%); 12 MiB/s, 209 objects/s recovering 2024-09-21T10:52:52.701 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:52.734 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:52.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:52.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:52.836 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:52.854 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":2690,"stamp":"2024-09-21T10:52:48.917742+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2831328012,"num_objects":43223,"num_object_clones":0,"num_object_copies":129669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28120,"num_objects_misplaced":9649,"num_objects_unfound":0,"num_objects_dirty":43223,"num_whiteouts":0,"num_read":277,"num_read_kb":599,"num_write":167702,"num_write_kb":6743423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31048,"num_bytes_recovered":2004021938,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75338,"ondisk_log_size":75338,"up":297,"acting":292,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":328,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":12531556,"kb_used_data":9325492,"kb_used_omap":141,"kb_used_meta":3205746,"kb_avail":456194204,"statfs":{"total":479975178240,"available":467142864896,"internally_reserved":0,"allocated":9549303808,"data_stored":9524360215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":144912,"internal_metadata":3282684400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-369557504,"num_objects":-5639,"num_object_clones":0,"num_object_copies":-16917,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3264,"num_objects_misplaced":-1328,"num_objects_unfound":0,"num_objects_dirty":-5639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5639,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3573,"num_bytes_recovered":223608832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002649"},"pg_stats":[{"pgid":"55.f","version":"1472'10472","reported_seq":16072,"reported_epoch":1472,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T10:52:46.708374+0000","last_change":"2024-09-21T10:52:25.539794+0000","last_active":"2024-09-21T10:52:46.708374+0000","last_peered":"2024-09-21T10:52:46.708374+0000","last_clean":"2024-09-21T10:49:43.822243+0000","last_became_active":"2024-09-21T10:51:28.555652+0000","last_became_peered":"2024-09-21T10:51:28.555652+0000","last_unstale":"2024-09-21T10:52:46.708374+0000","last_undegraded":"2024-09-21T10:52:25.537884+0000","last_fullsized":"2024-09-21T10:52:46.708374+0000","mapping_epoch":1448,"log_start":"1442'6340","ondisk_log_start":"1442'6340","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4132,"log_dups_size":0,"ondisk_log_size":4132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:30:34.790715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174325760,"num_objects":2660,"num_object_clones":0,"num_object_copies":7980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1992,"num_objects_misplaced":668,"num_objects_unfound":0,"num_objects_dirty":2660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10472,"num_write_kb":420224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2753,"num_bytes_recovered":180420608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,0],"avail_no_missing":["6","5","7"],"object_location_counts":[{"shards":"0,6,7","objects":1821},{"shards":"5,6,7","objects":839}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.e","version":"1472'10370","reported_seq":10513,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:46.371498+0000","last_change":"2024-09-21T10:52:16.415783+0000","last_active":"2024-09-21T10:52:46.371498+0000","last_peered":"2024-09-21T10:52:46.371498+0000","last_clean":"2024-09-21T10:52:46.371498+0000","last_became_active":"2024-09-21T10:52:16.397921+0000","last_became_peered":"2024-09-21T10:52:16.397921+0000","last_unstale":"2024-09-21T10:52:46.371498+0000","last_undegraded":"2024-09-21T10:52:46.371498+0000","last_fullsized":"2024-09-21T10:52:46.371498+0000","mapping_epoch":1463,"log_start":"1433'5656","ondisk_log_start":"1433'5656","created":1411,"last_epoch_clean":1464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4714,"log_dups_size":0,"ondisk_log_size":4714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:13.753306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173146112,"num_objects":2642,"num_object_clones":0,"num_object_copies":7926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10370,"num_write_kb":416384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.d","version":"1472'10697","reported_seq":15452,"reported_epoch":1472,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:52:46.274272+0000","last_change":"2024-09-21T10:52:26.803434+0000","last_active":"2024-09-21T10:52:46.274272+0000","last_peered":"2024-09-21T10:52:46.274272+0000","last_clean":"2024-09-21T10:50:20.259057+0000","last_became_active":"2024-09-21T10:52:26.653674+0000","last_became_peered":"2024-09-21T10:52:26.653674+0000","last_unstale":"2024-09-21T10:52:46.274272+0000","last_undegraded":"2024-09-21T10:52:26.638016+0000","last_fullsized":"2024-09-21T10:52:26.462648+0000","mapping_epoch":1466,"log_start":"1436'5826","ondisk_log_start":"1436'5826","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4871,"log_dups_size":0,"ondisk_log_size":4871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:31:18.206348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175636504,"num_objects":2681,"num_object_clones":0,"num_object_copies":8043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2681,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10697,"num_write_kb":428033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2331,"num_bytes_recovered":152436736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[4,3],"avail_no_missing":["4","0","6"],"object_location_counts":[{"shards":"0,3,4","objects":268},{"shards":"0,4,7","objects":2412},{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":4,"purged_snaps":[]},{"pgid":"55.c","version":"1472'10400","reported_seq":15038,"reported_epoch":1472,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T10:52:46.321171+0000","last_change":"2024-09-21T10:52:25.708024+0000","last_active":"2024-09-21T10:52:46.321171+0000","last_peered":"2024-09-21T10:52:46.321171+0000","last_clean":"2024-09-21T10:50:03.007234+0000","last_became_active":"2024-09-21T10:52:25.621852+0000","last_became_peered":"2024-09-21T10:52:25.621852+0000","last_unstale":"2024-09-21T10:52:46.321171+0000","last_undegraded":"2024-09-21T10:52:25.613400+0000","last_fullsized":"2024-09-21T10:52:46.321171+0000","mapping_epoch":1465,"log_start":"1437'5786","ondisk_log_start":"1437'5786","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4614,"log_dups_size":0,"ondisk_log_size":4614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179044352,"num_objects":2732,"num_object_clones":0,"num_object_copies":8196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2732,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10400,"num_write_kb":420224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2246,"num_bytes_recovered":147128320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,0],"avail_no_missing":["3","1","4"],"object_location_counts":[{"shards":"0,3,4","objects":881},{"shards":"3,4","objects":1851}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.b","version":"1472'10657","reported_seq":10728,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:46.721081+0000","last_change":"2024-09-21T10:48:52.934001+0000","last_active":"2024-09-21T10:52:46.721081+0000","last_peered":"2024-09-21T10:52:46.721081+0000","last_clean":"2024-09-21T10:52:46.721081+0000","last_became_active":"2024-09-21T10:48:52.933674+0000","last_became_peered":"2024-09-21T10:48:52.933674+0000","last_unstale":"2024-09-21T10:52:46.721081+0000","last_undegraded":"2024-09-21T10:52:46.721081+0000","last_fullsized":"2024-09-21T10:52:46.721081+0000","mapping_epoch":1411,"log_start":"1443'6506","ondisk_log_start":"1443'6506","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4151,"log_dups_size":0,"ondisk_log_size":4151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:13.178680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181075968,"num_objects":2763,"num_object_clones":0,"num_object_copies":8289,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10657,"num_write_kb":429440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.a","version":"1472'10422","reported_seq":11877,"reported_epoch":1472,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T10:52:46.381618+0000","last_change":"2024-09-21T10:52:26.731569+0000","last_active":"2024-09-21T10:52:46.381618+0000","last_peered":"2024-09-21T10:52:46.381618+0000","last_clean":"2024-09-21T10:50:03.021536+0000","last_became_active":"2024-09-21T10:52:26.699954+0000","last_became_peered":"2024-09-21T10:52:26.699954+0000","last_unstale":"2024-09-21T10:52:46.381618+0000","last_undegraded":"2024-09-21T10:52:26.664328+0000","last_fullsized":"2024-09-21T10:52:46.381618+0000","mapping_epoch":1466,"log_start":"1436'5751","ondisk_log_start":"1436'5751","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4671,"log_dups_size":0,"ondisk_log_size":4671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177733632,"num_objects":2712,"num_object_clones":0,"num_object_copies":8136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2712,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10422,"num_write_kb":420288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":655,"num_bytes_recovered":42926080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,4,6],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1848},{"shards":"3,4,6","objects":864}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.9","version":"1472'10443","reported_seq":19998,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:46.387331+0000","last_change":"2024-09-21T10:52:44.354764+0000","last_active":"2024-09-21T10:52:46.387331+0000","last_peered":"2024-09-21T10:52:46.387331+0000","last_clean":"2024-09-21T10:52:46.387331+0000","last_became_active":"2024-09-21T10:52:44.325813+0000","last_became_peered":"2024-09-21T10:52:44.325813+0000","last_unstale":"2024-09-21T10:52:46.387331+0000","last_undegraded":"2024-09-21T10:52:46.387331+0000","last_fullsized":"2024-09-21T10:52:46.387331+0000","mapping_epoch":1471,"log_start":"1437'5769","ondisk_log_start":"1437'5769","created":1411,"last_epoch_clean":1472,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4674,"log_dups_size":0,"ondisk_log_size":4674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:58.318369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178978816,"num_objects":2731,"num_object_clones":0,"num_object_copies":8193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10443,"num_write_kb":421568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4701,"num_bytes_recovered":297533440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.8","version":"1472'10328","reported_seq":18099,"reported_epoch":1472,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T10:52:46.629503+0000","last_change":"2024-09-21T10:52:25.496588+0000","last_active":"2024-09-21T10:52:46.629503+0000","last_peered":"2024-09-21T10:52:46.629503+0000","last_clean":"2024-09-21T10:49:43.813262+0000","last_became_active":"2024-09-21T10:51:28.546990+0000","last_became_peered":"2024-09-21T10:51:28.546990+0000","last_unstale":"2024-09-21T10:52:46.629503+0000","last_undegraded":"2024-09-21T10:52:25.490859+0000","last_fullsized":"2024-09-21T10:52:46.629503+0000","mapping_epoch":1448,"log_start":"1441'6234","ondisk_log_start":"1441'6234","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4094,"log_dups_size":0,"ondisk_log_size":4094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:51:45.867981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177340416,"num_objects":2706,"num_object_clones":0,"num_object_copies":8118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":813,"num_objects_misplaced":2706,"num_objects_unfound":0,"num_objects_dirty":2706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10328,"num_write_kb":417088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3825,"num_bytes_recovered":250675200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,0],"avail_no_missing":["6","0","5"],"object_location_counts":[{"shards":"0,3,6","objects":1893},{"shards":"0,5,6","objects":813}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.7","version":"1472'10600","reported_seq":10671,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:46.930565+0000","last_change":"2024-09-21T10:48:52.933230+0000","last_active":"2024-09-21T10:52:46.930565+0000","last_peered":"2024-09-21T10:52:46.930565+0000","last_clean":"2024-09-21T10:52:46.930565+0000","last_became_active":"2024-09-21T10:48:52.932944+0000","last_became_peered":"2024-09-21T10:48:52.932944+0000","last_unstale":"2024-09-21T10:52:46.930565+0000","last_undegraded":"2024-09-21T10:52:46.930565+0000","last_fullsized":"2024-09-21T10:52:46.930565+0000","mapping_epoch":1411,"log_start":"1432'5528","ondisk_log_start":"1432'5528","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":5072,"log_dups_size":0,"ondisk_log_size":5072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:16:43.711173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176160768,"num_objects":2688,"num_object_clones":0,"num_object_copies":8064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10600,"num_write_kb":425216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"55.6","version":"1472'10651","reported_seq":21409,"reported_epoch":1472,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-21T10:52:46.938199+0000","last_change":"2024-09-21T10:52:25.578778+0000","last_active":"2024-09-21T10:52:46.938199+0000","last_peered":"2024-09-21T10:52:46.938199+0000","last_clean":"2024-09-21T10:49:43.854914+0000","last_became_active":"2024-09-21T10:52:03.460583+0000","last_became_peered":"2024-09-21T10:52:03.460583+0000","last_unstale":"2024-09-21T10:52:46.938199+0000","last_undegraded":"2024-09-21T10:52:46.938199+0000","last_fullsized":"2024-09-21T10:52:03.308381+0000","mapping_epoch":1455,"log_start":"1432'5626","ondisk_log_start":"1432'5626","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":5025,"log_dups_size":0,"ondisk_log_size":5025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:40.566532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175177728,"num_objects":2673,"num_object_clones":0,"num_object_copies":8019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":908,"num_objects_unfound":0,"num_objects_dirty":2673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10651,"num_write_kb":426368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4208,"num_bytes_recovered":275316736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,1],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"1,4,7","objects":272},{"shards":"1,7","objects":1765},{"shards":"3,5,7","objects":636}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"55.5","version":"1472'10461","reported_seq":10582,"reported_epoch":1472,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-21T10:52:46.650081+0000","last_change":"2024-09-21T10:52:25.760897+0000","last_active":"2024-09-21T10:52:46.650081+0000","last_peered":"2024-09-21T10:52:46.650081+0000","last_clean":"2024-09-21T10:50:02.995767+0000","last_became_active":"2024-09-21T10:52:25.564312+0000","last_became_peered":"2024-09-21T10:52:25.564312+0000","last_unstale":"2024-09-21T10:52:46.650081+0000","last_undegraded":"2024-09-21T10:52:46.650081+0000","last_fullsized":"2024-09-21T10:52:46.650081+0000","mapping_epoch":1465,"log_start":"1442'6314","ondisk_log_start":"1442'6314","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4147,"log_dups_size":0,"ondisk_log_size":4147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:26:53.597959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175439872,"num_objects":2677,"num_object_clones":0,"num_object_copies":8031,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2677,"num_objects_unfound":0,"num_objects_dirty":2677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10461,"num_write_kb":420416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,0],"avail_no_missing":["6","0","1","3","4"],"object_location_counts":[{"shards":"0,1,6","objects":2677}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.4","version":"1472'10587","reported_seq":10658,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:46.799943+0000","last_change":"2024-09-21T10:48:52.934634+0000","last_active":"2024-09-21T10:52:46.799943+0000","last_peered":"2024-09-21T10:52:46.799943+0000","last_clean":"2024-09-21T10:52:46.799943+0000","last_became_active":"2024-09-21T10:48:52.934368+0000","last_became_peered":"2024-09-21T10:48:52.934368+0000","last_unstale":"2024-09-21T10:52:46.799943+0000","last_undegraded":"2024-09-21T10:52:46.799943+0000","last_fullsized":"2024-09-21T10:52:46.799943+0000","mapping_epoch":1411,"log_start":"1432'5545","ondisk_log_start":"1432'5545","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":5042,"log_dups_size":0,"ondisk_log_size":5042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:27:09.216667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179109888,"num_objects":2733,"num_object_clones":0,"num_object_copies":8199,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10587,"num_write_kb":426240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"55.3","version":"1472'10059","reported_seq":21280,"reported_epoch":1472,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:52:46.044631+0000","last_change":"2024-09-21T10:52:26.628130+0000","last_active":"2024-09-21T10:52:46.044631+0000","last_peered":"2024-09-21T10:52:46.044631+0000","last_clean":"2024-09-21T10:49:43.839828+0000","last_became_active":"2024-09-21T10:52:26.596220+0000","last_became_peered":"2024-09-21T10:52:26.596220+0000","last_unstale":"2024-09-21T10:52:46.044631+0000","last_undegraded":"2024-09-21T10:52:26.536372+0000","last_fullsized":"2024-09-21T10:52:26.462998+0000","mapping_epoch":1466,"log_start":"1437'5743","ondisk_log_start":"1437'5743","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4316,"log_dups_size":0,"ondisk_log_size":4316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175833088,"num_objects":2683,"num_object_clones":0,"num_object_copies":8049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2755,"num_objects_misplaced":831,"num_objects_unfound":0,"num_objects_dirty":2683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10059,"num_write_kb":407744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3692,"num_bytes_recovered":241827840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,0],"avail_no_missing":["1","5","6"],"object_location_counts":[{"shards":"0,1","objects":831},{"shards":"1,3","objects":949},{"shards":"1,5","objects":903}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"55.2","version":"1472'10270","reported_seq":20946,"reported_epoch":1472,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T10:52:46.025236+0000","last_change":"2024-09-21T10:52:26.812464+0000","last_active":"2024-09-21T10:52:46.025236+0000","last_peered":"2024-09-21T10:52:46.025236+0000","last_clean":"2024-09-21T10:50:02.974304+0000","last_became_active":"2024-09-21T10:52:26.694338+0000","last_became_peered":"2024-09-21T10:52:26.694338+0000","last_unstale":"2024-09-21T10:52:46.025236+0000","last_undegraded":"2024-09-21T10:52:26.678206+0000","last_fullsized":"2024-09-21T10:52:46.025236+0000","mapping_epoch":1466,"log_start":"1438'5906","ondisk_log_start":"1438'5906","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4364,"log_dups_size":0,"ondisk_log_size":4364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172490752,"num_objects":2632,"num_object_clones":0,"num_object_copies":7896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4458,"num_objects_misplaced":1859,"num_objects_unfound":0,"num_objects_dirty":2632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10270,"num_write_kb":412864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3822,"num_bytes_recovered":250347520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,4],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"0,1,4","objects":806},{"shards":"1,6","objects":1826}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"55.1","version":"1472'10543","reported_seq":11831,"reported_epoch":1472,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:52:46.900886+0000","last_change":"2024-09-21T10:52:15.413755+0000","last_active":"2024-09-21T10:52:46.900886+0000","last_peered":"2024-09-21T10:52:46.900886+0000","last_clean":"2024-09-21T10:50:20.316846+0000","last_became_active":"2024-09-21T10:52:03.442393+0000","last_became_peered":"2024-09-21T10:52:03.442393+0000","last_unstale":"2024-09-21T10:52:46.900886+0000","last_undegraded":"2024-09-21T10:52:15.413422+0000","last_fullsized":"2024-09-21T10:52:03.308195+0000","mapping_epoch":1455,"log_start":"1432'5512","ondisk_log_start":"1432'5512","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":5031,"log_dups_size":0,"ondisk_log_size":5031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:45:04.793364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180551680,"num_objects":2755,"num_object_clones":0,"num_object_copies":8265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10543,"num_write_kb":425536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":596,"num_bytes_recovered":20643840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":2755}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"55.0","version":"1472'10521","reported_seq":14971,"reported_epoch":1472,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T10:52:46.385606+0000","last_change":"2024-09-21T10:52:26.707855+0000","last_active":"2024-09-21T10:52:46.385606+0000","last_peered":"2024-09-21T10:52:46.385606+0000","last_clean":"2024-09-21T10:50:03.041207+0000","last_became_active":"2024-09-21T10:52:26.645887+0000","last_became_peered":"2024-09-21T10:52:26.645887+0000","last_unstale":"2024-09-21T10:52:46.385606+0000","last_undegraded":"2024-09-21T10:52:26.635700+0000","last_fullsized":"2024-09-21T10:52:46.385606+0000","mapping_epoch":1466,"log_start":"1436'5799","ondisk_log_start":"1436'5799","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4722,"log_dups_size":0,"ondisk_log_size":4722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178847744,"num_objects":2729,"num_object_clones":0,"num_object_copies":8187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2729,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10521,"num_write_kb":424000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2153,"num_bytes_recovered":141099008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,1],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"1,3,6","objects":859},{"shards":"3,6","objects":1870}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":1962,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.327789+0000","last_change":"2024-09-21T10:52:03.323899+0000","last_active":"2024-09-21T10:52:43.327789+0000","last_peered":"2024-09-21T10:52:43.327789+0000","last_clean":"2024-09-21T10:52:43.327789+0000","last_became_active":"2024-09-21T10:52:03.314626+0000","last_became_peered":"2024-09-21T10:52:03.314626+0000","last_unstale":"2024-09-21T10:52:43.327789+0000","last_undegraded":"2024-09-21T10:52:43.327789+0000","last_fullsized":"2024-09-21T10:52:43.327789+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:13.939806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1461,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.329164+0000","last_change":"2024-09-21T10:52:15.396755+0000","last_active":"2024-09-21T10:52:43.329164+0000","last_peered":"2024-09-21T10:52:43.329164+0000","last_clean":"2024-09-21T10:52:43.329164+0000","last_became_active":"2024-09-21T10:52:15.396417+0000","last_became_peered":"2024-09-21T10:52:15.396417+0000","last_unstale":"2024-09-21T10:52:43.329164+0000","last_undegraded":"2024-09-21T10:52:43.329164+0000","last_fullsized":"2024-09-21T10:52:43.329164+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:25.005574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":2253,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.326656+0000","last_change":"2024-09-21T10:52:15.400227+0000","last_active":"2024-09-21T10:52:44.326656+0000","last_peered":"2024-09-21T10:52:44.326656+0000","last_clean":"2024-09-21T10:52:44.326656+0000","last_became_active":"2024-09-21T10:52:15.399888+0000","last_became_peered":"2024-09-21T10:52:15.399888+0000","last_unstale":"2024-09-21T10:52:44.326656+0000","last_undegraded":"2024-09-21T10:52:44.326656+0000","last_fullsized":"2024-09-21T10:52:44.326656+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:42:52.565184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2270,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.322833+0000","last_change":"2024-09-21T10:51:27.458216+0000","last_active":"2024-09-21T10:52:44.322833+0000","last_peered":"2024-09-21T10:52:44.322833+0000","last_clean":"2024-09-21T10:52:44.322833+0000","last_became_active":"2024-09-21T10:51:27.457948+0000","last_became_peered":"2024-09-21T10:51:27.457948+0000","last_unstale":"2024-09-21T10:52:44.322833+0000","last_undegraded":"2024-09-21T10:52:44.322833+0000","last_fullsized":"2024-09-21T10:52:44.322833+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:25:29.584079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":100,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.336405+0000","last_change":"2024-09-21T10:49:44.878242+0000","last_active":"2024-09-21T10:52:43.336405+0000","last_peered":"2024-09-21T10:52:43.336405+0000","last_clean":"2024-09-21T10:52:43.336405+0000","last_became_active":"2024-09-21T10:49:44.866578+0000","last_became_peered":"2024-09-21T10:49:44.866578+0000","last_unstale":"2024-09-21T10:52:43.336405+0000","last_undegraded":"2024-09-21T10:52:43.336405+0000","last_fullsized":"2024-09-21T10:52:43.336405+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:17:24.145551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1467,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.329563+0000","last_change":"2024-09-21T10:51:27.460539+0000","last_active":"2024-09-21T10:52:43.329563+0000","last_peered":"2024-09-21T10:52:43.329563+0000","last_clean":"2024-09-21T10:52:43.329563+0000","last_became_active":"2024-09-21T10:51:27.460211+0000","last_became_peered":"2024-09-21T10:51:27.460211+0000","last_unstale":"2024-09-21T10:52:43.329563+0000","last_undegraded":"2024-09-21T10:52:43.329563+0000","last_fullsized":"2024-09-21T10:52:43.329563+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:14:15.700998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":139,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.330001+0000","last_change":"2024-09-21T10:52:15.405886+0000","last_active":"2024-09-21T10:52:43.330001+0000","last_peered":"2024-09-21T10:52:43.330001+0000","last_clean":"2024-09-21T10:52:43.330001+0000","last_became_active":"2024-09-21T10:52:15.397317+0000","last_became_peered":"2024-09-21T10:52:15.397317+0000","last_unstale":"2024-09-21T10:52:43.330001+0000","last_undegraded":"2024-09-21T10:52:43.330001+0000","last_fullsized":"2024-09-21T10:52:43.330001+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:43:48.813861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1458,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.329948+0000","last_change":"2024-09-21T10:09:06.560175+0000","last_active":"2024-09-21T10:52:43.329948+0000","last_peered":"2024-09-21T10:52:43.329948+0000","last_clean":"2024-09-21T10:52:43.329948+0000","last_became_active":"2024-09-21T10:09:06.558584+0000","last_became_peered":"2024-09-21T10:09:06.558584+0000","last_unstale":"2024-09-21T10:52:43.329948+0000","last_undegraded":"2024-09-21T10:52:43.329948+0000","last_fullsized":"2024-09-21T10:52:43.329948+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:02.918339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":48,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.329754+0000","last_change":"2024-09-21T10:51:27.437665+0000","last_active":"2024-09-21T10:52:43.329754+0000","last_peered":"2024-09-21T10:52:43.329754+0000","last_clean":"2024-09-21T10:52:43.329754+0000","last_became_active":"2024-09-21T10:51:27.436481+0000","last_became_peered":"2024-09-21T10:51:27.436481+0000","last_unstale":"2024-09-21T10:52:43.329754+0000","last_undegraded":"2024-09-21T10:52:43.329754+0000","last_fullsized":"2024-09-21T10:52:43.329754+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:07:50.267297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1025,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.321521+0000","last_change":"2024-09-21T10:51:27.459033+0000","last_active":"2024-09-21T10:52:44.321521+0000","last_peered":"2024-09-21T10:52:44.321521+0000","last_clean":"2024-09-21T10:52:44.321521+0000","last_became_active":"2024-09-21T10:51:27.458728+0000","last_became_peered":"2024-09-21T10:51:27.458728+0000","last_unstale":"2024-09-21T10:52:44.321521+0000","last_undegraded":"2024-09-21T10:52:44.321521+0000","last_fullsized":"2024-09-21T10:52:44.321521+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:20.133032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":1487,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.329715+0000","last_change":"2024-09-21T10:52:25.473550+0000","last_active":"2024-09-21T10:52:43.329715+0000","last_peered":"2024-09-21T10:52:43.329715+0000","last_clean":"2024-09-21T10:52:43.329715+0000","last_became_active":"2024-09-21T10:52:25.473105+0000","last_became_peered":"2024-09-21T10:52:25.473105+0000","last_unstale":"2024-09-21T10:52:43.329715+0000","last_undegraded":"2024-09-21T10:52:43.329715+0000","last_fullsized":"2024-09-21T10:52:43.329715+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:15:45.352518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2249,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.324073+0000","last_change":"2024-09-21T10:40:13.432531+0000","last_active":"2024-09-21T10:52:44.324073+0000","last_peered":"2024-09-21T10:52:44.324073+0000","last_clean":"2024-09-21T10:52:44.324073+0000","last_became_active":"2024-09-21T10:40:13.432210+0000","last_became_peered":"2024-09-21T10:40:13.432210+0000","last_unstale":"2024-09-21T10:52:44.324073+0000","last_undegraded":"2024-09-21T10:52:44.324073+0000","last_fullsized":"2024-09-21T10:52:44.324073+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:20:10.998607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2221,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.321069+0000","last_change":"2024-09-21T10:09:06.530394+0000","last_active":"2024-09-21T10:52:44.321069+0000","last_peered":"2024-09-21T10:52:44.321069+0000","last_clean":"2024-09-21T10:52:44.321069+0000","last_became_active":"2024-09-21T10:09:06.529307+0000","last_became_peered":"2024-09-21T10:09:06.529307+0000","last_unstale":"2024-09-21T10:52:44.321069+0000","last_undegraded":"2024-09-21T10:52:44.321069+0000","last_fullsized":"2024-09-21T10:52:44.321069+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:54.287397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.322034+0000","last_change":"2024-09-21T10:52:25.489896+0000","last_active":"2024-09-21T10:52:43.322034+0000","last_peered":"2024-09-21T10:52:43.322034+0000","last_clean":"2024-09-21T10:52:43.322034+0000","last_became_active":"2024-09-21T10:52:25.489154+0000","last_became_peered":"2024-09-21T10:52:25.489154+0000","last_unstale":"2024-09-21T10:52:43.322034+0000","last_undegraded":"2024-09-21T10:52:43.322034+0000","last_fullsized":"2024-09-21T10:52:43.322034+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:36:02.076931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2250,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.325915+0000","last_change":"2024-09-21T10:52:25.493309+0000","last_active":"2024-09-21T10:52:44.325915+0000","last_peered":"2024-09-21T10:52:44.325915+0000","last_clean":"2024-09-21T10:52:44.325915+0000","last_became_active":"2024-09-21T10:52:25.492568+0000","last_became_peered":"2024-09-21T10:52:25.492568+0000","last_unstale":"2024-09-21T10:52:44.325915+0000","last_undegraded":"2024-09-21T10:52:44.325915+0000","last_fullsized":"2024-09-21T10:52:44.325915+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:19:19.393370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1474,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.327854+0000","last_change":"2024-09-21T10:52:03.317474+0000","last_active":"2024-09-21T10:52:43.327854+0000","last_peered":"2024-09-21T10:52:43.327854+0000","last_clean":"2024-09-21T10:52:43.327854+0000","last_became_active":"2024-09-21T10:52:03.317001+0000","last_became_peered":"2024-09-21T10:52:03.317001+0000","last_unstale":"2024-09-21T10:52:43.327854+0000","last_undegraded":"2024-09-21T10:52:43.327854+0000","last_fullsized":"2024-09-21T10:52:43.327854+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:06:07.039763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2249,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.326138+0000","last_change":"2024-09-21T10:52:15.412246+0000","last_active":"2024-09-21T10:52:44.326138+0000","last_peered":"2024-09-21T10:52:44.326138+0000","last_clean":"2024-09-21T10:52:44.326138+0000","last_became_active":"2024-09-21T10:52:15.406204+0000","last_became_peered":"2024-09-21T10:52:15.406204+0000","last_unstale":"2024-09-21T10:52:44.326138+0000","last_undegraded":"2024-09-21T10:52:44.326138+0000","last_fullsized":"2024-09-21T10:52:44.326138+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:03.524324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2229,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.324299+0000","last_change":"2024-09-21T10:52:15.402822+0000","last_active":"2024-09-21T10:52:44.324299+0000","last_peered":"2024-09-21T10:52:44.324299+0000","last_clean":"2024-09-21T10:52:44.324299+0000","last_became_active":"2024-09-21T10:52:15.402548+0000","last_became_peered":"2024-09-21T10:52:15.402548+0000","last_unstale":"2024-09-21T10:52:44.324299+0000","last_undegraded":"2024-09-21T10:52:44.324299+0000","last_fullsized":"2024-09-21T10:52:44.324299+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:04:04.667598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2254,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.323022+0000","last_change":"2024-09-21T10:52:25.499172+0000","last_active":"2024-09-21T10:52:44.323022+0000","last_peered":"2024-09-21T10:52:44.323022+0000","last_clean":"2024-09-21T10:52:44.323022+0000","last_became_active":"2024-09-21T10:52:25.498835+0000","last_became_peered":"2024-09-21T10:52:25.498835+0000","last_unstale":"2024-09-21T10:52:44.323022+0000","last_undegraded":"2024-09-21T10:52:44.323022+0000","last_fullsized":"2024-09-21T10:52:44.323022+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:10:12.434470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.321971+0000","last_change":"2024-09-21T10:52:25.489774+0000","last_active":"2024-09-21T10:52:43.321971+0000","last_peered":"2024-09-21T10:52:43.321971+0000","last_clean":"2024-09-21T10:52:43.321971+0000","last_became_active":"2024-09-21T10:52:25.487788+0000","last_became_peered":"2024-09-21T10:52:25.487788+0000","last_unstale":"2024-09-21T10:52:43.321971+0000","last_undegraded":"2024-09-21T10:52:43.321971+0000","last_fullsized":"2024-09-21T10:52:43.321971+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:48.842419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":23,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.320768+0000","last_change":"2024-09-21T10:52:25.565465+0000","last_active":"2024-09-21T10:52:43.320768+0000","last_peered":"2024-09-21T10:52:43.320768+0000","last_clean":"2024-09-21T10:52:43.320768+0000","last_became_active":"2024-09-21T10:52:25.510131+0000","last_became_peered":"2024-09-21T10:52:25.510131+0000","last_unstale":"2024-09-21T10:52:43.320768+0000","last_undegraded":"2024-09-21T10:52:43.320768+0000","last_fullsized":"2024-09-21T10:52:43.320768+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:41:13.870503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":450,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2256,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.328069+0000","last_change":"2024-09-21T10:52:25.504787+0000","last_active":"2024-09-21T10:52:44.328069+0000","last_peered":"2024-09-21T10:52:44.328069+0000","last_clean":"2024-09-21T10:52:44.328069+0000","last_became_active":"2024-09-21T10:52:25.504198+0000","last_became_peered":"2024-09-21T10:52:25.504198+0000","last_unstale":"2024-09-21T10:52:44.328069+0000","last_undegraded":"2024-09-21T10:52:44.328069+0000","last_fullsized":"2024-09-21T10:52:44.328069+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:38:25.666224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":1491,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.330079+0000","last_change":"2024-09-21T10:44:14.415264+0000","last_active":"2024-09-21T10:52:43.330079+0000","last_peered":"2024-09-21T10:52:43.330079+0000","last_clean":"2024-09-21T10:52:43.330079+0000","last_became_active":"2024-09-21T10:44:14.407813+0000","last_became_peered":"2024-09-21T10:44:14.407813+0000","last_unstale":"2024-09-21T10:52:43.330079+0000","last_undegraded":"2024-09-21T10:52:43.330079+0000","last_fullsized":"2024-09-21T10:52:43.330079+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:55.933601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1444,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.330017+0000","last_change":"2024-09-21T10:07:56.001756+0000","last_active":"2024-09-21T10:52:43.330017+0000","last_peered":"2024-09-21T10:52:43.330017+0000","last_clean":"2024-09-21T10:52:43.330017+0000","last_became_active":"2024-09-21T10:07:56.000789+0000","last_became_peered":"2024-09-21T10:07:56.000789+0000","last_unstale":"2024-09-21T10:52:43.330017+0000","last_undegraded":"2024-09-21T10:52:43.330017+0000","last_fullsized":"2024-09-21T10:52:43.330017+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:56:04.302902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"346'149","reported_seq":906,"reported_epoch":1472,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:52:44.323019+0000","last_change":"2024-09-21T10:52:26.478603+0000","last_active":"2024-09-21T10:52:44.323019+0000","last_peered":"2024-09-21T10:52:44.323019+0000","last_clean":"2024-09-21T10:52:16.392213+0000","last_became_active":"2024-09-21T10:52:26.478167+0000","last_became_peered":"2024-09-21T10:52:26.478167+0000","last_unstale":"2024-09-21T10:52:44.323019+0000","last_undegraded":"2024-09-21T10:52:26.463682+0000","last_fullsized":"2024-09-21T10:52:26.463251+0000","mapping_epoch":1466,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1449,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1251389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,6],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":1},{"shards":"3,6","objects":-1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1459,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.329193+0000","last_change":"2024-09-21T10:52:25.473127+0000","last_active":"2024-09-21T10:52:43.329193+0000","last_peered":"2024-09-21T10:52:43.329193+0000","last_clean":"2024-09-21T10:52:43.329193+0000","last_became_active":"2024-09-21T10:52:25.472805+0000","last_became_peered":"2024-09-21T10:52:25.472805+0000","last_unstale":"2024-09-21T10:52:43.329193+0000","last_undegraded":"2024-09-21T10:52:43.329193+0000","last_fullsized":"2024-09-21T10:52:43.329193+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:13.038797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1505,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.327911+0000","last_change":"2024-09-21T10:52:25.493263+0000","last_active":"2024-09-21T10:52:43.327911+0000","last_peered":"2024-09-21T10:52:43.327911+0000","last_clean":"2024-09-21T10:52:43.327911+0000","last_became_active":"2024-09-21T10:52:25.492970+0000","last_became_peered":"2024-09-21T10:52:25.492970+0000","last_unstale":"2024-09-21T10:52:43.327911+0000","last_undegraded":"2024-09-21T10:52:43.327911+0000","last_fullsized":"2024-09-21T10:52:43.327911+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:43:40.305783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":100,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.322354+0000","last_change":"2024-09-21T10:52:25.513834+0000","last_active":"2024-09-21T10:52:44.322354+0000","last_peered":"2024-09-21T10:52:44.322354+0000","last_clean":"2024-09-21T10:52:44.322354+0000","last_became_active":"2024-09-21T10:52:25.475423+0000","last_became_peered":"2024-09-21T10:52:25.475423+0000","last_unstale":"2024-09-21T10:52:44.322354+0000","last_undegraded":"2024-09-21T10:52:44.322354+0000","last_fullsized":"2024-09-21T10:52:44.322354+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:10.216527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"346'149","reported_seq":891,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.335624+0000","last_change":"2024-09-21T10:52:24.459450+0000","last_active":"2024-09-21T10:52:43.335624+0000","last_peered":"2024-09-21T10:52:43.335624+0000","last_clean":"2024-09-21T10:52:43.335624+0000","last_became_active":"2024-09-21T10:52:15.405544+0000","last_became_peered":"2024-09-21T10:52:15.405544+0000","last_unstale":"2024-09-21T10:52:43.335624+0000","last_undegraded":"2024-09-21T10:52:43.335624+0000","last_fullsized":"2024-09-21T10:52:43.335624+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2221,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.326066+0000","last_change":"2024-09-21T10:52:25.494316+0000","last_active":"2024-09-21T10:52:44.326066+0000","last_peered":"2024-09-21T10:52:44.326066+0000","last_clean":"2024-09-21T10:52:44.326066+0000","last_became_active":"2024-09-21T10:52:25.494016+0000","last_became_peered":"2024-09-21T10:52:25.494016+0000","last_unstale":"2024-09-21T10:52:44.326066+0000","last_undegraded":"2024-09-21T10:52:44.326066+0000","last_fullsized":"2024-09-21T10:52:44.326066+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:07.859584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2222,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.322788+0000","last_change":"2024-09-21T10:07:56.006236+0000","last_active":"2024-09-21T10:52:44.322788+0000","last_peered":"2024-09-21T10:52:44.322788+0000","last_clean":"2024-09-21T10:52:44.322788+0000","last_became_active":"2024-09-21T10:07:56.004439+0000","last_became_peered":"2024-09-21T10:07:56.004439+0000","last_unstale":"2024-09-21T10:52:44.322788+0000","last_undegraded":"2024-09-21T10:52:44.322788+0000","last_fullsized":"2024-09-21T10:52:44.322788+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:39.422467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":2272,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.324467+0000","last_change":"2024-09-21T10:52:25.536401+0000","last_active":"2024-09-21T10:52:44.324467+0000","last_peered":"2024-09-21T10:52:44.324467+0000","last_clean":"2024-09-21T10:52:44.324467+0000","last_became_active":"2024-09-21T10:52:25.496480+0000","last_became_peered":"2024-09-21T10:52:25.496480+0000","last_unstale":"2024-09-21T10:52:44.324467+0000","last_undegraded":"2024-09-21T10:52:44.324467+0000","last_fullsized":"2024-09-21T10:52:44.324467+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:50:34.820373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"346'149","reported_seq":716,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.326616+0000","last_change":"2024-09-21T10:51:30.450301+0000","last_active":"2024-09-21T10:52:43.326616+0000","last_peered":"2024-09-21T10:52:43.326616+0000","last_clean":"2024-09-21T10:52:43.326616+0000","last_became_active":"2024-09-21T10:51:30.449925+0000","last_became_peered":"2024-09-21T10:51:30.449925+0000","last_unstale":"2024-09-21T10:52:43.326616+0000","last_undegraded":"2024-09-21T10:52:43.326616+0000","last_fullsized":"2024-09-21T10:52:43.326616+0000","mapping_epoch":1450,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1451,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:41.011085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:41.011085+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:38:33.451261+0000","scrub_duration":0.0061250310000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2187,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.321001+0000","last_change":"2024-09-21T10:51:27.431416+0000","last_active":"2024-09-21T10:52:43.321001+0000","last_peered":"2024-09-21T10:52:43.321001+0000","last_clean":"2024-09-21T10:52:43.321001+0000","last_became_active":"2024-09-21T10:51:27.431127+0000","last_became_peered":"2024-09-21T10:51:27.431127+0000","last_unstale":"2024-09-21T10:52:43.321001+0000","last_undegraded":"2024-09-21T10:52:43.321001+0000","last_fullsized":"2024-09-21T10:52:43.321001+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:27.721665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.320666+0000","last_change":"2024-09-21T10:52:25.468849+0000","last_active":"2024-09-21T10:52:43.320666+0000","last_peered":"2024-09-21T10:52:43.320666+0000","last_clean":"2024-09-21T10:52:43.320666+0000","last_became_active":"2024-09-21T10:52:25.468552+0000","last_became_peered":"2024-09-21T10:52:25.468552+0000","last_unstale":"2024-09-21T10:52:43.320666+0000","last_undegraded":"2024-09-21T10:52:43.320666+0000","last_fullsized":"2024-09-21T10:52:43.320666+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:36:41.692028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2272,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.321458+0000","last_change":"2024-09-21T10:51:27.436608+0000","last_active":"2024-09-21T10:52:44.321458+0000","last_peered":"2024-09-21T10:52:44.321458+0000","last_clean":"2024-09-21T10:52:44.321458+0000","last_became_active":"2024-09-21T10:51:27.434949+0000","last_became_peered":"2024-09-21T10:51:27.434949+0000","last_unstale":"2024-09-21T10:52:44.321458+0000","last_undegraded":"2024-09-21T10:52:44.321458+0000","last_fullsized":"2024-09-21T10:52:44.321458+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:49:23.439425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"346'149","reported_seq":759,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.326614+0000","last_change":"2024-09-21T10:52:28.590793+0000","last_active":"2024-09-21T10:52:43.326614+0000","last_peered":"2024-09-21T10:52:43.326614+0000","last_clean":"2024-09-21T10:52:43.326614+0000","last_became_active":"2024-09-21T10:52:28.588662+0000","last_became_peered":"2024-09-21T10:52:28.588662+0000","last_unstale":"2024-09-21T10:52:43.326614+0000","last_undegraded":"2024-09-21T10:52:43.326614+0000","last_fullsized":"2024-09-21T10:52:43.326614+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:44.018645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:44.018645+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:46.849427+0000","scrub_duration":0.0012789399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2227,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.321646+0000","last_change":"2024-09-21T10:51:27.440425+0000","last_active":"2024-09-21T10:52:44.321646+0000","last_peered":"2024-09-21T10:52:44.321646+0000","last_clean":"2024-09-21T10:52:44.321646+0000","last_became_active":"2024-09-21T10:51:27.439666+0000","last_became_peered":"2024-09-21T10:51:27.439666+0000","last_unstale":"2024-09-21T10:52:44.321646+0000","last_undegraded":"2024-09-21T10:52:44.321646+0000","last_fullsized":"2024-09-21T10:52:44.321646+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:56:47.410516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2295,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.321321+0000","last_change":"2024-09-21T10:52:03.314926+0000","last_active":"2024-09-21T10:52:44.321321+0000","last_peered":"2024-09-21T10:52:44.321321+0000","last_clean":"2024-09-21T10:52:44.321321+0000","last_became_active":"2024-09-21T10:52:03.314602+0000","last_became_peered":"2024-09-21T10:52:03.314602+0000","last_unstale":"2024-09-21T10:52:44.321321+0000","last_undegraded":"2024-09-21T10:52:44.321321+0000","last_fullsized":"2024-09-21T10:52:44.321321+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:13.774521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1450,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.323936+0000","last_change":"2024-09-21T10:52:25.502830+0000","last_active":"2024-09-21T10:52:44.323936+0000","last_peered":"2024-09-21T10:52:44.323936+0000","last_clean":"2024-09-21T10:52:44.323936+0000","last_became_active":"2024-09-21T10:52:25.502505+0000","last_became_peered":"2024-09-21T10:52:25.502505+0000","last_unstale":"2024-09-21T10:52:44.323936+0000","last_undegraded":"2024-09-21T10:52:44.323936+0000","last_fullsized":"2024-09-21T10:52:44.323936+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:01:14.965694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"346'149","reported_seq":814,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.329010+0000","last_change":"2024-09-21T10:52:24.458657+0000","last_active":"2024-09-21T10:52:43.329010+0000","last_peered":"2024-09-21T10:52:43.329010+0000","last_clean":"2024-09-21T10:52:43.329010+0000","last_became_active":"2024-09-21T10:51:30.436983+0000","last_became_peered":"2024-09-21T10:51:30.436983+0000","last_unstale":"2024-09-21T10:52:43.329010+0000","last_undegraded":"2024-09-21T10:52:43.329010+0000","last_fullsized":"2024-09-21T10:52:43.329010+0000","mapping_epoch":1450,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1451,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2289,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.321594+0000","last_change":"2024-09-21T10:52:15.397180+0000","last_active":"2024-09-21T10:52:44.321594+0000","last_peered":"2024-09-21T10:52:44.321594+0000","last_clean":"2024-09-21T10:52:44.321594+0000","last_became_active":"2024-09-21T10:52:15.396879+0000","last_became_peered":"2024-09-21T10:52:15.396879+0000","last_unstale":"2024-09-21T10:52:44.321594+0000","last_undegraded":"2024-09-21T10:52:44.321594+0000","last_fullsized":"2024-09-21T10:52:44.321594+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:03:07.162178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1457,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.330093+0000","last_change":"2024-09-21T10:51:27.436590+0000","last_active":"2024-09-21T10:52:43.330093+0000","last_peered":"2024-09-21T10:52:43.330093+0000","last_clean":"2024-09-21T10:52:43.330093+0000","last_became_active":"2024-09-21T10:51:27.435332+0000","last_became_peered":"2024-09-21T10:51:27.435332+0000","last_unstale":"2024-09-21T10:52:43.330093+0000","last_undegraded":"2024-09-21T10:52:43.330093+0000","last_fullsized":"2024-09-21T10:52:43.330093+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:05:42.885506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":184,"reported_epoch":1472,"state":"active+recovery_wait","last_fresh":"2024-09-21T10:52:44.321294+0000","last_change":"2024-09-21T10:52:26.680832+0000","last_active":"2024-09-21T10:52:44.321294+0000","last_peered":"2024-09-21T10:52:44.321294+0000","last_clean":"2024-09-21T10:51:45.437560+0000","last_became_active":"2024-09-21T10:52:03.315398+0000","last_became_peered":"2024-09-21T10:52:03.315398+0000","last_unstale":"2024-09-21T10:52:44.321294+0000","last_undegraded":"2024-09-21T10:52:44.321294+0000","last_fullsized":"2024-09-21T10:52:44.321294+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":["1","3","4","6"],"object_location_counts":[{"shards":"1,3,6","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"346'149","reported_seq":785,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.322119+0000","last_change":"2024-09-21T10:52:28.495319+0000","last_active":"2024-09-21T10:52:43.322119+0000","last_peered":"2024-09-21T10:52:43.322119+0000","last_clean":"2024-09-21T10:52:43.322119+0000","last_became_active":"2024-09-21T10:52:28.494923+0000","last_became_peered":"2024-09-21T10:52:28.494923+0000","last_unstale":"2024-09-21T10:52:43.322119+0000","last_undegraded":"2024-09-21T10:52:43.322119+0000","last_fullsized":"2024-09-21T10:52:43.322119+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:52:17.894787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:52:17.894787+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":175,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.330174+0000","last_change":"2024-09-21T10:52:25.475200+0000","last_active":"2024-09-21T10:52:43.330174+0000","last_peered":"2024-09-21T10:52:43.330174+0000","last_clean":"2024-09-21T10:52:43.330174+0000","last_became_active":"2024-09-21T10:52:25.474872+0000","last_became_peered":"2024-09-21T10:52:25.474872+0000","last_unstale":"2024-09-21T10:52:43.330174+0000","last_undegraded":"2024-09-21T10:52:43.330174+0000","last_fullsized":"2024-09-21T10:52:43.330174+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:36:34.207844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2246,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.320991+0000","last_change":"2024-09-21T10:52:03.325889+0000","last_active":"2024-09-21T10:52:44.320991+0000","last_peered":"2024-09-21T10:52:44.320991+0000","last_clean":"2024-09-21T10:52:44.320991+0000","last_became_active":"2024-09-21T10:52:03.325500+0000","last_became_peered":"2024-09-21T10:52:03.325500+0000","last_unstale":"2024-09-21T10:52:44.320991+0000","last_undegraded":"2024-09-21T10:52:44.320991+0000","last_fullsized":"2024-09-21T10:52:44.320991+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:20.708252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1032,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.322744+0000","last_change":"2024-09-21T10:51:27.457894+0000","last_active":"2024-09-21T10:52:44.322744+0000","last_peered":"2024-09-21T10:52:44.322744+0000","last_clean":"2024-09-21T10:52:44.322744+0000","last_became_active":"2024-09-21T10:51:27.457431+0000","last_became_peered":"2024-09-21T10:51:27.457431+0000","last_unstale":"2024-09-21T10:52:44.322744+0000","last_undegraded":"2024-09-21T10:52:44.322744+0000","last_fullsized":"2024-09-21T10:52:44.322744+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:03:33.550759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"346'149","reported_seq":902,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.322213+0000","last_change":"2024-09-21T10:52:15.396042+0000","last_active":"2024-09-21T10:52:44.322213+0000","last_peered":"2024-09-21T10:52:44.322213+0000","last_clean":"2024-09-21T10:52:44.322213+0000","last_became_active":"2024-09-21T10:52:15.395717+0000","last_became_peered":"2024-09-21T10:52:15.395717+0000","last_unstale":"2024-09-21T10:52:44.322213+0000","last_undegraded":"2024-09-21T10:52:44.322213+0000","last_fullsized":"2024-09-21T10:52:44.322213+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2271,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.322252+0000","last_change":"2024-09-21T10:52:25.476572+0000","last_active":"2024-09-21T10:52:44.322252+0000","last_peered":"2024-09-21T10:52:44.322252+0000","last_clean":"2024-09-21T10:52:44.322252+0000","last_became_active":"2024-09-21T10:52:25.475803+0000","last_became_peered":"2024-09-21T10:52:25.475803+0000","last_unstale":"2024-09-21T10:52:44.322252+0000","last_undegraded":"2024-09-21T10:52:44.322252+0000","last_fullsized":"2024-09-21T10:52:44.322252+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:06:44.191486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2203,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.328964+0000","last_change":"2024-09-21T10:52:03.326753+0000","last_active":"2024-09-21T10:52:43.328964+0000","last_peered":"2024-09-21T10:52:43.328964+0000","last_clean":"2024-09-21T10:52:43.328964+0000","last_became_active":"2024-09-21T10:52:03.326348+0000","last_became_peered":"2024-09-21T10:52:03.326348+0000","last_unstale":"2024-09-21T10:52:43.328964+0000","last_undegraded":"2024-09-21T10:52:43.328964+0000","last_fullsized":"2024-09-21T10:52:43.328964+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:57:38.100799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":126,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.322498+0000","last_change":"2024-09-21T10:51:27.426006+0000","last_active":"2024-09-21T10:52:43.322498+0000","last_peered":"2024-09-21T10:52:43.322498+0000","last_clean":"2024-09-21T10:52:43.322498+0000","last_became_active":"2024-09-21T10:51:27.425698+0000","last_became_peered":"2024-09-21T10:51:27.425698+0000","last_unstale":"2024-09-21T10:52:43.322498+0000","last_undegraded":"2024-09-21T10:52:43.322498+0000","last_fullsized":"2024-09-21T10:52:43.322498+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:35:13.210330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":20,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.325825+0000","last_change":"2024-09-21T10:52:15.401737+0000","last_active":"2024-09-21T10:52:43.325825+0000","last_peered":"2024-09-21T10:52:43.325825+0000","last_clean":"2024-09-21T10:52:43.325825+0000","last_became_active":"2024-09-21T10:52:15.400859+0000","last_became_peered":"2024-09-21T10:52:15.400859+0000","last_unstale":"2024-09-21T10:52:43.325825+0000","last_undegraded":"2024-09-21T10:52:43.325825+0000","last_fullsized":"2024-09-21T10:52:43.325825+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:15.723031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2225,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.325223+0000","last_change":"2024-09-21T10:49:44.865830+0000","last_active":"2024-09-21T10:52:44.325223+0000","last_peered":"2024-09-21T10:52:44.325223+0000","last_clean":"2024-09-21T10:52:44.325223+0000","last_became_active":"2024-09-21T10:49:44.865499+0000","last_became_peered":"2024-09-21T10:49:44.865499+0000","last_unstale":"2024-09-21T10:52:44.325223+0000","last_undegraded":"2024-09-21T10:52:44.325223+0000","last_fullsized":"2024-09-21T10:52:44.325223+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:19:22.013154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":1451,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.321398+0000","last_change":"2024-09-21T10:52:25.489550+0000","last_active":"2024-09-21T10:52:43.321398+0000","last_peered":"2024-09-21T10:52:43.321398+0000","last_clean":"2024-09-21T10:52:43.321398+0000","last_became_active":"2024-09-21T10:52:25.489171+0000","last_became_peered":"2024-09-21T10:52:25.489171+0000","last_unstale":"2024-09-21T10:52:43.321398+0000","last_undegraded":"2024-09-21T10:52:43.321398+0000","last_fullsized":"2024-09-21T10:52:43.321398+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:21.105058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":785,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.320769+0000","last_change":"2024-09-21T10:52:28.601906+0000","last_active":"2024-09-21T10:52:43.320769+0000","last_peered":"2024-09-21T10:52:43.320769+0000","last_clean":"2024-09-21T10:52:43.320769+0000","last_became_active":"2024-09-21T10:52:28.587447+0000","last_became_peered":"2024-09-21T10:52:28.587447+0000","last_unstale":"2024-09-21T10:52:43.320769+0000","last_undegraded":"2024-09-21T10:52:43.320769+0000","last_fullsized":"2024-09-21T10:52:43.320769+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:44.223578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:44.223578+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":74,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"346'149","reported_seq":733,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.326142+0000","last_change":"2024-09-21T10:51:30.453979+0000","last_active":"2024-09-21T10:52:43.326142+0000","last_peered":"2024-09-21T10:52:43.326142+0000","last_clean":"2024-09-21T10:52:43.326142+0000","last_became_active":"2024-09-21T10:51:30.450893+0000","last_became_peered":"2024-09-21T10:51:30.450893+0000","last_unstale":"2024-09-21T10:52:43.326142+0000","last_undegraded":"2024-09-21T10:52:43.326142+0000","last_fullsized":"2024-09-21T10:52:43.326142+0000","mapping_epoch":1450,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1451,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:42.008235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:42.008235+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:21:48.570809+0000","scrub_duration":0.011129834999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2222,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.322099+0000","last_change":"2024-09-21T10:09:06.538940+0000","last_active":"2024-09-21T10:52:44.322099+0000","last_peered":"2024-09-21T10:52:44.322099+0000","last_clean":"2024-09-21T10:52:44.322099+0000","last_became_active":"2024-09-21T10:09:06.538668+0000","last_became_peered":"2024-09-21T10:09:06.538668+0000","last_unstale":"2024-09-21T10:52:44.322099+0000","last_undegraded":"2024-09-21T10:52:44.322099+0000","last_fullsized":"2024-09-21T10:52:44.322099+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:42:44.299488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2221,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.325825+0000","last_change":"2024-09-21T10:52:25.493032+0000","last_active":"2024-09-21T10:52:44.325825+0000","last_peered":"2024-09-21T10:52:44.325825+0000","last_clean":"2024-09-21T10:52:44.325825+0000","last_became_active":"2024-09-21T10:52:25.492182+0000","last_became_peered":"2024-09-21T10:52:25.492182+0000","last_unstale":"2024-09-21T10:52:44.325825+0000","last_undegraded":"2024-09-21T10:52:44.325825+0000","last_fullsized":"2024-09-21T10:52:44.325825+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:12:57.118420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1463,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.329921+0000","last_change":"2024-09-21T10:52:15.402809+0000","last_active":"2024-09-21T10:52:43.329921+0000","last_peered":"2024-09-21T10:52:43.329921+0000","last_clean":"2024-09-21T10:52:43.329921+0000","last_became_active":"2024-09-21T10:52:15.402497+0000","last_became_peered":"2024-09-21T10:52:15.402497+0000","last_unstale":"2024-09-21T10:52:43.329921+0000","last_undegraded":"2024-09-21T10:52:43.329921+0000","last_fullsized":"2024-09-21T10:52:43.329921+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:07:35.713534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1474,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.329376+0000","last_change":"2024-09-21T10:44:14.399842+0000","last_active":"2024-09-21T10:52:43.329376+0000","last_peered":"2024-09-21T10:52:43.329376+0000","last_clean":"2024-09-21T10:52:43.329376+0000","last_became_active":"2024-09-21T10:44:14.399426+0000","last_became_peered":"2024-09-21T10:44:14.399426+0000","last_unstale":"2024-09-21T10:52:43.329376+0000","last_undegraded":"2024-09-21T10:52:43.329376+0000","last_fullsized":"2024-09-21T10:52:43.329376+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:26:45.362101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"346'149","reported_seq":902,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.324187+0000","last_change":"2024-09-21T10:52:16.404597+0000","last_active":"2024-09-21T10:52:44.324187+0000","last_peered":"2024-09-21T10:52:44.324187+0000","last_clean":"2024-09-21T10:52:44.324187+0000","last_became_active":"2024-09-21T10:52:16.404194+0000","last_became_peered":"2024-09-21T10:52:16.404194+0000","last_unstale":"2024-09-21T10:52:44.324187+0000","last_undegraded":"2024-09-21T10:52:44.324187+0000","last_fullsized":"2024-09-21T10:52:44.324187+0000","mapping_epoch":1463,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1464,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":793725,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1469,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.336414+0000","last_change":"2024-09-21T10:40:13.432387+0000","last_active":"2024-09-21T10:52:43.336414+0000","last_peered":"2024-09-21T10:52:43.336414+0000","last_clean":"2024-09-21T10:52:43.336414+0000","last_became_active":"2024-09-21T10:40:13.432053+0000","last_became_peered":"2024-09-21T10:40:13.432053+0000","last_unstale":"2024-09-21T10:52:43.336414+0000","last_undegraded":"2024-09-21T10:52:43.336414+0000","last_fullsized":"2024-09-21T10:52:43.336414+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:37.486101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1447,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.326349+0000","last_change":"2024-09-21T10:07:56.020791+0000","last_active":"2024-09-21T10:52:43.326349+0000","last_peered":"2024-09-21T10:52:43.326349+0000","last_clean":"2024-09-21T10:52:43.326349+0000","last_became_active":"2024-09-21T10:07:56.020205+0000","last_became_peered":"2024-09-21T10:07:56.020205+0000","last_unstale":"2024-09-21T10:52:43.326349+0000","last_undegraded":"2024-09-21T10:52:43.326349+0000","last_fullsized":"2024-09-21T10:52:43.326349+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:09.813166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"346'149","reported_seq":910,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.321642+0000","last_change":"2024-09-21T10:49:47.904295+0000","last_active":"2024-09-21T10:52:43.321642+0000","last_peered":"2024-09-21T10:52:43.321642+0000","last_clean":"2024-09-21T10:52:43.321642+0000","last_became_active":"2024-09-21T10:49:47.903650+0000","last_became_peered":"2024-09-21T10:49:47.903650+0000","last_unstale":"2024-09-21T10:52:43.321642+0000","last_undegraded":"2024-09-21T10:52:43.321642+0000","last_fullsized":"2024-09-21T10:52:43.321642+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":74,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1445,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.330033+0000","last_change":"2024-09-21T10:09:06.533271+0000","last_active":"2024-09-21T10:52:43.330033+0000","last_peered":"2024-09-21T10:52:43.330033+0000","last_clean":"2024-09-21T10:52:43.330033+0000","last_became_active":"2024-09-21T10:09:06.532041+0000","last_became_peered":"2024-09-21T10:09:06.532041+0000","last_unstale":"2024-09-21T10:52:43.330033+0000","last_undegraded":"2024-09-21T10:52:43.330033+0000","last_fullsized":"2024-09-21T10:52:43.330033+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:04.236375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":2250,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.324842+0000","last_change":"2024-09-21T10:51:27.529295+0000","last_active":"2024-09-21T10:52:44.324842+0000","last_peered":"2024-09-21T10:52:44.324842+0000","last_clean":"2024-09-21T10:52:44.324842+0000","last_became_active":"2024-09-21T10:51:27.446359+0000","last_became_peered":"2024-09-21T10:51:27.446359+0000","last_unstale":"2024-09-21T10:52:44.324842+0000","last_undegraded":"2024-09-21T10:52:44.324842+0000","last_fullsized":"2024-09-21T10:52:44.324842+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:17:08.788558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":680,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.325179+0000","last_change":"2024-09-21T10:51:27.481103+0000","last_active":"2024-09-21T10:52:44.325179+0000","last_peered":"2024-09-21T10:52:44.325179+0000","last_clean":"2024-09-21T10:52:44.325179+0000","last_became_active":"2024-09-21T10:51:27.480763+0000","last_became_peered":"2024-09-21T10:51:27.480763+0000","last_unstale":"2024-09-21T10:52:44.325179+0000","last_undegraded":"2024-09-21T10:52:44.325179+0000","last_fullsized":"2024-09-21T10:52:44.325179+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:35.375303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":785,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.323342+0000","last_change":"2024-09-21T10:49:44.864819+0000","last_active":"2024-09-21T10:52:44.323342+0000","last_peered":"2024-09-21T10:52:44.323342+0000","last_clean":"2024-09-21T10:52:44.323342+0000","last_became_active":"2024-09-21T10:49:44.864061+0000","last_became_peered":"2024-09-21T10:49:44.864061+0000","last_unstale":"2024-09-21T10:52:44.323342+0000","last_undegraded":"2024-09-21T10:52:44.323342+0000","last_fullsized":"2024-09-21T10:52:44.323342+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:37.266967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":29,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.325820+0000","last_change":"2024-09-21T10:52:25.492570+0000","last_active":"2024-09-21T10:52:43.325820+0000","last_peered":"2024-09-21T10:52:43.325820+0000","last_clean":"2024-09-21T10:52:43.325820+0000","last_became_active":"2024-09-21T10:52:25.486382+0000","last_became_peered":"2024-09-21T10:52:25.486382+0000","last_unstale":"2024-09-21T10:52:43.325820+0000","last_undegraded":"2024-09-21T10:52:43.325820+0000","last_fullsized":"2024-09-21T10:52:43.325820+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:06.702707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":133,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.321039+0000","last_change":"2024-09-21T10:49:44.874522+0000","last_active":"2024-09-21T10:52:43.321039+0000","last_peered":"2024-09-21T10:52:43.321039+0000","last_clean":"2024-09-21T10:52:43.321039+0000","last_became_active":"2024-09-21T10:49:44.863242+0000","last_became_peered":"2024-09-21T10:49:44.863242+0000","last_unstale":"2024-09-21T10:52:43.321039+0000","last_undegraded":"2024-09-21T10:52:43.321039+0000","last_fullsized":"2024-09-21T10:52:43.321039+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:09:02.066837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1479,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.329258+0000","last_change":"2024-09-21T10:52:03.315267+0000","last_active":"2024-09-21T10:52:43.329258+0000","last_peered":"2024-09-21T10:52:43.329258+0000","last_clean":"2024-09-21T10:52:43.329258+0000","last_became_active":"2024-09-21T10:52:03.314474+0000","last_became_peered":"2024-09-21T10:52:03.314474+0000","last_unstale":"2024-09-21T10:52:43.329258+0000","last_undegraded":"2024-09-21T10:52:43.329258+0000","last_fullsized":"2024-09-21T10:52:43.329258+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:00:57.791341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":21,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.321903+0000","last_change":"2024-09-21T10:52:25.518721+0000","last_active":"2024-09-21T10:52:43.321903+0000","last_peered":"2024-09-21T10:52:43.321903+0000","last_clean":"2024-09-21T10:52:43.321903+0000","last_became_active":"2024-09-21T10:52:25.472114+0000","last_became_peered":"2024-09-21T10:52:25.472114+0000","last_unstale":"2024-09-21T10:52:43.321903+0000","last_undegraded":"2024-09-21T10:52:43.321903+0000","last_fullsized":"2024-09-21T10:52:43.321903+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:33:48.471147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.320675+0000","last_change":"2024-09-21T10:52:25.494490+0000","last_active":"2024-09-21T10:52:43.320675+0000","last_peered":"2024-09-21T10:52:43.320675+0000","last_clean":"2024-09-21T10:52:43.320675+0000","last_became_active":"2024-09-21T10:52:25.494200+0000","last_became_peered":"2024-09-21T10:52:25.494200+0000","last_unstale":"2024-09-21T10:52:43.320675+0000","last_undegraded":"2024-09-21T10:52:43.320675+0000","last_fullsized":"2024-09-21T10:52:43.320675+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:15:21.030031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":21,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.321311+0000","last_change":"2024-09-21T10:52:25.479229+0000","last_active":"2024-09-21T10:52:43.321311+0000","last_peered":"2024-09-21T10:52:43.321311+0000","last_clean":"2024-09-21T10:52:43.321311+0000","last_became_active":"2024-09-21T10:52:25.470551+0000","last_became_peered":"2024-09-21T10:52:25.470551+0000","last_unstale":"2024-09-21T10:52:43.321311+0000","last_undegraded":"2024-09-21T10:52:43.321311+0000","last_fullsized":"2024-09-21T10:52:43.321311+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:15.458231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1491,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.327306+0000","last_change":"2024-09-21T10:51:27.517234+0000","last_active":"2024-09-21T10:52:43.327306+0000","last_peered":"2024-09-21T10:52:43.327306+0000","last_clean":"2024-09-21T10:52:43.327306+0000","last_became_active":"2024-09-21T10:51:27.431253+0000","last_became_peered":"2024-09-21T10:51:27.431253+0000","last_unstale":"2024-09-21T10:52:43.327306+0000","last_undegraded":"2024-09-21T10:52:43.327306+0000","last_fullsized":"2024-09-21T10:52:43.327306+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:35:42.249305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2205,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.324390+0000","last_change":"2024-09-21T10:52:25.497298+0000","last_active":"2024-09-21T10:52:44.324390+0000","last_peered":"2024-09-21T10:52:44.324390+0000","last_clean":"2024-09-21T10:52:44.324390+0000","last_became_active":"2024-09-21T10:52:25.496876+0000","last_became_peered":"2024-09-21T10:52:25.496876+0000","last_unstale":"2024-09-21T10:52:44.324390+0000","last_undegraded":"2024-09-21T10:52:44.324390+0000","last_fullsized":"2024-09-21T10:52:44.324390+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:16.587897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":116,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.321788+0000","last_change":"2024-09-21T10:44:14.398878+0000","last_active":"2024-09-21T10:52:43.321788+0000","last_peered":"2024-09-21T10:52:43.321788+0000","last_clean":"2024-09-21T10:52:43.321788+0000","last_became_active":"2024-09-21T10:44:14.398547+0000","last_became_peered":"2024-09-21T10:44:14.398547+0000","last_unstale":"2024-09-21T10:52:43.321788+0000","last_undegraded":"2024-09-21T10:52:43.321788+0000","last_fullsized":"2024-09-21T10:52:43.321788+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:13.008134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2253,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.326743+0000","last_change":"2024-09-21T10:52:25.498554+0000","last_active":"2024-09-21T10:52:44.326743+0000","last_peered":"2024-09-21T10:52:44.326743+0000","last_clean":"2024-09-21T10:52:44.326743+0000","last_became_active":"2024-09-21T10:52:25.498194+0000","last_became_peered":"2024-09-21T10:52:25.498194+0000","last_unstale":"2024-09-21T10:52:44.326743+0000","last_undegraded":"2024-09-21T10:52:44.326743+0000","last_fullsized":"2024-09-21T10:52:44.326743+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:47.115271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.321984+0000","last_change":"2024-09-21T10:52:25.489063+0000","last_active":"2024-09-21T10:52:43.321984+0000","last_peered":"2024-09-21T10:52:43.321984+0000","last_clean":"2024-09-21T10:52:43.321984+0000","last_became_active":"2024-09-21T10:52:25.488129+0000","last_became_peered":"2024-09-21T10:52:25.488129+0000","last_unstale":"2024-09-21T10:52:43.321984+0000","last_undegraded":"2024-09-21T10:52:43.321984+0000","last_fullsized":"2024-09-21T10:52:43.321984+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:36.996829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1431,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.320960+0000","last_change":"2024-09-21T10:49:44.864898+0000","last_active":"2024-09-21T10:52:43.320960+0000","last_peered":"2024-09-21T10:52:43.320960+0000","last_clean":"2024-09-21T10:52:43.320960+0000","last_became_active":"2024-09-21T10:49:44.862691+0000","last_became_peered":"2024-09-21T10:49:44.862691+0000","last_unstale":"2024-09-21T10:52:43.320960+0000","last_undegraded":"2024-09-21T10:52:43.320960+0000","last_fullsized":"2024-09-21T10:52:43.320960+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:57.809113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":122,"reported_epoch":1471,"state":"active+clean","last_fresh":"2024-09-21T10:52:43.321483+0000","last_change":"2024-09-21T10:44:14.418113+0000","last_active":"2024-09-21T10:52:43.321483+0000","last_peered":"2024-09-21T10:52:43.321483+0000","last_clean":"2024-09-21T10:52:43.321483+0000","last_became_active":"2024-09-21T10:44:14.398327+0000","last_became_peered":"2024-09-21T10:44:14.398327+0000","last_unstale":"2024-09-21T10:52:43.321483+0000","last_undegraded":"2024-09-21T10:52:43.321483+0000","last_fullsized":"2024-09-21T10:52:43.321483+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:08.810863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2234,"reported_epoch":1472,"state":"active+clean","last_fresh":"2024-09-21T10:52:44.323556+0000","last_change":"2024-09-21T10:44:14.398584+0000","last_active":"2024-09-21T10:52:44.323556+0000","last_peered":"2024-09-21T10:52:44.323556+0000","last_clean":"2024-09-21T10:52:44.323556+0000","last_became_active":"2024-09-21T10:44:14.398278+0000","last_became_peered":"2024-09-21T10:44:14.398278+0000","last_unstale":"2024-09-21T10:52:44.323556+0000","last_undegraded":"2024-09-21T10:52:44.323556+0000","last_fullsized":"2024-09-21T10:52:44.323556+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:19.681750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":55,"num_pg":16,"stat_sum":{"num_bytes":2830893080,"num_objects":43197,"num_object_clones":0,"num_object_copies":129591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28119,"num_objects_misplaced":9649,"num_objects_unfound":0,"num_objects_dirty":43197,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167481,"num_write_kb":6741633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30983,"num_bytes_recovered":2000355328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11888504832,"data_stored":11888492616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73640,"ondisk_log_size":73640,"up":48,"acting":44,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":429359,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":276,"num_read_kb":598,"num_write":157,"num_write_kb":1757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3655209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":32,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":95,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":11344,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":18188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234705,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3919084,"kb_used_data":3404836,"kb_used_omap":1,"kb_used_meta":514238,"kb_avail":89826068,"statfs":{"total":95995035648,"available":91981893632,"internally_reserved":0,"allocated":3486552064,"data_stored":3481564593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":526580171},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496342,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2427124,"kb_used_data":1752044,"kb_used_omap":1,"kb_used_meta":675006,"kb_avail":91318028,"statfs":{"total":95995035648,"available":93509660672,"internally_reserved":0,"allocated":1794093056,"data_stored":1789098516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":691206602},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757979,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019615,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1579128,"kb_used_data":956592,"kb_used_omap":1,"kb_used_meta":622526,"kb_avail":92166024,"statfs":{"total":95995035648,"available":94378008576,"internally_reserved":0,"allocated":979550208,"data_stored":974551548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":637467078},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369358,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2702192,"kb_used_data":1975904,"kb_used_omap":69,"kb_used_meta":726202,"kb_avail":91042960,"statfs":{"total":95995035648,"available":93227991040,"internally_reserved":0,"allocated":2023325696,"data_stored":2018338690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":743630978},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":2920577761814,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807958044,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1904028,"kb_used_data":1236116,"kb_used_omap":66,"kb_used_meta":667773,"kb_avail":91841124,"statfs":{"total":95995035648,"available":94045310976,"internally_reserved":0,"allocated":1265782784,"data_stored":1260806868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68589,"internal_metadata":683799571},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":2860448219683,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":892538880,"data_stored":892534808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1252327424,"data_stored":1252327424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1992691712,"data_stored":1992687640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":965873664,"data_stored":965869592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1535115264,"data_stored":1535115264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1777926144,"data_stored":1777926144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3472031744,"data_stored":3472031744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:52:52.893 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph status --format=json 2024-09-21T10:52:52.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:52.895 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:52.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:52.996 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:53.041 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:53.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:53.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:53.143 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:53.171 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:53.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:53.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:53.272 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:53.300 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:53.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:53.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:53.402 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:53.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:53.448 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:53.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:53.462 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:52:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:52:53] "GET /metrics HTTP/1.1" 200 36632 "" "Prometheus/2.51.0" 2024-09-21T10:52:53.548 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:53.580 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:53.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:53.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:53.682 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:53.712 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:53.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:53.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:53.814 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:53.857 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:53.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:53.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:53.958 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:54.007 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:54.030 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:54.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:54.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:54.133 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:54.163 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:54.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:54.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:54.265 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:54.301 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:54.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:54.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:54.403 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:54.436 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:54.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:54.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:54.538 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:54.568 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:54.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:54.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:54 smithi160 ceph-mon[101879]: pgmap v2692: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.5 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 603 op/s; 26281/120747 objects degraded (21.765%); 9234/120747 objects misplaced (7.647%); 21 MiB/s, 351 objects/s recovering 2024-09-21T10:52:54.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:54 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 26281/120747 objects degraded (21.765%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:52:54.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:54 smithi070 ceph-mon[107385]: pgmap v2692: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.5 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 603 op/s; 26281/120747 objects degraded (21.765%); 9234/120747 objects misplaced (7.647%); 21 MiB/s, 351 objects/s recovering 2024-09-21T10:52:54.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:54 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 26281/120747 objects degraded (21.765%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:52:54.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:54 smithi070 ceph-mon[112139]: pgmap v2692: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.5 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 603 op/s; 26281/120747 objects degraded (21.765%); 9234/120747 objects misplaced (7.647%); 21 MiB/s, 351 objects/s recovering 2024-09-21T10:52:54.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:54 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 26281/120747 objects degraded (21.765%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:52:54.670 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:54.700 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:54.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:54.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:54.752 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:52:54.752 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T10:52:54.753 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.753 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:172460 55.c e1472/1465 55:32ba02b7:::benchmark_data_smithi070_294394_object67119:head v 1472'10739, mlcod=0'0)", 2024-09-21T10:52:54.753 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:54.657403+0000", 2024-09-21T10:52:54.753 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.014657247999999999, 2024-09-21T10:52:54.753 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.014688302, 2024-09-21T10:52:54.753 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:52:54.753 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:52:54.753 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "started", 2024-09-21T10:52:54.753 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:52:54.753 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.753 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:52:54.753 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.657403+0000", 2024-09-21T10:52:54.753 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:52:54.754 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.754 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.754 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:52:54.754 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.657403+0000", 2024-09-21T10:52:54.754 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:52:54.754 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.754 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.754 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:52:54.754 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.657404+0000", 2024-09-21T10:52:54.754 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1100000000000004e-07 2024-09-21T10:52:54.754 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.754 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.754 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:52:54.755 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.657410+0000", 2024-09-21T10:52:54.755 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8640000000000001e-06 2024-09-21T10:52:54.755 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.755 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.755 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:52:54.755 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.657430+0000", 2024-09-21T10:52:54.755 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9936e-05 2024-09-21T10:52:54.755 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.755 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.755 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:52:54.755 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.657438+0000", 2024-09-21T10:52:54.755 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.4169999999999992e-06 2024-09-21T10:52:54.755 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.755 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.756 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:52:54.756 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.662860+0000", 2024-09-21T10:52:54.756 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.005422339 2024-09-21T10:52:54.756 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.756 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.756 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:52:54.756 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.662916+0000", 2024-09-21T10:52:54.756 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5994e-05 2024-09-21T10:52:54.756 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:52:54.756 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:52:54.756 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:52:54.756 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.756 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.757 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "MOSDPGPush(55.3 1472/1466 [PushOp(55:ccd8c0b9:::benchmark_data_smithi070_294394_object67139:head, version: 1423'4037, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(55:ccd8c0b9:::benchmark_data_smithi070_294394_object67139:head@1423'4037, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T10:52:54.757 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:54.666451+0000", 2024-09-21T10:52:54.757 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0056088580000000004, 2024-09-21T10:52:54.757 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0057341459999999999, 2024-09-21T10:52:54.757 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:52:54.757 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:52:54.757 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "started", 2024-09-21T10:52:54.757 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:52:54.757 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.757 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:52:54.757 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.666451+0000", 2024-09-21T10:52:54.757 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:52:54.757 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.757 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.758 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:52:54.758 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.666451+0000", 2024-09-21T10:52:54.758 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:52:54.758 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.758 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.758 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:52:54.758 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.666452+0000", 2024-09-21T10:52:54.758 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4099999999999997e-07 2024-09-21T10:52:54.758 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.758 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.758 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:52:54.758 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.666482+0000", 2024-09-21T10:52:54.758 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9856e-05 2024-09-21T10:52:54.759 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.759 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.759 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:52:54.759 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.666508+0000", 2024-09-21T10:52:54.759 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.588e-05 2024-09-21T10:52:54.759 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.759 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.759 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:52:54.759 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.666526+0000", 2024-09-21T10:52:54.759 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.774e-05 2024-09-21T10:52:54.759 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.759 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.759 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:52:54.759 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.666652+0000", 2024-09-21T10:52:54.760 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000125888 2024-09-21T10:52:54.760 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.760 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.760 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:52:54.760 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.666671+0000", 2024-09-21T10:52:54.760 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9439000000000001e-05 2024-09-21T10:52:54.760 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:52:54.760 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:52:54.760 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:52:54.760 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.760 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.760 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:172488 55.5 e1472/1465 55:a633ead1:::benchmark_data_smithi070_294394_object67147:head v 1472'10777, mlcod=1472'10774)", 2024-09-21T10:52:54.760 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:54.669471+0000", 2024-09-21T10:52:54.760 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0025891809999999999, 2024-09-21T10:52:54.761 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0027724020000000002, 2024-09-21T10:52:54.761 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:52:54.761 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:52:54.761 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "started", 2024-09-21T10:52:54.761 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:52:54.761 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.761 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:52:54.761 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.669471+0000", 2024-09-21T10:52:54.761 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:52:54.761 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.761 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.761 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:52:54.761 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.669471+0000", 2024-09-21T10:52:54.761 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:52:54.762 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.762 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.762 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:52:54.762 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.669472+0000", 2024-09-21T10:52:54.762 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0999999999999997e-07 2024-09-21T10:52:54.762 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.762 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.762 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:52:54.762 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.669478+0000", 2024-09-21T10:52:54.762 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9399999999999999e-06 2024-09-21T10:52:54.762 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.762 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.762 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:52:54.762 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.669488+0000", 2024-09-21T10:52:54.763 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0599e-05 2024-09-21T10:52:54.763 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.763 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.763 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:52:54.763 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.669495+0000", 2024-09-21T10:52:54.763 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9229999999999996e-06 2024-09-21T10:52:54.763 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.763 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.763 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:52:54.763 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.669554+0000", 2024-09-21T10:52:54.763 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9097e-05 2024-09-21T10:52:54.763 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.763 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.764 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:52:54.764 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.669577+0000", 2024-09-21T10:52:54.764 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2357e-05 2024-09-21T10:52:54.764 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:52:54.764 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:52:54.764 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:52:54.764 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.764 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.764 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "MOSDPGPush(55.f 1472/1448 [PushOp(55:f92aee44:::benchmark_data_smithi070_294394_object67149:head, version: 1423'4207, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(55:f92aee44:::benchmark_data_smithi070_294394_object67149:head@1423'4207, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T10:52:54.764 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:54.672015+0000", 2024-09-21T10:52:54.764 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 4.4981999999999997e-05, 2024-09-21T10:52:54.764 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000297285, 2024-09-21T10:52:54.764 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:52:54.764 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:52:54.765 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "started", 2024-09-21T10:52:54.765 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:52:54.765 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.765 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:52:54.765 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.672015+0000", 2024-09-21T10:52:54.765 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:52:54.765 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.765 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.765 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:52:54.765 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.672015+0000", 2024-09-21T10:52:54.765 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:52:54.765 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.765 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.765 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:52:54.766 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.672016+0000", 2024-09-21T10:52:54.766 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7199999999999997e-07 2024-09-21T10:52:54.766 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.766 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.766 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:52:54.766 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.672066+0000", 2024-09-21T10:52:54.766 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0068999999999998e-05 2024-09-21T10:52:54.766 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.766 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.766 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:52:54.766 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.672091+0000", 2024-09-21T10:52:54.766 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5055e-05 2024-09-21T10:52:54.766 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.766 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.767 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:52:54.767 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.672110+0000", 2024-09-21T10:52:54.767 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8295e-05 2024-09-21T10:52:54.767 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.767 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.767 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:52:54.767 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.672227+0000", 2024-09-21T10:52:54.767 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000117482 2024-09-21T10:52:54.767 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:52:54.767 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:52:54.767 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:52:54.767 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:54.672243+0000", 2024-09-21T10:52:54.767 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6024000000000001e-05 2024-09-21T10:52:54.767 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:52:54.768 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:52:54.768 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:52:54.768 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:52:54.768 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-09-21T10:52:54.768 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 4 2024-09-21T10:52:54.768 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:52:54.802 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:54.845 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:54.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:54.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:54.946 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:54.980 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:54.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:54.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:55.082 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:55.124 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:55.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:55.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:55.226 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:55.257 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:55.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:55.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:55.359 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:55.422 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:55.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:55.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:55.524 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:55.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:55.562 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:55.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:55.663 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:55.693 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:55.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:55.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:55.795 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:55.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:55.827 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:55.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:55.928 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:55.959 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:55.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:55.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:56.002 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:52:56.061 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:56.107 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:56.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:56.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:56.209 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:56.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:56.245 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:56.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:56.346 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:56.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:56.378 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:56.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:56.479 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:56.523 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:56.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:56.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:56.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:56 smithi160 ceph-mon[101879]: pgmap v2693: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.4 GiB data, 12 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 567 op/s; 25841/120090 objects degraded (21.518%); 9343/120090 objects misplaced (7.780%); 20 MiB/s, 339 objects/s recovering 2024-09-21T10:52:56.625 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:56.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:56 smithi070 ceph-mon[107385]: pgmap v2693: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.4 GiB data, 12 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 567 op/s; 25841/120090 objects degraded (21.518%); 9343/120090 objects misplaced (7.780%); 20 MiB/s, 339 objects/s recovering 2024-09-21T10:52:56.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:56 smithi070 ceph-mon[112139]: pgmap v2693: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.4 GiB data, 12 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 567 op/s; 25841/120090 objects degraded (21.518%); 9343/120090 objects misplaced (7.780%); 20 MiB/s, 339 objects/s recovering 2024-09-21T10:52:56.655 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:56.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:56.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:56.757 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:56.796 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:56.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:56.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:56.898 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:56.935 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:56.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:56.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:57.037 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:57.084 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:57.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:57.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:57.186 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:52:57.223 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:57.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:57.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:52:57.325 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:57.362 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:57.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:57.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:57.463 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:57.496 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:57.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:57.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:57.597 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:57.636 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:57.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:57.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:57.738 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:57.769 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:57.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:57.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:57.871 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:52:57.910 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:57.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:57.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:52:58.012 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:58.048 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:58.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:58.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:58.150 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:58.195 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:58.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:58.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:58.297 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:58.333 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:58.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:58.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:58.435 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:58.486 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:58.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:58.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:58.537 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:58 smithi070 ceph-mon[107385]: pgmap v2694: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.3 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 666 op/s; 24624/114144 objects degraded (21.573%); 8954/114144 objects misplaced (7.844%); 24 MiB/s, 401 objects/s recovering 2024-09-21T10:52:58.537 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:52:58 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 24624/114144 objects degraded (21.573%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:52:58.538 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:58 smithi070 ceph-mon[112139]: pgmap v2694: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.3 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 666 op/s; 24624/114144 objects degraded (21.573%); 8954/114144 objects misplaced (7.844%); 24 MiB/s, 401 objects/s recovering 2024-09-21T10:52:58.538 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:52:58 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 24624/114144 objects degraded (21.573%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:52:58.587 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:52:58.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:58 smithi160 ceph-mon[101879]: pgmap v2694: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.3 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 666 op/s; 24624/114144 objects degraded (21.573%); 8954/114144 objects misplaced (7.844%); 24 MiB/s, 401 objects/s recovering 2024-09-21T10:52:58.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:52:58 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 24624/114144 objects degraded (21.573%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:52:58.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:58.619 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:58.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:52:58.720 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:58.758 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:58.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:58.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:58.860 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:58.893 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:58.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:58.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:58.995 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:52:59.001 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:59.031 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:52:59.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:59.049 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:59.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:52:59.150 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:52:59.183 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:52:59.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:59.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:52:59.285 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:59.317 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:59.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:59.339 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:59.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:59.440 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:52:59.477 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:59.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:59.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:52:59.579 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:59.618 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:59.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:59.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:52:59.720 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:52:59.753 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:59.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:59.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:52:59.855 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:52:59.902 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:52:59.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:52:59.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:00.005 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:00.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:00.047 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:00.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:00.148 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:00.186 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:00.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:00.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:00.288 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:00.327 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:00.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:00.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:00.429 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:00.479 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:00.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:00.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:00.558 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:00 smithi070 ceph-mon[112139]: pgmap v2695: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 591 op/s; 23723/110520 objects degraded (21.465%); 8948/110520 objects misplaced (8.096%); 21 MiB/s, 341 objects/s recovering 2024-09-21T10:53:00.558 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:00 smithi070 ceph-mon[107385]: pgmap v2695: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 591 op/s; 23723/110520 objects degraded (21.465%); 8948/110520 objects misplaced (8.096%); 21 MiB/s, 341 objects/s recovering 2024-09-21T10:53:00.583 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:00 smithi160 ceph-mon[101879]: pgmap v2695: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 591 op/s; 23723/110520 objects degraded (21.465%); 8948/110520 objects misplaced (8.096%); 21 MiB/s, 341 objects/s recovering 2024-09-21T10:53:00.619 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:00.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:00.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:00.720 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:00.755 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:00.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:00.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:00.857 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:00.895 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:00.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:00.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:00.997 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:01.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:01.079 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:01.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:01.180 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:01.221 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:01.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:01.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:01.322 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:01.348 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:01.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:01.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:01.450 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:01.496 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:01.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:01.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:01.598 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:01.649 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:01.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:01.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:01.751 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:01.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:01.792 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:01.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:01.893 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:01.925 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:01.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:01.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:02.027 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:02.067 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:02.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:02.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:02.169 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:02.218 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:02.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:02.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:02.320 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:02.360 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:02.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:02.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:02.366 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T10:53:02.463 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:02.488 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:02.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:02.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:02.590 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:02.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:02 smithi070 ceph-mon[107385]: pgmap v2696: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 531 op/s; 23723/110520 objects degraded (21.465%); 8948/110520 objects misplaced (8.096%); 20 MiB/s, 319 objects/s recovering 2024-09-21T10:53:02.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:02 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/297791040' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:53:02.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:02 smithi070 ceph-mon[112139]: pgmap v2696: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 531 op/s; 23723/110520 objects degraded (21.465%); 8948/110520 objects misplaced (8.096%); 20 MiB/s, 319 objects/s recovering 2024-09-21T10:53:02.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:02 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/297791040' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:53:02.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:02.638 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:02.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:02.739 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:02.769 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:02.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:02.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:02 smithi160 ceph-mon[101879]: pgmap v2696: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 531 op/s; 23723/110520 objects degraded (21.465%); 8948/110520 objects misplaced (8.096%); 20 MiB/s, 319 objects/s recovering 2024-09-21T10:53:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:02 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/297791040' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:53:02.870 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:02.912 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:02.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:02.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:03.013 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:03.039 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:03.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:03.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:03.141 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:03.173 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:03.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:03.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:03.184 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:53:03.184 INFO:teuthology.orchestra.run.smithi070.stdout:{"fsid":"7c68bfbc-77fe-11ef-baf6-efdc52797490","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23723/110520 objects degraded (21.465%), 10 pgs degraded","count":10},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":3000,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":1473,"num_osds":8,"num_up_osds":8,"osd_up_since":1726913345,"num_in_osds":5,"osd_in_since":1726915886,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":86},{"state_name":"active+recovery_wait+degraded+remapped","count":6},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+remapped","count":1},{"state_name":"active+recovery_wait+remapped","count":1}],"num_pgs":99,"num_pools":5,"num_objects":36840,"data_bytes":2413011724,"bytes_used":11922272256,"bytes_avail":468052905984,"bytes_total":479975178240,"degraded_objects":23723,"degraded_total":110520,"degraded_ratio":0.21464893231994212,"misplaced_objects":8948,"misplaced_total":110520,"misplaced_ratio":0.080962721679334063,"recovering_objects_per_sec":319,"recovering_bytes_per_sec":20941592,"recovering_keys_per_sec":0,"num_objects_recovered":3837,"num_bytes_recovered":251461632,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":531},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi070.gansda","status":"up:active","gid":14652}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.70:8443/","prometheus":"http://172.21.15.70:9283/"}},"servicemap":{"epoch":36,"modified":"2024-09-21T10:49:02.825956+0000","services":{}},"progress_events":{}} 2024-09-21T10:53:03.275 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:03.311 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:03.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:03.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:03.413 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:03.453 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:03.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:03.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:03.554 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:03.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:03.590 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:03.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:53:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:53:03] "GET /metrics HTTP/1.1" 200 36632 "" "Prometheus/2.51.0" 2024-09-21T10:53:03.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:03 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:53:03.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:03 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/297791040' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:53:03.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:03 smithi070 ceph-mon[107385]: osdmap e1473: 8 total, 8 up, 5 in 2024-09-21T10:53:03.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:03 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:53:03.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:03 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:53:03.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:03 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/866826879' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:53:03.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:03 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 23723/110520 objects degraded (21.465%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:53:03.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:03 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:53:03.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:03 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/297791040' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:53:03.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:03 smithi070 ceph-mon[112139]: osdmap e1473: 8 total, 8 up, 5 in 2024-09-21T10:53:03.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:03 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:53:03.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:03 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:53:03.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:03 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/866826879' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:53:03.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:03 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 23723/110520 objects degraded (21.465%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:53:03.691 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:03.723 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:03.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:03.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:03.824 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:03 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:53:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:03 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/297791040' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:53:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:03 smithi160 ceph-mon[101879]: osdmap e1473: 8 total, 8 up, 5 in 2024-09-21T10:53:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:03 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:53:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:03 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:53:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:03 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/866826879' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:53:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:03 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 23723/110520 objects degraded (21.465%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:53:03.857 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:03.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:03.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:03.959 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:03.990 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:03.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:03.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:04.092 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:04.113 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:04.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:04.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:04.215 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:04.245 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:04.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:04.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:04.346 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:04.381 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:04.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:04.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:04.483 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:04.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:04.517 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:04.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:04.618 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:04.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:04 smithi070 ceph-mon[112139]: pgmap v2698: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.1 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 636 op/s; 22494/101646 objects degraded (22.130%); 8255/101646 objects misplaced (8.121%); 23 MiB/s, 362 objects/s recovering 2024-09-21T10:53:04.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:04 smithi070 ceph-mon[107385]: pgmap v2698: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.1 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 636 op/s; 22494/101646 objects degraded (22.130%); 8255/101646 objects misplaced (8.121%); 23 MiB/s, 362 objects/s recovering 2024-09-21T10:53:04.654 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:04.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:04.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:04.756 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:04.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:04.787 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:04.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:04.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:04 smithi160 ceph-mon[101879]: pgmap v2698: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.1 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 636 op/s; 22494/101646 objects degraded (22.130%); 8255/101646 objects misplaced (8.121%); 23 MiB/s, 362 objects/s recovering 2024-09-21T10:53:04.889 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:04.920 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:04.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:04.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:05.022 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:05.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:05.074 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:05.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:05.175 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:05.203 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:05.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:05.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:05.305 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:05.335 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:05.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:05.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:05.437 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:05.473 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:05.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:05.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:05.575 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:05.606 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:05.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:05.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:05.708 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:05.738 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:05.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:05.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:05.786 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:53:05.786 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:53:05.786 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:53:05.786 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:53:05.786 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:53:05.786 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:53:05.786 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:53:05.786 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:53:05.822 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:53:05.840 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:53:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:53:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:53:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:53:05.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:05.884 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:05.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:05.985 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:06.021 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:06.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:06.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:06.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:06.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:06.162 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:06.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:06.263 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:06.297 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:06.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:06.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:06.399 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:06.422 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-21T10:53:06.422 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:53:06.428 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:06.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:06.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:06.530 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:06.561 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:06.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:06.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:06.663 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:06.684 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:06 smithi070 ceph-mon[107385]: pgmap v2699: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.0 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 657 op/s; 21627/100344 objects degraded (21.553%); 8466/100344 objects misplaced (8.437%); 24 MiB/s, 383 objects/s recovering 2024-09-21T10:53:06.685 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:06 smithi070 ceph-mon[112139]: pgmap v2699: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.0 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 657 op/s; 21627/100344 objects degraded (21.553%); 8466/100344 objects misplaced (8.437%); 24 MiB/s, 383 objects/s recovering 2024-09-21T10:53:06.692 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:06.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:06.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:06.794 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:06.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:06.833 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:06.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:06.834 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:53:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:06 smithi160 ceph-mon[101879]: pgmap v2699: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.0 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 657 op/s; 21627/100344 objects degraded (21.553%); 8466/100344 objects misplaced (8.437%); 24 MiB/s, 383 objects/s recovering 2024-09-21T10:53:06.934 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:06.965 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:06.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:06.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:07.067 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:07.106 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:07.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:07.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:07.207 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:07.244 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:07.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:07.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:07.346 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:07.379 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:07.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:07.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:07.481 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:07.510 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:07.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:07.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:07.611 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:07.643 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:07.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:07.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:07.673 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:53:07.673 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:53:07.673 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:53:07.673 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:53:07.673 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:53:07.673 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:53:07.673 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:53:07.673 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:53:07.674 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:53:07.674 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:53:07.674 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:53:07.674 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:53:07.674 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:53:07.674 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:53:07.674 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:53:07.674 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:53:07.745 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:07.775 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:07.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:07.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:07 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4126897833' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:53:07.877 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:07.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:07 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4126897833' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:53:07.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:07 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4126897833' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:53:07.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:07.913 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:07.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:08.014 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:08.047 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:08.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:08.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:08.149 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:08.184 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:08.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:08.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:08.285 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:08.321 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:08.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:08.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:08.423 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:08.456 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:08.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:08.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:08.558 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:08.600 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:08.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:08.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:08.702 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:08.739 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:08.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:08.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:08.840 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:08 smithi160 ceph-mon[101879]: pgmap v2700: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.9 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 681 op/s; 20288/93693 objects degraded (21.654%); 8021/93693 objects misplaced (8.561%); 25 MiB/s, 396 objects/s recovering 2024-09-21T10:53:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:08 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 20288/93693 objects degraded (21.654%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:08.876 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:08.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:08.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:08.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:08 smithi070 ceph-mon[112139]: pgmap v2700: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.9 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 681 op/s; 20288/93693 objects degraded (21.654%); 8021/93693 objects misplaced (8.561%); 25 MiB/s, 396 objects/s recovering 2024-09-21T10:53:08.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:08 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 20288/93693 objects degraded (21.654%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:08.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:08 smithi070 ceph-mon[107385]: pgmap v2700: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.9 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 681 op/s; 20288/93693 objects degraded (21.654%); 8021/93693 objects misplaced (8.561%); 25 MiB/s, 396 objects/s recovering 2024-09-21T10:53:08.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:08 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 20288/93693 objects degraded (21.654%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:08.977 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:09.014 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:09.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:09.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:09.116 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:09.154 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:09.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:09.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:09.256 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:09.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:09.290 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:09.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:09.391 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:09.429 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:09.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:09.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:09.531 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:09.570 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:09.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:09.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:09.583 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:53:09.591 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:53:09.672 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:09.702 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:09.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:09.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:09.804 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:09.845 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:09.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:09.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:09.947 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:09.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:09.980 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:09.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:10.081 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:10.114 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:10.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:10.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:10.216 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:10.250 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:10.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:10.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:10.352 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:10.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:10.385 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:10.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:10.486 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:10.523 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:10.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:10.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:10.625 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:10.652 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:10.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:10.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:10.754 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:10.789 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:10.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:10.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:10.820 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:10 smithi070 ceph-mon[107385]: pgmap v2701: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.8 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 687 op/s; 19463/89871 objects degraded (21.657%); 8082/89871 objects misplaced (8.993%); 24 MiB/s, 395 objects/s recovering 2024-09-21T10:53:10.820 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:10 smithi070 ceph-mon[112139]: pgmap v2701: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.8 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 687 op/s; 19463/89871 objects degraded (21.657%); 8082/89871 objects misplaced (8.993%); 24 MiB/s, 395 objects/s recovering 2024-09-21T10:53:10.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:10 smithi160 ceph-mon[101879]: pgmap v2701: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.8 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 687 op/s; 19463/89871 objects degraded (21.657%); 8082/89871 objects misplaced (8.993%); 24 MiB/s, 395 objects/s recovering 2024-09-21T10:53:10.891 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:10.931 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:10.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:10.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:11.032 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:11.064 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:11.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:11.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:11.166 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:11.198 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:11.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:11.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:11.300 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:11.338 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:11.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:11.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:11.439 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:11.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:11.482 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:11.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:11.583 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:11.599 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:53:11.619 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:11.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:11.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:11.720 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:11.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:11.753 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:11.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:11.854 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:11.883 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:11.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:11.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:11.985 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:12.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:12.021 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:12.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:12.122 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:12.159 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:12.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:12.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:12.261 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:12.299 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:12.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:12.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:12.401 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:12.430 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:12.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:12.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:12.531 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:12.565 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:12.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:12.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:12.666 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:12.705 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:12.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:12.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:12.806 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:12.838 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:12.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:12.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:12 smithi160 ceph-mon[101879]: pgmap v2702: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.8 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 687 op/s; 19463/89871 objects degraded (21.657%); 8082/89871 objects misplaced (8.993%); 24 MiB/s, 395 objects/s recovering 2024-09-21T10:53:12.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:12 smithi070 ceph-mon[107385]: pgmap v2702: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.8 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 687 op/s; 19463/89871 objects degraded (21.657%); 8082/89871 objects misplaced (8.993%); 24 MiB/s, 395 objects/s recovering 2024-09-21T10:53:12.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:12 smithi070 ceph-mon[112139]: pgmap v2702: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.8 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 687 op/s; 19463/89871 objects degraded (21.657%); 8082/89871 objects misplaced (8.993%); 24 MiB/s, 395 objects/s recovering 2024-09-21T10:53:12.940 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:12.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:12.995 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:12.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:13.048 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:53:13.048 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:53:13.096 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:13.128 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:13.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:13.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:13.230 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:13.262 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:13.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:13.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:13.366 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:53:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:53:13] "GET /metrics HTTP/1.1" 200 36638 "" "Prometheus/2.51.0" 2024-09-21T10:53:13.368 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:13.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:13.410 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:13.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:13.511 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:13.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:13.546 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:13.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:13.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:13 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 19463/89871 objects degraded (21.657%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:13.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:13 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 19463/89871 objects degraded (21.657%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:13.646 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:13.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:13.684 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:13.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:13.785 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:13.825 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:13.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:13.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:13.926 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:13.945 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:53:13.954 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:13.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:13.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:14.056 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:14.087 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:14.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:14.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:14.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:13 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 19463/89871 objects degraded (21.657%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:14.189 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:14.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:14.224 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:14.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:14.325 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:14.357 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:14.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:14.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:14.459 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:14.487 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:14.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:14.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:14.589 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:14.602 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:14 smithi070 ceph-mon[112139]: pgmap v2703: 99 pgs: 2 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.6 GiB data, 9.8 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 700 op/s; 18378/79425 objects degraded (23.139%); 7360/79425 objects misplaced (9.267%); 20 MiB/s, 360 objects/s recovering 2024-09-21T10:53:14.633 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:14.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:14.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:14.735 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:14.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:14.766 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:14.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:14.867 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:14.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:14 smithi070 ceph-mon[107385]: pgmap v2703: 99 pgs: 2 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.6 GiB data, 9.8 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 700 op/s; 18378/79425 objects degraded (23.139%); 7360/79425 objects misplaced (9.267%); 20 MiB/s, 360 objects/s recovering 2024-09-21T10:53:14.903 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:14.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:14.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:15.004 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:15.043 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:15.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:15.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:15.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:14 smithi160 ceph-mon[101879]: pgmap v2703: 99 pgs: 2 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.6 GiB data, 9.8 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 700 op/s; 18378/79425 objects degraded (23.139%); 7360/79425 objects misplaced (9.267%); 20 MiB/s, 360 objects/s recovering 2024-09-21T10:53:15.144 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:15.178 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:15.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:15.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:15.280 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:15.304 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:15.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:15.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:15.406 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:15.441 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:15.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:15.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:15.543 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:15.577 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:15.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:15.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:15.617 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T10:53:15.679 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:15.712 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:15.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:15.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:15.814 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:15.850 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:15.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:15.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:15.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:15 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4164585910' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:53:15.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:15 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:53:15.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:15 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4164585910' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:53:15.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:15 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:53:15.951 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:15.983 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:15.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:15.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:16.085 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:16.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:15 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4164585910' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:53:16.093 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:15 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:53:16.121 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:16.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:16.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:16.224 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:16.245 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:16.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:16.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:16.347 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:16.385 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:16.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:16.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:16.487 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:16.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:16.523 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:16.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:16.624 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:16.650 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:16.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:16.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:16.751 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:16.787 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:16.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:16.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:16.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:16 smithi070 ceph-mon[107385]: pgmap v2704: 99 pgs: 2 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.6 GiB data, 9.7 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 658 op/s; 17629/77922 objects degraded (22.624%); 7608/77922 objects misplaced (9.764%); 21 MiB/s, 359 objects/s recovering 2024-09-21T10:53:16.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:16 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:53:16.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:16 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:53:16.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:16 smithi070 ceph-mon[107385]: osdmap e1474: 8 total, 8 up, 5 in 2024-09-21T10:53:16.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:16 smithi070 ceph-mon[112139]: pgmap v2704: 99 pgs: 2 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.6 GiB data, 9.7 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 658 op/s; 17629/77922 objects degraded (22.624%); 7608/77922 objects misplaced (9.764%); 21 MiB/s, 359 objects/s recovering 2024-09-21T10:53:16.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:16 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:53:16.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:16 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:53:16.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:16 smithi070 ceph-mon[112139]: osdmap e1474: 8 total, 8 up, 5 in 2024-09-21T10:53:16.889 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:16.922 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:16.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:16.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:17.024 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:17.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:17.047 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:17.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:17.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:16 smithi160 ceph-mon[101879]: pgmap v2704: 99 pgs: 2 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.6 GiB data, 9.7 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 658 op/s; 17629/77922 objects degraded (22.624%); 7608/77922 objects misplaced (9.764%); 21 MiB/s, 359 objects/s recovering 2024-09-21T10:53:17.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:16 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:53:17.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:16 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:53:17.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:16 smithi160 ceph-mon[101879]: osdmap e1474: 8 total, 8 up, 5 in 2024-09-21T10:53:17.149 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:17.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:17.186 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:17.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:17.287 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:17.322 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:17.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:17.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:17.423 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:17.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:17.459 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:17.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:17.560 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:17.597 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:17.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:17.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:17.698 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:17.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:17.737 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:17.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:17.838 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:17.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:17.864 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:17.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:17.964 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:17.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:18.000 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:18.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:18.100 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:18.136 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:18.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:18.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:18.238 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:18.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:18.285 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:18.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:18.386 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:18.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:18.435 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:18.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:18.536 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:18.571 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:18.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:18.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:18.673 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:18.709 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:18.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:18.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:18.811 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:18.846 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:18.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:18.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:18.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:18 smithi070 ceph-mon[107385]: pgmap v2706: 99 pgs: 2 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.4 GiB data, 9.3 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 761 op/s; 16664/70833 objects degraded (23.526%); 6912/70833 objects misplaced (9.758%); 20 MiB/s, 350 objects/s recovering 2024-09-21T10:53:18.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:18 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 16664/70833 objects degraded (23.526%), 9 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:18.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:18 smithi070 ceph-mon[112139]: pgmap v2706: 99 pgs: 2 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.4 GiB data, 9.3 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 761 op/s; 16664/70833 objects degraded (23.526%); 6912/70833 objects misplaced (9.758%); 20 MiB/s, 350 objects/s recovering 2024-09-21T10:53:18.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:18 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 16664/70833 objects degraded (23.526%), 9 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:18.948 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:18.977 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:18.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:18.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:19.078 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:19.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:18 smithi160 ceph-mon[101879]: pgmap v2706: 99 pgs: 2 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.4 GiB data, 9.3 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 761 op/s; 16664/70833 objects degraded (23.526%); 6912/70833 objects misplaced (9.758%); 20 MiB/s, 350 objects/s recovering 2024-09-21T10:53:19.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:18 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 16664/70833 objects degraded (23.526%), 9 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:19.116 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:19.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:19.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:19.218 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:19.252 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:19.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:19.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:19.354 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:19.362 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:53:19.371 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:53:19.382 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:19.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:19.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:19.484 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:19.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:19.530 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:19.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:19.631 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:19.663 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:19.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:19.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:19.762 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:53:19.763 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T10:53:19.765 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:19.792 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:19.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:19.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:19.893 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:19.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:19.927 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:19.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:20.028 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:20.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:20.069 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:20.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:20.170 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:20.203 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:20.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:20.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:20.305 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:20.343 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:20.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:20.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:20.445 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:20.481 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:20.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:20.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:20.553 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":2707,"stamp":"2024-09-21T10:53:18.932935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1447142156,"num_objects":22102,"num_object_clones":0,"num_object_copies":66306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15970,"num_objects_misplaced":7139,"num_objects_unfound":0,"num_objects_dirty":22102,"num_whiteouts":0,"num_read":277,"num_read_kb":599,"num_write":188823,"num_write_kb":6743423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42465,"num_bytes_recovered":2705781426,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75859,"ondisk_log_size":75859,"up":297,"acting":292,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":328,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":9477832,"kb_used_data":6179604,"kb_used_omap":141,"kb_used_meta":3297586,"kb_avail":459247928,"statfs":{"total":479975178240,"available":470269878272,"internally_reserved":0,"allocated":6327914496,"data_stored":6302897653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":144912,"internal_metadata":3376728560},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-514785280,"num_objects":-7855,"num_object_clones":0,"num_object_copies":-23565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3493,"num_objects_misplaced":-943,"num_objects_unfound":0,"num_objects_dirty":-7855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7855,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3620,"num_bytes_recovered":197722112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003732"},"pg_stats":[{"pgid":"55.f","version":"1474'11790","reported_seq":19070,"reported_epoch":1474,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T10:53:16.724735+0000","last_change":"2024-09-21T10:52:25.539794+0000","last_active":"2024-09-21T10:53:16.724735+0000","last_peered":"2024-09-21T10:53:16.724735+0000","last_clean":"2024-09-21T10:49:43.822243+0000","last_became_active":"2024-09-21T10:51:28.555652+0000","last_became_peered":"2024-09-21T10:51:28.555652+0000","last_unstale":"2024-09-21T10:53:16.724735+0000","last_undegraded":"2024-09-21T10:52:25.537884+0000","last_fullsized":"2024-09-21T10:53:16.724735+0000","mapping_epoch":1448,"log_start":"1445'7640","ondisk_log_start":"1445'7640","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4150,"log_dups_size":0,"ondisk_log_size":4150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:30:34.790715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87949312,"num_objects":1342,"num_object_clones":0,"num_object_copies":4026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1153,"num_objects_misplaced":189,"num_objects_unfound":0,"num_objects_dirty":1342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11790,"num_write_kb":420224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3592,"num_bytes_recovered":235405312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,0],"avail_no_missing":["6","5","7"],"object_location_counts":[{"shards":"0,6,7","objects":1342}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.e","version":"1474'11603","reported_seq":11750,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:16.537404+0000","last_change":"2024-09-21T10:52:16.415783+0000","last_active":"2024-09-21T10:53:16.537404+0000","last_peered":"2024-09-21T10:53:16.537404+0000","last_clean":"2024-09-21T10:53:16.537404+0000","last_became_active":"2024-09-21T10:52:16.397921+0000","last_became_peered":"2024-09-21T10:52:16.397921+0000","last_unstale":"2024-09-21T10:53:16.537404+0000","last_undegraded":"2024-09-21T10:53:16.537404+0000","last_fullsized":"2024-09-21T10:53:16.537404+0000","mapping_epoch":1463,"log_start":"1444'6856","ondisk_log_start":"1444'6856","created":1411,"last_epoch_clean":1464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:13.753306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92340224,"num_objects":1409,"num_object_clones":0,"num_object_copies":4227,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11603,"num_write_kb":416384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.d","version":"1474'11959","reported_seq":19238,"reported_epoch":1474,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:53:16.134266+0000","last_change":"2024-09-21T10:52:26.803434+0000","last_active":"2024-09-21T10:53:16.134266+0000","last_peered":"2024-09-21T10:53:16.134266+0000","last_clean":"2024-09-21T10:50:20.259057+0000","last_became_active":"2024-09-21T10:52:26.653674+0000","last_became_peered":"2024-09-21T10:52:26.653674+0000","last_unstale":"2024-09-21T10:53:16.134266+0000","last_undegraded":"2024-09-21T10:52:26.638016+0000","last_fullsized":"2024-09-21T10:52:26.462648+0000","mapping_epoch":1466,"log_start":"1444'6926","ondisk_log_start":"1444'6926","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":5033,"log_dups_size":0,"ondisk_log_size":5033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:31:18.206348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92930072,"num_objects":1419,"num_object_clones":0,"num_object_copies":4257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1419,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11959,"num_write_kb":428033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3592,"num_bytes_recovered":235077632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[4,3],"avail_no_missing":["4","0","6"],"object_location_counts":[{"shards":"0,3,4","objects":268},{"shards":"0,4,7","objects":1151}],"blocked_by":[],"up_primary":7,"acting_primary":4,"purged_snaps":[]},{"pgid":"55.c","version":"1474'11742","reported_seq":19067,"reported_epoch":1474,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T10:53:16.531544+0000","last_change":"2024-09-21T10:52:25.708024+0000","last_active":"2024-09-21T10:53:16.531544+0000","last_peered":"2024-09-21T10:53:16.531544+0000","last_clean":"2024-09-21T10:50:03.007234+0000","last_became_active":"2024-09-21T10:52:25.621852+0000","last_became_peered":"2024-09-21T10:52:25.621852+0000","last_unstale":"2024-09-21T10:53:16.531544+0000","last_undegraded":"2024-09-21T10:52:25.613400+0000","last_fullsized":"2024-09-21T10:53:16.531544+0000","mapping_epoch":1465,"log_start":"1444'6986","ondisk_log_start":"1444'6986","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4756,"log_dups_size":0,"ondisk_log_size":4756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91095040,"num_objects":1390,"num_object_clones":0,"num_object_copies":4170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11742,"num_write_kb":420224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3588,"num_bytes_recovered":235077632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,0],"avail_no_missing":["3","1","4"],"object_location_counts":[{"shards":"0,3,4","objects":881},{"shards":"3,4","objects":509}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.b","version":"1474'11957","reported_seq":12030,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:16.747048+0000","last_change":"2024-09-21T10:48:52.934001+0000","last_active":"2024-09-21T10:53:16.747048+0000","last_peered":"2024-09-21T10:53:16.747048+0000","last_clean":"2024-09-21T10:53:16.747048+0000","last_became_active":"2024-09-21T10:48:52.933674+0000","last_became_peered":"2024-09-21T10:48:52.933674+0000","last_unstale":"2024-09-21T10:53:16.747048+0000","last_undegraded":"2024-09-21T10:53:16.747048+0000","last_fullsized":"2024-09-21T10:53:16.747048+0000","mapping_epoch":1411,"log_start":"1445'7806","ondisk_log_start":"1445'7806","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4151,"log_dups_size":0,"ondisk_log_size":4151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:13.178680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95879168,"num_objects":1463,"num_object_clones":0,"num_object_copies":4389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11957,"num_write_kb":429440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.a","version":"1474'11754","reported_seq":15877,"reported_epoch":1474,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T10:53:16.529631+0000","last_change":"2024-09-21T10:52:26.731569+0000","last_active":"2024-09-21T10:53:16.529631+0000","last_peered":"2024-09-21T10:53:16.529631+0000","last_clean":"2024-09-21T10:50:03.021536+0000","last_became_active":"2024-09-21T10:52:26.699954+0000","last_became_peered":"2024-09-21T10:52:26.699954+0000","last_unstale":"2024-09-21T10:53:16.529631+0000","last_undegraded":"2024-09-21T10:52:26.664328+0000","last_fullsized":"2024-09-21T10:53:16.529631+0000","mapping_epoch":1466,"log_start":"1444'7051","ondisk_log_start":"1444'7051","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4703,"log_dups_size":0,"ondisk_log_size":4703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90439680,"num_objects":1380,"num_object_clones":0,"num_object_copies":4140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1380,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":420288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1987,"num_bytes_recovered":130220032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,4,6],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":516},{"shards":"3,4,6","objects":864}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.9","version":"1474'11742","reported_seq":21301,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:16.509300+0000","last_change":"2024-09-21T10:52:44.354764+0000","last_active":"2024-09-21T10:53:16.509300+0000","last_peered":"2024-09-21T10:53:16.509300+0000","last_clean":"2024-09-21T10:53:16.509300+0000","last_became_active":"2024-09-21T10:52:44.325813+0000","last_became_peered":"2024-09-21T10:52:44.325813+0000","last_unstale":"2024-09-21T10:53:16.509300+0000","last_undegraded":"2024-09-21T10:53:16.509300+0000","last_fullsized":"2024-09-21T10:53:16.509300+0000","mapping_epoch":1471,"log_start":"1444'6969","ondisk_log_start":"1444'6969","created":1411,"last_epoch_clean":1472,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4773,"log_dups_size":0,"ondisk_log_size":4773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:58.318369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93847552,"num_objects":1432,"num_object_clones":0,"num_object_copies":4296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11742,"num_write_kb":421568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4701,"num_bytes_recovered":297533440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.8","version":"1474'11588","reported_seq":20987,"reported_epoch":1474,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-21T10:53:16.739503+0000","last_change":"2024-09-21T10:53:07.722709+0000","last_active":"2024-09-21T10:53:16.739503+0000","last_peered":"2024-09-21T10:53:16.739503+0000","last_clean":"2024-09-21T10:49:43.813262+0000","last_became_active":"2024-09-21T10:51:28.546990+0000","last_became_peered":"2024-09-21T10:51:28.546990+0000","last_unstale":"2024-09-21T10:53:16.739503+0000","last_undegraded":"2024-09-21T10:53:16.739503+0000","last_fullsized":"2024-09-21T10:53:16.739503+0000","mapping_epoch":1448,"log_start":"1445'7434","ondisk_log_start":"1445'7434","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4154,"log_dups_size":0,"ondisk_log_size":4154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:51:45.867981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94765056,"num_objects":1446,"num_object_clones":0,"num_object_copies":4338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1446,"num_objects_unfound":0,"num_objects_dirty":1446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11588,"num_write_kb":417088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4638,"num_bytes_recovered":303955968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,0],"avail_no_missing":["6","0","3","5"],"object_location_counts":[{"shards":"0,3,6","objects":1446}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.7","version":"1474'11871","reported_seq":11945,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:16.934366+0000","last_change":"2024-09-21T10:48:52.933230+0000","last_active":"2024-09-21T10:53:16.934366+0000","last_peered":"2024-09-21T10:53:16.934366+0000","last_clean":"2024-09-21T10:53:16.934366+0000","last_became_active":"2024-09-21T10:48:52.932944+0000","last_became_peered":"2024-09-21T10:48:52.932944+0000","last_unstale":"2024-09-21T10:53:16.934366+0000","last_undegraded":"2024-09-21T10:53:16.934366+0000","last_fullsized":"2024-09-21T10:53:16.934366+0000","mapping_epoch":1411,"log_start":"1444'6828","ondisk_log_start":"1444'6828","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":5043,"log_dups_size":0,"ondisk_log_size":5043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:16:43.711173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92864512,"num_objects":1417,"num_object_clones":0,"num_object_copies":4251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11871,"num_write_kb":425216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"55.6","version":"1474'11976","reported_seq":25971,"reported_epoch":1474,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-21T10:53:16.971173+0000","last_change":"2024-09-21T10:52:25.578778+0000","last_active":"2024-09-21T10:53:16.971173+0000","last_peered":"2024-09-21T10:53:16.971173+0000","last_clean":"2024-09-21T10:49:43.854914+0000","last_became_active":"2024-09-21T10:52:03.460583+0000","last_became_peered":"2024-09-21T10:52:03.460583+0000","last_unstale":"2024-09-21T10:53:16.971173+0000","last_undegraded":"2024-09-21T10:53:16.971173+0000","last_fullsized":"2024-09-21T10:52:03.308381+0000","mapping_epoch":1455,"log_start":"1444'6926","ondisk_log_start":"1444'6926","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":5050,"log_dups_size":0,"ondisk_log_size":5050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:40.566532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88342528,"num_objects":1348,"num_object_clones":0,"num_object_copies":4044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11976,"num_write_kb":426368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5825,"num_bytes_recovered":334823424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,1],"avail_no_missing":["7","1","5"],"object_location_counts":[{"shards":"1,7","objects":1348}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"55.5","version":"1474'11771","reported_seq":11894,"reported_epoch":1474,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-21T10:53:16.775044+0000","last_change":"2024-09-21T10:52:25.760897+0000","last_active":"2024-09-21T10:53:16.775044+0000","last_peered":"2024-09-21T10:53:16.775044+0000","last_clean":"2024-09-21T10:50:02.995767+0000","last_became_active":"2024-09-21T10:52:25.564312+0000","last_became_peered":"2024-09-21T10:52:25.564312+0000","last_unstale":"2024-09-21T10:53:16.775044+0000","last_undegraded":"2024-09-21T10:53:16.775044+0000","last_fullsized":"2024-09-21T10:53:16.775044+0000","mapping_epoch":1465,"log_start":"1446'7714","ondisk_log_start":"1446'7714","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4057,"log_dups_size":0,"ondisk_log_size":4057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:26:53.597959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89587712,"num_objects":1367,"num_object_clones":0,"num_object_copies":4101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1367,"num_objects_unfound":0,"num_objects_dirty":1367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11771,"num_write_kb":420416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,0],"avail_no_missing":["6","0","1","3","4"],"object_location_counts":[{"shards":"0,1,6","objects":1367}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.4","version":"1474'11928","reported_seq":12002,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:16.951339+0000","last_change":"2024-09-21T10:48:52.934634+0000","last_active":"2024-09-21T10:53:16.951339+0000","last_peered":"2024-09-21T10:53:16.951339+0000","last_clean":"2024-09-21T10:53:16.951339+0000","last_became_active":"2024-09-21T10:48:52.934368+0000","last_became_peered":"2024-09-21T10:48:52.934368+0000","last_unstale":"2024-09-21T10:53:16.951339+0000","last_undegraded":"2024-09-21T10:53:16.951339+0000","last_fullsized":"2024-09-21T10:53:16.951339+0000","mapping_epoch":1411,"log_start":"1444'6845","ondisk_log_start":"1444'6845","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":5083,"log_dups_size":0,"ondisk_log_size":5083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:27:09.216667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91226112,"num_objects":1392,"num_object_clones":0,"num_object_copies":4176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11928,"num_write_kb":426240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"55.3","version":"1474'11496","reported_seq":26497,"reported_epoch":1474,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:53:18.705326+0000","last_change":"2024-09-21T10:52:26.628130+0000","last_active":"2024-09-21T10:53:18.705326+0000","last_peered":"2024-09-21T10:53:18.705326+0000","last_clean":"2024-09-21T10:49:43.839828+0000","last_became_active":"2024-09-21T10:52:26.596220+0000","last_became_peered":"2024-09-21T10:52:26.596220+0000","last_unstale":"2024-09-21T10:53:18.705326+0000","last_undegraded":"2024-09-21T10:52:26.536372+0000","last_fullsized":"2024-09-21T10:52:26.462998+0000","mapping_epoch":1466,"log_start":"1445'7143","ondisk_log_start":"1445'7143","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4353,"log_dups_size":0,"ondisk_log_size":4353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81657856,"num_objects":1246,"num_object_clones":0,"num_object_copies":3738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":416,"num_objects_misplaced":830,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11496,"num_write_kb":407744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5128,"num_bytes_recovered":335937536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,0],"avail_no_missing":["1","3","5","6"],"object_location_counts":[{"shards":"0,1","objects":830},{"shards":"1,3","objects":416}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"55.2","version":"1474'11717","reported_seq":25293,"reported_epoch":1474,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T10:53:18.706098+0000","last_change":"2024-09-21T10:52:26.812464+0000","last_active":"2024-09-21T10:53:18.706098+0000","last_peered":"2024-09-21T10:53:18.706098+0000","last_clean":"2024-09-21T10:50:02.974304+0000","last_became_active":"2024-09-21T10:52:26.694338+0000","last_became_peered":"2024-09-21T10:52:26.694338+0000","last_unstale":"2024-09-21T10:53:18.706098+0000","last_undegraded":"2024-09-21T10:52:26.678206+0000","last_fullsized":"2024-09-21T10:53:18.706098+0000","mapping_epoch":1466,"log_start":"1445'7406","ondisk_log_start":"1445'7406","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4311,"log_dups_size":0,"ondisk_log_size":4311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77660160,"num_objects":1185,"num_object_clones":0,"num_object_copies":3555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1563,"num_objects_misplaced":3307,"num_objects_unfound":0,"num_objects_dirty":1185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11717,"num_write_kb":412864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5270,"num_bytes_recovered":345243648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,4],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"0,1,4","objects":806},{"shards":"1,4,6","objects":1},{"shards":"1,6","objects":378}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"55.1","version":"1474'11858","reported_seq":13149,"reported_epoch":1474,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:53:16.961216+0000","last_change":"2024-09-21T10:52:15.413755+0000","last_active":"2024-09-21T10:53:16.961216+0000","last_peered":"2024-09-21T10:53:16.961216+0000","last_clean":"2024-09-21T10:50:20.316846+0000","last_became_active":"2024-09-21T10:52:03.442393+0000","last_became_peered":"2024-09-21T10:52:03.442393+0000","last_unstale":"2024-09-21T10:53:16.961216+0000","last_undegraded":"2024-09-21T10:52:15.413422+0000","last_fullsized":"2024-09-21T10:52:03.308195+0000","mapping_epoch":1455,"log_start":"1444'6812","ondisk_log_start":"1444'6812","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":5046,"log_dups_size":0,"ondisk_log_size":5046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:45:04.793364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94371840,"num_objects":1440,"num_object_clones":0,"num_object_copies":4320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11858,"num_write_kb":425536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":596,"num_bytes_recovered":20643840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":1440}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"55.0","version":"1474'11850","reported_seq":18964,"reported_epoch":1474,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T10:53:16.524231+0000","last_change":"2024-09-21T10:52:26.707855+0000","last_active":"2024-09-21T10:53:16.524231+0000","last_peered":"2024-09-21T10:53:16.524231+0000","last_clean":"2024-09-21T10:50:03.041207+0000","last_became_active":"2024-09-21T10:52:26.645887+0000","last_became_peered":"2024-09-21T10:52:26.645887+0000","last_unstale":"2024-09-21T10:53:16.524231+0000","last_undegraded":"2024-09-21T10:52:26.635700+0000","last_fullsized":"2024-09-21T10:53:16.524231+0000","mapping_epoch":1466,"log_start":"1444'7099","ondisk_log_start":"1444'7099","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4751,"log_dups_size":0,"ondisk_log_size":4751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91750400,"num_objects":1400,"num_object_clones":0,"num_object_copies":4200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11850,"num_write_kb":424000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3482,"num_bytes_recovered":228196352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,1],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"1,3,6","objects":859},{"shards":"3,6","objects":541}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":1964,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.375991+0000","last_change":"2024-09-21T10:52:03.323899+0000","last_active":"2024-09-21T10:53:02.375991+0000","last_peered":"2024-09-21T10:53:02.375991+0000","last_clean":"2024-09-21T10:53:02.375991+0000","last_became_active":"2024-09-21T10:52:03.314626+0000","last_became_peered":"2024-09-21T10:52:03.314626+0000","last_unstale":"2024-09-21T10:53:02.375991+0000","last_undegraded":"2024-09-21T10:53:02.375991+0000","last_fullsized":"2024-09-21T10:53:02.375991+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:13.939806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1464,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.376465+0000","last_change":"2024-09-21T10:52:15.396755+0000","last_active":"2024-09-21T10:53:02.376465+0000","last_peered":"2024-09-21T10:53:02.376465+0000","last_clean":"2024-09-21T10:53:02.376465+0000","last_became_active":"2024-09-21T10:52:15.396417+0000","last_became_peered":"2024-09-21T10:52:15.396417+0000","last_unstale":"2024-09-21T10:53:02.376465+0000","last_undegraded":"2024-09-21T10:53:02.376465+0000","last_fullsized":"2024-09-21T10:53:02.376465+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:25.005574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":2258,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.628860+0000","last_change":"2024-09-21T10:52:15.400227+0000","last_active":"2024-09-21T10:53:15.628860+0000","last_peered":"2024-09-21T10:53:15.628860+0000","last_clean":"2024-09-21T10:53:15.628860+0000","last_became_active":"2024-09-21T10:52:15.399888+0000","last_became_peered":"2024-09-21T10:52:15.399888+0000","last_unstale":"2024-09-21T10:53:15.628860+0000","last_undegraded":"2024-09-21T10:53:15.628860+0000","last_fullsized":"2024-09-21T10:53:15.628860+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:42:52.565184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2274,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.626706+0000","last_change":"2024-09-21T10:51:27.458216+0000","last_active":"2024-09-21T10:53:15.626706+0000","last_peered":"2024-09-21T10:53:15.626706+0000","last_clean":"2024-09-21T10:53:15.626706+0000","last_became_active":"2024-09-21T10:51:27.457948+0000","last_became_peered":"2024-09-21T10:51:27.457948+0000","last_unstale":"2024-09-21T10:53:15.626706+0000","last_undegraded":"2024-09-21T10:53:15.626706+0000","last_fullsized":"2024-09-21T10:53:15.626706+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:25:29.584079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":102,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.379707+0000","last_change":"2024-09-21T10:49:44.878242+0000","last_active":"2024-09-21T10:53:02.379707+0000","last_peered":"2024-09-21T10:53:02.379707+0000","last_clean":"2024-09-21T10:53:02.379707+0000","last_became_active":"2024-09-21T10:49:44.866578+0000","last_became_peered":"2024-09-21T10:49:44.866578+0000","last_unstale":"2024-09-21T10:53:02.379707+0000","last_undegraded":"2024-09-21T10:53:02.379707+0000","last_fullsized":"2024-09-21T10:53:02.379707+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:17:24.145551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1469,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.377717+0000","last_change":"2024-09-21T10:51:27.460539+0000","last_active":"2024-09-21T10:53:02.377717+0000","last_peered":"2024-09-21T10:53:02.377717+0000","last_clean":"2024-09-21T10:53:02.377717+0000","last_became_active":"2024-09-21T10:51:27.460211+0000","last_became_peered":"2024-09-21T10:51:27.460211+0000","last_unstale":"2024-09-21T10:53:02.377717+0000","last_undegraded":"2024-09-21T10:53:02.377717+0000","last_fullsized":"2024-09-21T10:53:02.377717+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:14:15.700998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":141,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.377020+0000","last_change":"2024-09-21T10:52:15.405886+0000","last_active":"2024-09-21T10:53:02.377020+0000","last_peered":"2024-09-21T10:53:02.377020+0000","last_clean":"2024-09-21T10:53:02.377020+0000","last_became_active":"2024-09-21T10:52:15.397317+0000","last_became_peered":"2024-09-21T10:52:15.397317+0000","last_unstale":"2024-09-21T10:53:02.377020+0000","last_undegraded":"2024-09-21T10:53:02.377020+0000","last_fullsized":"2024-09-21T10:53:02.377020+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:43:48.813861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1460,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.377014+0000","last_change":"2024-09-21T10:09:06.560175+0000","last_active":"2024-09-21T10:53:02.377014+0000","last_peered":"2024-09-21T10:53:02.377014+0000","last_clean":"2024-09-21T10:53:02.377014+0000","last_became_active":"2024-09-21T10:09:06.558584+0000","last_became_peered":"2024-09-21T10:09:06.558584+0000","last_unstale":"2024-09-21T10:53:02.377014+0000","last_undegraded":"2024-09-21T10:53:02.377014+0000","last_fullsized":"2024-09-21T10:53:02.377014+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:02.918339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":50,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.376947+0000","last_change":"2024-09-21T10:51:27.437665+0000","last_active":"2024-09-21T10:53:02.376947+0000","last_peered":"2024-09-21T10:53:02.376947+0000","last_clean":"2024-09-21T10:53:02.376947+0000","last_became_active":"2024-09-21T10:51:27.436481+0000","last_became_peered":"2024-09-21T10:51:27.436481+0000","last_unstale":"2024-09-21T10:53:02.376947+0000","last_undegraded":"2024-09-21T10:53:02.376947+0000","last_fullsized":"2024-09-21T10:53:02.376947+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:07:50.267297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1029,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.620603+0000","last_change":"2024-09-21T10:51:27.459033+0000","last_active":"2024-09-21T10:53:15.620603+0000","last_peered":"2024-09-21T10:53:15.620603+0000","last_clean":"2024-09-21T10:53:15.620603+0000","last_became_active":"2024-09-21T10:51:27.458728+0000","last_became_peered":"2024-09-21T10:51:27.458728+0000","last_unstale":"2024-09-21T10:53:15.620603+0000","last_undegraded":"2024-09-21T10:53:15.620603+0000","last_fullsized":"2024-09-21T10:53:15.620603+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:20.133032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":1489,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.377737+0000","last_change":"2024-09-21T10:52:25.473550+0000","last_active":"2024-09-21T10:53:02.377737+0000","last_peered":"2024-09-21T10:53:02.377737+0000","last_clean":"2024-09-21T10:53:02.377737+0000","last_became_active":"2024-09-21T10:52:25.473105+0000","last_became_peered":"2024-09-21T10:52:25.473105+0000","last_unstale":"2024-09-21T10:53:02.377737+0000","last_undegraded":"2024-09-21T10:53:02.377737+0000","last_fullsized":"2024-09-21T10:53:02.377737+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:15:45.352518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2253,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.634313+0000","last_change":"2024-09-21T10:40:13.432531+0000","last_active":"2024-09-21T10:53:15.634313+0000","last_peered":"2024-09-21T10:53:15.634313+0000","last_clean":"2024-09-21T10:53:15.634313+0000","last_became_active":"2024-09-21T10:40:13.432210+0000","last_became_peered":"2024-09-21T10:40:13.432210+0000","last_unstale":"2024-09-21T10:53:15.634313+0000","last_undegraded":"2024-09-21T10:53:15.634313+0000","last_fullsized":"2024-09-21T10:53:15.634313+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:20:10.998607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2225,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.620552+0000","last_change":"2024-09-21T10:09:06.530394+0000","last_active":"2024-09-21T10:53:15.620552+0000","last_peered":"2024-09-21T10:53:15.620552+0000","last_clean":"2024-09-21T10:53:15.620552+0000","last_became_active":"2024-09-21T10:09:06.529307+0000","last_became_peered":"2024-09-21T10:09:06.529307+0000","last_unstale":"2024-09-21T10:53:15.620552+0000","last_undegraded":"2024-09-21T10:53:15.620552+0000","last_fullsized":"2024-09-21T10:53:15.620552+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:54.287397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.371979+0000","last_change":"2024-09-21T10:52:25.489896+0000","last_active":"2024-09-21T10:53:02.371979+0000","last_peered":"2024-09-21T10:53:02.371979+0000","last_clean":"2024-09-21T10:53:02.371979+0000","last_became_active":"2024-09-21T10:52:25.489154+0000","last_became_peered":"2024-09-21T10:52:25.489154+0000","last_unstale":"2024-09-21T10:53:02.371979+0000","last_undegraded":"2024-09-21T10:53:02.371979+0000","last_fullsized":"2024-09-21T10:53:02.371979+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:36:02.076931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2254,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.625455+0000","last_change":"2024-09-21T10:52:25.493309+0000","last_active":"2024-09-21T10:53:15.625455+0000","last_peered":"2024-09-21T10:53:15.625455+0000","last_clean":"2024-09-21T10:53:15.625455+0000","last_became_active":"2024-09-21T10:52:25.492568+0000","last_became_peered":"2024-09-21T10:52:25.492568+0000","last_unstale":"2024-09-21T10:53:15.625455+0000","last_undegraded":"2024-09-21T10:53:15.625455+0000","last_fullsized":"2024-09-21T10:53:15.625455+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:19:19.393370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1477,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.619677+0000","last_change":"2024-09-21T10:52:03.317474+0000","last_active":"2024-09-21T10:53:15.619677+0000","last_peered":"2024-09-21T10:53:15.619677+0000","last_clean":"2024-09-21T10:53:15.619677+0000","last_became_active":"2024-09-21T10:52:03.317001+0000","last_became_peered":"2024-09-21T10:52:03.317001+0000","last_unstale":"2024-09-21T10:53:15.619677+0000","last_undegraded":"2024-09-21T10:53:15.619677+0000","last_fullsized":"2024-09-21T10:53:15.619677+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:06:07.039763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2253,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.625692+0000","last_change":"2024-09-21T10:52:15.412246+0000","last_active":"2024-09-21T10:53:15.625692+0000","last_peered":"2024-09-21T10:53:15.625692+0000","last_clean":"2024-09-21T10:53:15.625692+0000","last_became_active":"2024-09-21T10:52:15.406204+0000","last_became_peered":"2024-09-21T10:52:15.406204+0000","last_unstale":"2024-09-21T10:53:15.625692+0000","last_undegraded":"2024-09-21T10:53:15.625692+0000","last_fullsized":"2024-09-21T10:53:15.625692+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:03.524324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2233,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.624859+0000","last_change":"2024-09-21T10:52:15.402822+0000","last_active":"2024-09-21T10:53:15.624859+0000","last_peered":"2024-09-21T10:53:15.624859+0000","last_clean":"2024-09-21T10:53:15.624859+0000","last_became_active":"2024-09-21T10:52:15.402548+0000","last_became_peered":"2024-09-21T10:52:15.402548+0000","last_unstale":"2024-09-21T10:53:15.624859+0000","last_undegraded":"2024-09-21T10:53:15.624859+0000","last_fullsized":"2024-09-21T10:53:15.624859+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:04:04.667598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2258,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.626815+0000","last_change":"2024-09-21T10:52:25.499172+0000","last_active":"2024-09-21T10:53:15.626815+0000","last_peered":"2024-09-21T10:53:15.626815+0000","last_clean":"2024-09-21T10:53:15.626815+0000","last_became_active":"2024-09-21T10:52:25.498835+0000","last_became_peered":"2024-09-21T10:52:25.498835+0000","last_unstale":"2024-09-21T10:53:15.626815+0000","last_undegraded":"2024-09-21T10:53:15.626815+0000","last_fullsized":"2024-09-21T10:53:15.626815+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:10:12.434470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":19,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.371941+0000","last_change":"2024-09-21T10:52:25.489774+0000","last_active":"2024-09-21T10:53:02.371941+0000","last_peered":"2024-09-21T10:53:02.371941+0000","last_clean":"2024-09-21T10:53:02.371941+0000","last_became_active":"2024-09-21T10:52:25.487788+0000","last_became_peered":"2024-09-21T10:52:25.487788+0000","last_unstale":"2024-09-21T10:53:02.371941+0000","last_undegraded":"2024-09-21T10:53:02.371941+0000","last_fullsized":"2024-09-21T10:53:02.371941+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:48.842419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":25,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.370754+0000","last_change":"2024-09-21T10:52:25.565465+0000","last_active":"2024-09-21T10:53:02.370754+0000","last_peered":"2024-09-21T10:53:02.370754+0000","last_clean":"2024-09-21T10:53:02.370754+0000","last_became_active":"2024-09-21T10:52:25.510131+0000","last_became_peered":"2024-09-21T10:52:25.510131+0000","last_unstale":"2024-09-21T10:53:02.370754+0000","last_undegraded":"2024-09-21T10:53:02.370754+0000","last_fullsized":"2024-09-21T10:53:02.370754+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:41:13.870503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":450,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2260,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.626347+0000","last_change":"2024-09-21T10:52:25.504787+0000","last_active":"2024-09-21T10:53:15.626347+0000","last_peered":"2024-09-21T10:53:15.626347+0000","last_clean":"2024-09-21T10:53:15.626347+0000","last_became_active":"2024-09-21T10:52:25.504198+0000","last_became_peered":"2024-09-21T10:52:25.504198+0000","last_unstale":"2024-09-21T10:53:15.626347+0000","last_undegraded":"2024-09-21T10:53:15.626347+0000","last_fullsized":"2024-09-21T10:53:15.626347+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:38:25.666224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":1493,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.377215+0000","last_change":"2024-09-21T10:44:14.415264+0000","last_active":"2024-09-21T10:53:02.377215+0000","last_peered":"2024-09-21T10:53:02.377215+0000","last_clean":"2024-09-21T10:53:02.377215+0000","last_became_active":"2024-09-21T10:44:14.407813+0000","last_became_peered":"2024-09-21T10:44:14.407813+0000","last_unstale":"2024-09-21T10:53:02.377215+0000","last_undegraded":"2024-09-21T10:53:02.377215+0000","last_fullsized":"2024-09-21T10:53:02.377215+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:55.933601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1447,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.613811+0000","last_change":"2024-09-21T10:07:56.001756+0000","last_active":"2024-09-21T10:53:15.613811+0000","last_peered":"2024-09-21T10:53:15.613811+0000","last_clean":"2024-09-21T10:53:15.613811+0000","last_became_active":"2024-09-21T10:07:56.000789+0000","last_became_peered":"2024-09-21T10:07:56.000789+0000","last_unstale":"2024-09-21T10:53:15.613811+0000","last_undegraded":"2024-09-21T10:53:15.613811+0000","last_fullsized":"2024-09-21T10:53:15.613811+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:56:04.302902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"346'149","reported_seq":910,"reported_epoch":1474,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:53:15.625317+0000","last_change":"2024-09-21T10:52:26.478603+0000","last_active":"2024-09-21T10:53:15.625317+0000","last_peered":"2024-09-21T10:53:15.625317+0000","last_clean":"2024-09-21T10:52:16.392213+0000","last_became_active":"2024-09-21T10:52:26.478167+0000","last_became_peered":"2024-09-21T10:52:26.478167+0000","last_unstale":"2024-09-21T10:53:15.625317+0000","last_undegraded":"2024-09-21T10:52:26.463682+0000","last_fullsized":"2024-09-21T10:52:26.463251+0000","mapping_epoch":1466,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1449,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1251389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,6],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":1},{"shards":"3,6","objects":-1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1462,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.612810+0000","last_change":"2024-09-21T10:52:25.473127+0000","last_active":"2024-09-21T10:53:15.612810+0000","last_peered":"2024-09-21T10:53:15.612810+0000","last_clean":"2024-09-21T10:53:15.612810+0000","last_became_active":"2024-09-21T10:52:25.472805+0000","last_became_peered":"2024-09-21T10:52:25.472805+0000","last_unstale":"2024-09-21T10:53:15.612810+0000","last_undegraded":"2024-09-21T10:53:15.612810+0000","last_fullsized":"2024-09-21T10:53:15.612810+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:13.038797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1507,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.376134+0000","last_change":"2024-09-21T10:52:25.493263+0000","last_active":"2024-09-21T10:53:02.376134+0000","last_peered":"2024-09-21T10:53:02.376134+0000","last_clean":"2024-09-21T10:53:02.376134+0000","last_became_active":"2024-09-21T10:52:25.492970+0000","last_became_peered":"2024-09-21T10:52:25.492970+0000","last_unstale":"2024-09-21T10:53:02.376134+0000","last_undegraded":"2024-09-21T10:53:02.376134+0000","last_fullsized":"2024-09-21T10:53:02.376134+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:43:40.305783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":104,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.620552+0000","last_change":"2024-09-21T10:52:25.513834+0000","last_active":"2024-09-21T10:53:15.620552+0000","last_peered":"2024-09-21T10:53:15.620552+0000","last_clean":"2024-09-21T10:53:15.620552+0000","last_became_active":"2024-09-21T10:52:25.475423+0000","last_became_peered":"2024-09-21T10:52:25.475423+0000","last_unstale":"2024-09-21T10:53:15.620552+0000","last_undegraded":"2024-09-21T10:53:15.620552+0000","last_fullsized":"2024-09-21T10:53:15.620552+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:10.216527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"346'149","reported_seq":893,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.377552+0000","last_change":"2024-09-21T10:52:24.459450+0000","last_active":"2024-09-21T10:53:02.377552+0000","last_peered":"2024-09-21T10:53:02.377552+0000","last_clean":"2024-09-21T10:53:02.377552+0000","last_became_active":"2024-09-21T10:52:15.405544+0000","last_became_peered":"2024-09-21T10:52:15.405544+0000","last_unstale":"2024-09-21T10:53:02.377552+0000","last_undegraded":"2024-09-21T10:53:02.377552+0000","last_fullsized":"2024-09-21T10:53:02.377552+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2225,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.625618+0000","last_change":"2024-09-21T10:52:25.494316+0000","last_active":"2024-09-21T10:53:15.625618+0000","last_peered":"2024-09-21T10:53:15.625618+0000","last_clean":"2024-09-21T10:53:15.625618+0000","last_became_active":"2024-09-21T10:52:25.494016+0000","last_became_peered":"2024-09-21T10:52:25.494016+0000","last_unstale":"2024-09-21T10:53:15.625618+0000","last_undegraded":"2024-09-21T10:53:15.625618+0000","last_fullsized":"2024-09-21T10:53:15.625618+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:07.859584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2226,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.620737+0000","last_change":"2024-09-21T10:07:56.006236+0000","last_active":"2024-09-21T10:53:15.620737+0000","last_peered":"2024-09-21T10:53:15.620737+0000","last_clean":"2024-09-21T10:53:15.620737+0000","last_became_active":"2024-09-21T10:07:56.004439+0000","last_became_peered":"2024-09-21T10:07:56.004439+0000","last_unstale":"2024-09-21T10:53:15.620737+0000","last_undegraded":"2024-09-21T10:53:15.620737+0000","last_fullsized":"2024-09-21T10:53:15.620737+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:39.422467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":2276,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.625009+0000","last_change":"2024-09-21T10:52:25.536401+0000","last_active":"2024-09-21T10:53:15.625009+0000","last_peered":"2024-09-21T10:53:15.625009+0000","last_clean":"2024-09-21T10:53:15.625009+0000","last_became_active":"2024-09-21T10:52:25.496480+0000","last_became_peered":"2024-09-21T10:52:25.496480+0000","last_unstale":"2024-09-21T10:53:15.625009+0000","last_undegraded":"2024-09-21T10:53:15.625009+0000","last_fullsized":"2024-09-21T10:53:15.625009+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:50:34.820373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"346'149","reported_seq":718,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.375487+0000","last_change":"2024-09-21T10:51:30.450301+0000","last_active":"2024-09-21T10:53:02.375487+0000","last_peered":"2024-09-21T10:53:02.375487+0000","last_clean":"2024-09-21T10:53:02.375487+0000","last_became_active":"2024-09-21T10:51:30.449925+0000","last_became_peered":"2024-09-21T10:51:30.449925+0000","last_unstale":"2024-09-21T10:53:02.375487+0000","last_undegraded":"2024-09-21T10:53:02.375487+0000","last_fullsized":"2024-09-21T10:53:02.375487+0000","mapping_epoch":1450,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1451,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:41.011085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:41.011085+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:38:33.451261+0000","scrub_duration":0.0061250310000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2189,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.370748+0000","last_change":"2024-09-21T10:51:27.431416+0000","last_active":"2024-09-21T10:53:02.370748+0000","last_peered":"2024-09-21T10:53:02.370748+0000","last_clean":"2024-09-21T10:53:02.370748+0000","last_became_active":"2024-09-21T10:51:27.431127+0000","last_became_peered":"2024-09-21T10:51:27.431127+0000","last_unstale":"2024-09-21T10:53:02.370748+0000","last_undegraded":"2024-09-21T10:53:02.370748+0000","last_fullsized":"2024-09-21T10:53:02.370748+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:27.721665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.370730+0000","last_change":"2024-09-21T10:52:25.468849+0000","last_active":"2024-09-21T10:53:02.370730+0000","last_peered":"2024-09-21T10:53:02.370730+0000","last_clean":"2024-09-21T10:53:02.370730+0000","last_became_active":"2024-09-21T10:52:25.468552+0000","last_became_peered":"2024-09-21T10:52:25.468552+0000","last_unstale":"2024-09-21T10:53:02.370730+0000","last_undegraded":"2024-09-21T10:53:02.370730+0000","last_fullsized":"2024-09-21T10:53:02.370730+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:36:41.692028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2276,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.620560+0000","last_change":"2024-09-21T10:51:27.436608+0000","last_active":"2024-09-21T10:53:15.620560+0000","last_peered":"2024-09-21T10:53:15.620560+0000","last_clean":"2024-09-21T10:53:15.620560+0000","last_became_active":"2024-09-21T10:51:27.434949+0000","last_became_peered":"2024-09-21T10:51:27.434949+0000","last_unstale":"2024-09-21T10:53:15.620560+0000","last_undegraded":"2024-09-21T10:53:15.620560+0000","last_fullsized":"2024-09-21T10:53:15.620560+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:49:23.439425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"346'149","reported_seq":761,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.375846+0000","last_change":"2024-09-21T10:52:28.590793+0000","last_active":"2024-09-21T10:53:02.375846+0000","last_peered":"2024-09-21T10:53:02.375846+0000","last_clean":"2024-09-21T10:53:02.375846+0000","last_became_active":"2024-09-21T10:52:28.588662+0000","last_became_peered":"2024-09-21T10:52:28.588662+0000","last_unstale":"2024-09-21T10:53:02.375846+0000","last_undegraded":"2024-09-21T10:53:02.375846+0000","last_fullsized":"2024-09-21T10:53:02.375846+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:44.018645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:44.018645+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:46.849427+0000","scrub_duration":0.0012789399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2231,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.626221+0000","last_change":"2024-09-21T10:51:27.440425+0000","last_active":"2024-09-21T10:53:15.626221+0000","last_peered":"2024-09-21T10:53:15.626221+0000","last_clean":"2024-09-21T10:53:15.626221+0000","last_became_active":"2024-09-21T10:51:27.439666+0000","last_became_peered":"2024-09-21T10:51:27.439666+0000","last_unstale":"2024-09-21T10:53:15.626221+0000","last_undegraded":"2024-09-21T10:53:15.626221+0000","last_fullsized":"2024-09-21T10:53:15.626221+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:56:47.410516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2299,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.620338+0000","last_change":"2024-09-21T10:52:03.314926+0000","last_active":"2024-09-21T10:53:15.620338+0000","last_peered":"2024-09-21T10:53:15.620338+0000","last_clean":"2024-09-21T10:53:15.620338+0000","last_became_active":"2024-09-21T10:52:03.314602+0000","last_became_peered":"2024-09-21T10:52:03.314602+0000","last_unstale":"2024-09-21T10:53:15.620338+0000","last_undegraded":"2024-09-21T10:53:15.620338+0000","last_fullsized":"2024-09-21T10:53:15.620338+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:13.774521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1455,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.634150+0000","last_change":"2024-09-21T10:52:25.502830+0000","last_active":"2024-09-21T10:53:15.634150+0000","last_peered":"2024-09-21T10:53:15.634150+0000","last_clean":"2024-09-21T10:53:15.634150+0000","last_became_active":"2024-09-21T10:52:25.502505+0000","last_became_peered":"2024-09-21T10:52:25.502505+0000","last_unstale":"2024-09-21T10:53:15.634150+0000","last_undegraded":"2024-09-21T10:53:15.634150+0000","last_fullsized":"2024-09-21T10:53:15.634150+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:01:14.965694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"346'149","reported_seq":816,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.376366+0000","last_change":"2024-09-21T10:52:24.458657+0000","last_active":"2024-09-21T10:53:02.376366+0000","last_peered":"2024-09-21T10:53:02.376366+0000","last_clean":"2024-09-21T10:53:02.376366+0000","last_became_active":"2024-09-21T10:51:30.436983+0000","last_became_peered":"2024-09-21T10:51:30.436983+0000","last_unstale":"2024-09-21T10:53:02.376366+0000","last_undegraded":"2024-09-21T10:53:02.376366+0000","last_fullsized":"2024-09-21T10:53:02.376366+0000","mapping_epoch":1450,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1451,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2293,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.620697+0000","last_change":"2024-09-21T10:52:15.397180+0000","last_active":"2024-09-21T10:53:15.620697+0000","last_peered":"2024-09-21T10:53:15.620697+0000","last_clean":"2024-09-21T10:53:15.620697+0000","last_became_active":"2024-09-21T10:52:15.396879+0000","last_became_peered":"2024-09-21T10:52:15.396879+0000","last_unstale":"2024-09-21T10:53:15.620697+0000","last_undegraded":"2024-09-21T10:53:15.620697+0000","last_fullsized":"2024-09-21T10:53:15.620697+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:03:07.162178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1460,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.614094+0000","last_change":"2024-09-21T10:51:27.436590+0000","last_active":"2024-09-21T10:53:15.614094+0000","last_peered":"2024-09-21T10:53:15.614094+0000","last_clean":"2024-09-21T10:53:15.614094+0000","last_became_active":"2024-09-21T10:51:27.435332+0000","last_became_peered":"2024-09-21T10:51:27.435332+0000","last_unstale":"2024-09-21T10:53:15.614094+0000","last_undegraded":"2024-09-21T10:53:15.614094+0000","last_fullsized":"2024-09-21T10:53:15.614094+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:05:42.885506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":189,"reported_epoch":1474,"state":"active+recovery_wait","last_fresh":"2024-09-21T10:53:15.625934+0000","last_change":"2024-09-21T10:52:26.680832+0000","last_active":"2024-09-21T10:53:15.625934+0000","last_peered":"2024-09-21T10:53:15.625934+0000","last_clean":"2024-09-21T10:51:45.437560+0000","last_became_active":"2024-09-21T10:52:03.315398+0000","last_became_peered":"2024-09-21T10:52:03.315398+0000","last_unstale":"2024-09-21T10:53:15.625934+0000","last_undegraded":"2024-09-21T10:53:15.625934+0000","last_fullsized":"2024-09-21T10:53:15.625934+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":["1","3","4","6"],"object_location_counts":[{"shards":"1,3,6","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"346'149","reported_seq":787,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.370546+0000","last_change":"2024-09-21T10:52:28.495319+0000","last_active":"2024-09-21T10:53:02.370546+0000","last_peered":"2024-09-21T10:53:02.370546+0000","last_clean":"2024-09-21T10:53:02.370546+0000","last_became_active":"2024-09-21T10:52:28.494923+0000","last_became_peered":"2024-09-21T10:52:28.494923+0000","last_unstale":"2024-09-21T10:53:02.370546+0000","last_undegraded":"2024-09-21T10:53:02.370546+0000","last_fullsized":"2024-09-21T10:53:02.370546+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:52:17.894787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:52:17.894787+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":177,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.377088+0000","last_change":"2024-09-21T10:52:25.475200+0000","last_active":"2024-09-21T10:53:02.377088+0000","last_peered":"2024-09-21T10:53:02.377088+0000","last_clean":"2024-09-21T10:53:02.377088+0000","last_became_active":"2024-09-21T10:52:25.474872+0000","last_became_peered":"2024-09-21T10:52:25.474872+0000","last_unstale":"2024-09-21T10:53:02.377088+0000","last_undegraded":"2024-09-21T10:53:02.377088+0000","last_fullsized":"2024-09-21T10:53:02.377088+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:36:34.207844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2250,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.620249+0000","last_change":"2024-09-21T10:52:03.325889+0000","last_active":"2024-09-21T10:53:15.620249+0000","last_peered":"2024-09-21T10:53:15.620249+0000","last_clean":"2024-09-21T10:53:15.620249+0000","last_became_active":"2024-09-21T10:52:03.325500+0000","last_became_peered":"2024-09-21T10:52:03.325500+0000","last_unstale":"2024-09-21T10:53:15.620249+0000","last_undegraded":"2024-09-21T10:53:15.620249+0000","last_fullsized":"2024-09-21T10:53:15.620249+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:20.708252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1036,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.626642+0000","last_change":"2024-09-21T10:51:27.457894+0000","last_active":"2024-09-21T10:53:15.626642+0000","last_peered":"2024-09-21T10:53:15.626642+0000","last_clean":"2024-09-21T10:53:15.626642+0000","last_became_active":"2024-09-21T10:51:27.457431+0000","last_became_peered":"2024-09-21T10:51:27.457431+0000","last_unstale":"2024-09-21T10:53:15.626642+0000","last_undegraded":"2024-09-21T10:53:15.626642+0000","last_fullsized":"2024-09-21T10:53:15.626642+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:03:33.550759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"346'149","reported_seq":906,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.622618+0000","last_change":"2024-09-21T10:52:15.396042+0000","last_active":"2024-09-21T10:53:15.622618+0000","last_peered":"2024-09-21T10:53:15.622618+0000","last_clean":"2024-09-21T10:53:15.622618+0000","last_became_active":"2024-09-21T10:52:15.395717+0000","last_became_peered":"2024-09-21T10:52:15.395717+0000","last_unstale":"2024-09-21T10:53:15.622618+0000","last_undegraded":"2024-09-21T10:53:15.622618+0000","last_fullsized":"2024-09-21T10:53:15.622618+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2275,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.620538+0000","last_change":"2024-09-21T10:52:25.476572+0000","last_active":"2024-09-21T10:53:15.620538+0000","last_peered":"2024-09-21T10:53:15.620538+0000","last_clean":"2024-09-21T10:53:15.620538+0000","last_became_active":"2024-09-21T10:52:25.475803+0000","last_became_peered":"2024-09-21T10:52:25.475803+0000","last_unstale":"2024-09-21T10:53:15.620538+0000","last_undegraded":"2024-09-21T10:53:15.620538+0000","last_fullsized":"2024-09-21T10:53:15.620538+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:06:44.191486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2205,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.378571+0000","last_change":"2024-09-21T10:52:03.326753+0000","last_active":"2024-09-21T10:53:02.378571+0000","last_peered":"2024-09-21T10:53:02.378571+0000","last_clean":"2024-09-21T10:53:02.378571+0000","last_became_active":"2024-09-21T10:52:03.326348+0000","last_became_peered":"2024-09-21T10:52:03.326348+0000","last_unstale":"2024-09-21T10:53:02.378571+0000","last_undegraded":"2024-09-21T10:53:02.378571+0000","last_fullsized":"2024-09-21T10:53:02.378571+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:57:38.100799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":128,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.370521+0000","last_change":"2024-09-21T10:51:27.426006+0000","last_active":"2024-09-21T10:53:02.370521+0000","last_peered":"2024-09-21T10:53:02.370521+0000","last_clean":"2024-09-21T10:53:02.370521+0000","last_became_active":"2024-09-21T10:51:27.425698+0000","last_became_peered":"2024-09-21T10:51:27.425698+0000","last_unstale":"2024-09-21T10:53:02.370521+0000","last_undegraded":"2024-09-21T10:53:02.370521+0000","last_fullsized":"2024-09-21T10:53:02.370521+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:35:13.210330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":22,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.374677+0000","last_change":"2024-09-21T10:52:15.401737+0000","last_active":"2024-09-21T10:53:02.374677+0000","last_peered":"2024-09-21T10:53:02.374677+0000","last_clean":"2024-09-21T10:53:02.374677+0000","last_became_active":"2024-09-21T10:52:15.400859+0000","last_became_peered":"2024-09-21T10:52:15.400859+0000","last_unstale":"2024-09-21T10:53:02.374677+0000","last_undegraded":"2024-09-21T10:53:02.374677+0000","last_fullsized":"2024-09-21T10:53:02.374677+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:15.723031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2230,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.625206+0000","last_change":"2024-09-21T10:49:44.865830+0000","last_active":"2024-09-21T10:53:15.625206+0000","last_peered":"2024-09-21T10:53:15.625206+0000","last_clean":"2024-09-21T10:53:15.625206+0000","last_became_active":"2024-09-21T10:49:44.865499+0000","last_became_peered":"2024-09-21T10:49:44.865499+0000","last_unstale":"2024-09-21T10:53:15.625206+0000","last_undegraded":"2024-09-21T10:53:15.625206+0000","last_fullsized":"2024-09-21T10:53:15.625206+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:19:22.013154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":1453,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.371221+0000","last_change":"2024-09-21T10:52:25.489550+0000","last_active":"2024-09-21T10:53:02.371221+0000","last_peered":"2024-09-21T10:53:02.371221+0000","last_clean":"2024-09-21T10:53:02.371221+0000","last_became_active":"2024-09-21T10:52:25.489171+0000","last_became_peered":"2024-09-21T10:52:25.489171+0000","last_unstale":"2024-09-21T10:53:02.371221+0000","last_undegraded":"2024-09-21T10:53:02.371221+0000","last_fullsized":"2024-09-21T10:53:02.371221+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:21.105058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":787,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.370355+0000","last_change":"2024-09-21T10:52:28.601906+0000","last_active":"2024-09-21T10:53:02.370355+0000","last_peered":"2024-09-21T10:53:02.370355+0000","last_clean":"2024-09-21T10:53:02.370355+0000","last_became_active":"2024-09-21T10:52:28.587447+0000","last_became_peered":"2024-09-21T10:52:28.587447+0000","last_unstale":"2024-09-21T10:53:02.370355+0000","last_undegraded":"2024-09-21T10:53:02.370355+0000","last_fullsized":"2024-09-21T10:53:02.370355+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:44.223578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:44.223578+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":74,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"346'149","reported_seq":735,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.374801+0000","last_change":"2024-09-21T10:51:30.453979+0000","last_active":"2024-09-21T10:53:02.374801+0000","last_peered":"2024-09-21T10:53:02.374801+0000","last_clean":"2024-09-21T10:53:02.374801+0000","last_became_active":"2024-09-21T10:51:30.450893+0000","last_became_peered":"2024-09-21T10:51:30.450893+0000","last_unstale":"2024-09-21T10:53:02.374801+0000","last_undegraded":"2024-09-21T10:53:02.374801+0000","last_fullsized":"2024-09-21T10:53:02.374801+0000","mapping_epoch":1450,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1451,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:42.008235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:42.008235+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:21:48.570809+0000","scrub_duration":0.011129834999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2227,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.621476+0000","last_change":"2024-09-21T10:09:06.538940+0000","last_active":"2024-09-21T10:53:15.621476+0000","last_peered":"2024-09-21T10:53:15.621476+0000","last_clean":"2024-09-21T10:53:15.621476+0000","last_became_active":"2024-09-21T10:09:06.538668+0000","last_became_peered":"2024-09-21T10:09:06.538668+0000","last_unstale":"2024-09-21T10:53:15.621476+0000","last_undegraded":"2024-09-21T10:53:15.621476+0000","last_fullsized":"2024-09-21T10:53:15.621476+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:42:44.299488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2225,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.625363+0000","last_change":"2024-09-21T10:52:25.493032+0000","last_active":"2024-09-21T10:53:15.625363+0000","last_peered":"2024-09-21T10:53:15.625363+0000","last_clean":"2024-09-21T10:53:15.625363+0000","last_became_active":"2024-09-21T10:52:25.492182+0000","last_became_peered":"2024-09-21T10:52:25.492182+0000","last_unstale":"2024-09-21T10:53:15.625363+0000","last_undegraded":"2024-09-21T10:53:15.625363+0000","last_fullsized":"2024-09-21T10:53:15.625363+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:12:57.118420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1465,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.377181+0000","last_change":"2024-09-21T10:52:15.402809+0000","last_active":"2024-09-21T10:53:02.377181+0000","last_peered":"2024-09-21T10:53:02.377181+0000","last_clean":"2024-09-21T10:53:02.377181+0000","last_became_active":"2024-09-21T10:52:15.402497+0000","last_became_peered":"2024-09-21T10:52:15.402497+0000","last_unstale":"2024-09-21T10:53:02.377181+0000","last_undegraded":"2024-09-21T10:53:02.377181+0000","last_fullsized":"2024-09-21T10:53:02.377181+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:07:35.713534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1476,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.378731+0000","last_change":"2024-09-21T10:44:14.399842+0000","last_active":"2024-09-21T10:53:02.378731+0000","last_peered":"2024-09-21T10:53:02.378731+0000","last_clean":"2024-09-21T10:53:02.378731+0000","last_became_active":"2024-09-21T10:44:14.399426+0000","last_became_peered":"2024-09-21T10:44:14.399426+0000","last_unstale":"2024-09-21T10:53:02.378731+0000","last_undegraded":"2024-09-21T10:53:02.378731+0000","last_fullsized":"2024-09-21T10:53:02.378731+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:26:45.362101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"346'149","reported_seq":906,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.624270+0000","last_change":"2024-09-21T10:52:16.404597+0000","last_active":"2024-09-21T10:53:15.624270+0000","last_peered":"2024-09-21T10:53:15.624270+0000","last_clean":"2024-09-21T10:53:15.624270+0000","last_became_active":"2024-09-21T10:52:16.404194+0000","last_became_peered":"2024-09-21T10:52:16.404194+0000","last_unstale":"2024-09-21T10:53:15.624270+0000","last_undegraded":"2024-09-21T10:53:15.624270+0000","last_fullsized":"2024-09-21T10:53:15.624270+0000","mapping_epoch":1463,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1464,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":793725,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1471,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.379724+0000","last_change":"2024-09-21T10:40:13.432387+0000","last_active":"2024-09-21T10:53:02.379724+0000","last_peered":"2024-09-21T10:53:02.379724+0000","last_clean":"2024-09-21T10:53:02.379724+0000","last_became_active":"2024-09-21T10:40:13.432053+0000","last_became_peered":"2024-09-21T10:40:13.432053+0000","last_unstale":"2024-09-21T10:53:02.379724+0000","last_undegraded":"2024-09-21T10:53:02.379724+0000","last_fullsized":"2024-09-21T10:53:02.379724+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:37.486101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1449,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.375034+0000","last_change":"2024-09-21T10:07:56.020791+0000","last_active":"2024-09-21T10:53:02.375034+0000","last_peered":"2024-09-21T10:53:02.375034+0000","last_clean":"2024-09-21T10:53:02.375034+0000","last_became_active":"2024-09-21T10:07:56.020205+0000","last_became_peered":"2024-09-21T10:07:56.020205+0000","last_unstale":"2024-09-21T10:53:02.375034+0000","last_undegraded":"2024-09-21T10:53:02.375034+0000","last_fullsized":"2024-09-21T10:53:02.375034+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:09.813166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"346'149","reported_seq":912,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.370491+0000","last_change":"2024-09-21T10:49:47.904295+0000","last_active":"2024-09-21T10:53:02.370491+0000","last_peered":"2024-09-21T10:53:02.370491+0000","last_clean":"2024-09-21T10:53:02.370491+0000","last_became_active":"2024-09-21T10:49:47.903650+0000","last_became_peered":"2024-09-21T10:49:47.903650+0000","last_unstale":"2024-09-21T10:53:02.370491+0000","last_undegraded":"2024-09-21T10:53:02.370491+0000","last_fullsized":"2024-09-21T10:53:02.370491+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":74,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1447,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.377039+0000","last_change":"2024-09-21T10:09:06.533271+0000","last_active":"2024-09-21T10:53:02.377039+0000","last_peered":"2024-09-21T10:53:02.377039+0000","last_clean":"2024-09-21T10:53:02.377039+0000","last_became_active":"2024-09-21T10:09:06.532041+0000","last_became_peered":"2024-09-21T10:09:06.532041+0000","last_unstale":"2024-09-21T10:53:02.377039+0000","last_undegraded":"2024-09-21T10:53:02.377039+0000","last_fullsized":"2024-09-21T10:53:02.377039+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:04.236375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":2254,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.624348+0000","last_change":"2024-09-21T10:51:27.529295+0000","last_active":"2024-09-21T10:53:15.624348+0000","last_peered":"2024-09-21T10:53:15.624348+0000","last_clean":"2024-09-21T10:53:15.624348+0000","last_became_active":"2024-09-21T10:51:27.446359+0000","last_became_peered":"2024-09-21T10:51:27.446359+0000","last_unstale":"2024-09-21T10:53:15.624348+0000","last_undegraded":"2024-09-21T10:53:15.624348+0000","last_fullsized":"2024-09-21T10:53:15.624348+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:17:08.788558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":684,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.623548+0000","last_change":"2024-09-21T10:51:27.481103+0000","last_active":"2024-09-21T10:53:15.623548+0000","last_peered":"2024-09-21T10:53:15.623548+0000","last_clean":"2024-09-21T10:53:15.623548+0000","last_became_active":"2024-09-21T10:51:27.480763+0000","last_became_peered":"2024-09-21T10:51:27.480763+0000","last_unstale":"2024-09-21T10:53:15.623548+0000","last_undegraded":"2024-09-21T10:53:15.623548+0000","last_fullsized":"2024-09-21T10:53:15.623548+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:35.375303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":789,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.625664+0000","last_change":"2024-09-21T10:49:44.864819+0000","last_active":"2024-09-21T10:53:15.625664+0000","last_peered":"2024-09-21T10:53:15.625664+0000","last_clean":"2024-09-21T10:53:15.625664+0000","last_became_active":"2024-09-21T10:49:44.864061+0000","last_became_peered":"2024-09-21T10:49:44.864061+0000","last_unstale":"2024-09-21T10:53:15.625664+0000","last_undegraded":"2024-09-21T10:53:15.625664+0000","last_fullsized":"2024-09-21T10:53:15.625664+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:37.266967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":31,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.374370+0000","last_change":"2024-09-21T10:52:25.492570+0000","last_active":"2024-09-21T10:53:02.374370+0000","last_peered":"2024-09-21T10:53:02.374370+0000","last_clean":"2024-09-21T10:53:02.374370+0000","last_became_active":"2024-09-21T10:52:25.486382+0000","last_became_peered":"2024-09-21T10:52:25.486382+0000","last_unstale":"2024-09-21T10:53:02.374370+0000","last_undegraded":"2024-09-21T10:53:02.374370+0000","last_fullsized":"2024-09-21T10:53:02.374370+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:06.702707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":136,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.629581+0000","last_change":"2024-09-21T10:49:44.874522+0000","last_active":"2024-09-21T10:53:15.629581+0000","last_peered":"2024-09-21T10:53:15.629581+0000","last_clean":"2024-09-21T10:53:15.629581+0000","last_became_active":"2024-09-21T10:49:44.863242+0000","last_became_peered":"2024-09-21T10:49:44.863242+0000","last_unstale":"2024-09-21T10:53:15.629581+0000","last_undegraded":"2024-09-21T10:53:15.629581+0000","last_fullsized":"2024-09-21T10:53:15.629581+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:09:02.066837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1481,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.378631+0000","last_change":"2024-09-21T10:52:03.315267+0000","last_active":"2024-09-21T10:53:02.378631+0000","last_peered":"2024-09-21T10:53:02.378631+0000","last_clean":"2024-09-21T10:53:02.378631+0000","last_became_active":"2024-09-21T10:52:03.314474+0000","last_became_peered":"2024-09-21T10:52:03.314474+0000","last_unstale":"2024-09-21T10:53:02.378631+0000","last_undegraded":"2024-09-21T10:53:02.378631+0000","last_fullsized":"2024-09-21T10:53:02.378631+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:00:57.791341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":23,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.371900+0000","last_change":"2024-09-21T10:52:25.518721+0000","last_active":"2024-09-21T10:53:02.371900+0000","last_peered":"2024-09-21T10:53:02.371900+0000","last_clean":"2024-09-21T10:53:02.371900+0000","last_became_active":"2024-09-21T10:52:25.472114+0000","last_became_peered":"2024-09-21T10:52:25.472114+0000","last_unstale":"2024-09-21T10:53:02.371900+0000","last_undegraded":"2024-09-21T10:53:02.371900+0000","last_fullsized":"2024-09-21T10:53:02.371900+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:33:48.471147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.370247+0000","last_change":"2024-09-21T10:52:25.494490+0000","last_active":"2024-09-21T10:53:02.370247+0000","last_peered":"2024-09-21T10:53:02.370247+0000","last_clean":"2024-09-21T10:53:02.370247+0000","last_became_active":"2024-09-21T10:52:25.494200+0000","last_became_peered":"2024-09-21T10:52:25.494200+0000","last_unstale":"2024-09-21T10:53:02.370247+0000","last_undegraded":"2024-09-21T10:53:02.370247+0000","last_fullsized":"2024-09-21T10:53:02.370247+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:15:21.030031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":23,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.370897+0000","last_change":"2024-09-21T10:52:25.479229+0000","last_active":"2024-09-21T10:53:02.370897+0000","last_peered":"2024-09-21T10:53:02.370897+0000","last_clean":"2024-09-21T10:53:02.370897+0000","last_became_active":"2024-09-21T10:52:25.470551+0000","last_became_peered":"2024-09-21T10:52:25.470551+0000","last_unstale":"2024-09-21T10:53:02.370897+0000","last_undegraded":"2024-09-21T10:53:02.370897+0000","last_fullsized":"2024-09-21T10:53:02.370897+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:15.458231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1494,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:11.761902+0000","last_change":"2024-09-21T10:51:27.517234+0000","last_active":"2024-09-21T10:53:11.761902+0000","last_peered":"2024-09-21T10:53:11.761902+0000","last_clean":"2024-09-21T10:53:11.761902+0000","last_became_active":"2024-09-21T10:51:27.431253+0000","last_became_peered":"2024-09-21T10:51:27.431253+0000","last_unstale":"2024-09-21T10:53:11.761902+0000","last_undegraded":"2024-09-21T10:53:11.761902+0000","last_fullsized":"2024-09-21T10:53:11.761902+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:35:42.249305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2209,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.624936+0000","last_change":"2024-09-21T10:52:25.497298+0000","last_active":"2024-09-21T10:53:15.624936+0000","last_peered":"2024-09-21T10:53:15.624936+0000","last_clean":"2024-09-21T10:53:15.624936+0000","last_became_active":"2024-09-21T10:52:25.496876+0000","last_became_peered":"2024-09-21T10:52:25.496876+0000","last_unstale":"2024-09-21T10:53:15.624936+0000","last_undegraded":"2024-09-21T10:53:15.624936+0000","last_fullsized":"2024-09-21T10:53:15.624936+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:16.587897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":118,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.370634+0000","last_change":"2024-09-21T10:44:14.398878+0000","last_active":"2024-09-21T10:53:02.370634+0000","last_peered":"2024-09-21T10:53:02.370634+0000","last_clean":"2024-09-21T10:53:02.370634+0000","last_became_active":"2024-09-21T10:44:14.398547+0000","last_became_peered":"2024-09-21T10:44:14.398547+0000","last_unstale":"2024-09-21T10:53:02.370634+0000","last_undegraded":"2024-09-21T10:53:02.370634+0000","last_fullsized":"2024-09-21T10:53:02.370634+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:13.008134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2257,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.628869+0000","last_change":"2024-09-21T10:52:25.498554+0000","last_active":"2024-09-21T10:53:15.628869+0000","last_peered":"2024-09-21T10:53:15.628869+0000","last_clean":"2024-09-21T10:53:15.628869+0000","last_became_active":"2024-09-21T10:52:25.498194+0000","last_became_peered":"2024-09-21T10:52:25.498194+0000","last_unstale":"2024-09-21T10:53:15.628869+0000","last_undegraded":"2024-09-21T10:53:15.628869+0000","last_fullsized":"2024-09-21T10:53:15.628869+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:47.115271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.370770+0000","last_change":"2024-09-21T10:52:25.489063+0000","last_active":"2024-09-21T10:53:02.370770+0000","last_peered":"2024-09-21T10:53:02.370770+0000","last_clean":"2024-09-21T10:53:02.370770+0000","last_became_active":"2024-09-21T10:52:25.488129+0000","last_became_peered":"2024-09-21T10:52:25.488129+0000","last_unstale":"2024-09-21T10:53:02.370770+0000","last_undegraded":"2024-09-21T10:53:02.370770+0000","last_fullsized":"2024-09-21T10:53:02.370770+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:36.996829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1433,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.370880+0000","last_change":"2024-09-21T10:49:44.864898+0000","last_active":"2024-09-21T10:53:02.370880+0000","last_peered":"2024-09-21T10:53:02.370880+0000","last_clean":"2024-09-21T10:53:02.370880+0000","last_became_active":"2024-09-21T10:49:44.862691+0000","last_became_peered":"2024-09-21T10:49:44.862691+0000","last_unstale":"2024-09-21T10:53:02.370880+0000","last_undegraded":"2024-09-21T10:53:02.370880+0000","last_fullsized":"2024-09-21T10:53:02.370880+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:57.809113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":124,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.371248+0000","last_change":"2024-09-21T10:44:14.418113+0000","last_active":"2024-09-21T10:53:02.371248+0000","last_peered":"2024-09-21T10:53:02.371248+0000","last_clean":"2024-09-21T10:53:02.371248+0000","last_became_active":"2024-09-21T10:44:14.398327+0000","last_became_peered":"2024-09-21T10:44:14.398327+0000","last_unstale":"2024-09-21T10:53:02.371248+0000","last_undegraded":"2024-09-21T10:53:02.371248+0000","last_fullsized":"2024-09-21T10:53:02.371248+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:08.810863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2238,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.625808+0000","last_change":"2024-09-21T10:44:14.398584+0000","last_active":"2024-09-21T10:53:15.625808+0000","last_peered":"2024-09-21T10:53:15.625808+0000","last_clean":"2024-09-21T10:53:15.625808+0000","last_became_active":"2024-09-21T10:44:14.398278+0000","last_became_peered":"2024-09-21T10:44:14.398278+0000","last_unstale":"2024-09-21T10:53:15.625808+0000","last_undegraded":"2024-09-21T10:53:15.625808+0000","last_fullsized":"2024-09-21T10:53:15.625808+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:19.681750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":55,"num_pg":16,"stat_sum":{"num_bytes":1446707224,"num_objects":22076,"num_object_clones":0,"num_object_copies":66228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15969,"num_objects_misplaced":7139,"num_objects_unfound":0,"num_objects_dirty":22076,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":188602,"num_write_kb":6741633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42400,"num_bytes_recovered":2702114816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8378265600,"data_stored":8378253384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74161,"ondisk_log_size":74161,"up":48,"acting":44,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":429359,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":276,"num_read_kb":598,"num_write":157,"num_write_kb":1757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3655209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":32,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":95,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":11344,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":18188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234711,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3285840,"kb_used_data":2753540,"kb_used_omap":1,"kb_used_meta":532286,"kb_avail":90459312,"statfs":{"total":95995035648,"available":92630335488,"internally_reserved":0,"allocated":2819624960,"data_stored":2814622839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":545061323},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496348,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1688088,"kb_used_data":1007308,"kb_used_omap":1,"kb_used_meta":680766,"kb_avail":92057064,"statfs":{"total":95995035648,"available":94266433536,"internally_reserved":0,"allocated":1031483392,"data_stored":1026474202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":697104842},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757985,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019621,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1180184,"kb_used_data":539600,"kb_used_omap":1,"kb_used_meta":640574,"kb_avail":92564968,"statfs":{"total":95995035648,"available":94786527232,"internally_reserved":0,"allocated":552550400,"data_stored":547537090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":655948230},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369364,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1992140,"kb_used_data":1229376,"kb_used_omap":69,"kb_used_meta":762298,"kb_avail":91753012,"statfs":{"total":95995035648,"available":93955084288,"internally_reserved":0,"allocated":1258881024,"data_stored":1253879368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":780593282},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2920577761820,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807958051,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1331580,"kb_used_data":649780,"kb_used_omap":66,"kb_used_meta":681661,"kb_avail":92413572,"statfs":{"total":95995035648,"available":94631497728,"internally_reserved":0,"allocated":665374720,"data_stored":660384154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68589,"internal_metadata":698020883},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":2860448219689,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":699666432,"data_stored":699662360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":644349952,"data_stored":644349952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1220349952,"data_stored":1220345880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":538710016,"data_stored":538705944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1476132864,"data_stored":1476132864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1011154944,"data_stored":1011154944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2787901440,"data_stored":2787901440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:53:20.583 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:20.615 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:20.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:20.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:20.717 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:20.754 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:20.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:20.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:20.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:20 smithi070 ceph-mon[107385]: pgmap v2707: 99 pgs: 2 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.3 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 785 op/s; 15970/66306 objects degraded (24.085%); 7139/66306 objects misplaced (10.767%); 19 MiB/s, 361 objects/s recovering 2024-09-21T10:53:20.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:53:20.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:53:20.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:20 smithi070 ceph-mon[112139]: pgmap v2707: 99 pgs: 2 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.3 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 785 op/s; 15970/66306 objects degraded (24.085%); 7139/66306 objects misplaced (10.767%); 19 MiB/s, 361 objects/s recovering 2024-09-21T10:53:20.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:53:20.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:53:20.856 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:20.889 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:20.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:20.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:20.991 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:21.025 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:21.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:21.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:21.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:20 smithi160 ceph-mon[101879]: pgmap v2707: 99 pgs: 2 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.3 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 785 op/s; 15970/66306 objects degraded (24.085%); 7139/66306 objects misplaced (10.767%); 19 MiB/s, 361 objects/s recovering 2024-09-21T10:53:21.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:53:21.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:53:21.127 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:21.160 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:21.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:21.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:21.262 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:21.308 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:21.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:21.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:21.410 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:21.446 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:21.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:21.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:21.547 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:21.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:21.584 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:21.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:21.685 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:21.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:21.716 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:21.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:21.817 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:21.853 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:21.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:21.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:21.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:21 smithi070 ceph-mon[107385]: from='client.55285 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:53:21.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:21 smithi070 ceph-mon[112139]: from='client.55285 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:53:21.955 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:21.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:21.992 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:21.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:22.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:21 smithi160 ceph-mon[101879]: from='client.55285 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:53:22.092 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:22.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:22.114 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:22.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:22.215 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:22.257 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:22.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:22.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:22.359 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:22.399 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:22.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:22.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:22.501 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:22.535 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:22.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:22.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:22.637 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:22.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:22.672 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:22.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:22.772 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:22.804 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:22.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:22.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:22.905 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:22.936 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:22.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:22.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:23.038 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:23.052 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:22 smithi070 ceph-mon[107385]: pgmap v2708: 99 pgs: 2 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.3 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 785 op/s; 15970/66306 objects degraded (24.085%); 7139/66306 objects misplaced (10.767%); 19 MiB/s, 361 objects/s recovering 2024-09-21T10:53:23.052 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:22 smithi070 ceph-mon[112139]: pgmap v2708: 99 pgs: 2 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.3 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 785 op/s; 15970/66306 objects degraded (24.085%); 7139/66306 objects misplaced (10.767%); 19 MiB/s, 361 objects/s recovering 2024-09-21T10:53:23.074 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:23.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:23.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:23.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:22 smithi160 ceph-mon[101879]: pgmap v2708: 99 pgs: 2 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.3 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 785 op/s; 15970/66306 objects degraded (24.085%); 7139/66306 objects misplaced (10.767%); 19 MiB/s, 361 objects/s recovering 2024-09-21T10:53:23.176 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:23.214 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:23.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:23.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:23.316 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:23.347 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:23.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:23.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:23.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:53:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:53:23] "GET /metrics HTTP/1.1" 200 36627 "" "Prometheus/2.51.0" 2024-09-21T10:53:23.449 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:23.475 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:23.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:23.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:23.557 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:53:23.577 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:23.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:23.606 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:23.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:23.693 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:23 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 15970/66306 objects degraded (24.085%), 9 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:23.693 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:23 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 15970/66306 objects degraded (24.085%), 9 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:23.706 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:23.735 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:23.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:23.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:23.836 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:23.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:23.858 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:23.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:23.959 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:23.988 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:23.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:23.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:24.090 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:24.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:23 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 15970/66306 objects degraded (24.085%), 9 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:24.120 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:24.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:24.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:24.222 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:24.264 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:24.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:24.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:24.366 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:24.399 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:24.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:24.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:24.501 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:24.533 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:24.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:24.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:24.635 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:24.678 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:24.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:24.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:24.780 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:24.812 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:24.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:24.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:24.914 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:24.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:24.947 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:24.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:24.992 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:24 smithi160 ceph-mon[101879]: pgmap v2709: 99 pgs: 2 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.1 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 800 op/s; 14912/55410 objects degraded (26.912%); 6421/55410 objects misplaced (11.588%); 18 MiB/s, 357 objects/s recovering 2024-09-21T10:53:24.999 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:24 smithi070 ceph-mon[107385]: pgmap v2709: 99 pgs: 2 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.1 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 800 op/s; 14912/55410 objects degraded (26.912%); 6421/55410 objects misplaced (11.588%); 18 MiB/s, 357 objects/s recovering 2024-09-21T10:53:24.999 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:24 smithi070 ceph-mon[112139]: pgmap v2709: 99 pgs: 2 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.1 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 800 op/s; 14912/55410 objects degraded (26.912%); 6421/55410 objects misplaced (11.588%); 18 MiB/s, 357 objects/s recovering 2024-09-21T10:53:25.048 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:25.087 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:25.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:25.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:25.189 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:25.220 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:25.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:25.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:25.322 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:25.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:25.348 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:25.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:25.449 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:25.463 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:53:25.470 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:25.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:25.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:25.572 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:25.614 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:25.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:25.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:25.695 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:53:25.699 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:25 smithi070 ceph-mon[107385]: pgmap v2710: 99 pgs: 2 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.1 GiB data, 8.5 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 802 op/s; 14127/53829 objects degraded (26.244%); 6679/53829 objects misplaced (12.408%); 18 MiB/s, 360 objects/s recovering 2024-09-21T10:53:25.699 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:25 smithi070 ceph-mon[112139]: pgmap v2710: 99 pgs: 2 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.1 GiB data, 8.5 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 802 op/s; 14127/53829 objects degraded (26.244%); 6679/53829 objects misplaced (12.408%); 18 MiB/s, 360 objects/s recovering 2024-09-21T10:53:25.716 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:25.789 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:25.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:25.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:25.891 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:25.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:25.926 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:25.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:26.027 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:26.064 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:26.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:26.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:26.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:25 smithi160 ceph-mon[101879]: pgmap v2710: 99 pgs: 2 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.1 GiB data, 8.5 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 802 op/s; 14127/53829 objects degraded (26.244%); 6679/53829 objects misplaced (12.408%); 18 MiB/s, 360 objects/s recovering 2024-09-21T10:53:26.166 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:26.187 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:26.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:26.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:26.288 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:26.327 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:26.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:26.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:26.429 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:26.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:26.466 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:26.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:26.476 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:53:26.476 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:53:26.477 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:53:26.477 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:53:26.477 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:53:26.477 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:53:26.477 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:53:26.477 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:53:26.477 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:53:26.477 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:53:26.477 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:53:26.477 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:53:26.477 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:53:26.477 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:53:26.477 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:53:26.478 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:53:26.567 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:26.598 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:26.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:26.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:26.700 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:26.732 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:26.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:26.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:26.835 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:26.875 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:26.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:26.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:26.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:26 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/421342590' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:53:26.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:26 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/421342590' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:53:26.977 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:27.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:27.013 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:27.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:27.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:26 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/421342590' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:53:27.102 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:53:27.102 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T10:53:27.102 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T10:53:27.102 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T10:53:27.102 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.102 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:76003 55.8 55:10aa5eab:::benchmark_data_smithi070_294394_object76002:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1426)", 2024-09-21T10:53:27.102 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:49:43.855798+0000", 2024-09-21T10:53:27.103 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 223.24264229100001, 2024-09-21T10:53:27.103 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2879054860000001, 2024-09-21T10:53:27.103 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:53:27.103 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:53:27.103 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:53:27.103 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:53:27.103 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:53:27.103 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:53:27.103 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 76003 2024-09-21T10:53:27.103 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.103 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:53:27.104 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.104 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:53:27.104 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855798+0000", 2024-09-21T10:53:27.104 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.104 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.104 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.104 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:53:27.104 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855798+0000", 2024-09-21T10:53:27.104 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.104 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.104 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.104 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:53:27.104 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855800+0000", 2024-09-21T10:53:27.105 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7969999999999999e-06 2024-09-21T10:53:27.105 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.105 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.105 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:53:27.105 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855864+0000", 2024-09-21T10:53:27.105 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3825999999999995e-05 2024-09-21T10:53:27.105 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.105 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.105 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:53:27.106 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855881+0000", 2024-09-21T10:53:27.106 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6991999999999999e-05 2024-09-21T10:53:27.106 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.106 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.106 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:53:27.106 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855889+0000", 2024-09-21T10:53:27.106 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6910000000000003e-06 2024-09-21T10:53:27.106 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.106 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.106 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.106 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.856008+0000", 2024-09-21T10:53:27.106 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011862300000000001 2024-09-21T10:53:27.107 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.107 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.107 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:53:27.107 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.856021+0000", 2024-09-21T10:53:27.107 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3016e-05 2024-09-21T10:53:27.107 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.107 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.107 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.107 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.849652+0000", 2024-09-21T10:53:27.107 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99363043399999995 2024-09-21T10:53:27.107 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.107 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.107 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:53:27.108 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.849663+0000", 2024-09-21T10:53:27.108 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1222e-05 2024-09-21T10:53:27.108 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.108 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.108 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.108 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.114507+0000", 2024-09-21T10:53:27.108 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.264844214 2024-09-21T10:53:27.108 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.108 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.108 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:53:27.109 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.114727+0000", 2024-09-21T10:53:27.109 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00021980900000000001 2024-09-21T10:53:27.109 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.109 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.109 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:53:27.109 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.114900+0000", 2024-09-21T10:53:27.109 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017282199999999999 2024-09-21T10:53:27.109 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.109 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.109 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:53:27.109 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.123746+0000", 2024-09-21T10:53:27.109 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0088463229999999997 2024-09-21T10:53:27.109 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.109 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.110 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:53:27.110 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.123761+0000", 2024-09-21T10:53:27.110 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4737e-05 2024-09-21T10:53:27.110 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.110 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.110 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:53:27.110 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.140254+0000", 2024-09-21T10:53:27.110 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.016493628999999999 2024-09-21T10:53:27.110 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.110 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.110 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:53:27.110 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.140379+0000", 2024-09-21T10:53:27.110 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012441000000000001 2024-09-21T10:53:27.111 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.111 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.111 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:53:27.111 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.143704+0000", 2024-09-21T10:53:27.111 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0033249410000000001 2024-09-21T10:53:27.111 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.111 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:53:27.111 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.111 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.111 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.111 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:76029 55.8 55:192a45d5:::benchmark_data_smithi070_294394_object76028:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1427)", 2024-09-21T10:53:27.111 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:49:43.887515+0000", 2024-09-21T10:53:27.111 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 223.21092508500001, 2024-09-21T10:53:27.112 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2596707930000002, 2024-09-21T10:53:27.112 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:53:27.112 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:53:27.112 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:53:27.112 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:53:27.112 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:53:27.112 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:53:27.112 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 76029 2024-09-21T10:53:27.112 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.112 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:53:27.112 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.112 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:53:27.112 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887515+0000", 2024-09-21T10:53:27.113 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.113 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.113 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.113 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:53:27.113 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887515+0000", 2024-09-21T10:53:27.113 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.113 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.113 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.113 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:53:27.113 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887518+0000", 2024-09-21T10:53:27.113 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5490000000000001e-06 2024-09-21T10:53:27.113 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.113 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.113 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:53:27.114 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887543+0000", 2024-09-21T10:53:27.114 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5434000000000001e-05 2024-09-21T10:53:27.114 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.114 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.114 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:53:27.114 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887565+0000", 2024-09-21T10:53:27.114 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2099e-05 2024-09-21T10:53:27.114 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.114 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.114 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:53:27.114 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887575+0000", 2024-09-21T10:53:27.114 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7860000000000008e-06 2024-09-21T10:53:27.114 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.115 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.115 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.115 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887646+0000", 2024-09-21T10:53:27.115 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1037000000000004e-05 2024-09-21T10:53:27.115 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.115 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.115 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:53:27.115 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887658+0000", 2024-09-21T10:53:27.115 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1302e-05 2024-09-21T10:53:27.115 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.115 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.115 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.115 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.849687+0000", 2024-09-21T10:53:27.115 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.96202977000000001 2024-09-21T10:53:27.116 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.116 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.116 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:53:27.116 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.849695+0000", 2024-09-21T10:53:27.116 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4089999999999997e-06 2024-09-21T10:53:27.116 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.116 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.116 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.116 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.120438+0000", 2024-09-21T10:53:27.116 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2707431630000001 2024-09-21T10:53:27.116 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.116 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.116 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:53:27.116 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.120627+0000", 2024-09-21T10:53:27.116 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018910699999999999 2024-09-21T10:53:27.117 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.117 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.117 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:53:27.117 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:27.118 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.120803+0000", 2024-09-21T10:53:27.118 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017566500000000001 2024-09-21T10:53:27.118 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.118 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.118 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:53:27.118 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.127153+0000", 2024-09-21T10:53:27.119 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0063504479999999999 2024-09-21T10:53:27.119 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.119 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.119 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:53:27.119 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.127173+0000", 2024-09-21T10:53:27.119 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0123999999999999e-05 2024-09-21T10:53:27.119 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.119 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.119 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:53:27.119 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.143773+0000", 2024-09-21T10:53:27.119 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.016599635000000001 2024-09-21T10:53:27.119 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.119 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.120 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:53:27.120 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.143894+0000", 2024-09-21T10:53:27.120 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012071899999999999 2024-09-21T10:53:27.120 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.120 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.120 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:53:27.120 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.147186+0000", 2024-09-21T10:53:27.120 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0032925459999999999 2024-09-21T10:53:27.120 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.120 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:53:27.120 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.120 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.120 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.120 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:76034 55.8 55:1ffe6a40:::benchmark_data_smithi070_294394_object76033:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1427)", 2024-09-21T10:53:27.121 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:49:43.890819+0000", 2024-09-21T10:53:27.121 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 223.207621088, 2024-09-21T10:53:27.121 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2599814199999999, 2024-09-21T10:53:27.121 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:53:27.121 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:53:27.121 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:53:27.121 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:53:27.121 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:53:27.121 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:53:27.121 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 76034 2024-09-21T10:53:27.121 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.121 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:53:27.122 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.122 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:53:27.122 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.890819+0000", 2024-09-21T10:53:27.122 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.122 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.122 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.122 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:53:27.122 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.890819+0000", 2024-09-21T10:53:27.122 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.122 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.122 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.122 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:53:27.123 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.890822+0000", 2024-09-21T10:53:27.123 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5730000000000002e-06 2024-09-21T10:53:27.123 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.123 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.123 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:53:27.123 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.890851+0000", 2024-09-21T10:53:27.123 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.919e-05 2024-09-21T10:53:27.123 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.123 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.123 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:53:27.123 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.890873+0000", 2024-09-21T10:53:27.123 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1478999999999999e-05 2024-09-21T10:53:27.123 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.124 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.124 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:53:27.124 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.890884+0000", 2024-09-21T10:53:27.124 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0896e-05 2024-09-21T10:53:27.124 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.124 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.124 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.124 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.890961+0000", 2024-09-21T10:53:27.124 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7788999999999997e-05 2024-09-21T10:53:27.124 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.124 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.124 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:53:27.124 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.890973+0000", 2024-09-21T10:53:27.125 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2130000000000001e-05 2024-09-21T10:53:27.125 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.125 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.125 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.125 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.849716+0000", 2024-09-21T10:53:27.125 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.95874219199999999 2024-09-21T10:53:27.125 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.125 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.125 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:53:27.125 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.849723+0000", 2024-09-21T10:53:27.125 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2060000000000001e-06 2024-09-21T10:53:27.125 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.125 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.126 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.126 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.123858+0000", 2024-09-21T10:53:27.126 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.27413475 2024-09-21T10:53:27.126 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.126 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.126 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:53:27.126 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.124033+0000", 2024-09-21T10:53:27.126 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000175067 2024-09-21T10:53:27.126 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.126 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.126 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:53:27.126 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.124221+0000", 2024-09-21T10:53:27.127 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000188593 2024-09-21T10:53:27.127 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.127 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.127 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:53:27.127 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.127229+0000", 2024-09-21T10:53:27.127 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.003007895 2024-09-21T10:53:27.127 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.127 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.127 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:53:27.127 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.130344+0000", 2024-09-21T10:53:27.127 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0031151070000000002 2024-09-21T10:53:27.127 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.127 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.127 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:53:27.128 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.147244+0000", 2024-09-21T10:53:27.128 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.016899687 2024-09-21T10:53:27.128 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.128 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.128 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:53:27.128 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.147315+0000", 2024-09-21T10:53:27.128 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1049000000000004e-05 2024-09-21T10:53:27.128 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.128 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.128 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:53:27.128 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.150801+0000", 2024-09-21T10:53:27.129 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.003485817 2024-09-21T10:53:27.129 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.129 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:53:27.129 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.129 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.129 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.129 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:129295 55.1 e1446/1440 55:8efdf0b1:::benchmark_data_smithi070_294394_object23954:head v 1446'8143, mlcod=0'0)", 2024-09-21T10:53:27.129 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:22.707277+0000", 2024-09-21T10:53:27.129 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 124.391163209, 2024-09-21T10:53:27.129 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27558063199999999, 2024-09-21T10:53:27.130 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:53:27.130 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:53:27.130 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:53:27.130 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:53:27.130 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.130 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:53:27.130 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.707277+0000", 2024-09-21T10:53:27.130 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.130 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.131 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.131 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:53:27.131 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.707277+0000", 2024-09-21T10:53:27.131 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.131 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.131 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.131 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:53:27.131 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.707278+0000", 2024-09-21T10:53:27.131 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0299999999999997e-07 2024-09-21T10:53:27.131 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.131 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.131 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:53:27.131 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.707285+0000", 2024-09-21T10:53:27.131 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8850000000000001e-06 2024-09-21T10:53:27.132 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.132 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.132 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:53:27.132 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.707296+0000", 2024-09-21T10:53:27.132 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0993e-05 2024-09-21T10:53:27.132 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.132 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.132 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:53:27.132 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.707303+0000", 2024-09-21T10:53:27.132 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7120000000000003e-06 2024-09-21T10:53:27.132 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.132 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.132 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.133 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.945300+0000", 2024-09-21T10:53:27.133 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.23799742600000001 2024-09-21T10:53:27.133 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.133 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.133 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:53:27.133 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.945329+0000", 2024-09-21T10:53:27.133 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8379999999999999e-05 2024-09-21T10:53:27.133 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.133 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.133 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:53:27.133 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.982817+0000", 2024-09-21T10:53:27.133 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.037488847999999998 2024-09-21T10:53:27.133 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.134 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.134 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:53:27.134 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.982858+0000", 2024-09-21T10:53:27.134 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0485000000000003e-05 2024-09-21T10:53:27.134 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.134 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:53:27.134 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.134 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.134 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.134 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:129296 55.8 55:1cab724d:::benchmark_data_smithi070_294394_object23955:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:53:27.134 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:22.708169+0000", 2024-09-21T10:53:27.134 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 124.390271771, 2024-09-21T10:53:27.134 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.24355597700000001, 2024-09-21T10:53:27.135 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:53:27.135 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:53:27.135 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:53:27.135 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:53:27.135 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:53:27.135 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:53:27.135 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 129296 2024-09-21T10:53:27.135 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.135 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:53:27.135 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.135 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:53:27.135 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.708169+0000", 2024-09-21T10:53:27.135 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.136 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.136 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.136 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:53:27.136 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.708169+0000", 2024-09-21T10:53:27.136 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.136 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.136 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.136 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:53:27.136 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.708171+0000", 2024-09-21T10:53:27.136 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0150000000000002e-06 2024-09-21T10:53:27.136 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.136 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.136 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:53:27.137 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.708177+0000", 2024-09-21T10:53:27.137 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.852e-06 2024-09-21T10:53:27.137 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.137 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.137 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:53:27.137 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.708190+0000", 2024-09-21T10:53:27.137 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3148e-05 2024-09-21T10:53:27.137 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.137 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.137 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:53:27.137 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.708197+0000", 2024-09-21T10:53:27.137 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0040000000000002e-06 2024-09-21T10:53:27.137 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.137 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.137 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.138 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.708739+0000", 2024-09-21T10:53:27.138 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00054260300000000001 2024-09-21T10:53:27.138 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.138 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.138 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:53:27.138 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.708879+0000", 2024-09-21T10:53:27.138 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013970700000000001 2024-09-21T10:53:27.138 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.138 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.138 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:53:27.138 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.709019+0000", 2024-09-21T10:53:27.138 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013960900000000001 2024-09-21T10:53:27.139 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.139 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.139 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:53:27.139 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.713167+0000", 2024-09-21T10:53:27.139 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0041479009999999998 2024-09-21T10:53:27.139 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.139 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.139 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:53:27.139 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.714761+0000", 2024-09-21T10:53:27.139 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001594729 2024-09-21T10:53:27.139 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.139 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.139 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:53:27.140 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.949456+0000", 2024-09-21T10:53:27.140 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.23469445 2024-09-21T10:53:27.140 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.140 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.140 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:53:27.140 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.949604+0000", 2024-09-21T10:53:27.140 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000147894 2024-09-21T10:53:27.140 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.140 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.140 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:53:27.140 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.951725+0000", 2024-09-21T10:53:27.140 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0021210650000000001 2024-09-21T10:53:27.140 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.140 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:53:27.141 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.141 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.141 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.141 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:129298 55.5 55:aae0d80d:::benchmark_data_smithi070_294394_object23957:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:53:27.141 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:22.714780+0000", 2024-09-21T10:53:27.141 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 124.38366070799999, 2024-09-21T10:53:27.141 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.23622679199999999, 2024-09-21T10:53:27.141 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:53:27.141 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:53:27.141 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:53:27.141 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:53:27.141 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:53:27.141 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:53:27.142 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 129298 2024-09-21T10:53:27.142 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.142 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:53:27.142 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.142 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:53:27.142 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.714780+0000", 2024-09-21T10:53:27.142 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.142 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.142 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.142 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:53:27.142 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.714780+0000", 2024-09-21T10:53:27.142 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.142 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.143 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.143 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:53:27.143 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.714782+0000", 2024-09-21T10:53:27.143 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9489999999999999e-06 2024-09-21T10:53:27.143 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.143 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.143 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:53:27.143 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.714788+0000", 2024-09-21T10:53:27.143 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1210000000000003e-06 2024-09-21T10:53:27.143 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.143 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.143 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:53:27.143 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.714800+0000", 2024-09-21T10:53:27.144 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.219e-05 2024-09-21T10:53:27.144 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.144 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.144 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:53:27.144 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.714808+0000", 2024-09-21T10:53:27.144 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7749999999999993e-06 2024-09-21T10:53:27.144 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.144 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.144 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.144 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.714867+0000", 2024-09-21T10:53:27.144 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8956999999999998e-05 2024-09-21T10:53:27.144 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.144 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.144 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:53:27.145 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.714987+0000", 2024-09-21T10:53:27.145 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000120728 2024-09-21T10:53:27.145 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.145 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.145 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,3,6", 2024-09-21T10:53:27.145 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.715375+0000", 2024-09-21T10:53:27.145 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000387814 2024-09-21T10:53:27.145 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.145 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.145 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:53:27.145 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.947741+0000", 2024-09-21T10:53:27.145 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.232365505 2024-09-21T10:53:27.145 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.145 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.145 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:53:27.146 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.947799+0000", 2024-09-21T10:53:27.146 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7720999999999998e-05 2024-09-21T10:53:27.146 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.146 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.146 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:53:27.146 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.947853+0000", 2024-09-21T10:53:27.146 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4453000000000002e-05 2024-09-21T10:53:27.146 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.146 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.146 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:53:27.146 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.950846+0000", 2024-09-21T10:53:27.146 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0029934509999999998 2024-09-21T10:53:27.147 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.147 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.147 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:53:27.147 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.950899+0000", 2024-09-21T10:53:27.147 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2769000000000002e-05 2024-09-21T10:53:27.147 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.147 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.147 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:53:27.147 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.951007+0000", 2024-09-21T10:53:27.147 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000107359 2024-09-21T10:53:27.147 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.148 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:53:27.148 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.148 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.148 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.148 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "MOSDPGPush(55.2 1446/1434 [PushOp(55:42275384:::benchmark_data_smithi070_294394_object23949:head, version: 1416'1519, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(55:42275384:::benchmark_data_smithi070_294394_object23949:head@1416'1519, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T10:53:27.148 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:22.715198+0000", 2024-09-21T10:53:27.148 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 124.383242818, 2024-09-21T10:53:27.148 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.233190694, 2024-09-21T10:53:27.148 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:53:27.148 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:53:27.148 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "started", 2024-09-21T10:53:27.148 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:53:27.149 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.149 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:53:27.149 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.715198+0000", 2024-09-21T10:53:27.149 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.149 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.149 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.149 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:53:27.149 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.715198+0000", 2024-09-21T10:53:27.149 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.149 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.149 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.149 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:53:27.149 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.715198+0000", 2024-09-21T10:53:27.149 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6400000000000002e-07 2024-09-21T10:53:27.150 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.150 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.150 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:53:27.150 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.715228+0000", 2024-09-21T10:53:27.150 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9607e-05 2024-09-21T10:53:27.150 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.150 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.150 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:53:27.150 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.715247+0000", 2024-09-21T10:53:27.150 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8816000000000001e-05 2024-09-21T10:53:27.150 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.150 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.150 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:53:27.151 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.715262+0000", 2024-09-21T10:53:27.151 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5613000000000001e-05 2024-09-21T10:53:27.151 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.151 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.151 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.151 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.715366+0000", 2024-09-21T10:53:27.151 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000103769 2024-09-21T10:53:27.151 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.151 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.151 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:53:27.151 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.715386+0000", 2024-09-21T10:53:27.151 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0136000000000002e-05 2024-09-21T10:53:27.151 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.152 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.152 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:53:27.152 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.948388+0000", 2024-09-21T10:53:27.152 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.233002089 2024-09-21T10:53:27.152 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.152 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:53:27.152 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.152 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.152 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.152 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "MOSDPGPushReply(55.8 1446/1428 [PushReplyOp(55:1a4fe3ae:::benchmark_data_smithi070_294394_object37264:head)])", 2024-09-21T10:53:27.152 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:22.716411+0000", 2024-09-21T10:53:27.152 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 124.382029813, 2024-09-21T10:53:27.152 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.23640767900000001, 2024-09-21T10:53:27.153 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:53:27.153 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:53:27.153 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:53:27.153 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:53:27.153 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.153 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:53:27.153 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.716411+0000", 2024-09-21T10:53:27.153 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.153 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.153 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.153 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:53:27.153 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.716411+0000", 2024-09-21T10:53:27.153 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.154 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.154 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.154 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:53:27.154 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.716411+0000", 2024-09-21T10:53:27.154 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.85e-07 2024-09-21T10:53:27.154 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.154 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.154 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:53:27.154 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.716416+0000", 2024-09-21T10:53:27.154 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8189999999999999e-06 2024-09-21T10:53:27.154 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.154 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.154 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:53:27.155 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.945276+0000", 2024-09-21T10:53:27.155 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.22885976299999999 2024-09-21T10:53:27.155 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.155 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.155 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:53:27.155 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.945289+0000", 2024-09-21T10:53:27.155 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3278000000000001e-05 2024-09-21T10:53:27.155 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.155 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.155 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.155 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.945376+0000", 2024-09-21T10:53:27.155 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6266000000000004e-05 2024-09-21T10:53:27.155 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.156 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.156 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:53:27.156 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.952818+0000", 2024-09-21T10:53:27.156 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0074427679999999998 2024-09-21T10:53:27.156 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.156 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:53:27.156 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.156 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.156 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.156 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:129300 55.5 55:adccdd22:::benchmark_data_smithi070_294394_object23959:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:53:27.156 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:22.716445+0000", 2024-09-21T10:53:27.156 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 124.38199541100001, 2024-09-21T10:53:27.156 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.24455037499999999, 2024-09-21T10:53:27.157 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:53:27.157 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:53:27.157 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:53:27.157 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:53:27.157 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:53:27.157 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:53:27.157 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 129300 2024-09-21T10:53:27.157 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.157 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:53:27.157 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.157 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:53:27.157 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.716445+0000", 2024-09-21T10:53:27.157 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.157 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.158 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.158 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:53:27.158 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.716445+0000", 2024-09-21T10:53:27.158 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.158 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.158 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.158 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:53:27.158 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.716446+0000", 2024-09-21T10:53:27.158 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.322e-06 2024-09-21T10:53:27.158 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.158 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.158 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:53:27.159 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.716451+0000", 2024-09-21T10:53:27.159 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2429999999999999e-06 2024-09-21T10:53:27.159 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.159 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.159 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:53:27.159 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.945269+0000", 2024-09-21T10:53:27.159 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.22881818800000001 2024-09-21T10:53:27.159 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.159 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.159 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:53:27.159 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.945287+0000", 2024-09-21T10:53:27.159 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8023999999999999e-05 2024-09-21T10:53:27.159 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.160 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.160 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.160 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.951033+0000", 2024-09-21T10:53:27.160 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0057462950000000002 2024-09-21T10:53:27.160 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.160 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.160 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:53:27.160 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.951142+0000", 2024-09-21T10:53:27.160 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000108631 2024-09-21T10:53:27.160 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.160 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.160 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,3,6", 2024-09-21T10:53:27.160 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.951254+0000", 2024-09-21T10:53:27.161 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000112182 2024-09-21T10:53:27.161 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.161 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.161 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:53:27.161 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.960562+0000", 2024-09-21T10:53:27.161 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0093075840000000007 2024-09-21T10:53:27.161 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.161 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.161 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:53:27.161 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.960608+0000", 2024-09-21T10:53:27.161 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6307999999999998e-05 2024-09-21T10:53:27.161 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.161 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.162 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:53:27.162 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.960650+0000", 2024-09-21T10:53:27.162 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.1752000000000003e-05 2024-09-21T10:53:27.162 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.162 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.163 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:53:27.163 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.960827+0000", 2024-09-21T10:53:27.163 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017739700000000001 2024-09-21T10:53:27.163 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.163 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.163 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:53:27.163 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.960880+0000", 2024-09-21T10:53:27.164 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3146000000000002e-05 2024-09-21T10:53:27.164 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.164 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.164 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:53:27.164 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.960995+0000", 2024-09-21T10:53:27.164 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000115303 2024-09-21T10:53:27.164 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.164 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:53:27.164 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.164 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.164 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.164 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop_reply(client.35670.0:129298 55.5 e1446/1434 ondisk, result = 0)", 2024-09-21T10:53:27.165 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:22.718413+0000", 2024-09-21T10:53:27.165 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 124.38002793299999, 2024-09-21T10:53:27.165 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.229390868, 2024-09-21T10:53:27.165 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:53:27.165 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:53:27.165 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "started", 2024-09-21T10:53:27.165 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:53:27.165 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.165 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:53:27.165 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.718413+0000", 2024-09-21T10:53:27.165 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.165 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.165 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.166 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:53:27.166 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.718413+0000", 2024-09-21T10:53:27.166 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.166 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.166 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.166 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:53:27.166 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.718414+0000", 2024-09-21T10:53:27.166 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.46e-06 2024-09-21T10:53:27.166 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.166 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.166 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:53:27.166 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.718421+0000", 2024-09-21T10:53:27.166 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4850000000000003e-06 2024-09-21T10:53:27.167 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.167 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.167 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:53:27.167 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.945399+0000", 2024-09-21T10:53:27.167 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.226977014 2024-09-21T10:53:27.167 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.167 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.167 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:53:27.167 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.945406+0000", 2024-09-21T10:53:27.167 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0609999999999998e-06 2024-09-21T10:53:27.167 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.167 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.167 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.167 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.947791+0000", 2024-09-21T10:53:27.168 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0023850149999999999 2024-09-21T10:53:27.168 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.168 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.168 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:53:27.168 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.947795+0000", 2024-09-21T10:53:27.168 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7110000000000003e-06 2024-09-21T10:53:27.168 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.168 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.168 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:53:27.168 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.947803+0000", 2024-09-21T10:53:27.169 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1219999999999995e-06 2024-09-21T10:53:27.169 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.169 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:53:27.169 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.169 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.169 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.169 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:130858 55.8 55:178fc653:::benchmark_data_smithi070_294394_object25517:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:53:27.169 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:26.393677+0000", 2024-09-21T10:53:27.169 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 120.70476304499999, 2024-09-21T10:53:27.169 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0198451989999999, 2024-09-21T10:53:27.169 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:53:27.169 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:53:27.169 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:53:27.169 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:53:27.170 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:53:27.170 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:53:27.170 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 130858 2024-09-21T10:53:27.170 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.170 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:53:27.170 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.170 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:53:27.170 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.393677+0000", 2024-09-21T10:53:27.170 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.170 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.170 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.170 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:53:27.170 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.393677+0000", 2024-09-21T10:53:27.171 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.171 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.171 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.171 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:53:27.171 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.393679+0000", 2024-09-21T10:53:27.171 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7349999999999999e-06 2024-09-21T10:53:27.171 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.171 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.171 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:53:27.171 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.393686+0000", 2024-09-21T10:53:27.171 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4930000000000003e-06 2024-09-21T10:53:27.171 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.171 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.171 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:53:27.172 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.393701+0000", 2024-09-21T10:53:27.172 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5466999999999999e-05 2024-09-21T10:53:27.172 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.172 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.172 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:53:27.172 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.393710+0000", 2024-09-21T10:53:27.172 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.8789999999999992e-06 2024-09-21T10:53:27.172 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.172 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.172 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.172 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.395855+0000", 2024-09-21T10:53:27.172 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0021445420000000001 2024-09-21T10:53:27.172 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.172 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.173 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:53:27.173 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.395988+0000", 2024-09-21T10:53:27.173 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013296900000000001 2024-09-21T10:53:27.174 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.174 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.174 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:53:27.174 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.396195+0000", 2024-09-21T10:53:27.174 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000207035 2024-09-21T10:53:27.174 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.174 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.175 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.175 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.445846+0000", 2024-09-21T10:53:27.175 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.049651140000000003 2024-09-21T10:53:27.175 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.175 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.175 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:53:27.175 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.445859+0000", 2024-09-21T10:53:27.175 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3772e-05 2024-09-21T10:53:27.175 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.175 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.175 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.176 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.413508+0000", 2024-09-21T10:53:27.176 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.96764826299999995 2024-09-21T10:53:27.176 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.176 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.176 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:53:27.176 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.413523+0000", 2024-09-21T10:53:27.176 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4904e-05 2024-09-21T10:53:27.176 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.176 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:53:27.176 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.176 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.176 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.176 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:130866 55.8 55.11526d38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:53:27.176 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:26.400243+0000", 2024-09-21T10:53:27.177 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 120.69819709399999, 2024-09-21T10:53:27.177 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.013316393, 2024-09-21T10:53:27.177 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:53:27.177 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:53:27.177 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:53:27.177 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:53:27.177 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:53:27.177 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:53:27.177 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 130866 2024-09-21T10:53:27.177 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.177 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:53:27.177 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.177 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:53:27.177 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.400243+0000", 2024-09-21T10:53:27.178 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.178 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.178 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.178 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:53:27.178 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.400243+0000", 2024-09-21T10:53:27.178 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.178 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.178 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.178 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:53:27.178 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.400245+0000", 2024-09-21T10:53:27.178 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0829999999999998e-06 2024-09-21T10:53:27.178 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.178 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.179 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:53:27.179 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.400251+0000", 2024-09-21T10:53:27.179 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3410000000000002e-06 2024-09-21T10:53:27.179 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.179 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.179 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:53:27.179 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.400261+0000", 2024-09-21T10:53:27.179 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7210000000000006e-06 2024-09-21T10:53:27.179 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.179 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.179 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:53:27.179 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.400267+0000", 2024-09-21T10:53:27.179 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8560000000000001e-06 2024-09-21T10:53:27.180 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.180 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.180 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.180 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.446742+0000", 2024-09-21T10:53:27.180 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.046474840000000003 2024-09-21T10:53:27.180 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.180 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.180 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:53:27.180 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.446887+0000", 2024-09-21T10:53:27.180 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014484099999999999 2024-09-21T10:53:27.180 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.181 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.181 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.181 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.413550+0000", 2024-09-21T10:53:27.181 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.96666295099999999 2024-09-21T10:53:27.181 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.181 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.181 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:53:27.181 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.413560+0000", 2024-09-21T10:53:27.181 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7599999999999997e-06 2024-09-21T10:53:27.181 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.181 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:53:27.181 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.181 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.182 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.182 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:130872 55.8 55.862244f8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1447)", 2024-09-21T10:53:27.182 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:26.466075+0000", 2024-09-21T10:53:27.182 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 120.632365765, 2024-09-21T10:53:27.182 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.94751783499999997, 2024-09-21T10:53:27.182 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:53:27.182 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:53:27.182 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:53:27.182 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:53:27.182 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:53:27.182 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:53:27.183 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 130872 2024-09-21T10:53:27.183 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.183 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:53:27.183 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.183 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:53:27.183 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.466075+0000", 2024-09-21T10:53:27.183 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.183 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.183 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.183 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:53:27.183 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.466075+0000", 2024-09-21T10:53:27.183 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.183 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.184 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.184 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:53:27.184 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.466077+0000", 2024-09-21T10:53:27.184 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9479999999999998e-06 2024-09-21T10:53:27.184 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.184 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.184 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:53:27.184 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.466085+0000", 2024-09-21T10:53:27.184 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3850000000000002e-06 2024-09-21T10:53:27.184 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.184 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.184 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:53:27.185 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.466104+0000", 2024-09-21T10:53:27.185 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9218000000000001e-05 2024-09-21T10:53:27.185 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.185 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.185 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:53:27.185 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.466115+0000", 2024-09-21T10:53:27.185 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0271e-05 2024-09-21T10:53:27.185 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.185 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.185 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.185 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.466576+0000", 2024-09-21T10:53:27.186 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00046172299999999998 2024-09-21T10:53:27.186 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.186 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.186 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:53:27.186 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.466585+0000", 2024-09-21T10:53:27.186 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7509999999999997e-06 2024-09-21T10:53:27.186 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.186 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.186 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.186 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.413582+0000", 2024-09-21T10:53:27.186 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.946997333 2024-09-21T10:53:27.186 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.186 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.187 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:53:27.187 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.413593+0000", 2024-09-21T10:53:27.187 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0206e-05 2024-09-21T10:53:27.187 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.187 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:53:27.187 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.187 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.187 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.187 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:130874 55.5 55.1ccdde55 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1447)", 2024-09-21T10:53:27.187 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:26.472197+0000", 2024-09-21T10:53:27.187 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 120.626243035, 2024-09-21T10:53:27.187 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.94240631200000002, 2024-09-21T10:53:27.187 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:53:27.187 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:53:27.188 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:53:27.188 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:53:27.188 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:53:27.188 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:53:27.188 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 130874 2024-09-21T10:53:27.188 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.188 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:53:27.188 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.188 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:53:27.188 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.472197+0000", 2024-09-21T10:53:27.188 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.188 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.188 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.188 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:53:27.188 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.472197+0000", 2024-09-21T10:53:27.189 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.189 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.189 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.189 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:53:27.189 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.472200+0000", 2024-09-21T10:53:27.189 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3329999999999999e-06 2024-09-21T10:53:27.189 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.189 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.189 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:53:27.189 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.472209+0000", 2024-09-21T10:53:27.189 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.884e-06 2024-09-21T10:53:27.189 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.189 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.190 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:53:27.190 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.472231+0000", 2024-09-21T10:53:27.190 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2211000000000001e-05 2024-09-21T10:53:27.190 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.190 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.190 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:53:27.190 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.472242+0000", 2024-09-21T10:53:27.190 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1233000000000001e-05 2024-09-21T10:53:27.190 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.190 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.190 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.190 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.472325+0000", 2024-09-21T10:53:27.191 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3275000000000005e-05 2024-09-21T10:53:27.191 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.191 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.191 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:53:27.191 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.472338+0000", 2024-09-21T10:53:27.191 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2302e-05 2024-09-21T10:53:27.191 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.191 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.191 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.191 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.414595+0000", 2024-09-21T10:53:27.192 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.942257601 2024-09-21T10:53:27.192 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.192 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.192 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:53:27.192 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.414604+0000", 2024-09-21T10:53:27.192 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.473e-06 2024-09-21T10:53:27.192 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.192 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:53:27.192 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.192 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.192 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.193 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:156258 55.c e1464/1448 55:39292414:::benchmark_data_smithi070_294394_object50917:head v 1464'9744, mlcod=0'0)", 2024-09-21T10:53:27.193 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.338132+0000", 2024-09-21T10:53:27.193 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 62.760308868999999, 2024-09-21T10:53:27.193 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.15064587099999999, 2024-09-21T10:53:27.193 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:53:27.193 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:53:27.193 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:53:27.193 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:53:27.193 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.193 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:53:27.193 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.338132+0000", 2024-09-21T10:53:27.193 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.193 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.194 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.194 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:53:27.194 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.338132+0000", 2024-09-21T10:53:27.194 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.194 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.194 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.194 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:53:27.194 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.338133+0000", 2024-09-21T10:53:27.194 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7900000000000007e-07 2024-09-21T10:53:27.194 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.194 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.194 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:53:27.195 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.338140+0000", 2024-09-21T10:53:27.195 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1740000000000003e-06 2024-09-21T10:53:27.195 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.195 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.195 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:53:27.195 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.338152+0000", 2024-09-21T10:53:27.195 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2215000000000001e-05 2024-09-21T10:53:27.195 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.195 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.195 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:53:27.195 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.338159+0000", 2024-09-21T10:53:27.195 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3520000000000001e-06 2024-09-21T10:53:27.195 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.196 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.196 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.196 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.338240+0000", 2024-09-21T10:53:27.196 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1086000000000003e-05 2024-09-21T10:53:27.196 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.196 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.196 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:53:27.196 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.338267+0000", 2024-09-21T10:53:27.196 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6194999999999999e-05 2024-09-21T10:53:27.196 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.196 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.196 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:53:27.196 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.488714+0000", 2024-09-21T10:53:27.196 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.15044706499999999 2024-09-21T10:53:27.197 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.197 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.197 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:53:27.197 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.488777+0000", 2024-09-21T10:53:27.197 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3805000000000003e-05 2024-09-21T10:53:27.197 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.197 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:53:27.197 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.197 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.197 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.197 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:156268 55.f e1464/1448 55:f53808b0:::benchmark_data_smithi070_294394_object50927:head v 1464'9749, mlcod=0'0)", 2024-09-21T10:53:27.197 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.340846+0000", 2024-09-21T10:53:27.197 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 62.757594949000001, 2024-09-21T10:53:27.198 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.158773426, 2024-09-21T10:53:27.198 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:53:27.198 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:53:27.198 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:53:27.198 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:53:27.198 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.198 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:53:27.198 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.340846+0000", 2024-09-21T10:53:27.198 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.198 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.198 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.198 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:53:27.198 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.340846+0000", 2024-09-21T10:53:27.199 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.199 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.199 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.199 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:53:27.199 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.340847+0000", 2024-09-21T10:53:27.199 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3650000000000001e-06 2024-09-21T10:53:27.199 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.199 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.199 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:53:27.199 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.340855+0000", 2024-09-21T10:53:27.199 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7100000000000007e-06 2024-09-21T10:53:27.199 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.199 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.200 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:53:27.200 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.340868+0000", 2024-09-21T10:53:27.200 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3312000000000001e-05 2024-09-21T10:53:27.200 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.200 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.200 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:53:27.200 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.340876+0000", 2024-09-21T10:53:27.200 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5320000000000006e-06 2024-09-21T10:53:27.200 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.200 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.200 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.200 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.478844+0000", 2024-09-21T10:53:27.200 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13796760599999999 2024-09-21T10:53:27.200 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.201 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.201 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:53:27.201 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.478891+0000", 2024-09-21T10:53:27.201 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6912000000000001e-05 2024-09-21T10:53:27.201 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.201 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.201 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:53:27.201 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.499553+0000", 2024-09-21T10:53:27.201 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.020662064000000001 2024-09-21T10:53:27.201 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.201 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.202 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:53:27.202 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.499619+0000", 2024-09-21T10:53:27.202 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5925000000000003e-05 2024-09-21T10:53:27.202 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.202 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:53:27.202 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.202 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.202 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.202 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:156262 55.3 e1464/1448)", 2024-09-21T10:53:27.202 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.343780+0000", 2024-09-21T10:53:27.202 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 62.754660158, 2024-09-21T10:53:27.202 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13981411199999999, 2024-09-21T10:53:27.203 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:53:27.203 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:53:27.203 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:53:27.203 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:53:27.203 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.203 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:53:27.203 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.343780+0000", 2024-09-21T10:53:27.203 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.203 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.203 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.203 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:53:27.203 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.343780+0000", 2024-09-21T10:53:27.203 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.204 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.204 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.204 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:53:27.204 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.343782+0000", 2024-09-21T10:53:27.204 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.28e-06 2024-09-21T10:53:27.204 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.204 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.204 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:53:27.204 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.343790+0000", 2024-09-21T10:53:27.204 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9440000000000005e-06 2024-09-21T10:53:27.204 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.204 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.205 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:53:27.205 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.475488+0000", 2024-09-21T10:53:27.205 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13169877499999999 2024-09-21T10:53:27.205 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.205 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.205 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:53:27.205 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.475507+0000", 2024-09-21T10:53:27.205 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8768000000000001e-05 2024-09-21T10:53:27.205 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.205 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.206 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.206 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.483581+0000", 2024-09-21T10:53:27.206 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0080743680000000002 2024-09-21T10:53:27.206 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.206 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.206 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:53:27.206 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.483594+0000", 2024-09-21T10:53:27.206 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2977e-05 2024-09-21T10:53:27.206 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.206 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:53:27.206 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.207 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.207 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.207 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:156269 55.f e1464/1448 55:fb867c57:::benchmark_data_smithi070_294394_object50928:head v 1464'9750, mlcod=0'0)", 2024-09-21T10:53:27.207 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.344388+0000", 2024-09-21T10:53:27.207 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 62.754052831999999, 2024-09-21T10:53:27.207 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.156159983, 2024-09-21T10:53:27.207 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:53:27.207 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:53:27.207 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:53:27.207 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:53:27.207 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.207 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:53:27.207 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.344388+0000", 2024-09-21T10:53:27.208 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.208 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.208 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.208 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:53:27.208 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.344388+0000", 2024-09-21T10:53:27.208 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.208 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.208 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.208 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:53:27.208 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.344388+0000", 2024-09-21T10:53:27.208 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9999999999999996e-07 2024-09-21T10:53:27.208 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.208 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.208 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:53:27.209 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.344395+0000", 2024-09-21T10:53:27.209 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7240000000000003e-06 2024-09-21T10:53:27.209 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.209 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.209 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:53:27.209 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.473817+0000", 2024-09-21T10:53:27.209 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.129422075 2024-09-21T10:53:27.209 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.209 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.209 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:53:27.209 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.473834+0000", 2024-09-21T10:53:27.209 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6381000000000001e-05 2024-09-21T10:53:27.209 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.209 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.210 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.210 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.494817+0000", 2024-09-21T10:53:27.210 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.020983841999999999 2024-09-21T10:53:27.210 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.210 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.210 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:53:27.210 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.494841+0000", 2024-09-21T10:53:27.210 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3091999999999998e-05 2024-09-21T10:53:27.210 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.210 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.210 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:53:27.210 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.500504+0000", 2024-09-21T10:53:27.210 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0056635160000000004 2024-09-21T10:53:27.211 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.211 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.211 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:53:27.211 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.500548+0000", 2024-09-21T10:53:27.211 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3553000000000001e-05 2024-09-21T10:53:27.211 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.211 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:53:27.211 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.211 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.211 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.212 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:195880 55.c e1474/1465 55:341b656a:::benchmark_data_smithi070_294394_object90539:head v 1474'12198, mlcod=0'0)", 2024-09-21T10:53:27.212 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:53:26.307538+0000", 2024-09-21T10:53:27.212 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.79090265500000001, 2024-09-21T10:53:27.212 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.122286966, 2024-09-21T10:53:27.212 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:53:27.212 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:53:27.212 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:53:27.212 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:53:27.212 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.212 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:53:27.212 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.307538+0000", 2024-09-21T10:53:27.212 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.212 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.213 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.213 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:53:27.213 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.307538+0000", 2024-09-21T10:53:27.213 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.213 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.213 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.213 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:53:27.213 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.307538+0000", 2024-09-21T10:53:27.213 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8700000000000001e-07 2024-09-21T10:53:27.213 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.213 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.213 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:53:27.213 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.307542+0000", 2024-09-21T10:53:27.213 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0149999999999997e-06 2024-09-21T10:53:27.213 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.214 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.214 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:53:27.214 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.307550+0000", 2024-09-21T10:53:27.214 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4889999999999998e-06 2024-09-21T10:53:27.214 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.214 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.214 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:53:27.214 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.307555+0000", 2024-09-21T10:53:27.214 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4909999999999996e-06 2024-09-21T10:53:27.214 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.214 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.214 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.215 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.314879+0000", 2024-09-21T10:53:27.215 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0073239250000000002 2024-09-21T10:53:27.215 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.215 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.215 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:53:27.215 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.314904+0000", 2024-09-21T10:53:27.215 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4516999999999999e-05 2024-09-21T10:53:27.215 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.215 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.215 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:53:27.215 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.429774+0000", 2024-09-21T10:53:27.215 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.114870376 2024-09-21T10:53:27.215 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.216 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.216 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:53:27.216 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.429825+0000", 2024-09-21T10:53:27.216 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0766000000000001e-05 2024-09-21T10:53:27.216 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.216 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:53:27.216 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.216 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.216 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.216 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:195894 55.c e1474/1465 55:3dc3da2f:::benchmark_data_smithi070_294394_object90553:head v 1474'12199, mlcod=0'0)", 2024-09-21T10:53:27.216 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:53:26.309020+0000", 2024-09-21T10:53:27.216 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.78942059600000003, 2024-09-21T10:53:27.216 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.146994024, 2024-09-21T10:53:27.216 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:53:27.217 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:53:27.217 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:53:27.217 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:53:27.217 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.217 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:53:27.217 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.309020+0000", 2024-09-21T10:53:27.217 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.217 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.217 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.217 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:53:27.217 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.309020+0000", 2024-09-21T10:53:27.218 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:27.218 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.218 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.218 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:53:27.218 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.309021+0000", 2024-09-21T10:53:27.218 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9199999999999999e-07 2024-09-21T10:53:27.218 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.218 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.218 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:53:27.218 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.309028+0000", 2024-09-21T10:53:27.218 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.844e-06 2024-09-21T10:53:27.219 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.219 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.219 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:53:27.219 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.309040+0000", 2024-09-21T10:53:27.219 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2e-05 2024-09-21T10:53:27.219 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.219 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.219 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:53:27.219 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.309048+0000", 2024-09-21T10:53:27.219 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9619999999999993e-06 2024-09-21T10:53:27.219 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.219 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.219 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:27.219 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.443698+0000", 2024-09-21T10:53:27.220 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13465039300000001 2024-09-21T10:53:27.220 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.220 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.220 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:53:27.220 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.443727+0000", 2024-09-21T10:53:27.220 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9249000000000001e-05 2024-09-21T10:53:27.220 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.220 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.220 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:53:27.220 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.455970+0000", 2024-09-21T10:53:27.220 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012242602999999999 2024-09-21T10:53:27.220 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:27.220 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:27.220 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:53:27.221 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.456014+0000", 2024-09-21T10:53:27.221 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3980999999999998e-05 2024-09-21T10:53:27.221 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.221 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:53:27.221 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.221 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:27.221 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:53:27.221 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:53:27.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:27.223 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:27.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:27.324 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:27.363 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:27.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:27.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:27.465 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:27.504 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:27.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:27.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:27.606 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:27.636 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:27.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:27.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:27.738 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:27.778 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:27.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:27.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:27.879 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:27.923 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:27.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:27.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:28.025 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:28.059 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:28.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:28.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:28.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:27 smithi160 ceph-mon[101879]: pgmap v2711: 99 pgs: 2 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 973 MiB data, 8.1 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 916 op/s; 12974/46782 objects degraded (27.733%); 6216/46782 objects misplaced (13.287%); 21 MiB/s, 400 objects/s recovering 2024-09-21T10:53:28.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:27 smithi070 ceph-mon[107385]: pgmap v2711: 99 pgs: 2 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 973 MiB data, 8.1 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 916 op/s; 12974/46782 objects degraded (27.733%); 6216/46782 objects misplaced (13.287%); 21 MiB/s, 400 objects/s recovering 2024-09-21T10:53:28.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:27 smithi070 ceph-mon[112139]: pgmap v2711: 99 pgs: 2 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 973 MiB data, 8.1 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 916 op/s; 12974/46782 objects degraded (27.733%); 6216/46782 objects misplaced (13.287%); 21 MiB/s, 400 objects/s recovering 2024-09-21T10:53:28.161 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:28.193 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:28.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:28.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:28.294 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:28.327 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:28.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:28.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:28.429 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:28.464 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:28.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:28.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:28.489 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:53:28.567 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:28.598 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:28.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:28.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:28.700 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:28.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:28.739 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:28.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:28.840 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:28.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:28 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 12974/46782 objects degraded (27.733%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:28.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:28 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 12974/46782 objects degraded (27.733%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:28.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:28.883 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:28.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:28.984 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:29.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:29.015 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:29.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:29.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:28 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 12974/46782 objects degraded (27.733%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:29.117 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:29.159 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:29.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:29.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:29.261 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:29.291 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:29.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:29.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:29.393 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:29.421 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:29.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:29.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:29.523 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:29.556 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:29.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:29.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:29.657 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:29.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:29.687 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:29.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:29.788 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:29.826 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:29.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:29.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:29.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:29 smithi070 ceph-mon[107385]: pgmap v2712: 99 pgs: 1 active+recovery_wait+undersized+remapped, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 891 MiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 777 op/s; 12515/42831 objects degraded (29.219%); 6173/42831 objects misplaced (14.412%); 17 MiB/s, 356 objects/s recovering 2024-09-21T10:53:29.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:29 smithi070 ceph-mon[112139]: pgmap v2712: 99 pgs: 1 active+recovery_wait+undersized+remapped, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 891 MiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 777 op/s; 12515/42831 objects degraded (29.219%); 6173/42831 objects misplaced (14.412%); 17 MiB/s, 356 objects/s recovering 2024-09-21T10:53:29.928 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:29.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:29.964 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:29.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:29.965 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:29 smithi160 ceph-mon[101879]: pgmap v2712: 99 pgs: 1 active+recovery_wait+undersized+remapped, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 891 MiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 777 op/s; 12515/42831 objects degraded (29.219%); 6173/42831 objects misplaced (14.412%); 17 MiB/s, 356 objects/s recovering 2024-09-21T10:53:30.030 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:53:30.066 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:30.094 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:53:30.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:30.100 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:30.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:30.201 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:30.245 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:30.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:30.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:30.347 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:30.382 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:30.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:30.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:30.484 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:30.512 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:30.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:30.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:30.613 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:30.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:30.651 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:30.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:30.753 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:30.796 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:30.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:30.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:30.897 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:30.931 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:30.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:30.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:31.033 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:31.065 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:31.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:31.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:31.167 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:31.204 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:31.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:31.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:31.306 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:31.344 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:31.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:31.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:31.446 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:31.476 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:31.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:31.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:31.548 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:53:31.577 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:31.611 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:31.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:31.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:31.620 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:53:31.622 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T10:53:31.713 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:31.753 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:31.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:31.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:31.858 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:31.889 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:31.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:31.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:31.991 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:32.030 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:32.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:32.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:32.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:31 smithi070 ceph-mon[107385]: pgmap v2713: 99 pgs: 1 active+recovery_wait+undersized+remapped, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 891 MiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 651 op/s; 12515/42831 objects degraded (29.219%); 6173/42831 objects misplaced (14.412%); 15 MiB/s, 288 objects/s recovering 2024-09-21T10:53:32.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:31 smithi070 ceph-mon[107385]: from='client.35916 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:53:32.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:31 smithi070 ceph-mon[112139]: pgmap v2713: 99 pgs: 1 active+recovery_wait+undersized+remapped, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 891 MiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 651 op/s; 12515/42831 objects degraded (29.219%); 6173/42831 objects misplaced (14.412%); 15 MiB/s, 288 objects/s recovering 2024-09-21T10:53:32.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:31 smithi070 ceph-mon[112139]: from='client.35916 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:53:32.132 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:32.153 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:32.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:32.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:32.254 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:32.290 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:32.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:32.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:32.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:31 smithi160 ceph-mon[101879]: pgmap v2713: 99 pgs: 1 active+recovery_wait+undersized+remapped, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 891 MiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 651 op/s; 12515/42831 objects degraded (29.219%); 6173/42831 objects misplaced (14.412%); 15 MiB/s, 288 objects/s recovering 2024-09-21T10:53:32.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:31 smithi160 ceph-mon[101879]: from='client.35916 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:53:32.391 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:32.424 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:32.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:32.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:32.526 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:32.554 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:32.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:32.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:32.656 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:32.694 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:32.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:32.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:32.795 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:32.828 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:32.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:32.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:32.930 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:32.953 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:32.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:32.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:33.054 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:33.084 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:33.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:33.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:33.185 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:33.216 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:33.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:33.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:33.318 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:33.352 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:33.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:33.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:33.453 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:33.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:33.490 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:33.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:33.590 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:33.627 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:33.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:53:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:53:33] "GET /metrics HTTP/1.1" 200 36627 "" "Prometheus/2.51.0" 2024-09-21T10:53:33.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:33.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:33.729 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:33.763 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:33.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:33.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:33.865 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:33.895 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:33.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:33.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:33.997 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:34.023 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:34.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:34.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:34.125 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:34.165 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:34.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:34.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:34.267 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:34.302 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:34.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:34.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:34.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:34 smithi160 ceph-mon[101879]: pgmap v2714: 99 pgs: 1 active+recovery_wait+undersized+remapped, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 633 MiB data, 7.3 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 995 op/s; 11085/30459 objects degraded (36.393%); 5581/30459 objects misplaced (18.323%); 17 MiB/s, 352 objects/s recovering 2024-09-21T10:53:34.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:34 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 11085/30459 objects degraded (36.393%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:34.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:34 smithi070 ceph-mon[107385]: pgmap v2714: 99 pgs: 1 active+recovery_wait+undersized+remapped, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 633 MiB data, 7.3 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 995 op/s; 11085/30459 objects degraded (36.393%); 5581/30459 objects misplaced (18.323%); 17 MiB/s, 352 objects/s recovering 2024-09-21T10:53:34.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:34 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 11085/30459 objects degraded (36.393%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:34.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:34 smithi070 ceph-mon[112139]: pgmap v2714: 99 pgs: 1 active+recovery_wait+undersized+remapped, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 633 MiB data, 7.3 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 995 op/s; 11085/30459 objects degraded (36.393%); 5581/30459 objects misplaced (18.323%); 17 MiB/s, 352 objects/s recovering 2024-09-21T10:53:34.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:34 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 11085/30459 objects degraded (36.393%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:34.404 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:34.433 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:34.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:34.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:34.535 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:34.571 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:34.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:34.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:34.604 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":2713,"stamp":"2024-09-21T10:53:30.938913+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":934322956,"num_objects":14277,"num_object_clones":0,"num_object_copies":42831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12515,"num_objects_misplaced":6173,"num_objects_unfound":0,"num_objects_dirty":14277,"num_whiteouts":0,"num_read":277,"num_read_kb":599,"num_write":196648,"num_write_kb":6743423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45932,"num_bytes_recovered":2888495794,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75984,"ondisk_log_size":75984,"up":297,"acting":292,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":328,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":8254396,"kb_used_data":4900052,"kb_used_omap":141,"kb_used_meta":3354098,"kb_avail":460471364,"statfs":{"total":479975178240,"available":471522676736,"internally_reserved":0,"allocated":5017653248,"data_stored":4992636405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":144912,"internal_metadata":3434596848},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-512819200,"num_objects":-7825,"num_object_clones":0,"num_object_copies":-23475,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3455,"num_objects_misplaced":-966,"num_objects_unfound":0,"num_objects_dirty":-7825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7825,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3467,"num_bytes_recovered":182714368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005978"},"pg_stats":[{"pgid":"55.f","version":"1474'12267","reported_seq":19548,"reported_epoch":1474,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T10:53:26.800306+0000","last_change":"2024-09-21T10:52:25.539794+0000","last_active":"2024-09-21T10:53:26.800306+0000","last_peered":"2024-09-21T10:53:26.800306+0000","last_clean":"2024-09-21T10:49:43.822243+0000","last_became_active":"2024-09-21T10:51:28.555652+0000","last_became_peered":"2024-09-21T10:51:28.555652+0000","last_unstale":"2024-09-21T10:53:26.800306+0000","last_undegraded":"2024-09-21T10:52:25.537884+0000","last_fullsized":"2024-09-21T10:53:26.800306+0000","mapping_epoch":1448,"log_start":"1446'8140","ondisk_log_start":"1446'8140","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4127,"log_dups_size":0,"ondisk_log_size":4127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:30:34.790715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56688640,"num_objects":865,"num_object_clones":0,"num_object_copies":2595,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12267,"num_write_kb":420224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3592,"num_bytes_recovered":235405312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,0],"avail_no_missing":["6","5","7"],"object_location_counts":[{"shards":"0,6,7","objects":865}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.e","version":"1474'12125","reported_seq":12272,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:26.553569+0000","last_change":"2024-09-21T10:52:16.415783+0000","last_active":"2024-09-21T10:53:26.553569+0000","last_peered":"2024-09-21T10:53:26.553569+0000","last_clean":"2024-09-21T10:53:26.553569+0000","last_became_active":"2024-09-21T10:52:16.397921+0000","last_became_peered":"2024-09-21T10:52:16.397921+0000","last_unstale":"2024-09-21T10:53:26.553569+0000","last_undegraded":"2024-09-21T10:53:26.553569+0000","last_fullsized":"2024-09-21T10:53:26.553569+0000","mapping_epoch":1463,"log_start":"1445'7356","ondisk_log_start":"1445'7356","created":1411,"last_epoch_clean":1464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4769,"log_dups_size":0,"ondisk_log_size":4769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:13.753306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58130432,"num_objects":887,"num_object_clones":0,"num_object_copies":2661,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12125,"num_write_kb":416384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.d","version":"1474'12419","reported_seq":20619,"reported_epoch":1474,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:53:26.321487+0000","last_change":"2024-09-21T10:52:26.803434+0000","last_active":"2024-09-21T10:53:26.321487+0000","last_peered":"2024-09-21T10:53:26.321487+0000","last_clean":"2024-09-21T10:50:20.259057+0000","last_became_active":"2024-09-21T10:52:26.653674+0000","last_became_peered":"2024-09-21T10:52:26.653674+0000","last_unstale":"2024-09-21T10:53:26.321487+0000","last_undegraded":"2024-09-21T10:52:26.638016+0000","last_fullsized":"2024-09-21T10:52:26.462648+0000","mapping_epoch":1466,"log_start":"1445'7326","ondisk_log_start":"1445'7326","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":5093,"log_dups_size":0,"ondisk_log_size":5093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:31:18.206348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62783512,"num_objects":959,"num_object_clones":0,"num_object_copies":2877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5923,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":959,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12419,"num_write_kb":428033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4053,"num_bytes_recovered":265289728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[4,3],"avail_no_missing":["4","0","6"],"object_location_counts":[{"shards":"0,3,4","objects":268},{"shards":"0,4,7","objects":690},{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":4,"purged_snaps":[]},{"pgid":"55.c","version":"1474'12211","reported_seq":20476,"reported_epoch":1474,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T10:53:26.555247+0000","last_change":"2024-09-21T10:52:25.708024+0000","last_active":"2024-09-21T10:53:26.555247+0000","last_peered":"2024-09-21T10:53:26.555247+0000","last_clean":"2024-09-21T10:50:03.007234+0000","last_became_active":"2024-09-21T10:52:25.621852+0000","last_became_peered":"2024-09-21T10:52:25.621852+0000","last_unstale":"2024-09-21T10:53:26.555247+0000","last_undegraded":"2024-09-21T10:52:25.613400+0000","last_fullsized":"2024-09-21T10:53:26.555247+0000","mapping_epoch":1465,"log_start":"1445'7486","ondisk_log_start":"1445'7486","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4725,"log_dups_size":0,"ondisk_log_size":4725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60358656,"num_objects":921,"num_object_clones":0,"num_object_copies":2763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12211,"num_write_kb":420224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4057,"num_bytes_recovered":265814016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,0],"avail_no_missing":["3","1","4"],"object_location_counts":[{"shards":"0,3,4","objects":881},{"shards":"3,4","objects":40}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.b","version":"1474'12480","reported_seq":12554,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:26.791763+0000","last_change":"2024-09-21T10:48:52.934001+0000","last_active":"2024-09-21T10:53:26.791763+0000","last_peered":"2024-09-21T10:53:26.791763+0000","last_clean":"2024-09-21T10:53:26.791763+0000","last_became_active":"2024-09-21T10:48:52.933674+0000","last_became_peered":"2024-09-21T10:48:52.933674+0000","last_unstale":"2024-09-21T10:53:26.791763+0000","last_undegraded":"2024-09-21T10:53:26.791763+0000","last_fullsized":"2024-09-21T10:53:26.791763+0000","mapping_epoch":1411,"log_start":"1450'8406","ondisk_log_start":"1450'8406","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4074,"log_dups_size":0,"ondisk_log_size":4074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:13.178680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61603840,"num_objects":940,"num_object_clones":0,"num_object_copies":2820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12480,"num_write_kb":429440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.a","version":"1474'12238","reported_seq":17330,"reported_epoch":1474,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T10:53:26.541721+0000","last_change":"2024-09-21T10:52:26.731569+0000","last_active":"2024-09-21T10:53:26.541721+0000","last_peered":"2024-09-21T10:53:26.541721+0000","last_clean":"2024-09-21T10:50:03.021536+0000","last_became_active":"2024-09-21T10:52:26.699954+0000","last_became_peered":"2024-09-21T10:52:26.699954+0000","last_unstale":"2024-09-21T10:53:26.541721+0000","last_undegraded":"2024-09-21T10:52:26.664328+0000","last_fullsized":"2024-09-21T10:53:26.541721+0000","mapping_epoch":1466,"log_start":"1445'7451","ondisk_log_start":"1445'7451","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4787,"log_dups_size":0,"ondisk_log_size":4787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":896,"num_object_clones":0,"num_object_copies":2688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12238,"num_write_kb":420288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2471,"num_bytes_recovered":161939456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,4,6],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":32},{"shards":"3,4,6","objects":864}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.9","version":"1474'12230","reported_seq":21790,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:26.545415+0000","last_change":"2024-09-21T10:52:44.354764+0000","last_active":"2024-09-21T10:53:26.545415+0000","last_peered":"2024-09-21T10:53:26.545415+0000","last_clean":"2024-09-21T10:53:26.545415+0000","last_became_active":"2024-09-21T10:52:44.325813+0000","last_became_peered":"2024-09-21T10:52:44.325813+0000","last_unstale":"2024-09-21T10:53:26.545415+0000","last_undegraded":"2024-09-21T10:53:26.545415+0000","last_fullsized":"2024-09-21T10:53:26.545415+0000","mapping_epoch":1471,"log_start":"1445'7469","ondisk_log_start":"1445'7469","created":1411,"last_epoch_clean":1472,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4761,"log_dups_size":0,"ondisk_log_size":4761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:58.318369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61865984,"num_objects":944,"num_object_clones":0,"num_object_copies":2832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12230,"num_write_kb":421568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4701,"num_bytes_recovered":297533440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.8","version":"1474'12083","reported_seq":21482,"reported_epoch":1474,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-21T10:53:26.746368+0000","last_change":"2024-09-21T10:53:07.722709+0000","last_active":"2024-09-21T10:53:26.746368+0000","last_peered":"2024-09-21T10:53:26.746368+0000","last_clean":"2024-09-21T10:49:43.813262+0000","last_became_active":"2024-09-21T10:51:28.546990+0000","last_became_peered":"2024-09-21T10:51:28.546990+0000","last_unstale":"2024-09-21T10:53:26.746368+0000","last_undegraded":"2024-09-21T10:53:26.746368+0000","last_fullsized":"2024-09-21T10:53:26.746368+0000","mapping_epoch":1448,"log_start":"1446'7934","ondisk_log_start":"1446'7934","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4149,"log_dups_size":0,"ondisk_log_size":4149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:51:45.867981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62324736,"num_objects":951,"num_object_clones":0,"num_object_copies":2853,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":951,"num_objects_unfound":0,"num_objects_dirty":951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12083,"num_write_kb":417088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4638,"num_bytes_recovered":303955968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,0],"avail_no_missing":["6","0","3","5"],"object_location_counts":[{"shards":"0,3,6","objects":951}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.7","version":"1474'12378","reported_seq":12452,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:26.981961+0000","last_change":"2024-09-21T10:48:52.933230+0000","last_active":"2024-09-21T10:53:26.981961+0000","last_peered":"2024-09-21T10:53:26.981961+0000","last_clean":"2024-09-21T10:53:26.981961+0000","last_became_active":"2024-09-21T10:48:52.932944+0000","last_became_peered":"2024-09-21T10:48:52.932944+0000","last_unstale":"2024-09-21T10:53:26.981961+0000","last_undegraded":"2024-09-21T10:53:26.981961+0000","last_fullsized":"2024-09-21T10:53:26.981961+0000","mapping_epoch":1411,"log_start":"1445'7328","ondisk_log_start":"1445'7328","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":5050,"log_dups_size":0,"ondisk_log_size":5050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:16:43.711173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59637760,"num_objects":910,"num_object_clones":0,"num_object_copies":2730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12378,"num_write_kb":425216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"55.6","version":"1474'12437","reported_seq":27790,"reported_epoch":1474,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-21T10:53:26.995973+0000","last_change":"2024-09-21T10:52:25.578778+0000","last_active":"2024-09-21T10:53:26.995973+0000","last_peered":"2024-09-21T10:53:26.995973+0000","last_clean":"2024-09-21T10:49:43.854914+0000","last_became_active":"2024-09-21T10:52:03.460583+0000","last_became_peered":"2024-09-21T10:52:03.460583+0000","last_unstale":"2024-09-21T10:53:26.995973+0000","last_undegraded":"2024-09-21T10:53:26.995973+0000","last_fullsized":"2024-09-21T10:52:03.308381+0000","mapping_epoch":1455,"log_start":"1445'7426","ondisk_log_start":"1445'7426","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":5011,"log_dups_size":0,"ondisk_log_size":5011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:40.566532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58130432,"num_objects":887,"num_object_clones":0,"num_object_copies":2661,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12437,"num_write_kb":426368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6504,"num_bytes_recovered":334823424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,1],"avail_no_missing":["7","1","5"],"object_location_counts":[{"shards":"1,7","objects":887}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"55.5","version":"1474'12268","reported_seq":12391,"reported_epoch":1474,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-21T10:53:26.794336+0000","last_change":"2024-09-21T10:52:25.760897+0000","last_active":"2024-09-21T10:53:26.794336+0000","last_peered":"2024-09-21T10:53:26.794336+0000","last_clean":"2024-09-21T10:50:02.995767+0000","last_became_active":"2024-09-21T10:52:25.564312+0000","last_became_peered":"2024-09-21T10:52:25.564312+0000","last_unstale":"2024-09-21T10:53:26.794336+0000","last_undegraded":"2024-09-21T10:53:26.794336+0000","last_fullsized":"2024-09-21T10:53:26.794336+0000","mapping_epoch":1465,"log_start":"1446'8114","ondisk_log_start":"1446'8114","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4154,"log_dups_size":0,"ondisk_log_size":4154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:26:53.597959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57016320,"num_objects":870,"num_object_clones":0,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":870,"num_objects_unfound":0,"num_objects_dirty":870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12268,"num_write_kb":420416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,0],"avail_no_missing":["6","0","1","3","4"],"object_location_counts":[{"shards":"0,1,6","objects":870}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.4","version":"1474'12411","reported_seq":12483,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:26.988078+0000","last_change":"2024-09-21T10:48:52.934634+0000","last_active":"2024-09-21T10:53:26.988078+0000","last_peered":"2024-09-21T10:53:26.988078+0000","last_clean":"2024-09-21T10:53:26.988078+0000","last_became_active":"2024-09-21T10:48:52.934368+0000","last_became_peered":"2024-09-21T10:48:52.934368+0000","last_unstale":"2024-09-21T10:53:26.988078+0000","last_undegraded":"2024-09-21T10:53:26.988078+0000","last_fullsized":"2024-09-21T10:53:26.988078+0000","mapping_epoch":1411,"log_start":"1445'7345","ondisk_log_start":"1445'7345","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":5066,"log_dups_size":0,"ondisk_log_size":5066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:27:09.216667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59572224,"num_objects":909,"num_object_clones":0,"num_object_copies":2727,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12411,"num_write_kb":426240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"55.3","version":"1474'12004","reported_seq":27837,"reported_epoch":1474,"state":"active+recovery_wait+undersized+remapped","last_fresh":"2024-09-21T10:53:28.683484+0000","last_change":"2024-09-21T10:53:27.173985+0000","last_active":"2024-09-21T10:53:28.683484+0000","last_peered":"2024-09-21T10:53:28.683484+0000","last_clean":"2024-09-21T10:49:43.839828+0000","last_became_active":"2024-09-21T10:52:26.596220+0000","last_became_peered":"2024-09-21T10:52:26.596220+0000","last_unstale":"2024-09-21T10:53:28.683484+0000","last_undegraded":"2024-09-21T10:53:28.683484+0000","last_fullsized":"2024-09-21T10:52:26.462998+0000","mapping_epoch":1466,"log_start":"1446'7643","ondisk_log_start":"1446'7643","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4361,"log_dups_size":0,"ondisk_log_size":4361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48365568,"num_objects":738,"num_object_clones":0,"num_object_copies":2214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":738,"num_objects_unfound":0,"num_objects_dirty":738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12004,"num_write_kb":407744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5544,"num_bytes_recovered":363200512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,0],"avail_no_missing":["1","0","3","5","6"],"object_location_counts":[{"shards":"0,1","objects":738}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"55.2","version":"1474'12167","reported_seq":26642,"reported_epoch":1474,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T10:53:28.683919+0000","last_change":"2024-09-21T10:52:26.812464+0000","last_active":"2024-09-21T10:53:28.683919+0000","last_peered":"2024-09-21T10:53:28.683919+0000","last_clean":"2024-09-21T10:50:02.974304+0000","last_became_active":"2024-09-21T10:52:26.694338+0000","last_became_peered":"2024-09-21T10:52:26.694338+0000","last_unstale":"2024-09-21T10:53:28.683919+0000","last_undegraded":"2024-09-21T10:52:26.678206+0000","last_fullsized":"2024-09-21T10:53:28.683919+0000","mapping_epoch":1466,"log_start":"1446'7806","ondisk_log_start":"1446'7806","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4361,"log_dups_size":0,"ondisk_log_size":4361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48168960,"num_objects":735,"num_object_clones":0,"num_object_copies":2205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":735,"num_objects_misplaced":3614,"num_objects_unfound":0,"num_objects_dirty":735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12167,"num_write_kb":412864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5719,"num_bytes_recovered":374669312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,4],"avail_no_missing":["1","3","4"],"object_location_counts":[{"shards":"0,1,4","objects":735}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"55.1","version":"1474'12350","reported_seq":13640,"reported_epoch":1474,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:53:26.978624+0000","last_change":"2024-09-21T10:52:15.413755+0000","last_active":"2024-09-21T10:53:26.978624+0000","last_peered":"2024-09-21T10:53:26.978624+0000","last_clean":"2024-09-21T10:50:20.316846+0000","last_became_active":"2024-09-21T10:52:03.442393+0000","last_became_peered":"2024-09-21T10:52:03.442393+0000","last_unstale":"2024-09-21T10:53:26.978624+0000","last_undegraded":"2024-09-21T10:52:15.413422+0000","last_fullsized":"2024-09-21T10:52:03.308195+0000","mapping_epoch":1455,"log_start":"1445'7312","ondisk_log_start":"1445'7312","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":5038,"log_dups_size":0,"ondisk_log_size":5038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:45:04.793364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62128128,"num_objects":948,"num_object_clones":0,"num_object_copies":2844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12350,"num_write_kb":425536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":596,"num_bytes_recovered":20643840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":948}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"55.0","version":"1474'12359","reported_seq":20491,"reported_epoch":1474,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T10:53:26.539600+0000","last_change":"2024-09-21T10:52:26.707855+0000","last_active":"2024-09-21T10:53:26.539600+0000","last_peered":"2024-09-21T10:53:26.539600+0000","last_clean":"2024-09-21T10:50:03.041207+0000","last_became_active":"2024-09-21T10:52:26.645887+0000","last_became_peered":"2024-09-21T10:52:26.645887+0000","last_unstale":"2024-09-21T10:53:26.539600+0000","last_undegraded":"2024-09-21T10:52:26.635700+0000","last_fullsized":"2024-09-21T10:53:26.539600+0000","mapping_epoch":1466,"log_start":"1445'7599","ondisk_log_start":"1445'7599","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4760,"log_dups_size":0,"ondisk_log_size":4760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58392576,"num_objects":891,"num_object_clones":0,"num_object_copies":2673,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":891,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12359,"num_write_kb":424000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3991,"num_bytes_recovered":261554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,1],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"1,3,6","objects":859},{"shards":"3,6","objects":32}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":1964,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.375991+0000","last_change":"2024-09-21T10:52:03.323899+0000","last_active":"2024-09-21T10:53:02.375991+0000","last_peered":"2024-09-21T10:53:02.375991+0000","last_clean":"2024-09-21T10:53:02.375991+0000","last_became_active":"2024-09-21T10:52:03.314626+0000","last_became_peered":"2024-09-21T10:52:03.314626+0000","last_unstale":"2024-09-21T10:53:02.375991+0000","last_undegraded":"2024-09-21T10:53:02.375991+0000","last_fullsized":"2024-09-21T10:53:02.375991+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:13.939806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1464,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.376465+0000","last_change":"2024-09-21T10:52:15.396755+0000","last_active":"2024-09-21T10:53:02.376465+0000","last_peered":"2024-09-21T10:53:02.376465+0000","last_clean":"2024-09-21T10:53:02.376465+0000","last_became_active":"2024-09-21T10:52:15.396417+0000","last_became_peered":"2024-09-21T10:52:15.396417+0000","last_unstale":"2024-09-21T10:53:02.376465+0000","last_undegraded":"2024-09-21T10:53:02.376465+0000","last_fullsized":"2024-09-21T10:53:02.376465+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:25.005574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":2258,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.628860+0000","last_change":"2024-09-21T10:52:15.400227+0000","last_active":"2024-09-21T10:53:15.628860+0000","last_peered":"2024-09-21T10:53:15.628860+0000","last_clean":"2024-09-21T10:53:15.628860+0000","last_became_active":"2024-09-21T10:52:15.399888+0000","last_became_peered":"2024-09-21T10:52:15.399888+0000","last_unstale":"2024-09-21T10:53:15.628860+0000","last_undegraded":"2024-09-21T10:53:15.628860+0000","last_fullsized":"2024-09-21T10:53:15.628860+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:42:52.565184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2274,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.626706+0000","last_change":"2024-09-21T10:51:27.458216+0000","last_active":"2024-09-21T10:53:15.626706+0000","last_peered":"2024-09-21T10:53:15.626706+0000","last_clean":"2024-09-21T10:53:15.626706+0000","last_became_active":"2024-09-21T10:51:27.457948+0000","last_became_peered":"2024-09-21T10:51:27.457948+0000","last_unstale":"2024-09-21T10:53:15.626706+0000","last_undegraded":"2024-09-21T10:53:15.626706+0000","last_fullsized":"2024-09-21T10:53:15.626706+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:25:29.584079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":102,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.379707+0000","last_change":"2024-09-21T10:49:44.878242+0000","last_active":"2024-09-21T10:53:02.379707+0000","last_peered":"2024-09-21T10:53:02.379707+0000","last_clean":"2024-09-21T10:53:02.379707+0000","last_became_active":"2024-09-21T10:49:44.866578+0000","last_became_peered":"2024-09-21T10:49:44.866578+0000","last_unstale":"2024-09-21T10:53:02.379707+0000","last_undegraded":"2024-09-21T10:53:02.379707+0000","last_fullsized":"2024-09-21T10:53:02.379707+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:17:24.145551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1469,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.377717+0000","last_change":"2024-09-21T10:51:27.460539+0000","last_active":"2024-09-21T10:53:02.377717+0000","last_peered":"2024-09-21T10:53:02.377717+0000","last_clean":"2024-09-21T10:53:02.377717+0000","last_became_active":"2024-09-21T10:51:27.460211+0000","last_became_peered":"2024-09-21T10:51:27.460211+0000","last_unstale":"2024-09-21T10:53:02.377717+0000","last_undegraded":"2024-09-21T10:53:02.377717+0000","last_fullsized":"2024-09-21T10:53:02.377717+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:14:15.700998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":141,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.377020+0000","last_change":"2024-09-21T10:52:15.405886+0000","last_active":"2024-09-21T10:53:02.377020+0000","last_peered":"2024-09-21T10:53:02.377020+0000","last_clean":"2024-09-21T10:53:02.377020+0000","last_became_active":"2024-09-21T10:52:15.397317+0000","last_became_peered":"2024-09-21T10:52:15.397317+0000","last_unstale":"2024-09-21T10:53:02.377020+0000","last_undegraded":"2024-09-21T10:53:02.377020+0000","last_fullsized":"2024-09-21T10:53:02.377020+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:43:48.813861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1460,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.377014+0000","last_change":"2024-09-21T10:09:06.560175+0000","last_active":"2024-09-21T10:53:02.377014+0000","last_peered":"2024-09-21T10:53:02.377014+0000","last_clean":"2024-09-21T10:53:02.377014+0000","last_became_active":"2024-09-21T10:09:06.558584+0000","last_became_peered":"2024-09-21T10:09:06.558584+0000","last_unstale":"2024-09-21T10:53:02.377014+0000","last_undegraded":"2024-09-21T10:53:02.377014+0000","last_fullsized":"2024-09-21T10:53:02.377014+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:02.918339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":50,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.376947+0000","last_change":"2024-09-21T10:51:27.437665+0000","last_active":"2024-09-21T10:53:02.376947+0000","last_peered":"2024-09-21T10:53:02.376947+0000","last_clean":"2024-09-21T10:53:02.376947+0000","last_became_active":"2024-09-21T10:51:27.436481+0000","last_became_peered":"2024-09-21T10:51:27.436481+0000","last_unstale":"2024-09-21T10:53:02.376947+0000","last_undegraded":"2024-09-21T10:53:02.376947+0000","last_fullsized":"2024-09-21T10:53:02.376947+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:07:50.267297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1029,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.620603+0000","last_change":"2024-09-21T10:51:27.459033+0000","last_active":"2024-09-21T10:53:15.620603+0000","last_peered":"2024-09-21T10:53:15.620603+0000","last_clean":"2024-09-21T10:53:15.620603+0000","last_became_active":"2024-09-21T10:51:27.458728+0000","last_became_peered":"2024-09-21T10:51:27.458728+0000","last_unstale":"2024-09-21T10:53:15.620603+0000","last_undegraded":"2024-09-21T10:53:15.620603+0000","last_fullsized":"2024-09-21T10:53:15.620603+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:20.133032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":1489,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.377737+0000","last_change":"2024-09-21T10:52:25.473550+0000","last_active":"2024-09-21T10:53:02.377737+0000","last_peered":"2024-09-21T10:53:02.377737+0000","last_clean":"2024-09-21T10:53:02.377737+0000","last_became_active":"2024-09-21T10:52:25.473105+0000","last_became_peered":"2024-09-21T10:52:25.473105+0000","last_unstale":"2024-09-21T10:53:02.377737+0000","last_undegraded":"2024-09-21T10:53:02.377737+0000","last_fullsized":"2024-09-21T10:53:02.377737+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:15:45.352518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2253,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.634313+0000","last_change":"2024-09-21T10:40:13.432531+0000","last_active":"2024-09-21T10:53:15.634313+0000","last_peered":"2024-09-21T10:53:15.634313+0000","last_clean":"2024-09-21T10:53:15.634313+0000","last_became_active":"2024-09-21T10:40:13.432210+0000","last_became_peered":"2024-09-21T10:40:13.432210+0000","last_unstale":"2024-09-21T10:53:15.634313+0000","last_undegraded":"2024-09-21T10:53:15.634313+0000","last_fullsized":"2024-09-21T10:53:15.634313+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:20:10.998607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2225,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.620552+0000","last_change":"2024-09-21T10:09:06.530394+0000","last_active":"2024-09-21T10:53:15.620552+0000","last_peered":"2024-09-21T10:53:15.620552+0000","last_clean":"2024-09-21T10:53:15.620552+0000","last_became_active":"2024-09-21T10:09:06.529307+0000","last_became_peered":"2024-09-21T10:09:06.529307+0000","last_unstale":"2024-09-21T10:53:15.620552+0000","last_undegraded":"2024-09-21T10:53:15.620552+0000","last_fullsized":"2024-09-21T10:53:15.620552+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:54.287397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.371979+0000","last_change":"2024-09-21T10:52:25.489896+0000","last_active":"2024-09-21T10:53:02.371979+0000","last_peered":"2024-09-21T10:53:02.371979+0000","last_clean":"2024-09-21T10:53:02.371979+0000","last_became_active":"2024-09-21T10:52:25.489154+0000","last_became_peered":"2024-09-21T10:52:25.489154+0000","last_unstale":"2024-09-21T10:53:02.371979+0000","last_undegraded":"2024-09-21T10:53:02.371979+0000","last_fullsized":"2024-09-21T10:53:02.371979+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:36:02.076931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2254,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.625455+0000","last_change":"2024-09-21T10:52:25.493309+0000","last_active":"2024-09-21T10:53:15.625455+0000","last_peered":"2024-09-21T10:53:15.625455+0000","last_clean":"2024-09-21T10:53:15.625455+0000","last_became_active":"2024-09-21T10:52:25.492568+0000","last_became_peered":"2024-09-21T10:52:25.492568+0000","last_unstale":"2024-09-21T10:53:15.625455+0000","last_undegraded":"2024-09-21T10:53:15.625455+0000","last_fullsized":"2024-09-21T10:53:15.625455+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:19:19.393370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1477,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.619677+0000","last_change":"2024-09-21T10:52:03.317474+0000","last_active":"2024-09-21T10:53:15.619677+0000","last_peered":"2024-09-21T10:53:15.619677+0000","last_clean":"2024-09-21T10:53:15.619677+0000","last_became_active":"2024-09-21T10:52:03.317001+0000","last_became_peered":"2024-09-21T10:52:03.317001+0000","last_unstale":"2024-09-21T10:53:15.619677+0000","last_undegraded":"2024-09-21T10:53:15.619677+0000","last_fullsized":"2024-09-21T10:53:15.619677+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:06:07.039763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2253,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.625692+0000","last_change":"2024-09-21T10:52:15.412246+0000","last_active":"2024-09-21T10:53:15.625692+0000","last_peered":"2024-09-21T10:53:15.625692+0000","last_clean":"2024-09-21T10:53:15.625692+0000","last_became_active":"2024-09-21T10:52:15.406204+0000","last_became_peered":"2024-09-21T10:52:15.406204+0000","last_unstale":"2024-09-21T10:53:15.625692+0000","last_undegraded":"2024-09-21T10:53:15.625692+0000","last_fullsized":"2024-09-21T10:53:15.625692+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:03.524324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2233,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.624859+0000","last_change":"2024-09-21T10:52:15.402822+0000","last_active":"2024-09-21T10:53:15.624859+0000","last_peered":"2024-09-21T10:53:15.624859+0000","last_clean":"2024-09-21T10:53:15.624859+0000","last_became_active":"2024-09-21T10:52:15.402548+0000","last_became_peered":"2024-09-21T10:52:15.402548+0000","last_unstale":"2024-09-21T10:53:15.624859+0000","last_undegraded":"2024-09-21T10:53:15.624859+0000","last_fullsized":"2024-09-21T10:53:15.624859+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:04:04.667598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2258,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.626815+0000","last_change":"2024-09-21T10:52:25.499172+0000","last_active":"2024-09-21T10:53:15.626815+0000","last_peered":"2024-09-21T10:53:15.626815+0000","last_clean":"2024-09-21T10:53:15.626815+0000","last_became_active":"2024-09-21T10:52:25.498835+0000","last_became_peered":"2024-09-21T10:52:25.498835+0000","last_unstale":"2024-09-21T10:53:15.626815+0000","last_undegraded":"2024-09-21T10:53:15.626815+0000","last_fullsized":"2024-09-21T10:53:15.626815+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:10:12.434470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":19,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.371941+0000","last_change":"2024-09-21T10:52:25.489774+0000","last_active":"2024-09-21T10:53:02.371941+0000","last_peered":"2024-09-21T10:53:02.371941+0000","last_clean":"2024-09-21T10:53:02.371941+0000","last_became_active":"2024-09-21T10:52:25.487788+0000","last_became_peered":"2024-09-21T10:52:25.487788+0000","last_unstale":"2024-09-21T10:53:02.371941+0000","last_undegraded":"2024-09-21T10:53:02.371941+0000","last_fullsized":"2024-09-21T10:53:02.371941+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:48.842419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":25,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.370754+0000","last_change":"2024-09-21T10:52:25.565465+0000","last_active":"2024-09-21T10:53:02.370754+0000","last_peered":"2024-09-21T10:53:02.370754+0000","last_clean":"2024-09-21T10:53:02.370754+0000","last_became_active":"2024-09-21T10:52:25.510131+0000","last_became_peered":"2024-09-21T10:52:25.510131+0000","last_unstale":"2024-09-21T10:53:02.370754+0000","last_undegraded":"2024-09-21T10:53:02.370754+0000","last_fullsized":"2024-09-21T10:53:02.370754+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:41:13.870503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":450,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2260,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.626347+0000","last_change":"2024-09-21T10:52:25.504787+0000","last_active":"2024-09-21T10:53:15.626347+0000","last_peered":"2024-09-21T10:53:15.626347+0000","last_clean":"2024-09-21T10:53:15.626347+0000","last_became_active":"2024-09-21T10:52:25.504198+0000","last_became_peered":"2024-09-21T10:52:25.504198+0000","last_unstale":"2024-09-21T10:53:15.626347+0000","last_undegraded":"2024-09-21T10:53:15.626347+0000","last_fullsized":"2024-09-21T10:53:15.626347+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:38:25.666224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":1493,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.377215+0000","last_change":"2024-09-21T10:44:14.415264+0000","last_active":"2024-09-21T10:53:02.377215+0000","last_peered":"2024-09-21T10:53:02.377215+0000","last_clean":"2024-09-21T10:53:02.377215+0000","last_became_active":"2024-09-21T10:44:14.407813+0000","last_became_peered":"2024-09-21T10:44:14.407813+0000","last_unstale":"2024-09-21T10:53:02.377215+0000","last_undegraded":"2024-09-21T10:53:02.377215+0000","last_fullsized":"2024-09-21T10:53:02.377215+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:55.933601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1447,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.613811+0000","last_change":"2024-09-21T10:07:56.001756+0000","last_active":"2024-09-21T10:53:15.613811+0000","last_peered":"2024-09-21T10:53:15.613811+0000","last_clean":"2024-09-21T10:53:15.613811+0000","last_became_active":"2024-09-21T10:07:56.000789+0000","last_became_peered":"2024-09-21T10:07:56.000789+0000","last_unstale":"2024-09-21T10:53:15.613811+0000","last_undegraded":"2024-09-21T10:53:15.613811+0000","last_fullsized":"2024-09-21T10:53:15.613811+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:56:04.302902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"346'149","reported_seq":910,"reported_epoch":1474,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:53:15.625317+0000","last_change":"2024-09-21T10:52:26.478603+0000","last_active":"2024-09-21T10:53:15.625317+0000","last_peered":"2024-09-21T10:53:15.625317+0000","last_clean":"2024-09-21T10:52:16.392213+0000","last_became_active":"2024-09-21T10:52:26.478167+0000","last_became_peered":"2024-09-21T10:52:26.478167+0000","last_unstale":"2024-09-21T10:53:15.625317+0000","last_undegraded":"2024-09-21T10:52:26.463682+0000","last_fullsized":"2024-09-21T10:52:26.463251+0000","mapping_epoch":1466,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1449,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1251389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,6],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":1},{"shards":"3,6","objects":-1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1462,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.612810+0000","last_change":"2024-09-21T10:52:25.473127+0000","last_active":"2024-09-21T10:53:15.612810+0000","last_peered":"2024-09-21T10:53:15.612810+0000","last_clean":"2024-09-21T10:53:15.612810+0000","last_became_active":"2024-09-21T10:52:25.472805+0000","last_became_peered":"2024-09-21T10:52:25.472805+0000","last_unstale":"2024-09-21T10:53:15.612810+0000","last_undegraded":"2024-09-21T10:53:15.612810+0000","last_fullsized":"2024-09-21T10:53:15.612810+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:13.038797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1507,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.376134+0000","last_change":"2024-09-21T10:52:25.493263+0000","last_active":"2024-09-21T10:53:02.376134+0000","last_peered":"2024-09-21T10:53:02.376134+0000","last_clean":"2024-09-21T10:53:02.376134+0000","last_became_active":"2024-09-21T10:52:25.492970+0000","last_became_peered":"2024-09-21T10:52:25.492970+0000","last_unstale":"2024-09-21T10:53:02.376134+0000","last_undegraded":"2024-09-21T10:53:02.376134+0000","last_fullsized":"2024-09-21T10:53:02.376134+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:43:40.305783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":104,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.620552+0000","last_change":"2024-09-21T10:52:25.513834+0000","last_active":"2024-09-21T10:53:15.620552+0000","last_peered":"2024-09-21T10:53:15.620552+0000","last_clean":"2024-09-21T10:53:15.620552+0000","last_became_active":"2024-09-21T10:52:25.475423+0000","last_became_peered":"2024-09-21T10:52:25.475423+0000","last_unstale":"2024-09-21T10:53:15.620552+0000","last_undegraded":"2024-09-21T10:53:15.620552+0000","last_fullsized":"2024-09-21T10:53:15.620552+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:10.216527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"346'149","reported_seq":893,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.377552+0000","last_change":"2024-09-21T10:52:24.459450+0000","last_active":"2024-09-21T10:53:02.377552+0000","last_peered":"2024-09-21T10:53:02.377552+0000","last_clean":"2024-09-21T10:53:02.377552+0000","last_became_active":"2024-09-21T10:52:15.405544+0000","last_became_peered":"2024-09-21T10:52:15.405544+0000","last_unstale":"2024-09-21T10:53:02.377552+0000","last_undegraded":"2024-09-21T10:53:02.377552+0000","last_fullsized":"2024-09-21T10:53:02.377552+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2225,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.625618+0000","last_change":"2024-09-21T10:52:25.494316+0000","last_active":"2024-09-21T10:53:15.625618+0000","last_peered":"2024-09-21T10:53:15.625618+0000","last_clean":"2024-09-21T10:53:15.625618+0000","last_became_active":"2024-09-21T10:52:25.494016+0000","last_became_peered":"2024-09-21T10:52:25.494016+0000","last_unstale":"2024-09-21T10:53:15.625618+0000","last_undegraded":"2024-09-21T10:53:15.625618+0000","last_fullsized":"2024-09-21T10:53:15.625618+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:07.859584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2227,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.621007+0000","last_change":"2024-09-21T10:07:56.006236+0000","last_active":"2024-09-21T10:53:15.621007+0000","last_peered":"2024-09-21T10:53:15.621007+0000","last_clean":"2024-09-21T10:53:15.621007+0000","last_became_active":"2024-09-21T10:07:56.004439+0000","last_became_peered":"2024-09-21T10:07:56.004439+0000","last_unstale":"2024-09-21T10:53:15.621007+0000","last_undegraded":"2024-09-21T10:53:15.621007+0000","last_fullsized":"2024-09-21T10:53:15.621007+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:39.422467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":2276,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.625009+0000","last_change":"2024-09-21T10:52:25.536401+0000","last_active":"2024-09-21T10:53:15.625009+0000","last_peered":"2024-09-21T10:53:15.625009+0000","last_clean":"2024-09-21T10:53:15.625009+0000","last_became_active":"2024-09-21T10:52:25.496480+0000","last_became_peered":"2024-09-21T10:52:25.496480+0000","last_unstale":"2024-09-21T10:53:15.625009+0000","last_undegraded":"2024-09-21T10:53:15.625009+0000","last_fullsized":"2024-09-21T10:53:15.625009+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:50:34.820373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"346'149","reported_seq":718,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.375487+0000","last_change":"2024-09-21T10:51:30.450301+0000","last_active":"2024-09-21T10:53:02.375487+0000","last_peered":"2024-09-21T10:53:02.375487+0000","last_clean":"2024-09-21T10:53:02.375487+0000","last_became_active":"2024-09-21T10:51:30.449925+0000","last_became_peered":"2024-09-21T10:51:30.449925+0000","last_unstale":"2024-09-21T10:53:02.375487+0000","last_undegraded":"2024-09-21T10:53:02.375487+0000","last_fullsized":"2024-09-21T10:53:02.375487+0000","mapping_epoch":1450,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1451,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:41.011085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:41.011085+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:38:33.451261+0000","scrub_duration":0.0061250310000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2189,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.370748+0000","last_change":"2024-09-21T10:51:27.431416+0000","last_active":"2024-09-21T10:53:02.370748+0000","last_peered":"2024-09-21T10:53:02.370748+0000","last_clean":"2024-09-21T10:53:02.370748+0000","last_became_active":"2024-09-21T10:51:27.431127+0000","last_became_peered":"2024-09-21T10:51:27.431127+0000","last_unstale":"2024-09-21T10:53:02.370748+0000","last_undegraded":"2024-09-21T10:53:02.370748+0000","last_fullsized":"2024-09-21T10:53:02.370748+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:27.721665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.370730+0000","last_change":"2024-09-21T10:52:25.468849+0000","last_active":"2024-09-21T10:53:02.370730+0000","last_peered":"2024-09-21T10:53:02.370730+0000","last_clean":"2024-09-21T10:53:02.370730+0000","last_became_active":"2024-09-21T10:52:25.468552+0000","last_became_peered":"2024-09-21T10:52:25.468552+0000","last_unstale":"2024-09-21T10:53:02.370730+0000","last_undegraded":"2024-09-21T10:53:02.370730+0000","last_fullsized":"2024-09-21T10:53:02.370730+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:36:41.692028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2276,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.620560+0000","last_change":"2024-09-21T10:51:27.436608+0000","last_active":"2024-09-21T10:53:15.620560+0000","last_peered":"2024-09-21T10:53:15.620560+0000","last_clean":"2024-09-21T10:53:15.620560+0000","last_became_active":"2024-09-21T10:51:27.434949+0000","last_became_peered":"2024-09-21T10:51:27.434949+0000","last_unstale":"2024-09-21T10:53:15.620560+0000","last_undegraded":"2024-09-21T10:53:15.620560+0000","last_fullsized":"2024-09-21T10:53:15.620560+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:49:23.439425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"346'149","reported_seq":761,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.375846+0000","last_change":"2024-09-21T10:52:28.590793+0000","last_active":"2024-09-21T10:53:02.375846+0000","last_peered":"2024-09-21T10:53:02.375846+0000","last_clean":"2024-09-21T10:53:02.375846+0000","last_became_active":"2024-09-21T10:52:28.588662+0000","last_became_peered":"2024-09-21T10:52:28.588662+0000","last_unstale":"2024-09-21T10:53:02.375846+0000","last_undegraded":"2024-09-21T10:53:02.375846+0000","last_fullsized":"2024-09-21T10:53:02.375846+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:44.018645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:44.018645+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:46.849427+0000","scrub_duration":0.0012789399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2231,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.626221+0000","last_change":"2024-09-21T10:51:27.440425+0000","last_active":"2024-09-21T10:53:15.626221+0000","last_peered":"2024-09-21T10:53:15.626221+0000","last_clean":"2024-09-21T10:53:15.626221+0000","last_became_active":"2024-09-21T10:51:27.439666+0000","last_became_peered":"2024-09-21T10:51:27.439666+0000","last_unstale":"2024-09-21T10:53:15.626221+0000","last_undegraded":"2024-09-21T10:53:15.626221+0000","last_fullsized":"2024-09-21T10:53:15.626221+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:56:47.410516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2299,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.620338+0000","last_change":"2024-09-21T10:52:03.314926+0000","last_active":"2024-09-21T10:53:15.620338+0000","last_peered":"2024-09-21T10:53:15.620338+0000","last_clean":"2024-09-21T10:53:15.620338+0000","last_became_active":"2024-09-21T10:52:03.314602+0000","last_became_peered":"2024-09-21T10:52:03.314602+0000","last_unstale":"2024-09-21T10:53:15.620338+0000","last_undegraded":"2024-09-21T10:53:15.620338+0000","last_fullsized":"2024-09-21T10:53:15.620338+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:13.774521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1455,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.634150+0000","last_change":"2024-09-21T10:52:25.502830+0000","last_active":"2024-09-21T10:53:15.634150+0000","last_peered":"2024-09-21T10:53:15.634150+0000","last_clean":"2024-09-21T10:53:15.634150+0000","last_became_active":"2024-09-21T10:52:25.502505+0000","last_became_peered":"2024-09-21T10:52:25.502505+0000","last_unstale":"2024-09-21T10:53:15.634150+0000","last_undegraded":"2024-09-21T10:53:15.634150+0000","last_fullsized":"2024-09-21T10:53:15.634150+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:01:14.965694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"346'149","reported_seq":816,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.376366+0000","last_change":"2024-09-21T10:52:24.458657+0000","last_active":"2024-09-21T10:53:02.376366+0000","last_peered":"2024-09-21T10:53:02.376366+0000","last_clean":"2024-09-21T10:53:02.376366+0000","last_became_active":"2024-09-21T10:51:30.436983+0000","last_became_peered":"2024-09-21T10:51:30.436983+0000","last_unstale":"2024-09-21T10:53:02.376366+0000","last_undegraded":"2024-09-21T10:53:02.376366+0000","last_fullsized":"2024-09-21T10:53:02.376366+0000","mapping_epoch":1450,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1451,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2293,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.620697+0000","last_change":"2024-09-21T10:52:15.397180+0000","last_active":"2024-09-21T10:53:15.620697+0000","last_peered":"2024-09-21T10:53:15.620697+0000","last_clean":"2024-09-21T10:53:15.620697+0000","last_became_active":"2024-09-21T10:52:15.396879+0000","last_became_peered":"2024-09-21T10:52:15.396879+0000","last_unstale":"2024-09-21T10:53:15.620697+0000","last_undegraded":"2024-09-21T10:53:15.620697+0000","last_fullsized":"2024-09-21T10:53:15.620697+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:03:07.162178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1460,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.614094+0000","last_change":"2024-09-21T10:51:27.436590+0000","last_active":"2024-09-21T10:53:15.614094+0000","last_peered":"2024-09-21T10:53:15.614094+0000","last_clean":"2024-09-21T10:53:15.614094+0000","last_became_active":"2024-09-21T10:51:27.435332+0000","last_became_peered":"2024-09-21T10:51:27.435332+0000","last_unstale":"2024-09-21T10:53:15.614094+0000","last_undegraded":"2024-09-21T10:53:15.614094+0000","last_fullsized":"2024-09-21T10:53:15.614094+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:05:42.885506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":189,"reported_epoch":1474,"state":"active+recovery_wait","last_fresh":"2024-09-21T10:53:15.625934+0000","last_change":"2024-09-21T10:52:26.680832+0000","last_active":"2024-09-21T10:53:15.625934+0000","last_peered":"2024-09-21T10:53:15.625934+0000","last_clean":"2024-09-21T10:51:45.437560+0000","last_became_active":"2024-09-21T10:52:03.315398+0000","last_became_peered":"2024-09-21T10:52:03.315398+0000","last_unstale":"2024-09-21T10:53:15.625934+0000","last_undegraded":"2024-09-21T10:53:15.625934+0000","last_fullsized":"2024-09-21T10:53:15.625934+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":["1","3","4","6"],"object_location_counts":[{"shards":"1,3,6","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"346'149","reported_seq":787,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.370546+0000","last_change":"2024-09-21T10:52:28.495319+0000","last_active":"2024-09-21T10:53:02.370546+0000","last_peered":"2024-09-21T10:53:02.370546+0000","last_clean":"2024-09-21T10:53:02.370546+0000","last_became_active":"2024-09-21T10:52:28.494923+0000","last_became_peered":"2024-09-21T10:52:28.494923+0000","last_unstale":"2024-09-21T10:53:02.370546+0000","last_undegraded":"2024-09-21T10:53:02.370546+0000","last_fullsized":"2024-09-21T10:53:02.370546+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:52:17.894787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:52:17.894787+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":177,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.377088+0000","last_change":"2024-09-21T10:52:25.475200+0000","last_active":"2024-09-21T10:53:02.377088+0000","last_peered":"2024-09-21T10:53:02.377088+0000","last_clean":"2024-09-21T10:53:02.377088+0000","last_became_active":"2024-09-21T10:52:25.474872+0000","last_became_peered":"2024-09-21T10:52:25.474872+0000","last_unstale":"2024-09-21T10:53:02.377088+0000","last_undegraded":"2024-09-21T10:53:02.377088+0000","last_fullsized":"2024-09-21T10:53:02.377088+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:36:34.207844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2250,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.620249+0000","last_change":"2024-09-21T10:52:03.325889+0000","last_active":"2024-09-21T10:53:15.620249+0000","last_peered":"2024-09-21T10:53:15.620249+0000","last_clean":"2024-09-21T10:53:15.620249+0000","last_became_active":"2024-09-21T10:52:03.325500+0000","last_became_peered":"2024-09-21T10:52:03.325500+0000","last_unstale":"2024-09-21T10:53:15.620249+0000","last_undegraded":"2024-09-21T10:53:15.620249+0000","last_fullsized":"2024-09-21T10:53:15.620249+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:20.708252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1036,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.626642+0000","last_change":"2024-09-21T10:51:27.457894+0000","last_active":"2024-09-21T10:53:15.626642+0000","last_peered":"2024-09-21T10:53:15.626642+0000","last_clean":"2024-09-21T10:53:15.626642+0000","last_became_active":"2024-09-21T10:51:27.457431+0000","last_became_peered":"2024-09-21T10:51:27.457431+0000","last_unstale":"2024-09-21T10:53:15.626642+0000","last_undegraded":"2024-09-21T10:53:15.626642+0000","last_fullsized":"2024-09-21T10:53:15.626642+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:03:33.550759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"346'149","reported_seq":906,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.622618+0000","last_change":"2024-09-21T10:52:15.396042+0000","last_active":"2024-09-21T10:53:15.622618+0000","last_peered":"2024-09-21T10:53:15.622618+0000","last_clean":"2024-09-21T10:53:15.622618+0000","last_became_active":"2024-09-21T10:52:15.395717+0000","last_became_peered":"2024-09-21T10:52:15.395717+0000","last_unstale":"2024-09-21T10:53:15.622618+0000","last_undegraded":"2024-09-21T10:53:15.622618+0000","last_fullsized":"2024-09-21T10:53:15.622618+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2275,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.620538+0000","last_change":"2024-09-21T10:52:25.476572+0000","last_active":"2024-09-21T10:53:15.620538+0000","last_peered":"2024-09-21T10:53:15.620538+0000","last_clean":"2024-09-21T10:53:15.620538+0000","last_became_active":"2024-09-21T10:52:25.475803+0000","last_became_peered":"2024-09-21T10:52:25.475803+0000","last_unstale":"2024-09-21T10:53:15.620538+0000","last_undegraded":"2024-09-21T10:53:15.620538+0000","last_fullsized":"2024-09-21T10:53:15.620538+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:06:44.191486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2205,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.378571+0000","last_change":"2024-09-21T10:52:03.326753+0000","last_active":"2024-09-21T10:53:02.378571+0000","last_peered":"2024-09-21T10:53:02.378571+0000","last_clean":"2024-09-21T10:53:02.378571+0000","last_became_active":"2024-09-21T10:52:03.326348+0000","last_became_peered":"2024-09-21T10:52:03.326348+0000","last_unstale":"2024-09-21T10:53:02.378571+0000","last_undegraded":"2024-09-21T10:53:02.378571+0000","last_fullsized":"2024-09-21T10:53:02.378571+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:57:38.100799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":128,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.370521+0000","last_change":"2024-09-21T10:51:27.426006+0000","last_active":"2024-09-21T10:53:02.370521+0000","last_peered":"2024-09-21T10:53:02.370521+0000","last_clean":"2024-09-21T10:53:02.370521+0000","last_became_active":"2024-09-21T10:51:27.425698+0000","last_became_peered":"2024-09-21T10:51:27.425698+0000","last_unstale":"2024-09-21T10:53:02.370521+0000","last_undegraded":"2024-09-21T10:53:02.370521+0000","last_fullsized":"2024-09-21T10:53:02.370521+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:35:13.210330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":22,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.374677+0000","last_change":"2024-09-21T10:52:15.401737+0000","last_active":"2024-09-21T10:53:02.374677+0000","last_peered":"2024-09-21T10:53:02.374677+0000","last_clean":"2024-09-21T10:53:02.374677+0000","last_became_active":"2024-09-21T10:52:15.400859+0000","last_became_peered":"2024-09-21T10:52:15.400859+0000","last_unstale":"2024-09-21T10:53:02.374677+0000","last_undegraded":"2024-09-21T10:53:02.374677+0000","last_fullsized":"2024-09-21T10:53:02.374677+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:15.723031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2230,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.625206+0000","last_change":"2024-09-21T10:49:44.865830+0000","last_active":"2024-09-21T10:53:15.625206+0000","last_peered":"2024-09-21T10:53:15.625206+0000","last_clean":"2024-09-21T10:53:15.625206+0000","last_became_active":"2024-09-21T10:49:44.865499+0000","last_became_peered":"2024-09-21T10:49:44.865499+0000","last_unstale":"2024-09-21T10:53:15.625206+0000","last_undegraded":"2024-09-21T10:53:15.625206+0000","last_fullsized":"2024-09-21T10:53:15.625206+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:19:22.013154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":1453,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.371221+0000","last_change":"2024-09-21T10:52:25.489550+0000","last_active":"2024-09-21T10:53:02.371221+0000","last_peered":"2024-09-21T10:53:02.371221+0000","last_clean":"2024-09-21T10:53:02.371221+0000","last_became_active":"2024-09-21T10:52:25.489171+0000","last_became_peered":"2024-09-21T10:52:25.489171+0000","last_unstale":"2024-09-21T10:53:02.371221+0000","last_undegraded":"2024-09-21T10:53:02.371221+0000","last_fullsized":"2024-09-21T10:53:02.371221+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:21.105058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":787,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.370355+0000","last_change":"2024-09-21T10:52:28.601906+0000","last_active":"2024-09-21T10:53:02.370355+0000","last_peered":"2024-09-21T10:53:02.370355+0000","last_clean":"2024-09-21T10:53:02.370355+0000","last_became_active":"2024-09-21T10:52:28.587447+0000","last_became_peered":"2024-09-21T10:52:28.587447+0000","last_unstale":"2024-09-21T10:53:02.370355+0000","last_undegraded":"2024-09-21T10:53:02.370355+0000","last_fullsized":"2024-09-21T10:53:02.370355+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:44.223578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:44.223578+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":74,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"346'149","reported_seq":735,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.374801+0000","last_change":"2024-09-21T10:51:30.453979+0000","last_active":"2024-09-21T10:53:02.374801+0000","last_peered":"2024-09-21T10:53:02.374801+0000","last_clean":"2024-09-21T10:53:02.374801+0000","last_became_active":"2024-09-21T10:51:30.450893+0000","last_became_peered":"2024-09-21T10:51:30.450893+0000","last_unstale":"2024-09-21T10:53:02.374801+0000","last_undegraded":"2024-09-21T10:53:02.374801+0000","last_fullsized":"2024-09-21T10:53:02.374801+0000","mapping_epoch":1450,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1451,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:42.008235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:42.008235+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:21:48.570809+0000","scrub_duration":0.011129834999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2227,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.621476+0000","last_change":"2024-09-21T10:09:06.538940+0000","last_active":"2024-09-21T10:53:15.621476+0000","last_peered":"2024-09-21T10:53:15.621476+0000","last_clean":"2024-09-21T10:53:15.621476+0000","last_became_active":"2024-09-21T10:09:06.538668+0000","last_became_peered":"2024-09-21T10:09:06.538668+0000","last_unstale":"2024-09-21T10:53:15.621476+0000","last_undegraded":"2024-09-21T10:53:15.621476+0000","last_fullsized":"2024-09-21T10:53:15.621476+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:42:44.299488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2225,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.625363+0000","last_change":"2024-09-21T10:52:25.493032+0000","last_active":"2024-09-21T10:53:15.625363+0000","last_peered":"2024-09-21T10:53:15.625363+0000","last_clean":"2024-09-21T10:53:15.625363+0000","last_became_active":"2024-09-21T10:52:25.492182+0000","last_became_peered":"2024-09-21T10:52:25.492182+0000","last_unstale":"2024-09-21T10:53:15.625363+0000","last_undegraded":"2024-09-21T10:53:15.625363+0000","last_fullsized":"2024-09-21T10:53:15.625363+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:12:57.118420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1466,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.613625+0000","last_change":"2024-09-21T10:52:15.402809+0000","last_active":"2024-09-21T10:53:15.613625+0000","last_peered":"2024-09-21T10:53:15.613625+0000","last_clean":"2024-09-21T10:53:15.613625+0000","last_became_active":"2024-09-21T10:52:15.402497+0000","last_became_peered":"2024-09-21T10:52:15.402497+0000","last_unstale":"2024-09-21T10:53:15.613625+0000","last_undegraded":"2024-09-21T10:53:15.613625+0000","last_fullsized":"2024-09-21T10:53:15.613625+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:07:35.713534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1477,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.613087+0000","last_change":"2024-09-21T10:44:14.399842+0000","last_active":"2024-09-21T10:53:15.613087+0000","last_peered":"2024-09-21T10:53:15.613087+0000","last_clean":"2024-09-21T10:53:15.613087+0000","last_became_active":"2024-09-21T10:44:14.399426+0000","last_became_peered":"2024-09-21T10:44:14.399426+0000","last_unstale":"2024-09-21T10:53:15.613087+0000","last_undegraded":"2024-09-21T10:53:15.613087+0000","last_fullsized":"2024-09-21T10:53:15.613087+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:26:45.362101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"346'149","reported_seq":906,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.624270+0000","last_change":"2024-09-21T10:52:16.404597+0000","last_active":"2024-09-21T10:53:15.624270+0000","last_peered":"2024-09-21T10:53:15.624270+0000","last_clean":"2024-09-21T10:53:15.624270+0000","last_became_active":"2024-09-21T10:52:16.404194+0000","last_became_peered":"2024-09-21T10:52:16.404194+0000","last_unstale":"2024-09-21T10:53:15.624270+0000","last_undegraded":"2024-09-21T10:53:15.624270+0000","last_fullsized":"2024-09-21T10:53:15.624270+0000","mapping_epoch":1463,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1464,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":793725,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1472,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.614796+0000","last_change":"2024-09-21T10:40:13.432387+0000","last_active":"2024-09-21T10:53:15.614796+0000","last_peered":"2024-09-21T10:53:15.614796+0000","last_clean":"2024-09-21T10:53:15.614796+0000","last_became_active":"2024-09-21T10:40:13.432053+0000","last_became_peered":"2024-09-21T10:40:13.432053+0000","last_unstale":"2024-09-21T10:53:15.614796+0000","last_undegraded":"2024-09-21T10:53:15.614796+0000","last_fullsized":"2024-09-21T10:53:15.614796+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:37.486101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1449,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.375034+0000","last_change":"2024-09-21T10:07:56.020791+0000","last_active":"2024-09-21T10:53:02.375034+0000","last_peered":"2024-09-21T10:53:02.375034+0000","last_clean":"2024-09-21T10:53:02.375034+0000","last_became_active":"2024-09-21T10:07:56.020205+0000","last_became_peered":"2024-09-21T10:07:56.020205+0000","last_unstale":"2024-09-21T10:53:02.375034+0000","last_undegraded":"2024-09-21T10:53:02.375034+0000","last_fullsized":"2024-09-21T10:53:02.375034+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:09.813166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"346'149","reported_seq":912,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.370491+0000","last_change":"2024-09-21T10:49:47.904295+0000","last_active":"2024-09-21T10:53:02.370491+0000","last_peered":"2024-09-21T10:53:02.370491+0000","last_clean":"2024-09-21T10:53:02.370491+0000","last_became_active":"2024-09-21T10:49:47.903650+0000","last_became_peered":"2024-09-21T10:49:47.903650+0000","last_unstale":"2024-09-21T10:53:02.370491+0000","last_undegraded":"2024-09-21T10:53:02.370491+0000","last_fullsized":"2024-09-21T10:53:02.370491+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":74,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1448,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.614490+0000","last_change":"2024-09-21T10:09:06.533271+0000","last_active":"2024-09-21T10:53:15.614490+0000","last_peered":"2024-09-21T10:53:15.614490+0000","last_clean":"2024-09-21T10:53:15.614490+0000","last_became_active":"2024-09-21T10:09:06.532041+0000","last_became_peered":"2024-09-21T10:09:06.532041+0000","last_unstale":"2024-09-21T10:53:15.614490+0000","last_undegraded":"2024-09-21T10:53:15.614490+0000","last_fullsized":"2024-09-21T10:53:15.614490+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:04.236375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":2254,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.624348+0000","last_change":"2024-09-21T10:51:27.529295+0000","last_active":"2024-09-21T10:53:15.624348+0000","last_peered":"2024-09-21T10:53:15.624348+0000","last_clean":"2024-09-21T10:53:15.624348+0000","last_became_active":"2024-09-21T10:51:27.446359+0000","last_became_peered":"2024-09-21T10:51:27.446359+0000","last_unstale":"2024-09-21T10:53:15.624348+0000","last_undegraded":"2024-09-21T10:53:15.624348+0000","last_fullsized":"2024-09-21T10:53:15.624348+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:17:08.788558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":684,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.623548+0000","last_change":"2024-09-21T10:51:27.481103+0000","last_active":"2024-09-21T10:53:15.623548+0000","last_peered":"2024-09-21T10:53:15.623548+0000","last_clean":"2024-09-21T10:53:15.623548+0000","last_became_active":"2024-09-21T10:51:27.480763+0000","last_became_peered":"2024-09-21T10:51:27.480763+0000","last_unstale":"2024-09-21T10:53:15.623548+0000","last_undegraded":"2024-09-21T10:53:15.623548+0000","last_fullsized":"2024-09-21T10:53:15.623548+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:35.375303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":790,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.625708+0000","last_change":"2024-09-21T10:49:44.864819+0000","last_active":"2024-09-21T10:53:15.625708+0000","last_peered":"2024-09-21T10:53:15.625708+0000","last_clean":"2024-09-21T10:53:15.625708+0000","last_became_active":"2024-09-21T10:49:44.864061+0000","last_became_peered":"2024-09-21T10:49:44.864061+0000","last_unstale":"2024-09-21T10:53:15.625708+0000","last_undegraded":"2024-09-21T10:53:15.625708+0000","last_fullsized":"2024-09-21T10:53:15.625708+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:37.266967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":31,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.374370+0000","last_change":"2024-09-21T10:52:25.492570+0000","last_active":"2024-09-21T10:53:02.374370+0000","last_peered":"2024-09-21T10:53:02.374370+0000","last_clean":"2024-09-21T10:53:02.374370+0000","last_became_active":"2024-09-21T10:52:25.486382+0000","last_became_peered":"2024-09-21T10:52:25.486382+0000","last_unstale":"2024-09-21T10:53:02.374370+0000","last_undegraded":"2024-09-21T10:53:02.374370+0000","last_fullsized":"2024-09-21T10:53:02.374370+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:06.702707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":136,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.629581+0000","last_change":"2024-09-21T10:49:44.874522+0000","last_active":"2024-09-21T10:53:15.629581+0000","last_peered":"2024-09-21T10:53:15.629581+0000","last_clean":"2024-09-21T10:53:15.629581+0000","last_became_active":"2024-09-21T10:49:44.863242+0000","last_became_peered":"2024-09-21T10:49:44.863242+0000","last_unstale":"2024-09-21T10:53:15.629581+0000","last_undegraded":"2024-09-21T10:53:15.629581+0000","last_fullsized":"2024-09-21T10:53:15.629581+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:09:02.066837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1482,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.612881+0000","last_change":"2024-09-21T10:52:03.315267+0000","last_active":"2024-09-21T10:53:15.612881+0000","last_peered":"2024-09-21T10:53:15.612881+0000","last_clean":"2024-09-21T10:53:15.612881+0000","last_became_active":"2024-09-21T10:52:03.314474+0000","last_became_peered":"2024-09-21T10:52:03.314474+0000","last_unstale":"2024-09-21T10:53:15.612881+0000","last_undegraded":"2024-09-21T10:53:15.612881+0000","last_fullsized":"2024-09-21T10:53:15.612881+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:00:57.791341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":23,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.371900+0000","last_change":"2024-09-21T10:52:25.518721+0000","last_active":"2024-09-21T10:53:02.371900+0000","last_peered":"2024-09-21T10:53:02.371900+0000","last_clean":"2024-09-21T10:53:02.371900+0000","last_became_active":"2024-09-21T10:52:25.472114+0000","last_became_peered":"2024-09-21T10:52:25.472114+0000","last_unstale":"2024-09-21T10:53:02.371900+0000","last_undegraded":"2024-09-21T10:53:02.371900+0000","last_fullsized":"2024-09-21T10:53:02.371900+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:33:48.471147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.370247+0000","last_change":"2024-09-21T10:52:25.494490+0000","last_active":"2024-09-21T10:53:02.370247+0000","last_peered":"2024-09-21T10:53:02.370247+0000","last_clean":"2024-09-21T10:53:02.370247+0000","last_became_active":"2024-09-21T10:52:25.494200+0000","last_became_peered":"2024-09-21T10:52:25.494200+0000","last_unstale":"2024-09-21T10:53:02.370247+0000","last_undegraded":"2024-09-21T10:53:02.370247+0000","last_fullsized":"2024-09-21T10:53:02.370247+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:15:21.030031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":23,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.370897+0000","last_change":"2024-09-21T10:52:25.479229+0000","last_active":"2024-09-21T10:53:02.370897+0000","last_peered":"2024-09-21T10:53:02.370897+0000","last_clean":"2024-09-21T10:53:02.370897+0000","last_became_active":"2024-09-21T10:52:25.470551+0000","last_became_peered":"2024-09-21T10:52:25.470551+0000","last_unstale":"2024-09-21T10:53:02.370897+0000","last_undegraded":"2024-09-21T10:53:02.370897+0000","last_fullsized":"2024-09-21T10:53:02.370897+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:15.458231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1494,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:11.761902+0000","last_change":"2024-09-21T10:51:27.517234+0000","last_active":"2024-09-21T10:53:11.761902+0000","last_peered":"2024-09-21T10:53:11.761902+0000","last_clean":"2024-09-21T10:53:11.761902+0000","last_became_active":"2024-09-21T10:51:27.431253+0000","last_became_peered":"2024-09-21T10:51:27.431253+0000","last_unstale":"2024-09-21T10:53:11.761902+0000","last_undegraded":"2024-09-21T10:53:11.761902+0000","last_fullsized":"2024-09-21T10:53:11.761902+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:35:42.249305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2209,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.624936+0000","last_change":"2024-09-21T10:52:25.497298+0000","last_active":"2024-09-21T10:53:15.624936+0000","last_peered":"2024-09-21T10:53:15.624936+0000","last_clean":"2024-09-21T10:53:15.624936+0000","last_became_active":"2024-09-21T10:52:25.496876+0000","last_became_peered":"2024-09-21T10:52:25.496876+0000","last_unstale":"2024-09-21T10:53:15.624936+0000","last_undegraded":"2024-09-21T10:53:15.624936+0000","last_fullsized":"2024-09-21T10:53:15.624936+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:16.587897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":118,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.370634+0000","last_change":"2024-09-21T10:44:14.398878+0000","last_active":"2024-09-21T10:53:02.370634+0000","last_peered":"2024-09-21T10:53:02.370634+0000","last_clean":"2024-09-21T10:53:02.370634+0000","last_became_active":"2024-09-21T10:44:14.398547+0000","last_became_peered":"2024-09-21T10:44:14.398547+0000","last_unstale":"2024-09-21T10:53:02.370634+0000","last_undegraded":"2024-09-21T10:53:02.370634+0000","last_fullsized":"2024-09-21T10:53:02.370634+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:13.008134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2257,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.628869+0000","last_change":"2024-09-21T10:52:25.498554+0000","last_active":"2024-09-21T10:53:15.628869+0000","last_peered":"2024-09-21T10:53:15.628869+0000","last_clean":"2024-09-21T10:53:15.628869+0000","last_became_active":"2024-09-21T10:52:25.498194+0000","last_became_peered":"2024-09-21T10:52:25.498194+0000","last_unstale":"2024-09-21T10:53:15.628869+0000","last_undegraded":"2024-09-21T10:53:15.628869+0000","last_fullsized":"2024-09-21T10:53:15.628869+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:47.115271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.370770+0000","last_change":"2024-09-21T10:52:25.489063+0000","last_active":"2024-09-21T10:53:02.370770+0000","last_peered":"2024-09-21T10:53:02.370770+0000","last_clean":"2024-09-21T10:53:02.370770+0000","last_became_active":"2024-09-21T10:52:25.488129+0000","last_became_peered":"2024-09-21T10:52:25.488129+0000","last_unstale":"2024-09-21T10:53:02.370770+0000","last_undegraded":"2024-09-21T10:53:02.370770+0000","last_fullsized":"2024-09-21T10:53:02.370770+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:36.996829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1433,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.370880+0000","last_change":"2024-09-21T10:49:44.864898+0000","last_active":"2024-09-21T10:53:02.370880+0000","last_peered":"2024-09-21T10:53:02.370880+0000","last_clean":"2024-09-21T10:53:02.370880+0000","last_became_active":"2024-09-21T10:49:44.862691+0000","last_became_peered":"2024-09-21T10:49:44.862691+0000","last_unstale":"2024-09-21T10:53:02.370880+0000","last_undegraded":"2024-09-21T10:53:02.370880+0000","last_fullsized":"2024-09-21T10:53:02.370880+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:57.809113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":124,"reported_epoch":1473,"state":"active+clean","last_fresh":"2024-09-21T10:53:02.371248+0000","last_change":"2024-09-21T10:44:14.418113+0000","last_active":"2024-09-21T10:53:02.371248+0000","last_peered":"2024-09-21T10:53:02.371248+0000","last_clean":"2024-09-21T10:53:02.371248+0000","last_became_active":"2024-09-21T10:44:14.398327+0000","last_became_peered":"2024-09-21T10:44:14.398327+0000","last_unstale":"2024-09-21T10:53:02.371248+0000","last_undegraded":"2024-09-21T10:53:02.371248+0000","last_fullsized":"2024-09-21T10:53:02.371248+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:08.810863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2238,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:53:15.625808+0000","last_change":"2024-09-21T10:44:14.398584+0000","last_active":"2024-09-21T10:53:15.625808+0000","last_peered":"2024-09-21T10:53:15.625808+0000","last_clean":"2024-09-21T10:53:15.625808+0000","last_became_active":"2024-09-21T10:44:14.398278+0000","last_became_peered":"2024-09-21T10:44:14.398278+0000","last_unstale":"2024-09-21T10:53:15.625808+0000","last_undegraded":"2024-09-21T10:53:15.625808+0000","last_fullsized":"2024-09-21T10:53:15.625808+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:19.681750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":55,"num_pg":16,"stat_sum":{"num_bytes":933888024,"num_objects":14251,"num_object_clones":0,"num_object_copies":42753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12514,"num_objects_misplaced":6173,"num_objects_unfound":0,"num_objects_dirty":14251,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196427,"num_write_kb":6741633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45867,"num_bytes_recovered":2884829184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6886141952,"data_stored":6886129736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74286,"ondisk_log_size":74286,"up":48,"acting":44,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":429359,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":276,"num_read_kb":598,"num_write":157,"num_write_kb":1757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3655209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":32,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":95,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":11344,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":18188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234713,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3065420,"kb_used_data":2502724,"kb_used_omap":1,"kb_used_meta":562622,"kb_avail":90679732,"statfs":{"total":95995035648,"available":92856045568,"internally_reserved":0,"allocated":2562789376,"data_stored":2557787255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":576125387},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496350,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1400980,"kb_used_data":720204,"kb_used_omap":1,"kb_used_meta":680766,"kb_avail":92344172,"statfs":{"total":95995035648,"available":94560432128,"internally_reserved":0,"allocated":737488896,"data_stored":732479706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":697104842},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757987,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019623,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1040408,"kb_used_data":387536,"kb_used_omap":1,"kb_used_meta":652862,"kb_avail":92704744,"statfs":{"total":95995035648,"available":94929657856,"internally_reserved":0,"allocated":396836864,"data_stored":391823554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":668531142},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369366,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1648136,"kb_used_data":876032,"kb_used_omap":69,"kb_used_meta":772026,"kb_avail":92097016,"statfs":{"total":95995035648,"available":94307344384,"internally_reserved":0,"allocated":897056768,"data_stored":892055112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":790554754},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2920577761822,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807958053,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1099452,"kb_used_data":413556,"kb_used_omap":66,"kb_used_meta":685821,"kb_avail":92645700,"statfs":{"total":95995035648,"available":94869196800,"internally_reserved":0,"allocated":423481344,"data_stored":418490778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68589,"internal_metadata":702280723},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":2860448219692,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":536875008,"data_stored":536870936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":403898368,"data_stored":403898368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":856428544,"data_stored":856424472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":381227008,"data_stored":381222936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1456472064,"data_stored":1456472064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":719454208,"data_stored":719454208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2531786752,"data_stored":2531786752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:53:34.608 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph status --format=json 2024-09-21T10:53:34.673 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:34.704 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:34.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:34.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:34.805 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:34.835 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:34.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:34.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:34.937 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:34.984 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:34.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:34.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:35.086 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:35.120 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:35.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:35.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:35.317 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:35.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:35.359 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:35.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:35.462 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:35.507 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:35.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:35.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:35.609 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:35.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:35.642 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:35.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:35.743 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:35.774 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:35.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:35.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:35.876 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:35.909 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:35.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:35.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:36.011 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:36.044 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:36.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:36.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:36.146 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:36.180 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:36.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:36.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:36.281 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:36 smithi070 ceph-mon[107385]: pgmap v2715: 99 pgs: 1 active+recovery_wait+undersized+remapped, 2 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 601 MiB data, 7.2 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 736 op/s; 10834/28899 objects degraded (37.489%); 5060/28899 objects misplaced (17.509%); 13 MiB/s, 265 objects/s recovering 2024-09-21T10:53:36.281 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:53:36.281 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:36 smithi070 ceph-mon[112139]: pgmap v2715: 99 pgs: 1 active+recovery_wait+undersized+remapped, 2 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 601 MiB data, 7.2 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 736 op/s; 10834/28899 objects degraded (37.489%); 5060/28899 objects misplaced (17.509%); 13 MiB/s, 265 objects/s recovering 2024-09-21T10:53:36.281 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:53:36.282 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:36.307 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:53:36.316 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:53:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:36 smithi160 ceph-mon[101879]: pgmap v2715: 99 pgs: 1 active+recovery_wait+undersized+remapped, 2 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 601 MiB data, 7.2 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 736 op/s; 10834/28899 objects degraded (37.489%); 5060/28899 objects misplaced (17.509%); 13 MiB/s, 265 objects/s recovering 2024-09-21T10:53:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:53:36.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:36.342 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:36.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:36.443 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:36.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:36.480 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:36.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:36.581 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:36.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:36.621 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:36.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:36.721 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:36.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:36.755 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:36.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:36.856 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:36.893 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:36.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:36.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:36.995 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:37.025 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:37.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:37.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:37.076 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:53:37.076 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T10:53:37.076 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.076 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:205945 55.7 e1474/1411 55:e18dcc00:::benchmark_data_smithi070_294394_object100604:head v 1474'12997, mlcod=1474'12995)", 2024-09-21T10:53:37.076 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:53:37.070214+0000", 2024-09-21T10:53:37.076 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0028752510000000001, 2024-09-21T10:53:37.076 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0029041219999999999, 2024-09-21T10:53:37.076 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:53:37.076 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:53:37.076 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "started", 2024-09-21T10:53:37.076 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:53:37.076 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.077 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:53:37.077 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.070214+0000", 2024-09-21T10:53:37.077 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:37.077 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.077 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.077 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:53:37.077 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.070214+0000", 2024-09-21T10:53:37.077 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:37.077 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.077 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.077 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:53:37.077 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.070215+0000", 2024-09-21T10:53:37.077 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7300000000000005e-07 2024-09-21T10:53:37.077 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.078 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.078 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:53:37.078 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.070222+0000", 2024-09-21T10:53:37.078 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5880000000000003e-06 2024-09-21T10:53:37.078 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.078 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.078 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:53:37.078 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.070237+0000", 2024-09-21T10:53:37.078 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5118e-05 2024-09-21T10:53:37.078 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.078 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.078 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:53:37.078 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.070244+0000", 2024-09-21T10:53:37.079 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6140000000000003e-06 2024-09-21T10:53:37.079 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.079 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.079 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:37.079 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.070315+0000", 2024-09-21T10:53:37.079 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0401999999999994e-05 2024-09-21T10:53:37.079 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.079 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.079 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:53:37.079 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.070338+0000", 2024-09-21T10:53:37.079 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3524e-05 2024-09-21T10:53:37.079 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:37.079 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:53:37.080 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:37.080 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.080 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.080 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:205944 55.8 e1474/1448 55:15d7bfa1:::benchmark_data_smithi070_294394_object100603:head v 1474'12728, mlcod=1474'12726)", 2024-09-21T10:53:37.080 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:53:37.071152+0000", 2024-09-21T10:53:37.080 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.001937743, 2024-09-21T10:53:37.080 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0020244600000000001, 2024-09-21T10:53:37.080 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:53:37.080 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:53:37.080 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "started", 2024-09-21T10:53:37.080 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:53:37.080 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.080 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:53:37.080 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.071152+0000", 2024-09-21T10:53:37.081 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:37.081 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.081 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.081 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:53:37.081 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.071152+0000", 2024-09-21T10:53:37.081 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:37.081 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.081 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.081 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:53:37.081 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.071153+0000", 2024-09-21T10:53:37.081 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4600000000000004e-07 2024-09-21T10:53:37.081 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.081 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.082 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:53:37.082 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.071158+0000", 2024-09-21T10:53:37.082 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.891e-06 2024-09-21T10:53:37.082 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.082 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.082 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:53:37.082 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.071171+0000", 2024-09-21T10:53:37.082 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2942000000000001e-05 2024-09-21T10:53:37.082 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.082 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.082 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:53:37.082 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.071178+0000", 2024-09-21T10:53:37.082 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6069999999999996e-06 2024-09-21T10:53:37.082 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.082 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.083 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:37.083 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.071358+0000", 2024-09-21T10:53:37.083 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017986099999999999 2024-09-21T10:53:37.083 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.083 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.083 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:53:37.083 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.071417+0000", 2024-09-21T10:53:37.083 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8879999999999999e-05 2024-09-21T10:53:37.083 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:37.083 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:53:37.083 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:37.083 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.083 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.083 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:205928 55.2 55:4cb5b368:::benchmark_data_smithi070_294394_object100587:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1474)", 2024-09-21T10:53:37.084 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:53:37.053240+0000", 2024-09-21T10:53:37.084 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.019849072999999998, 2024-09-21T10:53:37.084 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.019983562, 2024-09-21T10:53:37.084 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:53:37.084 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:53:37.084 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:53:37.084 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:53:37.084 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:53:37.084 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:53:37.084 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 205928 2024-09-21T10:53:37.084 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.084 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:53:37.084 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.084 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:53:37.085 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.053240+0000", 2024-09-21T10:53:37.085 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:37.085 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.085 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.085 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:53:37.085 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.053240+0000", 2024-09-21T10:53:37.085 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:53:37.085 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.085 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.085 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:53:37.085 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.053242+0000", 2024-09-21T10:53:37.085 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.683e-06 2024-09-21T10:53:37.085 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.085 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.086 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:53:37.086 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.053248+0000", 2024-09-21T10:53:37.086 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7049999999999996e-06 2024-09-21T10:53:37.086 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.086 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.086 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:53:37.086 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.053265+0000", 2024-09-21T10:53:37.086 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7201999999999999e-05 2024-09-21T10:53:37.086 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.086 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.086 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:53:37.086 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.053274+0000", 2024-09-21T10:53:37.086 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2240000000000001e-06 2024-09-21T10:53:37.087 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.087 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.087 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:37.087 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.053366+0000", 2024-09-21T10:53:37.087 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1983000000000003e-05 2024-09-21T10:53:37.087 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.087 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.087 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:53:37.087 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.053973+0000", 2024-09-21T10:53:37.087 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000606318 2024-09-21T10:53:37.087 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.087 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.087 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:53:37.087 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.060076+0000", 2024-09-21T10:53:37.088 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0061034169999999999 2024-09-21T10:53:37.088 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.088 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.088 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:53:37.088 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.060167+0000", 2024-09-21T10:53:37.088 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0699999999999996e-05 2024-09-21T10:53:37.088 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.088 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.088 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,6,7", 2024-09-21T10:53:37.088 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.060304+0000", 2024-09-21T10:53:37.088 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013773600000000001 2024-09-21T10:53:37.088 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.088 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.088 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:53:37.089 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.066918+0000", 2024-09-21T10:53:37.089 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0066138359999999997 2024-09-21T10:53:37.089 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.089 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.089 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:53:37.089 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.066959+0000", 2024-09-21T10:53:37.089 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0419e-05 2024-09-21T10:53:37.089 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:53:37.089 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:53:37.089 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:53:37.089 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:37.069411+0000", 2024-09-21T10:53:37.089 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0024525079999999999 2024-09-21T10:53:37.089 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:37.089 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:53:37.090 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:37.090 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:53:37.090 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-09-21T10:53:37.090 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 3 2024-09-21T10:53:37.090 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:53:37.126 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:37.180 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:37.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:37.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:37.281 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:37.315 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:37.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:37.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:37.417 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:37.450 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:37.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:37.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:37.552 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:37.585 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:37.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:37.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:37.688 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:37.722 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:37.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:37.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:37.824 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:37.859 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:37.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:37.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:37.961 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:38.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:38.005 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:38.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:38.036 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T10:53:38.106 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:38.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:38.143 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:38.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:38.244 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:38.279 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:38.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:38.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:38.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:38 smithi160 ceph-mon[101879]: pgmap v2716: 99 pgs: 1 active+recovery_wait+undersized+remapped, 2 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 413 MiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 942 op/s; 9326/19899 objects degraded (46.867%); 4454/19899 objects misplaced (22.383%); 12 MiB/s, 249 objects/s recovering 2024-09-21T10:53:38.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:38 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1868081064' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:53:38.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:38 smithi070 ceph-mon[107385]: pgmap v2716: 99 pgs: 1 active+recovery_wait+undersized+remapped, 2 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 413 MiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 942 op/s; 9326/19899 objects degraded (46.867%); 4454/19899 objects misplaced (22.383%); 12 MiB/s, 249 objects/s recovering 2024-09-21T10:53:38.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:38 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1868081064' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:53:38.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:38 smithi070 ceph-mon[112139]: pgmap v2716: 99 pgs: 1 active+recovery_wait+undersized+remapped, 2 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 413 MiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 942 op/s; 9326/19899 objects degraded (46.867%); 4454/19899 objects misplaced (22.383%); 12 MiB/s, 249 objects/s recovering 2024-09-21T10:53:38.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:38 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1868081064' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:53:38.381 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:38.410 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:38.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:38.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:38.512 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:38.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:38.551 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:38.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:38.581 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:53:38.652 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:38.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:38.716 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:38.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:38.817 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:38.852 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:38.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:38.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:38.954 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:38.991 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:38.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:38.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:39.093 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:39.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:39.129 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:39.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:39.230 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:39.268 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:39.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:39.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:39.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:39 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:53:39.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:39 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1868081064' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:53:39.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:39 smithi160 ceph-mon[101879]: osdmap e1475: 8 total, 8 up, 5 in 2024-09-21T10:53:39.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:39 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 9326/19899 objects degraded (46.867%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:39.370 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:39.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:39 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:53:39.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:39 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1868081064' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:53:39.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:39 smithi070 ceph-mon[112139]: osdmap e1475: 8 total, 8 up, 5 in 2024-09-21T10:53:39.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:39 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 9326/19899 objects degraded (46.867%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:39.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:39 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:53:39.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:39 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1868081064' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:53:39.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:39 smithi070 ceph-mon[107385]: osdmap e1475: 8 total, 8 up, 5 in 2024-09-21T10:53:39.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:39 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 9326/19899 objects degraded (46.867%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:39.409 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:39.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:39.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:39.511 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:39.542 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:39.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:39.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:39.644 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:39.676 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:39.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:39.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:39.777 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:39.812 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:39.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:39.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:39.914 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:39.948 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:39.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:39.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:40.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:40 smithi070 ceph-mon[107385]: pgmap v2718: 99 pgs: 1 active+recovery_wait+undersized+remapped, 2 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 302 MiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 942 op/s; 9035/14544 objects degraded (62.122%); 3880/14544 objects misplaced (26.678%); 7.9 MiB/s, 207 objects/s recovering 2024-09-21T10:53:40.050 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:40.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:40.091 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:40.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:40.192 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:40.224 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:40.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:40.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:40.326 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:40.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:40 smithi160 ceph-mon[101879]: pgmap v2718: 99 pgs: 1 active+recovery_wait+undersized+remapped, 2 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 302 MiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 942 op/s; 9035/14544 objects degraded (62.122%); 3880/14544 objects misplaced (26.678%); 7.9 MiB/s, 207 objects/s recovering 2024-09-21T10:53:40.347 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:40 smithi070 ceph-mon[112139]: pgmap v2718: 99 pgs: 1 active+recovery_wait+undersized+remapped, 2 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 302 MiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 942 op/s; 9035/14544 objects degraded (62.122%); 3880/14544 objects misplaced (26.678%); 7.9 MiB/s, 207 objects/s recovering 2024-09-21T10:53:40.364 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:40.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:40.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:40.466 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:40.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:40.505 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:40.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:40.606 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:40.642 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:40.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:40.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:40.744 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:40.798 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:40.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:40.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:40.900 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:40.940 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:40.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:40.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:41.042 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:41.081 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:41.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:41.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:41.183 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:41.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:41.233 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:41.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:41.333 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:41.377 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:41.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:41.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:41.479 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:41.513 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:41.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:41.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:41.615 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:41.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:41.652 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:41.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:41.652 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Removed 105338 objects 2024-09-21T10:53:41.652 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Clean up completed and total clean up time :190.821 2024-09-21T10:53:41.753 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:41.774 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:41.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:41.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:41.876 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:41.898 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:41.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:41.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:42.000 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:42.012 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:53:42.025 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:42.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:42.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:42.126 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:42.148 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:42.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:42.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:42.250 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:42.274 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:42.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:42.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:42.275 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:42 smithi070 ceph-mon[107385]: pgmap v2719: 99 pgs: 1 active+recovery_wait+undersized+remapped, 2 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 302 MiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 942 op/s; 9035/14544 objects degraded (62.122%); 3880/14544 objects misplaced (26.678%); 7.9 MiB/s, 207 objects/s recovering 2024-09-21T10:53:42.275 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:42 smithi070 ceph-mon[112139]: pgmap v2719: 99 pgs: 1 active+recovery_wait+undersized+remapped, 2 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 302 MiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 942 op/s; 9035/14544 objects degraded (62.122%); 3880/14544 objects misplaced (26.678%); 7.9 MiB/s, 207 objects/s recovering 2024-09-21T10:53:42.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:42 smithi160 ceph-mon[101879]: pgmap v2719: 99 pgs: 1 active+recovery_wait+undersized+remapped, 2 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 302 MiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 942 op/s; 9035/14544 objects degraded (62.122%); 3880/14544 objects misplaced (26.678%); 7.9 MiB/s, 207 objects/s recovering 2024-09-21T10:53:42.375 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:42.397 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:42.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:42.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:42.498 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:42.520 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:42.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:42.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:42.553 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:53:42.553 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:53:42.553 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:53:42.553 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:53:42.553 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:53:42.553 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:53:42.554 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:53:42.554 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:53:42.554 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:53:42.554 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:53:42.554 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:53:42.554 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:53:42.562 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:53:42.562 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:53:42.581 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:53:42.581 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:53:42.621 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:42.643 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:42.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:42.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:42.745 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:42.766 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:42.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:42.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:42.862 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:53:42.862 INFO:teuthology.orchestra.run.smithi070.stdout:{"fsid":"7c68bfbc-77fe-11ef-baf6-efdc52797490","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9035/14544 objects degraded (62.122%), 8 pgs degraded, 5 pgs undersized","count":13},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":3040,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":1476,"num_osds":8,"num_up_osds":8,"osd_up_since":1726913345,"num_in_osds":5,"osd_in_since":1726915886,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":86},{"state_name":"active+recovery_wait+degraded+remapped","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovery_wait+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":1},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovery_wait+undersized+remapped","count":1}],"num_pgs":99,"num_pools":5,"num_objects":4848,"data_bytes":316384012,"bytes_used":6899580928,"bytes_avail":473075597312,"bytes_total":479975178240,"degraded_objects":9035,"degraded_total":14544,"degraded_ratio":0.62121837183718376,"misplaced_objects":3880,"misplaced_total":14544,"misplaced_ratio":0.2667766776677668,"recovering_objects_per_sec":207,"recovering_bytes_per_sec":8306745,"recovering_keys_per_sec":0,"num_objects_recovered":2080,"num_bytes_recovered":83099648,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":942},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi070.gansda","status":"up:active","gid":14652}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.70:8443/","prometheus":"http://172.21.15.70:9283/"}},"servicemap":{"epoch":37,"modified":"2024-09-21T10:53:20.935827+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-21T10:53:42.868 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:42.892 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:42.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:42.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:42.993 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:43.016 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:43.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:43.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:43.117 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:43.139 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:43.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:43.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:43.241 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:43.262 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:43.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:43.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:43.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:43 smithi160 ceph-mon[101879]: osdmap e1476: 8 total, 8 up, 5 in 2024-09-21T10:53:43.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:43 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2135036223' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:53:43.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:43 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1539270763' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:53:43.365 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:43.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:53:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:53:43] "GET /metrics HTTP/1.1" 200 36626 "" "Prometheus/2.51.0" 2024-09-21T10:53:43.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:43 smithi070 ceph-mon[107385]: osdmap e1476: 8 total, 8 up, 5 in 2024-09-21T10:53:43.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:43 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2135036223' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:53:43.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:43 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1539270763' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:53:43.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:43 smithi070 ceph-mon[112139]: osdmap e1476: 8 total, 8 up, 5 in 2024-09-21T10:53:43.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:43 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2135036223' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:53:43.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:43 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1539270763' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:53:43.386 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:43.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:43.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:43.489 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:43.510 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:43.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:43.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:43.612 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:43.633 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:43.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:43.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:43.735 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:43.756 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:43.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:43.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:43.772 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:53:43.858 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:43.879 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:43.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:43.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:43.981 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:44.001 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:44.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:44.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:44.103 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:44.125 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:44.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:44.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:44.226 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:44.250 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:44.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:44.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:44.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:44 smithi160 ceph-mon[101879]: pgmap v2721: 99 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+remapped, 2 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+undersized+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 26 MiB data, 5.7 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 1.15k op/s; 7718/1326 objects degraded (582.051%); 3257/1326 objects misplaced (245.626%); 5.4 MiB/s, 203 objects/s recovering 2024-09-21T10:53:44.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:44 smithi160 ceph-mon[101879]: osdmap e1477: 8 total, 8 up, 5 in 2024-09-21T10:53:44.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:44 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 7718/1326 objects degraded (582.051%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:44.351 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:44.373 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:44.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:44.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:44.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:44 smithi070 ceph-mon[107385]: pgmap v2721: 99 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+remapped, 2 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+undersized+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 26 MiB data, 5.7 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 1.15k op/s; 7718/1326 objects degraded (582.051%); 3257/1326 objects misplaced (245.626%); 5.4 MiB/s, 203 objects/s recovering 2024-09-21T10:53:44.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:44 smithi070 ceph-mon[107385]: osdmap e1477: 8 total, 8 up, 5 in 2024-09-21T10:53:44.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:44 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 7718/1326 objects degraded (582.051%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:44.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:44 smithi070 ceph-mon[112139]: pgmap v2721: 99 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+remapped, 2 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+undersized+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 26 MiB data, 5.7 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 1.15k op/s; 7718/1326 objects degraded (582.051%); 3257/1326 objects misplaced (245.626%); 5.4 MiB/s, 203 objects/s recovering 2024-09-21T10:53:44.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:44 smithi070 ceph-mon[112139]: osdmap e1477: 8 total, 8 up, 5 in 2024-09-21T10:53:44.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:44 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 7718/1326 objects degraded (582.051%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:44.475 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:44.496 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:44.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:44.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:44.598 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:44.620 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:44.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:44.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:44.721 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:44.742 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:44.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:44.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:44.844 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:44.865 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:44.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:44.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:44.967 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:44.988 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:44.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:44.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:45.090 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:45.112 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:45.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:45.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:45.214 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:45.236 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:45.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:45.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:45.338 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:45.360 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:45.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:45.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:45.462 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:45.484 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:45.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:45.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:45.586 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:45.608 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:45.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:45.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:45.709 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:45.731 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:45.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:45.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:45.833 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:45.856 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:45.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:45.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:45.958 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:45.979 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:45.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:45.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:46.081 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:46.104 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:46.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:46.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:46.206 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:46.228 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:46.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:46.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:46.309 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:46 smithi070 ceph-mon[107385]: pgmap v2723: 99 pgs: 1 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+remapped, 1 active+recovering+remapped, 3 active+recovery_wait+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.7 MiB data, 5.6 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 775 op/s; 7525/192 objects degraded (3919.271%); 2879/192 objects misplaced (1499.479%); 5.0 MiB/s, 152 objects/s recovering 2024-09-21T10:53:46.309 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:46 smithi070 ceph-mon[112139]: pgmap v2723: 99 pgs: 1 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+remapped, 1 active+recovering+remapped, 3 active+recovery_wait+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.7 MiB data, 5.6 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 775 op/s; 7525/192 objects degraded (3919.271%); 2879/192 objects misplaced (1499.479%); 5.0 MiB/s, 152 objects/s recovering 2024-09-21T10:53:46.329 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:46.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:46 smithi160 ceph-mon[101879]: pgmap v2723: 99 pgs: 1 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+remapped, 1 active+recovering+remapped, 3 active+recovery_wait+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.7 MiB data, 5.6 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 775 op/s; 7525/192 objects degraded (3919.271%); 2879/192 objects misplaced (1499.479%); 5.0 MiB/s, 152 objects/s recovering 2024-09-21T10:53:46.351 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:46.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:46.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:46.453 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:46.476 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:46.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:46.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:46.578 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:46.599 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:46.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:46.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:46.701 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:46.723 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:46.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:46.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:46.825 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:46.848 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:46.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:46.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:46.950 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:46.973 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:46.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:46.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:47.068 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:53:47.075 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:47.139 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:47.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:47.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:47.241 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:47.266 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:47.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:47.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:47.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:47 smithi160 ceph-mon[101879]: osdmap e1478: 8 total, 8 up, 5 in 2024-09-21T10:53:47.367 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:47.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:47 smithi070 ceph-mon[107385]: osdmap e1478: 8 total, 8 up, 5 in 2024-09-21T10:53:47.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:47 smithi070 ceph-mon[112139]: osdmap e1478: 8 total, 8 up, 5 in 2024-09-21T10:53:47.392 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:47.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:47.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:47.492 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-09-21T10:53:47.492 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-09-21T10:53:47.494 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:47.516 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:47.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:47.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:47.618 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:47.642 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:47.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:47.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:47.743 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:47.765 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:47.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:47.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:47.867 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:47.890 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:47.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:47.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:47.992 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:48.014 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:48.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:48.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:48.116 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:48.137 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:48.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:48.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:48.239 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:48.262 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:48.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:48.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:48.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:48 smithi160 ceph-mon[101879]: pgmap v2725: 99 pgs: 2 active+remapped+backfill_wait, 1 peering, 1 active+recovery_wait+undersized+remapped, 1 active+recovering+remapped, 4 active+recovery_wait+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 425 KiB data, 5.6 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 803 op/s; 7516/78 objects degraded (9635.897%); 2879/78 objects misplaced (3691.026%); 2.8 MiB/s, 459 objects/s recovering 2024-09-21T10:53:48.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:48 smithi160 ceph-mon[101879]: osdmap e1479: 8 total, 8 up, 5 in 2024-09-21T10:53:48.364 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:48.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:48 smithi070 ceph-mon[107385]: pgmap v2725: 99 pgs: 2 active+remapped+backfill_wait, 1 peering, 1 active+recovery_wait+undersized+remapped, 1 active+recovering+remapped, 4 active+recovery_wait+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 425 KiB data, 5.6 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 803 op/s; 7516/78 objects degraded (9635.897%); 2879/78 objects misplaced (3691.026%); 2.8 MiB/s, 459 objects/s recovering 2024-09-21T10:53:48.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:48 smithi070 ceph-mon[107385]: osdmap e1479: 8 total, 8 up, 5 in 2024-09-21T10:53:48.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:48 smithi070 ceph-mon[112139]: pgmap v2725: 99 pgs: 2 active+remapped+backfill_wait, 1 peering, 1 active+recovery_wait+undersized+remapped, 1 active+recovering+remapped, 4 active+recovery_wait+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 425 KiB data, 5.6 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 803 op/s; 7516/78 objects degraded (9635.897%); 2879/78 objects misplaced (3691.026%); 2.8 MiB/s, 459 objects/s recovering 2024-09-21T10:53:48.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:48 smithi070 ceph-mon[112139]: osdmap e1479: 8 total, 8 up, 5 in 2024-09-21T10:53:48.387 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:48.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:48.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:48.489 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:48.512 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:48.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:48.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:48.614 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:48.635 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:48.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:48.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:48.737 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:48.761 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:48.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:48.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:48.862 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:48.884 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:48.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:48.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:48.986 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:49.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:49.008 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:49.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:49.109 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:49.130 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:49.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:49.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:49.232 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:49.254 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:49.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:49.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:49.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:49 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 7516/78 objects degraded (9635.897%), 2 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:49.358 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:49.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:49 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 7516/78 objects degraded (9635.897%), 2 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:49.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:49 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 7516/78 objects degraded (9635.897%), 2 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:49.379 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:49.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:49.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:49.480 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:49.503 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:49.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:49.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:49.605 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:49.626 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:49.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:49.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:49.728 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:49.750 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:49.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:49.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:49.852 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:49.874 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:49.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:49.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:49.976 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:49.996 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:49.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:49.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:50.098 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:50.121 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:50.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:50.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:50.222 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:50.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:50.246 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:50.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:50.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:50 smithi160 ceph-mon[101879]: pgmap v2727: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 1 peering, 1 active+recovery_wait+undersized+remapped, 1 active+recovering+remapped, 4 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 425 KiB data, 5.6 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 69 op/s; 5233/78 objects degraded (6708.974%); 2879/78 objects misplaced (3691.026%); 2.1 MiB/s, 436 objects/s recovering 2024-09-21T10:53:50.347 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:50.368 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:50.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:50.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:50.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:50 smithi070 ceph-mon[107385]: pgmap v2727: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 1 peering, 1 active+recovery_wait+undersized+remapped, 1 active+recovering+remapped, 4 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 425 KiB data, 5.6 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 69 op/s; 5233/78 objects degraded (6708.974%); 2879/78 objects misplaced (3691.026%); 2.1 MiB/s, 436 objects/s recovering 2024-09-21T10:53:50.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:50 smithi070 ceph-mon[112139]: pgmap v2727: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 1 peering, 1 active+recovery_wait+undersized+remapped, 1 active+recovering+remapped, 4 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 425 KiB data, 5.6 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 69 op/s; 5233/78 objects degraded (6708.974%); 2879/78 objects misplaced (3691.026%); 2.1 MiB/s, 436 objects/s recovering 2024-09-21T10:53:50.471 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:50.493 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:50.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:50.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:50.595 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:50.617 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:50.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:50.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:50.719 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:50.740 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:50.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:50.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:50.842 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:50.863 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:50.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:50.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:50.965 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:50.987 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:50.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:50.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:51.089 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:51.111 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:51.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:51.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:51.162 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:51 smithi070 ceph-mon[107385]: osdmap e1480: 8 total, 8 up, 5 in 2024-09-21T10:53:51.162 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:53:51.162 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:53:51.162 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:51 smithi070 ceph-mon[112139]: osdmap e1480: 8 total, 8 up, 5 in 2024-09-21T10:53:51.163 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:53:51.163 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:53:51.213 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:51.236 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:51.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:51.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:51.289 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-21T10:53:51.289 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:53:51.337 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:51 smithi160 ceph-mon[101879]: osdmap e1480: 8 total, 8 up, 5 in 2024-09-21T10:53:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:53:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:53:51.358 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:51.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:51.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:51.460 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:51.482 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:51.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:51.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:51.508 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:53:51.508 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:53:51.585 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:51.606 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:51.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:51.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:51.708 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:51.730 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:51.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:51.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:51.832 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:51.853 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:51.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:51.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:51.955 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:51.977 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:51.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:51.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:52.078 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:52.100 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:52.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:52.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:52.202 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:52.224 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:52.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:52.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:52.282 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:52 smithi070 ceph-mon[107385]: pgmap v2729: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 1 peering, 1 active+recovery_wait+undersized+remapped, 1 active+recovering+remapped, 4 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 425 KiB data, 5.6 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 6 op/s; 5233/78 objects degraded (6708.974%); 2879/78 objects misplaced (3691.026%); 53 KiB/s, 404 objects/s recovering 2024-09-21T10:53:52.282 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:52 smithi070 ceph-mon[107385]: osdmap e1481: 8 total, 8 up, 5 in 2024-09-21T10:53:52.283 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:52 smithi070 ceph-mon[112139]: pgmap v2729: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 1 peering, 1 active+recovery_wait+undersized+remapped, 1 active+recovering+remapped, 4 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 425 KiB data, 5.6 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 6 op/s; 5233/78 objects degraded (6708.974%); 2879/78 objects misplaced (3691.026%); 53 KiB/s, 404 objects/s recovering 2024-09-21T10:53:52.283 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:52 smithi070 ceph-mon[112139]: osdmap e1481: 8 total, 8 up, 5 in 2024-09-21T10:53:52.303 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:53:52.313 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:53:52.322 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:53:52.325 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:52.337 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:53:52.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:52 smithi160 ceph-mon[101879]: pgmap v2729: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 1 peering, 1 active+recovery_wait+undersized+remapped, 1 active+recovering+remapped, 4 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 425 KiB data, 5.6 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 6 op/s; 5233/78 objects degraded (6708.974%); 2879/78 objects misplaced (3691.026%); 53 KiB/s, 404 objects/s recovering 2024-09-21T10:53:52.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:52 smithi160 ceph-mon[101879]: osdmap e1481: 8 total, 8 up, 5 in 2024-09-21T10:53:52.346 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:53:52.349 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:52.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:52.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:52.451 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:52.472 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:52.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:52.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:52.574 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:52.596 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:52.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:52.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:52.698 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:52.720 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:52.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:52.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:52.822 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:52.846 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:52.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:52.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:52.947 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:52.968 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:52.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:52.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:53.070 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:53.092 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:53.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:53.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:53.194 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:53.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:53.216 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:53.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:53.317 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:53.339 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:53.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:53.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:53.441 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:53.463 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:53.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:53.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:53.466 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:53:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:53:53] "GET /metrics HTTP/1.1" 200 36625 "" "Prometheus/2.51.0" 2024-09-21T10:53:53.564 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:53.586 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:53.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:53.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:53.688 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:53.709 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:53.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:53.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:53.811 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:53.833 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:53.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:53.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:53.934 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:53.955 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:53.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:53.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:54.057 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:54.078 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:54.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:54.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:54.179 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:54.202 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:54.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:54.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:54.303 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:54.325 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:54.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:54.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:54.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:54 smithi160 ceph-mon[101879]: pgmap v2731: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+remapped, 4 active+recovery_wait+remapped, 90 active+clean; 425 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail; 268/78 objects degraded (343.590%); 2879/78 objects misplaced (3691.026%); 0 B/s, 632 objects/s recovering 2024-09-21T10:53:54.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:54 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 268/78 objects degraded (343.590%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:54.356 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:54 smithi070 ceph-mon[107385]: pgmap v2731: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+remapped, 4 active+recovery_wait+remapped, 90 active+clean; 425 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail; 268/78 objects degraded (343.590%); 2879/78 objects misplaced (3691.026%); 0 B/s, 632 objects/s recovering 2024-09-21T10:53:54.357 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:54 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 268/78 objects degraded (343.590%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:54.357 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:54 smithi070 ceph-mon[112139]: pgmap v2731: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+remapped, 4 active+recovery_wait+remapped, 90 active+clean; 425 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail; 268/78 objects degraded (343.590%); 2879/78 objects misplaced (3691.026%); 0 B/s, 632 objects/s recovering 2024-09-21T10:53:54.357 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:54 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 268/78 objects degraded (343.590%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:53:54.426 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:54.448 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:54.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:54.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:54.550 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:54.572 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:54.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:54.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:54.674 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:54.696 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:54.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:54.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:54.797 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:54.818 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:54.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:54.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:54.920 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:54.941 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:54.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:54.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:55.043 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:55.066 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:55.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:55.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:55.168 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:55.190 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:55.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:55.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:55.292 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:55.313 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:55.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:55.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:55.415 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:55.438 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:55.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:55.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:55.540 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:55.564 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:55.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:55.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:55.665 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:55.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:55.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:55.788 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:55.810 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:55.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:55.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:55.912 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:55.934 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:55.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:55.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:56.036 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:56.058 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:56.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:56.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:56.160 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:56.182 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:56.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:56.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:56.284 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:56.306 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:56.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:56.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:56.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:56 smithi160 ceph-mon[101879]: pgmap v2732: 99 pgs: 1 active+recovery_wait+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 4 active+recovery_wait+remapped, 90 active+clean; 425 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail; 268/78 objects degraded (343.590%); 2879/78 objects misplaced (3691.026%); 0 B/s, 425 objects/s recovering 2024-09-21T10:53:56.351 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:56 smithi070 ceph-mon[107385]: pgmap v2732: 99 pgs: 1 active+recovery_wait+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 4 active+recovery_wait+remapped, 90 active+clean; 425 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail; 268/78 objects degraded (343.590%); 2879/78 objects misplaced (3691.026%); 0 B/s, 425 objects/s recovering 2024-09-21T10:53:56.351 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:56 smithi070 ceph-mon[112139]: pgmap v2732: 99 pgs: 1 active+recovery_wait+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 4 active+recovery_wait+remapped, 90 active+clean; 425 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail; 268/78 objects degraded (343.590%); 2879/78 objects misplaced (3691.026%); 0 B/s, 425 objects/s recovering 2024-09-21T10:53:56.408 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:56.432 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:56.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:56.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:56.533 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:56.555 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:56.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:56.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:56.584 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:53:56.656 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:56.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:56.680 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:56.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:56.781 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:56.802 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:56.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:56.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:56.904 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:56.926 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:56.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:56.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:57.028 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:57.049 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:57.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:57.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:57.152 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:57.173 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:57.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:57.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:57.275 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:57.297 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:57.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:57.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:57.399 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:57.421 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:57.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:57.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:57.523 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:53:57.544 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:57.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:57.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:53:57.646 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:57.667 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:57.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:57.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:57.769 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:57.791 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:57.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:57.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:57.893 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:57.919 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:57.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:57.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:57.948 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:53:57.948 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:53:57.949 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:53:57.949 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:53:57.964 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:53:57.965 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:53:57.982 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:53:57.982 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:53:57.999 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:53:57.999 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:53:58.015 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:53:58.016 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:53:58.021 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:58.032 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:53:58.032 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:53:58.042 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:58.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:58.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:58.049 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:53:58.049 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:53:58.090 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T10:53:58.144 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:58.165 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:58.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:58.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:58.267 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:58.290 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:58.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:58.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:58.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:58 smithi160 ceph-mon[101879]: pgmap v2733: 99 pgs: 1 active+recovery_wait+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 4 active+recovery_wait+remapped, 90 active+clean; 425 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail; 268/78 objects degraded (343.590%); 2879/78 objects misplaced (3691.026%); 0 B/s, 535 objects/s recovering 2024-09-21T10:53:58.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:58 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2177890889' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:53:58.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:58 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/416449653' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:53:58.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:58 smithi070 ceph-mon[107385]: pgmap v2733: 99 pgs: 1 active+recovery_wait+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 4 active+recovery_wait+remapped, 90 active+clean; 425 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail; 268/78 objects degraded (343.590%); 2879/78 objects misplaced (3691.026%); 0 B/s, 535 objects/s recovering 2024-09-21T10:53:58.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:58 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2177890889' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:53:58.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:58 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/416449653' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:53:58.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:58 smithi070 ceph-mon[112139]: pgmap v2733: 99 pgs: 1 active+recovery_wait+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 4 active+recovery_wait+remapped, 90 active+clean; 425 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail; 268/78 objects degraded (343.590%); 2879/78 objects misplaced (3691.026%); 0 B/s, 535 objects/s recovering 2024-09-21T10:53:58.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:58 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2177890889' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:53:58.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:58 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/416449653' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:53:58.391 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:58.412 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:58.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:58.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:58.515 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:58.537 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:58.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:58.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:58.638 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:53:58.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:58.662 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:58.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:53:58.764 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:58.786 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:58.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:58.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:58.887 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:53:58.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:58.912 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:58.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:53:59.013 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:59.036 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:59.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:59.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:59.138 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:53:59.162 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:59.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:59.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:53:59.264 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:53:59.287 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:59.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:59.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:53:59.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:59 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:53:59.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:59 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2177890889' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:53:59.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:53:59 smithi160 ceph-mon[101879]: osdmap e1482: 8 total, 8 up, 5 in 2024-09-21T10:53:59.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:59 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:53:59.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:59 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2177890889' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:53:59.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:53:59 smithi070 ceph-mon[107385]: osdmap e1482: 8 total, 8 up, 5 in 2024-09-21T10:53:59.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:59 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:53:59.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:59 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2177890889' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:53:59.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:53:59 smithi070 ceph-mon[112139]: osdmap e1482: 8 total, 8 up, 5 in 2024-09-21T10:53:59.388 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:53:59.410 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:53:59.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:59.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:53:59.512 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:59.536 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:59.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:59.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:59.637 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:59.660 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:59.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:59.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:53:59.762 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:53:59.786 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:59.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:59.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:53:59.888 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:53:59.912 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:53:59.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:53:59.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:00.014 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:00.036 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:00.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:00.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:00.138 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:00.161 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:00.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:00.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:00.249 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:54:00.250 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T10:54:00.263 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:00.284 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:00.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:00.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:00.343 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:00 smithi070 ceph-mon[107385]: pgmap v2735: 99 pgs: 1 active+recovery_wait+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 4 active+recovery_wait+remapped, 90 active+clean; 425 KiB data, 5.4 GiB used, 442 GiB / 447 GiB avail; 268/78 objects degraded (343.590%); 2879/78 objects misplaced (3691.026%); 0 B/s, 535 objects/s recovering 2024-09-21T10:54:00.344 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:00 smithi070 ceph-mon[112139]: pgmap v2735: 99 pgs: 1 active+recovery_wait+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 4 active+recovery_wait+remapped, 90 active+clean; 425 KiB data, 5.4 GiB used, 442 GiB / 447 GiB avail; 268/78 objects degraded (343.590%); 2879/78 objects misplaced (3691.026%); 0 B/s, 535 objects/s recovering 2024-09-21T10:54:00.386 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:00.408 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:00.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:00.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:00.510 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:00.532 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:00.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:00.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:00.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:00 smithi160 ceph-mon[101879]: pgmap v2735: 99 pgs: 1 active+recovery_wait+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 4 active+recovery_wait+remapped, 90 active+clean; 425 KiB data, 5.4 GiB used, 442 GiB / 447 GiB avail; 268/78 objects degraded (343.590%); 2879/78 objects misplaced (3691.026%); 0 B/s, 535 objects/s recovering 2024-09-21T10:54:00.633 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:00.655 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:00.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:00.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:00.757 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:00.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:00.778 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:00.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:00.879 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:00.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:00.902 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:00.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:01.006 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:01.028 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:01.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:01.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:01.099 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'unique_pool_2' removed 2024-09-21T10:54:01.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:01 smithi070 ceph-mon[107385]: from='client.45911 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:54:01.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:01 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/432904939' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:54:01.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:01 smithi070 ceph-mon[112139]: from='client.45911 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:54:01.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:01 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/432904939' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:54:01.130 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:01.168 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:01.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:01.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:01.207 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":2735,"stamp":"2024-09-21T10:53:58.947161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":434932,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":268,"num_objects_misplaced":2879,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":277,"num_read_kb":599,"num_write":210899,"num_write_kb":6743423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55490,"num_bytes_recovered":2989028018,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75835,"ondisk_log_size":75835,"up":297,"acting":295,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":327,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":5665096,"kb_used_data":2214988,"kb_used_omap":141,"kb_used_meta":3449906,"kb_avail":463060664,"statfs":{"total":479975178240,"available":474174119936,"internally_reserved":0,"allocated":2268147712,"data_stored":2242874041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":144912,"internal_metadata":3532704240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4284,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001914"},"pg_stats":[{"pgid":"55.f","version":"1475'13132","reported_seq":20420,"reported_epoch":1480,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-21T10:53:50.080115+0000","last_change":"2024-09-21T10:53:41.650201+0000","last_active":"2024-09-21T10:53:50.080115+0000","last_peered":"2024-09-21T10:53:50.080115+0000","last_clean":"2024-09-21T10:49:43.822243+0000","last_became_active":"2024-09-21T10:51:28.555652+0000","last_became_peered":"2024-09-21T10:51:28.555652+0000","last_unstale":"2024-09-21T10:53:50.080115+0000","last_undegraded":"2024-09-21T10:53:50.080115+0000","last_fullsized":"2024-09-21T10:53:50.080115+0000","mapping_epoch":1448,"log_start":"1454'9040","ondisk_log_start":"1454'9040","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4092,"log_dups_size":0,"ondisk_log_size":4092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:30:34.790715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13132,"num_write_kb":420224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3592,"num_bytes_recovered":235405312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,0],"avail_no_missing":["6","1","5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.e","version":"1475'13012","reported_seq":13174,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.082145+0000","last_change":"2024-09-21T10:52:16.415783+0000","last_active":"2024-09-21T10:53:51.082145+0000","last_peered":"2024-09-21T10:53:51.082145+0000","last_clean":"2024-09-21T10:53:51.082145+0000","last_became_active":"2024-09-21T10:52:16.397921+0000","last_became_peered":"2024-09-21T10:52:16.397921+0000","last_unstale":"2024-09-21T10:53:51.082145+0000","last_undegraded":"2024-09-21T10:53:51.082145+0000","last_fullsized":"2024-09-21T10:53:51.082145+0000","mapping_epoch":1463,"log_start":"1452'8256","ondisk_log_start":"1452'8256","created":1411,"last_epoch_clean":1464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4756,"log_dups_size":0,"ondisk_log_size":4756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:13.753306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":416384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.d","version":"1475'13378","reported_seq":26366,"reported_epoch":1481,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T10:53:56.360608+0000","last_change":"2024-09-21T10:53:49.827999+0000","last_active":"2024-09-21T10:53:56.360608+0000","last_peered":"2024-09-21T10:53:56.360608+0000","last_clean":"2024-09-21T10:50:20.259057+0000","last_became_active":"2024-09-21T10:52:26.653674+0000","last_became_peered":"2024-09-21T10:52:26.653674+0000","last_unstale":"2024-09-21T10:53:56.360608+0000","last_undegraded":"2024-09-21T10:52:26.638016+0000","last_fullsized":"2024-09-21T10:52:26.462648+0000","mapping_epoch":1466,"log_start":"1446'8326","ondisk_log_start":"1446'8326","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":5052,"log_dups_size":0,"ondisk_log_size":5052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:31:18.206348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13378,"num_write_kb":428033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5876,"num_bytes_recovered":310509568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[4,3],"avail_no_missing":["4","0","6"],"object_location_counts":[{"shards":"3,4","objects":-268}],"blocked_by":[],"up_primary":7,"acting_primary":4,"purged_snaps":[]},{"pgid":"55.c","version":"1475'13132","reported_seq":24666,"reported_epoch":1481,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:53:51.081424+0000","last_change":"2024-09-21T10:53:45.366645+0000","last_active":"2024-09-21T10:53:51.081424+0000","last_peered":"2024-09-21T10:53:51.081424+0000","last_clean":"2024-09-21T10:50:03.007234+0000","last_became_active":"2024-09-21T10:52:25.621852+0000","last_became_peered":"2024-09-21T10:52:25.621852+0000","last_unstale":"2024-09-21T10:53:51.081424+0000","last_undegraded":"2024-09-21T10:53:51.081424+0000","last_fullsized":"2024-09-21T10:53:51.081424+0000","mapping_epoch":1465,"log_start":"1452'8386","ondisk_log_start":"1452'8386","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4746,"log_dups_size":0,"ondisk_log_size":4746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13132,"num_write_kb":420224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5683,"num_bytes_recovered":268435456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,0],"avail_no_missing":["3","0","1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.b","version":"1475'13420","reported_seq":13501,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.078619+0000","last_change":"2024-09-21T10:48:52.934001+0000","last_active":"2024-09-21T10:53:50.078619+0000","last_peered":"2024-09-21T10:53:50.078619+0000","last_clean":"2024-09-21T10:53:50.078619+0000","last_became_active":"2024-09-21T10:48:52.933674+0000","last_became_peered":"2024-09-21T10:48:52.933674+0000","last_unstale":"2024-09-21T10:53:50.078619+0000","last_undegraded":"2024-09-21T10:53:50.078619+0000","last_fullsized":"2024-09-21T10:53:50.078619+0000","mapping_epoch":1411,"log_start":"1454'9306","ondisk_log_start":"1454'9306","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4114,"log_dups_size":0,"ondisk_log_size":4114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:13.178680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13420,"num_write_kb":429440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.a","version":"1475'13134","reported_seq":18304,"reported_epoch":1481,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-21T10:53:51.081904+0000","last_change":"2024-09-21T10:53:42.050786+0000","last_active":"2024-09-21T10:53:51.081904+0000","last_peered":"2024-09-21T10:53:51.081904+0000","last_clean":"2024-09-21T10:50:03.021536+0000","last_became_active":"2024-09-21T10:52:26.699954+0000","last_became_peered":"2024-09-21T10:52:26.699954+0000","last_unstale":"2024-09-21T10:53:51.081904+0000","last_undegraded":"2024-09-21T10:53:51.081904+0000","last_fullsized":"2024-09-21T10:53:51.081904+0000","mapping_epoch":1466,"log_start":"1452'8351","ondisk_log_start":"1452'8351","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4783,"log_dups_size":0,"ondisk_log_size":4783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13134,"num_write_kb":420288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2503,"num_bytes_recovered":164036608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,4,6],"avail_no_missing":["3","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.9","version":"1475'13174","reported_seq":22748,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.081789+0000","last_change":"2024-09-21T10:52:44.354764+0000","last_active":"2024-09-21T10:53:51.081789+0000","last_peered":"2024-09-21T10:53:51.081789+0000","last_clean":"2024-09-21T10:53:51.081789+0000","last_became_active":"2024-09-21T10:52:44.325813+0000","last_became_peered":"2024-09-21T10:52:44.325813+0000","last_unstale":"2024-09-21T10:53:51.081789+0000","last_undegraded":"2024-09-21T10:53:51.081789+0000","last_fullsized":"2024-09-21T10:53:51.081789+0000","mapping_epoch":1471,"log_start":"1452'8469","ondisk_log_start":"1452'8469","created":1411,"last_epoch_clean":1472,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4705,"log_dups_size":0,"ondisk_log_size":4705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:58.318369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13174,"num_write_kb":421568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4701,"num_bytes_recovered":297533440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.8","version":"1475'13034","reported_seq":22444,"reported_epoch":1480,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:53:50.077182+0000","last_change":"2024-09-21T10:53:41.154589+0000","last_active":"2024-09-21T10:53:50.077182+0000","last_peered":"2024-09-21T10:53:50.077182+0000","last_clean":"2024-09-21T10:49:43.813262+0000","last_became_active":"2024-09-21T10:51:28.546990+0000","last_became_peered":"2024-09-21T10:51:28.546990+0000","last_unstale":"2024-09-21T10:53:50.077182+0000","last_undegraded":"2024-09-21T10:53:50.077182+0000","last_fullsized":"2024-09-21T10:53:50.077182+0000","mapping_epoch":1448,"log_start":"1454'8934","ondisk_log_start":"1454'8934","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4100,"log_dups_size":0,"ondisk_log_size":4100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:51:45.867981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13034,"num_write_kb":417088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4638,"num_bytes_recovered":303955968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,0],"avail_no_missing":["6","0","1","3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.7","version":"1475'13288","reported_seq":13369,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.079133+0000","last_change":"2024-09-21T10:48:52.933230+0000","last_active":"2024-09-21T10:53:50.079133+0000","last_peered":"2024-09-21T10:53:50.079133+0000","last_clean":"2024-09-21T10:53:50.079133+0000","last_became_active":"2024-09-21T10:48:52.932944+0000","last_became_peered":"2024-09-21T10:48:52.932944+0000","last_unstale":"2024-09-21T10:53:50.079133+0000","last_undegraded":"2024-09-21T10:53:50.079133+0000","last_fullsized":"2024-09-21T10:53:50.079133+0000","mapping_epoch":1411,"log_start":"1450'8228","ondisk_log_start":"1450'8228","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":5060,"log_dups_size":0,"ondisk_log_size":5060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:16:43.711173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13288,"num_write_kb":425216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"55.6","version":"1475'13324","reported_seq":31260,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.079590+0000","last_change":"2024-09-21T10:53:43.092431+0000","last_active":"2024-09-21T10:53:50.079590+0000","last_peered":"2024-09-21T10:53:50.079590+0000","last_clean":"2024-09-21T10:53:50.079590+0000","last_became_active":"2024-09-21T10:53:43.057005+0000","last_became_peered":"2024-09-21T10:53:43.057005+0000","last_unstale":"2024-09-21T10:53:50.079590+0000","last_undegraded":"2024-09-21T10:53:50.079590+0000","last_fullsized":"2024-09-21T10:53:50.079590+0000","mapping_epoch":1476,"log_start":"1446'8226","ondisk_log_start":"1446'8226","created":1411,"last_epoch_clean":1477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":5098,"log_dups_size":0,"ondisk_log_size":5098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:07:11.785307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13324,"num_write_kb":426368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7787,"num_bytes_recovered":334823424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"55.5","version":"1475'13138","reported_seq":13268,"reported_epoch":1480,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-21T10:53:50.076776+0000","last_change":"2024-09-21T10:52:25.760897+0000","last_active":"2024-09-21T10:53:50.076776+0000","last_peered":"2024-09-21T10:53:50.076776+0000","last_clean":"2024-09-21T10:50:02.995767+0000","last_became_active":"2024-09-21T10:52:25.564312+0000","last_became_peered":"2024-09-21T10:52:25.564312+0000","last_unstale":"2024-09-21T10:53:50.076776+0000","last_undegraded":"2024-09-21T10:53:50.076776+0000","last_fullsized":"2024-09-21T10:53:50.076776+0000","mapping_epoch":1465,"log_start":"1454'9014","ondisk_log_start":"1454'9014","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4124,"log_dups_size":0,"ondisk_log_size":4124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:26:53.597959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13138,"num_write_kb":420416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,0],"avail_no_missing":["6","0","1","3","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.4","version":"1475'13320","reported_seq":13401,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.078395+0000","last_change":"2024-09-21T10:48:52.934634+0000","last_active":"2024-09-21T10:53:50.078395+0000","last_peered":"2024-09-21T10:53:50.078395+0000","last_clean":"2024-09-21T10:53:50.078395+0000","last_became_active":"2024-09-21T10:48:52.934368+0000","last_became_peered":"2024-09-21T10:48:52.934368+0000","last_unstale":"2024-09-21T10:53:50.078395+0000","last_undegraded":"2024-09-21T10:53:50.078395+0000","last_fullsized":"2024-09-21T10:53:50.078395+0000","mapping_epoch":1411,"log_start":"1449'8245","ondisk_log_start":"1449'8245","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":5075,"log_dups_size":0,"ondisk_log_size":5075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:27:09.216667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13320,"num_write_kb":426240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"55.3","version":"1475'12742","reported_seq":28591,"reported_epoch":1482,"state":"active+recovery_wait+undersized+remapped","last_fresh":"2024-09-21T10:53:58.093697+0000","last_change":"2024-09-21T10:53:27.173985+0000","last_active":"2024-09-21T10:53:58.093697+0000","last_peered":"2024-09-21T10:53:58.093697+0000","last_clean":"2024-09-21T10:49:43.839828+0000","last_became_active":"2024-09-21T10:52:26.596220+0000","last_became_peered":"2024-09-21T10:52:26.596220+0000","last_unstale":"2024-09-21T10:53:58.093697+0000","last_undegraded":"2024-09-21T10:53:58.093697+0000","last_fullsized":"2024-09-21T10:52:26.462998+0000","mapping_epoch":1466,"log_start":"1454'8443","ondisk_log_start":"1454'8443","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4299,"log_dups_size":0,"ondisk_log_size":4299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12742,"num_write_kb":407744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5544,"num_bytes_recovered":363200512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,0],"avail_no_missing":["1","0","3","5","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"55.2","version":"1475'12902","reported_seq":28863,"reported_epoch":1482,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-21T10:53:58.094265+0000","last_change":"2024-09-21T10:53:41.628094+0000","last_active":"2024-09-21T10:53:58.094265+0000","last_peered":"2024-09-21T10:53:58.094265+0000","last_clean":"2024-09-21T10:50:02.974304+0000","last_became_active":"2024-09-21T10:52:26.694338+0000","last_became_peered":"2024-09-21T10:52:26.694338+0000","last_unstale":"2024-09-21T10:53:58.094265+0000","last_undegraded":"2024-09-21T10:53:58.094265+0000","last_fullsized":"2024-09-21T10:53:58.094265+0000","mapping_epoch":1466,"log_start":"1454'8606","ondisk_log_start":"1454'8606","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4296,"log_dups_size":0,"ondisk_log_size":4296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2879,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12902,"num_write_kb":412864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6454,"num_bytes_recovered":422838272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,4],"avail_no_missing":["1","3","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"55.1","version":"1475'13298","reported_seq":19173,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.114223+0000","last_change":"2024-09-21T10:53:51.114223+0000","last_active":"2024-09-21T10:53:51.114223+0000","last_peered":"2024-09-21T10:53:51.114223+0000","last_clean":"2024-09-21T10:53:51.114223+0000","last_became_active":"2024-09-21T10:53:51.084474+0000","last_became_peered":"2024-09-21T10:53:51.084474+0000","last_unstale":"2024-09-21T10:53:51.114223+0000","last_undegraded":"2024-09-21T10:53:51.114223+0000","last_fullsized":"2024-09-21T10:53:51.114223+0000","mapping_epoch":1480,"log_start":"1446'8212","ondisk_log_start":"1446'8212","created":1411,"last_epoch_clean":1481,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":5086,"log_dups_size":0,"ondisk_log_size":5086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:16:58.199698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13298,"num_write_kb":425536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2879,"num_bytes_recovered":20643840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"55.0","version":"1475'13250","reported_seq":24949,"reported_epoch":1481,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:53:51.081587+0000","last_change":"2024-09-21T10:53:49.518270+0000","last_active":"2024-09-21T10:53:51.081587+0000","last_peered":"2024-09-21T10:53:51.081587+0000","last_clean":"2024-09-21T10:50:03.041207+0000","last_became_active":"2024-09-21T10:52:26.645887+0000","last_became_peered":"2024-09-21T10:52:26.645887+0000","last_unstale":"2024-09-21T10:53:51.081587+0000","last_undegraded":"2024-09-21T10:53:51.081587+0000","last_fullsized":"2024-09-21T10:53:51.081587+0000","mapping_epoch":1466,"log_start":"1452'8499","ondisk_log_start":"1452'8499","created":1411,"last_epoch_clean":1412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:48:51.913530+0000","last_clean_scrub_stamp":"2024-09-21T10:48:51.913530+0000","objects_scrubbed":0,"log_size":4751,"log_dups_size":0,"ondisk_log_size":4751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":424000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5766,"num_bytes_recovered":263651328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,1],"avail_no_missing":["3","1","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":1972,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.080303+0000","last_change":"2024-09-21T10:52:03.323899+0000","last_active":"2024-09-21T10:53:50.080303+0000","last_peered":"2024-09-21T10:53:50.080303+0000","last_clean":"2024-09-21T10:53:50.080303+0000","last_became_active":"2024-09-21T10:52:03.314626+0000","last_became_peered":"2024-09-21T10:52:03.314626+0000","last_unstale":"2024-09-21T10:53:50.080303+0000","last_undegraded":"2024-09-21T10:53:50.080303+0000","last_fullsized":"2024-09-21T10:53:50.080303+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:13.939806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1471,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.077572+0000","last_change":"2024-09-21T10:52:15.396755+0000","last_active":"2024-09-21T10:53:50.077572+0000","last_peered":"2024-09-21T10:53:50.077572+0000","last_clean":"2024-09-21T10:53:50.077572+0000","last_became_active":"2024-09-21T10:52:15.396417+0000","last_became_peered":"2024-09-21T10:52:15.396417+0000","last_unstale":"2024-09-21T10:53:50.077572+0000","last_undegraded":"2024-09-21T10:53:50.077572+0000","last_fullsized":"2024-09-21T10:53:50.077572+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:25.005574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":2272,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.082287+0000","last_change":"2024-09-21T10:52:15.400227+0000","last_active":"2024-09-21T10:53:51.082287+0000","last_peered":"2024-09-21T10:53:51.082287+0000","last_clean":"2024-09-21T10:53:51.082287+0000","last_became_active":"2024-09-21T10:52:15.399888+0000","last_became_peered":"2024-09-21T10:52:15.399888+0000","last_unstale":"2024-09-21T10:53:51.082287+0000","last_undegraded":"2024-09-21T10:53:51.082287+0000","last_fullsized":"2024-09-21T10:53:51.082287+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:42:52.565184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2288,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.081996+0000","last_change":"2024-09-21T10:51:27.458216+0000","last_active":"2024-09-21T10:53:51.081996+0000","last_peered":"2024-09-21T10:53:51.081996+0000","last_clean":"2024-09-21T10:53:51.081996+0000","last_became_active":"2024-09-21T10:51:27.457948+0000","last_became_peered":"2024-09-21T10:51:27.457948+0000","last_unstale":"2024-09-21T10:53:51.081996+0000","last_undegraded":"2024-09-21T10:53:51.081996+0000","last_fullsized":"2024-09-21T10:53:51.081996+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:25:29.584079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":110,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.081610+0000","last_change":"2024-09-21T10:49:44.878242+0000","last_active":"2024-09-21T10:53:51.081610+0000","last_peered":"2024-09-21T10:53:51.081610+0000","last_clean":"2024-09-21T10:53:51.081610+0000","last_became_active":"2024-09-21T10:49:44.866578+0000","last_became_peered":"2024-09-21T10:49:44.866578+0000","last_unstale":"2024-09-21T10:53:51.081610+0000","last_undegraded":"2024-09-21T10:53:51.081610+0000","last_fullsized":"2024-09-21T10:53:51.081610+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:17:24.145551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1476,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.079277+0000","last_change":"2024-09-21T10:51:27.460539+0000","last_active":"2024-09-21T10:53:50.079277+0000","last_peered":"2024-09-21T10:53:50.079277+0000","last_clean":"2024-09-21T10:53:50.079277+0000","last_became_active":"2024-09-21T10:51:27.460211+0000","last_became_peered":"2024-09-21T10:51:27.460211+0000","last_unstale":"2024-09-21T10:53:50.079277+0000","last_undegraded":"2024-09-21T10:53:50.079277+0000","last_fullsized":"2024-09-21T10:53:50.079277+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:14:15.700998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":149,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.079912+0000","last_change":"2024-09-21T10:52:15.405886+0000","last_active":"2024-09-21T10:53:51.079912+0000","last_peered":"2024-09-21T10:53:51.079912+0000","last_clean":"2024-09-21T10:53:51.079912+0000","last_became_active":"2024-09-21T10:52:15.397317+0000","last_became_peered":"2024-09-21T10:52:15.397317+0000","last_unstale":"2024-09-21T10:53:51.079912+0000","last_undegraded":"2024-09-21T10:53:51.079912+0000","last_fullsized":"2024-09-21T10:53:51.079912+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:43:48.813861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1467,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.078915+0000","last_change":"2024-09-21T10:09:06.560175+0000","last_active":"2024-09-21T10:53:50.078915+0000","last_peered":"2024-09-21T10:53:50.078915+0000","last_clean":"2024-09-21T10:53:50.078915+0000","last_became_active":"2024-09-21T10:09:06.558584+0000","last_became_peered":"2024-09-21T10:09:06.558584+0000","last_unstale":"2024-09-21T10:53:50.078915+0000","last_undegraded":"2024-09-21T10:53:50.078915+0000","last_fullsized":"2024-09-21T10:53:50.078915+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:02.918339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":57,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.078734+0000","last_change":"2024-09-21T10:51:27.437665+0000","last_active":"2024-09-21T10:53:50.078734+0000","last_peered":"2024-09-21T10:53:50.078734+0000","last_clean":"2024-09-21T10:53:50.078734+0000","last_became_active":"2024-09-21T10:51:27.436481+0000","last_became_peered":"2024-09-21T10:51:27.436481+0000","last_unstale":"2024-09-21T10:53:50.078734+0000","last_undegraded":"2024-09-21T10:53:50.078734+0000","last_fullsized":"2024-09-21T10:53:50.078734+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:07:50.267297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1045,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.095151+0000","last_change":"2024-09-21T10:51:27.459033+0000","last_active":"2024-09-21T10:53:58.095151+0000","last_peered":"2024-09-21T10:53:58.095151+0000","last_clean":"2024-09-21T10:53:58.095151+0000","last_became_active":"2024-09-21T10:51:27.458728+0000","last_became_peered":"2024-09-21T10:51:27.458728+0000","last_unstale":"2024-09-21T10:53:58.095151+0000","last_undegraded":"2024-09-21T10:53:58.095151+0000","last_fullsized":"2024-09-21T10:53:58.095151+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:20.133032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":1496,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.079432+0000","last_change":"2024-09-21T10:52:25.473550+0000","last_active":"2024-09-21T10:53:50.079432+0000","last_peered":"2024-09-21T10:53:50.079432+0000","last_clean":"2024-09-21T10:53:50.079432+0000","last_became_active":"2024-09-21T10:52:25.473105+0000","last_became_peered":"2024-09-21T10:52:25.473105+0000","last_unstale":"2024-09-21T10:53:50.079432+0000","last_undegraded":"2024-09-21T10:53:50.079432+0000","last_fullsized":"2024-09-21T10:53:50.079432+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:15:45.352518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2267,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.084294+0000","last_change":"2024-09-21T10:40:13.432531+0000","last_active":"2024-09-21T10:53:51.084294+0000","last_peered":"2024-09-21T10:53:51.084294+0000","last_clean":"2024-09-21T10:53:51.084294+0000","last_became_active":"2024-09-21T10:40:13.432210+0000","last_became_peered":"2024-09-21T10:40:13.432210+0000","last_unstale":"2024-09-21T10:53:51.084294+0000","last_undegraded":"2024-09-21T10:53:51.084294+0000","last_fullsized":"2024-09-21T10:53:51.084294+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:20:10.998607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2241,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.094140+0000","last_change":"2024-09-21T10:09:06.530394+0000","last_active":"2024-09-21T10:53:58.094140+0000","last_peered":"2024-09-21T10:53:58.094140+0000","last_clean":"2024-09-21T10:53:58.094140+0000","last_became_active":"2024-09-21T10:09:06.529307+0000","last_became_peered":"2024-09-21T10:09:06.529307+0000","last_unstale":"2024-09-21T10:53:58.094140+0000","last_undegraded":"2024-09-21T10:53:58.094140+0000","last_fullsized":"2024-09-21T10:53:58.094140+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:54.287397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.079506+0000","last_change":"2024-09-21T10:52:25.489896+0000","last_active":"2024-09-21T10:53:50.079506+0000","last_peered":"2024-09-21T10:53:50.079506+0000","last_clean":"2024-09-21T10:53:50.079506+0000","last_became_active":"2024-09-21T10:52:25.489154+0000","last_became_peered":"2024-09-21T10:52:25.489154+0000","last_unstale":"2024-09-21T10:53:50.079506+0000","last_undegraded":"2024-09-21T10:53:50.079506+0000","last_fullsized":"2024-09-21T10:53:50.079506+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:36:02.076931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2268,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.081768+0000","last_change":"2024-09-21T10:52:25.493309+0000","last_active":"2024-09-21T10:53:51.081768+0000","last_peered":"2024-09-21T10:53:51.081768+0000","last_clean":"2024-09-21T10:53:51.081768+0000","last_became_active":"2024-09-21T10:52:25.492568+0000","last_became_peered":"2024-09-21T10:52:25.492568+0000","last_unstale":"2024-09-21T10:53:51.081768+0000","last_undegraded":"2024-09-21T10:53:51.081768+0000","last_fullsized":"2024-09-21T10:53:51.081768+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:19:19.393370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1484,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.080389+0000","last_change":"2024-09-21T10:52:03.317474+0000","last_active":"2024-09-21T10:53:50.080389+0000","last_peered":"2024-09-21T10:53:50.080389+0000","last_clean":"2024-09-21T10:53:50.080389+0000","last_became_active":"2024-09-21T10:52:03.317001+0000","last_became_peered":"2024-09-21T10:52:03.317001+0000","last_unstale":"2024-09-21T10:53:50.080389+0000","last_undegraded":"2024-09-21T10:53:50.080389+0000","last_fullsized":"2024-09-21T10:53:50.080389+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:06:07.039763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2268,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.082527+0000","last_change":"2024-09-21T10:52:15.412246+0000","last_active":"2024-09-21T10:53:51.082527+0000","last_peered":"2024-09-21T10:53:51.082527+0000","last_clean":"2024-09-21T10:53:51.082527+0000","last_became_active":"2024-09-21T10:52:15.406204+0000","last_became_peered":"2024-09-21T10:52:15.406204+0000","last_unstale":"2024-09-21T10:53:51.082527+0000","last_undegraded":"2024-09-21T10:53:51.082527+0000","last_fullsized":"2024-09-21T10:53:51.082527+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:03.524324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2248,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.082794+0000","last_change":"2024-09-21T10:52:15.402822+0000","last_active":"2024-09-21T10:53:51.082794+0000","last_peered":"2024-09-21T10:53:51.082794+0000","last_clean":"2024-09-21T10:53:51.082794+0000","last_became_active":"2024-09-21T10:52:15.402548+0000","last_became_peered":"2024-09-21T10:52:15.402548+0000","last_unstale":"2024-09-21T10:53:51.082794+0000","last_undegraded":"2024-09-21T10:53:51.082794+0000","last_fullsized":"2024-09-21T10:53:51.082794+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:04:04.667598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2273,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.082079+0000","last_change":"2024-09-21T10:52:25.499172+0000","last_active":"2024-09-21T10:53:51.082079+0000","last_peered":"2024-09-21T10:53:51.082079+0000","last_clean":"2024-09-21T10:53:51.082079+0000","last_became_active":"2024-09-21T10:52:25.498835+0000","last_became_peered":"2024-09-21T10:52:25.498835+0000","last_unstale":"2024-09-21T10:53:51.082079+0000","last_undegraded":"2024-09-21T10:53:51.082079+0000","last_fullsized":"2024-09-21T10:53:51.082079+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:10:12.434470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":26,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.079431+0000","last_change":"2024-09-21T10:52:25.489774+0000","last_active":"2024-09-21T10:53:50.079431+0000","last_peered":"2024-09-21T10:53:50.079431+0000","last_clean":"2024-09-21T10:53:50.079431+0000","last_became_active":"2024-09-21T10:52:25.487788+0000","last_became_peered":"2024-09-21T10:52:25.487788+0000","last_unstale":"2024-09-21T10:53:50.079431+0000","last_undegraded":"2024-09-21T10:53:50.079431+0000","last_fullsized":"2024-09-21T10:53:50.079431+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:48.842419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":32,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.079346+0000","last_change":"2024-09-21T10:52:25.565465+0000","last_active":"2024-09-21T10:53:50.079346+0000","last_peered":"2024-09-21T10:53:50.079346+0000","last_clean":"2024-09-21T10:53:50.079346+0000","last_became_active":"2024-09-21T10:52:25.510131+0000","last_became_peered":"2024-09-21T10:52:25.510131+0000","last_unstale":"2024-09-21T10:53:50.079346+0000","last_undegraded":"2024-09-21T10:53:50.079346+0000","last_fullsized":"2024-09-21T10:53:50.079346+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:41:13.870503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":450,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2276,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.094662+0000","last_change":"2024-09-21T10:52:25.504787+0000","last_active":"2024-09-21T10:53:58.094662+0000","last_peered":"2024-09-21T10:53:58.094662+0000","last_clean":"2024-09-21T10:53:58.094662+0000","last_became_active":"2024-09-21T10:52:25.504198+0000","last_became_peered":"2024-09-21T10:52:25.504198+0000","last_unstale":"2024-09-21T10:53:58.094662+0000","last_undegraded":"2024-09-21T10:53:58.094662+0000","last_fullsized":"2024-09-21T10:53:58.094662+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:38:25.666224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":1501,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.080209+0000","last_change":"2024-09-21T10:44:14.415264+0000","last_active":"2024-09-21T10:53:50.080209+0000","last_peered":"2024-09-21T10:53:50.080209+0000","last_clean":"2024-09-21T10:53:50.080209+0000","last_became_active":"2024-09-21T10:44:14.407813+0000","last_became_peered":"2024-09-21T10:44:14.407813+0000","last_unstale":"2024-09-21T10:53:50.080209+0000","last_undegraded":"2024-09-21T10:53:50.080209+0000","last_fullsized":"2024-09-21T10:53:50.080209+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:55.933601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1454,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.078988+0000","last_change":"2024-09-21T10:07:56.001756+0000","last_active":"2024-09-21T10:53:50.078988+0000","last_peered":"2024-09-21T10:53:50.078988+0000","last_clean":"2024-09-21T10:53:50.078988+0000","last_became_active":"2024-09-21T10:07:56.000789+0000","last_became_peered":"2024-09-21T10:07:56.000789+0000","last_unstale":"2024-09-21T10:53:50.078988+0000","last_undegraded":"2024-09-21T10:53:50.078988+0000","last_fullsized":"2024-09-21T10:53:50.078988+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:56:04.302902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"346'149","reported_seq":936,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.082009+0000","last_change":"2024-09-21T10:53:47.069462+0000","last_active":"2024-09-21T10:53:51.082009+0000","last_peered":"2024-09-21T10:53:51.082009+0000","last_clean":"2024-09-21T10:53:51.082009+0000","last_became_active":"2024-09-21T10:53:47.069113+0000","last_became_peered":"2024-09-21T10:53:47.069113+0000","last_unstale":"2024-09-21T10:53:51.082009+0000","last_undegraded":"2024-09-21T10:53:51.082009+0000","last_fullsized":"2024-09-21T10:53:51.082009+0000","mapping_epoch":1478,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1479,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1579069,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1469,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.078633+0000","last_change":"2024-09-21T10:52:25.473127+0000","last_active":"2024-09-21T10:53:50.078633+0000","last_peered":"2024-09-21T10:53:50.078633+0000","last_clean":"2024-09-21T10:53:50.078633+0000","last_became_active":"2024-09-21T10:52:25.472805+0000","last_became_peered":"2024-09-21T10:52:25.472805+0000","last_unstale":"2024-09-21T10:53:50.078633+0000","last_undegraded":"2024-09-21T10:53:50.078633+0000","last_fullsized":"2024-09-21T10:53:50.078633+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:13.038797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1514,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.080422+0000","last_change":"2024-09-21T10:52:25.493263+0000","last_active":"2024-09-21T10:53:50.080422+0000","last_peered":"2024-09-21T10:53:50.080422+0000","last_clean":"2024-09-21T10:53:50.080422+0000","last_became_active":"2024-09-21T10:52:25.492970+0000","last_became_peered":"2024-09-21T10:52:25.492970+0000","last_unstale":"2024-09-21T10:53:50.080422+0000","last_undegraded":"2024-09-21T10:53:50.080422+0000","last_fullsized":"2024-09-21T10:53:50.080422+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:43:40.305783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":121,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.094965+0000","last_change":"2024-09-21T10:52:25.513834+0000","last_active":"2024-09-21T10:53:58.094965+0000","last_peered":"2024-09-21T10:53:58.094965+0000","last_clean":"2024-09-21T10:53:58.094965+0000","last_became_active":"2024-09-21T10:52:25.475423+0000","last_became_peered":"2024-09-21T10:52:25.475423+0000","last_unstale":"2024-09-21T10:53:58.094965+0000","last_undegraded":"2024-09-21T10:53:58.094965+0000","last_fullsized":"2024-09-21T10:53:58.094965+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:10.216527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"346'149","reported_seq":901,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.081050+0000","last_change":"2024-09-21T10:52:24.459450+0000","last_active":"2024-09-21T10:53:51.081050+0000","last_peered":"2024-09-21T10:53:51.081050+0000","last_clean":"2024-09-21T10:53:51.081050+0000","last_became_active":"2024-09-21T10:52:15.405544+0000","last_became_peered":"2024-09-21T10:52:15.405544+0000","last_unstale":"2024-09-21T10:53:51.081050+0000","last_undegraded":"2024-09-21T10:53:51.081050+0000","last_fullsized":"2024-09-21T10:53:51.081050+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2239,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.082444+0000","last_change":"2024-09-21T10:52:25.494316+0000","last_active":"2024-09-21T10:53:51.082444+0000","last_peered":"2024-09-21T10:53:51.082444+0000","last_clean":"2024-09-21T10:53:51.082444+0000","last_became_active":"2024-09-21T10:52:25.494016+0000","last_became_peered":"2024-09-21T10:52:25.494016+0000","last_unstale":"2024-09-21T10:53:51.082444+0000","last_undegraded":"2024-09-21T10:53:51.082444+0000","last_fullsized":"2024-09-21T10:53:51.082444+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:07.859584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2243,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.095159+0000","last_change":"2024-09-21T10:07:56.006236+0000","last_active":"2024-09-21T10:53:58.095159+0000","last_peered":"2024-09-21T10:53:58.095159+0000","last_clean":"2024-09-21T10:53:58.095159+0000","last_became_active":"2024-09-21T10:07:56.004439+0000","last_became_peered":"2024-09-21T10:07:56.004439+0000","last_unstale":"2024-09-21T10:53:58.095159+0000","last_undegraded":"2024-09-21T10:53:58.095159+0000","last_fullsized":"2024-09-21T10:53:58.095159+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:39.422467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":2290,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.082813+0000","last_change":"2024-09-21T10:52:25.536401+0000","last_active":"2024-09-21T10:53:51.082813+0000","last_peered":"2024-09-21T10:53:51.082813+0000","last_clean":"2024-09-21T10:53:51.082813+0000","last_became_active":"2024-09-21T10:52:25.496480+0000","last_became_peered":"2024-09-21T10:52:25.496480+0000","last_unstale":"2024-09-21T10:53:51.082813+0000","last_undegraded":"2024-09-21T10:53:51.082813+0000","last_fullsized":"2024-09-21T10:53:51.082813+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:50:34.820373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"346'149","reported_seq":726,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.084529+0000","last_change":"2024-09-21T10:51:30.450301+0000","last_active":"2024-09-21T10:53:51.084529+0000","last_peered":"2024-09-21T10:53:51.084529+0000","last_clean":"2024-09-21T10:53:51.084529+0000","last_became_active":"2024-09-21T10:51:30.449925+0000","last_became_peered":"2024-09-21T10:51:30.449925+0000","last_unstale":"2024-09-21T10:53:51.084529+0000","last_undegraded":"2024-09-21T10:53:51.084529+0000","last_fullsized":"2024-09-21T10:53:51.084529+0000","mapping_epoch":1450,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1451,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:41.011085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:41.011085+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:38:33.451261+0000","scrub_duration":0.0061250310000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2197,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.078004+0000","last_change":"2024-09-21T10:51:27.431416+0000","last_active":"2024-09-21T10:53:50.078004+0000","last_peered":"2024-09-21T10:53:50.078004+0000","last_clean":"2024-09-21T10:53:50.078004+0000","last_became_active":"2024-09-21T10:51:27.431127+0000","last_became_peered":"2024-09-21T10:51:27.431127+0000","last_unstale":"2024-09-21T10:53:50.078004+0000","last_undegraded":"2024-09-21T10:53:50.078004+0000","last_fullsized":"2024-09-21T10:53:50.078004+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:27.721665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.079270+0000","last_change":"2024-09-21T10:52:25.468849+0000","last_active":"2024-09-21T10:53:50.079270+0000","last_peered":"2024-09-21T10:53:50.079270+0000","last_clean":"2024-09-21T10:53:50.079270+0000","last_became_active":"2024-09-21T10:52:25.468552+0000","last_became_peered":"2024-09-21T10:52:25.468552+0000","last_unstale":"2024-09-21T10:53:50.079270+0000","last_undegraded":"2024-09-21T10:53:50.079270+0000","last_fullsized":"2024-09-21T10:53:50.079270+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:36:41.692028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2293,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.094593+0000","last_change":"2024-09-21T10:51:27.436608+0000","last_active":"2024-09-21T10:53:58.094593+0000","last_peered":"2024-09-21T10:53:58.094593+0000","last_clean":"2024-09-21T10:53:58.094593+0000","last_became_active":"2024-09-21T10:51:27.434949+0000","last_became_peered":"2024-09-21T10:51:27.434949+0000","last_unstale":"2024-09-21T10:53:58.094593+0000","last_undegraded":"2024-09-21T10:53:58.094593+0000","last_fullsized":"2024-09-21T10:53:58.094593+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:49:23.439425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"346'149","reported_seq":769,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.084759+0000","last_change":"2024-09-21T10:52:28.590793+0000","last_active":"2024-09-21T10:53:51.084759+0000","last_peered":"2024-09-21T10:53:51.084759+0000","last_clean":"2024-09-21T10:53:51.084759+0000","last_became_active":"2024-09-21T10:52:28.588662+0000","last_became_peered":"2024-09-21T10:52:28.588662+0000","last_unstale":"2024-09-21T10:53:51.084759+0000","last_undegraded":"2024-09-21T10:53:51.084759+0000","last_fullsized":"2024-09-21T10:53:51.084759+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:44.018645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:44.018645+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:46.849427+0000","scrub_duration":0.0012789399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2248,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.094537+0000","last_change":"2024-09-21T10:51:27.440425+0000","last_active":"2024-09-21T10:53:58.094537+0000","last_peered":"2024-09-21T10:53:58.094537+0000","last_clean":"2024-09-21T10:53:58.094537+0000","last_became_active":"2024-09-21T10:51:27.439666+0000","last_became_peered":"2024-09-21T10:51:27.439666+0000","last_unstale":"2024-09-21T10:53:58.094537+0000","last_undegraded":"2024-09-21T10:53:58.094537+0000","last_fullsized":"2024-09-21T10:53:58.094537+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:56:47.410516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2316,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.094362+0000","last_change":"2024-09-21T10:52:03.314926+0000","last_active":"2024-09-21T10:53:58.094362+0000","last_peered":"2024-09-21T10:53:58.094362+0000","last_clean":"2024-09-21T10:53:58.094362+0000","last_became_active":"2024-09-21T10:52:03.314602+0000","last_became_peered":"2024-09-21T10:52:03.314602+0000","last_unstale":"2024-09-21T10:53:58.094362+0000","last_undegraded":"2024-09-21T10:53:58.094362+0000","last_fullsized":"2024-09-21T10:53:58.094362+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:13.774521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1469,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.084237+0000","last_change":"2024-09-21T10:52:25.502830+0000","last_active":"2024-09-21T10:53:51.084237+0000","last_peered":"2024-09-21T10:53:51.084237+0000","last_clean":"2024-09-21T10:53:51.084237+0000","last_became_active":"2024-09-21T10:52:25.502505+0000","last_became_peered":"2024-09-21T10:52:25.502505+0000","last_unstale":"2024-09-21T10:53:51.084237+0000","last_undegraded":"2024-09-21T10:53:51.084237+0000","last_fullsized":"2024-09-21T10:53:51.084237+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:01:14.965694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"346'149","reported_seq":824,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.077728+0000","last_change":"2024-09-21T10:52:24.458657+0000","last_active":"2024-09-21T10:53:50.077728+0000","last_peered":"2024-09-21T10:53:50.077728+0000","last_clean":"2024-09-21T10:53:50.077728+0000","last_became_active":"2024-09-21T10:51:30.436983+0000","last_became_peered":"2024-09-21T10:51:30.436983+0000","last_unstale":"2024-09-21T10:53:50.077728+0000","last_undegraded":"2024-09-21T10:53:50.077728+0000","last_fullsized":"2024-09-21T10:53:50.077728+0000","mapping_epoch":1450,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1451,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2310,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.094834+0000","last_change":"2024-09-21T10:52:15.397180+0000","last_active":"2024-09-21T10:53:58.094834+0000","last_peered":"2024-09-21T10:53:58.094834+0000","last_clean":"2024-09-21T10:53:58.094834+0000","last_became_active":"2024-09-21T10:52:15.396879+0000","last_became_peered":"2024-09-21T10:52:15.396879+0000","last_unstale":"2024-09-21T10:53:58.094834+0000","last_undegraded":"2024-09-21T10:53:58.094834+0000","last_fullsized":"2024-09-21T10:53:58.094834+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:03:07.162178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1467,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.079065+0000","last_change":"2024-09-21T10:51:27.436590+0000","last_active":"2024-09-21T10:53:50.079065+0000","last_peered":"2024-09-21T10:53:50.079065+0000","last_clean":"2024-09-21T10:53:50.079065+0000","last_became_active":"2024-09-21T10:51:27.435332+0000","last_became_peered":"2024-09-21T10:51:27.435332+0000","last_unstale":"2024-09-21T10:53:50.079065+0000","last_undegraded":"2024-09-21T10:53:50.079065+0000","last_fullsized":"2024-09-21T10:53:50.079065+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:05:42.885506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":210,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.094366+0000","last_change":"2024-09-21T10:53:41.149328+0000","last_active":"2024-09-21T10:53:58.094366+0000","last_peered":"2024-09-21T10:53:58.094366+0000","last_clean":"2024-09-21T10:53:58.094366+0000","last_became_active":"2024-09-21T10:52:03.315398+0000","last_became_peered":"2024-09-21T10:52:03.315398+0000","last_unstale":"2024-09-21T10:53:58.094366+0000","last_undegraded":"2024-09-21T10:53:58.094366+0000","last_fullsized":"2024-09-21T10:53:58.094366+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:48:09.007053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"346'149","reported_seq":794,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.077684+0000","last_change":"2024-09-21T10:52:28.495319+0000","last_active":"2024-09-21T10:53:50.077684+0000","last_peered":"2024-09-21T10:53:50.077684+0000","last_clean":"2024-09-21T10:53:50.077684+0000","last_became_active":"2024-09-21T10:52:28.494923+0000","last_became_peered":"2024-09-21T10:52:28.494923+0000","last_unstale":"2024-09-21T10:53:50.077684+0000","last_undegraded":"2024-09-21T10:53:50.077684+0000","last_fullsized":"2024-09-21T10:53:50.077684+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:52:17.894787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:52:17.894787+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":184,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.080521+0000","last_change":"2024-09-21T10:52:25.475200+0000","last_active":"2024-09-21T10:53:50.080521+0000","last_peered":"2024-09-21T10:53:50.080521+0000","last_clean":"2024-09-21T10:53:50.080521+0000","last_became_active":"2024-09-21T10:52:25.474872+0000","last_became_peered":"2024-09-21T10:52:25.474872+0000","last_unstale":"2024-09-21T10:53:50.080521+0000","last_undegraded":"2024-09-21T10:53:50.080521+0000","last_fullsized":"2024-09-21T10:53:50.080521+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:36:34.207844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2266,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.094107+0000","last_change":"2024-09-21T10:52:03.325889+0000","last_active":"2024-09-21T10:53:58.094107+0000","last_peered":"2024-09-21T10:53:58.094107+0000","last_clean":"2024-09-21T10:53:58.094107+0000","last_became_active":"2024-09-21T10:52:03.325500+0000","last_became_peered":"2024-09-21T10:52:03.325500+0000","last_unstale":"2024-09-21T10:53:58.094107+0000","last_undegraded":"2024-09-21T10:53:58.094107+0000","last_fullsized":"2024-09-21T10:53:58.094107+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:20.708252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1050,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.081889+0000","last_change":"2024-09-21T10:51:27.457894+0000","last_active":"2024-09-21T10:53:51.081889+0000","last_peered":"2024-09-21T10:53:51.081889+0000","last_clean":"2024-09-21T10:53:51.081889+0000","last_became_active":"2024-09-21T10:51:27.457431+0000","last_became_peered":"2024-09-21T10:51:27.457431+0000","last_unstale":"2024-09-21T10:53:51.081889+0000","last_undegraded":"2024-09-21T10:53:51.081889+0000","last_fullsized":"2024-09-21T10:53:51.081889+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:03:33.550759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"346'149","reported_seq":923,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.095234+0000","last_change":"2024-09-21T10:52:15.396042+0000","last_active":"2024-09-21T10:53:58.095234+0000","last_peered":"2024-09-21T10:53:58.095234+0000","last_clean":"2024-09-21T10:53:58.095234+0000","last_became_active":"2024-09-21T10:52:15.395717+0000","last_became_peered":"2024-09-21T10:52:15.395717+0000","last_unstale":"2024-09-21T10:53:58.095234+0000","last_undegraded":"2024-09-21T10:53:58.095234+0000","last_fullsized":"2024-09-21T10:53:58.095234+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2292,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.094881+0000","last_change":"2024-09-21T10:52:25.476572+0000","last_active":"2024-09-21T10:53:58.094881+0000","last_peered":"2024-09-21T10:53:58.094881+0000","last_clean":"2024-09-21T10:53:58.094881+0000","last_became_active":"2024-09-21T10:52:25.475803+0000","last_became_peered":"2024-09-21T10:52:25.475803+0000","last_unstale":"2024-09-21T10:53:58.094881+0000","last_undegraded":"2024-09-21T10:53:58.094881+0000","last_fullsized":"2024-09-21T10:53:58.094881+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:06:44.191486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2213,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.078497+0000","last_change":"2024-09-21T10:52:03.326753+0000","last_active":"2024-09-21T10:53:50.078497+0000","last_peered":"2024-09-21T10:53:50.078497+0000","last_clean":"2024-09-21T10:53:50.078497+0000","last_became_active":"2024-09-21T10:52:03.326348+0000","last_became_peered":"2024-09-21T10:52:03.326348+0000","last_unstale":"2024-09-21T10:53:50.078497+0000","last_undegraded":"2024-09-21T10:53:50.078497+0000","last_fullsized":"2024-09-21T10:53:50.078497+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:57:38.100799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":135,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.077122+0000","last_change":"2024-09-21T10:51:27.426006+0000","last_active":"2024-09-21T10:53:50.077122+0000","last_peered":"2024-09-21T10:53:50.077122+0000","last_clean":"2024-09-21T10:53:50.077122+0000","last_became_active":"2024-09-21T10:51:27.425698+0000","last_became_peered":"2024-09-21T10:51:27.425698+0000","last_unstale":"2024-09-21T10:53:50.077122+0000","last_undegraded":"2024-09-21T10:53:50.077122+0000","last_fullsized":"2024-09-21T10:53:50.077122+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:35:13.210330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":29,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.079040+0000","last_change":"2024-09-21T10:52:15.401737+0000","last_active":"2024-09-21T10:53:50.079040+0000","last_peered":"2024-09-21T10:53:50.079040+0000","last_clean":"2024-09-21T10:53:50.079040+0000","last_became_active":"2024-09-21T10:52:15.400859+0000","last_became_peered":"2024-09-21T10:52:15.400859+0000","last_unstale":"2024-09-21T10:53:50.079040+0000","last_undegraded":"2024-09-21T10:53:50.079040+0000","last_fullsized":"2024-09-21T10:53:50.079040+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:15.723031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2244,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.082317+0000","last_change":"2024-09-21T10:49:44.865830+0000","last_active":"2024-09-21T10:53:51.082317+0000","last_peered":"2024-09-21T10:53:51.082317+0000","last_clean":"2024-09-21T10:53:51.082317+0000","last_became_active":"2024-09-21T10:49:44.865499+0000","last_became_peered":"2024-09-21T10:49:44.865499+0000","last_unstale":"2024-09-21T10:53:51.082317+0000","last_undegraded":"2024-09-21T10:53:51.082317+0000","last_fullsized":"2024-09-21T10:53:51.082317+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:19:22.013154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":1460,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.078126+0000","last_change":"2024-09-21T10:52:25.489550+0000","last_active":"2024-09-21T10:53:50.078126+0000","last_peered":"2024-09-21T10:53:50.078126+0000","last_clean":"2024-09-21T10:53:50.078126+0000","last_became_active":"2024-09-21T10:52:25.489171+0000","last_became_peered":"2024-09-21T10:52:25.489171+0000","last_unstale":"2024-09-21T10:53:50.078126+0000","last_undegraded":"2024-09-21T10:53:50.078126+0000","last_fullsized":"2024-09-21T10:53:50.078126+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:21.105058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":794,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.076992+0000","last_change":"2024-09-21T10:52:28.601906+0000","last_active":"2024-09-21T10:53:50.076992+0000","last_peered":"2024-09-21T10:53:50.076992+0000","last_clean":"2024-09-21T10:53:50.076992+0000","last_became_active":"2024-09-21T10:52:28.587447+0000","last_became_peered":"2024-09-21T10:52:28.587447+0000","last_unstale":"2024-09-21T10:53:50.076992+0000","last_undegraded":"2024-09-21T10:53:50.076992+0000","last_fullsized":"2024-09-21T10:53:50.076992+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:44.223578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:44.223578+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":74,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"346'149","reported_seq":743,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.083377+0000","last_change":"2024-09-21T10:51:30.453979+0000","last_active":"2024-09-21T10:53:51.083377+0000","last_peered":"2024-09-21T10:53:51.083377+0000","last_clean":"2024-09-21T10:53:51.083377+0000","last_became_active":"2024-09-21T10:51:30.450893+0000","last_became_peered":"2024-09-21T10:51:30.450893+0000","last_unstale":"2024-09-21T10:53:51.083377+0000","last_undegraded":"2024-09-21T10:53:51.083377+0000","last_fullsized":"2024-09-21T10:53:51.083377+0000","mapping_epoch":1450,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1451,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:42.008235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:42.008235+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:21:48.570809+0000","scrub_duration":0.011129834999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2243,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.094659+0000","last_change":"2024-09-21T10:09:06.538940+0000","last_active":"2024-09-21T10:53:58.094659+0000","last_peered":"2024-09-21T10:53:58.094659+0000","last_clean":"2024-09-21T10:53:58.094659+0000","last_became_active":"2024-09-21T10:09:06.538668+0000","last_became_peered":"2024-09-21T10:09:06.538668+0000","last_unstale":"2024-09-21T10:53:58.094659+0000","last_undegraded":"2024-09-21T10:53:58.094659+0000","last_fullsized":"2024-09-21T10:53:58.094659+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:42:44.299488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2239,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.081676+0000","last_change":"2024-09-21T10:52:25.493032+0000","last_active":"2024-09-21T10:53:51.081676+0000","last_peered":"2024-09-21T10:53:51.081676+0000","last_clean":"2024-09-21T10:53:51.081676+0000","last_became_active":"2024-09-21T10:52:25.492182+0000","last_became_peered":"2024-09-21T10:52:25.492182+0000","last_unstale":"2024-09-21T10:53:51.081676+0000","last_undegraded":"2024-09-21T10:53:51.081676+0000","last_fullsized":"2024-09-21T10:53:51.081676+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:12:57.118420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1473,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.080294+0000","last_change":"2024-09-21T10:52:15.402809+0000","last_active":"2024-09-21T10:53:50.080294+0000","last_peered":"2024-09-21T10:53:50.080294+0000","last_clean":"2024-09-21T10:53:50.080294+0000","last_became_active":"2024-09-21T10:52:15.402497+0000","last_became_peered":"2024-09-21T10:52:15.402497+0000","last_unstale":"2024-09-21T10:53:50.080294+0000","last_undegraded":"2024-09-21T10:53:50.080294+0000","last_fullsized":"2024-09-21T10:53:50.080294+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:07:35.713534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1484,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.078847+0000","last_change":"2024-09-21T10:44:14.399842+0000","last_active":"2024-09-21T10:53:50.078847+0000","last_peered":"2024-09-21T10:53:50.078847+0000","last_clean":"2024-09-21T10:53:50.078847+0000","last_became_active":"2024-09-21T10:44:14.399426+0000","last_became_peered":"2024-09-21T10:44:14.399426+0000","last_unstale":"2024-09-21T10:53:50.078847+0000","last_undegraded":"2024-09-21T10:53:50.078847+0000","last_fullsized":"2024-09-21T10:53:50.078847+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:26:45.362101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"346'149","reported_seq":920,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.081578+0000","last_change":"2024-09-21T10:52:16.404597+0000","last_active":"2024-09-21T10:53:51.081578+0000","last_peered":"2024-09-21T10:53:51.081578+0000","last_clean":"2024-09-21T10:53:51.081578+0000","last_became_active":"2024-09-21T10:52:16.404194+0000","last_became_peered":"2024-09-21T10:52:16.404194+0000","last_unstale":"2024-09-21T10:53:51.081578+0000","last_undegraded":"2024-09-21T10:53:51.081578+0000","last_fullsized":"2024-09-21T10:53:51.081578+0000","mapping_epoch":1463,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1464,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":793725,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1479,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.079656+0000","last_change":"2024-09-21T10:40:13.432387+0000","last_active":"2024-09-21T10:53:50.079656+0000","last_peered":"2024-09-21T10:53:50.079656+0000","last_clean":"2024-09-21T10:53:50.079656+0000","last_became_active":"2024-09-21T10:40:13.432053+0000","last_became_peered":"2024-09-21T10:40:13.432053+0000","last_unstale":"2024-09-21T10:53:50.079656+0000","last_undegraded":"2024-09-21T10:53:50.079656+0000","last_fullsized":"2024-09-21T10:53:50.079656+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:37.486101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1457,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.078757+0000","last_change":"2024-09-21T10:07:56.020791+0000","last_active":"2024-09-21T10:53:50.078757+0000","last_peered":"2024-09-21T10:53:50.078757+0000","last_clean":"2024-09-21T10:53:50.078757+0000","last_became_active":"2024-09-21T10:07:56.020205+0000","last_became_peered":"2024-09-21T10:07:56.020205+0000","last_unstale":"2024-09-21T10:53:50.078757+0000","last_undegraded":"2024-09-21T10:53:50.078757+0000","last_fullsized":"2024-09-21T10:53:50.078757+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:09.813166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"346'149","reported_seq":920,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.082020+0000","last_change":"2024-09-21T10:49:47.904295+0000","last_active":"2024-09-21T10:53:51.082020+0000","last_peered":"2024-09-21T10:53:51.082020+0000","last_clean":"2024-09-21T10:53:51.082020+0000","last_became_active":"2024-09-21T10:49:47.903650+0000","last_became_peered":"2024-09-21T10:49:47.903650+0000","last_unstale":"2024-09-21T10:53:51.082020+0000","last_undegraded":"2024-09-21T10:53:51.082020+0000","last_fullsized":"2024-09-21T10:53:51.082020+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":74,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1455,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.080122+0000","last_change":"2024-09-21T10:09:06.533271+0000","last_active":"2024-09-21T10:53:50.080122+0000","last_peered":"2024-09-21T10:53:50.080122+0000","last_clean":"2024-09-21T10:53:50.080122+0000","last_became_active":"2024-09-21T10:09:06.532041+0000","last_became_peered":"2024-09-21T10:09:06.532041+0000","last_unstale":"2024-09-21T10:53:50.080122+0000","last_undegraded":"2024-09-21T10:53:50.080122+0000","last_fullsized":"2024-09-21T10:53:50.080122+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:04.236375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":2268,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.082500+0000","last_change":"2024-09-21T10:51:27.529295+0000","last_active":"2024-09-21T10:53:51.082500+0000","last_peered":"2024-09-21T10:53:51.082500+0000","last_clean":"2024-09-21T10:53:51.082500+0000","last_became_active":"2024-09-21T10:51:27.446359+0000","last_became_peered":"2024-09-21T10:51:27.446359+0000","last_unstale":"2024-09-21T10:53:51.082500+0000","last_undegraded":"2024-09-21T10:53:51.082500+0000","last_fullsized":"2024-09-21T10:53:51.082500+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:17:08.788558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":701,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.095375+0000","last_change":"2024-09-21T10:51:27.481103+0000","last_active":"2024-09-21T10:53:58.095375+0000","last_peered":"2024-09-21T10:53:58.095375+0000","last_clean":"2024-09-21T10:53:58.095375+0000","last_became_active":"2024-09-21T10:51:27.480763+0000","last_became_peered":"2024-09-21T10:51:27.480763+0000","last_unstale":"2024-09-21T10:53:58.095375+0000","last_undegraded":"2024-09-21T10:53:58.095375+0000","last_fullsized":"2024-09-21T10:53:58.095375+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:35.375303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":804,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.082115+0000","last_change":"2024-09-21T10:49:44.864819+0000","last_active":"2024-09-21T10:53:51.082115+0000","last_peered":"2024-09-21T10:53:51.082115+0000","last_clean":"2024-09-21T10:53:51.082115+0000","last_became_active":"2024-09-21T10:49:44.864061+0000","last_became_peered":"2024-09-21T10:49:44.864061+0000","last_unstale":"2024-09-21T10:53:51.082115+0000","last_undegraded":"2024-09-21T10:53:51.082115+0000","last_fullsized":"2024-09-21T10:53:51.082115+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:37.266967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":38,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.078254+0000","last_change":"2024-09-21T10:52:25.492570+0000","last_active":"2024-09-21T10:53:50.078254+0000","last_peered":"2024-09-21T10:53:50.078254+0000","last_clean":"2024-09-21T10:53:50.078254+0000","last_became_active":"2024-09-21T10:52:25.486382+0000","last_became_peered":"2024-09-21T10:52:25.486382+0000","last_unstale":"2024-09-21T10:53:50.078254+0000","last_undegraded":"2024-09-21T10:53:50.078254+0000","last_fullsized":"2024-09-21T10:53:50.078254+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:06.702707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":143,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.081620+0000","last_change":"2024-09-21T10:49:44.874522+0000","last_active":"2024-09-21T10:53:50.081620+0000","last_peered":"2024-09-21T10:53:50.081620+0000","last_clean":"2024-09-21T10:53:50.081620+0000","last_became_active":"2024-09-21T10:49:44.863242+0000","last_became_peered":"2024-09-21T10:49:44.863242+0000","last_unstale":"2024-09-21T10:53:50.081620+0000","last_undegraded":"2024-09-21T10:53:50.081620+0000","last_fullsized":"2024-09-21T10:53:50.081620+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:09:02.066837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1489,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.078712+0000","last_change":"2024-09-21T10:52:03.315267+0000","last_active":"2024-09-21T10:53:50.078712+0000","last_peered":"2024-09-21T10:53:50.078712+0000","last_clean":"2024-09-21T10:53:50.078712+0000","last_became_active":"2024-09-21T10:52:03.314474+0000","last_became_peered":"2024-09-21T10:52:03.314474+0000","last_unstale":"2024-09-21T10:53:50.078712+0000","last_undegraded":"2024-09-21T10:53:50.078712+0000","last_fullsized":"2024-09-21T10:53:50.078712+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:00:57.791341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":30,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.079353+0000","last_change":"2024-09-21T10:52:25.518721+0000","last_active":"2024-09-21T10:53:50.079353+0000","last_peered":"2024-09-21T10:53:50.079353+0000","last_clean":"2024-09-21T10:53:50.079353+0000","last_became_active":"2024-09-21T10:52:25.472114+0000","last_became_peered":"2024-09-21T10:52:25.472114+0000","last_unstale":"2024-09-21T10:53:50.079353+0000","last_undegraded":"2024-09-21T10:53:50.079353+0000","last_fullsized":"2024-09-21T10:53:50.079353+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:33:48.471147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.076917+0000","last_change":"2024-09-21T10:52:25.494490+0000","last_active":"2024-09-21T10:53:50.076917+0000","last_peered":"2024-09-21T10:53:50.076917+0000","last_clean":"2024-09-21T10:53:50.076917+0000","last_became_active":"2024-09-21T10:52:25.494200+0000","last_became_peered":"2024-09-21T10:52:25.494200+0000","last_unstale":"2024-09-21T10:53:50.076917+0000","last_undegraded":"2024-09-21T10:53:50.076917+0000","last_fullsized":"2024-09-21T10:53:50.076917+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:15:21.030031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":30,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.078999+0000","last_change":"2024-09-21T10:52:25.479229+0000","last_active":"2024-09-21T10:53:50.078999+0000","last_peered":"2024-09-21T10:53:50.078999+0000","last_clean":"2024-09-21T10:53:50.078999+0000","last_became_active":"2024-09-21T10:52:25.470551+0000","last_became_peered":"2024-09-21T10:52:25.470551+0000","last_unstale":"2024-09-21T10:53:50.078999+0000","last_undegraded":"2024-09-21T10:53:50.078999+0000","last_fullsized":"2024-09-21T10:53:50.078999+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:15.458231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1501,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.077206+0000","last_change":"2024-09-21T10:51:27.517234+0000","last_active":"2024-09-21T10:53:50.077206+0000","last_peered":"2024-09-21T10:53:50.077206+0000","last_clean":"2024-09-21T10:53:50.077206+0000","last_became_active":"2024-09-21T10:51:27.431253+0000","last_became_peered":"2024-09-21T10:51:27.431253+0000","last_unstale":"2024-09-21T10:53:50.077206+0000","last_undegraded":"2024-09-21T10:53:50.077206+0000","last_fullsized":"2024-09-21T10:53:50.077206+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:35:42.249305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2223,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.082762+0000","last_change":"2024-09-21T10:52:25.497298+0000","last_active":"2024-09-21T10:53:51.082762+0000","last_peered":"2024-09-21T10:53:51.082762+0000","last_clean":"2024-09-21T10:53:51.082762+0000","last_became_active":"2024-09-21T10:52:25.496876+0000","last_became_peered":"2024-09-21T10:52:25.496876+0000","last_unstale":"2024-09-21T10:53:51.082762+0000","last_undegraded":"2024-09-21T10:53:51.082762+0000","last_fullsized":"2024-09-21T10:53:51.082762+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:16.587897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":126,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.085216+0000","last_change":"2024-09-21T10:44:14.398878+0000","last_active":"2024-09-21T10:53:50.085216+0000","last_peered":"2024-09-21T10:53:50.085216+0000","last_clean":"2024-09-21T10:53:50.085216+0000","last_became_active":"2024-09-21T10:44:14.398547+0000","last_became_peered":"2024-09-21T10:44:14.398547+0000","last_unstale":"2024-09-21T10:53:50.085216+0000","last_undegraded":"2024-09-21T10:53:50.085216+0000","last_fullsized":"2024-09-21T10:53:50.085216+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:13.008134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2272,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.082375+0000","last_change":"2024-09-21T10:52:25.498554+0000","last_active":"2024-09-21T10:53:51.082375+0000","last_peered":"2024-09-21T10:53:51.082375+0000","last_clean":"2024-09-21T10:53:51.082375+0000","last_became_active":"2024-09-21T10:52:25.498194+0000","last_became_peered":"2024-09-21T10:52:25.498194+0000","last_unstale":"2024-09-21T10:53:51.082375+0000","last_undegraded":"2024-09-21T10:53:51.082375+0000","last_fullsized":"2024-09-21T10:53:51.082375+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:47.115271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":26,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.077551+0000","last_change":"2024-09-21T10:52:25.489063+0000","last_active":"2024-09-21T10:53:50.077551+0000","last_peered":"2024-09-21T10:53:50.077551+0000","last_clean":"2024-09-21T10:53:50.077551+0000","last_became_active":"2024-09-21T10:52:25.488129+0000","last_became_peered":"2024-09-21T10:52:25.488129+0000","last_unstale":"2024-09-21T10:53:50.077551+0000","last_undegraded":"2024-09-21T10:53:50.077551+0000","last_fullsized":"2024-09-21T10:53:50.077551+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:36.996829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1441,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.081596+0000","last_change":"2024-09-21T10:49:44.864898+0000","last_active":"2024-09-21T10:53:50.081596+0000","last_peered":"2024-09-21T10:53:50.081596+0000","last_clean":"2024-09-21T10:53:50.081596+0000","last_became_active":"2024-09-21T10:49:44.862691+0000","last_became_peered":"2024-09-21T10:49:44.862691+0000","last_unstale":"2024-09-21T10:53:50.081596+0000","last_undegraded":"2024-09-21T10:53:50.081596+0000","last_fullsized":"2024-09-21T10:53:50.081596+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:57.809113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":132,"reported_epoch":1480,"state":"active+clean","last_fresh":"2024-09-21T10:53:50.078199+0000","last_change":"2024-09-21T10:44:14.418113+0000","last_active":"2024-09-21T10:53:50.078199+0000","last_peered":"2024-09-21T10:53:50.078199+0000","last_clean":"2024-09-21T10:53:50.078199+0000","last_became_active":"2024-09-21T10:44:14.398327+0000","last_became_peered":"2024-09-21T10:44:14.398327+0000","last_unstale":"2024-09-21T10:53:50.078199+0000","last_undegraded":"2024-09-21T10:53:50.078199+0000","last_fullsized":"2024-09-21T10:53:50.078199+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:08.810863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2253,"reported_epoch":1481,"state":"active+clean","last_fresh":"2024-09-21T10:53:51.082708+0000","last_change":"2024-09-21T10:44:14.398584+0000","last_active":"2024-09-21T10:53:51.082708+0000","last_peered":"2024-09-21T10:53:51.082708+0000","last_clean":"2024-09-21T10:53:51.082708+0000","last_became_active":"2024-09-21T10:44:14.398278+0000","last_became_peered":"2024-09-21T10:44:14.398278+0000","last_unstale":"2024-09-21T10:53:51.082708+0000","last_undegraded":"2024-09-21T10:53:51.082708+0000","last_fullsized":"2024-09-21T10:53:51.082708+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:19.681750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":55,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":268,"num_objects_misplaced":2879,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":210678,"num_write_kb":6741633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55424,"num_bytes_recovered":2985033728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3854700544,"data_stored":3854696472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74137,"ondisk_log_size":74137,"up":48,"acting":46,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":429359,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":276,"num_read_kb":598,"num_write":157,"num_write_kb":1757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":3982889,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2703360,"data_stored":2689152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":33,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":95,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":11344,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":18188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234719,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2549628,"kb_used_data":1964852,"kb_used_omap":1,"kb_used_meta":584766,"kb_avail":91195524,"statfs":{"total":95995035648,"available":93384216576,"internally_reserved":0,"allocated":2012008448,"data_stored":2006954815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":598800843},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496356,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":899340,"kb_used_data":196412,"kb_used_omap":1,"kb_used_meta":702910,"kb_avail":92845812,"statfs":{"total":95995035648,"available":95074111488,"internally_reserved":0,"allocated":201125888,"data_stored":196065186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":719780298},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757993,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019629,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":692748,"kb_used_data":30012,"kb_used_omap":1,"kb_used_meta":662718,"kb_avail":93052404,"statfs":{"total":95995035648,"available":95285661696,"internally_reserved":0,"allocated":30732288,"data_stored":25671538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":678623686},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369372,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":816180,"kb_used_data":12140,"kb_used_omap":69,"kb_used_meta":803962,"kb_avail":92928972,"statfs":{"total":95995035648,"available":95159267328,"internally_reserved":0,"allocated":12431360,"data_stored":7382264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":823257218},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2920577761828,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807958059,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":707200,"kb_used_data":11572,"kb_used_omap":66,"kb_used_meta":695549,"kb_avail":93037952,"statfs":{"total":95995035648,"available":95270862848,"internally_reserved":0,"allocated":11849728,"data_stored":6800238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68589,"internal_metadata":712242195},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":2860448219697,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":288755712,"data_stored":288751640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18284544,"data_stored":18284544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1371996160,"data_stored":1371996160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":188678144,"data_stored":188678144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1986985984,"data_stored":1986985984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:54:01.270 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:01.297 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:01.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:01.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:01.400 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:01.430 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:54:01.451 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:01.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:01.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:01.553 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:01.575 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:01.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:01.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:01.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:01 smithi160 ceph-mon[101879]: from='client.45911 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:54:01.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:01 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/432904939' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:54:01.677 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:01.698 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:01.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:01.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:01.800 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:01.822 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:01.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:01.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:01.925 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:01.947 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:01.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:01.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:02.049 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:02.075 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:02.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:02.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:02.176 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:02.198 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:02.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:02.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:02.300 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:02.321 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:02.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:02.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:02.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:02 smithi070 ceph-mon[107385]: pgmap v2736: 99 pgs: 1 active+recovery_wait+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 4 active+recovery_wait+remapped, 90 active+clean; 425 KiB data, 5.4 GiB used, 442 GiB / 447 GiB avail; 268/78 objects degraded (343.590%); 2879/78 objects misplaced (3691.026%); 0 B/s, 434 objects/s recovering 2024-09-21T10:54:02.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:02 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/432904939' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:54:02.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:02 smithi070 ceph-mon[107385]: osdmap e1483: 8 total, 8 up, 5 in 2024-09-21T10:54:02.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:02 smithi070 ceph-mon[112139]: pgmap v2736: 99 pgs: 1 active+recovery_wait+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 4 active+recovery_wait+remapped, 90 active+clean; 425 KiB data, 5.4 GiB used, 442 GiB / 447 GiB avail; 268/78 objects degraded (343.590%); 2879/78 objects misplaced (3691.026%); 0 B/s, 434 objects/s recovering 2024-09-21T10:54:02.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:02 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/432904939' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:54:02.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:02 smithi070 ceph-mon[112139]: osdmap e1483: 8 total, 8 up, 5 in 2024-09-21T10:54:02.423 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:02.450 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:02.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:02.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:02.551 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:02.573 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:02.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:02.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:02 smithi160 ceph-mon[101879]: pgmap v2736: 99 pgs: 1 active+recovery_wait+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 4 active+recovery_wait+remapped, 90 active+clean; 425 KiB data, 5.4 GiB used, 442 GiB / 447 GiB avail; 268/78 objects degraded (343.590%); 2879/78 objects misplaced (3691.026%); 0 B/s, 434 objects/s recovering 2024-09-21T10:54:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:02 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/432904939' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:54:02.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:02 smithi160 ceph-mon[101879]: osdmap e1483: 8 total, 8 up, 5 in 2024-09-21T10:54:02.675 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:02.696 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:02.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:02.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:02.756 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T10:54:02.757 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T10:54:02.757 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-09-21T10:54:02.757 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_3 16 2024-09-21T10:54:02.798 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:02.820 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:02.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:02.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:02.922 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:02.943 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:02.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:02.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:03.045 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:03.066 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:03.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:03.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:03.168 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:03.194 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:03.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:03.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:03.215 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:54:03.296 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:03.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:03.323 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:03.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:03.337 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:03 smithi070 ceph-mon[107385]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 268/78 objects degraded (343.590%), 1 pg degraded, 2 pgs undersized) 2024-09-21T10:54:03.337 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:03 smithi070 ceph-mon[112139]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 268/78 objects degraded (343.590%), 1 pg degraded, 2 pgs undersized) 2024-09-21T10:54:03.337 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:54:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:54:03] "GET /metrics HTTP/1.1" 200 36625 "" "Prometheus/2.51.0" 2024-09-21T10:54:03.424 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:03.450 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:03.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:03.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:03.552 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:03.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:03.573 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:03.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:03.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:03 smithi160 ceph-mon[101879]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 268/78 objects degraded (343.590%), 1 pg degraded, 2 pgs undersized) 2024-09-21T10:54:03.674 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:03.696 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:03.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:03.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:03.798 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:03.824 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:03.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:03.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:03.926 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:03.952 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:03.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:03.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:04.053 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:04.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:04.077 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:04.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:04.178 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:04.204 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:04.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:04.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:04.213 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:54:04.253 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:54:04.306 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:04.333 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:04.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:04.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:04.434 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:04.461 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:04.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:04.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:04.522 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:04 smithi070 ceph-mon[107385]: pgmap v2738: 83 pgs: 83 active+clean; 425 KiB data, 4.3 GiB used, 443 GiB / 447 GiB avail 2024-09-21T10:54:04.522 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:04 smithi070 ceph-mon[112139]: pgmap v2738: 83 pgs: 83 active+clean; 425 KiB data, 4.3 GiB used, 443 GiB / 447 GiB avail 2024-09-21T10:54:04.562 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:04.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:04.586 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:04.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:04.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:04 smithi160 ceph-mon[101879]: pgmap v2738: 83 pgs: 83 active+clean; 425 KiB data, 4.3 GiB used, 443 GiB / 447 GiB avail 2024-09-21T10:54:04.687 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:04.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:04.713 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:04.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:04.753 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:54:04.814 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:04.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:04.841 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:04.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:04.942 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:04.963 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:04.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:04.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:05.065 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:05.085 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:05.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:05.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:05.187 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:05.208 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:05.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:05.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:05.310 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:05.322 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:54:05.322 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T10:54:05.323 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T10:54:05.323 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T10:54:05.323 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.323 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:76000 55.3 55:cd844426:::benchmark_data_smithi070_294394_object75999:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1426)", 2024-09-21T10:54:05.323 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:49:43.855902+0000", 2024-09-21T10:54:05.323 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 260.62470195899999, 2024-09-21T10:54:05.323 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2807186910000001, 2024-09-21T10:54:05.323 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:54:05.324 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:54:05.324 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:54:05.324 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:54:05.324 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:54:05.324 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:54:05.324 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 76000 2024-09-21T10:54:05.324 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.324 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:54:05.325 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.325 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:54:05.325 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855902+0000", 2024-09-21T10:54:05.325 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.325 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.325 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.326 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:54:05.326 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855902+0000", 2024-09-21T10:54:05.326 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.326 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.326 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.326 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:54:05.326 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855904+0000", 2024-09-21T10:54:05.326 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1840000000000002e-06 2024-09-21T10:54:05.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.327 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:54:05.327 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855927+0000", 2024-09-21T10:54:05.327 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2909999999999999e-05 2024-09-21T10:54:05.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.327 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:54:05.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855957+0000", 2024-09-21T10:54:05.328 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9437e-05 2024-09-21T10:54:05.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:54:05.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855965+0000", 2024-09-21T10:54:05.329 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.4349999999999997e-06 2024-09-21T10:54:05.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.329 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.856021+0000", 2024-09-21T10:54:05.329 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6232e-05 2024-09-21T10:54:05.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:54:05.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.856214+0000", 2024-09-21T10:54:05.330 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00019308699999999999 2024-09-21T10:54:05.330 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.330 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,7", 2024-09-21T10:54:05.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.856386+0000", 2024-09-21T10:54:05.330 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000171621 2024-09-21T10:54:05.330 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.331 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.331 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.331 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.862577+0000", 2024-09-21T10:54:05.331 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0061906019999999999 2024-09-21T10:54:05.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.331 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.331 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.331 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.862589+0000", 2024-09-21T10:54:05.332 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2367e-05 2024-09-21T10:54:05.332 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.332 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.332 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.332 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.853010+0000", 2024-09-21T10:54:05.332 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99042137600000002 2024-09-21T10:54:05.332 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.332 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.332 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.333 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.853028+0000", 2024-09-21T10:54:05.333 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7776999999999999e-05 2024-09-21T10:54:05.333 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.333 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.333 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.333 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.116392+0000", 2024-09-21T10:54:05.333 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.263364207 2024-09-21T10:54:05.333 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.335 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.335 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:54:05.335 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.116926+0000", 2024-09-21T10:54:05.335 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053343100000000001 2024-09-21T10:54:05.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.335 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.335 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.335 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.136415+0000", 2024-09-21T10:54:05.336 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.019489623000000001 2024-09-21T10:54:05.336 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.336 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.336 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:54:05.336 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.136621+0000", 2024-09-21T10:54:05.336 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00020540200000000001 2024-09-21T10:54:05.336 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.336 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:54:05.336 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.337 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.337 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.337 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:76010 55.3 55:ca4bf1d6:::benchmark_data_smithi070_294394_object76009:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1427)", 2024-09-21T10:54:05.337 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:49:43.866567+0000", 2024-09-21T10:54:05.337 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 260.61403705599997, 2024-09-21T10:54:05.337 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2823839870000002, 2024-09-21T10:54:05.337 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:54:05.337 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:54:05.338 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:54:05.338 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:54:05.338 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:54:05.338 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:54:05.338 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 76010 2024-09-21T10:54:05.338 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.338 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:54:05.338 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.338 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:54:05.338 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.866567+0000", 2024-09-21T10:54:05.339 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.339 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:54:05.339 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.866567+0000", 2024-09-21T10:54:05.339 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.339 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:54:05.339 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.866568+0000", 2024-09-21T10:54:05.339 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5629999999999999e-06 2024-09-21T10:54:05.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.339 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:54:05.340 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.866589+0000", 2024-09-21T10:54:05.340 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0886999999999998e-05 2024-09-21T10:54:05.340 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.340 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.340 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:54:05.340 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.866600+0000", 2024-09-21T10:54:05.340 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0621e-05 2024-09-21T10:54:05.340 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.340 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.340 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:54:05.340 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.866608+0000", 2024-09-21T10:54:05.340 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9279999999999993e-06 2024-09-21T10:54:05.340 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.340 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.340 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.341 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.866999+0000", 2024-09-21T10:54:05.341 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000390734 2024-09-21T10:54:05.341 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.341 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.341 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.341 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.867008+0000", 2024-09-21T10:54:05.341 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2779999999999999e-06 2024-09-21T10:54:05.341 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.341 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.341 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.341 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.853163+0000", 2024-09-21T10:54:05.341 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.98615497699999999 2024-09-21T10:54:05.341 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.341 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.342 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.342 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.853170+0000", 2024-09-21T10:54:05.342 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5530000000000004e-06 2024-09-21T10:54:05.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.342 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.342 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.342 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.120718+0000", 2024-09-21T10:54:05.342 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2675472079999999 2024-09-21T10:54:05.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.342 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.342 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:54:05.342 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.120831+0000", 2024-09-21T10:54:05.342 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000113841 2024-09-21T10:54:05.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.343 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.343 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:54:05.343 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.120947+0000", 2024-09-21T10:54:05.343 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000115327 2024-09-21T10:54:05.343 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.343 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.343 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.343 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.136561+0000", 2024-09-21T10:54:05.343 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015613986 2024-09-21T10:54:05.343 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.343 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.343 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.343 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.136585+0000", 2024-09-21T10:54:05.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.429e-05 2024-09-21T10:54:05.344 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.344 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:54:05.344 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.144528+0000", 2024-09-21T10:54:05.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0079428559999999999 2024-09-21T10:54:05.344 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.344 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:54:05.344 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.144577+0000", 2024-09-21T10:54:05.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8749999999999999e-05 2024-09-21T10:54:05.344 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.344 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:54:05.345 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.148951+0000", 2024-09-21T10:54:05.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0043741880000000002 2024-09-21T10:54:05.345 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.345 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:54:05.345 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.345 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:76030 55.3 55:c3f9098d:::benchmark_data_smithi070_294394_object76029:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1427)", 2024-09-21T10:54:05.345 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:49:43.887718+0000", 2024-09-21T10:54:05.345 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 260.59288587999998, 2024-09-21T10:54:05.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2693207429999998, 2024-09-21T10:54:05.345 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:54:05.345 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:54:05.345 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:54:05.346 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:54:05.346 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:54:05.346 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:54:05.346 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 76030 2024-09-21T10:54:05.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.346 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:54:05.346 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:54:05.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887718+0000", 2024-09-21T10:54:05.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.346 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:54:05.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887718+0000", 2024-09-21T10:54:05.347 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.347 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:54:05.347 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887720+0000", 2024-09-21T10:54:05.347 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5880000000000001e-06 2024-09-21T10:54:05.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.347 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:54:05.347 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887741+0000", 2024-09-21T10:54:05.347 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0874e-05 2024-09-21T10:54:05.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.347 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:54:05.347 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887754+0000", 2024-09-21T10:54:05.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3876999999999999e-05 2024-09-21T10:54:05.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.348 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:54:05.348 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887761+0000", 2024-09-21T10:54:05.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8290000000000002e-06 2024-09-21T10:54:05.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.348 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.348 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887819+0000", 2024-09-21T10:54:05.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7469999999999997e-05 2024-09-21T10:54:05.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.348 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.349 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887828+0000", 2024-09-21T10:54:05.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7559999999999994e-06 2024-09-21T10:54:05.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.349 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.349 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.853198+0000", 2024-09-21T10:54:05.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.96536982999999998 2024-09-21T10:54:05.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.349 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.349 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.853205+0000", 2024-09-21T10:54:05.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1559999999999998e-06 2024-09-21T10:54:05.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.136704+0000", 2024-09-21T10:54:05.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.283498252 2024-09-21T10:54:05.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:54:05.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.136820+0000", 2024-09-21T10:54:05.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011636700000000001 2024-09-21T10:54:05.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:54:05.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.136933+0000", 2024-09-21T10:54:05.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000112872 2024-09-21T10:54:05.351 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.351 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.351 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.149001+0000", 2024-09-21T10:54:05.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012068146 2024-09-21T10:54:05.351 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.351 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:54:05.351 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.153155+0000", 2024-09-21T10:54:05.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0041541190000000004 2024-09-21T10:54:05.351 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.351 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.351 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.153219+0000", 2024-09-21T10:54:05.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3492999999999997e-05 2024-09-21T10:54:05.352 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.352 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.352 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:54:05.352 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.153287+0000", 2024-09-21T10:54:05.352 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7943999999999999e-05 2024-09-21T10:54:05.352 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.352 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.352 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:54:05.352 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.157039+0000", 2024-09-21T10:54:05.352 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00375217 2024-09-21T10:54:05.352 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.352 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:54:05.352 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.352 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.353 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.353 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:91257 55.2 55:43dc9448:::benchmark_data_smithi070_294394_object91256:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1432)", 2024-09-21T10:54:05.353 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:50:03.040864+0000", 2024-09-21T10:54:05.353 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 241.439739484, 2024-09-21T10:54:05.353 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7075791259999999, 2024-09-21T10:54:05.353 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:54:05.353 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:54:05.353 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:54:05.353 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:54:05.353 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:54:05.353 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:54:05.353 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 91257 2024-09-21T10:54:05.353 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.353 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:54:05.354 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.354 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:54:05.354 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.040864+0000", 2024-09-21T10:54:05.354 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.354 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.354 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:54:05.354 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.040864+0000", 2024-09-21T10:54:05.354 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.354 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.354 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:54:05.354 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.040866+0000", 2024-09-21T10:54:05.354 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5859999999999999e-06 2024-09-21T10:54:05.355 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.355 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.355 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:54:05.355 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.040893+0000", 2024-09-21T10:54:05.355 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7036999999999999e-05 2024-09-21T10:54:05.355 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.355 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.355 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:54:05.355 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.040904+0000", 2024-09-21T10:54:05.355 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0513000000000001e-05 2024-09-21T10:54:05.355 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.355 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.355 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:54:05.355 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.040910+0000", 2024-09-21T10:54:05.356 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4280000000000001e-06 2024-09-21T10:54:05.356 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.356 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.356 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.356 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.041137+0000", 2024-09-21T10:54:05.356 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000227297 2024-09-21T10:54:05.356 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.356 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.356 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:54:05.356 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.045948+0000", 2024-09-21T10:54:05.356 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0048108129999999997 2024-09-21T10:54:05.356 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.356 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.356 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:54:05.356 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.046101+0000", 2024-09-21T10:54:05.357 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015316599999999999 2024-09-21T10:54:05.357 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.357 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.357 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:54:05.357 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.050938+0000", 2024-09-21T10:54:05.357 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0048367210000000004 2024-09-21T10:54:05.357 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.357 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.357 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.357 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.065829+0000", 2024-09-21T10:54:05.357 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.014891051000000001 2024-09-21T10:54:05.357 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.357 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.358 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.065844+0000", 2024-09-21T10:54:05.358 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4579999999999999e-05 2024-09-21T10:54:05.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.358 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.257262+0000", 2024-09-21T10:54:05.358 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.191418597 2024-09-21T10:54:05.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.358 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.257272+0000", 2024-09-21T10:54:05.358 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5789999999999993e-06 2024-09-21T10:54:05.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.718417+0000", 2024-09-21T10:54:05.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.461144759 2024-09-21T10:54:05.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:54:05.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.723923+0000", 2024-09-21T10:54:05.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0055066669999999998 2024-09-21T10:54:05.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.748299+0000", 2024-09-21T10:54:05.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.024375267999999999 2024-09-21T10:54:05.360 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.360 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.360 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:54:05.360 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.748444+0000", 2024-09-21T10:54:05.360 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014506400000000001 2024-09-21T10:54:05.360 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.360 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:54:05.360 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.360 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.360 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.360 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:91271 55.2 55:4c7a1cbb:::benchmark_data_smithi070_294394_object91270:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1433)", 2024-09-21T10:54:05.360 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:50:03.087644+0000", 2024-09-21T10:54:05.360 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 241.392960155, 2024-09-21T10:54:05.360 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.665248112, 2024-09-21T10:54:05.361 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:54:05.361 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:54:05.361 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:54:05.361 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:54:05.361 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:54:05.361 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:54:05.361 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 91271 2024-09-21T10:54:05.361 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.361 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:54:05.361 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.361 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:54:05.361 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.087644+0000", 2024-09-21T10:54:05.361 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.361 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.361 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.362 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:54:05.362 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.087644+0000", 2024-09-21T10:54:05.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.362 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:54:05.362 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.087646+0000", 2024-09-21T10:54:05.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.204e-06 2024-09-21T10:54:05.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.362 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:54:05.362 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.087676+0000", 2024-09-21T10:54:05.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0414000000000001e-05 2024-09-21T10:54:05.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.363 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:54:05.363 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.087699+0000", 2024-09-21T10:54:05.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2390000000000001e-05 2024-09-21T10:54:05.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.363 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:54:05.363 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.087711+0000", 2024-09-21T10:54:05.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1701e-05 2024-09-21T10:54:05.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.363 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.363 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.087965+0000", 2024-09-21T10:54:05.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00025414200000000001 2024-09-21T10:54:05.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.364 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.364 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.087981+0000", 2024-09-21T10:54:05.364 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6010000000000001e-05 2024-09-21T10:54:05.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.364 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.364 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.257295+0000", 2024-09-21T10:54:05.364 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.169314252 2024-09-21T10:54:05.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.364 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.364 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.257301+0000", 2024-09-21T10:54:05.365 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.207e-06 2024-09-21T10:54:05.365 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.365 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.365 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.365 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.728462+0000", 2024-09-21T10:54:05.365 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.471161285 2024-09-21T10:54:05.365 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.365 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.365 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:54:05.365 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.728637+0000", 2024-09-21T10:54:05.365 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000174742 2024-09-21T10:54:05.365 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.365 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.365 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:54:05.366 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.729182+0000", 2024-09-21T10:54:05.366 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00054522299999999995 2024-09-21T10:54:05.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.366 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.366 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:54:05.366 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.748442+0000", 2024-09-21T10:54:05.366 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.019259486999999999 2024-09-21T10:54:05.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.366 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.366 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.366 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.748493+0000", 2024-09-21T10:54:05.366 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1008999999999999e-05 2024-09-21T10:54:05.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.366 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.367 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.367 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.748540+0000", 2024-09-21T10:54:05.367 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7389000000000003e-05 2024-09-21T10:54:05.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.367 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.367 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:54:05.367 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.748644+0000", 2024-09-21T10:54:05.367 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000103699 2024-09-21T10:54:05.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.367 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.367 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:54:05.367 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.752892+0000", 2024-09-21T10:54:05.367 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0042479579999999996 2024-09-21T10:54:05.368 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.368 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:54:05.368 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.368 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.368 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.368 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:91282 55.2 55:40800590:::benchmark_data_smithi070_294394_object91281:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1433)", 2024-09-21T10:54:05.368 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:50:03.100410+0000", 2024-09-21T10:54:05.368 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 241.380193529, 2024-09-21T10:54:05.368 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7104403500000001, 2024-09-21T10:54:05.368 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:54:05.368 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:54:05.368 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:54:05.368 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:54:05.368 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:54:05.369 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:54:05.369 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 91282 2024-09-21T10:54:05.369 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.369 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:54:05.369 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.369 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:54:05.369 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.100410+0000", 2024-09-21T10:54:05.369 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.369 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.369 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.369 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:54:05.369 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.100410+0000", 2024-09-21T10:54:05.369 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.369 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.370 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.370 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:54:05.370 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.100412+0000", 2024-09-21T10:54:05.370 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9549999999999999e-06 2024-09-21T10:54:05.370 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.370 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.370 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:54:05.370 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.100438+0000", 2024-09-21T10:54:05.370 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5133000000000001e-05 2024-09-21T10:54:05.370 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.370 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.370 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:54:05.370 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.100461+0000", 2024-09-21T10:54:05.371 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3764000000000002e-05 2024-09-21T10:54:05.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.371 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.371 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:54:05.371 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.100471+0000", 2024-09-21T10:54:05.371 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0200999999999999e-05 2024-09-21T10:54:05.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.371 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.371 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.371 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.100550+0000", 2024-09-21T10:54:05.371 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8767e-05 2024-09-21T10:54:05.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.371 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.371 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.372 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.100561+0000", 2024-09-21T10:54:05.372 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0545e-05 2024-09-21T10:54:05.372 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.372 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.372 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.372 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.257320+0000", 2024-09-21T10:54:05.372 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.15675908199999999 2024-09-21T10:54:05.372 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.372 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.372 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.372 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.257326+0000", 2024-09-21T10:54:05.372 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0209999999999996e-06 2024-09-21T10:54:05.372 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.372 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.373 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.373 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.752938+0000", 2024-09-21T10:54:05.373 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4956121549999999 2024-09-21T10:54:05.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.373 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:54:05.373 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.753091+0000", 2024-09-21T10:54:05.373 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015340300000000001 2024-09-21T10:54:05.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.373 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:54:05.373 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.753264+0000", 2024-09-21T10:54:05.373 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000172924 2024-09-21T10:54:05.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.374 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.374 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:54:05.374 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.766215+0000", 2024-09-21T10:54:05.374 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012951028 2024-09-21T10:54:05.374 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.374 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.374 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.374 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.781445+0000", 2024-09-21T10:54:05.374 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015229351 2024-09-21T10:54:05.374 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.374 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.374 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.374 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.803339+0000", 2024-09-21T10:54:05.374 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.021893968999999999 2024-09-21T10:54:05.375 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.375 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.375 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:54:05.375 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.803430+0000", 2024-09-21T10:54:05.375 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1318999999999998e-05 2024-09-21T10:54:05.375 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.375 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.375 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:54:05.375 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.810851+0000", 2024-09-21T10:54:05.375 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0074207329999999997 2024-09-21T10:54:05.375 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.375 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:54:05.375 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.376 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.376 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.376 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:91286 55.2 55:4def9a29:::benchmark_data_smithi070_294394_object91285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1433)", 2024-09-21T10:54:05.376 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:50:03.104675+0000", 2024-09-21T10:54:05.376 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 241.37592916899999, 2024-09-21T10:54:05.376 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7145347900000001, 2024-09-21T10:54:05.376 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:54:05.376 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:54:05.376 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:54:05.376 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:54:05.376 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:54:05.376 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:54:05.376 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 91286 2024-09-21T10:54:05.376 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.376 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:54:05.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.377 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:54:05.377 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.104675+0000", 2024-09-21T10:54:05.377 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.377 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.377 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:54:05.377 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.104675+0000", 2024-09-21T10:54:05.377 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.377 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.377 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:54:05.377 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.104676+0000", 2024-09-21T10:54:05.377 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.576e-06 2024-09-21T10:54:05.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.378 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.378 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:54:05.378 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.104694+0000", 2024-09-21T10:54:05.378 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7638999999999998e-05 2024-09-21T10:54:05.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.378 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.378 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:54:05.378 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.104704+0000", 2024-09-21T10:54:05.378 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0358e-05 2024-09-21T10:54:05.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.378 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.378 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:54:05.378 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.104709+0000", 2024-09-21T10:54:05.379 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1159999999999998e-06 2024-09-21T10:54:05.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.379 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.379 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.379 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.104755+0000", 2024-09-21T10:54:05.379 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5584000000000003e-05 2024-09-21T10:54:05.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.379 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.379 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.379 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.104761+0000", 2024-09-21T10:54:05.379 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6590000000000004e-06 2024-09-21T10:54:05.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.379 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.379 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.380 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.257345+0000", 2024-09-21T10:54:05.380 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.15258385399999999 2024-09-21T10:54:05.380 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.380 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.380 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.380 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.257350+0000", 2024-09-21T10:54:05.380 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8109999999999999e-06 2024-09-21T10:54:05.380 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.380 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.380 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.380 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.755959+0000", 2024-09-21T10:54:05.380 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4986089069999999 2024-09-21T10:54:05.380 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.380 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.381 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:54:05.381 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.756079+0000", 2024-09-21T10:54:05.381 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000119536 2024-09-21T10:54:05.381 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.381 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.381 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:54:05.381 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.756274+0000", 2024-09-21T10:54:05.381 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00019505400000000001 2024-09-21T10:54:05.381 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.381 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.381 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:54:05.381 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.766227+0000", 2024-09-21T10:54:05.381 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0099535519999999992 2024-09-21T10:54:05.381 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.382 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.382 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.382 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.785592+0000", 2024-09-21T10:54:05.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.019365054 2024-09-21T10:54:05.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.382 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.382 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.382 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.810900+0000", 2024-09-21T10:54:05.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.025307168000000001 2024-09-21T10:54:05.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.382 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.382 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:54:05.382 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.811297+0000", 2024-09-21T10:54:05.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00039743999999999999 2024-09-21T10:54:05.383 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.383 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.383 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:54:05.383 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.819210+0000", 2024-09-21T10:54:05.383 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0079124820000000002 2024-09-21T10:54:05.383 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.383 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:54:05.383 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.383 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.383 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.383 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:91298 55.2 55:4da05bce:::benchmark_data_smithi070_294394_object91297:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1433)", 2024-09-21T10:54:05.383 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:50:03.123418+0000", 2024-09-21T10:54:05.383 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 241.35718576900001, 2024-09-21T10:54:05.383 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7002024090000001, 2024-09-21T10:54:05.384 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:54:05.384 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:54:05.384 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:54:05.384 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:54:05.384 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:54:05.384 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:54:05.384 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 91298 2024-09-21T10:54:05.384 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.384 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:54:05.384 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.384 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:54:05.384 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123418+0000", 2024-09-21T10:54:05.384 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:54:05.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123418+0000", 2024-09-21T10:54:05.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:54:05.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123420+0000", 2024-09-21T10:54:05.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.556e-06 2024-09-21T10:54:05.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:54:05.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123438+0000", 2024-09-21T10:54:05.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8634000000000002e-05 2024-09-21T10:54:05.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:54:05.386 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123449+0000", 2024-09-21T10:54:05.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0781999999999999e-05 2024-09-21T10:54:05.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:54:05.386 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123454+0000", 2024-09-21T10:54:05.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3109999999999996e-06 2024-09-21T10:54:05.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.387 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123509+0000", 2024-09-21T10:54:05.387 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4549000000000002e-05 2024-09-21T10:54:05.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.387 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.387 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.387 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123520+0000", 2024-09-21T10:54:05.387 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1070000000000001e-05 2024-09-21T10:54:05.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.387 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.387 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.387 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.257549+0000", 2024-09-21T10:54:05.387 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.134029119 2024-09-21T10:54:05.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.387 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.387 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.388 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.257556+0000", 2024-09-21T10:54:05.388 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8040000000000004e-06 2024-09-21T10:54:05.388 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.388 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.388 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.388 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.759290+0000", 2024-09-21T10:54:05.388 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5017340020000001 2024-09-21T10:54:05.388 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.388 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.388 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:54:05.388 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.759477+0000", 2024-09-21T10:54:05.388 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000187097 2024-09-21T10:54:05.388 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.389 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.389 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:54:05.389 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.759660+0000", 2024-09-21T10:54:05.389 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018245499999999999 2024-09-21T10:54:05.389 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.389 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.389 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:54:05.389 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.775049+0000", 2024-09-21T10:54:05.389 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015389534999999999 2024-09-21T10:54:05.389 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.389 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.389 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.389 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.785632+0000", 2024-09-21T10:54:05.389 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.010582505000000001 2024-09-21T10:54:05.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.390 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.390 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.819272+0000", 2024-09-21T10:54:05.390 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.033639948000000003 2024-09-21T10:54:05.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.390 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:54:05.390 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.819359+0000", 2024-09-21T10:54:05.390 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7269999999999996e-05 2024-09-21T10:54:05.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.390 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:54:05.390 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.823621+0000", 2024-09-21T10:54:05.391 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0042617719999999996 2024-09-21T10:54:05.391 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.391 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:54:05.391 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.391 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.391 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.391 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:91299 55.3 55:c9f1c749:::benchmark_data_smithi070_294394_object91298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1433)", 2024-09-21T10:54:05.391 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:50:03.123772+0000", 2024-09-21T10:54:05.391 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 241.35683216499999, 2024-09-21T10:54:05.391 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6459212750000001, 2024-09-21T10:54:05.391 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:54:05.391 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:54:05.391 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:54:05.391 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:54:05.392 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:54:05.392 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:54:05.392 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 91299 2024-09-21T10:54:05.392 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.392 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:54:05.392 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.392 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:54:05.392 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123772+0000", 2024-09-21T10:54:05.392 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.392 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.392 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.392 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:54:05.392 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123772+0000", 2024-09-21T10:54:05.392 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.392 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.393 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.393 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:54:05.393 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123773+0000", 2024-09-21T10:54:05.393 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7599999999999996e-07 2024-09-21T10:54:05.393 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.393 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.393 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:54:05.393 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123792+0000", 2024-09-21T10:54:05.393 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9372999999999999e-05 2024-09-21T10:54:05.393 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.393 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.393 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:54:05.393 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123801+0000", 2024-09-21T10:54:05.393 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2660000000000007e-06 2024-09-21T10:54:05.394 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.394 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.394 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:54:05.394 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123805+0000", 2024-09-21T10:54:05.394 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2930000000000002e-06 2024-09-21T10:54:05.394 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.394 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.394 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.394 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123852+0000", 2024-09-21T10:54:05.394 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6292999999999999e-05 2024-09-21T10:54:05.394 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.394 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.394 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.394 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123860+0000", 2024-09-21T10:54:05.395 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5790000000000004e-06 2024-09-21T10:54:05.395 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.395 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.395 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.395 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.254002+0000", 2024-09-21T10:54:05.395 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13014203199999999 2024-09-21T10:54:05.395 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.395 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.395 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.395 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.254014+0000", 2024-09-21T10:54:05.395 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1302e-05 2024-09-21T10:54:05.395 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.395 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.395 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.396 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.598674+0000", 2024-09-21T10:54:05.396 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3446601089999999 2024-09-21T10:54:05.396 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.396 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.396 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:54:05.396 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.598843+0000", 2024-09-21T10:54:05.396 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016916300000000001 2024-09-21T10:54:05.396 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.396 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.396 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:54:05.396 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.599473+0000", 2024-09-21T10:54:05.396 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062992099999999997 2024-09-21T10:54:05.396 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.396 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.396 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.397 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.761869+0000", 2024-09-21T10:54:05.397 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.16239562599999999 2024-09-21T10:54:05.397 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.397 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.397 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.397 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.761909+0000", 2024-09-21T10:54:05.397 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0163999999999997e-05 2024-09-21T10:54:05.397 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.397 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.397 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:54:05.397 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.764195+0000", 2024-09-21T10:54:05.397 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002286293 2024-09-21T10:54:05.397 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.398 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.398 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:54:05.398 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.764537+0000", 2024-09-21T10:54:05.398 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000341892 2024-09-21T10:54:05.398 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.398 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.398 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:54:05.398 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.769693+0000", 2024-09-21T10:54:05.398 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0051561929999999999 2024-09-21T10:54:05.398 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.398 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:54:05.398 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.398 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.398 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.399 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:91300 55.3 55:c662b782:::benchmark_data_smithi070_294394_object91299:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1433)", 2024-09-21T10:54:05.399 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:50:03.123953+0000", 2024-09-21T10:54:05.399 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 241.356651081, 2024-09-21T10:54:05.399 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.67947549, 2024-09-21T10:54:05.399 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:54:05.399 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:54:05.399 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:54:05.399 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:54:05.399 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:54:05.399 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:54:05.399 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 91300 2024-09-21T10:54:05.399 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.399 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:54:05.399 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.400 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:54:05.400 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123953+0000", 2024-09-21T10:54:05.400 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.400 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.400 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.400 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:54:05.400 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123953+0000", 2024-09-21T10:54:05.400 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.400 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.400 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.400 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:54:05.400 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123954+0000", 2024-09-21T10:54:05.400 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8400000000000003e-07 2024-09-21T10:54:05.401 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.401 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.401 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:54:05.401 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123973+0000", 2024-09-21T10:54:05.401 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9658000000000001e-05 2024-09-21T10:54:05.401 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.401 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.401 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:54:05.401 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123981+0000", 2024-09-21T10:54:05.401 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8469999999999996e-06 2024-09-21T10:54:05.401 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.401 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.401 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:54:05.401 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.123985+0000", 2024-09-21T10:54:05.402 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9899999999999999e-06 2024-09-21T10:54:05.402 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.402 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.402 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.402 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.124026+0000", 2024-09-21T10:54:05.402 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.1102000000000001e-05 2024-09-21T10:54:05.402 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.402 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.402 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.402 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.124035+0000", 2024-09-21T10:54:05.402 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1579999999999994e-06 2024-09-21T10:54:05.402 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.402 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.402 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.403 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.254041+0000", 2024-09-21T10:54:05.403 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13000513499999999 2024-09-21T10:54:05.403 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.403 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.403 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.403 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.254059+0000", 2024-09-21T10:54:05.403 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8929000000000001e-05 2024-09-21T10:54:05.403 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.403 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.403 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.403 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.769844+0000", 2024-09-21T10:54:05.403 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.51578426 2024-09-21T10:54:05.403 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.403 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.404 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:54:05.404 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.769972+0000", 2024-09-21T10:54:05.404 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000128233 2024-09-21T10:54:05.404 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.404 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.404 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:54:05.404 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.770116+0000", 2024-09-21T10:54:05.404 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014439499999999999 2024-09-21T10:54:05.404 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.404 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.404 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:54:05.404 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.783082+0000", 2024-09-21T10:54:05.404 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012965829 2024-09-21T10:54:05.404 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.405 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.405 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.405 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.797652+0000", 2024-09-21T10:54:05.405 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.014569446 2024-09-21T10:54:05.405 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.405 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.405 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.405 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.798576+0000", 2024-09-21T10:54:05.405 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00092482000000000003 2024-09-21T10:54:05.405 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.405 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.405 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:54:05.405 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.798648+0000", 2024-09-21T10:54:05.405 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1123000000000003e-05 2024-09-21T10:54:05.405 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.406 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.406 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:54:05.406 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.803428+0000", 2024-09-21T10:54:05.406 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0047807809999999996 2024-09-21T10:54:05.406 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.406 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:54:05.406 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.406 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.406 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.406 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:91308 55.2 55:432b9bad:::benchmark_data_smithi070_294394_object91307:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1433)", 2024-09-21T10:54:05.406 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:50:03.151172+0000", 2024-09-21T10:54:05.406 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 241.32943177800001, 2024-09-21T10:54:05.406 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6770448820000001, 2024-09-21T10:54:05.406 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:54:05.407 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:54:05.407 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:54:05.407 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:54:05.407 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:54:05.407 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:54:05.407 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 91308 2024-09-21T10:54:05.407 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.407 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:54:05.407 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.407 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:54:05.407 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.151172+0000", 2024-09-21T10:54:05.407 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.407 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.407 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.408 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:54:05.408 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.151172+0000", 2024-09-21T10:54:05.408 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.408 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.408 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.408 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:54:05.408 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.151174+0000", 2024-09-21T10:54:05.408 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.46e-06 2024-09-21T10:54:05.408 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.408 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.408 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:54:05.408 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.151192+0000", 2024-09-21T10:54:05.408 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8454e-05 2024-09-21T10:54:05.408 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.409 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.409 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:54:05.409 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.151203+0000", 2024-09-21T10:54:05.409 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1241e-05 2024-09-21T10:54:05.409 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.409 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.409 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:54:05.409 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.151209+0000", 2024-09-21T10:54:05.409 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3480000000000003e-06 2024-09-21T10:54:05.409 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.409 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.409 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.409 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.151270+0000", 2024-09-21T10:54:05.409 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1665e-05 2024-09-21T10:54:05.409 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.410 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.410 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.410 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.151282+0000", 2024-09-21T10:54:05.410 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1701e-05 2024-09-21T10:54:05.410 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.410 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.410 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.410 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.257576+0000", 2024-09-21T10:54:05.410 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10629356 2024-09-21T10:54:05.410 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.410 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.410 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.410 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.257582+0000", 2024-09-21T10:54:05.411 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9349999999999999e-06 2024-09-21T10:54:05.411 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.411 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.411 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.411 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.763154+0000", 2024-09-21T10:54:05.411 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.505572508 2024-09-21T10:54:05.411 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.411 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.411 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:54:05.411 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.763319+0000", 2024-09-21T10:54:05.411 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016480800000000001 2024-09-21T10:54:05.411 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.411 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.411 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:54:05.412 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.763503+0000", 2024-09-21T10:54:05.412 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018448299999999999 2024-09-21T10:54:05.412 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.412 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.412 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:54:05.412 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.775057+0000", 2024-09-21T10:54:05.412 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.011553178000000001 2024-09-21T10:54:05.412 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.412 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.412 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.412 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.785667+0000", 2024-09-21T10:54:05.412 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.010610385 2024-09-21T10:54:05.412 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.412 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.412 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.413 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.823676+0000", 2024-09-21T10:54:05.413 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.038008844999999999 2024-09-21T10:54:05.413 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.413 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.413 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:54:05.413 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.823737+0000", 2024-09-21T10:54:05.413 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1110000000000003e-05 2024-09-21T10:54:05.413 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.413 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.413 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:54:05.413 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.828217+0000", 2024-09-21T10:54:05.413 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0044802009999999996 2024-09-21T10:54:05.413 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.413 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:54:05.414 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.414 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.414 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.414 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:130845 55.3 55:c4b635a2:::benchmark_data_smithi070_294394_object25504:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:54:05.414 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:26.363041+0000", 2024-09-21T10:54:05.414 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 158.11756274000001, 2024-09-21T10:54:05.414 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.266966327, 2024-09-21T10:54:05.414 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:54:05.414 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:54:05.414 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:54:05.414 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:54:05.414 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:54:05.414 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:54:05.415 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 130845 2024-09-21T10:54:05.415 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.415 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:54:05.415 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.415 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:54:05.415 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.363041+0000", 2024-09-21T10:54:05.415 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.415 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.418 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.418 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:54:05.418 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.363041+0000", 2024-09-21T10:54:05.418 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.418 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.418 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.418 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:54:05.418 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.363043+0000", 2024-09-21T10:54:05.419 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.53e-06 2024-09-21T10:54:05.419 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.419 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.419 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:54:05.419 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.363048+0000", 2024-09-21T10:54:05.419 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8369999999999996e-06 2024-09-21T10:54:05.419 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.419 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.419 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:54:05.419 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.363059+0000", 2024-09-21T10:54:05.419 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1355000000000001e-05 2024-09-21T10:54:05.419 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.419 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.419 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:54:05.420 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.363065+0000", 2024-09-21T10:54:05.420 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8080000000000001e-06 2024-09-21T10:54:05.420 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.420 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.420 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.420 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.366250+0000", 2024-09-21T10:54:05.420 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0031852289999999999 2024-09-21T10:54:05.420 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.420 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.420 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:54:05.420 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.375345+0000", 2024-09-21T10:54:05.420 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0090954079999999993 2024-09-21T10:54:05.420 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.421 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.421 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.421 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.396385+0000", 2024-09-21T10:54:05.421 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.021039869999999999 2024-09-21T10:54:05.421 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.421 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.421 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:54:05.421 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.396584+0000", 2024-09-21T10:54:05.421 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000198304 2024-09-21T10:54:05.421 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.421 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.421 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:54:05.421 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.396972+0000", 2024-09-21T10:54:05.421 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000388561 2024-09-21T10:54:05.422 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.422 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.422 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:54:05.422 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.413918+0000", 2024-09-21T10:54:05.422 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.016945913999999999 2024-09-21T10:54:05.422 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.422 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.422 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.422 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.418908+0000", 2024-09-21T10:54:05.422 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0049898119999999997 2024-09-21T10:54:05.422 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.422 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.422 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.422 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.455449+0000", 2024-09-21T10:54:05.423 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.036541487999999997 2024-09-21T10:54:05.423 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.423 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.423 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.423 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.455463+0000", 2024-09-21T10:54:05.423 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3332e-05 2024-09-21T10:54:05.423 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.423 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.423 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.423 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.429116+0000", 2024-09-21T10:54:05.423 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.97365345800000003 2024-09-21T10:54:05.423 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.423 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.423 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.424 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.429128+0000", 2024-09-21T10:54:05.424 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1681999999999999e-05 2024-09-21T10:54:05.424 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.424 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.424 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.424 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.629922+0000", 2024-09-21T10:54:05.424 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2007940420000001 2024-09-21T10:54:05.424 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.424 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.424 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:54:05.424 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.630008+0000", 2024-09-21T10:54:05.424 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5697000000000003e-05 2024-09-21T10:54:05.424 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.424 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:54:05.425 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.425 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.425 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.425 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:130863 55.3 55:c477e325:::benchmark_data_smithi070_294394_object25522:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:54:05.425 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:26.403411+0000", 2024-09-21T10:54:05.425 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 158.07719309300001, 2024-09-21T10:54:05.425 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4185171620000001, 2024-09-21T10:54:05.425 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:54:05.425 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:54:05.425 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:54:05.425 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:54:05.425 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:54:05.426 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:54:05.426 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 130863 2024-09-21T10:54:05.426 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.426 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:54:05.426 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.426 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:54:05.426 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.403411+0000", 2024-09-21T10:54:05.426 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.426 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.426 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.426 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:54:05.426 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.403411+0000", 2024-09-21T10:54:05.426 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.426 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.427 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.427 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:54:05.427 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.403413+0000", 2024-09-21T10:54:05.427 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3159999999999999e-06 2024-09-21T10:54:05.427 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.427 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.427 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:54:05.427 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.403421+0000", 2024-09-21T10:54:05.427 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7889999999999995e-06 2024-09-21T10:54:05.427 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.427 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.427 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:54:05.427 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.403437+0000", 2024-09-21T10:54:05.428 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5977e-05 2024-09-21T10:54:05.428 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.428 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.428 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:54:05.428 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.403446+0000", 2024-09-21T10:54:05.428 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6570000000000003e-06 2024-09-21T10:54:05.428 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.428 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.428 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.428 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.410837+0000", 2024-09-21T10:54:05.428 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0073913859999999998 2024-09-21T10:54:05.428 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.429 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.429 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:54:05.429 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.413854+0000", 2024-09-21T10:54:05.429 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0030171859999999998 2024-09-21T10:54:05.429 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.429 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.429 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.429 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.458342+0000", 2024-09-21T10:54:05.429 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.044487421999999999 2024-09-21T10:54:05.429 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.429 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.429 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.429 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.458353+0000", 2024-09-21T10:54:05.429 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1004e-05 2024-09-21T10:54:05.430 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.430 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.430 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.430 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.429161+0000", 2024-09-21T10:54:05.430 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.97080803299999996 2024-09-21T10:54:05.430 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.430 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.430 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.430 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.429169+0000", 2024-09-21T10:54:05.430 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2199999999999992e-06 2024-09-21T10:54:05.430 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.430 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.430 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.430 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.757697+0000", 2024-09-21T10:54:05.431 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.328527934 2024-09-21T10:54:05.431 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.431 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.431 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:54:05.431 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.758034+0000", 2024-09-21T10:54:05.431 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00033750499999999999 2024-09-21T10:54:05.431 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.431 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.431 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.431 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.778950+0000", 2024-09-21T10:54:05.431 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.020915321000000001 2024-09-21T10:54:05.431 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.431 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.431 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:54:05.432 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.779056+0000", 2024-09-21T10:54:05.432 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010603999999999999 2024-09-21T10:54:05.432 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.432 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.432 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,3,6", 2024-09-21T10:54:05.432 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.779524+0000", 2024-09-21T10:54:05.432 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00046834399999999999 2024-09-21T10:54:05.432 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.432 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.432 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.432 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.809552+0000", 2024-09-21T10:54:05.432 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.030028368999999999 2024-09-21T10:54:05.432 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.432 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.433 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:54:05.433 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.809565+0000", 2024-09-21T10:54:05.433 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2473000000000001e-05 2024-09-21T10:54:05.433 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.433 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.433 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.433 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.809594+0000", 2024-09-21T10:54:05.433 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9169000000000001e-05 2024-09-21T10:54:05.433 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.433 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.433 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.433 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.809627+0000", 2024-09-21T10:54:05.433 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2666999999999997e-05 2024-09-21T10:54:05.433 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.434 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.434 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:54:05.434 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.809881+0000", 2024-09-21T10:54:05.434 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00025474100000000001 2024-09-21T10:54:05.434 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.434 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.434 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:54:05.434 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.821928+0000", 2024-09-21T10:54:05.434 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012046609 2024-09-21T10:54:05.434 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.434 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:54:05.434 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.434 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.435 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.435 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:130868 55.3 55:c060ea53:::benchmark_data_smithi070_294394_object25527:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:54:05.435 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:26.414944+0000", 2024-09-21T10:54:05.435 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 158.065660189, 2024-09-21T10:54:05.435 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4176187740000001, 2024-09-21T10:54:05.435 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:54:05.435 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:54:05.435 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:54:05.435 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:54:05.435 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:54:05.435 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:54:05.435 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 130868 2024-09-21T10:54:05.435 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.435 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:54:05.436 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.436 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:54:05.436 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.414944+0000", 2024-09-21T10:54:05.436 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.436 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.436 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.436 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:54:05.436 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.414944+0000", 2024-09-21T10:54:05.436 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.436 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.436 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.436 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:54:05.436 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.414946+0000", 2024-09-21T10:54:05.436 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.821e-06 2024-09-21T10:54:05.437 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.437 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.437 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:54:05.437 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.414953+0000", 2024-09-21T10:54:05.437 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5800000000000003e-06 2024-09-21T10:54:05.437 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.437 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.437 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:54:05.437 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.414972+0000", 2024-09-21T10:54:05.437 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8644e-05 2024-09-21T10:54:05.437 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.437 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.437 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:54:05.437 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.414980+0000", 2024-09-21T10:54:05.438 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6009999999999995e-06 2024-09-21T10:54:05.438 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.438 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.438 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.438 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.415312+0000", 2024-09-21T10:54:05.438 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00033163099999999998 2024-09-21T10:54:05.438 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.438 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.438 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:54:05.438 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.418842+0000", 2024-09-21T10:54:05.438 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035301970000000001 2024-09-21T10:54:05.438 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.438 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.438 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.439 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.459012+0000", 2024-09-21T10:54:05.439 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.040169336 2024-09-21T10:54:05.439 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.439 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.439 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.439 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.459022+0000", 2024-09-21T10:54:05.439 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0879999999999999e-05 2024-09-21T10:54:05.439 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.439 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.439 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.439 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.429197+0000", 2024-09-21T10:54:05.439 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.97017467599999996 2024-09-21T10:54:05.439 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.439 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.440 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.440 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.429205+0000", 2024-09-21T10:54:05.440 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8800000000000008e-06 2024-09-21T10:54:05.440 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.440 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.440 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.440 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.758071+0000", 2024-09-21T10:54:05.440 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.328865768 2024-09-21T10:54:05.440 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.440 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.440 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:54:05.440 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.758459+0000", 2024-09-21T10:54:05.440 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000388551 2024-09-21T10:54:05.440 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.440 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.441 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.441 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.807195+0000", 2024-09-21T10:54:05.441 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.048735607 2024-09-21T10:54:05.441 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.441 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.441 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:54:05.441 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.807294+0000", 2024-09-21T10:54:05.441 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.8789000000000006e-05 2024-09-21T10:54:05.441 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.441 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.441 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,3,6", 2024-09-21T10:54:05.441 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.807447+0000", 2024-09-21T10:54:05.441 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015366699999999999 2024-09-21T10:54:05.442 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.442 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.442 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.442 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.821997+0000", 2024-09-21T10:54:05.442 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.014549805000000001 2024-09-21T10:54:05.442 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.442 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.442 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:54:05.442 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.822015+0000", 2024-09-21T10:54:05.442 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7459e-05 2024-09-21T10:54:05.442 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.442 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.442 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.442 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.822067+0000", 2024-09-21T10:54:05.443 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1848000000000002e-05 2024-09-21T10:54:05.443 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.443 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.443 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.443 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.826902+0000", 2024-09-21T10:54:05.443 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0048356809999999997 2024-09-21T10:54:05.443 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.443 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.443 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:54:05.443 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.827095+0000", 2024-09-21T10:54:05.443 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00019322100000000001 2024-09-21T10:54:05.443 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.443 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.443 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:54:05.444 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.832563+0000", 2024-09-21T10:54:05.444 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0054671320000000004 2024-09-21T10:54:05.444 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.444 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:54:05.444 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.444 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.444 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.444 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:156262 55.3 55:c72c6ba0:::benchmark_data_smithi070_294394_object50921:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1464)", 2024-09-21T10:54:05.444 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.317232+0000", 2024-09-21T10:54:05.444 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 100.163372399, 2024-09-21T10:54:05.444 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.319640197, 2024-09-21T10:54:05.444 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:54:05.444 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:54:05.444 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:54:05.445 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:54:05.445 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:54:05.445 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:54:05.445 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 156262 2024-09-21T10:54:05.445 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.445 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:54:05.445 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.445 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:54:05.445 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.317232+0000", 2024-09-21T10:54:05.445 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.445 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.445 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.445 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:54:05.445 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.317232+0000", 2024-09-21T10:54:05.446 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.446 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.446 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.446 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:54:05.446 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.317233+0000", 2024-09-21T10:54:05.446 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.84e-06 2024-09-21T10:54:05.446 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.446 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.446 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:54:05.446 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.317240+0000", 2024-09-21T10:54:05.446 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2790000000000004e-06 2024-09-21T10:54:05.446 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.446 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.447 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:54:05.447 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.317253+0000", 2024-09-21T10:54:05.447 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.378e-05 2024-09-21T10:54:05.447 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.447 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.447 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:54:05.447 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.317261+0000", 2024-09-21T10:54:05.447 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7400000000000004e-06 2024-09-21T10:54:05.447 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.447 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.447 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.447 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.317343+0000", 2024-09-21T10:54:05.447 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1835000000000005e-05 2024-09-21T10:54:05.447 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.448 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.448 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:54:05.448 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.321143+0000", 2024-09-21T10:54:05.448 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0038001250000000001 2024-09-21T10:54:05.448 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.448 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.448 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.448 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.343392+0000", 2024-09-21T10:54:05.448 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.022248375000000001 2024-09-21T10:54:05.448 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.448 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.448 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:54:05.448 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.343496+0000", 2024-09-21T10:54:05.448 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000104279 2024-09-21T10:54:05.449 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.449 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.449 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,3,6", 2024-09-21T10:54:05.449 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.343661+0000", 2024-09-21T10:54:05.449 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016517699999999999 2024-09-21T10:54:05.449 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.449 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.449 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.449 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.355131+0000", 2024-09-21T10:54:05.449 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.011469791 2024-09-21T10:54:05.449 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.449 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.449 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.449 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.355158+0000", 2024-09-21T10:54:05.449 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6951e-05 2024-09-21T10:54:05.450 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.450 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.450 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:54:05.450 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.356176+0000", 2024-09-21T10:54:05.450 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0010179410000000001 2024-09-21T10:54:05.450 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.450 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.450 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.450 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.472799+0000", 2024-09-21T10:54:05.450 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.116623622 2024-09-21T10:54:05.450 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.450 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.450 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.451 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.472815+0000", 2024-09-21T10:54:05.451 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6058000000000001e-05 2024-09-21T10:54:05.451 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.451 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.451 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.451 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.467557+0000", 2024-09-21T10:54:05.451 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99474126100000004 2024-09-21T10:54:05.451 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.451 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.451 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.451 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.467588+0000", 2024-09-21T10:54:05.451 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1421999999999998e-05 2024-09-21T10:54:05.451 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.451 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.451 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.452 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.627831+0000", 2024-09-21T10:54:05.452 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.160243092 2024-09-21T10:54:05.452 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.452 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.452 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for missing object", 2024-09-21T10:54:05.452 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.627985+0000", 2024-09-21T10:54:05.452 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000154327 2024-09-21T10:54:05.452 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.452 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.452 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.452 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.636715+0000", 2024-09-21T10:54:05.452 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0087296089999999993 2024-09-21T10:54:05.452 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.452 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.453 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:54:05.453 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.636872+0000", 2024-09-21T10:54:05.453 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015669300000000001 2024-09-21T10:54:05.453 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.453 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:54:05.453 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.453 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.453 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.453 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:156265 55.3 55:c8ae1fd0:::benchmark_data_smithi070_294394_object50924:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1464)", 2024-09-21T10:54:05.453 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.325973+0000", 2024-09-21T10:54:05.453 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 100.15463088, 2024-09-21T10:54:05.453 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3149953180000002, 2024-09-21T10:54:05.453 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:54:05.453 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:54:05.454 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:54:05.454 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:54:05.454 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:54:05.454 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:54:05.454 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 156265 2024-09-21T10:54:05.454 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.454 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:54:05.454 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.454 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:54:05.454 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.325973+0000", 2024-09-21T10:54:05.454 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.454 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.454 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.454 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:54:05.455 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.325973+0000", 2024-09-21T10:54:05.455 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.455 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.455 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.455 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:54:05.455 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.325975+0000", 2024-09-21T10:54:05.455 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8470000000000001e-06 2024-09-21T10:54:05.455 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.455 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.455 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:54:05.455 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.325980+0000", 2024-09-21T10:54:05.455 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.242e-06 2024-09-21T10:54:05.455 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.456 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.456 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:54:05.456 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.325995+0000", 2024-09-21T10:54:05.456 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5055000000000001e-05 2024-09-21T10:54:05.456 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.456 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.456 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:54:05.456 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.326002+0000", 2024-09-21T10:54:05.456 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0280000000000003e-06 2024-09-21T10:54:05.456 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.456 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.456 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.456 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.326145+0000", 2024-09-21T10:54:05.456 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014283199999999999 2024-09-21T10:54:05.457 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.457 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.457 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:54:05.457 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.327892+0000", 2024-09-21T10:54:05.457 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001747352 2024-09-21T10:54:05.457 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.457 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.457 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.457 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.351485+0000", 2024-09-21T10:54:05.457 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.023592318000000001 2024-09-21T10:54:05.457 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.457 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.457 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:54:05.457 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.351567+0000", 2024-09-21T10:54:05.458 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2572999999999997e-05 2024-09-21T10:54:05.458 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.458 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.458 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,3,6", 2024-09-21T10:54:05.458 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.351690+0000", 2024-09-21T10:54:05.458 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012269599999999999 2024-09-21T10:54:05.458 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.458 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.458 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.458 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.355179+0000", 2024-09-21T10:54:05.458 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0034887949999999998 2024-09-21T10:54:05.458 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.458 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.459 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:54:05.459 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.356185+0000", 2024-09-21T10:54:05.459 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0010059999999999999 2024-09-21T10:54:05.459 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.459 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.459 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.459 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.358107+0000", 2024-09-21T10:54:05.459 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001921989 2024-09-21T10:54:05.459 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.459 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.459 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.459 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.473066+0000", 2024-09-21T10:54:05.459 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.114959478 2024-09-21T10:54:05.459 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.460 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.460 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.460 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.473076+0000", 2024-09-21T10:54:05.460 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3810000000000005e-06 2024-09-21T10:54:05.460 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.460 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.460 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.460 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.469705+0000", 2024-09-21T10:54:05.460 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99662927800000001 2024-09-21T10:54:05.460 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.460 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.460 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.460 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.469716+0000", 2024-09-21T10:54:05.460 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0906e-05 2024-09-21T10:54:05.461 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.461 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.461 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.461 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.629933+0000", 2024-09-21T10:54:05.461 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1602172040000001 2024-09-21T10:54:05.461 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.461 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.461 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for missing object", 2024-09-21T10:54:05.461 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.630738+0000", 2024-09-21T10:54:05.461 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00080527400000000003 2024-09-21T10:54:05.461 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.461 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.461 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.461 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.640869+0000", 2024-09-21T10:54:05.462 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.01013057 2024-09-21T10:54:05.462 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.462 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.462 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:54:05.462 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.640968+0000", 2024-09-21T10:54:05.462 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9500000000000006e-05 2024-09-21T10:54:05.462 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.462 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:54:05.462 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.462 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.462 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.462 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:156289 55.3 55:cca4d5e4:::benchmark_data_smithi070_294394_object50948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1464)", 2024-09-21T10:54:05.462 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.379748+0000", 2024-09-21T10:54:05.463 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 100.100855861, 2024-09-21T10:54:05.463 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2985970020000002, 2024-09-21T10:54:05.463 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:54:05.463 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:54:05.463 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:54:05.463 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:54:05.463 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:54:05.463 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:54:05.463 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 156289 2024-09-21T10:54:05.463 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.463 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:54:05.463 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.463 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:54:05.463 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.379748+0000", 2024-09-21T10:54:05.464 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.464 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.464 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.464 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:54:05.464 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.379748+0000", 2024-09-21T10:54:05.464 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.464 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.464 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.464 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:54:05.464 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.379750+0000", 2024-09-21T10:54:05.464 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.922e-06 2024-09-21T10:54:05.464 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.464 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.464 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:54:05.465 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.379756+0000", 2024-09-21T10:54:05.465 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9739999999999999e-06 2024-09-21T10:54:05.465 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.465 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.465 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:54:05.465 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.379773+0000", 2024-09-21T10:54:05.465 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7363e-05 2024-09-21T10:54:05.465 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.465 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.465 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:54:05.465 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.379781+0000", 2024-09-21T10:54:05.465 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7959999999999996e-06 2024-09-21T10:54:05.465 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.465 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.466 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.466 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.379846+0000", 2024-09-21T10:54:05.466 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4603000000000005e-05 2024-09-21T10:54:05.466 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.466 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.466 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:54:05.466 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.380542+0000", 2024-09-21T10:54:05.466 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00069589599999999999 2024-09-21T10:54:05.466 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.466 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.466 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.466 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.473103+0000", 2024-09-21T10:54:05.466 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.092561056000000003 2024-09-21T10:54:05.466 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.467 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.467 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.467 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.473109+0000", 2024-09-21T10:54:05.467 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1829999999999999e-06 2024-09-21T10:54:05.467 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.467 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.467 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.467 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.469746+0000", 2024-09-21T10:54:05.467 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99663669200000005 2024-09-21T10:54:05.467 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.467 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.467 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.467 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.469755+0000", 2024-09-21T10:54:05.467 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5699999999999999e-06 2024-09-21T10:54:05.468 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.468 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.468 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.468 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.630804+0000", 2024-09-21T10:54:05.468 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1610483920000001 2024-09-21T10:54:05.468 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.468 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.468 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:54:05.468 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.631711+0000", 2024-09-21T10:54:05.468 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00090767300000000001 2024-09-21T10:54:05.468 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.468 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.468 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.468 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.646421+0000", 2024-09-21T10:54:05.469 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.014710246999999999 2024-09-21T10:54:05.469 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.469 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.469 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:54:05.469 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.646528+0000", 2024-09-21T10:54:05.469 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010703400000000001 2024-09-21T10:54:05.469 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.469 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.469 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,3,7", 2024-09-21T10:54:05.469 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.646672+0000", 2024-09-21T10:54:05.469 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014363599999999999 2024-09-21T10:54:05.469 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.469 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.469 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.470 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.659293+0000", 2024-09-21T10:54:05.470 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012620726000000001 2024-09-21T10:54:05.470 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.470 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.470 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.470 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.659342+0000", 2024-09-21T10:54:05.470 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9116000000000002e-05 2024-09-21T10:54:05.470 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.470 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.470 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.470 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.670514+0000", 2024-09-21T10:54:05.470 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.011172100000000001 2024-09-21T10:54:05.470 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.471 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.471 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:54:05.471 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.676561+0000", 2024-09-21T10:54:05.471 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.006047251 2024-09-21T10:54:05.471 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.471 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.471 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:54:05.471 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.676602+0000", 2024-09-21T10:54:05.471 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0967999999999998e-05 2024-09-21T10:54:05.471 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.471 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.471 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:54:05.471 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.678345+0000", 2024-09-21T10:54:05.471 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0017428039999999999 2024-09-21T10:54:05.471 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.472 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:54:05.472 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.472 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.472 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.472 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:156293 55.3 55:c7ab3785:::benchmark_data_smithi070_294394_object50952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1464)", 2024-09-21T10:54:05.472 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.387365+0000", 2024-09-21T10:54:05.472 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 100.093239213, 2024-09-21T10:54:05.472 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2933115700000002, 2024-09-21T10:54:05.472 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:54:05.472 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:54:05.472 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:54:05.472 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:54:05.472 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:54:05.472 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:54:05.473 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 156293 2024-09-21T10:54:05.473 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.473 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:54:05.473 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.473 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:54:05.473 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.387365+0000", 2024-09-21T10:54:05.473 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.473 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.473 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.473 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:54:05.473 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.387365+0000", 2024-09-21T10:54:05.473 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.473 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.473 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.474 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:54:05.474 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.387366+0000", 2024-09-21T10:54:05.474 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.457e-06 2024-09-21T10:54:05.474 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.474 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.474 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:54:05.474 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.387379+0000", 2024-09-21T10:54:05.474 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3186999999999999e-05 2024-09-21T10:54:05.474 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.474 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.474 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:54:05.474 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.387390+0000", 2024-09-21T10:54:05.474 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0549e-05 2024-09-21T10:54:05.474 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.475 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.475 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:54:05.475 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.387398+0000", 2024-09-21T10:54:05.475 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0450000000000001e-06 2024-09-21T10:54:05.475 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.475 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.475 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.475 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.387455+0000", 2024-09-21T10:54:05.475 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6753999999999999e-05 2024-09-21T10:54:05.476 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.476 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.476 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:54:05.476 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.388264+0000", 2024-09-21T10:54:05.476 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00080899100000000001 2024-09-21T10:54:05.476 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.476 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.476 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.476 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.473515+0000", 2024-09-21T10:54:05.476 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.085251407000000001 2024-09-21T10:54:05.477 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.477 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.477 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.477 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.473524+0000", 2024-09-21T10:54:05.477 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9970000000000004e-06 2024-09-21T10:54:05.477 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.477 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.477 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.477 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.469787+0000", 2024-09-21T10:54:05.477 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99626246799999996 2024-09-21T10:54:05.477 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.478 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.478 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.478 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.469794+0000", 2024-09-21T10:54:05.478 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.272e-06 2024-09-21T10:54:05.478 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.478 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.478 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.478 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.651119+0000", 2024-09-21T10:54:05.478 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1813249480000001 2024-09-21T10:54:05.478 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.478 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.478 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:54:05.478 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.651533+0000", 2024-09-21T10:54:05.478 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00041398299999999998 2024-09-21T10:54:05.479 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.479 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.479 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.479 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.663514+0000", 2024-09-21T10:54:05.479 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.011981466 2024-09-21T10:54:05.479 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.479 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.479 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:54:05.479 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.663604+0000", 2024-09-21T10:54:05.479 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9920000000000006e-05 2024-09-21T10:54:05.479 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.479 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.479 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,3,7", 2024-09-21T10:54:05.479 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.663726+0000", 2024-09-21T10:54:05.480 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012202899999999999 2024-09-21T10:54:05.480 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.480 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.480 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.480 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.670454+0000", 2024-09-21T10:54:05.480 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0067277719999999999 2024-09-21T10:54:05.480 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.480 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.480 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:54:05.480 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.679853+0000", 2024-09-21T10:54:05.480 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0093995080000000009 2024-09-21T10:54:05.480 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.481 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.481 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.481 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.679896+0000", 2024-09-21T10:54:05.481 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2920999999999998e-05 2024-09-21T10:54:05.481 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.481 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.481 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.481 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.679929+0000", 2024-09-21T10:54:05.481 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3009000000000002e-05 2024-09-21T10:54:05.481 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.481 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.481 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:54:05.481 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.679964+0000", 2024-09-21T10:54:05.481 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5077999999999999e-05 2024-09-21T10:54:05.482 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.482 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.482 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:54:05.482 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.680676+0000", 2024-09-21T10:54:05.482 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00071180899999999999 2024-09-21T10:54:05.482 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.482 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:54:05.482 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.482 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.482 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.482 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:156301 55.3 55:cbe6e631:::benchmark_data_smithi070_294394_object50960:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1465)", 2024-09-21T10:54:05.482 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.500014+0000", 2024-09-21T10:54:05.482 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 99.980590340000006, 2024-09-21T10:54:05.482 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2107678850000001, 2024-09-21T10:54:05.483 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:54:05.483 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:54:05.483 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:54:05.483 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:54:05.483 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:54:05.483 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:54:05.483 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 156301 2024-09-21T10:54:05.483 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.483 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:54:05.483 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.483 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:54:05.483 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.500014+0000", 2024-09-21T10:54:05.483 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.484 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.484 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.484 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:54:05.484 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.500014+0000", 2024-09-21T10:54:05.484 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.484 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.484 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.484 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:54:05.484 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.500016+0000", 2024-09-21T10:54:05.484 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.751e-06 2024-09-21T10:54:05.484 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.484 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.484 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:54:05.484 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.500026+0000", 2024-09-21T10:54:05.485 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4879999999999996e-06 2024-09-21T10:54:05.485 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.485 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.485 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:54:05.485 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.500048+0000", 2024-09-21T10:54:05.485 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2322000000000001e-05 2024-09-21T10:54:05.485 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.485 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.485 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:54:05.485 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.500065+0000", 2024-09-21T10:54:05.485 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6716000000000001e-05 2024-09-21T10:54:05.485 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.485 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.485 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.486 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.500146+0000", 2024-09-21T10:54:05.486 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1501000000000005e-05 2024-09-21T10:54:05.486 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.486 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.486 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.486 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.500158+0000", 2024-09-21T10:54:05.486 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1688000000000001e-05 2024-09-21T10:54:05.486 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.486 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.486 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.486 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.469817+0000", 2024-09-21T10:54:05.486 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.96965923899999995 2024-09-21T10:54:05.486 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.487 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.487 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.487 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.469823+0000", 2024-09-21T10:54:05.487 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4999999999999999e-06 2024-09-21T10:54:05.487 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.487 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.487 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.487 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.651577+0000", 2024-09-21T10:54:05.487 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.181754054 2024-09-21T10:54:05.487 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.487 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.487 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:54:05.487 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.654453+0000", 2024-09-21T10:54:05.487 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0028760859999999999 2024-09-21T10:54:05.488 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.488 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.488 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.488 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.673839+0000", 2024-09-21T10:54:05.488 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.019386417999999999 2024-09-21T10:54:05.488 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.488 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.488 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:54:05.488 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.673952+0000", 2024-09-21T10:54:05.488 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000112693 2024-09-21T10:54:05.488 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.488 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.488 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,3,7", 2024-09-21T10:54:05.488 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.674052+0000", 2024-09-21T10:54:05.489 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9528e-05 2024-09-21T10:54:05.489 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.489 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.489 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:54:05.489 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.679862+0000", 2024-09-21T10:54:05.489 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0058103829999999997 2024-09-21T10:54:05.489 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.489 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.489 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.489 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.680729+0000", 2024-09-21T10:54:05.489 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00086661599999999996 2024-09-21T10:54:05.489 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.489 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.490 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.490 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.680786+0000", 2024-09-21T10:54:05.490 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7385e-05 2024-09-21T10:54:05.490 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.490 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.490 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.490 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.708516+0000", 2024-09-21T10:54:05.490 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.027730138000000001 2024-09-21T10:54:05.490 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.490 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.490 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:54:05.490 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.708587+0000", 2024-09-21T10:54:05.490 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0601999999999999e-05 2024-09-21T10:54:05.490 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.491 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.491 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:54:05.491 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.710781+0000", 2024-09-21T10:54:05.491 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0021947770000000002 2024-09-21T10:54:05.491 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.491 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:54:05.491 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.491 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.491 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.491 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:156302 55.3 55:c0dc2955:::benchmark_data_smithi070_294394_object50961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1465)", 2024-09-21T10:54:05.491 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.501061+0000", 2024-09-21T10:54:05.491 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 99.979543340000006, 2024-09-21T10:54:05.491 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2119707480000002, 2024-09-21T10:54:05.491 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:54:05.492 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:54:05.492 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:54:05.492 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:54:05.492 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:54:05.492 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:54:05.492 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 156302 2024-09-21T10:54:05.492 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.492 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:54:05.492 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.492 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:54:05.492 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.501061+0000", 2024-09-21T10:54:05.492 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.492 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.492 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.493 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:54:05.493 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.501061+0000", 2024-09-21T10:54:05.493 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:54:05.493 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.493 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.493 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:54:05.493 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.501062+0000", 2024-09-21T10:54:05.493 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6020000000000001e-06 2024-09-21T10:54:05.493 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.493 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.493 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:54:05.493 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.501069+0000", 2024-09-21T10:54:05.493 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6889999999999998e-06 2024-09-21T10:54:05.493 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.494 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.494 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:54:05.494 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.501088+0000", 2024-09-21T10:54:05.494 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9321999999999999e-05 2024-09-21T10:54:05.494 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.494 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.494 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:54:05.494 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.501099+0000", 2024-09-21T10:54:05.494 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.075e-05 2024-09-21T10:54:05.494 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.494 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.494 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.494 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.501169+0000", 2024-09-21T10:54:05.494 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9797999999999998e-05 2024-09-21T10:54:05.495 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.495 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.495 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.495 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.501177+0000", 2024-09-21T10:54:05.495 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.568e-06 2024-09-21T10:54:05.495 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.495 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.495 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.495 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.469845+0000", 2024-09-21T10:54:05.495 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.96866734499999996 2024-09-21T10:54:05.495 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.495 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.495 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:54:05.496 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.469851+0000", 2024-09-21T10:54:05.496 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1619999999999996e-06 2024-09-21T10:54:05.496 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.496 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.496 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.496 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.656700+0000", 2024-09-21T10:54:05.496 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1868494380000001 2024-09-21T10:54:05.496 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.496 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.496 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:54:05.496 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.659223+0000", 2024-09-21T10:54:05.496 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025225790000000001 2024-09-21T10:54:05.496 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.496 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.496 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:54:05.497 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.680821+0000", 2024-09-21T10:54:05.497 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.021597813 2024-09-21T10:54:05.497 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.497 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.497 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:54:05.497 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.680881+0000", 2024-09-21T10:54:05.497 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0291999999999998e-05 2024-09-21T10:54:05.497 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.497 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.497 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,3,7", 2024-09-21T10:54:05.497 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.680964+0000", 2024-09-21T10:54:05.497 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3250000000000004e-05 2024-09-21T10:54:05.497 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.498 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.498 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.498 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.684121+0000", 2024-09-21T10:54:05.498 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0031563950000000002 2024-09-21T10:54:05.498 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.498 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.498 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:54:05.498 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.684139+0000", 2024-09-21T10:54:05.498 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8797999999999999e-05 2024-09-21T10:54:05.498 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.498 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.498 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.498 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.684744+0000", 2024-09-21T10:54:05.498 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00060464399999999999 2024-09-21T10:54:05.499 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.499 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.499 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:54:05.499 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.710836+0000", 2024-09-21T10:54:05.499 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.026091849 2024-09-21T10:54:05.499 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.499 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.499 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:54:05.499 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.710897+0000", 2024-09-21T10:54:05.499 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1260000000000006e-05 2024-09-21T10:54:05.499 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:54:05.499 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:54:05.499 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:54:05.499 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.713031+0000", 2024-09-21T10:54:05.500 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0021341939999999998 2024-09-21T10:54:05.500 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.500 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:54:05.500 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.500 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:54:05.500 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:54:05.500 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:54:05.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:05.501 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:05.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:05.602 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:05.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:54:05.628 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:05.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:54:05.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:05.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:05.729 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:05.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:05.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:54:05.853 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:05.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:05.875 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:05.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:05.977 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:05.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:05.999 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:05.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:06.100 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:06.115 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:54:06.176 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:06.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:06.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:06.277 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:06.299 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:06.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:06.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:06.401 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:06.424 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:06.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:06.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:06.526 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:06.551 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:06.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:06.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:06.652 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:06.664 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:06 smithi070 ceph-mon[107385]: pgmap v2739: 83 pgs: 83 active+clean; 425 KiB data, 4.3 GiB used, 443 GiB / 447 GiB avail 2024-09-21T10:54:06.664 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:54:06.664 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:54:06.664 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:06 smithi070 ceph-mon[112139]: pgmap v2739: 83 pgs: 83 active+clean; 425 KiB data, 4.3 GiB used, 443 GiB / 447 GiB avail 2024-09-21T10:54:06.664 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:54:06.665 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:54:06.677 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:06.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:06.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:06.779 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:06.801 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:06.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:06.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:06 smithi160 ceph-mon[101879]: pgmap v2739: 83 pgs: 83 active+clean; 425 KiB data, 4.3 GiB used, 443 GiB / 447 GiB avail 2024-09-21T10:54:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:54:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:54:06.902 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:06.924 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:06.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:06.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:07.025 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:07.048 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:07.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:07.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:07.149 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:07.171 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:07.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:07.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:07.273 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:07.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:07.294 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:07.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:07.395 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:07.417 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:07.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:07.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:07.519 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:07.541 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:07.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:07.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:07.643 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:07.666 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:07.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:07.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:07.768 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:07.791 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:07.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:07.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:07.870 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:54:07.870 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:54:07.880 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:54:07.891 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:54:07.892 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:54:07.893 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:07.917 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:54:07.917 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:54:07.919 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:07.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:07.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:07.922 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:54:07.939 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:54:07.939 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:54:07.955 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:54:07.956 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:54:07.973 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:54:07.973 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:54:07.989 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:54:07.989 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:54:08.005 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:54:08.005 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:54:08.021 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:08.042 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:08.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:08.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:08.144 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:08.166 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:08.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:08.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:08.268 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:08.289 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:08.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:08.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:08.391 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:08.412 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:08.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:08.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:08.507 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:08 smithi070 ceph-mon[107385]: pgmap v2740: 83 pgs: 83 active+clean; 425 KiB data, 4.3 GiB used, 443 GiB / 447 GiB avail 2024-09-21T10:54:08.508 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:08 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3750984123' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:54:08.508 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:08 smithi070 ceph-mon[112139]: pgmap v2740: 83 pgs: 83 active+clean; 425 KiB data, 4.3 GiB used, 443 GiB / 447 GiB avail 2024-09-21T10:54:08.508 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:08 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3750984123' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:54:08.513 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:08.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:08.535 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:08.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:08.636 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:08.658 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:08.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:08.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:08.747 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:08 smithi160 ceph-mon[101879]: pgmap v2740: 83 pgs: 83 active+clean; 425 KiB data, 4.3 GiB used, 443 GiB / 447 GiB avail 2024-09-21T10:54:08.747 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:08 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3750984123' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:54:08.759 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:08.781 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:08.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:08.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:08.882 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:08.904 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:08.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:08.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:09.005 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:09.027 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:09.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:09.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:09.128 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:09.150 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:09.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:09.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:09.252 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:09.273 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:09.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:09.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:09.375 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:09.397 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:09.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:09.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:09.499 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:09.521 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:09.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:09.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:09.622 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:09.644 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:09.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:09.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:09.746 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:09.767 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:09.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:09.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:09.870 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:09.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:09.892 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:09.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:09.993 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:10.013 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:10.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:10.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:10.115 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:10.136 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:10.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:10.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:10.237 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:10.258 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:10.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:10.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:10.360 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:10.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:10.382 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:10.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:10.448 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'unique_pool_3' created 2024-09-21T10:54:10.482 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:10.504 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:10.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:10.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:10.595 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:10 smithi070 ceph-mon[107385]: pgmap v2741: 83 pgs: 83 active+clean; 425 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-21T10:54:10.596 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:10 smithi070 ceph-mon[112139]: pgmap v2741: 83 pgs: 83 active+clean; 425 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-21T10:54:10.606 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:10.628 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:10.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:10.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:10.729 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:10.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:10.751 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:10.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:10.785 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:54:10.786 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T10:54:10.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:10 smithi160 ceph-mon[101879]: pgmap v2741: 83 pgs: 83 active+clean; 425 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-21T10:54:10.852 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:10.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:10.874 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:10.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:10.975 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:10.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:10.999 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:11.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:11.101 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:11.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:11.124 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:11.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:11.225 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:11.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:11.257 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:11.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:11.358 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:11.385 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:11.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:11.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:11.487 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:11.494 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:11 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/247505441' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T10:54:11.494 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:11 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T10:54:11.494 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:11 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-21T10:54:11.494 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:11 smithi070 ceph-mon[107385]: osdmap e1484: 8 total, 8 up, 5 in 2024-09-21T10:54:11.494 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:11 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:54:11.494 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:11 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:54:11.494 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:11 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/247505441' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T10:54:11.495 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:11 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T10:54:11.495 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:11 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-21T10:54:11.495 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:11 smithi070 ceph-mon[112139]: osdmap e1484: 8 total, 8 up, 5 in 2024-09-21T10:54:11.495 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:11 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:54:11.495 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:11 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:54:11.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:11.511 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:11.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:11.514 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":2742,"stamp":"2024-09-21T10:54:10.443722+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":434932,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":276,"num_read_kb":598,"num_write":221,"num_write_kb":1790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":3994290,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1698,"ondisk_log_size":1698,"up":249,"acting":249,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":250,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":3505928,"kb_used_data":68048,"kb_used_omap":141,"kb_used_meta":3437682,"kb_avail":465219832,"statfs":{"total":479975178240,"available":476385107968,"internally_reserved":0,"allocated":69681152,"data_stored":44352402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":144912,"internal_metadata":3520186864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-268,"num_objects_misplaced":-2879,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-210678,"num_write_kb":-6741633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-55424,"num_bytes_recovered":-2985033728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.496058"},"pg_stats":[{"pgid":"56.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T10:54:10.441789+0000","last_change":"2024-09-21T10:54:10.441789+0000","last_active":"2024-09-21T10:54:10.441789+0000","last_peered":"2024-09-21T10:54:10.441789+0000","last_clean":"2024-09-21T10:54:10.441789+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:54:10.441789+0000","last_undegraded":"2024-09-21T10:54:10.441789+0000","last_fullsized":"2024-09-21T10:54:10.441789+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T10:54:10.441789+0000","last_change":"2024-09-21T10:54:10.441789+0000","last_active":"2024-09-21T10:54:10.441789+0000","last_peered":"2024-09-21T10:54:10.441789+0000","last_clean":"2024-09-21T10:54:10.441789+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:54:10.441789+0000","last_undegraded":"2024-09-21T10:54:10.441789+0000","last_fullsized":"2024-09-21T10:54:10.441789+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T10:54:10.441789+0000","last_change":"2024-09-21T10:54:10.441789+0000","last_active":"2024-09-21T10:54:10.441789+0000","last_peered":"2024-09-21T10:54:10.441789+0000","last_clean":"2024-09-21T10:54:10.441789+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:54:10.441789+0000","last_undegraded":"2024-09-21T10:54:10.441789+0000","last_fullsized":"2024-09-21T10:54:10.441789+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T10:54:10.441789+0000","last_change":"2024-09-21T10:54:10.441789+0000","last_active":"2024-09-21T10:54:10.441789+0000","last_peered":"2024-09-21T10:54:10.441789+0000","last_clean":"2024-09-21T10:54:10.441789+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:54:10.441789+0000","last_undegraded":"2024-09-21T10:54:10.441789+0000","last_fullsized":"2024-09-21T10:54:10.441789+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T10:54:10.441789+0000","last_change":"2024-09-21T10:54:10.441789+0000","last_active":"2024-09-21T10:54:10.441789+0000","last_peered":"2024-09-21T10:54:10.441789+0000","last_clean":"2024-09-21T10:54:10.441789+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:54:10.441789+0000","last_undegraded":"2024-09-21T10:54:10.441789+0000","last_fullsized":"2024-09-21T10:54:10.441789+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T10:54:10.441789+0000","last_change":"2024-09-21T10:54:10.441789+0000","last_active":"2024-09-21T10:54:10.441789+0000","last_peered":"2024-09-21T10:54:10.441789+0000","last_clean":"2024-09-21T10:54:10.441789+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:54:10.441789+0000","last_undegraded":"2024-09-21T10:54:10.441789+0000","last_fullsized":"2024-09-21T10:54:10.441789+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T10:54:10.441789+0000","last_change":"2024-09-21T10:54:10.441789+0000","last_active":"2024-09-21T10:54:10.441789+0000","last_peered":"2024-09-21T10:54:10.441789+0000","last_clean":"2024-09-21T10:54:10.441789+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:54:10.441789+0000","last_undegraded":"2024-09-21T10:54:10.441789+0000","last_fullsized":"2024-09-21T10:54:10.441789+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T10:54:10.441789+0000","last_change":"2024-09-21T10:54:10.441789+0000","last_active":"2024-09-21T10:54:10.441789+0000","last_peered":"2024-09-21T10:54:10.441789+0000","last_clean":"2024-09-21T10:54:10.441789+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:54:10.441789+0000","last_undegraded":"2024-09-21T10:54:10.441789+0000","last_fullsized":"2024-09-21T10:54:10.441789+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T10:54:10.441789+0000","last_change":"2024-09-21T10:54:10.441789+0000","last_active":"2024-09-21T10:54:10.441789+0000","last_peered":"2024-09-21T10:54:10.441789+0000","last_clean":"2024-09-21T10:54:10.441789+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:54:10.441789+0000","last_undegraded":"2024-09-21T10:54:10.441789+0000","last_fullsized":"2024-09-21T10:54:10.441789+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T10:54:10.441789+0000","last_change":"2024-09-21T10:54:10.441789+0000","last_active":"2024-09-21T10:54:10.441789+0000","last_peered":"2024-09-21T10:54:10.441789+0000","last_clean":"2024-09-21T10:54:10.441789+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:54:10.441789+0000","last_undegraded":"2024-09-21T10:54:10.441789+0000","last_fullsized":"2024-09-21T10:54:10.441789+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T10:54:10.441789+0000","last_change":"2024-09-21T10:54:10.441789+0000","last_active":"2024-09-21T10:54:10.441789+0000","last_peered":"2024-09-21T10:54:10.441789+0000","last_clean":"2024-09-21T10:54:10.441789+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:54:10.441789+0000","last_undegraded":"2024-09-21T10:54:10.441789+0000","last_fullsized":"2024-09-21T10:54:10.441789+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T10:54:10.441789+0000","last_change":"2024-09-21T10:54:10.441789+0000","last_active":"2024-09-21T10:54:10.441789+0000","last_peered":"2024-09-21T10:54:10.441789+0000","last_clean":"2024-09-21T10:54:10.441789+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:54:10.441789+0000","last_undegraded":"2024-09-21T10:54:10.441789+0000","last_fullsized":"2024-09-21T10:54:10.441789+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T10:54:10.441789+0000","last_change":"2024-09-21T10:54:10.441789+0000","last_active":"2024-09-21T10:54:10.441789+0000","last_peered":"2024-09-21T10:54:10.441789+0000","last_clean":"2024-09-21T10:54:10.441789+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:54:10.441789+0000","last_undegraded":"2024-09-21T10:54:10.441789+0000","last_fullsized":"2024-09-21T10:54:10.441789+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T10:54:10.441789+0000","last_change":"2024-09-21T10:54:10.441789+0000","last_active":"2024-09-21T10:54:10.441789+0000","last_peered":"2024-09-21T10:54:10.441789+0000","last_clean":"2024-09-21T10:54:10.441789+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:54:10.441789+0000","last_undegraded":"2024-09-21T10:54:10.441789+0000","last_fullsized":"2024-09-21T10:54:10.441789+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T10:54:10.441789+0000","last_change":"2024-09-21T10:54:10.441789+0000","last_active":"2024-09-21T10:54:10.441789+0000","last_peered":"2024-09-21T10:54:10.441789+0000","last_clean":"2024-09-21T10:54:10.441789+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:54:10.441789+0000","last_undegraded":"2024-09-21T10:54:10.441789+0000","last_fullsized":"2024-09-21T10:54:10.441789+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T10:54:10.441789+0000","last_change":"2024-09-21T10:54:10.441789+0000","last_active":"2024-09-21T10:54:10.441789+0000","last_peered":"2024-09-21T10:54:10.441789+0000","last_clean":"2024-09-21T10:54:10.441789+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:54:10.441789+0000","last_undegraded":"2024-09-21T10:54:10.441789+0000","last_fullsized":"2024-09-21T10:54:10.441789+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":1974,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.093179+0000","last_change":"2024-09-21T10:52:03.323899+0000","last_active":"2024-09-21T10:53:58.093179+0000","last_peered":"2024-09-21T10:53:58.093179+0000","last_clean":"2024-09-21T10:53:58.093179+0000","last_became_active":"2024-09-21T10:52:03.314626+0000","last_became_peered":"2024-09-21T10:52:03.314626+0000","last_unstale":"2024-09-21T10:53:58.093179+0000","last_undegraded":"2024-09-21T10:53:58.093179+0000","last_fullsized":"2024-09-21T10:53:58.093179+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:13.939806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1473,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.094764+0000","last_change":"2024-09-21T10:52:15.396755+0000","last_active":"2024-09-21T10:53:58.094764+0000","last_peered":"2024-09-21T10:53:58.094764+0000","last_clean":"2024-09-21T10:53:58.094764+0000","last_became_active":"2024-09-21T10:52:15.396417+0000","last_became_peered":"2024-09-21T10:52:15.396417+0000","last_unstale":"2024-09-21T10:53:58.094764+0000","last_undegraded":"2024-09-21T10:53:58.094764+0000","last_fullsized":"2024-09-21T10:53:58.094764+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:25.005574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":2276,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.376023+0000","last_change":"2024-09-21T10:52:15.400227+0000","last_active":"2024-09-21T10:54:01.376023+0000","last_peered":"2024-09-21T10:54:01.376023+0000","last_clean":"2024-09-21T10:54:01.376023+0000","last_became_active":"2024-09-21T10:52:15.399888+0000","last_became_peered":"2024-09-21T10:52:15.399888+0000","last_unstale":"2024-09-21T10:54:01.376023+0000","last_undegraded":"2024-09-21T10:54:01.376023+0000","last_fullsized":"2024-09-21T10:54:01.376023+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:42:52.565184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2292,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.373876+0000","last_change":"2024-09-21T10:51:27.458216+0000","last_active":"2024-09-21T10:54:01.373876+0000","last_peered":"2024-09-21T10:54:01.373876+0000","last_clean":"2024-09-21T10:54:01.373876+0000","last_became_active":"2024-09-21T10:51:27.457948+0000","last_became_peered":"2024-09-21T10:51:27.457948+0000","last_unstale":"2024-09-21T10:54:01.373876+0000","last_undegraded":"2024-09-21T10:54:01.373876+0000","last_fullsized":"2024-09-21T10:54:01.373876+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:25:29.584079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":112,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.098230+0000","last_change":"2024-09-21T10:49:44.878242+0000","last_active":"2024-09-21T10:53:58.098230+0000","last_peered":"2024-09-21T10:53:58.098230+0000","last_clean":"2024-09-21T10:53:58.098230+0000","last_became_active":"2024-09-21T10:49:44.866578+0000","last_became_peered":"2024-09-21T10:49:44.866578+0000","last_unstale":"2024-09-21T10:53:58.098230+0000","last_undegraded":"2024-09-21T10:53:58.098230+0000","last_fullsized":"2024-09-21T10:53:58.098230+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:17:24.145551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1478,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.095134+0000","last_change":"2024-09-21T10:51:27.460539+0000","last_active":"2024-09-21T10:53:58.095134+0000","last_peered":"2024-09-21T10:53:58.095134+0000","last_clean":"2024-09-21T10:53:58.095134+0000","last_became_active":"2024-09-21T10:51:27.460211+0000","last_became_peered":"2024-09-21T10:51:27.460211+0000","last_unstale":"2024-09-21T10:53:58.095134+0000","last_undegraded":"2024-09-21T10:53:58.095134+0000","last_fullsized":"2024-09-21T10:53:58.095134+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:14:15.700998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":151,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.095523+0000","last_change":"2024-09-21T10:52:15.405886+0000","last_active":"2024-09-21T10:53:58.095523+0000","last_peered":"2024-09-21T10:53:58.095523+0000","last_clean":"2024-09-21T10:53:58.095523+0000","last_became_active":"2024-09-21T10:52:15.397317+0000","last_became_peered":"2024-09-21T10:52:15.397317+0000","last_unstale":"2024-09-21T10:53:58.095523+0000","last_undegraded":"2024-09-21T10:53:58.095523+0000","last_fullsized":"2024-09-21T10:53:58.095523+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:43:48.813861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1469,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.095473+0000","last_change":"2024-09-21T10:09:06.560175+0000","last_active":"2024-09-21T10:53:58.095473+0000","last_peered":"2024-09-21T10:53:58.095473+0000","last_clean":"2024-09-21T10:53:58.095473+0000","last_became_active":"2024-09-21T10:09:06.558584+0000","last_became_peered":"2024-09-21T10:09:06.558584+0000","last_unstale":"2024-09-21T10:53:58.095473+0000","last_undegraded":"2024-09-21T10:53:58.095473+0000","last_fullsized":"2024-09-21T10:53:58.095473+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:02.918339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":60,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.106516+0000","last_change":"2024-09-21T10:51:27.437665+0000","last_active":"2024-09-21T10:54:01.106516+0000","last_peered":"2024-09-21T10:54:01.106516+0000","last_clean":"2024-09-21T10:54:01.106516+0000","last_became_active":"2024-09-21T10:51:27.436481+0000","last_became_peered":"2024-09-21T10:51:27.436481+0000","last_unstale":"2024-09-21T10:54:01.106516+0000","last_undegraded":"2024-09-21T10:54:01.106516+0000","last_fullsized":"2024-09-21T10:54:01.106516+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:07:50.267297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1047,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.378024+0000","last_change":"2024-09-21T10:51:27.459033+0000","last_active":"2024-09-21T10:54:01.378024+0000","last_peered":"2024-09-21T10:54:01.378024+0000","last_clean":"2024-09-21T10:54:01.378024+0000","last_became_active":"2024-09-21T10:51:27.458728+0000","last_became_peered":"2024-09-21T10:51:27.458728+0000","last_unstale":"2024-09-21T10:54:01.378024+0000","last_undegraded":"2024-09-21T10:54:01.378024+0000","last_fullsized":"2024-09-21T10:54:01.378024+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:20.133032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":1498,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.095291+0000","last_change":"2024-09-21T10:52:25.473550+0000","last_active":"2024-09-21T10:53:58.095291+0000","last_peered":"2024-09-21T10:53:58.095291+0000","last_clean":"2024-09-21T10:53:58.095291+0000","last_became_active":"2024-09-21T10:52:25.473105+0000","last_became_peered":"2024-09-21T10:52:25.473105+0000","last_unstale":"2024-09-21T10:53:58.095291+0000","last_undegraded":"2024-09-21T10:53:58.095291+0000","last_fullsized":"2024-09-21T10:53:58.095291+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:15:45.352518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2271,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.376992+0000","last_change":"2024-09-21T10:40:13.432531+0000","last_active":"2024-09-21T10:54:01.376992+0000","last_peered":"2024-09-21T10:54:01.376992+0000","last_clean":"2024-09-21T10:54:01.376992+0000","last_became_active":"2024-09-21T10:40:13.432210+0000","last_became_peered":"2024-09-21T10:40:13.432210+0000","last_unstale":"2024-09-21T10:54:01.376992+0000","last_undegraded":"2024-09-21T10:54:01.376992+0000","last_fullsized":"2024-09-21T10:54:01.376992+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:20:10.998607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2244,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.373836+0000","last_change":"2024-09-21T10:09:06.530394+0000","last_active":"2024-09-21T10:54:01.373836+0000","last_peered":"2024-09-21T10:54:01.373836+0000","last_clean":"2024-09-21T10:54:01.373836+0000","last_became_active":"2024-09-21T10:09:06.529307+0000","last_became_peered":"2024-09-21T10:09:06.529307+0000","last_unstale":"2024-09-21T10:54:01.373836+0000","last_undegraded":"2024-09-21T10:54:01.373836+0000","last_fullsized":"2024-09-21T10:54:01.373836+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:54.287397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":28,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.091534+0000","last_change":"2024-09-21T10:52:25.489896+0000","last_active":"2024-09-21T10:53:58.091534+0000","last_peered":"2024-09-21T10:53:58.091534+0000","last_clean":"2024-09-21T10:53:58.091534+0000","last_became_active":"2024-09-21T10:52:25.489154+0000","last_became_peered":"2024-09-21T10:52:25.489154+0000","last_unstale":"2024-09-21T10:53:58.091534+0000","last_undegraded":"2024-09-21T10:53:58.091534+0000","last_fullsized":"2024-09-21T10:53:58.091534+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:36:02.076931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2272,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.373358+0000","last_change":"2024-09-21T10:52:25.493309+0000","last_active":"2024-09-21T10:54:01.373358+0000","last_peered":"2024-09-21T10:54:01.373358+0000","last_clean":"2024-09-21T10:54:01.373358+0000","last_became_active":"2024-09-21T10:52:25.492568+0000","last_became_peered":"2024-09-21T10:52:25.492568+0000","last_unstale":"2024-09-21T10:54:01.373358+0000","last_undegraded":"2024-09-21T10:54:01.373358+0000","last_fullsized":"2024-09-21T10:54:01.373358+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:19:19.393370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1486,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.093253+0000","last_change":"2024-09-21T10:52:03.317474+0000","last_active":"2024-09-21T10:53:58.093253+0000","last_peered":"2024-09-21T10:53:58.093253+0000","last_clean":"2024-09-21T10:53:58.093253+0000","last_became_active":"2024-09-21T10:52:03.317001+0000","last_became_peered":"2024-09-21T10:52:03.317001+0000","last_unstale":"2024-09-21T10:53:58.093253+0000","last_undegraded":"2024-09-21T10:53:58.093253+0000","last_fullsized":"2024-09-21T10:53:58.093253+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:06:07.039763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2272,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.374025+0000","last_change":"2024-09-21T10:52:15.412246+0000","last_active":"2024-09-21T10:54:01.374025+0000","last_peered":"2024-09-21T10:54:01.374025+0000","last_clean":"2024-09-21T10:54:01.374025+0000","last_became_active":"2024-09-21T10:52:15.406204+0000","last_became_peered":"2024-09-21T10:52:15.406204+0000","last_unstale":"2024-09-21T10:54:01.374025+0000","last_undegraded":"2024-09-21T10:54:01.374025+0000","last_fullsized":"2024-09-21T10:54:01.374025+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:03.524324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2252,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.373788+0000","last_change":"2024-09-21T10:52:15.402822+0000","last_active":"2024-09-21T10:54:01.373788+0000","last_peered":"2024-09-21T10:54:01.373788+0000","last_clean":"2024-09-21T10:54:01.373788+0000","last_became_active":"2024-09-21T10:52:15.402548+0000","last_became_peered":"2024-09-21T10:52:15.402548+0000","last_unstale":"2024-09-21T10:54:01.373788+0000","last_undegraded":"2024-09-21T10:54:01.373788+0000","last_fullsized":"2024-09-21T10:54:01.373788+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:04:04.667598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2277,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.374069+0000","last_change":"2024-09-21T10:52:25.499172+0000","last_active":"2024-09-21T10:54:01.374069+0000","last_peered":"2024-09-21T10:54:01.374069+0000","last_clean":"2024-09-21T10:54:01.374069+0000","last_became_active":"2024-09-21T10:52:25.498835+0000","last_became_peered":"2024-09-21T10:52:25.498835+0000","last_unstale":"2024-09-21T10:54:01.374069+0000","last_undegraded":"2024-09-21T10:54:01.374069+0000","last_fullsized":"2024-09-21T10:54:01.374069+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:10:12.434470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":28,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.091238+0000","last_change":"2024-09-21T10:52:25.489774+0000","last_active":"2024-09-21T10:53:58.091238+0000","last_peered":"2024-09-21T10:53:58.091238+0000","last_clean":"2024-09-21T10:53:58.091238+0000","last_became_active":"2024-09-21T10:52:25.487788+0000","last_became_peered":"2024-09-21T10:52:25.487788+0000","last_unstale":"2024-09-21T10:53:58.091238+0000","last_undegraded":"2024-09-21T10:53:58.091238+0000","last_fullsized":"2024-09-21T10:53:58.091238+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:48.842419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":34,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.091202+0000","last_change":"2024-09-21T10:52:25.565465+0000","last_active":"2024-09-21T10:53:58.091202+0000","last_peered":"2024-09-21T10:53:58.091202+0000","last_clean":"2024-09-21T10:53:58.091202+0000","last_became_active":"2024-09-21T10:52:25.510131+0000","last_became_peered":"2024-09-21T10:52:25.510131+0000","last_unstale":"2024-09-21T10:53:58.091202+0000","last_undegraded":"2024-09-21T10:53:58.091202+0000","last_fullsized":"2024-09-21T10:53:58.091202+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:41:13.870503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":450,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2278,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.376054+0000","last_change":"2024-09-21T10:52:25.504787+0000","last_active":"2024-09-21T10:54:01.376054+0000","last_peered":"2024-09-21T10:54:01.376054+0000","last_clean":"2024-09-21T10:54:01.376054+0000","last_became_active":"2024-09-21T10:52:25.504198+0000","last_became_peered":"2024-09-21T10:52:25.504198+0000","last_unstale":"2024-09-21T10:54:01.376054+0000","last_undegraded":"2024-09-21T10:54:01.376054+0000","last_fullsized":"2024-09-21T10:54:01.376054+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:38:25.666224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":1503,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.095598+0000","last_change":"2024-09-21T10:44:14.415264+0000","last_active":"2024-09-21T10:53:58.095598+0000","last_peered":"2024-09-21T10:53:58.095598+0000","last_clean":"2024-09-21T10:53:58.095598+0000","last_became_active":"2024-09-21T10:44:14.407813+0000","last_became_peered":"2024-09-21T10:44:14.407813+0000","last_unstale":"2024-09-21T10:53:58.095598+0000","last_undegraded":"2024-09-21T10:53:58.095598+0000","last_fullsized":"2024-09-21T10:53:58.095598+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:55.933601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1456,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.095377+0000","last_change":"2024-09-21T10:07:56.001756+0000","last_active":"2024-09-21T10:53:58.095377+0000","last_peered":"2024-09-21T10:53:58.095377+0000","last_clean":"2024-09-21T10:53:58.095377+0000","last_became_active":"2024-09-21T10:07:56.000789+0000","last_became_peered":"2024-09-21T10:07:56.000789+0000","last_unstale":"2024-09-21T10:53:58.095377+0000","last_undegraded":"2024-09-21T10:53:58.095377+0000","last_fullsized":"2024-09-21T10:53:58.095377+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:56:04.302902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"346'149","reported_seq":940,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.374034+0000","last_change":"2024-09-21T10:53:47.069462+0000","last_active":"2024-09-21T10:54:01.374034+0000","last_peered":"2024-09-21T10:54:01.374034+0000","last_clean":"2024-09-21T10:54:01.374034+0000","last_became_active":"2024-09-21T10:53:47.069113+0000","last_became_peered":"2024-09-21T10:53:47.069113+0000","last_unstale":"2024-09-21T10:54:01.374034+0000","last_undegraded":"2024-09-21T10:54:01.374034+0000","last_fullsized":"2024-09-21T10:54:01.374034+0000","mapping_epoch":1478,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1479,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1579069,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1471,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.095090+0000","last_change":"2024-09-21T10:52:25.473127+0000","last_active":"2024-09-21T10:53:58.095090+0000","last_peered":"2024-09-21T10:53:58.095090+0000","last_clean":"2024-09-21T10:53:58.095090+0000","last_became_active":"2024-09-21T10:52:25.472805+0000","last_became_peered":"2024-09-21T10:52:25.472805+0000","last_unstale":"2024-09-21T10:53:58.095090+0000","last_undegraded":"2024-09-21T10:53:58.095090+0000","last_fullsized":"2024-09-21T10:53:58.095090+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:13.038797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1516,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.093321+0000","last_change":"2024-09-21T10:52:25.493263+0000","last_active":"2024-09-21T10:53:58.093321+0000","last_peered":"2024-09-21T10:53:58.093321+0000","last_clean":"2024-09-21T10:53:58.093321+0000","last_became_active":"2024-09-21T10:52:25.492970+0000","last_became_peered":"2024-09-21T10:52:25.492970+0000","last_unstale":"2024-09-21T10:53:58.093321+0000","last_undegraded":"2024-09-21T10:53:58.093321+0000","last_fullsized":"2024-09-21T10:53:58.093321+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:43:40.305783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":123,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.378496+0000","last_change":"2024-09-21T10:52:25.513834+0000","last_active":"2024-09-21T10:54:01.378496+0000","last_peered":"2024-09-21T10:54:01.378496+0000","last_clean":"2024-09-21T10:54:01.378496+0000","last_became_active":"2024-09-21T10:52:25.475423+0000","last_became_peered":"2024-09-21T10:52:25.475423+0000","last_unstale":"2024-09-21T10:54:01.378496+0000","last_undegraded":"2024-09-21T10:54:01.378496+0000","last_fullsized":"2024-09-21T10:54:01.378496+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:10.216527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"346'149","reported_seq":903,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.095823+0000","last_change":"2024-09-21T10:52:24.459450+0000","last_active":"2024-09-21T10:53:58.095823+0000","last_peered":"2024-09-21T10:53:58.095823+0000","last_clean":"2024-09-21T10:53:58.095823+0000","last_became_active":"2024-09-21T10:52:15.405544+0000","last_became_peered":"2024-09-21T10:52:15.405544+0000","last_unstale":"2024-09-21T10:53:58.095823+0000","last_undegraded":"2024-09-21T10:53:58.095823+0000","last_fullsized":"2024-09-21T10:53:58.095823+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2244,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.373974+0000","last_change":"2024-09-21T10:52:25.494316+0000","last_active":"2024-09-21T10:54:01.373974+0000","last_peered":"2024-09-21T10:54:01.373974+0000","last_clean":"2024-09-21T10:54:01.373974+0000","last_became_active":"2024-09-21T10:52:25.494016+0000","last_became_peered":"2024-09-21T10:52:25.494016+0000","last_unstale":"2024-09-21T10:54:01.373974+0000","last_undegraded":"2024-09-21T10:54:01.373974+0000","last_fullsized":"2024-09-21T10:54:01.373974+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:07.859584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2245,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.378703+0000","last_change":"2024-09-21T10:07:56.006236+0000","last_active":"2024-09-21T10:54:01.378703+0000","last_peered":"2024-09-21T10:54:01.378703+0000","last_clean":"2024-09-21T10:54:01.378703+0000","last_became_active":"2024-09-21T10:07:56.004439+0000","last_became_peered":"2024-09-21T10:07:56.004439+0000","last_unstale":"2024-09-21T10:54:01.378703+0000","last_undegraded":"2024-09-21T10:54:01.378703+0000","last_fullsized":"2024-09-21T10:54:01.378703+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:39.422467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":2294,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.373986+0000","last_change":"2024-09-21T10:52:25.536401+0000","last_active":"2024-09-21T10:54:01.373986+0000","last_peered":"2024-09-21T10:54:01.373986+0000","last_clean":"2024-09-21T10:54:01.373986+0000","last_became_active":"2024-09-21T10:52:25.496480+0000","last_became_peered":"2024-09-21T10:52:25.496480+0000","last_unstale":"2024-09-21T10:54:01.373986+0000","last_undegraded":"2024-09-21T10:54:01.373986+0000","last_fullsized":"2024-09-21T10:54:01.373986+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:50:34.820373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"346'149","reported_seq":728,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.093073+0000","last_change":"2024-09-21T10:51:30.450301+0000","last_active":"2024-09-21T10:53:58.093073+0000","last_peered":"2024-09-21T10:53:58.093073+0000","last_clean":"2024-09-21T10:53:58.093073+0000","last_became_active":"2024-09-21T10:51:30.449925+0000","last_became_peered":"2024-09-21T10:51:30.449925+0000","last_unstale":"2024-09-21T10:53:58.093073+0000","last_undegraded":"2024-09-21T10:53:58.093073+0000","last_fullsized":"2024-09-21T10:53:58.093073+0000","mapping_epoch":1450,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1451,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:41.011085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:41.011085+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:38:33.451261+0000","scrub_duration":0.0061250310000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2199,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.090714+0000","last_change":"2024-09-21T10:51:27.431416+0000","last_active":"2024-09-21T10:53:58.090714+0000","last_peered":"2024-09-21T10:53:58.090714+0000","last_clean":"2024-09-21T10:53:58.090714+0000","last_became_active":"2024-09-21T10:51:27.431127+0000","last_became_peered":"2024-09-21T10:51:27.431127+0000","last_unstale":"2024-09-21T10:53:58.090714+0000","last_undegraded":"2024-09-21T10:53:58.090714+0000","last_fullsized":"2024-09-21T10:53:58.090714+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:27.721665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":28,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.091117+0000","last_change":"2024-09-21T10:52:25.468849+0000","last_active":"2024-09-21T10:53:58.091117+0000","last_peered":"2024-09-21T10:53:58.091117+0000","last_clean":"2024-09-21T10:53:58.091117+0000","last_became_active":"2024-09-21T10:52:25.468552+0000","last_became_peered":"2024-09-21T10:52:25.468552+0000","last_unstale":"2024-09-21T10:53:58.091117+0000","last_undegraded":"2024-09-21T10:53:58.091117+0000","last_fullsized":"2024-09-21T10:53:58.091117+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:36:41.692028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2295,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.377924+0000","last_change":"2024-09-21T10:51:27.436608+0000","last_active":"2024-09-21T10:54:01.377924+0000","last_peered":"2024-09-21T10:54:01.377924+0000","last_clean":"2024-09-21T10:54:01.377924+0000","last_became_active":"2024-09-21T10:51:27.434949+0000","last_became_peered":"2024-09-21T10:51:27.434949+0000","last_unstale":"2024-09-21T10:54:01.377924+0000","last_undegraded":"2024-09-21T10:54:01.377924+0000","last_fullsized":"2024-09-21T10:54:01.377924+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:49:23.439425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"346'149","reported_seq":771,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.092908+0000","last_change":"2024-09-21T10:52:28.590793+0000","last_active":"2024-09-21T10:53:58.092908+0000","last_peered":"2024-09-21T10:53:58.092908+0000","last_clean":"2024-09-21T10:53:58.092908+0000","last_became_active":"2024-09-21T10:52:28.588662+0000","last_became_peered":"2024-09-21T10:52:28.588662+0000","last_unstale":"2024-09-21T10:53:58.092908+0000","last_undegraded":"2024-09-21T10:53:58.092908+0000","last_fullsized":"2024-09-21T10:53:58.092908+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:44.018645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:44.018645+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:46.849427+0000","scrub_duration":0.0012789399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2250,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.375930+0000","last_change":"2024-09-21T10:51:27.440425+0000","last_active":"2024-09-21T10:54:01.375930+0000","last_peered":"2024-09-21T10:54:01.375930+0000","last_clean":"2024-09-21T10:54:01.375930+0000","last_became_active":"2024-09-21T10:51:27.439666+0000","last_became_peered":"2024-09-21T10:51:27.439666+0000","last_unstale":"2024-09-21T10:54:01.375930+0000","last_undegraded":"2024-09-21T10:54:01.375930+0000","last_fullsized":"2024-09-21T10:54:01.375930+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:56:47.410516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2318,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.377686+0000","last_change":"2024-09-21T10:52:03.314926+0000","last_active":"2024-09-21T10:54:01.377686+0000","last_peered":"2024-09-21T10:54:01.377686+0000","last_clean":"2024-09-21T10:54:01.377686+0000","last_became_active":"2024-09-21T10:52:03.314602+0000","last_became_peered":"2024-09-21T10:52:03.314602+0000","last_unstale":"2024-09-21T10:54:01.377686+0000","last_undegraded":"2024-09-21T10:54:01.377686+0000","last_fullsized":"2024-09-21T10:54:01.377686+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:13.774521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1473,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.376889+0000","last_change":"2024-09-21T10:52:25.502830+0000","last_active":"2024-09-21T10:54:01.376889+0000","last_peered":"2024-09-21T10:54:01.376889+0000","last_clean":"2024-09-21T10:54:01.376889+0000","last_became_active":"2024-09-21T10:52:25.502505+0000","last_became_peered":"2024-09-21T10:52:25.502505+0000","last_unstale":"2024-09-21T10:54:01.376889+0000","last_undegraded":"2024-09-21T10:54:01.376889+0000","last_fullsized":"2024-09-21T10:54:01.376889+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:01:14.965694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"346'149","reported_seq":826,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.094742+0000","last_change":"2024-09-21T10:52:24.458657+0000","last_active":"2024-09-21T10:53:58.094742+0000","last_peered":"2024-09-21T10:53:58.094742+0000","last_clean":"2024-09-21T10:53:58.094742+0000","last_became_active":"2024-09-21T10:51:30.436983+0000","last_became_peered":"2024-09-21T10:51:30.436983+0000","last_unstale":"2024-09-21T10:53:58.094742+0000","last_undegraded":"2024-09-21T10:53:58.094742+0000","last_fullsized":"2024-09-21T10:53:58.094742+0000","mapping_epoch":1450,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1451,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2312,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.378519+0000","last_change":"2024-09-21T10:52:15.397180+0000","last_active":"2024-09-21T10:54:01.378519+0000","last_peered":"2024-09-21T10:54:01.378519+0000","last_clean":"2024-09-21T10:54:01.378519+0000","last_became_active":"2024-09-21T10:52:15.396879+0000","last_became_peered":"2024-09-21T10:52:15.396879+0000","last_unstale":"2024-09-21T10:54:01.378519+0000","last_undegraded":"2024-09-21T10:54:01.378519+0000","last_fullsized":"2024-09-21T10:54:01.378519+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:03:07.162178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1469,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.095452+0000","last_change":"2024-09-21T10:51:27.436590+0000","last_active":"2024-09-21T10:53:58.095452+0000","last_peered":"2024-09-21T10:53:58.095452+0000","last_clean":"2024-09-21T10:53:58.095452+0000","last_became_active":"2024-09-21T10:51:27.435332+0000","last_became_peered":"2024-09-21T10:51:27.435332+0000","last_unstale":"2024-09-21T10:53:58.095452+0000","last_undegraded":"2024-09-21T10:53:58.095452+0000","last_fullsized":"2024-09-21T10:53:58.095452+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:05:42.885506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":212,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.375777+0000","last_change":"2024-09-21T10:53:41.149328+0000","last_active":"2024-09-21T10:54:01.375777+0000","last_peered":"2024-09-21T10:54:01.375777+0000","last_clean":"2024-09-21T10:54:01.375777+0000","last_became_active":"2024-09-21T10:52:03.315398+0000","last_became_peered":"2024-09-21T10:52:03.315398+0000","last_unstale":"2024-09-21T10:54:01.375777+0000","last_undegraded":"2024-09-21T10:54:01.375777+0000","last_fullsized":"2024-09-21T10:54:01.375777+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:48:09.007053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"346'149","reported_seq":796,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.090969+0000","last_change":"2024-09-21T10:52:28.495319+0000","last_active":"2024-09-21T10:53:58.090969+0000","last_peered":"2024-09-21T10:53:58.090969+0000","last_clean":"2024-09-21T10:53:58.090969+0000","last_became_active":"2024-09-21T10:52:28.494923+0000","last_became_peered":"2024-09-21T10:52:28.494923+0000","last_unstale":"2024-09-21T10:53:58.090969+0000","last_undegraded":"2024-09-21T10:53:58.090969+0000","last_fullsized":"2024-09-21T10:53:58.090969+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:52:17.894787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:52:17.894787+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":186,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.095881+0000","last_change":"2024-09-21T10:52:25.475200+0000","last_active":"2024-09-21T10:53:58.095881+0000","last_peered":"2024-09-21T10:53:58.095881+0000","last_clean":"2024-09-21T10:53:58.095881+0000","last_became_active":"2024-09-21T10:52:25.474872+0000","last_became_peered":"2024-09-21T10:52:25.474872+0000","last_unstale":"2024-09-21T10:53:58.095881+0000","last_undegraded":"2024-09-21T10:53:58.095881+0000","last_fullsized":"2024-09-21T10:53:58.095881+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:36:34.207844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2268,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.375032+0000","last_change":"2024-09-21T10:52:03.325889+0000","last_active":"2024-09-21T10:54:01.375032+0000","last_peered":"2024-09-21T10:54:01.375032+0000","last_clean":"2024-09-21T10:54:01.375032+0000","last_became_active":"2024-09-21T10:52:03.325500+0000","last_became_peered":"2024-09-21T10:52:03.325500+0000","last_unstale":"2024-09-21T10:54:01.375032+0000","last_undegraded":"2024-09-21T10:54:01.375032+0000","last_fullsized":"2024-09-21T10:54:01.375032+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:20.708252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1054,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.373780+0000","last_change":"2024-09-21T10:51:27.457894+0000","last_active":"2024-09-21T10:54:01.373780+0000","last_peered":"2024-09-21T10:54:01.373780+0000","last_clean":"2024-09-21T10:54:01.373780+0000","last_became_active":"2024-09-21T10:51:27.457431+0000","last_became_peered":"2024-09-21T10:51:27.457431+0000","last_unstale":"2024-09-21T10:54:01.373780+0000","last_undegraded":"2024-09-21T10:54:01.373780+0000","last_fullsized":"2024-09-21T10:54:01.373780+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:03:33.550759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"346'149","reported_seq":925,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.377496+0000","last_change":"2024-09-21T10:52:15.396042+0000","last_active":"2024-09-21T10:54:01.377496+0000","last_peered":"2024-09-21T10:54:01.377496+0000","last_clean":"2024-09-21T10:54:01.377496+0000","last_became_active":"2024-09-21T10:52:15.395717+0000","last_became_peered":"2024-09-21T10:52:15.395717+0000","last_unstale":"2024-09-21T10:54:01.377496+0000","last_undegraded":"2024-09-21T10:54:01.377496+0000","last_fullsized":"2024-09-21T10:54:01.377496+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2294,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.378399+0000","last_change":"2024-09-21T10:52:25.476572+0000","last_active":"2024-09-21T10:54:01.378399+0000","last_peered":"2024-09-21T10:54:01.378399+0000","last_clean":"2024-09-21T10:54:01.378399+0000","last_became_active":"2024-09-21T10:52:25.475803+0000","last_became_peered":"2024-09-21T10:52:25.475803+0000","last_unstale":"2024-09-21T10:54:01.378399+0000","last_undegraded":"2024-09-21T10:54:01.378399+0000","last_fullsized":"2024-09-21T10:54:01.378399+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:06:44.191486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2215,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.094945+0000","last_change":"2024-09-21T10:52:03.326753+0000","last_active":"2024-09-21T10:53:58.094945+0000","last_peered":"2024-09-21T10:53:58.094945+0000","last_clean":"2024-09-21T10:53:58.094945+0000","last_became_active":"2024-09-21T10:52:03.326348+0000","last_became_peered":"2024-09-21T10:52:03.326348+0000","last_unstale":"2024-09-21T10:53:58.094945+0000","last_undegraded":"2024-09-21T10:53:58.094945+0000","last_fullsized":"2024-09-21T10:53:58.094945+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:57:38.100799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":137,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.090636+0000","last_change":"2024-09-21T10:51:27.426006+0000","last_active":"2024-09-21T10:53:58.090636+0000","last_peered":"2024-09-21T10:53:58.090636+0000","last_clean":"2024-09-21T10:53:58.090636+0000","last_became_active":"2024-09-21T10:51:27.425698+0000","last_became_peered":"2024-09-21T10:51:27.425698+0000","last_unstale":"2024-09-21T10:53:58.090636+0000","last_undegraded":"2024-09-21T10:53:58.090636+0000","last_fullsized":"2024-09-21T10:53:58.090636+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:35:13.210330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":31,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.092736+0000","last_change":"2024-09-21T10:52:15.401737+0000","last_active":"2024-09-21T10:53:58.092736+0000","last_peered":"2024-09-21T10:53:58.092736+0000","last_clean":"2024-09-21T10:53:58.092736+0000","last_became_active":"2024-09-21T10:52:15.400859+0000","last_became_peered":"2024-09-21T10:52:15.400859+0000","last_unstale":"2024-09-21T10:53:58.092736+0000","last_undegraded":"2024-09-21T10:53:58.092736+0000","last_fullsized":"2024-09-21T10:53:58.092736+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:15.723031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2248,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.374529+0000","last_change":"2024-09-21T10:49:44.865830+0000","last_active":"2024-09-21T10:54:01.374529+0000","last_peered":"2024-09-21T10:54:01.374529+0000","last_clean":"2024-09-21T10:54:01.374529+0000","last_became_active":"2024-09-21T10:49:44.865499+0000","last_became_peered":"2024-09-21T10:49:44.865499+0000","last_unstale":"2024-09-21T10:54:01.374529+0000","last_undegraded":"2024-09-21T10:54:01.374529+0000","last_fullsized":"2024-09-21T10:54:01.374529+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:19:22.013154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":1462,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.090783+0000","last_change":"2024-09-21T10:52:25.489550+0000","last_active":"2024-09-21T10:53:58.090783+0000","last_peered":"2024-09-21T10:53:58.090783+0000","last_clean":"2024-09-21T10:53:58.090783+0000","last_became_active":"2024-09-21T10:52:25.489171+0000","last_became_peered":"2024-09-21T10:52:25.489171+0000","last_unstale":"2024-09-21T10:53:58.090783+0000","last_undegraded":"2024-09-21T10:53:58.090783+0000","last_fullsized":"2024-09-21T10:53:58.090783+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:21.105058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":796,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.090541+0000","last_change":"2024-09-21T10:52:28.601906+0000","last_active":"2024-09-21T10:53:58.090541+0000","last_peered":"2024-09-21T10:53:58.090541+0000","last_clean":"2024-09-21T10:53:58.090541+0000","last_became_active":"2024-09-21T10:52:28.587447+0000","last_became_peered":"2024-09-21T10:52:28.587447+0000","last_unstale":"2024-09-21T10:53:58.090541+0000","last_undegraded":"2024-09-21T10:53:58.090541+0000","last_fullsized":"2024-09-21T10:53:58.090541+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:44.223578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:44.223578+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":74,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"346'149","reported_seq":745,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.092593+0000","last_change":"2024-09-21T10:51:30.453979+0000","last_active":"2024-09-21T10:53:58.092593+0000","last_peered":"2024-09-21T10:53:58.092593+0000","last_clean":"2024-09-21T10:53:58.092593+0000","last_became_active":"2024-09-21T10:51:30.450893+0000","last_became_peered":"2024-09-21T10:51:30.450893+0000","last_unstale":"2024-09-21T10:53:58.092593+0000","last_undegraded":"2024-09-21T10:53:58.092593+0000","last_fullsized":"2024-09-21T10:53:58.092593+0000","mapping_epoch":1450,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1451,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:42.008235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:42.008235+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:21:48.570809+0000","scrub_duration":0.011129834999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2245,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.379470+0000","last_change":"2024-09-21T10:09:06.538940+0000","last_active":"2024-09-21T10:54:01.379470+0000","last_peered":"2024-09-21T10:54:01.379470+0000","last_clean":"2024-09-21T10:54:01.379470+0000","last_became_active":"2024-09-21T10:09:06.538668+0000","last_became_peered":"2024-09-21T10:09:06.538668+0000","last_unstale":"2024-09-21T10:54:01.379470+0000","last_undegraded":"2024-09-21T10:54:01.379470+0000","last_fullsized":"2024-09-21T10:54:01.379470+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:42:44.299488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2243,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.373270+0000","last_change":"2024-09-21T10:52:25.493032+0000","last_active":"2024-09-21T10:54:01.373270+0000","last_peered":"2024-09-21T10:54:01.373270+0000","last_clean":"2024-09-21T10:54:01.373270+0000","last_became_active":"2024-09-21T10:52:25.492182+0000","last_became_peered":"2024-09-21T10:52:25.492182+0000","last_unstale":"2024-09-21T10:54:01.373270+0000","last_undegraded":"2024-09-21T10:54:01.373270+0000","last_fullsized":"2024-09-21T10:54:01.373270+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:12:57.118420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1475,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.095739+0000","last_change":"2024-09-21T10:52:15.402809+0000","last_active":"2024-09-21T10:53:58.095739+0000","last_peered":"2024-09-21T10:53:58.095739+0000","last_clean":"2024-09-21T10:53:58.095739+0000","last_became_active":"2024-09-21T10:52:15.402497+0000","last_became_peered":"2024-09-21T10:52:15.402497+0000","last_unstale":"2024-09-21T10:53:58.095739+0000","last_undegraded":"2024-09-21T10:53:58.095739+0000","last_fullsized":"2024-09-21T10:53:58.095739+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:07:35.713534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1486,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.095209+0000","last_change":"2024-09-21T10:44:14.399842+0000","last_active":"2024-09-21T10:53:58.095209+0000","last_peered":"2024-09-21T10:53:58.095209+0000","last_clean":"2024-09-21T10:53:58.095209+0000","last_became_active":"2024-09-21T10:44:14.399426+0000","last_became_peered":"2024-09-21T10:44:14.399426+0000","last_unstale":"2024-09-21T10:53:58.095209+0000","last_undegraded":"2024-09-21T10:53:58.095209+0000","last_fullsized":"2024-09-21T10:53:58.095209+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:26:45.362101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"346'149","reported_seq":925,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.373217+0000","last_change":"2024-09-21T10:52:16.404597+0000","last_active":"2024-09-21T10:54:01.373217+0000","last_peered":"2024-09-21T10:54:01.373217+0000","last_clean":"2024-09-21T10:54:01.373217+0000","last_became_active":"2024-09-21T10:52:16.404194+0000","last_became_peered":"2024-09-21T10:52:16.404194+0000","last_unstale":"2024-09-21T10:54:01.373217+0000","last_undegraded":"2024-09-21T10:54:01.373217+0000","last_fullsized":"2024-09-21T10:54:01.373217+0000","mapping_epoch":1463,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1464,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":793725,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1481,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.098232+0000","last_change":"2024-09-21T10:40:13.432387+0000","last_active":"2024-09-21T10:53:58.098232+0000","last_peered":"2024-09-21T10:53:58.098232+0000","last_clean":"2024-09-21T10:53:58.098232+0000","last_became_active":"2024-09-21T10:40:13.432053+0000","last_became_peered":"2024-09-21T10:40:13.432053+0000","last_unstale":"2024-09-21T10:53:58.098232+0000","last_undegraded":"2024-09-21T10:53:58.098232+0000","last_fullsized":"2024-09-21T10:53:58.098232+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:37.486101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1459,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.092738+0000","last_change":"2024-09-21T10:07:56.020791+0000","last_active":"2024-09-21T10:53:58.092738+0000","last_peered":"2024-09-21T10:53:58.092738+0000","last_clean":"2024-09-21T10:53:58.092738+0000","last_became_active":"2024-09-21T10:07:56.020205+0000","last_became_peered":"2024-09-21T10:07:56.020205+0000","last_unstale":"2024-09-21T10:53:58.092738+0000","last_undegraded":"2024-09-21T10:53:58.092738+0000","last_fullsized":"2024-09-21T10:53:58.092738+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:09.813166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"346'149","reported_seq":922,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.090889+0000","last_change":"2024-09-21T10:49:47.904295+0000","last_active":"2024-09-21T10:53:58.090889+0000","last_peered":"2024-09-21T10:53:58.090889+0000","last_clean":"2024-09-21T10:53:58.090889+0000","last_became_active":"2024-09-21T10:49:47.903650+0000","last_became_peered":"2024-09-21T10:49:47.903650+0000","last_unstale":"2024-09-21T10:53:58.090889+0000","last_undegraded":"2024-09-21T10:53:58.090889+0000","last_fullsized":"2024-09-21T10:53:58.090889+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":74,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1457,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.095517+0000","last_change":"2024-09-21T10:09:06.533271+0000","last_active":"2024-09-21T10:53:58.095517+0000","last_peered":"2024-09-21T10:53:58.095517+0000","last_clean":"2024-09-21T10:53:58.095517+0000","last_became_active":"2024-09-21T10:09:06.532041+0000","last_became_peered":"2024-09-21T10:09:06.532041+0000","last_unstale":"2024-09-21T10:53:58.095517+0000","last_undegraded":"2024-09-21T10:53:58.095517+0000","last_fullsized":"2024-09-21T10:53:58.095517+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:04.236375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":2273,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.375470+0000","last_change":"2024-09-21T10:51:27.529295+0000","last_active":"2024-09-21T10:54:01.375470+0000","last_peered":"2024-09-21T10:54:01.375470+0000","last_clean":"2024-09-21T10:54:01.375470+0000","last_became_active":"2024-09-21T10:51:27.446359+0000","last_became_peered":"2024-09-21T10:51:27.446359+0000","last_unstale":"2024-09-21T10:54:01.375470+0000","last_undegraded":"2024-09-21T10:54:01.375470+0000","last_fullsized":"2024-09-21T10:54:01.375470+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:17:08.788558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":703,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.378476+0000","last_change":"2024-09-21T10:51:27.481103+0000","last_active":"2024-09-21T10:54:01.378476+0000","last_peered":"2024-09-21T10:54:01.378476+0000","last_clean":"2024-09-21T10:54:01.378476+0000","last_became_active":"2024-09-21T10:51:27.480763+0000","last_became_peered":"2024-09-21T10:51:27.480763+0000","last_unstale":"2024-09-21T10:54:01.378476+0000","last_undegraded":"2024-09-21T10:54:01.378476+0000","last_fullsized":"2024-09-21T10:54:01.378476+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:35.375303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":808,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.374573+0000","last_change":"2024-09-21T10:49:44.864819+0000","last_active":"2024-09-21T10:54:01.374573+0000","last_peered":"2024-09-21T10:54:01.374573+0000","last_clean":"2024-09-21T10:54:01.374573+0000","last_became_active":"2024-09-21T10:49:44.864061+0000","last_became_peered":"2024-09-21T10:49:44.864061+0000","last_unstale":"2024-09-21T10:54:01.374573+0000","last_undegraded":"2024-09-21T10:54:01.374573+0000","last_fullsized":"2024-09-21T10:54:01.374573+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:37.266967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":40,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.092337+0000","last_change":"2024-09-21T10:52:25.492570+0000","last_active":"2024-09-21T10:53:58.092337+0000","last_peered":"2024-09-21T10:53:58.092337+0000","last_clean":"2024-09-21T10:53:58.092337+0000","last_became_active":"2024-09-21T10:52:25.486382+0000","last_became_peered":"2024-09-21T10:52:25.486382+0000","last_unstale":"2024-09-21T10:53:58.092337+0000","last_undegraded":"2024-09-21T10:53:58.092337+0000","last_fullsized":"2024-09-21T10:53:58.092337+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:06.702707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":145,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.091689+0000","last_change":"2024-09-21T10:49:44.874522+0000","last_active":"2024-09-21T10:53:58.091689+0000","last_peered":"2024-09-21T10:53:58.091689+0000","last_clean":"2024-09-21T10:53:58.091689+0000","last_became_active":"2024-09-21T10:49:44.863242+0000","last_became_peered":"2024-09-21T10:49:44.863242+0000","last_unstale":"2024-09-21T10:53:58.091689+0000","last_undegraded":"2024-09-21T10:53:58.091689+0000","last_fullsized":"2024-09-21T10:53:58.091689+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:09:02.066837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1491,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.095106+0000","last_change":"2024-09-21T10:52:03.315267+0000","last_active":"2024-09-21T10:53:58.095106+0000","last_peered":"2024-09-21T10:53:58.095106+0000","last_clean":"2024-09-21T10:53:58.095106+0000","last_became_active":"2024-09-21T10:52:03.314474+0000","last_became_peered":"2024-09-21T10:52:03.314474+0000","last_unstale":"2024-09-21T10:53:58.095106+0000","last_undegraded":"2024-09-21T10:53:58.095106+0000","last_fullsized":"2024-09-21T10:53:58.095106+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:00:57.791341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":32,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.091220+0000","last_change":"2024-09-21T10:52:25.518721+0000","last_active":"2024-09-21T10:53:58.091220+0000","last_peered":"2024-09-21T10:53:58.091220+0000","last_clean":"2024-09-21T10:53:58.091220+0000","last_became_active":"2024-09-21T10:52:25.472114+0000","last_became_peered":"2024-09-21T10:52:25.472114+0000","last_unstale":"2024-09-21T10:53:58.091220+0000","last_undegraded":"2024-09-21T10:53:58.091220+0000","last_fullsized":"2024-09-21T10:53:58.091220+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:33:48.471147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":28,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.090482+0000","last_change":"2024-09-21T10:52:25.494490+0000","last_active":"2024-09-21T10:53:58.090482+0000","last_peered":"2024-09-21T10:53:58.090482+0000","last_clean":"2024-09-21T10:53:58.090482+0000","last_became_active":"2024-09-21T10:52:25.494200+0000","last_became_peered":"2024-09-21T10:52:25.494200+0000","last_unstale":"2024-09-21T10:53:58.090482+0000","last_undegraded":"2024-09-21T10:53:58.090482+0000","last_fullsized":"2024-09-21T10:53:58.090482+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:15:21.030031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":32,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.091395+0000","last_change":"2024-09-21T10:52:25.479229+0000","last_active":"2024-09-21T10:53:58.091395+0000","last_peered":"2024-09-21T10:53:58.091395+0000","last_clean":"2024-09-21T10:53:58.091395+0000","last_became_active":"2024-09-21T10:52:25.470551+0000","last_became_peered":"2024-09-21T10:52:25.470551+0000","last_unstale":"2024-09-21T10:53:58.091395+0000","last_undegraded":"2024-09-21T10:53:58.091395+0000","last_fullsized":"2024-09-21T10:53:58.091395+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:15.458231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1503,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.092186+0000","last_change":"2024-09-21T10:51:27.517234+0000","last_active":"2024-09-21T10:53:58.092186+0000","last_peered":"2024-09-21T10:53:58.092186+0000","last_clean":"2024-09-21T10:53:58.092186+0000","last_became_active":"2024-09-21T10:51:27.431253+0000","last_became_peered":"2024-09-21T10:51:27.431253+0000","last_unstale":"2024-09-21T10:53:58.092186+0000","last_undegraded":"2024-09-21T10:53:58.092186+0000","last_fullsized":"2024-09-21T10:53:58.092186+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:35:42.249305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2227,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.373888+0000","last_change":"2024-09-21T10:52:25.497298+0000","last_active":"2024-09-21T10:54:01.373888+0000","last_peered":"2024-09-21T10:54:01.373888+0000","last_clean":"2024-09-21T10:54:01.373888+0000","last_became_active":"2024-09-21T10:52:25.496876+0000","last_became_peered":"2024-09-21T10:52:25.496876+0000","last_unstale":"2024-09-21T10:54:01.373888+0000","last_undegraded":"2024-09-21T10:54:01.373888+0000","last_fullsized":"2024-09-21T10:54:01.373888+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:16.587897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":128,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.090986+0000","last_change":"2024-09-21T10:44:14.398878+0000","last_active":"2024-09-21T10:53:58.090986+0000","last_peered":"2024-09-21T10:53:58.090986+0000","last_clean":"2024-09-21T10:53:58.090986+0000","last_became_active":"2024-09-21T10:44:14.398547+0000","last_became_peered":"2024-09-21T10:44:14.398547+0000","last_unstale":"2024-09-21T10:53:58.090986+0000","last_undegraded":"2024-09-21T10:53:58.090986+0000","last_fullsized":"2024-09-21T10:53:58.090986+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:13.008134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2276,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.376118+0000","last_change":"2024-09-21T10:52:25.498554+0000","last_active":"2024-09-21T10:54:01.376118+0000","last_peered":"2024-09-21T10:54:01.376118+0000","last_clean":"2024-09-21T10:54:01.376118+0000","last_became_active":"2024-09-21T10:52:25.498194+0000","last_became_peered":"2024-09-21T10:52:25.498194+0000","last_unstale":"2024-09-21T10:54:01.376118+0000","last_undegraded":"2024-09-21T10:54:01.376118+0000","last_fullsized":"2024-09-21T10:54:01.376118+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:47.115271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":28,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.090838+0000","last_change":"2024-09-21T10:52:25.489063+0000","last_active":"2024-09-21T10:53:58.090838+0000","last_peered":"2024-09-21T10:53:58.090838+0000","last_clean":"2024-09-21T10:53:58.090838+0000","last_became_active":"2024-09-21T10:52:25.488129+0000","last_became_peered":"2024-09-21T10:52:25.488129+0000","last_unstale":"2024-09-21T10:53:58.090838+0000","last_undegraded":"2024-09-21T10:53:58.090838+0000","last_fullsized":"2024-09-21T10:53:58.090838+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:36.996829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1443,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.091589+0000","last_change":"2024-09-21T10:49:44.864898+0000","last_active":"2024-09-21T10:53:58.091589+0000","last_peered":"2024-09-21T10:53:58.091589+0000","last_clean":"2024-09-21T10:53:58.091589+0000","last_became_active":"2024-09-21T10:49:44.862691+0000","last_became_peered":"2024-09-21T10:49:44.862691+0000","last_unstale":"2024-09-21T10:53:58.091589+0000","last_undegraded":"2024-09-21T10:53:58.091589+0000","last_fullsized":"2024-09-21T10:53:58.091589+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:57.809113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":134,"reported_epoch":1482,"state":"active+clean","last_fresh":"2024-09-21T10:53:58.090794+0000","last_change":"2024-09-21T10:44:14.418113+0000","last_active":"2024-09-21T10:53:58.090794+0000","last_peered":"2024-09-21T10:53:58.090794+0000","last_clean":"2024-09-21T10:53:58.090794+0000","last_became_active":"2024-09-21T10:44:14.398327+0000","last_became_peered":"2024-09-21T10:44:14.398327+0000","last_unstale":"2024-09-21T10:53:58.090794+0000","last_undegraded":"2024-09-21T10:53:58.090794+0000","last_fullsized":"2024-09-21T10:53:58.090794+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:08.810863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2257,"reported_epoch":1483,"state":"active+clean","last_fresh":"2024-09-21T10:54:01.374695+0000","last_change":"2024-09-21T10:44:14.398584+0000","last_active":"2024-09-21T10:54:01.374695+0000","last_peered":"2024-09-21T10:54:01.374695+0000","last_clean":"2024-09-21T10:54:01.374695+0000","last_became_active":"2024-09-21T10:44:14.398278+0000","last_became_peered":"2024-09-21T10:44:14.398278+0000","last_unstale":"2024-09-21T10:54:01.374695+0000","last_undegraded":"2024-09-21T10:54:01.374695+0000","last_fullsized":"2024-09-21T10:54:01.374695+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:19.681750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":56,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":429359,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":276,"num_read_kb":598,"num_write":157,"num_write_kb":1757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":3982889,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":33,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":11344,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":18008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234721,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":580320,"kb_used_data":11924,"kb_used_omap":1,"kb_used_meta":568382,"kb_avail":93164832,"statfs":{"total":95995035648,"available":95400787968,"internally_reserved":0,"allocated":12210176,"data_stored":7141627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":582023627},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496358,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":715108,"kb_used_data":12184,"kb_used_omap":1,"kb_used_meta":702910,"kb_avail":93030044,"statfs":{"total":95995035648,"available":95262765056,"internally_reserved":0,"allocated":12476416,"data_stored":7404804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":719780298},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757995,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019631,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":682912,"kb_used_data":20184,"kb_used_omap":1,"kb_used_meta":662718,"kb_avail":93062240,"statfs":{"total":95995035648,"available":95295733760,"internally_reserved":0,"allocated":20668416,"data_stored":15596827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":678623686},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369375,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":820376,"kb_used_data":12172,"kb_used_omap":69,"kb_used_meta":808122,"kb_avail":92924776,"statfs":{"total":95995035648,"available":95154970624,"internally_reserved":0,"allocated":12464128,"data_stored":7400116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":827517058},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2920577761830,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807958061,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":707212,"kb_used_data":11584,"kb_used_omap":66,"kb_used_meta":695549,"kb_avail":93037940,"statfs":{"total":95995035648,"available":95270850560,"internally_reserved":0,"allocated":11862016,"data_stored":6809028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68589,"internal_metadata":712242195},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":2860448219700,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:54:11.518 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph status --format=json 2024-09-21T10:54:11.612 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:11.635 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:11.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:11.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:11.736 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:11.758 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:11.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:11.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:11 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/247505441' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T10:54:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:11 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T10:54:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:11 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-21T10:54:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:11 smithi160 ceph-mon[101879]: osdmap e1484: 8 total, 8 up, 5 in 2024-09-21T10:54:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:11 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:54:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:11 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:54:11.860 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:11.881 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:11.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:11.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:11.983 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:12.004 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:54:12.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:12.006 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:12.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:12.088 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:54:12.107 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:12.129 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:12.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:12.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:12.230 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:12.252 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:12.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:12.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:12.354 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:12.375 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:12.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:12.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:12.436 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-09-21T10:54:12.477 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:12.498 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:12.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:12.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:12.600 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:12.622 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:12.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:12.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:12.723 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:12.744 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:12.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:12.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:12 smithi160 ceph-mon[101879]: from='client.45935 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:54:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:12 smithi160 ceph-mon[101879]: pgmap v2743: 99 pgs: 16 unknown, 83 active+clean; 425 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-21T10:54:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:12 smithi160 ceph-mon[101879]: osdmap e1485: 8 total, 8 up, 5 in 2024-09-21T10:54:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:12 smithi160 ceph-mon[101879]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:54:12.846 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:12.868 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:12.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:12.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:12.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:12 smithi070 ceph-mon[112139]: from='client.45935 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:54:12.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:12 smithi070 ceph-mon[112139]: pgmap v2743: 99 pgs: 16 unknown, 83 active+clean; 425 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-21T10:54:12.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:12 smithi070 ceph-mon[112139]: osdmap e1485: 8 total, 8 up, 5 in 2024-09-21T10:54:12.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:12 smithi070 ceph-mon[112139]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:54:12.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:12 smithi070 ceph-mon[107385]: from='client.45935 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:54:12.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:12 smithi070 ceph-mon[107385]: pgmap v2743: 99 pgs: 16 unknown, 83 active+clean; 425 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-21T10:54:12.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:12 smithi070 ceph-mon[107385]: osdmap e1485: 8 total, 8 up, 5 in 2024-09-21T10:54:12.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:12 smithi070 ceph-mon[107385]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:54:12.970 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:12.991 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:12.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:12.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:13.093 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:13.114 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:13.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:13.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:13.216 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:13.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:13.238 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:13.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:13.254 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:54:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:54:13] "GET /metrics HTTP/1.1" 200 34187 "" "Prometheus/2.51.0" 2024-09-21T10:54:13.339 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:13.360 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:13.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:13.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:13.462 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:13.483 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:13.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:13.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:13.583 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:13 smithi070 ceph-mon[112139]: osdmap e1486: 8 total, 8 up, 5 in 2024-09-21T10:54:13.583 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:13 smithi070 ceph-mon[107385]: osdmap e1486: 8 total, 8 up, 5 in 2024-09-21T10:54:13.585 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:13.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:13.607 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:13.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:13.708 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:13.729 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:13.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:13.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:13.831 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:13 smithi160 ceph-mon[101879]: osdmap e1486: 8 total, 8 up, 5 in 2024-09-21T10:54:13.852 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:13.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:13.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:13.954 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:13.974 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:13.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:13.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:14.076 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:14.102 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:14.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:14.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:14.204 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:14.228 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:14.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:14.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:14.330 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:14.352 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:14.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:14.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:14.454 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:14.478 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:14.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:14.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:14.580 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:14.602 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:14.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:14.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:14.613 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:14 smithi070 ceph-mon[107385]: pgmap v2746: 99 pgs: 4 creating+peering, 8 unknown, 87 active+clean; 425 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-21T10:54:14.613 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:14 smithi070 ceph-mon[112139]: pgmap v2746: 99 pgs: 4 creating+peering, 8 unknown, 87 active+clean; 425 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-21T10:54:14.704 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:14.731 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:14.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:14.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:14.746 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:54:14.747 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:54:14.832 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:14 smithi160 ceph-mon[101879]: pgmap v2746: 99 pgs: 4 creating+peering, 8 unknown, 87 active+clean; 425 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-21T10:54:14.853 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:14.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:14.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:14.955 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:14.981 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:14.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:14.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:15.083 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:15.108 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:15.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:15.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:15.210 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:15.235 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:15.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:15.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:15.337 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:15.361 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:15.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:15.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:15.463 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:15.465 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T10:54:15.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:15.489 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:15.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:15.590 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:15.825 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:54:15.825 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:54:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:15 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/507273399' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:54:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:15 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:54:15.851 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:15.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:15.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:15.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:15 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/507273399' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:54:15.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:15 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:54:15.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:15 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/507273399' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:54:15.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:15 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:54:15.953 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:15.975 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:15.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:15.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:16.077 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:16.099 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:16.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:16.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:16.201 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:16.227 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:16.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:16.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:16.329 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:16.353 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:16.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:16.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:16.454 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:16.478 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:16.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:16.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:16.580 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:16.604 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:16.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:16.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:16.706 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:16.727 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:16.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:16.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:16.810 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:16 smithi070 ceph-mon[107385]: pgmap v2747: 99 pgs: 4 creating+peering, 95 active+clean; 425 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-21T10:54:16.811 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:16 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:54:16.811 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:16 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:54:16.811 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:16 smithi070 ceph-mon[107385]: osdmap e1487: 8 total, 8 up, 5 in 2024-09-21T10:54:16.811 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:16 smithi070 ceph-mon[112139]: pgmap v2747: 99 pgs: 4 creating+peering, 95 active+clean; 425 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-21T10:54:16.811 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:16 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:54:16.811 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:16 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:54:16.811 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:16 smithi070 ceph-mon[112139]: osdmap e1487: 8 total, 8 up, 5 in 2024-09-21T10:54:16.830 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:16.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:16 smithi160 ceph-mon[101879]: pgmap v2747: 99 pgs: 4 creating+peering, 95 active+clean; 425 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-21T10:54:16.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:16 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:54:16.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:16 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:54:16.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:16 smithi160 ceph-mon[101879]: osdmap e1487: 8 total, 8 up, 5 in 2024-09-21T10:54:16.856 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:16.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:16.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:16.958 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:16.983 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:16.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:16.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:17.084 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:17.106 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:17.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:17.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:17.207 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:17.228 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:17.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:17.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:17.330 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:17.356 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:17.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:17.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:17.458 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:17.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:17.480 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:17.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:17.580 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:17.601 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:17.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:17.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:17.703 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:17.728 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:17.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:17.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:17.830 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:17.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:17 smithi160 ceph-mon[101879]: 1.1 scrub starts 2024-09-21T10:54:17.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:17 smithi160 ceph-mon[101879]: 1.1 scrub ok 2024-09-21T10:54:17.851 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:17.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:17.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:17.865 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:54:17.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:17 smithi070 ceph-mon[112139]: 1.1 scrub starts 2024-09-21T10:54:17.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:17 smithi070 ceph-mon[112139]: 1.1 scrub ok 2024-09-21T10:54:17.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:17 smithi070 ceph-mon[107385]: 1.1 scrub starts 2024-09-21T10:54:17.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:17 smithi070 ceph-mon[107385]: 1.1 scrub ok 2024-09-21T10:54:17.953 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:17.979 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:17.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:17.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:18.081 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:18.107 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:18.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:18.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:18.208 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:18.234 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:18.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:18.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:18.336 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:18.350 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:54:18.358 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:18.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:18.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:18.460 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:18.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:18.487 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:18.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:18.588 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:18.610 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:18.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:18.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:18 smithi070 ceph-mon[107385]: pgmap v2749: 99 pgs: 99 active+clean; 372 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-21T10:54:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:18 smithi070 ceph-mon[107385]: 1.9 scrub starts 2024-09-21T10:54:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:18 smithi070 ceph-mon[107385]: 1.9 scrub ok 2024-09-21T10:54:18.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:18 smithi070 ceph-mon[112139]: pgmap v2749: 99 pgs: 99 active+clean; 372 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-21T10:54:18.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:18 smithi070 ceph-mon[112139]: 1.9 scrub starts 2024-09-21T10:54:18.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:18 smithi070 ceph-mon[112139]: 1.9 scrub ok 2024-09-21T10:54:18.711 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:18.737 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:18.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:18.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:18.832 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:54:18.838 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:18 smithi160 ceph-mon[101879]: pgmap v2749: 99 pgs: 99 active+clean; 372 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-21T10:54:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:18 smithi160 ceph-mon[101879]: 1.9 scrub starts 2024-09-21T10:54:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:18 smithi160 ceph-mon[101879]: 1.9 scrub ok 2024-09-21T10:54:18.863 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:18.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:18.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:18.966 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:18.991 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:18.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:18.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:19.092 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:19.114 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:19.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:19.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:19.216 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:19.242 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:19.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:19.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:19.344 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:19.365 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:19.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:19.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:19.466 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:19.491 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:19.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:19.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:19.592 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:19.618 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:19.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:19.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:19.720 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:19.741 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:19.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:19.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:19.843 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:19.869 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:19.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:19.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:19.970 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:19.992 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:19.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:19.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:20.094 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:20.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:20.120 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:20.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:20.220 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:20.256 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:20.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:20.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:20.364 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:20.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:20.400 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:20.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:20.401 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:54:20.501 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:20.524 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:20.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:20.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:20.627 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:20.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:20 smithi070 ceph-mon[112139]: pgmap v2750: 99 pgs: 99 active+clean; 372 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-21T10:54:20.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:54:20.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:54:20.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:20 smithi070 ceph-mon[107385]: pgmap v2750: 99 pgs: 99 active+clean; 372 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-21T10:54:20.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:54:20.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:54:20.654 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:20.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:20.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:20.759 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:20.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:20.794 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:20.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:20 smithi160 ceph-mon[101879]: pgmap v2750: 99 pgs: 99 active+clean; 372 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-21T10:54:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:54:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:54:20.895 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:20.923 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:20.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:20.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:21.031 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:21.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:21.059 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:21.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:21.160 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:21.189 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:21.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:21.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:21.291 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:21.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:21.322 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:21.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:21.423 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:21.454 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:21.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:21.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:21.555 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:21.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:21.579 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:21.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:21.680 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:21.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:21.704 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:21.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:21.772 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:21 smithi070 ceph-mon[107385]: 1.0 scrub starts 2024-09-21T10:54:21.772 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:21 smithi070 ceph-mon[107385]: 1.0 scrub ok 2024-09-21T10:54:21.773 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:21 smithi070 ceph-mon[112139]: 1.0 scrub starts 2024-09-21T10:54:21.773 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:21 smithi070 ceph-mon[112139]: 1.0 scrub ok 2024-09-21T10:54:21.805 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:21.835 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:21.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:21.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:21 smithi160 ceph-mon[101879]: 1.0 scrub starts 2024-09-21T10:54:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:21 smithi160 ceph-mon[101879]: 1.0 scrub ok 2024-09-21T10:54:21.937 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:21.959 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:21.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:21.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:22.061 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:22.083 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:22.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:22.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:22.185 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:22.212 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:22.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:22.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:22.239 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:54:22.239 INFO:teuthology.orchestra.run.smithi070.stdout:{"fsid":"7c68bfbc-77fe-11ef-baf6-efdc52797490","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":3079,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":1487,"num_osds":8,"num_up_osds":8,"osd_up_since":1726913345,"num_in_osds":5,"osd_in_since":1726915886,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":99}],"num_pgs":99,"num_pools":5,"num_objects":25,"data_bytes":381262,"bytes_used":3581886464,"bytes_avail":476393291776,"bytes_total":479975178240},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi070.gansda","status":"up:active","gid":14652}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.70:8443/","prometheus":"http://172.21.15.70:9283/"}},"servicemap":{"epoch":38,"modified":"2024-09-21T10:53:54.946329+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-21T10:54:22.313 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:22.339 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:22.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:22.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:22.441 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:22.466 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:22.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:22.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:22.569 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:22.593 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:22.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:22.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:22.696 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:22.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:22.725 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:22.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:22.788 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:22 smithi070 ceph-mon[107385]: pgmap v2751: 99 pgs: 99 active+clean; 372 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-21T10:54:22.789 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:22 smithi070 ceph-mon[107385]: 1.2 scrub starts 2024-09-21T10:54:22.789 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:22 smithi070 ceph-mon[107385]: 1.2 scrub ok 2024-09-21T10:54:22.789 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:22 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4063394992' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:54:22.789 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:22 smithi070 ceph-mon[112139]: pgmap v2751: 99 pgs: 99 active+clean; 372 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-21T10:54:22.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:22 smithi070 ceph-mon[112139]: 1.2 scrub starts 2024-09-21T10:54:22.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:22 smithi070 ceph-mon[112139]: 1.2 scrub ok 2024-09-21T10:54:22.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:22 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4063394992' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:54:22.827 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:22.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:22 smithi160 ceph-mon[101879]: pgmap v2751: 99 pgs: 99 active+clean; 372 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-21T10:54:22.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:22 smithi160 ceph-mon[101879]: 1.2 scrub starts 2024-09-21T10:54:22.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:22 smithi160 ceph-mon[101879]: 1.2 scrub ok 2024-09-21T10:54:22.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:22 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4063394992' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:54:22.851 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:22.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:22.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:22.953 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:22.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:22.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:23.079 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:23.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:23.101 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:23.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:23.192 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-21T10:54:23.192 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:54:23.202 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:23.228 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:23.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:23.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:23.330 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:23.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:23.353 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:23.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:23.455 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:23.477 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:54:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:54:23] "GET /metrics HTTP/1.1" 200 36540 "" "Prometheus/2.51.0" 2024-09-21T10:54:23.477 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:23 smithi070 ceph-mon[107385]: 1.5 scrub starts 2024-09-21T10:54:23.477 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:23 smithi070 ceph-mon[107385]: 1.5 scrub ok 2024-09-21T10:54:23.477 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:23 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1008571768' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:54:23.480 INFO:teuthology.orchestra.run.smithi070.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-09-21T10:54:23.481 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:23.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:23.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:23.583 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:23.605 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:23.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:23.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:23.707 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:23.728 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:23.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:23.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:23.733 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:23 smithi070 ceph-mon[112139]: 1.5 scrub starts 2024-09-21T10:54:23.733 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:23 smithi070 ceph-mon[112139]: 1.5 scrub ok 2024-09-21T10:54:23.734 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:23 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1008571768' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:54:23.734 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:23 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:54:23.734 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:23 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:54:23.831 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:23 smithi160 ceph-mon[101879]: 1.5 scrub starts 2024-09-21T10:54:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:23 smithi160 ceph-mon[101879]: 1.5 scrub ok 2024-09-21T10:54:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:23 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1008571768' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:54:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:23 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:54:23.857 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:23.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:23.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:23.959 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:23.984 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:23.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:23.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:24.086 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:24.109 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:24.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:24.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:24.211 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:24.233 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:24.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:24.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:24.335 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:24.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:24.362 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:24.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:24.463 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:24.488 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:24.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:24.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:24.590 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:24.615 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:24.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:24.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:24.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:24 smithi070 ceph-mon[107385]: pgmap v2752: 99 pgs: 99 active+clean; 215 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-21T10:54:24.629 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:24 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:54:24.630 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:24 smithi070 ceph-mon[107385]: osdmap e1488: 8 total, 8 up, 5 in 2024-09-21T10:54:24.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:24 smithi070 ceph-mon[112139]: pgmap v2752: 99 pgs: 99 active+clean; 215 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-21T10:54:24.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:24 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:54:24.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:24 smithi070 ceph-mon[112139]: osdmap e1488: 8 total, 8 up, 5 in 2024-09-21T10:54:24.716 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:24.741 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:24.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:24 smithi160 ceph-mon[101879]: pgmap v2752: 99 pgs: 99 active+clean; 215 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-21T10:54:24.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:24 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:54:24.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:24 smithi160 ceph-mon[101879]: osdmap e1488: 8 total, 8 up, 5 in 2024-09-21T10:54:24.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:24.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:24.844 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:24.866 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:24.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:24.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:24.967 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:24.995 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:24.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:24.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:25.097 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:25.118 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:25.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:25.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:25.219 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:25.240 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:25.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:25.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:25.342 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:25.364 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:25.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:25.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:25.465 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:25.488 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T10:54:25.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:25.491 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:25.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:25.532 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:25 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4016335358' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:54:25.532 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:25 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:54:25.542 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:54:25.542 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:54:25.561 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:54:25.562 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:54:25.579 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:54:25.579 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:54:25.580 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:25 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4016335358' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:54:25.580 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:25 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:54:25.580 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:25 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4016335358' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:54:25.580 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:25 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:54:25.592 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:25.595 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:54:25.595 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:54:25.612 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:54:25.612 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:54:25.614 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:25.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:25.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:25.629 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:54:25.629 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:54:25.645 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:54:25.645 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:54:25.661 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:54:25.661 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:54:25.716 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:25.738 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:25.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:25.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:25.842 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:25.863 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:25.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:25.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:25.965 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:25.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:25.987 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:25.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:26.088 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:26.109 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:26.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:26.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:26.211 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:26.232 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:26.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:26.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:26.334 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:26.344 DEBUG:teuthology.orchestra.run.smithi070:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-09-21T10:54:26.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:26.356 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:26.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:26.357 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T10:54:26.357 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T10:54:26.458 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:26.480 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:26.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:26.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:26.547 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:hints = 1 2024-09-21T10:54:26.547 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T10:54:26.547 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Object prefix: benchmark_data_smithi070_333437 2024-09-21T10:54:26.548 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:54:26.548 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 0 0 0 0 0 0 - 0 2024-09-21T10:54:26.581 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:26.615 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:26.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:26.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:26 smithi070 ceph-mon[107385]: pgmap v2754: 99 pgs: 99 active+clean; 215 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-21T10:54:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:26 smithi070 ceph-mon[107385]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:54:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:26 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:54:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:26 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:54:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:26 smithi070 ceph-mon[107385]: osdmap e1489: 8 total, 8 up, 5 in 2024-09-21T10:54:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:26 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3605373856' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:54:26.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:26 smithi070 ceph-mon[112139]: pgmap v2754: 99 pgs: 99 active+clean; 215 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-21T10:54:26.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:26 smithi070 ceph-mon[112139]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:54:26.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:26 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:54:26.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:26 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:54:26.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:26 smithi070 ceph-mon[112139]: osdmap e1489: 8 total, 8 up, 5 in 2024-09-21T10:54:26.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:26 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3605373856' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:54:26.717 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:26.757 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:26.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:26.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:26 smithi160 ceph-mon[101879]: pgmap v2754: 99 pgs: 99 active+clean; 215 KiB data, 3.3 GiB used, 444 GiB / 447 GiB avail 2024-09-21T10:54:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:26 smithi160 ceph-mon[101879]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:54:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:26 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:54:26.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:26 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:54:26.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:26 smithi160 ceph-mon[101879]: osdmap e1489: 8 total, 8 up, 5 in 2024-09-21T10:54:26.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:26 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3605373856' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:54:26.861 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:26.899 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:26.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:26.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:26.948 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:54:26.950 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:54:27.001 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:27.039 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:27.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:27.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:27.140 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:27.177 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:27.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:27.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:27.279 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:27.315 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:27.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:27.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:27.417 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:27.450 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:27.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:27.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:27.547 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 1 16 4115 4099 256.173 256.188 0.00271848 0.00389339 2024-09-21T10:54:27.552 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:27.588 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:27.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:27.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:27.690 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:27.727 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:27.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:27.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:27.829 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:27.863 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:27.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:27.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:27.965 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:27.999 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:28.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:28.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:28.101 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:28.138 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:28.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:28.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:28.239 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:28.270 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:28.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:28.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:28.372 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:28.407 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:28.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:28.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:28.509 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:28.543 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:28.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:28.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:28.547 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 2 16 7797 7781 243.139 230.125 0.00462996 0.00410394 2024-09-21T10:54:28.645 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:28.682 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:28.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:28.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:28.715 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:54:28.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:28 smithi070 ceph-mon[107385]: pgmap v2756: 99 pgs: 99 active+clean; 5.7 MiB data, 3.4 GiB used, 444 GiB / 447 GiB avail; 602 KiB/s wr, 1 op/s 2024-09-21T10:54:28.749 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:28 smithi070 ceph-mon[112139]: pgmap v2756: 99 pgs: 99 active+clean; 5.7 MiB data, 3.4 GiB used, 444 GiB / 447 GiB avail; 602 KiB/s wr, 1 op/s 2024-09-21T10:54:28.784 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:28.824 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:28.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:28.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:28 smithi160 ceph-mon[101879]: pgmap v2756: 99 pgs: 99 active+clean; 5.7 MiB data, 3.4 GiB used, 444 GiB / 447 GiB avail; 602 KiB/s wr, 1 op/s 2024-09-21T10:54:28.926 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:28.960 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:28.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:28.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:29.061 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:29.098 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:29.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:29.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:29.200 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:29.228 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:29.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:29.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:29.330 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:29.360 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:29.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:29.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:29.462 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:29.491 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:29.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:29.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:29.529 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:54:29.548 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 3 16 11060 11044 230.052 203.938 0.00390854 0.00433984 2024-09-21T10:54:29.593 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:29.622 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:29.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:29.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:29.724 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:29.757 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:29.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:29.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:29.859 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:29.891 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:29.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:29.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:29.992 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:30.029 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:30.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:30.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:30.131 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:30.168 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:30.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:30.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:30.270 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:30.307 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:30.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:30.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:30.409 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:30.439 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:30.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:30.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:30.541 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:30.548 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 4 16 14115 14099 220.27 190.938 0.00404732 0.0045334 2024-09-21T10:54:30.578 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:30.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:30.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:30.680 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:30.716 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:30.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:30.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:30.818 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:30 smithi160 ceph-mon[101879]: pgmap v2757: 99 pgs: 99 active+clean; 125 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 16 MiB/s wr, 240 op/s 2024-09-21T10:54:30.848 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:30.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:30.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:30.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:30 smithi070 ceph-mon[112139]: pgmap v2757: 99 pgs: 99 active+clean; 125 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 16 MiB/s wr, 240 op/s 2024-09-21T10:54:30.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:30 smithi070 ceph-mon[107385]: pgmap v2757: 99 pgs: 99 active+clean; 125 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 16 MiB/s wr, 240 op/s 2024-09-21T10:54:30.951 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:30.993 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:30.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:30.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:31.095 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:31.127 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:31.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:31.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:31.228 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:31.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:31.262 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:31.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:31.363 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:31.394 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:31.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:31.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:31.495 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:31.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:31.539 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:31.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:31.548 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 5 15 17049 17034 212.901 183.438 0.00727692 0.00469256 2024-09-21T10:54:31.641 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:31.679 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:31.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:31.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:31.781 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:31.811 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:31.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:31.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:31.913 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:31.949 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:31.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:31.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:32.051 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:32.080 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:32.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:32.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:32.182 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:32.220 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:32.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:32.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:32.322 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:32.352 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:32.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:32.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:32.454 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:32.485 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:32.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:32.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:32.501 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:54:32.501 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T10:54:32.548 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 6 16 19485 19469 202.781 152.188 0.00931278 0.0049254 2024-09-21T10:54:32.588 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:32.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:32.609 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:32.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:32.710 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:32.748 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:32.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:32.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:32.772 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:32 smithi070 ceph-mon[107385]: pgmap v2758: 99 pgs: 99 active+clean; 324 MiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 40 MiB/s wr, 645 op/s 2024-09-21T10:54:32.773 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:32 smithi070 ceph-mon[112139]: pgmap v2758: 99 pgs: 99 active+clean; 324 MiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 40 MiB/s wr, 645 op/s 2024-09-21T10:54:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:32 smithi160 ceph-mon[101879]: pgmap v2758: 99 pgs: 99 active+clean; 324 MiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 40 MiB/s wr, 645 op/s 2024-09-21T10:54:32.850 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:32.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:32.872 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:32.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:32.973 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:33.005 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:33.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:33.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:33.107 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:33.144 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:33.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:33.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:33.246 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:33.275 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:33.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:33.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:33.376 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:33.407 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:33.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:33.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:33.497 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:54:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:54:33] "GET /metrics HTTP/1.1" 200 36540 "" "Prometheus/2.51.0" 2024-09-21T10:54:33.509 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:33.544 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:33.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:33.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:33.548 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 7 16 21808 21792 194.551 145.188 0.00490681 0.00513413 2024-09-21T10:54:33.646 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:33.678 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:33.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:33.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:33.780 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:33.819 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:33.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:33.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:33 smithi160 ceph-mon[101879]: from='client.36018 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:54:33.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:33 smithi070 ceph-mon[107385]: from='client.36018 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:54:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:33 smithi070 ceph-mon[112139]: from='client.36018 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:54:33.922 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:33.958 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:33.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:33.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:34.060 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:34.096 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:34.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:34.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:34.198 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:34.228 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:34.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:34.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:34.330 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:34.369 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:34.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:34.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:34.470 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:34.497 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:34.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:34.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:34.548 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 8 16 24078 24062 187.966 141.875 0.0165215 0.00531473 2024-09-21T10:54:34.599 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:34.634 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:34.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:34.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:34.736 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:34.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:34.783 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:34.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:34 smithi160 ceph-mon[101879]: pgmap v2759: 99 pgs: 99 active+clean; 835 MiB data, 6.0 GiB used, 441 GiB / 447 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-21T10:54:34.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:34 smithi070 ceph-mon[107385]: pgmap v2759: 99 pgs: 99 active+clean; 835 MiB data, 6.0 GiB used, 441 GiB / 447 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-21T10:54:34.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:34 smithi070 ceph-mon[112139]: pgmap v2759: 99 pgs: 99 active+clean; 835 MiB data, 6.0 GiB used, 441 GiB / 447 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-21T10:54:34.885 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:34.924 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:34.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:34.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:35.025 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:35.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:35.047 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:35.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:35.148 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:35.178 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:35.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:35.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:35.281 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:35.320 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:35.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:35.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:35.421 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:35.457 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:35.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:35.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:35.548 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 9 16 25963 25947 180.17 117.812 0.00602651 0.00554432 2024-09-21T10:54:35.559 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:35.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:35.583 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:35.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:35.684 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:35.713 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:35.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:35.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:35.815 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:35.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:35.847 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:35.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:35.948 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:35.977 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:35.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:35.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:36.079 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:36.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:36.118 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:36.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:36.219 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:36.259 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:36.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:36.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:36.361 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:36.389 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:36.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:36.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:36.491 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:36.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:36.524 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:36.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:36.548 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 10 16 28093 28077 175.464 133.125 0.0051271 0.00569369 2024-09-21T10:54:36.625 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:36.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:36.647 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:36.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:36.748 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:36.783 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:36.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:36.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:36.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:36 smithi070 ceph-mon[107385]: pgmap v2760: 99 pgs: 99 active+clean; 1.2 GiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-09-21T10:54:36.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:54:36.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:54:36.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:36 smithi070 ceph-mon[112139]: pgmap v2760: 99 pgs: 99 active+clean; 1.2 GiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-09-21T10:54:36.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:54:36.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:54:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:36 smithi160 ceph-mon[101879]: pgmap v2760: 99 pgs: 99 active+clean; 1.2 GiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-09-21T10:54:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:54:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:54:36.885 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:36.913 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:36.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:36.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:37.018 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:37.054 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:37.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:37.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:37.155 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:37.190 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:37.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:37.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:37.229 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":2758,"stamp":"2024-09-21T10:54:30.955432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":340232887,"num_objects":5213,"num_object_clones":0,"num_object_copies":15639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5213,"num_whiteouts":0,"num_read":104,"num_read_kb":225,"num_write":5312,"num_write_kb":332788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":2652544,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6887,"ondisk_log_size":6887,"up":297,"acting":297,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":297,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":3972968,"kb_used_data":508720,"kb_used_omap":141,"kb_used_meta":3464050,"kb_avail":464752792,"statfs":{"total":479975178240,"available":475906859008,"internally_reserved":0,"allocated":520929280,"data_stored":495374635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":144912,"internal_metadata":3547187696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":340012634,"num_objects":5189,"num_object_clones":0,"num_object_copies":15567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5189,"num_whiteouts":0,"num_read":-34,"num_read_kb":-75,"num_write":5169,"num_write_kb":331876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002465"},"pg_stats":[{"pgid":"56.f","version":"1489'532","reported_seq":553,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:28.738909+0000","last_change":"2024-09-21T10:54:11.453036+0000","last_active":"2024-09-21T10:54:28.738909+0000","last_peered":"2024-09-21T10:54:28.738909+0000","last_clean":"2024-09-21T10:54:28.738909+0000","last_became_active":"2024-09-21T10:54:11.452748+0000","last_became_peered":"2024-09-21T10:54:11.452748+0000","last_unstale":"2024-09-21T10:54:28.738909+0000","last_undegraded":"2024-09-21T10:54:28.738909+0000","last_fullsized":"2024-09-21T10:54:28.738909+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:04:42.219393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34865152,"num_objects":532,"num_object_clones":0,"num_object_copies":1596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":532,"num_write_kb":34048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.e","version":"1489'120","reported_seq":134,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:27.013557+0000","last_change":"2024-09-21T10:54:11.452866+0000","last_active":"2024-09-21T10:54:27.013557+0000","last_peered":"2024-09-21T10:54:27.013557+0000","last_clean":"2024-09-21T10:54:27.013557+0000","last_became_active":"2024-09-21T10:54:11.452553+0000","last_became_peered":"2024-09-21T10:54:11.452553+0000","last_unstale":"2024-09-21T10:54:27.013557+0000","last_undegraded":"2024-09-21T10:54:27.013557+0000","last_fullsized":"2024-09-21T10:54:27.013557+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:02:31.756582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"56.d","version":"1489'636","reported_seq":657,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:29.183376+0000","last_change":"2024-09-21T10:54:11.455827+0000","last_active":"2024-09-21T10:54:29.183376+0000","last_peered":"2024-09-21T10:54:29.183376+0000","last_clean":"2024-09-21T10:54:29.183376+0000","last_became_active":"2024-09-21T10:54:11.455499+0000","last_became_peered":"2024-09-21T10:54:11.455499+0000","last_unstale":"2024-09-21T10:54:29.183376+0000","last_undegraded":"2024-09-21T10:54:29.183376+0000","last_fullsized":"2024-09-21T10:54:29.183376+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":636,"log_dups_size":0,"ondisk_log_size":636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:58:18.441343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41680896,"num_objects":636,"num_object_clones":0,"num_object_copies":1908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":636,"num_write_kb":40704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.c","version":"1489'493","reported_seq":514,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:28.739186+0000","last_change":"2024-09-21T10:54:11.452969+0000","last_active":"2024-09-21T10:54:28.739186+0000","last_peered":"2024-09-21T10:54:28.739186+0000","last_clean":"2024-09-21T10:54:28.739186+0000","last_became_active":"2024-09-21T10:54:11.452637+0000","last_became_peered":"2024-09-21T10:54:11.452637+0000","last_unstale":"2024-09-21T10:54:28.739186+0000","last_undegraded":"2024-09-21T10:54:28.739186+0000","last_fullsized":"2024-09-21T10:54:28.739186+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":493,"log_dups_size":0,"ondisk_log_size":493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:51:29.279090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32309248,"num_objects":493,"num_object_clones":0,"num_object_copies":1479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":493,"num_write_kb":31552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.b","version":"0'0","reported_seq":14,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806050+0000","last_change":"2024-09-21T10:54:11.455486+0000","last_active":"2024-09-21T10:54:23.806050+0000","last_peered":"2024-09-21T10:54:23.806050+0000","last_clean":"2024-09-21T10:54:23.806050+0000","last_became_active":"2024-09-21T10:54:11.455190+0000","last_became_peered":"2024-09-21T10:54:11.455190+0000","last_unstale":"2024-09-21T10:54:23.806050+0000","last_undegraded":"2024-09-21T10:54:23.806050+0000","last_fullsized":"2024-09-21T10:54:23.806050+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:50:03.321568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"56.a","version":"1489'525","reported_seq":546,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:28.739353+0000","last_change":"2024-09-21T10:54:11.454454+0000","last_active":"2024-09-21T10:54:28.739353+0000","last_peered":"2024-09-21T10:54:28.739353+0000","last_clean":"2024-09-21T10:54:28.739353+0000","last_became_active":"2024-09-21T10:54:11.454164+0000","last_became_peered":"2024-09-21T10:54:11.454164+0000","last_unstale":"2024-09-21T10:54:28.739353+0000","last_undegraded":"2024-09-21T10:54:28.739353+0000","last_fullsized":"2024-09-21T10:54:28.739353+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":525,"log_dups_size":0,"ondisk_log_size":525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:57:14.569670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34406400,"num_objects":525,"num_object_clones":0,"num_object_copies":1575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":525,"num_write_kb":33600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.9","version":"0'0","reported_seq":14,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.805824+0000","last_change":"2024-09-21T10:54:11.455480+0000","last_active":"2024-09-21T10:54:23.805824+0000","last_peered":"2024-09-21T10:54:23.805824+0000","last_clean":"2024-09-21T10:54:23.805824+0000","last_became_active":"2024-09-21T10:54:11.455158+0000","last_became_peered":"2024-09-21T10:54:11.455158+0000","last_unstale":"2024-09-21T10:54:23.805824+0000","last_undegraded":"2024-09-21T10:54:23.805824+0000","last_fullsized":"2024-09-21T10:54:23.805824+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:51:41.079284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"56.8","version":"1489'89","reported_seq":103,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:26.840673+0000","last_change":"2024-09-21T10:54:11.452926+0000","last_active":"2024-09-21T10:54:26.840673+0000","last_peered":"2024-09-21T10:54:26.840673+0000","last_clean":"2024-09-21T10:54:26.840673+0000","last_became_active":"2024-09-21T10:54:11.452630+0000","last_became_peered":"2024-09-21T10:54:11.452630+0000","last_unstale":"2024-09-21T10:54:26.840673+0000","last_undegraded":"2024-09-21T10:54:26.840673+0000","last_fullsized":"2024-09-21T10:54:26.840673+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:07:29.362532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5832704,"num_objects":89,"num_object_clones":0,"num_object_copies":267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89,"num_write_kb":5696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"56.7","version":"0'0","reported_seq":14,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806707+0000","last_change":"2024-09-21T10:54:11.454878+0000","last_active":"2024-09-21T10:54:23.806707+0000","last_peered":"2024-09-21T10:54:23.806707+0000","last_clean":"2024-09-21T10:54:23.806707+0000","last_became_active":"2024-09-21T10:54:11.454601+0000","last_became_peered":"2024-09-21T10:54:11.454601+0000","last_unstale":"2024-09-21T10:54:23.806707+0000","last_undegraded":"2024-09-21T10:54:23.806707+0000","last_fullsized":"2024-09-21T10:54:23.806707+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:17:37.975888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"56.6","version":"1489'127","reported_seq":141,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:27.022795+0000","last_change":"2024-09-21T10:54:11.453195+0000","last_active":"2024-09-21T10:54:27.022795+0000","last_peered":"2024-09-21T10:54:27.022795+0000","last_clean":"2024-09-21T10:54:27.022795+0000","last_became_active":"2024-09-21T10:54:11.452814+0000","last_became_peered":"2024-09-21T10:54:11.452814+0000","last_unstale":"2024-09-21T10:54:27.022795+0000","last_undegraded":"2024-09-21T10:54:27.022795+0000","last_fullsized":"2024-09-21T10:54:27.022795+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:41:21.445831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8323072,"num_objects":127,"num_object_clones":0,"num_object_copies":381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":127,"num_write_kb":8128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"56.5","version":"1489'632","reported_seq":653,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:29.183606+0000","last_change":"2024-09-21T10:54:11.454780+0000","last_active":"2024-09-21T10:54:29.183606+0000","last_peered":"2024-09-21T10:54:29.183606+0000","last_clean":"2024-09-21T10:54:29.183606+0000","last_became_active":"2024-09-21T10:54:11.454545+0000","last_became_peered":"2024-09-21T10:54:11.454545+0000","last_unstale":"2024-09-21T10:54:29.183606+0000","last_undegraded":"2024-09-21T10:54:29.183606+0000","last_fullsized":"2024-09-21T10:54:29.183606+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:55:33.056608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41418752,"num_objects":632,"num_object_clones":0,"num_object_copies":1896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":632,"num_write_kb":40448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.4","version":"0'0","reported_seq":14,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806159+0000","last_change":"2024-09-21T10:54:11.454777+0000","last_active":"2024-09-21T10:54:23.806159+0000","last_peered":"2024-09-21T10:54:23.806159+0000","last_clean":"2024-09-21T10:54:23.806159+0000","last_became_active":"2024-09-21T10:54:11.454490+0000","last_became_peered":"2024-09-21T10:54:11.454490+0000","last_unstale":"2024-09-21T10:54:23.806159+0000","last_undegraded":"2024-09-21T10:54:23.806159+0000","last_fullsized":"2024-09-21T10:54:23.806159+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:59.530447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"56.3","version":"1489'119","reported_seq":133,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:27.016603+0000","last_change":"2024-09-21T10:54:11.452259+0000","last_active":"2024-09-21T10:54:27.016603+0000","last_peered":"2024-09-21T10:54:27.016603+0000","last_clean":"2024-09-21T10:54:27.016603+0000","last_became_active":"2024-09-21T10:54:11.451997+0000","last_became_peered":"2024-09-21T10:54:11.451997+0000","last_unstale":"2024-09-21T10:54:27.016603+0000","last_undegraded":"2024-09-21T10:54:27.016603+0000","last_fullsized":"2024-09-21T10:54:27.016603+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:52:22.227114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7798784,"num_objects":119,"num_object_clones":0,"num_object_copies":357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119,"num_write_kb":7616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"56.2","version":"1489'632","reported_seq":653,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:29.177856+0000","last_change":"2024-09-21T10:54:11.451633+0000","last_active":"2024-09-21T10:54:29.177856+0000","last_peered":"2024-09-21T10:54:29.177856+0000","last_clean":"2024-09-21T10:54:29.177856+0000","last_became_active":"2024-09-21T10:54:11.451333+0000","last_became_peered":"2024-09-21T10:54:11.451333+0000","last_unstale":"2024-09-21T10:54:29.177856+0000","last_undegraded":"2024-09-21T10:54:29.177856+0000","last_fullsized":"2024-09-21T10:54:29.177856+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:17:20.683870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41418752,"num_objects":632,"num_object_clones":0,"num_object_copies":1896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":632,"num_write_kb":40448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.1","version":"1489'611","reported_seq":632,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:29.182925+0000","last_change":"2024-09-21T10:54:11.451499+0000","last_active":"2024-09-21T10:54:29.182925+0000","last_peered":"2024-09-21T10:54:29.182925+0000","last_clean":"2024-09-21T10:54:29.182925+0000","last_became_active":"2024-09-21T10:54:11.451204+0000","last_became_peered":"2024-09-21T10:54:11.451204+0000","last_unstale":"2024-09-21T10:54:29.182925+0000","last_undegraded":"2024-09-21T10:54:29.182925+0000","last_fullsized":"2024-09-21T10:54:29.182925+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":611,"log_dups_size":0,"ondisk_log_size":611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:43.059873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40042496,"num_objects":611,"num_object_clones":0,"num_object_copies":1833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":611,"num_write_kb":39104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.0","version":"1489'673","reported_seq":694,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:29.180347+0000","last_change":"2024-09-21T10:54:11.455255+0000","last_active":"2024-09-21T10:54:29.180347+0000","last_peered":"2024-09-21T10:54:29.180347+0000","last_clean":"2024-09-21T10:54:29.180347+0000","last_became_active":"2024-09-21T10:54:11.454741+0000","last_became_peered":"2024-09-21T10:54:11.454741+0000","last_unstale":"2024-09-21T10:54:29.180347+0000","last_undegraded":"2024-09-21T10:54:29.180347+0000","last_fullsized":"2024-09-21T10:54:29.180347+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:42:08.583105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44105728,"num_objects":673,"num_object_clones":0,"num_object_copies":2019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":673,"num_write_kb":43072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":1980,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.867445+0000","last_change":"2024-09-21T10:52:03.323899+0000","last_active":"2024-09-21T10:54:23.867445+0000","last_peered":"2024-09-21T10:54:23.867445+0000","last_clean":"2024-09-21T10:54:23.867445+0000","last_became_active":"2024-09-21T10:52:03.314626+0000","last_became_peered":"2024-09-21T10:52:03.314626+0000","last_unstale":"2024-09-21T10:54:23.867445+0000","last_undegraded":"2024-09-21T10:54:23.867445+0000","last_fullsized":"2024-09-21T10:54:23.867445+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:13.939806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1479,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.805928+0000","last_change":"2024-09-21T10:52:15.396755+0000","last_active":"2024-09-21T10:54:23.805928+0000","last_peered":"2024-09-21T10:54:23.805928+0000","last_clean":"2024-09-21T10:54:23.805928+0000","last_became_active":"2024-09-21T10:52:15.396417+0000","last_became_peered":"2024-09-21T10:52:15.396417+0000","last_unstale":"2024-09-21T10:54:23.805928+0000","last_undegraded":"2024-09-21T10:54:23.805928+0000","last_fullsized":"2024-09-21T10:54:23.805928+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:25.005574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":2288,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.567519+0000","last_change":"2024-09-21T10:52:15.400227+0000","last_active":"2024-09-21T10:54:25.567519+0000","last_peered":"2024-09-21T10:54:25.567519+0000","last_clean":"2024-09-21T10:54:25.567519+0000","last_became_active":"2024-09-21T10:52:15.399888+0000","last_became_peered":"2024-09-21T10:52:15.399888+0000","last_unstale":"2024-09-21T10:54:25.567519+0000","last_undegraded":"2024-09-21T10:54:25.567519+0000","last_fullsized":"2024-09-21T10:54:25.567519+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:42:52.565184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2304,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.566827+0000","last_change":"2024-09-21T10:51:27.458216+0000","last_active":"2024-09-21T10:54:25.566827+0000","last_peered":"2024-09-21T10:54:25.566827+0000","last_clean":"2024-09-21T10:54:25.566827+0000","last_became_active":"2024-09-21T10:51:27.457948+0000","last_became_peered":"2024-09-21T10:51:27.457948+0000","last_unstale":"2024-09-21T10:54:25.566827+0000","last_undegraded":"2024-09-21T10:54:25.566827+0000","last_fullsized":"2024-09-21T10:54:25.566827+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:25:29.584079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":118,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806575+0000","last_change":"2024-09-21T10:49:44.878242+0000","last_active":"2024-09-21T10:54:23.806575+0000","last_peered":"2024-09-21T10:54:23.806575+0000","last_clean":"2024-09-21T10:54:23.806575+0000","last_became_active":"2024-09-21T10:49:44.866578+0000","last_became_peered":"2024-09-21T10:49:44.866578+0000","last_unstale":"2024-09-21T10:54:23.806575+0000","last_undegraded":"2024-09-21T10:54:23.806575+0000","last_fullsized":"2024-09-21T10:54:23.806575+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:17:24.145551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1484,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806205+0000","last_change":"2024-09-21T10:51:27.460539+0000","last_active":"2024-09-21T10:54:23.806205+0000","last_peered":"2024-09-21T10:54:23.806205+0000","last_clean":"2024-09-21T10:54:23.806205+0000","last_became_active":"2024-09-21T10:51:27.460211+0000","last_became_peered":"2024-09-21T10:51:27.460211+0000","last_unstale":"2024-09-21T10:54:23.806205+0000","last_undegraded":"2024-09-21T10:54:23.806205+0000","last_fullsized":"2024-09-21T10:54:23.806205+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:14:15.700998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":157,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806404+0000","last_change":"2024-09-21T10:52:15.405886+0000","last_active":"2024-09-21T10:54:23.806404+0000","last_peered":"2024-09-21T10:54:23.806404+0000","last_clean":"2024-09-21T10:54:23.806404+0000","last_became_active":"2024-09-21T10:52:15.397317+0000","last_became_peered":"2024-09-21T10:52:15.397317+0000","last_unstale":"2024-09-21T10:54:23.806404+0000","last_undegraded":"2024-09-21T10:54:23.806404+0000","last_fullsized":"2024-09-21T10:54:23.806404+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:43:48.813861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1475,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806637+0000","last_change":"2024-09-21T10:09:06.560175+0000","last_active":"2024-09-21T10:54:23.806637+0000","last_peered":"2024-09-21T10:54:23.806637+0000","last_clean":"2024-09-21T10:54:23.806637+0000","last_became_active":"2024-09-21T10:09:06.558584+0000","last_became_peered":"2024-09-21T10:09:06.558584+0000","last_unstale":"2024-09-21T10:54:23.806637+0000","last_undegraded":"2024-09-21T10:54:23.806637+0000","last_fullsized":"2024-09-21T10:54:23.806637+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:02.918339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":66,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806233+0000","last_change":"2024-09-21T10:51:27.437665+0000","last_active":"2024-09-21T10:54:23.806233+0000","last_peered":"2024-09-21T10:54:23.806233+0000","last_clean":"2024-09-21T10:54:23.806233+0000","last_became_active":"2024-09-21T10:51:27.436481+0000","last_became_peered":"2024-09-21T10:51:27.436481+0000","last_unstale":"2024-09-21T10:54:23.806233+0000","last_undegraded":"2024-09-21T10:54:23.806233+0000","last_fullsized":"2024-09-21T10:54:23.806233+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:07:50.267297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1059,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.555771+0000","last_change":"2024-09-21T10:51:27.459033+0000","last_active":"2024-09-21T10:54:25.555771+0000","last_peered":"2024-09-21T10:54:25.555771+0000","last_clean":"2024-09-21T10:54:25.555771+0000","last_became_active":"2024-09-21T10:51:27.458728+0000","last_became_peered":"2024-09-21T10:51:27.458728+0000","last_unstale":"2024-09-21T10:54:25.555771+0000","last_undegraded":"2024-09-21T10:54:25.555771+0000","last_fullsized":"2024-09-21T10:54:25.555771+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:20.133032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":1504,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806480+0000","last_change":"2024-09-21T10:52:25.473550+0000","last_active":"2024-09-21T10:54:23.806480+0000","last_peered":"2024-09-21T10:54:23.806480+0000","last_clean":"2024-09-21T10:54:23.806480+0000","last_became_active":"2024-09-21T10:52:25.473105+0000","last_became_peered":"2024-09-21T10:52:25.473105+0000","last_unstale":"2024-09-21T10:54:23.806480+0000","last_undegraded":"2024-09-21T10:54:23.806480+0000","last_fullsized":"2024-09-21T10:54:23.806480+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:15:45.352518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2284,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.567332+0000","last_change":"2024-09-21T10:40:13.432531+0000","last_active":"2024-09-21T10:54:25.567332+0000","last_peered":"2024-09-21T10:54:25.567332+0000","last_clean":"2024-09-21T10:54:25.567332+0000","last_became_active":"2024-09-21T10:40:13.432210+0000","last_became_peered":"2024-09-21T10:40:13.432210+0000","last_unstale":"2024-09-21T10:54:25.567332+0000","last_undegraded":"2024-09-21T10:54:25.567332+0000","last_fullsized":"2024-09-21T10:54:25.567332+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:20:10.998607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2256,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.554976+0000","last_change":"2024-09-21T10:09:06.530394+0000","last_active":"2024-09-21T10:54:25.554976+0000","last_peered":"2024-09-21T10:54:25.554976+0000","last_clean":"2024-09-21T10:54:25.554976+0000","last_became_active":"2024-09-21T10:09:06.529307+0000","last_became_peered":"2024-09-21T10:09:06.529307+0000","last_unstale":"2024-09-21T10:54:25.554976+0000","last_undegraded":"2024-09-21T10:54:25.554976+0000","last_fullsized":"2024-09-21T10:54:25.554976+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:54.287397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":35,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.486419+0000","last_change":"2024-09-21T10:52:25.489896+0000","last_active":"2024-09-21T10:54:25.486419+0000","last_peered":"2024-09-21T10:54:25.486419+0000","last_clean":"2024-09-21T10:54:25.486419+0000","last_became_active":"2024-09-21T10:52:25.489154+0000","last_became_peered":"2024-09-21T10:52:25.489154+0000","last_unstale":"2024-09-21T10:54:25.486419+0000","last_undegraded":"2024-09-21T10:54:25.486419+0000","last_fullsized":"2024-09-21T10:54:25.486419+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:36:02.076931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2284,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.566261+0000","last_change":"2024-09-21T10:52:25.493309+0000","last_active":"2024-09-21T10:54:25.566261+0000","last_peered":"2024-09-21T10:54:25.566261+0000","last_clean":"2024-09-21T10:54:25.566261+0000","last_became_active":"2024-09-21T10:52:25.492568+0000","last_became_peered":"2024-09-21T10:52:25.492568+0000","last_unstale":"2024-09-21T10:54:25.566261+0000","last_undegraded":"2024-09-21T10:54:25.566261+0000","last_fullsized":"2024-09-21T10:54:25.566261+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:19:19.393370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1492,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.867484+0000","last_change":"2024-09-21T10:52:03.317474+0000","last_active":"2024-09-21T10:54:23.867484+0000","last_peered":"2024-09-21T10:54:23.867484+0000","last_clean":"2024-09-21T10:54:23.867484+0000","last_became_active":"2024-09-21T10:52:03.317001+0000","last_became_peered":"2024-09-21T10:52:03.317001+0000","last_unstale":"2024-09-21T10:54:23.867484+0000","last_undegraded":"2024-09-21T10:54:23.867484+0000","last_fullsized":"2024-09-21T10:54:23.867484+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:06:07.039763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2284,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.566904+0000","last_change":"2024-09-21T10:52:15.412246+0000","last_active":"2024-09-21T10:54:25.566904+0000","last_peered":"2024-09-21T10:54:25.566904+0000","last_clean":"2024-09-21T10:54:25.566904+0000","last_became_active":"2024-09-21T10:52:15.406204+0000","last_became_peered":"2024-09-21T10:52:15.406204+0000","last_unstale":"2024-09-21T10:54:25.566904+0000","last_undegraded":"2024-09-21T10:54:25.566904+0000","last_fullsized":"2024-09-21T10:54:25.566904+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:03.524324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2264,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.567740+0000","last_change":"2024-09-21T10:52:15.402822+0000","last_active":"2024-09-21T10:54:25.567740+0000","last_peered":"2024-09-21T10:54:25.567740+0000","last_clean":"2024-09-21T10:54:25.567740+0000","last_became_active":"2024-09-21T10:52:15.402548+0000","last_became_peered":"2024-09-21T10:52:15.402548+0000","last_unstale":"2024-09-21T10:54:25.567740+0000","last_undegraded":"2024-09-21T10:54:25.567740+0000","last_fullsized":"2024-09-21T10:54:25.567740+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:04:04.667598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2289,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.566121+0000","last_change":"2024-09-21T10:52:25.499172+0000","last_active":"2024-09-21T10:54:25.566121+0000","last_peered":"2024-09-21T10:54:25.566121+0000","last_clean":"2024-09-21T10:54:25.566121+0000","last_became_active":"2024-09-21T10:52:25.498835+0000","last_became_peered":"2024-09-21T10:52:25.498835+0000","last_unstale":"2024-09-21T10:54:25.566121+0000","last_undegraded":"2024-09-21T10:54:25.566121+0000","last_fullsized":"2024-09-21T10:54:25.566121+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:10:12.434470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":34,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.479090+0000","last_change":"2024-09-21T10:52:25.489774+0000","last_active":"2024-09-21T10:54:23.479090+0000","last_peered":"2024-09-21T10:54:23.479090+0000","last_clean":"2024-09-21T10:54:23.479090+0000","last_became_active":"2024-09-21T10:52:25.487788+0000","last_became_peered":"2024-09-21T10:52:25.487788+0000","last_unstale":"2024-09-21T10:54:23.479090+0000","last_undegraded":"2024-09-21T10:54:23.479090+0000","last_fullsized":"2024-09-21T10:54:23.479090+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:48.842419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":40,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.478348+0000","last_change":"2024-09-21T10:52:25.565465+0000","last_active":"2024-09-21T10:54:23.478348+0000","last_peered":"2024-09-21T10:54:23.478348+0000","last_clean":"2024-09-21T10:54:23.478348+0000","last_became_active":"2024-09-21T10:52:25.510131+0000","last_became_peered":"2024-09-21T10:52:25.510131+0000","last_unstale":"2024-09-21T10:54:23.478348+0000","last_undegraded":"2024-09-21T10:54:23.478348+0000","last_fullsized":"2024-09-21T10:54:23.478348+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:41:13.870503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":450,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2290,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.554240+0000","last_change":"2024-09-21T10:52:25.504787+0000","last_active":"2024-09-21T10:54:25.554240+0000","last_peered":"2024-09-21T10:54:25.554240+0000","last_clean":"2024-09-21T10:54:25.554240+0000","last_became_active":"2024-09-21T10:52:25.504198+0000","last_became_peered":"2024-09-21T10:52:25.504198+0000","last_unstale":"2024-09-21T10:54:25.554240+0000","last_undegraded":"2024-09-21T10:54:25.554240+0000","last_fullsized":"2024-09-21T10:54:25.554240+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:38:25.666224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":1509,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806577+0000","last_change":"2024-09-21T10:44:14.415264+0000","last_active":"2024-09-21T10:54:23.806577+0000","last_peered":"2024-09-21T10:54:23.806577+0000","last_clean":"2024-09-21T10:54:23.806577+0000","last_became_active":"2024-09-21T10:44:14.407813+0000","last_became_peered":"2024-09-21T10:44:14.407813+0000","last_unstale":"2024-09-21T10:54:23.806577+0000","last_undegraded":"2024-09-21T10:54:23.806577+0000","last_fullsized":"2024-09-21T10:54:23.806577+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:55.933601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1462,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806610+0000","last_change":"2024-09-21T10:07:56.001756+0000","last_active":"2024-09-21T10:54:23.806610+0000","last_peered":"2024-09-21T10:54:23.806610+0000","last_clean":"2024-09-21T10:54:23.806610+0000","last_became_active":"2024-09-21T10:07:56.000789+0000","last_became_peered":"2024-09-21T10:07:56.000789+0000","last_unstale":"2024-09-21T10:54:23.806610+0000","last_undegraded":"2024-09-21T10:54:23.806610+0000","last_fullsized":"2024-09-21T10:54:23.806610+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:56:04.302902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"346'149","reported_seq":952,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.567087+0000","last_change":"2024-09-21T10:53:47.069462+0000","last_active":"2024-09-21T10:54:25.567087+0000","last_peered":"2024-09-21T10:54:25.567087+0000","last_clean":"2024-09-21T10:54:25.567087+0000","last_became_active":"2024-09-21T10:53:47.069113+0000","last_became_peered":"2024-09-21T10:53:47.069113+0000","last_unstale":"2024-09-21T10:54:25.567087+0000","last_undegraded":"2024-09-21T10:54:25.567087+0000","last_fullsized":"2024-09-21T10:54:25.567087+0000","mapping_epoch":1478,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1479,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:55:05.610284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1579069,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1477,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806060+0000","last_change":"2024-09-21T10:52:25.473127+0000","last_active":"2024-09-21T10:54:23.806060+0000","last_peered":"2024-09-21T10:54:23.806060+0000","last_clean":"2024-09-21T10:54:23.806060+0000","last_became_active":"2024-09-21T10:52:25.472805+0000","last_became_peered":"2024-09-21T10:52:25.472805+0000","last_unstale":"2024-09-21T10:54:23.806060+0000","last_undegraded":"2024-09-21T10:54:23.806060+0000","last_fullsized":"2024-09-21T10:54:23.806060+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:13.038797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1523,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.567861+0000","last_change":"2024-09-21T10:52:25.493263+0000","last_active":"2024-09-21T10:54:25.567861+0000","last_peered":"2024-09-21T10:54:25.567861+0000","last_clean":"2024-09-21T10:54:25.567861+0000","last_became_active":"2024-09-21T10:52:25.492970+0000","last_became_peered":"2024-09-21T10:52:25.492970+0000","last_unstale":"2024-09-21T10:54:25.567861+0000","last_undegraded":"2024-09-21T10:54:25.567861+0000","last_fullsized":"2024-09-21T10:54:25.567861+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:43:40.305783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":135,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.554426+0000","last_change":"2024-09-21T10:52:25.513834+0000","last_active":"2024-09-21T10:54:25.554426+0000","last_peered":"2024-09-21T10:54:25.554426+0000","last_clean":"2024-09-21T10:54:25.554426+0000","last_became_active":"2024-09-21T10:52:25.475423+0000","last_became_peered":"2024-09-21T10:52:25.475423+0000","last_unstale":"2024-09-21T10:54:25.554426+0000","last_undegraded":"2024-09-21T10:54:25.554426+0000","last_fullsized":"2024-09-21T10:54:25.554426+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:10.216527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"346'149","reported_seq":921,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806319+0000","last_change":"2024-09-21T10:54:22.051517+0000","last_active":"2024-09-21T10:54:23.806319+0000","last_peered":"2024-09-21T10:54:23.806319+0000","last_clean":"2024-09-21T10:54:23.806319+0000","last_became_active":"2024-09-21T10:52:15.405544+0000","last_became_peered":"2024-09-21T10:52:15.405544+0000","last_unstale":"2024-09-21T10:54:23.806319+0000","last_undegraded":"2024-09-21T10:54:23.806319+0000","last_fullsized":"2024-09-21T10:54:23.806319+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:22.051403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:22.051403+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:15:18.487929+0000","scrub_duration":0.0016142979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2256,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.566257+0000","last_change":"2024-09-21T10:52:25.494316+0000","last_active":"2024-09-21T10:54:25.566257+0000","last_peered":"2024-09-21T10:54:25.566257+0000","last_clean":"2024-09-21T10:54:25.566257+0000","last_became_active":"2024-09-21T10:52:25.494016+0000","last_became_peered":"2024-09-21T10:52:25.494016+0000","last_unstale":"2024-09-21T10:54:25.566257+0000","last_undegraded":"2024-09-21T10:54:25.566257+0000","last_fullsized":"2024-09-21T10:54:25.566257+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:07.859584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2257,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.555649+0000","last_change":"2024-09-21T10:07:56.006236+0000","last_active":"2024-09-21T10:54:25.555649+0000","last_peered":"2024-09-21T10:54:25.555649+0000","last_clean":"2024-09-21T10:54:25.555649+0000","last_became_active":"2024-09-21T10:07:56.004439+0000","last_became_peered":"2024-09-21T10:07:56.004439+0000","last_unstale":"2024-09-21T10:54:25.555649+0000","last_undegraded":"2024-09-21T10:54:25.555649+0000","last_fullsized":"2024-09-21T10:54:25.555649+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:39.422467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":2306,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.566228+0000","last_change":"2024-09-21T10:52:25.536401+0000","last_active":"2024-09-21T10:54:25.566228+0000","last_peered":"2024-09-21T10:54:25.566228+0000","last_clean":"2024-09-21T10:54:25.566228+0000","last_became_active":"2024-09-21T10:52:25.496480+0000","last_became_peered":"2024-09-21T10:52:25.496480+0000","last_unstale":"2024-09-21T10:54:25.566228+0000","last_undegraded":"2024-09-21T10:54:25.566228+0000","last_fullsized":"2024-09-21T10:54:25.566228+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:50:34.820373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"346'149","reported_seq":734,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.867336+0000","last_change":"2024-09-21T10:51:30.450301+0000","last_active":"2024-09-21T10:54:23.867336+0000","last_peered":"2024-09-21T10:54:23.867336+0000","last_clean":"2024-09-21T10:54:23.867336+0000","last_became_active":"2024-09-21T10:51:30.449925+0000","last_became_peered":"2024-09-21T10:51:30.449925+0000","last_unstale":"2024-09-21T10:54:23.867336+0000","last_undegraded":"2024-09-21T10:54:23.867336+0000","last_fullsized":"2024-09-21T10:54:23.867336+0000","mapping_epoch":1450,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1451,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:41.011085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:41.011085+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:38:33.451261+0000","scrub_duration":0.0061250310000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2205,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.478382+0000","last_change":"2024-09-21T10:51:27.431416+0000","last_active":"2024-09-21T10:54:23.478382+0000","last_peered":"2024-09-21T10:54:23.478382+0000","last_clean":"2024-09-21T10:54:23.478382+0000","last_became_active":"2024-09-21T10:51:27.431127+0000","last_became_peered":"2024-09-21T10:51:27.431127+0000","last_unstale":"2024-09-21T10:54:23.478382+0000","last_undegraded":"2024-09-21T10:54:23.478382+0000","last_fullsized":"2024-09-21T10:54:23.478382+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:27.721665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":34,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.478232+0000","last_change":"2024-09-21T10:52:25.468849+0000","last_active":"2024-09-21T10:54:23.478232+0000","last_peered":"2024-09-21T10:54:23.478232+0000","last_clean":"2024-09-21T10:54:23.478232+0000","last_became_active":"2024-09-21T10:52:25.468552+0000","last_became_peered":"2024-09-21T10:52:25.468552+0000","last_unstale":"2024-09-21T10:54:23.478232+0000","last_undegraded":"2024-09-21T10:54:23.478232+0000","last_fullsized":"2024-09-21T10:54:23.478232+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:36:41.692028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2307,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.555712+0000","last_change":"2024-09-21T10:51:27.436608+0000","last_active":"2024-09-21T10:54:25.555712+0000","last_peered":"2024-09-21T10:54:25.555712+0000","last_clean":"2024-09-21T10:54:25.555712+0000","last_became_active":"2024-09-21T10:51:27.434949+0000","last_became_peered":"2024-09-21T10:51:27.434949+0000","last_unstale":"2024-09-21T10:54:25.555712+0000","last_undegraded":"2024-09-21T10:54:25.555712+0000","last_fullsized":"2024-09-21T10:54:25.555712+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:49:23.439425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"346'149","reported_seq":777,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.867417+0000","last_change":"2024-09-21T10:52:28.590793+0000","last_active":"2024-09-21T10:54:23.867417+0000","last_peered":"2024-09-21T10:54:23.867417+0000","last_clean":"2024-09-21T10:54:23.867417+0000","last_became_active":"2024-09-21T10:52:28.588662+0000","last_became_peered":"2024-09-21T10:52:28.588662+0000","last_unstale":"2024-09-21T10:54:23.867417+0000","last_undegraded":"2024-09-21T10:54:23.867417+0000","last_fullsized":"2024-09-21T10:54:23.867417+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:44.018645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:44.018645+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:46.849427+0000","scrub_duration":0.0012789399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2262,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.555301+0000","last_change":"2024-09-21T10:51:27.440425+0000","last_active":"2024-09-21T10:54:25.555301+0000","last_peered":"2024-09-21T10:54:25.555301+0000","last_clean":"2024-09-21T10:54:25.555301+0000","last_became_active":"2024-09-21T10:51:27.439666+0000","last_became_peered":"2024-09-21T10:51:27.439666+0000","last_unstale":"2024-09-21T10:54:25.555301+0000","last_undegraded":"2024-09-21T10:54:25.555301+0000","last_fullsized":"2024-09-21T10:54:25.555301+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:56:47.410516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2330,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.555603+0000","last_change":"2024-09-21T10:52:03.314926+0000","last_active":"2024-09-21T10:54:25.555603+0000","last_peered":"2024-09-21T10:54:25.555603+0000","last_clean":"2024-09-21T10:54:25.555603+0000","last_became_active":"2024-09-21T10:52:03.314602+0000","last_became_peered":"2024-09-21T10:52:03.314602+0000","last_unstale":"2024-09-21T10:54:25.555603+0000","last_undegraded":"2024-09-21T10:54:25.555603+0000","last_fullsized":"2024-09-21T10:54:25.555603+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:13.774521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1485,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.566276+0000","last_change":"2024-09-21T10:52:25.502830+0000","last_active":"2024-09-21T10:54:25.566276+0000","last_peered":"2024-09-21T10:54:25.566276+0000","last_clean":"2024-09-21T10:54:25.566276+0000","last_became_active":"2024-09-21T10:52:25.502505+0000","last_became_peered":"2024-09-21T10:52:25.502505+0000","last_unstale":"2024-09-21T10:54:25.566276+0000","last_undegraded":"2024-09-21T10:54:25.566276+0000","last_fullsized":"2024-09-21T10:54:25.566276+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:01:14.965694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"346'149","reported_seq":844,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.805821+0000","last_change":"2024-09-21T10:54:16.021952+0000","last_active":"2024-09-21T10:54:23.805821+0000","last_peered":"2024-09-21T10:54:23.805821+0000","last_clean":"2024-09-21T10:54:23.805821+0000","last_became_active":"2024-09-21T10:51:30.436983+0000","last_became_peered":"2024-09-21T10:51:30.436983+0000","last_unstale":"2024-09-21T10:54:23.805821+0000","last_undegraded":"2024-09-21T10:54:23.805821+0000","last_fullsized":"2024-09-21T10:54:23.805821+0000","mapping_epoch":1450,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1451,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:16.021843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:16.021843+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:41:43.225623+0000","scrub_duration":0.0036826160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2324,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.555900+0000","last_change":"2024-09-21T10:52:15.397180+0000","last_active":"2024-09-21T10:54:25.555900+0000","last_peered":"2024-09-21T10:54:25.555900+0000","last_clean":"2024-09-21T10:54:25.555900+0000","last_became_active":"2024-09-21T10:52:15.396879+0000","last_became_peered":"2024-09-21T10:52:15.396879+0000","last_unstale":"2024-09-21T10:54:25.555900+0000","last_undegraded":"2024-09-21T10:54:25.555900+0000","last_fullsized":"2024-09-21T10:54:25.555900+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:03:07.162178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1475,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806800+0000","last_change":"2024-09-21T10:51:27.436590+0000","last_active":"2024-09-21T10:54:23.806800+0000","last_peered":"2024-09-21T10:54:23.806800+0000","last_clean":"2024-09-21T10:54:23.806800+0000","last_became_active":"2024-09-21T10:51:27.435332+0000","last_became_peered":"2024-09-21T10:51:27.435332+0000","last_unstale":"2024-09-21T10:54:23.806800+0000","last_undegraded":"2024-09-21T10:54:23.806800+0000","last_fullsized":"2024-09-21T10:54:23.806800+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:05:42.885506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":224,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.555228+0000","last_change":"2024-09-21T10:53:41.149328+0000","last_active":"2024-09-21T10:54:25.555228+0000","last_peered":"2024-09-21T10:54:25.555228+0000","last_clean":"2024-09-21T10:54:25.555228+0000","last_became_active":"2024-09-21T10:52:03.315398+0000","last_became_peered":"2024-09-21T10:52:03.315398+0000","last_unstale":"2024-09-21T10:54:25.555228+0000","last_undegraded":"2024-09-21T10:54:25.555228+0000","last_fullsized":"2024-09-21T10:54:25.555228+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:48:09.007053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"346'149","reported_seq":814,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.478340+0000","last_change":"2024-09-21T10:54:21.445887+0000","last_active":"2024-09-21T10:54:23.478340+0000","last_peered":"2024-09-21T10:54:23.478340+0000","last_clean":"2024-09-21T10:54:23.478340+0000","last_became_active":"2024-09-21T10:52:28.494923+0000","last_became_peered":"2024-09-21T10:52:28.494923+0000","last_unstale":"2024-09-21T10:54:23.478340+0000","last_undegraded":"2024-09-21T10:54:23.478340+0000","last_fullsized":"2024-09-21T10:54:23.478340+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:21.445783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:21.445783+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:33:42.131030+0000","scrub_duration":0.0018624990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":192,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806835+0000","last_change":"2024-09-21T10:52:25.475200+0000","last_active":"2024-09-21T10:54:23.806835+0000","last_peered":"2024-09-21T10:54:23.806835+0000","last_clean":"2024-09-21T10:54:23.806835+0000","last_became_active":"2024-09-21T10:52:25.474872+0000","last_became_peered":"2024-09-21T10:52:25.474872+0000","last_unstale":"2024-09-21T10:54:23.806835+0000","last_undegraded":"2024-09-21T10:54:23.806835+0000","last_fullsized":"2024-09-21T10:54:23.806835+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:36:34.207844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2281,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.554716+0000","last_change":"2024-09-21T10:52:03.325889+0000","last_active":"2024-09-21T10:54:25.554716+0000","last_peered":"2024-09-21T10:54:25.554716+0000","last_clean":"2024-09-21T10:54:25.554716+0000","last_became_active":"2024-09-21T10:52:03.325500+0000","last_became_peered":"2024-09-21T10:52:03.325500+0000","last_unstale":"2024-09-21T10:54:25.554716+0000","last_undegraded":"2024-09-21T10:54:25.554716+0000","last_fullsized":"2024-09-21T10:54:25.554716+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:20.708252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1066,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.566779+0000","last_change":"2024-09-21T10:51:27.457894+0000","last_active":"2024-09-21T10:54:25.566779+0000","last_peered":"2024-09-21T10:54:25.566779+0000","last_clean":"2024-09-21T10:54:25.566779+0000","last_became_active":"2024-09-21T10:51:27.457431+0000","last_became_peered":"2024-09-21T10:51:27.457431+0000","last_unstale":"2024-09-21T10:54:25.566779+0000","last_undegraded":"2024-09-21T10:54:25.566779+0000","last_fullsized":"2024-09-21T10:54:25.566779+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:03:33.550759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"346'149","reported_seq":937,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.555667+0000","last_change":"2024-09-21T10:52:15.396042+0000","last_active":"2024-09-21T10:54:25.555667+0000","last_peered":"2024-09-21T10:54:25.555667+0000","last_clean":"2024-09-21T10:54:25.555667+0000","last_became_active":"2024-09-21T10:52:15.395717+0000","last_became_peered":"2024-09-21T10:52:15.395717+0000","last_unstale":"2024-09-21T10:54:25.555667+0000","last_undegraded":"2024-09-21T10:54:25.555667+0000","last_fullsized":"2024-09-21T10:54:25.555667+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:55:04.143095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2306,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.554533+0000","last_change":"2024-09-21T10:52:25.476572+0000","last_active":"2024-09-21T10:54:25.554533+0000","last_peered":"2024-09-21T10:54:25.554533+0000","last_clean":"2024-09-21T10:54:25.554533+0000","last_became_active":"2024-09-21T10:52:25.475803+0000","last_became_peered":"2024-09-21T10:52:25.475803+0000","last_unstale":"2024-09-21T10:54:25.554533+0000","last_undegraded":"2024-09-21T10:54:25.554533+0000","last_fullsized":"2024-09-21T10:54:25.554533+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:06:44.191486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2221,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.805953+0000","last_change":"2024-09-21T10:52:03.326753+0000","last_active":"2024-09-21T10:54:23.805953+0000","last_peered":"2024-09-21T10:54:23.805953+0000","last_clean":"2024-09-21T10:54:23.805953+0000","last_became_active":"2024-09-21T10:52:03.326348+0000","last_became_peered":"2024-09-21T10:52:03.326348+0000","last_unstale":"2024-09-21T10:54:23.805953+0000","last_undegraded":"2024-09-21T10:54:23.805953+0000","last_fullsized":"2024-09-21T10:54:23.805953+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:57:38.100799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":144,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.478004+0000","last_change":"2024-09-21T10:51:27.426006+0000","last_active":"2024-09-21T10:54:23.478004+0000","last_peered":"2024-09-21T10:54:23.478004+0000","last_clean":"2024-09-21T10:54:23.478004+0000","last_became_active":"2024-09-21T10:51:27.425698+0000","last_became_peered":"2024-09-21T10:51:27.425698+0000","last_unstale":"2024-09-21T10:54:23.478004+0000","last_undegraded":"2024-09-21T10:54:23.478004+0000","last_fullsized":"2024-09-21T10:54:23.478004+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:35:13.210330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":38,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.867023+0000","last_change":"2024-09-21T10:52:15.401737+0000","last_active":"2024-09-21T10:54:23.867023+0000","last_peered":"2024-09-21T10:54:23.867023+0000","last_clean":"2024-09-21T10:54:23.867023+0000","last_became_active":"2024-09-21T10:52:15.400859+0000","last_became_peered":"2024-09-21T10:52:15.400859+0000","last_unstale":"2024-09-21T10:54:23.867023+0000","last_undegraded":"2024-09-21T10:54:23.867023+0000","last_fullsized":"2024-09-21T10:54:23.867023+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:15.723031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2260,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.567016+0000","last_change":"2024-09-21T10:49:44.865830+0000","last_active":"2024-09-21T10:54:25.567016+0000","last_peered":"2024-09-21T10:54:25.567016+0000","last_clean":"2024-09-21T10:54:25.567016+0000","last_became_active":"2024-09-21T10:49:44.865499+0000","last_became_peered":"2024-09-21T10:49:44.865499+0000","last_unstale":"2024-09-21T10:54:25.567016+0000","last_undegraded":"2024-09-21T10:54:25.567016+0000","last_fullsized":"2024-09-21T10:54:25.567016+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:19:22.013154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":1468,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.478476+0000","last_change":"2024-09-21T10:52:25.489550+0000","last_active":"2024-09-21T10:54:23.478476+0000","last_peered":"2024-09-21T10:54:23.478476+0000","last_clean":"2024-09-21T10:54:23.478476+0000","last_became_active":"2024-09-21T10:52:25.489171+0000","last_became_peered":"2024-09-21T10:52:25.489171+0000","last_unstale":"2024-09-21T10:54:23.478476+0000","last_undegraded":"2024-09-21T10:54:23.478476+0000","last_fullsized":"2024-09-21T10:54:23.478476+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:21.105058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":814,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.477870+0000","last_change":"2024-09-21T10:54:20.458538+0000","last_active":"2024-09-21T10:54:23.477870+0000","last_peered":"2024-09-21T10:54:23.477870+0000","last_clean":"2024-09-21T10:54:23.477870+0000","last_became_active":"2024-09-21T10:52:28.587447+0000","last_became_peered":"2024-09-21T10:52:28.587447+0000","last_unstale":"2024-09-21T10:54:23.477870+0000","last_undegraded":"2024-09-21T10:54:23.477870+0000","last_fullsized":"2024-09-21T10:54:23.477870+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:20.458421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:20.458421+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:35.467492+0000","scrub_duration":0.001528569,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"346'149","reported_seq":751,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.867131+0000","last_change":"2024-09-21T10:51:30.453979+0000","last_active":"2024-09-21T10:54:23.867131+0000","last_peered":"2024-09-21T10:54:23.867131+0000","last_clean":"2024-09-21T10:54:23.867131+0000","last_became_active":"2024-09-21T10:51:30.450893+0000","last_became_peered":"2024-09-21T10:51:30.450893+0000","last_unstale":"2024-09-21T10:54:23.867131+0000","last_undegraded":"2024-09-21T10:54:23.867131+0000","last_fullsized":"2024-09-21T10:54:23.867131+0000","mapping_epoch":1450,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1451,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:42.008235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:42.008235+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:21:48.570809+0000","scrub_duration":0.011129834999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2257,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.555621+0000","last_change":"2024-09-21T10:09:06.538940+0000","last_active":"2024-09-21T10:54:25.555621+0000","last_peered":"2024-09-21T10:54:25.555621+0000","last_clean":"2024-09-21T10:54:25.555621+0000","last_became_active":"2024-09-21T10:09:06.538668+0000","last_became_peered":"2024-09-21T10:09:06.538668+0000","last_unstale":"2024-09-21T10:54:25.555621+0000","last_undegraded":"2024-09-21T10:54:25.555621+0000","last_fullsized":"2024-09-21T10:54:25.555621+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:42:44.299488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2256,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.566404+0000","last_change":"2024-09-21T10:52:25.493032+0000","last_active":"2024-09-21T10:54:25.566404+0000","last_peered":"2024-09-21T10:54:25.566404+0000","last_clean":"2024-09-21T10:54:25.566404+0000","last_became_active":"2024-09-21T10:52:25.492182+0000","last_became_peered":"2024-09-21T10:52:25.492182+0000","last_unstale":"2024-09-21T10:54:25.566404+0000","last_undegraded":"2024-09-21T10:54:25.566404+0000","last_fullsized":"2024-09-21T10:54:25.566404+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:12:57.118420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1481,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806795+0000","last_change":"2024-09-21T10:52:15.402809+0000","last_active":"2024-09-21T10:54:23.806795+0000","last_peered":"2024-09-21T10:54:23.806795+0000","last_clean":"2024-09-21T10:54:23.806795+0000","last_became_active":"2024-09-21T10:52:15.402497+0000","last_became_peered":"2024-09-21T10:52:15.402497+0000","last_unstale":"2024-09-21T10:54:23.806795+0000","last_undegraded":"2024-09-21T10:54:23.806795+0000","last_fullsized":"2024-09-21T10:54:23.806795+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:07:35.713534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1492,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806255+0000","last_change":"2024-09-21T10:44:14.399842+0000","last_active":"2024-09-21T10:54:23.806255+0000","last_peered":"2024-09-21T10:54:23.806255+0000","last_clean":"2024-09-21T10:54:23.806255+0000","last_became_active":"2024-09-21T10:44:14.399426+0000","last_became_peered":"2024-09-21T10:44:14.399426+0000","last_unstale":"2024-09-21T10:54:23.806255+0000","last_undegraded":"2024-09-21T10:54:23.806255+0000","last_fullsized":"2024-09-21T10:54:23.806255+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:26:45.362101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"346'149","reported_seq":937,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.566767+0000","last_change":"2024-09-21T10:52:16.404597+0000","last_active":"2024-09-21T10:54:25.566767+0000","last_peered":"2024-09-21T10:54:25.566767+0000","last_clean":"2024-09-21T10:54:25.566767+0000","last_became_active":"2024-09-21T10:52:16.404194+0000","last_became_peered":"2024-09-21T10:52:16.404194+0000","last_unstale":"2024-09-21T10:54:25.566767+0000","last_undegraded":"2024-09-21T10:54:25.566767+0000","last_fullsized":"2024-09-21T10:54:25.566767+0000","mapping_epoch":1463,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1464,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:55:01.626212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":793725,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1487,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806592+0000","last_change":"2024-09-21T10:40:13.432387+0000","last_active":"2024-09-21T10:54:23.806592+0000","last_peered":"2024-09-21T10:54:23.806592+0000","last_clean":"2024-09-21T10:54:23.806592+0000","last_became_active":"2024-09-21T10:40:13.432053+0000","last_became_peered":"2024-09-21T10:40:13.432053+0000","last_unstale":"2024-09-21T10:54:23.806592+0000","last_undegraded":"2024-09-21T10:54:23.806592+0000","last_fullsized":"2024-09-21T10:54:23.806592+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:37.486101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1465,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.867382+0000","last_change":"2024-09-21T10:07:56.020791+0000","last_active":"2024-09-21T10:54:23.867382+0000","last_peered":"2024-09-21T10:54:23.867382+0000","last_clean":"2024-09-21T10:54:23.867382+0000","last_became_active":"2024-09-21T10:07:56.020205+0000","last_became_peered":"2024-09-21T10:07:56.020205+0000","last_unstale":"2024-09-21T10:54:23.867382+0000","last_undegraded":"2024-09-21T10:54:23.867382+0000","last_fullsized":"2024-09-21T10:54:23.867382+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:09.813166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"346'149","reported_seq":940,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.477924+0000","last_change":"2024-09-21T10:54:17.418763+0000","last_active":"2024-09-21T10:54:23.477924+0000","last_peered":"2024-09-21T10:54:23.477924+0000","last_clean":"2024-09-21T10:54:23.477924+0000","last_became_active":"2024-09-21T10:49:47.903650+0000","last_became_peered":"2024-09-21T10:49:47.903650+0000","last_unstale":"2024-09-21T10:54:23.477924+0000","last_undegraded":"2024-09-21T10:54:23.477924+0000","last_fullsized":"2024-09-21T10:54:23.477924+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:17.418646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:17.418646+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:17.134488+0000","scrub_duration":0.00185648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1463,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806415+0000","last_change":"2024-09-21T10:09:06.533271+0000","last_active":"2024-09-21T10:54:23.806415+0000","last_peered":"2024-09-21T10:54:23.806415+0000","last_clean":"2024-09-21T10:54:23.806415+0000","last_became_active":"2024-09-21T10:09:06.532041+0000","last_became_peered":"2024-09-21T10:09:06.532041+0000","last_unstale":"2024-09-21T10:54:23.806415+0000","last_undegraded":"2024-09-21T10:54:23.806415+0000","last_fullsized":"2024-09-21T10:54:23.806415+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:04.236375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":2285,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.567122+0000","last_change":"2024-09-21T10:51:27.529295+0000","last_active":"2024-09-21T10:54:25.567122+0000","last_peered":"2024-09-21T10:54:25.567122+0000","last_clean":"2024-09-21T10:54:25.567122+0000","last_became_active":"2024-09-21T10:51:27.446359+0000","last_became_peered":"2024-09-21T10:51:27.446359+0000","last_unstale":"2024-09-21T10:54:25.567122+0000","last_undegraded":"2024-09-21T10:54:25.567122+0000","last_fullsized":"2024-09-21T10:54:25.567122+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:17:08.788558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":715,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.555849+0000","last_change":"2024-09-21T10:51:27.481103+0000","last_active":"2024-09-21T10:54:25.555849+0000","last_peered":"2024-09-21T10:54:25.555849+0000","last_clean":"2024-09-21T10:54:25.555849+0000","last_became_active":"2024-09-21T10:51:27.480763+0000","last_became_peered":"2024-09-21T10:51:27.480763+0000","last_unstale":"2024-09-21T10:54:25.555849+0000","last_undegraded":"2024-09-21T10:54:25.555849+0000","last_fullsized":"2024-09-21T10:54:25.555849+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:35.375303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":820,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.567137+0000","last_change":"2024-09-21T10:49:44.864819+0000","last_active":"2024-09-21T10:54:25.567137+0000","last_peered":"2024-09-21T10:54:25.567137+0000","last_clean":"2024-09-21T10:54:25.567137+0000","last_became_active":"2024-09-21T10:49:44.864061+0000","last_became_peered":"2024-09-21T10:49:44.864061+0000","last_unstale":"2024-09-21T10:54:25.567137+0000","last_undegraded":"2024-09-21T10:54:25.567137+0000","last_fullsized":"2024-09-21T10:54:25.567137+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:37.266967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":47,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.567277+0000","last_change":"2024-09-21T10:52:25.492570+0000","last_active":"2024-09-21T10:54:25.567277+0000","last_peered":"2024-09-21T10:54:25.567277+0000","last_clean":"2024-09-21T10:54:25.567277+0000","last_became_active":"2024-09-21T10:52:25.486382+0000","last_became_peered":"2024-09-21T10:52:25.486382+0000","last_unstale":"2024-09-21T10:54:25.567277+0000","last_undegraded":"2024-09-21T10:54:25.567277+0000","last_fullsized":"2024-09-21T10:54:25.567277+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:06.702707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":151,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.478598+0000","last_change":"2024-09-21T10:49:44.874522+0000","last_active":"2024-09-21T10:54:23.478598+0000","last_peered":"2024-09-21T10:54:23.478598+0000","last_clean":"2024-09-21T10:54:23.478598+0000","last_became_active":"2024-09-21T10:49:44.863242+0000","last_became_peered":"2024-09-21T10:49:44.863242+0000","last_unstale":"2024-09-21T10:54:23.478598+0000","last_undegraded":"2024-09-21T10:54:23.478598+0000","last_fullsized":"2024-09-21T10:54:23.478598+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:09:02.066837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1497,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806076+0000","last_change":"2024-09-21T10:52:03.315267+0000","last_active":"2024-09-21T10:54:23.806076+0000","last_peered":"2024-09-21T10:54:23.806076+0000","last_clean":"2024-09-21T10:54:23.806076+0000","last_became_active":"2024-09-21T10:52:03.314474+0000","last_became_peered":"2024-09-21T10:52:03.314474+0000","last_unstale":"2024-09-21T10:54:23.806076+0000","last_undegraded":"2024-09-21T10:54:23.806076+0000","last_fullsized":"2024-09-21T10:54:23.806076+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:00:57.791341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":38,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.479070+0000","last_change":"2024-09-21T10:52:25.518721+0000","last_active":"2024-09-21T10:54:23.479070+0000","last_peered":"2024-09-21T10:54:23.479070+0000","last_clean":"2024-09-21T10:54:23.479070+0000","last_became_active":"2024-09-21T10:52:25.472114+0000","last_became_peered":"2024-09-21T10:52:25.472114+0000","last_unstale":"2024-09-21T10:54:23.479070+0000","last_undegraded":"2024-09-21T10:54:23.479070+0000","last_fullsized":"2024-09-21T10:54:23.479070+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:33:48.471147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":34,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.477848+0000","last_change":"2024-09-21T10:52:25.494490+0000","last_active":"2024-09-21T10:54:23.477848+0000","last_peered":"2024-09-21T10:54:23.477848+0000","last_clean":"2024-09-21T10:54:23.477848+0000","last_became_active":"2024-09-21T10:52:25.494200+0000","last_became_peered":"2024-09-21T10:52:25.494200+0000","last_unstale":"2024-09-21T10:54:23.477848+0000","last_undegraded":"2024-09-21T10:54:23.477848+0000","last_fullsized":"2024-09-21T10:54:23.477848+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:15:21.030031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":38,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.479239+0000","last_change":"2024-09-21T10:52:25.479229+0000","last_active":"2024-09-21T10:54:23.479239+0000","last_peered":"2024-09-21T10:54:23.479239+0000","last_clean":"2024-09-21T10:54:23.479239+0000","last_became_active":"2024-09-21T10:52:25.470551+0000","last_became_peered":"2024-09-21T10:52:25.470551+0000","last_unstale":"2024-09-21T10:54:23.479239+0000","last_undegraded":"2024-09-21T10:54:23.479239+0000","last_fullsized":"2024-09-21T10:54:23.479239+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:15.458231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1509,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.866809+0000","last_change":"2024-09-21T10:51:27.517234+0000","last_active":"2024-09-21T10:54:23.866809+0000","last_peered":"2024-09-21T10:54:23.866809+0000","last_clean":"2024-09-21T10:54:23.866809+0000","last_became_active":"2024-09-21T10:51:27.431253+0000","last_became_peered":"2024-09-21T10:51:27.431253+0000","last_unstale":"2024-09-21T10:54:23.866809+0000","last_undegraded":"2024-09-21T10:54:23.866809+0000","last_fullsized":"2024-09-21T10:54:23.866809+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:35:42.249305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2239,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.566346+0000","last_change":"2024-09-21T10:52:25.497298+0000","last_active":"2024-09-21T10:54:25.566346+0000","last_peered":"2024-09-21T10:54:25.566346+0000","last_clean":"2024-09-21T10:54:25.566346+0000","last_became_active":"2024-09-21T10:52:25.496876+0000","last_became_peered":"2024-09-21T10:52:25.496876+0000","last_unstale":"2024-09-21T10:54:25.566346+0000","last_undegraded":"2024-09-21T10:54:25.566346+0000","last_fullsized":"2024-09-21T10:54:25.566346+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:16.587897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":134,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.478084+0000","last_change":"2024-09-21T10:44:14.398878+0000","last_active":"2024-09-21T10:54:23.478084+0000","last_peered":"2024-09-21T10:54:23.478084+0000","last_clean":"2024-09-21T10:54:23.478084+0000","last_became_active":"2024-09-21T10:44:14.398547+0000","last_became_peered":"2024-09-21T10:44:14.398547+0000","last_unstale":"2024-09-21T10:54:23.478084+0000","last_undegraded":"2024-09-21T10:54:23.478084+0000","last_fullsized":"2024-09-21T10:54:23.478084+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:13.008134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2288,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.566414+0000","last_change":"2024-09-21T10:52:25.498554+0000","last_active":"2024-09-21T10:54:25.566414+0000","last_peered":"2024-09-21T10:54:25.566414+0000","last_clean":"2024-09-21T10:54:25.566414+0000","last_became_active":"2024-09-21T10:52:25.498194+0000","last_became_peered":"2024-09-21T10:52:25.498194+0000","last_unstale":"2024-09-21T10:54:25.566414+0000","last_undegraded":"2024-09-21T10:54:25.566414+0000","last_fullsized":"2024-09-21T10:54:25.566414+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:47.115271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":34,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.478172+0000","last_change":"2024-09-21T10:52:25.489063+0000","last_active":"2024-09-21T10:54:23.478172+0000","last_peered":"2024-09-21T10:54:23.478172+0000","last_clean":"2024-09-21T10:54:23.478172+0000","last_became_active":"2024-09-21T10:52:25.488129+0000","last_became_peered":"2024-09-21T10:52:25.488129+0000","last_unstale":"2024-09-21T10:54:23.478172+0000","last_undegraded":"2024-09-21T10:54:23.478172+0000","last_fullsized":"2024-09-21T10:54:23.478172+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:36.996829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1449,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.478498+0000","last_change":"2024-09-21T10:49:44.864898+0000","last_active":"2024-09-21T10:54:23.478498+0000","last_peered":"2024-09-21T10:54:23.478498+0000","last_clean":"2024-09-21T10:54:23.478498+0000","last_became_active":"2024-09-21T10:49:44.862691+0000","last_became_peered":"2024-09-21T10:49:44.862691+0000","last_unstale":"2024-09-21T10:54:23.478498+0000","last_undegraded":"2024-09-21T10:54:23.478498+0000","last_fullsized":"2024-09-21T10:54:23.478498+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:57.809113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":140,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.478576+0000","last_change":"2024-09-21T10:44:14.418113+0000","last_active":"2024-09-21T10:54:23.478576+0000","last_peered":"2024-09-21T10:54:23.478576+0000","last_clean":"2024-09-21T10:54:23.478576+0000","last_became_active":"2024-09-21T10:44:14.398327+0000","last_became_peered":"2024-09-21T10:44:14.398327+0000","last_unstale":"2024-09-21T10:54:23.478576+0000","last_undegraded":"2024-09-21T10:54:23.478576+0000","last_fullsized":"2024-09-21T10:54:23.478576+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:08.810863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2269,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.567239+0000","last_change":"2024-09-21T10:44:14.398584+0000","last_active":"2024-09-21T10:54:25.567239+0000","last_peered":"2024-09-21T10:54:25.567239+0000","last_clean":"2024-09-21T10:54:25.567239+0000","last_became_active":"2024-09-21T10:44:14.398278+0000","last_became_peered":"2024-09-21T10:44:14.398278+0000","last_unstale":"2024-09-21T10:54:25.567239+0000","last_undegraded":"2024-09-21T10:54:25.567239+0000","last_fullsized":"2024-09-21T10:54:25.567239+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:19.681750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":56,"num_pg":16,"stat_sum":{"num_bytes":340066304,"num_objects":5189,"num_object_clones":0,"num_object_copies":15567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5189,"num_write_kb":332096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":718143488,"data_stored":718143488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5189,"ondisk_log_size":5189,"up":48,"acting":48,"num_store_stats":5},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":161010,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":225,"num_write":59,"num_write_kb":659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":2641143,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":33,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":11344,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":18008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234725,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":580416,"kb_used_data":12020,"kb_used_omap":1,"kb_used_meta":568382,"kb_avail":93164736,"statfs":{"total":95995035648,"available":95400689664,"internally_reserved":0,"allocated":12308480,"data_stored":7194752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":582023627},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496362,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":747204,"kb_used_data":44280,"kb_used_omap":1,"kb_used_meta":702910,"kb_avail":92997948,"statfs":{"total":95995035648,"available":95229898752,"internally_reserved":0,"allocated":45342720,"data_stored":40225929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":719780298},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757999,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019635,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":674692,"kb_used_data":11960,"kb_used_omap":1,"kb_used_meta":662718,"kb_avail":93070460,"statfs":{"total":95995035648,"available":95304151040,"internally_reserved":0,"allocated":12247040,"data_stored":7130272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":678623686},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369379,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1065908,"kb_used_data":243820,"kb_used_omap":69,"kb_used_meta":822010,"kb_avail":92679244,"statfs":{"total":95995035648,"available":94903545856,"internally_reserved":0,"allocated":249671680,"data_stored":244562489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":841738370},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2920577761834,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807958065,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":904748,"kb_used_data":196640,"kb_used_omap":66,"kb_used_meta":708029,"kb_avail":92840404,"statfs":{"total":95995035648,"available":95068573696,"internally_reserved":0,"allocated":201359360,"data_stored":196261193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68589,"internal_metadata":725021715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":2860448219704,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":274530304,"data_stored":274530304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":289931264,"data_stored":289931264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":61341696,"data_stored":61341696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":92340224,"data_stored":92340224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:54:37.292 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:37.326 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:37.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:37.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:37.428 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:37.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:37.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:37.549 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 11 15 30076 30061 170.785 124 0.0105148 0.00584928 2024-09-21T10:54:37.561 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:37.566 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:54:37.566 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T10:54:37.567 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T10:54:37.567 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T10:54:37.567 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:54:37.594 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:37.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:37.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:37.695 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:37.728 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:37.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:37.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:37.830 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:37.850 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:54:37.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:37.870 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:37.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:37.971 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:38.007 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:38.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:38.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:38.108 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:38.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:38.159 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:38.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:38.260 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:38.296 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:38.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:38.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:38.397 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:38.429 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:54:38.430 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:38.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:38.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:38.532 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:38.550 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 12 16 32189 32173 167.552 132 0.00848603 0.00596359 2024-09-21T10:54:38.569 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:38.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:38.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:38.671 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:38.699 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:38.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:38.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:38.801 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:38.831 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:38.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:38.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:38 smithi160 ceph-mon[101879]: pgmap v2761: 99 pgs: 99 active+clean; 1.4 GiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-09-21T10:54:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:38 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:54:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:38 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:54:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:38 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:54:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:38 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:54:38.845 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:38 smithi070 ceph-mon[107385]: pgmap v2761: 99 pgs: 99 active+clean; 1.4 GiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-09-21T10:54:38.845 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:38 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:54:38.845 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:38 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:54:38.845 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:38 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:54:38.845 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:38 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:54:38.845 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:38 smithi070 ceph-mon[112139]: pgmap v2761: 99 pgs: 99 active+clean; 1.4 GiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-09-21T10:54:38.845 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:38 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:54:38.846 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:38 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:54:38.846 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:38 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:54:38.846 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:38 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:54:38.879 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:54:38.933 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:38.972 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:38.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:38.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:39.074 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:39.109 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:39.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:39.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:39.211 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:39.238 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:39.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:39.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:39.340 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:39.379 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:39.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:39.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:39.481 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:39.518 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:39.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:39.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:39.549 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 13 16 34287 34271 164.749 131.125 0.00521936 0.00606476 2024-09-21T10:54:39.620 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:39.650 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:39.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:39.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:39.752 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:39.760 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:39 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:54:39.760 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:39 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:54:39.782 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:39.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:39.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:39 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:54:39.884 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:39.912 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:39.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:39.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:40.014 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:40.050 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:40.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:40.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:40.151 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:40.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:40.187 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:40.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:40.235 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:54:40.288 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:40.325 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:40.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:40.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:40.427 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:40.467 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:40.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:40.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:40.549 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 14 16 36318 36302 162.047 126.938 0.00608 0.00616617 2024-09-21T10:54:40.569 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:40.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:40.601 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:40.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:40.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:40 smithi070 ceph-mon[107385]: pgmap v2762: 99 pgs: 99 active+clean; 1.7 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-09-21T10:54:40.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:40 smithi070 ceph-mon[112139]: pgmap v2762: 99 pgs: 99 active+clean; 1.7 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-09-21T10:54:40.657 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:54:40.702 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:40.731 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:40.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:40.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:40.833 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:40 smithi160 ceph-mon[101879]: pgmap v2762: 99 pgs: 99 active+clean; 1.7 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-09-21T10:54:40.860 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:40.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:40.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:40.962 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:40.990 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:40.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:40.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:41.092 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:41.126 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:41.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:41.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:41.176 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:54:41.228 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:41.258 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:41.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:41.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:41.359 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:41.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:41.402 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:41.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:41.503 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:41.533 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:41.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:41.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:41.549 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 15 16 38279 38263 159.415 122.562 0.00676565 0.00626594 2024-09-21T10:54:41.635 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:41.674 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:41.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:41.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:41.776 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:41.805 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:41.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:41.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:41.907 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:41.936 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:41.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:41.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:42.038 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:42.068 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:42.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:42.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:42.170 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:42.212 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:42.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:42.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:42.314 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:42.344 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:42.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:42.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:42.446 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:42.472 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:42.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:42.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:42.549 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 16 16 40201 40185 156.958 120.125 0.00282741 0.00636638 2024-09-21T10:54:42.574 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:42.608 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:42.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:42.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:42.710 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:42.740 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:42.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:42.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:42 smithi160 ceph-mon[101879]: pgmap v2763: 99 pgs: 99 active+clean; 1.9 GiB data, 8.8 GiB used, 438 GiB / 447 GiB avail; 150 MiB/s wr, 2.40k op/s 2024-09-21T10:54:42.841 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:42.842 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:42 smithi070 ceph-mon[107385]: pgmap v2763: 99 pgs: 99 active+clean; 1.9 GiB data, 8.8 GiB used, 438 GiB / 447 GiB avail; 150 MiB/s wr, 2.40k op/s 2024-09-21T10:54:42.843 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:42 smithi070 ceph-mon[112139]: pgmap v2763: 99 pgs: 99 active+clean; 1.9 GiB data, 8.8 GiB used, 438 GiB / 447 GiB avail; 150 MiB/s wr, 2.40k op/s 2024-09-21T10:54:42.871 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:42.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:42.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:42.973 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:43.010 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:43.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:43.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:43.112 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:43.143 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:43.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:43.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:43.245 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:43.282 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:43.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:43.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:43.384 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:43.419 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:43.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:43.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:43.521 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:43.549 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 17 16 42005 41989 154.357 112.75 0.00939134 0.00647312 2024-09-21T10:54:43.563 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:43.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:43.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:43.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:54:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:54:43] "GET /metrics HTTP/1.1" 200 36593 "" "Prometheus/2.51.0" 2024-09-21T10:54:43.666 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:43.705 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:43.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:43.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:43.807 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:43.841 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:43.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:43.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:43.882 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:54:43.882 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:54:43.908 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:54:43.908 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:54:43.931 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:54:43.931 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:54:43.942 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:43.957 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:54:43.957 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:54:43.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:43.986 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:43.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:43.988 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:54:43.988 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:54:44.011 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:54:44.011 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:54:44.039 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:54:44.039 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:54:44.067 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:54:44.067 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:54:44.087 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:44.124 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:44.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:44.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:44.227 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:44.257 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:44.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:44.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:44.359 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:44.388 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:44.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:44.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:44.489 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:44.532 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:44.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:44.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:44.549 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 18 16 43806 43790 152.035 112.562 0.00754393 0.00657298 2024-09-21T10:54:44.599 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:44 smithi070 ceph-mon[107385]: pgmap v2764: 99 pgs: 99 active+clean; 2.2 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 160 MiB/s wr, 2.55k op/s 2024-09-21T10:54:44.599 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:44 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3695293363' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:54:44.599 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:44 smithi070 ceph-mon[112139]: pgmap v2764: 99 pgs: 99 active+clean; 2.2 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 160 MiB/s wr, 2.55k op/s 2024-09-21T10:54:44.599 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:44 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3695293363' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:54:44.634 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:44.674 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:44.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:44.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:44.776 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:44.808 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:44.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:44.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:44 smithi160 ceph-mon[101879]: pgmap v2764: 99 pgs: 99 active+clean; 2.2 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 160 MiB/s wr, 2.55k op/s 2024-09-21T10:54:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:44 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3695293363' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:54:44.910 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:44.948 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:44.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:44.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:45.050 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:45.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:45.087 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:45.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:45.188 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:45.214 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:45.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:45.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:45.316 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:45.348 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:45.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:45.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:45.451 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:45.481 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:45.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:45.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:45.549 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 19 16 45569 45553 149.832 110.188 0.00725464 0.00666908 2024-09-21T10:54:45.583 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:45.617 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:45.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:45.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:45.719 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:45.754 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:45.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:45.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:45.856 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:45.896 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:45.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:45.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:45.998 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:46.055 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:46.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:46.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:46.373 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:46.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:46.402 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:46.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:46.409 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T10:54:46.453 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:46 smithi070 ceph-mon[107385]: pgmap v2765: 99 pgs: 99 active+clean; 2.5 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 143 MiB/s wr, 2.28k op/s 2024-09-21T10:54:46.453 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:46 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1045953889' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:54:46.454 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:46 smithi070 ceph-mon[112139]: pgmap v2765: 99 pgs: 99 active+clean; 2.5 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 143 MiB/s wr, 2.28k op/s 2024-09-21T10:54:46.454 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:46 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1045953889' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:54:46.503 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:46.536 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:46.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:46.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:46.550 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:2024-09-21T10:54:46.548552+0000 min lat: 0.00129408 max lat: 0.164397 avg lat: 0.0067713 2024-09-21T10:54:46.550 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:54:46.550 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 20 16 47243 47227 147.571 104.625 0.00861129 0.0067713 2024-09-21T10:54:46.624 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:54:46.625 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T10:54:46.638 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:46.702 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:46.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:46.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:46.804 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:46.830 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:46.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:46.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:46 smithi160 ceph-mon[101879]: pgmap v2765: 99 pgs: 99 active+clean; 2.5 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 143 MiB/s wr, 2.28k op/s 2024-09-21T10:54:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:46 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1045953889' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:54:46.933 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:46.958 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:46.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:46.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:47.060 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:47.086 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:47.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:47.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:47.189 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:47.215 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:47.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:47.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:47.317 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:47.355 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:47.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:47.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:47.360 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:54:47.457 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:47.489 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:47.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:47.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:47.550 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 21 16 48946 48930 145.611 106.438 0.00922308 0.00686273 2024-09-21T10:54:47.591 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:47.629 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:47.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:47.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:47.731 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:47.766 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:47.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:47.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:47.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:47 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:54:47.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:47 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1045953889' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:54:47.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:47 smithi160 ceph-mon[101879]: osdmap e1490: 8 total, 8 up, 5 in 2024-09-21T10:54:47.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:47 smithi160 ceph-mon[101879]: from='client.36033 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:54:47.868 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:47.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:47 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:54:47.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:47 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1045953889' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:54:47.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:47 smithi070 ceph-mon[107385]: osdmap e1490: 8 total, 8 up, 5 in 2024-09-21T10:54:47.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:47 smithi070 ceph-mon[107385]: from='client.36033 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:54:47.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:47 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:54:47.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:47 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1045953889' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:54:47.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:47 smithi070 ceph-mon[112139]: osdmap e1490: 8 total, 8 up, 5 in 2024-09-21T10:54:47.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:47 smithi070 ceph-mon[112139]: from='client.36033 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:54:47.906 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:47.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:47.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:48.008 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:48.045 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:48.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:48.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:48.147 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:48.176 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:48.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:48.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:48.278 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:48.308 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:48.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:48.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:48.410 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:48.435 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:48.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:48.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:48.537 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:48.550 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 22 15 50332 50317 142.929 86.6875 0.111232 0.00697357 2024-09-21T10:54:48.569 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:48.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:48.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:48.670 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:48.708 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:48.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:48.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:48.810 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:48.837 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:48.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:48.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:48 smithi160 ceph-mon[101879]: pgmap v2767: 99 pgs: 99 active+clean; 2.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-09-21T10:54:48.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:48 smithi070 ceph-mon[112139]: pgmap v2767: 99 pgs: 99 active+clean; 2.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-09-21T10:54:48.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:48 smithi070 ceph-mon[107385]: pgmap v2767: 99 pgs: 99 active+clean; 2.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-09-21T10:54:48.939 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:48.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:48.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:49.066 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:49.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:49.105 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:49.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:49.206 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:49.235 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:49.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:49.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:49.337 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:49.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:49.367 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:49.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:49.468 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:49.492 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:49.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:49.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:49.550 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 23 16 51655 51639 140.307 82.625 0.00649351 0.007122 2024-09-21T10:54:49.595 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:49.622 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:49.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:49.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:49.724 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:49.752 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:49.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:49.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:49.853 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:49.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:49.894 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:49.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:49.995 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:50.036 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:50.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:50.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:50.138 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:50.177 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:50.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:50.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:50.279 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:50.305 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:50.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:50.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:50.407 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:50.438 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:50.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:50.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:50.540 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:50.550 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 24 16 53166 53150 138.395 94.4375 0.00553737 0.00722037 2024-09-21T10:54:50.570 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:50.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:50.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:50.672 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:50.688 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:50 smithi070 ceph-mon[107385]: pgmap v2768: 99 pgs: 99 active+clean; 2.9 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-09-21T10:54:50.688 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:54:50.688 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:54:50.690 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:50 smithi070 ceph-mon[112139]: pgmap v2768: 99 pgs: 99 active+clean; 2.9 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-09-21T10:54:50.690 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:54:50.690 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:54:50.702 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:50.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:50.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:50.804 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:50.836 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:50.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:50.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:50 smithi160 ceph-mon[101879]: pgmap v2768: 99 pgs: 99 active+clean; 2.9 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-09-21T10:54:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:54:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:54:50.938 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:50.982 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:50.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:50.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:51.084 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:51.123 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:51.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:51.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:51.225 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:51.252 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:51.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:51.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:51.354 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:51.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:51.391 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:51.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:51.493 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:51.521 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:51.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:51.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:51.550 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 25 16 54689 54673 136.667 95.1875 0.00949029 0.00731185 2024-09-21T10:54:51.623 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:51.655 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":2766,"stamp":"2024-09-21T10:54:46.408384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2822277815,"num_objects":43086,"num_object_clones":0,"num_object_copies":129258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43086,"num_whiteouts":0,"num_read":104,"num_read_kb":225,"num_write":43185,"num_write_kb":2756660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":2652544,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44760,"ondisk_log_size":44760,"up":297,"acting":297,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":297,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":11606936,"kb_used_data":7917808,"kb_used_omap":141,"kb_used_meta":3688946,"kb_avail":457118824,"statfs":{"total":479975178240,"available":468089675776,"internally_reserved":0,"allocated":8107835392,"data_stored":8082280747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":144912,"internal_metadata":3777481200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,2,11,4,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1501167616,"num_objects":22906,"num_object_clones":0,"num_object_copies":68718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22906,"num_write_kb":1465984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.451832"},"pg_stats":[{"pgid":"56.f","version":"1489'2616","reported_seq":2637,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:43.728230+0000","last_change":"2024-09-21T10:54:11.453036+0000","last_active":"2024-09-21T10:54:43.728230+0000","last_peered":"2024-09-21T10:54:43.728230+0000","last_clean":"2024-09-21T10:54:43.728230+0000","last_became_active":"2024-09-21T10:54:11.452748+0000","last_became_peered":"2024-09-21T10:54:11.452748+0000","last_unstale":"2024-09-21T10:54:43.728230+0000","last_undegraded":"2024-09-21T10:54:43.728230+0000","last_fullsized":"2024-09-21T10:54:43.728230+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":2616,"log_dups_size":0,"ondisk_log_size":2616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:04:42.219393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171442176,"num_objects":2616,"num_object_clones":0,"num_object_copies":7848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2616,"num_write_kb":167424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.e","version":"1489'2439","reported_seq":2453,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:42.023502+0000","last_change":"2024-09-21T10:54:11.452866+0000","last_active":"2024-09-21T10:54:42.023502+0000","last_peered":"2024-09-21T10:54:42.023502+0000","last_clean":"2024-09-21T10:54:42.023502+0000","last_became_active":"2024-09-21T10:54:11.452553+0000","last_became_peered":"2024-09-21T10:54:11.452553+0000","last_unstale":"2024-09-21T10:54:42.023502+0000","last_undegraded":"2024-09-21T10:54:42.023502+0000","last_fullsized":"2024-09-21T10:54:42.023502+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":2439,"log_dups_size":0,"ondisk_log_size":2439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:02:31.756582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159842304,"num_objects":2439,"num_object_clones":0,"num_object_copies":7317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2439,"num_write_kb":156096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"56.d","version":"1489'2733","reported_seq":2754,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:44.177604+0000","last_change":"2024-09-21T10:54:11.455827+0000","last_active":"2024-09-21T10:54:44.177604+0000","last_peered":"2024-09-21T10:54:44.177604+0000","last_clean":"2024-09-21T10:54:44.177604+0000","last_became_active":"2024-09-21T10:54:11.455499+0000","last_became_peered":"2024-09-21T10:54:11.455499+0000","last_unstale":"2024-09-21T10:54:44.177604+0000","last_undegraded":"2024-09-21T10:54:44.177604+0000","last_fullsized":"2024-09-21T10:54:44.177604+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":2733,"log_dups_size":0,"ondisk_log_size":2733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:58:18.441343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179109888,"num_objects":2733,"num_object_clones":0,"num_object_copies":8199,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2733,"num_write_kb":174912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.c","version":"1489'2740","reported_seq":2757,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:43.733831+0000","last_change":"2024-09-21T10:54:11.452969+0000","last_active":"2024-09-21T10:54:43.733831+0000","last_peered":"2024-09-21T10:54:43.733831+0000","last_clean":"2024-09-21T10:54:43.733831+0000","last_became_active":"2024-09-21T10:54:11.452637+0000","last_became_peered":"2024-09-21T10:54:11.452637+0000","last_unstale":"2024-09-21T10:54:43.733831+0000","last_undegraded":"2024-09-21T10:54:43.733831+0000","last_fullsized":"2024-09-21T10:54:43.733831+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":2740,"log_dups_size":0,"ondisk_log_size":2740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:51:29.279090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179568640,"num_objects":2740,"num_object_clones":0,"num_object_copies":8220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2740,"num_write_kb":175360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.b","version":"1489'2888","reported_seq":2898,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:46.379175+0000","last_change":"2024-09-21T10:54:11.455486+0000","last_active":"2024-09-21T10:54:46.379175+0000","last_peered":"2024-09-21T10:54:46.379175+0000","last_clean":"2024-09-21T10:54:46.379175+0000","last_became_active":"2024-09-21T10:54:11.455190+0000","last_became_peered":"2024-09-21T10:54:11.455190+0000","last_unstale":"2024-09-21T10:54:46.379175+0000","last_undegraded":"2024-09-21T10:54:46.379175+0000","last_fullsized":"2024-09-21T10:54:46.379175+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":2888,"log_dups_size":0,"ondisk_log_size":2888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:50:03.321568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189267968,"num_objects":2888,"num_object_clones":0,"num_object_copies":8664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2888,"num_write_kb":184832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"56.a","version":"1489'2601","reported_seq":2622,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:43.743589+0000","last_change":"2024-09-21T10:54:11.454454+0000","last_active":"2024-09-21T10:54:43.743589+0000","last_peered":"2024-09-21T10:54:43.743589+0000","last_clean":"2024-09-21T10:54:43.743589+0000","last_became_active":"2024-09-21T10:54:11.454164+0000","last_became_peered":"2024-09-21T10:54:11.454164+0000","last_unstale":"2024-09-21T10:54:43.743589+0000","last_undegraded":"2024-09-21T10:54:43.743589+0000","last_fullsized":"2024-09-21T10:54:43.743589+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":2601,"log_dups_size":0,"ondisk_log_size":2601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:57:14.569670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170459136,"num_objects":2601,"num_object_clones":0,"num_object_copies":7803,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2601,"num_write_kb":166464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.9","version":"1489'2941","reported_seq":2955,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:46.357194+0000","last_change":"2024-09-21T10:54:11.455480+0000","last_active":"2024-09-21T10:54:46.357194+0000","last_peered":"2024-09-21T10:54:46.357194+0000","last_clean":"2024-09-21T10:54:46.357194+0000","last_became_active":"2024-09-21T10:54:11.455158+0000","last_became_peered":"2024-09-21T10:54:11.455158+0000","last_unstale":"2024-09-21T10:54:46.357194+0000","last_undegraded":"2024-09-21T10:54:46.357194+0000","last_fullsized":"2024-09-21T10:54:46.357194+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":2941,"log_dups_size":0,"ondisk_log_size":2941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:51:41.079284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192741376,"num_objects":2941,"num_object_clones":0,"num_object_copies":8823,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2941,"num_write_kb":188224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"56.8","version":"1489'2483","reported_seq":2496,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:41.852186+0000","last_change":"2024-09-21T10:54:11.452926+0000","last_active":"2024-09-21T10:54:41.852186+0000","last_peered":"2024-09-21T10:54:41.852186+0000","last_clean":"2024-09-21T10:54:41.852186+0000","last_became_active":"2024-09-21T10:54:11.452630+0000","last_became_peered":"2024-09-21T10:54:11.452630+0000","last_unstale":"2024-09-21T10:54:41.852186+0000","last_undegraded":"2024-09-21T10:54:41.852186+0000","last_fullsized":"2024-09-21T10:54:41.852186+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":2483,"log_dups_size":0,"ondisk_log_size":2483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:07:29.362532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162725888,"num_objects":2483,"num_object_clones":0,"num_object_copies":7449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2483,"num_write_kb":158912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"56.7","version":"1489'2925","reported_seq":2939,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:46.363841+0000","last_change":"2024-09-21T10:54:11.454878+0000","last_active":"2024-09-21T10:54:46.363841+0000","last_peered":"2024-09-21T10:54:46.363841+0000","last_clean":"2024-09-21T10:54:46.363841+0000","last_became_active":"2024-09-21T10:54:11.454601+0000","last_became_peered":"2024-09-21T10:54:11.454601+0000","last_unstale":"2024-09-21T10:54:46.363841+0000","last_undegraded":"2024-09-21T10:54:46.363841+0000","last_fullsized":"2024-09-21T10:54:46.363841+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":2925,"log_dups_size":0,"ondisk_log_size":2925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:17:37.975888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191692800,"num_objects":2925,"num_object_clones":0,"num_object_copies":8775,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2925,"num_write_kb":187200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"56.6","version":"1489'2380","reported_seq":2394,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:42.020214+0000","last_change":"2024-09-21T10:54:11.453195+0000","last_active":"2024-09-21T10:54:42.020214+0000","last_peered":"2024-09-21T10:54:42.020214+0000","last_clean":"2024-09-21T10:54:42.020214+0000","last_became_active":"2024-09-21T10:54:11.452814+0000","last_became_peered":"2024-09-21T10:54:11.452814+0000","last_unstale":"2024-09-21T10:54:42.020214+0000","last_undegraded":"2024-09-21T10:54:42.020214+0000","last_fullsized":"2024-09-21T10:54:42.020214+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":2380,"log_dups_size":0,"ondisk_log_size":2380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:41:21.445831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155975680,"num_objects":2380,"num_object_clones":0,"num_object_copies":7140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2380,"num_write_kb":152320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"56.5","version":"1489'2744","reported_seq":2765,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:44.175562+0000","last_change":"2024-09-21T10:54:11.454780+0000","last_active":"2024-09-21T10:54:44.175562+0000","last_peered":"2024-09-21T10:54:44.175562+0000","last_clean":"2024-09-21T10:54:44.175562+0000","last_became_active":"2024-09-21T10:54:11.454545+0000","last_became_peered":"2024-09-21T10:54:11.454545+0000","last_unstale":"2024-09-21T10:54:44.175562+0000","last_undegraded":"2024-09-21T10:54:44.175562+0000","last_fullsized":"2024-09-21T10:54:44.175562+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":2744,"log_dups_size":0,"ondisk_log_size":2744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:55:33.056608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179830784,"num_objects":2744,"num_object_clones":0,"num_object_copies":8232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2744,"num_write_kb":175616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.4","version":"1489'3033","reported_seq":3047,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:46.359621+0000","last_change":"2024-09-21T10:54:11.454777+0000","last_active":"2024-09-21T10:54:46.359621+0000","last_peered":"2024-09-21T10:54:46.359621+0000","last_clean":"2024-09-21T10:54:46.359621+0000","last_became_active":"2024-09-21T10:54:11.454490+0000","last_became_peered":"2024-09-21T10:54:11.454490+0000","last_unstale":"2024-09-21T10:54:46.359621+0000","last_undegraded":"2024-09-21T10:54:46.359621+0000","last_fullsized":"2024-09-21T10:54:46.359621+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":3033,"log_dups_size":0,"ondisk_log_size":3033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:59.530447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198770688,"num_objects":3033,"num_object_clones":0,"num_object_copies":9099,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3033,"num_write_kb":194112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"56.3","version":"1489'2419","reported_seq":2433,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:42.037052+0000","last_change":"2024-09-21T10:54:11.452259+0000","last_active":"2024-09-21T10:54:42.037052+0000","last_peered":"2024-09-21T10:54:42.037052+0000","last_clean":"2024-09-21T10:54:42.037052+0000","last_became_active":"2024-09-21T10:54:11.451997+0000","last_became_peered":"2024-09-21T10:54:11.451997+0000","last_unstale":"2024-09-21T10:54:42.037052+0000","last_undegraded":"2024-09-21T10:54:42.037052+0000","last_fullsized":"2024-09-21T10:54:42.037052+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":2419,"log_dups_size":0,"ondisk_log_size":2419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:52:22.227114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158531584,"num_objects":2419,"num_object_clones":0,"num_object_copies":7257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2419,"num_write_kb":154816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"56.2","version":"1489'2731","reported_seq":2752,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:44.185688+0000","last_change":"2024-09-21T10:54:11.451633+0000","last_active":"2024-09-21T10:54:44.185688+0000","last_peered":"2024-09-21T10:54:44.185688+0000","last_clean":"2024-09-21T10:54:44.185688+0000","last_became_active":"2024-09-21T10:54:11.451333+0000","last_became_peered":"2024-09-21T10:54:11.451333+0000","last_unstale":"2024-09-21T10:54:44.185688+0000","last_undegraded":"2024-09-21T10:54:44.185688+0000","last_fullsized":"2024-09-21T10:54:44.185688+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":2731,"log_dups_size":0,"ondisk_log_size":2731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:17:20.683870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178978816,"num_objects":2731,"num_object_clones":0,"num_object_copies":8193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2731,"num_write_kb":174784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.1","version":"1489'2675","reported_seq":2696,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:44.190862+0000","last_change":"2024-09-21T10:54:11.451499+0000","last_active":"2024-09-21T10:54:44.190862+0000","last_peered":"2024-09-21T10:54:44.190862+0000","last_clean":"2024-09-21T10:54:44.190862+0000","last_became_active":"2024-09-21T10:54:11.451204+0000","last_became_peered":"2024-09-21T10:54:11.451204+0000","last_unstale":"2024-09-21T10:54:44.190862+0000","last_undegraded":"2024-09-21T10:54:44.190862+0000","last_fullsized":"2024-09-21T10:54:44.190862+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":2675,"log_dups_size":0,"ondisk_log_size":2675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:43.059873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175308800,"num_objects":2675,"num_object_clones":0,"num_object_copies":8025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2675,"num_write_kb":171200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.0","version":"1489'2714","reported_seq":2734,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:44.177580+0000","last_change":"2024-09-21T10:54:11.455255+0000","last_active":"2024-09-21T10:54:44.177580+0000","last_peered":"2024-09-21T10:54:44.177580+0000","last_clean":"2024-09-21T10:54:44.177580+0000","last_became_active":"2024-09-21T10:54:11.454741+0000","last_became_peered":"2024-09-21T10:54:11.454741+0000","last_unstale":"2024-09-21T10:54:44.177580+0000","last_undegraded":"2024-09-21T10:54:44.177580+0000","last_fullsized":"2024-09-21T10:54:44.177580+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":2714,"log_dups_size":0,"ondisk_log_size":2714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:42:08.583105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177864704,"num_objects":2714,"num_object_clones":0,"num_object_copies":8142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2714,"num_write_kb":173696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":1980,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.867445+0000","last_change":"2024-09-21T10:52:03.323899+0000","last_active":"2024-09-21T10:54:23.867445+0000","last_peered":"2024-09-21T10:54:23.867445+0000","last_clean":"2024-09-21T10:54:23.867445+0000","last_became_active":"2024-09-21T10:52:03.314626+0000","last_became_peered":"2024-09-21T10:52:03.314626+0000","last_unstale":"2024-09-21T10:54:23.867445+0000","last_undegraded":"2024-09-21T10:54:23.867445+0000","last_fullsized":"2024-09-21T10:54:23.867445+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:13.939806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1479,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.805928+0000","last_change":"2024-09-21T10:52:15.396755+0000","last_active":"2024-09-21T10:54:23.805928+0000","last_peered":"2024-09-21T10:54:23.805928+0000","last_clean":"2024-09-21T10:54:23.805928+0000","last_became_active":"2024-09-21T10:52:15.396417+0000","last_became_peered":"2024-09-21T10:52:15.396417+0000","last_unstale":"2024-09-21T10:54:23.805928+0000","last_undegraded":"2024-09-21T10:54:23.805928+0000","last_fullsized":"2024-09-21T10:54:23.805928+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:25.005574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":2288,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.567519+0000","last_change":"2024-09-21T10:52:15.400227+0000","last_active":"2024-09-21T10:54:25.567519+0000","last_peered":"2024-09-21T10:54:25.567519+0000","last_clean":"2024-09-21T10:54:25.567519+0000","last_became_active":"2024-09-21T10:52:15.399888+0000","last_became_peered":"2024-09-21T10:52:15.399888+0000","last_unstale":"2024-09-21T10:54:25.567519+0000","last_undegraded":"2024-09-21T10:54:25.567519+0000","last_fullsized":"2024-09-21T10:54:25.567519+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:42:52.565184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2304,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.566827+0000","last_change":"2024-09-21T10:51:27.458216+0000","last_active":"2024-09-21T10:54:25.566827+0000","last_peered":"2024-09-21T10:54:25.566827+0000","last_clean":"2024-09-21T10:54:25.566827+0000","last_became_active":"2024-09-21T10:51:27.457948+0000","last_became_peered":"2024-09-21T10:51:27.457948+0000","last_unstale":"2024-09-21T10:54:25.566827+0000","last_undegraded":"2024-09-21T10:54:25.566827+0000","last_fullsized":"2024-09-21T10:54:25.566827+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:25:29.584079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":118,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806575+0000","last_change":"2024-09-21T10:49:44.878242+0000","last_active":"2024-09-21T10:54:23.806575+0000","last_peered":"2024-09-21T10:54:23.806575+0000","last_clean":"2024-09-21T10:54:23.806575+0000","last_became_active":"2024-09-21T10:49:44.866578+0000","last_became_peered":"2024-09-21T10:49:44.866578+0000","last_unstale":"2024-09-21T10:54:23.806575+0000","last_undegraded":"2024-09-21T10:54:23.806575+0000","last_fullsized":"2024-09-21T10:54:23.806575+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:17:24.145551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1484,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806205+0000","last_change":"2024-09-21T10:51:27.460539+0000","last_active":"2024-09-21T10:54:23.806205+0000","last_peered":"2024-09-21T10:54:23.806205+0000","last_clean":"2024-09-21T10:54:23.806205+0000","last_became_active":"2024-09-21T10:51:27.460211+0000","last_became_peered":"2024-09-21T10:51:27.460211+0000","last_unstale":"2024-09-21T10:54:23.806205+0000","last_undegraded":"2024-09-21T10:54:23.806205+0000","last_fullsized":"2024-09-21T10:54:23.806205+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:14:15.700998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":157,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806404+0000","last_change":"2024-09-21T10:52:15.405886+0000","last_active":"2024-09-21T10:54:23.806404+0000","last_peered":"2024-09-21T10:54:23.806404+0000","last_clean":"2024-09-21T10:54:23.806404+0000","last_became_active":"2024-09-21T10:52:15.397317+0000","last_became_peered":"2024-09-21T10:52:15.397317+0000","last_unstale":"2024-09-21T10:54:23.806404+0000","last_undegraded":"2024-09-21T10:54:23.806404+0000","last_fullsized":"2024-09-21T10:54:23.806404+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:43:48.813861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1475,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806637+0000","last_change":"2024-09-21T10:09:06.560175+0000","last_active":"2024-09-21T10:54:23.806637+0000","last_peered":"2024-09-21T10:54:23.806637+0000","last_clean":"2024-09-21T10:54:23.806637+0000","last_became_active":"2024-09-21T10:09:06.558584+0000","last_became_peered":"2024-09-21T10:09:06.558584+0000","last_unstale":"2024-09-21T10:54:23.806637+0000","last_undegraded":"2024-09-21T10:54:23.806637+0000","last_fullsized":"2024-09-21T10:54:23.806637+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:02.918339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":66,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806233+0000","last_change":"2024-09-21T10:51:27.437665+0000","last_active":"2024-09-21T10:54:23.806233+0000","last_peered":"2024-09-21T10:54:23.806233+0000","last_clean":"2024-09-21T10:54:23.806233+0000","last_became_active":"2024-09-21T10:51:27.436481+0000","last_became_peered":"2024-09-21T10:51:27.436481+0000","last_unstale":"2024-09-21T10:54:23.806233+0000","last_undegraded":"2024-09-21T10:54:23.806233+0000","last_fullsized":"2024-09-21T10:54:23.806233+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:07:50.267297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1059,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.555771+0000","last_change":"2024-09-21T10:51:27.459033+0000","last_active":"2024-09-21T10:54:25.555771+0000","last_peered":"2024-09-21T10:54:25.555771+0000","last_clean":"2024-09-21T10:54:25.555771+0000","last_became_active":"2024-09-21T10:51:27.458728+0000","last_became_peered":"2024-09-21T10:51:27.458728+0000","last_unstale":"2024-09-21T10:54:25.555771+0000","last_undegraded":"2024-09-21T10:54:25.555771+0000","last_fullsized":"2024-09-21T10:54:25.555771+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:20.133032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":1504,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806480+0000","last_change":"2024-09-21T10:52:25.473550+0000","last_active":"2024-09-21T10:54:23.806480+0000","last_peered":"2024-09-21T10:54:23.806480+0000","last_clean":"2024-09-21T10:54:23.806480+0000","last_became_active":"2024-09-21T10:52:25.473105+0000","last_became_peered":"2024-09-21T10:52:25.473105+0000","last_unstale":"2024-09-21T10:54:23.806480+0000","last_undegraded":"2024-09-21T10:54:23.806480+0000","last_fullsized":"2024-09-21T10:54:23.806480+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:15:45.352518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2284,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.567332+0000","last_change":"2024-09-21T10:40:13.432531+0000","last_active":"2024-09-21T10:54:25.567332+0000","last_peered":"2024-09-21T10:54:25.567332+0000","last_clean":"2024-09-21T10:54:25.567332+0000","last_became_active":"2024-09-21T10:40:13.432210+0000","last_became_peered":"2024-09-21T10:40:13.432210+0000","last_unstale":"2024-09-21T10:54:25.567332+0000","last_undegraded":"2024-09-21T10:54:25.567332+0000","last_fullsized":"2024-09-21T10:54:25.567332+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:20:10.998607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2256,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.554976+0000","last_change":"2024-09-21T10:09:06.530394+0000","last_active":"2024-09-21T10:54:25.554976+0000","last_peered":"2024-09-21T10:54:25.554976+0000","last_clean":"2024-09-21T10:54:25.554976+0000","last_became_active":"2024-09-21T10:09:06.529307+0000","last_became_peered":"2024-09-21T10:09:06.529307+0000","last_unstale":"2024-09-21T10:54:25.554976+0000","last_undegraded":"2024-09-21T10:54:25.554976+0000","last_fullsized":"2024-09-21T10:54:25.554976+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:54.287397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":35,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.486419+0000","last_change":"2024-09-21T10:52:25.489896+0000","last_active":"2024-09-21T10:54:25.486419+0000","last_peered":"2024-09-21T10:54:25.486419+0000","last_clean":"2024-09-21T10:54:25.486419+0000","last_became_active":"2024-09-21T10:52:25.489154+0000","last_became_peered":"2024-09-21T10:52:25.489154+0000","last_unstale":"2024-09-21T10:54:25.486419+0000","last_undegraded":"2024-09-21T10:54:25.486419+0000","last_fullsized":"2024-09-21T10:54:25.486419+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:36:02.076931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2284,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.566261+0000","last_change":"2024-09-21T10:52:25.493309+0000","last_active":"2024-09-21T10:54:25.566261+0000","last_peered":"2024-09-21T10:54:25.566261+0000","last_clean":"2024-09-21T10:54:25.566261+0000","last_became_active":"2024-09-21T10:52:25.492568+0000","last_became_peered":"2024-09-21T10:52:25.492568+0000","last_unstale":"2024-09-21T10:54:25.566261+0000","last_undegraded":"2024-09-21T10:54:25.566261+0000","last_fullsized":"2024-09-21T10:54:25.566261+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:19:19.393370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1492,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.867484+0000","last_change":"2024-09-21T10:52:03.317474+0000","last_active":"2024-09-21T10:54:23.867484+0000","last_peered":"2024-09-21T10:54:23.867484+0000","last_clean":"2024-09-21T10:54:23.867484+0000","last_became_active":"2024-09-21T10:52:03.317001+0000","last_became_peered":"2024-09-21T10:52:03.317001+0000","last_unstale":"2024-09-21T10:54:23.867484+0000","last_undegraded":"2024-09-21T10:54:23.867484+0000","last_fullsized":"2024-09-21T10:54:23.867484+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:06:07.039763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2284,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.566904+0000","last_change":"2024-09-21T10:52:15.412246+0000","last_active":"2024-09-21T10:54:25.566904+0000","last_peered":"2024-09-21T10:54:25.566904+0000","last_clean":"2024-09-21T10:54:25.566904+0000","last_became_active":"2024-09-21T10:52:15.406204+0000","last_became_peered":"2024-09-21T10:52:15.406204+0000","last_unstale":"2024-09-21T10:54:25.566904+0000","last_undegraded":"2024-09-21T10:54:25.566904+0000","last_fullsized":"2024-09-21T10:54:25.566904+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:03.524324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2264,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.567740+0000","last_change":"2024-09-21T10:52:15.402822+0000","last_active":"2024-09-21T10:54:25.567740+0000","last_peered":"2024-09-21T10:54:25.567740+0000","last_clean":"2024-09-21T10:54:25.567740+0000","last_became_active":"2024-09-21T10:52:15.402548+0000","last_became_peered":"2024-09-21T10:52:15.402548+0000","last_unstale":"2024-09-21T10:54:25.567740+0000","last_undegraded":"2024-09-21T10:54:25.567740+0000","last_fullsized":"2024-09-21T10:54:25.567740+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:04:04.667598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2289,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.566121+0000","last_change":"2024-09-21T10:52:25.499172+0000","last_active":"2024-09-21T10:54:25.566121+0000","last_peered":"2024-09-21T10:54:25.566121+0000","last_clean":"2024-09-21T10:54:25.566121+0000","last_became_active":"2024-09-21T10:52:25.498835+0000","last_became_peered":"2024-09-21T10:52:25.498835+0000","last_unstale":"2024-09-21T10:54:25.566121+0000","last_undegraded":"2024-09-21T10:54:25.566121+0000","last_fullsized":"2024-09-21T10:54:25.566121+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:10:12.434470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":34,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.479090+0000","last_change":"2024-09-21T10:52:25.489774+0000","last_active":"2024-09-21T10:54:23.479090+0000","last_peered":"2024-09-21T10:54:23.479090+0000","last_clean":"2024-09-21T10:54:23.479090+0000","last_became_active":"2024-09-21T10:52:25.487788+0000","last_became_peered":"2024-09-21T10:52:25.487788+0000","last_unstale":"2024-09-21T10:54:23.479090+0000","last_undegraded":"2024-09-21T10:54:23.479090+0000","last_fullsized":"2024-09-21T10:54:23.479090+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:48.842419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":40,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.478348+0000","last_change":"2024-09-21T10:52:25.565465+0000","last_active":"2024-09-21T10:54:23.478348+0000","last_peered":"2024-09-21T10:54:23.478348+0000","last_clean":"2024-09-21T10:54:23.478348+0000","last_became_active":"2024-09-21T10:52:25.510131+0000","last_became_peered":"2024-09-21T10:52:25.510131+0000","last_unstale":"2024-09-21T10:54:23.478348+0000","last_undegraded":"2024-09-21T10:54:23.478348+0000","last_fullsized":"2024-09-21T10:54:23.478348+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:41:13.870503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":450,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2291,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.554289+0000","last_change":"2024-09-21T10:52:25.504787+0000","last_active":"2024-09-21T10:54:25.554289+0000","last_peered":"2024-09-21T10:54:25.554289+0000","last_clean":"2024-09-21T10:54:25.554289+0000","last_became_active":"2024-09-21T10:52:25.504198+0000","last_became_peered":"2024-09-21T10:52:25.504198+0000","last_unstale":"2024-09-21T10:54:25.554289+0000","last_undegraded":"2024-09-21T10:54:25.554289+0000","last_fullsized":"2024-09-21T10:54:25.554289+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:38:25.666224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":1509,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806577+0000","last_change":"2024-09-21T10:44:14.415264+0000","last_active":"2024-09-21T10:54:23.806577+0000","last_peered":"2024-09-21T10:54:23.806577+0000","last_clean":"2024-09-21T10:54:23.806577+0000","last_became_active":"2024-09-21T10:44:14.407813+0000","last_became_peered":"2024-09-21T10:44:14.407813+0000","last_unstale":"2024-09-21T10:54:23.806577+0000","last_undegraded":"2024-09-21T10:54:23.806577+0000","last_fullsized":"2024-09-21T10:54:23.806577+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:55.933601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1462,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806610+0000","last_change":"2024-09-21T10:07:56.001756+0000","last_active":"2024-09-21T10:54:23.806610+0000","last_peered":"2024-09-21T10:54:23.806610+0000","last_clean":"2024-09-21T10:54:23.806610+0000","last_became_active":"2024-09-21T10:07:56.000789+0000","last_became_peered":"2024-09-21T10:07:56.000789+0000","last_unstale":"2024-09-21T10:54:23.806610+0000","last_undegraded":"2024-09-21T10:54:23.806610+0000","last_fullsized":"2024-09-21T10:54:23.806610+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:56:04.302902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"346'149","reported_seq":952,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.567087+0000","last_change":"2024-09-21T10:53:47.069462+0000","last_active":"2024-09-21T10:54:25.567087+0000","last_peered":"2024-09-21T10:54:25.567087+0000","last_clean":"2024-09-21T10:54:25.567087+0000","last_became_active":"2024-09-21T10:53:47.069113+0000","last_became_peered":"2024-09-21T10:53:47.069113+0000","last_unstale":"2024-09-21T10:54:25.567087+0000","last_undegraded":"2024-09-21T10:54:25.567087+0000","last_fullsized":"2024-09-21T10:54:25.567087+0000","mapping_epoch":1478,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1479,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:55:05.610284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1579069,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1477,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806060+0000","last_change":"2024-09-21T10:52:25.473127+0000","last_active":"2024-09-21T10:54:23.806060+0000","last_peered":"2024-09-21T10:54:23.806060+0000","last_clean":"2024-09-21T10:54:23.806060+0000","last_became_active":"2024-09-21T10:52:25.472805+0000","last_became_peered":"2024-09-21T10:52:25.472805+0000","last_unstale":"2024-09-21T10:54:23.806060+0000","last_undegraded":"2024-09-21T10:54:23.806060+0000","last_fullsized":"2024-09-21T10:54:23.806060+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:13.038797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1523,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.567861+0000","last_change":"2024-09-21T10:52:25.493263+0000","last_active":"2024-09-21T10:54:25.567861+0000","last_peered":"2024-09-21T10:54:25.567861+0000","last_clean":"2024-09-21T10:54:25.567861+0000","last_became_active":"2024-09-21T10:52:25.492970+0000","last_became_peered":"2024-09-21T10:52:25.492970+0000","last_unstale":"2024-09-21T10:54:25.567861+0000","last_undegraded":"2024-09-21T10:54:25.567861+0000","last_fullsized":"2024-09-21T10:54:25.567861+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:43:40.305783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":135,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.554426+0000","last_change":"2024-09-21T10:52:25.513834+0000","last_active":"2024-09-21T10:54:25.554426+0000","last_peered":"2024-09-21T10:54:25.554426+0000","last_clean":"2024-09-21T10:54:25.554426+0000","last_became_active":"2024-09-21T10:52:25.475423+0000","last_became_peered":"2024-09-21T10:52:25.475423+0000","last_unstale":"2024-09-21T10:54:25.554426+0000","last_undegraded":"2024-09-21T10:54:25.554426+0000","last_fullsized":"2024-09-21T10:54:25.554426+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:10.216527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"346'149","reported_seq":921,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806319+0000","last_change":"2024-09-21T10:54:22.051517+0000","last_active":"2024-09-21T10:54:23.806319+0000","last_peered":"2024-09-21T10:54:23.806319+0000","last_clean":"2024-09-21T10:54:23.806319+0000","last_became_active":"2024-09-21T10:52:15.405544+0000","last_became_peered":"2024-09-21T10:52:15.405544+0000","last_unstale":"2024-09-21T10:54:23.806319+0000","last_undegraded":"2024-09-21T10:54:23.806319+0000","last_fullsized":"2024-09-21T10:54:23.806319+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:22.051403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:22.051403+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:15:18.487929+0000","scrub_duration":0.0016142979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2256,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.566257+0000","last_change":"2024-09-21T10:52:25.494316+0000","last_active":"2024-09-21T10:54:25.566257+0000","last_peered":"2024-09-21T10:54:25.566257+0000","last_clean":"2024-09-21T10:54:25.566257+0000","last_became_active":"2024-09-21T10:52:25.494016+0000","last_became_peered":"2024-09-21T10:52:25.494016+0000","last_unstale":"2024-09-21T10:54:25.566257+0000","last_undegraded":"2024-09-21T10:54:25.566257+0000","last_fullsized":"2024-09-21T10:54:25.566257+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:07.859584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2257,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.555649+0000","last_change":"2024-09-21T10:07:56.006236+0000","last_active":"2024-09-21T10:54:25.555649+0000","last_peered":"2024-09-21T10:54:25.555649+0000","last_clean":"2024-09-21T10:54:25.555649+0000","last_became_active":"2024-09-21T10:07:56.004439+0000","last_became_peered":"2024-09-21T10:07:56.004439+0000","last_unstale":"2024-09-21T10:54:25.555649+0000","last_undegraded":"2024-09-21T10:54:25.555649+0000","last_fullsized":"2024-09-21T10:54:25.555649+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:39.422467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":2307,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.566279+0000","last_change":"2024-09-21T10:52:25.536401+0000","last_active":"2024-09-21T10:54:25.566279+0000","last_peered":"2024-09-21T10:54:25.566279+0000","last_clean":"2024-09-21T10:54:25.566279+0000","last_became_active":"2024-09-21T10:52:25.496480+0000","last_became_peered":"2024-09-21T10:52:25.496480+0000","last_unstale":"2024-09-21T10:54:25.566279+0000","last_undegraded":"2024-09-21T10:54:25.566279+0000","last_fullsized":"2024-09-21T10:54:25.566279+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:50:34.820373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"346'149","reported_seq":734,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.867336+0000","last_change":"2024-09-21T10:51:30.450301+0000","last_active":"2024-09-21T10:54:23.867336+0000","last_peered":"2024-09-21T10:54:23.867336+0000","last_clean":"2024-09-21T10:54:23.867336+0000","last_became_active":"2024-09-21T10:51:30.449925+0000","last_became_peered":"2024-09-21T10:51:30.449925+0000","last_unstale":"2024-09-21T10:54:23.867336+0000","last_undegraded":"2024-09-21T10:54:23.867336+0000","last_fullsized":"2024-09-21T10:54:23.867336+0000","mapping_epoch":1450,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1451,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:41.011085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:41.011085+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:38:33.451261+0000","scrub_duration":0.0061250310000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2205,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.478382+0000","last_change":"2024-09-21T10:51:27.431416+0000","last_active":"2024-09-21T10:54:23.478382+0000","last_peered":"2024-09-21T10:54:23.478382+0000","last_clean":"2024-09-21T10:54:23.478382+0000","last_became_active":"2024-09-21T10:51:27.431127+0000","last_became_peered":"2024-09-21T10:51:27.431127+0000","last_unstale":"2024-09-21T10:54:23.478382+0000","last_undegraded":"2024-09-21T10:54:23.478382+0000","last_fullsized":"2024-09-21T10:54:23.478382+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:27.721665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":35,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.485991+0000","last_change":"2024-09-21T10:52:25.468849+0000","last_active":"2024-09-21T10:54:25.485991+0000","last_peered":"2024-09-21T10:54:25.485991+0000","last_clean":"2024-09-21T10:54:25.485991+0000","last_became_active":"2024-09-21T10:52:25.468552+0000","last_became_peered":"2024-09-21T10:52:25.468552+0000","last_unstale":"2024-09-21T10:54:25.485991+0000","last_undegraded":"2024-09-21T10:54:25.485991+0000","last_fullsized":"2024-09-21T10:54:25.485991+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:36:41.692028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2307,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.555712+0000","last_change":"2024-09-21T10:51:27.436608+0000","last_active":"2024-09-21T10:54:25.555712+0000","last_peered":"2024-09-21T10:54:25.555712+0000","last_clean":"2024-09-21T10:54:25.555712+0000","last_became_active":"2024-09-21T10:51:27.434949+0000","last_became_peered":"2024-09-21T10:51:27.434949+0000","last_unstale":"2024-09-21T10:54:25.555712+0000","last_undegraded":"2024-09-21T10:54:25.555712+0000","last_fullsized":"2024-09-21T10:54:25.555712+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:49:23.439425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"346'149","reported_seq":777,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.867417+0000","last_change":"2024-09-21T10:52:28.590793+0000","last_active":"2024-09-21T10:54:23.867417+0000","last_peered":"2024-09-21T10:54:23.867417+0000","last_clean":"2024-09-21T10:54:23.867417+0000","last_became_active":"2024-09-21T10:52:28.588662+0000","last_became_peered":"2024-09-21T10:52:28.588662+0000","last_unstale":"2024-09-21T10:54:23.867417+0000","last_undegraded":"2024-09-21T10:54:23.867417+0000","last_fullsized":"2024-09-21T10:54:23.867417+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:44.018645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:44.018645+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:46.849427+0000","scrub_duration":0.0012789399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2262,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.555301+0000","last_change":"2024-09-21T10:51:27.440425+0000","last_active":"2024-09-21T10:54:25.555301+0000","last_peered":"2024-09-21T10:54:25.555301+0000","last_clean":"2024-09-21T10:54:25.555301+0000","last_became_active":"2024-09-21T10:51:27.439666+0000","last_became_peered":"2024-09-21T10:51:27.439666+0000","last_unstale":"2024-09-21T10:54:25.555301+0000","last_undegraded":"2024-09-21T10:54:25.555301+0000","last_fullsized":"2024-09-21T10:54:25.555301+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:56:47.410516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2330,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.555603+0000","last_change":"2024-09-21T10:52:03.314926+0000","last_active":"2024-09-21T10:54:25.555603+0000","last_peered":"2024-09-21T10:54:25.555603+0000","last_clean":"2024-09-21T10:54:25.555603+0000","last_became_active":"2024-09-21T10:52:03.314602+0000","last_became_peered":"2024-09-21T10:52:03.314602+0000","last_unstale":"2024-09-21T10:54:25.555603+0000","last_undegraded":"2024-09-21T10:54:25.555603+0000","last_fullsized":"2024-09-21T10:54:25.555603+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:13.774521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1485,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.566276+0000","last_change":"2024-09-21T10:52:25.502830+0000","last_active":"2024-09-21T10:54:25.566276+0000","last_peered":"2024-09-21T10:54:25.566276+0000","last_clean":"2024-09-21T10:54:25.566276+0000","last_became_active":"2024-09-21T10:52:25.502505+0000","last_became_peered":"2024-09-21T10:52:25.502505+0000","last_unstale":"2024-09-21T10:54:25.566276+0000","last_undegraded":"2024-09-21T10:54:25.566276+0000","last_fullsized":"2024-09-21T10:54:25.566276+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:01:14.965694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"346'149","reported_seq":844,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.805821+0000","last_change":"2024-09-21T10:54:16.021952+0000","last_active":"2024-09-21T10:54:23.805821+0000","last_peered":"2024-09-21T10:54:23.805821+0000","last_clean":"2024-09-21T10:54:23.805821+0000","last_became_active":"2024-09-21T10:51:30.436983+0000","last_became_peered":"2024-09-21T10:51:30.436983+0000","last_unstale":"2024-09-21T10:54:23.805821+0000","last_undegraded":"2024-09-21T10:54:23.805821+0000","last_fullsized":"2024-09-21T10:54:23.805821+0000","mapping_epoch":1450,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1451,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:16.021843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:16.021843+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:41:43.225623+0000","scrub_duration":0.0036826160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2324,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.555900+0000","last_change":"2024-09-21T10:52:15.397180+0000","last_active":"2024-09-21T10:54:25.555900+0000","last_peered":"2024-09-21T10:54:25.555900+0000","last_clean":"2024-09-21T10:54:25.555900+0000","last_became_active":"2024-09-21T10:52:15.396879+0000","last_became_peered":"2024-09-21T10:52:15.396879+0000","last_unstale":"2024-09-21T10:54:25.555900+0000","last_undegraded":"2024-09-21T10:54:25.555900+0000","last_fullsized":"2024-09-21T10:54:25.555900+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:03:07.162178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1475,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806800+0000","last_change":"2024-09-21T10:51:27.436590+0000","last_active":"2024-09-21T10:54:23.806800+0000","last_peered":"2024-09-21T10:54:23.806800+0000","last_clean":"2024-09-21T10:54:23.806800+0000","last_became_active":"2024-09-21T10:51:27.435332+0000","last_became_peered":"2024-09-21T10:51:27.435332+0000","last_unstale":"2024-09-21T10:54:23.806800+0000","last_undegraded":"2024-09-21T10:54:23.806800+0000","last_fullsized":"2024-09-21T10:54:23.806800+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:05:42.885506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":224,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.555228+0000","last_change":"2024-09-21T10:53:41.149328+0000","last_active":"2024-09-21T10:54:25.555228+0000","last_peered":"2024-09-21T10:54:25.555228+0000","last_clean":"2024-09-21T10:54:25.555228+0000","last_became_active":"2024-09-21T10:52:03.315398+0000","last_became_peered":"2024-09-21T10:52:03.315398+0000","last_unstale":"2024-09-21T10:54:25.555228+0000","last_undegraded":"2024-09-21T10:54:25.555228+0000","last_fullsized":"2024-09-21T10:54:25.555228+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:48:09.007053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"346'149","reported_seq":815,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.485931+0000","last_change":"2024-09-21T10:54:21.445887+0000","last_active":"2024-09-21T10:54:25.485931+0000","last_peered":"2024-09-21T10:54:25.485931+0000","last_clean":"2024-09-21T10:54:25.485931+0000","last_became_active":"2024-09-21T10:52:28.494923+0000","last_became_peered":"2024-09-21T10:52:28.494923+0000","last_unstale":"2024-09-21T10:54:25.485931+0000","last_undegraded":"2024-09-21T10:54:25.485931+0000","last_fullsized":"2024-09-21T10:54:25.485931+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:21.445783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:21.445783+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:33:42.131030+0000","scrub_duration":0.0018624990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":193,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.568858+0000","last_change":"2024-09-21T10:52:25.475200+0000","last_active":"2024-09-21T10:54:25.568858+0000","last_peered":"2024-09-21T10:54:25.568858+0000","last_clean":"2024-09-21T10:54:25.568858+0000","last_became_active":"2024-09-21T10:52:25.474872+0000","last_became_peered":"2024-09-21T10:52:25.474872+0000","last_unstale":"2024-09-21T10:54:25.568858+0000","last_undegraded":"2024-09-21T10:54:25.568858+0000","last_fullsized":"2024-09-21T10:54:25.568858+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:36:34.207844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2281,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.554716+0000","last_change":"2024-09-21T10:52:03.325889+0000","last_active":"2024-09-21T10:54:25.554716+0000","last_peered":"2024-09-21T10:54:25.554716+0000","last_clean":"2024-09-21T10:54:25.554716+0000","last_became_active":"2024-09-21T10:52:03.325500+0000","last_became_peered":"2024-09-21T10:52:03.325500+0000","last_unstale":"2024-09-21T10:54:25.554716+0000","last_undegraded":"2024-09-21T10:54:25.554716+0000","last_fullsized":"2024-09-21T10:54:25.554716+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:20.708252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1066,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.566779+0000","last_change":"2024-09-21T10:51:27.457894+0000","last_active":"2024-09-21T10:54:25.566779+0000","last_peered":"2024-09-21T10:54:25.566779+0000","last_clean":"2024-09-21T10:54:25.566779+0000","last_became_active":"2024-09-21T10:51:27.457431+0000","last_became_peered":"2024-09-21T10:51:27.457431+0000","last_unstale":"2024-09-21T10:54:25.566779+0000","last_undegraded":"2024-09-21T10:54:25.566779+0000","last_fullsized":"2024-09-21T10:54:25.566779+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:03:33.550759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"346'149","reported_seq":937,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.555667+0000","last_change":"2024-09-21T10:52:15.396042+0000","last_active":"2024-09-21T10:54:25.555667+0000","last_peered":"2024-09-21T10:54:25.555667+0000","last_clean":"2024-09-21T10:54:25.555667+0000","last_became_active":"2024-09-21T10:52:15.395717+0000","last_became_peered":"2024-09-21T10:52:15.395717+0000","last_unstale":"2024-09-21T10:54:25.555667+0000","last_undegraded":"2024-09-21T10:54:25.555667+0000","last_fullsized":"2024-09-21T10:54:25.555667+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:55:04.143095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2306,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.554533+0000","last_change":"2024-09-21T10:52:25.476572+0000","last_active":"2024-09-21T10:54:25.554533+0000","last_peered":"2024-09-21T10:54:25.554533+0000","last_clean":"2024-09-21T10:54:25.554533+0000","last_became_active":"2024-09-21T10:52:25.475803+0000","last_became_peered":"2024-09-21T10:52:25.475803+0000","last_unstale":"2024-09-21T10:54:25.554533+0000","last_undegraded":"2024-09-21T10:54:25.554533+0000","last_fullsized":"2024-09-21T10:54:25.554533+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:06:44.191486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2221,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.805953+0000","last_change":"2024-09-21T10:52:03.326753+0000","last_active":"2024-09-21T10:54:23.805953+0000","last_peered":"2024-09-21T10:54:23.805953+0000","last_clean":"2024-09-21T10:54:23.805953+0000","last_became_active":"2024-09-21T10:52:03.326348+0000","last_became_peered":"2024-09-21T10:52:03.326348+0000","last_unstale":"2024-09-21T10:54:23.805953+0000","last_undegraded":"2024-09-21T10:54:23.805953+0000","last_fullsized":"2024-09-21T10:54:23.805953+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:57:38.100799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":144,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.478004+0000","last_change":"2024-09-21T10:51:27.426006+0000","last_active":"2024-09-21T10:54:23.478004+0000","last_peered":"2024-09-21T10:54:23.478004+0000","last_clean":"2024-09-21T10:54:23.478004+0000","last_became_active":"2024-09-21T10:51:27.425698+0000","last_became_peered":"2024-09-21T10:51:27.425698+0000","last_unstale":"2024-09-21T10:54:23.478004+0000","last_undegraded":"2024-09-21T10:54:23.478004+0000","last_fullsized":"2024-09-21T10:54:23.478004+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:35:13.210330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":38,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.867023+0000","last_change":"2024-09-21T10:52:15.401737+0000","last_active":"2024-09-21T10:54:23.867023+0000","last_peered":"2024-09-21T10:54:23.867023+0000","last_clean":"2024-09-21T10:54:23.867023+0000","last_became_active":"2024-09-21T10:52:15.400859+0000","last_became_peered":"2024-09-21T10:52:15.400859+0000","last_unstale":"2024-09-21T10:54:23.867023+0000","last_undegraded":"2024-09-21T10:54:23.867023+0000","last_fullsized":"2024-09-21T10:54:23.867023+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:15.723031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2260,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.567016+0000","last_change":"2024-09-21T10:49:44.865830+0000","last_active":"2024-09-21T10:54:25.567016+0000","last_peered":"2024-09-21T10:54:25.567016+0000","last_clean":"2024-09-21T10:54:25.567016+0000","last_became_active":"2024-09-21T10:49:44.865499+0000","last_became_peered":"2024-09-21T10:49:44.865499+0000","last_unstale":"2024-09-21T10:54:25.567016+0000","last_undegraded":"2024-09-21T10:54:25.567016+0000","last_fullsized":"2024-09-21T10:54:25.567016+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:19:22.013154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":1469,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.485981+0000","last_change":"2024-09-21T10:52:25.489550+0000","last_active":"2024-09-21T10:54:25.485981+0000","last_peered":"2024-09-21T10:54:25.485981+0000","last_clean":"2024-09-21T10:54:25.485981+0000","last_became_active":"2024-09-21T10:52:25.489171+0000","last_became_peered":"2024-09-21T10:52:25.489171+0000","last_unstale":"2024-09-21T10:54:25.485981+0000","last_undegraded":"2024-09-21T10:54:25.485981+0000","last_fullsized":"2024-09-21T10:54:25.485981+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:21.105058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":815,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.485628+0000","last_change":"2024-09-21T10:54:20.458538+0000","last_active":"2024-09-21T10:54:25.485628+0000","last_peered":"2024-09-21T10:54:25.485628+0000","last_clean":"2024-09-21T10:54:25.485628+0000","last_became_active":"2024-09-21T10:52:28.587447+0000","last_became_peered":"2024-09-21T10:52:28.587447+0000","last_unstale":"2024-09-21T10:54:25.485628+0000","last_undegraded":"2024-09-21T10:54:25.485628+0000","last_fullsized":"2024-09-21T10:54:25.485628+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:20.458421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:20.458421+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:35.467492+0000","scrub_duration":0.001528569,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"346'149","reported_seq":751,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.867131+0000","last_change":"2024-09-21T10:51:30.453979+0000","last_active":"2024-09-21T10:54:23.867131+0000","last_peered":"2024-09-21T10:54:23.867131+0000","last_clean":"2024-09-21T10:54:23.867131+0000","last_became_active":"2024-09-21T10:51:30.450893+0000","last_became_peered":"2024-09-21T10:51:30.450893+0000","last_unstale":"2024-09-21T10:54:23.867131+0000","last_undegraded":"2024-09-21T10:54:23.867131+0000","last_fullsized":"2024-09-21T10:54:23.867131+0000","mapping_epoch":1450,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1451,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:42.008235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:42.008235+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:21:48.570809+0000","scrub_duration":0.011129834999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2257,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.555621+0000","last_change":"2024-09-21T10:09:06.538940+0000","last_active":"2024-09-21T10:54:25.555621+0000","last_peered":"2024-09-21T10:54:25.555621+0000","last_clean":"2024-09-21T10:54:25.555621+0000","last_became_active":"2024-09-21T10:09:06.538668+0000","last_became_peered":"2024-09-21T10:09:06.538668+0000","last_unstale":"2024-09-21T10:54:25.555621+0000","last_undegraded":"2024-09-21T10:54:25.555621+0000","last_fullsized":"2024-09-21T10:54:25.555621+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:42:44.299488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2256,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.566404+0000","last_change":"2024-09-21T10:52:25.493032+0000","last_active":"2024-09-21T10:54:25.566404+0000","last_peered":"2024-09-21T10:54:25.566404+0000","last_clean":"2024-09-21T10:54:25.566404+0000","last_became_active":"2024-09-21T10:52:25.492182+0000","last_became_peered":"2024-09-21T10:52:25.492182+0000","last_unstale":"2024-09-21T10:54:25.566404+0000","last_undegraded":"2024-09-21T10:54:25.566404+0000","last_fullsized":"2024-09-21T10:54:25.566404+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:12:57.118420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1481,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806795+0000","last_change":"2024-09-21T10:52:15.402809+0000","last_active":"2024-09-21T10:54:23.806795+0000","last_peered":"2024-09-21T10:54:23.806795+0000","last_clean":"2024-09-21T10:54:23.806795+0000","last_became_active":"2024-09-21T10:52:15.402497+0000","last_became_peered":"2024-09-21T10:52:15.402497+0000","last_unstale":"2024-09-21T10:54:23.806795+0000","last_undegraded":"2024-09-21T10:54:23.806795+0000","last_fullsized":"2024-09-21T10:54:23.806795+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:07:35.713534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1492,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806255+0000","last_change":"2024-09-21T10:44:14.399842+0000","last_active":"2024-09-21T10:54:23.806255+0000","last_peered":"2024-09-21T10:54:23.806255+0000","last_clean":"2024-09-21T10:54:23.806255+0000","last_became_active":"2024-09-21T10:44:14.399426+0000","last_became_peered":"2024-09-21T10:44:14.399426+0000","last_unstale":"2024-09-21T10:54:23.806255+0000","last_undegraded":"2024-09-21T10:54:23.806255+0000","last_fullsized":"2024-09-21T10:54:23.806255+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:26:45.362101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"346'149","reported_seq":937,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.566767+0000","last_change":"2024-09-21T10:52:16.404597+0000","last_active":"2024-09-21T10:54:25.566767+0000","last_peered":"2024-09-21T10:54:25.566767+0000","last_clean":"2024-09-21T10:54:25.566767+0000","last_became_active":"2024-09-21T10:52:16.404194+0000","last_became_peered":"2024-09-21T10:52:16.404194+0000","last_unstale":"2024-09-21T10:54:25.566767+0000","last_undegraded":"2024-09-21T10:54:25.566767+0000","last_fullsized":"2024-09-21T10:54:25.566767+0000","mapping_epoch":1463,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1464,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:55:01.626212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":793725,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1487,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806592+0000","last_change":"2024-09-21T10:40:13.432387+0000","last_active":"2024-09-21T10:54:23.806592+0000","last_peered":"2024-09-21T10:54:23.806592+0000","last_clean":"2024-09-21T10:54:23.806592+0000","last_became_active":"2024-09-21T10:40:13.432053+0000","last_became_peered":"2024-09-21T10:40:13.432053+0000","last_unstale":"2024-09-21T10:54:23.806592+0000","last_undegraded":"2024-09-21T10:54:23.806592+0000","last_fullsized":"2024-09-21T10:54:23.806592+0000","mapping_epoch":1305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:37.486101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1465,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.867382+0000","last_change":"2024-09-21T10:07:56.020791+0000","last_active":"2024-09-21T10:54:23.867382+0000","last_peered":"2024-09-21T10:54:23.867382+0000","last_clean":"2024-09-21T10:54:23.867382+0000","last_became_active":"2024-09-21T10:07:56.020205+0000","last_became_peered":"2024-09-21T10:07:56.020205+0000","last_unstale":"2024-09-21T10:54:23.867382+0000","last_undegraded":"2024-09-21T10:54:23.867382+0000","last_fullsized":"2024-09-21T10:54:23.867382+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:09.813166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"346'149","reported_seq":940,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.477924+0000","last_change":"2024-09-21T10:54:17.418763+0000","last_active":"2024-09-21T10:54:23.477924+0000","last_peered":"2024-09-21T10:54:23.477924+0000","last_clean":"2024-09-21T10:54:23.477924+0000","last_became_active":"2024-09-21T10:49:47.903650+0000","last_became_peered":"2024-09-21T10:49:47.903650+0000","last_unstale":"2024-09-21T10:54:23.477924+0000","last_undegraded":"2024-09-21T10:54:23.477924+0000","last_fullsized":"2024-09-21T10:54:23.477924+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:17.418646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:17.418646+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:17.134488+0000","scrub_duration":0.00185648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1463,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806415+0000","last_change":"2024-09-21T10:09:06.533271+0000","last_active":"2024-09-21T10:54:23.806415+0000","last_peered":"2024-09-21T10:54:23.806415+0000","last_clean":"2024-09-21T10:54:23.806415+0000","last_became_active":"2024-09-21T10:09:06.532041+0000","last_became_peered":"2024-09-21T10:09:06.532041+0000","last_unstale":"2024-09-21T10:54:23.806415+0000","last_undegraded":"2024-09-21T10:54:23.806415+0000","last_fullsized":"2024-09-21T10:54:23.806415+0000","mapping_epoch":692,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:04.236375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":2285,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.567122+0000","last_change":"2024-09-21T10:51:27.529295+0000","last_active":"2024-09-21T10:54:25.567122+0000","last_peered":"2024-09-21T10:54:25.567122+0000","last_clean":"2024-09-21T10:54:25.567122+0000","last_became_active":"2024-09-21T10:51:27.446359+0000","last_became_peered":"2024-09-21T10:51:27.446359+0000","last_unstale":"2024-09-21T10:54:25.567122+0000","last_undegraded":"2024-09-21T10:54:25.567122+0000","last_fullsized":"2024-09-21T10:54:25.567122+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:17:08.788558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":715,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.555849+0000","last_change":"2024-09-21T10:51:27.481103+0000","last_active":"2024-09-21T10:54:25.555849+0000","last_peered":"2024-09-21T10:54:25.555849+0000","last_clean":"2024-09-21T10:54:25.555849+0000","last_became_active":"2024-09-21T10:51:27.480763+0000","last_became_peered":"2024-09-21T10:51:27.480763+0000","last_unstale":"2024-09-21T10:54:25.555849+0000","last_undegraded":"2024-09-21T10:54:25.555849+0000","last_fullsized":"2024-09-21T10:54:25.555849+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:35.375303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":820,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.567137+0000","last_change":"2024-09-21T10:49:44.864819+0000","last_active":"2024-09-21T10:54:25.567137+0000","last_peered":"2024-09-21T10:54:25.567137+0000","last_clean":"2024-09-21T10:54:25.567137+0000","last_became_active":"2024-09-21T10:49:44.864061+0000","last_became_peered":"2024-09-21T10:49:44.864061+0000","last_unstale":"2024-09-21T10:54:25.567137+0000","last_undegraded":"2024-09-21T10:54:25.567137+0000","last_fullsized":"2024-09-21T10:54:25.567137+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:37.266967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":47,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.567277+0000","last_change":"2024-09-21T10:52:25.492570+0000","last_active":"2024-09-21T10:54:25.567277+0000","last_peered":"2024-09-21T10:54:25.567277+0000","last_clean":"2024-09-21T10:54:25.567277+0000","last_became_active":"2024-09-21T10:52:25.486382+0000","last_became_peered":"2024-09-21T10:52:25.486382+0000","last_unstale":"2024-09-21T10:54:25.567277+0000","last_undegraded":"2024-09-21T10:54:25.567277+0000","last_fullsized":"2024-09-21T10:54:25.567277+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:06.702707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":151,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.478598+0000","last_change":"2024-09-21T10:49:44.874522+0000","last_active":"2024-09-21T10:54:23.478598+0000","last_peered":"2024-09-21T10:54:23.478598+0000","last_clean":"2024-09-21T10:54:23.478598+0000","last_became_active":"2024-09-21T10:49:44.863242+0000","last_became_peered":"2024-09-21T10:49:44.863242+0000","last_unstale":"2024-09-21T10:54:23.478598+0000","last_undegraded":"2024-09-21T10:54:23.478598+0000","last_fullsized":"2024-09-21T10:54:23.478598+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:09:02.066837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1497,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.806076+0000","last_change":"2024-09-21T10:52:03.315267+0000","last_active":"2024-09-21T10:54:23.806076+0000","last_peered":"2024-09-21T10:54:23.806076+0000","last_clean":"2024-09-21T10:54:23.806076+0000","last_became_active":"2024-09-21T10:52:03.314474+0000","last_became_peered":"2024-09-21T10:52:03.314474+0000","last_unstale":"2024-09-21T10:54:23.806076+0000","last_undegraded":"2024-09-21T10:54:23.806076+0000","last_fullsized":"2024-09-21T10:54:23.806076+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:00:57.791341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":39,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.486174+0000","last_change":"2024-09-21T10:52:25.518721+0000","last_active":"2024-09-21T10:54:25.486174+0000","last_peered":"2024-09-21T10:54:25.486174+0000","last_clean":"2024-09-21T10:54:25.486174+0000","last_became_active":"2024-09-21T10:52:25.472114+0000","last_became_peered":"2024-09-21T10:52:25.472114+0000","last_unstale":"2024-09-21T10:54:25.486174+0000","last_undegraded":"2024-09-21T10:54:25.486174+0000","last_fullsized":"2024-09-21T10:54:25.486174+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:33:48.471147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":35,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.485851+0000","last_change":"2024-09-21T10:52:25.494490+0000","last_active":"2024-09-21T10:54:25.485851+0000","last_peered":"2024-09-21T10:54:25.485851+0000","last_clean":"2024-09-21T10:54:25.485851+0000","last_became_active":"2024-09-21T10:52:25.494200+0000","last_became_peered":"2024-09-21T10:52:25.494200+0000","last_unstale":"2024-09-21T10:54:25.485851+0000","last_undegraded":"2024-09-21T10:54:25.485851+0000","last_fullsized":"2024-09-21T10:54:25.485851+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:15:21.030031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":39,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.486485+0000","last_change":"2024-09-21T10:52:25.479229+0000","last_active":"2024-09-21T10:54:25.486485+0000","last_peered":"2024-09-21T10:54:25.486485+0000","last_clean":"2024-09-21T10:54:25.486485+0000","last_became_active":"2024-09-21T10:52:25.470551+0000","last_became_peered":"2024-09-21T10:52:25.470551+0000","last_unstale":"2024-09-21T10:54:25.486485+0000","last_undegraded":"2024-09-21T10:54:25.486485+0000","last_fullsized":"2024-09-21T10:54:25.486485+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:15.458231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1509,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.866809+0000","last_change":"2024-09-21T10:51:27.517234+0000","last_active":"2024-09-21T10:54:23.866809+0000","last_peered":"2024-09-21T10:54:23.866809+0000","last_clean":"2024-09-21T10:54:23.866809+0000","last_became_active":"2024-09-21T10:51:27.431253+0000","last_became_peered":"2024-09-21T10:51:27.431253+0000","last_unstale":"2024-09-21T10:54:23.866809+0000","last_undegraded":"2024-09-21T10:54:23.866809+0000","last_fullsized":"2024-09-21T10:54:23.866809+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:35:42.249305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2240,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.566383+0000","last_change":"2024-09-21T10:52:25.497298+0000","last_active":"2024-09-21T10:54:25.566383+0000","last_peered":"2024-09-21T10:54:25.566383+0000","last_clean":"2024-09-21T10:54:25.566383+0000","last_became_active":"2024-09-21T10:52:25.496876+0000","last_became_peered":"2024-09-21T10:52:25.496876+0000","last_unstale":"2024-09-21T10:54:25.566383+0000","last_undegraded":"2024-09-21T10:54:25.566383+0000","last_fullsized":"2024-09-21T10:54:25.566383+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:16.587897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":134,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.478084+0000","last_change":"2024-09-21T10:44:14.398878+0000","last_active":"2024-09-21T10:54:23.478084+0000","last_peered":"2024-09-21T10:54:23.478084+0000","last_clean":"2024-09-21T10:54:23.478084+0000","last_became_active":"2024-09-21T10:44:14.398547+0000","last_became_peered":"2024-09-21T10:44:14.398547+0000","last_unstale":"2024-09-21T10:54:23.478084+0000","last_undegraded":"2024-09-21T10:54:23.478084+0000","last_fullsized":"2024-09-21T10:54:23.478084+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:13.008134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2288,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.566414+0000","last_change":"2024-09-21T10:52:25.498554+0000","last_active":"2024-09-21T10:54:25.566414+0000","last_peered":"2024-09-21T10:54:25.566414+0000","last_clean":"2024-09-21T10:54:25.566414+0000","last_became_active":"2024-09-21T10:52:25.498194+0000","last_became_peered":"2024-09-21T10:52:25.498194+0000","last_unstale":"2024-09-21T10:54:25.566414+0000","last_undegraded":"2024-09-21T10:54:25.566414+0000","last_fullsized":"2024-09-21T10:54:25.566414+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:47.115271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":34,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.478172+0000","last_change":"2024-09-21T10:52:25.489063+0000","last_active":"2024-09-21T10:54:23.478172+0000","last_peered":"2024-09-21T10:54:23.478172+0000","last_clean":"2024-09-21T10:54:23.478172+0000","last_became_active":"2024-09-21T10:52:25.488129+0000","last_became_peered":"2024-09-21T10:52:25.488129+0000","last_unstale":"2024-09-21T10:54:23.478172+0000","last_undegraded":"2024-09-21T10:54:23.478172+0000","last_fullsized":"2024-09-21T10:54:23.478172+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:36.996829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1449,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.478498+0000","last_change":"2024-09-21T10:49:44.864898+0000","last_active":"2024-09-21T10:54:23.478498+0000","last_peered":"2024-09-21T10:54:23.478498+0000","last_clean":"2024-09-21T10:54:23.478498+0000","last_became_active":"2024-09-21T10:49:44.862691+0000","last_became_peered":"2024-09-21T10:49:44.862691+0000","last_unstale":"2024-09-21T10:54:23.478498+0000","last_undegraded":"2024-09-21T10:54:23.478498+0000","last_fullsized":"2024-09-21T10:54:23.478498+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:57.809113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":140,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-09-21T10:54:23.478576+0000","last_change":"2024-09-21T10:44:14.418113+0000","last_active":"2024-09-21T10:54:23.478576+0000","last_peered":"2024-09-21T10:54:23.478576+0000","last_clean":"2024-09-21T10:54:23.478576+0000","last_became_active":"2024-09-21T10:44:14.398327+0000","last_became_peered":"2024-09-21T10:44:14.398327+0000","last_unstale":"2024-09-21T10:54:23.478576+0000","last_undegraded":"2024-09-21T10:54:23.478576+0000","last_fullsized":"2024-09-21T10:54:23.478576+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:08.810863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2269,"reported_epoch":1489,"state":"active+clean","last_fresh":"2024-09-21T10:54:25.567239+0000","last_change":"2024-09-21T10:44:14.398584+0000","last_active":"2024-09-21T10:54:25.567239+0000","last_peered":"2024-09-21T10:54:25.567239+0000","last_clean":"2024-09-21T10:54:25.567239+0000","last_became_active":"2024-09-21T10:44:14.398278+0000","last_became_peered":"2024-09-21T10:44:14.398278+0000","last_unstale":"2024-09-21T10:54:25.567239+0000","last_undegraded":"2024-09-21T10:54:25.567239+0000","last_fullsized":"2024-09-21T10:54:25.567239+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:19.681750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":56,"num_pg":16,"stat_sum":{"num_bytes":2822111232,"num_objects":43062,"num_object_clones":0,"num_object_copies":129186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43062,"num_write_kb":2755968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8231583744,"data_stored":8231583744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43062,"ondisk_log_size":43062,"up":48,"acting":48,"num_store_stats":5},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":161010,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":225,"num_write":59,"num_write_kb":659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":2641143,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":33,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":11344,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":18008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234728,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2306876,"kb_used_data":1694196,"kb_used_omap":1,"kb_used_meta":612670,"kb_avail":91438276,"statfs":{"total":95995035648,"available":93632794624,"internally_reserved":0,"allocated":1734856704,"data_stored":1729742976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":627374539},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,4,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496365,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2640320,"kb_used_data":1855352,"kb_used_omap":1,"kb_used_meta":784958,"kb_avail":91104832,"statfs":{"total":95995035648,"available":93291347968,"internally_reserved":0,"allocated":1899880448,"data_stored":1894763657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":803797450},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208758002,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019639,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2558208,"kb_used_data":1859384,"kb_used_omap":1,"kb_used_meta":698814,"kb_avail":91186944,"statfs":{"total":95995035648,"available":93375430656,"internally_reserved":0,"allocated":1904009216,"data_stored":1898892448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":715585990},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369382,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2037876,"kb_used_data":1189420,"kb_used_omap":69,"kb_used_meta":848378,"kb_avail":91707276,"statfs":{"total":95995035648,"available":93908250624,"internally_reserved":0,"allocated":1217966080,"data_stored":1212856889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":868739202},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2920577761837,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807958068,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2063656,"kb_used_data":1319456,"kb_used_omap":66,"kb_used_meta":744125,"kb_avail":91681496,"statfs":{"total":95995035648,"available":93881851904,"internally_reserved":0,"allocated":1351122944,"data_stored":1346024777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68589,"internal_metadata":761984019},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":2860448219707,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1389887488,"data_stored":1389887488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1243873280,"data_stored":1243873280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1925906432,"data_stored":1925906432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1910243328,"data_stored":1910243328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1761673216,"data_stored":1761673216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:54:51.661 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T10:54:51.661 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:54:51.661 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [1, 3, 4, 6, 7] 2024-09-21T10:54:51.661 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd out 4 2024-09-21T10:54:51.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:51.662 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:51.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:51.673 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:54:51.764 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:51.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:51.793 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:51.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:51.894 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:51.926 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:51.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:51.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:52.028 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:52.068 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:52.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:52.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:52.170 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:52.197 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:52.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:52.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:52.299 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:52.338 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:52.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:52.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:52.439 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:52.478 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:52.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:52.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:52.550 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 26 16 56199 56183 135.04 94.375 0.010634 0.00739957 2024-09-21T10:54:52.579 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:52 smithi070 ceph-mon[107385]: pgmap v2769: 99 pgs: 99 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-09-21T10:54:52.579 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:52 smithi070 ceph-mon[112139]: pgmap v2769: 99 pgs: 99 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-09-21T10:54:52.580 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:52.619 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:52.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:52.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:52.721 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:52.758 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:52.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:52.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:52 smithi160 ceph-mon[101879]: pgmap v2769: 99 pgs: 99 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-09-21T10:54:52.860 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:52.888 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:52.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:52.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:52.990 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:53.032 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:53.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:53.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:53.133 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:53.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:53.170 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:53.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:53.228 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:54:53.270 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:53.302 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:53.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:53.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:53.404 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:53.440 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:53.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:53.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:53.482 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:54:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:54:53] "GET /metrics HTTP/1.1" 200 36597 "" "Prometheus/2.51.0" 2024-09-21T10:54:53.542 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:53.550 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 27 16 57679 57663 133.464 92.5 0.00905347 0.00748837 2024-09-21T10:54:53.568 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:53.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:53.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:53.669 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:53.699 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:53.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:53.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:53.801 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:53.803 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:54:53.803 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:54:53.833 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:53.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:53.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:53.834 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:54:53.834 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:54:53.861 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:54:53.861 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:54:53.886 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:54:53.886 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:54:53.911 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:54:53.912 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:54:53.935 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:53.936 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:54:53.936 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:54:53.975 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:54:53.976 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:54:53.976 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:53.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:53.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:54.016 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:54:54.017 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:54:54.078 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:54.112 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:54.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:54.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:54.114 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:54:54.114 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T10:54:54.114 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T10:54:54.114 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [] 2024-09-21T10:54:54.114 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:54:54.215 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:54.248 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:54.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:54.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:54.350 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:54.390 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:54.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:54.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:54.492 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:54.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:54.527 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:54.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:54.551 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 28 16 59125 59109 131.922 90.375 0.0108258 0.00757482 2024-09-21T10:54:54.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:54 smithi070 ceph-mon[107385]: pgmap v2770: 99 pgs: 99 active+clean; 3.2 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-09-21T10:54:54.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:54 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/594770290' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:54:54.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:54 smithi070 ceph-mon[112139]: pgmap v2770: 99 pgs: 99 active+clean; 3.2 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-09-21T10:54:54.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:54 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/594770290' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:54:54.628 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:54.660 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:54.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:54.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:54.761 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:54.797 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:54.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:54.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:54 smithi160 ceph-mon[101879]: pgmap v2770: 99 pgs: 99 active+clean; 3.2 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-09-21T10:54:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:54 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/594770290' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:54:54.899 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:54.936 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:54.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:54.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:55.037 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:55.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:55.076 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:55.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:55.114 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:54:55.177 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:55.211 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:55.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:55.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:55.312 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:55.350 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:55.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:55.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:55.452 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:55.492 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:55.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:55.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:55.551 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 29 16 60524 60508 130.388 87.4375 0.0143704 0.0076647 2024-09-21T10:54:55.594 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:55.622 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:55.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:55.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:55.683 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:54:55.683 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:54:55.725 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:55.757 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:55.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:55.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:55.859 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:55.894 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:55.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:55.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:55.996 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:56.018 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:56.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:56.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:56.119 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:56.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:56.163 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:56.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:56.264 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:56.299 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:56.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:56.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:56.401 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:56.435 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:56.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:56.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:56.536 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:56.553 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 30 16 61816 61800 128.733 80.75 0.0235425 0.00776366 2024-09-21T10:54:56.573 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:56.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:56.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:56.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:56 smithi070 ceph-mon[107385]: pgmap v2771: 99 pgs: 99 active+clean; 3.5 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-09-21T10:54:56.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:56 smithi070 ceph-mon[112139]: pgmap v2771: 99 pgs: 99 active+clean; 3.5 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-09-21T10:54:56.675 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:56.705 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:56.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:56.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:56.807 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:56 smithi160 ceph-mon[101879]: pgmap v2771: 99 pgs: 99 active+clean; 3.5 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-09-21T10:54:56.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:56.851 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:56.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:56.954 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:57.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:57.002 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:57.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:57.102 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:57.130 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:57.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:57.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:57.231 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:57.259 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:57.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:57.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:57.362 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:57.389 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:57.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:57.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:57.490 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:57.516 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:57.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:57.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:57.551 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 31 16 63208 63192 127.387 87 0.00865415 0.00784478 2024-09-21T10:54:57.618 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:54:57.654 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:57.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:57.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:54:57.720 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:54:57.756 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:57.796 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:57.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:57.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:57.897 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:57.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:57.934 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:57.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:58.035 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:58.076 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:58.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:58.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:58.178 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:58.192 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:54:58.202 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:58.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:58.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:58.304 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:58.340 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:58.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:58.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:58.441 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:58.471 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:58.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:58.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:58.552 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 32 16 64411 64395 125.756 75.1875 0.0115483 0.00794703 2024-09-21T10:54:58.573 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:58.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:58.600 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:58.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:58.701 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:54:58.742 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:58.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:58.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:54:58.829 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:54:58 smithi070 ceph-mon[107385]: pgmap v2772: 99 pgs: 99 active+clean; 3.6 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-21T10:54:58.830 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:54:58 smithi070 ceph-mon[112139]: pgmap v2772: 99 pgs: 99 active+clean; 3.6 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-21T10:54:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:54:58 smithi160 ceph-mon[101879]: pgmap v2772: 99 pgs: 99 active+clean; 3.6 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-21T10:54:58.844 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:58.881 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:58.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:58.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:58.983 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:54:59.010 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:59.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:59.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:54:59.112 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:54:59.138 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:59.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:59.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:54:59.239 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:59.263 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:59.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:59.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:59.365 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:54:59.402 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:59.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:59.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:54:59.503 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:59.527 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:59.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:59.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:54:59.552 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 33 16 65601 65585 124.198 74.375 0.0382832 0.00803953 2024-09-21T10:54:59.611 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:54:59.629 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:54:59.665 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:59.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:59.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:54:59.766 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:54:59.806 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:54:59.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:59.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:54:59.908 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:54:59.942 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:54:59.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:54:59.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:00.044 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:00.073 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:00.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:00.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:00.175 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:00.204 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:00.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:00.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:00.306 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:00.344 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:00.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:00.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:00.446 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:00.454 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T10:55:00.455 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:00 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a[107362]: 2024-09-21T10:55:00.196+0000 7fad4a465640 -1 mon.a@0(leader).osd e1490 definitely_dead 0 2024-09-21T10:55:00.487 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:00.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:00.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:00.552 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 34 16 66818 66802 122.783 76.0625 0.00319775 0.00812072 2024-09-21T10:55:00.589 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:00.611 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:00.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:00.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:00.629 INFO:teuthology.orchestra.run.smithi070.stderr:marked out osd.4. 2024-09-21T10:55:00.712 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:00.734 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:00.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:00.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:00.837 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:00 smithi160 ceph-mon[101879]: pgmap v2773: 99 pgs: 99 active+clean; 3.8 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-21T10:55:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:00 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3529955220' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:55:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:00 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:55:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:00 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/719253372' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T10:55:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:00 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T10:55:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:00 smithi160 ceph-mon[101879]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-21T10:55:00.858 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:00.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:00.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:00.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:00 smithi070 ceph-mon[107385]: pgmap v2773: 99 pgs: 99 active+clean; 3.8 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-21T10:55:00.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:00 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3529955220' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:55:00.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:00 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:55:00.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:00 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/719253372' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T10:55:00.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:00 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T10:55:00.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:00 smithi070 ceph-mon[107385]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-21T10:55:00.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:00 smithi070 ceph-mon[112139]: pgmap v2773: 99 pgs: 99 active+clean; 3.8 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-21T10:55:00.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:00 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3529955220' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:55:00.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:00 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:55:00.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:00 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/719253372' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T10:55:00.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:00 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T10:55:00.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:00 smithi070 ceph-mon[112139]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-21T10:55:00.961 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:00.982 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:00.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:00.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:01.084 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:01.107 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:01.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:01.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:01.208 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:01.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:01.231 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:01.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:01.332 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:01.358 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:01.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:01.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:01.460 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:01.494 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:01.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:01.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:01.552 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 35 16 66818 66802 119.274 0 - 0.00812072 2024-09-21T10:55:01.596 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:01.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:01.619 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:01.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:01.718 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:01 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:55:01.719 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:01 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:55:01.719 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:01 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-21T10:55:01.719 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:01 smithi070 ceph-mon[107385]: osdmap e1491: 8 total, 8 up, 4 in 2024-09-21T10:55:01.719 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:01 smithi070 ceph-mon[107385]: osdmap e1492: 8 total, 8 up, 4 in 2024-09-21T10:55:01.719 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:01 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:55:01.719 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:01 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:55:01.719 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:01 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-21T10:55:01.720 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:01 smithi070 ceph-mon[112139]: osdmap e1491: 8 total, 8 up, 4 in 2024-09-21T10:55:01.720 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:01 smithi070 ceph-mon[112139]: osdmap e1492: 8 total, 8 up, 4 in 2024-09-21T10:55:01.720 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:01.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:01.744 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:01.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:01 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:55:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:01 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:55:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:01 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-21T10:55:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:01 smithi160 ceph-mon[101879]: osdmap e1491: 8 total, 8 up, 4 in 2024-09-21T10:55:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:01 smithi160 ceph-mon[101879]: osdmap e1492: 8 total, 8 up, 4 in 2024-09-21T10:55:01.845 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:01.866 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:01.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:01.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:01.968 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:01.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:01.994 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:01.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:02.096 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:02.122 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:02.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:02.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:02.224 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:02.253 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:02.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:02.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:02.355 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:02.383 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:02.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:02.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:02.485 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:02.531 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:02.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:02.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:02.552 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 36 16 66818 66802 115.961 0 - 0.00812072 2024-09-21T10:55:02.634 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:02.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:02.692 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:02.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:02.793 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:02.832 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:02.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:02.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:02 smithi160 ceph-mon[101879]: pgmap v2775: 99 pgs: 99 active+clean; 3.9 GiB data, 12 GiB used, 346 GiB / 358 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-21T10:55:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:02 smithi160 ceph-mon[101879]: osdmap e1493: 8 total, 8 up, 4 in 2024-09-21T10:55:02.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:02 smithi070 ceph-mon[107385]: pgmap v2775: 99 pgs: 99 active+clean; 3.9 GiB data, 12 GiB used, 346 GiB / 358 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-21T10:55:02.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:02 smithi070 ceph-mon[107385]: osdmap e1493: 8 total, 8 up, 4 in 2024-09-21T10:55:02.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:02 smithi070 ceph-mon[112139]: pgmap v2775: 99 pgs: 99 active+clean; 3.9 GiB data, 12 GiB used, 346 GiB / 358 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-21T10:55:02.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:02 smithi070 ceph-mon[112139]: osdmap e1493: 8 total, 8 up, 4 in 2024-09-21T10:55:02.934 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:02.968 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:02.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:02.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:03.070 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:03.104 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:03.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:03.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:03.206 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:03.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:03.243 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:03.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:03.344 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:03.381 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:03.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:03.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:03.483 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:03.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:03.519 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:03.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:03.552 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 37 16 67549 67533 114.062 15.2292 0.00906198 0.00876077 2024-09-21T10:55:03.620 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:55:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:55:03] "GET /metrics HTTP/1.1" 200 36597 "" "Prometheus/2.51.0" 2024-09-21T10:55:03.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:03.656 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:03.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:03.757 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:03.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:03.793 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:03.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:03.894 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:03.932 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:03.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:03.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:04.034 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:04.068 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:04.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:04.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:04.170 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:04.209 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:04.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:04.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:04.310 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:04.344 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:04.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:04.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:04.401 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:55:04.424 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:04 smithi070 ceph-mon[112139]: pgmap v2778: 99 pgs: 10 remapped+peering, 89 active+clean; 4.0 GiB data, 13 GiB used, 345 GiB / 358 GiB avail; 25 KiB/s rd, 70 MiB/s wr, 1.13k op/s; 70 KiB/s, 1 keys/s, 1 objects/s recovering 2024-09-21T10:55:04.424 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:04 smithi070 ceph-mon[112139]: osdmap e1494: 8 total, 8 up, 4 in 2024-09-21T10:55:04.424 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:04 smithi070 ceph-mon[112139]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:55:04.424 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:04 smithi070 ceph-mon[107385]: pgmap v2778: 99 pgs: 10 remapped+peering, 89 active+clean; 4.0 GiB data, 13 GiB used, 345 GiB / 358 GiB avail; 25 KiB/s rd, 70 MiB/s wr, 1.13k op/s; 70 KiB/s, 1 keys/s, 1 objects/s recovering 2024-09-21T10:55:04.424 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:04 smithi070 ceph-mon[107385]: osdmap e1494: 8 total, 8 up, 4 in 2024-09-21T10:55:04.424 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:04 smithi070 ceph-mon[107385]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:55:04.446 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:04.466 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:04.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:04.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:04.552 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 38 16 68409 68393 112.475 53.75 0.0132257 0.00888523 2024-09-21T10:55:04.569 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:04.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:04.603 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:04.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:04.704 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:04.734 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:04.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:04.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:04.836 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:04.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:04 smithi160 ceph-mon[101879]: pgmap v2778: 99 pgs: 10 remapped+peering, 89 active+clean; 4.0 GiB data, 13 GiB used, 345 GiB / 358 GiB avail; 25 KiB/s rd, 70 MiB/s wr, 1.13k op/s; 70 KiB/s, 1 keys/s, 1 objects/s recovering 2024-09-21T10:55:04.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:04 smithi160 ceph-mon[101879]: osdmap e1494: 8 total, 8 up, 4 in 2024-09-21T10:55:04.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:04 smithi160 ceph-mon[101879]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:55:04.868 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:04.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:04.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:04.969 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:05.007 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:05.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:05.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:05.110 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:05.143 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:05.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:05.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:05.245 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:05.284 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:05.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:05.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:05.387 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:05.438 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:05.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:05.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:05.541 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:05.553 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 39 16 69324 69308 111.054 57.1875 0.0108825 0.00899949 2024-09-21T10:55:05.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:05.580 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:05.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:05.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:05 smithi070 ceph-mon[107385]: osdmap e1495: 8 total, 8 up, 4 in 2024-09-21T10:55:05.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:05 smithi070 ceph-mon[112139]: osdmap e1495: 8 total, 8 up, 4 in 2024-09-21T10:55:05.680 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:05.714 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:05.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:05.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:05.816 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:05 smithi160 ceph-mon[101879]: osdmap e1495: 8 total, 8 up, 4 in 2024-09-21T10:55:05.853 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:05.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:05.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:05.955 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:05.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:05.993 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:05.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:06.094 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:06.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:06.134 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:06.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:06.234 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:06.271 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:06.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:06.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:06.373 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:06.406 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:06.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:06.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:06.508 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:06.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:06.547 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:06.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:06.553 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:2024-09-21T10:55:06.552513+0000 min lat: 0.00129408 max lat: 2.31521 avg lat: 0.00911127 2024-09-21T10:55:06.553 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:55:06.553 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 40 16 70220 70204 109.678 56 0.00949064 0.00911127 2024-09-21T10:55:06.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:06 smithi070 ceph-mon[112139]: pgmap v2781: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 peering, 10 remapped+peering, 82 active+clean; 4.1 GiB data, 13 GiB used, 345 GiB / 358 GiB avail; 37 KiB/s rd, 61 MiB/s wr, 1.01k op/s; 16861/202485 objects degraded (8.327%); 4108/202485 objects misplaced (2.029%); 697 KiB/s, 2 keys/s, 10 objects/s recovering 2024-09-21T10:55:06.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:06 smithi070 ceph-mon[112139]: Health check failed: Degraded data redundancy: 16861/202485 objects degraded (8.327%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:06.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:55:06.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:55:06.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:06 smithi070 ceph-mon[107385]: pgmap v2781: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 peering, 10 remapped+peering, 82 active+clean; 4.1 GiB data, 13 GiB used, 345 GiB / 358 GiB avail; 37 KiB/s rd, 61 MiB/s wr, 1.01k op/s; 16861/202485 objects degraded (8.327%); 4108/202485 objects misplaced (2.029%); 697 KiB/s, 2 keys/s, 10 objects/s recovering 2024-09-21T10:55:06.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:06 smithi070 ceph-mon[107385]: Health check failed: Degraded data redundancy: 16861/202485 objects degraded (8.327%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:06.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:55:06.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:55:06.648 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:06.688 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:06.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:06.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:06.790 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:06.831 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:06.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:06.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:06 smithi160 ceph-mon[101879]: pgmap v2781: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 peering, 10 remapped+peering, 82 active+clean; 4.1 GiB data, 13 GiB used, 345 GiB / 358 GiB avail; 37 KiB/s rd, 61 MiB/s wr, 1.01k op/s; 16861/202485 objects degraded (8.327%); 4108/202485 objects misplaced (2.029%); 697 KiB/s, 2 keys/s, 10 objects/s recovering 2024-09-21T10:55:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:06 smithi160 ceph-mon[101879]: Health check failed: Degraded data redundancy: 16861/202485 objects degraded (8.327%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:55:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:55:06.933 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:06.934 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:55:06.934 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:55:06.946 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:55:06.947 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:55:06.968 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:06.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:06.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:06.977 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:55:06.977 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:55:07.016 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:55:07.016 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:55:07.042 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:55:07.043 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:55:07.070 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:07.074 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:55:07.074 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:55:07.106 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:55:07.107 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:55:07.115 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:07.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:07.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:07.146 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:55:07.146 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:55:07.180 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:55:07.198 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:55:07.198 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:55:07.217 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:07.248 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:07.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:07.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:07.350 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:07.388 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:07.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:07.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:07.490 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:07.529 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:07.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:07.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:07.553 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 41 16 71098 71082 108.341 54.875 0.0209195 0.0092243 2024-09-21T10:55:07.631 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:07.663 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:07.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:07.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:07.765 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:07.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:07.805 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:07.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:07 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2029541222' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:55:07.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:07 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2029541222' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:55:07.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:07 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2029541222' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:55:07.906 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:07.944 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:07.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:07.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:08.046 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:08.075 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:08.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:08.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:08.176 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:08.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:08.213 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:08.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:08.314 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:08.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:08.349 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:08.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:08.450 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:08.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:08.482 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:08.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:08.553 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 42 16 72013 71997 107.123 57.1875 0.0208893 0.00932987 2024-09-21T10:55:08.584 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:08.617 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:08.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:08.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:08.719 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:08.758 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:08.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:08.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:08 smithi160 ceph-mon[101879]: pgmap v2782: 99 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 2 peering, 5 remapped+peering, 85 active+clean; 4.1 GiB data, 13 GiB used, 345 GiB / 358 GiB avail; 27 KiB/s rd, 50 MiB/s wr, 814 op/s; 25256/203868 objects degraded (12.388%); 4108/203868 objects misplaced (2.015%); 505 KiB/s, 1 keys/s, 7 objects/s recovering 2024-09-21T10:55:08.860 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:08.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:08 smithi070 ceph-mon[107385]: pgmap v2782: 99 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 2 peering, 5 remapped+peering, 85 active+clean; 4.1 GiB data, 13 GiB used, 345 GiB / 358 GiB avail; 27 KiB/s rd, 50 MiB/s wr, 814 op/s; 25256/203868 objects degraded (12.388%); 4108/203868 objects misplaced (2.015%); 505 KiB/s, 1 keys/s, 7 objects/s recovering 2024-09-21T10:55:08.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:08 smithi070 ceph-mon[112139]: pgmap v2782: 99 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 2 peering, 5 remapped+peering, 85 active+clean; 4.1 GiB data, 13 GiB used, 345 GiB / 358 GiB avail; 27 KiB/s rd, 50 MiB/s wr, 814 op/s; 25256/203868 objects degraded (12.388%); 4108/203868 objects misplaced (2.015%); 505 KiB/s, 1 keys/s, 7 objects/s recovering 2024-09-21T10:55:08.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:08.894 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:08.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:08.995 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:09.028 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:09.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:09.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:09.130 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:09.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:09.173 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:09.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:09.275 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:09.309 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:09.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:09.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:09.412 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:09.451 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:09.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:09.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:09.518 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:09 smithi070 ceph-mon[107385]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:55:09.518 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:09 smithi070 ceph-mon[112139]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:55:09.553 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:09.554 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 43 16 72910 72894 105.936 56.0625 0.0125461 0.00943411 2024-09-21T10:55:09.585 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:09.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:09.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:09.687 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:09.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:09.726 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:09.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:09.827 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:09 smithi160 ceph-mon[101879]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:55:09.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:09.860 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:09.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:09.961 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:09.997 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:09.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:09.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:10.099 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:10.126 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:10.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:10.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:10.228 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:10.264 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:55:10.272 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:10.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:10.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:10.375 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:10.406 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:10.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:10.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:10.508 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:10.545 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:10.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:10.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:10.553 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 44 16 73803 73787 104.796 55.8125 0.015972 0.00953587 2024-09-21T10:55:10.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:10 smithi070 ceph-mon[112139]: pgmap v2783: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 peering, 88 active+clean; 4.3 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 45 MiB/s wr, 714 op/s; 33576/210813 objects degraded (15.927%); 7930/210813 objects misplaced (3.762%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:55:10.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:10 smithi070 ceph-mon[107385]: pgmap v2783: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 peering, 88 active+clean; 4.3 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 45 MiB/s wr, 714 op/s; 33576/210813 objects degraded (15.927%); 7930/210813 objects misplaced (3.762%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:55:10.647 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:10.679 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:10.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:10.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:10.781 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:10.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:10.822 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:10.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:10.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:10 smithi160 ceph-mon[101879]: pgmap v2783: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 peering, 88 active+clean; 4.3 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 45 MiB/s wr, 714 op/s; 33576/210813 objects degraded (15.927%); 7930/210813 objects misplaced (3.762%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:55:10.856 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:55:10.923 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:10.955 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:10.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:10.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:11.057 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:11.089 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:11.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:11.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:11.191 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:11.231 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:11.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:11.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:11.250 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:55:11.334 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:11.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:11.365 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:11.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:11.466 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:11.497 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7] out_osds: [2, 5, 0, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:55:11.497 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:55:11.497 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_3 2024-09-21T10:55:11.498 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:55:11.503 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:11.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:11.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:11.554 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 45 16 74692 74676 103.701 55.5625 0.0160814 0.00963696 2024-09-21T10:55:11.605 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:11.635 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:11.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:11.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:11.737 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:11.773 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:11.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:11.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:11 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 33576/215061 objects degraded (15.612%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:11.875 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:11.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:11 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 33576/215061 objects degraded (15.612%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:11.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:11 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 33576/215061 objects degraded (15.612%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:11.915 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:11.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:11.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:12.017 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:12.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:12.054 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:12.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:12.155 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:12.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:12.195 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:12.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:12.296 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:12.337 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:12.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:12.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:12.438 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:12.475 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:12.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:12.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:12.502 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T10:55:12.554 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 46 16 75557 75541 102.622 54.0625 0.0119443 0.00973845 2024-09-21T10:55:12.577 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:12.613 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:12.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:12.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:12 smithi070 ceph-mon[107385]: pgmap v2784: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 4.4 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 47 MiB/s wr, 759 op/s; 33576/215061 objects degraded (15.612%); 7930/215061 objects misplaced (3.687%); 2.4 MiB/s, 37 objects/s recovering 2024-09-21T10:55:12.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:12 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/312336525' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:55:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:12 smithi070 ceph-mon[112139]: pgmap v2784: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 4.4 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 47 MiB/s wr, 759 op/s; 33576/215061 objects degraded (15.612%); 7930/215061 objects misplaced (3.687%); 2.4 MiB/s, 37 objects/s recovering 2024-09-21T10:55:12.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:12 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/312336525' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:55:12.716 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:12.752 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:12.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:12.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:12 smithi160 ceph-mon[101879]: pgmap v2784: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 4.4 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 47 MiB/s wr, 759 op/s; 33576/215061 objects degraded (15.612%); 7930/215061 objects misplaced (3.687%); 2.4 MiB/s, 37 objects/s recovering 2024-09-21T10:55:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:12 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/312336525' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:55:12.855 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:12.901 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:55:12.909 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:12.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:12.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:13.011 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:13.043 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:13.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:13.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:13.145 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:13.176 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:13.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:13.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:13.278 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:13.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:13.312 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:13.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:13.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:55:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:55:13] "GET /metrics HTTP/1.1" 200 36634 "" "Prometheus/2.51.0" 2024-09-21T10:55:13.412 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:13.447 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:13.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:13.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:13.550 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:13.556 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 47 16 76440 76424 101.609 55.1875 0.00434826 0.00983537 2024-09-21T10:55:13.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:13.588 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:13.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:13.627 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:55:13.689 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:13.731 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:13.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:13.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:13.738 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:13 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:55:13.738 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:13 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/312336525' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:55:13.738 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:13 smithi070 ceph-mon[112139]: osdmap e1496: 8 total, 8 up, 4 in 2024-09-21T10:55:13.742 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:13 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:55:13.742 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:13 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/312336525' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:55:13.742 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:13 smithi070 ceph-mon[107385]: osdmap e1496: 8 total, 8 up, 4 in 2024-09-21T10:55:13.833 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:13 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:55:13.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:13 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/312336525' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:55:13.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:13 smithi160 ceph-mon[101879]: osdmap e1496: 8 total, 8 up, 4 in 2024-09-21T10:55:13.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:13.872 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:13.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:13.973 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:14.005 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:14.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:14.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:14.107 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:14.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:14.146 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:14.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:14.247 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:14.278 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:14.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:14.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:14.380 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:14.422 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:14.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:14.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:14.524 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:14.558 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 48 16 77356 77340 100.685 57.25 0.00540844 0.00992519 2024-09-21T10:55:14.561 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:14.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:14.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:14.663 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:14.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:14.702 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:14.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:14.782 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:14 smithi070 ceph-mon[107385]: pgmap v2786: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 4.5 GiB data, 14 GiB used, 344 GiB / 358 GiB avail; 45 MiB/s wr, 715 op/s; 33576/220977 objects degraded (15.194%); 7789/220977 objects misplaced (3.525%); 3.0 MiB/s, 47 objects/s recovering 2024-09-21T10:55:14.782 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:14 smithi070 ceph-mon[112139]: pgmap v2786: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 4.5 GiB data, 14 GiB used, 344 GiB / 358 GiB avail; 45 MiB/s wr, 715 op/s; 33576/220977 objects degraded (15.194%); 7789/220977 objects misplaced (3.525%); 3.0 MiB/s, 47 objects/s recovering 2024-09-21T10:55:14.804 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:14 smithi160 ceph-mon[101879]: pgmap v2786: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 4.5 GiB data, 14 GiB used, 344 GiB / 358 GiB avail; 45 MiB/s wr, 715 op/s; 33576/220977 objects degraded (15.194%); 7789/220977 objects misplaced (3.525%); 3.0 MiB/s, 47 objects/s recovering 2024-09-21T10:55:14.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:14.844 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:14.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:14.945 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:14.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:14.983 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:14.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:15.084 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:15.119 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:15.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:15.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:15.221 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:15.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:15.257 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:15.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:15.358 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:15.394 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:15.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:15.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:15.496 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:15.538 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:15.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:15.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:15.556 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 49 16 78273 78257 99.8001 57.3125 0.00876023 0.0100141 2024-09-21T10:55:15.641 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:15.670 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:15.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:15.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:15.772 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:15.805 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:15.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:15.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:15.907 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:15.944 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:15.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:15.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:16.046 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:16.092 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:16.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:16.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:16.139 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:55:16.139 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:55:16.177 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:55:16.194 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:16.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:16.285 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:16.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:16.386 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:16.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:16.441 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:16.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:16.542 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:16.557 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 50 16 79163 79147 98.9149 55.625 0.0174252 0.0101034 2024-09-21T10:55:16.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:16.585 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:16.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:16.686 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:16.719 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:16.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:16.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:16.821 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:16.822 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:16 smithi070 ceph-mon[107385]: pgmap v2787: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 4.6 GiB data, 14 GiB used, 343 GiB / 358 GiB avail; 54 MiB/s wr, 863 op/s; 33576/228390 objects degraded (14.701%); 7639/228390 objects misplaced (3.345%); 3.5 MiB/s, 55 objects/s recovering 2024-09-21T10:55:16.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:16 smithi070 ceph-mon[112139]: pgmap v2787: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 4.6 GiB data, 14 GiB used, 343 GiB / 358 GiB avail; 54 MiB/s wr, 863 op/s; 33576/228390 objects degraded (14.701%); 7639/228390 objects misplaced (3.345%); 3.5 MiB/s, 55 objects/s recovering 2024-09-21T10:55:16.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:16 smithi160 ceph-mon[101879]: pgmap v2787: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 4.6 GiB data, 14 GiB used, 343 GiB / 358 GiB avail; 54 MiB/s wr, 863 op/s; 33576/228390 objects degraded (14.701%); 7639/228390 objects misplaced (3.345%); 3.5 MiB/s, 55 objects/s recovering 2024-09-21T10:55:16.866 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:55:16.885 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:16.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:16.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:16.986 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:17.026 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:17.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:17.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:17.129 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:17.167 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:17.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:17.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:17.269 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:17.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:17.304 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:17.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:17.405 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:17.442 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:17.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:17.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:17.543 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:17.557 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 51 16 80080 80064 98.0992 57.3125 0.00800395 0.010188 2024-09-21T10:55:17.583 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:17.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:17.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:17.685 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:17.712 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:17.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:17.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:17.814 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:17.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:17 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 33576/230070 objects degraded (14.594%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:17.850 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:17.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:17.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:17.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:17 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 33576/230070 objects degraded (14.594%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:17.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:17 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 33576/230070 objects degraded (14.594%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:17.952 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:17.980 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:17.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:17.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:18.082 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:18.116 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:18.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:18.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:18.218 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:18.249 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:18.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:18.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:18.350 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:18.391 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:18.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:18.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:18.492 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:18.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:18.536 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:18.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:18.557 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 52 16 80953 80937 97.2619 54.5625 0.0288984 0.0102761 2024-09-21T10:55:18.637 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:18.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:18.671 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:18.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:18.772 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:18.811 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:18.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:18.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:18 smithi160 ceph-mon[101879]: pgmap v2788: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 4.7 GiB data, 15 GiB used, 343 GiB / 358 GiB avail; 55 MiB/s wr, 873 op/s; 33576/230070 objects degraded (14.594%); 7639/230070 objects misplaced (3.320%); 3.5 MiB/s, 55 objects/s recovering 2024-09-21T10:55:18.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:18 smithi070 ceph-mon[107385]: pgmap v2788: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 4.7 GiB data, 15 GiB used, 343 GiB / 358 GiB avail; 55 MiB/s wr, 873 op/s; 33576/230070 objects degraded (14.594%); 7639/230070 objects misplaced (3.320%); 3.5 MiB/s, 55 objects/s recovering 2024-09-21T10:55:18.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:18 smithi070 ceph-mon[112139]: pgmap v2788: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 4.7 GiB data, 15 GiB used, 343 GiB / 358 GiB avail; 55 MiB/s wr, 873 op/s; 33576/230070 objects degraded (14.594%); 7639/230070 objects misplaced (3.320%); 3.5 MiB/s, 55 objects/s recovering 2024-09-21T10:55:18.913 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:18.940 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:18.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:18.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:19.042 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:19.081 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:19.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:19.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:19.145 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:55:19.146 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T10:55:19.183 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:19.216 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:19.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:19.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:19.318 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:19.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:19.364 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:19.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:19.465 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:19.499 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:19.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:19.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:19.557 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 53 16 81745 81729 96.3608 49.5 0.0192352 0.0103721 2024-09-21T10:55:19.602 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:19.643 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:19.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:19.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:19.745 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:19.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:19.780 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:19.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:19 smithi160 ceph-mon[101879]: from='client.55399 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:55:19.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:19 smithi070 ceph-mon[107385]: from='client.55399 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:55:19.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:19 smithi070 ceph-mon[112139]: from='client.55399 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:55:19.881 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:19.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:19.919 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:19.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:20.020 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:20.053 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:20.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:20.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:20.155 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:20.191 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:20.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:20.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:20.293 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:20.327 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:20.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:20.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:20.429 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:20.469 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:20.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:20.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:20.557 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 54 16 82676 82660 95.6538 58.1875 0.00797544 0.0104486 2024-09-21T10:55:20.571 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:20.617 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:20.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:20.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:20.718 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:20.759 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:20.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:20.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:20 smithi160 ceph-mon[101879]: pgmap v2789: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 4.8 GiB data, 15 GiB used, 343 GiB / 358 GiB avail; 56 MiB/s wr, 894 op/s; 33576/237645 objects degraded (14.129%); 7341/237645 objects misplaced (3.089%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T10:55:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:55:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:55:20.861 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:20.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:20 smithi070 ceph-mon[107385]: pgmap v2789: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 4.8 GiB data, 15 GiB used, 343 GiB / 358 GiB avail; 56 MiB/s wr, 894 op/s; 33576/237645 objects degraded (14.129%); 7341/237645 objects misplaced (3.089%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T10:55:20.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:55:20.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:55:20.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:20 smithi070 ceph-mon[112139]: pgmap v2789: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 4.8 GiB data, 15 GiB used, 343 GiB / 358 GiB avail; 56 MiB/s wr, 894 op/s; 33576/237645 objects degraded (14.129%); 7341/237645 objects misplaced (3.089%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T10:55:20.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:55:20.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:55:20.891 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:20.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:20.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:20.993 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:21.029 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:21.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:21.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:21.131 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:21.165 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:21.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:21.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:21.267 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:21.292 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:21.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:21.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:21.395 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:21.431 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:21.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:21.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:21.532 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:21.557 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 55 16 83622 83606 94.9896 59.125 0.0335033 0.010521 2024-09-21T10:55:21.567 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:21.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:21.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:21.669 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:21.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:21.708 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:21.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:21.809 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:21.844 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:21.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:21.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:21.946 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:21.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:21.987 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:21.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:22.087 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:22.126 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:22.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:22.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:22.228 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:22.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:22.270 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:22.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:22.371 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:22.401 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:22.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:22.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:22.502 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:22.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:22.544 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:22.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:22.557 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 56 16 84442 84426 94.2085 51.25 0.0314282 0.0106089 2024-09-21T10:55:22.645 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:22.670 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:22.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:22.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:22.772 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:22.809 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:22.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:22.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:22 smithi160 ceph-mon[101879]: pgmap v2790: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 4.9 GiB data, 15 GiB used, 342 GiB / 358 GiB avail; 56 MiB/s wr, 894 op/s; 33576/241896 objects degraded (13.880%); 7341/241896 objects misplaced (3.035%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T10:55:22.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:22 smithi070 ceph-mon[107385]: pgmap v2790: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 4.9 GiB data, 15 GiB used, 342 GiB / 358 GiB avail; 56 MiB/s wr, 894 op/s; 33576/241896 objects degraded (13.880%); 7341/241896 objects misplaced (3.035%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T10:55:22.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:22 smithi070 ceph-mon[112139]: pgmap v2790: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 4.9 GiB data, 15 GiB used, 342 GiB / 358 GiB avail; 56 MiB/s wr, 894 op/s; 33576/241896 objects degraded (13.880%); 7341/241896 objects misplaced (3.035%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T10:55:22.912 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:22.943 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:22.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:22.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:23.045 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:23.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:23.083 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:23.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:23.184 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:23.219 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:23.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:23.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:23.321 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:23.365 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:23.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:23.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:23.466 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:23.504 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:23.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:23.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:23.558 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 57 16 85296 85280 93.4921 53.375 0.00643674 0.0106904 2024-09-21T10:55:23.590 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:55:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:55:23] "GET /metrics HTTP/1.1" 200 36626 "" "Prometheus/2.51.0" 2024-09-21T10:55:23.606 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:23.639 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:23.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:23.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:23.741 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:23.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:23.772 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:23.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:23.825 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":2788,"stamp":"2024-09-21T10:55:16.967715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5024591363,"num_objects":76690,"num_object_clones":0,"num_object_copies":230070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33576,"num_objects_misplaced":7639,"num_objects_unfound":0,"num_objects_dirty":76690,"num_whiteouts":0,"num_read":173,"num_read_kb":375,"num_write":76828,"num_write_kb":4907692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":665,"num_bytes_recovered":42142632,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67344,"ondisk_log_size":67344,"up":297,"acting":287,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":297,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":374980608,"kb_used":15304428,"kb_used_data":12155912,"kb_used_omap":139,"kb_used_meta":3148340,"kb_avail":359676180,"statfs":{"total":383980142592,"available":368308408320,"internally_reserved":0,"allocated":12447653888,"data_stored":12426996139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":143318,"internal_metadata":3223900202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":572391424,"num_objects":8734,"num_object_clones":0,"num_object_copies":26202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8320,"num_objects_misplaced":3531,"num_objects_unfound":0,"num_objects_dirty":8734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8734,"num_write_kb":558976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":559,"num_bytes_recovered":36634624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003031"},"pg_stats":[{"pgid":"56.f","version":"1496'4741","reported_seq":5456,"reported_epoch":1496,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-21T10:55:13.822842+0000","last_change":"2024-09-21T10:55:02.692034+0000","last_active":"2024-09-21T10:55:13.822842+0000","last_peered":"2024-09-21T10:55:13.822842+0000","last_clean":"2024-09-21T10:55:00.435791+0000","last_became_active":"2024-09-21T10:55:02.645569+0000","last_became_peered":"2024-09-21T10:55:02.645569+0000","last_unstale":"2024-09-21T10:55:13.822842+0000","last_undegraded":"2024-09-21T10:55:13.822842+0000","last_fullsized":"2024-09-21T10:55:02.452183+0000","mapping_epoch":1492,"log_start":"1489'600","ondisk_log_start":"1489'600","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":4141,"log_dups_size":0,"ondisk_log_size":4141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310706176,"num_objects":4741,"num_object_clones":0,"num_object_copies":14223,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3803,"num_objects_unfound":0,"num_objects_dirty":4741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4741,"num_write_kb":303424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":22151168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7],"avail_no_missing":["1","4","7"],"object_location_counts":[{"shards":"1,4,7","objects":3803},{"shards":"1,7","objects":938}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.e","version":"1495'4633","reported_seq":4653,"reported_epoch":1495,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.059720+0000","last_change":"2024-09-21T10:54:11.452866+0000","last_active":"2024-09-21T10:55:12.059720+0000","last_peered":"2024-09-21T10:55:12.059720+0000","last_clean":"2024-09-21T10:55:12.059720+0000","last_became_active":"2024-09-21T10:54:11.452553+0000","last_became_peered":"2024-09-21T10:54:11.452553+0000","last_unstale":"2024-09-21T10:55:12.059720+0000","last_undegraded":"2024-09-21T10:55:12.059720+0000","last_fullsized":"2024-09-21T10:55:12.059720+0000","mapping_epoch":1484,"log_start":"1489'200","ondisk_log_start":"1489'200","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":4433,"log_dups_size":0,"ondisk_log_size":4433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:02:31.756582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303628288,"num_objects":4633,"num_object_clones":0,"num_object_copies":13899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4633,"num_write_kb":296512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"56.d","version":"1496'4901","reported_seq":4936,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:14.208055+0000","last_change":"2024-09-21T10:54:11.455827+0000","last_active":"2024-09-21T10:55:14.208055+0000","last_peered":"2024-09-21T10:55:14.208055+0000","last_clean":"2024-09-21T10:55:14.208055+0000","last_became_active":"2024-09-21T10:54:11.455499+0000","last_became_peered":"2024-09-21T10:54:11.455499+0000","last_unstale":"2024-09-21T10:55:14.208055+0000","last_undegraded":"2024-09-21T10:55:14.208055+0000","last_fullsized":"2024-09-21T10:55:14.208055+0000","mapping_epoch":1484,"log_start":"1489'1168","ondisk_log_start":"1489'1168","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":3733,"log_dups_size":0,"ondisk_log_size":3733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:58:18.441343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321191936,"num_objects":4901,"num_object_clones":0,"num_object_copies":14703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4901,"num_write_kb":313664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.c","version":"1496'4838","reported_seq":4875,"reported_epoch":1496,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:55:13.797181+0000","last_change":"2024-09-21T10:55:02.737453+0000","last_active":"2024-09-21T10:55:13.797181+0000","last_peered":"2024-09-21T10:55:13.797181+0000","last_clean":"2024-09-21T10:55:00.435595+0000","last_became_active":"2024-09-21T10:55:02.726777+0000","last_became_peered":"2024-09-21T10:55:02.726777+0000","last_unstale":"2024-09-21T10:55:13.797181+0000","last_undegraded":"2024-09-21T10:55:02.505088+0000","last_fullsized":"2024-09-21T10:55:02.458145+0000","mapping_epoch":1492,"log_start":"1489'637","ondisk_log_start":"1489'637","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":4201,"log_dups_size":0,"ondisk_log_size":4201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317063168,"num_objects":4838,"num_object_clones":0,"num_object_copies":14514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4838,"num_write_kb":309632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6],"avail_no_missing":["1","4","6"],"object_location_counts":[{"shards":"1,4,6","objects":4246},{"shards":"1,6","objects":592}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.b","version":"1496'4986","reported_seq":5014,"reported_epoch":1496,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:55:16.904883+0000","last_change":"2024-09-21T10:55:02.739752+0000","last_active":"2024-09-21T10:55:16.904883+0000","last_peered":"2024-09-21T10:55:16.904883+0000","last_clean":"2024-09-21T10:55:00.412261+0000","last_became_active":"2024-09-21T10:55:02.725641+0000","last_became_peered":"2024-09-21T10:55:02.725641+0000","last_unstale":"2024-09-21T10:55:16.904883+0000","last_undegraded":"2024-09-21T10:55:02.482139+0000","last_fullsized":"2024-09-21T10:55:02.452705+0000","mapping_epoch":1492,"log_start":"1489'800","ondisk_log_start":"1489'800","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":4186,"log_dups_size":0,"ondisk_log_size":4186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:13.502440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326762496,"num_objects":4986,"num_object_clones":0,"num_object_copies":14958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4986,"num_write_kb":319104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7],"avail_no_missing":["6","4","7"],"object_location_counts":[{"shards":"4,6,7","objects":4140},{"shards":"6,7","objects":846}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"56.a","version":"1496'4672","reported_seq":4709,"reported_epoch":1496,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:55:13.823584+0000","last_change":"2024-09-21T10:55:02.737448+0000","last_active":"2024-09-21T10:55:13.823584+0000","last_peered":"2024-09-21T10:55:13.823584+0000","last_clean":"2024-09-21T10:55:00.441157+0000","last_became_active":"2024-09-21T10:55:02.726286+0000","last_became_peered":"2024-09-21T10:55:02.726286+0000","last_unstale":"2024-09-21T10:55:13.823584+0000","last_undegraded":"2024-09-21T10:55:02.500096+0000","last_fullsized":"2024-09-21T10:55:02.452984+0000","mapping_epoch":1492,"log_start":"1489'500","ondisk_log_start":"1489'500","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":4172,"log_dups_size":0,"ondisk_log_size":4172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":4672,"num_object_clones":0,"num_object_copies":14016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4672,"num_write_kb":299008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6],"avail_no_missing":["1","4","6"],"object_location_counts":[{"shards":"1,4,6","objects":4084},{"shards":"1,6","objects":588}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.9","version":"1495'4646","reported_seq":4673,"reported_epoch":1495,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:55:12.020178+0000","last_change":"2024-09-21T10:55:02.734335+0000","last_active":"2024-09-21T10:55:12.020178+0000","last_peered":"2024-09-21T10:55:12.020178+0000","last_clean":"2024-09-21T10:55:00.439008+0000","last_became_active":"2024-09-21T10:55:02.723335+0000","last_became_peered":"2024-09-21T10:55:02.723335+0000","last_unstale":"2024-09-21T10:55:12.020178+0000","last_undegraded":"2024-09-21T10:55:02.487101+0000","last_fullsized":"2024-09-21T10:55:02.450136+0000","mapping_epoch":1492,"log_start":"1489'200","ondisk_log_start":"1489'200","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":4446,"log_dups_size":0,"ondisk_log_size":4446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:19:39.327001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304480256,"num_objects":4646,"num_object_clones":0,"num_object_copies":13938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4646,"num_write_kb":297344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6],"avail_no_missing":["7","4","6"],"object_location_counts":[{"shards":"4,6,7","objects":4142},{"shards":"6,7","objects":504}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"56.8","version":"1496'5020","reported_seq":5049,"reported_epoch":1496,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:55:16.837097+0000","last_change":"2024-09-21T10:55:02.664313+0000","last_active":"2024-09-21T10:55:16.837097+0000","last_peered":"2024-09-21T10:55:16.837097+0000","last_clean":"2024-09-21T10:55:00.450776+0000","last_became_active":"2024-09-21T10:55:02.656119+0000","last_became_peered":"2024-09-21T10:55:02.656119+0000","last_unstale":"2024-09-21T10:55:16.837097+0000","last_undegraded":"2024-09-21T10:55:02.486387+0000","last_fullsized":"2024-09-21T10:55:02.452708+0000","mapping_epoch":1492,"log_start":"1489'846","ondisk_log_start":"1489'846","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":4174,"log_dups_size":0,"ondisk_log_size":4174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:18:17.449312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328990720,"num_objects":5020,"num_object_clones":0,"num_object_copies":15060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5020,"num_write_kb":321280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3],"avail_no_missing":["6","3","4"],"object_location_counts":[{"shards":"3,4,6","objects":4255},{"shards":"3,6","objects":765}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"56.7","version":"1495'4681","reported_seq":4708,"reported_epoch":1495,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:55:12.064201+0000","last_change":"2024-09-21T10:55:02.718920+0000","last_active":"2024-09-21T10:55:12.064201+0000","last_peered":"2024-09-21T10:55:12.064201+0000","last_clean":"2024-09-21T10:55:00.433343+0000","last_became_active":"2024-09-21T10:55:02.708987+0000","last_became_peered":"2024-09-21T10:55:02.708987+0000","last_unstale":"2024-09-21T10:55:12.064201+0000","last_undegraded":"2024-09-21T10:55:02.495130+0000","last_fullsized":"2024-09-21T10:55:02.452035+0000","mapping_epoch":1492,"log_start":"1489'300","ondisk_log_start":"1489'300","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":4381,"log_dups_size":0,"ondisk_log_size":4381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:47:26.898013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306774016,"num_objects":4681,"num_object_clones":0,"num_object_copies":14043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4681,"num_write_kb":299584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6],"avail_no_missing":["7","4","6"],"object_location_counts":[{"shards":"4,6,7","objects":4178},{"shards":"6,7","objects":503}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"56.6","version":"1495'4632","reported_seq":5169,"reported_epoch":1495,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-21T10:55:12.090386+0000","last_change":"2024-09-21T10:55:02.668215+0000","last_active":"2024-09-21T10:55:12.090386+0000","last_peered":"2024-09-21T10:55:12.090386+0000","last_clean":"2024-09-21T10:55:00.451801+0000","last_became_active":"2024-09-21T10:55:02.631001+0000","last_became_peered":"2024-09-21T10:55:02.631001+0000","last_unstale":"2024-09-21T10:55:12.090386+0000","last_undegraded":"2024-09-21T10:55:12.090386+0000","last_fullsized":"2024-09-21T10:55:02.451689+0000","mapping_epoch":1492,"log_start":"1489'200","ondisk_log_start":"1489'200","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":4432,"log_dups_size":0,"ondisk_log_size":4432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:37:42.040624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303562752,"num_objects":4632,"num_object_clones":0,"num_object_copies":13896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3836,"num_objects_unfound":0,"num_objects_dirty":4632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4632,"num_write_kb":296448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":254,"num_bytes_recovered":16646144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3],"avail_no_missing":["7","3","4"],"object_location_counts":[{"shards":"3,4,7","objects":3836},{"shards":"3,7","objects":796}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"56.5","version":"1496'4868","reported_seq":4905,"reported_epoch":1496,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:55:14.231457+0000","last_change":"2024-09-21T10:55:02.666345+0000","last_active":"2024-09-21T10:55:14.231457+0000","last_peered":"2024-09-21T10:55:14.231457+0000","last_clean":"2024-09-21T10:55:00.429535+0000","last_became_active":"2024-09-21T10:55:02.650393+0000","last_became_peered":"2024-09-21T10:55:02.650393+0000","last_unstale":"2024-09-21T10:55:14.231457+0000","last_undegraded":"2024-09-21T10:55:02.480735+0000","last_fullsized":"2024-09-21T10:55:02.452671+0000","mapping_epoch":1492,"log_start":"1489'1171","ondisk_log_start":"1489'1171","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":3697,"log_dups_size":0,"ondisk_log_size":3697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319029248,"num_objects":4868,"num_object_clones":0,"num_object_copies":14604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4868,"num_write_kb":311552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"3,4,7","objects":4229},{"shards":"3,7","objects":639}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.4","version":"1496'4973","reported_seq":5002,"reported_epoch":1496,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:55:13.778339+0000","last_change":"2024-09-21T10:55:02.673694+0000","last_active":"2024-09-21T10:55:13.778339+0000","last_peered":"2024-09-21T10:55:13.778339+0000","last_clean":"2024-09-21T10:55:00.434021+0000","last_became_active":"2024-09-21T10:55:02.654994+0000","last_became_peered":"2024-09-21T10:55:02.654994+0000","last_unstale":"2024-09-21T10:55:13.778339+0000","last_undegraded":"2024-09-21T10:55:02.497232+0000","last_fullsized":"2024-09-21T10:55:02.455952+0000","mapping_epoch":1492,"log_start":"1489'793","ondisk_log_start":"1489'793","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":4180,"log_dups_size":0,"ondisk_log_size":4180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325910528,"num_objects":4973,"num_object_clones":0,"num_object_copies":14919,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4973,"num_write_kb":318272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6],"avail_no_missing":["1","4","6"],"object_location_counts":[{"shards":"1,4,6","objects":4302},{"shards":"1,6","objects":671}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.3","version":"1495'4671","reported_seq":4691,"reported_epoch":1495,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.064321+0000","last_change":"2024-09-21T10:54:11.452259+0000","last_active":"2024-09-21T10:55:12.064321+0000","last_peered":"2024-09-21T10:55:12.064321+0000","last_clean":"2024-09-21T10:55:12.064321+0000","last_became_active":"2024-09-21T10:54:11.451997+0000","last_became_peered":"2024-09-21T10:54:11.451997+0000","last_unstale":"2024-09-21T10:55:12.064321+0000","last_undegraded":"2024-09-21T10:55:12.064321+0000","last_fullsized":"2024-09-21T10:55:12.064321+0000","mapping_epoch":1484,"log_start":"1489'300","ondisk_log_start":"1489'300","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":4371,"log_dups_size":0,"ondisk_log_size":4371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:52:22.227114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306118656,"num_objects":4671,"num_object_clones":0,"num_object_copies":14013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4671,"num_write_kb":298944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"56.2","version":"1496'4827","reported_seq":4861,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:14.210297+0000","last_change":"2024-09-21T10:54:11.451633+0000","last_active":"2024-09-21T10:55:14.210297+0000","last_peered":"2024-09-21T10:55:14.210297+0000","last_clean":"2024-09-21T10:55:14.210297+0000","last_became_active":"2024-09-21T10:54:11.451333+0000","last_became_peered":"2024-09-21T10:54:11.451333+0000","last_unstale":"2024-09-21T10:55:14.210297+0000","last_undegraded":"2024-09-21T10:55:14.210297+0000","last_fullsized":"2024-09-21T10:55:14.210297+0000","mapping_epoch":1484,"log_start":"1489'1127","ondisk_log_start":"1489'1127","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":3700,"log_dups_size":0,"ondisk_log_size":3700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:17:20.683870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316342272,"num_objects":4827,"num_object_clones":0,"num_object_copies":14481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4827,"num_write_kb":308928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.1","version":"1496'4755","reported_seq":4790,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:14.206066+0000","last_change":"2024-09-21T10:54:11.451499+0000","last_active":"2024-09-21T10:55:14.206066+0000","last_peered":"2024-09-21T10:55:14.206066+0000","last_clean":"2024-09-21T10:55:14.206066+0000","last_became_active":"2024-09-21T10:54:11.451204+0000","last_became_peered":"2024-09-21T10:54:11.451204+0000","last_unstale":"2024-09-21T10:55:14.206066+0000","last_undegraded":"2024-09-21T10:55:14.206066+0000","last_fullsized":"2024-09-21T10:55:14.206066+0000","mapping_epoch":1484,"log_start":"1489'1067","ondisk_log_start":"1489'1067","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":3688,"log_dups_size":0,"ondisk_log_size":3688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:43.059873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311623680,"num_objects":4755,"num_object_clones":0,"num_object_copies":14265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4755,"num_write_kb":304320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.0","version":"1496'4821","reported_seq":4856,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:14.221139+0000","last_change":"2024-09-21T10:54:11.455255+0000","last_active":"2024-09-21T10:55:14.221139+0000","last_peered":"2024-09-21T10:55:14.221139+0000","last_clean":"2024-09-21T10:55:14.221139+0000","last_became_active":"2024-09-21T10:54:11.454741+0000","last_became_peered":"2024-09-21T10:54:11.454741+0000","last_unstale":"2024-09-21T10:55:14.221139+0000","last_undegraded":"2024-09-21T10:55:14.221139+0000","last_fullsized":"2024-09-21T10:55:14.221139+0000","mapping_epoch":1484,"log_start":"1489'1110","ondisk_log_start":"1489'1110","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":3711,"log_dups_size":0,"ondisk_log_size":3711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:42:08.583105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315949056,"num_objects":4821,"num_object_clones":0,"num_object_copies":14463,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4821,"num_write_kb":308544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":2002,"reported_epoch":1495,"state":"active+clean","last_fresh":"2024-09-21T10:55:04.361306+0000","last_change":"2024-09-21T10:55:01.487069+0000","last_active":"2024-09-21T10:55:04.361306+0000","last_peered":"2024-09-21T10:55:04.361306+0000","last_clean":"2024-09-21T10:55:04.361306+0000","last_became_active":"2024-09-21T10:55:01.468554+0000","last_became_peered":"2024-09-21T10:55:01.468554+0000","last_unstale":"2024-09-21T10:55:04.361306+0000","last_undegraded":"2024-09-21T10:55:04.361306+0000","last_fullsized":"2024-09-21T10:55:04.361306+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:59:55.312979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1459,"reported_epoch":1495,"state":"active+clean","last_fresh":"2024-09-21T10:55:04.369887+0000","last_change":"2024-09-21T10:55:01.479872+0000","last_active":"2024-09-21T10:55:04.369887+0000","last_peered":"2024-09-21T10:55:04.369887+0000","last_clean":"2024-09-21T10:55:04.369887+0000","last_became_active":"2024-09-21T10:55:01.479292+0000","last_became_peered":"2024-09-21T10:55:01.479292+0000","last_unstale":"2024-09-21T10:55:04.369887+0000","last_undegraded":"2024-09-21T10:55:04.369887+0000","last_fullsized":"2024-09-21T10:55:04.369887+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:36:53.565343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":2302,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.509794+0000","last_change":"2024-09-21T10:52:15.400227+0000","last_active":"2024-09-21T10:55:12.509794+0000","last_peered":"2024-09-21T10:55:12.509794+0000","last_clean":"2024-09-21T10:55:12.509794+0000","last_became_active":"2024-09-21T10:52:15.399888+0000","last_became_peered":"2024-09-21T10:52:15.399888+0000","last_unstale":"2024-09-21T10:55:12.509794+0000","last_undegraded":"2024-09-21T10:55:12.509794+0000","last_fullsized":"2024-09-21T10:55:12.509794+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:42:52.565184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2319,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.509059+0000","last_change":"2024-09-21T10:51:27.458216+0000","last_active":"2024-09-21T10:55:12.509059+0000","last_peered":"2024-09-21T10:55:12.509059+0000","last_clean":"2024-09-21T10:55:12.509059+0000","last_became_active":"2024-09-21T10:51:27.457948+0000","last_became_peered":"2024-09-21T10:51:27.457948+0000","last_unstale":"2024-09-21T10:55:12.509059+0000","last_undegraded":"2024-09-21T10:55:12.509059+0000","last_fullsized":"2024-09-21T10:55:12.509059+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:25:29.584079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":32,"reported_epoch":1495,"state":"active+clean","last_fresh":"2024-09-21T10:55:04.364507+0000","last_change":"2024-09-21T10:55:01.492728+0000","last_active":"2024-09-21T10:55:04.364507+0000","last_peered":"2024-09-21T10:55:04.364507+0000","last_clean":"2024-09-21T10:55:04.364507+0000","last_became_active":"2024-09-21T10:55:01.480502+0000","last_became_peered":"2024-09-21T10:55:01.480502+0000","last_unstale":"2024-09-21T10:55:04.364507+0000","last_undegraded":"2024-09-21T10:55:04.364507+0000","last_fullsized":"2024-09-21T10:55:04.364507+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:03:46.206970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1454,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.510088+0000","last_change":"2024-09-21T10:55:01.474486+0000","last_active":"2024-09-21T10:55:12.510088+0000","last_peered":"2024-09-21T10:55:12.510088+0000","last_clean":"2024-09-21T10:55:12.510088+0000","last_became_active":"2024-09-21T10:55:01.473850+0000","last_became_peered":"2024-09-21T10:55:01.473850+0000","last_unstale":"2024-09-21T10:55:12.510088+0000","last_undegraded":"2024-09-21T10:55:12.510088+0000","last_fullsized":"2024-09-21T10:55:12.510088+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:10.540522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":37,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.526165+0000","last_change":"2024-09-21T10:55:01.492881+0000","last_active":"2024-09-21T10:55:12.526165+0000","last_peered":"2024-09-21T10:55:12.526165+0000","last_clean":"2024-09-21T10:55:12.526165+0000","last_became_active":"2024-09-21T10:55:01.479714+0000","last_became_peered":"2024-09-21T10:55:01.479714+0000","last_unstale":"2024-09-21T10:55:12.526165+0000","last_undegraded":"2024-09-21T10:55:12.526165+0000","last_fullsized":"2024-09-21T10:55:12.526165+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:06:10.023859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":676,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.509454+0000","last_change":"2024-09-21T10:55:01.471821+0000","last_active":"2024-09-21T10:55:12.509454+0000","last_peered":"2024-09-21T10:55:12.509454+0000","last_clean":"2024-09-21T10:55:12.509454+0000","last_became_active":"2024-09-21T10:55:01.470979+0000","last_became_peered":"2024-09-21T10:55:01.470979+0000","last_unstale":"2024-09-21T10:55:12.509454+0000","last_undegraded":"2024-09-21T10:55:12.509454+0000","last_fullsized":"2024-09-21T10:55:12.509454+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:28:49.471629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":2039,"reported_epoch":1494,"state":"active+clean","last_fresh":"2024-09-21T10:55:03.351061+0000","last_change":"2024-09-21T10:55:01.475204+0000","last_active":"2024-09-21T10:55:03.351061+0000","last_peered":"2024-09-21T10:55:03.351061+0000","last_clean":"2024-09-21T10:55:03.351061+0000","last_became_active":"2024-09-21T10:55:01.474554+0000","last_became_peered":"2024-09-21T10:55:01.474554+0000","last_unstale":"2024-09-21T10:55:03.351061+0000","last_undegraded":"2024-09-21T10:55:03.351061+0000","last_fullsized":"2024-09-21T10:55:03.351061+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:24.849795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1080,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.526402+0000","last_change":"2024-09-21T10:55:01.466683+0000","last_active":"2024-09-21T10:55:12.526402+0000","last_peered":"2024-09-21T10:55:12.526402+0000","last_clean":"2024-09-21T10:55:12.526402+0000","last_became_active":"2024-09-21T10:55:01.465859+0000","last_became_peered":"2024-09-21T10:55:01.465859+0000","last_unstale":"2024-09-21T10:55:12.526402+0000","last_undegraded":"2024-09-21T10:55:12.526402+0000","last_fullsized":"2024-09-21T10:55:12.526402+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:34:59.495754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":21,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.516204+0000","last_change":"2024-09-21T10:55:01.472947+0000","last_active":"2024-09-21T10:55:12.516204+0000","last_peered":"2024-09-21T10:55:12.516204+0000","last_clean":"2024-09-21T10:55:12.516204+0000","last_became_active":"2024-09-21T10:55:01.471388+0000","last_became_peered":"2024-09-21T10:55:01.471388+0000","last_unstale":"2024-09-21T10:55:12.516204+0000","last_undegraded":"2024-09-21T10:55:12.516204+0000","last_fullsized":"2024-09-21T10:55:12.516204+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:33:06.060427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2305,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.512423+0000","last_change":"2024-09-21T10:55:01.472931+0000","last_active":"2024-09-21T10:55:12.512423+0000","last_peered":"2024-09-21T10:55:12.512423+0000","last_clean":"2024-09-21T10:55:12.512423+0000","last_became_active":"2024-09-21T10:55:01.469286+0000","last_became_peered":"2024-09-21T10:55:01.469286+0000","last_unstale":"2024-09-21T10:55:12.512423+0000","last_undegraded":"2024-09-21T10:55:12.512423+0000","last_fullsized":"2024-09-21T10:55:12.512423+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:27:59.942806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2277,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.513286+0000","last_change":"2024-09-21T10:55:01.477818+0000","last_active":"2024-09-21T10:55:12.513286+0000","last_peered":"2024-09-21T10:55:12.513286+0000","last_clean":"2024-09-21T10:55:12.513286+0000","last_became_active":"2024-09-21T10:55:01.477450+0000","last_became_peered":"2024-09-21T10:55:01.477450+0000","last_unstale":"2024-09-21T10:55:12.513286+0000","last_undegraded":"2024-09-21T10:55:12.513286+0000","last_fullsized":"2024-09-21T10:55:12.513286+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:38:51.872952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":41,"reported_epoch":1494,"state":"active+clean","last_fresh":"2024-09-21T10:55:03.359679+0000","last_change":"2024-09-21T10:52:25.489896+0000","last_active":"2024-09-21T10:55:03.359679+0000","last_peered":"2024-09-21T10:55:03.359679+0000","last_clean":"2024-09-21T10:55:03.359679+0000","last_became_active":"2024-09-21T10:52:25.489154+0000","last_became_peered":"2024-09-21T10:52:25.489154+0000","last_unstale":"2024-09-21T10:55:03.359679+0000","last_undegraded":"2024-09-21T10:55:03.359679+0000","last_fullsized":"2024-09-21T10:55:03.359679+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:36:02.076931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2299,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.509113+0000","last_change":"2024-09-21T10:52:25.493309+0000","last_active":"2024-09-21T10:55:12.509113+0000","last_peered":"2024-09-21T10:55:12.509113+0000","last_clean":"2024-09-21T10:55:12.509113+0000","last_became_active":"2024-09-21T10:52:25.492568+0000","last_became_peered":"2024-09-21T10:52:25.492568+0000","last_unstale":"2024-09-21T10:55:12.509113+0000","last_undegraded":"2024-09-21T10:55:12.509113+0000","last_fullsized":"2024-09-21T10:55:12.509113+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:19:19.393370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1499,"reported_epoch":1495,"state":"active+clean","last_fresh":"2024-09-21T10:55:04.361374+0000","last_change":"2024-09-21T10:52:03.317474+0000","last_active":"2024-09-21T10:55:04.361374+0000","last_peered":"2024-09-21T10:55:04.361374+0000","last_clean":"2024-09-21T10:55:04.361374+0000","last_became_active":"2024-09-21T10:52:03.317001+0000","last_became_peered":"2024-09-21T10:52:03.317001+0000","last_unstale":"2024-09-21T10:55:04.361374+0000","last_undegraded":"2024-09-21T10:55:04.361374+0000","last_fullsized":"2024-09-21T10:55:04.361374+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:06:07.039763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2298,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.510131+0000","last_change":"2024-09-21T10:52:15.412246+0000","last_active":"2024-09-21T10:55:12.510131+0000","last_peered":"2024-09-21T10:55:12.510131+0000","last_clean":"2024-09-21T10:55:12.510131+0000","last_became_active":"2024-09-21T10:52:15.406204+0000","last_became_peered":"2024-09-21T10:52:15.406204+0000","last_unstale":"2024-09-21T10:55:12.510131+0000","last_undegraded":"2024-09-21T10:55:12.510131+0000","last_fullsized":"2024-09-21T10:55:12.510131+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:03.524324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2278,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.510794+0000","last_change":"2024-09-21T10:52:15.402822+0000","last_active":"2024-09-21T10:55:12.510794+0000","last_peered":"2024-09-21T10:55:12.510794+0000","last_clean":"2024-09-21T10:55:12.510794+0000","last_became_active":"2024-09-21T10:52:15.402548+0000","last_became_peered":"2024-09-21T10:52:15.402548+0000","last_unstale":"2024-09-21T10:55:12.510794+0000","last_undegraded":"2024-09-21T10:55:12.510794+0000","last_fullsized":"2024-09-21T10:55:12.510794+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:04:04.667598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2303,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.510228+0000","last_change":"2024-09-21T10:52:25.499172+0000","last_active":"2024-09-21T10:55:12.510228+0000","last_peered":"2024-09-21T10:55:12.510228+0000","last_clean":"2024-09-21T10:55:12.510228+0000","last_became_active":"2024-09-21T10:52:25.498835+0000","last_became_peered":"2024-09-21T10:52:25.498835+0000","last_unstale":"2024-09-21T10:55:12.510228+0000","last_undegraded":"2024-09-21T10:55:12.510228+0000","last_fullsized":"2024-09-21T10:55:12.510228+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:10:12.434470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":41,"reported_epoch":1494,"state":"active+clean","last_fresh":"2024-09-21T10:55:03.359607+0000","last_change":"2024-09-21T10:52:25.489774+0000","last_active":"2024-09-21T10:55:03.359607+0000","last_peered":"2024-09-21T10:55:03.359607+0000","last_clean":"2024-09-21T10:55:03.359607+0000","last_became_active":"2024-09-21T10:52:25.487788+0000","last_became_peered":"2024-09-21T10:52:25.487788+0000","last_unstale":"2024-09-21T10:55:03.359607+0000","last_undegraded":"2024-09-21T10:55:03.359607+0000","last_fullsized":"2024-09-21T10:55:03.359607+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:48.842419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":62,"reported_epoch":1494,"state":"active+clean","last_fresh":"2024-09-21T10:55:03.351870+0000","last_change":"2024-09-21T10:55:01.490959+0000","last_active":"2024-09-21T10:55:03.351870+0000","last_peered":"2024-09-21T10:55:03.351870+0000","last_clean":"2024-09-21T10:55:03.351870+0000","last_became_active":"2024-09-21T10:55:01.466537+0000","last_became_peered":"2024-09-21T10:55:01.466537+0000","last_unstale":"2024-09-21T10:55:03.351870+0000","last_undegraded":"2024-09-21T10:55:03.351870+0000","last_fullsized":"2024-09-21T10:55:03.351870+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:39:39.829439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2305,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.516461+0000","last_change":"2024-09-21T10:52:25.504787+0000","last_active":"2024-09-21T10:55:12.516461+0000","last_peered":"2024-09-21T10:55:12.516461+0000","last_clean":"2024-09-21T10:55:12.516461+0000","last_became_active":"2024-09-21T10:52:25.504198+0000","last_became_peered":"2024-09-21T10:52:25.504198+0000","last_unstale":"2024-09-21T10:55:12.516461+0000","last_undegraded":"2024-09-21T10:55:12.516461+0000","last_fullsized":"2024-09-21T10:55:12.516461+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:38:25.666224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":1392,"reported_epoch":1494,"state":"active+clean","last_fresh":"2024-09-21T10:55:03.366817+0000","last_change":"2024-09-21T10:55:01.484570+0000","last_active":"2024-09-21T10:55:03.366817+0000","last_peered":"2024-09-21T10:55:03.366817+0000","last_clean":"2024-09-21T10:55:03.366817+0000","last_became_active":"2024-09-21T10:55:01.465514+0000","last_became_peered":"2024-09-21T10:55:01.465514+0000","last_unstale":"2024-09-21T10:55:03.366817+0000","last_undegraded":"2024-09-21T10:55:03.366817+0000","last_fullsized":"2024-09-21T10:55:03.366817+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:21:03.466224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":638,"reported_epoch":1495,"state":"active+clean","last_fresh":"2024-09-21T10:55:04.363533+0000","last_change":"2024-09-21T10:55:01.479588+0000","last_active":"2024-09-21T10:55:04.363533+0000","last_peered":"2024-09-21T10:55:04.363533+0000","last_clean":"2024-09-21T10:55:04.363533+0000","last_became_active":"2024-09-21T10:55:01.478467+0000","last_became_peered":"2024-09-21T10:55:01.478467+0000","last_unstale":"2024-09-21T10:55:04.363533+0000","last_undegraded":"2024-09-21T10:55:04.363533+0000","last_fullsized":"2024-09-21T10:55:04.363533+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:05.308443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"346'149","reported_seq":966,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.511134+0000","last_change":"2024-09-21T10:53:47.069462+0000","last_active":"2024-09-21T10:55:12.511134+0000","last_peered":"2024-09-21T10:55:12.511134+0000","last_clean":"2024-09-21T10:55:12.511134+0000","last_became_active":"2024-09-21T10:53:47.069113+0000","last_became_peered":"2024-09-21T10:53:47.069113+0000","last_unstale":"2024-09-21T10:55:12.511134+0000","last_undegraded":"2024-09-21T10:55:12.511134+0000","last_fullsized":"2024-09-21T10:55:12.511134+0000","mapping_epoch":1478,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1479,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1579069,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1460,"reported_epoch":1495,"state":"active+clean","last_fresh":"2024-09-21T10:55:04.360293+0000","last_change":"2024-09-21T10:55:01.468920+0000","last_active":"2024-09-21T10:55:04.360293+0000","last_peered":"2024-09-21T10:55:04.360293+0000","last_clean":"2024-09-21T10:55:04.360293+0000","last_became_active":"2024-09-21T10:55:01.468606+0000","last_became_peered":"2024-09-21T10:55:01.468606+0000","last_unstale":"2024-09-21T10:55:04.360293+0000","last_undegraded":"2024-09-21T10:55:04.360293+0000","last_fullsized":"2024-09-21T10:55:04.360293+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:31:02.135976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1530,"reported_epoch":1495,"state":"active+clean","last_fresh":"2024-09-21T10:55:04.361437+0000","last_change":"2024-09-21T10:52:25.493263+0000","last_active":"2024-09-21T10:55:04.361437+0000","last_peered":"2024-09-21T10:55:04.361437+0000","last_clean":"2024-09-21T10:55:04.361437+0000","last_became_active":"2024-09-21T10:52:25.492970+0000","last_became_peered":"2024-09-21T10:52:25.492970+0000","last_unstale":"2024-09-21T10:55:04.361437+0000","last_undegraded":"2024-09-21T10:55:04.361437+0000","last_fullsized":"2024-09-21T10:55:04.361437+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:43:40.305783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":160,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.514189+0000","last_change":"2024-09-21T10:55:01.488578+0000","last_active":"2024-09-21T10:55:12.514189+0000","last_peered":"2024-09-21T10:55:12.514189+0000","last_clean":"2024-09-21T10:55:12.514189+0000","last_became_active":"2024-09-21T10:55:01.477816+0000","last_became_peered":"2024-09-21T10:55:01.477816+0000","last_unstale":"2024-09-21T10:55:12.514189+0000","last_undegraded":"2024-09-21T10:55:12.514189+0000","last_fullsized":"2024-09-21T10:55:12.514189+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:01:01.361589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"346'149","reported_seq":640,"reported_epoch":1495,"state":"active+clean","last_fresh":"2024-09-21T10:55:04.396099+0000","last_change":"2024-09-21T10:55:04.394887+0000","last_active":"2024-09-21T10:55:04.396099+0000","last_peered":"2024-09-21T10:55:04.396099+0000","last_clean":"2024-09-21T10:55:04.396099+0000","last_became_active":"2024-09-21T10:55:04.394422+0000","last_became_peered":"2024-09-21T10:55:04.394422+0000","last_unstale":"2024-09-21T10:55:04.396099+0000","last_undegraded":"2024-09-21T10:55:04.396099+0000","last_fullsized":"2024-09-21T10:55:04.396099+0000","mapping_epoch":1494,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1495,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:22.051403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:22.051403+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2270,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.509183+0000","last_change":"2024-09-21T10:52:25.494316+0000","last_active":"2024-09-21T10:55:12.509183+0000","last_peered":"2024-09-21T10:55:12.509183+0000","last_clean":"2024-09-21T10:55:12.509183+0000","last_became_active":"2024-09-21T10:52:25.494016+0000","last_became_peered":"2024-09-21T10:52:25.494016+0000","last_unstale":"2024-09-21T10:55:12.509183+0000","last_undegraded":"2024-09-21T10:55:12.509183+0000","last_fullsized":"2024-09-21T10:55:12.509183+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:07.859584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2278,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.514363+0000","last_change":"2024-09-21T10:55:01.480701+0000","last_active":"2024-09-21T10:55:12.514363+0000","last_peered":"2024-09-21T10:55:12.514363+0000","last_clean":"2024-09-21T10:55:12.514363+0000","last_became_active":"2024-09-21T10:55:01.479997+0000","last_became_peered":"2024-09-21T10:55:01.479997+0000","last_unstale":"2024-09-21T10:55:12.514363+0000","last_undegraded":"2024-09-21T10:55:12.514363+0000","last_fullsized":"2024-09-21T10:55:12.514363+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:46:34.397469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":2332,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.517065+0000","last_change":"2024-09-21T10:55:01.480736+0000","last_active":"2024-09-21T10:55:12.517065+0000","last_peered":"2024-09-21T10:55:12.517065+0000","last_clean":"2024-09-21T10:55:12.517065+0000","last_became_active":"2024-09-21T10:55:01.471095+0000","last_became_peered":"2024-09-21T10:55:01.471095+0000","last_unstale":"2024-09-21T10:55:12.517065+0000","last_undegraded":"2024-09-21T10:55:12.517065+0000","last_fullsized":"2024-09-21T10:55:12.517065+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:49:16.475617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"346'149","reported_seq":758,"reported_epoch":1495,"state":"active+clean","last_fresh":"2024-09-21T10:55:04.385835+0000","last_change":"2024-09-21T10:55:04.385463+0000","last_active":"2024-09-21T10:55:04.385835+0000","last_peered":"2024-09-21T10:55:04.385835+0000","last_clean":"2024-09-21T10:55:04.385835+0000","last_became_active":"2024-09-21T10:55:04.385119+0000","last_became_peered":"2024-09-21T10:55:04.385119+0000","last_unstale":"2024-09-21T10:55:04.385835+0000","last_undegraded":"2024-09-21T10:55:04.385835+0000","last_fullsized":"2024-09-21T10:55:04.385835+0000","mapping_epoch":1494,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1495,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:41.011085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:41.011085+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:38:35.908119+0000","scrub_duration":0.0061250310000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2220,"reported_epoch":1494,"state":"active+clean","last_fresh":"2024-09-21T10:55:03.351130+0000","last_change":"2024-09-21T10:55:01.474981+0000","last_active":"2024-09-21T10:55:03.351130+0000","last_peered":"2024-09-21T10:55:03.351130+0000","last_clean":"2024-09-21T10:55:03.351130+0000","last_became_active":"2024-09-21T10:55:01.474594+0000","last_became_peered":"2024-09-21T10:55:01.474594+0000","last_unstale":"2024-09-21T10:55:03.351130+0000","last_undegraded":"2024-09-21T10:55:03.351130+0000","last_fullsized":"2024-09-21T10:55:03.351130+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:07:49.272842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":50,"reported_epoch":1494,"state":"active+clean","last_fresh":"2024-09-21T10:55:03.351796+0000","last_change":"2024-09-21T10:55:01.468733+0000","last_active":"2024-09-21T10:55:03.351796+0000","last_peered":"2024-09-21T10:55:03.351796+0000","last_clean":"2024-09-21T10:55:03.351796+0000","last_became_active":"2024-09-21T10:55:01.468372+0000","last_became_peered":"2024-09-21T10:55:01.468372+0000","last_unstale":"2024-09-21T10:55:03.351796+0000","last_undegraded":"2024-09-21T10:55:03.351796+0000","last_fullsized":"2024-09-21T10:55:03.351796+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:03:06.257682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2328,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.526310+0000","last_change":"2024-09-21T10:55:01.480857+0000","last_active":"2024-09-21T10:55:12.526310+0000","last_peered":"2024-09-21T10:55:12.526310+0000","last_clean":"2024-09-21T10:55:12.526310+0000","last_became_active":"2024-09-21T10:55:01.480202+0000","last_became_peered":"2024-09-21T10:55:01.480202+0000","last_unstale":"2024-09-21T10:55:12.526310+0000","last_undegraded":"2024-09-21T10:55:12.526310+0000","last_fullsized":"2024-09-21T10:55:12.526310+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:27.756242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"346'149","reported_seq":784,"reported_epoch":1495,"state":"active+clean","last_fresh":"2024-09-21T10:55:04.361212+0000","last_change":"2024-09-21T10:52:28.590793+0000","last_active":"2024-09-21T10:55:04.361212+0000","last_peered":"2024-09-21T10:55:04.361212+0000","last_clean":"2024-09-21T10:55:04.361212+0000","last_became_active":"2024-09-21T10:52:28.588662+0000","last_became_peered":"2024-09-21T10:52:28.588662+0000","last_unstale":"2024-09-21T10:55:04.361212+0000","last_undegraded":"2024-09-21T10:55:04.361212+0000","last_fullsized":"2024-09-21T10:55:04.361212+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:44.018645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:44.018645+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:46.849427+0000","scrub_duration":0.0012789399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2283,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.516314+0000","last_change":"2024-09-21T10:55:01.469713+0000","last_active":"2024-09-21T10:55:12.516314+0000","last_peered":"2024-09-21T10:55:12.516314+0000","last_clean":"2024-09-21T10:55:12.516314+0000","last_became_active":"2024-09-21T10:55:01.468470+0000","last_became_peered":"2024-09-21T10:55:01.468470+0000","last_unstale":"2024-09-21T10:55:12.516314+0000","last_undegraded":"2024-09-21T10:55:12.516314+0000","last_fullsized":"2024-09-21T10:55:12.516314+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:17:34.181766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2344,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.526084+0000","last_change":"2024-09-21T10:52:03.314926+0000","last_active":"2024-09-21T10:55:12.526084+0000","last_peered":"2024-09-21T10:55:12.526084+0000","last_clean":"2024-09-21T10:55:12.526084+0000","last_became_active":"2024-09-21T10:52:03.314602+0000","last_became_peered":"2024-09-21T10:52:03.314602+0000","last_unstale":"2024-09-21T10:55:12.526084+0000","last_undegraded":"2024-09-21T10:55:12.526084+0000","last_fullsized":"2024-09-21T10:55:12.526084+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:13.774521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1500,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.512272+0000","last_change":"2024-09-21T10:52:25.502830+0000","last_active":"2024-09-21T10:55:12.512272+0000","last_peered":"2024-09-21T10:55:12.512272+0000","last_clean":"2024-09-21T10:55:12.512272+0000","last_became_active":"2024-09-21T10:52:25.502505+0000","last_became_peered":"2024-09-21T10:52:25.502505+0000","last_unstale":"2024-09-21T10:55:12.512272+0000","last_undegraded":"2024-09-21T10:55:12.512272+0000","last_fullsized":"2024-09-21T10:55:12.512272+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:01:14.965694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"346'149","reported_seq":812,"reported_epoch":1495,"state":"active+clean","last_fresh":"2024-09-21T10:55:04.367965+0000","last_change":"2024-09-21T10:55:04.367965+0000","last_active":"2024-09-21T10:55:04.367965+0000","last_peered":"2024-09-21T10:55:04.367965+0000","last_clean":"2024-09-21T10:55:04.367965+0000","last_became_active":"2024-09-21T10:55:04.367565+0000","last_became_peered":"2024-09-21T10:55:04.367565+0000","last_unstale":"2024-09-21T10:55:04.367965+0000","last_undegraded":"2024-09-21T10:55:04.367965+0000","last_fullsized":"2024-09-21T10:55:04.367965+0000","mapping_epoch":1494,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1495,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:16.021843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:16.021843+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2338,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.526485+0000","last_change":"2024-09-21T10:52:15.397180+0000","last_active":"2024-09-21T10:55:12.526485+0000","last_peered":"2024-09-21T10:55:12.526485+0000","last_clean":"2024-09-21T10:55:12.526485+0000","last_became_active":"2024-09-21T10:52:15.396879+0000","last_became_peered":"2024-09-21T10:52:15.396879+0000","last_unstale":"2024-09-21T10:55:12.526485+0000","last_undegraded":"2024-09-21T10:55:12.526485+0000","last_fullsized":"2024-09-21T10:55:12.526485+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:03:07.162178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1440,"reported_epoch":1495,"state":"active+clean","last_fresh":"2024-09-21T10:55:04.362761+0000","last_change":"2024-09-21T10:55:01.479586+0000","last_active":"2024-09-21T10:55:04.362761+0000","last_peered":"2024-09-21T10:55:04.362761+0000","last_clean":"2024-09-21T10:55:04.362761+0000","last_became_active":"2024-09-21T10:55:01.478541+0000","last_became_peered":"2024-09-21T10:55:01.478541+0000","last_unstale":"2024-09-21T10:55:04.362761+0000","last_undegraded":"2024-09-21T10:55:04.362761+0000","last_fullsized":"2024-09-21T10:55:04.362761+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:51:10.154966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":239,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.516076+0000","last_change":"2024-09-21T10:53:41.149328+0000","last_active":"2024-09-21T10:55:12.516076+0000","last_peered":"2024-09-21T10:55:12.516076+0000","last_clean":"2024-09-21T10:55:12.516076+0000","last_became_active":"2024-09-21T10:52:03.315398+0000","last_became_peered":"2024-09-21T10:52:03.315398+0000","last_unstale":"2024-09-21T10:55:12.516076+0000","last_undegraded":"2024-09-21T10:55:12.516076+0000","last_fullsized":"2024-09-21T10:55:12.516076+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:48:09.007053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"346'149","reported_seq":838,"reported_epoch":1495,"state":"active+clean","last_fresh":"2024-09-21T10:55:04.366355+0000","last_change":"2024-09-21T10:55:04.366355+0000","last_active":"2024-09-21T10:55:04.366355+0000","last_peered":"2024-09-21T10:55:04.366355+0000","last_clean":"2024-09-21T10:55:04.366355+0000","last_became_active":"2024-09-21T10:55:04.365929+0000","last_became_peered":"2024-09-21T10:55:04.365929+0000","last_unstale":"2024-09-21T10:55:04.366355+0000","last_undegraded":"2024-09-21T10:55:04.366355+0000","last_fullsized":"2024-09-21T10:55:04.366355+0000","mapping_epoch":1494,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1495,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:21.445783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:21.445783+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:04:30.538721+0000","scrub_duration":0.0018624990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":20,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.520572+0000","last_change":"2024-09-21T10:55:01.469268+0000","last_active":"2024-09-21T10:55:12.520572+0000","last_peered":"2024-09-21T10:55:12.520572+0000","last_clean":"2024-09-21T10:55:12.520572+0000","last_became_active":"2024-09-21T10:55:01.468878+0000","last_became_peered":"2024-09-21T10:55:01.468878+0000","last_unstale":"2024-09-21T10:55:12.520572+0000","last_undegraded":"2024-09-21T10:55:12.520572+0000","last_fullsized":"2024-09-21T10:55:12.520572+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:03:24.884225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2295,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.512592+0000","last_change":"2024-09-21T10:52:03.325889+0000","last_active":"2024-09-21T10:55:12.512592+0000","last_peered":"2024-09-21T10:55:12.512592+0000","last_clean":"2024-09-21T10:55:12.512592+0000","last_became_active":"2024-09-21T10:52:03.325500+0000","last_became_peered":"2024-09-21T10:52:03.325500+0000","last_unstale":"2024-09-21T10:55:12.512592+0000","last_undegraded":"2024-09-21T10:55:12.512592+0000","last_fullsized":"2024-09-21T10:55:12.512592+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:20.708252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1081,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.508955+0000","last_change":"2024-09-21T10:51:27.457894+0000","last_active":"2024-09-21T10:55:12.508955+0000","last_peered":"2024-09-21T10:55:12.508955+0000","last_clean":"2024-09-21T10:55:12.508955+0000","last_became_active":"2024-09-21T10:51:27.457431+0000","last_became_peered":"2024-09-21T10:51:27.457431+0000","last_unstale":"2024-09-21T10:55:12.508955+0000","last_undegraded":"2024-09-21T10:55:12.508955+0000","last_fullsized":"2024-09-21T10:55:12.508955+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:03:33.550759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"346'149","reported_seq":961,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.513852+0000","last_change":"2024-09-21T10:55:04.370841+0000","last_active":"2024-09-21T10:55:12.513852+0000","last_peered":"2024-09-21T10:55:12.513852+0000","last_clean":"2024-09-21T10:55:12.513852+0000","last_became_active":"2024-09-21T10:55:04.370543+0000","last_became_peered":"2024-09-21T10:55:04.370543+0000","last_unstale":"2024-09-21T10:55:12.513852+0000","last_undegraded":"2024-09-21T10:55:12.513852+0000","last_fullsized":"2024-09-21T10:55:12.513852+0000","mapping_epoch":1494,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1495,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2327,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.514112+0000","last_change":"2024-09-21T10:55:01.480224+0000","last_active":"2024-09-21T10:55:12.514112+0000","last_peered":"2024-09-21T10:55:12.514112+0000","last_clean":"2024-09-21T10:55:12.514112+0000","last_became_active":"2024-09-21T10:55:01.479045+0000","last_became_peered":"2024-09-21T10:55:01.479045+0000","last_unstale":"2024-09-21T10:55:12.514112+0000","last_undegraded":"2024-09-21T10:55:12.514112+0000","last_fullsized":"2024-09-21T10:55:12.514112+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:10.816874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2199,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.509147+0000","last_change":"2024-09-21T10:55:01.471853+0000","last_active":"2024-09-21T10:55:12.509147+0000","last_peered":"2024-09-21T10:55:12.509147+0000","last_clean":"2024-09-21T10:55:12.509147+0000","last_became_active":"2024-09-21T10:55:01.470262+0000","last_became_peered":"2024-09-21T10:55:01.470262+0000","last_unstale":"2024-09-21T10:55:12.509147+0000","last_undegraded":"2024-09-21T10:55:12.509147+0000","last_fullsized":"2024-09-21T10:55:12.509147+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:22:50.746700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":150,"reported_epoch":1494,"state":"active+clean","last_fresh":"2024-09-21T10:55:03.351489+0000","last_change":"2024-09-21T10:51:27.426006+0000","last_active":"2024-09-21T10:55:03.351489+0000","last_peered":"2024-09-21T10:55:03.351489+0000","last_clean":"2024-09-21T10:55:03.351489+0000","last_became_active":"2024-09-21T10:51:27.425698+0000","last_became_peered":"2024-09-21T10:51:27.425698+0000","last_unstale":"2024-09-21T10:55:03.351489+0000","last_undegraded":"2024-09-21T10:55:03.351489+0000","last_fullsized":"2024-09-21T10:55:03.351489+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:35:13.210330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":54,"reported_epoch":1495,"state":"active+clean","last_fresh":"2024-09-21T10:55:04.363913+0000","last_change":"2024-09-21T10:55:01.472846+0000","last_active":"2024-09-21T10:55:04.363913+0000","last_peered":"2024-09-21T10:55:04.363913+0000","last_clean":"2024-09-21T10:55:04.363913+0000","last_became_active":"2024-09-21T10:55:01.472355+0000","last_became_peered":"2024-09-21T10:55:01.472355+0000","last_unstale":"2024-09-21T10:55:04.363913+0000","last_undegraded":"2024-09-21T10:55:04.363913+0000","last_fullsized":"2024-09-21T10:55:04.363913+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:54.663396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2282,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.510169+0000","last_change":"2024-09-21T10:55:01.474215+0000","last_active":"2024-09-21T10:55:12.510169+0000","last_peered":"2024-09-21T10:55:12.510169+0000","last_clean":"2024-09-21T10:55:12.510169+0000","last_became_active":"2024-09-21T10:55:01.473444+0000","last_became_peered":"2024-09-21T10:55:01.473444+0000","last_unstale":"2024-09-21T10:55:12.510169+0000","last_undegraded":"2024-09-21T10:55:12.510169+0000","last_fullsized":"2024-09-21T10:55:12.510169+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:30:26.025903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":1475,"reported_epoch":1494,"state":"active+clean","last_fresh":"2024-09-21T10:55:03.351199+0000","last_change":"2024-09-21T10:52:25.489550+0000","last_active":"2024-09-21T10:55:03.351199+0000","last_peered":"2024-09-21T10:55:03.351199+0000","last_clean":"2024-09-21T10:55:03.351199+0000","last_became_active":"2024-09-21T10:52:25.489171+0000","last_became_peered":"2024-09-21T10:52:25.489171+0000","last_unstale":"2024-09-21T10:55:03.351199+0000","last_undegraded":"2024-09-21T10:55:03.351199+0000","last_fullsized":"2024-09-21T10:55:03.351199+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:21.105058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":821,"reported_epoch":1494,"state":"active+clean","last_fresh":"2024-09-21T10:55:03.351359+0000","last_change":"2024-09-21T10:54:20.458538+0000","last_active":"2024-09-21T10:55:03.351359+0000","last_peered":"2024-09-21T10:55:03.351359+0000","last_clean":"2024-09-21T10:55:03.351359+0000","last_became_active":"2024-09-21T10:52:28.587447+0000","last_became_peered":"2024-09-21T10:52:28.587447+0000","last_unstale":"2024-09-21T10:55:03.351359+0000","last_undegraded":"2024-09-21T10:55:03.351359+0000","last_fullsized":"2024-09-21T10:55:03.351359+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:20.458421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:20.458421+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:35.467492+0000","scrub_duration":0.001528569,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"346'149","reported_seq":775,"reported_epoch":1495,"state":"active+clean","last_fresh":"2024-09-21T10:55:04.368995+0000","last_change":"2024-09-21T10:55:04.368448+0000","last_active":"2024-09-21T10:55:04.368995+0000","last_peered":"2024-09-21T10:55:04.368995+0000","last_clean":"2024-09-21T10:55:04.368995+0000","last_became_active":"2024-09-21T10:55:04.368107+0000","last_became_peered":"2024-09-21T10:55:04.368107+0000","last_unstale":"2024-09-21T10:55:04.368995+0000","last_undegraded":"2024-09-21T10:55:04.368995+0000","last_fullsized":"2024-09-21T10:55:04.368995+0000","mapping_epoch":1494,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1495,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:42.008235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:42.008235+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:07:30.729422+0000","scrub_duration":0.011129834999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2278,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.525422+0000","last_change":"2024-09-21T10:55:01.475118+0000","last_active":"2024-09-21T10:55:12.525422+0000","last_peered":"2024-09-21T10:55:12.525422+0000","last_clean":"2024-09-21T10:55:12.525422+0000","last_became_active":"2024-09-21T10:55:01.474515+0000","last_became_peered":"2024-09-21T10:55:01.474515+0000","last_unstale":"2024-09-21T10:55:12.525422+0000","last_undegraded":"2024-09-21T10:55:12.525422+0000","last_fullsized":"2024-09-21T10:55:12.525422+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:25:01.070263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2270,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.509031+0000","last_change":"2024-09-21T10:52:25.493032+0000","last_active":"2024-09-21T10:55:12.509031+0000","last_peered":"2024-09-21T10:55:12.509031+0000","last_clean":"2024-09-21T10:55:12.509031+0000","last_became_active":"2024-09-21T10:52:25.492182+0000","last_became_peered":"2024-09-21T10:52:25.492182+0000","last_unstale":"2024-09-21T10:55:12.509031+0000","last_undegraded":"2024-09-21T10:55:12.509031+0000","last_fullsized":"2024-09-21T10:55:12.509031+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:12:57.118420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1461,"reported_epoch":1495,"state":"active+clean","last_fresh":"2024-09-21T10:55:04.359598+0000","last_change":"2024-09-21T10:55:01.479104+0000","last_active":"2024-09-21T10:55:04.359598+0000","last_peered":"2024-09-21T10:55:04.359598+0000","last_clean":"2024-09-21T10:55:04.359598+0000","last_became_active":"2024-09-21T10:55:01.478380+0000","last_became_peered":"2024-09-21T10:55:01.478380+0000","last_unstale":"2024-09-21T10:55:04.359598+0000","last_undegraded":"2024-09-21T10:55:04.359598+0000","last_fullsized":"2024-09-21T10:55:04.359598+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:11.194951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1374,"reported_epoch":1494,"state":"active+clean","last_fresh":"2024-09-21T10:55:03.353216+0000","last_change":"2024-09-21T10:55:01.470469+0000","last_active":"2024-09-21T10:55:03.353216+0000","last_peered":"2024-09-21T10:55:03.353216+0000","last_clean":"2024-09-21T10:55:03.353216+0000","last_became_active":"2024-09-21T10:55:01.470097+0000","last_became_peered":"2024-09-21T10:55:01.470097+0000","last_unstale":"2024-09-21T10:55:03.353216+0000","last_undegraded":"2024-09-21T10:55:03.353216+0000","last_fullsized":"2024-09-21T10:55:03.353216+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:48:19.409017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"346'149","reported_seq":965,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.508938+0000","last_change":"2024-09-21T10:55:04.377180+0000","last_active":"2024-09-21T10:55:12.508938+0000","last_peered":"2024-09-21T10:55:12.508938+0000","last_clean":"2024-09-21T10:55:12.508938+0000","last_became_active":"2024-09-21T10:55:04.376659+0000","last_became_peered":"2024-09-21T10:55:04.376659+0000","last_unstale":"2024-09-21T10:55:12.508938+0000","last_undegraded":"2024-09-21T10:55:12.508938+0000","last_fullsized":"2024-09-21T10:55:12.508938+0000","mapping_epoch":1494,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1495,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1056413,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1310,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.513350+0000","last_change":"2024-09-21T10:55:01.477884+0000","last_active":"2024-09-21T10:55:12.513350+0000","last_peered":"2024-09-21T10:55:12.513350+0000","last_clean":"2024-09-21T10:55:12.513350+0000","last_became_active":"2024-09-21T10:55:01.477512+0000","last_became_peered":"2024-09-21T10:55:01.477512+0000","last_unstale":"2024-09-21T10:55:12.513350+0000","last_undegraded":"2024-09-21T10:55:12.513350+0000","last_fullsized":"2024-09-21T10:55:12.513350+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:58:06.062711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1481,"reported_epoch":1495,"state":"active+clean","last_fresh":"2024-09-21T10:55:04.362848+0000","last_change":"2024-09-21T10:55:01.480292+0000","last_active":"2024-09-21T10:55:04.362848+0000","last_peered":"2024-09-21T10:55:04.362848+0000","last_clean":"2024-09-21T10:55:04.362848+0000","last_became_active":"2024-09-21T10:55:01.479630+0000","last_became_peered":"2024-09-21T10:55:01.479630+0000","last_unstale":"2024-09-21T10:55:04.362848+0000","last_undegraded":"2024-09-21T10:55:04.362848+0000","last_fullsized":"2024-09-21T10:55:04.362848+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:42.598659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"346'149","reported_seq":946,"reported_epoch":1494,"state":"active+clean","last_fresh":"2024-09-21T10:55:03.353133+0000","last_change":"2024-09-21T10:54:17.418763+0000","last_active":"2024-09-21T10:55:03.353133+0000","last_peered":"2024-09-21T10:55:03.353133+0000","last_clean":"2024-09-21T10:55:03.353133+0000","last_became_active":"2024-09-21T10:49:47.903650+0000","last_became_peered":"2024-09-21T10:49:47.903650+0000","last_unstale":"2024-09-21T10:55:03.353133+0000","last_undegraded":"2024-09-21T10:55:03.353133+0000","last_fullsized":"2024-09-21T10:55:03.353133+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:17.418646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:17.418646+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:17.134488+0000","scrub_duration":0.00185648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":662,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.511474+0000","last_change":"2024-09-21T10:55:01.468438+0000","last_active":"2024-09-21T10:55:12.511474+0000","last_peered":"2024-09-21T10:55:12.511474+0000","last_clean":"2024-09-21T10:55:12.511474+0000","last_became_active":"2024-09-21T10:55:01.468044+0000","last_became_peered":"2024-09-21T10:55:01.468044+0000","last_unstale":"2024-09-21T10:55:12.511474+0000","last_undegraded":"2024-09-21T10:55:12.511474+0000","last_fullsized":"2024-09-21T10:55:12.511474+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:45:03.600664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":2310,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.509524+0000","last_change":"2024-09-21T10:55:01.484923+0000","last_active":"2024-09-21T10:55:12.509524+0000","last_peered":"2024-09-21T10:55:12.509524+0000","last_clean":"2024-09-21T10:55:12.509524+0000","last_became_active":"2024-09-21T10:55:01.471461+0000","last_became_peered":"2024-09-21T10:55:01.471461+0000","last_unstale":"2024-09-21T10:55:12.509524+0000","last_undegraded":"2024-09-21T10:55:12.509524+0000","last_fullsized":"2024-09-21T10:55:12.509524+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:54:11.289126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":736,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.514003+0000","last_change":"2024-09-21T10:55:01.478361+0000","last_active":"2024-09-21T10:55:12.514003+0000","last_peered":"2024-09-21T10:55:12.514003+0000","last_clean":"2024-09-21T10:55:12.514003+0000","last_became_active":"2024-09-21T10:55:01.478002+0000","last_became_peered":"2024-09-21T10:55:01.478002+0000","last_unstale":"2024-09-21T10:55:12.514003+0000","last_undegraded":"2024-09-21T10:55:12.514003+0000","last_fullsized":"2024-09-21T10:55:12.514003+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:33:01.481712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":834,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.511213+0000","last_change":"2024-09-21T10:49:44.864819+0000","last_active":"2024-09-21T10:55:12.511213+0000","last_peered":"2024-09-21T10:55:12.511213+0000","last_clean":"2024-09-21T10:55:12.511213+0000","last_became_active":"2024-09-21T10:49:44.864061+0000","last_became_peered":"2024-09-21T10:49:44.864061+0000","last_unstale":"2024-09-21T10:55:12.511213+0000","last_undegraded":"2024-09-21T10:55:12.511213+0000","last_fullsized":"2024-09-21T10:55:12.511213+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:37.266967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":54,"reported_epoch":1495,"state":"active+clean","last_fresh":"2024-09-21T10:55:04.362619+0000","last_change":"2024-09-21T10:52:25.492570+0000","last_active":"2024-09-21T10:55:04.362619+0000","last_peered":"2024-09-21T10:55:04.362619+0000","last_clean":"2024-09-21T10:55:04.362619+0000","last_became_active":"2024-09-21T10:52:25.486382+0000","last_became_peered":"2024-09-21T10:52:25.486382+0000","last_unstale":"2024-09-21T10:55:04.362619+0000","last_undegraded":"2024-09-21T10:55:04.362619+0000","last_fullsized":"2024-09-21T10:55:04.362619+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:06.702707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":157,"reported_epoch":1494,"state":"active+clean","last_fresh":"2024-09-21T10:55:03.352059+0000","last_change":"2024-09-21T10:49:44.874522+0000","last_active":"2024-09-21T10:55:03.352059+0000","last_peered":"2024-09-21T10:55:03.352059+0000","last_clean":"2024-09-21T10:55:03.352059+0000","last_became_active":"2024-09-21T10:49:44.863242+0000","last_became_peered":"2024-09-21T10:49:44.863242+0000","last_unstale":"2024-09-21T10:55:03.352059+0000","last_undegraded":"2024-09-21T10:55:03.352059+0000","last_fullsized":"2024-09-21T10:55:03.352059+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:09:02.066837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1475,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.513209+0000","last_change":"2024-09-21T10:55:01.477365+0000","last_active":"2024-09-21T10:55:12.513209+0000","last_peered":"2024-09-21T10:55:12.513209+0000","last_clean":"2024-09-21T10:55:12.513209+0000","last_became_active":"2024-09-21T10:55:01.476978+0000","last_became_peered":"2024-09-21T10:55:01.476978+0000","last_unstale":"2024-09-21T10:55:12.513209+0000","last_undegraded":"2024-09-21T10:55:12.513209+0000","last_fullsized":"2024-09-21T10:55:12.513209+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:28.893708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":58,"reported_epoch":1494,"state":"active+clean","last_fresh":"2024-09-21T10:55:03.359520+0000","last_change":"2024-09-21T10:55:01.496194+0000","last_active":"2024-09-21T10:55:03.359520+0000","last_peered":"2024-09-21T10:55:03.359520+0000","last_clean":"2024-09-21T10:55:03.359520+0000","last_became_active":"2024-09-21T10:55:01.469712+0000","last_became_peered":"2024-09-21T10:55:01.469712+0000","last_unstale":"2024-09-21T10:55:03.359520+0000","last_undegraded":"2024-09-21T10:55:03.359520+0000","last_fullsized":"2024-09-21T10:55:03.359520+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:17:45.286996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":50,"reported_epoch":1494,"state":"active+clean","last_fresh":"2024-09-21T10:55:03.351447+0000","last_change":"2024-09-21T10:55:01.462980+0000","last_active":"2024-09-21T10:55:03.351447+0000","last_peered":"2024-09-21T10:55:03.351447+0000","last_clean":"2024-09-21T10:55:03.351447+0000","last_became_active":"2024-09-21T10:55:01.462677+0000","last_became_peered":"2024-09-21T10:55:01.462677+0000","last_unstale":"2024-09-21T10:55:03.351447+0000","last_undegraded":"2024-09-21T10:55:03.351447+0000","last_fullsized":"2024-09-21T10:55:03.351447+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:20:13.489085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":58,"reported_epoch":1494,"state":"active+clean","last_fresh":"2024-09-21T10:55:03.362689+0000","last_change":"2024-09-21T10:55:01.494579+0000","last_active":"2024-09-21T10:55:03.362689+0000","last_peered":"2024-09-21T10:55:03.362689+0000","last_clean":"2024-09-21T10:55:03.362689+0000","last_became_active":"2024-09-21T10:55:01.469707+0000","last_became_peered":"2024-09-21T10:55:01.469707+0000","last_unstale":"2024-09-21T10:55:03.362689+0000","last_undegraded":"2024-09-21T10:55:03.362689+0000","last_fullsized":"2024-09-21T10:55:03.362689+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:36.380411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1516,"reported_epoch":1495,"state":"active+clean","last_fresh":"2024-09-21T10:55:04.370681+0000","last_change":"2024-09-21T10:51:27.517234+0000","last_active":"2024-09-21T10:55:04.370681+0000","last_peered":"2024-09-21T10:55:04.370681+0000","last_clean":"2024-09-21T10:55:04.370681+0000","last_became_active":"2024-09-21T10:51:27.431253+0000","last_became_peered":"2024-09-21T10:51:27.431253+0000","last_unstale":"2024-09-21T10:55:04.370681+0000","last_undegraded":"2024-09-21T10:55:04.370681+0000","last_fullsized":"2024-09-21T10:55:04.370681+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:35:42.249305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2254,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.510869+0000","last_change":"2024-09-21T10:52:25.497298+0000","last_active":"2024-09-21T10:55:12.510869+0000","last_peered":"2024-09-21T10:55:12.510869+0000","last_clean":"2024-09-21T10:55:12.510869+0000","last_became_active":"2024-09-21T10:52:25.496876+0000","last_became_peered":"2024-09-21T10:52:25.496876+0000","last_unstale":"2024-09-21T10:55:12.510869+0000","last_undegraded":"2024-09-21T10:55:12.510869+0000","last_fullsized":"2024-09-21T10:55:12.510869+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:16.587897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":149,"reported_epoch":1494,"state":"active+clean","last_fresh":"2024-09-21T10:55:03.353288+0000","last_change":"2024-09-21T10:55:01.465847+0000","last_active":"2024-09-21T10:55:03.353288+0000","last_peered":"2024-09-21T10:55:03.353288+0000","last_clean":"2024-09-21T10:55:03.353288+0000","last_became_active":"2024-09-21T10:55:01.464407+0000","last_became_peered":"2024-09-21T10:55:01.464407+0000","last_unstale":"2024-09-21T10:55:03.353288+0000","last_undegraded":"2024-09-21T10:55:03.353288+0000","last_fullsized":"2024-09-21T10:55:03.353288+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:59:36.163714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2302,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.509904+0000","last_change":"2024-09-21T10:52:25.498554+0000","last_active":"2024-09-21T10:55:12.509904+0000","last_peered":"2024-09-21T10:55:12.509904+0000","last_clean":"2024-09-21T10:55:12.509904+0000","last_became_active":"2024-09-21T10:52:25.498194+0000","last_became_peered":"2024-09-21T10:52:25.498194+0000","last_unstale":"2024-09-21T10:55:12.509904+0000","last_undegraded":"2024-09-21T10:55:12.509904+0000","last_fullsized":"2024-09-21T10:55:12.509904+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:47.115271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":41,"reported_epoch":1494,"state":"active+clean","last_fresh":"2024-09-21T10:55:03.350899+0000","last_change":"2024-09-21T10:52:25.489063+0000","last_active":"2024-09-21T10:55:03.350899+0000","last_peered":"2024-09-21T10:55:03.350899+0000","last_clean":"2024-09-21T10:55:03.350899+0000","last_became_active":"2024-09-21T10:52:25.488129+0000","last_became_peered":"2024-09-21T10:52:25.488129+0000","last_unstale":"2024-09-21T10:55:03.350899+0000","last_undegraded":"2024-09-21T10:55:03.350899+0000","last_fullsized":"2024-09-21T10:55:03.350899+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:36.996829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1455,"reported_epoch":1494,"state":"active+clean","last_fresh":"2024-09-21T10:55:03.351978+0000","last_change":"2024-09-21T10:49:44.864898+0000","last_active":"2024-09-21T10:55:03.351978+0000","last_peered":"2024-09-21T10:55:03.351978+0000","last_clean":"2024-09-21T10:55:03.351978+0000","last_became_active":"2024-09-21T10:49:44.862691+0000","last_became_peered":"2024-09-21T10:49:44.862691+0000","last_unstale":"2024-09-21T10:55:03.351978+0000","last_undegraded":"2024-09-21T10:55:03.351978+0000","last_fullsized":"2024-09-21T10:55:03.351978+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:57.809113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":161,"reported_epoch":1494,"state":"active+clean","last_fresh":"2024-09-21T10:55:03.351275+0000","last_change":"2024-09-21T10:55:01.498736+0000","last_active":"2024-09-21T10:55:03.351275+0000","last_peered":"2024-09-21T10:55:03.351275+0000","last_clean":"2024-09-21T10:55:03.351275+0000","last_became_active":"2024-09-21T10:55:01.475099+0000","last_became_peered":"2024-09-21T10:55:01.475099+0000","last_unstale":"2024-09-21T10:55:03.351275+0000","last_undegraded":"2024-09-21T10:55:03.351275+0000","last_fullsized":"2024-09-21T10:55:03.351275+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:51:50.219969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2283,"reported_epoch":1496,"state":"active+clean","last_fresh":"2024-09-21T10:55:12.511334+0000","last_change":"2024-09-21T10:44:14.398584+0000","last_active":"2024-09-21T10:55:12.511334+0000","last_peered":"2024-09-21T10:55:12.511334+0000","last_clean":"2024-09-21T10:55:12.511334+0000","last_became_active":"2024-09-21T10:44:14.398278+0000","last_became_peered":"2024-09-21T10:44:14.398278+0000","last_unstale":"2024-09-21T10:55:12.511334+0000","last_undegraded":"2024-09-21T10:55:12.511334+0000","last_fullsized":"2024-09-21T10:55:12.511334+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:19.681750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":56,"num_pg":16,"stat_sum":{"num_bytes":5024317440,"num_objects":76665,"num_object_clones":0,"num_object_copies":229995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33576,"num_objects_misplaced":7639,"num_objects_unfound":0,"num_objects_dirty":76665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76665,"num_write_kb":4906560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":38797312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15225454592,"data_stored":15225454592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65646,"ondisk_log_size":65646,"up":48,"acting":38,"num_store_stats":5},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":268350,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":173,"num_read_kb":375,"num_write":99,"num_write_kb":1099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3333191,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2641920,"data_stored":2624160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":33,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":12072,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":23492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234734,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3999008,"kb_used_data":3350232,"kb_used_omap":1,"kb_used_meta":648766,"kb_avail":89746144,"statfs":{"total":95995035648,"available":91900051456,"internally_reserved":0,"allocated":3430637568,"data_stored":3425477623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":664336843},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496372,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4782196,"kb_used_data":3938988,"kb_used_omap":1,"kb_used_meta":843198,"kb_avail":88962956,"statfs":{"total":95995035648,"available":91098066944,"internally_reserved":0,"allocated":4033523712,"data_stored":4028352772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":863435210},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208758009,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019645,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369388,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3249336,"kb_used_data":2360620,"kb_used_omap":69,"kb_used_meta":888634,"kb_avail":90495816,"statfs":{"total":95995035648,"available":92667715584,"internally_reserved":0,"allocated":2417274880,"data_stored":2412103940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":909961346},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2920577761843,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807958074,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3273888,"kb_used_data":2506072,"kb_used_omap":66,"kb_used_meta":767741,"kb_avail":90471264,"statfs":{"total":95995035648,"available":92642574336,"internally_reserved":0,"allocated":2566217728,"data_stored":2561061804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68589,"internal_metadata":786166803},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":2860448219713,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2568355840,"data_stored":2568355840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2424176640,"data_stored":2424176640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2739863552,"data_stored":2739863552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4042063872,"data_stored":4042063872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3450994688,"data_stored":3450994688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:55:23.830 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T10:55:23.830 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_3 pg_num 26 2024-09-21T10:55:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:23 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 33576/241896 objects degraded (13.880%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:23.864 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:55:23.874 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:23.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:23 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 33576/241896 objects degraded (13.880%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:23.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:23 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 33576/241896 objects degraded (13.880%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:23.919 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:23.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:23.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:24.021 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:24.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:24.072 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:24.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:24.173 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:24.199 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:24.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:24.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:24.301 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:24.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:24.344 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:24.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:24.445 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:24.478 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:24.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:24.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:24.558 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 58 15 86115 86100 92.7638 51.25 0.0183343 0.0107749 2024-09-21T10:55:24.580 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:24.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:24.613 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:24.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:24.703 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:24 smithi070 ceph-mon[112139]: pgmap v2791: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.0 GiB data, 16 GiB used, 342 GiB / 358 GiB avail; 53 MiB/s wr, 852 op/s; 33576/247737 objects degraded (13.553%); 7197/247737 objects misplaced (2.905%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:55:24.704 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:24 smithi070 ceph-mon[107385]: pgmap v2791: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.0 GiB data, 16 GiB used, 342 GiB / 358 GiB avail; 53 MiB/s wr, 852 op/s; 33576/247737 objects degraded (13.553%); 7197/247737 objects misplaced (2.905%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:55:24.714 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:24.727 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:55:24.728 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:55:24.743 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:55:24.744 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:55:24.744 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:24.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:24.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:24.783 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:55:24.783 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:55:24.818 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:55:24.818 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:55:24.846 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:24.859 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:55:24.859 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:55:24.880 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:24.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:24.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:24.887 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:55:24.887 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:55:24.919 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:55:24.919 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:55:24.962 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:55:24.962 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:55:24.982 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:25.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:25.029 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:25.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:25.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:24 smithi160 ceph-mon[101879]: pgmap v2791: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.0 GiB data, 16 GiB used, 342 GiB / 358 GiB avail; 53 MiB/s wr, 852 op/s; 33576/247737 objects degraded (13.553%); 7197/247737 objects misplaced (2.905%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:55:25.130 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:25.162 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:25.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:25.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:25.263 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:25.291 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:25.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:25.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:25.393 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:25.428 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:25.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:25.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:25.530 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:25.558 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 59 16 86941 86925 92.0655 51.5625 0.0296558 0.0108555 2024-09-21T10:55:25.564 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:25.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:25.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:25.666 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:25.691 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:25.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:25.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:25.793 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:25.810 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:55:25.826 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:25.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:25.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:25.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:25 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2727206172' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:55:25.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:25 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2727206172' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:55:25.928 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:25.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:25.970 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:25.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:26.071 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:26.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:25 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2727206172' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:55:26.106 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:26.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:26.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:26.207 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:26.244 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:26.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:26.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:26.346 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:26.387 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:26.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:26.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:26.490 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:26.534 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:26.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:26.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:26.558 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:2024-09-21T10:55:26.557151+0000 min lat: 0.00129408 max lat: 2.31521 avg lat: 0.010937 2024-09-21T10:55:26.558 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:55:26.558 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 60 16 87759 87743 91.3831 51.125 0.0510261 0.010937 2024-09-21T10:55:26.635 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:26.672 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:26.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:26.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:26.769 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:26 smithi070 ceph-mon[107385]: pgmap v2792: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.2 GiB data, 16 GiB used, 342 GiB / 358 GiB avail; 59 MiB/s wr, 944 op/s; 33576/255006 objects degraded (13.167%); 7044/255006 objects misplaced (2.762%); 3.9 MiB/s, 62 objects/s recovering 2024-09-21T10:55:26.769 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:26 smithi070 ceph-mon[112139]: pgmap v2792: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.2 GiB data, 16 GiB used, 342 GiB / 358 GiB avail; 59 MiB/s wr, 944 op/s; 33576/255006 objects degraded (13.167%); 7044/255006 objects misplaced (2.762%); 3.9 MiB/s, 62 objects/s recovering 2024-09-21T10:55:26.774 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:26.809 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:26.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:26.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:26.911 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:26.953 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:26.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:26.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:27.055 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:27.082 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:55:27.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:26 smithi160 ceph-mon[101879]: pgmap v2792: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.2 GiB data, 16 GiB used, 342 GiB / 358 GiB avail; 59 MiB/s wr, 944 op/s; 33576/255006 objects degraded (13.167%); 7044/255006 objects misplaced (2.762%); 3.9 MiB/s, 62 objects/s recovering 2024-09-21T10:55:27.125 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:27.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:27.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:27.227 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:27.259 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:27.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:27.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:27.361 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:27.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:27.398 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:27.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:27.499 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:27.541 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:27.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:27.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:27.558 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 61 16 88631 88615 90.7784 54.5 0.00966431 0.0110107 2024-09-21T10:55:27.643 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:27.688 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:27.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:27.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:27.789 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:27.823 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:27.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:27.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:27.925 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:27.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:27.964 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:27.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:28.065 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:28.105 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:28.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:28.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:28.207 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:28.244 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:28.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:28.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:28.346 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:28.384 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:55:28.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:28.389 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:28.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:28.491 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:28.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:28.526 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:28.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:28.558 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 62 16 89497 89481 90.1872 54.125 0.0132391 0.0110825 2024-09-21T10:55:28.619 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T10:55:28.627 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:28.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:28.692 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:28.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:28.792 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:28.822 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:28.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:28.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:28.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:28 smithi070 ceph-mon[112139]: pgmap v2793: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.2 GiB data, 16 GiB used, 341 GiB / 358 GiB avail; 49 MiB/s wr, 783 op/s; 33576/256596 objects degraded (13.085%); 7044/256596 objects misplaced (2.745%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T10:55:28.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:28 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 33576/256596 objects degraded (13.085%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:28.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:28 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2699505773' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:55:28.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:28 smithi070 ceph-mon[107385]: pgmap v2793: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.2 GiB data, 16 GiB used, 341 GiB / 358 GiB avail; 49 MiB/s wr, 783 op/s; 33576/256596 objects degraded (13.085%); 7044/256596 objects misplaced (2.745%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T10:55:28.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:28 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 33576/256596 objects degraded (13.085%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:28.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:28 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2699505773' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:55:28.924 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:28.966 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:28.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:28.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:29.068 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:29.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:28 smithi160 ceph-mon[101879]: pgmap v2793: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.2 GiB data, 16 GiB used, 341 GiB / 358 GiB avail; 49 MiB/s wr, 783 op/s; 33576/256596 objects degraded (13.085%); 7044/256596 objects misplaced (2.745%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T10:55:29.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:28 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 33576/256596 objects degraded (13.085%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:29.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:28 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2699505773' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:55:29.103 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:29.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:29.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:29.205 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:29.240 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:29.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:29.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:29.341 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:29.377 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:29.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:29.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:29.479 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:29.513 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:29.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:29.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:29.558 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 63 16 90376 90360 89.6277 54.9375 0.0127883 0.0111524 2024-09-21T10:55:29.615 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:29.649 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:29.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:29.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:29.751 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:29.790 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:29.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:29.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:29.856 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:29 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:55:29.856 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:29 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2699505773' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:55:29.856 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:29 smithi070 ceph-mon[112139]: osdmap e1497: 8 total, 8 up, 4 in 2024-09-21T10:55:29.856 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:29 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:55:29.857 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:29 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2699505773' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:55:29.857 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:29 smithi070 ceph-mon[107385]: osdmap e1497: 8 total, 8 up, 4 in 2024-09-21T10:55:29.892 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:29.941 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:29.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:29.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:30.044 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:30.081 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:30.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:30.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:30.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:29 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:55:30.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:29 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2699505773' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:55:30.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:29 smithi160 ceph-mon[101879]: osdmap e1497: 8 total, 8 up, 4 in 2024-09-21T10:55:30.184 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:30.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:30.227 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:30.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:30.328 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:30.362 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:30.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:30.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:30.464 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:30.496 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:30.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:30.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:30.546 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:55:30.559 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 64 16 91299 91283 89.1267 57.6875 0.0334675 0.0112141 2024-09-21T10:55:30.598 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:30.633 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:30.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:30.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:30.735 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:30.773 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:30.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:30.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:30.875 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:30.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:30 smithi070 ceph-mon[107385]: pgmap v2795: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.4 GiB data, 17 GiB used, 341 GiB / 358 GiB avail; 55 MiB/s wr, 872 op/s; 33576/263814 objects degraded (12.727%); 6738/263814 objects misplaced (2.554%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:55:30.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:30 smithi070 ceph-mon[112139]: pgmap v2795: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.4 GiB data, 17 GiB used, 341 GiB / 358 GiB avail; 55 MiB/s wr, 872 op/s; 33576/263814 objects degraded (12.727%); 6738/263814 objects misplaced (2.554%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:55:30.910 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:30.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:30.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:31.012 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:31.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:31.045 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:31.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:31.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:30 smithi160 ceph-mon[101879]: pgmap v2795: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.4 GiB data, 17 GiB used, 341 GiB / 358 GiB avail; 55 MiB/s wr, 872 op/s; 33576/263814 objects degraded (12.727%); 6738/263814 objects misplaced (2.554%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:55:31.145 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:31.191 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:31.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:31.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:31.293 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:31.339 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:31.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:31.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:31.441 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:31.475 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:31.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:31.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:31.560 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 65 16 92121 92105 88.5458 51.375 0.0317281 0.0112883 2024-09-21T10:55:31.577 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:31.606 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:31.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:31.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:31.707 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:31.755 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:31.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:31.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:31.858 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:31.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:31.892 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:31.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:31.899 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:55:31.994 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:32.039 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:32.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:32.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:32.141 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:32.182 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:32.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:32.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:32.283 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:32.314 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:32.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:32.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:32.416 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:32.455 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:32.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:32.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:32.557 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:32.562 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 66 16 93010 92994 88.0461 55.5625 0.0206059 0.0113521 2024-09-21T10:55:32.596 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:32.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:32.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:32.698 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:32.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:32.735 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:32.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:32.836 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:32.865 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:32.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:32.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:32.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:32 smithi070 ceph-mon[107385]: pgmap v2796: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.4 GiB data, 17 GiB used, 341 GiB / 358 GiB avail; 53 MiB/s wr, 855 op/s; 33576/267576 objects degraded (12.548%); 6738/267576 objects misplaced (2.518%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:55:32.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:32 smithi070 ceph-mon[112139]: pgmap v2796: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.4 GiB data, 17 GiB used, 341 GiB / 358 GiB avail; 53 MiB/s wr, 855 op/s; 33576/267576 objects degraded (12.548%); 6738/267576 objects misplaced (2.518%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:55:32.967 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:33.010 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:33.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:33.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:33.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:32 smithi160 ceph-mon[101879]: pgmap v2796: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.4 GiB data, 17 GiB used, 341 GiB / 358 GiB avail; 53 MiB/s wr, 855 op/s; 33576/267576 objects degraded (12.548%); 6738/267576 objects misplaced (2.518%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:55:33.112 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:33.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:33.149 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:33.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:33.250 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:33.272 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:33.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:33.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:33.373 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:33.395 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:33.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:33.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:33.497 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:33.528 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:33.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:33.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:33.551 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:55:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:55:33] "GET /metrics HTTP/1.1" 200 36626 "" "Prometheus/2.51.0" 2024-09-21T10:55:33.560 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 67 16 93598 93582 87.2805 36.75 0.0276223 0.0114513 2024-09-21T10:55:33.630 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:33.676 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:33.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:33.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:33.778 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:33.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:33.811 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:33.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:33.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:33 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 33576/267576 objects degraded (12.548%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:33 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 33576/267576 objects degraded (12.548%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:33.913 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:33.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:33.948 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:33.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:34.049 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:34.076 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:34.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:34.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:34.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:33 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 33576/267576 objects degraded (12.548%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:34.177 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:34.216 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:34.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:34.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:34.318 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:34.362 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:34.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:34.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:34.464 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:34.497 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:34.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:34.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:34.560 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 68 16 94488 94472 86.815 55.625 0.0179207 0.0115131 2024-09-21T10:55:34.598 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:34.649 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:34.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:34.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:34.751 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:34.781 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:34.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:34.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:34.862 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:34 smithi070 ceph-mon[107385]: pgmap v2797: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.6 GiB data, 17 GiB used, 340 GiB / 358 GiB avail; 54 MiB/s wr, 859 op/s; 33576/273519 objects degraded (12.276%); 6591/273519 objects misplaced (2.410%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:55:34.863 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:34 smithi070 ceph-mon[112139]: pgmap v2797: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.6 GiB data, 17 GiB used, 340 GiB / 358 GiB avail; 54 MiB/s wr, 859 op/s; 33576/273519 objects degraded (12.276%); 6591/273519 objects misplaced (2.410%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:55:34.883 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:34.926 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:34.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:34.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:35.028 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:35.060 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:35.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:35.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:35.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:34 smithi160 ceph-mon[101879]: pgmap v2797: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.6 GiB data, 17 GiB used, 340 GiB / 358 GiB avail; 54 MiB/s wr, 859 op/s; 33576/273519 objects degraded (12.276%); 6591/273519 objects misplaced (2.410%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:55:35.162 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:35.197 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:35.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:35.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:35.299 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:35.338 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:35.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:35.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:35.424 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:55:35.425 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T10:55:35.425 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T10:55:35.425 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T10:55:35.425 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.425 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:75997 55.9 55:9df18c11:::benchmark_data_smithi070_294394_object75996:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1426)", 2024-09-21T10:55:35.425 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:49:43.845377+0000", 2024-09-21T10:55:35.425 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 350.93866341699999, 2024-09-21T10:55:35.425 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3307690619999999, 2024-09-21T10:55:35.425 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:55:35.425 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:55:35.425 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:55:35.425 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:55:35.426 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:55:35.426 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:35.426 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 75997 2024-09-21T10:55:35.426 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.426 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:55:35.426 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.426 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:55:35.426 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.845377+0000", 2024-09-21T10:55:35.426 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.426 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.426 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.426 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:55:35.426 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.845377+0000", 2024-09-21T10:55:35.427 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.427 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.427 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.427 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:55:35.427 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.845378+0000", 2024-09-21T10:55:35.427 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.457e-06 2024-09-21T10:55:35.427 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.427 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.427 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:55:35.427 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.845402+0000", 2024-09-21T10:55:35.427 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3493e-05 2024-09-21T10:55:35.427 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.427 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.427 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:55:35.427 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.845414+0000", 2024-09-21T10:55:35.428 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2119999999999999e-05 2024-09-21T10:55:35.428 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.428 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.428 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:55:35.428 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.845422+0000", 2024-09-21T10:55:35.428 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6389999999999992e-06 2024-09-21T10:55:35.428 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.428 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.428 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.428 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.846209+0000", 2024-09-21T10:55:35.428 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00078733999999999998 2024-09-21T10:55:35.428 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.428 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.428 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:55:35.429 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.846319+0000", 2024-09-21T10:55:35.429 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000109648 2024-09-21T10:55:35.429 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.429 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.429 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:55:35.429 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.846443+0000", 2024-09-21T10:55:35.429 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012475 2024-09-21T10:55:35.429 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.429 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.429 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:55:35.429 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.853363+0000", 2024-09-21T10:55:35.429 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0069197119999999997 2024-09-21T10:55:35.429 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.429 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.430 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.430 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.857141+0000", 2024-09-21T10:55:35.430 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0037782850000000002 2024-09-21T10:55:35.430 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.430 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.430 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.430 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.868536+0000", 2024-09-21T10:55:35.430 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.011394824 2024-09-21T10:55:35.430 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.430 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.430 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.430 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.868546+0000", 2024-09-21T10:55:35.430 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0088000000000001e-05 2024-09-21T10:55:35.430 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.431 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.431 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.431 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.854392+0000", 2024-09-21T10:55:35.431 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.98584531900000005 2024-09-21T10:55:35.431 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.431 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.431 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.431 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.854406+0000", 2024-09-21T10:55:35.431 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.395e-05 2024-09-21T10:55:35.431 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.431 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.431 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.431 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.155653+0000", 2024-09-21T10:55:35.431 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.301247364 2024-09-21T10:55:35.432 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.432 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.432 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:55:35.432 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.156148+0000", 2024-09-21T10:55:35.432 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00049496399999999997 2024-09-21T10:55:35.432 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.432 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.432 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.432 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.176031+0000", 2024-09-21T10:55:35.432 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.019882983999999999 2024-09-21T10:55:35.432 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.432 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.432 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:55:35.432 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.176146+0000", 2024-09-21T10:55:35.433 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011512500000000001 2024-09-21T10:55:35.433 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.433 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:55:35.433 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.433 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.433 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.433 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:76007 55.9 55:9d80d777:::benchmark_data_smithi070_294394_object76006:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1427)", 2024-09-21T10:55:35.433 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:49:43.867005+0000", 2024-09-21T10:55:35.433 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 350.91703508000001, 2024-09-21T10:55:35.433 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3174965109999999, 2024-09-21T10:55:35.433 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:55:35.433 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:55:35.433 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:35.433 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:55:35.434 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:55:35.434 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:35.434 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 76007 2024-09-21T10:55:35.434 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.434 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:55:35.434 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.434 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:55:35.434 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.867005+0000", 2024-09-21T10:55:35.434 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.434 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.434 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.434 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:55:35.434 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.867005+0000", 2024-09-21T10:55:35.434 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.435 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.435 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.435 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:55:35.435 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.867006+0000", 2024-09-21T10:55:35.435 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6600000000000005e-07 2024-09-21T10:55:35.435 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.435 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.435 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:55:35.435 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.867035+0000", 2024-09-21T10:55:35.435 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8377e-05 2024-09-21T10:55:35.435 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.435 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.435 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:55:35.435 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.867043+0000", 2024-09-21T10:55:35.435 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9260000000000006e-06 2024-09-21T10:55:35.436 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.436 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.436 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:55:35.436 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.867050+0000", 2024-09-21T10:55:35.436 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1360000000000002e-06 2024-09-21T10:55:35.436 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.436 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.436 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.436 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.868571+0000", 2024-09-21T10:55:35.436 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0015209119999999999 2024-09-21T10:55:35.436 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.436 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.436 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.436 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.868577+0000", 2024-09-21T10:55:35.437 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0360000000000003e-06 2024-09-21T10:55:35.437 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.437 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.437 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.437 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.854433+0000", 2024-09-21T10:55:35.437 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.98585627200000003 2024-09-21T10:55:35.437 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.437 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.437 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.438 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.854440+0000", 2024-09-21T10:55:35.438 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1980000000000004e-06 2024-09-21T10:55:35.438 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.438 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.438 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.438 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.161898+0000", 2024-09-21T10:55:35.438 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.307458191 2024-09-21T10:55:35.438 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.438 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.438 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:55:35.439 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.162049+0000", 2024-09-21T10:55:35.439 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000150543 2024-09-21T10:55:35.439 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.439 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.439 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:55:35.439 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.162454+0000", 2024-09-21T10:55:35.439 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00040488999999999998 2024-09-21T10:55:35.439 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.439 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.439 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.439 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.176384+0000", 2024-09-21T10:55:35.439 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.013930840999999999 2024-09-21T10:55:35.439 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.440 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.440 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.440 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.176460+0000", 2024-09-21T10:55:35.440 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5462000000000004e-05 2024-09-21T10:55:35.440 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.440 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.440 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:55:35.440 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.180350+0000", 2024-09-21T10:55:35.440 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0038902300000000002 2024-09-21T10:55:35.440 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.440 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.440 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:55:35.440 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.180415+0000", 2024-09-21T10:55:35.440 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4327999999999998e-05 2024-09-21T10:55:35.441 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.441 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.441 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:55:35.441 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.184502+0000", 2024-09-21T10:55:35.441 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0040873029999999996 2024-09-21T10:55:35.441 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.441 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:55:35.441 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.441 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.441 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.441 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:76016 55.9 55:93835e2d:::benchmark_data_smithi070_294394_object76015:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1427)", 2024-09-21T10:55:35.441 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:49:43.877052+0000", 2024-09-21T10:55:35.441 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 350.90698844500002, 2024-09-21T10:55:35.441 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3249626640000001, 2024-09-21T10:55:35.442 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:55:35.442 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:55:35.442 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:35.442 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:55:35.442 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:55:35.442 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:35.442 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 76016 2024-09-21T10:55:35.442 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.442 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:55:35.442 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.443 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:55:35.443 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.877052+0000", 2024-09-21T10:55:35.443 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.443 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.443 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.443 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:55:35.443 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.877052+0000", 2024-09-21T10:55:35.444 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.444 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.444 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.444 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:55:35.444 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.877053+0000", 2024-09-21T10:55:35.444 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4410000000000001e-06 2024-09-21T10:55:35.444 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.444 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.444 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:55:35.445 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.877076+0000", 2024-09-21T10:55:35.445 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2129000000000001e-05 2024-09-21T10:55:35.445 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.445 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.445 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:55:35.445 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.877086+0000", 2024-09-21T10:55:35.445 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.027e-05 2024-09-21T10:55:35.445 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.445 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.445 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:55:35.446 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.877093+0000", 2024-09-21T10:55:35.446 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9650000000000002e-06 2024-09-21T10:55:35.446 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.446 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.446 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.446 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.877155+0000", 2024-09-21T10:55:35.446 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1878999999999995e-05 2024-09-21T10:55:35.446 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.447 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.447 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.447 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.877165+0000", 2024-09-21T10:55:35.447 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0406e-05 2024-09-21T10:55:35.447 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.447 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.447 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.447 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.854464+0000", 2024-09-21T10:55:35.448 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.97729942000000003 2024-09-21T10:55:35.448 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.448 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.448 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.448 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.854471+0000", 2024-09-21T10:55:35.448 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9979999999999997e-06 2024-09-21T10:55:35.448 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.448 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.449 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.449 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.176493+0000", 2024-09-21T10:55:35.449 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.322021382 2024-09-21T10:55:35.449 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.449 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.449 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:55:35.450 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.176648+0000", 2024-09-21T10:55:35.450 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015506799999999999 2024-09-21T10:55:35.450 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.450 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.450 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:55:35.450 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.176801+0000", 2024-09-21T10:55:35.450 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015295700000000001 2024-09-21T10:55:35.451 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.451 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.451 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.451 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.184582+0000", 2024-09-21T10:55:35.451 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0077810839999999997 2024-09-21T10:55:35.451 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.451 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.451 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:55:35.452 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.192506+0000", 2024-09-21T10:55:35.452 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0079242640000000003 2024-09-21T10:55:35.452 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.452 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.452 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.452 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.198152+0000", 2024-09-21T10:55:35.452 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0056459459999999998 2024-09-21T10:55:35.452 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.453 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.453 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:55:35.453 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.198203+0000", 2024-09-21T10:55:35.453 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0401000000000001e-05 2024-09-21T10:55:35.453 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.453 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.453 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:55:35.454 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.202015+0000", 2024-09-21T10:55:35.454 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0038120540000000001 2024-09-21T10:55:35.454 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.454 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:55:35.454 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.454 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.454 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.455 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:76033 55.9 55:996cb394:::benchmark_data_smithi070_294394_object76032:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1427)", 2024-09-21T10:55:35.455 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:49:43.889361+0000", 2024-09-21T10:55:35.455 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 350.89467964099998, 2024-09-21T10:55:35.455 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.319872497, 2024-09-21T10:55:35.455 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:55:35.455 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:55:35.456 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:35.456 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:55:35.456 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:55:35.456 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:35.456 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 76033 2024-09-21T10:55:35.456 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.456 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:55:35.456 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.456 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:55:35.456 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.889361+0000", 2024-09-21T10:55:35.456 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.456 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.457 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.457 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:55:35.457 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.889361+0000", 2024-09-21T10:55:35.457 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.457 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.457 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.457 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:55:35.457 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.889363+0000", 2024-09-21T10:55:35.457 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9309999999999998e-06 2024-09-21T10:55:35.457 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.457 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.457 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:55:35.457 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.889403+0000", 2024-09-21T10:55:35.457 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9892999999999999e-05 2024-09-21T10:55:35.457 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.458 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.458 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:55:35.458 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.889424+0000", 2024-09-21T10:55:35.458 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1866e-05 2024-09-21T10:55:35.458 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.458 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.458 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:55:35.458 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.889434+0000", 2024-09-21T10:55:35.458 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9019999999999994e-06 2024-09-21T10:55:35.458 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.458 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.458 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.459 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.889525+0000", 2024-09-21T10:55:35.459 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.111e-05 2024-09-21T10:55:35.459 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.459 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.459 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.459 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.889542+0000", 2024-09-21T10:55:35.459 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6654999999999999e-05 2024-09-21T10:55:35.459 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.459 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.459 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.460 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.854495+0000", 2024-09-21T10:55:35.460 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.96495315500000001 2024-09-21T10:55:35.460 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.460 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.460 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.460 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.854503+0000", 2024-09-21T10:55:35.460 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.379e-06 2024-09-21T10:55:35.460 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.460 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.460 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.460 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.184620+0000", 2024-09-21T10:55:35.460 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.330117459 2024-09-21T10:55:35.460 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.460 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.461 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:55:35.461 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.184796+0000", 2024-09-21T10:55:35.461 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017597700000000001 2024-09-21T10:55:35.461 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.461 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.461 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:55:35.461 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.184971+0000", 2024-09-21T10:55:35.461 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000174532 2024-09-21T10:55:35.461 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.461 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.461 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.461 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.198044+0000", 2024-09-21T10:55:35.461 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.013073316999999999 2024-09-21T10:55:35.461 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.462 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.462 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.462 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.202122+0000", 2024-09-21T10:55:35.462 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0040778999999999998 2024-09-21T10:55:35.462 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.462 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.462 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:55:35.462 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.205219+0000", 2024-09-21T10:55:35.462 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0030968200000000001 2024-09-21T10:55:35.462 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.462 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.462 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:55:35.462 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.205285+0000", 2024-09-21T10:55:35.463 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6509999999999998e-05 2024-09-21T10:55:35.463 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.463 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.463 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:55:35.463 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.209233+0000", 2024-09-21T10:55:35.463 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0039480909999999999 2024-09-21T10:55:35.463 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.463 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:55:35.463 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.463 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.463 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.463 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:76035 55.9 55:9c3027e7:::benchmark_data_smithi070_294394_object76034:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1427)", 2024-09-21T10:55:35.463 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:49:43.891615+0000", 2024-09-21T10:55:35.463 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 350.89242582000003, 2024-09-21T10:55:35.464 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3207209400000002, 2024-09-21T10:55:35.464 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:55:35.464 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:55:35.464 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:35.464 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:55:35.464 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:55:35.464 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:35.464 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 76035 2024-09-21T10:55:35.464 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.464 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:55:35.464 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.464 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:55:35.464 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.891615+0000", 2024-09-21T10:55:35.464 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.465 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.465 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.465 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:55:35.465 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.891615+0000", 2024-09-21T10:55:35.465 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.465 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.465 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.465 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:55:35.465 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.891617+0000", 2024-09-21T10:55:35.465 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5270000000000001e-06 2024-09-21T10:55:35.466 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.466 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.466 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:55:35.466 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.891645+0000", 2024-09-21T10:55:35.466 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7416e-05 2024-09-21T10:55:35.466 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.466 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.467 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:55:35.467 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.891663+0000", 2024-09-21T10:55:35.467 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8383e-05 2024-09-21T10:55:35.467 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.467 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.467 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:55:35.467 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.891673+0000", 2024-09-21T10:55:35.467 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9129999999999998e-06 2024-09-21T10:55:35.468 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.468 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.468 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.468 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.891759+0000", 2024-09-21T10:55:35.468 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6024000000000002e-05 2024-09-21T10:55:35.468 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.468 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.469 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.469 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.891770+0000", 2024-09-21T10:55:35.469 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.128e-05 2024-09-21T10:55:35.469 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.469 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.469 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.469 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.854525+0000", 2024-09-21T10:55:35.470 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.96275492200000001 2024-09-21T10:55:35.470 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.470 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.470 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.470 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.854532+0000", 2024-09-21T10:55:35.470 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2579999999999998e-06 2024-09-21T10:55:35.470 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.471 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.471 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.471 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.187401+0000", 2024-09-21T10:55:35.471 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3328687109999999 2024-09-21T10:55:35.471 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.471 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.472 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:55:35.472 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.187565+0000", 2024-09-21T10:55:35.472 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016365800000000001 2024-09-21T10:55:35.472 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.472 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.472 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:55:35.473 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.187729+0000", 2024-09-21T10:55:35.473 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000164613 2024-09-21T10:55:35.473 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.473 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.473 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.473 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.198084+0000", 2024-09-21T10:55:35.473 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.010354515 2024-09-21T10:55:35.474 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.474 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.474 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.474 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.209269+0000", 2024-09-21T10:55:35.474 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.011185179999999999 2024-09-21T10:55:35.474 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.475 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.475 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:55:35.475 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.209333+0000", 2024-09-21T10:55:35.475 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4067999999999997e-05 2024-09-21T10:55:35.475 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.475 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.476 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:55:35.476 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.209366+0000", 2024-09-21T10:55:35.476 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2623999999999998e-05 2024-09-21T10:55:35.476 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.476 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.476 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:55:35.477 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.212336+0000", 2024-09-21T10:55:35.477 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0029698480000000002 2024-09-21T10:55:35.477 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.477 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:55:35.477 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.477 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.478 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.478 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:91267 55.c 55:38532bcb:::benchmark_data_smithi070_294394_object91266:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1433)", 2024-09-21T10:55:35.478 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:50:03.077547+0000", 2024-09-21T10:55:35.478 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 331.70649312299997, 2024-09-21T10:55:35.478 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6119762259999999, 2024-09-21T10:55:35.478 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:55:35.479 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:55:35.479 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:35.479 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:55:35.479 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:55:35.479 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:35.479 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 91267 2024-09-21T10:55:35.480 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.480 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:55:35.480 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.480 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:55:35.480 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.077547+0000", 2024-09-21T10:55:35.480 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.480 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.480 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.481 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:55:35.481 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.077547+0000", 2024-09-21T10:55:35.481 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.481 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.481 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.481 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:55:35.482 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.077549+0000", 2024-09-21T10:55:35.482 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.04e-06 2024-09-21T10:55:35.482 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.482 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.482 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:55:35.482 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.077584+0000", 2024-09-21T10:55:35.482 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4246999999999999e-05 2024-09-21T10:55:35.483 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.483 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.483 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:55:35.483 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.077597+0000", 2024-09-21T10:55:35.483 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3782e-05 2024-09-21T10:55:35.483 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.483 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.483 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:55:35.483 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.077607+0000", 2024-09-21T10:55:35.483 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0057999999999999e-05 2024-09-21T10:55:35.483 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.483 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.484 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.484 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.082132+0000", 2024-09-21T10:55:35.484 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0045246820000000004 2024-09-21T10:55:35.484 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.484 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.484 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.484 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.082143+0000", 2024-09-21T10:55:35.484 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0882e-05 2024-09-21T10:55:35.484 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.484 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.484 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.484 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.251830+0000", 2024-09-21T10:55:35.484 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.169687273 2024-09-21T10:55:35.485 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.485 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.485 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.485 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.251842+0000", 2024-09-21T10:55:35.485 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1489e-05 2024-09-21T10:55:35.485 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.485 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.485 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.485 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.655591+0000", 2024-09-21T10:55:35.485 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4037494939999999 2024-09-21T10:55:35.485 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.485 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.486 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:55:35.486 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.655766+0000", 2024-09-21T10:55:35.486 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017465699999999999 2024-09-21T10:55:35.486 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.486 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.486 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:55:35.486 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.656347+0000", 2024-09-21T10:55:35.486 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00058121799999999999 2024-09-21T10:55:35.486 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.487 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.487 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.487 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.667684+0000", 2024-09-21T10:55:35.487 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.011336458000000001 2024-09-21T10:55:35.487 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.487 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.487 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:55:35.487 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.667702+0000", 2024-09-21T10:55:35.488 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8321999999999998e-05 2024-09-21T10:55:35.488 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.488 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.488 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.488 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.684567+0000", 2024-09-21T10:55:35.488 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.016865394999999998 2024-09-21T10:55:35.488 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.488 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.488 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:55:35.489 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.684627+0000", 2024-09-21T10:55:35.489 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9630000000000003e-05 2024-09-21T10:55:35.489 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.489 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.489 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:55:35.489 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.689523+0000", 2024-09-21T10:55:35.489 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0048965989999999997 2024-09-21T10:55:35.489 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.489 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:55:35.490 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.490 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.490 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.490 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:91276 55.0 55:068d4ef6:::benchmark_data_smithi070_294394_object91275:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1433)", 2024-09-21T10:55:35.490 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:50:03.092500+0000", 2024-09-21T10:55:35.490 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 331.69154073999999, 2024-09-21T10:55:35.490 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5339436580000001, 2024-09-21T10:55:35.491 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:55:35.491 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:55:35.491 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:35.491 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:55:35.491 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:55:35.491 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:35.491 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 91276 2024-09-21T10:55:35.491 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.491 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:55:35.492 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.492 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:55:35.492 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.092500+0000", 2024-09-21T10:55:35.492 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.492 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.492 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.492 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:55:35.492 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.092500+0000", 2024-09-21T10:55:35.493 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.493 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.493 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.493 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:55:35.493 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.092501+0000", 2024-09-21T10:55:35.493 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4300000000000001e-06 2024-09-21T10:55:35.493 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.493 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.493 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:55:35.494 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.092522+0000", 2024-09-21T10:55:35.494 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1047e-05 2024-09-21T10:55:35.494 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.494 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.494 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:55:35.494 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.092536+0000", 2024-09-21T10:55:35.494 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3746e-05 2024-09-21T10:55:35.494 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.495 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.495 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:55:35.495 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.092543+0000", 2024-09-21T10:55:35.495 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2359999999999998e-06 2024-09-21T10:55:35.495 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.495 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.495 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.495 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.092605+0000", 2024-09-21T10:55:35.496 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2049000000000002e-05 2024-09-21T10:55:35.496 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.496 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.496 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.496 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.092636+0000", 2024-09-21T10:55:35.496 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0465999999999999e-05 2024-09-21T10:55:35.496 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.496 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.496 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.497 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.247994+0000", 2024-09-21T10:55:35.497 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.15535863599999999 2024-09-21T10:55:35.497 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.497 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.497 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.497 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.248003+0000", 2024-09-21T10:55:35.497 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.037e-06 2024-09-21T10:55:35.497 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.497 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.498 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.498 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.598306+0000", 2024-09-21T10:55:35.498 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3503028340000001 2024-09-21T10:55:35.498 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.498 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.498 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:55:35.498 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.598777+0000", 2024-09-21T10:55:35.498 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000470583 2024-09-21T10:55:35.499 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.499 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.499 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:55:35.499 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.599321+0000", 2024-09-21T10:55:35.499 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00054458600000000001 2024-09-21T10:55:35.499 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.499 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.499 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:55:35.500 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.607219+0000", 2024-09-21T10:55:35.500 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0078978009999999994 2024-09-21T10:55:35.500 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.500 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.500 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.500 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.611017+0000", 2024-09-21T10:55:35.500 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0037974800000000002 2024-09-21T10:55:35.500 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.500 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.500 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.500 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.620289+0000", 2024-09-21T10:55:35.500 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0092722619999999999 2024-09-21T10:55:35.500 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.500 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.501 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:55:35.501 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.620530+0000", 2024-09-21T10:55:35.501 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00024090599999999999 2024-09-21T10:55:35.501 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.501 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.501 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:55:35.501 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.626443+0000", 2024-09-21T10:55:35.501 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0059135589999999997 2024-09-21T10:55:35.501 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.501 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:55:35.501 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.501 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.501 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.502 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:91278 55.0 55:030c66b6:::benchmark_data_smithi070_294394_object91277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1433)", 2024-09-21T10:55:35.502 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:50:03.092610+0000", 2024-09-21T10:55:35.502 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 331.69143058999998, 2024-09-21T10:55:35.502 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5441410179999999, 2024-09-21T10:55:35.502 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:55:35.502 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:55:35.502 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:35.502 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:55:35.502 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:55:35.502 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:35.502 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 91278 2024-09-21T10:55:35.502 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.503 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:55:35.503 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.503 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:55:35.503 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.092610+0000", 2024-09-21T10:55:35.503 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.503 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.503 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.503 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:55:35.503 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.092610+0000", 2024-09-21T10:55:35.503 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.503 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.503 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.503 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:55:35.503 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.092610+0000", 2024-09-21T10:55:35.504 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5799999999999999e-07 2024-09-21T10:55:35.504 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.504 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.504 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:55:35.504 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.092631+0000", 2024-09-21T10:55:35.504 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0520999999999999e-05 2024-09-21T10:55:35.504 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.504 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.504 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:55:35.504 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.092639+0000", 2024-09-21T10:55:35.504 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1389999999999995e-06 2024-09-21T10:55:35.504 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.504 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.504 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:55:35.505 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.092645+0000", 2024-09-21T10:55:35.505 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0650000000000004e-06 2024-09-21T10:55:35.505 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.505 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.505 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.505 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.092680+0000", 2024-09-21T10:55:35.505 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4718999999999997e-05 2024-09-21T10:55:35.505 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.505 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.505 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.505 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.092686+0000", 2024-09-21T10:55:35.505 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5710000000000003e-06 2024-09-21T10:55:35.505 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.505 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.506 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.506 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.248027+0000", 2024-09-21T10:55:35.506 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.15534089100000001 2024-09-21T10:55:35.506 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.506 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.506 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.506 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.248035+0000", 2024-09-21T10:55:35.506 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9370000000000004e-06 2024-09-21T10:55:35.506 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.506 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.506 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.506 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.611058+0000", 2024-09-21T10:55:35.506 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3630223539999999 2024-09-21T10:55:35.507 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.507 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.507 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:55:35.507 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.611248+0000", 2024-09-21T10:55:35.507 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000189984 2024-09-21T10:55:35.507 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.507 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.507 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:55:35.507 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.611458+0000", 2024-09-21T10:55:35.507 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000210232 2024-09-21T10:55:35.507 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.507 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.507 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.508 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.626526+0000", 2024-09-21T10:55:35.508 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015067888999999999 2024-09-21T10:55:35.508 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.508 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.508 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:55:35.508 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.626543+0000", 2024-09-21T10:55:35.508 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7147999999999999e-05 2024-09-21T10:55:35.508 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.508 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.508 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.508 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.631963+0000", 2024-09-21T10:55:35.508 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0054197020000000002 2024-09-21T10:55:35.508 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.509 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.509 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:55:35.509 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.632304+0000", 2024-09-21T10:55:35.509 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00034152800000000001 2024-09-21T10:55:35.509 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.509 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.509 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:55:35.509 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.636751+0000", 2024-09-21T10:55:35.509 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00444678 2024-09-21T10:55:35.509 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.509 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:55:35.509 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.509 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.509 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.510 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:91284 55.a 55:51dd32cb:::benchmark_data_smithi070_294394_object91283:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1433)", 2024-09-21T10:55:35.510 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:50:03.100842+0000", 2024-09-21T10:55:35.510 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 331.68319841099998, 2024-09-21T10:55:35.510 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.405876949, 2024-09-21T10:55:35.510 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:55:35.510 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:55:35.510 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:35.510 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:55:35.510 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:55:35.510 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:35.510 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 91284 2024-09-21T10:55:35.510 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.510 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:55:35.511 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.511 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:55:35.511 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.100842+0000", 2024-09-21T10:55:35.511 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.511 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.511 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.511 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:55:35.511 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.100842+0000", 2024-09-21T10:55:35.511 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.511 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.511 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.511 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:55:35.511 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.100843+0000", 2024-09-21T10:55:35.511 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2699999999999998e-07 2024-09-21T10:55:35.512 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.512 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.512 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:55:35.512 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.100863+0000", 2024-09-21T10:55:35.512 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0015999999999999e-05 2024-09-21T10:55:35.512 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.512 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.512 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:55:35.512 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.100877+0000", 2024-09-21T10:55:35.512 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4248e-05 2024-09-21T10:55:35.512 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.512 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.512 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:55:35.512 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.100884+0000", 2024-09-21T10:55:35.513 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7800000000000003e-06 2024-09-21T10:55:35.513 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.513 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.513 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.513 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.100954+0000", 2024-09-21T10:55:35.513 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9955000000000004e-05 2024-09-21T10:55:35.513 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.513 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.513 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.513 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.100965+0000", 2024-09-21T10:55:35.513 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1052e-05 2024-09-21T10:55:35.513 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.513 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.513 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.514 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.245704+0000", 2024-09-21T10:55:35.514 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.14473862600000001 2024-09-21T10:55:35.514 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.514 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.514 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.514 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.245714+0000", 2024-09-21T10:55:35.514 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0728e-05 2024-09-21T10:55:35.514 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.514 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.514 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.514 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.487422+0000", 2024-09-21T10:55:35.514 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.241707498 2024-09-21T10:55:35.514 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.515 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.515 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:55:35.515 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.487584+0000", 2024-09-21T10:55:35.515 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000161736 2024-09-21T10:55:35.515 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.515 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.515 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:55:35.515 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.488100+0000", 2024-09-21T10:55:35.515 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000516333 2024-09-21T10:55:35.515 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.515 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.515 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.515 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.496625+0000", 2024-09-21T10:55:35.516 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0085252290000000005 2024-09-21T10:55:35.516 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.516 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.516 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.516 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.501280+0000", 2024-09-21T10:55:35.516 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0046551910000000004 2024-09-21T10:55:35.516 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.516 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.516 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:55:35.516 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.501335+0000", 2024-09-21T10:55:35.516 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4778000000000003e-05 2024-09-21T10:55:35.516 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.516 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.516 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:55:35.517 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.501483+0000", 2024-09-21T10:55:35.517 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014753100000000001 2024-09-21T10:55:35.517 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.517 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.517 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:55:35.517 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.506719+0000", 2024-09-21T10:55:35.517 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0052363210000000004 2024-09-21T10:55:35.517 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.517 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:55:35.517 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.517 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.517 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.517 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:91294 55.a 55:5092edc6:::benchmark_data_smithi070_294394_object91293:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1433)", 2024-09-21T10:55:35.518 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:50:03.114359+0000", 2024-09-21T10:55:35.518 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 331.669681373, 2024-09-21T10:55:35.518 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4016950989999999, 2024-09-21T10:55:35.518 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:55:35.518 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:55:35.518 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:35.518 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:55:35.518 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:55:35.518 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:35.518 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 91294 2024-09-21T10:55:35.518 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.518 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:55:35.518 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.519 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:55:35.519 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.114359+0000", 2024-09-21T10:55:35.519 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.519 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.519 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.519 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:55:35.519 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.114359+0000", 2024-09-21T10:55:35.519 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.519 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.519 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.519 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:55:35.519 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.114360+0000", 2024-09-21T10:55:35.519 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3659999999999999e-06 2024-09-21T10:55:35.520 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.520 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.520 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:55:35.520 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.114394+0000", 2024-09-21T10:55:35.520 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3581999999999998e-05 2024-09-21T10:55:35.520 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.520 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.520 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:55:35.520 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.114409+0000", 2024-09-21T10:55:35.520 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5121e-05 2024-09-21T10:55:35.520 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.520 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.520 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:55:35.520 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.114416+0000", 2024-09-21T10:55:35.521 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0280000000000003e-06 2024-09-21T10:55:35.521 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.521 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.521 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.521 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.114489+0000", 2024-09-21T10:55:35.521 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3150000000000003e-05 2024-09-21T10:55:35.521 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.521 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.521 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.521 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.114499+0000", 2024-09-21T10:55:35.521 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.6269999999999995e-06 2024-09-21T10:55:35.521 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.521 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.521 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.522 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.245736+0000", 2024-09-21T10:55:35.522 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13123684899999999 2024-09-21T10:55:35.522 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.522 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.522 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.522 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.245742+0000", 2024-09-21T10:55:35.522 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7899999999999996e-06 2024-09-21T10:55:35.522 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.522 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.522 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.522 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.496663+0000", 2024-09-21T10:55:35.522 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2509210930000001 2024-09-21T10:55:35.522 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.523 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.523 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:55:35.523 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.496842+0000", 2024-09-21T10:55:35.523 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000179543 2024-09-21T10:55:35.523 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.523 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.523 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:55:35.523 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.497692+0000", 2024-09-21T10:55:35.523 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00084997600000000001 2024-09-21T10:55:35.523 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.523 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.524 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:55:35.524 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.506733+0000", 2024-09-21T10:55:35.524 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0090409240000000005 2024-09-21T10:55:35.524 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.524 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.524 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.524 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.506788+0000", 2024-09-21T10:55:35.524 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5127999999999998e-05 2024-09-21T10:55:35.524 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.524 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.524 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.524 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.510304+0000", 2024-09-21T10:55:35.524 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.003515804 2024-09-21T10:55:35.524 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.525 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.525 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:55:35.525 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.510707+0000", 2024-09-21T10:55:35.525 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00040316199999999999 2024-09-21T10:55:35.525 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.525 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.525 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:55:35.525 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.516054+0000", 2024-09-21T10:55:35.525 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0053469559999999999 2024-09-21T10:55:35.525 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.525 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:55:35.525 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.525 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.526 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.526 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:91303 55.0 55:053c6146:::benchmark_data_smithi070_294394_object91302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1433)", 2024-09-21T10:55:35.526 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:50:03.130601+0000", 2024-09-21T10:55:35.526 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 331.65343917299998, 2024-09-21T10:55:35.526 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5103891630000001, 2024-09-21T10:55:35.526 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:55:35.526 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:55:35.526 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:35.526 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:55:35.526 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:55:35.526 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:35.526 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 91303 2024-09-21T10:55:35.526 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.527 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:55:35.527 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.527 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:55:35.527 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.130601+0000", 2024-09-21T10:55:35.527 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.527 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.527 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.527 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:55:35.527 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.130601+0000", 2024-09-21T10:55:35.527 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.527 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.527 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.527 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:55:35.527 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.130602+0000", 2024-09-21T10:55:35.528 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6499999999999999e-07 2024-09-21T10:55:35.528 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.528 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.528 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:55:35.528 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.130618+0000", 2024-09-21T10:55:35.528 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6572999999999999e-05 2024-09-21T10:55:35.528 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.528 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.528 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:55:35.528 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.130626+0000", 2024-09-21T10:55:35.528 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6450000000000005e-06 2024-09-21T10:55:35.528 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.528 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.528 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:55:35.529 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.130630+0000", 2024-09-21T10:55:35.529 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7809999999999999e-06 2024-09-21T10:55:35.529 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.529 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.529 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.529 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.130685+0000", 2024-09-21T10:55:35.529 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4753000000000002e-05 2024-09-21T10:55:35.529 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.529 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.529 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.529 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.130692+0000", 2024-09-21T10:55:35.529 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7109999999999995e-06 2024-09-21T10:55:35.529 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.530 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.530 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.530 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.248060+0000", 2024-09-21T10:55:35.530 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.117367705 2024-09-21T10:55:35.530 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.530 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.530 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.530 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.248067+0000", 2024-09-21T10:55:35.530 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2679999999999997e-06 2024-09-21T10:55:35.530 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.530 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.530 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.530 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.615179+0000", 2024-09-21T10:55:35.530 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.367111712 2024-09-21T10:55:35.531 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.531 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.531 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:55:35.531 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.615544+0000", 2024-09-21T10:55:35.531 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000365235 2024-09-21T10:55:35.531 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.531 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.531 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:55:35.531 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.616032+0000", 2024-09-21T10:55:35.531 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000487925 2024-09-21T10:55:35.531 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.531 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.531 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:55:35.531 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.626548+0000", 2024-09-21T10:55:35.532 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.010516050000000001 2024-09-21T10:55:35.532 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.532 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.532 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.532 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.626593+0000", 2024-09-21T10:55:35.532 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5071000000000003e-05 2024-09-21T10:55:35.532 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.532 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.532 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.532 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.636832+0000", 2024-09-21T10:55:35.532 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.010238639000000001 2024-09-21T10:55:35.532 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.532 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.532 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:55:35.532 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.636949+0000", 2024-09-21T10:55:35.533 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000117445 2024-09-21T10:55:35.533 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.533 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.533 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:55:35.533 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.640990+0000", 2024-09-21T10:55:35.533 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0040410849999999998 2024-09-21T10:55:35.533 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.533 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:55:35.533 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.533 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.533 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.533 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:130861 55.c 55:3373a108:::benchmark_data_smithi070_294394_object25520:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:55:35.533 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:26.395479+0000", 2024-09-21T10:55:35.533 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 248.38856156400001, 2024-09-21T10:55:35.534 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3509853239999998, 2024-09-21T10:55:35.534 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:55:35.534 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:55:35.534 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:55:35.534 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:55:35.534 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:55:35.534 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:35.534 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 130861 2024-09-21T10:55:35.534 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.534 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:55:35.534 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.534 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:55:35.534 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.395479+0000", 2024-09-21T10:55:35.534 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.535 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.535 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.535 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:55:35.535 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.395479+0000", 2024-09-21T10:55:35.535 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.535 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.535 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.535 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:55:35.535 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.395480+0000", 2024-09-21T10:55:35.535 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5910000000000001e-06 2024-09-21T10:55:35.535 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.535 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.535 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:55:35.535 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.395485+0000", 2024-09-21T10:55:35.536 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5240000000000002e-06 2024-09-21T10:55:35.536 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.536 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.536 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:55:35.536 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.395497+0000", 2024-09-21T10:55:35.536 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1728e-05 2024-09-21T10:55:35.536 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.536 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.536 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:55:35.536 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.395503+0000", 2024-09-21T10:55:35.536 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8310000000000003e-06 2024-09-21T10:55:35.536 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.536 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.536 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.536 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.395568+0000", 2024-09-21T10:55:35.537 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4877999999999998e-05 2024-09-21T10:55:35.537 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.537 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.537 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:55:35.537 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.395721+0000", 2024-09-21T10:55:35.537 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015300700000000001 2024-09-21T10:55:35.537 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.537 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.537 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:55:35.537 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.396922+0000", 2024-09-21T10:55:35.537 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001200961 2024-09-21T10:55:35.537 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.538 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.538 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.538 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.412500+0000", 2024-09-21T10:55:35.538 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015577841 2024-09-21T10:55:35.538 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.538 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.538 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.538 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.412553+0000", 2024-09-21T10:55:35.538 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2957000000000001e-05 2024-09-21T10:55:35.538 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.539 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.539 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.539 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.453580+0000", 2024-09-21T10:55:35.539 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.041026405000000002 2024-09-21T10:55:35.539 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.539 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.539 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.539 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.453590+0000", 2024-09-21T10:55:35.539 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0397999999999999e-05 2024-09-21T10:55:35.539 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.539 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.539 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.539 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.421521+0000", 2024-09-21T10:55:35.539 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.967931231 2024-09-21T10:55:35.540 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.540 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.540 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.540 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.421535+0000", 2024-09-21T10:55:35.540 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3757000000000001e-05 2024-09-21T10:55:35.540 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.540 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.540 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.540 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.670551+0000", 2024-09-21T10:55:35.540 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2490162309999999 2024-09-21T10:55:35.540 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.540 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.540 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for missing object", 2024-09-21T10:55:35.540 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.670716+0000", 2024-09-21T10:55:35.541 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016505299999999999 2024-09-21T10:55:35.541 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.541 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.541 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.541 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.746286+0000", 2024-09-21T10:55:35.541 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.075570094000000004 2024-09-21T10:55:35.541 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.541 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.541 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:55:35.541 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.746464+0000", 2024-09-21T10:55:35.541 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000177837 2024-09-21T10:55:35.541 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.541 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:55:35.541 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.542 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.542 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.542 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:156258 55.c 55:39292414:::benchmark_data_smithi070_294394_object50917:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1464)", 2024-09-21T10:55:35.542 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.312622+0000", 2024-09-21T10:55:35.542 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 190.47141809600001, 2024-09-21T10:55:35.542 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.330110946, 2024-09-21T10:55:35.542 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:55:35.542 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:55:35.542 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:55:35.542 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:55:35.542 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:55:35.542 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:35.543 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 156258 2024-09-21T10:55:35.543 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.543 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:55:35.543 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.543 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:55:35.543 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.312622+0000", 2024-09-21T10:55:35.543 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.543 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.543 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.543 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:55:35.543 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.312622+0000", 2024-09-21T10:55:35.544 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.544 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.544 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.544 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:55:35.544 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.312625+0000", 2024-09-21T10:55:35.544 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6079999999999998e-06 2024-09-21T10:55:35.544 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.544 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.544 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:55:35.544 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.312632+0000", 2024-09-21T10:55:35.544 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7390000000000002e-06 2024-09-21T10:55:35.544 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.544 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.545 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:55:35.545 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.312650+0000", 2024-09-21T10:55:35.545 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7946000000000001e-05 2024-09-21T10:55:35.545 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.545 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.545 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:55:35.545 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.312660+0000", 2024-09-21T10:55:35.545 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0178e-05 2024-09-21T10:55:35.545 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.545 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.545 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.545 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.319581+0000", 2024-09-21T10:55:35.545 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0069215680000000003 2024-09-21T10:55:35.546 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.546 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.546 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:55:35.546 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.322223+0000", 2024-09-21T10:55:35.546 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0026414160000000002 2024-09-21T10:55:35.546 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.546 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.546 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.546 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.335700+0000", 2024-09-21T10:55:35.546 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.013477668 2024-09-21T10:55:35.546 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.546 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.546 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:55:35.547 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.335800+0000", 2024-09-21T10:55:35.547 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9123000000000006e-05 2024-09-21T10:55:35.547 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.547 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.547 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,1,4", 2024-09-21T10:55:35.547 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.335951+0000", 2024-09-21T10:55:35.547 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015160799999999999 2024-09-21T10:55:35.547 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.547 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.547 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:55:35.547 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.339558+0000", 2024-09-21T10:55:35.547 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0036068509999999999 2024-09-21T10:55:35.548 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.548 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.548 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.548 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.340760+0000", 2024-09-21T10:55:35.548 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0012021810000000001 2024-09-21T10:55:35.548 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.548 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.548 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.548 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.342017+0000", 2024-09-21T10:55:35.548 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001256704 2024-09-21T10:55:35.548 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.548 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.548 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.549 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.476857+0000", 2024-09-21T10:55:35.549 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13483968700000001 2024-09-21T10:55:35.549 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.549 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.549 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.549 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.476910+0000", 2024-09-21T10:55:35.549 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3829000000000002e-05 2024-09-21T10:55:35.549 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.549 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.549 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.549 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.642675+0000", 2024-09-21T10:55:35.549 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.165764507 2024-09-21T10:55:35.549 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.549 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.549 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:55:35.550 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.642733+0000", 2024-09-21T10:55:35.550 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8332999999999999e-05 2024-09-21T10:55:35.550 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.550 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:55:35.550 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.550 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.550 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.550 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:156286 55.c 55:3c4e6380:::benchmark_data_smithi070_294394_object50945:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1464)", 2024-09-21T10:55:35.550 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.368882+0000", 2024-09-21T10:55:35.550 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 190.415158088, 2024-09-21T10:55:35.550 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3537682069999999, 2024-09-21T10:55:35.550 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:55:35.550 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:55:35.551 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:35.551 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:55:35.551 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:55:35.551 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:35.551 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 156286 2024-09-21T10:55:35.551 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.551 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:55:35.551 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.551 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:55:35.551 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.368882+0000", 2024-09-21T10:55:35.551 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.552 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.552 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.552 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:55:35.552 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.368882+0000", 2024-09-21T10:55:35.552 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.552 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.552 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.552 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:55:35.552 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.368885+0000", 2024-09-21T10:55:35.552 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2979999999999999e-06 2024-09-21T10:55:35.552 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.552 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.553 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:55:35.553 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.368891+0000", 2024-09-21T10:55:35.553 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.708e-06 2024-09-21T10:55:35.553 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.553 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.553 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:55:35.553 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.368907+0000", 2024-09-21T10:55:35.553 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5437000000000001e-05 2024-09-21T10:55:35.553 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.553 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.553 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:55:35.553 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.368916+0000", 2024-09-21T10:55:35.553 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9160000000000007e-06 2024-09-21T10:55:35.553 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.554 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.554 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.554 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.368988+0000", 2024-09-21T10:55:35.554 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1964000000000004e-05 2024-09-21T10:55:35.554 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.554 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.554 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:55:35.554 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.369490+0000", 2024-09-21T10:55:35.554 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00050230499999999998 2024-09-21T10:55:35.554 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.554 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.554 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.554 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.478001+0000", 2024-09-21T10:55:35.555 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.108510617 2024-09-21T10:55:35.555 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.555 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.555 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.555 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.478010+0000", 2024-09-21T10:55:35.555 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.8339999999999994e-06 2024-09-21T10:55:35.555 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.555 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.555 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.555 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.710218+0000", 2024-09-21T10:55:35.555 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2322071860000001 2024-09-21T10:55:35.555 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.555 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.555 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:55:35.556 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.710354+0000", 2024-09-21T10:55:35.556 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013647799999999999 2024-09-21T10:55:35.556 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.556 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.556 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,4,7", 2024-09-21T10:55:35.556 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.710499+0000", 2024-09-21T10:55:35.556 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014466899999999999 2024-09-21T10:55:35.556 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.556 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.556 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.556 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.714807+0000", 2024-09-21T10:55:35.556 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0043086019999999999 2024-09-21T10:55:35.556 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.556 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.557 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.557 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.714852+0000", 2024-09-21T10:55:35.557 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.4433e-05 2024-09-21T10:55:35.557 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.557 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.557 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:55:35.557 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.714864+0000", 2024-09-21T10:55:35.557 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1946999999999999e-05 2024-09-21T10:55:35.557 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.557 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.557 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.557 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.720322+0000", 2024-09-21T10:55:35.557 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0054582290000000002 2024-09-21T10:55:35.557 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.558 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.558 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:55:35.558 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.720406+0000", 2024-09-21T10:55:35.558 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3776999999999994e-05 2024-09-21T10:55:35.558 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.558 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.558 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:55:35.558 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.722651+0000", 2024-09-21T10:55:35.558 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0022448070000000001 2024-09-21T10:55:35.558 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.559 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:55:35.559 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.559 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.559 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.559 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:156299 55.c 55:372c1ddf:::benchmark_data_smithi070_294394_object50958:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1464)", 2024-09-21T10:55:35.559 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.406138+0000", 2024-09-21T10:55:35.559 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 190.37790200500001, 2024-09-21T10:55:35.559 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3185818819999999, 2024-09-21T10:55:35.559 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:55:35.559 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:55:35.559 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:35.559 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:55:35.559 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:55:35.559 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:35.560 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 156299 2024-09-21T10:55:35.560 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.560 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:55:35.560 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.560 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:55:35.560 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.406138+0000", 2024-09-21T10:55:35.560 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.560 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.560 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.560 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:55:35.560 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.406138+0000", 2024-09-21T10:55:35.560 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.560 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.560 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.561 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:55:35.561 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.406140+0000", 2024-09-21T10:55:35.561 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.181e-06 2024-09-21T10:55:35.561 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.561 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.561 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:55:35.561 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.406143+0000", 2024-09-21T10:55:35.561 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9249999999999997e-06 2024-09-21T10:55:35.561 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.561 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.561 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:55:35.561 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.406155+0000", 2024-09-21T10:55:35.561 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1143e-05 2024-09-21T10:55:35.562 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.562 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.562 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:55:35.562 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.406163+0000", 2024-09-21T10:55:35.562 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5699999999999993e-06 2024-09-21T10:55:35.562 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.562 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.562 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.562 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.406234+0000", 2024-09-21T10:55:35.562 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0656999999999998e-05 2024-09-21T10:55:35.562 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.562 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.562 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:55:35.562 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.406708+0000", 2024-09-21T10:55:35.563 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00047407900000000003 2024-09-21T10:55:35.563 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.563 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.563 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.563 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.478035+0000", 2024-09-21T10:55:35.563 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.071327045000000006 2024-09-21T10:55:35.563 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.563 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.563 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.563 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.478041+0000", 2024-09-21T10:55:35.563 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.224e-06 2024-09-21T10:55:35.563 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.563 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.564 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.564 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.712816+0000", 2024-09-21T10:55:35.564 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.234774367 2024-09-21T10:55:35.564 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.564 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.564 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:55:35.564 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.712921+0000", 2024-09-21T10:55:35.564 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000105307 2024-09-21T10:55:35.564 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.564 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.564 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,4,7", 2024-09-21T10:55:35.564 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.713000+0000", 2024-09-21T10:55:35.564 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8954000000000003e-05 2024-09-21T10:55:35.564 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.565 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.565 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.565 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.714914+0000", 2024-09-21T10:55:35.565 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0019138530000000001 2024-09-21T10:55:35.565 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.565 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.565 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.565 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.714967+0000", 2024-09-21T10:55:35.565 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3718999999999998e-05 2024-09-21T10:55:35.565 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.565 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.565 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:55:35.565 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.714981+0000", 2024-09-21T10:55:35.565 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3464e-05 2024-09-21T10:55:35.566 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.566 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.566 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.566 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.722715+0000", 2024-09-21T10:55:35.566 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0077343689999999996 2024-09-21T10:55:35.566 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.566 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.566 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:55:35.566 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.722775+0000", 2024-09-21T10:55:35.566 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9599999999999999e-05 2024-09-21T10:55:35.566 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.566 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.566 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:55:35.567 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.724720+0000", 2024-09-21T10:55:35.567 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001945425 2024-09-21T10:55:35.567 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.567 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:55:35.567 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.567 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.567 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.567 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:156303 55.0 55:043b163c:::benchmark_data_smithi070_294394_object50962:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1465)", 2024-09-21T10:55:35.567 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.513934+0000", 2024-09-21T10:55:35.567 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 190.270105984, 2024-09-21T10:55:35.567 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2375139740000001, 2024-09-21T10:55:35.567 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:55:35.567 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:55:35.567 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:35.568 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:55:35.568 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:55:35.568 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:35.568 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 156303 2024-09-21T10:55:35.568 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.568 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:55:35.568 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.568 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:55:35.568 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.513934+0000", 2024-09-21T10:55:35.568 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.568 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.568 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.568 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:55:35.568 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.513934+0000", 2024-09-21T10:55:35.568 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.569 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.569 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.569 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:55:35.569 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.513936+0000", 2024-09-21T10:55:35.569 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1009999999999999e-06 2024-09-21T10:55:35.569 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.569 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.569 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:55:35.569 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.513944+0000", 2024-09-21T10:55:35.569 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9140000000000008e-06 2024-09-21T10:55:35.569 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.569 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.569 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:55:35.570 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.513963+0000", 2024-09-21T10:55:35.570 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8712999999999999e-05 2024-09-21T10:55:35.570 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.570 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.570 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:55:35.570 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.513972+0000", 2024-09-21T10:55:35.570 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1190000000000003e-06 2024-09-21T10:55:35.570 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.570 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.570 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.570 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.514061+0000", 2024-09-21T10:55:35.570 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9059000000000002e-05 2024-09-21T10:55:35.570 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.571 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.571 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.571 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.514073+0000", 2024-09-21T10:55:35.571 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1301000000000001e-05 2024-09-21T10:55:35.571 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.571 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.571 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.571 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.462210+0000", 2024-09-21T10:55:35.571 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.94813707000000003 2024-09-21T10:55:35.571 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.571 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.571 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.571 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.462220+0000", 2024-09-21T10:55:35.571 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0360000000000001e-05 2024-09-21T10:55:35.571 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.572 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.572 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.572 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.707189+0000", 2024-09-21T10:55:35.572 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2449689660000001 2024-09-21T10:55:35.572 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.572 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.572 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:55:35.572 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.707640+0000", 2024-09-21T10:55:35.572 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00045076399999999998 2024-09-21T10:55:35.572 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.572 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.572 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.572 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.723477+0000", 2024-09-21T10:55:35.573 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015836787000000001 2024-09-21T10:55:35.573 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.573 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.573 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:55:35.573 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.723634+0000", 2024-09-21T10:55:35.573 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015763100000000001 2024-09-21T10:55:35.573 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.573 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.573 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6,7", 2024-09-21T10:55:35.573 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.723760+0000", 2024-09-21T10:55:35.573 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012615099999999999 2024-09-21T10:55:35.573 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.573 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.573 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:55:35.574 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.733609+0000", 2024-09-21T10:55:35.574 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0098482320000000002 2024-09-21T10:55:35.574 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.574 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.574 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.574 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.748913+0000", 2024-09-21T10:55:35.574 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015304386999999999 2024-09-21T10:55:35.574 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.574 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.574 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.574 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.748958+0000", 2024-09-21T10:55:35.574 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.4951999999999999e-05 2024-09-21T10:55:35.575 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.575 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.575 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.575 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.749085+0000", 2024-09-21T10:55:35.575 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000127513 2024-09-21T10:55:35.575 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.575 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.575 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:55:35.575 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.749146+0000", 2024-09-21T10:55:35.575 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0296e-05 2024-09-21T10:55:35.575 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.575 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.575 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:55:35.575 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.751448+0000", 2024-09-21T10:55:35.576 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002302658 2024-09-21T10:55:35.576 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.576 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:55:35.576 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.576 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.576 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.576 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:156314 55.0 55:08dfcd2f:::benchmark_data_smithi070_294394_object50973:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1465)", 2024-09-21T10:55:35.576 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.554453+0000", 2024-09-21T10:55:35.576 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 190.22958725500001, 2024-09-21T10:55:35.576 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1993647900000002, 2024-09-21T10:55:35.576 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:55:35.576 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:55:35.576 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:35.577 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:55:35.577 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:55:35.577 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:35.577 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 156314 2024-09-21T10:55:35.577 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.577 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:55:35.577 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.577 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:55:35.577 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.554453+0000", 2024-09-21T10:55:35.577 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.577 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.577 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.577 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:55:35.577 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.554453+0000", 2024-09-21T10:55:35.577 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.578 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.578 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.578 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:55:35.578 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.554460+0000", 2024-09-21T10:55:35.578 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2200000000000003e-06 2024-09-21T10:55:35.578 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.578 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.578 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:55:35.578 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.554468+0000", 2024-09-21T10:55:35.578 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5379999999999997e-06 2024-09-21T10:55:35.578 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.578 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.578 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:55:35.579 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.554484+0000", 2024-09-21T10:55:35.579 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.577e-05 2024-09-21T10:55:35.579 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.579 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.579 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:55:35.579 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.554502+0000", 2024-09-21T10:55:35.579 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8406999999999998e-05 2024-09-21T10:55:35.579 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.579 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.579 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.579 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.554554+0000", 2024-09-21T10:55:35.579 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1579000000000002e-05 2024-09-21T10:55:35.579 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.579 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.580 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.580 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.554561+0000", 2024-09-21T10:55:35.580 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0249999999999997e-06 2024-09-21T10:55:35.580 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.580 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.580 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.580 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.462245+0000", 2024-09-21T10:55:35.580 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.90768446700000005 2024-09-21T10:55:35.580 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.580 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.580 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.580 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.462253+0000", 2024-09-21T10:55:35.580 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4019999999999996e-06 2024-09-21T10:55:35.580 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.581 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.581 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.581 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.709897+0000", 2024-09-21T10:55:35.581 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2476449220000001 2024-09-21T10:55:35.581 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.581 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.581 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:55:35.581 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.710321+0000", 2024-09-21T10:55:35.581 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000423972 2024-09-21T10:55:35.581 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.581 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.581 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.581 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.731108+0000", 2024-09-21T10:55:35.581 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.020786414 2024-09-21T10:55:35.582 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.582 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.582 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:55:35.582 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.731214+0000", 2024-09-21T10:55:35.582 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010618799999999999 2024-09-21T10:55:35.582 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.582 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.582 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6,7", 2024-09-21T10:55:35.582 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.731346+0000", 2024-09-21T10:55:35.582 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013200500000000001 2024-09-21T10:55:35.582 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.582 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.582 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:55:35.583 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.746498+0000", 2024-09-21T10:55:35.583 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015151553 2024-09-21T10:55:35.583 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.583 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.583 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.583 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.749001+0000", 2024-09-21T10:55:35.583 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025033249999999998 2024-09-21T10:55:35.583 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.583 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.583 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.583 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.749043+0000", 2024-09-21T10:55:35.583 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2175999999999997e-05 2024-09-21T10:55:35.583 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.583 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.584 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.584 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.751602+0000", 2024-09-21T10:55:35.584 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025585629999999998 2024-09-21T10:55:35.584 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.584 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.584 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:55:35.584 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.751663+0000", 2024-09-21T10:55:35.584 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1457999999999993e-05 2024-09-21T10:55:35.584 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.584 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.584 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:55:35.584 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.753818+0000", 2024-09-21T10:55:35.584 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002154806 2024-09-21T10:55:35.584 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.585 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:55:35.585 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.585 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.585 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.585 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:156321 55.0 55:0a755edd:::benchmark_data_smithi070_294394_object50980:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1466)", 2024-09-21T10:55:35.585 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:25.645767+0000", 2024-09-21T10:55:35.585 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 189.13827316199999, 2024-09-21T10:55:35.585 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1136162780000001, 2024-09-21T10:55:35.585 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:55:35.585 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:55:35.585 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:35.585 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:55:35.585 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:55:35.586 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:35.586 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 156321 2024-09-21T10:55:35.586 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.586 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:55:35.586 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.586 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:55:35.586 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.645767+0000", 2024-09-21T10:55:35.586 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.586 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.586 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.586 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:55:35.586 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.645767+0000", 2024-09-21T10:55:35.586 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.586 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.587 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.587 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:55:35.587 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.645769+0000", 2024-09-21T10:55:35.587 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.559e-06 2024-09-21T10:55:35.587 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.587 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.587 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:55:35.587 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.645775+0000", 2024-09-21T10:55:35.587 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8379999999999998e-06 2024-09-21T10:55:35.587 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.587 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.587 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:55:35.587 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.645788+0000", 2024-09-21T10:55:35.587 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.361e-05 2024-09-21T10:55:35.588 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.588 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.588 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:55:35.588 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.645796+0000", 2024-09-21T10:55:35.588 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9449999999999993e-06 2024-09-21T10:55:35.588 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.588 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.588 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.588 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.645861+0000", 2024-09-21T10:55:35.588 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4399000000000005e-05 2024-09-21T10:55:35.588 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.588 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.588 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.588 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.645868+0000", 2024-09-21T10:55:35.588 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0690000000000004e-06 2024-09-21T10:55:35.589 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.589 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.589 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.589 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.710350+0000", 2024-09-21T10:55:35.589 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.064481888 2024-09-21T10:55:35.589 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.589 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.589 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:55:35.589 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.710679+0000", 2024-09-21T10:55:35.589 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00032911699999999997 2024-09-21T10:55:35.589 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.589 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.589 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.589 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.738592+0000", 2024-09-21T10:55:35.590 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.027912939000000001 2024-09-21T10:55:35.590 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.590 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.590 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:55:35.590 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.738699+0000", 2024-09-21T10:55:35.590 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010733 2024-09-21T10:55:35.590 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.590 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.590 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6,7", 2024-09-21T10:55:35.590 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.738945+0000", 2024-09-21T10:55:35.590 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00024583500000000002 2024-09-21T10:55:35.590 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.590 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.590 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:55:35.591 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.748872+0000", 2024-09-21T10:55:35.591 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0099270450000000007 2024-09-21T10:55:35.591 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.591 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.591 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.591 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.751504+0000", 2024-09-21T10:55:35.591 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0026319249999999998 2024-09-21T10:55:35.591 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.591 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.591 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.591 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.751554+0000", 2024-09-21T10:55:35.591 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0661000000000002e-05 2024-09-21T10:55:35.591 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.592 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.592 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.592 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.756319+0000", 2024-09-21T10:55:35.592 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0047645250000000004 2024-09-21T10:55:35.592 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.592 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.592 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:55:35.592 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.756560+0000", 2024-09-21T10:55:35.592 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000241375 2024-09-21T10:55:35.592 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.592 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.592 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:55:35.592 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.759383+0000", 2024-09-21T10:55:35.592 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0028232180000000002 2024-09-21T10:55:35.593 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.593 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:55:35.593 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.593 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.593 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.593 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36012.0:66794 56.5 56:af9493d5:::benchmark_data_smithi070_333437_object66793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1490)", 2024-09-21T10:55:35.593 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:55:00.442566+0000", 2024-09-21T10:55:35.593 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 34.341474423000001, 2024-09-21T10:55:35.593 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2236836719999999, 2024-09-21T10:55:35.593 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:55:35.593 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:55:35.593 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:55:35.593 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:55:35.594 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36012", 2024-09-21T10:55:35.594 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T10:55:35.594 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 66794 2024-09-21T10:55:35.594 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.594 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:55:35.594 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.594 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:55:35.594 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442566+0000", 2024-09-21T10:55:35.594 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.594 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.594 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.594 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:55:35.594 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442566+0000", 2024-09-21T10:55:35.594 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.595 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.595 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.595 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:55:35.595 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442567+0000", 2024-09-21T10:55:35.595 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.8899999999999998e-07 2024-09-21T10:55:35.595 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.595 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.595 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:55:35.595 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442585+0000", 2024-09-21T10:55:35.595 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8022000000000001e-05 2024-09-21T10:55:35.595 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.595 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.595 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:55:35.596 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442596+0000", 2024-09-21T10:55:35.596 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0895e-05 2024-09-21T10:55:35.596 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.596 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.596 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:55:35.596 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442602+0000", 2024-09-21T10:55:35.596 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4019999999999999e-06 2024-09-21T10:55:35.596 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.596 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.596 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.596 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442671+0000", 2024-09-21T10:55:35.596 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8868999999999994e-05 2024-09-21T10:55:35.596 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.596 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.597 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:55:35.597 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442774+0000", 2024-09-21T10:55:35.597 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000102638 2024-09-21T10:55:35.597 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.597 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.597 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:55:35.597 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442886+0000", 2024-09-21T10:55:35.597 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000111853 2024-09-21T10:55:35.597 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.597 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.597 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:55:35.597 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.445148+0000", 2024-09-21T10:55:35.597 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0022620380000000001 2024-09-21T10:55:35.597 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.597 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.598 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.598 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465200+0000", 2024-09-21T10:55:35.598 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.020052745 2024-09-21T10:55:35.598 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.598 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.598 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.598 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465211+0000", 2024-09-21T10:55:35.598 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0713999999999999e-05 2024-09-21T10:55:35.598 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.598 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.598 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.598 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:01.454645+0000", 2024-09-21T10:55:35.598 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.98943445699999999 2024-09-21T10:55:35.599 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.599 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.599 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.599 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:01.454664+0000", 2024-09-21T10:55:35.599 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8576999999999998e-05 2024-09-21T10:55:35.599 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.599 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.599 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.599 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.666160+0000", 2024-09-21T10:55:35.599 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2114961719999999 2024-09-21T10:55:35.599 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.599 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.599 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:55:35.599 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.666250+0000", 2024-09-21T10:55:35.600 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9400999999999993e-05 2024-09-21T10:55:35.600 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.600 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:55:35.600 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.600 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.600 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.600 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36012.0:66807 56.5 56:a8efccf3:::benchmark_data_smithi070_333437_object66806:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1491)", 2024-09-21T10:55:35.600 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:55:00.465619+0000", 2024-09-21T10:55:35.600 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 34.318421041999997, 2024-09-21T10:55:35.600 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2238977850000001, 2024-09-21T10:55:35.600 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:55:35.600 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:55:35.600 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:35.600 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:55:35.600 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36012", 2024-09-21T10:55:35.601 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T10:55:35.601 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 66807 2024-09-21T10:55:35.601 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.601 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:55:35.601 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.601 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:55:35.601 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465619+0000", 2024-09-21T10:55:35.601 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.601 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.601 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.601 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:55:35.601 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465619+0000", 2024-09-21T10:55:35.601 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:55:35.601 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.602 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.602 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:55:35.602 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465621+0000", 2024-09-21T10:55:35.602 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.841e-06 2024-09-21T10:55:35.602 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.602 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.602 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:55:35.602 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465673+0000", 2024-09-21T10:55:35.602 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1858999999999999e-05 2024-09-21T10:55:35.602 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.602 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.602 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:55:35.602 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465694+0000", 2024-09-21T10:55:35.603 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1259000000000001e-05 2024-09-21T10:55:35.603 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.603 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.603 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:55:35.603 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465715+0000", 2024-09-21T10:55:35.603 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1141999999999998e-05 2024-09-21T10:55:35.603 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.603 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.603 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.603 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465867+0000", 2024-09-21T10:55:35.603 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000151569 2024-09-21T10:55:35.603 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.603 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.603 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.604 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465876+0000", 2024-09-21T10:55:35.604 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9139999999999997e-06 2024-09-21T10:55:35.604 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.604 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.604 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.604 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:01.454697+0000", 2024-09-21T10:55:35.604 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.98882113500000002 2024-09-21T10:55:35.604 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.604 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.604 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:55:35.604 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:01.454706+0000", 2024-09-21T10:55:35.604 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2080000000000006e-06 2024-09-21T10:55:35.604 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.604 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.605 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:55:35.605 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.670154+0000", 2024-09-21T10:55:35.605 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2154475469999999 2024-09-21T10:55:35.605 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.605 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.605 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:55:35.605 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.670320+0000", 2024-09-21T10:55:35.605 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016649899999999999 2024-09-21T10:55:35.605 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.605 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.605 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:55:35.605 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.670782+0000", 2024-09-21T10:55:35.605 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00046200700000000001 2024-09-21T10:55:35.605 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.606 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.606 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:55:35.606 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.677750+0000", 2024-09-21T10:55:35.606 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0069672229999999998 2024-09-21T10:55:35.606 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.606 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.606 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.606 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.678570+0000", 2024-09-21T10:55:35.606 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00082092899999999997 2024-09-21T10:55:35.606 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.606 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.606 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:35.606 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.685882+0000", 2024-09-21T10:55:35.606 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00731155 2024-09-21T10:55:35.607 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.607 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.607 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:55:35.607 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.685948+0000", 2024-09-21T10:55:35.607 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5481000000000005e-05 2024-09-21T10:55:35.607 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:55:35.607 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:55:35.607 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:55:35.607 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.689517+0000", 2024-09-21T10:55:35.607 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035696220000000002 2024-09-21T10:55:35.607 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.607 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:55:35.607 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.607 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:55:35.607 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:55:35.608 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:55:35.608 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:35.609 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 69 16 95323 95307 86.3131 52.1875 0.021892 0.0115803 2024-09-21T10:55:35.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:35.648 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:35.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:35.749 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:35.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:35.786 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:35.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:35.887 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:35.928 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:35.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:35.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:36.029 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:36.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:36.061 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:36.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:36.162 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:36.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:36.194 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:36.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:36.295 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:36.331 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:36.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:36.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:36.433 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:36.464 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:36.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:36.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:36.560 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 70 16 96203 96187 85.8657 55 0.00914345 0.01164 2024-09-21T10:55:36.566 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:36.602 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:36.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:36.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:36.649 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:36 smithi070 ceph-mon[107385]: pgmap v2798: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.7 GiB data, 18 GiB used, 340 GiB / 358 GiB avail; 53 MiB/s wr, 848 op/s; 33576/280476 objects degraded (11.971%); 6404/280476 objects misplaced (2.283%); 4.0 MiB/s, 63 objects/s recovering 2024-09-21T10:55:36.650 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:55:36.650 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:55:36.650 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:36 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4165326638' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:55:36.650 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:36 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:55:36.650 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:36 smithi070 ceph-mon[112139]: pgmap v2798: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.7 GiB data, 18 GiB used, 340 GiB / 358 GiB avail; 53 MiB/s wr, 848 op/s; 33576/280476 objects degraded (11.971%); 6404/280476 objects misplaced (2.283%); 4.0 MiB/s, 63 objects/s recovering 2024-09-21T10:55:36.650 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:55:36.650 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:55:36.650 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:36 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4165326638' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:55:36.651 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:36 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:55:36.651 INFO:teuthology.orchestra.run.smithi070.stderr:set pool 56 pg_num to 26 2024-09-21T10:55:36.704 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:36.741 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:36.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:36.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:36 smithi160 ceph-mon[101879]: pgmap v2798: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.7 GiB data, 18 GiB used, 340 GiB / 358 GiB avail; 53 MiB/s wr, 848 op/s; 33576/280476 objects degraded (11.971%); 6404/280476 objects misplaced (2.283%); 4.0 MiB/s, 63 objects/s recovering 2024-09-21T10:55:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:55:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:55:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:36 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4165326638' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:55:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:36 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:55:36.843 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:36.884 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:36.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:36.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:36.986 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:37.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:37.020 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:37.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:37.121 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:37.164 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:37.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:37.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:37.266 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:37.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:37.302 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:37.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:37.402 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:37.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:37.440 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:37.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:37.541 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:37.560 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 71 16 97108 97092 85.453 56.5625 0.0246982 0.0116972 2024-09-21T10:55:37.572 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:37.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:37.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:37.674 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:37.736 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:37.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:37.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:37.838 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:37.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:37 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-09-21T10:55:37.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:37 smithi070 ceph-mon[107385]: osdmap e1498: 8 total, 8 up, 4 in 2024-09-21T10:55:37.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:37 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T10:55:37.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:37 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-09-21T10:55:37.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:37 smithi070 ceph-mon[112139]: osdmap e1498: 8 total, 8 up, 4 in 2024-09-21T10:55:37.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:37 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T10:55:37.882 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:37.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:37.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:37.984 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:38.005 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:38.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:38.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:38.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:37 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-09-21T10:55:38.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:37 smithi160 ceph-mon[101879]: osdmap e1498: 8 total, 8 up, 4 in 2024-09-21T10:55:38.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:37 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T10:55:38.107 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:38.129 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:38.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:38.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:38.231 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:38.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:38.253 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:38.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:38.354 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:38.391 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:38.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:38.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:38.493 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:38.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:38.522 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:38.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:38.561 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 72 16 97262 97246 84.3999 9.625 0.00330093 0.0117914 2024-09-21T10:55:38.625 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:38.647 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:38.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:38.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:38.750 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:38.773 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:38.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:38.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:38.875 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:38.897 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:38.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:38.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:38.999 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:39.021 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:39.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:39.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:38 smithi160 ceph-mon[101879]: pgmap v2800: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.7 GiB data, 18 GiB used, 340 GiB / 358 GiB avail; 53 MiB/s wr, 855 op/s; 33576/282048 objects degraded (11.904%); 6404/282048 objects misplaced (2.271%); 3.7 MiB/s, 59 objects/s recovering 2024-09-21T10:55:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:38 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T10:55:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:38 smithi160 ceph-mon[101879]: osdmap e1499: 8 total, 8 up, 4 in 2024-09-21T10:55:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:38 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 33576/282048 objects degraded (11.904%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:38 smithi160 ceph-mon[101879]: osdmap e1500: 8 total, 8 up, 4 in 2024-09-21T10:55:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:38 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:55:39.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:39.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:38 smithi070 ceph-mon[112139]: pgmap v2800: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.7 GiB data, 18 GiB used, 340 GiB / 358 GiB avail; 53 MiB/s wr, 855 op/s; 33576/282048 objects degraded (11.904%); 6404/282048 objects misplaced (2.271%); 3.7 MiB/s, 59 objects/s recovering 2024-09-21T10:55:39.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:38 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T10:55:39.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:38 smithi070 ceph-mon[112139]: osdmap e1499: 8 total, 8 up, 4 in 2024-09-21T10:55:39.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:38 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 33576/282048 objects degraded (11.904%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:39.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:38 smithi070 ceph-mon[112139]: osdmap e1500: 8 total, 8 up, 4 in 2024-09-21T10:55:39.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:38 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:55:39.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:38 smithi070 ceph-mon[107385]: pgmap v2800: 99 pgs: 2 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.7 GiB data, 18 GiB used, 340 GiB / 358 GiB avail; 53 MiB/s wr, 855 op/s; 33576/282048 objects degraded (11.904%); 6404/282048 objects misplaced (2.271%); 3.7 MiB/s, 59 objects/s recovering 2024-09-21T10:55:39.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:38 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T10:55:39.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:38 smithi070 ceph-mon[107385]: osdmap e1499: 8 total, 8 up, 4 in 2024-09-21T10:55:39.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:38 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 33576/282048 objects degraded (11.904%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:39.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:38 smithi070 ceph-mon[107385]: osdmap e1500: 8 total, 8 up, 4 in 2024-09-21T10:55:39.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:38 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:55:39.143 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:39.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:39.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:39.245 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:39.266 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:39.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:39.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:39.368 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:39.397 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:39.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:39.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:39.499 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:39.542 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:39.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:39.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:39.560 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 73 16 97475 97459 83.4262 13.3125 0.0300441 0.0119813 2024-09-21T10:55:39.568 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:55:39.636 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:55:39.644 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:39.704 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:39.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:39.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:39.805 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:39.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:39.840 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:39.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:39.893 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:55:39.940 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:40.022 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:40.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:40.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:40.124 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:40.157 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:40.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:40.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:40.259 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:40.296 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:40.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:40.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:40.397 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:40.436 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:40.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:40.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:40.538 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:40.561 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 74 16 98640 98624 83.2827 72.8125 0.0146307 0.0120013 2024-09-21T10:55:40.572 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:40.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:40.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:40.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:40 smithi160 ceph-mon[101879]: pgmap v2803: 109 pgs: 1 active+recovery_wait+degraded, 1 peering, 9 unknown, 2 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.9 GiB data, 18 GiB used, 339 GiB / 358 GiB avail; 52 MiB/s wr, 824 op/s; 31470/288366 objects degraded (10.913%); 5989/288366 objects misplaced (2.077%); 6.3 MiB/s, 100 objects/s recovering 2024-09-21T10:55:40.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:40 smithi160 ceph-mon[101879]: osdmap e1501: 8 total, 8 up, 4 in 2024-09-21T10:55:40.612 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:40 smithi070 ceph-mon[112139]: pgmap v2803: 109 pgs: 1 active+recovery_wait+degraded, 1 peering, 9 unknown, 2 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.9 GiB data, 18 GiB used, 339 GiB / 358 GiB avail; 52 MiB/s wr, 824 op/s; 31470/288366 objects degraded (10.913%); 5989/288366 objects misplaced (2.077%); 6.3 MiB/s, 100 objects/s recovering 2024-09-21T10:55:40.613 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:40 smithi070 ceph-mon[112139]: osdmap e1501: 8 total, 8 up, 4 in 2024-09-21T10:55:40.613 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:40 smithi070 ceph-mon[107385]: pgmap v2803: 109 pgs: 1 active+recovery_wait+degraded, 1 peering, 9 unknown, 2 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.9 GiB data, 18 GiB used, 339 GiB / 358 GiB avail; 52 MiB/s wr, 824 op/s; 31470/288366 objects degraded (10.913%); 5989/288366 objects misplaced (2.077%); 6.3 MiB/s, 100 objects/s recovering 2024-09-21T10:55:40.613 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:40 smithi070 ceph-mon[107385]: osdmap e1501: 8 total, 8 up, 4 in 2024-09-21T10:55:40.674 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:40.710 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:40.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:40.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:40.812 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:40.855 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:40.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:40.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:40.957 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:40.988 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:40.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:40.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:41.090 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:41.128 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:41.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:41.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:41.230 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:41.271 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:41.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:41.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:41.373 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:41.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:41.412 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:41.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:41.514 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:41.554 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:41.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:41.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:41.560 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 75 16 99885 99869 83.2096 77.8125 0.00688776 0.0120122 2024-09-21T10:55:41.656 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:41.684 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:41.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:41.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:41.785 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:41.828 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:41.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:41.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:41.930 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:41.972 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:41.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:41.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:42.074 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:42.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:42.114 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:42.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:42.215 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:42.244 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:42.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:42.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:42.346 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:42.375 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:42.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:42.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:42.468 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7] out_osds: [2, 5, 0, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:55:42.469 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:55:42.469 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-21T10:55:42.469 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:55:42.469 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:55:42.469 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-21T10:55:42.476 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:42.515 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:55:42.516 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:55:42.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:42 smithi070 ceph-mon[107385]: pgmap v2805: 109 pgs: 2 active+recovery_wait+degraded, 2 peering, 5 unknown, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 5.9 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 51 MiB/s wr, 819 op/s; 29355/291126 objects degraded (10.083%); 5989/291126 objects misplaced (2.057%); 6.0 MiB/s, 96 objects/s recovering 2024-09-21T10:55:42.546 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:42 smithi070 ceph-mon[112139]: pgmap v2805: 109 pgs: 2 active+recovery_wait+degraded, 2 peering, 5 unknown, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 5.9 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 51 MiB/s wr, 819 op/s; 29355/291126 objects degraded (10.083%); 5989/291126 objects misplaced (2.057%); 6.0 MiB/s, 96 objects/s recovering 2024-09-21T10:55:42.547 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:55:42.547 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:55:42.559 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:42.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:42.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:42.561 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 76 16 101100 101084 83.1139 75.9375 0.0294386 0.0120254 2024-09-21T10:55:42.576 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:55:42.576 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:55:42.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:42 smithi160 ceph-mon[101879]: pgmap v2805: 109 pgs: 2 active+recovery_wait+degraded, 2 peering, 5 unknown, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 5.9 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 51 MiB/s wr, 819 op/s; 29355/291126 objects degraded (10.083%); 5989/291126 objects misplaced (2.057%); 6.0 MiB/s, 96 objects/s recovering 2024-09-21T10:55:42.604 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:55:42.605 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:55:42.632 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:55:42.632 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:55:42.661 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:42.667 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:55:42.667 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:55:42.697 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:42.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:42.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:42.699 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:55:42.699 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:55:42.732 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:55:42.732 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:55:42.799 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:42.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:42.834 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:42.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:42.935 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:42.971 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:42.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:42.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:43.073 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:43.106 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:55:43.106 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:55:43.111 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:43.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:43.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:43.214 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:43.266 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:43.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:43.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:43.340 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:55:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:55:43] "GET /metrics HTTP/1.1" 200 36627 "" "Prometheus/2.51.0" 2024-09-21T10:55:43.368 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:43.410 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:43.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:43.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:43.511 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:43.545 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:43.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:43.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:43.561 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 77 16 102231 102215 82.9524 70.6875 0.00590071 0.0120483 2024-09-21T10:55:43.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:43 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2624335332' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:55:43.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:43 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 29355/291126 objects degraded (10.083%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:43.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:43 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2624335332' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:55:43.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:43 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 29355/291126 objects degraded (10.083%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:43.634 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:43 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2624335332' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:55:43.635 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:43 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 29355/291126 objects degraded (10.083%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:43.647 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:43.688 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:43.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:43.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:43.789 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:43.822 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:43.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:43.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:43.924 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:43.951 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:55:43.977 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:43.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:43.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:44.079 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:44.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:44.128 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:44.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:44.228 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:44.256 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:44.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:44.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:44.357 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:44.378 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T10:55:44.385 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:44.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:44.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:44.487 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:44.517 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:44.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:44.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:44.561 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 78 16 103298 103282 82.7438 66.6875 0.00984196 0.0120806 2024-09-21T10:55:44.569 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:55:44.619 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:44.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:44 smithi070 ceph-mon[107385]: pgmap v2806: 109 pgs: 10 active+recovery_wait+degraded, 2 peering, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.0 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 49 MiB/s wr, 788 op/s; 32427/296247 objects degraded (10.946%); 2901/296247 objects misplaced (0.979%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:55:44.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:44 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2642936960' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:55:44.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:44 smithi070 ceph-mon[112139]: pgmap v2806: 109 pgs: 10 active+recovery_wait+degraded, 2 peering, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.0 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 49 MiB/s wr, 788 op/s; 32427/296247 objects degraded (10.946%); 2901/296247 objects misplaced (0.979%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:55:44.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:44 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2642936960' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:55:44.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:44.671 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:44.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:44.711 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:44 smithi160 ceph-mon[101879]: pgmap v2806: 109 pgs: 10 active+recovery_wait+degraded, 2 peering, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.0 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 49 MiB/s wr, 788 op/s; 32427/296247 objects degraded (10.946%); 2901/296247 objects misplaced (0.979%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:55:44.711 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:44 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2642936960' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:55:44.715 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:55:44.772 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:44.804 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:44.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:44.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:44.906 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:44.927 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:44.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:44.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:45.029 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:45.061 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:45.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:45.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:45.163 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:45.184 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:45.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:45.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:45.286 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:45.319 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:45.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:45.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:45.421 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:45.453 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:45.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:45.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:45.555 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:45.561 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 79 16 104409 104393 82.5753 69.4375 0.00743487 0.0121045 2024-09-21T10:55:45.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:45.588 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:45.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:45.689 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:45.719 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:45 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:55:45.719 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:45 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2642936960' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:55:45.719 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:45 smithi160 ceph-mon[101879]: osdmap e1502: 8 total, 8 up, 4 in 2024-09-21T10:55:45.726 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:45.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:45.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:45.828 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:45.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:45.868 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:45.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:45.869 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:55:45.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:45 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:55:45.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:45 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2642936960' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:55:45.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:45 smithi070 ceph-mon[107385]: osdmap e1502: 8 total, 8 up, 4 in 2024-09-21T10:55:45.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:45 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:55:45.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:45 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2642936960' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:55:45.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:45 smithi070 ceph-mon[112139]: osdmap e1502: 8 total, 8 up, 4 in 2024-09-21T10:55:45.969 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:46.006 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:46.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:46.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:46.108 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:46.135 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:46.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:46.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:46.237 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:46.270 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:46.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:46.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:46.372 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:46.401 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:46.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:46.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:46.503 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:46.526 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:46.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:46.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:46.561 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:2024-09-21T10:55:46.560285+0000 min lat: 0.00129408 max lat: 2.31521 avg lat: 0.0121375 2024-09-21T10:55:46.561 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:55:46.561 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 80 16 105441 105425 82.3493 64.5 0.00755458 0.0121375 2024-09-21T10:55:46.628 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:46.668 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:46.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:46.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:46.749 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:46 smithi160 ceph-mon[101879]: pgmap v2808: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.2 GiB data, 19 GiB used, 338 GiB / 358 GiB avail; 63 MiB/s wr, 1k op/s; 36648/305748 objects degraded (11.986%); 2665/305748 objects misplaced (0.872%); 4.7 MiB/s, 74 objects/s recovering 2024-09-21T10:55:46.771 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:46.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:46.815 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:46.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:46.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:46 smithi070 ceph-mon[107385]: pgmap v2808: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.2 GiB data, 19 GiB used, 338 GiB / 358 GiB avail; 63 MiB/s wr, 1k op/s; 36648/305748 objects degraded (11.986%); 2665/305748 objects misplaced (0.872%); 4.7 MiB/s, 74 objects/s recovering 2024-09-21T10:55:46.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:46 smithi070 ceph-mon[112139]: pgmap v2808: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.2 GiB data, 19 GiB used, 338 GiB / 358 GiB avail; 63 MiB/s wr, 1k op/s; 36648/305748 objects degraded (11.986%); 2665/305748 objects misplaced (0.872%); 4.7 MiB/s, 74 objects/s recovering 2024-09-21T10:55:46.916 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:46.937 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:46.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:46.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:47.039 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:47.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:47.061 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:47.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:47.162 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:47.197 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:47.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:47.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:47.298 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:47.326 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:47.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:47.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:47.428 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:47.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:47.478 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:47.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:47.561 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 81 16 106476 106460 82.1312 64.6875 0.0058814 0.0121709 2024-09-21T10:55:47.580 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:47.614 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:47.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:47.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:47.717 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:47.749 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:47.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:47.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:47.841 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:55:47.851 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:47.873 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:47.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:47.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:47.975 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:48.007 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:48.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:48.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:48.109 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:48.132 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:48.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:48.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:48.234 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:48.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:48.256 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:48.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:48.357 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:48.390 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:48.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:48.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:48.492 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:48.542 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:48.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:48.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:48.561 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 82 16 107481 107465 81.8956 62.8125 0.0166441 0.0122056 2024-09-21T10:55:48.644 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:48.680 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:48.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:48.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:48.771 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:48 smithi160 ceph-mon[101879]: pgmap v2809: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.2 GiB data, 19 GiB used, 338 GiB / 358 GiB avail; 45 MiB/s wr, 723 op/s; 36648/305748 objects degraded (11.986%); 2665/305748 objects misplaced (0.872%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T10:55:48.772 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:48 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 36648/305748 objects degraded (11.986%), 15 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:48.782 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:48.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:48.816 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:48.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:48.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:48 smithi070 ceph-mon[107385]: pgmap v2809: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.2 GiB data, 19 GiB used, 338 GiB / 358 GiB avail; 45 MiB/s wr, 723 op/s; 36648/305748 objects degraded (11.986%); 2665/305748 objects misplaced (0.872%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T10:55:48.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:48 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 36648/305748 objects degraded (11.986%), 15 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:48.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:48 smithi070 ceph-mon[112139]: pgmap v2809: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.2 GiB data, 19 GiB used, 338 GiB / 358 GiB avail; 45 MiB/s wr, 723 op/s; 36648/305748 objects degraded (11.986%); 2665/305748 objects misplaced (0.872%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T10:55:48.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:48 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 36648/305748 objects degraded (11.986%), 15 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:48.918 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:48.951 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:48.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:48.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:49.053 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:49.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:49.079 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:49.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:49.180 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:49.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:49.219 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:49.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:49.318 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:55:49.321 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:49.356 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:49.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:49.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:49.458 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:49.485 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:49.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:49.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:49.561 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 83 16 108610 108594 81.759 70.5625 0.00585199 0.0122262 2024-09-21T10:55:49.588 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:49.620 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:49.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:49.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:49.722 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:49.755 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:49.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:49.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:49.857 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:49.892 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:49.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:49.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:49.994 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:50.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:50.032 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:50.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:50.133 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:50.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:50.169 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:50.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:50.270 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:50.307 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:50.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:50.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:50.409 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:50.428 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:50 smithi160 ceph-mon[101879]: pgmap v2810: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.4 GiB data, 20 GiB used, 337 GiB / 358 GiB avail; 55 MiB/s wr, 885 op/s; 36648/316680 objects degraded (11.573%); 2481/316680 objects misplaced (0.783%); 2.8 MiB/s, 45 objects/s recovering 2024-09-21T10:55:50.428 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:55:50.428 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:55:50.447 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:50.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:50.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:50.549 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:50.561 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 84 16 109865 109849 81.7194 78.4375 0.0136929 0.0122318 2024-09-21T10:55:50.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:50.584 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:50.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:50.685 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:50.718 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:50.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:50.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:50.752 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:50 smithi070 ceph-mon[107385]: pgmap v2810: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.4 GiB data, 20 GiB used, 337 GiB / 358 GiB avail; 55 MiB/s wr, 885 op/s; 36648/316680 objects degraded (11.573%); 2481/316680 objects misplaced (0.783%); 2.8 MiB/s, 45 objects/s recovering 2024-09-21T10:55:50.752 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:55:50.752 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:55:50.753 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:50 smithi070 ceph-mon[112139]: pgmap v2810: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.4 GiB data, 20 GiB used, 337 GiB / 358 GiB avail; 55 MiB/s wr, 885 op/s; 36648/316680 objects degraded (11.573%); 2481/316680 objects misplaced (0.783%); 2.8 MiB/s, 45 objects/s recovering 2024-09-21T10:55:50.753 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:55:50.753 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:55:50.820 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:50.834 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:55:50.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:50.871 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:50.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:50.972 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:51.007 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:51.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:51.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:51.109 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:51.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:51.149 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:51.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:51.250 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:51.271 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:51.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:51.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:51.340 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:55:51.340 INFO:teuthology.orchestra.run.smithi160.stdout: "success": "" 2024-09-21T10:55:51.340 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:55:51.373 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:51.421 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:51.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:51.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:51.523 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:51.544 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:51.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:51.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:51.562 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 85 16 110828 110812 81.466 60.1875 0.00395323 0.0122625 2024-09-21T10:55:51.645 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:51.681 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:51.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:51.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:51.783 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:51.808 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:51.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:51.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:51.910 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:51.941 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:51.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:51.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:52.043 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:52.064 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:52.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:52.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:52.165 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:52.198 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:52.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:52.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:52.300 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:52.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:52.332 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:52.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:52.433 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:52.473 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:52.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:52.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:52.561 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 86 16 111894 111878 81.2934 66.625 0.0131226 0.0122958 2024-09-21T10:55:52.575 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:52.615 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:52.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:52.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:52.717 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:52.744 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:52.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:52.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:52.802 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:52 smithi160 ceph-mon[101879]: pgmap v2811: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.5 GiB data, 20 GiB used, 337 GiB / 358 GiB avail; 64 MiB/s wr, 1.02k op/s; 36648/321789 objects degraded (11.389%); 2481/321789 objects misplaced (0.771%); 2.7 MiB/s, 43 objects/s recovering 2024-09-21T10:55:52.846 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:52.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:52 smithi070 ceph-mon[107385]: pgmap v2811: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.5 GiB data, 20 GiB used, 337 GiB / 358 GiB avail; 64 MiB/s wr, 1.02k op/s; 36648/321789 objects degraded (11.389%); 2481/321789 objects misplaced (0.771%); 2.7 MiB/s, 43 objects/s recovering 2024-09-21T10:55:52.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:52 smithi070 ceph-mon[112139]: pgmap v2811: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.5 GiB data, 20 GiB used, 337 GiB / 358 GiB avail; 64 MiB/s wr, 1.02k op/s; 36648/321789 objects degraded (11.389%); 2481/321789 objects misplaced (0.771%); 2.7 MiB/s, 43 objects/s recovering 2024-09-21T10:55:52.879 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:52.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:52.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:52.981 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:53.020 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:53.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:53.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:53.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:53.148 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:53.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:53.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:53.250 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:53.288 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:53.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:53.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:53.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:55:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:55:53] "GET /metrics HTTP/1.1" 200 36626 "" "Prometheus/2.51.0" 2024-09-21T10:55:53.383 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:55:53.383 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:55:53.390 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:53.427 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:53.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:53.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:53.529 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:53.562 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 87 16 112866 112850 81.0572 60.75 0.00849587 0.0123321 2024-09-21T10:55:53.575 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:53.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:53.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:53.677 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:53.703 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:53 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 36648/321789 objects degraded (11.389%), 15 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:53.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:53 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 36648/321789 objects degraded (11.389%), 15 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:53.726 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:53 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 36648/321789 objects degraded (11.389%), 15 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:53.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:53.731 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:53.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:53.832 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:53.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:53.865 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:53.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:53.913 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:55:53.914 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:55:53.943 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:55:53.944 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:55:53.966 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:53.973 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:55:53.973 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:55:54.002 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:55:54.002 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:55:54.004 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:54.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:54.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:54.035 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:55:54.035 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:55:54.063 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:55:54.065 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:55:54.091 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:55:54.091 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:55:54.106 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:54.121 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:55:54.121 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:55:54.139 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:54.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:54.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:54.241 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:54.275 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:54.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:54.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:54.377 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:54.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:54.415 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:54.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:54.424 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:55:54.515 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:54.562 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 88 16 113959 113943 80.9121 68.3125 0.0480939 0.0123526 2024-09-21T10:55:54.564 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:54.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:54.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:54.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:54 smithi070 ceph-mon[112139]: pgmap v2812: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.7 GiB data, 21 GiB used, 337 GiB / 358 GiB avail; 68 MiB/s wr, 1.09k op/s; 36648/328983 objects degraded (11.140%); 2481/328983 objects misplaced (0.754%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:55:54.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:54 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2575052075' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:55:54.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:54 smithi070 ceph-mon[107385]: pgmap v2812: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.7 GiB data, 21 GiB used, 337 GiB / 358 GiB avail; 68 MiB/s wr, 1.09k op/s; 36648/328983 objects degraded (11.140%); 2481/328983 objects misplaced (0.754%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:55:54.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:54 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2575052075' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:55:54.666 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:54.700 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:54.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:54.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:54.786 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:54 smithi160 ceph-mon[101879]: pgmap v2812: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.7 GiB data, 21 GiB used, 337 GiB / 358 GiB avail; 68 MiB/s wr, 1.09k op/s; 36648/328983 objects degraded (11.140%); 2481/328983 objects misplaced (0.754%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:55:54.786 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:54 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2575052075' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:55:54.803 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:54.835 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:54.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:54.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:54.937 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:54.981 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:54.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:54.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:55.082 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:55.118 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:55.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:55.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:55.220 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:55.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:55.259 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:55.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:55.360 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:55.399 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:55.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:55.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:55.465 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:55:55.501 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:55.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:55.534 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:55.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:55.562 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 89 16 115131 115115 80.826 73.25 0.00530733 0.0123673 2024-09-21T10:55:55.634 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:55.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:55.675 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:55.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:55.776 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:55.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:55.815 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:55.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:55.916 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:55.950 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:55.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:55.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:56.051 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:56.094 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:56.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:56.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:56.195 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:56.228 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:56.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:56.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:56.330 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:56.372 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:56.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:56.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:56.474 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:56.510 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:56.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:56.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:56.562 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Total time run: 90.0079 2024-09-21T10:55:56.562 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Total writes made: 116218 2024-09-21T10:55:56.563 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Write size: 65536 2024-09-21T10:55:56.563 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Object size: 65536 2024-09-21T10:55:56.563 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Bandwidth (MB/sec): 80.6999 2024-09-21T10:55:56.563 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Stddev Bandwidth: 45.1701 2024-09-21T10:55:56.563 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Max bandwidth (MB/sec): 256.188 2024-09-21T10:55:56.563 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Min bandwidth (MB/sec): 0 2024-09-21T10:55:56.563 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Average IOPS: 1291 2024-09-21T10:55:56.563 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Stddev IOPS: 722.732 2024-09-21T10:55:56.563 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Max IOPS: 4099 2024-09-21T10:55:56.563 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Min IOPS: 0 2024-09-21T10:55:56.563 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Average Latency(s): 0.012387 2024-09-21T10:55:56.563 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Stddev Latency(s): 0.0326988 2024-09-21T10:55:56.563 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Max latency(s): 2.31521 2024-09-21T10:55:56.563 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Min latency(s): 0.00129408 2024-09-21T10:55:56.565 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T10:55:56.612 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:56.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:56.676 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:56.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:56.777 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:56.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:56.815 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:56.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:56 smithi160 ceph-mon[101879]: pgmap v2813: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.9 GiB data, 22 GiB used, 336 GiB / 358 GiB avail; 63 MiB/s wr, 1.01k op/s; 36648/337962 objects degraded (10.844%); 2279/337962 objects misplaced (0.674%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T10:55:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:56 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:55:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:56 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:55:56.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:56 smithi070 ceph-mon[107385]: pgmap v2813: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.9 GiB data, 22 GiB used, 336 GiB / 358 GiB avail; 63 MiB/s wr, 1.01k op/s; 36648/337962 objects degraded (10.844%); 2279/337962 objects misplaced (0.674%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T10:55:56.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:56 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:55:56.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:56 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:55:56.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:56 smithi070 ceph-mon[112139]: pgmap v2813: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.9 GiB data, 22 GiB used, 336 GiB / 358 GiB avail; 63 MiB/s wr, 1.01k op/s; 36648/337962 objects degraded (10.844%); 2279/337962 objects misplaced (0.674%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T10:55:56.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:56 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:55:56.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:56 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:55:56.917 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:56.955 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:56.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:56.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:57.054 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7] out_osds: [2, 5, 0, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:55:57.054 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:55:57.055 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_3 2024-09-21T10:55:57.055 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:55:57.057 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:57.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:57.107 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:57.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:57.208 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:57.235 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:57.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:57.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:57.337 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:57.365 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:57.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:57.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:57.447 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:55:57.468 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:57.529 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:57.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:57.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:57.631 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:55:57.637 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T10:55:57.684 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:57.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:57.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:55:57.787 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:57.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:57.832 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:57.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:57.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:57 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1280301648' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:55:57.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:57 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:55:57.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:57 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1280301648' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:55:57.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:57 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:55:57.933 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:57.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:57.972 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:57.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:58.073 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:57 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1280301648' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:55:58.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:57 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:55:58.105 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:58.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:58.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:58.207 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:55:58.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:58.254 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:58.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:55:58.355 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:58.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:58.392 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:58.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:58.493 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:58.522 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:58.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:58.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:58.624 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:58.662 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:58.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:58.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:58.669 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:55:58.764 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:58.765 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:58 smithi160 ceph-mon[101879]: pgmap v2814: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.9 GiB data, 22 GiB used, 336 GiB / 358 GiB avail; 56 MiB/s wr, 894 op/s; 36648/337962 objects degraded (10.844%); 2279/337962 objects misplaced (0.674%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T10:55:58.765 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:58 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:55:58.765 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:58 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:55:58.765 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:58 smithi160 ceph-mon[101879]: osdmap e1503: 8 total, 8 up, 4 in 2024-09-21T10:55:58.765 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:58 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:55:58.765 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:58 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:55:58.765 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:58 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 36648/337962 objects degraded (10.844%), 15 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:58.765 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:58 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:55:58.765 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:58 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:55:58.765 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:55:58 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:55:58.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:58.802 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:58.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:58.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:58 smithi070 ceph-mon[107385]: pgmap v2814: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.9 GiB data, 22 GiB used, 336 GiB / 358 GiB avail; 56 MiB/s wr, 894 op/s; 36648/337962 objects degraded (10.844%); 2279/337962 objects misplaced (0.674%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T10:55:58.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:58 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:55:58.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:58 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:55:58.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:58 smithi070 ceph-mon[107385]: osdmap e1503: 8 total, 8 up, 4 in 2024-09-21T10:55:58.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:58 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:55:58.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:58 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:55:58.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:58 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 36648/337962 objects degraded (10.844%), 15 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:58.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:58 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:55:58.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:58 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:55:58.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:55:58 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:55:58.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:58 smithi070 ceph-mon[112139]: pgmap v2814: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.9 GiB data, 22 GiB used, 336 GiB / 358 GiB avail; 56 MiB/s wr, 894 op/s; 36648/337962 objects degraded (10.844%); 2279/337962 objects misplaced (0.674%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T10:55:58.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:58 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:55:58.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:58 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:55:58.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:58 smithi070 ceph-mon[112139]: osdmap e1503: 8 total, 8 up, 4 in 2024-09-21T10:55:58.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:58 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:55:58.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:58 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:55:58.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:58 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 36648/337962 objects degraded (10.844%), 15 pgs degraded (PG_DEGRADED) 2024-09-21T10:55:58.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:58 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:55:58.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:58 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:55:58.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:55:58 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:55:58.903 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:55:58.927 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:55:58.927 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T10:55:58.927 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T10:55:58.927 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T10:55:58.927 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.927 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.35670.0:72212 55.a e1425/1411 55:51f2ed65:::benchmark_data_smithi070_294394_object72211:head v 1425'4475, mlcod=1425'4473)", 2024-09-21T10:55:58.927 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:49:40.176758+0000", 2024-09-21T10:55:58.927 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 378.745593982, 2024-09-21T10:55:58.928 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.29508102600000002, 2024-09-21T10:55:58.928 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:55:58.928 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:58.928 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:55:58.928 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.928 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:55:58.928 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.176758+0000", 2024-09-21T10:55:58.928 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:58.928 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.928 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.928 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:55:58.928 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.176758+0000", 2024-09-21T10:55:58.929 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:58.929 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.929 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.929 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:55:58.929 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.176757+0000", 2024-09-21T10:55:58.929 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:55:58.929 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.929 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.929 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:55:58.929 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.176788+0000", 2024-09-21T10:55:58.929 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0559000000000003e-05 2024-09-21T10:55:58.929 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.929 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.929 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:55:58.930 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.176800+0000", 2024-09-21T10:55:58.930 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1918e-05 2024-09-21T10:55:58.930 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.930 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.930 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:55:58.930 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.176807+0000", 2024-09-21T10:55:58.930 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2640000000000002e-06 2024-09-21T10:55:58.930 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.930 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.930 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:58.930 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.176854+0000", 2024-09-21T10:55:58.930 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6476e-05 2024-09-21T10:55:58.931 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.931 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.931 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:55:58.931 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.176873+0000", 2024-09-21T10:55:58.931 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9624999999999999e-05 2024-09-21T10:55:58.931 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.931 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.931 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:55:58.931 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.471800+0000", 2024-09-21T10:55:58.931 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.29492674299999999 2024-09-21T10:55:58.931 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.931 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.931 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:55:58.931 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.471839+0000", 2024-09-21T10:55:58.932 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9310999999999997e-05 2024-09-21T10:55:58.932 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:58.932 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:55:58.932 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:58.932 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.932 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.932 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.35670.0:72213 55.a e1425/1411 55:551f1baf:::benchmark_data_smithi070_294394_object72212:head v 1425'4476, mlcod=1425'4473)", 2024-09-21T10:55:58.932 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:49:40.178762+0000", 2024-09-21T10:55:58.932 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 378.743590348, 2024-09-21T10:55:58.932 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.293101098, 2024-09-21T10:55:58.932 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:55:58.932 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:58.932 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:55:58.932 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.933 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:55:58.933 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.178762+0000", 2024-09-21T10:55:58.933 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:58.933 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.933 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.933 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:55:58.933 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.178762+0000", 2024-09-21T10:55:58.933 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:58.933 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.933 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.933 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:55:58.933 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.178761+0000", 2024-09-21T10:55:58.933 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:55:58.933 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.934 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.934 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:55:58.934 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.178869+0000", 2024-09-21T10:55:58.934 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000108661 2024-09-21T10:55:58.934 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.934 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.934 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:55:58.934 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.178883+0000", 2024-09-21T10:55:58.934 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3338e-05 2024-09-21T10:55:58.934 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.934 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.934 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:55:58.935 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.178890+0000", 2024-09-21T10:55:58.935 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6159999999999996e-06 2024-09-21T10:55:58.935 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.935 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.935 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:58.935 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.179248+0000", 2024-09-21T10:55:58.935 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00035803700000000001 2024-09-21T10:55:58.935 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.935 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.935 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:55:58.935 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.179267+0000", 2024-09-21T10:55:58.935 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9004999999999998e-05 2024-09-21T10:55:58.935 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.935 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.936 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:55:58.936 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.471843+0000", 2024-09-21T10:55:58.936 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.292575843 2024-09-21T10:55:58.936 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.936 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.936 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:55:58.936 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.471863+0000", 2024-09-21T10:55:58.936 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9646000000000001e-05 2024-09-21T10:55:58.936 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:58.936 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:55:58.936 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:58.936 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.936 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.937 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.35670.0:72214 55.1 e1425/1411 55:8b604f8f:::benchmark_data_smithi070_294394_object72213:head v 1425'4519, mlcod=1425'4516)", 2024-09-21T10:55:58.937 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:49:40.179964+0000", 2024-09-21T10:55:58.937 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 378.74238826099997, 2024-09-21T10:55:58.937 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30231281900000001, 2024-09-21T10:55:58.937 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:55:58.937 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:58.937 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:55:58.937 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.937 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:55:58.937 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.179964+0000", 2024-09-21T10:55:58.937 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:58.937 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.937 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.938 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:55:58.938 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.179964+0000", 2024-09-21T10:55:58.938 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:58.938 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.938 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.938 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:55:58.938 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.179963+0000", 2024-09-21T10:55:58.938 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:55:58.938 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.938 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.938 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:55:58.938 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180001+0000", 2024-09-21T10:55:58.938 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8398000000000001e-05 2024-09-21T10:55:58.939 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.939 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.939 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:55:58.939 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180012+0000", 2024-09-21T10:55:58.939 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0916e-05 2024-09-21T10:55:58.939 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.939 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.939 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:55:58.939 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180020+0000", 2024-09-21T10:55:58.939 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2030000000000003e-06 2024-09-21T10:55:58.939 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.939 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.940 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:58.940 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180093+0000", 2024-09-21T10:55:58.940 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3728999999999998e-05 2024-09-21T10:55:58.940 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.940 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.940 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:55:58.940 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180114+0000", 2024-09-21T10:55:58.940 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0322999999999998e-05 2024-09-21T10:55:58.940 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.940 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.940 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:55:58.941 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.482168+0000", 2024-09-21T10:55:58.941 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30205438699999998 2024-09-21T10:55:58.941 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.941 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.941 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:55:58.941 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.482277+0000", 2024-09-21T10:55:58.941 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000108762 2024-09-21T10:55:58.941 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:58.941 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:55:58.942 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:58.942 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.942 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.942 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.35670.0:72216 55.c e1425/1411 55:33c7533d:::benchmark_data_smithi070_294394_object72215:head v 1425'4484, mlcod=1425'4481)", 2024-09-21T10:55:58.942 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:49:40.180613+0000", 2024-09-21T10:55:58.942 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 378.741738919, 2024-09-21T10:55:58.942 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.29111427499999998, 2024-09-21T10:55:58.942 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:55:58.943 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:58.943 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:55:58.943 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.943 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:55:58.943 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180613+0000", 2024-09-21T10:55:58.943 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:58.943 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.944 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.944 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:55:58.944 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180613+0000", 2024-09-21T10:55:58.944 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:58.944 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.944 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.944 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:55:58.944 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180612+0000", 2024-09-21T10:55:58.944 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:55:58.944 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.944 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.944 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:55:58.944 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180726+0000", 2024-09-21T10:55:58.945 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000113441 2024-09-21T10:55:58.945 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.945 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.945 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:55:58.945 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180749+0000", 2024-09-21T10:55:58.945 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2691999999999999e-05 2024-09-21T10:55:58.945 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.945 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.945 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:55:58.945 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180755+0000", 2024-09-21T10:55:58.945 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2179999999999996e-06 2024-09-21T10:55:58.945 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.945 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.946 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:58.946 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180798+0000", 2024-09-21T10:55:58.946 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3201000000000002e-05 2024-09-21T10:55:58.946 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.946 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.946 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:55:58.946 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180817+0000", 2024-09-21T10:55:58.946 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9369999999999999e-05 2024-09-21T10:55:58.946 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.946 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.946 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:55:58.946 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.471681+0000", 2024-09-21T10:55:58.947 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.29086372100000002 2024-09-21T10:55:58.947 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.947 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.947 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:55:58.947 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.471728+0000", 2024-09-21T10:55:58.947 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6445e-05 2024-09-21T10:55:58.947 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:58.947 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:55:58.947 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:58.947 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.947 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.947 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.35670.0:72217 55.9 e1425/1411 55:977475fe:::benchmark_data_smithi070_294394_object72216:head v 1425'4465, mlcod=1425'4463)", 2024-09-21T10:55:58.947 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:49:40.180772+0000", 2024-09-21T10:55:58.947 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 378.74158027099998, 2024-09-21T10:55:58.948 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30159776900000002, 2024-09-21T10:55:58.948 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:55:58.948 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:58.948 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:55:58.948 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.948 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:55:58.948 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180772+0000", 2024-09-21T10:55:58.948 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:58.948 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.948 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.948 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:55:58.948 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180772+0000", 2024-09-21T10:55:58.948 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:58.949 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.949 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.949 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:55:58.949 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180772+0000", 2024-09-21T10:55:58.949 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T10:55:58.949 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.949 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.949 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:55:58.949 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180802+0000", 2024-09-21T10:55:58.949 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9989e-05 2024-09-21T10:55:58.949 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.949 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.949 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:55:58.949 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180808+0000", 2024-09-21T10:55:58.950 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6549999999999998e-06 2024-09-21T10:55:58.950 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.950 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.950 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:55:58.950 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180812+0000", 2024-09-21T10:55:58.950 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2950000000000004e-06 2024-09-21T10:55:58.950 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.950 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.950 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:58.950 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180862+0000", 2024-09-21T10:55:58.950 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9017000000000002e-05 2024-09-21T10:55:58.950 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.951 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.951 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:55:58.951 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.180880+0000", 2024-09-21T10:55:58.951 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8374999999999999e-05 2024-09-21T10:55:58.951 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.951 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.951 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:55:58.951 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.482283+0000", 2024-09-21T10:55:58.951 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30140297199999999 2024-09-21T10:55:58.951 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.951 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.951 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:55:58.951 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.482370+0000", 2024-09-21T10:55:58.951 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6848000000000006e-05 2024-09-21T10:55:58.951 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:58.952 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:55:58.952 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:58.952 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.952 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.952 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.35670.0:72215 55.d e1425/1411 55:b697a537:::benchmark_data_smithi070_294394_object72214:head v 1425'4658, mlcod=1425'4656)", 2024-09-21T10:55:58.952 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:49:40.182534+0000", 2024-09-21T10:55:58.952 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 378.73981836799999, 2024-09-21T10:55:58.952 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.29417858400000002, 2024-09-21T10:55:58.952 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:55:58.952 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:58.952 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:55:58.952 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.952 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:55:58.952 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.182534+0000", 2024-09-21T10:55:58.952 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:58.953 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.953 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.953 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:55:58.953 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.182534+0000", 2024-09-21T10:55:58.953 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:58.953 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.953 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.953 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:55:58.953 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.182532+0000", 2024-09-21T10:55:58.953 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:55:58.953 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.953 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.953 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:55:58.954 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.182576+0000", 2024-09-21T10:55:58.954 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3553999999999997e-05 2024-09-21T10:55:58.954 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.954 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.954 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:55:58.954 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.471751+0000", 2024-09-21T10:55:58.954 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.28917540600000002 2024-09-21T10:55:58.954 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.954 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.954 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:55:58.954 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.471764+0000", 2024-09-21T10:55:58.954 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2503e-05 2024-09-21T10:55:58.954 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.954 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.955 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:58.955 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.471815+0000", 2024-09-21T10:55:58.955 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1384000000000001e-05 2024-09-21T10:55:58.955 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.955 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.955 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:55:58.955 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.471842+0000", 2024-09-21T10:55:58.955 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6721e-05 2024-09-21T10:55:58.955 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.955 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.955 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:55:58.955 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.476625+0000", 2024-09-21T10:55:58.955 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.004782573 2024-09-21T10:55:58.956 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.956 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.956 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:55:58.956 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:40.476712+0000", 2024-09-21T10:55:58.956 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7843000000000006e-05 2024-09-21T10:55:58.956 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:58.956 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:55:58.956 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:58.956 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.956 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.956 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:129299 55.d 55:b83dbe9e:::benchmark_data_smithi070_294394_object23958:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:55:58.956 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:22.715175+0000", 2024-09-21T10:55:58.956 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 276.20717741300001, 2024-09-21T10:55:58.957 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.25006946699999999, 2024-09-21T10:55:58.957 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:55:58.957 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:58.957 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:55:58.957 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:55:58.957 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:58.957 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 129299 2024-09-21T10:55:58.957 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.957 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:55:58.957 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.957 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:55:58.957 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.715175+0000", 2024-09-21T10:55:58.957 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:58.958 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.958 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.958 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:55:58.958 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.715175+0000", 2024-09-21T10:55:58.958 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:58.958 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.958 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.958 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:55:58.958 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.715173+0000", 2024-09-21T10:55:58.959 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:55:58.959 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.959 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.959 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:55:58.959 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.715180+0000", 2024-09-21T10:55:58.959 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0269999999999998e-06 2024-09-21T10:55:58.959 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.959 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.959 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:55:58.959 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.715193+0000", 2024-09-21T10:55:58.959 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3064000000000001e-05 2024-09-21T10:55:58.960 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.960 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.960 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:55:58.960 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.715200+0000", 2024-09-21T10:55:58.960 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9059999999999996e-06 2024-09-21T10:55:58.960 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.960 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.960 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:58.960 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.718977+0000", 2024-09-21T10:55:58.960 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0037772579999999999 2024-09-21T10:55:58.960 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.960 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.960 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:55:58.961 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.719152+0000", 2024-09-21T10:55:58.961 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017478 2024-09-21T10:55:58.961 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.961 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.961 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:55:58.961 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.719313+0000", 2024-09-21T10:55:58.961 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000161185 2024-09-21T10:55:58.961 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.961 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.961 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:55:58.961 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.722530+0000", 2024-09-21T10:55:58.961 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0032165689999999999 2024-09-21T10:55:58.961 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.961 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.962 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:58.962 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.726307+0000", 2024-09-21T10:55:58.962 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0037772439999999999 2024-09-21T10:55:58.962 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.962 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.962 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:58.962 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.960947+0000", 2024-09-21T10:55:58.962 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.23464001400000001 2024-09-21T10:55:58.962 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.962 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.962 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:55:58.962 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.960996+0000", 2024-09-21T10:55:58.962 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8819999999999997e-05 2024-09-21T10:55:58.962 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.963 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.963 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:55:58.963 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.965244+0000", 2024-09-21T10:55:58.963 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0042483529999999999 2024-09-21T10:55:58.963 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:58.963 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:55:58.963 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:58.963 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.963 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.963 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:129315 55.d 55:bf0d41f3:::benchmark_data_smithi070_294394_object23974:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:55:58.963 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:22.765937+0000", 2024-09-21T10:55:58.963 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 276.15641533500002, 2024-09-21T10:55:58.963 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20239199699999999, 2024-09-21T10:55:58.963 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:55:58.964 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:58.964 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:55:58.964 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:55:58.964 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:58.964 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 129315 2024-09-21T10:55:58.964 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.964 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:55:58.964 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.964 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:55:58.965 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.765937+0000", 2024-09-21T10:55:58.965 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:58.965 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.965 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.965 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:55:58.965 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.765937+0000", 2024-09-21T10:55:58.965 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:58.965 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.965 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.965 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:55:58.965 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.765935+0000", 2024-09-21T10:55:58.965 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:55:58.965 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.965 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.966 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:55:58.966 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.765945+0000", 2024-09-21T10:55:58.966 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8879999999999992e-06 2024-09-21T10:55:58.966 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.966 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.966 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:55:58.966 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.765959+0000", 2024-09-21T10:55:58.966 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3892e-05 2024-09-21T10:55:58.966 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.966 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.966 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:55:58.966 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.765967+0000", 2024-09-21T10:55:58.966 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4379999999999998e-06 2024-09-21T10:55:58.966 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.966 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.967 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:58.967 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.766010+0000", 2024-09-21T10:55:58.967 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3838999999999998e-05 2024-09-21T10:55:58.967 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.967 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.967 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:55:58.967 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.766151+0000", 2024-09-21T10:55:58.967 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014064700000000001 2024-09-21T10:55:58.967 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.967 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.967 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:55:58.967 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.766258+0000", 2024-09-21T10:55:58.967 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000107228 2024-09-21T10:55:58.967 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.968 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.968 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:55:58.968 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.768407+0000", 2024-09-21T10:55:58.968 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021486399999999998 2024-09-21T10:55:58.968 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.968 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.968 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:58.968 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.772605+0000", 2024-09-21T10:55:58.968 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0041983259999999996 2024-09-21T10:55:58.968 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.968 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.968 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:58.968 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.965314+0000", 2024-09-21T10:55:58.969 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.192708883 2024-09-21T10:55:58.969 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.969 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.969 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:55:58.969 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.965373+0000", 2024-09-21T10:55:58.969 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8919000000000003e-05 2024-09-21T10:55:58.969 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.969 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.969 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:55:58.969 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:22.968329+0000", 2024-09-21T10:55:58.969 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002955877 2024-09-21T10:55:58.969 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:58.969 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:55:58.969 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:58.969 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.970 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.970 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:130860 55.d 55:b4ee1c49:::benchmark_data_smithi070_294394_object25519:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:55:58.970 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:26.394761+0000", 2024-09-21T10:55:58.970 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 272.52759165600003, 2024-09-21T10:55:58.970 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3490294309999999, 2024-09-21T10:55:58.970 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:55:58.970 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:55:58.970 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:55:58.970 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:55:58.970 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:58.970 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 130860 2024-09-21T10:55:58.970 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.970 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:55:58.970 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.970 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:55:58.971 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.394761+0000", 2024-09-21T10:55:58.971 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:58.971 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.971 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.971 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:55:58.971 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.394761+0000", 2024-09-21T10:55:58.971 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:58.971 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.971 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.971 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:55:58.971 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.394758+0000", 2024-09-21T10:55:58.971 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:55:58.971 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.971 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.972 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:55:58.972 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.394768+0000", 2024-09-21T10:55:58.972 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2429999999999994e-06 2024-09-21T10:55:58.972 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.972 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.972 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:55:58.972 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.394788+0000", 2024-09-21T10:55:58.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:58.973 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:58.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:55:58.974 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9777000000000001e-05 2024-09-21T10:55:58.974 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.974 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.974 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:55:58.974 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.394796+0000", 2024-09-21T10:55:58.974 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8040000000000007e-06 2024-09-21T10:55:58.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.975 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.975 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:58.975 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.394852+0000", 2024-09-21T10:55:58.975 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5896999999999997e-05 2024-09-21T10:55:58.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.975 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.975 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:55:58.975 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.395006+0000", 2024-09-21T10:55:58.975 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015377699999999999 2024-09-21T10:55:58.976 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.976 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.976 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:55:58.976 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.395149+0000", 2024-09-21T10:55:58.976 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000143514 2024-09-21T10:55:58.976 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.976 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.976 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:55:58.977 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.397840+0000", 2024-09-21T10:55:58.977 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0026902039999999999 2024-09-21T10:55:58.977 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.977 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.977 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:58.977 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.414884+0000", 2024-09-21T10:55:58.977 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.017044549999999999 2024-09-21T10:55:58.977 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.977 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.977 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:58.977 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.429059+0000", 2024-09-21T10:55:58.977 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.014175128 2024-09-21T10:55:58.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.978 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.978 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:55:58.978 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.429074+0000", 2024-09-21T10:55:58.978 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4436e-05 2024-09-21T10:55:58.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.978 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.978 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:58.978 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.427445+0000", 2024-09-21T10:55:58.978 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99837087800000002 2024-09-21T10:55:58.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.978 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.979 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:55:58.979 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.427457+0000", 2024-09-21T10:55:58.979 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2643000000000001e-05 2024-09-21T10:55:58.979 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.979 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:58.979 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.691524+0000", 2024-09-21T10:55:58.979 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2640668900000001 2024-09-21T10:55:58.979 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.979 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for missing object", 2024-09-21T10:55:58.979 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.691640+0000", 2024-09-21T10:55:58.979 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000116205 2024-09-21T10:55:58.980 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.980 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.980 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:58.980 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.743637+0000", 2024-09-21T10:55:58.980 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.051996268999999998 2024-09-21T10:55:58.980 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.980 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.980 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:55:58.980 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.743790+0000", 2024-09-21T10:55:58.980 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015328100000000001 2024-09-21T10:55:58.980 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:58.980 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:55:58.980 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:58.981 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.981 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.981 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:130862 55.d 55:bd7eda03:::benchmark_data_smithi070_294394_object25521:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:55:58.981 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:26.398474+0000", 2024-09-21T10:55:58.981 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 272.523878212, 2024-09-21T10:55:58.981 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3783652179999999, 2024-09-21T10:55:58.981 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:55:58.981 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:55:58.981 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:55:58.981 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:55:58.981 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:58.981 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 130862 2024-09-21T10:55:58.981 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.981 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:55:58.982 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.982 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:55:58.982 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398474+0000", 2024-09-21T10:55:58.982 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:58.982 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.982 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.982 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:55:58.982 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398474+0000", 2024-09-21T10:55:58.982 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:58.982 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.982 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.982 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:55:58.982 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398472+0000", 2024-09-21T10:55:58.982 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:55:58.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.983 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.983 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:55:58.983 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398481+0000", 2024-09-21T10:55:58.983 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1589999999999993e-06 2024-09-21T10:55:58.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.983 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.983 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:55:58.983 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398495+0000", 2024-09-21T10:55:58.983 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4829e-05 2024-09-21T10:55:58.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.983 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.983 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:55:58.983 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398503+0000", 2024-09-21T10:55:58.984 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1499999999999999e-06 2024-09-21T10:55:58.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.984 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:58.985 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398549+0000", 2024-09-21T10:55:58.985 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5802e-05 2024-09-21T10:55:58.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.985 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.985 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:55:58.985 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398680+0000", 2024-09-21T10:55:58.985 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000130784 2024-09-21T10:55:58.986 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.986 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.986 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:55:58.986 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398797+0000", 2024-09-21T10:55:58.986 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000117408 2024-09-21T10:55:58.986 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.986 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.986 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:55:58.986 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.402741+0000", 2024-09-21T10:55:58.986 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0039439920000000003 2024-09-21T10:55:58.986 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.986 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.986 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:58.986 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.414917+0000", 2024-09-21T10:55:58.987 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012175314 2024-09-21T10:55:58.987 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.987 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:58.987 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.430736+0000", 2024-09-21T10:55:58.987 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.015818895999999999 2024-09-21T10:55:58.987 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.987 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:55:58.987 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.430747+0000", 2024-09-21T10:55:58.987 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1589e-05 2024-09-21T10:55:58.987 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.987 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:58.987 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.427498+0000", 2024-09-21T10:55:58.988 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99675095499999999 2024-09-21T10:55:58.988 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.988 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.988 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:55:58.988 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.427507+0000", 2024-09-21T10:55:58.988 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9830000000000002e-06 2024-09-21T10:55:58.988 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.988 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.988 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:58.988 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.750176+0000", 2024-09-21T10:55:58.988 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3226684440000001 2024-09-21T10:55:58.988 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.988 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.988 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for missing object", 2024-09-21T10:55:58.989 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.750312+0000", 2024-09-21T10:55:58.989 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013604399999999999 2024-09-21T10:55:58.989 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.989 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.989 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:58.989 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.776747+0000", 2024-09-21T10:55:58.989 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.026435199 2024-09-21T10:55:58.989 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.989 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.989 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:55:58.989 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.776839+0000", 2024-09-21T10:55:58.989 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2300999999999996e-05 2024-09-21T10:55:58.989 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:58.989 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:55:58.990 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:58.990 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.990 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.990 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:130864 55.d 55:bf76459a:::benchmark_data_smithi070_294394_object25523:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:55:58.990 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:26.398512+0000", 2024-09-21T10:55:58.990 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 272.52383971500001, 2024-09-21T10:55:58.990 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.415816907, 2024-09-21T10:55:58.990 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:55:58.990 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:55:58.990 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:55:58.990 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:55:58.990 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:58.990 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 130864 2024-09-21T10:55:58.991 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.991 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:55:58.991 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.991 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:55:58.991 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398512+0000", 2024-09-21T10:55:58.991 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:58.991 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.991 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.991 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:55:58.991 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398512+0000", 2024-09-21T10:55:58.991 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:58.991 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.991 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.992 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:55:58.992 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398512+0000", 2024-09-21T10:55:58.992 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T10:55:58.992 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.992 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.992 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:55:58.992 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398515+0000", 2024-09-21T10:55:58.992 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4679999999999999e-06 2024-09-21T10:55:58.992 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.992 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.992 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:55:58.992 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398530+0000", 2024-09-21T10:55:58.992 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5201000000000001e-05 2024-09-21T10:55:58.993 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.993 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.993 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:55:58.993 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398535+0000", 2024-09-21T10:55:58.993 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0250000000000002e-06 2024-09-21T10:55:58.993 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.993 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.993 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:58.993 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.400838+0000", 2024-09-21T10:55:58.993 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0023031710000000001 2024-09-21T10:55:58.994 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.994 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.994 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:55:58.994 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.400934+0000", 2024-09-21T10:55:58.994 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6107999999999995e-05 2024-09-21T10:55:58.994 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.994 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.994 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:55:58.994 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.401033+0000", 2024-09-21T10:55:58.994 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9264000000000004e-05 2024-09-21T10:55:58.994 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.994 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.994 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:55:58.994 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.402893+0000", 2024-09-21T10:55:58.995 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0018592560000000001 2024-09-21T10:55:58.995 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.995 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.995 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:58.995 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.430776+0000", 2024-09-21T10:55:58.995 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.027883080000000001 2024-09-21T10:55:58.995 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.995 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.995 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:55:58.995 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.430791+0000", 2024-09-21T10:55:58.995 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5741999999999999e-05 2024-09-21T10:55:58.995 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.995 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.995 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:58.995 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.427535+0000", 2024-09-21T10:55:58.996 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99674326099999999 2024-09-21T10:55:58.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.996 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.996 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:55:58.996 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.427543+0000", 2024-09-21T10:55:58.996 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5269999999999998e-06 2024-09-21T10:55:58.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.996 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.996 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:58.996 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.750345+0000", 2024-09-21T10:55:58.996 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.322801291 2024-09-21T10:55:58.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.996 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.996 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for missing object", 2024-09-21T10:55:58.997 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.750449+0000", 2024-09-21T10:55:58.997 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000104609 2024-09-21T10:55:58.997 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.997 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.997 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:58.997 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.814210+0000", 2024-09-21T10:55:58.997 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.063760777000000005 2024-09-21T10:55:58.997 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.997 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.997 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:55:58.997 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.814329+0000", 2024-09-21T10:55:58.997 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119499 2024-09-21T10:55:58.997 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:58.997 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:55:58.998 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:58.998 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.998 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.998 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:130888 55.d 55:b8d17701:::benchmark_data_smithi070_294394_object25547:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1449)", 2024-09-21T10:55:58.998 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:28.580660+0000", 2024-09-21T10:55:58.998 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 270.34169214600001, 2024-09-21T10:55:58.998 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.27647097700000001, 2024-09-21T10:55:58.998 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:55:58.998 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:58.998 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:55:58.998 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:55:58.998 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:58.998 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 130888 2024-09-21T10:55:58.998 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.999 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:55:58.999 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.999 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:55:58.999 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.580660+0000", 2024-09-21T10:55:58.999 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:58.999 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.999 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.999 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:55:58.999 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.580660+0000", 2024-09-21T10:55:58.999 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:58.999 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:58.999 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:58.999 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:55:59.000 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.580658+0000", 2024-09-21T10:55:59.000 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 2024-09-21T10:55:59.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.000 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:55:59.000 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.580669+0000", 2024-09-21T10:55:59.000 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1199000000000001e-05 2024-09-21T10:55:59.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.000 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:55:59.000 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.580685+0000", 2024-09-21T10:55:59.000 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5889999999999999e-05 2024-09-21T10:55:59.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.000 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:55:59.001 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.580703+0000", 2024-09-21T10:55:59.001 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8664e-05 2024-09-21T10:55:59.001 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.001 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.001 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.001 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.751302+0000", 2024-09-21T10:55:59.001 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.170598571 2024-09-21T10:55:59.001 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.001 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.001 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:55:59.001 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.751884+0000", 2024-09-21T10:55:59.001 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00058179499999999997 2024-09-21T10:55:59.001 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.001 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.001 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.001 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.781175+0000", 2024-09-21T10:55:59.002 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.029291467000000002 2024-09-21T10:55:59.002 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.002 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.002 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:55:59.002 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.781250+0000", 2024-09-21T10:55:59.002 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4361000000000002e-05 2024-09-21T10:55:59.002 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.002 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.002 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3,6", 2024-09-21T10:55:59.002 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.781630+0000", 2024-09-21T10:55:59.002 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00037988099999999999 2024-09-21T10:55:59.002 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.002 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.002 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:55:59.003 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.806861+0000", 2024-09-21T10:55:59.003 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.025231923 2024-09-21T10:55:59.003 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.003 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.003 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:59.003 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.814370+0000", 2024-09-21T10:55:59.003 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.007508163 2024-09-21T10:55:59.003 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.003 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.003 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:59.003 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.814421+0000", 2024-09-21T10:55:59.003 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1409000000000002e-05 2024-09-21T10:55:59.003 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.003 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.003 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:59.004 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.853936+0000", 2024-09-21T10:55:59.004 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.039515330000000001 2024-09-21T10:55:59.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.004 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.004 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:55:59.004 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.854028+0000", 2024-09-21T10:55:59.004 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1755000000000005e-05 2024-09-21T10:55:59.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.004 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.004 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:55:59.004 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.857131+0000", 2024-09-21T10:55:59.004 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0031028890000000002 2024-09-21T10:55:59.004 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:59.004 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:55:59.004 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:59.005 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.005 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.005 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:130901 55.d 55:be5bbebe:::benchmark_data_smithi070_294394_object25560:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1449)", 2024-09-21T10:55:59.005 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:28.672601+0000", 2024-09-21T10:55:59.005 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 270.24975141700003, 2024-09-21T10:55:59.005 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.188728971, 2024-09-21T10:55:59.005 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:55:59.005 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:59.005 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:55:59.005 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:55:59.005 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:59.005 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 130901 2024-09-21T10:55:59.005 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.005 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:55:59.006 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.006 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:55:59.006 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.672601+0000", 2024-09-21T10:55:59.006 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:59.006 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.006 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.006 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:55:59.006 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.672601+0000", 2024-09-21T10:55:59.006 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:59.006 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.006 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.006 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:55:59.006 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.672599+0000", 2024-09-21T10:55:59.007 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:55:59.007 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.007 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.007 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:55:59.007 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.672609+0000", 2024-09-21T10:55:59.007 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5559999999999996e-06 2024-09-21T10:55:59.007 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.007 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.007 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:55:59.007 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.672622+0000", 2024-09-21T10:55:59.007 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3148e-05 2024-09-21T10:55:59.007 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.007 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.007 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:55:59.007 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.672630+0000", 2024-09-21T10:55:59.008 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1720000000000007e-06 2024-09-21T10:55:59.008 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.008 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.008 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.008 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.751914+0000", 2024-09-21T10:55:59.008 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.079284350000000003 2024-09-21T10:55:59.008 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.008 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.008 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:55:59.008 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.752435+0000", 2024-09-21T10:55:59.008 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00052104899999999999 2024-09-21T10:55:59.008 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.008 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.009 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.009 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.795817+0000", 2024-09-21T10:55:59.009 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.043381757999999999 2024-09-21T10:55:59.009 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.009 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.009 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:55:59.009 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.795884+0000", 2024-09-21T10:55:59.009 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7234e-05 2024-09-21T10:55:59.009 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.009 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.009 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3,6", 2024-09-21T10:55:59.010 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.795982+0000", 2024-09-21T10:55:59.010 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8054999999999995e-05 2024-09-21T10:55:59.010 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.010 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.010 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:55:59.010 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.814327+0000", 2024-09-21T10:55:59.010 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.018344580999999999 2024-09-21T10:55:59.010 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.010 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.010 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:59.011 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.814478+0000", 2024-09-21T10:55:59.011 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015079700000000001 2024-09-21T10:55:59.011 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.011 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.011 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:59.011 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.814525+0000", 2024-09-21T10:55:59.011 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7110000000000001e-05 2024-09-21T10:55:59.011 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.011 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.011 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:59.011 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.858640+0000", 2024-09-21T10:55:59.011 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.044115190999999998 2024-09-21T10:55:59.011 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.011 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.011 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:55:59.011 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.858707+0000", 2024-09-21T10:55:59.012 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7298999999999994e-05 2024-09-21T10:55:59.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.012 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.012 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:55:59.012 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.861330+0000", 2024-09-21T10:55:59.012 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0026224769999999998 2024-09-21T10:55:59.012 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:59.012 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:55:59.012 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:59.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.012 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.012 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:147245 55.d 55:bdf3b662:::benchmark_data_smithi070_294394_object41904:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1457)", 2024-09-21T10:55:59.012 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:04.440486+0000", 2024-09-21T10:55:59.013 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 234.48186571299999, 2024-09-21T10:55:59.013 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.199205455, 2024-09-21T10:55:59.013 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:55:59.013 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:59.013 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:55:59.013 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:55:59.013 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:59.013 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 147245 2024-09-21T10:55:59.013 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.013 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:55:59.013 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.013 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:55:59.013 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:04.440486+0000", 2024-09-21T10:55:59.013 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:59.014 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.014 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.014 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:55:59.014 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:04.440486+0000", 2024-09-21T10:55:59.014 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:59.014 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.014 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.014 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:55:59.014 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:04.440485+0000", 2024-09-21T10:55:59.014 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:55:59.014 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.014 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.014 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:55:59.014 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:04.440492+0000", 2024-09-21T10:55:59.014 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3760000000000002e-06 2024-09-21T10:55:59.014 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.015 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.015 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:55:59.015 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:04.440504+0000", 2024-09-21T10:55:59.015 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1440000000000001e-05 2024-09-21T10:55:59.015 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.015 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.015 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:55:59.015 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:04.440510+0000", 2024-09-21T10:55:59.015 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7660000000000001e-06 2024-09-21T10:55:59.015 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.015 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.015 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.015 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:04.459793+0000", 2024-09-21T10:55:59.015 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.019282436999999999 2024-09-21T10:55:59.016 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.016 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:55:59.016 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:04.460365+0000", 2024-09-21T10:55:59.016 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00057215100000000002 2024-09-21T10:55:59.016 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.016 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.016 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:04.514686+0000", 2024-09-21T10:55:59.016 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.054320945000000002 2024-09-21T10:55:59.016 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.016 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:55:59.016 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:04.514840+0000", 2024-09-21T10:55:59.017 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000154232 2024-09-21T10:55:59.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.017 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.017 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3,6", 2024-09-21T10:55:59.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:04.514992+0000", 2024-09-21T10:55:59.017 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000151318 2024-09-21T10:55:59.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.017 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.017 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:55:59.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:04.520383+0000", 2024-09-21T10:55:59.017 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0053913320000000004 2024-09-21T10:55:59.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.017 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.017 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:59.019 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:04.520421+0000", 2024-09-21T10:55:59.019 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8593999999999997e-05 2024-09-21T10:55:59.019 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.019 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.019 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:59.019 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:04.533495+0000", 2024-09-21T10:55:59.019 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.013073738 2024-09-21T10:55:59.019 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.019 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.019 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:59.019 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:04.636191+0000", 2024-09-21T10:55:59.019 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102695484 2024-09-21T10:55:59.020 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.020 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.020 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:55:59.020 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:04.636289+0000", 2024-09-21T10:55:59.020 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8562999999999998e-05 2024-09-21T10:55:59.020 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.020 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.020 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:55:59.020 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:04.639692+0000", 2024-09-21T10:55:59.020 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003402698 2024-09-21T10:55:59.020 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:59.020 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:55:59.020 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:59.020 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.021 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.021 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:147735 55.d 55:b20fc495:::benchmark_data_smithi070_294394_object42394:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1458)", 2024-09-21T10:55:59.021 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:05.672091+0000", 2024-09-21T10:55:59.021 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 233.25026092900001, 2024-09-21T10:55:59.021 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.21110140699999999, 2024-09-21T10:55:59.021 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:55:59.021 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:59.021 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:55:59.021 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:55:59.021 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:59.021 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 147735 2024-09-21T10:55:59.021 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.021 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:55:59.021 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.022 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:55:59.022 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:05.672091+0000", 2024-09-21T10:55:59.022 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:59.022 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.022 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.022 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:55:59.022 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:05.672091+0000", 2024-09-21T10:55:59.022 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:59.022 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.022 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.022 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:55:59.022 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:05.672089+0000", 2024-09-21T10:55:59.022 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:55:59.023 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.023 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.023 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:55:59.023 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:05.672097+0000", 2024-09-21T10:55:59.023 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7689999999999996e-06 2024-09-21T10:55:59.023 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.023 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.023 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:55:59.023 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:05.672111+0000", 2024-09-21T10:55:59.023 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.38e-05 2024-09-21T10:55:59.023 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.023 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.023 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:55:59.023 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:05.672118+0000", 2024-09-21T10:55:59.024 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2840000000000001e-06 2024-09-21T10:55:59.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.024 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.024 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.024 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:05.672160+0000", 2024-09-21T10:55:59.024 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1703999999999999e-05 2024-09-21T10:55:59.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.024 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.024 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:55:59.024 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:05.672840+0000", 2024-09-21T10:55:59.024 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00068014599999999996 2024-09-21T10:55:59.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.024 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.024 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.025 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:05.680225+0000", 2024-09-21T10:55:59.025 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.007384959 2024-09-21T10:55:59.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.025 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.025 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:55:59.025 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:05.680338+0000", 2024-09-21T10:55:59.025 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011267199999999999 2024-09-21T10:55:59.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.025 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.025 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3,6", 2024-09-21T10:55:59.025 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:05.680486+0000", 2024-09-21T10:55:59.025 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000148317 2024-09-21T10:55:59.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.026 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.026 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:59.026 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:05.684027+0000", 2024-09-21T10:55:59.026 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003540905 2024-09-21T10:55:59.026 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.026 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.026 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:55:59.026 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:05.684046+0000", 2024-09-21T10:55:59.026 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9562e-05 2024-09-21T10:55:59.026 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.027 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.027 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:59.027 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:05.688860+0000", 2024-09-21T10:55:59.027 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0048134279999999998 2024-09-21T10:55:59.027 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.027 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.027 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:59.027 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:05.879846+0000", 2024-09-21T10:55:59.027 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.19098606800000001 2024-09-21T10:55:59.027 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.027 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.028 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:55:59.028 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:05.879904+0000", 2024-09-21T10:55:59.028 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7874999999999997e-05 2024-09-21T10:55:59.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.028 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.028 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:55:59.028 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:05.883193+0000", 2024-09-21T10:55:59.028 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0032888869999999999 2024-09-21T10:55:59.028 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:59.028 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:55:59.028 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:59.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.028 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.028 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:156264 55.d 55:b68e274d:::benchmark_data_smithi070_294394_object50923:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1464)", 2024-09-21T10:55:59.029 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:24.325793+0000", 2024-09-21T10:55:59.029 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 214.59655965799999, 2024-09-21T10:55:59.029 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4867916409999999, 2024-09-21T10:55:59.029 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:55:59.029 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:55:59.029 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:55:59.029 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:55:59.029 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:59.029 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 156264 2024-09-21T10:55:59.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.029 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:55:59.029 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.029 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:55:59.029 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.325793+0000", 2024-09-21T10:55:59.030 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:59.030 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.030 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:55:59.030 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.325793+0000", 2024-09-21T10:55:59.030 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:59.030 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.030 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:55:59.030 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.325791+0000", 2024-09-21T10:55:59.030 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:55:59.030 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.031 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:55:59.031 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.325798+0000", 2024-09-21T10:55:59.031 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3939999999999998e-06 2024-09-21T10:55:59.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.031 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.031 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:55:59.031 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.325814+0000", 2024-09-21T10:55:59.031 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5532e-05 2024-09-21T10:55:59.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.031 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.031 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:55:59.031 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.325824+0000", 2024-09-21T10:55:59.031 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9739999999999997e-06 2024-09-21T10:55:59.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.031 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.032 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.032 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.335430+0000", 2024-09-21T10:55:59.032 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0096062189999999992 2024-09-21T10:55:59.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.032 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.032 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:55:59.032 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.336139+0000", 2024-09-21T10:55:59.032 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00070871200000000001 2024-09-21T10:55:59.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.032 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.032 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.032 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.347686+0000", 2024-09-21T10:55:59.032 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.01154722 2024-09-21T10:55:59.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.033 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.033 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:55:59.033 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.347760+0000", 2024-09-21T10:55:59.033 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4179999999999998e-05 2024-09-21T10:55:59.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.033 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.033 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3,6", 2024-09-21T10:55:59.033 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.347846+0000", 2024-09-21T10:55:59.033 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5513000000000006e-05 2024-09-21T10:55:59.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.033 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.033 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:59.033 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.350917+0000", 2024-09-21T10:55:59.033 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0030714679999999999 2024-09-21T10:55:59.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.034 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.034 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:55:59.034 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.350930+0000", 2024-09-21T10:55:59.034 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3365e-05 2024-09-21T10:55:59.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.034 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.034 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:59.034 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.354685+0000", 2024-09-21T10:55:59.034 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003754856 2024-09-21T10:55:59.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.034 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.034 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.035 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.470185+0000", 2024-09-21T10:55:59.035 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.115499896 2024-09-21T10:55:59.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.035 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:55:59.035 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.470199+0000", 2024-09-21T10:55:59.035 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3498e-05 2024-09-21T10:55:59.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.035 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.035 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.466078+0000", 2024-09-21T10:55:59.035 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99587892300000003 2024-09-21T10:55:59.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.035 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:55:59.036 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.466088+0000", 2024-09-21T10:55:59.036 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0428000000000001e-05 2024-09-21T10:55:59.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.036 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.036 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.036 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.688170+0000", 2024-09-21T10:55:59.036 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.222082468 2024-09-21T10:55:59.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.036 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.036 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for missing object", 2024-09-21T10:55:59.036 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.688260+0000", 2024-09-21T10:55:59.036 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9816999999999997e-05 2024-09-21T10:55:59.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.037 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.037 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.812498+0000", 2024-09-21T10:55:59.037 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12423751299999999 2024-09-21T10:55:59.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.037 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:55:59.037 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.812584+0000", 2024-09-21T10:55:59.037 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6431e-05 2024-09-21T10:55:59.037 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:59.037 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:55:59.037 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:59.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.037 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:156285 55.d 55:b5785a76:::benchmark_data_smithi070_294394_object50944:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1464)", 2024-09-21T10:55:59.038 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:24.368417+0000", 2024-09-21T10:55:59.038 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 214.553935499, 2024-09-21T10:55:59.038 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4562967860000002, 2024-09-21T10:55:59.038 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:55:59.038 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:55:59.038 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:55:59.038 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:55:59.038 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:59.038 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 156285 2024-09-21T10:55:59.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.038 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:55:59.038 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.038 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:55:59.038 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.368417+0000", 2024-09-21T10:55:59.038 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:59.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.039 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.039 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:55:59.039 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.368417+0000", 2024-09-21T10:55:59.039 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:59.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.039 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.039 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:55:59.039 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.368415+0000", 2024-09-21T10:55:59.039 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:55:59.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.039 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.039 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:55:59.040 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.368422+0000", 2024-09-21T10:55:59.040 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4130000000000003e-06 2024-09-21T10:55:59.040 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.040 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.040 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:55:59.040 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.368435+0000", 2024-09-21T10:55:59.040 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3662999999999999e-05 2024-09-21T10:55:59.040 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.040 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.040 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:55:59.040 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.368442+0000", 2024-09-21T10:55:59.040 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4910000000000002e-06 2024-09-21T10:55:59.040 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.040 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.041 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.041 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.368497+0000", 2024-09-21T10:55:59.041 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5174999999999999e-05 2024-09-21T10:55:59.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.041 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.041 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:55:59.041 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.369528+0000", 2024-09-21T10:55:59.041 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0010314199999999999 2024-09-21T10:55:59.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.041 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.041 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.041 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.376862+0000", 2024-09-21T10:55:59.041 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0073339119999999997 2024-09-21T10:55:59.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.042 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:55:59.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.376932+0000", 2024-09-21T10:55:59.042 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9691999999999996e-05 2024-09-21T10:55:59.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.042 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3,6", 2024-09-21T10:55:59.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.377022+0000", 2024-09-21T10:55:59.042 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0498e-05 2024-09-21T10:55:59.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.042 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:55:59.043 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.380265+0000", 2024-09-21T10:55:59.043 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0032423859999999999 2024-09-21T10:55:59.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.043 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.043 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:59.043 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.380299+0000", 2024-09-21T10:55:59.043 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4088000000000003e-05 2024-09-21T10:55:59.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.044 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:59.044 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.384814+0000", 2024-09-21T10:55:59.044 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0045151929999999998 2024-09-21T10:55:59.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.044 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.044 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.470384+0000", 2024-09-21T10:55:59.044 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.085570246000000003 2024-09-21T10:55:59.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.045 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:55:59.045 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.470393+0000", 2024-09-21T10:55:59.045 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4880000000000007e-06 2024-09-21T10:55:59.045 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.045 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.045 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.045 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.466114+0000", 2024-09-21T10:55:59.045 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99572154000000002 2024-09-21T10:55:59.045 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.045 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.045 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:55:59.045 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.466122+0000", 2024-09-21T10:55:59.045 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1269999999999997e-06 2024-09-21T10:55:59.045 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.046 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.046 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.046 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.812761+0000", 2024-09-21T10:55:59.046 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3466391930000001 2024-09-21T10:55:59.046 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.046 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.046 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for missing object", 2024-09-21T10:55:59.046 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.812878+0000", 2024-09-21T10:55:59.046 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011688099999999999 2024-09-21T10:55:59.046 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.046 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.046 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.046 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.824639+0000", 2024-09-21T10:55:59.047 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011761751000000001 2024-09-21T10:55:59.047 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.047 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.047 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:55:59.047 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.824713+0000", 2024-09-21T10:55:59.047 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4164000000000003e-05 2024-09-21T10:55:59.047 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:59.047 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:55:59.047 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:59.047 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.047 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.047 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:156310 55.d 55:b751137b:::benchmark_data_smithi070_294394_object50969:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1465)", 2024-09-21T10:55:59.047 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:24.538751+0000", 2024-09-21T10:55:59.047 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 214.383601476, 2024-09-21T10:55:59.048 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3296688460000001, 2024-09-21T10:55:59.048 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:55:59.048 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:59.048 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:55:59.048 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:55:59.048 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:59.048 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 156310 2024-09-21T10:55:59.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.048 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:55:59.048 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.048 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:55:59.048 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.538751+0000", 2024-09-21T10:55:59.048 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:59.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.048 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.049 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:55:59.049 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.538751+0000", 2024-09-21T10:55:59.049 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:59.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.049 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.049 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:55:59.049 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.538748+0000", 2024-09-21T10:55:59.049 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 2024-09-21T10:55:59.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.049 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.049 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:55:59.049 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.538759+0000", 2024-09-21T10:55:59.049 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0808000000000001e-05 2024-09-21T10:55:59.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.050 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:55:59.050 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.538782+0000", 2024-09-21T10:55:59.050 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.268e-05 2024-09-21T10:55:59.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.050 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:55:59.050 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.538791+0000", 2024-09-21T10:55:59.050 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1589999999999999e-06 2024-09-21T10:55:59.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.050 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.050 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.538842+0000", 2024-09-21T10:55:59.051 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1545000000000002e-05 2024-09-21T10:55:59.051 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.051 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.051 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:55:59.051 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.538851+0000", 2024-09-21T10:55:59.051 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8959999999999992e-06 2024-09-21T10:55:59.051 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.051 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.051 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.051 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.466142+0000", 2024-09-21T10:55:59.051 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.92729063599999995 2024-09-21T10:55:59.051 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.051 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.051 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:55:59.052 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.466148+0000", 2024-09-21T10:55:59.052 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6679999999999998e-06 2024-09-21T10:55:59.052 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.052 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.052 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.052 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.812905+0000", 2024-09-21T10:55:59.052 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3467571700000001 2024-09-21T10:55:59.052 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.052 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.052 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:55:59.052 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.813970+0000", 2024-09-21T10:55:59.052 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0010657570000000001 2024-09-21T10:55:59.052 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.052 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.053 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.053 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.831790+0000", 2024-09-21T10:55:59.053 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.01781928 2024-09-21T10:55:59.053 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.053 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.053 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:55:59.053 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.831884+0000", 2024-09-21T10:55:59.053 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4627000000000001e-05 2024-09-21T10:55:59.053 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.053 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.053 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:55:59.053 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.832016+0000", 2024-09-21T10:55:59.053 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000131676 2024-09-21T10:55:59.054 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.054 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.054 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:55:59.054 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.849842+0000", 2024-09-21T10:55:59.054 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.017825769000000002 2024-09-21T10:55:59.054 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.054 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.054 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:59.054 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.849892+0000", 2024-09-21T10:55:59.054 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0278000000000002e-05 2024-09-21T10:55:59.054 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.054 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.054 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:59.055 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.865035+0000", 2024-09-21T10:55:59.055 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.015142689000000001 2024-09-21T10:55:59.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.055 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:55:59.055 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.865082+0000", 2024-09-21T10:55:59.055 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7398000000000003e-05 2024-09-21T10:55:59.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.055 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:55:59.055 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.868420+0000", 2024-09-21T10:55:59.055 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0033374279999999999 2024-09-21T10:55:59.055 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:59.055 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:55:59.056 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:59.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.056 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.056 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:156316 55.d 55:b4941ee3:::benchmark_data_smithi070_294394_object50975:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1465)", 2024-09-21T10:55:59.056 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:24.602976+0000", 2024-09-21T10:55:59.056 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 214.31937648600001, 2024-09-21T10:55:59.056 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2688839519999999, 2024-09-21T10:55:59.056 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:55:59.056 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:59.056 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:55:59.056 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:55:59.056 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:59.056 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 156316 2024-09-21T10:55:59.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.057 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:55:59.057 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.057 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:55:59.057 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.602976+0000", 2024-09-21T10:55:59.057 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:59.057 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.057 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.057 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:55:59.057 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.602976+0000", 2024-09-21T10:55:59.057 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:59.057 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.057 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.057 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:55:59.057 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.602974+0000", 2024-09-21T10:55:59.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:55:59.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.058 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.058 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:55:59.058 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.602983+0000", 2024-09-21T10:55:59.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0420000000000008e-06 2024-09-21T10:55:59.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.058 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.058 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:55:59.058 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.603000+0000", 2024-09-21T10:55:59.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6609e-05 2024-09-21T10:55:59.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.058 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.059 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:55:59.059 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.603007+0000", 2024-09-21T10:55:59.059 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7079999999999997e-06 2024-09-21T10:55:59.059 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.059 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.059 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.603048+0000", 2024-09-21T10:55:59.059 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1103999999999998e-05 2024-09-21T10:55:59.059 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.059 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:55:59.059 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.603056+0000", 2024-09-21T10:55:59.059 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6070000000000002e-06 2024-09-21T10:55:59.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.060 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.060 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.060 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.466166+0000", 2024-09-21T10:55:59.060 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.86311000800000004 2024-09-21T10:55:59.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.060 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.060 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:55:59.060 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.466172+0000", 2024-09-21T10:55:59.060 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8579999999999996e-06 2024-09-21T10:55:59.061 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.061 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.061 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.061 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.814030+0000", 2024-09-21T10:55:59.061 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.347857885 2024-09-21T10:55:59.061 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.061 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.061 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:55:59.061 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.814610+0000", 2024-09-21T10:55:59.061 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00058030699999999996 2024-09-21T10:55:59.061 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.062 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.062 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.840300+0000", 2024-09-21T10:55:59.062 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.025689740999999999 2024-09-21T10:55:59.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.062 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:55:59.062 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.840399+0000", 2024-09-21T10:55:59.062 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8877999999999998e-05 2024-09-21T10:55:59.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.062 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:55:59.062 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.840499+0000", 2024-09-21T10:55:59.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000100562 2024-09-21T10:55:59.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:55:59.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.855306+0000", 2024-09-21T10:55:59.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.014806487 2024-09-21T10:55:59.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:59.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.868479+0000", 2024-09-21T10:55:59.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.013173213 2024-09-21T10:55:59.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.064 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:59.064 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.868514+0000", 2024-09-21T10:55:59.064 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4925000000000002e-05 2024-09-21T10:55:59.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.064 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.064 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:55:59.064 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.868568+0000", 2024-09-21T10:55:59.064 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4514e-05 2024-09-21T10:55:59.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.064 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.064 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:55:59.064 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.871860+0000", 2024-09-21T10:55:59.064 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0032912369999999998 2024-09-21T10:55:59.065 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:59.065 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:55:59.065 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:59.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.065 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:156330 55.d 55:b71364a5:::benchmark_data_smithi070_294394_object50989:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1466)", 2024-09-21T10:55:59.065 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:25.802103+0000", 2024-09-21T10:55:59.065 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 213.120249255, 2024-09-21T10:55:59.065 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0730541469999999, 2024-09-21T10:55:59.065 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:55:59.065 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:55:59.065 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:55:59.065 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:55:59.066 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:55:59.066 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 156330 2024-09-21T10:55:59.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.066 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:55:59.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:55:59.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.802103+0000", 2024-09-21T10:55:59.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:59.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:55:59.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.802103+0000", 2024-09-21T10:55:59.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:55:59.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:55:59.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.802101+0000", 2024-09-21T10:55:59.067 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:55:59.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:55:59.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.802108+0000", 2024-09-21T10:55:59.067 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.973e-06 2024-09-21T10:55:59.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:55:59.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.802123+0000", 2024-09-21T10:55:59.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4664999999999999e-05 2024-09-21T10:55:59.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.068 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:55:59.068 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.802132+0000", 2024-09-21T10:55:59.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0799999999999995e-06 2024-09-21T10:55:59.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.068 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.068 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.802172+0000", 2024-09-21T10:55:59.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0259000000000002e-05 2024-09-21T10:55:59.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.069 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:55:59.069 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.802180+0000", 2024-09-21T10:55:59.069 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8529999999999992e-06 2024-09-21T10:55:59.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.069 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.069 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.814647+0000", 2024-09-21T10:55:59.069 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0124667549999999 2024-09-21T10:55:59.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.069 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:55:59.069 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.815703+0000", 2024-09-21T10:55:59.069 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001055708 2024-09-21T10:55:59.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.070 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.070 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:55:59.070 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.847320+0000", 2024-09-21T10:55:59.070 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.031617520000000003 2024-09-21T10:55:59.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.070 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.070 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:55:59.070 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.847431+0000", 2024-09-21T10:55:59.070 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000110976 2024-09-21T10:55:59.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.070 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.070 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:55:59.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.847571+0000", 2024-09-21T10:55:59.071 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014008400000000001 2024-09-21T10:55:59.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:55:59.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.862793+0000", 2024-09-21T10:55:59.071 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.01522224 2024-09-21T10:55:59.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:59.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.871910+0000", 2024-09-21T10:55:59.071 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0091163189999999995 2024-09-21T10:55:59.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.072 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:55:59.072 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.871943+0000", 2024-09-21T10:55:59.072 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3519000000000002e-05 2024-09-21T10:55:59.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.072 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:55:59.072 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.871999+0000", 2024-09-21T10:55:59.072 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6032000000000002e-05 2024-09-21T10:55:59.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:55:59.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:55:59.072 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:55:59.072 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.875157+0000", 2024-09-21T10:55:59.072 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0031577369999999999 2024-09-21T10:55:59.073 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:59.073 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:55:59.073 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:59.073 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:55:59.073 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:55:59.073 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:55:59.074 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:55:59.110 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:59.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:59.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:55:59.211 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:59.247 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:59.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:59.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:59.349 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:59.394 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:59.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:59.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:55:59.489 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:55:59.496 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:55:59.549 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:59.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:59.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:55:59.651 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:55:59.683 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:59.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:59.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:55:59.785 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:55:59.820 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:55:59.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:59.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:55:59.922 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:55:59.958 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:55:59.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:55:59.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:00.060 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:00.075 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:56:00.081 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:00.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:00.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:00.183 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:00.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:00.221 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:00.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:00.322 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:00.358 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:00.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:00.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:00.381 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:56:00.461 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:00.496 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:00.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:00.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:00.598 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:00.630 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:00.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:00.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:00.732 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:00.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:00.763 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:00.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:00 smithi160 ceph-mon[101879]: pgmap v2816: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 7.0 GiB data, 22 GiB used, 335 GiB / 358 GiB avail; 57 MiB/s wr, 938 op/s; 36555/343644 objects degraded (10.637%); 2279/343644 objects misplaced (0.663%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T10:56:00.864 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:00.894 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:00.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:00.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:00.996 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:01.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:01.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:01.074 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:56:01.114 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:00 smithi070 ceph-mon[107385]: pgmap v2816: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 7.0 GiB data, 22 GiB used, 335 GiB / 358 GiB avail; 57 MiB/s wr, 938 op/s; 36555/343644 objects degraded (10.637%); 2279/343644 objects misplaced (0.663%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T10:56:01.114 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:00 smithi070 ceph-mon[112139]: pgmap v2816: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 7.0 GiB data, 22 GiB used, 335 GiB / 358 GiB avail; 57 MiB/s wr, 938 op/s; 36555/343644 objects degraded (10.637%); 2279/343644 objects misplaced (0.663%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T10:56:01.131 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:01.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:01.173 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:01.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:01.274 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:01.315 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:01.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:01.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:01.417 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:01.448 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:01.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:01.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:01.550 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:01.580 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:01.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:01.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:01.682 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:01.723 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:01.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:01.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:01.825 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:01.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:01.848 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:01.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:01.949 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:01.984 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:01.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:01.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:02.086 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:02.124 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:02.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:02.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:02.226 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:02.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:02.262 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:02.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:02.363 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:02.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:02.401 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:02.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:02.403 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:56:02.403 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T10:56:02.403 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T10:56:02.404 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:56:02.502 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:02.535 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:02.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:02.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:02.636 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:02.672 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:02.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:02.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:02.774 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:02.808 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:02.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:02.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:02.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:02 smithi070 ceph-mon[112139]: pgmap v2817: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 7.0 GiB data, 22 GiB used, 335 GiB / 358 GiB avail; 102 B/s rd, 56 MiB/s wr, 1.05k op/s; 36260/344238 objects degraded (10.533%); 2061/344238 objects misplaced (0.599%); 5.0 MiB/s, 80 objects/s recovering 2024-09-21T10:56:02.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:02 smithi070 ceph-mon[107385]: pgmap v2817: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 7.0 GiB data, 22 GiB used, 335 GiB / 358 GiB avail; 102 B/s rd, 56 MiB/s wr, 1.05k op/s; 36260/344238 objects degraded (10.533%); 2061/344238 objects misplaced (0.599%); 5.0 MiB/s, 80 objects/s recovering 2024-09-21T10:56:02.910 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:02.942 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:02.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:02.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:03.044 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:03.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:03.080 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:03.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:02 smithi160 ceph-mon[101879]: pgmap v2817: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 7.0 GiB data, 22 GiB used, 335 GiB / 358 GiB avail; 102 B/s rd, 56 MiB/s wr, 1.05k op/s; 36260/344238 objects degraded (10.533%); 2061/344238 objects misplaced (0.599%); 5.0 MiB/s, 80 objects/s recovering 2024-09-21T10:56:03.180 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:03.221 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:03.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:03.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:03.323 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:03.356 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:03.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:03.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:03.458 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:03.468 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:56:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:56:03] "GET /metrics HTTP/1.1" 200 36626 "" "Prometheus/2.51.0" 2024-09-21T10:56:03.494 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:03.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:03.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:03.595 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:03.634 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:03.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:03.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:03.736 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:03.757 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:03.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:03.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:03.859 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:03.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:03 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 36260/344238 objects degraded (10.533%), 15 pgs degraded (PG_DEGRADED) 2024-09-21T10:56:03.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:03 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 36260/344238 objects degraded (10.533%), 15 pgs degraded (PG_DEGRADED) 2024-09-21T10:56:03.896 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:03.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:03.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:03.998 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:04.032 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:04.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:04.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:04.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:03 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 36260/344238 objects degraded (10.533%), 15 pgs degraded (PG_DEGRADED) 2024-09-21T10:56:04.134 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:04.154 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:04.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:04.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:04.256 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:04.302 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:04.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:04.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:04.403 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:04.430 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:04.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:04.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:04.532 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:04.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:04.569 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:04.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:04.670 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:04.709 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:04.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:04.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:04.810 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:04.848 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:04.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:04.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:04.867 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T10:56:04.868 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T10:56:04.949 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:04.976 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:04 smithi070 ceph-mon[107385]: pgmap v2818: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.9 GiB data, 22 GiB used, 336 GiB / 358 GiB avail; 102 B/s rd, 41 MiB/s wr, 1.00k op/s; 35416/338385 objects degraded (10.466%); 2061/338385 objects misplaced (0.609%); 10 MiB/s, 165 objects/s recovering 2024-09-21T10:56:04.976 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:04 smithi070 ceph-mon[112139]: pgmap v2818: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.9 GiB data, 22 GiB used, 336 GiB / 358 GiB avail; 102 B/s rd, 41 MiB/s wr, 1.00k op/s; 35416/338385 objects degraded (10.466%); 2061/338385 objects misplaced (0.609%); 10 MiB/s, 165 objects/s recovering 2024-09-21T10:56:04.978 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:04.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:04.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:05.080 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:05.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:04 smithi160 ceph-mon[101879]: pgmap v2818: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.9 GiB data, 22 GiB used, 336 GiB / 358 GiB avail; 102 B/s rd, 41 MiB/s wr, 1.00k op/s; 35416/338385 objects degraded (10.466%); 2061/338385 objects misplaced (0.609%); 10 MiB/s, 165 objects/s recovering 2024-09-21T10:56:05.117 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:05.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:05.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:05.219 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:05.252 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:05.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:05.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:05.354 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:05.411 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:05.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:05.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:05.512 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:05.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:05.546 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:05.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:05.647 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:05.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:05.681 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:05.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:05.782 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:05.817 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:05.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:05.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:05.919 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:05.928 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:56:05.929 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:56:05.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:05.956 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:05.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:05.966 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":2818,"stamp":"2024-09-21T10:56:02.984652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7390703131,"num_objects":112795,"num_object_clones":0,"num_object_copies":338385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35416,"num_objects_misplaced":2061,"num_objects_unfound":0,"num_objects_dirty":112795,"num_whiteouts":0,"num_read":174,"num_read_kb":376,"num_write":119831,"num_write_kb":7439085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4405,"num_bytes_recovered":287247272,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120185,"ondisk_log_size":120185,"up":327,"acting":323,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":327,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":374980608,"kb_used":22949516,"kb_used_data":19345380,"kb_used_omap":139,"kb_used_meta":3603636,"kb_avail":352031092,"statfs":{"total":383980142592,"available":360479838208,"internally_reserved":0,"allocated":19809669120,"data_stored":19788784123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":143318,"internal_metadata":3690123306},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,1,5,7,5,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":205324312,"num_objects":3134,"num_object_clones":0,"num_object_copies":9402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1232,"num_objects_misplaced":-420,"num_objects_unfound":0,"num_objects_dirty":3134,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10032,"num_write_kb":421249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1652,"num_bytes_recovered":108265472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004743"},"pg_stats":[{"pgid":"56.19","version":"1503'6754","reported_seq":7114,"reported_epoch":1503,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-21T10:56:02.140871+0000","last_change":"2024-09-21T10:55:38.468965+0000","last_active":"2024-09-21T10:56:02.140871+0000","last_peered":"2024-09-21T10:56:02.140871+0000","last_clean":"2024-09-21T10:55:00.439008+0000","last_became_active":"2024-09-21T10:55:38.464402+0000","last_became_peered":"2024-09-21T10:55:38.464402+0000","last_unstale":"2024-09-21T10:56:02.140871+0000","last_undegraded":"2024-09-21T10:55:38.423639+0000","last_fullsized":"2024-09-21T10:56:02.140871+0000","mapping_epoch":1499,"log_start":"1489'1600","ondisk_log_start":"1489'1600","created":1499,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":5154,"log_dups_size":0,"ondisk_log_size":5154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223936512,"num_objects":3417,"num_object_clones":0,"num_object_copies":10251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3735,"num_write_kb":228864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":159,"num_bytes_recovered":10420224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":["7","4","6"],"object_location_counts":[{"shards":"3,6,7","objects":1507},{"shards":"4,6,7","objects":1910}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"56.18","version":"1503'6882","reported_seq":7232,"reported_epoch":1503,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-21T10:56:01.933314+0000","last_change":"2024-09-21T10:55:38.448876+0000","last_active":"2024-09-21T10:56:01.933314+0000","last_peered":"2024-09-21T10:56:01.933314+0000","last_clean":"2024-09-21T10:55:00.450776+0000","last_became_active":"2024-09-21T10:55:38.415958+0000","last_became_peered":"2024-09-21T10:55:38.415958+0000","last_unstale":"2024-09-21T10:56:01.933314+0000","last_undegraded":"2024-09-21T10:55:38.410262+0000","last_fullsized":"2024-09-21T10:56:01.933314+0000","mapping_epoch":1499,"log_start":"1489'1946","ondisk_log_start":"1489'1946","created":1499,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230457344,"num_objects":3517,"num_object_clones":0,"num_object_copies":10551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1958,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3823,"num_write_kb":234848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":10027008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":["6","3","4"],"object_location_counts":[{"shards":"3,4,6","objects":1958},{"shards":"3,6,7","objects":1559}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"56.17","version":"1503'6797","reported_seq":7174,"reported_epoch":1503,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-21T10:56:02.212300+0000","last_change":"2024-09-21T10:55:38.475418+0000","last_active":"2024-09-21T10:56:02.212300+0000","last_peered":"2024-09-21T10:56:02.212300+0000","last_clean":"2024-09-21T10:55:00.433343+0000","last_became_active":"2024-09-21T10:55:38.471621+0000","last_became_peered":"2024-09-21T10:55:38.471621+0000","last_unstale":"2024-09-21T10:56:02.212300+0000","last_undegraded":"2024-09-21T10:55:38.428899+0000","last_fullsized":"2024-09-21T10:56:02.212300+0000","mapping_epoch":1499,"log_start":"1489'1700","ondisk_log_start":"1489'1700","created":1499,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":5097,"log_dups_size":0,"ondisk_log_size":5097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224559104,"num_objects":3427,"num_object_clones":0,"num_object_copies":10281,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1907,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3763,"num_write_kb":230048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":11010048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":["7","4","6"],"object_location_counts":[{"shards":"3,6,7","objects":1520},{"shards":"4,6,7","objects":1907}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"56.16","version":"1503'6753","reported_seq":8834,"reported_epoch":1503,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-21T10:56:02.150178+0000","last_change":"2024-09-21T10:55:38.466639+0000","last_active":"2024-09-21T10:56:02.150178+0000","last_peered":"2024-09-21T10:56:02.150178+0000","last_clean":"2024-09-21T10:55:00.451801+0000","last_became_active":"2024-09-21T10:55:38.462641+0000","last_became_peered":"2024-09-21T10:55:38.462641+0000","last_unstale":"2024-09-21T10:56:02.150178+0000","last_undegraded":"2024-09-21T10:55:38.401053+0000","last_fullsized":"2024-09-21T10:56:02.150178+0000","mapping_epoch":1499,"log_start":"1489'1600","ondisk_log_start":"1489'1600","created":1499,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":5153,"log_dups_size":0,"ondisk_log_size":5153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224788480,"num_objects":3430,"num_object_clones":0,"num_object_copies":10290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1563,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3732,"num_write_kb":229184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":509,"num_bytes_recovered":33357824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":["7","3","4"],"object_location_counts":[{"shards":"1,3,7","objects":1867},{"shards":"3,4,7","objects":1563}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"56.15","version":"1503'6834","reported_seq":7048,"reported_epoch":1503,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-21T10:55:59.361851+0000","last_change":"2024-09-21T10:55:38.446520+0000","last_active":"2024-09-21T10:55:59.361851+0000","last_peered":"2024-09-21T10:55:59.361851+0000","last_clean":"2024-09-21T10:55:00.429535+0000","last_became_active":"2024-09-21T10:55:38.442642+0000","last_became_peered":"2024-09-21T10:55:38.442642+0000","last_unstale":"2024-09-21T10:55:59.361851+0000","last_undegraded":"2024-09-21T10:55:38.414279+0000","last_fullsized":"2024-09-21T10:55:59.361851+0000","mapping_epoch":1499,"log_start":"1489'2371","ondisk_log_start":"1489'2371","created":1499,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":4463,"log_dups_size":0,"ondisk_log_size":4463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236683264,"num_objects":3612,"num_object_clones":0,"num_object_copies":10836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3770,"num_write_kb":236192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":5242880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"3,4,7","objects":2035},{"shards":"3,6,7","objects":1577}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.14","version":"1503'6885","reported_seq":7063,"reported_epoch":1503,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-21T10:55:58.969359+0000","last_change":"2024-09-21T10:55:38.452609+0000","last_active":"2024-09-21T10:55:58.969359+0000","last_peered":"2024-09-21T10:55:58.969359+0000","last_clean":"2024-09-21T10:55:00.434021+0000","last_became_active":"2024-09-21T10:55:38.428010+0000","last_became_peered":"2024-09-21T10:55:38.428010+0000","last_unstale":"2024-09-21T10:55:58.969359+0000","last_undegraded":"2024-09-21T10:55:38.421150+0000","last_fullsized":"2024-09-21T10:55:58.969359+0000","mapping_epoch":1499,"log_start":"1489'2093","ondisk_log_start":"1489'2093","created":1499,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":4792,"log_dups_size":0,"ondisk_log_size":4792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238944256,"num_objects":3646,"num_object_clones":0,"num_object_copies":10938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3774,"num_write_kb":237440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":4259840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":["1","4","6"],"object_location_counts":[{"shards":"1,4,6","objects":2131},{"shards":"1,6,7","objects":1515}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.13","version":"1503'6836","reported_seq":6874,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:56:02.215995+0000","last_change":"2024-09-21T10:55:38.415115+0000","last_active":"2024-09-21T10:56:02.215995+0000","last_peered":"2024-09-21T10:56:02.215995+0000","last_clean":"2024-09-21T10:56:02.215995+0000","last_became_active":"2024-09-21T10:55:38.412957+0000","last_became_peered":"2024-09-21T10:55:38.412957+0000","last_unstale":"2024-09-21T10:56:02.215995+0000","last_undegraded":"2024-09-21T10:56:02.215995+0000","last_fullsized":"2024-09-21T10:56:02.215995+0000","mapping_epoch":1499,"log_start":"1489'1700","ondisk_log_start":"1489'1700","created":1499,"last_epoch_clean":1500,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":5136,"log_dups_size":0,"ondisk_log_size":5136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229408768,"num_objects":3501,"num_object_clones":0,"num_object_copies":10503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3809,"num_write_kb":233888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"56.12","version":"1503'6766","reported_seq":6823,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:59.313044+0000","last_change":"2024-09-21T10:55:38.442266+0000","last_active":"2024-09-21T10:55:59.313044+0000","last_peered":"2024-09-21T10:55:59.313044+0000","last_clean":"2024-09-21T10:55:59.313044+0000","last_became_active":"2024-09-21T10:55:38.439234+0000","last_became_peered":"2024-09-21T10:55:38.439234+0000","last_unstale":"2024-09-21T10:55:59.313044+0000","last_undegraded":"2024-09-21T10:55:59.313044+0000","last_fullsized":"2024-09-21T10:55:59.313044+0000","mapping_epoch":1499,"log_start":"1489'2427","ondisk_log_start":"1489'2427","created":1499,"last_epoch_clean":1500,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":4339,"log_dups_size":0,"ondisk_log_size":4339,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232620032,"num_objects":3550,"num_object_clones":0,"num_object_copies":10650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3710,"num_write_kb":232288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.11","version":"1503'6636","reported_seq":6694,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:59.323959+0000","last_change":"2024-09-21T10:55:38.410701+0000","last_active":"2024-09-21T10:55:59.323959+0000","last_peered":"2024-09-21T10:55:59.323959+0000","last_clean":"2024-09-21T10:55:59.323959+0000","last_became_active":"2024-09-21T10:55:38.408835+0000","last_became_peered":"2024-09-21T10:55:38.408835+0000","last_unstale":"2024-09-21T10:55:59.323959+0000","last_undegraded":"2024-09-21T10:55:59.323959+0000","last_fullsized":"2024-09-21T10:55:59.323959+0000","mapping_epoch":1499,"log_start":"1489'2267","ondisk_log_start":"1489'2267","created":1499,"last_epoch_clean":1500,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":4369,"log_dups_size":0,"ondisk_log_size":4369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229408768,"num_objects":3501,"num_object_clones":0,"num_object_copies":10503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3649,"num_write_kb":228768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.10","version":"1503'6768","reported_seq":6825,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:59.336639+0000","last_change":"2024-09-21T10:55:38.442432+0000","last_active":"2024-09-21T10:55:59.336639+0000","last_peered":"2024-09-21T10:55:59.336639+0000","last_clean":"2024-09-21T10:55:59.336639+0000","last_became_active":"2024-09-21T10:55:38.439439+0000","last_became_peered":"2024-09-21T10:55:38.439439+0000","last_unstale":"2024-09-21T10:55:59.336639+0000","last_undegraded":"2024-09-21T10:55:59.336639+0000","last_fullsized":"2024-09-21T10:55:59.336639+0000","mapping_epoch":1499,"log_start":"1489'2410","ondisk_log_start":"1489'2410","created":1499,"last_epoch_clean":1500,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":4358,"log_dups_size":0,"ondisk_log_size":4358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231440384,"num_objects":3532,"num_object_clones":0,"num_object_copies":10596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3720,"num_write_kb":232032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.f","version":"1503'7235","reported_seq":11448,"reported_epoch":1503,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-21T10:55:58.982627+0000","last_change":"2024-09-21T10:55:02.692034+0000","last_active":"2024-09-21T10:55:58.982627+0000","last_peered":"2024-09-21T10:55:58.982627+0000","last_clean":"2024-09-21T10:55:00.435791+0000","last_became_active":"2024-09-21T10:55:02.645569+0000","last_became_peered":"2024-09-21T10:55:02.645569+0000","last_unstale":"2024-09-21T10:55:58.982627+0000","last_undegraded":"2024-09-21T10:55:58.982627+0000","last_fullsized":"2024-09-21T10:55:02.452183+0000","mapping_epoch":1492,"log_start":"1490'3351","ondisk_log_start":"1490'3351","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":3884,"log_dups_size":0,"ondisk_log_size":3884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456851456,"num_objects":6971,"num_object_clones":0,"num_object_copies":20913,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2061,"num_objects_unfound":0,"num_objects_dirty":6971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7235,"num_write_kb":454592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2080,"num_bytes_recovered":136314880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7],"avail_no_missing":["1","4","7"],"object_location_counts":[{"shards":"1,4,7","objects":2061},{"shards":"1,7","objects":4910}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.e","version":"1503'7588","reported_seq":7616,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:56:02.181670+0000","last_change":"2024-09-21T10:54:11.452866+0000","last_active":"2024-09-21T10:56:02.181670+0000","last_peered":"2024-09-21T10:56:02.181670+0000","last_clean":"2024-09-21T10:56:02.181670+0000","last_became_active":"2024-09-21T10:54:11.452553+0000","last_became_peered":"2024-09-21T10:54:11.452553+0000","last_unstale":"2024-09-21T10:56:02.181670+0000","last_undegraded":"2024-09-21T10:56:02.181670+0000","last_fullsized":"2024-09-21T10:56:02.181670+0000","mapping_epoch":1484,"log_start":"1490'3705","ondisk_log_start":"1490'3705","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":3883,"log_dups_size":0,"ondisk_log_size":3883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:02:31.756582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455999488,"num_objects":6958,"num_object_clones":0,"num_object_copies":20874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7588,"num_write_kb":465472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"56.d","version":"1503'7427","reported_seq":7479,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:59.345335+0000","last_change":"2024-09-21T10:54:11.455827+0000","last_active":"2024-09-21T10:55:59.345335+0000","last_peered":"2024-09-21T10:55:59.345335+0000","last_clean":"2024-09-21T10:55:59.345335+0000","last_became_active":"2024-09-21T10:54:11.455499+0000","last_became_peered":"2024-09-21T10:54:11.455499+0000","last_unstale":"2024-09-21T10:55:59.345335+0000","last_undegraded":"2024-09-21T10:55:59.345335+0000","last_fullsized":"2024-09-21T10:55:59.345335+0000","mapping_epoch":1484,"log_start":"1490'4076","ondisk_log_start":"1490'4076","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":3351,"log_dups_size":0,"ondisk_log_size":3351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:46:24.073885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":468975640,"num_objects":7157,"num_object_clones":0,"num_object_copies":21471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7157,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7427,"num_write_kb":466625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.c","version":"1503'7412","reported_seq":7463,"reported_epoch":1503,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:55:58.923310+0000","last_change":"2024-09-21T10:55:02.737453+0000","last_active":"2024-09-21T10:55:58.923310+0000","last_peered":"2024-09-21T10:55:58.923310+0000","last_clean":"2024-09-21T10:55:00.435595+0000","last_became_active":"2024-09-21T10:55:02.726777+0000","last_became_peered":"2024-09-21T10:55:02.726777+0000","last_unstale":"2024-09-21T10:55:58.923310+0000","last_undegraded":"2024-09-21T10:55:02.505088+0000","last_fullsized":"2024-09-21T10:55:02.458145+0000","mapping_epoch":1492,"log_start":"1490'3539","ondisk_log_start":"1490'3539","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":3873,"log_dups_size":0,"ondisk_log_size":3873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":468975616,"num_objects":7156,"num_object_clones":0,"num_object_copies":21468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7412,"num_write_kb":466176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6],"avail_no_missing":["1","4","6"],"object_location_counts":[{"shards":"1,4,6","objects":4118},{"shards":"1,6","objects":2910}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.b","version":"1503'7577","reported_seq":7612,"reported_epoch":1503,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:56:01.964675+0000","last_change":"2024-09-21T10:55:02.739752+0000","last_active":"2024-09-21T10:56:01.964675+0000","last_peered":"2024-09-21T10:56:01.964675+0000","last_clean":"2024-09-21T10:55:00.412261+0000","last_became_active":"2024-09-21T10:55:02.725641+0000","last_became_peered":"2024-09-21T10:55:02.725641+0000","last_unstale":"2024-09-21T10:56:01.964675+0000","last_undegraded":"2024-09-21T10:55:02.482139+0000","last_fullsized":"2024-09-21T10:55:02.452705+0000","mapping_epoch":1492,"log_start":"1490'3829","ondisk_log_start":"1490'3829","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":3748,"log_dups_size":0,"ondisk_log_size":3748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:13.502440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458293248,"num_objects":6993,"num_object_clones":0,"num_object_copies":20979,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7577,"num_write_kb":466240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7],"avail_no_missing":["6","4","7"],"object_location_counts":[{"shards":"4,6,7","objects":3848},{"shards":"6,7","objects":2853}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"56.a","version":"1503'7254","reported_seq":7304,"reported_epoch":1503,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:55:58.962644+0000","last_change":"2024-09-21T10:55:02.737448+0000","last_active":"2024-09-21T10:55:58.962644+0000","last_peered":"2024-09-21T10:55:58.962644+0000","last_clean":"2024-09-21T10:55:00.441157+0000","last_became_active":"2024-09-21T10:55:02.726286+0000","last_became_peered":"2024-09-21T10:55:02.726286+0000","last_unstale":"2024-09-21T10:55:58.962644+0000","last_undegraded":"2024-09-21T10:55:02.500096+0000","last_fullsized":"2024-09-21T10:55:02.452984+0000","mapping_epoch":1492,"log_start":"1490'3312","ondisk_log_start":"1490'3312","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":3942,"log_dups_size":0,"ondisk_log_size":3942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":460455936,"num_objects":7026,"num_object_clones":0,"num_object_copies":21078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7254,"num_write_kb":456960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6],"avail_no_missing":["1","4","6"],"object_location_counts":[{"shards":"1,4,6","objects":3970},{"shards":"1,6","objects":2942}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.9","version":"1503'6775","reported_seq":7082,"reported_epoch":1503,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-21T10:56:02.213992+0000","last_change":"2024-09-21T10:55:39.415864+0000","last_active":"2024-09-21T10:56:02.213992+0000","last_peered":"2024-09-21T10:56:02.213992+0000","last_clean":"2024-09-21T10:55:00.439008+0000","last_became_active":"2024-09-21T10:55:39.403857+0000","last_became_peered":"2024-09-21T10:55:39.403857+0000","last_unstale":"2024-09-21T10:56:02.213992+0000","last_undegraded":"2024-09-21T10:55:39.361606+0000","last_fullsized":"2024-09-21T10:56:02.213992+0000","mapping_epoch":1500,"log_start":"1489'1600","ondisk_log_start":"1489'1600","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":5175,"log_dups_size":0,"ondisk_log_size":5175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228851712,"num_objects":3492,"num_object_clones":0,"num_object_copies":10476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3756,"num_write_kb":231936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":8650752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":["7","4","6"],"object_location_counts":[{"shards":"3,6,7","objects":1551},{"shards":"4,6,7","objects":1941}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"56.8","version":"1503'6882","reported_seq":7233,"reported_epoch":1503,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-21T10:56:02.008879+0000","last_change":"2024-09-21T10:55:39.411491+0000","last_active":"2024-09-21T10:56:02.008879+0000","last_peered":"2024-09-21T10:56:02.008879+0000","last_clean":"2024-09-21T10:55:00.450776+0000","last_became_active":"2024-09-21T10:55:39.376957+0000","last_became_peered":"2024-09-21T10:55:39.376957+0000","last_unstale":"2024-09-21T10:56:02.008879+0000","last_undegraded":"2024-09-21T10:55:39.363795+0000","last_fullsized":"2024-09-21T10:56:02.008879+0000","mapping_epoch":1500,"log_start":"1489'1946","ondisk_log_start":"1489'1946","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230588416,"num_objects":3518,"num_object_clones":0,"num_object_copies":10554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3822,"num_write_kb":234912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":10027008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":["6","3","4"],"object_location_counts":[{"shards":"3,4,6","objects":1991},{"shards":"3,6,7","objects":1527}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"56.7","version":"1503'6862","reported_seq":7248,"reported_epoch":1503,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-21T10:56:02.161914+0000","last_change":"2024-09-21T10:55:39.404075+0000","last_active":"2024-09-21T10:56:02.161914+0000","last_peered":"2024-09-21T10:56:02.161914+0000","last_clean":"2024-09-21T10:55:00.433343+0000","last_became_active":"2024-09-21T10:55:39.379062+0000","last_became_peered":"2024-09-21T10:55:39.379062+0000","last_unstale":"2024-09-21T10:56:02.161914+0000","last_undegraded":"2024-09-21T10:55:39.363157+0000","last_fullsized":"2024-09-21T10:56:02.161914+0000","mapping_epoch":1500,"log_start":"1489'1700","ondisk_log_start":"1489'1700","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":5162,"log_dups_size":0,"ondisk_log_size":5162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228294656,"num_objects":3483,"num_object_clones":0,"num_object_copies":10449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3827,"num_write_kb":233952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":11272192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":["7","4","6"],"object_location_counts":[{"shards":"3,6,7","objects":1552},{"shards":"4,6,7","objects":1931}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"56.6","version":"1503'6796","reported_seq":8878,"reported_epoch":1503,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-21T10:56:02.122871+0000","last_change":"2024-09-21T10:55:39.393636+0000","last_active":"2024-09-21T10:56:02.122871+0000","last_peered":"2024-09-21T10:56:02.122871+0000","last_clean":"2024-09-21T10:55:00.451801+0000","last_became_active":"2024-09-21T10:55:39.368953+0000","last_became_peered":"2024-09-21T10:55:39.368953+0000","last_unstale":"2024-09-21T10:56:02.122871+0000","last_undegraded":"2024-09-21T10:55:39.363596+0000","last_fullsized":"2024-09-21T10:56:02.122871+0000","mapping_epoch":1500,"log_start":"1489'1600","ondisk_log_start":"1489'1600","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":5196,"log_dups_size":0,"ondisk_log_size":5196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227344384,"num_objects":3469,"num_object_clones":0,"num_object_copies":10407,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3775,"num_write_kb":231808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":509,"num_bytes_recovered":33357824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":["7","3","4"],"object_location_counts":[{"shards":"1,3,7","objects":1960},{"shards":"3,4,7","objects":1509}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"56.5","version":"1503'6793","reported_seq":7014,"reported_epoch":1503,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-21T10:55:59.360284+0000","last_change":"2024-09-21T10:55:39.407628+0000","last_active":"2024-09-21T10:55:59.360284+0000","last_peered":"2024-09-21T10:55:59.360284+0000","last_clean":"2024-09-21T10:55:00.429535+0000","last_became_active":"2024-09-21T10:55:39.403018+0000","last_became_peered":"2024-09-21T10:55:39.403018+0000","last_unstale":"2024-09-21T10:55:59.360284+0000","last_undegraded":"2024-09-21T10:55:39.365316+0000","last_fullsized":"2024-09-21T10:55:59.360284+0000","mapping_epoch":1500,"log_start":"1489'2371","ondisk_log_start":"1489'2371","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":4422,"log_dups_size":0,"ondisk_log_size":4422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233340928,"num_objects":3560,"num_object_clones":0,"num_object_copies":10680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3728,"num_write_kb":233248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":5505024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"3,4,7","objects":2031},{"shards":"3,6,7","objects":1529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.4","version":"1503'6883","reported_seq":7063,"reported_epoch":1503,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-21T10:55:58.977926+0000","last_change":"2024-09-21T10:55:39.410137+0000","last_active":"2024-09-21T10:55:58.977926+0000","last_peered":"2024-09-21T10:55:58.977926+0000","last_clean":"2024-09-21T10:55:00.434021+0000","last_became_active":"2024-09-21T10:55:39.403204+0000","last_became_peered":"2024-09-21T10:55:39.403204+0000","last_unstale":"2024-09-21T10:55:58.977926+0000","last_undegraded":"2024-09-21T10:55:39.366238+0000","last_fullsized":"2024-09-21T10:55:58.977926+0000","mapping_epoch":1500,"log_start":"1489'2093","ondisk_log_start":"1489'2093","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":4790,"log_dups_size":0,"ondisk_log_size":4790,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238682112,"num_objects":3642,"num_object_clones":0,"num_object_copies":10926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3772,"num_write_kb":237248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":4390912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":["1","4","6"],"object_location_counts":[{"shards":"1,4,6","objects":2039},{"shards":"1,6,7","objects":1603}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.3","version":"1503'6836","reported_seq":6874,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:56:02.082155+0000","last_change":"2024-09-21T10:55:38.402353+0000","last_active":"2024-09-21T10:56:02.082155+0000","last_peered":"2024-09-21T10:56:02.082155+0000","last_clean":"2024-09-21T10:56:02.082155+0000","last_became_active":"2024-09-21T10:55:38.400287+0000","last_became_peered":"2024-09-21T10:55:38.400287+0000","last_unstale":"2024-09-21T10:56:02.082155+0000","last_undegraded":"2024-09-21T10:56:02.082155+0000","last_fullsized":"2024-09-21T10:56:02.082155+0000","mapping_epoch":1484,"log_start":"1489'1700","ondisk_log_start":"1489'1700","created":1484,"last_epoch_clean":1500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":5136,"log_dups_size":0,"ondisk_log_size":5136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229539840,"num_objects":3502,"num_object_clones":0,"num_object_copies":10506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3808,"num_write_kb":233952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"56.2","version":"1503'6837","reported_seq":6893,"reported_epoch":1503,"state":"active+recovery_wait","last_fresh":"2024-09-21T10:55:59.357714+0000","last_change":"2024-09-21T10:55:38.439327+0000","last_active":"2024-09-21T10:55:59.357714+0000","last_peered":"2024-09-21T10:55:59.357714+0000","last_clean":"2024-09-21T10:55:37.633891+0000","last_became_active":"2024-09-21T10:55:38.390565+0000","last_became_peered":"2024-09-21T10:55:38.390565+0000","last_unstale":"2024-09-21T10:55:59.357714+0000","last_undegraded":"2024-09-21T10:55:59.357714+0000","last_fullsized":"2024-09-21T10:55:59.357714+0000","mapping_epoch":1484,"log_start":"1489'2427","ondisk_log_start":"1489'2427","created":1484,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":4410,"log_dups_size":0,"ondisk_log_size":4410,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235962368,"num_objects":3600,"num_object_clones":0,"num_object_copies":10800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3780,"num_write_kb":236192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":3600}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.1","version":"1503'6684","reported_seq":6742,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:59.284513+0000","last_change":"2024-09-21T10:55:38.390538+0000","last_active":"2024-09-21T10:55:59.284513+0000","last_peered":"2024-09-21T10:55:59.284513+0000","last_clean":"2024-09-21T10:55:59.284513+0000","last_became_active":"2024-09-21T10:55:38.386728+0000","last_became_peered":"2024-09-21T10:55:38.386728+0000","last_unstale":"2024-09-21T10:55:59.284513+0000","last_undegraded":"2024-09-21T10:55:59.284513+0000","last_fullsized":"2024-09-21T10:55:59.284513+0000","mapping_epoch":1484,"log_start":"1489'2267","ondisk_log_start":"1489'2267","created":1484,"last_epoch_clean":1500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":4417,"log_dups_size":0,"ondisk_log_size":4417,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231636992,"num_objects":3534,"num_object_clones":0,"num_object_copies":10602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3696,"num_write_kb":231392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.0","version":"1503'6775","reported_seq":6833,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:59.316029+0000","last_change":"2024-09-21T10:55:38.371134+0000","last_active":"2024-09-21T10:55:59.316029+0000","last_peered":"2024-09-21T10:55:59.316029+0000","last_clean":"2024-09-21T10:55:59.316029+0000","last_became_active":"2024-09-21T10:55:38.369012+0000","last_became_peered":"2024-09-21T10:55:38.369012+0000","last_unstale":"2024-09-21T10:55:59.316029+0000","last_undegraded":"2024-09-21T10:55:59.316029+0000","last_fullsized":"2024-09-21T10:55:59.316029+0000","mapping_epoch":1484,"log_start":"1489'2410","ondisk_log_start":"1489'2410","created":1484,"last_epoch_clean":1500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:54:10.441789+0000","last_clean_scrub_stamp":"2024-09-21T10:54:10.441789+0000","objects_scrubbed":0,"log_size":4365,"log_dups_size":0,"ondisk_log_size":4365,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234389504,"num_objects":3576,"num_object_clones":0,"num_object_copies":10728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3726,"num_write_kb":233696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":2010,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.386338+0000","last_change":"2024-09-21T10:55:01.487069+0000","last_active":"2024-09-21T10:55:44.386338+0000","last_peered":"2024-09-21T10:55:44.386338+0000","last_clean":"2024-09-21T10:55:44.386338+0000","last_became_active":"2024-09-21T10:55:01.468554+0000","last_became_peered":"2024-09-21T10:55:01.468554+0000","last_unstale":"2024-09-21T10:55:44.386338+0000","last_undegraded":"2024-09-21T10:55:44.386338+0000","last_fullsized":"2024-09-21T10:55:44.386338+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:59:55.312979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1466,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.388308+0000","last_change":"2024-09-21T10:55:01.479872+0000","last_active":"2024-09-21T10:55:44.388308+0000","last_peered":"2024-09-21T10:55:44.388308+0000","last_clean":"2024-09-21T10:55:44.388308+0000","last_became_active":"2024-09-21T10:55:01.479292+0000","last_became_peered":"2024-09-21T10:55:01.479292+0000","last_unstale":"2024-09-21T10:55:44.388308+0000","last_undegraded":"2024-09-21T10:55:44.388308+0000","last_fullsized":"2024-09-21T10:55:44.388308+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:36:53.565343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":2317,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.656038+0000","last_change":"2024-09-21T10:52:15.400227+0000","last_active":"2024-09-21T10:55:57.656038+0000","last_peered":"2024-09-21T10:55:57.656038+0000","last_clean":"2024-09-21T10:55:57.656038+0000","last_became_active":"2024-09-21T10:52:15.399888+0000","last_became_peered":"2024-09-21T10:52:15.399888+0000","last_unstale":"2024-09-21T10:55:57.656038+0000","last_undegraded":"2024-09-21T10:55:57.656038+0000","last_fullsized":"2024-09-21T10:55:57.656038+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:42:52.565184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2333,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.653346+0000","last_change":"2024-09-21T10:51:27.458216+0000","last_active":"2024-09-21T10:55:57.653346+0000","last_peered":"2024-09-21T10:55:57.653346+0000","last_clean":"2024-09-21T10:55:57.653346+0000","last_became_active":"2024-09-21T10:51:27.457948+0000","last_became_peered":"2024-09-21T10:51:27.457948+0000","last_unstale":"2024-09-21T10:55:57.653346+0000","last_undegraded":"2024-09-21T10:55:57.653346+0000","last_fullsized":"2024-09-21T10:55:57.653346+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:25:29.584079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":39,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.387127+0000","last_change":"2024-09-21T10:55:01.492728+0000","last_active":"2024-09-21T10:55:44.387127+0000","last_peered":"2024-09-21T10:55:44.387127+0000","last_clean":"2024-09-21T10:55:44.387127+0000","last_became_active":"2024-09-21T10:55:01.480502+0000","last_became_peered":"2024-09-21T10:55:01.480502+0000","last_unstale":"2024-09-21T10:55:44.387127+0000","last_undegraded":"2024-09-21T10:55:44.387127+0000","last_fullsized":"2024-09-21T10:55:44.387127+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:03:46.206970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1468,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.653723+0000","last_change":"2024-09-21T10:55:01.474486+0000","last_active":"2024-09-21T10:55:57.653723+0000","last_peered":"2024-09-21T10:55:57.653723+0000","last_clean":"2024-09-21T10:55:57.653723+0000","last_became_active":"2024-09-21T10:55:01.473850+0000","last_became_peered":"2024-09-21T10:55:01.473850+0000","last_unstale":"2024-09-21T10:55:57.653723+0000","last_undegraded":"2024-09-21T10:55:57.653723+0000","last_fullsized":"2024-09-21T10:55:57.653723+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:10.540522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":51,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.663846+0000","last_change":"2024-09-21T10:55:01.492881+0000","last_active":"2024-09-21T10:55:57.663846+0000","last_peered":"2024-09-21T10:55:57.663846+0000","last_clean":"2024-09-21T10:55:57.663846+0000","last_became_active":"2024-09-21T10:55:01.479714+0000","last_became_peered":"2024-09-21T10:55:01.479714+0000","last_unstale":"2024-09-21T10:55:57.663846+0000","last_undegraded":"2024-09-21T10:55:57.663846+0000","last_fullsized":"2024-09-21T10:55:57.663846+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:06:10.023859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":690,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.657956+0000","last_change":"2024-09-21T10:55:01.471821+0000","last_active":"2024-09-21T10:55:57.657956+0000","last_peered":"2024-09-21T10:55:57.657956+0000","last_clean":"2024-09-21T10:55:57.657956+0000","last_became_active":"2024-09-21T10:55:01.470979+0000","last_became_peered":"2024-09-21T10:55:01.470979+0000","last_unstale":"2024-09-21T10:55:57.657956+0000","last_undegraded":"2024-09-21T10:55:57.657956+0000","last_fullsized":"2024-09-21T10:55:57.657956+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:28:49.471629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":2047,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.378498+0000","last_change":"2024-09-21T10:55:01.475204+0000","last_active":"2024-09-21T10:55:44.378498+0000","last_peered":"2024-09-21T10:55:44.378498+0000","last_clean":"2024-09-21T10:55:44.378498+0000","last_became_active":"2024-09-21T10:55:01.474554+0000","last_became_peered":"2024-09-21T10:55:01.474554+0000","last_unstale":"2024-09-21T10:55:44.378498+0000","last_undegraded":"2024-09-21T10:55:44.378498+0000","last_fullsized":"2024-09-21T10:55:44.378498+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:24.849795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1095,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.664105+0000","last_change":"2024-09-21T10:55:01.466683+0000","last_active":"2024-09-21T10:55:57.664105+0000","last_peered":"2024-09-21T10:55:57.664105+0000","last_clean":"2024-09-21T10:55:57.664105+0000","last_became_active":"2024-09-21T10:55:01.465859+0000","last_became_peered":"2024-09-21T10:55:01.465859+0000","last_unstale":"2024-09-21T10:55:57.664105+0000","last_undegraded":"2024-09-21T10:55:57.664105+0000","last_fullsized":"2024-09-21T10:55:57.664105+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:34:59.495754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":35,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.659530+0000","last_change":"2024-09-21T10:55:01.472947+0000","last_active":"2024-09-21T10:55:57.659530+0000","last_peered":"2024-09-21T10:55:57.659530+0000","last_clean":"2024-09-21T10:55:57.659530+0000","last_became_active":"2024-09-21T10:55:01.471388+0000","last_became_peered":"2024-09-21T10:55:01.471388+0000","last_unstale":"2024-09-21T10:55:57.659530+0000","last_undegraded":"2024-09-21T10:55:57.659530+0000","last_fullsized":"2024-09-21T10:55:57.659530+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:33:06.060427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2319,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.670422+0000","last_change":"2024-09-21T10:55:01.472931+0000","last_active":"2024-09-21T10:55:57.670422+0000","last_peered":"2024-09-21T10:55:57.670422+0000","last_clean":"2024-09-21T10:55:57.670422+0000","last_became_active":"2024-09-21T10:55:01.469286+0000","last_became_peered":"2024-09-21T10:55:01.469286+0000","last_unstale":"2024-09-21T10:55:57.670422+0000","last_undegraded":"2024-09-21T10:55:57.670422+0000","last_fullsized":"2024-09-21T10:55:57.670422+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:27:59.942806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2291,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.645285+0000","last_change":"2024-09-21T10:55:01.477818+0000","last_active":"2024-09-21T10:55:57.645285+0000","last_peered":"2024-09-21T10:55:57.645285+0000","last_clean":"2024-09-21T10:55:57.645285+0000","last_became_active":"2024-09-21T10:55:01.477450+0000","last_became_peered":"2024-09-21T10:55:01.477450+0000","last_unstale":"2024-09-21T10:55:57.645285+0000","last_undegraded":"2024-09-21T10:55:57.645285+0000","last_fullsized":"2024-09-21T10:55:57.645285+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:38:51.872952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":49,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.383177+0000","last_change":"2024-09-21T10:52:25.489896+0000","last_active":"2024-09-21T10:55:44.383177+0000","last_peered":"2024-09-21T10:55:44.383177+0000","last_clean":"2024-09-21T10:55:44.383177+0000","last_became_active":"2024-09-21T10:52:25.489154+0000","last_became_peered":"2024-09-21T10:52:25.489154+0000","last_unstale":"2024-09-21T10:55:44.383177+0000","last_undegraded":"2024-09-21T10:55:44.383177+0000","last_fullsized":"2024-09-21T10:55:44.383177+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:36:02.076931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2313,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.652903+0000","last_change":"2024-09-21T10:52:25.493309+0000","last_active":"2024-09-21T10:55:57.652903+0000","last_peered":"2024-09-21T10:55:57.652903+0000","last_clean":"2024-09-21T10:55:57.652903+0000","last_became_active":"2024-09-21T10:52:25.492568+0000","last_became_peered":"2024-09-21T10:52:25.492568+0000","last_unstale":"2024-09-21T10:55:57.652903+0000","last_undegraded":"2024-09-21T10:55:57.652903+0000","last_fullsized":"2024-09-21T10:55:57.652903+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:19:19.393370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1507,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.386408+0000","last_change":"2024-09-21T10:52:03.317474+0000","last_active":"2024-09-21T10:55:44.386408+0000","last_peered":"2024-09-21T10:55:44.386408+0000","last_clean":"2024-09-21T10:55:44.386408+0000","last_became_active":"2024-09-21T10:52:03.317001+0000","last_became_peered":"2024-09-21T10:52:03.317001+0000","last_unstale":"2024-09-21T10:55:44.386408+0000","last_undegraded":"2024-09-21T10:55:44.386408+0000","last_fullsized":"2024-09-21T10:55:44.386408+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:06:07.039763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2313,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.653060+0000","last_change":"2024-09-21T10:52:15.412246+0000","last_active":"2024-09-21T10:55:57.653060+0000","last_peered":"2024-09-21T10:55:57.653060+0000","last_clean":"2024-09-21T10:55:57.653060+0000","last_became_active":"2024-09-21T10:52:15.406204+0000","last_became_peered":"2024-09-21T10:52:15.406204+0000","last_unstale":"2024-09-21T10:55:57.653060+0000","last_undegraded":"2024-09-21T10:55:57.653060+0000","last_fullsized":"2024-09-21T10:55:57.653060+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:03.524324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2292,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.655831+0000","last_change":"2024-09-21T10:52:15.402822+0000","last_active":"2024-09-21T10:55:57.655831+0000","last_peered":"2024-09-21T10:55:57.655831+0000","last_clean":"2024-09-21T10:55:57.655831+0000","last_became_active":"2024-09-21T10:52:15.402548+0000","last_became_peered":"2024-09-21T10:52:15.402548+0000","last_unstale":"2024-09-21T10:55:57.655831+0000","last_undegraded":"2024-09-21T10:55:57.655831+0000","last_fullsized":"2024-09-21T10:55:57.655831+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:04:04.667598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2318,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.653575+0000","last_change":"2024-09-21T10:52:25.499172+0000","last_active":"2024-09-21T10:55:57.653575+0000","last_peered":"2024-09-21T10:55:57.653575+0000","last_clean":"2024-09-21T10:55:57.653575+0000","last_became_active":"2024-09-21T10:52:25.498835+0000","last_became_peered":"2024-09-21T10:52:25.498835+0000","last_unstale":"2024-09-21T10:55:57.653575+0000","last_undegraded":"2024-09-21T10:55:57.653575+0000","last_fullsized":"2024-09-21T10:55:57.653575+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:10:12.434470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":49,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.383101+0000","last_change":"2024-09-21T10:52:25.489774+0000","last_active":"2024-09-21T10:55:44.383101+0000","last_peered":"2024-09-21T10:55:44.383101+0000","last_clean":"2024-09-21T10:55:44.383101+0000","last_became_active":"2024-09-21T10:52:25.487788+0000","last_became_peered":"2024-09-21T10:52:25.487788+0000","last_unstale":"2024-09-21T10:55:44.383101+0000","last_undegraded":"2024-09-21T10:55:44.383101+0000","last_fullsized":"2024-09-21T10:55:44.383101+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:48.842419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":70,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.379442+0000","last_change":"2024-09-21T10:55:01.490959+0000","last_active":"2024-09-21T10:55:44.379442+0000","last_peered":"2024-09-21T10:55:44.379442+0000","last_clean":"2024-09-21T10:55:44.379442+0000","last_became_active":"2024-09-21T10:55:01.466537+0000","last_became_peered":"2024-09-21T10:55:01.466537+0000","last_unstale":"2024-09-21T10:55:44.379442+0000","last_undegraded":"2024-09-21T10:55:44.379442+0000","last_fullsized":"2024-09-21T10:55:44.379442+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:39:39.829439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2319,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.660684+0000","last_change":"2024-09-21T10:52:25.504787+0000","last_active":"2024-09-21T10:55:57.660684+0000","last_peered":"2024-09-21T10:55:57.660684+0000","last_clean":"2024-09-21T10:55:57.660684+0000","last_became_active":"2024-09-21T10:52:25.504198+0000","last_became_peered":"2024-09-21T10:52:25.504198+0000","last_unstale":"2024-09-21T10:55:57.660684+0000","last_undegraded":"2024-09-21T10:55:57.660684+0000","last_fullsized":"2024-09-21T10:55:57.660684+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:38:25.666224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":1400,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.379654+0000","last_change":"2024-09-21T10:55:01.484570+0000","last_active":"2024-09-21T10:55:44.379654+0000","last_peered":"2024-09-21T10:55:44.379654+0000","last_clean":"2024-09-21T10:55:44.379654+0000","last_became_active":"2024-09-21T10:55:01.465514+0000","last_became_peered":"2024-09-21T10:55:01.465514+0000","last_unstale":"2024-09-21T10:55:44.379654+0000","last_undegraded":"2024-09-21T10:55:44.379654+0000","last_fullsized":"2024-09-21T10:55:44.379654+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:21:03.466224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":645,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.388160+0000","last_change":"2024-09-21T10:55:01.479588+0000","last_active":"2024-09-21T10:55:44.388160+0000","last_peered":"2024-09-21T10:55:44.388160+0000","last_clean":"2024-09-21T10:55:44.388160+0000","last_became_active":"2024-09-21T10:55:01.478467+0000","last_became_peered":"2024-09-21T10:55:01.478467+0000","last_unstale":"2024-09-21T10:55:44.388160+0000","last_undegraded":"2024-09-21T10:55:44.388160+0000","last_fullsized":"2024-09-21T10:55:44.388160+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:05.308443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"346'149","reported_seq":981,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.654707+0000","last_change":"2024-09-21T10:53:47.069462+0000","last_active":"2024-09-21T10:55:57.654707+0000","last_peered":"2024-09-21T10:55:57.654707+0000","last_clean":"2024-09-21T10:55:57.654707+0000","last_became_active":"2024-09-21T10:53:47.069113+0000","last_became_peered":"2024-09-21T10:53:47.069113+0000","last_unstale":"2024-09-21T10:55:57.654707+0000","last_undegraded":"2024-09-21T10:55:57.654707+0000","last_fullsized":"2024-09-21T10:55:57.654707+0000","mapping_epoch":1478,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1479,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1579069,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1467,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.385574+0000","last_change":"2024-09-21T10:55:01.468920+0000","last_active":"2024-09-21T10:55:44.385574+0000","last_peered":"2024-09-21T10:55:44.385574+0000","last_clean":"2024-09-21T10:55:44.385574+0000","last_became_active":"2024-09-21T10:55:01.468606+0000","last_became_peered":"2024-09-21T10:55:01.468606+0000","last_unstale":"2024-09-21T10:55:44.385574+0000","last_undegraded":"2024-09-21T10:55:44.385574+0000","last_fullsized":"2024-09-21T10:55:44.385574+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:31:02.135976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1537,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.386478+0000","last_change":"2024-09-21T10:52:25.493263+0000","last_active":"2024-09-21T10:55:44.386478+0000","last_peered":"2024-09-21T10:55:44.386478+0000","last_clean":"2024-09-21T10:55:44.386478+0000","last_became_active":"2024-09-21T10:52:25.492970+0000","last_became_peered":"2024-09-21T10:52:25.492970+0000","last_unstale":"2024-09-21T10:55:44.386478+0000","last_undegraded":"2024-09-21T10:55:44.386478+0000","last_fullsized":"2024-09-21T10:55:44.386478+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:43:40.305783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":175,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.646100+0000","last_change":"2024-09-21T10:55:01.488578+0000","last_active":"2024-09-21T10:55:57.646100+0000","last_peered":"2024-09-21T10:55:57.646100+0000","last_clean":"2024-09-21T10:55:57.646100+0000","last_became_active":"2024-09-21T10:55:01.477816+0000","last_became_peered":"2024-09-21T10:55:01.477816+0000","last_unstale":"2024-09-21T10:55:57.646100+0000","last_undegraded":"2024-09-21T10:55:57.646100+0000","last_fullsized":"2024-09-21T10:55:57.646100+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:01:01.361589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"346'149","reported_seq":647,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.386855+0000","last_change":"2024-09-21T10:55:04.394887+0000","last_active":"2024-09-21T10:55:44.386855+0000","last_peered":"2024-09-21T10:55:44.386855+0000","last_clean":"2024-09-21T10:55:44.386855+0000","last_became_active":"2024-09-21T10:55:04.394422+0000","last_became_peered":"2024-09-21T10:55:04.394422+0000","last_unstale":"2024-09-21T10:55:44.386855+0000","last_undegraded":"2024-09-21T10:55:44.386855+0000","last_fullsized":"2024-09-21T10:55:44.386855+0000","mapping_epoch":1494,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1495,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:22.051403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:22.051403+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2284,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.652984+0000","last_change":"2024-09-21T10:52:25.494316+0000","last_active":"2024-09-21T10:55:57.652984+0000","last_peered":"2024-09-21T10:55:57.652984+0000","last_clean":"2024-09-21T10:55:57.652984+0000","last_became_active":"2024-09-21T10:52:25.494016+0000","last_became_peered":"2024-09-21T10:52:25.494016+0000","last_unstale":"2024-09-21T10:55:57.652984+0000","last_undegraded":"2024-09-21T10:55:57.652984+0000","last_fullsized":"2024-09-21T10:55:57.652984+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:07.859584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2293,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.647001+0000","last_change":"2024-09-21T10:55:01.480701+0000","last_active":"2024-09-21T10:55:57.647001+0000","last_peered":"2024-09-21T10:55:57.647001+0000","last_clean":"2024-09-21T10:55:57.647001+0000","last_became_active":"2024-09-21T10:55:01.479997+0000","last_became_peered":"2024-09-21T10:55:01.479997+0000","last_unstale":"2024-09-21T10:55:57.647001+0000","last_undegraded":"2024-09-21T10:55:57.647001+0000","last_fullsized":"2024-09-21T10:55:57.647001+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:46:34.397469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":2346,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.656478+0000","last_change":"2024-09-21T10:55:01.480736+0000","last_active":"2024-09-21T10:55:57.656478+0000","last_peered":"2024-09-21T10:55:57.656478+0000","last_clean":"2024-09-21T10:55:57.656478+0000","last_became_active":"2024-09-21T10:55:01.471095+0000","last_became_peered":"2024-09-21T10:55:01.471095+0000","last_unstale":"2024-09-21T10:55:57.656478+0000","last_undegraded":"2024-09-21T10:55:57.656478+0000","last_fullsized":"2024-09-21T10:55:57.656478+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:49:16.475617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"346'149","reported_seq":765,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.386349+0000","last_change":"2024-09-21T10:55:04.385463+0000","last_active":"2024-09-21T10:55:44.386349+0000","last_peered":"2024-09-21T10:55:44.386349+0000","last_clean":"2024-09-21T10:55:44.386349+0000","last_became_active":"2024-09-21T10:55:04.385119+0000","last_became_peered":"2024-09-21T10:55:04.385119+0000","last_unstale":"2024-09-21T10:55:44.386349+0000","last_undegraded":"2024-09-21T10:55:44.386349+0000","last_fullsized":"2024-09-21T10:55:44.386349+0000","mapping_epoch":1494,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1495,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:41.011085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:41.011085+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:38:35.908119+0000","scrub_duration":0.0061250310000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2229,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.378528+0000","last_change":"2024-09-21T10:55:01.474981+0000","last_active":"2024-09-21T10:55:44.378528+0000","last_peered":"2024-09-21T10:55:44.378528+0000","last_clean":"2024-09-21T10:55:44.378528+0000","last_became_active":"2024-09-21T10:55:01.474594+0000","last_became_peered":"2024-09-21T10:55:01.474594+0000","last_unstale":"2024-09-21T10:55:44.378528+0000","last_undegraded":"2024-09-21T10:55:44.378528+0000","last_fullsized":"2024-09-21T10:55:44.378528+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:07:49.272842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":58,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.379358+0000","last_change":"2024-09-21T10:55:01.468733+0000","last_active":"2024-09-21T10:55:44.379358+0000","last_peered":"2024-09-21T10:55:44.379358+0000","last_clean":"2024-09-21T10:55:44.379358+0000","last_became_active":"2024-09-21T10:55:01.468372+0000","last_became_peered":"2024-09-21T10:55:01.468372+0000","last_unstale":"2024-09-21T10:55:44.379358+0000","last_undegraded":"2024-09-21T10:55:44.379358+0000","last_fullsized":"2024-09-21T10:55:44.379358+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:03:06.257682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2343,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.664051+0000","last_change":"2024-09-21T10:55:01.480857+0000","last_active":"2024-09-21T10:55:57.664051+0000","last_peered":"2024-09-21T10:55:57.664051+0000","last_clean":"2024-09-21T10:55:57.664051+0000","last_became_active":"2024-09-21T10:55:01.480202+0000","last_became_peered":"2024-09-21T10:55:01.480202+0000","last_unstale":"2024-09-21T10:55:57.664051+0000","last_undegraded":"2024-09-21T10:55:57.664051+0000","last_fullsized":"2024-09-21T10:55:57.664051+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:27.756242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"346'149","reported_seq":791,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.386248+0000","last_change":"2024-09-21T10:52:28.590793+0000","last_active":"2024-09-21T10:55:44.386248+0000","last_peered":"2024-09-21T10:55:44.386248+0000","last_clean":"2024-09-21T10:55:44.386248+0000","last_became_active":"2024-09-21T10:52:28.588662+0000","last_became_peered":"2024-09-21T10:52:28.588662+0000","last_unstale":"2024-09-21T10:55:44.386248+0000","last_undegraded":"2024-09-21T10:55:44.386248+0000","last_fullsized":"2024-09-21T10:55:44.386248+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:44.018645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:44.018645+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:46.849427+0000","scrub_duration":0.0012789399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2298,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.660010+0000","last_change":"2024-09-21T10:55:01.469713+0000","last_active":"2024-09-21T10:55:57.660010+0000","last_peered":"2024-09-21T10:55:57.660010+0000","last_clean":"2024-09-21T10:55:57.660010+0000","last_became_active":"2024-09-21T10:55:01.468470+0000","last_became_peered":"2024-09-21T10:55:01.468470+0000","last_unstale":"2024-09-21T10:55:57.660010+0000","last_undegraded":"2024-09-21T10:55:57.660010+0000","last_fullsized":"2024-09-21T10:55:57.660010+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:17:34.181766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2359,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.663788+0000","last_change":"2024-09-21T10:52:03.314926+0000","last_active":"2024-09-21T10:55:57.663788+0000","last_peered":"2024-09-21T10:55:57.663788+0000","last_clean":"2024-09-21T10:55:57.663788+0000","last_became_active":"2024-09-21T10:52:03.314602+0000","last_became_peered":"2024-09-21T10:52:03.314602+0000","last_unstale":"2024-09-21T10:55:57.663788+0000","last_undegraded":"2024-09-21T10:55:57.663788+0000","last_fullsized":"2024-09-21T10:55:57.663788+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:13.774521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1514,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.670290+0000","last_change":"2024-09-21T10:52:25.502830+0000","last_active":"2024-09-21T10:55:57.670290+0000","last_peered":"2024-09-21T10:55:57.670290+0000","last_clean":"2024-09-21T10:55:57.670290+0000","last_became_active":"2024-09-21T10:52:25.502505+0000","last_became_peered":"2024-09-21T10:52:25.502505+0000","last_unstale":"2024-09-21T10:55:57.670290+0000","last_undegraded":"2024-09-21T10:55:57.670290+0000","last_fullsized":"2024-09-21T10:55:57.670290+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:01:14.965694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"346'149","reported_seq":820,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.378914+0000","last_change":"2024-09-21T10:55:04.367965+0000","last_active":"2024-09-21T10:55:44.378914+0000","last_peered":"2024-09-21T10:55:44.378914+0000","last_clean":"2024-09-21T10:55:44.378914+0000","last_became_active":"2024-09-21T10:55:04.367565+0000","last_became_peered":"2024-09-21T10:55:04.367565+0000","last_unstale":"2024-09-21T10:55:44.378914+0000","last_undegraded":"2024-09-21T10:55:44.378914+0000","last_fullsized":"2024-09-21T10:55:44.378914+0000","mapping_epoch":1494,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1495,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:16.021843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:16.021843+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2353,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.664215+0000","last_change":"2024-09-21T10:52:15.397180+0000","last_active":"2024-09-21T10:55:57.664215+0000","last_peered":"2024-09-21T10:55:57.664215+0000","last_clean":"2024-09-21T10:55:57.664215+0000","last_became_active":"2024-09-21T10:52:15.396879+0000","last_became_peered":"2024-09-21T10:52:15.396879+0000","last_unstale":"2024-09-21T10:55:57.664215+0000","last_undegraded":"2024-09-21T10:55:57.664215+0000","last_fullsized":"2024-09-21T10:55:57.664215+0000","mapping_epoch":1462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:03:07.162178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1447,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.388089+0000","last_change":"2024-09-21T10:55:01.479586+0000","last_active":"2024-09-21T10:55:44.388089+0000","last_peered":"2024-09-21T10:55:44.388089+0000","last_clean":"2024-09-21T10:55:44.388089+0000","last_became_active":"2024-09-21T10:55:01.478541+0000","last_became_peered":"2024-09-21T10:55:01.478541+0000","last_unstale":"2024-09-21T10:55:44.388089+0000","last_undegraded":"2024-09-21T10:55:44.388089+0000","last_fullsized":"2024-09-21T10:55:44.388089+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:51:10.154966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":253,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.658001+0000","last_change":"2024-09-21T10:53:41.149328+0000","last_active":"2024-09-21T10:55:57.658001+0000","last_peered":"2024-09-21T10:55:57.658001+0000","last_clean":"2024-09-21T10:55:57.658001+0000","last_became_active":"2024-09-21T10:52:03.315398+0000","last_became_peered":"2024-09-21T10:52:03.315398+0000","last_unstale":"2024-09-21T10:55:57.658001+0000","last_undegraded":"2024-09-21T10:55:57.658001+0000","last_fullsized":"2024-09-21T10:55:57.658001+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:48:09.007053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"346'149","reported_seq":846,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.376605+0000","last_change":"2024-09-21T10:55:04.366355+0000","last_active":"2024-09-21T10:55:44.376605+0000","last_peered":"2024-09-21T10:55:44.376605+0000","last_clean":"2024-09-21T10:55:44.376605+0000","last_became_active":"2024-09-21T10:55:04.365929+0000","last_became_peered":"2024-09-21T10:55:04.365929+0000","last_unstale":"2024-09-21T10:55:44.376605+0000","last_undegraded":"2024-09-21T10:55:44.376605+0000","last_fullsized":"2024-09-21T10:55:44.376605+0000","mapping_epoch":1494,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1495,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:21.445783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:21.445783+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:04:30.538721+0000","scrub_duration":0.0018624990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":34,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.649211+0000","last_change":"2024-09-21T10:55:01.469268+0000","last_active":"2024-09-21T10:55:57.649211+0000","last_peered":"2024-09-21T10:55:57.649211+0000","last_clean":"2024-09-21T10:55:57.649211+0000","last_became_active":"2024-09-21T10:55:01.468878+0000","last_became_peered":"2024-09-21T10:55:01.468878+0000","last_unstale":"2024-09-21T10:55:57.649211+0000","last_undegraded":"2024-09-21T10:55:57.649211+0000","last_fullsized":"2024-09-21T10:55:57.649211+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:03:24.884225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2309,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.651985+0000","last_change":"2024-09-21T10:52:03.325889+0000","last_active":"2024-09-21T10:55:57.651985+0000","last_peered":"2024-09-21T10:55:57.651985+0000","last_clean":"2024-09-21T10:55:57.651985+0000","last_became_active":"2024-09-21T10:52:03.325500+0000","last_became_peered":"2024-09-21T10:52:03.325500+0000","last_unstale":"2024-09-21T10:55:57.651985+0000","last_undegraded":"2024-09-21T10:55:57.651985+0000","last_fullsized":"2024-09-21T10:55:57.651985+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:20.708252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1095,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.653256+0000","last_change":"2024-09-21T10:51:27.457894+0000","last_active":"2024-09-21T10:55:57.653256+0000","last_peered":"2024-09-21T10:55:57.653256+0000","last_clean":"2024-09-21T10:55:57.653256+0000","last_became_active":"2024-09-21T10:51:27.457431+0000","last_became_peered":"2024-09-21T10:51:27.457431+0000","last_unstale":"2024-09-21T10:55:57.653256+0000","last_undegraded":"2024-09-21T10:55:57.653256+0000","last_fullsized":"2024-09-21T10:55:57.653256+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:03:33.550759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"346'149","reported_seq":976,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.645646+0000","last_change":"2024-09-21T10:55:04.370841+0000","last_active":"2024-09-21T10:55:57.645646+0000","last_peered":"2024-09-21T10:55:57.645646+0000","last_clean":"2024-09-21T10:55:57.645646+0000","last_became_active":"2024-09-21T10:55:04.370543+0000","last_became_peered":"2024-09-21T10:55:04.370543+0000","last_unstale":"2024-09-21T10:55:57.645646+0000","last_undegraded":"2024-09-21T10:55:57.645646+0000","last_fullsized":"2024-09-21T10:55:57.645646+0000","mapping_epoch":1494,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1495,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2342,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.646011+0000","last_change":"2024-09-21T10:55:01.480224+0000","last_active":"2024-09-21T10:55:57.646011+0000","last_peered":"2024-09-21T10:55:57.646011+0000","last_clean":"2024-09-21T10:55:57.646011+0000","last_became_active":"2024-09-21T10:55:01.479045+0000","last_became_peered":"2024-09-21T10:55:01.479045+0000","last_unstale":"2024-09-21T10:55:57.646011+0000","last_undegraded":"2024-09-21T10:55:57.646011+0000","last_fullsized":"2024-09-21T10:55:57.646011+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:10.816874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2213,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.653490+0000","last_change":"2024-09-21T10:55:01.471853+0000","last_active":"2024-09-21T10:55:57.653490+0000","last_peered":"2024-09-21T10:55:57.653490+0000","last_clean":"2024-09-21T10:55:57.653490+0000","last_became_active":"2024-09-21T10:55:01.470262+0000","last_became_peered":"2024-09-21T10:55:01.470262+0000","last_unstale":"2024-09-21T10:55:57.653490+0000","last_undegraded":"2024-09-21T10:55:57.653490+0000","last_fullsized":"2024-09-21T10:55:57.653490+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:22:50.746700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":158,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.376028+0000","last_change":"2024-09-21T10:51:27.426006+0000","last_active":"2024-09-21T10:55:44.376028+0000","last_peered":"2024-09-21T10:55:44.376028+0000","last_clean":"2024-09-21T10:55:44.376028+0000","last_became_active":"2024-09-21T10:51:27.425698+0000","last_became_peered":"2024-09-21T10:51:27.425698+0000","last_unstale":"2024-09-21T10:55:44.376028+0000","last_undegraded":"2024-09-21T10:55:44.376028+0000","last_fullsized":"2024-09-21T10:55:44.376028+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:35:13.210330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":61,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.386966+0000","last_change":"2024-09-21T10:55:01.472846+0000","last_active":"2024-09-21T10:55:44.386966+0000","last_peered":"2024-09-21T10:55:44.386966+0000","last_clean":"2024-09-21T10:55:44.386966+0000","last_became_active":"2024-09-21T10:55:01.472355+0000","last_became_peered":"2024-09-21T10:55:01.472355+0000","last_unstale":"2024-09-21T10:55:44.386966+0000","last_undegraded":"2024-09-21T10:55:44.386966+0000","last_fullsized":"2024-09-21T10:55:44.386966+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:54.663396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2296,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.653802+0000","last_change":"2024-09-21T10:55:01.474215+0000","last_active":"2024-09-21T10:55:57.653802+0000","last_peered":"2024-09-21T10:55:57.653802+0000","last_clean":"2024-09-21T10:55:57.653802+0000","last_became_active":"2024-09-21T10:55:01.473444+0000","last_became_peered":"2024-09-21T10:55:01.473444+0000","last_unstale":"2024-09-21T10:55:57.653802+0000","last_undegraded":"2024-09-21T10:55:57.653802+0000","last_fullsized":"2024-09-21T10:55:57.653802+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:30:26.025903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":1483,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.378568+0000","last_change":"2024-09-21T10:52:25.489550+0000","last_active":"2024-09-21T10:55:44.378568+0000","last_peered":"2024-09-21T10:55:44.378568+0000","last_clean":"2024-09-21T10:55:44.378568+0000","last_became_active":"2024-09-21T10:52:25.489171+0000","last_became_peered":"2024-09-21T10:52:25.489171+0000","last_unstale":"2024-09-21T10:55:44.378568+0000","last_undegraded":"2024-09-21T10:55:44.378568+0000","last_fullsized":"2024-09-21T10:55:44.378568+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:21.105058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":829,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.375990+0000","last_change":"2024-09-21T10:54:20.458538+0000","last_active":"2024-09-21T10:55:44.375990+0000","last_peered":"2024-09-21T10:55:44.375990+0000","last_clean":"2024-09-21T10:55:44.375990+0000","last_became_active":"2024-09-21T10:52:28.587447+0000","last_became_peered":"2024-09-21T10:52:28.587447+0000","last_unstale":"2024-09-21T10:55:44.375990+0000","last_undegraded":"2024-09-21T10:55:44.375990+0000","last_fullsized":"2024-09-21T10:55:44.375990+0000","mapping_epoch":1468,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:20.458421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:20.458421+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:35.467492+0000","scrub_duration":0.001528569,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"346'149","reported_seq":782,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.385769+0000","last_change":"2024-09-21T10:55:04.368448+0000","last_active":"2024-09-21T10:55:44.385769+0000","last_peered":"2024-09-21T10:55:44.385769+0000","last_clean":"2024-09-21T10:55:44.385769+0000","last_became_active":"2024-09-21T10:55:04.368107+0000","last_became_peered":"2024-09-21T10:55:04.368107+0000","last_unstale":"2024-09-21T10:55:44.385769+0000","last_undegraded":"2024-09-21T10:55:44.385769+0000","last_fullsized":"2024-09-21T10:55:44.385769+0000","mapping_epoch":1494,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1495,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:42.008235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:42.008235+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:07:30.729422+0000","scrub_duration":0.011129834999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2293,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.668272+0000","last_change":"2024-09-21T10:55:01.475118+0000","last_active":"2024-09-21T10:55:57.668272+0000","last_peered":"2024-09-21T10:55:57.668272+0000","last_clean":"2024-09-21T10:55:57.668272+0000","last_became_active":"2024-09-21T10:55:01.474515+0000","last_became_peered":"2024-09-21T10:55:01.474515+0000","last_unstale":"2024-09-21T10:55:57.668272+0000","last_undegraded":"2024-09-21T10:55:57.668272+0000","last_fullsized":"2024-09-21T10:55:57.668272+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:25:01.070263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2284,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.652820+0000","last_change":"2024-09-21T10:52:25.493032+0000","last_active":"2024-09-21T10:55:57.652820+0000","last_peered":"2024-09-21T10:55:57.652820+0000","last_clean":"2024-09-21T10:55:57.652820+0000","last_became_active":"2024-09-21T10:52:25.492182+0000","last_became_peered":"2024-09-21T10:52:25.492182+0000","last_unstale":"2024-09-21T10:55:57.652820+0000","last_undegraded":"2024-09-21T10:55:57.652820+0000","last_fullsized":"2024-09-21T10:55:57.652820+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:12:57.118420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1468,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.386127+0000","last_change":"2024-09-21T10:55:01.479104+0000","last_active":"2024-09-21T10:55:44.386127+0000","last_peered":"2024-09-21T10:55:44.386127+0000","last_clean":"2024-09-21T10:55:44.386127+0000","last_became_active":"2024-09-21T10:55:01.478380+0000","last_became_peered":"2024-09-21T10:55:01.478380+0000","last_unstale":"2024-09-21T10:55:44.386127+0000","last_undegraded":"2024-09-21T10:55:44.386127+0000","last_fullsized":"2024-09-21T10:55:44.386127+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:11.194951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1382,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.379076+0000","last_change":"2024-09-21T10:55:01.470469+0000","last_active":"2024-09-21T10:55:44.379076+0000","last_peered":"2024-09-21T10:55:44.379076+0000","last_clean":"2024-09-21T10:55:44.379076+0000","last_became_active":"2024-09-21T10:55:01.470097+0000","last_became_peered":"2024-09-21T10:55:01.470097+0000","last_unstale":"2024-09-21T10:55:44.379076+0000","last_undegraded":"2024-09-21T10:55:44.379076+0000","last_fullsized":"2024-09-21T10:55:44.379076+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:48:19.409017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"346'149","reported_seq":979,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.652728+0000","last_change":"2024-09-21T10:55:04.377180+0000","last_active":"2024-09-21T10:55:57.652728+0000","last_peered":"2024-09-21T10:55:57.652728+0000","last_clean":"2024-09-21T10:55:57.652728+0000","last_became_active":"2024-09-21T10:55:04.376659+0000","last_became_peered":"2024-09-21T10:55:04.376659+0000","last_unstale":"2024-09-21T10:55:57.652728+0000","last_undegraded":"2024-09-21T10:55:57.652728+0000","last_fullsized":"2024-09-21T10:55:57.652728+0000","mapping_epoch":1494,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1495,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1056413,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1324,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.645238+0000","last_change":"2024-09-21T10:55:01.477884+0000","last_active":"2024-09-21T10:55:57.645238+0000","last_peered":"2024-09-21T10:55:57.645238+0000","last_clean":"2024-09-21T10:55:57.645238+0000","last_became_active":"2024-09-21T10:55:01.477512+0000","last_became_peered":"2024-09-21T10:55:01.477512+0000","last_unstale":"2024-09-21T10:55:57.645238+0000","last_undegraded":"2024-09-21T10:55:57.645238+0000","last_fullsized":"2024-09-21T10:55:57.645238+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:58:06.062711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1489,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.002345+0000","last_change":"2024-09-21T10:55:01.480292+0000","last_active":"2024-09-21T10:55:57.002345+0000","last_peered":"2024-09-21T10:55:57.002345+0000","last_clean":"2024-09-21T10:55:57.002345+0000","last_became_active":"2024-09-21T10:55:01.479630+0000","last_became_peered":"2024-09-21T10:55:01.479630+0000","last_unstale":"2024-09-21T10:55:57.002345+0000","last_undegraded":"2024-09-21T10:55:57.002345+0000","last_fullsized":"2024-09-21T10:55:57.002345+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:42.598659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"346'149","reported_seq":955,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.637247+0000","last_change":"2024-09-21T10:54:17.418763+0000","last_active":"2024-09-21T10:55:57.637247+0000","last_peered":"2024-09-21T10:55:57.637247+0000","last_clean":"2024-09-21T10:55:57.637247+0000","last_became_active":"2024-09-21T10:49:47.903650+0000","last_became_peered":"2024-09-21T10:49:47.903650+0000","last_unstale":"2024-09-21T10:55:57.637247+0000","last_undegraded":"2024-09-21T10:55:57.637247+0000","last_fullsized":"2024-09-21T10:55:57.637247+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:17.418646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:17.418646+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:17.134488+0000","scrub_duration":0.00185648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":676,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.655009+0000","last_change":"2024-09-21T10:55:01.468438+0000","last_active":"2024-09-21T10:55:57.655009+0000","last_peered":"2024-09-21T10:55:57.655009+0000","last_clean":"2024-09-21T10:55:57.655009+0000","last_became_active":"2024-09-21T10:55:01.468044+0000","last_became_peered":"2024-09-21T10:55:01.468044+0000","last_unstale":"2024-09-21T10:55:57.655009+0000","last_undegraded":"2024-09-21T10:55:57.655009+0000","last_fullsized":"2024-09-21T10:55:57.655009+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:45:03.600664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":2324,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.658043+0000","last_change":"2024-09-21T10:55:01.484923+0000","last_active":"2024-09-21T10:55:57.658043+0000","last_peered":"2024-09-21T10:55:57.658043+0000","last_clean":"2024-09-21T10:55:57.658043+0000","last_became_active":"2024-09-21T10:55:01.471461+0000","last_became_peered":"2024-09-21T10:55:01.471461+0000","last_unstale":"2024-09-21T10:55:57.658043+0000","last_undegraded":"2024-09-21T10:55:57.658043+0000","last_fullsized":"2024-09-21T10:55:57.658043+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:54:11.289126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":751,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.645782+0000","last_change":"2024-09-21T10:55:01.478361+0000","last_active":"2024-09-21T10:55:57.645782+0000","last_peered":"2024-09-21T10:55:57.645782+0000","last_clean":"2024-09-21T10:55:57.645782+0000","last_became_active":"2024-09-21T10:55:01.478002+0000","last_became_peered":"2024-09-21T10:55:01.478002+0000","last_unstale":"2024-09-21T10:55:57.645782+0000","last_undegraded":"2024-09-21T10:55:57.645782+0000","last_fullsized":"2024-09-21T10:55:57.645782+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:33:01.481712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":849,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.654750+0000","last_change":"2024-09-21T10:49:44.864819+0000","last_active":"2024-09-21T10:55:57.654750+0000","last_peered":"2024-09-21T10:55:57.654750+0000","last_clean":"2024-09-21T10:55:57.654750+0000","last_became_active":"2024-09-21T10:49:44.864061+0000","last_became_peered":"2024-09-21T10:49:44.864061+0000","last_unstale":"2024-09-21T10:55:57.654750+0000","last_undegraded":"2024-09-21T10:55:57.654750+0000","last_fullsized":"2024-09-21T10:55:57.654750+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:37.266967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":61,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.385083+0000","last_change":"2024-09-21T10:52:25.492570+0000","last_active":"2024-09-21T10:55:44.385083+0000","last_peered":"2024-09-21T10:55:44.385083+0000","last_clean":"2024-09-21T10:55:44.385083+0000","last_became_active":"2024-09-21T10:52:25.486382+0000","last_became_peered":"2024-09-21T10:52:25.486382+0000","last_unstale":"2024-09-21T10:55:44.385083+0000","last_undegraded":"2024-09-21T10:55:44.385083+0000","last_fullsized":"2024-09-21T10:55:44.385083+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:06.702707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":166,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.379664+0000","last_change":"2024-09-21T10:49:44.874522+0000","last_active":"2024-09-21T10:55:44.379664+0000","last_peered":"2024-09-21T10:55:44.379664+0000","last_clean":"2024-09-21T10:55:44.379664+0000","last_became_active":"2024-09-21T10:49:44.863242+0000","last_became_peered":"2024-09-21T10:49:44.863242+0000","last_unstale":"2024-09-21T10:55:44.379664+0000","last_undegraded":"2024-09-21T10:55:44.379664+0000","last_fullsized":"2024-09-21T10:55:44.379664+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:09:02.066837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1489,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.645189+0000","last_change":"2024-09-21T10:55:01.477365+0000","last_active":"2024-09-21T10:55:57.645189+0000","last_peered":"2024-09-21T10:55:57.645189+0000","last_clean":"2024-09-21T10:55:57.645189+0000","last_became_active":"2024-09-21T10:55:01.476978+0000","last_became_peered":"2024-09-21T10:55:01.476978+0000","last_unstale":"2024-09-21T10:55:57.645189+0000","last_undegraded":"2024-09-21T10:55:57.645189+0000","last_fullsized":"2024-09-21T10:55:57.645189+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:28.893708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":66,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.383025+0000","last_change":"2024-09-21T10:55:01.496194+0000","last_active":"2024-09-21T10:55:44.383025+0000","last_peered":"2024-09-21T10:55:44.383025+0000","last_clean":"2024-09-21T10:55:44.383025+0000","last_became_active":"2024-09-21T10:55:01.469712+0000","last_became_peered":"2024-09-21T10:55:01.469712+0000","last_unstale":"2024-09-21T10:55:44.383025+0000","last_undegraded":"2024-09-21T10:55:44.383025+0000","last_fullsized":"2024-09-21T10:55:44.383025+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:17:45.286996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":58,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.375960+0000","last_change":"2024-09-21T10:55:01.462980+0000","last_active":"2024-09-21T10:55:44.375960+0000","last_peered":"2024-09-21T10:55:44.375960+0000","last_clean":"2024-09-21T10:55:44.375960+0000","last_became_active":"2024-09-21T10:55:01.462677+0000","last_became_peered":"2024-09-21T10:55:01.462677+0000","last_unstale":"2024-09-21T10:55:44.375960+0000","last_undegraded":"2024-09-21T10:55:44.375960+0000","last_fullsized":"2024-09-21T10:55:44.375960+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:20:13.489085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":66,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.383773+0000","last_change":"2024-09-21T10:55:01.494579+0000","last_active":"2024-09-21T10:55:44.383773+0000","last_peered":"2024-09-21T10:55:44.383773+0000","last_clean":"2024-09-21T10:55:44.383773+0000","last_became_active":"2024-09-21T10:55:01.469707+0000","last_became_peered":"2024-09-21T10:55:01.469707+0000","last_unstale":"2024-09-21T10:55:44.383773+0000","last_undegraded":"2024-09-21T10:55:44.383773+0000","last_fullsized":"2024-09-21T10:55:44.383773+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:36.380411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1524,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.388569+0000","last_change":"2024-09-21T10:51:27.517234+0000","last_active":"2024-09-21T10:55:44.388569+0000","last_peered":"2024-09-21T10:55:44.388569+0000","last_clean":"2024-09-21T10:55:44.388569+0000","last_became_active":"2024-09-21T10:51:27.431253+0000","last_became_peered":"2024-09-21T10:51:27.431253+0000","last_unstale":"2024-09-21T10:55:44.388569+0000","last_undegraded":"2024-09-21T10:55:44.388569+0000","last_fullsized":"2024-09-21T10:55:44.388569+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:35:42.249305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2268,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.655921+0000","last_change":"2024-09-21T10:52:25.497298+0000","last_active":"2024-09-21T10:55:57.655921+0000","last_peered":"2024-09-21T10:55:57.655921+0000","last_clean":"2024-09-21T10:55:57.655921+0000","last_became_active":"2024-09-21T10:52:25.496876+0000","last_became_peered":"2024-09-21T10:52:25.496876+0000","last_unstale":"2024-09-21T10:55:57.655921+0000","last_undegraded":"2024-09-21T10:55:57.655921+0000","last_fullsized":"2024-09-21T10:55:57.655921+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:16.587897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":158,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.212768+0000","last_change":"2024-09-21T10:55:01.465847+0000","last_active":"2024-09-21T10:55:57.212768+0000","last_peered":"2024-09-21T10:55:57.212768+0000","last_clean":"2024-09-21T10:55:57.212768+0000","last_became_active":"2024-09-21T10:55:01.464407+0000","last_became_peered":"2024-09-21T10:55:01.464407+0000","last_unstale":"2024-09-21T10:55:57.212768+0000","last_undegraded":"2024-09-21T10:55:57.212768+0000","last_fullsized":"2024-09-21T10:55:57.212768+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:59:36.163714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2316,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.656144+0000","last_change":"2024-09-21T10:52:25.498554+0000","last_active":"2024-09-21T10:55:57.656144+0000","last_peered":"2024-09-21T10:55:57.656144+0000","last_clean":"2024-09-21T10:55:57.656144+0000","last_became_active":"2024-09-21T10:52:25.498194+0000","last_became_peered":"2024-09-21T10:52:25.498194+0000","last_unstale":"2024-09-21T10:55:57.656144+0000","last_undegraded":"2024-09-21T10:55:57.656144+0000","last_fullsized":"2024-09-21T10:55:57.656144+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:47.115271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":49,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.376513+0000","last_change":"2024-09-21T10:52:25.489063+0000","last_active":"2024-09-21T10:55:44.376513+0000","last_peered":"2024-09-21T10:55:44.376513+0000","last_clean":"2024-09-21T10:55:44.376513+0000","last_became_active":"2024-09-21T10:52:25.488129+0000","last_became_peered":"2024-09-21T10:52:25.488129+0000","last_unstale":"2024-09-21T10:55:44.376513+0000","last_undegraded":"2024-09-21T10:55:44.376513+0000","last_fullsized":"2024-09-21T10:55:44.376513+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:36.996829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1463,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:44.379576+0000","last_change":"2024-09-21T10:49:44.864898+0000","last_active":"2024-09-21T10:55:44.379576+0000","last_peered":"2024-09-21T10:55:44.379576+0000","last_clean":"2024-09-21T10:55:44.379576+0000","last_became_active":"2024-09-21T10:49:44.862691+0000","last_became_peered":"2024-09-21T10:49:44.862691+0000","last_unstale":"2024-09-21T10:55:44.379576+0000","last_undegraded":"2024-09-21T10:55:44.379576+0000","last_fullsized":"2024-09-21T10:55:44.379576+0000","mapping_epoch":1427,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:57.809113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":170,"reported_epoch":1502,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.212818+0000","last_change":"2024-09-21T10:55:01.498736+0000","last_active":"2024-09-21T10:55:57.212818+0000","last_peered":"2024-09-21T10:55:57.212818+0000","last_clean":"2024-09-21T10:55:57.212818+0000","last_became_active":"2024-09-21T10:55:01.475099+0000","last_became_peered":"2024-09-21T10:55:01.475099+0000","last_unstale":"2024-09-21T10:55:57.212818+0000","last_undegraded":"2024-09-21T10:55:57.212818+0000","last_fullsized":"2024-09-21T10:55:57.212818+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:51:50.219969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2298,"reported_epoch":1503,"state":"active+clean","last_fresh":"2024-09-21T10:55:57.654906+0000","last_change":"2024-09-21T10:44:14.398584+0000","last_active":"2024-09-21T10:55:57.654906+0000","last_peered":"2024-09-21T10:55:57.654906+0000","last_clean":"2024-09-21T10:55:57.654906+0000","last_became_active":"2024-09-21T10:44:14.398278+0000","last_became_peered":"2024-09-21T10:44:14.398278+0000","last_unstale":"2024-09-21T10:55:57.654906+0000","last_undegraded":"2024-09-21T10:55:57.654906+0000","last_fullsized":"2024-09-21T10:55:57.654906+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:19.681750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":56,"num_pg":26,"stat_sum":{"num_bytes":7390429208,"num_objects":112770,"num_object_clones":0,"num_object_copies":338310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35416,"num_objects_misplaced":2061,"num_objects_unfound":0,"num_objects_dirty":112770,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119668,"num_write_kb":7437953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4332,"num_bytes_recovered":283901952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22423547904,"data_stored":22423535688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118487,"ondisk_log_size":118487,"up":78,"acting":74,"num_store_stats":5},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":268350,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":173,"num_read_kb":375,"num_write":99,"num_write_kb":1099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3333191,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2641920,"data_stored":2624160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":33,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":12072,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":23492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234744,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6114724,"kb_used_data":5321564,"kb_used_omap":1,"kb_used_meta":793150,"kb_avail":87630428,"statfs":{"total":95995035648,"available":89733558272,"internally_reserved":0,"allocated":5449281536,"data_stored":5444058181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":812186059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496381,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6829352,"kb_used_data":5891808,"kb_used_omap":1,"kb_used_meta":937534,"kb_avail":86915800,"statfs":{"total":95995035648,"available":89001779200,"internally_reserved":0,"allocated":6033211392,"data_stored":6027984482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":960035274},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208758018,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019654,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369397,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5195308,"kb_used_data":4206304,"kb_used_omap":69,"kb_used_meta":988602,"kb_avail":88549844,"statfs":{"total":95995035648,"available":90675040256,"internally_reserved":0,"allocated":4307255296,"data_stored":4302028386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":1012328578},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":2920577761852,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807958083,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4810132,"kb_used_data":3925704,"kb_used_omap":66,"kb_used_meta":884349,"kb_avail":88935020,"statfs":{"total":95995035648,"available":91069460480,"internally_reserved":0,"allocated":4019920896,"data_stored":4014713074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68589,"internal_metadata":905573395},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,2,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":2860448219722,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3998941184,"data_stored":3998941184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4274327552,"data_stored":4274323480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2739863552,"data_stored":2739863552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6003036160,"data_stored":6003032088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5407379456,"data_stored":5407375384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:56:05.970 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_3 pgp_num 26 2024-09-21T10:56:06.057 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:06.081 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:06.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:06.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:56:06.183 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:06.208 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:06.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:06.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:06.214 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:56:06.214 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:56:06.256 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:56:06.257 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:56:06.257 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:56:06.286 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:56:06.286 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:56:06.310 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:06.320 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:56:06.320 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:56:06.337 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:06.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:06.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:06.350 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:56:06.350 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:56:06.352 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:56:06.383 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:56:06.383 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:56:06.413 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:56:06.413 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:56:06.439 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:06.444 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:56:06.444 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:56:06.467 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:06.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:06.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:06.569 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:06.580 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:56:06.601 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:06.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:06.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:06.703 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:06.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:06.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:06.840 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:06 smithi160 ceph-mon[101879]: from='client.55423 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:56:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:06 smithi160 ceph-mon[101879]: pgmap v2819: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.7 GiB data, 22 GiB used, 336 GiB / 358 GiB avail; 102 B/s rd, 22 MiB/s wr, 972 op/s; 34824/330315 objects degraded (10.543%); 1798/330315 objects misplaced (0.544%); 14 MiB/s, 230 objects/s recovering 2024-09-21T10:56:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:06 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2459135362' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:56:06.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:06 smithi070 ceph-mon[107385]: from='client.55423 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:56:06.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:06 smithi070 ceph-mon[107385]: pgmap v2819: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.7 GiB data, 22 GiB used, 336 GiB / 358 GiB avail; 102 B/s rd, 22 MiB/s wr, 972 op/s; 34824/330315 objects degraded (10.543%); 1798/330315 objects misplaced (0.544%); 14 MiB/s, 230 objects/s recovering 2024-09-21T10:56:06.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:06 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2459135362' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:56:06.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:06 smithi070 ceph-mon[112139]: from='client.55423 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:56:06.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:06 smithi070 ceph-mon[112139]: pgmap v2819: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.7 GiB data, 22 GiB used, 336 GiB / 358 GiB avail; 102 B/s rd, 22 MiB/s wr, 972 op/s; 34824/330315 objects degraded (10.543%); 1798/330315 objects misplaced (0.544%); 14 MiB/s, 230 objects/s recovering 2024-09-21T10:56:06.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:06 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2459135362' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:56:06.881 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:06.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:06.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:06.921 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:56:06.983 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:07.012 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:07.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:07.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:07.114 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:07.149 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:07.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:07.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:07.251 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:07.287 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:07.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:07.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:07.389 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:07.422 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:07.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:07.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:07.524 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:07.560 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:07.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:07.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:07.662 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:07.692 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:07.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:07.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:07.794 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:07.830 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:07.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:07.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:07.932 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:07.962 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:07.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:07.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:08.064 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:08.088 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:07 smithi160 ceph-mon[101879]: pgmap v2820: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.7 GiB data, 22 GiB used, 336 GiB / 358 GiB avail; 102 B/s rd, 22 MiB/s wr, 972 op/s; 34824/330315 objects degraded (10.543%); 1798/330315 objects misplaced (0.544%); 14 MiB/s, 230 objects/s recovering 2024-09-21T10:56:08.100 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:07 smithi070 ceph-mon[107385]: pgmap v2820: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.7 GiB data, 22 GiB used, 336 GiB / 358 GiB avail; 102 B/s rd, 22 MiB/s wr, 972 op/s; 34824/330315 objects degraded (10.543%); 1798/330315 objects misplaced (0.544%); 14 MiB/s, 230 objects/s recovering 2024-09-21T10:56:08.100 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:07 smithi070 ceph-mon[112139]: pgmap v2820: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.7 GiB data, 22 GiB used, 336 GiB / 358 GiB avail; 102 B/s rd, 22 MiB/s wr, 972 op/s; 34824/330315 objects degraded (10.543%); 1798/330315 objects misplaced (0.544%); 14 MiB/s, 230 objects/s recovering 2024-09-21T10:56:08.102 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:08.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:08.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:08.204 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:08.242 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:08.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:08.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:08.344 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:08.379 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:08.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:08.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:08.481 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:08.513 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:08.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:08.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:08.615 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:08.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:08.648 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:08.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:08.749 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:08.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:08.783 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:08.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:08.790 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:08 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 34824/330315 objects degraded (10.543%), 15 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:56:08.883 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:08.924 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:08.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:08.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:08.965 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:08 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 34824/330315 objects degraded (10.543%), 15 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:56:08.965 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:08 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 34824/330315 objects degraded (10.543%), 15 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:56:09.026 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:09.066 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:09.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:09.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:09.121 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:56:09.121 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:56:09.169 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:09.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:09.203 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:09.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:09.304 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:09.340 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:09.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:09.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:09.442 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:09.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:09.481 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:09.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:09.582 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:09.619 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:09.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:09.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:09.721 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:09.754 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:09.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:09.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:09.820 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:09 smithi160 ceph-mon[101879]: pgmap v2821: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.6 GiB data, 21 GiB used, 336 GiB / 358 GiB avail; 90 B/s rd, 8.2 MiB/s wr, 833 op/s; 33938/324213 objects degraded (10.468%); 1798/324213 objects misplaced (0.555%); 17 MiB/s, 272 objects/s recovering 2024-09-21T10:56:09.856 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:09.860 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:56:09.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:09.897 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:09.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:09.998 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:10.022 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:09 smithi070 ceph-mon[107385]: pgmap v2821: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.6 GiB data, 21 GiB used, 336 GiB / 358 GiB avail; 90 B/s rd, 8.2 MiB/s wr, 833 op/s; 33938/324213 objects degraded (10.468%); 1798/324213 objects misplaced (0.555%); 17 MiB/s, 272 objects/s recovering 2024-09-21T10:56:10.022 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:09 smithi070 ceph-mon[112139]: pgmap v2821: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.6 GiB data, 21 GiB used, 336 GiB / 358 GiB avail; 90 B/s rd, 8.2 MiB/s wr, 833 op/s; 33938/324213 objects degraded (10.468%); 1798/324213 objects misplaced (0.555%); 17 MiB/s, 272 objects/s recovering 2024-09-21T10:56:10.045 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:10.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:10.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:10.147 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:10.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:10.188 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:10.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:10.205 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:56:10.288 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:10.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:10.322 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:10.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:10.423 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:10.443 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:56:10.463 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:10.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:10.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:10.565 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:10.598 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:10.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:10.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:10.700 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:10.715 INFO:teuthology.orchestra.run.smithi070.stderr:set pool 56 pgp_num to 26 2024-09-21T10:56:10.732 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:10.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:10.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:10.834 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:10.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:10.874 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:10.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:10.975 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:11.011 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:11.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:11.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:11.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:10 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3162853943' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "26"} : dispatch 2024-09-21T10:56:11.100 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:10 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3162853943' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "26"} : dispatch 2024-09-21T10:56:11.100 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:10 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3162853943' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "26"} : dispatch 2024-09-21T10:56:11.113 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:11.151 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:11.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:11.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:11.253 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:11.287 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:11.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:11.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:11.389 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:11.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:11.427 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:11.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:11.528 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:11.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:11.567 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:11.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:11.572 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:56:11.669 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:11.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:11.701 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:11.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:11.729 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T10:56:11.747 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:11 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3162853943' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "26"}]': finished 2024-09-21T10:56:11.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:11 smithi070 ceph-mon[107385]: osdmap e1504: 8 total, 8 up, 4 in 2024-09-21T10:56:11.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:11 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1243095327' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:56:11.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:11 smithi070 ceph-mon[107385]: pgmap v2823: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.4 GiB data, 21 GiB used, 337 GiB / 358 GiB avail; 0 B/s wr, 944 op/s; 33307/315900 objects degraded (10.544%); 1490/315900 objects misplaced (0.472%); 22 MiB/s, 352 objects/s recovering 2024-09-21T10:56:11.748 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:11 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3162853943' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "26"}]': finished 2024-09-21T10:56:11.748 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:11 smithi070 ceph-mon[112139]: osdmap e1504: 8 total, 8 up, 4 in 2024-09-21T10:56:11.748 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:11 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1243095327' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:56:11.748 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:11 smithi070 ceph-mon[112139]: pgmap v2823: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.4 GiB data, 21 GiB used, 337 GiB / 358 GiB avail; 0 B/s wr, 944 op/s; 33307/315900 objects degraded (10.544%); 1490/315900 objects misplaced (0.472%); 22 MiB/s, 352 objects/s recovering 2024-09-21T10:56:11.803 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:11.835 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:11.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:11.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:11.936 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:11.968 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:11.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:11.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:12.033 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:11 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3162853943' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "26"}]': finished 2024-09-21T10:56:12.033 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:11 smithi160 ceph-mon[101879]: osdmap e1504: 8 total, 8 up, 4 in 2024-09-21T10:56:12.033 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:11 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1243095327' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:56:12.033 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:11 smithi160 ceph-mon[101879]: pgmap v2823: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.4 GiB data, 21 GiB used, 337 GiB / 358 GiB avail; 0 B/s wr, 944 op/s; 33307/315900 objects degraded (10.544%); 1490/315900 objects misplaced (0.472%); 22 MiB/s, 352 objects/s recovering 2024-09-21T10:56:12.070 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:12.104 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:12.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:12.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:12.198 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:56:12.198 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:56:12.206 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:12.247 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:12.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:12.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:12.350 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:12.388 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:12.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:12.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:12.489 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:12.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:12.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:12.628 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:12.665 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:12.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:12.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:12.767 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:12.804 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:12.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:12.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:12 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:56:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:12 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1243095327' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:56:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:12 smithi160 ceph-mon[101879]: osdmap e1505: 8 total, 8 up, 4 in 2024-09-21T10:56:12.906 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:12.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:12.943 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:12.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:13.044 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:13.074 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:13.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:13.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:13.102 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:56:13.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:12 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:56:13.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:12 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1243095327' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:56:13.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:12 smithi070 ceph-mon[107385]: osdmap e1505: 8 total, 8 up, 4 in 2024-09-21T10:56:13.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:12 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:56:13.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:12 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1243095327' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:56:13.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:12 smithi070 ceph-mon[112139]: osdmap e1505: 8 total, 8 up, 4 in 2024-09-21T10:56:13.176 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:13.207 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:13.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:13.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:13.310 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:13.365 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:13.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:13.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:13.467 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:13.494 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:56:13.518 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:13.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:13.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:13.568 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:56:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:56:13] "GET /metrics HTTP/1.1" 200 36637 "" "Prometheus/2.51.0" 2024-09-21T10:56:13.620 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:13.651 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:13.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:13.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:13.753 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:13.795 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:13.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:13.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:13.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:13 smithi070 ceph-mon[107385]: pgmap v2825: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.3 GiB data, 20 GiB used, 337 GiB / 358 GiB avail; 0 B/s wr, 880 op/s; 32348/309186 objects degraded (10.462%); 1490/309186 objects misplaced (0.482%); 22 MiB/s, 347 objects/s recovering 2024-09-21T10:56:13.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:13 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 33307/315900 objects degraded (10.544%), 15 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:56:13.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:13 smithi070 ceph-mon[112139]: pgmap v2825: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.3 GiB data, 20 GiB used, 337 GiB / 358 GiB avail; 0 B/s wr, 880 op/s; 32348/309186 objects degraded (10.462%); 1490/309186 objects misplaced (0.482%); 22 MiB/s, 347 objects/s recovering 2024-09-21T10:56:13.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:13 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 33307/315900 objects degraded (10.544%), 15 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:56:13.897 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:13.930 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:13.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:13.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:13.983 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:13 smithi160 ceph-mon[101879]: pgmap v2825: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.3 GiB data, 20 GiB used, 337 GiB / 358 GiB avail; 0 B/s wr, 880 op/s; 32348/309186 objects degraded (10.462%); 1490/309186 objects misplaced (0.482%); 22 MiB/s, 347 objects/s recovering 2024-09-21T10:56:13.984 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:13 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 33307/315900 objects degraded (10.544%), 15 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:56:14.032 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:14.065 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:14.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:14.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:14.167 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:14.205 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:14.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:14.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:14.253 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:56:14.253 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:56:14.286 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:56:14.286 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:56:14.308 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:14.318 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:56:14.318 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:56:14.340 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:14.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:14.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:14.354 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:56:14.354 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:56:14.380 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:56:14.380 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:56:14.406 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:56:14.406 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:56:14.436 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:56:14.437 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:56:14.441 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:14.466 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:56:14.467 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:56:14.472 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:14.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:14.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:14.574 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:14.606 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:14.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:14.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:14.701 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:56:14.708 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:14.768 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:14.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:14.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:14.870 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:14.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:14.906 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:14.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:15.006 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:15.031 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:14 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1980256291' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:56:15.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:15.040 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:15.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:15.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:14 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1980256291' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:56:15.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:14 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1980256291' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:56:15.141 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:15.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:15.174 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:15.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:15.275 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:15.312 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:15.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:15.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:15.414 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:15.448 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:15.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:15.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:15.489 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:56:15.489 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T10:56:15.490 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.490 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36012.0:134469 56.8 56:11a4bb1f:::benchmark_data_smithi070_333437_object18248:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1505)", 2024-09-21T10:56:15.490 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:56:15.472362+0000", 2024-09-21T10:56:15.490 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.013926031, 2024-09-21T10:56:15.490 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.013958481999999999, 2024-09-21T10:56:15.490 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:15.490 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:56:15.490 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:56:15.490 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36012", 2024-09-21T10:56:15.490 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T10:56:15.490 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 134469 2024-09-21T10:56:15.490 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.490 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:15.491 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.491 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:15.491 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.472362+0000", 2024-09-21T10:56:15.491 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:15.491 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.491 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.491 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:15.491 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.472362+0000", 2024-09-21T10:56:15.491 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:15.491 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.491 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.491 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:15.491 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.472360+0000", 2024-09-21T10:56:15.491 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:56:15.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.492 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.492 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:15.492 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.472368+0000", 2024-09-21T10:56:15.492 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3240000000000003e-06 2024-09-21T10:56:15.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.492 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.492 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:15.492 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.472386+0000", 2024-09-21T10:56:15.492 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8382000000000001e-05 2024-09-21T10:56:15.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.492 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.492 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:15.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.472395+0000", 2024-09-21T10:56:15.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.276e-06 2024-09-21T10:56:15.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:15.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.472452+0000", 2024-09-21T10:56:15.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6975999999999998e-05 2024-09-21T10:56:15.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:56:15.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.473197+0000", 2024-09-21T10:56:15.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000745096 2024-09-21T10:56:15.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.494 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:15.494 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.483280+0000", 2024-09-21T10:56:15.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.010083757 2024-09-21T10:56:15.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.494 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.494 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:56:15.494 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.483371+0000", 2024-09-21T10:56:15.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0451999999999994e-05 2024-09-21T10:56:15.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.494 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.494 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:56:15.494 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.483762+0000", 2024-09-21T10:56:15.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00039065999999999998 2024-09-21T10:56:15.495 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:15.495 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:15.495 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:15.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.495 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.495 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36012.0:134471 56.18 56:1d65ed17:::benchmark_data_smithi070_333437_object18250:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1505)", 2024-09-21T10:56:15.495 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:56:15.473897+0000", 2024-09-21T10:56:15.495 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.012390688, 2024-09-21T10:56:15.495 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.01251756, 2024-09-21T10:56:15.495 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:15.495 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for degraded object", 2024-09-21T10:56:15.495 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:56:15.495 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36012", 2024-09-21T10:56:15.495 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T10:56:15.496 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 134471 2024-09-21T10:56:15.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.496 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:15.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.496 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:15.496 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.473897+0000", 2024-09-21T10:56:15.496 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:15.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.496 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:15.496 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.473897+0000", 2024-09-21T10:56:15.496 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:15.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.497 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.497 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:15.497 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.473896+0000", 2024-09-21T10:56:15.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:56:15.497 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.497 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.497 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:15.497 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.473902+0000", 2024-09-21T10:56:15.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5130000000000002e-06 2024-09-21T10:56:15.497 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.497 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.497 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:15.497 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.473916+0000", 2024-09-21T10:56:15.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3373000000000001e-05 2024-09-21T10:56:15.497 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.498 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:15.498 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.473923+0000", 2024-09-21T10:56:15.498 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8199999999999997e-06 2024-09-21T10:56:15.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.498 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:15.498 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.474089+0000", 2024-09-21T10:56:15.498 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016538999999999999 2024-09-21T10:56:15.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.498 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:56:15.498 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.476919+0000", 2024-09-21T10:56:15.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0028306310000000001 2024-09-21T10:56:15.499 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:15.499 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:15.499 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:15.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.499 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.36012.0:134472 56.f e1505/1492 56:fefbfdbf:::benchmark_data_smithi070_333437_object18251:head v 1505'8228, mlcod=1489'37)", 2024-09-21T10:56:15.499 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:56:15.476375+0000", 2024-09-21T10:56:15.499 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0099122580000000002, 2024-09-21T10:56:15.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.01010399, 2024-09-21T10:56:15.499 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:15.499 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T10:56:15.499 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:15.500 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.500 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:15.500 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.476375+0000", 2024-09-21T10:56:15.500 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:15.500 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.500 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.500 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:15.500 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.476375+0000", 2024-09-21T10:56:15.500 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:15.500 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.500 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.500 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:15.500 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.476375+0000", 2024-09-21T10:56:15.500 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T10:56:15.500 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.501 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.501 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:15.501 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.476380+0000", 2024-09-21T10:56:15.501 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9599999999999999e-06 2024-09-21T10:56:15.501 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.501 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.501 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:15.501 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.476387+0000", 2024-09-21T10:56:15.501 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6749999999999996e-06 2024-09-21T10:56:15.501 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.501 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.501 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:15.501 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.476392+0000", 2024-09-21T10:56:15.501 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5860000000000004e-06 2024-09-21T10:56:15.502 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.502 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.502 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:15.502 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.476459+0000", 2024-09-21T10:56:15.502 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6602999999999999e-05 2024-09-21T10:56:15.502 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.502 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.502 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:56:15.502 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.476495+0000", 2024-09-21T10:56:15.502 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5729999999999998e-05 2024-09-21T10:56:15.502 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:15.502 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:15.502 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:15.502 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.503 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.503 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36012.0:134476 56.b 56:d5656baa:::benchmark_data_smithi070_333437_object18255:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1505)", 2024-09-21T10:56:15.503 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:56:15.477994+0000", 2024-09-21T10:56:15.503 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0082934710000000002, 2024-09-21T10:56:15.503 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0085457810000000006, 2024-09-21T10:56:15.503 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:15.503 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:56:15.503 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:56:15.503 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36012", 2024-09-21T10:56:15.504 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T10:56:15.504 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 134476 2024-09-21T10:56:15.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.504 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:15.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.504 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:15.504 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.477994+0000", 2024-09-21T10:56:15.504 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:15.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.504 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:15.504 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.477994+0000", 2024-09-21T10:56:15.505 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:15.505 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.505 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:15.505 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.477993+0000", 2024-09-21T10:56:15.505 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:56:15.505 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.505 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:15.505 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.477998+0000", 2024-09-21T10:56:15.505 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.746e-06 2024-09-21T10:56:15.505 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.505 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:15.506 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.478006+0000", 2024-09-21T10:56:15.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3909999999999998e-06 2024-09-21T10:56:15.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.506 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:15.506 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.478011+0000", 2024-09-21T10:56:15.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4929999999999998e-06 2024-09-21T10:56:15.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.506 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:15.506 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.479778+0000", 2024-09-21T10:56:15.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001766063 2024-09-21T10:56:15.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.507 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:56:15.507 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.479945+0000", 2024-09-21T10:56:15.507 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000167193 2024-09-21T10:56:15.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.507 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.507 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:56:15.507 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.480104+0000", 2024-09-21T10:56:15.507 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015932699999999999 2024-09-21T10:56:15.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.507 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.507 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:56:15.507 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.483405+0000", 2024-09-21T10:56:15.507 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0033005619999999999 2024-09-21T10:56:15.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.508 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.508 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:56:15.508 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.486043+0000", 2024-09-21T10:56:15.508 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0026385380000000002 2024-09-21T10:56:15.508 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:15.508 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:15.508 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:15.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.508 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.508 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDPGPushReply(56.18 1505/1499 [PushReplyOp(56:1d65ed17:::benchmark_data_smithi070_333437_object18250:head)])", 2024-09-21T10:56:15.508 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:56:15.480976+0000", 2024-09-21T10:56:15.508 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0053111570000000004, 2024-09-21T10:56:15.508 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0056388319999999999, 2024-09-21T10:56:15.508 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:15.509 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:56:15.509 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:15.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.509 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:15.509 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.480976+0000", 2024-09-21T10:56:15.509 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:15.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.509 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:15.509 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.480976+0000", 2024-09-21T10:56:15.509 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:15.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.509 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:15.510 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.480976+0000", 2024-09-21T10:56:15.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T10:56:15.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.510 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:15.510 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.480980+0000", 2024-09-21T10:56:15.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9380000000000002e-06 2024-09-21T10:56:15.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.510 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:15.510 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.480991+0000", 2024-09-21T10:56:15.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0898e-05 2024-09-21T10:56:15.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.511 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:15.511 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.480997+0000", 2024-09-21T10:56:15.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4500000000000001e-06 2024-09-21T10:56:15.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.511 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:15.511 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.481145+0000", 2024-09-21T10:56:15.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014751 2024-09-21T10:56:15.511 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:15.511 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:15.511 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:15.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.511 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.36012.0:134473 56.15 e1505/1499 56:a8524413:::benchmark_data_smithi070_333437_object18252:head v 1505'7343, mlcod=0'0)", 2024-09-21T10:56:15.512 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:56:15.483079+0000", 2024-09-21T10:56:15.512 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0032083469999999998, 2024-09-21T10:56:15.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0035848939999999999, 2024-09-21T10:56:15.512 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:15.512 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T10:56:15.512 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:15.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:15.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.483079+0000", 2024-09-21T10:56:15.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:15.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:15.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.483079+0000", 2024-09-21T10:56:15.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:15.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.513 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:15.513 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.483075+0000", 2024-09-21T10:56:15.513 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999957 2024-09-21T10:56:15.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.513 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:15.513 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.483087+0000", 2024-09-21T10:56:15.513 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2306999999999999e-05 2024-09-21T10:56:15.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.513 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:15.513 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.483097+0000", 2024-09-21T10:56:15.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0030000000000001e-05 2024-09-21T10:56:15.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.514 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:15.514 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.483103+0000", 2024-09-21T10:56:15.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9030000000000001e-06 2024-09-21T10:56:15.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.514 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:15.514 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.485205+0000", 2024-09-21T10:56:15.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021022340000000001 2024-09-21T10:56:15.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.514 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:56:15.515 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.485233+0000", 2024-09-21T10:56:15.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7866999999999999e-05 2024-09-21T10:56:15.515 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:15.515 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:15.515 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:15.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.515 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.36012.0:134468 56.4 e1505/1500 56:2686391d:::benchmark_data_smithi070_333437_object18247:head v 1505'7372, mlcod=0'0)", 2024-09-21T10:56:15.515 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:56:15.483471+0000", 2024-09-21T10:56:15.515 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0028165260000000002, 2024-09-21T10:56:15.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0032529379999999999, 2024-09-21T10:56:15.515 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:15.515 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T10:56:15.515 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:15.516 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.516 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:15.516 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.483471+0000", 2024-09-21T10:56:15.516 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:15.516 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.516 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.516 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:15.516 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.483471+0000", 2024-09-21T10:56:15.516 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:15.516 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.516 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.516 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:15.516 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.483470+0000", 2024-09-21T10:56:15.516 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:56:15.516 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.517 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:15.517 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.483477+0000", 2024-09-21T10:56:15.517 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7519999999999999e-06 2024-09-21T10:56:15.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.517 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:15.517 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.483488+0000", 2024-09-21T10:56:15.517 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0601999999999999e-05 2024-09-21T10:56:15.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.517 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:15.517 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.483493+0000", 2024-09-21T10:56:15.517 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6180000000000003e-06 2024-09-21T10:56:15.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:15.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.483551+0000", 2024-09-21T10:56:15.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7277000000000001e-05 2024-09-21T10:56:15.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:56:15.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.483570+0000", 2024-09-21T10:56:15.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9887000000000001e-05 2024-09-21T10:56:15.518 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:15.518 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:15.518 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:15.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.519 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.36012.0:134478 56.d e1505/1484 56:ba7ddfed:::benchmark_data_smithi070_333437_object18257:head v 1505'8467, mlcod=1505'8465)", 2024-09-21T10:56:15.519 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:56:15.485144+0000", 2024-09-21T10:56:15.519 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.001143902, 2024-09-21T10:56:15.519 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001633925, 2024-09-21T10:56:15.519 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:15.519 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T10:56:15.519 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:15.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.519 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:15.519 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.485144+0000", 2024-09-21T10:56:15.519 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:15.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:15.520 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.485144+0000", 2024-09-21T10:56:15.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:15.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:15.520 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.485143+0000", 2024-09-21T10:56:15.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:56:15.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:15.520 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.485150+0000", 2024-09-21T10:56:15.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9759999999999998e-06 2024-09-21T10:56:15.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:15.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.485161+0000", 2024-09-21T10:56:15.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1225e-05 2024-09-21T10:56:15.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:15.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.485168+0000", 2024-09-21T10:56:15.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7290000000000003e-06 2024-09-21T10:56:15.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:15.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.486568+0000", 2024-09-21T10:56:15.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0014002439999999999 2024-09-21T10:56:15.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.522 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.522 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:56:15.522 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.486587+0000", 2024-09-21T10:56:15.522 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9139000000000001e-05 2024-09-21T10:56:15.522 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:15.522 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:15.522 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:15.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.522 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.522 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.36012.0:134475 56.5 e1505/1500)", 2024-09-21T10:56:15.522 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:56:15.485913+0000", 2024-09-21T10:56:15.522 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.00037426199999999999, 2024-09-21T10:56:15.523 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00091646800000000004, 2024-09-21T10:56:15.523 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:15.523 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T10:56:15.523 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:15.523 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.523 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:15.523 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.485913+0000", 2024-09-21T10:56:15.523 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:15.523 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.523 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.523 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:15.523 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.485913+0000", 2024-09-21T10:56:15.523 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:15.523 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.524 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:15.524 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.485912+0000", 2024-09-21T10:56:15.524 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:56:15.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.524 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:15.524 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.485919+0000", 2024-09-21T10:56:15.524 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7399999999999998e-06 2024-09-21T10:56:15.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.524 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:15.524 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.485929+0000", 2024-09-21T10:56:15.524 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0263e-05 2024-09-21T10:56:15.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:15.525 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:15.525 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:15.525 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:56:15.485935+0000", 2024-09-21T10:56:15.525 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5809999999999996e-06 2024-09-21T10:56:15.525 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:15.525 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:15.525 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:15.525 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:15.525 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T10:56:15.525 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 9 2024-09-21T10:56:15.525 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:56:15.550 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:15.582 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:15.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:15.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:15.683 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:15.716 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:15.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:15.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:15.778 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:56:15.818 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:15.850 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:15.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:15.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:15.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:15 smithi070 ceph-mon[107385]: pgmap v2826: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.1 GiB data, 20 GiB used, 337 GiB / 358 GiB avail; 0 B/s wr, 1.24k op/s; 31702/300492 objects degraded (10.550%); 1148/300492 objects misplaced (0.382%); 29 MiB/s, 471 objects/s recovering 2024-09-21T10:56:15.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:15 smithi070 ceph-mon[112139]: pgmap v2826: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.1 GiB data, 20 GiB used, 337 GiB / 358 GiB avail; 0 B/s wr, 1.24k op/s; 31702/300492 objects degraded (10.550%); 1148/300492 objects misplaced (0.382%); 29 MiB/s, 471 objects/s recovering 2024-09-21T10:56:15.952 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:15.986 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:15.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:15.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:16.088 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:16.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:15 smithi160 ceph-mon[101879]: pgmap v2826: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.1 GiB data, 20 GiB used, 337 GiB / 358 GiB avail; 0 B/s wr, 1.24k op/s; 31702/300492 objects degraded (10.550%); 1148/300492 objects misplaced (0.382%); 29 MiB/s, 471 objects/s recovering 2024-09-21T10:56:16.126 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:16.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:16.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:16.227 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:16.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:16.263 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:16.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:16.285 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:56:16.365 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:16.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:16.400 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:16.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:16.501 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:16.529 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:16.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:16.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:16.542 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7] out_osds: [2, 5, 0, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:56:16.542 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:56:16.542 INFO:tasks.thrashosds.thrasher:Reweighting osd 3 to 0.2979121117504502 2024-09-21T10:56:16.542 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 3 0.2979121117504502 2024-09-21T10:56:16.632 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:16.650 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:56:16.681 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:16.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:16.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:16.782 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:16.813 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:16.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:16.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:16.915 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:16.937 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:16.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:16.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:17.039 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:17.074 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:17.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:17.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:17.176 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:17.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:17.215 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:17.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:17.316 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:17.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:17.353 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:17.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:17.454 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:17.486 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:17.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:17.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:17.588 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:17.627 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:17.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:17.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:17.659 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:56:17.728 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:17.755 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:17.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:17.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:17.857 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:17.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:17.897 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:17.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:17.998 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:18.038 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:18.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:18.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:18.141 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:18.180 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:18.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:18.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:18.279 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:18 smithi070 ceph-mon[107385]: pgmap v2827: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.1 GiB data, 20 GiB used, 337 GiB / 358 GiB avail; 0 B/s wr, 988 op/s; 31702/300492 objects degraded (10.550%); 1148/300492 objects misplaced (0.382%); 23 MiB/s, 360 objects/s recovering 2024-09-21T10:56:18.279 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:18 smithi070 ceph-mon[112139]: pgmap v2827: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.1 GiB data, 20 GiB used, 337 GiB / 358 GiB avail; 0 B/s wr, 988 op/s; 31702/300492 objects degraded (10.550%); 1148/300492 objects misplaced (0.382%); 23 MiB/s, 360 objects/s recovering 2024-09-21T10:56:18.282 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:18.310 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:18.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:18.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:18.313 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:56:18.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:18 smithi160 ceph-mon[101879]: pgmap v2827: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.1 GiB data, 20 GiB used, 337 GiB / 358 GiB avail; 0 B/s wr, 988 op/s; 31702/300492 objects degraded (10.550%); 1148/300492 objects misplaced (0.382%); 23 MiB/s, 360 objects/s recovering 2024-09-21T10:56:18.362 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:56:18.412 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:18.446 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:18.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:18.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:18.548 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:18.586 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:18.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:18.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:18.688 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:18.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:18.724 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:18.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:18.825 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:18.861 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:18.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:18.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:18.936 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:56:18.936 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T10:56:18.936 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T10:56:18.936 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T10:56:18.937 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:56:18.962 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:19.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:19.005 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:19.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:19.063 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T10:56:19.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:19 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 31702/300492 objects degraded (10.550%), 15 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:56:19.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:19 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3634146177' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:56:19.106 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:19.138 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:19.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:19.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:19.160 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:19 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 31702/300492 objects degraded (10.550%), 15 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:56:19.160 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:19 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3634146177' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:56:19.161 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:19 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 31702/300492 objects degraded (10.550%), 15 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:56:19.161 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:19 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3634146177' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:56:19.240 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:19.275 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:19.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:19.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:19.377 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:19.409 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:19.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:19.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:19.511 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:19.547 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:19.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:19.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:19.649 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:19.684 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:19.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:19.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:19.724 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:56:19.786 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:19.815 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:19.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:19.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:19.917 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:19.949 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:19.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:19.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:20.051 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:20.063 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:56:20.081 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:20.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:20.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:20.183 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:20.216 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:20.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:20.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:20.317 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:20.340 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:20 smithi160 ceph-mon[101879]: pgmap v2828: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.0 GiB data, 20 GiB used, 338 GiB / 358 GiB avail; 0 B/s wr, 888 op/s; 30563/293832 objects degraded (10.402%); 1148/293832 objects misplaced (0.391%); 23 MiB/s, 372 objects/s recovering 2024-09-21T10:56:20.340 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:20 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:56:20.340 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:20 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3634146177' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:56:20.340 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:20 smithi160 ceph-mon[101879]: osdmap e1506: 8 total, 8 up, 4 in 2024-09-21T10:56:20.354 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:20.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:20.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:20.358 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:20 smithi070 ceph-mon[107385]: pgmap v2828: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.0 GiB data, 20 GiB used, 338 GiB / 358 GiB avail; 0 B/s wr, 888 op/s; 30563/293832 objects degraded (10.402%); 1148/293832 objects misplaced (0.391%); 23 MiB/s, 372 objects/s recovering 2024-09-21T10:56:20.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:20 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:56:20.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:20 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3634146177' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:56:20.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:20 smithi070 ceph-mon[107385]: osdmap e1506: 8 total, 8 up, 4 in 2024-09-21T10:56:20.359 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:20 smithi070 ceph-mon[112139]: pgmap v2828: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.0 GiB data, 20 GiB used, 338 GiB / 358 GiB avail; 0 B/s wr, 888 op/s; 30563/293832 objects degraded (10.402%); 1148/293832 objects misplaced (0.391%); 23 MiB/s, 372 objects/s recovering 2024-09-21T10:56:20.359 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:20 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:56:20.359 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:20 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3634146177' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:56:20.359 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:20 smithi070 ceph-mon[112139]: osdmap e1506: 8 total, 8 up, 4 in 2024-09-21T10:56:20.457 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:20.494 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:20.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:20.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:20.595 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:20.628 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:20.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:20.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:20.729 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:20.765 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:20.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:20.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:20.867 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:20.898 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:20.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:20.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:21.000 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:21.032 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:21.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:21.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:21.135 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:21.173 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:21.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:21.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:21.275 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:21.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:21.310 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:21.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:21.411 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:21.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:21.445 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:21.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:21.545 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:21.577 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:21.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:21.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:21.669 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:56:21.669 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:56:21.679 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:21.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:21.718 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:21.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:21.758 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:56:21.758 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:56:21.759 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:56:21.759 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:56:21.820 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:21.855 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:21.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:21.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:21.957 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:21.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:21.981 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:21.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:22.082 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:22.133 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:22.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:22.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:22.235 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:22.264 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:56:22.264 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:56:22.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:22.269 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:22.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:22.300 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:56:22.301 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:56:22.331 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:56:22.331 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:56:22.368 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:56:22.368 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:56:22.370 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:22.393 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:56:22.393 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:56:22.395 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:22.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:22.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:22.410 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:56:22.410 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:56:22.426 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.3 to 0.297912 (4c43) 2024-09-21T10:56:22.447 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:56:22.447 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:56:22.482 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:56:22.482 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:56:22.497 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:22.518 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:22.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:22.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:22.619 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:22 smithi070 ceph-mon[107385]: pgmap v2830: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 5.8 GiB data, 19 GiB used, 338 GiB / 358 GiB avail; 0 B/s wr, 1.10k op/s; 29927/285393 objects degraded (10.486%); 733/285393 objects misplaced (0.257%); 28 MiB/s, 446 objects/s recovering 2024-09-21T10:56:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:22 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1153097503' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:56:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:22 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3698646768' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.2979121117504502} : dispatch 2024-09-21T10:56:22.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:22 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.2979121117504502} : dispatch 2024-09-21T10:56:22.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:22 smithi070 ceph-mon[112139]: pgmap v2830: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 5.8 GiB data, 19 GiB used, 338 GiB / 358 GiB avail; 0 B/s wr, 1.10k op/s; 29927/285393 objects degraded (10.486%); 733/285393 objects misplaced (0.257%); 28 MiB/s, 446 objects/s recovering 2024-09-21T10:56:22.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:22 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1153097503' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:56:22.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:22 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3698646768' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.2979121117504502} : dispatch 2024-09-21T10:56:22.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:22 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.2979121117504502} : dispatch 2024-09-21T10:56:22.642 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:22.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:22.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:22.743 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:22.764 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:22.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:22.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:22 smithi160 ceph-mon[101879]: pgmap v2830: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 5.8 GiB data, 19 GiB used, 338 GiB / 358 GiB avail; 0 B/s wr, 1.10k op/s; 29927/285393 objects degraded (10.486%); 733/285393 objects misplaced (0.257%); 28 MiB/s, 446 objects/s recovering 2024-09-21T10:56:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:22 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1153097503' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:56:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:22 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3698646768' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.2979121117504502} : dispatch 2024-09-21T10:56:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:22 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.2979121117504502} : dispatch 2024-09-21T10:56:22.866 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:22.887 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:22.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:22.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:22.989 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:23.010 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:23.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:23.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:23.112 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:23.133 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:23.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:23.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:23.235 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:23.256 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:23.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:23.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:23.358 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:23.381 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:23.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:23.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:23.437 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:56:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:56:23] "GET /metrics HTTP/1.1" 200 36633 "" "Prometheus/2.51.0" 2024-09-21T10:56:23.483 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:23.503 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:23.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:23.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:23.525 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:56:23.605 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:23.626 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:23.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:23.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:23.728 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:23.749 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:23.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:23.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:23 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.2979121117504502}]': finished 2024-09-21T10:56:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:23 smithi160 ceph-mon[101879]: osdmap e1507: 8 total, 8 up, 4 in 2024-09-21T10:56:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:23 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 29927/285393 objects degraded (10.486%), 15 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:56:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:23 smithi160 ceph-mon[101879]: osdmap e1508: 8 total, 8 up, 4 in 2024-09-21T10:56:23.851 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:23.872 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:23.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:23.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:23.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:23 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.2979121117504502}]': finished 2024-09-21T10:56:23.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:23 smithi070 ceph-mon[107385]: osdmap e1507: 8 total, 8 up, 4 in 2024-09-21T10:56:23.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:23 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 29927/285393 objects degraded (10.486%), 15 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:56:23.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:23 smithi070 ceph-mon[107385]: osdmap e1508: 8 total, 8 up, 4 in 2024-09-21T10:56:23.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:23 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.2979121117504502}]': finished 2024-09-21T10:56:23.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:23 smithi070 ceph-mon[112139]: osdmap e1507: 8 total, 8 up, 4 in 2024-09-21T10:56:23.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:23 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 29927/285393 objects degraded (10.486%), 15 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:56:23.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:23 smithi070 ceph-mon[112139]: osdmap e1508: 8 total, 8 up, 4 in 2024-09-21T10:56:23.973 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:23.995 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:23.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:23.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:24.096 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:24.118 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:24.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:24.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:24.220 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:24.241 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:24.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:24.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:24.343 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:24.364 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:24.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:24.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:24.466 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:24.509 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:24.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:24.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:24.611 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:24.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:24.643 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:24.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:24.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:24 smithi160 ceph-mon[101879]: pgmap v2832: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 5.7 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 0 B/s wr, 892 op/s; 28709/279054 objects degraded (10.288%); 733/279054 objects misplaced (0.263%); 27 MiB/s, 425 objects/s recovering 2024-09-21T10:56:24.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:24 smithi160 ceph-mon[101879]: osdmap e1509: 8 total, 8 up, 4 in 2024-09-21T10:56:24.744 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:24.783 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:24.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:24.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:24.806 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:24 smithi070 ceph-mon[107385]: pgmap v2832: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 5.7 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 0 B/s wr, 892 op/s; 28709/279054 objects degraded (10.288%); 733/279054 objects misplaced (0.263%); 27 MiB/s, 425 objects/s recovering 2024-09-21T10:56:24.806 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:24 smithi070 ceph-mon[107385]: osdmap e1509: 8 total, 8 up, 4 in 2024-09-21T10:56:24.806 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:24 smithi070 ceph-mon[112139]: pgmap v2832: 109 pgs: 12 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 5.7 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 0 B/s wr, 892 op/s; 28709/279054 objects degraded (10.288%); 733/279054 objects misplaced (0.263%); 27 MiB/s, 425 objects/s recovering 2024-09-21T10:56:24.806 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:24 smithi070 ceph-mon[112139]: osdmap e1509: 8 total, 8 up, 4 in 2024-09-21T10:56:24.885 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:24.908 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:24.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:24.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:25.009 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:25.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:25.032 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:25.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:25.133 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:25.156 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:25.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:25.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:25.259 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:25.281 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:25.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:25.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:25.382 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:25.407 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:25.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:25.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:25.509 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:25.543 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:25.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:25.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:25.645 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:25.667 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:25.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:25.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:25.768 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:25.797 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:25.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:25.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:25 smithi160 ceph-mon[101879]: 56.c starting backfill to osd.7 from (0'0,0'0] MAX to 1506'8854 2024-09-21T10:56:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:25 smithi160 ceph-mon[101879]: 56.18 starting backfill to osd.1 from (0'0,0'0] MAX to 1506'7514 2024-09-21T10:56:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:25 smithi160 ceph-mon[101879]: 56.8 starting backfill to osd.1 from (0'0,0'0] MAX to 1506'7556 2024-09-21T10:56:25.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:25 smithi160 ceph-mon[101879]: 56.b starting backfill to osd.1 from (0'0,0'0] MAX to 1506'8790 2024-09-21T10:56:25.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:25 smithi160 ceph-mon[101879]: 56.15 starting backfill to osd.1 from (0'0,0'0] MAX to 1506'7565 2024-09-21T10:56:25.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:25 smithi160 ceph-mon[101879]: 56.5 starting backfill to osd.1 from (0'0,0'0] MAX to 1506'7504 2024-09-21T10:56:25.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:25 smithi160 ceph-mon[101879]: 56.17 starting backfill to osd.1 from (0'0,0'0] MAX to 1506'7399 2024-09-21T10:56:25.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:25 smithi160 ceph-mon[101879]: 56.19 starting backfill to osd.1 from (0'0,0'0] MAX to 1506'7384 2024-09-21T10:56:25.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:25 smithi160 ceph-mon[101879]: 56.7 starting backfill to osd.1 from (0'0,0'0] MAX to 1506'7447 2024-09-21T10:56:25.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:25 smithi160 ceph-mon[101879]: 56.16 starting backfill to osd.6 from (0'0,0'0] MAX to 1506'7335 2024-09-21T10:56:25.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:25 smithi160 ceph-mon[101879]: 56.6 starting backfill to osd.6 from (0'0,0'0] MAX to 1506'7413 2024-09-21T10:56:25.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:25 smithi160 ceph-mon[101879]: 56.12 starting backfill to osd.6 from (0'0,0'0] MAX to 1506'7486 2024-09-21T10:56:25.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:25 smithi160 ceph-mon[101879]: 56.9 starting backfill to osd.1 from (0'0,0'0] MAX to 1506'7420 2024-09-21T10:56:25.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:25 smithi160 ceph-mon[101879]: 56.2 starting backfill to osd.6 from (0'0,0'0] MAX to 1506'7549 2024-09-21T10:56:25.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:25 smithi160 ceph-mon[101879]: 56.d starting backfill to osd.1 from (0'0,0'0] MAX to 1506'8904 2024-09-21T10:56:25.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:25 smithi160 ceph-mon[101879]: osdmap e1510: 8 total, 8 up, 4 in 2024-09-21T10:56:25.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[107385]: 56.c starting backfill to osd.7 from (0'0,0'0] MAX to 1506'8854 2024-09-21T10:56:25.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[107385]: 56.18 starting backfill to osd.1 from (0'0,0'0] MAX to 1506'7514 2024-09-21T10:56:25.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[107385]: 56.8 starting backfill to osd.1 from (0'0,0'0] MAX to 1506'7556 2024-09-21T10:56:25.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[107385]: 56.b starting backfill to osd.1 from (0'0,0'0] MAX to 1506'8790 2024-09-21T10:56:25.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[107385]: 56.15 starting backfill to osd.1 from (0'0,0'0] MAX to 1506'7565 2024-09-21T10:56:25.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[107385]: 56.5 starting backfill to osd.1 from (0'0,0'0] MAX to 1506'7504 2024-09-21T10:56:25.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[107385]: 56.17 starting backfill to osd.1 from (0'0,0'0] MAX to 1506'7399 2024-09-21T10:56:25.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[107385]: 56.19 starting backfill to osd.1 from (0'0,0'0] MAX to 1506'7384 2024-09-21T10:56:25.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[107385]: 56.7 starting backfill to osd.1 from (0'0,0'0] MAX to 1506'7447 2024-09-21T10:56:25.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[107385]: 56.16 starting backfill to osd.6 from (0'0,0'0] MAX to 1506'7335 2024-09-21T10:56:25.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[107385]: 56.6 starting backfill to osd.6 from (0'0,0'0] MAX to 1506'7413 2024-09-21T10:56:25.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[107385]: 56.12 starting backfill to osd.6 from (0'0,0'0] MAX to 1506'7486 2024-09-21T10:56:25.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[107385]: 56.9 starting backfill to osd.1 from (0'0,0'0] MAX to 1506'7420 2024-09-21T10:56:25.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[107385]: 56.2 starting backfill to osd.6 from (0'0,0'0] MAX to 1506'7549 2024-09-21T10:56:25.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[107385]: 56.d starting backfill to osd.1 from (0'0,0'0] MAX to 1506'8904 2024-09-21T10:56:25.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[107385]: osdmap e1510: 8 total, 8 up, 4 in 2024-09-21T10:56:25.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[112139]: 56.c starting backfill to osd.7 from (0'0,0'0] MAX to 1506'8854 2024-09-21T10:56:25.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[112139]: 56.18 starting backfill to osd.1 from (0'0,0'0] MAX to 1506'7514 2024-09-21T10:56:25.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[112139]: 56.8 starting backfill to osd.1 from (0'0,0'0] MAX to 1506'7556 2024-09-21T10:56:25.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[112139]: 56.b starting backfill to osd.1 from (0'0,0'0] MAX to 1506'8790 2024-09-21T10:56:25.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[112139]: 56.15 starting backfill to osd.1 from (0'0,0'0] MAX to 1506'7565 2024-09-21T10:56:25.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[112139]: 56.5 starting backfill to osd.1 from (0'0,0'0] MAX to 1506'7504 2024-09-21T10:56:25.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[112139]: 56.17 starting backfill to osd.1 from (0'0,0'0] MAX to 1506'7399 2024-09-21T10:56:25.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[112139]: 56.19 starting backfill to osd.1 from (0'0,0'0] MAX to 1506'7384 2024-09-21T10:56:25.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[112139]: 56.7 starting backfill to osd.1 from (0'0,0'0] MAX to 1506'7447 2024-09-21T10:56:25.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[112139]: 56.16 starting backfill to osd.6 from (0'0,0'0] MAX to 1506'7335 2024-09-21T10:56:25.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[112139]: 56.6 starting backfill to osd.6 from (0'0,0'0] MAX to 1506'7413 2024-09-21T10:56:25.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[112139]: 56.12 starting backfill to osd.6 from (0'0,0'0] MAX to 1506'7486 2024-09-21T10:56:25.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[112139]: 56.9 starting backfill to osd.1 from (0'0,0'0] MAX to 1506'7420 2024-09-21T10:56:25.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[112139]: 56.2 starting backfill to osd.6 from (0'0,0'0] MAX to 1506'7549 2024-09-21T10:56:25.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[112139]: 56.d starting backfill to osd.1 from (0'0,0'0] MAX to 1506'8904 2024-09-21T10:56:25.880 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:25 smithi070 ceph-mon[112139]: osdmap e1510: 8 total, 8 up, 4 in 2024-09-21T10:56:25.898 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:25.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:25.936 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:25.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:26.037 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:26.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:26.073 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:26.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:26.174 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:26.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:26.207 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:26.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:26.308 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:26.340 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:26.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:26.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:26.442 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:26.474 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:26.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:26.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:26.576 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:26.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:26.611 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:26.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:26.712 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:26.746 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:26.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:26.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:26 smithi160 ceph-mon[101879]: pgmap v2835: 109 pgs: 1 active+undersized+remapped, 2 remapped+peering, 11 active+recovery_wait+degraded, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 5.6 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 0 B/s wr, 827 op/s; 22876/275340 objects degraded (8.308%); 1220/275340 objects misplaced (0.443%); 28 MiB/s, 441 objects/s recovering 2024-09-21T10:56:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:26 smithi160 ceph-mon[101879]: osdmap e1511: 8 total, 8 up, 4 in 2024-09-21T10:56:26.848 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:26.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:26 smithi070 ceph-mon[112139]: pgmap v2835: 109 pgs: 1 active+undersized+remapped, 2 remapped+peering, 11 active+recovery_wait+degraded, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 5.6 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 0 B/s wr, 827 op/s; 22876/275340 objects degraded (8.308%); 1220/275340 objects misplaced (0.443%); 28 MiB/s, 441 objects/s recovering 2024-09-21T10:56:26.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:26 smithi070 ceph-mon[112139]: osdmap e1511: 8 total, 8 up, 4 in 2024-09-21T10:56:26.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:26 smithi070 ceph-mon[107385]: pgmap v2835: 109 pgs: 1 active+undersized+remapped, 2 remapped+peering, 11 active+recovery_wait+degraded, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 5.6 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 0 B/s wr, 827 op/s; 22876/275340 objects degraded (8.308%); 1220/275340 objects misplaced (0.443%); 28 MiB/s, 441 objects/s recovering 2024-09-21T10:56:26.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:26 smithi070 ceph-mon[107385]: osdmap e1511: 8 total, 8 up, 4 in 2024-09-21T10:56:26.882 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:26.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:26.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:26.984 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:27.015 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:27.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:27.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:27.119 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:27.150 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:27.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:27.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:27.251 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:27.284 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:27.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:27.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:27.386 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:27.415 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:27.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:27.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:27.517 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:27.553 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:27.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:27.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:27.655 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:27.691 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:27.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:27.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:27.793 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:27.825 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:27.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:27.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:27.927 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:27.956 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:27.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:27.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:28.058 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:28.095 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:28.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:28.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:28.197 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:28.228 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:28.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:28.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:28.330 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:28.352 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:28.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:28.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:28.454 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:28.482 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:28.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:28.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:28.523 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:56:28.523 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:56:28.583 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:28.620 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:28.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:28.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:28.722 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:28.750 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:28.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:28.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:28 smithi160 ceph-mon[101879]: pgmap v2838: 109 pgs: 1 active+undersized+remapped, 2 remapped+peering, 11 active+recovery_wait+degraded, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 5.6 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 0 B/s wr, 309 op/s; 22876/275340 objects degraded (8.308%); 1220/275340 objects misplaced (0.443%); 17 MiB/s, 274 objects/s recovering 2024-09-21T10:56:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:28 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 22876/275340 objects degraded (8.308%), 13 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:56:28.852 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:28.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:28 smithi070 ceph-mon[107385]: pgmap v2838: 109 pgs: 1 active+undersized+remapped, 2 remapped+peering, 11 active+recovery_wait+degraded, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 5.6 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 0 B/s wr, 309 op/s; 22876/275340 objects degraded (8.308%); 1220/275340 objects misplaced (0.443%); 17 MiB/s, 274 objects/s recovering 2024-09-21T10:56:28.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:28 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 22876/275340 objects degraded (8.308%), 13 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:56:28.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:28 smithi070 ceph-mon[112139]: pgmap v2838: 109 pgs: 1 active+undersized+remapped, 2 remapped+peering, 11 active+recovery_wait+degraded, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 5.6 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 0 B/s wr, 309 op/s; 22876/275340 objects degraded (8.308%); 1220/275340 objects misplaced (0.443%); 17 MiB/s, 274 objects/s recovering 2024-09-21T10:56:28.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:28 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 22876/275340 objects degraded (8.308%), 13 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:56:28.885 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:28.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:28.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:28.987 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:29.021 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:29.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:29.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:29.123 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:29.156 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:29.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:29.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:29.258 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:29.293 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:29.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:29.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:29.373 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:56:29.395 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:29.462 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:29.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:29.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:29.564 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:29.592 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:29.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:29.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:29.694 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:29.725 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:29.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:29.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:29.728 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:56:29.827 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:29.849 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:29.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:29.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:29.951 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:29.984 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:29.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:29.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:30.086 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:30.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:30.121 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:30.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:30.222 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:30.259 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:30.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:30.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:30.361 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:30.392 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:30.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:30.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:30.494 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:30.530 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:30.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:30.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:30.632 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:30.666 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:30.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:30.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:30.768 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:30.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:30.802 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:30.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:30 smithi160 ceph-mon[101879]: pgmap v2839: 109 pgs: 1 active+recovery_wait+remapped, 9 active+recovery_wait+degraded+remapped, 1 active+undersized+remapped, 2 remapped+peering, 2 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 87 active+clean; 5.5 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 0 B/s wr, 517 op/s; 22132/270324 objects degraded (8.187%); 37027/270324 objects misplaced (13.697%); 20 MiB/s, 329 objects/s recovering 2024-09-21T10:56:30.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:30 smithi070 ceph-mon[107385]: pgmap v2839: 109 pgs: 1 active+recovery_wait+remapped, 9 active+recovery_wait+degraded+remapped, 1 active+undersized+remapped, 2 remapped+peering, 2 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 87 active+clean; 5.5 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 0 B/s wr, 517 op/s; 22132/270324 objects degraded (8.187%); 37027/270324 objects misplaced (13.697%); 20 MiB/s, 329 objects/s recovering 2024-09-21T10:56:30.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:30 smithi070 ceph-mon[112139]: pgmap v2839: 109 pgs: 1 active+recovery_wait+remapped, 9 active+recovery_wait+degraded+remapped, 1 active+undersized+remapped, 2 remapped+peering, 2 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 87 active+clean; 5.5 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 0 B/s wr, 517 op/s; 22132/270324 objects degraded (8.187%); 37027/270324 objects misplaced (13.697%); 20 MiB/s, 329 objects/s recovering 2024-09-21T10:56:30.903 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:30.925 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:30.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:30.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:31.027 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:31.061 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:31.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:31.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:31.162 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:31.196 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:31.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:31.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:31.298 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:31.336 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:31.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:31.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:31.438 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:31.468 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:31.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:31.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:31.570 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:31.598 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:31.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:31.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:31.700 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:31.738 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:31.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:31.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:31.840 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:31.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:31.875 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:31.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:31.976 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:31.981 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:56:31.981 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:56:32.005 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:32.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:32.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:32.107 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:32.141 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:32.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:32.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:32.242 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:32.269 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:32.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:32.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:32.370 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:32.405 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:32.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:32.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:32.506 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:32.527 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:32.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:32.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:32.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:32 smithi160 ceph-mon[101879]: pgmap v2840: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 89 active+clean; 5.4 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 0 B/s wr, 500 op/s; 26061/266538 objects degraded (9.778%); 38160/266538 objects misplaced (14.317%); 13 MiB/s, 212 objects/s recovering 2024-09-21T10:56:32.629 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:32.661 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:32.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:32.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:32.724 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:56:32.763 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:32.795 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:32.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:32.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:32.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:32 smithi070 ceph-mon[107385]: pgmap v2840: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 89 active+clean; 5.4 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 0 B/s wr, 500 op/s; 26061/266538 objects degraded (9.778%); 38160/266538 objects misplaced (14.317%); 13 MiB/s, 212 objects/s recovering 2024-09-21T10:56:32.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:32 smithi070 ceph-mon[112139]: pgmap v2840: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 89 active+clean; 5.4 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 0 B/s wr, 500 op/s; 26061/266538 objects degraded (9.778%); 38160/266538 objects misplaced (14.317%); 13 MiB/s, 212 objects/s recovering 2024-09-21T10:56:32.897 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:32.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:32.937 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:32.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:33.038 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:33.067 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:56:33.079 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:33.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:33.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:33.151 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:56:33.182 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:33.229 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:33.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:33.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:33.331 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:33.366 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:33.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:33.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:33.434 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:56:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:56:33] "GET /metrics HTTP/1.1" 200 36633 "" "Prometheus/2.51.0" 2024-09-21T10:56:33.468 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:33.498 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:33.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:33.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:33.600 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:33.639 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:33.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:33.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:33.725 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:33 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 26061/266538 objects degraded (9.778%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:56:33.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:33 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 26061/266538 objects degraded (9.778%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:56:33.740 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:33.776 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:33.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:33.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:33 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 26061/266538 objects degraded (9.778%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:56:33.877 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:33.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:33.912 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:33.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:34.013 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:34.039 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:34.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:34.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:34.140 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:34.173 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:34.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:34.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:34.275 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:34.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:34.314 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:34.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:34.415 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:34.450 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:34.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:34.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:34.552 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:34.581 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:34.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:34.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:34.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:34 smithi160 ceph-mon[101879]: pgmap v2841: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 89 active+clean; 5.3 GiB data, 18 GiB used, 339 GiB / 358 GiB avail; 0 B/s wr, 710 op/s; 24624/258294 objects degraded (9.533%); 36812/258294 objects misplaced (14.252%); 22 MiB/s, 355 objects/s recovering 2024-09-21T10:56:34.683 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:34.715 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:34.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:34.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:34.756 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:34 smithi070 ceph-mon[112139]: pgmap v2841: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 89 active+clean; 5.3 GiB data, 18 GiB used, 339 GiB / 358 GiB avail; 0 B/s wr, 710 op/s; 24624/258294 objects degraded (9.533%); 36812/258294 objects misplaced (14.252%); 22 MiB/s, 355 objects/s recovering 2024-09-21T10:56:34.756 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:34 smithi070 ceph-mon[107385]: pgmap v2841: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 89 active+clean; 5.3 GiB data, 18 GiB used, 339 GiB / 358 GiB avail; 0 B/s wr, 710 op/s; 24624/258294 objects degraded (9.533%); 36812/258294 objects misplaced (14.252%); 22 MiB/s, 355 objects/s recovering 2024-09-21T10:56:34.817 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:34.844 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:34.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:34.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:34.946 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:34.980 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:34.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:34.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:35.083 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:35.117 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:56:35.118 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:35.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:35.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:35.220 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:35.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:35.257 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:35.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:35.358 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:35.407 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:35.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:35.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:35.460 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:56:35.509 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:35.540 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:35.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:35.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:35.641 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:35.663 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:35.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:35.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:35.755 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:56:35.755 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:56:35.765 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:35.800 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:35.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:35.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:35.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:56:35.902 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:35.938 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:35.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:35.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:36.040 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:36.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:36.075 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:36.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:36.176 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:36.203 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:36.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:36.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:36.304 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:36.335 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:36.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:36.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:36.437 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:36.468 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:36.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:36.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:36.493 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T10:56:36.570 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:36.605 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:36.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:36.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:36.708 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:36.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:36.742 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:36.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:36.820 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:36 smithi160 ceph-mon[101879]: pgmap v2842: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 89 active+clean; 5.1 GiB data, 18 GiB used, 340 GiB / 358 GiB avail; 0 B/s wr, 772 op/s; 24159/253044 objects degraded (9.547%); 36604/253044 objects misplaced (14.465%); 23 MiB/s, 365 objects/s recovering 2024-09-21T10:56:36.820 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:36 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/476259704' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:56:36.820 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:36 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:56:36.844 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:36.872 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:36.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:36.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:36.874 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:56:36.874 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:56:36.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:36 smithi070 ceph-mon[107385]: pgmap v2842: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 89 active+clean; 5.1 GiB data, 18 GiB used, 340 GiB / 358 GiB avail; 0 B/s wr, 772 op/s; 24159/253044 objects degraded (9.547%); 36604/253044 objects misplaced (14.465%); 23 MiB/s, 365 objects/s recovering 2024-09-21T10:56:36.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:36 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/476259704' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:56:36.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:36 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:56:36.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:36 smithi070 ceph-mon[112139]: pgmap v2842: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 89 active+clean; 5.1 GiB data, 18 GiB used, 340 GiB / 358 GiB avail; 0 B/s wr, 772 op/s; 24159/253044 objects degraded (9.547%); 36604/253044 objects misplaced (14.465%); 23 MiB/s, 365 objects/s recovering 2024-09-21T10:56:36.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:36 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/476259704' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:56:36.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:36 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:56:36.974 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:37.016 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:37.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:37.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:37.118 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:37.148 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:37.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:37.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:37.249 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:37.286 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:37.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:37.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:37.388 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:37.416 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:37.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:37.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:37.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:37 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:56:37.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:37 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:56:37.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:37 smithi070 ceph-mon[107385]: osdmap e1512: 8 total, 8 up, 4 in 2024-09-21T10:56:37.505 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:37 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:56:37.506 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:37 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:56:37.506 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:37 smithi070 ceph-mon[112139]: osdmap e1512: 8 total, 8 up, 4 in 2024-09-21T10:56:37.517 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:37.561 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:37.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:37.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:37.570 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:56:37.663 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:37.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:37.698 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:37.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:37.799 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:37.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:37.833 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:37.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:37.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:37 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:56:37.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:37 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:56:37.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:37 smithi160 ceph-mon[101879]: osdmap e1512: 8 total, 8 up, 4 in 2024-09-21T10:56:37.934 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:37.937 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:56:37.939 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7] out_osds: [2, 5, 0, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:56:37.939 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:56:37.939 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-21T10:56:37.940 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-21T10:56:37.980 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:37.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:37.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:38.082 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:38.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:38.117 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:38.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:38.173 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:56:38.174 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:56:38.209 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:56:38.209 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:56:38.218 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:38.241 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:56:38.241 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:56:38.252 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:38.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:38.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:38.280 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:56:38.280 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:56:38.314 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:56:38.314 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:56:38.319 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:56:38.344 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:56:38.344 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:56:38.354 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:38.375 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:56:38.375 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:56:38.416 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:56:38.417 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:56:38.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:38.420 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:38.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:38.521 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:38.551 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:38.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:38.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:38.603 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:38 smithi070 ceph-mon[107385]: pgmap v2844: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 89 active+clean; 5.1 GiB data, 18 GiB used, 340 GiB / 358 GiB avail; 0 B/s wr, 742 op/s; 24159/253044 objects degraded (9.547%); 36604/253044 objects misplaced (14.465%); 22 MiB/s, 351 objects/s recovering 2024-09-21T10:56:38.603 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:38 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4143616293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:56:38.603 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:38 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 24159/253044 objects degraded (9.547%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:56:38.603 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:38 smithi070 ceph-mon[112139]: pgmap v2844: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 89 active+clean; 5.1 GiB data, 18 GiB used, 340 GiB / 358 GiB avail; 0 B/s wr, 742 op/s; 24159/253044 objects degraded (9.547%); 36604/253044 objects misplaced (14.465%); 22 MiB/s, 351 objects/s recovering 2024-09-21T10:56:38.604 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:38 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4143616293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:56:38.604 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:38 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 24159/253044 objects degraded (9.547%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:56:38.653 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:38.691 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:38.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:38.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:38.793 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:38.831 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:38.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:38.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:38 smithi160 ceph-mon[101879]: pgmap v2844: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 89 active+clean; 5.1 GiB data, 18 GiB used, 340 GiB / 358 GiB avail; 0 B/s wr, 742 op/s; 24159/253044 objects degraded (9.547%); 36604/253044 objects misplaced (14.465%); 22 MiB/s, 351 objects/s recovering 2024-09-21T10:56:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:38 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4143616293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:56:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:38 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 24159/253044 objects degraded (9.547%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:56:38.933 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:38.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:38.974 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:38.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:39.075 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:39.106 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:39.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:39.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:39.208 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:39.230 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:39.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:39.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:39.332 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:39.367 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:39.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:39.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:39.469 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:39.505 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:39.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:39.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:39.607 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:39.643 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:39.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:39.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:39.745 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:39.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:39.775 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:39.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:39.855 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:56:39.876 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:39.911 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:39.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:39.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:40.013 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:40.047 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:40.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:40.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:40.149 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:40.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:40.185 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:40.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:40.286 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:40.320 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:40.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:40.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:40.422 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:40.449 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:40.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:40.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:40.551 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:40.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:40.586 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:40.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:40.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:40 smithi070 ceph-mon[107385]: pgmap v2845: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 89 active+clean; 5.0 GiB data, 18 GiB used, 340 GiB / 358 GiB avail; 0 B/s wr, 878 op/s; 22679/243948 objects degraded (9.297%); 35068/243948 objects misplaced (14.375%); 27 MiB/s, 424 objects/s recovering 2024-09-21T10:56:40.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:40 smithi070 ceph-mon[112139]: pgmap v2845: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 89 active+clean; 5.0 GiB data, 18 GiB used, 340 GiB / 358 GiB avail; 0 B/s wr, 878 op/s; 22679/243948 objects degraded (9.297%); 35068/243948 objects misplaced (14.375%); 27 MiB/s, 424 objects/s recovering 2024-09-21T10:56:40.687 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:40.719 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:40.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:40.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:40.820 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:40 smithi160 ceph-mon[101879]: pgmap v2845: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 89 active+clean; 5.0 GiB data, 18 GiB used, 340 GiB / 358 GiB avail; 0 B/s wr, 878 op/s; 22679/243948 objects degraded (9.297%); 35068/243948 objects misplaced (14.375%); 27 MiB/s, 424 objects/s recovering 2024-09-21T10:56:40.849 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:40.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:40.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:40.951 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:40.984 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:40.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:40.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:41.086 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:41.115 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:41.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:41.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:41.216 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:41.249 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:41.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:41.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:41.351 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:41.368 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:56:41.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:41.385 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:41.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:41.483 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:56:41.483 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T10:56:41.483 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T10:56:41.483 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T10:56:41.483 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.483 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:76013 55.6 55:62b4fd39:::benchmark_data_smithi070_294394_object76012:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1427)", 2024-09-21T10:56:41.483 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:49:43.870385+0000", 2024-09-21T10:56:41.483 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 417.60783239400001, 2024-09-21T10:56:41.483 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2166969070000002, 2024-09-21T10:56:41.484 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:41.484 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:56:41.484 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:56:41.484 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:56:41.484 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:56:41.484 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 76013 2024-09-21T10:56:41.484 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.484 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:41.484 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.484 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:41.484 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.870385+0000", 2024-09-21T10:56:41.484 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.484 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.485 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.485 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:41.485 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.870385+0000", 2024-09-21T10:56:41.485 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.485 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.485 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.485 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:41.485 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.870384+0000", 2024-09-21T10:56:41.485 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:56:41.485 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.485 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.485 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:41.485 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.870414+0000", 2024-09-21T10:56:41.485 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0156000000000001e-05 2024-09-21T10:56:41.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.486 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.486 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:41.486 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.870423+0000", 2024-09-21T10:56:41.486 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2699999999999993e-06 2024-09-21T10:56:41.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.486 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.486 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:41.486 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.870430+0000", 2024-09-21T10:56:41.486 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0360000000000001e-06 2024-09-21T10:56:41.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.486 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.486 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.486 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.870591+0000", 2024-09-21T10:56:41.487 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016091799999999999 2024-09-21T10:56:41.487 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.487 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.487 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.487 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.870602+0000", 2024-09-21T10:56:41.487 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0454e-05 2024-09-21T10:56:41.487 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.487 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.487 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.487 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:44.852669+0000", 2024-09-21T10:56:41.487 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98206790799999999 2024-09-21T10:56:41.487 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.488 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.488 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.488 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:44.852685+0000", 2024-09-21T10:56:41.488 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5865000000000001e-05 2024-09-21T10:56:41.488 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.488 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.488 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.488 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.066038+0000", 2024-09-21T10:56:41.488 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.213352574 2024-09-21T10:56:41.488 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.488 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.488 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:56:41.488 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.066244+0000", 2024-09-21T10:56:41.488 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00020609799999999999 2024-09-21T10:56:41.488 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.489 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.489 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:56:41.489 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.066433+0000", 2024-09-21T10:56:41.489 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00018909600000000001 2024-09-21T10:56:41.489 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.489 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.489 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:56:41.489 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.073362+0000", 2024-09-21T10:56:41.489 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.006928862 2024-09-21T10:56:41.489 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.489 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.489 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:56:41.490 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.079289+0000", 2024-09-21T10:56:41.490 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0059268510000000003 2024-09-21T10:56:41.490 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.490 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.490 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:56:41.490 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.084504+0000", 2024-09-21T10:56:41.490 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0052148359999999996 2024-09-21T10:56:41.490 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.490 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.490 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:56:41.490 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.084543+0000", 2024-09-21T10:56:41.490 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9851000000000002e-05 2024-09-21T10:56:41.490 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.491 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.491 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:56:41.491 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.087082+0000", 2024-09-21T10:56:41.491 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0025385429999999999 2024-09-21T10:56:41.491 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.491 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:41.491 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.491 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.491 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.491 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:76032 55.6 55:62366b66:::benchmark_data_smithi070_294394_object76031:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1427)", 2024-09-21T10:56:41.491 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:49:43.888229+0000", 2024-09-21T10:56:41.491 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 417.58998806099999, 2024-09-21T10:56:41.491 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2159101369999998, 2024-09-21T10:56:41.492 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:41.492 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:56:41.492 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:56:41.492 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:56:41.492 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:56:41.492 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 76032 2024-09-21T10:56:41.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.492 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:41.492 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.492 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:41.492 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.888229+0000", 2024-09-21T10:56:41.492 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.492 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:41.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.888229+0000", 2024-09-21T10:56:41.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:41.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.888229+0000", 2024-09-21T10:56:41.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:56:41.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:41.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.888300+0000", 2024-09-21T10:56:41.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1427999999999994e-05 2024-09-21T10:56:41.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:41.494 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.888308+0000", 2024-09-21T10:56:41.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9370000000000004e-06 2024-09-21T10:56:41.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.494 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.494 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:41.494 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.888313+0000", 2024-09-21T10:56:41.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3329999999999996e-06 2024-09-21T10:56:41.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.494 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.494 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.494 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.888364+0000", 2024-09-21T10:56:41.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1110999999999999e-05 2024-09-21T10:56:41.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.495 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.495 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.495 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:43.888373+0000", 2024-09-21T10:56:41.495 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8049999999999996e-06 2024-09-21T10:56:41.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.495 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.495 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.495 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:44.852810+0000", 2024-09-21T10:56:41.495 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96443674099999999 2024-09-21T10:56:41.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.495 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.495 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.495 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:44.852821+0000", 2024-09-21T10:56:41.496 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0567999999999999e-05 2024-09-21T10:56:41.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.496 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.496 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.070983+0000", 2024-09-21T10:56:41.496 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.218162725 2024-09-21T10:56:41.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.496 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:56:41.496 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.071093+0000", 2024-09-21T10:56:41.496 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000109423 2024-09-21T10:56:41.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.496 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:56:41.497 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.071208+0000", 2024-09-21T10:56:41.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000114925 2024-09-21T10:56:41.497 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:41.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.498 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:56:41.498 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.073472+0000", 2024-09-21T10:56:41.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022645640000000002 2024-09-21T10:56:41.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.499 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:56:41.499 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.087145+0000", 2024-09-21T10:56:41.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.013673087 2024-09-21T10:56:41.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.499 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:56:41.499 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.101970+0000", 2024-09-21T10:56:41.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.01482488 2024-09-21T10:56:41.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.500 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.500 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:56:41.500 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.102017+0000", 2024-09-21T10:56:41.500 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6446000000000002e-05 2024-09-21T10:56:41.500 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.500 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.500 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:56:41.500 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:49:46.104140+0000", 2024-09-21T10:56:41.500 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021229420000000001 2024-09-21T10:56:41.500 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.500 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:41.500 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.501 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.501 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.501 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:130857 55.1 55:845d4b54:::benchmark_data_smithi070_294394_object25516:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:56:41.501 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:26.391618+0000", 2024-09-21T10:56:41.501 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 315.086599138, 2024-09-21T10:56:41.501 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.316512366, 2024-09-21T10:56:41.501 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:41.501 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:56:41.501 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:56:41.501 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:56:41.501 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:56:41.501 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 130857 2024-09-21T10:56:41.502 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.502 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:41.502 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.502 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:41.502 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.391618+0000", 2024-09-21T10:56:41.502 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.502 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.502 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.502 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:41.502 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.391618+0000", 2024-09-21T10:56:41.502 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.502 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.502 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.503 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:41.503 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.391617+0000", 2024-09-21T10:56:41.503 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:56:41.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.503 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.503 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:41.503 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.391639+0000", 2024-09-21T10:56:41.503 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2328e-05 2024-09-21T10:56:41.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.503 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.503 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:41.503 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.391650+0000", 2024-09-21T10:56:41.503 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1178999999999999e-05 2024-09-21T10:56:41.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.504 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:41.504 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.391657+0000", 2024-09-21T10:56:41.504 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0339999999999999e-06 2024-09-21T10:56:41.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.504 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.504 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.391726+0000", 2024-09-21T10:56:41.504 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8800000000000005e-05 2024-09-21T10:56:41.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.504 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:56:41.505 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.391904+0000", 2024-09-21T10:56:41.505 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000178355 2024-09-21T10:56:41.505 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.505 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:56:41.505 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.392074+0000", 2024-09-21T10:56:41.505 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0001694 2024-09-21T10:56:41.505 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.505 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:56:41.505 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.395758+0000", 2024-09-21T10:56:41.505 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0036843029999999999 2024-09-21T10:56:41.505 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.505 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:56:41.506 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.399337+0000", 2024-09-21T10:56:41.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0035790639999999999 2024-09-21T10:56:41.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.506 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.506 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.431942+0000", 2024-09-21T10:56:41.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.032605122 2024-09-21T10:56:41.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.506 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.506 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.431958+0000", 2024-09-21T10:56:41.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5485000000000002e-05 2024-09-21T10:56:41.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.507 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.507 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.682756+0000", 2024-09-21T10:56:41.507 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.250798361 2024-09-21T10:56:41.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.507 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.507 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for missing object", 2024-09-21T10:56:41.507 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.682843+0000", 2024-09-21T10:56:41.507 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7089000000000006e-05 2024-09-21T10:56:41.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.507 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.507 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.507 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.708009+0000", 2024-09-21T10:56:41.507 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.025165722000000001 2024-09-21T10:56:41.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.508 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.508 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:56:41.508 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.708131+0000", 2024-09-21T10:56:41.508 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012173200000000001 2024-09-21T10:56:41.508 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.508 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:41.508 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.508 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.508 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:130869 55.1 55:84542c61:::benchmark_data_smithi070_294394_object25528:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:56:41.508 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:26.414265+0000", 2024-09-21T10:56:41.508 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 315.06395238699997, 2024-09-21T10:56:41.509 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.299359347, 2024-09-21T10:56:41.509 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:41.509 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:56:41.509 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:56:41.509 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:56:41.509 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:56:41.509 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 130869 2024-09-21T10:56:41.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.509 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:41.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.509 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:41.509 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.414265+0000", 2024-09-21T10:56:41.509 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.510 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:41.510 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.414265+0000", 2024-09-21T10:56:41.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.510 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:41.510 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.414263+0000", 2024-09-21T10:56:41.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:56:41.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.510 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:41.510 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.414277+0000", 2024-09-21T10:56:41.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4402e-05 2024-09-21T10:56:41.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.511 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:41.511 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.414290+0000", 2024-09-21T10:56:41.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2408999999999999e-05 2024-09-21T10:56:41.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.511 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:41.511 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.414299+0000", 2024-09-21T10:56:41.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1730000000000001e-06 2024-09-21T10:56:41.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.432143+0000", 2024-09-21T10:56:41.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.017843708999999999 2024-09-21T10:56:41.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.432152+0000", 2024-09-21T10:56:41.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6379999999999999e-06 2024-09-21T10:56:41.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.684814+0000", 2024-09-21T10:56:41.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.252661188 2024-09-21T10:56:41.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.513 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:56:41.513 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.684908+0000", 2024-09-21T10:56:41.513 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4697000000000005e-05 2024-09-21T10:56:41.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.513 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:56:41.513 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.685001+0000", 2024-09-21T10:56:41.513 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2702999999999997e-05 2024-09-21T10:56:41.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.513 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:56:41.513 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.690741+0000", 2024-09-21T10:56:41.513 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0057398099999999997 2024-09-21T10:56:41.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.514 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:56:41.514 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.693235+0000", 2024-09-21T10:56:41.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0024941310000000001 2024-09-21T10:56:41.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.514 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:56:41.514 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.712113+0000", 2024-09-21T10:56:41.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.018877636 2024-09-21T10:56:41.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.514 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:56:41.514 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.712146+0000", 2024-09-21T10:56:41.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3723000000000002e-05 2024-09-21T10:56:41.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.515 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:56:41.515 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.713624+0000", 2024-09-21T10:56:41.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001478221 2024-09-21T10:56:41.515 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.515 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:41.515 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.515 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:130875 55.1 55:88335384:::benchmark_data_smithi070_294394_object25534:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1447)", 2024-09-21T10:56:41.515 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:26.472863+0000", 2024-09-21T10:56:41.515 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 315.005354447, 2024-09-21T10:56:41.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2432479830000001, 2024-09-21T10:56:41.516 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:41.516 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:56:41.516 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:56:41.516 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:56:41.516 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:56:41.516 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 130875 2024-09-21T10:56:41.516 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.516 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:41.516 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.516 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:41.516 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.472863+0000", 2024-09-21T10:56:41.516 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.516 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.516 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.517 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:41.517 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.472863+0000", 2024-09-21T10:56:41.517 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.517 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:41.517 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.472861+0000", 2024-09-21T10:56:41.517 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:56:41.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.517 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:41.517 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.472869+0000", 2024-09-21T10:56:41.517 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9410000000000007e-06 2024-09-21T10:56:41.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:41.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.472885+0000", 2024-09-21T10:56:41.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5347000000000001e-05 2024-09-21T10:56:41.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:41.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.472892+0000", 2024-09-21T10:56:41.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4030000000000001e-06 2024-09-21T10:56:41.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.472939+0000", 2024-09-21T10:56:41.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7379999999999997e-05 2024-09-21T10:56:41.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.519 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.519 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.472949+0000", 2024-09-21T10:56:41.519 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3389999999999999e-06 2024-09-21T10:56:41.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.519 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.519 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.688178+0000", 2024-09-21T10:56:41.519 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.215229151 2024-09-21T10:56:41.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.519 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:56:41.519 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.688267+0000", 2024-09-21T10:56:41.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8834999999999999e-05 2024-09-21T10:56:41.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:56:41.520 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.688355+0000", 2024-09-21T10:56:41.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8072999999999995e-05 2024-09-21T10:56:41.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:56:41.520 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.693210+0000", 2024-09-21T10:56:41.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0048552969999999997 2024-09-21T10:56:41.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:56:41.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.693602+0000", 2024-09-21T10:56:41.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00039189599999999998 2024-09-21T10:56:41.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:56:41.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.713683+0000", 2024-09-21T10:56:41.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020080735999999998 2024-09-21T10:56:41.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:56:41.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.713765+0000", 2024-09-21T10:56:41.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2015999999999995e-05 2024-09-21T10:56:41.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.522 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.522 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:56:41.522 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.716111+0000", 2024-09-21T10:56:41.522 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0023462209999999999 2024-09-21T10:56:41.522 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.522 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:41.522 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.522 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.522 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:146722 55.6 55:62c1491e:::benchmark_data_smithi070_294394_object41381:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1454)", 2024-09-21T10:56:41.522 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:02.380351+0000", 2024-09-21T10:56:41.522 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 279.09786672899997, 2024-09-21T10:56:41.523 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2584458490000001, 2024-09-21T10:56:41.523 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:41.523 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:56:41.523 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:56:41.523 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:56:41.523 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:56:41.523 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 146722 2024-09-21T10:56:41.523 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.523 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:41.523 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.523 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:41.523 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.380351+0000", 2024-09-21T10:56:41.524 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.524 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:41.524 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.380351+0000", 2024-09-21T10:56:41.524 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.524 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:41.524 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.380350+0000", 2024-09-21T10:56:41.524 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:56:41.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.524 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:41.524 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.380354+0000", 2024-09-21T10:56:41.525 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3769999999999998e-06 2024-09-21T10:56:41.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.525 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.525 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:41.525 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.380368+0000", 2024-09-21T10:56:41.525 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3932e-05 2024-09-21T10:56:41.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.525 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.525 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:41.525 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.380375+0000", 2024-09-21T10:56:41.525 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5019999999999998e-06 2024-09-21T10:56:41.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.526 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.526 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.526 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.418880+0000", 2024-09-21T10:56:41.526 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.038505466000000002 2024-09-21T10:56:41.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.526 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.526 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:56:41.526 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.419547+0000", 2024-09-21T10:56:41.526 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00066679100000000002 2024-09-21T10:56:41.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.526 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.526 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.526 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.470894+0000", 2024-09-21T10:56:41.526 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.051346507 2024-09-21T10:56:41.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.527 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.527 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:56:41.527 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.470984+0000", 2024-09-21T10:56:41.527 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0383000000000005e-05 2024-09-21T10:56:41.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.527 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.527 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:56:41.527 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.471127+0000", 2024-09-21T10:56:41.527 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014285099999999999 2024-09-21T10:56:41.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.527 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.528 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:56:41.528 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.512854+0000", 2024-09-21T10:56:41.528 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.041727186999999999 2024-09-21T10:56:41.528 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.528 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.528 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.528 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.547436+0000", 2024-09-21T10:56:41.528 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.034581604000000002 2024-09-21T10:56:41.528 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.528 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.528 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.528 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.547449+0000", 2024-09-21T10:56:41.528 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3438000000000001e-05 2024-09-21T10:56:41.528 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.529 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.529 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.529 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.485584+0000", 2024-09-21T10:56:41.529 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.93813491199999999 2024-09-21T10:56:41.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.529 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.529 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for missing object", 2024-09-21T10:56:41.529 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.485670+0000", 2024-09-21T10:56:41.529 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5696000000000001e-05 2024-09-21T10:56:41.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.529 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.529 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.529 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.638739+0000", 2024-09-21T10:56:41.530 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.15306972299999999 2024-09-21T10:56:41.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.530 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.530 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:56:41.530 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.638797+0000", 2024-09-21T10:56:41.530 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7287999999999998e-05 2024-09-21T10:56:41.530 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.530 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:41.530 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.530 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.530 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:146729 55.6 55:608c543e:::benchmark_data_smithi070_294394_object41388:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1454)", 2024-09-21T10:56:41.530 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:02.396871+0000", 2024-09-21T10:56:41.531 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 279.08134691399999, 2024-09-21T10:56:41.531 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2891787770000001, 2024-09-21T10:56:41.531 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:41.531 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:56:41.531 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:56:41.531 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:56:41.531 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:56:41.531 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 146729 2024-09-21T10:56:41.531 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.531 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:41.531 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.531 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:41.532 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.396871+0000", 2024-09-21T10:56:41.532 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.532 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.532 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.532 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:41.532 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.396871+0000", 2024-09-21T10:56:41.532 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.532 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.532 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.532 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:41.532 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.396869+0000", 2024-09-21T10:56:41.532 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:56:41.532 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.532 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.533 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:41.533 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.396875+0000", 2024-09-21T10:56:41.533 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9680000000000003e-06 2024-09-21T10:56:41.533 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.533 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.533 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:41.533 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.396886+0000", 2024-09-21T10:56:41.533 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0462000000000001e-05 2024-09-21T10:56:41.533 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.533 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.533 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:41.533 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.396893+0000", 2024-09-21T10:56:41.533 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.075e-06 2024-09-21T10:56:41.533 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.534 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.534 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.419623+0000", 2024-09-21T10:56:41.534 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.022730345999999998 2024-09-21T10:56:41.534 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.534 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:56:41.534 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.420411+0000", 2024-09-21T10:56:41.534 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00078796900000000002 2024-09-21T10:56:41.534 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.534 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.534 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.490549+0000", 2024-09-21T10:56:41.534 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.070137946000000007 2024-09-21T10:56:41.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.535 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.535 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:56:41.535 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.490671+0000", 2024-09-21T10:56:41.535 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012188099999999999 2024-09-21T10:56:41.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.535 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.535 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:56:41.535 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.490846+0000", 2024-09-21T10:56:41.535 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017487500000000001 2024-09-21T10:56:41.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.535 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.535 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:56:41.535 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.512860+0000", 2024-09-21T10:56:41.536 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.022014140000000001 2024-09-21T10:56:41.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.536 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.536 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.536 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.547736+0000", 2024-09-21T10:56:41.536 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.034875818000000003 2024-09-21T10:56:41.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.536 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.536 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.536 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.547742+0000", 2024-09-21T10:56:41.536 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8379999999999998e-06 2024-09-21T10:56:41.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.536 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.536 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.537 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.638854+0000", 2024-09-21T10:56:41.537 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0911121210000001 2024-09-21T10:56:41.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.537 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for missing object", 2024-09-21T10:56:41.537 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.638991+0000", 2024-09-21T10:56:41.537 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000137224 2024-09-21T10:56:41.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.537 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.537 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.685940+0000", 2024-09-21T10:56:41.537 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.046949079999999997 2024-09-21T10:56:41.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.538 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:56:41.538 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.686049+0000", 2024-09-21T10:56:41.538 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000109361 2024-09-21T10:56:41.538 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.538 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:41.538 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.538 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.538 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:146730 55.6 55:6334822f:::benchmark_data_smithi070_294394_object41389:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1454)", 2024-09-21T10:56:41.538 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:02.400361+0000", 2024-09-21T10:56:41.538 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 279.077856236, 2024-09-21T10:56:41.538 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2884983160000001, 2024-09-21T10:56:41.538 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:41.539 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:56:41.539 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:56:41.539 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:56:41.539 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:56:41.539 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 146730 2024-09-21T10:56:41.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.539 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:41.539 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.539 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:41.539 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.400361+0000", 2024-09-21T10:56:41.539 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.539 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.539 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:41.540 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.400361+0000", 2024-09-21T10:56:41.540 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.540 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.540 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:41.540 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.400360+0000", 2024-09-21T10:56:41.540 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:56:41.540 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.540 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:41.540 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.400374+0000", 2024-09-21T10:56:41.540 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3937999999999999e-05 2024-09-21T10:56:41.540 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.540 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:41.541 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.400388+0000", 2024-09-21T10:56:41.542 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3903000000000001e-05 2024-09-21T10:56:41.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:41.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.400396+0000", 2024-09-21T10:56:41.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.892e-06 2024-09-21T10:56:41.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.420471+0000", 2024-09-21T10:56:41.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020075493999999999 2024-09-21T10:56:41.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:56:41.544 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.423010+0000", 2024-09-21T10:56:41.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0025383329999999998 2024-09-21T10:56:41.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.544 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.500569+0000", 2024-09-21T10:56:41.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.077559272999999998 2024-09-21T10:56:41.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:56:41.544 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.500686+0000", 2024-09-21T10:56:41.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000117562 2024-09-21T10:56:41.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.545 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.545 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:56:41.545 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.500875+0000", 2024-09-21T10:56:41.545 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00018836100000000001 2024-09-21T10:56:41.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.545 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.545 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:56:41.545 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.520179+0000", 2024-09-21T10:56:41.545 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.019303740999999999 2024-09-21T10:56:41.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.545 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.545 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.545 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.547770+0000", 2024-09-21T10:56:41.545 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.027591079000000001 2024-09-21T10:56:41.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.546 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.546 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.547775+0000", 2024-09-21T10:56:41.546 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4110000000000004e-06 2024-09-21T10:56:41.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.546 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.546 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.639033+0000", 2024-09-21T10:56:41.546 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.091257674 2024-09-21T10:56:41.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.546 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for missing object", 2024-09-21T10:56:41.547 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.639116+0000", 2024-09-21T10:56:41.547 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3152999999999995e-05 2024-09-21T10:56:41.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.547 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.547 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.688752+0000", 2024-09-21T10:56:41.547 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.049635771000000002 2024-09-21T10:56:41.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.547 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:56:41.547 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.688860+0000", 2024-09-21T10:56:41.547 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010795599999999999 2024-09-21T10:56:41.547 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.548 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:41.548 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.548 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.548 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.548 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:146734 55.6 55:6eeaca4a:::benchmark_data_smithi070_294394_object41393:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1454)", 2024-09-21T10:56:41.548 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:02.407189+0000", 2024-09-21T10:56:41.548 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 279.07102819900001, 2024-09-21T10:56:41.548 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.286442066, 2024-09-21T10:56:41.548 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:41.548 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:56:41.548 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:56:41.548 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:56:41.548 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:56:41.549 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 146734 2024-09-21T10:56:41.549 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.549 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:41.549 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.549 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:41.549 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.407189+0000", 2024-09-21T10:56:41.549 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.549 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.549 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.549 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:41.549 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.407189+0000", 2024-09-21T10:56:41.549 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.549 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.549 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.550 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:41.550 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.407189+0000", 2024-09-21T10:56:41.550 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T10:56:41.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.550 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:41.550 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.407192+0000", 2024-09-21T10:56:41.550 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.661e-06 2024-09-21T10:56:41.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.550 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:41.550 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.407197+0000", 2024-09-21T10:56:41.550 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2310000000000004e-06 2024-09-21T10:56:41.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.551 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.551 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:41.551 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.407201+0000", 2024-09-21T10:56:41.551 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1670000000000001e-06 2024-09-21T10:56:41.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.551 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.551 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.551 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.423053+0000", 2024-09-21T10:56:41.551 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.015852485999999999 2024-09-21T10:56:41.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.551 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.551 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:56:41.551 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.424099+0000", 2024-09-21T10:56:41.552 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0010458519999999999 2024-09-21T10:56:41.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.552 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.552 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.508628+0000", 2024-09-21T10:56:41.552 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.084528206999999994 2024-09-21T10:56:41.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.552 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:56:41.552 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.508733+0000", 2024-09-21T10:56:41.552 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000105766 2024-09-21T10:56:41.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.553 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:56:41.553 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.508833+0000", 2024-09-21T10:56:41.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9946999999999997e-05 2024-09-21T10:56:41.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.553 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:56:41.553 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.523612+0000", 2024-09-21T10:56:41.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.014778952 2024-09-21T10:56:41.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.553 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.553 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.547798+0000", 2024-09-21T10:56:41.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.024186181000000001 2024-09-21T10:56:41.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.547803+0000", 2024-09-21T10:56:41.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5199999999999999e-06 2024-09-21T10:56:41.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.639166+0000", 2024-09-21T10:56:41.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.091363597 2024-09-21T10:56:41.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for missing object", 2024-09-21T10:56:41.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.639287+0000", 2024-09-21T10:56:41.555 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000120599 2024-09-21T10:56:41.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.555 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.555 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.555 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.693534+0000", 2024-09-21T10:56:41.555 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.054246757999999999 2024-09-21T10:56:41.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.555 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.555 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:56:41.555 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.693631+0000", 2024-09-21T10:56:41.555 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7510999999999995e-05 2024-09-21T10:56:41.555 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.555 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:41.556 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.556 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.556 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.556 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:146738 55.6 55:63e86696:::benchmark_data_smithi070_294394_object41397:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1454)", 2024-09-21T10:56:41.556 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:02.413330+0000", 2024-09-21T10:56:41.556 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 279.06488702399997, 2024-09-21T10:56:41.556 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2826502529999999, 2024-09-21T10:56:41.556 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:41.556 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:56:41.556 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:56:41.556 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:56:41.556 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:56:41.556 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 146738 2024-09-21T10:56:41.556 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.556 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:41.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.557 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:41.557 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.413330+0000", 2024-09-21T10:56:41.557 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.557 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:41.557 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.413330+0000", 2024-09-21T10:56:41.557 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.557 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:41.557 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.413329+0000", 2024-09-21T10:56:41.558 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:56:41.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.558 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:41.558 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.413335+0000", 2024-09-21T10:56:41.558 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0009999999999998e-06 2024-09-21T10:56:41.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.558 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:41.558 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.413346+0000", 2024-09-21T10:56:41.558 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0199e-05 2024-09-21T10:56:41.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.559 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:41.559 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.413353+0000", 2024-09-21T10:56:41.559 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1860000000000003e-06 2024-09-21T10:56:41.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.559 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.559 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.559 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.424137+0000", 2024-09-21T10:56:41.559 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.010784401000000001 2024-09-21T10:56:41.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.559 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.559 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:56:41.559 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.425137+0000", 2024-09-21T10:56:41.559 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00099991199999999994 2024-09-21T10:56:41.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.560 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.516609+0000", 2024-09-21T10:56:41.560 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.091472328000000006 2024-09-21T10:56:41.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.560 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:56:41.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.516684+0000", 2024-09-21T10:56:41.560 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4868000000000002e-05 2024-09-21T10:56:41.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.560 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:56:41.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.516798+0000", 2024-09-21T10:56:41.561 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000113317 2024-09-21T10:56:41.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.561 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:56:41.561 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.527281+0000", 2024-09-21T10:56:41.561 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.010483233 2024-09-21T10:56:41.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.561 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.561 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.547825+0000", 2024-09-21T10:56:41.561 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020543766000000001 2024-09-21T10:56:41.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.561 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.562 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.547829+0000", 2024-09-21T10:56:41.562 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4490000000000001e-06 2024-09-21T10:56:41.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.562 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.562 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.562 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.639335+0000", 2024-09-21T10:56:41.562 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0915056139999999 2024-09-21T10:56:41.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.562 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.562 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for missing object", 2024-09-21T10:56:41.562 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.639455+0000", 2024-09-21T10:56:41.562 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119914 2024-09-21T10:56:41.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.563 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.563 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.563 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.695926+0000", 2024-09-21T10:56:41.563 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.056471806999999999 2024-09-21T10:56:41.563 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.563 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.563 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:56:41.563 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.695981+0000", 2024-09-21T10:56:41.563 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4290999999999999e-05 2024-09-21T10:56:41.563 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.563 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:41.563 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.563 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.563 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.564 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:146752 55.6 55:611d662e:::benchmark_data_smithi070_294394_object41411:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1454)", 2024-09-21T10:56:41.564 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:02.441233+0000", 2024-09-21T10:56:41.564 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 279.036984469, 2024-09-21T10:56:41.564 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2732895870000001, 2024-09-21T10:56:41.564 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:41.564 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:56:41.564 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:56:41.564 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:56:41.564 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:56:41.564 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 146752 2024-09-21T10:56:41.564 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.564 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:41.564 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.565 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:41.565 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.441233+0000", 2024-09-21T10:56:41.565 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.565 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.565 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:41.565 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.441233+0000", 2024-09-21T10:56:41.565 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.565 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.565 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:41.565 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.441228+0000", 2024-09-21T10:56:41.565 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999948 2024-09-21T10:56:41.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.566 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.566 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:41.566 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.441243+0000", 2024-09-21T10:56:41.566 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5234e-05 2024-09-21T10:56:41.566 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.566 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.566 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:41.566 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.441260+0000", 2024-09-21T10:56:41.566 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7045e-05 2024-09-21T10:56:41.566 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.566 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.566 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:41.566 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.441269+0000", 2024-09-21T10:56:41.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6999999999999997e-06 2024-09-21T10:56:41.567 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.567 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.567 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.567 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.441640+0000", 2024-09-21T10:56:41.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00037076500000000001 2024-09-21T10:56:41.567 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.567 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.567 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:56:41.567 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.442517+0000", 2024-09-21T10:56:41.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00087742699999999996 2024-09-21T10:56:41.567 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.567 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.567 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.568 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.523636+0000", 2024-09-21T10:56:41.568 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.081118862 2024-09-21T10:56:41.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.568 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.568 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:56:41.568 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.523710+0000", 2024-09-21T10:56:41.568 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4362000000000004e-05 2024-09-21T10:56:41.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.568 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.568 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:56:41.568 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.523811+0000", 2024-09-21T10:56:41.568 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010073099999999999 2024-09-21T10:56:41.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.568 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.569 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:56:41.569 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.540240+0000", 2024-09-21T10:56:41.569 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.016429348 2024-09-21T10:56:41.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.569 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.569 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.547851+0000", 2024-09-21T10:56:41.569 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.007610721 2024-09-21T10:56:41.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.569 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.569 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.547855+0000", 2024-09-21T10:56:41.569 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3370000000000001e-06 2024-09-21T10:56:41.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.570 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.570 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.570 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.639488+0000", 2024-09-21T10:56:41.570 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.09163269 2024-09-21T10:56:41.570 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.570 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.570 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for missing object", 2024-09-21T10:56:41.570 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.639629+0000", 2024-09-21T10:56:41.570 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014090399999999999 2024-09-21T10:56:41.570 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.570 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.570 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.570 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.714436+0000", 2024-09-21T10:56:41.571 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.074806579999999998 2024-09-21T10:56:41.571 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.571 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.571 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:56:41.571 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.714523+0000", 2024-09-21T10:56:41.571 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7078999999999998e-05 2024-09-21T10:56:41.571 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.571 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:41.571 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.571 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.571 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.571 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:146754 55.6 55:677ef63c:::benchmark_data_smithi070_294394_object41413:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1454)", 2024-09-21T10:56:41.571 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:02.441308+0000", 2024-09-21T10:56:41.572 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 279.03690901300001, 2024-09-21T10:56:41.572 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.29458509, 2024-09-21T10:56:41.572 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:41.572 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:56:41.572 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:56:41.572 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:56:41.572 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:56:41.572 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 146754 2024-09-21T10:56:41.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.572 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:41.573 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.573 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:41.573 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.441308+0000", 2024-09-21T10:56:41.573 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.573 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.573 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.573 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:41.573 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.441308+0000", 2024-09-21T10:56:41.573 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.573 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.573 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.573 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:41.573 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.441308+0000", 2024-09-21T10:56:41.574 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T10:56:41.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.574 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.574 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:41.574 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.441310+0000", 2024-09-21T10:56:41.574 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3290000000000001e-06 2024-09-21T10:56:41.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.574 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.574 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:41.574 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.441316+0000", 2024-09-21T10:56:41.574 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0799999999999996e-06 2024-09-21T10:56:41.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.574 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.574 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:41.575 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.441319+0000", 2024-09-21T10:56:41.575 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9230000000000004e-06 2024-09-21T10:56:41.575 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.575 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.575 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.575 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.442647+0000", 2024-09-21T10:56:41.575 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001327171 2024-09-21T10:56:41.575 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.575 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.575 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:56:41.577 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.445234+0000", 2024-09-21T10:56:41.577 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002587116 2024-09-21T10:56:41.577 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.577 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.578 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.578 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.532916+0000", 2024-09-21T10:56:41.578 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.087681718000000006 2024-09-21T10:56:41.578 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.578 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.578 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:56:41.578 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.533002+0000", 2024-09-21T10:56:41.578 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6941000000000007e-05 2024-09-21T10:56:41.578 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.578 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.578 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:56:41.578 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.533112+0000", 2024-09-21T10:56:41.578 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000109539 2024-09-21T10:56:41.578 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.579 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.579 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.579 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.547885+0000", 2024-09-21T10:56:41.579 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.014772759999999999 2024-09-21T10:56:41.579 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.579 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.579 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.579 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.547889+0000", 2024-09-21T10:56:41.579 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5299999999999998e-06 2024-09-21T10:56:41.579 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.579 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.579 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.579 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.640986+0000", 2024-09-21T10:56:41.580 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0930970799999999 2024-09-21T10:56:41.580 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.580 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.580 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for missing object", 2024-09-21T10:56:41.580 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.641180+0000", 2024-09-21T10:56:41.580 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000193845 2024-09-21T10:56:41.580 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.580 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.580 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.580 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.735809+0000", 2024-09-21T10:56:41.580 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.094629098999999994 2024-09-21T10:56:41.580 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.580 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.580 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:56:41.581 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.735894+0000", 2024-09-21T10:56:41.581 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4266999999999998e-05 2024-09-21T10:56:41.581 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.581 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:41.581 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.581 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.581 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.581 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:146758 55.6 55:6d3562fe:::benchmark_data_smithi070_294394_object41417:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1454)", 2024-09-21T10:56:41.581 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:02.445037+0000", 2024-09-21T10:56:41.581 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 279.03318075700003, 2024-09-21T10:56:41.581 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2454350249999999, 2024-09-21T10:56:41.581 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:41.581 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:56:41.582 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:56:41.582 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:56:41.582 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:56:41.582 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 146758 2024-09-21T10:56:41.582 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.582 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:41.582 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.582 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:41.582 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.445037+0000", 2024-09-21T10:56:41.582 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.582 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.582 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.582 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:41.582 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.445037+0000", 2024-09-21T10:56:41.583 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.583 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.583 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:41.583 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.445036+0000", 2024-09-21T10:56:41.583 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:56:41.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.583 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.583 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:41.583 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.445040+0000", 2024-09-21T10:56:41.583 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4889999999999997e-06 2024-09-21T10:56:41.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.583 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.583 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:41.584 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.445050+0000", 2024-09-21T10:56:41.584 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3340000000000008e-06 2024-09-21T10:56:41.584 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.584 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.584 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:41.584 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.445055+0000", 2024-09-21T10:56:41.584 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7150000000000003e-06 2024-09-21T10:56:41.584 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.584 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.584 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.584 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.455747+0000", 2024-09-21T10:56:41.584 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.010692141 2024-09-21T10:56:41.584 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.584 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.585 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:56:41.585 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.456332+0000", 2024-09-21T10:56:41.585 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00058443700000000004 2024-09-21T10:56:41.585 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.585 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.585 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.585 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.547913+0000", 2024-09-21T10:56:41.585 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.091581334 2024-09-21T10:56:41.585 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.585 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.585 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.585 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:02.547919+0000", 2024-09-21T10:56:41.585 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2719999999999997e-06 2024-09-21T10:56:41.585 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.586 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.586 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.586 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.641217+0000", 2024-09-21T10:56:41.586 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.09329842 2024-09-21T10:56:41.586 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.586 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.586 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:56:41.586 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.641337+0000", 2024-09-21T10:56:41.586 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011988400000000001 2024-09-21T10:56:41.586 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.586 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.586 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:56:41.587 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.641468+0000", 2024-09-21T10:56:41.587 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013095399999999999 2024-09-21T10:56:41.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.587 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.587 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:56:41.587 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.650531+0000", 2024-09-21T10:56:41.587 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0090633750000000003 2024-09-21T10:56:41.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.587 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.587 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:56:41.587 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.673919+0000", 2024-09-21T10:56:41.587 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.023387827999999999 2024-09-21T10:56:41.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.588 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.588 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:56:41.588 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.688915+0000", 2024-09-21T10:56:41.588 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.014995879 2024-09-21T10:56:41.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.588 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.588 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:56:41.588 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.688968+0000", 2024-09-21T10:56:41.588 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3211000000000003e-05 2024-09-21T10:56:41.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.588 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.588 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:56:41.588 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:03.690472+0000", 2024-09-21T10:56:41.588 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0015036909999999999 2024-09-21T10:56:41.589 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.589 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:41.589 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.589 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.589 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.589 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36012.0:66802 56.9 56:9f3f93eb:::benchmark_data_smithi070_333437_object66801:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1491)", 2024-09-21T10:56:41.589 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:55:00.456449+0000", 2024-09-21T10:56:41.589 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 101.021768087, 2024-09-21T10:56:41.589 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2869491179999999, 2024-09-21T10:56:41.589 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:41.589 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:56:41.589 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:56:41.589 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36012", 2024-09-21T10:56:41.589 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T10:56:41.590 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 66802 2024-09-21T10:56:41.590 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.590 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:41.590 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.590 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:41.590 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.456449+0000", 2024-09-21T10:56:41.590 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.590 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.590 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.590 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:41.590 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.456449+0000", 2024-09-21T10:56:41.590 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.590 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.591 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.591 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:41.591 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.456449+0000", 2024-09-21T10:56:41.591 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:56:41.591 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.591 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.591 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:41.591 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.456493+0000", 2024-09-21T10:56:41.591 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3843e-05 2024-09-21T10:56:41.591 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.591 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.591 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:41.591 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.456502+0000", 2024-09-21T10:56:41.591 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2529999999999993e-06 2024-09-21T10:56:41.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.592 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.592 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:41.592 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.456511+0000", 2024-09-21T10:56:41.592 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1470000000000007e-06 2024-09-21T10:56:41.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.592 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.592 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.592 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.463326+0000", 2024-09-21T10:56:41.592 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0068151749999999997 2024-09-21T10:56:41.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.592 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.592 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.592 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.463338+0000", 2024-09-21T10:56:41.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2162e-05 2024-09-21T10:56:41.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.593 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.593 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:01.451102+0000", 2024-09-21T10:56:41.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98776343200000005 2024-09-21T10:56:41.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.593 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.593 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:01.451163+0000", 2024-09-21T10:56:41.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1539999999999997e-05 2024-09-21T10:56:41.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.593 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.730257+0000", 2024-09-21T10:56:41.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2790933209999999 2024-09-21T10:56:41.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:56:41.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.732040+0000", 2024-09-21T10:56:41.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0017833669999999999 2024-09-21T10:56:41.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:56:41.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.732227+0000", 2024-09-21T10:56:41.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000186672 2024-09-21T10:56:41.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.595 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:56:41.595 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.739967+0000", 2024-09-21T10:56:41.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0077409169999999999 2024-09-21T10:56:41.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.595 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:56:41.595 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.739997+0000", 2024-09-21T10:56:41.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9128e-05 2024-09-21T10:56:41.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.595 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:56:41.595 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.740004+0000", 2024-09-21T10:56:41.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4599999999999997e-06 2024-09-21T10:56:41.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.596 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:56:41.596 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.740055+0000", 2024-09-21T10:56:41.596 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0637999999999999e-05 2024-09-21T10:56:41.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.596 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:56:41.596 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.743399+0000", 2024-09-21T10:56:41.596 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0033438000000000001 2024-09-21T10:56:41.596 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.596 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:41.596 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.597 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36012.0:66803 56.9 56:9f2c0e77:::benchmark_data_smithi070_333437_object66802:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1491)", 2024-09-21T10:56:41.597 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:55:00.460408+0000", 2024-09-21T10:56:41.597 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 101.017809927, 2024-09-21T10:56:41.597 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3194555189999999, 2024-09-21T10:56:41.597 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:41.597 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:56:41.597 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:56:41.597 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36012", 2024-09-21T10:56:41.597 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T10:56:41.597 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 66803 2024-09-21T10:56:41.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.597 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:41.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.597 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:41.598 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.460408+0000", 2024-09-21T10:56:41.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.598 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:41.598 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.460408+0000", 2024-09-21T10:56:41.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.598 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:41.598 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.460406+0000", 2024-09-21T10:56:41.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:56:41.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.599 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:41.599 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.460443+0000", 2024-09-21T10:56:41.599 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7419000000000002e-05 2024-09-21T10:56:41.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.599 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:41.599 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.460452+0000", 2024-09-21T10:56:41.599 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7450000000000001e-06 2024-09-21T10:56:41.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.599 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:41.599 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.460461+0000", 2024-09-21T10:56:41.599 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.636e-06 2024-09-21T10:56:41.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.600 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.600 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.463364+0000", 2024-09-21T10:56:41.600 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002903138 2024-09-21T10:56:41.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.600 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.600 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.463373+0000", 2024-09-21T10:56:41.600 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6440000000000006e-06 2024-09-21T10:56:41.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.601 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.601 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:01.451270+0000", 2024-09-21T10:56:41.601 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98789737399999999 2024-09-21T10:56:41.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:41.601 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:41.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:41.602 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.602 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.603 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.603 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:01.451281+0000", 2024-09-21T10:56:41.603 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0798000000000001e-05 2024-09-21T10:56:41.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.603 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.603 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.737264+0000", 2024-09-21T10:56:41.603 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2859834619999999 2024-09-21T10:56:41.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.603 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:56:41.603 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.737423+0000", 2024-09-21T10:56:41.604 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015861299999999999 2024-09-21T10:56:41.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:56:41.604 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.737595+0000", 2024-09-21T10:56:41.604 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017209199999999999 2024-09-21T10:56:41.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:56:41.604 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.747429+0000", 2024-09-21T10:56:41.604 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0098344829999999998 2024-09-21T10:56:41.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:56:41.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.747484+0000", 2024-09-21T10:56:41.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4518000000000002e-05 2024-09-21T10:56:41.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.605 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:56:41.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.774103+0000", 2024-09-21T10:56:41.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.026619166999999999 2024-09-21T10:56:41.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.605 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:56:41.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.774143+0000", 2024-09-21T10:56:41.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9691999999999999e-05 2024-09-21T10:56:41.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.606 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:56:41.606 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.779863+0000", 2024-09-21T10:56:41.606 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0057202379999999999 2024-09-21T10:56:41.606 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.606 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:41.606 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.606 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36012.0:66811 56.9 56:97893571:::benchmark_data_smithi070_333437_object66810:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1491)", 2024-09-21T10:56:41.606 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:55:00.478723+0000", 2024-09-21T10:56:41.606 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 100.99949429900001, 2024-09-21T10:56:41.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3042930190000002, 2024-09-21T10:56:41.607 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:41.607 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:56:41.607 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:56:41.607 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36012", 2024-09-21T10:56:41.607 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T10:56:41.607 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 66811 2024-09-21T10:56:41.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.607 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:41.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.607 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:41.607 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.478723+0000", 2024-09-21T10:56:41.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.608 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:41.608 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.478723+0000", 2024-09-21T10:56:41.608 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.608 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:41.608 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.478722+0000", 2024-09-21T10:56:41.608 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:56:41.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.608 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:41.608 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.478762+0000", 2024-09-21T10:56:41.608 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0435000000000002e-05 2024-09-21T10:56:41.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:41.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.478781+0000", 2024-09-21T10:56:41.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8802999999999999e-05 2024-09-21T10:56:41.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:41.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.478789+0000", 2024-09-21T10:56:41.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4619999999999996e-06 2024-09-21T10:56:41.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.478913+0000", 2024-09-21T10:56:41.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012334600000000001 2024-09-21T10:56:41.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.478924+0000", 2024-09-21T10:56:41.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0733999999999999e-05 2024-09-21T10:56:41.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:01.451308+0000", 2024-09-21T10:56:41.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.97238486700000004 2024-09-21T10:56:41.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:01.451318+0000", 2024-09-21T10:56:41.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5359999999999998e-06 2024-09-21T10:56:41.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.743468+0000", 2024-09-21T10:56:41.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.292149905 2024-09-21T10:56:41.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:56:41.612 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.743612+0000", 2024-09-21T10:56:41.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000143907 2024-09-21T10:56:41.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.612 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:56:41.612 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.743767+0000", 2024-09-21T10:56:41.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015484900000000001 2024-09-21T10:56:41.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.612 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:56:41.612 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.747491+0000", 2024-09-21T10:56:41.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0037247700000000001 2024-09-21T10:56:41.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:56:41.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.754904+0000", 2024-09-21T10:56:41.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0074122290000000002 2024-09-21T10:56:41.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:56:41.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.779912+0000", 2024-09-21T10:56:41.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.025008389999999998 2024-09-21T10:56:41.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:56:41.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.779995+0000", 2024-09-21T10:56:41.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2614999999999994e-05 2024-09-21T10:56:41.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:56:41.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.783016+0000", 2024-09-21T10:56:41.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0030216000000000002 2024-09-21T10:56:41.614 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.614 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:41.614 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.614 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36012.0:66812 56.7 56:eeaa5c20:::benchmark_data_smithi070_333437_object66811:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1491)", 2024-09-21T10:56:41.614 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:55:00.479101+0000", 2024-09-21T10:56:41.614 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 100.99911658400001, 2024-09-21T10:56:41.615 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2525598900000001, 2024-09-21T10:56:41.615 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:41.615 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:56:41.615 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:56:41.615 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36012", 2024-09-21T10:56:41.615 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T10:56:41.615 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 66812 2024-09-21T10:56:41.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.615 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:41.615 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.615 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:41.615 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.479101+0000", 2024-09-21T10:56:41.615 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.616 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:41.616 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.479101+0000", 2024-09-21T10:56:41.616 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.616 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:41.616 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.479100+0000", 2024-09-21T10:56:41.616 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T10:56:41.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.617 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:41.617 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.479162+0000", 2024-09-21T10:56:41.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2292000000000006e-05 2024-09-21T10:56:41.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.617 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:41.617 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.479174+0000", 2024-09-21T10:56:41.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1815e-05 2024-09-21T10:56:41.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.617 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:41.617 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.479181+0000", 2024-09-21T10:56:41.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3620000000000002e-06 2024-09-21T10:56:41.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.618 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.479228+0000", 2024-09-21T10:56:41.618 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7703000000000001e-05 2024-09-21T10:56:41.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.618 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.479238+0000", 2024-09-21T10:56:41.618 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6679999999999996e-06 2024-09-21T10:56:41.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:01.452258+0000", 2024-09-21T10:56:41.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.97302057399999997 2024-09-21T10:56:41.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.619 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:01.452270+0000", 2024-09-21T10:56:41.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1966e-05 2024-09-21T10:56:41.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.619 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.715624+0000", 2024-09-21T10:56:41.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2633537610000001 2024-09-21T10:56:41.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.620 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:56:41.620 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.717046+0000", 2024-09-21T10:56:41.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0014222029999999999 2024-09-21T10:56:41.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.620 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:56:41.620 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.717150+0000", 2024-09-21T10:56:41.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010349300000000001 2024-09-21T10:56:41.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.620 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:56:41.620 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.721635+0000", 2024-09-21T10:56:41.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0044847200000000002 2024-09-21T10:56:41.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.621 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:56:41.621 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.721645+0000", 2024-09-21T10:56:41.621 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0501e-05 2024-09-21T10:56:41.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.621 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:56:41.621 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.728972+0000", 2024-09-21T10:56:41.621 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0073264749999999998 2024-09-21T10:56:41.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.621 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:56:41.621 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.729034+0000", 2024-09-21T10:56:41.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2677999999999999e-05 2024-09-21T10:56:41.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.622 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:56:41.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.731661+0000", 2024-09-21T10:56:41.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0026265059999999998 2024-09-21T10:56:41.622 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.622 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:41.622 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.622 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36012.0:66815 56.9 56:9270d223:::benchmark_data_smithi070_333437_object66814:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1491)", 2024-09-21T10:56:41.623 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:55:00.482191+0000", 2024-09-21T10:56:41.623 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 100.996026301, 2024-09-21T10:56:41.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3072628229999999, 2024-09-21T10:56:41.623 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:41.623 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:56:41.623 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:56:41.623 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36012", 2024-09-21T10:56:41.623 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T10:56:41.623 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 66815 2024-09-21T10:56:41.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.623 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:41.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:41.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.482191+0000", 2024-09-21T10:56:41.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.624 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:41.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.482191+0000", 2024-09-21T10:56:41.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.624 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:41.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.482190+0000", 2024-09-21T10:56:41.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:56:41.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.625 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:41.625 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.482264+0000", 2024-09-21T10:56:41.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4000999999999998e-05 2024-09-21T10:56:41.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.625 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:41.625 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.482275+0000", 2024-09-21T10:56:41.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0708e-05 2024-09-21T10:56:41.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.625 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:41.625 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.482281+0000", 2024-09-21T10:56:41.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.708e-06 2024-09-21T10:56:41.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.626 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.626 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.626 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.482334+0000", 2024-09-21T10:56:41.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2317000000000001e-05 2024-09-21T10:56:41.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.626 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.626 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.626 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.482345+0000", 2024-09-21T10:56:41.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1212e-05 2024-09-21T10:56:41.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.626 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.626 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:01.451341+0000", 2024-09-21T10:56:41.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96899629300000001 2024-09-21T10:56:41.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:01.451350+0000", 2024-09-21T10:56:41.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2410000000000001e-06 2024-09-21T10:56:41.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.747516+0000", 2024-09-21T10:56:41.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2961652350000001 2024-09-21T10:56:41.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.628 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.628 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:56:41.628 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.747674+0000", 2024-09-21T10:56:41.628 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015796199999999999 2024-09-21T10:56:41.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.628 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.628 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:56:41.628 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.747850+0000", 2024-09-21T10:56:41.628 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017681500000000001 2024-09-21T10:56:41.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.628 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.628 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:56:41.628 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.754754+0000", 2024-09-21T10:56:41.628 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0069037539999999998 2024-09-21T10:56:41.629 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.629 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.629 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:56:41.629 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.756804+0000", 2024-09-21T10:56:41.629 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002049763 2024-09-21T10:56:41.629 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.629 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.629 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:56:41.629 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.786297+0000", 2024-09-21T10:56:41.629 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.029493035000000001 2024-09-21T10:56:41.629 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.629 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.629 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:56:41.630 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.786344+0000", 2024-09-21T10:56:41.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6941999999999999e-05 2024-09-21T10:56:41.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.630 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:56:41.630 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.789454+0000", 2024-09-21T10:56:41.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0031100360000000001 2024-09-21T10:56:41.630 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.630 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:41.630 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.630 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36012.0:66818 56.9 56:9931f1ba:::benchmark_data_smithi070_333437_object66817:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1491)", 2024-09-21T10:56:41.630 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:55:00.487172+0000", 2024-09-21T10:56:41.631 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 100.991045349, 2024-09-21T10:56:41.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3160899480000001, 2024-09-21T10:56:41.631 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:41.631 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:56:41.631 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:56:41.631 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36012", 2024-09-21T10:56:41.631 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T10:56:41.631 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 66818 2024-09-21T10:56:41.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.631 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:41.631 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:41.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.487172+0000", 2024-09-21T10:56:41.632 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.632 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:41.632 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.487172+0000", 2024-09-21T10:56:41.632 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.632 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:41.632 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.487170+0000", 2024-09-21T10:56:41.632 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:56:41.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.633 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:41.633 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.487223+0000", 2024-09-21T10:56:41.633 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2549999999999997e-05 2024-09-21T10:56:41.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.633 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.633 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:41.633 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.487232+0000", 2024-09-21T10:56:41.633 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9379999999999998e-06 2024-09-21T10:56:41.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.633 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.633 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:41.633 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.487236+0000", 2024-09-21T10:56:41.633 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5190000000000003e-06 2024-09-21T10:56:41.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.634 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.634 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.487280+0000", 2024-09-21T10:56:41.634 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4008999999999998e-05 2024-09-21T10:56:41.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.634 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.634 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:00.487288+0000", 2024-09-21T10:56:41.634 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9619999999999993e-06 2024-09-21T10:56:41.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.634 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.635 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:01.451374+0000", 2024-09-21T10:56:41.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96408550400000004 2024-09-21T10:56:41.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.635 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.635 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.635 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:01.451383+0000", 2024-09-21T10:56:41.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4409999999999999e-06 2024-09-21T10:56:41.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.635 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.635 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.635 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.750799+0000", 2024-09-21T10:56:41.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.299415354 2024-09-21T10:56:41.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.635 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:56:41.636 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.751022+0000", 2024-09-21T10:56:41.636 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00022297800000000001 2024-09-21T10:56:41.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:56:41.636 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.751275+0000", 2024-09-21T10:56:41.636 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025366899999999999 2024-09-21T10:56:41.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:56:41.636 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.754981+0000", 2024-09-21T10:56:41.636 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003706053 2024-09-21T10:56:41.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:56:41.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.757028+0000", 2024-09-21T10:56:41.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0020469160000000002 2024-09-21T10:56:41.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:56:41.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.800319+0000", 2024-09-21T10:56:41.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.043290225000000002 2024-09-21T10:56:41.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:56:41.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.800364+0000", 2024-09-21T10:56:41.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5263999999999999e-05 2024-09-21T10:56:41.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:56:41.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:02.803262+0000", 2024-09-21T10:56:41.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0028982420000000001 2024-09-21T10:56:41.638 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.638 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:41.638 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.638 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36012.0:97179 56.7 56:e284a7fd:::benchmark_data_smithi070_333437_object97178:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1499)", 2024-09-21T10:56:41.638 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:55:37.696861+0000", 2024-09-21T10:56:41.638 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 63.781356789999997, 2024-09-21T10:56:41.639 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.686342864, 2024-09-21T10:56:41.639 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:56:41.639 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:56:41.639 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:56:41.639 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36012", 2024-09-21T10:56:41.639 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T10:56:41.639 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 97179 2024-09-21T10:56:41.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.639 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:56:41.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.639 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:56:41.639 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:37.696861+0000", 2024-09-21T10:56:41.639 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.640 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.640 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:56:41.640 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:37.696861+0000", 2024-09-21T10:56:41.640 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:56:41.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.640 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.640 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:56:41.640 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:37.696860+0000", 2024-09-21T10:56:41.640 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T10:56:41.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.640 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.640 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:56:41.640 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:37.696902+0000", 2024-09-21T10:56:41.640 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1814000000000003e-05 2024-09-21T10:56:41.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.641 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:56:41.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:37.696908+0000", 2024-09-21T10:56:41.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1859999999999997e-06 2024-09-21T10:56:41.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.641 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:56:41.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:37.696913+0000", 2024-09-21T10:56:41.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7230000000000004e-06 2024-09-21T10:56:41.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.641 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:37.898420+0000", 2024-09-21T10:56:41.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20150755300000001 2024-09-21T10:56:41.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.642 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.642 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:37.898429+0000", 2024-09-21T10:56:41.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1419999999999993e-06 2024-09-21T10:56:41.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.642 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.642 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:38.364566+0000", 2024-09-21T10:56:41.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.46613732899999999 2024-09-21T10:56:41.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.642 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.643 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:56:41.643 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:38.364577+0000", 2024-09-21T10:56:41.643 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0839000000000001e-05 2024-09-21T10:56:41.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.643 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:56:41.643 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:39.383121+0000", 2024-09-21T10:56:41.643 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.018544656 2024-09-21T10:56:41.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:56:41.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:56:41.643 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:56:41.643 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:39.383204+0000", 2024-09-21T10:56:41.643 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2132000000000006e-05 2024-09-21T10:56:41.643 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.644 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:41.644 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.644 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:56:41.644 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:56:41.644 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:56:41.702 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:41.739 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:41.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:41.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:41.747 INFO:teuthology.orchestra.run.smithi070.stderr:no valid command found; 10 closest matches: 2024-09-21T10:56:41.748 INFO:teuthology.orchestra.run.smithi070.stderr:0 2024-09-21T10:56:41.748 INFO:teuthology.orchestra.run.smithi070.stderr:1 2024-09-21T10:56:41.748 INFO:teuthology.orchestra.run.smithi070.stderr:2 2024-09-21T10:56:41.748 INFO:teuthology.orchestra.run.smithi070.stderr:abort 2024-09-21T10:56:41.748 INFO:teuthology.orchestra.run.smithi070.stderr:assert 2024-09-21T10:56:41.748 INFO:teuthology.orchestra.run.smithi070.stderr:bench [] [] [] [] 2024-09-21T10:56:41.748 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:56:41.748 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs files list 2024-09-21T10:56:41.748 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs stats 2024-09-21T10:56:41.748 INFO:teuthology.orchestra.run.smithi070.stderr:bluestore allocator dump block 2024-09-21T10:56:41.748 INFO:teuthology.orchestra.run.smithi070.stderr:Error EINVAL: invalid command 2024-09-21T10:56:41.841 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:41.861 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:41.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:41.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:41.963 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:41.996 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:41.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:41.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:42.098 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:42.134 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:42.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:42.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:42.236 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:42.274 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:42.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:42.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:42.376 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:42.410 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:42.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:42.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:42.512 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:42.544 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:42.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:42.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:42.646 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:42.679 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:42.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:42.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:42.781 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:42.812 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:42.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:42.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:42 smithi160 ceph-mon[101879]: pgmap v2846: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 89 active+clean; 4.9 GiB data, 17 GiB used, 340 GiB / 358 GiB avail; 0 B/s wr, 930 op/s; 22212/238617 objects degraded (9.309%); 34805/238617 objects misplaced (14.586%); 27 MiB/s, 431 objects/s recovering 2024-09-21T10:56:42.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:42 smithi070 ceph-mon[107385]: pgmap v2846: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 89 active+clean; 4.9 GiB data, 17 GiB used, 340 GiB / 358 GiB avail; 0 B/s wr, 930 op/s; 22212/238617 objects degraded (9.309%); 34805/238617 objects misplaced (14.586%); 27 MiB/s, 431 objects/s recovering 2024-09-21T10:56:42.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:42 smithi070 ceph-mon[112139]: pgmap v2846: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 89 active+clean; 4.9 GiB data, 17 GiB used, 340 GiB / 358 GiB avail; 0 B/s wr, 930 op/s; 22212/238617 objects degraded (9.309%); 34805/238617 objects misplaced (14.586%); 27 MiB/s, 431 objects/s recovering 2024-09-21T10:56:42.914 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:42.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:42.936 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:42.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:42.996 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:56:43.037 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:43.073 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:43.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:43.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:43.174 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:43.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:43.212 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:43.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:43.313 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:43.346 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:43.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:43.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:43.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:56:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:56:43] "GET /metrics HTTP/1.1" 200 36657 "" "Prometheus/2.51.0" 2024-09-21T10:56:43.448 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:43.474 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:43.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:43.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:43.576 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:43.609 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:43.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:43.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:43.711 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:43.744 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:43.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:43.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:43 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 22212/238617 objects degraded (9.309%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:56:43.845 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:43.876 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:43.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:43.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:43.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:43 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 22212/238617 objects degraded (9.309%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:56:43.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:43 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 22212/238617 objects degraded (9.309%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:56:43.978 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:44.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:44.015 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:44.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:44.116 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:44.149 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:44.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:44.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:44.252 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:44.285 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:44.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:44.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:44.387 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:44.420 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:44.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:44.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:44.523 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:44.561 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:44.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:44.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:44.663 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:44.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:44.700 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:44.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:44.768 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:44 smithi070 ceph-mon[107385]: pgmap v2847: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 89 active+clean; 4.7 GiB data, 17 GiB used, 341 GiB / 358 GiB avail; 0 B/s wr, 960 op/s; 20673/229482 objects degraded (9.009%); 33275/229482 objects misplaced (14.500%); 28 MiB/s, 441 objects/s recovering 2024-09-21T10:56:44.769 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:44 smithi070 ceph-mon[112139]: pgmap v2847: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 89 active+clean; 4.7 GiB data, 17 GiB used, 341 GiB / 358 GiB avail; 0 B/s wr, 960 op/s; 20673/229482 objects degraded (9.009%); 33275/229482 objects misplaced (14.500%); 28 MiB/s, 441 objects/s recovering 2024-09-21T10:56:44.801 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:44.839 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:44.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:44.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:44 smithi160 ceph-mon[101879]: pgmap v2847: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 89 active+clean; 4.7 GiB data, 17 GiB used, 341 GiB / 358 GiB avail; 0 B/s wr, 960 op/s; 20673/229482 objects degraded (9.009%); 33275/229482 objects misplaced (14.500%); 28 MiB/s, 441 objects/s recovering 2024-09-21T10:56:44.941 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:44.976 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:44.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:44.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:45.078 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:45.113 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:45.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:45.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:45.215 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:45.249 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:45.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:45.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:45.351 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:45.388 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:45.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:45.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:45.490 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:45.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:45.525 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:45.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:45.581 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:56:45.582 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:56:45.582 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-21T10:56:45.626 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:45.672 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:56:45.702 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:45.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:45.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:45.804 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:45.845 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:45.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:45.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:45.947 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:45.985 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:45.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:45.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:46.087 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:46.119 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:46.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:46.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:46.220 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:46.257 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:46.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:46.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:46.359 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:46.392 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:46.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:46.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:46.494 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:46.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:46.535 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:46.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:46.536 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:56:46.548 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:56:46.636 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:46.670 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:46.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:46.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:46.772 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:46.807 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:46.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:46.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:46 smithi160 ceph-mon[101879]: pgmap v2848: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 89 active+clean; 4.6 GiB data, 17 GiB used, 341 GiB / 358 GiB avail; 0 B/s wr, 974 op/s; 20198/223791 objects degraded (9.025%); 32926/223791 objects misplaced (14.713%); 29 MiB/s, 457 objects/s recovering 2024-09-21T10:56:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:46 smithi160 ceph-mon[101879]: osdmap e1513: 8 total, 8 up, 4 in 2024-09-21T10:56:46.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:46 smithi070 ceph-mon[107385]: pgmap v2848: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 89 active+clean; 4.6 GiB data, 17 GiB used, 341 GiB / 358 GiB avail; 0 B/s wr, 974 op/s; 20198/223791 objects degraded (9.025%); 32926/223791 objects misplaced (14.713%); 29 MiB/s, 457 objects/s recovering 2024-09-21T10:56:46.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:46 smithi070 ceph-mon[107385]: osdmap e1513: 8 total, 8 up, 4 in 2024-09-21T10:56:46.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:46 smithi070 ceph-mon[112139]: pgmap v2848: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 89 active+clean; 4.6 GiB data, 17 GiB used, 341 GiB / 358 GiB avail; 0 B/s wr, 974 op/s; 20198/223791 objects degraded (9.025%); 32926/223791 objects misplaced (14.713%); 29 MiB/s, 457 objects/s recovering 2024-09-21T10:56:46.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:46 smithi070 ceph-mon[112139]: osdmap e1513: 8 total, 8 up, 4 in 2024-09-21T10:56:46.908 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:46.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:46.940 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:46.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:47.030 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:56:47.042 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:47.089 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:47.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:47.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:47.191 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:47.227 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:47.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:47.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:47.328 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:47.366 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:47.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:47.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:47.467 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:47.505 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:47.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:47.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:47.607 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:47.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:47 smithi070 ceph-mon[107385]: osdmap e1514: 8 total, 8 up, 4 in 2024-09-21T10:56:47.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:47 smithi070 ceph-mon[112139]: osdmap e1514: 8 total, 8 up, 4 in 2024-09-21T10:56:47.651 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:47.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:47.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:47.752 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:47.790 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:47.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:47.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:47.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:47 smithi160 ceph-mon[101879]: osdmap e1514: 8 total, 8 up, 4 in 2024-09-21T10:56:47.892 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:47.928 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:47.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:47.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:48.029 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:48.061 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:48.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:48.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:48.163 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:48.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:48.209 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:48.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:48.310 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:48.351 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:48.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:48.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:48.453 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:48.489 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:48.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:48.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:48.536 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T10:56:48.591 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:48.622 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:48.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:48.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:48.724 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:48.757 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:48.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:48.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:48.785 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:48 smithi070 ceph-mon[107385]: pgmap v2851: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 89 active+clean; 4.6 GiB data, 17 GiB used, 341 GiB / 358 GiB avail; 0 B/s wr, 839 op/s; 20198/223791 objects degraded (9.025%); 32926/223791 objects misplaced (14.713%); 24 MiB/s, 386 objects/s recovering 2024-09-21T10:56:48.785 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:48 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 20198/223791 objects degraded (9.025%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:56:48.786 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:48 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2133841452' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:56:48.786 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:48 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:56:48.786 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:48 smithi070 ceph-mon[112139]: pgmap v2851: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 89 active+clean; 4.6 GiB data, 17 GiB used, 341 GiB / 358 GiB avail; 0 B/s wr, 839 op/s; 20198/223791 objects degraded (9.025%); 32926/223791 objects misplaced (14.713%); 24 MiB/s, 386 objects/s recovering 2024-09-21T10:56:48.786 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:48 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 20198/223791 objects degraded (9.025%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:56:48.786 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:48 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2133841452' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:56:48.786 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:48 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:56:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:48 smithi160 ceph-mon[101879]: pgmap v2851: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 89 active+clean; 4.6 GiB data, 17 GiB used, 341 GiB / 358 GiB avail; 0 B/s wr, 839 op/s; 20198/223791 objects degraded (9.025%); 32926/223791 objects misplaced (14.713%); 24 MiB/s, 386 objects/s recovering 2024-09-21T10:56:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:48 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 20198/223791 objects degraded (9.025%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:56:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:48 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2133841452' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:56:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:48 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:56:48.859 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:48.893 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:48.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:48.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:48.994 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:49.032 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:49.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:49.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:49.134 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:49.169 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:49.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:49.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:49.271 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:49.307 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:49.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:49.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:49.408 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:49.443 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:49.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:49.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:49.545 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:49.585 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:49.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:49.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:49.687 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:49.721 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:49.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:49.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:49.823 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:49.831 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:49 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:56:49.831 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:49 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:56:49.831 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:49 smithi070 ceph-mon[107385]: osdmap e1515: 8 total, 8 up, 4 in 2024-09-21T10:56:49.831 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:49 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:56:49.831 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:49 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:56:49.831 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:49 smithi070 ceph-mon[112139]: osdmap e1515: 8 total, 8 up, 4 in 2024-09-21T10:56:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:49 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:56:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:49 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:56:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:49 smithi160 ceph-mon[101879]: osdmap e1515: 8 total, 8 up, 4 in 2024-09-21T10:56:49.871 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:49.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:49.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:49.973 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:50.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:50.014 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:50.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:50.115 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:50.155 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:50.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:50.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:50.256 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:50.288 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:50.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:50.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:50.390 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:50.421 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:50.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:50.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:50.522 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:50.556 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:50.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:50.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:50.658 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:50.694 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:50.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:50.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:50.796 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:50.829 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:50.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:50.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:50 smithi160 ceph-mon[101879]: pgmap v2853: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 90 active+clean; 4.3 GiB data, 16 GiB used, 341 GiB / 358 GiB avail; 0 B/s wr, 883 op/s; 18463/213564 objects degraded (8.645%); 31201/213564 objects misplaced (14.610%); 27 MiB/s, 426 objects/s recovering 2024-09-21T10:56:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:56:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:56:50.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:50 smithi070 ceph-mon[107385]: pgmap v2853: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 90 active+clean; 4.3 GiB data, 16 GiB used, 341 GiB / 358 GiB avail; 0 B/s wr, 883 op/s; 18463/213564 objects degraded (8.645%); 31201/213564 objects misplaced (14.610%); 27 MiB/s, 426 objects/s recovering 2024-09-21T10:56:50.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:56:50.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:56:50.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:50 smithi070 ceph-mon[112139]: pgmap v2853: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 90 active+clean; 4.3 GiB data, 16 GiB used, 341 GiB / 358 GiB avail; 0 B/s wr, 883 op/s; 18463/213564 objects degraded (8.645%); 31201/213564 objects misplaced (14.610%); 27 MiB/s, 426 objects/s recovering 2024-09-21T10:56:50.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:56:50.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:56:50.931 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:50.970 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:50.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:50.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:51.071 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:51.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:51.108 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:51.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:51.209 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:51.241 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:51.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:51.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:51.343 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:51.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:51.380 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:51.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:51.480 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:51.512 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:51.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:51.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:51.613 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:51.645 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:51.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:51.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:51.747 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:51.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:51.781 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:51.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:51.882 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:51.903 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:51.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:51.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:52.005 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:52.037 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:52.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:52.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:52.139 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:52.170 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:52.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:52.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:52.272 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:52.306 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:52.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:52.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:52.408 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:52.448 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:52.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:52.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:52.550 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:52.585 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:52.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:52.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:52.687 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:52.718 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:52.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:52.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:52.820 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:52 smithi160 ceph-mon[101879]: pgmap v2854: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 4.2 GiB data, 16 GiB used, 342 GiB / 358 GiB avail; 0 B/s wr, 914 op/s; 17352/207321 objects degraded (8.370%); 31581/207321 objects misplaced (15.233%); 26 MiB/s, 411 objects/s recovering 2024-09-21T10:56:52.855 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:52.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:52.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:52.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:52 smithi070 ceph-mon[107385]: pgmap v2854: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 4.2 GiB data, 16 GiB used, 342 GiB / 358 GiB avail; 0 B/s wr, 914 op/s; 17352/207321 objects degraded (8.370%); 31581/207321 objects misplaced (15.233%); 26 MiB/s, 411 objects/s recovering 2024-09-21T10:56:52.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:52 smithi070 ceph-mon[112139]: pgmap v2854: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 4.2 GiB data, 16 GiB used, 342 GiB / 358 GiB avail; 0 B/s wr, 914 op/s; 17352/207321 objects degraded (8.370%); 31581/207321 objects misplaced (15.233%); 26 MiB/s, 411 objects/s recovering 2024-09-21T10:56:52.957 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:52.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:52.998 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:52.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:53.032 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:56:53.033 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T10:56:53.033 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T10:56:53.033 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:56:53.099 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:53.131 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:53.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:53.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:53.233 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:53.273 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:53.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:53.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:53.374 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:56:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:56:53] "GET /metrics HTTP/1.1" 200 36661 "" "Prometheus/2.51.0" 2024-09-21T10:56:53.376 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:53.411 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:53.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:53.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:53.514 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:53.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:53.550 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:53.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:53.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:53 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 17352/207321 objects degraded (8.370%), 14 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:56:53.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:53 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 17352/207321 objects degraded (8.370%), 14 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:56:53.651 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:53.689 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:53.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:53.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:53.790 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:53.822 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:53.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:53.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:53 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 17352/207321 objects degraded (8.370%), 14 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:56:53.923 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:53.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:53.959 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:53.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:54.060 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:54.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:54.099 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:54.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:54.199 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:54.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:54.232 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:54.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:54.334 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:54.369 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:54.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:54.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:54.471 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:54.502 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:54.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:54.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:54.604 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:54.649 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:54.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:54.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:54.751 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:54.788 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:54.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:54.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:54 smithi160 ceph-mon[101879]: pgmap v2855: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 4.0 GiB data, 15 GiB used, 342 GiB / 358 GiB avail; 0 B/s wr, 1.21k op/s; 15556/196617 objects degraded (7.912%); 29788/196617 objects misplaced (15.150%); 36 MiB/s, 569 objects/s recovering 2024-09-21T10:56:54.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:54 smithi070 ceph-mon[107385]: pgmap v2855: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 4.0 GiB data, 15 GiB used, 342 GiB / 358 GiB avail; 0 B/s wr, 1.21k op/s; 15556/196617 objects degraded (7.912%); 29788/196617 objects misplaced (15.150%); 36 MiB/s, 569 objects/s recovering 2024-09-21T10:56:54.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:54 smithi070 ceph-mon[112139]: pgmap v2855: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 4.0 GiB data, 15 GiB used, 342 GiB / 358 GiB avail; 0 B/s wr, 1.21k op/s; 15556/196617 objects degraded (7.912%); 29788/196617 objects misplaced (15.150%); 36 MiB/s, 569 objects/s recovering 2024-09-21T10:56:54.890 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:54.923 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:54.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:54.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:55.025 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:55.059 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:55.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:55.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:55.161 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:55.192 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:55.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:55.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:55.294 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:55.324 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:55.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:55.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:55.425 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:55.470 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:55.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:55.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:55.472 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:56:55.573 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:55.605 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:55.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:55.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:55.707 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:55.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:55.746 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:55.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:55.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:55 smithi160 ceph-mon[101879]: osdmap e1516: 8 total, 8 up, 4 in 2024-09-21T10:56:55.847 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:55.874 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:55.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:55.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:55.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:55 smithi070 ceph-mon[107385]: osdmap e1516: 8 total, 8 up, 4 in 2024-09-21T10:56:55.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:55 smithi070 ceph-mon[112139]: osdmap e1516: 8 total, 8 up, 4 in 2024-09-21T10:56:55.977 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:56.008 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:56.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:56.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:56.110 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:56.146 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:56.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:56.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:56.248 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:56.283 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:56.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:56.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:56.290 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:56:56.385 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:56.430 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:56.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:56.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:56.532 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:56.574 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:56.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:56.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:56.676 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:56.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:56.711 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:56.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:56.812 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:56.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:56.839 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:56.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:56.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:56 smithi070 ceph-mon[107385]: pgmap v2857: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 3.9 GiB data, 15 GiB used, 343 GiB / 358 GiB avail; 0 B/s wr, 1.40k op/s; 15153/190068 objects degraded (7.972%); 29384/190068 objects misplaced (15.460%); 40 MiB/s, 633 objects/s recovering 2024-09-21T10:56:56.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:56 smithi070 ceph-mon[107385]: osdmap e1517: 8 total, 8 up, 4 in 2024-09-21T10:56:56.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:56 smithi070 ceph-mon[112139]: pgmap v2857: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 3.9 GiB data, 15 GiB used, 343 GiB / 358 GiB avail; 0 B/s wr, 1.40k op/s; 15153/190068 objects degraded (7.972%); 29384/190068 objects misplaced (15.460%); 40 MiB/s, 633 objects/s recovering 2024-09-21T10:56:56.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:56 smithi070 ceph-mon[112139]: osdmap e1517: 8 total, 8 up, 4 in 2024-09-21T10:56:56.940 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:56.974 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:56.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:56.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:57.076 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:57.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:56 smithi160 ceph-mon[101879]: pgmap v2857: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 3.9 GiB data, 15 GiB used, 343 GiB / 358 GiB avail; 0 B/s wr, 1.40k op/s; 15153/190068 objects degraded (7.972%); 29384/190068 objects misplaced (15.460%); 40 MiB/s, 633 objects/s recovering 2024-09-21T10:56:57.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:56 smithi160 ceph-mon[101879]: osdmap e1517: 8 total, 8 up, 4 in 2024-09-21T10:56:57.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:57.107 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:57.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:57.208 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:57.242 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:57.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:57.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:57.344 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:57.379 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:57.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:57.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:57.480 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:57.513 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:57.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:57.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:57.615 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:57.653 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:57.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:57.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:57.755 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:57.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:57.791 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:57.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:57.892 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:57.923 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:57.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:57.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:58.024 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:58.053 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:58.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:58.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:58.155 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:58.201 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:58.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:58.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:58.302 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:58.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:58.339 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:58.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:58.440 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:56:58.474 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:58.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:58.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:56:58.576 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:56:58.601 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:58.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:58.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:56:58.703 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:58.743 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:58.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:58.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:58.846 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:56:58.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:58 smithi070 ceph-mon[107385]: pgmap v2859: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 3.9 GiB data, 15 GiB used, 343 GiB / 358 GiB avail; 0 B/s wr, 978 op/s; 15153/190068 objects degraded (7.972%); 29384/190068 objects misplaced (15.460%); 26 MiB/s, 416 objects/s recovering 2024-09-21T10:56:58.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:58 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2504706178' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:56:58.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:58 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 15153/190068 objects degraded (7.972%), 14 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:56:58.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:58 smithi070 ceph-mon[112139]: pgmap v2859: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 3.9 GiB data, 15 GiB used, 343 GiB / 358 GiB avail; 0 B/s wr, 978 op/s; 15153/190068 objects degraded (7.972%); 29384/190068 objects misplaced (15.460%); 26 MiB/s, 416 objects/s recovering 2024-09-21T10:56:58.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:58 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2504706178' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:56:58.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:58 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 15153/190068 objects degraded (7.972%), 14 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:56:58.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:58.883 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:58.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:56:58.984 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:56:59.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:59.028 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:59.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:56:59.051 INFO:teuthology.orchestra.run.smithi070.stderr:no valid command found; 10 closest matches: 2024-09-21T10:56:59.052 INFO:teuthology.orchestra.run.smithi070.stderr:0 2024-09-21T10:56:59.052 INFO:teuthology.orchestra.run.smithi070.stderr:1 2024-09-21T10:56:59.052 INFO:teuthology.orchestra.run.smithi070.stderr:2 2024-09-21T10:56:59.052 INFO:teuthology.orchestra.run.smithi070.stderr:abort 2024-09-21T10:56:59.052 INFO:teuthology.orchestra.run.smithi070.stderr:assert 2024-09-21T10:56:59.052 INFO:teuthology.orchestra.run.smithi070.stderr:bench [] [] [] [] 2024-09-21T10:56:59.052 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:56:59.052 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs files list 2024-09-21T10:56:59.052 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs stats 2024-09-21T10:56:59.052 INFO:teuthology.orchestra.run.smithi070.stderr:bluestore allocator dump block 2024-09-21T10:56:59.052 INFO:teuthology.orchestra.run.smithi070.stderr:Error EINVAL: invalid command 2024-09-21T10:56:59.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:58 smithi160 ceph-mon[101879]: pgmap v2859: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 3.9 GiB data, 15 GiB used, 343 GiB / 358 GiB avail; 0 B/s wr, 978 op/s; 15153/190068 objects degraded (7.972%); 29384/190068 objects misplaced (15.460%); 26 MiB/s, 416 objects/s recovering 2024-09-21T10:56:59.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:58 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2504706178' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:56:59.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:58 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 15153/190068 objects degraded (7.972%), 14 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:56:59.129 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:59.167 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:59.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:59.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:59.269 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:56:59.306 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:59.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:59.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:56:59.407 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:56:59.468 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:59.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:59.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:56:59.569 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:59.593 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:59.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:59.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:59.695 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:56:59.728 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:56:59.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:59.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:56:59.830 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:56:59.864 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:56:59.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:56:59.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:56:59.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:56:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:56:59.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:56:59 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:56:59.968 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:00.005 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:00.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:00.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:00.054 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:57:00.054 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:57:00.054 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:57:00.054 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:57:00.054 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:57:00.055 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:57:00.055 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:57:00.055 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:57:00.055 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:57:00.055 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:57:00.056 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:57:00.056 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:57:00.056 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:57:00.056 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:57:00.056 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:57:00.056 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:57:00.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:56:59 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:57:00.106 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:00.139 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:00.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:00.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:00.240 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:00.274 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:00.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:00.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:00.376 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:00.418 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:00.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:00.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:00.521 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:00.558 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:00.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:00.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:00.660 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:00.690 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:00.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:00.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:00.792 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:00.844 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:00.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:00.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:00.853 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:00 smithi160 ceph-mon[101879]: pgmap v2860: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 92 active+clean; 3.7 GiB data, 15 GiB used, 343 GiB / 358 GiB avail; 0 B/s wr, 1.15k op/s; 13372/179610 objects degraded (7.445%); 27652/179610 objects misplaced (15.396%); 34 MiB/s, 547 objects/s recovering 2024-09-21T10:57:00.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:00 smithi070 ceph-mon[107385]: pgmap v2860: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 92 active+clean; 3.7 GiB data, 15 GiB used, 343 GiB / 358 GiB avail; 0 B/s wr, 1.15k op/s; 13372/179610 objects degraded (7.445%); 27652/179610 objects misplaced (15.396%); 34 MiB/s, 547 objects/s recovering 2024-09-21T10:57:00.860 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:00 smithi070 ceph-mon[112139]: pgmap v2860: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 92 active+clean; 3.7 GiB data, 15 GiB used, 343 GiB / 358 GiB avail; 0 B/s wr, 1.15k op/s; 13372/179610 objects degraded (7.445%); 27652/179610 objects misplaced (15.396%); 34 MiB/s, 547 objects/s recovering 2024-09-21T10:57:00.946 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:00.981 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:00.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:00.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:01.083 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:01.118 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:01.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:01.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:01.220 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:01.254 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:01.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:01.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:01.356 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:01.360 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:57:01.360 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:57:01.360 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-21T10:57:01.386 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:01.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:01.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:01.440 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:57:01.447 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:57:01.488 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:01.543 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:01.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:01.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:01.645 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:01.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:01.680 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:01.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:01.781 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:01.811 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:01.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:01.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:01.913 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:01.950 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:01.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:01.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:02.052 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:02.087 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:02.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:02.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:02.189 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:02.233 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:02.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:02.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:02.335 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:02.371 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:02.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:02.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:02.473 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:02.510 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:02.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:02.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:02.612 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:02.640 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:02.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:02.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:02.742 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:02.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:02.788 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:02.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:02.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:02 smithi070 ceph-mon[112139]: pgmap v2861: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.5 GiB data, 14 GiB used, 343 GiB / 358 GiB avail; 0 B/s wr, 957 op/s; 12438/173634 objects degraded (7.163%); 28082/173634 objects misplaced (16.173%); 24 MiB/s, 386 objects/s recovering 2024-09-21T10:57:02.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:02 smithi070 ceph-mon[107385]: pgmap v2861: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.5 GiB data, 14 GiB used, 343 GiB / 358 GiB avail; 0 B/s wr, 957 op/s; 12438/173634 objects degraded (7.163%); 28082/173634 objects misplaced (16.173%); 24 MiB/s, 386 objects/s recovering 2024-09-21T10:57:02.889 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:02.897 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:57:02.931 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:02.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:02.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:02.982 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:02 smithi160 ceph-mon[101879]: pgmap v2861: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.5 GiB data, 14 GiB used, 343 GiB / 358 GiB avail; 0 B/s wr, 957 op/s; 12438/173634 objects degraded (7.163%); 28082/173634 objects misplaced (16.173%); 24 MiB/s, 386 objects/s recovering 2024-09-21T10:57:03.035 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:03.055 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:03.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:03.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:03.157 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:03.190 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:03.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:03.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:03.291 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:03.327 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:03.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:03.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:03.428 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:03.463 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:03.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:03.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:03.565 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:03.600 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:03.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:03.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:03.609 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:57:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:57:03] "GET /metrics HTTP/1.1" 200 36661 "" "Prometheus/2.51.0" 2024-09-21T10:57:03.701 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:03.733 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:03.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:03.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:03.834 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:03.869 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:03.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:03.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:03.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:03 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 12438/173634 objects degraded (7.163%), 13 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:57:03.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:03 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 12438/173634 objects degraded (7.163%), 13 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:57:03.971 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:04.002 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:04.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:04.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:04.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:03 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 12438/173634 objects degraded (7.163%), 13 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:57:04.103 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:04.138 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:04.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:04.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:04.240 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:04.263 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:04.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:04.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:04.365 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:04.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:04.400 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:04.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:04.501 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:04.536 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:04.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:04.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:04.637 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:04.670 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:04.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:04.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:04.772 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:04.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:04.813 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:04.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:04.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:04 smithi160 ceph-mon[101879]: pgmap v2862: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.3 GiB data, 14 GiB used, 344 GiB / 358 GiB avail; 0 B/s wr, 1.07k op/s; 10711/162981 objects degraded (6.572%); 26295/162981 objects misplaced (16.134%); 30 MiB/s, 475 objects/s recovering 2024-09-21T10:57:04.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:04 smithi070 ceph-mon[112139]: pgmap v2862: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.3 GiB data, 14 GiB used, 344 GiB / 358 GiB avail; 0 B/s wr, 1.07k op/s; 10711/162981 objects degraded (6.572%); 26295/162981 objects misplaced (16.134%); 30 MiB/s, 475 objects/s recovering 2024-09-21T10:57:04.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:04 smithi070 ceph-mon[107385]: pgmap v2862: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.3 GiB data, 14 GiB used, 344 GiB / 358 GiB avail; 0 B/s wr, 1.07k op/s; 10711/162981 objects degraded (6.572%); 26295/162981 objects misplaced (16.134%); 30 MiB/s, 475 objects/s recovering 2024-09-21T10:57:04.914 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:04.947 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:04.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:04.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:05.049 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:05.110 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:05.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:05.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:05.212 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:05.243 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:05.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:05.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:05.345 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:05.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:05.384 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:05.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:05.486 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:05.521 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:05.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:05.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:05.623 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:05.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:05.657 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:05.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:05.758 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:05.790 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:05.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:05.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:05.892 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:05.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:05.927 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:05.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:06.028 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:06.063 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:06.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:06.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:06.165 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:06.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:06.199 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:06.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:06.300 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:06.333 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:06.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:06.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:06.371 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:06 smithi070 ceph-mon[112139]: pgmap v2863: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.2 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 0 B/s wr, 1.11k op/s; 10529/156663 objects degraded (6.721%); 25939/156663 objects misplaced (16.557%); 28 MiB/s, 454 objects/s recovering 2024-09-21T10:57:06.371 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:57:06.371 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:57:06.371 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:57:06.371 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:06 smithi070 ceph-mon[107385]: pgmap v2863: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.2 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 0 B/s wr, 1.11k op/s; 10529/156663 objects degraded (6.721%); 25939/156663 objects misplaced (16.557%); 28 MiB/s, 454 objects/s recovering 2024-09-21T10:57:06.371 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:57:06.371 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:57:06.372 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:57:06.435 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:06.438 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:06 smithi160 ceph-mon[101879]: pgmap v2863: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.2 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 0 B/s wr, 1.11k op/s; 10529/156663 objects degraded (6.721%); 25939/156663 objects misplaced (16.557%); 28 MiB/s, 454 objects/s recovering 2024-09-21T10:57:06.438 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:57:06.438 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:57:06.438 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:57:06.489 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:06.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:06.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:06.591 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:06.629 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:06.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:06.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:06.731 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:06.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:06.765 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:06.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:06.866 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:06.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:06.896 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:06.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:06.997 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:07.031 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:07.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:07.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:07.133 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:07.167 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:07.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:07.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:07.268 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:07.303 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:07.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:07.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:07.405 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:07.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:07.439 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:07.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:07.540 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:07.574 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:07.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:07.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:07.676 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:07.709 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:07.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:07.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:07.811 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:07.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:07.844 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:07.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:07.945 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:07.977 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:07.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:07.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:08.079 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:08.120 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:08.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:08.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:08.222 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:08.263 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:08.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:08.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:08.364 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:08.395 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:08.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:08.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:08.493 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:08 smithi070 ceph-mon[112139]: pgmap v2864: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.2 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 0 B/s wr, 974 op/s; 10529/156663 objects degraded (6.721%); 25939/156663 objects misplaced (16.557%); 25 MiB/s, 398 objects/s recovering 2024-09-21T10:57:08.493 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:08 smithi070 ceph-mon[107385]: pgmap v2864: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.2 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 0 B/s wr, 974 op/s; 10529/156663 objects degraded (6.721%); 25939/156663 objects misplaced (16.557%); 25 MiB/s, 398 objects/s recovering 2024-09-21T10:57:08.497 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:08.534 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:08.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:08.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:08 smithi160 ceph-mon[101879]: pgmap v2864: 109 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.2 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 0 B/s wr, 974 op/s; 10529/156663 objects degraded (6.721%); 25939/156663 objects misplaced (16.557%); 25 MiB/s, 398 objects/s recovering 2024-09-21T10:57:08.635 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:08.673 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:08.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:08.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:08.774 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:08.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:08.809 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:08.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:08.910 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:08.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:08.952 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:08.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:09.052 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:09.087 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:09.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:09.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:09.189 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:09.219 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:09.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:09.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:09.321 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:09.352 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:09.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:09.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:09.454 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:09.475 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:09.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:09.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:09.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:09 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 10529/156663 objects degraded (6.721%), 13 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:57:09.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:09 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 10529/156663 objects degraded (6.721%), 13 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:57:09.578 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:09.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:09 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 10529/156663 objects degraded (6.721%), 13 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:57:09.610 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:09.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:09.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:09.660 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:57:09.660 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:57:09.712 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:09.733 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:09.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:09.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:09.835 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:09.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:09.869 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:09.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:09.970 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:09.993 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:09.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:09.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:10.094 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:10.129 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:10.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:10.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:10.231 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:10.253 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:10.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:10.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:10.355 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:10.391 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:10.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:10.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:10.493 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:10.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:10.532 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:10.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:10.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:10 smithi160 ceph-mon[101879]: pgmap v2865: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.0 GiB data, 13 GiB used, 345 GiB / 358 GiB avail; 0 B/s wr, 1.25k op/s; 9420/145155 objects degraded (6.490%); 23543/145155 objects misplaced (16.219%); 29 MiB/s, 471 objects/s recovering 2024-09-21T10:57:10.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:10 smithi070 ceph-mon[107385]: pgmap v2865: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.0 GiB data, 13 GiB used, 345 GiB / 358 GiB avail; 0 B/s wr, 1.25k op/s; 9420/145155 objects degraded (6.490%); 23543/145155 objects misplaced (16.219%); 29 MiB/s, 471 objects/s recovering 2024-09-21T10:57:10.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:10 smithi070 ceph-mon[112139]: pgmap v2865: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.0 GiB data, 13 GiB used, 345 GiB / 358 GiB avail; 0 B/s wr, 1.25k op/s; 9420/145155 objects degraded (6.490%); 23543/145155 objects misplaced (16.219%); 29 MiB/s, 471 objects/s recovering 2024-09-21T10:57:10.633 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:10.667 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:10.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:10.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:10.769 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:10.806 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:10.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:10.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:10.908 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:10.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:10.943 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:10.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:11.044 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:11.067 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:11.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:11.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:11.169 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:11.204 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:11.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:11.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:11.306 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:11.329 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:11.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:11.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:11.431 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:11.460 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:11.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:11.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:11.562 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:11.594 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:11.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:11.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:11.696 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:11.731 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:11.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:11.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:11.833 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:11.870 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:11.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:11.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:11.972 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:12.004 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:12.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:12.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:12.105 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:12.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:12.133 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:12.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:12.234 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:12.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:12.269 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:12.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:12.370 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:12.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:12 smithi070 ceph-mon[107385]: pgmap v2866: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.8 GiB data, 12 GiB used, 345 GiB / 358 GiB avail; 0 B/s wr, 1.18k op/s; 9270/137088 objects degraded (6.762%); 23098/137088 objects misplaced (16.849%); 21 MiB/s, 345 objects/s recovering 2024-09-21T10:57:12.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:12 smithi070 ceph-mon[112139]: pgmap v2866: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.8 GiB data, 12 GiB used, 345 GiB / 358 GiB avail; 0 B/s wr, 1.18k op/s; 9270/137088 objects degraded (6.762%); 23098/137088 objects misplaced (16.849%); 21 MiB/s, 345 objects/s recovering 2024-09-21T10:57:12.391 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:12.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:12.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:12.493 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:12.524 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:12.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:12.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:12.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:12 smithi160 ceph-mon[101879]: pgmap v2866: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.8 GiB data, 12 GiB used, 345 GiB / 358 GiB avail; 0 B/s wr, 1.18k op/s; 9270/137088 objects degraded (6.762%); 23098/137088 objects misplaced (16.849%); 21 MiB/s, 345 objects/s recovering 2024-09-21T10:57:12.626 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:12.661 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:12.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:12.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:12.763 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:12.800 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:57:12.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:12.812 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:12.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:12.893 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:57:12.899 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:57:12.914 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:12.940 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:12.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:12.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:13.041 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:13.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:13.070 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:13.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:13.171 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:13.216 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:13.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:13.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:13.318 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:13.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:13.354 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:13.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:13.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:57:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:57:13] "GET /metrics HTTP/1.1" 200 36661 "" "Prometheus/2.51.0" 2024-09-21T10:57:13.455 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:13.493 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:13.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:13.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:13.596 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:13.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:13.636 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:13.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:13.737 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:13.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:13.770 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:13.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:13.870 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:13.903 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:13.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:13.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:14.005 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:14.044 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:14.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:14.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:14.146 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:14.186 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:14.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:14.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:14.205 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T10:57:14.288 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:14.323 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:14.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:14.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:14.425 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:14.459 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:14.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:14.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:14.561 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:14.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:14 smithi160 ceph-mon[101879]: pgmap v2867: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.5 GiB data, 12 GiB used, 346 GiB / 358 GiB avail; 0 B/s wr, 1.39k op/s; 9182/123717 objects degraded (7.422%); 19598/123717 objects misplaced (15.841%); 18 MiB/s, 303 objects/s recovering 2024-09-21T10:57:14.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:14 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 9182/123717 objects degraded (7.422%), 5 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:57:14.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:14 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/819996465' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:57:14.596 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:14.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:14.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:14 smithi070 ceph-mon[107385]: pgmap v2867: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.5 GiB data, 12 GiB used, 346 GiB / 358 GiB avail; 0 B/s wr, 1.39k op/s; 9182/123717 objects degraded (7.422%); 19598/123717 objects misplaced (15.841%); 18 MiB/s, 303 objects/s recovering 2024-09-21T10:57:14.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:14 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 9182/123717 objects degraded (7.422%), 5 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:57:14.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:14 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/819996465' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:57:14.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:14 smithi070 ceph-mon[112139]: pgmap v2867: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.5 GiB data, 12 GiB used, 346 GiB / 358 GiB avail; 0 B/s wr, 1.39k op/s; 9182/123717 objects degraded (7.422%); 19598/123717 objects misplaced (15.841%); 18 MiB/s, 303 objects/s recovering 2024-09-21T10:57:14.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:14 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 9182/123717 objects degraded (7.422%), 5 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:57:14.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:14 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/819996465' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:57:14.698 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:14.732 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:14.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:14.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:14.834 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:14.868 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:14.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:14.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:14.970 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:15.003 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:15.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:15.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:15.105 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:15.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:15.144 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:15.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:15.189 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:57:15.245 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:15.280 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:15.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:15.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:15.381 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:15.416 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:15.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:15.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:15.518 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:15.552 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:15.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:15.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:15.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:15 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:57:15.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:15 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/819996465' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:57:15.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:15 smithi160 ceph-mon[101879]: osdmap e1518: 8 total, 8 up, 4 in 2024-09-21T10:57:15.614 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:15 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:57:15.614 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:15 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/819996465' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:57:15.614 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:15 smithi070 ceph-mon[107385]: osdmap e1518: 8 total, 8 up, 4 in 2024-09-21T10:57:15.614 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:15 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:57:15.614 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:15 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/819996465' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:57:15.614 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:15 smithi070 ceph-mon[112139]: osdmap e1518: 8 total, 8 up, 4 in 2024-09-21T10:57:15.654 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:15.686 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:15.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:15.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:15.788 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:15.820 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:15.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:15.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:15.922 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:15.957 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:15.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:15.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:16.058 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:16.108 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:16.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:16.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:16.210 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:16.245 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:16.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:16.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:16.328 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:16 smithi070 ceph-mon[107385]: pgmap v2869: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.4 GiB data, 11 GiB used, 347 GiB / 358 GiB avail; 0 B/s wr, 1.35k op/s; 9037/116055 objects degraded (7.787%); 19203/116055 objects misplaced (16.546%); 7.8 MiB/s, 151 objects/s recovering 2024-09-21T10:57:16.328 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:16 smithi070 ceph-mon[112139]: pgmap v2869: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.4 GiB data, 11 GiB used, 347 GiB / 358 GiB avail; 0 B/s wr, 1.35k op/s; 9037/116055 objects degraded (7.787%); 19203/116055 objects misplaced (16.546%); 7.8 MiB/s, 151 objects/s recovering 2024-09-21T10:57:16.347 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:16.374 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:16.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:16.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:16.475 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:16.510 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:16.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:16.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:16.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:16 smithi160 ceph-mon[101879]: pgmap v2869: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.4 GiB data, 11 GiB used, 347 GiB / 358 GiB avail; 0 B/s wr, 1.35k op/s; 9037/116055 objects degraded (7.787%); 19203/116055 objects misplaced (16.546%); 7.8 MiB/s, 151 objects/s recovering 2024-09-21T10:57:16.611 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:16.641 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:16.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:16.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:16.743 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:16.782 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:16.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:16.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:16.884 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:16.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:16.919 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:16.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:17.020 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:17.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:17.058 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:17.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:17.159 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:17.181 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:57:17.196 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:17.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:17.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:17.298 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:17.330 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:17.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:17.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:17.432 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:17.474 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:17.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:17.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:17.526 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:57:17.526 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:57:17.559 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:57:17.559 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:57:17.576 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:17.590 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:57:17.591 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:57:17.611 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:17.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:17.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:17.621 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:57:17.622 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:57:17.650 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:57:17.651 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:57:17.684 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:57:17.684 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:57:17.713 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:17.714 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:57:17.714 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:57:17.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:17.750 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:17.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:17.751 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:57:17.751 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:57:17.852 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:17.884 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:17.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:17.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:17.986 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:18.028 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:18.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:18.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:18.130 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:18.165 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:18.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:18.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:18.267 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:18.291 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:18.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:18.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:18.393 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:18.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:18.429 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:18.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:18.530 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:18.569 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:18.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:18.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:18 smithi160 ceph-mon[101879]: pgmap v2870: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.4 GiB data, 11 GiB used, 347 GiB / 358 GiB avail; 0 B/s wr, 1.35k op/s; 9037/116055 objects degraded (7.787%); 19203/116055 objects misplaced (16.546%); 7.8 MiB/s, 151 objects/s recovering 2024-09-21T10:57:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:18 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1999616628' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:57:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:18 smithi070 ceph-mon[107385]: pgmap v2870: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.4 GiB data, 11 GiB used, 347 GiB / 358 GiB avail; 0 B/s wr, 1.35k op/s; 9037/116055 objects degraded (7.787%); 19203/116055 objects misplaced (16.546%); 7.8 MiB/s, 151 objects/s recovering 2024-09-21T10:57:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:18 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1999616628' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:57:18.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:18 smithi070 ceph-mon[112139]: pgmap v2870: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.4 GiB data, 11 GiB used, 347 GiB / 358 GiB avail; 0 B/s wr, 1.35k op/s; 9037/116055 objects degraded (7.787%); 19203/116055 objects misplaced (16.546%); 7.8 MiB/s, 151 objects/s recovering 2024-09-21T10:57:18.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:18 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1999616628' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:57:18.671 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:18.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:18.707 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:18.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:18.808 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:18.842 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:18.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:18.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:18.944 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:18.979 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:18.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:18.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:19.081 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:19.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:19.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:19.206 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:19.238 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:19.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:19.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:19.339 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:19.360 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:19.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:19.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:19.462 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:19.496 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:19.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:19.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:19.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:19 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 9037/116055 objects degraded (7.787%), 5 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:57:19.598 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:19.626 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:19.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:19.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:19.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:19 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 9037/116055 objects degraded (7.787%), 5 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:57:19.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:19 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 9037/116055 objects degraded (7.787%), 5 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:57:19.728 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:19.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:19.766 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:19.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:19.867 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:19.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:19.894 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:19.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:19.995 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:20.030 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:20.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:20.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:20.132 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:20.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:20.166 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:20.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:20.267 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:20.304 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:20.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:20.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:20.405 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:20.440 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:20.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:20.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:20.543 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:20.576 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:20.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:20.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:20.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:20 smithi160 ceph-mon[101879]: pgmap v2871: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.1 GiB data, 10 GiB used, 347 GiB / 358 GiB avail; 0 B/s wr, 1.43k op/s; 8596/102327 objects degraded (8.401%); 15997/102327 objects misplaced (15.633%); 940 KiB/s, 40 objects/s recovering 2024-09-21T10:57:20.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:20 smithi070 ceph-mon[107385]: pgmap v2871: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.1 GiB data, 10 GiB used, 347 GiB / 358 GiB avail; 0 B/s wr, 1.43k op/s; 8596/102327 objects degraded (8.401%); 15997/102327 objects misplaced (15.633%); 940 KiB/s, 40 objects/s recovering 2024-09-21T10:57:20.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:20 smithi070 ceph-mon[112139]: pgmap v2871: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.1 GiB data, 10 GiB used, 347 GiB / 358 GiB avail; 0 B/s wr, 1.43k op/s; 8596/102327 objects degraded (8.401%); 15997/102327 objects misplaced (15.633%); 940 KiB/s, 40 objects/s recovering 2024-09-21T10:57:20.679 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:20.713 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:20.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:20.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:20.815 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:20.846 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:20.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:20.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:20.948 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:20.981 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:20.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:20.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:21.083 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:21.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:21.128 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:21.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:21.229 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:21.260 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:21.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:21.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:21.362 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:21.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:21.414 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:21.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:21.515 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:21.555 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:21.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:21.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:57:21.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:57:21.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:57:21.657 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:21.691 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:21.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:21.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:21.793 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:21.826 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:21.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:21.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:21.928 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:21.975 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:21.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:21.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:22.077 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:22.109 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:22.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:22.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:22.211 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:22.252 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:22.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:22.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:22.328 INFO:teuthology.orchestra.run.smithi070.stderr:no valid command found; 10 closest matches: 2024-09-21T10:57:22.328 INFO:teuthology.orchestra.run.smithi070.stderr:0 2024-09-21T10:57:22.328 INFO:teuthology.orchestra.run.smithi070.stderr:1 2024-09-21T10:57:22.328 INFO:teuthology.orchestra.run.smithi070.stderr:2 2024-09-21T10:57:22.328 INFO:teuthology.orchestra.run.smithi070.stderr:abort 2024-09-21T10:57:22.328 INFO:teuthology.orchestra.run.smithi070.stderr:assert 2024-09-21T10:57:22.329 INFO:teuthology.orchestra.run.smithi070.stderr:bench [] [] [] [] 2024-09-21T10:57:22.329 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:57:22.329 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs files list 2024-09-21T10:57:22.329 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs stats 2024-09-21T10:57:22.329 INFO:teuthology.orchestra.run.smithi070.stderr:bluestore allocator dump block 2024-09-21T10:57:22.329 INFO:teuthology.orchestra.run.smithi070.stderr:Error EINVAL: invalid command 2024-09-21T10:57:22.354 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:22.388 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:22.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:22.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:22.436 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:22 smithi070 ceph-mon[107385]: pgmap v2872: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.9 GiB data, 9.8 GiB used, 348 GiB / 358 GiB avail; 0 B/s wr, 1.42k op/s; 8430/94473 objects degraded (8.923%); 15561/94473 objects misplaced (16.471%); 0 B/s, 31 objects/s recovering 2024-09-21T10:57:22.436 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:22 smithi070 ceph-mon[112139]: pgmap v2872: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.9 GiB data, 9.8 GiB used, 348 GiB / 358 GiB avail; 0 B/s wr, 1.42k op/s; 8430/94473 objects degraded (8.923%); 15561/94473 objects misplaced (16.471%); 0 B/s, 31 objects/s recovering 2024-09-21T10:57:22.490 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:22.521 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:22.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:22.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:22.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:22 smithi160 ceph-mon[101879]: pgmap v2872: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.9 GiB data, 9.8 GiB used, 348 GiB / 358 GiB avail; 0 B/s wr, 1.42k op/s; 8430/94473 objects degraded (8.923%); 15561/94473 objects misplaced (16.471%); 0 B/s, 31 objects/s recovering 2024-09-21T10:57:22.624 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:22.659 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:22.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:22.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:22.761 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:22.799 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:22.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:22.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:22.901 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:22.940 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:22.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:22.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:23.042 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:23.085 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:23.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:23.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:23.187 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:23.189 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:57:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:57:23] "GET /metrics HTTP/1.1" 200 36642 "" "Prometheus/2.51.0" 2024-09-21T10:57:23.218 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:23.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:23.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:23.320 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:23.353 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:23.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:23.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:23.455 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:23.490 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:23.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:23.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:23.592 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:23.627 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:23.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:23.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:23.657 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:57:23.658 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:57:23.658 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-21T10:57:23.729 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:23.757 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:23.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:23.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:23.859 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:23.899 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:57:23.900 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T10:57:23.900 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T10:57:23.900 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T10:57:23.900 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.900 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:76003 55.8 55:10aa5eab:::benchmark_data_smithi070_294394_object76002:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1426)", 2024-09-21T10:57:23.900 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:49:43.855798+0000", 2024-09-21T10:57:23.900 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 460.03851526300002, 2024-09-21T10:57:23.900 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2879054860000001, 2024-09-21T10:57:23.900 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:57:23.900 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:57:23.900 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:57:23.901 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:57:23.901 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:57:23.901 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:57:23.901 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 76003 2024-09-21T10:57:23.901 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.901 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:57:23.901 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.901 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:57:23.901 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855798+0000", 2024-09-21T10:57:23.901 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.901 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.901 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.901 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:57:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855798+0000", 2024-09-21T10:57:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:57:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855800+0000", 2024-09-21T10:57:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7969999999999999e-06 2024-09-21T10:57:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:57:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855864+0000", 2024-09-21T10:57:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3825999999999995e-05 2024-09-21T10:57:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.903 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:57:23.903 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855881+0000", 2024-09-21T10:57:23.903 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6991999999999999e-05 2024-09-21T10:57:23.903 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.903 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.903 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:57:23.903 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855889+0000", 2024-09-21T10:57:23.903 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6910000000000003e-06 2024-09-21T10:57:23.903 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.903 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.903 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.903 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.856008+0000", 2024-09-21T10:57:23.903 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011862300000000001 2024-09-21T10:57:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:57:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.856021+0000", 2024-09-21T10:57:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3016e-05 2024-09-21T10:57:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.849652+0000", 2024-09-21T10:57:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99363043399999995 2024-09-21T10:57:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:57:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.849663+0000", 2024-09-21T10:57:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1222e-05 2024-09-21T10:57:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.114507+0000", 2024-09-21T10:57:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.264844214 2024-09-21T10:57:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:57:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.114727+0000", 2024-09-21T10:57:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00021980900000000001 2024-09-21T10:57:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.906 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:57:23.906 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.114900+0000", 2024-09-21T10:57:23.906 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017282199999999999 2024-09-21T10:57:23.906 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.906 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.906 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:57:23.906 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.123746+0000", 2024-09-21T10:57:23.906 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0088463229999999997 2024-09-21T10:57:23.906 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.906 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.906 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:57:23.906 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.123761+0000", 2024-09-21T10:57:23.906 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4737e-05 2024-09-21T10:57:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:57:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.140254+0000", 2024-09-21T10:57:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.016493628999999999 2024-09-21T10:57:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:57:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.140379+0000", 2024-09-21T10:57:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012441000000000001 2024-09-21T10:57:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:57:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.143704+0000", 2024-09-21T10:57:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0033249410000000001 2024-09-21T10:57:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:57:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:76029 55.8 55:192a45d5:::benchmark_data_smithi070_294394_object76028:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1427)", 2024-09-21T10:57:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:49:43.887515+0000", 2024-09-21T10:57:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 460.00679805700003, 2024-09-21T10:57:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2596707930000002, 2024-09-21T10:57:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:57:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:57:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:57:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:57:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:57:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:57:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 76029 2024-09-21T10:57:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:57:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:57:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887515+0000", 2024-09-21T10:57:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:57:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887515+0000", 2024-09-21T10:57:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:57:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887518+0000", 2024-09-21T10:57:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5490000000000001e-06 2024-09-21T10:57:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:57:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887543+0000", 2024-09-21T10:57:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5434000000000001e-05 2024-09-21T10:57:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:57:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887565+0000", 2024-09-21T10:57:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2099e-05 2024-09-21T10:57:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:57:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887575+0000", 2024-09-21T10:57:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7860000000000008e-06 2024-09-21T10:57:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887646+0000", 2024-09-21T10:57:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1037000000000004e-05 2024-09-21T10:57:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:57:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887658+0000", 2024-09-21T10:57:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1302e-05 2024-09-21T10:57:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.849687+0000", 2024-09-21T10:57:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.96202977000000001 2024-09-21T10:57:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:57:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.849695+0000", 2024-09-21T10:57:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4089999999999997e-06 2024-09-21T10:57:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.120438+0000", 2024-09-21T10:57:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2707431630000001 2024-09-21T10:57:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:57:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.120627+0000", 2024-09-21T10:57:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018910699999999999 2024-09-21T10:57:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.914 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:57:23.914 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.120803+0000", 2024-09-21T10:57:23.914 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017566500000000001 2024-09-21T10:57:23.914 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.914 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.914 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:57:23.914 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.127153+0000", 2024-09-21T10:57:23.914 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0063504479999999999 2024-09-21T10:57:23.914 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.914 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.914 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:57:23.914 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.127173+0000", 2024-09-21T10:57:23.915 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0123999999999999e-05 2024-09-21T10:57:23.915 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.915 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.915 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:57:23.915 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.143773+0000", 2024-09-21T10:57:23.915 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.016599635000000001 2024-09-21T10:57:23.915 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.915 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.915 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:57:23.915 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.143894+0000", 2024-09-21T10:57:23.915 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012071899999999999 2024-09-21T10:57:23.915 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.915 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.915 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:57:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.147186+0000", 2024-09-21T10:57:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0032925459999999999 2024-09-21T10:57:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:57:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:76034 55.8 55:1ffe6a40:::benchmark_data_smithi070_294394_object76033:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1427)", 2024-09-21T10:57:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:49:43.890819+0000", 2024-09-21T10:57:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 460.00349405999998, 2024-09-21T10:57:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2599814199999999, 2024-09-21T10:57:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:57:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:57:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:57:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:57:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:57:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:57:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 76034 2024-09-21T10:57:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:57:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:57:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.890819+0000", 2024-09-21T10:57:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:57:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.890819+0000", 2024-09-21T10:57:23.918 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.918 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.918 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.918 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:57:23.918 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.890822+0000", 2024-09-21T10:57:23.918 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5730000000000002e-06 2024-09-21T10:57:23.918 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.918 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.918 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:57:23.918 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.890851+0000", 2024-09-21T10:57:23.918 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.919e-05 2024-09-21T10:57:23.918 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.918 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.919 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:57:23.919 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.890873+0000", 2024-09-21T10:57:23.919 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1478999999999999e-05 2024-09-21T10:57:23.919 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.919 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.919 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:57:23.919 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.890884+0000", 2024-09-21T10:57:23.919 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0896e-05 2024-09-21T10:57:23.919 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.919 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.919 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.919 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.890961+0000", 2024-09-21T10:57:23.919 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7788999999999997e-05 2024-09-21T10:57:23.920 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.920 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.920 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:57:23.920 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.890973+0000", 2024-09-21T10:57:23.920 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2130000000000001e-05 2024-09-21T10:57:23.920 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.920 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.920 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.920 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.849716+0000", 2024-09-21T10:57:23.920 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.95874219199999999 2024-09-21T10:57:23.920 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.920 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.920 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:57:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.849723+0000", 2024-09-21T10:57:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2060000000000001e-06 2024-09-21T10:57:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.123858+0000", 2024-09-21T10:57:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.27413475 2024-09-21T10:57:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:57:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.124033+0000", 2024-09-21T10:57:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000175067 2024-09-21T10:57:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.922 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:57:23.922 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.124221+0000", 2024-09-21T10:57:23.922 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000188593 2024-09-21T10:57:23.922 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.922 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.922 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:57:23.922 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.127229+0000", 2024-09-21T10:57:23.922 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.003007895 2024-09-21T10:57:23.922 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.922 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.922 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:57:23.922 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.130344+0000", 2024-09-21T10:57:23.922 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0031151070000000002 2024-09-21T10:57:23.922 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.923 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.923 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:57:23.923 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.147244+0000", 2024-09-21T10:57:23.923 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.016899687 2024-09-21T10:57:23.923 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.923 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.923 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:57:23.923 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.147315+0000", 2024-09-21T10:57:23.923 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1049000000000004e-05 2024-09-21T10:57:23.923 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.923 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.923 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:57:23.923 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.150801+0000", 2024-09-21T10:57:23.924 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.003485817 2024-09-21T10:57:23.924 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.924 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:57:23.924 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.924 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.924 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.924 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:129295 55.1 e1446/1440 55:8efdf0b1:::benchmark_data_smithi070_294394_object23954:head v 1446'8143, mlcod=0'0)", 2024-09-21T10:57:23.924 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:22.707277+0000", 2024-09-21T10:57:23.924 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 361.187036181, 2024-09-21T10:57:23.924 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27558063199999999, 2024-09-21T10:57:23.924 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:57:23.924 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:57:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:57:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:57:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:57:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.707277+0000", 2024-09-21T10:57:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:57:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.707277+0000", 2024-09-21T10:57:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:57:23.926 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.707278+0000", 2024-09-21T10:57:23.926 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0299999999999997e-07 2024-09-21T10:57:23.926 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.926 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.926 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:57:23.926 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.707285+0000", 2024-09-21T10:57:23.926 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8850000000000001e-06 2024-09-21T10:57:23.926 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.926 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.926 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:57:23.926 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.707296+0000", 2024-09-21T10:57:23.926 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0993e-05 2024-09-21T10:57:23.926 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:57:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.707303+0000", 2024-09-21T10:57:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7120000000000003e-06 2024-09-21T10:57:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.945300+0000", 2024-09-21T10:57:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.23799742600000001 2024-09-21T10:57:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:57:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.945329+0000", 2024-09-21T10:57:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8379999999999999e-05 2024-09-21T10:57:23.928 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.928 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.928 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:57:23.928 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.982817+0000", 2024-09-21T10:57:23.928 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.037488847999999998 2024-09-21T10:57:23.929 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.929 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.929 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:57:23.929 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.982858+0000", 2024-09-21T10:57:23.929 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0485000000000003e-05 2024-09-21T10:57:23.929 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.929 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:57:23.929 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.929 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.929 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.929 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:129296 55.8 55:1cab724d:::benchmark_data_smithi070_294394_object23955:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:57:23.929 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:22.708169+0000", 2024-09-21T10:57:23.929 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 361.186144743, 2024-09-21T10:57:23.930 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.24355597700000001, 2024-09-21T10:57:23.930 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:57:23.930 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:57:23.930 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:57:23.930 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:57:23.930 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:57:23.930 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:57:23.930 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 129296 2024-09-21T10:57:23.930 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.930 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:57:23.930 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.930 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:57:23.930 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.708169+0000", 2024-09-21T10:57:23.931 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.931 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.931 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.931 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:57:23.931 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.708169+0000", 2024-09-21T10:57:23.931 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.931 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.931 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.931 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:57:23.931 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.708171+0000", 2024-09-21T10:57:23.931 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0150000000000002e-06 2024-09-21T10:57:23.931 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.931 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:57:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.708177+0000", 2024-09-21T10:57:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.852e-06 2024-09-21T10:57:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:57:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.708190+0000", 2024-09-21T10:57:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3148e-05 2024-09-21T10:57:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:57:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.708197+0000", 2024-09-21T10:57:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0040000000000002e-06 2024-09-21T10:57:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.933 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.933 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.933 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.708739+0000", 2024-09-21T10:57:23.933 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00054260300000000001 2024-09-21T10:57:23.933 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.933 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.933 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:57:23.933 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.708879+0000", 2024-09-21T10:57:23.933 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013970700000000001 2024-09-21T10:57:23.933 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.933 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.933 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:57:23.933 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.709019+0000", 2024-09-21T10:57:23.934 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013960900000000001 2024-09-21T10:57:23.934 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.934 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.934 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:57:23.934 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.713167+0000", 2024-09-21T10:57:23.934 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0041479009999999998 2024-09-21T10:57:23.934 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.934 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.934 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:57:23.934 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.714761+0000", 2024-09-21T10:57:23.934 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001594729 2024-09-21T10:57:23.934 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.934 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:57:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.949456+0000", 2024-09-21T10:57:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.23469445 2024-09-21T10:57:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:57:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.949604+0000", 2024-09-21T10:57:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000147894 2024-09-21T10:57:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:57:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.951725+0000", 2024-09-21T10:57:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0021210650000000001 2024-09-21T10:57:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:57:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:129298 55.5 55:aae0d80d:::benchmark_data_smithi070_294394_object23957:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:57:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:22.714780+0000", 2024-09-21T10:57:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 361.17953368000002, 2024-09-21T10:57:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.23622679199999999, 2024-09-21T10:57:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:57:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:57:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:57:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:57:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:57:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:57:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 129298 2024-09-21T10:57:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:57:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:57:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.714780+0000", 2024-09-21T10:57:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:57:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.714780+0000", 2024-09-21T10:57:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:57:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.714782+0000", 2024-09-21T10:57:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9489999999999999e-06 2024-09-21T10:57:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:57:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.714788+0000", 2024-09-21T10:57:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1210000000000003e-06 2024-09-21T10:57:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:57:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.714800+0000", 2024-09-21T10:57:23.939 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.219e-05 2024-09-21T10:57:23.939 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.939 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.939 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:57:23.939 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.714808+0000", 2024-09-21T10:57:23.939 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7749999999999993e-06 2024-09-21T10:57:23.939 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.939 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.939 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.939 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.714867+0000", 2024-09-21T10:57:23.939 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8956999999999998e-05 2024-09-21T10:57:23.939 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.939 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:57:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.714987+0000", 2024-09-21T10:57:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000120728 2024-09-21T10:57:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,3,6", 2024-09-21T10:57:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.715375+0000", 2024-09-21T10:57:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000387814 2024-09-21T10:57:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:57:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.947741+0000", 2024-09-21T10:57:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.232365505 2024-09-21T10:57:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:57:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.947799+0000", 2024-09-21T10:57:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7720999999999998e-05 2024-09-21T10:57:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:57:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.947853+0000", 2024-09-21T10:57:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4453000000000002e-05 2024-09-21T10:57:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:57:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.950846+0000", 2024-09-21T10:57:23.942 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0029934509999999998 2024-09-21T10:57:23.942 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.942 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.942 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:57:23.942 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.950899+0000", 2024-09-21T10:57:23.942 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2769000000000002e-05 2024-09-21T10:57:23.942 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.942 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.942 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:57:23.942 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.951007+0000", 2024-09-21T10:57:23.942 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000107359 2024-09-21T10:57:23.942 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.942 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:57:23.943 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.943 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.943 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.943 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "MOSDPGPush(55.2 1446/1434 [PushOp(55:42275384:::benchmark_data_smithi070_294394_object23949:head, version: 1416'1519, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(55:42275384:::benchmark_data_smithi070_294394_object23949:head@1416'1519, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T10:57:23.943 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:22.715198+0000", 2024-09-21T10:57:23.943 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 361.17911579000003, 2024-09-21T10:57:23.943 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.233190694, 2024-09-21T10:57:23.943 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:57:23.943 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:57:23.943 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "started", 2024-09-21T10:57:23.943 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:57:23.943 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.943 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:57:23.944 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.715198+0000", 2024-09-21T10:57:23.944 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.944 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.944 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.944 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:57:23.944 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.715198+0000", 2024-09-21T10:57:23.944 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.944 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.944 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.944 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:57:23.944 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.715198+0000", 2024-09-21T10:57:23.944 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6400000000000002e-07 2024-09-21T10:57:23.944 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.944 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.945 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:57:23.945 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.715228+0000", 2024-09-21T10:57:23.945 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9607e-05 2024-09-21T10:57:23.945 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.945 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.945 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:57:23.945 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.715247+0000", 2024-09-21T10:57:23.945 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8816000000000001e-05 2024-09-21T10:57:23.945 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.945 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.945 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:57:23.945 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.715262+0000", 2024-09-21T10:57:23.945 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5613000000000001e-05 2024-09-21T10:57:23.945 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.715366+0000", 2024-09-21T10:57:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000103769 2024-09-21T10:57:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:57:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.715386+0000", 2024-09-21T10:57:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0136000000000002e-05 2024-09-21T10:57:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:57:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.948388+0000", 2024-09-21T10:57:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.233002089 2024-09-21T10:57:23.947 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.947 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:57:23.947 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.947 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.947 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.947 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "MOSDPGPushReply(55.8 1446/1428 [PushReplyOp(55:1a4fe3ae:::benchmark_data_smithi070_294394_object37264:head)])", 2024-09-21T10:57:23.947 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:22.716411+0000", 2024-09-21T10:57:23.947 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 361.17790278500001, 2024-09-21T10:57:23.947 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.23640767900000001, 2024-09-21T10:57:23.947 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:57:23.947 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:57:23.947 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:57:23.947 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:57:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:57:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.716411+0000", 2024-09-21T10:57:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:57:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.716411+0000", 2024-09-21T10:57:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:57:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.716411+0000", 2024-09-21T10:57:23.949 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.85e-07 2024-09-21T10:57:23.949 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.949 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.949 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:57:23.949 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.716416+0000", 2024-09-21T10:57:23.949 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8189999999999999e-06 2024-09-21T10:57:23.949 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.949 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.949 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:57:23.949 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.945276+0000", 2024-09-21T10:57:23.949 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.22885976299999999 2024-09-21T10:57:23.949 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.949 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.950 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:57:23.950 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.945289+0000", 2024-09-21T10:57:23.950 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3278000000000001e-05 2024-09-21T10:57:23.950 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.950 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.950 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.950 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.945376+0000", 2024-09-21T10:57:23.950 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6266000000000004e-05 2024-09-21T10:57:23.950 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.950 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.950 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:57:23.950 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.952818+0000", 2024-09-21T10:57:23.950 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0074427679999999998 2024-09-21T10:57:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:57:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:129300 55.5 55:adccdd22:::benchmark_data_smithi070_294394_object23959:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:57:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:22.716445+0000", 2024-09-21T10:57:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 361.17786838299997, 2024-09-21T10:57:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.24455037499999999, 2024-09-21T10:57:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:57:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:57:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:57:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:57:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:57:23.952 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:57:23.952 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 129300 2024-09-21T10:57:23.952 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.952 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:57:23.952 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.952 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:57:23.952 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.716445+0000", 2024-09-21T10:57:23.952 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.952 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.952 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.952 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:57:23.952 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.716445+0000", 2024-09-21T10:57:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:57:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.716446+0000", 2024-09-21T10:57:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.322e-06 2024-09-21T10:57:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:57:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.716451+0000", 2024-09-21T10:57:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2429999999999999e-06 2024-09-21T10:57:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:57:23.954 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.945269+0000", 2024-09-21T10:57:23.954 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.22881818800000001 2024-09-21T10:57:23.954 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.954 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.954 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:57:23.954 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.945287+0000", 2024-09-21T10:57:23.954 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8023999999999999e-05 2024-09-21T10:57:23.954 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.954 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.954 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.954 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.951033+0000", 2024-09-21T10:57:23.954 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0057462950000000002 2024-09-21T10:57:23.954 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:57:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.951142+0000", 2024-09-21T10:57:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000108631 2024-09-21T10:57:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,3,6", 2024-09-21T10:57:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.951254+0000", 2024-09-21T10:57:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000112182 2024-09-21T10:57:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:57:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.960562+0000", 2024-09-21T10:57:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0093075840000000007 2024-09-21T10:57:23.956 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.956 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.956 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:57:23.956 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.960608+0000", 2024-09-21T10:57:23.956 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6307999999999998e-05 2024-09-21T10:57:23.956 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.956 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.956 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:57:23.956 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.960650+0000", 2024-09-21T10:57:23.956 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.1752000000000003e-05 2024-09-21T10:57:23.956 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.956 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:23.957 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:23.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:23.958 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:57:23.958 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.960827+0000", 2024-09-21T10:57:23.958 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017739700000000001 2024-09-21T10:57:23.958 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.958 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.958 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:57:23.959 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.960880+0000", 2024-09-21T10:57:23.959 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3146000000000002e-05 2024-09-21T10:57:23.959 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.959 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.959 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:57:23.959 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.960995+0000", 2024-09-21T10:57:23.959 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000115303 2024-09-21T10:57:23.959 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.959 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:57:23.959 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.959 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.959 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.959 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop_reply(client.35670.0:129298 55.5 e1446/1434 ondisk, result = 0)", 2024-09-21T10:57:23.960 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:22.718413+0000", 2024-09-21T10:57:23.960 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 361.17590090499999, 2024-09-21T10:57:23.960 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.229390868, 2024-09-21T10:57:23.960 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:57:23.960 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:57:23.960 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "started", 2024-09-21T10:57:23.960 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:57:23.960 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.960 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:57:23.960 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.718413+0000", 2024-09-21T10:57:23.960 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.960 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.960 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:57:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.718413+0000", 2024-09-21T10:57:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:57:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.718414+0000", 2024-09-21T10:57:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.46e-06 2024-09-21T10:57:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:57:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.718421+0000", 2024-09-21T10:57:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4850000000000003e-06 2024-09-21T10:57:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.962 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.962 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:57:23.962 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.945399+0000", 2024-09-21T10:57:23.962 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.226977014 2024-09-21T10:57:23.962 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.962 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.962 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:57:23.962 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.945406+0000", 2024-09-21T10:57:23.962 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0609999999999998e-06 2024-09-21T10:57:23.962 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.962 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.962 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.962 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.947791+0000", 2024-09-21T10:57:23.963 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0023850149999999999 2024-09-21T10:57:23.963 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.963 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.963 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:57:23.963 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.947795+0000", 2024-09-21T10:57:23.963 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7110000000000003e-06 2024-09-21T10:57:23.963 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.963 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.963 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:57:23.963 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.947803+0000", 2024-09-21T10:57:23.963 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1219999999999995e-06 2024-09-21T10:57:23.963 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.963 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:57:23.963 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:130858 55.8 55:178fc653:::benchmark_data_smithi070_294394_object25517:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:57:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:26.393677+0000", 2024-09-21T10:57:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 357.50063601699998, 2024-09-21T10:57:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0198451989999999, 2024-09-21T10:57:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:57:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:57:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:57:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:57:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:57:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:57:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 130858 2024-09-21T10:57:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:57:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:57:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.393677+0000", 2024-09-21T10:57:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:57:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.393677+0000", 2024-09-21T10:57:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.966 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:57:23.966 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.393679+0000", 2024-09-21T10:57:23.966 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7349999999999999e-06 2024-09-21T10:57:23.966 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.966 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.966 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:57:23.966 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.393686+0000", 2024-09-21T10:57:23.966 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4930000000000003e-06 2024-09-21T10:57:23.966 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.966 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.966 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:57:23.966 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.393701+0000", 2024-09-21T10:57:23.966 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5466999999999999e-05 2024-09-21T10:57:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:57:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.393710+0000", 2024-09-21T10:57:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.8789999999999992e-06 2024-09-21T10:57:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.395855+0000", 2024-09-21T10:57:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0021445420000000001 2024-09-21T10:57:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:57:23.968 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.395988+0000", 2024-09-21T10:57:23.968 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013296900000000001 2024-09-21T10:57:23.968 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.968 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.968 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:57:23.968 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.396195+0000", 2024-09-21T10:57:23.968 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000207035 2024-09-21T10:57:23.968 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.968 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.968 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.968 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.445846+0000", 2024-09-21T10:57:23.968 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.049651140000000003 2024-09-21T10:57:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:57:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.445859+0000", 2024-09-21T10:57:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3772e-05 2024-09-21T10:57:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.413508+0000", 2024-09-21T10:57:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.96764826299999995 2024-09-21T10:57:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:57:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.413523+0000", 2024-09-21T10:57:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4904e-05 2024-09-21T10:57:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:57:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:130866 55.8 55.11526d38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:57:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:26.400243+0000", 2024-09-21T10:57:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 357.49407006600001, 2024-09-21T10:57:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.013316393, 2024-09-21T10:57:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:57:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:57:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:57:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:57:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:57:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:57:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 130866 2024-09-21T10:57:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:57:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:57:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.400243+0000", 2024-09-21T10:57:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:57:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.400243+0000", 2024-09-21T10:57:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:57:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.400245+0000", 2024-09-21T10:57:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0829999999999998e-06 2024-09-21T10:57:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:57:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.400251+0000", 2024-09-21T10:57:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3410000000000002e-06 2024-09-21T10:57:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:57:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.400261+0000", 2024-09-21T10:57:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7210000000000006e-06 2024-09-21T10:57:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:57:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.400267+0000", 2024-09-21T10:57:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8560000000000001e-06 2024-09-21T10:57:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.446742+0000", 2024-09-21T10:57:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.046474840000000003 2024-09-21T10:57:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.974 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:57:23.974 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.446887+0000", 2024-09-21T10:57:23.974 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014484099999999999 2024-09-21T10:57:23.974 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.974 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.974 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.974 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.413550+0000", 2024-09-21T10:57:23.974 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.96666295099999999 2024-09-21T10:57:23.974 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.974 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.974 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:57:23.974 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.413560+0000", 2024-09-21T10:57:23.974 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7599999999999997e-06 2024-09-21T10:57:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:57:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:130872 55.8 55.862244f8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1447)", 2024-09-21T10:57:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:26.466075+0000", 2024-09-21T10:57:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 357.42823873700002, 2024-09-21T10:57:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.94751783499999997, 2024-09-21T10:57:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:57:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:57:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:57:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:57:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:57:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:57:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 130872 2024-09-21T10:57:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:57:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:57:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.466075+0000", 2024-09-21T10:57:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:57:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.466075+0000", 2024-09-21T10:57:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.977 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.977 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.977 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:57:23.977 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.466077+0000", 2024-09-21T10:57:23.977 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9479999999999998e-06 2024-09-21T10:57:23.977 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.977 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.977 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:57:23.977 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.466085+0000", 2024-09-21T10:57:23.977 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3850000000000002e-06 2024-09-21T10:57:23.977 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.977 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.977 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:57:23.977 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.466104+0000", 2024-09-21T10:57:23.977 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9218000000000001e-05 2024-09-21T10:57:23.978 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.978 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.978 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:57:23.978 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.466115+0000", 2024-09-21T10:57:23.978 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0271e-05 2024-09-21T10:57:23.978 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.978 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.978 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.978 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.466576+0000", 2024-09-21T10:57:23.978 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00046172299999999998 2024-09-21T10:57:23.978 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.978 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.978 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:57:23.979 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.466585+0000", 2024-09-21T10:57:23.979 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7509999999999997e-06 2024-09-21T10:57:23.979 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.979 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.979 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.979 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.413582+0000", 2024-09-21T10:57:23.979 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.946997333 2024-09-21T10:57:23.979 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.979 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.979 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:57:23.979 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.413593+0000", 2024-09-21T10:57:23.979 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0206e-05 2024-09-21T10:57:23.979 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.979 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:57:23.980 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.980 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.980 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.980 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:130874 55.5 55.1ccdde55 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1447)", 2024-09-21T10:57:23.980 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:26.472197+0000", 2024-09-21T10:57:23.980 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 357.422116007, 2024-09-21T10:57:23.980 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.94240631200000002, 2024-09-21T10:57:23.980 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:57:23.980 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:57:23.980 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:57:23.980 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:57:23.980 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:57:23.980 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:57:23.980 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 130874 2024-09-21T10:57:23.981 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.981 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:57:23.981 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.981 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:57:23.981 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.472197+0000", 2024-09-21T10:57:23.981 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.981 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.981 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.981 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:57:23.981 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.472197+0000", 2024-09-21T10:57:23.981 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.981 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.981 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.982 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:57:23.982 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.472200+0000", 2024-09-21T10:57:23.982 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3329999999999999e-06 2024-09-21T10:57:23.982 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.982 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.982 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:57:23.982 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.472209+0000", 2024-09-21T10:57:23.982 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.884e-06 2024-09-21T10:57:23.982 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.982 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.982 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:57:23.982 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.472231+0000", 2024-09-21T10:57:23.982 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2211000000000001e-05 2024-09-21T10:57:23.983 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.983 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.983 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:57:23.983 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.472242+0000", 2024-09-21T10:57:23.983 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1233000000000001e-05 2024-09-21T10:57:23.983 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.983 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.983 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.983 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.472325+0000", 2024-09-21T10:57:23.983 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3275000000000005e-05 2024-09-21T10:57:23.983 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.983 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.983 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:57:23.983 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.472338+0000", 2024-09-21T10:57:23.984 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2302e-05 2024-09-21T10:57:23.984 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.984 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.984 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.984 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.414595+0000", 2024-09-21T10:57:23.984 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.942257601 2024-09-21T10:57:23.984 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.984 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.984 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:57:23.984 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.414604+0000", 2024-09-21T10:57:23.984 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.473e-06 2024-09-21T10:57:23.984 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.984 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:57:23.984 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.985 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.985 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.985 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:156258 55.c e1464/1448 55:39292414:::benchmark_data_smithi070_294394_object50917:head v 1464'9744, mlcod=0'0)", 2024-09-21T10:57:23.985 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.338132+0000", 2024-09-21T10:57:23.985 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 299.55618184100001, 2024-09-21T10:57:23.985 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.15064587099999999, 2024-09-21T10:57:23.985 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:57:23.985 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:57:23.985 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:57:23.985 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:57:23.985 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.985 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:57:23.985 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.338132+0000", 2024-09-21T10:57:23.985 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.986 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.986 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.986 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:57:23.986 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.338132+0000", 2024-09-21T10:57:23.986 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.986 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.986 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.986 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:57:23.986 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.338133+0000", 2024-09-21T10:57:23.986 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7900000000000007e-07 2024-09-21T10:57:23.986 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.986 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.986 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:57:23.987 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.338140+0000", 2024-09-21T10:57:23.987 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1740000000000003e-06 2024-09-21T10:57:23.987 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.987 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.987 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:57:23.987 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.338152+0000", 2024-09-21T10:57:23.987 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2215000000000001e-05 2024-09-21T10:57:23.987 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.987 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.987 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:57:23.987 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.338159+0000", 2024-09-21T10:57:23.987 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3520000000000001e-06 2024-09-21T10:57:23.987 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.987 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.988 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.988 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.338240+0000", 2024-09-21T10:57:23.988 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1086000000000003e-05 2024-09-21T10:57:23.988 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.988 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.988 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:57:23.988 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.338267+0000", 2024-09-21T10:57:23.988 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6194999999999999e-05 2024-09-21T10:57:23.988 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.988 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.988 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:57:23.988 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.488714+0000", 2024-09-21T10:57:23.988 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.15044706499999999 2024-09-21T10:57:23.988 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.989 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.989 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:57:23.989 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.488777+0000", 2024-09-21T10:57:23.989 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3805000000000003e-05 2024-09-21T10:57:23.989 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.989 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:57:23.989 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.989 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.989 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.989 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:156268 55.f e1464/1448 55:f53808b0:::benchmark_data_smithi070_294394_object50927:head v 1464'9749, mlcod=0'0)", 2024-09-21T10:57:23.989 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.340846+0000", 2024-09-21T10:57:23.989 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 299.55346792099999, 2024-09-21T10:57:23.989 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.158773426, 2024-09-21T10:57:23.989 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:57:23.990 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:57:23.990 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:57:23.990 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:57:23.990 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.990 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:57:23.990 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.340846+0000", 2024-09-21T10:57:23.990 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.990 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.990 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.990 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:57:23.990 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.340846+0000", 2024-09-21T10:57:23.990 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.991 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.991 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.991 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:57:23.991 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.340847+0000", 2024-09-21T10:57:23.991 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3650000000000001e-06 2024-09-21T10:57:23.991 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.991 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.991 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:57:23.991 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.340855+0000", 2024-09-21T10:57:23.991 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7100000000000007e-06 2024-09-21T10:57:23.991 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.991 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.991 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:57:23.991 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.340868+0000", 2024-09-21T10:57:23.992 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3312000000000001e-05 2024-09-21T10:57:23.992 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.992 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.992 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:57:23.992 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.340876+0000", 2024-09-21T10:57:23.992 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5320000000000006e-06 2024-09-21T10:57:23.992 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.992 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.992 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.992 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.478844+0000", 2024-09-21T10:57:23.992 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13796760599999999 2024-09-21T10:57:23.992 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.993 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.993 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:57:23.993 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.478891+0000", 2024-09-21T10:57:23.993 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6912000000000001e-05 2024-09-21T10:57:23.993 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.993 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.993 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:57:23.993 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.499553+0000", 2024-09-21T10:57:23.993 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.020662064000000001 2024-09-21T10:57:23.993 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.993 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.993 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:57:23.993 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.499619+0000", 2024-09-21T10:57:23.994 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5925000000000003e-05 2024-09-21T10:57:23.994 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.994 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:57:23.994 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:23.994 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.994 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.994 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:156262 55.3 e1464/1448)", 2024-09-21T10:57:23.995 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.343780+0000", 2024-09-21T10:57:23.995 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 299.55053313000002, 2024-09-21T10:57:23.995 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13981411199999999, 2024-09-21T10:57:23.995 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:57:23.996 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:57:23.996 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:57:23.996 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:57:23.996 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.996 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:57:23.997 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.343780+0000", 2024-09-21T10:57:23.997 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.997 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.997 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.997 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:57:23.997 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.343780+0000", 2024-09-21T10:57:23.997 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:23.997 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.997 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.998 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:57:23.998 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.343782+0000", 2024-09-21T10:57:23.998 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.28e-06 2024-09-21T10:57:23.998 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.998 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.998 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:57:23.998 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.343790+0000", 2024-09-21T10:57:23.998 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9440000000000005e-06 2024-09-21T10:57:23.998 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.998 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.998 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:57:23.998 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.475488+0000", 2024-09-21T10:57:23.998 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13169877499999999 2024-09-21T10:57:23.999 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.999 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.999 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:57:23.999 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.475507+0000", 2024-09-21T10:57:23.999 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8768000000000001e-05 2024-09-21T10:57:23.999 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.999 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.999 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:23.999 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.483581+0000", 2024-09-21T10:57:23.999 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0080743680000000002 2024-09-21T10:57:23.999 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:23.999 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:23.999 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:57:23.999 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.483594+0000", 2024-09-21T10:57:24.000 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2977e-05 2024-09-21T10:57:24.000 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:24.000 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:57:24.000 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:24.000 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.000 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.000 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:156269 55.f e1464/1448 55:fb867c57:::benchmark_data_smithi070_294394_object50928:head v 1464'9750, mlcod=0'0)", 2024-09-21T10:57:24.000 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.344388+0000", 2024-09-21T10:57:24.000 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 299.549925804, 2024-09-21T10:57:24.000 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.156159983, 2024-09-21T10:57:24.000 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:57:24.000 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:57:24.000 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:57:24.000 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:57:24.000 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.001 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:57:24.001 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.344388+0000", 2024-09-21T10:57:24.001 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:24.001 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.001 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.001 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:57:24.001 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.344388+0000", 2024-09-21T10:57:24.001 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:24.001 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.001 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.001 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:57:24.001 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.344388+0000", 2024-09-21T10:57:24.001 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9999999999999996e-07 2024-09-21T10:57:24.001 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.002 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.002 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:57:24.002 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.344395+0000", 2024-09-21T10:57:24.002 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7240000000000003e-06 2024-09-21T10:57:24.002 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.002 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.002 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:57:24.002 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.473817+0000", 2024-09-21T10:57:24.002 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.129422075 2024-09-21T10:57:24.002 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.002 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.002 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:57:24.002 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.473834+0000", 2024-09-21T10:57:24.002 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6381000000000001e-05 2024-09-21T10:57:24.003 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.003 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.003 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:24.003 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.494817+0000", 2024-09-21T10:57:24.003 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.020983841999999999 2024-09-21T10:57:24.003 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.003 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.003 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:57:24.003 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.494841+0000", 2024-09-21T10:57:24.003 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3091999999999998e-05 2024-09-21T10:57:24.003 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.003 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.003 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:57:24.003 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.500504+0000", 2024-09-21T10:57:24.004 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0056635160000000004 2024-09-21T10:57:24.004 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.004 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.004 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:57:24.004 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.500548+0000", 2024-09-21T10:57:24.004 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3553000000000001e-05 2024-09-21T10:57:24.004 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:24.005 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:57:24.005 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:24.005 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.005 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.005 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:195880 55.c e1474/1465 55:341b656a:::benchmark_data_smithi070_294394_object90539:head v 1474'12198, mlcod=0'0)", 2024-09-21T10:57:24.005 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:53:26.307538+0000", 2024-09-21T10:57:24.005 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 237.58677562700001, 2024-09-21T10:57:24.005 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.122286966, 2024-09-21T10:57:24.005 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:57:24.005 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:57:24.005 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:57:24.005 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:57:24.005 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.005 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:57:24.006 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.307538+0000", 2024-09-21T10:57:24.006 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:24.006 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.006 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.006 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:57:24.006 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.307538+0000", 2024-09-21T10:57:24.006 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:24.006 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.006 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.006 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:57:24.006 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.307538+0000", 2024-09-21T10:57:24.006 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8700000000000001e-07 2024-09-21T10:57:24.006 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.007 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.007 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:57:24.007 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.307542+0000", 2024-09-21T10:57:24.007 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0149999999999997e-06 2024-09-21T10:57:24.007 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.007 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.007 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:57:24.007 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.307550+0000", 2024-09-21T10:57:24.007 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4889999999999998e-06 2024-09-21T10:57:24.007 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.007 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.007 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:57:24.007 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.307555+0000", 2024-09-21T10:57:24.007 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4909999999999996e-06 2024-09-21T10:57:24.008 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.008 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.008 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:24.008 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.314879+0000", 2024-09-21T10:57:24.008 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0073239250000000002 2024-09-21T10:57:24.008 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.008 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.008 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:57:24.008 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.314904+0000", 2024-09-21T10:57:24.008 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4516999999999999e-05 2024-09-21T10:57:24.008 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.008 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.008 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:57:24.008 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.429774+0000", 2024-09-21T10:57:24.009 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.114870376 2024-09-21T10:57:24.009 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.009 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.009 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:57:24.009 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.429825+0000", 2024-09-21T10:57:24.009 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0766000000000001e-05 2024-09-21T10:57:24.009 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:24.009 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:57:24.009 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:24.009 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.009 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.009 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:195894 55.c e1474/1465 55:3dc3da2f:::benchmark_data_smithi070_294394_object90553:head v 1474'12199, mlcod=0'0)", 2024-09-21T10:57:24.009 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:53:26.309020+0000", 2024-09-21T10:57:24.009 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 237.585293568, 2024-09-21T10:57:24.010 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.146994024, 2024-09-21T10:57:24.010 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:57:24.010 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:57:24.010 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:57:24.010 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:57:24.010 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.010 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:57:24.010 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.309020+0000", 2024-09-21T10:57:24.010 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:24.010 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.010 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.010 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:57:24.010 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.309020+0000", 2024-09-21T10:57:24.010 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:57:24.011 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.011 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.011 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:57:24.011 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.309021+0000", 2024-09-21T10:57:24.011 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9199999999999999e-07 2024-09-21T10:57:24.011 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.011 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.011 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:57:24.011 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.309028+0000", 2024-09-21T10:57:24.011 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.844e-06 2024-09-21T10:57:24.011 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.011 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.011 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:57:24.011 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.309040+0000", 2024-09-21T10:57:24.012 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2e-05 2024-09-21T10:57:24.012 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.012 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.012 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:57:24.012 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.309048+0000", 2024-09-21T10:57:24.012 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9619999999999993e-06 2024-09-21T10:57:24.012 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.012 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.012 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:57:24.012 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.443698+0000", 2024-09-21T10:57:24.012 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13465039300000001 2024-09-21T10:57:24.012 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.012 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.012 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:57:24.013 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.443727+0000", 2024-09-21T10:57:24.013 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9249000000000001e-05 2024-09-21T10:57:24.013 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.013 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.013 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:57:24.013 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.455970+0000", 2024-09-21T10:57:24.013 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012242602999999999 2024-09-21T10:57:24.013 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:57:24.013 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:57:24.013 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:57:24.013 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.456014+0000", 2024-09-21T10:57:24.013 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3980999999999998e-05 2024-09-21T10:57:24.013 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:24.013 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:57:24.013 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:24.014 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:57:24.014 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:57:24.014 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:57:24.058 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:24.091 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:24.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:24.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:24.192 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:24.223 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:24.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:24.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:24.325 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:24.364 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:24.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:24.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:24.466 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:24.488 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:24.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:24.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:24.590 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:24 smithi160 ceph-mon[101879]: pgmap v2873: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.7 GiB data, 9.1 GiB used, 349 GiB / 358 GiB avail; 0 B/s wr, 1.41k op/s; 8012/81522 objects degraded (9.828%); 12526/81522 objects misplaced (15.365%); 0 B/s, 31 objects/s recovering 2024-09-21T10:57:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:24 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 8012/81522 objects degraded (9.828%), 5 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:57:24.620 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:24.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:24.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:24.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:24 smithi070 ceph-mon[107385]: pgmap v2873: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.7 GiB data, 9.1 GiB used, 349 GiB / 358 GiB avail; 0 B/s wr, 1.41k op/s; 8012/81522 objects degraded (9.828%); 12526/81522 objects misplaced (15.365%); 0 B/s, 31 objects/s recovering 2024-09-21T10:57:24.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:24 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 8012/81522 objects degraded (9.828%), 5 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:57:24.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:24 smithi070 ceph-mon[112139]: pgmap v2873: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.7 GiB data, 9.1 GiB used, 349 GiB / 358 GiB avail; 0 B/s wr, 1.41k op/s; 8012/81522 objects degraded (9.828%); 12526/81522 objects misplaced (15.365%); 0 B/s, 31 objects/s recovering 2024-09-21T10:57:24.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:24 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 8012/81522 objects degraded (9.828%), 5 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:57:24.722 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:24.739 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:57:24.754 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:24.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:24.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:24.856 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:24.887 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:24.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:24.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:24.978 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:57:24.989 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:25.083 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:25.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:25.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:25.185 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:25.200 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:57:25.256 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:57:25.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:25.261 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:25.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:25.362 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:25.449 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:25.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:25.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:25.551 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:25.578 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:25.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:25.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:25.680 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:25.727 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:25.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:25.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:25.829 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:25.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:25.869 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:25.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:25.971 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:26.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:26.017 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:26.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:26.118 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:26.155 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:26.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:26.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:26.257 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:26.293 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:26.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:26.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:26.395 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:26.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:26.435 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:26.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:26.536 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:26.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:26 smithi070 ceph-mon[107385]: pgmap v2874: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.5 GiB data, 8.5 GiB used, 349 GiB / 358 GiB avail; 0 B/s wr, 1.38k op/s; 7701/73740 objects degraded (10.443%); 12219/73740 objects misplaced (16.570%); 0 B/s, 31 objects/s recovering 2024-09-21T10:57:26.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:26 smithi070 ceph-mon[112139]: pgmap v2874: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.5 GiB data, 8.5 GiB used, 349 GiB / 358 GiB avail; 0 B/s wr, 1.38k op/s; 7701/73740 objects degraded (10.443%); 12219/73740 objects misplaced (16.570%); 0 B/s, 31 objects/s recovering 2024-09-21T10:57:26.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:26.569 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:26.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:26.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:26 smithi160 ceph-mon[101879]: pgmap v2874: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.5 GiB data, 8.5 GiB used, 349 GiB / 358 GiB avail; 0 B/s wr, 1.38k op/s; 7701/73740 objects degraded (10.443%); 12219/73740 objects misplaced (16.570%); 0 B/s, 31 objects/s recovering 2024-09-21T10:57:26.670 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:26.709 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:26.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:26.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:26.811 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:26.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:26.851 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:26.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:26.952 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:26.982 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:26.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:26.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:27.084 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:27.127 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:27.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:27.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:27.229 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:27.263 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:27.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:27.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:27.364 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:27.408 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:27.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:27.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:27.510 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:27.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:27.552 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:27.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:27.653 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:27.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:27.689 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:27.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:27.790 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:27.811 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:27.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:27.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:27.912 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:27.950 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:27.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:27.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:28.051 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:28.084 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:28.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:28.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:28.186 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:28.227 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:28.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:28.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:28.329 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:28.370 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:28.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:28.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:28.472 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:28.500 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:28 smithi070 ceph-mon[107385]: pgmap v2875: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.5 GiB data, 8.5 GiB used, 349 GiB / 358 GiB avail; 0 B/s wr, 1.17k op/s; 7701/73740 objects degraded (10.443%); 12219/73740 objects misplaced (16.570%); 0 B/s, 28 objects/s recovering 2024-09-21T10:57:28.500 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:28 smithi070 ceph-mon[112139]: pgmap v2875: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.5 GiB data, 8.5 GiB used, 349 GiB / 358 GiB avail; 0 B/s wr, 1.17k op/s; 7701/73740 objects degraded (10.443%); 12219/73740 objects misplaced (16.570%); 0 B/s, 28 objects/s recovering 2024-09-21T10:57:28.509 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:28.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:28.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:28.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:28 smithi160 ceph-mon[101879]: pgmap v2875: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.5 GiB data, 8.5 GiB used, 349 GiB / 358 GiB avail; 0 B/s wr, 1.17k op/s; 7701/73740 objects degraded (10.443%); 12219/73740 objects misplaced (16.570%); 0 B/s, 28 objects/s recovering 2024-09-21T10:57:28.611 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:28.631 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:28.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:28.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:28.733 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:28.765 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:28.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:28.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:28.867 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:28.915 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:28.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:28.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:29.017 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:29.048 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:29.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:29.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:29.150 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:29.188 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:29.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:29.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:29.255 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T10:57:29.290 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:29.320 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:29.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:29.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:29.421 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:29.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:29.465 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:29.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:29.506 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:29 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 7701/73740 objects degraded (10.443%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:57:29.506 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:29 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1710068671' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:57:29.507 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:29 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:57:29.507 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:29 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 7701/73740 objects degraded (10.443%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:57:29.507 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:29 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1710068671' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:57:29.507 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:29 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:57:29.566 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:29.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:29 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 7701/73740 objects degraded (10.443%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:57:29.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:29 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1710068671' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:57:29.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:29 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:57:29.600 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:29.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:29.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:29.702 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:29.737 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:29.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:29.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:29.839 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:29.871 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:29.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:29.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:29.973 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:30.012 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:30.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:30.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:30.114 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:30.150 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:30.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:30.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:30.197 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:57:30.199 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:57:30.251 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:30.284 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:30.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:30.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:30.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:30 smithi160 ceph-mon[101879]: pgmap v2876: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.2 GiB data, 7.9 GiB used, 350 GiB / 358 GiB avail; 0 B/s wr, 1.52k op/s; 6941/61134 objects degraded (11.354%); 9633/61134 objects misplaced (15.757%); 0 B/s, 28 objects/s recovering 2024-09-21T10:57:30.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:30 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:57:30.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:30 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:57:30.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:30 smithi160 ceph-mon[101879]: osdmap e1519: 8 total, 8 up, 4 in 2024-09-21T10:57:30.386 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:30.408 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:30.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:30.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:30.509 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:30.543 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:30.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:30.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:30.602 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:30 smithi070 ceph-mon[107385]: pgmap v2876: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.2 GiB data, 7.9 GiB used, 350 GiB / 358 GiB avail; 0 B/s wr, 1.52k op/s; 6941/61134 objects degraded (11.354%); 9633/61134 objects misplaced (15.757%); 0 B/s, 28 objects/s recovering 2024-09-21T10:57:30.602 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:30 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:57:30.602 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:30 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:57:30.602 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:30 smithi070 ceph-mon[107385]: osdmap e1519: 8 total, 8 up, 4 in 2024-09-21T10:57:30.602 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:30 smithi070 ceph-mon[112139]: pgmap v2876: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.2 GiB data, 7.9 GiB used, 350 GiB / 358 GiB avail; 0 B/s wr, 1.52k op/s; 6941/61134 objects degraded (11.354%); 9633/61134 objects misplaced (15.757%); 0 B/s, 28 objects/s recovering 2024-09-21T10:57:30.602 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:30 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:57:30.603 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:30 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:57:30.603 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:30 smithi070 ceph-mon[112139]: osdmap e1519: 8 total, 8 up, 4 in 2024-09-21T10:57:30.647 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:30.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:30.679 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:30.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:30.780 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:30.810 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:30.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:30.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:30.912 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:30.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:30.948 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:30.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:31.049 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:31.085 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:31.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:31.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:31.189 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:31.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:31.259 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:31.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:31.360 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:31.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:31.397 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:31.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:31.498 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:31.531 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:31.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:31.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:31.536 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:57:31.536 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:57:31.536 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:57:31.537 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:57:31.537 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:57:31.537 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:57:31.537 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:57:31.537 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:57:31.537 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:57:31.537 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:57:31.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:57:31.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:57:31.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:57:31.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:57:31.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:57:31.634 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:31.673 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:31.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:31.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:31.775 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:31.810 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:31.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:31.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:31.911 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:31.948 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:31.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:31.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:32.050 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:32.083 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:32.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:32.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:32.186 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:32.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:32.220 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:32.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:32.321 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:32.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:32.354 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:32.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:32.455 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:32.488 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:32.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:32.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:32.590 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:32.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:32 smithi160 ceph-mon[101879]: pgmap v2878: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.1 GiB data, 7.4 GiB used, 350 GiB / 358 GiB avail; 0 B/s wr, 1.35k op/s; 6330/53943 objects degraded (11.735%); 9633/53943 objects misplaced (17.858%); 0 B/s, 39 objects/s recovering 2024-09-21T10:57:32.625 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:32.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:32.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:32.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:32 smithi070 ceph-mon[107385]: pgmap v2878: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.1 GiB data, 7.4 GiB used, 350 GiB / 358 GiB avail; 0 B/s wr, 1.35k op/s; 6330/53943 objects degraded (11.735%); 9633/53943 objects misplaced (17.858%); 0 B/s, 39 objects/s recovering 2024-09-21T10:57:32.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:32 smithi070 ceph-mon[112139]: pgmap v2878: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.1 GiB data, 7.4 GiB used, 350 GiB / 358 GiB avail; 0 B/s wr, 1.35k op/s; 6330/53943 objects degraded (11.735%); 9633/53943 objects misplaced (17.858%); 0 B/s, 39 objects/s recovering 2024-09-21T10:57:32.643 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:57:32.727 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:32.758 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:32.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:32.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:32.859 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:32.887 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:32.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:32.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:32.989 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:33.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:33.030 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:33.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:33.100 INFO:teuthology.orchestra.run.smithi070.stderr:no valid command found; 10 closest matches: 2024-09-21T10:57:33.100 INFO:teuthology.orchestra.run.smithi070.stderr:0 2024-09-21T10:57:33.100 INFO:teuthology.orchestra.run.smithi070.stderr:1 2024-09-21T10:57:33.100 INFO:teuthology.orchestra.run.smithi070.stderr:2 2024-09-21T10:57:33.100 INFO:teuthology.orchestra.run.smithi070.stderr:abort 2024-09-21T10:57:33.100 INFO:teuthology.orchestra.run.smithi070.stderr:assert 2024-09-21T10:57:33.100 INFO:teuthology.orchestra.run.smithi070.stderr:bench [] [] [] [] 2024-09-21T10:57:33.100 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:57:33.100 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs files list 2024-09-21T10:57:33.100 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs stats 2024-09-21T10:57:33.100 INFO:teuthology.orchestra.run.smithi070.stderr:bluestore allocator dump block 2024-09-21T10:57:33.101 INFO:teuthology.orchestra.run.smithi070.stderr:Error EINVAL: invalid command 2024-09-21T10:57:33.131 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:33.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:33.173 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:33.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:33.274 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:33.309 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:33.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:33.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:33.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:57:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:57:33] "GET /metrics HTTP/1.1" 200 36642 "" "Prometheus/2.51.0" 2024-09-21T10:57:33.411 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:33.435 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:33.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:33.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:33.536 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:33.569 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:33.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:33.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:33.671 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:33.701 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:33.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:33.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:33.803 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:33.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:33.828 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:33.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:33.929 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:33.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:33.962 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:33.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:34.063 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:34.103 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:34.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:34.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:34.205 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:34.228 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:34.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:34.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:34.330 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:34.366 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:34.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:34.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:34.467 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:34.501 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:34.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:34.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:34.563 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:34 smithi070 ceph-mon[107385]: pgmap v2879: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 858 MiB data, 6.7 GiB used, 351 GiB / 358 GiB avail; 0 B/s wr, 1.34k op/s; 5503/41250 objects degraded (13.341%); 7119/41250 objects misplaced (17.258%); 0 B/s, 39 objects/s recovering 2024-09-21T10:57:34.563 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:34 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 5503/41250 objects degraded (13.341%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:57:34.563 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:34 smithi070 ceph-mon[112139]: pgmap v2879: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 858 MiB data, 6.7 GiB used, 351 GiB / 358 GiB avail; 0 B/s wr, 1.34k op/s; 5503/41250 objects degraded (13.341%); 7119/41250 objects misplaced (17.258%); 0 B/s, 39 objects/s recovering 2024-09-21T10:57:34.563 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:34 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 5503/41250 objects degraded (13.341%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:57:34.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:34 smithi160 ceph-mon[101879]: pgmap v2879: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 858 MiB data, 6.7 GiB used, 351 GiB / 358 GiB avail; 0 B/s wr, 1.34k op/s; 5503/41250 objects degraded (13.341%); 7119/41250 objects misplaced (17.258%); 0 B/s, 39 objects/s recovering 2024-09-21T10:57:34.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:34 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 5503/41250 objects degraded (13.341%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:57:34.603 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:34.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:34.642 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:34.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:34.743 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:34.771 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:34.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:34.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:34.873 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:34.906 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:34.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:34.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:35.007 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:35.038 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:35.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:35.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:35.140 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:35.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:35.177 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:35.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:35.278 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:35.310 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:35.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:35.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:35.412 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:35.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:35.451 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:35.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:35.552 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:35.584 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:35.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:35.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:35.686 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:35.726 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:35.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:35.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:35.828 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:35.859 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:35.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:35.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:35.961 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:35.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:35.995 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:35.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:36.096 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:36.130 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:36.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:36.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:36.232 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:36.266 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:36.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:36.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:36.368 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:36.405 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:36.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:36.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:36.507 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:36.529 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:36.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:36.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:36 smithi070 ceph-mon[107385]: pgmap v2880: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 698 MiB data, 6.2 GiB used, 351 GiB / 358 GiB avail; 0 B/s wr, 1.34k op/s; 4829/33552 objects degraded (14.393%); 7119/33552 objects misplaced (21.218%); 0 B/s, 45 objects/s recovering 2024-09-21T10:57:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:57:36.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:57:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:36 smithi070 ceph-mon[112139]: pgmap v2880: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 698 MiB data, 6.2 GiB used, 351 GiB / 358 GiB avail; 0 B/s wr, 1.34k op/s; 4829/33552 objects degraded (14.393%); 7119/33552 objects misplaced (21.218%); 0 B/s, 45 objects/s recovering 2024-09-21T10:57:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:57:36.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:57:36.631 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:36.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:36.665 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:36.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:36.766 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:36.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:36.789 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:36.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:36 smithi160 ceph-mon[101879]: pgmap v2880: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 698 MiB data, 6.2 GiB used, 351 GiB / 358 GiB avail; 0 B/s wr, 1.34k op/s; 4829/33552 objects degraded (14.393%); 7119/33552 objects misplaced (21.218%); 0 B/s, 45 objects/s recovering 2024-09-21T10:57:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:57:36.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:57:36.890 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:36.925 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:36.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:36.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:37.027 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:37.062 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:37.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:37.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:37.164 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:37.200 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:37.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:37.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:37.301 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:37.334 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:37.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:37.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:37.436 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:37.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:37.476 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:37.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:37.577 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:37.611 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:37.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:37.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:37.713 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:37.749 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:37.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:37.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:37.851 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:37.884 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:37.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:37.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:37.986 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:38.020 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:38.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:38.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:38.121 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:38.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:38.154 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:38.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:38.255 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:38.301 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:38.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:38.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:38.402 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:38.440 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:38.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:38.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:38.541 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:38.580 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:38.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:38.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:38 smithi070 ceph-mon[107385]: pgmap v2881: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 698 MiB data, 6.2 GiB used, 351 GiB / 358 GiB avail; 0 B/s wr, 1.34k op/s; 4829/33552 objects degraded (14.393%); 7119/33552 objects misplaced (21.218%); 0 B/s, 45 objects/s recovering 2024-09-21T10:57:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:38 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 4829/33552 objects degraded (14.393%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:57:38.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:38 smithi070 ceph-mon[112139]: pgmap v2881: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 698 MiB data, 6.2 GiB used, 351 GiB / 358 GiB avail; 0 B/s wr, 1.34k op/s; 4829/33552 objects degraded (14.393%); 7119/33552 objects misplaced (21.218%); 0 B/s, 45 objects/s recovering 2024-09-21T10:57:38.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:38 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 4829/33552 objects degraded (14.393%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:57:38.682 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:38.718 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:38.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:38.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:38.820 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:38 smithi160 ceph-mon[101879]: pgmap v2881: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 698 MiB data, 6.2 GiB used, 351 GiB / 358 GiB avail; 0 B/s wr, 1.34k op/s; 4829/33552 objects degraded (14.393%); 7119/33552 objects misplaced (21.218%); 0 B/s, 45 objects/s recovering 2024-09-21T10:57:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:38 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 4829/33552 objects degraded (14.393%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:57:38.854 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:38.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:38.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:38.956 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:38.991 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:38.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:38.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:39.092 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:39.125 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:39.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:39.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:39.227 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:39.261 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:39.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:39.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:39.362 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:39.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:39.397 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:39.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:39.498 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:39.534 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:39.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:39.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:39.636 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:39.675 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:39.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:39.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:39.776 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:39.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:39.809 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:39.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:39 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2471922671' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:57:39.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:39 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2471922671' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:57:39.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:39 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2471922671' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:57:39.910 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:39.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:39.936 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:39.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:40.037 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:40.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:40.071 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:40.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:40.172 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:40.210 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:40.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:40.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:40.311 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:40.346 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:40.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:40.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:40.448 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:40.486 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:40.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:40.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:40.587 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:40.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:40.618 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:40.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:40.719 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:40.752 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:40.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:40.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:40 smithi160 ceph-mon[101879]: pgmap v2882: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 432 MiB data, 5.5 GiB used, 352 GiB / 358 GiB avail; 0 B/s wr, 1.34k op/s; 3991/20820 objects degraded (19.169%); 4606/20820 objects misplaced (22.123%); 0 B/s, 45 objects/s recovering 2024-09-21T10:57:40.854 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:40.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:40 smithi070 ceph-mon[107385]: pgmap v2882: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 432 MiB data, 5.5 GiB used, 352 GiB / 358 GiB avail; 0 B/s wr, 1.34k op/s; 3991/20820 objects degraded (19.169%); 4606/20820 objects misplaced (22.123%); 0 B/s, 45 objects/s recovering 2024-09-21T10:57:40.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:40 smithi070 ceph-mon[112139]: pgmap v2882: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 432 MiB data, 5.5 GiB used, 352 GiB / 358 GiB avail; 0 B/s wr, 1.34k op/s; 3991/20820 objects degraded (19.169%); 4606/20820 objects misplaced (22.123%); 0 B/s, 45 objects/s recovering 2024-09-21T10:57:40.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:40.887 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:40.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:40.988 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:41.021 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:41.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:41.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:41.071 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Removed 116218 objects 2024-09-21T10:57:41.071 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Clean up completed and total clean up time :104.506 2024-09-21T10:57:41.123 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:41.144 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:41.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:41.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:41.246 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:41.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:41.268 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:41.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:41.369 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:41.390 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:41.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:41.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:41.492 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:41.513 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:41.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:41.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:41.540 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:57:41.540 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:57:41.540 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:57:41.540 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:57:41.540 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:57:41.540 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:57:41.540 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:57:41.540 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:57:41.541 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:57:41.541 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:57:41.541 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:57:41.541 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:57:41.541 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:57:41.541 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:57:41.541 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:57:41.541 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:57:41.598 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-09-21T10:57:41.598 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-09-21T10:57:41.615 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:41.665 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:41.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:41.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:41.766 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:41.788 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:41.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:41.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:41.890 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:41.912 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:41.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:41.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:42.014 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:42.035 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:42.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:42.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:42.137 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:42.159 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:42.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:42.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:42.260 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:42.281 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:42.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:42.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:42.383 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:42.404 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:42.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:42.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:42.506 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:42.527 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:42.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:42.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:42.629 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:42.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:42.651 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:42.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:42.752 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:42.773 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:42.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:42.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:42 smithi160 ceph-mon[101879]: pgmap v2883: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 273 MiB data, 5.0 GiB used, 353 GiB / 358 GiB avail; 0 B/s wr, 1.22k op/s; 3353/13170 objects degraded (25.459%); 4606/13170 objects misplaced (34.973%); 0 B/s, 37 objects/s recovering 2024-09-21T10:57:42.874 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:42.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:42 smithi070 ceph-mon[107385]: pgmap v2883: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 273 MiB data, 5.0 GiB used, 353 GiB / 358 GiB avail; 0 B/s wr, 1.22k op/s; 3353/13170 objects degraded (25.459%); 4606/13170 objects misplaced (34.973%); 0 B/s, 37 objects/s recovering 2024-09-21T10:57:42.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:42 smithi070 ceph-mon[112139]: pgmap v2883: 109 pgs: 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 273 MiB data, 5.0 GiB used, 353 GiB / 358 GiB avail; 0 B/s wr, 1.22k op/s; 3353/13170 objects degraded (25.459%); 4606/13170 objects misplaced (34.973%); 0 B/s, 37 objects/s recovering 2024-09-21T10:57:42.896 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:42.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:42.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:42.998 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:43.019 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:43.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:43.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:43.121 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:43.143 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:43.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:43.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:43.244 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:43.268 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:43.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:43.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:43.369 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:43.391 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:43.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:43.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:43.408 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:57:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:57:43] "GET /metrics HTTP/1.1" 200 36626 "" "Prometheus/2.51.0" 2024-09-21T10:57:43.493 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:43.514 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:43.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:43.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:43.616 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:43.638 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:43.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:43.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:43.740 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:43.762 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:43.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:43.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:43 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 3353/13170 objects degraded (25.459%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:57:43.864 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:43.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:43 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 3353/13170 objects degraded (25.459%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:57:43.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:43 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 3353/13170 objects degraded (25.459%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:57:43.885 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:43.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:43.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:43.987 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:44.010 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:44.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:44.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:44.112 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:44.133 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:44.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:44.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:44.235 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:44.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:44.259 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:44.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:44.360 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:44.381 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:44.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:44.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:44.483 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:44.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:44.506 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:44.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:44.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:44 smithi160 ceph-mon[101879]: pgmap v2884: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+undersized+remapped, 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 93 active+clean; 76 MiB data, 4.4 GiB used, 353 GiB / 358 GiB avail; 0 B/s wr, 1.42k op/s; 2664/3732 objects degraded (71.383%); 2754/3732 objects misplaced (73.794%); 0 B/s, 36 objects/s recovering 2024-09-21T10:57:44.606 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:44.628 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:44.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:44.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:44.730 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:44.752 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:44.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:44.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:44.854 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:44.877 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:44.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:44 smithi070 ceph-mon[107385]: pgmap v2884: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+undersized+remapped, 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 93 active+clean; 76 MiB data, 4.4 GiB used, 353 GiB / 358 GiB avail; 0 B/s wr, 1.42k op/s; 2664/3732 objects degraded (71.383%); 2754/3732 objects misplaced (73.794%); 0 B/s, 36 objects/s recovering 2024-09-21T10:57:44.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:44 smithi070 ceph-mon[112139]: pgmap v2884: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+undersized+remapped, 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 93 active+clean; 76 MiB data, 4.4 GiB used, 353 GiB / 358 GiB avail; 0 B/s wr, 1.42k op/s; 2664/3732 objects degraded (71.383%); 2754/3732 objects misplaced (73.794%); 0 B/s, 36 objects/s recovering 2024-09-21T10:57:44.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:44.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:44.979 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:45.002 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:45.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:45.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:45.103 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:45.130 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:45.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:45.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:45.232 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:45.255 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:45.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:45.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:45.357 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:45.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:45.381 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:45.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:45.482 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:45.504 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:45.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:45.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:45.606 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:45.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:45.629 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:45.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:45.730 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:45.752 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:45.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:45.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:45.854 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:45.881 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:45.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:45.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:45.982 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:46.004 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:46.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:46.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:46.076 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:57:46.076 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:57:46.106 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:46.132 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:46.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:46.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:46.233 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:46.256 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:46.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:46.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:46.358 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:46.380 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:46.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:46.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:46.482 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:46.505 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:46.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:46.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:46.606 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:46 smithi070 ceph-mon[107385]: pgmap v2885: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+undersized+remapped, 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 93 active+clean; 28 MiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 0 B/s wr, 1.14k op/s; 1517/1392 objects degraded (108.980%); 2754/1392 objects misplaced (197.845%); 0 B/s, 118 objects/s recovering 2024-09-21T10:57:46.607 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:46 smithi070 ceph-mon[112139]: pgmap v2885: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+undersized+remapped, 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 93 active+clean; 28 MiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 0 B/s wr, 1.14k op/s; 1517/1392 objects degraded (108.980%); 2754/1392 objects misplaced (197.845%); 0 B/s, 118 objects/s recovering 2024-09-21T10:57:46.607 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:46.630 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:46.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:46.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:46.637 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:57:46.637 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:57:46.731 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:46.754 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:46.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:46.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:46 smithi160 ceph-mon[101879]: pgmap v2885: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+undersized+remapped, 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 93 active+clean; 28 MiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 0 B/s wr, 1.14k op/s; 1517/1392 objects degraded (108.980%); 2754/1392 objects misplaced (197.845%); 0 B/s, 118 objects/s recovering 2024-09-21T10:57:46.856 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:46.878 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:46.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:46.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:46.887 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:57:46.915 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:57:46.980 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:47.007 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:47.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:47.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:47.108 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:47.130 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:47.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:47.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:47.232 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:47.253 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:47.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:47.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:47.355 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:47.381 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:47.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:47.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:47.451 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:57:47.483 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:47.508 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:47.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:47.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:47.610 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:47.634 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:47.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:47.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:47.663 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:57:47.736 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:47.760 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:47.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:47.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:47.862 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:47.887 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:47.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:47.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:47.988 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:48.013 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:48.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:48.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:48.115 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:48.137 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:48.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:48.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:48.238 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:48.264 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:48.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:48.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:48.366 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:48.388 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:48.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:48.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:48.490 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:48.514 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:48.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:48.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:48.580 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:48 smithi070 ceph-mon[107385]: pgmap v2886: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+undersized+remapped, 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 93 active+clean; 28 MiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 0 B/s wr, 929 op/s; 1517/1392 objects degraded (108.980%); 2754/1392 objects misplaced (197.845%); 0 B/s, 98 objects/s recovering 2024-09-21T10:57:48.580 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:48 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 1517/1392 objects degraded (108.980%), 4 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:57:48.580 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:48 smithi070 ceph-mon[107385]: osdmap e1520: 8 total, 8 up, 4 in 2024-09-21T10:57:48.580 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:48 smithi070 ceph-mon[112139]: pgmap v2886: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+undersized+remapped, 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 93 active+clean; 28 MiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 0 B/s wr, 929 op/s; 1517/1392 objects degraded (108.980%); 2754/1392 objects misplaced (197.845%); 0 B/s, 98 objects/s recovering 2024-09-21T10:57:48.581 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:48 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 1517/1392 objects degraded (108.980%), 4 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:57:48.581 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:48 smithi070 ceph-mon[112139]: osdmap e1520: 8 total, 8 up, 4 in 2024-09-21T10:57:48.616 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:48.640 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:48.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:48.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:48.742 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:48.765 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:48.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:48.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:48 smithi160 ceph-mon[101879]: pgmap v2886: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+undersized+remapped, 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 93 active+clean; 28 MiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 0 B/s wr, 929 op/s; 1517/1392 objects degraded (108.980%); 2754/1392 objects misplaced (197.845%); 0 B/s, 98 objects/s recovering 2024-09-21T10:57:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:48 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 1517/1392 objects degraded (108.980%), 4 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:57:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:48 smithi160 ceph-mon[101879]: osdmap e1520: 8 total, 8 up, 4 in 2024-09-21T10:57:48.868 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:48.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:48.892 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:48.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:48.993 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:49.017 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:49.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:49.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:49.118 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:49.140 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:49.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:49.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:49.242 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:49.263 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:49.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:49.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:49.365 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:49.387 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:49.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:49.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:49.489 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:49.514 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:49.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:49.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:49.616 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:49.640 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:49.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:49.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:49.742 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:49.747 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:57:49.768 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:49.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:49.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:49.871 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:49.893 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:49.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:49.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:49.997 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:50.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:50.020 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:50.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:50.122 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:50.143 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:50.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:50.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:50.245 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:50.266 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:50.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:50.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:50.368 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:50.391 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:50.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:50.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:50.449 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:50 smithi070 ceph-mon[112139]: pgmap v2888: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+undersized+remapped, 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 93 active+clean; 28 MiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 0 B/s wr, 691 op/s; 1517/1392 objects degraded (108.980%); 2754/1392 objects misplaced (197.845%); 0 B/s, 117 objects/s recovering 2024-09-21T10:57:50.449 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:50 smithi070 ceph-mon[112139]: osdmap e1521: 8 total, 8 up, 4 in 2024-09-21T10:57:50.449 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:57:50.449 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:50 smithi070 ceph-mon[107385]: pgmap v2888: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+undersized+remapped, 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 93 active+clean; 28 MiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 0 B/s wr, 691 op/s; 1517/1392 objects degraded (108.980%); 2754/1392 objects misplaced (197.845%); 0 B/s, 117 objects/s recovering 2024-09-21T10:57:50.449 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:50 smithi070 ceph-mon[107385]: osdmap e1521: 8 total, 8 up, 4 in 2024-09-21T10:57:50.449 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:57:50.493 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:50.517 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:50.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:50.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:50.619 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:50.640 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:50.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:50.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:50.742 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:50.764 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:50.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:50.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:50 smithi160 ceph-mon[101879]: pgmap v2888: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+undersized+remapped, 9 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 93 active+clean; 28 MiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 0 B/s wr, 691 op/s; 1517/1392 objects degraded (108.980%); 2754/1392 objects misplaced (197.845%); 0 B/s, 117 objects/s recovering 2024-09-21T10:57:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:50 smithi160 ceph-mon[101879]: osdmap e1521: 8 total, 8 up, 4 in 2024-09-21T10:57:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:57:50.866 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:50.887 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:50.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:50.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:50.989 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:51.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:51.011 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:51.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:51.077 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7] out_osds: [2, 5, 0, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:57:51.077 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:57:51.078 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-21T10:57:51.078 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:57:51.078 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:57:51.078 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-09-21T10:57:51.112 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:51.144 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:51.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:51.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:51.245 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:51.267 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:51.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:51.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:51.369 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:51.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:51.391 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:51.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:51.401 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'unique_pool_3' removed 2024-09-21T10:57:51.493 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:51.526 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:51.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:51.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:51 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1382211' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:57:51.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:51 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1382211' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:57:51.627 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:51.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:51.666 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:51.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:51.767 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:51.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:51.791 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:51.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:51.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:51 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1382211' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:57:51.892 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:51.915 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:51.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:51.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:52.017 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:52.038 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:52.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:52.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:52.140 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:52.161 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:52.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:52.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:52.262 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:52.284 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:52.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:52.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:52.386 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:52.409 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:52.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:52.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:52.410 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T10:57:52.511 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:52.533 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:52.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:52.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:52.635 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:52.656 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:52.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:52.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:52.759 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:52.781 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:52.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:52.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:52 smithi160 ceph-mon[101879]: pgmap v2890: 109 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+remapped, 9 active+recovery_wait+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 93 active+clean; 28 MiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 0 B/s wr, 117 op/s; 64/1392 objects degraded (4.598%); 2754/1392 objects misplaced (197.845%); 0 B/s, 423 objects/s recovering 2024-09-21T10:57:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:52 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1382211' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:57:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:52 smithi160 ceph-mon[101879]: osdmap e1522: 8 total, 8 up, 4 in 2024-09-21T10:57:52.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:52 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1856597143' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:57:52.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:52 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:57:52.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:52 smithi070 ceph-mon[107385]: pgmap v2890: 109 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+remapped, 9 active+recovery_wait+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 93 active+clean; 28 MiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 0 B/s wr, 117 op/s; 64/1392 objects degraded (4.598%); 2754/1392 objects misplaced (197.845%); 0 B/s, 423 objects/s recovering 2024-09-21T10:57:52.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:52 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1382211' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:57:52.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:52 smithi070 ceph-mon[107385]: osdmap e1522: 8 total, 8 up, 4 in 2024-09-21T10:57:52.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:52 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1856597143' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:57:52.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:52 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:57:52.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:52 smithi070 ceph-mon[112139]: pgmap v2890: 109 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+remapped, 9 active+recovery_wait+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 93 active+clean; 28 MiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 0 B/s wr, 117 op/s; 64/1392 objects degraded (4.598%); 2754/1392 objects misplaced (197.845%); 0 B/s, 423 objects/s recovering 2024-09-21T10:57:52.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:52 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1382211' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:57:52.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:52 smithi070 ceph-mon[112139]: osdmap e1522: 8 total, 8 up, 4 in 2024-09-21T10:57:52.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:52 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1856597143' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:57:52.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:52 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:57:52.883 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:52.904 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:52.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:52.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:53.007 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:53.034 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:53.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:53.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:53.135 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:53.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:53.158 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:53.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:53.259 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:53.285 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:53.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:53.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:53.386 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:53.413 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:53.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:53.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:53.514 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:53.539 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:53.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:53.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:53.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:57:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:57:53] "GET /metrics HTTP/1.1" 200 36622 "" "Prometheus/2.51.0" 2024-09-21T10:57:53.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:53 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:57:53.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:53 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:57:53.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:53 smithi070 ceph-mon[107385]: osdmap e1523: 8 total, 8 up, 4 in 2024-09-21T10:57:53.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:53 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 64/1392 objects degraded (4.598%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:57:53.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:53 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:57:53.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:53 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:57:53.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:53 smithi070 ceph-mon[112139]: osdmap e1523: 8 total, 8 up, 4 in 2024-09-21T10:57:53.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:53 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 64/1392 objects degraded (4.598%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:57:53.642 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:53.664 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:53.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:53.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:53.766 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:53.789 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:53.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:53.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:53 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:57:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:53 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:57:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:53 smithi160 ceph-mon[101879]: osdmap e1523: 8 total, 8 up, 4 in 2024-09-21T10:57:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:53 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 64/1392 objects degraded (4.598%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:57:53.891 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:53.917 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:53.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:53.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:54.019 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:54.044 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:54.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:54.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:54.146 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:54.171 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:54.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:54.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:54.273 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:54.299 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:54.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:54.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:54.401 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:54.426 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:54.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:54.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:54.528 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:54.554 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:54.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:54.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:54.656 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:54.667 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:54 smithi070 ceph-mon[107385]: pgmap v2893: 83 pgs: 83 active+clean; 268 KiB data, 4.1 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:57:54.667 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:54 smithi070 ceph-mon[107385]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 64/1392 objects degraded (4.598%), 2 pgs degraded, 2 pgs undersized) 2024-09-21T10:57:54.667 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:54 smithi070 ceph-mon[112139]: pgmap v2893: 83 pgs: 83 active+clean; 268 KiB data, 4.1 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:57:54.667 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:54 smithi070 ceph-mon[112139]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 64/1392 objects degraded (4.598%), 2 pgs degraded, 2 pgs undersized) 2024-09-21T10:57:54.676 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:54.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:54.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:54.743 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:54 smithi160 ceph-mon[101879]: pgmap v2893: 83 pgs: 83 active+clean; 268 KiB data, 4.1 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:57:54.743 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:54 smithi160 ceph-mon[101879]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 64/1392 objects degraded (4.598%), 2 pgs degraded, 2 pgs undersized) 2024-09-21T10:57:54.778 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:54.799 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:54.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:54.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:54.901 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:54.922 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:54.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:54.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:55.024 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:55.050 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:55.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:55.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:55.133 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T10:57:55.134 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T10:57:55.134 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-09-21T10:57:55.134 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_4 16 2024-09-21T10:57:55.152 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:55.197 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:55.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:55.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:55.299 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:55.320 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:55.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:55.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:55.421 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:55.442 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:55.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:55.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:55.544 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:55.565 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:55.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:55.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:55.658 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:57:55.666 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:55.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:55.689 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:55.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:55.790 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:55.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:55.813 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:55.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:55.915 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:55.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:55.937 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:55.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:56.038 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:56.060 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:56.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:56.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:56.162 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:56.184 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:56.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:56.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:56.286 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:56.307 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:56.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:56.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:56.408 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:57:56.429 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:56.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:56.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:57:56.479 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:57:56.532 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:56.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:56.554 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:56.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:56.655 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:56.677 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:56.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:56.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:56.756 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:56 smithi070 ceph-mon[107385]: pgmap v2894: 83 pgs: 83 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:57:56.756 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:56 smithi070 ceph-mon[112139]: pgmap v2894: 83 pgs: 83 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:57:56.778 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:56.800 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:56.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:56.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:56 smithi160 ceph-mon[101879]: pgmap v2894: 83 pgs: 83 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:57:56.903 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:56.924 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:56.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:56.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:57.026 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:57.047 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:57.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:57.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:57.149 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:57.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:57.173 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:57.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:57.274 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:57:57.296 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:57.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:57.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:57:57.398 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:57.421 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:57.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:57.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:57.523 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:57.545 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:57.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:57.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:57.647 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:57.669 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:57.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:57.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:57.770 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:57.792 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:57.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:57.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:57.894 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:57.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:57.916 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:57.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:58.017 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:58.041 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:58.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:58.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:58.143 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:58.167 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:58.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:58.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:58.269 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:58.294 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:58.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:58.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:58.396 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:58.419 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:58.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:58.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:58.496 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:57:58.520 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:57:58.543 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:58.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:58.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:57:58.645 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:58.666 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:58.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:58.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:58.716 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:57:58 smithi070 ceph-mon[107385]: pgmap v2895: 83 pgs: 83 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:57:58.717 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:57:58 smithi070 ceph-mon[112139]: pgmap v2895: 83 pgs: 83 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:57:58.768 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:58.794 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:58.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:58.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:57:58 smithi160 ceph-mon[101879]: pgmap v2895: 83 pgs: 83 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:57:58.896 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:58.918 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:58.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:58.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:59.001 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:57:59.020 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:59.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:59.045 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:59.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:57:59.147 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:57:59.169 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:59.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:59.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:57:59.271 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:59.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:59.298 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:59.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:59.399 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:59.426 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:59.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:59.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:59.528 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:57:59.551 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:59.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:59.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:57:59.653 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:57:59.676 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:59.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:59.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:57:59.778 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:57:59.802 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:57:59.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:59.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:57:59.904 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:57:59.926 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:57:59.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:57:59.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:00.028 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:00.050 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:00.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:00.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:00.151 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:00.173 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:00.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:00.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:00.274 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:00.296 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:00.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:00.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:00.390 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:58:00.390 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:58:00.398 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:00.419 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:00.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:00.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:00.521 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:00.542 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:00.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:00.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:00.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:00 smithi070 ceph-mon[107385]: pgmap v2896: 83 pgs: 83 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:00.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:00 smithi070 ceph-mon[112139]: pgmap v2896: 83 pgs: 83 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:00.644 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:00.665 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:00.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:00.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:00.767 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:00.794 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:00.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:00.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:00 smithi160 ceph-mon[101879]: pgmap v2896: 83 pgs: 83 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:00.896 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:00.917 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:00.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:00.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:01.018 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:01.041 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:01.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:01.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:01.062 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:58:01.062 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:58:01.070 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:58:01.070 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:58:01.086 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:58:01.086 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:58:01.103 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:58:01.103 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:58:01.119 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:58:01.119 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:58:01.135 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:58:01.136 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:58:01.143 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:01.152 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:58:01.153 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:58:01.164 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:01.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:01.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:01.170 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:58:01.170 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:58:01.266 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:01.292 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:01.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:01.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:01.394 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:01.418 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:01.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:01.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:01.519 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:01.542 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:01.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:01.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:01.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:01 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3564400364' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:58:01.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:01 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3564400364' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:58:01.644 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:01.667 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:01.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:01.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:01.769 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:01.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:01.791 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:01.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:01 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3564400364' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:58:01.893 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:01.915 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:01.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:01.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:01.967 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:58:01.967 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T10:58:01.967 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:58:02.016 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:02.038 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:02.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:02.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:02.140 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:02.162 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:02.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:02.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:02.264 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:02.290 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:02.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:02.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:02.391 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:02.413 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:02.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:02.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:02.514 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:02.540 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:02.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:02.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:02.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:02 smithi070 ceph-mon[107385]: pgmap v2897: 83 pgs: 83 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:02.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:02 smithi070 ceph-mon[112139]: pgmap v2897: 83 pgs: 83 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:02.642 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:02.666 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:02.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:02.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:02.720 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:58:02.768 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:02.789 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:02.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:02.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:02 smithi160 ceph-mon[101879]: pgmap v2897: 83 pgs: 83 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:02.891 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:02.918 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:02.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:02.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:03.019 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:03.045 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:03.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:03.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:03.146 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:03.168 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:03.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:03.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:03.269 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:03.290 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:03.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:03.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:03.392 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:03.418 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:03.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:03.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:03.427 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:58:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:58:03] "GET /metrics HTTP/1.1" 200 36622 "" "Prometheus/2.51.0" 2024-09-21T10:58:03.520 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:03.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:03.543 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:03.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:03.644 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:03.666 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:03.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:03.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:03.758 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:58:03.768 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:03.791 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:03.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:03.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:03.892 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:03.919 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:03.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:03.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:04.020 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:04.024 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:58:04.045 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:04.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:04.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:04.146 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:04.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:04.169 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:04.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:04.270 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:04.291 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:04.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:04.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:04.394 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:04.419 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:04.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:04.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:04.522 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:04.547 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:04.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:04.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:04.576 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:04 smithi070 ceph-mon[107385]: pgmap v2898: 83 pgs: 83 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:04.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:04 smithi070 ceph-mon[112139]: pgmap v2898: 83 pgs: 83 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:04.649 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:04.673 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:04.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:04.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:04.775 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:04.797 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:04.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:04.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:04.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:04 smithi160 ceph-mon[101879]: pgmap v2898: 83 pgs: 83 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:04.898 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:04.924 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:04.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:04.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:05.026 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:05.052 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:05.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:05.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:05.154 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:05.180 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:05.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:05.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:05.281 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:05.303 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:05.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:05.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:05.404 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:05.426 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'unique_pool_4' created 2024-09-21T10:58:05.426 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:05.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:05.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:05.528 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:05.550 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:05.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:05.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:05.620 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:05 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3464790991' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-21T10:58:05.621 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:58:05.621 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:58:05.621 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:05 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3464790991' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-21T10:58:05.621 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:58:05.621 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:58:05.652 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:05.674 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:05.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:05.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:05.742 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:58:05.776 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:05.804 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:05.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:05.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:05 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3464790991' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-21T10:58:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:58:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:58:05.907 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:05.930 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:05.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:05.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:06.032 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:06.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:06.058 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:06.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:06.158 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:06.184 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:06.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:06.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:06.282 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-09-21T10:58:06.285 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:06.347 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:06.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:06.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:06.449 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:06.473 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:06.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:06.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:06.575 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:06.598 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:06.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:06.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:06 smithi070 ceph-mon[107385]: pgmap v2899: 83 pgs: 83 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:06 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3464790991' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-21T10:58:06.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:06 smithi070 ceph-mon[107385]: osdmap e1524: 8 total, 8 up, 4 in 2024-09-21T10:58:06.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:06 smithi070 ceph-mon[112139]: pgmap v2899: 83 pgs: 83 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:06.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:06 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3464790991' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-21T10:58:06.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:06 smithi070 ceph-mon[112139]: osdmap e1524: 8 total, 8 up, 4 in 2024-09-21T10:58:06.700 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:06.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:06.725 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:06.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:06.826 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:06 smithi160 ceph-mon[101879]: pgmap v2899: 83 pgs: 83 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:06 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3464790991' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-21T10:58:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:06 smithi160 ceph-mon[101879]: osdmap e1524: 8 total, 8 up, 4 in 2024-09-21T10:58:06.846 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:06.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:06.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:06.948 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:06.971 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:06.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:06.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:07.048 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:58:07.073 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:07.094 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:07.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:07.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:07.195 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:07.220 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:07.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:07.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:07.322 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:07.347 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:07.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:07.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:07.449 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:07.473 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:07.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:07.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:07.575 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:07.598 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:07.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:07.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:07.700 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:07.723 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:07.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:07.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:07.797 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:07 smithi070 ceph-mon[112139]: osdmap e1525: 8 total, 8 up, 4 in 2024-09-21T10:58:07.798 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:07 smithi070 ceph-mon[107385]: osdmap e1525: 8 total, 8 up, 4 in 2024-09-21T10:58:07.816 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:58:07.824 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:07 smithi160 ceph-mon[101879]: osdmap e1525: 8 total, 8 up, 4 in 2024-09-21T10:58:07.845 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:07.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:07.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:07.947 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:07.970 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:07.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:07.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:08.071 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:08.093 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:08.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:08.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:08.154 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7] out_osds: [2, 5, 0, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:58:08.154 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:58:08.155 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-21T10:58:08.155 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd in 4 2024-09-21T10:58:08.195 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:08.217 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:08.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:08.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:08.319 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:08.340 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:08.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:08.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:08.441 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:08.446 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T10:58:08.464 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:08.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:08.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:08.565 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:08.589 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:08.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:08.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:08.691 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:08.713 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:08.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:08.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:08.722 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:08 smithi070 ceph-mon[107385]: pgmap v2902: 99 pgs: 16 unknown, 83 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:08.723 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:08 smithi070 ceph-mon[107385]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:58:08.723 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:08 smithi070 ceph-mon[107385]: osdmap e1526: 8 total, 8 up, 4 in 2024-09-21T10:58:08.723 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:08 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2581616512' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:58:08.723 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:08 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:58:08.723 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:08 smithi070 ceph-mon[112139]: pgmap v2902: 99 pgs: 16 unknown, 83 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:08.723 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:08 smithi070 ceph-mon[112139]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:58:08.723 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:08 smithi070 ceph-mon[112139]: osdmap e1526: 8 total, 8 up, 4 in 2024-09-21T10:58:08.723 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:08 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2581616512' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:58:08.723 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:08 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:58:08.815 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:08.828 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:58:08.828 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T10:58:08.828 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T10:58:08.828 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T10:58:08.828 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.828 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:76000 55.3 55:cd844426:::benchmark_data_smithi070_294394_object75999:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1426)", 2024-09-21T10:58:08.828 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:49:43.855902+0000", 2024-09-21T10:58:08.828 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 504.96747686200001, 2024-09-21T10:58:08.828 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2807186910000001, 2024-09-21T10:58:08.828 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:58:08.829 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:58:08.829 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:58:08.829 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:58:08.829 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:58:08.829 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:58:08.829 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 76000 2024-09-21T10:58:08.829 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.829 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:58:08.829 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.829 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:58:08.829 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855902+0000", 2024-09-21T10:58:08.829 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.829 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.830 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.830 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:58:08.830 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855902+0000", 2024-09-21T10:58:08.830 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.830 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.830 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.830 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:58:08.830 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855904+0000", 2024-09-21T10:58:08.830 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1840000000000002e-06 2024-09-21T10:58:08.830 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.830 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.830 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:58:08.830 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855927+0000", 2024-09-21T10:58:08.830 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2909999999999999e-05 2024-09-21T10:58:08.831 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.831 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.831 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:58:08.831 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855957+0000", 2024-09-21T10:58:08.831 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9437e-05 2024-09-21T10:58:08.831 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.831 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.831 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:58:08.831 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.855965+0000", 2024-09-21T10:58:08.831 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.4349999999999997e-06 2024-09-21T10:58:08.831 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.831 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.832 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.832 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.856021+0000", 2024-09-21T10:58:08.832 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6232e-05 2024-09-21T10:58:08.832 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.832 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.832 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:58:08.832 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.856214+0000", 2024-09-21T10:58:08.832 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00019308699999999999 2024-09-21T10:58:08.832 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.832 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.832 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 5,7", 2024-09-21T10:58:08.832 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.856386+0000", 2024-09-21T10:58:08.832 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000171621 2024-09-21T10:58:08.832 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.833 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.833 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.833 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.862577+0000", 2024-09-21T10:58:08.833 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0061906019999999999 2024-09-21T10:58:08.833 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.833 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.833 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.833 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.862589+0000", 2024-09-21T10:58:08.833 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2367e-05 2024-09-21T10:58:08.833 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.833 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.834 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.834 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.853010+0000", 2024-09-21T10:58:08.834 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99042137600000002 2024-09-21T10:58:08.834 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.834 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.834 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.834 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.853028+0000", 2024-09-21T10:58:08.834 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7776999999999999e-05 2024-09-21T10:58:08.834 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.834 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.835 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.835 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.116392+0000", 2024-09-21T10:58:08.835 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.263364207 2024-09-21T10:58:08.835 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.835 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.835 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:58:08.835 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.116926+0000", 2024-09-21T10:58:08.835 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053343100000000001 2024-09-21T10:58:08.836 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.836 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.836 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.836 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.136415+0000", 2024-09-21T10:58:08.836 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.019489623000000001 2024-09-21T10:58:08.836 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.836 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.836 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:58:08.836 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.136621+0000", 2024-09-21T10:58:08.837 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00020540200000000001 2024-09-21T10:58:08.837 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.837 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:58:08.837 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.837 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.837 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.837 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:76010 55.3 55:ca4bf1d6:::benchmark_data_smithi070_294394_object76009:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1427)", 2024-09-21T10:58:08.837 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:49:43.866567+0000", 2024-09-21T10:58:08.838 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 504.95681195899999, 2024-09-21T10:58:08.838 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2823839870000002, 2024-09-21T10:58:08.838 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:58:08.838 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:58:08.838 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:58:08.838 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:58:08.838 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:58:08.839 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:58:08.839 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 76010 2024-09-21T10:58:08.839 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.839 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:58:08.839 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.839 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:58:08.839 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.866567+0000", 2024-09-21T10:58:08.839 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.839 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.840 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.840 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:58:08.840 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.866567+0000", 2024-09-21T10:58:08.840 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.840 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.840 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.840 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:58:08.840 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.866568+0000", 2024-09-21T10:58:08.840 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5629999999999999e-06 2024-09-21T10:58:08.841 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.841 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.841 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:58:08.841 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.866589+0000", 2024-09-21T10:58:08.841 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0886999999999998e-05 2024-09-21T10:58:08.841 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.841 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.841 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:58:08.841 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.866600+0000", 2024-09-21T10:58:08.842 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0621e-05 2024-09-21T10:58:08.842 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.842 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.842 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:58:08.842 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.866608+0000", 2024-09-21T10:58:08.842 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9279999999999993e-06 2024-09-21T10:58:08.842 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.842 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.843 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.843 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.866999+0000", 2024-09-21T10:58:08.843 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000390734 2024-09-21T10:58:08.843 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.843 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.843 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.843 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.867008+0000", 2024-09-21T10:58:08.843 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2779999999999999e-06 2024-09-21T10:58:08.844 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.844 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.844 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.844 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.853163+0000", 2024-09-21T10:58:08.844 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.98615497699999999 2024-09-21T10:58:08.844 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.844 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.844 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.844 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.853170+0000", 2024-09-21T10:58:08.845 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5530000000000004e-06 2024-09-21T10:58:08.845 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.845 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.845 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.845 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.120718+0000", 2024-09-21T10:58:08.845 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2675472079999999 2024-09-21T10:58:08.845 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.845 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.846 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:58:08.846 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.120831+0000", 2024-09-21T10:58:08.846 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000113841 2024-09-21T10:58:08.846 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.846 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.846 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:58:08.846 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.120947+0000", 2024-09-21T10:58:08.846 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000115327 2024-09-21T10:58:08.846 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.846 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.846 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.847 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.136561+0000", 2024-09-21T10:58:08.847 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:08 smithi160 ceph-mon[101879]: pgmap v2902: 99 pgs: 16 unknown, 83 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:08.847 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:08 smithi160 ceph-mon[101879]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:58:08.847 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:08 smithi160 ceph-mon[101879]: osdmap e1526: 8 total, 8 up, 4 in 2024-09-21T10:58:08.847 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:08 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2581616512' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:58:08.847 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:08 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:58:08.848 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015613986 2024-09-21T10:58:08.848 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.848 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.848 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.848 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.136585+0000", 2024-09-21T10:58:08.848 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.429e-05 2024-09-21T10:58:08.848 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.848 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.848 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:58:08.848 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.144528+0000", 2024-09-21T10:58:08.849 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0079428559999999999 2024-09-21T10:58:08.849 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.849 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.849 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:58:08.849 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.144577+0000", 2024-09-21T10:58:08.849 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8749999999999999e-05 2024-09-21T10:58:08.849 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.849 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.849 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:58:08.849 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.148951+0000", 2024-09-21T10:58:08.849 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0043741880000000002 2024-09-21T10:58:08.849 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.850 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:58:08.850 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.850 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.850 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.850 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:76030 55.3 55:c3f9098d:::benchmark_data_smithi070_294394_object76029:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1427)", 2024-09-21T10:58:08.850 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:49:43.887718+0000", 2024-09-21T10:58:08.850 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 504.935660783, 2024-09-21T10:58:08.850 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2693207429999998, 2024-09-21T10:58:08.850 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:58:08.850 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:58:08.850 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:58:08.850 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:58:08.850 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:58:08.850 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:58:08.851 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 76030 2024-09-21T10:58:08.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.851 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:58:08.851 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.851 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:58:08.851 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887718+0000", 2024-09-21T10:58:08.851 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.851 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.851 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:58:08.851 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887718+0000", 2024-09-21T10:58:08.851 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.851 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.851 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:58:08.852 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887720+0000", 2024-09-21T10:58:08.852 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5880000000000001e-06 2024-09-21T10:58:08.852 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.852 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.852 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:58:08.852 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887741+0000", 2024-09-21T10:58:08.852 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0874e-05 2024-09-21T10:58:08.852 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.852 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.852 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:58:08.852 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887754+0000", 2024-09-21T10:58:08.852 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3876999999999999e-05 2024-09-21T10:58:08.852 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.852 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.852 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:58:08.853 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887761+0000", 2024-09-21T10:58:08.853 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8290000000000002e-06 2024-09-21T10:58:08.853 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.853 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.853 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.853 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887819+0000", 2024-09-21T10:58:08.853 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7469999999999997e-05 2024-09-21T10:58:08.853 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.853 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.853 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.853 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:43.887828+0000", 2024-09-21T10:58:08.853 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7559999999999994e-06 2024-09-21T10:58:08.853 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.854 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.854 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.854 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.853198+0000", 2024-09-21T10:58:08.854 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.96536982999999998 2024-09-21T10:58:08.854 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.854 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.854 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.854 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:44.853205+0000", 2024-09-21T10:58:08.854 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1559999999999998e-06 2024-09-21T10:58:08.854 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.854 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.854 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.854 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.136704+0000", 2024-09-21T10:58:08.855 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.283498252 2024-09-21T10:58:08.855 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.855 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.855 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:58:08.855 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.136820+0000", 2024-09-21T10:58:08.855 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011636700000000001 2024-09-21T10:58:08.855 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.855 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.855 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:58:08.855 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.136933+0000", 2024-09-21T10:58:08.855 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000112872 2024-09-21T10:58:08.855 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.856 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.856 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.856 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.149001+0000", 2024-09-21T10:58:08.856 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012068146 2024-09-21T10:58:08.856 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.856 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.856 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:58:08.856 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.153155+0000", 2024-09-21T10:58:08.856 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0041541190000000004 2024-09-21T10:58:08.856 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.856 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.856 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.856 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.153219+0000", 2024-09-21T10:58:08.856 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3492999999999997e-05 2024-09-21T10:58:08.857 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.857 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.857 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:58:08.857 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.153287+0000", 2024-09-21T10:58:08.857 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7943999999999999e-05 2024-09-21T10:58:08.857 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.857 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.857 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:58:08.857 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:49:46.157039+0000", 2024-09-21T10:58:08.857 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00375217 2024-09-21T10:58:08.857 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.858 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:58:08.858 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.858 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.858 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.858 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:91286 55.2 55:4def9a29:::benchmark_data_smithi070_294394_object91285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1433)", 2024-09-21T10:58:08.858 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:50:03.104675+0000", 2024-09-21T10:58:08.858 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 485.71870407199998, 2024-09-21T10:58:08.858 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7145347900000001, 2024-09-21T10:58:08.858 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:58:08.858 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:58:08.858 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:58:08.858 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:58:08.859 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:58:08.859 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:58:08.859 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 91286 2024-09-21T10:58:08.859 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.859 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:58:08.859 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.859 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:58:08.859 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.104675+0000", 2024-09-21T10:58:08.859 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.859 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.859 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.859 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:58:08.860 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.104675+0000", 2024-09-21T10:58:08.860 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.860 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.860 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.860 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:58:08.860 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.104676+0000", 2024-09-21T10:58:08.860 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.576e-06 2024-09-21T10:58:08.860 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.860 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.860 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:58:08.860 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.104694+0000", 2024-09-21T10:58:08.861 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7638999999999998e-05 2024-09-21T10:58:08.861 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.861 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.861 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:58:08.861 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.104704+0000", 2024-09-21T10:58:08.861 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0358e-05 2024-09-21T10:58:08.861 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.861 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.861 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:58:08.861 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.104709+0000", 2024-09-21T10:58:08.861 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1159999999999998e-06 2024-09-21T10:58:08.861 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.861 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.862 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.862 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.104755+0000", 2024-09-21T10:58:08.862 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5584000000000003e-05 2024-09-21T10:58:08.862 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.862 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.862 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.862 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.104761+0000", 2024-09-21T10:58:08.862 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6590000000000004e-06 2024-09-21T10:58:08.862 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.862 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.862 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.862 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.257345+0000", 2024-09-21T10:58:08.862 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.15258385399999999 2024-09-21T10:58:08.862 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.863 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.863 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.863 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:03.257350+0000", 2024-09-21T10:58:08.863 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8109999999999999e-06 2024-09-21T10:58:08.863 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.863 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.863 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.863 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.755959+0000", 2024-09-21T10:58:08.863 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4986089069999999 2024-09-21T10:58:08.863 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.863 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.864 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:58:08.864 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.756079+0000", 2024-09-21T10:58:08.864 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000119536 2024-09-21T10:58:08.864 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.864 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.864 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:58:08.864 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.756274+0000", 2024-09-21T10:58:08.864 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00019505400000000001 2024-09-21T10:58:08.864 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.864 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.864 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:58:08.864 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.766227+0000", 2024-09-21T10:58:08.864 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0099535519999999992 2024-09-21T10:58:08.865 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.865 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.865 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.865 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.785592+0000", 2024-09-21T10:58:08.865 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.019365054 2024-09-21T10:58:08.865 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.865 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.865 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.865 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.810900+0000", 2024-09-21T10:58:08.865 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.025307168000000001 2024-09-21T10:58:08.865 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.865 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.865 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:58:08.866 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.811297+0000", 2024-09-21T10:58:08.866 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00039743999999999999 2024-09-21T10:58:08.866 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.866 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.866 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:58:08.866 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:50:04.819210+0000", 2024-09-21T10:58:08.866 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0079124820000000002 2024-09-21T10:58:08.866 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.866 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:58:08.866 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.866 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.866 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.866 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:130845 55.3 55:c4b635a2:::benchmark_data_smithi070_294394_object25504:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:58:08.867 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:26.363041+0000", 2024-09-21T10:58:08.867 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 402.460337643, 2024-09-21T10:58:08.867 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.266966327, 2024-09-21T10:58:08.867 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:58:08.867 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:58:08.867 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:58:08.867 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:58:08.867 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:58:08.867 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:58:08.867 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 130845 2024-09-21T10:58:08.867 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.867 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:58:08.867 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.867 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:58:08.868 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.363041+0000", 2024-09-21T10:58:08.868 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.868 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.868 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:58:08.868 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.363041+0000", 2024-09-21T10:58:08.868 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.868 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.868 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:58:08.868 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.363043+0000", 2024-09-21T10:58:08.868 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.53e-06 2024-09-21T10:58:08.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.868 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.869 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:58:08.869 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.363048+0000", 2024-09-21T10:58:08.869 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8369999999999996e-06 2024-09-21T10:58:08.869 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.869 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.869 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:58:08.869 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.363059+0000", 2024-09-21T10:58:08.869 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1355000000000001e-05 2024-09-21T10:58:08.869 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.869 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.869 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:58:08.869 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.363065+0000", 2024-09-21T10:58:08.869 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8080000000000001e-06 2024-09-21T10:58:08.870 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.870 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.870 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.870 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.366250+0000", 2024-09-21T10:58:08.870 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0031852289999999999 2024-09-21T10:58:08.870 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.870 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.870 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:58:08.870 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.375345+0000", 2024-09-21T10:58:08.870 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0090954079999999993 2024-09-21T10:58:08.870 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.870 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.870 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.870 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.396385+0000", 2024-09-21T10:58:08.871 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.021039869999999999 2024-09-21T10:58:08.871 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.871 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.871 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:58:08.871 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.396584+0000", 2024-09-21T10:58:08.871 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000198304 2024-09-21T10:58:08.871 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.871 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.871 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:58:08.871 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.396972+0000", 2024-09-21T10:58:08.871 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000388561 2024-09-21T10:58:08.871 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.872 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.872 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:58:08.872 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.413918+0000", 2024-09-21T10:58:08.872 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.016945913999999999 2024-09-21T10:58:08.872 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.872 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.872 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.872 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.418908+0000", 2024-09-21T10:58:08.872 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0049898119999999997 2024-09-21T10:58:08.872 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.872 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.872 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.872 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.455449+0000", 2024-09-21T10:58:08.872 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.036541487999999997 2024-09-21T10:58:08.873 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.873 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.873 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.873 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.455463+0000", 2024-09-21T10:58:08.873 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3332e-05 2024-09-21T10:58:08.873 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.873 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.873 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.873 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.429116+0000", 2024-09-21T10:58:08.873 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.97365345800000003 2024-09-21T10:58:08.873 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.873 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.873 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.873 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.429128+0000", 2024-09-21T10:58:08.874 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1681999999999999e-05 2024-09-21T10:58:08.874 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.874 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.874 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.874 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.629922+0000", 2024-09-21T10:58:08.874 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2007940420000001 2024-09-21T10:58:08.874 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.874 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.874 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:58:08.874 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.630008+0000", 2024-09-21T10:58:08.875 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5697000000000003e-05 2024-09-21T10:58:08.875 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.875 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:58:08.875 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.875 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.875 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.875 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:130863 55.3 55:c477e325:::benchmark_data_smithi070_294394_object25522:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:58:08.875 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:26.403411+0000", 2024-09-21T10:58:08.875 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 402.41996799600003, 2024-09-21T10:58:08.875 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4185171620000001, 2024-09-21T10:58:08.875 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:58:08.875 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:58:08.875 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:58:08.876 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:58:08.876 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:58:08.876 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:58:08.876 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 130863 2024-09-21T10:58:08.876 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.876 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:58:08.876 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.876 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:58:08.876 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.403411+0000", 2024-09-21T10:58:08.876 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.876 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.876 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.876 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:58:08.877 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.403411+0000", 2024-09-21T10:58:08.877 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.877 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.877 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.877 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:58:08.877 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.403413+0000", 2024-09-21T10:58:08.877 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3159999999999999e-06 2024-09-21T10:58:08.877 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.877 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.877 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:58:08.877 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.403421+0000", 2024-09-21T10:58:08.877 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7889999999999995e-06 2024-09-21T10:58:08.877 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.877 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.878 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:58:08.878 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.403437+0000", 2024-09-21T10:58:08.878 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5977e-05 2024-09-21T10:58:08.878 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.878 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.878 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:58:08.878 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.403446+0000", 2024-09-21T10:58:08.878 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6570000000000003e-06 2024-09-21T10:58:08.878 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.878 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.879 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.879 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.410837+0000", 2024-09-21T10:58:08.879 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0073913859999999998 2024-09-21T10:58:08.879 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.879 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.879 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:58:08.879 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.413854+0000", 2024-09-21T10:58:08.879 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0030171859999999998 2024-09-21T10:58:08.879 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.879 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.879 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.879 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.458342+0000", 2024-09-21T10:58:08.879 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.044487421999999999 2024-09-21T10:58:08.879 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.880 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.880 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.880 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.458353+0000", 2024-09-21T10:58:08.880 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1004e-05 2024-09-21T10:58:08.880 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.880 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.880 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.880 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.429161+0000", 2024-09-21T10:58:08.880 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.97080803299999996 2024-09-21T10:58:08.880 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.880 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.880 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.880 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.429169+0000", 2024-09-21T10:58:08.881 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2199999999999992e-06 2024-09-21T10:58:08.881 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.881 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.881 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.881 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.757697+0000", 2024-09-21T10:58:08.881 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.328527934 2024-09-21T10:58:08.881 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.881 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.881 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:58:08.881 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.758034+0000", 2024-09-21T10:58:08.881 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00033750499999999999 2024-09-21T10:58:08.881 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.881 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.881 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.882 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.778950+0000", 2024-09-21T10:58:08.882 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.020915321000000001 2024-09-21T10:58:08.882 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.882 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.882 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:58:08.882 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.779056+0000", 2024-09-21T10:58:08.882 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010603999999999999 2024-09-21T10:58:08.882 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.882 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.882 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,3,6", 2024-09-21T10:58:08.882 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.779524+0000", 2024-09-21T10:58:08.882 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00046834399999999999 2024-09-21T10:58:08.882 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.883 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.883 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.883 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.809552+0000", 2024-09-21T10:58:08.883 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.030028368999999999 2024-09-21T10:58:08.883 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.883 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.883 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:58:08.883 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.809565+0000", 2024-09-21T10:58:08.883 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2473000000000001e-05 2024-09-21T10:58:08.883 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.883 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.883 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.883 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.809594+0000", 2024-09-21T10:58:08.884 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9169000000000001e-05 2024-09-21T10:58:08.884 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.884 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.884 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.884 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.809627+0000", 2024-09-21T10:58:08.884 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2666999999999997e-05 2024-09-21T10:58:08.884 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.884 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.884 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:58:08.884 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.809881+0000", 2024-09-21T10:58:08.884 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00025474100000000001 2024-09-21T10:58:08.884 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.884 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.885 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:58:08.885 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.821928+0000", 2024-09-21T10:58:08.885 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012046609 2024-09-21T10:58:08.885 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.885 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:58:08.885 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.885 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.885 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.885 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:130868 55.3 55:c060ea53:::benchmark_data_smithi070_294394_object25527:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:58:08.885 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:26.414944+0000", 2024-09-21T10:58:08.885 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 402.40843509199999, 2024-09-21T10:58:08.885 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4176187740000001, 2024-09-21T10:58:08.885 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:58:08.886 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:58:08.886 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:58:08.886 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:58:08.886 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:58:08.886 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:58:08.886 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 130868 2024-09-21T10:58:08.886 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.886 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:58:08.886 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.886 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:58:08.886 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.414944+0000", 2024-09-21T10:58:08.886 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.887 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.887 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.887 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:58:08.887 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.414944+0000", 2024-09-21T10:58:08.887 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.887 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.887 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.887 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:58:08.887 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.414946+0000", 2024-09-21T10:58:08.887 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.821e-06 2024-09-21T10:58:08.887 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.887 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.887 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:58:08.887 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.414953+0000", 2024-09-21T10:58:08.888 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5800000000000003e-06 2024-09-21T10:58:08.888 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.888 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.888 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:58:08.888 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.414972+0000", 2024-09-21T10:58:08.888 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8644e-05 2024-09-21T10:58:08.888 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.888 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.888 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:58:08.888 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.414980+0000", 2024-09-21T10:58:08.888 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6009999999999995e-06 2024-09-21T10:58:08.888 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.888 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.888 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.889 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.415312+0000", 2024-09-21T10:58:08.889 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00033163099999999998 2024-09-21T10:58:08.889 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.889 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.889 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:58:08.889 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.418842+0000", 2024-09-21T10:58:08.889 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035301970000000001 2024-09-21T10:58:08.889 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.889 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.889 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.889 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.459012+0000", 2024-09-21T10:58:08.889 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.040169336 2024-09-21T10:58:08.889 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.889 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.890 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.890 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.459022+0000", 2024-09-21T10:58:08.890 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0879999999999999e-05 2024-09-21T10:58:08.890 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.890 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.890 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.890 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.429197+0000", 2024-09-21T10:58:08.890 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.97017467599999996 2024-09-21T10:58:08.890 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.890 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.890 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.890 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.429205+0000", 2024-09-21T10:58:08.890 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8800000000000008e-06 2024-09-21T10:58:08.891 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.891 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.891 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.891 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.758071+0000", 2024-09-21T10:58:08.891 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.328865768 2024-09-21T10:58:08.891 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.891 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.891 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:58:08.891 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.758459+0000", 2024-09-21T10:58:08.891 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000388551 2024-09-21T10:58:08.891 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.891 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:08.893 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:08.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:08.894 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.894 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.807195+0000", 2024-09-21T10:58:08.894 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.048735607 2024-09-21T10:58:08.894 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.895 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.895 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:58:08.895 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.807294+0000", 2024-09-21T10:58:08.895 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.8789000000000006e-05 2024-09-21T10:58:08.895 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.895 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.895 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,3,6", 2024-09-21T10:58:08.895 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.807447+0000", 2024-09-21T10:58:08.895 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015366699999999999 2024-09-21T10:58:08.895 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.895 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.895 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.895 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.821997+0000", 2024-09-21T10:58:08.896 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.014549805000000001 2024-09-21T10:58:08.896 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.896 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.896 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:58:08.896 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.822015+0000", 2024-09-21T10:58:08.896 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7459e-05 2024-09-21T10:58:08.896 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.896 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.896 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.896 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.822067+0000", 2024-09-21T10:58:08.896 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1848000000000002e-05 2024-09-21T10:58:08.896 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.896 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.896 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.897 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.826902+0000", 2024-09-21T10:58:08.897 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0048356809999999997 2024-09-21T10:58:08.897 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.897 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.897 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:58:08.897 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.827095+0000", 2024-09-21T10:58:08.897 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00019322100000000001 2024-09-21T10:58:08.897 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.897 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.897 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:58:08.897 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:28.832563+0000", 2024-09-21T10:58:08.897 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0054671320000000004 2024-09-21T10:58:08.897 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.897 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:58:08.898 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.898 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.898 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.898 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:156262 55.3 55:c72c6ba0:::benchmark_data_smithi070_294394_object50921:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1464)", 2024-09-21T10:58:08.898 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.317232+0000", 2024-09-21T10:58:08.898 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 344.50614730199999, 2024-09-21T10:58:08.898 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.319640197, 2024-09-21T10:58:08.898 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:58:08.898 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:58:08.898 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:58:08.898 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:58:08.899 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:58:08.899 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:58:08.899 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 156262 2024-09-21T10:58:08.899 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.899 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:58:08.899 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.899 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:58:08.899 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.317232+0000", 2024-09-21T10:58:08.899 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.899 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.899 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.899 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:58:08.900 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.317232+0000", 2024-09-21T10:58:08.900 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.900 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.900 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.900 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:58:08.900 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.317233+0000", 2024-09-21T10:58:08.900 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.84e-06 2024-09-21T10:58:08.900 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.900 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.900 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:58:08.900 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.317240+0000", 2024-09-21T10:58:08.900 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2790000000000004e-06 2024-09-21T10:58:08.901 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.901 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.901 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:58:08.901 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.317253+0000", 2024-09-21T10:58:08.901 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.378e-05 2024-09-21T10:58:08.901 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.901 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.901 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:58:08.901 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.317261+0000", 2024-09-21T10:58:08.901 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7400000000000004e-06 2024-09-21T10:58:08.901 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.902 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.902 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.902 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.317343+0000", 2024-09-21T10:58:08.902 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1835000000000005e-05 2024-09-21T10:58:08.902 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.902 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.902 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:58:08.902 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.321143+0000", 2024-09-21T10:58:08.902 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0038001250000000001 2024-09-21T10:58:08.902 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.902 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.902 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.902 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.343392+0000", 2024-09-21T10:58:08.903 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.022248375000000001 2024-09-21T10:58:08.903 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.903 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.903 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:58:08.903 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.343496+0000", 2024-09-21T10:58:08.903 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000104279 2024-09-21T10:58:08.903 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.903 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.903 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,3,6", 2024-09-21T10:58:08.903 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.343661+0000", 2024-09-21T10:58:08.903 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016517699999999999 2024-09-21T10:58:08.903 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.903 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.904 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.904 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.355131+0000", 2024-09-21T10:58:08.904 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.011469791 2024-09-21T10:58:08.904 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.904 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.904 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.904 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.355158+0000", 2024-09-21T10:58:08.905 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6951e-05 2024-09-21T10:58:08.905 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.905 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.905 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:58:08.905 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.356176+0000", 2024-09-21T10:58:08.905 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0010179410000000001 2024-09-21T10:58:08.905 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.906 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.906 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.906 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.472799+0000", 2024-09-21T10:58:08.906 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.116623622 2024-09-21T10:58:08.906 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.906 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.906 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.906 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.472815+0000", 2024-09-21T10:58:08.906 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6058000000000001e-05 2024-09-21T10:58:08.906 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.906 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.907 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.907 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.467557+0000", 2024-09-21T10:58:08.907 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99474126100000004 2024-09-21T10:58:08.907 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.907 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.907 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.907 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.467588+0000", 2024-09-21T10:58:08.907 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1421999999999998e-05 2024-09-21T10:58:08.907 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.907 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.907 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.907 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.627831+0000", 2024-09-21T10:58:08.907 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.160243092 2024-09-21T10:58:08.908 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.908 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.908 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for missing object", 2024-09-21T10:58:08.908 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.627985+0000", 2024-09-21T10:58:08.908 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000154327 2024-09-21T10:58:08.908 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.908 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.908 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.908 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.636715+0000", 2024-09-21T10:58:08.908 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0087296089999999993 2024-09-21T10:58:08.908 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.908 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.908 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:58:08.908 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.636872+0000", 2024-09-21T10:58:08.909 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015669300000000001 2024-09-21T10:58:08.909 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.909 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:58:08.909 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.909 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.909 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.909 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:156265 55.3 55:c8ae1fd0:::benchmark_data_smithi070_294394_object50924:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1464)", 2024-09-21T10:58:08.909 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.325973+0000", 2024-09-21T10:58:08.909 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 344.49740578299998, 2024-09-21T10:58:08.909 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3149953180000002, 2024-09-21T10:58:08.909 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:58:08.909 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:58:08.909 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:58:08.909 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:58:08.910 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:58:08.910 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:58:08.910 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 156265 2024-09-21T10:58:08.910 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.910 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:58:08.910 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.910 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:58:08.910 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.325973+0000", 2024-09-21T10:58:08.910 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.910 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.910 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.910 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:58:08.911 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.325973+0000", 2024-09-21T10:58:08.911 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.911 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.911 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.911 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:58:08.911 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.325975+0000", 2024-09-21T10:58:08.911 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8470000000000001e-06 2024-09-21T10:58:08.911 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.911 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.911 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:58:08.911 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.325980+0000", 2024-09-21T10:58:08.911 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.242e-06 2024-09-21T10:58:08.911 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.911 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.912 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:58:08.912 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.325995+0000", 2024-09-21T10:58:08.912 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5055000000000001e-05 2024-09-21T10:58:08.912 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.912 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.912 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:58:08.912 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.326002+0000", 2024-09-21T10:58:08.912 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0280000000000003e-06 2024-09-21T10:58:08.912 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.912 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.912 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.912 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.326145+0000", 2024-09-21T10:58:08.912 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014283199999999999 2024-09-21T10:58:08.912 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.913 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.913 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:58:08.913 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.327892+0000", 2024-09-21T10:58:08.913 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001747352 2024-09-21T10:58:08.913 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.913 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.913 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.913 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.351485+0000", 2024-09-21T10:58:08.913 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.023592318000000001 2024-09-21T10:58:08.913 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.913 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.913 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:58:08.913 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.351567+0000", 2024-09-21T10:58:08.913 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2572999999999997e-05 2024-09-21T10:58:08.914 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.914 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.914 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,3,6", 2024-09-21T10:58:08.914 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.351690+0000", 2024-09-21T10:58:08.914 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012269599999999999 2024-09-21T10:58:08.914 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.914 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.914 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.914 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.355179+0000", 2024-09-21T10:58:08.914 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0034887949999999998 2024-09-21T10:58:08.914 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.914 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.914 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:58:08.915 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.356185+0000", 2024-09-21T10:58:08.915 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0010059999999999999 2024-09-21T10:58:08.915 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.915 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.915 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.915 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.358107+0000", 2024-09-21T10:58:08.915 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001921989 2024-09-21T10:58:08.915 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.915 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.915 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.915 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.473066+0000", 2024-09-21T10:58:08.915 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.114959478 2024-09-21T10:58:08.915 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.916 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.916 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.916 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.473076+0000", 2024-09-21T10:58:08.916 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3810000000000005e-06 2024-09-21T10:58:08.916 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.916 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.916 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.916 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.469705+0000", 2024-09-21T10:58:08.916 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99662927800000001 2024-09-21T10:58:08.916 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.916 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.916 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.916 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.469716+0000", 2024-09-21T10:58:08.917 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0906e-05 2024-09-21T10:58:08.917 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.917 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.917 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.917 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.629933+0000", 2024-09-21T10:58:08.917 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1602172040000001 2024-09-21T10:58:08.917 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.917 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.917 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for missing object", 2024-09-21T10:58:08.917 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.630738+0000", 2024-09-21T10:58:08.917 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00080527400000000003 2024-09-21T10:58:08.917 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.917 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.918 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.918 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.640869+0000", 2024-09-21T10:58:08.918 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.01013057 2024-09-21T10:58:08.918 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.918 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.918 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:58:08.918 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.640968+0000", 2024-09-21T10:58:08.918 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9500000000000006e-05 2024-09-21T10:58:08.918 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.918 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:58:08.918 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.918 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.918 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.919 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:156289 55.3 55:cca4d5e4:::benchmark_data_smithi070_294394_object50948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1464)", 2024-09-21T10:58:08.919 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.379748+0000", 2024-09-21T10:58:08.919 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 344.44363076399998, 2024-09-21T10:58:08.919 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2985970020000002, 2024-09-21T10:58:08.919 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:58:08.919 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:58:08.919 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:58:08.919 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:58:08.919 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:58:08.919 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:58:08.919 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 156289 2024-09-21T10:58:08.919 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.919 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:58:08.919 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.920 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:58:08.920 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.379748+0000", 2024-09-21T10:58:08.920 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.920 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.920 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.920 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:58:08.920 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.379748+0000", 2024-09-21T10:58:08.920 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.920 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.920 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.920 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:58:08.920 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.379750+0000", 2024-09-21T10:58:08.920 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.922e-06 2024-09-21T10:58:08.921 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.921 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.921 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:58:08.921 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.379756+0000", 2024-09-21T10:58:08.921 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9739999999999999e-06 2024-09-21T10:58:08.921 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.921 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.921 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:58:08.921 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.379773+0000", 2024-09-21T10:58:08.921 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7363e-05 2024-09-21T10:58:08.921 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.921 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.921 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:58:08.921 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.379781+0000", 2024-09-21T10:58:08.921 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7959999999999996e-06 2024-09-21T10:58:08.922 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.922 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.922 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.922 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.379846+0000", 2024-09-21T10:58:08.922 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4603000000000005e-05 2024-09-21T10:58:08.922 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.922 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.922 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:58:08.922 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.380542+0000", 2024-09-21T10:58:08.922 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00069589599999999999 2024-09-21T10:58:08.922 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.923 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.923 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.923 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.473103+0000", 2024-09-21T10:58:08.923 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.092561056000000003 2024-09-21T10:58:08.923 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.923 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.923 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.923 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.473109+0000", 2024-09-21T10:58:08.923 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1829999999999999e-06 2024-09-21T10:58:08.923 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.923 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.923 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.923 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.469746+0000", 2024-09-21T10:58:08.923 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99663669200000005 2024-09-21T10:58:08.924 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.924 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.924 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.924 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.469755+0000", 2024-09-21T10:58:08.924 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5699999999999999e-06 2024-09-21T10:58:08.924 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.924 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.924 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.924 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.630804+0000", 2024-09-21T10:58:08.924 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1610483920000001 2024-09-21T10:58:08.924 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.924 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.924 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:58:08.924 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.631711+0000", 2024-09-21T10:58:08.925 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00090767300000000001 2024-09-21T10:58:08.925 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.925 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.925 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.925 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.646421+0000", 2024-09-21T10:58:08.925 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.014710246999999999 2024-09-21T10:58:08.925 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.925 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.925 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:58:08.925 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.646528+0000", 2024-09-21T10:58:08.925 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010703400000000001 2024-09-21T10:58:08.925 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.925 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.925 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,3,7", 2024-09-21T10:58:08.926 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.646672+0000", 2024-09-21T10:58:08.926 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014363599999999999 2024-09-21T10:58:08.926 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.926 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.926 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.926 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.659293+0000", 2024-09-21T10:58:08.926 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012620726000000001 2024-09-21T10:58:08.926 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.926 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.926 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.926 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.659342+0000", 2024-09-21T10:58:08.926 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9116000000000002e-05 2024-09-21T10:58:08.926 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.927 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.927 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.927 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.670514+0000", 2024-09-21T10:58:08.927 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.011172100000000001 2024-09-21T10:58:08.927 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.927 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.927 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:58:08.927 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.676561+0000", 2024-09-21T10:58:08.927 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.006047251 2024-09-21T10:58:08.927 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.927 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.927 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:58:08.927 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.676602+0000", 2024-09-21T10:58:08.927 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0967999999999998e-05 2024-09-21T10:58:08.928 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.928 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.928 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:58:08.928 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.678345+0000", 2024-09-21T10:58:08.928 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0017428039999999999 2024-09-21T10:58:08.928 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.928 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:58:08.928 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.928 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.928 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.928 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:156293 55.3 55:c7ab3785:::benchmark_data_smithi070_294394_object50952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1464)", 2024-09-21T10:58:08.928 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.387365+0000", 2024-09-21T10:58:08.928 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 344.43601411600002, 2024-09-21T10:58:08.928 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2933115700000002, 2024-09-21T10:58:08.929 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:58:08.929 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:58:08.929 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:58:08.929 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:58:08.929 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:58:08.929 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:58:08.929 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 156293 2024-09-21T10:58:08.929 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.929 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:58:08.929 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.929 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:58:08.929 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.387365+0000", 2024-09-21T10:58:08.929 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.930 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.930 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.930 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:58:08.930 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.387365+0000", 2024-09-21T10:58:08.930 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.930 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.930 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.930 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:58:08.930 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.387366+0000", 2024-09-21T10:58:08.930 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.457e-06 2024-09-21T10:58:08.930 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.930 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.930 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:58:08.931 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.387379+0000", 2024-09-21T10:58:08.931 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3186999999999999e-05 2024-09-21T10:58:08.931 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.931 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.931 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:58:08.931 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.387390+0000", 2024-09-21T10:58:08.931 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0549e-05 2024-09-21T10:58:08.931 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.931 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.931 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:58:08.931 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.387398+0000", 2024-09-21T10:58:08.931 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0450000000000001e-06 2024-09-21T10:58:08.931 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.932 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.932 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.932 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.387455+0000", 2024-09-21T10:58:08.932 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6753999999999999e-05 2024-09-21T10:58:08.932 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.932 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.932 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:58:08.932 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.388264+0000", 2024-09-21T10:58:08.932 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00080899100000000001 2024-09-21T10:58:08.932 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.933 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.933 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.933 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.473515+0000", 2024-09-21T10:58:08.933 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.085251407000000001 2024-09-21T10:58:08.933 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.933 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.933 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.933 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.473524+0000", 2024-09-21T10:58:08.933 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9970000000000004e-06 2024-09-21T10:58:08.933 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.933 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.933 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.933 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.469787+0000", 2024-09-21T10:58:08.933 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99626246799999996 2024-09-21T10:58:08.934 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.934 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.934 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.934 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.469794+0000", 2024-09-21T10:58:08.934 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.272e-06 2024-09-21T10:58:08.934 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.934 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.934 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.934 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.651119+0000", 2024-09-21T10:58:08.934 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1813249480000001 2024-09-21T10:58:08.934 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.934 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.934 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:58:08.934 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.651533+0000", 2024-09-21T10:58:08.935 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00041398299999999998 2024-09-21T10:58:08.935 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.935 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.935 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.935 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.663514+0000", 2024-09-21T10:58:08.935 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.011981466 2024-09-21T10:58:08.935 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.935 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.935 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:58:08.935 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.663604+0000", 2024-09-21T10:58:08.935 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9920000000000006e-05 2024-09-21T10:58:08.935 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.935 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.935 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,3,7", 2024-09-21T10:58:08.936 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.663726+0000", 2024-09-21T10:58:08.936 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012202899999999999 2024-09-21T10:58:08.936 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.936 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.936 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.936 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.670454+0000", 2024-09-21T10:58:08.936 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0067277719999999999 2024-09-21T10:58:08.936 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.936 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.936 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:58:08.936 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.679853+0000", 2024-09-21T10:58:08.936 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0093995080000000009 2024-09-21T10:58:08.936 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.936 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.937 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.937 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.679896+0000", 2024-09-21T10:58:08.937 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2920999999999998e-05 2024-09-21T10:58:08.937 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.937 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.937 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.937 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.679929+0000", 2024-09-21T10:58:08.937 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3009000000000002e-05 2024-09-21T10:58:08.937 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.937 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.937 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:58:08.937 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.679964+0000", 2024-09-21T10:58:08.937 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5077999999999999e-05 2024-09-21T10:58:08.937 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.938 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.938 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:58:08.938 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.680676+0000", 2024-09-21T10:58:08.938 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00071180899999999999 2024-09-21T10:58:08.938 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.938 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:58:08.938 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.938 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.938 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.938 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:156301 55.3 55:cbe6e631:::benchmark_data_smithi070_294394_object50960:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1465)", 2024-09-21T10:58:08.938 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.500014+0000", 2024-09-21T10:58:08.938 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 344.32336524300001, 2024-09-21T10:58:08.938 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2107678850000001, 2024-09-21T10:58:08.938 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:58:08.939 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:58:08.939 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:58:08.939 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:58:08.939 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:58:08.939 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:58:08.939 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 156301 2024-09-21T10:58:08.939 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.939 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:58:08.939 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.939 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:58:08.939 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.500014+0000", 2024-09-21T10:58:08.939 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.939 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.939 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.940 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:58:08.940 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.500014+0000", 2024-09-21T10:58:08.940 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.940 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.940 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.940 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:58:08.940 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.500016+0000", 2024-09-21T10:58:08.940 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.751e-06 2024-09-21T10:58:08.940 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.940 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.940 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:58:08.940 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.500026+0000", 2024-09-21T10:58:08.940 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4879999999999996e-06 2024-09-21T10:58:08.940 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.941 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.941 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:58:08.941 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.500048+0000", 2024-09-21T10:58:08.941 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2322000000000001e-05 2024-09-21T10:58:08.941 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.941 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.941 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:58:08.941 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.500065+0000", 2024-09-21T10:58:08.941 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6716000000000001e-05 2024-09-21T10:58:08.941 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.941 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.941 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.941 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.500146+0000", 2024-09-21T10:58:08.941 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1501000000000005e-05 2024-09-21T10:58:08.942 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.942 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.942 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.942 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.500158+0000", 2024-09-21T10:58:08.942 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1688000000000001e-05 2024-09-21T10:58:08.942 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.942 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.942 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.942 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.469817+0000", 2024-09-21T10:58:08.942 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.96965923899999995 2024-09-21T10:58:08.942 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.942 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.942 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.943 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.469823+0000", 2024-09-21T10:58:08.943 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4999999999999999e-06 2024-09-21T10:58:08.943 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.943 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.943 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.943 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.651577+0000", 2024-09-21T10:58:08.943 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.181754054 2024-09-21T10:58:08.943 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.943 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.943 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:58:08.943 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.654453+0000", 2024-09-21T10:58:08.943 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0028760859999999999 2024-09-21T10:58:08.943 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.944 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.944 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.944 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.673839+0000", 2024-09-21T10:58:08.944 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.019386417999999999 2024-09-21T10:58:08.944 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.944 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.944 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:58:08.944 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.673952+0000", 2024-09-21T10:58:08.944 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000112693 2024-09-21T10:58:08.944 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.944 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.944 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,3,7", 2024-09-21T10:58:08.944 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.674052+0000", 2024-09-21T10:58:08.944 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9528e-05 2024-09-21T10:58:08.945 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.945 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.945 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:58:08.945 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.679862+0000", 2024-09-21T10:58:08.945 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0058103829999999997 2024-09-21T10:58:08.945 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.945 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.945 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.945 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.680729+0000", 2024-09-21T10:58:08.945 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00086661599999999996 2024-09-21T10:58:08.945 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.945 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.945 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.945 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.680786+0000", 2024-09-21T10:58:08.946 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7385e-05 2024-09-21T10:58:08.946 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.946 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.946 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.946 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.708516+0000", 2024-09-21T10:58:08.946 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.027730138000000001 2024-09-21T10:58:08.946 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.946 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.946 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:58:08.946 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.708587+0000", 2024-09-21T10:58:08.946 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0601999999999999e-05 2024-09-21T10:58:08.946 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.946 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.947 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:58:08.947 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.710781+0000", 2024-09-21T10:58:08.947 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0021947770000000002 2024-09-21T10:58:08.947 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.947 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:58:08.947 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.947 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.947 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.947 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:156302 55.3 55:c0dc2955:::benchmark_data_smithi070_294394_object50961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1465)", 2024-09-21T10:58:08.947 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.501061+0000", 2024-09-21T10:58:08.947 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 344.32231824299998, 2024-09-21T10:58:08.947 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2119707480000002, 2024-09-21T10:58:08.947 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:58:08.947 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:58:08.948 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:58:08.948 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:58:08.948 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:58:08.948 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:58:08.948 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 156302 2024-09-21T10:58:08.948 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.948 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:58:08.948 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.948 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:58:08.948 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.501061+0000", 2024-09-21T10:58:08.948 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.948 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.948 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.949 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:58:08.949 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.501061+0000", 2024-09-21T10:58:08.949 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.949 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.949 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.949 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:58:08.949 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.501062+0000", 2024-09-21T10:58:08.949 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6020000000000001e-06 2024-09-21T10:58:08.949 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.949 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.949 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:58:08.949 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.501069+0000", 2024-09-21T10:58:08.949 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6889999999999998e-06 2024-09-21T10:58:08.949 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.949 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.950 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:58:08.950 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.501088+0000", 2024-09-21T10:58:08.950 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9321999999999999e-05 2024-09-21T10:58:08.950 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.950 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.950 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:58:08.950 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.501099+0000", 2024-09-21T10:58:08.950 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.075e-05 2024-09-21T10:58:08.950 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.950 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.950 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.950 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.501169+0000", 2024-09-21T10:58:08.950 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9797999999999998e-05 2024-09-21T10:58:08.951 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.951 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.951 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.951 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.501177+0000", 2024-09-21T10:58:08.951 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.568e-06 2024-09-21T10:58:08.951 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.951 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.951 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.951 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.469845+0000", 2024-09-21T10:58:08.951 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.96866734499999996 2024-09-21T10:58:08.951 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.951 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.951 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.951 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.469851+0000", 2024-09-21T10:58:08.952 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1619999999999996e-06 2024-09-21T10:58:08.952 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.952 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.952 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.952 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.656700+0000", 2024-09-21T10:58:08.952 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1868494380000001 2024-09-21T10:58:08.952 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.952 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.952 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T10:58:08.952 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.659223+0000", 2024-09-21T10:58:08.952 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025225790000000001 2024-09-21T10:58:08.952 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.952 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.952 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.953 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.680821+0000", 2024-09-21T10:58:08.953 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.021597813 2024-09-21T10:58:08.953 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.953 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.953 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:58:08.953 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.680881+0000", 2024-09-21T10:58:08.953 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0291999999999998e-05 2024-09-21T10:58:08.953 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.953 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.953 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,3,7", 2024-09-21T10:58:08.953 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.680964+0000", 2024-09-21T10:58:08.953 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3250000000000004e-05 2024-09-21T10:58:08.953 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.953 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.954 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.954 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.684121+0000", 2024-09-21T10:58:08.954 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0031563950000000002 2024-09-21T10:58:08.954 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.954 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.954 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:58:08.954 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.684139+0000", 2024-09-21T10:58:08.954 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8797999999999999e-05 2024-09-21T10:58:08.954 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.954 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.954 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.954 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.684744+0000", 2024-09-21T10:58:08.954 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00060464399999999999 2024-09-21T10:58:08.955 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.955 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.955 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.955 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.710836+0000", 2024-09-21T10:58:08.955 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.026091849 2024-09-21T10:58:08.955 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.955 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.955 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:58:08.955 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.710897+0000", 2024-09-21T10:58:08.955 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1260000000000006e-05 2024-09-21T10:58:08.955 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.955 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.956 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:58:08.956 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.713031+0000", 2024-09-21T10:58:08.956 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0021341939999999998 2024-09-21T10:58:08.956 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.956 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:58:08.956 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.956 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.956 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.956 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36012.0:66793 56.a 56:5ae926c5:::benchmark_data_smithi070_333437_object66792:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1490)", 2024-09-21T10:58:08.956 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:55:00.441940+0000", 2024-09-21T10:58:08.956 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 188.38143913600001, 2024-09-21T10:58:08.956 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2963067540000002, 2024-09-21T10:58:08.956 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:58:08.956 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:58:08.957 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:58:08.957 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:58:08.957 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36012", 2024-09-21T10:58:08.957 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T10:58:08.957 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 66793 2024-09-21T10:58:08.957 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.957 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:58:08.957 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.957 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:58:08.957 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.441940+0000", 2024-09-21T10:58:08.957 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.957 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.957 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.957 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:58:08.958 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.441940+0000", 2024-09-21T10:58:08.958 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.958 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.958 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.958 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:58:08.958 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.441941+0000", 2024-09-21T10:58:08.958 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6220000000000001e-06 2024-09-21T10:58:08.958 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.958 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.958 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:58:08.958 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.441963+0000", 2024-09-21T10:58:08.958 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1175999999999999e-05 2024-09-21T10:58:08.958 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.959 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.959 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:58:08.959 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.441974+0000", 2024-09-21T10:58:08.959 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1545999999999999e-05 2024-09-21T10:58:08.959 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.959 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.959 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:58:08.959 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.441981+0000", 2024-09-21T10:58:08.959 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5010000000000001e-06 2024-09-21T10:58:08.959 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.960 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.960 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.960 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442031+0000", 2024-09-21T10:58:08.960 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0609000000000003e-05 2024-09-21T10:58:08.960 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.960 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.960 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:58:08.961 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442166+0000", 2024-09-21T10:58:08.961 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013523100000000001 2024-09-21T10:58:08.961 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.961 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.961 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:58:08.961 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442300+0000", 2024-09-21T10:58:08.961 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000133523 2024-09-21T10:58:08.961 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.961 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.962 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:58:08.962 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.444686+0000", 2024-09-21T10:58:08.962 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002386113 2024-09-21T10:58:08.962 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.962 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.962 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.962 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.456085+0000", 2024-09-21T10:58:08.962 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.011398508999999999 2024-09-21T10:58:08.962 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.962 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.962 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.962 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.456098+0000", 2024-09-21T10:58:08.962 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.34e-05 2024-09-21T10:58:08.963 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.963 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.963 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.963 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:01.465063+0000", 2024-09-21T10:58:08.963 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.008965248 2024-09-21T10:58:08.963 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.963 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.963 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.963 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:01.465079+0000", 2024-09-21T10:58:08.963 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6212e-05 2024-09-21T10:58:08.963 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.963 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.963 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.963 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.737218+0000", 2024-09-21T10:58:08.964 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.272138939 2024-09-21T10:58:08.964 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.964 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.964 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:58:08.964 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.738246+0000", 2024-09-21T10:58:08.964 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001028125 2024-09-21T10:58:08.964 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.964 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:58:08.964 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.964 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.964 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.964 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36012.0:66795 56.c 56:3e9cd853:::benchmark_data_smithi070_333437_object66794:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1490)", 2024-09-21T10:58:08.965 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:55:00.443235+0000", 2024-09-21T10:58:08.965 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 188.380143401, 2024-09-21T10:58:08.965 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.294099739, 2024-09-21T10:58:08.965 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:58:08.965 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:58:08.965 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:58:08.965 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:58:08.965 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36012", 2024-09-21T10:58:08.965 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T10:58:08.965 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 66795 2024-09-21T10:58:08.965 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.965 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:58:08.965 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.965 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:58:08.966 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.443235+0000", 2024-09-21T10:58:08.966 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.966 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.966 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.966 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:58:08.966 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.443235+0000", 2024-09-21T10:58:08.966 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.966 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.966 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.966 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:58:08.966 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.443237+0000", 2024-09-21T10:58:08.966 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2240000000000001e-06 2024-09-21T10:58:08.966 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.967 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.967 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:58:08.967 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.443260+0000", 2024-09-21T10:58:08.967 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3031999999999999e-05 2024-09-21T10:58:08.967 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.967 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.967 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:58:08.967 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.443274+0000", 2024-09-21T10:58:08.967 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4292e-05 2024-09-21T10:58:08.967 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.967 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.967 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:58:08.967 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.443281+0000", 2024-09-21T10:58:08.967 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2810000000000003e-06 2024-09-21T10:58:08.968 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.968 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.968 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.968 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.443338+0000", 2024-09-21T10:58:08.968 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7117000000000002e-05 2024-09-21T10:58:08.968 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.968 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.968 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:58:08.968 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.443471+0000", 2024-09-21T10:58:08.968 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013253700000000001 2024-09-21T10:58:08.968 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.968 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.968 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:58:08.969 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.443591+0000", 2024-09-21T10:58:08.969 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000120086 2024-09-21T10:58:08.969 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.969 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.969 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:58:08.969 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.446356+0000", 2024-09-21T10:58:08.969 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0027653320000000001 2024-09-21T10:58:08.969 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.969 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.969 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.969 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.463406+0000", 2024-09-21T10:58:08.969 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.017049877000000001 2024-09-21T10:58:08.969 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.969 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.969 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.970 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.463418+0000", 2024-09-21T10:58:08.970 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1858000000000001e-05 2024-09-21T10:58:08.970 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.970 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.970 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.970 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:01.470904+0000", 2024-09-21T10:58:08.970 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.007485937 2024-09-21T10:58:08.970 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.970 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.970 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.970 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:01.470919+0000", 2024-09-21T10:58:08.970 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4938e-05 2024-09-21T10:58:08.970 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.971 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.971 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.971 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.737240+0000", 2024-09-21T10:58:08.971 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2663211759999999 2024-09-21T10:58:08.971 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.971 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.971 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:58:08.971 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.737335+0000", 2024-09-21T10:58:08.971 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5051999999999997e-05 2024-09-21T10:58:08.971 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.971 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:58:08.971 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.971 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.971 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.972 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36012.0:66798 56.4 56:274b29bb:::benchmark_data_smithi070_333437_object66797:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1491)", 2024-09-21T10:58:08.972 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:55:00.456463+0000", 2024-09-21T10:58:08.972 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 188.36691609900001, 2024-09-21T10:58:08.972 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.235265987, 2024-09-21T10:58:08.972 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:58:08.972 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:58:08.972 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:58:08.972 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:58:08.972 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36012", 2024-09-21T10:58:08.972 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T10:58:08.972 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 66798 2024-09-21T10:58:08.972 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.972 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:58:08.972 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.973 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:58:08.973 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.456463+0000", 2024-09-21T10:58:08.973 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.973 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.973 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.973 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:58:08.973 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.456463+0000", 2024-09-21T10:58:08.973 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.973 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.973 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.973 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:58:08.973 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.456465+0000", 2024-09-21T10:58:08.973 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0159999999999998e-06 2024-09-21T10:58:08.974 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.974 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.974 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:58:08.974 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.456493+0000", 2024-09-21T10:58:08.974 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8039e-05 2024-09-21T10:58:08.974 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.974 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.974 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:58:08.974 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.456506+0000", 2024-09-21T10:58:08.974 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2782000000000001e-05 2024-09-21T10:58:08.974 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.974 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.974 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:58:08.975 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.456515+0000", 2024-09-21T10:58:08.975 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3780000000000007e-06 2024-09-21T10:58:08.975 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.975 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.975 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.975 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.463446+0000", 2024-09-21T10:58:08.975 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0069313090000000001 2024-09-21T10:58:08.975 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.975 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.975 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.975 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.463453+0000", 2024-09-21T10:58:08.976 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9739999999999996e-06 2024-09-21T10:58:08.976 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.976 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.976 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.976 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:01.470982+0000", 2024-09-21T10:58:08.976 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0075282860000001 2024-09-21T10:58:08.976 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.976 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.976 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.976 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:01.470990+0000", 2024-09-21T10:58:08.976 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.4579999999999993e-06 2024-09-21T10:58:08.976 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.976 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.976 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.977 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.666540+0000", 2024-09-21T10:58:08.977 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.195550052 2024-09-21T10:58:08.977 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.977 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.977 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:58:08.977 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.668836+0000", 2024-09-21T10:58:08.977 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0022957759999999998 2024-09-21T10:58:08.977 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.977 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.977 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:58:08.977 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.669242+0000", 2024-09-21T10:58:08.977 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00040637300000000002 2024-09-21T10:58:08.977 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.977 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.978 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.978 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.677177+0000", 2024-09-21T10:58:08.978 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0079349490000000002 2024-09-21T10:58:08.978 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.978 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.978 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:58:08.978 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.677192+0000", 2024-09-21T10:58:08.978 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4386000000000001e-05 2024-09-21T10:58:08.978 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.978 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.978 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.978 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.686919+0000", 2024-09-21T10:58:08.978 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0097278350000000006 2024-09-21T10:58:08.978 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.979 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.979 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:58:08.979 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.687438+0000", 2024-09-21T10:58:08.979 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00051895100000000003 2024-09-21T10:58:08.979 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.979 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.979 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:58:08.979 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.691729+0000", 2024-09-21T10:58:08.979 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0042904229999999998 2024-09-21T10:58:08.979 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.979 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:58:08.979 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.980 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.980 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.980 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36012.0:66804 56.a 56:5356f30b:::benchmark_data_smithi070_333437_object66803:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1491)", 2024-09-21T10:58:08.980 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:55:00.459423+0000", 2024-09-21T10:58:08.980 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 188.363955495, 2024-09-21T10:58:08.980 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3126952780000001, 2024-09-21T10:58:08.980 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:58:08.980 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:58:08.980 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:58:08.980 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:58:08.980 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36012", 2024-09-21T10:58:08.980 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T10:58:08.980 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 66804 2024-09-21T10:58:08.981 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.981 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:58:08.981 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.981 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:58:08.981 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.459423+0000", 2024-09-21T10:58:08.981 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.981 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.981 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.981 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:58:08.981 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.459423+0000", 2024-09-21T10:58:08.981 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.981 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.981 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.981 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:58:08.982 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.459425+0000", 2024-09-21T10:58:08.982 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7859999999999999e-06 2024-09-21T10:58:08.982 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.982 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.982 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:58:08.982 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.459567+0000", 2024-09-21T10:58:08.982 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014171999999999999 2024-09-21T10:58:08.982 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.982 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.982 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:58:08.982 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.459580+0000", 2024-09-21T10:58:08.982 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3046e-05 2024-09-21T10:58:08.982 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.982 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.983 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:58:08.983 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.459588+0000", 2024-09-21T10:58:08.983 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0460000000000006e-06 2024-09-21T10:58:08.983 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.983 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.983 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.983 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.461456+0000", 2024-09-21T10:58:08.983 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0018677710000000001 2024-09-21T10:58:08.983 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.983 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.983 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.984 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.461467+0000", 2024-09-21T10:58:08.984 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1494999999999999e-05 2024-09-21T10:58:08.984 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.984 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.984 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.984 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:01.467596+0000", 2024-09-21T10:58:08.984 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0061288740000001 2024-09-21T10:58:08.984 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.984 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.984 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.985 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:01.467608+0000", 2024-09-21T10:58:08.985 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2391e-05 2024-09-21T10:58:08.985 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.985 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.985 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.985 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.741242+0000", 2024-09-21T10:58:08.985 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2736333719999999 2024-09-21T10:58:08.985 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.985 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.985 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:58:08.985 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.741388+0000", 2024-09-21T10:58:08.985 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000146401 2024-09-21T10:58:08.986 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.986 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.986 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:58:08.986 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.741527+0000", 2024-09-21T10:58:08.986 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013844000000000001 2024-09-21T10:58:08.986 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.986 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.986 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:58:08.987 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.746459+0000", 2024-09-21T10:58:08.987 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0049319979999999999 2024-09-21T10:58:08.987 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.987 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.987 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.987 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.749038+0000", 2024-09-21T10:58:08.987 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025793460000000002 2024-09-21T10:58:08.987 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.987 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.987 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.987 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.768408+0000", 2024-09-21T10:58:08.988 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.019369938 2024-09-21T10:58:08.988 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.988 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.988 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:58:08.988 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.768462+0000", 2024-09-21T10:58:08.988 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4128000000000001e-05 2024-09-21T10:58:08.988 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.988 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.988 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:58:08.988 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.772119+0000", 2024-09-21T10:58:08.988 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0036565259999999998 2024-09-21T10:58:08.989 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.989 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:58:08.989 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.989 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.989 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.989 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36012.0:66813 56.4 56:2cb9f342:::benchmark_data_smithi070_333437_object66812:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1491)", 2024-09-21T10:58:08.989 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:55:00.480972+0000", 2024-09-21T10:58:08.989 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 188.34240687799999, 2024-09-21T10:58:08.989 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.214888475, 2024-09-21T10:58:08.989 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:58:08.989 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:58:08.989 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:58:08.989 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:58:08.990 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36012", 2024-09-21T10:58:08.990 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T10:58:08.990 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 66813 2024-09-21T10:58:08.990 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.990 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:58:08.990 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.990 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:58:08.990 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.480972+0000", 2024-09-21T10:58:08.990 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.990 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.990 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.990 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:58:08.990 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.480972+0000", 2024-09-21T10:58:08.991 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.991 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.991 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.991 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:58:08.991 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.480975+0000", 2024-09-21T10:58:08.991 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3960000000000002e-06 2024-09-21T10:58:08.991 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.991 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.991 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:58:08.991 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.481022+0000", 2024-09-21T10:58:08.991 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6669999999999999e-05 2024-09-21T10:58:08.991 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.991 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.991 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:58:08.992 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.481045+0000", 2024-09-21T10:58:08.992 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2982000000000001e-05 2024-09-21T10:58:08.992 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.992 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.992 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:58:08.992 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.481057+0000", 2024-09-21T10:58:08.992 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1738e-05 2024-09-21T10:58:08.992 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.992 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.992 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.992 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.481126+0000", 2024-09-21T10:58:08.992 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8758999999999997e-05 2024-09-21T10:58:08.992 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.992 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.992 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.993 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.481135+0000", 2024-09-21T10:58:08.993 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0569999999999999e-06 2024-09-21T10:58:08.993 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.993 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.993 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.993 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:01.471017+0000", 2024-09-21T10:58:08.993 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.98988205799999995 2024-09-21T10:58:08.993 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.993 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.993 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:08.993 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:01.471024+0000", 2024-09-21T10:58:08.993 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6210000000000004e-06 2024-09-21T10:58:08.993 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.993 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.993 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:08.994 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.677215+0000", 2024-09-21T10:58:08.994 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2061908749999999 2024-09-21T10:58:08.994 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.994 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.994 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:58:08.994 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.677382+0000", 2024-09-21T10:58:08.994 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016719900000000001 2024-09-21T10:58:08.994 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.994 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.994 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:58:08.994 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.677556+0000", 2024-09-21T10:58:08.994 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017326700000000001 2024-09-21T10:58:08.994 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.995 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.995 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:58:08.995 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.680555+0000", 2024-09-21T10:58:08.995 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0029997320000000002 2024-09-21T10:58:08.995 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.995 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.995 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.995 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.680834+0000", 2024-09-21T10:58:08.995 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00027831299999999998 2024-09-21T10:58:08.995 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.995 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.995 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:08.995 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.691838+0000", 2024-09-21T10:58:08.995 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.011004308000000001 2024-09-21T10:58:08.995 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.996 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.996 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:58:08.996 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.691896+0000", 2024-09-21T10:58:08.996 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7729999999999998e-05 2024-09-21T10:58:08.996 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.996 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.996 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:58:08.996 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.695860+0000", 2024-09-21T10:58:08.996 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0039647700000000003 2024-09-21T10:58:08.996 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.996 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:58:08.996 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:08.996 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.997 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.997 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36012.0:66817 56.c 56:3d750529:::benchmark_data_smithi070_333437_object66816:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1491)", 2024-09-21T10:58:08.997 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:55:00.485053+0000", 2024-09-21T10:58:08.997 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 188.33832537000001, 2024-09-21T10:58:08.997 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2921409619999999, 2024-09-21T10:58:08.997 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:58:08.997 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:58:08.997 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:58:08.997 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:58:08.997 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36012", 2024-09-21T10:58:08.997 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T10:58:08.997 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 66817 2024-09-21T10:58:08.997 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.998 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:58:08.998 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.998 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:58:08.998 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.485053+0000", 2024-09-21T10:58:08.998 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.998 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.998 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.998 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:58:08.998 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.485053+0000", 2024-09-21T10:58:08.998 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:08.998 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.998 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.998 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:58:08.999 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.485055+0000", 2024-09-21T10:58:08.999 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3999999999999999e-06 2024-09-21T10:58:08.999 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.999 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.999 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:58:08.999 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.485075+0000", 2024-09-21T10:58:08.999 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9805000000000001e-05 2024-09-21T10:58:08.999 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.999 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:08.999 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:58:08.999 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.485085+0000", 2024-09-21T10:58:08.999 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0373999999999999e-05 2024-09-21T10:58:08.999 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:08.999 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.000 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:58:09.000 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.485090+0000", 2024-09-21T10:58:09.000 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8269999999999997e-06 2024-09-21T10:58:09.000 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.000 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.000 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:09.000 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.485138+0000", 2024-09-21T10:58:09.000 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8529000000000002e-05 2024-09-21T10:58:09.000 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.000 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.000 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:09.000 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.485145+0000", 2024-09-21T10:58:09.000 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4520000000000002e-06 2024-09-21T10:58:09.000 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.000 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.001 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:09.001 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:01.470948+0000", 2024-09-21T10:58:09.001 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.98580295200000001 2024-09-21T10:58:09.001 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.001 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.001 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:09.001 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:01.470956+0000", 2024-09-21T10:58:09.001 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8590000000000005e-06 2024-09-21T10:58:09.001 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.001 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.001 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:09.001 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.740852+0000", 2024-09-21T10:58:09.001 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2698965719999999 2024-09-21T10:58:09.001 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.002 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.002 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:58:09.002 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.740975+0000", 2024-09-21T10:58:09.002 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000122912 2024-09-21T10:58:09.002 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.002 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.002 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:58:09.002 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.741400+0000", 2024-09-21T10:58:09.002 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00042462199999999999 2024-09-21T10:58:09.002 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.002 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.002 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:58:09.003 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.747590+0000", 2024-09-21T10:58:09.003 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0061901250000000003 2024-09-21T10:58:09.003 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.003 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.003 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:09.003 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.749098+0000", 2024-09-21T10:58:09.003 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001507752 2024-09-21T10:58:09.003 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.003 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.003 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:09.003 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.773526+0000", 2024-09-21T10:58:09.003 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.024428234 2024-09-21T10:58:09.003 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.004 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.004 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:58:09.004 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.773578+0000", 2024-09-21T10:58:09.004 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2268000000000002e-05 2024-09-21T10:58:09.004 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.004 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.004 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:58:09.004 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.777194+0000", 2024-09-21T10:58:09.004 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.003616279 2024-09-21T10:58:09.004 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:09.004 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:58:09.004 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:09.004 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.004 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.004 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36012.0:97191 56.4 56:23d4b74d:::benchmark_data_smithi070_333437_object97190:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1499)", 2024-09-21T10:58:09.005 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:55:37.698424+0000", 2024-09-21T10:58:09.005 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 151.12495504699999, 2024-09-21T10:58:09.005 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.729258682, 2024-09-21T10:58:09.005 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:58:09.005 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:58:09.005 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:58:09.005 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:58:09.005 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36012", 2024-09-21T10:58:09.005 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T10:58:09.005 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 97191 2024-09-21T10:58:09.005 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.005 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:58:09.005 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.005 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:58:09.006 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.698424+0000", 2024-09-21T10:58:09.006 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:09.006 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.006 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.006 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:58:09.006 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.698424+0000", 2024-09-21T10:58:09.006 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:58:09.006 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.006 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.006 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:58:09.006 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.698426+0000", 2024-09-21T10:58:09.006 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6919999999999998e-06 2024-09-21T10:58:09.007 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.007 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.007 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:58:09.007 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.698481+0000", 2024-09-21T10:58:09.007 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4224000000000001e-05 2024-09-21T10:58:09.007 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.007 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.007 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:58:09.007 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.698499+0000", 2024-09-21T10:58:09.007 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8474e-05 2024-09-21T10:58:09.007 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.007 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.007 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:58:09.007 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.698513+0000", 2024-09-21T10:58:09.007 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3604000000000001e-05 2024-09-21T10:58:09.008 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.008 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.008 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:09.008 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.815128+0000", 2024-09-21T10:58:09.008 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.116615566 2024-09-21T10:58:09.008 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.008 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.008 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:09.008 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.815139+0000", 2024-09-21T10:58:09.008 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0307e-05 2024-09-21T10:58:09.008 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.008 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.008 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:09.008 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:38.365220+0000", 2024-09-21T10:58:09.009 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.55008087299999997 2024-09-21T10:58:09.009 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.009 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.009 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:58:09.009 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:38.365232+0000", 2024-09-21T10:58:09.009 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2753e-05 2024-09-21T10:58:09.009 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.009 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.009 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:58:09.009 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.408400+0000", 2024-09-21T10:58:09.009 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0431676190000001 2024-09-21T10:58:09.009 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.009 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.009 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:58:09.010 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.408547+0000", 2024-09-21T10:58:09.010 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014689 2024-09-21T10:58:09.010 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.010 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.010 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:58:09.010 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.408875+0000", 2024-09-21T10:58:09.010 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00032823500000000002 2024-09-21T10:58:09.010 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.010 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.010 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:09.010 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.413776+0000", 2024-09-21T10:58:09.010 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0049010670000000003 2024-09-21T10:58:09.010 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.011 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.011 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:58:09.011 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.414038+0000", 2024-09-21T10:58:09.011 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000262377 2024-09-21T10:58:09.011 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.011 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.011 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:58:09.011 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.425603+0000", 2024-09-21T10:58:09.011 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.011564621000000001 2024-09-21T10:58:09.011 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.011 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.011 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:58:09.011 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.425664+0000", 2024-09-21T10:58:09.011 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0789999999999999e-05 2024-09-21T10:58:09.012 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:58:09.012 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:58:09.012 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:58:09.012 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.427682+0000", 2024-09-21T10:58:09.012 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0020185899999999998 2024-09-21T10:58:09.012 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:09.012 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:58:09.012 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:09.012 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:58:09.012 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:58:09.012 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:58:09.013 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:09.038 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:09.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:09.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:09.140 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:09.161 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:09.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:09.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:09.262 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:09.288 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:09.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:09.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:09.390 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:09.411 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:09.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:09.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:09.513 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:09.533 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:09.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:09.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:09.588 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:09 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:58:09.588 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:09 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:58:09.588 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:09 smithi070 ceph-mon[107385]: osdmap e1527: 8 total, 8 up, 4 in 2024-09-21T10:58:09.588 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:09 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:58:09.588 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:09 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:58:09.588 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:09 smithi070 ceph-mon[112139]: osdmap e1527: 8 total, 8 up, 4 in 2024-09-21T10:58:09.635 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:09.656 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:09.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:09.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:09.758 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:09.784 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:09.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:09.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:09 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:58:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:09 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:58:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:09 smithi160 ceph-mon[101879]: osdmap e1527: 8 total, 8 up, 4 in 2024-09-21T10:58:09.886 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:09.906 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:09.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:09.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:10.008 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:10.029 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:10.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:10.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:10.130 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:10.151 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:10.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:10.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:10.253 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:10.274 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:10.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:10.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:10.376 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:10.399 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:10.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:10.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:10.501 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:10.522 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:10.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:10.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:10.624 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:10.645 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:10.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:10.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:10.746 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:10.767 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:10.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:10.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:10.771 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:10 smithi070 ceph-mon[107385]: pgmap v2905: 99 pgs: 4 unknown, 95 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:10.772 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:10 smithi070 ceph-mon[112139]: pgmap v2905: 99 pgs: 4 unknown, 95 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:10.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:10 smithi160 ceph-mon[101879]: pgmap v2905: 99 pgs: 4 unknown, 95 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:10.869 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:10.893 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:10.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:10.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:10.995 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:11.016 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:11.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:11.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:11.118 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:11.139 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:11.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:11.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:11.241 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:11.266 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:11.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:11.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:11.368 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:11.394 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:11.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:11.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:11.496 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:11.520 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:11.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:11.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:11.529 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:58:11.622 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:11.643 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:11.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:11.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:11.744 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:11.770 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:11.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:11.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:11.872 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:11.894 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:11.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:11.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:11.996 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:12.021 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:58:12.022 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:12.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:12.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:12.124 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:12.150 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:12.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:12.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:12.252 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:12.274 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:12.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:12.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:12.376 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:12.398 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:12.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:12.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:12.500 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:12.522 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:12.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:12.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:12.623 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:12.650 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:12.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:12.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:12.723 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:12 smithi070 ceph-mon[107385]: pgmap v2906: 99 pgs: 99 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:12.723 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:12 smithi070 ceph-mon[112139]: pgmap v2906: 99 pgs: 99 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:12.752 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:12.778 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:12.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:12.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:12.792 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:58:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:12 smithi160 ceph-mon[101879]: pgmap v2906: 99 pgs: 99 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:12.880 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:12.901 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:12.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:12.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:13.003 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:13.029 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:13.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:13.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:13.131 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:13.152 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:13.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:13.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:13.182 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:58:13.183 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:58:13.199 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:58:13.199 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:58:13.215 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:58:13.215 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:58:13.231 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:58:13.231 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:58:13.248 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:58:13.248 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:58:13.248 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:58:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:58:13] "GET /metrics HTTP/1.1" 200 34203 "" "Prometheus/2.51.0" 2024-09-21T10:58:13.255 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:13.264 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:58:13.265 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:58:13.275 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:13.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:13.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:13.281 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:58:13.282 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:58:13.298 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:58:13.298 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:58:13.377 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:13.403 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:13.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:13.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:13.505 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:13.529 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:13.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:13.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:13.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:13 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/783999899' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:58:13.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:13 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/783999899' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:58:13.631 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:13.653 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:13.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:13.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:13.755 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:13.780 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:13.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:13.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:13 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/783999899' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:58:13.883 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:13.905 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:13.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:13.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:14.007 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:14.028 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:14.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:14.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:14.130 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:14.156 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:14.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:14.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:14.163 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:58:14.181 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:58:14.258 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:14.284 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:14.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:14.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:14.386 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:14.412 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:14.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:14.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:14.462 INFO:teuthology.orchestra.run.smithi070.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-09-21T10:58:14.513 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:14.534 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:14.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:14.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:14.636 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:14.662 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:14.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:14.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:14.764 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:14.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:14.787 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:14.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:14 smithi160 ceph-mon[101879]: pgmap v2907: 99 pgs: 99 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:14 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3978543129' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:58:14.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:14 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:58:14.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:14 smithi070 ceph-mon[107385]: pgmap v2907: 99 pgs: 99 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:14.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:14 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3978543129' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:58:14.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:14 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:58:14.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:14 smithi070 ceph-mon[112139]: pgmap v2907: 99 pgs: 99 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:14.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:14 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3978543129' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:58:14.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:14 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:58:14.888 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:14.914 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:14.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:14.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:15.016 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:15.038 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:15.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:15.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:15.140 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:15.161 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:15.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:15.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:15.263 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:15.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:15.290 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:15.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:15.391 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:15.417 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:15.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:15.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:15.519 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:15.542 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:15.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:15.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:15.644 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:15.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:15.666 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:15.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:15.739 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:15 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:58:15.739 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:15 smithi070 ceph-mon[112139]: osdmap e1528: 8 total, 8 up, 4 in 2024-09-21T10:58:15.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:15 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:58:15.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:15 smithi070 ceph-mon[107385]: osdmap e1528: 8 total, 8 up, 4 in 2024-09-21T10:58:15.767 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:15.788 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:15.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:15.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:15 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:58:15.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:15 smithi160 ceph-mon[101879]: osdmap e1528: 8 total, 8 up, 4 in 2024-09-21T10:58:15.890 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:15.911 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:15.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:15.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:16.012 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:16.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:16.039 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:16.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:16.132 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:58:16.140 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:16.196 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:16.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:16.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:16.297 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:16.323 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:16.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:16.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:16.425 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:16.446 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:16.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:16.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:16.548 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:16.567 DEBUG:teuthology.orchestra.run.smithi070:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-09-21T10:58:16.569 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T10:58:16.570 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T10:58:16.574 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:16.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:16.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:16.618 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:hints = 1 2024-09-21T10:58:16.619 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T10:58:16.619 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Object prefix: benchmark_data_smithi070_360420 2024-09-21T10:58:16.619 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:58:16.619 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 0 0 0 0 0 0 - 0 2024-09-21T10:58:16.676 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:16.714 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:16.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:16.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:16.728 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:16 smithi070 ceph-mon[107385]: pgmap v2909: 99 pgs: 99 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:16.728 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:16 smithi070 ceph-mon[107385]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:58:16.728 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:16 smithi070 ceph-mon[112139]: pgmap v2909: 99 pgs: 99 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:16.728 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:16 smithi070 ceph-mon[112139]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:58:16.817 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:16.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:16 smithi160 ceph-mon[101879]: pgmap v2909: 99 pgs: 99 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:16.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:16 smithi160 ceph-mon[101879]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:58:16.849 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:16.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:16.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:16.951 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:16.984 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:16.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:16.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:17.086 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:17.117 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:17.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:17.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:17.219 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:17.252 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:17.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:17.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:17.354 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:17.389 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:17.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:17.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:17.491 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:17.525 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:17.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:17.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:17.619 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 1 16 4080 4064 253.986 254 0.00532477 0.00392338 2024-09-21T10:58:17.627 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:17.663 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:17.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:17.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:17.764 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:17.799 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:17.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:17.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:17.901 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:17.935 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:17.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:17.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:18.037 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:18.068 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:18.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:18.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:18.170 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:18.204 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:18.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:18.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:18.305 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:18.341 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:18.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:18.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:18.443 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:18.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:18.476 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:18.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:18.577 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:18.607 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:18.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:18.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:18.619 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 2 16 7892 7876 246.108 238.25 0.00392291 0.00405606 2024-09-21T10:58:18.667 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:18 smithi070 ceph-mon[107385]: pgmap v2910: 99 pgs: 99 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:18.668 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:18 smithi070 ceph-mon[112139]: pgmap v2910: 99 pgs: 99 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:18.709 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:18.746 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:18.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:18.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:18.826 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:58:18.826 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:58:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:18 smithi160 ceph-mon[101879]: pgmap v2910: 99 pgs: 99 active+clean; 268 KiB data, 4.2 GiB used, 353 GiB / 358 GiB avail 2024-09-21T10:58:18.848 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:18.882 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:18.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:18.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:18.985 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:19.016 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:19.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:19.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:19.118 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:19.150 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:19.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:19.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:19.252 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:19.283 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:19.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:19.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:19.385 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:19.419 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:19.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:19.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:19.521 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:19.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:19.559 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:19.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:19.619 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 3 16 11305 11289 235.17 213.312 0.00579596 0.00424603 2024-09-21T10:58:19.660 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:19.694 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:19.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:19.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:19.796 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:19.830 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:19.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:19.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:19.931 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:19.967 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:19.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:19.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:20.068 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:20.104 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:20.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:20.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:20.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:19 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a[107362]: 2024-09-21T10:58:19.933+0000 7fad4a465640 -1 mon.a@0(leader).osd e1528 definitely_dead 0 2024-09-21T10:58:20.206 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:20.236 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:20.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:20.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:20.338 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:20.347 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:58:20.372 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:20.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:20.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:20.377 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:58:20.474 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:20.476 INFO:teuthology.orchestra.run.smithi070.stderr:marked in osd.4. 2024-09-21T10:58:20.516 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:20.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:20.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:20.617 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:20.619 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 4 16 14083 14067 219.78 173.625 0.00213447 0.00439888 2024-09-21T10:58:20.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:20 smithi070 ceph-mon[107385]: pgmap v2911: 99 pgs: 99 active+clean; 120 MiB data, 4.2 GiB used, 353 GiB / 358 GiB avail; 12 MiB/s wr, 191 op/s 2024-09-21T10:58:20.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:20 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4168142263' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T10:58:20.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:20 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T10:58:20.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:58:20.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:20 smithi070 ceph-mon[112139]: pgmap v2911: 99 pgs: 99 active+clean; 120 MiB data, 4.2 GiB used, 353 GiB / 358 GiB avail; 12 MiB/s wr, 191 op/s 2024-09-21T10:58:20.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:20 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4168142263' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T10:58:20.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:20 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T10:58:20.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:58:20.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:20.639 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:20.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:20.740 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:20.761 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:20.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:20.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:20 smithi160 ceph-mon[101879]: pgmap v2911: 99 pgs: 99 active+clean; 120 MiB data, 4.2 GiB used, 353 GiB / 358 GiB avail; 12 MiB/s wr, 191 op/s 2024-09-21T10:58:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:20 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4168142263' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T10:58:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:20 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T10:58:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:58:20.864 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:20.885 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:20.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:20.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:20.987 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:21.009 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:21.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:21.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:21.110 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:21.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:21.135 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:21.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:21.236 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:21.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:21.259 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:21.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:21.360 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:21.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:21.383 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:21.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:21.484 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:21.525 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:21.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:21.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:21.619 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 5 16 14083 14067 175.823 0 - 0.00439888 2024-09-21T10:58:21.627 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:21.649 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:21.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:21.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:21.750 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:21.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:21.772 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:21.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:21 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-21T10:58:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:21 smithi160 ceph-mon[101879]: osdmap e1529: 8 total, 8 up, 5 in 2024-09-21T10:58:21.844 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:21 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-21T10:58:21.844 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:21 smithi070 ceph-mon[112139]: osdmap e1529: 8 total, 8 up, 5 in 2024-09-21T10:58:21.845 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:21 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-21T10:58:21.845 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:21 smithi070 ceph-mon[107385]: osdmap e1529: 8 total, 8 up, 5 in 2024-09-21T10:58:21.874 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:21.897 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:21.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:21.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:21.999 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:22.021 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:22.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:22.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:22.123 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:22.146 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:22.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:22.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:22.248 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:22.270 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:22.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:22.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:22.372 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:22.394 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:22.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:22.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:22.489 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T10:58:22.496 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:22.526 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:22.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:22.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:22.620 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 6 16 14083 14067 146.518 0 - 0.00439888 2024-09-21T10:58:22.628 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:22.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:22.650 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:22.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:22.751 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:22.772 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:22.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:22.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:22.801 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:22 smithi070 ceph-mon[107385]: pgmap v2913: 99 pgs: 99 active+clean; 295 MiB data, 4.9 GiB used, 353 GiB / 358 GiB avail; 37 MiB/s wr, 588 op/s 2024-09-21T10:58:22.801 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:22 smithi070 ceph-mon[107385]: osdmap e1530: 8 total, 8 up, 5 in 2024-09-21T10:58:22.801 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:22 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1791006' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:58:22.801 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:22 smithi070 ceph-mon[112139]: pgmap v2913: 99 pgs: 99 active+clean; 295 MiB data, 4.9 GiB used, 353 GiB / 358 GiB avail; 37 MiB/s wr, 588 op/s 2024-09-21T10:58:22.801 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:22 smithi070 ceph-mon[112139]: osdmap e1530: 8 total, 8 up, 5 in 2024-09-21T10:58:22.801 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:22 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1791006' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:58:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:22 smithi160 ceph-mon[101879]: pgmap v2913: 99 pgs: 99 active+clean; 295 MiB data, 4.9 GiB used, 353 GiB / 358 GiB avail; 37 MiB/s wr, 588 op/s 2024-09-21T10:58:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:22 smithi160 ceph-mon[101879]: osdmap e1530: 8 total, 8 up, 5 in 2024-09-21T10:58:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:22 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1791006' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:58:22.875 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:22.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:22.905 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:22.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:23.006 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:23.035 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:23.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:23.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:23.137 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:23.165 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:23.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:23.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:23.267 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:23.295 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:23.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:23.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:23.397 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:23.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:23.442 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:23.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:23.485 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:58:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:58:23] "GET /metrics HTTP/1.1" 200 36607 "" "Prometheus/2.51.0" 2024-09-21T10:58:23.543 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:23.572 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:23.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:23.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:23.620 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 7 16 15505 15489 138.282 29.625 0.0119009 0.00722174 2024-09-21T10:58:23.676 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:23.713 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:23.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:23.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:23.815 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:23 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:58:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:23 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1791006' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:58:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:23 smithi160 ceph-mon[101879]: osdmap e1531: 8 total, 8 up, 5 in 2024-09-21T10:58:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:23 smithi160 ceph-mon[101879]: osdmap e1532: 8 total, 8 up, 5 in 2024-09-21T10:58:23.855 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:23.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:23.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:23.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:23 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:58:23.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:23 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1791006' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:58:23.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:23 smithi070 ceph-mon[107385]: osdmap e1531: 8 total, 8 up, 5 in 2024-09-21T10:58:23.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:23 smithi070 ceph-mon[107385]: osdmap e1532: 8 total, 8 up, 5 in 2024-09-21T10:58:23.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:23 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:58:23.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:23 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1791006' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:58:23.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:23 smithi070 ceph-mon[112139]: osdmap e1531: 8 total, 8 up, 5 in 2024-09-21T10:58:23.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:23 smithi070 ceph-mon[112139]: osdmap e1532: 8 total, 8 up, 5 in 2024-09-21T10:58:23.957 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:23.986 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:23.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:23.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:24.088 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:24.124 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:24.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:24.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:24.226 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:24.257 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:24.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:24.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:24.301 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:58:24.301 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:58:24.329 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:58:24.329 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:58:24.359 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:24.364 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:58:24.365 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:58:24.391 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:58:24.392 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:58:24.395 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:24.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:24.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:24.421 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:58:24.421 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:58:24.448 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T10:58:24.448 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:58:24.475 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:58:24.476 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:58:24.498 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:24.504 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:58:24.504 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:58:24.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:24.538 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:24.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:24.620 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 8 16 17567 17551 137.105 128.875 0.00873713 0.0072888 2024-09-21T10:58:24.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:24 smithi070 ceph-mon[107385]: pgmap v2916: 99 pgs: 18 peering, 11 remapped+peering, 4 unknown, 66 active+clean; 721 MiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 120 MiB/s wr, 1.92k op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:58:24.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:24 smithi070 ceph-mon[107385]: Health check failed: Reduced data availability: 13 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-09-21T10:58:24.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:24 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1163702892' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:58:24.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:24 smithi070 ceph-mon[107385]: osdmap e1533: 8 total, 8 up, 5 in 2024-09-21T10:58:24.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:24 smithi070 ceph-mon[112139]: pgmap v2916: 99 pgs: 18 peering, 11 remapped+peering, 4 unknown, 66 active+clean; 721 MiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 120 MiB/s wr, 1.92k op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:58:24.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:24 smithi070 ceph-mon[112139]: Health check failed: Reduced data availability: 13 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-09-21T10:58:24.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:24 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1163702892' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:58:24.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:24 smithi070 ceph-mon[112139]: osdmap e1533: 8 total, 8 up, 5 in 2024-09-21T10:58:24.639 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:24.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:24.678 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:24.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:24.743 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:24 smithi160 ceph-mon[101879]: pgmap v2916: 99 pgs: 18 peering, 11 remapped+peering, 4 unknown, 66 active+clean; 721 MiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 120 MiB/s wr, 1.92k op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:58:24.743 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:24 smithi160 ceph-mon[101879]: Health check failed: Reduced data availability: 13 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-09-21T10:58:24.743 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:24 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1163702892' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:58:24.743 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:24 smithi160 ceph-mon[101879]: osdmap e1533: 8 total, 8 up, 5 in 2024-09-21T10:58:24.779 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:24.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:24.821 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:24.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:24.922 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:24.956 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:24.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:24.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:25.058 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:25.080 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:25.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:25.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:25.182 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:25.212 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:25.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:25.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:25.314 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:25.343 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:25.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:25.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:25.445 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:25.473 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:25.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:25.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:25.575 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:25.615 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:25.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:25.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:25.620 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 9 16 19327 19311 134.092 110 0.0119267 0.00745228 2024-09-21T10:58:25.718 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:25.746 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:25.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:25.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:25 smithi160 ceph-mon[101879]: Health check failed: Degraded data redundancy: 1731/45831 objects degraded (3.777%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:58:25.848 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:25.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:25 smithi070 ceph-mon[107385]: Health check failed: Degraded data redundancy: 1731/45831 objects degraded (3.777%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:58:25.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:25 smithi070 ceph-mon[112139]: Health check failed: Degraded data redundancy: 1731/45831 objects degraded (3.777%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:58:25.884 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:25.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:25.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:25.937 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:58:25.966 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-21T10:58:25.986 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:26.017 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:26.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:26.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:26.119 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:26.162 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:26.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:26.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:26.263 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:26.286 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:26.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:26.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:26.388 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:26.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:26.427 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:26.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:26.527 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:26.557 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:26.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:26.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:26.620 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 10 16 21016 21000 131.239 105.562 0.00589979 0.00761381 2024-09-21T10:58:26.660 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:26.686 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:26.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:26.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:26.753 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:58:26.788 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:26.826 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:26.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:26.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:26 smithi160 ceph-mon[101879]: pgmap v2919: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 19 peering, 11 remapped+peering, 2 unknown, 64 active+clean; 953 MiB data, 9.1 GiB used, 438 GiB / 447 GiB avail; 165 MiB/s wr, 2.62k op/s; 1731/45831 objects degraded (3.777%); 669/45831 objects misplaced (1.460%); 3.4 MiB/s, 2 keys/s, 56 objects/s recovering 2024-09-21T10:58:26.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:26 smithi070 ceph-mon[107385]: pgmap v2919: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 19 peering, 11 remapped+peering, 2 unknown, 64 active+clean; 953 MiB data, 9.1 GiB used, 438 GiB / 447 GiB avail; 165 MiB/s wr, 2.62k op/s; 1731/45831 objects degraded (3.777%); 669/45831 objects misplaced (1.460%); 3.4 MiB/s, 2 keys/s, 56 objects/s recovering 2024-09-21T10:58:26.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:26 smithi070 ceph-mon[112139]: pgmap v2919: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 19 peering, 11 remapped+peering, 2 unknown, 64 active+clean; 953 MiB data, 9.1 GiB used, 438 GiB / 447 GiB avail; 165 MiB/s wr, 2.62k op/s; 1731/45831 objects degraded (3.777%); 669/45831 objects misplaced (1.460%); 3.4 MiB/s, 2 keys/s, 56 objects/s recovering 2024-09-21T10:58:26.928 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:26.969 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:26.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:26.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:27.071 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:27.111 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:27.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:27.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:27.213 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:27.241 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:27.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:27.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:27.343 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:27.381 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:27.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:27.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:27.433 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:58:27.483 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:27.524 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:27.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:27.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:27.620 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 11 16 22847 22831 129.71 114.438 0.00659879 0.00770493 2024-09-21T10:58:27.626 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:27.653 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:27.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:27.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:27.755 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:27.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:27.796 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:27.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:27.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:27 smithi070 ceph-mon[107385]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 13 pgs peering) 2024-09-21T10:58:27.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:27 smithi070 ceph-mon[112139]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 13 pgs peering) 2024-09-21T10:58:27.898 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:27.923 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:27.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:27.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:28.025 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:28.064 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:28.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:28.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:28.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:27 smithi160 ceph-mon[101879]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 13 pgs peering) 2024-09-21T10:58:28.166 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:28.192 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:28.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:28.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:28.294 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:28.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:28.332 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:28.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:28.433 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:28.462 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:28.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:28.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:28.564 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:28.606 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:28.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:28.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:28.620 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 12 16 24491 24475 127.463 102.75 0.00966005 0.00783966 2024-09-21T10:58:28.709 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:28.751 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:28.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:28.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:28.853 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:28.875 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:28.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:28.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:28.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:28 smithi070 ceph-mon[107385]: pgmap v2920: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 peering, 11 remapped+peering, 84 active+clean; 954 MiB data, 9.2 GiB used, 438 GiB / 447 GiB avail; 27 KiB/s rd, 122 MiB/s wr, 1.97k op/s; 1731/45834 objects degraded (3.777%); 669/45834 objects misplaced (1.460%); 2.6 MiB/s, 1 keys/s, 43 objects/s recovering 2024-09-21T10:58:28.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:28 smithi070 ceph-mon[112139]: pgmap v2920: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 peering, 11 remapped+peering, 84 active+clean; 954 MiB data, 9.2 GiB used, 438 GiB / 447 GiB avail; 27 KiB/s rd, 122 MiB/s wr, 1.97k op/s; 1731/45834 objects degraded (3.777%); 669/45834 objects misplaced (1.460%); 2.6 MiB/s, 1 keys/s, 43 objects/s recovering 2024-09-21T10:58:28.977 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:29.003 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:29.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:29.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:29.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:28 smithi160 ceph-mon[101879]: pgmap v2920: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 peering, 11 remapped+peering, 84 active+clean; 954 MiB data, 9.2 GiB used, 438 GiB / 447 GiB avail; 27 KiB/s rd, 122 MiB/s wr, 1.97k op/s; 1731/45834 objects degraded (3.777%); 669/45834 objects misplaced (1.460%); 2.6 MiB/s, 1 keys/s, 43 objects/s recovering 2024-09-21T10:58:29.105 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:29.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:29.144 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:29.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:29.168 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:58:29.245 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:29.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:29.268 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:29.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:29.369 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:29.391 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:29.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:29.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:29.493 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:29.526 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:29.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:29.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:29.620 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 13 16 25878 25862 124.326 86.6875 0.019072 0.00803545 2024-09-21T10:58:29.628 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:29.658 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:29.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:29.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:29.727 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:58:29.727 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T10:58:29.727 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T10:58:29.727 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T10:58:29.727 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:58:29.761 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:29.787 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:29.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:29.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:29.889 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:29.911 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:29.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:29.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:29.941 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:58:30.012 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:30.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:30.034 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:30.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:30.135 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:30.156 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:30.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:30.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:30.258 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:30.280 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:30.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:30.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:30.382 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:30.404 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:30.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:30.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:30.505 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:30.516 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:58:30.527 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:30.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:30.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:30.620 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 14 16 26100 26084 116.437 13.875 1.00958 0.00808225 2024-09-21T10:58:30.629 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:30.658 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:30.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:30.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:30.760 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:58:30.801 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:30.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:30.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:58:30.816 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:30 smithi070 ceph-mon[107385]: pgmap v2921: 99 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 1 peering, 88 active+clean; 1.2 GiB data, 9.8 GiB used, 437 GiB / 447 GiB avail; 23 KiB/s rd, 84 MiB/s wr, 1.36k op/s; 6963/61062 objects degraded (11.403%); 669/61062 objects misplaced (1.096%); 2.2 MiB/s, 36 objects/s recovering 2024-09-21T10:58:30.816 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:30 smithi070 ceph-mon[107385]: osdmap e1534: 8 total, 8 up, 5 in 2024-09-21T10:58:30.816 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:30 smithi070 ceph-mon[112139]: pgmap v2921: 99 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 1 peering, 88 active+clean; 1.2 GiB data, 9.8 GiB used, 437 GiB / 447 GiB avail; 23 KiB/s rd, 84 MiB/s wr, 1.36k op/s; 6963/61062 objects degraded (11.403%); 669/61062 objects misplaced (1.096%); 2.2 MiB/s, 36 objects/s recovering 2024-09-21T10:58:30.816 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:30 smithi070 ceph-mon[112139]: osdmap e1534: 8 total, 8 up, 5 in 2024-09-21T10:58:30.903 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:30.930 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:30.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:30.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:30.967 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7, 4] out_osds: [2, 5, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:58:30.967 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:58:30.968 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:58:30.968 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-21T10:58:30.968 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.5 2024-09-21T10:58:31.011 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:30 smithi160 ceph-mon[101879]: pgmap v2921: 99 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 1 peering, 88 active+clean; 1.2 GiB data, 9.8 GiB used, 437 GiB / 447 GiB avail; 23 KiB/s rd, 84 MiB/s wr, 1.36k op/s; 6963/61062 objects degraded (11.403%); 669/61062 objects misplaced (1.096%); 2.2 MiB/s, 36 objects/s recovering 2024-09-21T10:58:31.011 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:30 smithi160 ceph-mon[101879]: osdmap e1534: 8 total, 8 up, 5 in 2024-09-21T10:58:31.032 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:31.057 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:31.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:31.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:31.159 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:31.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:31.224 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:31.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:31.325 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:31.341 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 10:58:31 smithi160 systemd[1]: Stopping Ceph osd.5 for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:58:31.361 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:31.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:31.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:31.463 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:31.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:31.492 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:31.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:31.592 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:31.620 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 15 16 27589 27573 114.878 93.0625 0.015612 0.00869588 2024-09-21T10:58:31.663 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:31.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:31.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:31.765 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:31.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:31.817 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:31.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:31 smithi160 ceph-mon[101879]: osdmap e1535: 8 total, 8 up, 5 in 2024-09-21T10:58:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:58:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:31 smithi160 ceph-mon[101879]: osd.5 marked itself down and dead 2024-09-21T10:58:31.841 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 10:58:31 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[44885]: 2024-09-21T10:58:31.493+0000 7f1b88b29640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:58:31.842 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 10:58:31 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[44885]: 2024-09-21T10:58:31.493+0000 7f1b88b29640 -1 osd.5 1534 *** Got signal Terminated *** 2024-09-21T10:58:31.842 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 10:58:31 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[44885]: 2024-09-21T10:58:31.493+0000 7f1b88b29640 -1 osd.5 1534 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:58:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:31 smithi070 ceph-mon[107385]: osdmap e1535: 8 total, 8 up, 5 in 2024-09-21T10:58:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:58:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:31 smithi070 ceph-mon[107385]: osd.5 marked itself down and dead 2024-09-21T10:58:31.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:31 smithi070 ceph-mon[112139]: osdmap e1535: 8 total, 8 up, 5 in 2024-09-21T10:58:31.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:58:31.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:31 smithi070 ceph-mon[112139]: osd.5 marked itself down and dead 2024-09-21T10:58:31.918 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:31.948 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:31.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:31.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:32.049 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:32.075 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:32.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:32.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:32.177 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:32.221 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:32.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:32.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:32.323 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:32.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:32.370 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:32.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:32.471 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:32.497 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:32.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:32.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:32.599 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:32.621 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 16 16 29067 29051 113.471 92.375 0.0117671 0.00880304 2024-09-21T10:58:32.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:32.662 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:32.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:32.764 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:32.800 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:32.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:32.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:32.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:32 smithi070 ceph-mon[112139]: pgmap v2924: 99 pgs: 1 peering, 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 1.4 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 23 KiB/s rd, 89 MiB/s wr, 1.44k op/s; 6117/70617 objects degraded (8.662%); 786/70617 objects misplaced (1.113%); 9.1 MiB/s, 145 objects/s recovering 2024-09-21T10:58:32.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:32 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 6117/70617 objects degraded (8.662%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:58:32.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:32 smithi070 ceph-mon[112139]: osdmap e1536: 8 total, 7 up, 5 in 2024-09-21T10:58:32.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:32 smithi070 ceph-mon[107385]: pgmap v2924: 99 pgs: 1 peering, 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 1.4 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 23 KiB/s rd, 89 MiB/s wr, 1.44k op/s; 6117/70617 objects degraded (8.662%); 786/70617 objects misplaced (1.113%); 9.1 MiB/s, 145 objects/s recovering 2024-09-21T10:58:32.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:32 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 6117/70617 objects degraded (8.662%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:58:32.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:32 smithi070 ceph-mon[107385]: osdmap e1536: 8 total, 7 up, 5 in 2024-09-21T10:58:32.902 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:32.924 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:32.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:32.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:32.947 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:32 smithi160 ceph-mon[101879]: pgmap v2924: 99 pgs: 1 peering, 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 1.4 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 23 KiB/s rd, 89 MiB/s wr, 1.44k op/s; 6117/70617 objects degraded (8.662%); 786/70617 objects misplaced (1.113%); 9.1 MiB/s, 145 objects/s recovering 2024-09-21T10:58:32.947 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:32 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 6117/70617 objects degraded (8.662%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:58:32.947 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:32 smithi160 ceph-mon[101879]: osdmap e1536: 8 total, 7 up, 5 in 2024-09-21T10:58:32.948 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 10:58:32 smithi160 podman[218394]: 2024-09-21 10:58:32.630165767 +0000 UTC m=+1.289247014 container died f740263456411c12e331f612971f01fe31b23038cbd9696505c5431a8216b1ac (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=reef-6a1bdf8, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:58:33.026 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:33.049 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:33.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:33.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:33.151 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:33.176 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:33.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:33.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:33.278 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:33.300 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:33.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:33.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:33.341 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 10:58:33 smithi160 podman[218394]: 2024-09-21 10:58:33.143970443 +0000 UTC m=+1.803051682 container cleanup f740263456411c12e331f612971f01fe31b23038cbd9696505c5431a8216b1ac (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=reef-6a1bdf8, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731) 2024-09-21T10:58:33.341 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 10:58:33 smithi160 bash[218394]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5 2024-09-21T10:58:33.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:58:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:58:33] "GET /metrics HTTP/1.1" 200 36607 "" "Prometheus/2.51.0" 2024-09-21T10:58:33.403 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:33.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:33.427 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:33.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:33.528 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:33.550 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:33.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:33.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:33.621 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 17 16 29427 29411 108.119 22.5 0.00197058 0.00878375 2024-09-21T10:58:33.634 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T10:58:33.652 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:33.681 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:33.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:33.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:33.719 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 10:58:33 smithi160 podman[218557]: 2024-09-21 10:58:33.412628299 +0000 UTC m=+1.102186898 container remove f740263456411c12e331f612971f01fe31b23038cbd9696505c5431a8216b1ac (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=reef-6a1bdf8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-21T10:58:33.719 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:33 smithi160 ceph-mon[101879]: osdmap e1537: 8 total, 7 up, 5 in 2024-09-21T10:58:33.719 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:33 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2209360937' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:58:33.719 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:33 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:58:33.726 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:33 smithi070 ceph-mon[107385]: osdmap e1537: 8 total, 7 up, 5 in 2024-09-21T10:58:33.726 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:33 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2209360937' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:58:33.727 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:33 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:58:33.727 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:33 smithi070 ceph-mon[112139]: osdmap e1537: 8 total, 7 up, 5 in 2024-09-21T10:58:33.727 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:33 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2209360937' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:58:33.727 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:33 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:58:33.783 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:33.809 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:33.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:33.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:33.911 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:33.936 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:33.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:33.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:33.989 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:58:34.038 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:34.062 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:34.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:34.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:34.164 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:34.206 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:34.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:34.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:34.308 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:34.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:34.330 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:34.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:34.431 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:34.454 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:34.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:34.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:34.556 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:34.579 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:34.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:34.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:34.622 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 18 16 30685 30669 106.48 78.625 0.00399494 0.00938393 2024-09-21T10:58:34.680 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:34.732 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:34.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:34.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:34.834 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:34.874 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:34.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:34.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:34.884 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:34 smithi160 ceph-mon[101879]: pgmap v2927: 99 pgs: 1 peering, 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 1.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 111 MiB/s wr, 1.78k op/s; 6117/82473 objects degraded (7.417%); 786/82473 objects misplaced (0.953%); 11 MiB/s, 182 objects/s recovering 2024-09-21T10:58:34.884 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:34 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:58:34.884 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:34 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:58:34.884 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:34 smithi160 ceph-mon[101879]: osdmap e1538: 8 total, 7 up, 5 in 2024-09-21T10:58:34.886 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 10:58:34 smithi160 podman[218904]: 2024-09-21 10:58:34.603285195 +0000 UTC m=+0.037735181 image pull 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6 2024-09-21T10:58:34.886 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 10:58:34 smithi160 podman[218904]: 2024-09-21 10:58:34.752309241 +0000 UTC m=+0.186759199 container create 2f0cf126edfaec575c9add18f4bfc86bd164bc67982ae8755fb97ea3944e15a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=reef-6a1bdf8, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_BRANCH=HEAD) 2024-09-21T10:58:34.976 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:35.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:35.026 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:35.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:35.083 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:34 smithi070 ceph-mon[107385]: pgmap v2927: 99 pgs: 1 peering, 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 1.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 111 MiB/s wr, 1.78k op/s; 6117/82473 objects degraded (7.417%); 786/82473 objects misplaced (0.953%); 11 MiB/s, 182 objects/s recovering 2024-09-21T10:58:35.083 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:34 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:58:35.084 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:34 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:58:35.084 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:34 smithi070 ceph-mon[107385]: osdmap e1538: 8 total, 7 up, 5 in 2024-09-21T10:58:35.084 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:34 smithi070 ceph-mon[112139]: pgmap v2927: 99 pgs: 1 peering, 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 1.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 111 MiB/s wr, 1.78k op/s; 6117/82473 objects degraded (7.417%); 786/82473 objects misplaced (0.953%); 11 MiB/s, 182 objects/s recovering 2024-09-21T10:58:35.084 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:34 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:58:35.084 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:34 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:58:35.084 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:34 smithi070 ceph-mon[112139]: osdmap e1538: 8 total, 7 up, 5 in 2024-09-21T10:58:35.127 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:35.152 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 10:58:35 smithi160 podman[218904]: 2024-09-21 10:58:35.039351288 +0000 UTC m=+0.473801264 container init 2f0cf126edfaec575c9add18f4bfc86bd164bc67982ae8755fb97ea3944e15a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-6a1bdf8, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-21T10:58:35.153 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 10:58:35 smithi160 podman[218904]: 2024-09-21 10:58:35.047499197 +0000 UTC m=+0.481949159 container start 2f0cf126edfaec575c9add18f4bfc86bd164bc67982ae8755fb97ea3944e15a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=reef-6a1bdf8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-21T10:58:35.153 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:35.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:35.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:35.255 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:35.295 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:58:35.295 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:58:35.296 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:58:35.296 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:58:35.296 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:58:35.296 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:58:35.296 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:58:35.296 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:58:35.296 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:58:35.296 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:58:35.296 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T10:58:35.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:35.311 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:35.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:35.329 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T10:58:35.329 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:58:35.359 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:58:35.359 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:58:35.412 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:35.435 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:35.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:35.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:35.537 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:35.576 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:35.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:35.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:35.591 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 10:58:35 smithi160 podman[218904]: 2024-09-21 10:58:35.147544005 +0000 UTC m=+0.581993987 container attach 2f0cf126edfaec575c9add18f4bfc86bd164bc67982ae8755fb97ea3944e15a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=reef-6a1bdf8, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:58:35.621 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 19 16 31989 31973 105.165 81.5 0.0220691 0.00950385 2024-09-21T10:58:35.679 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:35.716 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:35.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:35.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:35.818 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:35.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:35.879 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:35.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:35.965 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:35 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3000018510' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:58:35.966 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:58:35.966 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:58:35.966 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 10:58:35 smithi160 podman[218904]: 2024-09-21 10:58:35.803557718 +0000 UTC m=+1.238007683 container died 2f0cf126edfaec575c9add18f4bfc86bd164bc67982ae8755fb97ea3944e15a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-6a1bdf8, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-21T10:58:35.973 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:35 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3000018510' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:58:35.973 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:58:35.973 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:58:35.974 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:35 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3000018510' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:58:35.974 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:58:35.974 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:58:35.980 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:36.005 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:36.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:36.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:36.107 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:36.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:36.152 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:36.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:36.253 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:36.278 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:36.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:36.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:36.380 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:36.416 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:36.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:36.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:36.518 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:36.544 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:36.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:36.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:36.621 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:2024-09-21T10:58:36.620133+0000 min lat: 0.00141297 max lat: 2.4606 avg lat: 0.0095514 2024-09-21T10:58:36.621 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:58:36.621 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 20 16 33501 33485 104.632 94.5 0.00921355 0.0095514 2024-09-21T10:58:36.646 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:36.687 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:36.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:36.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:36.789 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:36.816 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:36.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:36.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:36.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:36 smithi070 ceph-mon[112139]: pgmap v2929: 99 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.8 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 77 MiB/s wr, 1.23k op/s; 6117/86955 objects degraded (7.035%); 786/86955 objects misplaced (0.904%) 2024-09-21T10:58:36.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:36 smithi070 ceph-mon[107385]: pgmap v2929: 99 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.8 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 77 MiB/s wr, 1.23k op/s; 6117/86955 objects degraded (7.035%); 786/86955 objects misplaced (0.904%) 2024-09-21T10:58:36.918 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:36.952 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:36 smithi160 ceph-mon[101879]: pgmap v2929: 99 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.8 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 77 MiB/s wr, 1.23k op/s; 6117/86955 objects degraded (7.035%); 786/86955 objects misplaced (0.904%) 2024-09-21T10:58:36.954 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:36.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:36.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:36.975 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.5.service' 2024-09-21T10:58:37.056 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:37.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:37.110 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:37.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:37.141 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:58:37.142 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-21T10:58:37.211 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:37.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:37.238 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:37.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:37.339 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:37.387 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:37.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:37.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:37.422 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:58:37.423 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:58:37.489 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:37.517 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:37.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:37.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:37.619 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:37.621 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 21 16 34950 34934 103.961 90.5625 0.00643242 0.00961322 2024-09-21T10:58:37.664 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:37.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:37.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:37.766 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:37.791 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:37.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:37.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:37.877 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:58:37.892 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:37.947 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:37.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:37.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:38.048 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:38.075 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:38.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:38.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:38.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:37 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 6117/88464 objects degraded (6.915%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:58:38.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:37 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 6117/88464 objects degraded (6.915%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:58:38.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:37 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 6117/88464 objects degraded (6.915%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:58:38.178 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:38.210 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:38.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:38.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:38.312 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:38.346 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:38.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:38.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:38.448 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:38.474 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:38.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:38.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:38.576 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:38.603 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:38.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:38.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:38.621 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 22 16 36372 36356 103.275 88.875 0.0164673 0.00967691 2024-09-21T10:58:38.706 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:38.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:38.736 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:38.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:38.837 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:38.863 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:38.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:38.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:38.965 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:38.992 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:38.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:38.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:39.047 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:38 smithi070 ceph-mon[107385]: pgmap v2930: 99 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 1.8 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 62 MiB/s wr, 991 op/s; 6117/88464 objects degraded (6.915%); 8.2 MiB/s, 130 objects/s recovering 2024-09-21T10:58:39.047 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:38 smithi070 ceph-mon[112139]: pgmap v2930: 99 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 1.8 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 62 MiB/s wr, 991 op/s; 6117/88464 objects degraded (6.915%); 8.2 MiB/s, 130 objects/s recovering 2024-09-21T10:58:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:38 smithi160 ceph-mon[101879]: pgmap v2930: 99 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 1.8 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 62 MiB/s wr, 991 op/s; 6117/88464 objects degraded (6.915%); 8.2 MiB/s, 130 objects/s recovering 2024-09-21T10:58:39.094 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:39.137 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:39.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:39.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:39.212 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:58:39.213 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:58:39.239 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:39.281 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:39.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:39.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:39.383 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:39.409 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:39.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:39.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:39.424 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:58:39.512 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:39.539 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:39.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:39.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:39.621 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 23 16 37756 37740 102.546 86.5 0.0141031 0.00974593 2024-09-21T10:58:39.641 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:39.683 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:39.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:39.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:39.786 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:39.810 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:39.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:39.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:39.912 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:39.938 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:39.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:39.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:40.039 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:40.063 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:40.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:40.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:40.125 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:58:40.165 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:40.200 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:58:40.201 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:40.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:40.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:40.271 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:58:40.303 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:40.349 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:40.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:40.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:40.358 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:40 smithi070 ceph-mon[107385]: pgmap v2931: 99 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 2.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 81 MiB/s wr, 1.30k op/s; 5264/99534 objects degraded (5.289%); 236/99534 objects misplaced (0.237%); 12 MiB/s, 188 objects/s recovering 2024-09-21T10:58:40.358 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:40 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:58:40.358 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:40 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:58:40.358 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:40 smithi070 ceph-mon[112139]: pgmap v2931: 99 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 2.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 81 MiB/s wr, 1.30k op/s; 5264/99534 objects degraded (5.289%); 236/99534 objects misplaced (0.237%); 12 MiB/s, 188 objects/s recovering 2024-09-21T10:58:40.359 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:40 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:58:40.359 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:40 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:58:40.450 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:40.475 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:40.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:40.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:40.577 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:40.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:40 smithi160 ceph-mon[101879]: pgmap v2931: 99 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 2.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 81 MiB/s wr, 1.30k op/s; 5264/99534 objects degraded (5.289%); 236/99534 objects misplaced (0.237%); 12 MiB/s, 188 objects/s recovering 2024-09-21T10:58:40.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:40 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:58:40.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:40 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:58:40.615 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:40.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:40.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:40.621 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 24 16 39204 39188 102.043 90.5 0.00736335 0.00979473 2024-09-21T10:58:40.717 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:40.738 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:40.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:40.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:40.840 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:40.865 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:40.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:40.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:40.967 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:40.988 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:40.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:40.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:41.090 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:41.112 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:41.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:41.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:41.214 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:41.236 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:41.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:41.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:41.337 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:41.359 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:41.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:41.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:41.461 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:41.483 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:41.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:41.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:41.585 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:41.607 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:41.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:41.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:41.621 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 25 16 39596 39580 98.9415 24.5 0.00264163 0.00978052 2024-09-21T10:58:41.709 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:58:41.746 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:41.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:41.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:58:41.848 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:41.873 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:41.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:41.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:41.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:41 smithi070 ceph-mon[112139]: osdmap e1539: 8 total, 7 up, 5 in 2024-09-21T10:58:41.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:41 smithi070 ceph-mon[107385]: osdmap e1539: 8 total, 7 up, 5 in 2024-09-21T10:58:41.977 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:42.016 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:42.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:42.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:42.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:41 smithi160 ceph-mon[101879]: osdmap e1539: 8 total, 7 up, 5 in 2024-09-21T10:58:42.118 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:42.143 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7, 4] out_osds: [2, 5, 0] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-09-21T10:58:42.143 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:58:42.143 INFO:tasks.thrashosds.thrasher:Killing osd 6, live_osds are [0, 1, 2, 3, 4, 6, 7] 2024-09-21T10:58:42.143 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-21T10:58:42.144 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.6 2024-09-21T10:58:42.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:42.164 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:42.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:42.265 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:42.315 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:42.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:42.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:42.417 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:42.441 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:42.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:42.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:42.543 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:42.569 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:42.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:42.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:42.591 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 10:58:42 smithi160 systemd[1]: Stopping Ceph osd.6 for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:58:42.621 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 26 16 40826 40810 98.0925 76.875 0.020228 0.0101894 2024-09-21T10:58:42.671 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:42.713 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:42.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:42.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:42.815 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:42.850 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:42.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:42.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:42.951 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:42.977 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:42.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:42.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:43.079 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:42 smithi160 ceph-mon[101879]: pgmap v2933: 99 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 2.2 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 61 MiB/s wr, 971 op/s; 5264/105804 objects degraded (4.975%); 236/105804 objects misplaced (0.223%); 11 MiB/s, 175 objects/s recovering 2024-09-21T10:58:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:42 smithi160 ceph-mon[101879]: osdmap e1540: 8 total, 7 up, 5 in 2024-09-21T10:58:43.092 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 10:58:42 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[51187]: 2024-09-21T10:58:42.702+0000 7f81d78e2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:58:43.092 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 10:58:42 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[51187]: 2024-09-21T10:58:42.702+0000 7f81d78e2640 -1 osd.6 1540 *** Got signal Terminated *** 2024-09-21T10:58:43.092 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 10:58:42 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[51187]: 2024-09-21T10:58:42.702+0000 7f81d78e2640 -1 osd.6 1540 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:58:43.102 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:43.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:43.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:43.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:42 smithi070 ceph-mon[112139]: pgmap v2933: 99 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 2.2 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 61 MiB/s wr, 971 op/s; 5264/105804 objects degraded (4.975%); 236/105804 objects misplaced (0.223%); 11 MiB/s, 175 objects/s recovering 2024-09-21T10:58:43.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:42 smithi070 ceph-mon[112139]: osdmap e1540: 8 total, 7 up, 5 in 2024-09-21T10:58:43.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:42 smithi070 ceph-mon[107385]: pgmap v2933: 99 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 2.2 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 61 MiB/s wr, 971 op/s; 5264/105804 objects degraded (4.975%); 236/105804 objects misplaced (0.223%); 11 MiB/s, 175 objects/s recovering 2024-09-21T10:58:43.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:42 smithi070 ceph-mon[107385]: osdmap e1540: 8 total, 7 up, 5 in 2024-09-21T10:58:43.194 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:58:43.205 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:43.235 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:43.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:43.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:43.337 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:43.361 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:43.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:43.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:43.425 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:58:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:58:43] "GET /metrics HTTP/1.1" 200 36473 "" "Prometheus/2.51.0" 2024-09-21T10:58:43.464 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:43.491 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:43.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:43.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:43.592 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:43.617 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:43.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:43.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:43.621 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 27 16 42263 42247 97.7856 89.8125 0.0118753 0.0102204 2024-09-21T10:58:43.720 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:43.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:43.750 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:43.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:43.851 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:43.873 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:43.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:43.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:43.975 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:43.997 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:43.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:43.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:43 smithi160 ceph-mon[101879]: osd.6 marked itself down and dead 2024-09-21T10:58:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:43 smithi160 ceph-mon[101879]: pgmap v2935: 99 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 2.4 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 82 MiB/s wr, 1.31k op/s; 4402/118332 objects degraded (3.720%); 432/118332 objects misplaced (0.365%); 16 MiB/s, 258 objects/s recovering 2024-09-21T10:58:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:43 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 5264/105804 objects degraded (4.975%), 7 pgs degraded (PG_DEGRADED) 2024-09-21T10:58:44.100 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:44.117 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:43 smithi070 ceph-mon[107385]: osd.6 marked itself down and dead 2024-09-21T10:58:44.117 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:43 smithi070 ceph-mon[107385]: pgmap v2935: 99 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 2.4 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 82 MiB/s wr, 1.31k op/s; 4402/118332 objects degraded (3.720%); 432/118332 objects misplaced (0.365%); 16 MiB/s, 258 objects/s recovering 2024-09-21T10:58:44.117 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:43 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 5264/105804 objects degraded (4.975%), 7 pgs degraded (PG_DEGRADED) 2024-09-21T10:58:44.117 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:43 smithi070 ceph-mon[112139]: osd.6 marked itself down and dead 2024-09-21T10:58:44.117 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:43 smithi070 ceph-mon[112139]: pgmap v2935: 99 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 2.4 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 82 MiB/s wr, 1.31k op/s; 4402/118332 objects degraded (3.720%); 432/118332 objects misplaced (0.365%); 16 MiB/s, 258 objects/s recovering 2024-09-21T10:58:44.117 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:43 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 5264/105804 objects degraded (4.975%), 7 pgs degraded (PG_DEGRADED) 2024-09-21T10:58:44.121 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:44.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:44.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:44.223 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:44.244 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:44.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:44.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:44.346 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:44.367 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:44.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:44.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:44.468 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:44.490 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:44.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:44.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:44.591 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:44.614 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:44.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:44.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:44.621 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 28 16 42408 42392 94.6169 9.0625 0.00448473 0.0102232 2024-09-21T10:58:44.678 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 10:58:44 smithi160 podman[219839]: 2024-09-21 10:58:44.376071867 +0000 UTC m=+1.841143484 container died 03e6bb54f8b2c052b9ac3074460885e3041addbc99f6df6f583831db464c645c (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-6a1bdf8, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-21T10:58:44.716 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:44.752 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:44.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:44.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:44.854 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:44.876 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:44.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:44.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:44.967 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T10:58:44.967 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:44 smithi070 ceph-mon[107385]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:58:44.967 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:44 smithi070 ceph-mon[107385]: osdmap e1541: 8 total, 6 up, 5 in 2024-09-21T10:58:44.968 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:44 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3291246764' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:58:44.968 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:44 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:58:44.968 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:44 smithi070 ceph-mon[112139]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:58:44.968 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:44 smithi070 ceph-mon[112139]: osdmap e1541: 8 total, 6 up, 5 in 2024-09-21T10:58:44.968 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:44 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3291246764' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:58:44.968 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:44 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:58:44.978 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:45.000 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:45.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:45.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:45.065 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:44 smithi160 ceph-mon[101879]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:58:45.066 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:44 smithi160 ceph-mon[101879]: osdmap e1541: 8 total, 6 up, 5 in 2024-09-21T10:58:45.066 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:44 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3291246764' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:58:45.066 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:44 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:58:45.066 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 10:58:44 smithi160 podman[219839]: 2024-09-21 10:58:44.787248404 +0000 UTC m=+2.252320013 container cleanup 03e6bb54f8b2c052b9ac3074460885e3041addbc99f6df6f583831db464c645c (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=reef-6a1bdf8, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-21T10:58:45.066 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 10:58:44 smithi160 bash[219839]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6 2024-09-21T10:58:45.101 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:45.123 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:45.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:45.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:45.224 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:45.245 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:45.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:45.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:45.341 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 10:58:45 smithi160 podman[219893]: 2024-09-21 10:58:45.064189636 +0000 UTC m=+0.672980369 container remove 03e6bb54f8b2c052b9ac3074460885e3041addbc99f6df6f583831db464c645c (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=reef-6a1bdf8, ceph=True, org.label-schema.schema-version=1.0) 2024-09-21T10:58:45.346 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:45.368 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:45.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:45.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:45.470 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:45.492 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:45.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:45.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:45.594 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:45.616 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:45.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:45.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:45.621 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 29 16 42434 42418 91.4103 1.625 0.00240181 0.0104138 2024-09-21T10:58:45.644 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 10:58:45 smithi160 podman[220035]: 2024-09-21 10:58:45.286256967 +0000 UTC m=+0.016798546 image pull 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6 2024-09-21T10:58:45.644 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 10:58:45 smithi160 podman[220035]: 2024-09-21 10:58:45.415157968 +0000 UTC m=+0.145699535 container create 59bc0174a388883f09bab2790670e25f589174a4d272c5472dfcad49ac63bf9c (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-6a1bdf8, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-21T10:58:45.644 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 10:58:45 smithi160 podman[220035]: 2024-09-21 10:58:45.642709636 +0000 UTC m=+0.373251200 container init 59bc0174a388883f09bab2790670e25f589174a4d272c5472dfcad49ac63bf9c (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=reef-6a1bdf8, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-21T10:58:45.718 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:45.739 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:45.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:45.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:45.841 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:45.864 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:45.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:45.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:45.912 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:45 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:58:45.913 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:45 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:58:45.913 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:45 smithi160 ceph-mon[101879]: osdmap e1542: 8 total, 6 up, 5 in 2024-09-21T10:58:45.913 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:45 smithi160 ceph-mon[101879]: pgmap v2938: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 22 peering, 3 remapped, 11 stale+active+clean, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 59 active+clean; 2.5 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 88 MiB/s wr, 1.42k op/s; 2435/124260 objects degraded (1.960%); 432/124260 objects misplaced (0.348%); 12 MiB/s, 196 objects/s recovering 2024-09-21T10:58:45.913 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 10:58:45 smithi160 podman[220035]: 2024-09-21 10:58:45.648039718 +0000 UTC m=+0.378581280 container start 59bc0174a388883f09bab2790670e25f589174a4d272c5472dfcad49ac63bf9c (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=reef-6a1bdf8, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:58:45.913 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 10:58:45 smithi160 podman[220035]: 2024-09-21 10:58:45.751331532 +0000 UTC m=+0.481873093 container attach 59bc0174a388883f09bab2790670e25f589174a4d272c5472dfcad49ac63bf9c (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=reef-6a1bdf8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-21T10:58:45.965 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:45.994 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:45.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:45.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:46.096 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:46.117 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:46.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:46.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:46.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:45 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:58:46.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:45 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:58:46.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:45 smithi070 ceph-mon[107385]: osdmap e1542: 8 total, 6 up, 5 in 2024-09-21T10:58:46.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:45 smithi070 ceph-mon[107385]: pgmap v2938: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 22 peering, 3 remapped, 11 stale+active+clean, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 59 active+clean; 2.5 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 88 MiB/s wr, 1.42k op/s; 2435/124260 objects degraded (1.960%); 432/124260 objects misplaced (0.348%); 12 MiB/s, 196 objects/s recovering 2024-09-21T10:58:46.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:45 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:58:46.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:45 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:58:46.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:45 smithi070 ceph-mon[112139]: osdmap e1542: 8 total, 6 up, 5 in 2024-09-21T10:58:46.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:45 smithi070 ceph-mon[112139]: pgmap v2938: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 22 peering, 3 remapped, 11 stale+active+clean, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 59 active+clean; 2.5 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 88 MiB/s wr, 1.42k op/s; 2435/124260 objects degraded (1.960%); 432/124260 objects misplaced (0.348%); 12 MiB/s, 196 objects/s recovering 2024-09-21T10:58:46.219 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:46.241 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:46.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:46.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:46.282 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 10:58:45 smithi160 podman[220035]: 2024-09-21 10:58:45.910708957 +0000 UTC m=+0.641250517 container died 59bc0174a388883f09bab2790670e25f589174a4d272c5472dfcad49ac63bf9c (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6-deactivate, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=reef-6a1bdf8, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-21T10:58:46.342 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:46.363 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:46.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:46.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:46.465 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:46.487 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:46.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:46.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:46.588 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:46.610 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:46.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:46.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:46.622 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 30 16 42527 42511 88.557 5.8125 0.00239927 0.0107348 2024-09-21T10:58:46.711 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:46.734 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:46.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:46.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:46.835 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:46.856 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:46.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:46.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:46.958 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:46.979 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:46.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:46.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:47.081 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:47.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:46 smithi160 ceph-mon[101879]: Health check failed: Reduced data availability: 5 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-21T10:58:47.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:46 smithi160 ceph-mon[101879]: osdmap e1543: 8 total, 6 up, 5 in 2024-09-21T10:58:47.102 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:47.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:47.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:47.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:46 smithi070 ceph-mon[107385]: Health check failed: Reduced data availability: 5 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-21T10:58:47.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:46 smithi070 ceph-mon[107385]: osdmap e1543: 8 total, 6 up, 5 in 2024-09-21T10:58:47.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:46 smithi070 ceph-mon[112139]: Health check failed: Reduced data availability: 5 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-21T10:58:47.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:46 smithi070 ceph-mon[112139]: osdmap e1543: 8 total, 6 up, 5 in 2024-09-21T10:58:47.156 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:58:47.156 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:58:47.156 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:58:47.156 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:58:47.157 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:58:47.157 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:58:47.157 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:58:47.157 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:58:47.204 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:47.225 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:47.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:47.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:47.327 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:47.352 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:47.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:47.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:47.453 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:47.478 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:47.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:47.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:47.580 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:47.602 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:47.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:47.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:47.622 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 31 16 42527 42511 85.7003 0 - 0.0107348 2024-09-21T10:58:47.703 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:47.725 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:47.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:47.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:47.827 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:47.849 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:47.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:47.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:47.950 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:47.973 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:47.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:47.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:47.978 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:47 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3654375660' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:58:47.978 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:47 smithi070 ceph-mon[107385]: pgmap v2940: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 22 peering, 3 remapped, 11 stale+active+clean, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 59 active+clean; 2.5 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 67 MiB/s wr, 1.07k op/s; 2435/124260 objects degraded (1.960%); 432/124260 objects misplaced (0.348%); 10 MiB/s, 159 objects/s recovering 2024-09-21T10:58:47.978 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:47 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3654375660' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:58:47.979 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:47 smithi070 ceph-mon[112139]: pgmap v2940: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 22 peering, 3 remapped, 11 stale+active+clean, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 59 active+clean; 2.5 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 67 MiB/s wr, 1.07k op/s; 2435/124260 objects degraded (1.960%); 432/124260 objects misplaced (0.348%); 10 MiB/s, 159 objects/s recovering 2024-09-21T10:58:48.075 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:47 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3654375660' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:58:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:47 smithi160 ceph-mon[101879]: pgmap v2940: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 22 peering, 3 remapped, 11 stale+active+clean, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 59 active+clean; 2.5 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 67 MiB/s wr, 1.07k op/s; 2435/124260 objects degraded (1.960%); 432/124260 objects misplaced (0.348%); 10 MiB/s, 159 objects/s recovering 2024-09-21T10:58:48.096 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:48.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:48.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:48.198 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:48.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:48.220 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:48.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:48.321 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:48.342 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:48.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:48.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:48.376 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:58:48.376 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:58:48.376 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T10:58:48.376 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T10:58:48.394 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:58:48.394 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:58:48.444 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:48.465 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:48.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:48.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:48.567 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:48.596 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:48.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:48.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:48.622 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 32 16 42642 42626 83.2467 3.59375 0.00254745 0.011749 2024-09-21T10:58:48.698 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:48.728 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:48.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:48.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:48.830 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:48.856 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:48.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:48.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:48.958 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:48.983 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:48.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:48.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:49.027 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:48 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 2435/124260 objects degraded (1.960%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:58:49.027 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:48 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 2435/124260 objects degraded (1.960%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:58:49.054 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:58:49.054 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:58:49.085 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:49.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:48 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 2435/124260 objects degraded (1.960%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:58:49.113 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:49.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:49.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:49.214 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:49.243 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:49.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:49.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:49.345 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:49.368 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:49.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:49.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:49.390 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.6.service' 2024-09-21T10:58:49.443 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:58:49.443 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-21T10:58:49.470 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:49.511 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:49.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:49.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:49.613 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:49.622 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 33 16 43262 43246 81.8983 38.75 0.00241502 0.0121519 2024-09-21T10:58:49.637 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:49.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:49.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:49.738 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:49.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:49.764 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:49.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:49.824 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:49 smithi070 ceph-mon[107385]: pgmap v2941: 99 pgs: 1 activating+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+clean+remapped, 27 active+undersized, 2 active+recovering+undersized+degraded+remapped, 22 peering, 11 active+undersized+degraded, 29 active+clean; 2.6 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 32 MiB/s wr, 507 op/s; 26019/127467 objects degraded (20.412%); 5113/127467 objects misplaced (4.011%); 9.6 MiB/s, 151 objects/s recovering 2024-09-21T10:58:49.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:49 smithi070 ceph-mon[112139]: pgmap v2941: 99 pgs: 1 activating+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+clean+remapped, 27 active+undersized, 2 active+recovering+undersized+degraded+remapped, 22 peering, 11 active+undersized+degraded, 29 active+clean; 2.6 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 32 MiB/s wr, 507 op/s; 26019/127467 objects degraded (20.412%); 5113/127467 objects misplaced (4.011%); 9.6 MiB/s, 151 objects/s recovering 2024-09-21T10:58:49.865 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:49.889 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:49.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:49.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:49.980 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:58:49.991 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:50.053 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:50.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:50.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:50.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:49 smithi160 ceph-mon[101879]: pgmap v2941: 99 pgs: 1 activating+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+clean+remapped, 27 active+undersized, 2 active+recovering+undersized+degraded+remapped, 22 peering, 11 active+undersized+degraded, 29 active+clean; 2.6 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 32 MiB/s wr, 507 op/s; 26019/127467 objects degraded (20.412%); 5113/127467 objects misplaced (4.011%); 9.6 MiB/s, 151 objects/s recovering 2024-09-21T10:58:50.155 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:50.181 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:50.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:50.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:50.282 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:50.308 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:50.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:50.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:50.410 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:50.441 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:50.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:50.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:50.542 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:50.567 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:50.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:50.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:50.622 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 34 16 43755 43739 80.3957 30.8125 0.00282094 0.0123819 2024-09-21T10:58:50.669 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:50.691 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:50.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:50.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:50.696 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:58:50.696 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T10:58:50.696 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T10:58:50.696 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:58:50.793 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:50.815 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:50.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:50.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:50.916 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:50.941 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:50.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:50.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:51.042 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:51.055 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:58:51.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:51.065 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:51.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:51.166 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:51.192 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:51.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:51.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:51.293 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:51.315 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:51.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:51.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:51.417 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:51.441 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:51.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:51.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:51.501 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:58:51.543 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:51.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:51.578 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:51.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:51.622 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 35 16 44269 44253 79.0165 32.125 0.00269265 0.012612 2024-09-21T10:58:51.679 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:51.702 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:51.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:51.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:51.804 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:51.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:51.832 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:51.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:58:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:58:51.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:58:51.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:58:51.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:58:51.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:58:51.933 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:51.963 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:51.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:51.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:52.046 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:58:52.065 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:52.091 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:52.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:52.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:52.193 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:52.221 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:52.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:52.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:52.323 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:52.344 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:52.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:52.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:52.423 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:58:52.425 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:58:52.446 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:52.471 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:52.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:52.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:52.572 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:52.597 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:52.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:52.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:52.622 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 36 16 44766 44750 77.6844 31.0625 0.00225549 0.0128095 2024-09-21T10:58:52.699 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:52.728 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:52.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:52.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:52.830 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:52 smithi160 ceph-mon[101879]: pgmap v2942: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+clean+remapped, 44 active+undersized, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized+degraded, 29 active+clean; 2.6 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 28 MiB/s wr, 454 op/s; 33531/128331 objects degraded (26.129%); 5113/128331 objects misplaced (3.984%); 13 MiB/s, 199 objects/s recovering 2024-09-21T10:58:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:52 smithi160 ceph-mon[101879]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-09-21T10:58:52.861 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:52.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:52.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:52.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:52 smithi070 ceph-mon[112139]: pgmap v2942: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+clean+remapped, 44 active+undersized, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized+degraded, 29 active+clean; 2.6 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 28 MiB/s wr, 454 op/s; 33531/128331 objects degraded (26.129%); 5113/128331 objects misplaced (3.984%); 13 MiB/s, 199 objects/s recovering 2024-09-21T10:58:52.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:52 smithi070 ceph-mon[112139]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-09-21T10:58:52.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:52 smithi070 ceph-mon[107385]: pgmap v2942: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+clean+remapped, 44 active+undersized, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized+degraded, 29 active+clean; 2.6 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 28 MiB/s wr, 454 op/s; 33531/128331 objects degraded (26.129%); 5113/128331 objects misplaced (3.984%); 13 MiB/s, 199 objects/s recovering 2024-09-21T10:58:52.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:52 smithi070 ceph-mon[107385]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-09-21T10:58:52.962 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:52.988 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:52.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:52.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:53.090 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:53.115 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:53.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:53.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:53.217 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:53.240 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:53.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:53.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:53.277 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:58:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:58:53] "GET /metrics HTTP/1.1" 200 36501 "" "Prometheus/2.51.0" 2024-09-21T10:58:53.342 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:53.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:53.366 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:53.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:53.467 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:53.487 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:53.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:53.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:53.589 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:53.611 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:53.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:53.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:53.622 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 37 16 45276 45260 76.4463 31.875 0.00253354 0.0130275 2024-09-21T10:58:53.714 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:53.737 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:53.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:53.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:53.838 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:53.863 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:53.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:53.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:53.964 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:53.988 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:53.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:53.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:54.090 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:54.114 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:54.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:54.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:54.216 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:54.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:54.245 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:54.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:54.346 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:54.369 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:54.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:54.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:54.444 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7, 4] out_osds: [2, 5, 0] dead_osds: [5, 6] live_osds: [0, 1, 2, 3, 4, 7] 2024-09-21T10:58:54.444 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:58:54.444 INFO:tasks.thrashosds.thrasher:Reweighting osd 6 to 0.8683198499478187 2024-09-21T10:58:54.444 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 6 0.8683198499478187 2024-09-21T10:58:54.471 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:54.497 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:54.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:54.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:54.599 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:54.621 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:54.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:54.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:54.623 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 38 16 45823 45807 75.3341 34.1875 0.00285951 0.0132263 2024-09-21T10:58:54.652 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:54 smithi160 ceph-mon[101879]: pgmap v2943: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+undersized, 16 active+undersized+degraded, 29 active+clean; 2.7 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 16 MiB/s wr, 260 op/s; 34078/130773 objects degraded (26.059%); 5380/130773 objects misplaced (4.114%); 15 MiB/s, 238 objects/s recovering 2024-09-21T10:58:54.652 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:54 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 34078/130773 objects degraded (26.059%), 23 pgs degraded (PG_DEGRADED) 2024-09-21T10:58:54.687 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:54 smithi070 ceph-mon[107385]: pgmap v2943: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+undersized, 16 active+undersized+degraded, 29 active+clean; 2.7 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 16 MiB/s wr, 260 op/s; 34078/130773 objects degraded (26.059%); 5380/130773 objects misplaced (4.114%); 15 MiB/s, 238 objects/s recovering 2024-09-21T10:58:54.687 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:54 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 34078/130773 objects degraded (26.059%), 23 pgs degraded (PG_DEGRADED) 2024-09-21T10:58:54.687 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:54 smithi070 ceph-mon[112139]: pgmap v2943: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+undersized, 16 active+undersized+degraded, 29 active+clean; 2.7 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 16 MiB/s wr, 260 op/s; 34078/130773 objects degraded (26.059%); 5380/130773 objects misplaced (4.114%); 15 MiB/s, 238 objects/s recovering 2024-09-21T10:58:54.687 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:54 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 34078/130773 objects degraded (26.059%), 23 pgs degraded (PG_DEGRADED) 2024-09-21T10:58:54.724 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:54.745 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:54.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:54.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:54.847 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:54.871 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:54.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:54.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:54.973 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:54.994 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:54.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:54.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:55.096 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:55.121 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:55.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:55.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:55.223 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:55.248 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:55.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:55.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:55.350 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:55.373 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:55.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:55.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:55.475 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:55.498 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:55.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:55.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:55.600 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:55.622 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 39 16 46227 46211 74.0499 25.25 0.00278076 0.0134345 2024-09-21T10:58:55.625 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:55.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:55.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:55.727 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:55.760 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:55.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:55.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:55.861 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:55.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:55.889 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:55.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:55.990 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:56.014 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:56.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:56.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:56.116 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:56.147 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:56.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:56.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:56.249 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:56.275 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:56.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:56.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:56.377 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:56.405 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:56.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:56.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:56.466 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:56 smithi070 ceph-mon[107385]: pgmap v2944: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+degraded+remapped, 2 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+undersized, 16 active+undersized+degraded, 29 active+clean; 2.8 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 23 MiB/s wr, 375 op/s; 34721/135528 objects degraded (25.619%); 5855/135528 objects misplaced (4.320%); 12 MiB/s, 198 objects/s recovering 2024-09-21T10:58:56.467 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:56 smithi070 ceph-mon[112139]: pgmap v2944: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+degraded+remapped, 2 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+undersized, 16 active+undersized+degraded, 29 active+clean; 2.8 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 23 MiB/s wr, 375 op/s; 34721/135528 objects degraded (25.619%); 5855/135528 objects misplaced (4.320%); 12 MiB/s, 198 objects/s recovering 2024-09-21T10:58:56.507 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:56.530 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:56.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:56.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:56.622 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:2024-09-21T10:58:56.621770+0000 min lat: 0.00141297 max lat: 5.2078 avg lat: 0.0136557 2024-09-21T10:58:56.623 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:58:56.623 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 40 16 46662 46646 72.8783 27.1875 0.00241169 0.0136557 2024-09-21T10:58:56.632 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:56.655 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:56.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:56.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:56.756 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:56.784 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:56.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:56.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:56 smithi160 ceph-mon[101879]: pgmap v2944: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+degraded+remapped, 2 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+undersized, 16 active+undersized+degraded, 29 active+clean; 2.8 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 23 MiB/s wr, 375 op/s; 34721/135528 objects degraded (25.619%); 5855/135528 objects misplaced (4.320%); 12 MiB/s, 198 objects/s recovering 2024-09-21T10:58:56.885 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:56.910 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:56.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:56.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:57.011 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:57.040 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:57.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:57.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:57.143 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:57.174 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:57.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:57.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:57.275 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:57.307 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:57.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:57.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:57.409 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:57.440 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:57.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:57.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:57.542 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:57.571 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:57.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:57.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:57.622 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 41 16 47124 47108 71.805 28.875 0.00296146 0.0138655 2024-09-21T10:58:57.673 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:57.696 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:57.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:57.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:57.798 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:57.826 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:57.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:57.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:57.927 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:58:57.955 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:57.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:57.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:58:58.056 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:58.083 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:58.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:58.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:58.185 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:58.198 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:58 smithi070 ceph-mon[107385]: pgmap v2945: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+degraded+remapped, 2 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+undersized, 16 active+undersized+degraded, 29 active+clean; 2.8 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 21 MiB/s wr, 331 op/s; 34721/135528 objects degraded (25.619%); 5855/135528 objects misplaced (4.320%); 11 MiB/s, 175 objects/s recovering 2024-09-21T10:58:58.198 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:58 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:58:58.198 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:58 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:58:58.198 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:58 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/798820139' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:58:58.198 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:58 smithi070 ceph-mon[112139]: pgmap v2945: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+degraded+remapped, 2 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+undersized, 16 active+undersized+degraded, 29 active+clean; 2.8 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 21 MiB/s wr, 331 op/s; 34721/135528 objects degraded (25.619%); 5855/135528 objects misplaced (4.320%); 11 MiB/s, 175 objects/s recovering 2024-09-21T10:58:58.198 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:58 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:58:58.199 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:58 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:58:58.199 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:58 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/798820139' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:58:58.213 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:58.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:58.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:58.315 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:58.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:58.343 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:58.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:58.444 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:58.467 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:58.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:58.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:58.569 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:58.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:58 smithi160 ceph-mon[101879]: pgmap v2945: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+degraded+remapped, 2 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+undersized, 16 active+undersized+degraded, 29 active+clean; 2.8 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 21 MiB/s wr, 331 op/s; 34721/135528 objects degraded (25.619%); 5855/135528 objects misplaced (4.320%); 11 MiB/s, 175 objects/s recovering 2024-09-21T10:58:58.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:58 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:58:58.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:58 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:58:58.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:58 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/798820139' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:58:58.592 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:58.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:58.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:58.622 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 42 16 47676 47660 70.9167 34.5 0.00279431 0.0140438 2024-09-21T10:58:58.665 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:58:58.666 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:58:58.693 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:58.718 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:58.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:58.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:58.820 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:58.842 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:58.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:58.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:58.944 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:58.966 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:58.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:58.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:58.971 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:58:58.971 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:58:58.971 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:58:58.972 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:58:58.972 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:58:58.972 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:58:58.972 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:58:58.972 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:58:58.972 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:58:58.972 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:58:58.972 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:58:58.973 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T10:58:58.973 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T10:58:58.973 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:58:59.068 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:59.092 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:59.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:59.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:59.194 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:58:59.217 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:59.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:59.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:58:59.319 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:58:59.341 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:59.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:59.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:58:59.443 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:58:59.471 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:59.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:59.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:58:59.572 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:59.595 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:59.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:59.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:59.623 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 43 16 48052 48036 69.8139 23.5 0.147018 0.014297 2024-09-21T10:58:59.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:59 smithi070 ceph-mon[107385]: osdmap e1544: 8 total, 6 up, 5 in 2024-09-21T10:58:59.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:58:59 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 34721/135528 objects degraded (25.619%), 23 pgs degraded (PG_DEGRADED) 2024-09-21T10:58:59.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:59 smithi070 ceph-mon[112139]: osdmap e1544: 8 total, 6 up, 5 in 2024-09-21T10:58:59.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:58:59 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 34721/135528 objects degraded (25.619%), 23 pgs degraded (PG_DEGRADED) 2024-09-21T10:58:59.697 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:59.723 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:59.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:59.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:59.824 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:58:59.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:59 smithi160 ceph-mon[101879]: osdmap e1544: 8 total, 6 up, 5 in 2024-09-21T10:58:59.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:58:59 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 34721/135528 objects degraded (25.619%), 23 pgs degraded (PG_DEGRADED) 2024-09-21T10:58:59.848 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:58:59.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:59.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:58:59.950 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:58:59.970 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:58:59.982 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:58:59.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:58:59.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:58:59.995 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:59:00.084 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:00.109 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:00.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:00.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:00.211 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:00.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:00.238 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:00.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:00.339 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:00.366 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:00.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:00.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:00.468 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:00.492 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:00.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:00.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:00.594 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:00.615 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:00.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:00.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:00.623 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 44 16 48619 48603 69.0326 35.4375 0.00284751 0.0144474 2024-09-21T10:59:00.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:00 smithi070 ceph-mon[107385]: pgmap v2947: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 2 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+undersized, 16 active+undersized+degraded, 29 active+clean; 2.8 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 26 MiB/s wr, 420 op/s; 35787/140073 objects degraded (25.549%); 6046/140073 objects misplaced (4.316%); 15 MiB/s, 238 objects/s recovering 2024-09-21T10:59:00.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:00 smithi070 ceph-mon[107385]: osdmap e1545: 8 total, 6 up, 5 in 2024-09-21T10:59:00.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:00 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:59:00.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:00 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:59:00.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:00 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:59:00.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:00 smithi070 ceph-mon[112139]: pgmap v2947: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 2 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+undersized, 16 active+undersized+degraded, 29 active+clean; 2.8 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 26 MiB/s wr, 420 op/s; 35787/140073 objects degraded (25.549%); 6046/140073 objects misplaced (4.316%); 15 MiB/s, 238 objects/s recovering 2024-09-21T10:59:00.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:00 smithi070 ceph-mon[112139]: osdmap e1545: 8 total, 6 up, 5 in 2024-09-21T10:59:00.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:00 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:59:00.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:00 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:59:00.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:00 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:59:00.717 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:00.741 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:00.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:00.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:00 smithi160 ceph-mon[101879]: pgmap v2947: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 2 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+undersized, 16 active+undersized+degraded, 29 active+clean; 2.8 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 26 MiB/s wr, 420 op/s; 35787/140073 objects degraded (25.549%); 6046/140073 objects misplaced (4.316%); 15 MiB/s, 238 objects/s recovering 2024-09-21T10:59:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:00 smithi160 ceph-mon[101879]: osdmap e1545: 8 total, 6 up, 5 in 2024-09-21T10:59:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:00 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:59:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:00 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:59:00.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:00 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:59:00.843 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:00.866 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:00.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:00.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:00.967 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:00.989 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:00.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:00.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:01.090 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:01.120 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:01.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:01.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:01.222 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:01.250 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:01.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:01.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:01.351 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:01.373 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:01.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:01.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:01.418 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T10:59:01.476 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:01.501 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:01.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:01.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:01.581 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:01 smithi070 ceph-mon[107385]: pgmap v2949: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 2 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+undersized, 16 active+undersized+degraded, 29 active+clean; 2.9 GiB data, 15 GiB used, 433 GiB / 447 GiB avail; 34 MiB/s wr, 537 op/s; 36186/142617 objects degraded (25.373%); 6046/142617 objects misplaced (4.239%); 9.5 MiB/s, 152 objects/s recovering 2024-09-21T10:59:01.581 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:01 smithi070 ceph-mon[107385]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T10:59:01.581 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:01 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4065369037' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:59:01.582 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:01 smithi070 ceph-mon[112139]: pgmap v2949: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 2 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+undersized, 16 active+undersized+degraded, 29 active+clean; 2.9 GiB data, 15 GiB used, 433 GiB / 447 GiB avail; 34 MiB/s wr, 537 op/s; 36186/142617 objects degraded (25.373%); 6046/142617 objects misplaced (4.239%); 9.5 MiB/s, 152 objects/s recovering 2024-09-21T10:59:01.582 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:01 smithi070 ceph-mon[112139]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T10:59:01.582 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:01 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4065369037' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:59:01.603 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:01.623 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 45 15 49147 49132 68.2332 33.0625 0.00284529 0.0146051 2024-09-21T10:59:01.624 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:01.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:01.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:01.726 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:01.747 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:01.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:01.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:01 smithi160 ceph-mon[101879]: pgmap v2949: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 2 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+undersized, 16 active+undersized+degraded, 29 active+clean; 2.9 GiB data, 15 GiB used, 433 GiB / 447 GiB avail; 34 MiB/s wr, 537 op/s; 36186/142617 objects degraded (25.373%); 6046/142617 objects misplaced (4.239%); 9.5 MiB/s, 152 objects/s recovering 2024-09-21T10:59:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:01 smithi160 ceph-mon[101879]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T10:59:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:01 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4065369037' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:59:01.848 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:01.869 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:01.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:01.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:01.971 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:01.992 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:01.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:01.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:02.063 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:59:02.063 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:59:02.094 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:02.116 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:02.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:02.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:02.218 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:02.240 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:02.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:02.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:02.341 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:02.364 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:02.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:02.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:02.466 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:02.487 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:02.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:02.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:02.589 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:02.615 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:02.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:02.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:02.623 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 46 16 49600 49584 67.3639 28.25 0.00360096 0.0148107 2024-09-21T10:59:02.717 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:02.740 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:02.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:02.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:02.798 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:02 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:59:02.798 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:02 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4065369037' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:59:02.798 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:02 smithi070 ceph-mon[107385]: osdmap e1546: 8 total, 6 up, 5 in 2024-09-21T10:59:02.799 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:02 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:59:02.799 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:02 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4065369037' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:59:02.799 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:02 smithi070 ceph-mon[112139]: osdmap e1546: 8 total, 6 up, 5 in 2024-09-21T10:59:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:02 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:59:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:02 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4065369037' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:59:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:02 smithi160 ceph-mon[101879]: osdmap e1546: 8 total, 6 up, 5 in 2024-09-21T10:59:02.842 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:02.843 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:59:02.871 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:02.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:02.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:02.973 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:03.004 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:03.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:03.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:03.106 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:03.131 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:03.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:03.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:03.233 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:03.258 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:03.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:03.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:03.360 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:03.383 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:03.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:03.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:03.420 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:59:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:59:03] "GET /metrics HTTP/1.1" 200 36501 "" "Prometheus/2.51.0" 2024-09-21T10:59:03.485 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:03.508 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:03.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:03.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:03.610 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:03.623 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 47 16 50243 50227 66.7857 40.1875 0.00419349 0.0149432 2024-09-21T10:59:03.634 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:03.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:03.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:03.736 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:03.762 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:03.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:03.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:03 smithi160 ceph-mon[101879]: pgmap v2951: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 2 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+undersized, 16 active+undersized+degraded, 29 active+clean; 2.9 GiB data, 15 GiB used, 433 GiB / 447 GiB avail; 28 MiB/s wr, 442 op/s; 36186/142617 objects degraded (25.373%); 6046/142617 objects misplaced (4.239%); 13 MiB/s, 208 objects/s recovering 2024-09-21T10:59:03.913 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:03 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 36186/142617 objects degraded (25.373%), 23 pgs degraded (PG_DEGRADED) 2024-09-21T10:59:03.914 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:03.915 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:03 smithi070 ceph-mon[112139]: pgmap v2951: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 2 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+undersized, 16 active+undersized+degraded, 29 active+clean; 2.9 GiB data, 15 GiB used, 433 GiB / 447 GiB avail; 28 MiB/s wr, 442 op/s; 36186/142617 objects degraded (25.373%); 6046/142617 objects misplaced (4.239%); 13 MiB/s, 208 objects/s recovering 2024-09-21T10:59:03.915 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:03 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 36186/142617 objects degraded (25.373%), 23 pgs degraded (PG_DEGRADED) 2024-09-21T10:59:03.916 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:03 smithi070 ceph-mon[107385]: pgmap v2951: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 2 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+undersized, 16 active+undersized+degraded, 29 active+clean; 2.9 GiB data, 15 GiB used, 433 GiB / 447 GiB avail; 28 MiB/s wr, 442 op/s; 36186/142617 objects degraded (25.373%); 6046/142617 objects misplaced (4.239%); 13 MiB/s, 208 objects/s recovering 2024-09-21T10:59:03.916 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:03 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 36186/142617 objects degraded (25.373%), 23 pgs degraded (PG_DEGRADED) 2024-09-21T10:59:03.939 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:03.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:03.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:04.041 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:04.064 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:59:04.070 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:04.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:04.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:04.172 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:04.195 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:04.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:04.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:04.297 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:04.320 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:04.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:04.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:04.422 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:04.425 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.6 to 0.86832 (de4a) 2024-09-21T10:59:04.448 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:04.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:04.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:04.550 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:04.571 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:04.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:04.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:04.623 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 48 16 50700 50684 65.9893 28.5625 0.00502758 0.0150758 2024-09-21T10:59:04.673 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:04.694 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:04.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:04.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:04.796 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:04.817 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:04.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:04.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:04.824 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:59:04.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:04 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2560684975' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 0.8683198499478187} : dispatch 2024-09-21T10:59:04.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:04 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 0.8683198499478187} : dispatch 2024-09-21T10:59:04.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:04 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2560684975' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 0.8683198499478187} : dispatch 2024-09-21T10:59:04.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:04 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 0.8683198499478187} : dispatch 2024-09-21T10:59:04.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:04 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2560684975' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 0.8683198499478187} : dispatch 2024-09-21T10:59:04.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:04 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 0.8683198499478187} : dispatch 2024-09-21T10:59:04.920 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:04.942 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:04.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:04.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:05.043 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:05.066 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:05.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:05.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:05.167 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:05.190 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:05.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:05.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:05.292 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:05.313 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:05.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:05.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:05.415 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:05.440 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:05.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:05.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:05.543 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:05.564 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:05.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:05.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:05.623 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 49 16 50710 50694 64.6554 0.625 0.0023296 0.0150927 2024-09-21T10:59:05.666 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:05.688 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:05.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:05.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:05.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:05 smithi070 ceph-mon[112139]: pgmap v2952: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 44 active+undersized, 16 active+undersized+degraded, 29 active+clean; 3.0 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 37 MiB/s wr, 587 op/s; 34324/148827 objects degraded (23.063%); 9711/148827 objects misplaced (6.525%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:59:05.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:05 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 0.8683198499478187}]': finished 2024-09-21T10:59:05.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:05 smithi070 ceph-mon[112139]: osdmap e1547: 8 total, 6 up, 5 in 2024-09-21T10:59:05.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:59:05.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:59:05.705 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:05 smithi070 ceph-mon[107385]: pgmap v2952: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 44 active+undersized, 16 active+undersized+degraded, 29 active+clean; 3.0 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 37 MiB/s wr, 587 op/s; 34324/148827 objects degraded (23.063%); 9711/148827 objects misplaced (6.525%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:59:05.705 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:05 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 0.8683198499478187}]': finished 2024-09-21T10:59:05.705 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:05 smithi070 ceph-mon[107385]: osdmap e1547: 8 total, 6 up, 5 in 2024-09-21T10:59:05.705 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:59:05.705 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:59:05.727 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:59:05.727 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:59:05.790 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:05.811 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:05.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:05.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:05 smithi160 ceph-mon[101879]: pgmap v2952: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 44 active+undersized, 16 active+undersized+degraded, 29 active+clean; 3.0 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 37 MiB/s wr, 587 op/s; 34324/148827 objects degraded (23.063%); 9711/148827 objects misplaced (6.525%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:59:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:05 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 0.8683198499478187}]': finished 2024-09-21T10:59:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:05 smithi160 ceph-mon[101879]: osdmap e1547: 8 total, 6 up, 5 in 2024-09-21T10:59:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T10:59:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:59:05.913 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:05.935 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:05.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:05.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:06.037 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:06.058 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:06.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:06.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:06.160 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:06.182 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:06.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:06.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:06.283 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:06.305 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:06.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:06.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:06.406 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:06.415 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:59:06.428 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:06.429 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:59:06.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:06.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:06.531 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:06.553 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:06.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:06.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:06.623 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 50 16 50741 50725 63.401 1.9375 0.0043024 0.0152294 2024-09-21T10:59:06.655 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:06.676 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:06.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:06.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:06.688 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:59:06.730 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:06 smithi070 ceph-mon[107385]: osdmap e1548: 8 total, 6 up, 5 in 2024-09-21T10:59:06.731 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:06 smithi070 ceph-mon[107385]: osdmap e1549: 8 total, 6 up, 5 in 2024-09-21T10:59:06.731 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:06 smithi070 ceph-mon[112139]: osdmap e1548: 8 total, 6 up, 5 in 2024-09-21T10:59:06.731 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:06 smithi070 ceph-mon[112139]: osdmap e1549: 8 total, 6 up, 5 in 2024-09-21T10:59:06.778 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:06.802 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:06.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:06.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:06.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:06 smithi160 ceph-mon[101879]: osdmap e1548: 8 total, 6 up, 5 in 2024-09-21T10:59:06.811 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:06 smithi160 ceph-mon[101879]: osdmap e1549: 8 total, 6 up, 5 in 2024-09-21T10:59:06.903 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:06.926 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:06.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:06.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:07.027 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:07.049 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:07.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:07.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:07.150 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:07.171 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:07.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:07.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:07.208 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:59:07.273 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:07.294 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:07.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:07.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:07.396 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:07.419 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:07.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:07.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:07.521 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:07.543 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:07.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:07.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:07.623 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 51 16 50745 50729 62.1628 0.25 0.00399838 0.0152285 2024-09-21T10:59:07.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:07 smithi070 ceph-mon[107385]: pgmap v2955: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 44 active+undersized, 16 active+undersized+degraded, 29 active+clean; 3.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 27 MiB/s wr, 429 op/s; 34495/150342 objects degraded (22.944%); 9711/150342 objects misplaced (6.459%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:59:07.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:07 smithi070 ceph-mon[112139]: pgmap v2955: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 44 active+undersized, 16 active+undersized+degraded, 29 active+clean; 3.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 27 MiB/s wr, 429 op/s; 34495/150342 objects degraded (22.944%); 9711/150342 objects misplaced (6.459%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:59:07.645 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:07.667 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:07.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:07.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:07.769 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:07.790 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:07.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:07.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:07 smithi160 ceph-mon[101879]: pgmap v2955: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 44 active+undersized, 16 active+undersized+degraded, 29 active+clean; 3.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 27 MiB/s wr, 429 op/s; 34495/150342 objects degraded (22.944%); 9711/150342 objects misplaced (6.459%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:59:07.891 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:07.913 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:07.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:07.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:08.014 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:08.039 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:08.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:08.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:08.141 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:08.162 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:08.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:08.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:08.264 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:08.288 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:08.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:08.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:08.389 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:08.411 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:08.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:08.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:08.513 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:08.534 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:08.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:08.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:08.588 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:08 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 34495/150342 objects degraded (22.944%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T10:59:08.623 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 52 16 50745 50729 60.9673 0 - 0.0152285 2024-09-21T10:59:08.636 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:08.657 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:08.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:08.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:08.714 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:59:08.714 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:59:08.758 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:08.780 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:08.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:08.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:08.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:08 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 34495/150342 objects degraded (22.944%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T10:59:08.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:08 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 34495/150342 objects degraded (22.944%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T10:59:08.882 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:08.902 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:08.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:08.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:09.004 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:09.025 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:09.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:09.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:09.127 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:09.149 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:09.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:09.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:09.251 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:09.273 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:09.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:09.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:09.375 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:09.394 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:59:09.396 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:09.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:09.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:09.498 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:09.520 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:09.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:09.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:09.567 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:59:09.568 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:59:09.588 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:59:09.589 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:59:09.589 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:09 smithi070 ceph-mon[107385]: pgmap v2957: 99 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 44 active+undersized, 14 active+undersized+degraded, 29 active+clean; 3.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 32 MiB/s wr, 508 op/s; 35192/151773 objects degraded (23.187%); 7637/151773 objects misplaced (5.032%); 5.7 MiB/s, 91 objects/s recovering 2024-09-21T10:59:09.589 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:09 smithi070 ceph-mon[112139]: pgmap v2957: 99 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 44 active+undersized, 14 active+undersized+degraded, 29 active+clean; 3.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 32 MiB/s wr, 508 op/s; 35192/151773 objects degraded (23.187%); 7637/151773 objects misplaced (5.032%); 5.7 MiB/s, 91 objects/s recovering 2024-09-21T10:59:09.607 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:59:09.607 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:59:09.622 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:09.623 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:59:09.623 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:59:09.623 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 53 16 50915 50899 60.0174 5.3125 0.00315718 0.0166104 2024-09-21T10:59:09.641 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:59:09.641 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:59:09.641 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T10:59:09.641 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T10:59:09.653 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:09.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:09.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:09.659 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:59:09.659 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:59:09.667 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:59:09.755 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:09.778 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:09.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:09.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:09.800 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:09 smithi160 ceph-mon[101879]: pgmap v2957: 99 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 44 active+undersized, 14 active+undersized+degraded, 29 active+clean; 3.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 32 MiB/s wr, 508 op/s; 35192/151773 objects degraded (23.187%); 7637/151773 objects misplaced (5.032%); 5.7 MiB/s, 91 objects/s recovering 2024-09-21T10:59:09.880 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:09.904 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:09.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:09.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:10.006 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:10.029 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:10.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:10.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:10.131 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:10.155 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:10.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:10.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:10.204 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7, 4] out_osds: [2, 5, 0] dead_osds: [5, 6] live_osds: [0, 1, 2, 3, 4, 7] 2024-09-21T10:59:10.205 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:59:10.205 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-21T10:59:10.205 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:59:10.205 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:59:10.205 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-09-21T10:59:10.256 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:10.279 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:10.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:10.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:10.381 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:10.402 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:10.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:10.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:10.459 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T10:59:10.504 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:10.527 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:10.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:10.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:10.592 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:59:10.593 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:59:10.624 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 54 16 51377 51361 59.4406 28.875 0.0036024 0.0167723 2024-09-21T10:59:10.629 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:10.651 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:10.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:10.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:10.752 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:10.774 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:10.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:10.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:10.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:10 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3383544138' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:59:10.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:10 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2961018685' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:59:10.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:10 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:59:10.875 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:10.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:10 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3383544138' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:59:10.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:10 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2961018685' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:59:10.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:10 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:59:10.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:10 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3383544138' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:59:10.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:10 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2961018685' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:59:10.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:10 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:59:10.897 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:10.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:10.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:10.998 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:11.019 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:11.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:11.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:11.120 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:11.143 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:11.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:11.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:11.244 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:11.267 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:11.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:11.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:11.369 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:11.390 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:59:11.397 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:11.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:11.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:11.499 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:11.522 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:11.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:11.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:11.564 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:11 smithi160 ceph-mon[101879]: pgmap v2958: 99 pgs: 1 active+recovery_wait+remapped, 2 activating+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 12 MiB/s wr, 197 op/s; 34777/152376 objects degraded (22.823%); 6876/152376 objects misplaced (4.513%); 4.8 MiB/s, 76 objects/s recovering 2024-09-21T10:59:11.564 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:11 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:59:11.565 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:11 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:59:11.565 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:11 smithi160 ceph-mon[101879]: osdmap e1550: 8 total, 6 up, 5 in 2024-09-21T10:59:11.597 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:59:11.597 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T10:59:11.597 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T10:59:11.597 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T10:59:11.597 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:59:11.623 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:11.624 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 55 16 51821 51805 58.8644 27.75 0.130394 0.0169304 2024-09-21T10:59:11.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:11 smithi070 ceph-mon[107385]: pgmap v2958: 99 pgs: 1 active+recovery_wait+remapped, 2 activating+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 12 MiB/s wr, 197 op/s; 34777/152376 objects degraded (22.823%); 6876/152376 objects misplaced (4.513%); 4.8 MiB/s, 76 objects/s recovering 2024-09-21T10:59:11.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:11 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:59:11.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:11 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:59:11.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:11 smithi070 ceph-mon[107385]: osdmap e1550: 8 total, 6 up, 5 in 2024-09-21T10:59:11.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:11 smithi070 ceph-mon[112139]: pgmap v2958: 99 pgs: 1 active+recovery_wait+remapped, 2 activating+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 12 MiB/s wr, 197 op/s; 34777/152376 objects degraded (22.823%); 6876/152376 objects misplaced (4.513%); 4.8 MiB/s, 76 objects/s recovering 2024-09-21T10:59:11.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:11 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:59:11.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:11 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:59:11.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:11 smithi070 ceph-mon[112139]: osdmap e1550: 8 total, 6 up, 5 in 2024-09-21T10:59:11.648 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:11.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:11.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:11.750 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:11.771 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:11.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:11.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:11.873 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:11.895 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:11.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:11.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:11.997 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:12.019 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:12.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:12.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:12.121 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:12.144 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:12.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:12.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:12.246 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:12.275 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:12.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:12.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:12.314 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:59:12.376 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:12.427 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:12.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:12.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:12.529 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:12.551 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:12.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:12.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:12.593 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:59:12.620 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:59:12.624 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 56 16 52244 52228 58.2853 26.4375 0.00298117 0.0171096 2024-09-21T10:59:12.653 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:12.677 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:12.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:12.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:12.778 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:12.810 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:12.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:12.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:12.912 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:12.934 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:12.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:12.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:13.036 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:13.058 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:13.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:13.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:13.160 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:13.185 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:13.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:13.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:13.212 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:59:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:59:13] "GET /metrics HTTP/1.1" 200 36577 "" "Prometheus/2.51.0" 2024-09-21T10:59:13.223 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:59:13.223 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "" 2024-09-21T10:59:13.223 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:59:13.287 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:13.308 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:13.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:13.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:13.397 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:59:13.410 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:13.463 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:13.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:13.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:13.565 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:13.576 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:13 smithi070 ceph-mon[107385]: pgmap v2960: 99 pgs: 1 active+recovery_wait+remapped, 2 activating+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 6.1 MiB/s wr, 97 op/s; 34777/152376 objects degraded (22.823%); 6876/152376 objects misplaced (4.513%); 4.1 MiB/s, 65 objects/s recovering 2024-09-21T10:59:13.576 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:13 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 34777/152376 objects degraded (22.823%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T10:59:13.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:13 smithi070 ceph-mon[112139]: pgmap v2960: 99 pgs: 1 active+recovery_wait+remapped, 2 activating+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 6.1 MiB/s wr, 97 op/s; 34777/152376 objects degraded (22.823%); 6876/152376 objects misplaced (4.513%); 4.1 MiB/s, 65 objects/s recovering 2024-09-21T10:59:13.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:13 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 34777/152376 objects degraded (22.823%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T10:59:13.597 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:13.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:13.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:13.624 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 57 16 52707 52691 57.7704 28.9375 0.00289865 0.0172677 2024-09-21T10:59:13.700 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:13.721 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:13.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:13.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:13.823 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:13 smithi160 ceph-mon[101879]: pgmap v2960: 99 pgs: 1 active+recovery_wait+remapped, 2 activating+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 6.1 MiB/s wr, 97 op/s; 34777/152376 objects degraded (22.823%); 6876/152376 objects misplaced (4.513%); 4.1 MiB/s, 65 objects/s recovering 2024-09-21T10:59:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:13 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 34777/152376 objects degraded (22.823%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T10:59:13.845 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:13.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:13.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:13.946 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:13.978 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:13.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:13.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:14.045 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:59:14.080 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:14.107 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:14.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:14.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:14.209 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:14.232 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:14.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:14.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:14.334 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:14.356 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:14.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:14.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:14.458 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:14.481 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:14.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:14.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:14.583 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:14.617 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:14.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:14.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:14.624 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 58 16 53243 53227 57.3519 33.5 0.00333916 0.0174047 2024-09-21T10:59:14.692 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T10:59:14.693 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T10:59:14.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T10:59:14.693 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T10:59:14.693 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.693 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:130860 55.d 55:b4ee1c49:::benchmark_data_smithi070_294394_object25519:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:59:14.693 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:26.394761+0000", 2024-09-21T10:59:14.693 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 468.29267118799999, 2024-09-21T10:59:14.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3490294309999999, 2024-09-21T10:59:14.693 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:59:14.693 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:59:14.694 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:59:14.694 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:59:14.694 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:59:14.694 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 130860 2024-09-21T10:59:14.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.694 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:59:14.694 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.694 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:59:14.694 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.394761+0000", 2024-09-21T10:59:14.694 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.694 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:59:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.394761+0000", 2024-09-21T10:59:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:59:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.394758+0000", 2024-09-21T10:59:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:59:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:59:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.394768+0000", 2024-09-21T10:59:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2429999999999994e-06 2024-09-21T10:59:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:59:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.394788+0000", 2024-09-21T10:59:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9777000000000001e-05 2024-09-21T10:59:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:59:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.394796+0000", 2024-09-21T10:59:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8040000000000007e-06 2024-09-21T10:59:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.394852+0000", 2024-09-21T10:59:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5896999999999997e-05 2024-09-21T10:59:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:59:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.395006+0000", 2024-09-21T10:59:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015377699999999999 2024-09-21T10:59:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:59:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.395149+0000", 2024-09-21T10:59:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000143514 2024-09-21T10:59:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:59:14.698 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.397840+0000", 2024-09-21T10:59:14.698 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0026902039999999999 2024-09-21T10:59:14.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.698 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.698 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.414884+0000", 2024-09-21T10:59:14.698 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.017044549999999999 2024-09-21T10:59:14.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.698 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.698 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.429059+0000", 2024-09-21T10:59:14.698 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.014175128 2024-09-21T10:59:14.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:59:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.429074+0000", 2024-09-21T10:59:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4436e-05 2024-09-21T10:59:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.427445+0000", 2024-09-21T10:59:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99837087800000002 2024-09-21T10:59:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:59:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.427457+0000", 2024-09-21T10:59:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2643000000000001e-05 2024-09-21T10:59:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.691524+0000", 2024-09-21T10:59:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2640668900000001 2024-09-21T10:59:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for missing object", 2024-09-21T10:59:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.691640+0000", 2024-09-21T10:59:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000116205 2024-09-21T10:59:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.743637+0000", 2024-09-21T10:59:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.051996268999999998 2024-09-21T10:59:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:59:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.743790+0000", 2024-09-21T10:59:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015328100000000001 2024-09-21T10:59:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:59:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:130862 55.d 55:bd7eda03:::benchmark_data_smithi070_294394_object25521:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:59:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:26.398474+0000", 2024-09-21T10:59:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 468.28895774400002, 2024-09-21T10:59:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3783652179999999, 2024-09-21T10:59:14.702 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:59:14.702 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:59:14.702 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:59:14.702 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:59:14.702 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:59:14.702 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 130862 2024-09-21T10:59:14.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.702 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:59:14.702 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.702 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:59:14.702 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398474+0000", 2024-09-21T10:59:14.702 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:59:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398474+0000", 2024-09-21T10:59:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:59:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398472+0000", 2024-09-21T10:59:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:59:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:59:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398481+0000", 2024-09-21T10:59:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1589999999999993e-06 2024-09-21T10:59:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:59:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398495+0000", 2024-09-21T10:59:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4829e-05 2024-09-21T10:59:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:59:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398503+0000", 2024-09-21T10:59:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1499999999999999e-06 2024-09-21T10:59:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398549+0000", 2024-09-21T10:59:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5802e-05 2024-09-21T10:59:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:59:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398680+0000", 2024-09-21T10:59:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000130784 2024-09-21T10:59:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:59:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398797+0000", 2024-09-21T10:59:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000117408 2024-09-21T10:59:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:59:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.402741+0000", 2024-09-21T10:59:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0039439920000000003 2024-09-21T10:59:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.414917+0000", 2024-09-21T10:59:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012175314 2024-09-21T10:59:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.430736+0000", 2024-09-21T10:59:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.015818895999999999 2024-09-21T10:59:14.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.707 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:59:14.707 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.430747+0000", 2024-09-21T10:59:14.707 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1589e-05 2024-09-21T10:59:14.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.708 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.708 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.427498+0000", 2024-09-21T10:59:14.708 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99675095499999999 2024-09-21T10:59:14.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.708 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:59:14.708 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.427507+0000", 2024-09-21T10:59:14.708 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9830000000000002e-06 2024-09-21T10:59:14.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.750176+0000", 2024-09-21T10:59:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3226684440000001 2024-09-21T10:59:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for missing object", 2024-09-21T10:59:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.750312+0000", 2024-09-21T10:59:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013604399999999999 2024-09-21T10:59:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.776747+0000", 2024-09-21T10:59:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.026435199 2024-09-21T10:59:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:59:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.776839+0000", 2024-09-21T10:59:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2300999999999996e-05 2024-09-21T10:59:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:59:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:130864 55.d 55:bf76459a:::benchmark_data_smithi070_294394_object25523:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:59:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:51:26.398512+0000", 2024-09-21T10:59:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 468.28891924700002, 2024-09-21T10:59:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.415816907, 2024-09-21T10:59:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:59:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:59:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:59:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:59:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:59:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 130864 2024-09-21T10:59:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:59:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:59:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398512+0000", 2024-09-21T10:59:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:59:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398512+0000", 2024-09-21T10:59:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:59:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398512+0000", 2024-09-21T10:59:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T10:59:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:59:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398515+0000", 2024-09-21T10:59:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4679999999999999e-06 2024-09-21T10:59:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:59:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398530+0000", 2024-09-21T10:59:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5201000000000001e-05 2024-09-21T10:59:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:59:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.398535+0000", 2024-09-21T10:59:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0250000000000002e-06 2024-09-21T10:59:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.400838+0000", 2024-09-21T10:59:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0023031710000000001 2024-09-21T10:59:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:59:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.400934+0000", 2024-09-21T10:59:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6107999999999995e-05 2024-09-21T10:59:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:59:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.401033+0000", 2024-09-21T10:59:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9264000000000004e-05 2024-09-21T10:59:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:59:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.402893+0000", 2024-09-21T10:59:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0018592560000000001 2024-09-21T10:59:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.430776+0000", 2024-09-21T10:59:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.027883080000000001 2024-09-21T10:59:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:59:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:26.430791+0000", 2024-09-21T10:59:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5741999999999999e-05 2024-09-21T10:59:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.427535+0000", 2024-09-21T10:59:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99674326099999999 2024-09-21T10:59:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:59:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:27.427543+0000", 2024-09-21T10:59:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5269999999999998e-06 2024-09-21T10:59:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.750345+0000", 2024-09-21T10:59:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.322801291 2024-09-21T10:59:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for missing object", 2024-09-21T10:59:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.750449+0000", 2024-09-21T10:59:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000104609 2024-09-21T10:59:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.814210+0000", 2024-09-21T10:59:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.063760777000000005 2024-09-21T10:59:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:59:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:51:28.814329+0000", 2024-09-21T10:59:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119499 2024-09-21T10:59:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:59:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:156264 55.d 55:b68e274d:::benchmark_data_smithi070_294394_object50923:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1464)", 2024-09-21T10:59:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:24.325793+0000", 2024-09-21T10:59:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 410.36163919000001, 2024-09-21T10:59:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4867916409999999, 2024-09-21T10:59:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:59:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:59:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:59:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:59:14.719 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:59:14.719 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 156264 2024-09-21T10:59:14.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.719 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:59:14.719 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.719 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:59:14.719 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.325793+0000", 2024-09-21T10:59:14.719 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.719 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.719 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:59:14.719 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.325793+0000", 2024-09-21T10:59:14.719 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.720 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.720 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:59:14.720 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.325791+0000", 2024-09-21T10:59:14.720 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:59:14.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.720 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.720 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:59:14.720 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.325798+0000", 2024-09-21T10:59:14.720 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3939999999999998e-06 2024-09-21T10:59:14.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.720 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.720 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:59:14.720 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.325814+0000", 2024-09-21T10:59:14.720 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5532e-05 2024-09-21T10:59:14.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.721 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.721 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:59:14.721 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.325824+0000", 2024-09-21T10:59:14.721 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9739999999999997e-06 2024-09-21T10:59:14.721 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.721 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.721 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.721 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.335430+0000", 2024-09-21T10:59:14.721 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0096062189999999992 2024-09-21T10:59:14.721 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.721 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.721 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:59:14.721 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.336139+0000", 2024-09-21T10:59:14.721 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00070871200000000001 2024-09-21T10:59:14.722 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.722 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.722 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.722 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.347686+0000", 2024-09-21T10:59:14.722 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.01154722 2024-09-21T10:59:14.722 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.722 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.722 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:59:14.722 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.347760+0000", 2024-09-21T10:59:14.722 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4179999999999998e-05 2024-09-21T10:59:14.722 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.722 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.722 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3,6", 2024-09-21T10:59:14.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.347846+0000", 2024-09-21T10:59:14.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5513000000000006e-05 2024-09-21T10:59:14.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.723 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.350917+0000", 2024-09-21T10:59:14.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0030714679999999999 2024-09-21T10:59:14.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.723 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:59:14.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.350930+0000", 2024-09-21T10:59:14.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3365e-05 2024-09-21T10:59:14.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.723 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.354685+0000", 2024-09-21T10:59:14.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003754856 2024-09-21T10:59:14.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.470185+0000", 2024-09-21T10:59:14.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.115499896 2024-09-21T10:59:14.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:59:14.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.470199+0000", 2024-09-21T10:59:14.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3498e-05 2024-09-21T10:59:14.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.725 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.725 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.466078+0000", 2024-09-21T10:59:14.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99587892300000003 2024-09-21T10:59:14.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.725 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:59:14.725 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.466088+0000", 2024-09-21T10:59:14.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0428000000000001e-05 2024-09-21T10:59:14.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.725 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.725 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.688170+0000", 2024-09-21T10:59:14.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.222082468 2024-09-21T10:59:14.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.726 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for missing object", 2024-09-21T10:59:14.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.688260+0000", 2024-09-21T10:59:14.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9816999999999997e-05 2024-09-21T10:59:14.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.726 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.812498+0000", 2024-09-21T10:59:14.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12423751299999999 2024-09-21T10:59:14.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.726 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:59:14.727 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.812584+0000", 2024-09-21T10:59:14.727 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6431e-05 2024-09-21T10:59:14.727 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.727 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:59:14.727 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.727 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.727 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:156285 55.d 55:b5785a76:::benchmark_data_smithi070_294394_object50944:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1464)", 2024-09-21T10:59:14.727 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:24.368417+0000", 2024-09-21T10:59:14.727 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 410.31901503099999, 2024-09-21T10:59:14.727 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4562967860000002, 2024-09-21T10:59:14.727 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:59:14.727 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:59:14.727 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:59:14.728 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:59:14.728 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:59:14.728 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 156285 2024-09-21T10:59:14.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.728 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:59:14.728 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.728 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:59:14.728 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.368417+0000", 2024-09-21T10:59:14.728 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.728 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.728 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:59:14.728 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.368417+0000", 2024-09-21T10:59:14.728 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.729 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.729 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:59:14.729 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.368415+0000", 2024-09-21T10:59:14.729 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:59:14.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.729 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.729 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:59:14.729 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.368422+0000", 2024-09-21T10:59:14.729 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4130000000000003e-06 2024-09-21T10:59:14.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.729 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.729 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:59:14.729 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.368435+0000", 2024-09-21T10:59:14.730 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3662999999999999e-05 2024-09-21T10:59:14.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:59:14.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.368442+0000", 2024-09-21T10:59:14.730 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4910000000000002e-06 2024-09-21T10:59:14.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.368497+0000", 2024-09-21T10:59:14.730 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5174999999999999e-05 2024-09-21T10:59:14.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:59:14.731 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.369528+0000", 2024-09-21T10:59:14.731 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0010314199999999999 2024-09-21T10:59:14.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.731 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.376862+0000", 2024-09-21T10:59:14.731 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0073339119999999997 2024-09-21T10:59:14.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:59:14.731 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.376932+0000", 2024-09-21T10:59:14.731 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9691999999999996e-05 2024-09-21T10:59:14.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.732 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3,6", 2024-09-21T10:59:14.732 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.377022+0000", 2024-09-21T10:59:14.732 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0498e-05 2024-09-21T10:59:14.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.732 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.732 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:59:14.732 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.380265+0000", 2024-09-21T10:59:14.732 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0032423859999999999 2024-09-21T10:59:14.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.732 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.732 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.732 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.380299+0000", 2024-09-21T10:59:14.732 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4088000000000003e-05 2024-09-21T10:59:14.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.733 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.733 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.384814+0000", 2024-09-21T10:59:14.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0045151929999999998 2024-09-21T10:59:14.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.733 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.733 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.470384+0000", 2024-09-21T10:59:14.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.085570246000000003 2024-09-21T10:59:14.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.733 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:59:14.733 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.470393+0000", 2024-09-21T10:59:14.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4880000000000007e-06 2024-09-21T10:59:14.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.734 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.734 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.466114+0000", 2024-09-21T10:59:14.734 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99572154000000002 2024-09-21T10:59:14.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.734 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:59:14.734 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.466122+0000", 2024-09-21T10:59:14.734 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1269999999999997e-06 2024-09-21T10:59:14.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.734 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.734 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.812761+0000", 2024-09-21T10:59:14.735 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3466391930000001 2024-09-21T10:59:14.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.735 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.735 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for missing object", 2024-09-21T10:59:14.735 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.812878+0000", 2024-09-21T10:59:14.735 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011688099999999999 2024-09-21T10:59:14.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.735 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.735 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.735 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.824639+0000", 2024-09-21T10:59:14.735 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011761751000000001 2024-09-21T10:59:14.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.735 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.735 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:59:14.736 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.824713+0000", 2024-09-21T10:59:14.736 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4164000000000003e-05 2024-09-21T10:59:14.736 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.736 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:59:14.736 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.736 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.736 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:156310 55.d 55:b751137b:::benchmark_data_smithi070_294394_object50969:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1465)", 2024-09-21T10:59:14.736 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:24.538751+0000", 2024-09-21T10:59:14.736 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 410.14868100799998, 2024-09-21T10:59:14.736 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3296688460000001, 2024-09-21T10:59:14.736 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:59:14.736 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:59:14.736 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:59:14.737 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:59:14.737 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:59:14.737 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 156310 2024-09-21T10:59:14.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.737 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:59:14.737 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.737 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:59:14.737 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.538751+0000", 2024-09-21T10:59:14.737 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.737 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.737 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:59:14.737 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.538751+0000", 2024-09-21T10:59:14.737 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.738 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.738 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:59:14.738 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.538748+0000", 2024-09-21T10:59:14.738 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 2024-09-21T10:59:14.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.738 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.738 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:59:14.738 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.538759+0000", 2024-09-21T10:59:14.738 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0808000000000001e-05 2024-09-21T10:59:14.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.738 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.738 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:59:14.738 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.538782+0000", 2024-09-21T10:59:14.739 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.268e-05 2024-09-21T10:59:14.739 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.739 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.739 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:59:14.739 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.538791+0000", 2024-09-21T10:59:14.739 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1589999999999999e-06 2024-09-21T10:59:14.739 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.739 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.739 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.739 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.538842+0000", 2024-09-21T10:59:14.739 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1545000000000002e-05 2024-09-21T10:59:14.739 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.739 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.739 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:59:14.740 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.538851+0000", 2024-09-21T10:59:14.740 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8959999999999992e-06 2024-09-21T10:59:14.740 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.740 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.740 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.740 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.466142+0000", 2024-09-21T10:59:14.740 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.92729063599999995 2024-09-21T10:59:14.740 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.740 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.740 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:59:14.740 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.466148+0000", 2024-09-21T10:59:14.740 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6679999999999998e-06 2024-09-21T10:59:14.740 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.740 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.741 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.741 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.812905+0000", 2024-09-21T10:59:14.741 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3467571700000001 2024-09-21T10:59:14.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.741 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.741 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:59:14.741 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.813970+0000", 2024-09-21T10:59:14.741 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0010657570000000001 2024-09-21T10:59:14.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.741 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.741 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.741 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.831790+0000", 2024-09-21T10:59:14.741 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.01781928 2024-09-21T10:59:14.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.742 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.742 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:59:14.742 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.831884+0000", 2024-09-21T10:59:14.742 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4627000000000001e-05 2024-09-21T10:59:14.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.742 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.742 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:59:14.742 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.832016+0000", 2024-09-21T10:59:14.742 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000131676 2024-09-21T10:59:14.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.742 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.742 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:59:14.742 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.849842+0000", 2024-09-21T10:59:14.742 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.017825769000000002 2024-09-21T10:59:14.743 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.743 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.743 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.743 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.849892+0000", 2024-09-21T10:59:14.743 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0278000000000002e-05 2024-09-21T10:59:14.743 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.743 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.743 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.743 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.865035+0000", 2024-09-21T10:59:14.743 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.015142689000000001 2024-09-21T10:59:14.743 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.743 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.744 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:59:14.744 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.865082+0000", 2024-09-21T10:59:14.744 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7398000000000003e-05 2024-09-21T10:59:14.744 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.744 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.744 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:59:14.744 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.868420+0000", 2024-09-21T10:59:14.744 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0033374279999999999 2024-09-21T10:59:14.744 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.744 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:59:14.744 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.744 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.744 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.744 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:156316 55.d 55:b4941ee3:::benchmark_data_smithi070_294394_object50975:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1465)", 2024-09-21T10:59:14.745 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:24.602976+0000", 2024-09-21T10:59:14.745 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 410.08445601800003, 2024-09-21T10:59:14.745 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2688839519999999, 2024-09-21T10:59:14.745 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:59:14.745 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:59:14.745 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:59:14.745 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:59:14.745 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:59:14.745 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 156316 2024-09-21T10:59:14.745 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.745 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:59:14.745 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.745 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:59:14.745 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.602976+0000", 2024-09-21T10:59:14.745 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.746 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.746 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:59:14.746 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.602976+0000", 2024-09-21T10:59:14.746 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.746 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.746 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:59:14.746 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.602974+0000", 2024-09-21T10:59:14.746 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:59:14.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.746 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.746 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:59:14.746 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.602983+0000", 2024-09-21T10:59:14.747 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0420000000000008e-06 2024-09-21T10:59:14.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.747 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.747 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:59:14.747 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.603000+0000", 2024-09-21T10:59:14.747 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6609e-05 2024-09-21T10:59:14.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.747 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.747 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:59:14.747 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.603007+0000", 2024-09-21T10:59:14.747 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7079999999999997e-06 2024-09-21T10:59:14.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.747 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.747 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.748 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.603048+0000", 2024-09-21T10:59:14.748 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1103999999999998e-05 2024-09-21T10:59:14.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.748 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.748 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:59:14.748 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:24.603056+0000", 2024-09-21T10:59:14.748 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6070000000000002e-06 2024-09-21T10:59:14.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.748 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.748 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.748 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.466166+0000", 2024-09-21T10:59:14.748 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.86311000800000004 2024-09-21T10:59:14.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.748 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.749 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:59:14.749 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:14.750 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.466172+0000", 2024-09-21T10:59:14.750 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8579999999999996e-06 2024-09-21T10:59:14.751 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.751 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.751 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.751 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.814030+0000", 2024-09-21T10:59:14.751 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.347857885 2024-09-21T10:59:14.751 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.751 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.751 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:59:14.751 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.814610+0000", 2024-09-21T10:59:14.751 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00058030699999999996 2024-09-21T10:59:14.751 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.751 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.751 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.752 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.840300+0000", 2024-09-21T10:59:14.752 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.025689740999999999 2024-09-21T10:59:14.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.752 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.752 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:59:14.752 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.840399+0000", 2024-09-21T10:59:14.752 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8877999999999998e-05 2024-09-21T10:59:14.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.752 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.752 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:59:14.752 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.840499+0000", 2024-09-21T10:59:14.752 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000100562 2024-09-21T10:59:14.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.753 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.753 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:59:14.753 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.855306+0000", 2024-09-21T10:59:14.753 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.014806487 2024-09-21T10:59:14.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.753 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.753 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.753 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.868479+0000", 2024-09-21T10:59:14.753 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.013173213 2024-09-21T10:59:14.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.753 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.753 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.753 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.868514+0000", 2024-09-21T10:59:14.754 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4925000000000002e-05 2024-09-21T10:59:14.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.754 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.754 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:59:14.754 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.868568+0000", 2024-09-21T10:59:14.754 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4514e-05 2024-09-21T10:59:14.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.754 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.754 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:59:14.754 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.871860+0000", 2024-09-21T10:59:14.754 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0032912369999999998 2024-09-21T10:59:14.755 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.755 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:59:14.755 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.755 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.755 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35670.0:156330 55.d 55:b71364a5:::benchmark_data_smithi070_294394_object50989:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1466)", 2024-09-21T10:59:14.755 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:52:25.802103+0000", 2024-09-21T10:59:14.755 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 408.88532878699999, 2024-09-21T10:59:14.755 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0730541469999999, 2024-09-21T10:59:14.755 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:59:14.755 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:59:14.755 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:59:14.756 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35670", 2024-09-21T10:59:14.756 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:59:14.756 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 156330 2024-09-21T10:59:14.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.756 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:59:14.756 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.756 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:59:14.756 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.802103+0000", 2024-09-21T10:59:14.756 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.756 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.756 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:59:14.757 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.802103+0000", 2024-09-21T10:59:14.757 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.757 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.757 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.757 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:59:14.757 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.802101+0000", 2024-09-21T10:59:14.757 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:59:14.757 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.757 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.757 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:59:14.757 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.802108+0000", 2024-09-21T10:59:14.757 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.973e-06 2024-09-21T10:59:14.757 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.757 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.758 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:59:14.758 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.802123+0000", 2024-09-21T10:59:14.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4664999999999999e-05 2024-09-21T10:59:14.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.758 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:59:14.758 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.802132+0000", 2024-09-21T10:59:14.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0799999999999995e-06 2024-09-21T10:59:14.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.758 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.758 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.802172+0000", 2024-09-21T10:59:14.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0259000000000002e-05 2024-09-21T10:59:14.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:59:14.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:25.802180+0000", 2024-09-21T10:59:14.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8529999999999992e-06 2024-09-21T10:59:14.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.814647+0000", 2024-09-21T10:59:14.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0124667549999999 2024-09-21T10:59:14.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T10:59:14.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.815703+0000", 2024-09-21T10:59:14.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001055708 2024-09-21T10:59:14.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.760 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.760 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.847320+0000", 2024-09-21T10:59:14.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.031617520000000003 2024-09-21T10:59:14.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.760 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:59:14.760 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.847431+0000", 2024-09-21T10:59:14.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000110976 2024-09-21T10:59:14.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.760 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:59:14.761 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.847571+0000", 2024-09-21T10:59:14.761 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014008400000000001 2024-09-21T10:59:14.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.761 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:59:14.761 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.862793+0000", 2024-09-21T10:59:14.761 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.01522224 2024-09-21T10:59:14.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.761 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.761 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.871910+0000", 2024-09-21T10:59:14.761 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0091163189999999995 2024-09-21T10:59:14.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.761 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.762 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.871943+0000", 2024-09-21T10:59:14.762 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3519000000000002e-05 2024-09-21T10:59:14.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.762 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.762 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:59:14.762 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.871999+0000", 2024-09-21T10:59:14.762 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6032000000000002e-05 2024-09-21T10:59:14.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.762 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.762 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:59:14.762 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:52:26.875157+0000", 2024-09-21T10:59:14.762 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0031577369999999999 2024-09-21T10:59:14.762 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.762 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:59:14.763 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.763 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.763 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.763 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:29067 57.9 57:964842a6:::benchmark_data_smithi070_360420_object29066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1537)", 2024-09-21T10:59:14.763 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:58:32.636312+0000", 2024-09-21T10:59:14.763 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 42.051119380000003, 2024-09-21T10:59:14.763 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0001596829999999, 2024-09-21T10:59:14.763 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:59:14.763 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T10:59:14.763 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:59:14.763 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T10:59:14.763 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T10:59:14.763 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 29067 2024-09-21T10:59:14.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.764 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:59:14.764 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.764 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:59:14.764 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.636312+0000", 2024-09-21T10:59:14.764 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.764 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.764 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:59:14.764 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.636312+0000", 2024-09-21T10:59:14.764 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.764 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.764 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:59:14.765 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.636311+0000", 2024-09-21T10:59:14.765 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:59:14.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.765 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.765 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:59:14.765 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.637030+0000", 2024-09-21T10:59:14.765 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00071977200000000003 2024-09-21T10:59:14.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.765 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.765 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:59:14.765 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.637045+0000", 2024-09-21T10:59:14.765 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4175999999999999e-05 2024-09-21T10:59:14.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.765 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.766 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:59:14.766 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.637054+0000", 2024-09-21T10:59:14.766 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0489999999999993e-06 2024-09-21T10:59:14.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.766 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.766 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.637103+0000", 2024-09-21T10:59:14.766 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9271999999999999e-05 2024-09-21T10:59:14.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.766 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op must wait for map", 2024-09-21T10:59:14.766 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.637110+0000", 2024-09-21T10:59:14.766 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7989999999999996e-06 2024-09-21T10:59:14.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.767 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.767 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.648728+0000", 2024-09-21T10:59:14.767 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011618694000000001 2024-09-21T10:59:14.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.767 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:59:14.767 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.648738+0000", 2024-09-21T10:59:14.767 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0131e-05 2024-09-21T10:59:14.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.767 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.767 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.636419+0000", 2024-09-21T10:59:14.767 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98768005199999998 2024-09-21T10:59:14.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.768 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:59:14.768 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.636472+0000", 2024-09-21T10:59:14.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3508000000000003e-05 2024-09-21T10:59:14.768 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.768 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:59:14.768 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.768 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:29090 57.9 57:9c9b276d:::benchmark_data_smithi070_360420_object29089:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1537)", 2024-09-21T10:59:14.768 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:58:32.649590+0000", 2024-09-21T10:59:14.768 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 42.037841577000002, 2024-09-21T10:59:14.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99083443800000004, 2024-09-21T10:59:14.769 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:59:14.769 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:59:14.769 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:59:14.769 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T10:59:14.769 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T10:59:14.769 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 29090 2024-09-21T10:59:14.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.769 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:59:14.769 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.769 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:59:14.769 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.649590+0000", 2024-09-21T10:59:14.769 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.769 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.770 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:59:14.770 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.649590+0000", 2024-09-21T10:59:14.770 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.770 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.770 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:59:14.770 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.649588+0000", 2024-09-21T10:59:14.770 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 2024-09-21T10:59:14.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.770 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.770 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:59:14.770 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.649630+0000", 2024-09-21T10:59:14.770 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2567000000000001e-05 2024-09-21T10:59:14.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.771 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.771 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:59:14.771 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.649644+0000", 2024-09-21T10:59:14.771 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3338e-05 2024-09-21T10:59:14.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.771 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.771 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:59:14.771 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.649652+0000", 2024-09-21T10:59:14.771 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8439999999999998e-06 2024-09-21T10:59:14.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.771 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.771 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.771 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.649774+0000", 2024-09-21T10:59:14.772 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012225800000000001 2024-09-21T10:59:14.772 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.772 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.772 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:59:14.772 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.649784+0000", 2024-09-21T10:59:14.772 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7680000000000003e-06 2024-09-21T10:59:14.772 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.772 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.772 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.772 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.636744+0000", 2024-09-21T10:59:14.772 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98696035900000001 2024-09-21T10:59:14.772 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.772 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.773 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:59:14.773 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.636871+0000", 2024-09-21T10:59:14.773 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012668 2024-09-21T10:59:14.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.773 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:59:14.773 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.637029+0000", 2024-09-21T10:59:14.773 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015828899999999999 2024-09-21T10:59:14.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.773 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.773 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.640258+0000", 2024-09-21T10:59:14.773 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003229216 2024-09-21T10:59:14.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.774 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.774 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.774 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.640287+0000", 2024-09-21T10:59:14.774 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8941999999999998e-05 2024-09-21T10:59:14.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.774 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.774 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:59:14.774 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.640330+0000", 2024-09-21T10:59:14.774 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.295e-05 2024-09-21T10:59:14.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.774 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.774 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:59:14.774 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.640386+0000", 2024-09-21T10:59:14.774 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6097999999999997e-05 2024-09-21T10:59:14.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.775 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.775 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:59:14.775 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.640425+0000", 2024-09-21T10:59:14.775 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8421999999999999e-05 2024-09-21T10:59:14.775 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.775 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:59:14.775 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.775 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.775 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:29099 57.9 57:91a3677e:::benchmark_data_smithi070_360420_object29098:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1537)", 2024-09-21T10:59:14.775 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:58:32.658256+0000", 2024-09-21T10:59:14.775 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 42.029175244999998, 2024-09-21T10:59:14.776 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.983558767, 2024-09-21T10:59:14.776 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:59:14.776 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:59:14.776 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:59:14.776 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T10:59:14.776 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T10:59:14.776 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 29099 2024-09-21T10:59:14.776 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.776 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:59:14.776 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.776 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:59:14.776 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.658256+0000", 2024-09-21T10:59:14.776 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.776 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.777 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:59:14.777 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.658256+0000", 2024-09-21T10:59:14.777 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.777 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:59:14.777 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.658255+0000", 2024-09-21T10:59:14.777 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T10:59:14.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.777 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:59:14.777 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.659800+0000", 2024-09-21T10:59:14.777 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0015451600000000001 2024-09-21T10:59:14.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.778 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.778 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:59:14.778 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.659812+0000", 2024-09-21T10:59:14.778 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1970999999999999e-05 2024-09-21T10:59:14.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.778 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.778 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:59:14.778 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.659820+0000", 2024-09-21T10:59:14.778 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.259e-06 2024-09-21T10:59:14.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.778 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.778 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.779 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.660111+0000", 2024-09-21T10:59:14.779 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00029062400000000001 2024-09-21T10:59:14.779 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.779 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.779 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:59:14.779 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.660122+0000", 2024-09-21T10:59:14.779 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0842999999999999e-05 2024-09-21T10:59:14.779 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.779 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.779 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.779 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.638033+0000", 2024-09-21T10:59:14.779 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.97791153399999997 2024-09-21T10:59:14.779 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.780 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:59:14.780 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.638163+0000", 2024-09-21T10:59:14.780 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012933899999999999 2024-09-21T10:59:14.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.780 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:59:14.780 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.638264+0000", 2024-09-21T10:59:14.780 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000101463 2024-09-21T10:59:14.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.780 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.780 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.640323+0000", 2024-09-21T10:59:14.780 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0020588540000000002 2024-09-21T10:59:14.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.781 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.781 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.640491+0000", 2024-09-21T10:59:14.781 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016825600000000001 2024-09-21T10:59:14.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.781 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:59:14.781 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.641727+0000", 2024-09-21T10:59:14.781 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0012361900000000001 2024-09-21T10:59:14.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.781 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:59:14.781 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.641779+0000", 2024-09-21T10:59:14.782 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1511000000000002e-05 2024-09-21T10:59:14.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.782 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:59:14.782 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.641815+0000", 2024-09-21T10:59:14.782 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6361e-05 2024-09-21T10:59:14.782 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.782 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:59:14.782 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.782 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:29128 57.9 57:9035b4cc:::benchmark_data_smithi070_360420_object29127:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1537)", 2024-09-21T10:59:14.782 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:58:32.681201+0000", 2024-09-21T10:59:14.783 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 42.006230666, 2024-09-21T10:59:14.783 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96200238999999999, 2024-09-21T10:59:14.783 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:59:14.783 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:59:14.783 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:59:14.783 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T10:59:14.783 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T10:59:14.783 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 29128 2024-09-21T10:59:14.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.783 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:59:14.783 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.783 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:59:14.783 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681201+0000", 2024-09-21T10:59:14.783 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.784 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:59:14.784 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681201+0000", 2024-09-21T10:59:14.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.784 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:59:14.784 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681199+0000", 2024-09-21T10:59:14.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:59:14.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.784 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:59:14.784 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681241+0000", 2024-09-21T10:59:14.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1578000000000001e-05 2024-09-21T10:59:14.785 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.785 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.785 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:59:14.785 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681254+0000", 2024-09-21T10:59:14.785 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2886e-05 2024-09-21T10:59:14.785 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.785 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.785 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:59:14.785 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681261+0000", 2024-09-21T10:59:14.785 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1930000000000004e-06 2024-09-21T10:59:14.785 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.785 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.785 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.785 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681544+0000", 2024-09-21T10:59:14.786 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028305000000000002 2024-09-21T10:59:14.786 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.786 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.786 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:59:14.786 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681556+0000", 2024-09-21T10:59:14.786 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2459e-05 2024-09-21T10:59:14.786 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.786 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.786 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.786 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.639152+0000", 2024-09-21T10:59:14.786 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.95759554000000002 2024-09-21T10:59:14.786 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.786 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.786 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:59:14.786 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.639251+0000", 2024-09-21T10:59:14.787 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9067999999999994e-05 2024-09-21T10:59:14.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.787 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.787 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:59:14.787 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.639367+0000", 2024-09-21T10:59:14.787 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011600899999999999 2024-09-21T10:59:14.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.787 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.787 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:59:14.787 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.641824+0000", 2024-09-21T10:59:14.787 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0024572959999999999 2024-09-21T10:59:14.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.787 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.787 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.788 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.643074+0000", 2024-09-21T10:59:14.788 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0012495399999999999 2024-09-21T10:59:14.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.788 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.788 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.643130+0000", 2024-09-21T10:59:14.788 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6465999999999998e-05 2024-09-21T10:59:14.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.788 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:59:14.788 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.643173+0000", 2024-09-21T10:59:14.788 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2478000000000002e-05 2024-09-21T10:59:14.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.789 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:59:14.789 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.643203+0000", 2024-09-21T10:59:14.789 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0586999999999997e-05 2024-09-21T10:59:14.789 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.789 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:59:14.789 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.789 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.789 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.789 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:29129 57.9 57:9186962e:::benchmark_data_smithi070_360420_object29128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1537)", 2024-09-21T10:59:14.789 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:58:32.681277+0000", 2024-09-21T10:59:14.789 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 42.006154627999997, 2024-09-21T10:59:14.789 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96646752700000005, 2024-09-21T10:59:14.789 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:59:14.789 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:59:14.790 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:59:14.790 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T10:59:14.790 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T10:59:14.790 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 29129 2024-09-21T10:59:14.790 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.790 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:59:14.790 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.790 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:59:14.790 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681277+0000", 2024-09-21T10:59:14.790 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.790 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.790 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.790 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:59:14.791 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681277+0000", 2024-09-21T10:59:14.791 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.791 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.791 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.791 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:59:14.791 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681277+0000", 2024-09-21T10:59:14.791 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T10:59:14.791 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.791 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.791 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:59:14.791 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681304+0000", 2024-09-21T10:59:14.791 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7623999999999998e-05 2024-09-21T10:59:14.791 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.791 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.792 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:59:14.792 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681312+0000", 2024-09-21T10:59:14.792 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3769999999999998e-06 2024-09-21T10:59:14.792 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.792 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.792 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:59:14.792 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681317+0000", 2024-09-21T10:59:14.792 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3299999999999998e-06 2024-09-21T10:59:14.792 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.792 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.792 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.792 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681583+0000", 2024-09-21T10:59:14.792 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00026564200000000002 2024-09-21T10:59:14.792 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.793 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.793 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:59:14.793 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681589+0000", 2024-09-21T10:59:14.793 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7379999999999997e-06 2024-09-21T10:59:14.793 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.793 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.793 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.793 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.640530+0000", 2024-09-21T10:59:14.793 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.95894080400000004 2024-09-21T10:59:14.793 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.793 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.793 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:59:14.793 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.640729+0000", 2024-09-21T10:59:14.793 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019901000000000001 2024-09-21T10:59:14.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.794 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.794 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:59:14.794 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.640844+0000", 2024-09-21T10:59:14.794 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011447600000000001 2024-09-21T10:59:14.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.794 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.794 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:59:14.794 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.643080+0000", 2024-09-21T10:59:14.794 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022364339999999998 2024-09-21T10:59:14.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.794 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.794 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.794 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.645208+0000", 2024-09-21T10:59:14.795 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021281690000000001 2024-09-21T10:59:14.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.795 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.795 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.647662+0000", 2024-09-21T10:59:14.795 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0024534370000000002 2024-09-21T10:59:14.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.795 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:59:14.795 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.647719+0000", 2024-09-21T10:59:14.795 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7111000000000003e-05 2024-09-21T10:59:14.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.795 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:59:14.796 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.647745+0000", 2024-09-21T10:59:14.796 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5914e-05 2024-09-21T10:59:14.796 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.796 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:59:14.796 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.796 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.796 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.796 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:29144 57.9 57:919fd5ad:::benchmark_data_smithi070_360420_object29143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1537)", 2024-09-21T10:59:14.796 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:58:32.709679+0000", 2024-09-21T10:59:14.796 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 41.977752314, 2024-09-21T10:59:14.796 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.94539665399999995, 2024-09-21T10:59:14.796 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:59:14.796 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:59:14.796 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:59:14.797 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T10:59:14.797 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T10:59:14.797 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 29144 2024-09-21T10:59:14.797 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.797 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:59:14.797 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.797 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:59:14.797 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.709679+0000", 2024-09-21T10:59:14.797 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.797 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.797 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.797 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:59:14.797 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.709679+0000", 2024-09-21T10:59:14.797 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.798 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.798 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:59:14.798 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.709677+0000", 2024-09-21T10:59:14.798 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:59:14.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.798 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.798 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:59:14.798 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.709721+0000", 2024-09-21T10:59:14.798 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3291000000000003e-05 2024-09-21T10:59:14.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.798 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.798 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:59:14.799 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.709738+0000", 2024-09-21T10:59:14.799 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6813999999999999e-05 2024-09-21T10:59:14.799 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.799 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.799 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:59:14.799 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.709745+0000", 2024-09-21T10:59:14.799 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9289999999999998e-06 2024-09-21T10:59:14.799 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.799 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.799 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.799 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.709797+0000", 2024-09-21T10:59:14.799 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1066999999999997e-05 2024-09-21T10:59:14.799 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.799 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.800 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:59:14.800 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.709806+0000", 2024-09-21T10:59:14.800 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5969999999999998e-06 2024-09-21T10:59:14.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.800 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.800 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.800 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.641853+0000", 2024-09-21T10:59:14.800 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.93204697000000003 2024-09-21T10:59:14.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.800 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.800 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:59:14.800 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.641978+0000", 2024-09-21T10:59:14.800 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012522400000000001 2024-09-21T10:59:14.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.801 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.801 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:59:14.801 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.642117+0000", 2024-09-21T10:59:14.801 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013835400000000001 2024-09-21T10:59:14.801 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.801 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.801 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:59:14.801 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.645179+0000", 2024-09-21T10:59:14.801 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0030618450000000001 2024-09-21T10:59:14.801 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.801 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.801 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.801 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.645234+0000", 2024-09-21T10:59:14.801 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4979999999999999e-05 2024-09-21T10:59:14.802 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.802 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.802 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.802 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.654997+0000", 2024-09-21T10:59:14.802 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.009763384 2024-09-21T10:59:14.802 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.802 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.802 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:59:14.802 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.655038+0000", 2024-09-21T10:59:14.802 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1307999999999998e-05 2024-09-21T10:59:14.802 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.802 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.802 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:59:14.802 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.655076+0000", 2024-09-21T10:59:14.803 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7851000000000001e-05 2024-09-21T10:59:14.803 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.803 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:59:14.803 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.803 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.803 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.803 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:29201 57.9 57:93a056f6:::benchmark_data_smithi070_360420_object29200:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1537)", 2024-09-21T10:59:14.803 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:58:32.747625+0000", 2024-09-21T10:59:14.803 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 41.939806943000001, 2024-09-21T10:59:14.803 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.90758979299999998, 2024-09-21T10:59:14.803 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:59:14.803 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:59:14.803 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:59:14.803 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T10:59:14.804 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T10:59:14.804 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 29201 2024-09-21T10:59:14.804 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.804 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:59:14.804 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.804 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:59:14.804 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.747625+0000", 2024-09-21T10:59:14.804 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.804 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.804 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.804 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:59:14.804 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.747625+0000", 2024-09-21T10:59:14.804 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.804 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.805 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.805 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:59:14.805 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.747622+0000", 2024-09-21T10:59:14.805 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 2024-09-21T10:59:14.805 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.805 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.805 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:59:14.805 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.747673+0000", 2024-09-21T10:59:14.805 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0284000000000002e-05 2024-09-21T10:59:14.805 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.805 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.805 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:59:14.805 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.747690+0000", 2024-09-21T10:59:14.805 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7677999999999999e-05 2024-09-21T10:59:14.806 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.806 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.806 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:59:14.806 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.747704+0000", 2024-09-21T10:59:14.806 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.364e-05 2024-09-21T10:59:14.806 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.806 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.806 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.806 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.747763+0000", 2024-09-21T10:59:14.806 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8634999999999997e-05 2024-09-21T10:59:14.806 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.806 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.806 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:59:14.806 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.747772+0000", 2024-09-21T10:59:14.807 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1619999999999997e-06 2024-09-21T10:59:14.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.807 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.807 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.807 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.643228+0000", 2024-09-21T10:59:14.807 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.89545633199999997 2024-09-21T10:59:14.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.807 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.807 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:59:14.807 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.643345+0000", 2024-09-21T10:59:14.807 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000116978 2024-09-21T10:59:14.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.807 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.807 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:59:14.807 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.643469+0000", 2024-09-21T10:59:14.808 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012339600000000001 2024-09-21T10:59:14.808 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.808 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.808 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.808 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.645259+0000", 2024-09-21T10:59:14.808 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0017906899999999999 2024-09-21T10:59:14.808 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.808 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.808 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:59:14.808 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.652531+0000", 2024-09-21T10:59:14.808 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0072719259999999997 2024-09-21T10:59:14.808 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.808 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.809 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.809 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.655148+0000", 2024-09-21T10:59:14.809 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0026166129999999998 2024-09-21T10:59:14.809 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.809 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.809 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:59:14.809 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.655186+0000", 2024-09-21T10:59:14.809 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7758e-05 2024-09-21T10:59:14.809 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.809 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.809 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:59:14.809 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.655215+0000", 2024-09-21T10:59:14.809 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9013000000000002e-05 2024-09-21T10:59:14.809 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.810 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:59:14.810 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.810 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.810 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.810 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:29220 57.9 57:9be6f912:::benchmark_data_smithi070_360420_object29219:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1537)", 2024-09-21T10:59:14.810 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:58:32.754855+0000", 2024-09-21T10:59:14.810 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 41.932576445999999, 2024-09-21T10:59:14.810 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.90090863700000001, 2024-09-21T10:59:14.810 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:59:14.810 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:59:14.810 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:59:14.810 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T10:59:14.810 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T10:59:14.810 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 29220 2024-09-21T10:59:14.811 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.811 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:59:14.811 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.811 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:59:14.811 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.754855+0000", 2024-09-21T10:59:14.811 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.811 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.811 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.811 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:59:14.811 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.754855+0000", 2024-09-21T10:59:14.811 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.811 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.811 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.811 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:59:14.812 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.754853+0000", 2024-09-21T10:59:14.812 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:59:14.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.812 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.812 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:59:14.812 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.754908+0000", 2024-09-21T10:59:14.812 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4389999999999999e-05 2024-09-21T10:59:14.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.812 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.812 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:59:14.812 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.754926+0000", 2024-09-21T10:59:14.812 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7802999999999999e-05 2024-09-21T10:59:14.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.812 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.813 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:59:14.813 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.754934+0000", 2024-09-21T10:59:14.813 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2199999999999992e-06 2024-09-21T10:59:14.813 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.813 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.813 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.813 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.755006+0000", 2024-09-21T10:59:14.813 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2083999999999996e-05 2024-09-21T10:59:14.813 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.813 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.813 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:59:14.813 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.755018+0000", 2024-09-21T10:59:14.813 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1916000000000001e-05 2024-09-21T10:59:14.813 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.814 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.814 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.814 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.644250+0000", 2024-09-21T10:59:14.814 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88923207500000001 2024-09-21T10:59:14.814 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.814 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.814 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:59:14.814 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.644366+0000", 2024-09-21T10:59:14.814 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000115673 2024-09-21T10:59:14.814 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.814 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.814 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:59:14.814 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.644497+0000", 2024-09-21T10:59:14.814 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000131593 2024-09-21T10:59:14.815 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.815 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.815 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.815 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.647786+0000", 2024-09-21T10:59:14.815 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0032884500000000001 2024-09-21T10:59:14.815 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.815 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.815 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:59:14.815 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.652536+0000", 2024-09-21T10:59:14.815 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0047503959999999996 2024-09-21T10:59:14.815 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.815 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.815 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.816 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.655277+0000", 2024-09-21T10:59:14.816 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0027413709999999998 2024-09-21T10:59:14.816 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.816 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.816 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:59:14.816 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.655315+0000", 2024-09-21T10:59:14.816 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8065999999999998e-05 2024-09-21T10:59:14.816 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.816 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.816 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:59:14.816 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.655764+0000", 2024-09-21T10:59:14.816 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00044852899999999997 2024-09-21T10:59:14.816 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.816 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:59:14.816 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.817 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.817 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.817 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:29229 57.9 57:9730de02:::benchmark_data_smithi070_360420_object29228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1537)", 2024-09-21T10:59:14.817 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:58:32.763033+0000", 2024-09-21T10:59:14.817 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 41.924398453000002, 2024-09-21T10:59:14.817 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.89779595999999995, 2024-09-21T10:59:14.817 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:59:14.817 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:59:14.817 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:59:14.817 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T10:59:14.817 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T10:59:14.817 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 29229 2024-09-21T10:59:14.817 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.817 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:59:14.817 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.818 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:59:14.818 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.763033+0000", 2024-09-21T10:59:14.818 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.818 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.818 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.818 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:59:14.818 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.763033+0000", 2024-09-21T10:59:14.818 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.818 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.818 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.818 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:59:14.818 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.763032+0000", 2024-09-21T10:59:14.819 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:59:14.819 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.819 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.819 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:59:14.819 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.763063+0000", 2024-09-21T10:59:14.819 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1748000000000002e-05 2024-09-21T10:59:14.819 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.819 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.819 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:59:14.819 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.763077+0000", 2024-09-21T10:59:14.819 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3842999999999999e-05 2024-09-21T10:59:14.819 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.819 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.819 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:59:14.819 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.763085+0000", 2024-09-21T10:59:14.820 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2230000000000007e-06 2024-09-21T10:59:14.820 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.820 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.820 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.820 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.763144+0000", 2024-09-21T10:59:14.820 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9030000000000002e-05 2024-09-21T10:59:14.820 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.820 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.820 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:59:14.820 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.763155+0000", 2024-09-21T10:59:14.820 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0396e-05 2024-09-21T10:59:14.820 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.820 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.820 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.820 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.645278+0000", 2024-09-21T10:59:14.821 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88212362799999999 2024-09-21T10:59:14.821 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.821 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.821 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:59:14.821 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.645381+0000", 2024-09-21T10:59:14.821 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000102489 2024-09-21T10:59:14.821 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.821 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.821 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:59:14.821 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.645492+0000", 2024-09-21T10:59:14.821 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011087200000000001 2024-09-21T10:59:14.821 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.821 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.822 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.822 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.649988+0000", 2024-09-21T10:59:14.822 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0044962589999999998 2024-09-21T10:59:14.822 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.822 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.822 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:59:14.822 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.652541+0000", 2024-09-21T10:59:14.822 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002552478 2024-09-21T10:59:14.822 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.822 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.822 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.822 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.660744+0000", 2024-09-21T10:59:14.822 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0082031010000000008 2024-09-21T10:59:14.822 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.823 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.823 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:59:14.823 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.660792+0000", 2024-09-21T10:59:14.823 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8559e-05 2024-09-21T10:59:14.823 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.823 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.823 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:59:14.823 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.660829+0000", 2024-09-21T10:59:14.823 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7011999999999997e-05 2024-09-21T10:59:14.823 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.823 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:59:14.823 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.823 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.823 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.824 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:29303 57.9 57:928d9721:::benchmark_data_smithi070_360420_object29302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1537)", 2024-09-21T10:59:14.824 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:58:32.817075+0000", 2024-09-21T10:59:14.824 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 41.870356778000001, 2024-09-21T10:59:14.824 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.84419996200000003, 2024-09-21T10:59:14.824 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:59:14.824 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:59:14.824 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:59:14.824 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T10:59:14.824 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T10:59:14.824 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 29303 2024-09-21T10:59:14.824 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.824 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:59:14.824 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.824 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:59:14.825 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.817075+0000", 2024-09-21T10:59:14.825 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.825 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.825 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.825 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:59:14.825 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.817075+0000", 2024-09-21T10:59:14.825 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.825 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.825 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.825 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:59:14.825 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.817073+0000", 2024-09-21T10:59:14.825 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:59:14.825 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.825 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.825 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:59:14.825 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.817120+0000", 2024-09-21T10:59:14.826 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6910000000000003e-05 2024-09-21T10:59:14.826 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.826 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.826 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:59:14.826 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.817132+0000", 2024-09-21T10:59:14.826 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2159e-05 2024-09-21T10:59:14.826 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.826 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.826 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:59:14.826 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.817139+0000", 2024-09-21T10:59:14.826 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7769999999999997e-06 2024-09-21T10:59:14.826 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.826 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.827 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.827 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.817193+0000", 2024-09-21T10:59:14.827 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4282e-05 2024-09-21T10:59:14.827 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.827 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.827 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:59:14.827 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.817202+0000", 2024-09-21T10:59:14.827 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3810000000000005e-06 2024-09-21T10:59:14.827 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.827 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.827 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.827 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.646401+0000", 2024-09-21T10:59:14.827 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.82919829199999995 2024-09-21T10:59:14.827 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.827 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.828 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:59:14.828 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.646524+0000", 2024-09-21T10:59:14.828 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012292100000000001 2024-09-21T10:59:14.828 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.828 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.828 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:59:14.828 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.646656+0000", 2024-09-21T10:59:14.828 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000132807 2024-09-21T10:59:14.828 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.828 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.828 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.828 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.650022+0000", 2024-09-21T10:59:14.828 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003365982 2024-09-21T10:59:14.828 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.828 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.829 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:59:14.829 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.652583+0000", 2024-09-21T10:59:14.829 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0025600979999999998 2024-09-21T10:59:14.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.829 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.829 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.829 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.661198+0000", 2024-09-21T10:59:14.829 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0086153830000000008 2024-09-21T10:59:14.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.829 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.829 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:59:14.829 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.661241+0000", 2024-09-21T10:59:14.829 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2667999999999999e-05 2024-09-21T10:59:14.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.830 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.830 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:59:14.830 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.661275+0000", 2024-09-21T10:59:14.830 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4301999999999998e-05 2024-09-21T10:59:14.830 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.830 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:59:14.830 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.830 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.830 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.830 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:29311 57.9 57:9d968644:::benchmark_data_smithi070_360420_object29310:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1537)", 2024-09-21T10:59:14.830 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:58:32.822630+0000", 2024-09-21T10:59:14.830 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 41.864801913999997, 2024-09-21T10:59:14.830 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.84199829000000004, 2024-09-21T10:59:14.830 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:59:14.831 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:59:14.831 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:59:14.831 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T10:59:14.831 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T10:59:14.831 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 29311 2024-09-21T10:59:14.831 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.831 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:59:14.831 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.831 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:59:14.831 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.822630+0000", 2024-09-21T10:59:14.831 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.831 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.831 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.831 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:59:14.832 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:59:14.832 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.822630+0000", 2024-09-21T10:59:14.832 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.833 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.833 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.833 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:59:14.833 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.822627+0000", 2024-09-21T10:59:14.833 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 2024-09-21T10:59:14.833 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.833 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.833 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:59:14.833 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.822815+0000", 2024-09-21T10:59:14.833 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000187185 2024-09-21T10:59:14.833 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.833 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.833 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:59:14.833 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.822833+0000", 2024-09-21T10:59:14.834 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8711000000000001e-05 2024-09-21T10:59:14.834 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.834 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.834 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:59:14.834 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.822843+0000", 2024-09-21T10:59:14.834 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2839999999999995e-06 2024-09-21T10:59:14.834 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.834 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.834 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.834 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.822907+0000", 2024-09-21T10:59:14.834 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4676000000000002e-05 2024-09-21T10:59:14.834 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.834 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.835 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:59:14.835 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.822917+0000", 2024-09-21T10:59:14.835 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.73e-06 2024-09-21T10:59:14.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.835 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.835 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.835 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.647799+0000", 2024-09-21T10:59:14.835 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.82488186299999999 2024-09-21T10:59:14.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.835 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.835 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:59:14.835 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.647885+0000", 2024-09-21T10:59:14.835 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6119000000000007e-05 2024-09-21T10:59:14.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.836 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.836 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:59:14.836 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.648025+0000", 2024-09-21T10:59:14.836 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000140061 2024-09-21T10:59:14.836 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.836 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.836 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.836 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.652576+0000", 2024-09-21T10:59:14.836 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0045508110000000001 2024-09-21T10:59:14.836 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.836 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.836 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:59:14.836 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.652587+0000", 2024-09-21T10:59:14.836 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1263999999999999e-05 2024-09-21T10:59:14.837 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.837 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.837 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.837 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.664544+0000", 2024-09-21T10:59:14.837 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011956615 2024-09-21T10:59:14.837 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.837 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.837 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:59:14.837 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.664592+0000", 2024-09-21T10:59:14.837 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8281000000000001e-05 2024-09-21T10:59:14.837 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.837 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.837 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:59:14.837 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.664628+0000", 2024-09-21T10:59:14.838 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6096000000000001e-05 2024-09-21T10:59:14.838 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.838 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:59:14.838 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.838 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.838 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.838 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:29322 57.9 57:9edec8cc:::benchmark_data_smithi070_360420_object29321:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1537)", 2024-09-21T10:59:14.838 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:58:32.830884+0000", 2024-09-21T10:59:14.838 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 41.856547526, 2024-09-21T10:59:14.838 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.83385688599999996, 2024-09-21T10:59:14.838 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T10:59:14.838 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:59:14.838 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T10:59:14.838 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T10:59:14.839 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T10:59:14.839 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 29322 2024-09-21T10:59:14.839 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.839 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T10:59:14.839 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.839 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T10:59:14.839 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.830884+0000", 2024-09-21T10:59:14.839 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.839 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.839 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.839 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T10:59:14.839 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.830884+0000", 2024-09-21T10:59:14.839 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T10:59:14.839 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.840 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.840 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T10:59:14.840 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.830882+0000", 2024-09-21T10:59:14.840 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T10:59:14.840 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.840 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.840 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T10:59:14.840 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.831533+0000", 2024-09-21T10:59:14.840 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00065064699999999997 2024-09-21T10:59:14.840 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.840 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.840 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T10:59:14.840 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.831556+0000", 2024-09-21T10:59:14.840 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2490999999999999e-05 2024-09-21T10:59:14.841 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.841 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.841 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T10:59:14.841 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.831566+0000", 2024-09-21T10:59:14.841 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0284e-05 2024-09-21T10:59:14.841 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.841 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.841 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.841 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.831628+0000", 2024-09-21T10:59:14.841 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1719999999999999e-05 2024-09-21T10:59:14.841 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.841 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.841 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T10:59:14.841 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.831638+0000", 2024-09-21T10:59:14.842 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0226e-05 2024-09-21T10:59:14.842 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.842 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.842 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T10:59:14.842 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.648892+0000", 2024-09-21T10:59:14.842 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.81725446999999996 2024-09-21T10:59:14.842 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.842 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.842 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T10:59:14.842 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.648993+0000", 2024-09-21T10:59:14.842 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000100721 2024-09-21T10:59:14.842 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.842 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.842 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:59:14.843 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.649134+0000", 2024-09-21T10:59:14.843 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000140984 2024-09-21T10:59:14.843 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.843 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.843 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T10:59:14.843 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.652591+0000", 2024-09-21T10:59:14.843 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0034572639999999998 2024-09-21T10:59:14.843 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.843 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.843 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.843 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.652644+0000", 2024-09-21T10:59:14.843 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3244e-05 2024-09-21T10:59:14.843 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.843 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.844 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:14.844 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.664666+0000", 2024-09-21T10:59:14.844 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012021685000000001 2024-09-21T10:59:14.844 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.844 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.844 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T10:59:14.844 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.664710+0000", 2024-09-21T10:59:14.844 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3396000000000003e-05 2024-09-21T10:59:14.844 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T10:59:14.844 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T10:59:14.844 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T10:59:14.844 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.664741+0000", 2024-09-21T10:59:14.844 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1534e-05 2024-09-21T10:59:14.844 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.845 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:59:14.845 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.845 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T10:59:14.845 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T10:59:14.845 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T10:59:14.863 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:14.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:14.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:14.964 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:14.991 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:14.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:14.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:15.092 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:15.113 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:15.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:15.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:15.215 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:15.236 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:15.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:15.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:15.338 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:15.360 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:15.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:15.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:15.462 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:15.484 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:15.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:15.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:15.585 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:15.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:15 smithi160 ceph-mon[101879]: pgmap v2961: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.2 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 17 MiB/s wr, 279 op/s; 33981/157062 objects degraded (21.635%); 8276/157062 objects misplaced (5.269%); 5.5 MiB/s, 88 objects/s recovering 2024-09-21T10:59:15.608 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:15.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:15.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:15.624 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 59 16 53616 53600 56.7749 23.3125 0.00327437 0.0175784 2024-09-21T10:59:15.711 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:15.733 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:15.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:15.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:15.757 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:59:15.766 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:15 smithi070 ceph-mon[107385]: pgmap v2961: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.2 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 17 MiB/s wr, 279 op/s; 33981/157062 objects degraded (21.635%); 8276/157062 objects misplaced (5.269%); 5.5 MiB/s, 88 objects/s recovering 2024-09-21T10:59:15.767 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:15 smithi070 ceph-mon[112139]: pgmap v2961: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.2 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 17 MiB/s wr, 279 op/s; 33981/157062 objects degraded (21.635%); 8276/157062 objects misplaced (5.269%); 5.5 MiB/s, 88 objects/s recovering 2024-09-21T10:59:15.835 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:15.856 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:15.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:15.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:15.958 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:15.979 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:15.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:15.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:16.044 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:59:16.080 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:16.101 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:16.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:16.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:16.203 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:16.226 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:16.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:16.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:16.328 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:16.350 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:16.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:16.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:16.452 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:16.476 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:16.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:16.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:16.578 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:16.603 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:16.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:16.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:16.624 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:2024-09-21T10:59:16.623446+0000 min lat: 0.00141297 max lat: 5.2078 avg lat: 0.0177195 2024-09-21T10:59:16.624 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:59:16.624 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 60 16 54071 54055 56.3026 28.4375 0.096753 0.0177195 2024-09-21T10:59:16.660 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:59:16.660 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:59:16.677 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:59:16.678 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:59:16.694 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:59:16.694 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:59:16.705 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:16.712 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:59:16.712 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:59:16.730 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:16.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:16.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:16.738 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:59:16.738 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:59:16.738 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T10:59:16.738 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T10:59:16.757 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:59:16.757 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:59:16.832 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:16.855 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:16.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:16.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:16.957 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:16.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:16.987 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:16.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:17.088 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:17.110 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:17.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:17.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:17.212 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:17.232 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:17.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:17.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:17.334 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:17.358 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:17.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:17.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:17.460 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:17.481 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:17.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:17.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:17.583 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:17.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:17 smithi160 ceph-mon[101879]: pgmap v2962: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.2 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 17 MiB/s wr, 265 op/s; 34129/158262 objects degraded (21.565%); 8276/158262 objects misplaced (5.229%); 4.5 MiB/s, 71 objects/s recovering 2024-09-21T10:59:17.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:17 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2558096182' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:59:17.606 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:17.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:17.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:17.624 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 61 16 54517 54501 55.8365 27.875 0.00644677 0.0178748 2024-09-21T10:59:17.707 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:17.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:17.734 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:17.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:17.835 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:17.858 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:17.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:17.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:17.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:17 smithi070 ceph-mon[107385]: pgmap v2962: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.2 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 17 MiB/s wr, 265 op/s; 34129/158262 objects degraded (21.565%); 8276/158262 objects misplaced (5.229%); 4.5 MiB/s, 71 objects/s recovering 2024-09-21T10:59:17.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:17 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2558096182' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:59:17.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:17 smithi070 ceph-mon[112139]: pgmap v2962: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.2 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 17 MiB/s wr, 265 op/s; 34129/158262 objects degraded (21.565%); 8276/158262 objects misplaced (5.229%); 4.5 MiB/s, 71 objects/s recovering 2024-09-21T10:59:17.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:17 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2558096182' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:59:17.949 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:59:17.949 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:59:17.960 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:17.982 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:17.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:17.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:18.084 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:18.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:18.106 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:18.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:18.207 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:18.231 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:18.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:18.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:18.333 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:18.355 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:18.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:18.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:18.456 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:18.478 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:18.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:18.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:18.580 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:18 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 34129/158262 objects degraded (21.565%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T10:59:18.601 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:18.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:18.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:18.624 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 62 16 54982 54966 55.4046 29.0625 0.00253906 0.0180217 2024-09-21T10:59:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:18 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 34129/158262 objects degraded (21.565%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T10:59:18.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:18 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 34129/158262 objects degraded (21.565%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T10:59:18.702 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:18.725 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:18.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:18.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:18.742 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:59:18.743 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:59:18.787 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:59:18.827 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:18.849 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:18.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:18.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:18.950 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:18.983 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:18.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:18.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:19.017 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7, 4] out_osds: [2, 5, 0] dead_osds: [5, 6] live_osds: [0, 1, 2, 3, 4, 7] 2024-09-21T10:59:19.017 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:59:19.017 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-21T10:59:19.017 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:59:19.017 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:59:19.018 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-21T10:59:19.082 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:59:19.084 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:19.108 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:19.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:19.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:19.210 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:19.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:19.235 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:19.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:19.336 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:19.359 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:19.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:19.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:19.461 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:19.489 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:19.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:19.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:19.594 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:19.623 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:19.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:19.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:19.624 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 63 16 55396 55380 54.9358 25.875 0.00368779 0.018149 2024-09-21T10:59:19.725 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:19.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:19.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:19.773 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:59:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:19 smithi160 ceph-mon[101879]: pgmap v2963: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.3 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 19 MiB/s wr, 300 op/s; 34284/160803 objects degraded (21.320%); 8574/160803 objects misplaced (5.332%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:59:19.859 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:19.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:19 smithi070 ceph-mon[107385]: pgmap v2963: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.3 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 19 MiB/s wr, 300 op/s; 34284/160803 objects degraded (21.320%); 8574/160803 objects misplaced (5.332%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:59:19.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:19 smithi070 ceph-mon[112139]: pgmap v2963: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.3 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 19 MiB/s wr, 300 op/s; 34284/160803 objects degraded (21.320%); 8574/160803 objects misplaced (5.332%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:59:19.881 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:19.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:19.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:19.983 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:20.012 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:20.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:20.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:20.114 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:20.137 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:20.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:20.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:20.238 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:20.261 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:20.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:20.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:20.363 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:20.385 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:20.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:20.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:20.479 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T10:59:20.488 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:20.493 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:20 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1057485395' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:59:20.493 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:20 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:59:20.493 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:59:20.493 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:20 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1057485395' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:59:20.493 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:20 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:59:20.494 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:59:20.511 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:20.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:20.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:20.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:20 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1057485395' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:59:20.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:20 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:59:20.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:59:20.612 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:20.624 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 64 16 55818 55802 54.4895 26.375 0.00286237 0.0183162 2024-09-21T10:59:20.635 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:20.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:20.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:20.736 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:20.743 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:59:20.757 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:20.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:20.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:20.860 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:20.886 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:20.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:20.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:20.988 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:21.010 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:21.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:21.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:21.112 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:21.136 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:21.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:21.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:21.238 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:21.262 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:21.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:21.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:21.264 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:59:21.265 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:59:21.304 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:59:21.364 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:21.388 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:21.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:21.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:21.490 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:21.513 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:21.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:21.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:21 smithi160 ceph-mon[101879]: pgmap v2964: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.4 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 26 MiB/s wr, 419 op/s; 34675/164973 objects degraded (21.019%); 8876/164973 objects misplaced (5.380%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:59:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:21 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:59:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:21 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:59:21.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:21 smithi160 ceph-mon[101879]: osdmap e1551: 8 total, 6 up, 5 in 2024-09-21T10:59:21.614 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:21.625 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 65 16 56316 56300 54.13 31.125 0.00300307 0.0184401 2024-09-21T10:59:21.636 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:21.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:21.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:21.737 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:21.750 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:21 smithi070 ceph-mon[112139]: pgmap v2964: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.4 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 26 MiB/s wr, 419 op/s; 34675/164973 objects degraded (21.019%); 8876/164973 objects misplaced (5.380%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:59:21.751 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:21 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:59:21.751 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:21 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:59:21.751 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:21 smithi070 ceph-mon[112139]: osdmap e1551: 8 total, 6 up, 5 in 2024-09-21T10:59:21.751 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:21 smithi070 ceph-mon[107385]: pgmap v2964: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.4 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 26 MiB/s wr, 419 op/s; 34675/164973 objects degraded (21.019%); 8876/164973 objects misplaced (5.380%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:59:21.751 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:21 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:59:21.751 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:21 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:59:21.751 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:21 smithi070 ceph-mon[107385]: osdmap e1551: 8 total, 6 up, 5 in 2024-09-21T10:59:21.768 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:21.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:21.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:21.870 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:21.895 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:21.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:21.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:21.963 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:59:21.996 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:22.024 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:22.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:22.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:22.126 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:22.147 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:22.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:22.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:22.232 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:59:22.249 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:22.277 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:22.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:22.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:22.379 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:22.408 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:22.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:22.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:22.510 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:22.533 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:22.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:22.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:22.625 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 66 16 56735 56719 53.7066 26.1875 0.00432881 0.0185839 2024-09-21T10:59:22.628 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:59:22.628 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T10:59:22.628 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:59:22.635 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:22.658 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:22.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:22.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:22.760 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:22.784 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:22.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:22.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:22.886 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:22.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:22.918 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:22.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:23.019 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:23.045 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:23.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:23.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:23.149 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:23.173 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:23.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:23.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:23.231 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:59:23.255 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:59:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:59:23] "GET /metrics HTTP/1.1" 200 36580 "" "Prometheus/2.51.0" 2024-09-21T10:59:23.276 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:23.306 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:23.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:23.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:23.407 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:23.430 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:23.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:23.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:23.531 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:23.553 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:23.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:23.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:23.625 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 67 16 57210 57194 53.3481 29.6875 0.002692 0.0187117 2024-09-21T10:59:23.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:23 smithi070 ceph-mon[107385]: pgmap v2966: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.4 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 26 MiB/s wr, 419 op/s; 34675/164973 objects degraded (21.019%); 8876/164973 objects misplaced (5.380%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:59:23.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:23 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 34675/164973 objects degraded (21.019%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T10:59:23.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:23 smithi070 ceph-mon[112139]: pgmap v2966: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.4 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 26 MiB/s wr, 419 op/s; 34675/164973 objects degraded (21.019%); 8876/164973 objects misplaced (5.380%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:59:23.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:23 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 34675/164973 objects degraded (21.019%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T10:59:23.656 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:23.686 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:23.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:23.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:23.739 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:23 smithi160 ceph-mon[101879]: pgmap v2966: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.4 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 26 MiB/s wr, 419 op/s; 34675/164973 objects degraded (21.019%); 8876/164973 objects misplaced (5.380%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:59:23.740 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:23 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 34675/164973 objects degraded (21.019%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T10:59:23.788 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:23.820 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:23.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:23.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:23.922 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:23.947 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:23.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:23.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:24.048 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:24.072 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:24.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:24.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:24.171 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T10:59:24.171 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:59:24.173 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:24.199 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:24.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:24.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:24.301 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:24.328 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:24.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:24.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:24.430 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:24.444 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:59:24.445 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:59:24.462 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:24.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:24.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:24.463 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:59:24.463 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:59:24.486 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:59:24.486 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:59:24.504 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:59:24.505 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:59:24.527 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:59:24.528 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:59:24.528 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T10:59:24.528 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T10:59:24.550 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:59:24.550 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:59:24.563 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:24.586 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:24.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:24.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:24.625 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 68 16 57662 57646 52.979 28.25 0.356503 0.0188438 2024-09-21T10:59:24.688 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:24.711 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:24.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:24.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:24.814 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:24.835 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:24.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:24.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:24 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2006219377' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:59:24.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:24 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2006219377' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:59:24.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:24 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2006219377' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:59:24.937 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:24.960 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:24.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:24.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:25.061 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:25.083 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:25.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:25.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:25.185 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:25.207 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:25.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:25.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:25.309 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:25.332 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:25.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:25.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:25.434 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:25.458 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:25.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:25.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:25.559 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:25.580 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:25.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:25.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:25.625 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 69 16 58027 58011 52.5418 22.8125 0.00292012 0.018979 2024-09-21T10:59:25.682 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:25.707 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:25.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:25.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:25.809 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:25.831 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:25.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:25.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:25 smithi160 ceph-mon[101879]: pgmap v2967: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.4 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 24 MiB/s wr, 387 op/s; 34965/168702 objects degraded (20.726%); 9151/168702 objects misplaced (5.424%); 6.4 KiB/s, 0 objects/s recovering 2024-09-21T10:59:25.872 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:25 smithi070 ceph-mon[107385]: pgmap v2967: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.4 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 24 MiB/s wr, 387 op/s; 34965/168702 objects degraded (20.726%); 9151/168702 objects misplaced (5.424%); 6.4 KiB/s, 0 objects/s recovering 2024-09-21T10:59:25.872 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:25 smithi070 ceph-mon[112139]: pgmap v2967: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.4 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 24 MiB/s wr, 387 op/s; 34965/168702 objects degraded (20.726%); 9151/168702 objects misplaced (5.424%); 6.4 KiB/s, 0 objects/s recovering 2024-09-21T10:59:25.933 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:25.960 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:25.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:25.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:26.061 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:26.092 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:26.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:26.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:26.193 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:26.216 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:26.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:26.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:26.318 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:26.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:26.341 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:26.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:26.442 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:26.463 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:26.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:26.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:26.565 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:26.586 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:26.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:26.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:26.625 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 70 16 58418 58402 52.1402 24.4375 0.00259556 0.0191313 2024-09-21T10:59:26.666 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:59:26.667 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:59:26.688 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:26.713 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:59:26.731 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:26.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:26.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:26.832 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:26.862 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:26.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:26.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:26.964 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:26.993 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:26.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:26.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:27.094 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:27.125 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:27.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:27.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:27.226 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:27.252 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:27.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:27.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:27.354 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:27.377 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:27.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:27.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:27.479 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:27.504 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:27.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:27.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:27.606 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:27.625 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 71 16 58916 58900 51.8442 31.125 0.00309117 0.0192576 2024-09-21T10:59:27.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:27 smithi070 ceph-mon[107385]: pgmap v2968: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.5 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 28 MiB/s wr, 448 op/s; 35241/171720 objects degraded (20.522%); 9451/171720 objects misplaced (5.504%); 19 KiB/s, 0 objects/s recovering 2024-09-21T10:59:27.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:27 smithi070 ceph-mon[112139]: pgmap v2968: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.5 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 28 MiB/s wr, 448 op/s; 35241/171720 objects degraded (20.522%); 9451/171720 objects misplaced (5.504%); 19 KiB/s, 0 objects/s recovering 2024-09-21T10:59:27.628 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:27.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:27.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:27.729 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:27.751 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:27.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:27.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:27 smithi160 ceph-mon[101879]: pgmap v2968: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.5 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 28 MiB/s wr, 448 op/s; 35241/171720 objects degraded (20.522%); 9451/171720 objects misplaced (5.504%); 19 KiB/s, 0 objects/s recovering 2024-09-21T10:59:27.853 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:27.875 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:27.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:27.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:27.976 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:27.998 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:27.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:27.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:28.100 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:28.122 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:28.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:28.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:28.224 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:28.247 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:28.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:28.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:28.349 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:28.369 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:28.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:28.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:28.471 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:28.495 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:28.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:28.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:28.597 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:28.621 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:28.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:28.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:28.625 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 72 16 59335 59319 51.4878 26.1875 0.00362212 0.019383 2024-09-21T10:59:28.668 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:59:28.724 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:28.730 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:28 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 35241/171720 objects degraded (20.522%), 22 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:59:28.730 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:28 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 35241/171720 objects degraded (20.522%), 22 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:59:28.752 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:28.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:28.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:28 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 35241/171720 objects degraded (20.522%), 22 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:59:28.854 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:28.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:28.877 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:28.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:28.918 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:59:28.978 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:29.001 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:29.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:29.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:29.029 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:59:29.103 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:29.128 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:29.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:29.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:29.230 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:29.256 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:29.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:29.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:29.358 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:29.385 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:29.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:29.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:29.487 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:29.488 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T10:59:29.517 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:29.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:29.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:29.619 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:29.625 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 73 16 59736 59720 51.1258 25.0625 0.00301539 0.0195212 2024-09-21T10:59:29.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:29 smithi070 ceph-mon[107385]: pgmap v2969: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.5 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 28 MiB/s wr, 449 op/s; 35379/174291 objects degraded (20.299%); 9743/174291 objects misplaced (5.590%); 19 KiB/s, 0 objects/s recovering 2024-09-21T10:59:29.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:29 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3528261188' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:59:29.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:29 smithi070 ceph-mon[112139]: pgmap v2969: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.5 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 28 MiB/s wr, 449 op/s; 35379/174291 objects degraded (20.299%); 9743/174291 objects misplaced (5.590%); 19 KiB/s, 0 objects/s recovering 2024-09-21T10:59:29.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:29 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3528261188' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:59:29.642 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:29.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:29.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:29.743 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:29.772 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:29.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:29.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:29 smithi160 ceph-mon[101879]: pgmap v2969: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.5 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 28 MiB/s wr, 449 op/s; 35379/174291 objects degraded (20.299%); 9743/174291 objects misplaced (5.590%); 19 KiB/s, 0 objects/s recovering 2024-09-21T10:59:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:29 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3528261188' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:59:29.874 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:29.904 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:29.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:29.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:30.006 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:30.038 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:30.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:30.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:30.140 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:30.163 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:30.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:30.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:30.266 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:30.289 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:30.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:30.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:30.391 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:30.415 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:30.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:30.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:30.487 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:59:30.516 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:30.539 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:30.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:30.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:30.625 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 74 16 60256 60240 50.8741 32.5 0.00332418 0.0196176 2024-09-21T10:59:30.641 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:30.663 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:30.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:30.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:30.747 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:30 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:59:30.747 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:30 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3528261188' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:59:30.747 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:30 smithi070 ceph-mon[112139]: osdmap e1552: 8 total, 6 up, 5 in 2024-09-21T10:59:30.747 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:30 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:59:30.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:30 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3528261188' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:59:30.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:30 smithi070 ceph-mon[107385]: osdmap e1552: 8 total, 6 up, 5 in 2024-09-21T10:59:30.765 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:30.788 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:30.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:30.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:30 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:59:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:30 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3528261188' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:59:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:30 smithi160 ceph-mon[101879]: osdmap e1552: 8 total, 6 up, 5 in 2024-09-21T10:59:30.889 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:30.896 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7, 4] out_osds: [2, 5, 0] dead_osds: [5, 6] live_osds: [0, 1, 2, 3, 4, 7] 2024-09-21T10:59:30.896 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:59:30.896 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-21T10:59:30.896 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-21T10:59:30.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:30.911 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:30.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:31.012 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:31.032 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:31.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:31.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:31.134 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:31.157 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:31.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:31.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:31.258 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:31.280 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:31.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:31.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:31.382 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:31.405 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:31.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:31.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:31.506 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:31.531 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:31.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:31.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:31.625 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 75 16 60602 60586 50.4841 21.625 0.00329563 0.0197642 2024-09-21T10:59:31.634 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:31.654 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:31.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:31.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:31.756 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:31.779 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:31.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:31.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:31 smithi160 ceph-mon[101879]: pgmap v2971: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.6 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 28 MiB/s wr, 441 op/s; 35749/178116 objects degraded (20.071%); 9999/178116 objects misplaced (5.614%); 26 KiB/s, 0 objects/s recovering 2024-09-21T10:59:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:31 smithi070 ceph-mon[107385]: pgmap v2971: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.6 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 28 MiB/s wr, 441 op/s; 35749/178116 objects degraded (20.071%); 9999/178116 objects misplaced (5.614%); 26 KiB/s, 0 objects/s recovering 2024-09-21T10:59:31.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:31 smithi070 ceph-mon[112139]: pgmap v2971: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.6 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 28 MiB/s wr, 441 op/s; 35749/178116 objects degraded (20.071%); 9999/178116 objects misplaced (5.614%); 26 KiB/s, 0 objects/s recovering 2024-09-21T10:59:31.880 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:31.901 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:31.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:31.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:32.003 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:32.029 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:32.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:32.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:32.131 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:32.153 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:32.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:32.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:32.255 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:32.280 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:32.280 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:59:32.281 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:59:32.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:32.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:32.299 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:59:32.300 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:59:32.319 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:59:32.319 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:59:32.339 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:59:32.339 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:59:32.359 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:59:32.360 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:59:32.360 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T10:59:32.360 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T10:59:32.381 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:59:32.381 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:59:32.383 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:32.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:32.409 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:32.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:32.510 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:32.534 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:32.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:32.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:32.625 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 76 16 61143 61127 50.2647 33.8125 0.00294965 0.0198599 2024-09-21T10:59:32.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:32 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4012682316' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:59:32.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:32 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4012682316' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:59:32.636 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:32.665 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:32.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:32.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:32.767 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:32.788 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:32.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:32.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:32.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:32 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4012682316' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:59:32.890 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:32.913 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:32.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:32.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:33.014 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:33.044 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:33.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:33.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:33.145 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:33.168 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:33.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:33.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:33.269 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:33.291 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:33.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:33.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:33.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:59:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:59:33] "GET /metrics HTTP/1.1" 200 36580 "" "Prometheus/2.51.0" 2024-09-21T10:59:33.393 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:33.417 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:33.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:33.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:33.519 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:33.542 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:33.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:33.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:33.625 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 77 16 61586 61570 49.9715 27.6875 0.00315372 0.0199831 2024-09-21T10:59:33.645 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:33.669 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:33.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:33.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:33.771 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:33.795 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:33.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:33.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:33 smithi160 ceph-mon[101879]: pgmap v2972: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.6 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 27 MiB/s wr, 437 op/s; 35749/178116 objects degraded (20.071%); 9999/178116 objects misplaced (5.614%); 26 KiB/s, 0 objects/s recovering 2024-09-21T10:59:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:33 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 35749/178116 objects degraded (20.071%), 22 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:59:33.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:33 smithi070 ceph-mon[107385]: pgmap v2972: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.6 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 27 MiB/s wr, 437 op/s; 35749/178116 objects degraded (20.071%); 9999/178116 objects misplaced (5.614%); 26 KiB/s, 0 objects/s recovering 2024-09-21T10:59:33.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:33 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 35749/178116 objects degraded (20.071%), 22 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:59:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:33 smithi070 ceph-mon[112139]: pgmap v2972: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.6 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 27 MiB/s wr, 437 op/s; 35749/178116 objects degraded (20.071%); 9999/178116 objects misplaced (5.614%); 26 KiB/s, 0 objects/s recovering 2024-09-21T10:59:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:33 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 35749/178116 objects degraded (20.071%), 22 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:59:33.897 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:33.919 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:33.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:33.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:33.924 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:59:33.949 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:59:33.949 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:59:34.022 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:34.048 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:34.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:34.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:34.072 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:59:34.150 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:34.176 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:34.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:34.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:34.277 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:34.302 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:34.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:34.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:34.404 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:34.426 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:34.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:34.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:34.528 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:34.551 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:34.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:34.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:34.626 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 78 16 62083 62067 49.729 31.0625 0.00458199 0.0200835 2024-09-21T10:59:34.653 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:34.678 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:34.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:34.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:34.780 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:34.815 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:34.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:34.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:34.917 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:34.944 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:34.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:34.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:35.046 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:35.072 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:35.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:35.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:35.174 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T10:59:35.174 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:59:35.174 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:35.201 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:35.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:35.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:35.303 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:35.324 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:35.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:35.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:35.426 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:35.448 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:35.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:35.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:35.550 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:35.572 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:35.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:35.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:35.600 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:35 smithi070 ceph-mon[107385]: pgmap v2973: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.7 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 27 MiB/s wr, 439 op/s; 36036/181878 objects degraded (19.813%); 10272/181878 objects misplaced (5.648%); 26 KiB/s, 0 objects/s recovering 2024-09-21T10:59:35.601 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:59:35.601 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:35 smithi070 ceph-mon[112139]: pgmap v2973: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.7 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 27 MiB/s wr, 439 op/s; 36036/181878 objects degraded (19.813%); 10272/181878 objects misplaced (5.648%); 26 KiB/s, 0 objects/s recovering 2024-09-21T10:59:35.601 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:59:35.626 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 79 16 62507 62491 49.4349 26.5 0.00424349 0.0201828 2024-09-21T10:59:35.674 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:35.696 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:35.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:35.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:35.797 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:35.827 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:35.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:35.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:35 smithi160 ceph-mon[101879]: pgmap v2973: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.7 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 27 MiB/s wr, 439 op/s; 36036/181878 objects degraded (19.813%); 10272/181878 objects misplaced (5.648%); 26 KiB/s, 0 objects/s recovering 2024-09-21T10:59:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:59:35.929 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:35.950 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:59:35.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:35.951 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:35.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:36.053 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:36.076 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:36.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:36.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:36.177 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:36.198 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:36.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:36.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:36.301 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:36.322 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:36.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:36.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:36.358 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:59:36.396 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:59:36.424 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:36.447 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:36.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:36.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:36.549 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:36.574 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:36.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:36.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:36.626 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:2024-09-21T10:59:36.625107+0000 min lat: 0.00141297 max lat: 5.2078 avg lat: 0.020302 2024-09-21T10:59:36.626 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:59:36.626 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 80 16 62957 62941 49.1685 28.125 0.005307 0.020302 2024-09-21T10:59:36.676 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:36.698 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:36.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:36.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:36.800 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:36.824 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:36.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:36.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:36.926 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:36.953 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:36.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:36.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:37.055 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:37.077 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:37.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:37.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:37.179 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:37.200 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:37.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:37.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:37.302 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:37.330 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:37.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:37.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:37.337 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:59:37.347 INFO:teuthology.orchestra.run.smithi070.stderr:no valid command found; 10 closest matches: 2024-09-21T10:59:37.347 INFO:teuthology.orchestra.run.smithi070.stderr:0 2024-09-21T10:59:37.347 INFO:teuthology.orchestra.run.smithi070.stderr:1 2024-09-21T10:59:37.347 INFO:teuthology.orchestra.run.smithi070.stderr:2 2024-09-21T10:59:37.348 INFO:teuthology.orchestra.run.smithi070.stderr:abort 2024-09-21T10:59:37.348 INFO:teuthology.orchestra.run.smithi070.stderr:assert 2024-09-21T10:59:37.348 INFO:teuthology.orchestra.run.smithi070.stderr:bench [] [] [] [] 2024-09-21T10:59:37.348 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:59:37.348 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs files list 2024-09-21T10:59:37.348 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs stats 2024-09-21T10:59:37.348 INFO:teuthology.orchestra.run.smithi070.stderr:bluestore allocator dump block 2024-09-21T10:59:37.348 INFO:teuthology.orchestra.run.smithi070.stderr:Error EINVAL: invalid command 2024-09-21T10:59:37.432 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:37.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:37.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:37.559 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:37.583 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:37.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:37.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:37.626 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 81 16 63390 63374 48.8956 27.0625 0.00306332 0.0204118 2024-09-21T10:59:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:37 smithi070 ceph-mon[107385]: pgmap v2974: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 28 MiB/s wr, 444 op/s; 36343/185055 objects degraded (19.639%); 10558/185055 objects misplaced (5.705%); 26 KiB/s, 0 objects/s recovering 2024-09-21T10:59:37.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:37 smithi070 ceph-mon[112139]: pgmap v2974: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 28 MiB/s wr, 444 op/s; 36343/185055 objects degraded (19.639%); 10558/185055 objects misplaced (5.705%); 26 KiB/s, 0 objects/s recovering 2024-09-21T10:59:37.685 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:37.712 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:37.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:37.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:37.814 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:37.835 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:37.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:37.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:37.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:37 smithi160 ceph-mon[101879]: pgmap v2974: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 28 MiB/s wr, 444 op/s; 36343/185055 objects degraded (19.639%); 10558/185055 objects misplaced (5.705%); 26 KiB/s, 0 objects/s recovering 2024-09-21T10:59:37.937 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:37.962 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:37.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:37.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:38.064 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:38.087 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:38.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:38.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:38.189 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:38.212 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:38.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:38.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:38.314 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:38.335 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:38.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:38.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:38.437 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:38.472 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:38.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:38.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:38.574 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:38.595 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:38.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:38.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:38.626 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 82 16 63859 63843 48.6568 29.3125 0.00352169 0.020523 2024-09-21T10:59:38.697 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:38.719 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:38.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:38.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:38.773 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:38 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 36343/185055 objects degraded (19.639%), 22 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:59:38.774 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:38 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 36343/185055 objects degraded (19.639%), 22 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:59:38.795 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:59:38.795 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:59:38.796 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-21T10:59:38.821 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:38 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 36343/185055 objects degraded (19.639%), 22 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:59:38.851 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:38.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:38.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:38.952 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:38.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:38.975 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:38.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:39.076 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:39.099 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:39.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:39.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:39.201 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:39.227 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:39.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:39.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:39.329 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:39.353 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:39.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:39.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:39.455 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:39.480 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:39.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:39.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:39.582 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:39.604 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:39.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:39.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:39.626 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 83 16 64316 64300 48.4146 28.5625 0.0036844 0.0206029 2024-09-21T10:59:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:39 smithi070 ceph-mon[107385]: pgmap v2975: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 28 MiB/s wr, 442 op/s; 36499/187563 objects degraded (19.460%); 10844/187563 objects misplaced (5.782%); 26 KiB/s, 0 objects/s recovering 2024-09-21T10:59:39.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:39 smithi070 ceph-mon[112139]: pgmap v2975: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 28 MiB/s wr, 442 op/s; 36499/187563 objects degraded (19.460%); 10844/187563 objects misplaced (5.782%); 26 KiB/s, 0 objects/s recovering 2024-09-21T10:59:39.706 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:39.728 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:39.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:39.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:39.830 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:39 smithi160 ceph-mon[101879]: pgmap v2975: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 28 MiB/s wr, 442 op/s; 36499/187563 objects degraded (19.460%); 10844/187563 objects misplaced (5.782%); 26 KiB/s, 0 objects/s recovering 2024-09-21T10:59:39.862 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:59:39.862 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:59:39.863 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:39.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:39.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:39.967 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:40.004 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:40.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:40.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:40.107 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:40.141 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:40.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:40.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:40.243 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:40.274 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:40.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:40.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:40.375 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:40.399 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:40.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:40.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:40.501 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:40.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:40.529 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:40.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:40.627 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 84 16 64687 64671 48.1142 23.1875 0.0029188 0.0207415 2024-09-21T10:59:40.630 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:40.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:40.655 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:40.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:40.756 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:40.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:40.782 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:40.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:40.883 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:40.907 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:40.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:40.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:40.992 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:59:40.992 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:59:41.009 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:41.012 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:59:41.012 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:59:41.031 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:59:41.031 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:59:41.032 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:41.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:41.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:41.049 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:59:41.049 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:59:41.070 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:59:41.071 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:59:41.071 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T10:59:41.071 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T10:59:41.088 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:59:41.088 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:59:41.135 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:41.160 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:41.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:41.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:41.262 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:41.289 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:41.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:41.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:41.391 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:41.412 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:41.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:41.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:41.514 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:41.541 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:41.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:41.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:41.626 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 85 16 65153 65137 47.8908 29.125 0.26132 0.0208406 2024-09-21T10:59:41.643 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:41.665 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:41.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:41.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:41.703 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:41 smithi070 ceph-mon[107385]: pgmap v2976: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 31 MiB/s wr, 499 op/s; 36926/191823 objects degraded (19.250%); 11126/191823 objects misplaced (5.800%); 47 KiB/s, 0 objects/s recovering 2024-09-21T10:59:41.703 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:41 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2436871476' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:59:41.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:41 smithi070 ceph-mon[112139]: pgmap v2976: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 31 MiB/s wr, 499 op/s; 36926/191823 objects degraded (19.250%); 11126/191823 objects misplaced (5.800%); 47 KiB/s, 0 objects/s recovering 2024-09-21T10:59:41.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:41 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2436871476' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:59:41.766 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:41.790 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:41.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:41.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:41.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:41 smithi160 ceph-mon[101879]: pgmap v2976: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 31 MiB/s wr, 499 op/s; 36926/191823 objects degraded (19.250%); 11126/191823 objects misplaced (5.800%); 47 KiB/s, 0 objects/s recovering 2024-09-21T10:59:41.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:41 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2436871476' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:59:41.892 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:41.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:41.930 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:41.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:42.031 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:42.039 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:59:42.040 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:59:42.054 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:42.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:42.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:42.155 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:42.177 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:42.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:42.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:42.278 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:42.301 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:42.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:42.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:42.403 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:42.429 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:42.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:42.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:42.530 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:42.555 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:42.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:42.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:42.626 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 86 16 65620 65604 47.6733 29.1875 0.00623363 0.0209485 2024-09-21T10:59:42.658 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:42.684 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:42.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:42.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:42.785 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:42.812 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:42.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:42.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:42.914 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:42.935 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:42.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:42.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:43.036 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:43.059 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:43.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:43.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:43.161 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:43.182 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:43.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:43.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:43.284 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:43.307 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:43.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:43.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:43.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:59:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:59:43] "GET /metrics HTTP/1.1" 200 36579 "" "Prometheus/2.51.0" 2024-09-21T10:59:43.409 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:43.438 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:43.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:43.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:43.525 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T10:59:43.540 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:43.564 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:43.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:43.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:43.626 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 87 16 66081 66065 47.4565 28.8125 0.0029171 0.0210429 2024-09-21T10:59:43.665 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:43.688 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:43.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:43.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:43.790 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:43.823 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:43.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:43.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:43 smithi160 ceph-mon[101879]: pgmap v2977: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 24 MiB/s wr, 380 op/s; 36926/191823 objects degraded (19.250%); 11126/191823 objects misplaced (5.800%); 32 KiB/s, 0 objects/s recovering 2024-09-21T10:59:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:43 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/887999104' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:59:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:43 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 36926/191823 objects degraded (19.250%), 22 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:59:43.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:43 smithi070 ceph-mon[107385]: pgmap v2977: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 24 MiB/s wr, 380 op/s; 36926/191823 objects degraded (19.250%); 11126/191823 objects misplaced (5.800%); 32 KiB/s, 0 objects/s recovering 2024-09-21T10:59:43.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:43 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/887999104' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:59:43.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:43 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 36926/191823 objects degraded (19.250%), 22 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:59:43.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:43 smithi070 ceph-mon[112139]: pgmap v2977: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 24 MiB/s wr, 380 op/s; 36926/191823 objects degraded (19.250%); 11126/191823 objects misplaced (5.800%); 32 KiB/s, 0 objects/s recovering 2024-09-21T10:59:43.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:43 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/887999104' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:59:43.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:43 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 36926/191823 objects degraded (19.250%), 22 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:59:43.924 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:43.946 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:43.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:43.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:44.041 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:59:44.048 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:44.106 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:44.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:44.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:44.208 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:44.229 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:44.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:44.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:44.331 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:44.352 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:44.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:44.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:44.454 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:44.476 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:44.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:44.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:44.578 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:44.600 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:44.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:44.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:44.626 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 88 16 66498 66482 47.2134 26.0625 0.00320564 0.0211489 2024-09-21T10:59:44.701 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:44.729 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:44.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:44.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:44.831 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:44 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:59:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:44 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/887999104' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:59:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:44 smithi160 ceph-mon[101879]: osdmap e1553: 8 total, 6 up, 5 in 2024-09-21T10:59:44.854 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:44.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:44.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:44.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:44 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:59:44.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:44 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/887999104' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:59:44.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:44 smithi070 ceph-mon[112139]: osdmap e1553: 8 total, 6 up, 5 in 2024-09-21T10:59:44.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:44 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:59:44.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:44 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/887999104' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:59:44.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:44 smithi070 ceph-mon[107385]: osdmap e1553: 8 total, 6 up, 5 in 2024-09-21T10:59:44.955 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:44.976 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:44.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:44.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:45.078 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:45.105 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:45.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:45.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:45.207 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:45.229 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:45.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:45.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:45.330 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:45.352 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:45.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:45.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:45.454 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:45.476 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:45.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:45.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:45.577 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:45.601 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:45.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:45.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:45.627 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 89 16 66959 66943 47.0066 28.8125 0.00385584 0.0212368 2024-09-21T10:59:45.677 INFO:teuthology.orchestra.run.smithi070.stderr:no valid command found; 10 closest matches: 2024-09-21T10:59:45.677 INFO:teuthology.orchestra.run.smithi070.stderr:0 2024-09-21T10:59:45.677 INFO:teuthology.orchestra.run.smithi070.stderr:1 2024-09-21T10:59:45.677 INFO:teuthology.orchestra.run.smithi070.stderr:2 2024-09-21T10:59:45.677 INFO:teuthology.orchestra.run.smithi070.stderr:abort 2024-09-21T10:59:45.677 INFO:teuthology.orchestra.run.smithi070.stderr:assert 2024-09-21T10:59:45.677 INFO:teuthology.orchestra.run.smithi070.stderr:bench [] [] [] [] 2024-09-21T10:59:45.677 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:59:45.677 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs files list 2024-09-21T10:59:45.678 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs stats 2024-09-21T10:59:45.678 INFO:teuthology.orchestra.run.smithi070.stderr:bluestore allocator dump block 2024-09-21T10:59:45.678 INFO:teuthology.orchestra.run.smithi070.stderr:Error EINVAL: invalid command 2024-09-21T10:59:45.678 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:45 smithi070 ceph-mon[107385]: pgmap v2979: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 4.0 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 28 MiB/s wr, 454 op/s; 37219/195513 objects degraded (19.037%); 11406/195513 objects misplaced (5.834%); 38 KiB/s, 0 objects/s recovering 2024-09-21T10:59:45.678 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:45 smithi070 ceph-mon[112139]: pgmap v2979: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 4.0 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 28 MiB/s wr, 454 op/s; 37219/195513 objects degraded (19.037%); 11406/195513 objects misplaced (5.834%); 38 KiB/s, 0 objects/s recovering 2024-09-21T10:59:45.702 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:45.724 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:45.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:45.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:45.825 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:45.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:45 smithi160 ceph-mon[101879]: pgmap v2979: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 4.0 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 28 MiB/s wr, 454 op/s; 37219/195513 objects degraded (19.037%); 11406/195513 objects misplaced (5.834%); 38 KiB/s, 0 objects/s recovering 2024-09-21T10:59:45.847 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:45.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:45.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:45.948 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:45.973 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:45.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:45.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:46.075 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:46.099 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:46.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:46.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:46.200 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:46.223 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:46.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:46.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:46.297 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:59:46.297 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T10:59:46.297 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T10:59:46.297 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T10:59:46.297 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:59:46.324 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:46.347 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:46.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:46.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:46.449 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:46.472 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:46.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:46.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:46.574 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:46.597 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:46.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:46.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:46.626 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 90 16 67346 67330 46.753 24.1875 0.00331206 0.0213544 2024-09-21T10:59:46.698 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:46.720 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:46.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:46.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:46.821 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:46.843 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:46.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:46.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:46.944 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:46.965 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:46.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:46.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:47.063 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:59:47.067 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:47.088 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:47.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:47.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:47.095 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:59:47.123 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:59:47.189 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:47.212 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:47.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:47.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:47.314 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:47.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:47.336 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:47.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:47.437 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:47.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:47.458 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:47.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:47.558 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:47.579 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:47.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:47.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:47.627 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Total time run: 90.3574 2024-09-21T10:59:47.627 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Total writes made: 67346 2024-09-21T10:59:47.627 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Write size: 65536 2024-09-21T10:59:47.627 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Object size: 65536 2024-09-21T10:59:47.627 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Bandwidth (MB/sec): 46.5831 2024-09-21T10:59:47.627 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Stddev Bandwidth: 48.5184 2024-09-21T10:59:47.627 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Max bandwidth (MB/sec): 254 2024-09-21T10:59:47.628 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Min bandwidth (MB/sec): 0 2024-09-21T10:59:47.628 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Average IOPS: 745 2024-09-21T10:59:47.628 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Stddev IOPS: 776.299 2024-09-21T10:59:47.628 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Max IOPS: 4064 2024-09-21T10:59:47.628 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Min IOPS: 0 2024-09-21T10:59:47.628 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Average Latency(s): 0.0214192 2024-09-21T10:59:47.628 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Stddev Latency(s): 0.115539 2024-09-21T10:59:47.628 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Max latency(s): 5.2078 2024-09-21T10:59:47.628 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Min latency(s): 0.00141297 2024-09-21T10:59:47.681 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:47.682 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T10:59:47.713 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:47.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:47.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:47.716 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:59:47.717 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:59:47.717 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-21T10:59:47.815 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:47.839 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:47.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:47.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:47.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:47 smithi160 ceph-mon[101879]: pgmap v2980: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 4.0 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 28 MiB/s wr, 444 op/s; 37498/198387 objects degraded (18.901%); 11674/198387 objects misplaced (5.884%); 45 KiB/s, 0 objects/s recovering 2024-09-21T10:59:47.874 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:47 smithi070 ceph-mon[107385]: pgmap v2980: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 4.0 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 28 MiB/s wr, 444 op/s; 37498/198387 objects degraded (18.901%); 11674/198387 objects misplaced (5.884%); 45 KiB/s, 0 objects/s recovering 2024-09-21T10:59:47.874 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:47 smithi070 ceph-mon[112139]: pgmap v2980: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 4.0 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 28 MiB/s wr, 444 op/s; 37498/198387 objects degraded (18.901%); 11674/198387 objects misplaced (5.884%); 45 KiB/s, 0 objects/s recovering 2024-09-21T10:59:47.940 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:47.961 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:47.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:47.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:48.063 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:48.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:48.085 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:48.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:48.186 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:48.208 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:48.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:48.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:48.309 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:48.331 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:48.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:48.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:48.432 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:48.454 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:48.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:48.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:48.515 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:59:48.556 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:48.557 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:59:48.557 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:59:48.580 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:48.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:48.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:48.681 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:48.702 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:48.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:48.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:48.804 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:48.809 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:48 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 37498/198387 objects degraded (18.901%), 22 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-09-21T10:59:48.810 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:48 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 37498/198387 objects degraded (18.901%), 22 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-09-21T10:59:48.831 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:48.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:48.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:48 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 37498/198387 objects degraded (18.901%), 22 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-09-21T10:59:48.932 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:48.954 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:48.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:48.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:49.056 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:49.078 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:49.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:49.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:49.180 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:49.203 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:49.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:49.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:49.305 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:49.328 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:49.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:49.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:49.430 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:49.450 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:49.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:49.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:49.552 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:49.575 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:49.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:49.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:49.676 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:49.698 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:49.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:49.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:49.799 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:49.821 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:49.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:49.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:49 smithi160 ceph-mon[101879]: pgmap v2981: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 4.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 27 MiB/s wr, 430 op/s; 37630/200478 objects degraded (18.770%); 11892/200478 objects misplaced (5.932%); 45 KiB/s, 0 objects/s recovering 2024-09-21T10:59:49.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:49 smithi070 ceph-mon[107385]: pgmap v2981: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 4.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 27 MiB/s wr, 430 op/s; 37630/200478 objects degraded (18.770%); 11892/200478 objects misplaced (5.932%); 45 KiB/s, 0 objects/s recovering 2024-09-21T10:59:49.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:49 smithi070 ceph-mon[112139]: pgmap v2981: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 4.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 27 MiB/s wr, 430 op/s; 37630/200478 objects degraded (18.770%); 11892/200478 objects misplaced (5.932%); 45 KiB/s, 0 objects/s recovering 2024-09-21T10:59:49.923 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:49.944 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:49.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:49.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:50.046 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:50.067 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:50.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:50.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:50.169 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:50.191 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:50.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:50.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:50.292 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:50.315 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:50.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:50.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:50.418 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:50.438 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:50.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:50.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:50.540 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:50.563 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:50.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:50.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:50.665 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:50.686 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:50.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:50.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:50.788 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:50.813 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:50.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:50.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:59:50.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:59:50.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:59:50.916 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:50.938 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:50.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:50.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:51.039 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:51.061 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:51.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:51.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:51.163 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:51.185 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:51.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:51.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:51.287 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:51.309 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:51.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:51.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:51.411 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:51.432 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:51.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:51.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:51.533 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:51.556 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:51.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:51.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:51.658 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:51.680 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:51.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:51.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:51.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:51 smithi070 ceph-mon[107385]: pgmap v2982: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 4.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 102 B/s rd, 21 MiB/s wr, 371 op/s; 37642/201261 objects degraded (18.703%); 11932/201261 objects misplaced (5.929%); 429 KiB/s, 6 objects/s recovering 2024-09-21T10:59:51.753 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:51 smithi070 ceph-mon[112139]: pgmap v2982: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 4.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 102 B/s rd, 21 MiB/s wr, 371 op/s; 37642/201261 objects degraded (18.703%); 11932/201261 objects misplaced (5.929%); 429 KiB/s, 6 objects/s recovering 2024-09-21T10:59:51.781 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:51.802 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:51.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:51.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:51 smithi160 ceph-mon[101879]: pgmap v2982: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 4.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 102 B/s rd, 21 MiB/s wr, 371 op/s; 37642/201261 objects degraded (18.703%); 11932/201261 objects misplaced (5.929%); 429 KiB/s, 6 objects/s recovering 2024-09-21T10:59:51.904 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:51.925 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:51.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:51.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:52.026 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:52.048 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:52.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:52.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:52.149 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:52.170 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:52.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:52.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:52.272 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:52.295 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:52.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:52.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:52.396 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:52.417 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:52.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:52.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:52.519 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:52.540 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:52.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:52.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:52.642 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:52.665 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:52.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:52.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:52.729 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T10:59:52.729 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:59:52.749 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T10:59:52.750 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:59:52.767 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:52.770 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T10:59:52.770 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:59:52.788 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:52.788 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T10:59:52.788 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:59:52.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:52.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:52.805 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T10:59:52.805 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:59:52.806 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T10:59:52.806 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T10:59:52.824 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T10:59:52.824 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:59:52.890 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:52.912 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:52.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:52.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:53.013 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:53.035 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:53.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:53.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:53.136 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:53.158 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:53.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:53.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:53.259 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:53.280 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:53.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:53.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:53.382 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:53.403 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:53.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:53.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:53.462 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 10:59:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:10:59:53] "GET /metrics HTTP/1.1" 200 36576 "" "Prometheus/2.51.0" 2024-09-21T10:59:53.505 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:53.526 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:53.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:53.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:53.590 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T10:59:53.590 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T10:59:53.590 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T10:59:53.590 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T10:59:53.590 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.590 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:129295 55.1 e1446/1440 55:8efdf0b1:::benchmark_data_smithi070_294394_object23954:head v 1446'8143, mlcod=0'0)", 2024-09-21T10:59:53.590 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:22.707277+0000", 2024-09-21T10:59:53.590 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 509.921234293, 2024-09-21T10:59:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27558063199999999, 2024-09-21T10:59:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:59:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:59:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:59:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:59:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:59:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.707277+0000", 2024-09-21T10:59:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:59:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.707277+0000", 2024-09-21T10:59:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:59:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.707278+0000", 2024-09-21T10:59:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0299999999999997e-07 2024-09-21T10:59:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:59:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.707285+0000", 2024-09-21T10:59:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8850000000000001e-06 2024-09-21T10:59:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:59:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.707296+0000", 2024-09-21T10:59:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0993e-05 2024-09-21T10:59:53.593 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.593 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.593 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:59:53.593 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.707303+0000", 2024-09-21T10:59:53.593 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7120000000000003e-06 2024-09-21T10:59:53.593 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.593 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.593 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:59:53.593 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.945300+0000", 2024-09-21T10:59:53.593 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.23799742600000001 2024-09-21T10:59:53.593 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.593 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.593 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:59:53.593 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.945329+0000", 2024-09-21T10:59:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8379999999999999e-05 2024-09-21T10:59:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:59:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.982817+0000", 2024-09-21T10:59:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.037488847999999998 2024-09-21T10:59:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:59:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.982858+0000", 2024-09-21T10:59:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0485000000000003e-05 2024-09-21T10:59:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:59:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:129296 55.8 55:1cab724d:::benchmark_data_smithi070_294394_object23955:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:59:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:22.708169+0000", 2024-09-21T10:59:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 509.920342855, 2024-09-21T10:59:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.24355597700000001, 2024-09-21T10:59:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:59:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:59:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:59:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:59:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:59:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:59:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 129296 2024-09-21T10:59:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:59:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:59:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.708169+0000", 2024-09-21T10:59:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:59:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.708169+0000", 2024-09-21T10:59:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:59:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.708171+0000", 2024-09-21T10:59:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0150000000000002e-06 2024-09-21T10:59:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:59:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.708177+0000", 2024-09-21T10:59:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.852e-06 2024-09-21T10:59:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:59:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.708190+0000", 2024-09-21T10:59:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3148e-05 2024-09-21T10:59:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:59:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.708197+0000", 2024-09-21T10:59:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0040000000000002e-06 2024-09-21T10:59:53.598 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.598 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.598 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:59:53.598 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.708739+0000", 2024-09-21T10:59:53.598 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00054260300000000001 2024-09-21T10:59:53.598 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.598 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.598 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:59:53.598 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.708879+0000", 2024-09-21T10:59:53.598 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013970700000000001 2024-09-21T10:59:53.598 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.598 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.598 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:59:53.599 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.709019+0000", 2024-09-21T10:59:53.599 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013960900000000001 2024-09-21T10:59:53.599 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.599 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.599 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:59:53.599 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.713167+0000", 2024-09-21T10:59:53.599 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0041479009999999998 2024-09-21T10:59:53.599 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.599 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.599 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:53.599 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.714761+0000", 2024-09-21T10:59:53.599 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001594729 2024-09-21T10:59:53.599 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.599 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.599 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:53.600 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.949456+0000", 2024-09-21T10:59:53.600 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.23469445 2024-09-21T10:59:53.600 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.600 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.600 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:59:53.600 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.949604+0000", 2024-09-21T10:59:53.600 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000147894 2024-09-21T10:59:53.600 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.600 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.600 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:59:53.600 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.951725+0000", 2024-09-21T10:59:53.600 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0021210650000000001 2024-09-21T10:59:53.600 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.600 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:59:53.600 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.601 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.601 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.601 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:129298 55.5 55:aae0d80d:::benchmark_data_smithi070_294394_object23957:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:59:53.601 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:22.714780+0000", 2024-09-21T10:59:53.601 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 509.91373179200002, 2024-09-21T10:59:53.601 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.23622679199999999, 2024-09-21T10:59:53.601 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:59:53.601 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:59:53.601 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:59:53.601 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:59:53.601 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:59:53.601 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:59:53.601 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 129298 2024-09-21T10:59:53.601 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.601 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:59:53.601 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.602 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:59:53.602 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.714780+0000", 2024-09-21T10:59:53.602 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.602 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.602 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.602 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:59:53.602 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.714780+0000", 2024-09-21T10:59:53.602 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.602 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.602 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.602 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:59:53.602 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.714782+0000", 2024-09-21T10:59:53.602 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9489999999999999e-06 2024-09-21T10:59:53.602 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.602 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.603 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:59:53.603 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.714788+0000", 2024-09-21T10:59:53.603 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1210000000000003e-06 2024-09-21T10:59:53.603 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.603 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.603 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:59:53.603 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.714800+0000", 2024-09-21T10:59:53.603 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.219e-05 2024-09-21T10:59:53.603 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.603 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.603 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:59:53.603 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.714808+0000", 2024-09-21T10:59:53.603 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7749999999999993e-06 2024-09-21T10:59:53.603 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.603 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.604 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:59:53.604 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.714867+0000", 2024-09-21T10:59:53.604 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8956999999999998e-05 2024-09-21T10:59:53.604 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.604 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.604 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:59:53.604 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.714987+0000", 2024-09-21T10:59:53.604 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000120728 2024-09-21T10:59:53.604 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.604 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.604 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,3,6", 2024-09-21T10:59:53.604 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.715375+0000", 2024-09-21T10:59:53.604 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000387814 2024-09-21T10:59:53.604 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.604 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.605 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:53.605 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.947741+0000", 2024-09-21T10:59:53.605 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.232365505 2024-09-21T10:59:53.605 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.605 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.605 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:53.605 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.947799+0000", 2024-09-21T10:59:53.605 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7720999999999998e-05 2024-09-21T10:59:53.605 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.605 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.605 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:53.605 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.947853+0000", 2024-09-21T10:59:53.605 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4453000000000002e-05 2024-09-21T10:59:53.605 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.606 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.606 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:59:53.606 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.950846+0000", 2024-09-21T10:59:53.606 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0029934509999999998 2024-09-21T10:59:53.606 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.606 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.606 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:59:53.606 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.950899+0000", 2024-09-21T10:59:53.606 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2769000000000002e-05 2024-09-21T10:59:53.606 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.606 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.606 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:59:53.606 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.951007+0000", 2024-09-21T10:59:53.606 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000107359 2024-09-21T10:59:53.607 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.607 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:59:53.607 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.607 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.607 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.607 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "MOSDPGPush(55.2 1446/1434 [PushOp(55:42275384:::benchmark_data_smithi070_294394_object23949:head, version: 1416'1519, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(55:42275384:::benchmark_data_smithi070_294394_object23949:head@1416'1519, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T10:59:53.607 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:22.715198+0000", 2024-09-21T10:59:53.607 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 509.91331390200003, 2024-09-21T10:59:53.607 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.233190694, 2024-09-21T10:59:53.607 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:59:53.607 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:59:53.607 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "started", 2024-09-21T10:59:53.607 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:59:53.607 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.608 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:59:53.608 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.715198+0000", 2024-09-21T10:59:53.608 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.608 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.608 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.608 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:59:53.608 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.715198+0000", 2024-09-21T10:59:53.608 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.608 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.608 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.608 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:59:53.608 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.715198+0000", 2024-09-21T10:59:53.608 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6400000000000002e-07 2024-09-21T10:59:53.608 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.608 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.609 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:59:53.609 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.715228+0000", 2024-09-21T10:59:53.609 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9607e-05 2024-09-21T10:59:53.609 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.609 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.609 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:59:53.609 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.715247+0000", 2024-09-21T10:59:53.609 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8816000000000001e-05 2024-09-21T10:59:53.609 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.609 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.609 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:59:53.609 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.715262+0000", 2024-09-21T10:59:53.609 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5613000000000001e-05 2024-09-21T10:59:53.609 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.609 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.610 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:59:53.610 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.715366+0000", 2024-09-21T10:59:53.610 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000103769 2024-09-21T10:59:53.610 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.610 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.610 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:59:53.610 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.715386+0000", 2024-09-21T10:59:53.610 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0136000000000002e-05 2024-09-21T10:59:53.610 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.610 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.610 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:59:53.610 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.948388+0000", 2024-09-21T10:59:53.611 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.233002089 2024-09-21T10:59:53.611 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.611 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:59:53.611 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.611 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.611 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.611 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "MOSDPGPushReply(55.8 1446/1428 [PushReplyOp(55:1a4fe3ae:::benchmark_data_smithi070_294394_object37264:head)])", 2024-09-21T10:59:53.611 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:22.716411+0000", 2024-09-21T10:59:53.611 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 509.91210089700002, 2024-09-21T10:59:53.611 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.23640767900000001, 2024-09-21T10:59:53.611 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:59:53.611 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:59:53.611 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:59:53.611 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:59:53.611 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.612 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:59:53.612 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.716411+0000", 2024-09-21T10:59:53.612 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.612 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.612 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.612 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:59:53.612 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.716411+0000", 2024-09-21T10:59:53.612 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.612 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.612 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.612 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:59:53.612 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.716411+0000", 2024-09-21T10:59:53.612 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.85e-07 2024-09-21T10:59:53.612 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.612 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.613 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:59:53.613 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.716416+0000", 2024-09-21T10:59:53.613 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8189999999999999e-06 2024-09-21T10:59:53.613 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.613 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.613 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:59:53.613 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.945276+0000", 2024-09-21T10:59:53.613 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.22885976299999999 2024-09-21T10:59:53.613 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.613 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.613 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:59:53.613 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.945289+0000", 2024-09-21T10:59:53.613 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3278000000000001e-05 2024-09-21T10:59:53.613 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.613 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.614 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:59:53.614 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.945376+0000", 2024-09-21T10:59:53.614 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6266000000000004e-05 2024-09-21T10:59:53.614 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.614 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.614 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:59:53.614 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.952818+0000", 2024-09-21T10:59:53.614 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0074427679999999998 2024-09-21T10:59:53.614 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.614 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:59:53.614 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.614 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.614 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.614 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:129300 55.5 55:adccdd22:::benchmark_data_smithi070_294394_object23959:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:59:53.614 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:22.716445+0000", 2024-09-21T10:59:53.615 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 509.91206649499998, 2024-09-21T10:59:53.615 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.24455037499999999, 2024-09-21T10:59:53.615 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:59:53.615 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:59:53.615 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:59:53.615 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:59:53.615 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:59:53.615 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:59:53.615 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 129300 2024-09-21T10:59:53.615 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.615 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:59:53.615 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.615 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:59:53.615 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.716445+0000", 2024-09-21T10:59:53.615 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:59:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.716445+0000", 2024-09-21T10:59:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:59:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.716446+0000", 2024-09-21T10:59:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.322e-06 2024-09-21T10:59:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:59:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.716451+0000", 2024-09-21T10:59:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2429999999999999e-06 2024-09-21T10:59:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.617 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.617 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:59:53.617 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.945269+0000", 2024-09-21T10:59:53.617 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.22881818800000001 2024-09-21T10:59:53.617 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.617 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.617 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:59:53.617 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.945287+0000", 2024-09-21T10:59:53.617 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8023999999999999e-05 2024-09-21T10:59:53.617 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.617 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.617 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:59:53.617 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.951033+0000", 2024-09-21T10:59:53.617 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0057462950000000002 2024-09-21T10:59:53.618 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.618 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.618 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:59:53.618 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.951142+0000", 2024-09-21T10:59:53.618 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000108631 2024-09-21T10:59:53.618 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.618 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.618 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,3,6", 2024-09-21T10:59:53.618 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.951254+0000", 2024-09-21T10:59:53.618 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000112182 2024-09-21T10:59:53.618 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.618 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.618 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:53.618 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.960562+0000", 2024-09-21T10:59:53.619 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0093075840000000007 2024-09-21T10:59:53.619 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.619 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.619 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:53.619 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.960608+0000", 2024-09-21T10:59:53.619 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6307999999999998e-05 2024-09-21T10:59:53.619 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.619 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.619 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:59:53.619 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.960650+0000", 2024-09-21T10:59:53.619 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.1752000000000003e-05 2024-09-21T10:59:53.619 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.619 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.619 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T10:59:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.960827+0000", 2024-09-21T10:59:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017739700000000001 2024-09-21T10:59:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:59:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.960880+0000", 2024-09-21T10:59:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3146000000000002e-05 2024-09-21T10:59:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:59:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.960995+0000", 2024-09-21T10:59:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000115303 2024-09-21T10:59:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:59:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.621 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.621 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop_reply(client.35670.0:129298 55.5 e1446/1434 ondisk, result = 0)", 2024-09-21T10:59:53.621 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:22.718413+0000", 2024-09-21T10:59:53.621 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 509.91009901699999, 2024-09-21T10:59:53.621 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.229390868, 2024-09-21T10:59:53.621 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:59:53.621 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:59:53.621 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "started", 2024-09-21T10:59:53.621 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:59:53.621 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.621 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:59:53.621 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.718413+0000", 2024-09-21T10:59:53.621 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.621 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.621 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.622 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:59:53.622 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.718413+0000", 2024-09-21T10:59:53.622 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.622 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.622 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.622 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:59:53.622 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.718414+0000", 2024-09-21T10:59:53.622 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.46e-06 2024-09-21T10:59:53.622 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.622 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.622 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:59:53.622 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.718421+0000", 2024-09-21T10:59:53.622 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4850000000000003e-06 2024-09-21T10:59:53.622 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.623 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.623 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:59:53.623 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.945399+0000", 2024-09-21T10:59:53.623 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.226977014 2024-09-21T10:59:53.623 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.623 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.623 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:59:53.623 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.945406+0000", 2024-09-21T10:59:53.623 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0609999999999998e-06 2024-09-21T10:59:53.623 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.623 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.623 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:59:53.623 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.947791+0000", 2024-09-21T10:59:53.623 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0023850149999999999 2024-09-21T10:59:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:59:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.947795+0000", 2024-09-21T10:59:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7110000000000003e-06 2024-09-21T10:59:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:59:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:22.947803+0000", 2024-09-21T10:59:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1219999999999995e-06 2024-09-21T10:59:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:59:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:130858 55.8 55:178fc653:::benchmark_data_smithi070_294394_object25517:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:59:53.625 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:26.393677+0000", 2024-09-21T10:59:53.625 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 506.23483412899998, 2024-09-21T10:59:53.625 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0198451989999999, 2024-09-21T10:59:53.625 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:59:53.625 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:59:53.625 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:59:53.625 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:59:53.625 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:59:53.625 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:59:53.625 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 130858 2024-09-21T10:59:53.625 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.625 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:59:53.625 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.625 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:59:53.625 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.393677+0000", 2024-09-21T10:59:53.626 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.626 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.626 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.626 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:59:53.626 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.393677+0000", 2024-09-21T10:59:53.626 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.626 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.626 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.626 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:59:53.626 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.393679+0000", 2024-09-21T10:59:53.626 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7349999999999999e-06 2024-09-21T10:59:53.626 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.626 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.626 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:59:53.626 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.393686+0000", 2024-09-21T10:59:53.627 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4930000000000003e-06 2024-09-21T10:59:53.627 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.627 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.627 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:59:53.627 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.393701+0000", 2024-09-21T10:59:53.627 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5466999999999999e-05 2024-09-21T10:59:53.627 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.627 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.627 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:59:53.627 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.393710+0000", 2024-09-21T10:59:53.627 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.8789999999999992e-06 2024-09-21T10:59:53.627 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.627 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.627 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:59:53.628 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.395855+0000", 2024-09-21T10:59:53.628 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0021445420000000001 2024-09-21T10:59:53.628 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.628 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.628 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:59:53.628 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.395988+0000", 2024-09-21T10:59:53.628 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013296900000000001 2024-09-21T10:59:53.628 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.628 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.628 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:59:53.628 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.396195+0000", 2024-09-21T10:59:53.628 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000207035 2024-09-21T10:59:53.628 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.628 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.628 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:59:53.628 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.445846+0000", 2024-09-21T10:59:53.629 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.049651140000000003 2024-09-21T10:59:53.629 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.629 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.629 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:59:53.629 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.445859+0000", 2024-09-21T10:59:53.629 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3772e-05 2024-09-21T10:59:53.629 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.629 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.629 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:59:53.629 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.413508+0000", 2024-09-21T10:59:53.629 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.96764826299999995 2024-09-21T10:59:53.629 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.629 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.629 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:59:53.629 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.413523+0000", 2024-09-21T10:59:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4904e-05 2024-09-21T10:59:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:59:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:130866 55.8 55.11526d38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1446)", 2024-09-21T10:59:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:26.400243+0000", 2024-09-21T10:59:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 506.22826817800001, 2024-09-21T10:59:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.013316393, 2024-09-21T10:59:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:59:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:59:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:59:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:59:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:59:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:59:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 130866 2024-09-21T10:59:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:59:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:59:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.400243+0000", 2024-09-21T10:59:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:59:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.400243+0000", 2024-09-21T10:59:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:59:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.400245+0000", 2024-09-21T10:59:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0829999999999998e-06 2024-09-21T10:59:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:59:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.400251+0000", 2024-09-21T10:59:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3410000000000002e-06 2024-09-21T10:59:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:59:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.400261+0000", 2024-09-21T10:59:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7210000000000006e-06 2024-09-21T10:59:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.633 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.633 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:59:53.633 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.400267+0000", 2024-09-21T10:59:53.633 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8560000000000001e-06 2024-09-21T10:59:53.633 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.633 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.633 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:59:53.633 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.446742+0000", 2024-09-21T10:59:53.633 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.046474840000000003 2024-09-21T10:59:53.633 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.633 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.633 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:59:53.633 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.446887+0000", 2024-09-21T10:59:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014484099999999999 2024-09-21T10:59:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:59:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.413550+0000", 2024-09-21T10:59:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.96666295099999999 2024-09-21T10:59:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:59:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.413560+0000", 2024-09-21T10:59:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7599999999999997e-06 2024-09-21T10:59:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:59:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:130872 55.8 55.862244f8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1447)", 2024-09-21T10:59:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:26.466075+0000", 2024-09-21T10:59:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 506.16243684900002, 2024-09-21T10:59:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.94751783499999997, 2024-09-21T10:59:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:59:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:59:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:59:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:59:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:59:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:59:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 130872 2024-09-21T10:59:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:59:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:59:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.466075+0000", 2024-09-21T10:59:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:59:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.466075+0000", 2024-09-21T10:59:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:59:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.466077+0000", 2024-09-21T10:59:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9479999999999998e-06 2024-09-21T10:59:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.637 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.637 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:59:53.637 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.466085+0000", 2024-09-21T10:59:53.637 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3850000000000002e-06 2024-09-21T10:59:53.637 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.637 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.637 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:59:53.637 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.466104+0000", 2024-09-21T10:59:53.637 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9218000000000001e-05 2024-09-21T10:59:53.637 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.637 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.637 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:59:53.637 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.466115+0000", 2024-09-21T10:59:53.637 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0271e-05 2024-09-21T10:59:53.637 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.638 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.638 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:59:53.638 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.466576+0000", 2024-09-21T10:59:53.638 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00046172299999999998 2024-09-21T10:59:53.638 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.638 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.638 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:59:53.638 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.466585+0000", 2024-09-21T10:59:53.638 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7509999999999997e-06 2024-09-21T10:59:53.638 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.638 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.638 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:59:53.638 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.413582+0000", 2024-09-21T10:59:53.638 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.946997333 2024-09-21T10:59:53.638 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:59:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.413593+0000", 2024-09-21T10:59:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0206e-05 2024-09-21T10:59:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:59:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:130874 55.5 55.1ccdde55 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1447)", 2024-09-21T10:59:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:51:26.472197+0000", 2024-09-21T10:59:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 506.156314119, 2024-09-21T10:59:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.94240631200000002, 2024-09-21T10:59:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:59:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:59:53.640 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:59:53.640 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T10:59:53.640 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T10:59:53.640 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T10:59:53.640 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 130874 2024-09-21T10:59:53.640 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.640 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:59:53.640 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.640 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:59:53.640 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.472197+0000", 2024-09-21T10:59:53.640 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.640 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.640 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.640 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:59:53.640 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.472197+0000", 2024-09-21T10:59:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:59:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.472200+0000", 2024-09-21T10:59:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3329999999999999e-06 2024-09-21T10:59:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:59:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.472209+0000", 2024-09-21T10:59:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.884e-06 2024-09-21T10:59:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:59:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.472231+0000", 2024-09-21T10:59:53.642 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2211000000000001e-05 2024-09-21T10:59:53.642 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.642 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.642 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:59:53.642 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.472242+0000", 2024-09-21T10:59:53.642 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1233000000000001e-05 2024-09-21T10:59:53.642 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.642 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.642 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:59:53.642 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.472325+0000", 2024-09-21T10:59:53.642 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3275000000000005e-05 2024-09-21T10:59:53.642 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.642 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T10:59:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:26.472338+0000", 2024-09-21T10:59:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2302e-05 2024-09-21T10:59:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:59:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.414595+0000", 2024-09-21T10:59:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.942257601 2024-09-21T10:59:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:59:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:51:27.414604+0000", 2024-09-21T10:59:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.473e-06 2024-09-21T10:59:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:59:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:156258 55.c e1464/1448 55:39292414:::benchmark_data_smithi070_294394_object50917:head v 1464'9744, mlcod=0'0)", 2024-09-21T10:59:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.338132+0000", 2024-09-21T10:59:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 448.29037995300001, 2024-09-21T10:59:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.15064587099999999, 2024-09-21T10:59:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:59:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:59:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:59:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:59:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:59:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.338132+0000", 2024-09-21T10:59:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:59:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.338132+0000", 2024-09-21T10:59:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:59:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.338133+0000", 2024-09-21T10:59:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7900000000000007e-07 2024-09-21T10:59:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:59:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.338140+0000", 2024-09-21T10:59:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1740000000000003e-06 2024-09-21T10:59:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:59:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.338152+0000", 2024-09-21T10:59:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2215000000000001e-05 2024-09-21T10:59:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:59:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.338159+0000", 2024-09-21T10:59:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3520000000000001e-06 2024-09-21T10:59:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:59:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.338240+0000", 2024-09-21T10:59:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1086000000000003e-05 2024-09-21T10:59:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:59:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.338267+0000", 2024-09-21T10:59:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6194999999999999e-05 2024-09-21T10:59:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:59:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.488714+0000", 2024-09-21T10:59:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.15044706499999999 2024-09-21T10:59:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:59:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.488777+0000", 2024-09-21T10:59:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3805000000000003e-05 2024-09-21T10:59:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:59:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:156268 55.f e1464/1448 55:f53808b0:::benchmark_data_smithi070_294394_object50927:head v 1464'9749, mlcod=0'0)", 2024-09-21T10:59:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.340846+0000", 2024-09-21T10:59:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 448.28766603299999, 2024-09-21T10:59:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.158773426, 2024-09-21T10:59:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:59:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:59:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:59:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:59:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:59:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.340846+0000", 2024-09-21T10:59:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:59:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.340846+0000", 2024-09-21T10:59:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:59:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.340847+0000", 2024-09-21T10:59:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3650000000000001e-06 2024-09-21T10:59:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:59:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.340855+0000", 2024-09-21T10:59:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7100000000000007e-06 2024-09-21T10:59:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:59:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.340868+0000", 2024-09-21T10:59:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3312000000000001e-05 2024-09-21T10:59:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:59:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.340876+0000", 2024-09-21T10:59:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5320000000000006e-06 2024-09-21T10:59:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:59:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.478844+0000", 2024-09-21T10:59:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13796760599999999 2024-09-21T10:59:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:59:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.478891+0000", 2024-09-21T10:59:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6912000000000001e-05 2024-09-21T10:59:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:59:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.499553+0000", 2024-09-21T10:59:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.020662064000000001 2024-09-21T10:59:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:59:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.499619+0000", 2024-09-21T10:59:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5925000000000003e-05 2024-09-21T10:59:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:59:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:156262 55.3 e1464/1448)", 2024-09-21T10:59:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.343780+0000", 2024-09-21T10:59:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 448.28473124200002, 2024-09-21T10:59:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13981411199999999, 2024-09-21T10:59:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:59:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:59:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T10:59:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:59:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.653 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:59:53.653 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.343780+0000", 2024-09-21T10:59:53.653 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.653 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.653 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.653 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:59:53.653 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.343780+0000", 2024-09-21T10:59:53.653 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.653 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.653 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.653 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:59:53.653 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.343782+0000", 2024-09-21T10:59:53.653 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.28e-06 2024-09-21T10:59:53.653 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.653 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.653 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:59:53.654 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.343790+0000", 2024-09-21T10:59:53.654 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9440000000000005e-06 2024-09-21T10:59:53.654 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.654 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.654 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:59:53.654 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.475488+0000", 2024-09-21T10:59:53.654 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13169877499999999 2024-09-21T10:59:53.654 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.654 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.654 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:59:53.654 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.475507+0000", 2024-09-21T10:59:53.654 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8768000000000001e-05 2024-09-21T10:59:53.654 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.654 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.655 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:59:53.655 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.483581+0000", 2024-09-21T10:59:53.655 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0080743680000000002 2024-09-21T10:59:53.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.655 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:59:53.655 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.483594+0000", 2024-09-21T10:59:53.655 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2977e-05 2024-09-21T10:59:53.655 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.655 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:59:53.655 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:156269 55.f e1464/1448 55:fb867c57:::benchmark_data_smithi070_294394_object50928:head v 1464'9750, mlcod=0'0)", 2024-09-21T10:59:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.344388+0000", 2024-09-21T10:59:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 448.284123916, 2024-09-21T10:59:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.156159983, 2024-09-21T10:59:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:59:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:59:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:59:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:59:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:59:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.344388+0000", 2024-09-21T10:59:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:59:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.344388+0000", 2024-09-21T10:59:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:59:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.344388+0000", 2024-09-21T10:59:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9999999999999996e-07 2024-09-21T10:59:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:59:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.344395+0000", 2024-09-21T10:59:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7240000000000003e-06 2024-09-21T10:59:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.658 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:59:53.658 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.473817+0000", 2024-09-21T10:59:53.658 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.129422075 2024-09-21T10:59:53.658 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.658 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.658 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:59:53.658 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.473834+0000", 2024-09-21T10:59:53.658 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6381000000000001e-05 2024-09-21T10:59:53.658 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.658 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.658 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:59:53.658 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.494817+0000", 2024-09-21T10:59:53.658 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.020983841999999999 2024-09-21T10:59:53.658 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.659 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:59:53.659 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.494841+0000", 2024-09-21T10:59:53.659 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3091999999999998e-05 2024-09-21T10:59:53.659 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.659 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:59:53.659 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.500504+0000", 2024-09-21T10:59:53.659 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0056635160000000004 2024-09-21T10:59:53.659 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.659 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:59:53.659 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.500548+0000", 2024-09-21T10:59:53.659 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3553000000000001e-05 2024-09-21T10:59:53.659 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.660 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:59:53.660 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.660 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:195880 55.c e1474/1465 55:341b656a:::benchmark_data_smithi070_294394_object90539:head v 1474'12198, mlcod=0'0)", 2024-09-21T10:59:53.660 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:53:26.307538+0000", 2024-09-21T10:59:53.660 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 386.32097373900001, 2024-09-21T10:59:53.660 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.122286966, 2024-09-21T10:59:53.660 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:59:53.660 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:59:53.660 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:59:53.660 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:59:53.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.660 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:59:53.660 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.307538+0000", 2024-09-21T10:59:53.660 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.661 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.661 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.661 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:59:53.661 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.307538+0000", 2024-09-21T10:59:53.661 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.661 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.661 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.661 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:59:53.661 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.307538+0000", 2024-09-21T10:59:53.661 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8700000000000001e-07 2024-09-21T10:59:53.661 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.661 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.661 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:59:53.661 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.307542+0000", 2024-09-21T10:59:53.661 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0149999999999997e-06 2024-09-21T10:59:53.662 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.662 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.662 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:59:53.662 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.307550+0000", 2024-09-21T10:59:53.662 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4889999999999998e-06 2024-09-21T10:59:53.662 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.662 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.662 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:59:53.662 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.307555+0000", 2024-09-21T10:59:53.662 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4909999999999996e-06 2024-09-21T10:59:53.662 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.662 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.662 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:59:53.662 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.314879+0000", 2024-09-21T10:59:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0073239250000000002 2024-09-21T10:59:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:59:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.314904+0000", 2024-09-21T10:59:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4516999999999999e-05 2024-09-21T10:59:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:59:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.429774+0000", 2024-09-21T10:59:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.114870376 2024-09-21T10:59:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:59:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.429825+0000", 2024-09-21T10:59:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0766000000000001e-05 2024-09-21T10:59:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:59:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:195894 55.c e1474/1465 55:3dc3da2f:::benchmark_data_smithi070_294394_object90553:head v 1474'12199, mlcod=0'0)", 2024-09-21T10:59:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:53:26.309020+0000", 2024-09-21T10:59:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 386.31949168, 2024-09-21T10:59:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.146994024, 2024-09-21T10:59:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T10:59:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T10:59:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:59:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T10:59:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T10:59:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.309020+0000", 2024-09-21T10:59:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T10:59:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.309020+0000", 2024-09-21T10:59:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T10:59:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T10:59:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.309021+0000", 2024-09-21T10:59:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9199999999999999e-07 2024-09-21T10:59:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T10:59:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.309028+0000", 2024-09-21T10:59:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.844e-06 2024-09-21T10:59:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T10:59:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.309040+0000", 2024-09-21T10:59:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2e-05 2024-09-21T10:59:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T10:59:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.309048+0000", 2024-09-21T10:59:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9619999999999993e-06 2024-09-21T10:59:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T10:59:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.443698+0000", 2024-09-21T10:59:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13465039300000001 2024-09-21T10:59:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T10:59:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.443727+0000", 2024-09-21T10:59:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9249000000000001e-05 2024-09-21T10:59:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T10:59:53.668 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.455970+0000", 2024-09-21T10:59:53.668 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012242602999999999 2024-09-21T10:59:53.668 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T10:59:53.668 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T10:59:53.668 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T10:59:53.668 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.456014+0000", 2024-09-21T10:59:53.668 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3980999999999998e-05 2024-09-21T10:59:53.668 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.668 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:59:53.668 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.668 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T10:59:53.668 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T10:59:53.668 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T10:59:53.669 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:53.691 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:53.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:53.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:53.793 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:53.816 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:53.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:53.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:53 smithi160 ceph-mon[101879]: pgmap v2983: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 4.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 102 B/s rd, 21 MiB/s wr, 371 op/s; 37642/201261 objects degraded (18.703%); 11932/201261 objects misplaced (5.929%); 429 KiB/s, 6 objects/s recovering 2024-09-21T10:59:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:53 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1639295509' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:59:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:53 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 37642/201261 objects degraded (18.703%), 22 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-09-21T10:59:53.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:53 smithi070 ceph-mon[107385]: pgmap v2983: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 4.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 102 B/s rd, 21 MiB/s wr, 371 op/s; 37642/201261 objects degraded (18.703%); 11932/201261 objects misplaced (5.929%); 429 KiB/s, 6 objects/s recovering 2024-09-21T10:59:53.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:53 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1639295509' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:59:53.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:53 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 37642/201261 objects degraded (18.703%), 22 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-09-21T10:59:53.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:53 smithi070 ceph-mon[112139]: pgmap v2983: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 4.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 102 B/s rd, 21 MiB/s wr, 371 op/s; 37642/201261 objects degraded (18.703%); 11932/201261 objects misplaced (5.929%); 429 KiB/s, 6 objects/s recovering 2024-09-21T10:59:53.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:53 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1639295509' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:59:53.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:53 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 37642/201261 objects degraded (18.703%), 22 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-09-21T10:59:53.918 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:53.939 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:53.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:53.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:54.041 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:54.068 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:54.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:54.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:54.170 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:54.191 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:54.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:54.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:54.293 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:54.315 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:54.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:54.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:54.329 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:59:54.417 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:54.439 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:54.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:54.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:54.540 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:54.565 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:54.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:54.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:54.667 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:54.691 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:54.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:54.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:54.793 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:54.815 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:54.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:54.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:54.917 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:54.940 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:54.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:54.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:55.041 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:55.064 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:55.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:55.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:55.166 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:55.187 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:55.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:55.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:55.289 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:55.309 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:55.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:55.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:55.411 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:55.437 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:55.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:55.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:55.539 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:55.557 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 10:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=cleanup t=2024-09-21T10:59:55.130579274Z level=info msg="Completed cleanup jobs" duration=14.195814ms 2024-09-21T10:59:55.557 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 10:59:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=plugins.update.checker t=2024-09-21T10:59:55.26362114Z level=info msg="Update check succeeded" duration=57.909237ms 2024-09-21T10:59:55.565 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:55.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:55.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:55.567 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T10:59:55.668 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:55.689 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:55.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:55.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:55.704 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:55 smithi070 ceph-mon[107385]: pgmap v2984: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 4.1 GiB data, 18 GiB used, 430 GiB / 447 GiB avail; 94 B/s rd, 15 MiB/s wr, 326 op/s; 37414/199185 objects degraded (18.784%); 11771/199185 objects misplaced (5.910%); 782 KiB/s, 12 objects/s recovering 2024-09-21T10:59:55.704 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:55 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4254493379' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:59:55.704 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:55 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:59:55.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:55 smithi070 ceph-mon[112139]: pgmap v2984: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 4.1 GiB data, 18 GiB used, 430 GiB / 447 GiB avail; 94 B/s rd, 15 MiB/s wr, 326 op/s; 37414/199185 objects degraded (18.784%); 11771/199185 objects misplaced (5.910%); 782 KiB/s, 12 objects/s recovering 2024-09-21T10:59:55.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:55 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4254493379' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:59:55.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:55 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:59:55.791 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:55.814 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:55.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:55.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:55.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:55 smithi160 ceph-mon[101879]: pgmap v2984: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 4.1 GiB data, 18 GiB used, 430 GiB / 447 GiB avail; 94 B/s rd, 15 MiB/s wr, 326 op/s; 37414/199185 objects degraded (18.784%); 11771/199185 objects misplaced (5.910%); 782 KiB/s, 12 objects/s recovering 2024-09-21T10:59:55.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:55 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4254493379' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:59:55.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:55 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:59:55.916 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:55.941 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:55.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:55.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:56.042 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:56.064 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:56.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:56.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:56.165 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:56.187 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:56.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:56.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:56.289 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:56.320 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:56.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:56.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:56.422 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:56.447 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:56.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:56.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:56.548 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:56.572 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:56.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:56.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:56.673 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:56.695 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:56.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:56.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:56.796 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:56.817 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:56.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:56.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:56 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:59:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:56 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:59:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:56 smithi160 ceph-mon[101879]: osdmap e1554: 8 total, 6 up, 5 in 2024-09-21T10:59:56.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:56 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:59:56.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:56 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:59:56.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:56 smithi070 ceph-mon[112139]: osdmap e1554: 8 total, 6 up, 5 in 2024-09-21T10:59:56.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:56 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:59:56.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:56 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:59:56.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:56 smithi070 ceph-mon[107385]: osdmap e1554: 8 total, 6 up, 5 in 2024-09-21T10:59:56.919 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:56.940 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:56.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:56.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:57.041 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:57.064 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:57.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:57.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:57.165 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:57.184 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:59:57.184 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:59:57.187 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:57.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:57.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:57.289 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:57.310 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:57.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:57.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:57.412 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:57.436 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:57.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:57.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:57.537 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:57.559 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:57.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:57.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:57.660 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:57.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:57.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:57.784 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:57.805 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:57.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:57.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:57 smithi160 ceph-mon[101879]: pgmap v2986: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 4.0 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 102 B/s rd, 7.8 MiB/s wr, 279 op/s; 37136/197466 objects degraded (18.806%); 11590/197466 objects misplaced (5.869%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:59:57.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:57 smithi070 ceph-mon[107385]: pgmap v2986: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 4.0 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 102 B/s rd, 7.8 MiB/s wr, 279 op/s; 37136/197466 objects degraded (18.806%); 11590/197466 objects misplaced (5.869%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:59:57.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:57 smithi070 ceph-mon[112139]: pgmap v2986: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 4.0 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 102 B/s rd, 7.8 MiB/s wr, 279 op/s; 37136/197466 objects degraded (18.806%); 11590/197466 objects misplaced (5.869%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:59:57.907 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:57.929 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:57.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:57.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:57.956 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:59:58.031 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:58.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:58.052 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:58.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:58.154 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:59:58.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:58.175 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:58.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:59:58.276 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:58.299 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:58.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:58.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:58.400 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:58.425 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:58.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:58.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:58.527 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:58.549 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:58.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:58.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:58.651 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:58.671 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T10:59:58.677 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:58.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:58.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:58.779 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:58.801 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:58.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:58.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:58 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 37136/197466 objects degraded (18.806%), 22 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-09-21T10:59:58.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:58 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 37136/197466 objects degraded (18.806%), 22 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-09-21T10:59:58.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:58 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 37136/197466 objects degraded (18.806%), 22 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-09-21T10:59:58.902 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:58.923 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:58.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:58.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:59.025 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:59:59.046 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:59.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:59.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:59:59.148 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:59.169 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:59.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:59.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:59.185 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:59:59.271 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:59.293 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:59.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:59.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:59.352 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:59:59.352 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:59:59.352 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-21T10:59:59.394 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:59.423 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:59.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:59.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:59.525 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:59:59.551 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:59.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:59.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:59:59.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 10:59:59 smithi070 ceph-mon[107385]: pgmap v2987: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 4.0 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 102 B/s rd, 3.4 MiB/s wr, 255 op/s; 36983/196083 objects degraded (18.861%); 11436/196083 objects misplaced (5.832%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T10:59:59.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 10:59:59 smithi070 ceph-mon[112139]: pgmap v2987: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 4.0 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 102 B/s rd, 3.4 MiB/s wr, 255 op/s; 36983/196083 objects degraded (18.861%); 11436/196083 objects misplaced (5.832%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T10:59:59.652 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:59:59.678 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T10:59:59.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:59.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:59:59.780 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:59:59.803 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:59.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:59.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:59:59.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 10:59:59 smithi160 ceph-mon[101879]: pgmap v2987: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 4.0 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 102 B/s rd, 3.4 MiB/s wr, 255 op/s; 36983/196083 objects degraded (18.861%); 11436/196083 objects misplaced (5.832%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T10:59:59.905 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:59:59.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:59:59.928 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T10:59:59.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:59:59.974 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T11:00:00.029 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:00.050 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:00.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:00.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:00.152 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:00.174 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:00.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:00.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:00.275 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:00.304 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:00.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:00.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:00.359 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:00:00.406 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:00.407 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:00:00.431 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:00.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:00.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:00.533 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:00.556 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:00.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:00.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:00.657 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:00.679 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:00.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:00.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:00.780 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:00.802 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:00.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:00.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 36983/196083 objects degraded (18.861%), 22 pgs degraded, 56 pgs undersized 2024-09-21T11:00:00.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-21T11:00:00.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: daemon osd.5 on smithi160 is in unknown state 2024-09-21T11:00:00.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-21T11:00:00.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: [WRN] OSD_DOWN: 1 osds down 2024-09-21T11:00:00.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: osd.6 (root=default,host=smithi160) is down 2024-09-21T11:00:00.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: [WRN] PG_DEGRADED: Degraded data redundancy: 36983/196083 objects degraded (18.861%), 22 pgs degraded, 56 pgs undersized 2024-09-21T11:00:00.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 1.0 is stuck undersized for 73s, current state active+undersized, last acting [7,1] 2024-09-21T11:00:00.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 1.4 is stuck undersized for 73s, current state active+undersized, last acting [7,4] 2024-09-21T11:00:00.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 1.5 is stuck undersized for 73s, current state active+undersized, last acting [4,7] 2024-09-21T11:00:00.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 1.7 is stuck undersized for 73s, current state active+undersized, last acting [7,1] 2024-09-21T11:00:00.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 1.9 is stuck undersized for 73s, current state active+undersized, last acting [7,3] 2024-09-21T11:00:00.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 1.a is stuck undersized for 73s, current state active+undersized, last acting [4,1] 2024-09-21T11:00:00.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 2.1 is stuck undersized for 73s, current state active+undersized, last acting [1,4] 2024-09-21T11:00:00.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 2.2 is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,7] 2024-09-21T11:00:00.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 2.3 is stuck undersized for 73s, current state active+undersized+degraded, last acting [7,3] 2024-09-21T11:00:00.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 2.6 is stuck undersized for 73s, current state active+undersized, last acting [1,4] 2024-09-21T11:00:00.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 2.7 is stuck undersized for 73s, current state active+undersized, last acting [7,1] 2024-09-21T11:00:00.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 3.0 is stuck undersized for 73s, current state active+undersized, last acting [1,7] 2024-09-21T11:00:00.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 3.2 is stuck undersized for 73s, current state active+undersized, last acting [4,1] 2024-09-21T11:00:00.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 3.3 is stuck undersized for 73s, current state active+undersized, last acting [4,1] 2024-09-21T11:00:00.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 3.4 is stuck undersized for 73s, current state active+undersized, last acting [1,4] 2024-09-21T11:00:00.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 3.5 is stuck undersized for 73s, current state active+undersized, last acting [7,1] 2024-09-21T11:00:00.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 3.a is stuck undersized for 73s, current state active+undersized, last acting [4,1] 2024-09-21T11:00:00.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 3.14 is stuck undersized for 73s, current state active+undersized+degraded, last acting [4,7] 2024-09-21T11:00:00.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 3.17 is stuck undersized for 73s, current state active+undersized, last acting [1,4] 2024-09-21T11:00:00.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 3.1c is stuck undersized for 73s, current state active+undersized+degraded, last acting [4,1] 2024-09-21T11:00:00.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 3.1d is stuck undersized for 73s, current state active+undersized+degraded, last acting [4,1] 2024-09-21T11:00:00.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 3.1e is stuck undersized for 73s, current state active+undersized, last acting [1,7] 2024-09-21T11:00:00.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 3.1f is stuck undersized for 73s, current state active+undersized+degraded, last acting [4,7] 2024-09-21T11:00:00.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 4.0 is stuck undersized for 73s, current state active+undersized, last acting [7,1] 2024-09-21T11:00:00.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 4.1 is stuck undersized for 73s, current state active+undersized, last acting [4,7] 2024-09-21T11:00:00.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 4.5 is stuck undersized for 73s, current state active+undersized, last acting [4,7] 2024-09-21T11:00:00.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 4.c is stuck undersized for 73s, current state active+undersized, last acting [4,3] 2024-09-21T11:00:00.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 4.d is stuck undersized for 73s, current state active+undersized, last acting [4,1] 2024-09-21T11:00:00.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 4.e is stuck undersized for 73s, current state active+undersized, last acting [4,1] 2024-09-21T11:00:00.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 4.10 is stuck undersized for 73s, current state active+undersized, last acting [1,4] 2024-09-21T11:00:00.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 4.11 is stuck undersized for 73s, current state active+undersized, last acting [3,7] 2024-09-21T11:00:00.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 4.12 is stuck undersized for 73s, current state active+undersized, last acting [1,7] 2024-09-21T11:00:00.845 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 4.13 is stuck undersized for 73s, current state active+undersized, last acting [4,1] 2024-09-21T11:00:00.845 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 4.18 is stuck undersized for 73s, current state active+undersized, last acting [4,1] 2024-09-21T11:00:00.845 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 4.19 is stuck undersized for 73s, current state active+undersized, last acting [1,4] 2024-09-21T11:00:00.845 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 4.1a is stuck undersized for 73s, current state active+undersized, last acting [4,3] 2024-09-21T11:00:00.845 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 4.1d is stuck undersized for 73s, current state active+undersized, last acting [1,4] 2024-09-21T11:00:00.845 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 4.1f is stuck undersized for 73s, current state active+undersized, last acting [1,7] 2024-09-21T11:00:00.845 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 57.0 is stuck undersized for 95s, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,7] 2024-09-21T11:00:00.845 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 57.1 is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,7] 2024-09-21T11:00:00.845 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 57.3 is stuck undersized for 73s, current state active+undersized+degraded, last acting [3,7] 2024-09-21T11:00:00.845 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 57.5 is active+recovery_wait+undersized+degraded+remapped, acting [7,1] 2024-09-21T11:00:00.845 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 57.6 is active+recovery_wait+degraded, acting [7,4,1] 2024-09-21T11:00:00.845 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 57.7 is active+recovery_wait+undersized+degraded+remapped, acting [7,1] 2024-09-21T11:00:00.845 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 57.8 is active+recovery_wait+degraded+remapped, acting [7,4,1] 2024-09-21T11:00:00.846 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 57.9 is stuck undersized for 73s, current state active+undersized+degraded, last acting [4,1] 2024-09-21T11:00:00.846 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 57.b is active+recovery_wait+degraded+remapped, acting [4,1,7] 2024-09-21T11:00:00.846 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 57.c is active+recovery_wait+degraded+remapped, acting [4,7,1] 2024-09-21T11:00:00.846 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 57.d is active+recovering+undersized+degraded+remapped, acting [4,1] 2024-09-21T11:00:00.846 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 57.e is active+recovery_wait+undersized+degraded+remapped, acting [1,7] 2024-09-21T11:00:00.846 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: pg 57.f is stuck undersized for 73s, current state active+undersized+degraded, last acting [7,1] 2024-09-21T11:00:00.846 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:00 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T11:00:00.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 36983/196083 objects degraded (18.861%), 22 pgs degraded, 56 pgs undersized 2024-09-21T11:00:00.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-21T11:00:00.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: daemon osd.5 on smithi160 is in unknown state 2024-09-21T11:00:00.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-21T11:00:00.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: [WRN] OSD_DOWN: 1 osds down 2024-09-21T11:00:00.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: osd.6 (root=default,host=smithi160) is down 2024-09-21T11:00:00.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: [WRN] PG_DEGRADED: Degraded data redundancy: 36983/196083 objects degraded (18.861%), 22 pgs degraded, 56 pgs undersized 2024-09-21T11:00:00.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 1.0 is stuck undersized for 73s, current state active+undersized, last acting [7,1] 2024-09-21T11:00:00.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 1.4 is stuck undersized for 73s, current state active+undersized, last acting [7,4] 2024-09-21T11:00:00.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 1.5 is stuck undersized for 73s, current state active+undersized, last acting [4,7] 2024-09-21T11:00:00.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 1.7 is stuck undersized for 73s, current state active+undersized, last acting [7,1] 2024-09-21T11:00:00.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 1.9 is stuck undersized for 73s, current state active+undersized, last acting [7,3] 2024-09-21T11:00:00.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 1.a is stuck undersized for 73s, current state active+undersized, last acting [4,1] 2024-09-21T11:00:00.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 2.1 is stuck undersized for 73s, current state active+undersized, last acting [1,4] 2024-09-21T11:00:00.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 2.2 is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,7] 2024-09-21T11:00:00.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 2.3 is stuck undersized for 73s, current state active+undersized+degraded, last acting [7,3] 2024-09-21T11:00:00.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 2.6 is stuck undersized for 73s, current state active+undersized, last acting [1,4] 2024-09-21T11:00:00.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 2.7 is stuck undersized for 73s, current state active+undersized, last acting [7,1] 2024-09-21T11:00:00.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 3.0 is stuck undersized for 73s, current state active+undersized, last acting [1,7] 2024-09-21T11:00:00.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 3.2 is stuck undersized for 73s, current state active+undersized, last acting [4,1] 2024-09-21T11:00:00.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 3.3 is stuck undersized for 73s, current state active+undersized, last acting [4,1] 2024-09-21T11:00:00.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 3.4 is stuck undersized for 73s, current state active+undersized, last acting [1,4] 2024-09-21T11:00:00.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 3.5 is stuck undersized for 73s, current state active+undersized, last acting [7,1] 2024-09-21T11:00:00.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 3.a is stuck undersized for 73s, current state active+undersized, last acting [4,1] 2024-09-21T11:00:00.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 3.14 is stuck undersized for 73s, current state active+undersized+degraded, last acting [4,7] 2024-09-21T11:00:00.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 3.17 is stuck undersized for 73s, current state active+undersized, last acting [1,4] 2024-09-21T11:00:00.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 3.1c is stuck undersized for 73s, current state active+undersized+degraded, last acting [4,1] 2024-09-21T11:00:00.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 3.1d is stuck undersized for 73s, current state active+undersized+degraded, last acting [4,1] 2024-09-21T11:00:00.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 3.1e is stuck undersized for 73s, current state active+undersized, last acting [1,7] 2024-09-21T11:00:00.880 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 3.1f is stuck undersized for 73s, current state active+undersized+degraded, last acting [4,7] 2024-09-21T11:00:00.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 36983/196083 objects degraded (18.861%), 22 pgs degraded, 56 pgs undersized 2024-09-21T11:00:00.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-21T11:00:00.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: daemon osd.5 on smithi160 is in unknown state 2024-09-21T11:00:00.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-21T11:00:00.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: [WRN] OSD_DOWN: 1 osds down 2024-09-21T11:00:00.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: osd.6 (root=default,host=smithi160) is down 2024-09-21T11:00:00.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: [WRN] PG_DEGRADED: Degraded data redundancy: 36983/196083 objects degraded (18.861%), 22 pgs degraded, 56 pgs undersized 2024-09-21T11:00:00.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 1.0 is stuck undersized for 73s, current state active+undersized, last acting [7,1] 2024-09-21T11:00:00.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 1.4 is stuck undersized for 73s, current state active+undersized, last acting [7,4] 2024-09-21T11:00:00.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 1.5 is stuck undersized for 73s, current state active+undersized, last acting [4,7] 2024-09-21T11:00:00.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 1.7 is stuck undersized for 73s, current state active+undersized, last acting [7,1] 2024-09-21T11:00:00.881 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 1.9 is stuck undersized for 73s, current state active+undersized, last acting [7,3] 2024-09-21T11:00:00.882 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 1.a is stuck undersized for 73s, current state active+undersized, last acting [4,1] 2024-09-21T11:00:00.882 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 2.1 is stuck undersized for 73s, current state active+undersized, last acting [1,4] 2024-09-21T11:00:00.882 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 2.2 is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,7] 2024-09-21T11:00:00.882 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 2.3 is stuck undersized for 73s, current state active+undersized+degraded, last acting [7,3] 2024-09-21T11:00:00.882 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 2.6 is stuck undersized for 73s, current state active+undersized, last acting [1,4] 2024-09-21T11:00:00.882 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 2.7 is stuck undersized for 73s, current state active+undersized, last acting [7,1] 2024-09-21T11:00:00.882 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 3.0 is stuck undersized for 73s, current state active+undersized, last acting [1,7] 2024-09-21T11:00:00.882 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 3.2 is stuck undersized for 73s, current state active+undersized, last acting [4,1] 2024-09-21T11:00:00.882 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 3.3 is stuck undersized for 73s, current state active+undersized, last acting [4,1] 2024-09-21T11:00:00.882 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 3.4 is stuck undersized for 73s, current state active+undersized, last acting [1,4] 2024-09-21T11:00:00.882 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 3.5 is stuck undersized for 73s, current state active+undersized, last acting [7,1] 2024-09-21T11:00:00.882 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 3.a is stuck undersized for 73s, current state active+undersized, last acting [4,1] 2024-09-21T11:00:00.883 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 3.14 is stuck undersized for 73s, current state active+undersized+degraded, last acting [4,7] 2024-09-21T11:00:00.883 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 3.17 is stuck undersized for 73s, current state active+undersized, last acting [1,4] 2024-09-21T11:00:00.883 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 3.1c is stuck undersized for 73s, current state active+undersized+degraded, last acting [4,1] 2024-09-21T11:00:00.883 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 3.1d is stuck undersized for 73s, current state active+undersized+degraded, last acting [4,1] 2024-09-21T11:00:00.883 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 3.1e is stuck undersized for 73s, current state active+undersized, last acting [1,7] 2024-09-21T11:00:00.883 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 3.1f is stuck undersized for 73s, current state active+undersized+degraded, last acting [4,7] 2024-09-21T11:00:00.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 4.0 is stuck undersized for 73s, current state active+undersized, last acting [7,1] 2024-09-21T11:00:00.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 4.1 is stuck undersized for 73s, current state active+undersized, last acting [4,7] 2024-09-21T11:00:00.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 4.5 is stuck undersized for 73s, current state active+undersized, last acting [4,7] 2024-09-21T11:00:00.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 4.c is stuck undersized for 73s, current state active+undersized, last acting [4,3] 2024-09-21T11:00:00.883 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 4.d is stuck undersized for 73s, current state active+undersized, last acting [4,1] 2024-09-21T11:00:00.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 4.e is stuck undersized for 73s, current state active+undersized, last acting [4,1] 2024-09-21T11:00:00.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 4.10 is stuck undersized for 73s, current state active+undersized, last acting [1,4] 2024-09-21T11:00:00.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 4.11 is stuck undersized for 73s, current state active+undersized, last acting [3,7] 2024-09-21T11:00:00.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 4.12 is stuck undersized for 73s, current state active+undersized, last acting [1,7] 2024-09-21T11:00:00.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 4.13 is stuck undersized for 73s, current state active+undersized, last acting [4,1] 2024-09-21T11:00:00.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 4.18 is stuck undersized for 73s, current state active+undersized, last acting [4,1] 2024-09-21T11:00:00.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 4.19 is stuck undersized for 73s, current state active+undersized, last acting [1,4] 2024-09-21T11:00:00.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 4.1a is stuck undersized for 73s, current state active+undersized, last acting [4,3] 2024-09-21T11:00:00.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 4.1d is stuck undersized for 73s, current state active+undersized, last acting [1,4] 2024-09-21T11:00:00.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 4.1f is stuck undersized for 73s, current state active+undersized, last acting [1,7] 2024-09-21T11:00:00.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 57.0 is stuck undersized for 95s, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,7] 2024-09-21T11:00:00.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 57.1 is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,7] 2024-09-21T11:00:00.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 57.3 is stuck undersized for 73s, current state active+undersized+degraded, last acting [3,7] 2024-09-21T11:00:00.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 57.5 is active+recovery_wait+undersized+degraded+remapped, acting [7,1] 2024-09-21T11:00:00.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 57.6 is active+recovery_wait+degraded, acting [7,4,1] 2024-09-21T11:00:00.885 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 57.7 is active+recovery_wait+undersized+degraded+remapped, acting [7,1] 2024-09-21T11:00:00.885 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 57.8 is active+recovery_wait+degraded+remapped, acting [7,4,1] 2024-09-21T11:00:00.885 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 57.9 is stuck undersized for 73s, current state active+undersized+degraded, last acting [4,1] 2024-09-21T11:00:00.885 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 57.b is active+recovery_wait+degraded+remapped, acting [4,1,7] 2024-09-21T11:00:00.885 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 57.c is active+recovery_wait+degraded+remapped, acting [4,7,1] 2024-09-21T11:00:00.885 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 57.d is active+recovering+undersized+degraded+remapped, acting [4,1] 2024-09-21T11:00:00.885 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 57.e is active+recovery_wait+undersized+degraded+remapped, acting [1,7] 2024-09-21T11:00:00.885 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: pg 57.f is stuck undersized for 73s, current state active+undersized+degraded, last acting [7,1] 2024-09-21T11:00:00.885 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T11:00:00.885 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 4.0 is stuck undersized for 73s, current state active+undersized, last acting [7,1] 2024-09-21T11:00:00.885 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 4.1 is stuck undersized for 73s, current state active+undersized, last acting [4,7] 2024-09-21T11:00:00.885 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 4.5 is stuck undersized for 73s, current state active+undersized, last acting [4,7] 2024-09-21T11:00:00.885 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 4.c is stuck undersized for 73s, current state active+undersized, last acting [4,3] 2024-09-21T11:00:00.886 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 4.d is stuck undersized for 73s, current state active+undersized, last acting [4,1] 2024-09-21T11:00:00.886 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 4.e is stuck undersized for 73s, current state active+undersized, last acting [4,1] 2024-09-21T11:00:00.886 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 4.10 is stuck undersized for 73s, current state active+undersized, last acting [1,4] 2024-09-21T11:00:00.886 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 4.11 is stuck undersized for 73s, current state active+undersized, last acting [3,7] 2024-09-21T11:00:00.886 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 4.12 is stuck undersized for 73s, current state active+undersized, last acting [1,7] 2024-09-21T11:00:00.886 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 4.13 is stuck undersized for 73s, current state active+undersized, last acting [4,1] 2024-09-21T11:00:00.886 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 4.18 is stuck undersized for 73s, current state active+undersized, last acting [4,1] 2024-09-21T11:00:00.886 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 4.19 is stuck undersized for 73s, current state active+undersized, last acting [1,4] 2024-09-21T11:00:00.886 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 4.1a is stuck undersized for 73s, current state active+undersized, last acting [4,3] 2024-09-21T11:00:00.886 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 4.1d is stuck undersized for 73s, current state active+undersized, last acting [1,4] 2024-09-21T11:00:00.886 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 4.1f is stuck undersized for 73s, current state active+undersized, last acting [1,7] 2024-09-21T11:00:00.886 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 57.0 is stuck undersized for 95s, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,7] 2024-09-21T11:00:00.886 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 57.1 is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,7] 2024-09-21T11:00:00.886 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 57.3 is stuck undersized for 73s, current state active+undersized+degraded, last acting [3,7] 2024-09-21T11:00:00.887 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 57.5 is active+recovery_wait+undersized+degraded+remapped, acting [7,1] 2024-09-21T11:00:00.887 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 57.6 is active+recovery_wait+degraded, acting [7,4,1] 2024-09-21T11:00:00.887 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 57.7 is active+recovery_wait+undersized+degraded+remapped, acting [7,1] 2024-09-21T11:00:00.887 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 57.8 is active+recovery_wait+degraded+remapped, acting [7,4,1] 2024-09-21T11:00:00.887 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 57.9 is stuck undersized for 73s, current state active+undersized+degraded, last acting [4,1] 2024-09-21T11:00:00.887 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 57.b is active+recovery_wait+degraded+remapped, acting [4,1,7] 2024-09-21T11:00:00.887 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 57.c is active+recovery_wait+degraded+remapped, acting [4,7,1] 2024-09-21T11:00:00.887 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 57.d is active+recovering+undersized+degraded+remapped, acting [4,1] 2024-09-21T11:00:00.887 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 57.e is active+recovery_wait+undersized+degraded+remapped, acting [1,7] 2024-09-21T11:00:00.887 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: pg 57.f is stuck undersized for 73s, current state active+undersized+degraded, last acting [7,1] 2024-09-21T11:00:00.887 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:00 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T11:00:00.905 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:00.933 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:00.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:00.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:01.035 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:01.056 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:01.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:01.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:01.158 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:01.181 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:01.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:01.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:01.282 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:01.305 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:01.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:01.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:01.406 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:01.430 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:01.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:01.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:01.532 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:01.553 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:01.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:01.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:01.655 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:01.677 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:01.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:01.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:01.680 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:00:01.779 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:01.800 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:01.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:01.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:01 smithi160 ceph-mon[101879]: pgmap v2988: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 247 op/s; 36596/193848 objects degraded (18.879%); 11261/193848 objects misplaced (5.809%); 2.6 MiB/s, 42 objects/s recovering 2024-09-21T11:00:01.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:01 smithi070 ceph-mon[107385]: pgmap v2988: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 247 op/s; 36596/193848 objects degraded (18.879%); 11261/193848 objects misplaced (5.809%); 2.6 MiB/s, 42 objects/s recovering 2024-09-21T11:00:01.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:01 smithi070 ceph-mon[112139]: pgmap v2988: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 247 op/s; 36596/193848 objects degraded (18.879%); 11261/193848 objects misplaced (5.809%); 2.6 MiB/s, 42 objects/s recovering 2024-09-21T11:00:01.901 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:01.932 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:01.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:01.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:02.034 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:02.058 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:02.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:02.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:02.160 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:02.186 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:02.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:02.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:02.288 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:02.315 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:02.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:02.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:02.328 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T11:00:02.328 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T11:00:02.417 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:02.439 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:02.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:02.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:02.540 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:02.564 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:02.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:02.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:02.665 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:02.687 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:02.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:02.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:02.788 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:02.815 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:02.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:02.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:02.917 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:02.945 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:02.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:02.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:03.047 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:03.068 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:03.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:03.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:03.170 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:03.192 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:03.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:03.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:03.293 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:03.317 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:03.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:03.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:03.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:00:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:00:03] "GET /metrics HTTP/1.1" 200 36576 "" "Prometheus/2.51.0" 2024-09-21T11:00:03.418 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:03.439 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:03.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:03.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:03.541 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:03.562 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:03.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:03.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:03.663 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:03.689 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:03.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:03.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:03.790 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:03.812 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:03.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:03.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:03.829 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:03 smithi070 ceph-mon[107385]: pgmap v2989: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 247 op/s; 36596/193848 objects degraded (18.879%); 11261/193848 objects misplaced (5.809%); 2.6 MiB/s, 42 objects/s recovering 2024-09-21T11:00:03.830 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:03 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 36596/193848 objects degraded (18.879%), 22 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:03.830 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:03 smithi070 ceph-mon[112139]: pgmap v2989: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 247 op/s; 36596/193848 objects degraded (18.879%); 11261/193848 objects misplaced (5.809%); 2.6 MiB/s, 42 objects/s recovering 2024-09-21T11:00:03.830 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:03 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 36596/193848 objects degraded (18.879%), 22 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:03 smithi160 ceph-mon[101879]: pgmap v2989: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 247 op/s; 36596/193848 objects degraded (18.879%); 11261/193848 objects misplaced (5.809%); 2.6 MiB/s, 42 objects/s recovering 2024-09-21T11:00:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:03 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 36596/193848 objects degraded (18.879%), 22 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:03.914 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:03.939 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:03.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:03.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:04.040 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:04.061 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:04.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:04.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:04.162 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:04.187 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:04.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:04.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:04.288 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:04.309 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:04.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:04.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:04.410 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:04.434 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:04.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:04.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:04.535 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:04.556 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:04.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:04.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:04.658 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:04.679 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:04.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:04.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:04.707 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T11:00:04.781 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:04.804 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:04.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:04.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:04.905 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:04.933 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:04.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:04.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:05.034 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:05.055 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:05.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:05.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:05.157 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:05.181 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:05.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:05.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:05.283 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:05.305 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:05.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:05.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:05.407 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:05.428 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:05.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:05.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:05.529 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:05.554 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:05.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:05.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:05.656 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:05.677 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:05.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:05.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:05.778 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:05.802 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:05.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:05.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:05.841 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:05 smithi070 ceph-mon[107385]: pgmap v2990: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 247 op/s; 36369/191757 objects degraded (18.966%); 11109/191757 objects misplaced (5.793%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T11:00:05.841 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:00:05.841 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:05 smithi070 ceph-mon[112139]: pgmap v2990: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 247 op/s; 36369/191757 objects degraded (18.966%); 11109/191757 objects misplaced (5.793%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T11:00:05.841 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:00:05.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:05 smithi160 ceph-mon[101879]: pgmap v2990: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 247 op/s; 36369/191757 objects degraded (18.966%); 11109/191757 objects misplaced (5.793%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T11:00:05.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:00:05.904 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:05.927 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:05.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:05.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:06.029 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:06.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:06.059 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:06.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:06.159 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:06.184 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:06.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:06.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:06.286 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:06.310 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:06.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:06.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:06.412 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:06.442 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:06.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:06.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:06.539 INFO:teuthology.orchestra.run.smithi070.stderr:no valid command found; 10 closest matches: 2024-09-21T11:00:06.539 INFO:teuthology.orchestra.run.smithi070.stderr:0 2024-09-21T11:00:06.539 INFO:teuthology.orchestra.run.smithi070.stderr:1 2024-09-21T11:00:06.539 INFO:teuthology.orchestra.run.smithi070.stderr:2 2024-09-21T11:00:06.539 INFO:teuthology.orchestra.run.smithi070.stderr:abort 2024-09-21T11:00:06.539 INFO:teuthology.orchestra.run.smithi070.stderr:assert 2024-09-21T11:00:06.540 INFO:teuthology.orchestra.run.smithi070.stderr:bench [] [] [] [] 2024-09-21T11:00:06.540 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs debug_inject_read_zeros 2024-09-21T11:00:06.540 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs files list 2024-09-21T11:00:06.540 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs stats 2024-09-21T11:00:06.540 INFO:teuthology.orchestra.run.smithi070.stderr:bluestore allocator dump block 2024-09-21T11:00:06.540 INFO:teuthology.orchestra.run.smithi070.stderr:Error EINVAL: invalid command 2024-09-21T11:00:06.544 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:06.569 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:06.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:06.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:06.671 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:06.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:06.693 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:06.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:06.793 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:06.816 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:06.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:06.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:06.918 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:06.939 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:06.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:06.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:07.040 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:07.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:07.070 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:07.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:07.141 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T11:00:07.171 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:07.195 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:07.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:07.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:07.260 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:00:07.260 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:00:07.260 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:00:07.261 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:00:07.272 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:00:07.272 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:00:07.292 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:00:07.292 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:00:07.297 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:07.314 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:00:07.314 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:00:07.314 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T11:00:07.314 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T11:00:07.322 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:07.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:07.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:07.332 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:00:07.332 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:00:07.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:00:07.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:00:07.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:07 smithi070 ceph-mon[107385]: pgmap v2991: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 228 op/s; 36046/190029 objects degraded (18.969%); 10943/190029 objects misplaced (5.759%); 2.6 MiB/s, 42 objects/s recovering 2024-09-21T11:00:07.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:07 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3425496360' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:00:07.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:07 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:00:07.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:00:07.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:00:07.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:07 smithi070 ceph-mon[112139]: pgmap v2991: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 228 op/s; 36046/190029 objects degraded (18.969%); 10943/190029 objects misplaced (5.759%); 2.6 MiB/s, 42 objects/s recovering 2024-09-21T11:00:07.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:07 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3425496360' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:00:07.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:07 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:00:07.424 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:07.449 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:07.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:07.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:07.550 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:07.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:07.574 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:07.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:07.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:00:07.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:00:07.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:07 smithi160 ceph-mon[101879]: pgmap v2991: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 228 op/s; 36046/190029 objects degraded (18.969%); 10943/190029 objects misplaced (5.759%); 2.6 MiB/s, 42 objects/s recovering 2024-09-21T11:00:07.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:07 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3425496360' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:00:07.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:07 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:00:07.674 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:07.696 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:07.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:07.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:07.797 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:07.818 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:07.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:07.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:07.852 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T11:00:07.853 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T11:00:07.853 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.4 'trim stale osdmaps' 2024-09-21T11:00:07.920 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:07.959 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:07.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:07.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:08.060 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:08.083 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:08.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:08.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:08.185 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:08.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:08.207 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:08.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:08.308 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:08.329 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:08.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:08.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:08.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:08 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T11:00:08.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:08 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T11:00:08.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:08 smithi070 ceph-mon[112139]: osdmap e1555: 8 total, 6 up, 5 in 2024-09-21T11:00:08.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:08 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3466784485' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:00:08.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:08 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T11:00:08.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:08 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T11:00:08.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:08 smithi070 ceph-mon[107385]: osdmap e1555: 8 total, 6 up, 5 in 2024-09-21T11:00:08.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:08 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3466784485' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:00:08.431 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:08.452 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:08.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:08.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:08.553 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:08.575 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:08.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:08.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:08 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T11:00:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:08 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T11:00:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:08 smithi160 ceph-mon[101879]: osdmap e1555: 8 total, 6 up, 5 in 2024-09-21T11:00:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:08 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3466784485' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:00:08.676 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:08.699 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:08.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:08.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:08.801 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:08.821 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:08.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:08.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:08.923 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:08.946 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:08.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:08.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:09.048 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:09.069 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:09.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:09.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:09.170 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:09.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:09.193 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:09.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:09.293 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:09.314 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:09.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:09.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:09.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:09 smithi070 ceph-mon[107385]: pgmap v2993: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 250 op/s; 35899/188556 objects degraded (19.039%); 10788/188556 objects misplaced (5.721%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T11:00:09.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:09 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 36046/190029 objects degraded (18.969%), 22 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:09.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:09 smithi070 ceph-mon[112139]: pgmap v2993: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 250 op/s; 35899/188556 objects degraded (19.039%); 10788/188556 objects misplaced (5.721%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T11:00:09.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:09 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 36046/190029 objects degraded (18.969%), 22 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:09.416 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:09.439 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:09.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:09.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:09.541 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:09.564 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:09.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:09.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:09.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:09 smithi160 ceph-mon[101879]: pgmap v2993: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 250 op/s; 35899/188556 objects degraded (19.039%); 10788/188556 objects misplaced (5.721%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T11:00:09.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:09 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 36046/190029 objects degraded (18.969%), 22 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:09.666 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:09.690 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:09.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:09.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:09.792 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:09.812 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:09.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:09.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:09.914 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:09.936 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:09.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:09.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:10.037 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:10.055 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:00:10.055 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:00:10.062 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:10.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:10.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:10.081 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T11:00:10.082 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T11:00:10.164 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:10.189 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:10.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:10.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:10.291 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:10.324 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:10.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:10.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:10.425 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:10.447 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:10.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:10.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:10.549 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:10.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:10.579 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:10.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:10.680 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:10.705 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:10.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:10.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:10.807 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:10.828 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:10.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:10.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:10.930 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:10.955 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:10.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:10.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:11.058 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:11.086 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:11.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:11.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:11.134 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T11:00:11.189 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:11.213 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:11.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:11.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:11.314 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:11.335 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:11.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:11.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:11.437 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:11.460 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:11.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:11.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:11.463 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:11 smithi070 ceph-mon[112139]: pgmap v2994: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 255 op/s; 35492/186174 objects degraded (19.064%); 10646/186174 objects misplaced (5.718%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T11:00:11.561 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:11.586 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:11.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:11.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:11.687 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:11.709 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:11.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:11.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:11.810 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:11.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:11.833 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:11.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:11 smithi160 ceph-mon[101879]: pgmap v2994: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 255 op/s; 35492/186174 objects degraded (19.064%); 10646/186174 objects misplaced (5.718%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T11:00:11.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:11 smithi070 ceph-mon[107385]: pgmap v2994: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 255 op/s; 35492/186174 objects degraded (19.064%); 10646/186174 objects misplaced (5.718%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T11:00:11.934 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:11.955 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:11.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:11.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:12.057 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:12.078 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:12.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:12.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:12.082 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T11:00:12.179 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:12.202 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:12.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:12.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:12.303 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:12.324 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:12.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:12.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:12.425 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:12.447 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:12.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:12.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:12.500 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:00:12.549 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:12.570 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:12.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:12.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:12.671 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:12.697 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:12.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:12.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:12.799 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:12.822 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:12.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:12.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:12.924 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:12.946 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:12.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:12.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:13.048 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:13.071 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:13.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:13.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:13.173 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:13.196 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:13.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:13.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:13.250 INFO:teuthology.orchestra.run.smithi070.stderr:no valid command found; 10 closest matches: 2024-09-21T11:00:13.250 INFO:teuthology.orchestra.run.smithi070.stderr:0 2024-09-21T11:00:13.250 INFO:teuthology.orchestra.run.smithi070.stderr:1 2024-09-21T11:00:13.250 INFO:teuthology.orchestra.run.smithi070.stderr:2 2024-09-21T11:00:13.250 INFO:teuthology.orchestra.run.smithi070.stderr:abort 2024-09-21T11:00:13.250 INFO:teuthology.orchestra.run.smithi070.stderr:assert 2024-09-21T11:00:13.250 INFO:teuthology.orchestra.run.smithi070.stderr:bench [] [] [] [] 2024-09-21T11:00:13.250 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs debug_inject_read_zeros 2024-09-21T11:00:13.250 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs files list 2024-09-21T11:00:13.251 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs stats 2024-09-21T11:00:13.251 INFO:teuthology.orchestra.run.smithi070.stderr:bluestore allocator dump block 2024-09-21T11:00:13.251 INFO:teuthology.orchestra.run.smithi070.stderr:Error EINVAL: invalid command 2024-09-21T11:00:13.298 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:13.320 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:13.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:13.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:13.361 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:00:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:00:13] "GET /metrics HTTP/1.1" 200 36586 "" "Prometheus/2.51.0" 2024-09-21T11:00:13.422 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:13.445 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:13.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:13.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:13.471 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T11:00:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T11:00:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T11:00:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T11:00:13.472 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T11:00:13.547 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:13.569 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:13.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:13.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:13.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:13 smithi070 ceph-mon[107385]: pgmap v2995: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 255 op/s; 35492/186174 objects degraded (19.064%); 10646/186174 objects misplaced (5.718%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T11:00:13.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:13 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 35492/186174 objects degraded (19.064%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:13.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:13 smithi070 ceph-mon[112139]: pgmap v2995: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 255 op/s; 35492/186174 objects degraded (19.064%); 10646/186174 objects misplaced (5.718%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T11:00:13.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:13 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 35492/186174 objects degraded (19.064%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:13.671 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:13.693 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:13.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:13.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:13.795 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:13.819 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:13.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:13.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:13 smithi160 ceph-mon[101879]: pgmap v2995: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 255 op/s; 35492/186174 objects degraded (19.064%); 10646/186174 objects misplaced (5.718%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T11:00:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:13 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 35492/186174 objects degraded (19.064%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:13.920 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:13.947 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:13.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:13.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:14.048 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:14.075 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:14.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:14.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:14.177 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:14.199 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:14.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:14.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:14.300 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:14.322 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:14.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:14.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:14.424 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:14.446 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:14.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:14.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:14.548 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:14.569 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:14.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:14.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:14.670 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:14.693 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:14.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:14.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:14.795 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:14.822 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:14.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:14.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:14.923 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:14.945 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:14.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:14.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:15.003 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T11:00:15.003 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T11:00:15.047 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:15.068 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:15.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:15.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:15.170 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:15.195 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:15.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:15.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:15.297 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:15.319 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:15.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:15.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:15.420 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:15.443 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:15.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:15.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:15.545 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:15.567 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:15.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:15.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:15.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:15 smithi070 ceph-mon[107385]: pgmap v2996: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.7 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 262 op/s; 35257/183873 objects degraded (19.175%); 10470/183873 objects misplaced (5.694%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T11:00:15.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:15 smithi070 ceph-mon[112139]: pgmap v2996: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.7 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 262 op/s; 35257/183873 objects degraded (19.175%); 10470/183873 objects misplaced (5.694%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T11:00:15.670 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:15.692 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:15.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:15.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:15.793 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:15.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:15.822 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:15.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:15 smithi160 ceph-mon[101879]: pgmap v2996: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.7 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 262 op/s; 35257/183873 objects degraded (19.175%); 10470/183873 objects misplaced (5.694%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T11:00:15.911 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T11:00:15.923 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:15.976 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:15.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:15.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:16.078 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:16.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:16.108 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:16.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:16.209 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:16.231 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:16.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:16.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:16.333 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:16.358 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:16.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:16.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:16.459 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:16.486 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:16.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:16.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:16.588 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:16.611 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:16.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:16.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:16.713 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:16.736 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:16.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:16.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:16.837 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:16.858 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:16.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:16.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:16.960 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:16.981 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:16.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:16.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:17.083 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:17.103 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:17.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:17.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:17.168 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:00:17.205 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:17.228 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:17.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:17.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:17.330 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:17.357 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:17.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:17.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:17.458 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:17.488 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:17.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:17.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:17.590 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:17.612 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:17.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:17.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:17.713 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:17.740 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:17.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:17.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:17.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:17 smithi160 ceph-mon[101879]: pgmap v2997: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.7 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 267 op/s; 34889/181992 objects degraded (19.171%); 10292/181992 objects misplaced (5.655%); 3.2 MiB/s, 50 objects/s recovering 2024-09-21T11:00:17.841 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:17.863 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:17.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:17.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:17.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:17 smithi070 ceph-mon[107385]: pgmap v2997: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.7 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 267 op/s; 34889/181992 objects degraded (19.171%); 10292/181992 objects misplaced (5.655%); 3.2 MiB/s, 50 objects/s recovering 2024-09-21T11:00:17.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:17 smithi070 ceph-mon[112139]: pgmap v2997: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.7 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 267 op/s; 34889/181992 objects degraded (19.171%); 10292/181992 objects misplaced (5.655%); 3.2 MiB/s, 50 objects/s recovering 2024-09-21T11:00:17.965 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:17.987 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:17.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:17.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:18.089 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:18.111 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:18.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:18.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:18.213 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:18.234 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:18.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:18.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:18.336 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:18.364 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:18.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:18.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:18.466 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:18.474 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T11:00:18.492 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:18.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:18.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:18.594 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:18.615 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:18.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:18.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:18.717 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:18.738 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:18.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:18.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:18.840 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:18 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3419066182' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T11:00:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:18 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 34889/181992 objects degraded (19.171%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:18.861 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:18.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:18.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:18.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:18 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3419066182' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T11:00:18.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:18 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 34889/181992 objects degraded (19.171%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:18.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:18 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3419066182' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T11:00:18.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:18 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 34889/181992 objects degraded (19.171%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:18.963 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:18.989 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:18.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:18.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:19.090 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:19.111 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:19.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:19.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:19.213 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:19.241 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:19.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:19.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:19.343 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:19.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:19.367 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:19.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:19.467 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:19.489 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:19.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:19.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:19.590 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:19.612 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:19.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:19.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:19.714 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:19.739 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:19.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:19.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:19.840 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:19 smithi160 ceph-mon[101879]: pgmap v2998: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.7 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 288 op/s; 34681/180279 objects degraded (19.237%); 10101/180279 objects misplaced (5.603%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T11:00:19.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:19 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:00:19.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:19 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3419066182' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T11:00:19.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:19 smithi160 ceph-mon[101879]: osdmap e1556: 8 total, 6 up, 5 in 2024-09-21T11:00:19.862 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:19.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:19.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:19.869 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:19 smithi070 ceph-mon[112139]: pgmap v2998: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.7 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 288 op/s; 34681/180279 objects degraded (19.237%); 10101/180279 objects misplaced (5.603%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T11:00:19.869 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:19 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:00:19.869 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:19 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3419066182' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T11:00:19.869 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:19 smithi070 ceph-mon[112139]: osdmap e1556: 8 total, 6 up, 5 in 2024-09-21T11:00:19.869 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:19 smithi070 ceph-mon[107385]: pgmap v2998: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.7 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 288 op/s; 34681/180279 objects degraded (19.237%); 10101/180279 objects misplaced (5.603%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T11:00:19.869 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:19 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:00:19.869 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:19 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3419066182' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T11:00:19.869 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:19 smithi070 ceph-mon[107385]: osdmap e1556: 8 total, 6 up, 5 in 2024-09-21T11:00:19.964 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:19.986 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:19.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:19.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:20.004 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7, 4] out_osds: [2, 5, 0] dead_osds: [5, 6] live_osds: [0, 1, 2, 3, 4, 7] 2024-09-21T11:00:20.004 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T11:00:20.004 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_4 2024-09-21T11:00:20.004 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T11:00:20.087 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:20.112 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:20.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:20.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:20.214 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:20.238 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:20.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:20.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:20.340 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:20.371 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:20.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:20.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:20.444 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:00:20.444 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:00:20.464 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:00:20.464 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:00:20.473 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:20.484 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:00:20.484 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:00:20.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:20.497 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:20.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:20.502 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:00:20.502 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:00:20.519 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:00:20.520 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:00:20.520 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T11:00:20.520 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T11:00:20.538 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:00:20.538 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:00:20.598 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:20.620 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:20.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:20.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:20.721 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:20.744 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:20.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:20.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:20.805 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:00:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:00:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:20 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1704443087' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:00:20.846 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:20.868 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:20.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:20.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:20.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:00:20.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:20 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1704443087' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:00:20.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:00:20.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:20 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1704443087' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:00:20.970 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:20.992 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:20.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:20.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:21.093 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:21.116 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:21.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:21.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:21.218 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:21.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:21.247 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:21.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:21.348 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:21.359 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:00:21.376 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:21.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:21.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:21.477 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:21.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:21.503 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:21.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:21.604 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:21.630 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:21.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:21.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:21.732 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:21.758 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:21.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:21.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:21.810 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:21 smithi070 ceph-mon[107385]: pgmap v3000: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.6 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 284 op/s; 34234/177633 objects degraded (19.272%); 9944/177633 objects misplaced (5.598%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T11:00:21.810 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:21 smithi070 ceph-mon[112139]: pgmap v3000: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.6 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 284 op/s; 34234/177633 objects degraded (19.272%); 9944/177633 objects misplaced (5.598%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T11:00:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:21 smithi160 ceph-mon[101879]: pgmap v3000: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.6 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 284 op/s; 34234/177633 objects degraded (19.272%); 9944/177633 objects misplaced (5.598%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T11:00:21.861 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:21.885 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:21.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:21.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:21.890 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T11:00:21.987 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:22.009 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:22.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:22.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:22.111 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:22.134 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:22.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:22.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:22.235 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:22.256 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:22.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:22.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:22.358 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:22.382 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:22.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:22.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:22.484 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:22.506 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:22.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:22.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:22.566 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T11:00:22.567 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T11:00:22.569 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:00:22.608 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:22.631 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:22.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:22.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:22.732 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:22.754 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:22.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:22.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:22.856 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:22.882 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:22.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:22.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:22.984 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:23.007 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:23.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:23.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:23.109 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:23.130 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:23.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:23.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:23.231 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:23.253 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:23.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:23.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:23.355 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:23.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:00:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:00:23] "GET /metrics HTTP/1.1" 200 36587 "" "Prometheus/2.51.0" 2024-09-21T11:00:23.378 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:23.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:23.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:23.479 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:23.501 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:23.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:23.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:23.603 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:23.630 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:23.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:23.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:23.731 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:23.752 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:23.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:23.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:23 smithi160 ceph-mon[101879]: pgmap v3001: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.6 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 284 op/s; 34234/177633 objects degraded (19.272%); 9944/177633 objects misplaced (5.598%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T11:00:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:23 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 34234/177633 objects degraded (19.272%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:23.854 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:23.876 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:23.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:23.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:23.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:23 smithi070 ceph-mon[107385]: pgmap v3001: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.6 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 284 op/s; 34234/177633 objects degraded (19.272%); 9944/177633 objects misplaced (5.598%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T11:00:23.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:23 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 34234/177633 objects degraded (19.272%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:23.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:23 smithi070 ceph-mon[112139]: pgmap v3001: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.6 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 284 op/s; 34234/177633 objects degraded (19.272%); 9944/177633 objects misplaced (5.598%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T11:00:23.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:23 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 34234/177633 objects degraded (19.272%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:23.978 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:24.000 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:24.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:24.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:24.102 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:24.124 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:24.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:24.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:24.225 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:24.247 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:24.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:24.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:24.348 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:24.375 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:24.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:24.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:24.477 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:24.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:24.499 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:24.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:24.567 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T11:00:24.600 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:24.631 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:24.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:24.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:24.733 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:24.756 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:24.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:24.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:24.858 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:24.882 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:24.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:24.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:24.984 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:25.005 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:25.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:25.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:25.106 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:25.128 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:25.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:25.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:25.230 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:25.252 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:25.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:25.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:25.353 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:25.375 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:25.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:25.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:25.477 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:25.501 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:25.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:25.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:25.575 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:25 smithi070 ceph-mon[107385]: pgmap v3002: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.6 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 292 op/s; 33952/175095 objects degraded (19.391%); 9775/175095 objects misplaced (5.583%); 3.6 MiB/s, 56 objects/s recovering 2024-09-21T11:00:25.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:25 smithi070 ceph-mon[112139]: pgmap v3002: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.6 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 292 op/s; 33952/175095 objects degraded (19.391%); 9775/175095 objects misplaced (5.583%); 3.6 MiB/s, 56 objects/s recovering 2024-09-21T11:00:25.604 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:25.626 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:25.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:25.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:25.728 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:25.757 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:25.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:25.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:25 smithi160 ceph-mon[101879]: pgmap v3002: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.6 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 292 op/s; 33952/175095 objects degraded (19.391%); 9775/175095 objects misplaced (5.583%); 3.6 MiB/s, 56 objects/s recovering 2024-09-21T11:00:25.859 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:25.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:25.883 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:25.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:25.984 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:26.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:26.007 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:26.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:26.108 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:26.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:26.131 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:26.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:26.232 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:26.252 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:26.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:26.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:26.354 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:26.376 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:26.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:26.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:26.477 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:26.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:26.500 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:26.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:26.601 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:26.624 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:26.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:26.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:26.725 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:26.747 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:26.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:26.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:26.849 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:26.875 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:26.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:26.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:26.976 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:26.998 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:26.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:26.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:27.051 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T11:00:27.051 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T11:00:27.100 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:27.127 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:27.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:27.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:27.229 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:27.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:27.253 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:27.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:27.354 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:27.376 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:27.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:27.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:27.477 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:27.501 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:27.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:27.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:27.603 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:27.625 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:27.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:27.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:27.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:27 smithi070 ceph-mon[107385]: pgmap v3003: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.5 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 308 op/s; 33522/172740 objects degraded (19.406%); 9555/172740 objects misplaced (5.531%); 3.6 MiB/s, 57 objects/s recovering 2024-09-21T11:00:27.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:27 smithi070 ceph-mon[112139]: pgmap v3003: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.5 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 308 op/s; 33522/172740 objects degraded (19.406%); 9555/172740 objects misplaced (5.531%); 3.6 MiB/s, 57 objects/s recovering 2024-09-21T11:00:27.727 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:27.748 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:27.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:27.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:27 smithi160 ceph-mon[101879]: pgmap v3003: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.5 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 308 op/s; 33522/172740 objects degraded (19.406%); 9555/172740 objects misplaced (5.531%); 3.6 MiB/s, 57 objects/s recovering 2024-09-21T11:00:27.850 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:27.874 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:27.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:27.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:27.975 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:27.998 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:27.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:27.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:28.100 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:28.127 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:28.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:28.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:28.228 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:28.254 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:28.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:28.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:28.356 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:28.381 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:28.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:28.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:28.483 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:28.506 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:28.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:28.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:28.519 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T11:00:28.520 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T11:00:28.607 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:28.629 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:28.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:28.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:28.731 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:28.752 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:28.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:28.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:28.797 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:00:28.854 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:28.880 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:28.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:28.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:28.917 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:28 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:00:28.917 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:28 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:00:28.917 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:28 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T11:00:28.917 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:28 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T11:00:28.917 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:28 smithi070 ceph-mon[112139]: pgmap v3004: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.5 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 314 op/s; 33333/171003 objects degraded (19.493%); 9359/171003 objects misplaced (5.473%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T11:00:28.917 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:28 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:00:28.917 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:28 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 33522/172740 objects degraded (19.406%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:28.917 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:28 smithi070 ceph-mon[112139]: from='client.46376 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T11:00:28.917 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:28 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/712274961' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:00:28.918 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:28 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:00:28.918 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:28 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:00:28.918 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:28 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:00:28.918 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:28 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T11:00:28.918 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:28 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T11:00:28.918 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:28 smithi070 ceph-mon[107385]: pgmap v3004: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.5 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 314 op/s; 33333/171003 objects degraded (19.493%); 9359/171003 objects misplaced (5.473%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T11:00:28.918 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:28 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:00:28.918 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:28 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 33522/172740 objects degraded (19.406%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:28.918 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:28 smithi070 ceph-mon[107385]: from='client.46376 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T11:00:28.919 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:28 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/712274961' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:00:28.919 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:28 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:00:28.982 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:29.011 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:29.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:29.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:29.113 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:29.136 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:29.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:29.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:29.237 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:29.240 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T11:00:29.260 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:29.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:29.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:29.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:28 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:00:29.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:28 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:00:29.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:28 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T11:00:29.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:28 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T11:00:29.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:28 smithi160 ceph-mon[101879]: pgmap v3004: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.5 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 314 op/s; 33333/171003 objects degraded (19.493%); 9359/171003 objects misplaced (5.473%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T11:00:29.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:28 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:00:29.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:28 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 33522/172740 objects degraded (19.406%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:29.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:28 smithi160 ceph-mon[101879]: from='client.46376 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T11:00:29.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:28 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/712274961' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:00:29.342 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:28 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:00:29.361 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:29.391 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:29.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:29.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:29.493 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:29.517 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:29.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:29.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:29.618 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:29.642 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:29.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:29.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:29.650 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T11:00:29.744 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:29.770 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:29.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:29.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:29.872 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:29.894 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:29.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:29.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:29.995 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:30.019 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:30.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:30.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:30.083 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:29 smithi070 ceph-mon[107385]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-21T11:00:30.083 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:29 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:00:30.083 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:29 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T11:00:30.083 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:29 smithi070 ceph-mon[107385]: osdmap e1557: 8 total, 6 up, 5 in 2024-09-21T11:00:30.083 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:29 smithi070 ceph-mon[112139]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-21T11:00:30.083 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:29 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:00:30.083 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:29 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T11:00:30.083 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:29 smithi070 ceph-mon[112139]: osdmap e1557: 8 total, 6 up, 5 in 2024-09-21T11:00:30.121 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:30.142 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:30.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:30.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:30.244 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:30.266 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:30.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:30.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:30.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:29 smithi160 ceph-mon[101879]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-21T11:00:30.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:29 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:00:30.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:29 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T11:00:30.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:29 smithi160 ceph-mon[101879]: osdmap e1557: 8 total, 6 up, 5 in 2024-09-21T11:00:30.367 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:30.391 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:30.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:30.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:30.493 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:30.516 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:30.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:30.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:30.617 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:30.639 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:30.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:30.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:30.741 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:30.763 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:30.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:30.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:30.864 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:30.887 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:30.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:30.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:30.989 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:31.011 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:31.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:31.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:31.112 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:31.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:30 smithi070 ceph-mon[107385]: pgmap v3006: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.4 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 282 op/s; 33156/169305 objects degraded (19.584%); 9359/169305 objects misplaced (5.528%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T11:00:31.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:30 smithi070 ceph-mon[112139]: pgmap v3006: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.4 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 282 op/s; 33156/169305 objects degraded (19.584%); 9359/169305 objects misplaced (5.528%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T11:00:31.134 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:31.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:31.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:31.236 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:31.257 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:31.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:31.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:31.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:30 smithi160 ceph-mon[101879]: pgmap v3006: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.4 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 282 op/s; 33156/169305 objects degraded (19.584%); 9359/169305 objects misplaced (5.528%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T11:00:31.359 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:31.385 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:31.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:31.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:31.486 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:31.514 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:31.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:31.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:31.616 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:31.638 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:31.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:31.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:31.739 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:31.760 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:31.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:31.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:31.862 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:31.883 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:31.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:31.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:31.984 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:32.006 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:32.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:32.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:32.108 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:32.129 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:32.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:32.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:32.230 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:32.252 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:32.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:32.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:32.353 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:32.375 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:32.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:32.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:32.476 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:32.498 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:32.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:32.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:32.600 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:32.626 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:32.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:32.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:32.727 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:32.748 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:32.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:32.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:32.850 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:32.864 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":3004,"stamp":"2024-09-21T11:00:28.232741+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3734187547,"num_objects":57001,"num_object_clones":0,"num_object_copies":171003,"num_objects_missing_on_primary":469,"num_objects_missing":469,"num_objects_degraded":33333,"num_objects_misplaced":9359,"num_objects_unfound":0,"num_objects_dirty":57001,"num_whiteouts":0,"num_read":174,"num_read_kb":376,"num_write":77881,"num_write_kb":4311277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9134,"num_bytes_recovered":596189097,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68316,"ondisk_log_size":68316,"up":235,"acting":237,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":319,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":16935208,"kb_used_data":11285432,"kb_used_omap":141,"kb_used_meta":5649522,"kb_avail":451790552,"statfs":{"total":479975178240,"available":462633525248,"internally_reserved":0,"allocated":11556282368,"data_stored":11529578213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":144912,"internal_metadata":5785111024},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-202637312,"num_objects":-3092,"num_object_clones":0,"num_object_copies":-9276,"num_objects_missing_on_primary":-373,"num_objects_missing":-373,"num_objects_degraded":-1348,"num_objects_misplaced":-742,"num_objects_unfound":0,"num_objects_dirty":-3092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3092,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":576,"num_bytes_recovered":37748736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.822032"},"pg_stats":[{"pgid":"57.f","version":"1556'4931","reported_seq":4979,"reported_epoch":1556,"state":"active+undersized+degraded","last_fresh":"2024-09-21T11:00:27.334425+0000","last_change":"2024-09-21T10:58:44.721330+0000","last_active":"2024-09-21T11:00:27.334425+0000","last_peered":"2024-09-21T11:00:27.334425+0000","last_clean":"2024-09-21T10:58:43.715781+0000","last_became_active":"2024-09-21T10:58:44.721330+0000","last_became_peered":"2024-09-21T10:58:44.721330+0000","last_unstale":"2024-09-21T11:00:27.334425+0000","last_undegraded":"2024-09-21T10:58:44.714262+0000","last_fullsized":"2024-09-21T10:58:44.714154+0000","mapping_epoch":1541,"log_start":"1528'800","ondisk_log_start":"1528'800","created":1524,"last_epoch_clean":1525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":4131,"log_dups_size":0,"ondisk_log_size":4131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:14:35.842654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235470848,"num_objects":3593,"num_object_clones":0,"num_object_copies":10779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4931,"num_write_kb":272768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":3593}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"57.e","version":"1556'4714","reported_seq":4796,"reported_epoch":1556,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T11:00:24.303552+0000","last_change":"2024-09-21T10:59:06.547306+0000","last_active":"2024-09-21T11:00:24.303552+0000","last_peered":"2024-09-21T11:00:24.303552+0000","last_clean":"2024-09-21T10:58:43.700774+0000","last_became_active":"2024-09-21T10:59:06.542300+0000","last_became_peered":"2024-09-21T10:59:06.542300+0000","last_unstale":"2024-09-21T11:00:24.303552+0000","last_undegraded":"2024-09-21T10:59:06.452582+0000","last_fullsized":"2024-09-21T10:59:06.434803+0000","mapping_epoch":1548,"log_start":"1532'1000","ondisk_log_start":"1532'1000","created":1524,"last_epoch_clean":1525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":3714,"log_dups_size":0,"ondisk_log_size":3714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230424576,"num_objects":3516,"num_object_clones":0,"num_object_copies":10548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4714,"num_write_kb":263360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":392}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"57.d","version":"1556'4842","reported_seq":6175,"reported_epoch":1556,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T11:00:25.082450+0000","last_change":"2024-09-21T10:59:09.173585+0000","last_active":"2024-09-21T11:00:25.082450+0000","last_peered":"2024-09-21T11:00:25.082450+0000","last_clean":"2024-09-21T10:58:20.466678+0000","last_became_active":"2024-09-21T10:59:09.066402+0000","last_became_peered":"2024-09-21T10:59:09.066402+0000","last_unstale":"2024-09-21T11:00:25.082450+0000","last_undegraded":"2024-09-21T10:59:08.978402+0000","last_fullsized":"2024-09-21T10:59:06.432968+0000","mapping_epoch":1548,"log_start":"1528'800","ondisk_log_start":"1528'800","created":1524,"last_epoch_clean":1525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":4042,"log_dups_size":0,"ondisk_log_size":4042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:37:48.647745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234815512,"num_objects":3584,"num_object_clones":0,"num_object_copies":10752,"num_objects_missing_on_primary":233,"num_objects_missing":233,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3584,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4842,"num_write_kb":269569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":633,"num_bytes_recovered":41484288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1,4","objects":1547},{"shards":"1,7","objects":233}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"57.c","version":"1556'4782","reported_seq":7344,"reported_epoch":1556,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T11:00:25.089364+0000","last_change":"2024-09-21T10:58:48.262183+0000","last_active":"2024-09-21T11:00:25.089364+0000","last_peered":"2024-09-21T11:00:25.089364+0000","last_clean":"2024-09-21T10:58:43.677050+0000","last_became_active":"2024-09-21T10:58:45.788673+0000","last_became_peered":"2024-09-21T10:58:45.788673+0000","last_unstale":"2024-09-21T11:00:25.089364+0000","last_undegraded":"2024-09-21T10:58:45.737874+0000","last_fullsized":"2024-09-21T11:00:25.089364+0000","mapping_epoch":1542,"log_start":"1528'700","ondisk_log_start":"1528'700","created":1524,"last_epoch_clean":1540,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":4082,"log_dups_size":0,"ondisk_log_size":4082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:57:53.473443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231866368,"num_objects":3538,"num_object_clones":0,"num_object_copies":10614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1312,"num_objects_misplaced":2226,"num_objects_unfound":0,"num_objects_dirty":3538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4782,"num_write_kb":266240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1247,"num_bytes_recovered":81723392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7,1],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"1,4,7","objects":2226},{"shards":"4,7","objects":1312}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"57.b","version":"1556'4946","reported_seq":6340,"reported_epoch":1556,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T11:00:25.254240+0000","last_change":"2024-09-21T10:58:48.383673+0000","last_active":"2024-09-21T11:00:25.254240+0000","last_peered":"2024-09-21T11:00:25.254240+0000","last_clean":"2024-09-21T10:58:20.469888+0000","last_became_active":"2024-09-21T10:58:48.343642+0000","last_became_peered":"2024-09-21T10:58:48.343642+0000","last_unstale":"2024-09-21T11:00:25.254240+0000","last_undegraded":"2024-09-21T10:58:48.309206+0000","last_fullsized":"2024-09-21T11:00:25.254240+0000","mapping_epoch":1542,"log_start":"1528'900","ondisk_log_start":"1528'900","created":1524,"last_epoch_clean":1525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":4046,"log_dups_size":0,"ondisk_log_size":4046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:46:50.615845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237240320,"num_objects":3620,"num_object_clones":0,"num_object_copies":10860,"num_objects_missing_on_primary":235,"num_objects_missing":235,"num_objects_degraded":2034,"num_objects_misplaced":1821,"num_objects_unfound":0,"num_objects_dirty":3620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4946,"num_write_kb":274112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":664,"num_bytes_recovered":43515904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,7],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1,4","objects":1799},{"shards":"1,4,7","objects":1574},{"shards":"1,7","objects":235}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"57.a","version":"1556'4921","reported_seq":10291,"reported_epoch":1556,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:00:27.241069+0000","last_change":"2024-09-21T10:58:59.420184+0000","last_active":"2024-09-21T11:00:27.241069+0000","last_peered":"2024-09-21T11:00:27.241069+0000","last_clean":"2024-09-21T11:00:27.241069+0000","last_became_active":"2024-09-21T10:58:59.415394+0000","last_became_peered":"2024-09-21T10:58:59.415394+0000","last_unstale":"2024-09-21T11:00:27.241069+0000","last_undegraded":"2024-09-21T11:00:27.241069+0000","last_fullsized":"2024-09-21T11:00:27.241069+0000","mapping_epoch":1544,"log_start":"1528'800","ondisk_log_start":"1528'800","created":1524,"last_epoch_clean":1545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":4121,"log_dups_size":0,"ondisk_log_size":4121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:30:42.872604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236650496,"num_objects":3611,"num_object_clones":0,"num_object_copies":10833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3611,"num_objects_unfound":0,"num_objects_dirty":3611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4921,"num_write_kb":273024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2650,"num_bytes_recovered":173670400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,1],"avail_no_missing":["7","1","4"],"object_location_counts":[{"shards":"1,4,7","objects":3611}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"57.9","version":"1556'4798","reported_seq":6558,"reported_epoch":1556,"state":"active+undersized+degraded","last_fresh":"2024-09-21T11:00:25.107430+0000","last_change":"2024-09-21T10:58:44.728972+0000","last_active":"2024-09-21T11:00:25.107430+0000","last_peered":"2024-09-21T11:00:25.107430+0000","last_clean":"2024-09-21T10:58:43.705315+0000","last_became_active":"2024-09-21T10:58:44.728972+0000","last_became_peered":"2024-09-21T10:58:44.728972+0000","last_unstale":"2024-09-21T11:00:25.107430+0000","last_undegraded":"2024-09-21T10:58:44.715841+0000","last_fullsized":"2024-09-21T10:58:44.715729+0000","mapping_epoch":1541,"log_start":"1528'700","ondisk_log_start":"1528'700","created":1524,"last_epoch_clean":1538,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":4098,"log_dups_size":0,"ondisk_log_size":4098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:13:49.898142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232652800,"num_objects":3550,"num_object_clones":0,"num_object_copies":10650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3550,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4798,"num_write_kb":267136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":846,"num_bytes_recovered":55443456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":3550}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"57.8","version":"1556'4871","reported_seq":6339,"reported_epoch":1556,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T11:00:27.322454+0000","last_change":"2024-09-21T10:59:09.117812+0000","last_active":"2024-09-21T11:00:27.322454+0000","last_peered":"2024-09-21T11:00:27.322454+0000","last_clean":"2024-09-21T10:58:20.477216+0000","last_became_active":"2024-09-21T10:58:45.841910+0000","last_became_peered":"2024-09-21T10:58:45.841910+0000","last_unstale":"2024-09-21T11:00:27.322454+0000","last_undegraded":"2024-09-21T10:58:45.837108+0000","last_fullsized":"2024-09-21T11:00:27.322454+0000","mapping_epoch":1542,"log_start":"1528'800","ondisk_log_start":"1528'800","created":1524,"last_epoch_clean":1525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":4071,"log_dups_size":0,"ondisk_log_size":4071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:53:15.066341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227606528,"num_objects":3473,"num_object_clones":0,"num_object_copies":10419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1967,"num_objects_misplaced":1699,"num_objects_unfound":0,"num_objects_dirty":3473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4871,"num_write_kb":267008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":703,"num_bytes_recovered":46071808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,1],"avail_no_missing":["7"],"object_location_counts":[{"shards":"1,4,7","objects":1506},{"shards":"1,7","objects":193},{"shards":"4,7","objects":1774}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"57.7","version":"1556'4939","reported_seq":4993,"reported_epoch":1556,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T11:00:27.379045+0000","last_change":"2024-09-21T10:59:06.567534+0000","last_active":"2024-09-21T11:00:27.379045+0000","last_peered":"2024-09-21T11:00:27.379045+0000","last_clean":"2024-09-21T10:58:43.706565+0000","last_became_active":"2024-09-21T10:59:06.558404+0000","last_became_peered":"2024-09-21T10:59:06.558404+0000","last_unstale":"2024-09-21T11:00:27.379045+0000","last_undegraded":"2024-09-21T10:59:06.465380+0000","last_fullsized":"2024-09-21T10:59:06.436162+0000","mapping_epoch":1548,"log_start":"1528'800","ondisk_log_start":"1528'800","created":1524,"last_epoch_clean":1525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":4139,"log_dups_size":0,"ondisk_log_size":4139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:19:22.626591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235864064,"num_objects":3599,"num_object_clones":0,"num_object_copies":10797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4939,"num_write_kb":273216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":382}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"57.6","version":"1556'4838","reported_seq":6212,"reported_epoch":1556,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-21T11:00:27.393447+0000","last_change":"2024-09-21T10:59:05.494755+0000","last_active":"2024-09-21T11:00:27.393447+0000","last_peered":"2024-09-21T11:00:27.393447+0000","last_clean":"2024-09-21T10:58:20.477277+0000","last_became_active":"2024-09-21T10:59:05.489014+0000","last_became_peered":"2024-09-21T10:59:05.489014+0000","last_unstale":"2024-09-21T11:00:27.393447+0000","last_undegraded":"2024-09-21T10:59:05.484906+0000","last_fullsized":"2024-09-21T11:00:27.393447+0000","mapping_epoch":1547,"log_start":"1528'700","ondisk_log_start":"1528'700","created":1524,"last_epoch_clean":1525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":4138,"log_dups_size":0,"ondisk_log_size":4138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:35:38.607914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231735296,"num_objects":3536,"num_object_clones":0,"num_object_copies":10608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4838,"num_write_kb":267968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":654,"num_bytes_recovered":42860544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":["7"],"object_location_counts":[{"shards":"1,4,7","objects":1586},{"shards":"1,7","objects":199},{"shards":"4,7","objects":1751}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"57.5","version":"1556'4890","reported_seq":4943,"reported_epoch":1556,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T11:00:27.378900+0000","last_change":"2024-09-21T10:59:09.067611+0000","last_active":"2024-09-21T11:00:27.378900+0000","last_peered":"2024-09-21T11:00:27.378900+0000","last_clean":"2024-09-21T10:58:43.699982+0000","last_became_active":"2024-09-21T10:59:09.060408+0000","last_became_peered":"2024-09-21T10:59:09.060408+0000","last_unstale":"2024-09-21T11:00:27.378900+0000","last_undegraded":"2024-09-21T10:59:06.469346+0000","last_fullsized":"2024-09-21T10:59:06.436402+0000","mapping_epoch":1548,"log_start":"1528'800","ondisk_log_start":"1528'800","created":1524,"last_epoch_clean":1525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":4090,"log_dups_size":0,"ondisk_log_size":4090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:00:46.458877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228982784,"num_objects":3494,"num_object_clones":0,"num_object_copies":10482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4890,"num_write_kb":268288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":299}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"57.4","version":"1556'4836","reported_seq":4912,"reported_epoch":1556,"state":"active+clean","last_fresh":"2024-09-21T11:00:24.448025+0000","last_change":"2024-09-21T10:58:06.431419+0000","last_active":"2024-09-21T11:00:24.448025+0000","last_peered":"2024-09-21T11:00:24.448025+0000","last_clean":"2024-09-21T11:00:24.448025+0000","last_became_active":"2024-09-21T10:58:06.431076+0000","last_became_peered":"2024-09-21T10:58:06.431076+0000","last_unstale":"2024-09-21T11:00:24.448025+0000","last_undegraded":"2024-09-21T11:00:24.448025+0000","last_fullsized":"2024-09-21T11:00:24.448025+0000","mapping_epoch":1524,"log_start":"0'0","ondisk_log_start":"0'0","created":1524,"last_epoch_clean":1525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":4836,"log_dups_size":0,"ondisk_log_size":4836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:35.135602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233046016,"num_objects":3556,"num_object_clones":0,"num_object_copies":10668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4836,"num_write_kb":268544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"57.3","version":"1556'4951","reported_seq":5028,"reported_epoch":1556,"state":"active+undersized+degraded","last_fresh":"2024-09-21T11:00:24.501583+0000","last_change":"2024-09-21T10:58:44.722616+0000","last_active":"2024-09-21T11:00:24.501583+0000","last_peered":"2024-09-21T11:00:24.501583+0000","last_clean":"2024-09-21T10:58:43.705212+0000","last_became_active":"2024-09-21T10:58:44.722616+0000","last_became_peered":"2024-09-21T10:58:44.722616+0000","last_unstale":"2024-09-21T11:00:24.501583+0000","last_undegraded":"2024-09-21T10:58:44.711684+0000","last_fullsized":"2024-09-21T10:58:44.711583+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":1524,"last_epoch_clean":1525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":4951,"log_dups_size":0,"ondisk_log_size":4951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:29:36.728503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237043712,"num_objects":3617,"num_object_clones":0,"num_object_copies":10851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4951,"num_write_kb":274176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":3617}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"57.2","version":"1556'4742","reported_seq":6612,"reported_epoch":1556,"state":"active+clean","last_fresh":"2024-09-21T11:00:24.448069+0000","last_change":"2024-09-21T10:58:30.615008+0000","last_active":"2024-09-21T11:00:24.448069+0000","last_peered":"2024-09-21T11:00:24.448069+0000","last_clean":"2024-09-21T11:00:24.448069+0000","last_became_active":"2024-09-21T10:58:30.612027+0000","last_became_peered":"2024-09-21T10:58:30.612027+0000","last_unstale":"2024-09-21T11:00:24.448069+0000","last_undegraded":"2024-09-21T11:00:24.448069+0000","last_fullsized":"2024-09-21T11:00:24.448069+0000","mapping_epoch":1534,"log_start":"0'0","ondisk_log_start":"0'0","created":1524,"last_epoch_clean":1535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":4742,"log_dups_size":0,"ondisk_log_size":4742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:53:27.293726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226492416,"num_objects":3456,"num_object_clones":0,"num_object_copies":10368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4742,"num_write_kb":262336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":891,"num_bytes_recovered":58392576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"57.1","version":"1556'4794","reported_seq":4854,"reported_epoch":1556,"state":"active+undersized+degraded","last_fresh":"2024-09-21T11:00:24.258855+0000","last_change":"2024-09-21T10:58:44.718190+0000","last_active":"2024-09-21T11:00:24.258855+0000","last_peered":"2024-09-21T11:00:24.258855+0000","last_clean":"2024-09-21T10:58:43.668991+0000","last_became_active":"2024-09-21T10:58:44.718190+0000","last_became_peered":"2024-09-21T10:58:44.718190+0000","last_unstale":"2024-09-21T11:00:24.258855+0000","last_undegraded":"2024-09-21T10:58:44.710938+0000","last_fullsized":"2024-09-21T10:58:44.710825+0000","mapping_epoch":1541,"log_start":"1533'1100","ondisk_log_start":"1533'1100","created":1524,"last_epoch_clean":1525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":3694,"log_dups_size":0,"ondisk_log_size":3694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:47.033505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235405312,"num_objects":3592,"num_object_clones":0,"num_object_copies":10776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3592,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4794,"num_write_kb":268352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":3592}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"57.0","version":"1556'4923","reported_seq":6518,"reported_epoch":1556,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T11:00:24.297054+0000","last_change":"2024-09-21T10:59:09.138035+0000","last_active":"2024-09-21T11:00:24.297054+0000","last_peered":"2024-09-21T11:00:24.297054+0000","last_clean":"2024-09-21T10:58:20.470296+0000","last_became_active":"2024-09-21T10:58:22.923025+0000","last_became_peered":"2024-09-21T10:58:22.923025+0000","last_unstale":"2024-09-21T11:00:24.297054+0000","last_undegraded":"2024-09-21T10:58:22.501064+0000","last_fullsized":"2024-09-21T10:58:22.492610+0000","mapping_epoch":1530,"log_start":"1533'1200","ondisk_log_start":"1533'1200","created":1524,"last_epoch_clean":1525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":3723,"log_dups_size":0,"ondisk_log_size":3723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238616576,"num_objects":3641,"num_object_clones":0,"num_object_copies":10923,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4923,"num_write_kb":274048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":757,"num_bytes_recovered":49610752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":3513}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":2076,"reported_epoch":1555,"state":"active+undersized+degraded","last_fresh":"2024-09-21T11:00:07.144845+0000","last_change":"2024-09-21T10:58:44.735075+0000","last_active":"2024-09-21T11:00:07.144845+0000","last_peered":"2024-09-21T11:00:07.144845+0000","last_clean":"2024-09-21T10:58:14.890372+0000","last_became_active":"2024-09-21T10:58:44.735075+0000","last_became_peered":"2024-09-21T10:58:44.735075+0000","last_unstale":"2024-09-21T11:00:07.144845+0000","last_undegraded":"2024-09-21T10:58:44.721754+0000","last_fullsized":"2024-09-21T10:58:44.721648+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:59:48.033965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2328,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":43,"reported_epoch":1555,"state":"active+undersized","last_fresh":"2024-09-21T11:00:07.141955+0000","last_change":"2024-09-21T10:58:44.749774+0000","last_active":"2024-09-21T11:00:07.141955+0000","last_peered":"2024-09-21T11:00:07.141955+0000","last_clean":"2024-09-21T10:58:41.702579+0000","last_became_active":"2024-09-21T10:58:44.749774+0000","last_became_peered":"2024-09-21T10:58:44.749774+0000","last_unstale":"2024-09-21T11:00:07.141955+0000","last_undegraded":"2024-09-21T11:00:07.141955+0000","last_fullsized":"2024-09-21T10:58:44.717412+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:38:54.862673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":1930,"reported_epoch":1556,"state":"active+undersized","last_fresh":"2024-09-21T11:00:18.487825+0000","last_change":"2024-09-21T10:58:44.721956+0000","last_active":"2024-09-21T11:00:18.487825+0000","last_peered":"2024-09-21T11:00:18.487825+0000","last_clean":"2024-09-21T10:39:14.280711+0000","last_became_active":"2024-09-21T10:58:44.721956+0000","last_became_peered":"2024-09-21T10:58:44.721956+0000","last_unstale":"2024-09-21T11:00:18.487825+0000","last_undegraded":"2024-09-21T11:00:18.487825+0000","last_fullsized":"2024-09-21T10:58:44.714264+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:52:58.057120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2332,"reported_epoch":1556,"state":"active+undersized","last_fresh":"2024-09-21T11:00:18.486428+0000","last_change":"2024-09-21T10:58:44.731586+0000","last_active":"2024-09-21T11:00:18.486428+0000","last_peered":"2024-09-21T11:00:18.486428+0000","last_clean":"2024-09-21T10:58:41.709945+0000","last_became_active":"2024-09-21T10:58:44.731586+0000","last_became_peered":"2024-09-21T10:58:44.731586+0000","last_unstale":"2024-09-21T11:00:18.486428+0000","last_undegraded":"2024-09-21T11:00:18.486428+0000","last_fullsized":"2024-09-21T10:58:44.711669+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:48:04.938735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":47,"reported_epoch":1555,"state":"active+undersized+degraded","last_fresh":"2024-09-21T11:00:07.150561+0000","last_change":"2024-09-21T10:58:44.747748+0000","last_active":"2024-09-21T11:00:07.150561+0000","last_peered":"2024-09-21T11:00:07.150561+0000","last_clean":"2024-09-21T10:58:41.703399+0000","last_became_active":"2024-09-21T10:58:44.747748+0000","last_became_peered":"2024-09-21T10:58:44.747748+0000","last_unstale":"2024-09-21T11:00:07.150561+0000","last_undegraded":"2024-09-21T10:58:44.721624+0000","last_fullsized":"2024-09-21T10:58:44.721523+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:53.680877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":43,"reported_epoch":1555,"state":"active+undersized","last_fresh":"2024-09-21T11:00:07.144498+0000","last_change":"2024-09-21T10:58:44.733467+0000","last_active":"2024-09-21T11:00:07.144498+0000","last_peered":"2024-09-21T11:00:07.144498+0000","last_clean":"2024-09-21T10:58:41.704407+0000","last_became_active":"2024-09-21T10:58:44.733467+0000","last_became_peered":"2024-09-21T10:58:44.733467+0000","last_unstale":"2024-09-21T11:00:07.144498+0000","last_undegraded":"2024-09-21T11:00:07.144498+0000","last_fullsized":"2024-09-21T10:58:44.715744+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:29:44.068810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":47,"reported_epoch":1555,"state":"active+undersized+degraded","last_fresh":"2024-09-21T11:00:07.145081+0000","last_change":"2024-09-21T10:58:44.752106+0000","last_active":"2024-09-21T11:00:07.145081+0000","last_peered":"2024-09-21T11:00:07.145081+0000","last_clean":"2024-09-21T10:58:41.702664+0000","last_became_active":"2024-09-21T10:58:44.752106+0000","last_became_peered":"2024-09-21T10:58:44.752106+0000","last_unstale":"2024-09-21T11:00:07.145081+0000","last_undegraded":"2024-09-21T10:58:44.726297+0000","last_fullsized":"2024-09-21T10:58:44.726189+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:14:04.302987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":37,"reported_epoch":1555,"state":"active+clean","last_fresh":"2024-09-21T11:00:07.165466+0000","last_change":"2024-09-21T10:58:21.500009+0000","last_active":"2024-09-21T11:00:07.165466+0000","last_peered":"2024-09-21T11:00:07.165466+0000","last_clean":"2024-09-21T11:00:07.165466+0000","last_became_active":"2024-09-21T10:58:21.499699+0000","last_became_peered":"2024-09-21T10:58:21.499699+0000","last_unstale":"2024-09-21T11:00:07.165466+0000","last_undegraded":"2024-09-21T11:00:07.165466+0000","last_fullsized":"2024-09-21T11:00:07.165466+0000","mapping_epoch":1529,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:35:49.068038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":37,"reported_epoch":1555,"state":"active+clean","last_fresh":"2024-09-21T11:00:07.165423+0000","last_change":"2024-09-21T10:58:21.498765+0000","last_active":"2024-09-21T11:00:07.165423+0000","last_peered":"2024-09-21T11:00:07.165423+0000","last_clean":"2024-09-21T11:00:07.165423+0000","last_became_active":"2024-09-21T10:58:21.497240+0000","last_became_peered":"2024-09-21T10:58:21.497240+0000","last_unstale":"2024-09-21T11:00:07.165423+0000","last_undegraded":"2024-09-21T11:00:07.165423+0000","last_fullsized":"2024-09-21T11:00:07.165423+0000","mapping_epoch":1529,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:42:21.165885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1217,"reported_epoch":1556,"state":"active+clean","last_fresh":"2024-09-21T11:00:18.487314+0000","last_change":"2024-09-21T10:58:21.509655+0000","last_active":"2024-09-21T11:00:18.487314+0000","last_peered":"2024-09-21T11:00:18.487314+0000","last_clean":"2024-09-21T11:00:18.487314+0000","last_became_active":"2024-09-21T10:58:21.508983+0000","last_became_peered":"2024-09-21T10:58:21.508983+0000","last_unstale":"2024-09-21T11:00:18.487314+0000","last_undegraded":"2024-09-21T11:00:18.487314+0000","last_fullsized":"2024-09-21T11:00:18.487314+0000","mapping_epoch":1529,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:05.980043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":37,"reported_epoch":1555,"state":"active+clean","last_fresh":"2024-09-21T11:00:07.144609+0000","last_change":"2024-09-21T10:58:21.494272+0000","last_active":"2024-09-21T11:00:07.144609+0000","last_peered":"2024-09-21T11:00:07.144609+0000","last_clean":"2024-09-21T11:00:07.144609+0000","last_became_active":"2024-09-21T10:58:21.493939+0000","last_became_peered":"2024-09-21T10:58:21.493939+0000","last_unstale":"2024-09-21T11:00:07.144609+0000","last_undegraded":"2024-09-21T11:00:07.144609+0000","last_fullsized":"2024-09-21T11:00:07.144609+0000","mapping_epoch":1529,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:47:48.704699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2408,"reported_epoch":1556,"state":"active+undersized","last_fresh":"2024-09-21T11:00:18.487472+0000","last_change":"2024-09-21T10:58:44.731870+0000","last_active":"2024-09-21T11:00:18.487472+0000","last_peered":"2024-09-21T11:00:18.487472+0000","last_clean":"2024-09-21T10:58:41.711032+0000","last_became_active":"2024-09-21T10:58:44.731870+0000","last_became_peered":"2024-09-21T10:58:44.731870+0000","last_unstale":"2024-09-21T11:00:18.487472+0000","last_undegraded":"2024-09-21T11:00:18.487472+0000","last_fullsized":"2024-09-21T10:58:44.714252+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:55:40.433011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2406,"reported_epoch":1556,"state":"active+clean","last_fresh":"2024-09-21T11:00:18.487066+0000","last_change":"2024-09-21T10:58:21.518984+0000","last_active":"2024-09-21T11:00:18.487066+0000","last_peered":"2024-09-21T11:00:18.487066+0000","last_clean":"2024-09-21T11:00:18.487066+0000","last_became_active":"2024-09-21T10:58:21.518436+0000","last_became_peered":"2024-09-21T10:58:21.518436+0000","last_unstale":"2024-09-21T11:00:18.487066+0000","last_undegraded":"2024-09-21T11:00:18.487066+0000","last_fullsized":"2024-09-21T11:00:18.487066+0000","mapping_epoch":1529,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:19:25.745314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":123,"reported_epoch":1556,"state":"active+clean","last_fresh":"2024-09-21T11:00:18.487484+0000","last_change":"2024-09-21T10:58:21.498722+0000","last_active":"2024-09-21T11:00:18.487484+0000","last_peered":"2024-09-21T11:00:18.487484+0000","last_clean":"2024-09-21T11:00:18.487484+0000","last_became_active":"2024-09-21T10:58:21.498370+0000","last_became_peered":"2024-09-21T10:58:21.498370+0000","last_unstale":"2024-09-21T11:00:18.487484+0000","last_undegraded":"2024-09-21T11:00:18.487484+0000","last_fullsized":"2024-09-21T11:00:18.487484+0000","mapping_epoch":1529,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:49:33.751488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2344,"reported_epoch":1556,"state":"active+clean","last_fresh":"2024-09-21T11:00:18.486031+0000","last_change":"2024-09-21T10:58:21.509790+0000","last_active":"2024-09-21T11:00:18.486031+0000","last_peered":"2024-09-21T11:00:18.486031+0000","last_clean":"2024-09-21T11:00:18.486031+0000","last_became_active":"2024-09-21T10:58:21.509419+0000","last_became_peered":"2024-09-21T10:58:21.509419+0000","last_unstale":"2024-09-21T11:00:18.486031+0000","last_undegraded":"2024-09-21T11:00:18.486031+0000","last_fullsized":"2024-09-21T11:00:18.486031+0000","mapping_epoch":1529,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:33:35.355608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1550,"reported_epoch":1556,"state":"active+undersized","last_fresh":"2024-09-21T11:00:18.486983+0000","last_change":"2024-09-21T10:58:44.741842+0000","last_active":"2024-09-21T11:00:18.486983+0000","last_peered":"2024-09-21T11:00:18.486983+0000","last_clean":"2024-09-21T10:56:19.067629+0000","last_became_active":"2024-09-21T10:58:44.741842+0000","last_became_peered":"2024-09-21T10:58:44.741842+0000","last_unstale":"2024-09-21T11:00:18.486983+0000","last_undegraded":"2024-09-21T11:00:18.486983+0000","last_fullsized":"2024-09-21T10:58:44.714923+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:14:20.138349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2342,"reported_epoch":1556,"state":"active+undersized","last_fresh":"2024-09-21T11:00:18.485999+0000","last_change":"2024-09-21T10:58:44.755782+0000","last_active":"2024-09-21T11:00:18.485999+0000","last_peered":"2024-09-21T11:00:18.485999+0000","last_clean":"2024-09-21T10:58:41.709762+0000","last_became_active":"2024-09-21T10:58:44.755782+0000","last_became_peered":"2024-09-21T10:58:44.755782+0000","last_unstale":"2024-09-21T11:00:18.485999+0000","last_undegraded":"2024-09-21T11:00:18.485999+0000","last_fullsized":"2024-09-21T10:58:44.716123+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:10.397563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2273,"reported_epoch":1556,"state":"active+undersized","last_fresh":"2024-09-21T11:00:18.486654+0000","last_change":"2024-09-21T10:58:44.733187+0000","last_active":"2024-09-21T11:00:18.486654+0000","last_peered":"2024-09-21T11:00:18.486654+0000","last_clean":"2024-09-21T10:58:14.890290+0000","last_became_active":"2024-09-21T10:58:44.733187+0000","last_became_peered":"2024-09-21T10:58:44.733187+0000","last_unstale":"2024-09-21T11:00:18.486654+0000","last_undegraded":"2024-09-21T11:00:18.486654+0000","last_fullsized":"2024-09-21T10:58:44.714518+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:04:31.390581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2429,"reported_epoch":1556,"state":"active+undersized","last_fresh":"2024-09-21T11:00:18.476150+0000","last_change":"2024-09-21T10:58:44.716750+0000","last_active":"2024-09-21T11:00:18.476150+0000","last_peered":"2024-09-21T11:00:18.476150+0000","last_clean":"2024-09-21T10:58:41.708409+0000","last_became_active":"2024-09-21T10:58:44.716750+0000","last_became_peered":"2024-09-21T10:58:44.716750+0000","last_unstale":"2024-09-21T11:00:18.476150+0000","last_undegraded":"2024-09-21T11:00:18.476150+0000","last_fullsized":"2024-09-21T10:58:44.710874+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:57:10.925901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":122,"reported_epoch":1555,"state":"active+clean","last_fresh":"2024-09-21T11:00:07.147755+0000","last_change":"2024-09-21T10:58:21.498284+0000","last_active":"2024-09-21T11:00:07.147755+0000","last_peered":"2024-09-21T11:00:07.147755+0000","last_clean":"2024-09-21T11:00:07.147755+0000","last_became_active":"2024-09-21T10:58:21.497922+0000","last_became_peered":"2024-09-21T10:58:21.497922+0000","last_unstale":"2024-09-21T11:00:07.147755+0000","last_undegraded":"2024-09-21T11:00:07.147755+0000","last_fullsized":"2024-09-21T11:00:07.147755+0000","mapping_epoch":1529,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:50:33.970946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":140,"reported_epoch":1555,"state":"active+clean","last_fresh":"2024-09-21T11:00:07.143772+0000","last_change":"2024-09-21T10:58:21.542267+0000","last_active":"2024-09-21T11:00:07.143772+0000","last_peered":"2024-09-21T11:00:07.143772+0000","last_clean":"2024-09-21T11:00:07.143772+0000","last_became_active":"2024-09-21T10:58:21.521275+0000","last_became_peered":"2024-09-21T10:58:21.521275+0000","last_unstale":"2024-09-21T11:00:07.143772+0000","last_undegraded":"2024-09-21T11:00:07.143772+0000","last_fullsized":"2024-09-21T11:00:07.143772+0000","mapping_epoch":1529,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:16:23.438074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2431,"reported_epoch":1556,"state":"active+undersized","last_fresh":"2024-09-21T11:00:18.487528+0000","last_change":"2024-09-21T10:58:44.731487+0000","last_active":"2024-09-21T11:00:18.487528+0000","last_peered":"2024-09-21T11:00:18.487528+0000","last_clean":"2024-09-21T10:58:41.710517+0000","last_became_active":"2024-09-21T10:58:44.731487+0000","last_became_peered":"2024-09-21T10:58:44.731487+0000","last_unstale":"2024-09-21T11:00:18.487528+0000","last_undegraded":"2024-09-21T11:00:18.487528+0000","last_fullsized":"2024-09-21T10:58:44.715329+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:51:30.090288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":53,"reported_epoch":1555,"state":"active+undersized+degraded","last_fresh":"2024-09-21T11:00:07.144955+0000","last_change":"2024-09-21T10:58:44.735644+0000","last_active":"2024-09-21T11:00:07.144955+0000","last_peered":"2024-09-21T11:00:07.144955+0000","last_clean":"2024-09-21T10:58:41.704154+0000","last_became_active":"2024-09-21T10:58:44.735644+0000","last_became_peered":"2024-09-21T10:58:44.735644+0000","last_unstale":"2024-09-21T11:00:07.144955+0000","last_undegraded":"2024-09-21T10:58:44.724499+0000","last_fullsized":"2024-09-21T10:58:44.724382+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:03.215636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14268,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":43,"reported_epoch":1555,"state":"active+undersized","last_fresh":"2024-09-21T11:00:07.165197+0000","last_change":"2024-09-21T10:58:44.754089+0000","last_active":"2024-09-21T11:00:07.165197+0000","last_peered":"2024-09-21T11:00:07.165197+0000","last_clean":"2024-09-21T10:58:41.703812+0000","last_became_active":"2024-09-21T10:58:44.754089+0000","last_became_peered":"2024-09-21T10:58:44.754089+0000","last_unstale":"2024-09-21T11:00:07.165197+0000","last_undegraded":"2024-09-21T11:00:07.165197+0000","last_fullsized":"2024-09-21T10:58:44.719755+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:17:50.800103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"346'149","reported_seq":989,"reported_epoch":1555,"state":"active+undersized","last_fresh":"2024-09-21T11:00:07.141740+0000","last_change":"2024-09-21T10:58:44.735062+0000","last_active":"2024-09-21T11:00:07.141740+0000","last_peered":"2024-09-21T11:00:07.141740+0000","last_clean":"2024-09-21T10:58:41.701860+0000","last_became_active":"2024-09-21T10:58:44.735062+0000","last_became_peered":"2024-09-21T10:58:44.735062+0000","last_unstale":"2024-09-21T11:00:07.141740+0000","last_undegraded":"2024-09-21T11:00:07.141740+0000","last_fullsized":"2024-09-21T10:58:44.714364+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1906749,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":44,"reported_epoch":1556,"state":"active+undersized","last_fresh":"2024-09-21T11:00:18.489338+0000","last_change":"2024-09-21T10:58:44.724354+0000","last_active":"2024-09-21T11:00:18.489338+0000","last_peered":"2024-09-21T11:00:18.489338+0000","last_clean":"2024-09-21T10:58:41.703422+0000","last_became_active":"2024-09-21T10:58:44.724354+0000","last_became_peered":"2024-09-21T10:58:44.724354+0000","last_unstale":"2024-09-21T11:00:18.489338+0000","last_undegraded":"2024-09-21T11:00:18.489338+0000","last_fullsized":"2024-09-21T10:58:44.717123+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:46:48.946573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1565,"reported_epoch":1555,"state":"active+undersized","last_fresh":"2024-09-21T11:00:07.144063+0000","last_change":"2024-09-21T10:58:44.730085+0000","last_active":"2024-09-21T11:00:07.144063+0000","last_peered":"2024-09-21T11:00:07.144063+0000","last_clean":"2024-09-21T10:56:19.067690+0000","last_became_active":"2024-09-21T10:58:44.730085+0000","last_became_peered":"2024-09-21T10:58:44.730085+0000","last_unstale":"2024-09-21T11:00:07.144063+0000","last_undegraded":"2024-09-21T11:00:07.144063+0000","last_fullsized":"2024-09-21T10:58:44.718844+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:17:19.473681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":294,"reported_epoch":1556,"state":"active+clean","last_fresh":"2024-09-21T11:00:18.488061+0000","last_change":"2024-09-21T10:58:21.530763+0000","last_active":"2024-09-21T11:00:18.488061+0000","last_peered":"2024-09-21T11:00:18.488061+0000","last_clean":"2024-09-21T11:00:18.488061+0000","last_became_active":"2024-09-21T10:58:21.509219+0000","last_became_peered":"2024-09-21T10:58:21.509219+0000","last_unstale":"2024-09-21T11:00:18.488061+0000","last_undegraded":"2024-09-21T11:00:18.488061+0000","last_fullsized":"2024-09-21T11:00:18.488061+0000","mapping_epoch":1529,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:23:58.100189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"346'149","reported_seq":697,"reported_epoch":1555,"state":"active+undersized","last_fresh":"2024-09-21T11:00:07.143127+0000","last_change":"2024-09-21T10:58:44.731303+0000","last_active":"2024-09-21T11:00:07.143127+0000","last_peered":"2024-09-21T11:00:07.143127+0000","last_clean":"2024-09-21T10:58:41.703563+0000","last_became_active":"2024-09-21T10:58:44.731303+0000","last_became_peered":"2024-09-21T10:58:44.731303+0000","last_unstale":"2024-09-21T11:00:07.143127+0000","last_undegraded":"2024-09-21T11:00:07.143127+0000","last_fullsized":"2024-09-21T10:58:44.722782+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:22.051403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:22.051403+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2267,"reported_epoch":1556,"state":"active+undersized","last_fresh":"2024-09-21T11:00:18.482265+0000","last_change":"2024-09-21T10:58:44.725359+0000","last_active":"2024-09-21T11:00:18.482265+0000","last_peered":"2024-09-21T11:00:18.482265+0000","last_clean":"2024-09-21T10:58:41.699669+0000","last_became_active":"2024-09-21T10:58:44.725359+0000","last_became_peered":"2024-09-21T10:58:44.725359+0000","last_unstale":"2024-09-21T11:00:18.482265+0000","last_undegraded":"2024-09-21T11:00:18.482265+0000","last_fullsized":"2024-09-21T10:58:44.720129+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:09:05.546754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2412,"reported_epoch":1556,"state":"active+undersized","last_fresh":"2024-09-21T11:00:18.488291+0000","last_change":"2024-09-21T10:58:44.734195+0000","last_active":"2024-09-21T11:00:18.488291+0000","last_peered":"2024-09-21T11:00:18.488291+0000","last_clean":"2024-09-21T10:58:41.711401+0000","last_became_active":"2024-09-21T10:58:44.734195+0000","last_became_peered":"2024-09-21T10:58:44.734195+0000","last_unstale":"2024-09-21T11:00:18.488291+0000","last_undegraded":"2024-09-21T11:00:18.488291+0000","last_fullsized":"2024-09-21T10:58:44.715527+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:55:41.904047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":2465,"reported_epoch":1556,"state":"active+clean","last_fresh":"2024-09-21T11:00:18.476556+0000","last_change":"2024-09-21T10:58:21.532719+0000","last_active":"2024-09-21T11:00:18.476556+0000","last_peered":"2024-09-21T11:00:18.476556+0000","last_clean":"2024-09-21T11:00:18.476556+0000","last_became_active":"2024-09-21T10:58:21.508760+0000","last_became_peered":"2024-09-21T10:58:21.508760+0000","last_unstale":"2024-09-21T11:00:18.476556+0000","last_undegraded":"2024-09-21T11:00:18.476556+0000","last_fullsized":"2024-09-21T11:00:18.476556+0000","mapping_epoch":1529,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:18.937785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"346'149","reported_seq":840,"reported_epoch":1555,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:00:07.142919+0000","last_change":"2024-09-21T10:58:45.851409+0000","last_active":"2024-09-21T11:00:07.142919+0000","last_peered":"2024-09-21T11:00:07.142919+0000","last_clean":"2024-09-21T11:00:07.142919+0000","last_became_active":"2024-09-21T10:58:45.851001+0000","last_became_peered":"2024-09-21T10:58:45.851001+0000","last_unstale":"2024-09-21T11:00:07.142919+0000","last_undegraded":"2024-09-21T11:00:07.142919+0000","last_fullsized":"2024-09-21T11:00:07.142919+0000","mapping_epoch":1542,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1543,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:41.011085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:41.011085+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:10.580358+0000","scrub_duration":0.0061250310000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7,1],"avail_no_missing":["4","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2293,"reported_epoch":1555,"state":"active+clean","last_fresh":"2024-09-21T11:00:07.143574+0000","last_change":"2024-09-21T10:58:21.504753+0000","last_active":"2024-09-21T11:00:07.143574+0000","last_peered":"2024-09-21T11:00:07.143574+0000","last_clean":"2024-09-21T11:00:07.143574+0000","last_became_active":"2024-09-21T10:58:21.504445+0000","last_became_peered":"2024-09-21T10:58:21.504445+0000","last_unstale":"2024-09-21T11:00:07.143574+0000","last_undegraded":"2024-09-21T11:00:07.143574+0000","last_fullsized":"2024-09-21T11:00:07.143574+0000","mapping_epoch":1529,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:35:28.090630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":122,"reported_epoch":1555,"state":"active+clean","last_fresh":"2024-09-21T11:00:07.143654+0000","last_change":"2024-09-21T10:58:21.519613+0000","last_active":"2024-09-21T11:00:07.143654+0000","last_peered":"2024-09-21T11:00:07.143654+0000","last_clean":"2024-09-21T11:00:07.143654+0000","last_became_active":"2024-09-21T10:58:21.519300+0000","last_became_peered":"2024-09-21T10:58:21.519300+0000","last_unstale":"2024-09-21T11:00:07.143654+0000","last_undegraded":"2024-09-21T11:00:07.143654+0000","last_fullsized":"2024-09-21T11:00:07.143654+0000","mapping_epoch":1529,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:30:38.454563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2462,"reported_epoch":1556,"state":"active+undersized","last_fresh":"2024-09-21T11:00:18.487093+0000","last_change":"2024-09-21T10:58:44.736707+0000","last_active":"2024-09-21T11:00:18.487093+0000","last_peered":"2024-09-21T11:00:18.487093+0000","last_clean":"2024-09-21T10:58:41.710859+0000","last_became_active":"2024-09-21T10:58:44.736707+0000","last_became_peered":"2024-09-21T10:58:44.736707+0000","last_unstale":"2024-09-21T11:00:18.487093+0000","last_undegraded":"2024-09-21T11:00:18.487093+0000","last_fullsized":"2024-09-21T10:58:44.715739+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:59:33.129376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"346'149","reported_seq":761,"reported_epoch":1555,"state":"active+undersized","last_fresh":"2024-09-21T11:00:07.143935+0000","last_change":"2024-09-21T10:58:44.729467+0000","last_active":"2024-09-21T11:00:07.143935+0000","last_peered":"2024-09-21T11:00:07.143935+0000","last_clean":"2024-09-21T10:52:16.390617+0000","last_became_active":"2024-09-21T10:58:44.729467+0000","last_became_peered":"2024-09-21T10:58:44.729467+0000","last_unstale":"2024-09-21T11:00:07.143935+0000","last_undegraded":"2024-09-21T11:00:07.143935+0000","last_fullsized":"2024-09-21T10:58:44.718172+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1511,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:44.018645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:44.018645+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:25:43.141373+0000","scrub_duration":0.0012789399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2413,"reported_epoch":1556,"state":"active+clean","last_fresh":"2024-09-21T11:00:18.487351+0000","last_change":"2024-09-21T10:58:21.509504+0000","last_active":"2024-09-21T11:00:18.487351+0000","last_peered":"2024-09-21T11:00:18.487351+0000","last_clean":"2024-09-21T11:00:18.487351+0000","last_became_active":"2024-09-21T10:58:21.508149+0000","last_became_peered":"2024-09-21T10:58:21.508149+0000","last_unstale":"2024-09-21T11:00:18.487351+0000","last_undegraded":"2024-09-21T11:00:18.487351+0000","last_fullsized":"2024-09-21T11:00:18.487351+0000","mapping_epoch":1529,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:15:27.316337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2481,"reported_epoch":1556,"state":"active+clean","last_fresh":"2024-09-21T11:00:18.486877+0000","last_change":"2024-09-21T10:58:21.509350+0000","last_active":"2024-09-21T11:00:18.486877+0000","last_peered":"2024-09-21T11:00:18.486877+0000","last_clean":"2024-09-21T11:00:18.486877+0000","last_became_active":"2024-09-21T10:58:21.508670+0000","last_became_peered":"2024-09-21T10:58:21.508670+0000","last_unstale":"2024-09-21T11:00:18.486877+0000","last_undegraded":"2024-09-21T11:00:18.486877+0000","last_fullsized":"2024-09-21T11:00:18.486877+0000","mapping_epoch":1529,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:50:07.944177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1453,"reported_epoch":1555,"state":"active+undersized","last_fresh":"2024-09-21T11:00:07.143537+0000","last_change":"2024-09-21T10:58:44.727795+0000","last_active":"2024-09-21T11:00:07.143537+0000","last_peered":"2024-09-21T11:00:07.143537+0000","last_clean":"2024-09-21T10:52:16.391715+0000","last_became_active":"2024-09-21T10:58:44.727795+0000","last_became_peered":"2024-09-21T10:58:44.727795+0000","last_unstale":"2024-09-21T11:00:07.143537+0000","last_undegraded":"2024-09-21T11:00:07.143537+0000","last_fullsized":"2024-09-21T10:58:44.721466+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:46:04.781703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"346'149","reported_seq":842,"reported_epoch":1556,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:00:18.489474+0000","last_change":"2024-09-21T10:58:45.730097+0000","last_active":"2024-09-21T11:00:18.489474+0000","last_peered":"2024-09-21T11:00:18.489474+0000","last_clean":"2024-09-21T11:00:18.489474+0000","last_became_active":"2024-09-21T10:58:45.729634+0000","last_became_peered":"2024-09-21T10:58:45.729634+0000","last_unstale":"2024-09-21T11:00:18.489474+0000","last_undegraded":"2024-09-21T11:00:18.489474+0000","last_fullsized":"2024-09-21T11:00:18.489474+0000","mapping_epoch":1542,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1543,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:16.021843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:16.021843+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7,1],"avail_no_missing":["4","1","7"],"object_location_counts":[{"shards":"1,4,7","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2479,"reported_epoch":1556,"state":"active+clean","last_fresh":"2024-09-21T11:00:18.487287+0000","last_change":"2024-09-21T10:59:05.442601+0000","last_active":"2024-09-21T11:00:18.487287+0000","last_peered":"2024-09-21T11:00:18.487287+0000","last_clean":"2024-09-21T11:00:18.487287+0000","last_became_active":"2024-09-21T10:59:05.442310+0000","last_became_peered":"2024-09-21T10:59:05.442310+0000","last_unstale":"2024-09-21T11:00:18.487287+0000","last_undegraded":"2024-09-21T11:00:18.487287+0000","last_fullsized":"2024-09-21T11:00:18.487287+0000","mapping_epoch":1547,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:14.549258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":44,"reported_epoch":1556,"state":"active+undersized","last_fresh":"2024-09-21T11:00:18.508026+0000","last_change":"2024-09-21T10:58:44.745957+0000","last_active":"2024-09-21T11:00:18.508026+0000","last_peered":"2024-09-21T11:00:18.508026+0000","last_clean":"2024-09-21T10:58:41.703972+0000","last_became_active":"2024-09-21T10:58:44.745957+0000","last_became_peered":"2024-09-21T10:58:44.745957+0000","last_unstale":"2024-09-21T11:00:18.508026+0000","last_undegraded":"2024-09-21T11:00:18.508026+0000","last_fullsized":"2024-09-21T10:58:44.719466+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:06:12.336050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":376,"reported_epoch":1556,"state":"active+undersized+degraded","last_fresh":"2024-09-21T11:00:18.487095+0000","last_change":"2024-09-21T10:58:44.730985+0000","last_active":"2024-09-21T11:00:18.487095+0000","last_peered":"2024-09-21T11:00:18.487095+0000","last_clean":"2024-09-21T10:58:41.710261+0000","last_became_active":"2024-09-21T10:58:44.730985+0000","last_became_peered":"2024-09-21T10:58:44.730985+0000","last_unstale":"2024-09-21T11:00:18.487095+0000","last_undegraded":"2024-09-21T10:58:44.711579+0000","last_fullsized":"2024-09-21T10:58:44.711482+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:09:49.337655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"346'149","reported_seq":935,"reported_epoch":1555,"state":"active+clean","last_fresh":"2024-09-21T11:00:07.143169+0000","last_change":"2024-09-21T10:58:24.417753+0000","last_active":"2024-09-21T11:00:07.143169+0000","last_peered":"2024-09-21T11:00:07.143169+0000","last_clean":"2024-09-21T11:00:07.143169+0000","last_became_active":"2024-09-21T10:58:24.417349+0000","last_became_peered":"2024-09-21T10:58:24.417349+0000","last_unstale":"2024-09-21T11:00:07.143169+0000","last_undegraded":"2024-09-21T11:00:07.143169+0000","last_fullsized":"2024-09-21T11:00:07.143169+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:21.445783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:21.445783+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:29:40.046636+0000","scrub_duration":0.0018624990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":156,"reported_epoch":1556,"state":"active+clean","last_fresh":"2024-09-21T11:00:18.486552+0000","last_change":"2024-09-21T10:58:21.509483+0000","last_active":"2024-09-21T11:00:18.486552+0000","last_peered":"2024-09-21T11:00:18.486552+0000","last_clean":"2024-09-21T11:00:18.486552+0000","last_became_active":"2024-09-21T10:58:21.509105+0000","last_became_peered":"2024-09-21T10:58:21.509105+0000","last_unstale":"2024-09-21T11:00:18.486552+0000","last_undegraded":"2024-09-21T11:00:18.486552+0000","last_fullsized":"2024-09-21T11:00:18.486552+0000","mapping_epoch":1529,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:06:12.138826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2429,"reported_epoch":1556,"state":"active+undersized","last_fresh":"2024-09-21T11:00:18.486133+0000","last_change":"2024-09-21T10:58:44.741984+0000","last_active":"2024-09-21T11:00:18.486133+0000","last_peered":"2024-09-21T11:00:18.486133+0000","last_clean":"2024-09-21T10:58:41.709959+0000","last_became_active":"2024-09-21T10:58:44.741984+0000","last_became_peered":"2024-09-21T10:58:44.741984+0000","last_unstale":"2024-09-21T11:00:18.486133+0000","last_undegraded":"2024-09-21T11:00:18.486133+0000","last_fullsized":"2024-09-21T10:58:44.716350+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:50:18.501401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1094,"reported_epoch":1556,"state":"active+undersized","last_fresh":"2024-09-21T11:00:18.486891+0000","last_change":"2024-09-21T10:58:44.731671+0000","last_active":"2024-09-21T11:00:18.486891+0000","last_peered":"2024-09-21T11:00:18.486891+0000","last_clean":"2024-09-21T10:58:41.709991+0000","last_became_active":"2024-09-21T10:58:44.731671+0000","last_became_peered":"2024-09-21T10:58:44.731671+0000","last_unstale":"2024-09-21T11:00:18.486891+0000","last_undegraded":"2024-09-21T11:00:18.486891+0000","last_fullsized":"2024-09-21T10:58:44.712583+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:06:03.143774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"346'149","reported_seq":1115,"reported_epoch":1556,"state":"active+clean","last_fresh":"2024-09-21T11:00:18.486088+0000","last_change":"2024-09-21T10:59:05.439244+0000","last_active":"2024-09-21T11:00:18.486088+0000","last_peered":"2024-09-21T11:00:18.486088+0000","last_clean":"2024-09-21T11:00:18.486088+0000","last_became_active":"2024-09-21T10:59:05.438933+0000","last_became_peered":"2024-09-21T10:59:05.438933+0000","last_unstale":"2024-09-21T11:00:18.486088+0000","last_undegraded":"2024-09-21T11:00:18.486088+0000","last_fullsized":"2024-09-21T11:00:18.486088+0000","mapping_epoch":1547,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1548,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2457,"reported_epoch":1556,"state":"active+clean","last_fresh":"2024-09-21T11:00:18.487983+0000","last_change":"2024-09-21T10:58:21.509743+0000","last_active":"2024-09-21T11:00:18.487983+0000","last_peered":"2024-09-21T11:00:18.487983+0000","last_clean":"2024-09-21T11:00:18.487983+0000","last_became_active":"2024-09-21T10:58:21.508238+0000","last_became_peered":"2024-09-21T10:58:21.508238+0000","last_unstale":"2024-09-21T11:00:18.487983+0000","last_undegraded":"2024-09-21T11:00:18.487983+0000","last_fullsized":"2024-09-21T11:00:18.487983+0000","mapping_epoch":1529,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:53.366930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":38,"reported_epoch":1556,"state":"active+clean","last_fresh":"2024-09-21T11:00:18.489739+0000","last_change":"2024-09-21T10:58:21.518293+0000","last_active":"2024-09-21T11:00:18.489739+0000","last_peered":"2024-09-21T11:00:18.489739+0000","last_clean":"2024-09-21T11:00:18.489739+0000","last_became_active":"2024-09-21T10:58:21.515386+0000","last_became_peered":"2024-09-21T10:58:21.515386+0000","last_unstale":"2024-09-21T11:00:18.489739+0000","last_undegraded":"2024-09-21T11:00:18.489739+0000","last_fullsized":"2024-09-21T11:00:18.489739+0000","mapping_epoch":1529,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:08:35.368413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":214,"reported_epoch":1555,"state":"active+clean","last_fresh":"2024-09-21T11:00:17.465075+0000","last_change":"2024-09-21T10:51:27.426006+0000","last_active":"2024-09-21T11:00:17.465075+0000","last_peered":"2024-09-21T11:00:17.465075+0000","last_clean":"2024-09-21T11:00:17.465075+0000","last_became_active":"2024-09-21T10:51:27.425698+0000","last_became_peered":"2024-09-21T10:51:27.425698+0000","last_unstale":"2024-09-21T11:00:17.465075+0000","last_undegraded":"2024-09-21T11:00:17.465075+0000","last_fullsized":"2024-09-21T11:00:17.465075+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:35:13.210330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":121,"reported_epoch":1555,"state":"active+undersized","last_fresh":"2024-09-21T11:00:07.150162+0000","last_change":"2024-09-21T10:58:44.750614+0000","last_active":"2024-09-21T11:00:07.150162+0000","last_peered":"2024-09-21T11:00:07.150162+0000","last_clean":"2024-09-21T10:58:14.889597+0000","last_became_active":"2024-09-21T10:58:44.750614+0000","last_became_peered":"2024-09-21T10:58:44.750614+0000","last_unstale":"2024-09-21T11:00:07.150162+0000","last_undegraded":"2024-09-21T11:00:07.150162+0000","last_fullsized":"2024-09-21T10:58:44.726764+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:24.707888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2321,"reported_epoch":1555,"state":"active+undersized","last_fresh":"2024-09-21T11:00:07.144061+0000","last_change":"2024-09-21T10:58:44.748512+0000","last_active":"2024-09-21T11:00:07.144061+0000","last_peered":"2024-09-21T11:00:07.144061+0000","last_clean":"2024-09-21T10:58:14.889042+0000","last_became_active":"2024-09-21T10:58:44.748512+0000","last_became_peered":"2024-09-21T10:58:44.748512+0000","last_unstale":"2024-09-21T11:00:07.144061+0000","last_undegraded":"2024-09-21T11:00:07.144061+0000","last_fullsized":"2024-09-21T10:58:44.715769+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:41:44.453733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":1544,"reported_epoch":1555,"state":"active+undersized+degraded","last_fresh":"2024-09-21T11:00:07.143673+0000","last_change":"2024-09-21T10:58:44.722515+0000","last_active":"2024-09-21T11:00:07.143673+0000","last_peered":"2024-09-21T11:00:07.143673+0000","last_clean":"2024-09-21T10:58:42.407038+0000","last_became_active":"2024-09-21T10:58:44.722515+0000","last_became_peered":"2024-09-21T10:58:44.722515+0000","last_unstale":"2024-09-21T11:00:07.143673+0000","last_undegraded":"2024-09-21T10:58:44.712515+0000","last_fullsized":"2024-09-21T10:58:44.712401+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:10.249611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":890,"reported_epoch":1555,"state":"active+undersized","last_fresh":"2024-09-21T11:00:07.142573+0000","last_change":"2024-09-21T10:58:44.725921+0000","last_active":"2024-09-21T11:00:07.142573+0000","last_peered":"2024-09-21T11:00:07.142573+0000","last_clean":"2024-09-21T10:58:41.699976+0000","last_became_active":"2024-09-21T10:58:44.725921+0000","last_became_peered":"2024-09-21T10:58:44.725921+0000","last_unstale":"2024-09-21T11:00:07.142573+0000","last_undegraded":"2024-09-21T11:00:07.142573+0000","last_fullsized":"2024-09-21T10:58:44.720891+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:20.458421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:20.458421+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:10:27.439795+0000","scrub_duration":0.001528569,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"346'149","reported_seq":853,"reported_epoch":1555,"state":"active+undersized","last_fresh":"2024-09-21T11:00:07.143924+0000","last_change":"2024-09-21T10:58:44.747952+0000","last_active":"2024-09-21T11:00:07.143924+0000","last_peered":"2024-09-21T11:00:07.143924+0000","last_clean":"2024-09-21T10:58:17.129670+0000","last_became_active":"2024-09-21T10:58:44.747952+0000","last_became_peered":"2024-09-21T10:58:44.747952+0000","last_unstale":"2024-09-21T11:00:07.143924+0000","last_undegraded":"2024-09-21T11:00:07.143924+0000","last_fullsized":"2024-09-21T10:58:44.717716+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:42.008235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:42.008235+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:31:22.409860+0000","scrub_duration":0.011129834999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2408,"reported_epoch":1556,"state":"active+clean","last_fresh":"2024-09-21T11:00:18.486898+0000","last_change":"2024-09-21T10:58:21.510001+0000","last_active":"2024-09-21T11:00:18.486898+0000","last_peered":"2024-09-21T11:00:18.486898+0000","last_clean":"2024-09-21T11:00:18.486898+0000","last_became_active":"2024-09-21T10:58:21.509355+0000","last_became_peered":"2024-09-21T10:58:21.509355+0000","last_unstale":"2024-09-21T11:00:18.486898+0000","last_undegraded":"2024-09-21T11:00:18.486898+0000","last_fullsized":"2024-09-21T11:00:18.486898+0000","mapping_epoch":1529,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:46:28.865330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2316,"reported_epoch":1556,"state":"active+clean","last_fresh":"2024-09-21T11:00:18.486162+0000","last_change":"2024-09-21T10:58:21.515851+0000","last_active":"2024-09-21T11:00:18.486162+0000","last_peered":"2024-09-21T11:00:18.486162+0000","last_clean":"2024-09-21T11:00:18.486162+0000","last_became_active":"2024-09-21T10:58:21.515509+0000","last_became_peered":"2024-09-21T10:58:21.515509+0000","last_unstale":"2024-09-21T11:00:18.486162+0000","last_undegraded":"2024-09-21T11:00:18.486162+0000","last_fullsized":"2024-09-21T11:00:18.486162+0000","mapping_epoch":1529,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:40:44.164018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":43,"reported_epoch":1555,"state":"active+undersized","last_fresh":"2024-09-21T11:00:07.143510+0000","last_change":"2024-09-21T10:58:44.749385+0000","last_active":"2024-09-21T11:00:07.143510+0000","last_peered":"2024-09-21T11:00:07.143510+0000","last_clean":"2024-09-21T10:58:41.703967+0000","last_became_active":"2024-09-21T10:58:44.749385+0000","last_became_peered":"2024-09-21T10:58:44.749385+0000","last_unstale":"2024-09-21T11:00:07.143510+0000","last_undegraded":"2024-09-21T11:00:07.143510+0000","last_fullsized":"2024-09-21T10:58:44.720963+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:59:48.005300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":37,"reported_epoch":1555,"state":"active+clean","last_fresh":"2024-09-21T11:00:07.143530+0000","last_change":"2024-09-21T10:58:21.519569+0000","last_active":"2024-09-21T11:00:07.143530+0000","last_peered":"2024-09-21T11:00:07.143530+0000","last_clean":"2024-09-21T11:00:07.143530+0000","last_became_active":"2024-09-21T10:58:21.517976+0000","last_became_peered":"2024-09-21T10:58:21.517976+0000","last_unstale":"2024-09-21T11:00:07.143530+0000","last_undegraded":"2024-09-21T11:00:07.143530+0000","last_fullsized":"2024-09-21T11:00:07.143530+0000","mapping_epoch":1529,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:42.613119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"346'149","reported_seq":688,"reported_epoch":1555,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-21T11:00:07.141619+0000","last_change":"2024-09-21T10:58:45.841172+0000","last_active":"2024-09-21T11:00:07.141619+0000","last_peered":"2024-09-21T11:00:07.141619+0000","last_clean":"2024-09-21T10:54:46.424953+0000","last_became_active":"2024-09-21T10:58:45.840860+0000","last_became_peered":"2024-09-21T10:58:45.840860+0000","last_unstale":"2024-09-21T11:00:07.141619+0000","last_undegraded":"2024-09-21T11:00:07.141619+0000","last_fullsized":"2024-09-21T11:00:07.141619+0000","mapping_epoch":1542,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1514,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":793725,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,4,7","objects":-1},{"shards":"1,7","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":43,"reported_epoch":1555,"state":"active+undersized","last_fresh":"2024-09-21T11:00:07.150705+0000","last_change":"2024-09-21T10:58:44.747682+0000","last_active":"2024-09-21T11:00:07.150705+0000","last_peered":"2024-09-21T11:00:07.150705+0000","last_clean":"2024-09-21T10:58:41.703478+0000","last_became_active":"2024-09-21T10:58:44.747682+0000","last_became_peered":"2024-09-21T10:58:44.747682+0000","last_unstale":"2024-09-21T11:00:07.150705+0000","last_undegraded":"2024-09-21T11:00:07.150705+0000","last_fullsized":"2024-09-21T10:58:44.721229+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:34:15.604181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1540,"reported_epoch":1555,"state":"active+undersized","last_fresh":"2024-09-21T11:00:07.144170+0000","last_change":"2024-09-21T10:58:44.748950+0000","last_active":"2024-09-21T11:00:07.144170+0000","last_peered":"2024-09-21T11:00:07.144170+0000","last_clean":"2024-09-21T10:58:14.889149+0000","last_became_active":"2024-09-21T10:58:44.748950+0000","last_became_peered":"2024-09-21T10:58:44.748950+0000","last_unstale":"2024-09-21T11:00:07.144170+0000","last_undegraded":"2024-09-21T11:00:07.144170+0000","last_fullsized":"2024-09-21T10:58:44.719725+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:22:21.382872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"346'149","reported_seq":1016,"reported_epoch":1556,"state":"active+undersized","last_fresh":"2024-09-21T11:00:18.482212+0000","last_change":"2024-09-21T10:58:44.720907+0000","last_active":"2024-09-21T11:00:18.482212+0000","last_peered":"2024-09-21T11:00:18.482212+0000","last_clean":"2024-09-21T10:58:41.701314+0000","last_became_active":"2024-09-21T10:58:44.720907+0000","last_became_peered":"2024-09-21T10:58:44.720907+0000","last_unstale":"2024-09-21T11:00:18.482212+0000","last_undegraded":"2024-09-21T11:00:18.482212+0000","last_fullsized":"2024-09-21T10:58:44.712364+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:17.418646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:17.418646+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:33.791729+0000","scrub_duration":0.00185648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":43,"reported_epoch":1555,"state":"active+undersized","last_fresh":"2024-09-21T11:00:07.145194+0000","last_change":"2024-09-21T10:58:44.731307+0000","last_active":"2024-09-21T11:00:07.145194+0000","last_peered":"2024-09-21T11:00:07.145194+0000","last_clean":"2024-09-21T10:58:41.702754+0000","last_became_active":"2024-09-21T10:58:44.731307+0000","last_became_peered":"2024-09-21T10:58:44.731307+0000","last_unstale":"2024-09-21T11:00:07.145194+0000","last_undegraded":"2024-09-21T11:00:07.145194+0000","last_fullsized":"2024-09-21T10:58:44.716966+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:09:47.121689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":41,"reported_epoch":1555,"state":"active+clean","last_fresh":"2024-09-21T11:00:07.165279+0000","last_change":"2024-09-21T10:58:21.515138+0000","last_active":"2024-09-21T11:00:07.165279+0000","last_peered":"2024-09-21T11:00:07.165279+0000","last_clean":"2024-09-21T11:00:07.165279+0000","last_became_active":"2024-09-21T10:58:21.498250+0000","last_became_peered":"2024-09-21T10:58:21.498250+0000","last_unstale":"2024-09-21T11:00:07.165279+0000","last_undegraded":"2024-09-21T11:00:07.165279+0000","last_fullsized":"2024-09-21T11:00:07.165279+0000","mapping_epoch":1529,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:22:33.537236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":876,"reported_epoch":1556,"state":"active+undersized","last_fresh":"2024-09-21T11:00:18.486291+0000","last_change":"2024-09-21T10:58:44.731851+0000","last_active":"2024-09-21T11:00:18.486291+0000","last_peered":"2024-09-21T11:00:18.486291+0000","last_clean":"2024-09-21T10:58:41.710587+0000","last_became_active":"2024-09-21T10:58:44.731851+0000","last_became_peered":"2024-09-21T10:58:44.731851+0000","last_unstale":"2024-09-21T11:00:18.486291+0000","last_undegraded":"2024-09-21T11:00:18.486291+0000","last_fullsized":"2024-09-21T10:58:44.714426+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:49:02.476636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":758,"reported_epoch":1556,"state":"active+undersized","last_fresh":"2024-09-21T11:00:18.486704+0000","last_change":"2024-09-21T10:58:44.735629+0000","last_active":"2024-09-21T11:00:18.486704+0000","last_peered":"2024-09-21T11:00:18.486704+0000","last_clean":"2024-09-21T10:58:14.889657+0000","last_became_active":"2024-09-21T10:58:44.735629+0000","last_became_peered":"2024-09-21T10:58:44.735629+0000","last_unstale":"2024-09-21T11:00:18.486704+0000","last_undegraded":"2024-09-21T11:00:18.486704+0000","last_fullsized":"2024-09-21T10:58:44.715014+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:59:10.278777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1542,"reported_epoch":1556,"state":"active+undersized","last_fresh":"2024-09-21T11:00:18.486838+0000","last_change":"2024-09-21T10:58:44.730587+0000","last_active":"2024-09-21T11:00:18.486838+0000","last_peered":"2024-09-21T11:00:18.486838+0000","last_clean":"2024-09-21T10:52:16.682244+0000","last_became_active":"2024-09-21T10:58:44.730587+0000","last_became_peered":"2024-09-21T10:58:44.730587+0000","last_unstale":"2024-09-21T11:00:18.486838+0000","last_undegraded":"2024-09-21T11:00:18.486838+0000","last_fullsized":"2024-09-21T10:58:44.711142+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:24:31.440715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":253,"reported_epoch":1555,"state":"active+undersized+degraded","last_fresh":"2024-09-21T11:00:07.145203+0000","last_change":"2024-09-21T10:58:44.736308+0000","last_active":"2024-09-21T11:00:07.145203+0000","last_peered":"2024-09-21T11:00:07.145203+0000","last_clean":"2024-09-21T10:58:41.702325+0000","last_became_active":"2024-09-21T10:58:44.736308+0000","last_became_peered":"2024-09-21T10:58:44.736308+0000","last_unstale":"2024-09-21T11:00:07.145203+0000","last_undegraded":"2024-09-21T10:58:44.722474+0000","last_fullsized":"2024-09-21T10:58:44.722374+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:32:54.343333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":43,"reported_epoch":1555,"state":"active+undersized","last_fresh":"2024-09-21T11:00:07.143332+0000","last_change":"2024-09-21T10:58:44.741250+0000","last_active":"2024-09-21T11:00:07.143332+0000","last_peered":"2024-09-21T11:00:07.143332+0000","last_clean":"2024-09-21T10:58:41.703506+0000","last_became_active":"2024-09-21T10:58:44.741250+0000","last_became_peered":"2024-09-21T10:58:44.741250+0000","last_unstale":"2024-09-21T11:00:07.143332+0000","last_undegraded":"2024-09-21T11:00:07.143332+0000","last_fullsized":"2024-09-21T10:58:44.717644+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:50:14.596913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":147,"reported_epoch":1555,"state":"active+clean","last_fresh":"2024-09-21T11:00:07.147666+0000","last_change":"2024-09-21T10:58:21.533815+0000","last_active":"2024-09-21T11:00:07.147666+0000","last_peered":"2024-09-21T11:00:07.147666+0000","last_clean":"2024-09-21T11:00:07.147666+0000","last_became_active":"2024-09-21T10:58:21.504244+0000","last_became_peered":"2024-09-21T10:58:21.504244+0000","last_unstale":"2024-09-21T11:00:07.147666+0000","last_undegraded":"2024-09-21T11:00:07.147666+0000","last_fullsized":"2024-09-21T11:00:07.147666+0000","mapping_epoch":1529,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:27.734804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":137,"reported_epoch":1555,"state":"active+undersized","last_fresh":"2024-09-21T11:00:07.142444+0000","last_change":"2024-09-21T10:58:44.759025+0000","last_active":"2024-09-21T11:00:07.142444+0000","last_peered":"2024-09-21T11:00:07.142444+0000","last_clean":"2024-09-21T10:58:41.699865+0000","last_became_active":"2024-09-21T10:58:44.759025+0000","last_became_peered":"2024-09-21T10:58:44.759025+0000","last_unstale":"2024-09-21T11:00:07.142444+0000","last_undegraded":"2024-09-21T11:00:07.142444+0000","last_fullsized":"2024-09-21T10:58:44.720582+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:42:26.700889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":147,"reported_epoch":1555,"state":"active+clean","last_fresh":"2024-09-21T11:00:07.144235+0000","last_change":"2024-09-21T10:58:21.536591+0000","last_active":"2024-09-21T11:00:07.144235+0000","last_peered":"2024-09-21T11:00:07.144235+0000","last_clean":"2024-09-21T11:00:07.144235+0000","last_became_active":"2024-09-21T10:58:21.507725+0000","last_became_peered":"2024-09-21T10:58:21.507725+0000","last_unstale":"2024-09-21T11:00:07.144235+0000","last_undegraded":"2024-09-21T11:00:07.144235+0000","last_fullsized":"2024-09-21T11:00:07.144235+0000","mapping_epoch":1529,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:19:24.147762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1486,"reported_epoch":1556,"state":"active+undersized+degraded","last_fresh":"2024-09-21T11:00:18.485718+0000","last_change":"2024-09-21T10:58:44.742011+0000","last_active":"2024-09-21T11:00:18.485718+0000","last_peered":"2024-09-21T11:00:18.485718+0000","last_clean":"2024-09-21T10:51:09.456376+0000","last_became_active":"2024-09-21T10:58:44.742011+0000","last_became_peered":"2024-09-21T10:58:44.742011+0000","last_unstale":"2024-09-21T11:00:18.485718+0000","last_undegraded":"2024-09-21T10:58:44.715934+0000","last_fullsized":"2024-09-21T10:58:44.715831+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:58:03.721963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2250,"reported_epoch":1555,"state":"active+undersized","last_fresh":"2024-09-21T11:00:07.143943+0000","last_change":"2024-09-21T10:58:44.729690+0000","last_active":"2024-09-21T11:00:07.143943+0000","last_peered":"2024-09-21T11:00:07.143943+0000","last_clean":"2024-09-21T10:58:41.702422+0000","last_became_active":"2024-09-21T10:58:44.729690+0000","last_became_peered":"2024-09-21T10:58:44.729690+0000","last_unstale":"2024-09-21T11:00:07.143943+0000","last_undegraded":"2024-09-21T11:00:07.143943+0000","last_fullsized":"2024-09-21T10:58:44.718494+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:00:08.462052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":236,"reported_epoch":1555,"state":"active+undersized","last_fresh":"2024-09-21T11:00:07.142588+0000","last_change":"2024-09-21T10:58:44.725123+0000","last_active":"2024-09-21T11:00:07.142588+0000","last_peered":"2024-09-21T11:00:07.142588+0000","last_clean":"2024-09-21T10:58:41.701447+0000","last_became_active":"2024-09-21T10:58:44.725123+0000","last_became_peered":"2024-09-21T10:58:44.725123+0000","last_unstale":"2024-09-21T11:00:07.142588+0000","last_undegraded":"2024-09-21T11:00:07.142588+0000","last_fullsized":"2024-09-21T10:58:44.713831+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:51:14.337459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2298,"reported_epoch":1555,"state":"active+undersized","last_fresh":"2024-09-21T11:00:07.147432+0000","last_change":"2024-09-21T10:58:44.722341+0000","last_active":"2024-09-21T11:00:07.147432+0000","last_peered":"2024-09-21T11:00:07.147432+0000","last_clean":"2024-09-21T10:58:41.702957+0000","last_became_active":"2024-09-21T10:58:44.722341+0000","last_became_peered":"2024-09-21T10:58:44.722341+0000","last_unstale":"2024-09-21T11:00:07.147432+0000","last_undegraded":"2024-09-21T11:00:07.147432+0000","last_fullsized":"2024-09-21T10:58:44.717994+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:18:45.908751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":119,"reported_epoch":1555,"state":"active+undersized","last_fresh":"2024-09-21T11:00:07.143013+0000","last_change":"2024-09-21T10:58:44.727977+0000","last_active":"2024-09-21T11:00:07.143013+0000","last_peered":"2024-09-21T11:00:07.143013+0000","last_clean":"2024-09-21T10:58:41.702488+0000","last_became_active":"2024-09-21T10:58:44.727977+0000","last_became_peered":"2024-09-21T10:58:44.727977+0000","last_unstale":"2024-09-21T11:00:07.143013+0000","last_undegraded":"2024-09-21T11:00:07.143013+0000","last_fullsized":"2024-09-21T10:58:44.712433+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:56:17.977304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1524,"reported_epoch":1555,"state":"active+undersized","last_fresh":"2024-09-21T11:00:07.145071+0000","last_change":"2024-09-21T10:58:44.725203+0000","last_active":"2024-09-21T11:00:07.145071+0000","last_peered":"2024-09-21T11:00:07.145071+0000","last_clean":"2024-09-21T10:58:41.702243+0000","last_became_active":"2024-09-21T10:58:44.725203+0000","last_became_peered":"2024-09-21T10:58:44.725203+0000","last_unstale":"2024-09-21T11:00:07.145071+0000","last_undegraded":"2024-09-21T11:00:07.145071+0000","last_fullsized":"2024-09-21T10:58:44.712399+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:51:11.642818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":245,"reported_epoch":1555,"state":"active+undersized+degraded","last_fresh":"2024-09-21T11:00:07.143824+0000","last_change":"2024-09-21T10:58:44.734838+0000","last_active":"2024-09-21T11:00:07.143824+0000","last_peered":"2024-09-21T11:00:07.143824+0000","last_clean":"2024-09-21T10:58:41.701445+0000","last_became_active":"2024-09-21T10:58:44.734838+0000","last_became_peered":"2024-09-21T10:58:44.734838+0000","last_unstale":"2024-09-21T11:00:07.143824+0000","last_undegraded":"2024-09-21T10:58:44.719801+0000","last_fullsized":"2024-09-21T10:58:44.719541+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:54:36.389392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":102,"num_keys_recovered":30,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2405,"reported_epoch":1556,"state":"active+clean","last_fresh":"2024-09-21T11:00:18.476603+0000","last_change":"2024-09-21T10:44:14.398584+0000","last_active":"2024-09-21T11:00:18.476603+0000","last_peered":"2024-09-21T11:00:18.476603+0000","last_clean":"2024-09-21T11:00:18.476603+0000","last_became_active":"2024-09-21T10:44:14.398278+0000","last_became_peered":"2024-09-21T10:44:14.398278+0000","last_unstale":"2024-09-21T11:00:18.476603+0000","last_undegraded":"2024-09-21T11:00:18.476603+0000","last_fullsized":"2024-09-21T11:00:18.476603+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:19.681750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":57,"num_pg":16,"stat_sum":{"num_bytes":3733913624,"num_objects":56976,"num_object_clones":0,"num_object_copies":170928,"num_objects_missing_on_primary":468,"num_objects_missing":468,"num_objects_degraded":33316,"num_objects_misplaced":9357,"num_objects_unfound":0,"num_objects_dirty":56976,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":77718,"num_write_kb":4310145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9045,"num_bytes_recovered":592773120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11466715136,"data_stored":11466702920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66618,"ondisk_log_size":66618,"up":40,"acting":39,"num_store_stats":5},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":268350,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":173,"num_read_kb":375,"num_write":99,"num_write_kb":1099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3398183,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2703360,"data_stored":2689152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":24,"acting":27,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":19,"acting":19,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":17718,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":18008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":78,"acting":78,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":74,"acting":74,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234797,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4098840,"kb_used_data":3020816,"kb_used_omap":1,"kb_used_meta":1078014,"kb_avail":89646312,"statfs":{"total":95995035648,"available":91797823488,"internally_reserved":0,"allocated":3093315584,"data_stored":3088088745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":1103886795},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496413,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3400500,"kb_used_data":2189796,"kb_used_omap":1,"kb_used_meta":1210686,"kb_avail":90344652,"statfs":{"total":95995035648,"available":92512923648,"internally_reserved":0,"allocated":2242351104,"data_stored":2236868255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1239742922},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":163208758048,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019706,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3154448,"kb_used_data":2230856,"kb_used_omap":1,"kb_used_meta":923582,"kb_avail":90590704,"statfs":{"total":95995035648,"available":92764880896,"internally_reserved":0,"allocated":2284396544,"data_stored":2279177334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":945748422},"hb_peers":[0,1,2,3,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369450,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1934656,"kb_used_data":694776,"kb_used_omap":69,"kb_used_meta":1239802,"kb_avail":91810496,"statfs":{"total":95995035648,"available":94013947904,"internally_reserved":0,"allocated":711450624,"data_stored":706068880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":1269557378},"hb_peers":[0,1,2,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2920577761905,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807958136,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4346764,"kb_used_data":3149188,"kb_used_omap":66,"kb_used_meta":1197437,"kb_avail":89398388,"statfs":{"total":95995035648,"available":91543949312,"internally_reserved":0,"allocated":3224768512,"data_stored":3219374999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68589,"internal_metadata":1226175507},"hb_peers":[0,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":2860448219775,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3208056832,"data_stored":3208052760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":696582144,"data_stored":696582144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2234847232,"data_stored":2234843160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2257518592,"data_stored":2257518592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3069710336,"data_stored":3069706264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T11:00:32.868 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T11:00:32.868 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_4 pg_num 26 2024-09-21T11:00:32.873 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:32.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:32.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:32.975 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:32.998 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:32.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:32.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:33.061 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:00:33.062 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:00:33.080 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:00:33.080 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:00:33.100 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:33.100 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:00:33.101 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:00:33.119 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:00:33.126 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:00:33.126 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:00:33.128 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:33.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:33.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:33.149 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:00:33.149 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:00:33.150 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T11:00:33.150 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T11:00:33.161 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:00:33.171 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:00:33.171 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:00:33.231 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:33.252 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:33.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:33.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:33.354 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:33.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:00:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:00:33] "GET /metrics HTTP/1.1" 200 36587 "" "Prometheus/2.51.0" 2024-09-21T11:00:33.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:33 smithi070 ceph-mon[107385]: pgmap v3007: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.4 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 319 op/s; 32882/168231 objects degraded (19.546%); 9165/168231 objects misplaced (5.448%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T11:00:33.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:33 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3379409533' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:00:33.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:33 smithi070 ceph-mon[112139]: pgmap v3007: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.4 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 319 op/s; 32882/168231 objects degraded (19.546%); 9165/168231 objects misplaced (5.448%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T11:00:33.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:33 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3379409533' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:00:33.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:33.378 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:33.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:33.479 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:33.501 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:33.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:33.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:33.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:33 smithi160 ceph-mon[101879]: pgmap v3007: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.4 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 319 op/s; 32882/168231 objects degraded (19.546%); 9165/168231 objects misplaced (5.448%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T11:00:33.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:33 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3379409533' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:00:33.602 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:33.624 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:33.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:33.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:33.725 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:33.746 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:33.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:33.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:33.848 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:33.869 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:33.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:33.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:33.970 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:33.993 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:33.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:33.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:34.095 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:34.116 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:34.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:34.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:34.170 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T11:00:34.217 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:34.243 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:34.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:34.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:34.344 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:34.365 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:34.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:34.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:34.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:34 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 32882/168231 objects degraded (19.546%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:34.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:34 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 32882/168231 objects degraded (19.546%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:34.467 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:34.487 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:34.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:34.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:34.589 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:34.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:34 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 32882/168231 objects degraded (19.546%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:34.591 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 11:00:34 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=infra.usagestats t=2024-09-21T11:00:34.141822021Z level=info msg="Usage stats are ready to report" 2024-09-21T11:00:34.611 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:34.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:34.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:34.712 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:34.735 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:34.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:34.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:34.775 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T11:00:34.775 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T11:00:34.837 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:34.858 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:34.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:34.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:34.960 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:34.980 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:34.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:34.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:35.082 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:35.110 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:35.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:35.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:35.211 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:35.234 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:35.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:35.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:35.335 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:35.360 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:35.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:35.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:35.461 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:35.486 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:35.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:35.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:35.566 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:35 smithi070 ceph-mon[107385]: pgmap v3008: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.4 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 290 op/s; 32613/166548 objects degraded (19.582%); 8984/166548 objects misplaced (5.394%); 4.2 MiB/s, 67 objects/s recovering 2024-09-21T11:00:35.566 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:35 smithi070 ceph-mon[112139]: pgmap v3008: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.4 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 290 op/s; 32613/166548 objects degraded (19.582%); 8984/166548 objects misplaced (5.394%); 4.2 MiB/s, 67 objects/s recovering 2024-09-21T11:00:35.588 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:35.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:35 smithi160 ceph-mon[101879]: pgmap v3008: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.4 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 290 op/s; 32613/166548 objects degraded (19.582%); 8984/166548 objects misplaced (5.394%); 4.2 MiB/s, 67 objects/s recovering 2024-09-21T11:00:35.610 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:35.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:35.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:35.712 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:35.735 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:35.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:35.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:35.836 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:35.861 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:35.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:35.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:35.962 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:35.984 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:35.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:35.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:36.085 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:36.106 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:36.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:36.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:36.208 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:36.231 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:36.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:36.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:36.332 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:36.359 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:36.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:36.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:36.460 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:36.484 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:36.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:36.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:36.607 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:36.629 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:36.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:36.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:36.672 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:00:36.672 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:36 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:00:36.673 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:00:36.673 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:36 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:00:36.730 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:36.754 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:36.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:36.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:36.776 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T11:00:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:00:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:36 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:00:36.855 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:36.876 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:36.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:36.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:36.978 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:37.002 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:37.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:37.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:37.103 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:37.126 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:37.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:37.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:37.228 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:37.251 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T11:00:37.251 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T11:00:37.252 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T11:00:37.252 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T11:00:37.252 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:37.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:37.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:37.354 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:37.383 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:37.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:37.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:37.485 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:37.512 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:37.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:37.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:37.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:37 smithi160 ceph-mon[101879]: pgmap v3009: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.3 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 307 op/s; 32120/163668 objects degraded (19.625%); 8788/163668 objects misplaced (5.369%); 4.3 MiB/s, 68 objects/s recovering 2024-09-21T11:00:37.613 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:37.649 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:37.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:37.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:37.661 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:37 smithi070 ceph-mon[107385]: pgmap v3009: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.3 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 307 op/s; 32120/163668 objects degraded (19.625%); 8788/163668 objects misplaced (5.369%); 4.3 MiB/s, 68 objects/s recovering 2024-09-21T11:00:37.661 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:37 smithi070 ceph-mon[112139]: pgmap v3009: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.3 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 307 op/s; 32120/163668 objects degraded (19.625%); 8788/163668 objects misplaced (5.369%); 4.3 MiB/s, 68 objects/s recovering 2024-09-21T11:00:37.673 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:00:37.751 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:37.776 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:37.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:37.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:37.878 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:37.920 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:37.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:37.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:38.023 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:38.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:38.060 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:38.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:38.161 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:38.193 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:38.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:38.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:38.295 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:38.321 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:38.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:38.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:38.369 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T11:00:38.411 INFO:teuthology.orchestra.run.smithi070.stderr:set pool 57 pg_num to 26 2024-09-21T11:00:38.423 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:38.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:38.461 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:38.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:38.562 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:38.587 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:38.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:38.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:38 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3020496166' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T11:00:38.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:38 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T11:00:38.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:38 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3020496166' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T11:00:38.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:38 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T11:00:38.690 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:38.722 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:38.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:38.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:38.824 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:38 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3020496166' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T11:00:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:38 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T11:00:38.849 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:38.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:38.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:38.951 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:38.985 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:38.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:38.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:39.086 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:39.112 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:39.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:39.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:39.214 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:39.240 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:39.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:39.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:39.342 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:39.377 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:39.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:39.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:39.478 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:39.508 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:39.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:39.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:39.610 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:39 smithi070 ceph-mon[107385]: pgmap v3010: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.3 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 322 op/s; 31732/161328 objects degraded (19.669%); 8537/161328 objects misplaced (5.292%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T11:00:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:39 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-09-21T11:00:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:39 smithi070 ceph-mon[107385]: osdmap e1558: 8 total, 6 up, 5 in 2024-09-21T11:00:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:39 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 31732/161328 objects degraded (19.669%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:39.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:39 smithi070 ceph-mon[112139]: pgmap v3010: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.3 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 322 op/s; 31732/161328 objects degraded (19.669%); 8537/161328 objects misplaced (5.292%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T11:00:39.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:39 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-09-21T11:00:39.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:39 smithi070 ceph-mon[112139]: osdmap e1558: 8 total, 6 up, 5 in 2024-09-21T11:00:39.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:39 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 31732/161328 objects degraded (19.669%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:39.654 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:39.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:39.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:39.755 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:39.780 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:39.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:39.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:39 smithi160 ceph-mon[101879]: pgmap v3010: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.3 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 322 op/s; 31732/161328 objects degraded (19.669%); 8537/161328 objects misplaced (5.292%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T11:00:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:39 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-09-21T11:00:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:39 smithi160 ceph-mon[101879]: osdmap e1558: 8 total, 6 up, 5 in 2024-09-21T11:00:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:39 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 31732/161328 objects degraded (19.669%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:39.882 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:39.908 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:39.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:39.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:40.010 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:40.035 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:40.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:40.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:40.137 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:40.163 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:40.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:40.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:40.265 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:40.290 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:40.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:40.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:40.392 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:40.422 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:40.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:40.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:40.511 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:00:40.520 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:00:40.524 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:40.601 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:40.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:40.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:40.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:40 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T11:00:40.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:40 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T11:00:40.703 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:40.724 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:40.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:40.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:40.826 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:40 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T11:00:40.848 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:40.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:40.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:40.949 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:40.971 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:40.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:40.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:41.073 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:41.096 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:41.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:41.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:41.197 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:41.219 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:41.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:41.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:41.321 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:41.343 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:41.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:41.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:41.445 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:41.476 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:41.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:41.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:41.577 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:41.602 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:41.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:41.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:41.694 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:41 smithi070 ceph-mon[107385]: pgmap v3012: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.2 GiB data, 16 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 397 op/s; 31299/157371 objects degraded (19.889%); 8537/157371 objects misplaced (5.425%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T11:00:41.694 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:41 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T11:00:41.694 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:41 smithi070 ceph-mon[107385]: osdmap e1559: 8 total, 6 up, 5 in 2024-09-21T11:00:41.695 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:41 smithi070 ceph-mon[112139]: pgmap v3012: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.2 GiB data, 16 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 397 op/s; 31299/157371 objects degraded (19.889%); 8537/157371 objects misplaced (5.425%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T11:00:41.695 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:41 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T11:00:41.695 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:41 smithi070 ceph-mon[112139]: osdmap e1559: 8 total, 6 up, 5 in 2024-09-21T11:00:41.704 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:41.726 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:41.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:41.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:41.828 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:41.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:41 smithi160 ceph-mon[101879]: pgmap v3012: 99 pgs: 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.2 GiB data, 16 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 397 op/s; 31299/157371 objects degraded (19.889%); 8537/157371 objects misplaced (5.425%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T11:00:41.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:41 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T11:00:41.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:41 smithi160 ceph-mon[101879]: osdmap e1559: 8 total, 6 up, 5 in 2024-09-21T11:00:41.849 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:41.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:41.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:41.951 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:41.972 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:41.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:41.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:42.074 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:42.096 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:42.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:42.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:42.198 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:42.220 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:42.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:42.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:42.321 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:42.343 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:42.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:42.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:42.424 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T11:00:42.445 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:42.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:42.479 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:42.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:42.580 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:42.611 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:42.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:42.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:42.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:42 smithi070 ceph-mon[107385]: osdmap e1560: 8 total, 6 up, 5 in 2024-09-21T11:00:42.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:42 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4070252522' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:00:42.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:42 smithi070 ceph-mon[112139]: osdmap e1560: 8 total, 6 up, 5 in 2024-09-21T11:00:42.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:42 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4070252522' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:00:42.713 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:42.739 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:42.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:42.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:42 smithi160 ceph-mon[101879]: osdmap e1560: 8 total, 6 up, 5 in 2024-09-21T11:00:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:42 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4070252522' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:00:42.841 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:42.864 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:42.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:42.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:42.966 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:42.987 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:42.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:42.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:43.089 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:43.112 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:43.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:43.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:43.213 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:43.235 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:43.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:43.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:43.336 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:43.358 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:43.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:43.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:43.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:00:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:00:43] "GET /metrics HTTP/1.1" 200 36586 "" "Prometheus/2.51.0" 2024-09-21T11:00:43.459 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:43.493 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:43.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:43.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:43.595 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:43.620 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:43.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:43.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:43.722 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:43.747 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:43.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:43.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:43.781 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:43 smithi070 ceph-mon[107385]: pgmap v3015: 109 pgs: 10 unknown, 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 526 op/s; 30600/154194 objects degraded (19.845%); 8253/154194 objects misplaced (5.352%); 7.3 MiB/s, 117 objects/s recovering 2024-09-21T11:00:43.781 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:43 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:00:43.781 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:43 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4070252522' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T11:00:43.781 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:43 smithi070 ceph-mon[107385]: osdmap e1561: 8 total, 6 up, 5 in 2024-09-21T11:00:43.781 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:43 smithi070 ceph-mon[112139]: pgmap v3015: 109 pgs: 10 unknown, 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 526 op/s; 30600/154194 objects degraded (19.845%); 8253/154194 objects misplaced (5.352%); 7.3 MiB/s, 117 objects/s recovering 2024-09-21T11:00:43.781 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:43 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:00:43.782 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:43 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4070252522' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T11:00:43.782 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:43 smithi070 ceph-mon[112139]: osdmap e1561: 8 total, 6 up, 5 in 2024-09-21T11:00:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:43 smithi160 ceph-mon[101879]: pgmap v3015: 109 pgs: 10 unknown, 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 526 op/s; 30600/154194 objects degraded (19.845%); 8253/154194 objects misplaced (5.352%); 7.3 MiB/s, 117 objects/s recovering 2024-09-21T11:00:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:43 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:00:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:43 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4070252522' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T11:00:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:43 smithi160 ceph-mon[101879]: osdmap e1561: 8 total, 6 up, 5 in 2024-09-21T11:00:43.849 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:43.874 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:43.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:43.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:43.976 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:44.002 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:44.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:44.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:44.104 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:44.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:44.142 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:44.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:44.243 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:44.269 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:44.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:44.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:44.334 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7, 4] out_osds: [2, 5, 0] dead_osds: [5, 6] live_osds: [0, 1, 2, 3, 4, 7] 2024-09-21T11:00:44.334 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T11:00:44.334 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [1, 3, 6, 7, 4] 2024-09-21T11:00:44.335 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd out 4 2024-09-21T11:00:44.371 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:44.409 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:44.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:44.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:44.511 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:44.542 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:44.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:44.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:44.644 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:44.683 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:44.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:44.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:44.692 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:44 smithi070 ceph-mon[107385]: osdmap e1562: 8 total, 6 up, 5 in 2024-09-21T11:00:44.692 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:44 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 30600/154194 objects degraded (19.845%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:44.692 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:44 smithi070 ceph-mon[112139]: osdmap e1562: 8 total, 6 up, 5 in 2024-09-21T11:00:44.692 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:44 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 30600/154194 objects degraded (19.845%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:44.786 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:44.813 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:44.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:44.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:44 smithi160 ceph-mon[101879]: osdmap e1562: 8 total, 6 up, 5 in 2024-09-21T11:00:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:44 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 30600/154194 objects degraded (19.845%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:44.915 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:44.951 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:44.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:44.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:45.052 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:45.079 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:45.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:45.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:45.180 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:45.207 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:45.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:45.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:45.309 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:45.338 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:45.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:45.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:45.383 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T11:00:45.383 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T11:00:45.383 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T11:00:45.383 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T11:00:45.383 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T11:00:45.440 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:45.470 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:45.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:45.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:45.572 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:45.579 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:45 smithi070 ceph-mon[107385]: pgmap v3018: 109 pgs: 1 remapped+peering, 6 unknown, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 571 op/s; 29207/150513 objects degraded (19.405%); 7287/150513 objects misplaced (4.841%); 13 MiB/s, 212 objects/s recovering 2024-09-21T11:00:45.579 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:45 smithi070 ceph-mon[112139]: pgmap v3018: 109 pgs: 1 remapped+peering, 6 unknown, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 571 op/s; 29207/150513 objects degraded (19.405%); 7287/150513 objects misplaced (4.841%); 13 MiB/s, 212 objects/s recovering 2024-09-21T11:00:45.591 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:00:45.591 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:00:45.610 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:45.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:45.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:45.614 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:00:45.614 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:00:45.637 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:00:45.637 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:00:45.660 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:00:45.660 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:00:45.682 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:00:45.682 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:00:45.682 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T11:00:45.683 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T11:00:45.706 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:00:45.707 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:00:45.712 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:45.741 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:45.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:45.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:45.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:45 smithi160 ceph-mon[101879]: pgmap v3018: 109 pgs: 1 remapped+peering, 6 unknown, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+degraded, 42 active+undersized, 13 active+undersized+degraded, 31 active+clean; 3.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 571 op/s; 29207/150513 objects degraded (19.405%); 7287/150513 objects misplaced (4.841%); 13 MiB/s, 212 objects/s recovering 2024-09-21T11:00:45.843 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:45.869 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:45.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:45.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:45.971 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:46.003 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:46.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:46.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:46.105 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:46.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:46.141 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:46.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:46.242 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:46.268 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:46.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:46.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:46.369 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:46.390 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:46.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:46.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:46.491 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:46.520 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:46.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:46.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:46.621 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:46.648 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:46.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:46.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:46.693 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T11:00:46.718 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:46 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2933166112' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:00:46.718 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:46 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2933166112' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:00:46.749 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:46.775 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:46.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:46.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:46 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2933166112' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:00:46.876 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:46.902 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:46.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:46.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:47.004 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:47.030 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:47.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:47.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:47.132 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:47.157 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:47.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:47.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:47.259 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:47.285 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:47.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:47.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:47.387 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:47.425 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:47.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:47.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:47.527 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:47.552 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:47.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:47.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:47.653 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:47.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:47.691 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:47.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:47.792 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:47.806 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:47 smithi070 ceph-mon[107385]: pgmap v3019: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 remapped+peering, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+degraded, 42 active+undersized, 16 active+undersized+degraded, 32 active+clean; 2.9 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 555 op/s; 27689/144483 objects degraded (19.164%); 7094/144483 objects misplaced (4.910%); 15 MiB/s, 233 objects/s recovering 2024-09-21T11:00:47.807 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:47 smithi070 ceph-mon[112139]: pgmap v3019: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 remapped+peering, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+degraded, 42 active+undersized, 16 active+undersized+degraded, 32 active+clean; 2.9 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 555 op/s; 27689/144483 objects degraded (19.164%); 7094/144483 objects misplaced (4.910%); 15 MiB/s, 233 objects/s recovering 2024-09-21T11:00:47.827 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:47.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:47.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:47.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:47 smithi160 ceph-mon[101879]: pgmap v3019: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 remapped+peering, 3 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+degraded, 42 active+undersized, 16 active+undersized+degraded, 32 active+clean; 2.9 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 555 op/s; 27689/144483 objects degraded (19.164%); 7094/144483 objects misplaced (4.910%); 15 MiB/s, 233 objects/s recovering 2024-09-21T11:00:47.929 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:47.944 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T11:00:47.958 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:47.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:47.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:48.060 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:48.089 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:48.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:48.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:48.191 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:48.226 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:48.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:48.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:48.268 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:00:48.296 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T11:00:48.296 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T11:00:48.328 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:48.356 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:48.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:48.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:48.457 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:48.485 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:48.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:48.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:48.553 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:48 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 27689/144483 objects degraded (19.164%), 29 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:48.554 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:48 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 27689/144483 objects degraded (19.164%), 29 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:48.587 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:48.621 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:48.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:48.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:48.723 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:48.759 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:48.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:48.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:48 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 27689/144483 objects degraded (19.164%), 29 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:48.861 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:48.885 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:48.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:48.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:48.987 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:49.012 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:00:49.014 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:49.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:49.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:49.059 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:00:49.116 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:49.143 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:49.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:49.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:49.245 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:49.269 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:49.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:49.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:49.371 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:49.397 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:49.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:49.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:49.498 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:49.525 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:49.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:49.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:49.627 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:49.653 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:49.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:49.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:49.709 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:49 smithi070 ceph-mon[107385]: pgmap v3020: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+degraded, 42 active+undersized, 16 active+undersized+degraded, 32 active+clean; 2.8 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 782 op/s; 26715/138171 objects degraded (19.335%); 7492/138171 objects misplaced (5.422%); 25 MiB/s, 402 objects/s recovering 2024-09-21T11:00:49.709 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:49 smithi070 ceph-mon[112139]: pgmap v3020: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+degraded, 42 active+undersized, 16 active+undersized+degraded, 32 active+clean; 2.8 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 782 op/s; 26715/138171 objects degraded (19.335%); 7492/138171 objects misplaced (5.422%); 25 MiB/s, 402 objects/s recovering 2024-09-21T11:00:49.755 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:49.783 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:49.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:49.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:49 smithi160 ceph-mon[101879]: pgmap v3020: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+degraded, 42 active+undersized, 16 active+undersized+degraded, 32 active+clean; 2.8 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 782 op/s; 26715/138171 objects degraded (19.335%); 7492/138171 objects misplaced (5.422%); 25 MiB/s, 402 objects/s recovering 2024-09-21T11:00:49.884 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:49.921 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:49.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:49.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:50.023 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:50.047 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:50.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:50.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:50.149 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:50.175 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:50.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:50.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:50.277 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:50.298 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T11:00:50.314 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:50.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:50.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:50.416 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:50.444 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:50.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:50.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:50.546 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:50.577 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:50.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:50.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:50.679 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:50.712 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:50.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:50.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:50.813 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:00:50.842 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:50.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:50.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:50.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:00:50.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:00:50.944 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:50.969 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:50.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:50.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:51.071 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:51.105 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:51.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:51.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:51.206 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:51.233 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:51.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:51.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:51.335 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:51.361 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:51.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:51.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:51.463 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:51.491 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:51.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:51.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:51.593 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:51.628 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:51.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:51.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:51.730 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:51.756 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:51.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:51.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:51 smithi160 ceph-mon[101879]: pgmap v3021: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+degraded, 42 active+undersized, 16 active+undersized+degraded, 32 active+clean; 2.7 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 917 op/s; 25895/132177 objects degraded (19.591%); 7492/132177 objects misplaced (5.668%); 21 MiB/s, 343 objects/s recovering 2024-09-21T11:00:51.858 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:51.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:51 smithi070 ceph-mon[107385]: pgmap v3021: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+degraded, 42 active+undersized, 16 active+undersized+degraded, 32 active+clean; 2.7 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 917 op/s; 25895/132177 objects degraded (19.591%); 7492/132177 objects misplaced (5.668%); 21 MiB/s, 343 objects/s recovering 2024-09-21T11:00:51.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:51 smithi070 ceph-mon[112139]: pgmap v3021: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+degraded, 42 active+undersized, 16 active+undersized+degraded, 32 active+clean; 2.7 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 917 op/s; 25895/132177 objects degraded (19.591%); 7492/132177 objects misplaced (5.668%); 21 MiB/s, 343 objects/s recovering 2024-09-21T11:00:51.888 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:51.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:51.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:51.990 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:52.018 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:52.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:52.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:52.120 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:52.144 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:52.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:52.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:52.246 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:52.271 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:52.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:52.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:52.372 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:52.417 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:52.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:52.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:52.519 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:52.551 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:52.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:52.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:52.652 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:52.684 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:52.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:52.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:52.764 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:00:52.786 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:52.818 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:52.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:52.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:52.920 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:52.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:52.958 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:52.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:53.059 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:53.086 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:53.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:53.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:53.188 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:53.226 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:00:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:00:53] "GET /metrics HTTP/1.1" 200 36591 "" "Prometheus/2.51.0" 2024-09-21T11:00:53.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:53.227 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:53.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:53.328 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:53.370 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:53.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:53.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:53.473 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:53.479 INFO:teuthology.orchestra.run.smithi070.stderr:marked out osd.4. 2024-09-21T11:00:53.480 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a[107362]: 2024-09-21T11:00:53.285+0000 7fad4a465640 -1 mon.a@0(leader).osd e1562 definitely_dead 0 2024-09-21T11:00:53.480 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:53 smithi070 ceph-mon[107385]: pgmap v3022: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+degraded, 42 active+undersized, 16 active+undersized+degraded, 32 active+clean; 2.5 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 1.11k op/s; 23046/121518 objects degraded (18.965%); 7224/121518 objects misplaced (5.945%); 31 MiB/s, 503 objects/s recovering 2024-09-21T11:00:53.480 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:53 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1981850063' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:00:53.480 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:53 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:00:53.480 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:53 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4246437413' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T11:00:53.480 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:53 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T11:00:53.481 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:53 smithi070 ceph-mon[107385]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-21T11:00:53.481 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:53 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 23046/121518 objects degraded (18.965%), 30 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:53.481 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:53 smithi070 ceph-mon[112139]: pgmap v3022: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+degraded, 42 active+undersized, 16 active+undersized+degraded, 32 active+clean; 2.5 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 1.11k op/s; 23046/121518 objects degraded (18.965%); 7224/121518 objects misplaced (5.945%); 31 MiB/s, 503 objects/s recovering 2024-09-21T11:00:53.481 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:53 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1981850063' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:00:53.481 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:53 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:00:53.481 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:53 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4246437413' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T11:00:53.481 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:53 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T11:00:53.481 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:53 smithi070 ceph-mon[112139]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-21T11:00:53.481 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:53 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 23046/121518 objects degraded (18.965%), 30 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:53.482 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T11:00:53.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:53.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:53.616 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:53.638 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:53.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:53.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:53.739 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:53.760 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:53.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:53.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:53 smithi160 ceph-mon[101879]: pgmap v3022: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+degraded, 42 active+undersized, 16 active+undersized+degraded, 32 active+clean; 2.5 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 1.11k op/s; 23046/121518 objects degraded (18.965%); 7224/121518 objects misplaced (5.945%); 31 MiB/s, 503 objects/s recovering 2024-09-21T11:00:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:53 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1981850063' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:00:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:53 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:00:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:53 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4246437413' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T11:00:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:53 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T11:00:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:53 smithi160 ceph-mon[101879]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-21T11:00:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:53 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 23046/121518 objects degraded (18.965%), 30 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:53.862 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:53.884 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:53.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:53.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:53.986 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:54.008 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:54.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:54.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:54.109 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:54.131 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:54.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:54.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:54.232 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:54.253 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:54.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:54.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:54.318 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T11:00:54.318 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T11:00:54.355 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:54.376 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:54.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:54.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:54.477 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:54.506 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:54.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:54.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:54.608 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:54.630 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:54.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:54.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:54.732 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:54.753 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:54.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:54.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:54 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T11:00:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:54 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T11:00:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:54 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-21T11:00:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:54 smithi160 ceph-mon[101879]: osdmap e1563: 8 total, 6 up, 4 in 2024-09-21T11:00:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:54 smithi160 ceph-mon[101879]: 57.4 scrub starts 2024-09-21T11:00:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:54 smithi160 ceph-mon[101879]: 57.4 scrub ok 2024-09-21T11:00:54.855 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:54.876 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:54.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:54.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:54.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:54 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T11:00:54.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:54 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T11:00:54.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:54 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-21T11:00:54.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:54 smithi070 ceph-mon[107385]: osdmap e1563: 8 total, 6 up, 4 in 2024-09-21T11:00:54.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:54 smithi070 ceph-mon[107385]: 57.4 scrub starts 2024-09-21T11:00:54.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:54 smithi070 ceph-mon[107385]: 57.4 scrub ok 2024-09-21T11:00:54.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:54 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T11:00:54.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:54 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T11:00:54.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:54 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-21T11:00:54.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:54 smithi070 ceph-mon[112139]: osdmap e1563: 8 total, 6 up, 4 in 2024-09-21T11:00:54.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:54 smithi070 ceph-mon[112139]: 57.4 scrub starts 2024-09-21T11:00:54.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:54 smithi070 ceph-mon[112139]: 57.4 scrub ok 2024-09-21T11:00:54.978 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:55.000 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:55.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:55.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:55.102 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:55.124 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:55.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:55.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:55.226 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:55.246 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:55.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:55.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:55.348 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:55.372 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:55.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:55.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:55.474 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:55.505 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:55.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:55.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:55.606 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:55.631 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:55.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:55.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:55.733 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:55.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:55 smithi160 ceph-mon[101879]: pgmap v3024: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+degraded, 42 active+undersized, 16 active+undersized+degraded, 32 active+clean; 2.5 GiB data, 12 GiB used, 346 GiB / 358 GiB avail; 0 B/s wr, 966 op/s; 23046/121518 objects degraded (18.965%); 7224/121518 objects misplaced (5.945%); 28 MiB/s, 452 objects/s recovering 2024-09-21T11:00:55.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:55 smithi160 ceph-mon[101879]: osdmap e1564: 8 total, 6 up, 4 in 2024-09-21T11:00:55.754 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:55.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:55.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:55.855 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:55.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:55.878 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:55.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:55.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:55 smithi070 ceph-mon[107385]: pgmap v3024: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+degraded, 42 active+undersized, 16 active+undersized+degraded, 32 active+clean; 2.5 GiB data, 12 GiB used, 346 GiB / 358 GiB avail; 0 B/s wr, 966 op/s; 23046/121518 objects degraded (18.965%); 7224/121518 objects misplaced (5.945%); 28 MiB/s, 452 objects/s recovering 2024-09-21T11:00:55.879 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:55 smithi070 ceph-mon[107385]: osdmap e1564: 8 total, 6 up, 4 in 2024-09-21T11:00:55.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:55 smithi070 ceph-mon[112139]: pgmap v3024: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+degraded, 42 active+undersized, 16 active+undersized+degraded, 32 active+clean; 2.5 GiB data, 12 GiB used, 346 GiB / 358 GiB avail; 0 B/s wr, 966 op/s; 23046/121518 objects degraded (18.965%); 7224/121518 objects misplaced (5.945%); 28 MiB/s, 452 objects/s recovering 2024-09-21T11:00:55.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:55 smithi070 ceph-mon[112139]: osdmap e1564: 8 total, 6 up, 4 in 2024-09-21T11:00:55.980 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:56.010 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:56.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:56.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:56.112 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:56.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:56.153 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:56.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:56.253 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:56.281 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:56.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:56.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:56.384 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:56.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:56.421 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:56.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:56.522 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:56.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:56.557 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:56.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:56.598 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T11:00:56.658 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:56.691 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:56.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:56.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:56.787 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:56 smithi070 ceph-mon[107385]: osdmap e1565: 8 total, 6 up, 4 in 2024-09-21T11:00:56.787 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:56 smithi070 ceph-mon[107385]: 57.17 starting backfill to osd.3 from (0'0,0'0] MAX to 1562'5766 2024-09-21T11:00:56.787 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:56 smithi070 ceph-mon[107385]: 57.19 starting backfill to osd.7 from (0'0,1528'846] MAX to 1562'5676 2024-09-21T11:00:56.787 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:56 smithi070 ceph-mon[107385]: 57.9 starting backfill to osd.7 from (0'0,1528'846] MAX to 1562'5673 2024-09-21T11:00:56.787 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:56 smithi070 ceph-mon[107385]: 57.7 starting backfill to osd.3 from (0'0,0'0] MAX to 1562'5771 2024-09-21T11:00:56.787 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:56 smithi070 ceph-mon[112139]: osdmap e1565: 8 total, 6 up, 4 in 2024-09-21T11:00:56.787 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:56 smithi070 ceph-mon[112139]: 57.17 starting backfill to osd.3 from (0'0,0'0] MAX to 1562'5766 2024-09-21T11:00:56.788 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:56 smithi070 ceph-mon[112139]: 57.19 starting backfill to osd.7 from (0'0,1528'846] MAX to 1562'5676 2024-09-21T11:00:56.788 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:56 smithi070 ceph-mon[112139]: 57.9 starting backfill to osd.7 from (0'0,1528'846] MAX to 1562'5673 2024-09-21T11:00:56.788 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:56 smithi070 ceph-mon[112139]: 57.7 starting backfill to osd.3 from (0'0,0'0] MAX to 1562'5771 2024-09-21T11:00:56.793 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:56.830 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:56.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:56.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:56 smithi160 ceph-mon[101879]: osdmap e1565: 8 total, 6 up, 4 in 2024-09-21T11:00:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:56 smithi160 ceph-mon[101879]: 57.17 starting backfill to osd.3 from (0'0,0'0] MAX to 1562'5766 2024-09-21T11:00:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:56 smithi160 ceph-mon[101879]: 57.19 starting backfill to osd.7 from (0'0,1528'846] MAX to 1562'5676 2024-09-21T11:00:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:56 smithi160 ceph-mon[101879]: 57.9 starting backfill to osd.7 from (0'0,1528'846] MAX to 1562'5673 2024-09-21T11:00:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:56 smithi160 ceph-mon[101879]: 57.7 starting backfill to osd.3 from (0'0,0'0] MAX to 1562'5771 2024-09-21T11:00:56.844 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:00:56.844 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:00:56.867 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:00:56.867 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:00:56.890 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:00:56.890 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:00:56.914 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:00:56.915 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:00:56.932 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:56.943 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:00:56.943 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:00:56.989 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T11:00:56.990 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T11:00:56.991 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:00:56.991 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:00:56.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:56.992 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:56.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:57.093 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:57.130 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:57.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:57.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:57.231 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:57.260 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:57.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:57.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:57.362 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:57.396 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:57.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:57.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:57.497 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:57.525 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:57.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:57.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:57.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:57 smithi070 ceph-mon[107385]: pgmap v3027: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 remapped+peering, 15 unknown, 22 peering, 1 active+recovery_wait+remapped, 11 active+recovery_wait+degraded+remapped, 1 active+undersized+degraded+remapped, 23 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 8 active+undersized+degraded, 6 active+clean; 2.3 GiB data, 11 GiB used, 346 GiB / 358 GiB avail; 0 B/s wr, 680 op/s; 15272/110721 objects degraded (13.793%); 12870/110721 objects misplaced (11.624%); 35 MiB/s, 1 keys/s, 555 objects/s recovering 2024-09-21T11:00:57.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:57 smithi070 ceph-mon[107385]: Health check failed: Reduced data availability: 12 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T11:00:57.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:57 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1022324227' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:00:57.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:57 smithi070 ceph-mon[112139]: pgmap v3027: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 remapped+peering, 15 unknown, 22 peering, 1 active+recovery_wait+remapped, 11 active+recovery_wait+degraded+remapped, 1 active+undersized+degraded+remapped, 23 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 8 active+undersized+degraded, 6 active+clean; 2.3 GiB data, 11 GiB used, 346 GiB / 358 GiB avail; 0 B/s wr, 680 op/s; 15272/110721 objects degraded (13.793%); 12870/110721 objects misplaced (11.624%); 35 MiB/s, 1 keys/s, 555 objects/s recovering 2024-09-21T11:00:57.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:57 smithi070 ceph-mon[112139]: Health check failed: Reduced data availability: 12 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T11:00:57.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:57 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1022324227' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:00:57.628 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:57.667 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:57.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:57.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:57.768 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:57.795 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:57.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:57.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:57 smithi160 ceph-mon[101879]: pgmap v3027: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 remapped+peering, 15 unknown, 22 peering, 1 active+recovery_wait+remapped, 11 active+recovery_wait+degraded+remapped, 1 active+undersized+degraded+remapped, 23 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 8 active+undersized+degraded, 6 active+clean; 2.3 GiB data, 11 GiB used, 346 GiB / 358 GiB avail; 0 B/s wr, 680 op/s; 15272/110721 objects degraded (13.793%); 12870/110721 objects misplaced (11.624%); 35 MiB/s, 1 keys/s, 555 objects/s recovering 2024-09-21T11:00:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:57 smithi160 ceph-mon[101879]: Health check failed: Reduced data availability: 12 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T11:00:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:57 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1022324227' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:00:57.897 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:57.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:57.937 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:57.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:58.037 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:58.075 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:58.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:58.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:58.091 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T11:00:58.091 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T11:00:58.092 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:00:58.177 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:58.203 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:58.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:58.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:58.305 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:00:58.331 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:58.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:58.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:00:58.433 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:58.464 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:58.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:58.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:58.565 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:00:58.594 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:58.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:58.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:00:58.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:58 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 15272/110721 objects degraded (13.793%), 22 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:58.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:58 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 15272/110721 objects degraded (13.793%), 22 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:58.696 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:58.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:58.726 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:58.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:58.826 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:58 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 15272/110721 objects degraded (13.793%), 22 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-21T11:00:58.862 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:58.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:58.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:58.964 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:58.998 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:58.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:58.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:59.100 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:59.129 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:59.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:59.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:59.231 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:00:59.258 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:59.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:59.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:00:59.359 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:59.372 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T11:00:59.393 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:59.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:59.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:59.495 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:59.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:59.535 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:59.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:00:59.635 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:00:59.669 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:00:59 smithi070 ceph-mon[107385]: pgmap v3028: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 remapped+peering, 15 unknown, 22 peering, 1 active+recovery_wait+remapped, 11 active+recovery_wait+degraded+remapped, 1 active+undersized+degraded+remapped, 23 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 8 active+undersized+degraded, 6 active+clean; 2.3 GiB data, 11 GiB used, 346 GiB / 358 GiB avail; 0 B/s wr, 88 op/s; 15272/110721 objects degraded (13.793%); 12870/110721 objects misplaced (11.624%); 12 MiB/s, 1 keys/s, 190 objects/s recovering 2024-09-21T11:00:59.669 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:00:59 smithi070 ceph-mon[112139]: pgmap v3028: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 remapped+peering, 15 unknown, 22 peering, 1 active+recovery_wait+remapped, 11 active+recovery_wait+degraded+remapped, 1 active+undersized+degraded+remapped, 23 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 8 active+undersized+degraded, 6 active+clean; 2.3 GiB data, 11 GiB used, 346 GiB / 358 GiB avail; 0 B/s wr, 88 op/s; 15272/110721 objects degraded (13.793%); 12870/110721 objects misplaced (11.624%); 12 MiB/s, 1 keys/s, 190 objects/s recovering 2024-09-21T11:00:59.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:59.670 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:59.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:00:59.770 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:00:59.804 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:00:59.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:59.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:00:59.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:00:59 smithi160 ceph-mon[101879]: pgmap v3028: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 remapped+peering, 15 unknown, 22 peering, 1 active+recovery_wait+remapped, 11 active+recovery_wait+degraded+remapped, 1 active+undersized+degraded+remapped, 23 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 8 active+undersized+degraded, 6 active+clean; 2.3 GiB data, 11 GiB used, 346 GiB / 358 GiB avail; 0 B/s wr, 88 op/s; 15272/110721 objects degraded (13.793%); 12870/110721 objects misplaced (11.624%); 12 MiB/s, 1 keys/s, 190 objects/s recovering 2024-09-21T11:00:59.906 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:00:59.936 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:00:59.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:59.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:00.038 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:00.068 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:00.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:00.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:00.093 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T11:01:00.095 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T11:01:00.095 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T11:01:00.095 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T11:01:00.095 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T11:01:00.170 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:00.202 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:00.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:00.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:00.303 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:00.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:00.345 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:00.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:00.446 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:00.474 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:00.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:00.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:00.576 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:00.605 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:00.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:00.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:00.706 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:00.735 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:00.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:00.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:00.776 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:00 smithi070 ceph-mon[112139]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs inactive, 2 pgs peering) 2024-09-21T11:01:00.779 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:00 smithi070 ceph-mon[107385]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs inactive, 2 pgs peering) 2024-09-21T11:01:00.814 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T11:01:00.836 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:00 smithi160 ceph-mon[101879]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs inactive, 2 pgs peering) 2024-09-21T11:01:00.850 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:01:00.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:00.869 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:00.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:00.869 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:01:00.970 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:01.005 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:01.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:01.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:01.013 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7] out_osds: [2, 5, 0, 4] dead_osds: [5, 6] live_osds: [0, 1, 2, 3, 4, 7] 2024-09-21T11:01:01.013 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T11:01:01.013 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-21T11:01:01.013 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T11:01:01.107 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:01.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:01.142 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:01.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:01.243 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:01.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:01.280 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:01.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:01.381 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:01.419 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:01.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:01.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:01.521 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:01.561 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:01.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:01.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:01.646 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:01 smithi070 ceph-mon[107385]: pgmap v3029: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 12 active+recovery_wait+degraded+remapped, 1 active+undersized+degraded+remapped, 59 active+clean+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 2.1 GiB data, 11 GiB used, 347 GiB / 358 GiB avail; 0 B/s wr, 503 op/s; 19383/102093 objects degraded (18.986%); 17139/102093 objects misplaced (16.788%); 22 MiB/s, 1 keys/s, 366 objects/s recovering 2024-09-21T11:01:01.646 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:01 smithi070 ceph-mon[112139]: pgmap v3029: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 12 active+recovery_wait+degraded+remapped, 1 active+undersized+degraded+remapped, 59 active+clean+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 2.1 GiB data, 11 GiB used, 347 GiB / 358 GiB avail; 0 B/s wr, 503 op/s; 19383/102093 objects degraded (18.986%); 17139/102093 objects misplaced (16.788%); 22 MiB/s, 1 keys/s, 366 objects/s recovering 2024-09-21T11:01:01.663 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:01.693 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:01.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:01.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:01.797 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:01.827 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:01.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:01.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:01 smithi160 ceph-mon[101879]: pgmap v3029: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 12 active+recovery_wait+degraded+remapped, 1 active+undersized+degraded+remapped, 59 active+clean+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 2.1 GiB data, 11 GiB used, 347 GiB / 358 GiB avail; 0 B/s wr, 503 op/s; 19383/102093 objects degraded (18.986%); 17139/102093 objects misplaced (16.788%); 22 MiB/s, 1 keys/s, 366 objects/s recovering 2024-09-21T11:01:01.930 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:01.953 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:01.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:01.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:02.054 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:02.082 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:02.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:02.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:02.184 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:02.213 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:02.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:02.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:02.314 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:02.349 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:02.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:02.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:02.450 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:02.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:02.484 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:02.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:02.585 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:02.617 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:02.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:02.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:02.719 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:02.751 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:02.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:02.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:02.853 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:02.883 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:02.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:02.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:02.984 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:03.008 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:03.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:03.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:03.110 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:03.132 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:03.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:03.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:03.234 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:03.238 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:01:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:01:03] "GET /metrics HTTP/1.1" 200 36591 "" "Prometheus/2.51.0" 2024-09-21T11:01:03.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:03.256 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:03.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:03.357 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:03.378 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:03.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:03.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:03.479 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:03.484 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:01:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:03.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:03.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:03.605 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:03.634 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:03.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:03.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:03.736 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:03.766 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:03.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:03.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:03 smithi160 ceph-mon[101879]: pgmap v3030: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 11 active+recovery_wait+degraded+remapped, 60 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 1.9 GiB data, 11 GiB used, 347 GiB / 358 GiB avail; 0 B/s wr, 870 op/s; 18011/91416 objects degraded (19.702%); 14206/91416 objects misplaced (15.540%); 32 MiB/s, 1 keys/s, 517 objects/s recovering 2024-09-21T11:01:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:03 smithi160 ceph-mon[101879]: osdmap e1566: 8 total, 6 up, 4 in 2024-09-21T11:01:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:03 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 18011/91416 objects degraded (19.702%), 24 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T11:01:03.868 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:03.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:03 smithi070 ceph-mon[107385]: pgmap v3030: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 11 active+recovery_wait+degraded+remapped, 60 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 1.9 GiB data, 11 GiB used, 347 GiB / 358 GiB avail; 0 B/s wr, 870 op/s; 18011/91416 objects degraded (19.702%); 14206/91416 objects misplaced (15.540%); 32 MiB/s, 1 keys/s, 517 objects/s recovering 2024-09-21T11:01:03.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:03 smithi070 ceph-mon[107385]: osdmap e1566: 8 total, 6 up, 4 in 2024-09-21T11:01:03.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:03 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 18011/91416 objects degraded (19.702%), 24 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T11:01:03.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:03 smithi070 ceph-mon[112139]: pgmap v3030: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 11 active+recovery_wait+degraded+remapped, 60 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 1.9 GiB data, 11 GiB used, 347 GiB / 358 GiB avail; 0 B/s wr, 870 op/s; 18011/91416 objects degraded (19.702%); 14206/91416 objects misplaced (15.540%); 32 MiB/s, 1 keys/s, 517 objects/s recovering 2024-09-21T11:01:03.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:03 smithi070 ceph-mon[112139]: osdmap e1566: 8 total, 6 up, 4 in 2024-09-21T11:01:03.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:03 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 18011/91416 objects degraded (19.702%), 24 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T11:01:03.903 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:03.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:03.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:04.005 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:04.044 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:04.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:04.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:04.146 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:04.176 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:04.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:04.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:04.278 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:04.313 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:04.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:04.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:04.415 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:04.445 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:04.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:04.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:04.547 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:04.577 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:04.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:04.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:04.679 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:04.707 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:04.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:04.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:04.808 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:04.836 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:04.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:04.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:04.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:04 smithi160 ceph-mon[101879]: osdmap e1567: 8 total, 6 up, 4 in 2024-09-21T11:01:04.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:04 smithi070 ceph-mon[112139]: osdmap e1567: 8 total, 6 up, 4 in 2024-09-21T11:01:04.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:04 smithi070 ceph-mon[107385]: osdmap e1567: 8 total, 6 up, 4 in 2024-09-21T11:01:04.938 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:04.959 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:04.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:04.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:05.060 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:05.090 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:05.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:05.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:05.191 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:05.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:05.229 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:05.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:05.330 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:05.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:05.361 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:05.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:05.462 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:05.498 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:05.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:05.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:05.600 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:05.620 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:05.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:05.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:05.722 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:05.761 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:05.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:05.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:05 smithi160 ceph-mon[101879]: pgmap v3033: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 11 active+recovery_wait+degraded+remapped, 60 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 1.9 GiB data, 11 GiB used, 347 GiB / 358 GiB avail; 0 B/s wr, 804 op/s; 18011/91416 objects degraded (19.702%); 14206/91416 objects misplaced (15.540%); 23 MiB/s, 374 objects/s recovering 2024-09-21T11:01:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:01:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:01:05.863 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:05.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:05 smithi070 ceph-mon[107385]: pgmap v3033: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 11 active+recovery_wait+degraded+remapped, 60 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 1.9 GiB data, 11 GiB used, 347 GiB / 358 GiB avail; 0 B/s wr, 804 op/s; 18011/91416 objects degraded (19.702%); 14206/91416 objects misplaced (15.540%); 23 MiB/s, 374 objects/s recovering 2024-09-21T11:01:05.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:01:05.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:01:05.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:05 smithi070 ceph-mon[112139]: pgmap v3033: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 11 active+recovery_wait+degraded+remapped, 60 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 1.9 GiB data, 11 GiB used, 347 GiB / 358 GiB avail; 0 B/s wr, 804 op/s; 18011/91416 objects degraded (19.702%); 14206/91416 objects misplaced (15.540%); 23 MiB/s, 374 objects/s recovering 2024-09-21T11:01:05.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:01:05.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:01:05.892 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:05.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:05.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:05.994 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:06.025 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:06.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:06.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:06.127 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:06.150 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:06.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:06.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:06.251 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:06.282 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:06.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:06.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:06.383 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:06.410 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:06.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:06.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:06.512 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:06.539 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:06.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:06.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:06.640 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:06.669 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:06.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:06.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:06.770 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:06.800 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:06.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:06.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:06.901 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:06.931 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:06.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:06.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:07.032 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:07.061 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:07.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:07.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:07.163 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:07.197 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:07.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:07.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:07.298 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:07.328 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:07.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:07.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:07.429 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:07.457 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:07.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:07.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:07.559 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:07.587 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:07.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:07.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:07.689 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:07.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:07.727 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:07.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:07.828 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:07 smithi160 ceph-mon[101879]: pgmap v3034: 109 pgs: 2 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 7 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 1.5 GiB data, 9.9 GiB used, 348 GiB / 358 GiB avail; 0 B/s wr, 1.61k op/s; 11409/71967 objects degraded (15.853%); 13308/71967 objects misplaced (18.492%); 41 MiB/s, 679 objects/s recovering 2024-09-21T11:01:07.848 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:07.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:07.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:07.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:07 smithi070 ceph-mon[107385]: pgmap v3034: 109 pgs: 2 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 7 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 1.5 GiB data, 9.9 GiB used, 348 GiB / 358 GiB avail; 0 B/s wr, 1.61k op/s; 11409/71967 objects degraded (15.853%); 13308/71967 objects misplaced (18.492%); 41 MiB/s, 679 objects/s recovering 2024-09-21T11:01:07.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:07 smithi070 ceph-mon[112139]: pgmap v3034: 109 pgs: 2 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 7 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 1.5 GiB data, 9.9 GiB used, 348 GiB / 358 GiB avail; 0 B/s wr, 1.61k op/s; 11409/71967 objects degraded (15.853%); 13308/71967 objects misplaced (18.492%); 41 MiB/s, 679 objects/s recovering 2024-09-21T11:01:07.950 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:07.979 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:07.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:07.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:08.081 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:08.109 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:08.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:08.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:08.211 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:08.240 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:08.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:08.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:08.342 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:08.385 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:08.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:08.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:08.486 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:08.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:08.518 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:08.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:08.619 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:08.640 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:08.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:08.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:08.742 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:08.762 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:08.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:08.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:08 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 11409/71967 objects degraded (15.853%), 19 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T11:01:08.864 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:08.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:08 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 11409/71967 objects degraded (15.853%), 19 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T11:01:08.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:08 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 11409/71967 objects degraded (15.853%), 19 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T11:01:08.892 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:08.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:08.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:08.993 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:09.022 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:09.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:09.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:09.124 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:09.154 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:09.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:09.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:09.256 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:09.291 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:09.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:09.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:09.393 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:09.423 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:09.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:09.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:09.524 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:09.558 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:09.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:09.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:09.660 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:09.693 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:09.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:09.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:09.794 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:09.823 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:09.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:09.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:09.924 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:09.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:09.946 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:09.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:10.046 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:10.070 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:10.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:10.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:10.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:09 smithi160 ceph-mon[101879]: pgmap v3035: 109 pgs: 2 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 7 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 1.5 GiB data, 9.9 GiB used, 348 GiB / 358 GiB avail; 0 B/s wr, 1.25k op/s; 11409/71967 objects degraded (15.853%); 13308/71967 objects misplaced (18.492%); 31 MiB/s, 512 objects/s recovering 2024-09-21T11:01:10.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:09 smithi070 ceph-mon[107385]: pgmap v3035: 109 pgs: 2 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 7 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 1.5 GiB data, 9.9 GiB used, 348 GiB / 358 GiB avail; 0 B/s wr, 1.25k op/s; 11409/71967 objects degraded (15.853%); 13308/71967 objects misplaced (18.492%); 31 MiB/s, 512 objects/s recovering 2024-09-21T11:01:10.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:09 smithi070 ceph-mon[112139]: pgmap v3035: 109 pgs: 2 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 7 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 1.5 GiB data, 9.9 GiB used, 348 GiB / 358 GiB avail; 0 B/s wr, 1.25k op/s; 11409/71967 objects degraded (15.853%); 13308/71967 objects misplaced (18.492%); 31 MiB/s, 512 objects/s recovering 2024-09-21T11:01:10.172 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:10.202 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:10.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:10.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:10.304 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:10.344 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:10.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:10.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:10.446 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:10.468 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:10.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:10.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:10.570 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:10.603 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:10.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:10.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:10.704 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:10.733 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:10.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:10.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:10.834 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:10.865 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:10.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:10.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:10.966 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:11.000 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:11.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:11.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:11.101 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:11.136 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:11.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:11.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:11.237 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:11.267 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:11.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:11.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:11.369 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:11.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:11.411 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:11.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:11.511 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:11.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:11.543 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:11.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:11.644 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:11.667 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:11.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:11.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:11.769 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:11.798 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:11.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:11.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:11.899 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:11.934 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:11.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:11.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:12.035 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:12.075 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:12.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:12.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:12.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:11 smithi160 ceph-mon[101879]: pgmap v3036: 109 pgs: 3 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 6 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 1.2 GiB data, 9.5 GiB used, 348 GiB / 358 GiB avail; 0 B/s wr, 1.27k op/s; 9555/60804 objects degraded (15.714%); 11474/60804 objects misplaced (18.870%); 21 MiB/s, 349 objects/s recovering 2024-09-21T11:01:12.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:11 smithi070 ceph-mon[107385]: pgmap v3036: 109 pgs: 3 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 6 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 1.2 GiB data, 9.5 GiB used, 348 GiB / 358 GiB avail; 0 B/s wr, 1.27k op/s; 9555/60804 objects degraded (15.714%); 11474/60804 objects misplaced (18.870%); 21 MiB/s, 349 objects/s recovering 2024-09-21T11:01:12.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:11 smithi070 ceph-mon[112139]: pgmap v3036: 109 pgs: 3 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 6 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 1.2 GiB data, 9.5 GiB used, 348 GiB / 358 GiB avail; 0 B/s wr, 1.27k op/s; 9555/60804 objects degraded (15.714%); 11474/60804 objects misplaced (18.870%); 21 MiB/s, 349 objects/s recovering 2024-09-21T11:01:12.177 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:12.204 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:12.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:12.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:12.306 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:12.335 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:12.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:12.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:12.436 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:12.466 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:12.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:12.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:12.567 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:12.595 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:12.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:12.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:12.697 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:12.729 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:12.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:12.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:12.831 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:12.869 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:12.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:12.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:12.971 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:12.995 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:12.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:12.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:13.097 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:13.127 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:13.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:13.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:13.228 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:13.258 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:13.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:13.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:13.360 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:13.389 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:13.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:13.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:13.491 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:13.521 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:13.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:13.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:13.623 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:13.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:01:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:01:13] "GET /metrics HTTP/1.1" 200 36597 "" "Prometheus/2.51.0" 2024-09-21T11:01:13.648 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:13.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:13.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:13.750 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:13.783 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:13.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:13.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:13.885 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:13.918 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:13.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:13.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:13.926 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:13 smithi070 ceph-mon[107385]: pgmap v3037: 109 pgs: 3 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 6 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 1.0 GiB data, 9.1 GiB used, 349 GiB / 358 GiB avail; 0 B/s wr, 1.45k op/s; 8153/49413 objects degraded (16.500%); 9079/49413 objects misplaced (18.374%); 22 MiB/s, 367 objects/s recovering 2024-09-21T11:01:13.926 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:13 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 8153/49413 objects degraded (16.500%), 18 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T11:01:13.927 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:13 smithi070 ceph-mon[112139]: pgmap v3037: 109 pgs: 3 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 6 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 1.0 GiB data, 9.1 GiB used, 349 GiB / 358 GiB avail; 0 B/s wr, 1.45k op/s; 8153/49413 objects degraded (16.500%); 9079/49413 objects misplaced (18.374%); 22 MiB/s, 367 objects/s recovering 2024-09-21T11:01:13.927 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:13 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 8153/49413 objects degraded (16.500%), 18 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T11:01:14.019 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:14.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:14.050 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:14.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:14.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:13 smithi160 ceph-mon[101879]: pgmap v3037: 109 pgs: 3 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 6 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 1.0 GiB data, 9.1 GiB used, 349 GiB / 358 GiB avail; 0 B/s wr, 1.45k op/s; 8153/49413 objects degraded (16.500%); 9079/49413 objects misplaced (18.374%); 22 MiB/s, 367 objects/s recovering 2024-09-21T11:01:14.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:13 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 8153/49413 objects degraded (16.500%), 18 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T11:01:14.151 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:14.199 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:14.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:14.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:14.301 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:14.308 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:01:14.308 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:01:14.338 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:01:14.338 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:01:14.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:14.339 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:14.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:14.370 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:01:14.371 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:01:14.401 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:01:14.401 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:01:14.427 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:01:14.427 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:01:14.428 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T11:01:14.428 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T11:01:14.440 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:14.456 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:01:14.456 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:01:14.468 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:14.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:14.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:14.570 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:14.599 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:14.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:14.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:14.700 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:14.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:14.735 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:14.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:14.836 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:14.837 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:14 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1587811944' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:01:14.837 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:14 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1587811944' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:01:14.864 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:14.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:14.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:14.966 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:14.996 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:14.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:14.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:15.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:14 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1587811944' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:01:15.098 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:15.135 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:15.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:15.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:15.236 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:15.266 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:15.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:15.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:15.324 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T11:01:15.324 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T11:01:15.369 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:15.402 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:15.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:15.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:15.503 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:15.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:15.542 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:15.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:15.643 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:15.675 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:15.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:15.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:15.777 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:15.810 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:15.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:15.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:15.911 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:15.934 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:15 smithi070 ceph-mon[112139]: pgmap v3038: 109 pgs: 3 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 6 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 1.0 GiB data, 9.1 GiB used, 349 GiB / 358 GiB avail; 0 B/s wr, 1.31k op/s; 8153/49413 objects degraded (16.500%); 9079/49413 objects misplaced (18.374%); 20 MiB/s, 333 objects/s recovering 2024-09-21T11:01:15.934 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:15 smithi070 ceph-mon[107385]: pgmap v3038: 109 pgs: 3 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 6 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 1.0 GiB data, 9.1 GiB used, 349 GiB / 358 GiB avail; 0 B/s wr, 1.31k op/s; 8153/49413 objects degraded (16.500%); 9079/49413 objects misplaced (18.374%); 20 MiB/s, 333 objects/s recovering 2024-09-21T11:01:15.940 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:15.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:15.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:15.946 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:01:16.042 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:16.076 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:16.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:16.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:16.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:15 smithi160 ceph-mon[101879]: pgmap v3038: 109 pgs: 3 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 6 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 1.0 GiB data, 9.1 GiB used, 349 GiB / 358 GiB avail; 0 B/s wr, 1.31k op/s; 8153/49413 objects degraded (16.500%); 9079/49413 objects misplaced (18.374%); 20 MiB/s, 333 objects/s recovering 2024-09-21T11:01:16.178 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:16.208 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:16.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:16.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:16.310 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:16.347 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:16.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:16.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:16.449 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:16.481 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:16.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:16.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:16.582 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:16.619 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:16.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:16.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:16.691 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T11:01:16.720 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:16.753 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:16.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:16.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:16.840 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:16 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/535889826' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T11:01:16.841 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:16 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T11:01:16.841 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:16 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/535889826' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T11:01:16.841 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:16 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T11:01:16.854 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:16.886 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:16.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:16.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:16.988 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:17.025 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:17.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:17.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:17.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:16 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/535889826' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T11:01:17.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:16 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T11:01:17.126 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:17.162 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:17.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:17.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:17.216 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T11:01:17.217 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T11:01:17.264 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:17.295 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:17.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:17.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:17.325 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T11:01:17.397 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:17.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:17.434 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:17.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:17.535 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:17.576 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:17.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:17.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:17.678 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:17.710 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:17.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:17.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:17.812 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:17.843 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:17.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:17.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:17.945 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:17.966 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:17.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:17.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:18.068 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:18.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:17 smithi160 ceph-mon[101879]: pgmap v3039: 109 pgs: 3 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 6 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 476 MiB data, 8.0 GiB used, 350 GiB / 358 GiB avail; 0 B/s wr, 1.90k op/s; 4001/22905 objects degraded (17.468%); 4392/22905 objects misplaced (19.175%); 18 MiB/s, 327 objects/s recovering 2024-09-21T11:01:18.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:17 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:01:18.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:17 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T11:01:18.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:17 smithi160 ceph-mon[101879]: osdmap e1568: 8 total, 6 up, 4 in 2024-09-21T11:01:18.100 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:18.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:18.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:18.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:17 smithi070 ceph-mon[107385]: pgmap v3039: 109 pgs: 3 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 6 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 476 MiB data, 8.0 GiB used, 350 GiB / 358 GiB avail; 0 B/s wr, 1.90k op/s; 4001/22905 objects degraded (17.468%); 4392/22905 objects misplaced (19.175%); 18 MiB/s, 327 objects/s recovering 2024-09-21T11:01:18.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:17 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:01:18.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:17 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T11:01:18.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:17 smithi070 ceph-mon[107385]: osdmap e1568: 8 total, 6 up, 4 in 2024-09-21T11:01:18.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:17 smithi070 ceph-mon[112139]: pgmap v3039: 109 pgs: 3 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 6 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 476 MiB data, 8.0 GiB used, 350 GiB / 358 GiB avail; 0 B/s wr, 1.90k op/s; 4001/22905 objects degraded (17.468%); 4392/22905 objects misplaced (19.175%); 18 MiB/s, 327 objects/s recovering 2024-09-21T11:01:18.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:17 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:01:18.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:17 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T11:01:18.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:17 smithi070 ceph-mon[112139]: osdmap e1568: 8 total, 6 up, 4 in 2024-09-21T11:01:18.201 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:18.223 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:18.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:18.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:18.324 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:18.355 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:18.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:18.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:18.457 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:18.492 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:18.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:18.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:18.594 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:18.614 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:18.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:18.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:18.716 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:18.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:18.748 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:18.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:18.849 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:18.873 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:18.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:18.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:18.974 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:19.005 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:19.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:19.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:19.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:18 smithi160 ceph-mon[101879]: from='client.55777 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T11:01:19.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:18 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 4001/22905 objects degraded (17.468%), 18 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T11:01:19.107 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:19.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:18 smithi070 ceph-mon[107385]: from='client.55777 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T11:01:19.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:18 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 4001/22905 objects degraded (17.468%), 18 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T11:01:19.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:18 smithi070 ceph-mon[112139]: from='client.55777 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T11:01:19.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:18 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 4001/22905 objects degraded (17.468%), 18 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T11:01:19.137 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:19.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:19.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:19.238 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:19.273 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:19.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:19.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:19.375 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:19.408 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:19.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:19.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:19.475 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Removed 67346 objects 2024-09-21T11:01:19.475 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Clean up completed and total clean up time :91.7935 2024-09-21T11:01:19.511 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:19.532 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:19.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:19.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:19.634 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:19.655 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:19.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:19.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:19.757 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:19.778 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:19.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:19.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:19.880 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:19.901 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:19.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:19.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:20.003 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:20.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:20.025 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:20.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:20.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:19 smithi160 ceph-mon[101879]: pgmap v3041: 109 pgs: 3 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 6 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 476 MiB data, 8.0 GiB used, 350 GiB / 358 GiB avail; 0 B/s wr, 1.63k op/s; 4001/22905 objects degraded (17.468%); 4392/22905 objects misplaced (19.175%); 7.4 MiB/s, 149 objects/s recovering 2024-09-21T11:01:20.126 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:20.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:19 smithi070 ceph-mon[107385]: pgmap v3041: 109 pgs: 3 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 6 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 476 MiB data, 8.0 GiB used, 350 GiB / 358 GiB avail; 0 B/s wr, 1.63k op/s; 4001/22905 objects degraded (17.468%); 4392/22905 objects misplaced (19.175%); 7.4 MiB/s, 149 objects/s recovering 2024-09-21T11:01:20.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:19 smithi070 ceph-mon[112139]: pgmap v3041: 109 pgs: 3 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 6 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 16 active+undersized, 8 active+undersized+degraded, 8 active+clean; 476 MiB data, 8.0 GiB used, 350 GiB / 358 GiB avail; 0 B/s wr, 1.63k op/s; 4001/22905 objects degraded (17.468%); 4392/22905 objects misplaced (19.175%); 7.4 MiB/s, 149 objects/s recovering 2024-09-21T11:01:20.149 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:20.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:20.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:20.251 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:20.272 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:20.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:20.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:20.374 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:20.395 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:20.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:20.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:20.497 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:20.518 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:20.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:20.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:20.620 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:20.641 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:20.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:20.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:20.744 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:20.766 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:20.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:20.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:20.868 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:20.888 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:20.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:20.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:20.991 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:21.014 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:21.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:21.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:21.117 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:21.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:21.140 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:21.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:21.241 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:21.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:21.263 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:21.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:21.364 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:21.385 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:21.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:21.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:21.487 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:21.509 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:21.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:21.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:21.610 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:21.631 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:21.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:21.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:21.733 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:21.755 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:21.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:21.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:21.767 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:21 smithi070 ceph-mon[107385]: pgmap v3042: 109 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+remapped+backfill_wait, 3 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 18 active+undersized, 6 active+undersized+degraded, 8 active+clean; 202 MiB data, 7.4 GiB used, 350 GiB / 358 GiB avail; 0 B/s wr, 1.71k op/s; 1806/9747 objects degraded (18.529%); 2166/9747 objects misplaced (22.222%); 5.2 MiB/s, 153 objects/s recovering 2024-09-21T11:01:21.767 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:01:21.767 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:01:21.767 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:21 smithi070 ceph-mon[112139]: pgmap v3042: 109 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+remapped+backfill_wait, 3 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 18 active+undersized, 6 active+undersized+degraded, 8 active+clean; 202 MiB data, 7.4 GiB used, 350 GiB / 358 GiB avail; 0 B/s wr, 1.71k op/s; 1806/9747 objects degraded (18.529%); 2166/9747 objects misplaced (22.222%); 5.2 MiB/s, 153 objects/s recovering 2024-09-21T11:01:21.767 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:01:21.767 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:01:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:21 smithi160 ceph-mon[101879]: pgmap v3042: 109 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+remapped+backfill_wait, 3 active+recovery_wait+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 18 active+undersized, 6 active+undersized+degraded, 8 active+clean; 202 MiB data, 7.4 GiB used, 350 GiB / 358 GiB avail; 0 B/s wr, 1.71k op/s; 1806/9747 objects degraded (18.529%); 2166/9747 objects misplaced (22.222%); 5.2 MiB/s, 153 objects/s recovering 2024-09-21T11:01:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:01:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:01:21.857 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:21.878 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:21.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:21.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:21.980 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:22.001 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:22.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:22.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:22.103 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:22.124 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:22.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:22.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:22.225 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:22.247 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:22.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:22.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:22.349 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:22.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:22.371 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:22.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:22.472 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:22.494 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:22.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:22.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:22.596 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:22.599 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":3040,"stamp":"2024-09-21T11:01:16.685277+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":498904603,"num_objects":7635,"num_object_clones":0,"num_object_copies":22905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4001,"num_objects_misplaced":4392,"num_objects_unfound":0,"num_objects_dirty":7635,"num_whiteouts":0,"num_read":174,"num_read_kb":376,"num_write":124180,"num_write_kb":4177005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23420,"num_bytes_recovered":1496593023,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122689,"ondisk_log_size":122689,"up":228,"acting":300,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":300,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":374980608,"kb_used":8415624,"kb_used_data":3390880,"kb_used_omap":139,"kb_used_meta":5024436,"kb_avail":366564984,"statfs":{"total":383980142592,"available":375362543616,"internally_reserved":0,"allocated":3472261120,"data_stored":3454132621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":143318,"internal_metadata":5145022506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,4,3,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1071775744,"num_objects":-16354,"num_object_clones":0,"num_object_copies":-49062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-7408,"num_objects_misplaced":-8916,"num_objects_unfound":0,"num_objects_dirty":-16354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16354,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1492,"num_bytes_recovered":78053376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.437448"},"pg_stats":[{"pgid":"57.19","version":"1567'6508","reported_seq":8295,"reported_epoch":1567,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-21T11:01:14.357587+0000","last_change":"2024-09-21T11:00:55.586007+0000","last_active":"2024-09-21T11:01:14.357587+0000","last_peered":"2024-09-21T11:01:14.357587+0000","last_clean":"2024-09-21T10:58:43.705315+0000","last_became_active":"2024-09-21T11:00:55.574872+0000","last_became_peered":"2024-09-21T11:00:55.574872+0000","last_unstale":"2024-09-21T11:01:14.357587+0000","last_undegraded":"2024-09-21T11:00:55.505330+0000","last_fullsized":"2024-09-21T11:00:55.487332+0000","mapping_epoch":1564,"log_start":"1538'1939","ondisk_log_start":"1538'1939","created":1559,"last_epoch_clean":1538,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":4569,"log_dups_size":0,"ondisk_log_size":4569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19103744,"num_objects":292,"num_object_clones":0,"num_object_copies":876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":292,"num_objects_misplaced":292,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3883,"num_write_kb":133568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":423,"num_bytes_recovered":27721728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":292}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"57.18","version":"1567'6583","reported_seq":10444,"reported_epoch":1567,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-21T11:01:15.994259+0000","last_change":"2024-09-21T11:01:04.538760+0000","last_active":"2024-09-21T11:01:15.994259+0000","last_peered":"2024-09-21T11:01:15.994259+0000","last_clean":"2024-09-21T10:58:20.477216+0000","last_became_active":"2024-09-21T11:00:55.502885+0000","last_became_peered":"2024-09-21T11:00:55.502885+0000","last_unstale":"2024-09-21T11:01:15.994259+0000","last_undegraded":"2024-09-21T11:01:15.994259+0000","last_fullsized":"2024-09-21T11:01:15.994259+0000","mapping_epoch":1564,"log_start":"1538'1972","ondisk_log_start":"1538'1972","created":1559,"last_epoch_clean":1525,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":4611,"log_dups_size":0,"ondisk_log_size":4611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15269888,"num_objects":233,"num_object_clones":0,"num_object_copies":699,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":233,"num_objects_unfound":0,"num_objects_dirty":233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3939,"num_write_kb":133504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1326,"num_bytes_recovered":86867968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[{"shards":"1,4,7","objects":233}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"57.17","version":"1567'6756","reported_seq":8980,"reported_epoch":1567,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T11:01:15.977685+0000","last_change":"2024-09-21T11:00:55.604409+0000","last_active":"2024-09-21T11:01:15.977685+0000","last_peered":"2024-09-21T11:01:15.977685+0000","last_clean":"2024-09-21T10:58:43.706565+0000","last_became_active":"2024-09-21T11:00:55.583151+0000","last_became_peered":"2024-09-21T11:00:55.583151+0000","last_unstale":"2024-09-21T11:01:15.977685+0000","last_undegraded":"2024-09-21T11:00:55.577770+0000","last_fullsized":"2024-09-21T11:01:15.977685+0000","mapping_epoch":1564,"log_start":"1538'2292","ondisk_log_start":"1538'2292","created":1559,"last_epoch_clean":1525,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":4464,"log_dups_size":0,"ondisk_log_size":4464,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12320768,"num_objects":188,"num_object_clones":0,"num_object_copies":564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4081,"num_write_kb":136608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1071,"num_bytes_recovered":70123520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,4],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,4,7","objects":188}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"57.16","version":"1567'6629","reported_seq":10375,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:15.970645+0000","last_change":"2024-09-21T11:01:03.587900+0000","last_active":"2024-09-21T11:01:15.970645+0000","last_peered":"2024-09-21T11:01:15.970645+0000","last_clean":"2024-09-21T11:01:15.970645+0000","last_became_active":"2024-09-21T11:01:03.577864+0000","last_became_peered":"2024-09-21T11:01:03.577864+0000","last_unstale":"2024-09-21T11:01:15.970645+0000","last_undegraded":"2024-09-21T11:01:15.970645+0000","last_fullsized":"2024-09-21T11:01:15.970645+0000","mapping_epoch":1566,"log_start":"1538'2223","ondisk_log_start":"1538'2223","created":1559,"last_epoch_clean":1567,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":4406,"log_dups_size":0,"ondisk_log_size":4406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11501568,"num_objects":176,"num_object_clones":0,"num_object_copies":528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":176,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4012,"num_write_kb":133984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1297,"num_bytes_recovered":76054528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[{"shards":"1,4,7","objects":176}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"57.15","version":"1567'6635","reported_seq":8748,"reported_epoch":1567,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T11:01:16.003372+0000","last_change":"2024-09-21T11:00:55.551216+0000","last_active":"2024-09-21T11:01:16.003372+0000","last_peered":"2024-09-21T11:01:16.003372+0000","last_clean":"2024-09-21T10:58:43.699982+0000","last_became_active":"2024-09-21T11:00:55.517838+0000","last_became_peered":"2024-09-21T11:00:55.517838+0000","last_unstale":"2024-09-21T11:01:16.003372+0000","last_undegraded":"2024-09-21T11:00:55.485977+0000","last_fullsized":"2024-09-21T11:01:16.003372+0000","mapping_epoch":1564,"log_start":"1538'2061","ondisk_log_start":"1538'2061","created":1559,"last_epoch_clean":1525,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":4574,"log_dups_size":0,"ondisk_log_size":4574,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14254080,"num_objects":218,"num_object_clones":0,"num_object_copies":654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3975,"num_write_kb":134144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1016,"num_bytes_recovered":66584576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,4,7","objects":218}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"57.14","version":"1567'6600","reported_seq":6710,"reported_epoch":1567,"state":"active+clean","last_fresh":"2024-09-21T11:01:14.629892+0000","last_change":"2024-09-21T11:01:01.814929+0000","last_active":"2024-09-21T11:01:14.629892+0000","last_peered":"2024-09-21T11:01:14.629892+0000","last_clean":"2024-09-21T11:01:14.629892+0000","last_became_active":"2024-09-21T11:00:41.472172+0000","last_became_peered":"2024-09-21T11:00:41.472172+0000","last_unstale":"2024-09-21T11:01:14.629892+0000","last_undegraded":"2024-09-21T11:01:14.629892+0000","last_fullsized":"2024-09-21T11:01:14.629892+0000","mapping_epoch":1559,"log_start":"0'0","ondisk_log_start":"0'0","created":1559,"last_epoch_clean":1563,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":6600,"log_dups_size":0,"ondisk_log_size":6600,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-28T14:35:22.706761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16809984,"num_objects":257,"num_object_clones":0,"num_object_copies":771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3940,"num_write_kb":134272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"57.13","version":"1567'6694","reported_seq":6797,"reported_epoch":1567,"state":"active+undersized+degraded","last_fresh":"2024-09-21T11:01:14.632082+0000","last_change":"2024-09-21T11:00:41.446363+0000","last_active":"2024-09-21T11:01:14.632082+0000","last_peered":"2024-09-21T11:01:14.632082+0000","last_clean":"2024-09-21T10:58:43.705212+0000","last_became_active":"2024-09-21T11:00:41.446363+0000","last_became_peered":"2024-09-21T11:00:41.446363+0000","last_unstale":"2024-09-21T11:01:14.632082+0000","last_undegraded":"2024-09-21T11:00:41.439931+0000","last_fullsized":"2024-09-21T11:00:41.424013+0000","mapping_epoch":1559,"log_start":"0'0","ondisk_log_start":"0'0","created":1559,"last_epoch_clean":1525,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":6694,"log_dups_size":0,"ondisk_log_size":6694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T11:01:57.786874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19660800,"num_objects":300,"num_object_clones":0,"num_object_copies":900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":300,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3984,"num_write_kb":137088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":300}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"57.12","version":"1567'6415","reported_seq":8323,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:14.594343+0000","last_change":"2024-09-21T11:01:06.808877+0000","last_active":"2024-09-21T11:01:14.594343+0000","last_peered":"2024-09-21T11:01:14.594343+0000","last_clean":"2024-09-21T11:01:14.594343+0000","last_became_active":"2024-09-21T11:00:55.503806+0000","last_became_peered":"2024-09-21T11:00:55.503806+0000","last_unstale":"2024-09-21T11:01:14.594343+0000","last_undegraded":"2024-09-21T11:01:14.594343+0000","last_fullsized":"2024-09-21T11:01:14.594343+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":1559,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":6415,"log_dups_size":0,"ondisk_log_size":6415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic deep scrub scheduled @ 2024-10-01T01:52:21.437498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18448384,"num_objects":282,"num_object_clones":0,"num_object_copies":846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":282,"num_objects_unfound":0,"num_objects_dirty":282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3818,"num_write_kb":131168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":446,"num_bytes_recovered":29196288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,4],"avail_no_missing":["3","1","4"],"object_location_counts":[{"shards":"1,3,4","objects":282}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"57.11","version":"1567'6530","reported_seq":6618,"reported_epoch":1567,"state":"active+undersized+degraded","last_fresh":"2024-09-21T11:01:14.350858+0000","last_change":"2024-09-21T11:00:41.456311+0000","last_active":"2024-09-21T11:01:14.350858+0000","last_peered":"2024-09-21T11:01:14.350858+0000","last_clean":"2024-09-21T10:58:43.668991+0000","last_became_active":"2024-09-21T11:00:41.456311+0000","last_became_peered":"2024-09-21T11:00:41.456311+0000","last_unstale":"2024-09-21T11:01:14.350858+0000","last_undegraded":"2024-09-21T11:00:41.442392+0000","last_fullsized":"2024-09-21T11:00:41.423892+0000","mapping_epoch":1559,"log_start":"1538'1929","ondisk_log_start":"1538'1929","created":1559,"last_epoch_clean":1525,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":4601,"log_dups_size":0,"ondisk_log_size":4601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19759104,"num_objects":302,"num_object_clones":0,"num_object_copies":906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3892,"num_write_kb":134176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":302}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"57.10","version":"1567'6751","reported_seq":8376,"reported_epoch":1567,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T11:01:15.987107+0000","last_change":"2024-09-21T11:00:55.564377+0000","last_active":"2024-09-21T11:01:15.987107+0000","last_peered":"2024-09-21T11:01:15.987107+0000","last_clean":"2024-09-21T10:58:20.470296+0000","last_became_active":"2024-09-21T11:00:55.515411+0000","last_became_peered":"2024-09-21T11:00:55.515411+0000","last_unstale":"2024-09-21T11:01:15.987107+0000","last_undegraded":"2024-09-21T11:00:55.483417+0000","last_fullsized":"2024-09-21T11:01:15.987107+0000","mapping_epoch":1564,"log_start":"1538'2215","ondisk_log_start":"1538'2215","created":1559,"last_epoch_clean":1525,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":4536,"log_dups_size":0,"ondisk_log_size":4536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15237120,"num_objects":233,"num_object_clones":0,"num_object_copies":699,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":66,"num_objects_misplaced":167,"num_objects_unfound":0,"num_objects_dirty":233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4050,"num_write_kb":137024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":379,"num_bytes_recovered":24805376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,4,7","objects":233}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"57.f","version":"1567'8092","reported_seq":8151,"reported_epoch":1567,"state":"active+undersized+degraded","last_fresh":"2024-09-21T11:01:16.004438+0000","last_change":"2024-09-21T10:58:44.721330+0000","last_active":"2024-09-21T11:01:16.004438+0000","last_peered":"2024-09-21T11:01:16.004438+0000","last_clean":"2024-09-21T10:58:43.715781+0000","last_became_active":"2024-09-21T10:58:44.721330+0000","last_became_peered":"2024-09-21T10:58:44.721330+0000","last_unstale":"2024-09-21T11:01:16.004438+0000","last_undegraded":"2024-09-21T10:58:44.714262+0000","last_fullsized":"2024-09-21T10:58:44.714154+0000","mapping_epoch":1541,"log_start":"1557'5148","ondisk_log_start":"1557'5148","created":1524,"last_epoch_clean":1525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":2944,"log_dups_size":1160,"ondisk_log_size":2944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:14:35.842654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28311552,"num_objects":432,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8092,"num_write_kb":272768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":432}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"57.e","version":"1567'7703","reported_seq":10769,"reported_epoch":1567,"state":"active+recovering+degraded+remapped","last_fresh":"2024-09-21T11:01:14.360176+0000","last_change":"2024-09-21T11:01:02.810295+0000","last_active":"2024-09-21T11:01:14.360176+0000","last_peered":"2024-09-21T11:01:14.360176+0000","last_clean":"2024-09-21T10:58:43.700774+0000","last_became_active":"2024-09-21T11:00:55.530089+0000","last_became_peered":"2024-09-21T11:00:55.530089+0000","last_unstale":"2024-09-21T11:01:14.360176+0000","last_undegraded":"2024-09-21T11:00:55.495220+0000","last_fullsized":"2024-09-21T11:01:14.360176+0000","mapping_epoch":1564,"log_start":"1556'4722","ondisk_log_start":"1556'4722","created":1524,"last_epoch_clean":1525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":2981,"log_dups_size":986,"ondisk_log_size":2981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34537472,"num_objects":527,"num_object_clones":0,"num_object_copies":1581,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":527,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7703,"num_write_kb":263360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1477,"num_bytes_recovered":77070336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,4,7","objects":527}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"57.d","version":"1567'7816","reported_seq":13245,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:14.330558+0000","last_change":"2024-09-21T11:00:59.591348+0000","last_active":"2024-09-21T11:01:14.330558+0000","last_peered":"2024-09-21T11:01:14.330558+0000","last_clean":"2024-09-21T11:01:14.330558+0000","last_became_active":"2024-09-21T11:00:55.529258+0000","last_became_peered":"2024-09-21T11:00:55.529258+0000","last_unstale":"2024-09-21T11:01:14.330558+0000","last_undegraded":"2024-09-21T11:01:14.330558+0000","last_fullsized":"2024-09-21T11:01:14.330558+0000","mapping_epoch":1564,"log_start":"1556'4783","ondisk_log_start":"1556'4783","created":1524,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":3033,"log_dups_size":986,"ondisk_log_size":3033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:44:27.182159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39911448,"num_objects":610,"num_object_clones":0,"num_object_copies":1830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":610,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7816,"num_write_kb":269569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2669,"num_bytes_recovered":168099840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[{"shards":"1,4,7","objects":610}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"57.c","version":"1567'7928","reported_seq":11708,"reported_epoch":1567,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T11:01:15.988863+0000","last_change":"2024-09-21T11:00:55.979553+0000","last_active":"2024-09-21T11:01:15.988863+0000","last_peered":"2024-09-21T11:01:15.988863+0000","last_clean":"2024-09-21T10:58:43.677050+0000","last_became_active":"2024-09-21T11:00:55.783559+0000","last_became_peered":"2024-09-21T11:00:55.783559+0000","last_unstale":"2024-09-21T11:01:15.988863+0000","last_undegraded":"2024-09-21T11:00:55.767457+0000","last_fullsized":"2024-09-21T11:00:55.483720+0000","mapping_epoch":1564,"log_start":"1557'4967","ondisk_log_start":"1557'4967","created":1524,"last_epoch_clean":1540,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":2961,"log_dups_size":1160,"ondisk_log_size":2961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:28:03.079509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25690112,"num_objects":392,"num_object_clones":0,"num_object_copies":1176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":392,"num_objects_misplaced":710,"num_objects_unfound":0,"num_objects_dirty":392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7928,"num_write_kb":266240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1849,"num_bytes_recovered":121110528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":-318}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"57.b","version":"1567'8009","reported_seq":13505,"reported_epoch":1567,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-21T11:01:14.347356+0000","last_change":"2024-09-21T11:01:04.574057+0000","last_active":"2024-09-21T11:01:14.347356+0000","last_peered":"2024-09-21T11:01:14.347356+0000","last_clean":"2024-09-21T10:58:20.469888+0000","last_became_active":"2024-09-21T11:00:55.529264+0000","last_became_peered":"2024-09-21T11:00:55.529264+0000","last_unstale":"2024-09-21T11:01:14.347356+0000","last_undegraded":"2024-09-21T11:01:14.347356+0000","last_fullsized":"2024-09-21T11:01:14.347356+0000","mapping_epoch":1564,"log_start":"1556'4983","ondisk_log_start":"1556'4983","created":1524,"last_epoch_clean":1525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":3026,"log_dups_size":1044,"ondisk_log_size":3026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36503552,"num_objects":557,"num_object_clones":0,"num_object_copies":1671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":557,"num_objects_unfound":0,"num_objects_dirty":557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8009,"num_write_kb":274112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2710,"num_bytes_recovered":177602560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[{"shards":"1,4,7","objects":557}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"57.a","version":"1567'8157","reported_seq":13548,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:16.005756+0000","last_change":"2024-09-21T11:00:55.546257+0000","last_active":"2024-09-21T11:01:16.005756+0000","last_peered":"2024-09-21T11:01:16.005756+0000","last_clean":"2024-09-21T11:01:16.005756+0000","last_became_active":"2024-09-21T11:00:55.517071+0000","last_became_peered":"2024-09-21T11:00:55.517071+0000","last_unstale":"2024-09-21T11:01:16.005756+0000","last_undegraded":"2024-09-21T11:01:16.005756+0000","last_fullsized":"2024-09-21T11:01:16.005756+0000","mapping_epoch":1564,"log_start":"1557'5170","ondisk_log_start":"1557'5170","created":1524,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":2987,"log_dups_size":1160,"ondisk_log_size":2987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:35:46.017546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24576000,"num_objects":375,"num_object_clones":0,"num_object_copies":1125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":375,"num_objects_unfound":0,"num_objects_dirty":375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8157,"num_write_kb":273024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2650,"num_bytes_recovered":173670400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[{"shards":"1,4,7","objects":375}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"57.9","version":"1567'6520","reported_seq":8311,"reported_epoch":1567,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-21T11:01:14.250612+0000","last_change":"2024-09-21T11:00:55.589699+0000","last_active":"2024-09-21T11:01:14.250612+0000","last_peered":"2024-09-21T11:01:14.250612+0000","last_clean":"2024-09-21T10:58:43.705315+0000","last_became_active":"2024-09-21T11:00:55.574869+0000","last_became_peered":"2024-09-21T11:00:55.574869+0000","last_unstale":"2024-09-21T11:01:14.250612+0000","last_undegraded":"2024-09-21T11:00:55.498031+0000","last_fullsized":"2024-09-21T11:00:55.487165+0000","mapping_epoch":1564,"log_start":"1538'1840","ondisk_log_start":"1538'1840","created":1524,"last_epoch_clean":1538,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":4680,"log_dups_size":0,"ondisk_log_size":4680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18317312,"num_objects":279,"num_object_clones":0,"num_object_copies":837,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":279,"num_objects_misplaced":279,"num_objects_unfound":0,"num_objects_dirty":279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3894,"num_write_kb":133568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":424,"num_bytes_recovered":27721728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":279}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"57.8","version":"1567'6653","reported_seq":10553,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:15.968569+0000","last_change":"2024-09-21T11:01:02.809809+0000","last_active":"2024-09-21T11:01:15.968569+0000","last_peered":"2024-09-21T11:01:15.968569+0000","last_clean":"2024-09-21T11:01:15.968569+0000","last_became_active":"2024-09-21T11:00:55.502895+0000","last_became_peered":"2024-09-21T11:00:55.502895+0000","last_unstale":"2024-09-21T11:01:15.968569+0000","last_undegraded":"2024-09-21T11:01:15.968569+0000","last_fullsized":"2024-09-21T11:01:15.968569+0000","mapping_epoch":1564,"log_start":"1538'2149","ondisk_log_start":"1538'2149","created":1524,"last_epoch_clean":1566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":4504,"log_dups_size":0,"ondisk_log_size":4504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10682368,"num_objects":163,"num_object_clones":0,"num_object_copies":489,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":163,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4009,"num_write_kb":133504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1343,"num_bytes_recovered":88047616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[{"shards":"1,4,7","objects":163}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"57.7","version":"1567'6744","reported_seq":8957,"reported_epoch":1567,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T11:01:15.987523+0000","last_change":"2024-09-21T11:00:55.603949+0000","last_active":"2024-09-21T11:01:15.987523+0000","last_peered":"2024-09-21T11:01:15.987523+0000","last_clean":"2024-09-21T10:58:43.706565+0000","last_became_active":"2024-09-21T11:00:55.581096+0000","last_became_peered":"2024-09-21T11:00:55.581096+0000","last_unstale":"2024-09-21T11:01:15.987523+0000","last_undegraded":"2024-09-21T11:00:55.574889+0000","last_fullsized":"2024-09-21T11:01:15.987523+0000","mapping_epoch":1564,"log_start":"1538'2246","ondisk_log_start":"1538'2246","created":1524,"last_epoch_clean":1525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":4498,"log_dups_size":0,"ondisk_log_size":4498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13107200,"num_objects":200,"num_object_clones":0,"num_object_copies":600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4069,"num_write_kb":136608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1065,"num_bytes_recovered":69795840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,4],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,4,7","objects":200}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"57.6","version":"1567'6608","reported_seq":10366,"reported_epoch":1567,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-21T11:01:15.970632+0000","last_change":"2024-09-21T11:01:04.563468+0000","last_active":"2024-09-21T11:01:15.970632+0000","last_peered":"2024-09-21T11:01:15.970632+0000","last_clean":"2024-09-21T10:58:20.477277+0000","last_became_active":"2024-09-21T11:00:55.515734+0000","last_became_peered":"2024-09-21T11:00:55.515734+0000","last_unstale":"2024-09-21T11:01:15.970632+0000","last_undegraded":"2024-09-21T11:01:15.970632+0000","last_fullsized":"2024-09-21T11:01:15.970632+0000","mapping_epoch":1564,"log_start":"1538'2045","ondisk_log_start":"1538'2045","created":1524,"last_epoch_clean":1525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":4563,"log_dups_size":0,"ondisk_log_size":4563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12877824,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":196,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3990,"num_write_kb":133984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1306,"num_bytes_recovered":85622784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[{"shards":"1,4,7","objects":196}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"57.5","version":"1567'6684","reported_seq":8867,"reported_epoch":1567,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T11:01:15.933952+0000","last_change":"2024-09-21T11:00:55.552338+0000","last_active":"2024-09-21T11:01:15.933952+0000","last_peered":"2024-09-21T11:01:15.933952+0000","last_clean":"2024-09-21T10:58:43.699982+0000","last_became_active":"2024-09-21T11:00:55.521229+0000","last_became_peered":"2024-09-21T11:00:55.521229+0000","last_unstale":"2024-09-21T11:01:15.933952+0000","last_undegraded":"2024-09-21T11:00:55.494146+0000","last_fullsized":"2024-09-21T11:01:15.933952+0000","mapping_epoch":1564,"log_start":"1538'2156","ondisk_log_start":"1538'2156","created":1524,"last_epoch_clean":1525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":4528,"log_dups_size":0,"ondisk_log_size":4528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11042816,"num_objects":168,"num_object_clones":0,"num_object_copies":504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":134144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1050,"num_bytes_recovered":68812800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,4,7","objects":168}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"57.4","version":"1567'6568","reported_seq":6692,"reported_epoch":1567,"state":"active+clean","last_fresh":"2024-09-21T11:01:14.599106+0000","last_change":"2024-09-21T11:00:53.888239+0000","last_active":"2024-09-21T11:01:14.599106+0000","last_peered":"2024-09-21T11:01:14.599106+0000","last_clean":"2024-09-21T11:01:14.599106+0000","last_became_active":"2024-09-21T11:00:41.438358+0000","last_became_peered":"2024-09-21T11:00:41.438358+0000","last_unstale":"2024-09-21T11:01:14.599106+0000","last_undegraded":"2024-09-21T11:01:14.599106+0000","last_fullsized":"2024-09-21T11:01:14.599106+0000","mapping_epoch":1524,"log_start":"0'0","ondisk_log_start":"0'0","created":1524,"last_epoch_clean":1560,"parent":"0.0","parent_split_bits":0,"last_scrub":"1562'5705","last_scrub_stamp":"2024-09-21T11:00:53.888156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T11:00:53.888156+0000","objects_scrubbed":1128,"log_size":6568,"log_dups_size":0,"ondisk_log_size":6568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:33:36.198858+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17367040,"num_objects":265,"num_object_clones":0,"num_object_copies":795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":863,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"57.3","version":"1567'6700","reported_seq":6804,"reported_epoch":1567,"state":"active+undersized+degraded","last_fresh":"2024-09-21T11:01:14.585480+0000","last_change":"2024-09-21T11:00:41.440092+0000","last_active":"2024-09-21T11:01:14.585480+0000","last_peered":"2024-09-21T11:01:14.585480+0000","last_clean":"2024-09-21T10:58:43.705212+0000","last_became_active":"2024-09-21T11:00:41.440092+0000","last_became_peered":"2024-09-21T11:00:41.440092+0000","last_unstale":"2024-09-21T11:01:14.585480+0000","last_undegraded":"2024-09-21T11:00:41.420876+0000","last_fullsized":"2024-09-21T11:00:41.420726+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":1524,"last_epoch_clean":1525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":6700,"log_dups_size":0,"ondisk_log_size":6700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T11:01:56.759533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19267584,"num_objects":294,"num_object_clones":0,"num_object_copies":882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3990,"num_write_kb":137088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"57.2","version":"1567'6446","reported_seq":8354,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:14.547895+0000","last_change":"2024-09-21T11:01:03.803987+0000","last_active":"2024-09-21T11:01:14.547895+0000","last_peered":"2024-09-21T11:01:14.547895+0000","last_clean":"2024-09-21T11:01:14.547895+0000","last_became_active":"2024-09-21T11:00:55.503758+0000","last_became_peered":"2024-09-21T11:00:55.503758+0000","last_unstale":"2024-09-21T11:01:14.547895+0000","last_undegraded":"2024-09-21T11:01:14.547895+0000","last_fullsized":"2024-09-21T11:01:14.547895+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":1524,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":6446,"log_dups_size":0,"ondisk_log_size":6446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-28T18:19:01.978418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16416768,"num_objects":250,"num_object_clones":0,"num_object_copies":750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":250,"num_objects_unfound":0,"num_objects_dirty":250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3848,"num_write_kb":131168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":445,"num_bytes_recovered":29196288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,4],"avail_no_missing":["3","1","4"],"object_location_counts":[{"shards":"1,3,4","objects":250}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"57.1","version":"1567'6556","reported_seq":6645,"reported_epoch":1567,"state":"active+undersized+degraded","last_fresh":"2024-09-21T11:01:14.305190+0000","last_change":"2024-09-21T11:00:41.449894+0000","last_active":"2024-09-21T11:01:14.305190+0000","last_peered":"2024-09-21T11:01:14.305190+0000","last_clean":"2024-09-21T10:58:43.668991+0000","last_became_active":"2024-09-21T11:00:41.449894+0000","last_became_peered":"2024-09-21T11:00:41.449894+0000","last_unstale":"2024-09-21T11:01:14.305190+0000","last_undegraded":"2024-09-21T11:00:41.423257+0000","last_fullsized":"2024-09-21T11:00:41.423136+0000","mapping_epoch":1541,"log_start":"1538'1922","ondisk_log_start":"1538'1922","created":1524,"last_epoch_clean":1525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":4634,"log_dups_size":0,"ondisk_log_size":4634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18055168,"num_objects":275,"num_object_clones":0,"num_object_copies":825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3917,"num_write_kb":134176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":275}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"57.0","version":"1567'6837","reported_seq":8461,"reported_epoch":1567,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T11:01:15.984316+0000","last_change":"2024-09-21T11:00:55.563502+0000","last_active":"2024-09-21T11:01:15.984316+0000","last_peered":"2024-09-21T11:01:15.984316+0000","last_clean":"2024-09-21T10:58:20.470296+0000","last_became_active":"2024-09-21T11:00:55.516256+0000","last_became_peered":"2024-09-21T11:00:55.516256+0000","last_unstale":"2024-09-21T11:01:15.984316+0000","last_undegraded":"2024-09-21T11:00:55.486650+0000","last_fullsized":"2024-09-21T11:01:15.984316+0000","mapping_epoch":1564,"log_start":"1538'2369","ondisk_log_start":"1538'2369","created":1524,"last_epoch_clean":1525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:58:05.418285+0000","last_clean_scrub_stamp":"2024-09-21T10:58:05.418285+0000","objects_scrubbed":0,"log_size":4468,"log_dups_size":0,"ondisk_log_size":4468,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9601024,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":62,"num_objects_misplaced":84,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4135,"num_write_kb":137024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":378,"num_bytes_recovered":24805376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,4,7","objects":146}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":2005,"reported_epoch":1566,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:02.575021+0000","last_change":"2024-09-21T11:00:55.617268+0000","last_active":"2024-09-21T11:01:02.575021+0000","last_peered":"2024-09-21T11:01:02.575021+0000","last_clean":"2024-09-21T11:01:02.575021+0000","last_became_active":"2024-09-21T11:00:55.579088+0000","last_became_peered":"2024-09-21T11:00:55.579088+0000","last_unstale":"2024-09-21T11:01:02.575021+0000","last_undegraded":"2024-09-21T11:01:02.575021+0000","last_fullsized":"2024-09-21T11:01:02.575021+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:44:27.015942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[{"shards":"1,4,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1489,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.576506+0000","last_change":"2024-09-21T11:00:55.568154+0000","last_active":"2024-09-21T11:01:03.576506+0000","last_peered":"2024-09-21T11:01:03.576506+0000","last_clean":"2024-09-21T11:01:03.576506+0000","last_became_active":"2024-09-21T11:00:55.567225+0000","last_became_peered":"2024-09-21T11:00:55.567225+0000","last_unstale":"2024-09-21T11:01:03.576506+0000","last_undegraded":"2024-09-21T11:01:03.576506+0000","last_fullsized":"2024-09-21T11:01:03.576506+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:18:08.270631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":1953,"reported_epoch":1567,"state":"active+undersized","last_fresh":"2024-09-21T11:01:03.577306+0000","last_change":"2024-09-21T10:58:44.721956+0000","last_active":"2024-09-21T11:01:03.577306+0000","last_peered":"2024-09-21T11:01:03.577306+0000","last_clean":"2024-09-21T10:39:14.280711+0000","last_became_active":"2024-09-21T10:58:44.721956+0000","last_became_peered":"2024-09-21T10:58:44.721956+0000","last_unstale":"2024-09-21T11:01:03.577306+0000","last_undegraded":"2024-09-21T11:01:03.577306+0000","last_fullsized":"2024-09-21T10:58:44.714264+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:52:58.057120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2361,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.575991+0000","last_change":"2024-09-21T11:00:55.585455+0000","last_active":"2024-09-21T11:01:03.575991+0000","last_peered":"2024-09-21T11:01:03.575991+0000","last_clean":"2024-09-21T11:01:03.575991+0000","last_became_active":"2024-09-21T11:00:55.581911+0000","last_became_peered":"2024-09-21T11:00:55.581911+0000","last_unstale":"2024-09-21T11:01:03.575991+0000","last_undegraded":"2024-09-21T11:01:03.575991+0000","last_fullsized":"2024-09-21T11:01:03.575991+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:55:15.280582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":64,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.578739+0000","last_change":"2024-09-21T11:00:55.571707+0000","last_active":"2024-09-21T11:01:03.578739+0000","last_peered":"2024-09-21T11:01:03.578739+0000","last_clean":"2024-09-21T11:01:03.578739+0000","last_became_active":"2024-09-21T11:00:55.547187+0000","last_became_peered":"2024-09-21T11:00:55.547187+0000","last_unstale":"2024-09-21T11:01:03.578739+0000","last_undegraded":"2024-09-21T11:01:03.578739+0000","last_fullsized":"2024-09-21T11:01:03.578739+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:35:39.077011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[{"shards":"1,4,7","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1536,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.571591+0000","last_change":"2024-09-21T11:00:55.515853+0000","last_active":"2024-09-21T11:01:03.571591+0000","last_peered":"2024-09-21T11:01:03.571591+0000","last_clean":"2024-09-21T11:01:03.571591+0000","last_became_active":"2024-09-21T11:00:55.515163+0000","last_became_peered":"2024-09-21T11:00:55.515163+0000","last_unstale":"2024-09-21T11:01:03.571591+0000","last_undegraded":"2024-09-21T11:01:03.571591+0000","last_fullsized":"2024-09-21T11:01:03.571591+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:17:55.185652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,4],"avail_no_missing":["3","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":123,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.577128+0000","last_change":"2024-09-21T11:00:56.016454+0000","last_active":"2024-09-21T11:01:03.577128+0000","last_peered":"2024-09-21T11:01:03.577128+0000","last_clean":"2024-09-21T11:01:03.577128+0000","last_became_active":"2024-09-21T11:00:55.781595+0000","last_became_peered":"2024-09-21T11:00:55.781595+0000","last_unstale":"2024-09-21T11:01:03.577128+0000","last_undegraded":"2024-09-21T11:01:03.577128+0000","last_fullsized":"2024-09-21T11:01:03.577128+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:01:52.476042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[{"shards":"1,4,7","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":759,"reported_epoch":1567,"state":"active+clean","last_fresh":"2024-09-21T11:01:03.571449+0000","last_change":"2024-09-21T11:00:54.493719+0000","last_active":"2024-09-21T11:01:03.571449+0000","last_peered":"2024-09-21T11:01:03.571449+0000","last_clean":"2024-09-21T11:01:03.571449+0000","last_became_active":"2024-09-21T11:00:54.493365+0000","last_became_peered":"2024-09-21T11:00:54.493365+0000","last_unstale":"2024-09-21T11:01:03.571449+0000","last_undegraded":"2024-09-21T11:01:03.571449+0000","last_fullsized":"2024-09-21T11:01:03.571449+0000","mapping_epoch":1563,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:06:26.389408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":2097,"reported_epoch":1566,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:02.575302+0000","last_change":"2024-09-21T11:00:55.504589+0000","last_active":"2024-09-21T11:01:02.575302+0000","last_peered":"2024-09-21T11:01:02.575302+0000","last_clean":"2024-09-21T11:01:02.575302+0000","last_became_active":"2024-09-21T11:00:55.502024+0000","last_became_peered":"2024-09-21T11:00:55.502024+0000","last_unstale":"2024-09-21T11:01:02.575302+0000","last_undegraded":"2024-09-21T11:01:02.575302+0000","last_fullsized":"2024-09-21T11:01:02.575302+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:58:29.211098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1245,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.577336+0000","last_change":"2024-09-21T11:00:55.783298+0000","last_active":"2024-09-21T11:01:03.577336+0000","last_peered":"2024-09-21T11:01:03.577336+0000","last_clean":"2024-09-21T11:01:03.577336+0000","last_became_active":"2024-09-21T11:00:55.782157+0000","last_became_peered":"2024-09-21T11:00:55.782157+0000","last_unstale":"2024-09-21T11:01:03.577336+0000","last_undegraded":"2024-09-21T11:01:03.577336+0000","last_fullsized":"2024-09-21T11:01:03.577336+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:46:43.621342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":110,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.575575+0000","last_change":"2024-09-21T11:00:55.527760+0000","last_active":"2024-09-21T11:01:03.575575+0000","last_peered":"2024-09-21T11:01:03.575575+0000","last_clean":"2024-09-21T11:01:03.575575+0000","last_became_active":"2024-09-21T11:00:55.527357+0000","last_became_peered":"2024-09-21T11:00:55.527357+0000","last_unstale":"2024-09-21T11:01:03.575575+0000","last_undegraded":"2024-09-21T11:01:03.575575+0000","last_fullsized":"2024-09-21T11:01:03.575575+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:39:42.779064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2436,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.578742+0000","last_change":"2024-09-21T11:00:55.549947+0000","last_active":"2024-09-21T11:01:03.578742+0000","last_peered":"2024-09-21T11:01:03.578742+0000","last_clean":"2024-09-21T11:01:03.578742+0000","last_became_active":"2024-09-21T11:00:55.549560+0000","last_became_peered":"2024-09-21T11:00:55.549560+0000","last_unstale":"2024-09-21T11:01:03.578742+0000","last_undegraded":"2024-09-21T11:01:03.578742+0000","last_fullsized":"2024-09-21T11:01:03.578742+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:22:55.323455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2434,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.576163+0000","last_change":"2024-09-21T11:00:55.510076+0000","last_active":"2024-09-21T11:01:03.576163+0000","last_peered":"2024-09-21T11:01:03.576163+0000","last_clean":"2024-09-21T11:01:03.576163+0000","last_became_active":"2024-09-21T11:00:55.509382+0000","last_became_peered":"2024-09-21T11:00:55.509382+0000","last_unstale":"2024-09-21T11:01:03.576163+0000","last_undegraded":"2024-09-21T11:01:03.576163+0000","last_fullsized":"2024-09-21T11:01:03.576163+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:44:26.927743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3,4],"avail_no_missing":["1","3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":144,"reported_epoch":1566,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:02.578122+0000","last_change":"2024-09-21T11:00:55.538708+0000","last_active":"2024-09-21T11:01:02.578122+0000","last_peered":"2024-09-21T11:01:02.578122+0000","last_clean":"2024-09-21T11:01:02.578122+0000","last_became_active":"2024-09-21T11:00:55.533135+0000","last_became_peered":"2024-09-21T11:00:55.533135+0000","last_unstale":"2024-09-21T11:01:02.578122+0000","last_undegraded":"2024-09-21T11:01:02.578122+0000","last_fullsized":"2024-09-21T11:01:02.578122+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:46.032127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2373,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.576047+0000","last_change":"2024-09-21T11:00:55.567769+0000","last_active":"2024-09-21T11:01:03.576047+0000","last_peered":"2024-09-21T11:01:03.576047+0000","last_clean":"2024-09-21T11:01:03.576047+0000","last_became_active":"2024-09-21T11:00:55.566903+0000","last_became_peered":"2024-09-21T11:00:55.566903+0000","last_unstale":"2024-09-21T11:01:03.576047+0000","last_undegraded":"2024-09-21T11:01:03.576047+0000","last_fullsized":"2024-09-21T11:01:03.576047+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:55:39.331474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1573,"reported_epoch":1567,"state":"active+undersized","last_fresh":"2024-09-21T11:01:03.578392+0000","last_change":"2024-09-21T10:58:44.741842+0000","last_active":"2024-09-21T11:01:03.578392+0000","last_peered":"2024-09-21T11:01:03.578392+0000","last_clean":"2024-09-21T10:56:19.067629+0000","last_became_active":"2024-09-21T10:58:44.741842+0000","last_became_peered":"2024-09-21T10:58:44.741842+0000","last_unstale":"2024-09-21T11:01:03.578392+0000","last_undegraded":"2024-09-21T11:01:03.578392+0000","last_fullsized":"2024-09-21T10:58:44.714923+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:14:20.138349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2371,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.575861+0000","last_change":"2024-09-21T11:00:55.567320+0000","last_active":"2024-09-21T11:01:03.575861+0000","last_peered":"2024-09-21T11:01:03.575861+0000","last_clean":"2024-09-21T11:01:03.575861+0000","last_became_active":"2024-09-21T11:00:55.566390+0000","last_became_peered":"2024-09-21T11:00:55.566390+0000","last_unstale":"2024-09-21T11:01:03.575861+0000","last_undegraded":"2024-09-21T11:01:03.575861+0000","last_fullsized":"2024-09-21T11:01:03.575861+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:08.474800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2302,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.576978+0000","last_change":"2024-09-21T11:00:55.591606+0000","last_active":"2024-09-21T11:01:03.576978+0000","last_peered":"2024-09-21T11:01:03.576978+0000","last_clean":"2024-09-21T11:01:03.576978+0000","last_became_active":"2024-09-21T11:00:55.591189+0000","last_became_peered":"2024-09-21T11:00:55.591189+0000","last_unstale":"2024-09-21T11:01:03.576978+0000","last_undegraded":"2024-09-21T11:01:03.576978+0000","last_fullsized":"2024-09-21T11:01:03.576978+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:14:10.220545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2452,"reported_epoch":1567,"state":"active+undersized","last_fresh":"2024-09-21T11:01:03.571125+0000","last_change":"2024-09-21T10:58:44.716750+0000","last_active":"2024-09-21T11:01:03.571125+0000","last_peered":"2024-09-21T11:01:03.571125+0000","last_clean":"2024-09-21T10:58:41.708409+0000","last_became_active":"2024-09-21T10:58:44.716750+0000","last_became_peered":"2024-09-21T10:58:44.716750+0000","last_unstale":"2024-09-21T11:01:03.571125+0000","last_undegraded":"2024-09-21T11:01:03.571125+0000","last_fullsized":"2024-09-21T10:58:44.710874+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:57:10.925901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":143,"reported_epoch":1566,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:02.578061+0000","last_change":"2024-09-21T11:00:55.540199+0000","last_active":"2024-09-21T11:01:02.578061+0000","last_peered":"2024-09-21T11:01:02.578061+0000","last_clean":"2024-09-21T11:01:02.578061+0000","last_became_active":"2024-09-21T11:00:55.532419+0000","last_became_peered":"2024-09-21T11:00:55.532419+0000","last_unstale":"2024-09-21T11:01:02.578061+0000","last_undegraded":"2024-09-21T11:01:02.578061+0000","last_fullsized":"2024-09-21T11:01:02.578061+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:21:13.653686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":161,"reported_epoch":1566,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:02.576683+0000","last_change":"2024-09-21T11:00:55.501887+0000","last_active":"2024-09-21T11:01:02.576683+0000","last_peered":"2024-09-21T11:01:02.576683+0000","last_clean":"2024-09-21T11:01:02.576683+0000","last_became_active":"2024-09-21T11:00:55.500654+0000","last_became_peered":"2024-09-21T11:00:55.500654+0000","last_unstale":"2024-09-21T11:01:02.576683+0000","last_undegraded":"2024-09-21T11:01:02.576683+0000","last_fullsized":"2024-09-21T11:01:02.576683+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:21:09.155952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,4],"avail_no_missing":["7","3","4"],"object_location_counts":[{"shards":"3,4,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2453,"reported_epoch":1567,"state":"active+undersized","last_fresh":"2024-09-21T11:01:03.575828+0000","last_change":"2024-09-21T10:58:44.731487+0000","last_active":"2024-09-21T11:01:03.575828+0000","last_peered":"2024-09-21T11:01:03.575828+0000","last_clean":"2024-09-21T10:58:41.710517+0000","last_became_active":"2024-09-21T10:58:44.731487+0000","last_became_peered":"2024-09-21T10:58:44.731487+0000","last_unstale":"2024-09-21T11:01:03.575828+0000","last_undegraded":"2024-09-21T11:01:03.575828+0000","last_fullsized":"2024-09-21T10:58:44.715329+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:51:30.090288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":1470,"reported_epoch":1566,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:02.576433+0000","last_change":"2024-09-21T11:00:56.040727+0000","last_active":"2024-09-21T11:01:02.576433+0000","last_peered":"2024-09-21T11:01:02.576433+0000","last_clean":"2024-09-21T11:01:02.576433+0000","last_became_active":"2024-09-21T11:00:55.778108+0000","last_became_peered":"2024-09-21T11:00:55.778108+0000","last_unstale":"2024-09-21T11:01:02.576433+0000","last_undegraded":"2024-09-21T11:01:02.576433+0000","last_fullsized":"2024-09-21T11:01:02.576433+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:15:06.144823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":19024,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[{"shards":"1,4,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":642,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.577234+0000","last_change":"2024-09-21T11:00:55.530433+0000","last_active":"2024-09-21T11:01:03.577234+0000","last_peered":"2024-09-21T11:01:03.577234+0000","last_clean":"2024-09-21T11:01:03.577234+0000","last_became_active":"2024-09-21T11:00:55.529197+0000","last_became_peered":"2024-09-21T11:00:55.529197+0000","last_unstale":"2024-09-21T11:01:03.577234+0000","last_undegraded":"2024-09-21T11:01:03.577234+0000","last_fullsized":"2024-09-21T11:01:03.577234+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:32:36.035714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"346'149","reported_seq":1011,"reported_epoch":1566,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:02.576282+0000","last_change":"2024-09-21T11:00:55.781203+0000","last_active":"2024-09-21T11:01:02.576282+0000","last_peered":"2024-09-21T11:01:02.576282+0000","last_clean":"2024-09-21T11:01:02.576282+0000","last_became_active":"2024-09-21T11:00:55.778102+0000","last_became_peered":"2024-09-21T11:00:55.778102+0000","last_unstale":"2024-09-21T11:01:02.576282+0000","last_undegraded":"2024-09-21T11:01:02.576282+0000","last_fullsized":"2024-09-21T11:01:02.576282+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1906749,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1488,"reported_epoch":1566,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:02.574702+0000","last_change":"2024-09-21T11:00:55.576275+0000","last_active":"2024-09-21T11:01:02.574702+0000","last_peered":"2024-09-21T11:01:02.574702+0000","last_clean":"2024-09-21T11:01:02.574702+0000","last_became_active":"2024-09-21T11:00:55.573891+0000","last_became_peered":"2024-09-21T11:00:55.573891+0000","last_unstale":"2024-09-21T11:01:02.574702+0000","last_undegraded":"2024-09-21T11:01:02.574702+0000","last_fullsized":"2024-09-21T11:01:02.574702+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:47.341409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1577,"reported_epoch":1566,"state":"active+undersized","last_fresh":"2024-09-21T11:01:02.575565+0000","last_change":"2024-09-21T10:58:44.730085+0000","last_active":"2024-09-21T11:01:02.575565+0000","last_peered":"2024-09-21T11:01:02.575565+0000","last_clean":"2024-09-21T10:56:19.067690+0000","last_became_active":"2024-09-21T10:58:44.730085+0000","last_became_peered":"2024-09-21T10:58:44.730085+0000","last_unstale":"2024-09-21T11:01:02.575565+0000","last_undegraded":"2024-09-21T11:01:02.575565+0000","last_fullsized":"2024-09-21T10:58:44.718844+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:17:19.473681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":322,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.578646+0000","last_change":"2024-09-21T11:00:55.531863+0000","last_active":"2024-09-21T11:01:03.578646+0000","last_peered":"2024-09-21T11:01:03.578646+0000","last_clean":"2024-09-21T11:01:03.578646+0000","last_became_active":"2024-09-21T11:00:55.530945+0000","last_became_peered":"2024-09-21T11:00:55.530945+0000","last_unstale":"2024-09-21T11:01:03.578646+0000","last_undegraded":"2024-09-21T11:01:03.578646+0000","last_fullsized":"2024-09-21T11:01:03.578646+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:59:18.812299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[{"shards":"1,4,7","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"346'149","reported_seq":637,"reported_epoch":1566,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:02.576198+0000","last_change":"2024-09-21T11:00:55.547908+0000","last_active":"2024-09-21T11:01:02.576198+0000","last_peered":"2024-09-21T11:01:02.576198+0000","last_clean":"2024-09-21T11:01:02.576198+0000","last_became_active":"2024-09-21T11:00:55.517822+0000","last_became_peered":"2024-09-21T11:00:55.517822+0000","last_unstale":"2024-09-21T11:01:02.576198+0000","last_undegraded":"2024-09-21T11:01:02.576198+0000","last_fullsized":"2024-09-21T11:01:02.576198+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:22.051403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:22.051403+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2278,"reported_epoch":1566,"state":"active+undersized","last_fresh":"2024-09-21T11:01:02.576470+0000","last_change":"2024-09-21T10:58:44.725359+0000","last_active":"2024-09-21T11:01:02.576470+0000","last_peered":"2024-09-21T11:01:02.576470+0000","last_clean":"2024-09-21T10:58:41.699669+0000","last_became_active":"2024-09-21T10:58:44.725359+0000","last_became_peered":"2024-09-21T10:58:44.725359+0000","last_unstale":"2024-09-21T11:01:02.576470+0000","last_undegraded":"2024-09-21T11:01:02.576470+0000","last_fullsized":"2024-09-21T10:58:44.720129+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:09:05.546754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2440,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.579022+0000","last_change":"2024-09-21T11:00:55.513558+0000","last_active":"2024-09-21T11:01:03.579022+0000","last_peered":"2024-09-21T11:01:03.579022+0000","last_clean":"2024-09-21T11:01:03.579022+0000","last_became_active":"2024-09-21T11:00:55.512903+0000","last_became_peered":"2024-09-21T11:00:55.512903+0000","last_unstale":"2024-09-21T11:01:03.579022+0000","last_undegraded":"2024-09-21T11:01:03.579022+0000","last_fullsized":"2024-09-21T11:01:03.579022+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:11:57.971647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3,4],"avail_no_missing":["1","3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":2499,"reported_epoch":1567,"state":"active+clean","last_fresh":"2024-09-21T11:01:03.571614+0000","last_change":"2024-09-21T11:00:54.513276+0000","last_active":"2024-09-21T11:01:03.571614+0000","last_peered":"2024-09-21T11:01:03.571614+0000","last_clean":"2024-09-21T11:01:03.571614+0000","last_became_active":"2024-09-21T11:00:54.502716+0000","last_became_peered":"2024-09-21T11:00:54.502716+0000","last_unstale":"2024-09-21T11:01:03.571614+0000","last_undegraded":"2024-09-21T11:01:03.571614+0000","last_fullsized":"2024-09-21T11:01:03.571614+0000","mapping_epoch":1563,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:53:14.900173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"346'149","reported_seq":753,"reported_epoch":1566,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:02.577578+0000","last_change":"2024-09-21T11:00:55.537831+0000","last_active":"2024-09-21T11:01:02.577578+0000","last_peered":"2024-09-21T11:01:02.577578+0000","last_clean":"2024-09-21T11:01:02.577578+0000","last_became_active":"2024-09-21T11:00:55.530727+0000","last_became_peered":"2024-09-21T11:00:55.530727+0000","last_unstale":"2024-09-21T11:01:02.577578+0000","last_undegraded":"2024-09-21T11:01:02.577578+0000","last_fullsized":"2024-09-21T11:01:02.577578+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:41.011085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:41.011085+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:29:51.910262+0000","scrub_duration":0.0061250310000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2314,"reported_epoch":1566,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:02.575343+0000","last_change":"2024-09-21T11:00:55.504937+0000","last_active":"2024-09-21T11:01:02.575343+0000","last_peered":"2024-09-21T11:01:02.575343+0000","last_clean":"2024-09-21T11:01:02.575343+0000","last_became_active":"2024-09-21T11:00:55.502841+0000","last_became_peered":"2024-09-21T11:00:55.502841+0000","last_unstale":"2024-09-21T11:01:02.575343+0000","last_undegraded":"2024-09-21T11:01:02.575343+0000","last_fullsized":"2024-09-21T11:01:02.575343+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:04:24.946006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":144,"reported_epoch":1566,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:02.576455+0000","last_change":"2024-09-21T11:00:55.548899+0000","last_active":"2024-09-21T11:01:02.576455+0000","last_peered":"2024-09-21T11:01:02.576455+0000","last_clean":"2024-09-21T11:01:02.576455+0000","last_became_active":"2024-09-21T11:00:55.518409+0000","last_became_peered":"2024-09-21T11:00:55.518409+0000","last_unstale":"2024-09-21T11:01:02.576455+0000","last_undegraded":"2024-09-21T11:01:02.576455+0000","last_fullsized":"2024-09-21T11:01:02.576455+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:16:07.983523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2490,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.577354+0000","last_change":"2024-09-21T11:00:55.782899+0000","last_active":"2024-09-21T11:01:03.577354+0000","last_peered":"2024-09-21T11:01:03.577354+0000","last_clean":"2024-09-21T11:01:03.577354+0000","last_became_active":"2024-09-21T11:00:55.782000+0000","last_became_peered":"2024-09-21T11:00:55.782000+0000","last_unstale":"2024-09-21T11:01:03.577354+0000","last_undegraded":"2024-09-21T11:01:03.577354+0000","last_fullsized":"2024-09-21T11:01:03.577354+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:58:21.588298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"346'149","reported_seq":773,"reported_epoch":1566,"state":"active+undersized","last_fresh":"2024-09-21T11:01:02.575374+0000","last_change":"2024-09-21T10:58:44.729467+0000","last_active":"2024-09-21T11:01:02.575374+0000","last_peered":"2024-09-21T11:01:02.575374+0000","last_clean":"2024-09-21T10:52:16.390617+0000","last_became_active":"2024-09-21T10:58:44.729467+0000","last_became_peered":"2024-09-21T10:58:44.729467+0000","last_unstale":"2024-09-21T11:01:02.575374+0000","last_undegraded":"2024-09-21T11:01:02.575374+0000","last_fullsized":"2024-09-21T10:58:44.718172+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1511,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:44.018645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:44.018645+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:25:43.141373+0000","scrub_duration":0.0012789399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2442,"reported_epoch":1567,"state":"active+clean","last_fresh":"2024-09-21T11:01:03.575627+0000","last_change":"2024-09-21T11:00:54.509325+0000","last_active":"2024-09-21T11:01:03.575627+0000","last_peered":"2024-09-21T11:01:03.575627+0000","last_clean":"2024-09-21T11:01:03.575627+0000","last_became_active":"2024-09-21T11:00:54.508998+0000","last_became_peered":"2024-09-21T11:00:54.508998+0000","last_unstale":"2024-09-21T11:01:03.575627+0000","last_undegraded":"2024-09-21T11:01:03.575627+0000","last_fullsized":"2024-09-21T11:01:03.575627+0000","mapping_epoch":1563,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:39:41.166854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2509,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.576937+0000","last_change":"2024-09-21T11:00:55.782489+0000","last_active":"2024-09-21T11:01:03.576937+0000","last_peered":"2024-09-21T11:01:03.576937+0000","last_clean":"2024-09-21T11:01:03.576937+0000","last_became_active":"2024-09-21T11:00:55.781210+0000","last_became_peered":"2024-09-21T11:00:55.781210+0000","last_unstale":"2024-09-21T11:01:03.576937+0000","last_undegraded":"2024-09-21T11:01:03.576937+0000","last_fullsized":"2024-09-21T11:01:03.576937+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:44:37.556969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1465,"reported_epoch":1566,"state":"active+undersized","last_fresh":"2024-09-21T11:01:02.576333+0000","last_change":"2024-09-21T10:58:44.727795+0000","last_active":"2024-09-21T11:01:02.576333+0000","last_peered":"2024-09-21T11:01:02.576333+0000","last_clean":"2024-09-21T10:52:16.391715+0000","last_became_active":"2024-09-21T10:58:44.727795+0000","last_became_peered":"2024-09-21T10:58:44.727795+0000","last_unstale":"2024-09-21T11:01:02.576333+0000","last_undegraded":"2024-09-21T11:01:02.576333+0000","last_fullsized":"2024-09-21T10:58:44.721466+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:46:04.781703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"346'149","reported_seq":859,"reported_epoch":1566,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:02.575054+0000","last_change":"2024-09-21T11:00:55.575656+0000","last_active":"2024-09-21T11:01:02.575054+0000","last_peered":"2024-09-21T11:01:02.575054+0000","last_clean":"2024-09-21T11:01:02.575054+0000","last_became_active":"2024-09-21T11:00:55.573544+0000","last_became_peered":"2024-09-21T11:00:55.573544+0000","last_unstale":"2024-09-21T11:01:02.575054+0000","last_undegraded":"2024-09-21T11:01:02.575054+0000","last_fullsized":"2024-09-21T11:01:02.575054+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:16.021843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:16.021843+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[{"shards":"1,4,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2507,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.578476+0000","last_change":"2024-09-21T11:00:55.782867+0000","last_active":"2024-09-21T11:01:03.578476+0000","last_peered":"2024-09-21T11:01:03.578476+0000","last_clean":"2024-09-21T11:01:03.578476+0000","last_became_active":"2024-09-21T11:00:55.781632+0000","last_became_peered":"2024-09-21T11:00:55.781632+0000","last_unstale":"2024-09-21T11:01:03.578476+0000","last_undegraded":"2024-09-21T11:01:03.578476+0000","last_fullsized":"2024-09-21T11:01:03.578476+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:55:36.805713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1470,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.576618+0000","last_change":"2024-09-21T11:00:55.529566+0000","last_active":"2024-09-21T11:01:03.576618+0000","last_peered":"2024-09-21T11:01:03.576618+0000","last_clean":"2024-09-21T11:01:03.576618+0000","last_became_active":"2024-09-21T11:00:55.528339+0000","last_became_peered":"2024-09-21T11:00:55.528339+0000","last_unstale":"2024-09-21T11:01:03.576618+0000","last_undegraded":"2024-09-21T11:01:03.576618+0000","last_fullsized":"2024-09-21T11:01:03.576618+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:30:48.437923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":398,"reported_epoch":1567,"state":"active+undersized+degraded","last_fresh":"2024-09-21T11:01:03.575426+0000","last_change":"2024-09-21T10:58:44.730985+0000","last_active":"2024-09-21T11:01:03.575426+0000","last_peered":"2024-09-21T11:01:03.575426+0000","last_clean":"2024-09-21T10:58:41.710261+0000","last_became_active":"2024-09-21T10:58:44.730985+0000","last_became_peered":"2024-09-21T10:58:44.730985+0000","last_unstale":"2024-09-21T11:01:03.575426+0000","last_undegraded":"2024-09-21T10:58:44.711579+0000","last_fullsized":"2024-09-21T10:58:44.711482+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:09:49.337655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"346'149","reported_seq":957,"reported_epoch":1566,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:02.575089+0000","last_change":"2024-09-21T11:00:55.520154+0000","last_active":"2024-09-21T11:01:02.575089+0000","last_peered":"2024-09-21T11:01:02.575089+0000","last_clean":"2024-09-21T11:01:02.575089+0000","last_became_active":"2024-09-21T11:00:55.518484+0000","last_became_peered":"2024-09-21T11:00:55.518484+0000","last_unstale":"2024-09-21T11:01:02.575089+0000","last_undegraded":"2024-09-21T11:01:02.575089+0000","last_fullsized":"2024-09-21T11:01:02.575089+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:21.445783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:21.445783+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:26:48.796301+0000","scrub_duration":0.0018624990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":184,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.576735+0000","last_change":"2024-09-21T11:00:55.591612+0000","last_active":"2024-09-21T11:01:03.576735+0000","last_peered":"2024-09-21T11:01:03.576735+0000","last_clean":"2024-09-21T11:01:03.576735+0000","last_became_active":"2024-09-21T11:00:55.591207+0000","last_became_peered":"2024-09-21T11:00:55.591207+0000","last_unstale":"2024-09-21T11:01:03.576735+0000","last_undegraded":"2024-09-21T11:01:03.576735+0000","last_fullsized":"2024-09-21T11:01:03.576735+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:11:55.949333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2451,"reported_epoch":1567,"state":"active+undersized","last_fresh":"2024-09-21T11:01:03.576159+0000","last_change":"2024-09-21T10:58:44.741984+0000","last_active":"2024-09-21T11:01:03.576159+0000","last_peered":"2024-09-21T11:01:03.576159+0000","last_clean":"2024-09-21T10:58:41.709959+0000","last_became_active":"2024-09-21T10:58:44.741984+0000","last_became_peered":"2024-09-21T10:58:44.741984+0000","last_unstale":"2024-09-21T11:01:03.576159+0000","last_undegraded":"2024-09-21T11:01:03.576159+0000","last_fullsized":"2024-09-21T10:58:44.716350+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:50:18.501401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1123,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.575974+0000","last_change":"2024-09-21T11:00:55.585784+0000","last_active":"2024-09-21T11:01:03.575974+0000","last_peered":"2024-09-21T11:01:03.575974+0000","last_clean":"2024-09-21T11:01:03.575974+0000","last_became_active":"2024-09-21T11:00:55.582330+0000","last_became_peered":"2024-09-21T11:00:55.582330+0000","last_unstale":"2024-09-21T11:01:03.575974+0000","last_undegraded":"2024-09-21T11:01:03.575974+0000","last_fullsized":"2024-09-21T11:01:03.575974+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:34:03.183083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"346'149","reported_seq":1143,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.577109+0000","last_change":"2024-09-21T11:00:55.525493+0000","last_active":"2024-09-21T11:01:03.577109+0000","last_peered":"2024-09-21T11:01:03.577109+0000","last_clean":"2024-09-21T11:01:03.577109+0000","last_became_active":"2024-09-21T11:00:55.525078+0000","last_became_peered":"2024-09-21T11:00:55.525078+0000","last_unstale":"2024-09-21T11:01:03.577109+0000","last_undegraded":"2024-09-21T11:01:03.577109+0000","last_fullsized":"2024-09-21T11:01:03.577109+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2485,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.578475+0000","last_change":"2024-09-21T11:00:55.531392+0000","last_active":"2024-09-21T11:01:03.578475+0000","last_peered":"2024-09-21T11:01:03.578475+0000","last_clean":"2024-09-21T11:01:03.578475+0000","last_became_active":"2024-09-21T11:00:55.528516+0000","last_became_peered":"2024-09-21T11:00:55.528516+0000","last_unstale":"2024-09-21T11:01:03.578475+0000","last_undegraded":"2024-09-21T11:01:03.578475+0000","last_fullsized":"2024-09-21T11:01:03.578475+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:44:49.519148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2227,"reported_epoch":1566,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:02.574840+0000","last_change":"2024-09-21T11:00:55.575678+0000","last_active":"2024-09-21T11:01:02.574840+0000","last_peered":"2024-09-21T11:01:02.574840+0000","last_clean":"2024-09-21T11:01:02.574840+0000","last_became_active":"2024-09-21T11:00:55.573384+0000","last_became_peered":"2024-09-21T11:00:55.573384+0000","last_unstale":"2024-09-21T11:01:02.574840+0000","last_undegraded":"2024-09-21T11:01:02.574840+0000","last_fullsized":"2024-09-21T11:01:02.574840+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:03.697009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":225,"reported_epoch":1566,"state":"active+clean","last_fresh":"2024-09-21T11:01:02.576979+0000","last_change":"2024-09-21T10:51:27.426006+0000","last_active":"2024-09-21T11:01:02.576979+0000","last_peered":"2024-09-21T11:01:02.576979+0000","last_clean":"2024-09-21T11:01:02.576979+0000","last_became_active":"2024-09-21T10:51:27.425698+0000","last_became_peered":"2024-09-21T10:51:27.425698+0000","last_unstale":"2024-09-21T11:01:02.576979+0000","last_undegraded":"2024-09-21T11:01:02.576979+0000","last_fullsized":"2024-09-21T11:01:02.576979+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:35:13.210330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1389,"reported_epoch":1566,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:02.577429+0000","last_change":"2024-09-21T11:00:55.549787+0000","last_active":"2024-09-21T11:01:02.577429+0000","last_peered":"2024-09-21T11:01:02.577429+0000","last_clean":"2024-09-21T11:01:02.577429+0000","last_became_active":"2024-09-21T11:00:55.520045+0000","last_became_peered":"2024-09-21T11:00:55.520045+0000","last_unstale":"2024-09-21T11:01:02.577429+0000","last_undegraded":"2024-09-21T11:01:02.577429+0000","last_fullsized":"2024-09-21T11:01:02.577429+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:14:32.694688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2318,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.574852+0000","last_change":"2024-09-21T11:00:55.529203+0000","last_active":"2024-09-21T11:01:03.574852+0000","last_peered":"2024-09-21T11:01:03.574852+0000","last_clean":"2024-09-21T11:01:03.574852+0000","last_became_active":"2024-09-21T11:00:55.528652+0000","last_became_peered":"2024-09-21T11:00:55.528652+0000","last_unstale":"2024-09-21T11:01:03.574852+0000","last_undegraded":"2024-09-21T11:01:03.574852+0000","last_fullsized":"2024-09-21T11:01:03.574852+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:38:52.552159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":1556,"reported_epoch":1566,"state":"active+undersized+degraded","last_fresh":"2024-09-21T11:01:02.575450+0000","last_change":"2024-09-21T10:58:44.722515+0000","last_active":"2024-09-21T11:01:02.575450+0000","last_peered":"2024-09-21T11:01:02.575450+0000","last_clean":"2024-09-21T10:58:42.407038+0000","last_became_active":"2024-09-21T10:58:44.722515+0000","last_became_peered":"2024-09-21T10:58:44.722515+0000","last_unstale":"2024-09-21T11:01:02.575450+0000","last_undegraded":"2024-09-21T10:58:44.712515+0000","last_fullsized":"2024-09-21T10:58:44.712401+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:10.249611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":902,"reported_epoch":1566,"state":"active+undersized","last_fresh":"2024-09-21T11:01:02.576860+0000","last_change":"2024-09-21T10:58:44.725921+0000","last_active":"2024-09-21T11:01:02.576860+0000","last_peered":"2024-09-21T11:01:02.576860+0000","last_clean":"2024-09-21T10:58:41.699976+0000","last_became_active":"2024-09-21T10:58:44.725921+0000","last_became_peered":"2024-09-21T10:58:44.725921+0000","last_unstale":"2024-09-21T11:01:02.576860+0000","last_undegraded":"2024-09-21T11:01:02.576860+0000","last_fullsized":"2024-09-21T10:58:44.720891+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1469,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:20.458421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:20.458421+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:10:27.439795+0000","scrub_duration":0.001528569,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"346'149","reported_seq":711,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.574914+0000","last_change":"2024-09-21T11:00:55.527827+0000","last_active":"2024-09-21T11:01:03.574914+0000","last_peered":"2024-09-21T11:01:03.574914+0000","last_clean":"2024-09-21T11:01:03.574914+0000","last_became_active":"2024-09-21T11:00:55.527440+0000","last_became_peered":"2024-09-21T11:00:55.527440+0000","last_unstale":"2024-09-21T11:01:03.574914+0000","last_undegraded":"2024-09-21T11:01:03.574914+0000","last_fullsized":"2024-09-21T11:01:03.574914+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:42.008235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:42.008235+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:42:51.629757+0000","scrub_duration":0.011129834999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2437,"reported_epoch":1567,"state":"active+clean","last_fresh":"2024-09-21T11:01:03.577144+0000","last_change":"2024-09-21T11:00:54.511549+0000","last_active":"2024-09-21T11:01:03.577144+0000","last_peered":"2024-09-21T11:01:03.577144+0000","last_clean":"2024-09-21T11:01:03.577144+0000","last_became_active":"2024-09-21T11:00:54.510524+0000","last_became_peered":"2024-09-21T11:00:54.510524+0000","last_unstale":"2024-09-21T11:01:03.577144+0000","last_undegraded":"2024-09-21T11:01:03.577144+0000","last_fullsized":"2024-09-21T11:01:03.577144+0000","mapping_epoch":1563,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:11:20.054140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2345,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.576331+0000","last_change":"2024-09-21T11:00:55.567710+0000","last_active":"2024-09-21T11:01:03.576331+0000","last_peered":"2024-09-21T11:01:03.576331+0000","last_clean":"2024-09-21T11:01:03.576331+0000","last_became_active":"2024-09-21T11:00:55.566847+0000","last_became_peered":"2024-09-21T11:00:55.566847+0000","last_unstale":"2024-09-21T11:01:03.576331+0000","last_undegraded":"2024-09-21T11:01:03.576331+0000","last_fullsized":"2024-09-21T11:01:03.576331+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:31:26.145308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1491,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.578928+0000","last_change":"2024-09-21T11:00:55.545076+0000","last_active":"2024-09-21T11:01:03.578928+0000","last_peered":"2024-09-21T11:01:03.578928+0000","last_clean":"2024-09-21T11:01:03.578928+0000","last_became_active":"2024-09-21T11:00:55.544747+0000","last_became_peered":"2024-09-21T11:00:55.544747+0000","last_unstale":"2024-09-21T11:01:03.578928+0000","last_undegraded":"2024-09-21T11:01:03.578928+0000","last_fullsized":"2024-09-21T11:01:03.578928+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:07.698256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1432,"reported_epoch":1566,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:02.575534+0000","last_change":"2024-09-21T11:00:55.574806+0000","last_active":"2024-09-21T11:01:02.575534+0000","last_peered":"2024-09-21T11:01:02.575534+0000","last_clean":"2024-09-21T11:01:02.575534+0000","last_became_active":"2024-09-21T11:00:55.572750+0000","last_became_peered":"2024-09-21T11:00:55.572750+0000","last_unstale":"2024-09-21T11:01:02.575534+0000","last_undegraded":"2024-09-21T11:01:02.575534+0000","last_fullsized":"2024-09-21T11:01:02.575534+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:36.439306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"346'149","reported_seq":965,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.574861+0000","last_change":"2024-09-21T11:01:02.409463+0000","last_active":"2024-09-21T11:01:03.574861+0000","last_peered":"2024-09-21T11:01:03.574861+0000","last_clean":"2024-09-21T11:01:03.574861+0000","last_became_active":"2024-09-21T11:00:55.566385+0000","last_became_peered":"2024-09-21T11:00:55.566385+0000","last_unstale":"2024-09-21T11:01:03.574861+0000","last_undegraded":"2024-09-21T11:01:03.574861+0000","last_fullsized":"2024-09-21T11:01:03.574861+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T09:51:06.099351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1056413,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1392,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.578847+0000","last_change":"2024-09-21T11:00:55.550824+0000","last_active":"2024-09-21T11:01:03.578847+0000","last_peered":"2024-09-21T11:01:03.578847+0000","last_clean":"2024-09-21T11:01:03.578847+0000","last_became_active":"2024-09-21T11:00:55.550426+0000","last_became_peered":"2024-09-21T11:00:55.550426+0000","last_unstale":"2024-09-21T11:01:03.578847+0000","last_undegraded":"2024-09-21T11:01:03.578847+0000","last_fullsized":"2024-09-21T11:01:03.578847+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:55:15.443717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1537,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.575715+0000","last_change":"2024-09-21T11:00:55.511656+0000","last_active":"2024-09-21T11:01:03.575715+0000","last_peered":"2024-09-21T11:01:03.575715+0000","last_clean":"2024-09-21T11:01:03.575715+0000","last_became_active":"2024-09-21T11:00:55.511269+0000","last_became_peered":"2024-09-21T11:00:55.511269+0000","last_unstale":"2024-09-21T11:01:03.575715+0000","last_undegraded":"2024-09-21T11:01:03.575715+0000","last_fullsized":"2024-09-21T11:01:03.575715+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:53:11.193147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3,4],"avail_no_missing":["1","3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"346'149","reported_seq":1027,"reported_epoch":1566,"state":"active+undersized","last_fresh":"2024-09-21T11:01:02.575389+0000","last_change":"2024-09-21T10:58:44.720907+0000","last_active":"2024-09-21T11:01:02.575389+0000","last_peered":"2024-09-21T11:01:02.575389+0000","last_clean":"2024-09-21T10:58:41.701314+0000","last_became_active":"2024-09-21T10:58:44.720907+0000","last_became_peered":"2024-09-21T10:58:44.720907+0000","last_unstale":"2024-09-21T11:01:02.575389+0000","last_undegraded":"2024-09-21T11:01:02.575389+0000","last_fullsized":"2024-09-21T10:58:44.712364+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:17.418646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:17.418646+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:33.791729+0000","scrub_duration":0.00185648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":744,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.571646+0000","last_change":"2024-09-21T11:00:55.490690+0000","last_active":"2024-09-21T11:01:03.571646+0000","last_peered":"2024-09-21T11:01:03.571646+0000","last_clean":"2024-09-21T11:01:03.571646+0000","last_became_active":"2024-09-21T11:00:55.490139+0000","last_became_peered":"2024-09-21T11:00:55.490139+0000","last_unstale":"2024-09-21T11:01:03.571646+0000","last_undegraded":"2024-09-21T11:01:03.571646+0000","last_fullsized":"2024-09-21T11:01:03.571646+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:06:38.669796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,4],"avail_no_missing":["3","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":2261,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.576803+0000","last_change":"2024-09-21T11:00:55.527641+0000","last_active":"2024-09-21T11:01:03.576803+0000","last_peered":"2024-09-21T11:01:03.576803+0000","last_clean":"2024-09-21T11:01:03.576803+0000","last_became_active":"2024-09-21T11:00:55.527198+0000","last_became_peered":"2024-09-21T11:00:55.527198+0000","last_unstale":"2024-09-21T11:01:03.576803+0000","last_undegraded":"2024-09-21T11:01:03.576803+0000","last_fullsized":"2024-09-21T11:01:03.576803+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:52:26.209947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[{"shards":"1,4,7","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":905,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.578353+0000","last_change":"2024-09-21T11:00:55.530009+0000","last_active":"2024-09-21T11:01:03.578353+0000","last_peered":"2024-09-21T11:01:03.578353+0000","last_clean":"2024-09-21T11:01:03.578353+0000","last_became_active":"2024-09-21T11:00:55.528816+0000","last_became_peered":"2024-09-21T11:00:55.528816+0000","last_unstale":"2024-09-21T11:01:03.578353+0000","last_undegraded":"2024-09-21T11:01:03.578353+0000","last_fullsized":"2024-09-21T11:01:03.578353+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:47:02.758615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":787,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.576936+0000","last_change":"2024-09-21T11:00:55.782347+0000","last_active":"2024-09-21T11:01:03.576936+0000","last_peered":"2024-09-21T11:01:03.576936+0000","last_clean":"2024-09-21T11:01:03.576936+0000","last_became_active":"2024-09-21T11:00:55.781125+0000","last_became_peered":"2024-09-21T11:00:55.781125+0000","last_unstale":"2024-09-21T11:01:03.576936+0000","last_undegraded":"2024-09-21T11:01:03.576936+0000","last_fullsized":"2024-09-21T11:01:03.576936+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:04:41.603975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1565,"reported_epoch":1567,"state":"active+undersized","last_fresh":"2024-09-21T11:01:03.575175+0000","last_change":"2024-09-21T10:58:44.730587+0000","last_active":"2024-09-21T11:01:03.575175+0000","last_peered":"2024-09-21T11:01:03.575175+0000","last_clean":"2024-09-21T10:52:16.682244+0000","last_became_active":"2024-09-21T10:58:44.730587+0000","last_became_peered":"2024-09-21T10:58:44.730587+0000","last_unstale":"2024-09-21T11:01:03.575175+0000","last_undegraded":"2024-09-21T11:01:03.575175+0000","last_fullsized":"2024-09-21T10:58:44.711142+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:24:31.440715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":278,"reported_epoch":1566,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:02.577603+0000","last_change":"2024-09-21T11:00:55.575397+0000","last_active":"2024-09-21T11:01:02.577603+0000","last_peered":"2024-09-21T11:01:02.577603+0000","last_clean":"2024-09-21T11:01:02.577603+0000","last_became_active":"2024-09-21T11:00:55.548257+0000","last_became_peered":"2024-09-21T11:00:55.548257+0000","last_unstale":"2024-09-21T11:01:02.577603+0000","last_undegraded":"2024-09-21T11:01:02.577603+0000","last_fullsized":"2024-09-21T11:01:02.577603+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:51.093668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[{"shards":"1,4,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1564,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.576092+0000","last_change":"2024-09-21T11:00:55.586167+0000","last_active":"2024-09-21T11:01:03.576092+0000","last_peered":"2024-09-21T11:01:03.576092+0000","last_clean":"2024-09-21T11:01:03.576092+0000","last_became_active":"2024-09-21T11:00:55.585255+0000","last_became_peered":"2024-09-21T11:00:55.585255+0000","last_unstale":"2024-09-21T11:01:03.576092+0000","last_undegraded":"2024-09-21T11:01:03.576092+0000","last_fullsized":"2024-09-21T11:01:03.576092+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:22:02.067237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7,4],"avail_no_missing":["1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":168,"reported_epoch":1566,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:02.577987+0000","last_change":"2024-09-21T11:00:55.537880+0000","last_active":"2024-09-21T11:01:02.577987+0000","last_peered":"2024-09-21T11:01:02.577987+0000","last_clean":"2024-09-21T11:01:02.577987+0000","last_became_active":"2024-09-21T11:00:55.531271+0000","last_became_peered":"2024-09-21T11:00:55.531271+0000","last_unstale":"2024-09-21T11:01:02.577987+0000","last_undegraded":"2024-09-21T11:01:02.577987+0000","last_fullsized":"2024-09-21T11:01:02.577987+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:19.915846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[{"shards":"1,4,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":159,"reported_epoch":1567,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:03.571654+0000","last_change":"2024-09-21T11:00:55.563812+0000","last_active":"2024-09-21T11:01:03.571654+0000","last_peered":"2024-09-21T11:01:03.571654+0000","last_clean":"2024-09-21T11:01:03.571654+0000","last_became_active":"2024-09-21T11:00:55.554250+0000","last_became_peered":"2024-09-21T11:00:55.554250+0000","last_unstale":"2024-09-21T11:01:03.571654+0000","last_undegraded":"2024-09-21T11:01:03.571654+0000","last_fullsized":"2024-09-21T11:01:03.571654+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:20:04.366261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":168,"reported_epoch":1566,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:02.576723+0000","last_change":"2024-09-21T11:00:55.590938+0000","last_active":"2024-09-21T11:01:02.576723+0000","last_peered":"2024-09-21T11:01:02.576723+0000","last_clean":"2024-09-21T11:01:02.576723+0000","last_became_active":"2024-09-21T11:00:55.581470+0000","last_became_peered":"2024-09-21T11:00:55.581470+0000","last_unstale":"2024-09-21T11:01:02.576723+0000","last_undegraded":"2024-09-21T11:01:02.576723+0000","last_fullsized":"2024-09-21T11:01:02.576723+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:42:14.129767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[{"shards":"1,4,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1509,"reported_epoch":1567,"state":"active+undersized+degraded","last_fresh":"2024-09-21T11:01:03.575178+0000","last_change":"2024-09-21T10:58:44.742011+0000","last_active":"2024-09-21T11:01:03.575178+0000","last_peered":"2024-09-21T11:01:03.575178+0000","last_clean":"2024-09-21T10:51:09.456376+0000","last_became_active":"2024-09-21T10:58:44.742011+0000","last_became_peered":"2024-09-21T10:58:44.742011+0000","last_unstale":"2024-09-21T11:01:03.575178+0000","last_undegraded":"2024-09-21T10:58:44.715934+0000","last_fullsized":"2024-09-21T10:58:44.715831+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:58:03.721963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2262,"reported_epoch":1567,"state":"active+undersized","last_fresh":"2024-09-21T11:01:03.573159+0000","last_change":"2024-09-21T10:58:44.729690+0000","last_active":"2024-09-21T11:01:03.573159+0000","last_peered":"2024-09-21T11:01:03.573159+0000","last_clean":"2024-09-21T10:58:41.702422+0000","last_became_active":"2024-09-21T10:58:44.729690+0000","last_became_peered":"2024-09-21T10:58:44.729690+0000","last_unstale":"2024-09-21T11:01:03.573159+0000","last_undegraded":"2024-09-21T11:01:03.573159+0000","last_fullsized":"2024-09-21T10:58:44.718494+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:00:08.462052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":258,"reported_epoch":1566,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:02.575935+0000","last_change":"2024-09-21T11:00:55.576480+0000","last_active":"2024-09-21T11:01:02.575935+0000","last_peered":"2024-09-21T11:01:02.575935+0000","last_clean":"2024-09-21T11:01:02.575935+0000","last_became_active":"2024-09-21T11:00:55.574210+0000","last_became_peered":"2024-09-21T11:00:55.574210+0000","last_unstale":"2024-09-21T11:01:02.575935+0000","last_undegraded":"2024-09-21T11:01:02.575935+0000","last_fullsized":"2024-09-21T11:01:02.575935+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:34.831508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2310,"reported_epoch":1567,"state":"active+undersized","last_fresh":"2024-09-21T11:01:03.572397+0000","last_change":"2024-09-21T10:58:44.722341+0000","last_active":"2024-09-21T11:01:03.572397+0000","last_peered":"2024-09-21T11:01:03.572397+0000","last_clean":"2024-09-21T10:58:41.702957+0000","last_became_active":"2024-09-21T10:58:44.722341+0000","last_became_peered":"2024-09-21T10:58:44.722341+0000","last_unstale":"2024-09-21T11:01:03.572397+0000","last_undegraded":"2024-09-21T11:01:03.572397+0000","last_fullsized":"2024-09-21T10:58:44.717994+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:18:45.908751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":130,"reported_epoch":1566,"state":"active+undersized","last_fresh":"2024-09-21T11:01:02.574859+0000","last_change":"2024-09-21T10:58:44.727977+0000","last_active":"2024-09-21T11:01:02.574859+0000","last_peered":"2024-09-21T11:01:02.574859+0000","last_clean":"2024-09-21T10:58:41.702488+0000","last_became_active":"2024-09-21T10:58:44.727977+0000","last_became_peered":"2024-09-21T10:58:44.727977+0000","last_unstale":"2024-09-21T11:01:02.574859+0000","last_undegraded":"2024-09-21T11:01:02.574859+0000","last_fullsized":"2024-09-21T10:58:44.712433+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:56:17.977304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1536,"reported_epoch":1566,"state":"active+undersized","last_fresh":"2024-09-21T11:01:02.578185+0000","last_change":"2024-09-21T10:58:44.725203+0000","last_active":"2024-09-21T11:01:02.578185+0000","last_peered":"2024-09-21T11:01:02.578185+0000","last_clean":"2024-09-21T10:58:41.702243+0000","last_became_active":"2024-09-21T10:58:44.725203+0000","last_became_peered":"2024-09-21T10:58:44.725203+0000","last_unstale":"2024-09-21T11:01:02.578185+0000","last_undegraded":"2024-09-21T11:01:02.578185+0000","last_fullsized":"2024-09-21T10:58:44.712399+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:51:11.642818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":273,"reported_epoch":1566,"state":"active+clean+remapped","last_fresh":"2024-09-21T11:01:02.575534+0000","last_change":"2024-09-21T11:00:55.561829+0000","last_active":"2024-09-21T11:01:02.575534+0000","last_peered":"2024-09-21T11:01:02.575534+0000","last_clean":"2024-09-21T11:01:02.575534+0000","last_became_active":"2024-09-21T11:00:55.503317+0000","last_became_peered":"2024-09-21T11:00:55.503317+0000","last_unstale":"2024-09-21T11:01:02.575534+0000","last_undegraded":"2024-09-21T11:01:02.575534+0000","last_fullsized":"2024-09-21T11:01:02.575534+0000","mapping_epoch":1564,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1565,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:41:36.558253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[{"shards":"1,4,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2428,"reported_epoch":1567,"state":"active+clean","last_fresh":"2024-09-21T11:01:03.571559+0000","last_change":"2024-09-21T10:44:14.398584+0000","last_active":"2024-09-21T11:01:03.571559+0000","last_peered":"2024-09-21T11:01:03.571559+0000","last_clean":"2024-09-21T11:01:03.571559+0000","last_became_active":"2024-09-21T10:44:14.398278+0000","last_became_peered":"2024-09-21T10:44:14.398278+0000","last_unstale":"2024-09-21T11:01:03.571559+0000","last_undegraded":"2024-09-21T11:01:03.571559+0000","last_fullsized":"2024-09-21T11:01:03.571559+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:19.681750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":57,"num_pg":26,"stat_sum":{"num_bytes":498630680,"num_objects":7610,"num_object_clones":0,"num_object_copies":22830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3995,"num_objects_misplaced":4374,"num_objects_unfound":0,"num_objects_dirty":7610,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124017,"num_write_kb":4175873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23325,"num_bytes_recovered":1492910080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3693293568,"data_stored":3693281352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120991,"ondisk_log_size":120991,"up":56,"acting":70,"num_store_stats":5},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":268350,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":173,"num_read_kb":375,"num_write":99,"num_write_kb":1099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":3660871,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":22,"acting":30,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":17,"acting":21,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":17,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":21996,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":23450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":65,"acting":91,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":68,"acting":88,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234807,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1639160,"kb_used_data":416688,"kb_used_omap":1,"kb_used_meta":1222398,"kb_avail":92105992,"statfs":{"total":95995035648,"available":94316535808,"internally_reserved":0,"allocated":426688512,"data_stored":422726969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":1251736011},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496413,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3400500,"kb_used_data":2189796,"kb_used_omap":1,"kb_used_meta":1210686,"kb_avail":90344652,"statfs":{"total":95995035648,"available":92512923648,"internally_reserved":0,"allocated":2242351104,"data_stored":2236868255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1239742922},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":163208758048,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019716,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369460,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1400188,"kb_used_data":132532,"kb_used_omap":69,"kb_used_meta":1267578,"kb_avail":92344964,"statfs":{"total":95995035648,"available":94561243136,"internally_reserved":0,"allocated":135712768,"data_stored":131386166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":1298000002},"hb_peers":[0,1,2,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2920577761915,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807958146,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1975776,"kb_used_data":651864,"kb_used_omap":66,"kb_used_meta":1323773,"kb_avail":91769376,"statfs":{"total":95995035648,"available":93971841024,"internally_reserved":0,"allocated":667508736,"data_stored":663151231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68589,"internal_metadata":1355543571},"hb_peers":[0,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,4,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":2860448219785,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":567939072,"data_stored":567935000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":109445120,"data_stored":109445120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":349442048,"data_stored":349437976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2257518592,"data_stored":2257518592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":408948736,"data_stored":408944664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T11:01:22.604 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T11:01:22.604 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pg_num 21 2024-09-21T11:01:22.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:22.618 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:22.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:22.718 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:22.741 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:22.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:22.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:22.843 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:22.866 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:22.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:22.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:22.968 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:22.989 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:22.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:22.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:23.090 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:23.111 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:23.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:23.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:23.213 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:23.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:23.235 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:23.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:23.335 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:23.348 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:01:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:01:23] "GET /metrics HTTP/1.1" 200 36583 "" "Prometheus/2.51.0" 2024-09-21T11:01:23.357 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:23.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:23.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:23.459 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:23.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:23.482 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:23.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:23.583 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:23.607 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:23.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:23.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:23.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:23 smithi070 ceph-mon[112139]: pgmap v3043: 109 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+remapped+backfill_wait, 6 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 18 active+undersized, 6 active+undersized+degraded, 8 active+clean; 14 MiB data, 7.1 GiB used, 351 GiB / 358 GiB avail; 0 B/s wr, 1.64k op/s; 157/738 objects degraded (21.274%); 812/738 objects misplaced (110.027%); 473 KiB/s, 77 objects/s recovering 2024-09-21T11:01:23.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:23 smithi070 ceph-mon[107385]: pgmap v3043: 109 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+remapped+backfill_wait, 6 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 18 active+undersized, 6 active+undersized+degraded, 8 active+clean; 14 MiB data, 7.1 GiB used, 351 GiB / 358 GiB avail; 0 B/s wr, 1.64k op/s; 157/738 objects degraded (21.274%); 812/738 objects misplaced (110.027%); 473 KiB/s, 77 objects/s recovering 2024-09-21T11:01:23.709 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:23.730 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:23.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:23.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:23.831 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:23 smithi160 ceph-mon[101879]: pgmap v3043: 109 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+remapped+backfill_wait, 6 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 18 active+undersized, 6 active+undersized+degraded, 8 active+clean; 14 MiB data, 7.1 GiB used, 351 GiB / 358 GiB avail; 0 B/s wr, 1.64k op/s; 157/738 objects degraded (21.274%); 812/738 objects misplaced (110.027%); 473 KiB/s, 77 objects/s recovering 2024-09-21T11:01:23.853 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:23.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:23.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:23.943 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T11:01:23.944 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T11:01:23.950 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:01:23.950 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:01:23.955 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:23.976 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:23.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:23.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:24.078 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:24.100 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:24.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:24.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:24.201 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:24.224 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:24.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:24.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:24.326 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:24.347 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:24.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:24.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:24.449 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:24.471 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:24.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:24.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:24.573 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:24.595 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:24.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:24.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:24.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:24 smithi070 ceph-mon[107385]: osdmap e1569: 8 total, 6 up, 4 in 2024-09-21T11:01:24.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:24 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 157/738 objects degraded (21.274%), 12 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T11:01:24.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:24 smithi070 ceph-mon[112139]: osdmap e1569: 8 total, 6 up, 4 in 2024-09-21T11:01:24.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:24 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 157/738 objects degraded (21.274%), 12 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T11:01:24.697 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:24.718 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:24.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:24.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:24.820 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:24.840 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:24 smithi160 ceph-mon[101879]: osdmap e1569: 8 total, 6 up, 4 in 2024-09-21T11:01:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:24 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 157/738 objects degraded (21.274%), 12 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T11:01:24.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:24.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:24.942 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:24.965 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:24.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:24.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:25.067 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:25.088 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:25.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:25.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:25.190 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:25.211 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:25.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:25.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:25.313 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:25.334 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:25.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:25.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:25.370 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T11:01:25.436 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:25.458 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:25.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:25.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:25.560 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:25.582 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:25.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:25.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:25.683 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:25.705 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:25.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:25.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:25.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:25 smithi160 ceph-mon[101879]: pgmap v3045: 109 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+remapped+backfill_wait, 6 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 18 active+undersized, 6 active+undersized+degraded, 8 active+clean; 14 MiB data, 7.1 GiB used, 351 GiB / 358 GiB avail; 0 B/s wr, 947 op/s; 157/738 objects degraded (21.274%); 812/738 objects misplaced (110.027%); 0 B/s, 49 objects/s recovering 2024-09-21T11:01:25.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:25 smithi160 ceph-mon[101879]: osdmap e1570: 8 total, 6 up, 4 in 2024-09-21T11:01:25.806 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:25.827 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:25.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:25.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:25.849 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:25 smithi070 ceph-mon[107385]: pgmap v3045: 109 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+remapped+backfill_wait, 6 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 18 active+undersized, 6 active+undersized+degraded, 8 active+clean; 14 MiB data, 7.1 GiB used, 351 GiB / 358 GiB avail; 0 B/s wr, 947 op/s; 157/738 objects degraded (21.274%); 812/738 objects misplaced (110.027%); 0 B/s, 49 objects/s recovering 2024-09-21T11:01:25.849 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:25 smithi070 ceph-mon[107385]: osdmap e1570: 8 total, 6 up, 4 in 2024-09-21T11:01:25.850 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:25 smithi070 ceph-mon[112139]: pgmap v3045: 109 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+remapped+backfill_wait, 6 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 64 active+clean+remapped, 18 active+undersized, 6 active+undersized+degraded, 8 active+clean; 14 MiB data, 7.1 GiB used, 351 GiB / 358 GiB avail; 0 B/s wr, 947 op/s; 157/738 objects degraded (21.274%); 812/738 objects misplaced (110.027%); 0 B/s, 49 objects/s recovering 2024-09-21T11:01:25.850 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:25 smithi070 ceph-mon[112139]: osdmap e1570: 8 total, 6 up, 4 in 2024-09-21T11:01:25.872 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T11:01:25.929 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:25.979 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:25.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:25.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:26.081 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:26.102 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:26.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:26.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:26.203 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:26.225 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:26.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:26.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:26.327 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:26.349 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:26.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:26.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:26.450 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:26.471 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:26.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:26.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:26.573 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:26.595 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:26.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:26.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:26.696 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:26.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:26.720 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:26.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:26.821 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:26.843 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:26.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:26.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:26.945 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:26.966 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:26.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:26.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:27.068 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:27.090 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:27.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:27.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:27.192 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:27.214 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:27.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:27.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:27.316 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:27.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:27.343 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:27.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:27.444 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:27.465 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:27.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:27.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:27.567 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:27.589 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:27.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:27.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:27.595 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:27 smithi070 ceph-mon[112139]: pgmap v3047: 109 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 72 active+clean+remapped, 2 active+remapped+backfill_wait, 19 active+undersized, 5 active+undersized+degraded, 8 active+clean; 11 MiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 0 B/s wr, 953 op/s; 51/612 objects degraded (8.333%); 549/612 objects misplaced (89.706%); 0 B/s, 535 objects/s recovering 2024-09-21T11:01:27.596 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:27 smithi070 ceph-mon[107385]: pgmap v3047: 109 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 72 active+clean+remapped, 2 active+remapped+backfill_wait, 19 active+undersized, 5 active+undersized+degraded, 8 active+clean; 11 MiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 0 B/s wr, 953 op/s; 51/612 objects degraded (8.333%); 549/612 objects misplaced (89.706%); 0 B/s, 535 objects/s recovering 2024-09-21T11:01:27.691 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:27.723 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:27.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:27.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:27.825 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:27 smithi160 ceph-mon[101879]: pgmap v3047: 109 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 72 active+clean+remapped, 2 active+remapped+backfill_wait, 19 active+undersized, 5 active+undersized+degraded, 8 active+clean; 11 MiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 0 B/s wr, 953 op/s; 51/612 objects degraded (8.333%); 549/612 objects misplaced (89.706%); 0 B/s, 535 objects/s recovering 2024-09-21T11:01:27.846 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:27.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:27.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:27.947 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:27.968 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:27.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:27.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:28.070 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:28.073 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:01:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:01:28.096 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:28.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:28.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:28.103 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:01:28.104 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:01:28.119 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:01:28.121 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:01:28.122 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:01:28.153 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:01:28.153 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:01:28.161 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:01:28.172 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:01:28.172 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:01:28.172 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T11:01:28.172 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T11:01:28.190 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:01:28.191 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:01:28.197 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:28.221 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:28.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:28.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:28.323 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:28.345 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:28.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:28.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:28.425 INFO:teuthology.orchestra.run.smithi070.stderr:set pool 1 pg_num to 21 2024-09-21T11:01:28.446 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:28.469 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:28.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:28.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:28.570 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:28.592 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:28.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:28.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:28.694 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:28.715 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:28.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:28.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:28.816 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:28.837 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:28.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:28.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:28 smithi160 ceph-mon[101879]: osdmap e1571: 8 total, 6 up, 4 in 2024-09-21T11:01:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:28 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/284377643' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T11:01:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:28 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T11:01:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:28 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/711445591' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:01:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:28 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T11:01:28.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:28 smithi070 ceph-mon[107385]: osdmap e1571: 8 total, 6 up, 4 in 2024-09-21T11:01:28.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:28 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/284377643' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T11:01:28.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:28 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T11:01:28.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:28 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/711445591' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:01:28.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:28 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T11:01:28.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:28 smithi070 ceph-mon[112139]: osdmap e1571: 8 total, 6 up, 4 in 2024-09-21T11:01:28.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:28 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/284377643' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T11:01:28.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:28 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T11:01:28.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:28 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/711445591' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:01:28.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:28 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T11:01:28.939 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:28.960 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:28.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:28.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:29.062 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:29.085 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:29.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:29.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:29.186 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:29.207 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:29.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:29.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:29.309 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:29.331 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:29.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:29.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:29.432 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:29.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:29.455 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:29.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:29.551 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:29 smithi070 ceph-mon[112139]: pgmap v3049: 109 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 72 active+clean+remapped, 2 active+remapped+backfill_wait, 19 active+undersized, 5 active+undersized+degraded, 8 active+clean; 11 MiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 0 B/s wr, 8 op/s; 51/612 objects degraded (8.333%); 549/612 objects misplaced (89.706%); 0 B/s, 647 objects/s recovering 2024-09-21T11:01:29.551 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:29 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-21T11:01:29.551 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:29 smithi070 ceph-mon[112139]: osdmap e1572: 8 total, 6 up, 4 in 2024-09-21T11:01:29.551 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:29 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 51/612 objects degraded (8.333%), 5 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T11:01:29.551 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:29 smithi070 ceph-mon[107385]: pgmap v3049: 109 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 72 active+clean+remapped, 2 active+remapped+backfill_wait, 19 active+undersized, 5 active+undersized+degraded, 8 active+clean; 11 MiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 0 B/s wr, 8 op/s; 51/612 objects degraded (8.333%); 549/612 objects misplaced (89.706%); 0 B/s, 647 objects/s recovering 2024-09-21T11:01:29.551 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:29 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-21T11:01:29.551 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:29 smithi070 ceph-mon[107385]: osdmap e1572: 8 total, 6 up, 4 in 2024-09-21T11:01:29.551 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:29 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 51/612 objects degraded (8.333%), 5 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T11:01:29.557 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:29.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:29.580 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:29.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:29.676 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:29 smithi160 ceph-mon[101879]: pgmap v3049: 109 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 72 active+clean+remapped, 2 active+remapped+backfill_wait, 19 active+undersized, 5 active+undersized+degraded, 8 active+clean; 11 MiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 0 B/s wr, 8 op/s; 51/612 objects degraded (8.333%); 549/612 objects misplaced (89.706%); 0 B/s, 647 objects/s recovering 2024-09-21T11:01:29.677 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:29 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-21T11:01:29.677 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:29 smithi160 ceph-mon[101879]: osdmap e1572: 8 total, 6 up, 4 in 2024-09-21T11:01:29.677 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:29 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 51/612 objects degraded (8.333%), 5 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T11:01:29.681 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:29.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:29.703 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:29.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:29.804 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:29.826 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:29.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:29.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:29.863 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-09-21T11:01:29.863 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-09-21T11:01:29.927 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:29.950 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:29.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:29.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:30.051 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:30.072 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:30.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:30.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:30.173 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:30.194 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:30.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:30.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:30.296 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:30.317 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:30.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:30.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:30.372 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T11:01:30.372 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T11:01:30.419 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:30.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:30.441 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:30.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:30.541 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:30.562 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:30.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:30.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:30.664 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:30.684 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:30.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:30.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:30.786 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:30.807 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:30.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:30.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:30.909 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:30.931 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:30.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:30.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:31.033 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:31.056 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:31.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:31.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:31.158 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:31.180 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:31.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:31.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:31.226 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:30 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:01:31.226 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:30 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:01:31.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:30 smithi070 ceph-mon[107385]: pgmap v3051: 109 pgs: 1 active+recovering+undersized+remapped, 74 active+clean+remapped, 2 active+remapped+backfill_wait, 19 active+undersized, 5 active+undersized+degraded, 8 active+clean; 11 MiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 0 B/s wr, 8 op/s; 51/612 objects degraded (8.333%); 559/612 objects misplaced (91.340%); 0 B/s, 647 objects/s recovering 2024-09-21T11:01:31.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:30 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-21T11:01:31.227 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:30 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:01:31.227 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:30 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:01:31.227 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:30 smithi070 ceph-mon[112139]: pgmap v3051: 109 pgs: 1 active+recovering+undersized+remapped, 74 active+clean+remapped, 2 active+remapped+backfill_wait, 19 active+undersized, 5 active+undersized+degraded, 8 active+clean; 11 MiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 0 B/s wr, 8 op/s; 51/612 objects degraded (8.333%); 559/612 objects misplaced (91.340%); 0 B/s, 647 objects/s recovering 2024-09-21T11:01:31.227 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:30 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-21T11:01:31.282 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:31.303 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:31.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:31.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:31.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:30 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:01:31.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:30 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:01:31.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:30 smithi160 ceph-mon[101879]: pgmap v3051: 109 pgs: 1 active+recovering+undersized+remapped, 74 active+clean+remapped, 2 active+remapped+backfill_wait, 19 active+undersized, 5 active+undersized+degraded, 8 active+clean; 11 MiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 0 B/s wr, 8 op/s; 51/612 objects degraded (8.333%); 559/612 objects misplaced (91.340%); 0 B/s, 647 objects/s recovering 2024-09-21T11:01:31.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:30 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-21T11:01:31.405 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:31.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:31.428 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:31.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:31.528 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:31.551 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:31.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:31.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:31.652 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:31.673 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:31.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:31.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:31.774 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:31.796 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:31.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:31.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:31.898 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:31.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:31.921 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:31.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:32.022 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:32.043 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:32.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:32.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:32.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:31 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-21T11:01:32.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:31 smithi160 ceph-mon[101879]: osdmap e1573: 8 total, 6 up, 4 in 2024-09-21T11:01:32.145 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:32.166 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:32.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:32.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:32.183 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:31 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-21T11:01:32.183 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:31 smithi070 ceph-mon[107385]: osdmap e1573: 8 total, 6 up, 4 in 2024-09-21T11:01:32.183 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:31 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-21T11:01:32.183 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:31 smithi070 ceph-mon[112139]: osdmap e1573: 8 total, 6 up, 4 in 2024-09-21T11:01:32.268 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:32.289 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:32.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:32.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:32.373 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T11:01:32.376 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:01:32.390 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:32.438 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:32.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:32.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:32.539 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:32.560 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:32.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:32.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:32.629 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T11:01:32.629 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T11:01:32.629 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T11:01:32.629 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T11:01:32.629 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.629 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:156258 55.c 55:39292414:::benchmark_data_smithi070_294394_object50917:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1464)", 2024-09-21T11:01:32.629 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.312622+0000", 2024-09-21T11:01:32.629 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 548.29112645999999, 2024-09-21T11:01:32.629 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.330110946, 2024-09-21T11:01:32.630 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:01:32.630 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:01:32.630 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:01:32.630 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:01:32.630 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T11:01:32.630 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T11:01:32.630 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 156258 2024-09-21T11:01:32.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.630 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:01:32.630 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.630 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:01:32.630 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.312622+0000", 2024-09-21T11:01:32.630 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.631 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.631 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:01:32.631 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.312622+0000", 2024-09-21T11:01:32.631 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.631 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.631 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:01:32.631 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.312625+0000", 2024-09-21T11:01:32.631 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6079999999999998e-06 2024-09-21T11:01:32.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.631 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.631 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:01:32.631 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.312632+0000", 2024-09-21T11:01:32.632 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7390000000000002e-06 2024-09-21T11:01:32.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.632 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:01:32.632 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.312650+0000", 2024-09-21T11:01:32.632 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7946000000000001e-05 2024-09-21T11:01:32.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.632 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:01:32.632 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.312660+0000", 2024-09-21T11:01:32.632 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0178e-05 2024-09-21T11:01:32.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.633 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.633 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.319581+0000", 2024-09-21T11:01:32.633 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0069215680000000003 2024-09-21T11:01:32.633 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.633 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.633 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T11:01:32.633 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.322223+0000", 2024-09-21T11:01:32.633 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0026414160000000002 2024-09-21T11:01:32.633 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.633 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.633 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.633 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.335700+0000", 2024-09-21T11:01:32.634 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.013477668 2024-09-21T11:01:32.634 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.634 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.634 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:01:32.634 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.335800+0000", 2024-09-21T11:01:32.634 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9123000000000006e-05 2024-09-21T11:01:32.634 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.634 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.634 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,1,4", 2024-09-21T11:01:32.634 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.335951+0000", 2024-09-21T11:01:32.634 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015160799999999999 2024-09-21T11:01:32.634 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.634 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.634 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:01:32.635 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.339558+0000", 2024-09-21T11:01:32.635 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0036068509999999999 2024-09-21T11:01:32.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.635 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.635 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:01:32.635 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.340760+0000", 2024-09-21T11:01:32.635 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0012021810000000001 2024-09-21T11:01:32.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.635 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.635 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:01:32.635 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.342017+0000", 2024-09-21T11:01:32.635 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001256704 2024-09-21T11:01:32.636 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.636 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.636 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.476857+0000", 2024-09-21T11:01:32.636 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13483968700000001 2024-09-21T11:01:32.636 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.636 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:01:32.636 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.476910+0000", 2024-09-21T11:01:32.636 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3829000000000002e-05 2024-09-21T11:01:32.636 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.636 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.636 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.642675+0000", 2024-09-21T11:01:32.637 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.165764507 2024-09-21T11:01:32.637 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.637 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.637 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:01:32.637 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.642733+0000", 2024-09-21T11:01:32.637 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8332999999999999e-05 2024-09-21T11:01:32.637 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.637 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:01:32.637 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.637 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.637 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.637 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:156286 55.c 55:3c4e6380:::benchmark_data_smithi070_294394_object50945:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1464)", 2024-09-21T11:01:32.637 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.368882+0000", 2024-09-21T11:01:32.638 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 548.23486645200001, 2024-09-21T11:01:32.638 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3537682069999999, 2024-09-21T11:01:32.638 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:01:32.638 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:01:32.638 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:01:32.638 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:01:32.638 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T11:01:32.638 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T11:01:32.638 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 156286 2024-09-21T11:01:32.638 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.638 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:01:32.638 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.638 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:01:32.639 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.368882+0000", 2024-09-21T11:01:32.639 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.639 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.639 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:01:32.639 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.368882+0000", 2024-09-21T11:01:32.639 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.639 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.639 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:01:32.639 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.368885+0000", 2024-09-21T11:01:32.639 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2979999999999999e-06 2024-09-21T11:01:32.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.639 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.640 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:01:32.640 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.368891+0000", 2024-09-21T11:01:32.640 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.708e-06 2024-09-21T11:01:32.640 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.640 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.640 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:01:32.640 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.368907+0000", 2024-09-21T11:01:32.640 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5437000000000001e-05 2024-09-21T11:01:32.640 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.640 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.640 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:01:32.640 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.368916+0000", 2024-09-21T11:01:32.640 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9160000000000007e-06 2024-09-21T11:01:32.640 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.641 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.641 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.641 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.368988+0000", 2024-09-21T11:01:32.641 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1964000000000004e-05 2024-09-21T11:01:32.641 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.641 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.641 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T11:01:32.641 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.369490+0000", 2024-09-21T11:01:32.641 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00050230499999999998 2024-09-21T11:01:32.641 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.641 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.641 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.641 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.478001+0000", 2024-09-21T11:01:32.642 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.108510617 2024-09-21T11:01:32.642 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.642 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.642 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:01:32.642 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.478010+0000", 2024-09-21T11:01:32.642 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.8339999999999994e-06 2024-09-21T11:01:32.642 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.642 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.642 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.642 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.710218+0000", 2024-09-21T11:01:32.642 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2322071860000001 2024-09-21T11:01:32.642 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.642 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.643 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:01:32.643 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.710354+0000", 2024-09-21T11:01:32.643 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013647799999999999 2024-09-21T11:01:32.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.643 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.643 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,4,7", 2024-09-21T11:01:32.643 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.710499+0000", 2024-09-21T11:01:32.643 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014466899999999999 2024-09-21T11:01:32.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.643 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.643 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:01:32.643 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.714807+0000", 2024-09-21T11:01:32.643 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0043086019999999999 2024-09-21T11:01:32.644 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.644 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.644 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:01:32.644 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.714852+0000", 2024-09-21T11:01:32.644 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.4433e-05 2024-09-21T11:01:32.644 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.644 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.644 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:01:32.644 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.714864+0000", 2024-09-21T11:01:32.644 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1946999999999999e-05 2024-09-21T11:01:32.644 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.644 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.644 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:01:32.645 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.720322+0000", 2024-09-21T11:01:32.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0054582290000000002 2024-09-21T11:01:32.645 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.645 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.645 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:01:32.645 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.720406+0000", 2024-09-21T11:01:32.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3776999999999994e-05 2024-09-21T11:01:32.645 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.645 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.645 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:01:32.645 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.722651+0000", 2024-09-21T11:01:32.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0022448070000000001 2024-09-21T11:01:32.645 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.646 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:01:32.646 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.646 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.646 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.646 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:156303 55.0 55:043b163c:::benchmark_data_smithi070_294394_object50962:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1465)", 2024-09-21T11:01:32.646 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.513934+0000", 2024-09-21T11:01:32.646 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 548.089814348, 2024-09-21T11:01:32.646 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2375139740000001, 2024-09-21T11:01:32.646 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:01:32.646 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:01:32.646 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:01:32.646 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:01:32.646 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T11:01:32.647 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T11:01:32.647 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 156303 2024-09-21T11:01:32.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.647 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:01:32.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.647 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:01:32.647 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.513934+0000", 2024-09-21T11:01:32.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.647 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:01:32.647 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.513934+0000", 2024-09-21T11:01:32.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.648 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:01:32.648 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.513936+0000", 2024-09-21T11:01:32.648 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1009999999999999e-06 2024-09-21T11:01:32.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.648 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:01:32.648 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.513944+0000", 2024-09-21T11:01:32.648 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9140000000000008e-06 2024-09-21T11:01:32.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.648 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:01:32.648 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.513963+0000", 2024-09-21T11:01:32.648 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8712999999999999e-05 2024-09-21T11:01:32.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:01:32.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.513972+0000", 2024-09-21T11:01:32.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1190000000000003e-06 2024-09-21T11:01:32.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.514061+0000", 2024-09-21T11:01:32.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9059000000000002e-05 2024-09-21T11:01:32.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:01:32.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.514073+0000", 2024-09-21T11:01:32.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1301000000000001e-05 2024-09-21T11:01:32.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.650 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.650 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.462210+0000", 2024-09-21T11:01:32.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.94813707000000003 2024-09-21T11:01:32.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.650 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:01:32.650 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.462220+0000", 2024-09-21T11:01:32.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0360000000000001e-05 2024-09-21T11:01:32.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.650 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.707189+0000", 2024-09-21T11:01:32.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2449689660000001 2024-09-21T11:01:32.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.651 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T11:01:32.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.707640+0000", 2024-09-21T11:01:32.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00045076399999999998 2024-09-21T11:01:32.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.651 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.723477+0000", 2024-09-21T11:01:32.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015836787000000001 2024-09-21T11:01:32.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.652 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:01:32.652 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.723634+0000", 2024-09-21T11:01:32.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015763100000000001 2024-09-21T11:01:32.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.652 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.653 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6,7", 2024-09-21T11:01:32.654 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.723760+0000", 2024-09-21T11:01:32.654 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012615099999999999 2024-09-21T11:01:32.654 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.654 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.654 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:01:32.654 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.733609+0000", 2024-09-21T11:01:32.654 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0098482320000000002 2024-09-21T11:01:32.654 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.654 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.654 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:01:32.655 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.748913+0000", 2024-09-21T11:01:32.655 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015304386999999999 2024-09-21T11:01:32.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.655 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:01:32.655 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.748958+0000", 2024-09-21T11:01:32.655 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.4951999999999999e-05 2024-09-21T11:01:32.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.655 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:01:32.655 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.749085+0000", 2024-09-21T11:01:32.655 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000127513 2024-09-21T11:01:32.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.656 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:01:32.656 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.749146+0000", 2024-09-21T11:01:32.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0296e-05 2024-09-21T11:01:32.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.656 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:01:32.656 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.751448+0000", 2024-09-21T11:01:32.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002302658 2024-09-21T11:01:32.656 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.656 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:01:32.656 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.656 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.35670.0:156314 55.0 55:08dfcd2f:::benchmark_data_smithi070_294394_object50973:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1465)", 2024-09-21T11:01:32.657 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:52:24.554453+0000", 2024-09-21T11:01:32.657 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 548.04929561899996, 2024-09-21T11:01:32.657 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1993647900000002, 2024-09-21T11:01:32.657 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:01:32.657 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:01:32.657 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:01:32.657 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:01:32.657 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.35670", 2024-09-21T11:01:32.657 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/3008345115", 2024-09-21T11:01:32.657 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 156314 2024-09-21T11:01:32.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.657 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:01:32.657 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.657 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:01:32.658 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.554453+0000", 2024-09-21T11:01:32.658 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.658 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.658 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.658 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:01:32.658 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.554453+0000", 2024-09-21T11:01:32.658 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.658 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.658 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.658 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:01:32.658 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.554460+0000", 2024-09-21T11:01:32.658 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2200000000000003e-06 2024-09-21T11:01:32.658 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.658 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.659 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:01:32.659 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.554468+0000", 2024-09-21T11:01:32.659 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5379999999999997e-06 2024-09-21T11:01:32.659 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.659 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:01:32.659 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.554484+0000", 2024-09-21T11:01:32.659 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.577e-05 2024-09-21T11:01:32.659 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.659 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:01:32.659 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.554502+0000", 2024-09-21T11:01:32.659 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8406999999999998e-05 2024-09-21T11:01:32.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.660 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.660 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.554554+0000", 2024-09-21T11:01:32.660 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1579000000000002e-05 2024-09-21T11:01:32.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.660 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:01:32.660 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:24.554561+0000", 2024-09-21T11:01:32.660 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0249999999999997e-06 2024-09-21T11:01:32.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.660 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.660 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.462245+0000", 2024-09-21T11:01:32.660 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.90768446700000005 2024-09-21T11:01:32.661 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.661 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.661 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:01:32.661 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:25.462253+0000", 2024-09-21T11:01:32.661 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4019999999999996e-06 2024-09-21T11:01:32.661 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.661 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.661 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.661 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.709897+0000", 2024-09-21T11:01:32.661 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2476449220000001 2024-09-21T11:01:32.661 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.661 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.661 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for degraded object", 2024-09-21T11:01:32.662 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.710321+0000", 2024-09-21T11:01:32.662 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000423972 2024-09-21T11:01:32.662 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.662 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.662 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.662 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.731108+0000", 2024-09-21T11:01:32.662 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.020786414 2024-09-21T11:01:32.662 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.662 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.662 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:01:32.662 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.731214+0000", 2024-09-21T11:01:32.662 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010618799999999999 2024-09-21T11:01:32.662 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.663 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6,7", 2024-09-21T11:01:32.663 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.731346+0000", 2024-09-21T11:01:32.663 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013200500000000001 2024-09-21T11:01:32.663 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.663 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:01:32.663 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.746498+0000", 2024-09-21T11:01:32.663 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015151553 2024-09-21T11:01:32.663 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.663 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:01:32.663 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.749001+0000", 2024-09-21T11:01:32.663 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025033249999999998 2024-09-21T11:01:32.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.664 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.664 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:01:32.664 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.749043+0000", 2024-09-21T11:01:32.664 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2175999999999997e-05 2024-09-21T11:01:32.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.664 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.664 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:01:32.664 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.751602+0000", 2024-09-21T11:01:32.664 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025585629999999998 2024-09-21T11:01:32.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.664 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.665 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:01:32.665 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.751663+0000", 2024-09-21T11:01:32.665 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1457999999999993e-05 2024-09-21T11:01:32.665 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.665 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.665 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:01:32.665 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:52:26.753818+0000", 2024-09-21T11:01:32.665 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002154806 2024-09-21T11:01:32.665 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.665 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:01:32.665 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.665 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.665 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.665 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36012.0:66794 56.5 56:af9493d5:::benchmark_data_smithi070_333437_object66793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1490)", 2024-09-21T11:01:32.666 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:55:00.442566+0000", 2024-09-21T11:01:32.666 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 392.16118278699997, 2024-09-21T11:01:32.666 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2236836719999999, 2024-09-21T11:01:32.666 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:01:32.666 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:01:32.666 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:01:32.666 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:01:32.666 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36012", 2024-09-21T11:01:32.666 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T11:01:32.666 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 66794 2024-09-21T11:01:32.666 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.666 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:01:32.666 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.667 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:01:32.667 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442566+0000", 2024-09-21T11:01:32.667 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.667 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.667 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:01:32.667 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442566+0000", 2024-09-21T11:01:32.667 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.667 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.667 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:01:32.667 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442567+0000", 2024-09-21T11:01:32.667 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.8899999999999998e-07 2024-09-21T11:01:32.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.668 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.668 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:01:32.668 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442585+0000", 2024-09-21T11:01:32.668 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8022000000000001e-05 2024-09-21T11:01:32.668 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.668 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.668 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:01:32.668 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442596+0000", 2024-09-21T11:01:32.668 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0895e-05 2024-09-21T11:01:32.668 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.668 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.668 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:01:32.668 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442602+0000", 2024-09-21T11:01:32.668 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4019999999999999e-06 2024-09-21T11:01:32.669 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.669 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.669 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.669 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442671+0000", 2024-09-21T11:01:32.669 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8868999999999994e-05 2024-09-21T11:01:32.669 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.669 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.669 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:01:32.669 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442774+0000", 2024-09-21T11:01:32.669 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000102638 2024-09-21T11:01:32.669 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.669 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.669 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,7", 2024-09-21T11:01:32.669 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442886+0000", 2024-09-21T11:01:32.670 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000111853 2024-09-21T11:01:32.670 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.670 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.670 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:01:32.670 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.445148+0000", 2024-09-21T11:01:32.670 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0022620380000000001 2024-09-21T11:01:32.670 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.670 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.670 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.670 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465200+0000", 2024-09-21T11:01:32.670 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.020052745 2024-09-21T11:01:32.670 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.671 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.671 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:01:32.671 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465211+0000", 2024-09-21T11:01:32.671 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0713999999999999e-05 2024-09-21T11:01:32.671 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.671 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.671 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.671 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:01.454645+0000", 2024-09-21T11:01:32.671 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.98943445699999999 2024-09-21T11:01:32.671 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.671 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.671 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:01:32.671 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:01.454664+0000", 2024-09-21T11:01:32.671 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8576999999999998e-05 2024-09-21T11:01:32.672 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.672 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.672 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.672 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.666160+0000", 2024-09-21T11:01:32.672 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2114961719999999 2024-09-21T11:01:32.672 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.672 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.672 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:01:32.672 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.666250+0000", 2024-09-21T11:01:32.672 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9400999999999993e-05 2024-09-21T11:01:32.672 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.672 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:01:32.672 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.672 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.672 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.672 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36012.0:66807 56.5 56:a8efccf3:::benchmark_data_smithi070_333437_object66806:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1491)", 2024-09-21T11:01:32.673 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:55:00.465619+0000", 2024-09-21T11:01:32.673 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 392.13812940600002, 2024-09-21T11:01:32.673 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2238977850000001, 2024-09-21T11:01:32.673 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:01:32.673 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:01:32.673 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:01:32.673 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:01:32.673 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36012", 2024-09-21T11:01:32.673 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T11:01:32.673 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 66807 2024-09-21T11:01:32.673 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.673 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:01:32.673 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.674 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:01:32.674 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465619+0000", 2024-09-21T11:01:32.674 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.674 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.674 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.674 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:01:32.674 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465619+0000", 2024-09-21T11:01:32.674 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.674 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.674 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.674 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:01:32.674 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465621+0000", 2024-09-21T11:01:32.674 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.841e-06 2024-09-21T11:01:32.674 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.675 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.675 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:01:32.675 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465673+0000", 2024-09-21T11:01:32.675 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1858999999999999e-05 2024-09-21T11:01:32.675 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.675 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.675 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:01:32.675 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465694+0000", 2024-09-21T11:01:32.675 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1259000000000001e-05 2024-09-21T11:01:32.675 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.675 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.675 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:01:32.675 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465715+0000", 2024-09-21T11:01:32.675 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1141999999999998e-05 2024-09-21T11:01:32.676 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.676 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.676 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.676 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465867+0000", 2024-09-21T11:01:32.676 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000151569 2024-09-21T11:01:32.676 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.676 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.676 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:01:32.676 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465876+0000", 2024-09-21T11:01:32.676 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9139999999999997e-06 2024-09-21T11:01:32.676 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.676 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.676 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.677 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:01.454697+0000", 2024-09-21T11:01:32.677 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.98882113500000002 2024-09-21T11:01:32.677 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.677 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.677 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:01:32.677 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:01.454706+0000", 2024-09-21T11:01:32.677 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2080000000000006e-06 2024-09-21T11:01:32.677 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.677 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.677 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.677 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.670154+0000", 2024-09-21T11:01:32.677 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2154475469999999 2024-09-21T11:01:32.677 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.677 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.678 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:01:32.678 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.670320+0000", 2024-09-21T11:01:32.678 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016649899999999999 2024-09-21T11:01:32.678 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.678 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.678 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T11:01:32.678 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.670782+0000", 2024-09-21T11:01:32.678 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00046200700000000001 2024-09-21T11:01:32.678 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.678 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.678 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:01:32.678 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.677750+0000", 2024-09-21T11:01:32.678 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0069672229999999998 2024-09-21T11:01:32.679 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.679 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.679 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:01:32.679 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.678570+0000", 2024-09-21T11:01:32.679 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00082092899999999997 2024-09-21T11:01:32.679 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.679 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.679 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:01:32.679 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.685882+0000", 2024-09-21T11:01:32.679 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00731155 2024-09-21T11:01:32.679 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.679 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.679 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:01:32.680 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.685948+0000", 2024-09-21T11:01:32.680 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5481000000000005e-05 2024-09-21T11:01:32.680 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.680 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.680 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:01:32.680 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.689517+0000", 2024-09-21T11:01:32.680 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035696220000000002 2024-09-21T11:01:32.680 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.680 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:01:32.680 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.680 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.680 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.680 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36012.0:97176 56.5 56:a357387e:::benchmark_data_smithi070_333437_object97175:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1499)", 2024-09-21T11:01:32.680 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:55:37.706429+0000", 2024-09-21T11:01:32.681 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 354.89731967, 2024-09-21T11:01:32.681 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7010156360000002, 2024-09-21T11:01:32.681 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:01:32.681 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:01:32.681 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:01:32.681 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:01:32.681 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36012", 2024-09-21T11:01:32.681 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T11:01:32.681 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 97176 2024-09-21T11:01:32.681 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.681 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:01:32.681 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.681 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:01:32.682 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.706429+0000", 2024-09-21T11:01:32.682 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.682 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.682 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.682 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:01:32.682 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.706429+0000", 2024-09-21T11:01:32.682 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.682 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.682 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.682 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:01:32.682 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.706432+0000", 2024-09-21T11:01:32.682 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.146e-06 2024-09-21T11:01:32.683 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.683 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.683 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:01:32.683 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.706479+0000", 2024-09-21T11:01:32.683 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7190000000000001e-05 2024-09-21T11:01:32.683 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.683 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.683 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:01:32.683 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.706514+0000", 2024-09-21T11:01:32.683 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5027000000000002e-05 2024-09-21T11:01:32.683 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.683 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.684 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:01:32.684 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.706528+0000", 2024-09-21T11:01:32.684 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3933e-05 2024-09-21T11:01:32.684 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.684 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.684 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.684 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.797946+0000", 2024-09-21T11:01:32.684 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.091417409000000005 2024-09-21T11:01:32.684 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.684 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.684 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:01:32.684 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.797952+0000", 2024-09-21T11:01:32.684 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1009999999999997e-06 2024-09-21T11:01:32.685 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.685 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.685 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.685 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:38.361807+0000", 2024-09-21T11:01:32.685 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.56385527000000002 2024-09-21T11:01:32.685 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.685 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.685 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:01:32.685 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:38.361819+0000", 2024-09-21T11:01:32.685 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1719000000000001e-05 2024-09-21T11:01:32.685 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.685 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.685 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.685 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.407348+0000", 2024-09-21T11:01:32.685 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.045529425 2024-09-21T11:01:32.686 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.686 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.686 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:01:32.686 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.407445+0000", 2024-09-21T11:01:32.686 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.6416000000000007e-05 2024-09-21T11:01:32.686 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.686 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:01:32.686 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.686 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.686 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.686 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36012.0:97182 56.5 56:a2247821:::benchmark_data_smithi070_333437_object97181:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1499)", 2024-09-21T11:01:32.686 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:55:37.706622+0000", 2024-09-21T11:01:32.687 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 354.89712651999997, 2024-09-21T11:01:32.687 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.724287076, 2024-09-21T11:01:32.687 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:01:32.687 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:01:32.687 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:01:32.687 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:01:32.687 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36012", 2024-09-21T11:01:32.687 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T11:01:32.687 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 97182 2024-09-21T11:01:32.687 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.687 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:01:32.687 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.688 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:01:32.688 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.706622+0000", 2024-09-21T11:01:32.688 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.688 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.688 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.688 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:01:32.688 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.706622+0000", 2024-09-21T11:01:32.688 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.688 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.688 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.688 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:01:32.688 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.706623+0000", 2024-09-21T11:01:32.688 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5200000000000002e-07 2024-09-21T11:01:32.688 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.689 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.689 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:01:32.689 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.706650+0000", 2024-09-21T11:01:32.689 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6852e-05 2024-09-21T11:01:32.689 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.689 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.689 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:01:32.689 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.706660+0000", 2024-09-21T11:01:32.689 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0183999999999999e-05 2024-09-21T11:01:32.689 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.689 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.689 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:01:32.690 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.706665+0000", 2024-09-21T11:01:32.690 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6559999999999997e-06 2024-09-21T11:01:32.690 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.690 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.690 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.690 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.803733+0000", 2024-09-21T11:01:32.690 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.097067298999999996 2024-09-21T11:01:32.690 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.690 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.690 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:01:32.690 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.803749+0000", 2024-09-21T11:01:32.690 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6731999999999998e-05 2024-09-21T11:01:32.690 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.691 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.691 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.691 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:38.361843+0000", 2024-09-21T11:01:32.691 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.55809369399999997 2024-09-21T11:01:32.691 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.691 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.691 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:01:32.691 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:38.361850+0000", 2024-09-21T11:01:32.691 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0299999999999996e-06 2024-09-21T11:01:32.691 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.691 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.691 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.691 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.409049+0000", 2024-09-21T11:01:32.691 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0471991439999999 2024-09-21T11:01:32.692 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.692 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.692 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:01:32.692 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.409183+0000", 2024-09-21T11:01:32.692 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013339299999999999 2024-09-21T11:01:32.692 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.692 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.692 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T11:01:32.692 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.409487+0000", 2024-09-21T11:01:32.692 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00030424500000000003 2024-09-21T11:01:32.692 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.692 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.692 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:01:32.692 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.417173+0000", 2024-09-21T11:01:32.693 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0076856030000000001 2024-09-21T11:01:32.693 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.693 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.693 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:01:32.693 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.421310+0000", 2024-09-21T11:01:32.693 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.004137216 2024-09-21T11:01:32.693 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.693 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.693 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:01:32.693 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.428975+0000", 2024-09-21T11:01:32.693 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0076652789999999997 2024-09-21T11:01:32.693 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.693 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.693 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:01:32.694 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.429028+0000", 2024-09-21T11:01:32.694 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3059000000000001e-05 2024-09-21T11:01:32.694 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.694 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.694 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:01:32.694 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.430909+0000", 2024-09-21T11:01:32.694 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0018812379999999999 2024-09-21T11:01:32.694 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.694 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:01:32.694 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.694 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.694 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.694 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.46190.0:84902 57.3 57:cba41bdc:::benchmark_data_smithi070_360420_object17553:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1558)", 2024-09-21T11:01:32.695 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:00:40.417701+0000", 2024-09-21T11:01:32.695 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 52.186047514999998, 2024-09-21T11:01:32.695 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.17822523300000001, 2024-09-21T11:01:32.695 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:01:32.695 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:01:32.695 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:01:32.695 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:01:32.695 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.46190", 2024-09-21T11:01:32.695 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:01:32.695 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 84902 2024-09-21T11:01:32.695 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.695 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:01:32.695 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.695 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:01:32.696 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.417701+0000", 2024-09-21T11:01:32.696 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.696 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.696 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.696 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:01:32.696 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.417701+0000", 2024-09-21T11:01:32.696 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.696 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.696 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.696 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:01:32.696 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.417703+0000", 2024-09-21T11:01:32.696 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5719999999999999e-06 2024-09-21T11:01:32.696 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.696 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.696 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:01:32.697 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.417708+0000", 2024-09-21T11:01:32.697 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3299999999999998e-06 2024-09-21T11:01:32.697 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.697 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.697 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:01:32.697 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.417720+0000", 2024-09-21T11:01:32.697 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1884e-05 2024-09-21T11:01:32.697 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.697 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.697 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:01:32.697 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.417728+0000", 2024-09-21T11:01:32.697 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0250000000000002e-06 2024-09-21T11:01:32.697 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.697 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.698 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.698 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.417794+0000", 2024-09-21T11:01:32.698 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6248000000000007e-05 2024-09-21T11:01:32.698 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.698 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.698 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:01:32.698 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.417936+0000", 2024-09-21T11:01:32.698 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000141545 2024-09-21T11:01:32.698 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.698 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.698 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 7", 2024-09-21T11:01:32.698 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.418055+0000", 2024-09-21T11:01:32.698 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011948800000000001 2024-09-21T11:01:32.699 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.699 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.699 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:01:32.699 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.421101+0000", 2024-09-21T11:01:32.699 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0030453419999999999 2024-09-21T11:01:32.699 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.699 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.699 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.699 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.595914+0000", 2024-09-21T11:01:32.699 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.17481350300000001 2024-09-21T11:01:32.699 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.699 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.699 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:01:32.699 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.595926+0000", 2024-09-21T11:01:32.700 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2296000000000001e-05 2024-09-21T11:01:32.700 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.700 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:01:32.700 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.700 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.700 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.700 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.46190.0:84904 57.4 57:2ac37d6e:::benchmark_data_smithi070_360420_object17555:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1558)", 2024-09-21T11:01:32.700 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:00:40.419366+0000", 2024-09-21T11:01:32.700 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 52.184382372999998, 2024-09-21T11:01:32.700 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.17083596300000001, 2024-09-21T11:01:32.700 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:01:32.700 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:01:32.700 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:01:32.700 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:01:32.701 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.46190", 2024-09-21T11:01:32.701 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:01:32.701 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 84904 2024-09-21T11:01:32.701 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.701 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:01:32.701 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.701 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:01:32.701 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.419366+0000", 2024-09-21T11:01:32.701 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.701 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.701 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.701 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:01:32.701 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.419366+0000", 2024-09-21T11:01:32.702 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.702 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.702 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.702 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:01:32.702 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.419372+0000", 2024-09-21T11:01:32.702 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.959e-06 2024-09-21T11:01:32.702 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.702 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.702 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:01:32.702 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.419378+0000", 2024-09-21T11:01:32.702 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7629999999999997e-06 2024-09-21T11:01:32.702 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.702 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.702 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:01:32.703 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.419391+0000", 2024-09-21T11:01:32.703 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2532e-05 2024-09-21T11:01:32.703 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.703 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.703 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:01:32.703 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.419400+0000", 2024-09-21T11:01:32.703 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1840000000000005e-06 2024-09-21T11:01:32.703 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.703 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.703 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.703 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.419461+0000", 2024-09-21T11:01:32.703 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1273999999999996e-05 2024-09-21T11:01:32.703 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.704 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.704 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:01:32.704 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.419575+0000", 2024-09-21T11:01:32.704 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011343100000000001 2024-09-21T11:01:32.704 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.704 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.704 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,7", 2024-09-21T11:01:32.704 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.419674+0000", 2024-09-21T11:01:32.704 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9109000000000003e-05 2024-09-21T11:01:32.704 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.704 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.704 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:01:32.704 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.423179+0000", 2024-09-21T11:01:32.704 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035050810000000002 2024-09-21T11:01:32.704 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.705 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.705 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:01:32.705 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.424047+0000", 2024-09-21T11:01:32.705 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00086820099999999998 2024-09-21T11:01:32.705 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.705 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.705 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.705 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.590188+0000", 2024-09-21T11:01:32.705 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.16614068500000001 2024-09-21T11:01:32.705 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.705 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.705 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:01:32.705 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.590202+0000", 2024-09-21T11:01:32.705 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4744e-05 2024-09-21T11:01:32.706 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.706 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:01:32.706 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.706 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.706 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.706 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.46190.0:84908 57.4 57:2ff1ef3b:::benchmark_data_smithi070_360420_object17559:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1558)", 2024-09-21T11:01:32.706 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:00:40.424833+0000", 2024-09-21T11:01:32.706 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 52.178915336000003, 2024-09-21T11:01:32.706 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.16546303400000001, 2024-09-21T11:01:32.706 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:01:32.706 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:01:32.706 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:01:32.706 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:01:32.706 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.46190", 2024-09-21T11:01:32.707 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:01:32.707 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 84908 2024-09-21T11:01:32.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.707 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:01:32.707 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.707 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:01:32.707 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.424833+0000", 2024-09-21T11:01:32.707 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.707 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.707 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:01:32.707 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.424833+0000", 2024-09-21T11:01:32.707 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.708 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.708 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.708 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:01:32.708 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.424835+0000", 2024-09-21T11:01:32.708 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.094e-06 2024-09-21T11:01:32.708 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.708 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.708 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:01:32.708 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.424839+0000", 2024-09-21T11:01:32.708 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0500000000000002e-06 2024-09-21T11:01:32.708 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.708 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.708 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:01:32.708 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.424848+0000", 2024-09-21T11:01:32.709 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.8959999999999998e-06 2024-09-21T11:01:32.709 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.709 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.709 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:01:32.709 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.424853+0000", 2024-09-21T11:01:32.709 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0159999999999999e-06 2024-09-21T11:01:32.709 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.709 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.709 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.709 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.424900+0000", 2024-09-21T11:01:32.709 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6684999999999998e-05 2024-09-21T11:01:32.709 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.709 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.709 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:01:32.710 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.425031+0000", 2024-09-21T11:01:32.710 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013092400000000001 2024-09-21T11:01:32.710 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.710 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.710 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,7", 2024-09-21T11:01:32.710 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.425152+0000", 2024-09-21T11:01:32.710 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000121193 2024-09-21T11:01:32.710 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.710 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.710 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:01:32.710 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.428093+0000", 2024-09-21T11:01:32.710 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0029411789999999999 2024-09-21T11:01:32.710 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.711 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.711 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:01:32.711 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.428411+0000", 2024-09-21T11:01:32.711 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00031706699999999998 2024-09-21T11:01:32.711 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.711 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.711 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.711 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.590288+0000", 2024-09-21T11:01:32.711 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.161877623 2024-09-21T11:01:32.711 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.711 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.712 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:01:32.712 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.590296+0000", 2024-09-21T11:01:32.712 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3070000000000003e-06 2024-09-21T11:01:32.712 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.712 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:01:32.712 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.712 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.712 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.712 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.46190.0:84902 57.13 57:cba41bdc:::benchmark_data_smithi070_360420_object17553:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1559)", 2024-09-21T11:01:32.712 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:00:40.449158+0000", 2024-09-21T11:01:32.712 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 52.154590923999997, 2024-09-21T11:01:32.712 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.004033234, 2024-09-21T11:01:32.712 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:01:32.712 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:01:32.713 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:01:32.713 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:01:32.713 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.46190", 2024-09-21T11:01:32.713 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:01:32.713 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 84902 2024-09-21T11:01:32.713 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.713 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:01:32.713 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.713 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:01:32.713 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449158+0000", 2024-09-21T11:01:32.713 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.713 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.713 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.713 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:01:32.714 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449158+0000", 2024-09-21T11:01:32.714 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.714 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.714 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:01:32.714 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449161+0000", 2024-09-21T11:01:32.714 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.897e-06 2024-09-21T11:01:32.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.714 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.714 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:01:32.714 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449172+0000", 2024-09-21T11:01:32.714 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1392e-05 2024-09-21T11:01:32.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.715 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.715 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:01:32.715 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449203+0000", 2024-09-21T11:01:32.715 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0540000000000002e-05 2024-09-21T11:01:32.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.715 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.715 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:01:32.715 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449216+0000", 2024-09-21T11:01:32.715 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:32.717 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3678e-05 2024-09-21T11:01:32.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.717 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.717 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.717 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.596137+0000", 2024-09-21T11:01:32.717 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.14692064099999999 2024-09-21T11:01:32.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.717 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.717 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:01:32.717 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.596145+0000", 2024-09-21T11:01:32.717 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.909e-06 2024-09-21T11:01:32.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.718 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.718 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.718 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.453118+0000", 2024-09-21T11:01:32.718 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.85697302600000003 2024-09-21T11:01:32.718 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.718 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.718 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:01:32.718 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.453191+0000", 2024-09-21T11:01:32.718 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3151000000000005e-05 2024-09-21T11:01:32.718 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.718 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:01:32.718 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.718 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.719 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.719 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.46190.0:84904 57.14 57:2ac37d6e:::benchmark_data_smithi070_360420_object17555:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1559)", 2024-09-21T11:01:32.719 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:00:40.449248+0000", 2024-09-21T11:01:32.719 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 52.154500595999998, 2024-09-21T11:01:32.719 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.028813749, 2024-09-21T11:01:32.719 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:01:32.719 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:01:32.719 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:01:32.719 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:01:32.719 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.46190", 2024-09-21T11:01:32.719 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:01:32.719 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 84904 2024-09-21T11:01:32.719 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.720 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:01:32.720 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.720 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:01:32.720 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449248+0000", 2024-09-21T11:01:32.720 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.720 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.720 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.720 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:01:32.720 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449248+0000", 2024-09-21T11:01:32.720 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.720 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.720 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.720 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:01:32.720 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449249+0000", 2024-09-21T11:01:32.721 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1799999999999995e-07 2024-09-21T11:01:32.721 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.721 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.721 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:01:32.721 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449251+0000", 2024-09-21T11:01:32.721 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1789999999999998e-06 2024-09-21T11:01:32.721 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.721 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.721 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:01:32.721 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449257+0000", 2024-09-21T11:01:32.721 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2179999999999996e-06 2024-09-21T11:01:32.721 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.721 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.722 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:01:32.722 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449262+0000", 2024-09-21T11:01:32.722 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2310000000000004e-06 2024-09-21T11:01:32.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.722 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.722 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.722 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.590563+0000", 2024-09-21T11:01:32.722 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.14130077999999999 2024-09-21T11:01:32.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.722 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.722 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:01:32.722 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.590571+0000", 2024-09-21T11:01:32.722 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9200000000000004e-06 2024-09-21T11:01:32.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.723 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.723 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.723 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.478003+0000", 2024-09-21T11:01:32.723 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.88743153100000005 2024-09-21T11:01:32.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.723 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.723 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:01:32.723 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.478062+0000", 2024-09-21T11:01:32.723 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9372e-05 2024-09-21T11:01:32.723 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.723 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:01:32.723 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.724 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.724 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.46190.0:84908 57.14 57:2ff1ef3b:::benchmark_data_smithi070_360420_object17559:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1559)", 2024-09-21T11:01:32.724 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:00:40.449269+0000", 2024-09-21T11:01:32.724 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 52.154479246999998, 2024-09-21T11:01:32.724 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.035143608, 2024-09-21T11:01:32.724 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:01:32.724 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:01:32.724 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:01:32.724 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:01:32.724 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.46190", 2024-09-21T11:01:32.724 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:01:32.724 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 84908 2024-09-21T11:01:32.724 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.724 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:01:32.725 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.725 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:01:32.725 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449269+0000", 2024-09-21T11:01:32.725 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.725 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.725 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.725 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:01:32.725 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449269+0000", 2024-09-21T11:01:32.725 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.725 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.725 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.725 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:01:32.725 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449270+0000", 2024-09-21T11:01:32.725 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4200000000000002e-07 2024-09-21T11:01:32.726 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.726 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.726 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:01:32.726 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449271+0000", 2024-09-21T11:01:32.726 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5650000000000001e-06 2024-09-21T11:01:32.726 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.726 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.726 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:01:32.726 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449276+0000", 2024-09-21T11:01:32.726 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.87e-06 2024-09-21T11:01:32.726 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.726 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.726 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:01:32.727 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449280+0000", 2024-09-21T11:01:32.727 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0679999999999998e-06 2024-09-21T11:01:32.727 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.727 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.727 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.727 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.590591+0000", 2024-09-21T11:01:32.727 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.14131036599999999 2024-09-21T11:01:32.727 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.727 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.727 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:01:32.727 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.590596+0000", 2024-09-21T11:01:32.727 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4190000000000001e-06 2024-09-21T11:01:32.727 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.727 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.728 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.728 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.478340+0000", 2024-09-21T11:01:32.728 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.88774418499999996 2024-09-21T11:01:32.728 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.728 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.728 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for missing object", 2024-09-21T11:01:32.728 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.478458+0000", 2024-09-21T11:01:32.728 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000117686 2024-09-21T11:01:32.728 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.728 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.728 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.728 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.484331+0000", 2024-09-21T11:01:32.728 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0058731469999999996 2024-09-21T11:01:32.728 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.729 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.729 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:01:32.729 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.484413+0000", 2024-09-21T11:01:32.729 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1959999999999995e-05 2024-09-21T11:01:32.729 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.729 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:01:32.729 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.729 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.729 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.729 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.46190.0:84914 57.12 57:4f5dccec:::benchmark_data_smithi070_360420_object17565:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1559)", 2024-09-21T11:01:32.729 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:00:40.592927+0000", 2024-09-21T11:01:32.729 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 52.010821665000002, 2024-09-21T11:01:32.729 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.88331985400000002, 2024-09-21T11:01:32.729 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:01:32.730 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:01:32.730 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:01:32.730 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:01:32.730 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.46190", 2024-09-21T11:01:32.730 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:01:32.730 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 84914 2024-09-21T11:01:32.730 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.730 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:01:32.730 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.730 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:01:32.730 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.592927+0000", 2024-09-21T11:01:32.730 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.730 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.730 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.731 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:01:32.731 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.592927+0000", 2024-09-21T11:01:32.731 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.731 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.731 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.731 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:01:32.731 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.592929+0000", 2024-09-21T11:01:32.731 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2369999999999999e-06 2024-09-21T11:01:32.731 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.731 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.731 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:01:32.731 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.592942+0000", 2024-09-21T11:01:32.731 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2609000000000001e-05 2024-09-21T11:01:32.732 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.732 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.732 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:01:32.732 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.592969+0000", 2024-09-21T11:01:32.732 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7161999999999998e-05 2024-09-21T11:01:32.732 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.732 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.732 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:01:32.732 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.592984+0000", 2024-09-21T11:01:32.732 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4764e-05 2024-09-21T11:01:32.732 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.732 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.732 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.732 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.593075+0000", 2024-09-21T11:01:32.732 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0901000000000003e-05 2024-09-21T11:01:32.733 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.733 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.733 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:01:32.733 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.593086+0000", 2024-09-21T11:01:32.733 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0927999999999999e-05 2024-09-21T11:01:32.733 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.733 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.733 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.733 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.473920+0000", 2024-09-21T11:01:32.733 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.880834483 2024-09-21T11:01:32.733 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.733 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.733 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:01:32.733 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.474063+0000", 2024-09-21T11:01:32.734 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000142755 2024-09-21T11:01:32.734 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.734 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.734 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,4", 2024-09-21T11:01:32.734 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.474172+0000", 2024-09-21T11:01:32.734 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000108812 2024-09-21T11:01:32.734 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.734 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.734 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:01:32.734 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.476130+0000", 2024-09-21T11:01:32.734 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001958745 2024-09-21T11:01:32.734 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.734 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.735 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:01:32.735 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.476158+0000", 2024-09-21T11:01:32.735 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7339e-05 2024-09-21T11:01:32.735 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.735 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.735 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:01:32.735 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.476186+0000", 2024-09-21T11:01:32.735 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7902000000000002e-05 2024-09-21T11:01:32.735 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.735 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.735 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:01:32.735 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.476220+0000", 2024-09-21T11:01:32.735 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4038000000000002e-05 2024-09-21T11:01:32.735 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.736 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.736 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:01:32.736 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.476247+0000", 2024-09-21T11:01:32.736 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7178999999999998e-05 2024-09-21T11:01:32.736 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.736 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:01:32.736 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.736 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.736 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.736 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.46190.0:111593 57.12 57:49439ee5:::benchmark_data_smithi070_360420_object44244:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1567)", 2024-09-21T11:01:32.736 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:01:05.787639+0000", 2024-09-21T11:01:32.736 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 26.816109629, 2024-09-21T11:01:32.736 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.087192752999999998, 2024-09-21T11:01:32.736 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:01:32.737 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:01:32.737 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:01:32.737 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:01:32.737 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.46190", 2024-09-21T11:01:32.737 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:01:32.737 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 111593 2024-09-21T11:01:32.737 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.737 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:01:32.737 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.737 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:01:32.737 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.787639+0000", 2024-09-21T11:01:32.737 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.737 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.737 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.738 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:01:32.738 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.787639+0000", 2024-09-21T11:01:32.738 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.738 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.738 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.738 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:01:32.738 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.787641+0000", 2024-09-21T11:01:32.738 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.689e-06 2024-09-21T11:01:32.738 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.738 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.738 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:01:32.738 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.787647+0000", 2024-09-21T11:01:32.738 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0050000000000001e-06 2024-09-21T11:01:32.739 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.739 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.739 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:01:32.739 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.787661+0000", 2024-09-21T11:01:32.739 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4276e-05 2024-09-21T11:01:32.739 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.739 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.739 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:01:32.739 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.787670+0000", 2024-09-21T11:01:32.739 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0440000000000001e-06 2024-09-21T11:01:32.739 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.739 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.739 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.739 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.787733+0000", 2024-09-21T11:01:32.740 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2513000000000003e-05 2024-09-21T11:01:32.740 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.740 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.740 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:01:32.740 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.787854+0000", 2024-09-21T11:01:32.740 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000121168 2024-09-21T11:01:32.740 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.740 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.740 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,4", 2024-09-21T11:01:32.740 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.787948+0000", 2024-09-21T11:01:32.740 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4580000000000006e-05 2024-09-21T11:01:32.740 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.740 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.740 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:01:32.740 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.789542+0000", 2024-09-21T11:01:32.741 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001593934 2024-09-21T11:01:32.741 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.741 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.741 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:01:32.741 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.790272+0000", 2024-09-21T11:01:32.741 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00072944800000000005 2024-09-21T11:01:32.741 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.741 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.741 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:01:32.741 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.874635+0000", 2024-09-21T11:01:32.741 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.084363503000000006 2024-09-21T11:01:32.741 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.741 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.741 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:01:32.742 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.874736+0000", 2024-09-21T11:01:32.742 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000100992 2024-09-21T11:01:32.742 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.742 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.742 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:01:32.742 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.874832+0000", 2024-09-21T11:01:32.742 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5600999999999995e-05 2024-09-21T11:01:32.742 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.742 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:01:32.742 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.742 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.742 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.742 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.46190.0:111598 57.2 57:40597864:::benchmark_data_smithi070_360420_object44249:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1567)", 2024-09-21T11:01:32.742 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:01:05.787960+0000", 2024-09-21T11:01:32.743 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 26.815788964999999, 2024-09-21T11:01:32.743 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.087068255999999997, 2024-09-21T11:01:32.743 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:01:32.743 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:01:32.743 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:01:32.743 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:01:32.743 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.46190", 2024-09-21T11:01:32.743 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:01:32.743 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 111598 2024-09-21T11:01:32.743 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.743 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:01:32.743 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.743 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:01:32.744 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.787960+0000", 2024-09-21T11:01:32.744 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.744 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.744 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.744 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:01:32.744 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.787960+0000", 2024-09-21T11:01:32.744 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.744 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.744 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.744 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:01:32.744 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.787961+0000", 2024-09-21T11:01:32.744 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5799999999999998e-07 2024-09-21T11:01:32.744 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.744 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.744 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:01:32.745 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.787965+0000", 2024-09-21T11:01:32.745 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0409999999999999e-06 2024-09-21T11:01:32.745 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.745 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.745 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:01:32.745 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.787974+0000", 2024-09-21T11:01:32.745 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9220000000000003e-06 2024-09-21T11:01:32.745 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.745 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.745 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:01:32.745 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.787979+0000", 2024-09-21T11:01:32.745 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7599999999999999e-06 2024-09-21T11:01:32.745 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.746 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.746 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.746 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.788051+0000", 2024-09-21T11:01:32.746 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1582999999999995e-05 2024-09-21T11:01:32.746 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.746 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.746 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:01:32.746 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.788169+0000", 2024-09-21T11:01:32.746 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011846 2024-09-21T11:01:32.746 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.746 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.746 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,4", 2024-09-21T11:01:32.746 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.788282+0000", 2024-09-21T11:01:32.746 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000112099 2024-09-21T11:01:32.747 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.747 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.747 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:01:32.747 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.789867+0000", 2024-09-21T11:01:32.747 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00158511 2024-09-21T11:01:32.747 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.747 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.747 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:01:32.747 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.790607+0000", 2024-09-21T11:01:32.747 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00074058200000000005 2024-09-21T11:01:32.747 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.747 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.747 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:01:32.748 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.874871+0000", 2024-09-21T11:01:32.748 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.084263550000000007 2024-09-21T11:01:32.748 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.748 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.748 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:01:32.748 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.874946+0000", 2024-09-21T11:01:32.748 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4930000000000003e-05 2024-09-21T11:01:32.748 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.748 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.748 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:01:32.748 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:05.875028+0000", 2024-09-21T11:01:32.748 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2361000000000006e-05 2024-09-21T11:01:32.748 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.748 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:01:32.749 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.749 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.749 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.749 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.46190.0:118447 57.2 57:452f19cf:::benchmark_data_smithi070_360420_object51098:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1567)", 2024-09-21T11:01:32.749 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:01:10.373314+0000", 2024-09-21T11:01:32.749 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 22.230434283000001, 2024-09-21T11:01:32.749 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.089636080000000007, 2024-09-21T11:01:32.749 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:01:32.749 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:01:32.749 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:01:32.749 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:01:32.749 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.46190", 2024-09-21T11:01:32.749 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:01:32.750 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 118447 2024-09-21T11:01:32.750 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.750 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:01:32.750 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.750 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:01:32.750 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.373314+0000", 2024-09-21T11:01:32.750 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.750 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.750 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.750 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:01:32.750 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.373314+0000", 2024-09-21T11:01:32.750 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.750 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.751 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.751 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:01:32.751 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.373316+0000", 2024-09-21T11:01:32.751 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.516e-06 2024-09-21T11:01:32.751 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.751 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.751 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:01:32.751 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.373321+0000", 2024-09-21T11:01:32.751 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.78e-06 2024-09-21T11:01:32.751 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.751 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.751 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:01:32.751 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.373332+0000", 2024-09-21T11:01:32.752 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1562999999999999e-05 2024-09-21T11:01:32.752 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.752 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.752 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:01:32.752 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.373339+0000", 2024-09-21T11:01:32.752 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0910000000000004e-06 2024-09-21T11:01:32.752 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.752 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.752 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.752 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.373402+0000", 2024-09-21T11:01:32.752 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2388e-05 2024-09-21T11:01:32.752 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.752 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.752 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:01:32.753 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.373502+0000", 2024-09-21T11:01:32.753 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9946999999999997e-05 2024-09-21T11:01:32.753 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.753 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.753 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,4", 2024-09-21T11:01:32.753 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.373587+0000", 2024-09-21T11:01:32.753 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5630000000000005e-05 2024-09-21T11:01:32.753 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.753 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.753 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:01:32.753 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.375182+0000", 2024-09-21T11:01:32.753 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0015945499999999999 2024-09-21T11:01:32.753 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.753 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.753 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:01:32.754 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.375930+0000", 2024-09-21T11:01:32.754 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00074852100000000004 2024-09-21T11:01:32.754 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.754 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.754 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:01:32.754 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.462763+0000", 2024-09-21T11:01:32.754 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.086832723000000001 2024-09-21T11:01:32.754 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.754 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.754 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:01:32.754 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.462836+0000", 2024-09-21T11:01:32.754 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2571999999999996e-05 2024-09-21T11:01:32.754 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.755 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.755 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:01:32.755 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.462951+0000", 2024-09-21T11:01:32.755 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000114799 2024-09-21T11:01:32.755 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.755 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:01:32.755 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.755 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.755 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.755 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.46190.0:118451 57.14 57:2d1fad81:::benchmark_data_smithi070_360420_object51102:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1567)", 2024-09-21T11:01:32.755 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:01:10.373556+0000", 2024-09-21T11:01:32.755 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 22.230192881000001, 2024-09-21T11:01:32.755 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.089856197999999998, 2024-09-21T11:01:32.756 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:01:32.756 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:01:32.756 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:01:32.756 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:01:32.756 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.46190", 2024-09-21T11:01:32.756 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:01:32.756 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 118451 2024-09-21T11:01:32.756 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.756 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:01:32.756 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.756 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:01:32.756 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.373556+0000", 2024-09-21T11:01:32.756 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.756 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.756 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.757 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:01:32.757 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.373556+0000", 2024-09-21T11:01:32.757 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:01:32.757 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.757 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.757 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:01:32.757 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.373557+0000", 2024-09-21T11:01:32.757 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2900000000000003e-07 2024-09-21T11:01:32.757 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.757 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.757 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:01:32.757 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.373560+0000", 2024-09-21T11:01:32.757 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.405e-06 2024-09-21T11:01:32.758 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.758 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.758 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:01:32.758 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.373568+0000", 2024-09-21T11:01:32.758 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8099999999999998e-06 2024-09-21T11:01:32.758 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.758 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.758 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:01:32.758 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.373573+0000", 2024-09-21T11:01:32.758 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9239999999999998e-06 2024-09-21T11:01:32.758 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.759 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.759 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:01:32.759 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.373610+0000", 2024-09-21T11:01:32.759 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6958000000000001e-05 2024-09-21T11:01:32.759 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.759 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.759 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:01:32.759 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.373692+0000", 2024-09-21T11:01:32.759 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2479999999999996e-05 2024-09-21T11:01:32.759 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.759 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.759 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,7", 2024-09-21T11:01:32.759 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.373756+0000", 2024-09-21T11:01:32.760 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3471000000000003e-05 2024-09-21T11:01:32.760 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.760 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.760 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:01:32.760 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.375157+0000", 2024-09-21T11:01:32.760 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00140175 2024-09-21T11:01:32.760 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.760 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.760 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:01:32.760 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.376679+0000", 2024-09-21T11:01:32.760 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0015213589999999999 2024-09-21T11:01:32.760 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.760 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.760 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:01:32.761 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.463304+0000", 2024-09-21T11:01:32.761 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.086625062000000003 2024-09-21T11:01:32.761 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.761 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.761 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:01:32.761 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.463361+0000", 2024-09-21T11:01:32.761 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7062999999999999e-05 2024-09-21T11:01:32.761 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:01:32.761 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:01:32.761 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:01:32.761 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:01:10.463412+0000", 2024-09-21T11:01:32.761 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1187000000000003e-05 2024-09-21T11:01:32.761 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.762 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:01:32.762 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.762 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:01:32.762 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:01:32.762 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T11:01:32.783 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:32.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:32.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:32.884 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:32.906 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:32.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:32.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:32.967 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T11:01:33.007 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:33.028 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:33.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:33.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:33.130 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:33.152 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:33.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:33.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:33.253 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:33.275 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:33.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:33.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:33.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:32 smithi160 ceph-mon[101879]: osdmap e1574: 8 total, 6 up, 4 in 2024-09-21T11:01:33.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:32 smithi160 ceph-mon[101879]: pgmap v3054: 119 pgs: 2 peering, 5 remapped+peering, 10 unknown, 70 active+clean+remapped, 17 active+undersized, 5 active+undersized+degraded, 10 active+clean; 11 MiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 51/612 objects degraded (8.333%); 149/612 objects misplaced (24.346%); 0 B/s, 83 objects/s recovering 2024-09-21T11:01:33.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:32 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4249927639' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:01:33.377 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:33.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:01:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:01:33] "GET /metrics HTTP/1.1" 200 36583 "" "Prometheus/2.51.0" 2024-09-21T11:01:33.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:32 smithi070 ceph-mon[107385]: osdmap e1574: 8 total, 6 up, 4 in 2024-09-21T11:01:33.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:32 smithi070 ceph-mon[107385]: pgmap v3054: 119 pgs: 2 peering, 5 remapped+peering, 10 unknown, 70 active+clean+remapped, 17 active+undersized, 5 active+undersized+degraded, 10 active+clean; 11 MiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 51/612 objects degraded (8.333%); 149/612 objects misplaced (24.346%); 0 B/s, 83 objects/s recovering 2024-09-21T11:01:33.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:32 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4249927639' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:01:33.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:32 smithi070 ceph-mon[112139]: osdmap e1574: 8 total, 6 up, 4 in 2024-09-21T11:01:33.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:32 smithi070 ceph-mon[112139]: pgmap v3054: 119 pgs: 2 peering, 5 remapped+peering, 10 unknown, 70 active+clean+remapped, 17 active+undersized, 5 active+undersized+degraded, 10 active+clean; 11 MiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 51/612 objects degraded (8.333%); 149/612 objects misplaced (24.346%); 0 B/s, 83 objects/s recovering 2024-09-21T11:01:33.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:32 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4249927639' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:01:33.399 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:33.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:33.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:33.500 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:33.521 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:33.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:33.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:33.623 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:33.645 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:33.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:33.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:33.746 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:33.768 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:33.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:33.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:33.786 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T11:01:33.870 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:33.891 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:33.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:33.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:33.993 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:34.008 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:01:34.014 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:34.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:34.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:34.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:33 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:01:34.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:33 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4249927639' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T11:01:34.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:33 smithi160 ceph-mon[101879]: osdmap e1575: 8 total, 6 up, 4 in 2024-09-21T11:01:34.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:33 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 51/612 objects degraded (8.333%), 5 pgs degraded, 18 pgs undersized (PG_DEGRADED) 2024-09-21T11:01:34.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:33 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:01:34.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:33 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:01:34.116 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:34.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:33 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:01:34.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:33 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4249927639' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T11:01:34.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:33 smithi070 ceph-mon[107385]: osdmap e1575: 8 total, 6 up, 4 in 2024-09-21T11:01:34.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:33 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 51/612 objects degraded (8.333%), 5 pgs degraded, 18 pgs undersized (PG_DEGRADED) 2024-09-21T11:01:34.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:33 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:01:34.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:33 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:01:34.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:33 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:01:34.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:33 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4249927639' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T11:01:34.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:33 smithi070 ceph-mon[112139]: osdmap e1575: 8 total, 6 up, 4 in 2024-09-21T11:01:34.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:33 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 51/612 objects degraded (8.333%), 5 pgs degraded, 18 pgs undersized (PG_DEGRADED) 2024-09-21T11:01:34.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:33 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:01:34.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:33 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:01:34.137 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:34.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:34.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:34.238 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:34.261 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:34.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:34.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:34.363 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:34.384 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:34.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:34.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:34.486 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:34.508 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:34.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:34.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:34.610 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:34.631 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:34.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:34.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:34.732 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:34.757 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:34.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:34.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:34.858 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:34.863 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7] out_osds: [2, 5, 0, 4] dead_osds: [5, 6] live_osds: [0, 1, 2, 3, 4, 7] 2024-09-21T11:01:34.863 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T11:01:34.863 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-21T11:01:34.863 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd in 4 2024-09-21T11:01:34.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:34.880 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:34.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:34.981 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:35.002 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:35.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:35.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:35.103 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:35.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:35.126 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:35.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:35.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:34 smithi070 ceph-mon[107385]: osdmap e1576: 8 total, 6 up, 4 in 2024-09-21T11:01:35.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:34 smithi070 ceph-mon[107385]: pgmap v3057: 119 pgs: 2 peering, 5 remapped+peering, 10 unknown, 70 active+clean+remapped, 17 active+undersized, 5 active+undersized+degraded, 10 active+clean; 11 MiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 51/612 objects degraded (8.333%); 149/612 objects misplaced (24.346%); 0 B/s, 101 objects/s recovering 2024-09-21T11:01:35.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:34 smithi070 ceph-mon[112139]: osdmap e1576: 8 total, 6 up, 4 in 2024-09-21T11:01:35.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:34 smithi070 ceph-mon[112139]: pgmap v3057: 119 pgs: 2 peering, 5 remapped+peering, 10 unknown, 70 active+clean+remapped, 17 active+undersized, 5 active+undersized+degraded, 10 active+clean; 11 MiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 51/612 objects degraded (8.333%); 149/612 objects misplaced (24.346%); 0 B/s, 101 objects/s recovering 2024-09-21T11:01:35.227 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:35.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:35.250 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:35.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:35.298 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:34 smithi160 ceph-mon[101879]: osdmap e1576: 8 total, 6 up, 4 in 2024-09-21T11:01:35.298 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:34 smithi160 ceph-mon[101879]: pgmap v3057: 119 pgs: 2 peering, 5 remapped+peering, 10 unknown, 70 active+clean+remapped, 17 active+undersized, 5 active+undersized+degraded, 10 active+clean; 11 MiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 51/612 objects degraded (8.333%); 149/612 objects misplaced (24.346%); 0 B/s, 101 objects/s recovering 2024-09-21T11:01:35.352 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:35.373 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:35.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:35.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:35.475 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:35.501 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:35.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:35.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:35.603 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:35.628 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:35.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:35.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:35.730 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:35.750 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:35.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:35.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:35.852 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:35.878 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:35.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:35.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:35.979 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:36.000 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:36.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:36.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:36.102 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:36.128 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:36.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:36.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:36.230 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:36.251 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:36.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:36.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:01:36.353 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:36.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:01:36.378 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:36.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:01:36.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:36.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:36.383 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:01:36.385 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:01:36.480 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:36.501 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:36.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:36.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:36.513 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T11:01:36.603 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:36.629 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:36.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:36.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:36.731 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:36.752 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:36.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:36.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:36.854 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:36.875 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:36.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:36.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:36.977 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:36.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:37.000 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:37.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:37.101 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:37.117 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:36 smithi070 ceph-mon[107385]: pgmap v3058: 119 pgs: 83 active+clean+remapped, 21 active+undersized, 5 active+undersized+degraded, 10 active+clean; 11 MiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 51/612 objects degraded (8.333%); 150/612 objects misplaced (24.510%); 0 B/s, 76 objects/s recovering 2024-09-21T11:01:37.117 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:36 smithi070 ceph-mon[112139]: pgmap v3058: 119 pgs: 83 active+clean+remapped, 21 active+undersized, 5 active+undersized+degraded, 10 active+clean; 11 MiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 51/612 objects degraded (8.333%); 150/612 objects misplaced (24.510%); 0 B/s, 76 objects/s recovering 2024-09-21T11:01:37.126 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:37.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:37.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:37.229 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:37.255 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:37.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:37.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:37.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:36 smithi160 ceph-mon[101879]: pgmap v3058: 119 pgs: 83 active+clean+remapped, 21 active+undersized, 5 active+undersized+degraded, 10 active+clean; 11 MiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 51/612 objects degraded (8.333%); 150/612 objects misplaced (24.510%); 0 B/s, 76 objects/s recovering 2024-09-21T11:01:37.357 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:37.379 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:37.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:37.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:37.480 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:37.502 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:37.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:37.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:37.603 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:37.624 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:37.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:37.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:37.725 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:37.746 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:37.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:37.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:37.848 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:37.870 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:37.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:37.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:37.972 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:37.992 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'unique_pool_4' removed 2024-09-21T11:01:37.994 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:37.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:37.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:38.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:37 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3337274655' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T11:01:38.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:37 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T11:01:38.004 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:37 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3337274655' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T11:01:38.004 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:37 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T11:01:38.063 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T11:01:38.063 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T11:01:38.063 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T11:01:38.063 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T11:01:38.063 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:37 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3337274655' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T11:01:38.064 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:37 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T11:01:38.096 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:38.117 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:38.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:38.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:38.219 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:38.242 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:38.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:38.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:38.343 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:38.364 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:38.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:38.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:38.466 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:38.488 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:38.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:38.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:38.589 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:38.611 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:38.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:38.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:38.712 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:38.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:38.735 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:38.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:38.773 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T11:01:38.836 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:38.878 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:38.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:38.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:38.980 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:39.003 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:39.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:39.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:39.031 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:01:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:38 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T11:01:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:38 smithi160 ceph-mon[101879]: osdmap e1577: 8 total, 6 up, 4 in 2024-09-21T11:01:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:38 smithi160 ceph-mon[101879]: pgmap v3060: 93 pgs: 66 active+clean+remapped, 18 active+undersized, 3 active+undersized+degraded, 6 active+clean; 268 KiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 6/75 objects degraded (8.000%); 18/75 objects misplaced (24.000%) 2024-09-21T11:01:39.104 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:39.125 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:39.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:39.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:39.226 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:39.247 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:39.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:39.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:39.349 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:39.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:39.371 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:39.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:39.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:38 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T11:01:39.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:38 smithi070 ceph-mon[107385]: osdmap e1577: 8 total, 6 up, 4 in 2024-09-21T11:01:39.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:38 smithi070 ceph-mon[107385]: pgmap v3060: 93 pgs: 66 active+clean+remapped, 18 active+undersized, 3 active+undersized+degraded, 6 active+clean; 268 KiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 6/75 objects degraded (8.000%); 18/75 objects misplaced (24.000%) 2024-09-21T11:01:39.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:38 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T11:01:39.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:38 smithi070 ceph-mon[112139]: osdmap e1577: 8 total, 6 up, 4 in 2024-09-21T11:01:39.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:38 smithi070 ceph-mon[112139]: pgmap v3060: 93 pgs: 66 active+clean+remapped, 18 active+undersized, 3 active+undersized+degraded, 6 active+clean; 268 KiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 6/75 objects degraded (8.000%); 18/75 objects misplaced (24.000%) 2024-09-21T11:01:39.472 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:39.492 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:01:39.493 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:39.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:39.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:39.595 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:39.617 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:39.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:39.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:39.718 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:39.739 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:39.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:39.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:39.841 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:39.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:39.864 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:39.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:39.883 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T11:01:39.884 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T11:01:39.884 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_5 2024-09-21T11:01:39.884 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_5 16 2024-09-21T11:01:39.965 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:39.987 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:39.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:39.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:40.089 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:40.110 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:40.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:40.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:40.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:39 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 6/75 objects degraded (8.000%), 3 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-21T11:01:40.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:39 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 6/75 objects degraded (8.000%), 3 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-21T11:01:40.212 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:40.233 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:40.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:40.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:40.268 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:39 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 6/75 objects degraded (8.000%), 3 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-21T11:01:40.334 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:40.355 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:40.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:40.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:40.457 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:40.478 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:40.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:40.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:40.580 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:40.604 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:40.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:40.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:40.671 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T11:01:40.671 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T11:01:40.671 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T11:01:40.671 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T11:01:40.671 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T11:01:40.706 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:40.728 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:40.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:40.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:40.828 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:01:40.828 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:01:40.829 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:40.838 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:01:40.838 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:01:40.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:40.852 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:40.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:40.854 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:01:40.855 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:01:40.874 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:01:40.874 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:01:40.892 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:01:40.892 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:01:40.893 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T11:01:40.893 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T11:01:40.909 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:01:40.909 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:01:40.952 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:40.974 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:40.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:40.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:41.076 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:41.099 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:41.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:41.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:41.107 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:40 smithi070 ceph-mon[107385]: pgmap v3061: 93 pgs: 66 active+clean+remapped, 18 active+undersized, 3 active+undersized+degraded, 6 active+clean; 268 KiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 6/75 objects degraded (8.000%); 18/75 objects misplaced (24.000%) 2024-09-21T11:01:41.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:40 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/667758231' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:01:41.108 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:40 smithi070 ceph-mon[112139]: pgmap v3061: 93 pgs: 66 active+clean+remapped, 18 active+undersized, 3 active+undersized+degraded, 6 active+clean; 268 KiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 6/75 objects degraded (8.000%); 18/75 objects misplaced (24.000%) 2024-09-21T11:01:41.108 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:40 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/667758231' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:01:41.200 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:41.222 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:41.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:41.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:41.324 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:41.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:40 smithi160 ceph-mon[101879]: pgmap v3061: 93 pgs: 66 active+clean+remapped, 18 active+undersized, 3 active+undersized+degraded, 6 active+clean; 268 KiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 6/75 objects degraded (8.000%); 18/75 objects misplaced (24.000%) 2024-09-21T11:01:41.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:40 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/667758231' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:01:41.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:41.346 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:41.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:41.421 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T11:01:41.447 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:41.469 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:41.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:41.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:41.570 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:41.593 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:41.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:41.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:41.619 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:01:41.695 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:41.706 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:01:41.717 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:41.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:41.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:41.818 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:41.839 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:41.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:41.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:41.941 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:41.964 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:41.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:41.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:42.066 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:42.087 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:42.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:42.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:42.189 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:42.210 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:42.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:42.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:42.311 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:42.334 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:42.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:42.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:42.436 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:42.457 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:42.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:42.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:42.558 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:42.580 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:42.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:42.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:42.681 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:42.704 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:42.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:42.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:42.806 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:42.828 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:42.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:42.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:42.930 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:42.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:42.951 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:42.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:43.052 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:43.088 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:43.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:43.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:43.190 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:43.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:43.213 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:43.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:43.313 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:43.335 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:43.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:43.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:43.360 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T11:01:43.360 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T11:01:43.437 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:43.460 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:43.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:43.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:43.528 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:43 smithi070 ceph-mon[107385]: pgmap v3062: 93 pgs: 66 active+clean+remapped, 18 active+undersized, 3 active+undersized+degraded, 6 active+clean; 268 KiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 6/75 objects degraded (8.000%); 18/75 objects misplaced (24.000%) 2024-09-21T11:01:43.528 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:01:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:01:43] "GET /metrics HTTP/1.1" 200 36566 "" "Prometheus/2.51.0" 2024-09-21T11:01:43.528 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:43 smithi070 ceph-mon[112139]: pgmap v3062: 93 pgs: 66 active+clean+remapped, 18 active+undersized, 3 active+undersized+degraded, 6 active+clean; 268 KiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 6/75 objects degraded (8.000%); 18/75 objects misplaced (24.000%) 2024-09-21T11:01:43.562 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:43.583 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:43.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:43.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:43.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:43 smithi160 ceph-mon[101879]: pgmap v3062: 93 pgs: 66 active+clean+remapped, 18 active+undersized, 3 active+undersized+degraded, 6 active+clean; 268 KiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 6/75 objects degraded (8.000%); 18/75 objects misplaced (24.000%) 2024-09-21T11:01:43.685 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:43.707 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:43.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:43.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:43.808 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:43.829 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:43.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:43.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:43.928 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T11:01:43.928 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T11:01:43.931 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:43.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:43.954 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:43.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:44.055 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:44.059 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T11:01:44.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:44.077 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:44.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:44.179 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:44.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:44.201 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:44.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:44.269 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:01:44.302 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:44.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:44.324 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:44.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:44.425 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:44.447 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:44.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:44.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:44.548 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:44.570 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:44.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:44.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:44.671 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:44.692 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:44.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:44.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:44.794 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:44.815 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:44.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:44.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:44.917 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:44.938 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:44.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:44.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:45.023 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a[107362]: 2024-09-21T11:01:44.747+0000 7fad4a465640 -1 mon.a@0(leader).osd e1577 definitely_dead 0 2024-09-21T11:01:45.040 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:45.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:45.063 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:45.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:45.163 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:45.184 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:45.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:45.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:45.286 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:45.307 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:45.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:45.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:45.409 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:45.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:45.432 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:45.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:45.452 INFO:teuthology.orchestra.run.smithi070.stderr:marked in osd.4. 2024-09-21T11:01:45.534 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:45.555 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:45.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:45.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:45.601 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:45 smithi070 ceph-mon[107385]: pgmap v3063: 93 pgs: 66 active+clean+remapped, 18 active+undersized, 3 active+undersized+degraded, 6 active+clean; 268 KiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 6/75 objects degraded (8.000%); 18/75 objects misplaced (24.000%) 2024-09-21T11:01:45.601 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:45 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3920762021' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T11:01:45.601 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:45 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T11:01:45.602 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:45 smithi070 ceph-mon[112139]: pgmap v3063: 93 pgs: 66 active+clean+remapped, 18 active+undersized, 3 active+undersized+degraded, 6 active+clean; 268 KiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 6/75 objects degraded (8.000%); 18/75 objects misplaced (24.000%) 2024-09-21T11:01:45.602 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:45 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3920762021' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T11:01:45.602 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:45 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T11:01:45.657 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:45.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:45.679 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:45.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:45.727 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:45 smithi160 ceph-mon[101879]: pgmap v3063: 93 pgs: 66 active+clean+remapped, 18 active+undersized, 3 active+undersized+degraded, 6 active+clean; 268 KiB data, 6.9 GiB used, 351 GiB / 358 GiB avail; 6/75 objects degraded (8.000%); 18/75 objects misplaced (24.000%) 2024-09-21T11:01:45.727 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:45 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3920762021' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T11:01:45.727 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:45 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T11:01:45.780 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:45.801 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:45.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:45.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:45.903 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:45.925 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:45.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:45.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:45.929 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T11:01:45.949 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T11:01:45.949 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T11:01:46.027 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:46.049 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:46.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:46.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:46.150 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:46.172 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:46.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:46.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:46.275 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:46.298 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:46.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:46.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:46.350 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-21T11:01:46.399 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:46.420 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:46.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:46.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:46.522 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:46.535 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:46 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-21T11:01:46.535 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:46 smithi160 ceph-mon[101879]: osdmap e1578: 8 total, 6 up, 5 in 2024-09-21T11:01:46.535 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:46 smithi160 ceph-mon[101879]: osdmap e1579: 8 total, 6 up, 5 in 2024-09-21T11:01:46.543 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:46.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:46.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:46.555 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T11:01:46.618 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:46 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-21T11:01:46.618 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:46 smithi070 ceph-mon[112139]: osdmap e1578: 8 total, 6 up, 5 in 2024-09-21T11:01:46.618 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:46 smithi070 ceph-mon[112139]: osdmap e1579: 8 total, 6 up, 5 in 2024-09-21T11:01:46.618 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:46 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-21T11:01:46.618 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:46 smithi070 ceph-mon[107385]: osdmap e1578: 8 total, 6 up, 5 in 2024-09-21T11:01:46.618 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:46 smithi070 ceph-mon[107385]: osdmap e1579: 8 total, 6 up, 5 in 2024-09-21T11:01:46.645 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:46.665 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:46.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:46.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:46.748 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:01:46.766 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:46.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:46.788 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:46.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:46.888 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:46.909 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:46.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:46.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:47.011 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:47.032 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:47.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:47.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:47.134 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:47.155 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:47.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:47.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:47.256 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:47.277 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:47.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:47.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:47.379 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:47.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:47.402 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:47.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:47.455 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T11:01:47.475 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:47 smithi070 ceph-mon[107385]: pgmap v3065: 93 pgs: 5 peering, 25 remapped+peering, 37 active+clean+remapped, 18 active+undersized, 3 active+undersized+degraded, 5 active+clean; 268 KiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 6/75 objects degraded (8.000%); 4/75 objects misplaced (5.333%) 2024-09-21T11:01:47.475 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:47 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2413080563' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:01:47.475 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:47 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:01:47.475 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:47 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:01:47.475 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:47 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T11:01:47.475 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:47 smithi070 ceph-mon[107385]: osdmap e1580: 8 total, 6 up, 5 in 2024-09-21T11:01:47.475 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:47 smithi070 ceph-mon[112139]: pgmap v3065: 93 pgs: 5 peering, 25 remapped+peering, 37 active+clean+remapped, 18 active+undersized, 3 active+undersized+degraded, 5 active+clean; 268 KiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 6/75 objects degraded (8.000%); 4/75 objects misplaced (5.333%) 2024-09-21T11:01:47.476 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:47 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2413080563' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:01:47.476 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:47 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:01:47.476 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:47 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:01:47.476 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:47 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T11:01:47.476 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:47 smithi070 ceph-mon[112139]: osdmap e1580: 8 total, 6 up, 5 in 2024-09-21T11:01:47.503 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:47.525 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:47.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:47.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:47.627 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:47.649 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:47.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:47.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:47.750 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:47.771 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:47.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:47.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:47.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:47 smithi160 ceph-mon[101879]: pgmap v3065: 93 pgs: 5 peering, 25 remapped+peering, 37 active+clean+remapped, 18 active+undersized, 3 active+undersized+degraded, 5 active+clean; 268 KiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 6/75 objects degraded (8.000%); 4/75 objects misplaced (5.333%) 2024-09-21T11:01:47.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:47 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2413080563' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:01:47.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:47 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:01:47.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:47 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:01:47.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:47 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T11:01:47.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:47 smithi160 ceph-mon[101879]: osdmap e1580: 8 total, 6 up, 5 in 2024-09-21T11:01:47.873 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:47.895 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:47.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:47.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:47.896 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:01:47.997 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:48.019 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:48.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:48.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:48.121 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:48.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:48.143 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:48.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:48.244 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:48.266 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:48.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:48.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:48.368 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:48.390 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:48.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:48.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:48.420 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T11:01:48.420 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T11:01:48.476 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'unique_pool_5' created 2024-09-21T11:01:48.492 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:48.514 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:48.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:48.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:48.615 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:48.637 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:48.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:48.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:48.739 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:48.758 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:48 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/9215109' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-21T11:01:48.758 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:48 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-21T11:01:48.758 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:48 smithi160 ceph-mon[101879]: 1.11 scrub starts 2024-09-21T11:01:48.758 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:48 smithi160 ceph-mon[101879]: 1.11 scrub ok 2024-09-21T11:01:48.758 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:48 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:01:48.758 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:48 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:01:48.760 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:48.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:48.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:48.862 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:48.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:48 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/9215109' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-21T11:01:48.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:48 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-21T11:01:48.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:48 smithi070 ceph-mon[112139]: 1.11 scrub starts 2024-09-21T11:01:48.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:48 smithi070 ceph-mon[112139]: 1.11 scrub ok 2024-09-21T11:01:48.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:48 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:01:48.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:48 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:01:48.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:48 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/9215109' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-21T11:01:48.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:48 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-21T11:01:48.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:48 smithi070 ceph-mon[107385]: 1.11 scrub starts 2024-09-21T11:01:48.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:48 smithi070 ceph-mon[107385]: 1.11 scrub ok 2024-09-21T11:01:48.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:48 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:01:48.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:48 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:01:48.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:48.884 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:48.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:48.985 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:49.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:49.007 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:49.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:49.030 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T11:01:49.109 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:49.131 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:49.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:49.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:49.228 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:01:49.233 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:49.254 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:49.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:49.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:49.356 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:49.377 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:49.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:49.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:49.479 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:49.502 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:49.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:49.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:49.595 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:49 smithi070 ceph-mon[112139]: pgmap v3068: 93 pgs: 5 peering, 25 remapped+peering, 37 active+clean+remapped, 18 active+undersized, 3 active+undersized+degraded, 5 active+clean; 268 KiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 6/75 objects degraded (8.000%); 4/75 objects misplaced (5.333%) 2024-09-21T11:01:49.595 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:49 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-21T11:01:49.595 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:49 smithi070 ceph-mon[112139]: osdmap e1581: 8 total, 6 up, 5 in 2024-09-21T11:01:49.595 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:49 smithi070 ceph-mon[112139]: 1.b scrub starts 2024-09-21T11:01:49.595 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:49 smithi070 ceph-mon[112139]: 1.b scrub ok 2024-09-21T11:01:49.595 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:49 smithi070 ceph-mon[112139]: 1.12 scrub starts 2024-09-21T11:01:49.595 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:49 smithi070 ceph-mon[112139]: 1.12 scrub ok 2024-09-21T11:01:49.596 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:49 smithi070 ceph-mon[112139]: 1.1 scrub starts 2024-09-21T11:01:49.596 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:49 smithi070 ceph-mon[112139]: 1.1 scrub ok 2024-09-21T11:01:49.596 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:49 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T11:01:49.596 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:49 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T11:01:49.596 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:49 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:01:49.596 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:49 smithi070 ceph-mon[107385]: pgmap v3068: 93 pgs: 5 peering, 25 remapped+peering, 37 active+clean+remapped, 18 active+undersized, 3 active+undersized+degraded, 5 active+clean; 268 KiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 6/75 objects degraded (8.000%); 4/75 objects misplaced (5.333%) 2024-09-21T11:01:49.596 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:49 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-21T11:01:49.596 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:49 smithi070 ceph-mon[107385]: osdmap e1581: 8 total, 6 up, 5 in 2024-09-21T11:01:49.596 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:49 smithi070 ceph-mon[107385]: 1.b scrub starts 2024-09-21T11:01:49.596 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:49 smithi070 ceph-mon[107385]: 1.b scrub ok 2024-09-21T11:01:49.596 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:49 smithi070 ceph-mon[107385]: 1.12 scrub starts 2024-09-21T11:01:49.597 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:49 smithi070 ceph-mon[107385]: 1.12 scrub ok 2024-09-21T11:01:49.597 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:49 smithi070 ceph-mon[107385]: 1.1 scrub starts 2024-09-21T11:01:49.597 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:49 smithi070 ceph-mon[107385]: 1.1 scrub ok 2024-09-21T11:01:49.597 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:49 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T11:01:49.597 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:49 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T11:01:49.597 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:49 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:01:49.604 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:49.625 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:49.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:49.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:49.702 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:49 smithi160 ceph-mon[101879]: pgmap v3068: 93 pgs: 5 peering, 25 remapped+peering, 37 active+clean+remapped, 18 active+undersized, 3 active+undersized+degraded, 5 active+clean; 268 KiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 6/75 objects degraded (8.000%); 4/75 objects misplaced (5.333%) 2024-09-21T11:01:49.702 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:49 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-21T11:01:49.703 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:49 smithi160 ceph-mon[101879]: osdmap e1581: 8 total, 6 up, 5 in 2024-09-21T11:01:49.703 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:49 smithi160 ceph-mon[101879]: 1.b scrub starts 2024-09-21T11:01:49.703 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:49 smithi160 ceph-mon[101879]: 1.b scrub ok 2024-09-21T11:01:49.703 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:49 smithi160 ceph-mon[101879]: 1.12 scrub starts 2024-09-21T11:01:49.703 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:49 smithi160 ceph-mon[101879]: 1.12 scrub ok 2024-09-21T11:01:49.703 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:49 smithi160 ceph-mon[101879]: 1.1 scrub starts 2024-09-21T11:01:49.703 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:49 smithi160 ceph-mon[101879]: 1.1 scrub ok 2024-09-21T11:01:49.703 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:49 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T11:01:49.703 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:49 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T11:01:49.703 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:49 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:01:49.727 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:49.748 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:49.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:49.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:49.849 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:49.871 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:49.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:49.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:49.972 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:49.995 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:49.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:49.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:50.009 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-09-21T11:01:50.097 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:50.120 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:50.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:50.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:50.222 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:50.244 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:50.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:50.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:50.345 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:50.367 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:50.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:50.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:50.469 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:50.491 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:50.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:50.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:50 smithi160 ceph-mon[101879]: osdmap e1582: 8 total, 6 up, 5 in 2024-09-21T11:01:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:50 smithi160 ceph-mon[101879]: 1.2 scrub starts 2024-09-21T11:01:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:50 smithi160 ceph-mon[101879]: 1.2 scrub ok 2024-09-21T11:01:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:50 smithi160 ceph-mon[101879]: 1.8 scrub starts 2024-09-21T11:01:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:50 smithi160 ceph-mon[101879]: 1.8 scrub ok 2024-09-21T11:01:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:01:50.593 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:50.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:50.614 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:50.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:50.715 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:50.736 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:50.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:50.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:50.807 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T11:01:50.807 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T11:01:50.837 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:50.859 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:50.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:50.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:50.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:50 smithi070 ceph-mon[107385]: osdmap e1582: 8 total, 6 up, 5 in 2024-09-21T11:01:50.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:50 smithi070 ceph-mon[107385]: 1.2 scrub starts 2024-09-21T11:01:50.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:50 smithi070 ceph-mon[107385]: 1.2 scrub ok 2024-09-21T11:01:50.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:50 smithi070 ceph-mon[107385]: 1.8 scrub starts 2024-09-21T11:01:50.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:50 smithi070 ceph-mon[107385]: 1.8 scrub ok 2024-09-21T11:01:50.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:01:50.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:50 smithi070 ceph-mon[112139]: osdmap e1582: 8 total, 6 up, 5 in 2024-09-21T11:01:50.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:50 smithi070 ceph-mon[112139]: 1.2 scrub starts 2024-09-21T11:01:50.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:50 smithi070 ceph-mon[112139]: 1.2 scrub ok 2024-09-21T11:01:50.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:50 smithi070 ceph-mon[112139]: 1.8 scrub starts 2024-09-21T11:01:50.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:50 smithi070 ceph-mon[112139]: 1.8 scrub ok 2024-09-21T11:01:50.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:01:50.961 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:50.981 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:50.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:50.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:51.010 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:01:51.043 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T11:01:51.083 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:51.107 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:51.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:51.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:51.208 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:51.229 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:51.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:51.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:51.331 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:51.350 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7, 4] out_osds: [2, 5, 0] dead_osds: [5, 6] live_osds: [0, 1, 2, 3, 4, 7] 2024-09-21T11:01:51.350 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T11:01:51.350 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-21T11:01:51.350 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T11:01:51.351 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T11:01:51.351 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-21T11:01:51.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:51.353 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:51.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:51.453 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:51.471 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T11:01:51.478 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:51.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:51.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:51.528 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:51 smithi070 ceph-mon[112139]: pgmap v3071: 109 pgs: 6 creating+peering, 5 peering, 8 unknown, 25 remapped+peering, 25 active+clean+remapped, 19 active+undersized, 3 active+undersized+degraded, 18 active+clean; 250 KiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 6/75 objects degraded (8.000%); 3/75 objects misplaced (4.000%) 2024-09-21T11:01:51.529 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:51 smithi070 ceph-mon[112139]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T11:01:51.529 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:51 smithi070 ceph-mon[112139]: osdmap e1583: 8 total, 6 up, 5 in 2024-09-21T11:01:51.529 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:51 smithi070 ceph-mon[112139]: 1.3 scrub starts 2024-09-21T11:01:51.529 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:51 smithi070 ceph-mon[112139]: 1.3 scrub ok 2024-09-21T11:01:51.529 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:51 smithi070 ceph-mon[107385]: pgmap v3071: 109 pgs: 6 creating+peering, 5 peering, 8 unknown, 25 remapped+peering, 25 active+clean+remapped, 19 active+undersized, 3 active+undersized+degraded, 18 active+clean; 250 KiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 6/75 objects degraded (8.000%); 3/75 objects misplaced (4.000%) 2024-09-21T11:01:51.529 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:51 smithi070 ceph-mon[107385]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T11:01:51.529 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:51 smithi070 ceph-mon[107385]: osdmap e1583: 8 total, 6 up, 5 in 2024-09-21T11:01:51.529 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:51 smithi070 ceph-mon[107385]: 1.3 scrub starts 2024-09-21T11:01:51.529 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:51 smithi070 ceph-mon[107385]: 1.3 scrub ok 2024-09-21T11:01:51.580 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:51.603 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:51.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:51.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:51.704 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:51.727 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:51.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:51.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:51.757 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:01:51.757 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:01:51.766 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:01:51.766 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:01:51.786 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:01:51.786 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:01:51.802 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:01:51.802 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:01:51.818 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:01:51.819 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:01:51.819 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T11:01:51.819 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T11:01:51.828 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:51.835 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:01:51.835 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:01:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:51 smithi160 ceph-mon[101879]: pgmap v3071: 109 pgs: 6 creating+peering, 5 peering, 8 unknown, 25 remapped+peering, 25 active+clean+remapped, 19 active+undersized, 3 active+undersized+degraded, 18 active+clean; 250 KiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 6/75 objects degraded (8.000%); 3/75 objects misplaced (4.000%) 2024-09-21T11:01:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:51 smithi160 ceph-mon[101879]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T11:01:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:51 smithi160 ceph-mon[101879]: osdmap e1583: 8 total, 6 up, 5 in 2024-09-21T11:01:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:51 smithi160 ceph-mon[101879]: 1.3 scrub starts 2024-09-21T11:01:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:51 smithi160 ceph-mon[101879]: 1.3 scrub ok 2024-09-21T11:01:51.851 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:51.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:51.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:51.953 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:51.977 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:51.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:51.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:52.079 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:52.102 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:52.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:52.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:52.206 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:52.230 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:52.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:52.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:52.331 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:52.357 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:52.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:52.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:52.459 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:52.480 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:52.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:52.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:52.582 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:52.607 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:52.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:52.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:52.701 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:52 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2872904024' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:01:52.701 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:52 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2872904024' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:01:52.701 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T11:01:52.702 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T11:01:52.709 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:52.734 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:52.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:52.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:52.742 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:01:52.742 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:01:52.836 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:52 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2872904024' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:01:52.861 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:52.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:52.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:52.962 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:52.984 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:52.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:52.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:53.085 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:53.108 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:53.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:53.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:53.209 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:53.231 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:53.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:53.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:53.332 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:53.353 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:53.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:53.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:53.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:01:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:01:53] "GET /metrics HTTP/1.1" 200 36496 "" "Prometheus/2.51.0" 2024-09-21T11:01:53.455 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:53.481 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:53.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:53.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:53.583 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:53.608 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:53.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:53.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:53.655 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:53 smithi070 ceph-mon[112139]: pgmap v3073: 109 pgs: 6 creating+peering, 40 active+clean+remapped, 23 active+undersized, 3 active+undersized+degraded, 37 active+clean; 402 KiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 6/75 objects degraded (8.000%); 12/75 objects misplaced (16.000%) 2024-09-21T11:01:53.655 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:53 smithi070 ceph-mon[107385]: pgmap v3073: 109 pgs: 6 creating+peering, 40 active+clean+remapped, 23 active+undersized, 3 active+undersized+degraded, 37 active+clean; 402 KiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 6/75 objects degraded (8.000%); 12/75 objects misplaced (16.000%) 2024-09-21T11:01:53.710 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:53.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:53.732 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:53.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:53.833 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:53 smithi160 ceph-mon[101879]: pgmap v3073: 109 pgs: 6 creating+peering, 40 active+clean+remapped, 23 active+undersized, 3 active+undersized+degraded, 37 active+clean; 402 KiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 6/75 objects degraded (8.000%); 12/75 objects misplaced (16.000%) 2024-09-21T11:01:53.854 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:53.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:53.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:53.955 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:53.976 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:53.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:53.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:54.078 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:54.104 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:54.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:54.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:54.206 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:54.231 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:54.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:54.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:54.333 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:54.354 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:54.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:54.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:54.455 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:54.478 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:54.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:54.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:54.579 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:54.601 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:54.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:54.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:54.703 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T11:01:54.703 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:54.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:54 smithi070 ceph-mon[107385]: 1.14 scrub starts 2024-09-21T11:01:54.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:54 smithi070 ceph-mon[107385]: 1.14 scrub ok 2024-09-21T11:01:54.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:54 smithi070 ceph-mon[107385]: 1.d scrub starts 2024-09-21T11:01:54.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:54 smithi070 ceph-mon[107385]: 1.d scrub ok 2024-09-21T11:01:54.725 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:54 smithi070 ceph-mon[112139]: 1.14 scrub starts 2024-09-21T11:01:54.725 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:54 smithi070 ceph-mon[112139]: 1.14 scrub ok 2024-09-21T11:01:54.725 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:54 smithi070 ceph-mon[112139]: 1.d scrub starts 2024-09-21T11:01:54.725 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:54 smithi070 ceph-mon[112139]: 1.d scrub ok 2024-09-21T11:01:54.730 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:54.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:54.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:54.832 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:54 smithi160 ceph-mon[101879]: 1.14 scrub starts 2024-09-21T11:01:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:54 smithi160 ceph-mon[101879]: 1.14 scrub ok 2024-09-21T11:01:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:54 smithi160 ceph-mon[101879]: 1.d scrub starts 2024-09-21T11:01:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:54 smithi160 ceph-mon[101879]: 1.d scrub ok 2024-09-21T11:01:54.855 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:54.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:54.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:54.956 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:54.959 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:01:54.978 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:54.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:54.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:55.080 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:55.101 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:55.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:55.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:55.203 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:55.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:55.227 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:55.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:55.328 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:55.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:55.355 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:55.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:55.455 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:55.478 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:55.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:55.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:55.495 INFO:teuthology.orchestra.run.smithi070.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-09-21T11:01:55.579 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:55.600 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:55.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:55.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:55.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:55 smithi070 ceph-mon[107385]: pgmap v3074: 109 pgs: 6 creating+peering, 40 active+clean+remapped, 23 active+undersized, 3 active+undersized+degraded, 37 active+clean; 402 KiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 6/75 objects degraded (8.000%); 12/75 objects misplaced (16.000%) 2024-09-21T11:01:55.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:55 smithi070 ceph-mon[107385]: 1.13 scrub starts 2024-09-21T11:01:55.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:55 smithi070 ceph-mon[107385]: 1.c scrub starts 2024-09-21T11:01:55.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:55 smithi070 ceph-mon[107385]: 1.13 scrub ok 2024-09-21T11:01:55.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:55 smithi070 ceph-mon[107385]: 1.c scrub ok 2024-09-21T11:01:55.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:55 smithi070 ceph-mon[107385]: 1.5 scrub starts 2024-09-21T11:01:55.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:55 smithi070 ceph-mon[107385]: 1.5 scrub ok 2024-09-21T11:01:55.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:55 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/805562152' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T11:01:55.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:55 smithi070 ceph-mon[112139]: pgmap v3074: 109 pgs: 6 creating+peering, 40 active+clean+remapped, 23 active+undersized, 3 active+undersized+degraded, 37 active+clean; 402 KiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 6/75 objects degraded (8.000%); 12/75 objects misplaced (16.000%) 2024-09-21T11:01:55.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:55 smithi070 ceph-mon[112139]: 1.13 scrub starts 2024-09-21T11:01:55.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:55 smithi070 ceph-mon[112139]: 1.c scrub starts 2024-09-21T11:01:55.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:55 smithi070 ceph-mon[112139]: 1.13 scrub ok 2024-09-21T11:01:55.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:55 smithi070 ceph-mon[112139]: 1.c scrub ok 2024-09-21T11:01:55.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:55 smithi070 ceph-mon[112139]: 1.5 scrub starts 2024-09-21T11:01:55.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:55 smithi070 ceph-mon[112139]: 1.5 scrub ok 2024-09-21T11:01:55.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:55 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/805562152' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T11:01:55.703 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:55.729 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:55.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:55.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:55.831 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:55.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:55 smithi160 ceph-mon[101879]: pgmap v3074: 109 pgs: 6 creating+peering, 40 active+clean+remapped, 23 active+undersized, 3 active+undersized+degraded, 37 active+clean; 402 KiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 6/75 objects degraded (8.000%); 12/75 objects misplaced (16.000%) 2024-09-21T11:01:55.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:55 smithi160 ceph-mon[101879]: 1.13 scrub starts 2024-09-21T11:01:55.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:55 smithi160 ceph-mon[101879]: 1.c scrub starts 2024-09-21T11:01:55.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:55 smithi160 ceph-mon[101879]: 1.13 scrub ok 2024-09-21T11:01:55.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:55 smithi160 ceph-mon[101879]: 1.c scrub ok 2024-09-21T11:01:55.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:55 smithi160 ceph-mon[101879]: 1.5 scrub starts 2024-09-21T11:01:55.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:55 smithi160 ceph-mon[101879]: 1.5 scrub ok 2024-09-21T11:01:55.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:55 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/805562152' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T11:01:55.853 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:55.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:55.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:55.955 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:55.977 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:55.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:55.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:56.078 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:56.100 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:56.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:56.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:56.201 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:56.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:56.226 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:56.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:56.327 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:56.349 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:56.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:56.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:56.450 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:56.472 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:56.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:56.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:56.574 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:56.595 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:56.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:56.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:56.697 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:56.717 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:56.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:56.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:56.820 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:56.841 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:56 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/805562152' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T11:01:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:56 smithi160 ceph-mon[101879]: osdmap e1584: 8 total, 6 up, 5 in 2024-09-21T11:01:56.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:56.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:56.852 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:01:56.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:56 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/805562152' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T11:01:56.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:56 smithi070 ceph-mon[112139]: osdmap e1584: 8 total, 6 up, 5 in 2024-09-21T11:01:56.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:56 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/805562152' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T11:01:56.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:56 smithi070 ceph-mon[107385]: osdmap e1584: 8 total, 6 up, 5 in 2024-09-21T11:01:56.943 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:56.963 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:56.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:56.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:57.065 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:57.086 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:57.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:57.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:57.188 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:57.209 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:57.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:57.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:57.310 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:57.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:57.332 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:57.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:57.433 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:57.455 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:57.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:57.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:57.557 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:57.578 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:57.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:57.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:57.679 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:57.702 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:57.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:57.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:57.804 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:57.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:57.826 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:57.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:57.837 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:57 smithi070 ceph-mon[112139]: pgmap v3076: 109 pgs: 40 active+clean+remapped, 28 active+undersized, 3 active+undersized+degraded, 38 active+clean; 617 KiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 6/78 objects degraded (7.692%); 13/78 objects misplaced (16.667%) 2024-09-21T11:01:57.837 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:57 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 6/78 objects degraded (7.692%), 3 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-21T11:01:57.837 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:57 smithi070 ceph-mon[112139]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T11:01:57.837 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:57 smithi070 ceph-mon[107385]: pgmap v3076: 109 pgs: 40 active+clean+remapped, 28 active+undersized, 3 active+undersized+degraded, 38 active+clean; 617 KiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 6/78 objects degraded (7.692%); 13/78 objects misplaced (16.667%) 2024-09-21T11:01:57.837 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:57 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 6/78 objects degraded (7.692%), 3 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-21T11:01:57.838 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:57 smithi070 ceph-mon[107385]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T11:01:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:57 smithi160 ceph-mon[101879]: pgmap v3076: 109 pgs: 40 active+clean+remapped, 28 active+undersized, 3 active+undersized+degraded, 38 active+clean; 617 KiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 6/78 objects degraded (7.692%); 13/78 objects misplaced (16.667%) 2024-09-21T11:01:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:57 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 6/78 objects degraded (7.692%), 3 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-21T11:01:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:57 smithi160 ceph-mon[101879]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T11:01:57.927 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:57.949 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:57.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:57.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:58.051 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:58.074 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:58.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:58.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:58.175 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:58.197 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:58.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:58.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:58.273 DEBUG:teuthology.orchestra.run.smithi070:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-09-21T11:01:58.275 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T11:01:58.275 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T11:01:58.299 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:58.319 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:58.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:58.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:58.326 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:hints = 1 2024-09-21T11:01:58.326 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T11:01:58.326 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Object prefix: benchmark_data_smithi070_396992 2024-09-21T11:01:58.326 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T11:01:58.326 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 0 0 0 0 0 0 - 0 2024-09-21T11:01:58.421 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:58.457 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:58.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:58.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:58.559 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:58.594 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:58.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:58.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:58.696 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:01:58.730 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:58.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:58.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:01:58.832 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:58 smithi160 ceph-mon[101879]: 1.e scrub starts 2024-09-21T11:01:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:58 smithi160 ceph-mon[101879]: 1.e scrub ok 2024-09-21T11:01:58.863 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:58.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:58.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:58.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:58 smithi070 ceph-mon[112139]: 1.e scrub starts 2024-09-21T11:01:58.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:58 smithi070 ceph-mon[112139]: 1.e scrub ok 2024-09-21T11:01:58.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:58 smithi070 ceph-mon[107385]: 1.e scrub starts 2024-09-21T11:01:58.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:58 smithi070 ceph-mon[107385]: 1.e scrub ok 2024-09-21T11:01:58.965 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:58.998 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:58.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:59.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:59.100 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:01:59.135 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:59.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:59.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:01:59.237 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:59.269 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:59.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:59.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:59.326 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 1 16 4911 4895 305.919 305.938 0.00341153 0.00325868 2024-09-21T11:01:59.371 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:01:59.405 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:59.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:59.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:01:59.507 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:59.540 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:59.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:59.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:01:59.641 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:01:59.680 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:01:59.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:59.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:01:59.782 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:01:59.798 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:01:59 smithi070 ceph-mon[107385]: pgmap v3077: 109 pgs: 40 active+clean+remapped, 28 active+undersized, 3 active+undersized+degraded, 38 active+clean; 617 KiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 6/78 objects degraded (7.692%); 13/78 objects misplaced (16.667%) 2024-09-21T11:01:59.799 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:01:59 smithi070 ceph-mon[112139]: pgmap v3077: 109 pgs: 40 active+clean+remapped, 28 active+undersized, 3 active+undersized+degraded, 38 active+clean; 617 KiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 6/78 objects degraded (7.692%); 13/78 objects misplaced (16.667%) 2024-09-21T11:01:59.815 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:59.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:59.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:01:59.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:01:59 smithi160 ceph-mon[101879]: pgmap v3077: 109 pgs: 40 active+clean+remapped, 28 active+undersized, 3 active+undersized+degraded, 38 active+clean; 617 KiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 6/78 objects degraded (7.692%); 13/78 objects misplaced (16.667%) 2024-09-21T11:01:59.916 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:01:59.948 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:01:59.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:01:59.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:00.049 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:00.082 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:00.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:00.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:00.184 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:00.218 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:00.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:00.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:00.319 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:00.326 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 2 16 9393 9377 293.005 280.125 0.0036616 0.00340589 2024-09-21T11:02:00.356 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:00.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:00.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:00.458 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:00.493 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:00.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:00.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:00.594 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:00.625 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T11:02:00.625 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T11:02:00.626 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T11:02:00.626 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T11:02:00.631 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:00.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:00.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:00.733 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:00.769 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:00.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:00.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:00.801 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:00 smithi070 ceph-mon[107385]: 1.4 scrub starts 2024-09-21T11:02:00.801 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:00 smithi070 ceph-mon[107385]: 1.4 scrub ok 2024-09-21T11:02:00.801 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:00 smithi070 ceph-mon[112139]: 1.4 scrub starts 2024-09-21T11:02:00.801 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:00 smithi070 ceph-mon[112139]: 1.4 scrub ok 2024-09-21T11:02:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:00 smithi160 ceph-mon[101879]: 1.4 scrub starts 2024-09-21T11:02:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:00 smithi160 ceph-mon[101879]: 1.4 scrub ok 2024-09-21T11:02:00.870 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:00.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:00.906 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:00.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:01.007 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:01.039 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:01.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:01.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:01.100 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T11:02:01.100 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "" 2024-09-21T11:02:01.100 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T11:02:01.140 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:01.170 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:02:01.170 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:02:01.170 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:02:01.170 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:02:01.171 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:02:01.171 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:02:01.171 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:02:01.171 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:02:01.171 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:02:01.171 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:02:01.171 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:02:01.171 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T11:02:01.171 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T11:02:01.171 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:02:01.173 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:01.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:01.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:01.275 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:01.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:01.311 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:01.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:01.326 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 3 15 13322 13307 277.204 245.625 0.0033765 0.00360264 2024-09-21T11:02:01.411 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:01.445 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:01.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:01.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:01.544 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T11:02:01.547 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:01.583 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:01.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:01.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:01.685 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:01.716 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:01.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:01.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:01.818 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:01 smithi160 ceph-mon[101879]: pgmap v3078: 109 pgs: 40 active+clean+remapped, 22 active+undersized, 9 active+undersized+degraded, 38 active+clean; 168 MiB data, 8.1 GiB used, 439 GiB / 447 GiB avail; 17 MiB/s wr, 269 op/s; 1991/8127 objects degraded (24.499%); 13/8127 objects misplaced (0.160%) 2024-09-21T11:02:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:01 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2457857147' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:02:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:01 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/8087845' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:02:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:01 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:02:01.849 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:01.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:01.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:01.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:01 smithi070 ceph-mon[107385]: pgmap v3078: 109 pgs: 40 active+clean+remapped, 22 active+undersized, 9 active+undersized+degraded, 38 active+clean; 168 MiB data, 8.1 GiB used, 439 GiB / 447 GiB avail; 17 MiB/s wr, 269 op/s; 1991/8127 objects degraded (24.499%); 13/8127 objects misplaced (0.160%) 2024-09-21T11:02:01.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:01 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2457857147' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:02:01.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:01 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/8087845' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:02:01.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:01 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:02:01.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:01 smithi070 ceph-mon[112139]: pgmap v3078: 109 pgs: 40 active+clean+remapped, 22 active+undersized, 9 active+undersized+degraded, 38 active+clean; 168 MiB data, 8.1 GiB used, 439 GiB / 447 GiB avail; 17 MiB/s wr, 269 op/s; 1991/8127 objects degraded (24.499%); 13/8127 objects misplaced (0.160%) 2024-09-21T11:02:01.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:01 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2457857147' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:02:01.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:01 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/8087845' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:02:01.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:01 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:02:01.951 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:01.968 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T11:02:01.985 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:01.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:01.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:02.086 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:02.120 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:02.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:02.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:02.221 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:02.253 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:02.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:02.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:02.326 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 4 16 17189 17173 268.304 241.625 0.00304343 0.00372154 2024-09-21T11:02:02.356 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:02.388 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:02.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:02.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:02.490 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:02.522 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:02.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:02.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:02.624 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:02.658 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:02.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:02.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:02.760 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:02.792 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:02.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:02.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:02 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T11:02:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:02 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T11:02:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:02 smithi160 ceph-mon[101879]: osdmap e1585: 8 total, 6 up, 5 in 2024-09-21T11:02:02.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:02 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T11:02:02.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:02 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T11:02:02.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:02 smithi070 ceph-mon[107385]: osdmap e1585: 8 total, 6 up, 5 in 2024-09-21T11:02:02.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:02 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T11:02:02.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:02 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T11:02:02.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:02 smithi070 ceph-mon[112139]: osdmap e1585: 8 total, 6 up, 5 in 2024-09-21T11:02:02.894 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:02.928 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:02.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:02.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:03.030 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:03.063 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:03.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:03.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:03.165 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:03.199 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:03.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:03.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:03.268 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:02:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:02:03] "GET /metrics HTTP/1.1" 200 36496 "" "Prometheus/2.51.0" 2024-09-21T11:02:03.301 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:03.327 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 5 16 20785 20769 259.59 224.75 0.00344178 0.00384665 2024-09-21T11:02:03.335 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:03.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:03.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:03.349 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:02:03.437 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:03.470 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:03.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:03.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:03.573 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:03.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:03.610 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:03.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:03.711 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:03.745 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:03.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:03.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:03.822 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:03 smithi070 ceph-mon[107385]: pgmap v3080: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 493 MiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 61 MiB/s wr, 977 op/s; 4719/23700 objects degraded (19.911%); 13/23700 objects misplaced (0.055%) 2024-09-21T11:02:03.822 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:03 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 4719/23700 objects degraded (19.911%), 13 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-21T11:02:03.822 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:03 smithi070 ceph-mon[107385]: 1.6 scrub starts 2024-09-21T11:02:03.822 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:03 smithi070 ceph-mon[107385]: 1.6 scrub ok 2024-09-21T11:02:03.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:03 smithi070 ceph-mon[112139]: pgmap v3080: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 493 MiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 61 MiB/s wr, 977 op/s; 4719/23700 objects degraded (19.911%); 13/23700 objects misplaced (0.055%) 2024-09-21T11:02:03.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:03 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 4719/23700 objects degraded (19.911%), 13 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-21T11:02:03.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:03 smithi070 ceph-mon[112139]: 1.6 scrub starts 2024-09-21T11:02:03.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:03 smithi070 ceph-mon[112139]: 1.6 scrub ok 2024-09-21T11:02:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:03 smithi160 ceph-mon[101879]: pgmap v3080: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 493 MiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 61 MiB/s wr, 977 op/s; 4719/23700 objects degraded (19.911%); 13/23700 objects misplaced (0.055%) 2024-09-21T11:02:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:03 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 4719/23700 objects degraded (19.911%), 13 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-21T11:02:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:03 smithi160 ceph-mon[101879]: 1.6 scrub starts 2024-09-21T11:02:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:03 smithi160 ceph-mon[101879]: 1.6 scrub ok 2024-09-21T11:02:03.847 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:03.858 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T11:02:03.858 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T11:02:03.880 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:03.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:03.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:03.982 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:04.012 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:04.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:04.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:04.114 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:04.145 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:04.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:04.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:04.246 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:04.280 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:04.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:04.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:04.326 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 6 16 24006 23990 249.874 201.312 0.00609149 0.00399723 2024-09-21T11:02:04.382 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:04.414 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:04.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:04.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:04.516 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:04.553 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:04.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:04.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:04.654 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:04.686 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:04.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:04.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:04.787 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:04.816 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:04.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:04.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:04.917 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:04.952 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:04.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:04.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:05.054 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:05.088 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:05.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:05.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:05.190 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:05.223 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:05.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:05.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:05.305 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T11:02:05.325 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:05.327 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 7 16 27058 27042 241.409 190.75 0.00553769 0.00413715 2024-09-21T11:02:05.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:05.367 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:05.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:05.468 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:05.505 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:05.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:05.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:05.607 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:05.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:05 smithi070 ceph-mon[112139]: pgmap v3081: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 493 MiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 56 MiB/s wr, 894 op/s; 4719/23700 objects degraded (19.911%); 13/23700 objects misplaced (0.055%) 2024-09-21T11:02:05.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:02:05.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:02:05.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:05 smithi070 ceph-mon[107385]: pgmap v3081: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 493 MiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 56 MiB/s wr, 894 op/s; 4719/23700 objects degraded (19.911%); 13/23700 objects misplaced (0.055%) 2024-09-21T11:02:05.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:02:05.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:02:05.641 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:05.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:05.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:05.743 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:05.774 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:05.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:05.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:05 smithi160 ceph-mon[101879]: pgmap v3081: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 493 MiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 56 MiB/s wr, 894 op/s; 4719/23700 objects degraded (19.911%); 13/23700 objects misplaced (0.055%) 2024-09-21T11:02:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:02:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:02:05.859 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T11:02:05.876 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:05.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:05.938 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:05.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:06.038 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:06.059 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:06.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:06.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:06.161 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:06.193 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:06.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:06.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:06.295 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:06.326 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:06.327 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 8 16 29680 29664 231.716 163.875 0.00776419 0.00431149 2024-09-21T11:02:06.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:06.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:06.428 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:06.461 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:06.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:06.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:06.562 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:06.595 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:06.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:06.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:06.696 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:06.727 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:06.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:06.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:06.783 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:02:06.829 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:06.859 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:06.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:06.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:06.961 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:07.000 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:07.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:07.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:07.101 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:07.141 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:07.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:07.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:07.243 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:07.278 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:07.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:07.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:07.327 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 9 16 32524 32508 225.718 177.75 0.00672682 0.00442511 2024-09-21T11:02:07.379 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:07.415 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:07.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:07.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:07.516 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:07.553 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:07.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:07.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:07.621 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7, 4] out_osds: [2, 5, 0] dead_osds: [5, 6] live_osds: [0, 1, 2, 3, 4, 7] 2024-09-21T11:02:07.621 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T11:02:07.621 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-09-21T11:02:07.621 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T11:02:07.621 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T11:02:07.621 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-09-21T11:02:07.655 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:07.667 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:02:07.693 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:07.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:07.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:07.795 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:07.799 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:07 smithi070 ceph-mon[107385]: pgmap v3082: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 1.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 163 MiB/s wr, 2.60k op/s; 16180/78111 objects degraded (20.714%); 13/78111 objects misplaced (0.017%) 2024-09-21T11:02:07.800 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:07 smithi070 ceph-mon[112139]: pgmap v3082: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 1.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 163 MiB/s wr, 2.60k op/s; 16180/78111 objects degraded (20.714%); 13/78111 objects misplaced (0.017%) 2024-09-21T11:02:07.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:07.828 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:07.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:07.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:07 smithi160 ceph-mon[101879]: pgmap v3082: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 1.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 163 MiB/s wr, 2.60k op/s; 16180/78111 objects degraded (20.714%); 13/78111 objects misplaced (0.017%) 2024-09-21T11:02:07.928 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:07.971 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:07.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:07.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:07.986 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:02:08.074 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:08.101 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:08.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:08.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:08.203 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:08.242 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:08.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:08.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:08.327 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 10 16 35152 35136 219.57 164.25 0.00765771 0.00455019 2024-09-21T11:02:08.344 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:08.377 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:08.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:08.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:08.479 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:08.512 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:08.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:08.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:08.614 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:08.647 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:08.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:08.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:08.749 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:08.763 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:08 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 16180/78111 objects degraded (20.714%), 13 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-21T11:02:08.763 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:08 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 16180/78111 objects degraded (20.714%), 13 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-21T11:02:08.782 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:08.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:08.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:08 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 16180/78111 objects degraded (20.714%), 13 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-21T11:02:08.884 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:08.910 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:08.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:08.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:08.913 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T11:02:08.913 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T11:02:09.012 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:09.047 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:09.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:09.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:09.149 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:09.171 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:09.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:09.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:09.273 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:09.305 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:09.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:09.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:09.327 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 11 16 37465 37449 212.751 144.562 0.00269846 0.00469594 2024-09-21T11:02:09.407 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:09.428 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:09.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:09.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:09.529 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:09.562 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:09.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:09.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:09.664 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:09.699 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:09.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:09.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:09.801 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:09.838 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:09 smithi070 ceph-mon[107385]: pgmap v3083: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 1.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 162 MiB/s wr, 2.60k op/s; 16180/78111 objects degraded (20.714%); 13/78111 objects misplaced (0.017%) 2024-09-21T11:02:09.839 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:09 smithi070 ceph-mon[112139]: pgmap v3083: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 1.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 162 MiB/s wr, 2.60k op/s; 16180/78111 objects degraded (20.714%); 13/78111 objects misplaced (0.017%) 2024-09-21T11:02:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:09 smithi160 ceph-mon[101879]: pgmap v3083: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 1.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 162 MiB/s wr, 2.60k op/s; 16180/78111 objects degraded (20.714%); 13/78111 objects misplaced (0.017%) 2024-09-21T11:02:09.841 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:09.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:09.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:09.943 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:09.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:09.978 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:09.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:10.079 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:10.110 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:10.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:10.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:10.184 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T11:02:10.212 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:10.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:10.249 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:10.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:10.327 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 12 16 39703 39687 206.677 139.875 0.00557118 0.00483415 2024-09-21T11:02:10.350 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:10.382 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:10.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:10.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:10.484 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:10.512 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:10.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:10.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:10.614 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:10.651 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:10.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:10.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:10.735 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:02:10.736 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:02:10.753 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:10.769 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:02:10.769 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:02:10.786 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:10.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:10.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:10.798 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:02:10.799 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:02:10.822 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:02:10.822 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:02:10.888 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:10.902 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:02:10.902 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:02:10.903 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T11:02:10.903 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T11:02:10.920 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:10.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:10.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:10.934 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:02:10.934 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:02:11.023 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:11.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:11.053 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:11.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:11.154 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:11.187 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:11.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:11.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:11.289 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:11.321 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:11.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:11.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:11.327 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 13 16 41651 41635 200.144 121.75 0.003898 0.00499117 2024-09-21T11:02:11.423 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:11.458 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:11.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:11.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:11.560 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:11.593 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:11.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:11.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:11.694 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:11.727 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:11.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:11.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:11.829 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:11 smithi160 ceph-mon[101879]: pgmap v3084: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 2.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 187 MiB/s wr, 2.99k op/s; 21153/98013 objects degraded (21.582%); 13/98013 objects misplaced (0.013%) 2024-09-21T11:02:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:11 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/923309373' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:02:11.850 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:11.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:11.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:11.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:11 smithi070 ceph-mon[107385]: pgmap v3084: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 2.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 187 MiB/s wr, 2.99k op/s; 21153/98013 objects degraded (21.582%); 13/98013 objects misplaced (0.013%) 2024-09-21T11:02:11.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:11 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/923309373' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:02:11.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:11 smithi070 ceph-mon[112139]: pgmap v3084: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 2.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 187 MiB/s wr, 2.99k op/s; 21153/98013 objects degraded (21.582%); 13/98013 objects misplaced (0.013%) 2024-09-21T11:02:11.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:11 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/923309373' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:02:11.952 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:11.986 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:11.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:11.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:12.088 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:12.109 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:12.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:12.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:12.210 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:12.240 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:12.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:12.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:12.328 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 14 16 43379 43363 193.562 108 0.00648512 0.00516148 2024-09-21T11:02:12.342 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:12.373 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:12.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:12.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:12.475 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:12.497 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:12.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:12.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:12.599 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:12.631 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:12.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:12.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:12.733 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:12.756 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:12.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:12.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:12.857 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:12.887 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:12.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:12.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:12.989 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:13.011 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:13.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:13.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:13.112 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:13.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:13.144 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:13.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:13.245 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:13.266 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:13.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:13.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:13.328 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 15 16 45294 45278 188.636 119.688 0.0096615 0.00529682 2024-09-21T11:02:13.368 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:13.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:02:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:02:13] "GET /metrics HTTP/1.1" 200 36546 "" "Prometheus/2.51.0" 2024-09-21T11:02:13.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:13.402 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:13.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:13.503 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:13.536 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:13.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:13.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:13.638 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:13.667 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:13.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:13.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:13.759 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:13 smithi160 ceph-mon[101879]: pgmap v3085: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 2.4 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 181 MiB/s wr, 2.89k op/s; 24275/116925 objects degraded (20.761%); 13/116925 objects misplaced (0.011%) 2024-09-21T11:02:13.759 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:13 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 24275/116925 objects degraded (20.761%), 13 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-21T11:02:13.769 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:13.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:13.818 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:13.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:13.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:13 smithi070 ceph-mon[107385]: pgmap v3085: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 2.4 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 181 MiB/s wr, 2.89k op/s; 24275/116925 objects degraded (20.761%); 13/116925 objects misplaced (0.011%) 2024-09-21T11:02:13.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:13 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 24275/116925 objects degraded (20.761%), 13 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-21T11:02:13.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:13 smithi070 ceph-mon[112139]: pgmap v3085: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 2.4 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 181 MiB/s wr, 2.89k op/s; 24275/116925 objects degraded (20.761%); 13/116925 objects misplaced (0.011%) 2024-09-21T11:02:13.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:13 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 24275/116925 objects degraded (20.761%), 13 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-21T11:02:13.919 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:13.962 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:13.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:13.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:13.964 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T11:02:13.965 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T11:02:14.065 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:14.102 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:14.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:14.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:14.178 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:02:14.204 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:14.230 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:14.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:14.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:14.328 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 16 16 47308 47292 184.713 125.875 0.00602335 0.0054098 2024-09-21T11:02:14.331 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:14.359 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:14.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:14.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:14.461 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:14.467 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T11:02:14.467 INFO:teuthology.orchestra.run.smithi160.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T11:02:14.467 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T11:02:14.490 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:14.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:14.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:14.592 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:14.624 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:14.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:14.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:14.726 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:14.764 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:14.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:14.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:14.866 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:14.897 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:14.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:14.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:14.999 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:15.030 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:15.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:15.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:15.131 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:15.163 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:15.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:15.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:15.265 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:15.297 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:15.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:15.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:15.328 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 17 16 49330 49314 181.281 126.375 0.00511854 0.00551202 2024-09-21T11:02:15.398 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:15.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:15.420 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:15.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:15.521 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:15.543 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:15.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:15.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:15.645 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:15.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:15.681 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:15.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:15.781 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:15.819 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:15.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:15.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:15 smithi160 ceph-mon[101879]: pgmap v3086: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 2.4 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 162 MiB/s wr, 2.59k op/s; 24275/116925 objects degraded (20.761%); 13/116925 objects misplaced (0.011%) 2024-09-21T11:02:15.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:15 smithi070 ceph-mon[107385]: pgmap v3086: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 2.4 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 162 MiB/s wr, 2.59k op/s; 24275/116925 objects degraded (20.761%); 13/116925 objects misplaced (0.011%) 2024-09-21T11:02:15.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:15 smithi070 ceph-mon[112139]: pgmap v3086: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 2.4 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 162 MiB/s wr, 2.59k op/s; 24275/116925 objects degraded (20.761%); 13/116925 objects misplaced (0.011%) 2024-09-21T11:02:15.920 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:15.954 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:15.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:15.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:15.965 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T11:02:16.055 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:16.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:16.091 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:16.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:16.192 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:16.223 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:16.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:16.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:16.325 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:16.330 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 18 16 51310 51294 178.071 123.75 0.00656478 0.00561125 2024-09-21T11:02:16.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:16.380 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:16.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:16.481 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:16.513 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:16.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:16.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:16.615 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:16.648 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:16.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:16.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:16.749 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:16.781 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:16.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:16.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:16.883 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:16.914 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:16.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:16.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:17.015 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:17.047 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:17.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:17.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:17.148 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:17.179 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:17.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:17.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:17.281 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:17.318 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:17.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:17.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:17.329 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 19 16 53322 53306 175.317 125.75 0.00697305 0.00569949 2024-09-21T11:02:17.420 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:17.453 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:17.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:17.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:17.554 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:17.579 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:17.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:17.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:17.681 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:17.715 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:17.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:17.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:17.816 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:17.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:17 smithi160 ceph-mon[101879]: pgmap v3087: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 3.0 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 212 MiB/s wr, 3.40k op/s; 30310/146028 objects degraded (20.756%); 13/146028 objects misplaced (0.009%) 2024-09-21T11:02:17.842 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:17.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:17.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:17.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:17 smithi070 ceph-mon[107385]: pgmap v3087: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 3.0 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 212 MiB/s wr, 3.40k op/s; 30310/146028 objects degraded (20.756%); 13/146028 objects misplaced (0.009%) 2024-09-21T11:02:17.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:17 smithi070 ceph-mon[112139]: pgmap v3087: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 3.0 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 212 MiB/s wr, 3.40k op/s; 30310/146028 objects degraded (20.756%); 13/146028 objects misplaced (0.009%) 2024-09-21T11:02:17.944 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:17.980 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:17.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:17.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:18.082 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:18.114 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:18.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:18.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:18.216 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:18.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:18.247 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:18.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:18.330 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:2024-09-21T11:02:18.328781+0000 min lat: 0.00100109 max lat: 0.137376 avg lat: 0.00578622 2024-09-21T11:02:18.330 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T11:02:18.330 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 20 16 55290 55274 172.701 123 0.0122649 0.00578622 2024-09-21T11:02:18.348 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:18.382 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:18.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:18.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:18.484 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:18.507 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:18.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:18.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:18.608 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:18.643 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:18.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:18.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:18.745 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:18.781 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:18.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:18.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:18 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/397995994' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T11:02:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:18 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 30310/146028 objects degraded (20.756%), 13 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-21T11:02:18.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:18 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/397995994' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T11:02:18.899 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:18 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 30310/146028 objects degraded (20.756%), 13 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-21T11:02:18.900 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:18 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/397995994' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T11:02:18.900 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:18 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 30310/146028 objects degraded (20.756%), 13 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-21T11:02:18.900 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:18.931 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:18.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:18.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:19.033 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:19.067 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:19.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:19.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:19.169 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:19.200 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:19.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:19.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:19.301 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:19.322 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:19.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:19.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:19.329 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 21 16 57248 57232 170.304 122.375 0.00351834 0.0058628 2024-09-21T11:02:19.424 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:19.457 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:19.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:19.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:19.559 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:19.594 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:19.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:19.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:19.696 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:19.725 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:19.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:19.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:19.826 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:19 smithi160 ceph-mon[101879]: pgmap v3088: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 3.0 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 118 MiB/s wr, 1.89k op/s; 30310/146028 objects degraded (20.756%); 13/146028 objects misplaced (0.009%) 2024-09-21T11:02:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:19 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:02:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:19 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/397995994' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T11:02:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:19 smithi160 ceph-mon[101879]: osdmap e1586: 8 total, 6 up, 5 in 2024-09-21T11:02:19.859 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:19.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:19.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:19.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:19 smithi070 ceph-mon[107385]: pgmap v3088: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 3.0 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 118 MiB/s wr, 1.89k op/s; 30310/146028 objects degraded (20.756%); 13/146028 objects misplaced (0.009%) 2024-09-21T11:02:19.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:19 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:02:19.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:19 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/397995994' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T11:02:19.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:19 smithi070 ceph-mon[107385]: osdmap e1586: 8 total, 6 up, 5 in 2024-09-21T11:02:19.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:19 smithi070 ceph-mon[112139]: pgmap v3088: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 3.0 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 118 MiB/s wr, 1.89k op/s; 30310/146028 objects degraded (20.756%); 13/146028 objects misplaced (0.009%) 2024-09-21T11:02:19.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:19 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:02:19.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:19 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/397995994' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T11:02:19.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:19 smithi070 ceph-mon[112139]: osdmap e1586: 8 total, 6 up, 5 in 2024-09-21T11:02:19.961 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:19.993 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:19.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:19.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:20.094 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:20.116 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T11:02:20.127 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7, 4] out_osds: [2, 5, 0] dead_osds: [5, 6] live_osds: [0, 1, 2, 3, 4, 7] 2024-09-21T11:02:20.128 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T11:02:20.128 INFO:tasks.thrashosds.thrasher:Reviving osd 6 2024-09-21T11:02:20.128 INFO:tasks.cephadm.osd.6:Restarting osd.6 (starting--it wasn't running)... 2024-09-21T11:02:20.128 DEBUG:teuthology.orchestra.run.smithi160:osd.6> sudo journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.6.service 2024-09-21T11:02:20.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:20.130 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:20.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:20.173 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl start ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.6 2024-09-21T11:02:20.232 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:20.301 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:20.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:20.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:20.330 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 22 16 59332 59316 168.483 130.25 0.00481365 0.00592999 2024-09-21T11:02:20.403 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:20.432 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:20.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:20.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:20.528 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:20 smithi160 systemd[1]: Starting Ceph osd.6 for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T11:02:20.534 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:20.563 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:20.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:20.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:20.665 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:20.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:20.704 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:20.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:20.805 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:20.837 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:20.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:20.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:20.841 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:20 smithi160 podman[232580]: 2024-09-21 11:02:20.431933997 +0000 UTC m=+0.029618957 image pull 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6 2024-09-21T11:02:20.841 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:20 smithi160 podman[232580]: 2024-09-21 11:02:20.580939749 +0000 UTC m=+0.178624705 container create 00c98c3563b715b7a7d0e37639628b3889ba4c93ea87aded762164b2863d61b2 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, CEPH_POINT_RELEASE=, RELEASE=reef-6a1bdf8, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-21T11:02:20.942 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:20.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:20.985 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:20.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:21.086 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:21.116 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:21.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:21.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:21.218 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:21.253 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:21.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:21.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:21.330 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 23 16 61062 61046 165.859 108.125 0.00213693 0.00601693 2024-09-21T11:02:21.341 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:20 smithi160 podman[232580]: 2024-09-21 11:02:20.89254525 +0000 UTC m=+0.490230214 container init 00c98c3563b715b7a7d0e37639628b3889ba4c93ea87aded762164b2863d61b2 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6-activate, GIT_BRANCH=HEAD, RELEASE=reef-6a1bdf8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux ) 2024-09-21T11:02:21.341 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:20 smithi160 podman[232580]: 2024-09-21 11:02:20.901003116 +0000 UTC m=+0.498688071 container start 00c98c3563b715b7a7d0e37639628b3889ba4c93ea87aded762164b2863d61b2 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6-activate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=reef-6a1bdf8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS) 2024-09-21T11:02:21.341 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:20 smithi160 podman[232580]: 2024-09-21 11:02:20.992144681 +0000 UTC m=+0.589829635 container attach 00c98c3563b715b7a7d0e37639628b3889ba4c93ea87aded762164b2863d61b2 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6-activate, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, RELEASE=reef-6a1bdf8, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-21T11:02:21.355 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:21.384 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:21.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:21.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:21.486 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:21.520 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:21.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:21.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:21.622 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:21.663 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:21.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:21.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:21.745 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:21 smithi070 ceph-mon[107385]: pgmap v3090: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 3.3 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 132 MiB/s wr, 2.10k op/s; 34111/161181 objects degraded (21.163%); 13/161181 objects misplaced (0.008%) 2024-09-21T11:02:21.745 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:02:21.745 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:02:21.745 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:21 smithi070 ceph-mon[112139]: pgmap v3090: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 3.3 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 132 MiB/s wr, 2.10k op/s; 34111/161181 objects degraded (21.163%); 13/161181 objects misplaced (0.008%) 2024-09-21T11:02:21.745 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:02:21.745 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:02:21.765 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:21.796 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:21.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:21.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:21.831 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:21 smithi160 ceph-mon[101879]: pgmap v3090: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 3.3 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 132 MiB/s wr, 2.10k op/s; 34111/161181 objects degraded (21.163%); 13/161181 objects misplaced (0.008%) 2024-09-21T11:02:21.831 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:02:21.832 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:02:21.832 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:21 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6-activate[232610]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-21T11:02:21.832 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:21 smithi160 bash[232580]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-21T11:02:21.832 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:21 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6-activate[232610]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-21T11:02:21.832 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:21 smithi160 bash[232580]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-21T11:02:21.832 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:21 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6-activate[232610]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-21T11:02:21.832 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:21 smithi160 bash[232580]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-21T11:02:21.833 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:21 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6-activate[232610]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-21T11:02:21.833 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:21 smithi160 bash[232580]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-21T11:02:21.833 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:21 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6-activate[232610]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-21T11:02:21.833 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:21 smithi160 bash[232580]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-21T11:02:21.833 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:21 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6-activate[232610]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-21T11:02:21.833 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:21 smithi160 bash[232580]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-21T11:02:21.833 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:21 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6-activate[232610]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-21T11:02:21.833 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:21 smithi160 bash[232580]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-21T11:02:21.833 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:21 smithi160 podman[232580]: 2024-09-21 11:02:21.492837016 +0000 UTC m=+1.090521981 container died 00c98c3563b715b7a7d0e37639628b3889ba4c93ea87aded762164b2863d61b2 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6-activate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, RELEASE=reef-6a1bdf8, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-21T11:02:21.898 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:21.928 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:21.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:21.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:22.029 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:22.066 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:22.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:22.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:22.167 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:22.200 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:22.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:22.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:22.302 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:22.330 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:22.330 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 24 16 63022 63006 164.052 122.5 0.00716343 0.00609083 2024-09-21T11:02:22.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:22.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:22.431 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:22.453 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:22 smithi160 podman[232797]: 2024-09-21 11:02:22.155428726 +0000 UTC m=+0.651466898 container remove 00c98c3563b715b7a7d0e37639628b3889ba4c93ea87aded762164b2863d61b2 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=reef-6a1bdf8, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T11:02:22.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:22.463 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:22.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:22.564 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:22.597 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:22.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:22.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:22.699 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:22.727 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:22 smithi160 podman[232897]: 2024-09-21 11:02:22.354028341 +0000 UTC m=+0.031928838 image pull 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6 2024-09-21T11:02:22.728 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:22 smithi160 podman[232897]: 2024-09-21 11:02:22.490067826 +0000 UTC m=+0.167968309 container create e89916e49bf5a8b61c796656ed7e9aa11822ece713d8f262c14cd4044e15175c (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-6a1bdf8) 2024-09-21T11:02:22.728 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:22.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:22.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:22.830 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:22.858 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:22.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:22.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:22.884 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T11:02:22.960 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:22.989 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:22.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:22.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:23.090 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:23.091 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:22 smithi160 podman[232897]: 2024-09-21 11:02:22.726281789 +0000 UTC m=+0.404182280 container init e89916e49bf5a8b61c796656ed7e9aa11822ece713d8f262c14cd4044e15175c (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=reef-6a1bdf8, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-21T11:02:23.091 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:22 smithi160 podman[232897]: 2024-09-21 11:02:22.73435498 +0000 UTC m=+0.412255460 container start e89916e49bf5a8b61c796656ed7e9aa11822ece713d8f262c14cd4044e15175c (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-6a1bdf8) 2024-09-21T11:02:23.091 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:22 smithi160 bash[232897]: e89916e49bf5a8b61c796656ed7e9aa11822ece713d8f262c14cd4044e15175c 2024-09-21T11:02:23.091 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:22 smithi160 systemd[1]: Started Ceph osd.6 for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T11:02:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:23.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:23.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:23.198 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:02:23.222 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:23.248 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:02:23.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:23.266 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:23.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:23.330 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 25 16 64930 64914 162.259 119.25 0.00827749 0.00615874 2024-09-21T11:02:23.367 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:23.384 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:02:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:02:23] "GET /metrics HTTP/1.1" 200 36551 "" "Prometheus/2.51.0" 2024-09-21T11:02:23.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:23.395 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:23.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:23.496 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:23.525 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:23.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:23.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:23.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:23 smithi160 ceph-mon[101879]: pgmap v3091: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 3.6 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 123 MiB/s wr, 1.97k op/s; 36539/175962 objects degraded (20.765%); 13/175962 objects misplaced (0.007%) 2024-09-21T11:02:23.591 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:23 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:02:23.149+0000 7fab86a95640 -1 Falling back to public interface 2024-09-21T11:02:23.627 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:23.649 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:23 smithi070 ceph-mon[107385]: pgmap v3091: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 3.6 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 123 MiB/s wr, 1.97k op/s; 36539/175962 objects degraded (20.765%); 13/175962 objects misplaced (0.007%) 2024-09-21T11:02:23.649 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:23 smithi070 ceph-mon[112139]: pgmap v3091: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 3.6 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 123 MiB/s wr, 1.97k op/s; 36539/175962 objects degraded (20.765%); 13/175962 objects misplaced (0.007%) 2024-09-21T11:02:23.660 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:23.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:23.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:23.762 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:23.793 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:23.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:23.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:23.895 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:23.925 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:23.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:23.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:24.026 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:24.057 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:24.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:24.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:24.159 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:24.177 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T11:02:24.193 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:24.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:24.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:24.294 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:24.330 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 26 16 66831 66815 160.588 118.812 0.00537301 0.00622152 2024-09-21T11:02:24.338 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:24.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:24.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:24.440 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:24.470 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:24 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 36539/175962 objects degraded (20.765%), 13 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-21T11:02:24.471 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:24 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 36539/175962 objects degraded (20.765%), 13 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-21T11:02:24.471 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T11:02:24.471 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T11:02:24.471 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T11:02:24.472 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T11:02:24.472 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:02:24.472 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:195880 55.c e1474/1465 55:341b656a:::benchmark_data_smithi070_294394_object90539:head v 1474'12198, mlcod=0'0)", 2024-09-21T11:02:24.472 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:53:26.307538+0000", 2024-09-21T11:02:24.472 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 538.16009328799998, 2024-09-21T11:02:24.472 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.122286966, 2024-09-21T11:02:24.472 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:02:24.472 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:02:24.472 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:02:24.472 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:02:24.472 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:02:24.473 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:02:24.473 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.307538+0000", 2024-09-21T11:02:24.473 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:02:24.473 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:02:24.473 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:02:24.473 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:02:24.473 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.307538+0000", 2024-09-21T11:02:24.473 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:02:24.473 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:02:24.473 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:02:24.473 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:02:24.474 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.307538+0000", 2024-09-21T11:02:24.474 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8700000000000001e-07 2024-09-21T11:02:24.474 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:02:24.474 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:02:24.474 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:02:24.474 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.307542+0000", 2024-09-21T11:02:24.474 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0149999999999997e-06 2024-09-21T11:02:24.474 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:02:24.474 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:02:24.474 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:02:24.475 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.307550+0000", 2024-09-21T11:02:24.475 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4889999999999998e-06 2024-09-21T11:02:24.475 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:02:24.475 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:02:24.475 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:02:24.475 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.307555+0000", 2024-09-21T11:02:24.475 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4909999999999996e-06 2024-09-21T11:02:24.475 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:02:24.475 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:02:24.475 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:02:24.475 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.314879+0000", 2024-09-21T11:02:24.475 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0073239250000000002 2024-09-21T11:02:24.475 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:02:24.476 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:02:24.476 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:02:24.476 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.314904+0000", 2024-09-21T11:02:24.476 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4516999999999999e-05 2024-09-21T11:02:24.476 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:02:24.476 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:02:24.476 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:02:24.476 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.429774+0000", 2024-09-21T11:02:24.476 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.114870376 2024-09-21T11:02:24.476 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:02:24.476 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:02:24.476 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:02:24.476 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.429825+0000", 2024-09-21T11:02:24.477 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0766000000000001e-05 2024-09-21T11:02:24.477 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:02:24.477 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:02:24.477 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:02:24.477 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:02:24.477 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:02:24.477 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.35670.0:195894 55.c e1474/1465 55:3dc3da2f:::benchmark_data_smithi070_294394_object90553:head v 1474'12199, mlcod=0'0)", 2024-09-21T11:02:24.477 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:53:26.309020+0000", 2024-09-21T11:02:24.477 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 538.15861122900003, 2024-09-21T11:02:24.477 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.146994024, 2024-09-21T11:02:24.477 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:02:24.477 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:02:24.478 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:02:24.478 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:02:24.478 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:02:24.478 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:02:24.478 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.309020+0000", 2024-09-21T11:02:24.478 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:02:24.478 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:02:24.478 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:02:24.478 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:02:24.478 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.309020+0000", 2024-09-21T11:02:24.478 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:02:24.478 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:02:24.479 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:02:24.479 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:02:24.479 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.309021+0000", 2024-09-21T11:02:24.479 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9199999999999999e-07 2024-09-21T11:02:24.479 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:02:24.479 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:02:24.479 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:02:24.479 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.309028+0000", 2024-09-21T11:02:24.479 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.844e-06 2024-09-21T11:02:24.479 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:02:24.479 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:02:24.479 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:02:24.479 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.309040+0000", 2024-09-21T11:02:24.479 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2e-05 2024-09-21T11:02:24.480 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:02:24.480 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:02:24.480 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:02:24.480 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.309048+0000", 2024-09-21T11:02:24.480 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9619999999999993e-06 2024-09-21T11:02:24.480 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:02:24.480 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:02:24.480 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:02:24.480 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.443698+0000", 2024-09-21T11:02:24.480 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13465039300000001 2024-09-21T11:02:24.480 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:02:24.480 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:02:24.480 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:02:24.480 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.443727+0000", 2024-09-21T11:02:24.481 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9249000000000001e-05 2024-09-21T11:02:24.481 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:02:24.481 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:02:24.481 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:02:24.481 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.455970+0000", 2024-09-21T11:02:24.481 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012242602999999999 2024-09-21T11:02:24.481 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:02:24.481 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:02:24.481 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:02:24.481 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:53:26.456014+0000", 2024-09-21T11:02:24.481 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3980999999999998e-05 2024-09-21T11:02:24.481 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:02:24.481 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:02:24.481 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:02:24.482 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:02:24.482 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:02:24.482 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T11:02:24.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:24.483 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:24.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:24.583 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:24 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 36539/175962 objects degraded (20.765%), 13 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-21T11:02:24.606 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:24.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:24.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:24.708 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:24.744 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:24.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:24.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:24.845 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:24.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:24.878 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:24.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:24.979 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:25.008 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:25.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:25.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:25.110 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:25.148 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:25.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:25.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:25.250 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:25.289 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:25.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:25.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:25.330 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 27 16 68669 68653 158.894 114.875 0.00825104 0.00628909 2024-09-21T11:02:25.391 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:25.421 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:25.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:25.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:25.492 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:25 smithi160 ceph-mon[101879]: pgmap v3092: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 3.6 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 123 MiB/s wr, 1.97k op/s; 36539/175962 objects degraded (20.765%); 13/175962 objects misplaced (0.007%) 2024-09-21T11:02:25.522 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:25.546 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:25.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:25.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:25.648 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:25.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:25.686 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:25.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:25.787 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:25.822 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:25.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:25.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:25.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:25 smithi070 ceph-mon[107385]: pgmap v3092: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 3.6 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 123 MiB/s wr, 1.97k op/s; 36539/175962 objects degraded (20.765%); 13/175962 objects misplaced (0.007%) 2024-09-21T11:02:25.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:25 smithi070 ceph-mon[112139]: pgmap v3092: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 3.6 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 123 MiB/s wr, 1.97k op/s; 36539/175962 objects degraded (20.765%); 13/175962 objects misplaced (0.007%) 2024-09-21T11:02:25.924 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:25.946 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:25.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:25.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:26.048 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:26.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:26.089 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:26.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:26.190 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:26.225 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:26.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:26.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:26.327 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:26.331 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 28 16 70076 70060 156.36 87.9375 0.00774364 0.00639123 2024-09-21T11:02:26.363 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:26.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:26.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:26.465 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:26.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:26.499 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:26.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:26.600 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:26.635 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:26.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:26.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:26.737 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:26.765 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:26.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:26.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:26.803 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:02:26.836 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T11:02:26.867 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:26.910 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:26.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:26.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:27.012 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:27.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:27.049 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:27.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:27.150 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:27.187 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:27.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:27.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:27.289 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:27.326 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:27.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:27.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:27.330 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 29 16 71640 71624 154.339 97.75 0.00414302 0.00647473 2024-09-21T11:02:27.428 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:27.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:27.462 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:27.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:27.515 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:27 smithi070 ceph-mon[107385]: pgmap v3093: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 4.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 121 MiB/s wr, 1.93k op/s; 42335/204006 objects degraded (20.752%); 13/204006 objects misplaced (0.006%) 2024-09-21T11:02:27.515 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:27 smithi070 ceph-mon[112139]: pgmap v3093: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 4.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 121 MiB/s wr, 1.93k op/s; 42335/204006 objects degraded (20.752%); 13/204006 objects misplaced (0.006%) 2024-09-21T11:02:27.562 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:27.603 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:27.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:27.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:27.640 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:02:27.640 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:02:27.641 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:02:27.641 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:02:27.650 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:02:27.650 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:02:27.679 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:02:27.680 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:02:27.705 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:27.708 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:02:27.708 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:02:27.708 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T11:02:27.709 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T11:02:27.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:27.741 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:27.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:27.754 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:02:27.754 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:02:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:27 smithi160 ceph-mon[101879]: pgmap v3093: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 4.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 121 MiB/s wr, 1.93k op/s; 42335/204006 objects degraded (20.752%); 13/204006 objects misplaced (0.006%) 2024-09-21T11:02:27.842 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:27.871 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:27.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:27.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:27.973 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:28.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:28.009 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:28.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:28.110 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:28.148 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:28.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:28.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:28.250 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:28.281 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:28.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:28.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:28.330 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 30 16 73336 73320 152.727 106 0.00419873 0.00654344 2024-09-21T11:02:28.383 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:28.422 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:28.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:28.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:28.523 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:28.553 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:28.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:28.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:28.654 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:28.675 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:28.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:28.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:28.777 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:28.798 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:28.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:28.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:28 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2915869569' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:02:28.876 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:28 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2915869569' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:02:28.876 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:28 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2915869569' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:02:28.900 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:28.926 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:28.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:28.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:29.028 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:29.059 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:29.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:29.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:29.160 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:29.185 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:29.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:29.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:29.287 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:29.288 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T11:02:29.289 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T11:02:29.330 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:29.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:29.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:29.331 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 31 16 74717 74701 150.585 86.3125 0.00591855 0.00663667 2024-09-21T11:02:29.432 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:29.460 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:29.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:29.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:29.562 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:29.572 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:29 smithi070 ceph-mon[107385]: pgmap v3094: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 4.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 121 MiB/s wr, 1.93k op/s; 42335/204006 objects degraded (20.752%); 13/204006 objects misplaced (0.006%) 2024-09-21T11:02:29.572 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:29 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 42335/204006 objects degraded (20.752%), 13 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-21T11:02:29.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:29 smithi070 ceph-mon[112139]: pgmap v3094: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 4.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 121 MiB/s wr, 1.93k op/s; 42335/204006 objects degraded (20.752%); 13/204006 objects misplaced (0.006%) 2024-09-21T11:02:29.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:29 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 42335/204006 objects degraded (20.752%), 13 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-21T11:02:29.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:29.599 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:29.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:29.700 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:29.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:29.735 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:29.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:29.836 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:29.837 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:02:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:29 smithi160 ceph-mon[101879]: pgmap v3094: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 4.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 121 MiB/s wr, 1.93k op/s; 42335/204006 objects degraded (20.752%); 13/204006 objects misplaced (0.006%) 2024-09-21T11:02:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:29 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 42335/204006 objects degraded (20.752%), 13 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-21T11:02:29.878 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:29.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:29.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:29.979 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:30.000 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:30.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:30.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:30.102 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:30.135 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:30.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:30.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:30.237 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:30.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:30.266 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:30.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:30.331 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 32 16 76296 76280 148.963 98.6875 0.00496583 0.00670899 2024-09-21T11:02:30.367 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:30.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:30.392 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:30.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:30.437 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T11:02:30.493 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:30.522 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:30.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:30.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:30.624 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:30.652 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:30.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:30.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:30.707 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:30 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1241069743' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:02:30.707 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:30 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:02:30.754 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:30.774 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:30.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:30.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:30.876 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:30.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:30 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1241069743' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:02:30.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:30 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:02:30.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:30 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1241069743' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:02:30.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:30 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:02:30.901 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:30.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:30.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:31.002 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:31.045 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:31.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:31.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:31.147 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:31.173 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:31.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:31.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:31.231 INFO:teuthology.orchestra.run.smithi160.stderr:no valid command found; 10 closest matches: 2024-09-21T11:02:31.231 INFO:teuthology.orchestra.run.smithi160.stderr:0 2024-09-21T11:02:31.231 INFO:teuthology.orchestra.run.smithi160.stderr:1 2024-09-21T11:02:31.231 INFO:teuthology.orchestra.run.smithi160.stderr:2 2024-09-21T11:02:31.231 INFO:teuthology.orchestra.run.smithi160.stderr:abort 2024-09-21T11:02:31.231 INFO:teuthology.orchestra.run.smithi160.stderr:assert 2024-09-21T11:02:31.231 INFO:teuthology.orchestra.run.smithi160.stderr:bluestore allocator dump block 2024-09-21T11:02:31.231 INFO:teuthology.orchestra.run.smithi160.stderr:bluestore allocator fragmentation block 2024-09-21T11:02:31.232 INFO:teuthology.orchestra.run.smithi160.stderr:bluestore allocator score block 2024-09-21T11:02:31.232 INFO:teuthology.orchestra.run.smithi160.stderr:config diff 2024-09-21T11:02:31.232 INFO:teuthology.orchestra.run.smithi160.stderr:config diff get 2024-09-21T11:02:31.232 INFO:teuthology.orchestra.run.smithi160.stderr:admin_socket: invalid command 2024-09-21T11:02:31.275 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:31.290 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T11:02:31.315 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:31.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:31.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:31.330 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 33 16 77740 77724 147.184 90.25 0.0175129 0.00679004 2024-09-21T11:02:31.417 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:31.452 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:31.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:31.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:31.554 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:31.574 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:31.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:31.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:31.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:31 smithi160 ceph-mon[101879]: pgmap v3095: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 4.4 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 124 MiB/s wr, 1.99k op/s; 45308/215874 objects degraded (20.988%); 13/215874 objects misplaced (0.006%) 2024-09-21T11:02:31.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:31 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:02:31.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:31 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T11:02:31.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:31 smithi160 ceph-mon[101879]: osdmap e1587: 8 total, 6 up, 5 in 2024-09-21T11:02:31.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:31 smithi070 ceph-mon[107385]: pgmap v3095: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 4.4 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 124 MiB/s wr, 1.99k op/s; 45308/215874 objects degraded (20.988%); 13/215874 objects misplaced (0.006%) 2024-09-21T11:02:31.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:31 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:02:31.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:31 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T11:02:31.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:31 smithi070 ceph-mon[107385]: osdmap e1587: 8 total, 6 up, 5 in 2024-09-21T11:02:31.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:31 smithi070 ceph-mon[112139]: pgmap v3095: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 4.4 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 124 MiB/s wr, 1.99k op/s; 45308/215874 objects degraded (20.988%); 13/215874 objects misplaced (0.006%) 2024-09-21T11:02:31.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:31 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:02:31.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:31 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T11:02:31.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:31 smithi070 ceph-mon[112139]: osdmap e1587: 8 total, 6 up, 5 in 2024-09-21T11:02:31.677 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:31.704 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:31.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:31.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:31.806 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:31.849 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:31.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:31.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:31.951 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:31.986 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:31.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:31.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:32.008 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T11:02:32.009 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-09-21T11:02:32.088 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:32.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:32.116 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:32.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:32.217 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:32.245 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:32.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:32.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:32.331 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 34 16 79102 79086 145.358 85.125 0.00242807 0.00687263 2024-09-21T11:02:32.346 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:32.367 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:32.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:32.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:32.469 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:32.499 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:32.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:32.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:32.600 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:32.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:32.631 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:32.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:32.732 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:32.761 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:32.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:32.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:32.863 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:32.892 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:32.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:32.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:32.994 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:33.022 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:33.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:33.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:33.124 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:33.154 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:33.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:33.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:33.256 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:33.286 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:33.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:33.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:33.331 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 35 16 80813 80797 144.26 106.938 0.0066313 0.00692788 2024-09-21T11:02:33.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:02:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:02:33] "GET /metrics HTTP/1.1" 200 36551 "" "Prometheus/2.51.0" 2024-09-21T11:02:33.388 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:33.418 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:33.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:33.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:33.520 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:33.547 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:02:33.561 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:33.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:33.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:33.663 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:33.702 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:33.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:33.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:33.761 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:33 smithi070 ceph-mon[107385]: pgmap v3097: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 4.6 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 107 MiB/s wr, 1.71k op/s; 47197/227421 objects degraded (20.753%); 13/227421 objects misplaced (0.006%) 2024-09-21T11:02:33.761 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:33 smithi070 ceph-mon[112139]: pgmap v3097: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 4.6 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 107 MiB/s wr, 1.71k op/s; 47197/227421 objects degraded (20.753%); 13/227421 objects misplaced (0.006%) 2024-09-21T11:02:33.804 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:33.833 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:33.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:33.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:33 smithi160 ceph-mon[101879]: pgmap v3097: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 4.6 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 107 MiB/s wr, 1.71k op/s; 47197/227421 objects degraded (20.753%); 13/227421 objects misplaced (0.006%) 2024-09-21T11:02:33.935 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:33.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:33.965 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:33.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:34.065 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:34.092 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:34.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:34.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:34.194 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:34.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:34.224 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:34.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:34.324 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:34.331 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 36 16 82603 82587 143.36 111.875 0.00401993 0.00697062 2024-09-21T11:02:34.370 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:34.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:34.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:34.472 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:34.507 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:34.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:34.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:34.608 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:34.643 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:34.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:34.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:34.720 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:34 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 47197/227421 objects degraded (20.753%), 13 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-21T11:02:34.720 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:34 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 47197/227421 objects degraded (20.753%), 13 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-21T11:02:34.745 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:34.774 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:34.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:34.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:34 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 47197/227421 objects degraded (20.753%), 13 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-21T11:02:34.876 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:34.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:34.914 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:34.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:35.015 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:35.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:35.051 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:35.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:35.072 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T11:02:35.152 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:35.191 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:35.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:35.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:35.292 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:35.328 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:35.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:35.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:35.331 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 37 16 84347 84331 142.431 109 0.00657969 0.00701635 2024-09-21T11:02:35.430 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:35.455 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:35.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:35.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:35.557 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:35.607 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:35.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:35.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:35.639 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T11:02:35.639 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T11:02:35.709 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:35.742 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:35.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:35.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:35 smithi160 ceph-mon[101879]: pgmap v3098: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 4.6 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 107 MiB/s wr, 1.71k op/s; 47197/227421 objects degraded (20.753%); 13/227421 objects misplaced (0.006%) 2024-09-21T11:02:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:02:35.843 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:35.872 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:35.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:35.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:35 smithi070 ceph-mon[107385]: pgmap v3098: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 4.6 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 107 MiB/s wr, 1.71k op/s; 47197/227421 objects degraded (20.753%); 13/227421 objects misplaced (0.006%) 2024-09-21T11:02:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:02:35.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:35 smithi070 ceph-mon[112139]: pgmap v3098: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 4.6 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 107 MiB/s wr, 1.71k op/s; 47197/227421 objects degraded (20.753%); 13/227421 objects misplaced (0.006%) 2024-09-21T11:02:35.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:02:35.974 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:36.006 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:36.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:36.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:36.108 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:36.137 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:36.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:36.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:36.238 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:36.270 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:36.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:36.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:36.331 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 38 16 86015 85999 141.426 104.25 0.00820644 0.00706697 2024-09-21T11:02:36.373 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:36.406 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:36.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:36.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:36.508 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:36.554 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:36.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:36.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:36 smithi160 ceph-mon[101879]: from='osd.6 [v2:172.21.15.160:6808/3336777196,v1:172.21.15.160:6809/3336777196]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T11:02:36.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:36 smithi160 ceph-mon[101879]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T11:02:36.591 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:36 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:02:36.210+0000 7fab86a95640 -1 osd.6 1540 log_to_monitors true 2024-09-21T11:02:36.656 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:36.681 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:36.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:36.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:36.783 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:36.807 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:36.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:36.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:36.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:36 smithi070 ceph-mon[107385]: from='osd.6 [v2:172.21.15.160:6808/3336777196,v1:172.21.15.160:6809/3336777196]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T11:02:36.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:36 smithi070 ceph-mon[107385]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T11:02:36.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:36 smithi070 ceph-mon[112139]: from='osd.6 [v2:172.21.15.160:6808/3336777196,v1:172.21.15.160:6809/3336777196]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T11:02:36.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:36 smithi070 ceph-mon[112139]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T11:02:36.908 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:36.946 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:36.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:36.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:37.010 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T11:02:37.047 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:37.048 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:02:36 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:02:36.653+0000 7fab7a837640 -1 osd.6 1588 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T11:02:37.073 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:37.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:37.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:37.175 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:37.205 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:37.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:37.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:37.307 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:37.331 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 39 16 87365 87349 139.963 84.375 0.0123705 0.00714001 2024-09-21T11:02:37.337 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:37.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:37.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:37.370 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:02:37.439 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:37.469 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:37.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:37.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:37.571 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:37.591 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:37.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:37.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:37 smithi070 ceph-mon[107385]: pgmap v3099: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 5.1 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 99 MiB/s wr, 1.58k op/s; 52184/251319 objects degraded (20.764%); 13/251319 objects misplaced (0.005%) 2024-09-21T11:02:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:37 smithi070 ceph-mon[107385]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T11:02:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:37 smithi070 ceph-mon[107385]: osdmap e1588: 8 total, 6 up, 5 in 2024-09-21T11:02:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:37 smithi070 ceph-mon[107385]: from='osd.6 [v2:172.21.15.160:6808/3336777196,v1:172.21.15.160:6809/3336777196]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T11:02:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:37 smithi070 ceph-mon[107385]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T11:02:37.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:37 smithi070 ceph-mon[112139]: pgmap v3099: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 5.1 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 99 MiB/s wr, 1.58k op/s; 52184/251319 objects degraded (20.764%); 13/251319 objects misplaced (0.005%) 2024-09-21T11:02:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:37 smithi070 ceph-mon[112139]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T11:02:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:37 smithi070 ceph-mon[112139]: osdmap e1588: 8 total, 6 up, 5 in 2024-09-21T11:02:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:37 smithi070 ceph-mon[112139]: from='osd.6 [v2:172.21.15.160:6808/3336777196,v1:172.21.15.160:6809/3336777196]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T11:02:37.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:37 smithi070 ceph-mon[112139]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T11:02:37.693 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:37.716 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:37.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:37.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:37.717 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:02:37.805 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:37 smithi160 ceph-mon[101879]: pgmap v3099: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 5.1 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 99 MiB/s wr, 1.58k op/s; 52184/251319 objects degraded (20.764%); 13/251319 objects misplaced (0.005%) 2024-09-21T11:02:37.805 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:37 smithi160 ceph-mon[101879]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T11:02:37.805 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:37 smithi160 ceph-mon[101879]: osdmap e1588: 8 total, 6 up, 5 in 2024-09-21T11:02:37.805 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:37 smithi160 ceph-mon[101879]: from='osd.6 [v2:172.21.15.160:6808/3336777196,v1:172.21.15.160:6809/3336777196]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T11:02:37.805 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:37 smithi160 ceph-mon[101879]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T11:02:37.818 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:37.839 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:37.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:37.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:37.940 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:37.962 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:37.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:37.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:38.063 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:38.085 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:38.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:38.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:38.187 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:38.210 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:38.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:38.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:38.311 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:38.331 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:2024-09-21T11:02:38.330652+0000 min lat: 0.00100109 max lat: 0.230688 avg lat: 0.00715169 2024-09-21T11:02:38.331 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T11:02:38.331 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 40 16 87548 87532 136.75 11.4375 0.0032855 0.00715169 2024-09-21T11:02:38.332 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:38.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:38.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:38.434 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:38.452 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:02:38.453 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:02:38.457 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:38.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:38.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:38.483 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:02:38.483 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:02:38.505 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:02:38.505 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:02:38.521 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:02:38.521 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:02:38.522 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:38 smithi070 ceph-mon[112139]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T11:02:38.522 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:38 smithi070 ceph-mon[112139]: osd.6 [v2:172.21.15.160:6808/3336777196,v1:172.21.15.160:6809/3336777196] boot 2024-09-21T11:02:38.522 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:38 smithi070 ceph-mon[112139]: osdmap e1589: 8 total, 7 up, 5 in 2024-09-21T11:02:38.522 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:38 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T11:02:38.522 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:38 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2576070452' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:02:38.522 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:38 smithi070 ceph-mon[112139]: osdmap e1590: 8 total, 7 up, 5 in 2024-09-21T11:02:38.522 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:38 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 52184/251319 objects degraded (20.764%), 13 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-21T11:02:38.522 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:38 smithi070 ceph-mon[107385]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T11:02:38.522 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:38 smithi070 ceph-mon[107385]: osd.6 [v2:172.21.15.160:6808/3336777196,v1:172.21.15.160:6809/3336777196] boot 2024-09-21T11:02:38.522 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:38 smithi070 ceph-mon[107385]: osdmap e1589: 8 total, 7 up, 5 in 2024-09-21T11:02:38.522 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:38 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T11:02:38.522 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:38 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2576070452' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:02:38.523 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:38 smithi070 ceph-mon[107385]: osdmap e1590: 8 total, 7 up, 5 in 2024-09-21T11:02:38.523 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:38 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 52184/251319 objects degraded (20.764%), 13 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-21T11:02:38.537 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:02:38.538 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:02:38.538 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T11:02:38.554 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T11:02:38.554 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-21T11:02:38.559 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:38.570 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:02:38.570 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:02:38.580 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:38.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:38.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:38.584 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:38 smithi160 ceph-mon[101879]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T11:02:38.584 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:38 smithi160 ceph-mon[101879]: osd.6 [v2:172.21.15.160:6808/3336777196,v1:172.21.15.160:6809/3336777196] boot 2024-09-21T11:02:38.585 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:38 smithi160 ceph-mon[101879]: osdmap e1589: 8 total, 7 up, 5 in 2024-09-21T11:02:38.585 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:38 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T11:02:38.585 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:38 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2576070452' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:02:38.585 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:38 smithi160 ceph-mon[101879]: osdmap e1590: 8 total, 7 up, 5 in 2024-09-21T11:02:38.585 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:38 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 52184/251319 objects degraded (20.764%), 13 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-21T11:02:38.682 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:38.703 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:38.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:38.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:38.805 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:38.828 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:38.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:38.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:38.930 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:38.951 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:38.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:38.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:39.053 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:39.074 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:39.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:39.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:39.176 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:39.197 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:39.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:39.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:39.298 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:39.320 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:39.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:39.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:39.331 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 41 16 87548 87532 133.415 0 - 0.00715169 2024-09-21T11:02:39.422 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:39.446 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:39.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:39.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:39.547 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:39.570 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:39.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:39.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:39.672 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:39.674 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T11:02:39.675 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:39 smithi160 ceph-mon[101879]: pgmap v3102: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 5.1 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 94 MiB/s wr, 1.50k op/s; 52184/251319 objects degraded (20.764%); 13/251319 objects misplaced (0.005%) 2024-09-21T11:02:39.675 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:39 smithi160 ceph-mon[101879]: osdmap e1591: 8 total, 7 up, 5 in 2024-09-21T11:02:39.678 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T11:02:39.678 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T11:02:39.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:02:39.678 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.36567.0:87572 58.1 e1591/1590 58:81e544e2:::benchmark_data_smithi070_396992_object87571:head v 1591'5433, mlcod=1591'5430)", 2024-09-21T11:02:39.678 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:02:39.671417+0000", 2024-09-21T11:02:39.678 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.00036800599999999999, 2024-09-21T11:02:39.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00039291100000000002, 2024-09-21T11:02:39.679 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:02:39.679 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T11:02:39.679 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:02:39.679 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:02:39.679 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:02:39.679 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:39.671417+0000", 2024-09-21T11:02:39.679 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:02:39.679 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:02:39.679 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:02:39.679 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:02:39.679 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:39.671417+0000", 2024-09-21T11:02:39.680 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:02:39.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:02:39.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:02:39.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:02:39.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:39.671416+0000", 2024-09-21T11:02:39.680 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:02:39.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:02:39.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:02:39.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:02:39.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:39.671421+0000", 2024-09-21T11:02:39.680 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1649999999999997e-06 2024-09-21T11:02:39.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:02:39.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:02:39.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:02:39.681 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:39.671435+0000", 2024-09-21T11:02:39.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4038e-05 2024-09-21T11:02:39.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:02:39.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:02:39.681 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:02:39.681 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:39.671442+0000", 2024-09-21T11:02:39.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.348e-06 2024-09-21T11:02:39.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:02:39.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:02:39.681 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:02:39.681 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:39.671479+0000", 2024-09-21T11:02:39.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7236000000000001e-05 2024-09-21T11:02:39.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:02:39.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:02:39.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:02:39.682 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:39.671499+0000", 2024-09-21T11:02:39.682 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9703000000000001e-05 2024-09-21T11:02:39.682 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:02:39.682 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:02:39.682 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:02:39.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:02:39.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:02:39.682 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.36567.0:87614 58.b e1591/1590)", 2024-09-21T11:02:39.682 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:02:39.671656+0000", 2024-09-21T11:02:39.682 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.00012897200000000001, 2024-09-21T11:02:39.682 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00020973199999999999, 2024-09-21T11:02:39.682 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:02:39.682 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T11:02:39.683 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:02:39.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:02:39.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:02:39.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:39.671656+0000", 2024-09-21T11:02:39.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:02:39.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:02:39.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:02:39.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:02:39.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:39.671656+0000", 2024-09-21T11:02:39.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:02:39.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:02:39.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:02:39.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:02:39.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:39.671655+0000", 2024-09-21T11:02:39.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T11:02:39.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:02:39.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:02:39.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:02:39.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:39.671661+0000", 2024-09-21T11:02:39.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3000000000000001e-06 2024-09-21T11:02:39.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:02:39.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:02:39.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:02:39.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:39.671669+0000", 2024-09-21T11:02:39.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6200000000000005e-06 2024-09-21T11:02:39.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:02:39.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:02:39.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:02:39.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:39.671674+0000", 2024-09-21T11:02:39.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0830000000000003e-06 2024-09-21T11:02:39.685 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:02:39.685 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:02:39.685 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:02:39.685 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:02:39.685 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T11:02:39.685 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 2 2024-09-21T11:02:39.685 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T11:02:39.717 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:39.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:39.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:39.772 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:39 smithi070 ceph-mon[107385]: pgmap v3102: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 5.1 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 94 MiB/s wr, 1.50k op/s; 52184/251319 objects degraded (20.764%); 13/251319 objects misplaced (0.005%) 2024-09-21T11:02:39.772 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:39 smithi070 ceph-mon[107385]: osdmap e1591: 8 total, 7 up, 5 in 2024-09-21T11:02:39.772 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:39 smithi070 ceph-mon[112139]: pgmap v3102: 109 pgs: 40 active+clean+remapped, 18 active+undersized, 13 active+undersized+degraded, 38 active+clean; 5.1 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 94 MiB/s wr, 1.50k op/s; 52184/251319 objects degraded (20.764%); 13/251319 objects misplaced (0.005%) 2024-09-21T11:02:39.772 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:39 smithi070 ceph-mon[112139]: osdmap e1591: 8 total, 7 up, 5 in 2024-09-21T11:02:39.819 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:39.852 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:39.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:39.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:39.953 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:39.988 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:39.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:39.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:40.090 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:40.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:40.128 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:40.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:40.228 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:40.258 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:40.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:40.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:40.331 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 42 16 88494 88478 131.646 29.5625 0.00504214 0.00759118 2024-09-21T11:02:40.360 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:40.388 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:40.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:40.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:40.490 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:40.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:40.536 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:40.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:40.609 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T11:02:40.610 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T11:02:40.636 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:40.648 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:02:40.676 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:40.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:40.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:40.756 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:40 smithi070 ceph-mon[107385]: 58.b starting backfill to osd.6 from (0'0,0'0] MAX to 1588'5578 2024-09-21T11:02:40.757 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:40 smithi070 ceph-mon[107385]: 58.f starting backfill to osd.6 from (0'0,0'0] MAX to 1588'5362 2024-09-21T11:02:40.757 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:40 smithi070 ceph-mon[107385]: 58.1 starting backfill to osd.6 from (0'0,0'0] MAX to 1588'5428 2024-09-21T11:02:40.757 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:40 smithi070 ceph-mon[107385]: 58.8 starting backfill to osd.6 from (0'0,0'0] MAX to 1588'5460 2024-09-21T11:02:40.757 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:40 smithi070 ceph-mon[107385]: 58.3 starting backfill to osd.6 from (0'0,0'0] MAX to 1588'5451 2024-09-21T11:02:40.757 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:40 smithi070 ceph-mon[107385]: 58.a starting backfill to osd.6 from (0'0,0'0] MAX to 1588'5475 2024-09-21T11:02:40.757 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:40 smithi070 ceph-mon[107385]: 58.7 starting backfill to osd.6 from (0'0,0'0] MAX to 1588'5412 2024-09-21T11:02:40.757 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:40 smithi070 ceph-mon[107385]: 58.c starting backfill to osd.6 from (0'0,0'0] MAX to 1588'5598 2024-09-21T11:02:40.757 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:40 smithi070 ceph-mon[107385]: 58.d starting backfill to osd.6 from (0'0,0'0] MAX to 1588'5482 2024-09-21T11:02:40.757 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:40 smithi070 ceph-mon[112139]: 58.b starting backfill to osd.6 from (0'0,0'0] MAX to 1588'5578 2024-09-21T11:02:40.757 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:40 smithi070 ceph-mon[112139]: 58.f starting backfill to osd.6 from (0'0,0'0] MAX to 1588'5362 2024-09-21T11:02:40.757 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:40 smithi070 ceph-mon[112139]: 58.1 starting backfill to osd.6 from (0'0,0'0] MAX to 1588'5428 2024-09-21T11:02:40.758 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:40 smithi070 ceph-mon[112139]: 58.8 starting backfill to osd.6 from (0'0,0'0] MAX to 1588'5460 2024-09-21T11:02:40.758 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:40 smithi070 ceph-mon[112139]: 58.3 starting backfill to osd.6 from (0'0,0'0] MAX to 1588'5451 2024-09-21T11:02:40.758 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:40 smithi070 ceph-mon[112139]: 58.a starting backfill to osd.6 from (0'0,0'0] MAX to 1588'5475 2024-09-21T11:02:40.758 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:40 smithi070 ceph-mon[112139]: 58.7 starting backfill to osd.6 from (0'0,0'0] MAX to 1588'5412 2024-09-21T11:02:40.758 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:40 smithi070 ceph-mon[112139]: 58.c starting backfill to osd.6 from (0'0,0'0] MAX to 1588'5598 2024-09-21T11:02:40.758 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:40 smithi070 ceph-mon[112139]: 58.d starting backfill to osd.6 from (0'0,0'0] MAX to 1588'5482 2024-09-21T11:02:40.778 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:40.810 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:40.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:40.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:40 smithi160 ceph-mon[101879]: 58.b starting backfill to osd.6 from (0'0,0'0] MAX to 1588'5578 2024-09-21T11:02:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:40 smithi160 ceph-mon[101879]: 58.f starting backfill to osd.6 from (0'0,0'0] MAX to 1588'5362 2024-09-21T11:02:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:40 smithi160 ceph-mon[101879]: 58.1 starting backfill to osd.6 from (0'0,0'0] MAX to 1588'5428 2024-09-21T11:02:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:40 smithi160 ceph-mon[101879]: 58.8 starting backfill to osd.6 from (0'0,0'0] MAX to 1588'5460 2024-09-21T11:02:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:40 smithi160 ceph-mon[101879]: 58.3 starting backfill to osd.6 from (0'0,0'0] MAX to 1588'5451 2024-09-21T11:02:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:40 smithi160 ceph-mon[101879]: 58.a starting backfill to osd.6 from (0'0,0'0] MAX to 1588'5475 2024-09-21T11:02:40.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:40 smithi160 ceph-mon[101879]: 58.7 starting backfill to osd.6 from (0'0,0'0] MAX to 1588'5412 2024-09-21T11:02:40.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:40 smithi160 ceph-mon[101879]: 58.c starting backfill to osd.6 from (0'0,0'0] MAX to 1588'5598 2024-09-21T11:02:40.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:40 smithi160 ceph-mon[101879]: 58.d starting backfill to osd.6 from (0'0,0'0] MAX to 1588'5482 2024-09-21T11:02:40.911 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:40.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:40.947 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:40.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:41.048 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:41.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:41.078 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:41.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:41.179 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:41.214 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:41.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:41.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:41.316 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:41.331 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 43 16 89878 89862 130.596 86.5 0.00348882 0.00765085 2024-09-21T11:02:41.345 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:41.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:41.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:41.447 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:41.480 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:41.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:41.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:41.571 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:41 smithi070 ceph-mon[107385]: pgmap v3105: 109 pgs: 1 activating+undersized+degraded+remapped, 5 remapped+peering, 7 peering, 33 active+clean+remapped, 15 active+undersized, 6 active+undersized+degraded, 42 active+clean; 5.3 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 37 MiB/s wr, 596 op/s; 26693/258480 objects degraded (10.327%); 13/258480 objects misplaced (0.005%) 2024-09-21T11:02:41.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:41 smithi070 ceph-mon[112139]: pgmap v3105: 109 pgs: 1 activating+undersized+degraded+remapped, 5 remapped+peering, 7 peering, 33 active+clean+remapped, 15 active+undersized, 6 active+undersized+degraded, 42 active+clean; 5.3 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 37 MiB/s wr, 596 op/s; 26693/258480 objects degraded (10.327%); 13/258480 objects misplaced (0.005%) 2024-09-21T11:02:41.582 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:41.621 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:41.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:41.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:41.697 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:41 smithi160 ceph-mon[101879]: pgmap v3105: 109 pgs: 1 activating+undersized+degraded+remapped, 5 remapped+peering, 7 peering, 33 active+clean+remapped, 15 active+undersized, 6 active+undersized+degraded, 42 active+clean; 5.3 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 37 MiB/s wr, 596 op/s; 26693/258480 objects degraded (10.327%); 13/258480 objects misplaced (0.005%) 2024-09-21T11:02:41.722 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:41.758 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:41.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:41.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:41.860 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:41.890 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:41.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:41.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:41.992 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:42.033 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:42.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:42.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:42.135 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:42.171 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:42.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:42.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:42.273 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:42.300 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:42.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:42.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:42.331 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 44 16 91256 91240 129.586 86.125 0.00786711 0.0077123 2024-09-21T11:02:42.402 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:42.431 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:42.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:42.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:42.495 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T11:02:42.534 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:42.570 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:42.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:42.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:42.611 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T11:02:42.672 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:42.698 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:42.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:42.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:42.800 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:02:42.837 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:42.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:42.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:02:42.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:42 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/769925469' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:02:42.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:42 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:02:42.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:42 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/769925469' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:02:42.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:42 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:02:42.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:42 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/769925469' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:02:42.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:42 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:02:42.939 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:42.973 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:42.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:42.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:43.075 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:43.111 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:43.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:43.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:43.213 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:43.240 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:43.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:43.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:43.332 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 45 16 92566 92550 128.525 81.875 0.00464641 0.00777598 2024-09-21T11:02:43.342 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:43.371 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:43.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:43.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:43.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:02:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:02:43] "GET /metrics HTTP/1.1" 200 36549 "" "Prometheus/2.51.0" 2024-09-21T11:02:43.473 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:43.494 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:43.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:43.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:43.596 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:43.636 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:43.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:43.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:43.738 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:43.765 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:43.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:43.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:43.785 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:43 smithi070 ceph-mon[107385]: pgmap v3106: 109 pgs: 1 activating+undersized+degraded+remapped, 4 active+undersized+degraded+remapped+backfill_wait, 5 remapped+peering, 7 peering, 92 active+clean; 5.4 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 13 KiB/s rd, 48 MiB/s wr, 770 op/s; 27728/264645 objects degraded (10.477%); 330 KiB/s, 2 objects/s recovering 2024-09-21T11:02:43.785 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:43 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:02:43.785 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:43 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T11:02:43.785 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:43 smithi070 ceph-mon[107385]: osdmap e1592: 8 total, 7 up, 5 in 2024-09-21T11:02:43.785 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:43 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 27728/264645 objects degraded (10.477%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T11:02:43.785 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:43 smithi070 ceph-mon[112139]: pgmap v3106: 109 pgs: 1 activating+undersized+degraded+remapped, 4 active+undersized+degraded+remapped+backfill_wait, 5 remapped+peering, 7 peering, 92 active+clean; 5.4 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 13 KiB/s rd, 48 MiB/s wr, 770 op/s; 27728/264645 objects degraded (10.477%); 330 KiB/s, 2 objects/s recovering 2024-09-21T11:02:43.785 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:43 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:02:43.785 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:43 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T11:02:43.786 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:43 smithi070 ceph-mon[112139]: osdmap e1592: 8 total, 7 up, 5 in 2024-09-21T11:02:43.786 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:43 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 27728/264645 objects degraded (10.477%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T11:02:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:43 smithi160 ceph-mon[101879]: pgmap v3106: 109 pgs: 1 activating+undersized+degraded+remapped, 4 active+undersized+degraded+remapped+backfill_wait, 5 remapped+peering, 7 peering, 92 active+clean; 5.4 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 13 KiB/s rd, 48 MiB/s wr, 770 op/s; 27728/264645 objects degraded (10.477%); 330 KiB/s, 2 objects/s recovering 2024-09-21T11:02:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:43 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:02:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:43 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T11:02:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:43 smithi160 ceph-mon[101879]: osdmap e1592: 8 total, 7 up, 5 in 2024-09-21T11:02:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:43 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 27728/264645 objects degraded (10.477%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T11:02:43.868 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:43.901 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:43.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:43.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:44.003 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:44.032 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:44.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:44.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:44.134 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:44.166 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:44.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:44.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:44.268 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:44.309 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:44.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:44.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:44.332 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 46 16 93803 93787 127.412 77.3125 0.00972322 0.00784383 2024-09-21T11:02:44.411 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:02:44.446 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:44.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:44.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:02:44.547 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:44.579 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:44.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:44.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:44.681 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:44.710 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:44.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:44.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:44.812 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:44.843 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:44.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:44.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:44.945 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:44.978 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:44.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:44.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:45.031 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:02:45.064 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T11:02:45.064 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T11:02:45.064 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T11:02:45.065 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T11:02:45.065 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T11:02:45.080 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:02:45.118 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:45.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:45.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:02:45.220 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:45.261 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:45.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:45.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:45.332 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 47 16 95178 95162 126.529 85.9375 0.00843133 0.00789871 2024-09-21T11:02:45.363 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:02:45.396 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:45.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:45.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:02:45.498 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:45.527 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:45.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:45.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:45.629 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:45.658 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:45.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:45.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:45.760 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:45.799 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:45.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:45.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:45.830 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:45 smithi070 ceph-mon[107385]: pgmap v3108: 109 pgs: 1 activating+undersized+degraded+remapped, 4 active+undersized+degraded+remapped+backfill_wait, 5 remapped+peering, 7 peering, 92 active+clean; 5.4 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 12 KiB/s rd, 46 MiB/s wr, 749 op/s; 27728/264645 objects degraded (10.477%); 321 KiB/s, 2 objects/s recovering 2024-09-21T11:02:45.830 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:45 smithi070 ceph-mon[112139]: pgmap v3108: 109 pgs: 1 activating+undersized+degraded+remapped, 4 active+undersized+degraded+remapped+backfill_wait, 5 remapped+peering, 7 peering, 92 active+clean; 5.4 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 12 KiB/s rd, 46 MiB/s wr, 749 op/s; 27728/264645 objects degraded (10.477%); 321 KiB/s, 2 objects/s recovering 2024-09-21T11:02:45.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:45 smithi160 ceph-mon[101879]: pgmap v3108: 109 pgs: 1 activating+undersized+degraded+remapped, 4 active+undersized+degraded+remapped+backfill_wait, 5 remapped+peering, 7 peering, 92 active+clean; 5.4 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 12 KiB/s rd, 46 MiB/s wr, 749 op/s; 27728/264645 objects degraded (10.477%); 321 KiB/s, 2 objects/s recovering 2024-09-21T11:02:45.901 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:45.927 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:45.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:45.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:46.029 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:46.055 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:46.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:46.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:46.156 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:46.184 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:46.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:46.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:46.286 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:46.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:46.326 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:46.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:46.332 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 48 16 96537 96521 125.663 84.9375 0.00770953 0.00795261 2024-09-21T11:02:46.427 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:46.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:46.463 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:46.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:46.564 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:46.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:46.596 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:46.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:46.697 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:46.732 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:46.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:46.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:46.737 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7, 4] out_osds: [2, 5, 0] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 7, 6] 2024-09-21T11:02:46.737 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T11:02:46.737 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_5 2024-09-21T11:02:46.737 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T11:02:46.768 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T11:02:46.834 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:46.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:46.882 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:46.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:46.983 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:47.018 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:47.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:47.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:47.120 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:47.153 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:47.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:47.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:47.256 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:47.298 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:47.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:47.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:47.332 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 49 16 97829 97813 124.746 80.75 0.0244427 0.00801218 2024-09-21T11:02:47.400 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:47.425 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:47.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:47.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:47.526 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:47.568 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:47.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:47.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:47.670 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:47.696 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:47.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:47.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:47.798 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:47.831 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:47.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:47.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:47.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:47 smithi160 ceph-mon[101879]: pgmap v3109: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 5.8 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 9.6 KiB/s rd, 88 MiB/s wr, 1.42k op/s; 58579/284427 objects degraded (20.595%); 1.2 MiB/s, 17 objects/s recovering 2024-09-21T11:02:47.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:47 smithi070 ceph-mon[107385]: pgmap v3109: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 5.8 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 9.6 KiB/s rd, 88 MiB/s wr, 1.42k op/s; 58579/284427 objects degraded (20.595%); 1.2 MiB/s, 17 objects/s recovering 2024-09-21T11:02:47.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:47 smithi070 ceph-mon[112139]: pgmap v3109: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 5.8 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 9.6 KiB/s rd, 88 MiB/s wr, 1.42k op/s; 58579/284427 objects degraded (20.595%); 1.2 MiB/s, 17 objects/s recovering 2024-09-21T11:02:47.933 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:47.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:47.963 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:47.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:48.064 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:48.092 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:48.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:48.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:48.194 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:48.230 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:48.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:48.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:48.331 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:48.332 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 50 16 98898 98882 123.587 66.8125 0.00574695 0.00808607 2024-09-21T11:02:48.362 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:48.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:48.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:48.464 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:48.494 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:48.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:48.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:48.596 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:48.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:48.626 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:48.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:48 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 58579/284427 objects degraded (20.595%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:02:48.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:48 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 58579/284427 objects degraded (20.595%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:02:48.727 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:48.754 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:48.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:48.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:48 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 58579/284427 objects degraded (20.595%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:02:48.856 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:48.886 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:48.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:48.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:48.988 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:49.022 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:49.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:49.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:49.124 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:02:49.162 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:49.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:49.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:02:49.264 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:49.300 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:49.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:49.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:49.332 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 51 16 100084 100068 122.617 74.125 0.0227731 0.00815047 2024-09-21T11:02:49.334 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T11:02:49.401 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:49.434 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:49.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:49.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:49.535 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:49.562 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:49.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:49.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:49.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:49 smithi070 ceph-mon[112139]: pgmap v3110: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 5.8 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 8.5 KiB/s rd, 65 MiB/s wr, 1.05k op/s; 58579/284427 objects degraded (20.595%); 1.1 MiB/s, 15 objects/s recovering 2024-09-21T11:02:49.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:49 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T11:02:49.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:49 smithi070 ceph-mon[107385]: pgmap v3110: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 5.8 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 8.5 KiB/s rd, 65 MiB/s wr, 1.05k op/s; 58579/284427 objects degraded (20.595%); 1.1 MiB/s, 15 objects/s recovering 2024-09-21T11:02:49.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:49 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T11:02:49.664 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:49.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:49.709 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:49.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:49.810 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:49.834 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:49.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:49.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:49 smithi160 ceph-mon[101879]: pgmap v3110: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 5.8 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 8.5 KiB/s rd, 65 MiB/s wr, 1.05k op/s; 58579/284427 objects degraded (20.595%); 1.1 MiB/s, 15 objects/s recovering 2024-09-21T11:02:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:49 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T11:02:49.936 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:49.963 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:49.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:49.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:50.064 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:50.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:50.123 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:50.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:50.224 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:50.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:50.272 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:50.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:50.332 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 52 16 101150 101134 121.54 66.625 0.0160914 0.00822342 2024-09-21T11:02:50.374 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:50.402 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:50.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:50.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:50.506 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:50.534 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:50.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:50.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:50.636 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:50.659 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:50.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:50.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:50.677 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:02:50.678 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:02:50.761 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:50.789 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:50.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:50.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:50.890 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:50.918 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:02:50.918 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:02:50.918 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:02:50.918 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:02:50.919 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:50.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:50.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:51.010 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:02:51.010 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:02:51.010 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:02:51.010 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:02:51.010 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:02:51.010 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:02:51.021 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:51.059 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:51.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:51.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:51.160 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:51.191 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:51.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:51.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:51.293 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:51.297 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:02:51.310 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:02:51.310 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:02:51.332 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 53 16 102340 102324 120.651 74.375 0.0205588 0.00828366 2024-09-21T11:02:51.348 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:51.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:51.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:51.359 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:02:51.359 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:02:51.379 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:02:51.393 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:02:51.393 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:02:51.421 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:02:51.421 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:02:51.446 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:02:51.446 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:02:51.447 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T11:02:51.450 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:51.475 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T11:02:51.476 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T11:02:51.483 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:51.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:51.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:51.515 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:02:51.515 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:02:51.585 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:51.622 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:51.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:51.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:51.724 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:51.749 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:51.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:51.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:51.851 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:51.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:51 smithi070 ceph-mon[107385]: pgmap v3111: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.0 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 7.5 KiB/s rd, 73 MiB/s wr, 1.18k op/s; 60397/293757 objects degraded (20.560%); 1.8 MiB/s, 26 objects/s recovering 2024-09-21T11:02:51.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:51 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2844038361' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:02:51.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:51 smithi070 ceph-mon[112139]: pgmap v3111: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.0 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 7.5 KiB/s rd, 73 MiB/s wr, 1.18k op/s; 60397/293757 objects degraded (20.560%); 1.8 MiB/s, 26 objects/s recovering 2024-09-21T11:02:51.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:51 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2844038361' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:02:51.879 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:51.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:51.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:51.981 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:52.020 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:52.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:52.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:52.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:51 smithi160 ceph-mon[101879]: pgmap v3111: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.0 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 7.5 KiB/s rd, 73 MiB/s wr, 1.18k op/s; 60397/293757 objects degraded (20.560%); 1.8 MiB/s, 26 objects/s recovering 2024-09-21T11:02:52.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:51 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2844038361' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:02:52.122 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:52.147 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:52.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:52.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:52.249 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:52.275 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:52.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:52.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:52.332 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 54 16 103565 103549 119.834 76.5625 0.00420138 0.00833987 2024-09-21T11:02:52.377 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:52.427 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:52.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:52.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:52.529 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:52.555 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:52.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:52.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:52.656 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:52.685 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:52.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:52.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:52.787 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:52.814 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:52.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:52.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:52.916 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:52.943 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:52.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:52.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:53.044 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:53.071 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:53.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:53.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:53.173 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:02:53.207 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:53.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:53.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:02:53.309 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:53.332 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 55 16 104732 104716 118.981 72.9375 0.00740895 0.00840006 2024-09-21T11:02:53.337 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:53.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:53.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:53.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:02:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:02:53] "GET /metrics HTTP/1.1" 200 36585 "" "Prometheus/2.51.0" 2024-09-21T11:02:53.438 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:53.468 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:53.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:53.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:53.558 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T11:02:53.558 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T11:02:53.570 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:53.599 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:53.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:53.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:53.701 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:53.729 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:53.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:53.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:53.775 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:53 smithi070 ceph-mon[107385]: pgmap v3112: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.2 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 79 MiB/s wr, 1.27k op/s; 61895/302745 objects degraded (20.445%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T11:02:53.775 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:53 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 61895/302745 objects degraded (20.445%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:02:53.775 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:53 smithi070 ceph-mon[112139]: pgmap v3112: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.2 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 79 MiB/s wr, 1.27k op/s; 61895/302745 objects degraded (20.445%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T11:02:53.775 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:53 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 61895/302745 objects degraded (20.445%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:02:53.831 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:53.852 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:53.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:53.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:53.953 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:53.988 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:53.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:53.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:54.090 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:02:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:53 smithi160 ceph-mon[101879]: pgmap v3112: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.2 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 79 MiB/s wr, 1.27k op/s; 61895/302745 objects degraded (20.445%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T11:02:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:53 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 61895/302745 objects degraded (20.445%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:02:54.141 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:54.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:54.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:02:54.243 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:54.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:54.298 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:54.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:54.306 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:02:54.333 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 56 16 105886 105870 118.143 72.125 0.00485385 0.0084593 2024-09-21T11:02:54.400 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:02:54.424 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:54.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:54.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:02:54.526 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:02:54.561 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:54.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:54.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:02:54.663 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:54.694 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:54.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:54.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:54.796 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:54.856 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:54.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:54.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:54.958 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:54.984 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:54.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:54.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:55.086 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:55.112 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:55.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:55.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:55.214 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:55.239 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:55.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:55.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:55.334 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 57 16 107022 107006 117.316 71 0.00708771 0.00851903 2024-09-21T11:02:55.342 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:55.368 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:55.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:55.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:55.470 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:02:55.512 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:55.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:55.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:02:55.559 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T11:02:55.613 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:55.643 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:55.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:55.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:55.744 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:55.772 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:55.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:55.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:55.874 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:02:55.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:55 smithi070 ceph-mon[107385]: pgmap v3113: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.2 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 67 MiB/s wr, 1.08k op/s; 61895/302745 objects degraded (20.445%); 1.3 MiB/s, 21 objects/s recovering 2024-09-21T11:02:55.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:55 smithi070 ceph-mon[112139]: pgmap v3113: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.2 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 67 MiB/s wr, 1.08k op/s; 61895/302745 objects degraded (20.445%); 1.3 MiB/s, 21 objects/s recovering 2024-09-21T11:02:55.920 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:55.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:55.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:02:56.022 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:56.043 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:56.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:56.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:56.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:55 smithi160 ceph-mon[101879]: pgmap v3113: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.2 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 67 MiB/s wr, 1.08k op/s; 61895/302745 objects degraded (20.445%); 1.3 MiB/s, 21 objects/s recovering 2024-09-21T11:02:56.145 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:56.181 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:56.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:56.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:56.282 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:56.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:56.320 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:56.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:56.333 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 58 16 107940 107924 116.282 57.375 0.00627836 0.00859393 2024-09-21T11:02:56.421 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:56.447 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:56.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:56.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:56.549 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:02:56.582 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:56.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:56.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:02:56.684 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:56.701 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:56 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3788346294' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:02:56.721 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:56.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:56.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:56.815 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T11:02:56.822 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:56.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:56.857 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:56.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:56.958 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:57.001 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:57.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:57.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:57.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:56 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3788346294' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:02:57.103 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:57.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:56 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3788346294' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:02:57.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:57.145 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:57.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:57.246 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:57.278 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:57.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:57.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:57.334 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 59 16 108975 108959 115.408 64.6875 0.00910816 0.00865992 2024-09-21T11:02:57.380 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:57.400 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:57.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:57.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:57.502 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:57.530 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:57.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:57.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:57.631 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:57.653 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:57.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:57.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:57.754 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:57.775 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:57.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:57.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:57.877 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:57.902 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:57.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:57.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:57.956 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:57 smithi070 ceph-mon[107385]: pgmap v3114: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.5 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 96 MiB/s wr, 1.54k op/s; 64926/320073 objects degraded (20.285%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T11:02:57.957 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:57 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T11:02:57.957 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:57 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3788346294' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T11:02:57.957 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:57 smithi070 ceph-mon[107385]: osdmap e1593: 8 total, 7 up, 5 in 2024-09-21T11:02:57.957 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:57 smithi070 ceph-mon[112139]: pgmap v3114: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.5 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 96 MiB/s wr, 1.54k op/s; 64926/320073 objects degraded (20.285%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T11:02:57.957 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:57 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T11:02:57.957 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:57 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3788346294' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T11:02:57.957 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:57 smithi070 ceph-mon[112139]: osdmap e1593: 8 total, 7 up, 5 in 2024-09-21T11:02:57.959 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:57 smithi160 ceph-mon[101879]: pgmap v3114: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.5 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 96 MiB/s wr, 1.54k op/s; 64926/320073 objects degraded (20.285%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T11:02:57.959 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:57 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T11:02:57.959 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:57 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3788346294' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T11:02:57.959 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:57 smithi160 ceph-mon[101879]: osdmap e1593: 8 total, 7 up, 5 in 2024-09-21T11:02:58.004 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:58.032 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:58.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:58.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:58.134 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:58.165 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:58.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:58.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:58.267 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:02:58.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:58.304 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:58.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:02:58.334 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:2024-09-21T11:02:58.333102+0000 min lat: 0.00100109 max lat: 2.2389 avg lat: 0.00873614 2024-09-21T11:02:58.334 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T11:02:58.334 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 60 16 109857 109841 114.403 55.125 0.00575839 0.00873614 2024-09-21T11:02:58.406 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:58.435 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:58.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:58.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:58.537 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:58.561 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:58.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:58.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:58.662 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:58.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:58.705 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:58.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:58.806 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:02:58.833 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:58.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:58.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:02:58.935 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:58.965 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:58.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:58.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:59.067 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:59.105 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:59.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:59.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:59.207 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:59.245 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:59.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:59.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:02:59.334 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 61 16 111058 111042 113.758 75.0625 0.00591718 0.00878622 2024-09-21T11:02:59.347 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:02:59.384 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:59.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:59.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:02:59.486 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:02:59.515 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:59.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:59.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:02:59.617 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:02:59.650 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:59.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:59.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:02:59.752 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:02:59.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:59 smithi070 ceph-mon[107385]: pgmap v3116: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.5 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 74 MiB/s wr, 1.19k op/s; 64926/320073 objects degraded (20.285%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T11:02:59.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:02:59.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:59 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:02:59.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:02:59 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 64926/320073 objects degraded (20.285%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:02:59.753 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:59 smithi070 ceph-mon[112139]: pgmap v3116: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.5 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 74 MiB/s wr, 1.19k op/s; 64926/320073 objects degraded (20.285%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T11:02:59.753 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:59 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:02:59.753 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:59 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:02:59.753 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:02:59 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 64926/320073 objects degraded (20.285%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:02:59.783 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:02:59.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:59.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:02:59.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:59 smithi160 ceph-mon[101879]: pgmap v3116: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.5 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 74 MiB/s wr, 1.19k op/s; 64926/320073 objects degraded (20.285%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T11:02:59.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:59 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:02:59.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:59 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:02:59.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:02:59 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 64926/320073 objects degraded (20.285%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:02:59.885 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:02:59.913 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:02:59.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:02:59.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:00.015 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:00.047 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:00.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:00.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:00.148 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:00.183 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:00.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:00.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:00.285 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:00.316 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:00.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:00.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:00.334 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 62 16 112391 112375 113.267 83.3125 0.0101384 0.0088246 2024-09-21T11:03:00.417 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:00.454 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:00.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:00.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:00.556 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:00.587 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:00.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:00.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:00.689 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:00.718 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:00.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:00.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:00.820 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:00.850 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:00.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:00.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:00.952 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:00.987 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:00.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:00.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:01.089 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:01.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:01.121 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:01.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:01.204 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T11:03:01.222 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:01.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:01.289 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:01.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:01.334 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 63 16 113681 113665 112.748 80.625 0.0128211 0.00886478 2024-09-21T11:03:01.390 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:01.402 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:03:01.418 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:01.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:01.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:01.505 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T11:03:01.507 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:01 smithi070 ceph-mon[112139]: pgmap v3117: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.7 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 71 MiB/s wr, 1.14k op/s; 66469/327999 objects degraded (20.265%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T11:03:01.508 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:01 smithi070 ceph-mon[107385]: pgmap v3117: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.7 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 71 MiB/s wr, 1.14k op/s; 66469/327999 objects degraded (20.265%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T11:03:01.512 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T11:03:01.519 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:01.551 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:01.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:01.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:01.653 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:01.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:01.693 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:01.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:01.794 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:01.830 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:01.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:01.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:01 smithi160 ceph-mon[101879]: pgmap v3117: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.7 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 71 MiB/s wr, 1.14k op/s; 66469/327999 objects degraded (20.265%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T11:03:01.931 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:01.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:01.973 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:01.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:02.074 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:02.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:02.115 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:02.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:02.216 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:02.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:02.250 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:02.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:02.334 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 64 16 114949 114933 112.225 79.25 0.00912943 0.00890666 2024-09-21T11:03:02.350 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:02.387 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:02.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:02.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:02.489 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:02.515 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:02.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:02.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:02.607 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:02 smithi070 ceph-mon[107385]: from='client.36624 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T11:03:02.607 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:02 smithi070 ceph-mon[112139]: from='client.36624 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T11:03:02.616 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:02.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:02.654 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:02.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:02.755 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:02.787 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:02.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:02.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:02 smithi160 ceph-mon[101879]: from='client.36624 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T11:03:02.889 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:02.916 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:02.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:02.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:02.943 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":3117,"stamp":"2024-09-21T11:03:00.288696+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7163996011,"num_objects":109333,"num_object_clones":0,"num_object_copies":327999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":66469,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109333,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":109391,"num_write_kb":6995964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":597,"num_bytes_recovered":34824187,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70096,"ondisk_log_size":70096,"up":327,"acting":317,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":327,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":22196212,"kb_used_data":16663116,"kb_used_omap":268,"kb_used_meta":5532787,"kb_avail":446529548,"statfs":{"total":479975178240,"available":457246257152,"internally_reserved":0,"allocated":17063030784,"data_stored":17042119888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":274897,"internal_metadata":5665574447},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":748027904,"num_objects":11414,"num_object_clones":0,"num_object_copies":34242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11414,"num_write_kb":730496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":16384000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002904"},"pg_stats":[{"pgid":"58.f","version":"1593'6886","reported_seq":6921,"reported_epoch":1593,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-21T11:02:59.463733+0000","last_change":"2024-09-21T11:02:39.618216+0000","last_active":"2024-09-21T11:02:59.463733+0000","last_peered":"2024-09-21T11:02:59.463733+0000","last_clean":"2024-09-21T11:01:48.463662+0000","last_became_active":"2024-09-21T11:02:39.615272+0000","last_became_peered":"2024-09-21T11:02:39.615272+0000","last_unstale":"2024-09-21T11:02:59.463733+0000","last_undegraded":"2024-09-21T11:02:39.478968+0000","last_fullsized":"2024-09-21T11:02:39.458646+0000","mapping_epoch":1590,"log_start":"1585'3100","ondisk_log_start":"1585'3100","created":1581,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":3786,"log_dups_size":0,"ondisk_log_size":3786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451280896,"num_objects":6886,"num_object_clones":0,"num_object_copies":20658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6886,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6886,"num_write_kb":440704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":6886}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"58.e","version":"1593'6874","reported_seq":6908,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:59.487100+0000","last_change":"2024-09-21T11:01:49.479393+0000","last_active":"2024-09-21T11:02:59.487100+0000","last_peered":"2024-09-21T11:02:59.487100+0000","last_clean":"2024-09-21T11:02:59.487100+0000","last_became_active":"2024-09-21T11:01:49.479029+0000","last_became_peered":"2024-09-21T11:01:49.479029+0000","last_unstale":"2024-09-21T11:02:59.487100+0000","last_undegraded":"2024-09-21T11:02:59.487100+0000","last_fullsized":"2024-09-21T11:02:59.487100+0000","mapping_epoch":1581,"log_start":"1585'3100","ondisk_log_start":"1585'3100","created":1581,"last_epoch_clean":1582,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":3774,"log_dups_size":0,"ondisk_log_size":3774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:13:45.129221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450494464,"num_objects":6874,"num_object_clones":0,"num_object_copies":20622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6874,"num_write_kb":439936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"58.d","version":"1593'6959","reported_seq":6999,"reported_epoch":1593,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-21T11:02:59.475817+0000","last_change":"2024-09-21T11:02:39.629311+0000","last_active":"2024-09-21T11:02:59.475817+0000","last_peered":"2024-09-21T11:02:59.475817+0000","last_clean":"2024-09-21T11:01:48.463662+0000","last_became_active":"2024-09-21T11:02:39.610829+0000","last_became_peered":"2024-09-21T11:02:39.610829+0000","last_unstale":"2024-09-21T11:02:59.475817+0000","last_undegraded":"2024-09-21T11:02:39.629247+0000","last_fullsized":"2024-09-21T11:02:39.459532+0000","mapping_epoch":1590,"log_start":"1585'3200","ondisk_log_start":"1585'3200","created":1581,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":3759,"log_dups_size":0,"ondisk_log_size":3759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456065024,"num_objects":6959,"num_object_clones":0,"num_object_copies":20877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6959,"num_write_kb":445376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":6959}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"58.c","version":"1592'6764","reported_seq":6788,"reported_epoch":1592,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-21T11:02:55.573546+0000","last_change":"2024-09-21T11:02:39.614952+0000","last_active":"2024-09-21T11:02:55.573546+0000","last_peered":"2024-09-21T11:02:55.573546+0000","last_clean":"2024-09-21T11:01:48.463662+0000","last_became_active":"2024-09-21T11:02:39.605824+0000","last_became_peered":"2024-09-21T11:02:39.605824+0000","last_unstale":"2024-09-21T11:02:55.573546+0000","last_undegraded":"2024-09-21T11:02:39.496172+0000","last_fullsized":"2024-09-21T11:02:39.468015+0000","mapping_epoch":1590,"log_start":"1585'3000","ondisk_log_start":"1585'3000","created":1581,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":3764,"log_dups_size":0,"ondisk_log_size":3764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:38:37.251818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443285504,"num_objects":6764,"num_object_clones":0,"num_object_copies":20292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6764,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6764,"num_write_kb":432896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":6764}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"58.b","version":"1593'7056","reported_seq":7114,"reported_epoch":1593,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-21T11:02:59.431353+0000","last_change":"2024-09-21T11:02:39.713751+0000","last_active":"2024-09-21T11:02:59.431353+0000","last_peered":"2024-09-21T11:02:59.431353+0000","last_clean":"2024-09-21T11:01:48.463662+0000","last_became_active":"2024-09-21T11:02:39.572737+0000","last_became_peered":"2024-09-21T11:02:39.572737+0000","last_unstale":"2024-09-21T11:02:59.431353+0000","last_undegraded":"2024-09-21T11:02:39.478953+0000","last_fullsized":"2024-09-21T11:02:39.457970+0000","mapping_epoch":1590,"log_start":"1585'3300","ondisk_log_start":"1585'3300","created":1581,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":3756,"log_dups_size":0,"ondisk_log_size":3756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462422016,"num_objects":7056,"num_object_clones":0,"num_object_copies":21168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7056,"num_write_kb":451584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":524288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":7056}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"58.a","version":"1592'6745","reported_seq":6810,"reported_epoch":1592,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-21T11:02:56.000041+0000","last_change":"2024-09-21T11:02:39.669134+0000","last_active":"2024-09-21T11:02:56.000041+0000","last_peered":"2024-09-21T11:02:56.000041+0000","last_clean":"2024-09-21T11:01:48.463662+0000","last_became_active":"2024-09-21T11:02:39.543596+0000","last_became_peered":"2024-09-21T11:02:39.543596+0000","last_unstale":"2024-09-21T11:02:56.000041+0000","last_undegraded":"2024-09-21T11:02:39.557324+0000","last_fullsized":"2024-09-21T11:02:39.456546+0000","mapping_epoch":1590,"log_start":"1585'2900","ondisk_log_start":"1585'2900","created":1581,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":3845,"log_dups_size":0,"ondisk_log_size":3845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:39:32.983212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442040320,"num_objects":6745,"num_object_clones":0,"num_object_copies":20235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6733,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6745,"num_write_kb":431680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":851968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":6745}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"58.9","version":"1593'6951","reported_seq":7966,"reported_epoch":1593,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T11:02:59.672914+0000","last_change":"2024-09-21T11:02:39.727129+0000","last_active":"2024-09-21T11:02:59.672914+0000","last_peered":"2024-09-21T11:02:59.672914+0000","last_clean":"2024-09-21T11:01:48.463662+0000","last_became_active":"2024-09-21T11:02:39.665209+0000","last_became_peered":"2024-09-21T11:02:39.665209+0000","last_unstale":"2024-09-21T11:02:59.672914+0000","last_undegraded":"2024-09-21T11:02:39.484464+0000","last_fullsized":"2024-09-21T11:02:39.456055+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":1581,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":6951,"log_dups_size":0,"ondisk_log_size":6951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455540736,"num_objects":6951,"num_object_clones":0,"num_object_copies":20853,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6951,"num_write_kb":444864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":490,"num_bytes_recovered":32112640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1988}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"58.8","version":"1593'6887","reported_seq":6925,"reported_epoch":1593,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-21T11:02:59.468586+0000","last_change":"2024-09-21T11:02:39.632259+0000","last_active":"2024-09-21T11:02:59.468586+0000","last_peered":"2024-09-21T11:02:59.468586+0000","last_clean":"2024-09-21T11:01:48.463662+0000","last_became_active":"2024-09-21T11:02:39.592999+0000","last_became_peered":"2024-09-21T11:02:39.592999+0000","last_unstale":"2024-09-21T11:02:59.468586+0000","last_undegraded":"2024-09-21T11:02:39.480700+0000","last_fullsized":"2024-09-21T11:02:39.463570+0000","mapping_epoch":1590,"log_start":"1585'3100","ondisk_log_start":"1585'3100","created":1581,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":3787,"log_dups_size":0,"ondisk_log_size":3787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451346432,"num_objects":6887,"num_object_clones":0,"num_object_copies":20661,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6887,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6887,"num_write_kb":440768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":6887}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"58.7","version":"1592'6662","reported_seq":6689,"reported_epoch":1592,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-21T11:02:55.564002+0000","last_change":"2024-09-21T11:02:39.635951+0000","last_active":"2024-09-21T11:02:55.564002+0000","last_peered":"2024-09-21T11:02:55.564002+0000","last_clean":"2024-09-21T11:01:48.463662+0000","last_became_active":"2024-09-21T11:02:39.616773+0000","last_became_peered":"2024-09-21T11:02:39.616773+0000","last_unstale":"2024-09-21T11:02:55.564002+0000","last_undegraded":"2024-09-21T11:02:39.635856+0000","last_fullsized":"2024-09-21T11:02:39.466392+0000","mapping_epoch":1590,"log_start":"1585'2900","ondisk_log_start":"1585'2900","created":1581,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":3762,"log_dups_size":0,"ondisk_log_size":3762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:33.190931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436600832,"num_objects":6662,"num_object_clones":0,"num_object_copies":19986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6662,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6662,"num_write_kb":426368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":6662}],"blocked_by":[],"up_primary":6,"acting_primary":4,"purged_snaps":[]},{"pgid":"58.6","version":"1592'6744","reported_seq":6764,"reported_epoch":1592,"state":"active+clean","last_fresh":"2024-09-21T11:02:55.539162+0000","last_change":"2024-09-21T11:01:49.480589+0000","last_active":"2024-09-21T11:02:55.539162+0000","last_peered":"2024-09-21T11:02:55.539162+0000","last_clean":"2024-09-21T11:02:55.539162+0000","last_became_active":"2024-09-21T11:01:49.479800+0000","last_became_peered":"2024-09-21T11:01:49.479800+0000","last_unstale":"2024-09-21T11:02:55.539162+0000","last_undegraded":"2024-09-21T11:02:55.539162+0000","last_fullsized":"2024-09-21T11:02:55.539162+0000","mapping_epoch":1581,"log_start":"1585'2900","ondisk_log_start":"1585'2900","created":1581,"last_epoch_clean":1582,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":3844,"log_dups_size":0,"ondisk_log_size":3844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:40.465682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":441974784,"num_objects":6744,"num_object_clones":0,"num_object_copies":20232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6744,"num_write_kb":431616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"58.5","version":"1592'6757","reported_seq":6777,"reported_epoch":1592,"state":"active+clean","last_fresh":"2024-09-21T11:02:55.569188+0000","last_change":"2024-09-21T11:01:49.481001+0000","last_active":"2024-09-21T11:02:55.569188+0000","last_peered":"2024-09-21T11:02:55.569188+0000","last_clean":"2024-09-21T11:02:55.569188+0000","last_became_active":"2024-09-21T11:01:49.480685+0000","last_became_peered":"2024-09-21T11:01:49.480685+0000","last_unstale":"2024-09-21T11:02:55.569188+0000","last_undegraded":"2024-09-21T11:02:55.569188+0000","last_fullsized":"2024-09-21T11:02:55.569188+0000","mapping_epoch":1581,"log_start":"1585'3000","ondisk_log_start":"1585'3000","created":1581,"last_epoch_clean":1582,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":3757,"log_dups_size":0,"ondisk_log_size":3757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:04.173095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442826752,"num_objects":6757,"num_object_clones":0,"num_object_copies":20271,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6757,"num_write_kb":432448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"58.4","version":"1593'6949","reported_seq":6984,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:59.686421+0000","last_change":"2024-09-21T11:01:49.481522+0000","last_active":"2024-09-21T11:02:59.686421+0000","last_peered":"2024-09-21T11:02:59.686421+0000","last_clean":"2024-09-21T11:02:59.686421+0000","last_became_active":"2024-09-21T11:01:49.481200+0000","last_became_peered":"2024-09-21T11:01:49.481200+0000","last_unstale":"2024-09-21T11:02:59.686421+0000","last_undegraded":"2024-09-21T11:02:59.686421+0000","last_fullsized":"2024-09-21T11:02:59.686421+0000","mapping_epoch":1581,"log_start":"0'0","ondisk_log_start":"0'0","created":1581,"last_epoch_clean":1582,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":6949,"log_dups_size":0,"ondisk_log_size":6949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:45.593284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455409664,"num_objects":6949,"num_object_clones":0,"num_object_copies":20847,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6949,"num_write_kb":444736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"58.3","version":"1592'6690","reported_seq":6718,"reported_epoch":1592,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-21T11:02:55.547432+0000","last_change":"2024-09-21T11:02:39.623114+0000","last_active":"2024-09-21T11:02:55.547432+0000","last_peered":"2024-09-21T11:02:55.547432+0000","last_clean":"2024-09-21T11:01:48.463662+0000","last_became_active":"2024-09-21T11:02:39.572646+0000","last_became_peered":"2024-09-21T11:02:39.572646+0000","last_unstale":"2024-09-21T11:02:55.547432+0000","last_undegraded":"2024-09-21T11:02:39.491133+0000","last_fullsized":"2024-09-21T11:02:39.464218+0000","mapping_epoch":1590,"log_start":"1585'2900","ondisk_log_start":"1585'2900","created":1581,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":3790,"log_dups_size":0,"ondisk_log_size":3790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:52:36.727823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438435840,"num_objects":6690,"num_object_clones":0,"num_object_copies":20070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6690,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6690,"num_write_kb":428160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":6690}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"58.2","version":"1592'6689","reported_seq":6708,"reported_epoch":1592,"state":"active+clean","last_fresh":"2024-09-21T11:02:55.540864+0000","last_change":"2024-09-21T11:01:49.479201+0000","last_active":"2024-09-21T11:02:55.540864+0000","last_peered":"2024-09-21T11:02:55.540864+0000","last_clean":"2024-09-21T11:02:55.540864+0000","last_became_active":"2024-09-21T11:01:49.478956+0000","last_became_peered":"2024-09-21T11:01:49.478956+0000","last_unstale":"2024-09-21T11:02:55.540864+0000","last_undegraded":"2024-09-21T11:02:55.540864+0000","last_fullsized":"2024-09-21T11:02:55.540864+0000","mapping_epoch":1581,"log_start":"1585'2900","ondisk_log_start":"1585'2900","created":1581,"last_epoch_clean":1582,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":3789,"log_dups_size":0,"ondisk_log_size":3789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:13:56.307307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438370304,"num_objects":6689,"num_object_clones":0,"num_object_copies":20067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6689,"num_write_kb":428096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"58.1","version":"1593'6877","reported_seq":6915,"reported_epoch":1593,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-21T11:02:59.455748+0000","last_change":"2024-09-21T11:02:39.678928+0000","last_active":"2024-09-21T11:02:59.455748+0000","last_peered":"2024-09-21T11:02:59.455748+0000","last_clean":"2024-09-21T11:01:48.463662+0000","last_became_active":"2024-09-21T11:02:39.657244+0000","last_became_peered":"2024-09-21T11:02:39.657244+0000","last_unstale":"2024-09-21T11:02:59.455748+0000","last_undegraded":"2024-09-21T11:02:39.678821+0000","last_fullsized":"2024-09-21T11:02:39.460322+0000","mapping_epoch":1590,"log_start":"1585'3100","ondisk_log_start":"1585'3100","created":1581,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":3777,"log_dups_size":0,"ondisk_log_size":3777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450691072,"num_objects":6877,"num_object_clones":0,"num_object_copies":20631,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6877,"num_write_kb":440128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":6877}],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[]},{"pgid":"58.0","version":"1592'6818","reported_seq":6838,"reported_epoch":1592,"state":"active+clean","last_fresh":"2024-09-21T11:02:55.557783+0000","last_change":"2024-09-21T11:01:49.479198+0000","last_active":"2024-09-21T11:02:55.557783+0000","last_peered":"2024-09-21T11:02:55.557783+0000","last_clean":"2024-09-21T11:02:55.557783+0000","last_became_active":"2024-09-21T11:01:49.478919+0000","last_became_peered":"2024-09-21T11:01:49.478919+0000","last_unstale":"2024-09-21T11:02:55.557783+0000","last_undegraded":"2024-09-21T11:02:55.557783+0000","last_fullsized":"2024-09-21T11:02:55.557783+0000","mapping_epoch":1581,"log_start":"1585'3000","ondisk_log_start":"1585'3000","created":1581,"last_epoch_clean":1582,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":3818,"log_dups_size":0,"ondisk_log_size":3818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:28:28.922307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446824448,"num_objects":6818,"num_object_clones":0,"num_object_copies":20454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6818,"num_write_kb":436352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":2081,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.616304+0000","last_change":"2024-09-21T11:02:39.554122+0000","last_active":"2024-09-21T11:02:39.616304+0000","last_peered":"2024-09-21T11:02:39.616304+0000","last_clean":"2024-09-21T11:02:39.616304+0000","last_became_active":"2024-09-21T11:02:39.553373+0000","last_became_peered":"2024-09-21T11:02:39.553373+0000","last_unstale":"2024-09-21T11:02:39.616304+0000","last_undegraded":"2024-09-21T11:02:39.616304+0000","last_fullsized":"2024-09-21T11:02:39.616304+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:50.623242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1508,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.488573+0000","last_change":"2024-09-21T11:02:39.486670+0000","last_active":"2024-09-21T11:02:39.488573+0000","last_peered":"2024-09-21T11:02:39.488573+0000","last_clean":"2024-09-21T11:02:39.488573+0000","last_became_active":"2024-09-21T11:02:39.486248+0000","last_became_peered":"2024-09-21T11:02:39.486248+0000","last_unstale":"2024-09-21T11:02:39.488573+0000","last_undegraded":"2024-09-21T11:02:39.488573+0000","last_fullsized":"2024-09-21T11:02:39.488573+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:56:56.360708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":2251,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.460869+0000","last_change":"2024-09-21T11:02:38.484690+0000","last_active":"2024-09-21T11:02:39.460869+0000","last_peered":"2024-09-21T11:02:39.460869+0000","last_clean":"2024-09-21T11:02:39.460869+0000","last_became_active":"2024-09-21T11:02:38.484205+0000","last_became_peered":"2024-09-21T11:02:38.484205+0000","last_unstale":"2024-09-21T11:02:39.460869+0000","last_undegraded":"2024-09-21T11:02:39.460869+0000","last_fullsized":"2024-09-21T11:02:39.460869+0000","mapping_epoch":1589,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:51:46.102813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2425,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:56.721007+0000","last_change":"2024-09-21T11:02:39.488465+0000","last_active":"2024-09-21T11:02:56.721007+0000","last_peered":"2024-09-21T11:02:56.721007+0000","last_clean":"2024-09-21T11:02:56.721007+0000","last_became_active":"2024-09-21T11:02:39.488033+0000","last_became_peered":"2024-09-21T11:02:39.488033+0000","last_unstale":"2024-09-21T11:02:56.721007+0000","last_undegraded":"2024-09-21T11:02:56.721007+0000","last_fullsized":"2024-09-21T11:02:56.721007+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:16:30.263493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":86,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.506051+0000","last_change":"2024-09-21T11:02:39.502513+0000","last_active":"2024-09-21T11:02:39.506051+0000","last_peered":"2024-09-21T11:02:39.506051+0000","last_clean":"2024-09-21T11:02:39.506051+0000","last_became_active":"2024-09-21T11:02:39.502095+0000","last_became_peered":"2024-09-21T11:02:39.502095+0000","last_unstale":"2024-09-21T11:02:39.506051+0000","last_undegraded":"2024-09-21T11:02:39.506051+0000","last_fullsized":"2024-09-21T11:02:39.506051+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:50:46.078094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1555,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.483173+0000","last_change":"2024-09-21T11:02:39.481489+0000","last_active":"2024-09-21T11:02:39.483173+0000","last_peered":"2024-09-21T11:02:39.483173+0000","last_clean":"2024-09-21T11:02:39.483173+0000","last_became_active":"2024-09-21T11:02:39.481160+0000","last_became_peered":"2024-09-21T11:02:39.481160+0000","last_unstale":"2024-09-21T11:02:39.483173+0000","last_undegraded":"2024-09-21T11:02:39.483173+0000","last_fullsized":"2024-09-21T11:02:39.483173+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:28:36.699031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":86,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.506871+0000","last_change":"2024-09-21T11:02:39.502786+0000","last_active":"2024-09-21T11:02:39.506871+0000","last_peered":"2024-09-21T11:02:39.506871+0000","last_clean":"2024-09-21T11:02:39.506871+0000","last_became_active":"2024-09-21T11:02:39.498657+0000","last_became_peered":"2024-09-21T11:02:39.498657+0000","last_unstale":"2024-09-21T11:02:39.506871+0000","last_undegraded":"2024-09-21T11:02:39.506871+0000","last_fullsized":"2024-09-21T11:02:39.506871+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:30:40.747863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.465666+0000","last_change":"2024-09-21T11:01:46.490333+0000","last_active":"2024-09-21T11:02:39.465666+0000","last_peered":"2024-09-21T11:02:39.465666+0000","last_clean":"2024-09-21T11:02:39.465666+0000","last_became_active":"2024-09-21T11:01:46.490016+0000","last_became_peered":"2024-09-21T11:01:46.490016+0000","last_unstale":"2024-09-21T11:02:39.465666+0000","last_undegraded":"2024-09-21T11:02:39.465666+0000","last_fullsized":"2024-09-21T11:02:39.465666+0000","mapping_epoch":1578,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:47:34.742065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":66,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.467260+0000","last_change":"2024-09-21T11:01:47.475008+0000","last_active":"2024-09-21T11:02:39.467260+0000","last_peered":"2024-09-21T11:02:39.467260+0000","last_clean":"2024-09-21T11:02:39.467260+0000","last_became_active":"2024-09-21T11:01:47.474527+0000","last_became_peered":"2024-09-21T11:01:47.474527+0000","last_unstale":"2024-09-21T11:02:39.467260+0000","last_undegraded":"2024-09-21T11:02:39.467260+0000","last_fullsized":"2024-09-21T11:02:39.467260+0000","mapping_epoch":1579,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:01.364936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1304,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:56.719385+0000","last_change":"2024-09-21T11:01:47.466519+0000","last_active":"2024-09-21T11:02:56.719385+0000","last_peered":"2024-09-21T11:02:56.719385+0000","last_clean":"2024-09-21T11:02:56.719385+0000","last_became_active":"2024-09-21T11:01:47.465610+0000","last_became_peered":"2024-09-21T11:01:47.465610+0000","last_unstale":"2024-09-21T11:02:56.719385+0000","last_undegraded":"2024-09-21T11:02:56.719385+0000","last_fullsized":"2024-09-21T11:02:56.719385+0000","mapping_epoch":1579,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:15:11.771345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":66,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.464945+0000","last_change":"2024-09-21T11:01:47.471982+0000","last_active":"2024-09-21T11:02:39.464945+0000","last_peered":"2024-09-21T11:02:39.464945+0000","last_clean":"2024-09-21T11:02:39.464945+0000","last_became_active":"2024-09-21T11:01:47.471491+0000","last_became_peered":"2024-09-21T11:01:47.471491+0000","last_unstale":"2024-09-21T11:02:39.464945+0000","last_undegraded":"2024-09-21T11:02:39.464945+0000","last_fullsized":"2024-09-21T11:02:39.464945+0000","mapping_epoch":1579,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:40:19.986790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2501,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:56.721819+0000","last_change":"2024-09-21T11:02:39.489688+0000","last_active":"2024-09-21T11:02:56.721819+0000","last_peered":"2024-09-21T11:02:56.721819+0000","last_clean":"2024-09-21T11:02:56.721819+0000","last_became_active":"2024-09-21T11:02:39.489444+0000","last_became_peered":"2024-09-21T11:02:39.489444+0000","last_unstale":"2024-09-21T11:02:56.721819+0000","last_undegraded":"2024-09-21T11:02:56.721819+0000","last_fullsized":"2024-09-21T11:02:56.721819+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:35:11.865016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2494,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:56.721229+0000","last_change":"2024-09-21T11:01:46.489782+0000","last_active":"2024-09-21T11:02:56.721229+0000","last_peered":"2024-09-21T11:02:56.721229+0000","last_clean":"2024-09-21T11:02:56.721229+0000","last_became_active":"2024-09-21T11:01:46.488938+0000","last_became_peered":"2024-09-21T11:01:46.488938+0000","last_unstale":"2024-09-21T11:02:56.721229+0000","last_undegraded":"2024-09-21T11:02:56.721229+0000","last_fullsized":"2024-09-21T11:02:56.721229+0000","mapping_epoch":1578,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:35:18.021547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":179,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.457672+0000","last_change":"2024-09-21T11:01:46.491530+0000","last_active":"2024-09-21T11:02:39.457672+0000","last_peered":"2024-09-21T11:02:39.457672+0000","last_clean":"2024-09-21T11:02:39.457672+0000","last_became_active":"2024-09-21T11:01:46.491031+0000","last_became_peered":"2024-09-21T11:01:46.491031+0000","last_unstale":"2024-09-21T11:02:39.457672+0000","last_undegraded":"2024-09-21T11:02:39.457672+0000","last_fullsized":"2024-09-21T11:02:39.457672+0000","mapping_epoch":1578,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:53:33.778440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2432,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:56.716593+0000","last_change":"2024-09-21T11:01:46.488739+0000","last_active":"2024-09-21T11:02:56.716593+0000","last_peered":"2024-09-21T11:02:56.716593+0000","last_clean":"2024-09-21T11:02:56.716593+0000","last_became_active":"2024-09-21T11:01:46.488332+0000","last_became_peered":"2024-09-21T11:01:46.488332+0000","last_unstale":"2024-09-21T11:02:56.716593+0000","last_undegraded":"2024-09-21T11:02:56.716593+0000","last_fullsized":"2024-09-21T11:02:56.716593+0000","mapping_epoch":1578,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:03:15.153349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1536,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.460603+0000","last_change":"2024-09-21T11:02:38.481935+0000","last_active":"2024-09-21T11:02:39.460603+0000","last_peered":"2024-09-21T11:02:39.460603+0000","last_clean":"2024-09-21T11:02:39.460603+0000","last_became_active":"2024-09-21T11:02:38.481612+0000","last_became_peered":"2024-09-21T11:02:38.481612+0000","last_unstale":"2024-09-21T11:02:39.460603+0000","last_undegraded":"2024-09-21T11:02:39.460603+0000","last_fullsized":"2024-09-21T11:02:39.460603+0000","mapping_epoch":1589,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:07:24.439039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2435,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:56.716496+0000","last_change":"2024-09-21T11:02:39.481849+0000","last_active":"2024-09-21T11:02:56.716496+0000","last_peered":"2024-09-21T11:02:56.716496+0000","last_clean":"2024-09-21T11:02:56.716496+0000","last_became_active":"2024-09-21T11:02:39.481561+0000","last_became_peered":"2024-09-21T11:02:39.481561+0000","last_unstale":"2024-09-21T11:02:56.716496+0000","last_undegraded":"2024-09-21T11:02:56.716496+0000","last_fullsized":"2024-09-21T11:02:56.716496+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:51:57.860458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2257,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.616399+0000","last_change":"2024-09-21T11:02:39.554294+0000","last_active":"2024-09-21T11:02:39.616399+0000","last_peered":"2024-09-21T11:02:39.616399+0000","last_clean":"2024-09-21T11:02:39.616399+0000","last_became_active":"2024-09-21T11:02:39.553816+0000","last_became_peered":"2024-09-21T11:02:39.553816+0000","last_unstale":"2024-09-21T11:02:39.616399+0000","last_undegraded":"2024-09-21T11:02:39.616399+0000","last_fullsized":"2024-09-21T11:02:39.616399+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:29:21.923052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.14","version":"346'149","reported_seq":1078,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.504143+0000","last_change":"2024-09-21T11:02:39.499824+0000","last_active":"2024-09-21T11:02:39.504143+0000","last_peered":"2024-09-21T11:02:39.504143+0000","last_clean":"2024-09-21T11:02:39.504143+0000","last_became_active":"2024-09-21T11:02:39.499460+0000","last_became_peered":"2024-09-21T11:02:39.499460+0000","last_unstale":"2024-09-21T11:02:39.504143+0000","last_undegraded":"2024-09-21T11:02:39.504143+0000","last_fullsized":"2024-09-21T11:02:39.504143+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:53.724544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:53.724544+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:06:29.289800+0000","scrub_duration":0.0017779029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2512,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:56.727960+0000","last_change":"2024-09-21T11:02:38.470337+0000","last_active":"2024-09-21T11:02:56.727960+0000","last_peered":"2024-09-21T11:02:56.727960+0000","last_clean":"2024-09-21T11:02:56.727960+0000","last_became_active":"2024-09-21T11:02:38.470062+0000","last_became_peered":"2024-09-21T11:02:38.470062+0000","last_unstale":"2024-09-21T11:02:56.727960+0000","last_undegraded":"2024-09-21T11:02:56.727960+0000","last_fullsized":"2024-09-21T11:02:56.727960+0000","mapping_epoch":1589,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:59:04.810652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":178,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.457586+0000","last_change":"2024-09-21T11:01:46.491444+0000","last_active":"2024-09-21T11:02:39.457586+0000","last_peered":"2024-09-21T11:02:39.457586+0000","last_clean":"2024-09-21T11:02:39.457586+0000","last_became_active":"2024-09-21T11:01:46.491025+0000","last_became_peered":"2024-09-21T11:01:46.491025+0000","last_unstale":"2024-09-21T11:02:39.457586+0000","last_undegraded":"2024-09-21T11:02:39.457586+0000","last_fullsized":"2024-09-21T11:02:39.457586+0000","mapping_epoch":1578,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:39:31.407690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":196,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.457464+0000","last_change":"2024-09-21T11:01:46.473450+0000","last_active":"2024-09-21T11:02:39.457464+0000","last_peered":"2024-09-21T11:02:39.457464+0000","last_clean":"2024-09-21T11:02:39.457464+0000","last_became_active":"2024-09-21T11:01:46.473139+0000","last_became_peered":"2024-09-21T11:01:46.473139+0000","last_unstale":"2024-09-21T11:02:39.457464+0000","last_undegraded":"2024-09-21T11:02:39.457464+0000","last_fullsized":"2024-09-21T11:02:39.457464+0000","mapping_epoch":1578,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:04:49.188305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2513,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:56.720445+0000","last_change":"2024-09-21T11:02:38.484420+0000","last_active":"2024-09-21T11:02:56.720445+0000","last_peered":"2024-09-21T11:02:56.720445+0000","last_clean":"2024-09-21T11:02:56.720445+0000","last_became_active":"2024-09-21T11:02:38.484198+0000","last_became_peered":"2024-09-21T11:02:38.484198+0000","last_unstale":"2024-09-21T11:02:56.720445+0000","last_undegraded":"2024-09-21T11:02:56.720445+0000","last_fullsized":"2024-09-21T11:02:56.720445+0000","mapping_epoch":1589,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:38:38.340638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":92,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.509042+0000","last_change":"2024-09-21T11:02:39.505094+0000","last_active":"2024-09-21T11:02:39.509042+0000","last_peered":"2024-09-21T11:02:39.509042+0000","last_clean":"2024-09-21T11:02:39.509042+0000","last_became_active":"2024-09-21T11:02:39.503527+0000","last_became_peered":"2024-09-21T11:02:39.503527+0000","last_unstale":"2024-09-21T11:02:39.509042+0000","last_undegraded":"2024-09-21T11:02:39.509042+0000","last_fullsized":"2024-09-21T11:02:39.509042+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:12:34.916393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14268,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":661,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.502996+0000","last_change":"2024-09-21T11:02:39.492530+0000","last_active":"2024-09-21T11:02:39.502996+0000","last_peered":"2024-09-21T11:02:39.502996+0000","last_clean":"2024-09-21T11:02:39.502996+0000","last_became_active":"2024-09-21T11:02:39.491835+0000","last_became_peered":"2024-09-21T11:02:39.491835+0000","last_unstale":"2024-09-21T11:02:39.502996+0000","last_undegraded":"2024-09-21T11:02:39.502996+0000","last_fullsized":"2024-09-21T11:02:39.502996+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:37.157843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"346'149","reported_seq":1078,"reported_epoch":1592,"state":"active+clean","last_fresh":"2024-09-21T11:02:42.497033+0000","last_change":"2024-09-21T11:02:39.508294+0000","last_active":"2024-09-21T11:02:42.497033+0000","last_peered":"2024-09-21T11:02:42.497033+0000","last_clean":"2024-09-21T11:02:42.497033+0000","last_became_active":"2024-09-21T11:02:39.507981+0000","last_became_peered":"2024-09-21T11:02:39.507981+0000","last_unstale":"2024-09-21T11:02:42.497033+0000","last_undegraded":"2024-09-21T11:02:42.497033+0000","last_fullsized":"2024-09-21T11:02:42.497033+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:59.781270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:59.781270+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:10:14.119049+0000","scrub_duration":0.0031203189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1510,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.481094+0000","last_change":"2024-09-21T11:02:39.479530+0000","last_active":"2024-09-21T11:02:39.481094+0000","last_peered":"2024-09-21T11:02:39.481094+0000","last_clean":"2024-09-21T11:02:39.481094+0000","last_became_active":"2024-09-21T11:02:39.477997+0000","last_became_peered":"2024-09-21T11:02:39.477997+0000","last_unstale":"2024-09-21T11:02:39.481094+0000","last_undegraded":"2024-09-21T11:02:39.481094+0000","last_fullsized":"2024-09-21T11:02:39.481094+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:16:07.615963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1566,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.462814+0000","last_change":"2024-09-21T11:02:38.487978+0000","last_active":"2024-09-21T11:02:39.462814+0000","last_peered":"2024-09-21T11:02:39.462814+0000","last_clean":"2024-09-21T11:02:39.462814+0000","last_became_active":"2024-09-21T11:02:38.485556+0000","last_became_peered":"2024-09-21T11:02:38.485556+0000","last_unstale":"2024-09-21T11:02:39.462814+0000","last_undegraded":"2024-09-21T11:02:39.462814+0000","last_fullsized":"2024-09-21T11:02:39.462814+0000","mapping_epoch":1589,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:15:42.496020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":381,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:56.719669+0000","last_change":"2024-09-21T11:01:47.472195+0000","last_active":"2024-09-21T11:02:56.719669+0000","last_peered":"2024-09-21T11:02:56.719669+0000","last_clean":"2024-09-21T11:02:56.719669+0000","last_became_active":"2024-09-21T11:01:47.471752+0000","last_became_peered":"2024-09-21T11:01:47.471752+0000","last_unstale":"2024-09-21T11:02:56.719669+0000","last_undegraded":"2024-09-21T11:02:56.719669+0000","last_fullsized":"2024-09-21T11:02:56.719669+0000","mapping_epoch":1579,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:02:42.837537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"346'149","reported_seq":749,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.501834+0000","last_change":"2024-09-21T11:02:39.487908+0000","last_active":"2024-09-21T11:02:39.501834+0000","last_peered":"2024-09-21T11:02:39.501834+0000","last_clean":"2024-09-21T11:02:39.501834+0000","last_became_active":"2024-09-21T11:02:39.484101+0000","last_became_peered":"2024-09-21T11:02:39.484101+0000","last_unstale":"2024-09-21T11:02:39.501834+0000","last_undegraded":"2024-09-21T11:02:39.501834+0000","last_fullsized":"2024-09-21T11:02:39.501834+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:54.879276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:54.879276+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:43:23.977500+0000","scrub_duration":0.001843014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2313,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.456650+0000","last_change":"2024-09-21T11:02:38.474189+0000","last_active":"2024-09-21T11:02:39.456650+0000","last_peered":"2024-09-21T11:02:39.456650+0000","last_clean":"2024-09-21T11:02:39.456650+0000","last_became_active":"2024-09-21T11:02:38.473849+0000","last_became_peered":"2024-09-21T11:02:38.473849+0000","last_unstale":"2024-09-21T11:02:39.456650+0000","last_undegraded":"2024-09-21T11:02:39.456650+0000","last_fullsized":"2024-09-21T11:02:39.456650+0000","mapping_epoch":1589,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:19:36.681206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2505,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:56.719888+0000","last_change":"2024-09-21T11:02:39.477625+0000","last_active":"2024-09-21T11:02:56.719888+0000","last_peered":"2024-09-21T11:02:56.719888+0000","last_clean":"2024-09-21T11:02:56.719888+0000","last_became_active":"2024-09-21T11:02:39.476763+0000","last_became_peered":"2024-09-21T11:02:39.476763+0000","last_unstale":"2024-09-21T11:02:56.719888+0000","last_undegraded":"2024-09-21T11:02:56.719888+0000","last_fullsized":"2024-09-21T11:02:56.719888+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:26.214632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":2562,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:56.710023+0000","last_change":"2024-09-21T11:01:46.475454+0000","last_active":"2024-09-21T11:02:56.710023+0000","last_peered":"2024-09-21T11:02:56.710023+0000","last_clean":"2024-09-21T11:02:56.710023+0000","last_became_active":"2024-09-21T11:01:46.471699+0000","last_became_peered":"2024-09-21T11:01:46.471699+0000","last_unstale":"2024-09-21T11:02:56.710023+0000","last_undegraded":"2024-09-21T11:02:56.710023+0000","last_fullsized":"2024-09-21T11:02:56.710023+0000","mapping_epoch":1578,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:02.992002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"346'149","reported_seq":844,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.497571+0000","last_change":"2024-09-21T11:02:39.486915+0000","last_active":"2024-09-21T11:02:39.497571+0000","last_peered":"2024-09-21T11:02:39.497571+0000","last_clean":"2024-09-21T11:02:39.497571+0000","last_became_active":"2024-09-21T11:02:39.486300+0000","last_became_peered":"2024-09-21T11:02:39.486300+0000","last_unstale":"2024-09-21T11:02:39.497571+0000","last_undegraded":"2024-09-21T11:02:39.497571+0000","last_fullsized":"2024-09-21T11:02:39.497571+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:02:02.808006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:02:02.808006+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061250310000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2349,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.457542+0000","last_change":"2024-09-21T11:01:46.491422+0000","last_active":"2024-09-21T11:02:39.457542+0000","last_peered":"2024-09-21T11:02:39.457542+0000","last_clean":"2024-09-21T11:02:39.457542+0000","last_became_active":"2024-09-21T11:01:46.490992+0000","last_became_peered":"2024-09-21T11:01:46.490992+0000","last_unstale":"2024-09-21T11:02:39.457542+0000","last_undegraded":"2024-09-21T11:02:39.457542+0000","last_fullsized":"2024-09-21T11:02:39.457542+0000","mapping_epoch":1578,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:44:25.185915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":180,"reported_epoch":1592,"state":"active+clean","last_fresh":"2024-09-21T11:02:42.498639+0000","last_change":"2024-09-21T11:01:47.467160+0000","last_active":"2024-09-21T11:02:42.498639+0000","last_peered":"2024-09-21T11:02:42.498639+0000","last_clean":"2024-09-21T11:02:42.498639+0000","last_became_active":"2024-09-21T11:01:47.466714+0000","last_became_peered":"2024-09-21T11:01:47.466714+0000","last_unstale":"2024-09-21T11:02:42.498639+0000","last_undegraded":"2024-09-21T11:02:42.498639+0000","last_fullsized":"2024-09-21T11:02:42.498639+0000","mapping_epoch":1579,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:06:39.441034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2555,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:56.719283+0000","last_change":"2024-09-21T11:02:39.508042+0000","last_active":"2024-09-21T11:02:56.719283+0000","last_peered":"2024-09-21T11:02:56.719283+0000","last_clean":"2024-09-21T11:02:56.719283+0000","last_became_active":"2024-09-21T11:02:39.507808+0000","last_became_peered":"2024-09-21T11:02:39.507808+0000","last_unstale":"2024-09-21T11:02:56.719283+0000","last_undegraded":"2024-09-21T11:02:56.719283+0000","last_fullsized":"2024-09-21T11:02:56.719283+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:24.454114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"346'149","reported_seq":832,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.461139+0000","last_change":"2024-09-21T11:02:38.488107+0000","last_active":"2024-09-21T11:02:39.461139+0000","last_peered":"2024-09-21T11:02:39.461139+0000","last_clean":"2024-09-21T11:02:39.461139+0000","last_became_active":"2024-09-21T11:02:38.485060+0000","last_became_peered":"2024-09-21T11:02:38.485060+0000","last_unstale":"2024-09-21T11:02:39.461139+0000","last_undegraded":"2024-09-21T11:02:39.461139+0000","last_fullsized":"2024-09-21T11:02:39.461139+0000","mapping_epoch":1589,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1590,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:44.018645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:44.018645+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012789399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2502,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:56.720301+0000","last_change":"2024-09-21T11:01:46.488480+0000","last_active":"2024-09-21T11:02:56.720301+0000","last_peered":"2024-09-21T11:02:56.720301+0000","last_clean":"2024-09-21T11:02:56.720301+0000","last_became_active":"2024-09-21T11:01:46.487911+0000","last_became_peered":"2024-09-21T11:01:46.487911+0000","last_unstale":"2024-09-21T11:02:56.720301+0000","last_undegraded":"2024-09-21T11:02:56.720301+0000","last_fullsized":"2024-09-21T11:02:56.720301+0000","mapping_epoch":1578,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:01:01.299737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2569,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:56.719098+0000","last_change":"2024-09-21T11:01:46.489677+0000","last_active":"2024-09-21T11:02:56.719098+0000","last_peered":"2024-09-21T11:02:56.719098+0000","last_clean":"2024-09-21T11:02:56.719098+0000","last_became_active":"2024-09-21T11:01:46.488890+0000","last_became_peered":"2024-09-21T11:01:46.488890+0000","last_unstale":"2024-09-21T11:02:56.719098+0000","last_undegraded":"2024-09-21T11:02:56.719098+0000","last_fullsized":"2024-09-21T11:02:56.719098+0000","mapping_epoch":1578,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:50:27.887340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1454,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.463595+0000","last_change":"2024-09-21T11:02:38.480308+0000","last_active":"2024-09-21T11:02:39.463595+0000","last_peered":"2024-09-21T11:02:39.463595+0000","last_clean":"2024-09-21T11:02:39.463595+0000","last_became_active":"2024-09-21T11:02:38.480004+0000","last_became_peered":"2024-09-21T11:02:38.480004+0000","last_unstale":"2024-09-21T11:02:39.463595+0000","last_undegraded":"2024-09-21T11:02:39.463595+0000","last_fullsized":"2024-09-21T11:02:39.463595+0000","mapping_epoch":1589,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:40.913800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"346'149","reported_seq":895,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.481387+0000","last_change":"2024-09-21T11:02:39.480945+0000","last_active":"2024-09-21T11:02:39.481387+0000","last_peered":"2024-09-21T11:02:39.481387+0000","last_clean":"2024-09-21T11:02:39.481387+0000","last_became_active":"2024-09-21T11:02:39.479186+0000","last_became_peered":"2024-09-21T11:02:39.479186+0000","last_unstale":"2024-09-21T11:02:39.481387+0000","last_undegraded":"2024-09-21T11:02:39.481387+0000","last_fullsized":"2024-09-21T11:02:39.481387+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:48.918874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.918874+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:57:53.602750+0000","scrub_duration":0.001629134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2567,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:56.719474+0000","last_change":"2024-09-21T11:01:46.489833+0000","last_active":"2024-09-21T11:02:56.719474+0000","last_peered":"2024-09-21T11:02:56.719474+0000","last_clean":"2024-09-21T11:02:56.719474+0000","last_became_active":"2024-09-21T11:01:46.489065+0000","last_became_peered":"2024-09-21T11:01:46.489065+0000","last_unstale":"2024-09-21T11:02:56.719474+0000","last_undegraded":"2024-09-21T11:02:56.719474+0000","last_fullsized":"2024-09-21T11:02:56.719474+0000","mapping_epoch":1578,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:19:16.088782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1490,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.503077+0000","last_change":"2024-09-21T11:02:39.492700+0000","last_active":"2024-09-21T11:02:39.503077+0000","last_peered":"2024-09-21T11:02:39.503077+0000","last_clean":"2024-09-21T11:02:39.503077+0000","last_became_active":"2024-09-21T11:02:39.492243+0000","last_became_peered":"2024-09-21T11:02:39.492243+0000","last_unstale":"2024-09-21T11:02:39.503077+0000","last_undegraded":"2024-09-21T11:02:39.503077+0000","last_fullsized":"2024-09-21T11:02:39.503077+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:25:57.200203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":458,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:56.720148+0000","last_change":"2024-09-21T11:02:38.484988+0000","last_active":"2024-09-21T11:02:56.720148+0000","last_peered":"2024-09-21T11:02:56.720148+0000","last_clean":"2024-09-21T11:02:56.720148+0000","last_became_active":"2024-09-21T11:02:38.484642+0000","last_became_peered":"2024-09-21T11:02:38.484642+0000","last_unstale":"2024-09-21T11:02:56.720148+0000","last_undegraded":"2024-09-21T11:02:56.720148+0000","last_fullsized":"2024-09-21T11:02:56.720148+0000","mapping_epoch":1589,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:22:15.147952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"346'149","reported_seq":1014,"reported_epoch":1592,"state":"active+clean","last_fresh":"2024-09-21T11:02:42.496255+0000","last_change":"2024-09-21T11:01:49.769921+0000","last_active":"2024-09-21T11:02:42.496255+0000","last_peered":"2024-09-21T11:02:42.496255+0000","last_clean":"2024-09-21T11:02:42.496255+0000","last_became_active":"2024-09-21T11:01:47.467143+0000","last_became_peered":"2024-09-21T11:01:47.467143+0000","last_unstale":"2024-09-21T11:02:42.496255+0000","last_undegraded":"2024-09-21T11:02:42.496255+0000","last_fullsized":"2024-09-21T11:02:42.496255+0000","mapping_epoch":1579,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1580,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:49.769829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:49.769829+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:25:51.800563+0000","scrub_duration":0.0014744370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":244,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:56.722343+0000","last_change":"2024-09-21T11:01:46.479390+0000","last_active":"2024-09-21T11:02:56.722343+0000","last_peered":"2024-09-21T11:02:56.722343+0000","last_clean":"2024-09-21T11:02:56.722343+0000","last_became_active":"2024-09-21T11:01:46.473243+0000","last_became_peered":"2024-09-21T11:01:46.473243+0000","last_unstale":"2024-09-21T11:02:56.722343+0000","last_undegraded":"2024-09-21T11:02:56.722343+0000","last_fullsized":"2024-09-21T11:02:56.722343+0000","mapping_epoch":1578,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:17:04.280119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2511,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:56.716688+0000","last_change":"2024-09-21T11:02:38.485141+0000","last_active":"2024-09-21T11:02:56.716688+0000","last_peered":"2024-09-21T11:02:56.716688+0000","last_clean":"2024-09-21T11:02:56.716688+0000","last_became_active":"2024-09-21T11:02:38.484843+0000","last_became_peered":"2024-09-21T11:02:38.484843+0000","last_unstale":"2024-09-21T11:02:56.716688+0000","last_undegraded":"2024-09-21T11:02:56.716688+0000","last_fullsized":"2024-09-21T11:02:56.716688+0000","mapping_epoch":1589,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:39.971742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1188,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:56.721149+0000","last_change":"2024-09-21T11:02:39.488466+0000","last_active":"2024-09-21T11:02:56.721149+0000","last_peered":"2024-09-21T11:02:56.721149+0000","last_clean":"2024-09-21T11:02:56.721149+0000","last_became_active":"2024-09-21T11:02:39.487924+0000","last_became_peered":"2024-09-21T11:02:39.487924+0000","last_unstale":"2024-09-21T11:02:56.721149+0000","last_undegraded":"2024-09-21T11:02:56.721149+0000","last_fullsized":"2024-09-21T11:02:56.721149+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:57:39.804022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"346'149","reported_seq":1218,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:56.719671+0000","last_change":"2024-09-21T11:01:50.671425+0000","last_active":"2024-09-21T11:02:56.719671+0000","last_peered":"2024-09-21T11:02:56.719671+0000","last_clean":"2024-09-21T11:02:56.719671+0000","last_became_active":"2024-09-21T11:01:47.467181+0000","last_became_peered":"2024-09-21T11:01:47.467181+0000","last_unstale":"2024-09-21T11:02:56.719671+0000","last_undegraded":"2024-09-21T11:02:56.719671+0000","last_fullsized":"2024-09-21T11:02:56.719671+0000","mapping_epoch":1579,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1580,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:50.671365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:50.671365+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:31:06.721282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2545,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:56.719579+0000","last_change":"2024-09-21T11:01:46.490895+0000","last_active":"2024-09-21T11:02:56.719579+0000","last_peered":"2024-09-21T11:02:56.719579+0000","last_clean":"2024-09-21T11:02:56.719579+0000","last_became_active":"2024-09-21T11:01:46.490710+0000","last_became_peered":"2024-09-21T11:01:46.490710+0000","last_unstale":"2024-09-21T11:02:56.719579+0000","last_undegraded":"2024-09-21T11:02:56.719579+0000","last_fullsized":"2024-09-21T11:02:56.719579+0000","mapping_epoch":1578,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:06:33.208450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2239,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.464704+0000","last_change":"2024-09-21T11:01:47.474884+0000","last_active":"2024-09-21T11:02:39.464704+0000","last_peered":"2024-09-21T11:02:39.464704+0000","last_clean":"2024-09-21T11:02:39.464704+0000","last_became_active":"2024-09-21T11:01:47.474289+0000","last_became_peered":"2024-09-21T11:01:47.474289+0000","last_unstale":"2024-09-21T11:02:39.464704+0000","last_undegraded":"2024-09-21T11:02:39.464704+0000","last_fullsized":"2024-09-21T11:02:39.464704+0000","mapping_epoch":1579,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:26:11.223644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":251,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.457251+0000","last_change":"2024-09-21T10:51:27.426006+0000","last_active":"2024-09-21T11:02:39.457251+0000","last_peered":"2024-09-21T11:02:39.457251+0000","last_clean":"2024-09-21T11:02:39.457251+0000","last_became_active":"2024-09-21T10:51:27.425698+0000","last_became_peered":"2024-09-21T10:51:27.425698+0000","last_unstale":"2024-09-21T11:02:39.457251+0000","last_undegraded":"2024-09-21T11:02:39.457251+0000","last_fullsized":"2024-09-21T11:02:39.457251+0000","mapping_epoch":1447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:35:13.210330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":120,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.520888+0000","last_change":"2024-09-21T11:02:39.502922+0000","last_active":"2024-09-21T11:02:39.520888+0000","last_peered":"2024-09-21T11:02:39.520888+0000","last_clean":"2024-09-21T11:02:39.520888+0000","last_became_active":"2024-09-21T11:02:39.502633+0000","last_became_peered":"2024-09-21T11:02:39.502633+0000","last_unstale":"2024-09-21T11:02:39.520888+0000","last_undegraded":"2024-09-21T11:02:39.520888+0000","last_fullsized":"2024-09-21T11:02:39.520888+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:29:20.971522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2320,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.551879+0000","last_change":"2024-09-21T11:02:39.538572+0000","last_active":"2024-09-21T11:02:39.551879+0000","last_peered":"2024-09-21T11:02:39.551879+0000","last_clean":"2024-09-21T11:02:39.551879+0000","last_became_active":"2024-09-21T11:02:39.537393+0000","last_became_peered":"2024-09-21T11:02:39.537393+0000","last_unstale":"2024-09-21T11:02:39.551879+0000","last_undegraded":"2024-09-21T11:02:39.551879+0000","last_fullsized":"2024-09-21T11:02:39.551879+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:20:01.943762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":1591,"reported_epoch":1592,"state":"active+clean","last_fresh":"2024-09-21T11:02:42.496428+0000","last_change":"2024-09-21T11:02:38.477734+0000","last_active":"2024-09-21T11:02:42.496428+0000","last_peered":"2024-09-21T11:02:42.496428+0000","last_clean":"2024-09-21T11:02:42.496428+0000","last_became_active":"2024-09-21T11:02:38.477412+0000","last_became_peered":"2024-09-21T11:02:38.477412+0000","last_unstale":"2024-09-21T11:02:42.496428+0000","last_undegraded":"2024-09-21T11:02:42.496428+0000","last_fullsized":"2024-09-21T11:02:42.496428+0000","mapping_epoch":1589,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:51:14.563340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":943,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.457174+0000","last_change":"2024-09-21T11:02:38.478502+0000","last_active":"2024-09-21T11:02:39.457174+0000","last_peered":"2024-09-21T11:02:39.457174+0000","last_clean":"2024-09-21T11:02:39.457174+0000","last_became_active":"2024-09-21T11:02:38.478150+0000","last_became_peered":"2024-09-21T11:02:38.478150+0000","last_unstale":"2024-09-21T11:02:39.457174+0000","last_undegraded":"2024-09-21T11:02:39.457174+0000","last_fullsized":"2024-09-21T11:02:39.457174+0000","mapping_epoch":1589,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1590,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:20.458421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:20.458421+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001528569,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"346'149","reported_seq":861,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.551969+0000","last_change":"2024-09-21T11:02:39.538927+0000","last_active":"2024-09-21T11:02:39.551969+0000","last_peered":"2024-09-21T11:02:39.551969+0000","last_clean":"2024-09-21T11:02:39.551969+0000","last_became_active":"2024-09-21T11:02:39.538169+0000","last_became_peered":"2024-09-21T11:02:39.538169+0000","last_unstale":"2024-09-21T11:02:39.551969+0000","last_undegraded":"2024-09-21T11:02:39.551969+0000","last_fullsized":"2024-09-21T11:02:39.551969+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:42.008235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:42.008235+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:30:51.273619+0000","scrub_duration":0.011129834999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2497,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:56.722580+0000","last_change":"2024-09-21T11:01:46.484382+0000","last_active":"2024-09-21T11:02:56.722580+0000","last_peered":"2024-09-21T11:02:56.722580+0000","last_clean":"2024-09-21T11:02:56.722580+0000","last_became_active":"2024-09-21T11:01:46.484116+0000","last_became_peered":"2024-09-21T11:01:46.484116+0000","last_unstale":"2024-09-21T11:02:56.722580+0000","last_undegraded":"2024-09-21T11:02:56.722580+0000","last_fullsized":"2024-09-21T11:02:56.722580+0000","mapping_epoch":1578,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:48.924838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2405,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:56.716790+0000","last_change":"2024-09-21T11:01:46.488420+0000","last_active":"2024-09-21T11:02:56.716790+0000","last_peered":"2024-09-21T11:02:56.716790+0000","last_clean":"2024-09-21T11:02:56.716790+0000","last_became_active":"2024-09-21T11:01:46.488112+0000","last_became_peered":"2024-09-21T11:01:46.488112+0000","last_unstale":"2024-09-21T11:02:56.716790+0000","last_undegraded":"2024-09-21T11:02:56.716790+0000","last_fullsized":"2024-09-21T11:02:56.716790+0000","mapping_epoch":1578,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:34:11.692699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"346'149","reported_seq":1215,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:56.719293+0000","last_change":"2024-09-21T11:01:48.663405+0000","last_active":"2024-09-21T11:02:56.719293+0000","last_peered":"2024-09-21T11:02:56.719293+0000","last_clean":"2024-09-21T11:02:56.719293+0000","last_became_active":"2024-09-21T11:01:47.465393+0000","last_became_peered":"2024-09-21T11:01:47.465393+0000","last_unstale":"2024-09-21T11:02:56.719293+0000","last_undegraded":"2024-09-21T11:02:56.719293+0000","last_fullsized":"2024-09-21T11:02:56.719293+0000","mapping_epoch":1579,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1580,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:48.663349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.663349+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:09.574900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1511,"reported_epoch":1592,"state":"active+clean","last_fresh":"2024-09-21T11:02:42.500175+0000","last_change":"2024-09-21T11:02:39.484139+0000","last_active":"2024-09-21T11:02:42.500175+0000","last_peered":"2024-09-21T11:02:42.500175+0000","last_clean":"2024-09-21T11:02:42.500175+0000","last_became_active":"2024-09-21T11:02:39.483189+0000","last_became_peered":"2024-09-21T11:02:39.483189+0000","last_unstale":"2024-09-21T11:02:42.500175+0000","last_undegraded":"2024-09-21T11:02:42.500175+0000","last_fullsized":"2024-09-21T11:02:42.500175+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:15:49.908202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1444,"reported_epoch":1592,"state":"active+clean","last_fresh":"2024-09-21T11:02:42.498794+0000","last_change":"2024-09-21T11:01:47.475241+0000","last_active":"2024-09-21T11:02:42.498794+0000","last_peered":"2024-09-21T11:02:42.498794+0000","last_clean":"2024-09-21T11:02:42.498794+0000","last_became_active":"2024-09-21T11:01:47.474461+0000","last_became_peered":"2024-09-21T11:01:47.474461+0000","last_unstale":"2024-09-21T11:02:42.498794+0000","last_undegraded":"2024-09-21T11:02:42.498794+0000","last_fullsized":"2024-09-21T11:02:42.498794+0000","mapping_epoch":1579,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:41.117625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"346'149","reported_seq":684,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.592550+0000","last_change":"2024-09-21T11:02:39.522603+0000","last_active":"2024-09-21T11:02:39.592550+0000","last_peered":"2024-09-21T11:02:39.592550+0000","last_clean":"2024-09-21T11:02:39.592550+0000","last_became_active":"2024-09-21T11:02:39.522133+0000","last_became_peered":"2024-09-21T11:02:39.522133+0000","last_unstale":"2024-09-21T11:02:39.592550+0000","last_undegraded":"2024-09-21T11:02:39.592550+0000","last_fullsized":"2024-09-21T11:02:39.592550+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:49.947173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:49.947173+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1056413,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1412,"reported_epoch":1592,"state":"active+clean","last_fresh":"2024-09-21T11:02:42.500925+0000","last_change":"2024-09-21T11:02:39.502518+0000","last_active":"2024-09-21T11:02:42.500925+0000","last_peered":"2024-09-21T11:02:42.500925+0000","last_clean":"2024-09-21T11:02:42.500925+0000","last_became_active":"2024-09-21T11:02:39.502191+0000","last_became_peered":"2024-09-21T11:02:39.502191+0000","last_unstale":"2024-09-21T11:02:42.500925+0000","last_undegraded":"2024-09-21T11:02:42.500925+0000","last_fullsized":"2024-09-21T11:02:42.500925+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:50:18.110449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1539,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.551926+0000","last_change":"2024-09-21T11:02:39.538744+0000","last_active":"2024-09-21T11:02:39.551926+0000","last_peered":"2024-09-21T11:02:39.551926+0000","last_clean":"2024-09-21T11:02:39.551926+0000","last_became_active":"2024-09-21T11:02:39.537783+0000","last_became_peered":"2024-09-21T11:02:39.537783+0000","last_unstale":"2024-09-21T11:02:39.551926+0000","last_undegraded":"2024-09-21T11:02:39.551926+0000","last_fullsized":"2024-09-21T11:02:39.551926+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:46:56.553302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"346'149","reported_seq":1062,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.458672+0000","last_change":"2024-09-21T11:02:38.474085+0000","last_active":"2024-09-21T11:02:39.458672+0000","last_peered":"2024-09-21T11:02:39.458672+0000","last_clean":"2024-09-21T11:02:39.458672+0000","last_became_active":"2024-09-21T11:02:38.473593+0000","last_became_peered":"2024-09-21T11:02:38.473593+0000","last_unstale":"2024-09-21T11:02:39.458672+0000","last_undegraded":"2024-09-21T11:02:39.458672+0000","last_fullsized":"2024-09-21T11:02:39.458672+0000","mapping_epoch":1589,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1590,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:17.418646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:17.418646+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:59:13.530642+0000","scrub_duration":0.00185648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":764,"reported_epoch":1592,"state":"active+clean","last_fresh":"2024-09-21T11:02:42.501308+0000","last_change":"2024-09-21T11:02:39.501678+0000","last_active":"2024-09-21T11:02:42.501308+0000","last_peered":"2024-09-21T11:02:42.501308+0000","last_clean":"2024-09-21T11:02:42.501308+0000","last_became_active":"2024-09-21T11:02:39.497766+0000","last_became_peered":"2024-09-21T11:02:39.497766+0000","last_unstale":"2024-09-21T11:02:42.501308+0000","last_undegraded":"2024-09-21T11:02:42.501308+0000","last_fullsized":"2024-09-21T11:02:42.501308+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:09:25.807702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":71,"reported_epoch":1592,"state":"active+clean","last_fresh":"2024-09-21T11:02:42.498747+0000","last_change":"2024-09-21T11:01:47.470218+0000","last_active":"2024-09-21T11:02:42.498747+0000","last_peered":"2024-09-21T11:02:42.498747+0000","last_clean":"2024-09-21T11:02:42.498747+0000","last_became_active":"2024-09-21T11:01:47.469740+0000","last_became_peered":"2024-09-21T11:01:47.469740+0000","last_unstale":"2024-09-21T11:02:42.498747+0000","last_undegraded":"2024-09-21T11:02:42.498747+0000","last_fullsized":"2024-09-21T11:02:42.498747+0000","mapping_epoch":1579,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:10:20.640896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"346'149","reported_seq":845,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:41.523904+0000","last_change":"2024-09-21T11:02:39.485938+0000","last_active":"2024-09-21T11:02:41.523904+0000","last_peered":"2024-09-21T11:02:41.523904+0000","last_clean":"2024-09-21T11:02:41.523904+0000","last_became_active":"2024-09-21T11:02:39.485296+0000","last_became_peered":"2024-09-21T11:02:39.485296+0000","last_unstale":"2024-09-21T11:02:41.523904+0000","last_undegraded":"2024-09-21T11:02:41.523904+0000","last_fullsized":"2024-09-21T11:02:41.523904+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:57.851898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:57.851898+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061250310000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":970,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:56.719809+0000","last_change":"2024-09-21T11:02:39.480949+0000","last_active":"2024-09-21T11:02:56.719809+0000","last_peered":"2024-09-21T11:02:56.719809+0000","last_clean":"2024-09-21T11:02:56.719809+0000","last_became_active":"2024-09-21T11:02:39.480676+0000","last_became_peered":"2024-09-21T11:02:39.480676+0000","last_unstale":"2024-09-21T11:02:56.719809+0000","last_undegraded":"2024-09-21T11:02:56.719809+0000","last_fullsized":"2024-09-21T11:02:56.719809+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:56:31.332024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":742,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:41.523867+0000","last_change":"2024-09-21T11:02:39.539214+0000","last_active":"2024-09-21T11:02:41.523867+0000","last_peered":"2024-09-21T11:02:41.523867+0000","last_clean":"2024-09-21T11:02:41.523867+0000","last_became_active":"2024-09-21T11:02:39.538813+0000","last_became_peered":"2024-09-21T11:02:39.538813+0000","last_unstale":"2024-09-21T11:02:41.523867+0000","last_undegraded":"2024-09-21T11:02:41.523867+0000","last_fullsized":"2024-09-21T11:02:41.523867+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:15:05.230635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"346'149","reported_seq":833,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:41.523941+0000","last_change":"2024-09-21T11:02:38.481550+0000","last_active":"2024-09-21T11:02:41.523941+0000","last_peered":"2024-09-21T11:02:41.523941+0000","last_clean":"2024-09-21T11:02:41.523941+0000","last_became_active":"2024-09-21T11:02:38.480871+0000","last_became_peered":"2024-09-21T11:02:38.480871+0000","last_unstale":"2024-09-21T11:02:41.523941+0000","last_undegraded":"2024-09-21T11:02:41.523941+0000","last_fullsized":"2024-09-21T11:02:41.523941+0000","mapping_epoch":1589,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1590,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:44.018645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:44.018645+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012789399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":81,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.459822+0000","last_change":"2024-09-21T11:02:38.480266+0000","last_active":"2024-09-21T11:02:39.459822+0000","last_peered":"2024-09-21T11:02:39.459822+0000","last_clean":"2024-09-21T11:02:39.459822+0000","last_became_active":"2024-09-21T11:02:38.479954+0000","last_became_peered":"2024-09-21T11:02:38.479954+0000","last_unstale":"2024-09-21T11:02:39.459822+0000","last_undegraded":"2024-09-21T11:02:39.459822+0000","last_fullsized":"2024-09-21T11:02:39.459822+0000","mapping_epoch":1589,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:27:06.799723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":324,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.476720+0000","last_change":"2024-09-21T11:02:39.474034+0000","last_active":"2024-09-21T11:02:39.476720+0000","last_peered":"2024-09-21T11:02:39.476720+0000","last_clean":"2024-09-21T11:02:39.476720+0000","last_became_active":"2024-09-21T11:02:39.473689+0000","last_became_peered":"2024-09-21T11:02:39.473689+0000","last_unstale":"2024-09-21T11:02:39.476720+0000","last_undegraded":"2024-09-21T11:02:39.476720+0000","last_fullsized":"2024-09-21T11:02:39.476720+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:50:07.890331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"346'149","reported_seq":1080,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.509561+0000","last_change":"2024-09-21T11:02:39.508308+0000","last_active":"2024-09-21T11:02:39.509561+0000","last_peered":"2024-09-21T11:02:39.509561+0000","last_clean":"2024-09-21T11:02:39.509561+0000","last_became_active":"2024-09-21T11:02:39.507769+0000","last_became_peered":"2024-09-21T11:02:39.507769+0000","last_unstale":"2024-09-21T11:02:39.509561+0000","last_undegraded":"2024-09-21T11:02:39.509561+0000","last_fullsized":"2024-09-21T11:02:39.509561+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:54.708658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:54.708658+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:24:47.574601+0000","scrub_duration":0.0021035160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1584,"reported_epoch":1592,"state":"active+clean","last_fresh":"2024-09-21T11:02:42.498645+0000","last_change":"2024-09-21T11:02:39.489781+0000","last_active":"2024-09-21T11:02:42.498645+0000","last_peered":"2024-09-21T11:02:42.498645+0000","last_clean":"2024-09-21T11:02:42.498645+0000","last_became_active":"2024-09-21T11:02:39.489523+0000","last_became_peered":"2024-09-21T11:02:39.489523+0000","last_unstale":"2024-09-21T11:02:42.498645+0000","last_undegraded":"2024-09-21T11:02:42.498645+0000","last_fullsized":"2024-09-21T11:02:42.498645+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:39.794133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":204,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.457833+0000","last_change":"2024-09-21T11:01:47.476005+0000","last_active":"2024-09-21T11:02:39.457833+0000","last_peered":"2024-09-21T11:02:39.457833+0000","last_clean":"2024-09-21T11:02:39.457833+0000","last_became_active":"2024-09-21T11:01:47.475676+0000","last_became_peered":"2024-09-21T11:01:47.475676+0000","last_unstale":"2024-09-21T11:02:39.457833+0000","last_undegraded":"2024-09-21T11:02:39.457833+0000","last_fullsized":"2024-09-21T11:02:39.457833+0000","mapping_epoch":1579,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:36:39.495624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"346'149","reported_seq":749,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.501787+0000","last_change":"2024-09-21T11:02:39.487608+0000","last_active":"2024-09-21T11:02:39.501787+0000","last_peered":"2024-09-21T11:02:39.501787+0000","last_clean":"2024-09-21T11:02:39.501787+0000","last_became_active":"2024-09-21T11:02:39.479399+0000","last_became_peered":"2024-09-21T11:02:39.479399+0000","last_unstale":"2024-09-21T11:02:39.501787+0000","last_undegraded":"2024-09-21T11:02:39.501787+0000","last_fullsized":"2024-09-21T11:02:39.501787+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:53.896066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:53.896066+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:21:00.369599+0000","scrub_duration":0.001699578,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":204,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.471083+0000","last_change":"2024-09-21T11:02:39.470147+0000","last_active":"2024-09-21T11:02:39.471083+0000","last_peered":"2024-09-21T11:02:39.471083+0000","last_clean":"2024-09-21T11:02:39.471083+0000","last_became_active":"2024-09-21T11:02:39.468797+0000","last_became_peered":"2024-09-21T11:02:39.468797+0000","last_unstale":"2024-09-21T11:02:39.471083+0000","last_undegraded":"2024-09-21T11:02:39.471083+0000","last_fullsized":"2024-09-21T11:02:39.471083+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:13:04.375109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":204,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.460337+0000","last_change":"2024-09-21T11:01:47.466155+0000","last_active":"2024-09-21T11:02:39.460337+0000","last_peered":"2024-09-21T11:02:39.460337+0000","last_clean":"2024-09-21T11:02:39.460337+0000","last_became_active":"2024-09-21T11:01:47.465861+0000","last_became_peered":"2024-09-21T11:01:47.465861+0000","last_unstale":"2024-09-21T11:02:39.460337+0000","last_undegraded":"2024-09-21T11:02:39.460337+0000","last_fullsized":"2024-09-21T11:02:39.460337+0000","mapping_epoch":1579,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1580,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:02:51.671451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"346'149","reported_seq":1014,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.456876+0000","last_change":"2024-09-21T11:01:48.731990+0000","last_active":"2024-09-21T11:02:39.456876+0000","last_peered":"2024-09-21T11:02:39.456876+0000","last_clean":"2024-09-21T11:02:39.456876+0000","last_became_active":"2024-09-21T11:01:47.466476+0000","last_became_peered":"2024-09-21T11:01:47.466476+0000","last_unstale":"2024-09-21T11:02:39.456876+0000","last_undegraded":"2024-09-21T11:02:39.456876+0000","last_fullsized":"2024-09-21T11:02:39.456876+0000","mapping_epoch":1579,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1580,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:48.731903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.731903+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:09.568300+0000","scrub_duration":0.0015067920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1563,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.463328+0000","last_change":"2024-09-21T11:02:38.476969+0000","last_active":"2024-09-21T11:02:39.463328+0000","last_peered":"2024-09-21T11:02:39.463328+0000","last_clean":"2024-09-21T11:02:39.463328+0000","last_became_active":"2024-09-21T11:02:38.476551+0000","last_became_peered":"2024-09-21T11:02:38.476551+0000","last_unstale":"2024-09-21T11:02:39.463328+0000","last_undegraded":"2024-09-21T11:02:39.463328+0000","last_fullsized":"2024-09-21T11:02:39.463328+0000","mapping_epoch":1589,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:22:11.849771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2296,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.460035+0000","last_change":"2024-09-21T11:02:38.479112+0000","last_active":"2024-09-21T11:02:39.460035+0000","last_peered":"2024-09-21T11:02:39.460035+0000","last_clean":"2024-09-21T11:02:39.460035+0000","last_became_active":"2024-09-21T11:02:38.478833+0000","last_became_peered":"2024-09-21T11:02:38.478833+0000","last_unstale":"2024-09-21T11:02:39.460035+0000","last_undegraded":"2024-09-21T11:02:39.460035+0000","last_fullsized":"2024-09-21T11:02:39.460035+0000","mapping_epoch":1589,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:49:42.642978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"346'149","reported_seq":1215,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:56.719157+0000","last_change":"2024-09-21T11:01:54.706488+0000","last_active":"2024-09-21T11:02:56.719157+0000","last_peered":"2024-09-21T11:02:56.719157+0000","last_clean":"2024-09-21T11:02:56.719157+0000","last_became_active":"2024-09-21T11:01:47.468205+0000","last_became_peered":"2024-09-21T11:01:47.468205+0000","last_unstale":"2024-09-21T11:02:56.719157+0000","last_undegraded":"2024-09-21T11:02:56.719157+0000","last_fullsized":"2024-09-21T11:02:56.719157+0000","mapping_epoch":1579,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1580,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:54.706428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:54.706428+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:13:37.282221+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":304,"reported_epoch":1592,"state":"active+clean","last_fresh":"2024-09-21T11:02:42.496000+0000","last_change":"2024-09-21T11:02:39.471842+0000","last_active":"2024-09-21T11:02:42.496000+0000","last_peered":"2024-09-21T11:02:42.496000+0000","last_clean":"2024-09-21T11:02:42.496000+0000","last_became_active":"2024-09-21T11:02:39.471339+0000","last_became_peered":"2024-09-21T11:02:39.471339+0000","last_unstale":"2024-09-21T11:02:42.496000+0000","last_undegraded":"2024-09-21T11:02:42.496000+0000","last_fullsized":"2024-09-21T11:02:42.496000+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:53:53.996670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2344,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.457191+0000","last_change":"2024-09-21T11:02:38.476746+0000","last_active":"2024-09-21T11:02:39.457191+0000","last_peered":"2024-09-21T11:02:39.457191+0000","last_clean":"2024-09-21T11:02:39.457191+0000","last_became_active":"2024-09-21T11:02:38.476452+0000","last_became_peered":"2024-09-21T11:02:38.476452+0000","last_unstale":"2024-09-21T11:02:39.457191+0000","last_undegraded":"2024-09-21T11:02:39.457191+0000","last_fullsized":"2024-09-21T11:02:39.457191+0000","mapping_epoch":1589,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:08.806913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"346'149","reported_seq":942,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.456195+0000","last_change":"2024-09-21T11:02:38.473655+0000","last_active":"2024-09-21T11:02:39.456195+0000","last_peered":"2024-09-21T11:02:39.456195+0000","last_clean":"2024-09-21T11:02:39.456195+0000","last_became_active":"2024-09-21T11:02:38.473203+0000","last_became_peered":"2024-09-21T11:02:38.473203+0000","last_unstale":"2024-09-21T11:02:39.456195+0000","last_undegraded":"2024-09-21T11:02:39.456195+0000","last_fullsized":"2024-09-21T11:02:39.456195+0000","mapping_epoch":1589,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1590,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:20.458421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:20.458421+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001528569,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":165,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.459297+0000","last_change":"2024-09-21T11:02:38.478997+0000","last_active":"2024-09-21T11:02:39.459297+0000","last_peered":"2024-09-21T11:02:39.459297+0000","last_clean":"2024-09-21T11:02:39.459297+0000","last_became_active":"2024-09-21T11:02:38.478505+0000","last_became_peered":"2024-09-21T11:02:38.478505+0000","last_unstale":"2024-09-21T11:02:39.459297+0000","last_undegraded":"2024-09-21T11:02:39.459297+0000","last_fullsized":"2024-09-21T11:02:39.459297+0000","mapping_epoch":1589,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:41:30.759026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1570,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.457666+0000","last_change":"2024-09-21T11:02:38.476498+0000","last_active":"2024-09-21T11:02:39.457666+0000","last_peered":"2024-09-21T11:02:39.457666+0000","last_clean":"2024-09-21T11:02:39.457666+0000","last_became_active":"2024-09-21T11:02:38.476173+0000","last_became_peered":"2024-09-21T11:02:38.476173+0000","last_unstale":"2024-09-21T11:02:39.457666+0000","last_undegraded":"2024-09-21T11:02:39.457666+0000","last_fullsized":"2024-09-21T11:02:39.457666+0000","mapping_epoch":1589,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:49:26.618542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"346'149","reported_seq":895,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.481240+0000","last_change":"2024-09-21T11:02:39.479846+0000","last_active":"2024-09-21T11:02:39.481240+0000","last_peered":"2024-09-21T11:02:39.481240+0000","last_clean":"2024-09-21T11:02:39.481240+0000","last_became_active":"2024-09-21T11:02:39.478046+0000","last_became_peered":"2024-09-21T11:02:39.478046+0000","last_unstale":"2024-09-21T11:02:39.481240+0000","last_undegraded":"2024-09-21T11:02:39.481240+0000","last_fullsized":"2024-09-21T11:02:39.481240+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:47.955189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:47.955189+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:44:22.188751+0000","scrub_duration":0.0019884339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":318,"reported_epoch":1591,"state":"active+clean","last_fresh":"2024-09-21T11:02:39.469836+0000","last_change":"2024-09-21T11:02:39.468831+0000","last_active":"2024-09-21T11:02:39.469836+0000","last_peered":"2024-09-21T11:02:39.469836+0000","last_clean":"2024-09-21T11:02:39.469836+0000","last_became_active":"2024-09-21T11:02:39.468455+0000","last_became_peered":"2024-09-21T11:02:39.468455+0000","last_unstale":"2024-09-21T11:02:39.469836+0000","last_undegraded":"2024-09-21T11:02:39.469836+0000","last_fullsized":"2024-09-21T11:02:39.469836+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:34:59.495678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2481,"reported_epoch":1593,"state":"active+clean","last_fresh":"2024-09-21T11:02:56.714889+0000","last_change":"2024-09-21T10:44:14.398584+0000","last_active":"2024-09-21T11:02:56.714889+0000","last_peered":"2024-09-21T11:02:56.714889+0000","last_clean":"2024-09-21T11:02:56.714889+0000","last_became_active":"2024-09-21T10:44:14.398278+0000","last_became_peered":"2024-09-21T10:44:14.398278+0000","last_unstale":"2024-09-21T11:02:56.714889+0000","last_undegraded":"2024-09-21T11:02:56.714889+0000","last_fullsized":"2024-09-21T11:02:56.714889+0000","mapping_epoch":1369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:19.681750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":58,"num_pg":16,"stat_sum":{"num_bytes":7163609088,"num_objects":109308,"num_object_clones":0,"num_object_copies":327924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":66469,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109308,"num_write_kb":6995712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":515,"num_bytes_recovered":33751040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17098932224,"data_stored":17098932224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66908,"ondisk_log_size":66908,"up":48,"acting":38,"num_store_stats":5},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":381350,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1056413,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3129,"ondisk_log_size":3129,"up":63,"acting":63,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":16658,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":18008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3602,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234827,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6543876,"kb_used_data":5167228,"kb_used_omap":1,"kb_used_meta":1376638,"kb_avail":87201276,"statfs":{"total":95995035648,"available":89294106624,"internally_reserved":0,"allocated":5291241472,"data_stored":5287139355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":1409677771},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":1589,"seq":6824703033349,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":237436,"kb_used_data":117684,"kb_used_omap":128,"kb_used_meta":119615,"kb_avail":93507716,"statfs":{"total":95995035648,"available":95751901184,"internally_reserved":0,"allocated":120508416,"data_stored":115973446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":131575,"internal_metadata":122486281},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208758048,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019736,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5480580,"kb_used_data":4278332,"kb_used_omap":1,"kb_used_meta":1202238,"kb_avail":88264572,"statfs":{"total":95995035648,"available":90382921728,"internally_reserved":0,"allocated":4381011968,"data_stored":4376910376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1231092166},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369481,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3110388,"kb_used_data":1784428,"kb_used_omap":69,"kb_used_meta":1325882,"kb_avail":90634764,"statfs":{"total":95995035648,"available":92809998336,"internally_reserved":0,"allocated":1827254272,"data_stored":1823172012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":1357703298},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2920577761936,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807958167,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6823932,"kb_used_data":5315444,"kb_used_omap":66,"kb_used_meta":1508413,"kb_avail":86921220,"statfs":{"total":95995035648,"available":89007329280,"internally_reserved":0,"allocated":5443014656,"data_stored":5438924699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68589,"internal_metadata":1544614931},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":2860448219806,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1187,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5467406336,"data_stored":5467406336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1819738112,"data_stored":1819738112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4400480256,"data_stored":4400480256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":112459776,"data_stored":112459776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5298847744,"data_stored":5298847744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T11:03:02.948 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_5 pgp_num 16 2024-09-21T11:03:03.019 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:03.057 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:03.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:03.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:03.159 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:03.192 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:03.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:03.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:03.293 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:03.324 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:03.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:03.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:03.334 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 65 16 116238 116222 111.738 80.5625 0.0125434 0.00894544 2024-09-21T11:03:03.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:03:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:03:03] "GET /metrics HTTP/1.1" 200 36585 "" "Prometheus/2.51.0" 2024-09-21T11:03:03.426 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:03.459 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:03.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:03.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:03.561 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:03.591 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:03.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:03.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:03.693 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:03.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:03.729 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:03.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:03.812 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:03 smithi070 ceph-mon[112139]: pgmap v3118: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.8 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 70 MiB/s wr, 1.12k op/s; 67817/336237 objects degraded (20.169%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T11:03:03.812 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:03 smithi070 ceph-mon[107385]: pgmap v3118: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.8 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 70 MiB/s wr, 1.12k op/s; 67817/336237 objects degraded (20.169%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T11:03:03.830 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:03 smithi160 ceph-mon[101879]: pgmap v3118: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.8 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 70 MiB/s wr, 1.12k op/s; 67817/336237 objects degraded (20.169%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T11:03:03.864 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:03.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:03.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:03.966 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:04.001 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:04.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:04.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:04.003 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T11:03:04.004 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T11:03:04.103 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:04.136 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:04.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:04.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:04.238 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:04.274 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:04.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:04.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:04.334 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 66 16 117510 117494 111.249 79.5 0.0825616 0.00898311 2024-09-21T11:03:04.376 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:04.402 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:04.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:04.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:04.504 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:04.535 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:04.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:04.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:04.636 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:04.663 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:04.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:04.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:04.765 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:04.791 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:04.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:04.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:04.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:04 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 67817/336237 objects degraded (20.169%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:03:04.849 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:04 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 67817/336237 objects degraded (20.169%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:03:04.850 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:04 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 67817/336237 objects degraded (20.169%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:03:04.893 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:04.925 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:04.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:04.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:05.027 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:05.058 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:05.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:05.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:05.160 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:05.194 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:05.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:05.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:05.293 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T11:03:05.296 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:05.299 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:03:05.302 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:03:05.334 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 67 16 118798 118782 110.79 80.5 0.00966263 0.00902086 2024-09-21T11:03:05.335 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:05.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:05.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:05.437 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:05.474 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:05.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:05.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:05.575 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:05.614 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:05 smithi070 ceph-mon[107385]: pgmap v3119: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.8 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 70 MiB/s wr, 1.12k op/s; 67817/336237 objects degraded (20.169%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T11:03:05.614 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:03:05.615 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:03:05.615 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:05 smithi070 ceph-mon[112139]: pgmap v3119: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.8 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 70 MiB/s wr, 1.12k op/s; 67817/336237 objects degraded (20.169%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T11:03:05.615 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:03:05.615 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:03:05.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:05.617 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:05.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:05.718 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:05.755 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:05.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:05.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:05 smithi160 ceph-mon[101879]: pgmap v3119: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.8 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 70 MiB/s wr, 1.12k op/s; 67817/336237 objects degraded (20.169%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T11:03:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:03:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:03:05.858 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:05.899 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:05.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:05.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:06.001 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:06.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:06.043 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:06.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:06.144 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:06.173 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:06.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:06.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:06.275 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:06.302 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:06.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:06.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:06.334 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 68 16 120041 120025 110.303 77.6875 0.00936755 0.00906171 2024-09-21T11:03:06.405 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:06.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:06.435 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:06.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:06.536 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:06.564 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:06.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:06.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:06.665 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:06.703 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:06.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:06.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:06.805 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:06.841 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:06.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:06.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:06.943 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:06.970 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:06.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:06.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:07.072 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:07.115 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:07.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:07.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:07.217 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:07.232 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:03:07.232 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:03:07.253 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:07.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:07.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:07.261 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:03:07.261 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:03:07.291 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:03:07.291 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:03:07.320 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:03:07.321 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:03:07.334 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 69 16 121268 121252 109.816 76.6875 0.00367621 0.00909822 2024-09-21T11:03:07.347 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:03:07.347 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:03:07.348 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T11:03:07.354 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:07.376 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T11:03:07.376 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T11:03:07.389 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:07.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:07.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:07.403 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:03:07.403 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:03:07.492 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:07.525 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:07.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:07.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:07.627 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:07.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:07 smithi070 ceph-mon[107385]: pgmap v3120: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.2 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 74 MiB/s wr, 1.18k op/s; 71340/355545 objects degraded (20.065%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T11:03:07.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:07 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1915241824' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:03:07.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:07 smithi070 ceph-mon[112139]: pgmap v3120: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.2 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 74 MiB/s wr, 1.18k op/s; 71340/355545 objects degraded (20.065%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T11:03:07.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:07 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1915241824' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:03:07.663 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:07.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:07.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:07.764 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:07.796 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:07.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:07.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:07 smithi160 ceph-mon[101879]: pgmap v3120: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.2 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 74 MiB/s wr, 1.18k op/s; 71340/355545 objects degraded (20.065%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T11:03:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:07 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1915241824' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:03:07.897 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:07.927 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:07.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:07.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:08.029 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:08.066 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:08.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:08.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:08.168 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:08.195 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:08.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:08.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:08.297 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:08.326 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:08.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:08.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:08.335 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 70 16 122498 122482 109.345 76.875 0.0121242 0.00914108 2024-09-21T11:03:08.428 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:08.457 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:08.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:08.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:08.559 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:08.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:08.589 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:08.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:08.690 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:08.718 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:08.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:08.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:08.820 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:08.857 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:08.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:08.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:08.959 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:09.003 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:09.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:09.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:09.105 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:09.132 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:09.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:09.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:09.233 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:09.262 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:09.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:09.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:09.313 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T11:03:09.313 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T11:03:09.335 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 71 16 123761 123745 108.917 78.9375 0.0182479 0.00917706 2024-09-21T11:03:09.364 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:09.392 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:09.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:09.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:09.494 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:09.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:09.534 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:09.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:09.634 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:09.675 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:09.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:09.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:09.777 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:09.816 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:09.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:09.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:09 smithi160 ceph-mon[101879]: pgmap v3121: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.2 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 64 MiB/s wr, 1.02k op/s; 71340/355545 objects degraded (20.065%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T11:03:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:09 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 71340/355545 objects degraded (20.065%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:03:09.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:09 smithi070 ceph-mon[107385]: pgmap v3121: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.2 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 64 MiB/s wr, 1.02k op/s; 71340/355545 objects degraded (20.065%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T11:03:09.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:09 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 71340/355545 objects degraded (20.065%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:03:09.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:09 smithi070 ceph-mon[112139]: pgmap v3121: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.2 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 64 MiB/s wr, 1.02k op/s; 71340/355545 objects degraded (20.065%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T11:03:09.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:09 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 71340/355545 objects degraded (20.065%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:03:09.918 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:09.957 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:09.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:09.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:10.058 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:10.100 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:10.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:10.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:10.201 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:10.232 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:10.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:10.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:10.334 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:10.335 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 72 16 125046 125030 108.519 80.3125 0.00926572 0.00921027 2024-09-21T11:03:10.366 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:10.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:10.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:10.468 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:10.502 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:10.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:10.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:10.603 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:10.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:10.631 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:10.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:10.697 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:03:10.732 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:10.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:10.763 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:10.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:10.864 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:10.898 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:10.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:10.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:10.999 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:11.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:11.038 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:11.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:11.139 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:11.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:11.176 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:11.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:11.277 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:11.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:11.311 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:11.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:11.314 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T11:03:11.335 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 73 16 126362 126346 108.159 82.25 0.00803566 0.00924121 2024-09-21T11:03:11.412 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:11.438 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:11.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:11.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:11.461 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:11 smithi070 ceph-mon[112139]: pgmap v3122: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.4 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 77 MiB/s wr, 1.24k op/s; 73127/364701 objects degraded (20.051%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T11:03:11.461 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:11 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1054388162' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T11:03:11.461 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:11 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2180366947' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T11:03:11.469 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T11:03:11.471 INFO:teuthology.orchestra.run.smithi070.stderr:set pool 58 pgp_num to 16 2024-09-21T11:03:11.540 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:11.574 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:11.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:11.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:11.677 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:11.716 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:11.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:11.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:11.818 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:11 smithi160 ceph-mon[101879]: pgmap v3122: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.4 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 77 MiB/s wr, 1.24k op/s; 73127/364701 objects degraded (20.051%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T11:03:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:11 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1054388162' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T11:03:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:11 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2180366947' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T11:03:11.845 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:11.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:11.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:11.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:11 smithi070 ceph-mon[107385]: pgmap v3122: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.4 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 77 MiB/s wr, 1.24k op/s; 73127/364701 objects degraded (20.051%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T11:03:11.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:11 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1054388162' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T11:03:11.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:11 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2180366947' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T11:03:11.947 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:11.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:11.977 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:11.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:12.078 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:12.099 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:12.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:12.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:12.200 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:12.237 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:12.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:12.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:12.335 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 74 16 127518 127502 107.674 72.25 0.00781599 0.00928234 2024-09-21T11:03:12.339 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:12.363 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:12.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:12.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:12.465 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:12.505 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:12.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:12.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:12.606 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:12.637 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:12.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:12.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:12.739 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:12.768 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:12.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:12.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:12 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:03:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:12 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1054388162' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T11:03:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:12 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2180366947' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]': finished 2024-09-21T11:03:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:12 smithi160 ceph-mon[101879]: osdmap e1594: 8 total, 7 up, 5 in 2024-09-21T11:03:12.870 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:12.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:12 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:03:12.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:12 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1054388162' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T11:03:12.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:12 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2180366947' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]': finished 2024-09-21T11:03:12.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:12 smithi070 ceph-mon[107385]: osdmap e1594: 8 total, 7 up, 5 in 2024-09-21T11:03:12.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:12 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:03:12.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:12 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1054388162' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T11:03:12.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:12 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2180366947' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]': finished 2024-09-21T11:03:12.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:12 smithi070 ceph-mon[112139]: osdmap e1594: 8 total, 7 up, 5 in 2024-09-21T11:03:12.910 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:12.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:12.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:13.012 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:13.045 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:13.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:13.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:13.146 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:13.177 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:13.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:13.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:13.279 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:13.306 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:13.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:13.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:13.335 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 75 16 128700 128684 107.223 73.875 0.0280703 0.00932217 2024-09-21T11:03:13.410 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:13.447 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:13.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:13.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:13.472 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:03:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:03:13] "GET /metrics HTTP/1.1" 200 36576 "" "Prometheus/2.51.0" 2024-09-21T11:03:13.549 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:13.583 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:13.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:13.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:13.685 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:13.723 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:13.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:13.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:13.824 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:13 smithi160 ceph-mon[101879]: pgmap v3124: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.6 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 79 MiB/s wr, 1.27k op/s; 74724/374358 objects degraded (19.961%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T11:03:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:13 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 74724/374358 objects degraded (19.961%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:03:13.852 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:13.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:13.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:13.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:13 smithi070 ceph-mon[107385]: pgmap v3124: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.6 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 79 MiB/s wr, 1.27k op/s; 74724/374358 objects degraded (19.961%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T11:03:13.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:13 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 74724/374358 objects degraded (19.961%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:03:13.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:13 smithi070 ceph-mon[112139]: pgmap v3124: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.6 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 79 MiB/s wr, 1.27k op/s; 74724/374358 objects degraded (19.961%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T11:03:13.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:13 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 74724/374358 objects degraded (19.961%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:03:13.954 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:13.979 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:13.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:13.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:14.080 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:14.121 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:14.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:14.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:14.222 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:14.253 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:14.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:14.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:14.337 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 76 16 129888 129872 106.789 74.25 0.00707343 0.00935924 2024-09-21T11:03:14.355 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:14.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:14.378 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:14.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:14.478 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:14.515 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:14.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:14.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:14.618 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:14.647 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:14.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:14.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:14.748 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:14.775 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:14.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:14.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:14.877 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:14.905 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:14.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:14.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:15.006 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:15.044 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:15.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:15.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:15.146 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:15.170 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:15.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:15.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:15.273 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:15.301 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:15.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:15.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:15.336 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 77 16 131020 131004 106.321 70.75 0.0143125 0.00940101 2024-09-21T11:03:15.403 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:15.435 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:15.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:15.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:15.536 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:15.573 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:15.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:15.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:15.675 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:15.714 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:15.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:15.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:15.816 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:15 smithi160 ceph-mon[101879]: pgmap v3125: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.6 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 79 MiB/s wr, 1.27k op/s; 74724/374358 objects degraded (19.961%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T11:03:15.847 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:15.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:15.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:15.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:15 smithi070 ceph-mon[107385]: pgmap v3125: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.6 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 79 MiB/s wr, 1.27k op/s; 74724/374358 objects degraded (19.961%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T11:03:15.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:15 smithi070 ceph-mon[112139]: pgmap v3125: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.6 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 79 MiB/s wr, 1.27k op/s; 74724/374358 objects degraded (19.961%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T11:03:15.949 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:15.981 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:15.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:15.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:16.083 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:16.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:16.123 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:16.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:16.224 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:16.254 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:16.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:16.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:16.336 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 78 16 132231 132215 105.928 75.6875 0.0143644 0.00943612 2024-09-21T11:03:16.356 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:16.389 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:16.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:16.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:16.491 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:16.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:16.528 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:16.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:16.629 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:16.658 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:16.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:16.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:16.760 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:16.802 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:16.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:16.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:16.904 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:16.932 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:16.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:16.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:17.034 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:17.072 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:17.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:17.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:17.173 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:17.204 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:17.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:17.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:17.306 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:17.335 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:17.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:17.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:17.336 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 79 16 133337 133321 105.462 69.125 0.0114427 0.00947776 2024-09-21T11:03:17.437 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:17.462 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:17.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:17.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:17.563 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:17.591 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:17.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:17.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:17.693 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:17.721 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:17.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:17.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:17.823 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:17.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:17 smithi160 ceph-mon[101879]: pgmap v3126: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.0 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 77 MiB/s wr, 1.23k op/s; 77954/392487 objects degraded (19.862%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T11:03:17.852 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:17.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:17.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:17.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:17 smithi070 ceph-mon[107385]: pgmap v3126: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.0 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 77 MiB/s wr, 1.23k op/s; 77954/392487 objects degraded (19.862%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T11:03:17.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:17 smithi070 ceph-mon[112139]: pgmap v3126: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.0 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 77 MiB/s wr, 1.23k op/s; 77954/392487 objects degraded (19.862%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T11:03:17.954 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:17.981 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:17.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:17.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:18.083 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:18.110 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:18.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:18.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:18.212 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:18.240 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:18.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:18.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:18.336 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:2024-09-21T11:03:18.335122+0000 min lat: 0.00100109 max lat: 2.2389 avg lat: 0.009517 2024-09-21T11:03:18.336 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T11:03:18.336 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 80 16 134468 134452 105.028 70.6875 0.0154386 0.009517 2024-09-21T11:03:18.342 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:18.381 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:18.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:18.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:18.483 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:18.514 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:18.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:18.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:18.615 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:18.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:18 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 77954/392487 objects degraded (19.862%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:03:18.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:18 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 77954/392487 objects degraded (19.862%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:03:18.637 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:18.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:18.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:18.738 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:18.766 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:18.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:18.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:18 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 77954/392487 objects degraded (19.862%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:03:18.867 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:18.896 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:18.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:18.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:18.998 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:19.024 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:19.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:19.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:19.126 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:19.166 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:19.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:19.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:19.268 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:19.297 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:19.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:19.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:19.336 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 81 16 135553 135537 104.568 67.8125 0.0073356 0.00955753 2024-09-21T11:03:19.399 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:19.427 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:19.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:19.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:19.528 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:19.568 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:19.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:19.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:19.669 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:19.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:19.699 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:19.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:19.800 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:19 smithi160 ceph-mon[101879]: pgmap v3127: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.0 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 77 MiB/s wr, 1.23k op/s; 77954/392487 objects degraded (19.862%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T11:03:19.842 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:19.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:19.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:19.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:19 smithi070 ceph-mon[107385]: pgmap v3127: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.0 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 77 MiB/s wr, 1.23k op/s; 77954/392487 objects degraded (19.862%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T11:03:19.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:19 smithi070 ceph-mon[112139]: pgmap v3127: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.0 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 77 MiB/s wr, 1.23k op/s; 77954/392487 objects degraded (19.862%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T11:03:19.944 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:19.974 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:19.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:19.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:20.076 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:20.100 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:20.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:20.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:20.202 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:20.240 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:20.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:20.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:20.336 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 82 16 136670 136654 104.144 69.8125 0.0139643 0.00959787 2024-09-21T11:03:20.341 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:20.376 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:20.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:20.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:20.478 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:20.506 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:20.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:20.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:20.608 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:20.640 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:20.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:20.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:20.741 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:20.779 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:20.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:20.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:20.881 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:20.915 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:20.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:20.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:21.017 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:21.045 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:21.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:21.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:21.147 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:21.174 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:21.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:21.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:21.276 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:21.304 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:21.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:21.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:21.336 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 83 16 137904 137888 103.819 77.125 0.00808206 0.00962799 2024-09-21T11:03:21.406 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:21.447 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:21.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:21.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:21.549 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:21.578 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:21.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:21.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:21.680 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:21.710 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:21.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:21.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:21.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:21 smithi070 ceph-mon[107385]: pgmap v3128: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.2 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 75 MiB/s wr, 1.20k op/s; 79498/400752 objects degraded (19.837%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T11:03:21.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:03:21.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:03:21.725 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:21 smithi070 ceph-mon[112139]: pgmap v3128: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.2 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 75 MiB/s wr, 1.20k op/s; 79498/400752 objects degraded (19.837%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T11:03:21.725 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:03:21.725 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:03:21.812 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:21.841 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:21 smithi160 ceph-mon[101879]: pgmap v3128: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.2 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 75 MiB/s wr, 1.20k op/s; 79498/400752 objects degraded (19.837%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T11:03:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:03:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:03:21.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:21.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:21.943 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:21.981 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:21.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:21.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:22.083 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:22.119 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:22.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:22.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:22.221 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:22.246 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:22.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:22.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:22.336 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 84 16 139145 139129 103.506 77.5625 0.0485705 0.00965572 2024-09-21T11:03:22.349 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:22.390 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:22.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:22.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:22.491 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:22.520 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:22.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:22.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:22.622 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:22.656 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:22.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:22.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:22.757 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:22.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:22.789 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:22.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:22.891 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:22.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:22.932 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:22.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:23.033 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:23.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:23.064 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:23.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:23.165 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:23.196 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:23.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:23.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:23.298 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:23.329 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:23.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:23.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:23.336 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 85 16 140383 140367 103.199 77.375 0.0198954 0.00968539 2024-09-21T11:03:23.431 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:23.468 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:23 smithi070 ceph-mon[107385]: pgmap v3129: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.3 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 67 MiB/s wr, 1.07k op/s; 80877/409260 objects degraded (19.762%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T11:03:23.468 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:03:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:03:23] "GET /metrics HTTP/1.1" 200 36577 "" "Prometheus/2.51.0" 2024-09-21T11:03:23.469 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:23 smithi070 ceph-mon[112139]: pgmap v3129: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.3 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 67 MiB/s wr, 1.07k op/s; 80877/409260 objects degraded (19.762%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T11:03:23.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:23.470 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:23.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:23.572 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:23.604 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:23.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:23.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:23.706 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:23.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:23.737 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:23.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:23.838 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:23.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:23 smithi160 ceph-mon[101879]: pgmap v3129: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.3 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 67 MiB/s wr, 1.07k op/s; 80877/409260 objects degraded (19.762%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T11:03:23.882 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:23.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:23.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:23.984 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:24.017 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:24.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:24.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:24.086 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T11:03:24.086 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T11:03:24.086 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T11:03:24.086 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T11:03:24.119 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:24.145 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:24.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:24.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:24.246 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:24.272 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:24.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:24.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:24.336 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 86 16 141558 141542 102.852 73.4375 0.010183 0.00971796 2024-09-21T11:03:24.375 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:24.408 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:24.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:24.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:24.510 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:24.531 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:24.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:24.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:24.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:24 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 80877/409260 objects degraded (19.762%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:03:24.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:24 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 80877/409260 objects degraded (19.762%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:03:24.634 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:24.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:24.669 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:24.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:24.770 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:24.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:24.807 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:24.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:24 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 80877/409260 objects degraded (19.762%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:03:24.908 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:24.939 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:24.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:24.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:25.040 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:25.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:25.077 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:25.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:25.108 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T11:03:25.178 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:25.217 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:25.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:25.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:25.319 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:25.337 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 87 16 142589 142573 102.41 64.4375 0.0253172 0.00975959 2024-09-21T11:03:25.364 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:25.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:25.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:25.469 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:25.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:25.505 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:25.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:25.574 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:03:25.606 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:25.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:25 smithi070 ceph-mon[112139]: pgmap v3130: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.3 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 61 MiB/s wr, 969 op/s; 80877/409260 objects degraded (19.762%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T11:03:25.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:25 smithi070 ceph-mon[107385]: pgmap v3130: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.3 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 61 MiB/s wr, 969 op/s; 80877/409260 objects degraded (19.762%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T11:03:25.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:25.643 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:25.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:25.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:25 smithi160 ceph-mon[101879]: pgmap v3130: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.3 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 61 MiB/s wr, 969 op/s; 80877/409260 objects degraded (19.762%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T11:03:25.743 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:25.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:25.780 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:25.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:25.824 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T11:03:25.880 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:25.920 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:25.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:25.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:26.022 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:26.053 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:26.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:26.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:26.155 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:26.185 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:26.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:26.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:26.287 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:26.321 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:26.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:26.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:26.337 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 88 16 143745 143729 102.067 72.25 0.0212983 0.00979286 2024-09-21T11:03:26.423 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:26.457 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:26.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:26.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:26.559 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:26.590 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:26.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:26.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:26.692 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:26.731 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:26.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:26.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:26.829 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:26 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:03:26.829 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:26 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:03:26.829 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:26 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T11:03:26.829 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:26 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T11:03:26.829 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:26 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:03:26.829 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:26 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:03:26.829 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:26 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:03:26.829 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:26 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T11:03:26.829 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:26 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T11:03:26.830 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:26 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:03:26.833 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:26 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:03:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:26 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:03:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:26 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T11:03:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:26 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T11:03:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:26 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:03:26.868 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:26.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:26.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:26.970 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:27.000 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:27.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:27.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:27.102 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:27.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:27.137 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:27.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:27.169 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7, 4] out_osds: [2, 5, 0] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 7, 6] 2024-09-21T11:03:27.169 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T11:03:27.169 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-09-21T11:03:27.169 INFO:tasks.cephadm.osd.5:Restarting osd.5 (starting--it wasn't running)... 2024-09-21T11:03:27.170 DEBUG:teuthology.orchestra.run.smithi160:osd.5> sudo journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.5.service 2024-09-21T11:03:27.172 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl start ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.5 2024-09-21T11:03:27.226 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:03:27.238 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:27.245 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:03:27.287 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:27.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:27.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:27.337 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 89 16 144932 144916 101.754 74.1875 0.00587524 0.00982279 2024-09-21T11:03:27.389 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:27.431 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:27.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:27.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:27.533 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:27.562 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:27 smithi160 systemd[1]: Starting Ceph osd.5 for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T11:03:27.572 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:27.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:27.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:27.674 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:27.706 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:27.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:27.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:27.808 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:27 smithi160 ceph-mon[101879]: pgmap v3131: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.7 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 92 MiB/s wr, 1.47k op/s; 84166/427206 objects degraded (19.702%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T11:03:27.841 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:27 smithi160 podman[237658]: 2024-09-21 11:03:27.483200376 +0000 UTC m=+0.030658665 image pull 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6 2024-09-21T11:03:27.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:27.842 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:27.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:27.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:27 smithi070 ceph-mon[107385]: pgmap v3131: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.7 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 92 MiB/s wr, 1.47k op/s; 84166/427206 objects degraded (19.702%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T11:03:27.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:27 smithi070 ceph-mon[112139]: pgmap v3131: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.7 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 92 MiB/s wr, 1.47k op/s; 84166/427206 objects degraded (19.702%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T11:03:27.943 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:27.972 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:27.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:27.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:28.074 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:28.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:28.118 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:28.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:28.219 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:28.261 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:28.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:28.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:28.337 INFO:tasks.radosbench.radosbench.0.smithi070.stdout: 90 2 145956 145954 101.344 64.875 0.0344957 0.00986291 2024-09-21T11:03:28.363 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:28.384 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:28.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:28.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:28.486 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:28.508 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:28.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:28.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:28.610 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:28.630 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:28.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:28.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:28.732 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:28.754 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:28.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:28.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:28.803 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:03:28.803 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:03:28.803 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:03:28.803 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:03:28.803 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:03:28.803 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T11:03:28.804 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:03:28.804 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:03:28.804 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:03:28.804 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:03:28.804 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:03:28.804 INFO:teuthology.orchestra.run.smithi070.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T11:03:28.804 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T11:03:28.804 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:03:28.804 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:03:28.805 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:28 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/705624761' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:03:28.805 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:28 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 84166/427206 objects degraded (19.702%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:03:28.805 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:28 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/705624761' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:03:28.805 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:28 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 84166/427206 objects degraded (19.702%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:03:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:28 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/705624761' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:03:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:28 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 84166/427206 objects degraded (19.702%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:03:28.841 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:28 smithi160 podman[237658]: 2024-09-21 11:03:28.467372949 +0000 UTC m=+1.014831238 container create 3832d863b8c2bbd235853fe5bd25754fe3ee862f6e4aec4ec54be1756847d42d (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5-activate, org.label-schema.license=GPLv2, RELEASE=reef-6a1bdf8, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-21T11:03:28.856 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:28.877 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:28.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:28.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:28.978 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:28.999 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:29.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:29.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:29.101 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:29.122 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:29.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:29.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:29.224 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:29.245 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:29.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:29.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:29.337 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Total time run: 90.0199 2024-09-21T11:03:29.338 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Total writes made: 145956 2024-09-21T11:03:29.338 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Write size: 65536 2024-09-21T11:03:29.338 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Object size: 65536 2024-09-21T11:03:29.338 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Bandwidth (MB/sec): 101.336 2024-09-21T11:03:29.338 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Stddev Bandwidth: 51.0083 2024-09-21T11:03:29.338 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Max bandwidth (MB/sec): 305.938 2024-09-21T11:03:29.338 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Min bandwidth (MB/sec): 0 2024-09-21T11:03:29.338 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Average IOPS: 1621 2024-09-21T11:03:29.338 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Stddev IOPS: 816.133 2024-09-21T11:03:29.338 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Max IOPS: 4895 2024-09-21T11:03:29.339 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Min IOPS: 0 2024-09-21T11:03:29.339 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Average Latency(s): 0.00986325 2024-09-21T11:03:29.339 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Stddev Latency(s): 0.0244472 2024-09-21T11:03:29.339 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Max latency(s): 2.2389 2024-09-21T11:03:29.339 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Min latency(s): 0.00100109 2024-09-21T11:03:29.341 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T11:03:29.346 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:29.386 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:29.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:29.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:29.488 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:29.513 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:29.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:29.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:29.569 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T11:03:29.569 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T11:03:29.591 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:29 smithi160 podman[237658]: 2024-09-21 11:03:29.295911016 +0000 UTC m=+1.843369305 container init 3832d863b8c2bbd235853fe5bd25754fe3ee862f6e4aec4ec54be1756847d42d (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5-activate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=reef-6a1bdf8, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True) 2024-09-21T11:03:29.591 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:29 smithi160 podman[237658]: 2024-09-21 11:03:29.302912699 +0000 UTC m=+1.850370988 container start 3832d863b8c2bbd235853fe5bd25754fe3ee862f6e4aec4ec54be1756847d42d (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5-activate, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-6a1bdf8, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-21T11:03:29.591 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:29 smithi160 podman[237658]: 2024-09-21 11:03:29.412554697 +0000 UTC m=+1.960012996 container attach 3832d863b8c2bbd235853fe5bd25754fe3ee862f6e4aec4ec54be1756847d42d (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5-activate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-6a1bdf8, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-21T11:03:29.615 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:29.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:29.657 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:29.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:29.758 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:29.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:29.795 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:29.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:29.811 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:29 smithi070 ceph-mon[107385]: pgmap v3132: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.7 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 60 MiB/s wr, 964 op/s; 84166/427206 objects degraded (19.702%); 1.4 MiB/s, 21 objects/s recovering 2024-09-21T11:03:29.811 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:29 smithi070 ceph-mon[112139]: pgmap v3132: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.7 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 60 MiB/s wr, 964 op/s; 84166/427206 objects degraded (19.702%); 1.4 MiB/s, 21 objects/s recovering 2024-09-21T11:03:29.874 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:29 smithi160 ceph-mon[101879]: pgmap v3132: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.7 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 60 MiB/s wr, 964 op/s; 84166/427206 objects degraded (19.702%); 1.4 MiB/s, 21 objects/s recovering 2024-09-21T11:03:29.875 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:29 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5-activate[237766]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-21T11:03:29.875 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:29 smithi160 bash[237658]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-21T11:03:29.875 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:29 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5-activate[237766]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-21T11:03:29.875 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:29 smithi160 bash[237658]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-21T11:03:29.875 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:29 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5-activate[237766]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-21T11:03:29.875 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:29 smithi160 bash[237658]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-21T11:03:29.875 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:29 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5-activate[237766]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-21T11:03:29.875 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:29 smithi160 bash[237658]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-21T11:03:29.876 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:29 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5-activate[237766]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-21T11:03:29.876 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:29 smithi160 bash[237658]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-21T11:03:29.876 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:29 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5-activate[237766]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-21T11:03:29.876 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:29 smithi160 bash[237658]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-21T11:03:29.876 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:29 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5-activate[237766]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-21T11:03:29.876 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:29 smithi160 bash[237658]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-21T11:03:29.895 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:29.921 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:29.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:29.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:30.023 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:30.046 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:30.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:30.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:30.128 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:29 smithi160 podman[237658]: 2024-09-21 11:03:29.874256487 +0000 UTC m=+2.421714785 container died 3832d863b8c2bbd235853fe5bd25754fe3ee862f6e4aec4ec54be1756847d42d (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5-activate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=reef-6a1bdf8, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-21T11:03:30.147 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:30.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:30.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:30.275 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:30.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:30.310 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:30.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:30.411 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:30.453 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:30.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:30.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:30.554 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:30.654 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:30.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:30.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:30.757 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:30.798 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:30.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:30.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:30.900 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:30.931 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:30 smithi160 podman[237951]: 2024-09-21 11:03:30.614021872 +0000 UTC m=+0.729151952 container remove 3832d863b8c2bbd235853fe5bd25754fe3ee862f6e4aec4ec54be1756847d42d (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5-activate, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=reef-6a1bdf8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-21T11:03:30.942 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:30.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:30.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:31.043 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:31.070 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:31.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:31.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:31.172 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:31.198 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:31.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:31.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:31.283 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:30 smithi160 podman[238058]: 2024-09-21 11:03:30.835681914 +0000 UTC m=+0.027981453 image pull 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6 2024-09-21T11:03:31.283 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:30 smithi160 podman[238058]: 2024-09-21 11:03:30.968837698 +0000 UTC m=+0.161137217 container create 15bbf79846c01e94c9b0c2a2d137a4f2d09076f47610505782d875fe87051ac2 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=reef-6a1bdf8, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-21T11:03:31.283 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:31 smithi160 podman[238058]: 2024-09-21 11:03:31.280602394 +0000 UTC m=+0.472901931 container init 15bbf79846c01e94c9b0c2a2d137a4f2d09076f47610505782d875fe87051ac2 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=reef-6a1bdf8, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS) 2024-09-21T11:03:31.300 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:31.325 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:31.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:31.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:31.414 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T11:03:31.427 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:31.453 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:31.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:31.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:31.555 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:31.570 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T11:03:31.583 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:31.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:31.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:31.591 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:31 smithi160 podman[238058]: 2024-09-21 11:03:31.293516407 +0000 UTC m=+0.485815926 container start 15bbf79846c01e94c9b0c2a2d137a4f2d09076f47610505782d875fe87051ac2 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=reef-6a1bdf8, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-21T11:03:31.591 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:31 smithi160 bash[238058]: 15bbf79846c01e94c9b0c2a2d137a4f2d09076f47610505782d875fe87051ac2 2024-09-21T11:03:31.591 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:31 smithi160 systemd[1]: Started Ceph osd.5 for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T11:03:31.685 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:31.740 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:31.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:31.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:31.774 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:03:31.843 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:31.871 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:31.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:31.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:31 smithi070 ceph-mon[107385]: pgmap v3133: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.8 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 85 B/s rd, 71 MiB/s wr, 1.15k op/s; 85252/433236 objects degraded (19.678%); 2.3 MiB/s, 37 objects/s recovering 2024-09-21T11:03:31.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:31 smithi070 ceph-mon[112139]: pgmap v3133: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.8 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 85 B/s rd, 71 MiB/s wr, 1.15k op/s; 85252/433236 objects degraded (19.678%); 2.3 MiB/s, 37 objects/s recovering 2024-09-21T11:03:31.913 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:31 smithi160 ceph-mon[101879]: pgmap v3133: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.8 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 85 B/s rd, 71 MiB/s wr, 1.15k op/s; 85252/433236 objects degraded (19.678%); 2.3 MiB/s, 37 objects/s recovering 2024-09-21T11:03:31.913 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:31 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:03:31.608+0000 7f86c9407640 -1 Falling back to public interface 2024-09-21T11:03:31.974 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:32.013 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:32.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:32.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:32.114 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:32.140 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:32.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:32.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:32.242 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:32.308 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:32.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:32.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:32.410 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:32.444 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:32.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:32.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:32.545 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:32.572 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:32.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:32.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:32.674 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:32.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:32.708 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:32.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:32.808 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:32.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:32.841 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:32.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:32.941 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:32.975 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:32.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:32.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:33.077 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:33.098 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:33.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:33.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:33.200 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:33.233 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:33.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:33.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:33.335 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:33.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:33.365 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:33.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:33.465 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:33.491 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:33.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:33.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:33.593 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:33.609 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:03:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:03:33] "GET /metrics HTTP/1.1" 200 36577 "" "Prometheus/2.51.0" 2024-09-21T11:03:33.622 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:33.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:33.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:33.642 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:33 smithi160 ceph-mon[101879]: pgmap v3134: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.9 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 85 B/s rd, 65 MiB/s wr, 1.11k op/s; 85544/435258 objects degraded (19.654%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T11:03:33.643 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:33 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1630535568' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:03:33.643 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:33 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 85544/435258 objects degraded (19.654%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:03:33.724 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:33.750 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:33.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:33.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:33.833 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T11:03:33.852 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:33.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:33 smithi070 ceph-mon[107385]: pgmap v3134: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.9 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 85 B/s rd, 65 MiB/s wr, 1.11k op/s; 85544/435258 objects degraded (19.654%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T11:03:33.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:33 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1630535568' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:03:33.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:33 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 85544/435258 objects degraded (19.654%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:03:33.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:33 smithi070 ceph-mon[112139]: pgmap v3134: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.9 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 85 B/s rd, 65 MiB/s wr, 1.11k op/s; 85544/435258 objects degraded (19.654%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T11:03:33.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:33 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1630535568' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:03:33.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:33 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 85544/435258 objects degraded (19.654%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:03:33.880 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:33.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:33.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:33.982 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:34.026 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:34.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:34.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:34.099 INFO:teuthology.orchestra.run.smithi160.stderr:no valid command found; 10 closest matches: 2024-09-21T11:03:34.099 INFO:teuthology.orchestra.run.smithi160.stderr:0 2024-09-21T11:03:34.099 INFO:teuthology.orchestra.run.smithi160.stderr:1 2024-09-21T11:03:34.100 INFO:teuthology.orchestra.run.smithi160.stderr:2 2024-09-21T11:03:34.100 INFO:teuthology.orchestra.run.smithi160.stderr:abort 2024-09-21T11:03:34.100 INFO:teuthology.orchestra.run.smithi160.stderr:assert 2024-09-21T11:03:34.100 INFO:teuthology.orchestra.run.smithi160.stderr:bluefs debug_inject_read_zeros 2024-09-21T11:03:34.100 INFO:teuthology.orchestra.run.smithi160.stderr:bluefs files list 2024-09-21T11:03:34.100 INFO:teuthology.orchestra.run.smithi160.stderr:bluefs stats 2024-09-21T11:03:34.100 INFO:teuthology.orchestra.run.smithi160.stderr:bluestore bluefs device info [] 2024-09-21T11:03:34.100 INFO:teuthology.orchestra.run.smithi160.stderr:config diff 2024-09-21T11:03:34.100 INFO:teuthology.orchestra.run.smithi160.stderr:admin_socket: invalid command 2024-09-21T11:03:34.128 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:34.161 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:34.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:34.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:34.263 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:34.297 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:34.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:34.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:34.399 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:34.425 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:34.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:34.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:34.526 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:34.565 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:34.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:34.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:34.666 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:34.702 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:34.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:34.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:34.805 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:34.837 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:34.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:34.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:34.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:34 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:03:34.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:34 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1630535568' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T11:03:34.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:34 smithi070 ceph-mon[107385]: osdmap e1595: 8 total, 7 up, 5 in 2024-09-21T11:03:34.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:34 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:03:34.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:34 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1630535568' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T11:03:34.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:34 smithi070 ceph-mon[112139]: osdmap e1595: 8 total, 7 up, 5 in 2024-09-21T11:03:34.940 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:34.965 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:34.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:34.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:35.067 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:35.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:34 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:03:35.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:34 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1630535568' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T11:03:35.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:34 smithi160 ceph-mon[101879]: osdmap e1595: 8 total, 7 up, 5 in 2024-09-21T11:03:35.092 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:35.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:35.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:35.194 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:35.218 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:35.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:35.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:35.319 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:35.355 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:35.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:35.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:35.457 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:35.483 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:35.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:35.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:35.584 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:35.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:35.620 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:35.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:35.721 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:35.749 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:35.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:35.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:35.851 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:35 smithi070 ceph-mon[107385]: pgmap v3136: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.9 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 102 B/s rd, 60 MiB/s wr, 1.04k op/s; 85544/435258 objects degraded (19.654%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T11:03:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:03:35.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:35 smithi070 ceph-mon[112139]: pgmap v3136: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.9 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 102 B/s rd, 60 MiB/s wr, 1.04k op/s; 85544/435258 objects degraded (19.654%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T11:03:35.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:03:35.878 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:35.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:35.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:35.979 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:36.000 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:36.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:36.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:35 smithi160 ceph-mon[101879]: pgmap v3136: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.9 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 102 B/s rd, 60 MiB/s wr, 1.04k op/s; 85544/435258 objects degraded (19.654%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T11:03:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:03:36.101 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:36.139 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:36.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:36.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:36.241 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:36.266 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:36.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:36.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:36.367 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:36.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:36.395 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:36.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:36.496 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:36.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:36.518 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:36.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:36.619 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:36.646 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:36.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:36.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:36.748 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:36.776 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:36.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:36.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:36.877 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:36.901 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:36.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:36.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:37.002 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:37.040 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:37.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:37.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:37.142 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:37.163 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T11:03:37.164 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-09-21T11:03:37.165 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:37.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:37.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:37.267 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:37.288 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:37.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:37.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:37.390 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:37.438 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:37.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:37.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:37.540 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:37.583 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:37.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:37.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:37.685 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:37.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:37.721 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:37.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:37.822 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:37.859 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:37.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:37.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:37.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:37 smithi070 ceph-mon[112139]: pgmap v3137: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.5 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 102 B/s rd, 22 MiB/s wr, 987 op/s; 82328/419064 objects degraded (19.646%); 2.1 MiB/s, 32 objects/s recovering 2024-09-21T11:03:37.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:37 smithi070 ceph-mon[107385]: pgmap v3137: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.5 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 102 B/s rd, 22 MiB/s wr, 987 op/s; 82328/419064 objects degraded (19.646%); 2.1 MiB/s, 32 objects/s recovering 2024-09-21T11:03:37.961 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:37.998 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:37.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:37.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:38.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:37 smithi160 ceph-mon[101879]: pgmap v3137: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.5 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 102 B/s rd, 22 MiB/s wr, 987 op/s; 82328/419064 objects degraded (19.646%); 2.1 MiB/s, 32 objects/s recovering 2024-09-21T11:03:38.099 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:38.125 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:38.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:38.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:38.227 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:38.253 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:38.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:38.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:38.355 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:38.384 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:38.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:38.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:38.485 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:38.504 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T11:03:38.504 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T11:03:38.530 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:38.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:38.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:38.632 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:38.658 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:38.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:38.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:38.760 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:38.785 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:38.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:38.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:38.887 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:38.913 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:38.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:38.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:39.014 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:39.042 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:39.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:39.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:38 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 82328/419064 objects degraded (19.646%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:03:39.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:38 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 82328/419064 objects degraded (19.646%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:03:39.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:38 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 82328/419064 objects degraded (19.646%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T11:03:39.144 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:39.182 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:39.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:39.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:39.284 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:39.315 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:39.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:39.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:39.320 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T11:03:39.418 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:39.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:39.459 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:39.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:39.560 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:39.587 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:39.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:39.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:39.689 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:39.722 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:39.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:39.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:39.824 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:39.852 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:39.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:39.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:39.889 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:39 smithi070 ceph-mon[107385]: pgmap v3138: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.5 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 102 B/s rd, 22 MiB/s wr, 987 op/s; 82328/419064 objects degraded (19.646%); 2.1 MiB/s, 32 objects/s recovering 2024-09-21T11:03:39.890 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:39 smithi070 ceph-mon[112139]: pgmap v3138: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.5 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 102 B/s rd, 22 MiB/s wr, 987 op/s; 82328/419064 objects degraded (19.646%); 2.1 MiB/s, 32 objects/s recovering 2024-09-21T11:03:39.949 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:03:39.955 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:39.987 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:39.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:39.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:40.089 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:40.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:39 smithi160 ceph-mon[101879]: pgmap v3138: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.5 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 102 B/s rd, 22 MiB/s wr, 987 op/s; 82328/419064 objects degraded (19.646%); 2.1 MiB/s, 32 objects/s recovering 2024-09-21T11:03:40.092 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:39 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:03:39.620+0000 7f86c9407640 -1 osd.5 1534 log_to_monitors true 2024-09-21T11:03:40.123 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:40.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:40.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:40.224 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:40.250 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:40.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:40.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:40.352 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:40.382 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:40.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:40.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:40.484 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:40.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:40.513 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:40.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:40.614 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:40.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:40.645 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:40.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:40.745 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:40.758 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:03:40.767 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:40.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:40.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:40.869 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:40.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:40 smithi070 ceph-mon[107385]: from='osd.5 [v2:172.21.15.160:6816/1520073476,v1:172.21.15.160:6817/1520073476]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T11:03:40.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:40 smithi070 ceph-mon[107385]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T11:03:40.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:40 smithi070 ceph-mon[112139]: from='osd.5 [v2:172.21.15.160:6816/1520073476,v1:172.21.15.160:6817/1520073476]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T11:03:40.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:40 smithi070 ceph-mon[112139]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T11:03:40.891 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:40.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:40.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:40.993 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:41.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:41.015 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:41.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:41.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:40 smithi160 ceph-mon[101879]: from='osd.5 [v2:172.21.15.160:6816/1520073476,v1:172.21.15.160:6817/1520073476]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T11:03:41.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:40 smithi160 ceph-mon[101879]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T11:03:41.115 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:41.140 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:41.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:41.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:41.242 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:41.269 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:41.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:41.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:41.371 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:41.406 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:41.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:41.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:41.508 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:41.548 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:41.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:41.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:41.591 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:03:41 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:03:41.194+0000 7f86bd1a9640 -1 osd.5 1596 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T11:03:41.650 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:41.679 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:41.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:41.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:41.780 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:41.808 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:41.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:41.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:41.909 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:41.920 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T11:03:41.946 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:41 smithi070 ceph-mon[107385]: pgmap v3139: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.4 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 9.2 MiB/s wr, 1.01k op/s; 80649/411762 objects degraded (19.586%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T11:03:41.946 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:41 smithi070 ceph-mon[107385]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T11:03:41.946 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:41 smithi070 ceph-mon[107385]: osdmap e1596: 8 total, 7 up, 5 in 2024-09-21T11:03:41.946 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:41 smithi070 ceph-mon[107385]: from='osd.5 [v2:172.21.15.160:6816/1520073476,v1:172.21.15.160:6817/1520073476]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T11:03:41.946 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:41 smithi070 ceph-mon[107385]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T11:03:41.946 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:41 smithi070 ceph-mon[107385]: from='osd.5 ' entity='osd.5' 2024-09-21T11:03:41.947 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:41 smithi070 ceph-mon[112139]: pgmap v3139: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.4 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 9.2 MiB/s wr, 1.01k op/s; 80649/411762 objects degraded (19.586%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T11:03:41.947 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:41 smithi070 ceph-mon[112139]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T11:03:41.947 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:41 smithi070 ceph-mon[112139]: osdmap e1596: 8 total, 7 up, 5 in 2024-09-21T11:03:41.947 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:41 smithi070 ceph-mon[112139]: from='osd.5 [v2:172.21.15.160:6816/1520073476,v1:172.21.15.160:6817/1520073476]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T11:03:41.947 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:41 smithi070 ceph-mon[112139]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T11:03:41.947 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:41 smithi070 ceph-mon[112139]: from='osd.5 ' entity='osd.5' 2024-09-21T11:03:41.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:41.952 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:41.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:42.053 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:42.079 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:42.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:42.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:42.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:41 smithi160 ceph-mon[101879]: pgmap v3139: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.4 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 9.2 MiB/s wr, 1.01k op/s; 80649/411762 objects degraded (19.586%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T11:03:42.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:41 smithi160 ceph-mon[101879]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T11:03:42.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:41 smithi160 ceph-mon[101879]: osdmap e1596: 8 total, 7 up, 5 in 2024-09-21T11:03:42.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:41 smithi160 ceph-mon[101879]: from='osd.5 [v2:172.21.15.160:6816/1520073476,v1:172.21.15.160:6817/1520073476]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T11:03:42.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:41 smithi160 ceph-mon[101879]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-21T11:03:42.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:41 smithi160 ceph-mon[101879]: from='osd.5 ' entity='osd.5' 2024-09-21T11:03:42.164 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T11:03:42.182 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:42.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:42.246 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:42.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:42.347 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:42.374 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:42.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:42.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:42.476 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:42.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:42.524 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:42.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:42.531 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:03:42.626 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:42.653 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:42.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:42.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:42.754 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:42.764 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:03:42.789 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:42.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:42.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:42.890 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:42.932 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:42.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:42.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:43.034 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:43.059 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:43.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:43.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:43.160 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:43.190 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:43.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:43.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:43.291 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:43.323 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:43.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:43.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:43.328 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:43 smithi070 ceph-mon[107385]: osd.5 [v2:172.21.15.160:6816/1520073476,v1:172.21.15.160:6817/1520073476] boot 2024-09-21T11:03:43.328 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:43 smithi070 ceph-mon[107385]: osdmap e1597: 8 total, 8 up, 5 in 2024-09-21T11:03:43.328 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:43 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T11:03:43.328 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:43 smithi070 ceph-mon[107385]: pgmap v3142: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.2 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 0 B/s wr, 1.29k op/s; 79385/404214 objects degraded (19.639%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T11:03:43.329 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:43 smithi070 ceph-mon[112139]: osd.5 [v2:172.21.15.160:6816/1520073476,v1:172.21.15.160:6817/1520073476] boot 2024-09-21T11:03:43.329 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:43 smithi070 ceph-mon[112139]: osdmap e1597: 8 total, 8 up, 5 in 2024-09-21T11:03:43.329 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:43 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T11:03:43.329 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:43 smithi070 ceph-mon[112139]: pgmap v3142: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.2 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 0 B/s wr, 1.29k op/s; 79385/404214 objects degraded (19.639%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T11:03:43.329 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:03:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:03:43] "GET /metrics HTTP/1.1" 200 36583 "" "Prometheus/2.51.0" 2024-09-21T11:03:43.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:43 smithi160 ceph-mon[101879]: osd.5 [v2:172.21.15.160:6816/1520073476,v1:172.21.15.160:6817/1520073476] boot 2024-09-21T11:03:43.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:43 smithi160 ceph-mon[101879]: osdmap e1597: 8 total, 8 up, 5 in 2024-09-21T11:03:43.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:43 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T11:03:43.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:43 smithi160 ceph-mon[101879]: pgmap v3142: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.2 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 0 B/s wr, 1.29k op/s; 79385/404214 objects degraded (19.639%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T11:03:43.373 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:03:43.373 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:03:43.404 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:03:43.405 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:03:43.424 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:43.428 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:03:43.428 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:03:43.454 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:03:43.454 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:03:43.455 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:43.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:43.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:43.482 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:03:43.482 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:03:43.506 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T11:03:43.507 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T11:03:43.534 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T11:03:43.534 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T11:03:43.557 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:43.562 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:03:43.562 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:03:43.587 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:43.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:43.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:43.689 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:43.717 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:43.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:43.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:43.819 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:43.846 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:43.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:43.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:43.948 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:43.988 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:43.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:43.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:44.089 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:44.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:44.134 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:44.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:44.234 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:44.260 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:44.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:44.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:44.361 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:44.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:44 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/284121830' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:03:44.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:44 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 79385/404214 objects degraded (19.639%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T11:03:44.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:44 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/284121830' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:03:44.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:44 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 79385/404214 objects degraded (19.639%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T11:03:44.399 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:44.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:44.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:44.500 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:44.509 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:44 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/284121830' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:03:44.509 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:44 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 79385/404214 objects degraded (19.639%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T11:03:44.529 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:44.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:44.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:44.567 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T11:03:44.567 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T11:03:44.567 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T11:03:44.567 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T11:03:44.630 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:44.661 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:44.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:44.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:44.763 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:44.803 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:44.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:44.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:44.905 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:44.945 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:44.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:44.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:45.046 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:45.075 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:45.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:45.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:45.177 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:45.203 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:45.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:45.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:45.304 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:45.331 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:45 smithi070 ceph-mon[107385]: pgmap v3143: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.2 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 0 B/s wr, 618 op/s; 79385/404214 objects degraded (19.639%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T11:03:45.331 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:45 smithi070 ceph-mon[112139]: pgmap v3143: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.2 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 0 B/s wr, 618 op/s; 79385/404214 objects degraded (19.639%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T11:03:45.333 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:45.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:45.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:45.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:45 smithi160 ceph-mon[101879]: pgmap v3143: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 8.2 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 0 B/s wr, 618 op/s; 79385/404214 objects degraded (19.639%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T11:03:45.435 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:45.461 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:45.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:45.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:45.562 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:45.599 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:45.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:45.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:45.701 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:45.728 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:45.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:45.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:45.830 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:45.862 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:45.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:45.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:45.964 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:46.003 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:46.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:46.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:46.104 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:46.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:46.145 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:46.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:46.246 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:46.273 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:46.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:46.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:46.375 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:03:46.409 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:46.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:46.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:03:46.422 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T11:03:46.510 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:46.540 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:46.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:46.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:46.642 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:46.683 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:46.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:46.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:46.785 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:46.818 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:46.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:46.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:46.916 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T11:03:46.916 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T11:03:46.916 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T11:03:46.916 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [] 2024-09-21T11:03:46.916 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T11:03:46.920 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:46.948 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:46.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:46.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:47.050 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:03:47.088 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:47.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:47.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:03:47.190 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:47.213 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:47.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:47.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:47.315 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:47.342 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:47.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:47.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:47.363 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T11:03:47.444 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:03:47.482 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:47.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:47.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:03:47.584 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:47.618 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:47.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:47.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:47.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:47 smithi070 ceph-mon[112139]: pgmap v3144: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.9 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 0 B/s wr, 1.36k op/s; 75899/386358 objects degraded (19.645%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T11:03:47.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:47 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2607536884' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:03:47.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:47 smithi070 ceph-mon[107385]: pgmap v3144: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.9 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 0 B/s wr, 1.36k op/s; 75899/386358 objects degraded (19.645%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T11:03:47.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:47 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2607536884' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:03:47.720 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:47.746 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:47.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:47.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:47.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:47 smithi160 ceph-mon[101879]: pgmap v3144: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.9 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 0 B/s wr, 1.36k op/s; 75899/386358 objects degraded (19.645%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T11:03:47.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:47 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2607536884' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:03:47.847 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:47.882 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:47.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:47.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:47.918 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T11:03:47.984 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:48.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:48.023 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:48.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:48.115 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:03:48.124 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:48.157 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:48.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:48.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:48.258 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:48.284 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:48.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:48.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:48.386 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:48.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:48.414 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:48.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:48.515 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:48.551 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:48.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:48.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:48.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:48 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:03:48.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:48 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2607536884' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T11:03:48.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:48 smithi070 ceph-mon[112139]: osdmap e1598: 8 total, 8 up, 5 in 2024-09-21T11:03:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:48 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:03:48.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:48 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2607536884' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T11:03:48.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:48 smithi070 ceph-mon[107385]: osdmap e1598: 8 total, 8 up, 5 in 2024-09-21T11:03:48.653 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:48.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:48.683 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:48.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:48.785 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:48.819 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:48.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:48.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:48 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:03:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:48 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2607536884' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T11:03:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:48 smithi160 ceph-mon[101879]: osdmap e1598: 8 total, 8 up, 5 in 2024-09-21T11:03:48.921 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:48.952 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:48.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:48.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:49.054 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:49.082 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:49.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:49.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:49.183 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:49.212 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:49.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:49.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:49.313 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:49.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:49.342 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:49.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:49.409 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:03:49.443 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:49.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:49.481 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:49.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:49.534 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:49 smithi070 ceph-mon[107385]: pgmap v3146: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.9 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 0 B/s wr, 1.10k op/s; 75899/386358 objects degraded (19.645%); 936 KiB/s, 14 objects/s recovering 2024-09-21T11:03:49.534 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:49 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 75899/386358 objects degraded (19.645%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T11:03:49.534 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:49 smithi070 ceph-mon[112139]: pgmap v3146: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.9 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 0 B/s wr, 1.10k op/s; 75899/386358 objects degraded (19.645%); 936 KiB/s, 14 objects/s recovering 2024-09-21T11:03:49.534 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:49 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 75899/386358 objects degraded (19.645%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T11:03:49.582 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:03:49.621 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:49.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:49.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:03:49.723 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:49.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:49.752 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:49.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:49 smithi160 ceph-mon[101879]: pgmap v3146: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.9 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 0 B/s wr, 1.10k op/s; 75899/386358 objects degraded (19.645%); 936 KiB/s, 14 objects/s recovering 2024-09-21T11:03:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:49 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 75899/386358 objects degraded (19.645%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T11:03:49.853 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:49.887 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:49.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:49.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:49.989 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:50.018 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:50.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:50.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:50.120 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:50.147 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:50.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:50.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:50.249 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:50.284 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:50.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:50.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:50.386 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:50.398 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7, 4] out_osds: [2, 5, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 7, 6, 5] 2024-09-21T11:03:50.398 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T11:03:50.398 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T11:03:50.398 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.601213 2024-09-21T11:03:50.398 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 1 0.6012133668307378 2024-09-21T11:03:50.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:50.417 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:50.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:50.518 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:50.549 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:50.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:50.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:50.651 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:50.680 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:50.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:50.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:50.715 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T11:03:50.782 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:50.824 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:50.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:50.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:50.925 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:50.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:50.958 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:50.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:51.059 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:51.100 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:51.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:51.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:51.202 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:51.231 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:51.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:51.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:51.332 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:51.357 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:51.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:51.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:51.459 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:03:51.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:51.499 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:51.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:03:51.600 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:51 smithi070 ceph-mon[107385]: pgmap v3147: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.7 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 0 B/s wr, 1.11k op/s; 73783/377001 objects degraded (19.571%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T11:03:51.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:03:51.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:51 smithi070 ceph-mon[112139]: pgmap v3147: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.7 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 0 B/s wr, 1.11k op/s; 73783/377001 objects degraded (19.571%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T11:03:51.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:03:51.628 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:51.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:51.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:51.730 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:51.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:51.766 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:51.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:51 smithi160 ceph-mon[101879]: pgmap v3147: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.7 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 0 B/s wr, 1.11k op/s; 73783/377001 objects degraded (19.571%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T11:03:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:03:51.867 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:03:51.909 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:51.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:51.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:03:52.011 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:52.048 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:52.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:52.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:52.150 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:03:52.184 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:52.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:52.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:03:52.286 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:52.323 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:52.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:52.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:52.424 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:52.456 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:52.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:52.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:52.558 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:52.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:52.593 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:52.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:52.694 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:03:52.729 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:52.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:52.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:03:52.831 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:52.862 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:52.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:52.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:52.964 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:52.992 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:52.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:52.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:53.094 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:53.124 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:53.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:53.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:53.225 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:53.249 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:53.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:53.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:53.351 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:53.379 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:53.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:53.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:53.481 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:53.507 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:53.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:53.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:53.609 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:53.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:03:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:03:53] "GET /metrics HTTP/1.1" 200 36732 "" "Prometheus/2.51.0" 2024-09-21T11:03:53.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:53 smithi070 ceph-mon[107385]: pgmap v3148: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.5 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 0 B/s wr, 1.23k op/s; 72170/367242 objects degraded (19.652%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T11:03:53.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:53 smithi070 ceph-mon[112139]: pgmap v3148: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.5 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 0 B/s wr, 1.23k op/s; 72170/367242 objects degraded (19.652%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T11:03:53.639 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:53.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:53.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:53.741 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:53.780 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:53.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:53.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:53 smithi160 ceph-mon[101879]: pgmap v3148: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.5 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 0 B/s wr, 1.23k op/s; 72170/367242 objects degraded (19.652%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T11:03:53.882 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:53.914 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:53.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:53.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:54.015 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:54.037 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:54.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:54.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:54.139 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:54.173 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:54.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:54.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:54.275 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:03:54.303 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:54.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:54.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:03:54.404 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:54.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:54.436 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:54.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:54.478 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:54 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 72170/367242 objects degraded (19.652%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T11:03:54.479 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:54 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/27418562' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:03:54.479 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:54 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 72170/367242 objects degraded (19.652%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T11:03:54.479 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:54 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/27418562' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:03:54.536 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:54.571 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:54.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:54.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:54.673 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:54.707 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:54.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:54.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:54.809 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:54.832 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:54.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:54.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:54 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 72170/367242 objects degraded (19.652%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T11:03:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:54 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/27418562' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:03:54.933 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:54.972 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:54.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:54.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:55.074 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:55.105 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:55.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:55.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:55.206 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:55.233 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:55.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:55.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:55.335 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:55.371 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:55.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:55.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:55.417 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T11:03:55.418 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T11:03:55.418 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.418 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:171880 58.9 58:9604123c:::benchmark_data_smithi070_396992_object25921:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1598)", 2024-09-21T11:03:55.418 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:03:51.993675+0000", 2024-09-21T11:03:55.418 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.027575039999999999, 2024-09-21T11:03:55.418 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.027600725999999999, 2024-09-21T11:03:55.418 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:03:55.418 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:03:55.418 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "waiting for sub ops", 2024-09-21T11:03:55.418 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:03:55.418 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:03:55.418 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:03:55.419 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 171880 2024-09-21T11:03:55.419 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.419 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:03:55.419 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.419 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:03:55.419 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:51.993675+0000", 2024-09-21T11:03:55.419 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:03:55.419 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.419 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.419 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:03:55.419 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:51.993675+0000", 2024-09-21T11:03:55.419 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:03:55.419 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.420 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.420 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:03:55.420 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:51.993676+0000", 2024-09-21T11:03:55.420 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.689e-06 2024-09-21T11:03:55.420 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.420 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.420 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:03:55.420 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:51.993681+0000", 2024-09-21T11:03:55.420 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0119999999999996e-06 2024-09-21T11:03:55.420 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.420 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.420 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:03:55.420 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:51.993696+0000", 2024-09-21T11:03:55.421 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4756000000000001e-05 2024-09-21T11:03:55.421 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.421 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.421 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:03:55.421 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:51.993704+0000", 2024-09-21T11:03:55.421 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.571e-06 2024-09-21T11:03:55.421 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.421 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.421 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:03:55.421 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.001331+0000", 2024-09-21T11:03:55.421 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0076269989999999998 2024-09-21T11:03:55.421 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.421 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.421 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:03:55.421 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.001464+0000", 2024-09-21T11:03:55.422 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000133118 2024-09-21T11:03:55.422 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.422 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.422 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6", 2024-09-21T11:03:55.422 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.001588+0000", 2024-09-21T11:03:55.422 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000124268 2024-09-21T11:03:55.422 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:03:55.422 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:03:55.422 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:03:55.422 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.422 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.422 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:171884 58.9 58:9c967d22:::benchmark_data_smithi070_396992_object25925:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1598)", 2024-09-21T11:03:55.423 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:03:52.000106+0000", 2024-09-21T11:03:55.423 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.021143617999999999, 2024-09-21T11:03:55.423 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.021212715, 2024-09-21T11:03:55.423 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:03:55.423 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:03:55.423 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "waiting for sub ops", 2024-09-21T11:03:55.423 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:03:55.423 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:03:55.423 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:03:55.423 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 171884 2024-09-21T11:03:55.423 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.423 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:03:55.424 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.424 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:03:55.424 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.000106+0000", 2024-09-21T11:03:55.424 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:03:55.424 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.424 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.424 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:03:55.424 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.000106+0000", 2024-09-21T11:03:55.424 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:03:55.424 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.424 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.424 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:03:55.425 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.000107+0000", 2024-09-21T11:03:55.425 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4100000000000001e-06 2024-09-21T11:03:55.425 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.425 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.425 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:03:55.425 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.000113+0000", 2024-09-21T11:03:55.425 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5119999999999999e-06 2024-09-21T11:03:55.425 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.425 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.425 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:03:55.425 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.000127+0000", 2024-09-21T11:03:55.425 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4331000000000001e-05 2024-09-21T11:03:55.425 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.426 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.426 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:03:55.426 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.000135+0000", 2024-09-21T11:03:55.426 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7179999999999996e-06 2024-09-21T11:03:55.426 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.426 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.426 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:03:55.426 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.006640+0000", 2024-09-21T11:03:55.426 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0065048800000000002 2024-09-21T11:03:55.426 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.426 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.426 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:03:55.427 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.006782+0000", 2024-09-21T11:03:55.427 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014204300000000001 2024-09-21T11:03:55.427 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.427 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.427 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6", 2024-09-21T11:03:55.427 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.006871+0000", 2024-09-21T11:03:55.427 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9266999999999998e-05 2024-09-21T11:03:55.427 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:03:55.427 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:03:55.427 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:03:55.427 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.427 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.428 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:171899 58.9 58:93fdf65a:::benchmark_data_smithi070_396992_object25940:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1598)", 2024-09-21T11:03:55.428 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:03:52.008811+0000", 2024-09-21T11:03:55.428 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.012438947000000001, 2024-09-21T11:03:55.428 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012543772999999999, 2024-09-21T11:03:55.428 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:03:55.428 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:03:55.428 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "waiting for sub ops", 2024-09-21T11:03:55.428 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:03:55.428 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:03:55.428 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:03:55.428 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 171899 2024-09-21T11:03:55.428 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.428 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:03:55.429 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.429 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:03:55.429 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.008811+0000", 2024-09-21T11:03:55.429 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:03:55.429 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.429 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.429 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:03:55.429 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.008811+0000", 2024-09-21T11:03:55.429 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:03:55.429 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.429 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.429 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:03:55.429 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.008812+0000", 2024-09-21T11:03:55.430 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.6299999999999993e-07 2024-09-21T11:03:55.430 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.430 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.430 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:03:55.430 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.008816+0000", 2024-09-21T11:03:55.430 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2370000000000003e-06 2024-09-21T11:03:55.430 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.430 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.430 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:03:55.430 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.008827+0000", 2024-09-21T11:03:55.430 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.128e-05 2024-09-21T11:03:55.430 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.431 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.431 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:03:55.431 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.008834+0000", 2024-09-21T11:03:55.431 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8519999999999998e-06 2024-09-21T11:03:55.431 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.431 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.431 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:03:55.431 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.011611+0000", 2024-09-21T11:03:55.431 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0027774169999999999 2024-09-21T11:03:55.431 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.431 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.431 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:03:55.431 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.011739+0000", 2024-09-21T11:03:55.432 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000127281 2024-09-21T11:03:55.432 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.432 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.432 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6", 2024-09-21T11:03:55.432 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.011870+0000", 2024-09-21T11:03:55.432 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013112399999999999 2024-09-21T11:03:55.432 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:03:55.432 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:03:55.432 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:03:55.432 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.432 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.432 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:171901 58.9 58:9015d368:::benchmark_data_smithi070_396992_object25942:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1598)", 2024-09-21T11:03:55.432 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:03:52.008970+0000", 2024-09-21T11:03:55.432 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.012279201999999999, 2024-09-21T11:03:55.432 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012443602999999999, 2024-09-21T11:03:55.433 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:03:55.433 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:03:55.433 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "waiting for sub ops", 2024-09-21T11:03:55.433 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:03:55.433 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:03:55.433 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:03:55.433 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 171901 2024-09-21T11:03:55.433 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.433 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:03:55.433 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.433 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:03:55.433 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.008970+0000", 2024-09-21T11:03:55.433 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:03:55.433 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.433 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.434 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:03:55.434 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.008970+0000", 2024-09-21T11:03:55.434 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:03:55.434 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.434 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.434 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:03:55.434 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.008971+0000", 2024-09-21T11:03:55.434 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8299999999999996e-07 2024-09-21T11:03:55.434 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.434 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.434 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:03:55.434 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.008974+0000", 2024-09-21T11:03:55.434 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8049999999999998e-06 2024-09-21T11:03:55.435 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.435 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.435 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:03:55.435 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.008986+0000", 2024-09-21T11:03:55.435 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2252000000000001e-05 2024-09-21T11:03:55.435 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.435 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.435 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:03:55.435 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.008992+0000", 2024-09-21T11:03:55.435 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0419999999999999e-06 2024-09-21T11:03:55.435 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.435 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.435 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:03:55.435 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.016749+0000", 2024-09-21T11:03:55.436 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00775685 2024-09-21T11:03:55.436 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.436 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.436 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:03:55.436 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.016856+0000", 2024-09-21T11:03:55.436 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010670699999999999 2024-09-21T11:03:55.436 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.436 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.436 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6", 2024-09-21T11:03:55.436 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.016949+0000", 2024-09-21T11:03:55.436 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3613e-05 2024-09-21T11:03:55.436 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:03:55.436 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:03:55.437 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:03:55.437 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.437 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.437 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop_reply(client.36567.0:171880 58.9 e1598/1590)", 2024-09-21T11:03:55.437 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:03:52.014346+0000", 2024-09-21T11:03:55.437 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0069036669999999996, 2024-09-21T11:03:55.437 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0071282589999999996, 2024-09-21T11:03:55.437 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:03:55.437 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:03:55.437 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "queued for pg", 2024-09-21T11:03:55.437 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:03:55.437 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.437 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:03:55.438 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.014346+0000", 2024-09-21T11:03:55.438 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:03:55.438 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.438 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.438 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:03:55.438 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.014346+0000", 2024-09-21T11:03:55.438 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:03:55.438 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.438 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.438 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:03:55.438 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.014347+0000", 2024-09-21T11:03:55.438 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1429999999999999e-06 2024-09-21T11:03:55.438 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.438 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.439 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:03:55.439 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.014353+0000", 2024-09-21T11:03:55.439 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7339999999999996e-06 2024-09-21T11:03:55.439 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.439 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.439 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:03:55.439 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.014366+0000", 2024-09-21T11:03:55.439 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3302000000000001e-05 2024-09-21T11:03:55.439 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.439 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.439 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:03:55.439 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.014382+0000", 2024-09-21T11:03:55.439 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6140000000000001e-05 2024-09-21T11:03:55.440 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:03:55.440 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:03:55.440 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:03:55.440 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.440 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.440 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:171909 58.9 58.8a5918d9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1598)", 2024-09-21T11:03:55.440 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:03:52.015113+0000", 2024-09-21T11:03:55.440 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0061367210000000004, 2024-09-21T11:03:55.440 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0064021700000000004, 2024-09-21T11:03:55.440 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:03:55.440 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:03:55.440 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "queued for pg", 2024-09-21T11:03:55.440 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:03:55.440 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:03:55.441 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:03:55.441 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 171909 2024-09-21T11:03:55.441 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.441 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:03:55.441 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.441 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:03:55.441 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.015113+0000", 2024-09-21T11:03:55.441 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:03:55.441 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.441 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.441 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:03:55.441 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.015113+0000", 2024-09-21T11:03:55.441 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:03:55.442 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.442 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.442 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:03:55.442 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.015114+0000", 2024-09-21T11:03:55.442 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.438e-06 2024-09-21T11:03:55.442 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.442 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.442 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:03:55.442 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.015119+0000", 2024-09-21T11:03:55.442 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7679999999999999e-06 2024-09-21T11:03:55.442 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.442 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.442 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:03:55.442 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.015131+0000", 2024-09-21T11:03:55.443 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1894e-05 2024-09-21T11:03:55.443 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:03:55.443 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:03:55.443 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:03:55.443 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:52.015141+0000", 2024-09-21T11:03:55.443 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0468e-05 2024-09-21T11:03:55.443 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:03:55.443 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:03:55.443 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:03:55.443 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:03:55.443 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-09-21T11:03:55.443 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 6 2024-09-21T11:03:55.443 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T11:03:55.473 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:55.503 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:55.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:55.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:55.604 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:03:55.639 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:55.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:55.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:03:55.740 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:55.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:55 smithi160 ceph-mon[101879]: pgmap v3149: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.5 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 0 B/s wr, 1.23k op/s; 72170/367242 objects degraded (19.652%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T11:03:55.780 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:55.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:55.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:55.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:55 smithi070 ceph-mon[107385]: pgmap v3149: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.5 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 0 B/s wr, 1.23k op/s; 72170/367242 objects degraded (19.652%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T11:03:55.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:55 smithi070 ceph-mon[112139]: pgmap v3149: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.5 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 0 B/s wr, 1.23k op/s; 72170/367242 objects degraded (19.652%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T11:03:55.882 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:55.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:55.920 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:55.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:56.022 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:56.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:56.057 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:56.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:56.158 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:03:56.187 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:56.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:56.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:03:56.289 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:56.318 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:56.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:56.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:56.419 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:56.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:56.447 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:56.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:56.548 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:03:56.579 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:56.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:56.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:03:56.680 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:56.707 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:56.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:56.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:56.809 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:56.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:56.848 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:56.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:56.949 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:56.980 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:56.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:56.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:57.081 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:57.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:57.126 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:57.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:57.227 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:57.257 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:57.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:57.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:57.359 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:57.386 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:57.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:57.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:57.419 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:03:57.488 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:03:57.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:57.517 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:57.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:03:57.618 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:57.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:57 smithi070 ceph-mon[107385]: pgmap v3150: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.1 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 1.26k op/s; 68555/348522 objects degraded (19.670%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T11:03:57.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:57 smithi070 ceph-mon[112139]: pgmap v3150: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.1 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 1.26k op/s; 68555/348522 objects degraded (19.670%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T11:03:57.649 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:57.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:57.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:57.751 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:57.782 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:57.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:57.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:57 smithi160 ceph-mon[101879]: pgmap v3150: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.1 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 1.26k op/s; 68555/348522 objects degraded (19.670%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T11:03:57.883 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:57.912 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:57.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:57.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:58.013 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:58.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:58.042 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:58.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:58.143 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:58.181 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:58.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:58.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:58.283 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:03:58.304 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:58.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:58.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:03:58.406 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:58.442 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:58.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:58.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:58.518 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:03:58.519 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:03:58.519 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:03:58.519 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:03:58.519 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:03:58.519 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T11:03:58.519 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T11:03:58.519 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:03:58.519 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:03:58.519 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:03:58.519 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:03:58.520 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:03:58.520 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:03:58.520 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T11:03:58.520 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T11:03:58.520 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:03:58.544 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:58.570 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:58.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:58.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:58.672 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:03:58.698 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:58.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:58.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:03:58.800 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:03:58.823 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:58.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:58.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:03:58.925 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:58.954 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:58.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:58.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:59.056 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:03:59.077 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:03:59.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:59.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:03:59.179 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:03:59.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:59.218 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:59.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:03:59.319 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:03:59.340 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:59.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:59.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:03:59.441 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:03:59.480 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:59.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:59.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:03:59.581 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:59.617 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:59.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:59.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:59.719 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:59.751 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:59.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:59.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:59.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:59 smithi160 ceph-mon[101879]: pgmap v3151: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.1 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 1.15k op/s; 68555/348522 objects degraded (19.670%); 1.4 MiB/s, 23 objects/s recovering 2024-09-21T11:03:59.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:03:59 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 68555/348522 objects degraded (19.670%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T11:03:59.853 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:03:59.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:59 smithi070 ceph-mon[107385]: pgmap v3151: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.1 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 1.15k op/s; 68555/348522 objects degraded (19.670%); 1.4 MiB/s, 23 objects/s recovering 2024-09-21T11:03:59.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:03:59 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 68555/348522 objects degraded (19.670%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T11:03:59.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:59 smithi070 ceph-mon[112139]: pgmap v3151: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 7.1 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 1.15k op/s; 68555/348522 objects degraded (19.670%); 1.4 MiB/s, 23 objects/s recovering 2024-09-21T11:03:59.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:03:59 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 68555/348522 objects degraded (19.670%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T11:03:59.887 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:03:59.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:03:59.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:03:59.989 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:00.015 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:00.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:00.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:00.117 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:00.148 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:00.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:00.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:00.250 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:00.277 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:00.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:00.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:00.379 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:00.406 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:00.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:00.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:00.508 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:00.546 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:00.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:00.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:00.647 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:00.680 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:00.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:00.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:00.781 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:00.819 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:00.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:00.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:00.923 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:00.955 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:00.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:00.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:01.057 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:01.089 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:01.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:01.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:01.191 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:01.222 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:01.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:01.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:01.318 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T11:04:01.323 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:01.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:01.357 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:01.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:01.458 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:01.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:01.492 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:01.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:01.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:01 smithi070 ceph-mon[112139]: pgmap v3152: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.9 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 1.27k op/s; 66712/340434 objects degraded (19.596%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T11:04:01.547 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:01 smithi070 ceph-mon[107385]: pgmap v3152: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.9 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 1.27k op/s; 66712/340434 objects degraded (19.596%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T11:04:01.594 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:01.621 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:01.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:01.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:01.723 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:01.756 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:01.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:01.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:01 smithi160 ceph-mon[101879]: pgmap v3152: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.9 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 1.27k op/s; 66712/340434 objects degraded (19.596%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T11:04:01.858 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:01.887 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:01.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:01.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:01.988 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:02.015 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:02.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:02.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:02.087 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:04:02.116 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:02.133 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:04:02.157 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:02.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:02.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:02.262 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:02.289 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:02.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:02.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:02.391 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:02.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:02.434 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:02.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:02.535 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:02.561 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:02.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:02.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:02.663 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:02.697 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:02.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:02.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:02.799 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:02.836 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:02.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:02.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:02.908 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T11:04:02.937 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:02.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:02.975 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:02.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:03.076 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:03.097 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:03.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:03.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:03.199 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:04:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:04:03] "GET /metrics HTTP/1.1" 200 36732 "" "Prometheus/2.51.0" 2024-09-21T11:04:03.199 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:03.249 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:03.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:03.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:03.351 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:03.379 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:03.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:03.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:03.480 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:03.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:03.520 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:03.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:03.621 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:03.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:03 smithi070 ceph-mon[107385]: pgmap v3153: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.8 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s wr, 1.25k op/s; 65293/331938 objects degraded (19.670%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T11:04:03.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:03 smithi070 ceph-mon[112139]: pgmap v3153: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.8 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s wr, 1.25k op/s; 65293/331938 objects degraded (19.670%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T11:04:03.655 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:03.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:03.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:03.757 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:03.782 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:03.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:03.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:03 smithi160 ceph-mon[101879]: pgmap v3153: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.8 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s wr, 1.25k op/s; 65293/331938 objects degraded (19.670%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T11:04:03.884 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:03.909 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:03.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:03.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:04.010 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:04.051 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:04.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:04.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:04.152 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:04.197 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:04.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:04.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:04.299 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:04.333 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:04.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:04.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:04.435 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:04.464 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:04.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:04.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:04.566 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:04.592 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:04.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:04.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:04.619 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:04 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 65293/331938 objects degraded (19.670%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:04.619 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:04 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 65293/331938 objects degraded (19.670%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:04.694 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:04.720 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:04.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:04.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:04.822 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:04.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:04 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 65293/331938 objects degraded (19.670%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:04.850 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:04.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:04.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:04.952 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:04.990 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:04.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:04.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:05.092 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:05.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:05.129 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:05.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:05.230 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:05.256 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:05.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:05.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:05.358 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:05.391 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:05.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:05.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:05.493 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:05.523 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:05.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:05.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:05.625 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:05.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:05.660 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:05.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:05.714 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:05 smithi070 ceph-mon[107385]: pgmap v3154: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.8 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s wr, 980 op/s; 65293/331938 objects degraded (19.670%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T11:04:05.714 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:04:05.714 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:05 smithi070 ceph-mon[112139]: pgmap v3154: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.8 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s wr, 980 op/s; 65293/331938 objects degraded (19.670%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T11:04:05.715 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:04:05.761 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:05.790 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:05.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:05.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:05 smithi160 ceph-mon[101879]: pgmap v3154: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.8 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s wr, 980 op/s; 65293/331938 objects degraded (19.670%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T11:04:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:04:05.893 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:05.932 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:05.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:05.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:06.034 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:06.061 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:06.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:06.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:06.162 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:06.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:06.201 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:06.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:06.302 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:06.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:06.343 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:06.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:06.444 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:06.473 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:06.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:06.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:06.499 INFO:teuthology.orchestra.run.smithi070.stderr:set osd.1 primary-affinity to 0.601213 (8394012) 2024-09-21T11:04:06.575 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:06.599 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:06.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:06.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:06.702 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:06.723 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:06.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:06.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:06.825 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:06 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/403034609' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.6012133668307378} : dispatch 2024-09-21T11:04:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:06 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.6012133668307378} : dispatch 2024-09-21T11:04:06.848 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:06.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:06.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:06.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:06 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/403034609' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.6012133668307378} : dispatch 2024-09-21T11:04:06.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:06 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.6012133668307378} : dispatch 2024-09-21T11:04:06.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:06 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/403034609' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.6012133668307378} : dispatch 2024-09-21T11:04:06.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:06 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.6012133668307378} : dispatch 2024-09-21T11:04:06.950 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:06.974 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:06.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:06.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:07.076 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:07.098 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:07.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:07.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:07.200 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:07.223 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:07.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:07.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:07.324 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:07.347 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:07.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:07.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:07.449 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:07.470 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:07.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:07.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:07.516 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T11:04:07.571 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:07.599 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:07.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:07.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:07.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:07 smithi070 ceph-mon[107385]: pgmap v3155: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.4 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 0 B/s wr, 1.49k op/s; 61649/313476 objects degraded (19.666%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T11:04:07.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:07 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.6012133668307378}]': finished 2024-09-21T11:04:07.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:07 smithi070 ceph-mon[107385]: osdmap e1599: 8 total, 8 up, 5 in 2024-09-21T11:04:07.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:07 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3680084070' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:04:07.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:07 smithi070 ceph-mon[112139]: pgmap v3155: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.4 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 0 B/s wr, 1.49k op/s; 61649/313476 objects degraded (19.666%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T11:04:07.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:07 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.6012133668307378}]': finished 2024-09-21T11:04:07.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:07 smithi070 ceph-mon[112139]: osdmap e1599: 8 total, 8 up, 5 in 2024-09-21T11:04:07.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:07 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3680084070' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:04:07.701 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:07.723 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:07.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:07.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:07.825 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:07 smithi160 ceph-mon[101879]: pgmap v3155: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.4 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 0 B/s wr, 1.49k op/s; 61649/313476 objects degraded (19.666%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T11:04:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:07 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.6012133668307378}]': finished 2024-09-21T11:04:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:07 smithi160 ceph-mon[101879]: osdmap e1599: 8 total, 8 up, 5 in 2024-09-21T11:04:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:07 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3680084070' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:04:07.846 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:07.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:07.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:07.948 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:07.969 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:07.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:07.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:08.071 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:08.093 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:08.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:08.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:08.195 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:08.217 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:08.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:08.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:08.271 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T11:04:08.271 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T11:04:08.279 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:04:08.319 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:08.341 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:08.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:08.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:08.442 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:08.463 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:08.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:08.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:08.565 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:08.597 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:08.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:08.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:08.699 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:08.732 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:08.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:08.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:08.834 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:08 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T11:04:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:08 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3680084070' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T11:04:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:08 smithi160 ceph-mon[101879]: osdmap e1600: 8 total, 8 up, 5 in 2024-09-21T11:04:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:08 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 61649/313476 objects degraded (19.666%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:08.871 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:08.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:08.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:08.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:08 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T11:04:08.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:08 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3680084070' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T11:04:08.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:08 smithi070 ceph-mon[107385]: osdmap e1600: 8 total, 8 up, 5 in 2024-09-21T11:04:08.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:08 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 61649/313476 objects degraded (19.666%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:08.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:08 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T11:04:08.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:08 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3680084070' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T11:04:08.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:08 smithi070 ceph-mon[112139]: osdmap e1600: 8 total, 8 up, 5 in 2024-09-21T11:04:08.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:08 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 61649/313476 objects degraded (19.666%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:08.973 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:09.001 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:09.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:09.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:09.103 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:09.127 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:09.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:09.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:09.229 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:09.260 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:09.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:09.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:09.362 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:09.382 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:09.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:09.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:09.484 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:09.510 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:09.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:09.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:09.612 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:09.644 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:09.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:09.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:09.745 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:09.782 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:09.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:09.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:09.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:09 smithi160 ceph-mon[101879]: pgmap v3158: 109 pgs: 3 peering, 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 96 active+clean; 6.4 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 0 B/s wr, 1.12k op/s; 61649/313476 objects degraded (19.666%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T11:04:09.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:09 smithi160 ceph-mon[101879]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-21T11:04:09.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:09 smithi160 ceph-mon[101879]: osdmap e1601: 8 total, 8 up, 5 in 2024-09-21T11:04:09.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:09 smithi160 ceph-mon[101879]: 58.d continuing backfill to osd.6 from (1594'8000,1598'11814] MIN to 1598'11814 2024-09-21T11:04:09.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:09 smithi070 ceph-mon[107385]: pgmap v3158: 109 pgs: 3 peering, 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 96 active+clean; 6.4 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 0 B/s wr, 1.12k op/s; 61649/313476 objects degraded (19.666%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T11:04:09.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:09 smithi070 ceph-mon[107385]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-21T11:04:09.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:09 smithi070 ceph-mon[107385]: osdmap e1601: 8 total, 8 up, 5 in 2024-09-21T11:04:09.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:09 smithi070 ceph-mon[107385]: 58.d continuing backfill to osd.6 from (1594'8000,1598'11814] MIN to 1598'11814 2024-09-21T11:04:09.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:09 smithi070 ceph-mon[112139]: pgmap v3158: 109 pgs: 3 peering, 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 96 active+clean; 6.4 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 0 B/s wr, 1.12k op/s; 61649/313476 objects degraded (19.666%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T11:04:09.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:09 smithi070 ceph-mon[112139]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-21T11:04:09.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:09 smithi070 ceph-mon[112139]: osdmap e1601: 8 total, 8 up, 5 in 2024-09-21T11:04:09.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:09 smithi070 ceph-mon[112139]: 58.d continuing backfill to osd.6 from (1594'8000,1598'11814] MIN to 1598'11814 2024-09-21T11:04:09.884 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:09.910 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:09.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:09.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:10.012 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:10.036 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:10.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:10.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:10.138 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:10.172 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:10.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:10.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:10.274 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:10.300 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:10.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:10.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:10.402 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:10.429 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:10.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:10.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:10.530 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:10.557 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:10.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:10.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:10.658 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:10.694 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:10.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:10.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:10.796 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:10.818 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:10.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:10.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:10.919 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:10.951 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:10.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:10.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:11.053 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:11.082 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:11.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:11.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:11.184 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:11.212 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:11.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:11.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:11.313 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:11.340 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:11.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:11.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:11.441 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:11.466 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:11.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:11.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:11.568 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:11.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:11.609 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:11.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:11.710 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:11.733 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:11.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:11.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:11.835 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:11 smithi160 ceph-mon[101879]: pgmap v3160: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 3 peering, 9 active+undersized+degraded+remapped+backfill_wait, 96 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 0 B/s wr, 1.31k op/s; 60422/308376 objects degraded (19.594%); 4.6 MiB/s, 72 objects/s recovering 2024-09-21T11:04:11.861 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:11.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:11.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:11.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:11 smithi070 ceph-mon[107385]: pgmap v3160: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 3 peering, 9 active+undersized+degraded+remapped+backfill_wait, 96 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 0 B/s wr, 1.31k op/s; 60422/308376 objects degraded (19.594%); 4.6 MiB/s, 72 objects/s recovering 2024-09-21T11:04:11.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:11 smithi070 ceph-mon[112139]: pgmap v3160: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 3 peering, 9 active+undersized+degraded+remapped+backfill_wait, 96 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 0 B/s wr, 1.31k op/s; 60422/308376 objects degraded (19.594%); 4.6 MiB/s, 72 objects/s recovering 2024-09-21T11:04:11.963 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:11.987 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:11.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:11.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:12.088 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:12.114 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:12.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:12.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:12.216 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:12.241 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:12.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:12.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:12.343 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:12.364 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7, 4] out_osds: [2, 5, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 7, 6, 5] 2024-09-21T11:04:12.364 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T11:04:12.364 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-21T11:04:12.364 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd test-reweight-by-utilization 110 1.0 2024-09-21T11:04:12.380 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:12.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:12.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:12.482 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:12.515 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:12.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:12.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:12.616 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:12.644 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:12.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:12.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:12.746 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:12.771 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:12.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:12.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:12.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:12 smithi160 ceph-mon[101879]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-21T11:04:12.873 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:12.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:12 smithi070 ceph-mon[107385]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-21T11:04:12.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:12 smithi070 ceph-mon[112139]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-21T11:04:12.901 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:12.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:12.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:13.003 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:13.028 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:13.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:13.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:13.130 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:13.156 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:13.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:13.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:13.258 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:13.292 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:13.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:13.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:13.393 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:13.431 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:13.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:13.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:13.438 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:04:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:04:13] "GET /metrics HTTP/1.1" 200 36727 "" "Prometheus/2.51.0" 2024-09-21T11:04:13.533 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:13.562 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:13.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:13.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:13.664 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:13.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:13.703 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:13.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:13.804 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:13.830 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:13.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:13.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:13 smithi160 ceph-mon[101879]: pgmap v3161: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.1 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 12 KiB/s rd, 37 KiB/s wr, 659 op/s; 59199/301773 objects degraded (19.617%); 3.2 MiB/s, 50 objects/s recovering 2024-09-21T11:04:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:13 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 59199/301773 objects degraded (19.617%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:13.869 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:13 smithi070 ceph-mon[107385]: pgmap v3161: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.1 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 12 KiB/s rd, 37 KiB/s wr, 659 op/s; 59199/301773 objects degraded (19.617%); 3.2 MiB/s, 50 objects/s recovering 2024-09-21T11:04:13.869 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:13 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 59199/301773 objects degraded (19.617%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:13.869 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:13 smithi070 ceph-mon[112139]: pgmap v3161: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.1 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 12 KiB/s rd, 37 KiB/s wr, 659 op/s; 59199/301773 objects degraded (19.617%); 3.2 MiB/s, 50 objects/s recovering 2024-09-21T11:04:13.870 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:13 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 59199/301773 objects degraded (19.617%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:13.898 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T11:04:13.931 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:13.970 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:13.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:13.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:14.072 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:14.101 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:14.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:14.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:14.203 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:14.228 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:14.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:14.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:14.329 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:14.330 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:04:14.370 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:14.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:14.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:14.472 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:14.500 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:14.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:14.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:14.602 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:14.633 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:14.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:14.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:14.735 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:14.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:14.763 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:14.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:14.864 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:14.891 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:14.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:14.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:14.993 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:15.020 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:15.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:15.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:15.121 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:15.155 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:15.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:15.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:15.257 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:15.270 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T11:04:15.292 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:15.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:15.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:15.394 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:15.425 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:15.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:15.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:15.527 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:15.556 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:15.556 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:15 smithi070 ceph-mon[107385]: pgmap v3162: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.1 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 9.6 KiB/s rd, 28 KiB/s wr, 505 op/s; 59199/301773 objects degraded (19.617%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T11:04:15.557 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:15 smithi070 ceph-mon[112139]: pgmap v3162: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.1 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 9.6 KiB/s rd, 28 KiB/s wr, 505 op/s; 59199/301773 objects degraded (19.617%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T11:04:15.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:15.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:15.660 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:15.686 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:15.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:15.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:15.788 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:15.825 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:15.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:15.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:15 smithi160 ceph-mon[101879]: pgmap v3162: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 6.1 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 9.6 KiB/s rd, 28 KiB/s wr, 505 op/s; 59199/301773 objects degraded (19.617%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T11:04:15.927 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:15.955 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:15.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:15.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:16.057 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:16.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:16.089 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:16.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:16.190 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:16.218 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:16.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:16.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:16.320 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:16.354 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:16.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:16.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:16.456 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:16.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:16.495 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:16.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:16.591 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:04:16.596 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:16.648 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:04:16.655 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:16.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:16.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:16.757 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:16.786 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:16.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:16.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:16.888 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:16.918 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:16.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:16.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:17.020 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:17.047 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:17.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:17.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:17.148 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:17.176 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:17.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:17.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:17.278 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:17.304 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:17.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:17.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:17.406 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:17.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:17.442 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:17.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:17.543 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:17.574 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:17.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:17.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:17.676 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:17.715 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:17.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:17.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:17.729 INFO:teuthology.orchestra.run.smithi070.stdout:moved 55 / 327 (16.8196%) 2024-09-21T11:04:17.729 INFO:teuthology.orchestra.run.smithi070.stdout:avg 65.4 2024-09-21T11:04:17.729 INFO:teuthology.orchestra.run.smithi070.stdout:stddev 24.4344 -> 27.8223 (expected baseline 7.23326) 2024-09-21T11:04:17.729 INFO:teuthology.orchestra.run.smithi070.stdout:min osd.3 with 17 -> 21 pgs (0.259939 -> 0.321101 * mean) 2024-09-21T11:04:17.730 INFO:teuthology.orchestra.run.smithi070.stdout:max osd.1 with 80 -> 89 pgs (1.22324 -> 1.36086 * mean) 2024-09-21T11:04:17.730 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T11:04:17.730 INFO:teuthology.orchestra.run.smithi070.stdout:oload 110 2024-09-21T11:04:17.730 INFO:teuthology.orchestra.run.smithi070.stdout:max_change 1 2024-09-21T11:04:17.730 INFO:teuthology.orchestra.run.smithi070.stdout:max_change_osds 4 2024-09-21T11:04:17.730 INFO:teuthology.orchestra.run.smithi070.stdout:average_utilization 0.0443 2024-09-21T11:04:17.730 INFO:teuthology.orchestra.run.smithi070.stdout:overload_utilization 0.0487 2024-09-21T11:04:17.730 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 weight 0.8683 -> 1.0000 2024-09-21T11:04:17.730 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 weight 1.0000 -> 0.6806 2024-09-21T11:04:17.730 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 weight 0.2979 -> 0.4266 2024-09-21T11:04:17.730 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 weight 1.0000 -> 0.8206 2024-09-21T11:04:17.731 INFO:teuthology.orchestra.run.smithi070.stderr:no change 2024-09-21T11:04:17.817 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:17.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:17 smithi160 ceph-mon[101879]: pgmap v3163: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 5.8 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 8.5 KiB/s rd, 25 KiB/s wr, 1.15k op/s; 55593/283230 objects degraded (19.628%); 3.2 MiB/s, 50 objects/s recovering 2024-09-21T11:04:17.844 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:17.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:17.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:17.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:04:17 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T11:04:17.727+0000 7fd088d1d640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-09-21T11:04:17.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:17 smithi070 ceph-mon[107385]: pgmap v3163: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 5.8 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 8.5 KiB/s rd, 25 KiB/s wr, 1.15k op/s; 55593/283230 objects degraded (19.628%); 3.2 MiB/s, 50 objects/s recovering 2024-09-21T11:04:17.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:17 smithi070 ceph-mon[112139]: pgmap v3163: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 5.8 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 8.5 KiB/s rd, 25 KiB/s wr, 1.15k op/s; 55593/283230 objects degraded (19.628%); 3.2 MiB/s, 50 objects/s recovering 2024-09-21T11:04:17.946 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:17.973 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:17.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:17.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:18.074 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:18.101 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:18.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:18.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:18.202 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:18.243 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:18.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:18.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:18.345 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:18.372 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:18.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:18.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:18.474 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:18.505 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:18.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:18.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:18.607 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:18.635 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:18.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:18.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:18.737 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:18.762 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:18.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:18.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:18 smithi160 ceph-mon[101879]: from='client.46643 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T11:04:18.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:18 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 55593/283230 objects degraded (19.628%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:18.864 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:18.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:18 smithi070 ceph-mon[107385]: from='client.46643 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T11:04:18.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:18 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 55593/283230 objects degraded (19.628%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:18.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:18 smithi070 ceph-mon[112139]: from='client.46643 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T11:04:18.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:18 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 55593/283230 objects degraded (19.628%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:18.896 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:18.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:18.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:18.998 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:19.027 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:19.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:19.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:19.129 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:19.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:19.167 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:19.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:19.268 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:19.294 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:19.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:19.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:19.396 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:19.422 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:19.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:19.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:19.523 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:19.563 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:19.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:19.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:19.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:19 smithi070 ceph-mon[107385]: pgmap v3164: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 5.8 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 7.5 KiB/s rd, 22 KiB/s wr, 1.01k op/s; 55593/283230 objects degraded (19.628%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T11:04:19.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:19 smithi070 ceph-mon[112139]: pgmap v3164: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 5.8 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 7.5 KiB/s rd, 22 KiB/s wr, 1.01k op/s; 55593/283230 objects degraded (19.628%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T11:04:19.666 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:19.699 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:19.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:19.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:19.801 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:19.828 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:19.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:19.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:19 smithi160 ceph-mon[101879]: pgmap v3164: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 5.8 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 7.5 KiB/s rd, 22 KiB/s wr, 1.01k op/s; 55593/283230 objects degraded (19.628%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T11:04:19.930 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:19.964 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:19.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:19.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:20.065 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:20.092 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:20.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:20.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:20.194 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:20.225 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:20.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:20.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:20.326 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:20.360 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:20.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:20.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:20.461 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:20.498 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:20.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:20.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:20.600 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:20.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:20.636 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:20.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:20.737 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:20.768 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:20.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:20.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:20.862 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-21T11:04:20.862 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 110 1.0 2024-09-21T11:04:20.870 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:20.936 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:20.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:20.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:21.038 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:21.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:21.081 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:21.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:21.182 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:21.210 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:21.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:21.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:21.312 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:21.344 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:21.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:21.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:21.446 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:21.491 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:21.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:21.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:21.593 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:21.621 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:21.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:21.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:21.722 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:21.760 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:21.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:21.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:21 smithi160 ceph-mon[101879]: pgmap v3165: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 5.6 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 6.4 KiB/s rd, 19 KiB/s wr, 1.10k op/s; 53844/274695 objects degraded (19.601%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T11:04:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:04:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:04:21.875 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:21 smithi070 ceph-mon[107385]: pgmap v3165: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 5.6 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 6.4 KiB/s rd, 19 KiB/s wr, 1.10k op/s; 53844/274695 objects degraded (19.601%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T11:04:21.875 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:04:21.875 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:04:21.875 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:21 smithi070 ceph-mon[112139]: pgmap v3165: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 5.6 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 6.4 KiB/s rd, 19 KiB/s wr, 1.10k op/s; 53844/274695 objects degraded (19.601%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T11:04:21.876 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:04:21.876 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:04:21.891 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:21.927 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:21.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:21.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:22.029 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:22.066 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:22.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:22.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:22.167 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:22.171 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:04:22.171 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:04:22.197 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:04:22.197 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:04:22.208 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:22.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:22.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:22.221 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:04:22.221 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:04:22.247 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:04:22.247 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:04:22.271 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:04:22.271 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:04:22.297 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T11:04:22.298 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T11:04:22.310 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:22.321 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T11:04:22.322 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T11:04:22.339 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:22.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:22.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:22.357 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:04:22.357 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:04:22.431 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:22 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2227515414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:04:22.431 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:22 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2227515414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:04:22.442 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:22.482 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:22.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:22.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:22.584 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:22.611 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:22.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:22.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:22.713 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:22.748 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:22.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:22.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:22 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2227515414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:04:22.849 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:22.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:22.888 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:22.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:22.989 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:23.025 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:23.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:23.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:23.127 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:23.159 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:23.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:23.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:23.260 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:23.293 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:23.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:23.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:23.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:04:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:04:23] "GET /metrics HTTP/1.1" 200 36737 "" "Prometheus/2.51.0" 2024-09-21T11:04:23.394 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:23.432 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T11:04:23.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:23.433 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:23.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:23.534 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:23.571 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:23.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:23.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:23.672 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:23.714 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:23.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:23.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:23.815 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:23.820 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:23 smithi070 ceph-mon[107385]: pgmap v3166: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 5.4 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 6.2 KiB/s rd, 18 KiB/s wr, 1.23k op/s; 51938/264171 objects degraded (19.661%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T11:04:23.821 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:23 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/81762417' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T11:04:23.821 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:23 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T11:04:23.821 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:23 smithi070 ceph-mon[112139]: pgmap v3166: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 5.4 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 6.2 KiB/s rd, 18 KiB/s wr, 1.23k op/s; 51938/264171 objects degraded (19.661%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T11:04:23.821 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:23 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/81762417' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T11:04:23.821 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:23 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T11:04:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:23 smithi160 ceph-mon[101879]: pgmap v3166: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 5.4 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 6.2 KiB/s rd, 18 KiB/s wr, 1.23k op/s; 51938/264171 objects degraded (19.661%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T11:04:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:23 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/81762417' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T11:04:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:23 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T11:04:23.847 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:23.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:23.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:23.949 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:23.976 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:23.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:23.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:24.078 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:24.106 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:24.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:24.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:24.208 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:24.250 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:24.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:24.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:24.352 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:24.387 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:24.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:24.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:24.489 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:24.516 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:24.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:24.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:24.526 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:24 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:04:24.526 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:24 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T11:04:24.526 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:24 smithi070 ceph-mon[107385]: osdmap e1602: 8 total, 8 up, 5 in 2024-09-21T11:04:24.527 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:24 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 51938/264171 objects degraded (19.661%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:24.527 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:24 smithi070 ceph-mon[107385]: osdmap e1603: 8 total, 8 up, 5 in 2024-09-21T11:04:24.527 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:24 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:04:24.527 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:24 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T11:04:24.527 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:24 smithi070 ceph-mon[112139]: osdmap e1602: 8 total, 8 up, 5 in 2024-09-21T11:04:24.527 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:24 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 51938/264171 objects degraded (19.661%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:24.527 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:24 smithi070 ceph-mon[112139]: osdmap e1603: 8 total, 8 up, 5 in 2024-09-21T11:04:24.618 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:24.654 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:24.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:24.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:24.757 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:24.784 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:24.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:24.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:24 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:04:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:24 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T11:04:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:24 smithi160 ceph-mon[101879]: osdmap e1602: 8 total, 8 up, 5 in 2024-09-21T11:04:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:24 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 51938/264171 objects degraded (19.661%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:24 smithi160 ceph-mon[101879]: osdmap e1603: 8 total, 8 up, 5 in 2024-09-21T11:04:24.886 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:24.914 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:24.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:24.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:25.016 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:25.045 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:25.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:25.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:25.120 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:04:25.134 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T11:04:25.134 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T11:04:25.134 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T11:04:25.134 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T11:04:25.134 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.134 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36012.0:66794 56.5 56:af9493d5:::benchmark_data_smithi070_333437_object66793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1490)", 2024-09-21T11:04:25.134 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:55:00.442566+0000", 2024-09-21T11:04:25.134 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 564.68753149999998, 2024-09-21T11:04:25.134 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2236836719999999, 2024-09-21T11:04:25.134 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:04:25.135 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:04:25.135 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:04:25.135 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:04:25.135 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36012", 2024-09-21T11:04:25.135 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T11:04:25.135 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 66794 2024-09-21T11:04:25.135 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.135 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:04:25.135 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.135 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:04:25.135 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442566+0000", 2024-09-21T11:04:25.135 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.135 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.136 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.136 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:04:25.136 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442566+0000", 2024-09-21T11:04:25.136 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.136 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.136 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.136 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:04:25.136 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442567+0000", 2024-09-21T11:04:25.136 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.8899999999999998e-07 2024-09-21T11:04:25.136 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.136 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.136 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:04:25.136 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442585+0000", 2024-09-21T11:04:25.137 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8022000000000001e-05 2024-09-21T11:04:25.137 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.137 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.137 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:04:25.137 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442596+0000", 2024-09-21T11:04:25.137 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0895e-05 2024-09-21T11:04:25.137 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.137 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.137 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:04:25.137 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442602+0000", 2024-09-21T11:04:25.137 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4019999999999999e-06 2024-09-21T11:04:25.137 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.137 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.138 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.138 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442671+0000", 2024-09-21T11:04:25.138 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8868999999999994e-05 2024-09-21T11:04:25.138 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.138 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.138 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:04:25.138 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442774+0000", 2024-09-21T11:04:25.138 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000102638 2024-09-21T11:04:25.138 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.138 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.138 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,7", 2024-09-21T11:04:25.138 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.442886+0000", 2024-09-21T11:04:25.138 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000111853 2024-09-21T11:04:25.138 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.139 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.139 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:04:25.139 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.445148+0000", 2024-09-21T11:04:25.139 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0022620380000000001 2024-09-21T11:04:25.139 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.139 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.139 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.139 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465200+0000", 2024-09-21T11:04:25.139 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.020052745 2024-09-21T11:04:25.139 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.139 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.139 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:04:25.139 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465211+0000", 2024-09-21T11:04:25.139 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0713999999999999e-05 2024-09-21T11:04:25.140 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.140 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.140 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.140 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:01.454645+0000", 2024-09-21T11:04:25.140 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.98943445699999999 2024-09-21T11:04:25.140 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.140 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.140 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:04:25.140 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:01.454664+0000", 2024-09-21T11:04:25.140 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8576999999999998e-05 2024-09-21T11:04:25.140 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.140 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.140 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.140 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.666160+0000", 2024-09-21T11:04:25.140 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2114961719999999 2024-09-21T11:04:25.141 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.141 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.141 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:04:25.141 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.666250+0000", 2024-09-21T11:04:25.141 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9400999999999993e-05 2024-09-21T11:04:25.141 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.141 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:04:25.141 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.141 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.141 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.141 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36012.0:66807 56.5 56:a8efccf3:::benchmark_data_smithi070_333437_object66806:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1491)", 2024-09-21T11:04:25.141 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:55:00.465619+0000", 2024-09-21T11:04:25.141 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 564.66447811900002, 2024-09-21T11:04:25.141 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2238977850000001, 2024-09-21T11:04:25.142 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:04:25.142 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:04:25.142 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:04:25.142 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:04:25.142 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36012", 2024-09-21T11:04:25.142 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T11:04:25.142 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 66807 2024-09-21T11:04:25.142 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.142 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:04:25.142 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.142 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:04:25.142 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465619+0000", 2024-09-21T11:04:25.142 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.143 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.143 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.143 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:04:25.143 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465619+0000", 2024-09-21T11:04:25.143 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.143 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.143 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.143 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:04:25.143 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465621+0000", 2024-09-21T11:04:25.143 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.841e-06 2024-09-21T11:04:25.143 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.143 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.143 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:04:25.144 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465673+0000", 2024-09-21T11:04:25.144 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1858999999999999e-05 2024-09-21T11:04:25.144 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.144 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.144 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:04:25.144 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465694+0000", 2024-09-21T11:04:25.144 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1259000000000001e-05 2024-09-21T11:04:25.144 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.144 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.144 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:04:25.144 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465715+0000", 2024-09-21T11:04:25.144 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1141999999999998e-05 2024-09-21T11:04:25.144 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.144 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.145 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.145 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465867+0000", 2024-09-21T11:04:25.145 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000151569 2024-09-21T11:04:25.145 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.145 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.145 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:04:25.145 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:00.465876+0000", 2024-09-21T11:04:25.145 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9139999999999997e-06 2024-09-21T11:04:25.145 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.145 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.145 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.145 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:01.454697+0000", 2024-09-21T11:04:25.145 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.98882113500000002 2024-09-21T11:04:25.146 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.146 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.146 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:04:25.146 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:01.454706+0000", 2024-09-21T11:04:25.146 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2080000000000006e-06 2024-09-21T11:04:25.146 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.146 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.146 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.146 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.670154+0000", 2024-09-21T11:04:25.146 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2154475469999999 2024-09-21T11:04:25.146 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.146 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.146 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:04:25.146 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.670320+0000", 2024-09-21T11:04:25.147 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016649899999999999 2024-09-21T11:04:25.147 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.147 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.147 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T11:04:25.147 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.670782+0000", 2024-09-21T11:04:25.147 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00046200700000000001 2024-09-21T11:04:25.147 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.147 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.147 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:04:25.147 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.677750+0000", 2024-09-21T11:04:25.147 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0069672229999999998 2024-09-21T11:04:25.147 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.147 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.148 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:25.148 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.678570+0000", 2024-09-21T11:04:25.148 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:25.149 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00082092899999999997 2024-09-21T11:04:25.149 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.149 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.149 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:25.150 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.685882+0000", 2024-09-21T11:04:25.150 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00731155 2024-09-21T11:04:25.150 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.150 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.150 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:04:25.150 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.685948+0000", 2024-09-21T11:04:25.150 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5481000000000005e-05 2024-09-21T11:04:25.150 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.150 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.150 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:04:25.150 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:02.689517+0000", 2024-09-21T11:04:25.150 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035696220000000002 2024-09-21T11:04:25.151 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.151 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:04:25.151 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.151 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.151 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.151 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36012.0:97176 56.5 56:a357387e:::benchmark_data_smithi070_333437_object97175:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1499)", 2024-09-21T11:04:25.151 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:55:37.706429+0000", 2024-09-21T11:04:25.151 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 527.42366838299995, 2024-09-21T11:04:25.151 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7010156360000002, 2024-09-21T11:04:25.151 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:04:25.151 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:04:25.151 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:04:25.151 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:04:25.151 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36012", 2024-09-21T11:04:25.152 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T11:04:25.152 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 97176 2024-09-21T11:04:25.152 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.152 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:04:25.152 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.152 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:04:25.152 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.706429+0000", 2024-09-21T11:04:25.152 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.152 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.152 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.152 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:04:25.152 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.706429+0000", 2024-09-21T11:04:25.152 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.153 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.153 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.153 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:04:25.153 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.706432+0000", 2024-09-21T11:04:25.153 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.146e-06 2024-09-21T11:04:25.153 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.153 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.153 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:04:25.153 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.706479+0000", 2024-09-21T11:04:25.153 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7190000000000001e-05 2024-09-21T11:04:25.153 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.153 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.154 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:04:25.154 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.706514+0000", 2024-09-21T11:04:25.154 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5027000000000002e-05 2024-09-21T11:04:25.154 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.154 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.154 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:04:25.154 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.706528+0000", 2024-09-21T11:04:25.154 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3933e-05 2024-09-21T11:04:25.154 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.154 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.154 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.154 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.797946+0000", 2024-09-21T11:04:25.154 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.091417409000000005 2024-09-21T11:04:25.155 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.155 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.155 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:04:25.155 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.797952+0000", 2024-09-21T11:04:25.155 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1009999999999997e-06 2024-09-21T11:04:25.155 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.155 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.155 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.155 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:38.361807+0000", 2024-09-21T11:04:25.155 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.56385527000000002 2024-09-21T11:04:25.155 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.155 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.155 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:04:25.155 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:38.361819+0000", 2024-09-21T11:04:25.156 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1719000000000001e-05 2024-09-21T11:04:25.156 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.156 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.156 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.156 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.407348+0000", 2024-09-21T11:04:25.156 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.045529425 2024-09-21T11:04:25.156 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.156 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.156 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:04:25.156 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.407445+0000", 2024-09-21T11:04:25.156 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.6416000000000007e-05 2024-09-21T11:04:25.156 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.156 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:04:25.157 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.157 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.157 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.157 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36012.0:97182 56.5 56:a2247821:::benchmark_data_smithi070_333437_object97181:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1499)", 2024-09-21T11:04:25.157 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T10:55:37.706622+0000", 2024-09-21T11:04:25.157 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 527.42347523299998, 2024-09-21T11:04:25.157 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.724287076, 2024-09-21T11:04:25.157 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:04:25.157 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:04:25.157 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:04:25.157 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:04:25.157 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36012", 2024-09-21T11:04:25.158 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T11:04:25.158 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 97182 2024-09-21T11:04:25.158 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.158 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:04:25.158 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.158 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:04:25.158 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.706622+0000", 2024-09-21T11:04:25.158 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.158 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.158 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.158 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:04:25.158 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.706622+0000", 2024-09-21T11:04:25.158 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.159 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.159 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.159 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:04:25.159 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.706623+0000", 2024-09-21T11:04:25.159 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5200000000000002e-07 2024-09-21T11:04:25.159 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.159 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.159 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:04:25.159 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.706650+0000", 2024-09-21T11:04:25.159 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6852e-05 2024-09-21T11:04:25.159 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.159 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.160 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:04:25.160 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.706660+0000", 2024-09-21T11:04:25.160 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0183999999999999e-05 2024-09-21T11:04:25.160 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.160 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.160 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:04:25.160 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.706665+0000", 2024-09-21T11:04:25.160 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6559999999999997e-06 2024-09-21T11:04:25.160 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.160 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.160 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.160 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.803733+0000", 2024-09-21T11:04:25.160 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.097067298999999996 2024-09-21T11:04:25.160 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.161 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.161 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:04:25.161 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:37.803749+0000", 2024-09-21T11:04:25.161 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6731999999999998e-05 2024-09-21T11:04:25.161 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.161 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.161 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.161 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:38.361843+0000", 2024-09-21T11:04:25.161 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.55809369399999997 2024-09-21T11:04:25.161 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.161 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.161 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:04:25.161 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:38.361850+0000", 2024-09-21T11:04:25.161 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0299999999999996e-06 2024-09-21T11:04:25.162 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.162 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.162 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.162 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.409049+0000", 2024-09-21T11:04:25.162 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0471991439999999 2024-09-21T11:04:25.162 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.162 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.162 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:04:25.162 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.409183+0000", 2024-09-21T11:04:25.162 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013339299999999999 2024-09-21T11:04:25.162 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.162 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.162 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T11:04:25.163 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.409487+0000", 2024-09-21T11:04:25.163 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00030424500000000003 2024-09-21T11:04:25.163 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.163 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.163 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:04:25.163 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.417173+0000", 2024-09-21T11:04:25.163 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0076856030000000001 2024-09-21T11:04:25.163 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.163 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.163 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:25.163 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.421310+0000", 2024-09-21T11:04:25.163 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.004137216 2024-09-21T11:04:25.163 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.164 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.164 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:25.164 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.428975+0000", 2024-09-21T11:04:25.164 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0076652789999999997 2024-09-21T11:04:25.164 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.164 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.164 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:04:25.164 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.429028+0000", 2024-09-21T11:04:25.164 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3059000000000001e-05 2024-09-21T11:04:25.164 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.164 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.164 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:04:25.164 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T10:55:39.430909+0000", 2024-09-21T11:04:25.165 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0018812379999999999 2024-09-21T11:04:25.165 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.165 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:04:25.165 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.165 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.165 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.165 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.46190.0:84902 57.3 57:cba41bdc:::benchmark_data_smithi070_360420_object17553:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1558)", 2024-09-21T11:04:25.165 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:00:40.417701+0000", 2024-09-21T11:04:25.165 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 224.71239622799999, 2024-09-21T11:04:25.165 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.17822523300000001, 2024-09-21T11:04:25.165 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:04:25.165 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:04:25.165 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:04:25.166 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:04:25.166 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.46190", 2024-09-21T11:04:25.166 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:04:25.166 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 84902 2024-09-21T11:04:25.166 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.166 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:04:25.166 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.166 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:04:25.166 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.417701+0000", 2024-09-21T11:04:25.166 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.166 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.166 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.166 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:04:25.166 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.417701+0000", 2024-09-21T11:04:25.167 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.167 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.167 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.167 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:04:25.167 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.417703+0000", 2024-09-21T11:04:25.167 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5719999999999999e-06 2024-09-21T11:04:25.167 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.167 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.167 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:04:25.167 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.417708+0000", 2024-09-21T11:04:25.167 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3299999999999998e-06 2024-09-21T11:04:25.167 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.167 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.168 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:04:25.168 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.417720+0000", 2024-09-21T11:04:25.168 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1884e-05 2024-09-21T11:04:25.168 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.169 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.169 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:04:25.169 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.417728+0000", 2024-09-21T11:04:25.169 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0250000000000002e-06 2024-09-21T11:04:25.169 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.170 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.170 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.170 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.417794+0000", 2024-09-21T11:04:25.170 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6248000000000007e-05 2024-09-21T11:04:25.170 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.170 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.170 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:04:25.170 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.417936+0000", 2024-09-21T11:04:25.170 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000141545 2024-09-21T11:04:25.170 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.170 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.171 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 7", 2024-09-21T11:04:25.171 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.418055+0000", 2024-09-21T11:04:25.171 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011948800000000001 2024-09-21T11:04:25.171 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.171 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.171 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:04:25.171 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.421101+0000", 2024-09-21T11:04:25.171 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0030453419999999999 2024-09-21T11:04:25.171 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.171 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.171 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.171 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.595914+0000", 2024-09-21T11:04:25.172 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.17481350300000001 2024-09-21T11:04:25.172 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.172 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.172 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:04:25.172 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.595926+0000", 2024-09-21T11:04:25.172 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2296000000000001e-05 2024-09-21T11:04:25.172 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.172 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:04:25.172 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.172 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.172 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.172 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.46190.0:84904 57.4 57:2ac37d6e:::benchmark_data_smithi070_360420_object17555:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1558)", 2024-09-21T11:04:25.172 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:00:40.419366+0000", 2024-09-21T11:04:25.172 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 224.71073108600001, 2024-09-21T11:04:25.173 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.17083596300000001, 2024-09-21T11:04:25.173 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:04:25.173 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:04:25.173 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:04:25.173 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:04:25.173 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.46190", 2024-09-21T11:04:25.173 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:04:25.173 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 84904 2024-09-21T11:04:25.173 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.173 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:04:25.173 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.173 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:04:25.173 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.419366+0000", 2024-09-21T11:04:25.174 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.174 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.174 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.174 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:04:25.174 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.419366+0000", 2024-09-21T11:04:25.174 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.174 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.174 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.174 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:04:25.174 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.419372+0000", 2024-09-21T11:04:25.174 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.959e-06 2024-09-21T11:04:25.174 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.174 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.175 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:04:25.175 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.419378+0000", 2024-09-21T11:04:25.175 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7629999999999997e-06 2024-09-21T11:04:25.175 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.175 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.175 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:04:25.175 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.419391+0000", 2024-09-21T11:04:25.175 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2532e-05 2024-09-21T11:04:25.175 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.175 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.175 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:04:25.175 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.419400+0000", 2024-09-21T11:04:25.175 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1840000000000005e-06 2024-09-21T11:04:25.175 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.176 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.176 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.176 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.419461+0000", 2024-09-21T11:04:25.176 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1273999999999996e-05 2024-09-21T11:04:25.176 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.176 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.176 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:04:25.176 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.419575+0000", 2024-09-21T11:04:25.176 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011343100000000001 2024-09-21T11:04:25.176 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.176 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.176 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,7", 2024-09-21T11:04:25.176 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.419674+0000", 2024-09-21T11:04:25.177 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9109000000000003e-05 2024-09-21T11:04:25.177 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.177 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.177 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:04:25.177 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.423179+0000", 2024-09-21T11:04:25.177 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035050810000000002 2024-09-21T11:04:25.177 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.177 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.177 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:25.177 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.424047+0000", 2024-09-21T11:04:25.177 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00086820099999999998 2024-09-21T11:04:25.177 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.177 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.177 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.178 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.590188+0000", 2024-09-21T11:04:25.178 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.16614068500000001 2024-09-21T11:04:25.178 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.178 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.178 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:04:25.178 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.590202+0000", 2024-09-21T11:04:25.178 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4744e-05 2024-09-21T11:04:25.178 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.178 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:04:25.178 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.178 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.178 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.178 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.46190.0:84908 57.4 57:2ff1ef3b:::benchmark_data_smithi070_360420_object17559:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1558)", 2024-09-21T11:04:25.178 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:00:40.424833+0000", 2024-09-21T11:04:25.179 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 224.70526404899999, 2024-09-21T11:04:25.179 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.16546303400000001, 2024-09-21T11:04:25.179 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:04:25.179 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:04:25.179 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:04:25.179 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:04:25.179 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.46190", 2024-09-21T11:04:25.179 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:04:25.179 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 84908 2024-09-21T11:04:25.179 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.179 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:04:25.179 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.179 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:04:25.180 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.424833+0000", 2024-09-21T11:04:25.180 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.180 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.180 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.180 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:04:25.180 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.424833+0000", 2024-09-21T11:04:25.180 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.180 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.180 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.180 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:04:25.180 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.424835+0000", 2024-09-21T11:04:25.180 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.094e-06 2024-09-21T11:04:25.181 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.181 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.181 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:04:25.181 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.424839+0000", 2024-09-21T11:04:25.181 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0500000000000002e-06 2024-09-21T11:04:25.181 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.181 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.181 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:04:25.181 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.424848+0000", 2024-09-21T11:04:25.181 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.8959999999999998e-06 2024-09-21T11:04:25.181 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.181 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.181 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:04:25.181 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.424853+0000", 2024-09-21T11:04:25.181 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0159999999999999e-06 2024-09-21T11:04:25.182 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.182 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.182 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.182 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.424900+0000", 2024-09-21T11:04:25.182 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6684999999999998e-05 2024-09-21T11:04:25.182 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.182 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.182 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:04:25.182 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.425031+0000", 2024-09-21T11:04:25.182 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013092400000000001 2024-09-21T11:04:25.182 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.182 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.182 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,7", 2024-09-21T11:04:25.182 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.425152+0000", 2024-09-21T11:04:25.182 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000121193 2024-09-21T11:04:25.183 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.183 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.183 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:04:25.183 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.428093+0000", 2024-09-21T11:04:25.183 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0029411789999999999 2024-09-21T11:04:25.183 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.183 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.183 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:25.183 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.428411+0000", 2024-09-21T11:04:25.183 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00031706699999999998 2024-09-21T11:04:25.183 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.183 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.183 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.183 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.590288+0000", 2024-09-21T11:04:25.184 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.161877623 2024-09-21T11:04:25.184 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.184 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.184 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:04:25.184 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.590296+0000", 2024-09-21T11:04:25.184 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3070000000000003e-06 2024-09-21T11:04:25.184 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.184 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:04:25.184 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.184 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.184 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.184 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.46190.0:84902 57.13 57:cba41bdc:::benchmark_data_smithi070_360420_object17553:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1559)", 2024-09-21T11:04:25.184 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:00:40.449158+0000", 2024-09-21T11:04:25.184 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 224.68093963699999, 2024-09-21T11:04:25.185 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.004033234, 2024-09-21T11:04:25.185 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:04:25.185 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:04:25.185 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:04:25.185 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:04:25.185 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.46190", 2024-09-21T11:04:25.185 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:04:25.185 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 84902 2024-09-21T11:04:25.185 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.185 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:04:25.185 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.185 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:04:25.185 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449158+0000", 2024-09-21T11:04:25.185 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.185 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.186 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.186 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:04:25.186 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449158+0000", 2024-09-21T11:04:25.186 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.186 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.186 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.186 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:04:25.186 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449161+0000", 2024-09-21T11:04:25.186 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.897e-06 2024-09-21T11:04:25.186 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.186 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.186 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:04:25.186 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449172+0000", 2024-09-21T11:04:25.186 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1392e-05 2024-09-21T11:04:25.187 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.187 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.187 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:04:25.187 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449203+0000", 2024-09-21T11:04:25.187 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0540000000000002e-05 2024-09-21T11:04:25.187 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.187 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.187 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:04:25.187 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449216+0000", 2024-09-21T11:04:25.187 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3678e-05 2024-09-21T11:04:25.187 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.187 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.187 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.188 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.596137+0000", 2024-09-21T11:04:25.188 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.14692064099999999 2024-09-21T11:04:25.188 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.188 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.188 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:04:25.188 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.596145+0000", 2024-09-21T11:04:25.188 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.909e-06 2024-09-21T11:04:25.188 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.188 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.188 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.188 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.453118+0000", 2024-09-21T11:04:25.188 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.85697302600000003 2024-09-21T11:04:25.188 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.189 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.189 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:04:25.189 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.453191+0000", 2024-09-21T11:04:25.189 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3151000000000005e-05 2024-09-21T11:04:25.189 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.189 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:04:25.189 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.189 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.189 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.189 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.46190.0:84904 57.14 57:2ac37d6e:::benchmark_data_smithi070_360420_object17555:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1559)", 2024-09-21T11:04:25.189 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:00:40.449248+0000", 2024-09-21T11:04:25.189 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 224.680849309, 2024-09-21T11:04:25.189 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.028813749, 2024-09-21T11:04:25.190 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:04:25.190 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:04:25.190 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:04:25.190 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:04:25.190 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.46190", 2024-09-21T11:04:25.190 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:04:25.190 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 84904 2024-09-21T11:04:25.190 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.190 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:04:25.190 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.190 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:04:25.190 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449248+0000", 2024-09-21T11:04:25.190 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.191 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.191 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.191 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:04:25.191 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449248+0000", 2024-09-21T11:04:25.191 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.191 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.191 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.191 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:04:25.191 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449249+0000", 2024-09-21T11:04:25.191 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1799999999999995e-07 2024-09-21T11:04:25.191 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.191 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.191 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:04:25.192 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449251+0000", 2024-09-21T11:04:25.192 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1789999999999998e-06 2024-09-21T11:04:25.192 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.192 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.192 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:04:25.192 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449257+0000", 2024-09-21T11:04:25.192 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2179999999999996e-06 2024-09-21T11:04:25.192 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.192 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.192 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:04:25.192 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449262+0000", 2024-09-21T11:04:25.192 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2310000000000004e-06 2024-09-21T11:04:25.192 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.192 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.193 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.193 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.590563+0000", 2024-09-21T11:04:25.193 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.14130077999999999 2024-09-21T11:04:25.193 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.193 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.193 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:04:25.193 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.590571+0000", 2024-09-21T11:04:25.193 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9200000000000004e-06 2024-09-21T11:04:25.193 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.193 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.193 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.193 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.478003+0000", 2024-09-21T11:04:25.193 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.88743153100000005 2024-09-21T11:04:25.193 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.194 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.194 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:04:25.196 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.478062+0000", 2024-09-21T11:04:25.196 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9372e-05 2024-09-21T11:04:25.196 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.196 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:04:25.196 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.196 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.196 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.196 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.46190.0:84908 57.14 57:2ff1ef3b:::benchmark_data_smithi070_360420_object17559:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1559)", 2024-09-21T11:04:25.196 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:00:40.449269+0000", 2024-09-21T11:04:25.196 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 224.68082795999999, 2024-09-21T11:04:25.196 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.035143608, 2024-09-21T11:04:25.196 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:04:25.196 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:04:25.197 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:04:25.197 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:04:25.197 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.46190", 2024-09-21T11:04:25.197 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:04:25.197 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 84908 2024-09-21T11:04:25.197 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.197 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:04:25.197 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.197 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:04:25.197 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449269+0000", 2024-09-21T11:04:25.197 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.197 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.197 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.197 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:04:25.198 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449269+0000", 2024-09-21T11:04:25.198 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.198 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.198 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.198 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:04:25.198 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449270+0000", 2024-09-21T11:04:25.198 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4200000000000002e-07 2024-09-21T11:04:25.198 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.198 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.198 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:04:25.198 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449271+0000", 2024-09-21T11:04:25.198 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5650000000000001e-06 2024-09-21T11:04:25.198 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.199 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.199 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:04:25.199 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449276+0000", 2024-09-21T11:04:25.199 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.87e-06 2024-09-21T11:04:25.199 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.199 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.199 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:04:25.199 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449280+0000", 2024-09-21T11:04:25.199 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0679999999999998e-06 2024-09-21T11:04:25.199 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.199 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.199 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.199 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.590591+0000", 2024-09-21T11:04:25.199 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.14131036599999999 2024-09-21T11:04:25.200 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.200 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.200 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:04:25.200 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.590596+0000", 2024-09-21T11:04:25.200 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4190000000000001e-06 2024-09-21T11:04:25.200 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.200 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.200 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.200 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.478340+0000", 2024-09-21T11:04:25.200 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.88774418499999996 2024-09-21T11:04:25.200 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.200 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.200 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for missing object", 2024-09-21T11:04:25.201 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.478458+0000", 2024-09-21T11:04:25.201 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000117686 2024-09-21T11:04:25.201 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.201 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.201 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.201 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.484331+0000", 2024-09-21T11:04:25.201 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0058731469999999996 2024-09-21T11:04:25.201 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.201 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.201 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:04:25.201 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.484413+0000", 2024-09-21T11:04:25.201 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1959999999999995e-05 2024-09-21T11:04:25.201 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.201 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:04:25.202 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.202 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.202 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.202 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.46190.0:84914 57.12 57:4f5dccec:::benchmark_data_smithi070_360420_object17565:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1559)", 2024-09-21T11:04:25.202 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:00:40.592927+0000", 2024-09-21T11:04:25.202 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 224.53717037800001, 2024-09-21T11:04:25.202 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.88331985400000002, 2024-09-21T11:04:25.202 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:04:25.202 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:04:25.202 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:04:25.202 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:04:25.202 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.46190", 2024-09-21T11:04:25.202 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:04:25.203 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 84914 2024-09-21T11:04:25.203 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.203 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:04:25.203 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.203 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:04:25.203 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.592927+0000", 2024-09-21T11:04:25.203 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.203 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.203 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.203 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:04:25.203 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.592927+0000", 2024-09-21T11:04:25.203 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.203 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.204 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.204 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:04:25.204 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.592929+0000", 2024-09-21T11:04:25.204 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2369999999999999e-06 2024-09-21T11:04:25.204 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.204 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.204 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:04:25.204 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.592942+0000", 2024-09-21T11:04:25.204 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2609000000000001e-05 2024-09-21T11:04:25.204 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.204 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.204 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:04:25.204 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.592969+0000", 2024-09-21T11:04:25.204 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7161999999999998e-05 2024-09-21T11:04:25.205 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.205 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.205 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:04:25.205 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.592984+0000", 2024-09-21T11:04:25.205 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4764e-05 2024-09-21T11:04:25.205 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.205 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.205 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.205 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.593075+0000", 2024-09-21T11:04:25.205 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0901000000000003e-05 2024-09-21T11:04:25.205 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.205 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.205 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:04:25.206 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.593086+0000", 2024-09-21T11:04:25.206 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0927999999999999e-05 2024-09-21T11:04:25.206 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.206 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.206 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.206 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.473920+0000", 2024-09-21T11:04:25.206 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.880834483 2024-09-21T11:04:25.206 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.206 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.206 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:04:25.206 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.474063+0000", 2024-09-21T11:04:25.206 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000142755 2024-09-21T11:04:25.206 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.207 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.207 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,4", 2024-09-21T11:04:25.207 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.474172+0000", 2024-09-21T11:04:25.207 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000108812 2024-09-21T11:04:25.207 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.207 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.207 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:04:25.207 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.476130+0000", 2024-09-21T11:04:25.207 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001958745 2024-09-21T11:04:25.207 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.207 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.207 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:25.207 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.476158+0000", 2024-09-21T11:04:25.207 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7339e-05 2024-09-21T11:04:25.208 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.208 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.208 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:25.208 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.476186+0000", 2024-09-21T11:04:25.208 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7902000000000002e-05 2024-09-21T11:04:25.208 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.208 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.208 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:04:25.208 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.476220+0000", 2024-09-21T11:04:25.208 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4038000000000002e-05 2024-09-21T11:04:25.208 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.208 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.208 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:04:25.208 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.476247+0000", 2024-09-21T11:04:25.209 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7178999999999998e-05 2024-09-21T11:04:25.209 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.209 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:04:25.209 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.209 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.209 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.209 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:54298 58.4 58:2822487b:::benchmark_data_smithi070_396992_object54297:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1585)", 2024-09-21T11:04:25.209 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:02:17.814039+0000", 2024-09-21T11:04:25.209 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 127.31605800600001, 2024-09-21T11:04:25.209 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.099180247999999999, 2024-09-21T11:04:25.209 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:04:25.209 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:04:25.209 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:04:25.210 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:04:25.210 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:04:25.210 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:04:25.210 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 54298 2024-09-21T11:04:25.210 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.210 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:04:25.210 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.210 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:04:25.210 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.814039+0000", 2024-09-21T11:04:25.210 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.210 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.210 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.211 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:04:25.211 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.814039+0000", 2024-09-21T11:04:25.211 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.211 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.211 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.211 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:04:25.211 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.814041+0000", 2024-09-21T11:04:25.211 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6160000000000001e-06 2024-09-21T11:04:25.211 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.211 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.211 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:04:25.211 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.814063+0000", 2024-09-21T11:04:25.211 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2265000000000001e-05 2024-09-21T11:04:25.212 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.212 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.212 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:04:25.212 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.814076+0000", 2024-09-21T11:04:25.212 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2859e-05 2024-09-21T11:04:25.212 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.212 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.212 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:04:25.212 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.814084+0000", 2024-09-21T11:04:25.212 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5039999999999996e-06 2024-09-21T11:04:25.212 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.212 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.212 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.212 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.814148+0000", 2024-09-21T11:04:25.213 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4744000000000002e-05 2024-09-21T11:04:25.213 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.213 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.213 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:04:25.213 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.814292+0000", 2024-09-21T11:04:25.213 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014343799999999999 2024-09-21T11:04:25.213 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.213 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.213 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,7", 2024-09-21T11:04:25.213 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.814443+0000", 2024-09-21T11:04:25.213 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000151201 2024-09-21T11:04:25.213 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.213 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.213 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:25.214 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.913015+0000", 2024-09-21T11:04:25.214 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.098571703999999996 2024-09-21T11:04:25.214 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.214 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.214 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:25.214 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.913076+0000", 2024-09-21T11:04:25.214 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1131999999999997e-05 2024-09-21T11:04:25.214 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.214 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.214 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:04:25.214 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.913091+0000", 2024-09-21T11:04:25.214 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5316999999999999e-05 2024-09-21T11:04:25.214 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.215 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.215 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:04:25.215 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.913172+0000", 2024-09-21T11:04:25.215 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0928999999999997e-05 2024-09-21T11:04:25.215 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.215 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.215 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:04:25.215 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.913220+0000", 2024-09-21T11:04:25.215 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7539e-05 2024-09-21T11:04:25.215 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.215 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:04:25.215 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.215 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.215 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.216 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:54301 58.4 58:2ddcee4c:::benchmark_data_smithi070_396992_object54300:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1585)", 2024-09-21T11:04:25.216 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:02:17.814498+0000", 2024-09-21T11:04:25.216 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 127.315599241, 2024-09-21T11:04:25.216 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.098924687999999997, 2024-09-21T11:04:25.216 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:04:25.216 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:04:25.216 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:04:25.216 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:04:25.216 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:04:25.216 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:04:25.216 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 54301 2024-09-21T11:04:25.216 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.216 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:04:25.216 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.217 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:04:25.217 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.814498+0000", 2024-09-21T11:04:25.217 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.217 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.217 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.217 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:04:25.217 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.814498+0000", 2024-09-21T11:04:25.217 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.217 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.217 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.217 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:04:25.217 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.814499+0000", 2024-09-21T11:04:25.217 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.079e-06 2024-09-21T11:04:25.217 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.218 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.218 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:04:25.218 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.814521+0000", 2024-09-21T11:04:25.218 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1563e-05 2024-09-21T11:04:25.218 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.218 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.218 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:04:25.218 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.814533+0000", 2024-09-21T11:04:25.218 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2108e-05 2024-09-21T11:04:25.218 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.218 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.218 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:04:25.218 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.814541+0000", 2024-09-21T11:04:25.219 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0499999999999992e-06 2024-09-21T11:04:25.219 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.219 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.219 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.219 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.816289+0000", 2024-09-21T11:04:25.219 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0017481280000000001 2024-09-21T11:04:25.219 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.219 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.219 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:04:25.219 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.816530+0000", 2024-09-21T11:04:25.219 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00024073000000000001 2024-09-21T11:04:25.219 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.219 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.219 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,7", 2024-09-21T11:04:25.220 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.909390+0000", 2024-09-21T11:04:25.220 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.092860247000000007 2024-09-21T11:04:25.220 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.220 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.220 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:25.220 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.913309+0000", 2024-09-21T11:04:25.220 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0039192660000000002 2024-09-21T11:04:25.220 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.220 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.220 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:25.220 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.913357+0000", 2024-09-21T11:04:25.220 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7206000000000002e-05 2024-09-21T11:04:25.220 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.220 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.221 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:04:25.221 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.913375+0000", 2024-09-21T11:04:25.221 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8782000000000001e-05 2024-09-21T11:04:25.221 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.221 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.221 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:04:25.221 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.913403+0000", 2024-09-21T11:04:25.221 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7659999999999999e-05 2024-09-21T11:04:25.221 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.221 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.221 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:04:25.221 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:17.913423+0000", 2024-09-21T11:04:25.221 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9868999999999999e-05 2024-09-21T11:04:25.221 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.222 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:04:25.222 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.222 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.222 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.222 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:92626 58.9 58:914ab8cf:::benchmark_data_smithi070_396992_object92625:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1592)", 2024-09-21T11:04:25.222 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:02:43.375858+0000", 2024-09-21T11:04:25.222 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 101.7542394, 2024-09-21T11:04:25.222 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13687560300000001, 2024-09-21T11:04:25.222 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:04:25.222 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:04:25.222 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:04:25.222 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:04:25.222 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:04:25.223 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:04:25.223 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 92626 2024-09-21T11:04:25.223 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.223 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:04:25.223 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.223 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:04:25.223 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.375858+0000", 2024-09-21T11:04:25.223 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.223 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.223 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.223 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:04:25.223 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.375858+0000", 2024-09-21T11:04:25.223 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.223 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.224 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.224 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:04:25.224 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.375860+0000", 2024-09-21T11:04:25.224 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.556e-06 2024-09-21T11:04:25.224 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.224 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.224 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:04:25.224 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.375892+0000", 2024-09-21T11:04:25.224 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2665e-05 2024-09-21T11:04:25.224 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.224 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.224 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:04:25.224 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.375907+0000", 2024-09-21T11:04:25.225 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4253000000000001e-05 2024-09-21T11:04:25.225 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.225 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.225 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:04:25.225 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.375914+0000", 2024-09-21T11:04:25.225 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0330000000000003e-06 2024-09-21T11:04:25.225 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.225 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.225 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.225 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.384550+0000", 2024-09-21T11:04:25.225 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0086361489999999992 2024-09-21T11:04:25.225 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.225 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.226 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:04:25.226 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.384731+0000", 2024-09-21T11:04:25.226 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018118100000000001 2024-09-21T11:04:25.226 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.226 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.226 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6", 2024-09-21T11:04:25.226 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.384901+0000", 2024-09-21T11:04:25.226 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000170235 2024-09-21T11:04:25.226 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.226 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.226 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:25.226 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.400760+0000", 2024-09-21T11:04:25.226 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015859135 2024-09-21T11:04:25.227 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.227 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.227 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:04:25.227 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.400777+0000", 2024-09-21T11:04:25.227 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6884e-05 2024-09-21T11:04:25.227 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.227 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.227 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:25.227 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.509347+0000", 2024-09-21T11:04:25.227 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10856990399999999 2024-09-21T11:04:25.227 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.227 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.227 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:04:25.227 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.509411+0000", 2024-09-21T11:04:25.228 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4071000000000004e-05 2024-09-21T11:04:25.228 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.228 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.228 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:04:25.228 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.512734+0000", 2024-09-21T11:04:25.228 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.003322537 2024-09-21T11:04:25.228 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.228 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:04:25.228 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.228 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.228 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.228 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:92634 58.9 58:9bea1ee2:::benchmark_data_smithi070_396992_object92633:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1592)", 2024-09-21T11:04:25.228 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:02:43.381559+0000", 2024-09-21T11:04:25.229 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 101.74853819099999, 2024-09-21T11:04:25.229 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13366128399999999, 2024-09-21T11:04:25.229 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:04:25.229 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:04:25.229 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:04:25.229 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:04:25.229 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:04:25.229 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:04:25.229 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 92634 2024-09-21T11:04:25.229 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.229 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:04:25.229 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.229 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:04:25.230 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.381559+0000", 2024-09-21T11:04:25.230 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.230 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.230 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.230 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:04:25.230 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.381559+0000", 2024-09-21T11:04:25.230 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.230 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.230 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.230 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:04:25.230 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.381561+0000", 2024-09-21T11:04:25.230 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.984e-06 2024-09-21T11:04:25.230 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.231 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.231 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:04:25.231 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.381590+0000", 2024-09-21T11:04:25.231 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9094999999999998e-05 2024-09-21T11:04:25.231 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.231 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.231 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:04:25.231 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.381605+0000", 2024-09-21T11:04:25.231 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5038000000000001e-05 2024-09-21T11:04:25.231 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.231 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.231 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:04:25.232 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.381613+0000", 2024-09-21T11:04:25.232 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6729999999999992e-06 2024-09-21T11:04:25.232 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.232 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.232 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.232 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.387574+0000", 2024-09-21T11:04:25.232 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0059604929999999999 2024-09-21T11:04:25.232 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.232 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.232 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:04:25.232 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.387746+0000", 2024-09-21T11:04:25.232 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017235800000000001 2024-09-21T11:04:25.232 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.232 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.233 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6", 2024-09-21T11:04:25.233 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.387921+0000", 2024-09-21T11:04:25.233 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000174952 2024-09-21T11:04:25.233 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.233 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.233 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:04:25.233 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.400783+0000", 2024-09-21T11:04:25.233 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012862011 2024-09-21T11:04:25.233 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.233 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.233 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:25.233 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.400802+0000", 2024-09-21T11:04:25.233 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9127000000000002e-05 2024-09-21T11:04:25.234 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.234 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.234 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:25.234 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.512800+0000", 2024-09-21T11:04:25.234 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11199772600000001 2024-09-21T11:04:25.234 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.234 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.234 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:04:25.234 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.512857+0000", 2024-09-21T11:04:25.234 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7757999999999998e-05 2024-09-21T11:04:25.234 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.234 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.234 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:04:25.235 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:02:43.515221+0000", 2024-09-21T11:04:25.235 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0023630690000000002 2024-09-21T11:04:25.235 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.235 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:04:25.235 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.235 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.235 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.235 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "MOSDPGPushReply(58.9 1593/1590 [PushReplyOp(58:938d8a44:::benchmark_data_smithi070_396992_object11517:head)])", 2024-09-21T11:04:25.235 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:03:11.445622+0000", 2024-09-21T11:04:25.235 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 73.684475215000006, 2024-09-21T11:04:25.235 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.099615963000000002, 2024-09-21T11:04:25.235 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:04:25.235 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:04:25.236 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:04:25.236 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:04:25.236 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.236 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:04:25.236 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.445622+0000", 2024-09-21T11:04:25.236 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.236 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.236 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.236 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:04:25.236 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.445622+0000", 2024-09-21T11:04:25.236 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.236 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.236 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.236 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:04:25.236 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.445623+0000", 2024-09-21T11:04:25.237 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6300000000000005e-07 2024-09-21T11:04:25.237 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.237 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.237 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:04:25.237 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.445629+0000", 2024-09-21T11:04:25.237 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.139e-06 2024-09-21T11:04:25.237 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.237 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.237 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:04:25.237 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.445640+0000", 2024-09-21T11:04:25.237 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.094e-05 2024-09-21T11:04:25.237 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.237 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.238 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:04:25.238 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.445647+0000", 2024-09-21T11:04:25.238 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1919999999999999e-06 2024-09-21T11:04:25.238 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.238 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.238 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.238 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.465059+0000", 2024-09-21T11:04:25.238 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.019412276999999999 2024-09-21T11:04:25.238 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.238 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.238 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:04:25.238 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.545238+0000", 2024-09-21T11:04:25.238 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.080178752000000006 2024-09-21T11:04:25.239 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.239 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:04:25.239 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.239 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.239 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.239 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:126474 58.9 58:9ade3374:::benchmark_data_smithi070_396992_object126473:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1593)", 2024-09-21T11:04:25.239 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:03:11.446031+0000", 2024-09-21T11:04:25.239 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 73.684066122000004, 2024-09-21T11:04:25.239 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.099197902000000004, 2024-09-21T11:04:25.239 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:04:25.239 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:04:25.239 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:04:25.239 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:04:25.240 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:04:25.240 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:04:25.240 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 126474 2024-09-21T11:04:25.240 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.240 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:04:25.240 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.240 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:04:25.240 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.446031+0000", 2024-09-21T11:04:25.240 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.240 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.240 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.240 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:04:25.240 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.446031+0000", 2024-09-21T11:04:25.240 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.241 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.241 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.241 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:04:25.241 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.446033+0000", 2024-09-21T11:04:25.241 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7069999999999999e-06 2024-09-21T11:04:25.241 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.241 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.241 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:04:25.241 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.446055+0000", 2024-09-21T11:04:25.241 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2058000000000001e-05 2024-09-21T11:04:25.241 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.241 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.241 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:04:25.242 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.446069+0000", 2024-09-21T11:04:25.242 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3559e-05 2024-09-21T11:04:25.242 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.242 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.242 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:04:25.242 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.446076+0000", 2024-09-21T11:04:25.242 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4100000000000002e-06 2024-09-21T11:04:25.242 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.242 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.242 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.242 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.455248+0000", 2024-09-21T11:04:25.242 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0091717389999999999 2024-09-21T11:04:25.243 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.243 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.243 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:04:25.243 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.455431+0000", 2024-09-21T11:04:25.243 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000183203 2024-09-21T11:04:25.243 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.243 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.243 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6", 2024-09-21T11:04:25.243 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.455595+0000", 2024-09-21T11:04:25.243 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000164098 2024-09-21T11:04:25.243 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.243 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.243 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:25.243 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.471760+0000", 2024-09-21T11:04:25.244 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.016164609999999999 2024-09-21T11:04:25.244 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.244 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.244 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:25.244 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.503511+0000", 2024-09-21T11:04:25.244 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.03175099 2024-09-21T11:04:25.244 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.244 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.244 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:04:25.244 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.541600+0000", 2024-09-21T11:04:25.244 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.038089362000000002 2024-09-21T11:04:25.245 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.245 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.245 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:04:25.245 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.541691+0000", 2024-09-21T11:04:25.245 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1188999999999997e-05 2024-09-21T11:04:25.245 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.245 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.245 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:04:25.245 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.545229+0000", 2024-09-21T11:04:25.245 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035379769999999999 2024-09-21T11:04:25.245 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.245 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:04:25.245 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.245 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.246 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.246 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:126482 58.9 58:9b61eaa6:::benchmark_data_smithi070_396992_object126481:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1593)", 2024-09-21T11:04:25.246 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:03:11.451973+0000", 2024-09-21T11:04:25.246 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 73.678124581999995, 2024-09-21T11:04:25.246 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10290587800000001, 2024-09-21T11:04:25.246 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:04:25.246 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:04:25.246 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:04:25.246 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:04:25.246 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:04:25.246 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:04:25.246 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 126482 2024-09-21T11:04:25.246 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.247 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:04:25.247 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.247 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:04:25.247 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.451973+0000", 2024-09-21T11:04:25.247 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.247 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.247 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.247 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:04:25.247 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.451973+0000", 2024-09-21T11:04:25.247 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.247 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.247 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.247 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:04:25.247 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.451975+0000", 2024-09-21T11:04:25.248 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6500000000000001e-06 2024-09-21T11:04:25.248 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.248 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.248 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:04:25.248 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.451997+0000", 2024-09-21T11:04:25.248 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2033999999999999e-05 2024-09-21T11:04:25.248 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.248 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.248 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:04:25.248 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.452009+0000", 2024-09-21T11:04:25.248 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2165e-05 2024-09-21T11:04:25.248 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.248 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.248 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:04:25.249 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.452016+0000", 2024-09-21T11:04:25.249 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.605e-06 2024-09-21T11:04:25.249 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.249 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.249 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.249 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.459952+0000", 2024-09-21T11:04:25.249 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0079359029999999994 2024-09-21T11:04:25.249 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.249 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.249 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:04:25.249 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.460112+0000", 2024-09-21T11:04:25.249 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000159679 2024-09-21T11:04:25.249 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.249 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.250 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6", 2024-09-21T11:04:25.250 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.460279+0000", 2024-09-21T11:04:25.250 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000166937 2024-09-21T11:04:25.250 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.250 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.250 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:25.250 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.471803+0000", 2024-09-21T11:04:25.250 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.011523752999999999 2024-09-21T11:04:25.250 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.250 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.250 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:25.250 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.503553+0000", 2024-09-21T11:04:25.250 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.031750680000000003 2024-09-21T11:04:25.250 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.251 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.251 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:04:25.251 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.550953+0000", 2024-09-21T11:04:25.251 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.047400088999999999 2024-09-21T11:04:25.251 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.251 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.251 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:04:25.251 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.551789+0000", 2024-09-21T11:04:25.251 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00083604899999999995 2024-09-21T11:04:25.251 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.251 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.251 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:04:25.251 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:11.554879+0000", 2024-09-21T11:04:25.251 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.003089334 2024-09-21T11:04:25.252 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.252 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:04:25.252 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.252 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.252 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.252 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:135551 58.9 58:97d54893:::benchmark_data_smithi070_396992_object135550:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1594)", 2024-09-21T11:04:25.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:25.253 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:25.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:25.253 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:03:19.322880+0000", 2024-09-21T11:04:25.253 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 65.807217004999998, 2024-09-21T11:04:25.254 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.111543738, 2024-09-21T11:04:25.254 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:04:25.254 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:04:25.254 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:04:25.254 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:04:25.254 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:04:25.254 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:04:25.254 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 135551 2024-09-21T11:04:25.254 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.254 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:04:25.254 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.254 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:04:25.254 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.322880+0000", 2024-09-21T11:04:25.255 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.255 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.255 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.255 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:04:25.255 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.322880+0000", 2024-09-21T11:04:25.255 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.255 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.255 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.255 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:04:25.255 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.322882+0000", 2024-09-21T11:04:25.255 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.175e-06 2024-09-21T11:04:25.255 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.255 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.255 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:04:25.256 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.322901+0000", 2024-09-21T11:04:25.256 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9587999999999999e-05 2024-09-21T11:04:25.256 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.256 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.256 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:04:25.256 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.322910+0000", 2024-09-21T11:04:25.256 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3510000000000002e-06 2024-09-21T11:04:25.256 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.256 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.256 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:04:25.256 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.322914+0000", 2024-09-21T11:04:25.256 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7779999999999998e-06 2024-09-21T11:04:25.256 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.256 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.257 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.257 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.324312+0000", 2024-09-21T11:04:25.257 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0013972920000000001 2024-09-21T11:04:25.257 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.257 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.257 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:04:25.257 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.324451+0000", 2024-09-21T11:04:25.257 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013925999999999999 2024-09-21T11:04:25.257 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.257 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.257 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6", 2024-09-21T11:04:25.257 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.324552+0000", 2024-09-21T11:04:25.257 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000101587 2024-09-21T11:04:25.257 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.258 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.258 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:04:25.258 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.330085+0000", 2024-09-21T11:04:25.258 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0055324149999999997 2024-09-21T11:04:25.258 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.258 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.258 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:25.258 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.343588+0000", 2024-09-21T11:04:25.258 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.013502655000000001 2024-09-21T11:04:25.258 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.258 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.258 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:25.258 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.431430+0000", 2024-09-21T11:04:25.258 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.087842877999999999 2024-09-21T11:04:25.259 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.259 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.259 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:04:25.259 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.431498+0000", 2024-09-21T11:04:25.259 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7489000000000004e-05 2024-09-21T11:04:25.259 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.259 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.259 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:04:25.259 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.434424+0000", 2024-09-21T11:04:25.259 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0029262699999999999 2024-09-21T11:04:25.259 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.259 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:04:25.259 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.259 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.260 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.260 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:135552 58.9 58:9459c8dd:::benchmark_data_smithi070_396992_object135551:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1594)", 2024-09-21T11:04:25.260 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:03:19.324961+0000", 2024-09-21T11:04:25.260 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 65.805136665999996, 2024-09-21T11:04:25.260 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.112460513, 2024-09-21T11:04:25.260 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:04:25.260 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:04:25.260 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:04:25.260 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:04:25.260 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:04:25.260 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:04:25.260 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 135552 2024-09-21T11:04:25.260 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.260 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:04:25.261 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.261 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:04:25.261 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.324961+0000", 2024-09-21T11:04:25.261 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.261 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.261 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.261 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:04:25.261 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.324961+0000", 2024-09-21T11:04:25.261 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:04:25.261 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.261 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.261 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:04:25.261 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.324962+0000", 2024-09-21T11:04:25.262 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.64e-07 2024-09-21T11:04:25.262 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.262 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.262 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:04:25.262 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.324982+0000", 2024-09-21T11:04:25.262 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0363000000000001e-05 2024-09-21T11:04:25.262 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.262 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.262 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:04:25.262 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.324991+0000", 2024-09-21T11:04:25.262 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0880000000000001e-06 2024-09-21T11:04:25.262 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.262 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.263 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:04:25.263 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.324997+0000", 2024-09-21T11:04:25.263 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.161e-06 2024-09-21T11:04:25.263 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.263 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.263 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:04:25.263 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.327760+0000", 2024-09-21T11:04:25.263 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0027626550000000001 2024-09-21T11:04:25.263 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.263 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.263 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:04:25.263 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.327851+0000", 2024-09-21T11:04:25.263 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0858000000000003e-05 2024-09-21T11:04:25.263 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.264 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.264 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6", 2024-09-21T11:04:25.264 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.327942+0000", 2024-09-21T11:04:25.264 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0772000000000004e-05 2024-09-21T11:04:25.264 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.264 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.264 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:04:25.264 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.330252+0000", 2024-09-21T11:04:25.264 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0023099779999999999 2024-09-21T11:04:25.264 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.264 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.264 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:25.264 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.343618+0000", 2024-09-21T11:04:25.264 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.013365874999999999 2024-09-21T11:04:25.265 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.265 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.265 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:25.265 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.434478+0000", 2024-09-21T11:04:25.265 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.090860250000000004 2024-09-21T11:04:25.265 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.265 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.265 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:04:25.265 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.434541+0000", 2024-09-21T11:04:25.265 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3418999999999997e-05 2024-09-21T11:04:25.265 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:04:25.265 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:04:25.265 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:04:25.266 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:03:19.437421+0000", 2024-09-21T11:04:25.266 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0028801299999999998 2024-09-21T11:04:25.266 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.266 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:04:25.266 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.266 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:04:25.266 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:04:25.266 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T11:04:25.353 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:25.388 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:25.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:25.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:25.445 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T11:04:25.490 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:25.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:25.523 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:25.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:25.624 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:25.648 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:25.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:25.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:25.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:25 smithi160 ceph-mon[101879]: pgmap v3169: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 5.4 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 0 B/s wr, 793 op/s; 51938/264171 objects degraded (19.661%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T11:04:25.749 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:25.782 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:25.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:25.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:25.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:25 smithi070 ceph-mon[107385]: pgmap v3169: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 5.4 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 0 B/s wr, 793 op/s; 51938/264171 objects degraded (19.661%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T11:04:25.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:25 smithi070 ceph-mon[112139]: pgmap v3169: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 5.4 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 0 B/s wr, 793 op/s; 51938/264171 objects degraded (19.661%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T11:04:25.884 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:25.916 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:25.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:25.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:26.018 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:26.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:26.049 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:26.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:26.150 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:26.184 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:26.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:26.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:26.286 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:26.328 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:26.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:26.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:26.430 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:26.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:26.472 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:26.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:26.563 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T11:04:26.573 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:26.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:26.621 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:26.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:26.722 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:26.758 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:26 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T11:04:26.758 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:26 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T11:04:26.786 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:26.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:26.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:26 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T11:04:26.887 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:26.934 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:26.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:26.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:27.036 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:27.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:27.058 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:27.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:27.159 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:27.201 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:27.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:27.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:27.305 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:27.331 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:27.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:27.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:27.432 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:27.457 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:27.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:27.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:27.559 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:27.584 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:27.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:27.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:27.686 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:27.713 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:27.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:27.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:27.814 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:27.840 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:27.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:27.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:27 smithi160 ceph-mon[101879]: pgmap v3170: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 5.0 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.58k op/s; 48252/245262 objects degraded (19.674%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T11:04:27.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:27 smithi070 ceph-mon[107385]: pgmap v3170: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 5.0 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.58k op/s; 48252/245262 objects degraded (19.674%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T11:04:27.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:27 smithi070 ceph-mon[112139]: pgmap v3170: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 5.0 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.58k op/s; 48252/245262 objects degraded (19.674%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T11:04:27.942 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:27.971 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:27.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:27.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:28.073 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:28.098 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:28.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:28.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:28.200 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:28.226 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:28.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:28.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:28.328 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:28.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:28.350 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:28.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:28.451 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:28.473 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:28.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:28.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:28.575 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:28.611 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:28.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:28.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:28.713 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:28.743 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:28.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:28.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:28 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 48252/245262 objects degraded (19.674%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:28.845 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:28.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:28 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 48252/245262 objects degraded (19.674%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:28.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:28 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 48252/245262 objects degraded (19.674%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:28.884 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:28.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:28.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:28.987 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:29.013 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:29.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:29.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:29.116 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:29.142 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:29.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:29.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:29.244 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:29.273 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:29.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:29.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:29.375 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:29.396 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:29.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:29.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:29.498 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:29.526 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:29.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:29.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:29.628 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:29.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:29.678 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:29.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:29.779 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:29.819 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:29.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:29.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:29.830 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:29 smithi160 ceph-mon[101879]: pgmap v3171: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 5.0 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.23k op/s; 48252/245262 objects degraded (19.674%); 1.2 MiB/s, 18 objects/s recovering 2024-09-21T11:04:29.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:29 smithi070 ceph-mon[107385]: pgmap v3171: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 5.0 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.23k op/s; 48252/245262 objects degraded (19.674%); 1.2 MiB/s, 18 objects/s recovering 2024-09-21T11:04:29.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:29 smithi070 ceph-mon[112139]: pgmap v3171: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 5.0 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.23k op/s; 48252/245262 objects degraded (19.674%); 1.2 MiB/s, 18 objects/s recovering 2024-09-21T11:04:29.922 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:29.955 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:29.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:29.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:30.056 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:30.082 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:30.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:30.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:30.184 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:30.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:30.225 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:30.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:30.326 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:30.352 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:30.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:30.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:30.454 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:30.475 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:30.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:30.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:30.576 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:30.603 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:30.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:30.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:30.705 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:30.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:30.746 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:30.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:30.848 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:30.876 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:30.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:30.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:30.978 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:31.004 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:31.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:31.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:31.106 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:31.134 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:31.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:31.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:31.180 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T11:04:31.236 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:31.261 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:31.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:31.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:31.363 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:31.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:31.411 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:31.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:31.512 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:31.540 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:31.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:31.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:31.641 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:31.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:31.682 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:31.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:31.693 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:04:31.783 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:31.813 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:31.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:31.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:31.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:31 smithi070 ceph-mon[112139]: pgmap v3172: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 4.8 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 46769/238212 objects degraded (19.633%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T11:04:31.825 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:31 smithi070 ceph-mon[107385]: pgmap v3172: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 4.8 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 46769/238212 objects degraded (19.633%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T11:04:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:31 smithi160 ceph-mon[101879]: pgmap v3172: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 4.8 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 46769/238212 objects degraded (19.633%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T11:04:31.914 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:31.959 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:31.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:31.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:32.061 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:32.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:32.100 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:32.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:32.179 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:04:32.201 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:32.234 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:32.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:32.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:32.336 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:32.372 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:32.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:32.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:32.474 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:32.502 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:32.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:32.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:32.604 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:32.641 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:32.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:32.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:32.743 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:32.773 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:32.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:32.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:32.875 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:32.903 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:32.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:32.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:33.004 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:33.043 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:33.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:33.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:33.144 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:33.176 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:33.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:33.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:33.278 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:33.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:33.316 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:33.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:33.417 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:33.453 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:33.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:33.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:33.508 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:04:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:04:33] "GET /metrics HTTP/1.1" 200 36737 "" "Prometheus/2.51.0" 2024-09-21T11:04:33.554 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:33.576 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:33.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:33.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:33.678 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:33.699 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:33.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:33.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:33.801 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:33.824 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:33.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:33.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:33 smithi160 ceph-mon[101879]: pgmap v3173: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 4.7 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 1.31k op/s; 45145/229158 objects degraded (19.700%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T11:04:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:33 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'3':'26918','4':'54413','6':'65536','7':'45138'}"} : dispatch 2024-09-21T11:04:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:33 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 45145/229158 objects degraded (19.700%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:33.857 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:04:33.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:33 smithi070 ceph-mon[107385]: pgmap v3173: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 4.7 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 1.31k op/s; 45145/229158 objects degraded (19.700%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T11:04:33.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:33 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'3':'26918','4':'54413','6':'65536','7':'45138'}"} : dispatch 2024-09-21T11:04:33.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:33 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 45145/229158 objects degraded (19.700%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:33 smithi070 ceph-mon[112139]: pgmap v3173: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 4.7 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 1.31k op/s; 45145/229158 objects degraded (19.700%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T11:04:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:33 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'3':'26918','4':'54413','6':'65536','7':'45138'}"} : dispatch 2024-09-21T11:04:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:33 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 45145/229158 objects degraded (19.700%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:33.926 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:33.948 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:33.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:33.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:34.050 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:34.072 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:34.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:34.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:34.085 INFO:teuthology.orchestra.run.smithi070.stdout:moved 51 / 327 (15.5963%) 2024-09-21T11:04:34.085 INFO:teuthology.orchestra.run.smithi070.stdout:avg 65.4 2024-09-21T11:04:34.085 INFO:teuthology.orchestra.run.smithi070.stdout:stddev 24.4344 -> 27.3657 (expected baseline 7.23326) 2024-09-21T11:04:34.085 INFO:teuthology.orchestra.run.smithi070.stdout:min osd.3 with 17 -> 21 pgs (0.259939 -> 0.321101 * mean) 2024-09-21T11:04:34.085 INFO:teuthology.orchestra.run.smithi070.stdout:max osd.1 with 80 -> 87 pgs (1.22324 -> 1.33028 * mean) 2024-09-21T11:04:34.085 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T11:04:34.085 INFO:teuthology.orchestra.run.smithi070.stdout:oload 110 2024-09-21T11:04:34.085 INFO:teuthology.orchestra.run.smithi070.stdout:max_change 1 2024-09-21T11:04:34.085 INFO:teuthology.orchestra.run.smithi070.stdout:max_change_osds 4 2024-09-21T11:04:34.085 INFO:teuthology.orchestra.run.smithi070.stdout:average_utilization 0.0386 2024-09-21T11:04:34.085 INFO:teuthology.orchestra.run.smithi070.stdout:overload_utilization 0.0424 2024-09-21T11:04:34.085 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 weight 0.8683 -> 1.0000 2024-09-21T11:04:34.085 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 weight 1.0000 -> 0.6888 2024-09-21T11:04:34.086 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 weight 0.2979 -> 0.4107 2024-09-21T11:04:34.086 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 weight 1.0000 -> 0.8303 2024-09-21T11:04:34.173 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:34.196 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:34.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:34.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:34.297 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:34.319 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:34.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:34.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:34.421 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:34.442 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:34.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:34.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:34.544 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:34.569 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:34.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:34.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:34.674 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:34.695 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:34.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:34.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:34.797 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:34.818 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:34.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:34.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:34 smithi160 ceph-mon[101879]: from='client.46661 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T11:04:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'26918','4':'54413','6':'65536','7':'45138'}"}]': finished 2024-09-21T11:04:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:34 smithi160 ceph-mon[101879]: osdmap e1604: 8 total, 8 up, 5 in 2024-09-21T11:04:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:04:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:34 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:04:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:34 smithi160 ceph-mon[101879]: osdmap e1605: 8 total, 8 up, 5 in 2024-09-21T11:04:34.853 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:34 smithi070 ceph-mon[107385]: from='client.46661 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T11:04:34.853 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'26918','4':'54413','6':'65536','7':'45138'}"}]': finished 2024-09-21T11:04:34.853 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:34 smithi070 ceph-mon[107385]: osdmap e1604: 8 total, 8 up, 5 in 2024-09-21T11:04:34.853 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:04:34.853 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:34 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:04:34.853 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:34 smithi070 ceph-mon[107385]: osdmap e1605: 8 total, 8 up, 5 in 2024-09-21T11:04:34.853 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:34 smithi070 ceph-mon[112139]: from='client.46661 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T11:04:34.853 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'26918','4':'54413','6':'65536','7':'45138'}"}]': finished 2024-09-21T11:04:34.853 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:34 smithi070 ceph-mon[112139]: osdmap e1604: 8 total, 8 up, 5 in 2024-09-21T11:04:34.853 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:04:34.853 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:34 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:04:34.854 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:34 smithi070 ceph-mon[112139]: osdmap e1605: 8 total, 8 up, 5 in 2024-09-21T11:04:34.920 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:34.941 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:34.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:34.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:35.042 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:35.065 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:35.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:35.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:35.167 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:35.189 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:35.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:35.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:35.290 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:35.312 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:35.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:35.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:35.414 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:35.435 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:35.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:35.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:35.536 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:35.564 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:35.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:35.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:35.666 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:35.699 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:35.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:35.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:35.801 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:35.839 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:35.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:35.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:35 smithi160 ceph-mon[101879]: pgmap v3175: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 4.7 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 1.17k op/s; 45145/229158 objects degraded (19.700%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T11:04:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:35 smithi160 ceph-mon[101879]: 58.1 starting backfill to osd.4 from (0'0,0'0] MAX to 1603'13505 2024-09-21T11:04:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:35 smithi160 ceph-mon[101879]: 58.7 starting backfill to osd.1 from (0'0,0'0] MAX to 1603'13631 2024-09-21T11:04:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:35 smithi160 ceph-mon[101879]: 58.1 continuing backfill to osd.6 from (1595'9700,1603'13505] MIN to 1603'13505 2024-09-21T11:04:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:35 smithi160 ceph-mon[101879]: 58.7 continuing backfill to osd.6 from (1595'9800,1603'13631] MIN to 1603'13631 2024-09-21T11:04:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:04:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:04:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:35 smithi160 ceph-mon[101879]: osdmap e1606: 8 total, 8 up, 5 in 2024-09-21T11:04:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:35 smithi070 ceph-mon[107385]: pgmap v3175: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 4.7 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 1.17k op/s; 45145/229158 objects degraded (19.700%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T11:04:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:35 smithi070 ceph-mon[107385]: 58.1 starting backfill to osd.4 from (0'0,0'0] MAX to 1603'13505 2024-09-21T11:04:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:35 smithi070 ceph-mon[107385]: 58.7 starting backfill to osd.1 from (0'0,0'0] MAX to 1603'13631 2024-09-21T11:04:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:35 smithi070 ceph-mon[107385]: 58.1 continuing backfill to osd.6 from (1595'9700,1603'13505] MIN to 1603'13505 2024-09-21T11:04:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:35 smithi070 ceph-mon[107385]: 58.7 continuing backfill to osd.6 from (1595'9800,1603'13631] MIN to 1603'13631 2024-09-21T11:04:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:04:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:04:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:35 smithi070 ceph-mon[107385]: osdmap e1606: 8 total, 8 up, 5 in 2024-09-21T11:04:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:35 smithi070 ceph-mon[112139]: pgmap v3175: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+undersized+degraded+remapped+backfill_wait, 99 active+clean; 4.7 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 1.17k op/s; 45145/229158 objects degraded (19.700%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T11:04:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:35 smithi070 ceph-mon[112139]: 58.1 starting backfill to osd.4 from (0'0,0'0] MAX to 1603'13505 2024-09-21T11:04:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:35 smithi070 ceph-mon[112139]: 58.7 starting backfill to osd.1 from (0'0,0'0] MAX to 1603'13631 2024-09-21T11:04:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:35 smithi070 ceph-mon[112139]: 58.1 continuing backfill to osd.6 from (1595'9700,1603'13505] MIN to 1603'13505 2024-09-21T11:04:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:35 smithi070 ceph-mon[112139]: 58.7 continuing backfill to osd.6 from (1595'9800,1603'13631] MIN to 1603'13631 2024-09-21T11:04:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:04:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:04:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:35 smithi070 ceph-mon[112139]: osdmap e1606: 8 total, 8 up, 5 in 2024-09-21T11:04:35.941 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:35.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:35.971 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:35.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:36.071 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:36.109 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:36.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:36.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:36.211 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:36.234 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:36.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:36.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:36.336 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:36.368 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:36.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:36.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:36.468 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-21T11:04:36.468 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 110 1.0 2024-09-21T11:04:36.471 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:36.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:36.512 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:36.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:36.613 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:36 smithi070 ceph-mon[107385]: 58.4 starting backfill to osd.6 from (0'0,0'0] MAX to 1603'13692 2024-09-21T11:04:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:36 smithi070 ceph-mon[107385]: 58.e starting backfill to osd.4 from (0'0,0'0] MAX to 1603'13522 2024-09-21T11:04:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:36 smithi070 ceph-mon[107385]: 58.2 starting backfill to osd.3 from (0'0,0'0] MAX to 1603'13714 2024-09-21T11:04:36.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:36 smithi070 ceph-mon[107385]: 58.6 starting backfill to osd.6 from (0'0,0'0] MAX to 1603'13944 2024-09-21T11:04:36.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:36 smithi070 ceph-mon[112139]: 58.4 starting backfill to osd.6 from (0'0,0'0] MAX to 1603'13692 2024-09-21T11:04:36.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:36 smithi070 ceph-mon[112139]: 58.e starting backfill to osd.4 from (0'0,0'0] MAX to 1603'13522 2024-09-21T11:04:36.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:36 smithi070 ceph-mon[112139]: 58.2 starting backfill to osd.3 from (0'0,0'0] MAX to 1603'13714 2024-09-21T11:04:36.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:36 smithi070 ceph-mon[112139]: 58.6 starting backfill to osd.6 from (0'0,0'0] MAX to 1603'13944 2024-09-21T11:04:36.634 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:36.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:36.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:36.736 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:36.771 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:36.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:36.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:36.782 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:36 smithi160 ceph-mon[101879]: 58.4 starting backfill to osd.6 from (0'0,0'0] MAX to 1603'13692 2024-09-21T11:04:36.782 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:36 smithi160 ceph-mon[101879]: 58.e starting backfill to osd.4 from (0'0,0'0] MAX to 1603'13522 2024-09-21T11:04:36.782 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:36 smithi160 ceph-mon[101879]: 58.2 starting backfill to osd.3 from (0'0,0'0] MAX to 1603'13714 2024-09-21T11:04:36.782 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:36 smithi160 ceph-mon[101879]: 58.6 starting backfill to osd.6 from (0'0,0'0] MAX to 1603'13944 2024-09-21T11:04:36.873 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:36.899 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:36.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:36.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:37.001 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:37.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:37.036 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:37.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:37.137 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:37.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:37.176 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:37.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:37.276 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:37.309 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:37.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:37.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:37.410 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:37.449 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:37.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:37.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:37.551 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:37.578 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:37.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:37.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:37.679 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:37.716 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:37.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:37.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:37.817 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:37.840 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:37.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:37 smithi160 ceph-mon[101879]: pgmap v3178: 109 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 2 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 5 remapped+peering, 8 active+undersized+degraded+remapped+backfill_wait, 89 active+clean; 4.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 8.3 KiB/s rd, 24 KiB/s wr, 1.13k op/s; 42694/217962 objects degraded (19.588%); 18142/217962 objects misplaced (8.323%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T11:04:37.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:37 smithi160 ceph-mon[101879]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-09-21T11:04:37.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:37 smithi160 ceph-mon[101879]: osdmap e1607: 8 total, 8 up, 5 in 2024-09-21T11:04:37.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:37 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1909797063' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:04:37.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:37.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:37.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:37 smithi070 ceph-mon[112139]: pgmap v3178: 109 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 2 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 5 remapped+peering, 8 active+undersized+degraded+remapped+backfill_wait, 89 active+clean; 4.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 8.3 KiB/s rd, 24 KiB/s wr, 1.13k op/s; 42694/217962 objects degraded (19.588%); 18142/217962 objects misplaced (8.323%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T11:04:37.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:37 smithi070 ceph-mon[112139]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-09-21T11:04:37.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:37 smithi070 ceph-mon[112139]: osdmap e1607: 8 total, 8 up, 5 in 2024-09-21T11:04:37.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:37 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1909797063' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:04:37.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:37 smithi070 ceph-mon[107385]: pgmap v3178: 109 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 2 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 5 remapped+peering, 8 active+undersized+degraded+remapped+backfill_wait, 89 active+clean; 4.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 8.3 KiB/s rd, 24 KiB/s wr, 1.13k op/s; 42694/217962 objects degraded (19.588%); 18142/217962 objects misplaced (8.323%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T11:04:37.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:37 smithi070 ceph-mon[107385]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-09-21T11:04:37.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:37 smithi070 ceph-mon[107385]: osdmap e1607: 8 total, 8 up, 5 in 2024-09-21T11:04:37.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:37 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1909797063' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:04:37.942 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:37.972 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:37.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:37.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:38.074 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:38.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:38.102 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:38.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:38.146 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:04:38.146 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:04:38.147 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:04:38.147 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:04:38.147 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:04:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:04:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:04:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:04:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:04:38.147 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:04:38.148 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T11:04:38.148 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T11:04:38.148 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:04:38.148 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T11:04:38.148 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T11:04:38.148 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:04:38.203 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:38.241 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:38.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:38.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:38.342 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:38.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:38.387 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:38.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:38.488 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:38.513 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:38.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:38.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:38.615 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:38.652 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:38.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:38.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:38.754 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:38.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:38 smithi070 ceph-mon[107385]: osdmap e1608: 8 total, 8 up, 5 in 2024-09-21T11:04:38.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:38 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 42694/217962 objects degraded (19.588%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:38.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:38 smithi070 ceph-mon[112139]: osdmap e1608: 8 total, 8 up, 5 in 2024-09-21T11:04:38.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:38 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 42694/217962 objects degraded (19.588%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:38.798 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:38.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:38.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:38 smithi160 ceph-mon[101879]: osdmap e1608: 8 total, 8 up, 5 in 2024-09-21T11:04:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:38 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 42694/217962 objects degraded (19.588%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:38.900 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:38.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:38.944 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:38.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:39.045 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:39.080 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:39.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:39.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:39.182 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:39.207 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:39.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:39.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:39.229 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:04:39.309 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:39.335 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:39.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:39.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:39.436 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:39.465 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:39.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:39.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:39.567 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:39.612 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:39.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:39.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:39.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:39 smithi070 ceph-mon[112139]: pgmap v3181: 109 pgs: 7 peering, 2 active+clean+remapped, 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 2 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 4 remapped+peering, 8 active+undersized+degraded+remapped+backfill_wait, 81 active+clean; 4.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 18 KiB/s rd, 55 KiB/s wr, 946 op/s; 42694/217962 objects degraded (19.588%); 18142/217962 objects misplaced (8.323%); 5.2 MiB/s, 82 objects/s recovering 2024-09-21T11:04:39.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:39 smithi070 ceph-mon[112139]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-21T11:04:39.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:39 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:04:39.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:39 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:04:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:39 smithi070 ceph-mon[107385]: pgmap v3181: 109 pgs: 7 peering, 2 active+clean+remapped, 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 2 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 4 remapped+peering, 8 active+undersized+degraded+remapped+backfill_wait, 81 active+clean; 4.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 18 KiB/s rd, 55 KiB/s wr, 946 op/s; 42694/217962 objects degraded (19.588%); 18142/217962 objects misplaced (8.323%); 5.2 MiB/s, 82 objects/s recovering 2024-09-21T11:04:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:39 smithi070 ceph-mon[107385]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-21T11:04:39.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:39 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:04:39.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:39 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:04:39.713 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:39.750 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:39.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:39.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:39 smithi160 ceph-mon[101879]: pgmap v3181: 109 pgs: 7 peering, 2 active+clean+remapped, 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 2 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 4 remapped+peering, 8 active+undersized+degraded+remapped+backfill_wait, 81 active+clean; 4.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 18 KiB/s rd, 55 KiB/s wr, 946 op/s; 42694/217962 objects degraded (19.588%); 18142/217962 objects misplaced (8.323%); 5.2 MiB/s, 82 objects/s recovering 2024-09-21T11:04:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:39 smithi160 ceph-mon[101879]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-21T11:04:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:39 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:04:39.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:39 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:04:39.852 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:39.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:39.892 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:39.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:39.993 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:40.015 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:40.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:40.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:40.116 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:40.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:40.151 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:40.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:40.252 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:40.276 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:40.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:40.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:40.378 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:40.407 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:40.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:40.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:40.509 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:40.531 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:40.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:40.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:40.633 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:40.654 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:40.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:40.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:40.756 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:40.795 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:40.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:40.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:40.824 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T11:04:40.897 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:40.918 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:40.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:40.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:41.020 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:41.048 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:41.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:41.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:41.150 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:41.187 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:41.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:41.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:41.289 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:41.310 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:41.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:41.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:41.412 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:41.476 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:41.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:41.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:41.569 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T11:04:41.577 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:41.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:41 smithi160 ceph-mon[101879]: pgmap v3182: 109 pgs: 1 active+recovery_wait+remapped, 7 peering, 2 active+clean+remapped, 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 2 remapped+peering, 8 active+undersized+degraded+remapped+backfill_wait, 81 active+clean; 4.3 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 13 KiB/s rd, 38 KiB/s wr, 970 op/s; 41366/212397 objects degraded (19.476%); 26449/212397 objects misplaced (12.453%); 6.6 MiB/s, 105 objects/s recovering 2024-09-21T11:04:41.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:41 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1046430023' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:04:41.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:41 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:04:41.607 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:41.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:41.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:41.709 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:41.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:41.741 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:41.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:41.842 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:41.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:41 smithi070 ceph-mon[107385]: pgmap v3182: 109 pgs: 1 active+recovery_wait+remapped, 7 peering, 2 active+clean+remapped, 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 2 remapped+peering, 8 active+undersized+degraded+remapped+backfill_wait, 81 active+clean; 4.3 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 13 KiB/s rd, 38 KiB/s wr, 970 op/s; 41366/212397 objects degraded (19.476%); 26449/212397 objects misplaced (12.453%); 6.6 MiB/s, 105 objects/s recovering 2024-09-21T11:04:41.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:41 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1046430023' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:04:41.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:41 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:04:41.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:41 smithi070 ceph-mon[112139]: pgmap v3182: 109 pgs: 1 active+recovery_wait+remapped, 7 peering, 2 active+clean+remapped, 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 2 remapped+peering, 8 active+undersized+degraded+remapped+backfill_wait, 81 active+clean; 4.3 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 13 KiB/s rd, 38 KiB/s wr, 970 op/s; 41366/212397 objects degraded (19.476%); 26449/212397 objects misplaced (12.453%); 6.6 MiB/s, 105 objects/s recovering 2024-09-21T11:04:41.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:41 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1046430023' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:04:41.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:41 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:04:41.888 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:41.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:41.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:41.990 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:42.029 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:42.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:42.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:42.131 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:42.159 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:42.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:42.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:42.260 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:42.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:42.309 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:42.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:42.410 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:42.453 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:42.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:42.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:42.555 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:42.584 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:42.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:42.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:42.686 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:42.713 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:42.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:42.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:42 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:04:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:42 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T11:04:42.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:42 smithi160 ceph-mon[101879]: osdmap e1609: 8 total, 8 up, 5 in 2024-09-21T11:04:42.811 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T11:04:42.811 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T11:04:42.815 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:42.841 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:42.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:42.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:42.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:42 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:04:42.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:42 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T11:04:42.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:42 smithi070 ceph-mon[112139]: osdmap e1609: 8 total, 8 up, 5 in 2024-09-21T11:04:42.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:42 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:04:42.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:42 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T11:04:42.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:42 smithi070 ceph-mon[107385]: osdmap e1609: 8 total, 8 up, 5 in 2024-09-21T11:04:42.943 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:42.970 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:42.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:42.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:43.072 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:43.098 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:43.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:43.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:43.200 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:43.229 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:43.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:43.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:43.330 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:43.371 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:43.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:43.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:43.469 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:04:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:04:43] "GET /metrics HTTP/1.1" 200 36742 "" "Prometheus/2.51.0" 2024-09-21T11:04:43.473 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:43.493 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T11:04:43.499 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:43.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:43.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:43.601 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:43.627 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:43.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:43.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:43.729 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:43.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:43.751 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:43.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:43 smithi160 ceph-mon[101879]: pgmap v3184: 109 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 8 active+undersized+degraded+remapped+backfill_wait, 90 active+clean; 4.1 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 4.0 KiB/s rd, 12 KiB/s wr, 787 op/s; 39757/203829 objects degraded (19.505%); 25091/203829 objects misplaced (12.310%); 7.5 MiB/s, 121 objects/s recovering 2024-09-21T11:04:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:43 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 39757/203829 objects degraded (19.505%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:43.853 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:43.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:43 smithi070 ceph-mon[107385]: pgmap v3184: 109 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 8 active+undersized+degraded+remapped+backfill_wait, 90 active+clean; 4.1 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 4.0 KiB/s rd, 12 KiB/s wr, 787 op/s; 39757/203829 objects degraded (19.505%); 25091/203829 objects misplaced (12.310%); 7.5 MiB/s, 121 objects/s recovering 2024-09-21T11:04:43.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:43 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 39757/203829 objects degraded (19.505%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:43.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:43 smithi070 ceph-mon[112139]: pgmap v3184: 109 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 8 active+undersized+degraded+remapped+backfill_wait, 90 active+clean; 4.1 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 4.0 KiB/s rd, 12 KiB/s wr, 787 op/s; 39757/203829 objects degraded (19.505%); 25091/203829 objects misplaced (12.310%); 7.5 MiB/s, 121 objects/s recovering 2024-09-21T11:04:43.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:43 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 39757/203829 objects degraded (19.505%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:43.878 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:43.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:43.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:43.980 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:44.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:44.037 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:44.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:44.138 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:44.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:44.184 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:44.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:44.285 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:44.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:44.335 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:44.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:44.436 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:44.464 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:44.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:44.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:44.565 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:44.600 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:44.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:44.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:44.638 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:04:44.703 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:44.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:44.728 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:44.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:44.829 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:44.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:44.864 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:44.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:44.965 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:44.993 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:44.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:44.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:45.095 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:45.128 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:45.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:45.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:45.230 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:45.269 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:45.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:45.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:45.371 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:45.399 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:45.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:45.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:45.501 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:45.531 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:45.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:45.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:45.632 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:45.659 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:45.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:45.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:45.729 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:45 smithi070 ceph-mon[112139]: pgmap v3185: 109 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 8 active+undersized+degraded+remapped+backfill_wait, 90 active+clean; 4.1 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 3.1 KiB/s rd, 9.2 KiB/s wr, 606 op/s; 39757/203829 objects degraded (19.505%); 25091/203829 objects misplaced (12.310%); 5.8 MiB/s, 93 objects/s recovering 2024-09-21T11:04:45.729 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:45 smithi070 ceph-mon[107385]: pgmap v3185: 109 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 8 active+undersized+degraded+remapped+backfill_wait, 90 active+clean; 4.1 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 3.1 KiB/s rd, 9.2 KiB/s wr, 606 op/s; 39757/203829 objects degraded (19.505%); 25091/203829 objects misplaced (12.310%); 5.8 MiB/s, 93 objects/s recovering 2024-09-21T11:04:45.761 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:45.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:45.788 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:45.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:45.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:45 smithi160 ceph-mon[101879]: pgmap v3185: 109 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 8 active+undersized+degraded+remapped+backfill_wait, 90 active+clean; 4.1 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 3.1 KiB/s rd, 9.2 KiB/s wr, 606 op/s; 39757/203829 objects degraded (19.505%); 25091/203829 objects misplaced (12.310%); 5.8 MiB/s, 93 objects/s recovering 2024-09-21T11:04:45.889 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:45.927 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:45.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:45.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:46.028 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:46.068 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:46.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:46.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:46.170 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:46.198 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:46.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:46.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:46.299 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:46.328 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:46.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:46.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:46.430 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:46.465 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:46.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:46.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:46.567 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:46.577 INFO:teuthology.orchestra.run.smithi070.stdout:moved 59 / 327 (18.0428%) 2024-09-21T11:04:46.578 INFO:teuthology.orchestra.run.smithi070.stdout:avg 65.4 2024-09-21T11:04:46.578 INFO:teuthology.orchestra.run.smithi070.stdout:stddev 27.3657 -> 25.303 (expected baseline 7.23326) 2024-09-21T11:04:46.578 INFO:teuthology.orchestra.run.smithi070.stdout:min osd.3 with 21 -> 39 pgs (0.321101 -> 0.59633 * mean) 2024-09-21T11:04:46.578 INFO:teuthology.orchestra.run.smithi070.stdout:max osd.6 with 90 -> 100 pgs (1.37615 -> 1.52905 * mean) 2024-09-21T11:04:46.578 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T11:04:46.578 INFO:teuthology.orchestra.run.smithi070.stdout:oload 110 2024-09-21T11:04:46.578 INFO:teuthology.orchestra.run.smithi070.stdout:max_change 1 2024-09-21T11:04:46.578 INFO:teuthology.orchestra.run.smithi070.stdout:max_change_osds 4 2024-09-21T11:04:46.578 INFO:teuthology.orchestra.run.smithi070.stdout:average_utilization 0.0360 2024-09-21T11:04:46.578 INFO:teuthology.orchestra.run.smithi070.stdout:overload_utilization 0.0396 2024-09-21T11:04:46.578 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 weight 0.6888 -> 0.4793 2024-09-21T11:04:46.578 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 weight 0.4107 -> 0.5576 2024-09-21T11:04:46.579 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 weight 0.8303 -> 0.6888 2024-09-21T11:04:46.579 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 weight 1.0000 -> 0.6739 2024-09-21T11:04:46.599 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:46.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:46.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:46.701 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:46.724 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:46.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:46.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:46.826 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:46 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'44164','3':'36540','4':'45144','7':'31414'}"} : dispatch 2024-09-21T11:04:46.847 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:46.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:46.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:46.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:46 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'44164','3':'36540','4':'45144','7':'31414'}"} : dispatch 2024-09-21T11:04:46.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:46 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'44164','3':'36540','4':'45144','7':'31414'}"} : dispatch 2024-09-21T11:04:46.948 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:46.971 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:46.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:46.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:47.073 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:47.094 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:47.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:47.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:47.196 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:47.219 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:47.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:47.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:47.287 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-21T11:04:47.288 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 1000 3.0 2024-09-21T11:04:47.310 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:04:47.320 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:47.348 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:47.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:47.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:47.450 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:47.472 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:47.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:47.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:47.574 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:47.587 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:47 smithi070 ceph-mon[107385]: from='client.46679 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T11:04:47.587 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:47 smithi070 ceph-mon[107385]: pgmap v3186: 109 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 8 active+undersized+degraded+remapped+backfill_wait, 90 active+clean; 3.8 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 1.11k op/s; 36706/188670 objects degraded (19.455%); 22589/188670 objects misplaced (11.973%); 12 MiB/s, 187 objects/s recovering 2024-09-21T11:04:47.587 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:47 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'44164','3':'36540','4':'45144','7':'31414'}"}]': finished 2024-09-21T11:04:47.587 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:47 smithi070 ceph-mon[107385]: osdmap e1610: 8 total, 8 up, 5 in 2024-09-21T11:04:47.607 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:47.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:47.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:47.708 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:47.733 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:47.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:47.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:47.835 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:47.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:47 smithi160 ceph-mon[101879]: from='client.46679 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T11:04:47.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:47 smithi160 ceph-mon[101879]: pgmap v3186: 109 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 8 active+undersized+degraded+remapped+backfill_wait, 90 active+clean; 3.8 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 1.11k op/s; 36706/188670 objects degraded (19.455%); 22589/188670 objects misplaced (11.973%); 12 MiB/s, 187 objects/s recovering 2024-09-21T11:04:47.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:47 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'44164','3':'36540','4':'45144','7':'31414'}"}]': finished 2024-09-21T11:04:47.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:47 smithi160 ceph-mon[101879]: osdmap e1610: 8 total, 8 up, 5 in 2024-09-21T11:04:47.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:47.859 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:47.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:47.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:47 smithi070 ceph-mon[112139]: from='client.46679 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T11:04:47.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:47 smithi070 ceph-mon[112139]: pgmap v3186: 109 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 8 active+undersized+degraded+remapped+backfill_wait, 90 active+clean; 3.8 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 1.11k op/s; 36706/188670 objects degraded (19.455%); 22589/188670 objects misplaced (11.973%); 12 MiB/s, 187 objects/s recovering 2024-09-21T11:04:47.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:47 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'44164','3':'36540','4':'45144','7':'31414'}"}]': finished 2024-09-21T11:04:47.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:47 smithi070 ceph-mon[112139]: osdmap e1610: 8 total, 8 up, 5 in 2024-09-21T11:04:47.960 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:47.995 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:47.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:47.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:48.097 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:48.120 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:48.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:48.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:48.222 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:48.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:48.245 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:48.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:48.346 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:48.368 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:48.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:48.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:48.470 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:48.493 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:48.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:48.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:48.594 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:48.595 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:48 smithi070 ceph-mon[112139]: osdmap e1611: 8 total, 8 up, 5 in 2024-09-21T11:04:48.595 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:48 smithi070 ceph-mon[112139]: 58.8 continuing backfill to osd.6 from (1598'10721,1609'14260] MIN to 1609'14260 2024-09-21T11:04:48.595 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:48 smithi070 ceph-mon[112139]: 58.6 continuing backfill to osd.6 from (1598'10996,1609'14602] MIN to 1609'14602 2024-09-21T11:04:48.595 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:48 smithi070 ceph-mon[112139]: 58.3 continuing backfill to osd.6 from (1598'10842,1609'14382] MIN to 1609'14382 2024-09-21T11:04:48.595 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:48 smithi070 ceph-mon[112139]: 58.d starting backfill to osd.3 from (0'0,0'0] MAX to 1609'14429 2024-09-21T11:04:48.595 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:48 smithi070 ceph-mon[112139]: 58.7 continuing backfill to osd.1 from (1598'10684,1609'14309] MIN to 1609'14309 2024-09-21T11:04:48.596 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:48 smithi070 ceph-mon[107385]: osdmap e1611: 8 total, 8 up, 5 in 2024-09-21T11:04:48.596 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:48 smithi070 ceph-mon[107385]: 58.8 continuing backfill to osd.6 from (1598'10721,1609'14260] MIN to 1609'14260 2024-09-21T11:04:48.596 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:48 smithi070 ceph-mon[107385]: 58.6 continuing backfill to osd.6 from (1598'10996,1609'14602] MIN to 1609'14602 2024-09-21T11:04:48.596 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:48 smithi070 ceph-mon[107385]: 58.3 continuing backfill to osd.6 from (1598'10842,1609'14382] MIN to 1609'14382 2024-09-21T11:04:48.596 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:48 smithi070 ceph-mon[107385]: 58.d starting backfill to osd.3 from (0'0,0'0] MAX to 1609'14429 2024-09-21T11:04:48.596 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:48 smithi070 ceph-mon[107385]: 58.7 continuing backfill to osd.1 from (1598'10684,1609'14309] MIN to 1609'14309 2024-09-21T11:04:48.596 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:48 smithi070 ceph-mon[107385]: 58.7 starting backfill to osd.3 from (0'0,0'0] MAX to 1609'14309 2024-09-21T11:04:48.596 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:48 smithi070 ceph-mon[107385]: 58.e continuing backfill to osd.4 from (1594'7900,1609'14218] MIN to 1609'14218 2024-09-21T11:04:48.596 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:48 smithi070 ceph-mon[107385]: 58.e starting backfill to osd.6 from (0'0,0'0] MAX to 1609'14218 2024-09-21T11:04:48.596 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:48 smithi070 ceph-mon[107385]: 58.7 continuing backfill to osd.6 from (1598'10684,1609'14309] MIN to 1609'14309 2024-09-21T11:04:48.597 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:48 smithi070 ceph-mon[107385]: 58.d continuing backfill to osd.6 from (1598'10844,1609'14429] 58:b00128c3:::benchmark_data_smithi070_396992_object127114:head to 1609'14429 2024-09-21T11:04:48.597 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:48 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 36706/188670 objects degraded (19.455%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:48.617 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:48.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:48.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:48.671 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T11:04:48.694 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:04:48.719 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:48.757 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:48.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:48.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:48.859 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:48.874 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:48 smithi070 ceph-mon[112139]: 58.7 starting backfill to osd.3 from (0'0,0'0] MAX to 1609'14309 2024-09-21T11:04:48.875 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:48 smithi070 ceph-mon[112139]: 58.e continuing backfill to osd.4 from (1594'7900,1609'14218] MIN to 1609'14218 2024-09-21T11:04:48.875 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:48 smithi070 ceph-mon[112139]: 58.e starting backfill to osd.6 from (0'0,0'0] MAX to 1609'14218 2024-09-21T11:04:48.875 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:48 smithi070 ceph-mon[112139]: 58.7 continuing backfill to osd.6 from (1598'10684,1609'14309] MIN to 1609'14309 2024-09-21T11:04:48.875 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:48 smithi070 ceph-mon[112139]: 58.d continuing backfill to osd.6 from (1598'10844,1609'14429] 58:b00128c3:::benchmark_data_smithi070_396992_object127114:head to 1609'14429 2024-09-21T11:04:48.875 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:48 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 36706/188670 objects degraded (19.455%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:48.883 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:48 smithi160 ceph-mon[101879]: osdmap e1611: 8 total, 8 up, 5 in 2024-09-21T11:04:48.883 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:48 smithi160 ceph-mon[101879]: 58.8 continuing backfill to osd.6 from (1598'10721,1609'14260] MIN to 1609'14260 2024-09-21T11:04:48.883 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:48 smithi160 ceph-mon[101879]: 58.6 continuing backfill to osd.6 from (1598'10996,1609'14602] MIN to 1609'14602 2024-09-21T11:04:48.883 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:48 smithi160 ceph-mon[101879]: 58.3 continuing backfill to osd.6 from (1598'10842,1609'14382] MIN to 1609'14382 2024-09-21T11:04:48.883 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:48 smithi160 ceph-mon[101879]: 58.d starting backfill to osd.3 from (0'0,0'0] MAX to 1609'14429 2024-09-21T11:04:48.883 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:48 smithi160 ceph-mon[101879]: 58.7 continuing backfill to osd.1 from (1598'10684,1609'14309] MIN to 1609'14309 2024-09-21T11:04:48.883 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:48 smithi160 ceph-mon[101879]: 58.7 starting backfill to osd.3 from (0'0,0'0] MAX to 1609'14309 2024-09-21T11:04:48.883 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:48 smithi160 ceph-mon[101879]: 58.e continuing backfill to osd.4 from (1594'7900,1609'14218] MIN to 1609'14218 2024-09-21T11:04:48.883 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:48 smithi160 ceph-mon[101879]: 58.e starting backfill to osd.6 from (0'0,0'0] MAX to 1609'14218 2024-09-21T11:04:48.883 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:48 smithi160 ceph-mon[101879]: 58.7 continuing backfill to osd.6 from (1598'10684,1609'14309] MIN to 1609'14309 2024-09-21T11:04:48.883 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:48 smithi160 ceph-mon[101879]: 58.d continuing backfill to osd.6 from (1598'10844,1609'14429] 58:b00128c3:::benchmark_data_smithi070_396992_object127114:head to 1609'14429 2024-09-21T11:04:48.883 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:48 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 36706/188670 objects degraded (19.455%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:48.892 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:48.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:48.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:48.994 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:49.031 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:49.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:49.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:49.133 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:49.178 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:49.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:49.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:49.279 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:49.311 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:49.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:49.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:49.413 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:49.443 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:49.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:49.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:49.545 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:49.574 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:49.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:49.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:49.675 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:49.716 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:49.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:49.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:49.817 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:49.843 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:49.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:49.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:49.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:49 smithi070 ceph-mon[107385]: pgmap v3189: 109 pgs: 9 peering, 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 1 remapped+peering, 8 unknown, 8 active+undersized+degraded+remapped+backfill_wait, 73 active+clean; 3.8 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 7.4 KiB/s rd, 22 KiB/s wr, 752 op/s; 36706/188670 objects degraded (19.455%); 22589/188670 objects misplaced (11.973%); 8.6 MiB/s, 136 objects/s recovering 2024-09-21T11:04:49.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:49 smithi070 ceph-mon[107385]: Health check failed: Reduced data availability: 3 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T11:04:49.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:49 smithi070 ceph-mon[107385]: osdmap e1612: 8 total, 8 up, 5 in 2024-09-21T11:04:49.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:49 smithi070 ceph-mon[107385]: 58.0 starting backfill to osd.6 from (0'0,0'0] MAX to 1609'14514 2024-09-21T11:04:49.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:49 smithi070 ceph-mon[112139]: pgmap v3189: 109 pgs: 9 peering, 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 1 remapped+peering, 8 unknown, 8 active+undersized+degraded+remapped+backfill_wait, 73 active+clean; 3.8 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 7.4 KiB/s rd, 22 KiB/s wr, 752 op/s; 36706/188670 objects degraded (19.455%); 22589/188670 objects misplaced (11.973%); 8.6 MiB/s, 136 objects/s recovering 2024-09-21T11:04:49.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:49 smithi070 ceph-mon[112139]: Health check failed: Reduced data availability: 3 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T11:04:49.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:49 smithi070 ceph-mon[112139]: osdmap e1612: 8 total, 8 up, 5 in 2024-09-21T11:04:49.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:49 smithi070 ceph-mon[112139]: 58.0 starting backfill to osd.6 from (0'0,0'0] MAX to 1609'14514 2024-09-21T11:04:49.945 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:49.984 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:49.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:49.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:50.014 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:49 smithi160 ceph-mon[101879]: pgmap v3189: 109 pgs: 9 peering, 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 1 remapped+peering, 8 unknown, 8 active+undersized+degraded+remapped+backfill_wait, 73 active+clean; 3.8 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 7.4 KiB/s rd, 22 KiB/s wr, 752 op/s; 36706/188670 objects degraded (19.455%); 22589/188670 objects misplaced (11.973%); 8.6 MiB/s, 136 objects/s recovering 2024-09-21T11:04:50.015 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:49 smithi160 ceph-mon[101879]: Health check failed: Reduced data availability: 3 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T11:04:50.015 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:49 smithi160 ceph-mon[101879]: osdmap e1612: 8 total, 8 up, 5 in 2024-09-21T11:04:50.015 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:49 smithi160 ceph-mon[101879]: 58.0 starting backfill to osd.6 from (0'0,0'0] MAX to 1609'14514 2024-09-21T11:04:50.085 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:50.127 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:50.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:50.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:50.229 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:50.271 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:50.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:50.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:50.373 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:50.402 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:50.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:50.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:50.504 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:50.545 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:50.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:50.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:50.647 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:50.674 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T11:04:50.674 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T11:04:50.693 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:50.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:50.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:50.792 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:04:50.795 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:50.835 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:50.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:50.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:50.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:50 smithi070 ceph-mon[107385]: osdmap e1613: 8 total, 8 up, 5 in 2024-09-21T11:04:50.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:04:50.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:04:50.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:50 smithi070 ceph-mon[112139]: osdmap e1613: 8 total, 8 up, 5 in 2024-09-21T11:04:50.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:04:50.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:04:50.936 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:50.964 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:50.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:50.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:51.045 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:50 smithi160 ceph-mon[101879]: osdmap e1613: 8 total, 8 up, 5 in 2024-09-21T11:04:51.045 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:04:51.045 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:04:51.065 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:51.090 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:51.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:51.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:51.192 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:51.229 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:51.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:51.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:51.331 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:51.366 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:51.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:51.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:51.468 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:51.490 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:51.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:51.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:51.592 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:51.620 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:51.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:51.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:51.722 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:51.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:51.753 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:51.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:51.854 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:51.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:51 smithi070 ceph-mon[107385]: pgmap v3192: 109 pgs: 9 peering, 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded+remapped, 5 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 remapped+peering, 8 unknown, 7 active+undersized+degraded+remapped+backfill_wait, 70 active+clean; 3.8 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 12 KiB/s rd, 36 KiB/s wr, 355 op/s; 35437/184512 objects degraded (19.206%); 29386/184512 objects misplaced (15.926%); 3.1 MiB/s, 116 objects/s recovering 2024-09-21T11:04:51.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:51 smithi070 ceph-mon[107385]: osdmap e1614: 8 total, 8 up, 5 in 2024-09-21T11:04:51.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:51 smithi070 ceph-mon[112139]: pgmap v3192: 109 pgs: 9 peering, 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded+remapped, 5 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 remapped+peering, 8 unknown, 7 active+undersized+degraded+remapped+backfill_wait, 70 active+clean; 3.8 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 12 KiB/s rd, 36 KiB/s wr, 355 op/s; 35437/184512 objects degraded (19.206%); 29386/184512 objects misplaced (15.926%); 3.1 MiB/s, 116 objects/s recovering 2024-09-21T11:04:51.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:51 smithi070 ceph-mon[112139]: osdmap e1614: 8 total, 8 up, 5 in 2024-09-21T11:04:51.886 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:51.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:51.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:51.988 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:52.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:52.033 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:52.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:52.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:51 smithi160 ceph-mon[101879]: pgmap v3192: 109 pgs: 9 peering, 1 active+recovery_wait+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded+remapped, 5 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 remapped+peering, 8 unknown, 7 active+undersized+degraded+remapped+backfill_wait, 70 active+clean; 3.8 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 12 KiB/s rd, 36 KiB/s wr, 355 op/s; 35437/184512 objects degraded (19.206%); 29386/184512 objects misplaced (15.926%); 3.1 MiB/s, 116 objects/s recovering 2024-09-21T11:04:52.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:51 smithi160 ceph-mon[101879]: osdmap e1614: 8 total, 8 up, 5 in 2024-09-21T11:04:52.134 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:52.164 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:52.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:52.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:52.265 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:52.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:52.302 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:52.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:52.402 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:52.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:52.438 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:52.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:52.539 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:52.573 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:52.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:52.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:52.674 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:52.697 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:52.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:52.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:52.733 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:52 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2636088464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:04:52.734 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:52 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2636088464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:04:52.799 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:52.831 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:52.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:52.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:52.847 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:04:52.847 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:04:52.847 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:04:52.847 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:04:52.847 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:04:52.847 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T11:04:52.847 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T11:04:52.847 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:04:52.848 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:04:52.848 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:04:52.848 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:04:52.848 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:04:52.848 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T11:04:52.848 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T11:04:52.848 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:04:52.848 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:04:52.933 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:52.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:52.977 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:52.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:52.978 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:52 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2636088464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:04:53.079 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:53.115 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:53.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:53.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:53.216 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:53.247 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T11:04:53.265 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:53.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:53.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:53.367 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:53.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:53.410 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:53.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:53.511 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:53.540 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:53.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:53.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:53.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:04:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:04:53] "GET /metrics HTTP/1.1" 200 36769 "" "Prometheus/2.51.0" 2024-09-21T11:04:53.637 INFO:teuthology.orchestra.run.smithi070.stdout:moved 13 / 327 (3.97554%) 2024-09-21T11:04:53.638 INFO:teuthology.orchestra.run.smithi070.stdout:avg 65.4 2024-09-21T11:04:53.638 INFO:teuthology.orchestra.run.smithi070.stdout:stddev 25.303 -> 22.3652 (expected baseline 7.23326) 2024-09-21T11:04:53.638 INFO:teuthology.orchestra.run.smithi070.stdout:min osd.7 with 30 -> 27 pgs (0.458716 -> 0.412844 * mean) 2024-09-21T11:04:53.638 INFO:teuthology.orchestra.run.smithi070.stdout:max osd.6 with 100 -> 96 pgs (1.52905 -> 1.46789 * mean) 2024-09-21T11:04:53.638 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T11:04:53.638 INFO:teuthology.orchestra.run.smithi070.stdout:oload 1000 2024-09-21T11:04:53.638 INFO:teuthology.orchestra.run.smithi070.stdout:max_change 3 2024-09-21T11:04:53.638 INFO:teuthology.orchestra.run.smithi070.stdout:max_change_osds 4 2024-09-21T11:04:53.638 INFO:teuthology.orchestra.run.smithi070.stdout:average_utilization 0.0336 2024-09-21T11:04:53.638 INFO:teuthology.orchestra.run.smithi070.stdout:overload_utilization 0.3362 2024-09-21T11:04:53.638 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 weight 0.5576 -> 0.7003 2024-09-21T11:04:53.642 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:53.679 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:53.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:53.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:53.768 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:04:53.781 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:53.802 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:53.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:53.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:53.889 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:53 smithi160 ceph-mon[101879]: pgmap v3194: 109 pgs: 2 active+recovery_wait+remapped, 2 active+undersized+degraded+remapped, 21 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 7 active+undersized+degraded+remapped+backfill_wait, 71 active+clean; 3.6 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 5.3 KiB/s rd, 16 KiB/s wr, 663 op/s; 34460/179280 objects degraded (19.221%); 50158/179280 objects misplaced (27.977%); 18 MiB/s, 345 objects/s recovering 2024-09-21T11:04:53.889 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:53 smithi160 ceph-mon[101879]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 1 pg peering) 2024-09-21T11:04:53.890 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:53 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'3':'45893'}"} : dispatch 2024-09-21T11:04:53.890 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:53 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 34460/179280 objects degraded (19.221%), 10 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:53.904 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:53.925 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:53.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:53.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:54.027 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:54.048 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:54.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:54.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:54.122 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:53 smithi070 ceph-mon[112139]: pgmap v3194: 109 pgs: 2 active+recovery_wait+remapped, 2 active+undersized+degraded+remapped, 21 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 7 active+undersized+degraded+remapped+backfill_wait, 71 active+clean; 3.6 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 5.3 KiB/s rd, 16 KiB/s wr, 663 op/s; 34460/179280 objects degraded (19.221%); 50158/179280 objects misplaced (27.977%); 18 MiB/s, 345 objects/s recovering 2024-09-21T11:04:54.123 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:53 smithi070 ceph-mon[112139]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 1 pg peering) 2024-09-21T11:04:54.123 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:53 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'3':'45893'}"} : dispatch 2024-09-21T11:04:54.123 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:53 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 34460/179280 objects degraded (19.221%), 10 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:54.123 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:53 smithi070 ceph-mon[107385]: pgmap v3194: 109 pgs: 2 active+recovery_wait+remapped, 2 active+undersized+degraded+remapped, 21 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 7 active+undersized+degraded+remapped+backfill_wait, 71 active+clean; 3.6 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 5.3 KiB/s rd, 16 KiB/s wr, 663 op/s; 34460/179280 objects degraded (19.221%); 50158/179280 objects misplaced (27.977%); 18 MiB/s, 345 objects/s recovering 2024-09-21T11:04:54.123 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:53 smithi070 ceph-mon[107385]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 1 pg peering) 2024-09-21T11:04:54.123 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:53 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'3':'45893'}"} : dispatch 2024-09-21T11:04:54.123 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:53 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 34460/179280 objects degraded (19.221%), 10 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:54.150 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:54.171 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:54.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:54.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:54.272 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:54.293 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:54.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:54.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:54.395 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:54.416 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:54.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:54.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:54.518 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:54.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:54.542 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:54.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:54.548 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-21T11:04:54.548 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-09-21T11:04:54.643 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:54.678 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:54.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:54.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:54.779 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:54.824 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:54.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:54.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:54.853 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:54 smithi070 ceph-mon[112139]: from='client.46691 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T11:04:54.853 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:54 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'45893'}"}]': finished 2024-09-21T11:04:54.853 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:54 smithi070 ceph-mon[112139]: osdmap e1615: 8 total, 8 up, 5 in 2024-09-21T11:04:54.853 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:54 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:04:54.853 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:54 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:04:54.853 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:54 smithi070 ceph-mon[107385]: from='client.46691 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T11:04:54.854 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:54 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'45893'}"}]': finished 2024-09-21T11:04:54.854 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:54 smithi070 ceph-mon[107385]: osdmap e1615: 8 total, 8 up, 5 in 2024-09-21T11:04:54.854 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:54 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:04:54.854 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:54 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:04:54.925 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:54.962 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:54.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:54.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:55.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:54 smithi160 ceph-mon[101879]: from='client.46691 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T11:04:55.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:54 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'45893'}"}]': finished 2024-09-21T11:04:55.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:54 smithi160 ceph-mon[101879]: osdmap e1615: 8 total, 8 up, 5 in 2024-09-21T11:04:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:54 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:04:55.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:54 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:04:55.064 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:55.091 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:55.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:55.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:55.192 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:55.224 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:55.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:55.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:55.325 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:55.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:55.385 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:55.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:55.486 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:55.516 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:55.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:55.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:55.618 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:55.656 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:55.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:55.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:55.758 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:55.784 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T11:04:55.801 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:55.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:55.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:55.826 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:55 smithi070 ceph-mon[107385]: pgmap v3196: 109 pgs: 2 active+recovery_wait+remapped, 2 active+undersized+degraded+remapped, 21 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 7 active+undersized+degraded+remapped+backfill_wait, 71 active+clean; 3.6 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 4.4 KiB/s rd, 13 KiB/s wr, 548 op/s; 34460/179280 objects degraded (19.221%); 50158/179280 objects misplaced (27.977%); 15 MiB/s, 286 objects/s recovering 2024-09-21T11:04:55.827 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:55 smithi070 ceph-mon[107385]: osdmap e1616: 8 total, 8 up, 5 in 2024-09-21T11:04:55.827 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:55 smithi070 ceph-mon[107385]: 58.8 starting backfill to osd.3 from (0'0,0'0] MAX to 1614'14596 2024-09-21T11:04:55.827 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:55 smithi070 ceph-mon[107385]: 58.1 starting backfill to osd.3 from (0'0,0'0] MAX to 1614'14533 2024-09-21T11:04:55.827 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:55 smithi070 ceph-mon[107385]: 58.3 starting backfill to osd.3 from (0'0,0'0] MAX to 1614'14701 2024-09-21T11:04:55.827 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:55 smithi070 ceph-mon[107385]: 58.3 continuing backfill to osd.6 from (1598'10842,1614'14701] MIN to 1614'14701 2024-09-21T11:04:55.827 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:55 smithi070 ceph-mon[107385]: 58.1 continuing backfill to osd.6 from (1598'10659,1614'14533] MIN to 1614'14533 2024-09-21T11:04:55.827 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:55 smithi070 ceph-mon[107385]: 58.8 starting backfill to osd.6 from (1598'10721,1614'14593] MIN to 1614'14596 2024-09-21T11:04:55.827 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:55 smithi070 ceph-mon[112139]: pgmap v3196: 109 pgs: 2 active+recovery_wait+remapped, 2 active+undersized+degraded+remapped, 21 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 7 active+undersized+degraded+remapped+backfill_wait, 71 active+clean; 3.6 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 4.4 KiB/s rd, 13 KiB/s wr, 548 op/s; 34460/179280 objects degraded (19.221%); 50158/179280 objects misplaced (27.977%); 15 MiB/s, 286 objects/s recovering 2024-09-21T11:04:55.827 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:55 smithi070 ceph-mon[112139]: osdmap e1616: 8 total, 8 up, 5 in 2024-09-21T11:04:55.827 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:55 smithi070 ceph-mon[112139]: 58.8 starting backfill to osd.3 from (0'0,0'0] MAX to 1614'14596 2024-09-21T11:04:55.827 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:55 smithi070 ceph-mon[112139]: 58.1 starting backfill to osd.3 from (0'0,0'0] MAX to 1614'14533 2024-09-21T11:04:55.827 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:55 smithi070 ceph-mon[112139]: 58.3 starting backfill to osd.3 from (0'0,0'0] MAX to 1614'14701 2024-09-21T11:04:55.827 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:55 smithi070 ceph-mon[112139]: 58.3 continuing backfill to osd.6 from (1598'10842,1614'14701] MIN to 1614'14701 2024-09-21T11:04:55.828 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:55 smithi070 ceph-mon[112139]: 58.1 continuing backfill to osd.6 from (1598'10659,1614'14533] MIN to 1614'14533 2024-09-21T11:04:55.828 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:55 smithi070 ceph-mon[112139]: 58.8 starting backfill to osd.6 from (1598'10721,1614'14593] MIN to 1614'14596 2024-09-21T11:04:55.899 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:55 smithi160 ceph-mon[101879]: pgmap v3196: 109 pgs: 2 active+recovery_wait+remapped, 2 active+undersized+degraded+remapped, 21 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 7 active+undersized+degraded+remapped+backfill_wait, 71 active+clean; 3.6 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 4.4 KiB/s rd, 13 KiB/s wr, 548 op/s; 34460/179280 objects degraded (19.221%); 50158/179280 objects misplaced (27.977%); 15 MiB/s, 286 objects/s recovering 2024-09-21T11:04:55.899 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:55 smithi160 ceph-mon[101879]: osdmap e1616: 8 total, 8 up, 5 in 2024-09-21T11:04:55.899 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:55 smithi160 ceph-mon[101879]: 58.8 starting backfill to osd.3 from (0'0,0'0] MAX to 1614'14596 2024-09-21T11:04:55.899 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:55 smithi160 ceph-mon[101879]: 58.1 starting backfill to osd.3 from (0'0,0'0] MAX to 1614'14533 2024-09-21T11:04:55.900 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:55 smithi160 ceph-mon[101879]: 58.3 starting backfill to osd.3 from (0'0,0'0] MAX to 1614'14701 2024-09-21T11:04:55.900 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:55 smithi160 ceph-mon[101879]: 58.3 continuing backfill to osd.6 from (1598'10842,1614'14701] MIN to 1614'14701 2024-09-21T11:04:55.900 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:55 smithi160 ceph-mon[101879]: 58.1 continuing backfill to osd.6 from (1598'10659,1614'14533] MIN to 1614'14533 2024-09-21T11:04:55.900 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:55 smithi160 ceph-mon[101879]: 58.8 starting backfill to osd.6 from (1598'10721,1614'14593] MIN to 1614'14596 2024-09-21T11:04:55.903 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:55.944 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:55.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:55.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:56.046 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:04:56.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:56.103 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:56.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:04:56.204 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:56.256 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:56.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:56.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:56.324 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:04:56.358 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:56.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:56.396 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:56.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:56.497 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:56.538 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:56.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:56.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:56.640 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:56.675 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T11:04:56.675 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T11:04:56.675 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T11:04:56.675 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T11:04:56.675 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.675 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:29067 57.9 57:964842a6:::benchmark_data_smithi070_360420_object29066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1537)", 2024-09-21T11:04:56.675 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:58:32.636312+0000", 2024-09-21T11:04:56.675 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 383.18459554999998, 2024-09-21T11:04:56.676 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0001596829999999, 2024-09-21T11:04:56.676 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:04:56.676 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T11:04:56.676 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:04:56.676 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T11:04:56.676 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:04:56.676 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 29067 2024-09-21T11:04:56.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.676 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:04:56.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.676 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:04:56.676 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.636312+0000", 2024-09-21T11:04:56.676 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:04:56.677 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.636312+0000", 2024-09-21T11:04:56.677 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:04:56.677 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.636311+0000", 2024-09-21T11:04:56.677 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T11:04:56.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:04:56.677 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.637030+0000", 2024-09-21T11:04:56.677 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00071977200000000003 2024-09-21T11:04:56.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.678 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:04:56.678 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.637045+0000", 2024-09-21T11:04:56.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4175999999999999e-05 2024-09-21T11:04:56.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.678 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:04:56.678 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.637054+0000", 2024-09-21T11:04:56.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0489999999999993e-06 2024-09-21T11:04:56.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.679 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.679 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.637103+0000", 2024-09-21T11:04:56.679 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9271999999999999e-05 2024-09-21T11:04:56.679 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.679 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.679 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op must wait for map", 2024-09-21T11:04:56.679 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.637110+0000", 2024-09-21T11:04:56.679 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7989999999999996e-06 2024-09-21T11:04:56.679 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.679 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.679 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.679 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.648728+0000", 2024-09-21T11:04:56.679 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011618694000000001 2024-09-21T11:04:56.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:04:56.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.648738+0000", 2024-09-21T11:04:56.680 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0131e-05 2024-09-21T11:04:56.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.636419+0000", 2024-09-21T11:04:56.680 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98768005199999998 2024-09-21T11:04:56.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:04:56.681 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.636472+0000", 2024-09-21T11:04:56.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3508000000000003e-05 2024-09-21T11:04:56.681 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.681 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:04:56.681 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.681 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:29090 57.9 57:9c9b276d:::benchmark_data_smithi070_360420_object29089:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1537)", 2024-09-21T11:04:56.681 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:58:32.649590+0000", 2024-09-21T11:04:56.681 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 383.17131774699999, 2024-09-21T11:04:56.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99083443800000004, 2024-09-21T11:04:56.681 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:04:56.681 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:04:56.681 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:04:56.682 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T11:04:56.682 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:04:56.682 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 29090 2024-09-21T11:04:56.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.682 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:04:56.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:04:56.682 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.649590+0000", 2024-09-21T11:04:56.682 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:04:56.682 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.649590+0000", 2024-09-21T11:04:56.682 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:04:56.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.649588+0000", 2024-09-21T11:04:56.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 2024-09-21T11:04:56.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:04:56.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.649630+0000", 2024-09-21T11:04:56.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2567000000000001e-05 2024-09-21T11:04:56.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:04:56.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.649644+0000", 2024-09-21T11:04:56.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3338e-05 2024-09-21T11:04:56.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:04:56.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.649652+0000", 2024-09-21T11:04:56.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8439999999999998e-06 2024-09-21T11:04:56.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.649774+0000", 2024-09-21T11:04:56.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012225800000000001 2024-09-21T11:04:56.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:04:56.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.649784+0000", 2024-09-21T11:04:56.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7680000000000003e-06 2024-09-21T11:04:56.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.685 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.685 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.636744+0000", 2024-09-21T11:04:56.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98696035900000001 2024-09-21T11:04:56.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.685 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:04:56.685 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.636871+0000", 2024-09-21T11:04:56.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012668 2024-09-21T11:04:56.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.685 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6", 2024-09-21T11:04:56.686 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.637029+0000", 2024-09-21T11:04:56.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015828899999999999 2024-09-21T11:04:56.686 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.686 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.687 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.687 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.640258+0000", 2024-09-21T11:04:56.687 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003229216 2024-09-21T11:04:56.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.687 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.687 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.640287+0000", 2024-09-21T11:04:56.687 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8941999999999998e-05 2024-09-21T11:04:56.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.687 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:04:56.687 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.640330+0000", 2024-09-21T11:04:56.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.295e-05 2024-09-21T11:04:56.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.688 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:04:56.688 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.640386+0000", 2024-09-21T11:04:56.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6097999999999997e-05 2024-09-21T11:04:56.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.688 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:04:56.688 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.640425+0000", 2024-09-21T11:04:56.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8421999999999999e-05 2024-09-21T11:04:56.688 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.688 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:04:56.689 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.689 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.689 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:29099 57.9 57:91a3677e:::benchmark_data_smithi070_360420_object29098:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1537)", 2024-09-21T11:04:56.689 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:58:32.658256+0000", 2024-09-21T11:04:56.689 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 383.16265141500003, 2024-09-21T11:04:56.689 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.983558767, 2024-09-21T11:04:56.689 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:04:56.689 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:04:56.689 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:04:56.689 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T11:04:56.689 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:04:56.689 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 29099 2024-09-21T11:04:56.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.690 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:04:56.690 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.690 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:04:56.690 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.658256+0000", 2024-09-21T11:04:56.690 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.690 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.690 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:04:56.690 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.658256+0000", 2024-09-21T11:04:56.690 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.691 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:04:56.691 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.658255+0000", 2024-09-21T11:04:56.691 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:04:56.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.691 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:04:56.691 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.659800+0000", 2024-09-21T11:04:56.691 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0015451600000000001 2024-09-21T11:04:56.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.691 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:04:56.691 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.659812+0000", 2024-09-21T11:04:56.691 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1970999999999999e-05 2024-09-21T11:04:56.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.692 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:04:56.692 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.659820+0000", 2024-09-21T11:04:56.692 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.259e-06 2024-09-21T11:04:56.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.692 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.692 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.660111+0000", 2024-09-21T11:04:56.692 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00029062400000000001 2024-09-21T11:04:56.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.693 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.693 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:04:56.693 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.660122+0000", 2024-09-21T11:04:56.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0842999999999999e-05 2024-09-21T11:04:56.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.693 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.693 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.693 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.638033+0000", 2024-09-21T11:04:56.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.97791153399999997 2024-09-21T11:04:56.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.693 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.693 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:04:56.693 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.638163+0000", 2024-09-21T11:04:56.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012933899999999999 2024-09-21T11:04:56.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.694 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.694 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6", 2024-09-21T11:04:56.694 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.638264+0000", 2024-09-21T11:04:56.694 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000101463 2024-09-21T11:04:56.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.694 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.694 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.694 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.640323+0000", 2024-09-21T11:04:56.694 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0020588540000000002 2024-09-21T11:04:56.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.695 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.695 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.695 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.640491+0000", 2024-09-21T11:04:56.695 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016825600000000001 2024-09-21T11:04:56.695 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.695 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.695 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:04:56.695 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.641727+0000", 2024-09-21T11:04:56.695 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0012361900000000001 2024-09-21T11:04:56.695 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.696 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.696 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:04:56.696 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.641779+0000", 2024-09-21T11:04:56.696 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1511000000000002e-05 2024-09-21T11:04:56.696 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.696 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.696 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:04:56.696 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.641815+0000", 2024-09-21T11:04:56.696 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6361e-05 2024-09-21T11:04:56.696 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.696 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:04:56.696 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.696 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.697 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:29128 57.9 57:9035b4cc:::benchmark_data_smithi070_360420_object29127:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1537)", 2024-09-21T11:04:56.697 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:58:32.681201+0000", 2024-09-21T11:04:56.697 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 383.13970683600002, 2024-09-21T11:04:56.697 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96200238999999999, 2024-09-21T11:04:56.697 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:04:56.697 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:04:56.697 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:04:56.697 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T11:04:56.697 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:04:56.697 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 29128 2024-09-21T11:04:56.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.697 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:04:56.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.698 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:04:56.698 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681201+0000", 2024-09-21T11:04:56.698 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.698 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:04:56.698 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681201+0000", 2024-09-21T11:04:56.698 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.698 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:04:56.699 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681199+0000", 2024-09-21T11:04:56.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T11:04:56.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.699 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:04:56.699 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681241+0000", 2024-09-21T11:04:56.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1578000000000001e-05 2024-09-21T11:04:56.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.699 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:04:56.699 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681254+0000", 2024-09-21T11:04:56.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2886e-05 2024-09-21T11:04:56.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:04:56.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681261+0000", 2024-09-21T11:04:56.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1930000000000004e-06 2024-09-21T11:04:56.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681544+0000", 2024-09-21T11:04:56.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028305000000000002 2024-09-21T11:04:56.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:04:56.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681556+0000", 2024-09-21T11:04:56.701 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2459e-05 2024-09-21T11:04:56.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.701 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.701 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.639152+0000", 2024-09-21T11:04:56.701 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.95759554000000002 2024-09-21T11:04:56.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.701 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:04:56.701 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.639251+0000", 2024-09-21T11:04:56.701 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9067999999999994e-05 2024-09-21T11:04:56.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.701 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6", 2024-09-21T11:04:56.702 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.639367+0000", 2024-09-21T11:04:56.702 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011600899999999999 2024-09-21T11:04:56.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.702 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.702 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:04:56.702 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.641824+0000", 2024-09-21T11:04:56.702 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0024572959999999999 2024-09-21T11:04:56.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.702 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.702 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.702 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.643074+0000", 2024-09-21T11:04:56.702 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0012495399999999999 2024-09-21T11:04:56.703 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.703 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.703 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.703 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.643130+0000", 2024-09-21T11:04:56.703 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6465999999999998e-05 2024-09-21T11:04:56.703 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.703 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.703 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:04:56.703 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.643173+0000", 2024-09-21T11:04:56.703 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2478000000000002e-05 2024-09-21T11:04:56.703 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.703 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.703 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:04:56.703 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.643203+0000", 2024-09-21T11:04:56.704 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0586999999999997e-05 2024-09-21T11:04:56.704 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.704 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:04:56.704 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.704 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.704 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.704 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:29129 57.9 57:9186962e:::benchmark_data_smithi070_360420_object29128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1537)", 2024-09-21T11:04:56.704 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:58:32.681277+0000", 2024-09-21T11:04:56.704 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 383.13963079799998, 2024-09-21T11:04:56.704 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96646752700000005, 2024-09-21T11:04:56.704 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:04:56.704 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:04:56.704 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:04:56.705 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T11:04:56.705 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:04:56.705 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 29129 2024-09-21T11:04:56.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.705 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:04:56.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.705 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:04:56.705 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681277+0000", 2024-09-21T11:04:56.705 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.705 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:04:56.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681277+0000", 2024-09-21T11:04:56.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:04:56.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681277+0000", 2024-09-21T11:04:56.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T11:04:56.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:04:56.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681304+0000", 2024-09-21T11:04:56.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7623999999999998e-05 2024-09-21T11:04:56.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.707 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:04:56.707 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681312+0000", 2024-09-21T11:04:56.707 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3769999999999998e-06 2024-09-21T11:04:56.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.707 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:04:56.707 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681317+0000", 2024-09-21T11:04:56.707 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3299999999999998e-06 2024-09-21T11:04:56.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.707 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.708 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681583+0000", 2024-09-21T11:04:56.708 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00026564200000000002 2024-09-21T11:04:56.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.708 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:04:56.708 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.681589+0000", 2024-09-21T11:04:56.708 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7379999999999997e-06 2024-09-21T11:04:56.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.708 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.708 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.640530+0000", 2024-09-21T11:04:56.708 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.95894080400000004 2024-09-21T11:04:56.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.709 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:04:56.709 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.640729+0000", 2024-09-21T11:04:56.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019901000000000001 2024-09-21T11:04:56.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.709 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6", 2024-09-21T11:04:56.709 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.640844+0000", 2024-09-21T11:04:56.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011447600000000001 2024-09-21T11:04:56.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.709 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:04:56.709 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.643080+0000", 2024-09-21T11:04:56.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022364339999999998 2024-09-21T11:04:56.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.710 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.710 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.645208+0000", 2024-09-21T11:04:56.710 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021281690000000001 2024-09-21T11:04:56.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.710 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.710 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.647662+0000", 2024-09-21T11:04:56.710 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0024534370000000002 2024-09-21T11:04:56.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.710 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:04:56.710 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.647719+0000", 2024-09-21T11:04:56.711 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7111000000000003e-05 2024-09-21T11:04:56.711 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.711 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.711 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:04:56.711 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.647745+0000", 2024-09-21T11:04:56.711 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5914e-05 2024-09-21T11:04:56.711 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.711 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:04:56.711 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.711 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.711 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.711 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:29144 57.9 57:919fd5ad:::benchmark_data_smithi070_360420_object29143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1537)", 2024-09-21T11:04:56.711 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:58:32.709679+0000", 2024-09-21T11:04:56.711 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 383.11122848399998, 2024-09-21T11:04:56.711 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.94539665399999995, 2024-09-21T11:04:56.712 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:04:56.712 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:04:56.712 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:04:56.712 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T11:04:56.712 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:04:56.712 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 29144 2024-09-21T11:04:56.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.712 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:04:56.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:04:56.712 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.709679+0000", 2024-09-21T11:04:56.712 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:04:56.713 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.709679+0000", 2024-09-21T11:04:56.713 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.713 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:04:56.713 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.709677+0000", 2024-09-21T11:04:56.713 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T11:04:56.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.713 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:04:56.713 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.709721+0000", 2024-09-21T11:04:56.713 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3291000000000003e-05 2024-09-21T11:04:56.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.713 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:04:56.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.709738+0000", 2024-09-21T11:04:56.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6813999999999999e-05 2024-09-21T11:04:56.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:04:56.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.709745+0000", 2024-09-21T11:04:56.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9289999999999998e-06 2024-09-21T11:04:56.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.709797+0000", 2024-09-21T11:04:56.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1066999999999997e-05 2024-09-21T11:04:56.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:04:56.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.709806+0000", 2024-09-21T11:04:56.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5969999999999998e-06 2024-09-21T11:04:56.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.641853+0000", 2024-09-21T11:04:56.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.93204697000000003 2024-09-21T11:04:56.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:04:56.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.641978+0000", 2024-09-21T11:04:56.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012522400000000001 2024-09-21T11:04:56.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.716 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6", 2024-09-21T11:04:56.716 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.642117+0000", 2024-09-21T11:04:56.717 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013835400000000001 2024-09-21T11:04:56.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.717 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.717 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:04:56.717 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.645179+0000", 2024-09-21T11:04:56.718 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0030618450000000001 2024-09-21T11:04:56.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.718 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.718 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.645234+0000", 2024-09-21T11:04:56.718 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4979999999999999e-05 2024-09-21T11:04:56.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.718 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.718 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.654997+0000", 2024-09-21T11:04:56.718 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.009763384 2024-09-21T11:04:56.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.719 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:04:56.719 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.655038+0000", 2024-09-21T11:04:56.719 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1307999999999998e-05 2024-09-21T11:04:56.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.719 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.719 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:04:56.719 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.655076+0000", 2024-09-21T11:04:56.719 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7851000000000001e-05 2024-09-21T11:04:56.719 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.719 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:04:56.719 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.719 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.719 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:29201 57.9 57:93a056f6:::benchmark_data_smithi070_360420_object29200:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1537)", 2024-09-21T11:04:56.720 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:58:32.747625+0000", 2024-09-21T11:04:56.720 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 383.073283113, 2024-09-21T11:04:56.720 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.90758979299999998, 2024-09-21T11:04:56.720 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:04:56.720 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:04:56.720 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:04:56.720 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T11:04:56.720 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:04:56.720 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 29201 2024-09-21T11:04:56.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.720 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:04:56.720 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.721 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:04:56.721 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.747625+0000", 2024-09-21T11:04:56.721 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.721 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.721 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.721 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:04:56.721 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.747625+0000", 2024-09-21T11:04:56.721 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.721 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.721 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.721 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:04:56.721 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.747622+0000", 2024-09-21T11:04:56.721 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 2024-09-21T11:04:56.722 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.722 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.722 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:04:56.722 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.747673+0000", 2024-09-21T11:04:56.722 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0284000000000002e-05 2024-09-21T11:04:56.722 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.722 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.722 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:04:56.722 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.747690+0000", 2024-09-21T11:04:56.722 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7677999999999999e-05 2024-09-21T11:04:56.722 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.722 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.722 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:04:56.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.747704+0000", 2024-09-21T11:04:56.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.364e-05 2024-09-21T11:04:56.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.723 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.747763+0000", 2024-09-21T11:04:56.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8634999999999997e-05 2024-09-21T11:04:56.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.723 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:04:56.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.747772+0000", 2024-09-21T11:04:56.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1619999999999997e-06 2024-09-21T11:04:56.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.643228+0000", 2024-09-21T11:04:56.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.89545633199999997 2024-09-21T11:04:56.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:04:56.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.643345+0000", 2024-09-21T11:04:56.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000116978 2024-09-21T11:04:56.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6", 2024-09-21T11:04:56.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.643469+0000", 2024-09-21T11:04:56.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012339600000000001 2024-09-21T11:04:56.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.725 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.725 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.645259+0000", 2024-09-21T11:04:56.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0017906899999999999 2024-09-21T11:04:56.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.725 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:04:56.725 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.652531+0000", 2024-09-21T11:04:56.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0072719259999999997 2024-09-21T11:04:56.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.725 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.655148+0000", 2024-09-21T11:04:56.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0026166129999999998 2024-09-21T11:04:56.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.726 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:04:56.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.655186+0000", 2024-09-21T11:04:56.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7758e-05 2024-09-21T11:04:56.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.726 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:04:56.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.655215+0000", 2024-09-21T11:04:56.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9013000000000002e-05 2024-09-21T11:04:56.727 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.727 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:04:56.727 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.727 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.727 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDPGPush(57.d 1552/1548 [PushOp(57:be8b7064:::benchmark_data_smithi070_360420_object814:head, version: 1528'44, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(57:be8b7064:::benchmark_data_smithi070_360420_object814:head@1528'44, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T11:04:56.727 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:59:41.629941+0000", 2024-09-21T11:04:56.727 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 314.19096723000001, 2024-09-21T11:04:56.727 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.93329599399999996, 2024-09-21T11:04:56.727 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:04:56.727 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T11:04:56.727 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:04:56.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.727 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:04:56.728 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:41.629941+0000", 2024-09-21T11:04:56.728 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.728 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.728 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:04:56.728 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:41.629941+0000", 2024-09-21T11:04:56.728 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.728 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.728 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:04:56.728 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:41.629940+0000", 2024-09-21T11:04:56.728 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:04:56.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.729 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.729 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:04:56.729 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:41.629997+0000", 2024-09-21T11:04:56.729 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6929000000000003e-05 2024-09-21T11:04:56.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.729 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.729 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:04:56.729 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:41.630013+0000", 2024-09-21T11:04:56.729 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6393e-05 2024-09-21T11:04:56.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.729 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.729 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:04:56.729 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:41.630029+0000", 2024-09-21T11:04:56.729 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5739e-05 2024-09-21T11:04:56.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:42.542627+0000", 2024-09-21T11:04:56.730 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91259803399999995 2024-09-21T11:04:56.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:04:56.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:42.543204+0000", 2024-09-21T11:04:56.730 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00057712599999999996 2024-09-21T11:04:56.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:04:56.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:42.563237+0000", 2024-09-21T11:04:56.731 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020032702999999999 2024-09-21T11:04:56.731 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.731 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:04:56.731 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.731 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:67594 57.b 57:dd9f77c8:::benchmark_data_smithi070_360420_object245:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1553)", 2024-09-21T11:04:56.731 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:59:48.637537+0000", 2024-09-21T11:04:56.731 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 307.18337131499999, 2024-09-21T11:04:56.731 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.90511425400000001, 2024-09-21T11:04:56.731 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:04:56.731 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:04:56.731 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:04:56.732 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T11:04:56.732 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:04:56.732 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 67594 2024-09-21T11:04:56.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.732 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:04:56.732 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.732 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:04:56.732 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:48.637537+0000", 2024-09-21T11:04:56.732 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.732 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.732 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:04:56.732 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:48.637537+0000", 2024-09-21T11:04:56.732 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.733 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:04:56.733 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:48.637536+0000", 2024-09-21T11:04:56.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:04:56.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.733 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:04:56.733 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:48.637540+0000", 2024-09-21T11:04:56.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4859999999999999e-06 2024-09-21T11:04:56.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.733 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:04:56.733 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:48.637553+0000", 2024-09-21T11:04:56.734 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2412000000000001e-05 2024-09-21T11:04:56.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.734 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:04:56.734 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:48.637559+0000", 2024-09-21T11:04:56.734 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6390000000000003e-06 2024-09-21T11:04:56.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.734 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.734 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:48.693775+0000", 2024-09-21T11:04:56.734 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.056215862999999998 2024-09-21T11:04:56.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.735 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for missing object", 2024-09-21T11:04:56.735 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:48.697251+0000", 2024-09-21T11:04:56.735 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0034756869999999999 2024-09-21T11:04:56.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.735 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.735 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.735 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:48.752183+0000", 2024-09-21T11:04:56.735 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.054932188999999999 2024-09-21T11:04:56.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.735 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.735 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T11:04:56.735 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:48.755616+0000", 2024-09-21T11:04:56.735 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0034329540000000002 2024-09-21T11:04:56.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.736 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.736 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.736 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:49.493682+0000", 2024-09-21T11:04:56.736 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.73806554000000002 2024-09-21T11:04:56.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.736 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.736 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:04:56.736 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:49.496599+0000", 2024-09-21T11:04:56.736 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029172040000000001 2024-09-21T11:04:56.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.736 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.736 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,7", 2024-09-21T11:04:56.736 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:49.503175+0000", 2024-09-21T11:04:56.737 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0065759019999999998 2024-09-21T11:04:56.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.737 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.737 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:04:56.737 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:49.531686+0000", 2024-09-21T11:04:56.737 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.028511103999999999 2024-09-21T11:04:56.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.737 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.737 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.737 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:49.533114+0000", 2024-09-21T11:04:56.737 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001428106 2024-09-21T11:04:56.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.737 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.738 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.738 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:49.534079+0000", 2024-09-21T11:04:56.738 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00096502800000000004 2024-09-21T11:04:56.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.738 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.738 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:04:56.738 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:49.535738+0000", 2024-09-21T11:04:56.738 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001659141 2024-09-21T11:04:56.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.738 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.738 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:04:56.738 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:49.542651+0000", 2024-09-21T11:04:56.739 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0069127370000000004 2024-09-21T11:04:56.739 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.739 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:04:56.739 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.739 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.739 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.739 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:68089 57.b 57:dd0923f6:::benchmark_data_smithi070_360420_object740:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1553)", 2024-09-21T11:04:56.739 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:59:50.375371+0000", 2024-09-21T11:04:56.739 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 305.445536882, 2024-09-21T11:04:56.739 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91047509900000001, 2024-09-21T11:04:56.739 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:04:56.739 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:04:56.739 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:04:56.739 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T11:04:56.740 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:04:56.740 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 68089 2024-09-21T11:04:56.740 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.740 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:04:56.740 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.740 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:04:56.740 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:50.375371+0000", 2024-09-21T11:04:56.740 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.740 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.740 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.740 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:04:56.740 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:50.375371+0000", 2024-09-21T11:04:56.740 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.740 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.741 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.741 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:04:56.741 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:50.375370+0000", 2024-09-21T11:04:56.741 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:04:56.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.741 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.741 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:04:56.741 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:50.375376+0000", 2024-09-21T11:04:56.741 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7209999999999999e-06 2024-09-21T11:04:56.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.741 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.741 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:04:56.741 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:50.375388+0000", 2024-09-21T11:04:56.742 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2444e-05 2024-09-21T11:04:56.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.742 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.742 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:04:56.742 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:50.375396+0000", 2024-09-21T11:04:56.742 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9310000000000008e-06 2024-09-21T11:04:56.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.742 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.742 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.742 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:50.390075+0000", 2024-09-21T11:04:56.742 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.014678549000000001 2024-09-21T11:04:56.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.742 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.743 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for missing object", 2024-09-21T11:04:56.743 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:50.393386+0000", 2024-09-21T11:04:56.743 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0033116059999999999 2024-09-21T11:04:56.743 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.743 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.743 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.743 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:50.438012+0000", 2024-09-21T11:04:56.743 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.044625774999999999 2024-09-21T11:04:56.743 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.743 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.743 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T11:04:56.743 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:50.441468+0000", 2024-09-21T11:04:56.743 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003456228 2024-09-21T11:04:56.743 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.744 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.744 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.744 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:51.248859+0000", 2024-09-21T11:04:56.744 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.80739122200000002 2024-09-21T11:04:56.744 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.744 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.744 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:04:56.744 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:51.251660+0000", 2024-09-21T11:04:56.744 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002801048 2024-09-21T11:04:56.744 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.744 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.744 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,7", 2024-09-21T11:04:56.744 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:51.257154+0000", 2024-09-21T11:04:56.745 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0054934340000000002 2024-09-21T11:04:56.745 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.745 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.745 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:04:56.745 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:51.272596+0000", 2024-09-21T11:04:56.745 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.015441730000000001 2024-09-21T11:04:56.745 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.745 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.745 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.745 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:51.277323+0000", 2024-09-21T11:04:56.745 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0047272360000000001 2024-09-21T11:04:56.745 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.745 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.746 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.746 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:51.278266+0000", 2024-09-21T11:04:56.746 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00094360500000000003 2024-09-21T11:04:56.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.746 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.746 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:04:56.746 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:51.279765+0000", 2024-09-21T11:04:56.746 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001498306 2024-09-21T11:04:56.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.746 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.746 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:04:56.746 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:59:51.285846+0000", 2024-09-21T11:04:56.746 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.006081365 2024-09-21T11:04:56.747 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.747 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:04:56.747 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.747 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.747 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:219130 58.2 58:4c3b0c88:::benchmark_data_smithi070_396992_object73171:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1603)", 2024-09-21T11:04:56.747 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:33.496261+0000", 2024-09-21T11:04:56.747 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 22.324646856000001, 2024-09-21T11:04:56.747 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1229929470000002, 2024-09-21T11:04:56.747 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:04:56.747 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T11:04:56.747 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:04:56.747 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:04:56.748 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:04:56.748 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 219130 2024-09-21T11:04:56.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.748 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:04:56.748 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.748 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:04:56.748 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.496261+0000", 2024-09-21T11:04:56.748 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.748 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.748 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:04:56.748 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.496261+0000", 2024-09-21T11:04:56.748 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.748 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.749 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:04:56.749 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.496261+0000", 2024-09-21T11:04:56.749 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T11:04:56.749 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.749 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.749 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:04:56.749 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.496263+0000", 2024-09-21T11:04:56.749 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8779999999999998e-06 2024-09-21T11:04:56.749 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.749 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.749 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:04:56.749 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.496270+0000", 2024-09-21T11:04:56.749 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8199999999999999e-06 2024-09-21T11:04:56.749 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.750 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.750 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:04:56.750 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.496275+0000", 2024-09-21T11:04:56.750 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1089999999999997e-06 2024-09-21T11:04:56.750 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.750 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.750 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.750 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.496317+0000", 2024-09-21T11:04:56.750 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1937000000000002e-05 2024-09-21T11:04:56.750 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.750 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.750 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:04:56.750 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.496423+0000", 2024-09-21T11:04:56.750 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010550199999999999 2024-09-21T11:04:56.751 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.751 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.751 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,7", 2024-09-21T11:04:56.751 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.496521+0000", 2024-09-21T11:04:56.751 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8523999999999994e-05 2024-09-21T11:04:56.751 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.751 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.751 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.751 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.501813+0000", 2024-09-21T11:04:56.751 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.005291352 2024-09-21T11:04:56.751 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.751 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.751 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:04:56.751 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.501819+0000", 2024-09-21T11:04:56.752 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1449999999999996e-06 2024-09-21T11:04:56.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.752 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.752 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.752 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.543949+0000", 2024-09-21T11:04:56.752 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.042130244999999997 2024-09-21T11:04:56.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.752 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.752 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:04:56.752 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.543959+0000", 2024-09-21T11:04:56.752 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6250000000000002e-06 2024-09-21T11:04:56.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.752 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.753 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.753 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.526163+0000", 2024-09-21T11:04:56.753 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98220441000000003 2024-09-21T11:04:56.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.753 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.753 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:04:56.753 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.527342+0000", 2024-09-21T11:04:56.753 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0011790520000000001 2024-09-21T11:04:56.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.753 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.753 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.753 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.619188+0000", 2024-09-21T11:04:56.753 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.091845706 2024-09-21T11:04:56.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.753 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.754 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:04:56.754 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.619254+0000", 2024-09-21T11:04:56.754 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6134000000000001e-05 2024-09-21T11:04:56.754 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.754 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:04:56.754 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.754 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.754 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:219131 58.2 58:44c6283b:::benchmark_data_smithi070_396992_object73172:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1603)", 2024-09-21T11:04:56.754 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:33.497488+0000", 2024-09-21T11:04:56.754 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 22.323420178999999, 2024-09-21T11:04:56.754 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1220704669999999, 2024-09-21T11:04:56.754 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:04:56.755 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T11:04:56.755 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:04:56.755 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:04:56.755 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:04:56.755 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 219131 2024-09-21T11:04:56.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.755 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:04:56.755 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.755 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:04:56.755 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.497488+0000", 2024-09-21T11:04:56.755 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.755 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.755 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:04:56.756 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.497488+0000", 2024-09-21T11:04:56.756 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.756 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.756 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:04:56.756 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.497487+0000", 2024-09-21T11:04:56.756 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:04:56.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.756 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.756 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:04:56.756 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.497492+0000", 2024-09-21T11:04:56.756 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5820000000000001e-06 2024-09-21T11:04:56.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.757 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.757 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:04:56.757 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.497504+0000", 2024-09-21T11:04:56.757 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1603000000000001e-05 2024-09-21T11:04:56.757 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.757 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.757 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:04:56.757 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.497511+0000", 2024-09-21T11:04:56.757 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.148e-06 2024-09-21T11:04:56.757 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.757 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.757 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.757 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.499179+0000", 2024-09-21T11:04:56.757 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001668037 2024-09-21T11:04:56.757 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.757 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.758 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:04:56.758 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.499275+0000", 2024-09-21T11:04:56.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5736000000000006e-05 2024-09-21T11:04:56.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.758 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,7", 2024-09-21T11:04:56.758 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.499376+0000", 2024-09-21T11:04:56.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010124399999999999 2024-09-21T11:04:56.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.758 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:04:56.758 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.501823+0000", 2024-09-21T11:04:56.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0024471929999999999 2024-09-21T11:04:56.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.501937+0000", 2024-09-21T11:04:56.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000113743 2024-09-21T11:04:56.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.543986+0000", 2024-09-21T11:04:56.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.042049572 2024-09-21T11:04:56.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:04:56.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.543995+0000", 2024-09-21T11:04:56.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3790000000000006e-06 2024-09-21T11:04:56.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.760 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.760 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.534350+0000", 2024-09-21T11:04:56.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99035547199999996 2024-09-21T11:04:56.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.760 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:04:56.760 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.534361+0000", 2024-09-21T11:04:56.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0737000000000001e-05 2024-09-21T11:04:56.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.760 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.760 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.619518+0000", 2024-09-21T11:04:56.761 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0851569379999999 2024-09-21T11:04:56.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.761 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:04:56.761 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.619558+0000", 2024-09-21T11:04:56.761 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0209000000000001e-05 2024-09-21T11:04:56.761 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.761 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:04:56.761 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.761 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:219137 58.6 58:688d16c4:::benchmark_data_smithi070_396992_object73178:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1603)", 2024-09-21T11:04:56.761 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:33.505099+0000", 2024-09-21T11:04:56.762 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 22.315808603000001, 2024-09-21T11:04:56.762 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1234349250000002, 2024-09-21T11:04:56.762 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:04:56.762 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T11:04:56.762 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:04:56.762 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:04:56.762 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:04:56.762 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 219137 2024-09-21T11:04:56.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.762 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:04:56.762 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.762 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:04:56.762 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.505099+0000", 2024-09-21T11:04:56.763 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.763 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.763 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.763 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:04:56.763 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.505099+0000", 2024-09-21T11:04:56.763 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.763 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.763 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.763 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:04:56.763 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.505098+0000", 2024-09-21T11:04:56.763 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:04:56.763 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.763 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.764 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:04:56.764 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.505107+0000", 2024-09-21T11:04:56.764 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3790000000000006e-06 2024-09-21T11:04:56.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.764 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.764 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:04:56.764 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.505118+0000", 2024-09-21T11:04:56.764 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1562999999999999e-05 2024-09-21T11:04:56.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.764 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.764 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:04:56.764 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.505125+0000", 2024-09-21T11:04:56.764 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6950000000000003e-06 2024-09-21T11:04:56.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.765 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.765 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.765 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.505170+0000", 2024-09-21T11:04:56.765 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5532999999999999e-05 2024-09-21T11:04:56.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.765 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.765 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:04:56.765 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.505280+0000", 2024-09-21T11:04:56.765 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010936499999999999 2024-09-21T11:04:56.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.765 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.765 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,7", 2024-09-21T11:04:56.766 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.505383+0000", 2024-09-21T11:04:56.766 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000103391 2024-09-21T11:04:56.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.766 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:04:56.766 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.507870+0000", 2024-09-21T11:04:56.766 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002486556 2024-09-21T11:04:56.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.766 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.766 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.509821+0000", 2024-09-21T11:04:56.766 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0019513009999999999 2024-09-21T11:04:56.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.767 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.767 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.539798+0000", 2024-09-21T11:04:56.767 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.029977200999999998 2024-09-21T11:04:56.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.767 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:04:56.767 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.539809+0000", 2024-09-21T11:04:56.767 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1228e-05 2024-09-21T11:04:56.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.767 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.767 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.519794+0000", 2024-09-21T11:04:56.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.97998448500000002 2024-09-21T11:04:56.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.768 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:04:56.768 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.519808+0000", 2024-09-21T11:04:56.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3703000000000001e-05 2024-09-21T11:04:56.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.768 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.768 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.628397+0000", 2024-09-21T11:04:56.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.108589174 2024-09-21T11:04:56.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.769 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:04:56.769 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.628534+0000", 2024-09-21T11:04:56.769 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013744000000000001 2024-09-21T11:04:56.769 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.769 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:04:56.769 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.769 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.769 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:219161 58.6 58:68ba07c2:::benchmark_data_smithi070_396992_object73202:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1604)", 2024-09-21T11:04:56.769 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:33.552233+0000", 2024-09-21T11:04:56.769 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 22.268674444999998, 2024-09-21T11:04:56.769 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0861848489999999, 2024-09-21T11:04:56.769 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:04:56.770 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:04:56.770 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:04:56.770 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:04:56.770 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:04:56.770 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 219161 2024-09-21T11:04:56.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.770 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:04:56.770 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.770 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:04:56.770 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552233+0000", 2024-09-21T11:04:56.770 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.770 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.771 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:04:56.771 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552233+0000", 2024-09-21T11:04:56.771 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.771 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.771 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:04:56.771 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552232+0000", 2024-09-21T11:04:56.771 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:04:56.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.771 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.771 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:04:56.771 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552238+0000", 2024-09-21T11:04:56.771 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6729999999999997e-06 2024-09-21T11:04:56.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.772 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.772 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:04:56.772 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552249+0000", 2024-09-21T11:04:56.772 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0577999999999999e-05 2024-09-21T11:04:56.772 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.772 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.772 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:04:56.772 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552255+0000", 2024-09-21T11:04:56.772 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8000000000000001e-06 2024-09-21T11:04:56.772 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.772 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.772 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.772 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552316+0000", 2024-09-21T11:04:56.773 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0603000000000002e-05 2024-09-21T11:04:56.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.773 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:04:56.773 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552327+0000", 2024-09-21T11:04:56.773 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0618999999999999e-05 2024-09-21T11:04:56.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.773 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.773 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.521117+0000", 2024-09-21T11:04:56.773 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96879046300000005 2024-09-21T11:04:56.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.774 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:04:56.774 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.521126+0000", 2024-09-21T11:04:56.774 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1030000000000008e-06 2024-09-21T11:04:56.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.774 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.774 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.774 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.628825+0000", 2024-09-21T11:04:56.774 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.107698944 2024-09-21T11:04:56.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.774 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.774 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:04:56.774 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.628951+0000", 2024-09-21T11:04:56.774 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012609200000000001 2024-09-21T11:04:56.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.775 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.775 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6,7", 2024-09-21T11:04:56.775 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.629444+0000", 2024-09-21T11:04:56.775 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00049303699999999999 2024-09-21T11:04:56.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.775 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.775 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.775 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.635982+0000", 2024-09-21T11:04:56.775 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0065374029999999998 2024-09-21T11:04:56.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.775 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.775 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.775 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.638172+0000", 2024-09-21T11:04:56.776 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002190642 2024-09-21T11:04:56.776 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.776 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.776 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:04:56.776 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.638181+0000", 2024-09-21T11:04:56.776 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9190000000000005e-06 2024-09-21T11:04:56.776 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.776 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.776 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.776 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.638259+0000", 2024-09-21T11:04:56.776 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7583000000000006e-05 2024-09-21T11:04:56.776 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.776 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.776 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:04:56.777 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.638308+0000", 2024-09-21T11:04:56.777 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9605999999999999e-05 2024-09-21T11:04:56.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.777 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:04:56.777 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.638418+0000", 2024-09-21T11:04:56.777 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000109844 2024-09-21T11:04:56.777 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.777 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:04:56.777 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.778 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:219162 58.7 58:e30a89ab:::benchmark_data_smithi070_396992_object73203:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1604)", 2024-09-21T11:04:56.778 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:33.552486+0000", 2024-09-21T11:04:56.778 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 22.268421417999999, 2024-09-21T11:04:56.778 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0599404050000001, 2024-09-21T11:04:56.778 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:04:56.778 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:04:56.778 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:04:56.778 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:04:56.778 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:04:56.778 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 219162 2024-09-21T11:04:56.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.778 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:04:56.778 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.779 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:04:56.779 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552486+0000", 2024-09-21T11:04:56.779 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.779 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.779 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.779 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:04:56.779 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552486+0000", 2024-09-21T11:04:56.779 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.779 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.779 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.779 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:04:56.779 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552486+0000", 2024-09-21T11:04:56.779 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:04:56.779 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.780 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:04:56.780 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552491+0000", 2024-09-21T11:04:56.780 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0810000000000001e-06 2024-09-21T11:04:56.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.780 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:04:56.780 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552503+0000", 2024-09-21T11:04:56.780 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2281e-05 2024-09-21T11:04:56.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.780 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:04:56.780 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552513+0000", 2024-09-21T11:04:56.780 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6779999999999995e-06 2024-09-21T11:04:56.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.781 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.781 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552552+0000", 2024-09-21T11:04:56.781 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9700000000000003e-05 2024-09-21T11:04:56.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.781 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:04:56.781 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552557+0000", 2024-09-21T11:04:56.781 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.152e-06 2024-09-21T11:04:56.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.781 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.781 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.608525+0000", 2024-09-21T11:04:56.781 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0559673469999999 2024-09-21T11:04:56.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.782 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:04:56.782 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.608645+0000", 2024-09-21T11:04:56.782 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011995 2024-09-21T11:04:56.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.782 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6,7", 2024-09-21T11:04:56.782 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.608993+0000", 2024-09-21T11:04:56.782 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034864199999999997 2024-09-21T11:04:56.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.782 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:04:56.782 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.611354+0000", 2024-09-21T11:04:56.782 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0023607699999999999 2024-09-21T11:04:56.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.783 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.783 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.783 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.611659+0000", 2024-09-21T11:04:56.783 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00030510800000000001 2024-09-21T11:04:56.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.783 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.783 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.783 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.612102+0000", 2024-09-21T11:04:56.783 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000442433 2024-09-21T11:04:56.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.783 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.783 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.784 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.612291+0000", 2024-09-21T11:04:56.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00018962300000000001 2024-09-21T11:04:56.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.784 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:04:56.784 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.612330+0000", 2024-09-21T11:04:56.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8424999999999999e-05 2024-09-21T11:04:56.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.784 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:04:56.784 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.612427+0000", 2024-09-21T11:04:56.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7114999999999994e-05 2024-09-21T11:04:56.784 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.784 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:04:56.785 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.785 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.785 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.785 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:219166 58.2 58:4160bb5b:::benchmark_data_smithi070_396992_object73207:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1604)", 2024-09-21T11:04:56.785 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:33.558581+0000", 2024-09-21T11:04:56.785 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 22.262326601000002, 2024-09-21T11:04:56.785 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0708711590000002, 2024-09-21T11:04:56.785 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:04:56.785 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:04:56.785 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:04:56.785 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:04:56.785 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:04:56.785 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 219166 2024-09-21T11:04:56.786 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.786 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:04:56.786 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.786 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:04:56.786 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.558581+0000", 2024-09-21T11:04:56.786 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.786 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.786 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.786 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:04:56.786 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.558581+0000", 2024-09-21T11:04:56.786 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.786 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.786 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.786 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:04:56.787 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.558580+0000", 2024-09-21T11:04:56.787 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:04:56.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.787 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.787 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:04:56.787 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.558586+0000", 2024-09-21T11:04:56.787 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.711e-06 2024-09-21T11:04:56.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.787 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.787 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:04:56.787 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.558597+0000", 2024-09-21T11:04:56.787 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1251e-05 2024-09-21T11:04:56.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.788 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:04:56.788 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.558603+0000", 2024-09-21T11:04:56.788 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6289999999999998e-06 2024-09-21T11:04:56.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.788 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.788 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.558655+0000", 2024-09-21T11:04:56.788 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2666e-05 2024-09-21T11:04:56.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.788 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:04:56.788 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.558666+0000", 2024-09-21T11:04:56.788 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0740999999999999e-05 2024-09-21T11:04:56.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.789 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.789 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.789 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.534381+0000", 2024-09-21T11:04:56.789 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.97571475900000004 2024-09-21T11:04:56.789 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.789 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.789 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:04:56.789 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.534387+0000", 2024-09-21T11:04:56.789 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.263e-06 2024-09-21T11:04:56.789 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.789 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.789 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.789 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.619594+0000", 2024-09-21T11:04:56.790 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.085206838 2024-09-21T11:04:56.790 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.790 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.790 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:04:56.790 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.619740+0000", 2024-09-21T11:04:56.790 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000145582 2024-09-21T11:04:56.790 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.790 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.790 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,3,7", 2024-09-21T11:04:56.790 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.620170+0000", 2024-09-21T11:04:56.790 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00043068300000000001 2024-09-21T11:04:56.790 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.790 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.790 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.791 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.626730+0000", 2024-09-21T11:04:56.791 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0065597329999999999 2024-09-21T11:04:56.791 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.791 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.791 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:04:56.791 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.626739+0000", 2024-09-21T11:04:56.791 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0699999999999996e-06 2024-09-21T11:04:56.791 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.791 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.791 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.791 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.626768+0000", 2024-09-21T11:04:56.791 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8756999999999999e-05 2024-09-21T11:04:56.791 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.791 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.792 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.792 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.629320+0000", 2024-09-21T11:04:56.792 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002551967 2024-09-21T11:04:56.792 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.792 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.792 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:04:56.792 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.629354+0000", 2024-09-21T11:04:56.792 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3689999999999998e-05 2024-09-21T11:04:56.792 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.792 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.792 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:04:56.792 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.629452+0000", 2024-09-21T11:04:56.792 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8882000000000006e-05 2024-09-21T11:04:56.792 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.792 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:04:56.792 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.793 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.793 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.793 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:219171 58.2 58:4bc6147e:::benchmark_data_smithi070_396992_object73212:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1604)", 2024-09-21T11:04:56.793 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:33.567026+0000", 2024-09-21T11:04:56.793 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 22.253882374, 2024-09-21T11:04:56.793 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0626493610000001, 2024-09-21T11:04:56.793 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:04:56.793 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:04:56.793 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:04:56.793 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:04:56.793 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:04:56.793 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 219171 2024-09-21T11:04:56.793 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.793 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:04:56.793 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.794 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:04:56.794 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.567026+0000", 2024-09-21T11:04:56.794 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.794 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.794 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:04:56.794 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.567026+0000", 2024-09-21T11:04:56.794 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.794 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.794 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:04:56.794 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.567024+0000", 2024-09-21T11:04:56.794 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:04:56.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.795 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:04:56.795 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.567030+0000", 2024-09-21T11:04:56.795 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7520000000000002e-06 2024-09-21T11:04:56.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.795 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:04:56.795 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.567043+0000", 2024-09-21T11:04:56.795 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2942000000000001e-05 2024-09-21T11:04:56.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.795 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:04:56.796 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.567052+0000", 2024-09-21T11:04:56.796 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8780000000000004e-06 2024-09-21T11:04:56.796 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.796 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.796 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.796 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.567108+0000", 2024-09-21T11:04:56.796 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5476999999999998e-05 2024-09-21T11:04:56.796 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.796 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.796 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:04:56.796 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.567117+0000", 2024-09-21T11:04:56.796 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9229999999999997e-06 2024-09-21T11:04:56.796 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.796 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.796 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.796 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.534405+0000", 2024-09-21T11:04:56.797 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96728771599999996 2024-09-21T11:04:56.797 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.797 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.797 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:04:56.797 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.534411+0000", 2024-09-21T11:04:56.797 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1199999999999999e-06 2024-09-21T11:04:56.797 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.797 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.797 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.797 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.624395+0000", 2024-09-21T11:04:56.797 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0899832540000001 2024-09-21T11:04:56.797 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.797 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.797 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:04:56.798 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.624527+0000", 2024-09-21T11:04:56.798 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013255299999999999 2024-09-21T11:04:56.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.798 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.798 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,3,7", 2024-09-21T11:04:56.798 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.624672+0000", 2024-09-21T11:04:56.798 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014475999999999999 2024-09-21T11:04:56.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.798 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.798 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.798 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.626796+0000", 2024-09-21T11:04:56.798 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021244319999999999 2024-09-21T11:04:56.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.799 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.799 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.799 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.626824+0000", 2024-09-21T11:04:56.799 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7531000000000001e-05 2024-09-21T11:04:56.799 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.799 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.799 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:04:56.799 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.629012+0000", 2024-09-21T11:04:56.799 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021877450000000001 2024-09-21T11:04:56.799 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.799 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.799 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.799 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.629484+0000", 2024-09-21T11:04:56.799 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00047253900000000001 2024-09-21T11:04:56.799 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.799 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.800 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:04:56.800 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.629570+0000", 2024-09-21T11:04:56.800 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5729000000000005e-05 2024-09-21T11:04:56.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.800 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.800 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:04:56.800 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.629675+0000", 2024-09-21T11:04:56.800 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000105073 2024-09-21T11:04:56.800 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.800 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:04:56.800 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.800 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.800 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:219173 58.6 58:6a576e84:::benchmark_data_smithi070_396992_object73214:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1604)", 2024-09-21T11:04:56.800 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:33.572033+0000", 2024-09-21T11:04:56.801 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 22.248875134999999, 2024-09-21T11:04:56.801 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0691706920000001, 2024-09-21T11:04:56.801 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:04:56.801 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:04:56.801 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:04:56.801 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:04:56.801 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:04:56.801 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 219173 2024-09-21T11:04:56.801 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.801 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:04:56.801 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.801 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:04:56.801 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.572033+0000", 2024-09-21T11:04:56.801 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.801 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.802 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.802 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:04:56.802 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.572033+0000", 2024-09-21T11:04:56.802 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.802 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.802 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.802 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:04:56.802 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.572032+0000", 2024-09-21T11:04:56.802 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:04:56.802 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.802 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.802 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:04:56.802 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.572036+0000", 2024-09-21T11:04:56.802 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1899999999999997e-06 2024-09-21T11:04:56.803 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.803 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.803 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:04:56.803 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.572046+0000", 2024-09-21T11:04:56.803 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5170000000000005e-06 2024-09-21T11:04:56.803 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.803 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.803 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:04:56.803 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.572053+0000", 2024-09-21T11:04:56.803 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.041e-06 2024-09-21T11:04:56.803 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.803 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.803 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.804 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.572088+0000", 2024-09-21T11:04:56.804 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5231999999999997e-05 2024-09-21T11:04:56.804 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.804 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.804 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:04:56.804 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.572094+0000", 2024-09-21T11:04:56.804 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9789999999999998e-06 2024-09-21T11:04:56.804 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.804 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.804 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.804 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.521150+0000", 2024-09-21T11:04:56.804 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.94905558000000001 2024-09-21T11:04:56.804 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.804 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.804 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:04:56.805 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.521157+0000", 2024-09-21T11:04:56.805 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2490000000000004e-06 2024-09-21T11:04:56.805 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.805 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.805 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.805 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.633842+0000", 2024-09-21T11:04:56.805 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1126852979999999 2024-09-21T11:04:56.805 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.805 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.805 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:04:56.805 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.633953+0000", 2024-09-21T11:04:56.805 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011058200000000001 2024-09-21T11:04:56.805 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.806 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.806 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6,7", 2024-09-21T11:04:56.806 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.634074+0000", 2024-09-21T11:04:56.806 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000121488 2024-09-21T11:04:56.806 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.806 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.806 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:04:56.806 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.638189+0000", 2024-09-21T11:04:56.806 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0041143619999999999 2024-09-21T11:04:56.806 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.806 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.806 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.806 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.638224+0000", 2024-09-21T11:04:56.806 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5828999999999999e-05 2024-09-21T11:04:56.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.807 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.807 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.807 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.638463+0000", 2024-09-21T11:04:56.807 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000238373 2024-09-21T11:04:56.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.807 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.807 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.807 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.641050+0000", 2024-09-21T11:04:56.807 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0025870419999999999 2024-09-21T11:04:56.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.807 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.807 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:04:56.807 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.641099+0000", 2024-09-21T11:04:56.808 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9607999999999997e-05 2024-09-21T11:04:56.808 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.808 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.808 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:04:56.808 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.641203+0000", 2024-09-21T11:04:56.808 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000104094 2024-09-21T11:04:56.808 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.808 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:04:56.808 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.808 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.808 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.808 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:219174 58.6 58:6479ddda:::benchmark_data_smithi070_396992_object73215:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1605)", 2024-09-21T11:04:56.808 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:34.606152+0000", 2024-09-21T11:04:56.809 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 21.214755411999999, 2024-09-21T11:04:56.809 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.035756044, 2024-09-21T11:04:56.809 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:04:56.809 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:04:56.809 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:04:56.809 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:04:56.809 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:04:56.809 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 219174 2024-09-21T11:04:56.809 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.809 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:04:56.809 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.809 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:04:56.809 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.606152+0000", 2024-09-21T11:04:56.809 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.809 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.810 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.810 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:04:56.810 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.606152+0000", 2024-09-21T11:04:56.810 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.810 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.810 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.810 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:04:56.810 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.606151+0000", 2024-09-21T11:04:56.810 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:04:56.810 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.810 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.810 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:04:56.810 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.606158+0000", 2024-09-21T11:04:56.811 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1359999999999999e-06 2024-09-21T11:04:56.811 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.811 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.811 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:04:56.811 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.606171+0000", 2024-09-21T11:04:56.811 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2585999999999999e-05 2024-09-21T11:04:56.811 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.811 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.811 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:04:56.811 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.606177+0000", 2024-09-21T11:04:56.811 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0379999999999996e-06 2024-09-21T11:04:56.811 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.811 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.811 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.811 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.606230+0000", 2024-09-21T11:04:56.812 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3170000000000001e-05 2024-09-21T11:04:56.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.812 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.812 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:04:56.812 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.606243+0000", 2024-09-21T11:04:56.812 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2933e-05 2024-09-21T11:04:56.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.812 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.812 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.812 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.636017+0000", 2024-09-21T11:04:56.812 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0297743690000001 2024-09-21T11:04:56.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.812 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.812 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:04:56.812 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.636169+0000", 2024-09-21T11:04:56.812 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000151755 2024-09-21T11:04:56.813 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.813 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.813 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6,7", 2024-09-21T11:04:56.813 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.636297+0000", 2024-09-21T11:04:56.813 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000128374 2024-09-21T11:04:56.813 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.813 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.813 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:04:56.813 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.638473+0000", 2024-09-21T11:04:56.813 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021752199999999998 2024-09-21T11:04:56.813 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.813 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.813 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.813 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.638505+0000", 2024-09-21T11:04:56.813 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2549000000000003e-05 2024-09-21T11:04:56.814 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.814 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.814 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.814 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.641243+0000", 2024-09-21T11:04:56.814 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0027383080000000001 2024-09-21T11:04:56.814 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.814 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.814 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.814 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.641738+0000", 2024-09-21T11:04:56.814 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00049473300000000002 2024-09-21T11:04:56.814 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.814 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.814 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:04:56.815 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.641793+0000", 2024-09-21T11:04:56.815 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4389999999999999e-05 2024-09-21T11:04:56.815 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.815 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.815 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:04:56.815 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.641909+0000", 2024-09-21T11:04:56.815 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000116024 2024-09-21T11:04:56.815 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.815 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:04:56.815 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.815 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.815 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.815 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:235468 58.3 58:c6838012:::benchmark_data_smithi070_396992_object89509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1615)", 2024-09-21T11:04:56.816 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:53.657214+0000", 2024-09-21T11:04:56.816 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 2.163693925, 2024-09-21T11:04:56.816 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.097778777, 2024-09-21T11:04:56.816 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:04:56.816 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:04:56.816 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:04:56.816 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:04:56.816 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:04:56.816 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 235468 2024-09-21T11:04:56.816 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.816 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:04:56.816 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.816 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:04:56.816 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:53.657214+0000", 2024-09-21T11:04:56.817 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.817 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.817 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.817 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:04:56.817 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:53.657214+0000", 2024-09-21T11:04:56.817 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:04:56.817 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.817 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.817 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:04:56.817 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:53.657213+0000", 2024-09-21T11:04:56.817 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:04:56.817 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.817 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.818 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:04:56.818 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:53.657220+0000", 2024-09-21T11:04:56.818 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6510000000000001e-06 2024-09-21T11:04:56.818 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.818 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.818 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:04:56.818 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:53.657233+0000", 2024-09-21T11:04:56.818 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2300000000000001e-05 2024-09-21T11:04:56.818 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.818 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.818 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:04:56.818 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:53.657240+0000", 2024-09-21T11:04:56.818 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6629999999999993e-06 2024-09-21T11:04:56.819 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.819 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.819 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.819 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:53.660692+0000", 2024-09-21T11:04:56.819 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0034513740000000001 2024-09-21T11:04:56.819 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.819 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.819 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:04:56.819 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:53.660707+0000", 2024-09-21T11:04:56.819 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5597999999999999e-05 2024-09-21T11:04:56.819 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.819 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.819 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:04:56.819 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:54.751537+0000", 2024-09-21T11:04:56.820 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.090830199 2024-09-21T11:04:56.820 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.820 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.820 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:04:56.820 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:54.751632+0000", 2024-09-21T11:04:56.820 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4449000000000003e-05 2024-09-21T11:04:56.820 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.820 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.820 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,3,6", 2024-09-21T11:04:56.820 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:54.751737+0000", 2024-09-21T11:04:56.820 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000105552 2024-09-21T11:04:56.820 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.820 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.821 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.821 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:54.754565+0000", 2024-09-21T11:04:56.821 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0028272990000000001 2024-09-21T11:04:56.821 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.821 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.821 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.821 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:54.754627+0000", 2024-09-21T11:04:56.821 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1933000000000005e-05 2024-09-21T11:04:56.821 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.821 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.821 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:04:56.821 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:54.754635+0000", 2024-09-21T11:04:56.821 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3920000000000003e-06 2024-09-21T11:04:56.821 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.822 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.822 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:04:56.822 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:54.754794+0000", 2024-09-21T11:04:56.822 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015928900000000001 2024-09-21T11:04:56.822 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.822 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.822 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:04:56.822 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:54.754877+0000", 2024-09-21T11:04:56.822 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3166999999999998e-05 2024-09-21T11:04:56.822 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:04:56.822 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:04:56.822 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:04:56.822 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:54.754993+0000", 2024-09-21T11:04:56.823 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000115259 2024-09-21T11:04:56.823 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.823 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:04:56.823 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.823 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:04:56.823 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:04:56.823 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T11:04:56.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:56.824 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:56.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:56.925 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:56.938 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:56 smithi070 ceph-mon[107385]: osdmap e1617: 8 total, 8 up, 5 in 2024-09-21T11:04:56.938 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:56 smithi070 ceph-mon[112139]: osdmap e1617: 8 total, 8 up, 5 in 2024-09-21T11:04:56.962 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:56.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:56.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:57.016 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:04:57.064 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:57.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:56 smithi160 ceph-mon[101879]: osdmap e1617: 8 total, 8 up, 5 in 2024-09-21T11:04:57.092 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:57.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:57.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:57.194 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:57.226 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:57.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:57.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:57.329 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:57.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:57.371 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:57.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:57.472 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:57.498 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:57.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:57.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:57.600 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:57.636 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:57.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:57.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T11:04:57.737 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:57.766 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:57.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:57.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:57 smithi160 ceph-mon[101879]: pgmap v3199: 109 pgs: 4 peering, 2 active+recovery_wait+remapped, 3 active+undersized+degraded+remapped, 18 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 2 remapped+peering, 4 active+undersized+degraded+remapped+backfill_wait, 70 active+clean; 3.4 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 4.4 KiB/s rd, 13 KiB/s wr, 1.01k op/s; 24773/167271 objects degraded (14.810%); 39964/167271 objects misplaced (23.892%); 13 MiB/s, 300 objects/s recovering 2024-09-21T11:04:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:57 smithi160 ceph-mon[101879]: osdmap e1618: 8 total, 8 up, 5 in 2024-09-21T11:04:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:57 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2397956401' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:04:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:57 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:04:57.868 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:04:57.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:57 smithi070 ceph-mon[107385]: pgmap v3199: 109 pgs: 4 peering, 2 active+recovery_wait+remapped, 3 active+undersized+degraded+remapped, 18 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 2 remapped+peering, 4 active+undersized+degraded+remapped+backfill_wait, 70 active+clean; 3.4 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 4.4 KiB/s rd, 13 KiB/s wr, 1.01k op/s; 24773/167271 objects degraded (14.810%); 39964/167271 objects misplaced (23.892%); 13 MiB/s, 300 objects/s recovering 2024-09-21T11:04:57.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:57 smithi070 ceph-mon[107385]: osdmap e1618: 8 total, 8 up, 5 in 2024-09-21T11:04:57.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:57 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2397956401' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:04:57.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:57 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:04:57.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:57 smithi070 ceph-mon[112139]: pgmap v3199: 109 pgs: 4 peering, 2 active+recovery_wait+remapped, 3 active+undersized+degraded+remapped, 18 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 2 remapped+peering, 4 active+undersized+degraded+remapped+backfill_wait, 70 active+clean; 3.4 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 4.4 KiB/s rd, 13 KiB/s wr, 1.01k op/s; 24773/167271 objects degraded (14.810%); 39964/167271 objects misplaced (23.892%); 13 MiB/s, 300 objects/s recovering 2024-09-21T11:04:57.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:57 smithi070 ceph-mon[112139]: osdmap e1618: 8 total, 8 up, 5 in 2024-09-21T11:04:57.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:57 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2397956401' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:04:57.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:57 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:04:57.911 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:57.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:57.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:04:58.013 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:58.055 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:58.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:58.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:58.150 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T11:04:58.157 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:58.183 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:58.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:58.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:58.285 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:58.309 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:58.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:58.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:58.411 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:58.474 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:58.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:58.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:58.534 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:04:58.576 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:58.625 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:58.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:58.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:58.727 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:58.755 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:58.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:58.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:58.856 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:58.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:58.896 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:58.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:58.949 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:58 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:04:58.949 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:58 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T11:04:58.949 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:58 smithi160 ceph-mon[101879]: osdmap e1619: 8 total, 8 up, 5 in 2024-09-21T11:04:58.949 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:58 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 24773/167271 objects degraded (14.810%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:58.997 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:59.043 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:59.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:59.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:59.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:58 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:04:59.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:58 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T11:04:59.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:58 smithi070 ceph-mon[107385]: osdmap e1619: 8 total, 8 up, 5 in 2024-09-21T11:04:59.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:58 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 24773/167271 objects degraded (14.810%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:59.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:58 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:04:59.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:58 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T11:04:59.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:58 smithi070 ceph-mon[112139]: osdmap e1619: 8 total, 8 up, 5 in 2024-09-21T11:04:59.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:58 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 24773/167271 objects degraded (14.810%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T11:04:59.144 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:04:59.188 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:59.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:59.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:04:59.290 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:04:59.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:59.330 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:59.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:04:59.431 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:04:59.459 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:59.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:59.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:04:59.561 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:04:59.587 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:59.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:59.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:04:59.688 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:04:59.716 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:04:59.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:59.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:04:59.818 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:59.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:04:59 smithi160 ceph-mon[101879]: pgmap v3202: 109 pgs: 4 peering, 2 active+recovery_wait+remapped, 3 active+undersized+degraded+remapped, 18 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 2 remapped+peering, 4 active+undersized+degraded+remapped+backfill_wait, 70 active+clean; 3.4 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 1k op/s; 24773/167271 objects degraded (14.810%); 39960/167271 objects misplaced (23.889%); 80 KiB/s, 136 objects/s recovering 2024-09-21T11:04:59.860 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:04:59.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:04:59.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:04:59.961 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:04:59.984 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:04:59 smithi070 ceph-mon[107385]: pgmap v3202: 109 pgs: 4 peering, 2 active+recovery_wait+remapped, 3 active+undersized+degraded+remapped, 18 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 2 remapped+peering, 4 active+undersized+degraded+remapped+backfill_wait, 70 active+clean; 3.4 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 1k op/s; 24773/167271 objects degraded (14.810%); 39960/167271 objects misplaced (23.889%); 80 KiB/s, 136 objects/s recovering 2024-09-21T11:04:59.984 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:04:59 smithi070 ceph-mon[112139]: pgmap v3202: 109 pgs: 4 peering, 2 active+recovery_wait+remapped, 3 active+undersized+degraded+remapped, 18 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 2 remapped+peering, 4 active+undersized+degraded+remapped+backfill_wait, 70 active+clean; 3.4 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 1k op/s; 24773/167271 objects degraded (14.810%); 39960/167271 objects misplaced (23.889%); 80 KiB/s, 136 objects/s recovering 2024-09-21T11:05:00.003 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:00.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:00.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:00.105 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:00.138 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:00.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:00.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:00.240 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:00.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:00.269 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:00.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:00.370 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:00.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:00.392 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:00.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:00.493 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:00.540 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:00.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:00.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:00.642 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:00.647 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T11:05:00.647 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T11:05:00.648 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T11:05:00.648 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T11:05:00.663 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:00.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:00.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:00.765 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:00.793 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:00.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:00.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:00.895 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:00.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:00.924 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:00.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:01.025 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:01.055 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:01.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:01.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:01.157 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:01.192 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:01.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:01.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:01.294 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:01.325 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:01.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:01.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:01.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:00 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:05:01.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:00 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:05:01.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:00 smithi160 ceph-mon[101879]: pgmap v3203: 109 pgs: 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait, 78 active+clean; 3.3 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 901 op/s; 30087/160680 objects degraded (18.725%); 45219/160680 objects misplaced (28.142%); 0 B/s, 151 objects/s recovering 2024-09-21T11:05:01.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:00 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:05:01.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:00 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:05:01.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:00 smithi070 ceph-mon[107385]: pgmap v3203: 109 pgs: 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait, 78 active+clean; 3.3 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 901 op/s; 30087/160680 objects degraded (18.725%); 45219/160680 objects misplaced (28.142%); 0 B/s, 151 objects/s recovering 2024-09-21T11:05:01.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:00 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:05:01.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:00 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:05:01.378 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:00 smithi070 ceph-mon[112139]: pgmap v3203: 109 pgs: 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait, 78 active+clean; 3.3 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 901 op/s; 30087/160680 objects degraded (18.725%); 45219/160680 objects misplaced (28.142%); 0 B/s, 151 objects/s recovering 2024-09-21T11:05:01.428 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:01.467 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:01.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:01.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:01.569 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:01.596 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:01.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:01.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:01.698 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:01.719 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T11:05:01.737 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:01.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:01.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:01.839 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:01.864 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:01.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:01.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:01.966 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:01.991 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:01.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:01.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:02.093 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:02.109 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:05:02.137 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:02.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:02.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:02.239 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:02.268 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:02.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:02.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:02.370 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:02.392 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:02.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:02.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:02.493 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:02.523 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:02.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:02.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:02.625 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:02.654 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:02.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:02.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:02.756 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:02.790 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:02.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:02.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:02.892 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:02.929 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:02.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:02.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:03.030 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:03.064 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:03.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:03.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:03.166 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:03.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:03.194 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:03.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:03.295 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:03.327 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:03.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:03.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:03.429 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:03.464 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:03.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:03.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:03.565 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:03.587 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:03.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:03.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:03.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:03 smithi160 ceph-mon[101879]: pgmap v3204: 109 pgs: 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait, 78 active+clean; 3.1 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 28540/152349 objects degraded (18.733%); 42602/152349 objects misplaced (27.963%); 0 B/s, 49 objects/s recovering 2024-09-21T11:05:03.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:03 smithi070 ceph-mon[107385]: pgmap v3204: 109 pgs: 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait, 78 active+clean; 3.1 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 28540/152349 objects degraded (18.733%); 42602/152349 objects misplaced (27.963%); 0 B/s, 49 objects/s recovering 2024-09-21T11:05:03.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:05:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:05:03] "GET /metrics HTTP/1.1" 200 36769 "" "Prometheus/2.51.0" 2024-09-21T11:05:03.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:03 smithi070 ceph-mon[112139]: pgmap v3204: 109 pgs: 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait, 78 active+clean; 3.1 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 28540/152349 objects degraded (18.733%); 42602/152349 objects misplaced (27.963%); 0 B/s, 49 objects/s recovering 2024-09-21T11:05:03.690 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:03.711 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:03.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:03.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:03.813 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:03.835 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:03.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:03.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:03.937 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:03.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:03.960 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:03.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:04.061 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:04.082 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:04.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:04.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:04.184 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:04.206 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:04.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:04.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:04.308 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:04.329 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:04.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:04.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:04.431 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:04.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:04.462 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:04.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:04.521 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:04 smithi160 ceph-mon[101879]: osdmap e1620: 8 total, 8 up, 5 in 2024-09-21T11:05:04.521 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:04 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 28540/152349 objects degraded (18.733%), 10 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T11:05:04.562 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:04.602 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:04.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:04.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:04.704 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:04.733 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:04 smithi070 ceph-mon[107385]: osdmap e1620: 8 total, 8 up, 5 in 2024-09-21T11:05:04.733 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:04 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 28540/152349 objects degraded (18.733%), 10 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T11:05:04.736 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:04 smithi070 ceph-mon[112139]: osdmap e1620: 8 total, 8 up, 5 in 2024-09-21T11:05:04.736 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:04 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 28540/152349 objects degraded (18.733%), 10 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T11:05:04.746 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:04.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:04.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:04.848 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:04.876 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:04.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:04.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:04.978 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:05.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:05.021 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:05.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:05.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:05.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:05.171 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:05.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:05.272 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:05.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:05.302 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:05.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:05.403 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:05.443 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:05.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:05.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:05.545 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:05.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:05.586 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:05.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:05.611 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T11:05:05.611 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T11:05:05.611 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T11:05:05.611 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T11:05:05.611 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T11:05:05.688 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:05.702 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:05 smithi070 ceph-mon[107385]: pgmap v3206: 109 pgs: 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait, 78 active+clean; 3.1 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 647 op/s; 28540/152349 objects degraded (18.733%); 42602/152349 objects misplaced (27.963%); 0 B/s, 42 objects/s recovering 2024-09-21T11:05:05.703 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:05 smithi070 ceph-mon[107385]: osdmap e1621: 8 total, 8 up, 5 in 2024-09-21T11:05:05.703 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T11:05:05.703 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T11:05:05.703 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:05:05.703 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:05 smithi070 ceph-mon[112139]: pgmap v3206: 109 pgs: 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait, 78 active+clean; 3.1 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 647 op/s; 28540/152349 objects degraded (18.733%); 42602/152349 objects misplaced (27.963%); 0 B/s, 42 objects/s recovering 2024-09-21T11:05:05.703 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:05 smithi070 ceph-mon[112139]: osdmap e1621: 8 total, 8 up, 5 in 2024-09-21T11:05:05.703 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T11:05:05.703 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T11:05:05.703 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:05:05.727 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:05.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:05.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:05.828 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:05 smithi160 ceph-mon[101879]: pgmap v3206: 109 pgs: 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait, 78 active+clean; 3.1 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 647 op/s; 28540/152349 objects degraded (18.733%); 42602/152349 objects misplaced (27.963%); 0 B/s, 42 objects/s recovering 2024-09-21T11:05:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:05 smithi160 ceph-mon[101879]: osdmap e1621: 8 total, 8 up, 5 in 2024-09-21T11:05:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T11:05:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T11:05:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:05:05.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:05.864 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:05.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:05.876 INFO:teuthology.orchestra.run.smithi070.stdout:moved 0 / 327 (0%) 2024-09-21T11:05:05.877 INFO:teuthology.orchestra.run.smithi070.stdout:avg 65.4 2024-09-21T11:05:05.877 INFO:teuthology.orchestra.run.smithi070.stdout:stddev 22.3652 -> 22.3087 (expected baseline 7.23326) 2024-09-21T11:05:05.877 INFO:teuthology.orchestra.run.smithi070.stdout:min osd.7 with 27 -> 27 pgs (0.412844 -> 0.412844 * mean) 2024-09-21T11:05:05.877 INFO:teuthology.orchestra.run.smithi070.stdout:max osd.6 with 96 -> 96 pgs (1.46789 -> 1.46789 * mean) 2024-09-21T11:05:05.877 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T11:05:05.877 INFO:teuthology.orchestra.run.smithi070.stdout:oload 1000 2024-09-21T11:05:05.877 INFO:teuthology.orchestra.run.smithi070.stdout:max_change 1 2024-09-21T11:05:05.877 INFO:teuthology.orchestra.run.smithi070.stdout:max_change_osds 4 2024-09-21T11:05:05.877 INFO:teuthology.orchestra.run.smithi070.stdout:average_utilization 0.0286 2024-09-21T11:05:05.877 INFO:teuthology.orchestra.run.smithi070.stdout:overload_utilization 0.2864 2024-09-21T11:05:05.877 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 weight 0.7003 -> 0.7997 2024-09-21T11:05:05.877 INFO:teuthology.orchestra.run.smithi070.stderr:no change 2024-09-21T11:05:05.965 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:05.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:05.995 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:05.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:06.025 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:05:05 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T11:05:05.873+0000 7fd088d1d640 -1 mgr.server reply reply (1) Operation not permitted no change 2024-09-21T11:05:06.096 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:06.126 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:06.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:06.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:06.227 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:06.254 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:06.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:06.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:06.356 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:06.384 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:06.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:06.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:06.390 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T11:05:06.429 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T11:05:06.486 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:06.514 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:06.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:06.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:06.616 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:05:06.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:05:06.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:05:06.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:05:06.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:06.649 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:06.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:06.750 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:06.783 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:06.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:06.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:06.793 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:05:06.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:05:06.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:05:06.885 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:06.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:06.934 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:06.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:07.035 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:07.062 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:07.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:07.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:07.164 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:07.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:07.200 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:07.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:07.301 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:07.342 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:07.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:07.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:07.444 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:07.478 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:07.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:07.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:07.532 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:07 smithi070 ceph-mon[107385]: from='client.36747 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T11:05:07.532 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:07 smithi070 ceph-mon[107385]: pgmap v3208: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 2.8 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 1.15k op/s; 25060/139512 objects degraded (17.963%); 38961/139512 objects misplaced (27.927%); 6.7 MiB/s, 174 objects/s recovering 2024-09-21T11:05:07.533 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:07 smithi070 ceph-mon[112139]: from='client.36747 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T11:05:07.533 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:07 smithi070 ceph-mon[112139]: pgmap v3208: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 2.8 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 1.15k op/s; 25060/139512 objects degraded (17.963%); 38961/139512 objects misplaced (27.927%); 6.7 MiB/s, 174 objects/s recovering 2024-09-21T11:05:07.580 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:07.610 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:07.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:07.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:07.712 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:07.741 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:07.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:07.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:07 smithi160 ceph-mon[101879]: from='client.36747 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T11:05:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:07 smithi160 ceph-mon[101879]: pgmap v3208: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 2.8 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 1.15k op/s; 25060/139512 objects degraded (17.963%); 38961/139512 objects misplaced (27.927%); 6.7 MiB/s, 174 objects/s recovering 2024-09-21T11:05:07.843 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:07.883 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:07.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:07.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:07.984 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:08.025 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:08.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:08.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:08.126 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:08.159 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:08.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:08.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:08.261 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:08.293 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:08.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:08.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:08.395 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:08.448 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:08.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:08.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:08.550 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:08.575 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:08.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:08.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:08.677 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:08.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:08.715 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:08.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:08.816 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:08.844 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:08.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:08.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:08.947 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:08.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:08.979 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:08.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:09.041 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:05:09.042 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:05:09.047 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:05:09.065 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:05:09.065 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:05:09.066 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T11:05:09.066 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T11:05:09.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T11:05:09.066 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T11:05:09.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.066 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36012.0:97179 56.7 56:e284a7fd:::benchmark_data_smithi070_333437_object97178:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1499)", 2024-09-21T11:05:09.066 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:55:37.696861+0000", 2024-09-21T11:05:09.067 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 571.36329120699997, 2024-09-21T11:05:09.067 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.686342864, 2024-09-21T11:05:09.067 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:05:09.067 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T11:05:09.067 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:05:09.067 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36012", 2024-09-21T11:05:09.067 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/43219434", 2024-09-21T11:05:09.067 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 97179 2024-09-21T11:05:09.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.067 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:05:09.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:05:09.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:37.696861+0000", 2024-09-21T11:05:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:05:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:37.696861+0000", 2024-09-21T11:05:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:05:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:37.696860+0000", 2024-09-21T11:05:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T11:05:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.068 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:05:09.069 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:37.696902+0000", 2024-09-21T11:05:09.069 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1814000000000003e-05 2024-09-21T11:05:09.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.069 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:05:09.069 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:37.696908+0000", 2024-09-21T11:05:09.069 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1859999999999997e-06 2024-09-21T11:05:09.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.069 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:05:09.069 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:37.696913+0000", 2024-09-21T11:05:09.070 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7230000000000004e-06 2024-09-21T11:05:09.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.070 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.070 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.070 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:37.898420+0000", 2024-09-21T11:05:09.070 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20150755300000001 2024-09-21T11:05:09.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.070 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.070 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:05:09.070 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:37.898429+0000", 2024-09-21T11:05:09.070 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1419999999999993e-06 2024-09-21T11:05:09.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:38.364566+0000", 2024-09-21T11:05:09.071 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.46613732899999999 2024-09-21T11:05:09.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:05:09.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:38.364577+0000", 2024-09-21T11:05:09.071 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0839000000000001e-05 2024-09-21T11:05:09.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:39.383121+0000", 2024-09-21T11:05:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.018544656 2024-09-21T11:05:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:05:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:55:39.383204+0000", 2024-09-21T11:05:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2132000000000006e-05 2024-09-21T11:05:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:05:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:84903 57.18 57:1a2783a1:::benchmark_data_smithi070_360420_object17554:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1559)", 2024-09-21T11:05:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:00:40.450831+0000", 2024-09-21T11:05:09.072 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 268.609320712, 2024-09-21T11:05:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9987897640000001, 2024-09-21T11:05:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:05:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:05:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:05:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T11:05:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:05:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 84903 2024-09-21T11:05:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:05:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:05:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.450831+0000", 2024-09-21T11:05:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.074 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.074 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:05:09.074 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.450831+0000", 2024-09-21T11:05:09.074 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.074 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.074 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.074 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:05:09.074 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.450831+0000", 2024-09-21T11:05:09.074 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T11:05:09.074 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.074 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.074 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:05:09.074 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.450833+0000", 2024-09-21T11:05:09.074 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8920000000000001e-06 2024-09-21T11:05:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:05:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.450838+0000", 2024-09-21T11:05:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0170000000000004e-06 2024-09-21T11:05:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:05:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.450842+0000", 2024-09-21T11:05:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.755e-06 2024-09-21T11:05:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.075 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.622223+0000", 2024-09-21T11:05:09.076 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.171381743 2024-09-21T11:05:09.076 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.076 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.076 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:05:09.076 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.622232+0000", 2024-09-21T11:05:09.076 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3110000000000006e-06 2024-09-21T11:05:09.076 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.076 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.076 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.076 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.426926+0000", 2024-09-21T11:05:09.076 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8046946619999999 2024-09-21T11:05:09.076 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.076 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:05:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.426936+0000", 2024-09-21T11:05:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6169999999999996e-06 2024-09-21T11:05:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.444284+0000", 2024-09-21T11:05:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.017348403 2024-09-21T11:05:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:05:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.444394+0000", 2024-09-21T11:05:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000109795 2024-09-21T11:05:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.077 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-21T11:05:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.444560+0000", 2024-09-21T11:05:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016564600000000001 2024-09-21T11:05:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.449079+0000", 2024-09-21T11:05:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0045190559999999996 2024-09-21T11:05:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:05:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.449086+0000", 2024-09-21T11:05:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3629999999999996e-06 2024-09-21T11:05:09.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.449132+0000", 2024-09-21T11:05:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5802e-05 2024-09-21T11:05:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:05:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.449181+0000", 2024-09-21T11:05:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8937999999999998e-05 2024-09-21T11:05:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:05:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.449621+0000", 2024-09-21T11:05:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00044004999999999999 2024-09-21T11:05:09.079 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:05:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:84913 57.18 57:189fa363:::benchmark_data_smithi070_360420_object17564:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1559)", 2024-09-21T11:05:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:00:40.589206+0000", 2024-09-21T11:05:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 268.47094597400002, 2024-09-21T11:05:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8690907279999998, 2024-09-21T11:05:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:05:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:05:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:05:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T11:05:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:05:09.080 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 84913 2024-09-21T11:05:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:05:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:05:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.589206+0000", 2024-09-21T11:05:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:05:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.589206+0000", 2024-09-21T11:05:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:05:09.081 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.589204+0000", 2024-09-21T11:05:09.082 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:05:09.082 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.082 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.082 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:05:09.082 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.589212+0000", 2024-09-21T11:05:09.082 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2139999999999999e-06 2024-09-21T11:05:09.082 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.082 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.082 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:05:09.082 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.589223+0000", 2024-09-21T11:05:09.082 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1399999999999999e-05 2024-09-21T11:05:09.082 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.082 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:05:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.589232+0000", 2024-09-21T11:05:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1870000000000003e-06 2024-09-21T11:05:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.622249+0000", 2024-09-21T11:05:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.033016851 2024-09-21T11:05:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:05:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.622254+0000", 2024-09-21T11:05:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9420000000000003e-06 2024-09-21T11:05:09.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.426956+0000", 2024-09-21T11:05:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8047021640000001 2024-09-21T11:05:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:05:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.426962+0000", 2024-09-21T11:05:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8209999999999998e-06 2024-09-21T11:05:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.446121+0000", 2024-09-21T11:05:09.084 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0191587419999999 2024-09-21T11:05:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T11:05:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.449020+0000", 2024-09-21T11:05:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0028993199999999999 2024-09-21T11:05:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.452360+0000", 2024-09-21T11:05:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0033404849999999998 2024-09-21T11:05:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:05:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.452454+0000", 2024-09-21T11:05:09.085 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3280000000000001e-05 2024-09-21T11:05:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-21T11:05:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.452685+0000", 2024-09-21T11:05:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023139800000000001 2024-09-21T11:05:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:05:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.456964+0000", 2024-09-21T11:05:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0042792109999999998 2024-09-21T11:05:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.086 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.457870+0000", 2024-09-21T11:05:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00090521000000000004 2024-09-21T11:05:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.457926+0000", 2024-09-21T11:05:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6041000000000001e-05 2024-09-21T11:05:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:05:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.457976+0000", 2024-09-21T11:05:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0555e-05 2024-09-21T11:05:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:05:09.087 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.458297+0000", 2024-09-21T11:05:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00032048899999999997 2024-09-21T11:05:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:05:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:84916 57.5 57:a6fd8b58:::benchmark_data_smithi070_360420_object17567:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1559)", 2024-09-21T11:05:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:00:40.629113+0000", 2024-09-21T11:05:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 268.43103925100002, 2024-09-21T11:05:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.846688651, 2024-09-21T11:05:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:05:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:05:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:05:09.088 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T11:05:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:05:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 84916 2024-09-21T11:05:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:05:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:05:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.629113+0000", 2024-09-21T11:05:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:05:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.629113+0000", 2024-09-21T11:05:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.089 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:05:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.629111+0000", 2024-09-21T11:05:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T11:05:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:05:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.629121+0000", 2024-09-21T11:05:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8300000000000008e-06 2024-09-21T11:05:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:05:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.629132+0000", 2024-09-21T11:05:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1609e-05 2024-09-21T11:05:09.090 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:05:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.629139+0000", 2024-09-21T11:05:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5540000000000003e-06 2024-09-21T11:05:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.629181+0000", 2024-09-21T11:05:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2475000000000002e-05 2024-09-21T11:05:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:05:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.629190+0000", 2024-09-21T11:05:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2449999999999998e-06 2024-09-21T11:05:09.091 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.433253+0000", 2024-09-21T11:05:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.80406289500000006 2024-09-21T11:05:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:05:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.433266+0000", 2024-09-21T11:05:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3584000000000001e-05 2024-09-21T11:05:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.444118+0000", 2024-09-21T11:05:09.092 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0108519810000001 2024-09-21T11:05:09.093 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.093 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.093 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T11:05:09.093 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.444637+0000", 2024-09-21T11:05:09.093 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00051886499999999997 2024-09-21T11:05:09.093 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.093 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.093 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.093 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:09.094 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:05:09.095 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:05:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.465502+0000", 2024-09-21T11:05:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020864866999999999 2024-09-21T11:05:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:05:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.465584+0000", 2024-09-21T11:05:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2575000000000001e-05 2024-09-21T11:05:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-21T11:05:09.095 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.465865+0000", 2024-09-21T11:05:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028076000000000002 2024-09-21T11:05:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.467463+0000", 2024-09-21T11:05:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0015981039999999999 2024-09-21T11:05:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:05:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.469545+0000", 2024-09-21T11:05:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0020818500000000001 2024-09-21T11:05:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.096 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.474699+0000", 2024-09-21T11:05:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0051534240000000002 2024-09-21T11:05:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:05:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.474775+0000", 2024-09-21T11:05:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6031000000000004e-05 2024-09-21T11:05:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:05:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.475801+0000", 2024-09-21T11:05:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001026735 2024-09-21T11:05:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:05:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.097 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:84948 57.18 57:1ea98b34:::benchmark_data_smithi070_360420_object17599:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1560)", 2024-09-21T11:05:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:00:41.508218+0000", 2024-09-21T11:05:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 267.55193385500002, 2024-09-21T11:05:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9505037220000001, 2024-09-21T11:05:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:05:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:05:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:05:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T11:05:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:05:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 84948 2024-09-21T11:05:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:05:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.098 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:05:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.508218+0000", 2024-09-21T11:05:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:05:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.508218+0000", 2024-09-21T11:05:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:05:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.508217+0000", 2024-09-21T11:05:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:05:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.099 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:05:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.508222+0000", 2024-09-21T11:05:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6699999999999999e-06 2024-09-21T11:05:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:05:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.508233+0000", 2024-09-21T11:05:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.022e-05 2024-09-21T11:05:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:05:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.508239+0000", 2024-09-21T11:05:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6819999999999997e-06 2024-09-21T11:05:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.100 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.508287+0000", 2024-09-21T11:05:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7735999999999998e-05 2024-09-21T11:05:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:05:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.508294+0000", 2024-09-21T11:05:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3409999999999997e-06 2024-09-21T11:05:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.426980+0000", 2024-09-21T11:05:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.918685686 2024-09-21T11:05:09.101 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:05:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.426986+0000", 2024-09-21T11:05:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5509999999999999e-06 2024-09-21T11:05:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.449668+0000", 2024-09-21T11:05:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.02268208 2024-09-21T11:05:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T11:05:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.450193+0000", 2024-09-21T11:05:09.102 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00052498900000000001 2024-09-21T11:05:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.455091+0000", 2024-09-21T11:05:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0048979899999999996 2024-09-21T11:05:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:05:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.455186+0000", 2024-09-21T11:05:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5539999999999997e-05 2024-09-21T11:05:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-21T11:05:09.103 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.455384+0000", 2024-09-21T11:05:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019748300000000001 2024-09-21T11:05:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:05:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.457879+0000", 2024-09-21T11:05:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0024956359999999999 2024-09-21T11:05:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.458335+0000", 2024-09-21T11:05:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00045515699999999998 2024-09-21T11:05:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.104 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.105 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.458362+0000", 2024-09-21T11:05:09.105 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7668e-05 2024-09-21T11:05:09.105 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.105 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.105 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:05:09.105 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.458407+0000", 2024-09-21T11:05:09.105 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5241000000000003e-05 2024-09-21T11:05:09.105 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.105 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.105 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:05:09.105 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.458722+0000", 2024-09-21T11:05:09.105 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00031433500000000001 2024-09-21T11:05:09.105 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.105 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:05:09.106 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.106 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.106 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.106 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:84956 57.18 57:1d6cf45f:::benchmark_data_smithi070_360420_object17607:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1560)", 2024-09-21T11:05:09.106 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:00:41.512349+0000", 2024-09-21T11:05:09.106 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 267.54780311000002, 2024-09-21T11:05:09.106 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.95018297, 2024-09-21T11:05:09.106 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:05:09.106 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:05:09.106 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:05:09.106 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T11:05:09.106 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:05:09.106 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 84956 2024-09-21T11:05:09.106 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.107 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:05:09.107 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.107 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:05:09.107 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.512349+0000", 2024-09-21T11:05:09.107 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.107 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.107 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.107 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:05:09.107 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.512349+0000", 2024-09-21T11:05:09.107 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.107 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.107 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.107 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:05:09.107 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.512348+0000", 2024-09-21T11:05:09.108 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:05:09.108 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.108 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.108 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:05:09.108 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.512352+0000", 2024-09-21T11:05:09.108 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3239999999999996e-06 2024-09-21T11:05:09.108 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.108 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.108 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:05:09.108 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.512360+0000", 2024-09-21T11:05:09.108 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9950000000000005e-06 2024-09-21T11:05:09.108 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.108 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.108 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:05:09.109 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.512366+0000", 2024-09-21T11:05:09.109 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7910000000000001e-06 2024-09-21T11:05:09.109 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.109 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.109 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.109 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.512415+0000", 2024-09-21T11:05:09.109 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8634000000000002e-05 2024-09-21T11:05:09.109 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.109 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.109 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:05:09.109 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.512423+0000", 2024-09-21T11:05:09.109 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9039999999999992e-06 2024-09-21T11:05:09.109 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.109 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.110 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.110 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.427002+0000", 2024-09-21T11:05:09.110 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91457977099999999 2024-09-21T11:05:09.110 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.110 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.110 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:05:09.110 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.427008+0000", 2024-09-21T11:05:09.110 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4190000000000001e-06 2024-09-21T11:05:09.110 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.110 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.110 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.110 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.450228+0000", 2024-09-21T11:05:09.110 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0232196790000001 2024-09-21T11:05:09.111 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.111 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.111 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T11:05:09.111 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.451455+0000", 2024-09-21T11:05:09.111 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0012271890000000001 2024-09-21T11:05:09.111 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.111 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.111 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.111 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.456994+0000", 2024-09-21T11:05:09.111 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.005539647 2024-09-21T11:05:09.111 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.111 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.111 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:05:09.112 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.457051+0000", 2024-09-21T11:05:09.112 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6971000000000001e-05 2024-09-21T11:05:09.112 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.112 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.112 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-21T11:05:09.112 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.457185+0000", 2024-09-21T11:05:09.112 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013387800000000001 2024-09-21T11:05:09.112 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.112 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.112 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.112 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.458758+0000", 2024-09-21T11:05:09.112 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0015725719999999999 2024-09-21T11:05:09.112 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.113 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.113 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.113 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.458790+0000", 2024-09-21T11:05:09.113 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1738e-05 2024-09-21T11:05:09.113 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.113 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.113 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:05:09.113 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.461839+0000", 2024-09-21T11:05:09.113 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0030498309999999998 2024-09-21T11:05:09.113 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.113 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.113 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:05:09.113 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.461912+0000", 2024-09-21T11:05:09.113 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2309000000000002e-05 2024-09-21T11:05:09.113 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.114 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.114 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:05:09.114 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.462532+0000", 2024-09-21T11:05:09.114 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00062006500000000005 2024-09-21T11:05:09.114 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.114 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:05:09.114 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.114 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.114 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.114 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:84989 57.18 57:1b2d9fc9:::benchmark_data_smithi070_360420_object17640:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1560)", 2024-09-21T11:05:09.114 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:00:41.540067+0000", 2024-09-21T11:05:09.114 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 267.52008470200002, 2024-09-21T11:05:09.114 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9312866450000001, 2024-09-21T11:05:09.114 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:05:09.115 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:05:09.115 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:05:09.115 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T11:05:09.115 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:05:09.115 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 84989 2024-09-21T11:05:09.115 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.115 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:05:09.115 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.115 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:05:09.115 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.540067+0000", 2024-09-21T11:05:09.115 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.115 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.115 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.115 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:05:09.115 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.540067+0000", 2024-09-21T11:05:09.116 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.116 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.116 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.116 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:05:09.116 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.540066+0000", 2024-09-21T11:05:09.116 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:05:09.116 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.116 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.116 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:05:09.116 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.540071+0000", 2024-09-21T11:05:09.116 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.22e-06 2024-09-21T11:05:09.116 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.116 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.116 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:05:09.117 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.540082+0000", 2024-09-21T11:05:09.117 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0886e-05 2024-09-21T11:05:09.117 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.117 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.117 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:05:09.117 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.540089+0000", 2024-09-21T11:05:09.117 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6730000000000003e-06 2024-09-21T11:05:09.117 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.117 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.117 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.117 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.540165+0000", 2024-09-21T11:05:09.117 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6431000000000001e-05 2024-09-21T11:05:09.117 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.117 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.117 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:05:09.118 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.540174+0000", 2024-09-21T11:05:09.118 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6969999999999999e-06 2024-09-21T11:05:09.118 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.118 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.118 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.118 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.427027+0000", 2024-09-21T11:05:09.118 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88685251200000004 2024-09-21T11:05:09.118 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.118 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.118 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:05:09.118 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.427033+0000", 2024-09-21T11:05:09.118 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2060000000000004e-06 2024-09-21T11:05:09.118 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.118 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.119 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.119 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.458811+0000", 2024-09-21T11:05:09.119 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.031777978 2024-09-21T11:05:09.119 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.119 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.119 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T11:05:09.119 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.459351+0000", 2024-09-21T11:05:09.119 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00054042399999999996 2024-09-21T11:05:09.119 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.119 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.119 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.119 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.464277+0000", 2024-09-21T11:05:09.119 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0049253639999999998 2024-09-21T11:05:09.119 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.120 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.120 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:05:09.120 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.464379+0000", 2024-09-21T11:05:09.120 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000102583 2024-09-21T11:05:09.120 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.120 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.120 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-21T11:05:09.120 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.464632+0000", 2024-09-21T11:05:09.120 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025245600000000002 2024-09-21T11:05:09.120 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.120 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.120 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:05:09.120 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.468167+0000", 2024-09-21T11:05:09.120 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0035357209999999999 2024-09-21T11:05:09.120 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.121 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.121 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.121 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.470643+0000", 2024-09-21T11:05:09.121 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002475882 2024-09-21T11:05:09.121 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.121 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.121 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.121 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.470755+0000", 2024-09-21T11:05:09.121 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011183699999999999 2024-09-21T11:05:09.121 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.121 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.121 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:05:09.123 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:05:09.123 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:05:09.123 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.470805+0000", 2024-09-21T11:05:09.123 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9704999999999999e-05 2024-09-21T11:05:09.123 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.123 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.123 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:05:09.123 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.471354+0000", 2024-09-21T11:05:09.124 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00054911599999999999 2024-09-21T11:05:09.124 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.124 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:05:09.124 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.124 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.124 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.124 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:85034 57.18 57:18bea2e4:::benchmark_data_smithi070_360420_object17685:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1560)", 2024-09-21T11:05:09.124 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:00:41.614931+0000", 2024-09-21T11:05:09.124 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 267.44522136400002, 2024-09-21T11:05:09.124 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8569978009999999, 2024-09-21T11:05:09.124 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:05:09.124 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:05:09.124 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:05:09.124 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T11:05:09.125 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:05:09.125 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 85034 2024-09-21T11:05:09.125 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.125 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:05:09.125 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.125 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:05:09.125 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.614931+0000", 2024-09-21T11:05:09.125 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.125 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.125 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.125 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:05:09.125 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.614931+0000", 2024-09-21T11:05:09.125 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.125 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.126 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.126 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:05:09.126 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.614930+0000", 2024-09-21T11:05:09.126 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T11:05:09.126 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.126 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.126 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:05:09.126 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.614934+0000", 2024-09-21T11:05:09.126 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7749999999999999e-06 2024-09-21T11:05:09.126 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.126 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.126 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:05:09.126 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.614942+0000", 2024-09-21T11:05:09.126 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.276e-06 2024-09-21T11:05:09.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.127 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.127 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:05:09.127 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.614947+0000", 2024-09-21T11:05:09.127 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9479999999999999e-06 2024-09-21T11:05:09.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.127 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.127 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.127 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.614982+0000", 2024-09-21T11:05:09.127 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5445999999999999e-05 2024-09-21T11:05:09.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.127 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.127 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:05:09.128 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.614992+0000", 2024-09-21T11:05:09.128 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3230000000000004e-06 2024-09-21T11:05:09.128 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.128 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.128 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.128 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.427052+0000", 2024-09-21T11:05:09.128 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.81206049899999999 2024-09-21T11:05:09.128 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.128 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.128 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:05:09.128 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.427059+0000", 2024-09-21T11:05:09.128 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6950000000000003e-06 2024-09-21T11:05:09.128 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.128 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.128 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.129 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.459388+0000", 2024-09-21T11:05:09.129 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0323289659999999 2024-09-21T11:05:09.129 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.129 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.129 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T11:05:09.129 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.461825+0000", 2024-09-21T11:05:09.129 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002436737 2024-09-21T11:05:09.129 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.129 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.129 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.129 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.466857+0000", 2024-09-21T11:05:09.129 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0050320390000000003 2024-09-21T11:05:09.129 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.129 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.130 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:05:09.130 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.466942+0000", 2024-09-21T11:05:09.130 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5928999999999997e-05 2024-09-21T11:05:09.130 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.130 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.130 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-21T11:05:09.130 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.467178+0000", 2024-09-21T11:05:09.130 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023586200000000001 2024-09-21T11:05:09.130 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.130 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.130 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.130 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.470704+0000", 2024-09-21T11:05:09.130 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0035258149999999999 2024-09-21T11:05:09.130 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.131 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.131 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:05:09.131 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.470718+0000", 2024-09-21T11:05:09.131 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3920000000000001e-05 2024-09-21T11:05:09.131 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.131 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.131 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.131 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.471413+0000", 2024-09-21T11:05:09.131 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00069485499999999997 2024-09-21T11:05:09.131 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.131 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.131 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:05:09.131 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.471495+0000", 2024-09-21T11:05:09.131 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1886999999999997e-05 2024-09-21T11:05:09.132 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.132 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.132 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:05:09.132 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.471928+0000", 2024-09-21T11:05:09.132 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000433516 2024-09-21T11:05:09.132 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.132 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:05:09.132 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.132 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.132 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.132 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:54311 58.a 58:577c5622:::benchmark_data_smithi070_396992_object54310:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1585)", 2024-09-21T11:05:09.132 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:02:17.816986+0000", 2024-09-21T11:05:09.132 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 171.24316565699999, 2024-09-21T11:05:09.132 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.097134527999999998, 2024-09-21T11:05:09.133 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:05:09.133 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:05:09.133 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:05:09.133 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:05:09.133 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:05:09.133 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 54311 2024-09-21T11:05:09.133 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.133 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:05:09.133 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.133 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:05:09.133 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.816986+0000", 2024-09-21T11:05:09.133 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.133 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.133 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.133 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:05:09.134 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.816986+0000", 2024-09-21T11:05:09.134 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.134 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.134 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.134 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:05:09.134 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.816985+0000", 2024-09-21T11:05:09.134 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:05:09.134 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.134 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.134 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:05:09.134 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.817055+0000", 2024-09-21T11:05:09.134 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9688000000000001e-05 2024-09-21T11:05:09.134 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.134 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.134 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:05:09.135 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.817073+0000", 2024-09-21T11:05:09.135 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7907e-05 2024-09-21T11:05:09.135 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.135 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.135 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:05:09.135 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.817078+0000", 2024-09-21T11:05:09.135 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5439999999999998e-06 2024-09-21T11:05:09.135 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.135 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.135 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.135 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.818448+0000", 2024-09-21T11:05:09.135 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0013695280000000001 2024-09-21T11:05:09.135 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.135 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.136 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:05:09.136 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.818608+0000", 2024-09-21T11:05:09.136 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000160688 2024-09-21T11:05:09.136 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.136 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.136 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3", 2024-09-21T11:05:09.136 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.818774+0000", 2024-09-21T11:05:09.136 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016545500000000001 2024-09-21T11:05:09.136 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.136 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.136 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:05:09.136 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.820521+0000", 2024-09-21T11:05:09.136 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0017476130000000001 2024-09-21T11:05:09.136 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.137 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.137 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.137 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.913990+0000", 2024-09-21T11:05:09.137 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.093468564000000004 2024-09-21T11:05:09.137 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.137 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.137 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:05:09.137 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.914027+0000", 2024-09-21T11:05:09.137 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7416000000000002e-05 2024-09-21T11:05:09.137 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.137 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.137 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:05:09.137 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.914121+0000", 2024-09-21T11:05:09.137 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3364999999999998e-05 2024-09-21T11:05:09.137 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.138 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:05:09.138 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.138 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.138 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.138 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:54347 58.a 58:5080fd9e:::benchmark_data_smithi070_396992_object54346:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1585)", 2024-09-21T11:05:09.138 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:02:17.839385+0000", 2024-09-21T11:05:09.138 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 171.22076736, 2024-09-21T11:05:09.138 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.074932762, 2024-09-21T11:05:09.138 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:05:09.138 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:05:09.138 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:05:09.138 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:05:09.138 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:05:09.138 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 54347 2024-09-21T11:05:09.139 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.139 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:05:09.139 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.139 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:05:09.139 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.839385+0000", 2024-09-21T11:05:09.139 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.139 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.139 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.139 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:05:09.139 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.839385+0000", 2024-09-21T11:05:09.139 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.139 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.139 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.139 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:05:09.139 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.839383+0000", 2024-09-21T11:05:09.140 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:05:09.140 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.140 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.140 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:05:09.140 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.839423+0000", 2024-09-21T11:05:09.140 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9378000000000001e-05 2024-09-21T11:05:09.140 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.140 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.140 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:05:09.140 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.839434+0000", 2024-09-21T11:05:09.140 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0988000000000001e-05 2024-09-21T11:05:09.140 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.140 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.140 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:05:09.140 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.839439+0000", 2024-09-21T11:05:09.141 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1789999999999999e-06 2024-09-21T11:05:09.141 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.141 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.141 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.141 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.839475+0000", 2024-09-21T11:05:09.141 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6473999999999997e-05 2024-09-21T11:05:09.141 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.141 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.141 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:05:09.141 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.839605+0000", 2024-09-21T11:05:09.141 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013014100000000001 2024-09-21T11:05:09.141 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.141 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.141 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3", 2024-09-21T11:05:09.142 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.839767+0000", 2024-09-21T11:05:09.142 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000161819 2024-09-21T11:05:09.142 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.142 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.142 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:05:09.142 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.841523+0000", 2024-09-21T11:05:09.142 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0017561409999999999 2024-09-21T11:05:09.142 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.142 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.142 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.142 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.914178+0000", 2024-09-21T11:05:09.142 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.072655090000000006 2024-09-21T11:05:09.142 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.142 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.143 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:05:09.143 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.914236+0000", 2024-09-21T11:05:09.143 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7503999999999997e-05 2024-09-21T11:05:09.143 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.143 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.143 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:05:09.143 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.914317+0000", 2024-09-21T11:05:09.143 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1404999999999997e-05 2024-09-21T11:05:09.143 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.143 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:05:09.143 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.143 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.143 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.143 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:217099 58.a 58:53db01d7:::benchmark_data_smithi070_396992_object71140:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1603)", 2024-09-21T11:05:09.143 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:31.711874+0000", 2024-09-21T11:05:09.144 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 37.348278129000001, 2024-09-21T11:05:09.144 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.058318542000000001, 2024-09-21T11:05:09.144 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:05:09.144 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:05:09.144 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:05:09.144 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:05:09.144 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:05:09.144 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 217099 2024-09-21T11:05:09.144 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.144 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:05:09.144 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.144 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:05:09.144 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.711874+0000", 2024-09-21T11:05:09.144 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.144 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.145 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.145 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:05:09.145 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.711874+0000", 2024-09-21T11:05:09.145 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.145 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.145 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.145 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:05:09.145 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.711873+0000", 2024-09-21T11:05:09.145 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:05:09.145 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.145 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.145 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:05:09.145 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.711877+0000", 2024-09-21T11:05:09.145 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.16e-06 2024-09-21T11:05:09.145 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.146 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.146 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:05:09.146 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.711884+0000", 2024-09-21T11:05:09.146 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3189999999999997e-06 2024-09-21T11:05:09.146 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.146 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.146 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:05:09.146 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.711888+0000", 2024-09-21T11:05:09.146 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8700000000000002e-06 2024-09-21T11:05:09.146 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.146 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.146 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.146 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.711933+0000", 2024-09-21T11:05:09.146 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5052999999999997e-05 2024-09-21T11:05:09.146 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.147 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.147 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:05:09.147 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.712063+0000", 2024-09-21T11:05:09.147 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013026800000000001 2024-09-21T11:05:09.147 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.147 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.147 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", 2024-09-21T11:05:09.147 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.712193+0000", 2024-09-21T11:05:09.147 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000129558 2024-09-21T11:05:09.147 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.147 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.147 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:05:09.147 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.714291+0000", 2024-09-21T11:05:09.147 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0020982000000000002 2024-09-21T11:05:09.147 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.148 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.148 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.148 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.714820+0000", 2024-09-21T11:05:09.148 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00052909699999999997 2024-09-21T11:05:09.148 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.148 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.148 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.148 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.770044+0000", 2024-09-21T11:05:09.148 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.055223758999999997 2024-09-21T11:05:09.148 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.148 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.148 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:05:09.148 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.770093+0000", 2024-09-21T11:05:09.148 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9041e-05 2024-09-21T11:05:09.149 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.149 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.149 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:05:09.149 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.770192+0000", 2024-09-21T11:05:09.149 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9366999999999999e-05 2024-09-21T11:05:09.149 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.149 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:05:09.149 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.149 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.149 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.149 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:217103 58.a 58:54e92bc7:::benchmark_data_smithi070_396992_object71144:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1603)", 2024-09-21T11:05:09.149 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:31.714814+0000", 2024-09-21T11:05:09.149 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 37.345338259999998, 2024-09-21T11:05:09.149 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.064414930999999995, 2024-09-21T11:05:09.149 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:05:09.150 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:05:09.150 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:05:09.150 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:05:09.150 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:05:09.150 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 217103 2024-09-21T11:05:09.150 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.150 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:05:09.150 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.150 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:05:09.150 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.714814+0000", 2024-09-21T11:05:09.150 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.150 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.151 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.151 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:05:09.151 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.714814+0000", 2024-09-21T11:05:09.151 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.151 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.151 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:05:09.151 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.714812+0000", 2024-09-21T11:05:09.151 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:05:09.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.151 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.151 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:05:09.151 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.714819+0000", 2024-09-21T11:05:09.152 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5139999999999998e-06 2024-09-21T11:05:09.152 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.152 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.152 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:05:09.152 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.714833+0000", 2024-09-21T11:05:09.152 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.38e-05 2024-09-21T11:05:09.152 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.152 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.152 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:05:09.152 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.714841+0000", 2024-09-21T11:05:09.152 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.96e-06 2024-09-21T11:05:09.152 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.152 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.152 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.153 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.714870+0000", 2024-09-21T11:05:09.153 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9332999999999999e-05 2024-09-21T11:05:09.153 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.153 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.153 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:05:09.153 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.714949+0000", 2024-09-21T11:05:09.153 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8529000000000007e-05 2024-09-21T11:05:09.153 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.153 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.153 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", 2024-09-21T11:05:09.153 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.715062+0000", 2024-09-21T11:05:09.153 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011300500000000001 2024-09-21T11:05:09.153 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.153 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.153 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.153 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.770234+0000", 2024-09-21T11:05:09.154 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.055172713999999998 2024-09-21T11:05:09.154 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.154 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.154 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:05:09.154 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.770252+0000", 2024-09-21T11:05:09.154 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7567e-05 2024-09-21T11:05:09.154 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.154 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.154 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.154 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.779013+0000", 2024-09-21T11:05:09.154 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0087616250000000003 2024-09-21T11:05:09.154 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.154 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.155 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:05:09.155 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.779137+0000", 2024-09-21T11:05:09.155 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000124022 2024-09-21T11:05:09.155 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.155 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.155 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:05:09.155 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.779229+0000", 2024-09-21T11:05:09.155 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1111000000000002e-05 2024-09-21T11:05:09.155 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.155 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:05:09.155 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.155 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.155 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.155 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:237720 58.a 58:58d18b95:::benchmark_data_smithi070_396992_object91761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1618)", 2024-09-21T11:05:09.156 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:56.869368+0000", 2024-09-21T11:05:09.156 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 12.19078341, 2024-09-21T11:05:09.156 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060675688999999998, 2024-09-21T11:05:09.156 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:05:09.156 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:05:09.156 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:05:09.156 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:05:09.156 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:05:09.156 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 237720 2024-09-21T11:05:09.156 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.156 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:05:09.156 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.156 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:05:09.156 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:56.869368+0000", 2024-09-21T11:05:09.156 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.157 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.157 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.157 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:05:09.157 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:56.869368+0000", 2024-09-21T11:05:09.157 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.157 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.157 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.157 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:05:09.157 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:56.869367+0000", 2024-09-21T11:05:09.157 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:05:09.157 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.157 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.157 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:05:09.158 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:56.869373+0000", 2024-09-21T11:05:09.158 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8579999999999996e-06 2024-09-21T11:05:09.158 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.158 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.158 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:05:09.158 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:56.869383+0000", 2024-09-21T11:05:09.158 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0328e-05 2024-09-21T11:05:09.158 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.158 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.158 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:05:09.158 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:56.869389+0000", 2024-09-21T11:05:09.158 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2709999999999998e-06 2024-09-21T11:05:09.158 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.158 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.159 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.159 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:56.869438+0000", 2024-09-21T11:05:09.159 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8507000000000002e-05 2024-09-21T11:05:09.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.159 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.159 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:05:09.159 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:56.869622+0000", 2024-09-21T11:05:09.159 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000184563 2024-09-21T11:05:09.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.159 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.159 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", 2024-09-21T11:05:09.159 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:56.872470+0000", 2024-09-21T11:05:09.159 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0028472279999999998 2024-09-21T11:05:09.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.160 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.160 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.160 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:56.876191+0000", 2024-09-21T11:05:09.160 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0037213239999999998 2024-09-21T11:05:09.160 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.160 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.160 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:05:09.160 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:56.876858+0000", 2024-09-21T11:05:09.160 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00066745900000000002 2024-09-21T11:05:09.160 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.160 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.160 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.160 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:56.929896+0000", 2024-09-21T11:05:09.160 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.053037372999999999 2024-09-21T11:05:09.160 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.161 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.161 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:05:09.161 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:56.929951+0000", 2024-09-21T11:05:09.161 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4783e-05 2024-09-21T11:05:09.161 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.161 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.161 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:05:09.161 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:56.930044+0000", 2024-09-21T11:05:09.161 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3591000000000006e-05 2024-09-21T11:05:09.161 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.161 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:05:09.161 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.161 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.161 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.162 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:241152 58.a 58:5494570e:::benchmark_data_smithi070_396992_object95193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1619)", 2024-09-21T11:05:09.162 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:00.365098+0000", 2024-09-21T11:05:09.162 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 8.6950539849999995, 2024-09-21T11:05:09.162 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.077856499999999995, 2024-09-21T11:05:09.162 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:05:09.162 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:05:09.162 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:05:09.162 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:05:09.162 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:05:09.162 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 241152 2024-09-21T11:05:09.162 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.162 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:05:09.162 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.163 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:05:09.163 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.365098+0000", 2024-09-21T11:05:09.163 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.163 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.163 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.163 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:05:09.164 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:05:09.164 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:05:09.165 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.365098+0000", 2024-09-21T11:05:09.165 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.165 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.165 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.165 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:05:09.165 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.365096+0000", 2024-09-21T11:05:09.165 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:05:09.165 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.165 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.165 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:05:09.165 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.365103+0000", 2024-09-21T11:05:09.165 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.443e-06 2024-09-21T11:05:09.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.166 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.166 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:05:09.166 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.365116+0000", 2024-09-21T11:05:09.166 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3023000000000001e-05 2024-09-21T11:05:09.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.166 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.166 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:05:09.166 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.365123+0000", 2024-09-21T11:05:09.166 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3429999999999998e-06 2024-09-21T11:05:09.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.166 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.166 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.167 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.365486+0000", 2024-09-21T11:05:09.167 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000362987 2024-09-21T11:05:09.167 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.167 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.167 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:05:09.167 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.368785+0000", 2024-09-21T11:05:09.167 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0032987139999999999 2024-09-21T11:05:09.167 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.167 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.167 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", 2024-09-21T11:05:09.167 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.369328+0000", 2024-09-21T11:05:09.167 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00054261299999999995 2024-09-21T11:05:09.167 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.167 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.167 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.168 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.375700+0000", 2024-09-21T11:05:09.168 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0063721130000000004 2024-09-21T11:05:09.168 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.168 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.168 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:05:09.168 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.375708+0000", 2024-09-21T11:05:09.168 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8189999999999998e-06 2024-09-21T11:05:09.168 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.168 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.168 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.168 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.442771+0000", 2024-09-21T11:05:09.168 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.067062347999999994 2024-09-21T11:05:09.168 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.168 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.168 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:05:09.168 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.442856+0000", 2024-09-21T11:05:09.169 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5641000000000002e-05 2024-09-21T11:05:09.169 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.169 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.169 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:05:09.169 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.442954+0000", 2024-09-21T11:05:09.169 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7973999999999994e-05 2024-09-21T11:05:09.169 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.169 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:05:09.169 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.169 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.169 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.169 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:241961 58.a 58:5900b4a9:::benchmark_data_smithi070_396992_object96002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1619)", 2024-09-21T11:05:09.169 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:01.183748+0000", 2024-09-21T11:05:09.169 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 7.8764043680000002, 2024-09-21T11:05:09.170 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.042916202000000001, 2024-09-21T11:05:09.170 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:05:09.170 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:05:09.170 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:05:09.170 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:05:09.170 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:05:09.170 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 241961 2024-09-21T11:05:09.170 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.170 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:05:09.170 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.170 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:05:09.170 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.183748+0000", 2024-09-21T11:05:09.170 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.170 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.170 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.171 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:05:09.171 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.183748+0000", 2024-09-21T11:05:09.171 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.171 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.171 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.171 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:05:09.171 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.183746+0000", 2024-09-21T11:05:09.171 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:05:09.171 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.171 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.171 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:05:09.171 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.183753+0000", 2024-09-21T11:05:09.171 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.576e-06 2024-09-21T11:05:09.171 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.171 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.172 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:05:09.172 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.183773+0000", 2024-09-21T11:05:09.172 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9575000000000001e-05 2024-09-21T11:05:09.172 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.172 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.172 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:05:09.172 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.183780+0000", 2024-09-21T11:05:09.172 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0489999999999998e-06 2024-09-21T11:05:09.172 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.172 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.172 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.172 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.184020+0000", 2024-09-21T11:05:09.172 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023945800000000001 2024-09-21T11:05:09.172 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.173 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.173 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:05:09.173 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.184148+0000", 2024-09-21T11:05:09.173 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012879300000000001 2024-09-21T11:05:09.173 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.173 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.173 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", 2024-09-21T11:05:09.173 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.184270+0000", 2024-09-21T11:05:09.173 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000121647 2024-09-21T11:05:09.173 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.173 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.173 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:05:09.173 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.187430+0000", 2024-09-21T11:05:09.173 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003159963 2024-09-21T11:05:09.173 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.173 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.174 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.174 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.192045+0000", 2024-09-21T11:05:09.174 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0046145969999999998 2024-09-21T11:05:09.174 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.174 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.174 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.174 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.225461+0000", 2024-09-21T11:05:09.174 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.033416135999999999 2024-09-21T11:05:09.174 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.174 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.174 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:05:09.174 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.225924+0000", 2024-09-21T11:05:09.174 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00046339800000000001 2024-09-21T11:05:09.174 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.174 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.175 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:05:09.175 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.226664+0000", 2024-09-21T11:05:09.175 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00073965100000000002 2024-09-21T11:05:09.175 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.175 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:05:09.175 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.175 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.175 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.175 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:241962 58.a 58:5b72528f:::benchmark_data_smithi070_396992_object96003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1619)", 2024-09-21T11:05:09.175 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:01.183960+0000", 2024-09-21T11:05:09.175 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 7.8761915330000001, 2024-09-21T11:05:09.175 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.042952921999999998, 2024-09-21T11:05:09.175 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:05:09.175 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:05:09.175 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:05:09.176 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:05:09.176 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:05:09.176 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 241962 2024-09-21T11:05:09.176 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.176 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:05:09.176 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.176 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:05:09.176 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.183960+0000", 2024-09-21T11:05:09.176 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.176 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.176 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.176 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:05:09.176 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.183960+0000", 2024-09-21T11:05:09.176 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.176 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.177 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.177 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:05:09.177 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.183960+0000", 2024-09-21T11:05:09.177 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:05:09.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.177 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.177 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:05:09.177 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.183965+0000", 2024-09-21T11:05:09.177 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0860000000000001e-06 2024-09-21T11:05:09.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.177 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.177 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:05:09.177 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.183974+0000", 2024-09-21T11:05:09.177 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5650000000000007e-06 2024-09-21T11:05:09.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.177 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.178 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:05:09.178 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.183982+0000", 2024-09-21T11:05:09.178 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6380000000000004e-06 2024-09-21T11:05:09.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.178 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.178 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.178 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.187692+0000", 2024-09-21T11:05:09.178 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0037103399999999999 2024-09-21T11:05:09.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.178 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.178 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:05:09.178 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.188027+0000", 2024-09-21T11:05:09.178 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00033459899999999999 2024-09-21T11:05:09.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.179 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", 2024-09-21T11:05:09.179 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.188415+0000", 2024-09-21T11:05:09.179 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00038862 2024-09-21T11:05:09.179 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.179 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.179 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.192070+0000", 2024-09-21T11:05:09.179 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0036548689999999998 2024-09-21T11:05:09.179 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.179 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:05:09.179 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.192092+0000", 2024-09-21T11:05:09.179 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2135000000000001e-05 2024-09-21T11:05:09.179 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.180 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.180 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.180 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.226778+0000", 2024-09-21T11:05:09.180 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.034685894000000002 2024-09-21T11:05:09.180 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.180 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.180 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:05:09.180 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.226825+0000", 2024-09-21T11:05:09.180 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7110999999999997e-05 2024-09-21T11:05:09.180 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.180 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.180 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:05:09.180 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:01.226913+0000", 2024-09-21T11:05:09.180 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7865e-05 2024-09-21T11:05:09.180 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.181 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:05:09.181 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.181 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.181 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:244080 58.a 58:5105e019:::benchmark_data_smithi070_396992_object98121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1619)", 2024-09-21T11:05:09.181 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:03.220591+0000", 2024-09-21T11:05:09.181 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 5.8395613559999999, 2024-09-21T11:05:09.181 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.064642745000000001, 2024-09-21T11:05:09.181 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:05:09.181 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:05:09.181 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:05:09.181 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:05:09.181 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:05:09.182 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 244080 2024-09-21T11:05:09.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.182 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:05:09.182 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.182 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:05:09.182 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.220591+0000", 2024-09-21T11:05:09.182 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.182 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.182 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:05:09.182 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.220591+0000", 2024-09-21T11:05:09.182 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.182 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.182 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:05:09.182 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.220589+0000", 2024-09-21T11:05:09.183 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T11:05:09.183 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.183 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.183 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:05:09.183 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.220597+0000", 2024-09-21T11:05:09.183 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9440000000000005e-06 2024-09-21T11:05:09.183 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.183 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.183 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:05:09.183 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.220612+0000", 2024-09-21T11:05:09.183 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.537e-05 2024-09-21T11:05:09.183 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.183 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.183 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:05:09.183 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.220619+0000", 2024-09-21T11:05:09.184 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.294e-06 2024-09-21T11:05:09.184 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.184 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.184 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.184 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.220668+0000", 2024-09-21T11:05:09.184 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8690000000000003e-05 2024-09-21T11:05:09.184 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.184 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.184 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:05:09.184 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.220972+0000", 2024-09-21T11:05:09.184 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00030425500000000002 2024-09-21T11:05:09.184 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.184 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.184 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", 2024-09-21T11:05:09.185 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.221314+0000", 2024-09-21T11:05:09.185 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034205699999999999 2024-09-21T11:05:09.185 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.185 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.185 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.185 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.226865+0000", 2024-09-21T11:05:09.185 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0055506540000000004 2024-09-21T11:05:09.185 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.185 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.185 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:05:09.185 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.228071+0000", 2024-09-21T11:05:09.185 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00120634 2024-09-21T11:05:09.185 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.185 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.185 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.185 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.285083+0000", 2024-09-21T11:05:09.186 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.057011586000000003 2024-09-21T11:05:09.186 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.186 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.186 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:05:09.186 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.285132+0000", 2024-09-21T11:05:09.186 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9141999999999998e-05 2024-09-21T11:05:09.186 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.186 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.186 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:05:09.186 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.285233+0000", 2024-09-21T11:05:09.186 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000101087 2024-09-21T11:05:09.186 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.186 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:05:09.186 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.187 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.187 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.187 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:244085 58.a 58:5529bd05:::benchmark_data_smithi070_396992_object98126:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1619)", 2024-09-21T11:05:09.187 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:03.229366+0000", 2024-09-21T11:05:09.187 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 5.8307860290000004, 2024-09-21T11:05:09.187 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.063603385999999998, 2024-09-21T11:05:09.187 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:05:09.187 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:05:09.187 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:05:09.187 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:05:09.187 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:05:09.187 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 244085 2024-09-21T11:05:09.187 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.187 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:05:09.188 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.188 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:05:09.188 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229366+0000", 2024-09-21T11:05:09.188 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.188 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.188 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.188 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:05:09.188 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229366+0000", 2024-09-21T11:05:09.188 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.188 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.188 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.188 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:05:09.188 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229364+0000", 2024-09-21T11:05:09.188 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T11:05:09.188 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.189 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.189 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:05:09.189 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229372+0000", 2024-09-21T11:05:09.189 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5939999999999996e-06 2024-09-21T11:05:09.189 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.189 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.189 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:05:09.189 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229391+0000", 2024-09-21T11:05:09.189 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9494999999999999e-05 2024-09-21T11:05:09.189 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.189 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.189 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:05:09.189 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229399+0000", 2024-09-21T11:05:09.189 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8730000000000007e-06 2024-09-21T11:05:09.190 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.190 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.190 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.190 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229473+0000", 2024-09-21T11:05:09.190 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4270000000000006e-05 2024-09-21T11:05:09.190 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.190 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.190 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:05:09.190 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229590+0000", 2024-09-21T11:05:09.190 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011636700000000001 2024-09-21T11:05:09.190 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.190 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.190 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", 2024-09-21T11:05:09.191 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229740+0000", 2024-09-21T11:05:09.191 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000150118 2024-09-21T11:05:09.191 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.191 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.191 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:05:09.191 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.236236+0000", 2024-09-21T11:05:09.191 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0064965609999999997 2024-09-21T11:05:09.191 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.191 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.191 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.191 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.237063+0000", 2024-09-21T11:05:09.191 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00082698600000000004 2024-09-21T11:05:09.191 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.192 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.192 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.192 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.292822+0000", 2024-09-21T11:05:09.192 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.055758421000000002 2024-09-21T11:05:09.192 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.192 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.192 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:05:09.192 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.292876+0000", 2024-09-21T11:05:09.192 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3973e-05 2024-09-21T11:05:09.192 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.192 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.192 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:05:09.192 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.292969+0000", 2024-09-21T11:05:09.192 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3496000000000001e-05 2024-09-21T11:05:09.192 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.193 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:05:09.193 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.193 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.193 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.193 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:244088 58.a 58:56c2f617:::benchmark_data_smithi070_396992_object98129:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1619)", 2024-09-21T11:05:09.193 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:03.229413+0000", 2024-09-21T11:05:09.193 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 5.8307388610000004, 2024-09-21T11:05:09.193 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.065123039999999993, 2024-09-21T11:05:09.193 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:05:09.193 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:05:09.193 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:05:09.193 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:05:09.193 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:05:09.194 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 244088 2024-09-21T11:05:09.194 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.194 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:05:09.194 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.194 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:05:09.194 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229413+0000", 2024-09-21T11:05:09.194 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.194 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.194 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.194 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:05:09.194 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229413+0000", 2024-09-21T11:05:09.194 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.194 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.194 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.195 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:05:09.195 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229413+0000", 2024-09-21T11:05:09.195 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T11:05:09.195 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.195 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.195 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:05:09.195 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229416+0000", 2024-09-21T11:05:09.195 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5829999999999998e-06 2024-09-21T11:05:09.195 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.195 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.195 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:05:09.195 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229423+0000", 2024-09-21T11:05:09.195 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.199e-06 2024-09-21T11:05:09.195 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.195 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.196 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:05:09.196 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229428+0000", 2024-09-21T11:05:09.196 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1229999999999999e-06 2024-09-21T11:05:09.196 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.196 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.196 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.196 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.232821+0000", 2024-09-21T11:05:09.196 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0033924860000000001 2024-09-21T11:05:09.196 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.196 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.196 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:05:09.196 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.232951+0000", 2024-09-21T11:05:09.196 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013008799999999999 2024-09-21T11:05:09.196 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.197 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.197 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", 2024-09-21T11:05:09.197 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.233071+0000", 2024-09-21T11:05:09.197 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011995899999999999 2024-09-21T11:05:09.197 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.197 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.197 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:05:09.197 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.236467+0000", 2024-09-21T11:05:09.197 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003395595 2024-09-21T11:05:09.197 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.197 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.197 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.197 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.237169+0000", 2024-09-21T11:05:09.197 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00070235399999999998 2024-09-21T11:05:09.197 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.197 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.198 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.198 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.294371+0000", 2024-09-21T11:05:09.198 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.057201895000000003 2024-09-21T11:05:09.198 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.198 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.198 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:05:09.198 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.294428+0000", 2024-09-21T11:05:09.198 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6697999999999998e-05 2024-09-21T11:05:09.198 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.198 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.198 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:05:09.198 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.294536+0000", 2024-09-21T11:05:09.198 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010857 2024-09-21T11:05:09.198 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.199 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:05:09.199 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.199 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.199 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.199 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:249275 58.a 58:5383d281:::benchmark_data_smithi070_396992_object103316:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1621)", 2024-09-21T11:05:09.199 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:08.997720+0000", 2024-09-21T11:05:09.199 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.062432255999999998, 2024-09-21T11:05:09.199 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.042693738000000002, 2024-09-21T11:05:09.199 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:05:09.199 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:05:09.199 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:05:09.199 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:05:09.199 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:05:09.199 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 249275 2024-09-21T11:05:09.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.200 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:05:09.200 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.200 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:05:09.200 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:08.997720+0000", 2024-09-21T11:05:09.200 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.200 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.200 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:05:09.200 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:08.997720+0000", 2024-09-21T11:05:09.200 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:05:09.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.200 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.200 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:05:09.200 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:08.997718+0000", 2024-09-21T11:05:09.201 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:05:09.201 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.201 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.201 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:05:09.201 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:08.997725+0000", 2024-09-21T11:05:09.201 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1230000000000002e-06 2024-09-21T11:05:09.201 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.201 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.201 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:05:09.201 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:08.997741+0000", 2024-09-21T11:05:09.201 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5858e-05 2024-09-21T11:05:09.201 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.201 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.201 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:05:09.202 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:08.997749+0000", 2024-09-21T11:05:09.202 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5700000000000004e-06 2024-09-21T11:05:09.202 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.202 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.202 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:05:09.202 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:09.020818+0000", 2024-09-21T11:05:09.202 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.023069138999999999 2024-09-21T11:05:09.202 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.202 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.202 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:05:09.202 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:09.020937+0000", 2024-09-21T11:05:09.202 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119481 2024-09-21T11:05:09.202 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.203 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.203 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", 2024-09-21T11:05:09.203 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:09.021051+0000", 2024-09-21T11:05:09.203 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011385 2024-09-21T11:05:09.203 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.203 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.203 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.203 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:09.027539+0000", 2024-09-21T11:05:09.203 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0064874540000000001 2024-09-21T11:05:09.203 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.203 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.203 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:05:09.203 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:09.028890+0000", 2024-09-21T11:05:09.203 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001351489 2024-09-21T11:05:09.203 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.204 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.204 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:05:09.204 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:09.040267+0000", 2024-09-21T11:05:09.204 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011377257999999999 2024-09-21T11:05:09.204 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.204 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.204 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:05:09.204 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:09.040318+0000", 2024-09-21T11:05:09.204 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0407e-05 2024-09-21T11:05:09.204 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:05:09.204 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:05:09.204 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:05:09.204 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:09.040413+0000", 2024-09-21T11:05:09.204 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5650999999999996e-05 2024-09-21T11:05:09.204 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.205 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:05:09.205 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.205 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:05:09.205 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:05:09.205 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T11:05:09.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:09.206 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:09.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:09.206 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T11:05:09.206 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T11:05:09.207 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T11:05:09.207 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T11:05:09.219 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:05:09.219 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:05:09.307 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:09.341 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:09.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:09.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:09.443 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:09.472 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:09.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:09.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:09.574 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:09.604 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:09.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:09.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:09.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:09 smithi070 ceph-mon[107385]: pgmap v3209: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 2.8 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 881 op/s; 25060/139512 objects degraded (17.963%); 38961/139512 objects misplaced (27.927%); 6.7 MiB/s, 134 objects/s recovering 2024-09-21T11:05:09.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:09 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 25060/139512 objects degraded (17.963%), 9 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T11:05:09.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:09 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2704229167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:05:09.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:09 smithi070 ceph-mon[112139]: pgmap v3209: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 2.8 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 881 op/s; 25060/139512 objects degraded (17.963%); 38961/139512 objects misplaced (27.927%); 6.7 MiB/s, 134 objects/s recovering 2024-09-21T11:05:09.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:09 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 25060/139512 objects degraded (17.963%), 9 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T11:05:09.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:09 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2704229167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:05:09.706 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:09.722 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:09 smithi160 ceph-mon[101879]: pgmap v3209: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 2.8 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 881 op/s; 25060/139512 objects degraded (17.963%); 38961/139512 objects misplaced (27.927%); 6.7 MiB/s, 134 objects/s recovering 2024-09-21T11:05:09.722 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:09 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 25060/139512 objects degraded (17.963%), 9 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T11:05:09.722 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:09 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2704229167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:05:09.735 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:09.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:09.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:09.836 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:09.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:09.870 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:09.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:09.971 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:10.005 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:10.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:10.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:10.106 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:10.147 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:10.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:10.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:10.249 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:10.277 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T11:05:10.295 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:10.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:10.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:10.397 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:10.447 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:10.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:10.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:10.549 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:10.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:10.611 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:10.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:10.664 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:05:10.712 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:10.767 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:10.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:10.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:10.869 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:10.913 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:10.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:10.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:11.015 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:11.053 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:11.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:11.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:11.155 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:11.195 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:11.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:11.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:11.297 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:11.324 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:11.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:11.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:11.426 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:11.467 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:11.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:11.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:11.569 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:11.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:11 smithi160 ceph-mon[101879]: pgmap v3210: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 2.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 825 op/s; 23332/132543 objects degraded (17.603%); 37339/132543 objects misplaced (28.171%); 11 MiB/s, 205 objects/s recovering 2024-09-21T11:05:11.607 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:11.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:11.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:11.701 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:11 smithi070 ceph-mon[107385]: pgmap v3210: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 2.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 825 op/s; 23332/132543 objects degraded (17.603%); 37339/132543 objects misplaced (28.171%); 11 MiB/s, 205 objects/s recovering 2024-09-21T11:05:11.702 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:11 smithi070 ceph-mon[112139]: pgmap v3210: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 2.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 825 op/s; 23332/132543 objects degraded (17.603%); 37339/132543 objects misplaced (28.171%); 11 MiB/s, 205 objects/s recovering 2024-09-21T11:05:11.709 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:11.744 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:11.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:11.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:11.846 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:11.878 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:11.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:11.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:11.980 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:12.011 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:12.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:12.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:12.113 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:12.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:12.153 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:12.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:12.254 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:12.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:12.296 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:12.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:12.397 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:12.438 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T11:05:12.442 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:12.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:12.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:12.505 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:12 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3917764160' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:05:12.505 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:12 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:05:12.545 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:12.572 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:12.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:12.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:12.590 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T11:05:12.674 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:12.726 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:12.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:12.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:12.827 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:12.875 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:12.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:12.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:12.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:12 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3917764160' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:05:12.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:12 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:05:12.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:12 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3917764160' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:05:12.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:12 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:05:12.978 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:13.006 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:13.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:13.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:13.108 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:13.135 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:13.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:13.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:13.237 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:13.274 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:13.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:13.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:13.376 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:13.417 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:13.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:13.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:13.424 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:05:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:05:13] "GET /metrics HTTP/1.1" 200 36747 "" "Prometheus/2.51.0" 2024-09-21T11:05:13.446 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7, 4] out_osds: [2, 5, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 7, 6, 5] 2024-09-21T11:05:13.446 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T11:05:13.446 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-21T11:05:13.446 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T11:05:13.446 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T11:05:13.446 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-21T11:05:13.458 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T11:05:13.458 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T11:05:13.519 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:13.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:13.554 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:13.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:13.655 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:13.686 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:13.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:13.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:13.788 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:13.818 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:13.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:13.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:13 smithi160 ceph-mon[101879]: pgmap v3211: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 2.5 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 21712/123471 objects degraded (17.585%); 34510/123471 objects misplaced (27.950%); 10 MiB/s, 183 objects/s recovering 2024-09-21T11:05:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:13 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T11:05:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:13 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T11:05:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:13 smithi160 ceph-mon[101879]: osdmap e1622: 8 total, 8 up, 5 in 2024-09-21T11:05:13.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:13 smithi070 ceph-mon[112139]: pgmap v3211: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 2.5 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 21712/123471 objects degraded (17.585%); 34510/123471 objects misplaced (27.950%); 10 MiB/s, 183 objects/s recovering 2024-09-21T11:05:13.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:13 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T11:05:13.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:13 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T11:05:13.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:13 smithi070 ceph-mon[112139]: osdmap e1622: 8 total, 8 up, 5 in 2024-09-21T11:05:13.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:13 smithi070 ceph-mon[107385]: pgmap v3211: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 2.5 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 1.08k op/s; 21712/123471 objects degraded (17.585%); 34510/123471 objects misplaced (27.950%); 10 MiB/s, 183 objects/s recovering 2024-09-21T11:05:13.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:13 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T11:05:13.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:13 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T11:05:13.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:13 smithi070 ceph-mon[107385]: osdmap e1622: 8 total, 8 up, 5 in 2024-09-21T11:05:13.920 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:13.960 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:13.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:13.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:14.062 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:14.107 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:14.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:14.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:14.209 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:14.239 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:14.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:14.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:14.340 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:14.365 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:14.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:14.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:14.467 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:14.507 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:14.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:14.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:14.609 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:14.632 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:14.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:14.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:14.734 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:14.762 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:14.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:14.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:14.831 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:14 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 21712/123471 objects degraded (17.585%), 9 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T11:05:14.864 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:14.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:14 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 21712/123471 objects degraded (17.585%), 9 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T11:05:14.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:14 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 21712/123471 objects degraded (17.585%), 9 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T11:05:14.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:14.891 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:14.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:14.993 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:15.031 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:15.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:15.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:15.124 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:05:15.133 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:15.164 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:15.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:15.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:15.231 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T11:05:15.256 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:05:15.266 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:15.296 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:15.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:15.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:15.397 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:15.457 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:15 smithi160 ceph-mon[101879]: pgmap v3213: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 2.5 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 968 op/s; 21712/123471 objects degraded (17.585%); 34510/123471 objects misplaced (27.950%); 9.0 MiB/s, 165 objects/s recovering 2024-09-21T11:05:15.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:15.461 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:15.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:15.561 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:15 smithi070 ceph-mon[107385]: pgmap v3213: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 2.5 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 968 op/s; 21712/123471 objects degraded (17.585%); 34510/123471 objects misplaced (27.950%); 9.0 MiB/s, 165 objects/s recovering 2024-09-21T11:05:15.561 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:15 smithi070 ceph-mon[112139]: pgmap v3213: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 2.5 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 968 op/s; 21712/123471 objects degraded (17.585%); 34510/123471 objects misplaced (27.950%); 9.0 MiB/s, 165 objects/s recovering 2024-09-21T11:05:15.562 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:15.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:15.629 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:15.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:15.731 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:15.758 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:15.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:15.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:15.859 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:15.902 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:15.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:15.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:16.004 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:16.045 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:16.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:16.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:16.146 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:16.169 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:05:16.173 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:16.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:16.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:16.276 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:16.301 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:16.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:16.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:16.404 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:16.445 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:16.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:16.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:16.547 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:16.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:16.584 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:16.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:16.685 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:16.713 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:16.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:16.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:16.815 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:16.841 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:16.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:16.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:16.943 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:16.971 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T11:05:16.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:16.983 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:16.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:17.084 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:17.099 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:05:17.100 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:05:17.123 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:17.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:17.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:17.137 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:05:17.138 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:05:17.170 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:05:17.171 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:05:17.202 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:05:17.202 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:05:17.225 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:17.248 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:05:17.248 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:05:17.266 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:17.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:17.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:17.278 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:05:17.280 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T11:05:17.280 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T11:05:17.319 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T11:05:17.320 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T11:05:17.350 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:05:17.351 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:05:17.368 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:17.401 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:17.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:17.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:17.502 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:17.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:17.549 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:17.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:17.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:17 smithi070 ceph-mon[107385]: pgmap v3214: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 2.2 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 1.02k op/s; 18698/108945 objects degraded (17.163%); 30213/108945 objects misplaced (27.732%); 7.2 MiB/s, 115 objects/s recovering 2024-09-21T11:05:17.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:17 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2191130555' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:05:17.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:17 smithi070 ceph-mon[112139]: pgmap v3214: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 2.2 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 1.02k op/s; 18698/108945 objects degraded (17.163%); 30213/108945 objects misplaced (27.732%); 7.2 MiB/s, 115 objects/s recovering 2024-09-21T11:05:17.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:17 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2191130555' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:05:17.650 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:17.693 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:17.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:17.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:17.794 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:17.831 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:17.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:17.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:17.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:17 smithi160 ceph-mon[101879]: pgmap v3214: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 2.2 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 1.02k op/s; 18698/108945 objects degraded (17.163%); 30213/108945 objects misplaced (27.732%); 7.2 MiB/s, 115 objects/s recovering 2024-09-21T11:05:17.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:17 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2191130555' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:05:17.934 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:17.976 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:17.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:17.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:18.078 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:18.118 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:18.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:18.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:18.220 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:18.253 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:18.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:18.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:18.355 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:18.384 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:18.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:18.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:18.486 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:18.512 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:18.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:18.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:18.614 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:18.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:18.660 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:18.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:18.761 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:18.800 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:18.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:18.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:18.902 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:18.929 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:18.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:18.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:19.031 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:19.070 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:19.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:19.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:19.172 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:19.221 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:19.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:19.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:19.322 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:19.361 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:19.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:19.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:19.463 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:19.491 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:19.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:19.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:19.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:19 smithi160 ceph-mon[101879]: pgmap v3215: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 2.2 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 1.05k op/s; 18422/108117 objects degraded (17.039%); 30213/108117 objects misplaced (27.945%); 7.2 MiB/s, 115 objects/s recovering 2024-09-21T11:05:19.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:19 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 18422/108117 objects degraded (17.039%), 9 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T11:05:19.594 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:19.635 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:19.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:19.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:19.737 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:19.768 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:19.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:19.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:19.777 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:19 smithi070 ceph-mon[107385]: pgmap v3215: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 2.2 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 1.05k op/s; 18422/108117 objects degraded (17.039%); 30213/108117 objects misplaced (27.945%); 7.2 MiB/s, 115 objects/s recovering 2024-09-21T11:05:19.777 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:19 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 18422/108117 objects degraded (17.039%), 9 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T11:05:19.777 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:19 smithi070 ceph-mon[112139]: pgmap v3215: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 2.2 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 1.05k op/s; 18422/108117 objects degraded (17.039%); 30213/108117 objects misplaced (27.945%); 7.2 MiB/s, 115 objects/s recovering 2024-09-21T11:05:19.777 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:19 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 18422/108117 objects degraded (17.039%), 9 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T11:05:19.870 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:19.902 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:19.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:19.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:20.004 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:20.044 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:20.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:20.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:20.145 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:20.175 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:20.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:20.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:20.276 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:20.315 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:20.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:20.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:20.417 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:20.449 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:20.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:20.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:20.500 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T11:05:20.500 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T11:05:20.500 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T11:05:20.501 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T11:05:20.501 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T11:05:20.551 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:20.578 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:20.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:20.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:20.679 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:20.719 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:20.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:20.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:20.821 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:20.860 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:20.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:20.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:20.962 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:20.999 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:21.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:21.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:21.101 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:21.136 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:21.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:21.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:21.238 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:21.267 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:21.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:21.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:21.368 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:21.413 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:21.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:21.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:21.514 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:21.519 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T11:05:21.547 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:21 smithi070 ceph-mon[107385]: pgmap v3216: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 2.1 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 1.03k op/s; 16846/101667 objects degraded (16.570%); 28707/101667 objects misplaced (28.236%); 7.6 MiB/s, 121 objects/s recovering 2024-09-21T11:05:21.547 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:05:21.547 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:21 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:05:21.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:21 smithi070 ceph-mon[112139]: pgmap v3216: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 2.1 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 1.03k op/s; 16846/101667 objects degraded (16.570%); 28707/101667 objects misplaced (28.236%); 7.6 MiB/s, 121 objects/s recovering 2024-09-21T11:05:21.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:05:21.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:21 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:05:21.554 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:21.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:21.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:21.656 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:21.693 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:21.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:21.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:21.794 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:21.802 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:05:21.832 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:21.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:21.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:21 smithi160 ceph-mon[101879]: pgmap v3216: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 2.1 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 1.03k op/s; 16846/101667 objects degraded (16.570%); 28707/101667 objects misplaced (28.236%); 7.6 MiB/s, 121 objects/s recovering 2024-09-21T11:05:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:05:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:21 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:05:21.934 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:21.964 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:21.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:21.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:22.066 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:22.091 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:22.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:22.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:22.192 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:22.226 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:22.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:22.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:22.328 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:22.355 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:22.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:22.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:22.409 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T11:05:22.457 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:22.483 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:22.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:22.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:22.584 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:22.622 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:22.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:22.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:22.703 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:22 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1983311528' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T11:05:22.703 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:22 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1983311528' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T11:05:22.724 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:22.753 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:22.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:22.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:22 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1983311528' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T11:05:22.855 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:22.888 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:22.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:22.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:22.990 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:23.025 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:23.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:23.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:23.127 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:23.163 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:23.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:23.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:23.265 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:23.302 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:23.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:23.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:23.403 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:23.431 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:23.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:23.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:23.533 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:23.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:23.559 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:23.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:23.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:05:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:05:23] "GET /metrics HTTP/1.1" 200 36755 "" "Prometheus/2.51.0" 2024-09-21T11:05:23.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:23 smithi070 ceph-mon[107385]: pgmap v3217: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 1.9 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 1.02k op/s; 15223/92865 objects degraded (16.393%); 26015/92865 objects misplaced (28.014%); 7.6 MiB/s, 121 objects/s recovering 2024-09-21T11:05:23.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:23 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:05:23.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:23 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1983311528' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T11:05:23.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:23 smithi070 ceph-mon[107385]: osdmap e1623: 8 total, 8 up, 5 in 2024-09-21T11:05:23.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:23 smithi070 ceph-mon[112139]: pgmap v3217: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 1.9 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 1.02k op/s; 15223/92865 objects degraded (16.393%); 26015/92865 objects misplaced (28.014%); 7.6 MiB/s, 121 objects/s recovering 2024-09-21T11:05:23.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:23 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:05:23.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:23 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1983311528' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T11:05:23.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:23 smithi070 ceph-mon[112139]: osdmap e1623: 8 total, 8 up, 5 in 2024-09-21T11:05:23.660 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:23.690 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:23.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:23.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:23.781 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:23 smithi160 ceph-mon[101879]: pgmap v3217: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 1.9 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 1.02k op/s; 15223/92865 objects degraded (16.393%); 26015/92865 objects misplaced (28.014%); 7.6 MiB/s, 121 objects/s recovering 2024-09-21T11:05:23.781 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:23 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:05:23.782 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:23 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1983311528' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T11:05:23.782 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:23 smithi160 ceph-mon[101879]: osdmap e1623: 8 total, 8 up, 5 in 2024-09-21T11:05:23.792 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:23.807 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T11:05:23.816 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:23.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:23.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:23.918 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:23.940 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:23.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:23.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:23.956 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T11:05:23.956 INFO:teuthology.orchestra.run.smithi160.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T11:05:23.956 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T11:05:24.042 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:24.065 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:24.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:24.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:24.167 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:24.189 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:24.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:24.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:24.292 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:24.314 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:24.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:24.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:24.416 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:24.439 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:24.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:24.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:24.540 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:24.569 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:24.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:24.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:24 smithi160 ceph-mon[101879]: osdmap e1624: 8 total, 8 up, 5 in 2024-09-21T11:05:24.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:24 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 15223/92865 objects degraded (16.393%), 9 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T11:05:24.671 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:24.721 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:24.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:24.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:24.776 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:05:24.823 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:24.849 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:24.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:24.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:24.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:24 smithi070 ceph-mon[107385]: osdmap e1624: 8 total, 8 up, 5 in 2024-09-21T11:05:24.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:24 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 15223/92865 objects degraded (16.393%), 9 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T11:05:24.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:24 smithi070 ceph-mon[112139]: osdmap e1624: 8 total, 8 up, 5 in 2024-09-21T11:05:24.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:24 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 15223/92865 objects degraded (16.393%), 9 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T11:05:24.950 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:24.990 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:24.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:24.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:25.092 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:25.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:25.154 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:25.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:25.255 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:25.288 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:25.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:25.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:25.390 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:25.427 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:25.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:25.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:25.529 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:25.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:25.566 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:25.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:25.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:25 smithi070 ceph-mon[107385]: pgmap v3220: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 1.9 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 669 op/s; 15223/92865 objects degraded (16.393%); 26015/92865 objects misplaced (28.014%); 4.8 MiB/s, 77 objects/s recovering 2024-09-21T11:05:25.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:25 smithi070 ceph-mon[107385]: osdmap e1625: 8 total, 8 up, 5 in 2024-09-21T11:05:25.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:25 smithi070 ceph-mon[112139]: pgmap v3220: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 1.9 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 669 op/s; 15223/92865 objects degraded (16.393%); 26015/92865 objects misplaced (28.014%); 4.8 MiB/s, 77 objects/s recovering 2024-09-21T11:05:25.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:25 smithi070 ceph-mon[112139]: osdmap e1625: 8 total, 8 up, 5 in 2024-09-21T11:05:25.667 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:25.705 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:25.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:25.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:25.807 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:25 smithi160 ceph-mon[101879]: pgmap v3220: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait, 79 active+clean; 1.9 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 669 op/s; 15223/92865 objects degraded (16.393%); 26015/92865 objects misplaced (28.014%); 4.8 MiB/s, 77 objects/s recovering 2024-09-21T11:05:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:25 smithi160 ceph-mon[101879]: osdmap e1625: 8 total, 8 up, 5 in 2024-09-21T11:05:25.844 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:25.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:25.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:25.946 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:25.987 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:25.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:25.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:26.089 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:26.096 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:05:26.096 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:05:26.118 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:26.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:26.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:26.124 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:05:26.124 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:05:26.150 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:05:26.150 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:05:26.178 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:05:26.179 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:05:26.206 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:05:26.206 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:05:26.220 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:26.233 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T11:05:26.233 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T11:05:26.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:26.257 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:26.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:26.265 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T11:05:26.265 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T11:05:26.296 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:05:26.296 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:05:26.358 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:26.395 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:26.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:26.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:26.497 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:26.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:26.529 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:26.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:26.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:26 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4250301502' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:05:26.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:26 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4250301502' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:05:26.631 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:26.668 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:26.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:26.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:26.769 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:26.811 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:26.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:26.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:26.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:26 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4250301502' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:05:26.906 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T11:05:26.913 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:26.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:26.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:27.089 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:27.125 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:27.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:27.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:27.227 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:27.266 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:27.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:27.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:27.368 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:27.395 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:27.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:27.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:27.497 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:27.527 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:27.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:27.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:27.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:27 smithi160 ceph-mon[101879]: pgmap v3222: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 6 active+undersized+degraded+remapped+backfill_wait, 80 active+clean; 1.6 GiB data, 9.6 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 1.17k op/s; 12176/80580 objects degraded (15.110%); 22300/80580 objects misplaced (27.674%); 12 MiB/s, 197 objects/s recovering 2024-09-21T11:05:27.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:27 smithi070 ceph-mon[107385]: pgmap v3222: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 6 active+undersized+degraded+remapped+backfill_wait, 80 active+clean; 1.6 GiB data, 9.6 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 1.17k op/s; 12176/80580 objects degraded (15.110%); 22300/80580 objects misplaced (27.674%); 12 MiB/s, 197 objects/s recovering 2024-09-21T11:05:27.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:27 smithi070 ceph-mon[112139]: pgmap v3222: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 6 active+undersized+degraded+remapped+backfill_wait, 80 active+clean; 1.6 GiB data, 9.6 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 1.17k op/s; 12176/80580 objects degraded (15.110%); 22300/80580 objects misplaced (27.674%); 12 MiB/s, 197 objects/s recovering 2024-09-21T11:05:27.630 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:27.671 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:27.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:27.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:27.772 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:27.797 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:27.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:27.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:27.899 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:27.940 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:05:27.940 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:27.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:27.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:28.013 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T11:05:28.013 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T11:05:28.042 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:28.085 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:28.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:28.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:28.187 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:28.228 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:28.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:28.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:28.330 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:28.363 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:28.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:28.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:28.465 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:28.494 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:28.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:28.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:28.595 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:28.631 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:28.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:28.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:28.733 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:28.768 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:28.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:28.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:28.784 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T11:05:28.870 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:28.896 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:28.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:28.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:28.998 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:29.038 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:29.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:29.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:29.140 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:29.166 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:29.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:29.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:29.208 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:05:29.268 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:29.296 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:29.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:29.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:29.398 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:29.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:29.445 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:29.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:29.545 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:29.572 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:29.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:29.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:29.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:29 smithi070 ceph-mon[107385]: pgmap v3223: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 6 active+undersized+degraded+remapped+backfill_wait, 80 active+clean; 1.6 GiB data, 9.4 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 726 op/s; 11911/79782 objects degraded (14.929%); 22300/79782 objects misplaced (27.951%); 12 MiB/s, 197 objects/s recovering 2024-09-21T11:05:29.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:29 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 11911/79782 objects degraded (14.929%), 8 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T11:05:29.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:29 smithi070 ceph-mon[112139]: pgmap v3223: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 6 active+undersized+degraded+remapped+backfill_wait, 80 active+clean; 1.6 GiB data, 9.4 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 726 op/s; 11911/79782 objects degraded (14.929%); 22300/79782 objects misplaced (27.951%); 12 MiB/s, 197 objects/s recovering 2024-09-21T11:05:29.632 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:29 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 11911/79782 objects degraded (14.929%), 8 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T11:05:29.649 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7, 4] out_osds: [2, 5, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 7, 6, 5] 2024-09-21T11:05:29.649 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T11:05:29.649 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-21T11:05:29.649 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T11:05:29.649 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T11:05:29.650 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-21T11:05:29.674 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:29.693 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:29 smithi160 ceph-mon[101879]: pgmap v3223: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 6 active+undersized+degraded+remapped+backfill_wait, 80 active+clean; 1.6 GiB data, 9.4 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 726 op/s; 11911/79782 objects degraded (14.929%); 22300/79782 objects misplaced (27.951%); 12 MiB/s, 197 objects/s recovering 2024-09-21T11:05:29.693 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:29 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 11911/79782 objects degraded (14.929%), 8 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T11:05:29.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:29.734 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:29.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:29.835 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:29.855 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T11:05:29.889 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:29.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:29.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:29.990 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:30.043 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:30.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:30.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:30.145 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:30.175 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:05:30.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:30.182 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:30.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:30.283 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:30.319 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:30.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:30.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:30.420 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:30.439 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T11:05:30.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:30.451 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:30.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:30.552 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:30.592 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:30.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:30.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:30.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:30 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3840248729' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:05:30.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:30 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3840248729' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:05:30.694 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:30.722 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:30.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:30.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:30.824 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:30 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3840248729' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:05:30.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:30.867 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:30.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:30.968 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:30.996 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:30.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:30.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:31.098 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:31.126 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:31.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:31.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:31.228 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:31.259 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:31.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:31.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:31.361 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:31.389 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:31.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:31.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:31.492 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:31.519 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:31.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:31.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:31.621 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:31.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:31 smithi070 ceph-mon[107385]: pgmap v3224: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 7 active+undersized+degraded+remapped+backfill_wait, 80 active+clean; 1.5 GiB data, 9.1 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 830 op/s; 11077/73119 objects degraded (15.149%); 20753/73119 objects misplaced (28.382%); 9.3 MiB/s, 149 objects/s recovering 2024-09-21T11:05:31.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:31 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:05:31.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:31 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3840248729' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T11:05:31.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:31 smithi070 ceph-mon[107385]: osdmap e1626: 8 total, 8 up, 5 in 2024-09-21T11:05:31.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:31 smithi070 ceph-mon[112139]: pgmap v3224: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 7 active+undersized+degraded+remapped+backfill_wait, 80 active+clean; 1.5 GiB data, 9.1 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 830 op/s; 11077/73119 objects degraded (15.149%); 20753/73119 objects misplaced (28.382%); 9.3 MiB/s, 149 objects/s recovering 2024-09-21T11:05:31.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:31 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:05:31.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:31 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3840248729' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T11:05:31.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:31 smithi070 ceph-mon[112139]: osdmap e1626: 8 total, 8 up, 5 in 2024-09-21T11:05:31.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:31.681 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:31.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:31.782 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:31.809 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:31.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:31.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:31 smithi160 ceph-mon[101879]: pgmap v3224: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 7 active+undersized+degraded+remapped+backfill_wait, 80 active+clean; 1.5 GiB data, 9.1 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 830 op/s; 11077/73119 objects degraded (15.149%); 20753/73119 objects misplaced (28.382%); 9.3 MiB/s, 149 objects/s recovering 2024-09-21T11:05:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:31 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:05:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:31 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3840248729' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T11:05:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:31 smithi160 ceph-mon[101879]: osdmap e1626: 8 total, 8 up, 5 in 2024-09-21T11:05:31.911 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:31.942 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:31.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:31.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:32.044 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:32.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:32.074 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:32.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:32.175 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:32.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:32.216 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:32.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:32.317 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:32.355 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:32.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:32.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:32.457 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:32.486 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:32.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:32.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:32.588 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:32.624 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:32.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:32.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:32.726 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:32.765 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:32.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:32.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:32.867 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:32.901 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:32.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:32.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:33.003 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:33.037 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:33.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:33.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:33.140 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:33.174 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:33.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:33.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:33.267 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:05:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:05:33] "GET /metrics HTTP/1.1" 200 36755 "" "Prometheus/2.51.0" 2024-09-21T11:05:33.276 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:33.277 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:05:33.298 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:05:33.298 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:05:33.302 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:33.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:33.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:33.321 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:05:33.322 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:05:33.345 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:05:33.346 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:05:33.368 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:05:33.368 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:05:33.400 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:05:33.401 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:05:33.404 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:33.427 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T11:05:33.427 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T11:05:33.437 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:33.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:33.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:33.464 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T11:05:33.465 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T11:05:33.489 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:05:33.489 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:05:33.539 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:33.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:33.565 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:33.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:33.576 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:33 smithi160 ceph-mon[101879]: pgmap v3226: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 7 active+undersized+degraded+remapped+backfill_wait, 80 active+clean; 1.3 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 1.19k op/s; 9286/64410 objects degraded (14.417%); 18065/64410 objects misplaced (28.047%); 12 MiB/s, 193 objects/s recovering 2024-09-21T11:05:33.576 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:33 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2183535450' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:05:33.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:33 smithi070 ceph-mon[107385]: pgmap v3226: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 7 active+undersized+degraded+remapped+backfill_wait, 80 active+clean; 1.3 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 1.19k op/s; 9286/64410 objects degraded (14.417%); 18065/64410 objects misplaced (28.047%); 12 MiB/s, 193 objects/s recovering 2024-09-21T11:05:33.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:33 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2183535450' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:05:33.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:33 smithi070 ceph-mon[112139]: pgmap v3226: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 7 active+undersized+degraded+remapped+backfill_wait, 80 active+clean; 1.3 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 1.19k op/s; 9286/64410 objects degraded (14.417%); 18065/64410 objects misplaced (28.047%); 12 MiB/s, 193 objects/s recovering 2024-09-21T11:05:33.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:33 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2183535450' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:05:33.666 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:33.714 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T11:05:33.716 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:33.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:33.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:33.819 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:33.847 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:33.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:33.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:33.949 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:33.975 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:33.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:33.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:34.077 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:34.103 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:34.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:34.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:34.205 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:34.230 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T11:05:34.230 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T11:05:34.230 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T11:05:34.230 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T11:05:34.231 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:34.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:34.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:34.333 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:34.373 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:34.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:34.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:34.474 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:34.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:34.502 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:34.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:34.603 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:34.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:34.630 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:34.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:34.731 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:34.759 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:34.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:34.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:34 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 9286/64410 objects degraded (14.417%), 8 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T11:05:34.860 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:34.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:34 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 9286/64410 objects degraded (14.417%), 8 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T11:05:34.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:34 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 9286/64410 objects degraded (14.417%), 8 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T11:05:34.896 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:34.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:34.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:34.998 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:35.023 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:35.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:35.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:35.125 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:35.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:35.152 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:35.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:35.253 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:35.285 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:35.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:35.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:35.387 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:35.417 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:35.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:35.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:35.519 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:35.555 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:35.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:35.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:35.639 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:05:35.658 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:35.684 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:35.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:35.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:35.745 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:35 smithi160 ceph-mon[101879]: pgmap v3227: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 7 active+undersized+degraded+remapped+backfill_wait, 80 active+clean; 1.3 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 955 op/s; 9286/64410 objects degraded (14.417%); 18065/64410 objects misplaced (28.047%); 9.8 MiB/s, 156 objects/s recovering 2024-09-21T11:05:35.745 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:05:35.745 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:05:35.766 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:35 smithi070 ceph-mon[107385]: pgmap v3227: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 7 active+undersized+degraded+remapped+backfill_wait, 80 active+clean; 1.3 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 955 op/s; 9286/64410 objects degraded (14.417%); 18065/64410 objects misplaced (28.047%); 9.8 MiB/s, 156 objects/s recovering 2024-09-21T11:05:35.766 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:05:35.766 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:05:35.766 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:35 smithi070 ceph-mon[112139]: pgmap v3227: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 7 active+undersized+degraded+remapped+backfill_wait, 80 active+clean; 1.3 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 955 op/s; 9286/64410 objects degraded (14.417%); 18065/64410 objects misplaced (28.047%); 9.8 MiB/s, 156 objects/s recovering 2024-09-21T11:05:35.766 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:05:35.766 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:05:35.767 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T11:05:35.786 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:35.833 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:35.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:35.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:35.935 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:35.958 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:35.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:35.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:36.060 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:36.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:36.087 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:36.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:36.188 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:36.212 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:36.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:36.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:36.284 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T11:05:36.284 INFO:teuthology.orchestra.run.smithi160.stdout: "success": "" 2024-09-21T11:05:36.284 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T11:05:36.314 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:36.336 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:36.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:36.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:36.438 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:36.465 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:36.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:36.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:36.508 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:05:36.567 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:36.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:36.619 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:36.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:36.720 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:36.754 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:36.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:36.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:36.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:36 smithi160 ceph-mon[101879]: osdmap e1627: 8 total, 8 up, 5 in 2024-09-21T11:05:36.856 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:36.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:36 smithi070 ceph-mon[107385]: osdmap e1627: 8 total, 8 up, 5 in 2024-09-21T11:05:36.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:36 smithi070 ceph-mon[112139]: osdmap e1627: 8 total, 8 up, 5 in 2024-09-21T11:05:36.892 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:36.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:36.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:36.993 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:37.028 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:37.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:37.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:37.130 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:37.170 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:37.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:37.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:37.273 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:37.318 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:37.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:37.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:37.419 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:37.450 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:37.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:37.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:37.542 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T11:05:37.552 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:37.559 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:37 smithi070 ceph-mon[107385]: pgmap v3229: 109 pgs: 1 peering, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 7 active+undersized+degraded+remapped+backfill_wait, 80 active+clean; 1.0 GiB data, 8.1 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 1.21k op/s; 7205/50703 objects degraded (14.210%); 13906/50703 objects misplaced (27.426%); 5.7 MiB/s, 91 objects/s recovering 2024-09-21T11:05:37.560 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:37 smithi070 ceph-mon[107385]: osdmap e1628: 8 total, 8 up, 5 in 2024-09-21T11:05:37.560 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:37 smithi070 ceph-mon[112139]: pgmap v3229: 109 pgs: 1 peering, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 7 active+undersized+degraded+remapped+backfill_wait, 80 active+clean; 1.0 GiB data, 8.1 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 1.21k op/s; 7205/50703 objects degraded (14.210%); 13906/50703 objects misplaced (27.426%); 5.7 MiB/s, 91 objects/s recovering 2024-09-21T11:05:37.560 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:37 smithi070 ceph-mon[112139]: osdmap e1628: 8 total, 8 up, 5 in 2024-09-21T11:05:37.594 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:37.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:37.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:37.696 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:37.732 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:37.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:37.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:37.834 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:37.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:37 smithi160 ceph-mon[101879]: pgmap v3229: 109 pgs: 1 peering, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 7 active+undersized+degraded+remapped+backfill_wait, 80 active+clean; 1.0 GiB data, 8.1 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 1.21k op/s; 7205/50703 objects degraded (14.210%); 13906/50703 objects misplaced (27.426%); 5.7 MiB/s, 91 objects/s recovering 2024-09-21T11:05:37.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:37 smithi160 ceph-mon[101879]: osdmap e1628: 8 total, 8 up, 5 in 2024-09-21T11:05:37.855 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:05:37.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:37.883 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:37.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:37.984 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:38.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:38.017 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:38.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:38.118 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:38.157 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:38.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:38.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:38.259 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:38.290 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:38.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:38.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:38.392 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:38.435 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:38.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:38.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:38.467 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T11:05:38.537 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:38.570 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:38.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:38.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:38.613 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:38 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4269350508' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:05:38.613 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:38 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:05:38.613 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:38 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4269350508' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:05:38.614 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:38 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:05:38.691 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:38.737 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:38.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:38.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:38.839 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:38 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4269350508' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:05:38.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:38 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:05:38.866 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:38.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:38.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:38.968 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:38.997 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:38.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:38.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:39.099 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:39.127 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:39.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:39.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:39.229 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:39.264 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:39.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:39.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:39.366 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:39.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:39.396 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:39.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:39.497 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:39.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:39.523 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:39.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:39.624 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:39.657 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:39.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:39.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:39.759 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:39.784 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:39.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:39.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:39 smithi160 ceph-mon[101879]: pgmap v3231: 109 pgs: 1 peering, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 7 active+undersized+degraded+remapped+backfill_wait, 80 active+clean; 1.0 GiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 979 op/s; 6937/49899 objects degraded (13.902%); 13906/49899 objects misplaced (27.868%); 5.8 MiB/s, 93 objects/s recovering 2024-09-21T11:05:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:39 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:05:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:39 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T11:05:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:39 smithi160 ceph-mon[101879]: osdmap e1629: 8 total, 8 up, 5 in 2024-09-21T11:05:39.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:39 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 6937/49899 objects degraded (13.902%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T11:05:39.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:39 smithi070 ceph-mon[107385]: pgmap v3231: 109 pgs: 1 peering, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 7 active+undersized+degraded+remapped+backfill_wait, 80 active+clean; 1.0 GiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 979 op/s; 6937/49899 objects degraded (13.902%); 13906/49899 objects misplaced (27.868%); 5.8 MiB/s, 93 objects/s recovering 2024-09-21T11:05:39.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:39 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:05:39.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:39 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T11:05:39.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:39 smithi070 ceph-mon[107385]: osdmap e1629: 8 total, 8 up, 5 in 2024-09-21T11:05:39.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:39 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 6937/49899 objects degraded (13.902%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T11:05:39.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:39 smithi070 ceph-mon[112139]: pgmap v3231: 109 pgs: 1 peering, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 7 active+undersized+degraded+remapped+backfill_wait, 80 active+clean; 1.0 GiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 979 op/s; 6937/49899 objects degraded (13.902%); 13906/49899 objects misplaced (27.868%); 5.8 MiB/s, 93 objects/s recovering 2024-09-21T11:05:39.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:39 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:05:39.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:39 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T11:05:39.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:39 smithi070 ceph-mon[112139]: osdmap e1629: 8 total, 8 up, 5 in 2024-09-21T11:05:39.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:39 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 6937/49899 objects degraded (13.902%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T11:05:39.886 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:39.907 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:39.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:39.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:40.009 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:40.030 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:40.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:40.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:40.132 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:40.156 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:40.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:40.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:40.258 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:40.280 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:40.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:40.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:40.382 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:40.406 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:40.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:40.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:40.508 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:40.512 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T11:05:40.512 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T11:05:40.512 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T11:05:40.512 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T11:05:40.512 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T11:05:40.521 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:40 smithi160 ceph-mon[101879]: osdmap e1630: 8 total, 8 up, 5 in 2024-09-21T11:05:40.531 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:40.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:40.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:40.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:40 smithi070 ceph-mon[107385]: osdmap e1630: 8 total, 8 up, 5 in 2024-09-21T11:05:40.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:40 smithi070 ceph-mon[112139]: osdmap e1630: 8 total, 8 up, 5 in 2024-09-21T11:05:40.632 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:40.658 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:40.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:40.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:40.760 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:40.786 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:40.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:40.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:40.887 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:40.921 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:40.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:40.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:41.023 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:41.052 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:41.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:41.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:41.154 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:41.185 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:41.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:41.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:41.287 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:41.329 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:41.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:41.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:41.426 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T11:05:41.431 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:41.477 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:41.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:41.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:41.494 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:05:41.495 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:05:41.521 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:05:41.522 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:05:41.550 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:05:41.551 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:05:41.579 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:41.585 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:05:41.585 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:05:41.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:41.620 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:41.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:41.621 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:05:41.621 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:05:41.656 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T11:05:41.657 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T11:05:41.696 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T11:05:41.696 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T11:05:41.722 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:41.730 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:05:41.730 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:05:41.731 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:41 smithi070 ceph-mon[107385]: pgmap v3234: 109 pgs: 2 peering, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 6 active+undersized+degraded+remapped+backfill_wait, 80 active+clean; 926 MiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 912 op/s; 5449/44484 objects degraded (12.249%); 11778/44484 objects misplaced (26.477%); 17 MiB/s, 274 objects/s recovering 2024-09-21T11:05:41.731 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:41 smithi070 ceph-mon[107385]: osdmap e1631: 8 total, 8 up, 5 in 2024-09-21T11:05:41.731 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:41 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2271085507' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:05:41.731 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:41 smithi070 ceph-mon[112139]: pgmap v3234: 109 pgs: 2 peering, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 6 active+undersized+degraded+remapped+backfill_wait, 80 active+clean; 926 MiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 912 op/s; 5449/44484 objects degraded (12.249%); 11778/44484 objects misplaced (26.477%); 17 MiB/s, 274 objects/s recovering 2024-09-21T11:05:41.731 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:41 smithi070 ceph-mon[112139]: osdmap e1631: 8 total, 8 up, 5 in 2024-09-21T11:05:41.731 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:41 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2271085507' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:05:41.738 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:05:41.740 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:41 smithi160 ceph-mon[101879]: pgmap v3234: 109 pgs: 2 peering, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 6 active+undersized+degraded+remapped+backfill_wait, 80 active+clean; 926 MiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 912 op/s; 5449/44484 objects degraded (12.249%); 11778/44484 objects misplaced (26.477%); 17 MiB/s, 274 objects/s recovering 2024-09-21T11:05:41.740 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:41 smithi160 ceph-mon[101879]: osdmap e1631: 8 total, 8 up, 5 in 2024-09-21T11:05:41.740 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:41 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2271085507' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:05:41.760 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:41.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:41.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:41.780 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T11:05:41.862 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:41.920 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:41.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:41.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:42.022 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:42.048 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:42.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:42.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:42.145 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:05:42.149 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:42.174 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:42.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:42.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:42.224 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7, 4] out_osds: [2, 5, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 7, 6, 5] 2024-09-21T11:05:42.224 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T11:05:42.224 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-09-21T11:05:42.224 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd in 0 2024-09-21T11:05:42.276 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:42.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:42.312 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:42.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:42.413 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:42.445 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:42.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:42.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:42.547 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:42.592 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:42.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:42.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:42.694 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:42.734 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:42.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:42.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:42.836 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:42.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:42.878 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:42.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:42.979 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:43.013 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:43.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:43.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:43.115 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:43.150 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:43.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:43.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:43.252 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:43.294 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:43.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:43.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:43.319 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:05:43.378 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:05:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:05:43] "GET /metrics HTTP/1.1" 200 36761 "" "Prometheus/2.51.0" 2024-09-21T11:05:43.396 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:43.427 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:43.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:43.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:43.528 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:43.556 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:43.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:43.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:43.658 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:43.685 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:43.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:43.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:43.788 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:43.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:43.812 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:43.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:43.822 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:43 smithi070 ceph-mon[107385]: pgmap v3236: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 peering, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait, 81 active+clean; 785 MiB data, 7.4 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 688 op/s; 3853/37734 objects degraded (10.211%); 9666/37734 objects misplaced (25.616%); 17 MiB/s, 270 objects/s recovering 2024-09-21T11:05:43.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:43 smithi070 ceph-mon[112139]: pgmap v3236: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 peering, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait, 81 active+clean; 785 MiB data, 7.4 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 688 op/s; 3853/37734 objects degraded (10.211%); 9666/37734 objects misplaced (25.616%); 17 MiB/s, 270 objects/s recovering 2024-09-21T11:05:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:43 smithi160 ceph-mon[101879]: pgmap v3236: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 peering, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait, 81 active+clean; 785 MiB data, 7.4 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 688 op/s; 3853/37734 objects degraded (10.211%); 9666/37734 objects misplaced (25.616%); 17 MiB/s, 270 objects/s recovering 2024-09-21T11:05:43.913 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:43.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:43.937 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:43.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:44.038 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:44.062 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:44.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:44.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:44.165 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:44.187 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:44.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:44.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:44.289 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:44.311 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:44.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:44.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:44.413 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:44.436 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:44.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:44.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:44.532 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T11:05:44.538 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:44.606 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:44.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:44.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:44.708 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:44.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:44.740 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:44.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:44.826 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:44 smithi070 ceph-mon[107385]: osdmap e1632: 8 total, 8 up, 5 in 2024-09-21T11:05:44.827 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:44 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 3853/37734 objects degraded (10.211%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T11:05:44.827 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:44 smithi070 ceph-mon[112139]: osdmap e1632: 8 total, 8 up, 5 in 2024-09-21T11:05:44.827 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:44 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 3853/37734 objects degraded (10.211%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T11:05:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:44 smithi160 ceph-mon[101879]: osdmap e1632: 8 total, 8 up, 5 in 2024-09-21T11:05:44.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:44 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 3853/37734 objects degraded (10.211%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T11:05:44.841 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:44.867 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:44.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:44.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:44.969 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:44.990 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:44.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:44.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:45.091 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:45.116 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:45.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:45.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:45.218 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:45.240 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:45.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:45.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:45.341 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:45.364 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:45.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:45.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:45.466 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:45.489 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:45.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:45.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:45.590 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:45.617 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:45.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:45.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:45.719 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:45.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:45.748 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:45.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:45.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:45 smithi160 ceph-mon[101879]: pgmap v3238: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 peering, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait, 81 active+clean; 785 MiB data, 7.4 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 689 op/s; 3853/37734 objects degraded (10.211%); 9666/37734 objects misplaced (25.616%); 17 MiB/s, 270 objects/s recovering 2024-09-21T11:05:45.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:45 smithi160 ceph-mon[101879]: osdmap e1633: 8 total, 8 up, 5 in 2024-09-21T11:05:45.849 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:45.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:45 smithi070 ceph-mon[107385]: pgmap v3238: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 peering, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait, 81 active+clean; 785 MiB data, 7.4 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 689 op/s; 3853/37734 objects degraded (10.211%); 9666/37734 objects misplaced (25.616%); 17 MiB/s, 270 objects/s recovering 2024-09-21T11:05:45.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:45 smithi070 ceph-mon[107385]: osdmap e1633: 8 total, 8 up, 5 in 2024-09-21T11:05:45.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:45 smithi070 ceph-mon[112139]: pgmap v3238: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 peering, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait, 81 active+clean; 785 MiB data, 7.4 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 689 op/s; 3853/37734 objects degraded (10.211%); 9666/37734 objects misplaced (25.616%); 17 MiB/s, 270 objects/s recovering 2024-09-21T11:05:45.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:45 smithi070 ceph-mon[112139]: osdmap e1633: 8 total, 8 up, 5 in 2024-09-21T11:05:45.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:45.879 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:45.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:45.980 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:46.002 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:46.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:46.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:46.104 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:46.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:46.134 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:46.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:46.235 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:46.246 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:05:46.259 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:46.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:46.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:46.361 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:46.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:46.383 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:46.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:46.484 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:46.518 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T11:05:46.524 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:46.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:46.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:46.626 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:46.654 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:46.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:46.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:46.756 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:46.779 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:46.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:46.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:46.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:46 smithi160 ceph-mon[101879]: osdmap e1634: 8 total, 8 up, 5 in 2024-09-21T11:05:46.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:46 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3816966742' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:05:46.854 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:46 smithi070 ceph-mon[107385]: osdmap e1634: 8 total, 8 up, 5 in 2024-09-21T11:05:46.854 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:46 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3816966742' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:05:46.854 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:46 smithi070 ceph-mon[112139]: osdmap e1634: 8 total, 8 up, 5 in 2024-09-21T11:05:46.854 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:46 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3816966742' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:05:46.881 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:46.905 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:46.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:46.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:47.006 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:47.029 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:47.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:47.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:47.130 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:47.155 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:47.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:47.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:47.257 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:47.283 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:47.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:47.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:47.376 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T11:05:47.376 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T11:05:47.385 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:47.407 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:47.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:47.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:47.482 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:47 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a[107362]: 2024-09-21T11:05:47.174+0000 7fad4a465640 -1 mon.a@0(leader).osd e1635 definitely_dead 0 2024-09-21T11:05:47.509 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:47.523 INFO:teuthology.orchestra.run.smithi070.stderr:marked in osd.0. 2024-09-21T11:05:47.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:47.542 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:47.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:47.643 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:47.666 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:47.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:47.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:47.756 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:47 smithi070 ceph-mon[107385]: pgmap v3241: 109 pgs: 1 active+undersized+remapped, 1 peering, 2 active+undersized+degraded+remapped+backfilling, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait, 82 active+clean; 574 MiB data, 7.1 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 958 op/s; 1839/27633 objects degraded (6.655%); 6321/27633 objects misplaced (22.875%); 17 MiB/s, 279 objects/s recovering 2024-09-21T11:05:47.756 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:47 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T11:05:47.756 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:47 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3816966742' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T11:05:47.756 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:47 smithi070 ceph-mon[107385]: osdmap e1635: 8 total, 8 up, 5 in 2024-09-21T11:05:47.756 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:47 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/246124958' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-21T11:05:47.756 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:47 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-21T11:05:47.756 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:47 smithi070 ceph-mon[112139]: pgmap v3241: 109 pgs: 1 active+undersized+remapped, 1 peering, 2 active+undersized+degraded+remapped+backfilling, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait, 82 active+clean; 574 MiB data, 7.1 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 958 op/s; 1839/27633 objects degraded (6.655%); 6321/27633 objects misplaced (22.875%); 17 MiB/s, 279 objects/s recovering 2024-09-21T11:05:47.756 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:47 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T11:05:47.756 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:47 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3816966742' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T11:05:47.756 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:47 smithi070 ceph-mon[112139]: osdmap e1635: 8 total, 8 up, 5 in 2024-09-21T11:05:47.757 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:47 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/246124958' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-21T11:05:47.757 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:47 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-21T11:05:47.768 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:47.785 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:47 smithi160 ceph-mon[101879]: pgmap v3241: 109 pgs: 1 active+undersized+remapped, 1 peering, 2 active+undersized+degraded+remapped+backfilling, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait, 82 active+clean; 574 MiB data, 7.1 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 958 op/s; 1839/27633 objects degraded (6.655%); 6321/27633 objects misplaced (22.875%); 17 MiB/s, 279 objects/s recovering 2024-09-21T11:05:47.785 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:47 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T11:05:47.785 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:47 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3816966742' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T11:05:47.785 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:47 smithi160 ceph-mon[101879]: osdmap e1635: 8 total, 8 up, 5 in 2024-09-21T11:05:47.785 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:47 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/246124958' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-21T11:05:47.785 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:47 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-21T11:05:47.792 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:47.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:47.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:47.894 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:47.915 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:47.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:47.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:48.016 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:48.038 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:48.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:48.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:48.133 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T11:05:48.140 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:48.162 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:48.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:48.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:48.264 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:48.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:48.289 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:48.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:48.390 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:48.413 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:48.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:48.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:48.515 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:48.546 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:48.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:48.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:48.648 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:48.670 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:48.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:48.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:48.772 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:48.794 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:48.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:48.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:48 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-09-21T11:05:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:48 smithi160 ceph-mon[101879]: osdmap e1636: 8 total, 8 up, 6 in 2024-09-21T11:05:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:48 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 1839/27633 objects degraded (6.655%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T11:05:48.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:48 smithi160 ceph-mon[101879]: osdmap e1637: 8 total, 8 up, 6 in 2024-09-21T11:05:48.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:48 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-09-21T11:05:48.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:48 smithi070 ceph-mon[107385]: osdmap e1636: 8 total, 8 up, 6 in 2024-09-21T11:05:48.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:48 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 1839/27633 objects degraded (6.655%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T11:05:48.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:48 smithi070 ceph-mon[107385]: osdmap e1637: 8 total, 8 up, 6 in 2024-09-21T11:05:48.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:48 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-09-21T11:05:48.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:48 smithi070 ceph-mon[112139]: osdmap e1636: 8 total, 8 up, 6 in 2024-09-21T11:05:48.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:48 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 1839/27633 objects degraded (6.655%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T11:05:48.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:48 smithi070 ceph-mon[112139]: osdmap e1637: 8 total, 8 up, 6 in 2024-09-21T11:05:48.896 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:48.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:48.919 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:48.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:49.019 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:49.045 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:49.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:49.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:49.147 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:49.171 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:49.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:49.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:49.274 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:49.296 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:49.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:49.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:49.398 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:49.421 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:49.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:49.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:49.506 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T11:05:49.522 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:49.591 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-09-21T11:05:49.629 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:49.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:49.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:49.731 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:49.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:49.767 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:49.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:49 smithi160 ceph-mon[101879]: pgmap v3244: 109 pgs: 1 active+undersized+remapped, 1 peering, 1 active+undersized+degraded+remapped+backfilling, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait, 83 active+clean; 561 MiB data, 6.9 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 895 op/s; 1688/26985 objects degraded (6.255%); 6321/26985 objects misplaced (23.424%); 18 MiB/s, 288 objects/s recovering 2024-09-21T11:05:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:49 smithi160 ceph-mon[101879]: 58.1 starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17444 2024-09-21T11:05:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:49 smithi160 ceph-mon[101879]: 58.2 starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17803 2024-09-21T11:05:49.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:49 smithi160 ceph-mon[101879]: 58.0 starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17929 2024-09-21T11:05:49.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:49 smithi160 ceph-mon[101879]: 58.3 starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17707 2024-09-21T11:05:49.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:49 smithi160 ceph-mon[101879]: 58.6 starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17965 2024-09-21T11:05:49.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:49 smithi160 ceph-mon[101879]: 58.1 continuing backfill to osd.6 from (1603'13059,1635'17444] MIN to 1635'17444 2024-09-21T11:05:49.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:49 smithi160 ceph-mon[101879]: 58.2 continuing backfill to osd.3 from (1603'13566,1635'17803] 58:493b3968:::benchmark_data_smithi070_396992_object127732:head to 1635'17803 2024-09-21T11:05:49.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:49 smithi160 ceph-mon[101879]: 58.0 continuing backfill to osd.6 from (1603'13761,1635'17929] MIN to 1635'17929 2024-09-21T11:05:49.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:49 smithi160 ceph-mon[101879]: 58.3 continuing backfill to osd.3 from (1603'13442,1635'17707] MIN to 1635'17707 2024-09-21T11:05:49.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:49 smithi160 ceph-mon[101879]: 58.3 continuing backfill to osd.6 from (1603'13442,1635'17707] MIN to 1635'17707 2024-09-21T11:05:49.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:49 smithi160 ceph-mon[101879]: 58.6 continuing backfill to osd.6 from (1603'13796,1635'17965] MIN to 1635'17965 2024-09-21T11:05:49.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:49 smithi160 ceph-mon[101879]: osdmap e1638: 8 total, 8 up, 6 in 2024-09-21T11:05:49.868 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:49.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:49 smithi070 ceph-mon[107385]: pgmap v3244: 109 pgs: 1 active+undersized+remapped, 1 peering, 1 active+undersized+degraded+remapped+backfilling, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait, 83 active+clean; 561 MiB data, 6.9 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 895 op/s; 1688/26985 objects degraded (6.255%); 6321/26985 objects misplaced (23.424%); 18 MiB/s, 288 objects/s recovering 2024-09-21T11:05:49.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:49 smithi070 ceph-mon[107385]: 58.1 starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17444 2024-09-21T11:05:49.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:49 smithi070 ceph-mon[107385]: 58.2 starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17803 2024-09-21T11:05:49.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:49 smithi070 ceph-mon[107385]: 58.0 starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17929 2024-09-21T11:05:49.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:49 smithi070 ceph-mon[107385]: 58.3 starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17707 2024-09-21T11:05:49.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:49 smithi070 ceph-mon[107385]: 58.6 starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17965 2024-09-21T11:05:49.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:49 smithi070 ceph-mon[107385]: 58.1 continuing backfill to osd.6 from (1603'13059,1635'17444] MIN to 1635'17444 2024-09-21T11:05:49.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:49 smithi070 ceph-mon[107385]: 58.2 continuing backfill to osd.3 from (1603'13566,1635'17803] 58:493b3968:::benchmark_data_smithi070_396992_object127732:head to 1635'17803 2024-09-21T11:05:49.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:49 smithi070 ceph-mon[107385]: 58.0 continuing backfill to osd.6 from (1603'13761,1635'17929] MIN to 1635'17929 2024-09-21T11:05:49.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:49 smithi070 ceph-mon[107385]: 58.3 continuing backfill to osd.3 from (1603'13442,1635'17707] MIN to 1635'17707 2024-09-21T11:05:49.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:49 smithi070 ceph-mon[107385]: 58.3 continuing backfill to osd.6 from (1603'13442,1635'17707] MIN to 1635'17707 2024-09-21T11:05:49.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:49 smithi070 ceph-mon[107385]: 58.6 continuing backfill to osd.6 from (1603'13796,1635'17965] MIN to 1635'17965 2024-09-21T11:05:49.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:49 smithi070 ceph-mon[107385]: osdmap e1638: 8 total, 8 up, 6 in 2024-09-21T11:05:49.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:49 smithi070 ceph-mon[112139]: pgmap v3244: 109 pgs: 1 active+undersized+remapped, 1 peering, 1 active+undersized+degraded+remapped+backfilling, 16 active+clean+remapped, 5 active+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait, 83 active+clean; 561 MiB data, 6.9 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 895 op/s; 1688/26985 objects degraded (6.255%); 6321/26985 objects misplaced (23.424%); 18 MiB/s, 288 objects/s recovering 2024-09-21T11:05:49.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:49 smithi070 ceph-mon[112139]: 58.1 starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17444 2024-09-21T11:05:49.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:49 smithi070 ceph-mon[112139]: 58.2 starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17803 2024-09-21T11:05:49.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:49 smithi070 ceph-mon[112139]: 58.0 starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17929 2024-09-21T11:05:49.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:49 smithi070 ceph-mon[112139]: 58.3 starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17707 2024-09-21T11:05:49.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:49 smithi070 ceph-mon[112139]: 58.6 starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17965 2024-09-21T11:05:49.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:49 smithi070 ceph-mon[112139]: 58.1 continuing backfill to osd.6 from (1603'13059,1635'17444] MIN to 1635'17444 2024-09-21T11:05:49.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:49 smithi070 ceph-mon[112139]: 58.2 continuing backfill to osd.3 from (1603'13566,1635'17803] 58:493b3968:::benchmark_data_smithi070_396992_object127732:head to 1635'17803 2024-09-21T11:05:49.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:49 smithi070 ceph-mon[112139]: 58.0 continuing backfill to osd.6 from (1603'13761,1635'17929] MIN to 1635'17929 2024-09-21T11:05:49.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:49 smithi070 ceph-mon[112139]: 58.3 continuing backfill to osd.3 from (1603'13442,1635'17707] MIN to 1635'17707 2024-09-21T11:05:49.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:49 smithi070 ceph-mon[112139]: 58.3 continuing backfill to osd.6 from (1603'13442,1635'17707] MIN to 1635'17707 2024-09-21T11:05:49.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:49 smithi070 ceph-mon[112139]: 58.6 continuing backfill to osd.6 from (1603'13796,1635'17965] MIN to 1635'17965 2024-09-21T11:05:49.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:49 smithi070 ceph-mon[112139]: osdmap e1638: 8 total, 8 up, 6 in 2024-09-21T11:05:49.893 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:49.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:49.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:49.995 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:50.020 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:50.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:50.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:50.122 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:50.150 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:50.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:50.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:50.153 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:05:50.157 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:05:50.157 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:05:50.187 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:05:50.187 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:05:50.196 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:05:50.211 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:05:50.211 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:05:50.233 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:05:50.233 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:05:50.252 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:50.253 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:05:50.253 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:05:50.275 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T11:05:50.275 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T11:05:50.276 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:50.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:50.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:50.296 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T11:05:50.296 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T11:05:50.316 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:05:50.316 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:05:50.377 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:50.402 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:50.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:50.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:50.503 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:50.531 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:50.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:50.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:50.632 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:50.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:50.665 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:50.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:50.766 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:50.793 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:50.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:50.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:50 smithi160 ceph-mon[101879]: 58.8 starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17511 2024-09-21T11:05:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:50 smithi160 ceph-mon[101879]: 58.c starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17817 2024-09-21T11:05:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:50 smithi160 ceph-mon[101879]: 58.f starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17621 2024-09-21T11:05:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:50 smithi160 ceph-mon[101879]: 58.d starting backfill to osd.0 from (0'0,0'0] MAX to 1634'17788 2024-09-21T11:05:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:50 smithi160 ceph-mon[101879]: 58.a starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17717 2024-09-21T11:05:50.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:50 smithi160 ceph-mon[101879]: 58.5 starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17915 2024-09-21T11:05:50.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:50 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2011086775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:05:50.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:05:50.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:05:50.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:50 smithi160 ceph-mon[101879]: osdmap e1639: 8 total, 8 up, 6 in 2024-09-21T11:05:50.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:50 smithi070 ceph-mon[107385]: 58.8 starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17511 2024-09-21T11:05:50.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:50 smithi070 ceph-mon[107385]: 58.c starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17817 2024-09-21T11:05:50.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:50 smithi070 ceph-mon[107385]: 58.f starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17621 2024-09-21T11:05:50.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:50 smithi070 ceph-mon[107385]: 58.d starting backfill to osd.0 from (0'0,0'0] MAX to 1634'17788 2024-09-21T11:05:50.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:50 smithi070 ceph-mon[107385]: 58.a starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17717 2024-09-21T11:05:50.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:50 smithi070 ceph-mon[107385]: 58.5 starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17915 2024-09-21T11:05:50.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:50 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2011086775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:05:50.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:05:50.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:05:50.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:50 smithi070 ceph-mon[107385]: osdmap e1639: 8 total, 8 up, 6 in 2024-09-21T11:05:50.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:50 smithi070 ceph-mon[112139]: 58.8 starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17511 2024-09-21T11:05:50.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:50 smithi070 ceph-mon[112139]: 58.c starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17817 2024-09-21T11:05:50.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:50 smithi070 ceph-mon[112139]: 58.f starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17621 2024-09-21T11:05:50.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:50 smithi070 ceph-mon[112139]: 58.d starting backfill to osd.0 from (0'0,0'0] MAX to 1634'17788 2024-09-21T11:05:50.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:50 smithi070 ceph-mon[112139]: 58.a starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17717 2024-09-21T11:05:50.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:50 smithi070 ceph-mon[112139]: 58.5 starting backfill to osd.0 from (0'0,0'0] MAX to 1635'17915 2024-09-21T11:05:50.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:50 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2011086775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:05:50.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:05:50.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:05:50.879 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:50 smithi070 ceph-mon[112139]: osdmap e1639: 8 total, 8 up, 6 in 2024-09-21T11:05:50.895 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:50.917 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:50.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:50.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:51.019 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:51.042 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:51.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:51.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:51.143 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:51.166 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:51.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:51.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:51.268 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:51.291 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:51.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:51.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:51.392 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:51.414 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:51.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:51.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:51.516 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:51.540 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:51.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:51.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:51.641 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:51.675 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:51.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:51.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:51.777 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:51.806 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:51.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:51.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:51 smithi160 ceph-mon[101879]: pgmap v3247: 109 pgs: 1 active+remapped+backfilling, 2 active+undersized+remapped, 1 active+undersized+degraded+remapped, 12 active+clean+remapped, 8 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 84 active+clean; 523 MiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 205 op/s; 1086/25164 objects degraded (4.316%); 6680/25164 objects misplaced (26.546%); 13 MiB/s, 204 objects/s recovering 2024-09-21T11:05:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:51 smithi160 ceph-mon[101879]: osdmap e1640: 8 total, 8 up, 6 in 2024-09-21T11:05:51.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:51 smithi070 ceph-mon[107385]: pgmap v3247: 109 pgs: 1 active+remapped+backfilling, 2 active+undersized+remapped, 1 active+undersized+degraded+remapped, 12 active+clean+remapped, 8 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 84 active+clean; 523 MiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 205 op/s; 1086/25164 objects degraded (4.316%); 6680/25164 objects misplaced (26.546%); 13 MiB/s, 204 objects/s recovering 2024-09-21T11:05:51.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:51 smithi070 ceph-mon[107385]: osdmap e1640: 8 total, 8 up, 6 in 2024-09-21T11:05:51.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:51 smithi070 ceph-mon[112139]: pgmap v3247: 109 pgs: 1 active+remapped+backfilling, 2 active+undersized+remapped, 1 active+undersized+degraded+remapped, 12 active+clean+remapped, 8 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 84 active+clean; 523 MiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 205 op/s; 1086/25164 objects degraded (4.316%); 6680/25164 objects misplaced (26.546%); 13 MiB/s, 204 objects/s recovering 2024-09-21T11:05:51.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:51 smithi070 ceph-mon[112139]: osdmap e1640: 8 total, 8 up, 6 in 2024-09-21T11:05:51.909 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:51.931 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:51.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:51.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:52.033 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:52.056 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:52.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:52.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:52.157 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:52.180 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:52.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:52.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:52.282 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:52.305 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:52.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:52.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:52.407 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:52.430 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:52.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:52.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:52.532 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:52.558 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:52.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:52.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:52.660 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:52.695 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:52.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:52.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:52.797 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:52.820 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:52.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:52.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:52 smithi160 ceph-mon[101879]: 1.6 scrub starts 2024-09-21T11:05:52.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:52 smithi160 ceph-mon[101879]: osdmap e1641: 8 total, 8 up, 6 in 2024-09-21T11:05:52.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:52 smithi070 ceph-mon[107385]: 1.6 scrub starts 2024-09-21T11:05:52.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:52 smithi070 ceph-mon[107385]: osdmap e1641: 8 total, 8 up, 6 in 2024-09-21T11:05:52.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:52 smithi070 ceph-mon[112139]: 1.6 scrub starts 2024-09-21T11:05:52.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:52 smithi070 ceph-mon[112139]: osdmap e1641: 8 total, 8 up, 6 in 2024-09-21T11:05:52.922 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:52.946 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:52.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:52.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:53.047 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:53.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:53.069 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:53.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:53.170 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:53.192 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:53.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:53.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:53.294 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:53.317 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:53.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:53.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:53.419 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:53.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:53.441 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:53.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:53.542 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:53.552 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:05:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:05:53] "GET /metrics HTTP/1.1" 200 36764 "" "Prometheus/2.51.0" 2024-09-21T11:05:53.568 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:53.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:53.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:53.670 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:53.691 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:53.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:53.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:53.793 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:53.815 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:53.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:53.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:53 smithi160 ceph-mon[101879]: pgmap v3250: 109 pgs: 2 peering, 2 active+remapped+backfilling, 2 active+undersized+remapped, 2 active+undersized+degraded+remapped, 3 active+clean+remapped, 6 active+remapped+backfill_wait, 92 active+clean; 493 MiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 273 op/s; 1028/23700 objects degraded (4.338%); 6074/23700 objects misplaced (25.629%); 34 MiB/s, 552 objects/s recovering 2024-09-21T11:05:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:53 smithi160 ceph-mon[101879]: 1.e scrub starts 2024-09-21T11:05:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:53 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 1028/23700 objects degraded (4.338%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T11:05:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:53 smithi160 ceph-mon[101879]: osdmap e1642: 8 total, 8 up, 6 in 2024-09-21T11:05:53.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:53 smithi070 ceph-mon[107385]: pgmap v3250: 109 pgs: 2 peering, 2 active+remapped+backfilling, 2 active+undersized+remapped, 2 active+undersized+degraded+remapped, 3 active+clean+remapped, 6 active+remapped+backfill_wait, 92 active+clean; 493 MiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 273 op/s; 1028/23700 objects degraded (4.338%); 6074/23700 objects misplaced (25.629%); 34 MiB/s, 552 objects/s recovering 2024-09-21T11:05:53.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:53 smithi070 ceph-mon[107385]: 1.e scrub starts 2024-09-21T11:05:53.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:53 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 1028/23700 objects degraded (4.338%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T11:05:53.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:53 smithi070 ceph-mon[107385]: osdmap e1642: 8 total, 8 up, 6 in 2024-09-21T11:05:53.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:53 smithi070 ceph-mon[112139]: pgmap v3250: 109 pgs: 2 peering, 2 active+remapped+backfilling, 2 active+undersized+remapped, 2 active+undersized+degraded+remapped, 3 active+clean+remapped, 6 active+remapped+backfill_wait, 92 active+clean; 493 MiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 273 op/s; 1028/23700 objects degraded (4.338%); 6074/23700 objects misplaced (25.629%); 34 MiB/s, 552 objects/s recovering 2024-09-21T11:05:53.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:53 smithi070 ceph-mon[112139]: 1.e scrub starts 2024-09-21T11:05:53.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:53 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 1028/23700 objects degraded (4.338%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T11:05:53.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:53 smithi070 ceph-mon[112139]: osdmap e1642: 8 total, 8 up, 6 in 2024-09-21T11:05:53.916 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:53.937 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:53.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:53.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:54.039 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:54.061 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:54.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:54.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:54.162 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:54.185 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:54.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:54.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:54.287 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:54.308 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:54.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:54.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:54.410 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:54.431 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:54.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:54.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:54.533 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:54.562 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:54.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:54.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:54.564 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T11:05:54.592 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7, 4, 0] out_osds: [2, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 7, 6, 5] 2024-09-21T11:05:54.593 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T11:05:54.593 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-21T11:05:54.593 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T11:05:54.663 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:54.692 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:54.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:54.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:54.794 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:54.815 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:54.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:54.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:54.917 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:54.938 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:54.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:54.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:55.040 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:55.061 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:55.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:55.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:55.162 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:55.186 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:55.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:55.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:55.287 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:55.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:55.314 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:55.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:55.415 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:55.437 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:55.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:55.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:55.539 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:55.569 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:55.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:55.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:55.671 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:55.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:55.694 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:55.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:55.729 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T11:05:55.730 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T11:05:55.730 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T11:05:55.730 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T11:05:55.796 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:55.817 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:55.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:55.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:55.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:55 smithi160 ceph-mon[101879]: pgmap v3253: 109 pgs: 2 peering, 2 active+remapped+backfilling, 2 active+undersized+remapped, 2 active+undersized+degraded+remapped, 3 active+clean+remapped, 6 active+remapped+backfill_wait, 92 active+clean; 493 MiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 121 op/s; 1028/23700 objects degraded (4.338%); 6074/23700 objects misplaced (25.629%); 24 MiB/s, 380 objects/s recovering 2024-09-21T11:05:55.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:55 smithi160 ceph-mon[101879]: osdmap e1643: 8 total, 8 up, 6 in 2024-09-21T11:05:55.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:55 smithi070 ceph-mon[107385]: pgmap v3253: 109 pgs: 2 peering, 2 active+remapped+backfilling, 2 active+undersized+remapped, 2 active+undersized+degraded+remapped, 3 active+clean+remapped, 6 active+remapped+backfill_wait, 92 active+clean; 493 MiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 121 op/s; 1028/23700 objects degraded (4.338%); 6074/23700 objects misplaced (25.629%); 24 MiB/s, 380 objects/s recovering 2024-09-21T11:05:55.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:55 smithi070 ceph-mon[107385]: osdmap e1643: 8 total, 8 up, 6 in 2024-09-21T11:05:55.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:55 smithi070 ceph-mon[112139]: pgmap v3253: 109 pgs: 2 peering, 2 active+remapped+backfilling, 2 active+undersized+remapped, 2 active+undersized+degraded+remapped, 3 active+clean+remapped, 6 active+remapped+backfill_wait, 92 active+clean; 493 MiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 121 op/s; 1028/23700 objects degraded (4.338%); 6074/23700 objects misplaced (25.629%); 24 MiB/s, 380 objects/s recovering 2024-09-21T11:05:55.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:55 smithi070 ceph-mon[112139]: osdmap e1643: 8 total, 8 up, 6 in 2024-09-21T11:05:55.920 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:55.941 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:55.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:55.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:56.043 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:56.065 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:56.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:56.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:56.167 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:56.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:56.189 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:56.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:56.290 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:56.311 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:56.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:56.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:56.413 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:56.436 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:56.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:56.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:56.537 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:56.559 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:56.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:56.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:56.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:56 smithi160 ceph-mon[101879]: osdmap e1644: 8 total, 8 up, 6 in 2024-09-21T11:05:56.661 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:56.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:56.694 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:56.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:56.795 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:56.818 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:56.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:56.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:56.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:56 smithi070 ceph-mon[107385]: osdmap e1644: 8 total, 8 up, 6 in 2024-09-21T11:05:56.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:56 smithi070 ceph-mon[112139]: osdmap e1644: 8 total, 8 up, 6 in 2024-09-21T11:05:56.920 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:56.941 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:56.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:56.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:57.043 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:57.064 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:57.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:57.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:57.166 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:05:57.188 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:57.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:57.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:05:57.290 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:57.312 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:57.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:57.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:57.414 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:57.435 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:57.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:57.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:57.537 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:57.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:57.560 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:57.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:57.661 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:57.690 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:57.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:57.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:57.791 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:57.823 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:57.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:57.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:57 smithi160 ceph-mon[101879]: pgmap v3256: 109 pgs: 1 peering, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 104 active+clean; 430 MiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 250 op/s; 862/20688 objects degraded (4.167%); 2606/20688 objects misplaced (12.597%); 40 MiB/s, 643 objects/s recovering 2024-09-21T11:05:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:57 smithi160 ceph-mon[101879]: osdmap e1645: 8 total, 8 up, 6 in 2024-09-21T11:05:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:57 smithi160 ceph-mon[101879]: 1.7 scrub starts 2024-09-21T11:05:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:57 smithi160 ceph-mon[101879]: 1.7 scrub ok 2024-09-21T11:05:57.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:57 smithi070 ceph-mon[107385]: pgmap v3256: 109 pgs: 1 peering, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 104 active+clean; 430 MiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 250 op/s; 862/20688 objects degraded (4.167%); 2606/20688 objects misplaced (12.597%); 40 MiB/s, 643 objects/s recovering 2024-09-21T11:05:57.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:57 smithi070 ceph-mon[107385]: osdmap e1645: 8 total, 8 up, 6 in 2024-09-21T11:05:57.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:57 smithi070 ceph-mon[107385]: 1.7 scrub starts 2024-09-21T11:05:57.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:57 smithi070 ceph-mon[107385]: 1.7 scrub ok 2024-09-21T11:05:57.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:57 smithi070 ceph-mon[112139]: pgmap v3256: 109 pgs: 1 peering, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 104 active+clean; 430 MiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 250 op/s; 862/20688 objects degraded (4.167%); 2606/20688 objects misplaced (12.597%); 40 MiB/s, 643 objects/s recovering 2024-09-21T11:05:57.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:57 smithi070 ceph-mon[112139]: osdmap e1645: 8 total, 8 up, 6 in 2024-09-21T11:05:57.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:57 smithi070 ceph-mon[112139]: 1.7 scrub starts 2024-09-21T11:05:57.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:57 smithi070 ceph-mon[112139]: 1.7 scrub ok 2024-09-21T11:05:57.925 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:05:57.963 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:57.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:57.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:05:58.064 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:58.097 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:58.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:58.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:58.199 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:58.223 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:58.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:58.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:58.324 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:58.355 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:58.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:58.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:58.457 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:58.478 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:58.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:58.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:58.580 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:58.608 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:58.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:58.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:58.709 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:05:58.748 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:58.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:58.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:05:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:58 smithi160 ceph-mon[101879]: osdmap e1646: 8 total, 8 up, 6 in 2024-09-21T11:05:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:58 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 862/20688 objects degraded (4.167%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T11:05:58.849 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:58.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:58 smithi070 ceph-mon[107385]: osdmap e1646: 8 total, 8 up, 6 in 2024-09-21T11:05:58.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:58 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 862/20688 objects degraded (4.167%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T11:05:58.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:58 smithi070 ceph-mon[112139]: osdmap e1646: 8 total, 8 up, 6 in 2024-09-21T11:05:58.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:58 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 862/20688 objects degraded (4.167%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T11:05:58.878 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:58.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:58.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:58.980 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:59.006 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:59.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:59.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:59.108 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:05:59.143 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:05:59.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:59.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:05:59.245 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:59.274 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:59.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:59.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:05:59.376 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:59.406 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:59.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:59.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:59.448 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T11:05:59.508 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:05:59.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:59.540 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:59.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:05:59.557 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:05:59.641 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:05:59.651 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:05:59.665 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:59.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:59.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:05:59.669 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:59 smithi070 ceph-mon[107385]: 1.0 scrub starts 2024-09-21T11:05:59.669 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:59 smithi070 ceph-mon[107385]: 1.0 scrub ok 2024-09-21T11:05:59.669 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:05:59 smithi070 ceph-mon[107385]: pgmap v3259: 109 pgs: 1 peering, 2 active+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 105 active+clean; 421 MiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 287 op/s; 862/20244 objects degraded (4.258%); 1682/20244 objects misplaced (8.309%); 47 MiB/s, 754 objects/s recovering 2024-09-21T11:05:59.669 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:59 smithi070 ceph-mon[112139]: 1.0 scrub starts 2024-09-21T11:05:59.669 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:59 smithi070 ceph-mon[112139]: 1.0 scrub ok 2024-09-21T11:05:59.669 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:05:59 smithi070 ceph-mon[112139]: pgmap v3259: 109 pgs: 1 peering, 2 active+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 105 active+clean; 421 MiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 287 op/s; 862/20244 objects degraded (4.258%); 1682/20244 objects misplaced (8.309%); 47 MiB/s, 754 objects/s recovering 2024-09-21T11:05:59.767 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:05:59.800 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:59.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:59.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:05:59.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:59 smithi160 ceph-mon[101879]: 1.0 scrub starts 2024-09-21T11:05:59.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:59 smithi160 ceph-mon[101879]: 1.0 scrub ok 2024-09-21T11:05:59.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:05:59 smithi160 ceph-mon[101879]: pgmap v3259: 109 pgs: 1 peering, 2 active+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 105 active+clean; 421 MiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 287 op/s; 862/20244 objects degraded (4.258%); 1682/20244 objects misplaced (8.309%); 47 MiB/s, 754 objects/s recovering 2024-09-21T11:05:59.902 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:05:59.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:05:59.933 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:05:59.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:00.034 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:00.080 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:00.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:00.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:00.182 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:00.214 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:00.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:00.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:00.316 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:00.346 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:00.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:00.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:00.448 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:00.482 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:00.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:00.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:00.549 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:00 smithi070 ceph-mon[107385]: 1.f scrub starts 2024-09-21T11:06:00.550 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:00 smithi070 ceph-mon[107385]: 1.f scrub ok 2024-09-21T11:06:00.584 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:00.627 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:00.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:00.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:00.729 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:00.758 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:00.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:00.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:00 smithi160 ceph-mon[101879]: 1.f scrub starts 2024-09-21T11:06:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:00 smithi160 ceph-mon[101879]: 1.f scrub ok 2024-09-21T11:06:00.843 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:00 smithi070 ceph-mon[112139]: 1.f scrub starts 2024-09-21T11:06:00.843 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:00 smithi070 ceph-mon[112139]: 1.f scrub ok 2024-09-21T11:06:00.860 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:00.896 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:00.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:00.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:00.998 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:01.025 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:01.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:01.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:01.126 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:01.173 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:01.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:01.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:01.275 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:01.310 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:01.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:01.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:01.412 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:01.449 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:01.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:01.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:01.550 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:01.564 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is set 2024-09-21T11:06:01.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:01.585 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:01.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:01.686 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:01.728 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:01.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:01.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:01.830 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:01.834 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:01 smithi070 ceph-mon[107385]: 1.10 scrub starts 2024-09-21T11:06:01.834 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:01 smithi070 ceph-mon[107385]: 1.10 scrub ok 2024-09-21T11:06:01.834 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:01 smithi070 ceph-mon[107385]: pgmap v3260: 109 pgs: 1 peering, 2 active+undersized+degraded+remapped, 106 active+clean; 308 MiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 505 op/s; 695/14820 objects degraded (4.690%); 1147/14820 objects misplaced (7.740%); 36 MiB/s, 580 objects/s recovering 2024-09-21T11:06:01.834 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:01 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/567544585' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T11:06:01.834 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:01 smithi070 ceph-mon[107385]: 1.6 scrub starts 2024-09-21T11:06:01.834 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:01 smithi070 ceph-mon[107385]: 1.6 scrub ok 2024-09-21T11:06:01.834 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:01 smithi070 ceph-mon[112139]: 1.10 scrub starts 2024-09-21T11:06:01.835 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:01 smithi070 ceph-mon[112139]: 1.10 scrub ok 2024-09-21T11:06:01.835 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:01 smithi070 ceph-mon[112139]: pgmap v3260: 109 pgs: 1 peering, 2 active+undersized+degraded+remapped, 106 active+clean; 308 MiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 505 op/s; 695/14820 objects degraded (4.690%); 1147/14820 objects misplaced (7.740%); 36 MiB/s, 580 objects/s recovering 2024-09-21T11:06:01.835 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:01 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/567544585' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T11:06:01.835 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:01 smithi070 ceph-mon[112139]: 1.6 scrub starts 2024-09-21T11:06:01.835 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:01 smithi070 ceph-mon[112139]: 1.6 scrub ok 2024-09-21T11:06:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:01 smithi160 ceph-mon[101879]: 1.10 scrub starts 2024-09-21T11:06:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:01 smithi160 ceph-mon[101879]: 1.10 scrub ok 2024-09-21T11:06:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:01 smithi160 ceph-mon[101879]: pgmap v3260: 109 pgs: 1 peering, 2 active+undersized+degraded+remapped, 106 active+clean; 308 MiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 505 op/s; 695/14820 objects degraded (4.690%); 1147/14820 objects misplaced (7.740%); 36 MiB/s, 580 objects/s recovering 2024-09-21T11:06:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:01 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/567544585' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T11:06:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:01 smithi160 ceph-mon[101879]: 1.6 scrub starts 2024-09-21T11:06:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:01 smithi160 ceph-mon[101879]: 1.6 scrub ok 2024-09-21T11:06:01.863 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:01.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:01.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:01.965 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:02.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:02.004 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:02.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:02.105 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:02.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:02.142 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:02.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:02.243 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:02.279 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:02.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:02.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:02.381 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:02.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:02.413 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:02.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:02.456 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Removed 145956 objects 2024-09-21T11:06:02.456 INFO:tasks.radosbench.radosbench.0.smithi070.stdout:Clean up completed and total clean up time :153.114 2024-09-21T11:06:02.515 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:02.536 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:02.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:02.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:02.638 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:02.659 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:02.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:02.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:02.761 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:02.783 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:02.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:02.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:02 smithi160 ceph-mon[101879]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:06:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:02 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/567544585' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T11:06:02.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:02 smithi160 ceph-mon[101879]: osdmap e1647: 8 total, 8 up, 6 in 2024-09-21T11:06:02.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:02 smithi070 ceph-mon[107385]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:06:02.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:02 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/567544585' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T11:06:02.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:02 smithi070 ceph-mon[107385]: osdmap e1647: 8 total, 8 up, 6 in 2024-09-21T11:06:02.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:02 smithi070 ceph-mon[112139]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:06:02.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:02 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/567544585' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T11:06:02.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:02 smithi070 ceph-mon[112139]: osdmap e1647: 8 total, 8 up, 6 in 2024-09-21T11:06:02.884 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:02.906 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:02.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:02.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:03.007 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:03.029 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:03.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:03.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:03.131 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:03.151 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:03.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:03.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:03.253 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:03.274 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:03.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:03.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:03.376 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:03.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:06:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:06:03] "GET /metrics HTTP/1.1" 200 36764 "" "Prometheus/2.51.0" 2024-09-21T11:06:03.400 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:03.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:03.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:03.501 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:03.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:03.524 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:03.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:03.625 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:03.647 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:03.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:03.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:03.748 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:03.774 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:03.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:03.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:03.804 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:06:03.819 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:03 smithi070 ceph-mon[107385]: pgmap v3262: 109 pgs: 2 active+undersized+degraded+remapped, 107 active+clean; 171 MiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 688 op/s; 387/8268 objects degraded (4.681%); 531/8268 objects misplaced (6.422%); 8.4 MiB/s, 134 objects/s recovering 2024-09-21T11:06:03.819 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:03 smithi070 ceph-mon[107385]: Health check update: Degraded data redundancy: 387/8268 objects degraded (4.681%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T11:06:03.819 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:03 smithi070 ceph-mon[112139]: pgmap v3262: 109 pgs: 2 active+undersized+degraded+remapped, 107 active+clean; 171 MiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 688 op/s; 387/8268 objects degraded (4.681%); 531/8268 objects misplaced (6.422%); 8.4 MiB/s, 134 objects/s recovering 2024-09-21T11:06:03.819 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:03 smithi070 ceph-mon[112139]: Health check update: Degraded data redundancy: 387/8268 objects degraded (4.681%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T11:06:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:03 smithi160 ceph-mon[101879]: pgmap v3262: 109 pgs: 2 active+undersized+degraded+remapped, 107 active+clean; 171 MiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 688 op/s; 387/8268 objects degraded (4.681%); 531/8268 objects misplaced (6.422%); 8.4 MiB/s, 134 objects/s recovering 2024-09-21T11:06:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:03 smithi160 ceph-mon[101879]: Health check update: Degraded data redundancy: 387/8268 objects degraded (4.681%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T11:06:03.876 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:03.898 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:03.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:03.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:03.990 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T11:06:03.991 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T11:06:04.000 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:04.025 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:04.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:04.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:04.126 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:04.153 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:04.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:04.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:04.255 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:04.280 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:04.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:04.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:04.382 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:04.403 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:04.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:04.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:04.505 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:04.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:04.530 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:04.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:04.631 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:04.653 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:04.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:04.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:04.754 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:04.775 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:04.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:04.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:04.877 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:04.898 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:04.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:04.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:05.000 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:05.022 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:05.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:05.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:05.073 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T11:06:05.123 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:05.144 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:05.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:05.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:05.245 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:05.267 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:05.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:05.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:05.369 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:05.391 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:05.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:05.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:05.493 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:05.515 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:05.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:05.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:05.617 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:05.638 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:05.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:05.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:05.682 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":3262,"stamp":"2024-09-21T11:06:02.346086+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179315196,"num_objects":2756,"num_object_clones":0,"num_object_copies":8268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":387,"num_objects_misplaced":531,"num_objects_unfound":0,"num_objects_dirty":2756,"num_whiteouts":0,"num_read":87,"num_read_kb":188,"num_write":289297,"num_write_kb":9341767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20810,"num_bytes_recovered":1270123918,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76231,"ondisk_log_size":76231,"up":326,"acting":325,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":335,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":6912908,"kb_used_data":730260,"kb_used_omap":397,"kb_used_meta":6182130,"kb_avail":555558004,"statfs":{"total":575970213888,"available":568891396096,"internally_reserved":0,"allocated":747786240,"data_stored":720995487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":406920,"internal_metadata":6330501752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,2,10,2,7,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-271345874,"num_objects":-4140,"num_object_clones":0,"num_object_copies":-12420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-475,"num_objects_misplaced":-2075,"num_objects_unfound":0,"num_objects_dirty":-4140,"num_whiteouts":0,"num_read":-17,"num_read_kb":-37,"num_write":4131,"num_write_kb":-109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":810,"num_bytes_recovered":52949986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002876"},"pg_stats":[{"pgid":"58.f","version":"1646'17916","reported_seq":23544,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:59.684570+0000","last_change":"2024-09-21T11:05:54.545340+0000","last_active":"2024-09-21T11:05:59.684570+0000","last_peered":"2024-09-21T11:05:59.684570+0000","last_clean":"2024-09-21T11:05:59.684570+0000","last_became_active":"2024-09-21T11:05:54.515248+0000","last_became_peered":"2024-09-21T11:05:54.515248+0000","last_unstale":"2024-09-21T11:05:59.684570+0000","last_undegraded":"2024-09-21T11:05:59.684570+0000","last_fullsized":"2024-09-21T11:05:59.684570+0000","mapping_epoch":1642,"log_start":"1603'13264","ondisk_log_start":"1603'13264","created":1581,"last_epoch_clean":1643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":4652,"log_dups_size":0,"ondisk_log_size":4652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:25:26.383716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14024704,"num_objects":214,"num_object_clones":0,"num_object_copies":642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17916,"num_write_kb":580160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2728,"num_bytes_recovered":178782208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"58.e","version":"1646'17795","reported_seq":19447,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:59.924566+0000","last_change":"2024-09-21T11:05:51.544819+0000","last_active":"2024-09-21T11:05:59.924566+0000","last_peered":"2024-09-21T11:05:59.924566+0000","last_clean":"2024-09-21T11:05:59.924566+0000","last_became_active":"2024-09-21T11:05:51.515954+0000","last_became_peered":"2024-09-21T11:05:51.515954+0000","last_unstale":"2024-09-21T11:05:59.924566+0000","last_undegraded":"2024-09-21T11:05:59.924566+0000","last_fullsized":"2024-09-21T11:05:59.924566+0000","mapping_epoch":1639,"log_start":"1601'12591","ondisk_log_start":"1601'12591","created":1581,"last_epoch_clean":1640,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":5204,"log_dups_size":0,"ondisk_log_size":5204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:09:19.439202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14221312,"num_objects":217,"num_object_clones":0,"num_object_copies":651,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17795,"num_write_kb":576384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":489,"num_bytes_recovered":31981568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"58.d","version":"1646'18100","reported_seq":20903,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:59.915895+0000","last_change":"2024-09-21T11:05:54.574999+0000","last_active":"2024-09-21T11:05:59.915895+0000","last_peered":"2024-09-21T11:05:59.915895+0000","last_clean":"2024-09-21T11:05:59.915895+0000","last_became_active":"2024-09-21T11:05:54.515301+0000","last_became_peered":"2024-09-21T11:05:54.515301+0000","last_unstale":"2024-09-21T11:05:59.915895+0000","last_undegraded":"2024-09-21T11:05:59.915895+0000","last_fullsized":"2024-09-21T11:05:59.915895+0000","mapping_epoch":1642,"log_start":"1603'13544","ondisk_log_start":"1603'13544","created":1581,"last_epoch_clean":1643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":4556,"log_dups_size":0,"ondisk_log_size":4556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:28:57.905130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13434904,"num_objects":206,"num_object_clones":0,"num_object_copies":618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18100,"num_write_kb":585729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1034,"num_bytes_recovered":67633200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"58.c","version":"1646'18204","reported_seq":22004,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:06:01.025448+0000","last_change":"2024-09-21T11:05:55.542645+0000","last_active":"2024-09-21T11:06:01.025448+0000","last_peered":"2024-09-21T11:06:01.025448+0000","last_clean":"2024-09-21T11:06:01.025448+0000","last_became_active":"2024-09-21T11:05:55.520121+0000","last_became_peered":"2024-09-21T11:05:55.520121+0000","last_unstale":"2024-09-21T11:06:01.025448+0000","last_undegraded":"2024-09-21T11:06:01.025448+0000","last_fullsized":"2024-09-21T11:06:01.025448+0000","mapping_epoch":1643,"log_start":"1603'13627","ondisk_log_start":"1603'13627","created":1581,"last_epoch_clean":1644,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":4577,"log_dups_size":0,"ondisk_log_size":4577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:59:00.880044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7340032,"num_objects":112,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18204,"num_write_kb":586112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1839,"num_bytes_recovered":120520704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"58.b","version":"1646'18042","reported_seq":20750,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:59.696764+0000","last_change":"2024-09-21T11:05:45.532263+0000","last_active":"2024-09-21T11:05:59.696764+0000","last_peered":"2024-09-21T11:05:59.696764+0000","last_clean":"2024-09-21T11:05:59.696764+0000","last_became_active":"2024-09-21T11:05:45.510076+0000","last_became_peered":"2024-09-21T11:05:45.510076+0000","last_unstale":"2024-09-21T11:05:59.696764+0000","last_undegraded":"2024-09-21T11:05:59.696764+0000","last_fullsized":"2024-09-21T11:05:59.696764+0000","mapping_epoch":1633,"log_start":"1603'13438","ondisk_log_start":"1603'13438","created":1581,"last_epoch_clean":1634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":4604,"log_dups_size":0,"ondisk_log_size":4604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:00.566333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15597568,"num_objects":238,"num_object_clones":0,"num_object_copies":714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18042,"num_write_kb":584960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1272,"num_bytes_recovered":83361792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"58.a","version":"1646'18008","reported_seq":20496,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:59.672088+0000","last_change":"2024-09-21T11:05:57.580335+0000","last_active":"2024-09-21T11:05:59.672088+0000","last_peered":"2024-09-21T11:05:59.672088+0000","last_clean":"2024-09-21T11:05:59.672088+0000","last_became_active":"2024-09-21T11:05:57.530021+0000","last_became_peered":"2024-09-21T11:05:57.530021+0000","last_unstale":"2024-09-21T11:05:59.672088+0000","last_undegraded":"2024-09-21T11:05:59.672088+0000","last_fullsized":"2024-09-21T11:05:59.672088+0000","mapping_epoch":1645,"log_start":"1603'13566","ondisk_log_start":"1603'13566","created":1581,"last_epoch_clean":1646,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":4442,"log_dups_size":0,"ondisk_log_size":4442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:20:03.292605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13762560,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18008,"num_write_kb":582976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1176,"num_bytes_recovered":77070336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"58.9","version":"1646'18046","reported_seq":29120,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:59.931548+0000","last_change":"2024-09-21T11:05:04.490170+0000","last_active":"2024-09-21T11:05:59.931548+0000","last_peered":"2024-09-21T11:05:59.931548+0000","last_clean":"2024-09-21T11:05:59.931548+0000","last_became_active":"2024-09-21T11:05:04.416131+0000","last_became_peered":"2024-09-21T11:05:04.416131+0000","last_unstale":"2024-09-21T11:05:59.931548+0000","last_undegraded":"2024-09-21T11:05:59.931548+0000","last_fullsized":"2024-09-21T11:05:59.931548+0000","mapping_epoch":1620,"log_start":"1601'12745","ondisk_log_start":"1601'12745","created":1581,"last_epoch_clean":1621,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":5301,"log_dups_size":0,"ondisk_log_size":5301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:20:04.696973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13238272,"num_objects":202,"num_object_clones":0,"num_object_copies":606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18046,"num_write_kb":583936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5454,"num_bytes_recovered":269418496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"58.8","version":"1646'17769","reported_seq":21830,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:59.705322+0000","last_change":"2024-09-21T11:05:53.538429+0000","last_active":"2024-09-21T11:05:59.705322+0000","last_peered":"2024-09-21T11:05:59.705322+0000","last_clean":"2024-09-21T11:05:59.705322+0000","last_became_active":"2024-09-21T11:05:53.511766+0000","last_became_peered":"2024-09-21T11:05:53.511766+0000","last_unstale":"2024-09-21T11:05:59.705322+0000","last_undegraded":"2024-09-21T11:05:59.705322+0000","last_fullsized":"2024-09-21T11:05:59.705322+0000","mapping_epoch":1641,"log_start":"1603'13221","ondisk_log_start":"1603'13221","created":1581,"last_epoch_clean":1642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:31:57.485362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16711680,"num_objects":255,"num_object_clones":0,"num_object_copies":765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17769,"num_write_kb":576768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1451,"num_bytes_recovered":94961664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"58.7","version":"1646'18027","reported_seq":20198,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:59.905938+0000","last_change":"2024-09-21T11:05:49.614289+0000","last_active":"2024-09-21T11:05:59.905938+0000","last_peered":"2024-09-21T11:05:59.905938+0000","last_clean":"2024-09-21T11:05:59.905938+0000","last_became_active":"2024-09-21T11:05:49.528860+0000","last_became_peered":"2024-09-21T11:05:49.528860+0000","last_unstale":"2024-09-21T11:05:59.905938+0000","last_undegraded":"2024-09-21T11:05:59.905938+0000","last_fullsized":"2024-09-21T11:05:59.905938+0000","mapping_epoch":1637,"log_start":"1603'13484","ondisk_log_start":"1603'13484","created":1581,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":4543,"log_dups_size":0,"ondisk_log_size":4543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:36:59.935606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13434880,"num_objects":205,"num_object_clones":0,"num_object_copies":615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18027,"num_write_kb":583424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":513,"num_bytes_recovered":33619968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"58.6","version":"1647'18452","reported_seq":20021,"reported_epoch":1647,"state":"active+clean","last_fresh":"2024-09-21T11:06:01.831202+0000","last_change":"2024-09-21T11:05:56.556257+0000","last_active":"2024-09-21T11:06:01.831202+0000","last_peered":"2024-09-21T11:06:01.831202+0000","last_clean":"2024-09-21T11:06:01.831202+0000","last_became_active":"2024-09-21T11:05:56.524179+0000","last_became_peered":"2024-09-21T11:05:56.524179+0000","last_unstale":"2024-09-21T11:06:01.831202+0000","last_undegraded":"2024-09-21T11:06:01.831202+0000","last_fullsized":"2024-09-21T11:06:01.831202+0000","mapping_epoch":1644,"log_start":"1612'14658","ondisk_log_start":"1612'14658","created":1581,"last_epoch_clean":1645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":3794,"log_dups_size":0,"ondisk_log_size":3794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:00:00.208129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3407872,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18452,"num_write_kb":592128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":482,"num_bytes_recovered":31588352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"58.5","version":"1646'18325","reported_seq":19337,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:06:01.024720+0000","last_change":"2024-09-21T11:05:56.560595+0000","last_active":"2024-09-21T11:06:01.024720+0000","last_peered":"2024-09-21T11:06:01.024720+0000","last_clean":"2024-09-21T11:06:01.024720+0000","last_became_active":"2024-09-21T11:05:56.524945+0000","last_became_peered":"2024-09-21T11:05:56.524945+0000","last_unstale":"2024-09-21T11:06:01.024720+0000","last_undegraded":"2024-09-21T11:06:01.024720+0000","last_fullsized":"2024-09-21T11:06:01.024720+0000","mapping_epoch":1644,"log_start":"1603'13690","ondisk_log_start":"1603'13690","created":1581,"last_epoch_clean":1645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":4635,"log_dups_size":0,"ondisk_log_size":4635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:50:16.641697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8192000,"num_objects":125,"num_object_clones":0,"num_object_copies":375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18325,"num_write_kb":590400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":29949952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"58.4","version":"1647'18237","reported_seq":19449,"reported_epoch":1647,"state":"active+clean","last_fresh":"2024-09-21T11:06:01.809676+0000","last_change":"2024-09-21T11:05:49.559489+0000","last_active":"2024-09-21T11:06:01.809676+0000","last_peered":"2024-09-21T11:06:01.809676+0000","last_clean":"2024-09-21T11:06:01.809676+0000","last_became_active":"2024-09-21T11:05:49.527597+0000","last_became_peered":"2024-09-21T11:05:49.527597+0000","last_unstale":"2024-09-21T11:06:01.809676+0000","last_undegraded":"2024-09-21T11:06:01.809676+0000","last_fullsized":"2024-09-21T11:06:01.809676+0000","mapping_epoch":1637,"log_start":"1614'14485","ondisk_log_start":"1614'14485","created":1581,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":3752,"log_dups_size":0,"ondisk_log_size":3752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:28:11.652152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3473408,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18237,"num_write_kb":585280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":534,"num_bytes_recovered":34996224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"58.3","version":"1646'18114","reported_seq":18229,"reported_epoch":1646,"state":"active+undersized+degraded+remapped","last_fresh":"2024-09-21T11:06:01.012746+0000","last_change":"2024-09-21T11:05:48.647320+0000","last_active":"2024-09-21T11:06:01.012746+0000","last_peered":"2024-09-21T11:06:01.012746+0000","last_clean":"2024-09-21T11:01:48.463662+0000","last_became_active":"2024-09-21T11:05:48.623525+0000","last_became_peered":"2024-09-21T11:05:48.623525+0000","last_unstale":"2024-09-21T11:06:01.012746+0000","last_undegraded":"2024-09-21T11:05:48.548520+0000","last_fullsized":"2024-09-21T11:05:48.508674+0000","mapping_epoch":1636,"log_start":"1603'13442","ondisk_log_start":"1603'13442","created":1581,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":4672,"log_dups_size":0,"ondisk_log_size":4672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:48.522093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9437184,"num_objects":144,"num_object_clones":0,"num_object_copies":432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":144,"num_objects_misplaced":288,"num_objects_unfound":0,"num_objects_dirty":144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18114,"num_write_kb":584256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":144}],"blocked_by":[],"up_primary":3,"acting_primary":4,"purged_snaps":[]},{"pgid":"58.2","version":"1646'18158","reported_seq":24065,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:06:01.009224+0000","last_change":"2024-09-21T11:05:52.538502+0000","last_active":"2024-09-21T11:06:01.009224+0000","last_peered":"2024-09-21T11:06:01.009224+0000","last_clean":"2024-09-21T11:06:01.009224+0000","last_became_active":"2024-09-21T11:05:52.516412+0000","last_became_peered":"2024-09-21T11:05:52.516412+0000","last_unstale":"2024-09-21T11:06:01.009224+0000","last_undegraded":"2024-09-21T11:06:01.009224+0000","last_fullsized":"2024-09-21T11:06:01.009224+0000","mapping_epoch":1640,"log_start":"1603'13566","ondisk_log_start":"1603'13566","created":1581,"last_epoch_clean":1641,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":4592,"log_dups_size":0,"ondisk_log_size":4592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:05:29.826855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8519680,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18158,"num_write_kb":585216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2797,"num_bytes_recovered":183304192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"58.1","version":"1646'17711","reported_seq":17882,"reported_epoch":1646,"state":"active+undersized+degraded+remapped","last_fresh":"2024-09-21T11:05:59.696911+0000","last_change":"2024-09-21T11:05:48.645814+0000","last_active":"2024-09-21T11:05:59.696911+0000","last_peered":"2024-09-21T11:05:59.696911+0000","last_clean":"2024-09-21T11:01:48.463662+0000","last_became_active":"2024-09-21T11:05:48.606730+0000","last_became_peered":"2024-09-21T11:05:48.606730+0000","last_unstale":"2024-09-21T11:05:59.696911+0000","last_undegraded":"2024-09-21T11:05:48.528379+0000","last_fullsized":"2024-09-21T11:05:48.506599+0000","mapping_epoch":1636,"log_start":"1603'13059","ondisk_log_start":"1603'13059","created":1581,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":4652,"log_dups_size":0,"ondisk_log_size":4652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15925248,"num_objects":243,"num_object_clones":0,"num_object_copies":729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":243,"num_objects_misplaced":243,"num_objects_unfound":0,"num_objects_dirty":243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17711,"num_write_kb":574528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":243}],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[]},{"pgid":"58.0","version":"1646'18280","reported_seq":19811,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:06:01.020969+0000","last_change":"2024-09-21T11:05:50.553547+0000","last_active":"2024-09-21T11:06:01.020969+0000","last_peered":"2024-09-21T11:06:01.020969+0000","last_clean":"2024-09-21T11:06:01.020969+0000","last_became_active":"2024-09-21T11:05:50.519712+0000","last_became_peered":"2024-09-21T11:05:50.519712+0000","last_unstale":"2024-09-21T11:06:01.020969+0000","last_undegraded":"2024-09-21T11:06:01.020969+0000","last_fullsized":"2024-09-21T11:06:01.020969+0000","mapping_epoch":1638,"log_start":"1603'13761","ondisk_log_start":"1603'13761","created":1581,"last_epoch_clean":1639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T11:01:48.463662+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.463662+0000","objects_scrubbed":0,"log_size":4519,"log_dups_size":0,"ondisk_log_size":4519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:16:13.470607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8126464,"num_objects":124,"num_object_clones":0,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18280,"num_write_kb":588928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":475,"num_bytes_recovered":31129600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":36,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.527786+0000","last_change":"2024-09-21T11:05:48.544347+0000","last_active":"2024-09-21T11:05:57.527786+0000","last_peered":"2024-09-21T11:05:57.527786+0000","last_clean":"2024-09-21T11:05:57.527786+0000","last_became_active":"2024-09-21T11:05:48.529869+0000","last_became_peered":"2024-09-21T11:05:48.529869+0000","last_unstale":"2024-09-21T11:05:57.527786+0000","last_undegraded":"2024-09-21T11:05:57.527786+0000","last_fullsized":"2024-09-21T11:05:57.527786+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:18:15.635403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1564,"reported_epoch":1645,"state":"active+clean","last_fresh":"2024-09-21T11:05:56.519468+0000","last_change":"2024-09-21T11:02:39.486670+0000","last_active":"2024-09-21T11:05:56.519468+0000","last_peered":"2024-09-21T11:05:56.519468+0000","last_clean":"2024-09-21T11:05:56.519468+0000","last_became_active":"2024-09-21T11:02:39.486248+0000","last_became_peered":"2024-09-21T11:02:39.486248+0000","last_unstale":"2024-09-21T11:05:56.519468+0000","last_undegraded":"2024-09-21T11:05:56.519468+0000","last_fullsized":"2024-09-21T11:05:56.519468+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:56:56.360708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":72,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.530520+0000","last_change":"2024-09-21T11:04:54.654602+0000","last_active":"2024-09-21T11:05:57.530520+0000","last_peered":"2024-09-21T11:05:57.530520+0000","last_clean":"2024-09-21T11:05:57.530520+0000","last_became_active":"2024-09-21T11:04:54.654231+0000","last_became_peered":"2024-09-21T11:04:54.654231+0000","last_unstale":"2024-09-21T11:05:57.530520+0000","last_undegraded":"2024-09-21T11:05:57.530520+0000","last_fullsized":"2024-09-21T11:05:57.530520+0000","mapping_epoch":1615,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:53:49.828685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":79,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.529322+0000","last_change":"2024-09-21T11:05:48.533874+0000","last_active":"2024-09-21T11:05:57.529322+0000","last_peered":"2024-09-21T11:05:57.529322+0000","last_clean":"2024-09-21T11:05:57.529322+0000","last_became_active":"2024-09-21T11:05:48.533558+0000","last_became_peered":"2024-09-21T11:05:48.533558+0000","last_unstale":"2024-09-21T11:05:57.529322+0000","last_undegraded":"2024-09-21T11:05:57.529322+0000","last_fullsized":"2024-09-21T11:05:57.529322+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:01:28.336817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":141,"reported_epoch":1645,"state":"active+clean","last_fresh":"2024-09-21T11:05:56.522023+0000","last_change":"2024-09-21T11:02:39.502513+0000","last_active":"2024-09-21T11:05:56.522023+0000","last_peered":"2024-09-21T11:05:56.522023+0000","last_clean":"2024-09-21T11:05:56.522023+0000","last_became_active":"2024-09-21T11:02:39.502095+0000","last_became_peered":"2024-09-21T11:02:39.502095+0000","last_unstale":"2024-09-21T11:05:56.522023+0000","last_undegraded":"2024-09-21T11:05:56.522023+0000","last_fullsized":"2024-09-21T11:05:56.522023+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:50:46.078094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1665,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.529433+0000","last_change":"2024-09-21T11:05:48.528936+0000","last_active":"2024-09-21T11:05:57.529433+0000","last_peered":"2024-09-21T11:05:57.529433+0000","last_clean":"2024-09-21T11:05:57.529433+0000","last_became_active":"2024-09-21T11:05:48.528588+0000","last_became_peered":"2024-09-21T11:05:48.528588+0000","last_unstale":"2024-09-21T11:05:57.529433+0000","last_undegraded":"2024-09-21T11:05:57.529433+0000","last_fullsized":"2024-09-21T11:05:57.529433+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:20:53.087780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":142,"reported_epoch":1645,"state":"active+clean","last_fresh":"2024-09-21T11:05:56.520611+0000","last_change":"2024-09-21T11:02:39.502786+0000","last_active":"2024-09-21T11:05:56.520611+0000","last_peered":"2024-09-21T11:05:56.520611+0000","last_clean":"2024-09-21T11:05:56.520611+0000","last_became_active":"2024-09-21T11:02:39.498657+0000","last_became_peered":"2024-09-21T11:02:39.498657+0000","last_unstale":"2024-09-21T11:05:56.520611+0000","last_undegraded":"2024-09-21T11:05:56.520611+0000","last_fullsized":"2024-09-21T11:05:56.520611+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:30:40.747863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":868,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.529217+0000","last_change":"2024-09-21T11:05:48.527087+0000","last_active":"2024-09-21T11:05:57.529217+0000","last_peered":"2024-09-21T11:05:57.529217+0000","last_clean":"2024-09-21T11:05:57.529217+0000","last_became_active":"2024-09-21T11:05:48.526633+0000","last_became_peered":"2024-09-21T11:05:48.526633+0000","last_unstale":"2024-09-21T11:05:57.529217+0000","last_undegraded":"2024-09-21T11:05:57.529217+0000","last_fullsized":"2024-09-21T11:05:57.529217+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:06:08.774518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":30,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.527327+0000","last_change":"2024-09-21T11:05:48.622455+0000","last_active":"2024-09-21T11:05:57.527327+0000","last_peered":"2024-09-21T11:05:57.527327+0000","last_clean":"2024-09-21T11:05:57.527327+0000","last_became_active":"2024-09-21T11:05:48.559270+0000","last_became_peered":"2024-09-21T11:05:48.559270+0000","last_unstale":"2024-09-21T11:05:57.527327+0000","last_undegraded":"2024-09-21T11:05:57.527327+0000","last_fullsized":"2024-09-21T11:05:57.527327+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:37:01.281922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":79,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.530096+0000","last_change":"2024-09-21T11:05:48.527392+0000","last_active":"2024-09-21T11:05:57.530096+0000","last_peered":"2024-09-21T11:05:57.530096+0000","last_clean":"2024-09-21T11:05:57.530096+0000","last_became_active":"2024-09-21T11:05:48.527078+0000","last_became_peered":"2024-09-21T11:05:48.527078+0000","last_unstale":"2024-09-21T11:05:57.530096+0000","last_undegraded":"2024-09-21T11:05:57.530096+0000","last_fullsized":"2024-09-21T11:05:57.530096+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:58:33.681474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":149,"reported_epoch":1645,"state":"active+clean","last_fresh":"2024-09-21T11:05:56.520186+0000","last_change":"2024-09-21T11:05:48.614724+0000","last_active":"2024-09-21T11:05:56.520186+0000","last_peered":"2024-09-21T11:05:56.520186+0000","last_clean":"2024-09-21T11:05:56.520186+0000","last_became_active":"2024-09-21T11:05:48.614414+0000","last_became_peered":"2024-09-21T11:05:48.614414+0000","last_unstale":"2024-09-21T11:05:56.520186+0000","last_undegraded":"2024-09-21T11:05:56.520186+0000","last_fullsized":"2024-09-21T11:05:56.520186+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:14:09.665271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":82,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.530488+0000","last_change":"2024-09-21T11:04:47.619509+0000","last_active":"2024-09-21T11:05:57.530488+0000","last_peered":"2024-09-21T11:05:57.530488+0000","last_clean":"2024-09-21T11:05:57.530488+0000","last_became_active":"2024-09-21T11:04:47.618898+0000","last_became_peered":"2024-09-21T11:04:47.618898+0000","last_unstale":"2024-09-21T11:05:57.530488+0000","last_undegraded":"2024-09-21T11:05:57.530488+0000","last_fullsized":"2024-09-21T11:05:57.530488+0000","mapping_epoch":1610,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1611,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:28:19.899682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2608,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.531598+0000","last_change":"2024-09-21T11:04:34.535674+0000","last_active":"2024-09-21T11:05:57.531598+0000","last_peered":"2024-09-21T11:05:57.531598+0000","last_clean":"2024-09-21T11:05:57.531598+0000","last_became_active":"2024-09-21T11:04:34.535301+0000","last_became_peered":"2024-09-21T11:04:34.535301+0000","last_unstale":"2024-09-21T11:05:57.531598+0000","last_undegraded":"2024-09-21T11:05:57.531598+0000","last_fullsized":"2024-09-21T11:05:57.531598+0000","mapping_epoch":1604,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:47.024005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":30,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.527041+0000","last_change":"2024-09-21T11:05:48.620814+0000","last_active":"2024-09-21T11:05:57.527041+0000","last_peered":"2024-09-21T11:05:57.527041+0000","last_clean":"2024-09-21T11:05:57.527041+0000","last_became_active":"2024-09-21T11:05:48.564386+0000","last_became_peered":"2024-09-21T11:05:48.564386+0000","last_unstale":"2024-09-21T11:05:57.527041+0000","last_undegraded":"2024-09-21T11:05:57.527041+0000","last_fullsized":"2024-09-21T11:05:57.527041+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:49:31.385344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":89,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.528849+0000","last_change":"2024-09-21T11:05:48.548641+0000","last_active":"2024-09-21T11:05:57.528849+0000","last_peered":"2024-09-21T11:05:57.528849+0000","last_clean":"2024-09-21T11:05:57.528849+0000","last_became_active":"2024-09-21T11:05:48.548100+0000","last_became_peered":"2024-09-21T11:05:48.548100+0000","last_unstale":"2024-09-21T11:05:57.528849+0000","last_undegraded":"2024-09-21T11:05:57.528849+0000","last_fullsized":"2024-09-21T11:05:57.528849+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:21:43.172860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1610,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.531112+0000","last_change":"2024-09-21T11:05:48.530856+0000","last_active":"2024-09-21T11:05:57.531112+0000","last_peered":"2024-09-21T11:05:57.531112+0000","last_clean":"2024-09-21T11:05:57.531112+0000","last_became_active":"2024-09-21T11:05:48.530496+0000","last_became_peered":"2024-09-21T11:05:48.530496+0000","last_unstale":"2024-09-21T11:05:57.531112+0000","last_undegraded":"2024-09-21T11:05:57.531112+0000","last_fullsized":"2024-09-21T11:05:57.531112+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:15.620293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":94,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.529006+0000","last_change":"2024-09-21T11:04:34.523861+0000","last_active":"2024-09-21T11:05:57.529006+0000","last_peered":"2024-09-21T11:05:57.529006+0000","last_clean":"2024-09-21T11:05:57.529006+0000","last_became_active":"2024-09-21T11:04:34.523542+0000","last_became_peered":"2024-09-21T11:04:34.523542+0000","last_unstale":"2024-09-21T11:05:57.529006+0000","last_undegraded":"2024-09-21T11:05:57.529006+0000","last_fullsized":"2024-09-21T11:05:57.529006+0000","mapping_epoch":1604,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:35.990638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":30,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.527693+0000","last_change":"2024-09-21T11:05:48.528637+0000","last_active":"2024-09-21T11:05:57.527693+0000","last_peered":"2024-09-21T11:05:57.527693+0000","last_clean":"2024-09-21T11:05:57.527693+0000","last_became_active":"2024-09-21T11:05:48.528215+0000","last_became_peered":"2024-09-21T11:05:48.528215+0000","last_unstale":"2024-09-21T11:05:57.527693+0000","last_undegraded":"2024-09-21T11:05:57.527693+0000","last_fullsized":"2024-09-21T11:05:57.527693+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:43:21.451145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"346'149","reported_seq":872,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.529682+0000","last_change":"2024-09-21T11:04:57.686888+0000","last_active":"2024-09-21T11:05:57.529682+0000","last_peered":"2024-09-21T11:05:57.529682+0000","last_clean":"2024-09-21T11:05:57.529682+0000","last_became_active":"2024-09-21T11:04:57.685991+0000","last_became_peered":"2024-09-21T11:04:57.685991+0000","last_unstale":"2024-09-21T11:05:57.529682+0000","last_undegraded":"2024-09-21T11:05:57.529682+0000","last_fullsized":"2024-09-21T11:05:57.529682+0000","mapping_epoch":1618,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1619,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:53.724544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:53.724544+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:25:59.029231+0000","scrub_duration":0.0017779029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2633,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.529526+0000","last_change":"2024-09-21T11:05:48.529027+0000","last_active":"2024-09-21T11:05:57.529526+0000","last_peered":"2024-09-21T11:05:57.529526+0000","last_clean":"2024-09-21T11:05:57.529526+0000","last_became_active":"2024-09-21T11:05:48.528250+0000","last_became_peered":"2024-09-21T11:05:48.528250+0000","last_unstale":"2024-09-21T11:05:57.529526+0000","last_undegraded":"2024-09-21T11:05:57.529526+0000","last_fullsized":"2024-09-21T11:05:57.529526+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:17:41.340887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":28,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.526932+0000","last_change":"2024-09-21T11:05:49.532647+0000","last_active":"2024-09-21T11:05:57.526932+0000","last_peered":"2024-09-21T11:05:57.526932+0000","last_clean":"2024-09-21T11:05:57.526932+0000","last_became_active":"2024-09-21T11:05:49.532232+0000","last_became_peered":"2024-09-21T11:05:49.532232+0000","last_unstale":"2024-09-21T11:05:57.526932+0000","last_undegraded":"2024-09-21T11:05:57.526932+0000","last_fullsized":"2024-09-21T11:05:57.526932+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:30.002838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":359,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.530680+0000","last_change":"2024-09-21T11:05:48.645446+0000","last_active":"2024-09-21T11:05:57.530680+0000","last_peered":"2024-09-21T11:05:57.530680+0000","last_clean":"2024-09-21T11:05:57.530680+0000","last_became_active":"2024-09-21T11:05:48.538329+0000","last_became_peered":"2024-09-21T11:05:48.538329+0000","last_unstale":"2024-09-21T11:05:57.530680+0000","last_undegraded":"2024-09-21T11:05:57.530680+0000","last_fullsized":"2024-09-21T11:05:57.530680+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:16:14.942520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2594,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.530391+0000","last_change":"2024-09-21T11:05:49.544761+0000","last_active":"2024-09-21T11:05:57.530391+0000","last_peered":"2024-09-21T11:05:57.530391+0000","last_clean":"2024-09-21T11:05:57.530391+0000","last_became_active":"2024-09-21T11:05:49.544461+0000","last_became_peered":"2024-09-21T11:05:49.544461+0000","last_unstale":"2024-09-21T11:05:57.530391+0000","last_undegraded":"2024-09-21T11:05:57.530391+0000","last_fullsized":"2024-09-21T11:05:57.530391+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:34:15.842665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":148,"reported_epoch":1645,"state":"active+clean","last_fresh":"2024-09-21T11:05:56.520528+0000","last_change":"2024-09-21T11:02:39.505094+0000","last_active":"2024-09-21T11:05:56.520528+0000","last_peered":"2024-09-21T11:05:56.520528+0000","last_clean":"2024-09-21T11:05:56.520528+0000","last_became_active":"2024-09-21T11:02:39.503527+0000","last_became_peered":"2024-09-21T11:02:39.503527+0000","last_unstale":"2024-09-21T11:05:56.520528+0000","last_undegraded":"2024-09-21T11:05:56.520528+0000","last_fullsized":"2024-09-21T11:05:56.520528+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:12:34.916393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14268,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":717,"reported_epoch":1645,"state":"active+clean","last_fresh":"2024-09-21T11:05:56.520219+0000","last_change":"2024-09-21T11:02:39.492530+0000","last_active":"2024-09-21T11:05:56.520219+0000","last_peered":"2024-09-21T11:05:56.520219+0000","last_clean":"2024-09-21T11:05:56.520219+0000","last_became_active":"2024-09-21T11:02:39.491835+0000","last_became_peered":"2024-09-21T11:02:39.491835+0000","last_unstale":"2024-09-21T11:05:56.520219+0000","last_undegraded":"2024-09-21T11:05:56.520219+0000","last_fullsized":"2024-09-21T11:05:56.520219+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:37.157843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"346'149","reported_seq":872,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.529779+0000","last_change":"2024-09-21T11:04:57.686320+0000","last_active":"2024-09-21T11:05:57.529779+0000","last_peered":"2024-09-21T11:05:57.529779+0000","last_clean":"2024-09-21T11:05:57.529779+0000","last_became_active":"2024-09-21T11:04:57.684869+0000","last_became_peered":"2024-09-21T11:04:57.684869+0000","last_unstale":"2024-09-21T11:05:57.529779+0000","last_undegraded":"2024-09-21T11:05:57.529779+0000","last_fullsized":"2024-09-21T11:05:57.529779+0000","mapping_epoch":1618,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1619,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:59.781270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:59.781270+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:14:02.855091+0000","scrub_duration":0.0031203189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1583,"reported_epoch":1645,"state":"active+clean","last_fresh":"2024-09-21T11:05:56.520270+0000","last_change":"2024-09-21T11:05:48.525548+0000","last_active":"2024-09-21T11:05:56.520270+0000","last_peered":"2024-09-21T11:05:56.520270+0000","last_clean":"2024-09-21T11:05:56.520270+0000","last_became_active":"2024-09-21T11:05:48.525247+0000","last_became_peered":"2024-09-21T11:05:48.525247+0000","last_unstale":"2024-09-21T11:05:56.520270+0000","last_undegraded":"2024-09-21T11:05:56.520270+0000","last_fullsized":"2024-09-21T11:05:56.520270+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:51:18.193438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1640,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.531162+0000","last_change":"2024-09-21T11:04:47.606958+0000","last_active":"2024-09-21T11:05:57.531162+0000","last_peered":"2024-09-21T11:05:57.531162+0000","last_clean":"2024-09-21T11:05:57.531162+0000","last_became_active":"2024-09-21T11:04:47.606471+0000","last_became_peered":"2024-09-21T11:04:47.606471+0000","last_unstale":"2024-09-21T11:05:57.531162+0000","last_undegraded":"2024-09-21T11:05:57.531162+0000","last_fullsized":"2024-09-21T11:05:57.531162+0000","mapping_epoch":1610,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1611,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:37:09.730613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":34,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.527219+0000","last_change":"2024-09-21T11:05:48.624344+0000","last_active":"2024-09-21T11:05:57.527219+0000","last_peered":"2024-09-21T11:05:57.527219+0000","last_clean":"2024-09-21T11:05:57.527219+0000","last_became_active":"2024-09-21T11:05:48.564758+0000","last_became_peered":"2024-09-21T11:05:48.564758+0000","last_unstale":"2024-09-21T11:05:57.527219+0000","last_undegraded":"2024-09-21T11:05:57.527219+0000","last_fullsized":"2024-09-21T11:05:57.527219+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:32.746530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"346'149","reported_seq":831,"reported_epoch":1645,"state":"active+clean","last_fresh":"2024-09-21T11:05:56.522090+0000","last_change":"2024-09-21T11:05:51.520639+0000","last_active":"2024-09-21T11:05:56.522090+0000","last_peered":"2024-09-21T11:05:56.522090+0000","last_clean":"2024-09-21T11:05:56.522090+0000","last_became_active":"2024-09-21T11:05:51.519581+0000","last_became_peered":"2024-09-21T11:05:51.519581+0000","last_unstale":"2024-09-21T11:05:56.522090+0000","last_undegraded":"2024-09-21T11:05:56.522090+0000","last_fullsized":"2024-09-21T11:05:56.522090+0000","mapping_epoch":1639,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1640,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:54.879276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:54.879276+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:53.844022+0000","scrub_duration":0.001843014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":30,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.526769+0000","last_change":"2024-09-21T11:05:48.530892+0000","last_active":"2024-09-21T11:05:57.526769+0000","last_peered":"2024-09-21T11:05:57.526769+0000","last_clean":"2024-09-21T11:05:57.526769+0000","last_became_active":"2024-09-21T11:05:48.530518+0000","last_became_peered":"2024-09-21T11:05:48.530518+0000","last_unstale":"2024-09-21T11:05:57.526769+0000","last_undegraded":"2024-09-21T11:05:57.526769+0000","last_fullsized":"2024-09-21T11:05:57.526769+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:02:23.025808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2539,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.531085+0000","last_change":"2024-09-21T11:04:47.682972+0000","last_active":"2024-09-21T11:05:57.531085+0000","last_peered":"2024-09-21T11:05:57.531085+0000","last_clean":"2024-09-21T11:05:57.531085+0000","last_became_active":"2024-09-21T11:04:47.681183+0000","last_became_peered":"2024-09-21T11:04:47.681183+0000","last_unstale":"2024-09-21T11:05:57.531085+0000","last_undegraded":"2024-09-21T11:05:57.531085+0000","last_fullsized":"2024-09-21T11:05:57.531085+0000","mapping_epoch":1610,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1611,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:37:12.641763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":2692,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.531075+0000","last_change":"2024-09-21T11:05:48.647083+0000","last_active":"2024-09-21T11:05:57.531075+0000","last_peered":"2024-09-21T11:05:57.531075+0000","last_clean":"2024-09-21T11:05:57.531075+0000","last_became_active":"2024-09-21T11:05:48.543037+0000","last_became_peered":"2024-09-21T11:05:48.543037+0000","last_unstale":"2024-09-21T11:05:57.531075+0000","last_undegraded":"2024-09-21T11:05:57.531075+0000","last_fullsized":"2024-09-21T11:05:57.531075+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:00:57.802995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"346'149","reported_seq":940,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:06:00.912194+0000","last_change":"2024-09-21T11:06:00.912194+0000","last_active":"2024-09-21T11:06:00.912194+0000","last_peered":"2024-09-21T11:06:00.912194+0000","last_clean":"2024-09-21T11:06:00.912194+0000","last_became_active":"2024-09-21T11:05:51.518873+0000","last_became_peered":"2024-09-21T11:05:51.518873+0000","last_unstale":"2024-09-21T11:06:00.912194+0000","last_undegraded":"2024-09-21T11:06:00.912194+0000","last_fullsized":"2024-09-21T11:06:00.912194+0000","mapping_epoch":1639,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1640,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:06:00.911909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:06:00.911909+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:16:28.318208+0000","scrub_duration":0.002144838,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":30,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.527150+0000","last_change":"2024-09-21T11:05:48.537936+0000","last_active":"2024-09-21T11:05:57.527150+0000","last_peered":"2024-09-21T11:05:57.527150+0000","last_clean":"2024-09-21T11:05:57.527150+0000","last_became_active":"2024-09-21T11:05:48.537615+0000","last_became_peered":"2024-09-21T11:05:48.537615+0000","last_unstale":"2024-09-21T11:05:57.527150+0000","last_undegraded":"2024-09-21T11:05:57.527150+0000","last_fullsized":"2024-09-21T11:05:57.527150+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:03:45.353800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":244,"reported_epoch":1645,"state":"active+clean","last_fresh":"2024-09-21T11:05:56.525305+0000","last_change":"2024-09-21T11:05:48.528383+0000","last_active":"2024-09-21T11:05:56.525305+0000","last_peered":"2024-09-21T11:05:56.525305+0000","last_clean":"2024-09-21T11:05:56.525305+0000","last_became_active":"2024-09-21T11:05:48.527903+0000","last_became_peered":"2024-09-21T11:05:48.527903+0000","last_unstale":"2024-09-21T11:05:56.525305+0000","last_undegraded":"2024-09-21T11:05:56.525305+0000","last_fullsized":"2024-09-21T11:05:56.525305+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:59:25.065009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":30,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.526641+0000","last_change":"2024-09-21T11:05:48.531015+0000","last_active":"2024-09-21T11:05:57.526641+0000","last_peered":"2024-09-21T11:05:57.526641+0000","last_clean":"2024-09-21T11:05:57.526641+0000","last_became_active":"2024-09-21T11:05:48.530729+0000","last_became_peered":"2024-09-21T11:05:48.530729+0000","last_unstale":"2024-09-21T11:05:57.526641+0000","last_undegraded":"2024-09-21T11:05:57.526641+0000","last_fullsized":"2024-09-21T11:05:57.526641+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:26.948930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"346'149","reported_seq":944,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.531215+0000","last_change":"2024-09-21T11:05:57.003066+0000","last_active":"2024-09-21T11:05:57.531215+0000","last_peered":"2024-09-21T11:05:57.531215+0000","last_clean":"2024-09-21T11:05:57.531215+0000","last_became_active":"2024-09-21T11:05:51.519155+0000","last_became_peered":"2024-09-21T11:05:51.519155+0000","last_unstale":"2024-09-21T11:05:57.531215+0000","last_undegraded":"2024-09-21T11:05:57.531215+0000","last_fullsized":"2024-09-21T11:05:57.531215+0000","mapping_epoch":1639,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1640,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:05:57.002981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:05:57.002981+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:01:09.602087+0000","scrub_duration":0.0017979249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":30,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.527741+0000","last_change":"2024-09-21T11:05:48.524466+0000","last_active":"2024-09-21T11:05:57.527741+0000","last_peered":"2024-09-21T11:05:57.527741+0000","last_clean":"2024-09-21T11:05:57.527741+0000","last_became_active":"2024-09-21T11:05:48.524124+0000","last_became_peered":"2024-09-21T11:05:48.524124+0000","last_unstale":"2024-09-21T11:05:57.527741+0000","last_undegraded":"2024-09-21T11:05:57.527741+0000","last_fullsized":"2024-09-21T11:05:57.527741+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:20:46.313419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.526845+0000","last_change":"2024-09-21T11:05:48.531055+0000","last_active":"2024-09-21T11:05:57.526845+0000","last_peered":"2024-09-21T11:05:57.526845+0000","last_clean":"2024-09-21T11:05:57.526845+0000","last_became_active":"2024-09-21T11:05:48.530743+0000","last_became_peered":"2024-09-21T11:05:48.530743+0000","last_unstale":"2024-09-21T11:05:57.526845+0000","last_undegraded":"2024-09-21T11:05:57.526845+0000","last_fullsized":"2024-09-21T11:05:57.526845+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:09:04.625920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":77,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.529014+0000","last_change":"2024-09-21T11:05:48.528180+0000","last_active":"2024-09-21T11:05:57.529014+0000","last_peered":"2024-09-21T11:05:57.529014+0000","last_clean":"2024-09-21T11:05:57.529014+0000","last_became_active":"2024-09-21T11:05:48.527801+0000","last_became_peered":"2024-09-21T11:05:48.527801+0000","last_unstale":"2024-09-21T11:05:57.529014+0000","last_undegraded":"2024-09-21T11:05:57.529014+0000","last_fullsized":"2024-09-21T11:05:57.529014+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:36:53.170186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"346'149","reported_seq":966,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.527047+0000","last_change":"2024-09-21T11:05:51.518615+0000","last_active":"2024-09-21T11:05:57.527047+0000","last_peered":"2024-09-21T11:05:57.527047+0000","last_clean":"2024-09-21T11:05:57.527047+0000","last_became_active":"2024-09-21T11:05:51.518248+0000","last_became_peered":"2024-09-21T11:05:51.518248+0000","last_unstale":"2024-09-21T11:05:57.527047+0000","last_undegraded":"2024-09-21T11:05:57.527047+0000","last_fullsized":"2024-09-21T11:05:57.527047+0000","mapping_epoch":1639,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1640,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:48.918874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.918874+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:00.316316+0000","scrub_duration":0.001629134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":76,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.529883+0000","last_change":"2024-09-21T11:05:49.521052+0000","last_active":"2024-09-21T11:05:57.529883+0000","last_peered":"2024-09-21T11:05:57.529883+0000","last_clean":"2024-09-21T11:05:57.529883+0000","last_became_active":"2024-09-21T11:05:49.520760+0000","last_became_peered":"2024-09-21T11:05:49.520760+0000","last_unstale":"2024-09-21T11:05:57.529883+0000","last_undegraded":"2024-09-21T11:05:57.529883+0000","last_fullsized":"2024-09-21T11:05:57.529883+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:49:48.255066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1554,"reported_epoch":1645,"state":"active+clean","last_fresh":"2024-09-21T11:05:56.520368+0000","last_change":"2024-09-21T11:05:48.549704+0000","last_active":"2024-09-21T11:05:56.520368+0000","last_peered":"2024-09-21T11:05:56.520368+0000","last_clean":"2024-09-21T11:05:56.520368+0000","last_became_active":"2024-09-21T11:05:48.549374+0000","last_became_peered":"2024-09-21T11:05:48.549374+0000","last_unstale":"2024-09-21T11:05:56.520368+0000","last_undegraded":"2024-09-21T11:05:56.520368+0000","last_fullsized":"2024-09-21T11:05:56.520368+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:57.002097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":582,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.530016+0000","last_change":"2024-09-21T11:05:49.692048+0000","last_active":"2024-09-21T11:05:57.530016+0000","last_peered":"2024-09-21T11:05:57.530016+0000","last_clean":"2024-09-21T11:05:57.530016+0000","last_became_active":"2024-09-21T11:05:49.548699+0000","last_became_peered":"2024-09-21T11:05:49.548699+0000","last_unstale":"2024-09-21T11:05:57.530016+0000","last_undegraded":"2024-09-21T11:05:57.530016+0000","last_fullsized":"2024-09-21T11:05:57.530016+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:30.701647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":95,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"346'149","reported_seq":848,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.527502+0000","last_change":"2024-09-21T11:05:51.521261+0000","last_active":"2024-09-21T11:05:57.527502+0000","last_peered":"2024-09-21T11:05:57.527502+0000","last_clean":"2024-09-21T11:05:57.527502+0000","last_became_active":"2024-09-21T11:05:51.520925+0000","last_became_peered":"2024-09-21T11:05:51.520925+0000","last_unstale":"2024-09-21T11:05:57.527502+0000","last_undegraded":"2024-09-21T11:05:57.527502+0000","last_fullsized":"2024-09-21T11:05:57.527502+0000","mapping_epoch":1639,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1640,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:49.769829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:49.769829+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":37,"num_write":10,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":134174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":28,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.527654+0000","last_change":"2024-09-21T11:05:49.531290+0000","last_active":"2024-09-21T11:05:57.527654+0000","last_peered":"2024-09-21T11:05:57.527654+0000","last_clean":"2024-09-21T11:05:57.527654+0000","last_became_active":"2024-09-21T11:05:49.531016+0000","last_became_peered":"2024-09-21T11:05:49.531016+0000","last_unstale":"2024-09-21T11:05:57.527654+0000","last_undegraded":"2024-09-21T11:05:57.527654+0000","last_fullsized":"2024-09-21T11:05:57.527654+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:01:39.614195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2574,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.530005+0000","last_change":"2024-09-21T11:05:49.538825+0000","last_active":"2024-09-21T11:05:57.530005+0000","last_peered":"2024-09-21T11:05:57.530005+0000","last_clean":"2024-09-21T11:05:57.530005+0000","last_became_active":"2024-09-21T11:05:49.538488+0000","last_became_peered":"2024-09-21T11:05:49.538488+0000","last_unstale":"2024-09-21T11:05:57.530005+0000","last_undegraded":"2024-09-21T11:05:57.530005+0000","last_fullsized":"2024-09-21T11:05:57.530005+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:14:02.932113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":79,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.529365+0000","last_change":"2024-09-21T11:05:48.533847+0000","last_active":"2024-09-21T11:05:57.529365+0000","last_peered":"2024-09-21T11:05:57.529365+0000","last_clean":"2024-09-21T11:05:57.529365+0000","last_became_active":"2024-09-21T11:05:48.533544+0000","last_became_peered":"2024-09-21T11:05:48.533544+0000","last_unstale":"2024-09-21T11:05:57.529365+0000","last_undegraded":"2024-09-21T11:05:57.529365+0000","last_fullsized":"2024-09-21T11:05:57.529365+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:06:32.575001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"346'149","reported_seq":1282,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.530875+0000","last_change":"2024-09-21T11:04:37.545585+0000","last_active":"2024-09-21T11:05:57.530875+0000","last_peered":"2024-09-21T11:05:57.530875+0000","last_clean":"2024-09-21T11:05:57.530875+0000","last_became_active":"2024-09-21T11:04:37.545040+0000","last_became_peered":"2024-09-21T11:04:37.545040+0000","last_unstale":"2024-09-21T11:05:57.530875+0000","last_undegraded":"2024-09-21T11:05:57.530875+0000","last_fullsized":"2024-09-21T11:05:57.530875+0000","mapping_epoch":1607,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1608,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:50.671365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:50.671365+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:46.202953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":30,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.527367+0000","last_change":"2024-09-21T11:05:48.531154+0000","last_active":"2024-09-21T11:05:57.527367+0000","last_peered":"2024-09-21T11:05:57.527367+0000","last_clean":"2024-09-21T11:05:57.527367+0000","last_became_active":"2024-09-21T11:05:48.530845+0000","last_became_peered":"2024-09-21T11:05:48.530845+0000","last_unstale":"2024-09-21T11:05:57.527367+0000","last_undegraded":"2024-09-21T11:05:57.527367+0000","last_fullsized":"2024-09-21T11:05:57.527367+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:45.623351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":30,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.526779+0000","last_change":"2024-09-21T11:05:48.605628+0000","last_active":"2024-09-21T11:05:57.526779+0000","last_peered":"2024-09-21T11:05:57.526779+0000","last_clean":"2024-09-21T11:05:57.526779+0000","last_became_active":"2024-09-21T11:05:48.539248+0000","last_became_peered":"2024-09-21T11:05:48.539248+0000","last_unstale":"2024-09-21T11:05:57.526779+0000","last_undegraded":"2024-09-21T11:05:57.526779+0000","last_fullsized":"2024-09-21T11:05:57.526779+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:52:57.906544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":324,"reported_epoch":1645,"state":"active+clean","last_fresh":"2024-09-21T11:05:56.524993+0000","last_change":"2024-09-21T11:05:48.522804+0000","last_active":"2024-09-21T11:05:56.524993+0000","last_peered":"2024-09-21T11:05:56.524993+0000","last_clean":"2024-09-21T11:05:56.524993+0000","last_became_active":"2024-09-21T11:05:48.522467+0000","last_became_peered":"2024-09-21T11:05:48.522467+0000","last_unstale":"2024-09-21T11:05:56.524993+0000","last_undegraded":"2024-09-21T11:05:56.524993+0000","last_fullsized":"2024-09-21T11:05:56.524993+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:59:21.055560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":176,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.530613+0000","last_change":"2024-09-21T11:02:39.502922+0000","last_active":"2024-09-21T11:05:57.530613+0000","last_peered":"2024-09-21T11:05:57.530613+0000","last_clean":"2024-09-21T11:05:57.530613+0000","last_became_active":"2024-09-21T11:02:39.502633+0000","last_became_peered":"2024-09-21T11:02:39.502633+0000","last_unstale":"2024-09-21T11:05:57.530613+0000","last_undegraded":"2024-09-21T11:05:57.530613+0000","last_fullsized":"2024-09-21T11:05:57.530613+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:29:20.971522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":72,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.528958+0000","last_change":"2024-09-21T11:04:54.651475+0000","last_active":"2024-09-21T11:05:57.528958+0000","last_peered":"2024-09-21T11:05:57.528958+0000","last_clean":"2024-09-21T11:05:57.528958+0000","last_became_active":"2024-09-21T11:04:54.651179+0000","last_became_peered":"2024-09-21T11:04:54.651179+0000","last_unstale":"2024-09-21T11:05:57.528958+0000","last_undegraded":"2024-09-21T11:05:57.528958+0000","last_fullsized":"2024-09-21T11:05:57.528958+0000","mapping_epoch":1615,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:49.630408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":1482,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.529115+0000","last_change":"2024-09-21T11:05:49.568420+0000","last_active":"2024-09-21T11:05:57.529115+0000","last_peered":"2024-09-21T11:05:57.529115+0000","last_clean":"2024-09-21T11:05:57.529115+0000","last_became_active":"2024-09-21T11:05:49.523386+0000","last_became_peered":"2024-09-21T11:05:49.523386+0000","last_unstale":"2024-09-21T11:05:57.529115+0000","last_undegraded":"2024-09-21T11:05:57.529115+0000","last_fullsized":"2024-09-21T11:05:57.529115+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:25.230371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":811,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:58.110061+0000","last_change":"2024-09-21T11:05:58.110061+0000","last_active":"2024-09-21T11:05:58.110061+0000","last_peered":"2024-09-21T11:05:58.110061+0000","last_clean":"2024-09-21T11:05:58.110061+0000","last_became_active":"2024-09-21T11:05:51.518948+0000","last_became_peered":"2024-09-21T11:05:51.518948+0000","last_unstale":"2024-09-21T11:05:58.110061+0000","last_undegraded":"2024-09-21T11:05:58.110061+0000","last_fullsized":"2024-09-21T11:05:58.110061+0000","mapping_epoch":1639,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1640,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:05:58.110003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:05:58.110003+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:59.832936+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"346'149","reported_seq":951,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.530397+0000","last_change":"2024-09-21T11:05:51.519229+0000","last_active":"2024-09-21T11:05:57.530397+0000","last_peered":"2024-09-21T11:05:57.530397+0000","last_clean":"2024-09-21T11:05:57.530397+0000","last_became_active":"2024-09-21T11:05:51.518878+0000","last_became_peered":"2024-09-21T11:05:51.518878+0000","last_unstale":"2024-09-21T11:05:57.530397+0000","last_undegraded":"2024-09-21T11:05:57.530397+0000","last_fullsized":"2024-09-21T11:05:57.530397+0000","mapping_epoch":1639,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1640,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:42.008235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:42.008235+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:25:31.810536+0000","scrub_duration":0.011129834999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2604,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.531228+0000","last_change":"2024-09-21T11:01:46.484382+0000","last_active":"2024-09-21T11:05:57.531228+0000","last_peered":"2024-09-21T11:05:57.531228+0000","last_clean":"2024-09-21T11:05:57.531228+0000","last_became_active":"2024-09-21T11:01:46.484116+0000","last_became_peered":"2024-09-21T11:01:46.484116+0000","last_unstale":"2024-09-21T11:05:57.531228+0000","last_undegraded":"2024-09-21T11:05:57.531228+0000","last_fullsized":"2024-09-21T11:05:57.531228+0000","mapping_epoch":1578,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:48.924838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2519,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.530792+0000","last_change":"2024-09-21T11:04:47.603875+0000","last_active":"2024-09-21T11:05:57.530792+0000","last_peered":"2024-09-21T11:05:57.530792+0000","last_clean":"2024-09-21T11:05:57.530792+0000","last_became_active":"2024-09-21T11:04:47.603387+0000","last_became_peered":"2024-09-21T11:04:47.603387+0000","last_unstale":"2024-09-21T11:05:57.530792+0000","last_undegraded":"2024-09-21T11:05:57.530792+0000","last_fullsized":"2024-09-21T11:05:57.530792+0000","mapping_epoch":1610,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1611,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:50:20.313909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"346'149","reported_seq":1279,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.530475+0000","last_change":"2024-09-21T11:04:37.545030+0000","last_active":"2024-09-21T11:05:57.530475+0000","last_peered":"2024-09-21T11:05:57.530475+0000","last_clean":"2024-09-21T11:05:57.530475+0000","last_became_active":"2024-09-21T11:04:37.544275+0000","last_became_peered":"2024-09-21T11:04:37.544275+0000","last_unstale":"2024-09-21T11:05:57.530475+0000","last_undegraded":"2024-09-21T11:05:57.530475+0000","last_fullsized":"2024-09-21T11:05:57.530475+0000","mapping_epoch":1607,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1608,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:48.663349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.663349+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:05:32.798146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1575,"reported_epoch":1645,"state":"active+clean","last_fresh":"2024-09-21T11:05:56.521667+0000","last_change":"2024-09-21T11:04:47.634670+0000","last_active":"2024-09-21T11:05:56.521667+0000","last_peered":"2024-09-21T11:05:56.521667+0000","last_clean":"2024-09-21T11:05:56.521667+0000","last_became_active":"2024-09-21T11:04:47.632604+0000","last_became_peered":"2024-09-21T11:04:47.632604+0000","last_unstale":"2024-09-21T11:05:56.521667+0000","last_undegraded":"2024-09-21T11:05:56.521667+0000","last_fullsized":"2024-09-21T11:05:56.521667+0000","mapping_epoch":1610,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1611,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:53:06.603611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1499,"reported_epoch":1645,"state":"active+clean","last_fresh":"2024-09-21T11:05:56.525783+0000","last_change":"2024-09-21T11:04:54.664382+0000","last_active":"2024-09-21T11:05:56.525783+0000","last_peered":"2024-09-21T11:05:56.525783+0000","last_clean":"2024-09-21T11:05:56.525783+0000","last_became_active":"2024-09-21T11:04:54.664028+0000","last_became_peered":"2024-09-21T11:04:54.664028+0000","last_unstale":"2024-09-21T11:05:56.525783+0000","last_undegraded":"2024-09-21T11:05:56.525783+0000","last_fullsized":"2024-09-21T11:05:56.525783+0000","mapping_epoch":1615,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:10:38.686448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"346'149","reported_seq":1059,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.528832+0000","last_change":"2024-09-21T11:04:56.693547+0000","last_active":"2024-09-21T11:05:57.528832+0000","last_peered":"2024-09-21T11:05:57.528832+0000","last_clean":"2024-09-21T11:05:57.528832+0000","last_became_active":"2024-09-21T11:04:56.693184+0000","last_became_peered":"2024-09-21T11:04:56.693184+0000","last_unstale":"2024-09-21T11:05:57.528832+0000","last_undegraded":"2024-09-21T11:05:57.528832+0000","last_fullsized":"2024-09-21T11:05:57.528832+0000","mapping_epoch":1617,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1618,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:49.947173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:49.947173+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1319101,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1514,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.529884+0000","last_change":"2024-09-21T11:05:48.528120+0000","last_active":"2024-09-21T11:05:57.529884+0000","last_peered":"2024-09-21T11:05:57.529884+0000","last_clean":"2024-09-21T11:05:57.529884+0000","last_became_active":"2024-09-21T11:05:48.527862+0000","last_became_peered":"2024-09-21T11:05:48.527862+0000","last_unstale":"2024-09-21T11:05:57.529884+0000","last_undegraded":"2024-09-21T11:05:57.529884+0000","last_fullsized":"2024-09-21T11:05:57.529884+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:37:53.048201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1604,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.530299+0000","last_change":"2024-09-21T11:04:47.621289+0000","last_active":"2024-09-21T11:05:57.530299+0000","last_peered":"2024-09-21T11:05:57.530299+0000","last_clean":"2024-09-21T11:05:57.530299+0000","last_became_active":"2024-09-21T11:04:47.621009+0000","last_became_peered":"2024-09-21T11:04:47.621009+0000","last_unstale":"2024-09-21T11:05:57.530299+0000","last_undegraded":"2024-09-21T11:05:57.530299+0000","last_fullsized":"2024-09-21T11:05:57.530299+0000","mapping_epoch":1610,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1611,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:03:55.536034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"346'149","reported_seq":1117,"reported_epoch":1645,"state":"active+clean","last_fresh":"2024-09-21T11:05:56.525692+0000","last_change":"2024-09-21T11:02:38.474085+0000","last_active":"2024-09-21T11:05:56.525692+0000","last_peered":"2024-09-21T11:05:56.525692+0000","last_clean":"2024-09-21T11:05:56.525692+0000","last_became_active":"2024-09-21T11:02:38.473593+0000","last_became_peered":"2024-09-21T11:02:38.473593+0000","last_unstale":"2024-09-21T11:05:56.525692+0000","last_undegraded":"2024-09-21T11:05:56.525692+0000","last_fullsized":"2024-09-21T11:05:56.525692+0000","mapping_epoch":1589,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1590,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:17.418646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:17.418646+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:59:13.530642+0000","scrub_duration":0.00185648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":819,"reported_epoch":1645,"state":"active+clean","last_fresh":"2024-09-21T11:05:56.520691+0000","last_change":"2024-09-21T11:02:39.501678+0000","last_active":"2024-09-21T11:05:56.520691+0000","last_peered":"2024-09-21T11:05:56.520691+0000","last_clean":"2024-09-21T11:05:56.520691+0000","last_became_active":"2024-09-21T11:02:39.497766+0000","last_became_peered":"2024-09-21T11:02:39.497766+0000","last_unstale":"2024-09-21T11:05:56.520691+0000","last_undegraded":"2024-09-21T11:05:56.520691+0000","last_fullsized":"2024-09-21T11:05:56.520691+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:09:25.807702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":97,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.529102+0000","last_change":"2024-09-21T11:05:48.649119+0000","last_active":"2024-09-21T11:05:57.529102+0000","last_peered":"2024-09-21T11:05:57.529102+0000","last_clean":"2024-09-21T11:05:57.529102+0000","last_became_active":"2024-09-21T11:05:48.550467+0000","last_became_peered":"2024-09-21T11:05:48.550467+0000","last_unstale":"2024-09-21T11:05:57.529102+0000","last_undegraded":"2024-09-21T11:05:57.529102+0000","last_fullsized":"2024-09-21T11:05:57.529102+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:46:06.305352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"346'149","reported_seq":929,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.530920+0000","last_change":"2024-09-21T11:05:53.509398+0000","last_active":"2024-09-21T11:05:57.530920+0000","last_peered":"2024-09-21T11:05:57.530920+0000","last_clean":"2024-09-21T11:05:57.530920+0000","last_became_active":"2024-09-21T11:05:51.519353+0000","last_became_peered":"2024-09-21T11:05:51.519353+0000","last_unstale":"2024-09-21T11:05:57.530920+0000","last_undegraded":"2024-09-21T11:05:57.530920+0000","last_fullsized":"2024-09-21T11:05:57.530920+0000","mapping_epoch":1639,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1640,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:57.851898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:57.851898+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061250310000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":30,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.527232+0000","last_change":"2024-09-21T11:05:48.622817+0000","last_active":"2024-09-21T11:05:57.527232+0000","last_peered":"2024-09-21T11:05:57.527232+0000","last_clean":"2024-09-21T11:05:57.527232+0000","last_became_active":"2024-09-21T11:05:48.559669+0000","last_became_peered":"2024-09-21T11:05:48.559669+0000","last_unstale":"2024-09-21T11:05:57.527232+0000","last_undegraded":"2024-09-21T11:05:57.527232+0000","last_fullsized":"2024-09-21T11:05:57.527232+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:25:03.540564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":82,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.530126+0000","last_change":"2024-09-21T11:04:47.608909+0000","last_active":"2024-09-21T11:05:57.530126+0000","last_peered":"2024-09-21T11:05:57.530126+0000","last_clean":"2024-09-21T11:05:57.530126+0000","last_became_active":"2024-09-21T11:04:47.608245+0000","last_became_peered":"2024-09-21T11:04:47.608245+0000","last_unstale":"2024-09-21T11:05:57.530126+0000","last_undegraded":"2024-09-21T11:05:57.530126+0000","last_fullsized":"2024-09-21T11:05:57.530126+0000","mapping_epoch":1610,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1611,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:37:10.069192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"346'149","reported_seq":945,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:59.940956+0000","last_change":"2024-09-21T11:05:59.940956+0000","last_active":"2024-09-21T11:05:59.940956+0000","last_peered":"2024-09-21T11:05:59.940956+0000","last_clean":"2024-09-21T11:05:59.940956+0000","last_became_active":"2024-09-21T11:05:51.519597+0000","last_became_peered":"2024-09-21T11:05:51.519597+0000","last_unstale":"2024-09-21T11:05:59.940956+0000","last_undegraded":"2024-09-21T11:05:59.940956+0000","last_fullsized":"2024-09-21T11:05:59.940956+0000","mapping_epoch":1639,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1640,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:05:59.940755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:05:59.940755+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:23:05.975909+0000","scrub_duration":0.0034290169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":164,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.530472+0000","last_change":"2024-09-21T11:05:48.531760+0000","last_active":"2024-09-21T11:05:57.530472+0000","last_peered":"2024-09-21T11:05:57.530472+0000","last_clean":"2024-09-21T11:05:57.530472+0000","last_became_active":"2024-09-21T11:05:48.530864+0000","last_became_peered":"2024-09-21T11:05:48.530864+0000","last_unstale":"2024-09-21T11:05:57.530472+0000","last_undegraded":"2024-09-21T11:05:57.530472+0000","last_fullsized":"2024-09-21T11:05:57.530472+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:58:10.489194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":392,"reported_epoch":1645,"state":"active+clean","last_fresh":"2024-09-21T11:05:56.525522+0000","last_change":"2024-09-21T11:04:47.610105+0000","last_active":"2024-09-21T11:05:56.525522+0000","last_peered":"2024-09-21T11:05:56.525522+0000","last_clean":"2024-09-21T11:05:56.525522+0000","last_became_active":"2024-09-21T11:04:47.594894+0000","last_became_peered":"2024-09-21T11:04:47.594894+0000","last_unstale":"2024-09-21T11:05:56.525522+0000","last_undegraded":"2024-09-21T11:05:56.525522+0000","last_fullsized":"2024-09-21T11:05:56.525522+0000","mapping_epoch":1610,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1611,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:26:45.561726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"346'149","reported_seq":870,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.529570+0000","last_change":"2024-09-21T11:04:57.687191+0000","last_active":"2024-09-21T11:05:57.529570+0000","last_peered":"2024-09-21T11:05:57.529570+0000","last_clean":"2024-09-21T11:05:57.529570+0000","last_became_active":"2024-09-21T11:04:57.684412+0000","last_became_peered":"2024-09-21T11:04:57.684412+0000","last_unstale":"2024-09-21T11:05:57.529570+0000","last_undegraded":"2024-09-21T11:05:57.529570+0000","last_fullsized":"2024-09-21T11:05:57.529570+0000","mapping_epoch":1618,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1619,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:54.708658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:54.708658+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:59:04.967843+0000","scrub_duration":0.0021035160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1639,"reported_epoch":1645,"state":"active+clean","last_fresh":"2024-09-21T11:05:56.520539+0000","last_change":"2024-09-21T11:02:39.489781+0000","last_active":"2024-09-21T11:05:56.520539+0000","last_peered":"2024-09-21T11:05:56.520539+0000","last_clean":"2024-09-21T11:05:56.520539+0000","last_became_active":"2024-09-21T11:02:39.489523+0000","last_became_peered":"2024-09-21T11:02:39.489523+0000","last_unstale":"2024-09-21T11:05:56.520539+0000","last_undegraded":"2024-09-21T11:05:56.520539+0000","last_fullsized":"2024-09-21T11:05:56.520539+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:39.794133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":111,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.531378+0000","last_change":"2024-09-21T11:05:48.624546+0000","last_active":"2024-09-21T11:05:57.531378+0000","last_peered":"2024-09-21T11:05:57.531378+0000","last_clean":"2024-09-21T11:05:57.531378+0000","last_became_active":"2024-09-21T11:05:48.526229+0000","last_became_peered":"2024-09-21T11:05:48.526229+0000","last_unstale":"2024-09-21T11:05:57.531378+0000","last_undegraded":"2024-09-21T11:05:57.531378+0000","last_fullsized":"2024-09-21T11:05:57.531378+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:19:14.810382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"346'149","reported_seq":831,"reported_epoch":1645,"state":"active+clean","last_fresh":"2024-09-21T11:05:56.521191+0000","last_change":"2024-09-21T11:05:51.519974+0000","last_active":"2024-09-21T11:05:56.521191+0000","last_peered":"2024-09-21T11:05:56.521191+0000","last_clean":"2024-09-21T11:05:56.521191+0000","last_became_active":"2024-09-21T11:05:51.519253+0000","last_became_peered":"2024-09-21T11:05:51.519253+0000","last_unstale":"2024-09-21T11:05:56.521191+0000","last_undegraded":"2024-09-21T11:05:56.521191+0000","last_fullsized":"2024-09-21T11:05:56.521191+0000","mapping_epoch":1639,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1640,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:53.896066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:53.896066+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:35:33.679485+0000","scrub_duration":0.001699578,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":257,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.530088+0000","last_change":"2024-09-21T11:04:54.652631+0000","last_active":"2024-09-21T11:05:57.530088+0000","last_peered":"2024-09-21T11:05:57.530088+0000","last_clean":"2024-09-21T11:05:57.530088+0000","last_became_active":"2024-09-21T11:04:54.652230+0000","last_became_peered":"2024-09-21T11:04:54.652230+0000","last_unstale":"2024-09-21T11:05:57.530088+0000","last_undegraded":"2024-09-21T11:05:57.530088+0000","last_fullsized":"2024-09-21T11:05:57.530088+0000","mapping_epoch":1615,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:30:42.182255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":188,"reported_epoch":1645,"state":"active+clean","last_fresh":"2024-09-21T11:05:56.521160+0000","last_change":"2024-09-21T11:05:49.664478+0000","last_active":"2024-09-21T11:05:56.521160+0000","last_peered":"2024-09-21T11:05:56.521160+0000","last_clean":"2024-09-21T11:05:56.521160+0000","last_became_active":"2024-09-21T11:05:49.552565+0000","last_became_peered":"2024-09-21T11:05:49.552565+0000","last_unstale":"2024-09-21T11:05:56.521160+0000","last_undegraded":"2024-09-21T11:05:56.521160+0000","last_fullsized":"2024-09-21T11:05:56.521160+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:34.323282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"346'149","reported_seq":848,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.527565+0000","last_change":"2024-09-21T11:05:51.521406+0000","last_active":"2024-09-21T11:05:57.527565+0000","last_peered":"2024-09-21T11:05:57.527565+0000","last_clean":"2024-09-21T11:05:57.527565+0000","last_became_active":"2024-09-21T11:05:51.521078+0000","last_became_peered":"2024-09-21T11:05:51.521078+0000","last_unstale":"2024-09-21T11:05:57.527565+0000","last_undegraded":"2024-09-21T11:05:57.527565+0000","last_fullsized":"2024-09-21T11:05:57.527565+0000","mapping_epoch":1639,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1640,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:48.731903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.731903+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":38,"num_write":10,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":134175,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1688,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.529743+0000","last_change":"2024-09-21T11:05:49.569869+0000","last_active":"2024-09-21T11:05:57.529743+0000","last_peered":"2024-09-21T11:05:57.529743+0000","last_clean":"2024-09-21T11:05:57.529743+0000","last_became_active":"2024-09-21T11:05:49.526490+0000","last_became_peered":"2024-09-21T11:05:49.526490+0000","last_unstale":"2024-09-21T11:05:57.529743+0000","last_undegraded":"2024-09-21T11:05:57.529743+0000","last_fullsized":"2024-09-21T11:05:57.529743+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:34:32.411382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2348,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.531181+0000","last_change":"2024-09-21T11:05:48.530620+0000","last_active":"2024-09-21T11:05:57.531181+0000","last_peered":"2024-09-21T11:05:57.531181+0000","last_clean":"2024-09-21T11:05:57.531181+0000","last_became_active":"2024-09-21T11:05:48.530322+0000","last_became_peered":"2024-09-21T11:05:48.530322+0000","last_unstale":"2024-09-21T11:05:57.531181+0000","last_undegraded":"2024-09-21T11:05:57.531181+0000","last_fullsized":"2024-09-21T11:05:57.531181+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:39:20.664818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"346'149","reported_seq":1279,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.530368+0000","last_change":"2024-09-21T11:04:37.544703+0000","last_active":"2024-09-21T11:05:57.530368+0000","last_peered":"2024-09-21T11:05:57.530368+0000","last_clean":"2024-09-21T11:05:57.530368+0000","last_became_active":"2024-09-21T11:04:37.543859+0000","last_became_peered":"2024-09-21T11:04:37.543859+0000","last_unstale":"2024-09-21T11:05:57.530368+0000","last_undegraded":"2024-09-21T11:05:57.530368+0000","last_fullsized":"2024-09-21T11:05:57.530368+0000","mapping_epoch":1607,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1608,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:54.706428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:54.706428+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:28:17.451615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":359,"reported_epoch":1645,"state":"active+clean","last_fresh":"2024-09-21T11:05:56.525850+0000","last_change":"2024-09-21T11:02:39.471842+0000","last_active":"2024-09-21T11:05:56.525850+0000","last_peered":"2024-09-21T11:05:56.525850+0000","last_clean":"2024-09-21T11:05:56.525850+0000","last_became_active":"2024-09-21T11:02:39.471339+0000","last_became_peered":"2024-09-21T11:02:39.471339+0000","last_unstale":"2024-09-21T11:05:56.525850+0000","last_undegraded":"2024-09-21T11:05:56.525850+0000","last_fullsized":"2024-09-21T11:05:56.525850+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:53:53.996670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":30,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.527129+0000","last_change":"2024-09-21T11:05:48.527219+0000","last_active":"2024-09-21T11:05:57.527129+0000","last_peered":"2024-09-21T11:05:57.527129+0000","last_clean":"2024-09-21T11:05:57.527129+0000","last_became_active":"2024-09-21T11:05:48.526896+0000","last_became_peered":"2024-09-21T11:05:48.526896+0000","last_unstale":"2024-09-21T11:05:57.527129+0000","last_undegraded":"2024-09-21T11:05:57.527129+0000","last_fullsized":"2024-09-21T11:05:57.527129+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:26:48.839604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"346'149","reported_seq":803,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.526964+0000","last_change":"2024-09-21T11:05:51.519440+0000","last_active":"2024-09-21T11:05:57.526964+0000","last_peered":"2024-09-21T11:05:57.526964+0000","last_clean":"2024-09-21T11:05:57.526964+0000","last_became_active":"2024-09-21T11:05:51.519086+0000","last_became_peered":"2024-09-21T11:05:51.519086+0000","last_unstale":"2024-09-21T11:05:57.526964+0000","last_undegraded":"2024-09-21T11:05:57.526964+0000","last_fullsized":"2024-09-21T11:05:57.526964+0000","mapping_epoch":1639,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1640,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:20.458421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:20.458421+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":37,"num_write":10,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":134175,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":28,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.527714+0000","last_change":"2024-09-21T11:05:49.535560+0000","last_active":"2024-09-21T11:05:57.527714+0000","last_peered":"2024-09-21T11:05:57.527714+0000","last_clean":"2024-09-21T11:05:57.527714+0000","last_became_active":"2024-09-21T11:05:49.532436+0000","last_became_peered":"2024-09-21T11:05:49.532436+0000","last_unstale":"2024-09-21T11:05:57.527714+0000","last_undegraded":"2024-09-21T11:05:57.527714+0000","last_fullsized":"2024-09-21T11:05:57.527714+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:26:49.441136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1626,"reported_epoch":1645,"state":"active+clean","last_fresh":"2024-09-21T11:05:56.525436+0000","last_change":"2024-09-21T11:02:38.476498+0000","last_active":"2024-09-21T11:05:56.525436+0000","last_peered":"2024-09-21T11:05:56.525436+0000","last_clean":"2024-09-21T11:05:56.525436+0000","last_became_active":"2024-09-21T11:02:38.476173+0000","last_became_peered":"2024-09-21T11:02:38.476173+0000","last_unstale":"2024-09-21T11:05:56.525436+0000","last_undegraded":"2024-09-21T11:05:56.525436+0000","last_fullsized":"2024-09-21T11:05:56.525436+0000","mapping_epoch":1589,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:49:26.618542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"346'149","reported_seq":966,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.526875+0000","last_change":"2024-09-21T11:05:51.519837+0000","last_active":"2024-09-21T11:05:57.526875+0000","last_peered":"2024-09-21T11:05:57.526875+0000","last_clean":"2024-09-21T11:05:57.526875+0000","last_became_active":"2024-09-21T11:05:51.519497+0000","last_became_peered":"2024-09-21T11:05:51.519497+0000","last_unstale":"2024-09-21T11:05:57.526875+0000","last_undegraded":"2024-09-21T11:05:57.526875+0000","last_fullsized":"2024-09-21T11:05:57.526875+0000","mapping_epoch":1639,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1640,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:47.955189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:47.955189+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:19:10.109394+0000","scrub_duration":0.0019884339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":374,"reported_epoch":1645,"state":"active+clean","last_fresh":"2024-09-21T11:05:56.525845+0000","last_change":"2024-09-21T11:02:39.468831+0000","last_active":"2024-09-21T11:05:56.525845+0000","last_peered":"2024-09-21T11:05:56.525845+0000","last_clean":"2024-09-21T11:05:56.525845+0000","last_became_active":"2024-09-21T11:02:39.468455+0000","last_became_peered":"2024-09-21T11:02:39.468455+0000","last_unstale":"2024-09-21T11:05:56.525845+0000","last_undegraded":"2024-09-21T11:05:56.525845+0000","last_fullsized":"2024-09-21T11:05:56.525845+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:34:59.495678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2595,"reported_epoch":1646,"state":"active+clean","last_fresh":"2024-09-21T11:05:57.530247+0000","last_change":"2024-09-21T11:04:47.608591+0000","last_active":"2024-09-21T11:05:57.530247+0000","last_peered":"2024-09-21T11:05:57.530247+0000","last_clean":"2024-09-21T11:05:57.530247+0000","last_became_active":"2024-09-21T11:04:47.607812+0000","last_became_peered":"2024-09-21T11:04:47.607812+0000","last_unstale":"2024-09-21T11:05:57.530247+0000","last_undegraded":"2024-09-21T11:05:57.530247+0000","last_fullsized":"2024-09-21T11:05:57.530247+0000","mapping_epoch":1610,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1611,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:51:16.441279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":58,"num_pg":16,"stat_sum":{"num_bytes":178847768,"num_objects":2730,"num_object_clones":0,"num_object_copies":8190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":387,"num_objects_misplaced":531,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":289184,"num_write_kb":9341185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20702,"num_bytes_recovered":1268383792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":541929472,"data_stored":541917256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73043,"ondisk_log_size":73043,"up":48,"acting":46,"num_store_stats":6},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":461855,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":187,"num_write":49,"num_write_kb":549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1721625,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3129,"ondisk_log_size":3129,"up":63,"acting":63,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":95,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":18406,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":17314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3602,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":95,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234864,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1659604,"kb_used_data":26188,"kb_used_omap":1,"kb_used_meta":1633406,"kb_avail":92085548,"statfs":{"total":95995035648,"available":94295601152,"internally_reserved":0,"allocated":26816512,"data_stored":22355218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":1672608203},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1589,"seq":6824703033386,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":412580,"kb_used_data":94300,"kb_used_omap":128,"kb_used_meta":318143,"kb_avail":93332572,"statfs":{"total":95995035648,"available":95572553728,"internally_reserved":0,"allocated":96563200,"data_stored":92086689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":131575,"internal_metadata":325778953},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":6859062771741,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019773,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1506468,"kb_used_data":145884,"kb_used_omap":1,"kb_used_meta":1360574,"kb_avail":92238684,"statfs":{"total":95995035648,"available":94452412416,"internally_reserved":0,"allocated":149385216,"data_stored":144908730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1393228230},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369517,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1562576,"kb_used_data":119944,"kb_used_omap":69,"kb_used_meta":1442490,"kb_avail":92182576,"statfs":{"total":95995035648,"available":94394957824,"internally_reserved":0,"allocated":122822656,"data_stored":118361338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":1477109890},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,2,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2920577761972,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807958203,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":942212,"kb_used_data":173820,"kb_used_omap":119,"kb_used_meta":768264,"kb_avail":92802940,"statfs":{"total":95995035648,"available":95030210560,"internally_reserved":0,"allocated":177991680,"data_stored":173541941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":121896,"internal_metadata":786703320},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":666,"seq":2860448219842,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":829468,"kb_used_data":170124,"kb_used_omap":76,"kb_used_meta":659251,"kb_avail":92915684,"statfs":{"total":95995035648,"available":95145660416,"internally_reserved":0,"allocated":174206976,"data_stored":169741571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78716,"internal_metadata":675073156},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1187,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":136450048,"data_stored":136445976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":141033472,"data_stored":141033472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":95162368,"data_stored":95158296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":117248000,"data_stored":117243928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":35323904,"data_stored":35323904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16711680,"data_stored":16711680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T11:06:05.687 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pgp_num 21 2024-09-21T11:06:05.740 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:05.762 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:05.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:05.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:05 smithi160 ceph-mon[101879]: from='client.36828 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T11:06:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:05 smithi160 ceph-mon[101879]: pgmap v3263: 109 pgs: 2 active+undersized+degraded+remapped, 107 active+clean; 171 MiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 527 op/s; 387/8268 objects degraded (4.681%); 531/8268 objects misplaced (6.422%); 6.5 MiB/s, 103 objects/s recovering 2024-09-21T11:06:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T11:06:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:06:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:06:05.864 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:05.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:05 smithi070 ceph-mon[112139]: from='client.36828 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T11:06:05.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:05 smithi070 ceph-mon[112139]: pgmap v3263: 109 pgs: 2 active+undersized+degraded+remapped, 107 active+clean; 171 MiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 527 op/s; 387/8268 objects degraded (4.681%); 531/8268 objects misplaced (6.422%); 6.5 MiB/s, 103 objects/s recovering 2024-09-21T11:06:05.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T11:06:05.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:06:05.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:06:05.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:05 smithi070 ceph-mon[107385]: from='client.36828 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T11:06:05.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:05 smithi070 ceph-mon[107385]: pgmap v3263: 109 pgs: 2 active+undersized+degraded+remapped, 107 active+clean; 171 MiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 527 op/s; 387/8268 objects degraded (4.681%); 531/8268 objects misplaced (6.422%); 6.5 MiB/s, 103 objects/s recovering 2024-09-21T11:06:05.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T11:06:05.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:06:05.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:06:05.885 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:05.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:05.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:05.911 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:06:05.911 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:06:05.911 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:06:05.911 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:06:05.922 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:06:05.922 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:06:05.939 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:06:05.939 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:06:05.955 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:06:05.958 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:06:05.971 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T11:06:05.971 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T11:06:05.987 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:05.987 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T11:06:05.988 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T11:06:06.003 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:06:06.004 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:06:06.008 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:06.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:06.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:06.110 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:06.132 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:06.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:06.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:06.233 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:06.254 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:06.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:06.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:06.356 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:06.379 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:06.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:06.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:06.480 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:06.502 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:06.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:06.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:06.604 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:06.625 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:06.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:06.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:06.726 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:06.749 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:06.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:06.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:06.824 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:06 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3069430653' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:06:06.825 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:06 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T11:06:06.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:06 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3069430653' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:06:06.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:06 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T11:06:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:06 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3069430653' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:06:06.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:06 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T11:06:06.850 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:06.872 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:06.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:06.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:06.974 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:06.996 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:06.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:06.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:07.098 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:07.119 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:07.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:07.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:07.221 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:07.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:07.243 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:07.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:07.344 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:07.366 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:07.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:07.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:07.468 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:07.490 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:07.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:07.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:07.529 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:06:07.546 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:06:07.592 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:07.614 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:07.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:07.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:07.715 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:07.736 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:07.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:07.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:07.805 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:07 smithi070 ceph-mon[112139]: pgmap v3264: 109 pgs: 2 active+undersized+remapped, 107 active+clean; 7.0 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 747 op/s; 2.6 MiB/s, 41 objects/s recovering 2024-09-21T11:06:07.805 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:07 smithi070 ceph-mon[112139]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 387/8268 objects degraded (4.681%), 2 pgs degraded) 2024-09-21T11:06:07.805 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:07 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T11:06:07.806 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:07 smithi070 ceph-mon[112139]: osdmap e1648: 8 total, 8 up, 6 in 2024-09-21T11:06:07.806 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:07 smithi070 ceph-mon[107385]: pgmap v3264: 109 pgs: 2 active+undersized+remapped, 107 active+clean; 7.0 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 747 op/s; 2.6 MiB/s, 41 objects/s recovering 2024-09-21T11:06:07.806 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:07 smithi070 ceph-mon[107385]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 387/8268 objects degraded (4.681%), 2 pgs degraded) 2024-09-21T11:06:07.806 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:07 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T11:06:07.806 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:07 smithi070 ceph-mon[107385]: osdmap e1648: 8 total, 8 up, 6 in 2024-09-21T11:06:07.838 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:07 smithi160 ceph-mon[101879]: pgmap v3264: 109 pgs: 2 active+undersized+remapped, 107 active+clean; 7.0 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 747 op/s; 2.6 MiB/s, 41 objects/s recovering 2024-09-21T11:06:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:07 smithi160 ceph-mon[101879]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 387/8268 objects degraded (4.681%), 2 pgs degraded) 2024-09-21T11:06:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:07 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T11:06:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:07 smithi160 ceph-mon[101879]: osdmap e1648: 8 total, 8 up, 6 in 2024-09-21T11:06:07.859 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:07.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:07.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:07.960 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:07.983 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:07.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:07.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:08.084 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:08.106 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:08.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:08.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:08.208 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:08.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:08.231 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:08.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:08.332 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:08.355 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:08.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:08.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:08.457 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:08.483 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:08.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:08.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:08.585 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:08.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:08.609 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:08.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:08.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:08 smithi070 ceph-mon[107385]: osdmap e1649: 8 total, 8 up, 6 in 2024-09-21T11:06:08.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:08 smithi070 ceph-mon[112139]: osdmap e1649: 8 total, 8 up, 6 in 2024-09-21T11:06:08.710 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:08.731 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:08.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:08.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:08.832 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:08.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:08 smithi160 ceph-mon[101879]: osdmap e1649: 8 total, 8 up, 6 in 2024-09-21T11:06:08.858 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:08.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:08.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:08.960 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:08.985 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:08.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:08.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:09.004 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T11:06:09.004 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T11:06:09.087 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:09.108 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:09.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:09.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:09.210 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:09.231 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:09.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:09.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:09.333 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:09.354 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:09.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:09.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:09.456 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:09.480 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:09.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:09.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:09.510 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T11:06:09.582 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:09.608 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:09.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:09.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:09.710 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:09.736 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:09.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:09.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:09.838 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:09.839 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:09 smithi160 ceph-mon[101879]: pgmap v3267: 109 pgs: 1 peering, 2 active+undersized+remapped, 106 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 507 op/s 2024-09-21T11:06:09.839 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:09 smithi160 ceph-mon[101879]: osdmap e1650: 8 total, 8 up, 6 in 2024-09-21T11:06:09.859 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:09.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:09.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:09.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:09 smithi070 ceph-mon[107385]: pgmap v3267: 109 pgs: 1 peering, 2 active+undersized+remapped, 106 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 507 op/s 2024-09-21T11:06:09.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:09 smithi070 ceph-mon[107385]: osdmap e1650: 8 total, 8 up, 6 in 2024-09-21T11:06:09.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:09 smithi070 ceph-mon[112139]: pgmap v3267: 109 pgs: 1 peering, 2 active+undersized+remapped, 106 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 507 op/s 2024-09-21T11:06:09.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:09 smithi070 ceph-mon[112139]: osdmap e1650: 8 total, 8 up, 6 in 2024-09-21T11:06:09.961 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:09.982 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:09.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:09.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:10.084 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:10.109 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:10.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:10.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:10.211 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:10.232 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:10.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:10.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:10.334 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:10.359 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:10.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:10.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:10.460 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:10.484 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:10.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:10.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:10.586 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:10.612 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:10.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:10.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:10.714 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:10.738 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:10.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:10.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:10.840 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:10.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:10 smithi160 ceph-mon[101879]: osdmap e1651: 8 total, 8 up, 6 in 2024-09-21T11:06:10.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:10 smithi160 ceph-mon[101879]: osdmap e1652: 8 total, 8 up, 6 in 2024-09-21T11:06:10.863 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:10.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:10.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:10.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:10 smithi070 ceph-mon[107385]: osdmap e1651: 8 total, 8 up, 6 in 2024-09-21T11:06:10.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:10 smithi070 ceph-mon[107385]: osdmap e1652: 8 total, 8 up, 6 in 2024-09-21T11:06:10.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:10 smithi070 ceph-mon[112139]: osdmap e1651: 8 total, 8 up, 6 in 2024-09-21T11:06:10.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:10 smithi070 ceph-mon[112139]: osdmap e1652: 8 total, 8 up, 6 in 2024-09-21T11:06:10.965 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:10.986 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:10.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:10.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:11.055 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T11:06:11.088 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:11.110 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:11.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:11.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:11.212 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:11.233 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:11.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:11.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:11.335 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:11.356 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:11.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:11.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:11.458 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:11.479 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:11.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:11.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:11.581 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:11.603 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:11.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:11.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:11.705 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:11.727 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:11.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:11.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:11.829 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:11.850 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:11.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:11.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:11.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:11 smithi070 ceph-mon[112139]: pgmap v3270: 109 pgs: 6 peering, 2 active+undersized+remapped, 101 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 26 op/s; 80 KiB/s, 0 objects/s recovering 2024-09-21T11:06:11.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:11 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:06:11.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:11 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:06:11.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:11 smithi070 ceph-mon[107385]: pgmap v3270: 109 pgs: 6 peering, 2 active+undersized+remapped, 101 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 26 op/s; 80 KiB/s, 0 objects/s recovering 2024-09-21T11:06:11.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:11 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:06:11.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:11 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:06:11.951 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:11.973 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:11.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:11.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:12.074 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:12.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:11 smithi160 ceph-mon[101879]: pgmap v3270: 109 pgs: 6 peering, 2 active+undersized+remapped, 101 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 26 op/s; 80 KiB/s, 0 objects/s recovering 2024-09-21T11:06:12.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:11 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:06:12.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:11 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:06:12.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:12.098 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:12.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:12.199 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:12.221 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:12.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:12.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:12.322 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:12.346 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:12.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:12.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:12.448 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:12.474 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:12.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:12.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:12.576 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:12.602 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:12.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:12.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:12.704 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:12.730 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:12.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:12.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:12.832 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:12.853 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:12.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:12.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:12.954 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:12.975 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:12.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:12.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:13.045 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:06:13.077 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:13.082 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:06:13.105 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:13.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:13.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:13.175 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:06:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:06:13] "GET /metrics HTTP/1.1" 200 36746 "" "Prometheus/2.51.0" 2024-09-21T11:06:13.207 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:13.227 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:13.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:13.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:13.329 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:13.354 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:13.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:13.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:13.456 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:13.478 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:13.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:13.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:13.579 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:13.600 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:13.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:13.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:13.602 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is set 2024-09-21T11:06:13.602 INFO:teuthology.orchestra.run.smithi070.stderr:set pool 1 pgp_num to 21 2024-09-21T11:06:13.702 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:13.726 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:13.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:13.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:13.828 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:13.849 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:13.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:13.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:13.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:13 smithi070 ceph-mon[107385]: pgmap v3272: 109 pgs: 5 peering, 2 active+undersized+remapped, 102 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-21T11:06:13.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:13 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2653244071' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:06:13.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:13 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4147371219' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-21T11:06:13.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:13 smithi070 ceph-mon[112139]: pgmap v3272: 109 pgs: 5 peering, 2 active+undersized+remapped, 102 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-21T11:06:13.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:13 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2653244071' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:06:13.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:13 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4147371219' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-21T11:06:13.950 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:13.976 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:13.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:13.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:14.078 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:14.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:13 smithi160 ceph-mon[101879]: pgmap v3272: 109 pgs: 5 peering, 2 active+undersized+remapped, 102 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-21T11:06:14.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:13 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2653244071' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:06:14.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:13 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4147371219' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-21T11:06:14.099 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:14.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:14.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:14.201 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:14.222 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:14.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:14.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:14.324 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:14.350 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:14.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:14.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:14.452 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:14.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:14.477 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:14.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:14.578 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:14.599 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:14.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:14.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:14.702 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:14.724 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:14.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:14.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:14.825 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:14.847 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:14.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:14.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:14.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:14 smithi070 ceph-mon[107385]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:06:14.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:14 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2653244071' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T11:06:14.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:14 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4147371219' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-21T11:06:14.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:14 smithi070 ceph-mon[107385]: osdmap e1653: 8 total, 8 up, 6 in 2024-09-21T11:06:14.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:14 smithi070 ceph-mon[112139]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:06:14.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:14 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2653244071' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T11:06:14.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:14 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4147371219' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-21T11:06:14.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:14 smithi070 ceph-mon[112139]: osdmap e1653: 8 total, 8 up, 6 in 2024-09-21T11:06:14.949 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:14.971 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:14.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:14.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:15.072 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:15.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:14 smithi160 ceph-mon[101879]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:06:15.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:14 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2653244071' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T11:06:15.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:14 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4147371219' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-21T11:06:15.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:14 smithi160 ceph-mon[101879]: osdmap e1653: 8 total, 8 up, 6 in 2024-09-21T11:06:15.095 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:15.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:15.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:15.197 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:15.223 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:15.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:15.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:15.324 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:15.349 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:15.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:15.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:15.451 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:15.473 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:15.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:15.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:15.496 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_5 2024-09-21T11:06:15.496 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-09-21T11:06:15.574 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:15.600 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:15.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:15.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:15.701 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:15.727 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:15.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:15.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:15.787 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:15 smithi070 ceph-mon[107385]: pgmap v3274: 109 pgs: 5 peering, 2 active+undersized+remapped, 102 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-09-21T11:06:15.787 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:15 smithi070 ceph-mon[112139]: pgmap v3274: 109 pgs: 5 peering, 2 active+undersized+remapped, 102 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-09-21T11:06:15.828 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:15.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:15.850 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:15.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:15.951 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:15.977 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:15.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:15.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:16.079 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:16.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:15 smithi160 ceph-mon[101879]: pgmap v3274: 109 pgs: 5 peering, 2 active+undersized+remapped, 102 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-09-21T11:06:16.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:16.101 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:16.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:16.201 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:16.227 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:16.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:16.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:16.329 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:16.356 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:16.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:16.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:16.457 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:16.479 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:16.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:16.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:16.580 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:16.601 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:16.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:16.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:16.703 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:16.724 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:16.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:16.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:16.826 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:16.852 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:16.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:16.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:16.954 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:16.979 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:16.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:16.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:17.081 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:17.103 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:17.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:17.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:17.205 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:17.231 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:17.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:17.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:17.333 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:17.355 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:17.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:17.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:17.457 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:17.477 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:17.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:17.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:17.579 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:17.601 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:17.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:17.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:17.702 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:17.728 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:17.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:17.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:17.830 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:17.854 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:17.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:17.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:17.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:17 smithi070 ceph-mon[112139]: pgmap v3275: 109 pgs: 2 active+undersized+remapped, 107 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 47 KiB/s, 0 objects/s recovering 2024-09-21T11:06:17.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:17 smithi070 ceph-mon[107385]: pgmap v3275: 109 pgs: 2 active+undersized+remapped, 107 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 47 KiB/s, 0 objects/s recovering 2024-09-21T11:06:17.955 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:17.977 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:17.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:17.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:18.067 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T11:06:18.079 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:18.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:17 smithi160 ceph-mon[101879]: pgmap v3275: 109 pgs: 2 active+undersized+remapped, 107 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 47 KiB/s, 0 objects/s recovering 2024-09-21T11:06:18.104 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:18.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:18.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:18.205 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:18.226 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:18.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:18.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:18.326 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:06:18.326 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:06:18.328 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:18.343 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:06:18.343 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:06:18.354 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:18.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:18.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:18.359 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:06:18.360 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:06:18.375 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:06:18.375 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:06:18.392 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:06:18.392 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:06:18.409 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T11:06:18.409 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T11:06:18.425 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T11:06:18.425 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T11:06:18.442 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:06:18.442 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:06:18.443 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:06:18.456 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:18.481 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:18.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:18.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:18.583 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:18.604 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:18.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:18.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:18.706 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:18.730 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:18.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:18.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:18.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:18 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3421013911' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:06:18.796 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:18 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3421013911' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:06:18.832 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:18.854 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:18.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:18.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:18.955 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:18.976 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:18.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:18.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:19.078 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:19.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:18 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3421013911' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:06:19.104 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:19.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:19.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:19.206 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:19.227 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:19.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:19.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:19.329 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:19.350 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:19.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:19.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:19.451 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:19.478 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:19.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:19.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:19.579 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:19.601 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:19.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:19.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:19.643 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T11:06:19.644 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T11:06:19.644 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T11:06:19.644 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T11:06:19.644 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.644 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:285219 58.a 58:588dcfd1:::benchmark_data_smithi070_396992_object139260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1645)", 2024-09-21T11:06:19.644 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:56.563762+0000", 2024-09-21T11:06:19.644 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 23.077049408000001, 2024-09-21T11:06:19.644 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.026254811, 2024-09-21T11:06:19.644 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:19.644 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:19.644 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:19.645 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:19.645 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:19.645 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:19.645 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 285219 2024-09-21T11:06:19.645 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.645 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:19.645 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.645 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:19.645 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.563762+0000", 2024-09-21T11:06:19.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.645 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.645 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.645 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:19.645 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.563762+0000", 2024-09-21T11:06:19.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.646 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.646 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.646 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:19.646 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.563764+0000", 2024-09-21T11:06:19.646 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5260000000000001e-06 2024-09-21T11:06:19.646 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.646 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.646 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:19.646 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.563767+0000", 2024-09-21T11:06:19.646 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2590000000000001e-06 2024-09-21T11:06:19.646 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.646 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.646 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:19.646 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.563775+0000", 2024-09-21T11:06:19.646 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0199999999999994e-06 2024-09-21T11:06:19.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.647 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:19.647 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.563780+0000", 2024-09-21T11:06:19.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1490000000000002e-06 2024-09-21T11:06:19.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.647 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.647 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.563836+0000", 2024-09-21T11:06:19.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5578999999999998e-05 2024-09-21T11:06:19.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.647 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:19.647 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.563844+0000", 2024-09-21T11:06:19.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3739999999999998e-06 2024-09-21T11:06:19.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.648 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.648 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.573839+0000", 2024-09-21T11:06:19.648 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0099947279999999 2024-09-21T11:06:19.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.648 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:19.648 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.573929+0000", 2024-09-21T11:06:19.648 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9790999999999995e-05 2024-09-21T11:06:19.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.648 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,7", 2024-09-21T11:06:19.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.574175+0000", 2024-09-21T11:06:19.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000246023 2024-09-21T11:06:19.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:19.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.583478+0000", 2024-09-21T11:06:19.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0093032700000000006 2024-09-21T11:06:19.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.585998+0000", 2024-09-21T11:06:19.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025197319999999998 2024-09-21T11:06:19.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.589946+0000", 2024-09-21T11:06:19.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0039485190000000002 2024-09-21T11:06:19.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.650 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:19.650 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.589985+0000", 2024-09-21T11:06:19.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8682e-05 2024-09-21T11:06:19.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.650 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:19.650 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.590017+0000", 2024-09-21T11:06:19.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2159000000000002e-05 2024-09-21T11:06:19.650 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.650 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:19.650 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.651 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:285226 58.a 58:59325d5f:::benchmark_data_smithi070_396992_object139267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1645)", 2024-09-21T11:06:19.651 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:56.570441+0000", 2024-09-21T11:06:19.651 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 23.070371153, 2024-09-21T11:06:19.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.019698357, 2024-09-21T11:06:19.651 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:19.651 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:19.651 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:19.651 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:19.651 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:19.651 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:19.651 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 285226 2024-09-21T11:06:19.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.652 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:19.652 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.652 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:19.652 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.570441+0000", 2024-09-21T11:06:19.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.652 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.652 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:19.652 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.570441+0000", 2024-09-21T11:06:19.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.652 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.652 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:19.652 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.570442+0000", 2024-09-21T11:06:19.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4160000000000001e-06 2024-09-21T11:06:19.653 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.653 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.653 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:19.653 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.570447+0000", 2024-09-21T11:06:19.653 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6580000000000001e-06 2024-09-21T11:06:19.653 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.653 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.653 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:19.653 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.570461+0000", 2024-09-21T11:06:19.653 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4144e-05 2024-09-21T11:06:19.653 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.653 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.653 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:19.653 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.570468+0000", 2024-09-21T11:06:19.653 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6769999999999995e-06 2024-09-21T11:06:19.653 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.653 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.654 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.654 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.570531+0000", 2024-09-21T11:06:19.654 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2287999999999997e-05 2024-09-21T11:06:19.654 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.654 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.654 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:19.654 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.570540+0000", 2024-09-21T11:06:19.654 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0529999999999996e-06 2024-09-21T11:06:19.654 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.654 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.654 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.654 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.580549+0000", 2024-09-21T11:06:19.654 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.010009629 2024-09-21T11:06:19.654 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.654 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.654 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:19.655 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.580644+0000", 2024-09-21T11:06:19.655 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4118999999999998e-05 2024-09-21T11:06:19.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.655 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,7", 2024-09-21T11:06:19.655 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.580719+0000", 2024-09-21T11:06:19.655 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5693999999999997e-05 2024-09-21T11:06:19.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.655 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:19.655 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.583560+0000", 2024-09-21T11:06:19.655 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0028408359999999998 2024-09-21T11:06:19.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.655 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.655 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.588140+0000", 2024-09-21T11:06:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.004580008 2024-09-21T11:06:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.590073+0000", 2024-09-21T11:06:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001933057 2024-09-21T11:06:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.590112+0000", 2024-09-21T11:06:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8559000000000001e-05 2024-09-21T11:06:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.657 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:19.657 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.590139+0000", 2024-09-21T11:06:19.657 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7219000000000001e-05 2024-09-21T11:06:19.657 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.657 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:19.657 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.658 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.658 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.658 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:285240 58.a 58:5943288e:::benchmark_data_smithi070_396992_object139281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1645)", 2024-09-21T11:06:19.658 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:56.579583+0000", 2024-09-21T11:06:19.658 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 23.061228400000001, 2024-09-21T11:06:19.658 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.013241684, 2024-09-21T11:06:19.658 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:19.658 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:19.658 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:19.658 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:19.658 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:19.658 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:19.658 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 285240 2024-09-21T11:06:19.658 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.658 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:19.658 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.659 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:19.659 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.579583+0000", 2024-09-21T11:06:19.659 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.659 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.659 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:19.659 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.579583+0000", 2024-09-21T11:06:19.659 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.659 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.659 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:19.659 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.579585+0000", 2024-09-21T11:06:19.659 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3680000000000001e-06 2024-09-21T11:06:19.659 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.660 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:19.660 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.579588+0000", 2024-09-21T11:06:19.660 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4309999999999998e-06 2024-09-21T11:06:19.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.660 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:19.660 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.579598+0000", 2024-09-21T11:06:19.660 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0326000000000001e-05 2024-09-21T11:06:19.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.660 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:19.660 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.579605+0000", 2024-09-21T11:06:19.660 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3350000000000003e-06 2024-09-21T11:06:19.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.660 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.661 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.579657+0000", 2024-09-21T11:06:19.661 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1860999999999997e-05 2024-09-21T11:06:19.661 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.661 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.661 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:19.661 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.579664+0000", 2024-09-21T11:06:19.661 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6249999999999998e-06 2024-09-21T11:06:19.661 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.661 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.661 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.661 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.581487+0000", 2024-09-21T11:06:19.661 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.00182237 2024-09-21T11:06:19.661 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.661 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.661 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:19.662 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.581595+0000", 2024-09-21T11:06:19.662 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000108686 2024-09-21T11:06:19.662 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.662 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.662 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,7", 2024-09-21T11:06:19.662 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.581717+0000", 2024-09-21T11:06:19.662 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000121382 2024-09-21T11:06:19.662 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.662 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.662 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:19.662 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.585942+0000", 2024-09-21T11:06:19.662 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0042249210000000004 2024-09-21T11:06:19.662 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.662 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.663 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.589863+0000", 2024-09-21T11:06:19.663 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0039210920000000002 2024-09-21T11:06:19.663 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.663 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.663 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.592734+0000", 2024-09-21T11:06:19.663 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0028717370000000001 2024-09-21T11:06:19.663 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.663 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:19.663 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.592781+0000", 2024-09-21T11:06:19.663 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6010999999999997e-05 2024-09-21T11:06:19.663 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.663 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:19.664 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.592825+0000", 2024-09-21T11:06:19.664 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.4539000000000002e-05 2024-09-21T11:06:19.664 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.664 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:19.664 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.664 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.664 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:285249 58.a 58:54f7e629:::benchmark_data_smithi070_396992_object139290:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1645)", 2024-09-21T11:06:19.664 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:56.588424+0000", 2024-09-21T11:06:19.664 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 23.052387639999999, 2024-09-21T11:06:19.664 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0045879259999999, 2024-09-21T11:06:19.664 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:19.664 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:19.664 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:19.664 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:19.664 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:19.665 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:19.665 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 285249 2024-09-21T11:06:19.665 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.665 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:19.665 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.665 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:19.665 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.588424+0000", 2024-09-21T11:06:19.665 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.665 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.665 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.665 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:19.665 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.588424+0000", 2024-09-21T11:06:19.665 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.665 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.665 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.666 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:19.666 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.588426+0000", 2024-09-21T11:06:19.666 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5570000000000001e-06 2024-09-21T11:06:19.666 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.666 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.666 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:19.666 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.588434+0000", 2024-09-21T11:06:19.666 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3270000000000001e-06 2024-09-21T11:06:19.666 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.666 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.666 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:19.666 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.588448+0000", 2024-09-21T11:06:19.666 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4173e-05 2024-09-21T11:06:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.588456+0000", 2024-09-21T11:06:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6769999999999995e-06 2024-09-21T11:06:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.588513+0000", 2024-09-21T11:06:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7266999999999999e-05 2024-09-21T11:06:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.588522+0000", 2024-09-21T11:06:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3340000000000008e-06 2024-09-21T11:06:19.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.668 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.668 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.668 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.582535+0000", 2024-09-21T11:06:19.668 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99401276100000002 2024-09-21T11:06:19.668 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.668 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.668 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:19.668 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.582642+0000", 2024-09-21T11:06:19.668 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000106449 2024-09-21T11:06:19.668 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.668 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.668 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,7", 2024-09-21T11:06:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.582754+0000", 2024-09-21T11:06:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011267 2024-09-21T11:06:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.585945+0000", 2024-09-21T11:06:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0031903470000000001 2024-09-21T11:06:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.589911+0000", 2024-09-21T11:06:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0039667050000000001 2024-09-21T11:06:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.669 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.670 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.592922+0000", 2024-09-21T11:06:19.670 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0030109410000000001 2024-09-21T11:06:19.670 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.670 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.670 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:19.670 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.592975+0000", 2024-09-21T11:06:19.670 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3112000000000002e-05 2024-09-21T11:06:19.670 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.670 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.670 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:19.670 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.593012+0000", 2024-09-21T11:06:19.670 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6606000000000002e-05 2024-09-21T11:06:19.670 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:285271 58.a 58:5ef52b4e:::benchmark_data_smithi070_396992_object139312:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1645)", 2024-09-21T11:06:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:56.606865+0000", 2024-09-21T11:06:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 23.033946324999999, 2024-09-21T11:06:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.98627759199999998, 2024-09-21T11:06:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:19.671 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 285271 2024-09-21T11:06:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.606865+0000", 2024-09-21T11:06:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.606865+0000", 2024-09-21T11:06:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.672 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.606867+0000", 2024-09-21T11:06:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5999999999999999e-06 2024-09-21T11:06:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.606872+0000", 2024-09-21T11:06:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6550000000000003e-06 2024-09-21T11:06:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.606884+0000", 2024-09-21T11:06:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1958e-05 2024-09-21T11:06:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.673 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.606891+0000", 2024-09-21T11:06:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.396e-06 2024-09-21T11:06:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.606939+0000", 2024-09-21T11:06:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8084000000000002e-05 2024-09-21T11:06:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.606947+0000", 2024-09-21T11:06:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.695e-06 2024-09-21T11:06:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.674 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.583607+0000", 2024-09-21T11:06:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.97665987700000001 2024-09-21T11:06:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.583733+0000", 2024-09-21T11:06:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012601499999999999 2024-09-21T11:06:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,7", 2024-09-21T11:06:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.583854+0000", 2024-09-21T11:06:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000121023 2024-09-21T11:06:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.675 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.586882+0000", 2024-09-21T11:06:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0030282489999999998 2024-09-21T11:06:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.592689+0000", 2024-09-21T11:06:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0058069180000000003 2024-09-21T11:06:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.593056+0000", 2024-09-21T11:06:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00036660000000000002 2024-09-21T11:06:19.676 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.593105+0000", 2024-09-21T11:06:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9421e-05 2024-09-21T11:06:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.593143+0000", 2024-09-21T11:06:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8101e-05 2024-09-21T11:06:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.677 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:285273 58.a 58:52769141:::benchmark_data_smithi070_396992_object139314:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1645)", 2024-09-21T11:06:19.678 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:56.606919+0000", 2024-09-21T11:06:19.678 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 23.033892763000001, 2024-09-21T11:06:19.678 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99301413699999996, 2024-09-21T11:06:19.678 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:19.678 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:19.678 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:19.678 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:19.678 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:19.678 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:19.678 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 285273 2024-09-21T11:06:19.678 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.678 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:19.678 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.678 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:19.679 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.606919+0000", 2024-09-21T11:06:19.679 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.679 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.679 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.679 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:19.679 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.606919+0000", 2024-09-21T11:06:19.679 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.679 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.679 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.679 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:19.679 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.606920+0000", 2024-09-21T11:06:19.679 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2699999999999999e-07 2024-09-21T11:06:19.679 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.679 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.680 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:19.680 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.606922+0000", 2024-09-21T11:06:19.680 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8119999999999999e-06 2024-09-21T11:06:19.680 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.680 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.680 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:19.680 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.606931+0000", 2024-09-21T11:06:19.680 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1759999999999993e-06 2024-09-21T11:06:19.680 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.680 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.680 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:19.680 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.606937+0000", 2024-09-21T11:06:19.680 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1770000000000003e-06 2024-09-21T11:06:19.680 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.681 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.681 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.681 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.606966+0000", 2024-09-21T11:06:19.681 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9459e-05 2024-09-21T11:06:19.681 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.681 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.681 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:19.681 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.606971+0000", 2024-09-21T11:06:19.681 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.169e-06 2024-09-21T11:06:19.681 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.681 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.681 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.681 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.584732+0000", 2024-09-21T11:06:19.682 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.97776079800000004 2024-09-21T11:06:19.682 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.682 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.682 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:19.682 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.584834+0000", 2024-09-21T11:06:19.682 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000102087 2024-09-21T11:06:19.682 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.682 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.682 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,7", 2024-09-21T11:06:19.682 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.584949+0000", 2024-09-21T11:06:19.682 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000114645 2024-09-21T11:06:19.682 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.682 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.682 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:19.682 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.588148+0000", 2024-09-21T11:06:19.683 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0031988120000000001 2024-09-21T11:06:19.683 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.683 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.683 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.683 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.592866+0000", 2024-09-21T11:06:19.683 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0047185730000000002 2024-09-21T11:06:19.683 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.683 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.683 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.683 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.599793+0000", 2024-09-21T11:06:19.683 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0069268949999999998 2024-09-21T11:06:19.683 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.683 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.683 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:19.684 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.599890+0000", 2024-09-21T11:06:19.684 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.6359000000000004e-05 2024-09-21T11:06:19.684 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.684 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.684 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:19.684 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.599933+0000", 2024-09-21T11:06:19.684 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3547999999999997e-05 2024-09-21T11:06:19.684 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.684 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:19.684 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.684 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.684 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.684 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:285305 58.a 58:591327c3:::benchmark_data_smithi070_396992_object139346:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1645)", 2024-09-21T11:06:19.685 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:56.634195+0000", 2024-09-21T11:06:19.685 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 23.006617216999999, 2024-09-21T11:06:19.685 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.96587442899999998, 2024-09-21T11:06:19.685 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:19.685 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:19.685 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:19.685 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:19.685 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:19.685 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:19.685 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 285305 2024-09-21T11:06:19.685 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.685 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:19.685 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.685 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:19.685 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.634195+0000", 2024-09-21T11:06:19.685 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.686 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.686 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.686 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:19.686 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.634195+0000", 2024-09-21T11:06:19.686 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.686 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.686 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.686 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:19.686 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.634196+0000", 2024-09-21T11:06:19.686 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5820000000000001e-06 2024-09-21T11:06:19.686 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.686 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.686 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:19.687 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.634201+0000", 2024-09-21T11:06:19.687 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8629999999999999e-06 2024-09-21T11:06:19.687 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.687 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.687 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:19.687 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.634214+0000", 2024-09-21T11:06:19.687 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2733e-05 2024-09-21T11:06:19.687 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.687 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.687 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:19.687 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.634221+0000", 2024-09-21T11:06:19.687 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.61e-06 2024-09-21T11:06:19.687 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.687 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.687 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.687 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.634341+0000", 2024-09-21T11:06:19.688 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000119337 2024-09-21T11:06:19.688 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.688 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.688 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:19.688 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.634349+0000", 2024-09-21T11:06:19.688 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8809999999999996e-06 2024-09-21T11:06:19.688 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.688 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.688 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.688 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.586036+0000", 2024-09-21T11:06:19.688 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.95168719300000004 2024-09-21T11:06:19.688 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.688 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.688 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:19.688 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.586102+0000", 2024-09-21T11:06:19.689 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6576999999999996e-05 2024-09-21T11:06:19.689 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.689 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.689 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,7", 2024-09-21T11:06:19.689 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.586170+0000", 2024-09-21T11:06:19.689 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7506e-05 2024-09-21T11:06:19.689 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.689 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.689 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:19.689 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.589870+0000", 2024-09-21T11:06:19.689 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0037003090000000002 2024-09-21T11:06:19.689 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.689 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.689 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.690 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.596083+0000", 2024-09-21T11:06:19.690 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0062124520000000003 2024-09-21T11:06:19.690 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.690 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.690 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.690 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.599978+0000", 2024-09-21T11:06:19.690 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0038955219999999998 2024-09-21T11:06:19.690 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.690 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.690 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:19.690 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.600034+0000", 2024-09-21T11:06:19.690 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5897999999999999e-05 2024-09-21T11:06:19.690 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.690 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.690 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:19.691 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.600069+0000", 2024-09-21T11:06:19.691 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4965999999999997e-05 2024-09-21T11:06:19.691 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.691 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:19.691 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.691 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.691 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.691 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:285334 58.a 58:5fb79388:::benchmark_data_smithi070_396992_object139375:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1645)", 2024-09-21T11:06:19.691 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:56.657492+0000", 2024-09-21T11:06:19.691 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 22.983319584, 2024-09-21T11:06:19.691 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.94272302399999997, 2024-09-21T11:06:19.691 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:19.691 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:19.692 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:19.692 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:19.692 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:19.692 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:19.692 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 285334 2024-09-21T11:06:19.692 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.692 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:19.692 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.692 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:19.692 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.657492+0000", 2024-09-21T11:06:19.692 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.692 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.692 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.692 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:19.692 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.657492+0000", 2024-09-21T11:06:19.693 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.693 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.693 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.693 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:19.693 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.657494+0000", 2024-09-21T11:06:19.693 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4780000000000001e-06 2024-09-21T11:06:19.693 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.693 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.693 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:19.693 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.657498+0000", 2024-09-21T11:06:19.693 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5909999999999998e-06 2024-09-21T11:06:19.693 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.693 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.693 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:19.693 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.657511+0000", 2024-09-21T11:06:19.693 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3093e-05 2024-09-21T11:06:19.694 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.694 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.694 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:19.694 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.657519+0000", 2024-09-21T11:06:19.694 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3359999999999997e-06 2024-09-21T11:06:19.694 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.694 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.694 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.694 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.657609+0000", 2024-09-21T11:06:19.694 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0546999999999999e-05 2024-09-21T11:06:19.694 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.694 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.694 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:19.694 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.657618+0000", 2024-09-21T11:06:19.695 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7409999999999998e-06 2024-09-21T11:06:19.695 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.695 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.695 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.695 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.586921+0000", 2024-09-21T11:06:19.695 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.92930344200000004 2024-09-21T11:06:19.695 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.695 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.695 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:19.695 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.587027+0000", 2024-09-21T11:06:19.695 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010553299999999999 2024-09-21T11:06:19.695 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.695 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.695 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,7", 2024-09-21T11:06:19.696 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.587154+0000", 2024-09-21T11:06:19.696 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000126993 2024-09-21T11:06:19.696 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.696 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.696 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:19.696 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.590024+0000", 2024-09-21T11:06:19.696 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0028704580000000002 2024-09-21T11:06:19.696 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.696 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.696 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.696 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.596135+0000", 2024-09-21T11:06:19.696 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0061103520000000003 2024-09-21T11:06:19.696 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.696 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.696 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.697 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.600116+0000", 2024-09-21T11:06:19.697 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0039814380000000003 2024-09-21T11:06:19.697 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.697 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.697 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:19.697 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.600176+0000", 2024-09-21T11:06:19.697 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9580999999999998e-05 2024-09-21T11:06:19.697 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.697 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.697 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:19.697 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.600215+0000", 2024-09-21T11:06:19.697 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9440999999999997e-05 2024-09-21T11:06:19.697 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.697 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:19.697 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.697 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.698 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.698 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:285382 58.a 58:517b7ab8:::benchmark_data_smithi070_396992_object139423:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1645)", 2024-09-21T11:06:19.698 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:56.692264+0000", 2024-09-21T11:06:19.698 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 22.948548078999998, 2024-09-21T11:06:19.698 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.90815624299999997, 2024-09-21T11:06:19.698 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:19.698 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:19.698 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:19.698 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:19.698 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:19.698 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:19.698 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 285382 2024-09-21T11:06:19.698 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.698 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:19.699 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.699 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:19.699 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.692264+0000", 2024-09-21T11:06:19.699 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.699 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.699 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.699 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:19.699 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.692264+0000", 2024-09-21T11:06:19.699 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.699 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.699 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.699 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:19.699 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.692265+0000", 2024-09-21T11:06:19.699 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.46e-06 2024-09-21T11:06:19.700 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.700 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.700 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:19.700 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.692270+0000", 2024-09-21T11:06:19.700 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.78e-06 2024-09-21T11:06:19.700 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.700 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.700 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:19.700 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.692286+0000", 2024-09-21T11:06:19.700 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6201e-05 2024-09-21T11:06:19.700 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.700 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.700 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:19.700 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.692294+0000", 2024-09-21T11:06:19.700 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1489999999999994e-06 2024-09-21T11:06:19.700 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.701 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.701 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.701 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.692365+0000", 2024-09-21T11:06:19.701 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0640000000000001e-05 2024-09-21T11:06:19.701 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.701 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.701 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:19.701 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.692382+0000", 2024-09-21T11:06:19.701 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7328000000000001e-05 2024-09-21T11:06:19.701 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.701 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.701 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.701 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.588173+0000", 2024-09-21T11:06:19.701 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.89579099299999998 2024-09-21T11:06:19.702 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.702 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.702 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:19.702 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.588233+0000", 2024-09-21T11:06:19.702 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9815999999999998e-05 2024-09-21T11:06:19.702 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.702 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.702 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,7", 2024-09-21T11:06:19.702 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.588295+0000", 2024-09-21T11:06:19.702 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1472999999999999e-05 2024-09-21T11:06:19.702 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.702 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.702 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:19.702 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.590028+0000", 2024-09-21T11:06:19.702 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001733456 2024-09-21T11:06:19.702 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.703 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.703 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.703 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.596211+0000", 2024-09-21T11:06:19.703 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0061831539999999997 2024-09-21T11:06:19.703 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.703 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.703 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.703 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.600263+0000", 2024-09-21T11:06:19.703 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0040522040000000002 2024-09-21T11:06:19.703 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.703 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.703 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:19.703 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.600377+0000", 2024-09-21T11:06:19.703 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000113502 2024-09-21T11:06:19.703 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.704 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.704 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:19.704 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.600420+0000", 2024-09-21T11:06:19.704 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3087000000000003e-05 2024-09-21T11:06:19.704 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.704 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:19.704 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.704 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.704 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.704 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:285389 58.a 58:5321eb12:::benchmark_data_smithi070_396992_object139430:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1645)", 2024-09-21T11:06:19.704 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:56.696615+0000", 2024-09-21T11:06:19.704 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 22.944196756, 2024-09-21T11:06:19.704 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.90731865499999997, 2024-09-21T11:06:19.704 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:19.704 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:19.704 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:19.705 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:19.705 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:19.705 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:19.705 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 285389 2024-09-21T11:06:19.705 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.705 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:19.705 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.705 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:19.705 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.696615+0000", 2024-09-21T11:06:19.705 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.705 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.705 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.705 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:19.705 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.696615+0000", 2024-09-21T11:06:19.706 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.706 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.706 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.706 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:19.706 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.696617+0000", 2024-09-21T11:06:19.706 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5e-06 2024-09-21T11:06:19.706 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.706 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.706 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:19.706 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.696621+0000", 2024-09-21T11:06:19.706 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.1309999999999999e-06 2024-09-21T11:06:19.706 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.706 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.706 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:19.707 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.696637+0000", 2024-09-21T11:06:19.707 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6348999999999999e-05 2024-09-21T11:06:19.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.707 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.707 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:19.707 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.696644+0000", 2024-09-21T11:06:19.707 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4919999999999996e-06 2024-09-21T11:06:19.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.707 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.707 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.707 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.696695+0000", 2024-09-21T11:06:19.707 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0584000000000002e-05 2024-09-21T11:06:19.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.707 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.708 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:19.708 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.696703+0000", 2024-09-21T11:06:19.708 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2130000000000008e-06 2024-09-21T11:06:19.708 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.708 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.708 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.708 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.588956+0000", 2024-09-21T11:06:19.708 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.89225263600000004 2024-09-21T11:06:19.708 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.708 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.708 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:19.708 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.589015+0000", 2024-09-21T11:06:19.708 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8838000000000001e-05 2024-09-21T11:06:19.708 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.709 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.709 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,7", 2024-09-21T11:06:19.709 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.589091+0000", 2024-09-21T11:06:19.709 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5883000000000005e-05 2024-09-21T11:06:19.709 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.709 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.709 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:19.709 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.591546+0000", 2024-09-21T11:06:19.709 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0024552710000000002 2024-09-21T11:06:19.709 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.709 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.709 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:19.711 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.711 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.596310+0000", 2024-09-21T11:06:19.711 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0047644430000000002 2024-09-21T11:06:19.711 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.711 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.711 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.711 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.603848+0000", 2024-09-21T11:06:19.711 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0075374040000000002 2024-09-21T11:06:19.711 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.711 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.711 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:19.711 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.603893+0000", 2024-09-21T11:06:19.711 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5466000000000001e-05 2024-09-21T11:06:19.711 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.712 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.712 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:19.712 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.603934+0000", 2024-09-21T11:06:19.712 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0445000000000003e-05 2024-09-21T11:06:19.712 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.712 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:19.712 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.712 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.712 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.712 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:285391 58.a 58:54f4ce74:::benchmark_data_smithi070_396992_object139432:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1645)", 2024-09-21T11:06:19.712 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:56.698661+0000", 2024-09-21T11:06:19.712 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 22.942151243000001, 2024-09-21T11:06:19.712 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.91047981700000002, 2024-09-21T11:06:19.712 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:19.713 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:19.713 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:19.713 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:19.713 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:19.713 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:19.713 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 285391 2024-09-21T11:06:19.713 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.713 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:19.713 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.713 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:19.713 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.698661+0000", 2024-09-21T11:06:19.713 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.713 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.713 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.714 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:19.714 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.698661+0000", 2024-09-21T11:06:19.714 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.714 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.714 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:19.714 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.698662+0000", 2024-09-21T11:06:19.714 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6270000000000001e-06 2024-09-21T11:06:19.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.714 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.714 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:19.714 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.698671+0000", 2024-09-21T11:06:19.714 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.8540000000000003e-06 2024-09-21T11:06:19.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.715 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.715 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:19.715 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.698686+0000", 2024-09-21T11:06:19.715 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5437000000000001e-05 2024-09-21T11:06:19.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.715 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.715 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:19.715 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.698695+0000", 2024-09-21T11:06:19.715 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1499999999999999e-06 2024-09-21T11:06:19.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.715 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.715 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.715 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.698755+0000", 2024-09-21T11:06:19.715 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0238999999999997e-05 2024-09-21T11:06:19.716 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.716 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.716 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:19.716 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.698764+0000", 2024-09-21T11:06:19.716 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5340000000000005e-06 2024-09-21T11:06:19.716 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.716 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.716 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.716 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.590188+0000", 2024-09-21T11:06:19.716 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.89142343700000004 2024-09-21T11:06:19.716 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.716 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.716 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:19.716 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.590298+0000", 2024-09-21T11:06:19.716 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011044000000000001 2024-09-21T11:06:19.716 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.717 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.717 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,7", 2024-09-21T11:06:19.717 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.590365+0000", 2024-09-21T11:06:19.717 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6767999999999995e-05 2024-09-21T11:06:19.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.717 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.717 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:19.717 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.592836+0000", 2024-09-21T11:06:19.717 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0024705590000000002 2024-09-21T11:06:19.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.717 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.717 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.717 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.596359+0000", 2024-09-21T11:06:19.718 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.003523123 2024-09-21T11:06:19.718 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.718 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.718 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.718 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.609049+0000", 2024-09-21T11:06:19.718 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012690582000000001 2024-09-21T11:06:19.718 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.718 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.718 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:19.718 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.609097+0000", 2024-09-21T11:06:19.718 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8235999999999997e-05 2024-09-21T11:06:19.718 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.718 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.718 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:19.718 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.609140+0000", 2024-09-21T11:06:19.718 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2830999999999997e-05 2024-09-21T11:06:19.719 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.719 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:19.719 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.719 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.719 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.719 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:285394 58.a 58:526c2044:::benchmark_data_smithi070_396992_object139435:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1645)", 2024-09-21T11:06:19.719 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:56.700852+0000", 2024-09-21T11:06:19.719 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 22.939959348999999, 2024-09-21T11:06:19.719 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.90890858200000002, 2024-09-21T11:06:19.719 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:19.719 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:19.719 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:19.719 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:19.719 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:19.719 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:19.719 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 285394 2024-09-21T11:06:19.720 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.720 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:19.720 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.720 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:19.720 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.700852+0000", 2024-09-21T11:06:19.720 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.720 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.720 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.720 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:19.720 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.700852+0000", 2024-09-21T11:06:19.720 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.720 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.720 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.720 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:19.721 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.700854+0000", 2024-09-21T11:06:19.721 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3850000000000001e-06 2024-09-21T11:06:19.721 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.721 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.721 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:19.721 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.700858+0000", 2024-09-21T11:06:19.721 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6739999999999996e-06 2024-09-21T11:06:19.721 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.721 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.721 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:19.721 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.700870+0000", 2024-09-21T11:06:19.721 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1956999999999999e-05 2024-09-21T11:06:19.721 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.721 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.721 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:19.721 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.700878+0000", 2024-09-21T11:06:19.722 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8359999999999992e-06 2024-09-21T11:06:19.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.722 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.722 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.722 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.700933+0000", 2024-09-21T11:06:19.722 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4598999999999997e-05 2024-09-21T11:06:19.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.722 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.722 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:19.722 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.700942+0000", 2024-09-21T11:06:19.722 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7630000000000006e-06 2024-09-21T11:06:19.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.722 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.722 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.722 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.591583+0000", 2024-09-21T11:06:19.723 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.89064102999999994 2024-09-21T11:06:19.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.723 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.723 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:19.723 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.591672+0000", 2024-09-21T11:06:19.723 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9815999999999995e-05 2024-09-21T11:06:19.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.723 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.723 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,7", 2024-09-21T11:06:19.723 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.591737+0000", 2024-09-21T11:06:19.723 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4308999999999997e-05 2024-09-21T11:06:19.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.723 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.723 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:19.723 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.593024+0000", 2024-09-21T11:06:19.724 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001286861 2024-09-21T11:06:19.724 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.724 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.724 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.724 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.596406+0000", 2024-09-21T11:06:19.724 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0033820740000000001 2024-09-21T11:06:19.724 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.724 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.724 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.724 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.609672+0000", 2024-09-21T11:06:19.724 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.013266758999999999 2024-09-21T11:06:19.724 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.724 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.724 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:19.724 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.609723+0000", 2024-09-21T11:06:19.725 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0235000000000003e-05 2024-09-21T11:06:19.725 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.725 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.725 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:19.725 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.609761+0000", 2024-09-21T11:06:19.725 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8284000000000001e-05 2024-09-21T11:06:19.725 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.725 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:19.725 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.725 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.725 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.725 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:285431 58.a 58:5e2ee56f:::benchmark_data_smithi070_396992_object139472:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1645)", 2024-09-21T11:06:19.725 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:56.734710+0000", 2024-09-21T11:06:19.725 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 22.906101457999998, 2024-09-21T11:06:19.725 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.87516325299999997, 2024-09-21T11:06:19.725 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:19.726 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:19.726 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:19.726 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:19.726 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:19.726 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:19.726 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 285431 2024-09-21T11:06:19.726 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.726 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:19.726 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.726 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:19.726 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.734710+0000", 2024-09-21T11:06:19.726 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.726 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.726 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.726 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:19.727 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.734710+0000", 2024-09-21T11:06:19.727 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.727 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.727 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.727 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:19.727 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.734712+0000", 2024-09-21T11:06:19.727 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.548e-06 2024-09-21T11:06:19.727 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.727 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.727 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:19.727 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.734717+0000", 2024-09-21T11:06:19.727 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7269999999999998e-06 2024-09-21T11:06:19.727 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.727 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.727 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:19.727 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.734731+0000", 2024-09-21T11:06:19.728 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4066000000000001e-05 2024-09-21T11:06:19.728 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.728 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.728 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:19.728 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.734738+0000", 2024-09-21T11:06:19.728 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1679999999999998e-06 2024-09-21T11:06:19.728 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.728 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.728 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.728 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.734791+0000", 2024-09-21T11:06:19.728 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3495000000000002e-05 2024-09-21T11:06:19.728 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.728 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.728 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:19.729 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.734798+0000", 2024-09-21T11:06:19.729 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0690000000000004e-06 2024-09-21T11:06:19.729 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.729 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.729 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.729 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.593182+0000", 2024-09-21T11:06:19.729 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.85838385100000003 2024-09-21T11:06:19.729 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.729 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.729 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:19.729 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.593282+0000", 2024-09-21T11:06:19.729 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9365999999999997e-05 2024-09-21T11:06:19.729 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.729 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.729 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,7", 2024-09-21T11:06:19.730 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.593400+0000", 2024-09-21T11:06:19.730 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000118267 2024-09-21T11:06:19.730 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.730 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.730 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:19.730 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.596322+0000", 2024-09-21T11:06:19.730 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002921875 2024-09-21T11:06:19.730 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.730 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.730 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.730 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.599710+0000", 2024-09-21T11:06:19.730 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0033886900000000002 2024-09-21T11:06:19.730 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.730 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.730 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.731 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.609796+0000", 2024-09-21T11:06:19.731 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.01008509 2024-09-21T11:06:19.731 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.731 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.731 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:19.731 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.609839+0000", 2024-09-21T11:06:19.731 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3438e-05 2024-09-21T11:06:19.731 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.731 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.731 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:19.731 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.609874+0000", 2024-09-21T11:06:19.731 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4603000000000001e-05 2024-09-21T11:06:19.731 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.732 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:19.732 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.732 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.732 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.732 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:285436 58.a 58:539ee972:::benchmark_data_smithi070_396992_object139477:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1645)", 2024-09-21T11:06:19.732 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:56.740040+0000", 2024-09-21T11:06:19.732 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 22.900772107000002, 2024-09-21T11:06:19.732 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.86996446500000002, 2024-09-21T11:06:19.732 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:19.732 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:19.732 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:19.732 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:19.732 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:19.732 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:19.733 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 285436 2024-09-21T11:06:19.733 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.733 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:19.733 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.733 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:19.733 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.740040+0000", 2024-09-21T11:06:19.733 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.733 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.733 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.733 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:19.733 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.740040+0000", 2024-09-21T11:06:19.733 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.733 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.733 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.733 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:19.734 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.740041+0000", 2024-09-21T11:06:19.734 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3939999999999999e-06 2024-09-21T11:06:19.734 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.734 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.734 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:19.734 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.740045+0000", 2024-09-21T11:06:19.734 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6789999999999998e-06 2024-09-21T11:06:19.734 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.734 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.734 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:19.734 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.740056+0000", 2024-09-21T11:06:19.734 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1035e-05 2024-09-21T11:06:19.734 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.734 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.735 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:19.735 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.740063+0000", 2024-09-21T11:06:19.735 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7750000000000004e-06 2024-09-21T11:06:19.735 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.735 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.735 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.735 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.740120+0000", 2024-09-21T11:06:19.735 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7426000000000002e-05 2024-09-21T11:06:19.735 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.735 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.735 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:19.735 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.740129+0000", 2024-09-21T11:06:19.735 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2019999999999993e-06 2024-09-21T11:06:19.735 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.736 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.736 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.736 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.594487+0000", 2024-09-21T11:06:19.736 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.85435741399999998 2024-09-21T11:06:19.736 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.736 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.736 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:19.736 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.594588+0000", 2024-09-21T11:06:19.736 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010129299999999999 2024-09-21T11:06:19.736 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.736 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.736 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,7", 2024-09-21T11:06:19.736 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.594693+0000", 2024-09-21T11:06:19.736 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000105102 2024-09-21T11:06:19.736 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.737 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.737 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:19.737 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.596327+0000", 2024-09-21T11:06:19.737 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0016339499999999999 2024-09-21T11:06:19.737 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.737 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.737 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.737 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.599756+0000", 2024-09-21T11:06:19.737 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0034286059999999998 2024-09-21T11:06:19.737 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.737 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.737 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.737 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.609908+0000", 2024-09-21T11:06:19.738 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.010152715 2024-09-21T11:06:19.738 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.738 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.738 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:19.738 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.609962+0000", 2024-09-21T11:06:19.738 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3711999999999997e-05 2024-09-21T11:06:19.738 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.738 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.738 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:19.738 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.610004+0000", 2024-09-21T11:06:19.738 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2162000000000001e-05 2024-09-21T11:06:19.738 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.738 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:19.738 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.739 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.739 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.739 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:285444 58.a 58:54c40fa2:::benchmark_data_smithi070_396992_object139485:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1645)", 2024-09-21T11:06:19.739 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:56.749858+0000", 2024-09-21T11:06:19.739 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 22.890954016999999, 2024-09-21T11:06:19.739 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.86025808800000003, 2024-09-21T11:06:19.739 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:19.739 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:19.739 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:19.739 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:19.739 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:19.739 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:19.739 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 285444 2024-09-21T11:06:19.739 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.740 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:19.740 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.740 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:19.740 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.749858+0000", 2024-09-21T11:06:19.740 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.740 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.740 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.740 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:19.740 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.749858+0000", 2024-09-21T11:06:19.740 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.740 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.740 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.740 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:19.740 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.749859+0000", 2024-09-21T11:06:19.741 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4589999999999999e-06 2024-09-21T11:06:19.741 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.741 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.741 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:19.741 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.749864+0000", 2024-09-21T11:06:19.741 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9989999999999999e-06 2024-09-21T11:06:19.741 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.741 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.741 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:19.741 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.749878+0000", 2024-09-21T11:06:19.741 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3692999999999999e-05 2024-09-21T11:06:19.741 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.741 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.741 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:19.742 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.749886+0000", 2024-09-21T11:06:19.742 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.627e-06 2024-09-21T11:06:19.742 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.742 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.742 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.742 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.749941+0000", 2024-09-21T11:06:19.742 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5495000000000003e-05 2024-09-21T11:06:19.742 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.742 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.742 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:19.742 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.749949+0000", 2024-09-21T11:06:19.742 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.4400000000000005e-06 2024-09-21T11:06:19.743 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.743 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.743 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.743 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.596434+0000", 2024-09-21T11:06:19.743 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.84648495700000004 2024-09-21T11:06:19.743 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.743 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.743 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:19.743 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.596526+0000", 2024-09-21T11:06:19.743 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1100000000000005e-05 2024-09-21T11:06:19.743 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.743 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.743 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,7", 2024-09-21T11:06:19.744 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.596627+0000", 2024-09-21T11:06:19.744 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010130500000000001 2024-09-21T11:06:19.744 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.744 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.744 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:19.744 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.599722+0000", 2024-09-21T11:06:19.744 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0030954770000000001 2024-09-21T11:06:19.744 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.744 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.744 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.744 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.603978+0000", 2024-09-21T11:06:19.744 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0042557070000000001 2024-09-21T11:06:19.744 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.744 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.745 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.745 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.610040+0000", 2024-09-21T11:06:19.745 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0060618959999999998 2024-09-21T11:06:19.745 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.745 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.745 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:19.745 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.610082+0000", 2024-09-21T11:06:19.745 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.1627e-05 2024-09-21T11:06:19.745 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.745 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.745 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:19.745 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.610116+0000", 2024-09-21T11:06:19.745 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4306e-05 2024-09-21T11:06:19.745 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.745 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:19.746 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.746 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.746 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.746 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:285482 58.a 58:55f9b13f:::benchmark_data_smithi070_396992_object139523:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1645)", 2024-09-21T11:06:19.746 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:56.848841+0000", 2024-09-21T11:06:19.746 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 22.791970847000002, 2024-09-21T11:06:19.746 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.76387068199999997, 2024-09-21T11:06:19.746 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:19.746 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:19.746 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:19.746 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:19.746 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:19.746 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:19.746 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 285482 2024-09-21T11:06:19.747 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.747 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:19.747 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.747 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:19.747 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.848841+0000", 2024-09-21T11:06:19.747 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.747 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.747 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.747 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:19.747 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.848841+0000", 2024-09-21T11:06:19.747 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.747 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.747 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.747 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:19.748 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.848842+0000", 2024-09-21T11:06:19.748 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.528e-06 2024-09-21T11:06:19.748 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.748 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.748 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:19.748 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.848847+0000", 2024-09-21T11:06:19.748 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5519999999999998e-06 2024-09-21T11:06:19.748 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.748 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.748 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:19.748 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.848861+0000", 2024-09-21T11:06:19.748 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3529999999999999e-05 2024-09-21T11:06:19.748 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.748 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.748 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:19.749 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.848868+0000", 2024-09-21T11:06:19.749 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6809999999999998e-06 2024-09-21T11:06:19.749 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.749 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.749 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.749 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.848915+0000", 2024-09-21T11:06:19.749 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6690000000000002e-05 2024-09-21T11:06:19.749 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.749 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.749 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:19.749 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:56.848922+0000", 2024-09-21T11:06:19.749 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.024e-06 2024-09-21T11:06:19.749 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.749 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.750 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.750 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.598100+0000", 2024-09-21T11:06:19.750 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.74917837200000004 2024-09-21T11:06:19.750 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.750 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.750 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:19.750 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.598204+0000", 2024-09-21T11:06:19.750 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000104157 2024-09-21T11:06:19.750 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.750 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.750 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,7", 2024-09-21T11:06:19.750 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.598317+0000", 2024-09-21T11:06:19.750 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000112481 2024-09-21T11:06:19.751 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.751 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.751 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:19.751 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.600081+0000", 2024-09-21T11:06:19.751 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0017642809999999999 2024-09-21T11:06:19.751 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.751 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.751 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.751 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.604013+0000", 2024-09-21T11:06:19.751 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0039314320000000003 2024-09-21T11:06:19.751 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.751 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.751 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.751 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.612630+0000", 2024-09-21T11:06:19.752 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0086175130000000003 2024-09-21T11:06:19.752 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.752 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.752 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:19.752 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.612677+0000", 2024-09-21T11:06:19.752 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6876999999999999e-05 2024-09-21T11:06:19.752 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.752 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.752 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:19.752 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:57.612712+0000", 2024-09-21T11:06:19.752 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4564000000000003e-05 2024-09-21T11:06:19.752 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.752 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:19.752 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.752 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.753 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.753 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.36567.0:286578 58.c e1646/1643 58:33f71f3b:::benchmark_data_smithi070_396992_object140619:head v 1646'17993, mlcod=1646'17991)", 2024-09-21T11:06:19.753 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:58.438087+0000", 2024-09-21T11:06:19.753 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 21.202725188999999, 2024-09-21T11:06:19.753 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.043541133000000003, 2024-09-21T11:06:19.753 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:19.753 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:19.753 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:19.753 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:19.753 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.753 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:19.753 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.438087+0000", 2024-09-21T11:06:19.753 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.753 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.754 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.754 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:19.754 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.438087+0000", 2024-09-21T11:06:19.754 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.754 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.754 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.754 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:19.754 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.438087+0000", 2024-09-21T11:06:19.754 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1699999999999997e-07 2024-09-21T11:06:19.754 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.754 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.754 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:19.754 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.438092+0000", 2024-09-21T11:06:19.755 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6439999999999999e-06 2024-09-21T11:06:19.755 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.755 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.755 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:19.755 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.438101+0000", 2024-09-21T11:06:19.755 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0350000000000007e-06 2024-09-21T11:06:19.755 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.755 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.755 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:19.755 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.438107+0000", 2024-09-21T11:06:19.755 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2960000000000004e-06 2024-09-21T11:06:19.755 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.755 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.755 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.756 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.438174+0000", 2024-09-21T11:06:19.756 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7032000000000005e-05 2024-09-21T11:06:19.756 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.756 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.756 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:19.756 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.438192+0000", 2024-09-21T11:06:19.756 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7679000000000001e-05 2024-09-21T11:06:19.756 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.756 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.756 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:19.756 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.481543+0000", 2024-09-21T11:06:19.756 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.043351318 2024-09-21T11:06:19.756 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.756 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.757 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:19.757 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.481628+0000", 2024-09-21T11:06:19.757 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.4412000000000004e-05 2024-09-21T11:06:19.757 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.757 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:19.757 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.757 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.757 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.757 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:286580 58.a 58:5efcc6d6:::benchmark_data_smithi070_396992_object140621:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1646)", 2024-09-21T11:06:19.757 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:58.438502+0000", 2024-09-21T11:06:19.757 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 21.202309979999999, 2024-09-21T11:06:19.757 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.057715071999999999, 2024-09-21T11:06:19.757 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:19.757 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:19.758 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:19.758 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:19.758 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:19.758 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:19.758 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 286580 2024-09-21T11:06:19.758 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.758 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:19.758 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.758 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:19.758 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.438502+0000", 2024-09-21T11:06:19.758 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.758 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.758 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.758 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:19.759 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.438502+0000", 2024-09-21T11:06:19.759 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.759 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.759 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.759 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:19.759 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.438504+0000", 2024-09-21T11:06:19.759 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9970000000000001e-06 2024-09-21T11:06:19.759 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.759 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.759 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:19.759 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.438508+0000", 2024-09-21T11:06:19.759 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.1620000000000001e-06 2024-09-21T11:06:19.759 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.760 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.760 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:19.760 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.438519+0000", 2024-09-21T11:06:19.760 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0859999999999999e-05 2024-09-21T11:06:19.760 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.760 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.760 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:19.760 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.438525+0000", 2024-09-21T11:06:19.760 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6479999999999997e-06 2024-09-21T11:06:19.760 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.760 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.760 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.760 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.439319+0000", 2024-09-21T11:06:19.761 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00079310999999999997 2024-09-21T11:06:19.761 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.761 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.761 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:19.761 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.439675+0000", 2024-09-21T11:06:19.761 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00035597 2024-09-21T11:06:19.761 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.762 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.762 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,7", 2024-09-21T11:06:19.762 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.439975+0000", 2024-09-21T11:06:19.762 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00030006500000000002 2024-09-21T11:06:19.762 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.762 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.762 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.762 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.445548+0000", 2024-09-21T11:06:19.762 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0055729600000000001 2024-09-21T11:06:19.762 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.762 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.763 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:19.763 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.445556+0000", 2024-09-21T11:06:19.763 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.8000000000000004e-06 2024-09-21T11:06:19.763 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.763 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.763 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.763 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.495832+0000", 2024-09-21T11:06:19.763 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.050275713999999999 2024-09-21T11:06:19.763 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.763 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.763 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:19.763 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.496019+0000", 2024-09-21T11:06:19.764 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018691500000000001 2024-09-21T11:06:19.764 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.764 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.764 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:19.764 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.496217+0000", 2024-09-21T11:06:19.764 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00019787099999999999 2024-09-21T11:06:19.764 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.764 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:19.764 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.764 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.764 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.764 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:286581 58.a 58:55487840:::benchmark_data_smithi070_396992_object140622:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1646)", 2024-09-21T11:06:19.764 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:58.438543+0000", 2024-09-21T11:06:19.764 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 21.202268932999999, 2024-09-21T11:06:19.764 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0577866, 2024-09-21T11:06:19.765 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:19.765 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:19.765 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:19.765 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:19.765 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:19.765 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:19.765 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 286581 2024-09-21T11:06:19.765 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.765 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:19.765 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.765 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:19.765 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.438543+0000", 2024-09-21T11:06:19.765 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.765 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.766 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.766 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:19.766 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.438543+0000", 2024-09-21T11:06:19.766 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.766 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.766 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.766 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:19.766 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.438543+0000", 2024-09-21T11:06:19.766 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.32e-07 2024-09-21T11:06:19.766 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.766 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.766 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:19.766 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.438545+0000", 2024-09-21T11:06:19.767 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8309999999999999e-06 2024-09-21T11:06:19.767 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.767 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.767 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:19.767 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.438551+0000", 2024-09-21T11:06:19.767 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7590000000000003e-06 2024-09-21T11:06:19.767 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.767 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.767 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:19.767 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.438555+0000", 2024-09-21T11:06:19.767 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.4220000000000002e-06 2024-09-21T11:06:19.767 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.768 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.768 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.768 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.442900+0000", 2024-09-21T11:06:19.768 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0043444549999999997 2024-09-21T11:06:19.768 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.768 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.768 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:19.768 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.443003+0000", 2024-09-21T11:06:19.768 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000103077 2024-09-21T11:06:19.768 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.768 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.768 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,7", 2024-09-21T11:06:19.768 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.443104+0000", 2024-09-21T11:06:19.768 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000101396 2024-09-21T11:06:19.769 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.769 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.769 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:19.769 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.484194+0000", 2024-09-21T11:06:19.769 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.041089674 2024-09-21T11:06:19.769 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.769 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.769 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.769 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.484238+0000", 2024-09-21T11:06:19.769 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.4107000000000003e-05 2024-09-21T11:06:19.769 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.769 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.769 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:19.769 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.496255+0000", 2024-09-21T11:06:19.770 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012017392 2024-09-21T11:06:19.770 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.770 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.770 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:19.770 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.496299+0000", 2024-09-21T11:06:19.770 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3139000000000002e-05 2024-09-21T11:06:19.770 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.770 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.770 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:19.770 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.496329+0000", 2024-09-21T11:06:19.770 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0916e-05 2024-09-21T11:06:19.770 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.770 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:19.770 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.771 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.771 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.771 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.36567.0:286589 58.2 e1646/1640 58:41a0cb49:::benchmark_data_smithi070_396992_object140630:head v 1646'17973, mlcod=1646'17971)", 2024-09-21T11:06:19.771 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:58.444003+0000", 2024-09-21T11:06:19.771 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 21.196808970999999, 2024-09-21T11:06:19.771 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.043155368, 2024-09-21T11:06:19.771 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:19.771 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:19.771 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:19.771 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:19.771 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.771 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:19.771 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.444003+0000", 2024-09-21T11:06:19.771 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.772 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.772 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.772 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:19.772 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.444003+0000", 2024-09-21T11:06:19.772 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:19.772 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.772 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.772 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:19.772 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.444003+0000", 2024-09-21T11:06:19.772 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9999999999999998e-07 2024-09-21T11:06:19.772 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.772 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.772 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:19.772 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.444007+0000", 2024-09-21T11:06:19.772 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.827e-06 2024-09-21T11:06:19.773 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.773 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.773 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:19.773 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.444014+0000", 2024-09-21T11:06:19.773 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7700000000000004e-06 2024-09-21T11:06:19.773 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.773 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.773 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:19.773 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.444018+0000", 2024-09-21T11:06:19.773 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0820000000000001e-06 2024-09-21T11:06:19.773 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.773 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.774 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:19.774 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.445532+0000", 2024-09-21T11:06:19.774 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001514449 2024-09-21T11:06:19.774 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.774 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.774 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:19.774 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.445549+0000", 2024-09-21T11:06:19.774 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6578e-05 2024-09-21T11:06:19.774 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.774 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.774 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:19.774 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.487124+0000", 2024-09-21T11:06:19.774 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.041575286000000003 2024-09-21T11:06:19.774 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:19.775 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:19.775 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:19.775 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:58.487158+0000", 2024-09-21T11:06:19.775 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3875999999999999e-05 2024-09-21T11:06:19.775 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.775 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:19.775 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.775 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:19.775 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:19.775 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T11:06:19.801 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:19.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:19.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:19.870 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:19 smithi070 ceph-mon[107385]: pgmap v3276: 109 pgs: 2 active+undersized+remapped, 107 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:19.870 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:19 smithi070 ceph-mon[112139]: pgmap v3276: 109 pgs: 2 active+undersized+remapped, 107 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:19.902 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:19.928 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:19.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:19.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:20.030 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:20.051 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:20.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:20.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:20.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:19 smithi160 ceph-mon[101879]: pgmap v3276: 109 pgs: 2 active+undersized+remapped, 107 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:20.153 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:20.173 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:20.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:20.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:20.275 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:20.295 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:20.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:20.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:20.397 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:20.420 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:20.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:20.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:20.496 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7, 4, 0] out_osds: [2, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 7, 6, 5] 2024-09-21T11:06:20.497 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T11:06:20.521 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:20.560 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:20.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:20.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:20.662 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:20.686 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:20.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:20.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:20.788 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:20.811 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:20.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:20.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:20.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:20 smithi070 ceph-mon[107385]: osdmap e1654: 8 total, 8 up, 6 in 2024-09-21T11:06:20.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:06:20.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:06:20.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:20 smithi070 ceph-mon[107385]: osdmap e1655: 8 total, 8 up, 6 in 2024-09-21T11:06:20.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:20 smithi070 ceph-mon[112139]: osdmap e1654: 8 total, 8 up, 6 in 2024-09-21T11:06:20.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:06:20.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:06:20.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:20 smithi070 ceph-mon[112139]: osdmap e1655: 8 total, 8 up, 6 in 2024-09-21T11:06:20.912 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:20.935 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:20.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:20.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:21.036 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:21.057 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:21.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:21.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:21.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:20 smithi160 ceph-mon[101879]: osdmap e1654: 8 total, 8 up, 6 in 2024-09-21T11:06:21.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:06:21.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:06:21.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:20 smithi160 ceph-mon[101879]: osdmap e1655: 8 total, 8 up, 6 in 2024-09-21T11:06:21.160 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:21.181 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:21.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:21.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:21.274 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:06:21.283 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:21.307 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:21.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:21.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:21.409 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:21.430 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:21.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:21.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:21.532 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:21.555 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:21.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:21.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:21.656 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:21.678 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:21.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:21.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:21.780 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:21.801 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:21.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:21.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:21.903 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:21.924 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:21.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:21.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:22.026 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:22.047 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:22.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:22.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:22.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:21 smithi160 ceph-mon[101879]: pgmap v3278: 109 pgs: 1 peering, 1 active+undersized+remapped, 107 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:22.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:21 smithi070 ceph-mon[112139]: pgmap v3278: 109 pgs: 1 peering, 1 active+undersized+remapped, 107 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:22.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:21 smithi070 ceph-mon[107385]: pgmap v3278: 109 pgs: 1 peering, 1 active+undersized+remapped, 107 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:22.149 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:22.170 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:22.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:22.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:22.271 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:22.273 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T11:06:22.294 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:22.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:22.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:22.395 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:22.417 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:22.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:22.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:22.519 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:22.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:22.543 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:22.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:22.645 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:22.667 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:22.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:22.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:22.768 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:22.789 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:22.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:22.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:22.891 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:22.913 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:22.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:22.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:23.015 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:23.036 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:23.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:23.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:23.138 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:23.159 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:23.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:23.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:23.261 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:23.284 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:23.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:23.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:23.290 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T11:06:23.312 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:06:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:06:23] "GET /metrics HTTP/1.1" 200 36678 "" "Prometheus/2.51.0" 2024-09-21T11:06:23.385 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:23.406 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:23.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:23.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:23.508 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:23.530 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:23.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:23.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:23.631 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:23.645 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'unique_pool_5' removed 2024-09-21T11:06:23.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:23.662 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:23.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:23.763 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:23.785 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:23.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:23.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:23.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:23 smithi070 ceph-mon[112139]: pgmap v3280: 109 pgs: 1 peering, 108 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:23.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:23 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/852499488' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T11:06:23.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:23 smithi070 ceph-mon[107385]: pgmap v3280: 109 pgs: 1 peering, 108 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:23.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:23 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/852499488' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T11:06:23.887 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:23.908 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:23.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:23.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:24.010 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:24.032 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:24.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:24.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:24.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:23 smithi160 ceph-mon[101879]: pgmap v3280: 109 pgs: 1 peering, 108 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:24.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:23 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/852499488' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T11:06:24.135 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:24.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:24.157 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:24.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:24.258 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:24.280 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:24.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:24.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:24.382 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:24.403 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:24.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:24.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:24.504 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:24.528 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:24.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:24.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:24.630 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:24.654 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:24.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:24.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:24.755 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:24.777 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:24.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:24.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:24.828 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:24 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/852499488' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T11:06:24.828 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:24 smithi070 ceph-mon[107385]: osdmap e1656: 8 total, 8 up, 6 in 2024-09-21T11:06:24.828 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:24 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/852499488' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T11:06:24.829 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:24 smithi070 ceph-mon[112139]: osdmap e1656: 8 total, 8 up, 6 in 2024-09-21T11:06:24.850 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-21T11:06:24.850 INFO:teuthology.task.print:**** done end radosbench.yaml 2024-09-21T11:06:24.850 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-09-21T11:06:24.851 INFO:tasks.workunit:Pulling workunits from ref reef 2024-09-21T11:06:24.851 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-21T11:06:24.851 DEBUG:teuthology.orchestra.run.smithi070:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-21T11:06:24.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:24.878 INFO:teuthology.orchestra.run.smithi070.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-21T11:06:24.878 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-21T11:06:24.879 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:24.942 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:24.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:24.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:24.970 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-21T11:06:24.970 DEBUG:teuthology.orchestra.run.smithi070:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-21T11:06:25.025 INFO:tasks.workunit:timeout=3h 2024-09-21T11:06:25.026 INFO:tasks.workunit:cleanup=True 2024-09-21T11:06:25.026 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch reef https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2024-09-21T11:06:25.043 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:25.065 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:25.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:25.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:25.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:24 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/852499488' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T11:06:25.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:24 smithi160 ceph-mon[101879]: osdmap e1656: 8 total, 8 up, 6 in 2024-09-21T11:06:25.093 INFO:tasks.workunit.client.0.smithi070.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-21T11:06:25.168 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:25.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:25.189 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:25.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:25.290 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:25.313 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:25.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:25.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:25.415 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:25.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:25.438 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:25.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:25.539 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:25.560 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:25.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:25.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:25.662 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:25.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:25.684 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:25.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:25.770 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:25 smithi070 ceph-mon[107385]: pgmap v3282: 93 pgs: 93 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:25.770 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:25 smithi070 ceph-mon[112139]: pgmap v3282: 93 pgs: 93 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:25.784 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:25.807 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:25.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:25.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:25.909 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:25.935 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:25.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:25.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:25.992 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:25 smithi160 ceph-mon[101879]: pgmap v3282: 93 pgs: 93 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:26.037 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:26.062 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:26.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:26.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:26.163 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:26.184 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:26.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:26.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:26.286 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:26.307 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:26.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:26.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:26.409 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:26.431 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:26.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:26.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:26.533 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:26.554 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:26.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:26.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:26.656 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:26.677 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:26.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:26.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:26.757 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:06:26.757 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:06:26.757 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:06:26.779 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:26.810 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:26.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:26.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:26.912 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:26.933 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:26.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:26.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:27.035 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:27.056 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:27.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:27.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:27.157 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:27.179 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:27.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:27.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:27.281 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:27.303 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:27.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:27.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:27.405 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:27.425 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:27.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:27.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:27.527 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:27.549 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:27.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:27.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:27.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:27 smithi160 ceph-mon[101879]: pgmap v3283: 93 pgs: 93 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:27.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:27 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:06:27.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:27 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:06:27.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:27 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/757611221' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:06:27.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:27 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:06:27.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:27 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T11:06:27.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:27 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T11:06:27.592 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:27 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:06:27.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:27 smithi070 ceph-mon[112139]: pgmap v3283: 93 pgs: 93 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:27.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:27 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:06:27.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:27 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:06:27.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:27 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/757611221' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:06:27.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:27 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:06:27.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:27 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T11:06:27.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:27 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T11:06:27.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:27 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:06:27.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:27 smithi070 ceph-mon[107385]: pgmap v3283: 93 pgs: 93 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:27.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:27 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:06:27.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:27 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:06:27.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:27 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/757611221' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:06:27.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:27 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:06:27.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:27 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T11:06:27.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:27 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T11:06:27.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:27 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:06:27.651 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:27.656 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T11:06:27.675 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:27.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:27.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:27.777 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:27.798 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:27.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:27.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:27.900 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:27.922 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:27.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:27.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:28.024 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:28.046 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:28.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:28.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:28.148 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:28.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:28.169 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:28.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:28.270 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:28.293 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:28.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:28.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:28.395 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:28.416 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:28.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:28.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:28.517 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:28.540 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:28.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:28.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:28.641 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:28.664 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:28.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:28.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:28.766 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:28.788 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:28.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:28.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:28 smithi160 ceph-mon[101879]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:06:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:28 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T11:06:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:28 smithi160 ceph-mon[101879]: osdmap e1657: 8 total, 8 up, 6 in 2024-09-21T11:06:28.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:28 smithi070 ceph-mon[112139]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:06:28.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:28 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T11:06:28.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:28 smithi070 ceph-mon[112139]: osdmap e1657: 8 total, 8 up, 6 in 2024-09-21T11:06:28.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:28 smithi070 ceph-mon[107385]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T11:06:28.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:28 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T11:06:28.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:28 smithi070 ceph-mon[107385]: osdmap e1657: 8 total, 8 up, 6 in 2024-09-21T11:06:28.890 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:28.911 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:28.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:28.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:29.013 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:29.034 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:29.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:29.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:29.136 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:29.158 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:29.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:29.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:29.260 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:29.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:29.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:29.382 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:29.405 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:29.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:29.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:29.507 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:29.530 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:29.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:29.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:29.632 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:29.654 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:29.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:29.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:29.756 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:29.780 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:29.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:29.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:29.882 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:29.903 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:29.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:29.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:29.939 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:29 smithi070 ceph-mon[107385]: pgmap v3285: 93 pgs: 93 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:29.939 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:29 smithi070 ceph-mon[107385]: 1.12 scrub starts 2024-09-21T11:06:29.939 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:29 smithi070 ceph-mon[107385]: 1.12 scrub ok 2024-09-21T11:06:29.939 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:29 smithi070 ceph-mon[112139]: pgmap v3285: 93 pgs: 93 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:29.939 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:29 smithi070 ceph-mon[112139]: 1.12 scrub starts 2024-09-21T11:06:29.939 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:29 smithi070 ceph-mon[112139]: 1.12 scrub ok 2024-09-21T11:06:30.005 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:30.029 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:30.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:30.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:30.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:29 smithi160 ceph-mon[101879]: pgmap v3285: 93 pgs: 93 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:30.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:29 smithi160 ceph-mon[101879]: 1.12 scrub starts 2024-09-21T11:06:30.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:29 smithi160 ceph-mon[101879]: 1.12 scrub ok 2024-09-21T11:06:30.131 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:30.154 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:30.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:30.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:30.256 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:30.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:30.280 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:30.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:30.381 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:30.403 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:30.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:30.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:30.505 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:30.528 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:30.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:30.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:30.630 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:30.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:30.655 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:30.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:30.756 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:30.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:30.781 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:30.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:30.882 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:30.904 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:30.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:30.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:31.006 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:31.028 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:31.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:31.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:31.130 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:31.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:31.153 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:31.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:31.254 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:31.276 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:31.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:31.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:31.378 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:31.404 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:31.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:31.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:31.456 DEBUG:teuthology.orchestra.run.smithi070:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-21T11:06:31.500 INFO:tasks.workunit.client.0.smithi070.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-21T11:06:31.502 INFO:tasks.workunit.client.0.smithi070.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T11:06:31.502 INFO:tasks.workunit.client.0.smithi070.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-21T11:06:31.506 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:31.527 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:31.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:31.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:31.629 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:31.651 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:31.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:31.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:31.743 INFO:tasks.workunit.client.0.smithi070.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-21T11:06:31.752 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:31.774 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:31.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:31.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:31.794 INFO:tasks.workunit.client.0.smithi070.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-21T11:06:31.835 INFO:tasks.workunit.client.0.smithi070.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T11:06:31.837 INFO:tasks.workunit.client.0.smithi070.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T11:06:31.837 INFO:tasks.workunit.client.0.smithi070.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-21T11:06:31.876 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:31 smithi070 ceph-mon[107385]: pgmap v3286: 93 pgs: 93 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:31.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:31 smithi070 ceph-mon[112139]: pgmap v3286: 93 pgs: 93 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:31.878 INFO:tasks.workunit.client.0.smithi070.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T11:06:31.898 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:31.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:31.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:31.903 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T11:06:31.904 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-21T11:06:31.981 INFO:tasks.workunit:Running workunits matching cls/test_cls_rbd.sh on client.0... 2024-09-21T11:06:31.982 INFO:tasks.workunit:Running workunit cls/test_cls_rbd.sh... 2024-09-21T11:06:31.982 DEBUG:teuthology.orchestra.run.smithi070:workunit test cls/test_cls_rbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh 2024-09-21T11:06:32.000 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:32.026 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:32.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:32.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:32.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:31 smithi160 ceph-mon[101879]: pgmap v3286: 93 pgs: 93 active+clean; 430 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:32.128 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:32.150 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:32.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:32.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:32.252 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:32.273 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:32.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:32.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:32.347 INFO:tasks.workunit.client.0.smithi070.stdout:Running main() from gmock_main.cc 2024-09-21T11:06:32.347 INFO:tasks.workunit.client.0.smithi070.stdout:[==========] Running 70 tests from 1 test suite. 2024-09-21T11:06:32.348 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] Global test environment set-up. 2024-09-21T11:06:32.348 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 70 tests from TestClsRbd 2024-09-21T11:06:32.375 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:32.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:32.404 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:32.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:32.505 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:32.530 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:32.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:32.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:32.632 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:32.657 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:32.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:32.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:32.664 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T11:06:32.758 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:32.780 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:32.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:32.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:32.882 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:32.903 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:32.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:32.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:33.005 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:33.011 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:32 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3191768746' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T11:06:33.012 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:32 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T11:06:33.012 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:32 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3191768746' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T11:06:33.012 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:32 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T11:06:33.026 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:33.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:33.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:33.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:32 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3191768746' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T11:06:33.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:32 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T11:06:33.128 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:33.149 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:33.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:33.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:33.251 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:33.277 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:33.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:33.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:33.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:06:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:06:33] "GET /metrics HTTP/1.1" 200 36678 "" "Prometheus/2.51.0" 2024-09-21T11:06:33.379 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:33.401 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:33.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:33.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:33.503 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:33.519 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.get_all_features 2024-09-21T11:06:33.538 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:33.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:33.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:33.547 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.get_all_features (28 ms) 2024-09-21T11:06:33.547 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.copyup 2024-09-21T11:06:33.639 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:33.661 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:33.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:33.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:33.717 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.copyup (170 ms) 2024-09-21T11:06:33.717 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.sparse_copyup 2024-09-21T11:06:33.763 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:33.765 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.sparse_copyup (48 ms) 2024-09-21T11:06:33.766 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.get_and_set_id 2024-09-21T11:06:33.773 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.get_and_set_id (8 ms) 2024-09-21T11:06:33.773 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.add_remove_child 2024-09-21T11:06:33.786 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.add_remove_child (13 ms) 2024-09-21T11:06:33.786 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.directory_methods 2024-09-21T11:06:33.787 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:33.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:33.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:33.813 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.directory_methods (27 ms) 2024-09-21T11:06:33.814 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.create 2024-09-21T11:06:33.827 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.create (14 ms) 2024-09-21T11:06:33.827 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.get_features 2024-09-21T11:06:33.829 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.get_features (2 ms) 2024-09-21T11:06:33.829 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.get_object_prefix 2024-09-21T11:06:33.831 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.get_object_prefix (2 ms) 2024-09-21T11:06:33.831 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.get_create_timestamp 2024-09-21T11:06:33.833 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.get_create_timestamp (2 ms) 2024-09-21T11:06:33.833 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.get_access_timestamp 2024-09-21T11:06:33.834 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.get_access_timestamp (1 ms) 2024-09-21T11:06:33.834 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.get_modify_timestamp 2024-09-21T11:06:33.835 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.get_modify_timestamp (2 ms) 2024-09-21T11:06:33.836 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.get_data_pool 2024-09-21T11:06:33.839 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.get_data_pool (3 ms) 2024-09-21T11:06:33.839 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.get_size 2024-09-21T11:06:33.842 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.get_size (4 ms) 2024-09-21T11:06:33.843 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.set_size 2024-09-21T11:06:33.847 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.set_size (4 ms) 2024-09-21T11:06:33.847 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.protection_status 2024-09-21T11:06:33.864 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.protection_status (17 ms) 2024-09-21T11:06:33.864 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.snapshot_limits 2024-09-21T11:06:33.871 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.snapshot_limits (7 ms) 2024-09-21T11:06:33.871 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.parents_v1 2024-09-21T11:06:33.888 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:33.890 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:06:33.891 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:06:33.891 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:33 smithi070 ceph-mon[107385]: pgmap v3287: 93 pgs: 93 active+clean; 404 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:33.892 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:33 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T11:06:33.892 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:33 smithi070 ceph-mon[107385]: osdmap e1658: 8 total, 8 up, 6 in 2024-09-21T11:06:33.892 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:33 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1238721849' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi070-431558-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T11:06:33.892 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:33 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1238721849' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi070-431558-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T11:06:33.892 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:33 smithi070 ceph-mon[107385]: osdmap e1659: 8 total, 8 up, 6 in 2024-09-21T11:06:33.892 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:33 smithi070 ceph-mon[112139]: pgmap v3287: 93 pgs: 93 active+clean; 404 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:33.892 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:33 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T11:06:33.892 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:33 smithi070 ceph-mon[112139]: osdmap e1658: 8 total, 8 up, 6 in 2024-09-21T11:06:33.892 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:33 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1238721849' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi070-431558-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T11:06:33.892 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:33 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1238721849' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi070-431558-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T11:06:33.892 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:33 smithi070 ceph-mon[112139]: osdmap e1659: 8 total, 8 up, 6 in 2024-09-21T11:06:33.894 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:06:33.894 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:06:33.910 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:33.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:33.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:33.913 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.parents_v1 (41 ms) 2024-09-21T11:06:33.913 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.parents_v2 2024-09-21T11:06:33.914 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:06:33.914 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:06:33.932 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:06:33.932 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:06:33.941 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.parents_v2 (29 ms) 2024-09-21T11:06:33.941 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.snapshots 2024-09-21T11:06:33.949 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:06:33.949 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:06:33.963 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.snapshots (22 ms) 2024-09-21T11:06:33.964 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.snapid_race 2024-09-21T11:06:33.967 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.snapid_race (4 ms) 2024-09-21T11:06:33.967 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.stripingv2 2024-09-21T11:06:33.968 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T11:06:33.968 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T11:06:33.975 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.stripingv2 (8 ms) 2024-09-21T11:06:33.976 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.object_map_save 2024-09-21T11:06:33.978 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.object_map_save (3 ms) 2024-09-21T11:06:33.978 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.object_map_resize 2024-09-21T11:06:33.983 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.object_map_resize (5 ms) 2024-09-21T11:06:33.983 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.object_map_update 2024-09-21T11:06:33.985 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T11:06:33.985 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T11:06:33.988 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.object_map_update (5 ms) 2024-09-21T11:06:33.988 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.object_map_load_enoent 2024-09-21T11:06:33.988 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.object_map_load_enoent (0 ms) 2024-09-21T11:06:33.988 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.object_map_snap_add 2024-09-21T11:06:33.992 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.object_map_snap_add (5 ms) 2024-09-21T11:06:33.993 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.object_map_snap_remove 2024-09-21T11:06:33.996 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.object_map_snap_remove (3 ms) 2024-09-21T11:06:33.996 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.flags 2024-09-21T11:06:34.002 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.flags (6 ms) 2024-09-21T11:06:34.002 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.metadata 2024-09-21T11:06:34.004 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:06:34.004 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:06:34.011 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:34.035 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:34.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:34.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:34.036 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.metadata (33 ms) 2024-09-21T11:06:34.036 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.set_features 2024-09-21T11:06:34.041 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.set_features (7 ms) 2024-09-21T11:06:34.042 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.mirror 2024-09-21T11:06:34.074 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.mirror (31 ms) 2024-09-21T11:06:34.074 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.mirror_image 2024-09-21T11:06:34.088 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.mirror_image (15 ms) 2024-09-21T11:06:34.088 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.mirror_image_status 2024-09-21T11:06:34.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:33 smithi160 ceph-mon[101879]: pgmap v3287: 93 pgs: 93 active+clean; 404 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:34.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:33 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T11:06:34.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:33 smithi160 ceph-mon[101879]: osdmap e1658: 8 total, 8 up, 6 in 2024-09-21T11:06:34.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:33 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1238721849' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi070-431558-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T11:06:34.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:33 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1238721849' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi070-431558-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T11:06:34.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:33 smithi160 ceph-mon[101879]: osdmap e1659: 8 total, 8 up, 6 in 2024-09-21T11:06:34.114 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T11:06:34.114 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T11:06:34.136 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:34.158 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:34.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:34.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:34.260 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:34.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:34.284 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:34.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:34.385 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:34.407 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:34.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:34.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:34.509 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:34.536 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:34.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:34.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:34.553 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T11:06:34.638 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:34.660 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:34.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:34.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:34.762 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:34.784 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:34.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:34.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:34.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:34 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2309549027' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:06:34.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:34 smithi070 ceph-mon[107385]: osdmap e1660: 8 total, 8 up, 6 in 2024-09-21T11:06:34.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:34 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2309549027' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:06:34.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:34 smithi070 ceph-mon[112139]: osdmap e1660: 8 total, 8 up, 6 in 2024-09-21T11:06:34.886 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:34.909 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:34.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:34.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:35.011 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:35.033 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:35.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:35.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:35.045 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:06:35.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:34 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2309549027' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:06:35.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:34 smithi160 ceph-mon[101879]: osdmap e1660: 8 total, 8 up, 6 in 2024-09-21T11:06:35.136 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:35.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:35.166 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:35.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:35.268 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:35.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:35.292 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:35.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:35.393 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:35.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:35.417 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:35.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:35.518 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:35.543 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:35.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:35.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:35.646 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:35.660 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T11:06:35.678 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:35.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:35.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:35.779 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:35.808 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:35.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:35.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:35.821 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:35 smithi070 ceph-mon[112139]: pgmap v3290: 125 pgs: 32 unknown, 93 active+clean; 404 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:35.821 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:06:35.821 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:35 smithi070 ceph-mon[112139]: osdmap e1661: 8 total, 8 up, 6 in 2024-09-21T11:06:35.821 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:35 smithi070 ceph-mon[107385]: pgmap v3290: 125 pgs: 32 unknown, 93 active+clean; 404 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:35.821 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:06:35.822 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:35 smithi070 ceph-mon[107385]: osdmap e1661: 8 total, 8 up, 6 in 2024-09-21T11:06:35.847 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T11:06:35.910 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:35.934 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:35.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:35.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:36.036 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:36.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:36.060 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:36.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:35 smithi160 ceph-mon[101879]: pgmap v3290: 125 pgs: 32 unknown, 93 active+clean; 404 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:06:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:35 smithi160 ceph-mon[101879]: osdmap e1661: 8 total, 8 up, 6 in 2024-09-21T11:06:36.161 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:36.182 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:36.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:36.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:36.284 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:36.306 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:36.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:36.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:36.359 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.mirror_image_status (2271 ms) 2024-09-21T11:06:36.359 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.mirror_image_map 2024-09-21T11:06:36.407 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:36.429 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:36.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:36.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:36.442 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.mirror_image_map (83 ms) 2024-09-21T11:06:36.442 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.mirror_instances 2024-09-21T11:06:36.449 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.mirror_instances (7 ms) 2024-09-21T11:06:36.449 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.mirror_snapshot 2024-09-21T11:06:36.460 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.mirror_snapshot (12 ms) 2024-09-21T11:06:36.461 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.group_dir_list 2024-09-21T11:06:36.463 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.group_dir_list (2 ms) 2024-09-21T11:06:36.463 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.group_dir_add 2024-09-21T11:06:36.465 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.group_dir_add (2 ms) 2024-09-21T11:06:36.465 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.dir_add_already_existing 2024-09-21T11:06:36.467 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.dir_add_already_existing (2 ms) 2024-09-21T11:06:36.467 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.group_dir_rename 2024-09-21T11:06:36.472 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.group_dir_rename (5 ms) 2024-09-21T11:06:36.472 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.group_dir_remove 2024-09-21T11:06:36.475 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.group_dir_remove (3 ms) 2024-09-21T11:06:36.475 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.group_dir_remove_missing 2024-09-21T11:06:36.479 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.group_dir_remove_missing (4 ms) 2024-09-21T11:06:36.479 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.group_image_add 2024-09-21T11:06:36.482 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.group_image_add (3 ms) 2024-09-21T11:06:36.482 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.group_image_remove 2024-09-21T11:06:36.485 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.group_image_remove (3 ms) 2024-09-21T11:06:36.485 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.group_image_list 2024-09-21T11:06:36.487 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.group_image_list (2 ms) 2024-09-21T11:06:36.487 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.group_image_clean 2024-09-21T11:06:36.491 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.group_image_clean (4 ms) 2024-09-21T11:06:36.491 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.image_group_add 2024-09-21T11:06:36.493 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.image_group_add (2 ms) 2024-09-21T11:06:36.493 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.image_group_remove 2024-09-21T11:06:36.496 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.image_group_remove (3 ms) 2024-09-21T11:06:36.496 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.image_group_get 2024-09-21T11:06:36.498 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.image_group_get (2 ms) 2024-09-21T11:06:36.498 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_name 2024-09-21T11:06:36.500 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.group_snap_set_empty_name (2 ms) 2024-09-21T11:06:36.500 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_id 2024-09-21T11:06:36.502 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.group_snap_set_empty_id (2 ms) 2024-09-21T11:06:36.502 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_id 2024-09-21T11:06:36.504 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_id (2 ms) 2024-09-21T11:06:36.504 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_name 2024-09-21T11:06:36.506 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_name (3 ms) 2024-09-21T11:06:36.507 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.group_snap_set 2024-09-21T11:06:36.509 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-871-g6a1bdf85/rpm/el9/BUILD/ceph-18.2.4-871-g6a1bdf85/src/test/cls_rbd/test_cls_rbd.cc:2681: Failure 2024-09-21T11:06:36.509 INFO:tasks.workunit.client.0.smithi070.stdout:Expected equality of these values: 2024-09-21T11:06:36.509 INFO:tasks.workunit.client.0.smithi070.stdout: 1U 2024-09-21T11:06:36.509 INFO:tasks.workunit.client.0.smithi070.stdout: Which is: 1 2024-09-21T11:06:36.510 INFO:tasks.workunit.client.0.smithi070.stdout: keys.size() 2024-09-21T11:06:36.510 INFO:tasks.workunit.client.0.smithi070.stdout: Which is: 3 2024-09-21T11:06:36.510 INFO:tasks.workunit.client.0.smithi070.stdout:[ FAILED ] TestClsRbd.group_snap_set (2 ms) 2024-09-21T11:06:36.510 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.group_snap_list 2024-09-21T11:06:36.512 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.group_snap_list (3 ms) 2024-09-21T11:06:36.512 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.group_snap_list_max_return 2024-09-21T11:06:36.531 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:36.532 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.group_snap_list_max_return (20 ms) 2024-09-21T11:06:36.532 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.group_snap_list_max_read 2024-09-21T11:06:36.554 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:36.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:36.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:36.657 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:36.678 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:36.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:36.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:36.733 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.group_snap_list_max_read (201 ms) 2024-09-21T11:06:36.734 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.group_snap_remove 2024-09-21T11:06:36.737 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.group_snap_remove (4 ms) 2024-09-21T11:06:36.737 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.group_snap_get_by_id 2024-09-21T11:06:36.739 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.group_snap_get_by_id (2 ms) 2024-09-21T11:06:36.739 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.trash_methods 2024-09-21T11:06:36.745 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.trash_methods (7 ms) 2024-09-21T11:06:36.746 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.op_features 2024-09-21T11:06:36.753 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.op_features (7 ms) 2024-09-21T11:06:36.753 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.clone_parent 2024-09-21T11:06:36.773 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.clone_parent (20 ms) 2024-09-21T11:06:36.773 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.clone_parent_ns 2024-09-21T11:06:36.780 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:36.781 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.clone_parent_ns (8 ms) 2024-09-21T11:06:36.781 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.clone_child 2024-09-21T11:06:36.791 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.clone_child (10 ms) 2024-09-21T11:06:36.791 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.namespace_methods 2024-09-21T11:06:36.799 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.namespace_methods (7 ms) 2024-09-21T11:06:36.799 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.migration 2024-09-21T11:06:36.802 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:36.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:36.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:36.808 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.migration (9 ms) 2024-09-21T11:06:36.808 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.migration_v1 2024-09-21T11:06:36.818 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.migration_v1 (10 ms) 2024-09-21T11:06:36.818 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.assert_snapc_seq 2024-09-21T11:06:36.904 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:36.926 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:36.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:36.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:37.028 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:37.049 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:37.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:37.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:37.151 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:37.173 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:37.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:37.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:37.275 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:37.296 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:37.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:37.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:37.398 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:37.419 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:37.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:37.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:37.521 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:37.542 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:37.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:37.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:37.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:37 smithi160 ceph-mon[101879]: pgmap v3293: 125 pgs: 1 active+undersized+remapped, 12 unknown, 112 active+clean; 431 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 2.1 MiB/s rd, 1.1 MiB/s wr, 143 op/s; 33 KiB/s, 0 objects/s recovering 2024-09-21T11:06:37.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:37 smithi160 ceph-mon[101879]: osdmap e1662: 8 total, 8 up, 6 in 2024-09-21T11:06:37.644 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:37.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:37.666 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:37.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:37.669 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:06:37.767 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:37.790 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:37.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:37.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:37.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:37 smithi070 ceph-mon[107385]: pgmap v3293: 125 pgs: 1 active+undersized+remapped, 12 unknown, 112 active+clean; 431 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 2.1 MiB/s rd, 1.1 MiB/s wr, 143 op/s; 33 KiB/s, 0 objects/s recovering 2024-09-21T11:06:37.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:37 smithi070 ceph-mon[107385]: osdmap e1662: 8 total, 8 up, 6 in 2024-09-21T11:06:37.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:37 smithi070 ceph-mon[112139]: pgmap v3293: 125 pgs: 1 active+undersized+remapped, 12 unknown, 112 active+clean; 431 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 2.1 MiB/s rd, 1.1 MiB/s wr, 143 op/s; 33 KiB/s, 0 objects/s recovering 2024-09-21T11:06:37.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:37 smithi070 ceph-mon[112139]: osdmap e1662: 8 total, 8 up, 6 in 2024-09-21T11:06:37.892 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:37.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:37.914 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:37.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:38.015 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:38.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:38.037 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:38.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:38.138 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T11:06:38.159 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:38.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:38.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T11:06:38.261 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T11:06:38.282 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:38.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:38.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T11:06:38.384 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:38.406 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:38.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:38.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:38.508 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:38.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:38.529 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:38.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:38.617 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:06:38.630 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:38.655 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:38.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:38.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:38.689 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.assert_snapc_seq (1871 ms) 2024-09-21T11:06:38.689 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestClsRbd.sparsify 2024-09-21T11:06:38.734 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestClsRbd.sparsify (44 ms) 2024-09-21T11:06:38.756 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:38.779 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:38.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:38.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:38.880 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:38.902 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:38.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:38.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:38.947 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:38 smithi070 ceph-mon[112139]: osdmap e1663: 8 total, 8 up, 6 in 2024-09-21T11:06:38.947 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:38 smithi070 ceph-mon[112139]: 1.12 scrub starts 2024-09-21T11:06:38.948 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:38 smithi070 ceph-mon[112139]: 1.12 scrub ok 2024-09-21T11:06:38.948 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:38 smithi070 ceph-mon[107385]: osdmap e1663: 8 total, 8 up, 6 in 2024-09-21T11:06:38.948 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:38 smithi070 ceph-mon[107385]: 1.12 scrub starts 2024-09-21T11:06:38.948 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:38 smithi070 ceph-mon[107385]: 1.12 scrub ok 2024-09-21T11:06:39.005 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T11:06:39.028 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:39.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:39.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T11:06:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:38 smithi160 ceph-mon[101879]: osdmap e1663: 8 total, 8 up, 6 in 2024-09-21T11:06:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:38 smithi160 ceph-mon[101879]: 1.12 scrub starts 2024-09-21T11:06:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:38 smithi160 ceph-mon[101879]: 1.12 scrub ok 2024-09-21T11:06:39.121 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T11:06:39.130 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:39.156 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:39.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:39.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:39.258 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:39.283 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:39.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:39.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:39.384 DEBUG:teuthology.orchestra.run.smithi070:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T11:06:39.406 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: docker: command not found 2024-09-21T11:06:39.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:39.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi070 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T11:06:39.508 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T11:06:39.535 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:39.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:39.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T11:06:39.637 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T11:06:39.662 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:39.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:39.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T11:06:39.691 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T11:06:39.708 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 70 tests from TestClsRbd (5214 ms total) 2024-09-21T11:06:39.708 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T11:06:39.708 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] Global test environment tear-down 2024-09-21T11:06:39.709 INFO:tasks.workunit.client.0.smithi070.stdout:[==========] 70 tests from 1 test suite ran. (7361 ms total) 2024-09-21T11:06:39.709 INFO:tasks.workunit.client.0.smithi070.stdout:[ PASSED ] 69 tests. 2024-09-21T11:06:39.709 INFO:tasks.workunit.client.0.smithi070.stdout:[ FAILED ] 1 test, listed below: 2024-09-21T11:06:39.709 INFO:tasks.workunit.client.0.smithi070.stdout:[ FAILED ] TestClsRbd.group_snap_set 2024-09-21T11:06:39.709 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-21T11:06:39.709 INFO:tasks.workunit.client.0.smithi070.stdout: 1 FAILED TEST 2024-09-21T11:06:39.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:39.743 INFO:tasks.workunit:Stopping ['cls/test_cls_rbd.sh'] on client.0... 2024-09-21T11:06:39.744 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-21T11:06:39.764 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T11:06:39.787 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:39.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:39.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T11:06:39.841 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:39 smithi070 ceph-mon[107385]: pgmap v3296: 125 pgs: 1 active+undersized+remapped, 124 active+clean; 431 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 4.3 MiB/s rd, 2.1 MiB/s wr, 3.41k op/s; 33 KiB/s, 0 objects/s recovering 2024-09-21T11:06:39.842 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:39 smithi070 ceph-mon[107385]: osdmap e1664: 8 total, 8 up, 6 in 2024-09-21T11:06:39.842 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:39 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4136425708' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T11:06:39.842 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:39 smithi070 ceph-mon[112139]: pgmap v3296: 125 pgs: 1 active+undersized+remapped, 124 active+clean; 431 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 4.3 MiB/s rd, 2.1 MiB/s wr, 3.41k op/s; 33 KiB/s, 0 objects/s recovering 2024-09-21T11:06:39.842 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:39 smithi070 ceph-mon[112139]: osdmap e1664: 8 total, 8 up, 6 in 2024-09-21T11:06:39.842 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:39 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4136425708' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T11:06:39.890 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T11:06:39.912 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-21T11:06:39.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:06:39.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T11:06:39.954 INFO:tasks.thrashosds:joining thrashosds 2024-09-21T11:06:40.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:39 smithi160 ceph-mon[101879]: pgmap v3296: 125 pgs: 1 active+undersized+remapped, 124 active+clean; 431 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 4.3 MiB/s rd, 2.1 MiB/s wr, 3.41k op/s; 33 KiB/s, 0 objects/s recovering 2024-09-21T11:06:40.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:39 smithi160 ceph-mon[101879]: osdmap e1664: 8 total, 8 up, 6 in 2024-09-21T11:06:40.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:39 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4136425708' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T11:06:40.463 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T11:06:40.696 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:40 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4136425708' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T11:06:40.696 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:40 smithi070 ceph-mon[112139]: osdmap e1665: 8 total, 8 up, 6 in 2024-09-21T11:06:41.002 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:40 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4136425708' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T11:06:41.002 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:40 smithi070 ceph-mon[107385]: osdmap e1665: 8 total, 8 up, 6 in 2024-09-21T11:06:41.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:40 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4136425708' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T11:06:41.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:40 smithi160 ceph-mon[101879]: osdmap e1665: 8 total, 8 up, 6 in 2024-09-21T11:06:41.223 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:06:41.224 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:06:41.697 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T11:06:41.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:41 smithi070 ceph-mon[107385]: pgmap v3299: 93 pgs: 1 unknown, 8 peering, 84 active+clean; 404 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:41.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:41 smithi070 ceph-mon[107385]: osdmap e1666: 8 total, 8 up, 6 in 2024-09-21T11:06:41.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:41 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1603648458' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:06:41.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:41 smithi070 ceph-mon[112139]: pgmap v3299: 93 pgs: 1 unknown, 8 peering, 84 active+clean; 404 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:41.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:41 smithi070 ceph-mon[112139]: osdmap e1666: 8 total, 8 up, 6 in 2024-09-21T11:06:41.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:41 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1603648458' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:06:42.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:41 smithi160 ceph-mon[101879]: pgmap v3299: 93 pgs: 1 unknown, 8 peering, 84 active+clean; 404 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:42.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:41 smithi160 ceph-mon[101879]: osdmap e1666: 8 total, 8 up, 6 in 2024-09-21T11:06:42.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:41 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1603648458' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:06:43.025 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:42 smithi070 ceph-mon[107385]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T11:06:43.025 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:42 smithi070 ceph-mon[107385]: Cluster is now healthy 2024-09-21T11:06:43.025 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:42 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1603648458' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T11:06:43.025 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:42 smithi070 ceph-mon[107385]: osdmap e1667: 8 total, 8 up, 6 in 2024-09-21T11:06:43.025 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:42 smithi070 ceph-mon[112139]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T11:06:43.025 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:42 smithi070 ceph-mon[112139]: Cluster is now healthy 2024-09-21T11:06:43.026 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:42 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1603648458' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T11:06:43.026 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:42 smithi070 ceph-mon[112139]: osdmap e1667: 8 total, 8 up, 6 in 2024-09-21T11:06:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:42 smithi160 ceph-mon[101879]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T11:06:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:42 smithi160 ceph-mon[101879]: Cluster is now healthy 2024-09-21T11:06:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:42 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1603648458' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T11:06:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:42 smithi160 ceph-mon[101879]: osdmap e1667: 8 total, 8 up, 6 in 2024-09-21T11:06:43.313 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:06:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:06:43] "GET /metrics HTTP/1.1" 200 36669 "" "Prometheus/2.51.0" 2024-09-21T11:06:43.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:43 smithi070 ceph-mon[107385]: pgmap v3302: 93 pgs: 2 active+undersized+remapped, 1 unknown, 8 peering, 82 active+clean; 404 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:43.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:43 smithi070 ceph-mon[107385]: osdmap e1668: 8 total, 8 up, 6 in 2024-09-21T11:06:43.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:43 smithi070 ceph-mon[112139]: pgmap v3302: 93 pgs: 2 active+undersized+remapped, 1 unknown, 8 peering, 82 active+clean; 404 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:43.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:43 smithi070 ceph-mon[112139]: osdmap e1668: 8 total, 8 up, 6 in 2024-09-21T11:06:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:43 smithi160 ceph-mon[101879]: pgmap v3302: 93 pgs: 2 active+undersized+remapped, 1 unknown, 8 peering, 82 active+clean; 404 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:43 smithi160 ceph-mon[101879]: osdmap e1668: 8 total, 8 up, 6 in 2024-09-21T11:06:44.779 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T11:06:44.977 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:44 smithi070 ceph-mon[107385]: osdmap e1669: 8 total, 8 up, 6 in 2024-09-21T11:06:44.977 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:44 smithi070 ceph-mon[112139]: osdmap e1669: 8 total, 8 up, 6 in 2024-09-21T11:06:45.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:44 smithi160 ceph-mon[101879]: osdmap e1669: 8 total, 8 up, 6 in 2024-09-21T11:06:45.353 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0: {} 2024-09-21T11:06:45.354 INFO:teuthology.orchestra.run.smithi070.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T11:06:45.371 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1: {} 2024-09-21T11:06:45.371 INFO:teuthology.orchestra.run.smithi070.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T11:06:45.387 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2: {} 2024-09-21T11:06:45.387 INFO:teuthology.orchestra.run.smithi070.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T11:06:45.403 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3: {} 2024-09-21T11:06:45.403 INFO:teuthology.orchestra.run.smithi070.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T11:06:45.419 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4: {} 2024-09-21T11:06:45.419 INFO:teuthology.orchestra.run.smithi070.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T11:06:45.438 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5: {} 2024-09-21T11:06:45.438 INFO:teuthology.orchestra.run.smithi070.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T11:06:45.452 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6: {} 2024-09-21T11:06:45.452 INFO:teuthology.orchestra.run.smithi070.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T11:06:45.470 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7: {} 2024-09-21T11:06:45.470 INFO:teuthology.orchestra.run.smithi070.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T11:06:45.564 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T11:06:45.564 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T11:06:45.564 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T11:06:45.564 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T11:06:45.564 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T11:06:45.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:45 smithi070 ceph-mon[107385]: 1.2 scrub starts 2024-09-21T11:06:45.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:45 smithi070 ceph-mon[107385]: 1.2 scrub ok 2024-09-21T11:06:45.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:45 smithi070 ceph-mon[107385]: pgmap v3305: 93 pgs: 2 active+undersized+remapped, 1 unknown, 8 peering, 82 active+clean; 404 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:45.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:45 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2988033576' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:06:45.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:45 smithi070 ceph-mon[112139]: 1.2 scrub starts 2024-09-21T11:06:45.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:45 smithi070 ceph-mon[112139]: 1.2 scrub ok 2024-09-21T11:06:45.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:45 smithi070 ceph-mon[112139]: pgmap v3305: 93 pgs: 2 active+undersized+remapped, 1 unknown, 8 peering, 82 active+clean; 404 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:45.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:45 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2988033576' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:06:46.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:45 smithi160 ceph-mon[101879]: 1.2 scrub starts 2024-09-21T11:06:46.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:45 smithi160 ceph-mon[101879]: 1.2 scrub ok 2024-09-21T11:06:46.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:45 smithi160 ceph-mon[101879]: pgmap v3305: 93 pgs: 2 active+undersized+remapped, 1 unknown, 8 peering, 82 active+clean; 404 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:46.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:45 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2988033576' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T11:06:46.731 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T11:06:47.073 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:46 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1215069486' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T11:06:47.073 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:46 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T11:06:47.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:46 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1215069486' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T11:06:47.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:46 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T11:06:47.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:46 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1215069486' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T11:06:47.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:46 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T11:06:48.000 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:47 smithi070 ceph-mon[107385]: pgmap v3306: 93 pgs: 2 active+undersized+remapped, 91 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:48.000 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:47 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T11:06:48.000 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:47 smithi070 ceph-mon[107385]: osdmap e1670: 8 total, 8 up, 6 in 2024-09-21T11:06:48.000 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:47 smithi070 ceph-mon[112139]: pgmap v3306: 93 pgs: 2 active+undersized+remapped, 91 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:48.000 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:47 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T11:06:48.001 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:47 smithi070 ceph-mon[112139]: osdmap e1670: 8 total, 8 up, 6 in 2024-09-21T11:06:48.007 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:06:48.028 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T11:06:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:47 smithi160 ceph-mon[101879]: pgmap v3306: 93 pgs: 2 active+undersized+remapped, 91 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:47 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T11:06:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:47 smithi160 ceph-mon[101879]: osdmap e1670: 8 total, 8 up, 6 in 2024-09-21T11:06:48.376 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T11:06:49.096 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:06:49.096 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:06:49.981 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:49 smithi070 ceph-mon[112139]: pgmap v3308: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:49.981 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:49 smithi070 ceph-mon[107385]: pgmap v3308: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:50.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:49 smithi160 ceph-mon[101879]: pgmap v3308: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:51.738 INFO:teuthology.orchestra.run.smithi070.stderr:noscrub is unset 2024-09-21T11:06:51.738 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:51 smithi070 ceph-mon[107385]: pgmap v3309: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:51.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:06:51.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:06:51.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:51 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3203293479' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:06:51.739 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:51 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:06:51.739 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:51 smithi070 ceph-mon[112139]: pgmap v3309: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:51.739 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:06:51.739 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:06:51.739 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:51 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3203293479' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:06:51.739 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:51 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:06:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:51 smithi160 ceph-mon[101879]: pgmap v3309: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:06:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:06:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:51 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3203293479' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:06:51.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:51 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T11:06:53.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:52 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T11:06:53.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:52 smithi070 ceph-mon[107385]: osdmap e1671: 8 total, 8 up, 6 in 2024-09-21T11:06:53.005 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:52 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T11:06:53.005 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:52 smithi070 ceph-mon[112139]: osdmap e1671: 8 total, 8 up, 6 in 2024-09-21T11:06:53.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:52 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T11:06:53.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:52 smithi160 ceph-mon[101879]: osdmap e1671: 8 total, 8 up, 6 in 2024-09-21T11:06:53.299 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:06:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:06:53] "GET /metrics HTTP/1.1" 200 34283 "" "Prometheus/2.51.0" 2024-09-21T11:06:53.518 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T11:06:53.518 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T11:06:53.518 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T11:06:53.518 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T11:06:53.518 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.518 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:235449 58.8 58:1f16ff14:::benchmark_data_smithi070_396992_object89490:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1614)", 2024-09-21T11:06:53.519 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:04:53.631352+0000", 2024-09-21T11:06:53.519 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 119.883198484, 2024-09-21T11:06:53.519 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.119697275, 2024-09-21T11:06:53.519 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:53.519 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:53.519 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:06:53.519 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:53.519 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:53.519 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:53.519 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235449 2024-09-21T11:06:53.519 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.519 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:53.519 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.520 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:53.520 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631352+0000", 2024-09-21T11:06:53.520 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.520 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.520 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.520 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:53.520 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631352+0000", 2024-09-21T11:06:53.520 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.520 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.520 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.520 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:53.520 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631354+0000", 2024-09-21T11:06:53.520 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.64e-06 2024-09-21T11:06:53.520 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.521 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.521 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:53.521 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631359+0000", 2024-09-21T11:06:53.521 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3980000000000001e-06 2024-09-21T11:06:53.521 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.521 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.521 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:53.521 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631381+0000", 2024-09-21T11:06:53.521 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2833999999999999e-05 2024-09-21T11:06:53.521 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.521 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.521 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:53.522 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631390+0000", 2024-09-21T11:06:53.522 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3590000000000008e-06 2024-09-21T11:06:53.522 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.522 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.522 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.522 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.632465+0000", 2024-09-21T11:06:53.522 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00107569 2024-09-21T11:06:53.522 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.522 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.522 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:53.522 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.632582+0000", 2024-09-21T11:06:53.522 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011671399999999999 2024-09-21T11:06:53.522 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.523 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.523 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,6", 2024-09-21T11:06:53.523 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.632686+0000", 2024-09-21T11:06:53.523 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000104108 2024-09-21T11:06:53.523 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.523 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.523 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.523 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.649651+0000", 2024-09-21T11:06:53.523 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.016964858999999999 2024-09-21T11:06:53.523 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.523 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.523 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:53.523 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.649662+0000", 2024-09-21T11:06:53.523 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0492999999999999e-05 2024-09-21T11:06:53.524 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.524 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.524 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.524 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.750958+0000", 2024-09-21T11:06:53.524 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1012967220000001 2024-09-21T11:06:53.524 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.524 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.524 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:53.524 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.751050+0000", 2024-09-21T11:06:53.524 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1458000000000004e-05 2024-09-21T11:06:53.524 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.524 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:53.524 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.524 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.525 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.525 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:235450 58.8 58:1c8979f6:::benchmark_data_smithi070_396992_object89491:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1614)", 2024-09-21T11:06:53.525 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:04:53.631405+0000", 2024-09-21T11:06:53.525 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 119.883146095, 2024-09-21T11:06:53.525 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.120069628, 2024-09-21T11:06:53.525 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:53.525 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:53.525 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:06:53.525 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:53.525 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:53.525 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:53.525 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235450 2024-09-21T11:06:53.526 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.526 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:53.526 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.526 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:53.526 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631405+0000", 2024-09-21T11:06:53.526 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.526 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.526 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.526 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:53.526 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631405+0000", 2024-09-21T11:06:53.526 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.526 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.526 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.527 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:53.527 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631405+0000", 2024-09-21T11:06:53.527 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5799999999999999e-07 2024-09-21T11:06:53.527 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.527 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.527 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:53.527 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631409+0000", 2024-09-21T11:06:53.527 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.089e-06 2024-09-21T11:06:53.527 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.527 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.527 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:53.527 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631417+0000", 2024-09-21T11:06:53.527 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1179999999999992e-06 2024-09-21T11:06:53.527 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.528 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.528 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:53.528 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631422+0000", 2024-09-21T11:06:53.528 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6239999999999999e-06 2024-09-21T11:06:53.528 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.528 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.528 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.528 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.634516+0000", 2024-09-21T11:06:53.528 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0030942040000000001 2024-09-21T11:06:53.528 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.528 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.528 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:53.528 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.634639+0000", 2024-09-21T11:06:53.529 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012290700000000001 2024-09-21T11:06:53.529 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.529 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.529 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,6", 2024-09-21T11:06:53.529 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.634756+0000", 2024-09-21T11:06:53.529 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000116879 2024-09-21T11:06:53.529 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.529 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.529 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.529 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.649689+0000", 2024-09-21T11:06:53.529 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.014932288 2024-09-21T11:06:53.529 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.529 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.529 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:53.529 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.649697+0000", 2024-09-21T11:06:53.530 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5750000000000001e-06 2024-09-21T11:06:53.530 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.530 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.530 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.530 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.751406+0000", 2024-09-21T11:06:53.530 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.101708686 2024-09-21T11:06:53.530 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.530 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.530 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:53.530 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.751474+0000", 2024-09-21T11:06:53.530 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8700000000000003e-05 2024-09-21T11:06:53.530 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.531 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:53.531 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.531 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.531 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.531 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:235451 58.8 58:1d7c78d8:::benchmark_data_smithi070_396992_object89492:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1614)", 2024-09-21T11:06:53.531 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:04:53.631431+0000", 2024-09-21T11:06:53.531 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 119.883120034, 2024-09-21T11:06:53.531 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1351109159999999, 2024-09-21T11:06:53.531 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:53.531 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:53.531 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:06:53.531 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:53.531 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:53.532 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:53.532 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235451 2024-09-21T11:06:53.532 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.532 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:53.532 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.532 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:53.532 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631431+0000", 2024-09-21T11:06:53.532 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.532 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.532 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.532 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:53.532 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631431+0000", 2024-09-21T11:06:53.532 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.533 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.533 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.533 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:53.533 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631431+0000", 2024-09-21T11:06:53.533 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9399999999999995e-07 2024-09-21T11:06:53.533 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.533 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.533 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:53.533 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631434+0000", 2024-09-21T11:06:53.533 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.43e-06 2024-09-21T11:06:53.533 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.533 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.533 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:53.534 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631441+0000", 2024-09-21T11:06:53.534 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6399999999999997e-06 2024-09-21T11:06:53.534 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.534 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.534 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:53.534 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631447+0000", 2024-09-21T11:06:53.534 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3399999999999997e-06 2024-09-21T11:06:53.534 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.534 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.534 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.534 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.638055+0000", 2024-09-21T11:06:53.534 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0066081930000000001 2024-09-21T11:06:53.535 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.535 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.535 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:53.535 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.638351+0000", 2024-09-21T11:06:53.535 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00029561600000000001 2024-09-21T11:06:53.535 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.535 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.535 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,6", 2024-09-21T11:06:53.535 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.639288+0000", 2024-09-21T11:06:53.535 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00093712899999999998 2024-09-21T11:06:53.536 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.536 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.536 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.536 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.649726+0000", 2024-09-21T11:06:53.536 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.010438578 2024-09-21T11:06:53.536 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.536 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.536 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:53.536 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.649734+0000", 2024-09-21T11:06:53.536 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.678e-06 2024-09-21T11:06:53.536 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.536 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.536 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.536 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.751533+0000", 2024-09-21T11:06:53.537 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1017984919999999 2024-09-21T11:06:53.537 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.537 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.537 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for missing object", 2024-09-21T11:06:53.537 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.751654+0000", 2024-09-21T11:06:53.537 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012138000000000001 2024-09-21T11:06:53.537 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.537 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.537 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.537 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.766418+0000", 2024-09-21T11:06:53.537 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.014763940999999999 2024-09-21T11:06:53.537 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.537 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.537 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:53.538 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.766542+0000", 2024-09-21T11:06:53.538 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000124005 2024-09-21T11:06:53.538 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.538 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:53.538 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.538 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.538 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.538 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:235453 58.1 58:8c494647:::benchmark_data_smithi070_396992_object89494:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1614)", 2024-09-21T11:06:53.538 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:04:53.631454+0000", 2024-09-21T11:06:53.538 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 119.883096806, 2024-09-21T11:06:53.538 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1240480150000001, 2024-09-21T11:06:53.538 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:53.539 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:53.539 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:06:53.539 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:53.539 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:53.539 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:53.539 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235453 2024-09-21T11:06:53.539 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.539 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:53.539 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.539 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:53.539 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631454+0000", 2024-09-21T11:06:53.539 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.539 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.539 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.540 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:53.540 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631454+0000", 2024-09-21T11:06:53.540 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.540 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.540 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.540 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:53.540 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631455+0000", 2024-09-21T11:06:53.540 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.4000000000000002e-07 2024-09-21T11:06:53.540 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.540 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.540 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:53.540 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631457+0000", 2024-09-21T11:06:53.540 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2979999999999999e-06 2024-09-21T11:06:53.540 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.541 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.541 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:53.541 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631464+0000", 2024-09-21T11:06:53.541 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8299999999999998e-06 2024-09-21T11:06:53.541 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.541 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.541 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:53.541 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631469+0000", 2024-09-21T11:06:53.541 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0170000000000004e-06 2024-09-21T11:06:53.541 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.541 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.541 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.541 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631869+0000", 2024-09-21T11:06:53.541 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00040066 2024-09-21T11:06:53.541 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.542 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.542 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:53.542 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.632006+0000", 2024-09-21T11:06:53.542 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000136348 2024-09-21T11:06:53.542 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.542 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.542 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,6,7", 2024-09-21T11:06:53.542 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.632125+0000", 2024-09-21T11:06:53.542 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011964999999999999 2024-09-21T11:06:53.542 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.542 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.542 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:53.542 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.641319+0000", 2024-09-21T11:06:53.542 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0091932349999999993 2024-09-21T11:06:53.543 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.543 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.543 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.543 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.654205+0000", 2024-09-21T11:06:53.543 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012886195 2024-09-21T11:06:53.543 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.543 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.543 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:53.543 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.654214+0000", 2024-09-21T11:06:53.543 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5759999999999995e-06 2024-09-21T11:06:53.543 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.543 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.543 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.543 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.755422+0000", 2024-09-21T11:06:53.544 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.101207442 2024-09-21T11:06:53.544 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.544 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.544 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:53.544 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.755502+0000", 2024-09-21T11:06:53.544 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0323999999999999e-05 2024-09-21T11:06:53.544 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.544 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:53.544 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.544 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.544 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.544 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:235454 58.8 58:1dd9e6f2:::benchmark_data_smithi070_396992_object89495:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1614)", 2024-09-21T11:06:53.544 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:04:53.631556+0000", 2024-09-21T11:06:53.545 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 119.882994654, 2024-09-21T11:06:53.545 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.134390198, 2024-09-21T11:06:53.545 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:53.545 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:53.545 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:06:53.545 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:53.545 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:53.545 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:53.545 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235454 2024-09-21T11:06:53.545 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.545 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:53.545 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.546 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:53.546 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631556+0000", 2024-09-21T11:06:53.546 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.546 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.546 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.546 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:53.546 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631556+0000", 2024-09-21T11:06:53.546 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.546 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.546 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.546 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:53.546 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631557+0000", 2024-09-21T11:06:53.546 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5499999999999997e-07 2024-09-21T11:06:53.546 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.547 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.547 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:53.547 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631561+0000", 2024-09-21T11:06:53.547 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.861e-06 2024-09-21T11:06:53.547 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.547 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.547 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:53.547 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631572+0000", 2024-09-21T11:06:53.547 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1187e-05 2024-09-21T11:06:53.547 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.547 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.548 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:53.548 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.631578+0000", 2024-09-21T11:06:53.548 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.003e-06 2024-09-21T11:06:53.548 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.549 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.549 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.549 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.642010+0000", 2024-09-21T11:06:53.549 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.010431419000000001 2024-09-21T11:06:53.549 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.549 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.549 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:53.549 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.642147+0000", 2024-09-21T11:06:53.549 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013762799999999999 2024-09-21T11:06:53.549 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.549 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.549 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,6", 2024-09-21T11:06:53.549 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.642268+0000", 2024-09-21T11:06:53.550 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000120364 2024-09-21T11:06:53.550 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.550 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.550 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.550 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.649762+0000", 2024-09-21T11:06:53.550 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0074947269999999996 2024-09-21T11:06:53.550 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.550 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.550 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:53.550 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.649770+0000", 2024-09-21T11:06:53.550 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.824e-06 2024-09-21T11:06:53.550 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.550 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.550 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.551 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.751692+0000", 2024-09-21T11:06:53.551 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.101921763 2024-09-21T11:06:53.551 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.551 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.551 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for missing object", 2024-09-21T11:06:53.551 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.751774+0000", 2024-09-21T11:06:53.551 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1737999999999996e-05 2024-09-21T11:06:53.551 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.551 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.551 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.551 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.765856+0000", 2024-09-21T11:06:53.551 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.014082910000000001 2024-09-21T11:06:53.551 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.552 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.552 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:53.552 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.765946+0000", 2024-09-21T11:06:53.552 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0019000000000007e-05 2024-09-21T11:06:53.552 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.552 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:53.552 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.552 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.552 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.552 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:235457 58.8 58:15b1704f:::benchmark_data_smithi070_396992_object89498:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1614)", 2024-09-21T11:06:53.552 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:04:53.645488+0000", 2024-09-21T11:06:53.552 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 119.869062599, 2024-09-21T11:06:53.552 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.120015905, 2024-09-21T11:06:53.552 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:53.553 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:53.553 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:53.553 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:53.553 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:53.553 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:53.553 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235457 2024-09-21T11:06:53.553 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.553 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:53.553 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.553 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:53.553 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.645488+0000", 2024-09-21T11:06:53.553 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.553 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.554 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.554 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:53.554 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.645488+0000", 2024-09-21T11:06:53.554 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.554 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.554 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.554 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:53.554 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.645490+0000", 2024-09-21T11:06:53.554 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6899999999999999e-06 2024-09-21T11:06:53.554 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.554 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.554 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:53.554 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.645496+0000", 2024-09-21T11:06:53.554 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7570000000000001e-06 2024-09-21T11:06:53.555 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.555 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.555 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:53.555 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.645508+0000", 2024-09-21T11:06:53.555 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2258e-05 2024-09-21T11:06:53.555 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.555 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.555 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:53.555 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.645516+0000", 2024-09-21T11:06:53.555 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0879999999999995e-06 2024-09-21T11:06:53.555 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.555 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.555 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.555 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.649798+0000", 2024-09-21T11:06:53.556 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0042818630000000003 2024-09-21T11:06:53.556 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.556 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.556 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:53.556 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.649805+0000", 2024-09-21T11:06:53.556 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8759999999999999e-06 2024-09-21T11:06:53.556 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.556 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.556 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.556 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.751932+0000", 2024-09-21T11:06:53.556 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1021274240000001 2024-09-21T11:06:53.556 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.556 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.556 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:53.557 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.752050+0000", 2024-09-21T11:06:53.557 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000117429 2024-09-21T11:06:53.557 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.557 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.557 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,4,6", 2024-09-21T11:06:53.557 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.752182+0000", 2024-09-21T11:06:53.557 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000132194 2024-09-21T11:06:53.557 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.557 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.557 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.557 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.757255+0000", 2024-09-21T11:06:53.557 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0050730130000000003 2024-09-21T11:06:53.557 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.557 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.558 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:53.558 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.759726+0000", 2024-09-21T11:06:53.558 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0024705719999999999 2024-09-21T11:06:53.558 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.558 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.558 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.558 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.760105+0000", 2024-09-21T11:06:53.558 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00037963500000000002 2024-09-21T11:06:53.558 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.558 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.558 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.558 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.765324+0000", 2024-09-21T11:06:53.558 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0052185979999999996 2024-09-21T11:06:53.558 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.559 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.559 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:53.559 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.765391+0000", 2024-09-21T11:06:53.559 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7226999999999999e-05 2024-09-21T11:06:53.559 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.559 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.559 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:53.559 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.765504+0000", 2024-09-21T11:06:53.559 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000113281 2024-09-21T11:06:53.559 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.559 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:53.559 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.559 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.560 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.560 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:235462 58.8 58:1db8c9d0:::benchmark_data_smithi070_396992_object89503:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1614)", 2024-09-21T11:06:53.560 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:04:53.646590+0000", 2024-09-21T11:06:53.560 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 119.867960498, 2024-09-21T11:06:53.560 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1195718189999999, 2024-09-21T11:06:53.560 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:53.560 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:53.560 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:53.560 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:53.560 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:53.560 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:53.560 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235462 2024-09-21T11:06:53.560 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.560 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:53.561 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.561 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:53.561 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.646590+0000", 2024-09-21T11:06:53.561 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.561 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.561 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.561 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:53.561 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.646590+0000", 2024-09-21T11:06:53.561 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.561 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.561 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.561 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:53.561 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.646592+0000", 2024-09-21T11:06:53.561 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3480000000000001e-06 2024-09-21T11:06:53.562 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.562 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.562 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:53.562 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.646596+0000", 2024-09-21T11:06:53.562 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.4460000000000003e-06 2024-09-21T11:06:53.562 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.562 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.562 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:53.562 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.646608+0000", 2024-09-21T11:06:53.562 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1474000000000001e-05 2024-09-21T11:06:53.562 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.562 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.562 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:53.563 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.646615+0000", 2024-09-21T11:06:53.563 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1670000000000002e-06 2024-09-21T11:06:53.563 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.563 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.563 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.563 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.649830+0000", 2024-09-21T11:06:53.563 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0032155159999999999 2024-09-21T11:06:53.563 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.563 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.563 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:53.563 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.649838+0000", 2024-09-21T11:06:53.563 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7030000000000006e-06 2024-09-21T11:06:53.563 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.563 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.564 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.564 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.754475+0000", 2024-09-21T11:06:53.564 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1046370190000001 2024-09-21T11:06:53.564 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.564 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.564 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:53.564 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.754562+0000", 2024-09-21T11:06:53.564 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7028999999999996e-05 2024-09-21T11:06:53.564 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.564 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.564 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,4,6", 2024-09-21T11:06:53.564 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.754708+0000", 2024-09-21T11:06:53.564 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014552599999999999 2024-09-21T11:06:53.565 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.565 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.565 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.565 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.757294+0000", 2024-09-21T11:06:53.565 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025863169999999999 2024-09-21T11:06:53.565 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.565 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.565 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:53.565 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.759738+0000", 2024-09-21T11:06:53.565 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002443598 2024-09-21T11:06:53.565 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.565 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.565 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.565 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.762801+0000", 2024-09-21T11:06:53.565 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.003063049 2024-09-21T11:06:53.566 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.566 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.566 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.566 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.765985+0000", 2024-09-21T11:06:53.566 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0031843069999999999 2024-09-21T11:06:53.566 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.566 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.566 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:53.566 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.766049+0000", 2024-09-21T11:06:53.566 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4323000000000001e-05 2024-09-21T11:06:53.566 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.566 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.566 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:53.567 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.766162+0000", 2024-09-21T11:06:53.567 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000112997 2024-09-21T11:06:53.567 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.567 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:53.567 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.567 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.567 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.567 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:235465 58.1 58:8b2b1985:::benchmark_data_smithi070_396992_object89506:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1615)", 2024-09-21T11:06:53.567 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:04:53.655408+0000", 2024-09-21T11:06:53.567 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 119.859143317, 2024-09-21T11:06:53.567 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.112193005, 2024-09-21T11:06:53.567 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:53.567 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:53.567 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:53.568 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:53.568 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:53.568 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:53.568 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235465 2024-09-21T11:06:53.568 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.568 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:53.568 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.568 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:53.568 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.655408+0000", 2024-09-21T11:06:53.568 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.568 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.568 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.568 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:53.568 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.655408+0000", 2024-09-21T11:06:53.568 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.568 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.569 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.569 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:53.569 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.655409+0000", 2024-09-21T11:06:53.569 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.367e-06 2024-09-21T11:06:53.569 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.569 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.569 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:53.569 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.655413+0000", 2024-09-21T11:06:53.569 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.788e-06 2024-09-21T11:06:53.569 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.569 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.569 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:53.569 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.655423+0000", 2024-09-21T11:06:53.569 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0227999999999999e-05 2024-09-21T11:06:53.570 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.570 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.570 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:53.570 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.655431+0000", 2024-09-21T11:06:53.570 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9470000000000003e-06 2024-09-21T11:06:53.570 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.570 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.570 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.570 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.655475+0000", 2024-09-21T11:06:53.570 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.4174000000000001e-05 2024-09-21T11:06:53.571 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.571 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.571 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:53.571 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.655482+0000", 2024-09-21T11:06:53.571 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8129999999999998e-06 2024-09-21T11:06:53.571 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.571 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.571 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.571 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.755759+0000", 2024-09-21T11:06:53.571 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1002767499999999 2024-09-21T11:06:53.571 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.571 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.571 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:53.572 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.755870+0000", 2024-09-21T11:06:53.572 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000111731 2024-09-21T11:06:53.572 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.572 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.572 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,6,7", 2024-09-21T11:06:53.572 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.755991+0000", 2024-09-21T11:06:53.572 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000121042 2024-09-21T11:06:53.572 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.572 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.572 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.572 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.760794+0000", 2024-09-21T11:06:53.572 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0048026120000000004 2024-09-21T11:06:53.572 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.573 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.573 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.573 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.760847+0000", 2024-09-21T11:06:53.573 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2584999999999999e-05 2024-09-21T11:06:53.573 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.573 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.573 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:53.573 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.767394+0000", 2024-09-21T11:06:53.573 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0065476509999999998 2024-09-21T11:06:53.573 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.573 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.573 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.573 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.767438+0000", 2024-09-21T11:06:53.574 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3470000000000002e-05 2024-09-21T11:06:53.574 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.574 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.574 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:53.574 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.767489+0000", 2024-09-21T11:06:53.574 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0968999999999999e-05 2024-09-21T11:06:53.574 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.574 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.574 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:53.574 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.767601+0000", 2024-09-21T11:06:53.574 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000111878 2024-09-21T11:06:53.574 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.574 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:53.575 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.575 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.575 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.575 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:235467 58.1 58:8bb65d96:::benchmark_data_smithi070_396992_object89508:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1615)", 2024-09-21T11:06:53.575 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:04:53.655749+0000", 2024-09-21T11:06:53.575 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 119.858802428, 2024-09-21T11:06:53.575 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.112152585, 2024-09-21T11:06:53.575 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:53.575 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:53.575 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:53.575 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:53.575 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:53.575 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:53.575 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235467 2024-09-21T11:06:53.575 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.576 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:53.576 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.576 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:53.576 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.655749+0000", 2024-09-21T11:06:53.576 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.576 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.576 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.576 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:53.576 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.655749+0000", 2024-09-21T11:06:53.576 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.576 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.576 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.576 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:53.576 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.655749+0000", 2024-09-21T11:06:53.577 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7000000000000003e-07 2024-09-21T11:06:53.577 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.577 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.577 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:53.577 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.655754+0000", 2024-09-21T11:06:53.577 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2869999999999998e-06 2024-09-21T11:06:53.577 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.577 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.577 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:53.577 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.655765+0000", 2024-09-21T11:06:53.577 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0978999999999999e-05 2024-09-21T11:06:53.577 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.577 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.578 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:53.578 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.655772+0000", 2024-09-21T11:06:53.578 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1300000000000003e-06 2024-09-21T11:06:53.578 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.578 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.578 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.578 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.655826+0000", 2024-09-21T11:06:53.578 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4153000000000001e-05 2024-09-21T11:06:53.578 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.578 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.578 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:53.578 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.655834+0000", 2024-09-21T11:06:53.578 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2199999999999992e-06 2024-09-21T11:06:53.578 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.579 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.579 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.579 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.758159+0000", 2024-09-21T11:06:53.579 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.102324707 2024-09-21T11:06:53.579 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.579 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.579 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:53.579 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.758277+0000", 2024-09-21T11:06:53.579 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000117648 2024-09-21T11:06:53.579 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.579 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.579 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,6,7", 2024-09-21T11:06:53.579 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.758405+0000", 2024-09-21T11:06:53.579 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000128221 2024-09-21T11:06:53.580 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.580 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.580 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:53.580 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.767400+0000", 2024-09-21T11:06:53.580 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0089952369999999997 2024-09-21T11:06:53.580 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.580 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.580 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.580 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.767650+0000", 2024-09-21T11:06:53.580 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00025048300000000002 2024-09-21T11:06:53.580 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.580 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.580 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.580 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.767694+0000", 2024-09-21T11:06:53.580 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3352000000000001e-05 2024-09-21T11:06:53.580 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.581 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.581 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.581 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.767736+0000", 2024-09-21T11:06:53.581 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2505e-05 2024-09-21T11:06:53.581 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.581 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.581 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:53.581 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.767790+0000", 2024-09-21T11:06:53.581 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3884000000000001e-05 2024-09-21T11:06:53.581 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.581 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.581 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:53.581 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.767901+0000", 2024-09-21T11:06:53.581 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000110909 2024-09-21T11:06:53.582 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.582 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:53.582 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.582 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.582 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.582 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:235469 58.1 58:846aa98e:::benchmark_data_smithi070_396992_object89510:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1615)", 2024-09-21T11:06:53.582 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:04:53.655968+0000", 2024-09-21T11:06:53.582 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 119.858583035, 2024-09-21T11:06:53.582 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.119198192, 2024-09-21T11:06:53.582 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:53.582 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:53.582 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:53.582 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:53.583 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:53.583 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:53.583 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235469 2024-09-21T11:06:53.583 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.583 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:53.583 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.583 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:53.583 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.655968+0000", 2024-09-21T11:06:53.583 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.583 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.584 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.584 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:53.584 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.655968+0000", 2024-09-21T11:06:53.584 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.584 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.584 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.584 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:53.584 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.655969+0000", 2024-09-21T11:06:53.584 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.8199999999999998e-07 2024-09-21T11:06:53.584 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.584 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.584 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:53.584 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.655973+0000", 2024-09-21T11:06:53.584 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9740000000000004e-06 2024-09-21T11:06:53.585 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.585 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.585 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:53.585 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.655986+0000", 2024-09-21T11:06:53.585 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3152e-05 2024-09-21T11:06:53.585 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.585 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.585 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:53.585 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.655993+0000", 2024-09-21T11:06:53.585 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2969999999999998e-06 2024-09-21T11:06:53.585 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.585 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.585 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.586 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.656040+0000", 2024-09-21T11:06:53.586 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6635999999999999e-05 2024-09-21T11:06:53.586 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.586 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.586 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:53.586 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.656047+0000", 2024-09-21T11:06:53.586 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5460000000000003e-06 2024-09-21T11:06:53.586 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.586 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.586 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.586 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.760879+0000", 2024-09-21T11:06:53.586 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1048312010000001 2024-09-21T11:06:53.587 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.587 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.587 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:53.587 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.760994+0000", 2024-09-21T11:06:53.587 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011489600000000001 2024-09-21T11:06:53.587 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.587 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.587 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,6,7", 2024-09-21T11:06:53.587 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.761116+0000", 2024-09-21T11:06:53.587 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000122514 2024-09-21T11:06:53.587 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.587 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.587 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:53.588 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.767612+0000", 2024-09-21T11:06:53.588 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0064955339999999999 2024-09-21T11:06:53.588 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.588 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.588 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.588 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.767941+0000", 2024-09-21T11:06:53.588 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00032988700000000001 2024-09-21T11:06:53.588 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.588 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.588 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.588 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.774922+0000", 2024-09-21T11:06:53.588 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0069801289999999998 2024-09-21T11:06:53.588 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.588 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.589 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.589 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.774976+0000", 2024-09-21T11:06:53.589 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4125000000000001e-05 2024-09-21T11:06:53.589 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.589 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.589 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:53.589 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.775036+0000", 2024-09-21T11:06:53.589 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0769e-05 2024-09-21T11:06:53.589 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.589 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.589 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:53.589 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.775166+0000", 2024-09-21T11:06:53.589 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012965000000000001 2024-09-21T11:06:53.589 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.590 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:53.590 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.590 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.590 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.590 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:235471 58.1 58:8632999b:::benchmark_data_smithi070_396992_object89512:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1615)", 2024-09-21T11:06:53.590 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:04:53.664751+0000", 2024-09-21T11:06:53.590 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 119.849799586, 2024-09-21T11:06:53.590 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1141385589999999, 2024-09-21T11:06:53.590 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:53.590 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:53.590 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:53.590 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:53.590 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235471 2024-09-21T11:06:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.664751+0000", 2024-09-21T11:06:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.664751+0000", 2024-09-21T11:06:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.591 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.664753+0000", 2024-09-21T11:06:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6840000000000001e-06 2024-09-21T11:06:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.664758+0000", 2024-09-21T11:06:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.383e-06 2024-09-21T11:06:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.664770+0000", 2024-09-21T11:06:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1379000000000001e-05 2024-09-21T11:06:53.592 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.593 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.593 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:53.593 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.664781+0000", 2024-09-21T11:06:53.593 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0899e-05 2024-09-21T11:06:53.593 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.593 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.593 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.593 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.664959+0000", 2024-09-21T11:06:53.593 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017847499999999999 2024-09-21T11:06:53.593 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.593 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.593 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.664968+0000", 2024-09-21T11:06:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6519999999999995e-06 2024-09-21T11:06:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.764472+0000", 2024-09-21T11:06:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.099503785 2024-09-21T11:06:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.764611+0000", 2024-09-21T11:06:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013968699999999999 2024-09-21T11:06:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.594 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,6,7", 2024-09-21T11:06:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.764744+0000", 2024-09-21T11:06:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000133118 2024-09-21T11:06:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.767988+0000", 2024-09-21T11:06:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.003243541 2024-09-21T11:06:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.775188+0000", 2024-09-21T11:06:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0071998110000000004 2024-09-21T11:06:53.595 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.775259+0000", 2024-09-21T11:06:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0765000000000004e-05 2024-09-21T11:06:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.778723+0000", 2024-09-21T11:06:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0034643310000000002 2024-09-21T11:06:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.778773+0000", 2024-09-21T11:06:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0173000000000002e-05 2024-09-21T11:06:53.596 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.778890+0000", 2024-09-21T11:06:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000116876 2024-09-21T11:06:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:235476 58.1 58:8d88b02c:::benchmark_data_smithi070_396992_object89517:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1615)", 2024-09-21T11:06:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:04:53.672711+0000", 2024-09-21T11:06:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 119.84183985, 2024-09-21T11:06:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1087017450000001, 2024-09-21T11:06:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:53.597 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:53.598 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:53.598 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:53.598 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:53.598 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:53.598 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235476 2024-09-21T11:06:53.598 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.598 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:53.598 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.598 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:53.598 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.672711+0000", 2024-09-21T11:06:53.598 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.598 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.598 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.599 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:53.599 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.672711+0000", 2024-09-21T11:06:53.599 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.599 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.599 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.599 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:53.599 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.672713+0000", 2024-09-21T11:06:53.599 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4139999999999999e-06 2024-09-21T11:06:53.599 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.600 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.600 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:53.600 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.672717+0000", 2024-09-21T11:06:53.600 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.4360000000000004e-06 2024-09-21T11:06:53.600 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.600 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.600 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:53.600 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.672727+0000", 2024-09-21T11:06:53.600 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0074e-05 2024-09-21T11:06:53.600 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.600 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.600 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:53.601 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.672735+0000", 2024-09-21T11:06:53.601 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.79e-06 2024-09-21T11:06:53.601 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.601 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.601 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.601 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.672806+0000", 2024-09-21T11:06:53.601 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0946999999999997e-05 2024-09-21T11:06:53.601 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.601 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.601 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:53.601 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.672818+0000", 2024-09-21T11:06:53.602 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1976e-05 2024-09-21T11:06:53.602 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.602 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.602 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.602 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.768018+0000", 2024-09-21T11:06:53.602 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0952000769999999 2024-09-21T11:06:53.602 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.602 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.602 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:53.602 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.768137+0000", 2024-09-21T11:06:53.602 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011910599999999999 2024-09-21T11:06:53.603 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.603 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.603 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,6,7", 2024-09-21T11:06:53.603 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.768261+0000", 2024-09-21T11:06:53.603 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012403800000000001 2024-09-21T11:06:53.603 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.603 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.603 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:53.603 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.775195+0000", 2024-09-21T11:06:53.603 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0069341130000000004 2024-09-21T11:06:53.603 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.603 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.604 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.604 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.775309+0000", 2024-09-21T11:06:53.604 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011409 2024-09-21T11:06:53.604 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.604 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.604 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.604 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.775625+0000", 2024-09-21T11:06:53.604 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00031573300000000001 2024-09-21T11:06:53.604 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.604 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.604 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.604 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.781122+0000", 2024-09-21T11:06:53.605 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0054971380000000004 2024-09-21T11:06:53.605 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.605 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.605 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:53.605 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.781285+0000", 2024-09-21T11:06:53.605 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000162802 2024-09-21T11:06:53.605 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.605 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.605 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:53.605 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.781413+0000", 2024-09-21T11:06:53.605 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000128011 2024-09-21T11:06:53.605 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.605 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:53.606 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.606 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.606 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.606 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:235494 58.1 58:80c090a1:::benchmark_data_smithi070_396992_object89535:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1615)", 2024-09-21T11:06:53.606 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:04:53.697249+0000", 2024-09-21T11:06:53.606 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 119.817301969, 2024-09-21T11:06:53.606 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.108064172, 2024-09-21T11:06:53.606 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:53.606 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:53.606 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:53.606 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:53.607 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:53.607 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:53.607 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 235494 2024-09-21T11:06:53.607 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.607 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:53.607 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.607 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:53.607 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.697249+0000", 2024-09-21T11:06:53.607 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.607 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.607 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.607 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:53.608 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.697249+0000", 2024-09-21T11:06:53.608 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.608 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.608 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.608 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:53.608 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.697250+0000", 2024-09-21T11:06:53.608 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0299999999999999e-07 2024-09-21T11:06:53.608 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.608 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.608 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:53.608 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.697252+0000", 2024-09-21T11:06:53.608 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5670000000000002e-06 2024-09-21T11:06:53.609 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.609 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.609 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:53.609 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.697259+0000", 2024-09-21T11:06:53.609 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7050000000000002e-06 2024-09-21T11:06:53.609 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.609 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.609 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:53.609 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.697262+0000", 2024-09-21T11:06:53.609 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5659999999999998e-06 2024-09-21T11:06:53.609 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.609 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.610 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.610 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.697297+0000", 2024-09-21T11:06:53.610 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4332000000000003e-05 2024-09-21T11:06:53.610 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.610 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.610 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:53.610 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:53.697302+0000", 2024-09-21T11:06:53.610 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7670000000000003e-06 2024-09-21T11:06:53.610 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.610 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.610 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.610 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.771715+0000", 2024-09-21T11:06:53.611 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0744130030000001 2024-09-21T11:06:53.611 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.611 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.611 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:53.611 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.771822+0000", 2024-09-21T11:06:53.611 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000107037 2024-09-21T11:06:53.611 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.611 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.611 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,6,7", 2024-09-21T11:06:53.611 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.771931+0000", 2024-09-21T11:06:53.611 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000109453 2024-09-21T11:06:53.612 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.612 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.612 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.612 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.775730+0000", 2024-09-21T11:06:53.612 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0037990060000000002 2024-09-21T11:06:53.612 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.612 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.612 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.612 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.775762+0000", 2024-09-21T11:06:53.612 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1850000000000002e-05 2024-09-21T11:06:53.612 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.612 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.613 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:53.613 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.778687+0000", 2024-09-21T11:06:53.613 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00292524 2024-09-21T11:06:53.613 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.613 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.613 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.613 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.805109+0000", 2024-09-21T11:06:53.613 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.026422054 2024-09-21T11:06:53.613 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.613 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.613 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:53.613 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.805174+0000", 2024-09-21T11:06:53.614 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5160999999999995e-05 2024-09-21T11:06:53.614 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.614 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.614 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:53.614 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:54.805313+0000", 2024-09-21T11:06:53.614 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013882799999999999 2024-09-21T11:06:53.614 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.614 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:53.614 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.614 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.614 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.614 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:263994 58.f 58:f5724202:::benchmark_data_smithi070_396992_object118035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-09-21T11:06:53.615 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:23.379411+0000", 2024-09-21T11:06:53.615 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 90.135140050000004, 2024-09-21T11:06:53.615 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.060775971, 2024-09-21T11:06:53.615 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:53.615 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:53.615 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:06:53.615 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:53.615 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:53.615 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:53.615 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 263994 2024-09-21T11:06:53.615 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.615 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:23.379411+0000", 2024-09-21T11:06:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:23.379411+0000", 2024-09-21T11:06:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.616 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:53.617 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:23.379412+0000", 2024-09-21T11:06:53.617 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1500000000000003e-07 2024-09-21T11:06:53.617 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.617 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.617 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:53.617 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:23.379416+0000", 2024-09-21T11:06:53.617 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.4669999999999998e-06 2024-09-21T11:06:53.617 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.617 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.617 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:53.617 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:23.379427+0000", 2024-09-21T11:06:53.617 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0691e-05 2024-09-21T11:06:53.618 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.618 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.618 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:53.618 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:23.379434+0000", 2024-09-21T11:06:53.618 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8510000000000001e-06 2024-09-21T11:06:53.618 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.618 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.618 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.618 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:23.382494+0000", 2024-09-21T11:06:53.618 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0030605609999999998 2024-09-21T11:06:53.618 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.619 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.619 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:53.619 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:23.382624+0000", 2024-09-21T11:06:53.619 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012924 2024-09-21T11:06:53.619 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.619 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.619 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T11:06:53.619 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:23.382759+0000", 2024-09-21T11:06:53.619 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013505099999999999 2024-09-21T11:06:53.619 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.619 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.619 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:23.384193+0000", 2024-09-21T11:06:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0014338529999999999 2024-09-21T11:06:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:23.386633+0000", 2024-09-21T11:06:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002440593 2024-09-21T11:06:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:23.421375+0000", 2024-09-21T11:06:53.620 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.034741727 2024-09-21T11:06:53.621 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.621 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.621 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:53.621 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:23.421387+0000", 2024-09-21T11:06:53.621 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1939e-05 2024-09-21T11:06:53.621 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.621 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.621 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.621 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:24.440124+0000", 2024-09-21T11:06:53.621 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.01873719 2024-09-21T11:06:53.621 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.621 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.622 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:53.622 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:24.440187+0000", 2024-09-21T11:06:53.622 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2892999999999996e-05 2024-09-21T11:06:53.622 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.622 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:53.622 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.622 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.622 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.622 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:264046 58.f 58:f4ba5a17:::benchmark_data_smithi070_396992_object118087:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1624)", 2024-09-21T11:06:53.622 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:23.458118+0000", 2024-09-21T11:06:53.623 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 90.056432907000001, 2024-09-21T11:06:53.623 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99126260399999999, 2024-09-21T11:06:53.623 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:53.623 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:53.623 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:53.623 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:53.623 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:53.623 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:53.623 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 264046 2024-09-21T11:06:53.623 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.623 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:53.623 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:23.458118+0000", 2024-09-21T11:06:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:23.458118+0000", 2024-09-21T11:06:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:53.624 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:23.458119+0000", 2024-09-21T11:06:53.625 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2929999999999999e-06 2024-09-21T11:06:53.625 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.625 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.625 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:53.625 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:23.458124+0000", 2024-09-21T11:06:53.625 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2089999999999999e-06 2024-09-21T11:06:53.625 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.625 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.625 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:53.625 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:23.458135+0000", 2024-09-21T11:06:53.625 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0978000000000001e-05 2024-09-21T11:06:53.625 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.626 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.626 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:53.626 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:23.458142+0000", 2024-09-21T11:06:53.626 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0079999999999996e-06 2024-09-21T11:06:53.626 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.626 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.626 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.626 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:23.458242+0000", 2024-09-21T11:06:53.626 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000100661 2024-09-21T11:06:53.626 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.626 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.626 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:53.627 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:23.458250+0000", 2024-09-21T11:06:53.627 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9659999999999996e-06 2024-09-21T11:06:53.627 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.627 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.627 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.627 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:24.440574+0000", 2024-09-21T11:06:53.627 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.982323901 2024-09-21T11:06:53.627 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.627 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.627 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:53.627 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:24.440647+0000", 2024-09-21T11:06:53.627 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2650000000000004e-05 2024-09-21T11:06:53.628 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.628 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.628 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 6,7", 2024-09-21T11:06:53.628 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:24.440723+0000", 2024-09-21T11:06:53.628 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6521999999999997e-05 2024-09-21T11:06:53.628 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.628 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.628 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:53.628 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:24.446948+0000", 2024-09-21T11:06:53.628 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0062244869999999999 2024-09-21T11:06:53.628 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.629 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.629 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.629 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:24.446989+0000", 2024-09-21T11:06:53.629 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0995000000000003e-05 2024-09-21T11:06:53.629 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.629 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.629 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.629 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:24.449303+0000", 2024-09-21T11:06:53.629 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0023147350000000001 2024-09-21T11:06:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:24.449335+0000", 2024-09-21T11:06:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1930000000000001e-05 2024-09-21T11:06:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:24.449381+0000", 2024-09-21T11:06:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5269000000000003e-05 2024-09-21T11:06:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.630 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:278844 58.8 58:1f5bb5c6:::benchmark_data_smithi070_396992_object132885:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1630)", 2024-09-21T11:06:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:39.509241+0000", 2024-09-21T11:06:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 74.005310191000007, 2024-09-21T11:06:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0231375279999999, 2024-09-21T11:06:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 278844 2024-09-21T11:06:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.509241+0000", 2024-09-21T11:06:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.509241+0000", 2024-09-21T11:06:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.632 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:53.633 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.509242+0000", 2024-09-21T11:06:53.633 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2810000000000001e-06 2024-09-21T11:06:53.633 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.633 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.633 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:53.633 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.509247+0000", 2024-09-21T11:06:53.633 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9300000000000002e-06 2024-09-21T11:06:53.633 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.633 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.633 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:53.633 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.509259+0000", 2024-09-21T11:06:53.633 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1623999999999999e-05 2024-09-21T11:06:53.633 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.633 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.509267+0000", 2024-09-21T11:06:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2789999999999998e-06 2024-09-21T11:06:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.509331+0000", 2024-09-21T11:06:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3874000000000005e-05 2024-09-21T11:06:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.509339+0000", 2024-09-21T11:06:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5159999999999994e-06 2024-09-21T11:06:53.634 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.525085+0000", 2024-09-21T11:06:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.015745924 2024-09-21T11:06:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.525211+0000", 2024-09-21T11:06:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012532400000000001 2024-09-21T11:06:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,6", 2024-09-21T11:06:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.525295+0000", 2024-09-21T11:06:53.635 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.4971999999999999e-05 2024-09-21T11:06:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.527809+0000", 2024-09-21T11:06:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025139279999999999 2024-09-21T11:06:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.530043+0000", 2024-09-21T11:06:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002233914 2024-09-21T11:06:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.636 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.532302+0000", 2024-09-21T11:06:53.637 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0022582610000000001 2024-09-21T11:06:53.637 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.637 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.638 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:53.638 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.532340+0000", 2024-09-21T11:06:53.638 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8356999999999999e-05 2024-09-21T11:06:53.638 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.638 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.638 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:53.638 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.532378+0000", 2024-09-21T11:06:53.638 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8343999999999997e-05 2024-09-21T11:06:53.638 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.638 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:53.638 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:278848 58.8 58:159a3b74:::benchmark_data_smithi070_396992_object132889:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1630)", 2024-09-21T11:06:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:39.511132+0000", 2024-09-21T11:06:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 74.003418581999995, 2024-09-21T11:06:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.021464967, 2024-09-21T11:06:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:53.639 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 278848 2024-09-21T11:06:53.640 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.640 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:53.640 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.640 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:53.640 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.511132+0000", 2024-09-21T11:06:53.640 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.640 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.640 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.640 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:53.640 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.511132+0000", 2024-09-21T11:06:53.640 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.640 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.511134+0000", 2024-09-21T11:06:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1599999999999999e-06 2024-09-21T11:06:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.511138+0000", 2024-09-21T11:06:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5549999999999996e-06 2024-09-21T11:06:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.511151+0000", 2024-09-21T11:06:53.641 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2659e-05 2024-09-21T11:06:53.642 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.642 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.642 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:53.642 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.511159+0000", 2024-09-21T11:06:53.642 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8739999999999995e-06 2024-09-21T11:06:53.642 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.642 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.642 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.642 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.511218+0000", 2024-09-21T11:06:53.642 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9877000000000003e-05 2024-09-21T11:06:53.642 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.642 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.642 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.511227+0000", 2024-09-21T11:06:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.636e-06 2024-09-21T11:06:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.527847+0000", 2024-09-21T11:06:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.016620187 2024-09-21T11:06:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.527938+0000", 2024-09-21T11:06:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0897999999999996e-05 2024-09-21T11:06:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.643 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,6", 2024-09-21T11:06:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.528017+0000", 2024-09-21T11:06:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8301999999999997e-05 2024-09-21T11:06:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.532498+0000", 2024-09-21T11:06:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0044816730000000003 2024-09-21T11:06:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.532539+0000", 2024-09-21T11:06:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0581000000000003e-05 2024-09-21T11:06:53.644 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.532546+0000", 2024-09-21T11:06:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.469e-06 2024-09-21T11:06:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.532575+0000", 2024-09-21T11:06:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8756e-05 2024-09-21T11:06:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:53.645 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.532597+0000", 2024-09-21T11:06:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.234e-05 2024-09-21T11:06:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:278878 58.8 58:150966d6:::benchmark_data_smithi070_396992_object132919:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1630)", 2024-09-21T11:06:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:39.541489+0000", 2024-09-21T11:06:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 73.973062120999998, 2024-09-21T11:06:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99123509200000004, 2024-09-21T11:06:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:53.646 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 278878 2024-09-21T11:06:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.541489+0000", 2024-09-21T11:06:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.647 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.541489+0000", 2024-09-21T11:06:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.541490+0000", 2024-09-21T11:06:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2449999999999999e-06 2024-09-21T11:06:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.541494+0000", 2024-09-21T11:06:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3279999999999999e-06 2024-09-21T11:06:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.541506+0000", 2024-09-21T11:06:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1236000000000001e-05 2024-09-21T11:06:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.541511+0000", 2024-09-21T11:06:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4140000000000002e-06 2024-09-21T11:06:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.541570+0000", 2024-09-21T11:06:53.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8867999999999999e-05 2024-09-21T11:06:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.541581+0000", 2024-09-21T11:06:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.063e-05 2024-09-21T11:06:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.530092+0000", 2024-09-21T11:06:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.98851103500000004 2024-09-21T11:06:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:53.650 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.530221+0000", 2024-09-21T11:06:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012898799999999999 2024-09-21T11:06:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,6", 2024-09-21T11:06:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.530355+0000", 2024-09-21T11:06:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000134726 2024-09-21T11:06:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.532603+0000", 2024-09-21T11:06:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002247537 2024-09-21T11:06:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.532637+0000", 2024-09-21T11:06:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4323000000000003e-05 2024-09-21T11:06:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.532678+0000", 2024-09-21T11:06:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0735999999999998e-05 2024-09-21T11:06:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.532704+0000", 2024-09-21T11:06:53.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.635e-05 2024-09-21T11:06:53.653 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.653 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.653 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:53.653 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.532724+0000", 2024-09-21T11:06:53.653 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9675999999999999e-05 2024-09-21T11:06:53.653 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.653 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:53.653 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.653 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.653 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.653 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:278885 58.8 58:14609fa4:::benchmark_data_smithi070_396992_object132926:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1630)", 2024-09-21T11:06:53.654 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:39.547494+0000", 2024-09-21T11:06:53.654 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 73.967056634000002, 2024-09-21T11:06:53.654 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99576049700000002, 2024-09-21T11:06:53.654 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:53.654 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:53.654 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:53.654 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:53.654 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:53.654 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:53.654 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 278885 2024-09-21T11:06:53.654 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.654 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:53.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.655 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:53.655 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.547494+0000", 2024-09-21T11:06:53.655 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.655 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:53.655 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.547494+0000", 2024-09-21T11:06:53.655 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.547495+0000", 2024-09-21T11:06:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0610000000000001e-06 2024-09-21T11:06:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.547500+0000", 2024-09-21T11:06:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.4710000000000001e-06 2024-09-21T11:06:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.547512+0000", 2024-09-21T11:06:53.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2352e-05 2024-09-21T11:06:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.547519+0000", 2024-09-21T11:06:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7580000000000003e-06 2024-09-21T11:06:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.547573+0000", 2024-09-21T11:06:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.448e-05 2024-09-21T11:06:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.657 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:53.658 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:39.547582+0000", 2024-09-21T11:06:53.658 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.8249999999999994e-06 2024-09-21T11:06:53.658 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.658 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.658 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.658 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.532759+0000", 2024-09-21T11:06:53.658 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.98517706800000004 2024-09-21T11:06:53.658 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.658 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.658 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:53.658 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.532873+0000", 2024-09-21T11:06:53.658 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011383900000000001 2024-09-21T11:06:53.658 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.659 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 3,6", 2024-09-21T11:06:53.659 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.532991+0000", 2024-09-21T11:06:53.659 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000117855 2024-09-21T11:06:53.659 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.659 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.659 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.539035+0000", 2024-09-21T11:06:53.660 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0060434829999999997 2024-09-21T11:06:53.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.660 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.660 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.539123+0000", 2024-09-21T11:06:53.660 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.8321999999999999e-05 2024-09-21T11:06:53.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.660 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:53.660 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.543143+0000", 2024-09-21T11:06:53.661 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0040199190000000003 2024-09-21T11:06:53.661 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.661 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.661 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:53.661 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.543209+0000", 2024-09-21T11:06:53.661 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5974000000000002e-05 2024-09-21T11:06:53.661 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.661 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.661 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:53.661 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:40.543255+0000", 2024-09-21T11:06:53.662 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6090000000000001e-05 2024-09-21T11:06:53.662 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.662 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:53.662 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.662 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.662 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.662 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:284834 58.f 58:fdf181b3:::benchmark_data_smithi070_396992_object138875:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1642)", 2024-09-21T11:06:53.662 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:53.549048+0000", 2024-09-21T11:06:53.662 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 59.965502516000001, 2024-09-21T11:06:53.662 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99670904400000004, 2024-09-21T11:06:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:06:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:06:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:06:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:06:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:06:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:06:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 284834 2024-09-21T11:06:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:06:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:06:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:53.549048+0000", 2024-09-21T11:06:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:06:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:53.549048+0000", 2024-09-21T11:06:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:06:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:06:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:53.549050+0000", 2024-09-21T11:06:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.062e-06 2024-09-21T11:06:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:06:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:53.549055+0000", 2024-09-21T11:06:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0479999999999998e-06 2024-09-21T11:06:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:06:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:53.549066+0000", 2024-09-21T11:06:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1833e-05 2024-09-21T11:06:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:06:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:53.549074+0000", 2024-09-21T11:06:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0360000000000006e-06 2024-09-21T11:06:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:53.549141+0000", 2024-09-21T11:06:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6768999999999997e-05 2024-09-21T11:06:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:06:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:53.549149+0000", 2024-09-21T11:06:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9109999999999993e-06 2024-09-21T11:06:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:06:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:54.542607+0000", 2024-09-21T11:06:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99345767200000001 2024-09-21T11:06:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:06:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:54.542725+0000", 2024-09-21T11:06:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000118273 2024-09-21T11:06:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,6", 2024-09-21T11:06:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:54.542803+0000", 2024-09-21T11:06:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7778e-05 2024-09-21T11:06:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:54.545591+0000", 2024-09-21T11:06:53.668 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0027885219999999999 2024-09-21T11:06:53.668 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.668 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.668 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:06:53.668 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:54.545654+0000", 2024-09-21T11:06:53.668 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2562000000000002e-05 2024-09-21T11:06:53.668 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.668 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.668 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:06:53.668 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:54.545668+0000", 2024-09-21T11:06:53.668 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.361e-05 2024-09-21T11:06:53.668 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.668 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.669 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:06:53.669 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:54.545715+0000", 2024-09-21T11:06:53.669 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.7676000000000002e-05 2024-09-21T11:06:53.669 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:06:53.669 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:06:53.669 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:06:53.669 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:54.545757+0000", 2024-09-21T11:06:53.669 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2292000000000001e-05 2024-09-21T11:06:53.669 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.669 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:53.669 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.669 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:06:53.669 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:06:53.669 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T11:06:53.802 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:53 smithi070 ceph-mon[112139]: pgmap v3311: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:53.803 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:53 smithi070 ceph-mon[107385]: pgmap v3311: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:54.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:53 smithi160 ceph-mon[101879]: pgmap v3311: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:54.671 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T11:06:54.746 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T11:06:55.078 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:54 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3729989414' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T11:06:55.078 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:54 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3729989414' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T11:06:55.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:54 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3729989414' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T11:06:55.098 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T11:06:55.401 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:06:55.422 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:06:55.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:55 smithi070 ceph-mon[112139]: pgmap v3312: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:55.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:55 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3729989414' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T11:06:55.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:55 smithi070 ceph-mon[112139]: osdmap e1672: 8 total, 8 up, 6 in 2024-09-21T11:06:55.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:55 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 7]} : dispatch 2024-09-21T11:06:55.824 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:55 smithi070 ceph-mon[107385]: pgmap v3312: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:55.824 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:55 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3729989414' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T11:06:55.824 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:55 smithi070 ceph-mon[107385]: osdmap e1672: 8 total, 8 up, 6 in 2024-09-21T11:06:55.824 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:55 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 7]} : dispatch 2024-09-21T11:06:55.842 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 4 1 2024-09-21T11:06:56.010 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:55 smithi160 ceph-mon[101879]: pgmap v3312: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:56.010 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:55 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3729989414' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T11:06:56.010 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:55 smithi160 ceph-mon[101879]: osdmap e1672: 8 total, 8 up, 6 in 2024-09-21T11:06:56.010 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:55 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 7]} : dispatch 2024-09-21T11:06:57.083 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:56 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 7]}]': finished 2024-09-21T11:06:57.083 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:56 smithi070 ceph-mon[107385]: osdmap e1673: 8 total, 8 up, 6 in 2024-09-21T11:06:57.083 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:56 smithi070 ceph-mon[107385]: osdmap e1674: 8 total, 8 up, 6 in 2024-09-21T11:06:57.083 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:56 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 7]}]': finished 2024-09-21T11:06:57.084 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:56 smithi070 ceph-mon[112139]: osdmap e1673: 8 total, 8 up, 6 in 2024-09-21T11:06:57.084 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:56 smithi070 ceph-mon[112139]: osdmap e1674: 8 total, 8 up, 6 in 2024-09-21T11:06:57.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:56 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 7]}]': finished 2024-09-21T11:06:57.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:56 smithi160 ceph-mon[101879]: osdmap e1673: 8 total, 8 up, 6 in 2024-09-21T11:06:57.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:56 smithi160 ceph-mon[101879]: osdmap e1674: 8 total, 8 up, 6 in 2024-09-21T11:06:57.315 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:06:57.762 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:57 smithi070 ceph-mon[112139]: pgmap v3315: 93 pgs: 7 peering, 86 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:57.763 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:57 smithi070 ceph-mon[112139]: osdmap e1675: 8 total, 8 up, 6 in 2024-09-21T11:06:57.763 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:57 smithi070 ceph-mon[107385]: pgmap v3315: 93 pgs: 7 peering, 86 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:57.763 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:57 smithi070 ceph-mon[107385]: osdmap e1675: 8 total, 8 up, 6 in 2024-09-21T11:06:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:57 smithi160 ceph-mon[101879]: pgmap v3315: 93 pgs: 7 peering, 86 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:06:58.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:57 smithi160 ceph-mon[101879]: osdmap e1675: 8 total, 8 up, 6 in 2024-09-21T11:06:59.765 INFO:teuthology.orchestra.run.smithi070.stderr:nodeep-scrub is unset 2024-09-21T11:07:00.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:59 smithi160 ceph-mon[101879]: pgmap v3318: 93 pgs: 1 active+undersized+remapped, 8 peering, 84 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:00.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:59 smithi160 ceph-mon[101879]: osdmap e1676: 8 total, 8 up, 6 in 2024-09-21T11:07:00.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:59 smithi160 ceph-mon[101879]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T11:07:00.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:06:59 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/371072580' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:07:00.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:59 smithi070 ceph-mon[107385]: pgmap v3318: 93 pgs: 1 active+undersized+remapped, 8 peering, 84 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:00.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:59 smithi070 ceph-mon[107385]: osdmap e1676: 8 total, 8 up, 6 in 2024-09-21T11:07:00.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:59 smithi070 ceph-mon[107385]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T11:07:00.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:06:59 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/371072580' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:07:00.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:59 smithi070 ceph-mon[112139]: pgmap v3318: 93 pgs: 1 active+undersized+remapped, 8 peering, 84 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:00.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:59 smithi070 ceph-mon[112139]: osdmap e1676: 8 total, 8 up, 6 in 2024-09-21T11:07:00.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:59 smithi070 ceph-mon[112139]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T11:07:00.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:06:59 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/371072580' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T11:07:00.805 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:00 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/371072580' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T11:07:00.805 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:00 smithi070 ceph-mon[107385]: osdmap e1677: 8 total, 8 up, 6 in 2024-09-21T11:07:00.806 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:00 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/371072580' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T11:07:00.806 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:00 smithi070 ceph-mon[112139]: osdmap e1677: 8 total, 8 up, 6 in 2024-09-21T11:07:01.008 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T11:07:01.008 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T11:07:01.009 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T11:07:01.009 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T11:07:01.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:00 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/371072580' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T11:07:01.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:00 smithi160 ceph-mon[101879]: osdmap e1677: 8 total, 8 up, 6 in 2024-09-21T11:07:01.772 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T11:07:02.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:01 smithi160 ceph-mon[101879]: pgmap v3321: 93 pgs: 1 active+undersized+remapped, 1 peering, 91 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:02.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:01 smithi160 ceph-mon[101879]: 1.e deep-scrub starts 2024-09-21T11:07:02.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:01 smithi160 ceph-mon[101879]: 1.e deep-scrub ok 2024-09-21T11:07:02.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:01 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1144208255' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T11:07:02.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:01 smithi070 ceph-mon[107385]: pgmap v3321: 93 pgs: 1 active+undersized+remapped, 1 peering, 91 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:02.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:01 smithi070 ceph-mon[107385]: 1.e deep-scrub starts 2024-09-21T11:07:02.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:01 smithi070 ceph-mon[107385]: 1.e deep-scrub ok 2024-09-21T11:07:02.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:01 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1144208255' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T11:07:02.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:01 smithi070 ceph-mon[112139]: pgmap v3321: 93 pgs: 1 active+undersized+remapped, 1 peering, 91 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:02.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:01 smithi070 ceph-mon[112139]: 1.e deep-scrub starts 2024-09-21T11:07:02.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:01 smithi070 ceph-mon[112139]: 1.e deep-scrub ok 2024-09-21T11:07:02.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:01 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1144208255' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T11:07:02.327 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T11:07:03.009 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:02 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1144208255' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T11:07:03.010 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:02 smithi070 ceph-mon[107385]: osdmap e1678: 8 total, 8 up, 6 in 2024-09-21T11:07:03.010 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:02 smithi070 ceph-mon[107385]: pgmap v3323: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T11:07:03.010 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:02 smithi070 ceph-mon[107385]: osdmap e1679: 8 total, 8 up, 6 in 2024-09-21T11:07:03.010 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:02 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1144208255' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T11:07:03.010 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:02 smithi070 ceph-mon[112139]: osdmap e1678: 8 total, 8 up, 6 in 2024-09-21T11:07:03.010 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:02 smithi070 ceph-mon[112139]: pgmap v3323: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T11:07:03.010 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:02 smithi070 ceph-mon[112139]: osdmap e1679: 8 total, 8 up, 6 in 2024-09-21T11:07:03.029 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T11:07:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:02 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1144208255' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T11:07:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:02 smithi160 ceph-mon[101879]: osdmap e1678: 8 total, 8 up, 6 in 2024-09-21T11:07:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:02 smithi160 ceph-mon[101879]: pgmap v3323: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T11:07:03.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:02 smithi160 ceph-mon[101879]: osdmap e1679: 8 total, 8 up, 6 in 2024-09-21T11:07:03.142 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:07:03.263 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:07:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:07:03] "GET /metrics HTTP/1.1" 200 34283 "" "Prometheus/2.51.0" 2024-09-21T11:07:03.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:03 smithi070 ceph-mon[107385]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T11:07:03.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:03 smithi070 ceph-mon[107385]: Cluster is now healthy 2024-09-21T11:07:03.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:03 smithi070 ceph-mon[107385]: osdmap e1680: 8 total, 8 up, 6 in 2024-09-21T11:07:03.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:03 smithi070 ceph-mon[112139]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T11:07:03.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:03 smithi070 ceph-mon[112139]: Cluster is now healthy 2024-09-21T11:07:03.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:03 smithi070 ceph-mon[112139]: osdmap e1680: 8 total, 8 up, 6 in 2024-09-21T11:07:03.937 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:07:04.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:03 smithi160 ceph-mon[101879]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T11:07:04.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:03 smithi160 ceph-mon[101879]: Cluster is now healthy 2024-09-21T11:07:04.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:03 smithi160 ceph-mon[101879]: osdmap e1680: 8 total, 8 up, 6 in 2024-09-21T11:07:04.899 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:04 smithi070 ceph-mon[112139]: pgmap v3326: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T11:07:04.900 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:04 smithi070 ceph-mon[112139]: osdmap e1681: 8 total, 8 up, 6 in 2024-09-21T11:07:04.900 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:04 smithi070 ceph-mon[107385]: pgmap v3326: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T11:07:04.900 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:04 smithi070 ceph-mon[107385]: osdmap e1681: 8 total, 8 up, 6 in 2024-09-21T11:07:05.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:04 smithi160 ceph-mon[101879]: pgmap v3326: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T11:07:05.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:04 smithi160 ceph-mon[101879]: osdmap e1681: 8 total, 8 up, 6 in 2024-09-21T11:07:05.632 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T11:07:05.632 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T11:07:05.632 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T11:07:05.632 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T11:07:05.633 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T11:07:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:07:06.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:05 smithi160 ceph-mon[101879]: osdmap e1682: 8 total, 8 up, 6 in 2024-09-21T11:07:06.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:07:06.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:05 smithi070 ceph-mon[107385]: osdmap e1682: 8 total, 8 up, 6 in 2024-09-21T11:07:06.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:07:06.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:05 smithi070 ceph-mon[112139]: osdmap e1682: 8 total, 8 up, 6 in 2024-09-21T11:07:06.678 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T11:07:06.959 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:06 smithi070 ceph-mon[107385]: pgmap v3329: 93 pgs: 1 peering, 92 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:06.960 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:06 smithi070 ceph-mon[112139]: pgmap v3329: 93 pgs: 1 peering, 92 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:07.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:06 smithi160 ceph-mon[101879]: pgmap v3329: 93 pgs: 1 peering, 92 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:07.217 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:07:07.805 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T11:07:08.078 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:07 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4108889141' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T11:07:08.079 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:07 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T11:07:08.079 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:07 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4108889141' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T11:07:08.079 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:07 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T11:07:08.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:07 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4108889141' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T11:07:08.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:07 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T11:07:09.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:08 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T11:07:09.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:08 smithi160 ceph-mon[101879]: osdmap e1683: 8 total, 8 up, 6 in 2024-09-21T11:07:09.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:08 smithi160 ceph-mon[101879]: 1.e scrub starts 2024-09-21T11:07:09.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:08 smithi160 ceph-mon[101879]: 1.e scrub ok 2024-09-21T11:07:09.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:08 smithi160 ceph-mon[101879]: pgmap v3331: 93 pgs: 1 peering, 92 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T11:07:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:08 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T11:07:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:08 smithi070 ceph-mon[107385]: osdmap e1683: 8 total, 8 up, 6 in 2024-09-21T11:07:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:08 smithi070 ceph-mon[107385]: 1.e scrub starts 2024-09-21T11:07:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:08 smithi070 ceph-mon[107385]: 1.e scrub ok 2024-09-21T11:07:09.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:08 smithi070 ceph-mon[107385]: pgmap v3331: 93 pgs: 1 peering, 92 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T11:07:09.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:08 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T11:07:09.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:08 smithi070 ceph-mon[112139]: osdmap e1683: 8 total, 8 up, 6 in 2024-09-21T11:07:09.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:08 smithi070 ceph-mon[112139]: 1.e scrub starts 2024-09-21T11:07:09.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:08 smithi070 ceph-mon[112139]: 1.e scrub ok 2024-09-21T11:07:09.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:08 smithi070 ceph-mon[112139]: pgmap v3331: 93 pgs: 1 peering, 92 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T11:07:09.477 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T11:07:09.477 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T11:07:09.988 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T11:07:09.988 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T11:07:09.989 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T11:07:09.989 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [] 2024-09-21T11:07:09.989 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T11:07:10.263 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:07:10.697 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T11:07:11.474 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:11 smithi070 ceph-mon[107385]: pgmap v3332: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T11:07:11.514 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:11 smithi070 ceph-mon[112139]: pgmap v3332: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T11:07:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:11 smithi160 ceph-mon[101879]: pgmap v3332: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T11:07:12.449 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:07:12.589 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T11:07:12.590 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T11:07:13.391 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:07:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:07:13] "GET /metrics HTTP/1.1" 200 34258 "" "Prometheus/2.51.0" 2024-09-21T11:07:13.412 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":3333,"stamp":"2024-09-21T11:07:12.362592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":386923,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":83,"num_write_kb":252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":1665282,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3188,"ondisk_log_size":3188,"up":279,"acting":279,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":279,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":6280168,"kb_used_data":62928,"kb_used_omap":397,"kb_used_meta":6216754,"kb_avail":556190744,"statfs":{"total":575970213888,"available":569539321856,"internally_reserved":0,"allocated":64438272,"data_stored":36357855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":406920,"internal_metadata":6365956728},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.588446"},"pg_stats":[{"pgid":"3.1f","version":"71'4","reported_seq":110,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822222+0000","last_change":"2024-09-21T11:05:48.544347+0000","last_active":"2024-09-21T11:07:07.822222+0000","last_peered":"2024-09-21T11:07:07.822222+0000","last_clean":"2024-09-21T11:07:07.822222+0000","last_became_active":"2024-09-21T11:05:48.529869+0000","last_became_peered":"2024-09-21T11:05:48.529869+0000","last_unstale":"2024-09-21T11:07:07.822222+0000","last_undegraded":"2024-09-21T11:07:07.822222+0000","last_fullsized":"2024-09-21T11:07:07.822222+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:18:15.635403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1601,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.781019+0000","last_change":"2024-09-21T11:02:39.486670+0000","last_active":"2024-09-21T11:07:05.781019+0000","last_peered":"2024-09-21T11:07:05.781019+0000","last_clean":"2024-09-21T11:07:05.781019+0000","last_became_active":"2024-09-21T11:02:39.486248+0000","last_became_peered":"2024-09-21T11:02:39.486248+0000","last_unstale":"2024-09-21T11:07:05.781019+0000","last_undegraded":"2024-09-21T11:07:05.781019+0000","last_fullsized":"2024-09-21T11:07:05.781019+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:56:56.360708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":147,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.823268+0000","last_change":"2024-09-21T11:04:54.654602+0000","last_active":"2024-09-21T11:07:07.823268+0000","last_peered":"2024-09-21T11:07:07.823268+0000","last_clean":"2024-09-21T11:07:07.823268+0000","last_became_active":"2024-09-21T11:04:54.654231+0000","last_became_peered":"2024-09-21T11:04:54.654231+0000","last_unstale":"2024-09-21T11:07:07.823268+0000","last_undegraded":"2024-09-21T11:07:07.823268+0000","last_fullsized":"2024-09-21T11:07:07.823268+0000","mapping_epoch":1615,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:53:49.828685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":153,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822383+0000","last_change":"2024-09-21T11:05:48.533874+0000","last_active":"2024-09-21T11:07:07.822383+0000","last_peered":"2024-09-21T11:07:07.822383+0000","last_clean":"2024-09-21T11:07:07.822383+0000","last_became_active":"2024-09-21T11:05:48.533558+0000","last_became_peered":"2024-09-21T11:05:48.533558+0000","last_unstale":"2024-09-21T11:07:07.822383+0000","last_undegraded":"2024-09-21T11:07:07.822383+0000","last_fullsized":"2024-09-21T11:07:07.822383+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:01:28.336817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":179,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.781948+0000","last_change":"2024-09-21T11:02:39.502513+0000","last_active":"2024-09-21T11:07:05.781948+0000","last_peered":"2024-09-21T11:07:05.781948+0000","last_clean":"2024-09-21T11:07:05.781948+0000","last_became_active":"2024-09-21T11:02:39.502095+0000","last_became_peered":"2024-09-21T11:02:39.502095+0000","last_unstale":"2024-09-21T11:07:05.781948+0000","last_undegraded":"2024-09-21T11:07:05.781948+0000","last_fullsized":"2024-09-21T11:07:05.781948+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:50:46.078094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":15,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:06.039504+0000","last_change":"2024-09-21T11:07:02.780603+0000","last_active":"2024-09-21T11:07:06.039504+0000","last_peered":"2024-09-21T11:07:06.039504+0000","last_clean":"2024-09-21T11:07:06.039504+0000","last_became_active":"2024-09-21T11:07:02.780260+0000","last_became_peered":"2024-09-21T11:07:02.780260+0000","last_unstale":"2024-09-21T11:07:06.039504+0000","last_undegraded":"2024-09-21T11:07:06.039504+0000","last_fullsized":"2024-09-21T11:07:06.039504+0000","mapping_epoch":1678,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:50.715390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":179,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.781457+0000","last_change":"2024-09-21T11:02:39.502786+0000","last_active":"2024-09-21T11:07:05.781457+0000","last_peered":"2024-09-21T11:07:05.781457+0000","last_clean":"2024-09-21T11:07:05.781457+0000","last_became_active":"2024-09-21T11:02:39.498657+0000","last_became_peered":"2024-09-21T11:02:39.498657+0000","last_unstale":"2024-09-21T11:07:05.781457+0000","last_undegraded":"2024-09-21T11:07:05.781457+0000","last_fullsized":"2024-09-21T11:07:05.781457+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:30:40.747863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":15,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:06.039529+0000","last_change":"2024-09-21T11:07:02.780932+0000","last_active":"2024-09-21T11:07:06.039529+0000","last_peered":"2024-09-21T11:07:06.039529+0000","last_clean":"2024-09-21T11:07:06.039529+0000","last_became_active":"2024-09-21T11:07:02.780451+0000","last_became_peered":"2024-09-21T11:07:02.780451+0000","last_unstale":"2024-09-21T11:07:06.039529+0000","last_undegraded":"2024-09-21T11:07:06.039529+0000","last_fullsized":"2024-09-21T11:07:06.039529+0000","mapping_epoch":1678,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:59:32.916443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":104,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822731+0000","last_change":"2024-09-21T11:05:48.622455+0000","last_active":"2024-09-21T11:07:07.822731+0000","last_peered":"2024-09-21T11:07:07.822731+0000","last_clean":"2024-09-21T11:07:07.822731+0000","last_became_active":"2024-09-21T11:05:48.559270+0000","last_became_peered":"2024-09-21T11:05:48.559270+0000","last_unstale":"2024-09-21T11:07:07.822731+0000","last_undegraded":"2024-09-21T11:07:07.822731+0000","last_fullsized":"2024-09-21T11:07:07.822731+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:37:01.281922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":153,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.823056+0000","last_change":"2024-09-21T11:05:48.527392+0000","last_active":"2024-09-21T11:07:07.823056+0000","last_peered":"2024-09-21T11:07:07.823056+0000","last_clean":"2024-09-21T11:07:07.823056+0000","last_became_active":"2024-09-21T11:05:48.527078+0000","last_became_peered":"2024-09-21T11:05:48.527078+0000","last_unstale":"2024-09-21T11:07:07.823056+0000","last_undegraded":"2024-09-21T11:07:07.823056+0000","last_fullsized":"2024-09-21T11:07:07.823056+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:58:33.681474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":186,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.781726+0000","last_change":"2024-09-21T11:05:48.614724+0000","last_active":"2024-09-21T11:07:05.781726+0000","last_peered":"2024-09-21T11:07:05.781726+0000","last_clean":"2024-09-21T11:07:05.781726+0000","last_became_active":"2024-09-21T11:05:48.614414+0000","last_became_peered":"2024-09-21T11:05:48.614414+0000","last_unstale":"2024-09-21T11:07:05.781726+0000","last_undegraded":"2024-09-21T11:07:05.781726+0000","last_fullsized":"2024-09-21T11:07:05.781726+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:14:09.665271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":156,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822108+0000","last_change":"2024-09-21T11:04:47.619509+0000","last_active":"2024-09-21T11:07:07.822108+0000","last_peered":"2024-09-21T11:07:07.822108+0000","last_clean":"2024-09-21T11:07:07.822108+0000","last_became_active":"2024-09-21T11:04:47.618898+0000","last_became_peered":"2024-09-21T11:04:47.618898+0000","last_unstale":"2024-09-21T11:07:07.822108+0000","last_undegraded":"2024-09-21T11:07:07.822108+0000","last_fullsized":"2024-09-21T11:07:07.822108+0000","mapping_epoch":1610,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1611,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:28:19.899682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2690,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822791+0000","last_change":"2024-09-21T11:07:02.781730+0000","last_active":"2024-09-21T11:07:07.822791+0000","last_peered":"2024-09-21T11:07:07.822791+0000","last_clean":"2024-09-21T11:07:07.822791+0000","last_became_active":"2024-09-21T11:07:02.781410+0000","last_became_peered":"2024-09-21T11:07:02.781410+0000","last_unstale":"2024-09-21T11:07:07.822791+0000","last_undegraded":"2024-09-21T11:07:07.822791+0000","last_fullsized":"2024-09-21T11:07:07.822791+0000","mapping_epoch":1678,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:18:26.326185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":118,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822743+0000","last_change":"2024-09-21T11:06:55.763204+0000","last_active":"2024-09-21T11:07:07.822743+0000","last_peered":"2024-09-21T11:07:07.822743+0000","last_clean":"2024-09-21T11:07:07.822743+0000","last_became_active":"2024-09-21T11:06:55.762743+0000","last_became_peered":"2024-09-21T11:06:55.762743+0000","last_unstale":"2024-09-21T11:07:07.822743+0000","last_undegraded":"2024-09-21T11:07:07.822743+0000","last_fullsized":"2024-09-21T11:07:07.822743+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:25:04.804191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":170,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822016+0000","last_change":"2024-09-21T11:06:33.533570+0000","last_active":"2024-09-21T11:07:07.822016+0000","last_peered":"2024-09-21T11:07:07.822016+0000","last_clean":"2024-09-21T11:07:07.822016+0000","last_became_active":"2024-09-21T11:06:33.533236+0000","last_became_peered":"2024-09-21T11:06:33.533236+0000","last_unstale":"2024-09-21T11:07:07.822016+0000","last_undegraded":"2024-09-21T11:07:07.822016+0000","last_fullsized":"2024-09-21T11:07:07.822016+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:50:25.103638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1647,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.778865+0000","last_change":"2024-09-21T11:05:48.530856+0000","last_active":"2024-09-21T11:07:05.778865+0000","last_peered":"2024-09-21T11:07:05.778865+0000","last_clean":"2024-09-21T11:07:05.778865+0000","last_became_active":"2024-09-21T11:05:48.530496+0000","last_became_peered":"2024-09-21T11:05:48.530496+0000","last_unstale":"2024-09-21T11:07:05.778865+0000","last_undegraded":"2024-09-21T11:07:05.778865+0000","last_fullsized":"2024-09-21T11:07:05.778865+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:15.620293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":169,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822103+0000","last_change":"2024-09-21T11:04:34.523861+0000","last_active":"2024-09-21T11:07:07.822103+0000","last_peered":"2024-09-21T11:07:07.822103+0000","last_clean":"2024-09-21T11:07:07.822103+0000","last_became_active":"2024-09-21T11:04:34.523542+0000","last_became_peered":"2024-09-21T11:04:34.523542+0000","last_unstale":"2024-09-21T11:07:07.822103+0000","last_undegraded":"2024-09-21T11:07:07.822103+0000","last_fullsized":"2024-09-21T11:07:07.822103+0000","mapping_epoch":1604,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:35.990638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":104,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822123+0000","last_change":"2024-09-21T11:05:48.528637+0000","last_active":"2024-09-21T11:07:07.822123+0000","last_peered":"2024-09-21T11:07:07.822123+0000","last_clean":"2024-09-21T11:07:07.822123+0000","last_became_active":"2024-09-21T11:05:48.528215+0000","last_became_peered":"2024-09-21T11:05:48.528215+0000","last_unstale":"2024-09-21T11:07:07.822123+0000","last_undegraded":"2024-09-21T11:07:07.822123+0000","last_fullsized":"2024-09-21T11:07:07.822123+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:43:21.451145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"346'149","reported_seq":1157,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.783470+0000","last_change":"2024-09-21T11:07:05.783174+0000","last_active":"2024-09-21T11:07:05.783470+0000","last_peered":"2024-09-21T11:07:05.783470+0000","last_clean":"2024-09-21T11:07:05.783470+0000","last_became_active":"2024-09-21T11:07:05.782768+0000","last_became_peered":"2024-09-21T11:07:05.782768+0000","last_unstale":"2024-09-21T11:07:05.783470+0000","last_undegraded":"2024-09-21T11:07:05.783470+0000","last_fullsized":"2024-09-21T11:07:05.783470+0000","mapping_epoch":1681,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1682,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:53.724544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:53.724544+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:21:35.685593+0000","scrub_duration":0.0017779029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2714,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822640+0000","last_change":"2024-09-21T11:06:55.761046+0000","last_active":"2024-09-21T11:07:07.822640+0000","last_peered":"2024-09-21T11:07:07.822640+0000","last_clean":"2024-09-21T11:07:07.822640+0000","last_became_active":"2024-09-21T11:06:55.760680+0000","last_became_peered":"2024-09-21T11:06:55.760680+0000","last_unstale":"2024-09-21T11:07:07.822640+0000","last_undegraded":"2024-09-21T11:07:07.822640+0000","last_fullsized":"2024-09-21T11:07:07.822640+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:44:27.436994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.780728+0000","last_change":"2024-09-21T11:06:55.761083+0000","last_active":"2024-09-21T11:07:05.780728+0000","last_peered":"2024-09-21T11:07:05.780728+0000","last_clean":"2024-09-21T11:07:05.780728+0000","last_became_active":"2024-09-21T11:06:55.760730+0000","last_became_peered":"2024-09-21T11:06:55.760730+0000","last_unstale":"2024-09-21T11:07:05.780728+0000","last_undegraded":"2024-09-21T11:07:05.780728+0000","last_fullsized":"2024-09-21T11:07:05.780728+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:59:21.731681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":27,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.779867+0000","last_change":"2024-09-21T11:06:55.768569+0000","last_active":"2024-09-21T11:07:05.779867+0000","last_peered":"2024-09-21T11:07:05.779867+0000","last_clean":"2024-09-21T11:07:05.779867+0000","last_became_active":"2024-09-21T11:06:55.761500+0000","last_became_peered":"2024-09-21T11:06:55.761500+0000","last_unstale":"2024-09-21T11:07:05.779867+0000","last_undegraded":"2024-09-21T11:07:05.779867+0000","last_fullsized":"2024-09-21T11:07:05.779867+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:55:27.336715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":60,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.823345+0000","last_change":"2024-09-21T11:06:33.540524+0000","last_active":"2024-09-21T11:07:07.823345+0000","last_peered":"2024-09-21T11:07:07.823345+0000","last_clean":"2024-09-21T11:07:07.823345+0000","last_became_active":"2024-09-21T11:06:33.540173+0000","last_became_peered":"2024-09-21T11:06:33.540173+0000","last_unstale":"2024-09-21T11:07:07.823345+0000","last_undegraded":"2024-09-21T11:07:07.823345+0000","last_fullsized":"2024-09-21T11:07:07.823345+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:44:03.323025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":185,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.781240+0000","last_change":"2024-09-21T11:02:39.505094+0000","last_active":"2024-09-21T11:07:05.781240+0000","last_peered":"2024-09-21T11:07:05.781240+0000","last_clean":"2024-09-21T11:07:05.781240+0000","last_became_active":"2024-09-21T11:02:39.503527+0000","last_became_peered":"2024-09-21T11:02:39.503527+0000","last_unstale":"2024-09-21T11:07:05.781240+0000","last_undegraded":"2024-09-21T11:07:05.781240+0000","last_fullsized":"2024-09-21T11:07:05.781240+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:12:34.916393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14268,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":754,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.781335+0000","last_change":"2024-09-21T11:02:39.492530+0000","last_active":"2024-09-21T11:07:05.781335+0000","last_peered":"2024-09-21T11:07:05.781335+0000","last_clean":"2024-09-21T11:07:05.781335+0000","last_became_active":"2024-09-21T11:02:39.491835+0000","last_became_peered":"2024-09-21T11:02:39.491835+0000","last_unstale":"2024-09-21T11:07:05.781335+0000","last_undegraded":"2024-09-21T11:07:05.781335+0000","last_fullsized":"2024-09-21T11:07:05.781335+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:37.157843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"346'149","reported_seq":959,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822892+0000","last_change":"2024-09-21T11:06:58.762749+0000","last_active":"2024-09-21T11:07:07.822892+0000","last_peered":"2024-09-21T11:07:07.822892+0000","last_clean":"2024-09-21T11:07:07.822892+0000","last_became_active":"2024-09-21T11:06:58.762458+0000","last_became_peered":"2024-09-21T11:06:58.762458+0000","last_unstale":"2024-09-21T11:07:07.822892+0000","last_undegraded":"2024-09-21T11:07:07.822892+0000","last_fullsized":"2024-09-21T11:07:07.822892+0000","mapping_epoch":1675,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1676,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:59.781270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:59.781270+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:28:38.465822+0000","scrub_duration":0.0031203189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1621,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.781999+0000","last_change":"2024-09-21T11:05:48.525548+0000","last_active":"2024-09-21T11:07:05.781999+0000","last_peered":"2024-09-21T11:07:05.781999+0000","last_clean":"2024-09-21T11:07:05.781999+0000","last_became_active":"2024-09-21T11:05:48.525247+0000","last_became_peered":"2024-09-21T11:05:48.525247+0000","last_unstale":"2024-09-21T11:07:05.781999+0000","last_undegraded":"2024-09-21T11:07:05.781999+0000","last_fullsized":"2024-09-21T11:07:05.781999+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:51:18.193438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1695,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.778889+0000","last_change":"2024-09-21T11:06:55.761287+0000","last_active":"2024-09-21T11:07:05.778889+0000","last_peered":"2024-09-21T11:07:05.778889+0000","last_clean":"2024-09-21T11:07:05.778889+0000","last_became_active":"2024-09-21T11:06:55.760836+0000","last_became_peered":"2024-09-21T11:06:55.760836+0000","last_unstale":"2024-09-21T11:07:05.778889+0000","last_undegraded":"2024-09-21T11:07:05.778889+0000","last_fullsized":"2024-09-21T11:07:05.778889+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:07:51.834315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":108,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822900+0000","last_change":"2024-09-21T11:05:48.624344+0000","last_active":"2024-09-21T11:07:07.822900+0000","last_peered":"2024-09-21T11:07:07.822900+0000","last_clean":"2024-09-21T11:07:07.822900+0000","last_became_active":"2024-09-21T11:05:48.564758+0000","last_became_peered":"2024-09-21T11:05:48.564758+0000","last_unstale":"2024-09-21T11:07:07.822900+0000","last_undegraded":"2024-09-21T11:07:07.822900+0000","last_fullsized":"2024-09-21T11:07:07.822900+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:32.746530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"346'149","reported_seq":869,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.782053+0000","last_change":"2024-09-21T11:05:51.520639+0000","last_active":"2024-09-21T11:07:05.782053+0000","last_peered":"2024-09-21T11:07:05.782053+0000","last_clean":"2024-09-21T11:07:05.782053+0000","last_became_active":"2024-09-21T11:05:51.519581+0000","last_became_peered":"2024-09-21T11:05:51.519581+0000","last_unstale":"2024-09-21T11:07:05.782053+0000","last_undegraded":"2024-09-21T11:07:05.782053+0000","last_fullsized":"2024-09-21T11:07:05.782053+0000","mapping_epoch":1639,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1640,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:54.879276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:54.879276+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:53.844022+0000","scrub_duration":0.001843014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":53,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.821787+0000","last_change":"2024-09-21T11:06:55.759358+0000","last_active":"2024-09-21T11:07:07.821787+0000","last_peered":"2024-09-21T11:07:07.821787+0000","last_clean":"2024-09-21T11:07:07.821787+0000","last_became_active":"2024-09-21T11:06:55.759040+0000","last_became_peered":"2024-09-21T11:06:55.759040+0000","last_unstale":"2024-09-21T11:07:07.821787+0000","last_undegraded":"2024-09-21T11:07:07.821787+0000","last_fullsized":"2024-09-21T11:07:07.821787+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:59:39.297517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":60,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.823519+0000","last_change":"2024-09-21T11:06:33.532315+0000","last_active":"2024-09-21T11:07:07.823519+0000","last_peered":"2024-09-21T11:07:07.823519+0000","last_clean":"2024-09-21T11:07:07.823519+0000","last_became_active":"2024-09-21T11:06:33.532024+0000","last_became_peered":"2024-09-21T11:06:33.532024+0000","last_unstale":"2024-09-21T11:07:07.823519+0000","last_undegraded":"2024-09-21T11:07:07.823519+0000","last_fullsized":"2024-09-21T11:07:07.823519+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:35:36.295788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":2777,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822699+0000","last_change":"2024-09-21T11:06:55.766194+0000","last_active":"2024-09-21T11:07:07.822699+0000","last_peered":"2024-09-21T11:07:07.822699+0000","last_clean":"2024-09-21T11:07:07.822699+0000","last_became_active":"2024-09-21T11:06:55.761357+0000","last_became_peered":"2024-09-21T11:06:55.761357+0000","last_unstale":"2024-09-21T11:07:07.822699+0000","last_undegraded":"2024-09-21T11:07:07.822699+0000","last_fullsized":"2024-09-21T11:07:07.822699+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:19:18.903358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"346'149","reported_seq":994,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.779085+0000","last_change":"2024-09-21T11:06:43.715659+0000","last_active":"2024-09-21T11:07:05.779085+0000","last_peered":"2024-09-21T11:07:05.779085+0000","last_clean":"2024-09-21T11:07:05.779085+0000","last_became_active":"2024-09-21T11:06:43.715321+0000","last_became_peered":"2024-09-21T11:06:43.715321+0000","last_unstale":"2024-09-21T11:07:05.779085+0000","last_undegraded":"2024-09-21T11:07:05.779085+0000","last_fullsized":"2024-09-21T11:07:05.779085+0000","mapping_epoch":1668,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1669,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:06:00.911909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:06:00.911909+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:15:40.470846+0000","scrub_duration":0.002144838,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":104,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822563+0000","last_change":"2024-09-21T11:05:48.537936+0000","last_active":"2024-09-21T11:07:07.822563+0000","last_peered":"2024-09-21T11:07:07.822563+0000","last_clean":"2024-09-21T11:07:07.822563+0000","last_became_active":"2024-09-21T11:05:48.537615+0000","last_became_peered":"2024-09-21T11:05:48.537615+0000","last_unstale":"2024-09-21T11:07:07.822563+0000","last_undegraded":"2024-09-21T11:07:07.822563+0000","last_fullsized":"2024-09-21T11:07:07.822563+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:03:45.353800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":282,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:06.374164+0000","last_change":"2024-09-21T11:05:48.528383+0000","last_active":"2024-09-21T11:07:06.374164+0000","last_peered":"2024-09-21T11:07:06.374164+0000","last_clean":"2024-09-21T11:07:06.374164+0000","last_became_active":"2024-09-21T11:05:48.527903+0000","last_became_peered":"2024-09-21T11:05:48.527903+0000","last_unstale":"2024-09-21T11:07:06.374164+0000","last_undegraded":"2024-09-21T11:07:06.374164+0000","last_fullsized":"2024-09-21T11:07:06.374164+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:59:25.065009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":104,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.821948+0000","last_change":"2024-09-21T11:05:48.531015+0000","last_active":"2024-09-21T11:07:07.821948+0000","last_peered":"2024-09-21T11:07:07.821948+0000","last_clean":"2024-09-21T11:07:07.821948+0000","last_became_active":"2024-09-21T11:05:48.530729+0000","last_became_peered":"2024-09-21T11:05:48.530729+0000","last_unstale":"2024-09-21T11:07:07.821948+0000","last_undegraded":"2024-09-21T11:07:07.821948+0000","last_fullsized":"2024-09-21T11:07:07.821948+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:26.948930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"346'149","reported_seq":1032,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.778954+0000","last_change":"2024-09-21T11:06:58.762593+0000","last_active":"2024-09-21T11:07:05.778954+0000","last_peered":"2024-09-21T11:07:05.778954+0000","last_clean":"2024-09-21T11:07:05.778954+0000","last_became_active":"2024-09-21T11:06:58.762117+0000","last_became_peered":"2024-09-21T11:06:58.762117+0000","last_unstale":"2024-09-21T11:07:05.778954+0000","last_undegraded":"2024-09-21T11:07:05.778954+0000","last_fullsized":"2024-09-21T11:07:05.778954+0000","mapping_epoch":1675,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1676,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:05:57.002981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:05:57.002981+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:58:47.258964+0000","scrub_duration":0.0017979249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":67,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.823533+0000","last_change":"2024-09-21T11:07:02.779315+0000","last_active":"2024-09-21T11:07:07.823533+0000","last_peered":"2024-09-21T11:07:07.823533+0000","last_clean":"2024-09-21T11:07:07.823533+0000","last_became_active":"2024-09-21T11:07:02.778969+0000","last_became_peered":"2024-09-21T11:07:02.778969+0000","last_unstale":"2024-09-21T11:07:07.823533+0000","last_undegraded":"2024-09-21T11:07:07.823533+0000","last_fullsized":"2024-09-21T11:07:07.823533+0000","mapping_epoch":1678,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:56.329739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":104,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822041+0000","last_change":"2024-09-21T11:05:48.531055+0000","last_active":"2024-09-21T11:07:07.822041+0000","last_peered":"2024-09-21T11:07:07.822041+0000","last_clean":"2024-09-21T11:07:07.822041+0000","last_became_active":"2024-09-21T11:05:48.530743+0000","last_became_peered":"2024-09-21T11:05:48.530743+0000","last_unstale":"2024-09-21T11:07:07.822041+0000","last_undegraded":"2024-09-21T11:07:07.822041+0000","last_fullsized":"2024-09-21T11:07:07.822041+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:09:04.625920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":166,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822027+0000","last_change":"2024-09-21T11:06:55.759946+0000","last_active":"2024-09-21T11:07:07.822027+0000","last_peered":"2024-09-21T11:07:07.822027+0000","last_clean":"2024-09-21T11:07:07.822027+0000","last_became_active":"2024-09-21T11:06:55.759640+0000","last_became_peered":"2024-09-21T11:06:55.759640+0000","last_unstale":"2024-09-21T11:07:07.822027+0000","last_undegraded":"2024-09-21T11:07:07.822027+0000","last_fullsized":"2024-09-21T11:07:07.822027+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:32.254167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"346'149","reported_seq":1042,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822593+0000","last_change":"2024-09-21T11:05:51.518615+0000","last_active":"2024-09-21T11:07:07.822593+0000","last_peered":"2024-09-21T11:07:07.822593+0000","last_clean":"2024-09-21T11:07:07.822593+0000","last_became_active":"2024-09-21T11:05:51.518248+0000","last_became_peered":"2024-09-21T11:05:51.518248+0000","last_unstale":"2024-09-21T11:07:07.822593+0000","last_undegraded":"2024-09-21T11:07:07.822593+0000","last_fullsized":"2024-09-21T11:07:07.822593+0000","mapping_epoch":1639,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1640,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:48.918874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.918874+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:46.326025+0000","scrub_duration":0.001629134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":158,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822978+0000","last_change":"2024-09-21T11:06:33.535421+0000","last_active":"2024-09-21T11:07:07.822978+0000","last_peered":"2024-09-21T11:07:07.822978+0000","last_clean":"2024-09-21T11:07:07.822978+0000","last_became_active":"2024-09-21T11:06:33.535041+0000","last_became_peered":"2024-09-21T11:06:33.535041+0000","last_unstale":"2024-09-21T11:07:07.822978+0000","last_undegraded":"2024-09-21T11:07:07.822978+0000","last_fullsized":"2024-09-21T11:07:07.822978+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:56:17.413998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1592,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.781427+0000","last_change":"2024-09-21T11:05:48.549704+0000","last_active":"2024-09-21T11:07:05.781427+0000","last_peered":"2024-09-21T11:07:05.781427+0000","last_clean":"2024-09-21T11:07:05.781427+0000","last_became_active":"2024-09-21T11:05:48.549374+0000","last_became_peered":"2024-09-21T11:05:48.549374+0000","last_unstale":"2024-09-21T11:07:05.781427+0000","last_undegraded":"2024-09-21T11:07:05.781427+0000","last_fullsized":"2024-09-21T11:07:05.781427+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:57.002097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":656,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.823589+0000","last_change":"2024-09-21T11:05:49.692048+0000","last_active":"2024-09-21T11:07:07.823589+0000","last_peered":"2024-09-21T11:07:07.823589+0000","last_clean":"2024-09-21T11:07:07.823589+0000","last_became_active":"2024-09-21T11:05:49.548699+0000","last_became_peered":"2024-09-21T11:05:49.548699+0000","last_unstale":"2024-09-21T11:07:07.823589+0000","last_undegraded":"2024-09-21T11:07:07.823589+0000","last_fullsized":"2024-09-21T11:07:07.823589+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:30.701647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":95,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"346'149","reported_seq":956,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.783452+0000","last_change":"2024-09-21T11:07:05.782919+0000","last_active":"2024-09-21T11:07:05.783452+0000","last_peered":"2024-09-21T11:07:05.783452+0000","last_clean":"2024-09-21T11:07:05.783452+0000","last_became_active":"2024-09-21T11:07:05.782486+0000","last_became_peered":"2024-09-21T11:07:05.782486+0000","last_unstale":"2024-09-21T11:07:05.783452+0000","last_undegraded":"2024-09-21T11:07:05.783452+0000","last_fullsized":"2024-09-21T11:07:05.783452+0000","mapping_epoch":1681,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1682,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:06:44.106093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:06:44.106093+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:58:14.998229+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":67,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.823283+0000","last_change":"2024-09-21T11:06:55.759286+0000","last_active":"2024-09-21T11:07:07.823283+0000","last_peered":"2024-09-21T11:07:07.823283+0000","last_clean":"2024-09-21T11:07:07.823283+0000","last_became_active":"2024-09-21T11:06:55.758907+0000","last_became_peered":"2024-09-21T11:06:55.758907+0000","last_unstale":"2024-09-21T11:07:07.823283+0000","last_undegraded":"2024-09-21T11:07:07.823283+0000","last_fullsized":"2024-09-21T11:07:07.823283+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:43:23.517348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2611,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.777896+0000","last_change":"2024-09-21T11:05:49.538825+0000","last_active":"2024-09-21T11:07:05.777896+0000","last_peered":"2024-09-21T11:07:05.777896+0000","last_clean":"2024-09-21T11:07:05.777896+0000","last_became_active":"2024-09-21T11:05:49.538488+0000","last_became_peered":"2024-09-21T11:05:49.538488+0000","last_unstale":"2024-09-21T11:07:05.777896+0000","last_undegraded":"2024-09-21T11:07:05.777896+0000","last_fullsized":"2024-09-21T11:07:05.777896+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:14:02.932113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":154,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822463+0000","last_change":"2024-09-21T11:05:48.533847+0000","last_active":"2024-09-21T11:07:07.822463+0000","last_peered":"2024-09-21T11:07:07.822463+0000","last_clean":"2024-09-21T11:07:07.822463+0000","last_became_active":"2024-09-21T11:05:48.533544+0000","last_became_peered":"2024-09-21T11:05:48.533544+0000","last_unstale":"2024-09-21T11:07:07.822463+0000","last_undegraded":"2024-09-21T11:07:07.822463+0000","last_fullsized":"2024-09-21T11:07:07.822463+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:06:32.575001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"346'149","reported_seq":1319,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.778168+0000","last_change":"2024-09-21T11:04:37.545585+0000","last_active":"2024-09-21T11:07:05.778168+0000","last_peered":"2024-09-21T11:07:05.778168+0000","last_clean":"2024-09-21T11:07:05.778168+0000","last_became_active":"2024-09-21T11:04:37.545040+0000","last_became_peered":"2024-09-21T11:04:37.545040+0000","last_unstale":"2024-09-21T11:07:05.778168+0000","last_undegraded":"2024-09-21T11:07:05.778168+0000","last_fullsized":"2024-09-21T11:07:05.778168+0000","mapping_epoch":1607,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1608,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:50.671365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:50.671365+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:46.202953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":111,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822985+0000","last_change":"2024-09-21T11:06:55.762480+0000","last_active":"2024-09-21T11:07:07.822985+0000","last_peered":"2024-09-21T11:07:07.822985+0000","last_clean":"2024-09-21T11:07:07.822985+0000","last_became_active":"2024-09-21T11:06:55.761904+0000","last_became_peered":"2024-09-21T11:06:55.761904+0000","last_unstale":"2024-09-21T11:07:07.822985+0000","last_undegraded":"2024-09-21T11:07:07.822985+0000","last_fullsized":"2024-09-21T11:07:07.822985+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:22:02.009669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":118,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822418+0000","last_change":"2024-09-21T11:07:02.780287+0000","last_active":"2024-09-21T11:07:07.822418+0000","last_peered":"2024-09-21T11:07:07.822418+0000","last_clean":"2024-09-21T11:07:07.822418+0000","last_became_active":"2024-09-21T11:07:02.779916+0000","last_became_peered":"2024-09-21T11:07:02.779916+0000","last_unstale":"2024-09-21T11:07:07.822418+0000","last_undegraded":"2024-09-21T11:07:07.822418+0000","last_fullsized":"2024-09-21T11:07:07.822418+0000","mapping_epoch":1678,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:10.990224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":371,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.780730+0000","last_change":"2024-09-21T11:06:33.531901+0000","last_active":"2024-09-21T11:07:05.780730+0000","last_peered":"2024-09-21T11:07:05.780730+0000","last_clean":"2024-09-21T11:07:05.780730+0000","last_became_active":"2024-09-21T11:06:33.531562+0000","last_became_peered":"2024-09-21T11:06:33.531562+0000","last_unstale":"2024-09-21T11:07:05.780730+0000","last_undegraded":"2024-09-21T11:07:05.780730+0000","last_fullsized":"2024-09-21T11:07:05.780730+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:04.767469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":213,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.778435+0000","last_change":"2024-09-21T11:02:39.502922+0000","last_active":"2024-09-21T11:07:05.778435+0000","last_peered":"2024-09-21T11:07:05.778435+0000","last_clean":"2024-09-21T11:07:05.778435+0000","last_became_active":"2024-09-21T11:02:39.502633+0000","last_became_peered":"2024-09-21T11:02:39.502633+0000","last_unstale":"2024-09-21T11:07:05.778435+0000","last_undegraded":"2024-09-21T11:07:05.778435+0000","last_fullsized":"2024-09-21T11:07:05.778435+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:29:20.971522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":147,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.821884+0000","last_change":"2024-09-21T11:04:54.651475+0000","last_active":"2024-09-21T11:07:07.821884+0000","last_peered":"2024-09-21T11:07:07.821884+0000","last_clean":"2024-09-21T11:07:07.821884+0000","last_became_active":"2024-09-21T11:04:54.651179+0000","last_became_peered":"2024-09-21T11:04:54.651179+0000","last_unstale":"2024-09-21T11:07:07.821884+0000","last_undegraded":"2024-09-21T11:07:07.821884+0000","last_fullsized":"2024-09-21T11:07:07.821884+0000","mapping_epoch":1615,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:49.630408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":25,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.779822+0000","last_change":"2024-09-21T11:06:55.766388+0000","last_active":"2024-09-21T11:07:05.779822+0000","last_peered":"2024-09-21T11:07:05.779822+0000","last_clean":"2024-09-21T11:07:05.779822+0000","last_became_active":"2024-09-21T11:06:55.760643+0000","last_became_peered":"2024-09-21T11:06:55.760643+0000","last_unstale":"2024-09-21T11:07:05.779822+0000","last_undegraded":"2024-09-21T11:07:05.779822+0000","last_fullsized":"2024-09-21T11:07:05.779822+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:51:16.360005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":884,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.779947+0000","last_change":"2024-09-21T11:06:58.762931+0000","last_active":"2024-09-21T11:07:05.779947+0000","last_peered":"2024-09-21T11:07:05.779947+0000","last_clean":"2024-09-21T11:07:05.779947+0000","last_became_active":"2024-09-21T11:06:58.762397+0000","last_became_peered":"2024-09-21T11:06:58.762397+0000","last_unstale":"2024-09-21T11:07:05.779947+0000","last_undegraded":"2024-09-21T11:07:05.779947+0000","last_fullsized":"2024-09-21T11:07:05.779947+0000","mapping_epoch":1675,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1676,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:05:58.110003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:05:58.110003+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:59:21.260999+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"346'149","reported_seq":988,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.778536+0000","last_change":"2024-09-21T11:05:51.519229+0000","last_active":"2024-09-21T11:07:05.778536+0000","last_peered":"2024-09-21T11:07:05.778536+0000","last_clean":"2024-09-21T11:07:05.778536+0000","last_became_active":"2024-09-21T11:05:51.518878+0000","last_became_peered":"2024-09-21T11:05:51.518878+0000","last_unstale":"2024-09-21T11:07:05.778536+0000","last_undegraded":"2024-09-21T11:07:05.778536+0000","last_fullsized":"2024-09-21T11:07:05.778536+0000","mapping_epoch":1639,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1640,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:42.008235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:42.008235+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:25:31.810536+0000","scrub_duration":0.011129834999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2679,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.823376+0000","last_change":"2024-09-21T11:01:46.484382+0000","last_active":"2024-09-21T11:07:07.823376+0000","last_peered":"2024-09-21T11:07:07.823376+0000","last_clean":"2024-09-21T11:07:07.823376+0000","last_became_active":"2024-09-21T11:01:46.484116+0000","last_became_peered":"2024-09-21T11:01:46.484116+0000","last_unstale":"2024-09-21T11:07:07.823376+0000","last_undegraded":"2024-09-21T11:07:07.823376+0000","last_fullsized":"2024-09-21T11:07:07.823376+0000","mapping_epoch":1578,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:48.924838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":53,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.821672+0000","last_change":"2024-09-21T11:06:55.760878+0000","last_active":"2024-09-21T11:07:07.821672+0000","last_peered":"2024-09-21T11:07:07.821672+0000","last_clean":"2024-09-21T11:07:07.821672+0000","last_became_active":"2024-09-21T11:06:55.760619+0000","last_became_peered":"2024-09-21T11:06:55.760619+0000","last_unstale":"2024-09-21T11:07:07.821672+0000","last_undegraded":"2024-09-21T11:07:07.821672+0000","last_fullsized":"2024-09-21T11:07:07.821672+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:05:47.078808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"346'149","reported_seq":1312,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822342+0000","last_change":"2024-09-21T11:06:10.596860+0000","last_active":"2024-09-21T11:07:07.822342+0000","last_peered":"2024-09-21T11:07:07.822342+0000","last_clean":"2024-09-21T11:07:07.822342+0000","last_became_active":"2024-09-21T11:06:10.596469+0000","last_became_peered":"2024-09-21T11:06:10.596469+0000","last_unstale":"2024-09-21T11:07:07.822342+0000","last_undegraded":"2024-09-21T11:07:07.822342+0000","last_fullsized":"2024-09-21T11:07:07.822342+0000","mapping_epoch":1651,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1652,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:48.663349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.663349+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:02:37.313323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1621,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.781842+0000","last_change":"2024-09-21T11:06:33.535747+0000","last_active":"2024-09-21T11:07:05.781842+0000","last_peered":"2024-09-21T11:07:05.781842+0000","last_clean":"2024-09-21T11:07:05.781842+0000","last_became_active":"2024-09-21T11:06:33.535181+0000","last_became_peered":"2024-09-21T11:06:33.535181+0000","last_unstale":"2024-09-21T11:07:05.781842+0000","last_undegraded":"2024-09-21T11:07:05.781842+0000","last_fullsized":"2024-09-21T11:07:05.781842+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:52.206420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:06.039470+0000","last_change":"2024-09-21T11:07:02.780831+0000","last_active":"2024-09-21T11:07:06.039470+0000","last_peered":"2024-09-21T11:07:06.039470+0000","last_clean":"2024-09-21T11:07:06.039470+0000","last_became_active":"2024-09-21T11:07:02.780124+0000","last_became_peered":"2024-09-21T11:07:02.780124+0000","last_unstale":"2024-09-21T11:07:06.039470+0000","last_undegraded":"2024-09-21T11:07:06.039470+0000","last_fullsized":"2024-09-21T11:07:06.039470+0000","mapping_epoch":1678,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:09.477669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"346'149","reported_seq":1136,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.821972+0000","last_change":"2024-09-21T11:04:56.693547+0000","last_active":"2024-09-21T11:07:07.821972+0000","last_peered":"2024-09-21T11:07:07.821972+0000","last_clean":"2024-09-21T11:07:07.821972+0000","last_became_active":"2024-09-21T11:04:56.693184+0000","last_became_peered":"2024-09-21T11:04:56.693184+0000","last_unstale":"2024-09-21T11:07:07.821972+0000","last_undegraded":"2024-09-21T11:07:07.821972+0000","last_fullsized":"2024-09-21T11:07:07.821972+0000","mapping_epoch":1617,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1618,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:49.947173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:49.947173+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T11:07:16.574658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1319101,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":15,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:06.039570+0000","last_change":"2024-09-21T11:07:02.779174+0000","last_active":"2024-09-21T11:07:06.039570+0000","last_peered":"2024-09-21T11:07:06.039570+0000","last_clean":"2024-09-21T11:07:06.039570+0000","last_became_active":"2024-09-21T11:07:02.778864+0000","last_became_peered":"2024-09-21T11:07:02.778864+0000","last_unstale":"2024-09-21T11:07:06.039570+0000","last_undegraded":"2024-09-21T11:07:06.039570+0000","last_fullsized":"2024-09-21T11:07:06.039570+0000","mapping_epoch":1678,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:59:52.183861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1650,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.778044+0000","last_change":"2024-09-21T11:06:33.533209+0000","last_active":"2024-09-21T11:07:05.778044+0000","last_peered":"2024-09-21T11:07:05.778044+0000","last_clean":"2024-09-21T11:07:05.778044+0000","last_became_active":"2024-09-21T11:06:33.532865+0000","last_became_peered":"2024-09-21T11:06:33.532865+0000","last_unstale":"2024-09-21T11:07:05.778044+0000","last_undegraded":"2024-09-21T11:07:05.778044+0000","last_fullsized":"2024-09-21T11:07:05.778044+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:38:03.274980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"346'149","reported_seq":1155,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.780096+0000","last_change":"2024-09-21T11:02:38.474085+0000","last_active":"2024-09-21T11:07:05.780096+0000","last_peered":"2024-09-21T11:07:05.780096+0000","last_clean":"2024-09-21T11:07:05.780096+0000","last_became_active":"2024-09-21T11:02:38.473593+0000","last_became_peered":"2024-09-21T11:02:38.473593+0000","last_unstale":"2024-09-21T11:07:05.780096+0000","last_undegraded":"2024-09-21T11:07:05.780096+0000","last_fullsized":"2024-09-21T11:07:05.780096+0000","mapping_epoch":1589,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1590,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:17.418646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:17.418646+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:59:13.530642+0000","scrub_duration":0.00185648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":856,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.781488+0000","last_change":"2024-09-21T11:02:39.501678+0000","last_active":"2024-09-21T11:07:05.781488+0000","last_peered":"2024-09-21T11:07:05.781488+0000","last_clean":"2024-09-21T11:07:05.781488+0000","last_became_active":"2024-09-21T11:02:39.497766+0000","last_became_peered":"2024-09-21T11:02:39.497766+0000","last_unstale":"2024-09-21T11:07:05.781488+0000","last_undegraded":"2024-09-21T11:07:05.781488+0000","last_fullsized":"2024-09-21T11:07:05.781488+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:09:25.807702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":172,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822835+0000","last_change":"2024-09-21T11:05:48.649119+0000","last_active":"2024-09-21T11:07:07.822835+0000","last_peered":"2024-09-21T11:07:07.822835+0000","last_clean":"2024-09-21T11:07:07.822835+0000","last_became_active":"2024-09-21T11:05:48.550467+0000","last_became_peered":"2024-09-21T11:05:48.550467+0000","last_unstale":"2024-09-21T11:07:07.822835+0000","last_undegraded":"2024-09-21T11:07:07.822835+0000","last_fullsized":"2024-09-21T11:07:07.822835+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:46:06.305352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"346'149","reported_seq":941,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.942573+0000","last_change":"2024-09-21T11:07:07.942671+0000","last_active":"2024-09-21T11:07:07.942573+0000","last_peered":"2024-09-21T11:07:07.942573+0000","last_clean":"2024-09-21T11:07:07.942573+0000","last_became_active":"2024-09-21T11:07:05.782775+0000","last_became_peered":"2024-09-21T11:07:05.782775+0000","last_unstale":"2024-09-21T11:07:07.942573+0000","last_undegraded":"2024-09-21T11:07:07.942573+0000","last_fullsized":"2024-09-21T11:07:07.942573+0000","mapping_epoch":1681,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1682,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:07:07.942557+0000","last_deep_scrub":"346'149","last_deep_scrub_stamp":"2024-09-21T11:07:00.489593+0000","last_clean_scrub_stamp":"2024-09-21T11:07:07.942557+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:25:44.786324+0000","scrub_duration":0.0019991430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":104,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822647+0000","last_change":"2024-09-21T11:05:48.622817+0000","last_active":"2024-09-21T11:07:07.822647+0000","last_peered":"2024-09-21T11:07:07.822647+0000","last_clean":"2024-09-21T11:07:07.822647+0000","last_became_active":"2024-09-21T11:05:48.559669+0000","last_became_peered":"2024-09-21T11:05:48.559669+0000","last_unstale":"2024-09-21T11:07:07.822647+0000","last_undegraded":"2024-09-21T11:07:07.822647+0000","last_fullsized":"2024-09-21T11:07:07.822647+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:25:03.540564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":157,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.823133+0000","last_change":"2024-09-21T11:04:47.608909+0000","last_active":"2024-09-21T11:07:07.823133+0000","last_peered":"2024-09-21T11:07:07.823133+0000","last_clean":"2024-09-21T11:07:07.823133+0000","last_became_active":"2024-09-21T11:04:47.608245+0000","last_became_peered":"2024-09-21T11:04:47.608245+0000","last_unstale":"2024-09-21T11:07:07.823133+0000","last_undegraded":"2024-09-21T11:07:07.823133+0000","last_fullsized":"2024-09-21T11:07:07.823133+0000","mapping_epoch":1610,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1611,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:37:10.069192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"346'149","reported_seq":999,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.804032+0000","last_change":"2024-09-21T11:06:10.590388+0000","last_active":"2024-09-21T11:07:07.804032+0000","last_peered":"2024-09-21T11:07:07.804032+0000","last_clean":"2024-09-21T11:07:07.804032+0000","last_became_active":"2024-09-21T11:06:10.589444+0000","last_became_peered":"2024-09-21T11:06:10.589444+0000","last_unstale":"2024-09-21T11:07:07.804032+0000","last_undegraded":"2024-09-21T11:07:07.804032+0000","last_fullsized":"2024-09-21T11:07:07.804032+0000","mapping_epoch":1651,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1652,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:05:59.940755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:05:59.940755+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:23:37.430188+0000","scrub_duration":0.0034290169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":210,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.778372+0000","last_change":"2024-09-21T11:06:55.757992+0000","last_active":"2024-09-21T11:07:05.778372+0000","last_peered":"2024-09-21T11:07:05.778372+0000","last_clean":"2024-09-21T11:07:05.778372+0000","last_became_active":"2024-09-21T11:06:55.757384+0000","last_became_peered":"2024-09-21T11:06:55.757384+0000","last_unstale":"2024-09-21T11:07:05.778372+0000","last_undegraded":"2024-09-21T11:07:05.778372+0000","last_fullsized":"2024-09-21T11:07:05.778372+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:59:44.041307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":430,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.780670+0000","last_change":"2024-09-21T11:04:47.610105+0000","last_active":"2024-09-21T11:07:05.780670+0000","last_peered":"2024-09-21T11:07:05.780670+0000","last_clean":"2024-09-21T11:07:05.780670+0000","last_became_active":"2024-09-21T11:04:47.594894+0000","last_became_peered":"2024-09-21T11:04:47.594894+0000","last_unstale":"2024-09-21T11:07:05.780670+0000","last_undegraded":"2024-09-21T11:07:05.780670+0000","last_fullsized":"2024-09-21T11:07:05.780670+0000","mapping_epoch":1610,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1611,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:26:45.561726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"346'149","reported_seq":959,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822808+0000","last_change":"2024-09-21T11:06:10.594793+0000","last_active":"2024-09-21T11:07:07.822808+0000","last_peered":"2024-09-21T11:07:07.822808+0000","last_clean":"2024-09-21T11:07:07.822808+0000","last_became_active":"2024-09-21T11:06:10.594289+0000","last_became_peered":"2024-09-21T11:06:10.594289+0000","last_unstale":"2024-09-21T11:07:07.822808+0000","last_undegraded":"2024-09-21T11:07:07.822808+0000","last_fullsized":"2024-09-21T11:07:07.822808+0000","mapping_epoch":1651,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1652,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:54.708658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:54.708658+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:07.379791+0000","scrub_duration":0.0021035160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1686,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:06.039492+0000","last_change":"2024-09-21T11:06:40.705586+0000","last_active":"2024-09-21T11:07:06.039492+0000","last_peered":"2024-09-21T11:07:06.039492+0000","last_clean":"2024-09-21T11:07:06.039492+0000","last_became_active":"2024-09-21T11:06:40.705193+0000","last_became_peered":"2024-09-21T11:06:40.705193+0000","last_unstale":"2024-09-21T11:07:06.039492+0000","last_undegraded":"2024-09-21T11:07:06.039492+0000","last_fullsized":"2024-09-21T11:07:06.039492+0000","mapping_epoch":1665,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1666,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:25:37.486777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":38,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.780175+0000","last_change":"2024-09-21T11:07:02.784493+0000","last_active":"2024-09-21T11:07:05.780175+0000","last_peered":"2024-09-21T11:07:05.780175+0000","last_clean":"2024-09-21T11:07:05.780175+0000","last_became_active":"2024-09-21T11:07:02.781586+0000","last_became_peered":"2024-09-21T11:07:02.781586+0000","last_unstale":"2024-09-21T11:07:05.780175+0000","last_undegraded":"2024-09-21T11:07:05.780175+0000","last_fullsized":"2024-09-21T11:07:05.780175+0000","mapping_epoch":1678,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:06:42.604214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"346'149","reported_seq":858,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.779901+0000","last_change":"2024-09-21T11:06:58.765920+0000","last_active":"2024-09-21T11:07:05.779901+0000","last_peered":"2024-09-21T11:07:05.779901+0000","last_clean":"2024-09-21T11:07:05.779901+0000","last_became_active":"2024-09-21T11:06:58.765490+0000","last_became_peered":"2024-09-21T11:06:58.765490+0000","last_unstale":"2024-09-21T11:07:05.779901+0000","last_undegraded":"2024-09-21T11:07:05.779901+0000","last_fullsized":"2024-09-21T11:07:05.779901+0000","mapping_epoch":1675,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1676,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:53.896066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:53.896066+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:51.227905+0000","scrub_duration":0.001699578,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.780105+0000","last_change":"2024-09-21T11:06:55.758724+0000","last_active":"2024-09-21T11:07:05.780105+0000","last_peered":"2024-09-21T11:07:05.780105+0000","last_clean":"2024-09-21T11:07:05.780105+0000","last_became_active":"2024-09-21T11:06:55.758381+0000","last_became_peered":"2024-09-21T11:06:55.758381+0000","last_unstale":"2024-09-21T11:07:05.780105+0000","last_undegraded":"2024-09-21T11:07:05.780105+0000","last_fullsized":"2024-09-21T11:07:05.780105+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:03:29.158109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":25,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.780522+0000","last_change":"2024-09-21T11:06:55.771951+0000","last_active":"2024-09-21T11:07:05.780522+0000","last_peered":"2024-09-21T11:07:05.780522+0000","last_clean":"2024-09-21T11:07:05.780522+0000","last_became_active":"2024-09-21T11:06:55.763633+0000","last_became_peered":"2024-09-21T11:06:55.763633+0000","last_unstale":"2024-09-21T11:07:05.780522+0000","last_undegraded":"2024-09-21T11:07:05.780522+0000","last_fullsized":"2024-09-21T11:07:05.780522+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:59:18.987831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"346'149","reported_seq":925,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.823439+0000","last_change":"2024-09-21T11:06:58.762411+0000","last_active":"2024-09-21T11:07:07.823439+0000","last_peered":"2024-09-21T11:07:07.823439+0000","last_clean":"2024-09-21T11:07:07.823439+0000","last_became_active":"2024-09-21T11:06:58.761920+0000","last_became_peered":"2024-09-21T11:06:58.761920+0000","last_unstale":"2024-09-21T11:07:07.823439+0000","last_undegraded":"2024-09-21T11:07:07.823439+0000","last_fullsized":"2024-09-21T11:07:07.823439+0000","mapping_epoch":1675,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1676,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:06:37.734784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:06:37.734784+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:03:24.507956+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1724,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.777689+0000","last_change":"2024-09-21T11:05:49.569869+0000","last_active":"2024-09-21T11:07:05.777689+0000","last_peered":"2024-09-21T11:07:05.777689+0000","last_clean":"2024-09-21T11:07:05.777689+0000","last_became_active":"2024-09-21T11:05:49.526490+0000","last_became_peered":"2024-09-21T11:05:49.526490+0000","last_unstale":"2024-09-21T11:07:05.777689+0000","last_undegraded":"2024-09-21T11:07:05.777689+0000","last_fullsized":"2024-09-21T11:07:05.777689+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:34:32.411382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":53,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822428+0000","last_change":"2024-09-21T11:06:55.761306+0000","last_active":"2024-09-21T11:07:07.822428+0000","last_peered":"2024-09-21T11:07:07.822428+0000","last_clean":"2024-09-21T11:07:07.822428+0000","last_became_active":"2024-09-21T11:06:55.760835+0000","last_became_peered":"2024-09-21T11:06:55.760835+0000","last_unstale":"2024-09-21T11:07:07.822428+0000","last_undegraded":"2024-09-21T11:07:07.822428+0000","last_fullsized":"2024-09-21T11:07:07.822428+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:00.071274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"346'149","reported_seq":1326,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822749+0000","last_change":"2024-09-21T11:07:05.782862+0000","last_active":"2024-09-21T11:07:07.822749+0000","last_peered":"2024-09-21T11:07:07.822749+0000","last_clean":"2024-09-21T11:07:07.822749+0000","last_became_active":"2024-09-21T11:07:05.782468+0000","last_became_peered":"2024-09-21T11:07:05.782468+0000","last_unstale":"2024-09-21T11:07:07.822749+0000","last_undegraded":"2024-09-21T11:07:07.822749+0000","last_fullsized":"2024-09-21T11:07:07.822749+0000","mapping_epoch":1681,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1682,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:54.706428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:54.706428+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:26:13.823650+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":396,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.780237+0000","last_change":"2024-09-21T11:02:39.471842+0000","last_active":"2024-09-21T11:07:05.780237+0000","last_peered":"2024-09-21T11:07:05.780237+0000","last_clean":"2024-09-21T11:07:05.780237+0000","last_became_active":"2024-09-21T11:02:39.471339+0000","last_became_peered":"2024-09-21T11:02:39.471339+0000","last_unstale":"2024-09-21T11:07:05.780237+0000","last_undegraded":"2024-09-21T11:07:05.780237+0000","last_fullsized":"2024-09-21T11:07:05.780237+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:53:53.996670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":111,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822822+0000","last_change":"2024-09-21T11:06:55.762258+0000","last_active":"2024-09-21T11:07:07.822822+0000","last_peered":"2024-09-21T11:07:07.822822+0000","last_clean":"2024-09-21T11:07:07.822822+0000","last_became_active":"2024-09-21T11:06:55.761824+0000","last_became_peered":"2024-09-21T11:06:55.761824+0000","last_unstale":"2024-09-21T11:07:07.822822+0000","last_undegraded":"2024-09-21T11:07:07.822822+0000","last_fullsized":"2024-09-21T11:07:07.822822+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:43:09.930665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"346'149","reported_seq":858,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.780641+0000","last_change":"2024-09-21T11:06:10.587582+0000","last_active":"2024-09-21T11:07:05.780641+0000","last_peered":"2024-09-21T11:07:05.780641+0000","last_clean":"2024-09-21T11:07:05.780641+0000","last_became_active":"2024-09-21T11:06:10.587163+0000","last_became_peered":"2024-09-21T11:06:10.587163+0000","last_unstale":"2024-09-21T11:07:05.780641+0000","last_undegraded":"2024-09-21T11:07:05.780641+0000","last_fullsized":"2024-09-21T11:07:05.780641+0000","mapping_epoch":1651,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1652,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:06:00.073278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:06:00.073278+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:32:25.683870+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":123,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.823200+0000","last_change":"2024-09-21T11:06:55.762794+0000","last_active":"2024-09-21T11:07:07.823200+0000","last_peered":"2024-09-21T11:07:07.823200+0000","last_clean":"2024-09-21T11:07:07.823200+0000","last_became_active":"2024-09-21T11:06:55.760109+0000","last_became_peered":"2024-09-21T11:06:55.760109+0000","last_unstale":"2024-09-21T11:07:07.823200+0000","last_undegraded":"2024-09-21T11:07:07.823200+0000","last_fullsized":"2024-09-21T11:07:07.823200+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:41:23.127841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1663,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.780210+0000","last_change":"2024-09-21T11:02:38.476498+0000","last_active":"2024-09-21T11:07:05.780210+0000","last_peered":"2024-09-21T11:07:05.780210+0000","last_clean":"2024-09-21T11:07:05.780210+0000","last_became_active":"2024-09-21T11:02:38.476173+0000","last_became_peered":"2024-09-21T11:02:38.476173+0000","last_unstale":"2024-09-21T11:07:05.780210+0000","last_undegraded":"2024-09-21T11:07:05.780210+0000","last_fullsized":"2024-09-21T11:07:05.780210+0000","mapping_epoch":1589,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:49:26.618542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"346'149","reported_seq":1051,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822502+0000","last_change":"2024-09-21T11:06:10.596644+0000","last_active":"2024-09-21T11:07:07.822502+0000","last_peered":"2024-09-21T11:07:07.822502+0000","last_clean":"2024-09-21T11:07:07.822502+0000","last_became_active":"2024-09-21T11:06:10.596309+0000","last_became_peered":"2024-09-21T11:06:10.596309+0000","last_unstale":"2024-09-21T11:07:07.822502+0000","last_undegraded":"2024-09-21T11:07:07.822502+0000","last_fullsized":"2024-09-21T11:07:07.822502+0000","mapping_epoch":1651,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1652,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:47.955189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:47.955189+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:42:30.922196+0000","scrub_duration":0.0019884339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":411,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.780048+0000","last_change":"2024-09-21T11:02:39.468831+0000","last_active":"2024-09-21T11:07:05.780048+0000","last_peered":"2024-09-21T11:07:05.780048+0000","last_clean":"2024-09-21T11:07:05.780048+0000","last_became_active":"2024-09-21T11:02:39.468455+0000","last_became_peered":"2024-09-21T11:02:39.468455+0000","last_unstale":"2024-09-21T11:07:05.780048+0000","last_undegraded":"2024-09-21T11:07:05.780048+0000","last_fullsized":"2024-09-21T11:07:05.780048+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:34:59.495678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2676,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.823211+0000","last_change":"2024-09-21T11:06:33.530305+0000","last_active":"2024-09-21T11:07:07.823211+0000","last_peered":"2024-09-21T11:07:07.823211+0000","last_clean":"2024-09-21T11:07:07.823211+0000","last_became_active":"2024-09-21T11:06:33.529962+0000","last_became_peered":"2024-09-21T11:06:33.529962+0000","last_unstale":"2024-09-21T11:07:07.823211+0000","last_undegraded":"2024-09-21T11:07:07.823211+0000","last_fullsized":"2024-09-21T11:07:07.823211+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:41.527340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":381350,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1646781,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3129,"ondisk_log_size":3129,"up":63,"acting":63,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":95,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":18406,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":17314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3602,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234878,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1647724,"kb_used_data":10208,"kb_used_omap":1,"kb_used_meta":1637502,"kb_avail":92097428,"statfs":{"total":95995035648,"available":94307766272,"internally_reserved":0,"allocated":10452992,"data_stored":5766638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":1676802507},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1589,"seq":6824703033400,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":329076,"kb_used_data":10788,"kb_used_omap":128,"kb_used_meta":318143,"kb_avail":93416076,"statfs":{"total":95995035648,"available":95658061824,"internally_reserved":0,"allocated":11046912,"data_stored":6357495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":131575,"internal_metadata":325778953},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":6859062771755,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019787,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1371056,"kb_used_data":10468,"kb_used_omap":1,"kb_used_meta":1360574,"kb_avail":92374096,"statfs":{"total":95995035648,"available":94591074304,"internally_reserved":0,"allocated":10719232,"data_stored":6029838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1393228230},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369531,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1449184,"kb_used_data":10768,"kb_used_omap":69,"kb_used_meta":1438330,"kb_avail":92295968,"statfs":{"total":95995035648,"available":94511071232,"internally_reserved":0,"allocated":11026432,"data_stored":6352206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":1472850050},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2920577761986,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807958217,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":800792,"kb_used_data":10184,"kb_used_omap":119,"kb_used_meta":790472,"kb_avail":92944360,"statfs":{"total":95995035648,"available":95175024640,"internally_reserved":0,"allocated":10428416,"data_stored":5761697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":121896,"internal_metadata":809444312},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":666,"seq":2860448219856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":682336,"kb_used_data":10512,"kb_used_omap":76,"kb_used_meta":671731,"kb_avail":93062816,"statfs":{"total":95995035648,"available":95296323584,"internally_reserved":0,"allocated":10764288,"data_stored":6089981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78716,"internal_metadata":687852676},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1187,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T11:07:13.420 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T11:07:13.651 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:13 smithi070 ceph-mon[112139]: pgmap v3333: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T11:07:13.651 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:13 smithi070 ceph-mon[112139]: from='client.36942 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T11:07:13.651 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:13 smithi070 ceph-mon[107385]: pgmap v3333: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T11:07:13.651 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:13 smithi070 ceph-mon[107385]: from='client.36942 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T11:07:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:13 smithi160 ceph-mon[101879]: pgmap v3333: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T11:07:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:13 smithi160 ceph-mon[101879]: from='client.36942 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T11:07:14.825 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T11:07:14.825 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T11:07:14.825 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T11:07:14.826 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T11:07:14.830 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:07:15.519 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:15 smithi070 ceph-mon[107385]: pgmap v3334: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T11:07:15.520 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:15 smithi070 ceph-mon[112139]: pgmap v3334: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T11:07:15.541 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T11:07:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:15 smithi160 ceph-mon[101879]: pgmap v3334: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T11:07:16.208 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:07:17.769 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:17 smithi070 ceph-mon[107385]: pgmap v3335: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T11:07:17.770 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:17 smithi070 ceph-mon[112139]: pgmap v3335: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T11:07:17.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:17 smithi160 ceph-mon[101879]: pgmap v3335: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T11:07:18.041 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T11:07:18.042 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T11:07:19.024 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T11:07:19.024 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T11:07:19.024 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T11:07:19.024 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T11:07:19.024 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T11:07:19.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:19 smithi070 ceph-mon[107385]: from='client.36948 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T11:07:19.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:19 smithi070 ceph-mon[107385]: pgmap v3336: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:19.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:19 smithi070 ceph-mon[112139]: from='client.36948 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T11:07:19.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:19 smithi070 ceph-mon[112139]: pgmap v3336: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:19.696 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T11:07:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:19 smithi160 ceph-mon[101879]: from='client.36948 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T11:07:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:19 smithi160 ceph-mon[101879]: pgmap v3336: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:20.152 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":3335,"stamp":"2024-09-21T11:07:16.363356+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":386923,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":83,"num_write_kb":252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":1665282,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3188,"ondisk_log_size":3188,"up":279,"acting":279,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":279,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":6280168,"kb_used_data":62928,"kb_used_omap":397,"kb_used_meta":6216754,"kb_avail":556190744,"statfs":{"total":575970213888,"available":569539321856,"internally_reserved":0,"allocated":64438272,"data_stored":36357855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":406920,"internal_metadata":6365956728},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002061"},"pg_stats":[{"pgid":"3.1f","version":"71'4","reported_seq":110,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822222+0000","last_change":"2024-09-21T11:05:48.544347+0000","last_active":"2024-09-21T11:07:07.822222+0000","last_peered":"2024-09-21T11:07:07.822222+0000","last_clean":"2024-09-21T11:07:07.822222+0000","last_became_active":"2024-09-21T11:05:48.529869+0000","last_became_peered":"2024-09-21T11:05:48.529869+0000","last_unstale":"2024-09-21T11:07:07.822222+0000","last_undegraded":"2024-09-21T11:07:07.822222+0000","last_fullsized":"2024-09-21T11:07:07.822222+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:18:15.635403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1601,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.781019+0000","last_change":"2024-09-21T11:02:39.486670+0000","last_active":"2024-09-21T11:07:05.781019+0000","last_peered":"2024-09-21T11:07:05.781019+0000","last_clean":"2024-09-21T11:07:05.781019+0000","last_became_active":"2024-09-21T11:02:39.486248+0000","last_became_peered":"2024-09-21T11:02:39.486248+0000","last_unstale":"2024-09-21T11:07:05.781019+0000","last_undegraded":"2024-09-21T11:07:05.781019+0000","last_fullsized":"2024-09-21T11:07:05.781019+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:56:56.360708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":147,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.823268+0000","last_change":"2024-09-21T11:04:54.654602+0000","last_active":"2024-09-21T11:07:07.823268+0000","last_peered":"2024-09-21T11:07:07.823268+0000","last_clean":"2024-09-21T11:07:07.823268+0000","last_became_active":"2024-09-21T11:04:54.654231+0000","last_became_peered":"2024-09-21T11:04:54.654231+0000","last_unstale":"2024-09-21T11:07:07.823268+0000","last_undegraded":"2024-09-21T11:07:07.823268+0000","last_fullsized":"2024-09-21T11:07:07.823268+0000","mapping_epoch":1615,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:53:49.828685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":153,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822383+0000","last_change":"2024-09-21T11:05:48.533874+0000","last_active":"2024-09-21T11:07:07.822383+0000","last_peered":"2024-09-21T11:07:07.822383+0000","last_clean":"2024-09-21T11:07:07.822383+0000","last_became_active":"2024-09-21T11:05:48.533558+0000","last_became_peered":"2024-09-21T11:05:48.533558+0000","last_unstale":"2024-09-21T11:07:07.822383+0000","last_undegraded":"2024-09-21T11:07:07.822383+0000","last_fullsized":"2024-09-21T11:07:07.822383+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:01:28.336817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":179,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.781948+0000","last_change":"2024-09-21T11:02:39.502513+0000","last_active":"2024-09-21T11:07:05.781948+0000","last_peered":"2024-09-21T11:07:05.781948+0000","last_clean":"2024-09-21T11:07:05.781948+0000","last_became_active":"2024-09-21T11:02:39.502095+0000","last_became_peered":"2024-09-21T11:02:39.502095+0000","last_unstale":"2024-09-21T11:07:05.781948+0000","last_undegraded":"2024-09-21T11:07:05.781948+0000","last_fullsized":"2024-09-21T11:07:05.781948+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:50:46.078094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":15,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:06.039504+0000","last_change":"2024-09-21T11:07:02.780603+0000","last_active":"2024-09-21T11:07:06.039504+0000","last_peered":"2024-09-21T11:07:06.039504+0000","last_clean":"2024-09-21T11:07:06.039504+0000","last_became_active":"2024-09-21T11:07:02.780260+0000","last_became_peered":"2024-09-21T11:07:02.780260+0000","last_unstale":"2024-09-21T11:07:06.039504+0000","last_undegraded":"2024-09-21T11:07:06.039504+0000","last_fullsized":"2024-09-21T11:07:06.039504+0000","mapping_epoch":1678,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:50.715390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":179,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.781457+0000","last_change":"2024-09-21T11:02:39.502786+0000","last_active":"2024-09-21T11:07:05.781457+0000","last_peered":"2024-09-21T11:07:05.781457+0000","last_clean":"2024-09-21T11:07:05.781457+0000","last_became_active":"2024-09-21T11:02:39.498657+0000","last_became_peered":"2024-09-21T11:02:39.498657+0000","last_unstale":"2024-09-21T11:07:05.781457+0000","last_undegraded":"2024-09-21T11:07:05.781457+0000","last_fullsized":"2024-09-21T11:07:05.781457+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:30:40.747863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":15,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:06.039529+0000","last_change":"2024-09-21T11:07:02.780932+0000","last_active":"2024-09-21T11:07:06.039529+0000","last_peered":"2024-09-21T11:07:06.039529+0000","last_clean":"2024-09-21T11:07:06.039529+0000","last_became_active":"2024-09-21T11:07:02.780451+0000","last_became_peered":"2024-09-21T11:07:02.780451+0000","last_unstale":"2024-09-21T11:07:06.039529+0000","last_undegraded":"2024-09-21T11:07:06.039529+0000","last_fullsized":"2024-09-21T11:07:06.039529+0000","mapping_epoch":1678,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:59:32.916443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":104,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822731+0000","last_change":"2024-09-21T11:05:48.622455+0000","last_active":"2024-09-21T11:07:07.822731+0000","last_peered":"2024-09-21T11:07:07.822731+0000","last_clean":"2024-09-21T11:07:07.822731+0000","last_became_active":"2024-09-21T11:05:48.559270+0000","last_became_peered":"2024-09-21T11:05:48.559270+0000","last_unstale":"2024-09-21T11:07:07.822731+0000","last_undegraded":"2024-09-21T11:07:07.822731+0000","last_fullsized":"2024-09-21T11:07:07.822731+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:37:01.281922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":153,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.823056+0000","last_change":"2024-09-21T11:05:48.527392+0000","last_active":"2024-09-21T11:07:07.823056+0000","last_peered":"2024-09-21T11:07:07.823056+0000","last_clean":"2024-09-21T11:07:07.823056+0000","last_became_active":"2024-09-21T11:05:48.527078+0000","last_became_peered":"2024-09-21T11:05:48.527078+0000","last_unstale":"2024-09-21T11:07:07.823056+0000","last_undegraded":"2024-09-21T11:07:07.823056+0000","last_fullsized":"2024-09-21T11:07:07.823056+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:58:33.681474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":186,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.781726+0000","last_change":"2024-09-21T11:05:48.614724+0000","last_active":"2024-09-21T11:07:05.781726+0000","last_peered":"2024-09-21T11:07:05.781726+0000","last_clean":"2024-09-21T11:07:05.781726+0000","last_became_active":"2024-09-21T11:05:48.614414+0000","last_became_peered":"2024-09-21T11:05:48.614414+0000","last_unstale":"2024-09-21T11:07:05.781726+0000","last_undegraded":"2024-09-21T11:07:05.781726+0000","last_fullsized":"2024-09-21T11:07:05.781726+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:14:09.665271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":156,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822108+0000","last_change":"2024-09-21T11:04:47.619509+0000","last_active":"2024-09-21T11:07:07.822108+0000","last_peered":"2024-09-21T11:07:07.822108+0000","last_clean":"2024-09-21T11:07:07.822108+0000","last_became_active":"2024-09-21T11:04:47.618898+0000","last_became_peered":"2024-09-21T11:04:47.618898+0000","last_unstale":"2024-09-21T11:07:07.822108+0000","last_undegraded":"2024-09-21T11:07:07.822108+0000","last_fullsized":"2024-09-21T11:07:07.822108+0000","mapping_epoch":1610,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1611,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:28:19.899682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2690,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822791+0000","last_change":"2024-09-21T11:07:02.781730+0000","last_active":"2024-09-21T11:07:07.822791+0000","last_peered":"2024-09-21T11:07:07.822791+0000","last_clean":"2024-09-21T11:07:07.822791+0000","last_became_active":"2024-09-21T11:07:02.781410+0000","last_became_peered":"2024-09-21T11:07:02.781410+0000","last_unstale":"2024-09-21T11:07:07.822791+0000","last_undegraded":"2024-09-21T11:07:07.822791+0000","last_fullsized":"2024-09-21T11:07:07.822791+0000","mapping_epoch":1678,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:18:26.326185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":118,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822743+0000","last_change":"2024-09-21T11:06:55.763204+0000","last_active":"2024-09-21T11:07:07.822743+0000","last_peered":"2024-09-21T11:07:07.822743+0000","last_clean":"2024-09-21T11:07:07.822743+0000","last_became_active":"2024-09-21T11:06:55.762743+0000","last_became_peered":"2024-09-21T11:06:55.762743+0000","last_unstale":"2024-09-21T11:07:07.822743+0000","last_undegraded":"2024-09-21T11:07:07.822743+0000","last_fullsized":"2024-09-21T11:07:07.822743+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:25:04.804191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":171,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822057+0000","last_change":"2024-09-21T11:06:33.533570+0000","last_active":"2024-09-21T11:07:07.822057+0000","last_peered":"2024-09-21T11:07:07.822057+0000","last_clean":"2024-09-21T11:07:07.822057+0000","last_became_active":"2024-09-21T11:06:33.533236+0000","last_became_peered":"2024-09-21T11:06:33.533236+0000","last_unstale":"2024-09-21T11:07:07.822057+0000","last_undegraded":"2024-09-21T11:07:07.822057+0000","last_fullsized":"2024-09-21T11:07:07.822057+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:50:25.103638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1647,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.778865+0000","last_change":"2024-09-21T11:05:48.530856+0000","last_active":"2024-09-21T11:07:05.778865+0000","last_peered":"2024-09-21T11:07:05.778865+0000","last_clean":"2024-09-21T11:07:05.778865+0000","last_became_active":"2024-09-21T11:05:48.530496+0000","last_became_peered":"2024-09-21T11:05:48.530496+0000","last_unstale":"2024-09-21T11:07:05.778865+0000","last_undegraded":"2024-09-21T11:07:05.778865+0000","last_fullsized":"2024-09-21T11:07:05.778865+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:15.620293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":169,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822103+0000","last_change":"2024-09-21T11:04:34.523861+0000","last_active":"2024-09-21T11:07:07.822103+0000","last_peered":"2024-09-21T11:07:07.822103+0000","last_clean":"2024-09-21T11:07:07.822103+0000","last_became_active":"2024-09-21T11:04:34.523542+0000","last_became_peered":"2024-09-21T11:04:34.523542+0000","last_unstale":"2024-09-21T11:07:07.822103+0000","last_undegraded":"2024-09-21T11:07:07.822103+0000","last_fullsized":"2024-09-21T11:07:07.822103+0000","mapping_epoch":1604,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:35.990638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":104,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822123+0000","last_change":"2024-09-21T11:05:48.528637+0000","last_active":"2024-09-21T11:07:07.822123+0000","last_peered":"2024-09-21T11:07:07.822123+0000","last_clean":"2024-09-21T11:07:07.822123+0000","last_became_active":"2024-09-21T11:05:48.528215+0000","last_became_peered":"2024-09-21T11:05:48.528215+0000","last_unstale":"2024-09-21T11:07:07.822123+0000","last_undegraded":"2024-09-21T11:07:07.822123+0000","last_fullsized":"2024-09-21T11:07:07.822123+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:43:21.451145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"346'149","reported_seq":1157,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.783470+0000","last_change":"2024-09-21T11:07:05.783174+0000","last_active":"2024-09-21T11:07:05.783470+0000","last_peered":"2024-09-21T11:07:05.783470+0000","last_clean":"2024-09-21T11:07:05.783470+0000","last_became_active":"2024-09-21T11:07:05.782768+0000","last_became_peered":"2024-09-21T11:07:05.782768+0000","last_unstale":"2024-09-21T11:07:05.783470+0000","last_undegraded":"2024-09-21T11:07:05.783470+0000","last_fullsized":"2024-09-21T11:07:05.783470+0000","mapping_epoch":1681,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1682,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:53.724544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:53.724544+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:21:35.685593+0000","scrub_duration":0.0017779029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2714,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822640+0000","last_change":"2024-09-21T11:06:55.761046+0000","last_active":"2024-09-21T11:07:07.822640+0000","last_peered":"2024-09-21T11:07:07.822640+0000","last_clean":"2024-09-21T11:07:07.822640+0000","last_became_active":"2024-09-21T11:06:55.760680+0000","last_became_peered":"2024-09-21T11:06:55.760680+0000","last_unstale":"2024-09-21T11:07:07.822640+0000","last_undegraded":"2024-09-21T11:07:07.822640+0000","last_fullsized":"2024-09-21T11:07:07.822640+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:44:27.436994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.780728+0000","last_change":"2024-09-21T11:06:55.761083+0000","last_active":"2024-09-21T11:07:05.780728+0000","last_peered":"2024-09-21T11:07:05.780728+0000","last_clean":"2024-09-21T11:07:05.780728+0000","last_became_active":"2024-09-21T11:06:55.760730+0000","last_became_peered":"2024-09-21T11:06:55.760730+0000","last_unstale":"2024-09-21T11:07:05.780728+0000","last_undegraded":"2024-09-21T11:07:05.780728+0000","last_fullsized":"2024-09-21T11:07:05.780728+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:59:21.731681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":27,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.779867+0000","last_change":"2024-09-21T11:06:55.768569+0000","last_active":"2024-09-21T11:07:05.779867+0000","last_peered":"2024-09-21T11:07:05.779867+0000","last_clean":"2024-09-21T11:07:05.779867+0000","last_became_active":"2024-09-21T11:06:55.761500+0000","last_became_peered":"2024-09-21T11:06:55.761500+0000","last_unstale":"2024-09-21T11:07:05.779867+0000","last_undegraded":"2024-09-21T11:07:05.779867+0000","last_fullsized":"2024-09-21T11:07:05.779867+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:55:27.336715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":60,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.823345+0000","last_change":"2024-09-21T11:06:33.540524+0000","last_active":"2024-09-21T11:07:07.823345+0000","last_peered":"2024-09-21T11:07:07.823345+0000","last_clean":"2024-09-21T11:07:07.823345+0000","last_became_active":"2024-09-21T11:06:33.540173+0000","last_became_peered":"2024-09-21T11:06:33.540173+0000","last_unstale":"2024-09-21T11:07:07.823345+0000","last_undegraded":"2024-09-21T11:07:07.823345+0000","last_fullsized":"2024-09-21T11:07:07.823345+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:44:03.323025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":185,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.781240+0000","last_change":"2024-09-21T11:02:39.505094+0000","last_active":"2024-09-21T11:07:05.781240+0000","last_peered":"2024-09-21T11:07:05.781240+0000","last_clean":"2024-09-21T11:07:05.781240+0000","last_became_active":"2024-09-21T11:02:39.503527+0000","last_became_peered":"2024-09-21T11:02:39.503527+0000","last_unstale":"2024-09-21T11:07:05.781240+0000","last_undegraded":"2024-09-21T11:07:05.781240+0000","last_fullsized":"2024-09-21T11:07:05.781240+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:12:34.916393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14268,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":754,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.781335+0000","last_change":"2024-09-21T11:02:39.492530+0000","last_active":"2024-09-21T11:07:05.781335+0000","last_peered":"2024-09-21T11:07:05.781335+0000","last_clean":"2024-09-21T11:07:05.781335+0000","last_became_active":"2024-09-21T11:02:39.491835+0000","last_became_peered":"2024-09-21T11:02:39.491835+0000","last_unstale":"2024-09-21T11:07:05.781335+0000","last_undegraded":"2024-09-21T11:07:05.781335+0000","last_fullsized":"2024-09-21T11:07:05.781335+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:37.157843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"346'149","reported_seq":959,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822892+0000","last_change":"2024-09-21T11:06:58.762749+0000","last_active":"2024-09-21T11:07:07.822892+0000","last_peered":"2024-09-21T11:07:07.822892+0000","last_clean":"2024-09-21T11:07:07.822892+0000","last_became_active":"2024-09-21T11:06:58.762458+0000","last_became_peered":"2024-09-21T11:06:58.762458+0000","last_unstale":"2024-09-21T11:07:07.822892+0000","last_undegraded":"2024-09-21T11:07:07.822892+0000","last_fullsized":"2024-09-21T11:07:07.822892+0000","mapping_epoch":1675,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1676,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:59.781270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:59.781270+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:28:38.465822+0000","scrub_duration":0.0031203189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1621,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.781999+0000","last_change":"2024-09-21T11:05:48.525548+0000","last_active":"2024-09-21T11:07:05.781999+0000","last_peered":"2024-09-21T11:07:05.781999+0000","last_clean":"2024-09-21T11:07:05.781999+0000","last_became_active":"2024-09-21T11:05:48.525247+0000","last_became_peered":"2024-09-21T11:05:48.525247+0000","last_unstale":"2024-09-21T11:07:05.781999+0000","last_undegraded":"2024-09-21T11:07:05.781999+0000","last_fullsized":"2024-09-21T11:07:05.781999+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:51:18.193438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1695,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.778889+0000","last_change":"2024-09-21T11:06:55.761287+0000","last_active":"2024-09-21T11:07:05.778889+0000","last_peered":"2024-09-21T11:07:05.778889+0000","last_clean":"2024-09-21T11:07:05.778889+0000","last_became_active":"2024-09-21T11:06:55.760836+0000","last_became_peered":"2024-09-21T11:06:55.760836+0000","last_unstale":"2024-09-21T11:07:05.778889+0000","last_undegraded":"2024-09-21T11:07:05.778889+0000","last_fullsized":"2024-09-21T11:07:05.778889+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:07:51.834315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":108,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822900+0000","last_change":"2024-09-21T11:05:48.624344+0000","last_active":"2024-09-21T11:07:07.822900+0000","last_peered":"2024-09-21T11:07:07.822900+0000","last_clean":"2024-09-21T11:07:07.822900+0000","last_became_active":"2024-09-21T11:05:48.564758+0000","last_became_peered":"2024-09-21T11:05:48.564758+0000","last_unstale":"2024-09-21T11:07:07.822900+0000","last_undegraded":"2024-09-21T11:07:07.822900+0000","last_fullsized":"2024-09-21T11:07:07.822900+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:32.746530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"346'149","reported_seq":869,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.782053+0000","last_change":"2024-09-21T11:05:51.520639+0000","last_active":"2024-09-21T11:07:05.782053+0000","last_peered":"2024-09-21T11:07:05.782053+0000","last_clean":"2024-09-21T11:07:05.782053+0000","last_became_active":"2024-09-21T11:05:51.519581+0000","last_became_peered":"2024-09-21T11:05:51.519581+0000","last_unstale":"2024-09-21T11:07:05.782053+0000","last_undegraded":"2024-09-21T11:07:05.782053+0000","last_fullsized":"2024-09-21T11:07:05.782053+0000","mapping_epoch":1639,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1640,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:54.879276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:54.879276+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:53.844022+0000","scrub_duration":0.001843014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":53,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.821787+0000","last_change":"2024-09-21T11:06:55.759358+0000","last_active":"2024-09-21T11:07:07.821787+0000","last_peered":"2024-09-21T11:07:07.821787+0000","last_clean":"2024-09-21T11:07:07.821787+0000","last_became_active":"2024-09-21T11:06:55.759040+0000","last_became_peered":"2024-09-21T11:06:55.759040+0000","last_unstale":"2024-09-21T11:07:07.821787+0000","last_undegraded":"2024-09-21T11:07:07.821787+0000","last_fullsized":"2024-09-21T11:07:07.821787+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:59:39.297517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":60,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.823519+0000","last_change":"2024-09-21T11:06:33.532315+0000","last_active":"2024-09-21T11:07:07.823519+0000","last_peered":"2024-09-21T11:07:07.823519+0000","last_clean":"2024-09-21T11:07:07.823519+0000","last_became_active":"2024-09-21T11:06:33.532024+0000","last_became_peered":"2024-09-21T11:06:33.532024+0000","last_unstale":"2024-09-21T11:07:07.823519+0000","last_undegraded":"2024-09-21T11:07:07.823519+0000","last_fullsized":"2024-09-21T11:07:07.823519+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:35:36.295788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":2777,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822699+0000","last_change":"2024-09-21T11:06:55.766194+0000","last_active":"2024-09-21T11:07:07.822699+0000","last_peered":"2024-09-21T11:07:07.822699+0000","last_clean":"2024-09-21T11:07:07.822699+0000","last_became_active":"2024-09-21T11:06:55.761357+0000","last_became_peered":"2024-09-21T11:06:55.761357+0000","last_unstale":"2024-09-21T11:07:07.822699+0000","last_undegraded":"2024-09-21T11:07:07.822699+0000","last_fullsized":"2024-09-21T11:07:07.822699+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:19:18.903358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"346'149","reported_seq":994,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.779085+0000","last_change":"2024-09-21T11:06:43.715659+0000","last_active":"2024-09-21T11:07:05.779085+0000","last_peered":"2024-09-21T11:07:05.779085+0000","last_clean":"2024-09-21T11:07:05.779085+0000","last_became_active":"2024-09-21T11:06:43.715321+0000","last_became_peered":"2024-09-21T11:06:43.715321+0000","last_unstale":"2024-09-21T11:07:05.779085+0000","last_undegraded":"2024-09-21T11:07:05.779085+0000","last_fullsized":"2024-09-21T11:07:05.779085+0000","mapping_epoch":1668,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1669,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:06:00.911909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:06:00.911909+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:15:40.470846+0000","scrub_duration":0.002144838,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":104,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822563+0000","last_change":"2024-09-21T11:05:48.537936+0000","last_active":"2024-09-21T11:07:07.822563+0000","last_peered":"2024-09-21T11:07:07.822563+0000","last_clean":"2024-09-21T11:07:07.822563+0000","last_became_active":"2024-09-21T11:05:48.537615+0000","last_became_peered":"2024-09-21T11:05:48.537615+0000","last_unstale":"2024-09-21T11:07:07.822563+0000","last_undegraded":"2024-09-21T11:07:07.822563+0000","last_fullsized":"2024-09-21T11:07:07.822563+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:03:45.353800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":282,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:06.374164+0000","last_change":"2024-09-21T11:05:48.528383+0000","last_active":"2024-09-21T11:07:06.374164+0000","last_peered":"2024-09-21T11:07:06.374164+0000","last_clean":"2024-09-21T11:07:06.374164+0000","last_became_active":"2024-09-21T11:05:48.527903+0000","last_became_peered":"2024-09-21T11:05:48.527903+0000","last_unstale":"2024-09-21T11:07:06.374164+0000","last_undegraded":"2024-09-21T11:07:06.374164+0000","last_fullsized":"2024-09-21T11:07:06.374164+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:59:25.065009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":104,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.821948+0000","last_change":"2024-09-21T11:05:48.531015+0000","last_active":"2024-09-21T11:07:07.821948+0000","last_peered":"2024-09-21T11:07:07.821948+0000","last_clean":"2024-09-21T11:07:07.821948+0000","last_became_active":"2024-09-21T11:05:48.530729+0000","last_became_peered":"2024-09-21T11:05:48.530729+0000","last_unstale":"2024-09-21T11:07:07.821948+0000","last_undegraded":"2024-09-21T11:07:07.821948+0000","last_fullsized":"2024-09-21T11:07:07.821948+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:26.948930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"346'149","reported_seq":1032,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.778954+0000","last_change":"2024-09-21T11:06:58.762593+0000","last_active":"2024-09-21T11:07:05.778954+0000","last_peered":"2024-09-21T11:07:05.778954+0000","last_clean":"2024-09-21T11:07:05.778954+0000","last_became_active":"2024-09-21T11:06:58.762117+0000","last_became_peered":"2024-09-21T11:06:58.762117+0000","last_unstale":"2024-09-21T11:07:05.778954+0000","last_undegraded":"2024-09-21T11:07:05.778954+0000","last_fullsized":"2024-09-21T11:07:05.778954+0000","mapping_epoch":1675,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1676,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:05:57.002981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:05:57.002981+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:58:47.258964+0000","scrub_duration":0.0017979249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":67,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.823533+0000","last_change":"2024-09-21T11:07:02.779315+0000","last_active":"2024-09-21T11:07:07.823533+0000","last_peered":"2024-09-21T11:07:07.823533+0000","last_clean":"2024-09-21T11:07:07.823533+0000","last_became_active":"2024-09-21T11:07:02.778969+0000","last_became_peered":"2024-09-21T11:07:02.778969+0000","last_unstale":"2024-09-21T11:07:07.823533+0000","last_undegraded":"2024-09-21T11:07:07.823533+0000","last_fullsized":"2024-09-21T11:07:07.823533+0000","mapping_epoch":1678,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:56.329739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":104,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822041+0000","last_change":"2024-09-21T11:05:48.531055+0000","last_active":"2024-09-21T11:07:07.822041+0000","last_peered":"2024-09-21T11:07:07.822041+0000","last_clean":"2024-09-21T11:07:07.822041+0000","last_became_active":"2024-09-21T11:05:48.530743+0000","last_became_peered":"2024-09-21T11:05:48.530743+0000","last_unstale":"2024-09-21T11:07:07.822041+0000","last_undegraded":"2024-09-21T11:07:07.822041+0000","last_fullsized":"2024-09-21T11:07:07.822041+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:09:04.625920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":166,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822027+0000","last_change":"2024-09-21T11:06:55.759946+0000","last_active":"2024-09-21T11:07:07.822027+0000","last_peered":"2024-09-21T11:07:07.822027+0000","last_clean":"2024-09-21T11:07:07.822027+0000","last_became_active":"2024-09-21T11:06:55.759640+0000","last_became_peered":"2024-09-21T11:06:55.759640+0000","last_unstale":"2024-09-21T11:07:07.822027+0000","last_undegraded":"2024-09-21T11:07:07.822027+0000","last_fullsized":"2024-09-21T11:07:07.822027+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:32.254167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"346'149","reported_seq":1042,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822593+0000","last_change":"2024-09-21T11:05:51.518615+0000","last_active":"2024-09-21T11:07:07.822593+0000","last_peered":"2024-09-21T11:07:07.822593+0000","last_clean":"2024-09-21T11:07:07.822593+0000","last_became_active":"2024-09-21T11:05:51.518248+0000","last_became_peered":"2024-09-21T11:05:51.518248+0000","last_unstale":"2024-09-21T11:07:07.822593+0000","last_undegraded":"2024-09-21T11:07:07.822593+0000","last_fullsized":"2024-09-21T11:07:07.822593+0000","mapping_epoch":1639,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1640,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:48.918874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.918874+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:46.326025+0000","scrub_duration":0.001629134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":158,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822978+0000","last_change":"2024-09-21T11:06:33.535421+0000","last_active":"2024-09-21T11:07:07.822978+0000","last_peered":"2024-09-21T11:07:07.822978+0000","last_clean":"2024-09-21T11:07:07.822978+0000","last_became_active":"2024-09-21T11:06:33.535041+0000","last_became_peered":"2024-09-21T11:06:33.535041+0000","last_unstale":"2024-09-21T11:07:07.822978+0000","last_undegraded":"2024-09-21T11:07:07.822978+0000","last_fullsized":"2024-09-21T11:07:07.822978+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:56:17.413998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1592,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.781427+0000","last_change":"2024-09-21T11:05:48.549704+0000","last_active":"2024-09-21T11:07:05.781427+0000","last_peered":"2024-09-21T11:07:05.781427+0000","last_clean":"2024-09-21T11:07:05.781427+0000","last_became_active":"2024-09-21T11:05:48.549374+0000","last_became_peered":"2024-09-21T11:05:48.549374+0000","last_unstale":"2024-09-21T11:07:05.781427+0000","last_undegraded":"2024-09-21T11:07:05.781427+0000","last_fullsized":"2024-09-21T11:07:05.781427+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:57.002097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":656,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.823589+0000","last_change":"2024-09-21T11:05:49.692048+0000","last_active":"2024-09-21T11:07:07.823589+0000","last_peered":"2024-09-21T11:07:07.823589+0000","last_clean":"2024-09-21T11:07:07.823589+0000","last_became_active":"2024-09-21T11:05:49.548699+0000","last_became_peered":"2024-09-21T11:05:49.548699+0000","last_unstale":"2024-09-21T11:07:07.823589+0000","last_undegraded":"2024-09-21T11:07:07.823589+0000","last_fullsized":"2024-09-21T11:07:07.823589+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:30.701647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":95,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"346'149","reported_seq":956,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.783452+0000","last_change":"2024-09-21T11:07:05.782919+0000","last_active":"2024-09-21T11:07:05.783452+0000","last_peered":"2024-09-21T11:07:05.783452+0000","last_clean":"2024-09-21T11:07:05.783452+0000","last_became_active":"2024-09-21T11:07:05.782486+0000","last_became_peered":"2024-09-21T11:07:05.782486+0000","last_unstale":"2024-09-21T11:07:05.783452+0000","last_undegraded":"2024-09-21T11:07:05.783452+0000","last_fullsized":"2024-09-21T11:07:05.783452+0000","mapping_epoch":1681,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1682,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:06:44.106093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:06:44.106093+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:58:14.998229+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":67,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.823283+0000","last_change":"2024-09-21T11:06:55.759286+0000","last_active":"2024-09-21T11:07:07.823283+0000","last_peered":"2024-09-21T11:07:07.823283+0000","last_clean":"2024-09-21T11:07:07.823283+0000","last_became_active":"2024-09-21T11:06:55.758907+0000","last_became_peered":"2024-09-21T11:06:55.758907+0000","last_unstale":"2024-09-21T11:07:07.823283+0000","last_undegraded":"2024-09-21T11:07:07.823283+0000","last_fullsized":"2024-09-21T11:07:07.823283+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:43:23.517348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2611,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.777896+0000","last_change":"2024-09-21T11:05:49.538825+0000","last_active":"2024-09-21T11:07:05.777896+0000","last_peered":"2024-09-21T11:07:05.777896+0000","last_clean":"2024-09-21T11:07:05.777896+0000","last_became_active":"2024-09-21T11:05:49.538488+0000","last_became_peered":"2024-09-21T11:05:49.538488+0000","last_unstale":"2024-09-21T11:07:05.777896+0000","last_undegraded":"2024-09-21T11:07:05.777896+0000","last_fullsized":"2024-09-21T11:07:05.777896+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:14:02.932113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":154,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822463+0000","last_change":"2024-09-21T11:05:48.533847+0000","last_active":"2024-09-21T11:07:07.822463+0000","last_peered":"2024-09-21T11:07:07.822463+0000","last_clean":"2024-09-21T11:07:07.822463+0000","last_became_active":"2024-09-21T11:05:48.533544+0000","last_became_peered":"2024-09-21T11:05:48.533544+0000","last_unstale":"2024-09-21T11:07:07.822463+0000","last_undegraded":"2024-09-21T11:07:07.822463+0000","last_fullsized":"2024-09-21T11:07:07.822463+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:06:32.575001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"346'149","reported_seq":1319,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.778168+0000","last_change":"2024-09-21T11:04:37.545585+0000","last_active":"2024-09-21T11:07:05.778168+0000","last_peered":"2024-09-21T11:07:05.778168+0000","last_clean":"2024-09-21T11:07:05.778168+0000","last_became_active":"2024-09-21T11:04:37.545040+0000","last_became_peered":"2024-09-21T11:04:37.545040+0000","last_unstale":"2024-09-21T11:07:05.778168+0000","last_undegraded":"2024-09-21T11:07:05.778168+0000","last_fullsized":"2024-09-21T11:07:05.778168+0000","mapping_epoch":1607,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1608,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:50.671365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:50.671365+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:46.202953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":111,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822985+0000","last_change":"2024-09-21T11:06:55.762480+0000","last_active":"2024-09-21T11:07:07.822985+0000","last_peered":"2024-09-21T11:07:07.822985+0000","last_clean":"2024-09-21T11:07:07.822985+0000","last_became_active":"2024-09-21T11:06:55.761904+0000","last_became_peered":"2024-09-21T11:06:55.761904+0000","last_unstale":"2024-09-21T11:07:07.822985+0000","last_undegraded":"2024-09-21T11:07:07.822985+0000","last_fullsized":"2024-09-21T11:07:07.822985+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:22:02.009669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":118,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822418+0000","last_change":"2024-09-21T11:07:02.780287+0000","last_active":"2024-09-21T11:07:07.822418+0000","last_peered":"2024-09-21T11:07:07.822418+0000","last_clean":"2024-09-21T11:07:07.822418+0000","last_became_active":"2024-09-21T11:07:02.779916+0000","last_became_peered":"2024-09-21T11:07:02.779916+0000","last_unstale":"2024-09-21T11:07:07.822418+0000","last_undegraded":"2024-09-21T11:07:07.822418+0000","last_fullsized":"2024-09-21T11:07:07.822418+0000","mapping_epoch":1678,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:10.990224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":371,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.780730+0000","last_change":"2024-09-21T11:06:33.531901+0000","last_active":"2024-09-21T11:07:05.780730+0000","last_peered":"2024-09-21T11:07:05.780730+0000","last_clean":"2024-09-21T11:07:05.780730+0000","last_became_active":"2024-09-21T11:06:33.531562+0000","last_became_peered":"2024-09-21T11:06:33.531562+0000","last_unstale":"2024-09-21T11:07:05.780730+0000","last_undegraded":"2024-09-21T11:07:05.780730+0000","last_fullsized":"2024-09-21T11:07:05.780730+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:04.767469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":213,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.778435+0000","last_change":"2024-09-21T11:02:39.502922+0000","last_active":"2024-09-21T11:07:05.778435+0000","last_peered":"2024-09-21T11:07:05.778435+0000","last_clean":"2024-09-21T11:07:05.778435+0000","last_became_active":"2024-09-21T11:02:39.502633+0000","last_became_peered":"2024-09-21T11:02:39.502633+0000","last_unstale":"2024-09-21T11:07:05.778435+0000","last_undegraded":"2024-09-21T11:07:05.778435+0000","last_fullsized":"2024-09-21T11:07:05.778435+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:29:20.971522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":147,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.821884+0000","last_change":"2024-09-21T11:04:54.651475+0000","last_active":"2024-09-21T11:07:07.821884+0000","last_peered":"2024-09-21T11:07:07.821884+0000","last_clean":"2024-09-21T11:07:07.821884+0000","last_became_active":"2024-09-21T11:04:54.651179+0000","last_became_peered":"2024-09-21T11:04:54.651179+0000","last_unstale":"2024-09-21T11:07:07.821884+0000","last_undegraded":"2024-09-21T11:07:07.821884+0000","last_fullsized":"2024-09-21T11:07:07.821884+0000","mapping_epoch":1615,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:49.630408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":25,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.779822+0000","last_change":"2024-09-21T11:06:55.766388+0000","last_active":"2024-09-21T11:07:05.779822+0000","last_peered":"2024-09-21T11:07:05.779822+0000","last_clean":"2024-09-21T11:07:05.779822+0000","last_became_active":"2024-09-21T11:06:55.760643+0000","last_became_peered":"2024-09-21T11:06:55.760643+0000","last_unstale":"2024-09-21T11:07:05.779822+0000","last_undegraded":"2024-09-21T11:07:05.779822+0000","last_fullsized":"2024-09-21T11:07:05.779822+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:51:16.360005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":884,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.779947+0000","last_change":"2024-09-21T11:06:58.762931+0000","last_active":"2024-09-21T11:07:05.779947+0000","last_peered":"2024-09-21T11:07:05.779947+0000","last_clean":"2024-09-21T11:07:05.779947+0000","last_became_active":"2024-09-21T11:06:58.762397+0000","last_became_peered":"2024-09-21T11:06:58.762397+0000","last_unstale":"2024-09-21T11:07:05.779947+0000","last_undegraded":"2024-09-21T11:07:05.779947+0000","last_fullsized":"2024-09-21T11:07:05.779947+0000","mapping_epoch":1675,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1676,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:05:58.110003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:05:58.110003+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:59:21.260999+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"346'149","reported_seq":988,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.778536+0000","last_change":"2024-09-21T11:05:51.519229+0000","last_active":"2024-09-21T11:07:05.778536+0000","last_peered":"2024-09-21T11:07:05.778536+0000","last_clean":"2024-09-21T11:07:05.778536+0000","last_became_active":"2024-09-21T11:05:51.518878+0000","last_became_peered":"2024-09-21T11:05:51.518878+0000","last_unstale":"2024-09-21T11:07:05.778536+0000","last_undegraded":"2024-09-21T11:07:05.778536+0000","last_fullsized":"2024-09-21T11:07:05.778536+0000","mapping_epoch":1639,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1640,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:42.008235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:42.008235+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:25:31.810536+0000","scrub_duration":0.011129834999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2679,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.823376+0000","last_change":"2024-09-21T11:01:46.484382+0000","last_active":"2024-09-21T11:07:07.823376+0000","last_peered":"2024-09-21T11:07:07.823376+0000","last_clean":"2024-09-21T11:07:07.823376+0000","last_became_active":"2024-09-21T11:01:46.484116+0000","last_became_peered":"2024-09-21T11:01:46.484116+0000","last_unstale":"2024-09-21T11:07:07.823376+0000","last_undegraded":"2024-09-21T11:07:07.823376+0000","last_fullsized":"2024-09-21T11:07:07.823376+0000","mapping_epoch":1578,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:48.924838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":53,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.821672+0000","last_change":"2024-09-21T11:06:55.760878+0000","last_active":"2024-09-21T11:07:07.821672+0000","last_peered":"2024-09-21T11:07:07.821672+0000","last_clean":"2024-09-21T11:07:07.821672+0000","last_became_active":"2024-09-21T11:06:55.760619+0000","last_became_peered":"2024-09-21T11:06:55.760619+0000","last_unstale":"2024-09-21T11:07:07.821672+0000","last_undegraded":"2024-09-21T11:07:07.821672+0000","last_fullsized":"2024-09-21T11:07:07.821672+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:05:47.078808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"346'149","reported_seq":1312,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822342+0000","last_change":"2024-09-21T11:06:10.596860+0000","last_active":"2024-09-21T11:07:07.822342+0000","last_peered":"2024-09-21T11:07:07.822342+0000","last_clean":"2024-09-21T11:07:07.822342+0000","last_became_active":"2024-09-21T11:06:10.596469+0000","last_became_peered":"2024-09-21T11:06:10.596469+0000","last_unstale":"2024-09-21T11:07:07.822342+0000","last_undegraded":"2024-09-21T11:07:07.822342+0000","last_fullsized":"2024-09-21T11:07:07.822342+0000","mapping_epoch":1651,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1652,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:48.663349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.663349+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:02:37.313323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1621,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.781842+0000","last_change":"2024-09-21T11:06:33.535747+0000","last_active":"2024-09-21T11:07:05.781842+0000","last_peered":"2024-09-21T11:07:05.781842+0000","last_clean":"2024-09-21T11:07:05.781842+0000","last_became_active":"2024-09-21T11:06:33.535181+0000","last_became_peered":"2024-09-21T11:06:33.535181+0000","last_unstale":"2024-09-21T11:07:05.781842+0000","last_undegraded":"2024-09-21T11:07:05.781842+0000","last_fullsized":"2024-09-21T11:07:05.781842+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:52.206420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:06.039470+0000","last_change":"2024-09-21T11:07:02.780831+0000","last_active":"2024-09-21T11:07:06.039470+0000","last_peered":"2024-09-21T11:07:06.039470+0000","last_clean":"2024-09-21T11:07:06.039470+0000","last_became_active":"2024-09-21T11:07:02.780124+0000","last_became_peered":"2024-09-21T11:07:02.780124+0000","last_unstale":"2024-09-21T11:07:06.039470+0000","last_undegraded":"2024-09-21T11:07:06.039470+0000","last_fullsized":"2024-09-21T11:07:06.039470+0000","mapping_epoch":1678,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:09.477669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"346'149","reported_seq":1136,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.821972+0000","last_change":"2024-09-21T11:04:56.693547+0000","last_active":"2024-09-21T11:07:07.821972+0000","last_peered":"2024-09-21T11:07:07.821972+0000","last_clean":"2024-09-21T11:07:07.821972+0000","last_became_active":"2024-09-21T11:04:56.693184+0000","last_became_peered":"2024-09-21T11:04:56.693184+0000","last_unstale":"2024-09-21T11:07:07.821972+0000","last_undegraded":"2024-09-21T11:07:07.821972+0000","last_fullsized":"2024-09-21T11:07:07.821972+0000","mapping_epoch":1617,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1618,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:49.947173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:49.947173+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T11:07:16.574658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1319101,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":15,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:06.039570+0000","last_change":"2024-09-21T11:07:02.779174+0000","last_active":"2024-09-21T11:07:06.039570+0000","last_peered":"2024-09-21T11:07:06.039570+0000","last_clean":"2024-09-21T11:07:06.039570+0000","last_became_active":"2024-09-21T11:07:02.778864+0000","last_became_peered":"2024-09-21T11:07:02.778864+0000","last_unstale":"2024-09-21T11:07:06.039570+0000","last_undegraded":"2024-09-21T11:07:06.039570+0000","last_fullsized":"2024-09-21T11:07:06.039570+0000","mapping_epoch":1678,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:59:52.183861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1650,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.778044+0000","last_change":"2024-09-21T11:06:33.533209+0000","last_active":"2024-09-21T11:07:05.778044+0000","last_peered":"2024-09-21T11:07:05.778044+0000","last_clean":"2024-09-21T11:07:05.778044+0000","last_became_active":"2024-09-21T11:06:33.532865+0000","last_became_peered":"2024-09-21T11:06:33.532865+0000","last_unstale":"2024-09-21T11:07:05.778044+0000","last_undegraded":"2024-09-21T11:07:05.778044+0000","last_fullsized":"2024-09-21T11:07:05.778044+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:38:03.274980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"346'149","reported_seq":1155,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.780096+0000","last_change":"2024-09-21T11:02:38.474085+0000","last_active":"2024-09-21T11:07:05.780096+0000","last_peered":"2024-09-21T11:07:05.780096+0000","last_clean":"2024-09-21T11:07:05.780096+0000","last_became_active":"2024-09-21T11:02:38.473593+0000","last_became_peered":"2024-09-21T11:02:38.473593+0000","last_unstale":"2024-09-21T11:07:05.780096+0000","last_undegraded":"2024-09-21T11:07:05.780096+0000","last_fullsized":"2024-09-21T11:07:05.780096+0000","mapping_epoch":1589,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1590,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:17.418646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:17.418646+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:59:13.530642+0000","scrub_duration":0.00185648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":856,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.781488+0000","last_change":"2024-09-21T11:02:39.501678+0000","last_active":"2024-09-21T11:07:05.781488+0000","last_peered":"2024-09-21T11:07:05.781488+0000","last_clean":"2024-09-21T11:07:05.781488+0000","last_became_active":"2024-09-21T11:02:39.497766+0000","last_became_peered":"2024-09-21T11:02:39.497766+0000","last_unstale":"2024-09-21T11:07:05.781488+0000","last_undegraded":"2024-09-21T11:07:05.781488+0000","last_fullsized":"2024-09-21T11:07:05.781488+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:09:25.807702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":172,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822835+0000","last_change":"2024-09-21T11:05:48.649119+0000","last_active":"2024-09-21T11:07:07.822835+0000","last_peered":"2024-09-21T11:07:07.822835+0000","last_clean":"2024-09-21T11:07:07.822835+0000","last_became_active":"2024-09-21T11:05:48.550467+0000","last_became_peered":"2024-09-21T11:05:48.550467+0000","last_unstale":"2024-09-21T11:07:07.822835+0000","last_undegraded":"2024-09-21T11:07:07.822835+0000","last_fullsized":"2024-09-21T11:07:07.822835+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:46:06.305352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"346'149","reported_seq":941,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.942573+0000","last_change":"2024-09-21T11:07:07.942671+0000","last_active":"2024-09-21T11:07:07.942573+0000","last_peered":"2024-09-21T11:07:07.942573+0000","last_clean":"2024-09-21T11:07:07.942573+0000","last_became_active":"2024-09-21T11:07:05.782775+0000","last_became_peered":"2024-09-21T11:07:05.782775+0000","last_unstale":"2024-09-21T11:07:07.942573+0000","last_undegraded":"2024-09-21T11:07:07.942573+0000","last_fullsized":"2024-09-21T11:07:07.942573+0000","mapping_epoch":1681,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1682,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:07:07.942557+0000","last_deep_scrub":"346'149","last_deep_scrub_stamp":"2024-09-21T11:07:00.489593+0000","last_clean_scrub_stamp":"2024-09-21T11:07:07.942557+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:25:44.786324+0000","scrub_duration":0.0019991430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":104,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822647+0000","last_change":"2024-09-21T11:05:48.622817+0000","last_active":"2024-09-21T11:07:07.822647+0000","last_peered":"2024-09-21T11:07:07.822647+0000","last_clean":"2024-09-21T11:07:07.822647+0000","last_became_active":"2024-09-21T11:05:48.559669+0000","last_became_peered":"2024-09-21T11:05:48.559669+0000","last_unstale":"2024-09-21T11:07:07.822647+0000","last_undegraded":"2024-09-21T11:07:07.822647+0000","last_fullsized":"2024-09-21T11:07:07.822647+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:25:03.540564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":157,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.823133+0000","last_change":"2024-09-21T11:04:47.608909+0000","last_active":"2024-09-21T11:07:07.823133+0000","last_peered":"2024-09-21T11:07:07.823133+0000","last_clean":"2024-09-21T11:07:07.823133+0000","last_became_active":"2024-09-21T11:04:47.608245+0000","last_became_peered":"2024-09-21T11:04:47.608245+0000","last_unstale":"2024-09-21T11:07:07.823133+0000","last_undegraded":"2024-09-21T11:07:07.823133+0000","last_fullsized":"2024-09-21T11:07:07.823133+0000","mapping_epoch":1610,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1611,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:37:10.069192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"346'149","reported_seq":999,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.804032+0000","last_change":"2024-09-21T11:06:10.590388+0000","last_active":"2024-09-21T11:07:07.804032+0000","last_peered":"2024-09-21T11:07:07.804032+0000","last_clean":"2024-09-21T11:07:07.804032+0000","last_became_active":"2024-09-21T11:06:10.589444+0000","last_became_peered":"2024-09-21T11:06:10.589444+0000","last_unstale":"2024-09-21T11:07:07.804032+0000","last_undegraded":"2024-09-21T11:07:07.804032+0000","last_fullsized":"2024-09-21T11:07:07.804032+0000","mapping_epoch":1651,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1652,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:05:59.940755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:05:59.940755+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:23:37.430188+0000","scrub_duration":0.0034290169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":210,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.778372+0000","last_change":"2024-09-21T11:06:55.757992+0000","last_active":"2024-09-21T11:07:05.778372+0000","last_peered":"2024-09-21T11:07:05.778372+0000","last_clean":"2024-09-21T11:07:05.778372+0000","last_became_active":"2024-09-21T11:06:55.757384+0000","last_became_peered":"2024-09-21T11:06:55.757384+0000","last_unstale":"2024-09-21T11:07:05.778372+0000","last_undegraded":"2024-09-21T11:07:05.778372+0000","last_fullsized":"2024-09-21T11:07:05.778372+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:59:44.041307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":430,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.780670+0000","last_change":"2024-09-21T11:04:47.610105+0000","last_active":"2024-09-21T11:07:05.780670+0000","last_peered":"2024-09-21T11:07:05.780670+0000","last_clean":"2024-09-21T11:07:05.780670+0000","last_became_active":"2024-09-21T11:04:47.594894+0000","last_became_peered":"2024-09-21T11:04:47.594894+0000","last_unstale":"2024-09-21T11:07:05.780670+0000","last_undegraded":"2024-09-21T11:07:05.780670+0000","last_fullsized":"2024-09-21T11:07:05.780670+0000","mapping_epoch":1610,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1611,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:26:45.561726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"346'149","reported_seq":960,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822846+0000","last_change":"2024-09-21T11:06:10.594793+0000","last_active":"2024-09-21T11:07:07.822846+0000","last_peered":"2024-09-21T11:07:07.822846+0000","last_clean":"2024-09-21T11:07:07.822846+0000","last_became_active":"2024-09-21T11:06:10.594289+0000","last_became_peered":"2024-09-21T11:06:10.594289+0000","last_unstale":"2024-09-21T11:07:07.822846+0000","last_undegraded":"2024-09-21T11:07:07.822846+0000","last_fullsized":"2024-09-21T11:07:07.822846+0000","mapping_epoch":1651,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1652,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:54.708658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:54.708658+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:07.379791+0000","scrub_duration":0.0021035160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1686,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:06.039492+0000","last_change":"2024-09-21T11:06:40.705586+0000","last_active":"2024-09-21T11:07:06.039492+0000","last_peered":"2024-09-21T11:07:06.039492+0000","last_clean":"2024-09-21T11:07:06.039492+0000","last_became_active":"2024-09-21T11:06:40.705193+0000","last_became_peered":"2024-09-21T11:06:40.705193+0000","last_unstale":"2024-09-21T11:07:06.039492+0000","last_undegraded":"2024-09-21T11:07:06.039492+0000","last_fullsized":"2024-09-21T11:07:06.039492+0000","mapping_epoch":1665,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1666,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:25:37.486777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":38,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.780175+0000","last_change":"2024-09-21T11:07:02.784493+0000","last_active":"2024-09-21T11:07:05.780175+0000","last_peered":"2024-09-21T11:07:05.780175+0000","last_clean":"2024-09-21T11:07:05.780175+0000","last_became_active":"2024-09-21T11:07:02.781586+0000","last_became_peered":"2024-09-21T11:07:02.781586+0000","last_unstale":"2024-09-21T11:07:05.780175+0000","last_undegraded":"2024-09-21T11:07:05.780175+0000","last_fullsized":"2024-09-21T11:07:05.780175+0000","mapping_epoch":1678,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:06:42.604214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"346'149","reported_seq":858,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.779901+0000","last_change":"2024-09-21T11:06:58.765920+0000","last_active":"2024-09-21T11:07:05.779901+0000","last_peered":"2024-09-21T11:07:05.779901+0000","last_clean":"2024-09-21T11:07:05.779901+0000","last_became_active":"2024-09-21T11:06:58.765490+0000","last_became_peered":"2024-09-21T11:06:58.765490+0000","last_unstale":"2024-09-21T11:07:05.779901+0000","last_undegraded":"2024-09-21T11:07:05.779901+0000","last_fullsized":"2024-09-21T11:07:05.779901+0000","mapping_epoch":1675,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1676,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:53.896066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:53.896066+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:51.227905+0000","scrub_duration":0.001699578,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.780105+0000","last_change":"2024-09-21T11:06:55.758724+0000","last_active":"2024-09-21T11:07:05.780105+0000","last_peered":"2024-09-21T11:07:05.780105+0000","last_clean":"2024-09-21T11:07:05.780105+0000","last_became_active":"2024-09-21T11:06:55.758381+0000","last_became_peered":"2024-09-21T11:06:55.758381+0000","last_unstale":"2024-09-21T11:07:05.780105+0000","last_undegraded":"2024-09-21T11:07:05.780105+0000","last_fullsized":"2024-09-21T11:07:05.780105+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:03:29.158109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":25,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.780522+0000","last_change":"2024-09-21T11:06:55.771951+0000","last_active":"2024-09-21T11:07:05.780522+0000","last_peered":"2024-09-21T11:07:05.780522+0000","last_clean":"2024-09-21T11:07:05.780522+0000","last_became_active":"2024-09-21T11:06:55.763633+0000","last_became_peered":"2024-09-21T11:06:55.763633+0000","last_unstale":"2024-09-21T11:07:05.780522+0000","last_undegraded":"2024-09-21T11:07:05.780522+0000","last_fullsized":"2024-09-21T11:07:05.780522+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:59:18.987831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"346'149","reported_seq":925,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.823439+0000","last_change":"2024-09-21T11:06:58.762411+0000","last_active":"2024-09-21T11:07:07.823439+0000","last_peered":"2024-09-21T11:07:07.823439+0000","last_clean":"2024-09-21T11:07:07.823439+0000","last_became_active":"2024-09-21T11:06:58.761920+0000","last_became_peered":"2024-09-21T11:06:58.761920+0000","last_unstale":"2024-09-21T11:07:07.823439+0000","last_undegraded":"2024-09-21T11:07:07.823439+0000","last_fullsized":"2024-09-21T11:07:07.823439+0000","mapping_epoch":1675,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1676,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:06:37.734784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:06:37.734784+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:03:24.507956+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1724,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.777689+0000","last_change":"2024-09-21T11:05:49.569869+0000","last_active":"2024-09-21T11:07:05.777689+0000","last_peered":"2024-09-21T11:07:05.777689+0000","last_clean":"2024-09-21T11:07:05.777689+0000","last_became_active":"2024-09-21T11:05:49.526490+0000","last_became_peered":"2024-09-21T11:05:49.526490+0000","last_unstale":"2024-09-21T11:07:05.777689+0000","last_undegraded":"2024-09-21T11:07:05.777689+0000","last_fullsized":"2024-09-21T11:07:05.777689+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:34:32.411382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":53,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822428+0000","last_change":"2024-09-21T11:06:55.761306+0000","last_active":"2024-09-21T11:07:07.822428+0000","last_peered":"2024-09-21T11:07:07.822428+0000","last_clean":"2024-09-21T11:07:07.822428+0000","last_became_active":"2024-09-21T11:06:55.760835+0000","last_became_peered":"2024-09-21T11:06:55.760835+0000","last_unstale":"2024-09-21T11:07:07.822428+0000","last_undegraded":"2024-09-21T11:07:07.822428+0000","last_fullsized":"2024-09-21T11:07:07.822428+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:00.071274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"346'149","reported_seq":1326,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822749+0000","last_change":"2024-09-21T11:07:05.782862+0000","last_active":"2024-09-21T11:07:07.822749+0000","last_peered":"2024-09-21T11:07:07.822749+0000","last_clean":"2024-09-21T11:07:07.822749+0000","last_became_active":"2024-09-21T11:07:05.782468+0000","last_became_peered":"2024-09-21T11:07:05.782468+0000","last_unstale":"2024-09-21T11:07:07.822749+0000","last_undegraded":"2024-09-21T11:07:07.822749+0000","last_fullsized":"2024-09-21T11:07:07.822749+0000","mapping_epoch":1681,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1682,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:54.706428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:54.706428+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:26:13.823650+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":396,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.780237+0000","last_change":"2024-09-21T11:02:39.471842+0000","last_active":"2024-09-21T11:07:05.780237+0000","last_peered":"2024-09-21T11:07:05.780237+0000","last_clean":"2024-09-21T11:07:05.780237+0000","last_became_active":"2024-09-21T11:02:39.471339+0000","last_became_peered":"2024-09-21T11:02:39.471339+0000","last_unstale":"2024-09-21T11:07:05.780237+0000","last_undegraded":"2024-09-21T11:07:05.780237+0000","last_fullsized":"2024-09-21T11:07:05.780237+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:53:53.996670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":111,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822822+0000","last_change":"2024-09-21T11:06:55.762258+0000","last_active":"2024-09-21T11:07:07.822822+0000","last_peered":"2024-09-21T11:07:07.822822+0000","last_clean":"2024-09-21T11:07:07.822822+0000","last_became_active":"2024-09-21T11:06:55.761824+0000","last_became_peered":"2024-09-21T11:06:55.761824+0000","last_unstale":"2024-09-21T11:07:07.822822+0000","last_undegraded":"2024-09-21T11:07:07.822822+0000","last_fullsized":"2024-09-21T11:07:07.822822+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:43:09.930665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"346'149","reported_seq":858,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.780641+0000","last_change":"2024-09-21T11:06:10.587582+0000","last_active":"2024-09-21T11:07:05.780641+0000","last_peered":"2024-09-21T11:07:05.780641+0000","last_clean":"2024-09-21T11:07:05.780641+0000","last_became_active":"2024-09-21T11:06:10.587163+0000","last_became_peered":"2024-09-21T11:06:10.587163+0000","last_unstale":"2024-09-21T11:07:05.780641+0000","last_undegraded":"2024-09-21T11:07:05.780641+0000","last_fullsized":"2024-09-21T11:07:05.780641+0000","mapping_epoch":1651,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1652,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:06:00.073278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:06:00.073278+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:32:25.683870+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":123,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.823200+0000","last_change":"2024-09-21T11:06:55.762794+0000","last_active":"2024-09-21T11:07:07.823200+0000","last_peered":"2024-09-21T11:07:07.823200+0000","last_clean":"2024-09-21T11:07:07.823200+0000","last_became_active":"2024-09-21T11:06:55.760109+0000","last_became_peered":"2024-09-21T11:06:55.760109+0000","last_unstale":"2024-09-21T11:07:07.823200+0000","last_undegraded":"2024-09-21T11:07:07.823200+0000","last_fullsized":"2024-09-21T11:07:07.823200+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:41:23.127841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1663,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.780210+0000","last_change":"2024-09-21T11:02:38.476498+0000","last_active":"2024-09-21T11:07:05.780210+0000","last_peered":"2024-09-21T11:07:05.780210+0000","last_clean":"2024-09-21T11:07:05.780210+0000","last_became_active":"2024-09-21T11:02:38.476173+0000","last_became_peered":"2024-09-21T11:02:38.476173+0000","last_unstale":"2024-09-21T11:07:05.780210+0000","last_undegraded":"2024-09-21T11:07:05.780210+0000","last_fullsized":"2024-09-21T11:07:05.780210+0000","mapping_epoch":1589,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:49:26.618542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"346'149","reported_seq":1051,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.822502+0000","last_change":"2024-09-21T11:06:10.596644+0000","last_active":"2024-09-21T11:07:07.822502+0000","last_peered":"2024-09-21T11:07:07.822502+0000","last_clean":"2024-09-21T11:07:07.822502+0000","last_became_active":"2024-09-21T11:06:10.596309+0000","last_became_peered":"2024-09-21T11:06:10.596309+0000","last_unstale":"2024-09-21T11:07:07.822502+0000","last_undegraded":"2024-09-21T11:07:07.822502+0000","last_fullsized":"2024-09-21T11:07:07.822502+0000","mapping_epoch":1651,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1652,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:47.955189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:47.955189+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:42:30.922196+0000","scrub_duration":0.0019884339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":411,"reported_epoch":1682,"state":"active+clean","last_fresh":"2024-09-21T11:07:05.780048+0000","last_change":"2024-09-21T11:02:39.468831+0000","last_active":"2024-09-21T11:07:05.780048+0000","last_peered":"2024-09-21T11:07:05.780048+0000","last_clean":"2024-09-21T11:07:05.780048+0000","last_became_active":"2024-09-21T11:02:39.468455+0000","last_became_peered":"2024-09-21T11:02:39.468455+0000","last_unstale":"2024-09-21T11:07:05.780048+0000","last_undegraded":"2024-09-21T11:07:05.780048+0000","last_fullsized":"2024-09-21T11:07:05.780048+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:34:59.495678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2676,"reported_epoch":1683,"state":"active+clean","last_fresh":"2024-09-21T11:07:07.823211+0000","last_change":"2024-09-21T11:06:33.530305+0000","last_active":"2024-09-21T11:07:07.823211+0000","last_peered":"2024-09-21T11:07:07.823211+0000","last_clean":"2024-09-21T11:07:07.823211+0000","last_became_active":"2024-09-21T11:06:33.529962+0000","last_became_peered":"2024-09-21T11:06:33.529962+0000","last_unstale":"2024-09-21T11:07:07.823211+0000","last_undegraded":"2024-09-21T11:07:07.823211+0000","last_fullsized":"2024-09-21T11:07:07.823211+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:41.527340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":381350,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1646781,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3129,"ondisk_log_size":3129,"up":63,"acting":63,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":95,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":18406,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":17314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3602,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234878,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1647724,"kb_used_data":10208,"kb_used_omap":1,"kb_used_meta":1637502,"kb_avail":92097428,"statfs":{"total":95995035648,"available":94307766272,"internally_reserved":0,"allocated":10452992,"data_stored":5766638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":1676802507},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1589,"seq":6824703033400,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":329076,"kb_used_data":10788,"kb_used_omap":128,"kb_used_meta":318143,"kb_avail":93416076,"statfs":{"total":95995035648,"available":95658061824,"internally_reserved":0,"allocated":11046912,"data_stored":6357495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":131575,"internal_metadata":325778953},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":6859062771756,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019788,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1371056,"kb_used_data":10468,"kb_used_omap":1,"kb_used_meta":1360574,"kb_avail":92374096,"statfs":{"total":95995035648,"available":94591074304,"internally_reserved":0,"allocated":10719232,"data_stored":6029838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1393228230},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369532,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1449184,"kb_used_data":10768,"kb_used_omap":69,"kb_used_meta":1438330,"kb_avail":92295968,"statfs":{"total":95995035648,"available":94511071232,"internally_reserved":0,"allocated":11026432,"data_stored":6352206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":1472850050},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":2920577761987,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807958218,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":800792,"kb_used_data":10184,"kb_used_omap":119,"kb_used_meta":790472,"kb_avail":92944360,"statfs":{"total":95995035648,"available":95175024640,"internally_reserved":0,"allocated":10428416,"data_stored":5761697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":121896,"internal_metadata":809444312},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":666,"seq":2860448219857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":682336,"kb_used_data":10512,"kb_used_omap":76,"kb_used_meta":671731,"kb_avail":93062816,"statfs":{"total":95995035648,"available":95296323584,"internally_reserved":0,"allocated":10764288,"data_stored":6089981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78716,"internal_metadata":687852676},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1187,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T11:07:20.156 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T11:07:20.157 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T11:07:20.157 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-21T11:07:20.157 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-21T11:07:20.227 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:07:20.727 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:07:20.727 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:07:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:07:21.013 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:07:21.732 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:21 smithi070 ceph-mon[112139]: pgmap v3337: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:21.733 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:21 smithi070 ceph-mon[112139]: 1.8 scrub starts 2024-09-21T11:07:21.733 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:21 smithi070 ceph-mon[112139]: 1.8 scrub ok 2024-09-21T11:07:21.733 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:21 smithi070 ceph-mon[107385]: pgmap v3337: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:21.733 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:21 smithi070 ceph-mon[107385]: 1.8 scrub starts 2024-09-21T11:07:21.733 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:21 smithi070 ceph-mon[107385]: 1.8 scrub ok 2024-09-21T11:07:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:21 smithi160 ceph-mon[101879]: pgmap v3337: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:21.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:21 smithi160 ceph-mon[101879]: 1.8 scrub starts 2024-09-21T11:07:21.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:21 smithi160 ceph-mon[101879]: 1.8 scrub ok 2024-09-21T11:07:23.254 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:07:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:07:23] "GET /metrics HTTP/1.1" 200 34260 "" "Prometheus/2.51.0" 2024-09-21T11:07:23.259 INFO:teuthology.orchestra.run.smithi070.stderr:no valid command found; 10 closest matches: 2024-09-21T11:07:23.259 INFO:teuthology.orchestra.run.smithi070.stderr:0 2024-09-21T11:07:23.259 INFO:teuthology.orchestra.run.smithi070.stderr:1 2024-09-21T11:07:23.259 INFO:teuthology.orchestra.run.smithi070.stderr:2 2024-09-21T11:07:23.259 INFO:teuthology.orchestra.run.smithi070.stderr:abort 2024-09-21T11:07:23.259 INFO:teuthology.orchestra.run.smithi070.stderr:assert 2024-09-21T11:07:23.259 INFO:teuthology.orchestra.run.smithi070.stderr:bench [] [] [] [] 2024-09-21T11:07:23.259 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs debug_inject_read_zeros 2024-09-21T11:07:23.259 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs files list 2024-09-21T11:07:23.259 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs stats 2024-09-21T11:07:23.259 INFO:teuthology.orchestra.run.smithi070.stderr:bluestore allocator dump block 2024-09-21T11:07:23.259 INFO:teuthology.orchestra.run.smithi070.stderr:Error EINVAL: invalid command 2024-09-21T11:07:23.526 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:23 smithi070 ceph-mon[107385]: pgmap v3338: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:23.526 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:23 smithi070 ceph-mon[112139]: pgmap v3338: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:23 smithi160 ceph-mon[101879]: pgmap v3338: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:23.847 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T11:07:23.847 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T11:07:23.847 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T11:07:23.847 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T11:07:23.847 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.847 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.46190.0:84908 57.14 57:2ff1ef3b:::benchmark_data_smithi070_360420_object17559:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1559)", 2024-09-21T11:07:23.847 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:00:40.449269+0000", 2024-09-21T11:07:23.847 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 403.39415769499999, 2024-09-21T11:07:23.847 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.035143608, 2024-09-21T11:07:23.848 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:07:23.848 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:07:23.848 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:07:23.848 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:07:23.848 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.46190", 2024-09-21T11:07:23.848 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:07:23.848 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 84908 2024-09-21T11:07:23.848 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.848 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:07:23.848 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.848 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:07:23.848 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449269+0000", 2024-09-21T11:07:23.848 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.848 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.848 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.848 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:07:23.849 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449269+0000", 2024-09-21T11:07:23.849 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.849 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.849 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.849 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:07:23.849 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449270+0000", 2024-09-21T11:07:23.849 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4200000000000002e-07 2024-09-21T11:07:23.849 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.849 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.849 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:07:23.849 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449271+0000", 2024-09-21T11:07:23.849 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5650000000000001e-06 2024-09-21T11:07:23.849 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.849 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.849 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:07:23.850 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449276+0000", 2024-09-21T11:07:23.850 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.87e-06 2024-09-21T11:07:23.850 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.850 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.850 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:07:23.850 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.449280+0000", 2024-09-21T11:07:23.850 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0679999999999998e-06 2024-09-21T11:07:23.850 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.850 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.850 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.850 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.590591+0000", 2024-09-21T11:07:23.850 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.14131036599999999 2024-09-21T11:07:23.850 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.850 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.850 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:07:23.851 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:40.590596+0000", 2024-09-21T11:07:23.851 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4190000000000001e-06 2024-09-21T11:07:23.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.851 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.851 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.851 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.478340+0000", 2024-09-21T11:07:23.851 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.88774418499999996 2024-09-21T11:07:23.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.851 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.851 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for missing object", 2024-09-21T11:07:23.851 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.478458+0000", 2024-09-21T11:07:23.851 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000117686 2024-09-21T11:07:23.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.852 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.852 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.852 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.484331+0000", 2024-09-21T11:07:23.852 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0058731469999999996 2024-09-21T11:07:23.852 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.852 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.852 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:07:23.852 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:00:41.484413+0000", 2024-09-21T11:07:23.852 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1959999999999995e-05 2024-09-21T11:07:23.852 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.852 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:07:23.852 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.852 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.852 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.853 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:219142 58.e 58:75211456:::benchmark_data_smithi070_396992_object73183:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1603)", 2024-09-21T11:07:23.853 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:04:33.510241+0000", 2024-09-21T11:07:23.853 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 170.333186146, 2024-09-21T11:07:23.853 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1668667699999999, 2024-09-21T11:07:23.853 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:07:23.853 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:07:23.853 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:07:23.853 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:07:23.853 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:07:23.853 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:23.853 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 219142 2024-09-21T11:07:23.853 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.853 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:07:23.853 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.853 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:07:23.854 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:33.510241+0000", 2024-09-21T11:07:23.854 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.854 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.854 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.854 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:07:23.854 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:33.510241+0000", 2024-09-21T11:07:23.854 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.854 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.854 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.854 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:07:23.854 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:33.510243+0000", 2024-09-21T11:07:23.854 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7379999999999999e-06 2024-09-21T11:07:23.854 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.854 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.855 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:07:23.855 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:33.510248+0000", 2024-09-21T11:07:23.855 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4049999999999999e-06 2024-09-21T11:07:23.855 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.855 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.855 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:07:23.855 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:33.510266+0000", 2024-09-21T11:07:23.855 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7473e-05 2024-09-21T11:07:23.855 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.855 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.855 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:07:23.855 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:33.510272+0000", 2024-09-21T11:07:23.855 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6819999999999997e-06 2024-09-21T11:07:23.855 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.856 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.856 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.856 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:33.510346+0000", 2024-09-21T11:07:23.856 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3362000000000007e-05 2024-09-21T11:07:23.856 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.856 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.856 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:07:23.856 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:33.510493+0000", 2024-09-21T11:07:23.856 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000146997 2024-09-21T11:07:23.856 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.856 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.856 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,7", 2024-09-21T11:07:23.856 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:33.510633+0000", 2024-09-21T11:07:23.857 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014000500000000001 2024-09-21T11:07:23.857 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.857 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.857 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:07:23.857 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:33.513896+0000", 2024-09-21T11:07:23.857 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0032636340000000001 2024-09-21T11:07:23.857 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.857 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.857 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:23.857 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:33.515990+0000", 2024-09-21T11:07:23.857 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0020932289999999998 2024-09-21T11:07:23.857 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.857 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.858 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.858 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:33.522140+0000", 2024-09-21T11:07:23.858 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0061504029999999996 2024-09-21T11:07:23.858 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.858 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.858 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:07:23.858 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:33.522152+0000", 2024-09-21T11:07:23.858 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2411e-05 2024-09-21T11:07:23.858 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.858 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.858 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.858 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.512660+0000", 2024-09-21T11:07:23.858 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.99050738000000005 2024-09-21T11:07:23.858 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.859 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.859 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:07:23.859 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.512673+0000", 2024-09-21T11:07:23.859 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3643e-05 2024-09-21T11:07:23.859 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.859 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.859 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.859 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.659014+0000", 2024-09-21T11:07:23.859 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1463402810000001 2024-09-21T11:07:23.859 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.859 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.859 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for missing object", 2024-09-21T11:07:23.859 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.659132+0000", 2024-09-21T11:07:23.859 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011846 2024-09-21T11:07:23.860 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.860 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.860 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.860 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.677047+0000", 2024-09-21T11:07:23.860 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.017915041999999999 2024-09-21T11:07:23.860 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.860 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.860 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:07:23.860 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.677108+0000", 2024-09-21T11:07:23.860 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0625000000000003e-05 2024-09-21T11:07:23.860 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.860 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:07:23.860 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.860 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.860 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.861 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:219150 58.e 58:75ff93b0:::benchmark_data_smithi070_396992_object73191:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1604)", 2024-09-21T11:07:23.861 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:04:33.535764+0000", 2024-09-21T11:07:23.862 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 170.30766285499999, 2024-09-21T11:07:23.862 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1414995170000002, 2024-09-21T11:07:23.862 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:07:23.862 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:07:23.862 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:23.862 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:07:23.862 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:07:23.862 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:23.862 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 219150 2024-09-21T11:07:23.862 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.862 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:07:23.862 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.862 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:07:23.863 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:33.535764+0000", 2024-09-21T11:07:23.863 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.863 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.863 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.863 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:07:23.863 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:33.535764+0000", 2024-09-21T11:07:23.863 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.863 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.863 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.863 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:07:23.863 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:33.535766+0000", 2024-09-21T11:07:23.863 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7320000000000001e-06 2024-09-21T11:07:23.863 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.863 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.864 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:07:23.864 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:33.535772+0000", 2024-09-21T11:07:23.864 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9139999999999996e-06 2024-09-21T11:07:23.864 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.864 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.864 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:07:23.864 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:33.535788+0000", 2024-09-21T11:07:23.864 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5523999999999999e-05 2024-09-21T11:07:23.864 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.864 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.864 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:07:23.864 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:33.535796+0000", 2024-09-21T11:07:23.864 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5420000000000005e-06 2024-09-21T11:07:23.864 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.865 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.865 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.865 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:33.535863+0000", 2024-09-21T11:07:23.865 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6958000000000005e-05 2024-09-21T11:07:23.865 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.865 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.865 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:07:23.865 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:33.535874+0000", 2024-09-21T11:07:23.865 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0988000000000001e-05 2024-09-21T11:07:23.865 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.865 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.865 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.865 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.512702+0000", 2024-09-21T11:07:23.865 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.97682791000000002 2024-09-21T11:07:23.866 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.866 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.866 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:07:23.866 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.512710+0000", 2024-09-21T11:07:23.866 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1939999999999998e-06 2024-09-21T11:07:23.866 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.866 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.866 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.866 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.659467+0000", 2024-09-21T11:07:23.866 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.146756434 2024-09-21T11:07:23.866 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.866 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.866 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:07:23.866 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.659593+0000", 2024-09-21T11:07:23.867 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012690299999999999 2024-09-21T11:07:23.867 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.867 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.867 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,4,7", 2024-09-21T11:07:23.867 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.659732+0000", 2024-09-21T11:07:23.867 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000138651 2024-09-21T11:07:23.867 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.867 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.867 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:23.867 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.666648+0000", 2024-09-21T11:07:23.867 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0069162329999999999 2024-09-21T11:07:23.867 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.867 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.867 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:07:23.868 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.666700+0000", 2024-09-21T11:07:23.868 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1643e-05 2024-09-21T11:07:23.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.868 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.868 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:23.868 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.669429+0000", 2024-09-21T11:07:23.868 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002729184 2024-09-21T11:07:23.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.868 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.868 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:23.868 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.677143+0000", 2024-09-21T11:07:23.868 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0077139019999999999 2024-09-21T11:07:23.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.868 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.868 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:07:23.869 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.677177+0000", 2024-09-21T11:07:23.869 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3781000000000001e-05 2024-09-21T11:07:23.869 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.869 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.869 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:07:23.869 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.677264+0000", 2024-09-21T11:07:23.869 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7023999999999999e-05 2024-09-21T11:07:23.869 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.869 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:07:23.869 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.869 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.869 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.869 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:219122 58.4 58:288b92f9:::benchmark_data_smithi070_396992_object73163:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e1605)", 2024-09-21T11:07:23.869 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:04:34.513474+0000", 2024-09-21T11:07:23.870 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.32995356699999, 2024-09-21T11:07:23.870 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.182222729, 2024-09-21T11:07:23.870 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:07:23.870 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:07:23.870 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:07:23.870 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:07:23.870 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:07:23.870 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:23.870 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 219122 2024-09-21T11:07:23.870 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.870 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:07:23.870 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.870 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:07:23.870 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.513474+0000", 2024-09-21T11:07:23.870 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.871 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.871 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.871 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:07:23.871 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.513474+0000", 2024-09-21T11:07:23.871 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.871 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.871 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.871 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:07:23.871 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.513475+0000", 2024-09-21T11:07:23.871 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5969999999999999e-06 2024-09-21T11:07:23.871 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.871 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.871 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:07:23.871 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.513481+0000", 2024-09-21T11:07:23.872 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4609999999999999e-06 2024-09-21T11:07:23.872 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.872 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.872 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:07:23.872 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514002+0000", 2024-09-21T11:07:23.872 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00052086799999999996 2024-09-21T11:07:23.872 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.872 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.872 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:07:23.872 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514011+0000", 2024-09-21T11:07:23.872 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7909999999999999e-06 2024-09-21T11:07:23.872 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.872 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.872 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.872 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514113+0000", 2024-09-21T11:07:23.873 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000101477 2024-09-21T11:07:23.873 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.873 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.873 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:07:23.873 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514121+0000", 2024-09-21T11:07:23.873 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7579999999999992e-06 2024-09-21T11:07:23.873 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.873 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.873 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.873 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.695619+0000", 2024-09-21T11:07:23.873 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.181498739 2024-09-21T11:07:23.873 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.873 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.873 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:07:23.873 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.695696+0000", 2024-09-21T11:07:23.874 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7038000000000003e-05 2024-09-21T11:07:23.874 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.874 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:07:23.874 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.874 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.874 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.874 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:219125 58.4 58:2b0d7fc3:::benchmark_data_smithi070_396992_object73166:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e1605)", 2024-09-21T11:07:23.874 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:04:34.514030+0000", 2024-09-21T11:07:23.874 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.329396731, 2024-09-21T11:07:23.874 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.181997491, 2024-09-21T11:07:23.874 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:07:23.874 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:07:23.874 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:07:23.875 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:07:23.875 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:07:23.875 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:23.875 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 219125 2024-09-21T11:07:23.875 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.875 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:07:23.875 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.875 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:07:23.875 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514030+0000", 2024-09-21T11:07:23.875 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.875 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.875 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.875 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:07:23.875 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514030+0000", 2024-09-21T11:07:23.875 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.876 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.876 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.876 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:07:23.876 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514031+0000", 2024-09-21T11:07:23.876 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0200000000000001e-07 2024-09-21T11:07:23.876 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.876 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.876 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:07:23.876 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514035+0000", 2024-09-21T11:07:23.876 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7629999999999998e-06 2024-09-21T11:07:23.876 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.876 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.876 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:07:23.877 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514043+0000", 2024-09-21T11:07:23.877 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5259999999999993e-06 2024-09-21T11:07:23.877 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.877 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.877 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:07:23.877 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514049+0000", 2024-09-21T11:07:23.877 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4600000000000002e-06 2024-09-21T11:07:23.877 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.877 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.877 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.877 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514163+0000", 2024-09-21T11:07:23.877 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000114227 2024-09-21T11:07:23.877 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.877 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.878 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:07:23.878 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514172+0000", 2024-09-21T11:07:23.878 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5290000000000008e-06 2024-09-21T11:07:23.878 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.878 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.878 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.878 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.695950+0000", 2024-09-21T11:07:23.878 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.181778553 2024-09-21T11:07:23.878 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.878 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.878 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:07:23.878 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.696028+0000", 2024-09-21T11:07:23.878 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7731000000000005e-05 2024-09-21T11:07:23.878 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.878 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:07:23.879 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.879 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.879 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.879 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:219132 58.4 58:27675f58:::benchmark_data_smithi070_396992_object73173:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e1605)", 2024-09-21T11:07:23.879 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:04:34.514057+0000", 2024-09-21T11:07:23.879 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.32936988399999, 2024-09-21T11:07:23.879 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.182059985, 2024-09-21T11:07:23.879 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:07:23.879 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:07:23.879 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:07:23.879 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:07:23.879 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:07:23.879 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:23.879 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 219132 2024-09-21T11:07:23.880 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.880 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:07:23.880 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.880 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:07:23.880 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514057+0000", 2024-09-21T11:07:23.880 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.880 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.880 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.880 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:07:23.880 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514057+0000", 2024-09-21T11:07:23.880 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.880 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.880 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.880 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:07:23.881 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514058+0000", 2024-09-21T11:07:23.881 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.15e-07 2024-09-21T11:07:23.881 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.881 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.881 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:07:23.881 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514060+0000", 2024-09-21T11:07:23.881 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0190000000000001e-06 2024-09-21T11:07:23.881 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.881 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.881 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:07:23.881 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514066+0000", 2024-09-21T11:07:23.881 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8259999999999998e-06 2024-09-21T11:07:23.881 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.882 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.882 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:07:23.882 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514071+0000", 2024-09-21T11:07:23.882 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9490000000000004e-06 2024-09-21T11:07:23.882 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.882 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.882 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.882 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514216+0000", 2024-09-21T11:07:23.882 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014584799999999999 2024-09-21T11:07:23.882 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.882 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.882 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:07:23.882 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514223+0000", 2024-09-21T11:07:23.882 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6080000000000001e-06 2024-09-21T11:07:23.883 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.883 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.883 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.883 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.696062+0000", 2024-09-21T11:07:23.883 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1818389490000001 2024-09-21T11:07:23.883 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.883 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.883 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:07:23.883 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.696117+0000", 2024-09-21T11:07:23.883 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5371000000000002e-05 2024-09-21T11:07:23.883 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.883 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:07:23.883 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.883 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.884 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.884 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:219138 58.4 58:24f3f817:::benchmark_data_smithi070_396992_object73179:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e1605)", 2024-09-21T11:07:23.884 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:04:34.514125+0000", 2024-09-21T11:07:23.884 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.32930269400001, 2024-09-21T11:07:23.884 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.18207806, 2024-09-21T11:07:23.884 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:07:23.884 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:07:23.884 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:07:23.884 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:07:23.884 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:07:23.884 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:23.884 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 219138 2024-09-21T11:07:23.884 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.884 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:07:23.884 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.885 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:07:23.885 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514125+0000", 2024-09-21T11:07:23.885 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.885 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.885 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.885 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:07:23.885 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514125+0000", 2024-09-21T11:07:23.885 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.885 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.885 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.885 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:07:23.885 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514125+0000", 2024-09-21T11:07:23.885 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1999999999999998e-07 2024-09-21T11:07:23.885 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.885 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.885 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:07:23.886 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514129+0000", 2024-09-21T11:07:23.886 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7589999999999999e-06 2024-09-21T11:07:23.886 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.886 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.886 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:07:23.886 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514138+0000", 2024-09-21T11:07:23.886 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.8170000000000005e-06 2024-09-21T11:07:23.886 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.886 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.886 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:07:23.886 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514146+0000", 2024-09-21T11:07:23.886 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7619999999999995e-06 2024-09-21T11:07:23.886 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.886 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.887 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.887 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514254+0000", 2024-09-21T11:07:23.887 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010850599999999999 2024-09-21T11:07:23.887 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.887 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.887 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:07:23.887 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514261+0000", 2024-09-21T11:07:23.887 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3620000000000002e-06 2024-09-21T11:07:23.887 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.887 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.887 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.887 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.696144+0000", 2024-09-21T11:07:23.887 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.181883341 2024-09-21T11:07:23.887 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.888 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.888 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:07:23.888 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.696203+0000", 2024-09-21T11:07:23.888 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8693000000000002e-05 2024-09-21T11:07:23.888 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.888 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:07:23.888 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.888 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.888 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.888 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:219169 58.4 58:22f31fc6:::benchmark_data_smithi070_396992_object73210:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1605)", 2024-09-21T11:07:23.888 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:04:34.514157+0000", 2024-09-21T11:07:23.888 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.32927059299999, 2024-09-21T11:07:23.888 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.220695839, 2024-09-21T11:07:23.888 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:07:23.888 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:07:23.889 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:23.889 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:07:23.889 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:07:23.889 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:23.889 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 219169 2024-09-21T11:07:23.889 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.889 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:07:23.889 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.889 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:07:23.889 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514157+0000", 2024-09-21T11:07:23.889 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.889 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.889 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.889 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:07:23.889 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514157+0000", 2024-09-21T11:07:23.890 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.890 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.890 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.890 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:07:23.890 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514157+0000", 2024-09-21T11:07:23.890 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5900000000000002e-07 2024-09-21T11:07:23.890 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.890 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.890 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:07:23.890 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514159+0000", 2024-09-21T11:07:23.890 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.176e-06 2024-09-21T11:07:23.890 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.890 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.890 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:07:23.891 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514166+0000", 2024-09-21T11:07:23.891 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3609999999999997e-06 2024-09-21T11:07:23.891 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.891 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.891 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:07:23.891 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514170+0000", 2024-09-21T11:07:23.891 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.549e-06 2024-09-21T11:07:23.891 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.891 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.891 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.891 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514288+0000", 2024-09-21T11:07:23.891 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011773 2024-09-21T11:07:23.891 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.891 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.892 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:07:23.892 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.514295+0000", 2024-09-21T11:07:23.892 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6020000000000002e-06 2024-09-21T11:07:23.892 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.892 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.892 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.892 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.696232+0000", 2024-09-21T11:07:23.892 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1819362330000001 2024-09-21T11:07:23.892 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.892 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.892 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:07:23.892 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.696315+0000", 2024-09-21T11:07:23.892 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3616999999999996e-05 2024-09-21T11:07:23.892 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.893 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.893 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,6,7", 2024-09-21T11:07:23.893 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.696442+0000", 2024-09-21T11:07:23.893 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012676500000000001 2024-09-21T11:07:23.893 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.893 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.893 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:23.893 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.712024+0000", 2024-09-21T11:07:23.893 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015581920000000001 2024-09-21T11:07:23.893 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.893 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.893 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:07:23.893 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.712067+0000", 2024-09-21T11:07:23.893 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3473999999999997e-05 2024-09-21T11:07:23.894 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.894 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.894 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:23.894 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.720919+0000", 2024-09-21T11:07:23.894 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0088514119999999995 2024-09-21T11:07:23.894 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.894 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.894 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:23.894 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.734667+0000", 2024-09-21T11:07:23.894 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0137483 2024-09-21T11:07:23.894 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.894 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.894 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:07:23.894 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.734732+0000", 2024-09-21T11:07:23.895 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4418999999999994e-05 2024-09-21T11:07:23.895 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.895 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.895 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:07:23.895 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.734852+0000", 2024-09-21T11:07:23.895 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000120822 2024-09-21T11:07:23.895 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.895 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:07:23.895 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.895 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.895 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.895 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:219179 58.4 58:2509c6af:::benchmark_data_smithi070_396992_object73220:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1605)", 2024-09-21T11:07:23.895 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:04:34.639131+0000", 2024-09-21T11:07:23.895 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.204295753, 2024-09-21T11:07:23.896 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0998811180000001, 2024-09-21T11:07:23.896 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:07:23.896 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:07:23.896 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:23.896 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:07:23.896 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:07:23.896 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:23.896 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 219179 2024-09-21T11:07:23.896 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.896 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:07:23.896 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.896 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:07:23.896 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.639131+0000", 2024-09-21T11:07:23.896 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.896 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.897 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.897 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:07:23.897 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.639131+0000", 2024-09-21T11:07:23.897 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.897 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.897 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.897 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:07:23.897 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.639133+0000", 2024-09-21T11:07:23.897 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.282e-06 2024-09-21T11:07:23.897 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.897 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.897 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:07:23.897 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.639137+0000", 2024-09-21T11:07:23.897 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.4349999999999999e-06 2024-09-21T11:07:23.897 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.898 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.898 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:07:23.898 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.639148+0000", 2024-09-21T11:07:23.898 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1024e-05 2024-09-21T11:07:23.898 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.898 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.898 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:07:23.898 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.639153+0000", 2024-09-21T11:07:23.898 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3009999999999997e-06 2024-09-21T11:07:23.898 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.898 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.898 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.898 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.639202+0000", 2024-09-21T11:07:23.898 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8334000000000002e-05 2024-09-21T11:07:23.898 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.899 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.899 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:07:23.899 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:34.639209+0000", 2024-09-21T11:07:23.899 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0620000000000003e-06 2024-09-21T11:07:23.899 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.899 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.899 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.899 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.700827+0000", 2024-09-21T11:07:23.899 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0616176349999999 2024-09-21T11:07:23.899 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.899 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.899 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:07:23.899 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.700931+0000", 2024-09-21T11:07:23.899 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000104808 2024-09-21T11:07:23.900 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.900 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.900 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 4,6,7", 2024-09-21T11:07:23.900 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.701018+0000", 2024-09-21T11:07:23.900 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6958000000000003e-05 2024-09-21T11:07:23.900 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.900 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.900 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:23.900 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.712058+0000", 2024-09-21T11:07:23.900 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.011040210999999999 2024-09-21T11:07:23.900 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.900 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.900 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:07:23.900 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.720884+0000", 2024-09-21T11:07:23.900 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0088258899999999994 2024-09-21T11:07:23.900 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.901 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.901 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:23.901 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.720971+0000", 2024-09-21T11:07:23.901 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6507000000000004e-05 2024-09-21T11:07:23.901 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.901 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.901 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:23.901 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.738844+0000", 2024-09-21T11:07:23.901 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.017873515999999999 2024-09-21T11:07:23.901 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.901 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.901 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:07:23.901 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.738906+0000", 2024-09-21T11:07:23.901 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1168999999999997e-05 2024-09-21T11:07:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:07:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:04:35.739013+0000", 2024-09-21T11:07:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000106986 2024-09-21T11:07:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:07:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:244205 58.9 58:97807ee7:::benchmark_data_smithi070_396992_object98246:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1619)", 2024-09-21T11:07:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:03.376525+0000", 2024-09-21T11:07:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 140.46690175200001, 2024-09-21T11:07:23.902 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0687687750000001, 2024-09-21T11:07:23.903 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:07:23.903 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:07:23.903 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:07:23.903 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:07:23.903 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:07:23.903 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:23.903 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 244205 2024-09-21T11:07:23.903 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.903 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:07:23.903 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.903 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:07:23.903 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.376525+0000", 2024-09-21T11:07:23.903 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.903 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:07:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.376525+0000", 2024-09-21T11:07:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:07:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.376527+0000", 2024-09-21T11:07:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0640000000000001e-06 2024-09-21T11:07:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:07:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.376530+0000", 2024-09-21T11:07:23.904 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9260000000000002e-06 2024-09-21T11:07:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:07:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.376540+0000", 2024-09-21T11:07:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5529999999999998e-06 2024-09-21T11:07:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:07:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.376545+0000", 2024-09-21T11:07:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.361e-06 2024-09-21T11:07:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.376590+0000", 2024-09-21T11:07:23.905 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5049000000000002e-05 2024-09-21T11:07:23.906 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.906 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.906 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:07:23.906 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.376673+0000", 2024-09-21T11:07:23.906 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2787999999999994e-05 2024-09-21T11:07:23.906 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.906 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.906 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6", 2024-09-21T11:07:23.906 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.376750+0000", 2024-09-21T11:07:23.906 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6741000000000003e-05 2024-09-21T11:07:23.906 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.906 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.906 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:07:23.906 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.379092+0000", 2024-09-21T11:07:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0023418060000000001 2024-09-21T11:07:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.381815+0000", 2024-09-21T11:07:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0027232659999999998 2024-09-21T11:07:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.398830+0000", 2024-09-21T11:07:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.017014511 2024-09-21T11:07:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:07:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.398844+0000", 2024-09-21T11:07:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4711e-05 2024-09-21T11:07:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.445219+0000", 2024-09-21T11:07:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.046374769 2024-09-21T11:07:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:07:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.445294+0000", 2024-09-21T11:07:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5229999999999996e-05 2024-09-21T11:07:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:07:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:244206 58.9 58:987df1cf:::benchmark_data_smithi070_396992_object98247:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1619)", 2024-09-21T11:07:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:03.377969+0000", 2024-09-21T11:07:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 140.465458361, 2024-09-21T11:07:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0676627160000001, 2024-09-21T11:07:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:07:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:07:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:07:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:07:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:07:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:23.909 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 244206 2024-09-21T11:07:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:07:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:07:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.377969+0000", 2024-09-21T11:07:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:07:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.377969+0000", 2024-09-21T11:07:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:07:23.910 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.377970+0000", 2024-09-21T11:07:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.8999999999999995e-07 2024-09-21T11:07:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:07:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.377973+0000", 2024-09-21T11:07:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3529999999999999e-06 2024-09-21T11:07:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:07:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.377983+0000", 2024-09-21T11:07:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4900000000000006e-06 2024-09-21T11:07:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:07:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.377987+0000", 2024-09-21T11:07:23.911 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6750000000000001e-06 2024-09-21T11:07:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.379124+0000", 2024-09-21T11:07:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001136544 2024-09-21T11:07:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:07:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.379244+0000", 2024-09-21T11:07:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000120087 2024-09-21T11:07:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6", 2024-09-21T11:07:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.379402+0000", 2024-09-21T11:07:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000158033 2024-09-21T11:07:23.912 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:07:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.394466+0000", 2024-09-21T11:07:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015063758999999999 2024-09-21T11:07:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.399207+0000", 2024-09-21T11:07:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0047411320000000003 2024-09-21T11:07:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:07:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.399216+0000", 2024-09-21T11:07:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9889999999999998e-06 2024-09-21T11:07:23.913 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.914 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.914 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.914 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.445587+0000", 2024-09-21T11:07:23.914 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0463710690000001 2024-09-21T11:07:23.914 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.914 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.914 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:07:23.914 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.445632+0000", 2024-09-21T11:07:23.914 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.4694999999999998e-05 2024-09-21T11:07:23.914 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.914 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:07:23.914 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.914 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.914 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.915 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:244207 58.9 58:935c82d5:::benchmark_data_smithi070_396992_object98248:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1619)", 2024-09-21T11:07:23.915 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:03.381304+0000", 2024-09-21T11:07:23.915 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 140.46212269599999, 2024-09-21T11:07:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.064389153, 2024-09-21T11:07:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:07:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:07:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:07:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:07:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:07:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 244207 2024-09-21T11:07:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:07:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:07:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.381304+0000", 2024-09-21T11:07:23.916 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:07:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.381304+0000", 2024-09-21T11:07:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:07:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.381306+0000", 2024-09-21T11:07:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4419999999999999e-06 2024-09-21T11:07:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:07:23.917 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.381311+0000", 2024-09-21T11:07:23.918 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2730000000000002e-06 2024-09-21T11:07:23.918 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.918 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.918 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:07:23.918 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.381323+0000", 2024-09-21T11:07:23.918 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2148e-05 2024-09-21T11:07:23.918 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.918 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.918 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:07:23.918 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.381331+0000", 2024-09-21T11:07:23.918 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7479999999999993e-06 2024-09-21T11:07:23.918 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.918 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.918 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.919 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.382091+0000", 2024-09-21T11:07:23.919 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000760369 2024-09-21T11:07:23.919 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.919 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.919 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:07:23.919 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.382227+0000", 2024-09-21T11:07:23.919 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000135449 2024-09-21T11:07:23.919 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.919 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.919 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6", 2024-09-21T11:07:23.919 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.382351+0000", 2024-09-21T11:07:23.919 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000123657 2024-09-21T11:07:23.919 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.919 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.920 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:07:23.920 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.394473+0000", 2024-09-21T11:07:23.920 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012122166 2024-09-21T11:07:23.920 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.920 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.920 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.920 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.399253+0000", 2024-09-21T11:07:23.920 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.004780678 2024-09-21T11:07:23.920 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.920 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.920 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:07:23.920 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.399262+0000", 2024-09-21T11:07:23.920 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0949999999999996e-06 2024-09-21T11:07:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.445653+0000", 2024-09-21T11:07:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0463916129999999 2024-09-21T11:07:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:07:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.445694+0000", 2024-09-21T11:07:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0515000000000001e-05 2024-09-21T11:07:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:07:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.921 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.922 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:244211 58.9 58:95dc5161:::benchmark_data_smithi070_396992_object98252:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1619)", 2024-09-21T11:07:23.922 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:03.381479+0000", 2024-09-21T11:07:23.922 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 140.46194821, 2024-09-21T11:07:23.922 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.089266603, 2024-09-21T11:07:23.922 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:07:23.922 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:07:23.922 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T11:07:23.922 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:07:23.922 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:07:23.922 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:23.922 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 244211 2024-09-21T11:07:23.922 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.922 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:07:23.923 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.923 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:07:23.923 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.381479+0000", 2024-09-21T11:07:23.923 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.923 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.923 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.923 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:07:23.923 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.381479+0000", 2024-09-21T11:07:23.923 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.923 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.923 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.923 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:07:23.923 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.381480+0000", 2024-09-21T11:07:23.923 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9500000000000001e-07 2024-09-21T11:07:23.924 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.924 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.924 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:07:23.924 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.381483+0000", 2024-09-21T11:07:23.924 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.704e-06 2024-09-21T11:07:23.924 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.924 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.924 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:07:23.924 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.381492+0000", 2024-09-21T11:07:23.924 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1929999999999993e-06 2024-09-21T11:07:23.924 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.924 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.924 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:07:23.924 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.381498+0000", 2024-09-21T11:07:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0079999999999999e-06 2024-09-21T11:07:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.391442+0000", 2024-09-21T11:07:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0099440380000000005 2024-09-21T11:07:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:07:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.391562+0000", 2024-09-21T11:07:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000120485 2024-09-21T11:07:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6", 2024-09-21T11:07:23.925 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.391689+0000", 2024-09-21T11:07:23.926 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012641 2024-09-21T11:07:23.926 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.926 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.926 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.926 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.399325+0000", 2024-09-21T11:07:23.926 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0076358939999999998 2024-09-21T11:07:23.926 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.926 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.926 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:07:23.926 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.399330+0000", 2024-09-21T11:07:23.926 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.879e-06 2024-09-21T11:07:23.926 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.926 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.926 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.445777+0000", 2024-09-21T11:07:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.046446985 2024-09-21T11:07:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for missing object", 2024-09-21T11:07:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.446002+0000", 2024-09-21T11:07:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00022478299999999999 2024-09-21T11:07:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.470669+0000", 2024-09-21T11:07:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.024666519000000001 2024-09-21T11:07:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.927 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.928 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:07:23.928 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.470746+0000", 2024-09-21T11:07:23.928 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6909999999999994e-05 2024-09-21T11:07:23.928 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.928 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:07:23.928 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.928 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.928 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.928 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:244216 58.9 58:9600a0ee:::benchmark_data_smithi070_396992_object98257:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1619)", 2024-09-21T11:07:23.928 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:03.387459+0000", 2024-09-21T11:07:23.928 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 140.455968297, 2024-09-21T11:07:23.928 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.083458569, 2024-09-21T11:07:23.928 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:07:23.928 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:07:23.928 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:23.929 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:07:23.929 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:07:23.929 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:23.929 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 244216 2024-09-21T11:07:23.929 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.929 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:07:23.929 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.929 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:07:23.929 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.387459+0000", 2024-09-21T11:07:23.929 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.929 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.929 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.929 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:07:23.930 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.387459+0000", 2024-09-21T11:07:23.930 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.930 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.930 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.930 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:07:23.930 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.387460+0000", 2024-09-21T11:07:23.930 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5519999999999999e-06 2024-09-21T11:07:23.930 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.930 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.930 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:07:23.930 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.387466+0000", 2024-09-21T11:07:23.930 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6219999999999997e-06 2024-09-21T11:07:23.930 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.931 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.931 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:07:23.931 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.387477+0000", 2024-09-21T11:07:23.931 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0560000000000001e-05 2024-09-21T11:07:23.931 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.931 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.931 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:07:23.931 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.387484+0000", 2024-09-21T11:07:23.931 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7260000000000002e-06 2024-09-21T11:07:23.931 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.931 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.931 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.931 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.399357+0000", 2024-09-21T11:07:23.931 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.011872417999999999 2024-09-21T11:07:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:07:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.399364+0000", 2024-09-21T11:07:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4100000000000002e-06 2024-09-21T11:07:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.446035+0000", 2024-09-21T11:07:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.046670429 2024-09-21T11:07:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:07:23.932 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.446135+0000", 2024-09-21T11:07:23.933 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000100388 2024-09-21T11:07:23.933 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.933 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.933 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6", 2024-09-21T11:07:23.933 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.446254+0000", 2024-09-21T11:07:23.933 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000119413 2024-09-21T11:07:23.933 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.933 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.933 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:23.933 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.452022+0000", 2024-09-21T11:07:23.933 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0057673450000000001 2024-09-21T11:07:23.933 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.933 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.933 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:07:23.934 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.461712+0000", 2024-09-21T11:07:23.934 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0096904969999999993 2024-09-21T11:07:23.934 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.934 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.934 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:23.934 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.470783+0000", 2024-09-21T11:07:23.934 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0090710159999999995 2024-09-21T11:07:23.934 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.934 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.934 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:07:23.934 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.470840+0000", 2024-09-21T11:07:23.934 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6711e-05 2024-09-21T11:07:23.934 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.934 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:07:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.470917+0000", 2024-09-21T11:07:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7482000000000001e-05 2024-09-21T11:07:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:07:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:244235 58.9 58:9e3fa55b:::benchmark_data_smithi070_396992_object98276:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1620)", 2024-09-21T11:07:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:03.416041+0000", 2024-09-21T11:07:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 140.427386377, 2024-09-21T11:07:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.055055697, 2024-09-21T11:07:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:07:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:07:23.935 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:07:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:07:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 244235 2024-09-21T11:07:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:07:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:07:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.416041+0000", 2024-09-21T11:07:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:07:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.416041+0000", 2024-09-21T11:07:23.936 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:07:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.416042+0000", 2024-09-21T11:07:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.322e-06 2024-09-21T11:07:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:07:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.416048+0000", 2024-09-21T11:07:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7790000000000001e-06 2024-09-21T11:07:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:07:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.416062+0000", 2024-09-21T11:07:23.937 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.454e-05 2024-09-21T11:07:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:07:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.416071+0000", 2024-09-21T11:07:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5490000000000006e-06 2024-09-21T11:07:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.424659+0000", 2024-09-21T11:07:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0085884269999999992 2024-09-21T11:07:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:07:23.938 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.424669+0000", 2024-09-21T11:07:23.939 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9739999999999997e-06 2024-09-21T11:07:23.939 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.939 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.939 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.939 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.448966+0000", 2024-09-21T11:07:23.939 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.024297051 2024-09-21T11:07:23.939 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.939 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.939 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:07:23.939 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.449102+0000", 2024-09-21T11:07:23.939 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013562599999999999 2024-09-21T11:07:23.939 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.939 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6", 2024-09-21T11:07:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.449255+0000", 2024-09-21T11:07:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000153125 2024-09-21T11:07:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.461699+0000", 2024-09-21T11:07:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012443641 2024-09-21T11:07:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:07:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.461717+0000", 2024-09-21T11:07:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8508999999999998e-05 2024-09-21T11:07:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.940 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.470952+0000", 2024-09-21T11:07:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0092349489999999992 2024-09-21T11:07:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:07:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.471011+0000", 2024-09-21T11:07:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8619999999999998e-05 2024-09-21T11:07:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:07:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.471097+0000", 2024-09-21T11:07:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5585000000000001e-05 2024-09-21T11:07:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.941 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:07:23.942 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.942 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.942 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.942 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:244236 58.9 58:961a9d18:::benchmark_data_smithi070_396992_object98277:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1620)", 2024-09-21T11:07:23.942 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:03.416943+0000", 2024-09-21T11:07:23.942 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 140.42648440900001, 2024-09-21T11:07:23.942 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0600050809999999, 2024-09-21T11:07:23.942 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:07:23.942 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:07:23.942 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:23.942 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:07:23.942 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:07:23.942 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:23.942 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 244236 2024-09-21T11:07:23.943 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.943 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:07:23.943 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.943 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:07:23.943 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.416943+0000", 2024-09-21T11:07:23.943 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.943 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.943 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.943 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:07:23.943 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.416943+0000", 2024-09-21T11:07:23.943 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.943 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.943 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.943 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:07:23.944 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.416944+0000", 2024-09-21T11:07:23.944 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.23e-07 2024-09-21T11:07:23.944 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.945 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.945 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:07:23.945 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.416947+0000", 2024-09-21T11:07:23.945 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8179999999999997e-06 2024-09-21T11:07:23.945 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:07:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.416956+0000", 2024-09-21T11:07:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.4300000000000006e-06 2024-09-21T11:07:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:07:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.416962+0000", 2024-09-21T11:07:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1709999999999999e-06 2024-09-21T11:07:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.435708+0000", 2024-09-21T11:07:23.946 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.018746432 2024-09-21T11:07:23.947 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.947 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.947 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:07:23.947 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.435718+0000", 2024-09-21T11:07:23.947 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2289999999999992e-06 2024-09-21T11:07:23.947 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.947 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.947 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.947 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.452230+0000", 2024-09-21T11:07:23.947 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0165119520000001 2024-09-21T11:07:23.947 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.947 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.947 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:07:23.947 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.452367+0000", 2024-09-21T11:07:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013741099999999999 2024-09-21T11:07:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6", 2024-09-21T11:07:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.452503+0000", 2024-09-21T11:07:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013585499999999999 2024-09-21T11:07:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.461890+0000", 2024-09-21T11:07:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0093870050000000003 2024-09-21T11:07:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:07:23.948 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.464830+0000", 2024-09-21T11:07:23.949 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002940579 2024-09-21T11:07:23.949 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.949 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.949 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:23.949 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.476804+0000", 2024-09-21T11:07:23.949 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.011973246999999999 2024-09-21T11:07:23.949 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.949 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.949 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:07:23.949 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.476859+0000", 2024-09-21T11:07:23.949 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5636000000000001e-05 2024-09-21T11:07:23.949 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.949 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.949 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:07:23.950 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.476948+0000", 2024-09-21T11:07:23.950 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.8492999999999995e-05 2024-09-21T11:07:23.950 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.950 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:07:23.950 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.950 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.950 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.950 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:244239 58.9 58:92586350:::benchmark_data_smithi070_396992_object98280:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1620)", 2024-09-21T11:07:23.950 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:03.417278+0000", 2024-09-21T11:07:23.950 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 140.42614875500001, 2024-09-21T11:07:23.950 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0598711409999999, 2024-09-21T11:07:23.950 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:07:23.950 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:07:23.950 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:07:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:07:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 244239 2024-09-21T11:07:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:07:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:07:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.417278+0000", 2024-09-21T11:07:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:07:23.951 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.417278+0000", 2024-09-21T11:07:23.952 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.952 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.952 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.952 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:07:23.952 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.417279+0000", 2024-09-21T11:07:23.952 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0070000000000001e-06 2024-09-21T11:07:23.952 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.952 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.952 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:07:23.952 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.417284+0000", 2024-09-21T11:07:23.952 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.882e-06 2024-09-21T11:07:23.952 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.952 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.952 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:07:23.952 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.417300+0000", 2024-09-21T11:07:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5387999999999999e-05 2024-09-21T11:07:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:07:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.417307+0000", 2024-09-21T11:07:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5970000000000003e-06 2024-09-21T11:07:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.435743+0000", 2024-09-21T11:07:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.018435610000000002 2024-09-21T11:07:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:07:23.953 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.435750+0000", 2024-09-21T11:07:23.954 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.25e-06 2024-09-21T11:07:23.954 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.954 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.954 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.954 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.455556+0000", 2024-09-21T11:07:23.954 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.019806035 2024-09-21T11:07:23.954 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.954 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.954 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:07:23.954 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.455664+0000", 2024-09-21T11:07:23.954 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000108163 2024-09-21T11:07:23.954 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.954 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.954 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6", 2024-09-21T11:07:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.455800+0000", 2024-09-21T11:07:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013549699999999999 2024-09-21T11:07:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.461926+0000", 2024-09-21T11:07:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0061256629999999999 2024-09-21T11:07:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:07:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.464845+0000", 2024-09-21T11:07:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0029191450000000002 2024-09-21T11:07:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.955 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:23.956 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.477000+0000", 2024-09-21T11:07:23.956 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012155416 2024-09-21T11:07:23.956 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.956 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.956 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:07:23.956 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.477058+0000", 2024-09-21T11:07:23.956 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8208000000000002e-05 2024-09-21T11:07:23.956 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.956 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.956 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:07:23.956 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.477150+0000", 2024-09-21T11:07:23.956 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1279999999999993e-05 2024-09-21T11:07:23.956 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.956 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:07:23.957 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.957 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.957 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.957 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:244266 58.9 58:91256c05:::benchmark_data_smithi070_396992_object98307:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1620)", 2024-09-21T11:07:23.957 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:03.443405+0000", 2024-09-21T11:07:23.957 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 140.40002185399999, 2024-09-21T11:07:23.957 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0340413509999999, 2024-09-21T11:07:23.957 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:07:23.957 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:07:23.957 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:23.957 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:07:23.957 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:07:23.957 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:23.957 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 244266 2024-09-21T11:07:23.958 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.958 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:07:23.958 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.958 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:07:23.958 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.443405+0000", 2024-09-21T11:07:23.958 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.958 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.958 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.958 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:07:23.958 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.443405+0000", 2024-09-21T11:07:23.958 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.958 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.958 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.959 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:07:23.959 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.443407+0000", 2024-09-21T11:07:23.959 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4559999999999999e-06 2024-09-21T11:07:23.959 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.959 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.959 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:07:23.959 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.443412+0000", 2024-09-21T11:07:23.959 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9849999999999997e-06 2024-09-21T11:07:23.959 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.959 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.959 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:07:23.959 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.443423+0000", 2024-09-21T11:07:23.959 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1674e-05 2024-09-21T11:07:23.959 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.960 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.960 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:07:23.960 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.443431+0000", 2024-09-21T11:07:23.960 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.08e-06 2024-09-21T11:07:23.960 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.960 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.960 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.960 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.443501+0000", 2024-09-21T11:07:23.960 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0125999999999999e-05 2024-09-21T11:07:23.960 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.960 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.960 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:07:23.960 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:03.443511+0000", 2024-09-21T11:07:23.960 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0437e-05 2024-09-21T11:07:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.458579+0000", 2024-09-21T11:07:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.015067935 2024-09-21T11:07:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:07:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.458706+0000", 2024-09-21T11:07:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000126792 2024-09-21T11:07:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 1,6", 2024-09-21T11:07:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.458838+0000", 2024-09-21T11:07:23.961 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013238900000000001 2024-09-21T11:07:23.962 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.962 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.962 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:23.962 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.464898+0000", 2024-09-21T11:07:23.962 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0060602720000000002 2024-09-21T11:07:23.962 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.962 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.962 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:07:23.962 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.464914+0000", 2024-09-21T11:07:23.962 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5889999999999999e-05 2024-09-21T11:07:23.962 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.962 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.962 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:23.962 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.477313+0000", 2024-09-21T11:07:23.963 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.012398665999999999 2024-09-21T11:07:23.963 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.963 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.963 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:07:23.963 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.477378+0000", 2024-09-21T11:07:23.963 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5418000000000003e-05 2024-09-21T11:07:23.963 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.963 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.963 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:07:23.963 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:04.477447+0000", 2024-09-21T11:07:23.963 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8231000000000005e-05 2024-09-21T11:07:23.963 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.963 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:07:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:284830 58.d 58:b17b9152:::benchmark_data_smithi070_396992_object138871:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1642)", 2024-09-21T11:07:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:53.542198+0000", 2024-09-21T11:07:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 90.301229254000006, 2024-09-21T11:07:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.043232656, 2024-09-21T11:07:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:07:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:07:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:07:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:07:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 284830 2024-09-21T11:07:23.964 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:07:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:07:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:53.542198+0000", 2024-09-21T11:07:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:07:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:53.542198+0000", 2024-09-21T11:07:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:07:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:53.542200+0000", 2024-09-21T11:07:23.965 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6729999999999999e-06 2024-09-21T11:07:23.966 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.966 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.966 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:07:23.966 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:53.542206+0000", 2024-09-21T11:07:23.966 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3069999999999999e-06 2024-09-21T11:07:23.966 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.966 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.966 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:07:23.966 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:53.542218+0000", 2024-09-21T11:07:23.966 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2228e-05 2024-09-21T11:07:23.966 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.966 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.966 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:07:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:53.542226+0000", 2024-09-21T11:07:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3839999999999999e-06 2024-09-21T11:07:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:53.542304+0000", 2024-09-21T11:07:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8887999999999994e-05 2024-09-21T11:07:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:07:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:53.542314+0000", 2024-09-21T11:07:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.8579999999999995e-06 2024-09-21T11:07:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.967 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.968 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:54.571953+0000", 2024-09-21T11:07:23.968 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0296384240000001 2024-09-21T11:07:23.968 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.968 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.968 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:07:23.968 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:54.572062+0000", 2024-09-21T11:07:23.968 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010882000000000001 2024-09-21T11:07:23.968 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.968 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.968 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,4", 2024-09-21T11:07:23.968 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:54.572230+0000", 2024-09-21T11:07:23.968 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016833500000000001 2024-09-21T11:07:23.968 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.968 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.968 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:54.575205+0000", 2024-09-21T11:07:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0029752070000000001 2024-09-21T11:07:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:07:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:54.585233+0000", 2024-09-21T11:07:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0100275 2024-09-21T11:07:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:54.585340+0000", 2024-09-21T11:07:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010770800000000001 2024-09-21T11:07:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.969 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:07:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:54.585394+0000", 2024-09-21T11:07:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4138999999999998e-05 2024-09-21T11:07:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:07:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:54.585431+0000", 2024-09-21T11:07:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6185e-05 2024-09-21T11:07:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:07:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.36567.0:284845 58.d 58:b77c06c5:::benchmark_data_smithi070_396992_object138886:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1642)", 2024-09-21T11:07:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T11:05:53.566954+0000", 2024-09-21T11:07:23.970 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 90.276473358000004, 2024-09-21T11:07:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0187329570000001, 2024-09-21T11:07:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T11:07:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T11:07:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T11:07:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.36567", 2024-09-21T11:07:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 284845 2024-09-21T11:07:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T11:07:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T11:07:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:53.566954+0000", 2024-09-21T11:07:23.971 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T11:07:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:53.566954+0000", 2024-09-21T11:07:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T11:07:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T11:07:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:53.566955+0000", 2024-09-21T11:07:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4109999999999999e-06 2024-09-21T11:07:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T11:07:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:53.566960+0000", 2024-09-21T11:07:23.972 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9030000000000003e-06 2024-09-21T11:07:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T11:07:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:53.566973+0000", 2024-09-21T11:07:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3117e-05 2024-09-21T11:07:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T11:07:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:53.566980+0000", 2024-09-21T11:07:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.579e-06 2024-09-21T11:07:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.973 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:53.567061+0000", 2024-09-21T11:07:23.974 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1427999999999994e-05 2024-09-21T11:07:23.974 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.974 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.974 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for peered", 2024-09-21T11:07:23.974 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:53.567070+0000", 2024-09-21T11:07:23.974 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.4859999999999997e-06 2024-09-21T11:07:23.974 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.974 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.974 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T11:07:23.974 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:54.575271+0000", 2024-09-21T11:07:23.974 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0082017130000001 2024-09-21T11:07:23.974 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.974 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.974 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T11:07:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:54.575362+0000", 2024-09-21T11:07:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0724999999999996e-05 2024-09-21T11:07:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 0,4", 2024-09-21T11:07:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:54.575469+0000", 2024-09-21T11:07:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010681200000000001 2024-09-21T11:07:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T11:07:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:54.585444+0000", 2024-09-21T11:07:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0099751749999999993 2024-09-21T11:07:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.975 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:54.585490+0000", 2024-09-21T11:07:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5849000000000001e-05 2024-09-21T11:07:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:54.585592+0000", 2024-09-21T11:07:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000101734 2024-09-21T11:07:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T11:07:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:54.585648+0000", 2024-09-21T11:07:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6118e-05 2024-09-21T11:07:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T11:07:23.976 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T11:07:23.977 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T11:07:23.977 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T11:05:54.585687+0000", 2024-09-21T11:07:23.977 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8906999999999999e-05 2024-09-21T11:07:23.977 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.977 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:07:23.977 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.977 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T11:07:23.977 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T11:07:23.977 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T11:07:24.863 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T11:07:25.078 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:07:25.394 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T11:07:25.394 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T11:07:25.394 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-21T11:07:25.531 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:25 smithi160 ceph-mon[101879]: pgmap v3339: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:25.598 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:07:25.677 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:25 smithi070 ceph-mon[107385]: pgmap v3339: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:25.677 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:25 smithi070 ceph-mon[112139]: pgmap v3339: 93 pgs: 93 active+clean; 378 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:26.750 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T11:07:26.750 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T11:07:26.750 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T11:07:26.750 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T11:07:27.314 INFO:teuthology.orchestra.run.smithi070.stderr:no valid command found; 10 closest matches: 2024-09-21T11:07:27.314 INFO:teuthology.orchestra.run.smithi070.stderr:0 2024-09-21T11:07:27.315 INFO:teuthology.orchestra.run.smithi070.stderr:1 2024-09-21T11:07:27.315 INFO:teuthology.orchestra.run.smithi070.stderr:2 2024-09-21T11:07:27.315 INFO:teuthology.orchestra.run.smithi070.stderr:abort 2024-09-21T11:07:27.315 INFO:teuthology.orchestra.run.smithi070.stderr:assert 2024-09-21T11:07:27.315 INFO:teuthology.orchestra.run.smithi070.stderr:bench [] [] [] [] 2024-09-21T11:07:27.315 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs debug_inject_read_zeros 2024-09-21T11:07:27.315 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs files list 2024-09-21T11:07:27.315 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs stats 2024-09-21T11:07:27.315 INFO:teuthology.orchestra.run.smithi070.stderr:bluestore allocator dump block 2024-09-21T11:07:27.315 INFO:teuthology.orchestra.run.smithi070.stderr:Error EINVAL: invalid command 2024-09-21T11:07:27.555 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T11:07:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:27 smithi160 ceph-mon[101879]: pgmap v3340: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:27 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T11:07:27.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:27 smithi070 ceph-mon[107385]: pgmap v3340: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:27.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:27 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T11:07:27.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:27 smithi070 ceph-mon[112139]: pgmap v3340: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:27.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:27 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T11:07:28.119 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T11:07:28.119 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T11:07:28.119 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-21T11:07:28.411 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:07:28.924 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:07:29.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:29 smithi160 ceph-mon[101879]: pgmap v3341: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:29.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:29 smithi070 ceph-mon[107385]: pgmap v3341: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:29.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:29 smithi070 ceph-mon[112139]: pgmap v3341: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:30.723 INFO:teuthology.orchestra.run.smithi070.stderr:no valid command found; 10 closest matches: 2024-09-21T11:07:30.724 INFO:teuthology.orchestra.run.smithi070.stderr:0 2024-09-21T11:07:30.724 INFO:teuthology.orchestra.run.smithi070.stderr:1 2024-09-21T11:07:30.724 INFO:teuthology.orchestra.run.smithi070.stderr:2 2024-09-21T11:07:30.724 INFO:teuthology.orchestra.run.smithi070.stderr:abort 2024-09-21T11:07:30.724 INFO:teuthology.orchestra.run.smithi070.stderr:assert 2024-09-21T11:07:30.724 INFO:teuthology.orchestra.run.smithi070.stderr:bench [] [] [] [] 2024-09-21T11:07:30.724 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs debug_inject_read_zeros 2024-09-21T11:07:30.724 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs files list 2024-09-21T11:07:30.724 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs stats 2024-09-21T11:07:30.724 INFO:teuthology.orchestra.run.smithi070.stderr:bluestore allocator dump block 2024-09-21T11:07:30.724 INFO:teuthology.orchestra.run.smithi070.stderr:Error EINVAL: invalid command 2024-09-21T11:07:30.793 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T11:07:30.793 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T11:07:30.793 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T11:07:30.793 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T11:07:30.793 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T11:07:31.475 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T11:07:31.475 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T11:07:31.476 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-21T11:07:31.484 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T11:07:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:31 smithi160 ceph-mon[101879]: pgmap v3342: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:31.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:31 smithi070 ceph-mon[107385]: pgmap v3342: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:31.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:31 smithi070 ceph-mon[112139]: pgmap v3342: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:32.079 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:07:32.344 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:07:33.448 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:07:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:07:33] "GET /metrics HTTP/1.1" 200 34260 "" "Prometheus/2.51.0" 2024-09-21T11:07:33.836 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:33 smithi160 ceph-mon[101879]: pgmap v3343: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:33.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:33 smithi070 ceph-mon[107385]: pgmap v3343: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:33 smithi070 ceph-mon[112139]: pgmap v3343: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:34.300 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T11:07:34.300 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T11:07:34.300 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T11:07:34.300 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T11:07:34.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.300 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:29067 57.9 57:964842a6:::benchmark_data_smithi070_360420_object29066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1537)", 2024-09-21T11:07:34.300 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:58:32.636312+0000", 2024-09-21T11:07:34.301 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 541.66036282499999, 2024-09-21T11:07:34.301 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0001596829999999, 2024-09-21T11:07:34.301 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:34.301 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T11:07:34.301 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:34.301 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T11:07:34.301 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:07:34.301 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 29067 2024-09-21T11:07:34.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.301 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:34.301 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.301 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:34.301 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.636312+0000", 2024-09-21T11:07:34.301 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.302 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.302 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.302 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:34.302 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.636312+0000", 2024-09-21T11:07:34.302 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.302 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.302 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.302 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:34.302 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.636311+0000", 2024-09-21T11:07:34.302 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T11:07:34.302 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.302 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.302 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:34.303 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.637030+0000", 2024-09-21T11:07:34.303 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00071977200000000003 2024-09-21T11:07:34.303 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.303 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.303 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:34.303 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.637045+0000", 2024-09-21T11:07:34.303 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4175999999999999e-05 2024-09-21T11:07:34.303 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.303 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.303 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:34.303 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.637054+0000", 2024-09-21T11:07:34.303 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0489999999999993e-06 2024-09-21T11:07:34.303 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.304 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.304 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.304 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.637103+0000", 2024-09-21T11:07:34.304 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9271999999999999e-05 2024-09-21T11:07:34.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.304 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.304 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op must wait for map", 2024-09-21T11:07:34.304 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.637110+0000", 2024-09-21T11:07:34.304 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7989999999999996e-06 2024-09-21T11:07:34.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.304 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.304 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.304 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.648728+0000", 2024-09-21T11:07:34.304 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011618694000000001 2024-09-21T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.648738+0000", 2024-09-21T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0131e-05 2024-09-21T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.636419+0000", 2024-09-21T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98768005199999998 2024-09-21T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.636472+0000", 2024-09-21T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3508000000000003e-05 2024-09-21T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:29090 57.9 57:9c9b276d:::benchmark_data_smithi070_360420_object29089:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1537)", 2024-09-21T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:58:32.649590+0000", 2024-09-21T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 541.64708502200006, 2024-09-21T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99083443800000004, 2024-09-21T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T11:07:34.307 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:07:34.307 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 29090 2024-09-21T11:07:34.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.307 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:34.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.307 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:34.307 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.649590+0000", 2024-09-21T11:07:34.307 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.307 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:34.307 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.649590+0000", 2024-09-21T11:07:34.307 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.308 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.308 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:34.308 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.649588+0000", 2024-09-21T11:07:34.308 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 2024-09-21T11:07:34.308 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.308 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.308 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:34.308 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.649630+0000", 2024-09-21T11:07:34.308 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2567000000000001e-05 2024-09-21T11:07:34.308 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.308 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.308 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:34.308 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.649644+0000", 2024-09-21T11:07:34.308 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3338e-05 2024-09-21T11:07:34.308 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.309 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.309 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:34.309 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.649652+0000", 2024-09-21T11:07:34.309 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8439999999999998e-06 2024-09-21T11:07:34.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.309 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.309 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.309 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.649774+0000", 2024-09-21T11:07:34.309 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012225800000000001 2024-09-21T11:07:34.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.309 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.309 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:34.309 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.649784+0000", 2024-09-21T11:07:34.309 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7680000000000003e-06 2024-09-21T11:07:34.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.310 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.310 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.310 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.636744+0000", 2024-09-21T11:07:34.310 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98696035900000001 2024-09-21T11:07:34.310 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.310 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.310 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:34.310 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.636871+0000", 2024-09-21T11:07:34.310 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012668 2024-09-21T11:07:34.310 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.310 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.310 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6", 2024-09-21T11:07:34.310 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.637029+0000", 2024-09-21T11:07:34.310 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015828899999999999 2024-09-21T11:07:34.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.311 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.311 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.640258+0000", 2024-09-21T11:07:34.311 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003229216 2024-09-21T11:07:34.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.311 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.311 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.640287+0000", 2024-09-21T11:07:34.311 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8941999999999998e-05 2024-09-21T11:07:34.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.311 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:34.311 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.640330+0000", 2024-09-21T11:07:34.312 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.295e-05 2024-09-21T11:07:34.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.312 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.312 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:34.312 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.640386+0000", 2024-09-21T11:07:34.312 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6097999999999997e-05 2024-09-21T11:07:34.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.312 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.312 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:34.312 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.640425+0000", 2024-09-21T11:07:34.312 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8421999999999999e-05 2024-09-21T11:07:34.312 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.312 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:34.312 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.313 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.313 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:29099 57.9 57:91a3677e:::benchmark_data_smithi070_360420_object29098:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1537)", 2024-09-21T11:07:34.313 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T10:58:32.658256+0000", 2024-09-21T11:07:34.313 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 541.63841868999998, 2024-09-21T11:07:34.313 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.983558767, 2024-09-21T11:07:34.313 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:34.313 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:34.313 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:34.313 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T11:07:34.314 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:07:34.314 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 29099 2024-09-21T11:07:34.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.314 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:34.314 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.314 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:34.314 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.658256+0000", 2024-09-21T11:07:34.314 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.315 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:34.315 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.658256+0000", 2024-09-21T11:07:34.315 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.315 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:34.315 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.658255+0000", 2024-09-21T11:07:34.315 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:07:34.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.315 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:34.316 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.659800+0000", 2024-09-21T11:07:34.316 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0015451600000000001 2024-09-21T11:07:34.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.316 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.316 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:34.316 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.659812+0000", 2024-09-21T11:07:34.316 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1970999999999999e-05 2024-09-21T11:07:34.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.316 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.316 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:34.316 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.659820+0000", 2024-09-21T11:07:34.316 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.259e-06 2024-09-21T11:07:34.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.660111+0000", 2024-09-21T11:07:34.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00029062400000000001 2024-09-21T11:07:34.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:34.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:32.660122+0000", 2024-09-21T11:07:34.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0842999999999999e-05 2024-09-21T11:07:34.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.638033+0000", 2024-09-21T11:07:34.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.97791153399999997 2024-09-21T11:07:34.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.318 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.318 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:34.318 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.638163+0000", 2024-09-21T11:07:34.318 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012933899999999999 2024-09-21T11:07:34.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.318 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.318 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6", 2024-09-21T11:07:34.318 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.638264+0000", 2024-09-21T11:07:34.318 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000101463 2024-09-21T11:07:34.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.318 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.318 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.318 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.640323+0000", 2024-09-21T11:07:34.319 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0020588540000000002 2024-09-21T11:07:34.319 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.319 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.319 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.319 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.640491+0000", 2024-09-21T11:07:34.319 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016825600000000001 2024-09-21T11:07:34.319 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.319 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.319 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:34.319 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.641727+0000", 2024-09-21T11:07:34.319 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0012361900000000001 2024-09-21T11:07:34.319 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.319 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.320 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:34.320 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.641779+0000", 2024-09-21T11:07:34.320 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1511000000000002e-05 2024-09-21T11:07:34.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.320 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.320 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:34.320 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T10:58:33.641815+0000", 2024-09-21T11:07:34.320 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6361e-05 2024-09-21T11:07:34.320 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.320 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:34.320 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.320 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.320 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:219130 58.2 58:4c3b0c88:::benchmark_data_smithi070_396992_object73171:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1603)", 2024-09-21T11:07:34.321 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:33.496261+0000", 2024-09-21T11:07:34.321 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 180.800414131, 2024-09-21T11:07:34.321 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1229929470000002, 2024-09-21T11:07:34.321 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:34.321 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T11:07:34.321 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:34.321 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:34.321 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:34.321 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 219130 2024-09-21T11:07:34.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.321 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:34.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.321 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:34.321 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.496261+0000", 2024-09-21T11:07:34.322 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.322 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.322 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:34.322 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.496261+0000", 2024-09-21T11:07:34.322 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.322 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.322 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:34.322 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.496261+0000", 2024-09-21T11:07:34.322 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T11:07:34.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.322 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.322 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:34.323 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.496263+0000", 2024-09-21T11:07:34.323 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8779999999999998e-06 2024-09-21T11:07:34.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.323 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:34.323 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.496270+0000", 2024-09-21T11:07:34.323 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8199999999999999e-06 2024-09-21T11:07:34.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.323 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:34.323 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.496275+0000", 2024-09-21T11:07:34.323 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1089999999999997e-06 2024-09-21T11:07:34.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.323 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.323 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.496317+0000", 2024-09-21T11:07:34.324 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1937000000000002e-05 2024-09-21T11:07:34.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.324 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:34.324 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.496423+0000", 2024-09-21T11:07:34.324 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010550199999999999 2024-09-21T11:07:34.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.324 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,7", 2024-09-21T11:07:34.324 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.496521+0000", 2024-09-21T11:07:34.324 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8523999999999994e-05 2024-09-21T11:07:34.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.324 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.325 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.501813+0000", 2024-09-21T11:07:34.325 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.005291352 2024-09-21T11:07:34.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.325 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.325 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:34.325 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.501819+0000", 2024-09-21T11:07:34.325 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1449999999999996e-06 2024-09-21T11:07:34.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.325 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.325 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.325 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.543949+0000", 2024-09-21T11:07:34.325 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.042130244999999997 2024-09-21T11:07:34.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.325 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.325 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:34.326 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.543959+0000", 2024-09-21T11:07:34.326 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6250000000000002e-06 2024-09-21T11:07:34.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.326 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.326 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.526163+0000", 2024-09-21T11:07:34.326 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98220441000000003 2024-09-21T11:07:34.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.326 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:34.326 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.527342+0000", 2024-09-21T11:07:34.326 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0011790520000000001 2024-09-21T11:07:34.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.327 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.327 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.619188+0000", 2024-09-21T11:07:34.327 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.091845706 2024-09-21T11:07:34.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.327 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.327 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:34.327 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.619254+0000", 2024-09-21T11:07:34.327 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6134000000000001e-05 2024-09-21T11:07:34.327 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.327 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:34.327 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.327 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.327 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:219131 58.2 58:44c6283b:::benchmark_data_smithi070_396992_object73172:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1603)", 2024-09-21T11:07:34.328 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:33.497488+0000", 2024-09-21T11:07:34.328 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 180.79918745399999, 2024-09-21T11:07:34.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1220704669999999, 2024-09-21T11:07:34.328 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:34.328 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T11:07:34.328 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:34.328 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:34.328 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:34.328 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 219131 2024-09-21T11:07:34.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.328 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:34.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.328 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:34.328 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.497488+0000", 2024-09-21T11:07:34.329 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.329 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.329 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:34.329 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.497488+0000", 2024-09-21T11:07:34.329 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.329 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.329 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:34.329 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.497487+0000", 2024-09-21T11:07:34.329 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:07:34.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.329 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.330 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:34.330 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.497492+0000", 2024-09-21T11:07:34.330 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5820000000000001e-06 2024-09-21T11:07:34.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.330 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:34.330 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.497504+0000", 2024-09-21T11:07:34.330 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1603000000000001e-05 2024-09-21T11:07:34.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.330 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:34.330 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.497511+0000", 2024-09-21T11:07:34.330 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.148e-06 2024-09-21T11:07:34.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.331 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.331 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.499179+0000", 2024-09-21T11:07:34.331 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001668037 2024-09-21T11:07:34.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.331 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:34.331 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.499275+0000", 2024-09-21T11:07:34.331 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5736000000000006e-05 2024-09-21T11:07:34.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.331 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,7", 2024-09-21T11:07:34.332 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.499376+0000", 2024-09-21T11:07:34.332 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010124399999999999 2024-09-21T11:07:34.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.332 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:34.332 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.501823+0000", 2024-09-21T11:07:34.332 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0024471929999999999 2024-09-21T11:07:34.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.332 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.332 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.501937+0000", 2024-09-21T11:07:34.332 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000113743 2024-09-21T11:07:34.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.333 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.543986+0000", 2024-09-21T11:07:34.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.042049572 2024-09-21T11:07:34.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.333 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:34.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.543995+0000", 2024-09-21T11:07:34.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3790000000000006e-06 2024-09-21T11:07:34.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.333 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.534350+0000", 2024-09-21T11:07:34.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99035547199999996 2024-09-21T11:07:34.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.334 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:34.334 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.534361+0000", 2024-09-21T11:07:34.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0737000000000001e-05 2024-09-21T11:07:34.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.334 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.334 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.334 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.619518+0000", 2024-09-21T11:07:34.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0851569379999999 2024-09-21T11:07:34.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.334 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.334 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:34.334 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.619558+0000", 2024-09-21T11:07:34.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0209000000000001e-05 2024-09-21T11:07:34.335 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.335 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:34.335 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.335 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.335 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.335 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:219137 58.6 58:688d16c4:::benchmark_data_smithi070_396992_object73178:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1603)", 2024-09-21T11:07:34.335 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:33.505099+0000", 2024-09-21T11:07:34.335 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 180.791575878, 2024-09-21T11:07:34.335 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1234349250000002, 2024-09-21T11:07:34.335 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:34.335 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T11:07:34.335 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:34.335 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:34.335 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:34.335 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 219137 2024-09-21T11:07:34.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.336 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:34.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.336 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:34.336 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.505099+0000", 2024-09-21T11:07:34.336 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.336 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:34.336 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.505099+0000", 2024-09-21T11:07:34.336 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.337 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:34.337 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.505098+0000", 2024-09-21T11:07:34.337 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:07:34.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.337 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:34.337 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.505107+0000", 2024-09-21T11:07:34.337 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3790000000000006e-06 2024-09-21T11:07:34.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.337 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:34.337 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.505118+0000", 2024-09-21T11:07:34.337 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1562999999999999e-05 2024-09-21T11:07:34.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.338 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:34.338 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.505125+0000", 2024-09-21T11:07:34.338 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6950000000000003e-06 2024-09-21T11:07:34.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.338 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.338 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.338 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.505170+0000", 2024-09-21T11:07:34.338 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5532999999999999e-05 2024-09-21T11:07:34.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.338 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.338 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:34.338 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.505280+0000", 2024-09-21T11:07:34.338 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010936499999999999 2024-09-21T11:07:34.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.339 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.339 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,7", 2024-09-21T11:07:34.339 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.505383+0000", 2024-09-21T11:07:34.339 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000103391 2024-09-21T11:07:34.339 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.339 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.339 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:34.339 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.507870+0000", 2024-09-21T11:07:34.339 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002486556 2024-09-21T11:07:34.339 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.339 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.339 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.339 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.509821+0000", 2024-09-21T11:07:34.339 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0019513009999999999 2024-09-21T11:07:34.339 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.340 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.340 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.340 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.539798+0000", 2024-09-21T11:07:34.340 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.029977200999999998 2024-09-21T11:07:34.340 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.340 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.340 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:34.340 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.539809+0000", 2024-09-21T11:07:34.340 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1228e-05 2024-09-21T11:07:34.340 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.340 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.340 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.340 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.519794+0000", 2024-09-21T11:07:34.340 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.97998448500000002 2024-09-21T11:07:34.340 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.340 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.341 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:34.341 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.519808+0000", 2024-09-21T11:07:34.341 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3703000000000001e-05 2024-09-21T11:07:34.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.341 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.341 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.341 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.628397+0000", 2024-09-21T11:07:34.341 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.108589174 2024-09-21T11:07:34.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.341 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.341 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:34.341 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.628534+0000", 2024-09-21T11:07:34.341 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013744000000000001 2024-09-21T11:07:34.342 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.342 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:34.342 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.342 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.342 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.342 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:219161 58.6 58:68ba07c2:::benchmark_data_smithi070_396992_object73202:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1604)", 2024-09-21T11:07:34.342 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:33.552233+0000", 2024-09-21T11:07:34.342 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 180.74444172, 2024-09-21T11:07:34.342 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0861848489999999, 2024-09-21T11:07:34.342 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:34.342 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:34.342 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:34.342 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:34.342 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:34.343 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 219161 2024-09-21T11:07:34.343 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.343 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:34.343 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.343 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:34.343 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552233+0000", 2024-09-21T11:07:34.343 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.343 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.343 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.343 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:34.343 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552233+0000", 2024-09-21T11:07:34.343 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.343 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.343 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.343 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:34.344 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552232+0000", 2024-09-21T11:07:34.344 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:07:34.344 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.344 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.344 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:34.344 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552238+0000", 2024-09-21T11:07:34.344 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6729999999999997e-06 2024-09-21T11:07:34.344 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.344 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.344 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:34.344 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552249+0000", 2024-09-21T11:07:34.344 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0577999999999999e-05 2024-09-21T11:07:34.344 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.344 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.344 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:34.345 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552255+0000", 2024-09-21T11:07:34.345 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8000000000000001e-06 2024-09-21T11:07:34.345 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.345 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.345 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.345 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552316+0000", 2024-09-21T11:07:34.345 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0603000000000002e-05 2024-09-21T11:07:34.345 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.345 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.345 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:34.345 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552327+0000", 2024-09-21T11:07:34.345 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0618999999999999e-05 2024-09-21T11:07:34.345 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.345 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.346 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.346 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.521117+0000", 2024-09-21T11:07:34.346 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96879046300000005 2024-09-21T11:07:34.346 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.346 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.346 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:34.346 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.521126+0000", 2024-09-21T11:07:34.346 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1030000000000008e-06 2024-09-21T11:07:34.346 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.346 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.346 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.346 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.628825+0000", 2024-09-21T11:07:34.346 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.107698944 2024-09-21T11:07:34.346 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.347 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.347 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:34.347 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.628951+0000", 2024-09-21T11:07:34.347 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012609200000000001 2024-09-21T11:07:34.347 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.347 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.347 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6,7", 2024-09-21T11:07:34.347 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.629444+0000", 2024-09-21T11:07:34.347 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00049303699999999999 2024-09-21T11:07:34.347 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.347 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.347 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.347 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.635982+0000", 2024-09-21T11:07:34.347 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0065374029999999998 2024-09-21T11:07:34.348 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.348 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.348 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.348 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.638172+0000", 2024-09-21T11:07:34.348 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002190642 2024-09-21T11:07:34.348 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.348 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.348 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:34.348 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.638181+0000", 2024-09-21T11:07:34.348 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9190000000000005e-06 2024-09-21T11:07:34.348 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.348 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.348 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.348 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.638259+0000", 2024-09-21T11:07:34.349 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7583000000000006e-05 2024-09-21T11:07:34.349 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.349 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.349 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:34.349 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.638308+0000", 2024-09-21T11:07:34.349 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9605999999999999e-05 2024-09-21T11:07:34.349 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.349 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.349 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:34.349 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.638418+0000", 2024-09-21T11:07:34.349 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000109844 2024-09-21T11:07:34.349 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.349 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:34.349 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.349 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.350 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.350 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:219162 58.7 58:e30a89ab:::benchmark_data_smithi070_396992_object73203:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1604)", 2024-09-21T11:07:34.350 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:33.552486+0000", 2024-09-21T11:07:34.350 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 180.74418869300001, 2024-09-21T11:07:34.350 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0599404050000001, 2024-09-21T11:07:34.350 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:34.350 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:34.350 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:34.350 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:34.350 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:34.350 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 219162 2024-09-21T11:07:34.350 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.350 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:34.350 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.350 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:34.351 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552486+0000", 2024-09-21T11:07:34.351 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.351 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.351 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.351 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:34.351 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552486+0000", 2024-09-21T11:07:34.351 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.351 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.351 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.351 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:34.351 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552486+0000", 2024-09-21T11:07:34.351 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:07:34.351 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.352 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.352 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:34.352 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552491+0000", 2024-09-21T11:07:34.352 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0810000000000001e-06 2024-09-21T11:07:34.352 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.352 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.352 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:34.352 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552503+0000", 2024-09-21T11:07:34.352 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2281e-05 2024-09-21T11:07:34.352 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.352 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.352 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:34.352 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552513+0000", 2024-09-21T11:07:34.352 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6779999999999995e-06 2024-09-21T11:07:34.352 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.353 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.353 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.353 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552552+0000", 2024-09-21T11:07:34.353 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9700000000000003e-05 2024-09-21T11:07:34.353 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.353 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.353 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:34.353 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.552557+0000", 2024-09-21T11:07:34.353 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.152e-06 2024-09-21T11:07:34.353 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.353 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.353 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.353 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.608525+0000", 2024-09-21T11:07:34.353 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0559673469999999 2024-09-21T11:07:34.353 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.354 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.354 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:34.354 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.608645+0000", 2024-09-21T11:07:34.354 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011995 2024-09-21T11:07:34.354 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.354 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.354 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6,7", 2024-09-21T11:07:34.354 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.608993+0000", 2024-09-21T11:07:34.354 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034864199999999997 2024-09-21T11:07:34.354 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.354 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.354 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:34.354 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.611354+0000", 2024-09-21T11:07:34.356 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0023607699999999999 2024-09-21T11:07:34.356 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.356 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.356 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.356 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.611659+0000", 2024-09-21T11:07:34.356 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00030510800000000001 2024-09-21T11:07:34.356 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.356 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.356 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.356 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.612102+0000", 2024-09-21T11:07:34.356 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000442433 2024-09-21T11:07:34.356 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.357 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.357 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.357 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.612291+0000", 2024-09-21T11:07:34.357 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00018962300000000001 2024-09-21T11:07:34.357 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.357 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.357 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:34.357 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.612330+0000", 2024-09-21T11:07:34.357 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8424999999999999e-05 2024-09-21T11:07:34.357 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.357 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.357 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:34.357 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.612427+0000", 2024-09-21T11:07:34.357 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7114999999999994e-05 2024-09-21T11:07:34.358 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.358 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:34.358 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.358 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.358 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.358 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:219166 58.2 58:4160bb5b:::benchmark_data_smithi070_396992_object73207:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1604)", 2024-09-21T11:07:34.358 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:33.558581+0000", 2024-09-21T11:07:34.358 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 180.73809387599999, 2024-09-21T11:07:34.358 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0708711590000002, 2024-09-21T11:07:34.358 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:34.358 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:34.358 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:34.358 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:34.358 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:34.359 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 219166 2024-09-21T11:07:34.359 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.359 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:34.359 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.359 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:34.359 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.558581+0000", 2024-09-21T11:07:34.359 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.359 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.359 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.359 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:34.359 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.558581+0000", 2024-09-21T11:07:34.359 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.359 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.359 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.360 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:34.360 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.558580+0000", 2024-09-21T11:07:34.360 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:07:34.360 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.360 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.360 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:34.360 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.558586+0000", 2024-09-21T11:07:34.360 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.711e-06 2024-09-21T11:07:34.360 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.360 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.360 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:34.360 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.558597+0000", 2024-09-21T11:07:34.360 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1251e-05 2024-09-21T11:07:34.360 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.361 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:34.361 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.558603+0000", 2024-09-21T11:07:34.361 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6289999999999998e-06 2024-09-21T11:07:34.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.361 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.361 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.558655+0000", 2024-09-21T11:07:34.361 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2666e-05 2024-09-21T11:07:34.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.361 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:34.361 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.558666+0000", 2024-09-21T11:07:34.361 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0740999999999999e-05 2024-09-21T11:07:34.362 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.362 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.362 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.362 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.534381+0000", 2024-09-21T11:07:34.362 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.97571475900000004 2024-09-21T11:07:34.362 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.362 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.362 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:34.362 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.534387+0000", 2024-09-21T11:07:34.362 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.263e-06 2024-09-21T11:07:34.362 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.362 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.362 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.362 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.619594+0000", 2024-09-21T11:07:34.363 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.085206838 2024-09-21T11:07:34.363 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.363 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.363 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:34.363 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.619740+0000", 2024-09-21T11:07:34.363 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000145582 2024-09-21T11:07:34.363 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.363 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.363 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,3,7", 2024-09-21T11:07:34.363 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.620170+0000", 2024-09-21T11:07:34.363 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00043068300000000001 2024-09-21T11:07:34.363 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.363 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.363 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.364 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.626730+0000", 2024-09-21T11:07:34.364 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0065597329999999999 2024-09-21T11:07:34.364 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.364 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.364 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:34.364 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.626739+0000", 2024-09-21T11:07:34.364 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0699999999999996e-06 2024-09-21T11:07:34.364 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.364 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.364 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.364 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.626768+0000", 2024-09-21T11:07:34.364 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8756999999999999e-05 2024-09-21T11:07:34.364 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.364 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.364 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.365 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.629320+0000", 2024-09-21T11:07:34.365 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002551967 2024-09-21T11:07:34.365 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.365 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.365 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:34.365 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.629354+0000", 2024-09-21T11:07:34.365 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3689999999999998e-05 2024-09-21T11:07:34.365 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.365 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.365 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:34.365 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.629452+0000", 2024-09-21T11:07:34.365 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8882000000000006e-05 2024-09-21T11:07:34.365 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.365 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:34.366 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.366 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.366 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.366 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:219171 58.2 58:4bc6147e:::benchmark_data_smithi070_396992_object73212:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1604)", 2024-09-21T11:07:34.366 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:33.567026+0000", 2024-09-21T11:07:34.366 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 180.72964964900001, 2024-09-21T11:07:34.366 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0626493610000001, 2024-09-21T11:07:34.366 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:34.366 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:34.366 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:34.366 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:34.366 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:34.366 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 219171 2024-09-21T11:07:34.367 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.367 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:34.367 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.367 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:34.367 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.567026+0000", 2024-09-21T11:07:34.367 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.367 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.367 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.367 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:34.367 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.567026+0000", 2024-09-21T11:07:34.367 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.367 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.367 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.367 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:34.368 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.567024+0000", 2024-09-21T11:07:34.368 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:07:34.368 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.368 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.368 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:34.368 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.567030+0000", 2024-09-21T11:07:34.368 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7520000000000002e-06 2024-09-21T11:07:34.368 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.368 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.368 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:34.368 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.567043+0000", 2024-09-21T11:07:34.368 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2942000000000001e-05 2024-09-21T11:07:34.368 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.368 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.368 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:34.369 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.567052+0000", 2024-09-21T11:07:34.369 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8780000000000004e-06 2024-09-21T11:07:34.369 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.369 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.369 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.369 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.567108+0000", 2024-09-21T11:07:34.369 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5476999999999998e-05 2024-09-21T11:07:34.369 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.369 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.369 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:34.369 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.567117+0000", 2024-09-21T11:07:34.369 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9229999999999997e-06 2024-09-21T11:07:34.369 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.370 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.370 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.370 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.534405+0000", 2024-09-21T11:07:34.370 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96728771599999996 2024-09-21T11:07:34.370 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.370 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.370 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:34.370 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.534411+0000", 2024-09-21T11:07:34.370 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1199999999999999e-06 2024-09-21T11:07:34.370 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.370 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.370 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.370 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.624395+0000", 2024-09-21T11:07:34.370 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0899832540000001 2024-09-21T11:07:34.371 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.371 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.371 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:34.371 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.624527+0000", 2024-09-21T11:07:34.371 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013255299999999999 2024-09-21T11:07:34.371 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.371 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.371 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,3,7", 2024-09-21T11:07:34.371 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.624672+0000", 2024-09-21T11:07:34.371 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014475999999999999 2024-09-21T11:07:34.371 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.371 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.371 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.371 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.626796+0000", 2024-09-21T11:07:34.372 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021244319999999999 2024-09-21T11:07:34.372 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.372 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.372 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.372 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.626824+0000", 2024-09-21T11:07:34.372 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7531000000000001e-05 2024-09-21T11:07:34.372 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.372 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.372 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:34.372 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.629012+0000", 2024-09-21T11:07:34.372 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021877450000000001 2024-09-21T11:07:34.372 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.372 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.372 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.372 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.629484+0000", 2024-09-21T11:07:34.373 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00047253900000000001 2024-09-21T11:07:34.373 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.373 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.373 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:34.373 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.629570+0000", 2024-09-21T11:07:34.373 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5729000000000005e-05 2024-09-21T11:07:34.373 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.373 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.373 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:34.373 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.629675+0000", 2024-09-21T11:07:34.373 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000105073 2024-09-21T11:07:34.373 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.373 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:34.373 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.374 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.374 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.374 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:219173 58.6 58:6a576e84:::benchmark_data_smithi070_396992_object73214:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1604)", 2024-09-21T11:07:34.374 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:33.572033+0000", 2024-09-21T11:07:34.374 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 180.72464241, 2024-09-21T11:07:34.374 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0691706920000001, 2024-09-21T11:07:34.374 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:34.374 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:34.374 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:34.374 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:34.374 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:34.374 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 219173 2024-09-21T11:07:34.374 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.375 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:34.375 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.375 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:34.375 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.572033+0000", 2024-09-21T11:07:34.375 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.375 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.375 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.375 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:34.375 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.572033+0000", 2024-09-21T11:07:34.375 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.375 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.375 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.375 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:34.375 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.572032+0000", 2024-09-21T11:07:34.376 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:07:34.376 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.376 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.376 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:34.376 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.572036+0000", 2024-09-21T11:07:34.376 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1899999999999997e-06 2024-09-21T11:07:34.376 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.376 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.376 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:34.376 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.572046+0000", 2024-09-21T11:07:34.376 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5170000000000005e-06 2024-09-21T11:07:34.376 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.376 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.377 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:34.377 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.572053+0000", 2024-09-21T11:07:34.377 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.041e-06 2024-09-21T11:07:34.377 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.377 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.377 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.377 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.572088+0000", 2024-09-21T11:07:34.377 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5231999999999997e-05 2024-09-21T11:07:34.377 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.377 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.377 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:34.377 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.572094+0000", 2024-09-21T11:07:34.377 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9789999999999998e-06 2024-09-21T11:07:34.377 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.377 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.378 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.378 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.521150+0000", 2024-09-21T11:07:34.378 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.94905558000000001 2024-09-21T11:07:34.378 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.378 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.378 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:34.378 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.521157+0000", 2024-09-21T11:07:34.378 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2490000000000004e-06 2024-09-21T11:07:34.378 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.378 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.378 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.378 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.633842+0000", 2024-09-21T11:07:34.378 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1126852979999999 2024-09-21T11:07:34.379 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.379 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.379 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:34.379 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.633953+0000", 2024-09-21T11:07:34.379 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011058200000000001 2024-09-21T11:07:34.379 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.379 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.379 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6,7", 2024-09-21T11:07:34.379 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.634074+0000", 2024-09-21T11:07:34.379 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000121488 2024-09-21T11:07:34.379 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.379 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.379 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:34.379 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.638189+0000", 2024-09-21T11:07:34.379 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0041143619999999999 2024-09-21T11:07:34.380 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.380 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.380 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.380 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.638224+0000", 2024-09-21T11:07:34.380 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5828999999999999e-05 2024-09-21T11:07:34.380 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.380 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.380 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.380 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.638463+0000", 2024-09-21T11:07:34.380 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000238373 2024-09-21T11:07:34.380 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.380 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.380 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.380 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.641050+0000", 2024-09-21T11:07:34.381 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0025870419999999999 2024-09-21T11:07:34.381 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.381 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.381 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:34.381 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.641099+0000", 2024-09-21T11:07:34.381 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9607999999999997e-05 2024-09-21T11:07:34.381 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.381 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.381 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:34.381 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.641203+0000", 2024-09-21T11:07:34.381 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000104094 2024-09-21T11:07:34.381 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.381 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:34.381 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.381 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.382 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.382 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:219174 58.6 58:6479ddda:::benchmark_data_smithi070_396992_object73215:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1605)", 2024-09-21T11:07:34.382 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:34.606152+0000", 2024-09-21T11:07:34.382 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 179.690522687, 2024-09-21T11:07:34.382 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.035756044, 2024-09-21T11:07:34.382 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:34.382 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:34.382 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:34.382 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:34.382 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:34.382 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 219174 2024-09-21T11:07:34.382 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.382 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:34.382 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.383 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:34.383 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.606152+0000", 2024-09-21T11:07:34.383 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.383 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.383 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.383 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:34.383 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.606152+0000", 2024-09-21T11:07:34.383 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.383 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.383 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.383 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:34.383 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.606151+0000", 2024-09-21T11:07:34.383 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:07:34.384 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.384 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.384 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:34.384 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.606158+0000", 2024-09-21T11:07:34.384 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1359999999999999e-06 2024-09-21T11:07:34.384 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.384 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.384 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:34.384 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.606171+0000", 2024-09-21T11:07:34.384 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2585999999999999e-05 2024-09-21T11:07:34.384 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.384 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.384 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:34.384 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.606177+0000", 2024-09-21T11:07:34.384 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0379999999999996e-06 2024-09-21T11:07:34.385 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.385 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.385 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.385 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.606230+0000", 2024-09-21T11:07:34.385 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3170000000000001e-05 2024-09-21T11:07:34.385 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.385 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.385 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:34.385 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.606243+0000", 2024-09-21T11:07:34.385 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2933e-05 2024-09-21T11:07:34.385 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.385 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.385 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.386 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.636017+0000", 2024-09-21T11:07:34.386 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0297743690000001 2024-09-21T11:07:34.386 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.386 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.386 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:34.386 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.636169+0000", 2024-09-21T11:07:34.386 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000151755 2024-09-21T11:07:34.386 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.386 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.386 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,6,7", 2024-09-21T11:07:34.386 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.636297+0000", 2024-09-21T11:07:34.386 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000128374 2024-09-21T11:07:34.386 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.386 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.387 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:34.387 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.638473+0000", 2024-09-21T11:07:34.387 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021752199999999998 2024-09-21T11:07:34.387 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.387 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.387 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.387 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.638505+0000", 2024-09-21T11:07:34.387 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2549000000000003e-05 2024-09-21T11:07:34.387 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.387 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.387 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.387 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.641243+0000", 2024-09-21T11:07:34.387 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0027383080000000001 2024-09-21T11:07:34.387 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.388 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.388 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.388 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.641738+0000", 2024-09-21T11:07:34.388 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00049473300000000002 2024-09-21T11:07:34.388 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.388 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.388 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:34.388 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.641793+0000", 2024-09-21T11:07:34.388 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4389999999999999e-05 2024-09-21T11:07:34.388 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.388 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.388 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:34.388 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:35.641909+0000", 2024-09-21T11:07:34.388 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000116024 2024-09-21T11:07:34.388 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.389 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:34.389 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.389 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.389 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.389 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:235468 58.3 58:c6838012:::benchmark_data_smithi070_396992_object89509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1615)", 2024-09-21T11:07:34.389 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:53.657214+0000", 2024-09-21T11:07:34.389 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 160.6394612, 2024-09-21T11:07:34.389 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.097778777, 2024-09-21T11:07:34.389 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:34.389 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:34.389 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:34.389 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:34.389 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:34.389 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 235468 2024-09-21T11:07:34.390 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.390 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:34.390 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.390 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:34.390 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:53.657214+0000", 2024-09-21T11:07:34.390 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.390 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.390 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.390 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:34.390 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:53.657214+0000", 2024-09-21T11:07:34.390 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.390 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.390 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.390 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:34.391 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:53.657213+0000", 2024-09-21T11:07:34.391 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:07:34.391 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.391 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.391 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:34.391 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:53.657220+0000", 2024-09-21T11:07:34.391 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6510000000000001e-06 2024-09-21T11:07:34.391 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.391 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.391 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:34.391 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:53.657233+0000", 2024-09-21T11:07:34.391 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2300000000000001e-05 2024-09-21T11:07:34.391 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.391 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.392 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:34.392 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:53.657240+0000", 2024-09-21T11:07:34.392 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6629999999999993e-06 2024-09-21T11:07:34.392 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.392 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.392 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.392 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:53.660692+0000", 2024-09-21T11:07:34.392 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0034513740000000001 2024-09-21T11:07:34.392 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.392 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.392 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:34.392 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:53.660707+0000", 2024-09-21T11:07:34.392 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5597999999999999e-05 2024-09-21T11:07:34.392 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.393 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.393 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.393 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:54.751537+0000", 2024-09-21T11:07:34.393 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.090830199 2024-09-21T11:07:34.393 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.393 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.393 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:34.393 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:54.751632+0000", 2024-09-21T11:07:34.393 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4449000000000003e-05 2024-09-21T11:07:34.393 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.393 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.393 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,3,6", 2024-09-21T11:07:34.394 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:54.751737+0000", 2024-09-21T11:07:34.394 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000105552 2024-09-21T11:07:34.394 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.394 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.394 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.394 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:54.754565+0000", 2024-09-21T11:07:34.394 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0028272990000000001 2024-09-21T11:07:34.394 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.394 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.394 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.394 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:54.754627+0000", 2024-09-21T11:07:34.394 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1933000000000005e-05 2024-09-21T11:07:34.394 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.394 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.395 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:34.395 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:54.754635+0000", 2024-09-21T11:07:34.395 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3920000000000003e-06 2024-09-21T11:07:34.395 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.395 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.395 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.395 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:54.754794+0000", 2024-09-21T11:07:34.395 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015928900000000001 2024-09-21T11:07:34.395 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.395 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.395 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:34.395 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:54.754877+0000", 2024-09-21T11:07:34.395 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3166999999999998e-05 2024-09-21T11:07:34.395 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.395 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.396 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:34.396 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:54.754993+0000", 2024-09-21T11:07:34.396 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000115259 2024-09-21T11:07:34.396 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.396 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:34.396 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.396 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.396 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.396 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:275529 58.c 58:3eb3369e:::benchmark_data_smithi070_396992_object129570:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1627)", 2024-09-21T11:07:34.396 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:35.488792+0000", 2024-09-21T11:07:34.396 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 118.807883571, 2024-09-21T11:07:34.396 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.999575451, 2024-09-21T11:07:34.396 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:34.397 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:34.397 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:34.397 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:34.397 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:34.397 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 275529 2024-09-21T11:07:34.397 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.397 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:34.397 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.397 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:34.397 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:35.488792+0000", 2024-09-21T11:07:34.397 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.397 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.397 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.397 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:34.397 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:35.488792+0000", 2024-09-21T11:07:34.398 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.398 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.398 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.398 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:34.398 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:35.488790+0000", 2024-09-21T11:07:34.398 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:07:34.398 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.398 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.398 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:34.398 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:35.488796+0000", 2024-09-21T11:07:34.398 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9359999999999996e-06 2024-09-21T11:07:34.398 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.398 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.399 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:34.399 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:35.488805+0000", 2024-09-21T11:07:34.399 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6580000000000008e-06 2024-09-21T11:07:34.399 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.399 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.399 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:34.399 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:35.488811+0000", 2024-09-21T11:07:34.399 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0530000000000003e-06 2024-09-21T11:07:34.399 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.399 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.399 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.399 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:35.489722+0000", 2024-09-21T11:07:34.399 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00091135800000000002 2024-09-21T11:07:34.399 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.400 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.400 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:34.400 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:35.489730+0000", 2024-09-21T11:07:34.400 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1389999999999995e-06 2024-09-21T11:07:34.400 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.400 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.401 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.401 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:36.481597+0000", 2024-09-21T11:07:34.401 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99186702299999996 2024-09-21T11:07:34.402 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.402 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.402 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:34.402 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:36.481716+0000", 2024-09-21T11:07:34.402 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000118317 2024-09-21T11:07:34.402 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.402 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.402 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 6,7", 2024-09-21T11:07:34.402 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:36.481799+0000", 2024-09-21T11:07:34.402 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3290999999999999e-05 2024-09-21T11:07:34.402 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.402 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.402 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:34.403 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:36.488143+0000", 2024-09-21T11:07:34.403 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0063439459999999996 2024-09-21T11:07:34.403 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.403 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.403 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.403 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:36.488240+0000", 2024-09-21T11:07:34.403 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7040000000000006e-05 2024-09-21T11:07:34.403 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.403 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.403 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.403 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:36.488278+0000", 2024-09-21T11:07:34.403 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8379e-05 2024-09-21T11:07:34.403 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.404 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.404 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:34.404 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:36.488327+0000", 2024-09-21T11:07:34.404 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8744e-05 2024-09-21T11:07:34.404 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.404 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.404 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:34.404 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:36.488367+0000", 2024-09-21T11:07:34.404 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9854999999999997e-05 2024-09-21T11:07:34.404 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.404 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:34.404 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.404 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.404 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.405 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:285142 58.5 58:a53e4794:::benchmark_data_smithi070_396992_object139183:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1644)", 2024-09-21T11:07:34.405 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:55.554247+0000", 2024-09-21T11:07:34.405 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 98.742427874000001, 2024-09-21T11:07:34.405 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0087835199999999, 2024-09-21T11:07:34.405 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:34.405 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:34.405 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:34.405 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:34.405 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:34.405 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 285142 2024-09-21T11:07:34.405 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.405 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:34.405 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.405 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:34.405 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.554247+0000", 2024-09-21T11:07:34.406 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.406 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.406 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.406 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:34.406 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.554247+0000", 2024-09-21T11:07:34.406 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.406 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.406 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.406 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:34.406 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.554246+0000", 2024-09-21T11:07:34.406 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:07:34.406 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.406 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.406 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:34.407 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.554252+0000", 2024-09-21T11:07:34.407 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5300000000000004e-06 2024-09-21T11:07:34.407 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.407 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.407 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:34.407 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.554265+0000", 2024-09-21T11:07:34.407 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3311e-05 2024-09-21T11:07:34.407 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.407 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.407 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:34.407 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.554272+0000", 2024-09-21T11:07:34.407 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1930000000000004e-06 2024-09-21T11:07:34.407 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.408 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.408 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.408 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.554328+0000", 2024-09-21T11:07:34.408 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5747e-05 2024-09-21T11:07:34.408 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.408 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.408 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:34.408 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.554337+0000", 2024-09-21T11:07:34.408 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2499999999999995e-06 2024-09-21T11:07:34.408 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.408 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.408 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.408 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.557817+0000", 2024-09-21T11:07:34.408 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0034799889999999 2024-09-21T11:07:34.409 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.409 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.409 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:34.409 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.557929+0000", 2024-09-21T11:07:34.409 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000111477 2024-09-21T11:07:34.409 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.409 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.409 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T11:07:34.409 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.557993+0000", 2024-09-21T11:07:34.409 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4462999999999996e-05 2024-09-21T11:07:34.409 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.409 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.409 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.409 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.560690+0000", 2024-09-21T11:07:34.410 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0026967810000000001 2024-09-21T11:07:34.410 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.410 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.410 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:34.410 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.562876+0000", 2024-09-21T11:07:34.410 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021865249999999999 2024-09-21T11:07:34.410 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.410 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.410 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.410 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.562950+0000", 2024-09-21T11:07:34.410 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3623999999999998e-05 2024-09-21T11:07:34.410 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.410 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.410 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:34.411 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.562995+0000", 2024-09-21T11:07:34.411 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4413000000000003e-05 2024-09-21T11:07:34.411 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.411 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.411 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:34.411 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.563031+0000", 2024-09-21T11:07:34.411 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6291000000000002e-05 2024-09-21T11:07:34.411 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.411 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:34.411 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.411 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.411 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.411 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:285167 58.5 58:aa8cf1bf:::benchmark_data_smithi070_396992_object139208:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1644)", 2024-09-21T11:07:34.411 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:55.577023+0000", 2024-09-21T11:07:34.411 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 98.719652652999997, 2024-09-21T11:07:34.412 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99706349400000005, 2024-09-21T11:07:34.412 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:34.412 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:34.412 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:34.412 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:34.412 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:34.412 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 285167 2024-09-21T11:07:34.412 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.412 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:34.412 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.412 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:34.412 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.577023+0000", 2024-09-21T11:07:34.412 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.412 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.413 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.413 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:34.413 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.577023+0000", 2024-09-21T11:07:34.413 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.413 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.413 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.413 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:34.413 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.577021+0000", 2024-09-21T11:07:34.413 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:07:34.413 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.413 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.413 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:34.413 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.577027+0000", 2024-09-21T11:07:34.413 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2009999999999998e-06 2024-09-21T11:07:34.414 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.414 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.414 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:34.414 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.577036+0000", 2024-09-21T11:07:34.414 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5179999999999993e-06 2024-09-21T11:07:34.414 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.414 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.414 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:34.414 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.577042+0000", 2024-09-21T11:07:34.414 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1829999999999999e-06 2024-09-21T11:07:34.414 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.414 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.414 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.414 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.577080+0000", 2024-09-21T11:07:34.415 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7931e-05 2024-09-21T11:07:34.415 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.415 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.415 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:34.415 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.577103+0000", 2024-09-21T11:07:34.415 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2461000000000001e-05 2024-09-21T11:07:34.415 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.415 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.415 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.415 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.560722+0000", 2024-09-21T11:07:34.415 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.983619735 2024-09-21T11:07:34.415 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.415 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.416 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:34.416 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.560827+0000", 2024-09-21T11:07:34.416 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000104694 2024-09-21T11:07:34.416 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.416 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.416 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T11:07:34.416 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.560889+0000", 2024-09-21T11:07:34.416 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1960999999999998e-05 2024-09-21T11:07:34.416 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.416 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.416 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.416 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.565390+0000", 2024-09-21T11:07:34.416 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0045012630000000001 2024-09-21T11:07:34.416 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.417 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.417 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.417 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.565417+0000", 2024-09-21T11:07:34.417 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6556000000000001e-05 2024-09-21T11:07:34.417 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.417 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.417 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:34.417 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.573870+0000", 2024-09-21T11:07:34.417 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0084528469999999994 2024-09-21T11:07:34.417 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.417 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.417 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:34.417 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.574048+0000", 2024-09-21T11:07:34.418 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000177923 2024-09-21T11:07:34.418 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.418 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.418 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:34.418 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.574086+0000", 2024-09-21T11:07:34.418 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8371000000000002e-05 2024-09-21T11:07:34.418 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.418 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:34.418 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.418 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.418 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.418 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:285168 58.5 58:abac6849:::benchmark_data_smithi070_396992_object139209:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1644)", 2024-09-21T11:07:34.418 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:55.577872+0000", 2024-09-21T11:07:34.418 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 98.718803184999999, 2024-09-21T11:07:34.419 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99638369500000001, 2024-09-21T11:07:34.419 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:34.419 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:34.419 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:34.419 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:34.419 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:34.419 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 285168 2024-09-21T11:07:34.419 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.419 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:34.419 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.419 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:34.419 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.577872+0000", 2024-09-21T11:07:34.419 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.419 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.419 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.420 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:34.420 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.577872+0000", 2024-09-21T11:07:34.420 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.420 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.420 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.420 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:34.420 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.577871+0000", 2024-09-21T11:07:34.420 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:07:34.420 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.420 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.420 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:34.420 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.577875+0000", 2024-09-21T11:07:34.420 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1189999999999999e-06 2024-09-21T11:07:34.420 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.421 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.421 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:34.421 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.577884+0000", 2024-09-21T11:07:34.421 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6670000000000002e-06 2024-09-21T11:07:34.421 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.421 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.421 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:34.421 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.577890+0000", 2024-09-21T11:07:34.421 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9220000000000002e-06 2024-09-21T11:07:34.421 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.421 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.421 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.421 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.577924+0000", 2024-09-21T11:07:34.421 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4589999999999999e-05 2024-09-21T11:07:34.421 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.422 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.422 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:34.422 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.577932+0000", 2024-09-21T11:07:34.422 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0620000000000003e-06 2024-09-21T11:07:34.422 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.422 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.422 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.422 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.563059+0000", 2024-09-21T11:07:34.422 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98512711100000006 2024-09-21T11:07:34.422 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.422 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.422 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:34.422 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.563157+0000", 2024-09-21T11:07:34.423 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8011999999999996e-05 2024-09-21T11:07:34.423 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.423 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.423 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T11:07:34.423 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.563255+0000", 2024-09-21T11:07:34.423 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7979000000000005e-05 2024-09-21T11:07:34.423 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.423 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.423 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.423 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.565441+0000", 2024-09-21T11:07:34.423 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021861760000000002 2024-09-21T11:07:34.423 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.423 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.423 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.424 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.567415+0000", 2024-09-21T11:07:34.424 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0019736749999999998 2024-09-21T11:07:34.424 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.424 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.424 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:34.424 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.574170+0000", 2024-09-21T11:07:34.424 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.006755484 2024-09-21T11:07:34.424 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.424 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.424 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:34.424 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.574212+0000", 2024-09-21T11:07:34.424 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2290999999999999e-05 2024-09-21T11:07:34.424 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.424 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.425 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:34.425 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.574256+0000", 2024-09-21T11:07:34.425 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3409999999999999e-05 2024-09-21T11:07:34.425 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.425 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:34.425 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.425 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.425 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.425 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:285170 58.5 58:a0a24874:::benchmark_data_smithi070_396992_object139211:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1644)", 2024-09-21T11:07:34.425 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:55.578388+0000", 2024-09-21T11:07:34.425 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 98.718286704999997, 2024-09-21T11:07:34.425 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99823310099999996, 2024-09-21T11:07:34.425 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:34.426 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:34.426 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:34.426 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:34.426 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:34.426 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 285170 2024-09-21T11:07:34.426 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.426 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:34.426 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.426 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:34.426 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.578388+0000", 2024-09-21T11:07:34.426 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.426 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.426 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.426 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:34.427 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.578388+0000", 2024-09-21T11:07:34.427 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.427 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.427 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.427 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:34.427 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.578388+0000", 2024-09-21T11:07:34.427 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T11:07:34.427 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.427 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.427 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:34.427 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.578392+0000", 2024-09-21T11:07:34.427 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1579999999999998e-06 2024-09-21T11:07:34.427 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.427 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.428 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:34.428 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.578400+0000", 2024-09-21T11:07:34.428 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0790000000000001e-06 2024-09-21T11:07:34.428 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.428 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.428 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:34.428 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.578405+0000", 2024-09-21T11:07:34.428 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.95e-06 2024-09-21T11:07:34.428 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.428 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.428 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.428 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.578449+0000", 2024-09-21T11:07:34.428 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4292000000000002e-05 2024-09-21T11:07:34.428 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.429 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.429 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:34.429 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.578459+0000", 2024-09-21T11:07:34.429 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6779999999999995e-06 2024-09-21T11:07:34.429 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.429 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.429 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.429 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.565459+0000", 2024-09-21T11:07:34.429 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98699982600000002 2024-09-21T11:07:34.429 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.429 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.429 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:34.429 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.565552+0000", 2024-09-21T11:07:34.429 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3113000000000001e-05 2024-09-21T11:07:34.429 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.430 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.430 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T11:07:34.430 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.565649+0000", 2024-09-21T11:07:34.430 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.747e-05 2024-09-21T11:07:34.430 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.430 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.430 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.430 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.569507+0000", 2024-09-21T11:07:34.430 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0038576550000000002 2024-09-21T11:07:34.430 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.430 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.430 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.430 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.571588+0000", 2024-09-21T11:07:34.430 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0020814430000000001 2024-09-21T11:07:34.431 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.431 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.431 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:34.431 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.574262+0000", 2024-09-21T11:07:34.431 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0026736210000000002 2024-09-21T11:07:34.431 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.431 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.431 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:34.431 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.574301+0000", 2024-09-21T11:07:34.431 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8457000000000001e-05 2024-09-21T11:07:34.431 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.431 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.431 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:34.431 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.576622+0000", 2024-09-21T11:07:34.432 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0023210119999999999 2024-09-21T11:07:34.432 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.432 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:34.432 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.432 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.432 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.432 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:285173 58.5 58:a722ebc9:::benchmark_data_smithi070_396992_object139214:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1644)", 2024-09-21T11:07:34.432 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:55.581217+0000", 2024-09-21T11:07:34.432 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 98.715458427000002, 2024-09-21T11:07:34.432 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99550844599999999, 2024-09-21T11:07:34.432 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:34.432 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:34.432 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:34.432 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:34.433 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:34.433 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 285173 2024-09-21T11:07:34.433 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.433 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:34.433 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.433 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:34.433 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.581217+0000", 2024-09-21T11:07:34.433 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.433 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.433 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.433 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:34.433 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.581217+0000", 2024-09-21T11:07:34.433 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.433 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.433 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.434 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:34.434 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.581216+0000", 2024-09-21T11:07:34.434 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:07:34.434 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.434 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.434 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:34.434 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.581219+0000", 2024-09-21T11:07:34.434 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6059999999999999e-06 2024-09-21T11:07:34.434 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.434 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.434 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:34.434 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.581230+0000", 2024-09-21T11:07:34.434 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0655e-05 2024-09-21T11:07:34.434 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.435 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.435 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:34.435 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.581236+0000", 2024-09-21T11:07:34.435 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.127e-06 2024-09-21T11:07:34.435 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.435 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.435 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.435 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.581294+0000", 2024-09-21T11:07:34.435 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8026000000000003e-05 2024-09-21T11:07:34.435 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.435 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.435 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:34.435 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.581302+0000", 2024-09-21T11:07:34.435 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.542e-06 2024-09-21T11:07:34.435 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.436 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.436 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.436 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.567436+0000", 2024-09-21T11:07:34.436 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.986134446 2024-09-21T11:07:34.436 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.436 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.436 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:34.436 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.567526+0000", 2024-09-21T11:07:34.436 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0129000000000004e-05 2024-09-21T11:07:34.436 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.436 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.436 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T11:07:34.436 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.567628+0000", 2024-09-21T11:07:34.436 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010158300000000001 2024-09-21T11:07:34.436 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.437 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.437 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.437 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.569537+0000", 2024-09-21T11:07:34.437 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001908896 2024-09-21T11:07:34.437 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.437 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.437 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.437 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.571623+0000", 2024-09-21T11:07:34.437 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0020859860000000002 2024-09-21T11:07:34.437 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.437 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.437 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:34.437 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.576634+0000", 2024-09-21T11:07:34.437 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0050112259999999997 2024-09-21T11:07:34.438 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.438 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.438 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:34.438 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.576683+0000", 2024-09-21T11:07:34.438 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8912000000000002e-05 2024-09-21T11:07:34.438 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.438 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.438 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:34.438 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.576725+0000", 2024-09-21T11:07:34.438 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2237999999999998e-05 2024-09-21T11:07:34.438 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.438 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:34.438 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.438 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.439 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.439 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:285189 58.5 58:a0d40733:::benchmark_data_smithi070_396992_object139230:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1644)", 2024-09-21T11:07:34.439 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:55.593871+0000", 2024-09-21T11:07:34.439 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 98.702804670999996, 2024-09-21T11:07:34.439 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98292177400000003, 2024-09-21T11:07:34.439 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:34.439 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:34.439 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:34.439 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:34.439 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:34.439 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 285189 2024-09-21T11:07:34.439 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.439 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:34.439 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.439 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:34.440 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.593871+0000", 2024-09-21T11:07:34.440 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.440 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.440 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.440 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:34.440 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.593871+0000", 2024-09-21T11:07:34.440 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:34.440 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.440 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.440 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:34.440 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.593869+0000", 2024-09-21T11:07:34.440 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:07:34.440 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.440 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.441 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:34.441 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.593879+0000", 2024-09-21T11:07:34.441 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6169999999999996e-06 2024-09-21T11:07:34.441 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.441 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.441 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:34.441 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.593890+0000", 2024-09-21T11:07:34.441 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1608e-05 2024-09-21T11:07:34.441 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.441 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.441 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:34.441 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.593897+0000", 2024-09-21T11:07:34.441 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7399999999999998e-06 2024-09-21T11:07:34.441 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.442 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.442 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.442 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.593944+0000", 2024-09-21T11:07:34.442 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7082000000000001e-05 2024-09-21T11:07:34.442 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.442 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.442 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:34.442 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.593965+0000", 2024-09-21T11:07:34.442 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0457000000000001e-05 2024-09-21T11:07:34.442 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.442 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.442 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:34.442 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.569557+0000", 2024-09-21T11:07:34.443 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.97559292799999997 2024-09-21T11:07:34.443 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.443 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.443 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:34.443 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.569657+0000", 2024-09-21T11:07:34.443 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9742999999999997e-05 2024-09-21T11:07:34.443 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.443 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.443 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T11:07:34.443 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.569741+0000", 2024-09-21T11:07:34.443 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3448000000000005e-05 2024-09-21T11:07:34.443 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.443 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.443 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.444 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.573737+0000", 2024-09-21T11:07:34.444 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0039958349999999997 2024-09-21T11:07:34.444 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.444 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.444 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:34.444 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.574122+0000", 2024-09-21T11:07:34.444 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00038598700000000002 2024-09-21T11:07:34.444 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.444 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.444 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:34.444 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.576732+0000", 2024-09-21T11:07:34.444 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002609414 2024-09-21T11:07:34.444 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.444 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.444 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:34.445 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.576764+0000", 2024-09-21T11:07:34.445 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2422999999999998e-05 2024-09-21T11:07:34.445 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:34.445 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:34.445 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:34.445 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.576792+0000", 2024-09-21T11:07:34.445 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7957e-05 2024-09-21T11:07:34.445 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.445 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:34.445 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.445 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:34.445 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:34.445 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T11:07:35.662 INFO:teuthology.orchestra.run.smithi070.stderr:no valid command found; 10 closest matches: 2024-09-21T11:07:35.662 INFO:teuthology.orchestra.run.smithi070.stderr:0 2024-09-21T11:07:35.662 INFO:teuthology.orchestra.run.smithi070.stderr:1 2024-09-21T11:07:35.662 INFO:teuthology.orchestra.run.smithi070.stderr:2 2024-09-21T11:07:35.663 INFO:teuthology.orchestra.run.smithi070.stderr:abort 2024-09-21T11:07:35.663 INFO:teuthology.orchestra.run.smithi070.stderr:assert 2024-09-21T11:07:35.663 INFO:teuthology.orchestra.run.smithi070.stderr:bench [] [] [] [] 2024-09-21T11:07:35.663 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs debug_inject_read_zeros 2024-09-21T11:07:35.663 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs files list 2024-09-21T11:07:35.663 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs stats 2024-09-21T11:07:35.663 INFO:teuthology.orchestra.run.smithi070.stderr:bluestore allocator dump block 2024-09-21T11:07:35.663 INFO:teuthology.orchestra.run.smithi070.stderr:Error EINVAL: invalid command 2024-09-21T11:07:35.716 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:35 smithi160 ceph-mon[101879]: pgmap v3344: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:35.716 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:07:35.717 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:07:35.717 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:07:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:35 smithi070 ceph-mon[107385]: pgmap v3344: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:07:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:07:35.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:07:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:35 smithi070 ceph-mon[112139]: pgmap v3344: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:07:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:07:35.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:07:36.030 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T11:07:36.399 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T11:07:36.399 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T11:07:36.399 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.4 'trim stale osdmaps' 2024-09-21T11:07:36.417 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:07:36.596 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:07:37.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:37 smithi160 ceph-mon[101879]: pgmap v3345: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:37.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:37 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:07:37.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:37 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:07:37.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:37 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T11:07:37.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:37 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T11:07:37.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:37 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:07:37.729 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:37 smithi070 ceph-mon[107385]: pgmap v3345: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:37.729 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:37 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:07:37.729 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:37 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:07:37.729 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:37 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T11:07:37.729 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:37 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T11:07:37.729 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:37 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:07:37.729 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:37 smithi070 ceph-mon[112139]: pgmap v3345: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:37.729 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:37 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:07:37.730 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:37 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:07:37.730 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:37 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T11:07:37.730 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:37 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T11:07:37.730 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:37 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:07:38.171 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T11:07:38.171 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T11:07:38.171 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T11:07:38.171 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T11:07:38.242 INFO:teuthology.orchestra.run.smithi070.stderr:no valid command found; 10 closest matches: 2024-09-21T11:07:38.242 INFO:teuthology.orchestra.run.smithi070.stderr:0 2024-09-21T11:07:38.242 INFO:teuthology.orchestra.run.smithi070.stderr:1 2024-09-21T11:07:38.242 INFO:teuthology.orchestra.run.smithi070.stderr:2 2024-09-21T11:07:38.242 INFO:teuthology.orchestra.run.smithi070.stderr:abort 2024-09-21T11:07:38.242 INFO:teuthology.orchestra.run.smithi070.stderr:assert 2024-09-21T11:07:38.242 INFO:teuthology.orchestra.run.smithi070.stderr:bench [] [] [] [] 2024-09-21T11:07:38.242 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs debug_inject_read_zeros 2024-09-21T11:07:38.242 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs files list 2024-09-21T11:07:38.242 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs stats 2024-09-21T11:07:38.242 INFO:teuthology.orchestra.run.smithi070.stderr:bluestore allocator dump block 2024-09-21T11:07:38.243 INFO:teuthology.orchestra.run.smithi070.stderr:Error EINVAL: invalid command 2024-09-21T11:07:38.856 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T11:07:39.055 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:07:39.222 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T11:07:39.222 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T11:07:39.223 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.0 'trim stale osdmaps' 2024-09-21T11:07:39.425 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:07:39.485 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:39 smithi160 ceph-mon[101879]: pgmap v3346: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:39.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:39 smithi070 ceph-mon[107385]: pgmap v3346: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:39.546 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:39 smithi070 ceph-mon[112139]: pgmap v3346: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:40.810 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T11:07:40.811 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T11:07:40.811 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T11:07:40.811 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T11:07:40.811 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T11:07:41.098 INFO:teuthology.orchestra.run.smithi070.stderr:no valid command found; 10 closest matches: 2024-09-21T11:07:41.098 INFO:teuthology.orchestra.run.smithi070.stderr:0 2024-09-21T11:07:41.098 INFO:teuthology.orchestra.run.smithi070.stderr:1 2024-09-21T11:07:41.098 INFO:teuthology.orchestra.run.smithi070.stderr:2 2024-09-21T11:07:41.098 INFO:teuthology.orchestra.run.smithi070.stderr:abort 2024-09-21T11:07:41.098 INFO:teuthology.orchestra.run.smithi070.stderr:assert 2024-09-21T11:07:41.098 INFO:teuthology.orchestra.run.smithi070.stderr:bench [] [] [] [] 2024-09-21T11:07:41.099 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs debug_inject_read_zeros 2024-09-21T11:07:41.099 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs files list 2024-09-21T11:07:41.099 INFO:teuthology.orchestra.run.smithi070.stderr:bluefs stats 2024-09-21T11:07:41.099 INFO:teuthology.orchestra.run.smithi070.stderr:bluestore allocator dump block 2024-09-21T11:07:41.099 INFO:teuthology.orchestra.run.smithi070.stderr:Error EINVAL: invalid command 2024-09-21T11:07:41.460 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T11:07:41.658 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:07:41.779 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:41 smithi160 ceph-mon[101879]: pgmap v3347: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:41.850 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:41 smithi070 ceph-mon[107385]: pgmap v3347: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:41.851 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:41 smithi070 ceph-mon[112139]: pgmap v3347: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:41.871 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T11:07:41.871 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T11:07:43.421 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:07:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:07:43] "GET /metrics HTTP/1.1" 200 34247 "" "Prometheus/2.51.0" 2024-09-21T11:07:43.430 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T11:07:43.430 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T11:07:43.430 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T11:07:43.430 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T11:07:43.430 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.430 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:84903 57.18 57:1a2783a1:::benchmark_data_smithi070_360420_object17554:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1559)", 2024-09-21T11:07:43.430 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:00:40.450831+0000", 2024-09-21T11:07:43.430 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 422.94755521000002, 2024-09-21T11:07:43.430 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9987897640000001, 2024-09-21T11:07:43.430 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:43.431 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:43.431 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:43.431 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T11:07:43.431 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:07:43.431 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 84903 2024-09-21T11:07:43.431 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.431 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:43.431 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.431 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:43.431 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.450831+0000", 2024-09-21T11:07:43.431 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.431 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.431 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.431 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:43.431 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.450831+0000", 2024-09-21T11:07:43.432 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.432 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.432 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.432 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:43.432 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.450831+0000", 2024-09-21T11:07:43.432 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T11:07:43.432 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.432 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.432 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:43.432 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.450833+0000", 2024-09-21T11:07:43.432 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8920000000000001e-06 2024-09-21T11:07:43.432 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.432 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.433 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:43.433 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.450838+0000", 2024-09-21T11:07:43.433 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0170000000000004e-06 2024-09-21T11:07:43.433 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.433 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.433 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:43.433 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.450842+0000", 2024-09-21T11:07:43.433 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.755e-06 2024-09-21T11:07:43.433 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.433 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.433 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.433 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.622223+0000", 2024-09-21T11:07:43.433 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.171381743 2024-09-21T11:07:43.433 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.434 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.434 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:43.434 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.622232+0000", 2024-09-21T11:07:43.434 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3110000000000006e-06 2024-09-21T11:07:43.434 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.434 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.434 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.434 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.426926+0000", 2024-09-21T11:07:43.434 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8046946619999999 2024-09-21T11:07:43.434 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.434 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.434 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:43.434 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.426936+0000", 2024-09-21T11:07:43.435 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6169999999999996e-06 2024-09-21T11:07:43.435 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.435 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.435 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.435 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.444284+0000", 2024-09-21T11:07:43.435 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.017348403 2024-09-21T11:07:43.435 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.435 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.435 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:43.435 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.444394+0000", 2024-09-21T11:07:43.435 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000109795 2024-09-21T11:07:43.435 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.435 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.435 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-21T11:07:43.435 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.444560+0000", 2024-09-21T11:07:43.436 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016564600000000001 2024-09-21T11:07:43.436 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.436 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.436 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.436 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.449079+0000", 2024-09-21T11:07:43.436 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0045190559999999996 2024-09-21T11:07:43.436 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.436 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.436 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:43.436 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.449086+0000", 2024-09-21T11:07:43.436 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3629999999999996e-06 2024-09-21T11:07:43.436 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.436 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.436 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.437 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.449132+0000", 2024-09-21T11:07:43.437 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5802e-05 2024-09-21T11:07:43.437 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.437 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.437 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:43.437 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.449181+0000", 2024-09-21T11:07:43.437 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8937999999999998e-05 2024-09-21T11:07:43.437 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.437 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.437 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:43.437 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.449621+0000", 2024-09-21T11:07:43.437 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00044004999999999999 2024-09-21T11:07:43.437 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.437 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:43.437 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.438 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.438 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.438 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:84913 57.18 57:189fa363:::benchmark_data_smithi070_360420_object17564:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1559)", 2024-09-21T11:07:43.438 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:00:40.589206+0000", 2024-09-21T11:07:43.438 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 422.80918047199998, 2024-09-21T11:07:43.438 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8690907279999998, 2024-09-21T11:07:43.438 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:43.438 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:43.438 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:43.438 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T11:07:43.438 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:07:43.438 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 84913 2024-09-21T11:07:43.438 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.438 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:43.439 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.439 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:43.439 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.589206+0000", 2024-09-21T11:07:43.439 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.439 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.439 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.439 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:43.439 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.589206+0000", 2024-09-21T11:07:43.439 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.439 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.439 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.439 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:43.439 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.589204+0000", 2024-09-21T11:07:43.439 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:07:43.440 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.440 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.440 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:43.440 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.589212+0000", 2024-09-21T11:07:43.440 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2139999999999999e-06 2024-09-21T11:07:43.440 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.440 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.440 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:43.440 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.589223+0000", 2024-09-21T11:07:43.440 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1399999999999999e-05 2024-09-21T11:07:43.440 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.440 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.440 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:43.440 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.589232+0000", 2024-09-21T11:07:43.440 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1870000000000003e-06 2024-09-21T11:07:43.441 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.441 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.441 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.441 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.622249+0000", 2024-09-21T11:07:43.441 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.033016851 2024-09-21T11:07:43.441 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.441 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.441 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:43.441 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.622254+0000", 2024-09-21T11:07:43.441 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9420000000000003e-06 2024-09-21T11:07:43.441 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.441 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.441 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.441 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.426956+0000", 2024-09-21T11:07:43.442 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8047021640000001 2024-09-21T11:07:43.442 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.442 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.442 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:43.442 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.426962+0000", 2024-09-21T11:07:43.442 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8209999999999998e-06 2024-09-21T11:07:43.442 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.442 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.442 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.442 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.446121+0000", 2024-09-21T11:07:43.442 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0191587419999999 2024-09-21T11:07:43.442 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.442 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.442 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T11:07:43.443 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.449020+0000", 2024-09-21T11:07:43.443 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0028993199999999999 2024-09-21T11:07:43.443 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.443 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.443 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.443 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.452360+0000", 2024-09-21T11:07:43.443 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0033404849999999998 2024-09-21T11:07:43.443 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.443 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.443 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:43.443 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.452454+0000", 2024-09-21T11:07:43.443 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3280000000000001e-05 2024-09-21T11:07:43.443 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.443 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.444 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-21T11:07:43.444 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.452685+0000", 2024-09-21T11:07:43.444 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023139800000000001 2024-09-21T11:07:43.444 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.444 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.444 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:43.444 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.456964+0000", 2024-09-21T11:07:43.444 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0042792109999999998 2024-09-21T11:07:43.444 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.444 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.444 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.444 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.457870+0000", 2024-09-21T11:07:43.444 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00090521000000000004 2024-09-21T11:07:43.445 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.445 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.445 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.445 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.457926+0000", 2024-09-21T11:07:43.445 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6041000000000001e-05 2024-09-21T11:07:43.445 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.445 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.445 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:43.445 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.457976+0000", 2024-09-21T11:07:43.445 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0555e-05 2024-09-21T11:07:43.445 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.445 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.445 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:43.445 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.458297+0000", 2024-09-21T11:07:43.446 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00032048899999999997 2024-09-21T11:07:43.446 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.446 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:43.446 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.446 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.446 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.446 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:84916 57.5 57:a6fd8b58:::benchmark_data_smithi070_360420_object17567:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1559)", 2024-09-21T11:07:43.446 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:00:40.629113+0000", 2024-09-21T11:07:43.446 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 422.76927374899998, 2024-09-21T11:07:43.446 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.846688651, 2024-09-21T11:07:43.446 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:43.446 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:43.446 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:43.446 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T11:07:43.447 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:07:43.447 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 84916 2024-09-21T11:07:43.447 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.447 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:43.447 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.447 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:43.447 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.629113+0000", 2024-09-21T11:07:43.447 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.447 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.447 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.447 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:43.447 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.629113+0000", 2024-09-21T11:07:43.447 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.447 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.448 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.448 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:43.448 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.629111+0000", 2024-09-21T11:07:43.448 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T11:07:43.448 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.448 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.448 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:43.448 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.629121+0000", 2024-09-21T11:07:43.448 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8300000000000008e-06 2024-09-21T11:07:43.448 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.448 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.448 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:43.448 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.629132+0000", 2024-09-21T11:07:43.449 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1609e-05 2024-09-21T11:07:43.449 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.449 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.449 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:43.449 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.629139+0000", 2024-09-21T11:07:43.449 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5540000000000003e-06 2024-09-21T11:07:43.449 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.449 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.449 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.449 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.629181+0000", 2024-09-21T11:07:43.449 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2475000000000002e-05 2024-09-21T11:07:43.449 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.449 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.449 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:43.450 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:40.629190+0000", 2024-09-21T11:07:43.450 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2449999999999998e-06 2024-09-21T11:07:43.450 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.450 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.450 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.450 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.433253+0000", 2024-09-21T11:07:43.450 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.80406289500000006 2024-09-21T11:07:43.450 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.450 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.450 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:43.450 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.433266+0000", 2024-09-21T11:07:43.450 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3584000000000001e-05 2024-09-21T11:07:43.451 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.451 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.451 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.451 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.444118+0000", 2024-09-21T11:07:43.451 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0108519810000001 2024-09-21T11:07:43.451 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.451 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.451 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T11:07:43.451 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.444637+0000", 2024-09-21T11:07:43.451 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00051886499999999997 2024-09-21T11:07:43.451 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.451 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.451 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.451 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.465502+0000", 2024-09-21T11:07:43.452 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020864866999999999 2024-09-21T11:07:43.452 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.452 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.452 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:43.452 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.465584+0000", 2024-09-21T11:07:43.452 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2575000000000001e-05 2024-09-21T11:07:43.452 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.452 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.452 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-21T11:07:43.452 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.465865+0000", 2024-09-21T11:07:43.452 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028076000000000002 2024-09-21T11:07:43.452 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.452 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.453 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.453 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.467463+0000", 2024-09-21T11:07:43.453 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0015981039999999999 2024-09-21T11:07:43.453 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.453 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.453 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:43.453 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.469545+0000", 2024-09-21T11:07:43.453 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0020818500000000001 2024-09-21T11:07:43.453 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.453 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.453 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.453 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.474699+0000", 2024-09-21T11:07:43.453 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0051534240000000002 2024-09-21T11:07:43.453 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.453 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.454 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:43.454 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.474775+0000", 2024-09-21T11:07:43.454 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6031000000000004e-05 2024-09-21T11:07:43.454 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.454 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.454 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:43.454 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.475801+0000", 2024-09-21T11:07:43.454 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001026735 2024-09-21T11:07:43.454 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.454 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:43.454 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.454 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.455 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.455 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:84948 57.18 57:1ea98b34:::benchmark_data_smithi070_360420_object17599:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1560)", 2024-09-21T11:07:43.455 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:00:41.508218+0000", 2024-09-21T11:07:43.455 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 421.89016835299998, 2024-09-21T11:07:43.455 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9505037220000001, 2024-09-21T11:07:43.455 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:43.455 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:43.455 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:43.455 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T11:07:43.455 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:07:43.455 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 84948 2024-09-21T11:07:43.455 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.455 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:43.455 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.456 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:43.456 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.508218+0000", 2024-09-21T11:07:43.456 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.456 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.456 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.456 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:43.456 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.508218+0000", 2024-09-21T11:07:43.456 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.456 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.456 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.456 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:43.456 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.508217+0000", 2024-09-21T11:07:43.456 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:07:43.457 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.457 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.457 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:43.457 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.508222+0000", 2024-09-21T11:07:43.457 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6699999999999999e-06 2024-09-21T11:07:43.457 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.457 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.457 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:43.457 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.508233+0000", 2024-09-21T11:07:43.457 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.022e-05 2024-09-21T11:07:43.457 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.457 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.457 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:43.458 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.508239+0000", 2024-09-21T11:07:43.458 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6819999999999997e-06 2024-09-21T11:07:43.458 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.458 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.458 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.458 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.508287+0000", 2024-09-21T11:07:43.458 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7735999999999998e-05 2024-09-21T11:07:43.458 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.458 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.459 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:43.459 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.508294+0000", 2024-09-21T11:07:43.459 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3409999999999997e-06 2024-09-21T11:07:43.459 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.459 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.459 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.459 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.426980+0000", 2024-09-21T11:07:43.459 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.918685686 2024-09-21T11:07:43.459 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.459 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.459 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:43.459 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.426986+0000", 2024-09-21T11:07:43.459 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5509999999999999e-06 2024-09-21T11:07:43.460 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.460 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.460 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.460 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.449668+0000", 2024-09-21T11:07:43.460 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.02268208 2024-09-21T11:07:43.460 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.460 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.460 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T11:07:43.460 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.450193+0000", 2024-09-21T11:07:43.460 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00052498900000000001 2024-09-21T11:07:43.460 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.460 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.460 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.461 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.455091+0000", 2024-09-21T11:07:43.461 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0048979899999999996 2024-09-21T11:07:43.461 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.461 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.461 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:43.461 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.455186+0000", 2024-09-21T11:07:43.461 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5539999999999997e-05 2024-09-21T11:07:43.461 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.461 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.461 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-21T11:07:43.461 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.455384+0000", 2024-09-21T11:07:43.461 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019748300000000001 2024-09-21T11:07:43.462 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.462 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.462 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:43.462 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.457879+0000", 2024-09-21T11:07:43.462 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0024956359999999999 2024-09-21T11:07:43.462 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.462 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.462 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.462 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.458335+0000", 2024-09-21T11:07:43.462 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00045515699999999998 2024-09-21T11:07:43.462 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.462 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.462 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.463 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.458362+0000", 2024-09-21T11:07:43.463 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7668e-05 2024-09-21T11:07:43.463 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.463 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.463 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:43.463 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.458407+0000", 2024-09-21T11:07:43.463 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5241000000000003e-05 2024-09-21T11:07:43.463 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.463 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.463 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:43.463 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.458722+0000", 2024-09-21T11:07:43.463 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00031433500000000001 2024-09-21T11:07:43.463 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.463 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:43.464 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.464 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.464 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.464 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:84956 57.18 57:1d6cf45f:::benchmark_data_smithi070_360420_object17607:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1560)", 2024-09-21T11:07:43.464 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:00:41.512349+0000", 2024-09-21T11:07:43.464 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 421.88603760799998, 2024-09-21T11:07:43.464 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.95018297, 2024-09-21T11:07:43.464 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:43.464 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:43.464 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:43.464 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T11:07:43.464 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:07:43.464 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 84956 2024-09-21T11:07:43.464 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.464 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:43.465 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.465 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:43.465 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.512349+0000", 2024-09-21T11:07:43.465 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.465 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.465 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.465 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:43.465 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.512349+0000", 2024-09-21T11:07:43.465 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.465 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.465 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.465 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:43.465 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.512348+0000", 2024-09-21T11:07:43.465 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:07:43.466 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.466 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.466 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:43.466 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.512352+0000", 2024-09-21T11:07:43.466 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3239999999999996e-06 2024-09-21T11:07:43.466 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.466 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.466 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:43.466 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.512360+0000", 2024-09-21T11:07:43.466 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9950000000000005e-06 2024-09-21T11:07:43.466 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.466 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.466 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:43.466 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.512366+0000", 2024-09-21T11:07:43.467 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7910000000000001e-06 2024-09-21T11:07:43.467 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.467 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.467 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.467 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.512415+0000", 2024-09-21T11:07:43.467 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8634000000000002e-05 2024-09-21T11:07:43.467 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.467 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.467 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:43.467 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.512423+0000", 2024-09-21T11:07:43.467 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9039999999999992e-06 2024-09-21T11:07:43.467 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.467 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.467 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.468 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.427002+0000", 2024-09-21T11:07:43.468 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91457977099999999 2024-09-21T11:07:43.468 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.468 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.468 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:43.468 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.427008+0000", 2024-09-21T11:07:43.468 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4190000000000001e-06 2024-09-21T11:07:43.468 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.468 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.468 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.468 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.450228+0000", 2024-09-21T11:07:43.468 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0232196790000001 2024-09-21T11:07:43.468 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.468 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.468 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T11:07:43.469 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.451455+0000", 2024-09-21T11:07:43.469 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0012271890000000001 2024-09-21T11:07:43.469 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.469 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.469 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.469 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.456994+0000", 2024-09-21T11:07:43.469 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.005539647 2024-09-21T11:07:43.469 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.469 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.469 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:43.469 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.457051+0000", 2024-09-21T11:07:43.469 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6971000000000001e-05 2024-09-21T11:07:43.469 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.469 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.469 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-21T11:07:43.470 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.457185+0000", 2024-09-21T11:07:43.470 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013387800000000001 2024-09-21T11:07:43.470 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.470 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.470 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.470 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.458758+0000", 2024-09-21T11:07:43.470 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0015725719999999999 2024-09-21T11:07:43.470 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.470 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.470 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.470 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.458790+0000", 2024-09-21T11:07:43.470 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1738e-05 2024-09-21T11:07:43.470 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.471 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.471 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:43.471 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.461839+0000", 2024-09-21T11:07:43.471 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0030498309999999998 2024-09-21T11:07:43.471 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.471 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.471 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:43.471 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.461912+0000", 2024-09-21T11:07:43.471 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2309000000000002e-05 2024-09-21T11:07:43.471 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.471 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.471 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:43.471 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.462532+0000", 2024-09-21T11:07:43.471 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00062006500000000005 2024-09-21T11:07:43.471 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.472 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:43.472 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.472 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.472 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.472 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:84989 57.18 57:1b2d9fc9:::benchmark_data_smithi070_360420_object17640:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1560)", 2024-09-21T11:07:43.472 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:00:41.540067+0000", 2024-09-21T11:07:43.472 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 421.85831919999998, 2024-09-21T11:07:43.472 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9312866450000001, 2024-09-21T11:07:43.472 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:43.472 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:43.472 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:43.472 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T11:07:43.472 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:07:43.472 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 84989 2024-09-21T11:07:43.473 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.473 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:43.473 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.473 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:43.473 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.540067+0000", 2024-09-21T11:07:43.473 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.473 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.473 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.473 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:43.473 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.540067+0000", 2024-09-21T11:07:43.473 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.473 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.473 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.473 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:43.474 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.540066+0000", 2024-09-21T11:07:43.474 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:07:43.474 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.474 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.474 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:43.474 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.540071+0000", 2024-09-21T11:07:43.474 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.22e-06 2024-09-21T11:07:43.474 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.474 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.474 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:43.474 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.540082+0000", 2024-09-21T11:07:43.474 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0886e-05 2024-09-21T11:07:43.474 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.474 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.475 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:43.475 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.540089+0000", 2024-09-21T11:07:43.475 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6730000000000003e-06 2024-09-21T11:07:43.475 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.475 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.475 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.475 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.540165+0000", 2024-09-21T11:07:43.475 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6431000000000001e-05 2024-09-21T11:07:43.475 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.475 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.475 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:43.475 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.540174+0000", 2024-09-21T11:07:43.475 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6969999999999999e-06 2024-09-21T11:07:43.475 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.476 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.476 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.476 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.427027+0000", 2024-09-21T11:07:43.476 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88685251200000004 2024-09-21T11:07:43.476 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.476 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.476 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:43.476 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.427033+0000", 2024-09-21T11:07:43.476 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2060000000000004e-06 2024-09-21T11:07:43.476 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.476 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.476 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.476 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.458811+0000", 2024-09-21T11:07:43.476 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.031777978 2024-09-21T11:07:43.477 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.477 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.477 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T11:07:43.477 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.459351+0000", 2024-09-21T11:07:43.477 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00054042399999999996 2024-09-21T11:07:43.477 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.477 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.477 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.477 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.464277+0000", 2024-09-21T11:07:43.477 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0049253639999999998 2024-09-21T11:07:43.477 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.477 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.477 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:43.477 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.464379+0000", 2024-09-21T11:07:43.478 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000102583 2024-09-21T11:07:43.478 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.478 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.478 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-21T11:07:43.478 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.464632+0000", 2024-09-21T11:07:43.478 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025245600000000002 2024-09-21T11:07:43.478 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.478 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.478 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:43.478 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.468167+0000", 2024-09-21T11:07:43.478 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0035357209999999999 2024-09-21T11:07:43.478 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.478 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.478 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.479 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.470643+0000", 2024-09-21T11:07:43.479 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002475882 2024-09-21T11:07:43.479 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.479 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.479 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.479 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.470755+0000", 2024-09-21T11:07:43.479 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011183699999999999 2024-09-21T11:07:43.479 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.479 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.479 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:43.479 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.470805+0000", 2024-09-21T11:07:43.479 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9704999999999999e-05 2024-09-21T11:07:43.479 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.479 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.480 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:43.480 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.471354+0000", 2024-09-21T11:07:43.480 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00054911599999999999 2024-09-21T11:07:43.480 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.480 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:43.480 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.480 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.480 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.480 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46190.0:85034 57.18 57:18bea2e4:::benchmark_data_smithi070_360420_object17685:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1560)", 2024-09-21T11:07:43.480 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:00:41.614931+0000", 2024-09-21T11:07:43.480 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 421.78345586199998, 2024-09-21T11:07:43.480 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8569978009999999, 2024-09-21T11:07:43.480 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:43.480 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:43.481 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:43.481 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46190", 2024-09-21T11:07:43.481 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1997608010", 2024-09-21T11:07:43.481 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 85034 2024-09-21T11:07:43.481 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.481 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:43.481 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.481 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:43.481 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.614931+0000", 2024-09-21T11:07:43.481 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.481 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.481 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.481 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:43.481 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.614931+0000", 2024-09-21T11:07:43.481 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.482 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.482 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.482 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:43.482 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.614930+0000", 2024-09-21T11:07:43.482 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T11:07:43.482 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.482 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.482 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:43.482 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.614934+0000", 2024-09-21T11:07:43.482 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7749999999999999e-06 2024-09-21T11:07:43.482 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.482 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.482 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:43.482 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.614942+0000", 2024-09-21T11:07:43.483 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.276e-06 2024-09-21T11:07:43.483 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.483 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.483 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:43.483 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.614947+0000", 2024-09-21T11:07:43.483 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9479999999999999e-06 2024-09-21T11:07:43.483 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.483 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.483 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.483 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.614982+0000", 2024-09-21T11:07:43.483 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5445999999999999e-05 2024-09-21T11:07:43.483 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.483 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.483 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:43.484 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:41.614992+0000", 2024-09-21T11:07:43.484 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3230000000000004e-06 2024-09-21T11:07:43.484 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.484 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.484 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.484 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.427052+0000", 2024-09-21T11:07:43.484 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.81206049899999999 2024-09-21T11:07:43.484 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.484 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.484 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:43.484 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:42.427059+0000", 2024-09-21T11:07:43.484 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6950000000000003e-06 2024-09-21T11:07:43.484 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.484 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.484 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.486 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.459388+0000", 2024-09-21T11:07:43.486 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0323289659999999 2024-09-21T11:07:43.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.486 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.486 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-21T11:07:43.486 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.461825+0000", 2024-09-21T11:07:43.486 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002436737 2024-09-21T11:07:43.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.486 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.486 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.486 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.466857+0000", 2024-09-21T11:07:43.486 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0050320390000000003 2024-09-21T11:07:43.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.487 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.487 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:43.487 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.466942+0000", 2024-09-21T11:07:43.487 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5928999999999997e-05 2024-09-21T11:07:43.487 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.487 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.487 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-21T11:07:43.487 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.467178+0000", 2024-09-21T11:07:43.487 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023586200000000001 2024-09-21T11:07:43.487 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.487 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.487 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.487 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.470704+0000", 2024-09-21T11:07:43.487 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0035258149999999999 2024-09-21T11:07:43.488 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.488 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.488 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:43.488 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.470718+0000", 2024-09-21T11:07:43.488 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3920000000000001e-05 2024-09-21T11:07:43.488 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.488 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.488 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.488 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.471413+0000", 2024-09-21T11:07:43.488 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00069485499999999997 2024-09-21T11:07:43.488 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.488 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.488 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:43.489 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.471495+0000", 2024-09-21T11:07:43.489 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1886999999999997e-05 2024-09-21T11:07:43.489 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.489 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.489 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:43.489 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:00:43.471928+0000", 2024-09-21T11:07:43.489 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000433516 2024-09-21T11:07:43.489 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.489 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:43.489 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.489 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.489 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.489 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:54311 58.a 58:577c5622:::benchmark_data_smithi070_396992_object54310:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1585)", 2024-09-21T11:07:43.489 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:02:17.816986+0000", 2024-09-21T11:07:43.489 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 325.58140015499998, 2024-09-21T11:07:43.490 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.097134527999999998, 2024-09-21T11:07:43.490 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:43.490 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:43.490 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:43.490 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:43.490 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:43.490 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 54311 2024-09-21T11:07:43.490 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.490 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:43.490 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.490 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:43.490 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.816986+0000", 2024-09-21T11:07:43.490 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.491 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.491 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.491 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:43.491 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.816986+0000", 2024-09-21T11:07:43.491 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.491 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.491 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.491 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:43.491 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.816985+0000", 2024-09-21T11:07:43.491 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:07:43.491 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.491 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.491 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:43.491 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.817055+0000", 2024-09-21T11:07:43.491 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9688000000000001e-05 2024-09-21T11:07:43.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.492 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.492 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:43.492 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.817073+0000", 2024-09-21T11:07:43.492 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7907e-05 2024-09-21T11:07:43.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.492 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.492 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:43.492 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.817078+0000", 2024-09-21T11:07:43.492 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5439999999999998e-06 2024-09-21T11:07:43.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.492 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.492 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.492 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.818448+0000", 2024-09-21T11:07:43.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0013695280000000001 2024-09-21T11:07:43.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:43.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.818608+0000", 2024-09-21T11:07:43.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000160688 2024-09-21T11:07:43.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3", 2024-09-21T11:07:43.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.818774+0000", 2024-09-21T11:07:43.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016545500000000001 2024-09-21T11:07:43.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:43.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.820521+0000", 2024-09-21T11:07:43.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0017476130000000001 2024-09-21T11:07:43.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.494 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.494 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.494 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.913990+0000", 2024-09-21T11:07:43.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.093468564000000004 2024-09-21T11:07:43.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.494 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.494 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:43.494 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.914027+0000", 2024-09-21T11:07:43.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7416000000000002e-05 2024-09-21T11:07:43.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.494 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.494 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:43.495 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.914121+0000", 2024-09-21T11:07:43.495 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3364999999999998e-05 2024-09-21T11:07:43.495 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.495 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:43.495 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.495 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.495 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:54347 58.a 58:5080fd9e:::benchmark_data_smithi070_396992_object54346:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1585)", 2024-09-21T11:07:43.495 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:02:17.839385+0000", 2024-09-21T11:07:43.495 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 325.55900185799999, 2024-09-21T11:07:43.495 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.074932762, 2024-09-21T11:07:43.495 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:43.496 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:43.496 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:43.496 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:43.496 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:43.496 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 54347 2024-09-21T11:07:43.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.496 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:43.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.496 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:43.496 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.839385+0000", 2024-09-21T11:07:43.496 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.496 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:43.497 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.839385+0000", 2024-09-21T11:07:43.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.497 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.497 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.497 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:43.497 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.839383+0000", 2024-09-21T11:07:43.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:07:43.497 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.497 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.497 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:43.497 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.839423+0000", 2024-09-21T11:07:43.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9378000000000001e-05 2024-09-21T11:07:43.497 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.497 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.497 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:43.498 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.839434+0000", 2024-09-21T11:07:43.498 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0988000000000001e-05 2024-09-21T11:07:43.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.498 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:43.498 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.839439+0000", 2024-09-21T11:07:43.498 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1789999999999999e-06 2024-09-21T11:07:43.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.498 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.498 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.839475+0000", 2024-09-21T11:07:43.498 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6473999999999997e-05 2024-09-21T11:07:43.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.499 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:43.499 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.839605+0000", 2024-09-21T11:07:43.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013014100000000001 2024-09-21T11:07:43.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.499 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3", 2024-09-21T11:07:43.499 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.839767+0000", 2024-09-21T11:07:43.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000161819 2024-09-21T11:07:43.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.499 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:43.499 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.841523+0000", 2024-09-21T11:07:43.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0017561409999999999 2024-09-21T11:07:43.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.500 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.500 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.500 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.914178+0000", 2024-09-21T11:07:43.500 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.072655090000000006 2024-09-21T11:07:43.500 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.500 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.500 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:43.500 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.914236+0000", 2024-09-21T11:07:43.500 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7503999999999997e-05 2024-09-21T11:07:43.500 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.500 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.500 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:43.500 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:17.914317+0000", 2024-09-21T11:07:43.500 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1404999999999997e-05 2024-09-21T11:07:43.501 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.501 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:43.501 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.501 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.501 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.501 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:217103 58.a 58:54e92bc7:::benchmark_data_smithi070_396992_object71144:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1603)", 2024-09-21T11:07:43.501 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:31.714814+0000", 2024-09-21T11:07:43.501 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 191.683572758, 2024-09-21T11:07:43.501 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.064414930999999995, 2024-09-21T11:07:43.501 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:43.501 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:43.501 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:43.501 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:43.501 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:43.502 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 217103 2024-09-21T11:07:43.502 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.502 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:43.502 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.502 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:43.502 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.714814+0000", 2024-09-21T11:07:43.502 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.502 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.502 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.502 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:43.502 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.714814+0000", 2024-09-21T11:07:43.502 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.502 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.502 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.503 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:43.503 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.714812+0000", 2024-09-21T11:07:43.503 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:07:43.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.503 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.503 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:43.503 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.714819+0000", 2024-09-21T11:07:43.503 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5139999999999998e-06 2024-09-21T11:07:43.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.503 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.503 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:43.503 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.714833+0000", 2024-09-21T11:07:43.503 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.38e-05 2024-09-21T11:07:43.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.504 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:43.504 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.714841+0000", 2024-09-21T11:07:43.504 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.96e-06 2024-09-21T11:07:43.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.504 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.504 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.714870+0000", 2024-09-21T11:07:43.504 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9332999999999999e-05 2024-09-21T11:07:43.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.504 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:43.504 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.714949+0000", 2024-09-21T11:07:43.504 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8529000000000007e-05 2024-09-21T11:07:43.505 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.505 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", 2024-09-21T11:07:43.505 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.715062+0000", 2024-09-21T11:07:43.505 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011300500000000001 2024-09-21T11:07:43.505 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.505 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.505 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.770234+0000", 2024-09-21T11:07:43.505 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.055172713999999998 2024-09-21T11:07:43.505 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.505 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:43.505 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.770252+0000", 2024-09-21T11:07:43.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7567e-05 2024-09-21T11:07:43.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.506 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.506 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.779013+0000", 2024-09-21T11:07:43.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0087616250000000003 2024-09-21T11:07:43.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.506 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:43.506 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.779137+0000", 2024-09-21T11:07:43.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000124022 2024-09-21T11:07:43.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.507 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:43.507 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:31.779229+0000", 2024-09-21T11:07:43.507 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1111000000000002e-05 2024-09-21T11:07:43.507 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.507 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:43.507 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.507 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.507 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:241152 58.a 58:5494570e:::benchmark_data_smithi070_396992_object95193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1619)", 2024-09-21T11:07:43.507 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:00.365098+0000", 2024-09-21T11:07:43.507 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 163.03328848300001, 2024-09-21T11:07:43.507 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.077856499999999995, 2024-09-21T11:07:43.507 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:43.507 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:43.508 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:43.508 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:43.508 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:43.508 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 241152 2024-09-21T11:07:43.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.508 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:43.508 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.508 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:43.508 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.365098+0000", 2024-09-21T11:07:43.508 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.508 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.508 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:43.508 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.365098+0000", 2024-09-21T11:07:43.509 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.509 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:43.509 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.365096+0000", 2024-09-21T11:07:43.509 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:07:43.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.509 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:43.509 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.365103+0000", 2024-09-21T11:07:43.509 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.443e-06 2024-09-21T11:07:43.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.509 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:43.510 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.365116+0000", 2024-09-21T11:07:43.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3023000000000001e-05 2024-09-21T11:07:43.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.510 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:43.510 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.365123+0000", 2024-09-21T11:07:43.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3429999999999998e-06 2024-09-21T11:07:43.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.510 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.510 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.365486+0000", 2024-09-21T11:07:43.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000362987 2024-09-21T11:07:43.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.511 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:43.511 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.368785+0000", 2024-09-21T11:07:43.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0032987139999999999 2024-09-21T11:07:43.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.511 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", 2024-09-21T11:07:43.511 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.369328+0000", 2024-09-21T11:07:43.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00054261299999999995 2024-09-21T11:07:43.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.511 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.511 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.375700+0000", 2024-09-21T11:07:43.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0063721130000000004 2024-09-21T11:07:43.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:43.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.375708+0000", 2024-09-21T11:07:43.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8189999999999998e-06 2024-09-21T11:07:43.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.442771+0000", 2024-09-21T11:07:43.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.067062347999999994 2024-09-21T11:07:43.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.513 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:43.513 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.442856+0000", 2024-09-21T11:07:43.513 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5641000000000002e-05 2024-09-21T11:07:43.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.513 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:43.513 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:00.442954+0000", 2024-09-21T11:07:43.513 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7973999999999994e-05 2024-09-21T11:07:43.513 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.513 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:43.513 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.514 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:244080 58.a 58:5105e019:::benchmark_data_smithi070_396992_object98121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1619)", 2024-09-21T11:07:43.514 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:03.220591+0000", 2024-09-21T11:07:43.514 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 160.17779585400001, 2024-09-21T11:07:43.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.064642745000000001, 2024-09-21T11:07:43.514 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:43.514 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:43.514 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:43.514 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:43.514 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:43.514 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 244080 2024-09-21T11:07:43.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.514 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:43.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.515 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:43.515 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.220591+0000", 2024-09-21T11:07:43.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.515 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:43.515 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.220591+0000", 2024-09-21T11:07:43.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.515 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:43.515 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.220589+0000", 2024-09-21T11:07:43.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T11:07:43.516 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.516 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.516 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:43.516 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.220597+0000", 2024-09-21T11:07:43.516 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9440000000000005e-06 2024-09-21T11:07:43.516 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.516 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.516 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:43.516 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.220612+0000", 2024-09-21T11:07:43.516 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.537e-05 2024-09-21T11:07:43.516 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.516 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.516 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:43.517 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.220619+0000", 2024-09-21T11:07:43.517 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.294e-06 2024-09-21T11:07:43.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.517 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.517 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.220668+0000", 2024-09-21T11:07:43.517 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8690000000000003e-05 2024-09-21T11:07:43.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.517 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:43.517 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.220972+0000", 2024-09-21T11:07:43.517 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00030425500000000002 2024-09-21T11:07:43.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", 2024-09-21T11:07:43.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.221314+0000", 2024-09-21T11:07:43.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034205699999999999 2024-09-21T11:07:43.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.226865+0000", 2024-09-21T11:07:43.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0055506540000000004 2024-09-21T11:07:43.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:43.519 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.228071+0000", 2024-09-21T11:07:43.519 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00120634 2024-09-21T11:07:43.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.519 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.519 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.285083+0000", 2024-09-21T11:07:43.519 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.057011586000000003 2024-09-21T11:07:43.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.519 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:43.519 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.285132+0000", 2024-09-21T11:07:43.519 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9141999999999998e-05 2024-09-21T11:07:43.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:43.520 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.285233+0000", 2024-09-21T11:07:43.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000101087 2024-09-21T11:07:43.520 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.520 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:43.520 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.520 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:244085 58.a 58:5529bd05:::benchmark_data_smithi070_396992_object98126:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1619)", 2024-09-21T11:07:43.520 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:03.229366+0000", 2024-09-21T11:07:43.520 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 160.16902052699999, 2024-09-21T11:07:43.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.063603385999999998, 2024-09-21T11:07:43.520 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:43.521 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:43.521 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:43.521 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:43.521 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:43.521 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 244085 2024-09-21T11:07:43.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.521 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:43.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:43.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229366+0000", 2024-09-21T11:07:43.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:43.522 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229366+0000", 2024-09-21T11:07:43.522 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.522 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.522 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:43.522 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229364+0000", 2024-09-21T11:07:43.522 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T11:07:43.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.522 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.522 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:43.522 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229372+0000", 2024-09-21T11:07:43.522 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5939999999999996e-06 2024-09-21T11:07:43.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.522 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.523 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:43.523 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229391+0000", 2024-09-21T11:07:43.523 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9494999999999999e-05 2024-09-21T11:07:43.523 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.523 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.523 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:43.523 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229399+0000", 2024-09-21T11:07:43.523 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8730000000000007e-06 2024-09-21T11:07:43.523 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.523 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.523 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.523 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229473+0000", 2024-09-21T11:07:43.523 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4270000000000006e-05 2024-09-21T11:07:43.523 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.523 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.524 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:43.524 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229590+0000", 2024-09-21T11:07:43.524 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011636700000000001 2024-09-21T11:07:43.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.524 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", 2024-09-21T11:07:43.524 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229740+0000", 2024-09-21T11:07:43.524 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000150118 2024-09-21T11:07:43.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.524 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:43.524 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.236236+0000", 2024-09-21T11:07:43.525 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0064965609999999997 2024-09-21T11:07:43.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.525 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.525 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.525 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.237063+0000", 2024-09-21T11:07:43.525 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00082698600000000004 2024-09-21T11:07:43.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.525 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.525 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.525 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.292822+0000", 2024-09-21T11:07:43.525 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.055758421000000002 2024-09-21T11:07:43.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.525 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.525 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:43.526 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.292876+0000", 2024-09-21T11:07:43.526 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3973e-05 2024-09-21T11:07:43.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.526 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.526 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:43.526 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.292969+0000", 2024-09-21T11:07:43.526 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3496000000000001e-05 2024-09-21T11:07:43.526 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.526 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:43.526 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.526 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.526 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:244088 58.a 58:56c2f617:::benchmark_data_smithi070_396992_object98129:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1619)", 2024-09-21T11:07:43.526 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:03.229413+0000", 2024-09-21T11:07:43.526 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 160.16897335900001, 2024-09-21T11:07:43.527 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.065123039999999993, 2024-09-21T11:07:43.527 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:43.527 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:43.527 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:43.527 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:43.527 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:43.527 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 244088 2024-09-21T11:07:43.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.527 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:43.527 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.527 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:43.527 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229413+0000", 2024-09-21T11:07:43.527 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.528 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.528 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.528 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:43.528 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229413+0000", 2024-09-21T11:07:43.528 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.528 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.528 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.528 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:43.528 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229413+0000", 2024-09-21T11:07:43.528 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T11:07:43.528 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.528 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.528 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:43.528 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229416+0000", 2024-09-21T11:07:43.529 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5829999999999998e-06 2024-09-21T11:07:43.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.529 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.529 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:43.529 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229423+0000", 2024-09-21T11:07:43.529 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.199e-06 2024-09-21T11:07:43.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.529 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.529 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:43.529 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.229428+0000", 2024-09-21T11:07:43.529 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1229999999999999e-06 2024-09-21T11:07:43.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.529 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.529 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.529 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.232821+0000", 2024-09-21T11:07:43.530 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0033924860000000001 2024-09-21T11:07:43.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.530 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.530 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:43.530 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.232951+0000", 2024-09-21T11:07:43.530 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013008799999999999 2024-09-21T11:07:43.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.530 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.530 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", 2024-09-21T11:07:43.530 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.233071+0000", 2024-09-21T11:07:43.530 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011995899999999999 2024-09-21T11:07:43.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.530 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.530 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:43.531 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.236467+0000", 2024-09-21T11:07:43.531 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003395595 2024-09-21T11:07:43.531 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.531 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.531 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.531 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.237169+0000", 2024-09-21T11:07:43.531 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00070235399999999998 2024-09-21T11:07:43.531 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.531 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.531 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.531 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.294371+0000", 2024-09-21T11:07:43.531 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.057201895000000003 2024-09-21T11:07:43.531 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.531 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.532 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:43.532 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.294428+0000", 2024-09-21T11:07:43.532 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6697999999999998e-05 2024-09-21T11:07:43.532 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.532 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.532 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:43.532 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:03.294536+0000", 2024-09-21T11:07:43.532 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010857 2024-09-21T11:07:43.532 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.532 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:43.532 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.532 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.532 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.532 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:274468 58.a 58:507d8d22:::benchmark_data_smithi070_396992_object128509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1626)", 2024-09-21T11:07:43.533 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:34.418785+0000", 2024-09-21T11:07:43.533 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 128.97960140699999, 2024-09-21T11:07:43.533 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.074562790000000004, 2024-09-21T11:07:43.533 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:43.533 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:43.533 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:43.533 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:43.533 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:43.533 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 274468 2024-09-21T11:07:43.533 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.533 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:43.533 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.533 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:43.533 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:34.418785+0000", 2024-09-21T11:07:43.534 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.534 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.534 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:43.534 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:34.418785+0000", 2024-09-21T11:07:43.534 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.534 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.534 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:43.534 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:34.418784+0000", 2024-09-21T11:07:43.534 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:07:43.534 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.534 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:43.534 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:34.418789+0000", 2024-09-21T11:07:43.535 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1510000000000003e-06 2024-09-21T11:07:43.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.535 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.535 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:43.535 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:34.418798+0000", 2024-09-21T11:07:43.535 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0240000000000003e-06 2024-09-21T11:07:43.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.535 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.535 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:43.535 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:34.418803+0000", 2024-09-21T11:07:43.535 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4890000000000003e-06 2024-09-21T11:07:43.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.535 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.536 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.536 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:34.420160+0000", 2024-09-21T11:07:43.536 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0013562789999999999 2024-09-21T11:07:43.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.536 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.536 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:43.536 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:34.420285+0000", 2024-09-21T11:07:43.536 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000125926 2024-09-21T11:07:43.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.536 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.536 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", 2024-09-21T11:07:43.536 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:34.420403+0000", 2024-09-21T11:07:43.536 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011805499999999999 2024-09-21T11:07:43.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.536 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.537 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:43.537 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:34.424549+0000", 2024-09-21T11:07:43.537 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0041452349999999997 2024-09-21T11:07:43.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.537 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.537 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:34.425942+0000", 2024-09-21T11:07:43.537 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0013931810000000001 2024-09-21T11:07:43.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.537 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.537 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:34.493191+0000", 2024-09-21T11:07:43.537 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.067248611 2024-09-21T11:07:43.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.538 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:43.538 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:34.493246+0000", 2024-09-21T11:07:43.538 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5065999999999998e-05 2024-09-21T11:07:43.538 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.538 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:43.538 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:34.493348+0000", 2024-09-21T11:07:43.538 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000102193 2024-09-21T11:07:43.538 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.538 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:43.538 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.538 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.538 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:278112 58.a 58:581e4285:::benchmark_data_smithi070_396992_object132153:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1629)", 2024-09-21T11:07:43.539 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:38.751016+0000", 2024-09-21T11:07:43.539 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 124.647370866, 2024-09-21T11:07:43.539 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.094552412000000002, 2024-09-21T11:07:43.539 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:43.539 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:43.539 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:43.539 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:43.539 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:43.539 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 278112 2024-09-21T11:07:43.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.539 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:43.539 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.539 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:43.539 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.751016+0000", 2024-09-21T11:07:43.540 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.540 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.540 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:43.540 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.751016+0000", 2024-09-21T11:07:43.540 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.540 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.540 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:43.540 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.751014+0000", 2024-09-21T11:07:43.540 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:07:43.540 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.540 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:43.541 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.751020+0000", 2024-09-21T11:07:43.541 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2589999999999997e-06 2024-09-21T11:07:43.541 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.541 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.541 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:43.541 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.751036+0000", 2024-09-21T11:07:43.541 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6127999999999999e-05 2024-09-21T11:07:43.541 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.541 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.541 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:43.541 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.751044+0000", 2024-09-21T11:07:43.541 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4379999999999998e-06 2024-09-21T11:07:43.541 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.541 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.542 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.542 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.755847+0000", 2024-09-21T11:07:43.542 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0048034490000000004 2024-09-21T11:07:43.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.542 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.542 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:43.542 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.756005+0000", 2024-09-21T11:07:43.542 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015763999999999999 2024-09-21T11:07:43.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.542 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.542 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", 2024-09-21T11:07:43.542 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.756130+0000", 2024-09-21T11:07:43.542 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012558599999999999 2024-09-21T11:07:43.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.762498+0000", 2024-09-21T11:07:43.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0063674589999999998 2024-09-21T11:07:43.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:43.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.762503+0000", 2024-09-21T11:07:43.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3310000000000003e-06 2024-09-21T11:07:43.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.845261+0000", 2024-09-21T11:07:43.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.082757647000000004 2024-09-21T11:07:43.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:43.544 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.845318+0000", 2024-09-21T11:07:43.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6919000000000002e-05 2024-09-21T11:07:43.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:43.544 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.845568+0000", 2024-09-21T11:07:43.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025018499999999999 2024-09-21T11:07:43.544 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.544 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:43.544 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.545 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.545 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:278115 58.a 58:5af2b6fa:::benchmark_data_smithi070_396992_object132156:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1629)", 2024-09-21T11:07:43.545 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:38.755273+0000", 2024-09-21T11:07:43.545 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 124.64311354199999, 2024-09-21T11:07:43.545 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.093948772999999999, 2024-09-21T11:07:43.545 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:43.545 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:43.545 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:43.545 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:43.545 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:43.545 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 278115 2024-09-21T11:07:43.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.545 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:43.545 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.546 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:43.546 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.755273+0000", 2024-09-21T11:07:43.546 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.546 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:43.546 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.755273+0000", 2024-09-21T11:07:43.546 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.546 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:43.546 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.755272+0000", 2024-09-21T11:07:43.546 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:07:43.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.547 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:43.547 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.755277+0000", 2024-09-21T11:07:43.547 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1460000000000004e-06 2024-09-21T11:07:43.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.547 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:43.547 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.755304+0000", 2024-09-21T11:07:43.547 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6883999999999999e-05 2024-09-21T11:07:43.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.547 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:43.547 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.755310+0000", 2024-09-21T11:07:43.547 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5400000000000001e-06 2024-09-21T11:07:43.548 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.548 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.548 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.548 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.759921+0000", 2024-09-21T11:07:43.548 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0046107250000000004 2024-09-21T11:07:43.548 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.548 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.548 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:43.548 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.760020+0000", 2024-09-21T11:07:43.548 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9072000000000003e-05 2024-09-21T11:07:43.548 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.548 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.548 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", 2024-09-21T11:07:43.548 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.760093+0000", 2024-09-21T11:07:43.549 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2381999999999999e-05 2024-09-21T11:07:43.549 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.549 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.549 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.549 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.762928+0000", 2024-09-21T11:07:43.549 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0028353179999999999 2024-09-21T11:07:43.549 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.549 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.549 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:43.549 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.766813+0000", 2024-09-21T11:07:43.549 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0038855539999999998 2024-09-21T11:07:43.549 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.549 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.549 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.550 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.849117+0000", 2024-09-21T11:07:43.550 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.082303813000000003 2024-09-21T11:07:43.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.550 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:43.550 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.849161+0000", 2024-09-21T11:07:43.550 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3535999999999998e-05 2024-09-21T11:07:43.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.550 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:43.550 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.849222+0000", 2024-09-21T11:07:43.550 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0803e-05 2024-09-21T11:07:43.550 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.551 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:43.551 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.551 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.551 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:278116 58.a 58:50149a44:::benchmark_data_smithi070_396992_object132157:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1629)", 2024-09-21T11:07:43.551 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:38.755675+0000", 2024-09-21T11:07:43.551 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 124.642711758, 2024-09-21T11:07:43.551 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.093755818000000005, 2024-09-21T11:07:43.551 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:43.551 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:43.551 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:43.551 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:43.551 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:43.551 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 278116 2024-09-21T11:07:43.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.552 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:43.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.552 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:43.552 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.755675+0000", 2024-09-21T11:07:43.552 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.552 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:43.552 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.755675+0000", 2024-09-21T11:07:43.552 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.552 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:43.552 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.755674+0000", 2024-09-21T11:07:43.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:07:43.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.553 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:43.553 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.755679+0000", 2024-09-21T11:07:43.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1170000000000003e-06 2024-09-21T11:07:43.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.553 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:43.553 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.755688+0000", 2024-09-21T11:07:43.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7469999999999994e-06 2024-09-21T11:07:43.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:43.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.755701+0000", 2024-09-21T11:07:43.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3321e-05 2024-09-21T11:07:43.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.762946+0000", 2024-09-21T11:07:43.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0072449180000000004 2024-09-21T11:07:43.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:43.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.763022+0000", 2024-09-21T11:07:43.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6062000000000005e-05 2024-09-21T11:07:43.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.555 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.555 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", 2024-09-21T11:07:43.555 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.763093+0000", 2024-09-21T11:07:43.555 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1019000000000006e-05 2024-09-21T11:07:43.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.555 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.555 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.555 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.766796+0000", 2024-09-21T11:07:43.555 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0037030280000000001 2024-09-21T11:07:43.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.555 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.555 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:43.556 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.766849+0000", 2024-09-21T11:07:43.556 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2803000000000002e-05 2024-09-21T11:07:43.556 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.556 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.556 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.556 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.849268+0000", 2024-09-21T11:07:43.556 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.082419495999999995 2024-09-21T11:07:43.556 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.556 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.556 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:43.556 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.849320+0000", 2024-09-21T11:07:43.556 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2080999999999998e-05 2024-09-21T11:07:43.556 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.556 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.556 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:43.557 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.849430+0000", 2024-09-21T11:07:43.557 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000110004 2024-09-21T11:07:43.557 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.557 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:43.557 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.557 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:278119 58.a 58:51178c57:::benchmark_data_smithi070_396992_object132160:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1629)", 2024-09-21T11:07:43.557 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:38.763352+0000", 2024-09-21T11:07:43.557 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 124.63503486, 2024-09-21T11:07:43.557 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.089594847000000005, 2024-09-21T11:07:43.557 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:43.557 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:43.558 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:43.558 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:43.558 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:43.558 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 278119 2024-09-21T11:07:43.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.558 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:43.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.558 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:43.558 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.763352+0000", 2024-09-21T11:07:43.558 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.558 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:43.558 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.763352+0000", 2024-09-21T11:07:43.559 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.559 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.559 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:43.559 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.763351+0000", 2024-09-21T11:07:43.559 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:07:43.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.559 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.559 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:43.559 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.763356+0000", 2024-09-21T11:07:43.559 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6069999999999998e-06 2024-09-21T11:07:43.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.559 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.559 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:43.559 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.763368+0000", 2024-09-21T11:07:43.560 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1429e-05 2024-09-21T11:07:43.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.560 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:43.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.763375+0000", 2024-09-21T11:07:43.560 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.182e-06 2024-09-21T11:07:43.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.560 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.765005+0000", 2024-09-21T11:07:43.560 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001630711 2024-09-21T11:07:43.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.560 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:43.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.765080+0000", 2024-09-21T11:07:43.561 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5006999999999995e-05 2024-09-21T11:07:43.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.561 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", 2024-09-21T11:07:43.561 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.765157+0000", 2024-09-21T11:07:43.561 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6209000000000002e-05 2024-09-21T11:07:43.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.561 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.561 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.766961+0000", 2024-09-21T11:07:43.561 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001804162 2024-09-21T11:07:43.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.562 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:43.562 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.767021+0000", 2024-09-21T11:07:43.562 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0510000000000002e-05 2024-09-21T11:07:43.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.562 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.562 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:43.562 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.852778+0000", 2024-09-21T11:07:43.562 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.085756597000000004 2024-09-21T11:07:43.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.562 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.562 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:43.562 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.852831+0000", 2024-09-21T11:07:43.562 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3353999999999998e-05 2024-09-21T11:07:43.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.563 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.563 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:43.563 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:38.852946+0000", 2024-09-21T11:07:43.563 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000115114 2024-09-21T11:07:43.563 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.563 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:43.563 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.563 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.563 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.563 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDPGPushReply(58.a 1629/1590 [PushReplyOp(58:500f59ea:::benchmark_data_smithi070_396992_object140904:head)])", 2024-09-21T11:07:43.563 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:39.322307+0000", 2024-09-21T11:07:43.563 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 124.07607960199999, 2024-09-21T11:07:43.563 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.099159406000000005, 2024-09-21T11:07:43.563 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:43.564 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T11:07:43.564 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:43.564 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.564 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:43.564 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:39.322307+0000", 2024-09-21T11:07:43.564 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.564 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.564 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.564 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:43.564 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:39.322307+0000", 2024-09-21T11:07:43.564 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:43.564 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.564 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.564 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:43.565 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:39.322306+0000", 2024-09-21T11:07:43.565 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:07:43.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.565 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.565 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:43.565 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:39.322312+0000", 2024-09-21T11:07:43.565 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.105e-06 2024-09-21T11:07:43.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.565 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.565 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:43.565 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:39.322326+0000", 2024-09-21T11:07:43.565 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.345e-05 2024-09-21T11:07:43.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.566 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.566 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:43.566 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:39.322333+0000", 2024-09-21T11:07:43.566 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5510000000000002e-06 2024-09-21T11:07:43.566 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.566 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.566 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:43.566 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:39.421268+0000", 2024-09-21T11:07:43.566 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098935175 2024-09-21T11:07:43.566 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:43.566 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:43.566 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:43.566 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:39.421466+0000", 2024-09-21T11:07:43.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019786099999999999 2024-09-21T11:07:43.567 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.567 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:43.567 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.567 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:43.567 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:43.567 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T11:07:43.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:43 smithi160 ceph-mon[101879]: pgmap v3348: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:43.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:43 smithi070 ceph-mon[107385]: pgmap v3348: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:43.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:43 smithi070 ceph-mon[112139]: pgmap v3348: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:44.719 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T11:07:44.916 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:07:45.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:45 smithi160 ceph-mon[101879]: pgmap v3349: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:45.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:45 smithi070 ceph-mon[107385]: pgmap v3349: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:45.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:45 smithi070 ceph-mon[112139]: pgmap v3349: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:46.651 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T11:07:46.651 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T11:07:46.651 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T11:07:46.651 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T11:07:46.873 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-21T11:07:46.873 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-09-21T11:07:46.873 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd in 5 2024-09-21T11:07:47.084 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:07:47.327 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T11:07:47.478 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:47 smithi070 ceph-mon[107385]: pgmap v3350: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:47.478 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:47 smithi070 ceph-mon[112139]: pgmap v3350: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:47.527 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:07:47.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:47 smithi160 ceph-mon[101879]: pgmap v3350: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:49.272 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T11:07:49.272 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T11:07:49.272 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T11:07:49.272 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T11:07:49.272 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T11:07:49.377 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:49 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a[107362]: 2024-09-21T11:07:49.113+0000 7fad4a465640 -1 mon.a@0(leader).osd e1683 definitely_dead 0 2024-09-21T11:07:49.432 INFO:teuthology.orchestra.run.smithi070.stderr:marked in osd.5. 2024-09-21T11:07:49.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:49 smithi160 ceph-mon[101879]: pgmap v3351: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:49.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:49 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/4266955334' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-21T11:07:49.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:49 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-21T11:07:49.813 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:49 smithi070 ceph-mon[107385]: pgmap v3351: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:49.813 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:49 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/4266955334' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-21T11:07:49.813 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:49 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-21T11:07:49.814 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:49 smithi070 ceph-mon[112139]: pgmap v3351: 93 pgs: 93 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:49.814 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:49 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/4266955334' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-21T11:07:49.814 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:49 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-21T11:07:49.922 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T11:07:50.119 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:07:50.135 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-09-21T11:07:50.135 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-21T11:07:50.135 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-09-21T11:07:50.135 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd in 2 2024-09-21T11:07:50.329 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:07:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:50 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-21T11:07:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:50 smithi160 ceph-mon[101879]: osdmap e1684: 8 total, 8 up, 7 in 2024-09-21T11:07:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:50 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:07:50.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:50 smithi160 ceph-mon[101879]: osdmap e1685: 8 total, 8 up, 7 in 2024-09-21T11:07:50.697 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:50 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-21T11:07:50.697 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:50 smithi070 ceph-mon[112139]: osdmap e1684: 8 total, 8 up, 7 in 2024-09-21T11:07:50.697 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:50 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:07:50.697 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:50 smithi070 ceph-mon[112139]: osdmap e1685: 8 total, 8 up, 7 in 2024-09-21T11:07:50.698 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:50 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-21T11:07:50.698 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:50 smithi070 ceph-mon[107385]: osdmap e1684: 8 total, 8 up, 7 in 2024-09-21T11:07:50.698 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:50 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:07:50.698 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:50 smithi070 ceph-mon[107385]: osdmap e1685: 8 total, 8 up, 7 in 2024-09-21T11:07:51.564 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:51 smithi160 ceph-mon[101879]: pgmap v3353: 93 pgs: 15 peering, 78 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:51.564 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:51 smithi160 ceph-mon[101879]: osdmap e1686: 8 total, 8 up, 7 in 2024-09-21T11:07:51.711 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:51 smithi070 ceph-mon[107385]: pgmap v3353: 93 pgs: 15 peering, 78 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:51.711 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:51 smithi070 ceph-mon[107385]: osdmap e1686: 8 total, 8 up, 7 in 2024-09-21T11:07:51.711 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:51 smithi070 ceph-mon[112139]: pgmap v3353: 93 pgs: 15 peering, 78 active+clean; 582 KiB data, 6.0 GiB used, 530 GiB / 536 GiB avail 2024-09-21T11:07:51.712 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:51 smithi070 ceph-mon[112139]: osdmap e1686: 8 total, 8 up, 7 in 2024-09-21T11:07:51.815 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T11:07:51.815 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T11:07:51.815 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T11:07:51.815 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T11:07:51.815 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.815 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:87532 58.7 58.afd54b97 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e1589)", 2024-09-21T11:07:51.815 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:02:37.469322+0000", 2024-09-21T11:07:51.815 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 314.34229041700002, 2024-09-21T11:07:51.815 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.002815215, 2024-09-21T11:07:51.816 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:51.816 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T11:07:51.816 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:51.816 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:51.816 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:51.816 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 87532 2024-09-21T11:07:51.816 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.816 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:51.816 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.816 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:51.817 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.469322+0000", 2024-09-21T11:07:51.817 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.817 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.817 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.817 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:51.817 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.469322+0000", 2024-09-21T11:07:51.817 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.817 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.817 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.817 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:51.817 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.469321+0000", 2024-09-21T11:07:51.817 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:07:51.817 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.817 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.818 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:51.818 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.469484+0000", 2024-09-21T11:07:51.818 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016368099999999999 2024-09-21T11:07:51.818 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.818 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.818 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:51.818 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.469507+0000", 2024-09-21T11:07:51.818 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2113999999999999e-05 2024-09-21T11:07:51.818 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.818 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.818 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:51.818 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.469516+0000", 2024-09-21T11:07:51.818 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0299999999999999e-06 2024-09-21T11:07:51.819 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.819 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.819 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.819 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.478523+0000", 2024-09-21T11:07:51.819 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0090072239999999994 2024-09-21T11:07:51.819 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.819 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.819 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:51.819 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.478531+0000", 2024-09-21T11:07:51.819 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0800000000000006e-06 2024-09-21T11:07:51.819 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.819 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.819 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.820 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:38.472130+0000", 2024-09-21T11:07:51.820 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99359867700000004 2024-09-21T11:07:51.820 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.820 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.820 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:51.820 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:38.472138+0000", 2024-09-21T11:07:51.820 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0679999999999997e-06 2024-09-21T11:07:51.820 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.821 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:51.821 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.821 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.821 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.821 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:87535 58.1 58.179cb551 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e1589)", 2024-09-21T11:07:51.821 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:02:37.469540+0000", 2024-09-21T11:07:51.821 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 314.34207296900001, 2024-09-21T11:07:51.822 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99163556500000005, 2024-09-21T11:07:51.822 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:51.822 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T11:07:51.822 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:51.822 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:51.823 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:51.823 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 87535 2024-09-21T11:07:51.823 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.823 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:51.824 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.824 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:51.824 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.469540+0000", 2024-09-21T11:07:51.824 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.824 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.824 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.824 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:51.824 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.469540+0000", 2024-09-21T11:07:51.824 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.824 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.824 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.825 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:51.825 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.469539+0000", 2024-09-21T11:07:51.825 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T11:07:51.825 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.825 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.825 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:51.825 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.469605+0000", 2024-09-21T11:07:51.825 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5742999999999997e-05 2024-09-21T11:07:51.825 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.826 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.826 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:51.826 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.469613+0000", 2024-09-21T11:07:51.826 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2360000000000004e-06 2024-09-21T11:07:51.826 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.826 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.826 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:51.826 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.469619+0000", 2024-09-21T11:07:51.826 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1500000000000004e-06 2024-09-21T11:07:51.826 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.827 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.827 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.827 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.478724+0000", 2024-09-21T11:07:51.827 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0091047960000000001 2024-09-21T11:07:51.827 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.827 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.827 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:51.827 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.478730+0000", 2024-09-21T11:07:51.827 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5169999999999999e-06 2024-09-21T11:07:51.827 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.827 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.827 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.828 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:38.461165+0000", 2024-09-21T11:07:51.828 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98243536899999995 2024-09-21T11:07:51.828 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.828 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.828 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:51.828 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:38.461175+0000", 2024-09-21T11:07:51.828 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0414000000000001e-05 2024-09-21T11:07:51.828 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.828 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:51.828 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.828 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.828 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.828 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:87542 58.7 58.ccad6587 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1589)", 2024-09-21T11:07:51.829 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:02:37.483700+0000", 2024-09-21T11:07:51.829 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 314.32791318, 2024-09-21T11:07:51.829 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98846070900000005, 2024-09-21T11:07:51.829 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:51.829 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T11:07:51.829 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:51.829 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:51.829 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:51.829 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 87542 2024-09-21T11:07:51.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.829 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:51.829 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.829 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:51.829 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.483700+0000", 2024-09-21T11:07:51.829 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.830 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.830 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.830 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:51.830 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.483700+0000", 2024-09-21T11:07:51.830 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.830 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.830 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.830 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:51.830 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.483698+0000", 2024-09-21T11:07:51.830 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:07:51.830 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.830 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.830 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:51.831 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.483759+0000", 2024-09-21T11:07:51.831 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0865000000000001e-05 2024-09-21T11:07:51.831 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.831 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.831 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:51.831 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.483770+0000", 2024-09-21T11:07:51.831 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0726e-05 2024-09-21T11:07:51.831 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.831 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.831 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:51.831 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.483779+0000", 2024-09-21T11:07:51.831 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7849999999999997e-06 2024-09-21T11:07:51.831 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.831 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.832 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.832 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.483906+0000", 2024-09-21T11:07:51.832 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012746099999999999 2024-09-21T11:07:51.832 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.832 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.832 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:51.832 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.483914+0000", 2024-09-21T11:07:51.832 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9580000000000007e-06 2024-09-21T11:07:51.832 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.832 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.832 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.832 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:38.472155+0000", 2024-09-21T11:07:51.832 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98824039900000005 2024-09-21T11:07:51.832 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.833 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.833 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:51.834 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:38.472160+0000", 2024-09-21T11:07:51.834 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8250000000000001e-06 2024-09-21T11:07:51.834 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.834 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:51.834 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.834 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.835 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.835 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:87548 58.1 58.4becb4a1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1589)", 2024-09-21T11:07:51.835 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:02:37.500622+0000", 2024-09-21T11:07:51.835 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 314.31099069700002, 2024-09-21T11:07:51.835 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96058235800000003, 2024-09-21T11:07:51.835 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:51.835 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T11:07:51.835 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:51.835 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:51.835 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:51.835 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 87548 2024-09-21T11:07:51.836 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.836 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:51.836 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.836 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:51.836 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.500622+0000", 2024-09-21T11:07:51.836 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.836 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.836 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.836 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:51.837 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.500622+0000", 2024-09-21T11:07:51.837 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.837 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.837 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.837 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:51.837 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.500620+0000", 2024-09-21T11:07:51.837 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T11:07:51.838 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.838 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.838 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:51.838 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.500680+0000", 2024-09-21T11:07:51.838 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9540000000000003e-05 2024-09-21T11:07:51.838 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.839 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.839 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:51.839 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.500698+0000", 2024-09-21T11:07:51.839 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8050000000000002e-05 2024-09-21T11:07:51.839 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.839 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.839 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:51.840 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.500705+0000", 2024-09-21T11:07:51.840 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0149999999999998e-06 2024-09-21T11:07:51.840 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.840 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.840 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.840 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.500744+0000", 2024-09-21T11:07:51.841 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9326000000000003e-05 2024-09-21T11:07:51.841 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.841 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.841 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:51.841 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:37.500751+0000", 2024-09-21T11:07:51.841 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3270000000000003e-06 2024-09-21T11:07:51.842 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.842 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.842 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.842 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:38.461197+0000", 2024-09-21T11:07:51.842 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96044581299999998 2024-09-21T11:07:51.842 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.843 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.843 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:51.843 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:02:38.461204+0000", 2024-09-21T11:07:51.843 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3429999999999998e-06 2024-09-21T11:07:51.843 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.843 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:51.844 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.844 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.844 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.844 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDPGPush(58.9 1594/1590 [PushOp(58:9d7f89a0:::benchmark_data_smithi070_396992_object14078:head, version: 1584'895, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(58:9d7f89a0:::benchmark_data_smithi070_396992_object14078:head@1584'895, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T11:07:51.844 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:03:17.123563+0000", 2024-09-21T11:07:51.844 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 274.68804966300002, 2024-09-21T11:07:51.844 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16743028800000001, 2024-09-21T11:07:51.845 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:51.845 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T11:07:51.845 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:51.845 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.845 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:51.845 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:03:17.123563+0000", 2024-09-21T11:07:51.846 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.846 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.846 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.846 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:51.846 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:03:17.123563+0000", 2024-09-21T11:07:51.846 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.846 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.847 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.847 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:51.847 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:03:17.123562+0000", 2024-09-21T11:07:51.847 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:07:51.847 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.847 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.848 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:51.848 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:03:17.123632+0000", 2024-09-21T11:07:51.848 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9614000000000001e-05 2024-09-21T11:07:51.848 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.848 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.848 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:51.849 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:03:17.123653+0000", 2024-09-21T11:07:51.849 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1628e-05 2024-09-21T11:07:51.849 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.849 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.849 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:51.849 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:03:17.123668+0000", 2024-09-21T11:07:51.849 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4207e-05 2024-09-21T11:07:51.850 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.850 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.850 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.850 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:03:17.287714+0000", 2024-09-21T11:07:51.850 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.164046211 2024-09-21T11:07:51.850 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.850 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.851 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:51.851 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:03:17.287725+0000", 2024-09-21T11:07:51.851 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1256e-05 2024-09-21T11:07:51.851 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.851 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.851 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:51.851 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:03:17.290993+0000", 2024-09-21T11:07:51.852 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0032683019999999998 2024-09-21T11:07:51.852 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.852 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:51.852 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.852 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.852 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.852 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:219122 58.4 58.9f49d114 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e1604)", 2024-09-21T11:07:51.853 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:33.525219+0000", 2024-09-21T11:07:51.853 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 198.28639338799999, 2024-09-21T11:07:51.853 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99499420400000005, 2024-09-21T11:07:51.853 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:51.853 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T11:07:51.853 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:51.854 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:51.854 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:51.854 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 219122 2024-09-21T11:07:51.854 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.854 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:51.854 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.854 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:51.855 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.525219+0000", 2024-09-21T11:07:51.855 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.855 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.855 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.855 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:51.855 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.525219+0000", 2024-09-21T11:07:51.856 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.856 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.856 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:51.856 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.525218+0000", 2024-09-21T11:07:51.856 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:07:51.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.857 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.857 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:51.857 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.525224+0000", 2024-09-21T11:07:51.857 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0440000000000001e-06 2024-09-21T11:07:51.857 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.857 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.857 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:51.858 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.525233+0000", 2024-09-21T11:07:51.858 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1810000000000007e-06 2024-09-21T11:07:51.858 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.858 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.858 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:51.858 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.525242+0000", 2024-09-21T11:07:51.858 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4160000000000004e-06 2024-09-21T11:07:51.859 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.859 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.859 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.859 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.527728+0000", 2024-09-21T11:07:51.859 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002486416 2024-09-21T11:07:51.859 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.859 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.860 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:51.860 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.527735+0000", 2024-09-21T11:07:51.860 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4699999999999996e-06 2024-09-21T11:07:51.860 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.860 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.860 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.861 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.520206+0000", 2024-09-21T11:07:51.861 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99247047200000005 2024-09-21T11:07:51.861 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.861 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.861 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:51.861 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.520214+0000", 2024-09-21T11:07:51.861 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7190000000000001e-06 2024-09-21T11:07:51.862 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.862 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:51.862 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.862 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.862 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.862 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:219125 58.4 58.c3feb0d4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e1604)", 2024-09-21T11:07:51.863 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:33.525255+0000", 2024-09-21T11:07:51.863 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 198.28635826600001, 2024-09-21T11:07:51.863 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.994982001, 2024-09-21T11:07:51.863 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:51.863 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T11:07:51.863 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:51.863 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:51.863 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:51.864 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 219125 2024-09-21T11:07:51.864 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.864 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:51.864 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.864 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:51.864 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.525255+0000", 2024-09-21T11:07:51.865 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.865 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.865 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.865 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:51.865 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.525255+0000", 2024-09-21T11:07:51.865 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.865 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.866 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.866 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:51.866 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.525254+0000", 2024-09-21T11:07:51.866 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T11:07:51.866 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.866 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.866 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:51.867 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.525257+0000", 2024-09-21T11:07:51.867 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.638e-06 2024-09-21T11:07:51.867 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.867 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.867 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:51.867 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.525263+0000", 2024-09-21T11:07:51.867 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.562e-06 2024-09-21T11:07:51.868 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.868 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.868 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:51.868 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.525268+0000", 2024-09-21T11:07:51.868 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.746e-06 2024-09-21T11:07:51.868 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.869 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.869 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.869 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.527770+0000", 2024-09-21T11:07:51.869 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0025021990000000001 2024-09-21T11:07:51.869 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.869 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.870 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:51.870 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.527777+0000", 2024-09-21T11:07:51.870 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7889999999999997e-06 2024-09-21T11:07:51.870 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.870 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.870 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.871 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.520230+0000", 2024-09-21T11:07:51.871 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99245334399999996 2024-09-21T11:07:51.871 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.871 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.871 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:51.871 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.520237+0000", 2024-09-21T11:07:51.872 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1360000000000002e-06 2024-09-21T11:07:51.872 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.872 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:51.872 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.872 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.872 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.872 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:219132 58.4 58.1afae6e4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e1604)", 2024-09-21T11:07:51.873 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:33.525300+0000", 2024-09-21T11:07:51.873 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 198.28631299099999, 2024-09-21T11:07:51.873 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99495808699999999, 2024-09-21T11:07:51.873 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:51.873 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T11:07:51.873 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:51.873 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:51.873 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:51.874 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 219132 2024-09-21T11:07:51.874 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.874 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:51.874 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.874 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:51.874 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.525300+0000", 2024-09-21T11:07:51.874 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.874 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.875 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.875 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:51.875 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.525300+0000", 2024-09-21T11:07:51.875 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.875 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.875 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.876 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:51.876 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.525299+0000", 2024-09-21T11:07:51.876 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T11:07:51.876 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.876 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.876 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:51.876 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.525303+0000", 2024-09-21T11:07:51.876 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2550000000000002e-06 2024-09-21T11:07:51.877 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.877 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.877 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:51.877 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.525310+0000", 2024-09-21T11:07:51.877 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3629999999999996e-06 2024-09-21T11:07:51.878 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.878 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.878 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:51.878 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.525315+0000", 2024-09-21T11:07:51.878 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.276e-06 2024-09-21T11:07:51.878 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.879 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.879 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.879 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.527797+0000", 2024-09-21T11:07:51.879 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0024821969999999998 2024-09-21T11:07:51.879 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.879 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.879 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:51.880 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.527885+0000", 2024-09-21T11:07:51.880 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7367000000000006e-05 2024-09-21T11:07:51.880 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.880 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.880 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.880 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.520252+0000", 2024-09-21T11:07:51.880 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99236706100000005 2024-09-21T11:07:51.881 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.881 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.881 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:51.881 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.520258+0000", 2024-09-21T11:07:51.881 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0589999999999999e-06 2024-09-21T11:07:51.881 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.881 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:51.882 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.882 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.882 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.882 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:219138 58.4 58.e81fcf24 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e1604)", 2024-09-21T11:07:51.882 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:33.525450+0000", 2024-09-21T11:07:51.882 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 198.286162644, 2024-09-21T11:07:51.883 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99482884900000002, 2024-09-21T11:07:51.883 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:51.883 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T11:07:51.883 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:51.883 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:51.884 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:51.885 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 219138 2024-09-21T11:07:51.887 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.887 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:51.888 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.889 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:51.889 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.525450+0000", 2024-09-21T11:07:51.890 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.890 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.891 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.891 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:51.891 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.525450+0000", 2024-09-21T11:07:51.892 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.892 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.893 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.893 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:51.893 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.525450+0000", 2024-09-21T11:07:51.893 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T11:07:51.893 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.893 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.893 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:51.894 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.525453+0000", 2024-09-21T11:07:51.894 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.49e-06 2024-09-21T11:07:51.894 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.894 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.894 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:51.894 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.525462+0000", 2024-09-21T11:07:51.894 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0440000000000001e-06 2024-09-21T11:07:51.895 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.895 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.895 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:51.895 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.525468+0000", 2024-09-21T11:07:51.895 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0249999999999999e-06 2024-09-21T11:07:51.895 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.895 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.896 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.896 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.527936+0000", 2024-09-21T11:07:51.896 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0024681130000000001 2024-09-21T11:07:51.896 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.896 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.896 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:51.897 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.527942+0000", 2024-09-21T11:07:51.897 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8449999999999999e-06 2024-09-21T11:07:51.897 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.897 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.897 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.897 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.520273+0000", 2024-09-21T11:07:51.897 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99233097299999995 2024-09-21T11:07:51.898 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.898 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.898 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:51.898 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.520279+0000", 2024-09-21T11:07:51.898 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9390000000000002e-06 2024-09-21T11:07:51.898 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.898 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:51.899 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.899 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.899 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.899 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:219169 58.4 58.63f8cf44 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1604)", 2024-09-21T11:07:51.899 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:04:33.564115+0000", 2024-09-21T11:07:51.899 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 198.24749767399999, 2024-09-21T11:07:51.900 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.95618401200000003, 2024-09-21T11:07:51.900 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:51.900 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T11:07:51.900 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:51.900 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:51.900 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:51.900 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 219169 2024-09-21T11:07:51.901 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.901 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:51.901 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.901 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:51.901 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.564115+0000", 2024-09-21T11:07:51.901 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.901 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.902 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.902 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:51.902 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.564115+0000", 2024-09-21T11:07:51.902 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.902 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.902 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.903 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:51.903 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.564114+0000", 2024-09-21T11:07:51.903 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:07:51.903 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.903 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.903 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:51.903 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.564122+0000", 2024-09-21T11:07:51.904 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9699999999999999e-06 2024-09-21T11:07:51.904 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.904 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.904 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:51.904 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.564133+0000", 2024-09-21T11:07:51.904 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.112e-05 2024-09-21T11:07:51.905 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.905 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.905 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:51.905 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.564139+0000", 2024-09-21T11:07:51.905 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5670000000000002e-06 2024-09-21T11:07:51.905 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.905 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.906 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.906 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.564176+0000", 2024-09-21T11:07:51.906 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7443e-05 2024-09-21T11:07:51.906 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.906 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.906 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:51.907 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:33.564181+0000", 2024-09-21T11:07:51.907 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4260000000000002e-06 2024-09-21T11:07:51.907 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.907 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.907 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.907 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.520293+0000", 2024-09-21T11:07:51.908 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.95611197299999995 2024-09-21T11:07:51.908 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.908 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.908 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:51.908 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:04:34.520299+0000", 2024-09-21T11:07:51.908 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.772e-06 2024-09-21T11:07:51.909 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.909 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:51.909 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.909 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.909 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.909 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:285137 58.6 58:6e5c6547:::benchmark_data_smithi070_396992_object139178:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1644)", 2024-09-21T11:07:51.909 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:55.546126+0000", 2024-09-21T11:07:51.910 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 116.265486822, 2024-09-21T11:07:51.910 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.010406967, 2024-09-21T11:07:51.910 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:51.910 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:51.910 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:51.910 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:51.911 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:51.911 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 285137 2024-09-21T11:07:51.911 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.911 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:51.911 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.911 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:51.911 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.546126+0000", 2024-09-21T11:07:51.911 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.911 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.912 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.912 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:51.912 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.546126+0000", 2024-09-21T11:07:51.912 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.912 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.912 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.912 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:51.912 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.546125+0000", 2024-09-21T11:07:51.912 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:07:51.912 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.912 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.912 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:51.912 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.546130+0000", 2024-09-21T11:07:51.912 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4750000000000001e-06 2024-09-21T11:07:51.913 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.913 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.913 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:51.913 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.546141+0000", 2024-09-21T11:07:51.913 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1143e-05 2024-09-21T11:07:51.913 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.913 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.913 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:51.913 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.546148+0000", 2024-09-21T11:07:51.913 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.007e-06 2024-09-21T11:07:51.913 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.913 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.913 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.913 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.546212+0000", 2024-09-21T11:07:51.914 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3419999999999999e-05 2024-09-21T11:07:51.914 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.914 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.914 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:51.914 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.546221+0000", 2024-09-21T11:07:51.914 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0929999999999992e-06 2024-09-21T11:07:51.914 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.914 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.914 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.914 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.549160+0000", 2024-09-21T11:07:51.914 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0029394490000001 2024-09-21T11:07:51.914 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.914 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.915 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:51.915 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.549305+0000", 2024-09-21T11:07:51.915 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014459299999999999 2024-09-21T11:07:51.915 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.915 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.915 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T11:07:51.915 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.549733+0000", 2024-09-21T11:07:51.915 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00042815600000000001 2024-09-21T11:07:51.915 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.915 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.915 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:51.915 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.556383+0000", 2024-09-21T11:07:51.915 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0066493840000000004 2024-09-21T11:07:51.915 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.916 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.916 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:51.916 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.556438+0000", 2024-09-21T11:07:51.916 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5804000000000003e-05 2024-09-21T11:07:51.916 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.916 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.916 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:51.916 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.556448+0000", 2024-09-21T11:07:51.916 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8949999999999993e-06 2024-09-21T11:07:51.916 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.916 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.916 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:51.916 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.556494+0000", 2024-09-21T11:07:51.916 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5596000000000002e-05 2024-09-21T11:07:51.917 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.917 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.917 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:51.917 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.556533+0000", 2024-09-21T11:07:51.917 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9128999999999998e-05 2024-09-21T11:07:51.917 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.917 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:51.917 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.917 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.917 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.917 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:285155 58.6 58:69d89e74:::benchmark_data_smithi070_396992_object139196:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1644)", 2024-09-21T11:07:51.917 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:55.569531+0000", 2024-09-21T11:07:51.917 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 116.24208142099999, 2024-09-21T11:07:51.917 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.993659187, 2024-09-21T11:07:51.918 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:51.918 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:51.918 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:51.918 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:51.918 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:51.918 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 285155 2024-09-21T11:07:51.918 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.918 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:51.918 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.918 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:51.918 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.569531+0000", 2024-09-21T11:07:51.918 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.918 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.919 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.919 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:51.919 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.569531+0000", 2024-09-21T11:07:51.919 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.919 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.919 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.919 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:51.919 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.569529+0000", 2024-09-21T11:07:51.919 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T11:07:51.919 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.919 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.919 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:51.919 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.569537+0000", 2024-09-21T11:07:51.919 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1289999999999998e-06 2024-09-21T11:07:51.920 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.920 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.920 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:51.920 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.569551+0000", 2024-09-21T11:07:51.920 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4255e-05 2024-09-21T11:07:51.920 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.920 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.920 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:51.920 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.569558+0000", 2024-09-21T11:07:51.920 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7460000000000003e-06 2024-09-21T11:07:51.920 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.920 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.921 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.921 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.569600+0000", 2024-09-21T11:07:51.921 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2113999999999997e-05 2024-09-21T11:07:51.921 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.921 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.921 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:51.921 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.569609+0000", 2024-09-21T11:07:51.922 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7650000000000005e-06 2024-09-21T11:07:51.922 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.922 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.922 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.922 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.556552+0000", 2024-09-21T11:07:51.922 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98694208999999999 2024-09-21T11:07:51.922 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.922 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.922 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:51.923 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.556626+0000", 2024-09-21T11:07:51.923 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4585999999999994e-05 2024-09-21T11:07:51.923 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.923 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.923 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T11:07:51.923 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.556705+0000", 2024-09-21T11:07:51.923 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8894999999999996e-05 2024-09-21T11:07:51.923 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.923 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.924 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:51.924 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.559444+0000", 2024-09-21T11:07:51.924 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002739305 2024-09-21T11:07:51.924 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.924 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.924 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:51.925 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.561504+0000", 2024-09-21T11:07:51.925 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0020594070000000001 2024-09-21T11:07:51.925 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.925 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.925 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:51.925 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.563102+0000", 2024-09-21T11:07:51.925 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001598323 2024-09-21T11:07:51.925 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.926 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.926 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:51.926 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.563152+0000", 2024-09-21T11:07:51.926 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9845999999999997e-05 2024-09-21T11:07:51.926 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.926 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.926 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:51.927 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.563191+0000", 2024-09-21T11:07:51.927 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8682e-05 2024-09-21T11:07:51.927 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.927 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:51.927 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.927 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.927 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.928 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:285156 58.6 58:6bbb7c52:::benchmark_data_smithi070_396992_object139197:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1644)", 2024-09-21T11:07:51.928 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:55.569712+0000", 2024-09-21T11:07:51.928 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 116.24190031099999, 2024-09-21T11:07:51.928 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99369606899999996, 2024-09-21T11:07:51.928 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:51.928 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:51.929 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:51.929 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:51.929 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:51.929 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 285156 2024-09-21T11:07:51.929 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.929 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:51.929 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.929 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:51.929 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.569712+0000", 2024-09-21T11:07:51.929 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.929 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.929 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.930 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:51.930 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.569712+0000", 2024-09-21T11:07:51.930 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.930 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.930 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.930 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:51.930 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.569712+0000", 2024-09-21T11:07:51.930 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:07:51.930 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.930 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.930 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:51.930 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.569716+0000", 2024-09-21T11:07:51.930 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2869999999999998e-06 2024-09-21T11:07:51.930 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.931 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.931 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:51.931 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.569726+0000", 2024-09-21T11:07:51.931 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0475000000000001e-05 2024-09-21T11:07:51.931 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.931 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.931 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:51.931 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.569732+0000", 2024-09-21T11:07:51.931 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8409999999999996e-06 2024-09-21T11:07:51.931 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.931 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.931 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.931 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.569766+0000", 2024-09-21T11:07:51.931 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3312999999999998e-05 2024-09-21T11:07:51.932 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.932 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.932 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:51.932 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.569770+0000", 2024-09-21T11:07:51.932 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3889999999999998e-06 2024-09-21T11:07:51.932 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.932 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.932 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.932 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.558078+0000", 2024-09-21T11:07:51.932 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98830792000000001 2024-09-21T11:07:51.932 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.932 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.932 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:51.932 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.558168+0000", 2024-09-21T11:07:51.933 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9802999999999994e-05 2024-09-21T11:07:51.933 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.933 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.933 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T11:07:51.933 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.558243+0000", 2024-09-21T11:07:51.933 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5104999999999993e-05 2024-09-21T11:07:51.933 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.933 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.933 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:51.933 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.563058+0000", 2024-09-21T11:07:51.933 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00481483 2024-09-21T11:07:51.934 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.934 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.934 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:51.935 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.563267+0000", 2024-09-21T11:07:51.935 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00020924 2024-09-21T11:07:51.935 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.935 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.935 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:51.935 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.563324+0000", 2024-09-21T11:07:51.935 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7070000000000001e-05 2024-09-21T11:07:51.935 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.935 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.935 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:51.935 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.563371+0000", 2024-09-21T11:07:51.935 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6585000000000002e-05 2024-09-21T11:07:51.935 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.935 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.936 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:51.936 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.563409+0000", 2024-09-21T11:07:51.936 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8000999999999998e-05 2024-09-21T11:07:51.936 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.936 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:51.936 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.936 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.936 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.936 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:285177 58.6 58:619501ad:::benchmark_data_smithi070_396992_object139218:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1644)", 2024-09-21T11:07:51.936 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:55.583338+0000", 2024-09-21T11:07:51.936 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 116.228274481, 2024-09-21T11:07:51.936 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98675431199999997, 2024-09-21T11:07:51.936 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:51.936 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:51.937 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:51.937 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:51.937 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:51.937 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 285177 2024-09-21T11:07:51.937 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.937 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:51.937 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.937 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:51.937 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.583338+0000", 2024-09-21T11:07:51.937 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.937 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.937 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.937 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:51.938 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.583338+0000", 2024-09-21T11:07:51.938 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.938 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.938 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.938 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:51.938 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.583337+0000", 2024-09-21T11:07:51.938 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:07:51.938 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.938 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.938 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:51.938 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.583343+0000", 2024-09-21T11:07:51.938 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0140000000000004e-06 2024-09-21T11:07:51.938 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.939 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.939 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:51.939 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.583354+0000", 2024-09-21T11:07:51.939 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0434999999999999e-05 2024-09-21T11:07:51.939 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.939 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.939 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:51.939 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.583360+0000", 2024-09-21T11:07:51.939 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2820000000000002e-06 2024-09-21T11:07:51.939 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.939 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.939 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.939 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.583408+0000", 2024-09-21T11:07:51.939 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8418999999999998e-05 2024-09-21T11:07:51.940 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.940 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.940 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:51.940 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.583416+0000", 2024-09-21T11:07:51.940 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3939999999999998e-06 2024-09-21T11:07:51.940 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.940 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.940 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.940 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.559570+0000", 2024-09-21T11:07:51.940 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.97615485800000001 2024-09-21T11:07:51.940 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.940 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.940 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:51.940 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.559677+0000", 2024-09-21T11:07:51.941 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010615699999999999 2024-09-21T11:07:51.941 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.941 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.941 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T11:07:51.941 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.559817+0000", 2024-09-21T11:07:51.941 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014008099999999999 2024-09-21T11:07:51.941 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.941 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.941 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:51.941 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.563249+0000", 2024-09-21T11:07:51.941 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0034326920000000002 2024-09-21T11:07:51.941 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.941 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.941 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:51.942 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.563424+0000", 2024-09-21T11:07:51.942 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017477399999999999 2024-09-21T11:07:51.942 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.942 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.942 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:51.942 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.569968+0000", 2024-09-21T11:07:51.942 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0065440189999999999 2024-09-21T11:07:51.942 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.942 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.942 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:51.942 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.570031+0000", 2024-09-21T11:07:51.942 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2784000000000001e-05 2024-09-21T11:07:51.942 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.943 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.943 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:51.943 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.570093+0000", 2024-09-21T11:07:51.943 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1668000000000007e-05 2024-09-21T11:07:51.943 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.943 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:51.943 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.943 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.943 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.943 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:285180 58.6 58:6b3e3423:::benchmark_data_smithi070_396992_object139221:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1644)", 2024-09-21T11:07:51.943 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:55.587277+0000", 2024-09-21T11:07:51.943 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 116.224335967, 2024-09-21T11:07:51.943 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98295173700000005, 2024-09-21T11:07:51.943 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:51.944 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:51.944 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:51.944 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:51.944 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:51.944 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 285180 2024-09-21T11:07:51.944 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.944 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:51.944 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.944 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:51.944 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.587277+0000", 2024-09-21T11:07:51.944 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.944 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.944 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.944 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:51.945 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.587277+0000", 2024-09-21T11:07:51.945 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.945 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.945 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.945 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:51.945 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.587275+0000", 2024-09-21T11:07:51.945 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T11:07:51.945 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.945 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.945 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:51.945 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.587285+0000", 2024-09-21T11:07:51.945 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0541e-05 2024-09-21T11:07:51.945 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.945 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.945 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:51.946 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.587297+0000", 2024-09-21T11:07:51.946 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1671000000000001e-05 2024-09-21T11:07:51.946 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.946 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.946 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:51.946 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.587302+0000", 2024-09-21T11:07:51.946 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1789999999999999e-06 2024-09-21T11:07:51.946 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.946 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.946 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.946 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.587351+0000", 2024-09-21T11:07:51.946 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8976e-05 2024-09-21T11:07:51.946 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.947 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.947 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:51.947 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.587360+0000", 2024-09-21T11:07:51.947 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2e-06 2024-09-21T11:07:51.947 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.947 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.947 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.947 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.561546+0000", 2024-09-21T11:07:51.947 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.97418575100000004 2024-09-21T11:07:51.947 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.947 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.947 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:51.947 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.561641+0000", 2024-09-21T11:07:51.947 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4900000000000003e-05 2024-09-21T11:07:51.948 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.948 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.948 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T11:07:51.948 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.561715+0000", 2024-09-21T11:07:51.948 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4198999999999999e-05 2024-09-21T11:07:51.948 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.948 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.948 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:51.948 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.563429+0000", 2024-09-21T11:07:51.948 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001713687 2024-09-21T11:07:51.948 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.948 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.948 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:51.948 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.565707+0000", 2024-09-21T11:07:51.949 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022779100000000002 2024-09-21T11:07:51.949 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.949 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.949 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:51.949 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.570143+0000", 2024-09-21T11:07:51.949 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0044361729999999999 2024-09-21T11:07:51.949 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.949 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.949 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:51.949 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.570190+0000", 2024-09-21T11:07:51.949 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6974999999999997e-05 2024-09-21T11:07:51.949 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.949 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.949 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:51.950 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.570229+0000", 2024-09-21T11:07:51.950 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8482999999999997e-05 2024-09-21T11:07:51.950 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.950 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:51.950 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.950 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.950 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.950 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:285190 58.6 58:6eaaee39:::benchmark_data_smithi070_396992_object139231:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1644)", 2024-09-21T11:07:51.950 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:55.594038+0000", 2024-09-21T11:07:51.950 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 116.217574511, 2024-09-21T11:07:51.950 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.9763212, 2024-09-21T11:07:51.950 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:51.950 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:51.950 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:51.951 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:51.951 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:51.951 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 285190 2024-09-21T11:07:51.952 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.952 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:51.952 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.952 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:51.952 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.594038+0000", 2024-09-21T11:07:51.952 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.952 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.952 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.952 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:51.952 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.594038+0000", 2024-09-21T11:07:51.952 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.952 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.952 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.953 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:51.953 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.594037+0000", 2024-09-21T11:07:51.953 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-21T11:07:51.953 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.953 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.953 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:51.953 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.594044+0000", 2024-09-21T11:07:51.953 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6399999999999997e-06 2024-09-21T11:07:51.953 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.953 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.953 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:51.953 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.594060+0000", 2024-09-21T11:07:51.953 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5979999999999999e-05 2024-09-21T11:07:51.953 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.954 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.954 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:51.954 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.594068+0000", 2024-09-21T11:07:51.954 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.401e-06 2024-09-21T11:07:51.954 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.954 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.954 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.954 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.594121+0000", 2024-09-21T11:07:51.954 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3174000000000003e-05 2024-09-21T11:07:51.954 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.954 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.954 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:51.954 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.594129+0000", 2024-09-21T11:07:51.954 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1929999999999993e-06 2024-09-21T11:07:51.955 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.955 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.955 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.955 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.563455+0000", 2024-09-21T11:07:51.955 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96932588399999997 2024-09-21T11:07:51.955 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.955 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.955 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:51.955 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.563577+0000", 2024-09-21T11:07:51.955 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012164500000000001 2024-09-21T11:07:51.955 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.955 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.955 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T11:07:51.956 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.563706+0000", 2024-09-21T11:07:51.956 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012940799999999999 2024-09-21T11:07:51.956 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.956 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.956 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:51.956 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.565777+0000", 2024-09-21T11:07:51.956 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0020707859999999998 2024-09-21T11:07:51.956 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.956 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.956 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:51.956 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.569907+0000", 2024-09-21T11:07:51.956 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0041298519999999998 2024-09-21T11:07:51.956 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.957 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.957 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:51.957 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.570274+0000", 2024-09-21T11:07:51.957 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00036757800000000002 2024-09-21T11:07:51.957 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.957 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.957 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:51.957 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.570323+0000", 2024-09-21T11:07:51.957 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8401e-05 2024-09-21T11:07:51.957 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.957 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.957 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:51.957 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.570359+0000", 2024-09-21T11:07:51.958 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6959000000000003e-05 2024-09-21T11:07:51.958 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.958 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:51.958 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.958 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.958 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.958 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:285191 58.6 58:6205befc:::benchmark_data_smithi070_396992_object139232:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1644)", 2024-09-21T11:07:51.958 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:55.594356+0000", 2024-09-21T11:07:51.958 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 116.217257096, 2024-09-21T11:07:51.958 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.97617330999999996, 2024-09-21T11:07:51.958 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:51.958 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:51.958 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:51.959 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:51.959 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:51.959 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 285191 2024-09-21T11:07:51.959 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.959 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:51.959 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.959 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:51.959 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.594356+0000", 2024-09-21T11:07:51.959 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.959 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.959 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.959 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:51.959 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.594356+0000", 2024-09-21T11:07:51.959 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.960 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.960 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.960 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:51.960 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.594355+0000", 2024-09-21T11:07:51.960 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:07:51.960 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.960 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.960 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:51.960 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.594363+0000", 2024-09-21T11:07:51.960 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0620000000000001e-06 2024-09-21T11:07:51.960 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.960 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.960 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:51.960 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.594375+0000", 2024-09-21T11:07:51.960 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2437e-05 2024-09-21T11:07:51.961 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.961 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.961 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:51.961 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.594383+0000", 2024-09-21T11:07:51.961 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6459999999999993e-06 2024-09-21T11:07:51.961 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.961 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.961 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.961 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.594450+0000", 2024-09-21T11:07:51.961 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7234e-05 2024-09-21T11:07:51.961 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.961 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.961 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:51.961 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.594457+0000", 2024-09-21T11:07:51.962 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8600000000000004e-06 2024-09-21T11:07:51.962 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.962 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.962 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.962 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.565810+0000", 2024-09-21T11:07:51.962 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.97135344899999998 2024-09-21T11:07:51.962 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.962 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.962 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:51.962 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.565956+0000", 2024-09-21T11:07:51.962 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014576300000000001 2024-09-21T11:07:51.962 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.962 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.962 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T11:07:51.963 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.566102+0000", 2024-09-21T11:07:51.963 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014578700000000001 2024-09-21T11:07:51.963 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.963 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.963 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:51.963 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.570104+0000", 2024-09-21T11:07:51.963 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0040016510000000002 2024-09-21T11:07:51.963 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.963 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.963 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:51.963 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.570392+0000", 2024-09-21T11:07:51.963 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028820099999999997 2024-09-21T11:07:51.963 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.964 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.964 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:51.964 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.570448+0000", 2024-09-21T11:07:51.964 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6140999999999997e-05 2024-09-21T11:07:51.964 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.964 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.964 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:51.964 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.570493+0000", 2024-09-21T11:07:51.964 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5019999999999999e-05 2024-09-21T11:07:51.964 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.964 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.964 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:51.964 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.570529+0000", 2024-09-21T11:07:51.965 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5988000000000002e-05 2024-09-21T11:07:51.965 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.965 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:51.965 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.965 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.965 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.965 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:285197 58.6 58:61d45b41:::benchmark_data_smithi070_396992_object139238:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1644)", 2024-09-21T11:07:51.965 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:55.605001+0000", 2024-09-21T11:07:51.965 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 116.206612244, 2024-09-21T11:07:51.965 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96581929899999996, 2024-09-21T11:07:51.965 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:51.965 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:51.965 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:51.965 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:51.966 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:51.966 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 285197 2024-09-21T11:07:51.966 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.966 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:51.966 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.966 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:51.966 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.605001+0000", 2024-09-21T11:07:51.966 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.966 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.966 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.966 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:51.966 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.605001+0000", 2024-09-21T11:07:51.966 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.967 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.967 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.967 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:51.967 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.604999+0000", 2024-09-21T11:07:51.967 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:07:51.967 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.967 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.967 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:51.967 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.605006+0000", 2024-09-21T11:07:51.967 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.584e-06 2024-09-21T11:07:51.967 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.967 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.967 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:51.967 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.605018+0000", 2024-09-21T11:07:51.968 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2017e-05 2024-09-21T11:07:51.968 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.968 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.968 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:51.968 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.605026+0000", 2024-09-21T11:07:51.968 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8590000000000005e-06 2024-09-21T11:07:51.968 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.968 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.968 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.969 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.605067+0000", 2024-09-21T11:07:51.969 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1307000000000003e-05 2024-09-21T11:07:51.969 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.969 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.969 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:51.969 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.605074+0000", 2024-09-21T11:07:51.969 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2760000000000003e-06 2024-09-21T11:07:51.969 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.969 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.969 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.969 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.567940+0000", 2024-09-21T11:07:51.969 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96286536700000003 2024-09-21T11:07:51.969 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.969 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.970 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:51.970 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.568027+0000", 2024-09-21T11:07:51.970 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7360000000000004e-05 2024-09-21T11:07:51.970 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.970 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.970 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T11:07:51.970 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.568113+0000", 2024-09-21T11:07:51.970 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6174000000000005e-05 2024-09-21T11:07:51.970 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.970 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.970 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:51.970 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.570238+0000", 2024-09-21T11:07:51.970 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021247309999999999 2024-09-21T11:07:51.970 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.971 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.971 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:51.971 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.570417+0000", 2024-09-21T11:07:51.971 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017863800000000001 2024-09-21T11:07:51.971 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.971 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.971 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:51.971 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.570587+0000", 2024-09-21T11:07:51.971 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000169984 2024-09-21T11:07:51.971 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.971 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.971 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:51.971 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.570641+0000", 2024-09-21T11:07:51.972 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4060000000000001e-05 2024-09-21T11:07:51.972 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.972 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.972 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:51.972 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.570820+0000", 2024-09-21T11:07:51.972 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017926300000000001 2024-09-21T11:07:51.972 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.972 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:51.972 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.972 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.972 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.972 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:285202 58.6 58:60252d4e:::benchmark_data_smithi070_396992_object139243:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1644)", 2024-09-21T11:07:51.972 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:55.610652+0000", 2024-09-21T11:07:51.973 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 116.20096079699999, 2024-09-21T11:07:51.973 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96860272000000003, 2024-09-21T11:07:51.973 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:51.973 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:51.973 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:51.973 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:51.973 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:51.973 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 285202 2024-09-21T11:07:51.973 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.973 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:51.973 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.973 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:51.973 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.610652+0000", 2024-09-21T11:07:51.973 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.973 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.974 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.974 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:51.974 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.610652+0000", 2024-09-21T11:07:51.974 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.974 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.974 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.974 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:51.974 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.610651+0000", 2024-09-21T11:07:51.974 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:07:51.974 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.974 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.974 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:51.974 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.610656+0000", 2024-09-21T11:07:51.975 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2989999999999996e-06 2024-09-21T11:07:51.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.975 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.975 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:51.975 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.610667+0000", 2024-09-21T11:07:51.975 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1103e-05 2024-09-21T11:07:51.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.975 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.975 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:51.975 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.610675+0000", 2024-09-21T11:07:51.975 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2910000000000002e-06 2024-09-21T11:07:51.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.975 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.976 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.976 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.610713+0000", 2024-09-21T11:07:51.976 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8705999999999998e-05 2024-09-21T11:07:51.976 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.976 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.976 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:51.976 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.610721+0000", 2024-09-21T11:07:51.976 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0520000000000004e-06 2024-09-21T11:07:51.976 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.976 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.976 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.976 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.570855+0000", 2024-09-21T11:07:51.976 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96013457499999999 2024-09-21T11:07:51.976 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.977 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.977 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:51.977 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.570944+0000", 2024-09-21T11:07:51.977 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9377999999999997e-05 2024-09-21T11:07:51.977 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.977 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.977 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T11:07:51.977 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.571041+0000", 2024-09-21T11:07:51.977 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6145999999999997e-05 2024-09-21T11:07:51.977 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.977 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.977 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:51.977 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.576558+0000", 2024-09-21T11:07:51.977 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0055174580000000003 2024-09-21T11:07:51.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.978 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.978 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:51.978 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.576592+0000", 2024-09-21T11:07:51.978 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3849000000000001e-05 2024-09-21T11:07:51.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.978 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.978 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:51.978 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.579185+0000", 2024-09-21T11:07:51.978 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0025932030000000001 2024-09-21T11:07:51.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.978 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.978 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:51.979 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.579222+0000", 2024-09-21T11:07:51.979 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6698e-05 2024-09-21T11:07:51.979 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.979 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:51.979 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.579255+0000", 2024-09-21T11:07:51.979 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2903e-05 2024-09-21T11:07:51.979 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.979 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:51.979 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.979 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.979 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36567.0:285216 58.6 58:6f0bf70b:::benchmark_data_smithi070_396992_object139257:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1644)", 2024-09-21T11:07:51.979 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:05:55.649537+0000", 2024-09-21T11:07:51.980 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 116.162075655, 2024-09-21T11:07:51.980 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.93056374900000005, 2024-09-21T11:07:51.980 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:07:51.980 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T11:07:51.980 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T11:07:51.980 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36567", 2024-09-21T11:07:51.980 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.70:0/1405990020", 2024-09-21T11:07:51.980 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 285216 2024-09-21T11:07:51.980 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.980 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:07:51.980 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.980 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:07:51.980 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.649537+0000", 2024-09-21T11:07:51.980 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.981 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.981 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.981 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:07:51.981 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.649537+0000", 2024-09-21T11:07:51.981 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:07:51.981 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.981 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.981 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:07:51.981 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.649536+0000", 2024-09-21T11:07:51.981 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-21T11:07:51.981 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.981 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.981 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:07:51.982 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.649543+0000", 2024-09-21T11:07:51.982 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.554e-06 2024-09-21T11:07:51.982 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.982 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.982 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:07:51.982 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.649555+0000", 2024-09-21T11:07:51.982 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2245e-05 2024-09-21T11:07:51.982 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.982 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.982 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:07:51.982 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.649561+0000", 2024-09-21T11:07:51.982 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5219999999999998e-06 2024-09-21T11:07:51.982 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.982 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.983 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.983 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.649610+0000", 2024-09-21T11:07:51.983 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9468999999999997e-05 2024-09-21T11:07:51.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.983 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.983 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-21T11:07:51.983 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:55.649621+0000", 2024-09-21T11:07:51.983 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0862999999999999e-05 2024-09-21T11:07:51.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.983 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.983 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:07:51.983 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.572800+0000", 2024-09-21T11:07:51.983 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.92317875400000005 2024-09-21T11:07:51.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.984 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:07:51.984 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.572892+0000", 2024-09-21T11:07:51.984 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1558999999999995e-05 2024-09-21T11:07:51.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.984 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-21T11:07:51.984 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.572993+0000", 2024-09-21T11:07:51.984 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010107699999999999 2024-09-21T11:07:51.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.984 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T11:07:51.984 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.579261+0000", 2024-09-21T11:07:51.985 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0062681719999999998 2024-09-21T11:07:51.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.985 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.985 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:51.985 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.579430+0000", 2024-09-21T11:07:51.985 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016880000000000001 2024-09-21T11:07:51.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.985 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.985 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T11:07:51.985 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.579992+0000", 2024-09-21T11:07:51.985 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00056248699999999997 2024-09-21T11:07:51.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.985 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.985 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T11:07:51.986 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.580067+0000", 2024-09-21T11:07:51.986 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4653000000000005e-05 2024-09-21T11:07:51.986 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:07:51.986 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:07:51.986 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:07:51.986 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:05:56.580101+0000", 2024-09-21T11:07:51.986 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4140999999999997e-05 2024-09-21T11:07:51.986 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.986 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:51.986 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.986 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:07:51.986 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:07:51.986 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T11:07:52.449 INFO:teuthology.orchestra.run.smithi070.stderr:marked in osd.2. 2024-09-21T11:07:52.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:52 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1373324477' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-21T11:07:52.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a[107362]: 2024-09-21T11:07:52.239+0000 7fad4a465640 -1 mon.a@0(leader).osd e1686 definitely_dead 0 2024-09-21T11:07:52.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:52 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1373324477' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-21T11:07:52.709 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:52 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1373324477' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-21T11:07:52.730 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T11:07:52.927 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:07:53.218 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-09-21T11:07:53.219 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-21T11:07:53.418 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:07:53.449 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:07:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:07:53] "GET /metrics HTTP/1.1" 200 34250 "" "Prometheus/2.51.0" 2024-09-21T11:07:53.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:53 smithi160 ceph-mon[101879]: pgmap v3356: 93 pgs: 2 active+undersized+remapped, 3 remapped+peering, 15 peering, 73 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 11 B/s, 1 objects/s recovering 2024-09-21T11:07:53.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:53 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1373324477' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-21T11:07:53.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:53 smithi160 ceph-mon[101879]: osdmap e1687: 8 total, 8 up, 8 in 2024-09-21T11:07:53.722 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:53 smithi070 ceph-mon[107385]: pgmap v3356: 93 pgs: 2 active+undersized+remapped, 3 remapped+peering, 15 peering, 73 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 11 B/s, 1 objects/s recovering 2024-09-21T11:07:53.722 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:53 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1373324477' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-21T11:07:53.722 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:53 smithi070 ceph-mon[107385]: osdmap e1687: 8 total, 8 up, 8 in 2024-09-21T11:07:53.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:53 smithi070 ceph-mon[112139]: pgmap v3356: 93 pgs: 2 active+undersized+remapped, 3 remapped+peering, 15 peering, 73 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 11 B/s, 1 objects/s recovering 2024-09-21T11:07:53.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:53 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1373324477' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-21T11:07:53.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:53 smithi070 ceph-mon[112139]: osdmap e1687: 8 total, 8 up, 8 in 2024-09-21T11:07:54.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:54 smithi070 ceph-mon[107385]: osdmap e1688: 8 total, 8 up, 8 in 2024-09-21T11:07:54.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:54 smithi070 ceph-mon[112139]: osdmap e1688: 8 total, 8 up, 8 in 2024-09-21T11:07:54.635 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T11:07:54.635 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T11:07:54.635 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T11:07:54.635 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T11:07:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:54 smithi160 ceph-mon[101879]: osdmap e1688: 8 total, 8 up, 8 in 2024-09-21T11:07:55.262 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T11:07:55.282 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a: {} 2024-09-21T11:07:55.283 INFO:teuthology.orchestra.run.smithi070.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T11:07:55.291 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b: {} 2024-09-21T11:07:55.291 INFO:teuthology.orchestra.run.smithi070.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T11:07:55.300 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c: {} 2024-09-21T11:07:55.300 INFO:teuthology.orchestra.run.smithi070.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T11:07:55.465 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:07:55.588 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:55 smithi160 ceph-mon[101879]: pgmap v3359: 93 pgs: 2 active+undersized+remapped, 4 remapped+peering, 12 peering, 75 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 22 B/s, 2 objects/s recovering 2024-09-21T11:07:55.588 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:55 smithi160 ceph-mon[101879]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T11:07:55.589 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:55 smithi160 ceph-mon[101879]: osdmap e1689: 8 total, 8 up, 8 in 2024-09-21T11:07:55.589 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:55 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/500026417' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T11:07:55.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:55 smithi070 ceph-mon[107385]: pgmap v3359: 93 pgs: 2 active+undersized+remapped, 4 remapped+peering, 12 peering, 75 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 22 B/s, 2 objects/s recovering 2024-09-21T11:07:55.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:55 smithi070 ceph-mon[107385]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T11:07:55.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:55 smithi070 ceph-mon[107385]: osdmap e1689: 8 total, 8 up, 8 in 2024-09-21T11:07:55.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:55 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/500026417' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T11:07:55.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:55 smithi070 ceph-mon[112139]: pgmap v3359: 93 pgs: 2 active+undersized+remapped, 4 remapped+peering, 12 peering, 75 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 22 B/s, 2 objects/s recovering 2024-09-21T11:07:55.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:55 smithi070 ceph-mon[112139]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T11:07:55.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:55 smithi070 ceph-mon[112139]: osdmap e1689: 8 total, 8 up, 8 in 2024-09-21T11:07:55.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:55 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/500026417' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T11:07:55.931 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T11:07:56.133 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:07:56.562 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:56 smithi070 ceph-mon[107385]: osdmap e1690: 8 total, 8 up, 8 in 2024-09-21T11:07:56.562 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:56 smithi070 ceph-mon[112139]: osdmap e1690: 8 total, 8 up, 8 in 2024-09-21T11:07:56.671 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:56 smithi160 ceph-mon[101879]: osdmap e1690: 8 total, 8 up, 8 in 2024-09-21T11:07:57.185 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T11:07:57.185 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T11:07:57.185 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T11:07:57.185 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T11:07:57.185 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T11:07:57.499 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:57 smithi160 ceph-mon[101879]: pgmap v3362: 93 pgs: 1 peering, 4 active+undersized+remapped, 3 remapped+peering, 85 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 129 KiB/s, 1 objects/s recovering 2024-09-21T11:07:57.499 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:57 smithi160 ceph-mon[101879]: osdmap e1691: 8 total, 8 up, 8 in 2024-09-21T11:07:57.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:57 smithi070 ceph-mon[107385]: pgmap v3362: 93 pgs: 1 peering, 4 active+undersized+remapped, 3 remapped+peering, 85 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 129 KiB/s, 1 objects/s recovering 2024-09-21T11:07:57.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:57 smithi070 ceph-mon[107385]: osdmap e1691: 8 total, 8 up, 8 in 2024-09-21T11:07:57.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:57 smithi070 ceph-mon[112139]: pgmap v3362: 93 pgs: 1 peering, 4 active+undersized+remapped, 3 remapped+peering, 85 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 129 KiB/s, 1 objects/s recovering 2024-09-21T11:07:57.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:57 smithi070 ceph-mon[112139]: osdmap e1691: 8 total, 8 up, 8 in 2024-09-21T11:07:58.168 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T11:07:58.371 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T11:07:58.477 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T11:07:58.781 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:58 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1199078470' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T11:07:58.781 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:58 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T11:07:58.781 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:58 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1199078470' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T11:07:58.782 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:58 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T11:07:58.825 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:58 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1199078470' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T11:07:58.825 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:58 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T11:07:59.189 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 0 1 2024-09-21T11:07:59.390 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:07:59.536 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:59 smithi160 ceph-mon[101879]: pgmap v3364: 93 pgs: 2 peering, 3 active+undersized+remapped, 1 remapped+peering, 87 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 105 KiB/s, 2 keys/s, 1 objects/s recovering 2024-09-21T11:07:59.537 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:59 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T11:07:59.537 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:07:59 smithi160 ceph-mon[101879]: osdmap e1692: 8 total, 8 up, 8 in 2024-09-21T11:07:59.843 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:59 smithi070 ceph-mon[107385]: pgmap v3364: 93 pgs: 2 peering, 3 active+undersized+remapped, 1 remapped+peering, 87 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 105 KiB/s, 2 keys/s, 1 objects/s recovering 2024-09-21T11:07:59.843 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:59 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T11:07:59.843 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:07:59 smithi070 ceph-mon[107385]: osdmap e1692: 8 total, 8 up, 8 in 2024-09-21T11:07:59.844 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:59 smithi070 ceph-mon[112139]: pgmap v3364: 93 pgs: 2 peering, 3 active+undersized+remapped, 1 remapped+peering, 87 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 105 KiB/s, 2 keys/s, 1 objects/s recovering 2024-09-21T11:07:59.844 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:59 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T11:07:59.844 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:07:59 smithi070 ceph-mon[112139]: osdmap e1692: 8 total, 8 up, 8 in 2024-09-21T11:08:00.116 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T11:08:00.116 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T11:08:00.116 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T11:08:00.116 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T11:08:00.116 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.117 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDPGPush(2.3 1685/1684 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 63'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@63'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T11:08:00.117 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:07:50.446870+0000", 2024-09-21T11:08:00.117 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 9.6659922910000002, 2024-09-21T11:08:00.117 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025111099999999999, 2024-09-21T11:08:00.117 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:08:00.117 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T11:08:00.117 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:08:00.118 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.118 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:08:00.118 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.446870+0000", 2024-09-21T11:08:00.118 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:08:00.118 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.119 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.119 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:08:00.119 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.446870+0000", 2024-09-21T11:08:00.119 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:08:00.119 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.119 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.119 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:08:00.120 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.446869+0000", 2024-09-21T11:08:00.120 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T11:08:00.120 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.120 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.120 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:08:00.120 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.446892+0000", 2024-09-21T11:08:00.121 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3112000000000002e-05 2024-09-21T11:08:00.121 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.121 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.121 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:08:00.121 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.446907+0000", 2024-09-21T11:08:00.121 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4831000000000001e-05 2024-09-21T11:08:00.121 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.122 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.122 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:08:00.122 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.446923+0000", 2024-09-21T11:08:00.122 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5695000000000001e-05 2024-09-21T11:08:00.122 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.122 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.123 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:08:00.123 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.446988+0000", 2024-09-21T11:08:00.123 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5319000000000003e-05 2024-09-21T11:08:00.123 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.123 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.123 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:08:00.124 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.446999+0000", 2024-09-21T11:08:00.124 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0655e-05 2024-09-21T11:08:00.124 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.124 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.124 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:08:00.124 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.447121+0000", 2024-09-21T11:08:00.124 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012219800000000001 2024-09-21T11:08:00.125 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:08:00.125 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:08:00.125 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:08:00.125 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.125 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.125 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDPGPush(3.d 1685/1684 [PushOp(3:b50e409b:::mds_snaptable:head, version: 71'1, data_included: [0~70], data_size: 70, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:b50e409b:::mds_snaptable:head@71'1, size: 70, copy_subset: [0~70], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:70, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T11:08:00.125 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:07:50.447686+0000", 2024-09-21T11:08:00.126 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 9.6651760919999994, 2024-09-21T11:08:00.126 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028939200000000002, 2024-09-21T11:08:00.126 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:08:00.126 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T11:08:00.126 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:08:00.126 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.126 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:08:00.126 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.447686+0000", 2024-09-21T11:08:00.126 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:08:00.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.127 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.127 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:08:00.127 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.447686+0000", 2024-09-21T11:08:00.127 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:08:00.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.127 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.127 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:08:00.127 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.447685+0000", 2024-09-21T11:08:00.127 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T11:08:00.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.127 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.127 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:08:00.128 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.447704+0000", 2024-09-21T11:08:00.128 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8595999999999999e-05 2024-09-21T11:08:00.128 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.128 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.128 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:08:00.128 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.447718+0000", 2024-09-21T11:08:00.128 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4005999999999999e-05 2024-09-21T11:08:00.128 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.128 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.128 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:08:00.128 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.447732+0000", 2024-09-21T11:08:00.128 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.381e-05 2024-09-21T11:08:00.128 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.128 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.128 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:08:00.129 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.447782+0000", 2024-09-21T11:08:00.129 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0705000000000003e-05 2024-09-21T11:08:00.129 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.129 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.129 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:08:00.129 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.447793+0000", 2024-09-21T11:08:00.129 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0359e-05 2024-09-21T11:08:00.129 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.129 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.129 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:08:00.129 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.447975+0000", 2024-09-21T11:08:00.129 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000182554 2024-09-21T11:08:00.129 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:08:00.129 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:08:00.130 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:08:00.130 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.130 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.130 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDPGPush(3.10 1685/1684 [PushOp(3:0d82a743:::600.00000000:head, version: 71'2, data_included: [], data_size: 0, omap_header_size: 274, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:0d82a743:::600.00000000:head@71'2, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T11:08:00.130 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:07:50.451028+0000", 2024-09-21T11:08:00.130 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 9.6618344300000008, 2024-09-21T11:08:00.130 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000191707, 2024-09-21T11:08:00.130 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:08:00.130 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T11:08:00.130 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:08:00.130 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.130 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:08:00.130 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.451028+0000", 2024-09-21T11:08:00.130 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:08:00.131 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.131 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.131 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:08:00.131 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.451028+0000", 2024-09-21T11:08:00.131 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:08:00.131 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.131 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.131 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:08:00.131 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.451027+0000", 2024-09-21T11:08:00.131 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T11:08:00.131 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.131 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.131 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:08:00.132 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.451037+0000", 2024-09-21T11:08:00.132 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9680000000000001e-06 2024-09-21T11:08:00.132 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.132 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.132 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:08:00.132 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.451050+0000", 2024-09-21T11:08:00.132 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3127e-05 2024-09-21T11:08:00.132 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.132 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.132 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:08:00.132 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.451060+0000", 2024-09-21T11:08:00.132 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6209999999999999e-06 2024-09-21T11:08:00.132 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.132 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.133 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:08:00.133 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.451114+0000", 2024-09-21T11:08:00.133 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4017999999999997e-05 2024-09-21T11:08:00.133 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.133 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.133 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:08:00.133 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.451122+0000", 2024-09-21T11:08:00.133 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7840000000000004e-06 2024-09-21T11:08:00.133 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.133 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.134 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:08:00.134 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.451219+0000", 2024-09-21T11:08:00.134 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7808999999999998e-05 2024-09-21T11:08:00.134 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:08:00.134 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:08:00.134 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:08:00.134 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.134 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.134 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDPGPush(3.10 1685/1684 [PushOp(3:09401a81:::601.00000000:head, version: 71'4, data_included: [], data_size: 0, omap_header_size: 274, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:09401a81:::601.00000000:head@71'4, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T11:08:00.134 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:07:50.451891+0000", 2024-09-21T11:08:00.134 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 9.6609710339999992, 2024-09-21T11:08:00.134 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017396999999999999, 2024-09-21T11:08:00.134 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:08:00.135 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T11:08:00.135 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:08:00.135 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.135 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:08:00.135 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.451891+0000", 2024-09-21T11:08:00.135 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:08:00.135 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.135 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.135 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:08:00.135 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.451891+0000", 2024-09-21T11:08:00.135 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:08:00.135 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.135 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.135 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:08:00.136 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.451890+0000", 2024-09-21T11:08:00.136 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T11:08:00.136 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.136 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.136 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:08:00.136 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.451899+0000", 2024-09-21T11:08:00.136 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8009999999999993e-06 2024-09-21T11:08:00.136 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.136 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.136 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:08:00.136 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.451912+0000", 2024-09-21T11:08:00.136 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2852e-05 2024-09-21T11:08:00.136 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.136 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.137 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:08:00.137 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.451922+0000", 2024-09-21T11:08:00.137 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8600000000000005e-06 2024-09-21T11:08:00.137 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.137 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.137 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:08:00.137 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.451971+0000", 2024-09-21T11:08:00.137 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8837e-05 2024-09-21T11:08:00.137 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.137 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.137 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:08:00.137 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.451978+0000", 2024-09-21T11:08:00.137 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0600000000000002e-06 2024-09-21T11:08:00.137 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.138 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.138 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:08:00.138 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.452065+0000", 2024-09-21T11:08:00.138 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7142e-05 2024-09-21T11:08:00.138 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:08:00.138 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:08:00.138 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:08:00.138 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.138 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.138 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDPGPush(3.10 1685/1684 [PushOp(3:0bd6d154:::602.00000000:head, version: 71'6, data_included: [], data_size: 0, omap_header_size: 274, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:0bd6d154:::602.00000000:head@71'6, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T11:08:00.138 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:07:50.452782+0000", 2024-09-21T11:08:00.138 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 9.6600803099999997, 2024-09-21T11:08:00.138 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015449499999999999, 2024-09-21T11:08:00.138 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:08:00.139 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T11:08:00.139 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:08:00.139 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.139 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:08:00.139 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.452782+0000", 2024-09-21T11:08:00.139 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:08:00.139 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.139 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.139 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:08:00.139 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.452782+0000", 2024-09-21T11:08:00.139 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:08:00.139 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.139 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.139 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:08:00.140 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.452781+0000", 2024-09-21T11:08:00.140 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T11:08:00.140 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.140 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.140 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:08:00.140 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.452791+0000", 2024-09-21T11:08:00.140 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5109999999999992e-06 2024-09-21T11:08:00.140 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.140 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.140 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:08:00.140 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.452803+0000", 2024-09-21T11:08:00.140 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1854e-05 2024-09-21T11:08:00.140 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.140 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.141 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:08:00.141 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.452811+0000", 2024-09-21T11:08:00.141 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.704e-06 2024-09-21T11:08:00.141 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.141 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.141 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:08:00.141 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.452855+0000", 2024-09-21T11:08:00.141 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3677000000000002e-05 2024-09-21T11:08:00.141 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.141 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.141 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:08:00.141 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.452862+0000", 2024-09-21T11:08:00.141 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6699999999999997e-06 2024-09-21T11:08:00.141 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.141 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.142 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:08:00.142 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.452936+0000", 2024-09-21T11:08:00.142 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4664000000000002e-05 2024-09-21T11:08:00.142 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:08:00.142 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:08:00.142 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:08:00.142 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.142 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.142 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDPGPush(3.10 1685/1684 [PushOp(3:0d89b25e:::603.00000000:head, version: 71'8, data_included: [], data_size: 0, omap_header_size: 274, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:0d89b25e:::603.00000000:head@71'8, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T11:08:00.142 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:07:50.453603+0000", 2024-09-21T11:08:00.142 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 9.6592592150000005, 2024-09-21T11:08:00.142 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015013900000000001, 2024-09-21T11:08:00.142 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:08:00.143 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T11:08:00.143 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:08:00.143 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.143 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:08:00.143 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.453603+0000", 2024-09-21T11:08:00.143 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:08:00.143 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.143 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.143 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:08:00.143 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.453603+0000", 2024-09-21T11:08:00.143 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:08:00.143 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.143 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.143 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:08:00.144 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.453602+0000", 2024-09-21T11:08:00.144 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T11:08:00.144 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.144 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.144 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:08:00.144 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.453610+0000", 2024-09-21T11:08:00.144 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6240000000000002e-06 2024-09-21T11:08:00.144 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.144 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.144 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:08:00.144 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.453620+0000", 2024-09-21T11:08:00.144 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5559999999999996e-06 2024-09-21T11:08:00.144 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.144 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.144 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:08:00.144 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.453626+0000", 2024-09-21T11:08:00.145 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9940000000000003e-06 2024-09-21T11:08:00.145 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.145 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.145 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:08:00.145 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.453670+0000", 2024-09-21T11:08:00.145 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3665999999999998e-05 2024-09-21T11:08:00.145 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.145 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.145 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:08:00.145 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.453677+0000", 2024-09-21T11:08:00.145 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4230000000000002e-06 2024-09-21T11:08:00.145 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.145 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.145 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:08:00.146 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.453753+0000", 2024-09-21T11:08:00.146 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6380999999999999e-05 2024-09-21T11:08:00.146 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:08:00.146 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:08:00.146 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:08:00.146 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.146 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.146 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDPGPush(3.1f 1685/1684 [PushOp(3:ff5b34d6:::1.00000000:head, version: 71'2, data_included: [], data_size: 0, omap_header_size: 274, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:ff5b34d6:::1.00000000:head@71'2, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T11:08:00.146 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:07:50.455281+0000", 2024-09-21T11:08:00.146 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 9.6575811809999994, 2024-09-21T11:08:00.146 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00018932199999999999, 2024-09-21T11:08:00.146 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:08:00.146 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T11:08:00.147 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:08:00.147 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.147 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:08:00.147 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.455281+0000", 2024-09-21T11:08:00.147 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:08:00.147 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.147 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.147 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:08:00.147 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.455281+0000", 2024-09-21T11:08:00.147 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:08:00.147 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.147 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.147 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:08:00.148 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.455280+0000", 2024-09-21T11:08:00.148 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T11:08:00.148 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.148 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.148 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:08:00.148 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.455290+0000", 2024-09-21T11:08:00.148 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8220000000000002e-06 2024-09-21T11:08:00.148 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.148 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.148 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:08:00.148 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.455302+0000", 2024-09-21T11:08:00.148 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2397e-05 2024-09-21T11:08:00.148 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.148 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.149 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:08:00.149 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.455311+0000", 2024-09-21T11:08:00.149 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3930000000000008e-06 2024-09-21T11:08:00.149 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.149 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.149 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:08:00.149 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.455368+0000", 2024-09-21T11:08:00.149 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6730000000000001e-05 2024-09-21T11:08:00.149 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.149 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.149 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:08:00.149 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.455375+0000", 2024-09-21T11:08:00.149 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1069999999999999e-06 2024-09-21T11:08:00.150 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.150 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.150 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:08:00.150 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.455470+0000", 2024-09-21T11:08:00.150 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5531000000000004e-05 2024-09-21T11:08:00.150 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:08:00.150 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:08:00.150 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:08:00.150 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.150 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.150 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDPGPush(3.1f 1685/1684 [PushOp(3:fe07bba1:::100.00000000.inode:head, version: 71'3, data_included: [0~582], data_size: 582, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fe07bba1:::100.00000000.inode:head@71'3, size: 582, copy_subset: [0~582], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:582, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T11:08:00.150 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:07:50.456162+0000", 2024-09-21T11:08:00.151 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 9.6566998430000002, 2024-09-21T11:08:00.151 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000205529, 2024-09-21T11:08:00.151 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:08:00.151 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T11:08:00.151 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:08:00.151 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.151 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:08:00.151 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.456162+0000", 2024-09-21T11:08:00.151 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:08:00.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.151 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.151 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:08:00.151 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.456162+0000", 2024-09-21T11:08:00.152 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:08:00.152 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.152 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.152 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:08:00.152 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.456162+0000", 2024-09-21T11:08:00.152 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T11:08:00.152 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.152 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.152 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:08:00.152 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.456170+0000", 2024-09-21T11:08:00.152 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8100000000000004e-06 2024-09-21T11:08:00.152 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.152 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.152 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:08:00.152 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.456183+0000", 2024-09-21T11:08:00.153 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.217e-05 2024-09-21T11:08:00.153 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.153 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.153 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:08:00.153 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.456199+0000", 2024-09-21T11:08:00.153 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6056999999999999e-05 2024-09-21T11:08:00.153 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.153 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.153 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:08:00.153 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.456236+0000", 2024-09-21T11:08:00.153 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7333000000000003e-05 2024-09-21T11:08:00.153 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.153 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.153 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:08:00.153 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.456242+0000", 2024-09-21T11:08:00.154 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7860000000000002e-06 2024-09-21T11:08:00.154 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.154 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.154 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:08:00.154 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.456368+0000", 2024-09-21T11:08:00.154 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012594 2024-09-21T11:08:00.154 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:08:00.154 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:08:00.154 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:08:00.154 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.154 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.154 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDPGPush(3.1c 1685/1684 [PushOp(3:3ed09add:::607.00000000:head, version: 71'2, data_included: [], data_size: 0, omap_header_size: 274, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:3ed09add:::607.00000000:head@71'2, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T11:08:00.154 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:07:50.457418+0000", 2024-09-21T11:08:00.155 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 9.6554444089999993, 2024-09-21T11:08:00.155 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00021017099999999999, 2024-09-21T11:08:00.155 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:08:00.155 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T11:08:00.155 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:08:00.155 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.155 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:08:00.155 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.457418+0000", 2024-09-21T11:08:00.155 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:08:00.155 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.155 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.155 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:08:00.155 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.457418+0000", 2024-09-21T11:08:00.155 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:08:00.155 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.156 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.156 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:08:00.156 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.457417+0000", 2024-09-21T11:08:00.156 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T11:08:00.156 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.156 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.156 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:08:00.156 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.457427+0000", 2024-09-21T11:08:00.156 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8309999999999996e-06 2024-09-21T11:08:00.156 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.156 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.156 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:08:00.156 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.457440+0000", 2024-09-21T11:08:00.157 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2914e-05 2024-09-21T11:08:00.157 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.157 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.157 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:08:00.158 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.457450+0000", 2024-09-21T11:08:00.158 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8949999999999993e-06 2024-09-21T11:08:00.158 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.158 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.158 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:08:00.158 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.457506+0000", 2024-09-21T11:08:00.158 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6514999999999997e-05 2024-09-21T11:08:00.158 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.158 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.158 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:08:00.158 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.457518+0000", 2024-09-21T11:08:00.158 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1426e-05 2024-09-21T11:08:00.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.159 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.159 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:08:00.159 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.457628+0000", 2024-09-21T11:08:00.159 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000110246 2024-09-21T11:08:00.159 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:08:00.159 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:08:00.159 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:08:00.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.159 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.159 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDPGPush(3.1d 1685/1684 [PushOp(3:b871830b:::605.00000000:head, version: 71'2, data_included: [], data_size: 0, omap_header_size: 274, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:b871830b:::605.00000000:head@71'2, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T11:08:00.160 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:07:50.459103+0000", 2024-09-21T11:08:00.160 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 9.6537586100000006, 2024-09-21T11:08:00.160 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00021010899999999999, 2024-09-21T11:08:00.160 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:08:00.160 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T11:08:00.160 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:08:00.160 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.160 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:08:00.160 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.459103+0000", 2024-09-21T11:08:00.160 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:08:00.160 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.160 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.161 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:08:00.161 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.459103+0000", 2024-09-21T11:08:00.161 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:08:00.161 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.161 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.161 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:08:00.161 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.459103+0000", 2024-09-21T11:08:00.161 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T11:08:00.161 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.161 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.162 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:08:00.162 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.459112+0000", 2024-09-21T11:08:00.162 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1050000000000001e-06 2024-09-21T11:08:00.162 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.162 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.162 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:08:00.162 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.459125+0000", 2024-09-21T11:08:00.162 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2644e-05 2024-09-21T11:08:00.162 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.162 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.162 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:08:00.163 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.459133+0000", 2024-09-21T11:08:00.163 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1380000000000007e-06 2024-09-21T11:08:00.163 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.163 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.163 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:08:00.163 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.459189+0000", 2024-09-21T11:08:00.163 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6023000000000002e-05 2024-09-21T11:08:00.163 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.163 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.163 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:08:00.163 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.459200+0000", 2024-09-21T11:08:00.164 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0855e-05 2024-09-21T11:08:00.164 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.164 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.164 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:08:00.164 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.459314+0000", 2024-09-21T11:08:00.164 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011398100000000001 2024-09-21T11:08:00.164 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:08:00.164 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:08:00.164 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:08:00.164 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.165 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.165 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDPGPush(2.2 1685/1684 [PushOp(2:5cea7035:::rbd_info:head, version: 65'2, data_included: [0~19], data_size: 19, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:5cea7035:::rbd_info:head@65'2, size: 19, copy_subset: [0~19], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:19, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T11:08:00.165 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:07:50.459112+0000", 2024-09-21T11:08:00.165 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 9.6537501880000001, 2024-09-21T11:08:00.165 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000274128, 2024-09-21T11:08:00.165 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:08:00.165 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T11:08:00.165 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:08:00.165 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.165 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:08:00.165 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.459112+0000", 2024-09-21T11:08:00.165 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:08:00.165 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.165 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.166 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:08:00.166 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.459112+0000", 2024-09-21T11:08:00.166 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:08:00.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.166 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.166 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:08:00.166 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.459111+0000", 2024-09-21T11:08:00.166 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-21T11:08:00.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.166 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.166 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:08:00.166 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.459122+0000", 2024-09-21T11:08:00.166 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1314e-05 2024-09-21T11:08:00.167 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.167 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.167 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:08:00.167 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.459146+0000", 2024-09-21T11:08:00.167 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3595999999999999e-05 2024-09-21T11:08:00.167 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.167 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.167 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:08:00.167 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.459161+0000", 2024-09-21T11:08:00.167 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4781e-05 2024-09-21T11:08:00.167 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.167 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.168 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:08:00.168 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.459246+0000", 2024-09-21T11:08:00.168 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5369000000000002e-05 2024-09-21T11:08:00.168 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.168 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.168 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:08:00.168 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.459264+0000", 2024-09-21T11:08:00.168 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7847e-05 2024-09-21T11:08:00.168 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.168 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.168 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:08:00.168 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:50.459386+0000", 2024-09-21T11:08:00.168 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000122092 2024-09-21T11:08:00.168 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:08:00.168 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:08:00.169 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:08:00.169 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.169 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.169 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDPGPush(1.8 1686/1685 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'116, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'116, size: 262688, copy_subset: [0~262688], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:262688, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T11:08:00.169 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T11:07:51.583550+0000", 2024-09-21T11:08:00.169 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 8.5293124440000003, 2024-09-21T11:08:00.169 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00068683100000000005, 2024-09-21T11:08:00.169 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T11:08:00.169 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T11:08:00.169 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T11:08:00.169 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.169 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T11:08:00.169 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:51.583550+0000", 2024-09-21T11:08:00.169 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:08:00.170 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.170 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.170 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T11:08:00.170 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:51.583550+0000", 2024-09-21T11:08:00.170 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T11:08:00.170 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.170 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.170 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T11:08:00.170 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:51.583549+0000", 2024-09-21T11:08:00.170 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-21T11:08:00.170 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.171 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.171 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T11:08:00.171 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:51.583841+0000", 2024-09-21T11:08:00.171 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00029250599999999999 2024-09-21T11:08:00.171 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.171 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.171 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T11:08:00.171 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:51.583857+0000", 2024-09-21T11:08:00.171 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5546e-05 2024-09-21T11:08:00.171 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.171 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.171 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T11:08:00.171 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:51.583869+0000", 2024-09-21T11:08:00.172 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2082000000000001e-05 2024-09-21T11:08:00.172 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.172 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.172 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T11:08:00.172 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:51.583917+0000", 2024-09-21T11:08:00.172 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7970000000000003e-05 2024-09-21T11:08:00.172 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.172 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.172 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T11:08:00.172 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:51.583926+0000", 2024-09-21T11:08:00.172 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7499999999999992e-06 2024-09-21T11:08:00.172 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T11:08:00.172 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T11:08:00.172 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T11:08:00.172 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T11:07:51.584236+0000", 2024-09-21T11:08:00.173 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00031061400000000002 2024-09-21T11:08:00.173 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:08:00.173 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:08:00.173 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:08:00.173 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T11:08:00.173 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T11:08:00.173 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T11:08:00.605 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:00 smithi070 ceph-mon[112139]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T11:08:00.606 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:00 smithi070 ceph-mon[112139]: Cluster is now healthy 2024-09-21T11:08:00.606 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:00 smithi070 ceph-mon[107385]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T11:08:00.606 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:00 smithi070 ceph-mon[107385]: Cluster is now healthy 2024-09-21T11:08:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:00 smithi160 ceph-mon[101879]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T11:08:00.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:00 smithi160 ceph-mon[101879]: Cluster is now healthy 2024-09-21T11:08:01.529 INFO:teuthology.orchestra.run.smithi070.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-21T11:08:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:01 smithi160 ceph-mon[101879]: pgmap v3366: 93 pgs: 2 peering, 91 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 87 KiB/s, 1 keys/s, 1 objects/s recovering 2024-09-21T11:08:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:01 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1056371516' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T11:08:01.845 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:01 smithi070 ceph-mon[112139]: pgmap v3366: 93 pgs: 2 peering, 91 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 87 KiB/s, 1 keys/s, 1 objects/s recovering 2024-09-21T11:08:01.846 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:01 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1056371516' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T11:08:01.846 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:01 smithi070 ceph-mon[107385]: pgmap v3366: 93 pgs: 2 peering, 91 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 87 KiB/s, 1 keys/s, 1 objects/s recovering 2024-09-21T11:08:01.846 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:01 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1056371516' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T11:08:02.236 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T11:08:02.436 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:08:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:02 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1056371516' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-21T11:08:02.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:02 smithi160 ceph-mon[101879]: osdmap e1693: 8 total, 8 up, 8 in 2024-09-21T11:08:02.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:02 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1056371516' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-21T11:08:02.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:02 smithi070 ceph-mon[107385]: osdmap e1693: 8 total, 8 up, 8 in 2024-09-21T11:08:02.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:02 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1056371516' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-21T11:08:02.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:02 smithi070 ceph-mon[112139]: osdmap e1693: 8 total, 8 up, 8 in 2024-09-21T11:08:03.540 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:08:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:08:03] "GET /metrics HTTP/1.1" 200 34250 "" "Prometheus/2.51.0" 2024-09-21T11:08:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:03 smithi160 ceph-mon[101879]: pgmap v3368: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 5 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T11:08:03.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:03 smithi070 ceph-mon[112139]: pgmap v3368: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 5 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T11:08:03.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:03 smithi070 ceph-mon[107385]: pgmap v3368: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 5 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T11:08:04.544 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T11:08:04.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:04 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2931886017' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T11:08:04.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:04 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T11:08:04.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:04 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2931886017' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T11:08:04.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:04 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T11:08:04.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:04 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2931886017' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T11:08:04.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:04 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T11:08:05.330 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 1 1 2024-09-21T11:08:05.537 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:08:05.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:05 smithi070 ceph-mon[107385]: pgmap v3369: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:05.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:05 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T11:08:05.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:05 smithi070 ceph-mon[107385]: osdmap e1694: 8 total, 8 up, 8 in 2024-09-21T11:08:05.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:08:05.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:05 smithi070 ceph-mon[112139]: pgmap v3369: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:05.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:05 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T11:08:05.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:05 smithi070 ceph-mon[112139]: osdmap e1694: 8 total, 8 up, 8 in 2024-09-21T11:08:05.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:08:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:05 smithi160 ceph-mon[101879]: pgmap v3369: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:05 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T11:08:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:05 smithi160 ceph-mon[101879]: osdmap e1694: 8 total, 8 up, 8 in 2024-09-21T11:08:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:08:07.553 INFO:teuthology.orchestra.run.smithi070.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-21T11:08:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:07 smithi160 ceph-mon[101879]: pgmap v3371: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:07 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1582395505' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T11:08:07.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:07 smithi070 ceph-mon[112139]: pgmap v3371: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:07.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:07 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1582395505' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T11:08:07.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:07 smithi070 ceph-mon[107385]: pgmap v3371: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:07.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:07 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1582395505' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T11:08:08.290 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T11:08:08.488 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:08:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:08 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1582395505' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T11:08:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:08 smithi160 ceph-mon[101879]: osdmap e1695: 8 total, 8 up, 8 in 2024-09-21T11:08:08.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:08 smithi160 ceph-mon[101879]: osdmap e1696: 8 total, 8 up, 8 in 2024-09-21T11:08:08.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:08 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1582395505' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T11:08:08.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:08 smithi070 ceph-mon[107385]: osdmap e1695: 8 total, 8 up, 8 in 2024-09-21T11:08:08.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:08 smithi070 ceph-mon[107385]: osdmap e1696: 8 total, 8 up, 8 in 2024-09-21T11:08:08.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:08 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1582395505' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T11:08:08.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:08 smithi070 ceph-mon[112139]: osdmap e1695: 8 total, 8 up, 8 in 2024-09-21T11:08:08.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:08 smithi070 ceph-mon[112139]: osdmap e1696: 8 total, 8 up, 8 in 2024-09-21T11:08:09.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:09 smithi070 ceph-mon[107385]: pgmap v3373: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:09.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:09 smithi070 ceph-mon[112139]: pgmap v3373: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:09 smithi160 ceph-mon[101879]: pgmap v3373: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:11.575 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T11:08:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:11 smithi160 ceph-mon[101879]: pgmap v3375: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:11 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/644487370' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T11:08:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:11 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T11:08:11.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:11 smithi070 ceph-mon[107385]: pgmap v3375: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:11.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:11 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/644487370' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T11:08:11.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:11 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T11:08:11.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:11 smithi070 ceph-mon[112139]: pgmap v3375: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:11.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:11 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/644487370' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T11:08:11.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:11 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T11:08:12.257 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 2 1 2024-09-21T11:08:12.460 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:08:12.841 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:12 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T11:08:12.841 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:12 smithi070 ceph-mon[107385]: osdmap e1697: 8 total, 8 up, 8 in 2024-09-21T11:08:12.841 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:12 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T11:08:12.841 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:12 smithi070 ceph-mon[112139]: osdmap e1697: 8 total, 8 up, 8 in 2024-09-21T11:08:12.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:12 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T11:08:12.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:12 smithi160 ceph-mon[101879]: osdmap e1697: 8 total, 8 up, 8 in 2024-09-21T11:08:13.578 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:08:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:08:13] "GET /metrics HTTP/1.1" 200 34248 "" "Prometheus/2.51.0" 2024-09-21T11:08:13.829 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:13 smithi070 ceph-mon[107385]: pgmap v3377: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:13.830 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:13 smithi070 ceph-mon[112139]: pgmap v3377: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:13 smithi160 ceph-mon[101879]: pgmap v3377: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:14.587 INFO:teuthology.orchestra.run.smithi070.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-21T11:08:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:14 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/373062518' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T11:08:14.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:14 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T11:08:14.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:14 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/373062518' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T11:08:14.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:14 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T11:08:14.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:14 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/373062518' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T11:08:14.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:14 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T11:08:15.250 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T11:08:15.449 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:08:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:15 smithi160 ceph-mon[101879]: pgmap v3378: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:15 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-21T11:08:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:15 smithi160 ceph-mon[101879]: osdmap e1698: 8 total, 8 up, 8 in 2024-09-21T11:08:15.863 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:15 smithi070 ceph-mon[112139]: pgmap v3378: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:15.863 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:15 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-21T11:08:15.864 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:15 smithi070 ceph-mon[112139]: osdmap e1698: 8 total, 8 up, 8 in 2024-09-21T11:08:15.864 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:15 smithi070 ceph-mon[107385]: pgmap v3378: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:15.864 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:15 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-21T11:08:15.864 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:15 smithi070 ceph-mon[107385]: osdmap e1698: 8 total, 8 up, 8 in 2024-09-21T11:08:17.587 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T11:08:17.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:17 smithi070 ceph-mon[107385]: pgmap v3380: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:17.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:17 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3995015673' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T11:08:17.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:17 smithi070 ceph-mon[112139]: pgmap v3380: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:17.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:17 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3995015673' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T11:08:18.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:17 smithi160 ceph-mon[101879]: pgmap v3380: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:18.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:17 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3995015673' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T11:08:18.187 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 3 1 2024-09-21T11:08:18.382 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:08:18.776 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:18 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3995015673' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T11:08:18.776 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:18 smithi070 ceph-mon[107385]: osdmap e1699: 8 total, 8 up, 8 in 2024-09-21T11:08:18.777 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:18 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3995015673' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T11:08:18.777 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:18 smithi070 ceph-mon[112139]: osdmap e1699: 8 total, 8 up, 8 in 2024-09-21T11:08:19.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:18 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3995015673' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T11:08:19.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:18 smithi160 ceph-mon[101879]: osdmap e1699: 8 total, 8 up, 8 in 2024-09-21T11:08:19.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:19 smithi070 ceph-mon[112139]: pgmap v3382: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:19.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:19 smithi070 ceph-mon[107385]: pgmap v3382: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:20.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:19 smithi160 ceph-mon[101879]: pgmap v3382: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:20.603 INFO:teuthology.orchestra.run.smithi070.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-21T11:08:20.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:20 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3045526197' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T11:08:20.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:08:20.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:20 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3045526197' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T11:08:20.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:08:21.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:20 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3045526197' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T11:08:21.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:08:21.302 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 4 1 2024-09-21T11:08:21.496 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:08:21.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:21 smithi070 ceph-mon[107385]: pgmap v3383: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:21.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:21 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3045526197' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-21T11:08:21.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:21 smithi070 ceph-mon[107385]: osdmap e1700: 8 total, 8 up, 8 in 2024-09-21T11:08:21.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:21 smithi070 ceph-mon[112139]: pgmap v3383: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:21.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:21 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3045526197' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-21T11:08:21.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:21 smithi070 ceph-mon[112139]: osdmap e1700: 8 total, 8 up, 8 in 2024-09-21T11:08:22.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:21 smithi160 ceph-mon[101879]: pgmap v3383: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:22.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:21 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3045526197' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-21T11:08:22.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:21 smithi160 ceph-mon[101879]: osdmap e1700: 8 total, 8 up, 8 in 2024-09-21T11:08:23.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:08:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:08:23] "GET /metrics HTTP/1.1" 200 34247 "" "Prometheus/2.51.0" 2024-09-21T11:08:23.606 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T11:08:23.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:23 smithi070 ceph-mon[107385]: pgmap v3385: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:23.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:23 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1276091239' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T11:08:23.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:23 smithi070 ceph-mon[112139]: pgmap v3385: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:23.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:23 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1276091239' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T11:08:24.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:23 smithi160 ceph-mon[101879]: pgmap v3385: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:24.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:23 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1276091239' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T11:08:24.333 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 4 1 2024-09-21T11:08:24.539 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:08:24.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:24 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1276091239' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T11:08:24.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:24 smithi070 ceph-mon[107385]: osdmap e1701: 8 total, 8 up, 8 in 2024-09-21T11:08:24.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:24 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1276091239' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T11:08:24.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:24 smithi070 ceph-mon[112139]: osdmap e1701: 8 total, 8 up, 8 in 2024-09-21T11:08:25.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:24 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1276091239' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T11:08:25.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:24 smithi160 ceph-mon[101879]: osdmap e1701: 8 total, 8 up, 8 in 2024-09-21T11:08:25.825 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:25 smithi070 ceph-mon[107385]: pgmap v3387: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:25.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:25 smithi070 ceph-mon[112139]: pgmap v3387: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:25.993 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:25 smithi160 ceph-mon[101879]: pgmap v3387: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:26.625 INFO:teuthology.orchestra.run.smithi070.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-21T11:08:26.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:26 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3377757054' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T11:08:26.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:26 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T11:08:26.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:26 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3377757054' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T11:08:26.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:26 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T11:08:27.016 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:26 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3377757054' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T11:08:27.016 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:26 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T11:08:27.363 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T11:08:27.564 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:08:27.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:27 smithi070 ceph-mon[107385]: pgmap v3388: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:27.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:27 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-21T11:08:27.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:27 smithi070 ceph-mon[107385]: osdmap e1702: 8 total, 8 up, 8 in 2024-09-21T11:08:27.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:27 smithi070 ceph-mon[112139]: pgmap v3388: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:27.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:27 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-21T11:08:27.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:27 smithi070 ceph-mon[112139]: osdmap e1702: 8 total, 8 up, 8 in 2024-09-21T11:08:28.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:27 smithi160 ceph-mon[101879]: pgmap v3388: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:28.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:27 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-21T11:08:28.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:27 smithi160 ceph-mon[101879]: osdmap e1702: 8 total, 8 up, 8 in 2024-09-21T11:08:29.630 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T11:08:30.032 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:29 smithi070 ceph-mon[107385]: pgmap v3390: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:30.032 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:29 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1737849759' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T11:08:30.032 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:29 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T11:08:30.032 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:29 smithi070 ceph-mon[112139]: pgmap v3390: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:30.033 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:29 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1737849759' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T11:08:30.033 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:29 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T11:08:30.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:29 smithi160 ceph-mon[101879]: pgmap v3390: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:30.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:29 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1737849759' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T11:08:30.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:29 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T11:08:30.456 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 7 1 2024-09-21T11:08:30.654 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:08:30.774 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:30 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T11:08:30.774 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:30 smithi070 ceph-mon[107385]: osdmap e1703: 8 total, 8 up, 8 in 2024-09-21T11:08:30.775 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:30 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T11:08:30.775 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:30 smithi070 ceph-mon[112139]: osdmap e1703: 8 total, 8 up, 8 in 2024-09-21T11:08:31.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:30 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T11:08:31.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:30 smithi160 ceph-mon[101879]: osdmap e1703: 8 total, 8 up, 8 in 2024-09-21T11:08:31.836 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:31 smithi070 ceph-mon[107385]: pgmap v3392: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:31.836 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:31 smithi070 ceph-mon[112139]: pgmap v3392: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:32.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:31 smithi160 ceph-mon[101879]: pgmap v3392: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:32.649 INFO:teuthology.orchestra.run.smithi070.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-21T11:08:32.942 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:32 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3164719966' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T11:08:32.943 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:32 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T11:08:32.943 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:32 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3164719966' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T11:08:32.943 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:32 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T11:08:33.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:32 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3164719966' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T11:08:33.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:32 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T11:08:33.285 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:08:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:08:33] "GET /metrics HTTP/1.1" 200 34247 "" "Prometheus/2.51.0" 2024-09-21T11:08:33.307 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T11:08:33.503 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:08:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:33 smithi070 ceph-mon[112139]: pgmap v3393: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:33 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-21T11:08:33.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:33 smithi070 ceph-mon[112139]: osdmap e1704: 8 total, 8 up, 8 in 2024-09-21T11:08:33.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:33 smithi070 ceph-mon[107385]: pgmap v3393: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:33.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:33 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-21T11:08:33.878 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:33 smithi070 ceph-mon[107385]: osdmap e1704: 8 total, 8 up, 8 in 2024-09-21T11:08:34.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:33 smithi160 ceph-mon[101879]: pgmap v3393: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:34.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:33 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-21T11:08:34.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:33 smithi160 ceph-mon[101879]: osdmap e1704: 8 total, 8 up, 8 in 2024-09-21T11:08:35.653 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T11:08:35.978 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:35 smithi070 ceph-mon[107385]: pgmap v3395: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:35.979 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:08:35.979 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:35 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1402195488' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T11:08:35.979 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:35 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T11:08:35.979 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:35 smithi070 ceph-mon[112139]: pgmap v3395: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:35.980 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:08:35.980 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:35 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1402195488' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T11:08:35.980 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:35 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T11:08:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:35 smithi160 ceph-mon[101879]: pgmap v3395: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:36.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:08:36.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:35 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1402195488' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T11:08:36.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:35 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T11:08:36.312 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 6 1 2024-09-21T11:08:36.509 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:08:36.907 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:36 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T11:08:36.907 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:36 smithi070 ceph-mon[112139]: osdmap e1705: 8 total, 8 up, 8 in 2024-09-21T11:08:36.908 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:36 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T11:08:36.908 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:36 smithi070 ceph-mon[107385]: osdmap e1705: 8 total, 8 up, 8 in 2024-09-21T11:08:37.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:36 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T11:08:37.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:36 smithi160 ceph-mon[101879]: osdmap e1705: 8 total, 8 up, 8 in 2024-09-21T11:08:37.732 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:37 smithi160 ceph-mon[101879]: pgmap v3397: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:37.732 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:37 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T11:08:37.921 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:37 smithi070 ceph-mon[112139]: pgmap v3397: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:37.922 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:37 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T11:08:37.922 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:37 smithi070 ceph-mon[107385]: pgmap v3397: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:37.922 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:37 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T11:08:38.666 INFO:teuthology.orchestra.run.smithi070.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-21T11:08:38.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:38 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1830123399' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T11:08:38.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:38 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1830123399' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T11:08:39.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:38 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1830123399' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T11:08:39.867 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T11:08:39.948 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:39 smithi160 ceph-mon[101879]: pgmap v3398: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:39.948 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:39 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1830123399' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-21T11:08:39.948 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:39 smithi160 ceph-mon[101879]: osdmap e1706: 8 total, 8 up, 8 in 2024-09-21T11:08:40.086 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:08:40.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:39 smithi070 ceph-mon[107385]: pgmap v3398: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:40.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:39 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1830123399' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-21T11:08:40.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:39 smithi070 ceph-mon[107385]: osdmap e1706: 8 total, 8 up, 8 in 2024-09-21T11:08:40.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:39 smithi070 ceph-mon[112139]: pgmap v3398: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:40.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:39 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1830123399' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-21T11:08:40.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:39 smithi070 ceph-mon[112139]: osdmap e1706: 8 total, 8 up, 8 in 2024-09-21T11:08:41.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:41 smithi070 ceph-mon[107385]: pgmap v3400: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:41.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:41 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:08:41.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:41 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:08:41.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:41 smithi070 ceph-mon[112139]: pgmap v3400: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:41.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:41 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:08:41.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:41 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:08:42.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:41 smithi160 ceph-mon[101879]: pgmap v3400: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:42.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:41 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:08:42.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:41 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:08:42.675 INFO:teuthology.orchestra.run.smithi070.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T11:08:42.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:42 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3192698080' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T11:08:42.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:42 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3192698080' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T11:08:43.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:42 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3192698080' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T11:08:43.198 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:08:43 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:08:43] "GET /metrics HTTP/1.1" 200 34246 "" "Prometheus/2.51.0" 2024-09-21T11:08:43.605 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 5 1 2024-09-21T11:08:43.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:43 smithi070 ceph-mon[107385]: pgmap v3401: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:43.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:43 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3192698080' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T11:08:43.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:43 smithi070 ceph-mon[107385]: osdmap e1707: 8 total, 8 up, 8 in 2024-09-21T11:08:43.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:43 smithi070 ceph-mon[112139]: pgmap v3401: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:43.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:43 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3192698080' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T11:08:43.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:43 smithi070 ceph-mon[112139]: osdmap e1707: 8 total, 8 up, 8 in 2024-09-21T11:08:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:43 smithi160 ceph-mon[101879]: pgmap v3401: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:43 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3192698080' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T11:08:44.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:43 smithi160 ceph-mon[101879]: osdmap e1707: 8 total, 8 up, 8 in 2024-09-21T11:08:44.417 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:08:45.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:45 smithi070 ceph-mon[107385]: pgmap v3403: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:45.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:45 smithi070 ceph-mon[112139]: pgmap v3403: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:46.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:45 smithi160 ceph-mon[101879]: pgmap v3403: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:48.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:47 smithi070 ceph-mon[107385]: pgmap v3404: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:48.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:47 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:08:48.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:47 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:08:48.007 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:47 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T11:08:48.007 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:47 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T11:08:48.007 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:47 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:08:48.007 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:47 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3733175227' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T11:08:48.007 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:47 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T11:08:48.007 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:47 smithi070 ceph-mon[112139]: pgmap v3404: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:48.007 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:47 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:08:48.007 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:47 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:08:48.007 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:47 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T11:08:48.008 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:47 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T11:08:48.008 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:47 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:08:48.008 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:47 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3733175227' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T11:08:48.008 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:47 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T11:08:48.008 INFO:teuthology.orchestra.run.smithi070.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-21T11:08:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:47 smithi160 ceph-mon[101879]: pgmap v3404: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:47 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:08:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:47 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:08:48.091 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:47 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T11:08:48.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:47 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T11:08:48.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:47 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' 2024-09-21T11:08:48.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:47 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3733175227' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T11:08:48.092 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:47 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T11:08:48.743 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-21T11:08:48.743 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-21T11:08:48.743 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-21T11:08:48.743 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-21T11:08:48.743 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T11:08:48.743 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T11:08:48.939 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:08:49.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:49 smithi070 ceph-mon[107385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-21T11:08:49.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:49 smithi070 ceph-mon[107385]: osdmap e1708: 8 total, 8 up, 8 in 2024-09-21T11:08:49.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:49 smithi070 ceph-mon[107385]: pgmap v3406: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:49.299 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:49 smithi070 ceph-mon[112139]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-21T11:08:49.299 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:49 smithi070 ceph-mon[112139]: osdmap e1708: 8 total, 8 up, 8 in 2024-09-21T11:08:49.299 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:49 smithi070 ceph-mon[112139]: pgmap v3406: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:49.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:49 smithi160 ceph-mon[101879]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-21T11:08:49.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:49 smithi160 ceph-mon[101879]: osdmap e1708: 8 total, 8 up, 8 in 2024-09-21T11:08:49.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:49 smithi160 ceph-mon[101879]: pgmap v3406: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:50.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:50 smithi070 ceph-mon[107385]: osdmap e1709: 8 total, 8 up, 8 in 2024-09-21T11:08:50.127 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:50 smithi070 ceph-mon[112139]: osdmap e1709: 8 total, 8 up, 8 in 2024-09-21T11:08:50.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:50 smithi160 ceph-mon[101879]: osdmap e1709: 8 total, 8 up, 8 in 2024-09-21T11:08:50.835 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T11:08:50.836 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":1709,"fsid":"7c68bfbc-77fe-11ef-baf6-efdc52797490","created":"2024-09-21T09:47:59.216934+0000","modified":"2024-09-21T11:08:48.999846+0000","last_up_change":"2024-09-21T11:03:42.160423+0000","last_in_change":"2024-09-21T11:07:52.443629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":24,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":59,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:51:05.514472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":21,"pg_placement_num":21,"pg_placement_num_target":21,"pg_num_target":21,"pg_num_pending":21,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"1653","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"1573","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8999999761581421,"score_stable":1.8999999761581421,"optimal_score":1,"raw_score_acting":1.8999999761581421,"raw_score_stable":1.8999999761581421,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:53:41.830233+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-21T09:54:52.551642+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-21T09:54:52.861791+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6870ff8-14eb-4110-957a-00c73645b181","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":660,"up_from":666,"up_thru":1690,"down_at":660,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":452014366},{"type":"v1","addr":"172.21.15.70:6803","nonce":452014366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":452014366},{"type":"v1","addr":"172.21.15.70:6805","nonce":452014366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":452014366},{"type":"v1","addr":"172.21.15.70:6809","nonce":452014366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":452014366},{"type":"v1","addr":"172.21.15.70:6807","nonce":452014366}]},"public_addr":"172.21.15.70:6803/452014366","cluster_addr":"172.21.15.70:6805/452014366","heartbeat_back_addr":"172.21.15.70:6809/452014366","heartbeat_front_addr":"172.21.15.70:6807/452014366","state":["exists","up"]},{"osd":1,"uuid":"b1349e31-7493-489f-823a-55edb866d25f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":669,"up_from":674,"up_thru":1695,"down_at":669,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3938187175},{"type":"v1","addr":"172.21.15.70:6811","nonce":3938187175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3938187175},{"type":"v1","addr":"172.21.15.70:6813","nonce":3938187175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":3938187175},{"type":"v1","addr":"172.21.15.70:6817","nonce":3938187175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3938187175},{"type":"v1","addr":"172.21.15.70:6815","nonce":3938187175}]},"public_addr":"172.21.15.70:6811/3938187175","cluster_addr":"172.21.15.70:6813/3938187175","heartbeat_back_addr":"172.21.15.70:6817/3938187175","heartbeat_front_addr":"172.21.15.70:6815/3938187175","state":["exists","up"]},{"osd":2,"uuid":"ac4496ba-a860-4b09-bb38-7f433e081541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":677,"up_from":680,"up_thru":1690,"down_at":677,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":890146607},{"type":"v1","addr":"172.21.15.70:6819","nonce":890146607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":890146607},{"type":"v1","addr":"172.21.15.70:6821","nonce":890146607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":890146607},{"type":"v1","addr":"172.21.15.70:6825","nonce":890146607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":890146607},{"type":"v1","addr":"172.21.15.70:6823","nonce":890146607}]},"public_addr":"172.21.15.70:6819/890146607","cluster_addr":"172.21.15.70:6821/890146607","heartbeat_back_addr":"172.21.15.70:6825/890146607","heartbeat_front_addr":"172.21.15.70:6823/890146607","state":["exists","up"]},{"osd":3,"uuid":"1eedf4ce-057c-4576-ae5b-51d3595df195","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":689,"up_from":692,"up_thru":1689,"down_at":689,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3858266770},{"type":"v1","addr":"172.21.15.70:6827","nonce":3858266770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3858266770},{"type":"v1","addr":"172.21.15.70:6829","nonce":3858266770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3858266770},{"type":"v1","addr":"172.21.15.70:6833","nonce":3858266770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3858266770},{"type":"v1","addr":"172.21.15.70:6831","nonce":3858266770}]},"public_addr":"172.21.15.70:6827/3858266770","cluster_addr":"172.21.15.70:6829/3858266770","heartbeat_back_addr":"172.21.15.70:6833/3858266770","heartbeat_front_addr":"172.21.15.70:6831/3858266770","state":["exists","up"]},{"osd":4,"uuid":"a4b728b8-97a4-478f-8a11-ba590c0213e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":1690,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6801","nonce":3634505869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6803","nonce":3634505869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6807","nonce":3634505869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6805","nonce":3634505869}]},"public_addr":"172.21.15.160:6801/3634505869","cluster_addr":"172.21.15.160:6803/3634505869","heartbeat_back_addr":"172.21.15.160:6807/3634505869","heartbeat_front_addr":"172.21.15.160:6805/3634505869","state":["exists","up"]},{"osd":5,"uuid":"56915347-9254-4e94-b202-1f16d09e7946","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":38,"last_clean_end":1534,"up_from":1597,"up_thru":1708,"down_at":1536,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":1520073476},{"type":"v1","addr":"172.21.15.160:6817","nonce":1520073476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":1520073476},{"type":"v1","addr":"172.21.15.160:6819","nonce":1520073476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":1520073476},{"type":"v1","addr":"172.21.15.160:6823","nonce":1520073476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":1520073476},{"type":"v1","addr":"172.21.15.160:6821","nonce":1520073476}]},"public_addr":"172.21.15.160:6817/1520073476","cluster_addr":"172.21.15.160:6819/1520073476","heartbeat_back_addr":"172.21.15.160:6823/1520073476","heartbeat_front_addr":"172.21.15.160:6821/1520073476","state":["exists","up"]},{"osd":6,"uuid":"79e6de8c-b78b-40e2-9229-0275a873d40e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":46,"last_clean_end":1540,"up_from":1589,"up_thru":1687,"down_at":1541,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3336777196},{"type":"v1","addr":"172.21.15.160:6809","nonce":3336777196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3336777196},{"type":"v1","addr":"172.21.15.160:6811","nonce":3336777196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3336777196},{"type":"v1","addr":"172.21.15.160:6815","nonce":3336777196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3336777196},{"type":"v1","addr":"172.21.15.160:6813","nonce":3336777196}]},"public_addr":"172.21.15.160:6809/3336777196","cluster_addr":"172.21.15.160:6811/3336777196","heartbeat_back_addr":"172.21.15.160:6815/3336777196","heartbeat_front_addr":"172.21.15.160:6813/3336777196","state":["exists","up"]},{"osd":7,"uuid":"72245c8a-2fdb-4445-b378-16569f337fcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":1690,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6825","nonce":601673919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6827","nonce":601673919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6831","nonce":601673919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6829","nonce":601673919}]},"public_addr":"172.21.15.160:6825/601673919","cluster_addr":"172.21.15.160:6827/601673919","heartbeat_back_addr":"172.21.15.160:6831/601673919","heartbeat_front_addr":"172.21.15.160:6829/601673919","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:15.134515+0000","dead_epoch":660},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:39.191791+0000","dead_epoch":669},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:51:02.934581+0000","dead_epoch":676},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:51:26.514393+0000","dead_epoch":689},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:51:48.329249+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:12.268576+0000","dead_epoch":1534},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:36.370859+0000","dead_epoch":1540},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:59.888857+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.d","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/2422341096":"2024-09-22T10:00:49.897297+0000","172.21.15.160:0/1742426098":"2024-09-22T09:58:10.497359+0000","172.21.15.70:0/1905911777":"2024-09-22T09:53:31.134530+0000","172.21.15.160:6834/2915531836":"2024-09-22T09:58:10.497359+0000","172.21.15.70:0/4150778424":"2024-09-22T09:48:32.256720+0000","172.21.15.70:0/2308708361":"2024-09-22T10:00:49.897297+0000","172.21.15.70:6800/177993484":"2024-09-22T09:48:32.256720+0000","172.21.15.70:6800/1520336634":"2024-09-22T09:53:31.134530+0000","172.21.15.160:0/1014888319":"2024-09-22T09:58:10.497359+0000","172.21.15.70:6800/2142025682":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/1469701388":"2024-09-22T09:53:31.134530+0000","172.21.15.70:6801/603216094":"2024-09-22T10:00:49.897297+0000","172.21.15.160:0/19285887":"2024-09-22T09:58:10.497359+0000","172.21.15.70:6801/177993484":"2024-09-22T09:48:32.256720+0000","172.21.15.160:6835/2915531836":"2024-09-22T09:58:10.497359+0000","172.21.15.70:0/2521911561":"2024-09-22T09:48:32.256720+0000","172.21.15.70:0/2918782890":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/1012316198":"2024-09-22T09:48:32.256720+0000","172.21.15.70:0/1102418977":"2024-09-21T11:13:38.558014+0000","172.21.15.70:0/749465179":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/595094674":"2024-09-22T09:57:34.443100+0000","172.21.15.70:0/2386870039":"2024-09-22T09:57:34.443100+0000","172.21.15.70:0/3159615643":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/1066794139":"2024-09-22T09:53:31.134530+0000","172.21.15.70:0/2110605013":"2024-09-22T09:57:34.443100+0000","172.21.15.70:6801/1520336634":"2024-09-22T09:53:31.134530+0000","172.21.15.160:0/2751391809":"2024-09-22T09:58:10.497359+0000","172.21.15.160:0/3012862160":"2024-09-22T09:58:10.497359+0000","172.21.15.70:0/752126770":"2024-09-22T10:00:49.897297+0000","172.21.15.70:6801/2142025682":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/1635833554":"2024-09-22T10:00:49.897297+0000","172.21.15.70:6800/3639085022":"2024-09-22T09:57:34.443100+0000","172.21.15.70:0/2141342771":"2024-09-22T09:53:31.134530+0000","172.21.15.70:6801/3639085022":"2024-09-22T09:57:34.443100+0000","172.21.15.70:0/47418078":"2024-09-22T09:57:34.443100+0000","172.21.15.160:0/2880106921":"2024-09-22T09:58:10.497359+0000","172.21.15.70:6800/603216094":"2024-09-22T10:00:49.897297+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":1}]},{"pool":6,"snaps":[{"begin":2,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":1}]},{"pool":8,"snaps":[{"begin":2,"length":1}]},{"pool":9,"snaps":[{"begin":2,"length":1}]},{"pool":10,"snaps":[{"begin":2,"length":1}]},{"pool":11,"snaps":[{"begin":2,"length":1}]},{"pool":14,"snaps":[{"begin":7,"length":2}]},{"pool":16,"snaps":[{"begin":35,"length":3},{"begin":45,"length":3}]},{"pool":17,"snaps":[{"begin":22,"length":1}]},{"pool":18,"snaps":[{"begin":2,"length":1}]},{"pool":52,"snaps":[{"begin":104,"length":2}]},{"pool":59,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T11:08:51.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:51 smithi070 ceph-mon[107385]: pgmap v3408: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:51.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:51 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:08:51.127 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:51 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2040693271' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T11:08:51.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:51 smithi070 ceph-mon[112139]: pgmap v3408: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:51.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:51 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:08:51.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:51 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2040693271' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T11:08:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:51 smithi160 ceph-mon[101879]: pgmap v3408: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:51 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:08:51.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:51 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2040693271' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T11:08:51.424 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T11:08:51.424 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T11:08:51.619 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:08:53.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:08:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:08:53] "GET /metrics HTTP/1.1" 200 34248 "" "Prometheus/2.51.0" 2024-09-21T11:08:53.525 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T11:08:53.525 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":1709,"fsid":"7c68bfbc-77fe-11ef-baf6-efdc52797490","created":"2024-09-21T09:47:59.216934+0000","modified":"2024-09-21T11:08:48.999846+0000","last_up_change":"2024-09-21T11:03:42.160423+0000","last_in_change":"2024-09-21T11:07:52.443629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":24,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":59,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:51:05.514472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":21,"pg_placement_num":21,"pg_placement_num_target":21,"pg_num_target":21,"pg_num_pending":21,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"1653","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"1573","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8999999761581421,"score_stable":1.8999999761581421,"optimal_score":1,"raw_score_acting":1.8999999761581421,"raw_score_stable":1.8999999761581421,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:53:41.830233+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-21T09:54:52.551642+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-21T09:54:52.861791+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6870ff8-14eb-4110-957a-00c73645b181","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":660,"up_from":666,"up_thru":1690,"down_at":660,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":452014366},{"type":"v1","addr":"172.21.15.70:6803","nonce":452014366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":452014366},{"type":"v1","addr":"172.21.15.70:6805","nonce":452014366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":452014366},{"type":"v1","addr":"172.21.15.70:6809","nonce":452014366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":452014366},{"type":"v1","addr":"172.21.15.70:6807","nonce":452014366}]},"public_addr":"172.21.15.70:6803/452014366","cluster_addr":"172.21.15.70:6805/452014366","heartbeat_back_addr":"172.21.15.70:6809/452014366","heartbeat_front_addr":"172.21.15.70:6807/452014366","state":["exists","up"]},{"osd":1,"uuid":"b1349e31-7493-489f-823a-55edb866d25f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":669,"up_from":674,"up_thru":1695,"down_at":669,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3938187175},{"type":"v1","addr":"172.21.15.70:6811","nonce":3938187175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3938187175},{"type":"v1","addr":"172.21.15.70:6813","nonce":3938187175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":3938187175},{"type":"v1","addr":"172.21.15.70:6817","nonce":3938187175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3938187175},{"type":"v1","addr":"172.21.15.70:6815","nonce":3938187175}]},"public_addr":"172.21.15.70:6811/3938187175","cluster_addr":"172.21.15.70:6813/3938187175","heartbeat_back_addr":"172.21.15.70:6817/3938187175","heartbeat_front_addr":"172.21.15.70:6815/3938187175","state":["exists","up"]},{"osd":2,"uuid":"ac4496ba-a860-4b09-bb38-7f433e081541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":677,"up_from":680,"up_thru":1690,"down_at":677,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":890146607},{"type":"v1","addr":"172.21.15.70:6819","nonce":890146607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":890146607},{"type":"v1","addr":"172.21.15.70:6821","nonce":890146607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":890146607},{"type":"v1","addr":"172.21.15.70:6825","nonce":890146607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":890146607},{"type":"v1","addr":"172.21.15.70:6823","nonce":890146607}]},"public_addr":"172.21.15.70:6819/890146607","cluster_addr":"172.21.15.70:6821/890146607","heartbeat_back_addr":"172.21.15.70:6825/890146607","heartbeat_front_addr":"172.21.15.70:6823/890146607","state":["exists","up"]},{"osd":3,"uuid":"1eedf4ce-057c-4576-ae5b-51d3595df195","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":689,"up_from":692,"up_thru":1689,"down_at":689,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3858266770},{"type":"v1","addr":"172.21.15.70:6827","nonce":3858266770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3858266770},{"type":"v1","addr":"172.21.15.70:6829","nonce":3858266770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3858266770},{"type":"v1","addr":"172.21.15.70:6833","nonce":3858266770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3858266770},{"type":"v1","addr":"172.21.15.70:6831","nonce":3858266770}]},"public_addr":"172.21.15.70:6827/3858266770","cluster_addr":"172.21.15.70:6829/3858266770","heartbeat_back_addr":"172.21.15.70:6833/3858266770","heartbeat_front_addr":"172.21.15.70:6831/3858266770","state":["exists","up"]},{"osd":4,"uuid":"a4b728b8-97a4-478f-8a11-ba590c0213e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":1690,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6801","nonce":3634505869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6803","nonce":3634505869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6807","nonce":3634505869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":3634505869},{"type":"v1","addr":"172.21.15.160:6805","nonce":3634505869}]},"public_addr":"172.21.15.160:6801/3634505869","cluster_addr":"172.21.15.160:6803/3634505869","heartbeat_back_addr":"172.21.15.160:6807/3634505869","heartbeat_front_addr":"172.21.15.160:6805/3634505869","state":["exists","up"]},{"osd":5,"uuid":"56915347-9254-4e94-b202-1f16d09e7946","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":38,"last_clean_end":1534,"up_from":1597,"up_thru":1708,"down_at":1536,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":1520073476},{"type":"v1","addr":"172.21.15.160:6817","nonce":1520073476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":1520073476},{"type":"v1","addr":"172.21.15.160:6819","nonce":1520073476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":1520073476},{"type":"v1","addr":"172.21.15.160:6823","nonce":1520073476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":1520073476},{"type":"v1","addr":"172.21.15.160:6821","nonce":1520073476}]},"public_addr":"172.21.15.160:6817/1520073476","cluster_addr":"172.21.15.160:6819/1520073476","heartbeat_back_addr":"172.21.15.160:6823/1520073476","heartbeat_front_addr":"172.21.15.160:6821/1520073476","state":["exists","up"]},{"osd":6,"uuid":"79e6de8c-b78b-40e2-9229-0275a873d40e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":46,"last_clean_end":1540,"up_from":1589,"up_thru":1687,"down_at":1541,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3336777196},{"type":"v1","addr":"172.21.15.160:6809","nonce":3336777196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3336777196},{"type":"v1","addr":"172.21.15.160:6811","nonce":3336777196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3336777196},{"type":"v1","addr":"172.21.15.160:6815","nonce":3336777196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3336777196},{"type":"v1","addr":"172.21.15.160:6813","nonce":3336777196}]},"public_addr":"172.21.15.160:6809/3336777196","cluster_addr":"172.21.15.160:6811/3336777196","heartbeat_back_addr":"172.21.15.160:6815/3336777196","heartbeat_front_addr":"172.21.15.160:6813/3336777196","state":["exists","up"]},{"osd":7,"uuid":"72245c8a-2fdb-4445-b378-16569f337fcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":1690,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6825","nonce":601673919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6827","nonce":601673919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6831","nonce":601673919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":601673919},{"type":"v1","addr":"172.21.15.160:6829","nonce":601673919}]},"public_addr":"172.21.15.160:6825/601673919","cluster_addr":"172.21.15.160:6827/601673919","heartbeat_back_addr":"172.21.15.160:6831/601673919","heartbeat_front_addr":"172.21.15.160:6829/601673919","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:15.134515+0000","dead_epoch":660},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:39.191791+0000","dead_epoch":669},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:51:02.934581+0000","dead_epoch":676},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:51:26.514393+0000","dead_epoch":689},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:51:48.329249+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:12.268576+0000","dead_epoch":1534},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:36.370859+0000","dead_epoch":1540},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:59.888857+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.d","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/2422341096":"2024-09-22T10:00:49.897297+0000","172.21.15.70:0/1635833554":"2024-09-22T10:00:49.897297+0000","172.21.15.70:0/2308708361":"2024-09-22T10:00:49.897297+0000","172.21.15.70:0/1102418977":"2024-09-21T11:13:38.558014+0000","172.21.15.70:0/749465179":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/4150778424":"2024-09-22T09:48:32.256720+0000","172.21.15.160:6834/2915531836":"2024-09-22T09:58:10.497359+0000","172.21.15.70:0/2918782890":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/1012316198":"2024-09-22T09:48:32.256720+0000","172.21.15.70:6801/603216094":"2024-09-22T10:00:49.897297+0000","172.21.15.160:0/19285887":"2024-09-22T09:58:10.497359+0000","172.21.15.70:0/1469701388":"2024-09-22T09:53:31.134530+0000","172.21.15.70:6800/177993484":"2024-09-22T09:48:32.256720+0000","172.21.15.70:6800/1520336634":"2024-09-22T09:53:31.134530+0000","172.21.15.160:0/1014888319":"2024-09-22T09:58:10.497359+0000","172.21.15.70:6800/2142025682":"2024-09-22T09:48:55.219103+0000","172.21.15.70:6801/177993484":"2024-09-22T09:48:32.256720+0000","172.21.15.160:6835/2915531836":"2024-09-22T09:58:10.497359+0000","172.21.15.70:0/2521911561":"2024-09-22T09:48:32.256720+0000","172.21.15.70:0/595094674":"2024-09-22T09:57:34.443100+0000","172.21.15.70:0/2386870039":"2024-09-22T09:57:34.443100+0000","172.21.15.70:0/3159615643":"2024-09-22T09:48:55.219103+0000","172.21.15.70:0/1066794139":"2024-09-22T09:53:31.134530+0000","172.21.15.70:0/2110605013":"2024-09-22T09:57:34.443100+0000","172.21.15.70:6801/1520336634":"2024-09-22T09:53:31.134530+0000","172.21.15.160:0/2751391809":"2024-09-22T09:58:10.497359+0000","172.21.15.160:0/3012862160":"2024-09-22T09:58:10.497359+0000","172.21.15.70:0/752126770":"2024-09-22T10:00:49.897297+0000","172.21.15.70:6801/2142025682":"2024-09-22T09:48:55.219103+0000","172.21.15.70:6800/3639085022":"2024-09-22T09:57:34.443100+0000","172.21.15.70:0/2141342771":"2024-09-22T09:53:31.134530+0000","172.21.15.70:6801/3639085022":"2024-09-22T09:57:34.443100+0000","172.21.15.70:6800/603216094":"2024-09-22T10:00:49.897297+0000","172.21.15.160:0/2880106921":"2024-09-22T09:58:10.497359+0000","172.21.15.70:0/47418078":"2024-09-22T09:57:34.443100+0000","172.21.15.160:0/1742426098":"2024-09-22T09:58:10.497359+0000","172.21.15.70:0/1905911777":"2024-09-22T09:53:31.134530+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":1}]},{"pool":6,"snaps":[{"begin":2,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":1}]},{"pool":8,"snaps":[{"begin":2,"length":1}]},{"pool":9,"snaps":[{"begin":2,"length":1}]},{"pool":10,"snaps":[{"begin":2,"length":1}]},{"pool":11,"snaps":[{"begin":2,"length":1}]},{"pool":14,"snaps":[{"begin":7,"length":2}]},{"pool":16,"snaps":[{"begin":35,"length":3},{"begin":45,"length":3}]},{"pool":17,"snaps":[{"begin":22,"length":1}]},{"pool":18,"snaps":[{"begin":2,"length":1}]},{"pool":52,"snaps":[{"begin":104,"length":2}]},{"pool":59,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T11:08:53.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:53 smithi160 ceph-mon[101879]: pgmap v3409: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:53.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:53 smithi070 ceph-mon[107385]: pgmap v3409: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:53.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:53 smithi070 ceph-mon[112139]: pgmap v3409: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:54.280 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T11:08:54.280 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T11:08:54.281 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T11:08:54.281 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T11:08:54.281 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T11:08:54.282 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T11:08:54.282 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T11:08:54.283 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T11:08:54.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:54 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1194033249' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T11:08:54.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:54 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1194033249' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T11:08:54.676 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:08:54.678 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:08:54.694 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:08:54.704 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:08:54.708 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:08:54.708 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:08:54.720 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:08:54.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:54 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1194033249' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T11:08:55.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:55 smithi070 ceph-mon[107385]: pgmap v3410: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:55.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:55 smithi070 ceph-mon[112139]: pgmap v3410: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:55.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:55 smithi160 ceph-mon[101879]: pgmap v3410: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:56.334 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:08:56.486 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:56 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-21T11:08:56.487 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:56 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]} : dispatch 2024-09-21T11:08:56.487 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:56 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-21T11:08:56.487 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:56 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]} : dispatch 2024-09-21T11:08:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:56 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-21T11:08:56.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:56 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]} : dispatch 2024-09-21T11:08:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:57 smithi160 ceph-mon[101879]: pgmap v3411: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:57 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-21T11:08:57.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:57 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-09-21T11:08:57.842 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:57 smithi160 ceph-mon[101879]: osdmap e1710: 8 total, 8 up, 8 in 2024-09-21T11:08:57.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:57 smithi070 ceph-mon[107385]: pgmap v3411: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:57.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:57 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-21T11:08:57.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:57 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-09-21T11:08:57.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:57 smithi070 ceph-mon[107385]: osdmap e1710: 8 total, 8 up, 8 in 2024-09-21T11:08:57.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:57 smithi070 ceph-mon[112139]: pgmap v3411: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:57.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:57 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-21T11:08:57.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:57 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-09-21T11:08:57.878 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:57 smithi070 ceph-mon[112139]: osdmap e1710: 8 total, 8 up, 8 in 2024-09-21T11:08:58.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:58 smithi070 ceph-mon[107385]: osdmap e1711: 8 total, 8 up, 8 in 2024-09-21T11:08:58.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:58 smithi070 ceph-mon[112139]: osdmap e1711: 8 total, 8 up, 8 in 2024-09-21T11:08:58.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:58 smithi160 ceph-mon[101879]: osdmap e1711: 8 total, 8 up, 8 in 2024-09-21T11:08:59.816 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:08:59 smithi070 ceph-mon[107385]: pgmap v3414: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:59.817 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:08:59 smithi070 ceph-mon[112139]: pgmap v3414: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:08:59.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:08:59 smithi160 ceph-mon[101879]: pgmap v3414: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:01.682 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:01 smithi070 ceph-mon[112139]: pgmap v3415: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:01.682 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:01 smithi070 ceph-mon[107385]: pgmap v3415: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:01.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:01 smithi160 ceph-mon[101879]: pgmap v3415: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:03.358 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:09:03 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:09:03] "GET /metrics HTTP/1.1" 200 34248 "" "Prometheus/2.51.0" 2024-09-21T11:09:03.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:03 smithi070 ceph-mon[107385]: pgmap v3416: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:03.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:03 smithi070 ceph-mon[112139]: pgmap v3416: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:03.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:03 smithi160 ceph-mon[101879]: pgmap v3416: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:04.600 INFO:teuthology.orchestra.run.smithi070.stdout:231928234900 2024-09-21T11:09:04.600 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T11:09:05.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:05 smithi070 ceph-mon[107385]: pgmap v3417: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:05.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:05 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:09:05.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:05 smithi070 ceph-mon[112139]: pgmap v3417: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:05.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:05 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:09:05.759 INFO:teuthology.orchestra.run.smithi070.stdout:2860448219880 2024-09-21T11:09:05.760 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T11:09:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:05 smithi160 ceph-mon[101879]: pgmap v3417: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:05.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:05 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:09:07.709 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:07 smithi070 ceph-mon[107385]: pgmap v3418: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:07.709 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:07 smithi070 ceph-mon[112139]: pgmap v3418: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:07.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:07 smithi160 ceph-mon[101879]: pgmap v3418: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:08.249 INFO:teuthology.orchestra.run.smithi070.stdout:6824703033424 2024-09-21T11:09:08.250 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T11:09:09.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:09 smithi160 ceph-mon[101879]: pgmap v3419: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:09.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:09 smithi070 ceph-mon[107385]: pgmap v3419: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:09.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:09 smithi070 ceph-mon[112139]: pgmap v3419: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:10.322 INFO:teuthology.orchestra.run.smithi070.stdout:2972117369554 2024-09-21T11:09:10.323 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T11:09:11.016 INFO:teuthology.orchestra.run.smithi070.stdout:128849019811 2024-09-21T11:09:11.016 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T11:09:11.393 INFO:teuthology.orchestra.run.smithi070.stdout:6859062771779 2024-09-21T11:09:11.394 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T11:09:11.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:11 smithi070 ceph-mon[112139]: pgmap v3420: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:11.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:11 smithi070 ceph-mon[107385]: pgmap v3420: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:11.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:11 smithi160 ceph-mon[101879]: pgmap v3420: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:12.314 INFO:teuthology.orchestra.run.smithi070.stdout:2894807958242 2024-09-21T11:09:12.315 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T11:09:12.843 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:09:12.850 INFO:teuthology.orchestra.run.smithi070.stdout:2920577762011 2024-09-21T11:09:12.851 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T11:09:12.872 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:09:12.878 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:09:12.880 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:09:12.880 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:09:12.890 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:09:13.349 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:09:13.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:09:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:09:13] "GET /metrics HTTP/1.1" 200 34251 "" "Prometheus/2.51.0" 2024-09-21T11:09:13.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:13 smithi160 ceph-mon[101879]: pgmap v3421: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:13.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:13 smithi070 ceph-mon[107385]: pgmap v3421: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:13.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:13 smithi070 ceph-mon[112139]: pgmap v3421: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:15.644 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:15 smithi070 ceph-mon[107385]: pgmap v3422: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:15.645 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:15 smithi070 ceph-mon[112139]: pgmap v3422: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:15.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:15 smithi160 ceph-mon[101879]: pgmap v3422: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:16.226 INFO:teuthology.orchestra.run.smithi070.stdout:6859062771780 2024-09-21T11:09:16.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:16 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2088028612' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T11:09:16.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:16 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2088028612' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T11:09:16.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:16 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2088028612' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T11:09:17.804 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:17 smithi070 ceph-mon[107385]: pgmap v3423: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:17.804 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:17 smithi070 ceph-mon[112139]: pgmap v3423: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:17.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:17 smithi160 ceph-mon[101879]: pgmap v3423: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:19.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:19 smithi070 ceph-mon[107385]: pgmap v3424: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:19.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:19 smithi070 ceph-mon[112139]: pgmap v3424: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:19.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:19 smithi160 ceph-mon[101879]: pgmap v3424: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:20.019 INFO:teuthology.orchestra.run.smithi070.stdout:2894807958243 2024-09-21T11:09:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:20 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2341535807' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T11:09:20.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:20 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:09:20.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:20 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2341535807' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T11:09:20.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:20 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:09:20.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:20 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2341535807' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T11:09:20.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:20 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:09:21.826 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:21 smithi070 ceph-mon[107385]: pgmap v3425: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:21.827 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:21 smithi070 ceph-mon[112139]: pgmap v3425: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:21.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:21 smithi160 ceph-mon[101879]: pgmap v3425: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:22.063 INFO:tasks.cephadm.ceph_manager.ceph:need seq 6859062771779 got 6859062771780 for osd.5 2024-09-21T11:09:22.063 DEBUG:teuthology.parallel:result is None 2024-09-21T11:09:22.384 INFO:teuthology.orchestra.run.smithi070.stdout:6824703033426 2024-09-21T11:09:22.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:22 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/2067523026' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T11:09:22.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:22 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/2067523026' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T11:09:22.877 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:22 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/2067523026' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T11:09:23.114 INFO:tasks.cephadm.ceph_manager.ceph:need seq 6824703033424 got 6824703033426 for osd.6 2024-09-21T11:09:23.114 DEBUG:teuthology.parallel:result is None 2024-09-21T11:09:23.377 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:09:23 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:09:23] "GET /metrics HTTP/1.1" 200 34248 "" "Prometheus/2.51.0" 2024-09-21T11:09:23.471 INFO:teuthology.orchestra.run.smithi070.stdout:2972117369558 2024-09-21T11:09:23.665 INFO:teuthology.orchestra.run.smithi070.stdout:231928234905 2024-09-21T11:09:23.666 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:23 smithi070 ceph-mon[107385]: pgmap v3426: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:23.666 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:23 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3398230610' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T11:09:23.666 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:23 smithi070 ceph-mon[112139]: pgmap v3426: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:23.666 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:23 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3398230610' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T11:09:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:23 smithi160 ceph-mon[101879]: pgmap v3426: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:23.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:23 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3398230610' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T11:09:24.413 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234900 got 231928234905 for osd.7 2024-09-21T11:09:24.413 DEBUG:teuthology.parallel:result is None 2024-09-21T11:09:24.664 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:24 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1293358143' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T11:09:24.664 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:24 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1293358143' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T11:09:24.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:24 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1293358143' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T11:09:25.305 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2894807958242 got 2894807958243 for osd.1 2024-09-21T11:09:25.305 DEBUG:teuthology.parallel:result is None 2024-09-21T11:09:25.392 INFO:teuthology.orchestra.run.smithi070.stdout:128849019814 2024-09-21T11:09:25.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:25 smithi070 ceph-mon[107385]: pgmap v3427: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:25.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:25 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/1072521568' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T11:09:25.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:25 smithi070 ceph-mon[112139]: pgmap v3427: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:25.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:25 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/1072521568' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T11:09:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:25 smithi160 ceph-mon[101879]: pgmap v3427: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:25.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:25 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/1072521568' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T11:09:25.994 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849019811 got 128849019814 for osd.4 2024-09-21T11:09:25.994 DEBUG:teuthology.parallel:result is None 2024-09-21T11:09:26.481 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2972117369554 got 2972117369558 for osd.3 2024-09-21T11:09:26.481 DEBUG:teuthology.parallel:result is None 2024-09-21T11:09:26.892 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:09:27.556 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:27 smithi070 ceph-mon[107385]: pgmap v3428: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:27.557 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:27 smithi070 ceph-mon[112139]: pgmap v3428: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:27.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:27 smithi160 ceph-mon[101879]: pgmap v3428: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:28.445 INFO:teuthology.orchestra.run.smithi070.stdout:2860448219884 2024-09-21T11:09:28.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:28 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3100304379' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T11:09:28.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:28 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3100304379' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T11:09:28.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:28 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3100304379' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T11:09:29.091 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2860448219880 got 2860448219884 for osd.0 2024-09-21T11:09:29.091 DEBUG:teuthology.parallel:result is None 2024-09-21T11:09:29.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:29 smithi070 ceph-mon[112139]: pgmap v3429: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:29.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:29 smithi070 ceph-mon[107385]: pgmap v3429: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:29.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:29 smithi160 ceph-mon[101879]: pgmap v3429: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:30.201 INFO:teuthology.orchestra.run.smithi070.stdout:2920577762014 2024-09-21T11:09:30.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:30 smithi160 ceph-mon[101879]: from='client.? 172.21.15.70:0/3225924824' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T11:09:30.874 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:30 smithi070 ceph-mon[107385]: from='client.? 172.21.15.70:0/3225924824' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T11:09:30.874 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:30 smithi070 ceph-mon[112139]: from='client.? 172.21.15.70:0/3225924824' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T11:09:30.897 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2920577762011 got 2920577762014 for osd.2 2024-09-21T11:09:30.897 DEBUG:teuthology.parallel:result is None 2024-09-21T11:09:30.897 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T11:09:30.897 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T11:09:31.095 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:09:31.836 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:31 smithi070 ceph-mon[112139]: pgmap v3430: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:31.837 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:31 smithi070 ceph-mon[107385]: pgmap v3430: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:31.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:31 smithi160 ceph-mon[101879]: pgmap v3430: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:33.180 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T11:09:33.181 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T11:09:33.488 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:09:33 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: ::ffff:172.21.15.160 - - [21/Sep/2024:11:09:33] "GET /metrics HTTP/1.1" 200 34248 "" "Prometheus/2.51.0" 2024-09-21T11:09:33.772 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:33 smithi070 ceph-mon[107385]: pgmap v3431: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:33.772 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:33 smithi070 ceph-mon[112139]: pgmap v3431: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:33.793 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":3431,"stamp":"2024-09-21T11:09:32.397005+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595941,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":347449,"num_keys_recovered":50,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3188,"ondisk_log_size":3188,"up":279,"acting":279,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":279,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6428108,"kb_used_data":86104,"kb_used_omap":575,"kb_used_meta":6341312,"kb_avail":743533108,"statfs":{"total":767960285184,"available":761377902592,"internally_reserved":0,"allocated":88170496,"data_stored":49760943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":589295,"internal_metadata":6493504017},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003151"},"pg_stats":[{"pgid":"3.1f","version":"71'4","reported_seq":193,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.581709+0000","last_change":"2024-09-21T11:07:53.469858+0000","last_active":"2024-09-21T11:08:57.581709+0000","last_peered":"2024-09-21T11:08:57.581709+0000","last_clean":"2024-09-21T11:08:57.581709+0000","last_became_active":"2024-09-21T11:07:53.464219+0000","last_became_peered":"2024-09-21T11:07:53.464219+0000","last_unstale":"2024-09-21T11:08:57.581709+0000","last_undegraded":"2024-09-21T11:08:57.581709+0000","last_fullsized":"2024-09-21T11:08:57.581709+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:50:02.669630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4074,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1630,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.457558+0000","last_change":"2024-09-21T11:02:39.486670+0000","last_active":"2024-09-21T11:08:56.457558+0000","last_peered":"2024-09-21T11:08:56.457558+0000","last_clean":"2024-09-21T11:08:56.457558+0000","last_became_active":"2024-09-21T11:02:39.486248+0000","last_became_peered":"2024-09-21T11:02:39.486248+0000","last_unstale":"2024-09-21T11:08:56.457558+0000","last_undegraded":"2024-09-21T11:08:56.457558+0000","last_fullsized":"2024-09-21T11:08:56.457558+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:56:56.360708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":211,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.465386+0000","last_change":"2024-09-21T11:07:53.472545+0000","last_active":"2024-09-21T11:08:57.465386+0000","last_peered":"2024-09-21T11:08:57.465386+0000","last_clean":"2024-09-21T11:08:57.465386+0000","last_became_active":"2024-09-21T11:07:53.472156+0000","last_became_peered":"2024-09-21T11:07:53.472156+0000","last_unstale":"2024-09-21T11:08:57.465386+0000","last_undegraded":"2024-09-21T11:08:57.465386+0000","last_fullsized":"2024-09-21T11:08:57.465386+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:49:41.030319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":224,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.464923+0000","last_change":"2024-09-21T11:07:53.474533+0000","last_active":"2024-09-21T11:08:57.464923+0000","last_peered":"2024-09-21T11:08:57.464923+0000","last_clean":"2024-09-21T11:08:57.464923+0000","last_became_active":"2024-09-21T11:07:53.474107+0000","last_became_peered":"2024-09-21T11:07:53.474107+0000","last_unstale":"2024-09-21T11:08:57.464923+0000","last_undegraded":"2024-09-21T11:08:57.464923+0000","last_fullsized":"2024-09-21T11:08:57.464923+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:47:08.181090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":195,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.466058+0000","last_change":"2024-09-21T11:08:49.019024+0000","last_active":"2024-09-21T11:08:56.466058+0000","last_peered":"2024-09-21T11:08:56.466058+0000","last_clean":"2024-09-21T11:08:56.466058+0000","last_became_active":"2024-09-21T11:08:49.018435+0000","last_became_peered":"2024-09-21T11:08:49.018435+0000","last_unstale":"2024-09-21T11:08:56.466058+0000","last_undegraded":"2024-09-21T11:08:56.466058+0000","last_fullsized":"2024-09-21T11:08:56.466058+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:11:55.300993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":43,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.458100+0000","last_change":"2024-09-21T11:07:02.780603+0000","last_active":"2024-09-21T11:08:56.458100+0000","last_peered":"2024-09-21T11:08:56.458100+0000","last_clean":"2024-09-21T11:08:56.458100+0000","last_became_active":"2024-09-21T11:07:02.780260+0000","last_became_peered":"2024-09-21T11:07:02.780260+0000","last_unstale":"2024-09-21T11:08:56.458100+0000","last_undegraded":"2024-09-21T11:08:56.458100+0000","last_fullsized":"2024-09-21T11:08:56.458100+0000","mapping_epoch":1678,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:50.715390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":195,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.466149+0000","last_change":"2024-09-21T11:08:49.018833+0000","last_active":"2024-09-21T11:08:56.466149+0000","last_peered":"2024-09-21T11:08:56.466149+0000","last_clean":"2024-09-21T11:08:56.466149+0000","last_became_active":"2024-09-21T11:08:49.018474+0000","last_became_peered":"2024-09-21T11:08:49.018474+0000","last_unstale":"2024-09-21T11:08:56.466149+0000","last_undegraded":"2024-09-21T11:08:56.466149+0000","last_fullsized":"2024-09-21T11:08:56.466149+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:47:30.376303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":43,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.457630+0000","last_change":"2024-09-21T11:07:02.780932+0000","last_active":"2024-09-21T11:08:56.457630+0000","last_peered":"2024-09-21T11:08:56.457630+0000","last_clean":"2024-09-21T11:08:56.457630+0000","last_became_active":"2024-09-21T11:07:02.780451+0000","last_became_peered":"2024-09-21T11:07:02.780451+0000","last_unstale":"2024-09-21T11:08:56.457630+0000","last_undegraded":"2024-09-21T11:08:56.457630+0000","last_fullsized":"2024-09-21T11:08:56.457630+0000","mapping_epoch":1678,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:59:32.916443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":161,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.582300+0000","last_change":"2024-09-21T11:05:48.622455+0000","last_active":"2024-09-21T11:08:57.582300+0000","last_peered":"2024-09-21T11:08:57.582300+0000","last_clean":"2024-09-21T11:08:57.582300+0000","last_became_active":"2024-09-21T11:05:48.559270+0000","last_became_peered":"2024-09-21T11:05:48.559270+0000","last_unstale":"2024-09-21T11:08:57.582300+0000","last_undegraded":"2024-09-21T11:08:57.582300+0000","last_fullsized":"2024-09-21T11:08:57.582300+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:37:01.281922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":58,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.462406+0000","last_change":"2024-09-21T11:07:53.469868+0000","last_active":"2024-09-21T11:08:57.462406+0000","last_peered":"2024-09-21T11:08:57.462406+0000","last_clean":"2024-09-21T11:08:57.462406+0000","last_became_active":"2024-09-21T11:07:53.469589+0000","last_became_peered":"2024-09-21T11:07:53.469589+0000","last_unstale":"2024-09-21T11:08:57.462406+0000","last_undegraded":"2024-09-21T11:08:57.462406+0000","last_fullsized":"2024-09-21T11:08:57.462406+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:29:38.395222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":224,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.458226+0000","last_change":"2024-09-21T11:07:53.468703+0000","last_active":"2024-09-21T11:08:56.458226+0000","last_peered":"2024-09-21T11:08:56.458226+0000","last_clean":"2024-09-21T11:08:56.458226+0000","last_became_active":"2024-09-21T11:07:53.468332+0000","last_became_peered":"2024-09-21T11:07:53.468332+0000","last_unstale":"2024-09-21T11:08:56.458226+0000","last_undegraded":"2024-09-21T11:08:56.458226+0000","last_fullsized":"2024-09-21T11:08:56.458226+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:02:14.741210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":213,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.465081+0000","last_change":"2024-09-21T11:04:47.619509+0000","last_active":"2024-09-21T11:08:57.465081+0000","last_peered":"2024-09-21T11:08:57.465081+0000","last_clean":"2024-09-21T11:08:57.465081+0000","last_became_active":"2024-09-21T11:04:47.618898+0000","last_became_peered":"2024-09-21T11:04:47.618898+0000","last_unstale":"2024-09-21T11:08:57.465081+0000","last_undegraded":"2024-09-21T11:08:57.465081+0000","last_fullsized":"2024-09-21T11:08:57.465081+0000","mapping_epoch":1610,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1611,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:28:19.899682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2747,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:58.062635+0000","last_change":"2024-09-21T11:07:02.781730+0000","last_active":"2024-09-21T11:08:58.062635+0000","last_peered":"2024-09-21T11:08:58.062635+0000","last_clean":"2024-09-21T11:08:58.062635+0000","last_became_active":"2024-09-21T11:07:02.781410+0000","last_became_peered":"2024-09-21T11:07:02.781410+0000","last_unstale":"2024-09-21T11:08:58.062635+0000","last_undegraded":"2024-09-21T11:08:58.062635+0000","last_fullsized":"2024-09-21T11:08:58.062635+0000","mapping_epoch":1678,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:18:26.326185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":175,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.582482+0000","last_change":"2024-09-21T11:06:55.763204+0000","last_active":"2024-09-21T11:08:57.582482+0000","last_peered":"2024-09-21T11:08:57.582482+0000","last_clean":"2024-09-21T11:08:57.582482+0000","last_became_active":"2024-09-21T11:06:55.762743+0000","last_became_peered":"2024-09-21T11:06:55.762743+0000","last_unstale":"2024-09-21T11:08:57.582482+0000","last_undegraded":"2024-09-21T11:08:57.582482+0000","last_fullsized":"2024-09-21T11:08:57.582482+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:25:04.804191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":227,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.464858+0000","last_change":"2024-09-21T11:06:33.533570+0000","last_active":"2024-09-21T11:08:57.464858+0000","last_peered":"2024-09-21T11:08:57.464858+0000","last_clean":"2024-09-21T11:08:57.464858+0000","last_became_active":"2024-09-21T11:06:33.533236+0000","last_became_peered":"2024-09-21T11:06:33.533236+0000","last_unstale":"2024-09-21T11:08:57.464858+0000","last_undegraded":"2024-09-21T11:08:57.464858+0000","last_fullsized":"2024-09-21T11:08:57.464858+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:50:25.103638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1685,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.458404+0000","last_change":"2024-09-21T11:07:50.448460+0000","last_active":"2024-09-21T11:08:56.458404+0000","last_peered":"2024-09-21T11:08:56.458404+0000","last_clean":"2024-09-21T11:08:56.458404+0000","last_became_active":"2024-09-21T11:07:50.448156+0000","last_became_peered":"2024-09-21T11:07:50.448156+0000","last_unstale":"2024-09-21T11:08:56.458404+0000","last_undegraded":"2024-09-21T11:08:56.458404+0000","last_fullsized":"2024-09-21T11:08:56.458404+0000","mapping_epoch":1684,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1685,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:35.275772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":226,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.464946+0000","last_change":"2024-09-21T11:04:34.523861+0000","last_active":"2024-09-21T11:08:57.464946+0000","last_peered":"2024-09-21T11:08:57.464946+0000","last_clean":"2024-09-21T11:08:57.464946+0000","last_became_active":"2024-09-21T11:04:34.523542+0000","last_became_peered":"2024-09-21T11:04:34.523542+0000","last_unstale":"2024-09-21T11:08:57.464946+0000","last_undegraded":"2024-09-21T11:08:57.464946+0000","last_fullsized":"2024-09-21T11:08:57.464946+0000","mapping_epoch":1604,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:35.990638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":168,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.582507+0000","last_change":"2024-09-21T11:07:50.443304+0000","last_active":"2024-09-21T11:08:57.582507+0000","last_peered":"2024-09-21T11:08:57.582507+0000","last_clean":"2024-09-21T11:08:57.582507+0000","last_became_active":"2024-09-21T11:07:50.442989+0000","last_became_peered":"2024-09-21T11:07:50.442989+0000","last_unstale":"2024-09-21T11:08:57.582507+0000","last_undegraded":"2024-09-21T11:08:57.582507+0000","last_fullsized":"2024-09-21T11:08:57.582507+0000","mapping_epoch":1684,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1685,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:05.194134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"346'149","reported_seq":1159,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.462545+0000","last_change":"2024-09-21T11:07:56.474420+0000","last_active":"2024-09-21T11:08:57.462545+0000","last_peered":"2024-09-21T11:08:57.462545+0000","last_clean":"2024-09-21T11:08:57.462545+0000","last_became_active":"2024-09-21T11:07:56.474030+0000","last_became_peered":"2024-09-21T11:07:56.474030+0000","last_unstale":"2024-09-21T11:08:57.462545+0000","last_undegraded":"2024-09-21T11:08:57.462545+0000","last_fullsized":"2024-09-21T11:08:57.462545+0000","mapping_epoch":1690,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1691,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:53.724544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:53.724544+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:09:40.699968+0000","scrub_duration":0.0017779029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2771,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.465483+0000","last_change":"2024-09-21T11:06:55.761046+0000","last_active":"2024-09-21T11:08:57.465483+0000","last_peered":"2024-09-21T11:08:57.465483+0000","last_clean":"2024-09-21T11:08:57.465483+0000","last_became_active":"2024-09-21T11:06:55.760680+0000","last_became_peered":"2024-09-21T11:06:55.760680+0000","last_unstale":"2024-09-21T11:08:57.465483+0000","last_undegraded":"2024-09-21T11:08:57.465483+0000","last_fullsized":"2024-09-21T11:08:57.465483+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:44:27.436994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":37,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.466021+0000","last_change":"2024-09-21T11:07:50.443352+0000","last_active":"2024-09-21T11:08:56.466021+0000","last_peered":"2024-09-21T11:08:56.466021+0000","last_clean":"2024-09-21T11:08:56.466021+0000","last_became_active":"2024-09-21T11:07:50.443042+0000","last_became_peered":"2024-09-21T11:07:50.443042+0000","last_unstale":"2024-09-21T11:08:56.466021+0000","last_undegraded":"2024-09-21T11:08:56.466021+0000","last_fullsized":"2024-09-21T11:08:56.466021+0000","mapping_epoch":1684,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1685,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:12:03.053403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":56,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.822770+0000","last_change":"2024-09-21T11:06:55.768569+0000","last_active":"2024-09-21T11:08:57.822770+0000","last_peered":"2024-09-21T11:08:57.822770+0000","last_clean":"2024-09-21T11:08:57.822770+0000","last_became_active":"2024-09-21T11:06:55.761500+0000","last_became_peered":"2024-09-21T11:06:55.761500+0000","last_unstale":"2024-09-21T11:08:57.822770+0000","last_undegraded":"2024-09-21T11:08:57.822770+0000","last_fullsized":"2024-09-21T11:08:57.822770+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:55:27.336715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":124,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:58.062152+0000","last_change":"2024-09-21T11:07:53.463418+0000","last_active":"2024-09-21T11:08:58.062152+0000","last_peered":"2024-09-21T11:08:58.062152+0000","last_clean":"2024-09-21T11:08:58.062152+0000","last_became_active":"2024-09-21T11:07:53.463101+0000","last_became_peered":"2024-09-21T11:07:53.463101+0000","last_unstale":"2024-09-21T11:08:58.062152+0000","last_undegraded":"2024-09-21T11:08:58.062152+0000","last_fullsized":"2024-09-21T11:08:58.062152+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:39:31.345302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":214,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.458326+0000","last_change":"2024-09-21T11:02:39.505094+0000","last_active":"2024-09-21T11:08:56.458326+0000","last_peered":"2024-09-21T11:08:56.458326+0000","last_clean":"2024-09-21T11:08:56.458326+0000","last_became_active":"2024-09-21T11:02:39.503527+0000","last_became_peered":"2024-09-21T11:02:39.503527+0000","last_unstale":"2024-09-21T11:08:56.458326+0000","last_undegraded":"2024-09-21T11:08:56.458326+0000","last_fullsized":"2024-09-21T11:08:56.458326+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:12:34.916393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14268,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":783,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.457884+0000","last_change":"2024-09-21T11:02:39.492530+0000","last_active":"2024-09-21T11:08:56.457884+0000","last_peered":"2024-09-21T11:08:56.457884+0000","last_clean":"2024-09-21T11:08:56.457884+0000","last_became_active":"2024-09-21T11:02:39.491835+0000","last_became_peered":"2024-09-21T11:02:39.491835+0000","last_unstale":"2024-09-21T11:08:56.457884+0000","last_undegraded":"2024-09-21T11:08:56.457884+0000","last_fullsized":"2024-09-21T11:08:56.457884+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:37.157843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"346'149","reported_seq":1026,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.465100+0000","last_change":"2024-09-21T11:07:53.470728+0000","last_active":"2024-09-21T11:08:57.465100+0000","last_peered":"2024-09-21T11:08:57.465100+0000","last_clean":"2024-09-21T11:08:57.465100+0000","last_became_active":"2024-09-21T11:07:53.469520+0000","last_became_peered":"2024-09-21T11:07:53.469520+0000","last_unstale":"2024-09-21T11:08:57.465100+0000","last_undegraded":"2024-09-21T11:08:57.465100+0000","last_fullsized":"2024-09-21T11:08:57.465100+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:59.781270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:59.781270+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:56:37.036782+0000","scrub_duration":0.0031203189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1659,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.457888+0000","last_change":"2024-09-21T11:07:50.448728+0000","last_active":"2024-09-21T11:08:56.457888+0000","last_peered":"2024-09-21T11:08:56.457888+0000","last_clean":"2024-09-21T11:08:56.457888+0000","last_became_active":"2024-09-21T11:07:50.448383+0000","last_became_peered":"2024-09-21T11:07:50.448383+0000","last_unstale":"2024-09-21T11:08:56.457888+0000","last_undegraded":"2024-09-21T11:08:56.457888+0000","last_fullsized":"2024-09-21T11:08:56.457888+0000","mapping_epoch":1684,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1685,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:24.726052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1733,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.462599+0000","last_change":"2024-09-21T11:07:53.464587+0000","last_active":"2024-09-21T11:08:57.462599+0000","last_peered":"2024-09-21T11:08:57.462599+0000","last_clean":"2024-09-21T11:08:57.462599+0000","last_became_active":"2024-09-21T11:07:53.464295+0000","last_became_peered":"2024-09-21T11:07:53.464295+0000","last_unstale":"2024-09-21T11:08:57.462599+0000","last_undegraded":"2024-09-21T11:08:57.462599+0000","last_fullsized":"2024-09-21T11:08:57.462599+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:08:04.525051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":165,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.582595+0000","last_change":"2024-09-21T11:05:48.624344+0000","last_active":"2024-09-21T11:08:57.582595+0000","last_peered":"2024-09-21T11:08:57.582595+0000","last_clean":"2024-09-21T11:08:57.582595+0000","last_became_active":"2024-09-21T11:05:48.564758+0000","last_became_peered":"2024-09-21T11:05:48.564758+0000","last_unstale":"2024-09-21T11:08:57.582595+0000","last_undegraded":"2024-09-21T11:08:57.582595+0000","last_fullsized":"2024-09-21T11:08:57.582595+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:32.746530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"346'149","reported_seq":923,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.458561+0000","last_change":"2024-09-21T11:07:56.474206+0000","last_active":"2024-09-21T11:08:56.458561+0000","last_peered":"2024-09-21T11:08:56.458561+0000","last_clean":"2024-09-21T11:08:56.458561+0000","last_became_active":"2024-09-21T11:07:56.473654+0000","last_became_peered":"2024-09-21T11:07:56.473654+0000","last_unstale":"2024-09-21T11:08:56.458561+0000","last_undegraded":"2024-09-21T11:08:56.458561+0000","last_fullsized":"2024-09-21T11:08:56.458561+0000","mapping_epoch":1690,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1691,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:54.879276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:54.879276+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:39:06.647929+0000","scrub_duration":0.001843014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":110,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.464746+0000","last_change":"2024-09-21T11:06:55.759358+0000","last_active":"2024-09-21T11:08:57.464746+0000","last_peered":"2024-09-21T11:08:57.464746+0000","last_clean":"2024-09-21T11:08:57.464746+0000","last_became_active":"2024-09-21T11:06:55.759040+0000","last_became_peered":"2024-09-21T11:06:55.759040+0000","last_unstale":"2024-09-21T11:08:57.464746+0000","last_undegraded":"2024-09-21T11:08:57.464746+0000","last_fullsized":"2024-09-21T11:08:57.464746+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:59:39.297517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":117,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:58.062135+0000","last_change":"2024-09-21T11:06:33.532315+0000","last_active":"2024-09-21T11:08:58.062135+0000","last_peered":"2024-09-21T11:08:58.062135+0000","last_clean":"2024-09-21T11:08:58.062135+0000","last_became_active":"2024-09-21T11:06:33.532024+0000","last_became_peered":"2024-09-21T11:06:33.532024+0000","last_unstale":"2024-09-21T11:08:58.062135+0000","last_undegraded":"2024-09-21T11:08:58.062135+0000","last_fullsized":"2024-09-21T11:08:58.062135+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:35:36.295788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":2834,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.465336+0000","last_change":"2024-09-21T11:06:55.766194+0000","last_active":"2024-09-21T11:08:57.465336+0000","last_peered":"2024-09-21T11:08:57.465336+0000","last_clean":"2024-09-21T11:08:57.465336+0000","last_became_active":"2024-09-21T11:06:55.761357+0000","last_became_peered":"2024-09-21T11:06:55.761357+0000","last_unstale":"2024-09-21T11:08:57.465336+0000","last_undegraded":"2024-09-21T11:08:57.465336+0000","last_fullsized":"2024-09-21T11:08:57.465336+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:19:18.903358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"346'149","reported_seq":1023,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.462410+0000","last_change":"2024-09-21T11:06:43.715659+0000","last_active":"2024-09-21T11:08:57.462410+0000","last_peered":"2024-09-21T11:08:57.462410+0000","last_clean":"2024-09-21T11:08:57.462410+0000","last_became_active":"2024-09-21T11:06:43.715321+0000","last_became_peered":"2024-09-21T11:06:43.715321+0000","last_unstale":"2024-09-21T11:08:57.462410+0000","last_undegraded":"2024-09-21T11:08:57.462410+0000","last_fullsized":"2024-09-21T11:08:57.462410+0000","mapping_epoch":1668,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1669,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:06:00.911909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:06:00.911909+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:15:40.470846+0000","scrub_duration":0.002144838,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":168,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.582212+0000","last_change":"2024-09-21T11:07:50.444005+0000","last_active":"2024-09-21T11:08:57.582212+0000","last_peered":"2024-09-21T11:08:57.582212+0000","last_clean":"2024-09-21T11:08:57.582212+0000","last_became_active":"2024-09-21T11:07:50.443662+0000","last_became_peered":"2024-09-21T11:07:50.443662+0000","last_unstale":"2024-09-21T11:08:57.582212+0000","last_undegraded":"2024-09-21T11:08:57.582212+0000","last_fullsized":"2024-09-21T11:08:57.582212+0000","mapping_epoch":1684,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1685,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:12:43.497347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":311,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.822892+0000","last_change":"2024-09-21T11:05:48.528383+0000","last_active":"2024-09-21T11:08:57.822892+0000","last_peered":"2024-09-21T11:08:57.822892+0000","last_clean":"2024-09-21T11:08:57.822892+0000","last_became_active":"2024-09-21T11:05:48.527903+0000","last_became_peered":"2024-09-21T11:05:48.527903+0000","last_unstale":"2024-09-21T11:08:57.822892+0000","last_undegraded":"2024-09-21T11:08:57.822892+0000","last_fullsized":"2024-09-21T11:08:57.822892+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:59:25.065009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2607,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:58.062966+0000","last_change":"2024-09-21T11:08:08.575131+0000","last_active":"2024-09-21T11:08:58.062966+0000","last_peered":"2024-09-21T11:08:58.062966+0000","last_clean":"2024-09-21T11:08:58.062966+0000","last_became_active":"2024-09-21T11:08:08.574829+0000","last_became_peered":"2024-09-21T11:08:08.574829+0000","last_unstale":"2024-09-21T11:08:58.062966+0000","last_undegraded":"2024-09-21T11:08:58.062966+0000","last_fullsized":"2024-09-21T11:08:58.062966+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:15:57.943663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"346'149","reported_seq":1061,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.462681+0000","last_change":"2024-09-21T11:06:58.762593+0000","last_active":"2024-09-21T11:08:57.462681+0000","last_peered":"2024-09-21T11:08:57.462681+0000","last_clean":"2024-09-21T11:08:57.462681+0000","last_became_active":"2024-09-21T11:06:58.762117+0000","last_became_peered":"2024-09-21T11:06:58.762117+0000","last_unstale":"2024-09-21T11:08:57.462681+0000","last_undegraded":"2024-09-21T11:08:57.462681+0000","last_fullsized":"2024-09-21T11:08:57.462681+0000","mapping_epoch":1675,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1676,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:05:57.002981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:05:57.002981+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:58:47.258964+0000","scrub_duration":0.0017979249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":131,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:58.062753+0000","last_change":"2024-09-21T11:07:50.446812+0000","last_active":"2024-09-21T11:08:58.062753+0000","last_peered":"2024-09-21T11:08:58.062753+0000","last_clean":"2024-09-21T11:08:58.062753+0000","last_became_active":"2024-09-21T11:07:50.446435+0000","last_became_peered":"2024-09-21T11:07:50.446435+0000","last_unstale":"2024-09-21T11:08:58.062753+0000","last_undegraded":"2024-09-21T11:08:58.062753+0000","last_fullsized":"2024-09-21T11:08:58.062753+0000","mapping_epoch":1684,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1685,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:16:31.855319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2621,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:58.062868+0000","last_change":"2024-09-21T11:08:08.560254+0000","last_active":"2024-09-21T11:08:58.062868+0000","last_peered":"2024-09-21T11:08:58.062868+0000","last_clean":"2024-09-21T11:08:58.062868+0000","last_became_active":"2024-09-21T11:08:08.559925+0000","last_became_peered":"2024-09-21T11:08:08.559925+0000","last_unstale":"2024-09-21T11:08:58.062868+0000","last_undegraded":"2024-09-21T11:08:58.062868+0000","last_fullsized":"2024-09-21T11:08:58.062868+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:49:16.328026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":194,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.466129+0000","last_change":"2024-09-21T11:08:49.017651+0000","last_active":"2024-09-21T11:08:56.466129+0000","last_peered":"2024-09-21T11:08:56.466129+0000","last_clean":"2024-09-21T11:08:56.466129+0000","last_became_active":"2024-09-21T11:08:49.017325+0000","last_became_peered":"2024-09-21T11:08:49.017325+0000","last_unstale":"2024-09-21T11:08:56.466129+0000","last_undegraded":"2024-09-21T11:08:56.466129+0000","last_fullsized":"2024-09-21T11:08:56.466129+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:46:21.745857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"346'149","reported_seq":1058,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.465822+0000","last_change":"2024-09-21T11:08:49.015549+0000","last_active":"2024-09-21T11:08:56.465822+0000","last_peered":"2024-09-21T11:08:56.465822+0000","last_clean":"2024-09-21T11:08:56.465822+0000","last_became_active":"2024-09-21T11:08:49.015214+0000","last_became_peered":"2024-09-21T11:08:49.015214+0000","last_unstale":"2024-09-21T11:08:56.465822+0000","last_undegraded":"2024-09-21T11:08:56.465822+0000","last_fullsized":"2024-09-21T11:08:56.465822+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:48.918874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.918874+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:05:38.152071+0000","scrub_duration":0.001629134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":142,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:58.062802+0000","last_change":"2024-09-21T11:08:08.574895+0000","last_active":"2024-09-21T11:08:58.062802+0000","last_peered":"2024-09-21T11:08:58.062802+0000","last_clean":"2024-09-21T11:08:58.062802+0000","last_became_active":"2024-09-21T11:08:08.574606+0000","last_became_peered":"2024-09-21T11:08:08.574606+0000","last_unstale":"2024-09-21T11:08:58.062802+0000","last_undegraded":"2024-09-21T11:08:58.062802+0000","last_fullsized":"2024-09-21T11:08:58.062802+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:35:27.634442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1621,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.458019+0000","last_change":"2024-09-21T11:05:48.549704+0000","last_active":"2024-09-21T11:08:56.458019+0000","last_peered":"2024-09-21T11:08:56.458019+0000","last_clean":"2024-09-21T11:08:56.458019+0000","last_became_active":"2024-09-21T11:05:48.549374+0000","last_became_peered":"2024-09-21T11:05:48.549374+0000","last_unstale":"2024-09-21T11:08:56.458019+0000","last_undegraded":"2024-09-21T11:08:56.458019+0000","last_fullsized":"2024-09-21T11:08:56.458019+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:57.002097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":672,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.465933+0000","last_change":"2024-09-21T11:08:49.015428+0000","last_active":"2024-09-21T11:08:56.465933+0000","last_peered":"2024-09-21T11:08:56.465933+0000","last_clean":"2024-09-21T11:08:56.465933+0000","last_became_active":"2024-09-21T11:08:49.014725+0000","last_became_peered":"2024-09-21T11:08:49.014725+0000","last_unstale":"2024-09-21T11:08:56.465933+0000","last_undegraded":"2024-09-21T11:08:56.465933+0000","last_fullsized":"2024-09-21T11:08:56.465933+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:50:21.944099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":95,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"346'149","reported_seq":985,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.821900+0000","last_change":"2024-09-21T11:07:05.782919+0000","last_active":"2024-09-21T11:08:57.821900+0000","last_peered":"2024-09-21T11:08:57.821900+0000","last_clean":"2024-09-21T11:08:57.821900+0000","last_became_active":"2024-09-21T11:07:05.782486+0000","last_became_peered":"2024-09-21T11:07:05.782486+0000","last_unstale":"2024-09-21T11:08:57.821900+0000","last_undegraded":"2024-09-21T11:08:57.821900+0000","last_fullsized":"2024-09-21T11:08:57.821900+0000","mapping_epoch":1681,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1682,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:06:44.106093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:06:44.106093+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:58:14.998229+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.825547+0000","last_change":"2024-09-21T11:08:57.825277+0000","last_active":"2024-09-21T11:08:57.825547+0000","last_peered":"2024-09-21T11:08:57.825547+0000","last_clean":"2024-09-21T11:08:57.825547+0000","last_became_active":"2024-09-21T11:08:57.825021+0000","last_became_peered":"2024-09-21T11:08:57.825021+0000","last_unstale":"2024-09-21T11:08:57.825547+0000","last_undegraded":"2024-09-21T11:08:57.825547+0000","last_fullsized":"2024-09-21T11:08:57.825547+0000","mapping_epoch":1710,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1711,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:57:48.857063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2599,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:58.062638+0000","last_change":"2024-09-21T11:08:08.575089+0000","last_active":"2024-09-21T11:08:58.062638+0000","last_peered":"2024-09-21T11:08:58.062638+0000","last_clean":"2024-09-21T11:08:58.062638+0000","last_became_active":"2024-09-21T11:08:08.574703+0000","last_became_peered":"2024-09-21T11:08:08.574703+0000","last_unstale":"2024-09-21T11:08:58.062638+0000","last_undegraded":"2024-09-21T11:08:58.062638+0000","last_fullsized":"2024-09-21T11:08:58.062638+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:18:10.557195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":58,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.462088+0000","last_change":"2024-09-21T11:07:53.471395+0000","last_active":"2024-09-21T11:08:57.462088+0000","last_peered":"2024-09-21T11:08:57.462088+0000","last_clean":"2024-09-21T11:08:57.462088+0000","last_became_active":"2024-09-21T11:07:53.471078+0000","last_became_peered":"2024-09-21T11:07:53.471078+0000","last_unstale":"2024-09-21T11:08:57.462088+0000","last_undegraded":"2024-09-21T11:08:57.462088+0000","last_fullsized":"2024-09-21T11:08:57.462088+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:57:46.357558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"346'149","reported_seq":1279,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:58.062909+0000","last_change":"2024-09-21T11:08:08.558631+0000","last_active":"2024-09-21T11:08:58.062909+0000","last_peered":"2024-09-21T11:08:58.062909+0000","last_clean":"2024-09-21T11:08:58.062909+0000","last_became_active":"2024-09-21T11:08:08.558303+0000","last_became_peered":"2024-09-21T11:08:08.558303+0000","last_unstale":"2024-09-21T11:08:58.062909+0000","last_undegraded":"2024-09-21T11:08:58.062909+0000","last_fullsized":"2024-09-21T11:08:58.062909+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:50.671365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:50.671365+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:29:56.261115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":175,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.581667+0000","last_change":"2024-09-21T11:07:53.461508+0000","last_active":"2024-09-21T11:08:57.581667+0000","last_peered":"2024-09-21T11:08:57.581667+0000","last_clean":"2024-09-21T11:08:57.581667+0000","last_became_active":"2024-09-21T11:07:53.461184+0000","last_became_peered":"2024-09-21T11:07:53.461184+0000","last_unstale":"2024-09-21T11:08:57.581667+0000","last_undegraded":"2024-09-21T11:08:57.581667+0000","last_fullsized":"2024-09-21T11:08:57.581667+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:23:45.693130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":175,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.581913+0000","last_change":"2024-09-21T11:07:02.780287+0000","last_active":"2024-09-21T11:08:57.581913+0000","last_peered":"2024-09-21T11:08:57.581913+0000","last_clean":"2024-09-21T11:08:57.581913+0000","last_became_active":"2024-09-21T11:07:02.779916+0000","last_became_peered":"2024-09-21T11:07:02.779916+0000","last_unstale":"2024-09-21T11:08:57.581913+0000","last_undegraded":"2024-09-21T11:08:57.581913+0000","last_fullsized":"2024-09-21T11:08:57.581913+0000","mapping_epoch":1678,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:10.990224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":400,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.458752+0000","last_change":"2024-09-21T11:06:33.531901+0000","last_active":"2024-09-21T11:08:56.458752+0000","last_peered":"2024-09-21T11:08:56.458752+0000","last_clean":"2024-09-21T11:08:56.458752+0000","last_became_active":"2024-09-21T11:06:33.531562+0000","last_became_peered":"2024-09-21T11:06:33.531562+0000","last_unstale":"2024-09-21T11:08:56.458752+0000","last_undegraded":"2024-09-21T11:08:56.458752+0000","last_fullsized":"2024-09-21T11:08:56.458752+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:04.767469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":242,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.458103+0000","last_change":"2024-09-21T11:02:39.502922+0000","last_active":"2024-09-21T11:08:56.458103+0000","last_peered":"2024-09-21T11:08:56.458103+0000","last_clean":"2024-09-21T11:08:56.458103+0000","last_became_active":"2024-09-21T11:02:39.502633+0000","last_became_peered":"2024-09-21T11:02:39.502633+0000","last_unstale":"2024-09-21T11:08:56.458103+0000","last_undegraded":"2024-09-21T11:08:56.458103+0000","last_fullsized":"2024-09-21T11:08:56.458103+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:29:20.971522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":211,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.465061+0000","last_change":"2024-09-21T11:07:50.441026+0000","last_active":"2024-09-21T11:08:57.465061+0000","last_peered":"2024-09-21T11:08:57.465061+0000","last_clean":"2024-09-21T11:08:57.465061+0000","last_became_active":"2024-09-21T11:07:50.440650+0000","last_became_peered":"2024-09-21T11:07:50.440650+0000","last_unstale":"2024-09-21T11:08:57.465061+0000","last_undegraded":"2024-09-21T11:08:57.465061+0000","last_fullsized":"2024-09-21T11:08:57.465061+0000","mapping_epoch":1684,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1685,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:49:56.547116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":41,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.465992+0000","last_change":"2024-09-21T11:08:49.016227+0000","last_active":"2024-09-21T11:08:56.465992+0000","last_peered":"2024-09-21T11:08:56.465992+0000","last_clean":"2024-09-21T11:08:56.465992+0000","last_became_active":"2024-09-21T11:08:49.015010+0000","last_became_peered":"2024-09-21T11:08:49.015010+0000","last_unstale":"2024-09-21T11:08:56.465992+0000","last_undegraded":"2024-09-21T11:08:56.465992+0000","last_fullsized":"2024-09-21T11:08:56.465992+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:36:50.932413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":913,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.822216+0000","last_change":"2024-09-21T11:06:58.762931+0000","last_active":"2024-09-21T11:08:57.822216+0000","last_peered":"2024-09-21T11:08:57.822216+0000","last_clean":"2024-09-21T11:08:57.822216+0000","last_became_active":"2024-09-21T11:06:58.762397+0000","last_became_peered":"2024-09-21T11:06:58.762397+0000","last_unstale":"2024-09-21T11:08:57.822216+0000","last_undegraded":"2024-09-21T11:08:57.822216+0000","last_fullsized":"2024-09-21T11:08:57.822216+0000","mapping_epoch":1675,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1676,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:05:58.110003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:05:58.110003+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:59:21.260999+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"346'149","reported_seq":1006,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.465898+0000","last_change":"2024-09-21T11:08:49.015707+0000","last_active":"2024-09-21T11:08:56.465898+0000","last_peered":"2024-09-21T11:08:56.465898+0000","last_clean":"2024-09-21T11:08:56.465898+0000","last_became_active":"2024-09-21T11:08:49.015083+0000","last_became_peered":"2024-09-21T11:08:49.015083+0000","last_unstale":"2024-09-21T11:08:56.465898+0000","last_undegraded":"2024-09-21T11:08:56.465898+0000","last_fullsized":"2024-09-21T11:08:56.465898+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:42.008235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:42.008235+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:23:59.402474+0000","scrub_duration":0.011129834999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2736,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:58.063053+0000","last_change":"2024-09-21T11:01:46.484382+0000","last_active":"2024-09-21T11:08:58.063053+0000","last_peered":"2024-09-21T11:08:58.063053+0000","last_clean":"2024-09-21T11:08:58.063053+0000","last_became_active":"2024-09-21T11:01:46.484116+0000","last_became_peered":"2024-09-21T11:01:46.484116+0000","last_unstale":"2024-09-21T11:08:58.063053+0000","last_undegraded":"2024-09-21T11:08:58.063053+0000","last_fullsized":"2024-09-21T11:08:58.063053+0000","mapping_epoch":1578,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:48.924838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":110,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.464739+0000","last_change":"2024-09-21T11:06:55.760878+0000","last_active":"2024-09-21T11:08:57.464739+0000","last_peered":"2024-09-21T11:08:57.464739+0000","last_clean":"2024-09-21T11:08:57.464739+0000","last_became_active":"2024-09-21T11:06:55.760619+0000","last_became_peered":"2024-09-21T11:06:55.760619+0000","last_unstale":"2024-09-21T11:08:57.464739+0000","last_undegraded":"2024-09-21T11:08:57.464739+0000","last_fullsized":"2024-09-21T11:08:57.464739+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:05:47.078808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"346'149","reported_seq":1284,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:58.062799+0000","last_change":"2024-09-21T11:08:08.562618+0000","last_active":"2024-09-21T11:08:58.062799+0000","last_peered":"2024-09-21T11:08:58.062799+0000","last_clean":"2024-09-21T11:08:58.062799+0000","last_became_active":"2024-09-21T11:08:08.562125+0000","last_became_peered":"2024-09-21T11:08:08.562125+0000","last_unstale":"2024-09-21T11:08:58.062799+0000","last_undegraded":"2024-09-21T11:08:58.062799+0000","last_fullsized":"2024-09-21T11:08:58.062799+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:48.663349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.663349+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:45.603439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1650,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.458071+0000","last_change":"2024-09-21T11:06:33.535747+0000","last_active":"2024-09-21T11:08:56.458071+0000","last_peered":"2024-09-21T11:08:56.458071+0000","last_clean":"2024-09-21T11:08:56.458071+0000","last_became_active":"2024-09-21T11:06:33.535181+0000","last_became_peered":"2024-09-21T11:06:33.535181+0000","last_unstale":"2024-09-21T11:08:56.458071+0000","last_undegraded":"2024-09-21T11:08:56.458071+0000","last_fullsized":"2024-09-21T11:08:56.458071+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:52.206420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":53,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.462398+0000","last_change":"2024-09-21T11:07:53.468531+0000","last_active":"2024-09-21T11:08:57.462398+0000","last_peered":"2024-09-21T11:08:57.462398+0000","last_clean":"2024-09-21T11:08:57.462398+0000","last_became_active":"2024-09-21T11:07:53.468225+0000","last_became_peered":"2024-09-21T11:07:53.468225+0000","last_unstale":"2024-09-21T11:08:57.462398+0000","last_undegraded":"2024-09-21T11:08:57.462398+0000","last_fullsized":"2024-09-21T11:08:57.462398+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:36:06.139781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"346'149","reported_seq":1160,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.465777+0000","last_change":"2024-09-21T11:08:49.016904+0000","last_active":"2024-09-21T11:08:56.465777+0000","last_peered":"2024-09-21T11:08:56.465777+0000","last_clean":"2024-09-21T11:08:56.465777+0000","last_became_active":"2024-09-21T11:08:49.016201+0000","last_became_peered":"2024-09-21T11:08:49.016201+0000","last_unstale":"2024-09-21T11:08:56.465777+0000","last_undegraded":"2024-09-21T11:08:56.465777+0000","last_fullsized":"2024-09-21T11:08:56.465777+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:07:20.603775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:07:20.603775+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:20:50.045499+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":53,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.462351+0000","last_change":"2024-09-21T11:07:53.468717+0000","last_active":"2024-09-21T11:08:57.462351+0000","last_peered":"2024-09-21T11:08:57.462351+0000","last_clean":"2024-09-21T11:08:57.462351+0000","last_became_active":"2024-09-21T11:07:53.468381+0000","last_became_peered":"2024-09-21T11:07:53.468381+0000","last_unstale":"2024-09-21T11:08:57.462351+0000","last_undegraded":"2024-09-21T11:08:57.462351+0000","last_fullsized":"2024-09-21T11:08:57.462351+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:24.353490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1679,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.462493+0000","last_change":"2024-09-21T11:06:33.533209+0000","last_active":"2024-09-21T11:08:57.462493+0000","last_peered":"2024-09-21T11:08:57.462493+0000","last_clean":"2024-09-21T11:08:57.462493+0000","last_became_active":"2024-09-21T11:06:33.532865+0000","last_became_peered":"2024-09-21T11:06:33.532865+0000","last_unstale":"2024-09-21T11:08:57.462493+0000","last_undegraded":"2024-09-21T11:08:57.462493+0000","last_fullsized":"2024-09-21T11:08:57.462493+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:38:03.274980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"346'149","reported_seq":1209,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.822465+0000","last_change":"2024-09-21T11:07:56.474339+0000","last_active":"2024-09-21T11:08:57.822465+0000","last_peered":"2024-09-21T11:08:57.822465+0000","last_clean":"2024-09-21T11:08:57.822465+0000","last_became_active":"2024-09-21T11:07:56.473980+0000","last_became_peered":"2024-09-21T11:07:56.473980+0000","last_unstale":"2024-09-21T11:08:57.822465+0000","last_undegraded":"2024-09-21T11:08:57.822465+0000","last_fullsized":"2024-09-21T11:08:57.822465+0000","mapping_epoch":1690,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1691,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:17.418646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:17.418646+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:09:30.940593+0000","scrub_duration":0.00185648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":885,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.458465+0000","last_change":"2024-09-21T11:02:39.501678+0000","last_active":"2024-09-21T11:08:56.458465+0000","last_peered":"2024-09-21T11:08:56.458465+0000","last_clean":"2024-09-21T11:08:56.458465+0000","last_became_active":"2024-09-21T11:02:39.497766+0000","last_became_peered":"2024-09-21T11:02:39.497766+0000","last_unstale":"2024-09-21T11:08:56.458465+0000","last_undegraded":"2024-09-21T11:08:56.458465+0000","last_fullsized":"2024-09-21T11:08:56.458465+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:09:25.807702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":229,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.465228+0000","last_change":"2024-09-21T11:05:48.649119+0000","last_active":"2024-09-21T11:08:57.465228+0000","last_peered":"2024-09-21T11:08:57.465228+0000","last_clean":"2024-09-21T11:08:57.465228+0000","last_became_active":"2024-09-21T11:05:48.550467+0000","last_became_peered":"2024-09-21T11:05:48.550467+0000","last_unstale":"2024-09-21T11:08:57.465228+0000","last_undegraded":"2024-09-21T11:08:57.465228+0000","last_fullsized":"2024-09-21T11:08:57.465228+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:46:06.305352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"346'149","reported_seq":970,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.462341+0000","last_change":"2024-09-21T11:07:07.942671+0000","last_active":"2024-09-21T11:08:57.462341+0000","last_peered":"2024-09-21T11:08:57.462341+0000","last_clean":"2024-09-21T11:08:57.462341+0000","last_became_active":"2024-09-21T11:07:05.782775+0000","last_became_peered":"2024-09-21T11:07:05.782775+0000","last_unstale":"2024-09-21T11:08:57.462341+0000","last_undegraded":"2024-09-21T11:08:57.462341+0000","last_fullsized":"2024-09-21T11:08:57.462341+0000","mapping_epoch":1681,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1682,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:07:07.942557+0000","last_deep_scrub":"346'149","last_deep_scrub_stamp":"2024-09-21T11:07:00.489593+0000","last_clean_scrub_stamp":"2024-09-21T11:07:07.942557+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:25:44.786324+0000","scrub_duration":0.0019991430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":161,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.582243+0000","last_change":"2024-09-21T11:05:48.622817+0000","last_active":"2024-09-21T11:08:57.582243+0000","last_peered":"2024-09-21T11:08:57.582243+0000","last_clean":"2024-09-21T11:08:57.582243+0000","last_became_active":"2024-09-21T11:05:48.559669+0000","last_became_peered":"2024-09-21T11:05:48.559669+0000","last_unstale":"2024-09-21T11:08:57.582243+0000","last_undegraded":"2024-09-21T11:08:57.582243+0000","last_fullsized":"2024-09-21T11:08:57.582243+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:25:03.540564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":172,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.466071+0000","last_change":"2024-09-21T11:08:49.017101+0000","last_active":"2024-09-21T11:08:56.466071+0000","last_peered":"2024-09-21T11:08:56.466071+0000","last_clean":"2024-09-21T11:08:56.466071+0000","last_became_active":"2024-09-21T11:08:49.016479+0000","last_became_peered":"2024-09-21T11:08:49.016479+0000","last_unstale":"2024-09-21T11:08:56.466071+0000","last_undegraded":"2024-09-21T11:08:56.466071+0000","last_fullsized":"2024-09-21T11:08:56.466071+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:54.839925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"346'149","reported_seq":1045,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.462808+0000","last_change":"2024-09-21T11:07:53.466166+0000","last_active":"2024-09-21T11:08:57.462808+0000","last_peered":"2024-09-21T11:08:57.462808+0000","last_clean":"2024-09-21T11:08:57.462808+0000","last_became_active":"2024-09-21T11:07:53.465779+0000","last_became_peered":"2024-09-21T11:07:53.465779+0000","last_unstale":"2024-09-21T11:08:57.462808+0000","last_undegraded":"2024-09-21T11:08:57.462808+0000","last_fullsized":"2024-09-21T11:08:57.462808+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:05:59.940755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:05:59.940755+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:39:39.406723+0000","scrub_duration":0.0034290169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":239,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.462678+0000","last_change":"2024-09-21T11:06:55.757992+0000","last_active":"2024-09-21T11:08:57.462678+0000","last_peered":"2024-09-21T11:08:57.462678+0000","last_clean":"2024-09-21T11:08:57.462678+0000","last_became_active":"2024-09-21T11:06:55.757384+0000","last_became_peered":"2024-09-21T11:06:55.757384+0000","last_unstale":"2024-09-21T11:08:57.462678+0000","last_undegraded":"2024-09-21T11:08:57.462678+0000","last_fullsized":"2024-09-21T11:08:57.462678+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:59:44.041307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":472,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.822997+0000","last_change":"2024-09-21T11:07:50.450388+0000","last_active":"2024-09-21T11:08:57.822997+0000","last_peered":"2024-09-21T11:08:57.822997+0000","last_clean":"2024-09-21T11:08:57.822997+0000","last_became_active":"2024-09-21T11:07:50.443015+0000","last_became_peered":"2024-09-21T11:07:50.443015+0000","last_unstale":"2024-09-21T11:08:57.822997+0000","last_undegraded":"2024-09-21T11:08:57.822997+0000","last_fullsized":"2024-09-21T11:08:57.822997+0000","mapping_epoch":1684,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1685,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:25:51.176230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"346'149","reported_seq":1017,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.465002+0000","last_change":"2024-09-21T11:06:10.594793+0000","last_active":"2024-09-21T11:08:57.465002+0000","last_peered":"2024-09-21T11:08:57.465002+0000","last_clean":"2024-09-21T11:08:57.465002+0000","last_became_active":"2024-09-21T11:06:10.594289+0000","last_became_peered":"2024-09-21T11:06:10.594289+0000","last_unstale":"2024-09-21T11:08:57.465002+0000","last_undegraded":"2024-09-21T11:08:57.465002+0000","last_fullsized":"2024-09-21T11:08:57.465002+0000","mapping_epoch":1651,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1652,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:54.708658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:54.708658+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:07.379791+0000","scrub_duration":0.0021035160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1715,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.462600+0000","last_change":"2024-09-21T11:06:40.705586+0000","last_active":"2024-09-21T11:08:57.462600+0000","last_peered":"2024-09-21T11:08:57.462600+0000","last_clean":"2024-09-21T11:08:57.462600+0000","last_became_active":"2024-09-21T11:06:40.705193+0000","last_became_peered":"2024-09-21T11:06:40.705193+0000","last_unstale":"2024-09-21T11:08:57.462600+0000","last_undegraded":"2024-09-21T11:08:57.462600+0000","last_fullsized":"2024-09-21T11:08:57.462600+0000","mapping_epoch":1665,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1666,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:25:37.486777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":67,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.822908+0000","last_change":"2024-09-21T11:07:02.784493+0000","last_active":"2024-09-21T11:08:57.822908+0000","last_peered":"2024-09-21T11:08:57.822908+0000","last_clean":"2024-09-21T11:08:57.822908+0000","last_became_active":"2024-09-21T11:07:02.781586+0000","last_became_peered":"2024-09-21T11:07:02.781586+0000","last_unstale":"2024-09-21T11:08:57.822908+0000","last_undegraded":"2024-09-21T11:08:57.822908+0000","last_fullsized":"2024-09-21T11:08:57.822908+0000","mapping_epoch":1678,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:06:42.604214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"346'149","reported_seq":887,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.822769+0000","last_change":"2024-09-21T11:06:58.765920+0000","last_active":"2024-09-21T11:08:57.822769+0000","last_peered":"2024-09-21T11:08:57.822769+0000","last_clean":"2024-09-21T11:08:57.822769+0000","last_became_active":"2024-09-21T11:06:58.765490+0000","last_became_peered":"2024-09-21T11:06:58.765490+0000","last_unstale":"2024-09-21T11:08:57.822769+0000","last_undegraded":"2024-09-21T11:08:57.822769+0000","last_fullsized":"2024-09-21T11:08:57.822769+0000","mapping_epoch":1675,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1676,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:53.896066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:53.896066+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:51.227905+0000","scrub_duration":0.001699578,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":37,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.466295+0000","last_change":"2024-09-21T11:08:49.016220+0000","last_active":"2024-09-21T11:08:56.466295+0000","last_peered":"2024-09-21T11:08:56.466295+0000","last_clean":"2024-09-21T11:08:56.466295+0000","last_became_active":"2024-09-21T11:08:49.015909+0000","last_became_peered":"2024-09-21T11:08:49.015909+0000","last_unstale":"2024-09-21T11:08:56.466295+0000","last_undegraded":"2024-09-21T11:08:56.466295+0000","last_fullsized":"2024-09-21T11:08:56.466295+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:10:28.511316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":54,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.823015+0000","last_change":"2024-09-21T11:06:55.771951+0000","last_active":"2024-09-21T11:08:57.823015+0000","last_peered":"2024-09-21T11:08:57.823015+0000","last_clean":"2024-09-21T11:08:57.823015+0000","last_became_active":"2024-09-21T11:06:55.763633+0000","last_became_peered":"2024-09-21T11:06:55.763633+0000","last_unstale":"2024-09-21T11:08:57.823015+0000","last_undegraded":"2024-09-21T11:08:57.823015+0000","last_fullsized":"2024-09-21T11:08:57.823015+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:59:18.987831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"346'149","reported_seq":962,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.466098+0000","last_change":"2024-09-21T11:07:53.460517+0000","last_active":"2024-09-21T11:08:56.466098+0000","last_peered":"2024-09-21T11:08:56.466098+0000","last_clean":"2024-09-21T11:08:56.466098+0000","last_became_active":"2024-09-21T11:07:53.460095+0000","last_became_peered":"2024-09-21T11:07:53.460095+0000","last_unstale":"2024-09-21T11:08:56.466098+0000","last_undegraded":"2024-09-21T11:08:56.466098+0000","last_fullsized":"2024-09-21T11:08:56.466098+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:06:37.734784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:06:37.734784+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:21:30.091664+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1772,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.457362+0000","last_change":"2024-09-21T11:07:50.454445+0000","last_active":"2024-09-21T11:08:56.457362+0000","last_peered":"2024-09-21T11:08:56.457362+0000","last_clean":"2024-09-21T11:08:56.457362+0000","last_became_active":"2024-09-21T11:07:50.445182+0000","last_became_peered":"2024-09-21T11:07:50.445182+0000","last_unstale":"2024-09-21T11:08:56.457362+0000","last_undegraded":"2024-09-21T11:08:56.457362+0000","last_fullsized":"2024-09-21T11:08:56.457362+0000","mapping_epoch":1684,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1685,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:21:52.363073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":117,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.823461+0000","last_change":"2024-09-21T11:08:57.823171+0000","last_active":"2024-09-21T11:08:57.823461+0000","last_peered":"2024-09-21T11:08:57.823461+0000","last_clean":"2024-09-21T11:08:57.823461+0000","last_became_active":"2024-09-21T11:08:57.822855+0000","last_became_peered":"2024-09-21T11:08:57.822855+0000","last_unstale":"2024-09-21T11:08:57.823461+0000","last_undegraded":"2024-09-21T11:08:57.823461+0000","last_fullsized":"2024-09-21T11:08:57.823461+0000","mapping_epoch":1710,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1711,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:09:31.574671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"346'149","reported_seq":1284,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:58.062792+0000","last_change":"2024-09-21T11:08:08.558636+0000","last_active":"2024-09-21T11:08:58.062792+0000","last_peered":"2024-09-21T11:08:58.062792+0000","last_clean":"2024-09-21T11:08:58.062792+0000","last_became_active":"2024-09-21T11:08:08.558332+0000","last_became_peered":"2024-09-21T11:08:08.558332+0000","last_unstale":"2024-09-21T11:08:58.062792+0000","last_undegraded":"2024-09-21T11:08:58.062792+0000","last_fullsized":"2024-09-21T11:08:58.062792+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:54.706428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:54.706428+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:01:04.548955+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":425,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.458983+0000","last_change":"2024-09-21T11:02:39.471842+0000","last_active":"2024-09-21T11:08:56.458983+0000","last_peered":"2024-09-21T11:08:56.458983+0000","last_clean":"2024-09-21T11:08:56.458983+0000","last_became_active":"2024-09-21T11:02:39.471339+0000","last_became_peered":"2024-09-21T11:02:39.471339+0000","last_unstale":"2024-09-21T11:08:56.458983+0000","last_undegraded":"2024-09-21T11:08:56.458983+0000","last_fullsized":"2024-09-21T11:08:56.458983+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:53:53.996670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":168,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.582570+0000","last_change":"2024-09-21T11:06:55.762258+0000","last_active":"2024-09-21T11:08:57.582570+0000","last_peered":"2024-09-21T11:08:57.582570+0000","last_clean":"2024-09-21T11:08:57.582570+0000","last_became_active":"2024-09-21T11:06:55.761824+0000","last_became_peered":"2024-09-21T11:06:55.761824+0000","last_unstale":"2024-09-21T11:08:57.582570+0000","last_undegraded":"2024-09-21T11:08:57.582570+0000","last_fullsized":"2024-09-21T11:08:57.582570+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:43:09.930665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"346'149","reported_seq":876,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.465828+0000","last_change":"2024-09-21T11:08:49.019809+0000","last_active":"2024-09-21T11:08:56.465828+0000","last_peered":"2024-09-21T11:08:56.465828+0000","last_clean":"2024-09-21T11:08:56.465828+0000","last_became_active":"2024-09-21T11:08:49.019265+0000","last_became_peered":"2024-09-21T11:08:49.019265+0000","last_unstale":"2024-09-21T11:08:56.465828+0000","last_undegraded":"2024-09-21T11:08:56.465828+0000","last_fullsized":"2024-09-21T11:08:56.465828+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:06:00.073278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:06:00.073278+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:47:43.307376+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":180,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.582127+0000","last_change":"2024-09-21T11:06:55.762794+0000","last_active":"2024-09-21T11:08:57.582127+0000","last_peered":"2024-09-21T11:08:57.582127+0000","last_clean":"2024-09-21T11:08:57.582127+0000","last_became_active":"2024-09-21T11:06:55.760109+0000","last_became_peered":"2024-09-21T11:06:55.760109+0000","last_unstale":"2024-09-21T11:08:57.582127+0000","last_undegraded":"2024-09-21T11:08:57.582127+0000","last_fullsized":"2024-09-21T11:08:57.582127+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:41:23.127841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1679,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.466073+0000","last_change":"2024-09-21T11:08:49.016929+0000","last_active":"2024-09-21T11:08:56.466073+0000","last_peered":"2024-09-21T11:08:56.466073+0000","last_clean":"2024-09-21T11:08:56.466073+0000","last_became_active":"2024-09-21T11:08:49.016610+0000","last_became_peered":"2024-09-21T11:08:49.016610+0000","last_unstale":"2024-09-21T11:08:56.466073+0000","last_undegraded":"2024-09-21T11:08:56.466073+0000","last_fullsized":"2024-09-21T11:08:56.466073+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:00:04.990132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"346'149","reported_seq":1118,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.581654+0000","last_change":"2024-09-21T11:07:53.469227+0000","last_active":"2024-09-21T11:08:57.581654+0000","last_peered":"2024-09-21T11:08:57.581654+0000","last_clean":"2024-09-21T11:08:57.581654+0000","last_became_active":"2024-09-21T11:07:53.468868+0000","last_became_peered":"2024-09-21T11:07:53.468868+0000","last_unstale":"2024-09-21T11:08:57.581654+0000","last_undegraded":"2024-09-21T11:08:57.581654+0000","last_fullsized":"2024-09-21T11:08:57.581654+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:47.955189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:47.955189+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:45:44.145119+0000","scrub_duration":0.0019884339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":455,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.459032+0000","last_change":"2024-09-21T11:07:53.474774+0000","last_active":"2024-09-21T11:08:56.459032+0000","last_peered":"2024-09-21T11:08:56.459032+0000","last_clean":"2024-09-21T11:08:56.459032+0000","last_became_active":"2024-09-21T11:07:53.465568+0000","last_became_peered":"2024-09-21T11:07:53.465568+0000","last_unstale":"2024-09-21T11:08:56.459032+0000","last_undegraded":"2024-09-21T11:08:56.459032+0000","last_fullsized":"2024-09-21T11:08:56.459032+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:17.170792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":170,"num_keys_recovered":50,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2733,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.465312+0000","last_change":"2024-09-21T11:06:33.530305+0000","last_active":"2024-09-21T11:08:57.465312+0000","last_peered":"2024-09-21T11:08:57.465312+0000","last_clean":"2024-09-21T11:08:57.465312+0000","last_became_active":"2024-09-21T11:06:33.529962+0000","last_became_peered":"2024-09-21T11:06:33.529962+0000","last_unstale":"2024-09-21T11:08:57.465312+0000","last_undegraded":"2024-09-21T11:08:57.465312+0000","last_fullsized":"2024-09-21T11:08:57.465312+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:41.527340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3129,"ondisk_log_size":3129,"up":63,"acting":63,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":95,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":19674,"num_keys_recovered":50,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3602,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234907,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1648080,"kb_used_data":10560,"kb_used_omap":1,"kb_used_meta":1637502,"kb_avail":92097072,"statfs":{"total":95995035648,"available":94307401728,"internally_reserved":0,"allocated":10813440,"data_stored":6001606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":1676802507},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1589,"seq":6824703033429,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":333256,"kb_used_data":10872,"kb_used_omap":128,"kb_used_meta":322239,"kb_avail":93411896,"statfs":{"total":95995035648,"available":95653781504,"internally_reserved":0,"allocated":11132928,"data_stored":6329159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":131575,"internal_metadata":329973257},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":1597,"seq":6859062771784,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":50820,"kb_used_data":10804,"kb_used_omap":101,"kb_used_meta":39898,"kb_avail":93694332,"statfs":{"total":95995035648,"available":95942995968,"internally_reserved":0,"allocated":11063296,"data_stored":6259993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103790,"internal_metadata":40856210},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019816,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1371148,"kb_used_data":10556,"kb_used_omap":1,"kb_used_meta":1360574,"kb_avail":92374004,"statfs":{"total":95995035648,"available":94590980096,"internally_reserved":0,"allocated":10809344,"data_stored":6001536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1393228230},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369560,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1449528,"kb_used_data":11116,"kb_used_omap":69,"kb_used_meta":1438330,"kb_avail":92295624,"statfs":{"total":95995035648,"available":94510718976,"internally_reserved":0,"allocated":11382784,"data_stored":6587104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":1472850050},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":680,"seq":2920577762015,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87296,"kb_used_data":10800,"kb_used_omap":76,"kb_used_meta":76403,"kb_avail":93657856,"statfs":{"total":95995035648,"available":95905644544,"internally_reserved":0,"allocated":11059200,"data_stored":6259950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78585,"internal_metadata":78236935},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807958246,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":801140,"kb_used_data":10536,"kb_used_omap":119,"kb_used_meta":790472,"kb_avail":92944012,"statfs":{"total":95995035648,"available":95174668288,"internally_reserved":0,"allocated":10788864,"data_stored":5996665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":121896,"internal_metadata":809444312},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":666,"seq":2860448219885,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":686840,"kb_used_data":10860,"kb_used_omap":76,"kb_used_meta":675891,"kb_avail":93058312,"statfs":{"total":95995035648,"available":95291711488,"internally_reserved":0,"allocated":11120640,"data_stored":6324930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78716,"internal_metadata":692112516},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1187,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T11:09:33.797 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T11:09:33.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:33 smithi160 ceph-mon[101879]: pgmap v3431: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:33.992 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T11:09:34.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:34 smithi070 ceph-mon[107385]: from='client.37218 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T11:09:34.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:34 smithi070 ceph-mon[112139]: from='client.37218 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T11:09:34.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:34 smithi160 ceph-mon[101879]: from='client.37218 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T11:09:35.782 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:35 smithi070 ceph-mon[107385]: pgmap v3432: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:35.782 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:35 smithi070 ceph-mon[107385]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:09:35.782 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:35 smithi070 ceph-mon[112139]: pgmap v3432: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:35.782 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:35 smithi070 ceph-mon[112139]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:09:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:35 smithi160 ceph-mon[101879]: pgmap v3432: 93 pgs: 93 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-09-21T11:09:35.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:35 smithi160 ceph-mon[101879]: from='mgr.34107 172.21.15.70:0/2526968734' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T11:09:35.944 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T11:09:35.945 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-21T11:09:36.565 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":3432,"stamp":"2024-09-21T11:09:34.397625+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595941,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":347449,"num_keys_recovered":50,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3188,"ondisk_log_size":3188,"up":279,"acting":279,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":279,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6428108,"kb_used_data":86104,"kb_used_omap":575,"kb_used_meta":6341312,"kb_avail":743533108,"statfs":{"total":767960285184,"available":761377902592,"internally_reserved":0,"allocated":88170496,"data_stored":49760943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":589295,"internal_metadata":6493504017},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003324"},"pg_stats":[{"pgid":"3.1f","version":"71'4","reported_seq":193,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.581709+0000","last_change":"2024-09-21T11:07:53.469858+0000","last_active":"2024-09-21T11:08:57.581709+0000","last_peered":"2024-09-21T11:08:57.581709+0000","last_clean":"2024-09-21T11:08:57.581709+0000","last_became_active":"2024-09-21T11:07:53.464219+0000","last_became_peered":"2024-09-21T11:07:53.464219+0000","last_unstale":"2024-09-21T11:08:57.581709+0000","last_undegraded":"2024-09-21T11:08:57.581709+0000","last_fullsized":"2024-09-21T11:08:57.581709+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:50:02.669630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4074,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1630,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.457558+0000","last_change":"2024-09-21T11:02:39.486670+0000","last_active":"2024-09-21T11:08:56.457558+0000","last_peered":"2024-09-21T11:08:56.457558+0000","last_clean":"2024-09-21T11:08:56.457558+0000","last_became_active":"2024-09-21T11:02:39.486248+0000","last_became_peered":"2024-09-21T11:02:39.486248+0000","last_unstale":"2024-09-21T11:08:56.457558+0000","last_undegraded":"2024-09-21T11:08:56.457558+0000","last_fullsized":"2024-09-21T11:08:56.457558+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:56:56.360708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":211,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.465386+0000","last_change":"2024-09-21T11:07:53.472545+0000","last_active":"2024-09-21T11:08:57.465386+0000","last_peered":"2024-09-21T11:08:57.465386+0000","last_clean":"2024-09-21T11:08:57.465386+0000","last_became_active":"2024-09-21T11:07:53.472156+0000","last_became_peered":"2024-09-21T11:07:53.472156+0000","last_unstale":"2024-09-21T11:08:57.465386+0000","last_undegraded":"2024-09-21T11:08:57.465386+0000","last_fullsized":"2024-09-21T11:08:57.465386+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:49:41.030319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":224,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.464923+0000","last_change":"2024-09-21T11:07:53.474533+0000","last_active":"2024-09-21T11:08:57.464923+0000","last_peered":"2024-09-21T11:08:57.464923+0000","last_clean":"2024-09-21T11:08:57.464923+0000","last_became_active":"2024-09-21T11:07:53.474107+0000","last_became_peered":"2024-09-21T11:07:53.474107+0000","last_unstale":"2024-09-21T11:08:57.464923+0000","last_undegraded":"2024-09-21T11:08:57.464923+0000","last_fullsized":"2024-09-21T11:08:57.464923+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:47:08.181090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":195,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.466058+0000","last_change":"2024-09-21T11:08:49.019024+0000","last_active":"2024-09-21T11:08:56.466058+0000","last_peered":"2024-09-21T11:08:56.466058+0000","last_clean":"2024-09-21T11:08:56.466058+0000","last_became_active":"2024-09-21T11:08:49.018435+0000","last_became_peered":"2024-09-21T11:08:49.018435+0000","last_unstale":"2024-09-21T11:08:56.466058+0000","last_undegraded":"2024-09-21T11:08:56.466058+0000","last_fullsized":"2024-09-21T11:08:56.466058+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:11:55.300993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":43,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.458100+0000","last_change":"2024-09-21T11:07:02.780603+0000","last_active":"2024-09-21T11:08:56.458100+0000","last_peered":"2024-09-21T11:08:56.458100+0000","last_clean":"2024-09-21T11:08:56.458100+0000","last_became_active":"2024-09-21T11:07:02.780260+0000","last_became_peered":"2024-09-21T11:07:02.780260+0000","last_unstale":"2024-09-21T11:08:56.458100+0000","last_undegraded":"2024-09-21T11:08:56.458100+0000","last_fullsized":"2024-09-21T11:08:56.458100+0000","mapping_epoch":1678,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:50.715390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":195,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.466149+0000","last_change":"2024-09-21T11:08:49.018833+0000","last_active":"2024-09-21T11:08:56.466149+0000","last_peered":"2024-09-21T11:08:56.466149+0000","last_clean":"2024-09-21T11:08:56.466149+0000","last_became_active":"2024-09-21T11:08:49.018474+0000","last_became_peered":"2024-09-21T11:08:49.018474+0000","last_unstale":"2024-09-21T11:08:56.466149+0000","last_undegraded":"2024-09-21T11:08:56.466149+0000","last_fullsized":"2024-09-21T11:08:56.466149+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:47:30.376303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":43,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.457630+0000","last_change":"2024-09-21T11:07:02.780932+0000","last_active":"2024-09-21T11:08:56.457630+0000","last_peered":"2024-09-21T11:08:56.457630+0000","last_clean":"2024-09-21T11:08:56.457630+0000","last_became_active":"2024-09-21T11:07:02.780451+0000","last_became_peered":"2024-09-21T11:07:02.780451+0000","last_unstale":"2024-09-21T11:08:56.457630+0000","last_undegraded":"2024-09-21T11:08:56.457630+0000","last_fullsized":"2024-09-21T11:08:56.457630+0000","mapping_epoch":1678,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:59:32.916443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":161,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.582300+0000","last_change":"2024-09-21T11:05:48.622455+0000","last_active":"2024-09-21T11:08:57.582300+0000","last_peered":"2024-09-21T11:08:57.582300+0000","last_clean":"2024-09-21T11:08:57.582300+0000","last_became_active":"2024-09-21T11:05:48.559270+0000","last_became_peered":"2024-09-21T11:05:48.559270+0000","last_unstale":"2024-09-21T11:08:57.582300+0000","last_undegraded":"2024-09-21T11:08:57.582300+0000","last_fullsized":"2024-09-21T11:08:57.582300+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:37:01.281922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":58,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.462406+0000","last_change":"2024-09-21T11:07:53.469868+0000","last_active":"2024-09-21T11:08:57.462406+0000","last_peered":"2024-09-21T11:08:57.462406+0000","last_clean":"2024-09-21T11:08:57.462406+0000","last_became_active":"2024-09-21T11:07:53.469589+0000","last_became_peered":"2024-09-21T11:07:53.469589+0000","last_unstale":"2024-09-21T11:08:57.462406+0000","last_undegraded":"2024-09-21T11:08:57.462406+0000","last_fullsized":"2024-09-21T11:08:57.462406+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:29:38.395222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":224,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.458226+0000","last_change":"2024-09-21T11:07:53.468703+0000","last_active":"2024-09-21T11:08:56.458226+0000","last_peered":"2024-09-21T11:08:56.458226+0000","last_clean":"2024-09-21T11:08:56.458226+0000","last_became_active":"2024-09-21T11:07:53.468332+0000","last_became_peered":"2024-09-21T11:07:53.468332+0000","last_unstale":"2024-09-21T11:08:56.458226+0000","last_undegraded":"2024-09-21T11:08:56.458226+0000","last_fullsized":"2024-09-21T11:08:56.458226+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:02:14.741210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":213,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.465081+0000","last_change":"2024-09-21T11:04:47.619509+0000","last_active":"2024-09-21T11:08:57.465081+0000","last_peered":"2024-09-21T11:08:57.465081+0000","last_clean":"2024-09-21T11:08:57.465081+0000","last_became_active":"2024-09-21T11:04:47.618898+0000","last_became_peered":"2024-09-21T11:04:47.618898+0000","last_unstale":"2024-09-21T11:08:57.465081+0000","last_undegraded":"2024-09-21T11:08:57.465081+0000","last_fullsized":"2024-09-21T11:08:57.465081+0000","mapping_epoch":1610,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1611,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:28:19.899682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2747,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:58.062635+0000","last_change":"2024-09-21T11:07:02.781730+0000","last_active":"2024-09-21T11:08:58.062635+0000","last_peered":"2024-09-21T11:08:58.062635+0000","last_clean":"2024-09-21T11:08:58.062635+0000","last_became_active":"2024-09-21T11:07:02.781410+0000","last_became_peered":"2024-09-21T11:07:02.781410+0000","last_unstale":"2024-09-21T11:08:58.062635+0000","last_undegraded":"2024-09-21T11:08:58.062635+0000","last_fullsized":"2024-09-21T11:08:58.062635+0000","mapping_epoch":1678,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:18:26.326185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":175,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.582482+0000","last_change":"2024-09-21T11:06:55.763204+0000","last_active":"2024-09-21T11:08:57.582482+0000","last_peered":"2024-09-21T11:08:57.582482+0000","last_clean":"2024-09-21T11:08:57.582482+0000","last_became_active":"2024-09-21T11:06:55.762743+0000","last_became_peered":"2024-09-21T11:06:55.762743+0000","last_unstale":"2024-09-21T11:08:57.582482+0000","last_undegraded":"2024-09-21T11:08:57.582482+0000","last_fullsized":"2024-09-21T11:08:57.582482+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:25:04.804191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":227,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.464858+0000","last_change":"2024-09-21T11:06:33.533570+0000","last_active":"2024-09-21T11:08:57.464858+0000","last_peered":"2024-09-21T11:08:57.464858+0000","last_clean":"2024-09-21T11:08:57.464858+0000","last_became_active":"2024-09-21T11:06:33.533236+0000","last_became_peered":"2024-09-21T11:06:33.533236+0000","last_unstale":"2024-09-21T11:08:57.464858+0000","last_undegraded":"2024-09-21T11:08:57.464858+0000","last_fullsized":"2024-09-21T11:08:57.464858+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:50:25.103638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1685,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.458404+0000","last_change":"2024-09-21T11:07:50.448460+0000","last_active":"2024-09-21T11:08:56.458404+0000","last_peered":"2024-09-21T11:08:56.458404+0000","last_clean":"2024-09-21T11:08:56.458404+0000","last_became_active":"2024-09-21T11:07:50.448156+0000","last_became_peered":"2024-09-21T11:07:50.448156+0000","last_unstale":"2024-09-21T11:08:56.458404+0000","last_undegraded":"2024-09-21T11:08:56.458404+0000","last_fullsized":"2024-09-21T11:08:56.458404+0000","mapping_epoch":1684,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1685,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:35.275772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":226,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.464946+0000","last_change":"2024-09-21T11:04:34.523861+0000","last_active":"2024-09-21T11:08:57.464946+0000","last_peered":"2024-09-21T11:08:57.464946+0000","last_clean":"2024-09-21T11:08:57.464946+0000","last_became_active":"2024-09-21T11:04:34.523542+0000","last_became_peered":"2024-09-21T11:04:34.523542+0000","last_unstale":"2024-09-21T11:08:57.464946+0000","last_undegraded":"2024-09-21T11:08:57.464946+0000","last_fullsized":"2024-09-21T11:08:57.464946+0000","mapping_epoch":1604,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:35.990638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":168,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.582507+0000","last_change":"2024-09-21T11:07:50.443304+0000","last_active":"2024-09-21T11:08:57.582507+0000","last_peered":"2024-09-21T11:08:57.582507+0000","last_clean":"2024-09-21T11:08:57.582507+0000","last_became_active":"2024-09-21T11:07:50.442989+0000","last_became_peered":"2024-09-21T11:07:50.442989+0000","last_unstale":"2024-09-21T11:08:57.582507+0000","last_undegraded":"2024-09-21T11:08:57.582507+0000","last_fullsized":"2024-09-21T11:08:57.582507+0000","mapping_epoch":1684,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1685,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:05.194134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"346'149","reported_seq":1159,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.462545+0000","last_change":"2024-09-21T11:07:56.474420+0000","last_active":"2024-09-21T11:08:57.462545+0000","last_peered":"2024-09-21T11:08:57.462545+0000","last_clean":"2024-09-21T11:08:57.462545+0000","last_became_active":"2024-09-21T11:07:56.474030+0000","last_became_peered":"2024-09-21T11:07:56.474030+0000","last_unstale":"2024-09-21T11:08:57.462545+0000","last_undegraded":"2024-09-21T11:08:57.462545+0000","last_fullsized":"2024-09-21T11:08:57.462545+0000","mapping_epoch":1690,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1691,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:53.724544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:53.724544+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:09:40.699968+0000","scrub_duration":0.0017779029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2771,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.465483+0000","last_change":"2024-09-21T11:06:55.761046+0000","last_active":"2024-09-21T11:08:57.465483+0000","last_peered":"2024-09-21T11:08:57.465483+0000","last_clean":"2024-09-21T11:08:57.465483+0000","last_became_active":"2024-09-21T11:06:55.760680+0000","last_became_peered":"2024-09-21T11:06:55.760680+0000","last_unstale":"2024-09-21T11:08:57.465483+0000","last_undegraded":"2024-09-21T11:08:57.465483+0000","last_fullsized":"2024-09-21T11:08:57.465483+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:44:27.436994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":37,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.466021+0000","last_change":"2024-09-21T11:07:50.443352+0000","last_active":"2024-09-21T11:08:56.466021+0000","last_peered":"2024-09-21T11:08:56.466021+0000","last_clean":"2024-09-21T11:08:56.466021+0000","last_became_active":"2024-09-21T11:07:50.443042+0000","last_became_peered":"2024-09-21T11:07:50.443042+0000","last_unstale":"2024-09-21T11:08:56.466021+0000","last_undegraded":"2024-09-21T11:08:56.466021+0000","last_fullsized":"2024-09-21T11:08:56.466021+0000","mapping_epoch":1684,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1685,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:12:03.053403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":56,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.822770+0000","last_change":"2024-09-21T11:06:55.768569+0000","last_active":"2024-09-21T11:08:57.822770+0000","last_peered":"2024-09-21T11:08:57.822770+0000","last_clean":"2024-09-21T11:08:57.822770+0000","last_became_active":"2024-09-21T11:06:55.761500+0000","last_became_peered":"2024-09-21T11:06:55.761500+0000","last_unstale":"2024-09-21T11:08:57.822770+0000","last_undegraded":"2024-09-21T11:08:57.822770+0000","last_fullsized":"2024-09-21T11:08:57.822770+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:55:27.336715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":124,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:58.062152+0000","last_change":"2024-09-21T11:07:53.463418+0000","last_active":"2024-09-21T11:08:58.062152+0000","last_peered":"2024-09-21T11:08:58.062152+0000","last_clean":"2024-09-21T11:08:58.062152+0000","last_became_active":"2024-09-21T11:07:53.463101+0000","last_became_peered":"2024-09-21T11:07:53.463101+0000","last_unstale":"2024-09-21T11:08:58.062152+0000","last_undegraded":"2024-09-21T11:08:58.062152+0000","last_fullsized":"2024-09-21T11:08:58.062152+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:39:31.345302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"425'23","reported_seq":214,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.458326+0000","last_change":"2024-09-21T11:02:39.505094+0000","last_active":"2024-09-21T11:08:56.458326+0000","last_peered":"2024-09-21T11:08:56.458326+0000","last_clean":"2024-09-21T11:08:56.458326+0000","last_became_active":"2024-09-21T11:02:39.503527+0000","last_became_peered":"2024-09-21T11:02:39.503527+0000","last_unstale":"2024-09-21T11:08:56.458326+0000","last_undegraded":"2024-09-21T11:08:56.458326+0000","last_fullsized":"2024-09-21T11:08:56.458326+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:12:34.916393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14268,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":783,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.457884+0000","last_change":"2024-09-21T11:02:39.492530+0000","last_active":"2024-09-21T11:08:56.457884+0000","last_peered":"2024-09-21T11:08:56.457884+0000","last_clean":"2024-09-21T11:08:56.457884+0000","last_became_active":"2024-09-21T11:02:39.491835+0000","last_became_peered":"2024-09-21T11:02:39.491835+0000","last_unstale":"2024-09-21T11:08:56.457884+0000","last_undegraded":"2024-09-21T11:08:56.457884+0000","last_fullsized":"2024-09-21T11:08:56.457884+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:37.157843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"346'149","reported_seq":1026,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.465100+0000","last_change":"2024-09-21T11:07:53.470728+0000","last_active":"2024-09-21T11:08:57.465100+0000","last_peered":"2024-09-21T11:08:57.465100+0000","last_clean":"2024-09-21T11:08:57.465100+0000","last_became_active":"2024-09-21T11:07:53.469520+0000","last_became_peered":"2024-09-21T11:07:53.469520+0000","last_unstale":"2024-09-21T11:08:57.465100+0000","last_undegraded":"2024-09-21T11:08:57.465100+0000","last_fullsized":"2024-09-21T11:08:57.465100+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:59.781270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:59.781270+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:56:37.036782+0000","scrub_duration":0.0031203189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1659,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.457888+0000","last_change":"2024-09-21T11:07:50.448728+0000","last_active":"2024-09-21T11:08:56.457888+0000","last_peered":"2024-09-21T11:08:56.457888+0000","last_clean":"2024-09-21T11:08:56.457888+0000","last_became_active":"2024-09-21T11:07:50.448383+0000","last_became_peered":"2024-09-21T11:07:50.448383+0000","last_unstale":"2024-09-21T11:08:56.457888+0000","last_undegraded":"2024-09-21T11:08:56.457888+0000","last_fullsized":"2024-09-21T11:08:56.457888+0000","mapping_epoch":1684,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1685,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:24.726052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1733,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.462599+0000","last_change":"2024-09-21T11:07:53.464587+0000","last_active":"2024-09-21T11:08:57.462599+0000","last_peered":"2024-09-21T11:08:57.462599+0000","last_clean":"2024-09-21T11:08:57.462599+0000","last_became_active":"2024-09-21T11:07:53.464295+0000","last_became_peered":"2024-09-21T11:07:53.464295+0000","last_unstale":"2024-09-21T11:08:57.462599+0000","last_undegraded":"2024-09-21T11:08:57.462599+0000","last_fullsized":"2024-09-21T11:08:57.462599+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:08:04.525051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028998199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":165,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.582595+0000","last_change":"2024-09-21T11:05:48.624344+0000","last_active":"2024-09-21T11:08:57.582595+0000","last_peered":"2024-09-21T11:08:57.582595+0000","last_clean":"2024-09-21T11:08:57.582595+0000","last_became_active":"2024-09-21T11:05:48.564758+0000","last_became_peered":"2024-09-21T11:05:48.564758+0000","last_unstale":"2024-09-21T11:08:57.582595+0000","last_undegraded":"2024-09-21T11:08:57.582595+0000","last_fullsized":"2024-09-21T11:08:57.582595+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:32.746530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"346'149","reported_seq":923,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.458561+0000","last_change":"2024-09-21T11:07:56.474206+0000","last_active":"2024-09-21T11:08:56.458561+0000","last_peered":"2024-09-21T11:08:56.458561+0000","last_clean":"2024-09-21T11:08:56.458561+0000","last_became_active":"2024-09-21T11:07:56.473654+0000","last_became_peered":"2024-09-21T11:07:56.473654+0000","last_unstale":"2024-09-21T11:08:56.458561+0000","last_undegraded":"2024-09-21T11:08:56.458561+0000","last_fullsized":"2024-09-21T11:08:56.458561+0000","mapping_epoch":1690,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1691,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:54.879276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:54.879276+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:39:06.647929+0000","scrub_duration":0.001843014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":110,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.464746+0000","last_change":"2024-09-21T11:06:55.759358+0000","last_active":"2024-09-21T11:08:57.464746+0000","last_peered":"2024-09-21T11:08:57.464746+0000","last_clean":"2024-09-21T11:08:57.464746+0000","last_became_active":"2024-09-21T11:06:55.759040+0000","last_became_peered":"2024-09-21T11:06:55.759040+0000","last_unstale":"2024-09-21T11:08:57.464746+0000","last_undegraded":"2024-09-21T11:08:57.464746+0000","last_fullsized":"2024-09-21T11:08:57.464746+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:59:39.297517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":117,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:58.062135+0000","last_change":"2024-09-21T11:06:33.532315+0000","last_active":"2024-09-21T11:08:58.062135+0000","last_peered":"2024-09-21T11:08:58.062135+0000","last_clean":"2024-09-21T11:08:58.062135+0000","last_became_active":"2024-09-21T11:06:33.532024+0000","last_became_peered":"2024-09-21T11:06:33.532024+0000","last_unstale":"2024-09-21T11:08:58.062135+0000","last_undegraded":"2024-09-21T11:08:58.062135+0000","last_fullsized":"2024-09-21T11:08:58.062135+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:35:36.295788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":2834,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.465336+0000","last_change":"2024-09-21T11:06:55.766194+0000","last_active":"2024-09-21T11:08:57.465336+0000","last_peered":"2024-09-21T11:08:57.465336+0000","last_clean":"2024-09-21T11:08:57.465336+0000","last_became_active":"2024-09-21T11:06:55.761357+0000","last_became_peered":"2024-09-21T11:06:55.761357+0000","last_unstale":"2024-09-21T11:08:57.465336+0000","last_undegraded":"2024-09-21T11:08:57.465336+0000","last_fullsized":"2024-09-21T11:08:57.465336+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:19:18.903358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"346'149","reported_seq":1023,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.462410+0000","last_change":"2024-09-21T11:06:43.715659+0000","last_active":"2024-09-21T11:08:57.462410+0000","last_peered":"2024-09-21T11:08:57.462410+0000","last_clean":"2024-09-21T11:08:57.462410+0000","last_became_active":"2024-09-21T11:06:43.715321+0000","last_became_peered":"2024-09-21T11:06:43.715321+0000","last_unstale":"2024-09-21T11:08:57.462410+0000","last_undegraded":"2024-09-21T11:08:57.462410+0000","last_fullsized":"2024-09-21T11:08:57.462410+0000","mapping_epoch":1668,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1669,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:06:00.911909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:06:00.911909+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:15:40.470846+0000","scrub_duration":0.002144838,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":168,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.582212+0000","last_change":"2024-09-21T11:07:50.444005+0000","last_active":"2024-09-21T11:08:57.582212+0000","last_peered":"2024-09-21T11:08:57.582212+0000","last_clean":"2024-09-21T11:08:57.582212+0000","last_became_active":"2024-09-21T11:07:50.443662+0000","last_became_peered":"2024-09-21T11:07:50.443662+0000","last_unstale":"2024-09-21T11:08:57.582212+0000","last_undegraded":"2024-09-21T11:08:57.582212+0000","last_fullsized":"2024-09-21T11:08:57.582212+0000","mapping_epoch":1684,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1685,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:12:43.497347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":311,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.822892+0000","last_change":"2024-09-21T11:05:48.528383+0000","last_active":"2024-09-21T11:08:57.822892+0000","last_peered":"2024-09-21T11:08:57.822892+0000","last_clean":"2024-09-21T11:08:57.822892+0000","last_became_active":"2024-09-21T11:05:48.527903+0000","last_became_peered":"2024-09-21T11:05:48.527903+0000","last_unstale":"2024-09-21T11:08:57.822892+0000","last_undegraded":"2024-09-21T11:08:57.822892+0000","last_fullsized":"2024-09-21T11:08:57.822892+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:59:25.065009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032031200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2607,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:58.062966+0000","last_change":"2024-09-21T11:08:08.575131+0000","last_active":"2024-09-21T11:08:58.062966+0000","last_peered":"2024-09-21T11:08:58.062966+0000","last_clean":"2024-09-21T11:08:58.062966+0000","last_became_active":"2024-09-21T11:08:08.574829+0000","last_became_peered":"2024-09-21T11:08:08.574829+0000","last_unstale":"2024-09-21T11:08:58.062966+0000","last_undegraded":"2024-09-21T11:08:58.062966+0000","last_fullsized":"2024-09-21T11:08:58.062966+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:15:57.943663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"346'149","reported_seq":1061,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.462681+0000","last_change":"2024-09-21T11:06:58.762593+0000","last_active":"2024-09-21T11:08:57.462681+0000","last_peered":"2024-09-21T11:08:57.462681+0000","last_clean":"2024-09-21T11:08:57.462681+0000","last_became_active":"2024-09-21T11:06:58.762117+0000","last_became_peered":"2024-09-21T11:06:58.762117+0000","last_unstale":"2024-09-21T11:08:57.462681+0000","last_undegraded":"2024-09-21T11:08:57.462681+0000","last_fullsized":"2024-09-21T11:08:57.462681+0000","mapping_epoch":1675,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1676,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:05:57.002981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:05:57.002981+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:58:47.258964+0000","scrub_duration":0.0017979249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":131,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:58.062753+0000","last_change":"2024-09-21T11:07:50.446812+0000","last_active":"2024-09-21T11:08:58.062753+0000","last_peered":"2024-09-21T11:08:58.062753+0000","last_clean":"2024-09-21T11:08:58.062753+0000","last_became_active":"2024-09-21T11:07:50.446435+0000","last_became_peered":"2024-09-21T11:07:50.446435+0000","last_unstale":"2024-09-21T11:08:58.062753+0000","last_undegraded":"2024-09-21T11:08:58.062753+0000","last_fullsized":"2024-09-21T11:08:58.062753+0000","mapping_epoch":1684,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1685,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:16:31.855319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2621,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:58.062868+0000","last_change":"2024-09-21T11:08:08.560254+0000","last_active":"2024-09-21T11:08:58.062868+0000","last_peered":"2024-09-21T11:08:58.062868+0000","last_clean":"2024-09-21T11:08:58.062868+0000","last_became_active":"2024-09-21T11:08:08.559925+0000","last_became_peered":"2024-09-21T11:08:08.559925+0000","last_unstale":"2024-09-21T11:08:58.062868+0000","last_undegraded":"2024-09-21T11:08:58.062868+0000","last_fullsized":"2024-09-21T11:08:58.062868+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:49:16.328026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498381,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":194,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.466129+0000","last_change":"2024-09-21T11:08:49.017651+0000","last_active":"2024-09-21T11:08:56.466129+0000","last_peered":"2024-09-21T11:08:56.466129+0000","last_clean":"2024-09-21T11:08:56.466129+0000","last_became_active":"2024-09-21T11:08:49.017325+0000","last_became_peered":"2024-09-21T11:08:49.017325+0000","last_unstale":"2024-09-21T11:08:56.466129+0000","last_undegraded":"2024-09-21T11:08:56.466129+0000","last_fullsized":"2024-09-21T11:08:56.466129+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:46:21.745857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"346'149","reported_seq":1058,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.465822+0000","last_change":"2024-09-21T11:08:49.015549+0000","last_active":"2024-09-21T11:08:56.465822+0000","last_peered":"2024-09-21T11:08:56.465822+0000","last_clean":"2024-09-21T11:08:56.465822+0000","last_became_active":"2024-09-21T11:08:49.015214+0000","last_became_peered":"2024-09-21T11:08:49.015214+0000","last_unstale":"2024-09-21T11:08:56.465822+0000","last_undegraded":"2024-09-21T11:08:56.465822+0000","last_fullsized":"2024-09-21T11:08:56.465822+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:48.918874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.918874+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:05:38.152071+0000","scrub_duration":0.001629134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":142,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:58.062802+0000","last_change":"2024-09-21T11:08:08.574895+0000","last_active":"2024-09-21T11:08:58.062802+0000","last_peered":"2024-09-21T11:08:58.062802+0000","last_clean":"2024-09-21T11:08:58.062802+0000","last_became_active":"2024-09-21T11:08:08.574606+0000","last_became_peered":"2024-09-21T11:08:08.574606+0000","last_unstale":"2024-09-21T11:08:58.062802+0000","last_undegraded":"2024-09-21T11:08:58.062802+0000","last_fullsized":"2024-09-21T11:08:58.062802+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:35:27.634442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1621,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.458019+0000","last_change":"2024-09-21T11:05:48.549704+0000","last_active":"2024-09-21T11:08:56.458019+0000","last_peered":"2024-09-21T11:08:56.458019+0000","last_clean":"2024-09-21T11:08:56.458019+0000","last_became_active":"2024-09-21T11:05:48.549374+0000","last_became_peered":"2024-09-21T11:05:48.549374+0000","last_unstale":"2024-09-21T11:08:56.458019+0000","last_undegraded":"2024-09-21T11:08:56.458019+0000","last_fullsized":"2024-09-21T11:08:56.458019+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:57.002097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":672,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.465933+0000","last_change":"2024-09-21T11:08:49.015428+0000","last_active":"2024-09-21T11:08:56.465933+0000","last_peered":"2024-09-21T11:08:56.465933+0000","last_clean":"2024-09-21T11:08:56.465933+0000","last_became_active":"2024-09-21T11:08:49.014725+0000","last_became_peered":"2024-09-21T11:08:49.014725+0000","last_unstale":"2024-09-21T11:08:56.465933+0000","last_undegraded":"2024-09-21T11:08:56.465933+0000","last_fullsized":"2024-09-21T11:08:56.465933+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:50:21.944099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":95,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"346'149","reported_seq":985,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.821900+0000","last_change":"2024-09-21T11:07:05.782919+0000","last_active":"2024-09-21T11:08:57.821900+0000","last_peered":"2024-09-21T11:08:57.821900+0000","last_clean":"2024-09-21T11:08:57.821900+0000","last_became_active":"2024-09-21T11:07:05.782486+0000","last_became_peered":"2024-09-21T11:07:05.782486+0000","last_unstale":"2024-09-21T11:08:57.821900+0000","last_undegraded":"2024-09-21T11:08:57.821900+0000","last_fullsized":"2024-09-21T11:08:57.821900+0000","mapping_epoch":1681,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1682,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:06:44.106093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:06:44.106093+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:58:14.998229+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.825547+0000","last_change":"2024-09-21T11:08:57.825277+0000","last_active":"2024-09-21T11:08:57.825547+0000","last_peered":"2024-09-21T11:08:57.825547+0000","last_clean":"2024-09-21T11:08:57.825547+0000","last_became_active":"2024-09-21T11:08:57.825021+0000","last_became_peered":"2024-09-21T11:08:57.825021+0000","last_unstale":"2024-09-21T11:08:57.825547+0000","last_undegraded":"2024-09-21T11:08:57.825547+0000","last_fullsized":"2024-09-21T11:08:57.825547+0000","mapping_epoch":1710,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1711,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:57:48.857063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2599,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:58.062638+0000","last_change":"2024-09-21T11:08:08.575089+0000","last_active":"2024-09-21T11:08:58.062638+0000","last_peered":"2024-09-21T11:08:58.062638+0000","last_clean":"2024-09-21T11:08:58.062638+0000","last_became_active":"2024-09-21T11:08:08.574703+0000","last_became_peered":"2024-09-21T11:08:08.574703+0000","last_unstale":"2024-09-21T11:08:58.062638+0000","last_undegraded":"2024-09-21T11:08:58.062638+0000","last_fullsized":"2024-09-21T11:08:58.062638+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:18:10.557195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":58,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.462088+0000","last_change":"2024-09-21T11:07:53.471395+0000","last_active":"2024-09-21T11:08:57.462088+0000","last_peered":"2024-09-21T11:08:57.462088+0000","last_clean":"2024-09-21T11:08:57.462088+0000","last_became_active":"2024-09-21T11:07:53.471078+0000","last_became_peered":"2024-09-21T11:07:53.471078+0000","last_unstale":"2024-09-21T11:08:57.462088+0000","last_undegraded":"2024-09-21T11:08:57.462088+0000","last_fullsized":"2024-09-21T11:08:57.462088+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:57:46.357558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000473385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"346'149","reported_seq":1279,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:58.062909+0000","last_change":"2024-09-21T11:08:08.558631+0000","last_active":"2024-09-21T11:08:58.062909+0000","last_peered":"2024-09-21T11:08:58.062909+0000","last_clean":"2024-09-21T11:08:58.062909+0000","last_became_active":"2024-09-21T11:08:08.558303+0000","last_became_peered":"2024-09-21T11:08:08.558303+0000","last_unstale":"2024-09-21T11:08:58.062909+0000","last_undegraded":"2024-09-21T11:08:58.062909+0000","last_fullsized":"2024-09-21T11:08:58.062909+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":3,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:50.671365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:50.671365+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:29:56.261115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":175,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.581667+0000","last_change":"2024-09-21T11:07:53.461508+0000","last_active":"2024-09-21T11:08:57.581667+0000","last_peered":"2024-09-21T11:08:57.581667+0000","last_clean":"2024-09-21T11:08:57.581667+0000","last_became_active":"2024-09-21T11:07:53.461184+0000","last_became_peered":"2024-09-21T11:07:53.461184+0000","last_unstale":"2024-09-21T11:08:57.581667+0000","last_undegraded":"2024-09-21T11:08:57.581667+0000","last_fullsized":"2024-09-21T11:08:57.581667+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:23:45.693130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":175,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.581913+0000","last_change":"2024-09-21T11:07:02.780287+0000","last_active":"2024-09-21T11:08:57.581913+0000","last_peered":"2024-09-21T11:08:57.581913+0000","last_clean":"2024-09-21T11:08:57.581913+0000","last_became_active":"2024-09-21T11:07:02.779916+0000","last_became_peered":"2024-09-21T11:07:02.779916+0000","last_unstale":"2024-09-21T11:08:57.581913+0000","last_undegraded":"2024-09-21T11:08:57.581913+0000","last_fullsized":"2024-09-21T11:08:57.581913+0000","mapping_epoch":1678,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:10.990224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":400,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.458752+0000","last_change":"2024-09-21T11:06:33.531901+0000","last_active":"2024-09-21T11:08:56.458752+0000","last_peered":"2024-09-21T11:08:56.458752+0000","last_clean":"2024-09-21T11:08:56.458752+0000","last_became_active":"2024-09-21T11:06:33.531562+0000","last_became_peered":"2024-09-21T11:06:33.531562+0000","last_unstale":"2024-09-21T11:08:56.458752+0000","last_undegraded":"2024-09-21T11:08:56.458752+0000","last_fullsized":"2024-09-21T11:08:56.458752+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:04.767469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034537800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":242,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.458103+0000","last_change":"2024-09-21T11:02:39.502922+0000","last_active":"2024-09-21T11:08:56.458103+0000","last_peered":"2024-09-21T11:08:56.458103+0000","last_clean":"2024-09-21T11:08:56.458103+0000","last_became_active":"2024-09-21T11:02:39.502633+0000","last_became_peered":"2024-09-21T11:02:39.502633+0000","last_unstale":"2024-09-21T11:08:56.458103+0000","last_undegraded":"2024-09-21T11:08:56.458103+0000","last_fullsized":"2024-09-21T11:08:56.458103+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:29:20.971522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":211,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.465061+0000","last_change":"2024-09-21T11:07:50.441026+0000","last_active":"2024-09-21T11:08:57.465061+0000","last_peered":"2024-09-21T11:08:57.465061+0000","last_clean":"2024-09-21T11:08:57.465061+0000","last_became_active":"2024-09-21T11:07:50.440650+0000","last_became_peered":"2024-09-21T11:07:50.440650+0000","last_unstale":"2024-09-21T11:08:57.465061+0000","last_undegraded":"2024-09-21T11:08:57.465061+0000","last_fullsized":"2024-09-21T11:08:57.465061+0000","mapping_epoch":1684,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1685,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:49:56.547116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":41,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.465992+0000","last_change":"2024-09-21T11:08:49.016227+0000","last_active":"2024-09-21T11:08:56.465992+0000","last_peered":"2024-09-21T11:08:56.465992+0000","last_clean":"2024-09-21T11:08:56.465992+0000","last_became_active":"2024-09-21T11:08:49.015010+0000","last_became_peered":"2024-09-21T11:08:49.015010+0000","last_unstale":"2024-09-21T11:08:56.465992+0000","last_undegraded":"2024-09-21T11:08:56.465992+0000","last_fullsized":"2024-09-21T11:08:56.465992+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:42.179141+0000","last_clean_scrub_stamp":"2024-09-21T09:53:42.179141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:36:50.932413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308344,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"346'149","reported_seq":913,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.822216+0000","last_change":"2024-09-21T11:06:58.762931+0000","last_active":"2024-09-21T11:08:57.822216+0000","last_peered":"2024-09-21T11:08:57.822216+0000","last_clean":"2024-09-21T11:08:57.822216+0000","last_became_active":"2024-09-21T11:06:58.762397+0000","last_became_peered":"2024-09-21T11:06:58.762397+0000","last_unstale":"2024-09-21T11:08:57.822216+0000","last_undegraded":"2024-09-21T11:08:57.822216+0000","last_fullsized":"2024-09-21T11:08:57.822216+0000","mapping_epoch":1675,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1676,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:05:58.110003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:05:58.110003+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:59:21.260999+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"346'149","reported_seq":1006,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.465898+0000","last_change":"2024-09-21T11:08:49.015707+0000","last_active":"2024-09-21T11:08:56.465898+0000","last_peered":"2024-09-21T11:08:56.465898+0000","last_clean":"2024-09-21T11:08:56.465898+0000","last_became_active":"2024-09-21T11:08:49.015083+0000","last_became_peered":"2024-09-21T11:08:49.015083+0000","last_unstale":"2024-09-21T11:08:56.465898+0000","last_undegraded":"2024-09-21T11:08:56.465898+0000","last_fullsized":"2024-09-21T11:08:56.465898+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:49:42.008235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:49:42.008235+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:23:59.402474+0000","scrub_duration":0.011129834999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2736,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:58.063053+0000","last_change":"2024-09-21T11:01:46.484382+0000","last_active":"2024-09-21T11:08:58.063053+0000","last_peered":"2024-09-21T11:08:58.063053+0000","last_clean":"2024-09-21T11:08:58.063053+0000","last_became_active":"2024-09-21T11:01:46.484116+0000","last_became_peered":"2024-09-21T11:01:46.484116+0000","last_unstale":"2024-09-21T11:08:58.063053+0000","last_undegraded":"2024-09-21T11:08:58.063053+0000","last_fullsized":"2024-09-21T11:08:58.063053+0000","mapping_epoch":1578,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1579,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:48.924838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":110,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.464739+0000","last_change":"2024-09-21T11:06:55.760878+0000","last_active":"2024-09-21T11:08:57.464739+0000","last_peered":"2024-09-21T11:08:57.464739+0000","last_clean":"2024-09-21T11:08:57.464739+0000","last_became_active":"2024-09-21T11:06:55.760619+0000","last_became_peered":"2024-09-21T11:06:55.760619+0000","last_unstale":"2024-09-21T11:08:57.464739+0000","last_undegraded":"2024-09-21T11:08:57.464739+0000","last_fullsized":"2024-09-21T11:08:57.464739+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:05:47.078808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"346'149","reported_seq":1284,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:58.062799+0000","last_change":"2024-09-21T11:08:08.562618+0000","last_active":"2024-09-21T11:08:58.062799+0000","last_peered":"2024-09-21T11:08:58.062799+0000","last_clean":"2024-09-21T11:08:58.062799+0000","last_became_active":"2024-09-21T11:08:08.562125+0000","last_became_peered":"2024-09-21T11:08:08.562125+0000","last_unstale":"2024-09-21T11:08:58.062799+0000","last_undegraded":"2024-09-21T11:08:58.062799+0000","last_fullsized":"2024-09-21T11:08:58.062799+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:48.663349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:48.663349+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:45.603439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1650,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.458071+0000","last_change":"2024-09-21T11:06:33.535747+0000","last_active":"2024-09-21T11:08:56.458071+0000","last_peered":"2024-09-21T11:08:56.458071+0000","last_clean":"2024-09-21T11:08:56.458071+0000","last_became_active":"2024-09-21T11:06:33.535181+0000","last_became_peered":"2024-09-21T11:06:33.535181+0000","last_unstale":"2024-09-21T11:08:56.458071+0000","last_undegraded":"2024-09-21T11:08:56.458071+0000","last_fullsized":"2024-09-21T11:08:56.458071+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:52.206420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":53,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.462398+0000","last_change":"2024-09-21T11:07:53.468531+0000","last_active":"2024-09-21T11:08:57.462398+0000","last_peered":"2024-09-21T11:08:57.462398+0000","last_clean":"2024-09-21T11:08:57.462398+0000","last_became_active":"2024-09-21T11:07:53.468225+0000","last_became_peered":"2024-09-21T11:07:53.468225+0000","last_unstale":"2024-09-21T11:08:57.462398+0000","last_undegraded":"2024-09-21T11:08:57.462398+0000","last_fullsized":"2024-09-21T11:08:57.462398+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:36:06.139781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"346'149","reported_seq":1160,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.465777+0000","last_change":"2024-09-21T11:08:49.016904+0000","last_active":"2024-09-21T11:08:56.465777+0000","last_peered":"2024-09-21T11:08:56.465777+0000","last_clean":"2024-09-21T11:08:56.465777+0000","last_became_active":"2024-09-21T11:08:49.016201+0000","last_became_peered":"2024-09-21T11:08:49.016201+0000","last_unstale":"2024-09-21T11:08:56.465777+0000","last_undegraded":"2024-09-21T11:08:56.465777+0000","last_fullsized":"2024-09-21T11:08:56.465777+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:07:20.603775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:07:20.603775+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:20:50.045499+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":53,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.462351+0000","last_change":"2024-09-21T11:07:53.468717+0000","last_active":"2024-09-21T11:08:57.462351+0000","last_peered":"2024-09-21T11:08:57.462351+0000","last_clean":"2024-09-21T11:08:57.462351+0000","last_became_active":"2024-09-21T11:07:53.468381+0000","last_became_peered":"2024-09-21T11:07:53.468381+0000","last_unstale":"2024-09-21T11:08:57.462351+0000","last_undegraded":"2024-09-21T11:08:57.462351+0000","last_fullsized":"2024-09-21T11:08:57.462351+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:24.353490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1679,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.462493+0000","last_change":"2024-09-21T11:06:33.533209+0000","last_active":"2024-09-21T11:08:57.462493+0000","last_peered":"2024-09-21T11:08:57.462493+0000","last_clean":"2024-09-21T11:08:57.462493+0000","last_became_active":"2024-09-21T11:06:33.532865+0000","last_became_peered":"2024-09-21T11:06:33.532865+0000","last_unstale":"2024-09-21T11:08:57.462493+0000","last_undegraded":"2024-09-21T11:08:57.462493+0000","last_fullsized":"2024-09-21T11:08:57.462493+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:38:03.274980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"346'149","reported_seq":1209,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.822465+0000","last_change":"2024-09-21T11:07:56.474339+0000","last_active":"2024-09-21T11:08:57.822465+0000","last_peered":"2024-09-21T11:08:57.822465+0000","last_clean":"2024-09-21T11:08:57.822465+0000","last_became_active":"2024-09-21T11:07:56.473980+0000","last_became_peered":"2024-09-21T11:07:56.473980+0000","last_unstale":"2024-09-21T11:08:57.822465+0000","last_undegraded":"2024-09-21T11:08:57.822465+0000","last_fullsized":"2024-09-21T11:08:57.822465+0000","mapping_epoch":1690,"log_start":"0'0","ondisk_log_start":"0'0","created":1419,"last_epoch_clean":1691,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T10:54:17.418646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T10:54:17.418646+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:09:30.940593+0000","scrub_duration":0.00185648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":885,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.458465+0000","last_change":"2024-09-21T11:02:39.501678+0000","last_active":"2024-09-21T11:08:56.458465+0000","last_peered":"2024-09-21T11:08:56.458465+0000","last_clean":"2024-09-21T11:08:56.458465+0000","last_became_active":"2024-09-21T11:02:39.497766+0000","last_became_peered":"2024-09-21T11:02:39.497766+0000","last_unstale":"2024-09-21T11:08:56.458465+0000","last_undegraded":"2024-09-21T11:08:56.458465+0000","last_fullsized":"2024-09-21T11:08:56.458465+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:09:25.807702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":229,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.465228+0000","last_change":"2024-09-21T11:05:48.649119+0000","last_active":"2024-09-21T11:08:57.465228+0000","last_peered":"2024-09-21T11:08:57.465228+0000","last_clean":"2024-09-21T11:08:57.465228+0000","last_became_active":"2024-09-21T11:05:48.550467+0000","last_became_peered":"2024-09-21T11:05:48.550467+0000","last_unstale":"2024-09-21T11:08:57.465228+0000","last_undegraded":"2024-09-21T11:08:57.465228+0000","last_fullsized":"2024-09-21T11:08:57.465228+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:46:06.305352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"346'149","reported_seq":970,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.462341+0000","last_change":"2024-09-21T11:07:07.942671+0000","last_active":"2024-09-21T11:08:57.462341+0000","last_peered":"2024-09-21T11:08:57.462341+0000","last_clean":"2024-09-21T11:08:57.462341+0000","last_became_active":"2024-09-21T11:07:05.782775+0000","last_became_peered":"2024-09-21T11:07:05.782775+0000","last_unstale":"2024-09-21T11:08:57.462341+0000","last_undegraded":"2024-09-21T11:08:57.462341+0000","last_fullsized":"2024-09-21T11:08:57.462341+0000","mapping_epoch":1681,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1682,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:07:07.942557+0000","last_deep_scrub":"346'149","last_deep_scrub_stamp":"2024-09-21T11:07:00.489593+0000","last_clean_scrub_stamp":"2024-09-21T11:07:07.942557+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:25:44.786324+0000","scrub_duration":0.0019991430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":161,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.582243+0000","last_change":"2024-09-21T11:05:48.622817+0000","last_active":"2024-09-21T11:08:57.582243+0000","last_peered":"2024-09-21T11:08:57.582243+0000","last_clean":"2024-09-21T11:08:57.582243+0000","last_became_active":"2024-09-21T11:05:48.559669+0000","last_became_peered":"2024-09-21T11:05:48.559669+0000","last_unstale":"2024-09-21T11:08:57.582243+0000","last_undegraded":"2024-09-21T11:08:57.582243+0000","last_fullsized":"2024-09-21T11:08:57.582243+0000","mapping_epoch":1636,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:25:03.540564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":172,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.466071+0000","last_change":"2024-09-21T11:08:49.017101+0000","last_active":"2024-09-21T11:08:56.466071+0000","last_peered":"2024-09-21T11:08:56.466071+0000","last_clean":"2024-09-21T11:08:56.466071+0000","last_became_active":"2024-09-21T11:08:49.016479+0000","last_became_peered":"2024-09-21T11:08:49.016479+0000","last_unstale":"2024-09-21T11:08:56.466071+0000","last_undegraded":"2024-09-21T11:08:56.466071+0000","last_fullsized":"2024-09-21T11:08:56.466071+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:54.839925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"346'149","reported_seq":1045,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.462808+0000","last_change":"2024-09-21T11:07:53.466166+0000","last_active":"2024-09-21T11:08:57.462808+0000","last_peered":"2024-09-21T11:08:57.462808+0000","last_clean":"2024-09-21T11:08:57.462808+0000","last_became_active":"2024-09-21T11:07:53.465779+0000","last_became_peered":"2024-09-21T11:07:53.465779+0000","last_unstale":"2024-09-21T11:08:57.462808+0000","last_undegraded":"2024-09-21T11:08:57.462808+0000","last_fullsized":"2024-09-21T11:08:57.462808+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:05:59.940755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:05:59.940755+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:39:39.406723+0000","scrub_duration":0.0034290169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":239,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.462678+0000","last_change":"2024-09-21T11:06:55.757992+0000","last_active":"2024-09-21T11:08:57.462678+0000","last_peered":"2024-09-21T11:08:57.462678+0000","last_clean":"2024-09-21T11:08:57.462678+0000","last_became_active":"2024-09-21T11:06:55.757384+0000","last_became_peered":"2024-09-21T11:06:55.757384+0000","last_unstale":"2024-09-21T11:08:57.462678+0000","last_undegraded":"2024-09-21T11:08:57.462678+0000","last_fullsized":"2024-09-21T11:08:57.462678+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:59:44.041307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":472,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.822997+0000","last_change":"2024-09-21T11:07:50.450388+0000","last_active":"2024-09-21T11:08:57.822997+0000","last_peered":"2024-09-21T11:08:57.822997+0000","last_clean":"2024-09-21T11:08:57.822997+0000","last_became_active":"2024-09-21T11:07:50.443015+0000","last_became_peered":"2024-09-21T11:07:50.443015+0000","last_unstale":"2024-09-21T11:08:57.822997+0000","last_undegraded":"2024-09-21T11:08:57.822997+0000","last_fullsized":"2024-09-21T11:08:57.822997+0000","mapping_epoch":1684,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1685,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:25:51.176230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"346'149","reported_seq":1017,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.465002+0000","last_change":"2024-09-21T11:06:10.594793+0000","last_active":"2024-09-21T11:08:57.465002+0000","last_peered":"2024-09-21T11:08:57.465002+0000","last_clean":"2024-09-21T11:08:57.465002+0000","last_became_active":"2024-09-21T11:06:10.594289+0000","last_became_peered":"2024-09-21T11:06:10.594289+0000","last_unstale":"2024-09-21T11:08:57.465002+0000","last_undegraded":"2024-09-21T11:08:57.465002+0000","last_fullsized":"2024-09-21T11:08:57.465002+0000","mapping_epoch":1651,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1652,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:54.708658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:54.708658+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:07.379791+0000","scrub_duration":0.0021035160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1715,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.462600+0000","last_change":"2024-09-21T11:06:40.705586+0000","last_active":"2024-09-21T11:08:57.462600+0000","last_peered":"2024-09-21T11:08:57.462600+0000","last_clean":"2024-09-21T11:08:57.462600+0000","last_became_active":"2024-09-21T11:06:40.705193+0000","last_became_peered":"2024-09-21T11:06:40.705193+0000","last_unstale":"2024-09-21T11:08:57.462600+0000","last_undegraded":"2024-09-21T11:08:57.462600+0000","last_fullsized":"2024-09-21T11:08:57.462600+0000","mapping_epoch":1665,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1666,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:25:37.486777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":67,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.822908+0000","last_change":"2024-09-21T11:07:02.784493+0000","last_active":"2024-09-21T11:08:57.822908+0000","last_peered":"2024-09-21T11:08:57.822908+0000","last_clean":"2024-09-21T11:08:57.822908+0000","last_became_active":"2024-09-21T11:07:02.781586+0000","last_became_peered":"2024-09-21T11:07:02.781586+0000","last_unstale":"2024-09-21T11:08:57.822908+0000","last_undegraded":"2024-09-21T11:08:57.822908+0000","last_fullsized":"2024-09-21T11:08:57.822908+0000","mapping_epoch":1678,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:06:42.604214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"346'149","reported_seq":887,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.822769+0000","last_change":"2024-09-21T11:06:58.765920+0000","last_active":"2024-09-21T11:08:57.822769+0000","last_peered":"2024-09-21T11:08:57.822769+0000","last_clean":"2024-09-21T11:08:57.822769+0000","last_became_active":"2024-09-21T11:06:58.765490+0000","last_became_peered":"2024-09-21T11:06:58.765490+0000","last_unstale":"2024-09-21T11:08:57.822769+0000","last_undegraded":"2024-09-21T11:08:57.822769+0000","last_fullsized":"2024-09-21T11:08:57.822769+0000","mapping_epoch":1675,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1676,"parent":"0.0","parent_split_bits":4,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:53.896066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:53.896066+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:51.227905+0000","scrub_duration":0.001699578,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":37,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.466295+0000","last_change":"2024-09-21T11:08:49.016220+0000","last_active":"2024-09-21T11:08:56.466295+0000","last_peered":"2024-09-21T11:08:56.466295+0000","last_clean":"2024-09-21T11:08:56.466295+0000","last_became_active":"2024-09-21T11:08:49.015909+0000","last_became_peered":"2024-09-21T11:08:49.015909+0000","last_unstale":"2024-09-21T11:08:56.466295+0000","last_undegraded":"2024-09-21T11:08:56.466295+0000","last_fullsized":"2024-09-21T11:08:56.466295+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:10:28.511316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":54,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.823015+0000","last_change":"2024-09-21T11:06:55.771951+0000","last_active":"2024-09-21T11:08:57.823015+0000","last_peered":"2024-09-21T11:08:57.823015+0000","last_clean":"2024-09-21T11:08:57.823015+0000","last_became_active":"2024-09-21T11:06:55.763633+0000","last_became_peered":"2024-09-21T11:06:55.763633+0000","last_unstale":"2024-09-21T11:08:57.823015+0000","last_undegraded":"2024-09-21T11:08:57.823015+0000","last_fullsized":"2024-09-21T11:08:57.823015+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:59:18.987831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"346'149","reported_seq":962,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.466098+0000","last_change":"2024-09-21T11:07:53.460517+0000","last_active":"2024-09-21T11:08:56.466098+0000","last_peered":"2024-09-21T11:08:56.466098+0000","last_clean":"2024-09-21T11:08:56.466098+0000","last_became_active":"2024-09-21T11:07:53.460095+0000","last_became_peered":"2024-09-21T11:07:53.460095+0000","last_unstale":"2024-09-21T11:08:56.466098+0000","last_undegraded":"2024-09-21T11:08:56.466098+0000","last_fullsized":"2024-09-21T11:08:56.466098+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:06:37.734784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:06:37.734784+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:21:30.091664+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":1772,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.457362+0000","last_change":"2024-09-21T11:07:50.454445+0000","last_active":"2024-09-21T11:08:56.457362+0000","last_peered":"2024-09-21T11:08:56.457362+0000","last_clean":"2024-09-21T11:08:56.457362+0000","last_became_active":"2024-09-21T11:07:50.445182+0000","last_became_peered":"2024-09-21T11:07:50.445182+0000","last_unstale":"2024-09-21T11:08:56.457362+0000","last_undegraded":"2024-09-21T11:08:56.457362+0000","last_fullsized":"2024-09-21T11:08:56.457362+0000","mapping_epoch":1684,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1685,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:21:52.363073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":117,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.823461+0000","last_change":"2024-09-21T11:08:57.823171+0000","last_active":"2024-09-21T11:08:57.823461+0000","last_peered":"2024-09-21T11:08:57.823461+0000","last_clean":"2024-09-21T11:08:57.823461+0000","last_became_active":"2024-09-21T11:08:57.822855+0000","last_became_peered":"2024-09-21T11:08:57.822855+0000","last_unstale":"2024-09-21T11:08:57.823461+0000","last_undegraded":"2024-09-21T11:08:57.823461+0000","last_fullsized":"2024-09-21T11:08:57.823461+0000","mapping_epoch":1710,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1711,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:09:31.574671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"346'149","reported_seq":1284,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:58.062792+0000","last_change":"2024-09-21T11:08:08.558636+0000","last_active":"2024-09-21T11:08:58.062792+0000","last_peered":"2024-09-21T11:08:58.062792+0000","last_clean":"2024-09-21T11:08:58.062792+0000","last_became_active":"2024-09-21T11:08:08.558332+0000","last_became_peered":"2024-09-21T11:08:08.558332+0000","last_unstale":"2024-09-21T11:08:58.062792+0000","last_undegraded":"2024-09-21T11:08:58.062792+0000","last_fullsized":"2024-09-21T11:08:58.062792+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:54.706428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:54.706428+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:01:04.548955+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":425,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.458983+0000","last_change":"2024-09-21T11:02:39.471842+0000","last_active":"2024-09-21T11:08:56.458983+0000","last_peered":"2024-09-21T11:08:56.458983+0000","last_clean":"2024-09-21T11:08:56.458983+0000","last_became_active":"2024-09-21T11:02:39.471339+0000","last_became_peered":"2024-09-21T11:02:39.471339+0000","last_unstale":"2024-09-21T11:08:56.458983+0000","last_undegraded":"2024-09-21T11:08:56.458983+0000","last_fullsized":"2024-09-21T11:08:56.458983+0000","mapping_epoch":1590,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:53:53.996670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":168,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.582570+0000","last_change":"2024-09-21T11:06:55.762258+0000","last_active":"2024-09-21T11:08:57.582570+0000","last_peered":"2024-09-21T11:08:57.582570+0000","last_clean":"2024-09-21T11:08:57.582570+0000","last_became_active":"2024-09-21T11:06:55.761824+0000","last_became_peered":"2024-09-21T11:06:55.761824+0000","last_unstale":"2024-09-21T11:08:57.582570+0000","last_undegraded":"2024-09-21T11:08:57.582570+0000","last_fullsized":"2024-09-21T11:08:57.582570+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:43:09.930665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"346'149","reported_seq":876,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.465828+0000","last_change":"2024-09-21T11:08:49.019809+0000","last_active":"2024-09-21T11:08:56.465828+0000","last_peered":"2024-09-21T11:08:56.465828+0000","last_clean":"2024-09-21T11:08:56.465828+0000","last_became_active":"2024-09-21T11:08:49.019265+0000","last_became_peered":"2024-09-21T11:08:49.019265+0000","last_unstale":"2024-09-21T11:08:56.465828+0000","last_undegraded":"2024-09-21T11:08:56.465828+0000","last_fullsized":"2024-09-21T11:08:56.465828+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:06:00.073278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:06:00.073278+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:47:43.307376+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":180,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.582127+0000","last_change":"2024-09-21T11:06:55.762794+0000","last_active":"2024-09-21T11:08:57.582127+0000","last_peered":"2024-09-21T11:08:57.582127+0000","last_clean":"2024-09-21T11:08:57.582127+0000","last_became_active":"2024-09-21T11:06:55.760109+0000","last_became_peered":"2024-09-21T11:06:55.760109+0000","last_unstale":"2024-09-21T11:08:57.582127+0000","last_undegraded":"2024-09-21T11:08:57.582127+0000","last_fullsized":"2024-09-21T11:08:57.582127+0000","mapping_epoch":1672,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1673,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:41:23.127841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1679,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.466073+0000","last_change":"2024-09-21T11:08:49.016929+0000","last_active":"2024-09-21T11:08:56.466073+0000","last_peered":"2024-09-21T11:08:56.466073+0000","last_clean":"2024-09-21T11:08:56.466073+0000","last_became_active":"2024-09-21T11:08:49.016610+0000","last_became_peered":"2024-09-21T11:08:49.016610+0000","last_unstale":"2024-09-21T11:08:56.466073+0000","last_undegraded":"2024-09-21T11:08:56.466073+0000","last_fullsized":"2024-09-21T11:08:56.466073+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:00:04.990132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"346'149","reported_seq":1118,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.581654+0000","last_change":"2024-09-21T11:07:53.469227+0000","last_active":"2024-09-21T11:08:57.581654+0000","last_peered":"2024-09-21T11:08:57.581654+0000","last_clean":"2024-09-21T11:08:57.581654+0000","last_became_active":"2024-09-21T11:07:53.468868+0000","last_became_peered":"2024-09-21T11:07:53.468868+0000","last_unstale":"2024-09-21T11:08:57.581654+0000","last_undegraded":"2024-09-21T11:08:57.581654+0000","last_fullsized":"2024-09-21T11:08:57.581654+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":1573,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'149","last_scrub_stamp":"2024-09-21T11:01:47.955189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:06.099351+0000","last_clean_scrub_stamp":"2024-09-21T11:01:47.955189+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:45:44.145119+0000","scrub_duration":0.0019884339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":455,"reported_epoch":1710,"state":"active+clean","last_fresh":"2024-09-21T11:08:56.459032+0000","last_change":"2024-09-21T11:07:53.474774+0000","last_active":"2024-09-21T11:08:56.459032+0000","last_peered":"2024-09-21T11:08:56.459032+0000","last_clean":"2024-09-21T11:08:56.459032+0000","last_became_active":"2024-09-21T11:07:53.465568+0000","last_became_peered":"2024-09-21T11:07:53.465568+0000","last_unstale":"2024-09-21T11:08:56.459032+0000","last_undegraded":"2024-09-21T11:08:56.459032+0000","last_fullsized":"2024-09-21T11:08:56.459032+0000","mapping_epoch":1687,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1688,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:52.601790+0000","last_clean_scrub_stamp":"2024-09-21T09:54:52.601790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:17.170792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":170,"num_keys_recovered":50,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2733,"reported_epoch":1711,"state":"active+clean","last_fresh":"2024-09-21T11:08:57.465312+0000","last_change":"2024-09-21T11:06:33.530305+0000","last_active":"2024-09-21T11:08:57.465312+0000","last_peered":"2024-09-21T11:08:57.465312+0000","last_clean":"2024-09-21T11:08:57.465312+0000","last_became_active":"2024-09-21T11:06:33.529962+0000","last_became_peered":"2024-09-21T11:06:33.529962+0000","last_unstale":"2024-09-21T11:08:57.465312+0000","last_undegraded":"2024-09-21T11:08:57.465312+0000","last_fullsized":"2024-09-21T11:08:57.465312+0000","mapping_epoch":1658,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1659,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:54:53.603549+0000","last_clean_scrub_stamp":"2024-09-21T09:54:53.603549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:41.527340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3129,"ondisk_log_size":3129,"up":63,"acting":63,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":95,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":19674,"num_keys_recovered":50,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3602,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234907,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1648080,"kb_used_data":10560,"kb_used_omap":1,"kb_used_meta":1637502,"kb_avail":92097072,"statfs":{"total":95995035648,"available":94307401728,"internally_reserved":0,"allocated":10813440,"data_stored":6001606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":1676802507},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1589,"seq":6824703033429,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":333256,"kb_used_data":10872,"kb_used_omap":128,"kb_used_meta":322239,"kb_avail":93411896,"statfs":{"total":95995035648,"available":95653781504,"internally_reserved":0,"allocated":11132928,"data_stored":6329159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":131575,"internal_metadata":329973257},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":1597,"seq":6859062771784,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":50820,"kb_used_data":10804,"kb_used_omap":101,"kb_used_meta":39898,"kb_avail":93694332,"statfs":{"total":95995035648,"available":95942995968,"internally_reserved":0,"allocated":11063296,"data_stored":6259993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103790,"internal_metadata":40856210},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019816,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1371148,"kb_used_data":10556,"kb_used_omap":1,"kb_used_meta":1360574,"kb_avail":92374004,"statfs":{"total":95995035648,"available":94590980096,"internally_reserved":0,"allocated":10809344,"data_stored":6001536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1393228230},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":692,"seq":2972117369561,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1449528,"kb_used_data":11116,"kb_used_omap":69,"kb_used_meta":1438330,"kb_avail":92295624,"statfs":{"total":95995035648,"available":94510718976,"internally_reserved":0,"allocated":11382784,"data_stored":6587104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71550,"internal_metadata":1472850050},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":680,"seq":2920577762016,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87296,"kb_used_data":10800,"kb_used_omap":76,"kb_used_meta":76403,"kb_avail":93657856,"statfs":{"total":95995035648,"available":95905644544,"internally_reserved":0,"allocated":11059200,"data_stored":6259950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78585,"internal_metadata":78236935},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":674,"seq":2894807958247,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":801140,"kb_used_data":10536,"kb_used_omap":119,"kb_used_meta":790472,"kb_avail":92944012,"statfs":{"total":95995035648,"available":95174668288,"internally_reserved":0,"allocated":10788864,"data_stored":5996665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":121896,"internal_metadata":809444312},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":666,"seq":2860448219885,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":686840,"kb_used_data":10860,"kb_used_omap":76,"kb_used_meta":675891,"kb_avail":93058312,"statfs":{"total":95995035648,"available":95291711488,"internally_reserved":0,"allocated":11120640,"data_stored":6324930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78716,"internal_metadata":692112516},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2415,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1187,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T11:09:36.569 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T11:09:36.569 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cls/test_cls_rbd.sh) on smithi070 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2024-09-21T11:09:36.738 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=943fb8f39010442ea2d1cf42991d56a6 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cls/test_cls_rbd.sh) on smithi070 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2024-09-21T11:09:36.741 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-21T11:09:36.753 INFO:tasks.cephadm:Teardown begin 2024-09-21T11:09:36.753 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T11:09:36.785 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T11:09:36.831 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-21T11:09:36.832 DEBUG:teuthology.orchestra.run.smithi070:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T11:09:36.848 DEBUG:teuthology.orchestra.run.smithi160:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T11:09:36.869 INFO:tasks.cephadm:Stopping all daemons... 2024-09-21T11:09:36.870 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-21T11:09:36.870 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mon.a 2024-09-21T11:09:37.214 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:36 smithi070 systemd[1]: Stopping Ceph mon.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T11:09:37.627 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:09:37 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:11:09:37] ENGINE Bus STOPPING 2024-09-21T11:09:37.627 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:37 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a[107362]: 2024-09-21T11:09:37.213+0000 7fad4fc70640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T11:09:37.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:37 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a[107362]: 2024-09-21T11:09:37.213+0000 7fad4fc70640 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-09-21T11:09:37.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:37 smithi070 podman[446631]: 2024-09-21 11:09:37.419174182 +0000 UTC m=+0.328663248 container died 4a7b667a403311f40b95be0411d7385e1161f266e1871482d304082744d55eed (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-21T11:09:37.914 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:09:37 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:11:09:37] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T11:09:37.914 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:09:37 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:11:09:37] ENGINE Bus STOPPED 2024-09-21T11:09:37.914 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:09:37 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:11:09:37] ENGINE Bus STARTING 2024-09-21T11:09:37.914 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:09:37 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:11:09:37] ENGINE Serving on http://:::9283 2024-09-21T11:09:37.915 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:09:37 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:11:09:37] ENGINE Bus STARTED 2024-09-21T11:09:38.216 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:37 smithi070 podman[446631]: 2024-09-21 11:09:37.912786858 +0000 UTC m=+0.822275956 container cleanup 4a7b667a403311f40b95be0411d7385e1161f266e1871482d304082744d55eed (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-21T11:09:38.216 INFO:journalctl@ceph.mon.a.smithi070.stdout:Sep 21 11:09:37 smithi070 bash[446631]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-a 2024-09-21T11:09:38.336 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mon.a.service' 2024-09-21T11:09:38.388 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T11:09:38.389 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-21T11:09:38.389 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-21T11:09:38.389 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mon.c 2024-09-21T11:09:38.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:38 smithi070 systemd[1]: Stopping Ceph mon.c for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T11:09:39.071 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:09:38 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:11:09:38] ENGINE Bus STOPPING 2024-09-21T11:09:39.071 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:09:38 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:11:09:38] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T11:09:39.071 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:09:38 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:11:09:38] ENGINE Bus STOPPED 2024-09-21T11:09:39.071 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:09:38 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:11:09:38] ENGINE Bus STARTING 2024-09-21T11:09:39.071 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:09:38 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:11:09:38] ENGINE Serving on http://:::9283 2024-09-21T11:09:39.072 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:09:38 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: [21/Sep/2024:11:09:38] ENGINE Bus STARTED 2024-09-21T11:09:39.072 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:38 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-c[112116]: 2024-09-21T11:09:38.685+0000 7f6a820a2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T11:09:39.072 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:38 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-c[112116]: 2024-09-21T11:09:38.685+0000 7f6a820a2640 -1 mon.c@2(peon) e4 *** Got Signal Terminated *** 2024-09-21T11:09:39.072 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:38 smithi070 podman[446777]: 2024-09-21 11:09:38.789073793 +0000 UTC m=+0.220436302 container died e154380b80a2464f25105dce2a8896a64240f04aa62fcb8628f82ad9410e8275 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-21T11:09:39.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:39 smithi070 podman[446777]: 2024-09-21 11:09:39.284674224 +0000 UTC m=+0.716036724 container cleanup e154380b80a2464f25105dce2a8896a64240f04aa62fcb8628f82ad9410e8275 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T11:09:39.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Sep 21 11:09:39 smithi070 bash[446777]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-c 2024-09-21T11:09:39.657 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mon.c.service' 2024-09-21T11:09:39.690 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T11:09:39.690 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-21T11:09:39.690 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-21T11:09:39.691 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mon.b 2024-09-21T11:09:39.989 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:39 smithi160 systemd[1]: Stopping Ceph mon.b for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T11:09:40.264 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:39 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-b[101856]: 2024-09-21T11:09:39.985+0000 7fdce6f16640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T11:09:40.264 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:39 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-b[101856]: 2024-09-21T11:09:39.985+0000 7fdce6f16640 -1 mon.b@1(peon) e4 *** Got Signal Terminated *** 2024-09-21T11:09:40.547 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:40 smithi160 podman[260622]: 2024-09-21 11:09:40.263072802 +0000 UTC m=+0.374466954 container died 86e88af21f363c6d8bfd177fd8ebb63fb182a6bd6d5acfa40ef9d00474af8514 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=main-62da54b, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-21T11:09:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:40 smithi160 podman[260622]: 2024-09-21 11:09:40.768746284 +0000 UTC m=+0.880140437 container cleanup 86e88af21f363c6d8bfd177fd8ebb63fb182a6bd6d5acfa40ef9d00474af8514 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-b, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=main-62da54b, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-21T11:09:40.841 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 21 11:09:40 smithi160 bash[260622]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mon-b 2024-09-21T11:09:41.207 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mon.b.service' 2024-09-21T11:09:41.245 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T11:09:41.245 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-21T11:09:41.245 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-21T11:09:41.245 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mgr.y 2024-09-21T11:09:41.558 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:09:41 smithi070 systemd[1]: Stopping Ceph mgr.y for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T11:09:41.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:09:41 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T11:09:41.555+0000 7fd0e6fbb640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T11:09:41.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:09:41 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y[89775]: 2024-09-21T11:09:41.555+0000 7fd0e6fbb640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-21T11:09:41.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:09:41 smithi070 podman[446921]: 2024-09-21 11:09:41.59872862 +0000 UTC m=+0.155127375 container died e3d390893cff7dfefe7f03b68d7aea1d181d3d58a08908234a15f82c29160fc9 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=main-62da54b, ceph=True) 2024-09-21T11:09:42.350 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:09:42 smithi070 podman[446921]: 2024-09-21 11:09:42.097182734 +0000 UTC m=+0.653581490 container cleanup e3d390893cff7dfefe7f03b68d7aea1d181d3d58a08908234a15f82c29160fc9 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-21T11:09:42.350 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Sep 21 11:09:42 smithi070 bash[446921]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-y 2024-09-21T11:09:42.474 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mgr.y.service' 2024-09-21T11:09:42.508 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T11:09:42.508 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-21T11:09:42.508 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-21T11:09:42.509 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mgr.x 2024-09-21T11:09:42.833 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 11:09:42 smithi160 systemd[1]: Stopping Ceph mgr.x for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T11:09:43.091 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 11:09:42 smithi160 podman[260755]: 2024-09-21 11:09:42.831753307 +0000 UTC m=+0.133785485 container died d952652a9683dd915b748da0bf7687e8f6d5e67da3c90433a88b3a5358afa4c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-62da54b) 2024-09-21T11:09:43.591 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 11:09:43 smithi160 podman[260755]: 2024-09-21 11:09:43.357225267 +0000 UTC m=+0.659257438 container cleanup d952652a9683dd915b748da0bf7687e8f6d5e67da3c90433a88b3a5358afa4c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x, GIT_CLEAN=True, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-21T11:09:43.592 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 21 11:09:43 smithi160 bash[260755]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-mgr-x 2024-09-21T11:09:43.851 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@mgr.x.service' 2024-09-21T11:09:43.886 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T11:09:43.887 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-21T11:09:43.887 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-21T11:09:43.887 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.0 2024-09-21T11:09:44.188 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 11:09:43 smithi070 systemd[1]: Stopping Ceph osd.0 for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T11:09:44.627 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 11:09:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0[124972]: 2024-09-21T11:09:44.185+0000 7fa7ac8aa640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T11:09:44.627 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 11:09:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0[124972]: 2024-09-21T11:09:44.185+0000 7fa7ac8aa640 -1 osd.0 1711 *** Got signal Terminated *** 2024-09-21T11:09:44.627 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 11:09:44 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0[124972]: 2024-09-21T11:09:44.185+0000 7fa7ac8aa640 -1 osd.0 1711 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T11:09:49.877 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 11:09:49 smithi070 podman[447051]: 2024-09-21 11:09:49.591432656 +0000 UTC m=+5.510782951 container died eedacb586f7cd3a818326f1341db47909276349880368abce57bf2152e01a536 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, RELEASE=main-62da54b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-21T11:09:50.320 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 11:09:50 smithi070 podman[447051]: 2024-09-21 11:09:50.059388436 +0000 UTC m=+5.978738721 container cleanup eedacb586f7cd3a818326f1341db47909276349880368abce57bf2152e01a536 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-21T11:09:50.320 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 11:09:50 smithi070 bash[447051]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0 2024-09-21T11:09:50.627 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 11:09:50 smithi070 podman[447070]: 2024-09-21 11:09:50.319053526 +0000 UTC m=+0.723640281 container remove eedacb586f7cd3a818326f1341db47909276349880368abce57bf2152e01a536 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-21T11:09:50.890 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 11:09:50 smithi070 podman[447164]: 2024-09-21 11:09:50.528637473 +0000 UTC m=+0.015896726 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T11:09:50.890 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 11:09:50 smithi070 podman[447164]: 2024-09-21 11:09:50.628122856 +0000 UTC m=+0.115382096 container create bcd9f682a69c8be2a34f142e4f3fe0ca91d13d901072c73c33a90e2eac9badd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-21T11:09:50.891 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 11:09:50 smithi070 podman[447164]: 2024-09-21 11:09:50.889154518 +0000 UTC m=+0.376413762 container init bcd9f682a69c8be2a34f142e4f3fe0ca91d13d901072c73c33a90e2eac9badd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, RELEASE=main-62da54b, io.buildah.version=1.37.2) 2024-09-21T11:09:51.378 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 11:09:50 smithi070 podman[447164]: 2024-09-21 11:09:50.894241126 +0000 UTC m=+0.381500367 container start bcd9f682a69c8be2a34f142e4f3fe0ca91d13d901072c73c33a90e2eac9badd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-21T11:09:51.378 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 11:09:51 smithi070 podman[447164]: 2024-09-21 11:09:51.047525139 +0000 UTC m=+0.534784399 container attach bcd9f682a69c8be2a34f142e4f3fe0ca91d13d901072c73c33a90e2eac9badd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, RELEASE=main-62da54b, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-21T11:09:52.269 INFO:journalctl@ceph.osd.0.smithi070.stdout:Sep 21 11:09:51 smithi070 podman[447164]: 2024-09-21 11:09:51.931750806 +0000 UTC m=+1.419010063 container died bcd9f682a69c8be2a34f142e4f3fe0ca91d13d901072c73c33a90e2eac9badd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-21T11:09:52.682 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.0.service' 2024-09-21T11:09:52.721 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T11:09:52.721 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-21T11:09:52.722 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-21T11:09:52.722 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.1 2024-09-21T11:09:53.041 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 11:09:52 smithi070 systemd[1]: Stopping Ceph osd.1 for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T11:09:53.377 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 11:09:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1[130215]: 2024-09-21T11:09:53.038+0000 7eff7b4e2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T11:09:53.377 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 11:09:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1[130215]: 2024-09-21T11:09:53.038+0000 7eff7b4e2640 -1 osd.1 1711 *** Got signal Terminated *** 2024-09-21T11:09:53.377 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 11:09:53 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1[130215]: 2024-09-21T11:09:53.038+0000 7eff7b4e2640 -1 osd.1 1711 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T11:09:55.591 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 11:09:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=cleanup t=2024-09-21T11:09:55.130285819Z level=info msg="Completed cleanup jobs" duration=13.426155ms 2024-09-21T11:09:55.592 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 11:09:55 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=plugins.update.checker t=2024-09-21T11:09:55.265133395Z level=info msg="Update check succeeded" duration=59.38756ms 2024-09-21T11:09:58.872 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 11:09:58 smithi070 podman[447296]: 2024-09-21 11:09:58.566535302 +0000 UTC m=+5.629032223 container died 853f3bb7fa25dc85edf4d6051f583ebb6d21c87e669766fed690887303f3d031 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-21T11:09:59.127 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 11:09:59 smithi070 podman[447296]: 2024-09-21 11:09:59.06206593 +0000 UTC m=+6.124562839 container cleanup 853f3bb7fa25dc85edf4d6051f583ebb6d21c87e669766fed690887303f3d031 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True) 2024-09-21T11:09:59.128 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 11:09:59 smithi070 bash[447296]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1 2024-09-21T11:09:59.627 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 11:09:59 smithi070 podman[447315]: 2024-09-21 11:09:59.321822744 +0000 UTC m=+0.748206625 container remove 853f3bb7fa25dc85edf4d6051f583ebb6d21c87e669766fed690887303f3d031 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=) 2024-09-21T11:09:59.910 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 11:09:59 smithi070 podman[447408]: 2024-09-21 11:09:59.530436712 +0000 UTC m=+0.015975692 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T11:09:59.910 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 11:09:59 smithi070 podman[447408]: 2024-09-21 11:09:59.67317081 +0000 UTC m=+0.158709774 container create ce98e056489c1a347ed9f101a5ae0af04bd1258163506d8cbc89a37fc2fa2bad (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-21T11:09:59.911 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 11:09:59 smithi070 podman[447408]: 2024-09-21 11:09:59.90913664 +0000 UTC m=+0.394675616 container init ce98e056489c1a347ed9f101a5ae0af04bd1258163506d8cbc89a37fc2fa2bad (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T11:10:00.232 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 11:09:59 smithi070 podman[447408]: 2024-09-21 11:09:59.91390917 +0000 UTC m=+0.399448135 container start ce98e056489c1a347ed9f101a5ae0af04bd1258163506d8cbc89a37fc2fa2bad (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, RELEASE=main-62da54b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-21T11:10:00.232 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 11:10:00 smithi070 podman[447408]: 2024-09-21 11:10:00.000534076 +0000 UTC m=+0.486073055 container attach ce98e056489c1a347ed9f101a5ae0af04bd1258163506d8cbc89a37fc2fa2bad (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-62da54b, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T11:10:00.232 INFO:journalctl@ceph.osd.1.smithi070.stdout:Sep 21 11:10:00 smithi070 podman[447408]: 2024-09-21 11:10:00.101034587 +0000 UTC m=+0.586573552 container died ce98e056489c1a347ed9f101a5ae0af04bd1258163506d8cbc89a37fc2fa2bad (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-21T11:10:00.747 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.1.service' 2024-09-21T11:10:00.787 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T11:10:00.787 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-21T11:10:00.788 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-21T11:10:00.788 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.2 2024-09-21T11:10:01.127 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 11:10:00 smithi070 systemd[1]: Stopping Ceph osd.2 for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T11:10:01.627 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 11:10:01 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2[136847]: 2024-09-21T11:10:01.143+0000 7fd6606c5640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T11:10:01.628 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 11:10:01 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2[136847]: 2024-09-21T11:10:01.143+0000 7fd6606c5640 -1 osd.2 1711 *** Got signal Terminated *** 2024-09-21T11:10:01.628 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 11:10:01 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2[136847]: 2024-09-21T11:10:01.143+0000 7fd6606c5640 -1 osd.2 1711 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T11:10:06.976 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 11:10:06 smithi070 podman[447548]: 2024-09-21 11:10:06.684599817 +0000 UTC m=+5.673208113 container died 014c9efbc75d658bac4169810fb03f1d72b3958c43f05cc8da67d59f9e854ffd (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-21T11:10:07.377 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 11:10:07 smithi070 podman[447548]: 2024-09-21 11:10:07.14019773 +0000 UTC m=+6.128806017 container cleanup 014c9efbc75d658bac4169810fb03f1d72b3958c43f05cc8da67d59f9e854ffd (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-21T11:10:07.377 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 11:10:07 smithi070 bash[447548]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2 2024-09-21T11:10:07.787 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 11:10:07 smithi070 podman[447571]: 2024-09-21 11:10:07.458317957 +0000 UTC m=+0.765724523 container remove 014c9efbc75d658bac4169810fb03f1d72b3958c43f05cc8da67d59f9e854ffd (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2, RELEASE=main-62da54b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-21T11:10:08.038 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 11:10:07 smithi070 podman[447669]: 2024-09-21 11:10:07.689303684 +0000 UTC m=+0.016141538 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T11:10:08.038 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 11:10:07 smithi070 podman[447669]: 2024-09-21 11:10:07.803421396 +0000 UTC m=+0.130259237 container create ac493d2e60316b78fee969b893c02c80d43d6d88b505abecda8e4f5d4e8ed2b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-21T11:10:08.377 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 11:10:08 smithi070 podman[447669]: 2024-09-21 11:10:08.036771916 +0000 UTC m=+0.363609759 container init ac493d2e60316b78fee969b893c02c80d43d6d88b505abecda8e4f5d4e8ed2b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=main-62da54b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T11:10:08.377 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 11:10:08 smithi070 podman[447669]: 2024-09-21 11:10:08.041636712 +0000 UTC m=+0.368474558 container start ac493d2e60316b78fee969b893c02c80d43d6d88b505abecda8e4f5d4e8ed2b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-62da54b) 2024-09-21T11:10:08.378 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 11:10:08 smithi070 podman[447669]: 2024-09-21 11:10:08.136561736 +0000 UTC m=+0.463399579 container attach ac493d2e60316b78fee969b893c02c80d43d6d88b505abecda8e4f5d4e8ed2b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-09-21T11:10:08.378 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 11:10:08 smithi070 podman[447669]: 2024-09-21 11:10:08.235477247 +0000 UTC m=+0.562315105 container died ac493d2e60316b78fee969b893c02c80d43d6d88b505abecda8e4f5d4e8ed2b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T11:10:08.843 INFO:journalctl@ceph.osd.2.smithi070.stdout:Sep 21 11:10:08 smithi070 podman[447712]: 2024-09-21 11:10:08.840035921 +0000 UTC m=+0.599010646 container remove ac493d2e60316b78fee969b893c02c80d43d6d88b505abecda8e4f5d4e8ed2b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T11:10:08.864 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.2.service' 2024-09-21T11:10:08.902 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T11:10:08.903 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-21T11:10:08.903 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-21T11:10:08.903 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.3 2024-09-21T11:10:09.127 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 11:10:08 smithi070 systemd[1]: Stopping Ceph osd.3 for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T11:10:09.627 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 11:10:09 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3[143088]: 2024-09-21T11:10:09.227+0000 7fa7c9e4e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T11:10:09.627 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 11:10:09 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3[143088]: 2024-09-21T11:10:09.227+0000 7fa7c9e4e640 -1 osd.3 1711 *** Got signal Terminated *** 2024-09-21T11:10:09.627 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 11:10:09 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3[143088]: 2024-09-21T11:10:09.227+0000 7fa7c9e4e640 -1 osd.3 1711 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T11:10:11.591 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:11 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:11.203+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.766971+0000 front 2024-09-21T11:09:47.766857+0000 (oldest deadline 2024-09-21T11:10:10.666802+0000) 2024-09-21T11:10:11.591 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:11 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:11.440+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:46.805589+0000 front 2024-09-21T11:09:46.805756+0000 (oldest deadline 2024-09-21T11:10:10.905795+0000) 2024-09-21T11:10:12.591 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:12 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:12.251+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.766971+0000 front 2024-09-21T11:09:47.766857+0000 (oldest deadline 2024-09-21T11:10:10.666802+0000) 2024-09-21T11:10:12.591 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:12 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:12.461+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:46.805589+0000 front 2024-09-21T11:09:46.805756+0000 (oldest deadline 2024-09-21T11:10:10.905795+0000) 2024-09-21T11:10:13.127 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 11:10:12 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3[143088]: 2024-09-21T11:10:12.793+0000 7fa7c245b640 -1 osd.3 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:48.777484+0000 front 2024-09-21T11:09:48.777387+0000 (oldest deadline 2024-09-21T11:10:12.277598+0000) 2024-09-21T11:10:13.482 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:13.238+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.766971+0000 front 2024-09-21T11:09:47.766857+0000 (oldest deadline 2024-09-21T11:10:10.666802+0000) 2024-09-21T11:10:13.482 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 11:10:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T11:10:13.098Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.70:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.70:8765: connect: connection refused" 2024-09-21T11:10:13.483 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 11:10:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T11:10:13.098Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.70:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.70:8765: connect: connection refused" 2024-09-21T11:10:13.483 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 11:10:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T11:10:13.098Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.70:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.70:8765: connect: connection refused" 2024-09-21T11:10:13.483 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 11:10:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T11:10:13.103Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=smb msg="Unable to refresh target groups" err="Get \"http://172.21.15.70:8765/sd/prometheus/sd-config?service=smb\": dial tcp 172.21.15.70:8765: connect: connection refused" 2024-09-21T11:10:13.483 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 11:10:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T11:10:13.103Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.70:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.70:8765: connect: connection refused" 2024-09-21T11:10:13.484 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 11:10:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T11:10:13.103Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.70:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.70:8765: connect: connection refused" 2024-09-21T11:10:13.484 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 11:10:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T11:10:13.103Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.70:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.70:8765: connect: connection refused" 2024-09-21T11:10:13.841 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:13.479+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:46.805589+0000 front 2024-09-21T11:09:46.805756+0000 (oldest deadline 2024-09-21T11:10:10.905795+0000) 2024-09-21T11:10:14.127 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 11:10:13 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3[143088]: 2024-09-21T11:10:13.815+0000 7fa7c245b640 -1 osd.3 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:48.777484+0000 front 2024-09-21T11:09:48.777387+0000 (oldest deadline 2024-09-21T11:10:12.277598+0000) 2024-09-21T11:10:14.261 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 21 11:10:13 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-4[38539]: 2024-09-21T11:10:13.870+0000 7f48b3dd7640 -1 osd.4 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.693760+0000 front 2024-09-21T11:09:47.693863+0000 (oldest deadline 2024-09-21T11:10:12.993622+0000) 2024-09-21T11:10:14.527 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:14 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:14.259+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.766971+0000 front 2024-09-21T11:09:47.766857+0000 (oldest deadline 2024-09-21T11:10:10.666802+0000) 2024-09-21T11:10:14.841 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:14 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:14.525+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:46.805589+0000 front 2024-09-21T11:09:46.805756+0000 (oldest deadline 2024-09-21T11:10:10.905795+0000) 2024-09-21T11:10:14.842 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:14 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:14.683+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:15.098 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 11:10:14 smithi070 podman[447795]: 2024-09-21 11:10:14.793378995 +0000 UTC m=+5.668036064 container died 862f41fda2c3a1ede15feb1b50d8d410e975f82fec81188500e80680ac209fd2 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731) 2024-09-21T11:10:15.216 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 21 11:10:14 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-4[38539]: 2024-09-21T11:10:14.860+0000 7f48b3dd7640 -1 osd.4 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.693760+0000 front 2024-09-21T11:09:47.693863+0000 (oldest deadline 2024-09-21T11:10:12.993622+0000) 2024-09-21T11:10:15.378 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 11:10:15 smithi070 podman[447795]: 2024-09-21 11:10:15.258496038 +0000 UTC m=+6.133153096 container cleanup 862f41fda2c3a1ede15feb1b50d8d410e975f82fec81188500e80680ac209fd2 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-62da54b, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T11:10:15.378 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 11:10:15 smithi070 bash[447795]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3 2024-09-21T11:10:15.482 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:15 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:15.215+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.766971+0000 front 2024-09-21T11:09:47.766857+0000 (oldest deadline 2024-09-21T11:10:10.666802+0000) 2024-09-21T11:10:15.841 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:15 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:15.687+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:15.841 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:15 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:15.481+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:46.805589+0000 front 2024-09-21T11:09:46.805756+0000 (oldest deadline 2024-09-21T11:10:10.905795+0000) 2024-09-21T11:10:15.868 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 11:10:15 smithi070 podman[447815]: 2024-09-21 11:10:15.551453432 +0000 UTC m=+0.759028432 container remove 862f41fda2c3a1ede15feb1b50d8d410e975f82fec81188500e80680ac209fd2 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T11:10:16.127 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 11:10:15 smithi070 podman[447909]: 2024-09-21 11:10:15.77063748 +0000 UTC m=+0.015882300 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T11:10:16.127 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 11:10:15 smithi070 podman[447909]: 2024-09-21 11:10:15.894211567 +0000 UTC m=+0.139456412 container create 2fe670c78ba66f2a36d6c8eee774a2fb924bec91273474e2ffb90b6e5678b003 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, RELEASE=main-62da54b, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-21T11:10:16.200 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 21 11:10:15 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-4[38539]: 2024-09-21T11:10:15.875+0000 7f48b3dd7640 -1 osd.4 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.693760+0000 front 2024-09-21T11:09:47.693863+0000 (oldest deadline 2024-09-21T11:10:12.993622+0000) 2024-09-21T11:10:16.459 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:16 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:16.198+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.766971+0000 front 2024-09-21T11:09:47.766857+0000 (oldest deadline 2024-09-21T11:10:10.666802+0000) 2024-09-21T11:10:16.618 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 11:10:16 smithi070 podman[447909]: 2024-09-21 11:10:16.129698608 +0000 UTC m=+0.374943420 container init 2fe670c78ba66f2a36d6c8eee774a2fb924bec91273474e2ffb90b6e5678b003 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=main-62da54b) 2024-09-21T11:10:16.618 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 11:10:16 smithi070 podman[447909]: 2024-09-21 11:10:16.134584401 +0000 UTC m=+0.379829201 container start 2fe670c78ba66f2a36d6c8eee774a2fb924bec91273474e2ffb90b6e5678b003 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, org.label-schema.build-date=20240731, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-21T11:10:16.618 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 11:10:16 smithi070 podman[447909]: 2024-09-21 11:10:16.213000276 +0000 UTC m=+0.458245080 container attach 2fe670c78ba66f2a36d6c8eee774a2fb924bec91273474e2ffb90b6e5678b003 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-21T11:10:16.619 INFO:journalctl@ceph.osd.3.smithi070.stdout:Sep 21 11:10:16 smithi070 podman[447909]: 2024-09-21 11:10:16.321583471 +0000 UTC m=+0.566828277 container died 2fe670c78ba66f2a36d6c8eee774a2fb924bec91273474e2ffb90b6e5678b003 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, GIT_CLEAN=True, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-21T11:10:16.841 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:16 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:16.457+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:46.805589+0000 front 2024-09-21T11:09:46.805756+0000 (oldest deadline 2024-09-21T11:10:10.905795+0000) 2024-09-21T11:10:16.841 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:16 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:16.666+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:16.895 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.3.service' 2024-09-21T11:10:16.933 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T11:10:16.934 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-21T11:10:16.934 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-21T11:10:16.934 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.4 2024-09-21T11:10:17.263 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:17 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:17.154+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.766971+0000 front 2024-09-21T11:09:47.766857+0000 (oldest deadline 2024-09-21T11:10:10.666802+0000) 2024-09-21T11:10:17.263 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 21 11:10:16 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-4[38539]: 2024-09-21T11:10:16.922+0000 7f48b3dd7640 -1 osd.4 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.693760+0000 front 2024-09-21T11:09:47.693863+0000 (oldest deadline 2024-09-21T11:10:12.993622+0000) 2024-09-21T11:10:17.263 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 21 11:10:16 smithi160 systemd[1]: Stopping Ceph osd.4 for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T11:10:17.264 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 21 11:10:17 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-4[38539]: 2024-09-21T11:10:17.261+0000 7f48b77c3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T11:10:17.264 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 21 11:10:17 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-4[38539]: 2024-09-21T11:10:17.261+0000 7f48b77c3640 -1 osd.4 1711 *** Got signal Terminated *** 2024-09-21T11:10:17.264 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 21 11:10:17 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-4[38539]: 2024-09-21T11:10:17.261+0000 7f48b77c3640 -1 osd.4 1711 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T11:10:17.591 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:17 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:17.485+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:46.805589+0000 front 2024-09-21T11:09:46.805756+0000 (oldest deadline 2024-09-21T11:10:10.905795+0000) 2024-09-21T11:10:17.949 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:17 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:17.665+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:18.341 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 21 11:10:17 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-4[38539]: 2024-09-21T11:10:17.948+0000 7f48b3dd7640 -1 osd.4 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.693760+0000 front 2024-09-21T11:09:47.693863+0000 (oldest deadline 2024-09-21T11:10:12.993622+0000) 2024-09-21T11:10:18.341 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:18 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:18.142+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.766971+0000 front 2024-09-21T11:09:47.766857+0000 (oldest deadline 2024-09-21T11:10:10.666802+0000) 2024-09-21T11:10:18.841 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:18 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:18.707+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:18.841 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:18 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:18.534+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:46.805589+0000 front 2024-09-21T11:09:46.805756+0000 (oldest deadline 2024-09-21T11:10:10.905795+0000) 2024-09-21T11:10:19.341 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 21 11:10:18 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-4[38539]: 2024-09-21T11:10:18.995+0000 7f48b3dd7640 -1 osd.4 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.693760+0000 front 2024-09-21T11:09:47.693863+0000 (oldest deadline 2024-09-21T11:10:12.993622+0000) 2024-09-21T11:10:19.342 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:19 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:19.100+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.766971+0000 front 2024-09-21T11:09:47.766857+0000 (oldest deadline 2024-09-21T11:10:10.666802+0000) 2024-09-21T11:10:19.841 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:19 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:19.678+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:19.841 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:19 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:19.572+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:46.805589+0000 front 2024-09-21T11:09:46.805756+0000 (oldest deadline 2024-09-21T11:10:10.905795+0000) 2024-09-21T11:10:19.842 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:19 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:19.572+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.806652+0000 front 2024-09-21T11:09:56.806607+0000 (oldest deadline 2024-09-21T11:10:19.007176+0000) 2024-09-21T11:10:20.341 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:20 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:20.089+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.766971+0000 front 2024-09-21T11:09:47.766857+0000 (oldest deadline 2024-09-21T11:10:10.666802+0000) 2024-09-21T11:10:20.342 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 21 11:10:19 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-4[38539]: 2024-09-21T11:10:19.976+0000 7f48b3dd7640 -1 osd.4 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.693760+0000 front 2024-09-21T11:09:47.693863+0000 (oldest deadline 2024-09-21T11:10:12.993622+0000) 2024-09-21T11:10:20.841 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:20 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:20.645+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:20.841 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:20 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:20.548+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:46.805589+0000 front 2024-09-21T11:09:46.805756+0000 (oldest deadline 2024-09-21T11:10:10.905795+0000) 2024-09-21T11:10:20.842 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:20 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:20.548+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.806652+0000 front 2024-09-21T11:09:56.806607+0000 (oldest deadline 2024-09-21T11:10:19.007176+0000) 2024-09-21T11:10:21.341 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 21 11:10:20 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-4[38539]: 2024-09-21T11:10:20.957+0000 7f48b3dd7640 -1 osd.4 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.693760+0000 front 2024-09-21T11:09:47.693863+0000 (oldest deadline 2024-09-21T11:10:12.993622+0000) 2024-09-21T11:10:21.341 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:21 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:21.066+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.766971+0000 front 2024-09-21T11:09:47.766857+0000 (oldest deadline 2024-09-21T11:10:10.666802+0000) 2024-09-21T11:10:21.342 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:21 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:21.066+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:55.868120+0000 front 2024-09-21T11:09:55.868088+0000 (oldest deadline 2024-09-21T11:10:20.567998+0000) 2024-09-21T11:10:21.841 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:21 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:21.503+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:46.805589+0000 front 2024-09-21T11:09:46.805756+0000 (oldest deadline 2024-09-21T11:10:10.905795+0000) 2024-09-21T11:10:21.841 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:21 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:21.503+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.806652+0000 front 2024-09-21T11:09:56.806607+0000 (oldest deadline 2024-09-21T11:10:19.007176+0000) 2024-09-21T11:10:21.841 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:21 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:21.619+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:21.842 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:21 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:21.619+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.645887+0000 front 2024-09-21T11:09:56.645899+0000 (oldest deadline 2024-09-21T11:10:20.745851+0000) 2024-09-21T11:10:22.341 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 21 11:10:22 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-4[38539]: 2024-09-21T11:10:22.000+0000 7f48b3dd7640 -1 osd.4 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.693760+0000 front 2024-09-21T11:09:47.693863+0000 (oldest deadline 2024-09-21T11:10:12.993622+0000) 2024-09-21T11:10:22.342 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:22 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:22.086+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.766971+0000 front 2024-09-21T11:09:47.766857+0000 (oldest deadline 2024-09-21T11:10:10.666802+0000) 2024-09-21T11:10:22.342 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:22 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:22.086+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:55.868120+0000 front 2024-09-21T11:09:55.868088+0000 (oldest deadline 2024-09-21T11:10:20.567998+0000) 2024-09-21T11:10:22.762 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:22 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:22.595+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:22.762 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:22 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:22.595+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.645887+0000 front 2024-09-21T11:09:56.645899+0000 (oldest deadline 2024-09-21T11:10:20.745851+0000) 2024-09-21T11:10:22.762 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:22 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:22.509+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:46.805589+0000 front 2024-09-21T11:09:46.805756+0000 (oldest deadline 2024-09-21T11:10:10.905795+0000) 2024-09-21T11:10:22.762 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:22 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:22.509+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.806652+0000 front 2024-09-21T11:09:56.806607+0000 (oldest deadline 2024-09-21T11:10:19.007176+0000) 2024-09-21T11:10:23.077 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 21 11:10:22 smithi160 podman[260906]: 2024-09-21 11:10:22.760149539 +0000 UTC m=+5.631528498 container died 7325a075c488f6c7188d09713381ea804b7e986e00dc6b51ff816b7bd794c273 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-4, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=reef-6a1bdf8, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-21T11:10:23.341 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:23 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:23.126+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.766971+0000 front 2024-09-21T11:09:47.766857+0000 (oldest deadline 2024-09-21T11:10:10.666802+0000) 2024-09-21T11:10:23.341 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:23 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:23.126+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:55.868120+0000 front 2024-09-21T11:09:55.868088+0000 (oldest deadline 2024-09-21T11:10:20.567998+0000) 2024-09-21T11:10:23.342 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 21 11:10:23 smithi160 podman[260906]: 2024-09-21 11:10:23.251666025 +0000 UTC m=+6.123044984 container cleanup 7325a075c488f6c7188d09713381ea804b7e986e00dc6b51ff816b7bd794c273 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-4, GIT_CLEAN=True, RELEASE=reef-6a1bdf8, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-21T11:10:23.342 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 21 11:10:23 smithi160 bash[260906]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-4 2024-09-21T11:10:23.841 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 21 11:10:23 smithi160 podman[260928]: 2024-09-21 11:10:23.553456853 +0000 UTC m=+0.784580558 container remove 7325a075c488f6c7188d09713381ea804b7e986e00dc6b51ff816b7bd794c273 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-4, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-6a1bdf8, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T11:10:23.841 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:23 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:23.532+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:46.805589+0000 front 2024-09-21T11:09:46.805756+0000 (oldest deadline 2024-09-21T11:10:10.905795+0000) 2024-09-21T11:10:23.842 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:23 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:23.532+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.806652+0000 front 2024-09-21T11:09:56.806607+0000 (oldest deadline 2024-09-21T11:10:19.007176+0000) 2024-09-21T11:10:23.842 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:23 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:23.575+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:23.842 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:23 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:23.575+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.645887+0000 front 2024-09-21T11:09:56.645899+0000 (oldest deadline 2024-09-21T11:10:20.745851+0000) 2024-09-21T11:10:24.114 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 21 11:10:23 smithi160 podman[261020]: 2024-09-21 11:10:23.764698164 +0000 UTC m=+0.016096908 image pull 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6 2024-09-21T11:10:24.115 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 21 11:10:23 smithi160 podman[261020]: 2024-09-21 11:10:23.884546019 +0000 UTC m=+0.135944751 container create 4212c2cb80f536ad485c68c59d8ee2e8e8f49e7d7202a62127bfb989845a3e9d (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-4-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, RELEASE=reef-6a1bdf8, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-21T11:10:24.504 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:24.112+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.766971+0000 front 2024-09-21T11:09:47.766857+0000 (oldest deadline 2024-09-21T11:10:10.666802+0000) 2024-09-21T11:10:24.504 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:24.112+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:55.868120+0000 front 2024-09-21T11:09:55.868088+0000 (oldest deadline 2024-09-21T11:10:20.567998+0000) 2024-09-21T11:10:24.504 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 21 11:10:24 smithi160 podman[261020]: 2024-09-21 11:10:24.123547684 +0000 UTC m=+0.374946420 container init 4212c2cb80f536ad485c68c59d8ee2e8e8f49e7d7202a62127bfb989845a3e9d (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-4-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=reef-6a1bdf8, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-21T11:10:24.505 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 21 11:10:24 smithi160 podman[261020]: 2024-09-21 11:10:24.128198462 +0000 UTC m=+0.379597195 container start 4212c2cb80f536ad485c68c59d8ee2e8e8f49e7d7202a62127bfb989845a3e9d (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-4-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-6a1bdf8, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-21T11:10:24.505 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 21 11:10:24 smithi160 podman[261020]: 2024-09-21 11:10:24.231825062 +0000 UTC m=+0.483223811 container attach 4212c2cb80f536ad485c68c59d8ee2e8e8f49e7d7202a62127bfb989845a3e9d (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-4-deactivate, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, RELEASE=reef-6a1bdf8, org.label-schema.build-date=20240731) 2024-09-21T11:10:24.505 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 21 11:10:24 smithi160 podman[261020]: 2024-09-21 11:10:24.352983194 +0000 UTC m=+0.604381932 container died 4212c2cb80f536ad485c68c59d8ee2e8e8f49e7d7202a62127bfb989845a3e9d (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-4-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=reef-6a1bdf8, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS) 2024-09-21T11:10:24.841 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:24.502+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:46.805589+0000 front 2024-09-21T11:09:46.805756+0000 (oldest deadline 2024-09-21T11:10:10.905795+0000) 2024-09-21T11:10:24.841 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:24.502+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.806652+0000 front 2024-09-21T11:09:56.806607+0000 (oldest deadline 2024-09-21T11:10:19.007176+0000) 2024-09-21T11:10:24.842 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:24.547+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:24.842 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:24 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:24.547+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.645887+0000 front 2024-09-21T11:09:56.645899+0000 (oldest deadline 2024-09-21T11:10:20.745851+0000) 2024-09-21T11:10:24.964 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.4.service' 2024-09-21T11:10:25.003 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T11:10:25.004 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-21T11:10:25.004 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-21T11:10:25.004 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.5 2024-09-21T11:10:25.341 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:25.065+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.766971+0000 front 2024-09-21T11:09:47.766857+0000 (oldest deadline 2024-09-21T11:10:10.666802+0000) 2024-09-21T11:10:25.341 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:25.065+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:55.868120+0000 front 2024-09-21T11:09:55.868088+0000 (oldest deadline 2024-09-21T11:10:20.567998+0000) 2024-09-21T11:10:25.342 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:25 smithi160 systemd[1]: Stopping Ceph osd.5 for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T11:10:25.841 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:25.484+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:46.805589+0000 front 2024-09-21T11:09:46.805756+0000 (oldest deadline 2024-09-21T11:10:10.905795+0000) 2024-09-21T11:10:25.841 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:25.484+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.806652+0000 front 2024-09-21T11:09:56.806607+0000 (oldest deadline 2024-09-21T11:10:19.007176+0000) 2024-09-21T11:10:25.842 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:25.583+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:25.842 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:25.583+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.645887+0000 front 2024-09-21T11:09:56.645899+0000 (oldest deadline 2024-09-21T11:10:20.745851+0000) 2024-09-21T11:10:25.842 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:25.340+0000 7f86c63a0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T11:10:25.842 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:25.340+0000 7f86c63a0640 -1 osd.5 1711 *** Got signal Terminated *** 2024-09-21T11:10:25.842 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:25 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:25.340+0000 7f86c63a0640 -1 osd.5 1711 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T11:10:26.341 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:26 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:26.057+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.766971+0000 front 2024-09-21T11:09:47.766857+0000 (oldest deadline 2024-09-21T11:10:10.666802+0000) 2024-09-21T11:10:26.341 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:26 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:26.057+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:55.868120+0000 front 2024-09-21T11:09:55.868088+0000 (oldest deadline 2024-09-21T11:10:20.567998+0000) 2024-09-21T11:10:26.841 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:26 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:26.534+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:46.805589+0000 front 2024-09-21T11:09:46.805756+0000 (oldest deadline 2024-09-21T11:10:10.905795+0000) 2024-09-21T11:10:26.841 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:26 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:26.534+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.806652+0000 front 2024-09-21T11:09:56.806607+0000 (oldest deadline 2024-09-21T11:10:19.007176+0000) 2024-09-21T11:10:26.842 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:26 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:26.545+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:26.842 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:26 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:26.545+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.645887+0000 front 2024-09-21T11:09:56.645899+0000 (oldest deadline 2024-09-21T11:10:20.745851+0000) 2024-09-21T11:10:27.341 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:27 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:27.067+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.766971+0000 front 2024-09-21T11:09:47.766857+0000 (oldest deadline 2024-09-21T11:10:10.666802+0000) 2024-09-21T11:10:27.341 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:27 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:27.067+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:55.868120+0000 front 2024-09-21T11:09:55.868088+0000 (oldest deadline 2024-09-21T11:10:20.567998+0000) 2024-09-21T11:10:27.841 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:27 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:27.521+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:27.841 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:27 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:27.521+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.645887+0000 front 2024-09-21T11:09:56.645899+0000 (oldest deadline 2024-09-21T11:10:20.745851+0000) 2024-09-21T11:10:27.842 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:27 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:27.497+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:46.805589+0000 front 2024-09-21T11:09:46.805756+0000 (oldest deadline 2024-09-21T11:10:10.905795+0000) 2024-09-21T11:10:27.842 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:27 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:27.497+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.806652+0000 front 2024-09-21T11:09:56.806607+0000 (oldest deadline 2024-09-21T11:10:19.007176+0000) 2024-09-21T11:10:27.842 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:27 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:27.497+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:04.208749+0000 front 2024-09-21T11:10:04.208767+0000 (oldest deadline 2024-09-21T11:10:27.108550+0000) 2024-09-21T11:10:28.506 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:28 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:28.103+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.766971+0000 front 2024-09-21T11:09:47.766857+0000 (oldest deadline 2024-09-21T11:10:10.666802+0000) 2024-09-21T11:10:28.507 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:28 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:28.103+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:55.868120+0000 front 2024-09-21T11:09:55.868088+0000 (oldest deadline 2024-09-21T11:10:20.567998+0000) 2024-09-21T11:10:28.841 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:28 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:28.571+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:28.841 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:28 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:28.571+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.645887+0000 front 2024-09-21T11:09:56.645899+0000 (oldest deadline 2024-09-21T11:10:20.745851+0000) 2024-09-21T11:10:28.842 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:28 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:28.504+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:46.805589+0000 front 2024-09-21T11:09:46.805756+0000 (oldest deadline 2024-09-21T11:10:10.905795+0000) 2024-09-21T11:10:28.842 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:28 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:28.504+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.806652+0000 front 2024-09-21T11:09:56.806607+0000 (oldest deadline 2024-09-21T11:10:19.007176+0000) 2024-09-21T11:10:28.842 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:28 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:28.504+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:04.208749+0000 front 2024-09-21T11:10:04.208767+0000 (oldest deadline 2024-09-21T11:10:27.108550+0000) 2024-09-21T11:10:29.495 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:29 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:29.102+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.766971+0000 front 2024-09-21T11:09:47.766857+0000 (oldest deadline 2024-09-21T11:10:10.666802+0000) 2024-09-21T11:10:29.495 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:29 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:29.102+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:55.868120+0000 front 2024-09-21T11:09:55.868088+0000 (oldest deadline 2024-09-21T11:10:20.567998+0000) 2024-09-21T11:10:29.495 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:29 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:29.102+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:05.069816+0000 front 2024-09-21T11:10:05.069760+0000 (oldest deadline 2024-09-21T11:10:28.569642+0000) 2024-09-21T11:10:29.841 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:29 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:29.609+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:29.841 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:29 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:29.609+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.645887+0000 front 2024-09-21T11:09:56.645899+0000 (oldest deadline 2024-09-21T11:10:20.745851+0000) 2024-09-21T11:10:29.842 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:29 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:29.493+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:46.805589+0000 front 2024-09-21T11:09:46.805756+0000 (oldest deadline 2024-09-21T11:10:10.905795+0000) 2024-09-21T11:10:29.842 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:29 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:29.493+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.806652+0000 front 2024-09-21T11:09:56.806607+0000 (oldest deadline 2024-09-21T11:10:19.007176+0000) 2024-09-21T11:10:29.842 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:29 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:29.493+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:04.208749+0000 front 2024-09-21T11:10:04.208767+0000 (oldest deadline 2024-09-21T11:10:27.108550+0000) 2024-09-21T11:10:30.507 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:30 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:30.121+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:47.766971+0000 front 2024-09-21T11:09:47.766857+0000 (oldest deadline 2024-09-21T11:10:10.666802+0000) 2024-09-21T11:10:30.507 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:30 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:30.121+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:55.868120+0000 front 2024-09-21T11:09:55.868088+0000 (oldest deadline 2024-09-21T11:10:20.567998+0000) 2024-09-21T11:10:30.507 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:30 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5[238088]: 2024-09-21T11:10:30.121+0000 7f86be9ac640 -1 osd.5 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:05.069816+0000 front 2024-09-21T11:10:05.069760+0000 (oldest deadline 2024-09-21T11:10:28.569642+0000) 2024-09-21T11:10:30.841 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:30 smithi160 podman[261148]: 2024-09-21 11:10:30.701806688 +0000 UTC m=+5.477161480 container died 15bbf79846c01e94c9b0c2a2d137a4f2d09076f47610505782d875fe87051ac2 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=reef-6a1bdf8, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T11:10:30.842 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:30 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:30.504+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:46.805589+0000 front 2024-09-21T11:09:46.805756+0000 (oldest deadline 2024-09-21T11:10:10.905795+0000) 2024-09-21T11:10:30.842 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:30 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:30.504+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.806652+0000 front 2024-09-21T11:09:56.806607+0000 (oldest deadline 2024-09-21T11:10:19.007176+0000) 2024-09-21T11:10:30.842 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:30 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:30.504+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:04.208749+0000 front 2024-09-21T11:10:04.208767+0000 (oldest deadline 2024-09-21T11:10:27.108550+0000) 2024-09-21T11:10:30.842 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:30 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:30.583+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:30.842 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:30 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:30.583+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.645887+0000 front 2024-09-21T11:09:56.645899+0000 (oldest deadline 2024-09-21T11:10:20.745851+0000) 2024-09-21T11:10:30.842 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:30 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:30.583+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:04.246810+0000 front 2024-09-21T11:10:04.246852+0000 (oldest deadline 2024-09-21T11:10:30.146719+0000) 2024-09-21T11:10:31.341 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:31 smithi160 podman[261148]: 2024-09-21 11:10:31.146553469 +0000 UTC m=+5.921908294 container cleanup 15bbf79846c01e94c9b0c2a2d137a4f2d09076f47610505782d875fe87051ac2 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5, io.buildah.version=1.37.2, RELEASE=reef-6a1bdf8, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T11:10:31.341 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:31 smithi160 bash[261148]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5 2024-09-21T11:10:31.790 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:31 smithi160 podman[261170]: 2024-09-21 11:10:31.481327778 +0000 UTC m=+0.778690574 container remove 15bbf79846c01e94c9b0c2a2d137a4f2d09076f47610505782d875fe87051ac2 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=reef-6a1bdf8, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-21T11:10:31.791 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:31 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:31.552+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:46.805589+0000 front 2024-09-21T11:09:46.805756+0000 (oldest deadline 2024-09-21T11:10:10.905795+0000) 2024-09-21T11:10:31.791 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:31 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:31.552+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.806652+0000 front 2024-09-21T11:09:56.806607+0000 (oldest deadline 2024-09-21T11:10:19.007176+0000) 2024-09-21T11:10:31.791 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:31 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:31.552+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:04.208749+0000 front 2024-09-21T11:10:04.208767+0000 (oldest deadline 2024-09-21T11:10:27.108550+0000) 2024-09-21T11:10:31.791 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:31 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:31.591+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:31.791 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:31 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:31.591+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.645887+0000 front 2024-09-21T11:09:56.645899+0000 (oldest deadline 2024-09-21T11:10:20.745851+0000) 2024-09-21T11:10:31.792 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:31 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:31.591+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:04.246810+0000 front 2024-09-21T11:10:04.246852+0000 (oldest deadline 2024-09-21T11:10:30.146719+0000) 2024-09-21T11:10:32.092 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:31 smithi160 podman[261263]: 2024-09-21 11:10:31.691728108 +0000 UTC m=+0.015906165 image pull 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6 2024-09-21T11:10:32.092 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:31 smithi160 podman[261263]: 2024-09-21 11:10:31.790536982 +0000 UTC m=+0.114715072 container create c5d7a878882f98c5b89c6a793c48b02dcf4c66d9d9ebbb26bb88f9da25323720 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-6a1bdf8, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-21T11:10:32.092 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:32 smithi160 podman[261263]: 2024-09-21 11:10:32.026851792 +0000 UTC m=+0.351029850 container init c5d7a878882f98c5b89c6a793c48b02dcf4c66d9d9ebbb26bb88f9da25323720 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, CEPH_POINT_RELEASE=, RELEASE=reef-6a1bdf8, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-21T11:10:32.092 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:32 smithi160 podman[261263]: 2024-09-21 11:10:32.031788864 +0000 UTC m=+0.355966915 container start c5d7a878882f98c5b89c6a793c48b02dcf4c66d9d9ebbb26bb88f9da25323720 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , RELEASE=reef-6a1bdf8, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T11:10:32.556 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:32 smithi160 podman[261263]: 2024-09-21 11:10:32.126311682 +0000 UTC m=+0.450489739 container attach c5d7a878882f98c5b89c6a793c48b02dcf4c66d9d9ebbb26bb88f9da25323720 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-6a1bdf8, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-21T11:10:32.557 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 21 11:10:32 smithi160 podman[261263]: 2024-09-21 11:10:32.214147603 +0000 UTC m=+0.538325663 container died c5d7a878882f98c5b89c6a793c48b02dcf4c66d9d9ebbb26bb88f9da25323720 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=reef-6a1bdf8) 2024-09-21T11:10:32.841 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:32 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:32.630+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:32.841 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:32 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:32.630+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.645887+0000 front 2024-09-21T11:09:56.645899+0000 (oldest deadline 2024-09-21T11:10:20.745851+0000) 2024-09-21T11:10:32.841 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:32 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:32.630+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:04.246810+0000 front 2024-09-21T11:10:04.246852+0000 (oldest deadline 2024-09-21T11:10:30.146719+0000) 2024-09-21T11:10:32.842 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:32 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:32.573+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:46.805589+0000 front 2024-09-21T11:09:46.805756+0000 (oldest deadline 2024-09-21T11:10:10.905795+0000) 2024-09-21T11:10:32.842 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:32 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:32.573+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.806652+0000 front 2024-09-21T11:09:56.806607+0000 (oldest deadline 2024-09-21T11:10:19.007176+0000) 2024-09-21T11:10:32.842 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:32 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:32.573+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:04.208749+0000 front 2024-09-21T11:10:04.208767+0000 (oldest deadline 2024-09-21T11:10:27.108550+0000) 2024-09-21T11:10:32.878 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.5.service' 2024-09-21T11:10:32.917 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T11:10:32.917 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-21T11:10:32.917 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-21T11:10:32.917 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.6 2024-09-21T11:10:33.236 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:32 smithi160 systemd[1]: Stopping Ceph osd.6 for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T11:10:33.236 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:33 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:33.234+0000 7fab83a2e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T11:10:33.554 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:33 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:33.234+0000 7fab83a2e640 -1 osd.6 1711 *** Got signal Terminated *** 2024-09-21T11:10:33.554 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:33 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:33.234+0000 7fab83a2e640 -1 osd.6 1711 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T11:10:33.554 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:33 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:33.552+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:46.805589+0000 front 2024-09-21T11:09:46.805756+0000 (oldest deadline 2024-09-21T11:10:10.905795+0000) 2024-09-21T11:10:33.555 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:33 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:33.552+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.806652+0000 front 2024-09-21T11:09:56.806607+0000 (oldest deadline 2024-09-21T11:10:19.007176+0000) 2024-09-21T11:10:33.555 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:33 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:33.552+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:04.208749+0000 front 2024-09-21T11:10:04.208767+0000 (oldest deadline 2024-09-21T11:10:27.108550+0000) 2024-09-21T11:10:33.841 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:33 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:33.671+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:33.841 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:33 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:33.671+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.645887+0000 front 2024-09-21T11:09:56.645899+0000 (oldest deadline 2024-09-21T11:10:20.745851+0000) 2024-09-21T11:10:33.841 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:33 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:33.671+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:04.246810+0000 front 2024-09-21T11:10:04.246852+0000 (oldest deadline 2024-09-21T11:10:30.146719+0000) 2024-09-21T11:10:34.841 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:34 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:34.703+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:34.841 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:34 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:34.703+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.645887+0000 front 2024-09-21T11:09:56.645899+0000 (oldest deadline 2024-09-21T11:10:20.745851+0000) 2024-09-21T11:10:34.842 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:34 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:34.703+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:04.246810+0000 front 2024-09-21T11:10:04.246852+0000 (oldest deadline 2024-09-21T11:10:30.146719+0000) 2024-09-21T11:10:34.842 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:34 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:34.509+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:46.805589+0000 front 2024-09-21T11:09:46.805756+0000 (oldest deadline 2024-09-21T11:10:10.905795+0000) 2024-09-21T11:10:34.842 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:34 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:34.509+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.806652+0000 front 2024-09-21T11:09:56.806607+0000 (oldest deadline 2024-09-21T11:10:19.007176+0000) 2024-09-21T11:10:34.842 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:34 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:34.509+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:04.208749+0000 front 2024-09-21T11:10:04.208767+0000 (oldest deadline 2024-09-21T11:10:27.108550+0000) 2024-09-21T11:10:35.754 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:35 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:35.494+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:46.805589+0000 front 2024-09-21T11:09:46.805756+0000 (oldest deadline 2024-09-21T11:10:10.905795+0000) 2024-09-21T11:10:35.755 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:35 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:35.494+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.806652+0000 front 2024-09-21T11:09:56.806607+0000 (oldest deadline 2024-09-21T11:10:19.007176+0000) 2024-09-21T11:10:35.755 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:35 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:35.494+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:04.208749+0000 front 2024-09-21T11:10:04.208767+0000 (oldest deadline 2024-09-21T11:10:27.108550+0000) 2024-09-21T11:10:36.091 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:35 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:35.753+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:36.091 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:35 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:35.753+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.645887+0000 front 2024-09-21T11:09:56.645899+0000 (oldest deadline 2024-09-21T11:10:20.745851+0000) 2024-09-21T11:10:36.091 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:35 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:35.753+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:04.246810+0000 front 2024-09-21T11:10:04.246852+0000 (oldest deadline 2024-09-21T11:10:30.146719+0000) 2024-09-21T11:10:36.792 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:36 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:36.541+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:46.805589+0000 front 2024-09-21T11:09:46.805756+0000 (oldest deadline 2024-09-21T11:10:10.905795+0000) 2024-09-21T11:10:36.793 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:36 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:36.541+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.806652+0000 front 2024-09-21T11:09:56.806607+0000 (oldest deadline 2024-09-21T11:10:19.007176+0000) 2024-09-21T11:10:36.793 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:36 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:36.541+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:04.208749+0000 front 2024-09-21T11:10:04.208767+0000 (oldest deadline 2024-09-21T11:10:27.108550+0000) 2024-09-21T11:10:37.091 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:36 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:36.790+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:37.091 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:36 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:36.790+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.645887+0000 front 2024-09-21T11:09:56.645899+0000 (oldest deadline 2024-09-21T11:10:20.745851+0000) 2024-09-21T11:10:37.091 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:36 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:36.790+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:04.246810+0000 front 2024-09-21T11:10:04.246852+0000 (oldest deadline 2024-09-21T11:10:30.146719+0000) 2024-09-21T11:10:37.841 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:37.774+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:37.841 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:37.774+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.645887+0000 front 2024-09-21T11:09:56.645899+0000 (oldest deadline 2024-09-21T11:10:20.745851+0000) 2024-09-21T11:10:37.841 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:37.774+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:04.246810+0000 front 2024-09-21T11:10:04.246852+0000 (oldest deadline 2024-09-21T11:10:30.146719+0000) 2024-09-21T11:10:37.842 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:37.557+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:46.805589+0000 front 2024-09-21T11:09:46.805756+0000 (oldest deadline 2024-09-21T11:10:10.905795+0000) 2024-09-21T11:10:37.842 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:37.557+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.806652+0000 front 2024-09-21T11:09:56.806607+0000 (oldest deadline 2024-09-21T11:10:19.007176+0000) 2024-09-21T11:10:37.842 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:37 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6[232910]: 2024-09-21T11:10:37.557+0000 7fab7c03a640 -1 osd.6 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:04.208749+0000 front 2024-09-21T11:10:04.208767+0000 (oldest deadline 2024-09-21T11:10:27.108550+0000) 2024-09-21T11:10:39.080 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:38 smithi160 podman[261390]: 2024-09-21 11:10:38.784005689 +0000 UTC m=+5.651288542 container died e89916e49bf5a8b61c796656ed7e9aa11822ece713d8f262c14cd4044e15175c (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=reef-6a1bdf8, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T11:10:39.080 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:38 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:38.760+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:39.080 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:38 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:38.760+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.645887+0000 front 2024-09-21T11:09:56.645899+0000 (oldest deadline 2024-09-21T11:10:20.745851+0000) 2024-09-21T11:10:39.080 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:38 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:38.760+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:04.246810+0000 front 2024-09-21T11:10:04.246852+0000 (oldest deadline 2024-09-21T11:10:30.146719+0000) 2024-09-21T11:10:39.080 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:38 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:38.760+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6830 osd.3 since back 2024-09-21T11:10:12.848546+0000 front 2024-09-21T11:10:12.848490+0000 (oldest deadline 2024-09-21T11:10:38.748408+0000) 2024-09-21T11:10:39.341 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:39 smithi160 podman[261390]: 2024-09-21 11:10:39.274627221 +0000 UTC m=+6.141910075 container cleanup e89916e49bf5a8b61c796656ed7e9aa11822ece713d8f262c14cd4044e15175c (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=reef-6a1bdf8, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T11:10:39.341 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:39 smithi160 bash[261390]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6 2024-09-21T11:10:39.841 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:39 smithi160 podman[261411]: 2024-09-21 11:10:39.578918909 +0000 UTC m=+0.797064607 container remove e89916e49bf5a8b61c796656ed7e9aa11822ece713d8f262c14cd4044e15175c (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-6a1bdf8, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-21T11:10:39.841 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:39 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:39.714+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:39.842 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:39 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:39.714+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.645887+0000 front 2024-09-21T11:09:56.645899+0000 (oldest deadline 2024-09-21T11:10:20.745851+0000) 2024-09-21T11:10:39.842 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:39 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:39.714+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:04.246810+0000 front 2024-09-21T11:10:04.246852+0000 (oldest deadline 2024-09-21T11:10:30.146719+0000) 2024-09-21T11:10:39.842 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:39 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:39.714+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6830 osd.3 since back 2024-09-21T11:10:12.848546+0000 front 2024-09-21T11:10:12.848490+0000 (oldest deadline 2024-09-21T11:10:38.748408+0000) 2024-09-21T11:10:40.214 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:39 smithi160 podman[261504]: 2024-09-21 11:10:39.82561268 +0000 UTC m=+0.015828765 image pull 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6 2024-09-21T11:10:40.215 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:39 smithi160 podman[261504]: 2024-09-21 11:10:39.960652272 +0000 UTC m=+0.150868344 container create c8cf2fb5aa54ee54d7f84022026a9d560d9888f1bfac3fe3314cafc73d44cbb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6-deactivate, org.label-schema.vendor=CentOS, RELEASE=reef-6a1bdf8, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True) 2024-09-21T11:10:40.215 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:40 smithi160 podman[261504]: 2024-09-21 11:10:40.21313363 +0000 UTC m=+0.403349711 container init c8cf2fb5aa54ee54d7f84022026a9d560d9888f1bfac3fe3314cafc73d44cbb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6-deactivate, org.label-schema.vendor=CentOS, RELEASE=reef-6a1bdf8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.license=GPLv2) 2024-09-21T11:10:40.591 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:40 smithi160 podman[261504]: 2024-09-21 11:10:40.218408232 +0000 UTC m=+0.408624303 container start c8cf2fb5aa54ee54d7f84022026a9d560d9888f1bfac3fe3314cafc73d44cbb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6-deactivate, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=reef-6a1bdf8, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T11:10:40.591 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:40 smithi160 podman[261504]: 2024-09-21 11:10:40.321375355 +0000 UTC m=+0.511591439 container attach c8cf2fb5aa54ee54d7f84022026a9d560d9888f1bfac3fe3314cafc73d44cbb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=reef-6a1bdf8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-21T11:10:40.592 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 21 11:10:40 smithi160 podman[261504]: 2024-09-21 11:10:40.404438192 +0000 UTC m=+0.594654273 container died c8cf2fb5aa54ee54d7f84022026a9d560d9888f1bfac3fe3314cafc73d44cbb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-6a1bdf8, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2) 2024-09-21T11:10:41.026 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:40 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:40.743+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:41.027 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:40 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:40.743+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.645887+0000 front 2024-09-21T11:09:56.645899+0000 (oldest deadline 2024-09-21T11:10:20.745851+0000) 2024-09-21T11:10:41.027 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:40 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:40.743+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:04.246810+0000 front 2024-09-21T11:10:04.246852+0000 (oldest deadline 2024-09-21T11:10:30.146719+0000) 2024-09-21T11:10:41.027 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:40 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:40.743+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6830 osd.3 since back 2024-09-21T11:10:12.848546+0000 front 2024-09-21T11:10:12.848490+0000 (oldest deadline 2024-09-21T11:10:38.748408+0000) 2024-09-21T11:10:41.049 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.6.service' 2024-09-21T11:10:41.089 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T11:10:41.089 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-21T11:10:41.090 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-21T11:10:41.090 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.7 2024-09-21T11:10:41.341 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:41 smithi160 systemd[1]: Stopping Ceph osd.7 for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T11:10:41.841 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:41 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:41.462+0000 7fe7031d3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T11:10:41.842 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:41 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:41.462+0000 7fe7031d3640 -1 osd.7 1711 *** Got signal Terminated *** 2024-09-21T11:10:41.842 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:41 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:41.462+0000 7fe7031d3640 -1 osd.7 1711 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T11:10:41.842 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:41 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:41.699+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:41.842 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:41 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:41.699+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.645887+0000 front 2024-09-21T11:09:56.645899+0000 (oldest deadline 2024-09-21T11:10:20.745851+0000) 2024-09-21T11:10:41.843 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:41 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:41.699+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:04.246810+0000 front 2024-09-21T11:10:04.246852+0000 (oldest deadline 2024-09-21T11:10:30.146719+0000) 2024-09-21T11:10:41.843 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:41 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:41.699+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6830 osd.3 since back 2024-09-21T11:10:12.848546+0000 front 2024-09-21T11:10:12.848490+0000 (oldest deadline 2024-09-21T11:10:38.748408+0000) 2024-09-21T11:10:43.091 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:42 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:42.735+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:43.091 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:42 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:42.735+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.645887+0000 front 2024-09-21T11:09:56.645899+0000 (oldest deadline 2024-09-21T11:10:20.745851+0000) 2024-09-21T11:10:43.092 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:42 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:42.735+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:04.246810+0000 front 2024-09-21T11:10:04.246852+0000 (oldest deadline 2024-09-21T11:10:30.146719+0000) 2024-09-21T11:10:43.092 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:42 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:42.735+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6830 osd.3 since back 2024-09-21T11:10:12.848546+0000 front 2024-09-21T11:10:12.848490+0000 (oldest deadline 2024-09-21T11:10:38.748408+0000) 2024-09-21T11:10:44.091 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:43 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:43.709+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:44.092 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:43 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:43.709+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.645887+0000 front 2024-09-21T11:09:56.645899+0000 (oldest deadline 2024-09-21T11:10:20.745851+0000) 2024-09-21T11:10:44.092 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:43 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:43.709+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:04.246810+0000 front 2024-09-21T11:10:04.246852+0000 (oldest deadline 2024-09-21T11:10:30.146719+0000) 2024-09-21T11:10:44.092 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:43 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:43.709+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6830 osd.3 since back 2024-09-21T11:10:12.848546+0000 front 2024-09-21T11:10:12.848490+0000 (oldest deadline 2024-09-21T11:10:38.748408+0000) 2024-09-21T11:10:45.091 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:44 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:44.675+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:45.091 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:44 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:44.675+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.645887+0000 front 2024-09-21T11:09:56.645899+0000 (oldest deadline 2024-09-21T11:10:20.745851+0000) 2024-09-21T11:10:45.091 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:44 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:44.675+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:04.246810+0000 front 2024-09-21T11:10:04.246852+0000 (oldest deadline 2024-09-21T11:10:30.146719+0000) 2024-09-21T11:10:45.092 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:44 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:44.675+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6830 osd.3 since back 2024-09-21T11:10:12.848546+0000 front 2024-09-21T11:10:12.848490+0000 (oldest deadline 2024-09-21T11:10:38.748408+0000) 2024-09-21T11:10:46.091 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:45 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:45.639+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-09-21T11:09:49.045099+0000 front 2024-09-21T11:09:49.045196+0000 (oldest deadline 2024-09-21T11:10:13.745123+0000) 2024-09-21T11:10:46.091 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:45 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:45.639+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-09-21T11:09:56.645887+0000 front 2024-09-21T11:09:56.645899+0000 (oldest deadline 2024-09-21T11:10:20.745851+0000) 2024-09-21T11:10:46.091 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:45 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:45.639+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-09-21T11:10:04.246810+0000 front 2024-09-21T11:10:04.246852+0000 (oldest deadline 2024-09-21T11:10:30.146719+0000) 2024-09-21T11:10:46.092 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:45 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7[57138]: 2024-09-21T11:10:45.639+0000 7fe6fb7df640 -1 osd.7 1711 heartbeat_check: no reply from 172.21.15.70:6830 osd.3 since back 2024-09-21T11:10:12.848546+0000 front 2024-09-21T11:10:12.848490+0000 (oldest deadline 2024-09-21T11:10:38.748408+0000) 2024-09-21T11:10:47.449 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:47 smithi160 podman[261631]: 2024-09-21 11:10:47.112797609 +0000 UTC m=+5.804273764 container died e405b21a61461b6cb4815c1dbc7f0f6ce3e4f98de7bb17cc257122ae3b7262eb (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7, RELEASE=reef-6a1bdf8, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-21T11:10:47.841 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:47 smithi160 podman[261631]: 2024-09-21 11:10:47.636277252 +0000 UTC m=+6.327753395 container cleanup e405b21a61461b6cb4815c1dbc7f0f6ce3e4f98de7bb17cc257122ae3b7262eb (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=reef-6a1bdf8, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T11:10:47.841 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:47 smithi160 bash[261631]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7 2024-09-21T11:10:48.281 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:47 smithi160 podman[261653]: 2024-09-21 11:10:47.870978324 +0000 UTC m=+0.751231429 container remove e405b21a61461b6cb4815c1dbc7f0f6ce3e4f98de7bb17cc257122ae3b7262eb (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=reef-6a1bdf8, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T11:10:48.567 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:48 smithi160 podman[261747]: 2024-09-21 11:10:48.280312852 +0000 UTC m=+0.102422976 container create 3218f96295c5846799ea2374456a5252c7960722138c1ee2e6df2a6b522eb5c0 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=reef-6a1bdf8) 2024-09-21T11:10:48.567 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:48 smithi160 podman[261747]: 2024-09-21 11:10:48.193962025 +0000 UTC m=+0.016072161 image pull 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6 2024-09-21T11:10:48.567 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:48 smithi160 podman[261747]: 2024-09-21 11:10:48.47416213 +0000 UTC m=+0.296272254 container init 3218f96295c5846799ea2374456a5252c7960722138c1ee2e6df2a6b522eb5c0 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=reef-6a1bdf8, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2) 2024-09-21T11:10:48.567 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:48 smithi160 podman[261747]: 2024-09-21 11:10:48.478924873 +0000 UTC m=+0.301035001 container start 3218f96295c5846799ea2374456a5252c7960722138c1ee2e6df2a6b522eb5c0 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7-deactivate, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-6a1bdf8, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-21T11:10:48.568 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:48 smithi160 podman[261747]: 2024-09-21 11:10:48.565623702 +0000 UTC m=+0.387733841 container attach 3218f96295c5846799ea2374456a5252c7960722138c1ee2e6df2a6b522eb5c0 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, RELEASE=reef-6a1bdf8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T11:10:48.842 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:48 smithi160 podman[261747]: 2024-09-21 11:10:48.665107554 +0000 UTC m=+0.487217694 container died 3218f96295c5846799ea2374456a5252c7960722138c1ee2e6df2a6b522eb5c0 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=reef-6a1bdf8, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-21T11:10:49.272 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 21 11:10:49 smithi160 podman[261790]: 2024-09-21 11:10:49.269043012 +0000 UTC m=+0.598061531 container remove 3218f96295c5846799ea2374456a5252c7960722138c1ee2e6df2a6b522eb5c0 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=reef-6a1bdf8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-09-21T11:10:49.288 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@osd.7.service' 2024-09-21T11:10:49.326 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T11:10:49.326 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-21T11:10:49.327 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-21T11:10:49.327 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@prometheus.a 2024-09-21T11:10:49.591 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 11:10:49 smithi160 systemd[1]: Stopping Ceph prometheus.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T11:10:49.966 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 11:10:49 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T11:10:49.648Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T11:10:49.966 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 11:10:49 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T11:10:49.649Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-21T11:10:49.966 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 11:10:49 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T11:10:49.649Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-21T11:10:49.966 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 11:10:49 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T11:10:49.649Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T11:10:49.966 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 11:10:49 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T11:10:49.649Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-21T11:10:49.967 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 11:10:49 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T11:10:49.649Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-21T11:10:49.967 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 11:10:49 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T11:10:49.649Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T11:10:49.967 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 11:10:49 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T11:10:49.649Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-21T11:10:49.967 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 11:10:49 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T11:10:49.649Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-21T11:10:49.967 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 11:10:49 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T11:10:49.653Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T11:10:49.967 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 11:10:49 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T11:10:49.654Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-21T11:10:49.967 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 11:10:49 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a[88001]: ts=2024-09-21T11:10:49.654Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-21T11:10:49.967 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 11:10:49 smithi160 podman[261876]: 2024-09-21 11:10:49.659510673 +0000 UTC m=+0.120168986 container died 648952e57ad7c6f8ab76d6b076f2f02f51e6d413608e94503b4161582b451c94 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T11:10:50.299 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 11:10:50 smithi160 podman[261876]: 2024-09-21 11:10:50.071731337 +0000 UTC m=+0.532389650 container cleanup 648952e57ad7c6f8ab76d6b076f2f02f51e6d413608e94503b4161582b451c94 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T11:10:50.299 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 11:10:50 smithi160 bash[261876]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T11:10:50.534 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@prometheus.a.service' 2024-09-21T11:10:50.555 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 11:10:50 smithi160 podman[261894]: 2024-09-21 11:10:50.298071367 +0000 UTC m=+0.636421557 container remove 648952e57ad7c6f8ab76d6b076f2f02f51e6d413608e94503b4161582b451c94 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T11:10:50.555 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 11:10:50 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-21T11:10:50.556 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 11:10:50 smithi160 systemd[1]: Stopped Ceph prometheus.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T11:10:50.556 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 21 11:10:50 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@prometheus.a.service: Consumed 14.615s CPU time. 2024-09-21T11:10:50.567 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T11:10:50.568 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-21T11:10:50.568 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 --force --keep-logs 2024-09-21T11:10:50.722 INFO:teuthology.orchestra.run.smithi070.stdout:Deleting cluster with fsid: 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T11:10:52.377 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 11:10:52 smithi070 systemd[1]: Stopping Ceph alertmanager.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T11:10:52.775 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 11:10:52 smithi070 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a[102272]: ts=2024-09-21T11:10:52.475Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-21T11:10:52.775 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 11:10:52 smithi070 podman[448335]: 2024-09-21 11:10:52.516217229 +0000 UTC m=+0.185960069 container died 89ae95182419b722e9c485b51402bf6e6d9ebfa1c9224996f6e5a432fdb6bcc5 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T11:10:53.127 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 11:10:52 smithi070 podman[448335]: 2024-09-21 11:10:52.964206552 +0000 UTC m=+0.633949386 container cleanup 89ae95182419b722e9c485b51402bf6e6d9ebfa1c9224996f6e5a432fdb6bcc5 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T11:10:53.127 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 11:10:52 smithi070 bash[448335]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a 2024-09-21T11:10:53.627 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 11:10:53 smithi070 podman[448353]: 2024-09-21 11:10:53.240519241 +0000 UTC m=+0.721736963 container remove 89ae95182419b722e9c485b51402bf6e6d9ebfa1c9224996f6e5a432fdb6bcc5 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T11:10:53.627 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 11:10:53 smithi070 podman[448353]: 2024-09-21 11:10:53.316349061 +0000 UTC m=+0.797566738 volume remove 901253e1476e56a94630dcca7c87830472087b89ed5d8cce13b6869131348e87 2024-09-21T11:10:53.627 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 11:10:53 smithi070 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-21T11:10:53.627 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 11:10:53 smithi070 systemd[1]: Stopped Ceph alertmanager.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T11:10:53.628 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Sep 21 11:10:53 smithi070 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@alertmanager.a.service: Consumed 5.684s CPU time. 2024-09-21T11:10:53.908 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 11:10:53 smithi070 systemd[1]: Stopping Ceph node-exporter.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T11:10:54.178 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 11:10:53 smithi070 podman[448466]: 2024-09-21 11:10:53.906741557 +0000 UTC m=+0.115297529 container died 5d5d4b026070e1daa98aaabd67bb75cb9b6117c727068865f51afce896f70b8c (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T11:10:54.627 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 11:10:54 smithi070 podman[448466]: 2024-09-21 11:10:54.36049366 +0000 UTC m=+0.569049673 container cleanup 5d5d4b026070e1daa98aaabd67bb75cb9b6117c727068865f51afce896f70b8c (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T11:10:54.627 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 11:10:54 smithi070 bash[448466]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a 2024-09-21T11:10:54.956 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 11:10:54 smithi070 podman[448483]: 2024-09-21 11:10:54.628430351 +0000 UTC m=+0.725553214 container remove 5d5d4b026070e1daa98aaabd67bb75cb9b6117c727068865f51afce896f70b8c (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T11:10:54.957 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 11:10:54 smithi070 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-21T11:10:54.957 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 11:10:54 smithi070 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-21T11:10:54.957 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 11:10:54 smithi070 systemd[1]: Stopped Ceph node-exporter.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T11:10:54.957 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Sep 21 11:10:54 smithi070 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@node-exporter.a.service: Consumed 11.604s CPU time. 2024-09-21T11:11:03.354 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 --force --keep-logs 2024-09-21T11:11:03.501 INFO:teuthology.orchestra.run.smithi160.stdout:Deleting cluster with fsid: 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T11:11:05.341 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 11:11:04 smithi160 systemd[1]: Stopping Ceph node-exporter.b for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T11:11:05.341 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 11:11:05 smithi160 podman[262309]: 2024-09-21 11:11:05.088938423 +0000 UTC m=+0.104000791 container died 19b8e0e8ee0c3481a50904ebdbdd7cc113b8158dc83b644d97b69573f5338ede (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T11:11:05.736 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 11:11:05 smithi160 podman[262309]: 2024-09-21 11:11:05.459080907 +0000 UTC m=+0.474143267 container cleanup 19b8e0e8ee0c3481a50904ebdbdd7cc113b8158dc83b644d97b69573f5338ede (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T11:11:05.736 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 11:11:05 smithi160 bash[262309]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b 2024-09-21T11:11:06.047 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 11:11:05 smithi160 podman[262326]: 2024-09-21 11:11:05.735267745 +0000 UTC m=+0.646147392 container remove 19b8e0e8ee0c3481a50904ebdbdd7cc113b8158dc83b644d97b69573f5338ede (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T11:11:06.047 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 11:11:05 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-21T11:11:06.047 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 11:11:05 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-21T11:11:06.047 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 11:11:05 smithi160 systemd[1]: Stopped Ceph node-exporter.b for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T11:11:06.047 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 21 11:11:05 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@node-exporter.b.service: Consumed 11.100s CPU time. 2024-09-21T11:11:06.313 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 11:11:06 smithi160 systemd[1]: Stopping Ceph grafana.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490... 2024-09-21T11:11:06.313 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 11:11:06 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=server t=2024-09-21T11:11:06.29138847Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-09-21T11:11:06.314 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 11:11:06 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=tracing t=2024-09-21T11:11:06.291759934Z level=info msg="Closing tracing" 2024-09-21T11:11:06.314 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 11:11:06 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=ticker t=2024-09-21T11:11:06.291776406Z level=info msg=stopped last_tick=2024-09-21T11:11:00Z 2024-09-21T11:11:06.314 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 11:11:06 smithi160 ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a[92987]: logger=grafana-apiserver t=2024-09-21T11:11:06.291837863Z level=info msg="StorageObjectCountTracker pruner is exiting" 2024-09-21T11:11:06.314 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 11:11:06 smithi160 podman[262440]: 2024-09-21 11:11:06.312041132 +0000 UTC m=+0.128701232 container died daa2d8dfab48af09378046f2da6d57eaf11d749ac73ce79ab86f59b17549b177 (image=quay.io/ceph/grafana:10.4.8, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-21T11:11:06.951 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 11:11:06 smithi160 podman[262440]: 2024-09-21 11:11:06.713589176 +0000 UTC m=+0.530249271 container cleanup daa2d8dfab48af09378046f2da6d57eaf11d749ac73ce79ab86f59b17549b177 (image=quay.io/ceph/grafana:10.4.8, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-21T11:11:06.952 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 11:11:06 smithi160 bash[262440]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a 2024-09-21T11:11:07.280 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 11:11:06 smithi160 podman[262458]: 2024-09-21 11:11:06.949885701 +0000 UTC m=+0.638128142 container remove daa2d8dfab48af09378046f2da6d57eaf11d749ac73ce79ab86f59b17549b177 (image=quay.io/ceph/grafana:10.4.8, name=ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-21T11:11:07.280 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 11:11:07 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@grafana.a.service: Deactivated successfully. 2024-09-21T11:11:07.280 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 11:11:07 smithi160 systemd[1]: Stopped Ceph grafana.a for 7c68bfbc-77fe-11ef-baf6-efdc52797490. 2024-09-21T11:11:07.281 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 21 11:11:07 smithi160 systemd[1]: ceph-7c68bfbc-77fe-11ef-baf6-efdc52797490@grafana.a.service: Consumed 49.186s CPU time. 2024-09-21T11:11:11.655 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T11:11:11.688 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T11:11:11.715 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-21T11:11:11.717 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/crash to /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913663/remote/smithi070/crash 2024-09-21T11:11:11.718 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/crash -- . 2024-09-21T11:11:11.770 INFO:teuthology.orchestra.run.smithi070.stderr:tar: /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T11:11:11.771 INFO:teuthology.orchestra.run.smithi070.stderr:tar: Error is not recoverable: exiting now 2024-09-21T11:11:11.773 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/crash to /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913663/remote/smithi160/crash 2024-09-21T11:11:11.774 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/crash -- . 2024-09-21T11:11:11.817 INFO:teuthology.orchestra.run.smithi160.stderr:tar: /var/lib/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T11:11:11.817 INFO:teuthology.orchestra.run.smithi160.stderr:tar: Error is not recoverable: exiting now 2024-09-21T11:11:11.818 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-21T11:11:11.819 DEBUG:teuthology.orchestra.run.smithi070:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v MON_DOWN | egrep -v 'out of quorum' | egrep -v PG_AVAILABILITY | head -n 1 2024-09-21T11:11:11.907 INFO:teuthology.orchestra.run.smithi070.stdout:2024-09-21T09:53:43.186423+0000 mon.a (mon.0) 693 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T11:11:11.912 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-09-21T11:11:11.912 INFO:tasks.cephadm:Compressing logs... 2024-09-21T11:11:11.912 DEBUG:teuthology.orchestra.run.smithi070:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T11:11:11.915 DEBUG:teuthology.orchestra.run.smithi160:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T11:11:11.952 INFO:teuthology.orchestra.run.smithi160.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-21T11:11:11.952 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-osd.7.log 2024-09-21T11:11:11.952 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-volume.log 2024-09-21T11:11:11.953 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose --/var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-osd.7.log: /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-osd.4.log 2024-09-21T11:11:11.954 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph.audit.log 2024-09-21T11:11:11.954 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph.cephadm.log 2024-09-21T11:11:11.954 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-osd.6.log 2024-09-21T11:11:11.955 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-osd.5.log 2024-09-21T11:11:11.955 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose/var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-osd.6.log: -- /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph.log 2024-09-21T11:11:11.956 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-mgr.x.log 2024-09-21T11:11:11.956 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph.log: gzip -5 --verbose -- /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-mon.b.log 2024-09-21T11:11:11.956 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-mds.foo.smithi160.byyhfz.log 2024-09-21T11:11:11.956 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-mgr.x.log: /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T11:11:11.956 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-mds.foo.smithi160.byyhfz.log: /var/log/ceph/cephadm.log: 83.8% -- replaced with /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-mds.foo.smithi160.byyhfz.log.gz 2024-09-21T11:11:11.976 INFO:teuthology.orchestra.run.smithi070.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-21T11:11:11.976 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-osd.0.log 2024-09-21T11:11:11.978 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-osd.1.log 2024-09-21T11:11:11.978 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose/var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-osd.0.log: -- /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-volume.log 2024-09-21T11:11:11.978 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph.audit.log 2024-09-21T11:11:11.978 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-mon.c.log 2024-09-21T11:11:11.978 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph.cephadm.log 2024-09-21T11:11:11.978 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-mon.a.log 2024-09-21T11:11:11.980 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-osd.2.log 2024-09-21T11:11:11.981 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-mgr.y.log 2024-09-21T11:11:11.981 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph.log 2024-09-21T11:11:11.981 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-osd.3.log 2024-09-21T11:11:11.981 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph.log: gzip -5 --verbose -- /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-mds.foo.smithi070.gansda.log 2024-09-21T11:11:11.981 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T11:11:11.981 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-mds.foo.smithi070.gansda.log: 86.2% -- replaced with /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-mds.foo.smithi070.gansda.log.gz 2024-09-21T11:11:11.996 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/cephadm.log: 92.8% -- replaced with /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph.log.gz 2024-09-21T11:11:12.136 INFO:teuthology.orchestra.run.smithi070.stderr: 92.0% -- replaced with /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph.cephadm.log.gz 2024-09-21T11:11:12.161 INFO:teuthology.orchestra.run.smithi160.stderr: 85.9% -- replaced with /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph.cephadm.log.gz 2024-09-21T11:11:12.269 INFO:teuthology.orchestra.run.smithi070.stderr: 94.4% -- replaced with /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph.audit.log.gz 2024-09-21T11:11:12.277 INFO:teuthology.orchestra.run.smithi070.stderr: 94.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T11:11:12.286 INFO:teuthology.orchestra.run.smithi160.stderr: 87.2% -- replaced with /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph.log.gz 2024-09-21T11:11:12.463 INFO:teuthology.orchestra.run.smithi070.stderr: 91.2% -- replaced with /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-volume.log.gz 2024-09-21T11:11:12.465 INFO:teuthology.orchestra.run.smithi160.stderr: 91.3% -- replaced with /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-volume.log.gz 2024-09-21T11:11:12.492 INFO:teuthology.orchestra.run.smithi160.stderr: 90.7% -- replaced with /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph.audit.log.gz 2024-09-21T11:11:12.646 INFO:teuthology.orchestra.run.smithi160.stderr: 90.3% -- replaced with /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-mgr.x.log.gz 2024-09-21T11:11:13.209 INFO:teuthology.orchestra.run.smithi160.stderr: 90.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T11:11:20.590 INFO:teuthology.orchestra.run.smithi070.stderr: 89.9% -- replaced with /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-mgr.y.log.gz 2024-09-21T11:11:48.877 INFO:teuthology.orchestra.run.smithi160.stderr: 92.5% -- replaced with /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-mon.b.log.gz 2024-09-21T11:12:07.244 INFO:teuthology.orchestra.run.smithi070.stderr: 91.9% -- replaced with /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-mon.c.log.gz 2024-09-21T11:12:37.302 INFO:teuthology.orchestra.run.smithi070.stderr: 94.4% -- replaced with /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-osd.2.log.gz 2024-09-21T11:12:38.532 INFO:teuthology.orchestra.run.smithi070.stderr: 90.6% -- replaced with /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-mon.a.log.gz 2024-09-21T11:17:15.887 INFO:teuthology.orchestra.run.smithi160.stderr: 93.9% -- replaced with /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-osd.5.log.gz 2024-09-21T11:23:04.680 INFO:teuthology.orchestra.run.smithi070.stderr: 93.1% -- replaced with /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-osd.0.log.gz 2024-09-21T11:31:03.993 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-osd.6.log.gz 2024-09-21T11:33:36.624 INFO:teuthology.orchestra.run.smithi070.stderr: 92.9% -- replaced with /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-osd.3.log.gz 2024-09-21T11:34:14.526 INFO:teuthology.orchestra.run.smithi070.stderr: 92.9% -- replaced with /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-osd.1.log.gz 2024-09-21T11:34:14.565 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-21T11:34:14.565 INFO:teuthology.orchestra.run.smithi070.stderr:real 23m2.636s 2024-09-21T11:34:14.565 INFO:teuthology.orchestra.run.smithi070.stderr:user 8m34.123s 2024-09-21T11:34:14.565 INFO:teuthology.orchestra.run.smithi070.stderr:sys 0m26.440s 2024-09-21T11:34:19.560 INFO:teuthology.orchestra.run.smithi160.stderr: 93.4% -- replaced with /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-osd.4.log.gz 2024-09-21T11:34:52.589 INFO:teuthology.orchestra.run.smithi160.stderr: 92.9% -- replaced with /var/log/ceph/7c68bfbc-77fe-11ef-baf6-efdc52797490/ceph-osd.7.log.gz 2024-09-21T11:34:52.635 INFO:teuthology.orchestra.run.smithi160.stderr: 2024-09-21T11:34:52.635 INFO:teuthology.orchestra.run.smithi160.stderr:real 23m40.705s 2024-09-21T11:34:52.635 INFO:teuthology.orchestra.run.smithi160.stderr:user 7m58.864s 2024-09-21T11:34:52.635 INFO:teuthology.orchestra.run.smithi160.stderr:sys 0m25.638s 2024-09-21T11:34:52.636 INFO:tasks.cephadm:Archiving logs... 2024-09-21T11:34:52.636 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913663/remote/smithi070/log 2024-09-21T11:34:52.637 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T11:36:09.148 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913663/remote/smithi160/log 2024-09-21T11:36:09.149 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T11:37:15.353 INFO:tasks.cephadm:Removing cluster... 2024-09-21T11:37:15.354 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 --force 2024-09-21T11:37:15.946 INFO:teuthology.orchestra.run.smithi070.stdout:Deleting cluster with fsid: 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T11:37:16.753 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7c68bfbc-77fe-11ef-baf6-efdc52797490 --force 2024-09-21T11:37:17.395 INFO:teuthology.orchestra.run.smithi160.stdout:Deleting cluster with fsid: 7c68bfbc-77fe-11ef-baf6-efdc52797490 2024-09-21T11:37:18.168 INFO:tasks.cephadm:Removing cephadm ... 2024-09-21T11:37:18.169 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T11:37:18.188 DEBUG:teuthology.orchestra.run.smithi160:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T11:37:18.213 INFO:tasks.cephadm:Teardown complete 2024-09-21T11:37:18.214 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-21T11:37:18.224 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-21T11:37:18.224 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T11:37:18.230 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T11:37:18.346 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-21T11:37:18.356 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-21T11:37:18.356 DEBUG:teuthology.orchestra.run.smithi070:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T11:37:18.359 DEBUG:teuthology.orchestra.run.smithi160:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T11:37:18.373 INFO:teuthology.orchestra.run.smithi070.stderr:bash: line 1: ntpq: command not found 2024-09-21T11:37:18.373 INFO:teuthology.orchestra.run.smithi160.stderr:bash: line 1: ntpq: command not found 2024-09-21T11:37:18.440 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T11:37:18.441 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2024-09-21T11:37:18.441 INFO:teuthology.orchestra.run.smithi070.stdout:^* hv01.front.sepia.ceph.com 3 6 377 3 -248us[ -246us] +/- 57ms 2024-09-21T11:37:18.441 INFO:teuthology.orchestra.run.smithi070.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 4 +257us[ +259us] +/- 52ms 2024-09-21T11:37:18.441 INFO:teuthology.orchestra.run.smithi070.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 0 +25us[ +25us] +/- 69ms 2024-09-21T11:37:18.441 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T11:37:18.443 INFO:teuthology.orchestra.run.smithi160.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T11:37:18.443 INFO:teuthology.orchestra.run.smithi160.stdout:=============================================================================== 2024-09-21T11:37:18.443 INFO:teuthology.orchestra.run.smithi160.stdout:^* hv01.front.sepia.ceph.com 3 6 377 8 -252us[ -248us] +/- 57ms 2024-09-21T11:37:18.443 INFO:teuthology.orchestra.run.smithi160.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 1 +263us[ +263us] +/- 52ms 2024-09-21T11:37:18.443 INFO:teuthology.orchestra.run.smithi160.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 77 +26us[ +61us] +/- 68ms 2024-09-21T11:37:18.444 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T11:37:18.464 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-21T11:37:18.473 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-21T11:37:18.474 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-21T11:37:18.502 DEBUG:teuthology.orchestra.run.smithi070:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T11:37:18.616 DEBUG:teuthology.orchestra.run.smithi160:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T11:37:18.809 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T11:37:18.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:37:18.839 DEBUG:teuthology.orchestra.run.smithi160:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T11:37:18.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:37:18.867 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-21T11:37:18.889 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-21T11:37:18.945 INFO:teuthology.task.internal:Duration was 6961.528140 seconds 2024-09-21T11:37:18.945 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-21T11:37:18.978 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-21T11:37:18.978 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T11:37:18.981 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T11:37:19.040 INFO:teuthology.orchestra.run.smithi160.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T11:37:19.056 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T11:37:19.425 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-21T11:37:19.426 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi070.front.sepia.ceph.com 2024-09-21T11:37:19.426 DEBUG:teuthology.orchestra.run.smithi070:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T11:37:19.455 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi160.front.sepia.ceph.com 2024-09-21T11:37:19.455 DEBUG:teuthology.orchestra.run.smithi160:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T11:37:19.487 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-21T11:37:19.487 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T11:37:19.497 DEBUG:teuthology.orchestra.run.smithi160:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T11:37:19.554 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-21T11:37:19.554 DEBUG:teuthology.orchestra.run.smithi070:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T11:37:19.568 DEBUG:teuthology.orchestra.run.smithi160:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T11:37:21.431 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-21T11:37:21.443 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-21T11:37:21.444 DEBUG:teuthology.orchestra.run.smithi070:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T11:37:21.475 DEBUG:teuthology.orchestra.run.smithi160:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T11:37:21.504 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-21T11:37:21.513 DEBUG:teuthology.orchestra.run.smithi070:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T11:37:21.517 DEBUG:teuthology.orchestra.run.smithi160:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T11:37:21.551 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = core 2024-09-21T11:37:21.594 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern = core 2024-09-21T11:37:21.629 DEBUG:teuthology.orchestra.run.smithi070:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T11:37:21.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:37:21.644 DEBUG:teuthology.orchestra.run.smithi160:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T11:37:21.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:37:21.660 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-21T11:37:21.670 INFO:teuthology.task.internal:Transferring archived files... 2024-09-21T11:37:21.671 DEBUG:teuthology.misc:Transferring archived files from smithi070:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913663/remote/smithi070 2024-09-21T11:37:21.671 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T11:37:21.754 DEBUG:teuthology.misc:Transferring archived files from smithi160:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913663/remote/smithi160 2024-09-21T11:37:21.754 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T11:37:21.798 INFO:teuthology.task.internal:Removing archive directory... 2024-09-21T11:37:21.798 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T11:37:21.801 DEBUG:teuthology.orchestra.run.smithi160:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T11:37:21.845 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-21T11:37:21.978 INFO:teuthology.task.internal:Not uploading archives. 2024-09-21T11:37:21.978 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-21T11:37:22.053 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-21T11:37:22.054 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T11:37:22.056 DEBUG:teuthology.orchestra.run.smithi160:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T11:37:22.072 INFO:teuthology.orchestra.run.smithi160.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 11:37 /home/ubuntu/cephtest 2024-09-21T11:37:22.081 INFO:teuthology.orchestra.run.smithi070.stdout: 262169 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 21 11:37 /home/ubuntu/cephtest 2024-09-21T11:37:22.081 INFO:teuthology.orchestra.run.smithi070.stdout: 417188 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 21 11:06 /home/ubuntu/cephtest/mnt.0 2024-09-21T11:37:22.081 INFO:teuthology.orchestra.run.smithi070.stdout: 417218 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 21 11:06 /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T11:37:22.081 INFO:teuthology.orchestra.run.smithi070.stdout: 426239 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 11:06 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-21T11:37:22.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:37:22.082 INFO:teuthology.orchestra.run.smithi070.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-21T11:37:22.083 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi070 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-21T11:37:22.089 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-21T11:37:22.099 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-21T11:37:22.129 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi070 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-21T11:37:22.132 INFO:teuthology.run:Summary data: description: upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} duration: 6961.528140306473 failure_reason: 'Command failed (workunit test cls/test_cls_rbd.sh) on smithi070 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=943fb8f39010442ea2d1cf42991d56a6 status: fail success: false 2024-09-21T11:37:22.133 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T11:37:22.220 INFO:teuthology.run:FAIL